GitLab is an awesome mess

I’ve been managing my company’s GitLab instance for about a month or two, both on the front and on the server side. The instance is a Premium one, not so big (around 300 users with 1500-ish active projects), and is installed via Omnibus package (the easiest method available). So, here is some of my thoughts about it.

The good §

Beside the sluggish frontend, the gigantic amount of bloated features I’d rather implement myself or never touch (e.g. Terraform managed state backend, AutoDevOps, SAST, K8s agent, Pages, Container scanning), GitLab actually does lots of things right.

Firstly, I like its micro-component architecture. You have Gitaly as the middle-man for Git storage, Gitlab-Shell specifically to manage SSH access, and so on. It allows each part to horizontally scale independently of the other, which is always a plus. Also, Geo , for me personally with a month of experimenting, is a great Disaster Recovery implementation. In comparison, Gitea, at the time of writing, straightforwardly doesn’t scale, at all.

Then, there is the famous CI/CD to talk about. Putting a few misleading predefined CI/CD variables, and occasional weird YAML merging behaviors aside, the .gitlab-ci.yml schema is pretty clean, well-defined and flexible. My only complaint, as of now, is that it’s still a YAML document. default:, include: and !reference[] exist, but, at the end of the day, no YAML anchors or fancy features inside the YAML engine can save you from a big, ugly CI/CD configuration¹. GitLab Runner is, in my opinion, also superior in its feature set compared to GitHub or Drone’s ones.

The bad §

Paid features §

Just look at the feature comparison table , it’s obvious that the Free tier comes with so many bloated, unnecessary features, while lacking what I deem fundamental ones. More features are being locked down in Premium and Ultimate tiers as days go by, while the price for them increase every single year. Considering the terms of their Buyer Based Tiering model , I guess it somehow makes sense. I still wish to have CODEOWNERS, Merge Train, Geo and Audit Log in Free tier though.

Shipping unfinished features on new releases §

A great example of this is Batched background migrations , enabled in version 13.12. You have to wait for all the batched migrations to complete, before moving on to the next upgrade. There wouldn’t be a story to tell if things went that smoothly. The CI/CD table conversion job fails easily on large instances (mine already did, and you can find countless people complaining about it on the Internet). So, how do you suppose to fix the failed batched migrations? There was no Retry button on the WebUI until version 14.3, and the gitlab-rake command to manually rerun the batched migrations didn’t exist until 14.1.

Another example, just recently, is the introduction of admin_mode scope for Personal Access Token in version 15.8. All admin tokens have this scope injected to them automatically, while there are no way to set or unset it through GitLab API. What it means is that all my Pulumi automation code to manage admin access tokens is now broken, because Pulumi will try to recreate the tokens every time it runs, due to the mismatched scopes in state. How nice!

Tokens §

Since I mentioned tokens, let’s take a look at all the different ways for your automations to access private GitLab resources².

• CI_JOB_TOKEN is the unique one and is nicely implemented, security wise, but its fixed scopes are so limited it turns out to be useless most of the time³.
• I wonder why Deploy Keys even exists. It doesn’t enforce expiration date, so would quickly become stale and unmanageable. And I find people usually use it the same way as Deploy Tokens .
• Personal Access Token , Project Access Token and Group Access Token are a mess. They are powerful, but can quickly be forgotten once created, as the usual case is to generate one and set it in a CI/CD variable. With the future 16.0 release, all your old, poorly maintained pipelines might suddenly break out of nowhere as their lost tokens expire.
• To make things worse, instance-level access token isn’t a thing⁴, so you get stuck with bot-like accounts for cross top-level groups automations, which occupy license seats and waste $19 $29 USD/month each on your Premium plan⁵.

Afterword §

Beside technical knowledge, these 2 months of hands-on experience with GitLab also gave me some good memories. All those times pulling all-nighters with co-workers to upgrade GitLab, gossiping about random things while waiting for the pre-upgrade backup to finish, it was really fun. I, overall, enjoy being a GitLab instance administrator (who doesn’t like almighty power over all the company’s projects, to be honest 😄).

That’s all for me. Thanks for reading until the end!