GitLab

How GitLab runs

GitLab runs the official Enterprise Edition image gitlab/gitlab-ee as the gitlab Deployment in the gitlab namespace. Its license is under Admin > Subscription. Keep the gitlab-ee image on every upgrade: a gitlab-ce tag turns the instance into Community Edition and drops the licensed features.

• Settings: ConfigMap gitlab-config, key base.rb, loaded through GITLAB_OMNIBUS_CONFIG. Secrets reach it as environment variables from Kubernetes Secrets, read in base.rb with ENV[...]; never write a secret into the ConfigMap.
• Data: PVC gitlab, mounted at /home/git/data (repositories, shared, uploads, builds, backups). The files belong to uid/gid 1000, so the container command deletes the image’s ubuntu user (uid 1000) before init and base.rb sets user['uid'] and user['gid'] to 1000.
• State: PVC gitlab-state holds /etc/gitlab (including gitlab-secrets.json) and /var/opt/gitlab (Redis, Gitaly runtime). gitlab-secrets.json holds the keys for data GitLab encrypts; a copy is in Secret gitlab-secrets-json, to refresh whenever the file changes.
• SSH host keys: Secret gitlab-ssh-host-keys, copied into /etc/gitlab by the host-keys init container. Clients connect to gitlab-ssh.nrp-nautilus.io:30622 (Service gitlab-ssh).
• Database: the Postgres cluster gitlab-db (Zalando operator). Registry: the separate gitlab-reg Deployment; GitLab signs its tokens with the key in Secret ssl-key. Pages: port 8090 behind the *.pages.nrp-nautilus.io ingress.
• Placement: region us-west, like gitlab-db and gitlab-reg, preferring the SDSC nodes. From other regions the volume reads several times slower and Gitaly calls time out.
• Resources: every container in the namespace sets requests; limits are optional. GitLab requests 8 CPUs and 8Gi with a 32Gi memory limit.

To change a setting, edit base.rb in ConfigMap gitlab-config, then restart with kubectl -n gitlab rollout restart deploy/gitlab (about 5 minutes of downtime). Check the result in the rendered /var/opt/gitlab/gitlab-rails/etc/gitlab.yml inside the pod.

Upgrading GitLab

1. Pick the next version on the upgrade path and read its upgrade notes. Never skip a required stop. Before moving past one, wait until Admin > Monitoring > Background migrations shows nothing pending.

2. If the upgrade could need a rollback, copy the database first, while GitLab is stopped:

   kubectl -n gitlab scale deploy/gitlab --replicas=0
   kubectl -n gitlab exec gitlab-db-0 -c postgres -- psql -U postgres \
     -c "create database gitlab_before_upgrade template gitlab_production owner gitlab"

   Drop the copy once the upgrade is confirmed.

3. Set the new tag on both containers; the Deployment restarts the pod (strategy Recreate), which runs gitlab-ctl reconfigure and the database migrations before starting:

   kubectl -n gitlab set image deploy/gitlab gitlab=gitlab/gitlab-ee:<version>-ee.0 host-keys=gitlab/gitlab-ee:<version>-ee.0
   kubectl -n gitlab logs -f deploy/gitlab -c gitlab   # wait for "gitlab Reconfigured!"

4. Check the sign-in page, a git clone over SSH, a Pages site, a registry login and the agent endpoint (see below).

5. Upgrade the registry to the version that release ships: it is set in config/software/registry.rb of omnibus-gitlab at the same tag (19.1.9 ships v4.40.1). The registry stores images in S3 without a metadata database, so the update is a rolling image change:

   kubectl -n gitlab set image deploy/gitlab-reg registry=registry.gitlab.com/gitlab-org/build/cng/gitlab-container-registry:v<version>-gitlab

Use version tags in the manifests, not digests.

Backups

The image has no scheduler, so the container command starts busybox crond with one job: gitlab-backup create at 04:00 Pacific, skipping ci_secure_files. The tarball (about 250 GB) is uploaded to the Google Cloud Storage bucket nautilus-gitlab, and local copies older than two days are deleted. The job writes to /var/log/gitlab/backup.log in the pod. gitlab-secrets.json and the SSH host keys are not part of the tarball; restoring needs them too.

Kubernetes integration and agents

The GitLab agent server (KAS) runs in the GitLab pod (gitlab_kas['enable'] = true in base.rb). Agents connect to wss://gitlab.nrp-nautilus.io/-/kubernetes-agent/ and CI/CD jobs reach them through https://gitlab.nrp-nautilus.io/-/kubernetes-agent/k8s-proxy/; the pod’s nginx serves both paths, and the ingress carries the WebSockets. KAS has the same version as GitLab, so it moves with each upgrade; agents should be updated to the same version. Its keys are in gitlab-secrets.json.

Users install the agent in their own namespace with namespace-scoped rights (the chart’s default cluster-admin binding is turned off), as described in K8s GitLab Integration. The older certificate-based integration stays available through the certificate_based_clusters feature flag.

To check KAS: gitlab-ctl status gitlab-kas in the pod, and a plain request to the agent endpoint, which answers 426 Upgrade Required when the path works:

curl -s -o /dev/null -w '%{http_code}\n' https://gitlab.nrp-nautilus.io/-/kubernetes-agent/