GitLab Runners

Cluster operations — Reference for the administrators and operators who run Nautilus.

Runner managers (Kubernetes executor)

Most CI/CD jobs run through runner managers deployed in the gitlab namespace:

Deployment	Replicas	Job tags	Notes
`gitlab-runner-general`	8	none (untagged jobs)
`gitlab-runner-gitlab`	1	none (untagged jobs)
`gitlab-runner-sdsc`	1	none (untagged jobs)
`gitlab-runner-arm64`	1	`arm64`	runs on arm64 nodes
`gitlab-runner-dind`	1	`docker`, `dind`, `pinned`	Docker-in-Docker sidecar (`docker:28.5.2-dind`)
`gitlab-runner-arnoc`, `gitlab-runner-ue4`, `gitlab-runner-docker`	0		kept scaled to 0

Each manager registers itself when it starts (init container init-runner, gitlab-runner register). Jobs run as pods in the gitlab namespace on nodes labeled gitlab-worker=true (toleration nautilus.io/noceph), by default with 1 CPU and 50Gi requested and limits of 16 CPU and 50Gi, and share an S3 cache in bucket runners-cache on s3-west.

Keep the runners at the GitLab version (major and minor). To upgrade, set the image of both containers:

for d in general arm64 gitlab sdsc dind arnoc ue4 docker; do
  kubectl -n gitlab set image deploy/gitlab-runner-$d \
    init-runner=gitlab/gitlab-runner:<version> runner=gitlab/gitlab-runner:<version>
done

A manager that restarts aborts the jobs it is running, so check for running jobs first. The manager containers set requests (runner 100m CPU and 128Mi, init container 50m and 64Mi).

Restore GitLab Runner KubeVirt VMs (docker:dind, docker tag)

Minimal guide for the production GitLab runners: docker executor, docker:dind image, docker tag. The docker-tagged runners are KubeVirt VMs not Kubernetes Pods. The configuration for each VM is in a secret in the namespace. Duplicate and get a new runner token from Gitlab for new secret. Duplicate runner DataVolume before applying VM.

Running a Runner VM

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: runner2
  namespace: gitlab
spec:
  runStrategy: Always
  dataVolumeTemplates:
  - metadata:
      name: runner2-root
    spec:
      source:
        http:
          url: https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-amd64.img
      pvc:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 20Gi
        storageClassName: rook-ceph-block
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
                - key: topology.kubernetes.io/zone
                  operator: In
                  values:
                  - ucsd
                  - ucsd-nrp
                  - ucsd-sdsc
      accessCredentials:
      - sshPublicKey:
          propagationMethod:
            configDrive: {}
          source:
            secret:
              secretName: runners-pub-keys
      architecture: amd64
      domain:
        cpu:
          cores: 12
        devices:
          autoattachGraphicsDevice: true
          autoattachSerialConsole: true
          disks:
          - disk:
              bus: virtio
            name: harddrive
          - disk:
              bus: virtio
            name: rootdisk
            bootOrder: 1
          - disk:
              bus: virtio
            name: cloudinit
        machine:
          type: q35
        resources:
          limits:
            memory: 48Gi
            cpu: 12
          requests:
            memory: 48Gi
            cpu: 12
      volumes:
      - dataVolume:
          name: runner2-root
        name: rootdisk
      - emptyDisk:
          capacity: 300Gi
        name: harddrive
      - cloudInitConfigDrive:
          secretRef:
            name: runner2-userdata
        name: cloudinit

Verify

virtctl ssh ubuntu@runner1 -n gitlab -- sudo gitlab-runner status
virtctl ssh ubuntu@runner2 -n gitlab -- sudo gitlab-runner status
# Both: executor=docker, image=docker:dind, tag_list=["docker"], S3 cache in config.toml

SSH (after ssh-add your key or use password): virtctl ssh ubuntu@runner1 -n gitlab / virtctl ssh ubuntu@runner2 -n gitlab.

This work was supported in part by National Science Foundation (NSF) awards CNS-1730158, ACI-1540112, ACI-1541349, OAC-1826967, OAC-2112167, CNS-2100237, CNS-2120019.