GitLab/Gitlab Runner

GitLab Runner

Cloud Runners Plattform Evaluation Security Evaluation Shared Runners Trusted Runners External resources: GitLab Runner Admin menu GitLab CI metrics

GitLab Runner is an application that works with GitLab to run CI/CD jobs in a pipeline.^[1] For more information see the official GitLab Runner documentation. For information on how GitLab runners are used in production deployments, see Deploying services to production.

GitLab Runner types

Our GitLab instance offers different types of CI Runners:

Shared GitLab Runners: General purpose CI workers. These Runners execute jobs for a wide range of projects inside the /repos group in GitLab. If access to this kind of Runner is needed, consider moving your git repository to the /repos group and make yourself familiar with the details under Shared GitLab Runners. Shared Runners use the tag wmcs but also accept untagged jobs.

Trusted GitLab Runners: These offer a platform for CI jobs with additional security needs (like building production artifacts). These Runners run in WMF production infrastructure and access to these Runners is gated and restricted. Access has to be requested on per-project basis, so please take a look on Trusted GitLab Runners on how to get access. Trusted Runners use the tag trusted and do not accept untagged jobs.

Cloud Runners: Instance-wide CI workers. They are running on a managed Kubernetes in Digital Ocean. They are available instance wide for every project. Cloud Runners use the tag cloud (and kubernetes) but accept untagged jobs as well.

Memory-optimized Cloud Runners: These behave just like regular Cloud runners but feature more RAM than their counterpart. These runners are useful for more intensive building/packaging jobs. To utilize these runners, add the memory-optimized tag.

Permission matrix

The following table explains which features and permissions are available for which type of runner.


Runner Type	Trusted Runners	Dockerfile Runners	Shared Runner	Cloud Runners
access level	access has to be requested per project in projects.json	access has to be requested per project in projects.json and is restricted to a small set of infrastructure/foundational container projects	all projects in /repos have access	all projects have access
allowed images	only images from WMF registry `docker-registry.wikimedia.org/*/`	only images from WMF registry `docker-registry.wikimedia.org/*/`	images from WMF registry, popular upstream Dockerhub images (like debian, golang, node, python), GitLab upstream images, wmflabs images. See `profile::gitlab::runner::allowed_images`	no restriction
allowed Container build frontends	blubber frontend	blubber and Dockerfile frontend	blubber frontend	blubber and Dockerfile frontend
publish images to wmf registry	Yes (using credentials from jwt-authorizer)	Yes (using credentials from jwt-authorizer)	No	No
allowed job sources	Jobs from protected tags and branches only	Jobs from protected tags and branches only	Jobs from all branches or tags	Jobs from all branches or tags
allowed WMF services	GitLab, apt host, doc host, docker registry, Kubernetes staging (see `profile::gitlab::runner::allowed_services`)	GitLab, apt host, doc host, docker registry, Kubernetes staging (see `profile::gitlab::runner::allowed_services`)	GitLab, Zuul	none
allowed external services	Https services using the webproxy, Dockerhub is blocked	Https services using the webproxy	All public https services	All public https services

Adding new runners (end-users)

For special use-cases new runners can be added to any project by end-users. The token needed to register the runner can be obtained under project Settings > CI/CD > Runners (expand). Use the button "New project runner" to add a runner. The registration token will be shown once, so store the token in a secure place. Usage of the registration token varies depending on the type of runner (Docker, VM, Kubernetes). See register a runner upstream documentation. Please note this runners are self-managed, so you have to make sure to install updates and stay compatible with the GitLab server version.

Adding new runners (administrators)

This process is new since GitLab 16, see https://phabricator.wikimedia.org/T344951 and https://docs.gitlab.com/ee/ci/runners/new_creation_workflow.html

The process of adding Shared, Trusted or instance-wide runners is different from adding self-managed project runners. The tokens and registration are created programmatically to ensure all runners use the correct configuration (protection status, tags, description and untagged jobs). The tool used to create the tokens is https://gitlab.wikimedia.org/repos/releng/gitlab-settings. The actual registration happens in Puppet or Terraform, depending on the Runner.

To generate new tokens, use the runners generate-tokens command. The script will walk you over every runner configured in runner-config.json. Please note the output of the script contains private tokens.

Example run:

./runners  generate-tokens

Please add the following tokens to private puppet.
The token should be added as profile::gitlab::runner::token in either private puppet or WMCS puppet master.
The token for cloud runners should be stored as TF_VAR_runner_token in https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/settings/ci_cd.
The tokens are secrets, do not store them anywhere else!

Do you want to create a new token for shared_runners [y/N]: n
Skipping shared_runners

Do you want to create a new token for trusted_runners [y/N]: y
trusted_runners token: <removed>

As mentioned in the script output, the tokens are then used in Puppet or GitLab CI deployments. So for Shared and Trusted Runners configured by puppet, the token has to added as profile::gitlab::runner::token to private puppet. Runners in Digital Ocean use Terraform for deployments, so the token is added as TF_VAR_runner_token in https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/settings/ci_cd.

New Runners should pick up a updated token with the next puppet run or Terraform deployment. Existing Runners have to be un-registered and re-registered manually by setting profile::gitlab::runner::ensure to absent, run puppet and setting it to present again. Also the script does not rotate old tokens, it just creates to tokens. So manual deletion of Runners in https://gitlab.wikimedia.org/admin/runners may be required.

Evaluation and Design

Evaluation sub-pages on the right menu offer more insights into the design and security considerations.

↑ https://docs.gitlab.com/runner/