GitLab/Gitlab Runner

From Wikitech

GitLab Runner is an application that works with GitLab to run CI/CD jobs in a pipeline.[1] For more information see the official GitLab Runner documentation. For information on how GitLab runners are used in production deployments, see Deploying services to production.

GitLab Runner types

Our GitLab instance offers different types of CI Runners:

Shared GitLab Runners
General purpose CI workers. These Runners execute jobs for a wide range of projects inside the /repos group in GitLab. If access to this kind of Runner is needed, consider moving your git repository to the /repos group and make yourself familiar with the details under Shared GitLab Runners. Shared Runners use the tag wmcs but also accept untagged jobs.
Trusted GitLab Runners
These offer a platform for CI jobs with additional security needs (like building production artifacts). These Runners run in WMF production infrastructure and access to these Runners is gated and restricted. Access has to be requested on per-project basis, so please take a look on Trusted GitLab Runners on how to get access. Trusted Runners use the tag trusted and do not accept untagged jobs.
Cloud Runners
Instance-wide CI workers. They are running on a managed Kubernetes in Digital Ocean. They are available instance wide for every project. Cloud Runners use the tag cloud (and kubernetes) but accept untagged jobs as well.
Memory-optimized Cloud Runners
These behave just like regular Cloud runners but feature more RAM than their counterpart. These runners are useful for more intensive building/packaging jobs. To utilize these runners, add the memory-optimized tag.

Permission matrix

The following table explains which features and permissions are available for which type of runner.

Runner Type Trusted Runners Dockerfile Runners Shared Runner Cloud Runners
access level access has to be requested per project in projects.json access has to be requested per project in projects.json and is restricted to a small set of infrastructure/foundational container projects all projects in /repos have access all projects have access
allowed images only images from WMF registry**/* only images from WMF registry**/* images from WMF registry, popular upstream Dockerhub images (like debian, golang, node, python), GitLab upstream images, wmflabs images. See profile::gitlab::runner::allowed_images no restriction
allowed Container build frontends blubber frontend blubber and Dockerfile frontend blubber frontend blubber and Dockerfile frontend
publish images to wmf registry Yes (using credentials from jwt-authorizer) Yes (using credentials from jwt-authorizer) No No
allowed job sources Jobs from protected tags and branches only Jobs from protected tags and branches only Jobs from all branches or tags Jobs from all branches or tags
allowed WMF services GitLab, apt host, doc host, docker registry, Kubernetes staging (see profile::gitlab::runner::allowed_services) GitLab, apt host, doc host, docker registry, Kubernetes staging (see profile::gitlab::runner::allowed_services) GitLab, Zuul none
allowed external services Https services using the webproxy, Dockerhub is blocked Https services using the webproxy All public https services All public https services

Adding new runners (end-users)

For special use-cases new runners can be added to any project by end-users. The token needed to register the runner can be obtained under project Settings > CI/CD > Runners (expand). Use the button "New project runner" to add a runner. The registration token will be shown once, so store the token in a secure place. Usage of the registration token varies depending on the type of runner (Docker, VM, Kubernetes). See register a runner upstream documentation. Please note this runners are self-managed, so you have to make sure to install updates and stay compatible with the GitLab server version.

Adding new runners (administrators)

The process of adding Shared, Trusted or instance-wide runners is different from adding self-managed project runners. The tokens and registration are created programmatically to ensure all runners use the correct configuration (protection status, tags, description and untagged jobs). The tool used to create the tokens is The actual registration happens in Puppet or Terraform, depending on the Runner.

To generate new tokens, use the runners generate-tokens command. The script will walk you over every runner configured in runner-config.json. Please note the output of the script contains private tokens.

Example run:

./runners  generate-tokens

Please add the following tokens to private puppet.
The token should be added as profile::gitlab::runner::token in either private puppet or WMCS puppet master.
The token for cloud runners should be stored as TF_VAR_runner_token in
The tokens are secrets, do not store them anywhere else!

Do you want to create a new token for shared_runners [y/N]: n
Skipping shared_runners

Do you want to create a new token for trusted_runners [y/N]: y
trusted_runners token: <removed>

As mentioned in the script output, the tokens are then used in Puppet or GitLab CI deployments. So for Shared and Trusted Runners configured by puppet, the token has to added as profile::gitlab::runner::token to private puppet. Runners in Digital Ocean use Terraform for deployments, so the token is added as TF_VAR_runner_token in

New Runners should pick up a updated token with the next puppet run or Terraform deployment. Existing Runners have to be un-registered and re-registered manually by setting profile::gitlab::runner::ensure to absent, run puppet and setting it to present again. Also the script does not rotate old tokens, it just creates to tokens. So manual deletion of Runners in may be required.

Evaluation and Design

Evaluation sub-pages on the right menu offer more insights into the design and security considerations.