GitLab/Gitlab Runner
| GitLab Runner |
|---|
 |
|
GitLab Runner is an application that works with GitLab to run CI/CD jobs in a pipeline.[1] For more information see the official GitLab Runner documentation. For information on how GitLab runners are used in production deployments, see Deploying services to production.
GitLab Runner types
Our GitLab instance offers different types of CI Runners:
- Shared GitLab Runners
- General purpose CI workers. These Runners execute jobs for a wide range of projects inside the
/reposgroup in GitLab. If access to this kind of Runner is needed, consider moving your git repository to the/reposgroup and make yourself familiar with the details under Shared GitLab Runners. Shared Runners use the tagwmcsbut also accept untagged jobs.
- Trusted GitLab Runners
- These offer a platform for CI jobs with additional security needs (like building production artifacts). These Runners run in WMF production infrastructure and access to these Runners is gated and restricted. Access has to be requested on per-project basis, so please take a look on Trusted GitLab Runners on how to get access. Trusted Runners use the tag
trustedand do not accept untagged jobs.
- Cloud Runners
- Instance-wide CI workers. They are running on a managed Kubernetes in Digital Ocean. They are available instance wide for every project. Cloud Runners use the tag
cloud(andkubernetes) but accept untagged jobs as well.
- Memory-optimized Cloud Runners
- These behave just like regular Cloud runners but feature more RAM than their counterpart. These runners are useful for more intensive building/packaging jobs. To utilize these runners, add the
memory-optimizedtag. Regular runners are allocated 1GB.memory-optimizedrunners get 5.7GB.
Permission matrix
The following table explains which features and permissions are available for which type of runner.
| Runner Type | Trusted Runners | Dockerfile Runners | Shared Runner | Cloud Runners |
|---|---|---|---|---|
| access level | access has to be requested per project in projects.json | access has to be requested per project in projects.json and is restricted to a small set of infrastructure/foundational container projects | all projects in /repos have access | all projects have access |
| allowed images | only images from WMF registry docker-registry.wikimedia.org/**/*
|
only images from WMF registry docker-registry.wikimedia.org/**/*
|
images from WMF registry, popular upstream Dockerhub images (like debian, golang, node, python), GitLab upstream images, wmflabs images. See profile::gitlab::runner::allowed_images in operations/puppet/hieradata/cloud.yaml
|
no restriction |
| allowed Container build frontends | blubber frontend | blubber and Dockerfile frontend | blubber frontend | blubber and Dockerfile frontend |
| publish images to wmf registry | Yes (using credentials from jwt-authorizer) | Yes (using credentials from jwt-authorizer) | No | No |
| allowed job sources | Jobs from protected tags and branches only | Jobs from protected tags and branches only | Jobs from all branches or tags | Jobs from all branches or tags |
| allowed WMF services | GitLab, apt host, doc host, docker registry, Kubernetes staging (see profile::gitlab::runner::allowed_services)
|
GitLab, apt host, doc host, docker registry, Kubernetes staging (see profile::gitlab::runner::allowed_services)
|
GitLab, Zuul | none |
| allowed external services | Https services using the webproxy, Dockerhub is blocked | Https services using the webproxy | All public https services | All public https services |
Adding new runners (end-users)
For special use-cases new runners can be added to any project by end-users. The token needed to register the runner can be obtained under project Settings > CI/CD > Runners (expand). Use the button "New project runner" to add a runner. The registration token will be shown once, so store the token in a secure place. Usage of the registration token varies depending on the type of runner (Docker, VM, Kubernetes). See register a runner upstream documentation. Please note this runners are self-managed, so you have to make sure to install updates and stay compatible with the GitLab server version.
Adding new runners (administrators)
The process of adding Shared, Trusted or instance-wide runners is different from adding self-managed project runners. The tokens and registration are created programmatically to ensure all runners use the correct configuration (protection status, tags, description and untagged jobs). The tool used to create the tokens is https://gitlab.wikimedia.org/repos/releng/gitlab-settings. The actual registration happens in Puppet or Terraform, depending on the Runner.
To generate new tokens, use the runners generate-tokens command. The script will walk you over every runner configured in runner-config.json. Please note the output of the script contains private tokens.
Example run:
./runners generate-tokens
Please add the following tokens to private puppet.
The token should be added as profile::gitlab::runner::token in either private puppet or WMCS puppet master.
The token for cloud runners should be stored as TF_VAR_runner_token in https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/settings/ci_cd.
The tokens are secrets, do not store them anywhere else!
Do you want to create a new token for shared_runners [y/N]: n
Skipping shared_runners
Do you want to create a new token for trusted_runners [y/N]: y
trusted_runners token: <removed>
As mentioned in the script output, the tokens are then used in Puppet or GitLab CI deployments. So for Shared and Trusted Runners configured by puppet, the token has to added as profile::gitlab::runner::token to private puppet. Runners in Digital Ocean use Terraform for deployments, so the token is added as TF_VAR_runner_token in https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/settings/ci_cd.
New Runners should pick up a updated token with the next puppet run or Terraform deployment. Existing Runners have to be un-registered and re-registered manually by setting profile::gitlab::runner::ensure to absent, run puppet and setting it to present again. Also the script does not rotate old tokens, it just creates to tokens. So manual deletion of Runners in https://gitlab.wikimedia.org/admin/runners may be required.
Evaluation and Design
Evaluation sub-pages on the right menu offer more insights into the design and security considerations.