GitLab/Gitlab Runner/Trusted Runners

Trusted GitLab Runners offer CI with additional security and trust requirements. In contrast to the Shared Runners, which run in WMCS, Trusted Runners live inside WMF infrastructure. With this approach, SRE team has full control over the instance and who has access. Furthermore customization, like scaling, other disks and NICs can be done outside of the bounds of WMCS. Beside that, Shared Runners and Trusted Runners use the same puppet code (role(gitlab_runner)) with slightly different hiera configuration.

The current Trusted Runner cluster consist of six physical machines:

gitlab-runner1002.eqiad.wmnet
gitlab-runner1003.eqiad.wmnet
gitlab-runner1004.eqiad.wmnet
gitlab-runner2002.codfw.wmnet
gitlab-runner2003.codfw.wmnet
gitlab-runner2004.codfw.wmnet

See also GitLab/Gitlab Runner/Security Evaluation for all of the security measures enabled for Trusted Runners.

Request access to Trusted Runners

Access to this Runners is gated and restricted. No project has access to Trusted Runners by default. Access has to be requested on project basis. Please use the following Phabricator task template to create a access request: Task Template

Please make sure to check your project settings and especially who has maintainer permissions. You also must protect your main branch. As described in the Security Evaluation maintainer permissions ("merge", "+2") are needed to execute jobs on the Trusted Runners. This also means everyone with maintainer permissions can execute such jobs.

Access to Trusted Runners is managed by https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner project. This is the root project where all Trusted Runners are registered. To request access, add your project to the projects.json file and open a merge request. Please add the full project path (the portion after https://gitlab.wikimedia.org/) and the reason why access is needed.

This merge request will get reviewed and the project will be added to Trusted Runners. Adding and removing projects is automated by a python script which uses the projects.json .

Using Trusted Runners

To run a job on Trusted Runners:

The project has to be authorized to use Trusted Runners (see above).
The job must specify the trusted tag.
The job must be for a protected branch or tag. Otherwise the job will get stuck.

For example:

trusted-build-job:
  stage: trusted-build
  image: docker-registry.wikimedia.org/bullseye:20221127
  script:
    - echo "Compiling the code..."
    - echo "Compile complete."
  tags:
    - trusted # run job on Trusted Runners
  rules:
    - if: $CI_COMMIT_BRANCH == "main"

For this job to successfully run on Trusted Runners, the "main" branch must be protected.

Troubleshooting

A common pattern is to build and publish an image when a protected tag is created. This is usually expressed like the following:

trusted-build-job:
 ...
 tags: 
   - trusted 
 rules: 
   - if: $CI_COMMIT_TAG && $CI_COMMIT_REF_PROTECTED

If you find that your job is getting stuck, make sure that tags are protected in your project configuration. The easiest approach is to protect all tags by using the "*" wildcard. Note that protecting a tag after a job is created for it won't unstick the stuck job. You will need to cancel the job and create a new tag.

Dockerfile support

Support for container builds based on Dockerfiles was added in T357612. Projects in projects.json can be defined with "dockerfile": true to get added to Trusted Runner with Dockerfile support. However this is available for a small group of special-purpose projects only (base images, spacial CI images or infrastructure images). dockerfile defaults to false as the majority of images should be build with the secure blubber build frontend.

Monitoring

Trusted Runners export Prometheus metrics. Dashboards are available in Grafana: gitlab-ci-overview and gitlab-runner-detail

Further Read

Related task: T295481