GitLab/Gitlab Runner/Trusted Runners

From Wikitech

Trusted GitLab Runners offer CI with additional security and trust requirements. In contrast to the Shared Runners, which run in WMCS, Trusted Runners live inside WMF infrastructure. With this approach, SRE team has full control over the instance and who has access. Furthermore customization, like scaling, other disks and NICs can be done outside of the bounds of WMCS. Beside that, Shared Runners and Trusted Runners use the same puppet code (role(gitlab_runner)) with slightly different hiera configuration.

The current Trusted Runner cluster consist of six physical machines:

  • gitlab-runner1002.eqiad.wmnet
  • gitlab-runner1003.eqiad.wmnet
  • gitlab-runner1004.eqiad.wmnet
  • gitlab-runner2002.codfw.wmnet
  • gitlab-runner2003.codfw.wmnet
  • gitlab-runner2004.codfw.wmnet

See also GitLab/Gitlab Runner/Security Evaluation for all of the security measures enabled for Trusted Runners.

Request access to Trusted Runners

Access to this Runners is gated and restricted. No project has access to Trusted Runners by default. Access has to be requested on project basis. Please use the following Phabricator task template to create a access request: Task Template

Please make sure to check your project settings and especially who has maintainer permissions. You also must protect your main branch. As described in the Security Evaluation maintainer permissions ("merge", "+2") are needed to execute jobs on the Trusted Runners. Reversely this also means everyone with maintainer permissions can execute such jobs.

Access to Trusted Runners is managed by https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner project. This is the root project where all Trusted Runners are registered. To request access, add your project to the projects.json file and open a merge request. Please add the id (go to Settings > General in your project), the name of the project and a reason why access is needed.

This merge request will get reviewed and and the project will be added to Trusted Runners. Adding and removing projects is automated by a python script which uses the projects.json .

Using Trusted Runners

To run a job on Trusted Runners, the project has to be authorized to use Trusted Runners (see above). Jobs can select runners based on tags. Currently Trusted Runners use the tag trusted. So all jobs which need Trusted Runners should specify this tag:

trusted-build-job:
  stage: trusted-build
  image: docker-registry.wikimedia.org/bullseye:20221127
  script:
    - echo "Compiling the code..."
    - echo "Compile complete."
  tags:
    - trusted # run job on Trusted Runners
  rules:
    - if: $CI_COMMIT_BRANCH == "main"

Please note Trusted Runners accept jobs from protected branches only. So make sure to protect the main branch and to trigger this job on protected branches only. Otherwise the jobs will get stuck.

Monitoring

Trusted Runners export Prometheus metrics. Dashboards are available in Grafana: gitlab-ci-overview and gitlab-runner-detail

Further Read

Related task: T295481