This page contains SRE related topics for GitLab. For GitLab application-specific information, user documentation, and policy, please see mw:GitLab on mediawiki.org.
GitLab is reachable at https://gitlab.wikimedia.org/. We run multiple instances of GitLab:
- gitlab2002 runs production GitLab serving https://gitlab.wikimedia.org/
- gitlab1004 runs a passive GitLab replica serving https://gitlab-replica.wikimedia.org/
- gitlab1003 runs a passive GitLab replica serving https://gitlab-replica-old.wikimedia.org/
- gitlab-prod-1002.devtools.eqiad1.wikimedia.cloud runs a production-like test instance in WMCS/VPS serving https://gitlab.devtools.wmcloud.org/
Import code to GitLab
See instruction in MediaWiki.
gitlab1003, gitlab1004, gitlab2002 and test instance gitlab-prod-1001 are setup using puppet. The configuration currently lives in profile::gitlab. Former configuration from gitlab-ansible was migrated completely to puppet (see T283076). GitLab is installed as a Omnibus installation on all instances. So all GitLab components are installed using the official packages and are executed on a single host. The reasons for this setup can be found in the Initialization docs in Mediawiki.
GitLab login is implemented with SSO using the CAS/SSO. So users will be redirected to idp.wikimedia.org (idp.wmcloud.org on WMCS/VPS) to login to the SSO portal. Authentication is currently open to all users with a Wikimedia developer account for the production instance. Access to the replica and test instance is restricted to WMF/NDA groups.
Configuring and customizing the GitLab instance
GitLabs configuration can be adjusted in multiple places and can be separated in instance configuration (needs a service restart) and application configuration (can be changed during runtime in UI/API).
The GitLab instance configuration is stored in
/etc/gitlab/gitlab.rb and contains options like the URL, Prometheus options and OIDC settings. This file is managed by puppet: gitlab.rb.erb. Changes to this file require a service restart. See also the upstream template https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template for all possible configuration values.
Application configuration can be changed in the application settings API and appearance API (and the admin UI, but that is discouraged). This settings contain limits for projects, quotas, admin settings, logos and welcome texts. The WMF GitLab instances application configuration is managed by a set of Python scripts in repos/releng/gitlab-settings. The project contains individual scripts to adjust the application settings, the appearance or the project setup. Most scripts have a
update command to properly adjust all settings for each GitLab host.
WIP. The gitlab-settings script is executed manually when configuration changes are needed. There are efforts to run the scripts with a timer on the GitLab host.
GitLab offers CI/CD capabilities. For our current Runner documentation, see GitLab/Gitlab Runner.
GitLab hosts (production and replica) are running in eqiad/codfw datacenter. Each hosts have two public IPv4 and two public IPv6 addresses. The first address pair is used for general access to the machine (mostly ssh and system monitoring). The second address pair is needed to run a dedicated SSH daemon (for git pull and push) independent from the existing management SSH daemon. Furthermore the second address is used to serve https traffic without LVS. This ensures that GitLab (thus deployments and rollbacks) is independent from LVS. The second address pair is called "service address" or
service_ip sometimes in puppet/docs.
The primary address is not required to be public, but due to networking conventions one private and one public IP/VLAN is not encouraged currently (see T310265).
GitLab Runner machines have a single, private IPv4 and IPv6 address only. Some (Trusted) Runners are running in eqiad/codfw datacenter, other Runners are in WMCS
See Help:SSH_Fingerprints/gitlab.wikimedia.org for an overview of all fingerprints at once.
Each gitlab server has 4 IPs on the same network interface (see above). One IPv4 and one IPv6 is used for the standard sshd that admins use to connect to the individual backend (gitlab1001.wikimedia.org/gitlab2001.wikimedia.org) and one IPv4 and IPv6 is used for the service address (gitlab.wikimedia.org).
If you connect to the service as a user you _should_ expect to see the one for the service IP but currently you will see the one for the backend you are connecting to. Currently this is gitlab1004 but it could change when we switch data centers or fail over.
We are looking into getting a new configuration option into gitlab upstream to properly fix this. Meanwhile you can find fingerprints linked on Help:SSH_Fingerprints/gitlab.wikimedia.org.
also see the status of this ticket: phab:T296944
How to create or migrate a repo / group / project
See mw:GitLab/Hosting a project on GitLab for full user documentation.
Bootstrap a new GitLab instance
To setup a new GitLab instance from scratch the following steps are needed. To setup a new GitLab test instance, use GitLab/Test_Instance.
- Make sure to have a properly sized host and four SSDs (two for root partition and two for backups)
- Request a second public address mapped to the host (for https and git ssh)
- Set hiera keys
profile::gitlab::service_ip_v6with new public address
profile::gitlab::service_namewith desired service name
- add new host either to
- Add A, AAA and PTR records to DNS repo mapping to new address
- Assign puppet
role::gitlabto new instance and wait for puppet run/force puppet run on new instance
- Configure backup partitions and raid by running
- This script configures the second raid
md1on the third and forth disk. Make sure to double check disk names used in the script (
/dev/sdd) with actual disk names. Disk names may change randomly during reimages.
- If the disk names don't align, try reimage the host a second time
- This script configures the second raid
- apply gitlab-settings
- optional: trigger restore or failover
- optional: register runners