Jump to content

GitLab/Failover

From Wikitech
If there is sufficient time before the maintenance add it to next week's Tech News

GitLab has a active host and one or more replicas. The replicas are cold-standby currently, meaning they don't serve any production traffic and hold up to 24h old data. For maintenance or in case of emergency it is possible to failover the active host to a replica. This page describes the process broadly.

This process is dangerous and can cause data loss!
This is a double documentation that address two states of our Gitlab instances, while https://phabricator.wikimedia.org/T425441 is still open.

The process takes around 1h to 2h (depending on backup size). During that time GitLab is not available.

Prerequisites

The host to failover to should be a proper GitLab replica, meaning:

  • has a second IPv4 and IPv6 address configured as profile::gitlab::service_ip_v4 and profile::gitlab::service_ip_v6
  • is running the puppet role(gitlab)
  • has enough disk space

DNS model during the CDN transition (T425441)

GitLab is moving behind the CDN (T425441). Which world you are in changes what a failover must touch, and both worlds are covered below.

Quick test: compare dig +short gitlab.wikimedia.org and dig +short gitlab-ssh.wikimedia.org. If they return the same address, the public name still points directly at the GitLab host (pre-migration). If they differ, gitlab.wikimedia.org is on the CDN (post-migration) and only the internal records move during a failover.

This split is the intended end state, not an anomaly. After the migration the two names deliberately resolve to different addresses: HTTPS (web UI, API, git over https) enters through the CDN edge (text-lb), where ATS forwards it to the active host via the discovery CNAME and the Envoy sidecar on port 8443, while git over SSH bypasses the CDN entirely and lands on the host's own service IP. gitlab.wikimedia.org returning CDN addresses and gitlab-ssh.wikimedia.org returning a host address is precisely how a healthy post-migration setup looks.

The records involved:

  • gitlab.discovery.wmnet, gitlab-replica-a.discovery.wmnet, gitlab-replica-b.discovery.wmnet: static CNAMEs under $ORIGIN discovery.wmnet. in templates/wmnet of the operations/dns repository, each pointing at a host's public FQDN. ATS (the CDN backend layer) resolves these to route the public HTTPS names to the right host, on port 8443 through the Envoy TLS sidecar. They exist and are safe to swap in both worlds.
  • gitlab-ssh.wikimedia.org, gitlab-ssh-replica-a|b.wikimedia.org: public A/AAAA records (in templates/wikimedia.org) pointing directly at the GitLab hosts. Git over SSH uses these in both worlds.
  • gitlab.wikimedia.org, gitlab-replica-a|b.wikimedia.org: public A/AAAA records. Pre-migration they point directly at the hosts and must be swapped during a failover. Post-migration they belong to the CDN (text-lb), never move during a failover, and must not be touched.

A failover therefore always swaps the discovery CNAMEs and the gitlab-ssh records; while gitlab is not yet behind the CDN it additionally swaps the public gitlab A/AAAA records (transition: remove this sentence and the marked steps below after T425441).

Planned Failover

A planned failover means the old production instance is responding and working properly and doing a recent backup is possible. There is no data loss. The following steps are needed to failover to a new host.

Before failover

A full checklist can be found in one of the failover tasks: T358559.

Changes to prepare in advance

Prepare the following patches, but do not merge them until the cookbook prompts you:

  • configure new host with profile::gitlab::service_name: 'gitlab.wikimedia.org' (or other name if switching replicas, example change 802150)
  • configure new host in profile::gitlab::active_host (example change 802150, not needed for replicas)
  • one operations/dns change containing all the record swaps for both hosts, detailed below.

Example DNS change for a failover of gitlab.wikimedia.org from gitlab1004 to gitlab1003 (which becomes the new primary while gitlab1004 takes over gitlab-replica-a; substitute your actual hosts and check the current addresses in the zonefile).

In templates/wmnet, under $ORIGIN discovery.wmnet. (both worlds):

-gitlab            60 IN CNAME gitlab1004.wikimedia.org.
-gitlab-replica-a  60 IN CNAME gitlab1003.wikimedia.org.
+gitlab            60 IN CNAME gitlab1003.wikimedia.org.
+gitlab-replica-a  60 IN CNAME gitlab1004.wikimedia.org.
 gitlab-replica-b  60 IN CNAME gitlab2002.wikimedia.org.

In templates/wikimedia.org, swap the A/AAAA records of the two gitlab-ssh names (both worlds; values as of this writing, verify against the zonefile):

-gitlab-ssh 300 IN A 208.80.154.145
-gitlab-ssh 300 IN AAAA 2620:0:861:2:208:80:154:145
-gitlab-ssh-replica-a 300 IN A 208.80.154.15
-gitlab-ssh-replica-a 300 IN AAAA 2620:0:861:1:208:80:154:15
+gitlab-ssh 300 IN A 208.80.154.15
+gitlab-ssh 300 IN AAAA 2620:0:861:1:208:80:154:15
+gitlab-ssh-replica-a 300 IN A 208.80.154.145
+gitlab-ssh-replica-a 300 IN AAAA 2620:0:861:2:208:80:154:145

(transition: remove after T425441) While gitlab is not yet behind the CDN, the same patch must also swap the public HTTPS records in templates/wikimedia.org, mirroring the gitlab-ssh swap above for gitlab and gitlab-replica-a (A and AAAA, same addresses as their gitlab-ssh counterparts; example change 802473). Check for PTR records as the old procedure did.

Once gitlab is behind the CDN, do not touch the gitlab.wikimedia.org / gitlab-replica-*.wikimedia.org A/AAAA records: they point at text-lb and stay where they are. No PTR changes are needed either, since the hosts keep their own addresses.

Note: the TTL on the discovery CNAMEs is being lowered to 60 seconds (gerrit:c/operations/dns/+/1310135, in review at the time of writing). If that change is not merged yet, account for the current, longer TTL when timing the switch.

During failover (using cookbook)

The cookbook will handle all of the manual steps of the failover, except for merging any puppet or DNS changes. Be sure to make sure that you have the --switch-from-host and --switch-to-host hosts correct. Running the cookbook with the --dry-run flag will let you see the steps handled without making any changes. The cookbook will take about 4+ hours. Ensure that it is run in tmux/screen.

  • Start gitlab failover cookbook on the cumin host with cookbook sre.gitlab.failover --switch-from-host <current gitlab host> --switch-to-host <new gitlab host> -t Tzzzz
  • Confirm that the intended migration is correct (e.g., that gitlab.wikimedia.org, gitlab-replica-a.wikimedia.org and gitlab-replica-b.wikimedia.org will be on the correct machine at the end)
  • When prompted to, merge the change prepared earlier to set profile::gitlab::service_name: 'gitlab.wikimedia.org'
  • When prompted to, merge the DNS change prepared earlier, then run authdns-update on the DNS master (following DNS#Changing records in a zonefile)
  • Test that the new gitlab instance looks correct.

What the cookbook verifies after the DNS merge depends on its version:

  • Current cookbook (transition: remove after T425441): snapshots the public IPs of the two external_url names at startup and asserts they have swapped. This check only works while the public names point directly at the hosts; it does not look at the discovery or gitlab-ssh records, so verify those manually with the spot checks below.
  • Updated cookbook (gerrit:c/operations/cookbooks/+/1320977, with fixes in gerrit:c/operations/cookbooks/+/1320978): snapshots the authoritative discovery CNAME targets and the gitlab-ssh IPs at startup, wipes the internal recursor cache for the four names via sre.dns.wipe-cache, then asserts each name holds the pre-migration value of its counterpart. If a name still resolves to its old value you are offered a retry, so you can fix the merge or wait out caches without aborting the run. Pre-migration, additionally verify the public HTTPS swap manually (transition: remove after T425441).

Manual spot checks, in addition to what the cookbook verifies:

# Both worlds: the discovery name must point at the new primary
dig +short CNAME gitlab.discovery.wmnet @10.3.0.1

# Both worlds: git over SSH must answer from the new primary
ssh -T git@gitlab-ssh.wikimedia.org   # expect: Welcome to GitLab, @username!

# Both worlds: HTTPS must reach the new primary
curl -sI https://gitlab.wikimedia.org/explore   # expect HTTP 200 (x-cache header present once behind the CDN)

(transition: remove after T425441) Pre-migration, also confirm the public records swapped: dig +short gitlab.wikimedia.org must now return the new primary's address (the same as gitlab-ssh.wikimedia.org).

During failover (manual steps)

  • pause all GitLab Runners
  • stop puppet on old host with sudo disable-puppet "Failover in progress"
  • stop write access on nginx and ssh-gitlab on old host with gitlab-ctl stop nginx and systemctl stop ssh-gitlab
  • create full backup on old host:
/usr/bin/gitlab-backup create CRON=1 STRATEGY=copy SKIP=artifacts GZIP_RSYNCABLE="true" GITLAB_BACKUP_MAX_CONCURRENCY="4" GITLAB_BACKUP_MAX_STORAGE_CONCURRENCY="1"
  • sync backup, on to new host: /usr/bin/rsync -avp /srv/gitlab-backup/ rsync://<NEW_HOST>.wikimedia.org/data-backup
  • configure new host with profile::gitlab::service_name: 'gitlab.wikimedia.org' (example change 802150)
  • configure new host in profile::gitlab::active_host (example change 802150)
  • trigger restore on new host /srv/gitlab-backup/gitlab-restore.sh
  • merge the single prepared operations/dns patch (discovery CNAMEs + gitlab-ssh records, plus the public gitlab records pre-migration (transition: remove after T425441)) and run authdns-update
  • wipe the recursor cache: sudo cookbook sre.dns.wipe-cache gitlab.discovery.wmnet gitlab-replica-a.discovery.wmnet gitlab-ssh.wikimedia.org gitlab-ssh-replica-a.wikimedia.org (pre-migration, append the public names too; transition: remove after T425441)
  • verify DNS with the dig / ssh / curl spot checks from the cookbook section above
  • verify installation (login, push, pull, look at metrics)
  • run puppet on new host
  • enable puppet on old host with sudo enable-puppet "Failover in progress"
  • unpause all GitLab Runners
  • announce end of downtime

After failover

  • Make sure new instance works properly (login, trigger a CI job)
  • Check if timers for backups are present on production host
  • Check if timers for restore are present on replicas only

Unplanned Failover

A unplanned failover means the old production instance is not responding/lost and it is not possible to create a backup is possible. There is up to 24 hours of data loss GitLab.

Get as new data as possible

Check the age of the backup in bacula and on the existing replicas. If the backup is reasonably new, use this backup (make sure to check GitLab/Backup and Restore#Fetch backups from bacula). If that backup is too old, try to manually schedule a database dump and rsync the git repositories. However this is not an automated step and needs more planning.

During failover

The following steps assume that the old host is not available anymore and a replica with the most recent ("latest") backup is used to failover:

  • configure new host with profile::gitlab::service_name: 'gitlab.wikimedia.org' (example change 802150)
  • configure new host in profile::gitlab::active_host (example change 802150)
  • if needed, trigger a restore on new host /srv/gitlab-backup/gitlab-restore.sh (not needed if new backup can't be created)
  • point gitlab.discovery.wmnet (CNAME in templates/wmnet) and the gitlab-ssh.wikimedia.org A/AAAA records (in templates/wikimedia.org) at the new host (pre-migration, also repoint the public gitlab.wikimedia.org A/AAAA records; transition: remove after T425441), then run authdns-update
  • wipe the recursor cache for the affected names with sre.dns.wipe-cache and verify with dig; the records of the dead host's replica name can be cleaned up afterwards
  • verify installation (login, push, pull, look at metrics)
  • run puppet on new host

Aborting Failover/Rollback

Sometimes, you will need to abort the failover while the work is in progress. These instructions do not apply once you have finalised the failover process, switching back at that point will require working the full process again. Find the point that your need to start at, and work the remainder of the instructions.

I've already merged some changes

After you merge changes, there are two possible situations. In both cases, this situation should be resolved before restarting gitlab and removing the deploy page.

I've merged the DNS change

This should be the least likely scenario, since all previous steps should have gone correctly. Aborting the failover at this point opens the possibility that writes to the new host have started, and may create a split-brain situation if you decide to switch back. In almost all cases, you will be better off leaving the host as it is, since this is the last step of the migration. If you do need to abort the migration at this point, you should STRONGLY consider following the full failover process. Be careful here. Data can be lost! If you're ok with potentially losing data, then revert your DNS change (restoring the discovery CNAMEs, the gitlab-ssh A/AAAA records and, pre-migration, the public gitlab records (transition: remove after T425441)), run authdns-update, wipe the recursor cache for the affected names with sre.dns.wipe-cache, verify with the dig / ssh / curl spot checks, and follow the two sections below (reverting the puppet change, and restoring access to the old gitlab instance).

I've merged the puppet change

If the DNS change has not been merged yet, then no changes will have been made to the new system which means they should both be in sync. In this case, revert your change that switches the profile::gitlab::service_name and profile::gitlab::active_host, run puppet on both hosts, and then proceed with restoring access to gitlab with the instructions below.

I haven't merged anything yet

Up to the point where you merge changes, rolling back is fairly simple, since we haven't changed anything on either hosts, only gone as far as creating a backup:

  • Restart gitlab on the host that will remain as the primary: sudo gitlab-ctl restart; sudo systemctl start gitlab-ssh; sudo gitlab-ctl depoloy-page up
  • Restart puppet on both hosts: sudo enable-puppet
  • Check if you need to remove any backup files that were created, in order to not fill up the disk