GitLab/Cheat Sheet

More or less loose collection of GitLab commands that might be helpful (or were helpful at some point)

GitLab

Restarts

Reboot GitLab instance:

cookbook sre.hosts.reboot-single gitlab1004.wikimedia.org

Restart GitLab service:

gitlab-ctl restart

Reload GitLab config:

gitlab-ctl reconfigure

Restart (git) SSH daemon:

systemctl restart ssh-gitlab

Check GitLab status:

gitlab-ctl status

Get GitLab logs (see also https://docs.gitlab.com/omnibus/settings/logs.html#tail-logs-in-a-console-on-the-server):

sudo gitlab-ctl tail

Backup and Restore

Trigger a full backup:

systemctl start full-backup

Trigger a config backup:

systemctl start config-backup

Get logs of backup:

journalctl -u full-backup

Sync backup to replica (assuming gitlab1003 is the replica):

systemctl start rsync-data-backup-gitlab1003.wikimedia.org.service
systemctl start rsync-config-backup-gitlab1003.wikimedia.org.service

Trigger a restore on replica:

systemctl start backup-restore

Check status of restore:

systemctl status backup-restore.service

Check logs of restore:

journalctl -u backup-restore.service

Postgres

Connect to the embedded Postgres database:

sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql -d gitlabhq_production

gitlab-runner

Restarts

Gracefully reboot all GitLab Runner:

cookbook sre.gitlab.reboot-runner --reason "reason" --alias gitlab-runner reboot

Restart gitlab-runner service:

systemctl restart gitlab-runner

Depool/pause a runner

Run the depool command on the host (/usr/local/bin/depool)

Pool/unpause a runner

Run the pool command on the host (/usr/local/bin/pool)

Register/Unregister:

Unregister Runner (note, this can also be done by setting profile::gitlab::runner::ensure to absent and run puppet):

gitlab-runner unregister --name `hostname -f`

Unregister Trusted Runner (different because running as non-root:

su -s /bin/bash gitlab-runner -c "gitlab-runner unregister --name `hostname -f`"

Register Runner (handled by puppet):

run-puppet-agent

Clean docker cache:

Cleanup old docker volumes in /var/lib/docker:

/usr/share/gitlab-runner/clear-docker-cache

Registering to a replica:

If you need to test something on a replica using a runner, it can be done easily

1. Pause the runner on the admin interface of the primary (https://gitlab.wikimedia.org/admin/runners)
2. Disable puppet:

   disable-puppet "Re-registering with replica temporarily"
   

3. Stop the runner

   systemctl stop gitlab-runner
   

4. Change the host the runner registers with in `config.toml`, `config-template.toml`, `managed.toml`, and `registration.toml`

   sudo sed -i -e 's/gitlab.wikimedia.org/gitlab-replica-old.wikimedia.org/' /etc/gitlab-runner/*.toml
   

5. Start the runner:

   systemctl start gitlab-runner
   

After this, you can re-enable puppet to register the runner with the primary again

Run commands against all WMCS Runners (cloudcumin)

cloudcumin host can be used to run commands against all WMCS Runners. Login to cloudcumin:

sudo cumin 'O{project:gitlab-runners name:^runner.*}' 'command'

For example show all runner system_ids:

sudo cumin 'O{project:gitlab-runners name:^runner.*}' 'cat /etc/gitlab-runner/.runner_system_id'

Depool/pause all WMCS runners

Login to cloudcumin and run:

sudo cumin 'O{project:gitlab-runners name:^runner.*}' 'depool'

Pool/unpause all WMCS runners

Login to cloudcumin and run:

sudo cumin 'O{project:gitlab-runners name:^runner.*}' 'pool'