GitLab/Migration
SSH hostname migration: gitlab-ssh.wikimedia.org
As part of moving GitLab's web traffic behind the Wikimedia CDN (T425441), SSH access to GitLab is moving to a dedicated hostname: gitlab-ssh.wikimedia.org. HTTPS (web UI, clones over https://, API) stays on gitlab.wikimedia.org and is not affected.
Both hostnames currently resolve to the same server and serve the same SSH host key, so you can migrate at any time. SSH on gitlab.wikimedia.org keeps working until the cut-over date (announced on wikitech-l and via a GitLab banner); after that, SSH will only work via gitlab-ssh.wikimedia.org.
Recommended: rewrite with git insteadOf
A single git config rewrites all your GitLab SSH remotes at once — existing and future, across every repository — without touching each one:
git config --global url."git@gitlab-ssh.wikimedia.org:".insteadOf "git@gitlab.wikimedia.org:"
This leaves your remotes untouched on disk; git substitutes the hostname at run time. Verify it resolves as expected:
$ git ls-remote --get-url git@gitlab.wikimedia.org:repos/releng/scap.git
git@gitlab-ssh.wikimedia.org:repos/releng/scap.git
Alternative: rewrite the remotes in place
A helper script (P93905) can scan a directory tree and rewrite many repositories interactively (with --dry-run and an optional --update-known-hosts that derives the new known_hosts entry from your already-trusted one):
./migrate_gitlab_remotes.py --workdir ~/code --dry-run
HTTPS remotes are left untouched by all of the above.
Host key
gitlab-ssh.wikimedia.org serves the same host key as gitlab.wikimedia.org — there is no new fingerprint to verify. If you connect without a known_hosts entry for the new name, recent OpenSSH will tell you the key is already known under its other name:
This host key is known by the following other names/addresses:
~/.ssh/known_hosts: gitlab.wikimedia.org
If you have a custom ~/.ssh/config
If you have a Host gitlab.wikimedia.org block (specific IdentityFile, IdentitiesOnly, etc.), add the new name to the same line so it keeps applying:
Host gitlab.wikimedia.org gitlab-ssh.wikimedia.org
User yourusername
IdentityFile ~/.ssh/your_key
IdentitiesOnly yes
wmf-laptop users: re-run the config generator
Upgrading the wmf-laptop package (1.0.6+) only refreshes the template at /usr/share/wmf-laptop/ssh-client-config. It does not rewrite your ~/.ssh/config. Re-run wmf-update-ssh-config after the upgrade so the WMF section is regenerated with the gitlab-ssh.wikimedia.org block. If you skip this, gitlab-ssh.wikimedia.org falls into the generic Host *.wikimedia.org block (production key + bastion ProxyCommand) and SSH fails with "Permission denied (publickey)".
Verify:
ssh -G gitlab-ssh.wikimedia.org | grep -iE 'identityfile|proxycommand'
# correct: your cloud key, no ProxyCommand
# stale config: PRODUCTION_KEY + a bastion ProxyCommand -> re-run wmf-update-ssh-config
glab (GitLab CLI)
glab picks the GitLab host from your git remotes. Once a remote uses gitlab-ssh.wikimedia.org (directly or via the insteadOf rewrite), glab no longer recognises a known host:
none of the git remotes configured for this repository point to a known GitLab host. Configured remotes: gitlab-ssh.wikimedia.org
Add a second host block in ~/.config/glab-cli/config.yml mirroring the gitlab.wikimedia.org one but keyed on the new name. Keep api_host on gitlab.wikimedia.org (only the SSH host changed):
hosts:
gitlab.wikimedia.org:
token: <your token>
api_host: gitlab.wikimedia.org
ssh_host: gitlab-ssh.wikimedia.org
git_protocol: ssh
api_protocol: https
user: <you>
gitlab-ssh.wikimedia.org:
token: <your token>
api_host: gitlab.wikimedia.org
ssh_host: gitlab-ssh.wikimedia.org
git_protocol: ssh
api_protocol: https
user: <you>
(Unrelated gotcha seen on a fresh glab install: a top-level host: gitlab.com in config.yml makes glab hit gitlab.com; set it to gitlab.wikimedia.org.)
GerritLab
Ensure that you're using at least version 2.2.0 of GerritLab.
Configure GerritLab so that it knows how to map gitlab-ssh.wikimedia.org to https://gitlab.wikimedia.org.
git config --global gerritlab.gitlab-ssh.wikimedia.org.url https://gitlab.wikimedia.org
Troubleshooting: "Permission denied (publickey)"
If SSH to gitlab-ssh.wikimedia.org fails with "Permission denied (publickey)" while gitlab.wikimedia.org still works, the problem is client-side: both names point to the same server with the same authorized keys, so the key that works for one must simply be offered for the other. Check, in order:
- Stale config (wmf-laptop): re-run
wmf-update-ssh-config(see above) and re-check withssh -G. - Key not registered: run
ssh -T git@gitlab.wikimedia.organdssh -T git@gitlab-ssh.wikimedia.org. If both fail, the key your agent offers is not in your GitLab account; add it under Preferences > SSH Keys. - Custom SSH agent (e.g. 1Password): a top
Host *block withIdentityAgent+IdentitiesOnly yestakes precedence over the per-host block and can make the agent offer the wrong key for the new hostname. Make sure the key that works forgitlab.wikimedia.orgis also offered forgitlab-ssh.wikimedia.org(in 1Password, addgitlab-ssh.wikimedia.orgto that key's allowed hosts).
Use ssh -vvv git@gitlab-ssh.wikimedia.org to see which key is offered and whether the server accepts it.
CI and automation
Update anything that clones from GitLab over SSH outside your workstation: CI job definitions, deployment scripts, cron jobs, bot accounts, and documentation. Search for git@gitlab.wikimedia.org in your project's configuration. The insteadOf trick also works in a job's git config if rewriting URLs is impractical.
Questions
Comments and questions on T425441.