Yubikey-SSH

This procedure describes how to store your production SSH credentials on a Yubikey physical 2FA device. Doing so provides an additional layer of protection against having your key compromised via your laptop. It's recommended, but not required, for anyone with production SSH access. These instructions are written for a vanilla Ubuntu or Debian installation, and will probably require adjustment for Mac OS or other Linux environments.

When you're done, your production SSH private key will be stored only on your Yubikey. That means if it's lost or damaged, you won't be able to SSH to production without asking a colleague to help you authorize another key. If your Yubikey is lost or stolen, it will still be protected by your PIN, but you should immediately contact SREs to have the key revoked.

You can also use the same Yubikey as your 2FA device for Gmail and some other services; that isn't covered in these instructions.

Prerequisites

You already have a PGP key. It can be the same key you'll use for Pwstore, but it doesn't need to be signed by anyone yet.
You either already have SSH access to production, or don't mind waiting until your Yubikey arrives in the mail (in which case your onboarding buddy will help you with the parts of this procedure that need SSH access).

Instructions

Email techsupport@wikimedia.org and ask them to send you a Yubikey.
Done? Great, come back in a couple of days when you get a package in the mail.
Oh hey, welcome back! Your new Yubikey should be in its original packaging from Yubico. (If it's unwrapped or appears tampered with, stop. If an attacker has compromised your Yubikey, they can gain access to prod and harm our users.) Unwrap it and plug it into your laptop, then visit https://yubico.com/genuine and follow instructions, to make sure you've got the real thing and it's working correctly. It should say "verification complete" at the end.
Install the SmartCard daemon: sudo apt install scdaemon. For MacOS, you can follow https://docs.releng.linuxfoundation.org/en/latest/gpg.html#setup-gpg-agent.
Once the installation is done, unplug your Yubikey and plug it back in. Leave it plugged in for the rest of the instructions.
The default user PIN and administrator PIN are 123456 and 12345678, respectively. You'll be asked for the user PIN the first time you use your SSH key after the Yubikey loses power -- that is, after each time you plug it in or reboot. If you enter the wrong user PIN three times in a row, you'll need the admin PIN to unlock the Yubikey, including to change the user PIN. Change both PINs away from the defaults now:
1. Run gpg --change-pin, which will drop you into an interactive prompt.
2. Choose 1 - change PIN.
3. When prompted, enter first the old user PIN (123456) and then your new user PIN twice. Despite the misleading name, your "PIN" doesn't need to be a number; you should use a strong password.
4. Choose 3 - change Admin PIN.
5. When prompted, enter first the old admin PIN (12345678) and then your new admin PIN twice. Use a different strong password.
6. Unlike the user PIN, you'll only rarely need to use the admin PIN, so consider storing it in a password manager or writing it down in a safe place along with your recovery codes for other services. (If you lose both PINs, you'll be able to reset the Yubikey, but your SSH key will be permanently lost.)
7. Enter q to quit.
Add an authentication subkey to your PGP key:
1. Run gpg --expert --edit-key <your PGP key ID>, which will drop you into another interactive prompt.
2. Type the command addkey.
3. When asked what kind of key you want, choose (8) RSA (set your own capabilities).
4. The default capabilities are Sign and Encrypt; you only want the capability Authenticate, so disable Sign and Encrypt and enable Authenticate by entering S, then E, then A.
5. Next you'll be asked what keysize you want. For a recent-model Yubikey (4 or 5, as shown on the packaging) enter 4096 bits.
6. Expiration: You can enter 0 for no expiration.
7. Confirm this twice and you'll be asked for your PGP password.
8. Next, still at the GPG prompt, select the new subkey with the command key 2.
9. Move the subkey onto your Yubikey with the command keytocard.
10. When asked where you want to store the key, choose (3) Authentication key (you'll be asked for your PGP password and then the admin pin for your YubiKey from above)
11. When you return to the gpg> prompt, type the command quit to quit, and y to save changes.
Next, authorize your new SSH key:
1. To see the public key, run SSH_AUTH_SOCK=/run/user/$(id -u)/gnupg/S.gpg-agent.ssh ssh-add -L. For MacOS, it should be something like SSH_AUTH_SOCK=/Users/${USER}/.gnupg/S.gpg-agent.ssh ssh-add -L. The output should be a single long line beginning with ssh-rsa; set it aside.
2. In the operations/puppet repository, open modules/admin/data/data.yaml in your editor.
3. Find the stanza that begins with your shell username and contains an ssh_keys: list. If it doesn't exist yet, stop here and ask your onboarding buddy for help.
4. Add your new public key, retrieved above, as a second list entry below the old one -- that is, prefixed with - at the same level of indentation. (Don't delete the old one yet; you'll still need it for the next steps.)
5. Commit your change, have it reviewed by your onboarding buddy, then +2 and submit.
6. SSH to puppeserver1001.eqiad.wmnet (at this point, you're still using your old credentials) and run sudo puppet-merge. Verify that your change is the only diff, then confirm the merge. You may want to leave this window open, in case you get stuck and can't log back in.
7. Now SSH to your bastion of choice and run sudo run-puppet-agent. You may want to leave this window open too, in case you get stuck and can't log back in.
Configure SSH to use the new key instead of the old one:
1. On your laptop, open ~/.ssh/config in your editor. If it doesn't exist yet, stop here and ask your onboarding buddy for help, or refer to Production_access#Setting_up_your_SSH_config and modify it as follows.
2. In the Host stanzas for the bastion and for the rest of production, comment out the IdentityFile statement for your existing private key by prepending a #. (We won't change any other hosts, like Gerrit -- remember, your SSH credentials for production shouldn't be reused for anything else.)
3. In its place, add IdentityAgent /run/user/%i/gnupg/S.gpg-agent.ssh to both stanzas (or for MacOS, the path aforementioned).
4. If you have IdentitiesOnly yes, comment it out in both stanzas. (If you have IdentitiesOnly yes set overall, you’ll have to remove it – it’s not enough to add IdentitiesOnly no to the Host stanzas for production.)
5. Save and close the file.
SSH to the bastion again. If you succeed, your new Yubikey-only credentials are working! If you can't get in, or if you get a Password: prompt, trying again with ssh -v may help to debug.
Once you're happily SSHing around with your new key, deauthorize your old one, so that your Yubikey is required to access production. (If both keys work, the Yubikey provides no protection, just extra attack surface.)
1. Open the puppet repository and edit modules/admin/data/data.yaml again.
2. Find your stanza and delete the entry in ssh_keys: for your old key.
3. Commit your change, have it reviewed by your onboarding buddy, then +2 and submit.
4. SSH to puppetserver1001.eqiad.wmnet and run sudo puppet-merge.
5. SSH to the bastion and run sudo run-puppet-agent.
6. SSH to the same bastion one more time, to verify that the new config is correct.
Optional, but recommended: For added security, configure your Yubikey to require you to physically touch it each time you use it to authenticate. With this setting, you can leave the Yubikey plugged into your computer without worrying about a remote attacker using it to access prod; using your Yubikey will require physical presence at your machine (or knowledge of your admin PIN to turn the setting back off.)
1. Install Yubikey Manager.
  - For Debian/Ubuntu: sudo apt install yubikey-manager
2. Remove and reinsert the Yubikey after installing the Yubikey Manager
3. Run ykman --version.
4. Set the touch policy; the correct command depends on your Yubikey Manager version.
  - For ykman version 4.x (Debian bookworm): Run ykman openpgp keys set-touch aut cached
  - For ykman version 3.x (Ubuntu 19.10+, Debian bullseye+): Run ykman openpgp set-touch aut cached. You'll need to touch your Yubikey once each time you SSH. Even though you need to authenticate once to the bastion and once to the destination host, your Yubikey will cache the touch for fifteen seconds.
  - For ykman version 2.x: Run ykman openpgp touch aut on. You'll need to touch your Yubikey twice each time you SSH, once to authenticate to the bastion and once to authenticate to the destination host.
  - If there is an error Failed connecting to YubiKey 5 [OTP+FIDO+CCID]. Make sure the application have the required permissions. then try unplugging the YubiKey and plugging it back in.
5. Enter the admin PIN you selected earlier.
6. Type y to confirm.
7. Try SSHing to puppetserver1001.eqiad.wmnet again to test the new setting. When the LED on the Yubikey blinks, touch it. From now on, you'll need to do this each time you SSH. If the Yubikey ever blinks unexpectedly when you aren't trying to use it, you should become suspicious.
Optional cleanup: You can now delete the commented-out lines from your .ssh/config. You can also delete the old key from your .ssh/ directory if you like.

Resetting your Yubikey

You can reset the OpenPGP app on the Yubikey by running ykman openpgp reset, and typing y to confirm. This will delete all PGP keys stored on your Yubikey, and reset your PIN and Admin PIN to the defaults above.

More information