Jump to content

Yubikey-SSH-FIDO

From Wikitech

This page describes how to generate FIDO/U2F authenticator-backed SSH keys, and use them for production access to WMF systems, enabling 2FA for SSH. With this setup you can create an FIDO-backed SSH key, which consists of two parts:

  • A private SSH key file very similar to a traditional SSH key. The public key gets added to Puppet (data.yaml) and is distributed to the servers you have access to, the private key gets loaded into your SSH agent. The difference to a classic SSH key is that it's not usable without the hardware-backed part described below and thus protected against data exfiltration (like an exploit fetching your ~/.ssh directory). This part of the key is also sometimes referred to as a key handle.
  • The second part is the private key stored on the FIDO token and cannot be extracted from it.

Each of these key parts isn't usable by itself. The actual SSH key is derived by the FIDO token whenever authentication is requested. Each attempt needs to be confirmed by touching/tapping the token.

The support is native to contemporary OpenSSH releases and complemented by an external library used to access the tokens called libfido.

Prerequisites

  • You need a FIDO-compatible security key. If you are a WMF staff member, you can reach out to ITS to obtain one. The following keys are known to be compatible have been shipped by ITS over the past years:
    • YubiKey 5 (various models)
    • YubiKey 4 (various models)

Since FIDO is a standard there's also plenty of other options, compatibility can be checked at https://fidoalliance.org/certification/fido-certified-products/

Support for FIDO U2F was added in OpenSSH 8.2 and important features were added in OpenSSH 8.9, so using 8.9 or later is recommended.

The version of OpenSSH shipped in MacOS does not enable FIDO. Instead OpenSSH needs to be installed from Homebrew (https://brew.sh/)

Limitations

You cannot connect to systems still running Debian 10/Buster using FIDO keys. If you need this access, please see #Access to systems running Debian 10/Buster.

The version of OpenSSH deployed in our switches doesn't support FIDO keys across the installed base. If you have network access, please see #Access to network devices.

Instructions

1. Generate new SSH key pair(s)

For our use of FIDO SSH tokens, the PIN isn't used in the default path. Still it makes sense to change the PIN towards a custom one away from the factory default. If you use a Yubikey, simply install the Yubikey manager (yubikey-manager on Debian/Ubuntu or Fedora, ykman on Homebrew) and run 

ykman fido access change-pin

The next step depends on the capabilities of your FIDO token. (Hint: If you are unsure of your Yubikey model, you can often find the version using ykinfo -v or ykman info.) If you have a Yubikey 5 or different modern FIDO key, connect it to your computer and execute the following to create a new keypair (replace TOKENNAME with a descriptive name of your token which be appended to the public part of your key. You can also omit passing the argument, which will default to USER@HOSTNAME, but if you have more than one FIDO token it's recommended to add a descriptive name like Yubikey-USB-C or Yubikey-USB-A. 

   ssh-keygen -t ed25519-sk -C TOKENNAME

If you have a Yubikey 4 or an older model which doesn't support ed25519 keys, run the following: 

   ssh-keygen -t ecdsa-sk -C TOKENNAME

You will be prompted to touch the security key when it starts flashing.

ssh-keygen defaults to FIDO-backed keys which are stored on the FIDO token with a key handle stored on disk, there are some other key options which are not relevant to the use of FIDO tokens when accessing Wikimedia systems, which are listed at Yubikey-SSH-FIDO#Non-standard key types.

You will then be prompted for the filename to use. It is best to use a name you can associate with the physical device, especially if you have multiple Yubikeys. For example "yubi_black" and "yubi_colour" so you can tell them apart.

Next you will be prompted for a passphrase. With a FIDO-backed key, adding a passphrase to the key handle part of your key is significantly less important compared to a traditional SSH, but it still makes sense to add a passphrase as defense in depth (and the passphrase will only be requested once when you load it into your SSH agent).

ssh-keygen will create two files on your local computer:

  • A public key which can be added to 'authorized-keys' on remote systems as before
  • A private key, which is the key handle to your FIDO token

If you have multiple Yubikeys repeat the above steps multiple times, generating a keypair for each Yubikey. Make sure only one Yubikey is connected to the computer at any time.

2. Add the public keys to puppet repo for production access

Now the public key(s) generated in the first step needs to be registered in the Wikimedia Puppet repository (which ensures that the key is installed to the authorized_keys of the Wikimedia servers you can access).

Simply open a task in Phabricator and add the SRE-Access-Requests tag. There's a rotation of SREs handling the tickets.

If you are familiar with the Wikimedia Puppet repository you can also go ahead and prepare a patch with your key change. The YAML file in question is modules/admin/data/data.yaml, simply search for your username and add your new key to the ssh_keys: field.

When adding multiple keys it makes sense to have a description after each that helps identify it. That makes it simpler to remove one of them in the future if a particular key is lost/damaged.

3. Modify your SSH config file to use the new keys and test access

The last thing to do is adjust your local SSH config to use the new keys when connecting to production systems.

It is recommended to use default default SSH client config as provided by wmf-laptop, it covers all the settings to use FIDO SSH access smoothly.

To use your new key simply:

  • add your new key to your SSH agent via ssh-add .ssh/id_KEYNAME
  • update the IdentityFile directive in your SSH config to the new key handle file you created

If you generated multiple keys you can add them all by using several IdentityFile statements, each on a separate line.

You can now SSH into a server you're allowed to. Your first login will not only require one token tap, but two. This has the following background: SSH access to Wikimedia is routed via so-called bastion hosts (these servers are named bast*).

Every SSH connection to a production host other than the bastion hosts goes via two SSH connections:

  • A connection to the configured bastion host over the internet
  • A second connection through that bastion session to the destination server

There is however an OpenSSH feature which mostly avoids needing to tap twice: SSH Multiplexing allows the reuse of an existing TCP connection for multiple SSH sessions, which means that:

  • The first SSH connection (often also called the master connection) will require tapping twice
  • As long as one SSH connection is open (or if ControlPersist is used, all further connections will reuse the TCP connection an only require a token tap for the connection to the target server

If you use the default SSH Wikimedia configuration this will work out of the box, if not please see #I need to tap my token twice for every connection attempt.

4. Complete the migration by removing your old SSH key(s)

Once your tests are complete and you're happy with the new setup, you can reopen the Phabricator task that added your key and request the deletion of your old non-FIDO key.

Tips and tricks

Getting Desktop notifications that a touch might be required

If you are running a Debian Trixie and later system, you can get easy notifications on your window manager using libnotify and yubikey-touch-detector. The following assumes that $XDG_CONFIG_HOME=$HOME/.config, adjust accordingly. 

apt install yubikey-touch-detector

mkdir $HOME/.config/yubikey-touch-detector/

cp /usr/share/doc/yubikey-touch-detector/examples/service.conf.example $HOME/.config/yubikey-touch-detector/

systemctl --user enable --now yubikey-touch-detector.service

Feel free to edit the config file if needed, but the defaults are pretty sane.

Special cases

Access to systems running Debian 10/Buster

We have handful of legacy systems running on Debian 10/Buster. The version of OpenSSH in Debian Buster is too old to support FIDO SSH keys. If access to those systems is needed, there's the option to add an additional buster_ssh_keys: entry in data.yaml. The key configured there will be installed on Buster nodes instead of the main key config entry containing the FIDO key(s).

Access to network devices

FIDO isn't supported in the OpenSSH version of all our network devices yet, as such we'll remain using classic SSH keys until the hardware is upgraded (keys for SSH access are managed via the Homer repository and not the Puppet repository). You can use the following config snippet in ~/.ssh/config to configure a different key for accessing the network devices: 

Host cr* lsw* asw* msw* mr* asw* cloudsw* ssw* fasw* pfw*
  User $USER
  IdentityFile ~/.ssh/$ECC_KEY
  ProxyCommand ssh -a -W %h:%p $BASTION

The public key should be added under 'users' in common.yaml in the homer public repo.

Non-standard key types

ssh-keygen defaults to keys which are stored on the FIDO token with a key handle stored on disk (and what is also used for accessing Wikimedia systems). There are additional possible key types which can be generated by passing an option during ssh-keygen, which are not in use when accessing Wikimedia systems:

  • The verify-required option requires the user to enter their FIDO PIN every time they connect to a remote host. While this adds an additional layer of security, it would be too cumbersome for day-to-day SSH access (but it might be an option for some systems with elevated sensitivity in a later step).
  • If the resident option is used, this means that the key handle isn't stored on disk, but also on the FIDO token. While there are some special use cases for this (e.g. using a token on multiple computers), this isn't a setup that is supported for accessing Wikimedia systems.

Troubleshooting

I need to tap my token twice for every connection attempt

SSH access to Wikimedia is routed via so-called bastion hosts (these servers are named bast*).

Every SSH connection to a production host other than the bastion hosts goes via two SSH connections:

  • A connection to the configured bastion host over the internet
  • A second connection through that bastion session to the destination server

If this is done in the simplest way then every time you connect to a remote host it will require two presses of the FIDO token, one for each of these connections. This can be confusing; while the token will blink twice, there's usually no other indication of needing to tap a second time.

There is however an OpenSSH feature which mostly avoids this: SSH Multiplexing allows the reuse of an existing TCP connection for multiple SSH sessions, which means that:

  • The first SSH connection (often also called the master connection) will require tapping twice
  • As long as one SSH connection is open (or if ControlPersist is used (see below)), all further connections will reuse the TCP connection an only require a token tap for the connection to the target server

Two configuration settings are required to make use of SSH multiplexing:

  • The connection to the bastions needs to enable a SSH multiplexing by configuring a ControlMaster (setting it to auto is the recommended setting)
  • The connection to the bastions needs to configure a ControlPath (which is the socket used for connection multiplexing)
  • (Optional) If ControlPersist is set to a time frame (e.g. 60m), then the master connection will remain open in the background and avoid the need to double-tap if you have only made one SSH connection and closed it before opening a new one.

Our default SSH client config as provided by wmf-laptop already uses SSH multiplexing. As an example, however, the below shows the specific commands that are used to ensure SSH multiplexing and proxying work: 

   Host bast1003.wikimedia.org
       ControlMaster auto
       ControlPath ~/.ssh/control-%C
       ControlPersist 10m
       IdentitiesOnly yes
       IdentityFile ~/.ssh/yubi_colour
       IdentityFile ~/.ssh/yubi_black
       ProxyCommand none

   Host *.wikimedia.org
       IdentitiesOnly yes
       IdentityFile ~/.ssh/yubi_colour
       IdentityFile ~/.ssh/yubi_black
       ProxyCommand ssh -a -W %h:%p bast1003.wikimedia.org

References / background

A few useful references with more background:

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Yubikey-SSH-FIDO&oldid=2368792"