Overview

Key creation

Authentication servers

There are two authentication servers, auth1001.eqiad.wmment and auth2001.codfw.wmnet. Each is equipped with a YubiHSM which has been configured to only allow validation, but no creation of keys (keys can only be generated on the third HSM attached to a personalisation notebook stored in OIT, see above).

With YubiHSMs, the keys are generated on the HSM (which also ensures that quality randomness is used by using the hardware RNG on chip). Each HSM supports different mode flags (e.g. creating/verifying etc) (by default an HSM supports the full range of operations ).

There are two methods to store the keys; internally on the HSM or in the form of AEAD blocks (authenticated encryption with associated data). We're using the AEAD storage since we want to use two auth servers/HSMs for redundancy.

To verify and decrypt an AEAD block, a symmetric key and a random nonce are required. These are stored securely on the YubiHSM devices, i.e. they cannot be read by an attacker who compromised the authentication servers. The encrypted AEAD blocks can thus be stored on the local filesystem of the authentication servers (and replicated between the auth servers)

An application validating an OTP passes the OTP and the AEAD to the YubiHSM, which then decrypts the key, validates it and returns the result along with a timestamp to counter replay attacks. These counters are stored on the validation servers (and also needs to be replicated among them). These keys are then used to personalise Yubikeys with the created keys (using ykpersonalize).

All AEADs are replicated between auth1001 and auth2001 using rsync.

Each authentication server runs two low level service

yhsm-daemon mediates the concurrent access to the USB serial interface of the YubiHSM (puppetised via yubiauth::yhsm_daemon)
yhsm-yubikey-ksm is the YubiHSM backend for the validation service (running the actual validation of the OTP) (puppetised via yubiauth::yhsm_yubikey_ksm)

In addition each authentication server runs vubikey-val, the higher level validation server (it also takes care of replicating counter values between the auth servers to prevent replay attacks). It's implemented in PHP and served via Apache and runs a local MySQL database. It runs a local replication server to synchronise OTP counters between the two servers.

When logging in on one of the bastion hosts, the two authentication servers are queried by pam_yubico (first version at https://gerrit.wikimedia.org/r/#/c/282160/ with using HTTPS and accessing both auth servers TBD in followup commits)

Enabling 2fa access will be a gradual process: For all members of the "yubiauth" group an additional authentication method (using the OpenSSH configuration directive AuthenticationMethods) 2fa will be required in addition to the SSH key. Once all users in the production cluster are converted to 2fa, the group membership check can be dropped. https://gerrit.wikimedia.org/r/#/c/281630/

I'm planning to add the Yubico key ID to modules/admin/data/data.yaml, so that /etc/yubikeys (a list of all permitted Yubikeys used by pam_yubico) can be generated from these values. This would also add the membership in the yubiauth group.