This is an outline of Fundraising system authentication and two-factor authentication as of 10/2015.

goals

meet 2FA and password requirements for PCI
easy to support
flexibility and resiliency

design decisions

SSH key authentication is standard and secure
2FA on bastions with single factor on internal hosts is acceptable because internal hosts are only accessible via bastions
Yubikeys with Yubico cloud API are easy to administer and use
Kerberos passwords are a well-integrated on-premise option that enforces PCI password standards
Shadow passwords can be used as a fallback for SREs if network is inaccessible
Radius->Auth Server->Yubico API limits outbound firewall policy requirements

authentication scheme overview

all servers use ssh public keys as the primary authentication method
bastion server ssh requires 2FA

first factor is the ssh public key
second factor is "a password" satisfied by yubikey, kerberos, or local shadow password

all servers use the same yubikey->kerberos->shadow-password scheme where passwords are used

yubikeys are authenticated via radius server passthrough to the yubico cloud API
kerberos preempts local shadow passwords
fallback to shadow password authentication as backup if auth servers are inaccessible

firewall policies enforce bastion use and restrict access to internal hosts by unix group
users, groups, ssh keys, unix passwords, and yubikey user mappings are all managed with puppet
users can change their kerberos password from any host

modified PAM password backend

authentication-related configuration

all hosts

/etc/ssh/sshd_config

We enable 2FA for bastion servers, first checking ssh public key and then password:

AuthenticationMethods publickey,keyboard-interactive

We disable 2FA for other servers:

AuthenticationMethods publickey

Note that OpenSSH can be configured to behave differently based on various session criteria using the Match directive. For example to do a soft rollout, we initially enabled 2FA only for members of the sysops group:

AuthenticationMethods publickey

Match Group sysops
	AuthenticationMethods publickey,keyboard-interactive

/etc/pam.d/sshd

With OpenSSH set to use PAM, this is the next step with keyboard-interactive. Our config just forwards to pam.d/common-auth.

@include common-auth

/etc/pam.d/common-auth

This is the default password handling sequence, which is used for ssh, terminal, sudo, etc.

auth [success=ignore default=1]  pam_exec.so expose_authtok quiet /usr/local/bin/yubikey_otp_filter
auth [success=3 default=2]       pam_radius_auth.so
auth [success=2 default=ignore]  pam_krb5.so minimum_uid=500 use_first_pass
auth [success=1 default=ignore]  pam_unix.so use_first_pass
auth requisite                   pam_deny.so

/etc/pam.d/common-passwd

User-accessed password utilities administer kerberos passwords.

password  [success=1 default=ignore]    pam_krb5.so minimum_uid=500
password  requisite                     pam_deny.s
password  required                      pam_permit.so

/usr/local/bin/yubikey_otp_filter

This is the simple script pam_exec.so invokes in an auth context.

#!/usr/bin/perl
use Sys::Syslog;

if (<STDIN> =~ /^[a-z]{44}[\x00\r\n]*$/) {
    printlog("$ENV{'PAM_USER'} Yubikey OTP detected");
    exit 0;
} else {
    printlog("$ENV{'PAM_USER'} non-Yubikey password");
    exit 1;
}

sub printlog {
    Sys::Syslog::setlogsock('unix');
    Sys::Syslog::openlog('pam_yubikey_filter','ndelay,pid','auth');
    Sys::Syslog::syslog('info',shift);
    Sys::Syslog::closelog();
}

/etc/pam_radius_auth.conf

The radius client configuration tries two possible servers with a 3 second timeout.

# server      secret    timeout
10.1.1.5      pa$$word  3
10.2.0.5      pa$$word  3

/etc/krb5.conf

The kerberos client config tries two possible kdc servers for authentication. Password admin is directed to the master.

WMF-FUNDRAISING = {
    kdc = 10.1.1.5
    kdc = 10.2.0.5
    admin_server = 10.1.1.5
}

auth servers

Authentication servers have the above settings plus kerberos and freeradius server configuration. The kerberos and freeradius settings are fairly generic, and mostly beyond the scope of this overview.

/etc/freeradius/users

Freeradius is configured to act as a simple pass through to the yubico cloud API. The server is configured to use PAM to authenticate users. This means every user must exist on the auth server, though login can be disabled.

DEFAULT Auth-Type := PAM

/etc/pam/radiusd

This configuration comes into play with freeradius using PAM to authenticate users. This is where pam_yubico makes the call out to the yubico cloud API.

auth required pam_yubico.so id=16 debug authfile=/etc/yubikey_mappings url=https://api.yubico.com/wsapi/2.0/verify?id=%d&otp=%s

/etc/yubikey_mappings

This file maps unix username to yubikey public ID, which are included as the first 12 characters of every OTP.

jg:ccccccebebeb
bk:ccccccebebeb
...

administration

users and groups are administered via (frack)puppet/hieradata/user/*.yaml
yubikey user mappings and ssh public keys are in (frack)puppet-private
shadow passwords are in (frack)puppet-private as hashes
kerberos kdc is administered on the frauth server, users can change their passwords from any host via passwd command

user experience

ssh directly to bastion

jg@laptop:~> ssh frbast-eqiad.wikimedia.org
(jg@frbast) Password: 

frbast1002 fundraising bastion host (role::bastion)

ssh to internal host, via tunnel through bastion

jg@laptop:~> ssh frlog1002
(jg@frbast) Password:

frlog1002 fundraising logger host (role::logger)