User:Elukey/Analytics/Kerberos

Why Kerberos?

We use it mostly for Hadoop, since it is the only reliable way to add proper authentication for users/daemons/etc..

Current setup

We have two hosts:

krb1001.eqiad.wmnet
krb2001.codfw.wmnet

Both hosts are running a kdc and a kadmind daemon, meanwhile only one kpropd. In puppet we set up the master and its replica, usually 1001 and 2001 respectively.

Daemons and their roles

KDC (Key Distribution Centre) - a daemon that authenticate a principal returning a TGT (Ticket Granting Ticket) and acting as TGS (Ticket Granting Service)
kadmin - a daemon that offers a cli to modify/read the Kerberos database (add principals, generate keytabs, etc..)
kprop - a cli util that runs periodically on the master node (via systemd timer) propagating changes to the database to the replicas.
kpropd - a daemon running only on all the replicas, able to get updates sent by kprop and apply the to the local database.

Bootstrap a kerberos host

There are two roles defined in puppet for a kerberos node:

1) master

2) replica

The most common operation should be bootstrapping a new replica, but if you are building a cluster from scratch a new master will need to be created as well.

Master node

The master node is the one that holds the authoritative kerberos database containing the credentials for a certain realm. The only realm supported up to now is the WIKIMEDIA one. One you have applied the role::kerberos::kdc to the master node, everything should install/configure nicely but the kerberos daemons shouldn't be able to start correctly due to the absence of a properly bootstrapped database. In order to create a database, execute the following on the master node:

sudo kdb5_util create -s -r WIKIMEDIA

You will have to enter a password. The one of the WIKIMEDIA realm is stored in pwstore.

At this point you should be able to start the kdc and kadmind daemons.

Replica node

The replica node needs to be bootstrapped as the master node, but it also need something more to allow replication with kprop/kpropd to work. Everything should already be deployed by puppet, but there are some manual steps needed:

log on the master node (not the replica, it is important) and create host principals for the replica. For example:

Generate principal and keytab for the replica, and create the keytab in a tmp location

elukey@krb1001:~$ sudo kadmin.local
Authenticating as principal host/admin@WIKIMEDIA with password.
kadmin.local: addprinc -randkey host/krb2001.codfw.wmnet@WIKIMEDIA
kadmin.local: ktadd -k /tmp/krb2001.keytab host/krb2001.codfw.wmnet@WIKIMEDIA

Stop krb5-kdc and krb5-admin-server on the replica host
Copy the /tmp/krb2001.keytab keytab to the replica node, and rename it /etc/krb5.keytab
Restart the daemons

At this point it should be possible to execute the replicate-krb-database.service on the master node to see if database replication works correctly. The service is a very basic script that dumps the current status of the Kerberos database and sends it to each replica listed in hiera via kprop. On Every replica the kpropd daemon should be listening on port 754 to get updates from the master (puppet should take care of setting up kpropd with its related acls).

Good docs to use as reference

https://web.mit.edu/kerberos/krb5-devel/doc/admin/install_kdc.html

Backups

There are two main workflows to save the Kerberos database data:

replicate-krb-database (systemd timer) that periodically dumps the database and replicates to all the replicas via kprop.
backup-kdc-database (systemd timer), running on all nodes, that periodically takes a snapshot of the local database and save it under /srv.

Why do we need two? If the master database gets corrupted/inconsistent and replicated to all the replicas, then we end up having the wrong data everywhere. Having a local list of database snapshots (especially on the master) should help in having one good version of the database to restore.

The master db snapshots are also saved periodically in Bacula.

Create principals and keytabs

In the Kerberos world, every user is called 'principal' and it may be related to a real human user or a service. In the former case, the user simply kinit, input a password and receive a TGT (that will be used later on to get service access etc..). In the latter, there is a convenient way to store the password to allow a transparent service bootstrap without any manual password input by the operator, namely the keytab.

Create a principal for a real user

In this case, the workflow is to generate the principal with a temporary password, and then allow the user to change it as soon as possible. This is currently possible logging on the current kadmin primary (see Hiera kerberos_kadmin_server_primar or just log in in one of the krb nodes, there is a bit motd that explains what servers should not be used) and using the command manage_principals.py like the following:

sudo manage_principals.py get elukey
sudo manage_principals.py create elukey --email_address=ltoscano@wikimedia.org
etc..

The second use case is very interesting, since it handles the workflow described above:

A principal is created via kadmin.local.
A temporary random password is set, together with its expiry in seconds.
An email is sent to the email_address provided, with instructions about how to change the password. It is sufficient to log in on a node that is a kerberos client, run kinit and follow the instructions.

Create a keytab for a service

In order to properly authenticate a service, we need to:

create the service principal (example: hdfs/analytics1028.eqiad.wmnet@WIKIMEDIA)
create the keytab for the principal (that eventually will be stored on the the host to be used by a daemon/service).

There is a script on the krb nodes that can help with this: generate_keytabs.py. It takes as input a file listing, for every row, what to create (principals, keytabs, etc..) and it stores the output (e.g. keytab files) to a known location by default under /srv.

The keytabs are deployed in puppet via the secret module (see profile::kerberos::keytabs) so they need to be copied to puppetmaster1001 in some way. There is a handy rsync server on the krb hosts that offers a module to copy srv only from puppetmaster1001, using user+password (see /srv/kerberos/rsync_secrets_file on the krb hosts). For example, you can do on puppetmaster1001:

mkdir /home/elukey/keytabs
rsync kerb@krb1001.eqiad.wment::srv-keytabs ./keytabs
chown -R gitpuppet:gitpuppet ./keytabs
sudo cp -a ./keytabs/* /srv/private/modules/secret/secrets/kerberos/keytabs/
rm -rf /home/elukey/keytabs

Another option is of course to rsync directly to the right location in the private repo and chown -R gitpuppet:gitpuppet.

List of principals and their keytabs for Hadoop

The following configuration files are the ones used for the Hadoop test cluster. They are meant to be used with the generate_keytabs.py script.

Hadoop Master node

analytics1028.eqiad.wmnet,create_princ,HTTP
analytics1028.eqiad.wmnet,create_princ,hdfs
analytics1028.eqiad.wmnet,create_princ,yarn
analytics1028.eqiad.wmnet,create_princ,mapred

analytics1028.eqiad.wmnet,create_keytab,HTTP
analytics1028.eqiad.wmnet,create_keytab,hdfs
analytics1028.eqiad.wmnet,create_keytab,yarn
analytics1028.eqiad.wmnet,create_keytab,mapred

Hadoop Master (standby) node

analytics1029.eqiad.wmnet,create_princ,HTTP
analytics1029.eqiad.wmnet,create_princ,hdfs
analytics1029.eqiad.wmnet,create_princ,yarn
analytics1029.eqiad.wmnet,create_princ,mapred

analytics1029.eqiad.wmnet,create_keytab,HTTP
analytics1029.eqiad.wmnet,create_keytab,hdfs
analytics1029.eqiad.wmnet,create_keytab,yarn
analytics1029.eqiad.wmnet,create_keytab,mapred

Hadoop worker node

analytics1031.eqiad.wmnet,create_princ,HTTP
analytics1031.eqiad.wmnet,create_princ,hdfs
analytics1031.eqiad.wmnet,create_princ,yarn

analytics1031.eqiad.wmnet,create_keytab,yarn
analytics1031.eqiad.wmnet,create_keytab,hdfs
analytics1031.eqiad.wmnet,create_keytab,HTTP

Hadoop coordinator (please note the special need for Oozie to have a single keytab for two principals)

analytics1030.eqiad.wmnet,create_princ,HTTP
analytics1030.eqiad.wmnet,create_princ,hdfs
analytics1030.eqiad.wmnet,create_princ,yarn
analytics1030.eqiad.wmnet,create_princ,hive
analytics1030.eqiad.wmnet,create_princ,oozie
analytics1030.eqiad.wmnet,create_princ,HTTP-oozie
analytics1030.eqiad.wmnet,create_princ,analytics

analytics1030.eqiad.wmnet,create_keytab,HTTP
analytics1030.eqiad.wmnet,create_keytab,hdfs
analytics1030.eqiad.wmnet,create_keytab,yarn
analytics1030.eqiad.wmnet,create_keytab,hive
analytics1030.eqiad.wmnet,create_keytab,oozie
analytics1030.eqiad.wmnet,create_keytab,HTTP-oozie
analytics1030.eqiad.wmnet,merge_keytab,HTTP-oozie,oozie
analytics1030.eqiad.wmnet,create_keytab,analytics

Druid worker (only kerberos credentials to access HDFS, not to authenticate Druid's clients)

analytics1041.eqiad.wmnet,create_princ,druid

analytics1041.eqiad.wmnet,create_keytab,druid

Hadoop UIs

analytics1039.eqiad.wmnet,create_princ,HTTP
analytics1039.eqiad.wmnet,create_princ,hue

analytics1039.eqiad.wmnet,create_keytab,HTTP
analytics1039.eqiad.wmnet,create_keytab,hue