From Wikitech
Jump to navigation Jump to search

Why Kerberos?

We use it mostly for Hadoop, since it is the only reliable way to add proper authentication for users/daemons/etc..

Current setup

We have two hosts:

  • krb1001.eqiad.wmnet
  • krb2001.codfw.wmnet

Both hosts are running a kdc and a kadmind daemon, meanwhile only one kpropd. In puppet we set up the master and its replica, usually 1001 and 2001 respectively.

Daemons and their roles

  • KDC (Key Distribution Centre) - a daemon that authenticate a principal returning a TGT (Ticket Granting Ticket) and acting as TGS (Ticket Granting Service)
  • kadmin - a daemon that offers a cli to modify/read the Kerberos database (add principals, generate keytabs, etc..)
  • kprop - a cli util that runs periodically on the master node (via systemd timer) propagating changes to the database to the replicas.
  • kpropd - a daemon running only on all the replicas, able to get updates sent by kprop and apply the to the local database.

Bootstrap a kerberos host

There are two roles defined in puppet for a kerberos node:

1) master

2) replica

The most common operation should be bootstrapping a new replica, but if you are building a cluster from scratch a new master will need to be created as well.

Master node

The master node is the one that holds the authoritative kerberos database containing the credentials for a certain realm. The only realm supported up to now is the WIKIMEDIA one. One you have applied the role::kerberos::kdc to the master node, everything should install/configure nicely but the kerberos daemons shouldn't be able to start correctly due to the absence of a properly bootstrapped database. In order to create a database, execute the following on the master node:

sudo kdb5_util create -s -r WIKIMEDIA

You will have to enter a password. The one of the WIKIMEDIA realm is stored in pwstore.

At this point you should be able to start the kdc and kadmind daemons.

Replica node

The replica node needs to be bootstrapped as the master node, but it also need something more to allow replication with kprop/kpropd to work. Everything should already be deployed by puppet, but there are some manual steps needed:

  • log on the master node (not the replica, it is important) and create host principals for the replica. For example:
  • Generate principal and keytab for the replica, and create the keytab in a tmp location
    elukey@krb1001:~$ sudo kadmin.local
    Authenticating as principal host/admin@WIKIMEDIA with password.
    kadmin.local: addprinc -randkey host/krb2001.codfw.wmnet@WIKIMEDIA
    kadmin.local: ktadd -k /tmp/krb2001.keytab host/krb2001.codfw.wmnet@WIKIMEDIA
  • Stop krb5-kdc and krb5-admin-server on the replica host
  • Copy the /tmp/krb2001.keytab keytab to the replica node, and rename it /etc/krb5.keytab
  • Restart the daemons

At this point it should be possible to execute the replicate-krb-database.service on the master node to see if database replication works correctly. The service is a very basic script that dumps the current status of the Kerberos database and sends it to each replica listed in hiera via kprop. On Every replica the kpropd daemon should be listening on port 754 to get updates from the master (puppet should take care of setting up kpropd with its related acls).

Good docs to use as reference


There are two main workflows to save the Kerberos database data:

  • replicate-krb-database (systemd timer) that periodically dumps the database and replicates to all the replicas via kprop.
  • backup-kdc-database (systemd timer), running on all nodes, that periodically takes a snapshot of the local database and save it under /srv.

Why do we need two? If the master database gets corrupted/inconsistent and replicated to all the replicas, then we end up having the wrong data everywhere. Having a local list of database snapshots (especially on the master) should help in having one good version of the database to restore.

The master db snapshots are also saved periodically in Bacula.

Create principals and keytabs

In the Kerberos world, every user is called 'principal' and it may be related to a real human user or a service. In the former case, the user simply kinit, input a password and receive a TGT (that will be used later on to get service access etc..). In the latter, there is a convenient way to store the password to allow a transparent service bootstrap without any manual password input by the operator, namely the keytab.

Create a principal for a real user

In this case, the workflow is to generate the principal with a temporary password, and then allow the user to change it as soon as possible. This is currently possible logging on the current kadmin primary (see Hiera kerberos_kadmin_server_primar or just log in in one of the krb nodes, there is a bit motd that explains what servers should not be used) and using the command like the following:

  • sudo get elukey
  • sudo create elukey
  • etc..

The second use case is very interesting, since it handles the workflow described above:

  1. A principal is created via kadmin.local.
  2. A temporary random password is set, together with its expiry in seconds.
  3. An email is sent to the email_address provided, with instructions about how to change the password. It is sufficient to log in on a node that is a kerberos client, run kinit and follow the instructions.

Create a keytab for a service

In order to properly authenticate a service, we need to:

  • create the service principal (example: hdfs/analytics1028.eqiad.wmnet@WIKIMEDIA)
  • create the keytab for the principal (that eventually will be stored on the the host to be used by a daemon/service).

There is a script on the krb nodes that can help with this: It takes as input a file listing, for every row, what to create (principals, keytabs, etc..) and it stores the output (e.g. keytab files) to a known location by default under /srv.

The keytabs are deployed in puppet via the secret module (see profile::kerberos::keytabs) so they need to be copied to puppetmaster1001 in some way. There is a handy rsync server on the krb hosts that offers a module to copy srv only from puppetmaster1001, using user+password (see /srv/kerberos/rsync_secrets_file on the krb hosts). For example, you can do on puppetmaster1001:

mkdir /home/elukey/keytabs
rsync kerb@krb1001.eqiad.wment::srv-keytabs ./keytabs
chown -R gitpuppet:gitpuppet ./keytabs
sudo cp -a ./keytabs/* /srv/private/modules/secret/secrets/kerberos/keytabs/
rm -rf /home/elukey/keytabs

Another option is of course to rsync directly to the right location in the private repo and chown -R gitpuppet:gitpuppet.

List of principals and their keytabs for Hadoop

The following configuration files are the ones used for the Hadoop test cluster. They are meant to be used with the script.

Hadoop Master node



Hadoop Master (standby) node



Hadoop worker node



Hadoop coordinator (please note the special need for Oozie to have a single keytab for two principals)



Druid worker (only kerberos credentials to access HDFS, not to authenticate Druid's clients)



Hadoop UIs