From Wikitech
Jump to navigation Jump to search

cergen is a Python command line tool and library to help with managing and generating asymmetric key pairs and x509 certificates for use with TLS/SSL encryption and authentication.

The openssl CLI is useful and very featureful, but fairly complicated. The Java keytool CLI serves a similar purpose, but outputs files in a different format than openssl. cergen allows you to declare desired certificates and keys in a YAML manifest, and then idempotently generate them into files of various formats for use with different technologies.

cergen was originally based on cassandra-ca-manager, and expanded to generate more file formats and to integrate with custom CA implementations, including Puppet CA.

cergen is installed on puppetmaster frontends that are also CAs. It is intended to be used there to generate certificates signed by our Puppet CA, and committed to the ops puppet private repository's secret module.

For more information, see the cergen README.

Usage on Puppet CA host

cergen production certificate manifest files are checked into Puppet private. On puppetmaster1001, these can be found at /srv/private/modules/secret/secrets/certificates/certificate.manifests.d/. *.certs.yaml files in this directory declare certificates and keys that should be generated and managed, as well as any CAs that will be used to sign those certificates.

Most likely, you'll be using the Puppet CA to sign your certificates. The puppet_ca signer is declared in puppet_ca.certs.yaml:

# This is our Puppet CA.  It will be used to sign other certificates.
  class_name: puppet
  hostname: puppetmaster1001.eqiad.wmnet

Here puppet_ca is the name of the authority. You can refer to puppet_ca as the authority for the certificates you want to generate and have signed by our Puppet CA. In kafka_jumbo.certs.yaml, we declare a certificate for use by Kafka brokers in the jumbo-eqiad cluster:

# Certificates for the kafka-jumbo cluster.
  authority: puppet_ca
    password: 'XXXXXXXX'
    algorithm: ec

With the CA and certificates declared, we can now use cergen to check the status of our certificates. We keep generated files in /srv/private/modules/secret/secrets/certificates, so we need to provide that as the --base-path= option.

$ cergen --base-path=/srv/private/modules/secret/secrets/certificates /srv/private/modules/secret/secrets/certificates/certificate.manifests.d

Status of certificates ['kafka_jumbo-eqiad_broker']

Certificate(kafka_jumbo-eqiad_broker, authorities=[PuppetCA(puppetmaster1001.eqiad.wmnet_8140)]):
	/srv/private/modules/secret/secrets/certificates/kafka_jumbo-eqiad_broker/kafka_jumbo-eqiad_broker.key.private.pem: PRESENT (mtime: 2017-12-05T14:47:45.105453)
	/srv/private/modules/secret/secrets/certificates/kafka_jumbo-eqiad_broker/kafka_jumbo-eqiad_broker.key.public.pem: PRESENT (mtime: 2017-12-05T14:47:45.105453)
	/srv/private/modules/secret/secrets/certificates/kafka_jumbo-eqiad_broker/kafka_jumbo-eqiad_broker.crt.pem: PRESENT (mtime: 2017-12-05T14:47:46.417451)
	/srv/private/modules/secret/secrets/certificates/kafka_jumbo-eqiad_broker/kafka_jumbo-eqiad_broker.keystore.p12: PRESENT (mtime: 2017-12-05T14:47:46.433451)
	/srv/private/modules/secret/secrets/certificates/kafka_jumbo-eqiad_broker/kafka_jumbo-eqiad_broker.keystore.jks: PRESENT (mtime: 2017-12-05T14:47:47.261450)
	/srv/private/modules/secret/secrets/certificates/kafka_jumbo-eqiad_broker/truststore.jks: PRESENT (mtime: 2017-12-05T14:47:47.649449)

Without the --generate flag, cergen will just print out a list of certificates it is managing, and the status of the expected files. By providing --generate, cergen will attempt to generate any files for certificates declared in its manifests that are not PRESENT. For these certificates, the key.private.pem and the crt.pem files are considered the canonical sources of data. The other files are subordinate to those. E.g., if the keystore.jks file is missing, it will be re-generated from the key.private.pem and crt.pem files. However, if either crt.pem or key.private.pem is missing, you will need to manually generate a brand new key and certificate by removing the files and running again with --generate.

NOTE: If you are regenerating a Puppet signed certificate, you must first remove the certificate from the Puppet CA. puppet cert clean <common_name> should do it.

Certificates that declare authority: puppet_ca will be auto-signed by our Puppet CA. If you need to remove a Puppet signed cert, remember to also destroy that cert using the puppet cert CLI, e.g. puppet cert destroy kafka_jumbo-eqiad_broker.

To avoid messing with certificates you are not concerned with, you can instruct cergen to only select specific certificates by using the -c flag. E.g.

cergen --generate -c 'kafka.*' ... might select all certificates who's names start with 'kafka'. See cergen --helpfor more options.


On the puppetmaster (eg: puppetmaster1001.eqiad.wmnet), create /srv/private/modules/secret/secrets/certificates/certificate.manifests.d/SERVICENAME.certs.yaml as follows:

  authority: puppet_ca
  expiry: null
  alt_names: ["SERVICENAME.svc.eqiad.wmnet","*.whatever.org", ...]
    algorithm: ec

Note: the service name / common name (i.e. SERVICENAME.discovery.wmnet) will be implicitly included in alt_names, no need to duplicate it there as well.

By setting a key.password, cergen will output encrypted private key files. If you need an unencrypted private key file you can omit the key.password and skip the openssl command below.

cergen -c 'SERVICENAME.*' --generate --base-path=/srv/private/modules/secret/secrets/certificates /srv/private/modules/secret/secrets/certificates/certificate.manifests.d

# For certificates with key.password, run:
openssl ec -in modules/secret/secrets/certificates/SERVICENAME.discovery.wmnet/SERVICENAME.discovery.wmnet.key.private.pem -out /srv/private/modules/secret/secrets/ssl/SERVICENAME.discovery.wmnet.key
# openssl will ask for YOURSECRETPASSWORD at this point

# For certificates without key.password, just copy the private key over:
cp ./modules/secret/secrets/certificates/SERVICENAME.discovery.wmnet/SERVICENAME.discovery.wmnet.key.private.pem /srv/private/modules/secret/secrets/ssl/SERVICENAME.discovery.wmnet.key
git add /modules/secret/secrets/ssl/SERVICENAME.discovery.wmnet.key

If everything went well, the public key is now under /srv/private/modules/secret/secrets/certificates/SERVICENAME.discovery.wmnet/SERVICENAME.discovery.wmnet.crt.pem. If your service is using sslcert::certificate with "use_cergen" then there's nothing else to do other than commit your changes to /srv/private.

Otherwise copy the public key to the operations/puppet repository under files/ssl/SERVICENAME.discovery.wmnet.crt Notice the extension difference! The extension is .crt.pem on the puppetmaster, but it needs to be just .crt in operations/puppet.

The private key is /srv/private/modules/secret/secrets/ssl/SERVICENAME.discovery.wmnet.key. There's nothing else to do other than commit your changes to /srv/private.

In order for pcc to work properly, a dummy secret key (NOT the real one) needs to be added to the labs/private repository under modules/secret/secrets/ssl/SERVICENAME.discovery.wmnet.key. Please note that the labs/private repository is NOT private as the name might suggest, it is public. The key you add should look like this:


See https://gerrit.wikimedia.org/r/#/c/labs/private/+/523929/ as an example.

Update a certificate

  • Update the YAML file /srv/private/modules/secret/secrets/certificates/certificate.manifests.d/SERVICENAME.certs.yaml
  • Remove the old certificate from the Puppet CA
  puppet cert clean <common_name>
  • Remove the following files from /srv/private/modules/secret/secrets/certificates/SERVICENAME.discovery.wmnet/:

SERVICENAME.discovery.wmnet.crt.pem SERVICENAME.discovery.wmnet.csr.pem SERVICENAME.discovery.wmnet.keystore.jks SERVICENAME.discovery.wmnet.keystore.p12 truststore.jks

  • Update the certificate with:

cergen -c 'SERVICENAME.*' --generate --base-path=/srv/private/modules/secret/secrets/certificates /srv/private/modules/secret/secrets/certificates/certificate.manifests.d

  • The new cert should be under /srv/private/modules/secret/secrets/certificates/SERVICENAME.discovery.wmnet/SERVICENAME.discovery.wmnet.crt.pem.
  • Add the updated certificate to puppet under modules/profile/files/ssl/SERVICENAME.discovery.wmnet.crt.

Distribution via Puppet

Once you've generated your certificate and key files using cergen, you should commit them, as well as any changes you've made to certificate.manifests.d/. Then you can use the puppet secret template function to render the contents of the files into locations for use by your production service.

file { "/path/to/my/my_certificate_keystore.jks":
    content => secret('certificates/my_certificate/my_certificate.keystore.jks'),

See also the profile::kafka::broker class for another example.

Future work

It would be very convenient if the manual steps of declaring a certificate, generating it, and distributing the desired files were all abstracted away into a single handy puppet define. This might be possible, but is complicated. It might look something like this:

cergen::certificate { 'kafka_jumbo-eqiad_broker':
    destination => '/etc/kafka/ssl',
    properties  => {
      'authority' => 'puppet_ca',
      'key' => {
          'algorithm' => 'rsa',
          'password'  => hiera('passwords::blabla'),

This would allow us to declare a certificate in the class/node/location where it should be deployed. This is easier said than done, as Puppet signed certificates must be created and signed on the same host as the Puppet CA (you can't just use the Puppet CA HTTP API). Creating a define like above would require

  • declare the certificate on the host
  • Using exported resources, realize the rendered cergen manifest.yaml to the Puppet CA host (e.g. puppetmaster1001)
  • Use the generate() puppet function to run the cergen command on the puppetmaster and commit the resulting certificate files public Puppet, and private key files to the puppet private repo.
  • Pull the certificate files using source and key files from puppet private using the secret() function.

I wrote a PoC to do this, but it is totally untested and probably would not work as is.

The original Phabricator ticket tracking this work was T166167.