The root CA is an offline signing CA. The root key is only stored on the root ca server and in (TODO: some offline location). When creating new intermediate certificates one will need to manully generate the on the root ca server and then copy the certificate content into puppet.
Creating an intermediate certificate is mostly a manual process however we do make use of puppet to ensure the certificate is created with the correct parameters
debmonitor: Used for connections between debmonitor clients and the internal debmonitor client auth projected daemon.
discovery: Used for connections between ATS and back-end origin servers.
kafka: Used by Kafka brokers to secure connections between clients (producers/consumers) and them, and to authenticate each other when starting up.
etcd: Used by etcd cluster members (peers) to authenticate each other and to secure etcd traffic between peers and clients. Currently only in use for one etcd cluster.
Adding a new intermediate
To add a new certificate we first neeed to add an entry to the
profile::pki::root_ca::intermediates: array in hiera for the root CA server.
profile::pki::root_ca::intermediates: - debmonitor - discovery - kafka + - test
Once you have done this you will need to run puppet and the puppet root ca server (like
pki-root1001.eqiad.wmnet) to create the public and private key pair. They should be created in
/etc/cfssl/ssl/$intermediate_name and you should also see the puppet output.
$ sudo puppet agent -t Info: Using configured environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Retrieving locales Info: Loading facts Info: Caching catalog for pki-root.pki.eqiad1.wikimedia.cloud Info: Applying configuration version '(e739fdec88) Jbond - P:pki::root_ca: use correct title for intermediate certs' Notice: The LDAP client stack for this host is: sssd/sudo Notice: /Stage[main]/Profile::Ldap::Client::Labs/Notify[LDAP client stack]/message: defined 'message' as 'The LDAP client stack for this host is: sssd/sudo' Notice: /Stage[main]/Profile::Pki::Root_ca/Cfssl::Cert[WMF_test_intermediate_ca]/File[/etc/cfssl/ssl/WMF_test_intermediate_ca]/ensure: created (corrective) Notice: /Stage[main]/Profile::Pki::Root_ca/Cfssl::Cert[WMF_test_intermediate_ca]/Exec[Generate cert WMF_test_intermediate_ca]/returns: executed successfully (corrective) Notice: /Stage[main]/Profile::Pki::Root_ca/Cfssl::Cert[WMF_test_intermediate_ca]/File[/etc/cfssl/ssl/WMF_test_intermediate_ca/WMF_test_intermediate_ca.pem]/mode: mode changed '0644' to '0440' (corrective) Notice: /Stage[main]/Profile::Pki::Root_ca/Cfssl::Cert[WMF_test_intermediate_ca]/File[/etc/cfssl/ssl/WMF_test_intermediate_ca/WMF_test_intermediate_ca-key.pem]/mode: mode changed '0600' to '0440' (corrective) Notice: /Stage[main]/Profile::Pki::Root_ca/Cfssl::Cert[WMF_test_intermediate_ca]/File[/etc/cfssl/ssl/WMF_test_intermediate_ca/WMF_test_intermediate_ca.csr]/mode: mode changed '0644' to '0440' (corrective) Notice: Applied catalog in 4.94 seconds
Once you have generated the new certificate you will need to:
1) Copy the certificate file to the public puppet repo (
2) Copy the private key to the secret store (currently puppet private repo
3) Add an entry under the
profile::pki::multirootca::intermediates: key. Note the location of the certificate and key must go in the specified paths for puppet to find them e.g.
profile::pki::multirootca::intermediates: test: ocsp_port: 10001 # The following parameters are optional and override the defaults nets: -192.0.2.0/24 auth_keys: default_auth: key: $custom_key type: standard profile: database: expiry: 8760h usages: - 'digital signature' - 'key encipherment' - 'server auth' - 'client auth'
Renewing a new intermediate
The root pki root server will automaticity renew intermediated certificates it manages and resign the public key on disk on the root CA server however it will be required to upload the certificate to the puppet repo e.g.
profile/pki/intermediates/$CA.pem. Further at the time of writing certificate bundles are only ever fetched once by puppet. As such you will need to run a cumin command to delete the current intermediate certificates and have puppet re-download the new one.
$ cumin 'R:cfssl::cert%label = $CA' 'rm /etc/cfssl/$CA_chain.pem'
This is not ideal and is an area we shuold improve. some possible solutions
- use puppet to fetch the resource via http directly. This would rely on the `last-modified` header so would automaticity update the bundles but would also cause a lot of unnecessary head requests to check this value on every puppet run
- make the re-querement to store the public intermidate cert in profile/pki/intermediates/ a MUST. This would allow other modules to pull the intermediate directly from puppet but ties the puppet code to WMF specifics
- Look to add bundle support to the genkey and sign commands https://github.com/cloudflare/cfssl/issues/779. Upstream doesn't seem to have moved on this so it may mean maintaining a patch
- reimplement the genkey and sign commands using the api directly, which does support the bundle option. Requires maintaining our own puppet provider/type
A signing profile is a way of specfying particular options that are to be used when generating certificates from a particular intermediate CA. These commonly include certificate duration and a specific auth_key but may also include a number of other parameters that are set out here.
Adding a new signing profile
There are three steps required in order to create a new signing profile.
- Add a new entry for the signing profile to the
profile::pki::multirootca::intermediateshash in puppet. This is in the file
hieradata/role/common/pki/multirootca.yaml. It should specify at least the
auth_keyvalue, which is a reference to the actual key in the private repo.
- Add a new auth key for this signing profile to the
profile::pki::multirootca::default_auth_keys:hash in the private repo. This is in the file
hieradata/role/common/pki/multirootca.yaml. A suitable value can be generated with
openssl rand -hex 8.
- Add a dummy value of the same to the
labs/privaterepo to allow the puppet compiler to refer to it too.
If you are required to revoke a certificate you can use the
cfssl-certs revoke command which expects the public certificate file as input
$ cfssl-certs -vvvv revoke -R cessationOfOperation /etc/cfssl/signers/debmonitor_discovery_wmnet/ca/debmonitor_discovery_wmnet.pem DEBUG:root:running: /usr/bin/cfssl revoke -db-config /etc/cfssl/db.conf -serial 436540461805550888668031556404920484612797891140 -aki 3bada271e634bd1bfc80bf35718391d0ef691336 -reason cessationOfOperation DEBUG:root:b''
OCSP responses are generated on the multicaroot server however the OCSP signing certificate is created on the Root CA server in a similar way to the the intermediates i.e. it is generated using the cfssl::cert resource and files are maintained on the root ca server. however you will need to copy the certificate to the puppet repo (
modules/profile/files/pki/ROOT/$CN.pem) and secret store (
pki/ROOT/$CN.pem). This is only an issue when boot strapping or if you need to extend the life of the ocsp certificate
When installing a new root CA or in the unfortunate event of having to revoke and replace the current root CA one must follow a number of boot strapping steps to create the initial root ca key. To this end, setting
profile::pki::root_ca::bootstrap: true will instruct the root CA host to initialize the keypair. At the next puppet run you'll find
/etc/cfssl/signers/<CA NAME>/ca/ .
Once you have the key you will need to upload the public part (
ca.pem) of the key to the puppet/operations repo recommended to place in
modules/profile/files/pki/ROOT/$CN.pem and the private