Jump to content

PKI/CA Operations

From Wikitech
< PKI
(Redirected from PKI/root ca)

The root CA is an offline signing CA. The root key is only stored on the root ca server and in (TODO: some offline location). When creating new intermediate certificates one will need to manually generate the new intermediate certificate on the root ca server and then copy the certificate content into puppet.

Intermediate Certificates

Creating an intermediate certificate is mostly a manual process however we do make use of puppet to ensure the certificate is created with the correct parameters

Adding a new intermediate

To add a new certificate we first neeed to add an entry to the profile::pki::root_ca::intermediates: array in hiera for the root CA server. 

profile::pki::root_ca::intermediates:
  - debmonitor
  - discovery
  - kafka
+ - test

Once you have done this you will need to run puppet and the puppet root ca server (like pki-root1001.eqiad.wmnet) to create the public and private key pair. They should be created in /etc/cfssl/ssl/$intermediate_name and you should also see the puppet output. 

$ sudo puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for pki-root.pki.eqiad1.wikimedia.cloud
Info: Applying configuration version '(e739fdec88) Jbond - P:pki::root_ca: use correct title for intermediate certs'
Notice: The LDAP client stack for this host is: sssd/sudo
Notice: /Stage[main]/Profile::Ldap::Client::Labs/Notify[LDAP client stack]/message: defined 'message' as 'The LDAP client stack for this host is: sssd/sudo'
Notice: /Stage[main]/Profile::Pki::Root_ca/Cfssl::Cert[WMF_test_intermediate_ca]/File[/etc/cfssl/ssl/WMF_test_intermediate_ca]/ensure: created (corrective)
Notice: /Stage[main]/Profile::Pki::Root_ca/Cfssl::Cert[WMF_test_intermediate_ca]/Exec[Generate cert WMF_test_intermediate_ca]/returns: executed successfully (corrective)
Notice: /Stage[main]/Profile::Pki::Root_ca/Cfssl::Cert[WMF_test_intermediate_ca]/File[/etc/cfssl/ssl/WMF_test_intermediate_ca/WMF_test_intermediate_ca.pem]/mode: mode changed '0644' to '0440' (corrective)
Notice: /Stage[main]/Profile::Pki::Root_ca/Cfssl::Cert[WMF_test_intermediate_ca]/File[/etc/cfssl/ssl/WMF_test_intermediate_ca/WMF_test_intermediate_ca-key.pem]/mode: mode changed '0600' to '0440' (corrective)
Notice: /Stage[main]/Profile::Pki::Root_ca/Cfssl::Cert[WMF_test_intermediate_ca]/File[/etc/cfssl/ssl/WMF_test_intermediate_ca/WMF_test_intermediate_ca.csr]/mode: mode changed '0644' to '0440' (corrective)
Notice: Applied catalog in 4.94 seconds

Once you have generated the new certificate you will need to:

1) Copy the certificate file to the public puppet repo (modules/profile/files/pki/intermediates/$intermediate_name.pem).

2) Copy the private key to the secret store (currently puppet private repo modules/secret/secrets/pki/intermediates/$intermediate_name-key.pem). For example: 

(export INTERMEDIATE=aux ; sudo SSH_AUTH_SOCK=/run/keyholder/proxy.sock \
    scp -3 pki-root1001.eqiad.wmnet:/etc/cfssl/ssl/${INTERMEDIATE}/${INTERMEDIATE}-key.pem \
           puppetmaster1001.eqiad.wmnet:/srv/private/modules/secret/secrets/pki/intermediates)

3) Add a dummy private key to the labs/private repo (echo nosecret > modules/secret/secrets/pki/intermediates/$intermediate_name-key.pem).

4) Add an entry under the profile::pki::multirootca::intermediates: key. Note the location of the certificate and key must go in the specified paths for puppet to find them e.g. 

profile::pki::multirootca::intermediates:
  test:
    ocsp_port: 10001
    # The following parameters are optional and override the defaults
    nets:
      -192.0.2.0/24
    auth_keys:
      default_auth:
        key: $custom_key
        type: standard
    profile:
      database:
        expiry: 8760h
        usages:
          - 'digital signature'
          - 'key encipherment'
          - 'server auth'
          - 'client auth'

Renewing a new intermediate

The root pki root server will automatically renew intermediate certificates it manages and resign the public key on disk on the root CA server however it will be required to upload the certificate to the puppet repo e.g. profile/pki/intermediates/$CA.pem. Further at the time of writing certificate bundles are only ever fetched once by puppet. As such you will need to run a cumin command to delete the current intermediate certificates and have puppet re-download the new one. 

$ cumin 'R:cfssl::cert%label = $CA' 'rm /etc/cfssl/$CA_chain.pem'

This is not ideal and is an area we should improve. Some possible solutions:

  • use puppet to fetch the resource via http directly. This would rely on the `last-modified` header so it would automatically update the bundles but would also cause a lot of unnecessary head requests to check this value on every puppet run
  • make it mandatory to store the public intermediate cert in profile/pki/intermediates/. This would allow other modules to pull the intermediate certificates directly from puppet but ties the puppet code to WMF specifics
  • Look to add bundle support to the genkey and sign commands https://github.com/cloudflare/cfssl/issues/779. Upstream doesn't seem to have moved on this so it may mean maintaining a patch
  • reimplement the genkey and sign commands using the api directly, which does support the bundle option. Requires maintaining our own puppet provider/type

Signing Profiles

A signing profile is a way of specifying particular options that are to be used when generating certificates from a particular intermediate CA. These commonly include certificate duration and a specific auth_key but may also include a number of other parameters that are documented on Cloudflare's github.

Adding a new signing profile

There are three steps required in order to create a new signing profile.

  • Add a new entry for the signing profile to the profile::pki::multirootca::intermediates hash in puppet. This is in the file hieradata/role/common/pki/multirootca.yaml . It should specify at least the auth_key value, which is a reference to the actual key in the private repo.
  • Add a new auth key for this signing profile to the profile::pki::multirootca::default_auth_keys: hash in the private repo. This is in the file hieradata/role/common/pki/multirootca.yaml. A suitable value can be generated with openssl rand -hex 8.
  • Add a dummy value of the same to the labs/private repo to allow the puppet compiler to refer to it too.

Revoking certificates

If you are required to revoke a certificate you can use the cfssl-certs revoke command which expects the public certificate file as input 

$ cfssl-certs -vvvv revoke -R cessationOfOperation /etc/cfssl/signers/debmonitor_discovery_wmnet/ca/debmonitor_discovery_wmnet.pem
DEBUG:root:running: /usr/bin/cfssl revoke -db-config /etc/cfssl/db.conf -serial 436540461805550888668031556404920484612797891140 -aki 3bada271e634bd1bfc80bf35718391d0ef691336 -reason cessationOfOperation
DEBUG:root:b''

OCSP

OCSP responses are generated on the multicaroot server however the OCSP signing certificate is created on the Root CA server in a similar way to the the intermediates i.e. it is generated using the cfssl::cert resource and files are maintained on the root ca server. however you will need to copy the certificate to the puppet repo (modules/profile/files/pki/ROOT/$CN.pem) and secret store (pki/ROOT/$CN.pem). This is only an issue when boot strapping or if you need to extend the life of the ocsp certificate


New server/HW refresh

When adding a new server it should be as easy as giving it the pki::multirootca role and imaging it. However the IP addresses of the pki servers are hardcoded in Kubernetes network policies (firewall rules), so it needs some co-ordination with the various k8s clusters owners. See https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/898759 for an example change adding pki2002.

Alt DNS names

At the time of writing the pki::multirootca use the puppet agent certificates to provide authentication. As the pki service listens on pki.discovery.wmnet we make use of the puppet dns_alt_names configuration. This can cause problems if rebuilding the server as this options is not currently supported by the reimage scripts. As such it is recommended to follow the following steps when (re)building

  • first move the host into the spare::system role this allows you to use the reimage scripts to rebuild the host and prevents it getting stuck
  • once up move the host into the pki::multirootca role and run puppet. you should see a change like the following (although puppet will fail)
--- /etc/puppet/puppet.conf.d/10-main.conf      2021-03-25 11:19:13.680926176 +0000
+++ /tmp/puppet-file20210330-30308-1y7a7nd      2021-03-30 11:46:43.552449866 +0000
@@ -14,7 +14,7 @@
 server = puppet
 
 ca_server = puppetmaster1001.eqiad.wmnet
-
+dns_alt_names = pki.discovery.wmnet
 daemonize = false
 http_connect_timeout = 60
 http_read_timeout = 960
  • once this is in-place run the sre.puppet.renew-cert cookbook to regenerate the new cert
$ sudo  cookbook sre.puppet.renew-cert --allow-alt-names  pki1001.eqiad.wmnet
  • finally run puppet on the pki servers

Boot strapping

When installing a new root CA or in the unfortunate event of having to revoke and replace the current root CA one must follow a number of boot strapping steps to create the initial root ca key. To this end, setting profile::pki::root_ca::bootstrap: true will instruct the root CA host to initialize the keypair. At the next puppet run you'll find ca.pem and ca-key.pem in /etc/cfssl/signers/<CA NAME>/ca/ .

Once you have the key you will need to upload the public part (ca.pem) of the key to the puppet/operations repo recommended to place in modules/profile/files/pki/ROOT/$CN.pem and the private

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=PKI/CA_Operations&oldid=2138101"