PKI/Clients

We currently use CFSSL to provide and manage PKI solutions. Clients are able to make use of the CFSSL API endpoint available at https://pki.discovery.wmnet/, however this endpoint is protected via TLS Client authentications and currently requires users to use the puppet agent certificate. The puppet agent certificate was chosen as it is something we already issue when a host is provisioned or reimaged; we can change this at a later date or configure additional intermediate CAs which have different authentication mechanisms. In addition to the client auth requirement, API requests also need to be signed with hmac using a secret key (available in the puppet private repo).

Puppet integration

Puppet hosts can start using the PKI infrastructure by adding the profile::pki::client to their role. This will configure the host with the required CFSSL configuration to be able to request new certificates for the configured intermediate certificates.

Creating certificates

When a user wants to create a new certificate, they will first need to decide which intermediate CA they want to use. Note that each realm of operations using client auth should have its own intermediate CA and you should not use cross functional CA's. For example: debmonitor, DB connections, and conftool should all have their own intermediate CA's. If you need to create a new intermediate CA, see the CA Operations Page. Unless the defaults have been changed, all certificates will be issued with an expiry of 672h and set to autorenew 11 days before they expire.

Once you know which intermediate CA you want to use to issue your certificate, you will need to know which profile to use. By default, intermediate certificates are created with two profiles. The default profile is for client auth and the server profile is for daemon services. If you have created additional profiles for your CA, you can use them as below. The below example assumes we want to create two certificates in the "WMF testing CA", one using the default profile and one using the server profile.

profile::pki::get_cert

The simplest interface to for creating certificates is the profile::pki::get_cert function. This function has a similar signature to cfssl::cert, but it also returns the path of the cert, the private key, and the CA bundle (chained). As you most often need to have these paths for other parts of puppet configurations, this is likely the most useful interface.

class profile::foobar () {
    $ssl_paths = profile::pki::get_cert('WMF_testing_CA')
    $config = {
      ssl_cert = $ssl_paths['cert']
      ssl_key = $ssl_paths['key']
    }
    file {'/etc/ssl_config.json':
      ensure => file,
      content = $config.to_json
    }
 }

To also fetch the CA bundle:

profile::pki::get_cert('WMF_testing_CA', $facts['fqdn']) >> {
  'cert'    => '/etc/cfssl/ssl/WMF_testing_CA/WMF_testing_CA_$fqdn.pem',      # owned by root
  'key'     => '/etc/cfssl/ssl/WMF_testing_CA/WMF_testing_CA_$fqdn-key.pem',  # owned by root
  'chain'   => '/etc/cfssl/ssl/WMF_testing_CA/WMF_testing_CA_chain.pem',      # owned by root
  'chained' => '/etc/cfssl/ssl/WMF_testing_CA/WMF_testing_CA_chained.pem',    # owned by root
}

To fetch the CA bundle, changing the outputdir and user:

profile::pki::get_cert(
  'WMF_testing_CA',
  $facts['fqdn'],
  {'outdir' => '/etc/foobar', 'owner' => 'foobar'}
) >> {
  'cert'    => '/etc/foobar/WMF_testing_CA_$fqdn.pem',     # owned by foobar
  'key'     => '/etc/foobar/WMF_testing_CA_$fqdn-key.pem', # owned by foobar
  'chain'   => '/etc/foobar/WMF_testing_CA_chain.pem',     # owned by foobar
  'chained' => '/etc/foobar/WMF_testing_CA_chained.pem',   # owned by foobar
}

If you use the outdir directory, make sure it is declared in puppet, and its owner should have write permissions (e.g. 0700). If the directory is missing, the chained certificate generation will cause the puppet run to fail.

Hiera

We can also add certificates by adding the configuration to hiera via the profile::pki::client::certs. Entries here are passed directly to cfssl::cert and support all parameters for that resource. In the following example we add provide_chain: true:

profile::pki::client::certs:
  '%{facts.networking.fqdn}':
    # label is escaped with regsubst('[^\w\-]', '_', 'G')
    label: WMF_testing_CA
    provide_chain: true
  foobar.example.com:
    label: WMF_test_intermediate_ca
    provide_chain: true
    profile: server

As mentioned above certificates are set with a short TTL and are set to auto expire; as such you may need to auto restart some service when the certificate is renewed. For this you can use the notify_service parameter.

profile::pki::client::certs:
  foobar.example.com:
    label: WMF_test_intermediate_ca
    provide_chain: true
    profile: server
    notify_service: foobar

In the above example, Service['foobar'] will be notified when the certificate is renewed.

Puppet

Sometimes it is required to have more complex relationships or keep resources in specific manifests to ease debugging and readability. As such it is also possible to use the cfssl::cert resource directly.

class profile::foobar () {
  cfssl::cert{ $facts['networking']['fqdn']:
    label         => WMF_testing_CA,
    provide_chain => true,
  }
  cfssl::cert{ 'foobar.example.com':
    label         => WMF_testing_CA,
    provide_chain => true,
    profile       => 'server',
  }
}

Defining the resource in puppet allows for more complex relations as indicated in the following example:

class profile::foobar () {
  exec {'foobar_exec':
    command     => 'run a command every time the client cert is renewed'
    refreshonly => true,
  }
  service {'foobar':
    ensure => 'running',
  }
  cfssl::cert{ $facts['networking']['fqdn']:
    label         => WMF_testing_CA,
    provide_chain => true,
    notify        => Exec['foobar_exec'],
  }
  cfssl::cert{ 'foobar.example.com':
    label          => WMF_testing_CA,
    provide_chain  => true,
    profile        => 'server',
    notify_service => 'foobar',
    # We could also use the following however notify_service is a bit more optimal
    # notify        => Service['foobar'],
  }
  service {'another_foobar':
    ensure => 'running',
    subscribe => Cfssl::Cert['foobar.example.com'],
  }
}

Proxy Signing

The PKI client can start a cfssl api service and proxy the signing request to the main multicaroot instance. The proxy server still needs to have access to a valid puppet agent certificate to use for outbound connections to the multiroot ca server, but it offers an http endpoint locally which does not have these restrictions. This allows production puppet hosts to provide PKI services to non-puppet hosts. The main driver for this was to allow k8s hosts to provide PKI services to their containers over localhost. To enable the proxy services simply add the following configuration to hiera:

profile::pki::client::enable_proxy: true
# By default the proxy service listens on http://127.0.0.1:8888/
# This can be changed with the following settings however one should 
# be very cautious in considering having this listen on anything other then localhost 
profile::pki::client::listen_addr: '127.0.0.1'
profile::pki::client::listen_port: 8888

CLI Interactions

Users are also able interact with CFSSL using the cfssl command line tool. The instructions below assume you are running commands from a puppet-enabled host in the production network. The pki server is protected via TLS mutual auth using the puppet CA; in addition there is an additional secret token held in /etc/cfssl/client-cfssl.conf.

First, create a certificate signing request in json format e.g. crs.json

First, create a certificate signing request in json format (e.g. crs.json):

{
  "hosts": [
    "cumin1001.eqiad.wmnet"
  ],
  "key": {
    "algo": "ecdsa",
    "size": 256
  },
  "names": [
    {
      "organisation": "SnakeOil", 
      "country": "uk"
    }
  ]
}

Once this is in place you can send it to the pki server for a signature with the following command:

$ /usr/bin/cfssl gencert \
  -config /etc/cfssl/client-cfssl.conf \
  -tls-remote-ca $(facter -p puppet_config.localcacert) \
  -mutual-tls-client-cert $(facter -p puppet_config.hostcert) \
  -mutual-tls-client-key $(facter -p puppet_config.hostprivkey) \
  -label ${ca_label} -profile server \  
  csr.json

${ca_label} is the name of the intermediate; see PKI/CA Operations#Current intermediates.

This command will produce a json response with the certificate, key and csr. To automatically parse out json you can pipe the command to cfssljson -bare $dir/$basename e.g.

$ /usr/bin/cfssl gencert \
  -config /etc/cfssl/client-cfssl.conf \
  -tls-remote-ca $(facter -p puppet_config.localcacert) \
  -mutual-tls-client-cert $(facter -p puppet_config.hostcert) \
  -mutual-tls-client-key $(facter -p puppet_config.hostprivkey) \
  -label ${ca_label} -profile server \  
  csr.json | /usr/bin/cfssljson -bare /path/to/certs/basename
$ ls -la /path/to/certs/
drwxr-xr-x  2 root root 4096 Jun 28 12:11 .
drwx------ 11 root root 4096 Jun 28 12:11 ..
-rw-r--r--  1 root root  395 Jun 28 12:11 basename.csr
-rw-------  1 root root  227 Jun 28 12:11 basename-key.pem
-rw-r--r--  1 root root 1054 Jun 28 12:11 basename.pem

Users can also re-sign the certificate by using pretty much the same command, changing gencert to sign and using the current pem certificate instead of the json csr:

$ /usr/bin/cfssl sign \
  -config /etc/cfssl/client-cfssl.conf \
  -tls-remote-ca $(facter -p puppet_config.localcacert) \
  -mutual-tls-client-cert $(facter -p puppet_config.hostcert) \
  -mutual-tls-client-key $(facter -p puppet_config.hostprivkey) \
  -label ${ca_label} -profile server \  
  /path/to/certs/basename.pem| /usr/bin/cfssljson -bare /path/to/certs/basename

Kubernetes Integration

Kubernetes clusters may use a combination of cert-manager and cfssl-issuer to provide certificates for workload inside the cluster.