The cloud puppet masters have a tiered puppetmaster set up. At the top level there are the cloudinfra-internal-puppetmaster puppetmasters. Theses provide a puppet environment which is used for the supporting infrastructure used by the reset of cloud. including acting as the puppet masters for the cloud not puppetmatsers. As such we will refer to this infrastructure as the master of masters (MoMs).

Puppet MoMs

Theses servers should be configured in a similar manager to the stand-alone puppet masters
All servers in the cloudinfra project including the MoM's should configure the primary MoM puppetmaster as the puppetmaster: and puppet_ca_server: in the projects hierarchy
The MoMs are clients of them self this means that the SSL key materiel in /var/lib/puppet/ssl will be signed by the key material in /var/lib/puppet/server/ssl

Cloud puppet masters

Theses servers are configured in a similar manner to the production puppet masters however unlike the production puppet masters which are clients of its own CA. The cloud puppet masters are clients of the MoMs. Theses servers exists in the cloudinfra project so should already be configured to use the MoMs as there puppetmaster and CA.

SSL

The big difference the cloud puppet masters have over both the production puppet masters and the MoMs is that the have two different SSL CA's in play. This is because they are puppet agents to the MoM's (first CA) and the puppet CA to the cloud agents (second CA). All puppet masters are both agents and CA's however they are normally an agent of there own CA, this is the major thing that differentiates theses puppet masters.

This means that:

key material in /var/lib/puppet/ssl is only valid for talking to the MoM infrastructure
key material in /var/lib/puppet/server/ssl is only valid for talking to the cloud agents
key material in theses two directories should be different and should not be able to verify certs from each other

Agent operations

When agents talk to the puppet master they talk to what ever is configured as the server and ca_server options in /etc/puppet/puppet.conf. Which in turn are controlled via the puppetmaster: and puppet_ca_server: hiera keys. This should ultimately resolve to a web site configuered on the puppet masters which:

performs client auth using apache
load balances the connection across all configured puppetmasters
from the backend vhost: proxies connections to the passanger application

If the agent has SSL issue connecting with the puppet master you can use curl to debug any client auth issues with the following command. Im not sure why this gives A HTTP 500 however if things are misconfigured it will give a 403. Further you can drop the --head option to dump the entire catalogue in yaml

$ sudo curl -H 'Accept: yaml'  --tlsv1 --cacert /var/lib/puppet/ssl/certs/ca.pem --cert /var/lib/puppet/ssl/certs/$(hostname -f).pem --key /var/lib/puppet/ssl/private_keys/$(hostname -f).pem --head https://$(sudo facter -p puppet_server):8140/production/catalog/$(hostname -f)
HTTP/1.1 500 Internal Server Error
Date: Tue, 17 Nov 2020 10:35:18 GMT
Server: Apache
X-Puppet-Version: 5.5.10
X-Powered-By: Phusion Passenger 5.0.30
Content-Length: 142
Status: 500 Internal Server Error
Backend-Timing: D=24852 t=1605609318492550
Content-Type: application/json; charset=utf-8
Connection: close

If this don't work the first thing i would try is to delete the agent certificate and regenerate it. If this dosn;t work then the puppet master probably has its SSL misconfigured.

puppet master vhost validation

To validate the SSL configuration on the vhosts you need to make sure that all of the client certificates used in the vhost configueration are associated with the correct CA certificate. A quick way to do this is with the following script

#!/bin/bash
base_dir=/etc/apache2/sites-enabled
expected_ca_file=/var/lib/puppet/server/ssl/certs/ca.pem
program_exit=0
for vhost in "${base_dir}"/*
do
        cert_file=$(awk '$1=="SSLCertificateFile" {print $2}' "${vhost}")
        key_file=$(awk '$1=="SSLCertificateKeyFile" {print $2}' "${vhost}")
        ca_file=$(awk '$1=="SSLCACertificateFile" {print $2}' "${vhost}")
        if [[ -z ${cert_file} ]] || [[ -z ${key_file} ]] || [[ -z ${ca_file} ]]
        then
                echo "WARN:${vhost}: unable to get SSL paramters"
                continue
        fi
        if [ "${ca_file}" != "${expected_ca_file}" ]
        then
                echo "ERROR:${vhost}: unexpected SSLCACertificateFile (${ca_file} != ${expected_ca_file})"
                program_exit=1
                continue
        fi
        cert_modulus=$(openssl x509 -noout -modulus -in ${cert_file} | openssl md5)
        key_modulus=$(openssl rsa -noout -modulus -in ${key_file} | openssl md5)
        if [ "${cert_modulus}" != "${key_modulus}" ]
        then
                echo "ERROR:${vhost}: Public private key dont match"
                program_exit=1
                continue
        fi
        openssl verify -CAfile "${expected_ca_file}" "${cert_file}" &> /dev/null
        if [ $? -ne 0 ]
        then
                echo "ERROR:${vhost}: keys not signed by local puppet ca ${expected_ca_file}"
                program_exit=1
                continue
        fi
        echo "PASS:${vhost} OK!"
done

HA and its config

In order to provide HA for the puppet infrastructure we use the standard hostname puppet as the default puppetmaster. this hostname resolves to a site specific (in production) or cloud specific puppetmaster via a CNAME. This means that the puppet masters servicing cloud nodes need to have a certificate which is valid for this hostname. With the current puppet code we specificity generate a puppet certificate and store it in the private repo secrets module as per the puppetmaster::puppet_ca module. To generate such a certificate you can use the following command

$ sudo puppet ca generate --dns-alt-names puppet,puppetmaster,puppet.$(hostname -d) puppetmaster.$(hostname -d)

after this is complete you should upload the files in /var/lib/puppet/server/ssl/{certs,private_keys}/puppetmaster.$(hostname -d).pem to the correct secrets location in the private repo on the MoMs

For the cloud puppet masters we also need to generate a client certificate for its self signed by its own puppet CA. Remember the certs in /var/lib/puppet/ssl are signed by the MoM servers and we need some agent certs under /var/lib/puppet/server/ssl

$ sudo puppet ca generate --dns-alt-names puppet,puppetmaster,puppet.$(hostname -d),puppetmaster.$(hostname -d) ($hostname -f)

HTTPd config

Currently the intermediate puppetmaster vhos on cloud e.g. puppetmaster-01.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud have the SSL config pointing to the agent ssl dir /var/lib/puppet/ssl however i think they should be more correctly pointing to the server ssl dir /var/lib/puppet/server/ssl. I feel this is something we need to fix however the last attempt caused issues with production. This currently seems to be working in cloud but i feel like its an there is an issue here waiting to break