Portal:Cloud VPS/Admin/Runbooks/PuppetCertificateAboutToExpire
This happens when a Puppet certificate is about to expire. It might be either a normal host certificate or a CA certificate.
Error / Incident
This usually comes in the form of an alert in alertmanager.
The alert will include the CN for the certificate that's expiring.
Debugging
The alert will tell you also how much is left for the certificate to expire in the summary, for example:
summary: Puppet CA certificate Puppet CA: clouddb-services-puppetmaster-01.clouddb-services.eqiad.wmflabs is about to expire in 27d 3h 0m 43s
Common issues
Host certificate about to expire
If a host certificate is about to expire, run the wmcs.vps.refresh_puppet_certs cookbook to renew it. (This usually means that the host is quite old, so consider replacing it with one that uses a newer OS version!)
If the cookbook fails with "The certificate retrieved from the master does not match the agent's private key", try cleaning up the cert manually from the puppet server:
# puppetserver ca clean --certname {FQDN}
Then from the puppet client delete the old cert and re-run Puppet to generate a new one:
# rm /var/lib/puppet/ssl/{FQDN}.pem
# run-puppet-agent
CA certificate about to expire
Sometimes it's the Puppet CA certificate itself that's about to expire. This is happening if the certificate CN starts with Puppet CA:.
To renew that certificate, follow Help:Standalone_puppetmaster#Renewing_puppetserver_CA_certificate.
Related information
Old incidents
- T354714 - [clouddb-service-puppetmaster-2] Renew puppet CA certificates