Acme-chief is an application resulting from the Wikimedia Hackathon 2018 that is to be used to centrally request configured TLS certificates from ACME servers, then make the public and private parts available to authorised API users.
See T235252 for how to set this up for a Cloud VPS project - particularly the service account creation subtask which needs to be performed by the cloud administrators.
In production this is already set up to manage production DNS, most people probably just want to know to find the certificate configuration in the hieradata/role/common/acme_chief.yaml file in operations/puppet.git.
If acme-chief is having issues, you should also check the Let's Encrypt status page to make sure it isn't having an outage or maintenance.
Acme-chief production environment is composed of one active instance and at least one passive instance. The hiera key
profile::acme_chief::active flags an instance an active while passive instances are listed in an array called
The active instance is responsible of running both the acme-chief service (
acme-chief.service) and the puppet file API service (
nginx.service). A passive instance idly runs the puppet file API service but not
acme-chief.service. TLS material is synchronized between instances using the oneshot systemd service
acme-chief-certs-sync.service. This service is triggered by a systemd timer every 30 minutes on the active host.
Replacing the active instance
- Create the new Ganeti VM.
- Set its role as
acme_chiefon site.pp and make sure that's listed as a passive instance on
- Once the new instance is up and runnning arm keyholder for SSH access
run-puppet-agenton the current active instance
- Trigger the TLS material sync service on the active instance — either directly on the host or via Cumin:
$ sudo systemctl start acme-chief-certs-sync.service
- The new instance should now have a current copy of the TLS material in
/var/lib/acme-chief/certs. You can verify any of the certificates. For example:
$ openssl x509 -noout -dates -issuer -in /var/lib/acme-chief/certs/unified/live/ec-prime256v1.crt
- Disable puppet on the old active instance and on every acme-chief client
$ sudo -i cumin 'R:acme_chief::cert' "disable-puppet 'acmechief maintenance'"
acme-chief.serviceon the old active instance
- Set the new instance as active in
profile::acme_chief::activeand remove it from
run-puppet-agenton the new active instance. After Puppet is done
acme-chief.serviceshould be up and running in the new instance.
- Re-enable puppet on the acme-chief clients:
$ sudo -i cumin 'R:acme_chief::cert' "enable-puppet 'acmechief maintenance'"
- Decommission the old instance using the sre.hosts.decommission cookbook