Fundraising/techops/procedures/services-lets encrypt setup

Ideal setup

Ideally, you will deploy LE certs for a service in two steps. First you will add the certificate to the LE config, and then you'll add it for any service to reference. Here is an example of doing this for nginx.

Let's Encrypt addition

In the letsencrypt::certbot class of the role/service manifest, add the new site to the domains array. An example from frdev:

class { 'letsencrypt::certbot':
    domains             => [
        $civicrm_website_hostname,
        $payments_listener_website_hostname,
        $civi_sa_website_hostname,
        'new.ssl.site.frdev.wikimedia.org',
    ],
}

Then:

Deploy the puppet code.
Run the inital commands in the first part of cert_bot_do_all_the_things to open up the iptables rules and set the proxy setting
Run: certbot certonly --agree-tos -m fr-tech-ops@wikimedia.org --webroot -w /var/www/letsencrypt -d site.fqdn
Run cert_bot_do_all_the_things to verify it will run fine in regular circumstances and to close the firewall rules

Service addition

After you have set up the certificate config and successfully created the certificate, then you can add the references to the certificate or have the service use the certificate.

In the nginx case that would mean adding the https site to the proper manifest:

nginx::https_site { 'new.ssl.site.frdev.wikimedia.org':
    settings go here
}

Chicken and Egg of initial setup

When setting up the LE setup, we need to be able to accept connections on port 80 via Nginx. However, we can't have the site set up in Nginx yet or the service will fail to start due to the missing certificate. If that happens, here are the steps to perform.

Recovery from referencing the cert too early

As root

run the inital commands in the first part of cert_bot_do_all_the_things to open up the iptables rules and set the proxy setting
stop nginx
stop puppet
kill puppet cron
rm host specific file from /etc/nginx/sites-enabled
rm any temp LE files/dirs from testing
- rm /etc/ssl/certs/site.fqdn.bundle.pem
- rm -rf /etc/letsencrypt/live/site.fqdn
run: certbot certonly --agree-tos -m fr-tech-ops@wikimedia.org --webroot -w /var/www/letsencrypt -d site.fqdn
run the iptables commands in cert_bot_do_all_the_things to close the firewall rules
run puppet

At this point, you should have the certificate generated, puppet runs succeeding, and nginx running again.