This page contains information on how to administer and replace SSL certificates for Toolforge.
Certificates are usually valid for 1 year, and they should be renewed at least 2 weeks prior to expiration date.
There are currently 2 certificates in use.
- *.wmflabs.org (also know as star.wmflabs.org)
- *.tools.wmflabs.org (also known as star.tools.wmflabs.org)
This certificate is in use by Toolforge web proxies and by other web proxies in WMCS, and is physically deployed in several servers:
In the case of Toolforge servers, the private key is hosted in the project puppetmaster (tools-puppetmaster-*).
In the case of proxy servers in the project-proxy tenant, the private key is hosted directly in the server (non-puppeticed).
This is used for the k8s master and docker registry servers in Toolforge:
These aren't centrally managed yet, but should be! TODO: What does this mean?
Renewing a certificate
The process for both certificates is very similar, and also could be complex.
- request certificate renewal, usually with DCops and/or Traffic team. Involves purchase approval, etc. May take a couple of weeks.
- the private key is added to the /srv/private repo in a prod puppetmaster without replacing the old one. Usually with a new. prefix in the filename. This is usually done by whoever purchases the certificate.
- once the public key is received, an operations/puppet.git repo patch should be stagged into gerrit. This is usually done by whoever purchases the certificate. This patch is not merged yet.
- ensure affected servers/services are working correctly previous to further operations.
- disable/stop puppet agent in all affected servers, i.e, the servers running the webservers using the certificate to be renewed.
- merge public key patch into the operations/puppet.git repo.
- replace the old private key with the new one in the /srv/private repo in production puppetmaster.
- refresh/rebase Toolforge puppetmaster repos (use git pull --rebase): /var/lib/git/operations/puppet and /var/lib/git/labs/private
- manually copy (scp) the private key from /srv/private from a production puppetmaster (puppetmaster1001.eqiad.wmnet for example) into Toolforge own puppetmaster
- replace the old private key with the new one in the private repo in Toolforge puppetmaster (/var/lib/git/labs/private), and do a local git commit (tag it with [local] in the commit msg). Check owner and permissions.
- if required by the certificate you are renewing, scp private key to nova-proxy servers and put it into /etc/ssl/private.
- enable and run puppet agent in one of the server to see if all public/private keys are in place. Restart nginx to see if it can start with no issues with the new certificate.
- if all was fine in the previous test, continue to all other servers.
- in the case of k8s master, restart kube-apiserver as well.
TODO: generate copy-paste commands for simplicity!
A collection of example Phabricator tasks:
- T206223 *.wmflabs.org cert needs renewing
- T160187 ssl certificate/key update: *.tools.wmflabs.org (expires on 2017-03-24)
- T174611 update *.wmflabs.org by 2017-10-16