Portal:Toolforge/Admin/SSL certificates

From Wikitech
Jump to navigation Jump to search

This page contains information on how to administer and replace SSL certificates for Toolforge.

Certificates are usually valid for 1 year, and they should be renewed at least 2 weeks prior to expiration date.


There are currently 2 certificates in use.

  • *.wmflabs.org (also know as star.wmflabs.org)
  • *.tools.wmflabs.org (also known as star.tools.wmflabs.org)


This certificate is in use by Toolforge web proxies and by other web proxies in WMCS, and is physically deployed in several servers:

  • tools-static-*.tools.eqiad.wmflabs
  • tools-proxy-*.tools.eqiad.wmflabs
  • novaproxy-*.project-proxy.eqiad.wmflabs

In the case of Toolforge servers, the private key is hosted in the project puppetmaster (tools-puppetmaster-*).

In the case of proxy servers in the project-proxy tenant, the private key is hosted directly in the server (non-puppeticed).

Example URL using this SSL certificate: https://tools.wmflabs.org/


This is used for the k8s master and docker registry servers in Toolforge:

  • tools-k8s-master-*.tools.eqiad.wmflabs
  • tools-docker-registry-*.tools.eqiad.wmflabs

These aren't centrally managed yet, but should be! TODO: What does this mean?

Example URL using this SSL certificate: https://docker-registry.tools.wmflabs.org/

Renewing a certificate

The process for both certificates is very similar, and also could be complex.

  1. request certificate renewal, usually with DCops and/or Traffic team. Involves purchase approval, etc. May take a couple of weeks.
  2. the private key is added to the /srv/private repo in a prod puppetmaster without replacing the old one. Usually with a new. prefix in the filename. This is usually done by whoever purchases the certificate.
  3. once the public key is received, an operations/puppet.git repo patch should be stagged into gerrit. This is usually done by whoever purchases the certificate. This patch is not merged yet.
  4. ensure affected servers/services are working correctly previous to further operations.
  5. disable/stop puppet agent in all affected servers, i.e, the servers running the webservers using the certificate to be renewed.
  6. merge public key patch into the operations/puppet.git repo.
  7. replace the old private key with the new one in the /srv/private repo in production puppetmaster.
  8. refresh/rebase Toolforge puppetmaster repos (use git pull --rebase): /var/lib/git/operations/puppet and /var/lib/git/labs/private
  9. manually copy (scp) the private key from /srv/private from a production puppetmaster (puppetmaster1001.eqiad.wmnet for example) into Toolforge own puppetmaster
  10. replace the old private key with the new one in the private repo in Toolforge puppetmaster (/var/lib/git/labs/private), and do a local git commit (tag it with [local] in the commit msg). Check owner and permissions.
  11. if required by the certificate you are renewing, scp private key to nova-proxy servers and put it into /etc/ssl/private.
  12. enable and run puppet agent in one of the server to see if all public/private keys are in place. Restart nginx to see if it can start with no issues with the new certificate.
  13. if all was fine in the previous test, continue to all other servers.
  14. in the case of k8s master, restart kube-apiserver as well.

Example of replacing *.wmflabs.org

A collection of example Phabricator tasks:

TODO: add command examples

Example of replacing *.tools.wmflabs.org

A collection of example Phabricator tasks:

Concrete example commands used for T223332:

aborrero@labpuppetmaster1001:~$ sudo cumin "O{project:tools name:^tools-k8s-master*} OR O{project:tools name:^tools-docker-registry*}" "puppet agent --disable 'T223332 - arturo upgrading SSL cert'"
3 hosts will be targeted:
Confirm to continue [y/n]? y
===== NO OUTPUT =====                                                                                                                       
PASS:  |███████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:06<00:00,  2.53s/hosts]     
FAIL:  |                                                                                               |   0% (0/3) [00:06<?, ?hosts/s]     
100.0% (3/3) success ratio (>= 100.0% threshold) for command: 'puppet agent --d...rading SSL cert''.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.

aborrero@puppetmaster1001:~ $ sudo puppet-merge
Fetching new commits from https://gerrit.wikimedia.org/r/operations/puppet
--- a/files/ssl/star.tools.wmflabs.org.crt
+++ b/files/ssl/star.tools.wmflabs.org.crt
@@ -1,17 +1,17 @@

aborrero@puppetmaster1001:/srv/private $ sudo git rm modules/secret/secrets/ssl/star.tools.wmflabs.org.key
rm 'modules/secret/secrets/ssl/star.tools.wmflabs.org.key'
aborrero@puppetmaster1001:/srv/private $ sudo git mv modules/secret/secrets/ssl/new.star.tools.wmflabs.org.key modules/secret/secrets/ssl/star.tools.wmflabs.org.key
aborrero@puppetmaster1001:/srv/private $ sudo git commit -as

aborrero@puppetmaster1001:/srv/private $ cp modules/secret/secrets/ssl/star.tools.wmflabs.org.key ~
arturo@laptop:~ $ ssh puppetmaster1001.eqiad.wmnet 'cat star.tools.wmflabs.org.key' | ssh tools-puppetmaster-01.eqiad.wmflabs 'cat > start.tools.wmflabs.org.key'
aborrero@puppetmaster1001:~$ rm star.tools.wmflabs.org.key

aborrero@tools-puppetmaster-01:~$ sudo git-sync-upstream
aborrero@tools-puppetmaster-01:/var/lib/git/labs/private$ sudo mv ~/start.tools.wmflabs.org.key modules/secret/secrets/ssl/star.tools.wmflabs.org.key
aborrero@tools-puppetmaster-01:/var/lib/git/labs/private$ sudo chown gitpuppet:gitpuppet modules/secret/secrets/ssl/star.tools.wmflabs.org.key 
aborrero@tools-puppetmaster-01:/var/lib/git/labs/private$ sudo git commit -as

aborrero@tools-docker-registry-04:~$ sudo puppet agent --enable
aborrero@tools-docker-registry-04:~$ sudo puppet agent -tv
Notice: /Stage[main]/Profile::Toolforge::Docker::Registry/Sslcert::Certificate[star.tools.wmflabs.org]/File[/etc/ssl/localcerts/star.tools.wmflabs.org.crt]/content: 
--- /etc/ssl/localcerts/star.tools.wmflabs.org.crt	2019-01-15 14:17:16.445645012 +0000
+++ /tmp/puppet-file20190520-3514-3gbome	2019-05-20 11:21:09.458580334 +0000
@@ -1,17 +1,17 @@
Notice: Applied catalog in 5.70 seconds

aborrero@tools-docker-registry-04:~$ sudo systemctl restart nginx.service 
aborrero@tools-docker-registry-04:~$ sudo systemctl status nginx.service
aborrero@tools-docker-registry-03:~$ sudo puppet agent --enable
aborrero@tools-docker-registry-03:~$ sudo puppet agent -tv
aborrero@tools-docker-registry-03:~$ sudo systemctl restart nginx.service 
aborrero@tools-docker-registry-03:~$ sudo systemctl status nginx.service
aborrero@tools-k8s-master-01:~$ sudo puppet agent --enable
aborrero@tools-k8s-master-01:~$ sudo puppet agent -tv
aborrero@tools-k8s-master-01:~$ sudo systemctl restart kube-apiserver.service

See also