Portal:Toolforge/Admin/SSL certificates

From Wikitech
Jump to navigation Jump to search

This page contains information on how to administer and replace SSL certificates for Toolforge.

Certificates are usually valid for 1 year, and they should be renewed at least 2 weeks prior to expiration date.

Usage

There are currently 2 certificates in use.

  • *.wmflabs.org (also know as star.wmflabs.org)
  • *.tools.wmflabs.org (also known as star.tools.wmflabs.org)

*.wmflabs.org

This certificate is in use by Toolforge web proxies and by other web proxies in WMCS, and is physically deployed in several servers:

  • tools-static-*.tools.eqiad.wmflabs
  • tools-proxy-*.tools.eqiad.wmflabs
  • novaproxy-*.project-proxy.eqiad.wmflabs

In the case of Toolforge servers, the private key is hosted in the project puppetmaster (tools-puppetmaster-*).

In the case of proxy servers in the project-proxy tenant, the private key is hosted directly in the server (non-puppeticed).

*.tools.wmflabs.org

This is used for the k8s master and docker registry servers in Toolforge:

  • tools-k8s-master-*.tools.eqiad.wmflabs
  • tools-docker-registry-*.tools.eqiad.wmflabs

These aren't centrally managed yet, but should be! TODO: What does this mean?

Renewing a certificate

The process for both certificates is very similar, and also could be complex.

  1. request certificate renewal, usually with DCops and/or Traffic team. Involves purchase approval, etc. May take a couple of weeks.
  2. the private key is added to the /srv/private repo in a prod puppetmaster without replacing the old one. Usually with a new. prefix in the filename. This is usually done by whoever purchases the certificate.
  3. once the public key is received, an operations/puppet.git repo patch should be stagged into gerrit. This is usually done by whoever purchases the certificate. This patch is not merged yet.
  4. ensure affected servers/services are working correctly previous to further operations.
  5. disable/stop puppet agent in all affected servers, i.e, the servers running the webservers using the certificate to be renewed.
  6. merge public key patch into the operations/puppet.git repo.
  7. replace the old private key with the new one in the /srv/private repo in production puppetmaster.
  8. refresh/rebase Toolforge puppetmaster repos (use git pull --rebase): /var/lib/git/operations/puppet and /var/lib/git/labs/private
  9. manually copy (scp) the private key from /srv/private from a production puppetmaster (puppetmaster1001.eqiad.wmnet for example) into Toolforge own puppetmaster
  10. replace the old private key with the new one in the private repo in Toolforge puppetmaster (/var/lib/git/labs/private), and do a local git commit (tag it with [local] in the commit msg). Check owner and permissions.
  11. if required by the certificate you are renewing, scp private key to nova-proxy servers and put it into /etc/ssl/private.
  12. enable and run puppet agent in one of the server to see if all public/private keys are in place. Restart nginx to see if it can start with no issues with the new certificate.
  13. if all was fine in the previous test, continue to all other servers.
  14. in the case of k8s master, restart kube-apiserver as well.

TODO: generate copy-paste commands for simplicity!

Examples

A collection of example Phabricator tasks:

See also