This page attempts to document how encryption is utilised in the Foundation production environment i.e. service to service
The vast majority of services utilising encryption do so over TLS and often make use of the certificates issued by the Puppet CA. The puppet CA is [ab]used as we know that every server in the production environment already has a certificate issued from a centrally managed location. This makes it easy to issue and revoke certificates however it also make it more difficult to compartmentalise different security domains.
Services wishing to support client authentication with the certificates issued via the puppet CA will need to add the Puppet CA public certificate to there listed of trusted root certificates. Puppet ensures that the CA public certificate is available on all nodes at the following locations
- /var/lib/puppet/ssl/certs/ca.pem i.e.
sudo puppet config print localcacert.
- /etc/ssl/certs/Puppet_Internal_CA.pem (symlink managed by update-ca-certificates)
Clients preforming TLS authentication will need to be configured with the public and private client certificates issued by puppet. The file content of theses files is not managed by the puppet repo but instead issued and stored on the server when it is first imaged. you can use the following commands to show the location of the public and private certificates.
puppet config print hostcert
puppet config print hostprivkey
however these locations are only accessible to the
root users. As such it is common for services to copy theses files to there own configuration folder via
bacula clients are configured to use TLS with the puppet agent certificates exposed via base::expose_puppet_certs. Further they are configured to to verify their peer. The backup server is also configured with TLS however they are configured not to verify the peer however connections are also secured with a password.
The debmonitor server is available via https://debmonitor.wikimedia.org/ using the global sign wildcard certificate. This is served via the caching infrastructure which fetches the actual content from https://debmonitor.discovery.wmnet, this endpoint is configured with a certificate issued via cergen. This site is configured to perform SSL client verifiication and sets appropriate headers which are later used by the django application. On the client side the puppet agent certificates are exposed using
debmonitor-client and configured via
The kafka brokeer is configured to use a certificate generated by cergen much like the
sslcert::certificate list below. However kafka is a java daemon and as such needs the java keystore file which is not currently supported by
sslcert::certificate. Therefore the
profile::kafaka::broker class takes care of copying the java keystore file from the private repo. Kafka is also configured to request SSL client auth trusting certificates signed by the Puppet CA. The following clients make use of the kafka broker
- varnishkafka via
profile::cache::kafka::eventloggingwhich both include
kube-apiservice service is configured with a certificicate genrated by cergen and copied into place using
sslcert::certificate. The kubernetes clients are also configured with SSL client certificates issued via puppet and copied into place using the
k8s::ssl class instead of the
expose_puppet_certs resource. However the kube api server is not configured with
client-ca-file so it is unclear if client authentication is being preformed
The rsync class has an option to wrap rsync commands in an stunnel. If this is enabled then the server will be configured with an stunnel daemon which proxies commands to the rsync process. the cron job or systemd time used to actually rsync the data will also need to be configured as an stunnel client
Rsyslog runs on every machine and sending logs to the
syslog::centralserver role. Both the central server and the sending nodes are configured to use the puppet agent certificates to establish a TLS connection and use the puppet CA as the trusted authority
We also have a number of services configured with standard SSL server certificates. As theses certificates are signed by and internal CA they are not for public consumption and often not utilised by humans. The most common use of theses certificates is to allow for the cacheing and LVS layers to securely talk with the backends services. Services wishing to make to have a Server certificate will first need to generate the certificate using cergen.
Cergen automatically copies files into the the private repo under
sslcert::certificate expects the public certificate to exist in the public puppet-repo in the files directory and the private key to exist in the private repo under the ssl folder
sslcert::certificate is a puppet resources used by many services to copy a certificate from the puppet private repo. In the majority of cases the certificate copied is one generated with cergen however it could also be a commercial certificate such as the wildcard globalsign certs. Theses certificates are often used to provide SSL endpoints for internal services. The following is a list of services which use the
- profile::openldap_corp & profile::docker::registry
profile::tlsproxy::envoy has similar functionality to tlsproxy::localssl however but uses Envoy instead of nginx. It uses
sslcert::certificate to distribute certificates and is intended as a replacement to tlsproxy::localssl. The following roles us this profile
- etherpad -
- grafana -
- graphite::production -
- logstash -
- logstash7 -
- mediawiki::mainenance -
- microsites::peopleweb -
- otrs -
- phabricator -
- planet -
- puppetboard -
- releases -
- restbase::production -
- requesttracker -
- webperf::processors_and_site -
- webserver_misc_apps -
- webserver_misc_static -
tlsproxy::localssl is used to create an TLS termination revers proxy to a service listening on localhost. It is often used to configure backend services which will be fetched by the caching layer. In this case it uses the sslcert::certificate resource to distribute a certificate created by cergen and signed by the puppet CA. however it is also used to distribute certificates created by Acme-chief which are signed by the lets encrypt CA. The following services use tls::proxy
- [https://github.com/wikimedia/puppet/blob/production/modules/profile/manifests/mediawiki/jobrunner_tls.pp#L11-L24 videoscaler
profile::cache::ssl::unified class wraps
tlsproxy::localssl and is primarily used to configure the production on the caching infrastructure caching servers with the global sign wildcard certificate.
profile::mediawiki::webserver class wraps
tlsproxy::localssl and is used to configure TLS termination on the mediawiki servers. The following services use this class
- api server
Apache Traffic Server (ATS)
In order to ensure Cross DC traffic is encrypted we configure the ATS servers to talk to the origin servers (applayer) via TLS. We configure
proxy.config.ssl.client.CA.cert.filename to Puppet_Internal_CA.pem. This allows it to connect to and validate backend services such as grafana.discovery.wmnet. which exposes uses envoy for TLS termination end exposes its puppet CA certificate via sslcert::certificate
Our databases are configured with a TSL enabled endpoints using the puppet agent certificates exposed via
base::expose_puppet_certs. Our config dose not specify set
ssl-mode as such it is configured with the default preferred option. This means clients will attempt to connect over TLS but will fall back to an unencrypted connection if thats not possible.
Currently TLS is used for mariadb to mariadb connections for things like replication as well as some adhoc clients such as the promethious exporter
Mcrouter does not use the Puppet CA instead it has its own CA managed by cergen more information can be found on the mcrouter page. Mcrouter in eqiad establishes TLS connections to 4 Mcrouters in codfw to allow cross-dc replication when needed. Same thing happens vice-versa of course. TLS is not used from client to Mcrouter or from Mcrouter to Memcached.
Hadoop does not use the Puppet CA instead it has its own CA managed by cergen. The certificates issues by the hadoop CA are used to provide TLS authentication to the connections between Mappers/Shufflers/Reducers during Map-Reduce jobs.
Cassandra does not use the Puppet CA instead it has its own CA managed by by a python script cassandra-ca-manager (a precurser to cergen). The certificates issues by the Cassandra CA are used to provide TLS authentication to the connections between the cassandra nodes.