cert-manager adds (besides others) a Certificate CRD to the Kubernetes clusters which automates obtaining and renewing of TLS certificates. A so called Issuer component, the cfssl-issuer is used as a bridge to our PKI which does the actual signing.
- cert-manager: source, Docker images, helm chart
- cfssl-issuer: source, Docker image, helm chart, helm chart (CRDs)
All components are installed to clusters using the
install_cert_manager toggle in helmfile.d/admin_ng/helmfile.yaml
While cert-manager is deployed with default config, cluster operators need to provide at least one CFSSL ClusterIssuer/Issuer object that defines URL, credentials and configuration of the PKI server as well as which label (CFSSL wording for intermediate CA) and signing profile/policy to use.
This is all taken care of by the cfssl-issuer helm-chart. By default it will use the
discovery label/intermediate and the
k8s profile (which is, apart from the auth key, equal to the default server profile. See the PKI docs for details). For staging clusters the profile
k8s_staging should be used which limits the certificates to 24h expiry.
If you add another profile for a new cluster, remember to file a change like the following to add the ferm rules to PKI intermediate nodes: https://gerrit.wikimedia.org/r/c/operations/puppet/+/807502
This is work in progress. We should have a helper function in the helm charts common_templates that does this in a generic way. If you feel the need for a certificate right now, please drop a line to ServiceOps.
All deployers can request certificates from within the cluster by adding a
Certificate object to their services helm chart, like:
apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: testcert spec: secretName: testcert-tls-certificate dnsNames: - testcert.discovery.wmnet - testcert.svc.codfw.wmnet - testcert.svc.eqiad.wmnet issuerRef: name: discovery group: cfssl-issuer.wikimedia.org kind: ClusterIssuer
spec.issuerRef references the CFSSL Issuer to use, so it should not be changed in general.
The Certificate object will be validated via a webhook and, if valid, trigger cert-manager flow.  cert-manager now creates a CertificateRequest object containing a CRT which is then signed via the cfssl-issuer / PKI and ultimately ends in a
kubernetes.io/tls Secret with the name
testcert-tls-certificate being created in the same namespace as the Certificate object.
The Secret object contains the data fields
tls.key which can be used in Pods like described in https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets-as-files-from-a-pod
The Istio-Ingressgateway requires TLS certificates to be created in it's Kubernetes namespace (
istio-system). As deployers don't have access to this namespace, certificate objects will be automatically added during namespace creation by helmfile.d/admin_ng/helmfile_namespace_certs.yaml if
namespace_certificates is enabled for a cluster in helmfile.d/admin_ng/helmfile.yaml.
A Grafana dashboard can be found at: https://grafana-rw.wikimedia.org/d/vo5tiJTnz/cert-manager
To build a new cfssl-issuer, you need to add a new SemVer tag to it's repository, update the Dockerfile.template and changelog in the production-images repository and then build a new image using docker-pgk. The process is described in Kubernetes/Images#Production images