Kubernetes/cert-manager

Kubernetes

• Clusters
• Docker Image building
• Deployments
• Deployment Charts
• kubectl
• kubectl/Cheat Sheet
• Service ports
• Service ips
• Education and learning resources
• Troubleshooting
• Resource requests and limits

Policies and administration

• Administration
• Add a new service
• Remove a service
• Creating a new cluster
• Adding or removing nodes
• Adding or removing control-planes
• Infrastructure upgrade policy
• OCI Image Versioning Policy
• Upstream Helm charts policy
• Packages

Infrastructure

• Calico
• cert-manager
• ChartMuseum
• Deployment pipeline
• Docker-registry
• Dragonfly
• Helm packaging
• Ingress
• Istio
• Logging
• Metrics
• Container Runtime Engine

Local development

• Minikube
• Local-charts

Related

• Kubernetes on Toolforge
• Mirror of the Kubernetes v1.16 docs
• Mediawiki On Kubernetes

[]

Introduction

cert-manager adds (besides others) a Certificate CRD to the Kubernetes clusters which automates obtaining and renewing of TLS certificates. A so called Issuer component, the cfssl-issuer is used as a bridge to our PKI which does the actual signing.

Components:

• cert-manager: source, Docker images, helm chart
• cfssl-issuer: source, Docker image, helm chart, helm chart (CRDs)

All components are installed to clusters using the install_cert_manager toggle in helmfile.d/admin_ng/helmfile.yaml

Configuration

While cert-manager is deployed with default config, cluster operators need to provide at least one CFSSL ClusterIssuer/Issuer object that defines URL, credentials and configuration of the PKI server as well as which label (CFSSL wording for intermediate CA) and signing profile/policy to use.

This is all taken care of by the cfssl-issuer helm-chart. By default it will use the discovery label/intermediate and the k8s profile (which is, apart from the auth key, equal to the default server profile. See the PKI docs for details). For staging clusters the profile k8s_staging should be used which limits the certificates to 24h expiry.

If you add another profile for a new cluster, remember to file a change like the following to add the ferm rules to PKI intermediate nodes: https://gerrit.wikimedia.org/r/c/operations/puppet/+/807502

Usage

This is work in progress. We should have a helper function in the helm charts common_templates that does this in a generic way.

If you feel the need for a certificate right now, please drop a line to ServiceOps.

All deployers can request certificates from within the cluster by adding a Certificate object to their services helm chart, like:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: testcert
spec:
  secretName: testcert-tls-certificate
  dnsNames:
    - testcert.discovery.wmnet
    - testcert.svc.codfw.wmnet
    - testcert.svc.eqiad.wmnet
  issuerRef:
    name: discovery
    group: cfssl-issuer.wikimedia.org
    kind: ClusterIssuer

spec.issuerRef references the CFSSL Issuer to use, so it should not be changed in general.

The Certificate object will be validated via a webhook and, if valid, trigger cert-manager flow. cert-manager now creates a CertificateRequest object containing a CRT which is then signed via the cfssl-issuer / PKI and ultimately ends in a kubernetes.io/tls Secret with the name testcert-tls-certificate being created in the same namespace as the Certificate object.

The Secret object contains the data fields tls.crt and tls.key which can be used in Pods like described in https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets-as-files-from-a-pod

Istio-Ingressgateway

The Istio-Ingressgateway requires TLS certificates to be created in it's Kubernetes namespace (istio-system). As deployers don't have access to this namespace, certificate objects will be automatically added during namespace creation by helmfile.d/admin_ng/helmfile_namespace_certs.yaml if namespace_certificates is enabled for a cluster in helmfile.d/admin_ng/helmfile.yaml.

Monitoring

A Grafana dashboard can be found at: https://grafana.wikimedia.org/d/vo5tiJTnz/cert-manager

cfssl-issuer

The cfssl-issuer based off of cert-managers sample-external-issuer project.

Repo update

Tags:

• tags prefixed by sample-external-issuer correspond to upsteam tags
• tags without this prefix are our own.

Merging from upstream:

• add the upstream as remote
• push the current upstream/main to the upstream/main branch of the gerrit repo (to make gerrit know about all upstream commits)
• branch off of the main branch
• git merge the upstream changes you want
• resolve conflicts
• git review your merge commit

Testing

End to end the issuer requires a kubernetes cluster, so this is not possible to do in CI currently. There are Make targets to run end to end testing in a local kind cluster, though.

Building cfssl-issuer

To build a new cfssl-issuer, you need to add a new SemVer tag to it's repository, update the Dockerfile.template and changelog in the production-images repository and then build a new image using docker-pgk. The process is described in Kubernetes/Images#Production images