Portal:Toolforge/Admin/Kubernetes/Certificates

Tracked in Phabricator
Task T292238

This page contains information on certificates (PKI, X.509, etc) for the Toolforge Kubernetes cluster.

General considerations

Kubernetes includes an internal CA which is the main one we use for cluster operations.

By default, kubernetes issued certificates are valid for 1 year. After that period, they should be renewed.

The internal kubernetes CA, generated at deployment time by kubadm expires after 10 years. The current CA is good until Nov 3 14:13:50 2029 GMT

Worth noting that etcd servers don't use the kubernetes CA, but use the puppetmaster CA instead.

Most certs can be checked for expiration with sudo kubeadm certs check-expiration on a control plane node.

External API access

We have certain entities contacting external the kubernetes API. The authorization/authentication access is managed using a kubernetes ServiceAccount and a x509 certificate. The x509 certificate encodes the ServiceAccount name in the Subject field.

Some examples of this:

tools-prometheus uses this external API access to scrape metrics.
TODO: any other example?

Operations

Disable Puppet on the affected notes to make this whole operation more atomic, and no puppet client see the private repo without content

Certificates for this use case can be generated using a custom script we have: wmcs-k8s-get-cert .

Usually, the generated cert will be copy&pasted into the private puppet repo to be used as a secret in a puppet module or profile.

Renewing the certificate is just generating a new one and replacing the old one.

Example workflow for replacing tools-prometheus k8s certificate:

user@cloudcumin1001:~# sudo cumin "O{project:tools name:tools-prometheus}" 'disable-puppet "T12345 refreshing certificates"'

user@tools-k8s-control-3:~$ sudo -i wmcs-k8s-get-cert prometheus
/tmp/tmp.9k9N7ksn6K/server-cert.pem
/tmp/tmp.9k9N7ksn6K/server-key.pem
user@tools-k8s-control-3:~$ sudo cat /tmp/tmp.9k9N7ksn6K/server-cert.pem
-----BEGIN CERTIFICATE-----
MIIDYTCCA[...]
-----END CERTIFICATE-----
user@tools-k8s-control-3:~$ sudo cat /tmp/tmp.9k9N7ksn6K/server-key.pem
-----BEGIN RSA PRIVATE KEY-----
MIIEpQIBA[...]
-----END RSA PRIVATE KEY-----

root@tools-puppetmaster-02:/var/lib/git/labs/private# vim modules/secret/secrets/ssl/toolforge-k8s-prometheus.key
# copy paste here the private key
root@tools-puppetmaster-02:/var/lib/git/labs/private# git commit -a
# Write the task you are working on in the commit and any details you find relevant you are done!

user@laptop:~/git/wmf/operations/puppet$ nano modules/profile/files/ssl/toolforge-k8s-prometheus.crt
create a patch similar to https://gerrit.wikimedia.org/r/c/operations/puppet/+/926484

user@cloudcumin1001:~# sudo cumin "O{project:tools name:tools-prometheus}" 'enable-puppet "T12345 refreshing certificates"'

Internal API requests

Sometimes the Kubernetes API server makes requests to other services. For example:

custom webhooks
the internal metrics server (i.e, what kubectl top uses)

In general Kubernetes requires those requests to be encrypted and verified via TLS certificates. Historically we used to generate certificates for those using the Kubenetes internal CA, because that was possible and the easiest method. However due to changes in the Kubernetes certificates API, that is no longer possible. These days the modern approach for these is to use cert-manager to generate those certificates.

cert-manager

You will need to generate a certificate for the service. Self-signed certificates will work fine. Something like this:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: metrics-server-api-tls
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: metrics-server-api-tls
spec:
  dnsNames:
    - "metrics-server.metrics.svc"
  secretName: metrics-server-api-tls
  revisionHistoryLimit: 1
  issuerRef:
    name: metrics-server-api-tls
    kind: Issuer
    group: cert-manager.io

This will generate a certificate for that DNS name and save it to a secret called metrics-server-api-tls. To see the status of the certificate, use kubectl describe certificate:

taavi@tools-sgebastion-11:~ $ kubectl describe certificate -n metrics metrics-server-api-tls
Name:         metrics-server-api-tls
Namespace:    metrics
Labels:       app=raw
              app.kubernetes.io/managed-by=Helm
              chart=raw-0.3.0
              heritage=Helm
              release=metrics-server-api-certs
Annotations:  meta.helm.sh/release-name: metrics-server-api-certs
              meta.helm.sh/release-namespace: metrics
API Version:  cert-manager.io/v1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2023-02-17T11:15:56Z
  Generation:          1
  Managed Fields:
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:revision:
    Manager:      cert-manager-certificates-issuing
    Operation:    Update
    Time:         2023-02-17T11:15:56Z
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    Manager:         helm
    Operation:       Update
    Time:            2023-02-17T11:15:56Z
  Resource Version:  1059036361
  UID:               2e6e087e-06d9-4da5-b8ed-6109dbc38d6e
Spec:
  Dns Names:
    metrics-server.metrics.svc
  Issuer Ref:
    Group:                 cert-manager.io
    Kind:                  Issuer
    Name:                  metrics-server-api-tls
  Revision History Limit:  1
  Secret Name:             metrics-server-api-tls
Status:
  Conditions:
    Last Transition Time:  2023-02-17T11:15:56Z
    Message:               Certificate is up to date and has not expired
    Observed Generation:   1
    Reason:                Ready
    Status:                True
    Type:                  Ready
  Not After:               2023-05-18T11:15:56Z
  Not Before:              2023-02-17T11:15:56Z
  Renewal Time:            2023-04-18T11:15:56Z
  Revision:                1
Events:
  Type    Reason     Age   From                                       Message
  ----    ------     ----  ----                                       -------
  Normal  Issuing    17m   cert-manager-certificates-trigger          Issuing certificate as Secret does not exist
  Normal  Generated  17m   cert-manager-certificates-key-manager      Stored new private key in temporary Secret resource "metrics-server-api-tls-hv7gg"
  Normal  Requested  17m   cert-manager-certificates-request-manager  Created new CertificateRequest resource "metrics-server-api-tls-cm8gn"
  Normal  Issuing    17m   cert-manager-certificates-issuing          The certificate has been successfully issued

On the api configuration (so usually either the webhook or APIService object), use cert-manager's CAInjector feature:

  annotations:
    # syntax: namespace/secret-name
    cert-manager.io/inject-ca-from: "metrics/metrics-server-api-tls"

Cert-manager will automatically renew the certificate when it has 1/3 of its lifetime remaining. If the service does not automatically re-load the changed certificate, you can use stakater/reloader to restart the deployment when the certificates change.

Node/kubelet certs

This information has been moved to Portal:Toolforge/Admin/Kubernetes

Tool certs

This information has been moved to Portal:Toolforge/Admin/Kubernetes

Etcd certs

All etcd servers use puppetmaster issued certificates (puppet node certificates). The etcd service will only allow communication from clients presenting a certificate signed by the same CA. This means kubernetes components that contact etcd should use puppet node certificates.

In the puppet profile controlling this, we have a mechanism to refresh the certificate and restart the etcd daemon if the puppet node certificate changes (it is reissued or whatever).