Kubernetes/Infrastructure Upgrade Procedure

Kubernetes

• Clusters
• Docker Image building
• Deployments
• Deployment Charts
• Sextant
• Charlie
• kubectl
• kubectl/Cheat Sheet
• Service ports
• Service ips
• Education and learning resources
• Troubleshooting
• Resource requests and limits

Administration

• Administration
• Add a new service
• Remove a service
• Creating a new cluster
• Adding or removing nodes
• Adding or removing control-planes
• Infrastructure Upgrade Procedure
• Point upgrades
• Debian Packages

Policies

• Infrastructure upgrade policy
• OCI Image Versioning Policy
• Upstream Helm charts policy

Infrastructure

• Calico
• cert-manager
• ChartMuseum
• Deployment pipeline
• Docker-registry
• Dragonfly
• Helm packaging
• Ingress
• Istio
• Logging
• Metrics
• Container Runtime Engine

Local development

• Minikube
• Local-charts

Related

• Kubernetes on Toolforge
• Mirror of the Kubernetes v1.16 docs
• Mediawiki On Kubernetes

[]

This describes the general procedure used to plan and execute previous kubernetes upgrades. This is just a very high level overview of the required steps and there will potentially be special cases and things to consider with each individual upgrade. Also all of this is not set in stone and should be amended/extended along the way.

General process

• Create an umbrella Phab task for planning and coordination
• Ensure clusters are running the latest patch level version
• Set a target k8s version, based on the compatibility matrix of all components
• Define if/which components are in need of an upgrade (to satisfy the dependency matrix between k8s and all components)
  • Can this update be done before the k8s upgrade (e.g. is the components target version compatible with the currently running k8s version)? If so, upgrade the component right away to decouple it from the k8s upgrade.
• Update deployment-charts CI to validate against the chosen target k8s version, see: Add_CI_validation_for_a_new_Kubernetes_version
• Read all k8s changelogs (see below) between current and target k8s version
• Read the changelogs for all components
• Re-iterate over the k8s and component changelogs and work through the action required items, gather relevant details and incorporate necessary changes into debian packages, container images, puppet, helm charts, ...
• Build required debian packages and container images
• Upgrade staging clusters
• Ensure eventual metric changes are reflected in grafana dashboards and alerts
• Upgrade production clusters

Reading changelogs

For reading the changelogs go version-by version. In kubernetes case just read the minor version changelogs as these will entail all changes introduced by patch level versions in between. Copy changes that seem relevant into the phab task. If in doubt better copy more then less. You can always remove or strike it later. Keep/include issue numbers/references where possible.

Especially look out for:

• Breaking changes/Urgent Upgrade Nodes (usually marked with: "No, really, you MUST read this before you upgrade")
• "Action Required" marks
• Deprecated APIs
• Changes to metrics/metric names (these happen from time to time and collecting them right away makes it easier to follow up later)

Classify the changes by 'Action Required' and 'Note', where 'Action Required' may contain 'Metrics changes'.

• 'Action Required': Contains items you think require additional work like changes to the debian packages, puppet codebase, helm charts, etc.
• 'Note': Contains items worth mentioning. Like new features being added that would unblock some task or would be nice to integrate, changes in behavior that might interfere with our use or stuff that just seems interesting at first read and you want to follow up on.

Template per k8s minor version:

[] [[ https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-9.9.md | v9.9.99 ]]
* **Action Required**
  * **Deprecated APIs**
  * **Metrics changes**
* **Node**