Kubernetes/Point upgrades

The steps below detail the procedure to perform a point upgrade for Kubernetes (e.g. 1.31.4 to 1.31.14).

Build

Follow the instructions at Kubernetes/Packages to build a new version of the Kubernetes packages.

Ensure that you copy the resulting packages to the apt server, and copy the packages for each of the distributions the new packages will be deployed on.

Deploy to workers

It's preferable to first deploy the package changes on a staging cluster (e.g. staging-codfw), before deploying elsewhere. Deploy changes to kubernetes workers first, and ensure they become ready after the upgrade, before proceeding to upgrade the control plane.

On the deployment server, use debdeploy to generate a spec, and then deploy to the kubernetes workers in the cluster you're working with (consider deploying to a single node first, before proceeding with larger node groups):

sudo debdeploy deploy -u 2026-05-28-kubernetes-node.yaml -Q "P{kubestage100*}"

kubectl and kube-proxy should automatically restart when the package is deployed. Double check that this happened correctly on the affected nodes, and that the nodes became schedulable.

Deploying in moderately-sized batches is a good idea, to ensure that the entire cluster doesn't experience networking stuttering at the same time. Restarting kube-proxy will cause nodes to recheck iptables rules, and this can cause a few seconds of latency.

Deploy to the control plane

Upgrading the control plane is mechanically similar to upgrading the workers, but the apiserver service will not automatically restart when the new package is deployed.

First, upgrade the package:

sudo debdeploy deploy -u 2026-05-28-kubernetes-node.yaml -Q "P{kubestagemaster100*}"

Then, use the kube-apiserver-safe-restart service to restart the apiserver on each of the control plane nodes. This service uses a lockfile in etcd to ensure that only one control plane apiserver is restarting at any given time.

sudo cumin -b 1 kubestagemaster[1003-1005].eqiad.wmnet 'sudo systemctl restart kube-apiserver-safe-restart'

Ensure that the staging cluster is operating as expected, and proceed to upgrade the remaining clusters.

Deploying to other clusters

Before upgrading other clusters, make a note of your intent in #wikimedia-k8s-sig, per the discussion in the SIG meeting. It's been established that the team that operates kubernetes for the Wikikube clusters can proceed with these upgrades after such a notification.

Consider tagging the cluster owners in your message.