This page contains historical information. It may be outdated or unreliable.

Toolforge Kubernetes 1.23 upgrade

prep

[x] run prepare upgrade cookbook
[x] downtime project via https://prometheus-alerts.wikimedia.org
[x] update topic on -cloud

control nodes

tools-k8s-control-4

[x] run upgrade node cookbook
[x] check that services start healthy

tools-k8s-control-5

[x] run upgrade node cookbook
[x] check that services start healthy

tools-k8s-control-6

[x] run upgrade node cookbook
[x] check that services start healthy

worker nodes

run upgrade node cookbook for each. it's ok to do a couple in parallel

[x] tools-k8s-worker-30 [x] tools-k8s-worker-31 [x] tools-k8s-worker-32

tools-k8s-worker-33 tools-k8s-worker-34 tools-k8s-worker-35 tools-k8s-worker-36 tools-k8s-worker-37 tools-k8s-worker-38 tools-k8s-worker-39 tools-k8s-worker-40 tools-k8s-worker-41 tools-k8s-worker-42 tools-k8s-worker-43 tools-k8s-worker-44 tools-k8s-worker-45 tools-k8s-worker-46 tools-k8s-worker-47 tools-k8s-worker-48 tools-k8s-worker-49 tools-k8s-worker-50

tools-k8s-worker-51 tools-k8s-worker-52 tools-k8s-worker-53 tools-k8s-worker-54 tools-k8s-worker-55 tools-k8s-worker-56 tools-k8s-worker-57 tools-k8s-worker-58 tools-k8s-worker-59 tools-k8s-worker-60 tools-k8s-worker-61 tools-k8s-worker-62 tools-k8s-worker-64 tools-k8s-worker-65 tools-k8s-worker-66 tools-k8s-worker-67 tools-k8s-worker-68 tools-k8s-worker-69 tools-k8s-worker-70

tools-k8s-worker-71 tools-k8s-worker-72 tools-k8s-worker-73 tools-k8s-worker-74 tools-k8s-worker-75 tools-k8s-worker-76 tools-k8s-worker-77 tools-k8s-worker-78 tools-k8s-worker-79 tools-k8s-worker-80 tools-k8s-worker-81 tools-k8s-worker-82 tools-k8s-worker-83 tools-k8s-worker-84 tools-k8s-worker-85 tools-k8s-worker-86 tools-k8s-worker-87 tools-k8s-worker-88

ingress nodes

[x] kubectl -n ingress-nginx-gen2 scale deployment ingress-nginx-gen2-controller --replicas=2
run upgrade node cookbook for each:

[x] tools-k8s-ingress-4 [x] tools-k8s-ingress-5 [x] tools-k8s-ingress-6

[x] revert afterwards: kubectl -n ingress-nginx-gen2 scale deployment ingress-nginx-gen2-controller --replicas=3

cleanup

[x] remove downtime
[x] revert topic change