Toolforge Kubernetes 1.26 upgrade

Parent task: phab:T327025

prep

[x] run prepare upgrade cookbook
[x] downtime project via https://prometheus-alerts.wikimedia.org
[x] update topic on -cloud

control nodes

tools-k8s-control-7

[x] run upgrade node cookbook
[x] check that services start healthy
[x] depool control-8 and -9 via haproxy, check that control-7 is still doing ok

tools-k8s-control-8

[x] run upgrade node cookbook
[x] check that services start healthy

tools-k8s-control-9

[x] run upgrade node cookbook
[x] check that services start healthy

worker nodes

run upgrade node cookbook for each. it's ok to do a couple in parallel

[x] tools-k8s-worker-nfs-1
[x] tools-k8s-worker-nfs-2
[x] tools-k8s-worker-nfs-3
[x] tools-k8s-worker-nfs-5
[x] tools-k8s-worker-nfs-6
[x] tools-k8s-worker-nfs-7
[x] tools-k8s-worker-nfs-8
[x] tools-k8s-worker-nfs-9
[x] tools-k8s-worker-nfs-10
[x] tools-k8s-worker-nfs-11
[x] tools-k8s-worker-nfs-12
[x] tools-k8s-worker-nfs-13
[x] tools-k8s-worker-nfs-14
[x] tools-k8s-worker-nfs-16
[x] tools-k8s-worker-nfs-17
[x] tools-k8s-worker-nfs-19
[x] tools-k8s-worker-nfs-20
[x] tools-k8s-worker-nfs-21
[x] tools-k8s-worker-nfs-22
[x] tools-k8s-worker-nfs-23
[x] tools-k8s-worker-nfs-24
[x] tools-k8s-worker-nfs-26
[x] tools-k8s-worker-nfs-27
[x] tools-k8s-worker-nfs-28
[x] tools-k8s-worker-nfs-29
[x] tools-k8s-worker-nfs-30
[x] tools-k8s-worker-nfs-31
[x] tools-k8s-worker-nfs-32
[x] tools-k8s-worker-nfs-33
[x] tools-k8s-worker-nfs-34
[x] tools-k8s-worker-nfs-35
[x] tools-k8s-worker-nfs-36
[x] tools-k8s-worker-nfs-37
[x] tools-k8s-worker-nfs-38
[x] tools-k8s-worker-nfs-39
[x] tools-k8s-worker-nfs-40
[x] tools-k8s-worker-nfs-41
[x] tools-k8s-worker-nfs-42
[x] tools-k8s-worker-nfs-43
[x] tools-k8s-worker-nfs-44
[x] tools-k8s-worker-nfs-45
[x] tools-k8s-worker-nfs-46
[x] tools-k8s-worker-nfs-47
[x] tools-k8s-worker-nfs-48
[x] tools-k8s-worker-nfs-49
[x] tools-k8s-worker-nfs-50
[x] tools-k8s-worker-nfs-53
[x] tools-k8s-worker-nfs-54
[x] tools-k8s-worker-nfs-55
[x] tools-k8s-worker-nfs-56
[x] tools-k8s-worker-nfs-57
[x] tools-k8s-worker-nfs-58
[x] tools-k8s-worker-nfs-60
[x] tools-k8s-worker-nfs-61
[x] tools-k8s-worker-nfs-62
[x] tools-k8s-worker-nfs-63
[x] tools-k8s-worker-nfs-64
[x] tools-k8s-worker-102
[x] tools-k8s-worker-103
[x] tools-k8s-worker-105
[x] tools-k8s-worker-106
[x] tools-k8s-worker-107
[x] tools-k8s-worker-108

ingress nodes

[x] kubectl -n ingress-nginx-gen2 scale deployment ingress-nginx-gen2-controller --replicas=2
run upgrade node cookbook for each:
[x] tools-k8s-ingress-7
[x] tools-k8s-ingress-8
[x] tools-k8s-ingress-9
[x] revert afterwards: kubectl -n ingress-nginx-gen2 scale deployment ingress-nginx-gen2-controller --replicas=3

cleanup

[x] remove downtime
[x] revert topic change