Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity

Toolforge Admin

• Toolforge user docs
• Toolforge admin docs
• System overview
• Reference: all WMCS-owned services

Admin tasks and tools

• List of admin tasks
• ToolsDB
• OpenStack Browser
• Infrastructure tools

Failover

• Toolschecker
• Redis
• Prometheus

Kubernetes

• Certificates
• Deploying
• Docker-registry
• Jobs framework
• Labels
• Networking and ingress
• Pod tracing

Kubernetes services

• Jobs framework
• Build service
• Envvars service
• Harbor

Packaging

• .deb packages

Runbooks and emergency guides

• Portal:Toolforge/Admin/Runbooks

Other

• Exim and Email operations

[]

The procedures in this runbook require admin permissions to complete.

The ToolforgeKubernetesCapacity alert fires when the Toolforge Kubernetes cluster is close to running out of a certain resource (CPU or memory). In general, this is caused either by:

• Natural increased usage (in which case the fix is to simply provision more capacity)
• Something misbehaving and taking up more resources than it should

Debugging

Dashboard

This panel has an overview of the cluster capacity. The main thing to worry about is "CPU requests" staying below "CPU allocatable".

Locating new workloads

On a bastion run as your own user:

$ kubectl get cronjob -A --sort-by=.metadata.creationTimestamp
$ kubectl get deployment -A --sort-by=.metadata.creationTimestamp

Fixing

Provisioning new nodes

On a cloudcumin:

$ sudo cookbook wmcs.toolforge.add_k8s_node --cluster-name tools --role worker

If you need to provision more than one node at once, it is safe to start a new cookbook run after the first one has created the VM.

Related information

• Portal:Toolforge/Admin/Kubernetes

Old incidents

• The cluster ran out of capacity on 2023-12-01 when cron jobs could not execute fast enough during a cluster reboot after several tools had migrated from the Grid Engine to K8s.