Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady

Toolforge Admin

• Toolforge user docs
• Toolforge admin docs
• System overview
• Reference: all WMCS-owned services

Admin tasks and tools

• List of admin tasks
• Toolforge roots and Toolforge admins
• ToolsDB
• OpenStack Browser
• Infrastructure tools

Failover

• Redis
• Prometheus

Toolforge components

• Jobs Service
• Build service
• Envvars service
• Components Service
• Logs Service
• Full list
• Deploying a component or cli

Packaging

• Packaging guide

Kubernetes infrastructure

• Certificates
• Deploying a new cluster
• Docker-registry
• Labels
• Networking and ingress
• Pod tracing

Runbooks and emergency guides

• Portal:Toolforge/Admin/Runbooks
• Incident response process

Other

• Exim and Email operations

[]

The procedures in this runbook require admin permissions to complete.

The ToolforgeKubernetesNodeNotReady can happen for three reasons in two different urgency levels (see node.yaml):

• critical - One k8s worker node is flagged as not ready
• page - More than five nodes are flagged as not ready
• page - There's no metric to alert on (prometheus is missing the metric)

Debugging

On a bastion run as your own user:

$ kubectl sudo get node
$ kubectl sudo describe node <node>

Related information

• Portal:Toolforge/Admin/Kubernetes