Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown

Toolforge Admin

• Toolforge user docs
• Toolforge admin docs
• System overview
• Reference: all WMCS-owned services

Admin tasks and tools

• List of admin tasks
• Toolforge roots and Toolforge admins
• ToolsDB
• OpenStack Browser
• Infrastructure tools

Failover

• Toolschecker
• Redis
• Prometheus

Kubernetes

• Certificates
• Deploying
• Docker-registry
• Jobs framework
• Labels
• Networking and ingress
• Pod tracing

Kubernetes services

• Jobs framework
• Build service
• Envvars service
• Harbor

Packaging

• .deb packages

Runbooks and emergency guides

• Portal:Toolforge/Admin/Runbooks
• Incident response process

Other

• Exim and Email operations

[]

This happens when one of the haproxy servers is down from that haproxy instance (one of the hosts haproxy redirects too, not to be confused with the haproxy server itself).

The procedures in this runbook require admin permissions to complete.

Error / Incident

In the alert you should see the instance (the haproxy host), the backend (the web service being proxied) and the server (the server the requests are proxied to). That might get you started on where to look.

Debugging

TBD

Common issues

Add new issues here when you encounter them!

Related information

• See HAProxy for generic information like commands/etc.

Old incidents

Add any incident tasks here!