Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyUnknown

Toolforge Admin

• Toolforge user docs
• Toolforge admin docs
• System overview
• Reference: all WMCS-owned services

Admin tasks and tools

• List of admin tasks
• Toolforge roots and Toolforge admins
• ToolsDB
• OpenStack Browser
• Infrastructure tools

Failover

• Toolschecker
• Redis
• Prometheus

Kubernetes

• Certificates
• Deploying
• Docker-registry
• Jobs framework
• Labels
• Networking and ingress
• Pod tracing

Kubernetes services

• Jobs framework
• Build service
• Envvars service
• Harbor

Packaging

• .deb packages

Runbooks and emergency guides

• Portal:Toolforge/Admin/Runbooks
• Incident response process

Other

• Exim and Email operations

[]

This happens when prometheus has no data from haproy on the server status (server_up).

The procedures in this runbook require admin permissions to complete.

Error / Incident

This usually comes in the form of an alert in alertmanager.

There you will get which project (tools, toolsbeta, ...) is the one it's failing for.

Debugging

This is a tricky one and it will be related to the way we gather metrics on tools/toolsbeta.

Note that this is not directly related to the metricsinfra monitoring project, but toolforge's own setup.

You can start by going to the project's prometheus page and trying to get the stats there, example for tools:

https://tools-prometheus.wmflabs.org/tools/classic/graph?g0.range_input=1h&g0.expr=sum(up%7Bjob%3D%22tekton-pipelines-controller%22%7D)&g0.tab=1

Common issues

Add new issues here when you encounter them!

Related information

• See HAProxy for generic information like commands/etc.
• Karma UI for cloud vps (use project=tools or project=toolsbeta for filtering)
• Tools prometheus
• Toolsbeta prometheus
• Alerts repository
• Toolforge admin docs
• See also any other *Unknown runbooks.

Old incidents

Add any incident tasks here!