Portal:Toolforge/Admin/Runbooks/MaintainDBUsersDown

The procedures in this runbook require admin permissions to complete.

The MaintainDBUsersDown happens when maintain-dbusers is not reachable by prometheus.

Debugging

Maintain-dbusers lives in one of the cloudcontrol nodes, to find out which:

02:41 PM ~/Work/wikimedia/operations-puppet (production|✔)
dcaro@acme$ git grep maintain_dbusers_primary:
hieradata/common.yaml:wmcs_maintain_dbusers_primary: cloudcontrol1007.eqiad.wmnet

It should be up and running as a systemd service and listening on port 9090:

root@cloudcontrol1007:~# systemctl status maintain-dbusers.service 
● maintain-dbusers.service - Maintain wikireplica and toolsdb accounts for tools users, tools and paws users
     Loaded: loaded (/lib/systemd/system/maintain-dbusers.service; static)
     Active: active (running) since Wed 2025-11-12 13:01:28 UTC; 42min ago
   Main PID: 3483570 (maintain-dbuser)
      Tasks: 2 (limit: 617442)
     Memory: 39.2M
        CPU: 1min 43.333s
     CGroup: /system.slice/maintain-dbusers.service
             └─3483570 /usr/bin/python3 /usr/local/sbin/maintain-dbusers maintain

root@cloudcontrol1007:~# curl http://127.0.0.1:9090
# HELP python_gc_objects_collected_total Objects collected during gc
# TYPE python_gc_objects_collected_total counter
python_gc_objects_collected_total{generation="0"} 291139.0
python_gc_objects_collected_total{generation="1"} 26545.0
python_gc_objects_collected_total{generation="2"} 205289.0
# HELP python_gc_objects_uncollectable_total Uncollectable objects found during GC
# TYPE python_gc_objects_uncollectable_total counter
python_gc_objects_uncollectable_total{generation="0"} 0.0
python_gc_objects_uncollectable_total{generation="1"} 0.0
python_gc_objects_uncollectable_total{generation="2"} 0.0
# HELP python_gc_collections_total Number of times this generation was collected
# TYPE python_gc_collections_total counter
python_gc_collections_total{generation="0"} 4026.0
python_gc_collections_total{generation="1"} 366.0
python_gc_collections_total{generation="2"} 33.0
# HELP python_info Python platform information
# TYPE python_info gauge
python_info{implementation="CPython",major="3",minor="11",patchlevel="2",version="3.11.2"} 1.0
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 2.144256e+08
# HELP process_resident_memory_bytes Resident memory size in bytes.
...

Common issues

Related information

Support contacts

Old incidents

Add here any new incidents