Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors

The procedures in this runbook require admin permissions to complete.

The MaintainDBUsersManyError happens when maintain-dbusers is having many errors when trying to operate.

Debugging

Maintain-dbusers lives in one of the cloudcontrol nodes, to find out which:

02:41 PM ~/Work/wikimedia/operations-puppet (production|✔)
dcaro@acme$ git grep maintain_dbusers_primary:
hieradata/common.yaml:wmcs_maintain_dbusers_primary: cloudcontrol1007.eqiad.wmnet

It should be up and running as a systemd service and listening on port 9090:

root@cloudcontrol1007:~# systemctl status maintain-dbusers.service 
● maintain-dbusers.service - Maintain wikireplica and toolsdb accounts for tools users, tools and paws users
     Loaded: loaded (/lib/systemd/system/maintain-dbusers.service; static)
     Active: active (running) since Wed 2025-11-12 13:01:28 UTC; 42min ago
   Main PID: 3483570 (maintain-dbuser)
      Tasks: 2 (limit: 617442)
     Memory: 39.2M
        CPU: 1min 43.333s
     CGroup: /system.slice/maintain-dbusers.service
             └─3483570 /usr/bin/python3 /usr/local/sbin/maintain-dbusers maintain

You can check the logs to see if there were any errors:

root@cloudcontrol1007:~# journalctl -u maintain-dbusers.service 
...

You can try to restart it, or run it manually to see why it's getting stuck (might be LDAP hanging, as it has to fetch all the users from there).

Common issues

Related information

Support contacts

Old incidents

Add here any new incidents