Jump to content

Stuck global renames

From Wikitech

Reporting a stuck rename

Sometimes you'll find that a rename becomes stuck at a project (see for example phab:T365533). In that case, please follow these instructions:

  • Wait at least 3 hours since the rename started. The system will automatically retry failed renames a few times.
  • Please open a task in Phabricator (this link wil automatically add the right tags). In there please:
  • post a link to the Special:GlobalRenameProgress link to the account whose rename has become stuck;
  • mention the old and new user name.

Fixing a stuck rename

A developer with production shell access should run the fixStuckGlobalRename.php script on the deployment server (and not on maintenance server, as before) to requeue the job: 

mwscript-k8s --comment="<Phab ticket>" -f -- extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=<...> --logwiki=metawiki '<old username>' '<new username>'

Former workflow

NOTE: The old way of doing this on the maintenance server still works for now but will go way soon. See the command below. 

mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=<...> --logwiki=metawiki '<old username>' '<new username>'
  • comment: a description of the action being performed. You can just link to the phabricator ticket of the task as the comment (if it has one).
  • wiki: the database name of the wiki where the rename job has become stuck, e.g. ptwiki. If it is stuck on multiple wikis, use the one that comes first alphabetically.
  • logwiki: the database name of the wiki where the global rename was started. This will always be metawiki.
  • --ignorestatus is a parameter that can be added to fix a global rename that's stuck in running state even though the job is not actually running (so trying to run the script without this switch gives skipping duplicate rename from...). This used to happen a lot in the past due to fragile error handling; shouldn't be the case anymore. Use with care; could make a mess if the job really is running.

Once you are done, log in the production SAL that you've run the script, by issuing an IRC command in #wikimedia-operations such as:

  • !log <Phabricator task ID> Ran mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=ptwiki --logwiki=metawiki 'OldName' 'NewName'

In case of doubt, ask the maintainers of CentralAuth.

Debugging a stuck rename

We don't usually do this because it is time-consuming and these days stuck renames are pretty much one-off infrastructure issues (jobs getting lost, which cannot be proven or disproven via logs). But if there are more stuck renames than usual and you suspect something is wrong, or requeueing doesn't help, check Logstash:

  • Searching for the old or new username in the CentralAuth log channel (dashboard) should catch the more obvious things and narrow down the time range (based on when the job ran on the preceding wikis).
  • Searching for the job runner URL might reveal more elusive errors like exceptions and DB timeouts (dashboard; try to filter by wiki and/or time range).
Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Stuck_global_renames&oldid=2230425"