Incidents/20151005-MediaWiki
Appearance
Summary
Typo in MediaWiki config change took all of the wikis down for around a minute.
Timeline
Times are approximate and based on Alex's IRC logs.
- 23:37:35 Alex merges 243728, to change mw:Extension:WikidataPageBanner config to include all namespaces on ruwikivoyage
- The change is not ideal, but looks harmless enough and passes automated checks (e.g. syntax)...
- 23:37:52 sync-file starts
- 23:38:09 sync-file finishes
- 23:38:34 First report of fatals in #wikimedia-operations from MaxSem, users begin to notice in other channels
- 23:38:57 Alex resets on tin, starts sync-file again
- 23:39:14 sync-file finishes
Conclusions
- There are exactly two 'r's in
array
. - Adding an extra r (
arrray()
) will form a valid PHP function name so passes a PHP syntax check.
Actionables
Explicit next steps to prevent this from happening again as much as possible, with Phabricator tasks linked for every step.
- mediawiki-config needs better test coverage - task T114725