Incident documentation/20151005-MediaWiki

From Wikitech
Jump to: navigation, search

Summary

Typo in MediaWiki config change took all of the wikis down for around a minute.

Timeline

Times are approximate and based on Alex's IRC logs.

  • 23:37:35 Alex merges 243728, to change mw:Extension:WikidataPageBanner config to include all namespaces on ruwikivoyage
    • The change is not ideal, but looks harmless enough and passes automated checks (e.g. syntax)...
  • 23:37:52 sync-file starts
  • 23:38:09 sync-file finishes
  • 23:38:34 First report of fatals in #wikimedia-operations from MaxSem, users begin to notice in other channels
  • 23:38:57 Alex resets on tin, starts sync-file again
  • 23:39:14 sync-file finishes

Conclusions

  • There are exactly two 'r's in array.
  • Adding an extra r (arrray()) will form a valid PHP function name so passes a PHP syntax check.

Actionables

Explicit next steps to prevent this from happening again as much as possible, with Phabricator tasks linked for every step.

  • Status:    Not done - mediawiki-config needs better test coverage - Task T114725