Incidents/2018-12-12 Train-1.33.0-wmf.8

From Wikitech

Summary

There were several problems with the 2018-12-11 1.33.0-wmf.8 train (phab:T206662) resulting in 1 rollback.

Stats

Issues

  • phab:T211769 Wikimedia\Rdbms\Database::selectSQLText called from ApiBase::filterIDs with incorrect parameters: $conds must be a string or an array
  • phab:T211774 Full table scans on oldimage table
  • phab:T211805 Call to a member function format() on a non-object (boolean)
  • phab:T211806 Passing in the "body" request option as an array to send a POST request has been deprecated
  • phab:T211885 ErrorException from line 47 of /srv/mediawiki/php-1.33.0-wmf.8/extensions/Kartographer/includes/ApiQueryMapData.php: PHP Warning: data error
  • phab:T211886 Fatal error: request has exceeded memory limit in /srv/mediawiki/php-1.33.0-wmf.8/vendor/guzzlehttp/psr7/src/Stream.php on line 97

Timeline

This is a step by step outline of what happened to cause the incident and how it was remedied.

2018-12-11

  • 13:57 zeljkof: starting branch cut for 1.33.0-wmf.8
  • 14:43 <zfilipin@deploy1001> Pruned MediaWiki: 1.33.0-wmf.1 (duration: 11m 40s)
  • 14:46 <zfilipin@deploy1001> Pruned MediaWiki: 1.33.0-wmf.2 (duration: 03m 12s)
  • 14:50 <zfilipin@deploy1001> Pruned MediaWiki: 1.33.0-wmf.3 (duration: 02m 51s)
  • 15:31 <zfilipin@deploy1001> Finished scap: testwiki to php-1.33.0-wmf.8 and rebuild l10n cache (duration: 36m 08s)
  • 15:49 <zfilipin@deploy1001> rebuilt and synchronized wikiversions files: group0 to 1.33.0-wmf.8

End of Day Summary

  • Group0: 1.33.0-wmf.8
  • Group1: 1.33.0-wmf.6
  • Group2: 1.33.0-wmf.6
  • Status: Yes Done

2018-12-12

  • 14:16 <zfilipin@deploy1001> Synchronized php: group1 wikis to 1.33.0-wmf.8 (duration: 00m 51s)
  • 14:29 <zfilipin@deploy1001> rebuilt and synchronized wikiversions files: Revert "group1 wikis to 1.33.0-wmf.8"
  • 14:35 phab:T211769 Wikimedia\Rdbms\Database::selectSQLText called from ApiBase::filterIDs with incorrect parameters: $conds must be a string or an array
  • 15:31 phab:T211769 Full table scans on oldimage table
  • 15:37 phab:T211769 Wikimedia\Rdbms\Database::selectSQLText called from ApiBase::filterIDs with incorrect parameters: $conds must be a string or an array
  • 17:02 <zfilipin@deploy1001> Synchronized php: group1 wikis to 1.33.0-wmf.8 (duration: 00m 50s)
  • 17:11 phab:T211769 Full table scans on oldimage table
  • 18:35 phab:T211805 Call to a member function format() on a non-object (boolean)
  • 18:47 phab:T211806 Passing in the "body" request option as an array to send a POST request has been deprecated
  • 21:48 phab:T211805 Call to a member function format() on a non-object (boolean)

End of Day Summary

  • Group0: 1.33.0-wmf.8
  • Group1: 1.33.0-wmf.8
  • Group2: 1.33.0-wmf.6
  • Status: Yes Done

2018-12-13

  • 01:00 phab:T211806 Passing in the "body" request option as an array to send a POST request has been deprecated
  • 14:21 <zfilipin@deploy1001> rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.8
  • 14:46 phab:T211885 ErrorException from line 47 of /srv/mediawiki/php-1.33.0-wmf.8/extensions/Kartographer/includes/ApiQueryMapData.php: PHP Warning: data error
  • 15:03 phab:T211886 Fatal error: request has exceeded memory limit in /srv/mediawiki/php-1.33.0-wmf.8/vendor/guzzlehttp/psr7/src/Stream.php on line 97
  • 22:40 phab:T211886 Fatal error: request has exceeded memory limit in /srv/mediawiki/php-1.33.0-wmf.8/vendor/guzzlehttp/psr7/src/Stream.php on line 97

End of Day Summary

  • Group0: 1.33.0-wmf.8
  • Group1: 1.33.0-wmf.8
  • Group2: 1.33.0-wmf.8
  • Status: Yes Done

2018-12-18

  • 11:36 phab:T211885 ErrorException from line 47 of /srv/mediawiki/php-1.33.0-wmf.8/extensions/Kartographer/includes/ApiQueryMapData.php: PHP Warning: data error

Conclusions

What weakness did we learn about and how can we address them?

Links to relevant documentation

Actionables

Explicit next steps to prevent this from happening again as much as possible, with Phabricator tasks linked for every step.

NOTE: Please add the #wikimedia-incident Phabricator project to these follow-up tasks and move them to the "follow-up/actionable" column.

  • To do #1 (TODO: Create task)
  • To do #2 (TODO: Create task)