Server Admin Log/Archive 29

From Wikitech

2016-05-31

  • 23:57 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Set collation to uca-it for it.wikipedia (T136647) (duration: 00m 25s)
  • 23:54 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.3/extensions/GeoData/includes/Hooks.php: Don't index non-Earth coordinates (T136559) (duration: 00m 23s)
  • 23:47 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.4/extensions/GeoData/includes/Hooks.php: Don't index non-Earth coordinates (T136559) (duration: 00m 24s)
  • 23:45 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.3/extensions/VisualEditor/modules/ve-mw/init/ve.init.MWWelcomeDialog.js: ve.init.MWWelcomeDialog: Fix keyboard focus on dialog actions (T135808) (duration: 00m 22s)
  • 23:43 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.4/extensions/VisualEditor/modules/ve-mw/init/ve.init.MWWelcomeDialog.js: ve.init.MWWelcomeDialog: Fix keyboard focus on dialog actions (T135808) (duration: 00m 23s)
  • 23:37 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.3/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: Turn off textcat subtest of search satisfaction (T134319) (duration: 00m 23s)
  • 23:33 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.4/extensions/Echo/modules/ui/mw.echo.ui.NotificationItemWidget.js: Adjust styling for Special:Notification items (T136572, 2/2) (duration: 00m 24s)
  • 23:33 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.4/extensions/Echo/modules/styles/: Adjust styling for Special:Notification items (T136572, 1/2) (duration: 00m 30s)
  • 23:28 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.4/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: Turn off textcat subtest of search satisfaction (T134319) (duration: 00m 30s)
  • 22:38 tgr: running "mwscript sql.php --wiki=zerowiki /srv/mediawiki/php-1.28.0-wmf.4/maintenance/archives/patch-bot_passwords.sql" for T135074
  • 21:47 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.28.0-wmf.4
  • 21:43 logmsgbot: thcipriani@tin Synchronized wmf-config/abusefilter.php: Workaround for T136644 (duration: 00m 30s)
  • 21:34 awight: Enable PayPal Express Checkout in paymentswiki config.
  • 21:33 urandom: Bouncing restbase on restbase1015.eqiad.wmnet
  • 21:32 urandom: Bouncing restbase on restbase1013.eqiad.wmnet
  • 21:32 urandom: Bouncing restbase on restbase1012.eqiad.wmnet
  • 21:30 urandom: Bouncing restbase on restbase1009.eqiad.wmnet
  • 21:29 urandom: Bouncing restbase on restbase1008.eqiad.wmnet
  • 21:28 urandom: Bouncing restbase on restbase1010.eqiad
  • 20:58 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 back to 1.28.0-wmf.3
  • 20:55 thcipriani: rolling back group0 wmf.4 for T136644 too much log spam
  • 20:39 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.28.0-wmf.4
  • 20:35 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.3/extensions/Wikidata/extensions/Wikibase/view/resources/jquery/ui/jquery.ui.tagadata.js: Update Wikibase (duration: 00m 23s)
  • 20:33 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.4/extensions/Wikidata/extensions/Wikibase/view/resources/jquery/ui/jquery.ui.tagadata.js: Update Wikibase (duration: 00m 30s)
  • 20:28 bblack: depooled reboot of cp3043 - T126062
  • 20:23 bblack: depooled reboot of cp3042 - T126062
  • 20:13 logmsgbot: thcipriani@tin Finished scap: testwiki to php-1.28.0-wmf.4 and rebuild l10n cache (duration: 50m 09s)
  • 20:10 urandom: disabling cql binary transport on restbase1007-c
  • 20:09 bblack: depooled reboot of cp3041 - T126062
  • 20:02 bblack: depooled reboot of cp3033 (not 3032) - T126062
  • 20:02 bblack: depooled reboot of cp3032 - T126062
  • 19:57 bblack: depooled reboot of cp3031 - T126062
  • 19:50 bblack: depooled reboot of cp3030 - T126062
  • 19:45 bblack: restarting cp* nginxes for config update
  • 19:23 logmsgbot: thcipriani@tin Started scap: testwiki to php-1.28.0-wmf.4 and rebuild l10n cache
  • 19:14 urandom: Restart restbase1007-c.eqiad.wmnet because reasons
  • 18:13 Krinkle: mwscript deleteEqualMessages.php --wiki hrwikibooks (T45917)
  • 18:07 ejegg: updated payments-wiki from e6807395d7687d521070b83d159b77b242e5c04f to 5bb160e9898224e1d7d0a5c57fe408edb998a262
  • 17:26 jynus: restarting mysqls at sanitarium, some transitional lag on labs
  • 17:17 bblack: depooled reboot of cp3040 - T126062
  • 17:03 thcipriani: starting branch-cut for mediawiki and extensions for version 1.28.0-wmf.4
  • 16:52 elukey: disabling puppet on mc10* hosts as prep step for https://gerrit.wikimedia.org/r/#/c/291916. Memcached 1.4.25 will be deployed to mc1010 as part of a perf. test (T129963)
  • 16:15 urandom: Upgrade of restbase1007-{a,b,c} complete : T126629
  • 16:10 urandom: Upgrading Cassandra to 2.2.6 on restbase1007.eqiad.wmnet : T126629
  • 16:07 bblack: depooling cp3032 to investigate T126062
  • 16:06 urandom: Stopping restbase1007-{a,b,c} in preparation for upgrade : T126629
  • 15:51 urandom: Disabling puppet in preparation for upgrade on restbase1007, 1010, and 1011 : T126629
  • 15:10 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.3/extensions/EventBus/EventBus.hooks.php: SWAT: Replace wfUrlEncode with rawurlencode (duration: 00m 27s)
  • 15:07 moritzm: rebooting mendelevium (ticket.wikimedia.org) for update to Linux 4.4
  • 15:06 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Math: Enable MathML everywhere but private wikis (duration: 00m 34s)
  • 15:03 bblack: restarting all frontend caches for new memory params (randomized order, ~1-min spacing, ~2h to completion) - T135384
  • 14:54 moritzm: rolling reboot of scb systems in codfw for Linux 4.4 upgrade
  • 12:21 moritzm: continue rolling reboot of mc2* systems for Linux 4.4 upgrade
  • 08:50 mobrovac: restbase deploy end of fcd62e1
  • 08:34 mobrovac: restbase deploy start of fcd62e1
  • 08:04 jynus: db1049> megacli -PDOnline -PhysDrv '[32:4]' -a0
  • 07:46 mobrovac: change-prop deploying 980f65c
  • 07:15 jynus: db1049> megacli -PDOffline -PhysDrv '[32:4]' -a0
  • 06:20 moritzm: upgrading hhvm in eqiad (also picking up updated versions of icu and lcms)
  • 02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue May 31 02:31:36 UTC 2016 (duration 5m 54s)
  • 02:25 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.3) (duration: 09m 30s)

2016-05-30

  • 23:04 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Update BetaFeatures whitelist (duration: 00m 32s)
  • 18:50 Krinkle: mwscript deleteEqualMessages.php --wiki nvwiki (T45917)
  • 18:15 logmsgbot: jzerebecki@tin Synchronized php-1.28.0-wmf.3/extensions/Wikidata/vendor/wikibase/javascript-api/WikibaseJavaScriptApi.php: Wikidata WikibaseJavaScriptApi: Fix getLocationAgnosticMwApi behavior in Internet Explorer b6ae82c71af3d9361cfb9e8d4e6e45bcd5ee9b26 2 of 2 T136543 (duration: 00m 24s)
  • 18:14 logmsgbot: jzerebecki@tin Synchronized php-1.28.0-wmf.3/extensions/Wikidata/vendor/wikibase/javascript-api/src/getLocationAgnosticMwApi.js: Wikidata WikibaseJavaScriptApi: Fix getLocationAgnosticMwApi behavior in Internet Explorer b6ae82c71af3d9361cfb9e8d4e6e45bcd5ee9b26 1 of 2 T136543 (duration: 00m 26s)
  • 17:41 ori: Synced composer.{json,lock} and multiversion for I5ac86f190b
  • 17:27 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.3/includes/api/ApiQueryRevisions.php: T136375 (duration: 00m 52s)
  • 15:37 logmsgbot: dcausse@tin Synchronized wmf-config/CommonSettings.php: Make VE RB URLs domain-relative (duration: 00m 26s)
  • 15:17 logmsgbot: dcausse@tin Synchronized wmf-config/InitialiseSettings.php: Changetags should be granted only to sysops and bots in ruwiki (duration: 00m 26s)
  • 15:10 logmsgbot: dcausse@tin Synchronized wmf-config: Send wmf.4 search and ttmserver traffic to codfw (duration: 00m 33s)
  • 14:07 moritzm: rolling reboot of mc2* to Linux 4.4
  • 13:53 logmsgbot: jmm@tin Synchronized wmf-config/CommonSettings.php: firejail security hardening for image scalers (duration: 00m 38s)
  • 13:52 moritzm: enable firejail on image scalers
  • 12:59 gehel: disabling warmers elasticsearch codfw cluster (T133125)
  • 12:53 hashar: Upgrading Zuul 1cc37f7..66c8e52 T128569
  • 12:35 gehel: re-enabling puppet on elasticsearch codfw cluster (T133125)
  • 12:07 gehel: nginx restarted on elasticsearch codfw cluster (T133125)
  • 11:42 moritzm: upgrading hhvm in codfw (also picking up updated versions of icu and lcms)
  • 10:57 moritzm: upgrading hhvm on remaining canaries (also picking up updated versions of icu and lcms)
  • 10:35 hashar: Restarted Zuul.
  • 10:31 moritzm: upgrading hhvm on mw1017 (also picking up updated versions of icu and lcms)
  • 10:30 hashar: Zuul deadlocked :(
  • 10:12 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Reduce db1071 load (duration: 00m 48s)
  • 09:57 moritzm: installing libidn security updates on jessie systems
  • 09:09 gehel: shutting down elasticsearch on codfw for upgrade (T133125)
  • 08:44 volans: Align thread_pool_max_threads to my.cnf value on 1 slave/shard in eqiad (db1065,db1076,db1078,db1040,db1026,db1061,db1039) T133333
  • 08:27 gehel: starting elasticsearch upgrade on codfw (T133125)
  • 02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon May 30 02:30:46 UTC 2016 (duration 5m 52s)
  • 02:24 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.3) (duration: 09m 04s)

2016-05-29

  • 02:58 urandom: Bootsrapping restbase2004-b.codfw.wmet : T134016
  • 02:27 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun May 29 02:27:27 UTC 2016 (duration 5m 27s)
  • 02:22 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.3) (duration: 08m 35s)

2016-05-28

  • 19:47 hoo: Updated Wikidata's property suggester with data from Monday's json dump and removed the external identifiers as a workaround for T132839
  • 13:59 urandom: Bootstrapping restbase1011-c.eqiad.wmnet : T134016
  • 02:26 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat May 28 02:26:10 UTC 2016 (duration 5m 47s)
  • 02:20 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.3) (duration: 08m 20s)

2016-05-27

  • 23:58 ori: Forcing a Puppet run on logstash*
  • 23:44 legoktm: ran extensions/GlobalBlocking/fixGlobalBlockWhitelist.php for T56496
  • 23:15 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.3/includes/actions/RollbackAction.php: RollbackAction: Don't return true, causes '1' to be output (duration: 00m 34s)
  • 22:50 legoktm: mwscript initSiteStats.php --wiki=lrcwiki --update for T109635
  • 19:49 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.3/includes/Linker.php: Iba17ce55ff9 (duration: 00m 25s)
  • 19:49 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.3/includes/actions/RollbackAction.php: Iba17ce55ff9 (duration: 00m 31s)
  • 19:06 paravoid: un-draining esams
  • 18:15 thcipriani: restarted zuul due to deadlock issue
  • 17:10 volans: Stop slave, stop mysql and shutdown es2017 and es2019 for hardware maintenance T130702
  • 16:35 dcausse: elasticsearch in codfw: creating jamwiki index
  • 16:34 volans: Align runtime MySQL max_connections on codfw masters with the my.cnf ones T133333
  • 16:04 paravoid: shutting down ms-fe3001/ms-fe3002
  • 15:26 paravoid: draining esams for network maintenance
  • 15:02 logmsgbot: krenair@tin Synchronized php-1.28.0-wmf.3/extensions/CentralNotice/resources/subscribing: rv due to T136387 (duration: 00m 36s)
  • 14:36 paravoid: restarting pybal on lvs3003/lvs3004
  • 14:19 mobrovac: change-prop deployed 3747ebd
  • 13:54 urandom: Bootstrapping restbase2003-b.codfw.wmnet : T134016
  • 12:27 hashar: CI/Zuul deadlocked quickly due to a dependency set on a repository not known to Zuul
  • 12:22 volans: Align runtime MySQL configurations on codfw slaves with the my.cnf ones T133333
  • 11:27 elukey: restarted jmxtrans on kafka10* hosts
  • 10:11 volans: restarting MySQL on db2038 to test change 286858 - T133333
  • 10:04 moritzm: disable firejail test on mw1153, all went well, but rather revert back since it's Friday and enable this along with the other image scalers on Monday
  • 09:14 moritzm: enable firejail for image scaling on mw1153 as a canary
  • 08:57 volans: Set sync_binlog=1 on db2011 (m2) and monitoring it. T133333
  • 08:00 elukey: restarted memcached on mc1009 to collect metrics for T129963
  • 07:04 _joe_: updating HHVM on the remaining hosts (mira, wasat, snapshot1*)
  • 02:33 ejegg: rolled back payments from de0398a244094f2bad6bc70eefce8388a616e575 to e6807395d7687d521070b83d159b77b242e5c04f
  • 02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri May 27 02:31:01 UTC 2016 (duration 9m 12s)
  • 02:21 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.3) (duration: 07m 50s)
  • 01:18 ejegg: updated payments from e6807395d7687d521070b83d159b77b242e5c04f to de0398a244094f2bad6bc70eefce8388a616e575
  • 00:48 mutante: purging pk.wikimedia.org from varnish, cache_text non-eqiad backends, then frontends
  • 00:48 logmsgbot: krenair@tin Synchronized php-1.28.0-wmf.3/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.ArticleTargetLoader.js: https://gerrit.wikimedia.org/r/#/c/291145/ (duration: 00m 23s)
  • 00:30 logmsgbot: krenair@tin Synchronized php-1.28.0-wmf.3/extensions/WikimediaEvents/extension.json: https://gerrit.wikimedia.org/r/#/c/291143/1 (duration: 00m 28s)
  • 00:28 mutante: purging pk.wikimedia.org from varnish, cache_text eqiad backends

2016-05-26

  • 23:56 logmsgbot: krenair@tin Synchronized php-1.28.0-wmf.3/extensions/CentralNotice/resources/subscribing: https://gerrit.wikimedia.org/r/#/c/291120/1 (duration: 00m 24s)
  • 23:54 ejegg: updated SmashPig from f0bf4385afac65a27f99c5f657c3d0931c991fa8 to 90757321a3bfa1045202e06e3dd1960a0043493a
  • 23:40 logmsgbot: krenair@tin Synchronized php-1.28.0-wmf.3/extensions/VisualEditor/modules/ve-mw/dm/models/ve.dm.MWTransclusionModel.js: https://gerrit.wikimedia.org/r/#/c/290994/ (duration: 00m 25s)
  • 23:36 logmsgbot: krenair@tin Synchronized php-1.28.0-wmf.3/extensions/Math/modules/ve-math/ve.ui.MWMathContextItem.js: touch (duration: 00m 27s)
  • 23:33 logmsgbot: krenair@tin Synchronized php-1.28.0-wmf.3/extensions/Math/modules/ve-math/ve.ui.MWMathContextItem.js: https://gerrit.wikimedia.org/r/#/c/290971/ (duration: 00m 28s)
  • 23:14 logmsgbot: krenair@tin Synchronized php-1.28.0-wmf.3/extensions/OATHAuth/special/SpecialOATHEnable.php: https://gerrit.wikimedia.org/r/#/c/291007/ (duration: 00m 39s)
  • 22:30 urandom: Bootstrapping restbase2007-c.codfw.wmnet : T134016
  • 21:59 logmsgbot: ori@tin Synchronized php-1.28.0-wmf.3/includes/api/ApiStashEdit.php: 8521b7b069: Send edit stash metrics for cache attempts (duration: 00m 25s)
  • 21:50 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.2/includes/api/ApiStashEdit.php: Bail out in ApiStashEdit for bots for sanity (duration: 00m 24s)
  • 21:49 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.3/includes/api/ApiStashEdit.php: Bail out in ApiStashEdit for bots for sanity (duration: 00m 25s)
  • 21:43 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.3/includes/title/MediaWikiTitleCodec.php: TitleParser: In formatTitle(), don't throw exceptions on bad namespaces - T136352, T136356 (duration: 00m 28s)
  • 21:12 mutante: running update-ubuntu-mirror on carbon to check for T136307
  • 20:38 logmsgbot: twentyafterfour@tin Synchronized php-1.28.0-wmf.3/includes/specials/SpecialSearch.php: deploy hotfix for itwiki search T136356 (duration: 00m 23s)
  • 19:41 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.3
  • 19:31 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.3/includes/api/ApiStashEdit.php: 9a9ec26d25 (duration: 00m 24s)
  • 19:28 Dereckson: mwscript initSiteStats.php --wiki fowiki --update (T136353)
  • 17:56 mdholloway: mobileapps finished deploying 5ce4f31 (n.b. last deployment, on 23 May, appears to have re-deployed b8c396a)
  • 17:37 mdholloway: starting mobileapps deployment
  • 15:45 jynus: applying schema change to s3 hosts echo wikis T135699
  • 15:31 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.3/resources/src/mediawiki.special/mediawiki.special.search.css: SWAT: [[gerrit:290710|Fix regression: text color in .mw-search-result-data (duration: 00m 27s)
  • 15:27 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: CommonSettings: cleanup temp cache file if rename fails (duration: 00m 30s)
  • 15:20 kart_: Update cxserver to b431aef
  • 15:12 urandom: Starting cleanup of restbase1012-a.eqiad.wmnet
  • 15:07 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.3/extensions/EventBus: SWAT: Use getPrefixedURL and getPrefixedDBkey instead of getText (duration: 00m 35s)
  • 15:06 urandom: Bootstrapping restbase1014-c.eqiad.wmnet : T134016
  • 13:52 jynus: restarting es2017 for kernel upgrade
  • 12:39 jynus: powercycling es2017
  • 12:31 jynus: updating user table on labswiki to fix incorrect encoding T131630
  • 12:07 moritzm: rolling reboot of restbase-test cluster
  • 11:55 moritzm: rebooting mx2001 for kernel update to Linux 4.4
  • 09:58 _joe_: starting updateCollations.php forced run on all wikis with uca category collation
  • 09:28 _joe_: all traffic serving appservers are now running with libicu52 (T86096)
  • 09:07 jynus: converting user table on labswiki to utf8
  • 09:05 mobrovac: restbase deployment end of bd38b1b
  • 08:58 logmsgbot: dcausse@tin Synchronized wmf-config/CirrusSearch-labs.php: Cirrus: disable the safeifier in labs (duration: 00m 25s)
  • 08:55 mobrovac: restbase deployment start of bd38b1b
  • 08:53 logmsgbot: dcausse@tin Synchronized wmf-config/CirrusSearch-labs.php: Cirrus: disable the safeifier in labs (duration: 02m 36s)
  • 08:45 moritzm: powercycling snapshot1004 (stuck after reboot)
  • 08:36 moritzm: installing openssh security updates on trusty systems
  • 08:15 _joe_: upgrading hhvm on eqiad's appserver cluster, (T86096)
  • 07:50 _joe_: upgrading hhvm on eqiad's api cluster, (T86096)
  • 07:43 mobrovac: restbase starting partial mobile-sections dump of enwiki for T135571 on restbase1009
  • 07:36 _joe_: upgrading hhvm on eqiad jobrunners, tin + terbium (T86096)
  • 07:31 dcausse: elastic: updating cirrussearch warmers on eqiad and codfw
  • 07:29 _joe_: upgrading hhvm on the eqiad imagescalers, T86096
  • 06:41 _joe_: upgrading hhvm on the eqiad canaries, T86096
  • 05:50 _joe_: starting upgrades of hhvm to newer libicu in codfw (T86096)
  • 03:05 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu May 26 03:05:45 UTC 2016 (duration 9m 27s)
  • 02:56 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.3) (duration: 15m 49s)
  • 02:24 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.2) (duration: 10m 33s)
  • 01:47 mutante: mw1249 - restart hhvm
  • 00:57 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.3/extensions/WikimediaMessages/i18n/wikimedia: Add i18n messages for new Support and Safety group (duration: 00m 26s)
  • 00:52 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.3/includes/api/ApiMain.php: 01e68e966413c (duration: 00m 29s)
  • 00:27 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu May 26 00:27:50 UTC 2016 (duration 10m 15s)
  • 00:17 logmsgbot: dereckson@tin scap sync-l10n completed (1.28.0-wmf.3) (duration: 16m 15s)

2016-05-25

  • 23:46 logmsgbot: dereckson@tin scap sync-l10n completed (1.28.0-wmf.2) (duration: 10m 12s)
  • 23:44 urandom: Bootstrapping restbase2005-b.codfw.wmnet : T134016
  • 22:05 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: wikidata back to 1.28.0-wmf.3
  • 22:04 twentyafterfour: re-re-verting wikidata back to wmf.3
  • 22:00 logmsgbot: twentyafterfour@tin Synchronized php-1.28.0-wmf.3: (no message) (duration: 08m 18s)
  • 21:52 twentyafterfour: syncing https://gerrit.wikimedia.org/r/#/c/290789/ and https://gerrit.wikimedia.org/r/#/c/290799/
  • 19:31 urandom: Start cleanup of restbase2001-b.codfw.wmnet : T1340116
  • 19:14 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: roll back wikidata to wmf.2
  • 19:07 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.3
  • 18:58 mobrovac: restbase running a partial mobile-sections dump of eswiki for T135571 on restbase1009
  • 18:25 bblack: restarting varnish-frontend on cp3048 with lg_dirty_multi:6 unpuppetized - T135384
  • 18:04 logmsgbot: aaron@tin Synchronized wmf-config/db-eqiad.php: Fix slave lag wait calls for read-only ES clusters (duration: 00m 27s)
  • 18:03 logmsgbot: aaron@tin Synchronized wmf-config/db-codfw.php: Fix slave lag wait calls for read-only ES clusters (duration: 00m 23s)
  • 18:02 logmsgbot: aaron@tin scap aborted: file wmf-config/db-codfw.php Fix slave lag wait calls for read-only ES clusters (duration: 02m 53s)
  • 18:00 logmsgbot: aaron@tin Started scap: file wmf-config/db-codfw.php Fix slave lag wait calls for read-only ES clusters
  • 17:56 bblack: rolling restart of global text frontend memory caches for upsizing - reduce spacing to 5 mins, will finish notably faster - T135384
  • 17:25 logmsgbot: dpatrick@tin Synchronized wmf-config/InitialiseSettings.php: Enabling OATHAuth on CentralAuth wikis (duration: 00m 24s)
  • 16:40 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1056 after hardware maintenance (duration: 00m 23s)
  • 16:13 jynus: reloading haproxy config to repool db1047
  • 16:13 volans: kill very long query on db1047 (ID 89274525, client disconnected) T136214
  • 15:54 hashar: puppet fails on gallium (Precise) due to E: Unable to locate package firejail
  • 15:33 bblack: rolling restart of global text frontend memory caches for upsizing - 15 min spacing, ~8H to completion - T135384
  • 15:28 jynus: applying third schema change on x1 hosts T135699
  • 15:27 _joe_: imaging mw1261, with debian jessie
  • 15:25 logmsgbot: thcipriani@tin Synchronized portals: SWAT: T136019 updating portal stats. (duration: 00m 26s)
  • 15:24 logmsgbot: thcipriani@tin Synchronized portals/prod/wikipedia.org/assets: SWAT: T136019 updating portal stats. (duration: 00m 23s)
  • 15:05 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Math: Enable MathML on all wikibooks (duration: 00m 29s)
  • 15:01 mobrovac: restbase cassandra truncated local_group_globaldomain_T_mathoid_svg.data
  • 15:00 urandom: Bootstrap restbase1012-c.eqiad.wmnet : T134016
  • 14:50 moritzm: restarted hhvm on mw1116 and mw1117 (got stuck, output of hhvm-dump-debug available)
  • 14:25 jynus: shutting down db1056 for hardware maintenance
  • 14:24 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1056 for hardware maintenance (duration: 00m 24s)
  • 14:23 urandom: Starting cleanup on restbase1010-a.eqiad.wmnet : T134016
  • 13:31 logmsgbot: oblivian@tin Synchronized wmf-config/InitialiseSettings.php: revert uca-ta collation (duration: 00m 31s)
  • 11:12 mobrovac: restbase deploy end of 8f39fa4
  • 11:02 mobrovac: restbase deploy start of 8f39fa4
  • 09:42 elukey: restarted gmetad on uranium to test if new memcached metrics would be picked up (T129963)
  • 09:26 akosiaris: mw1114's logs and ganglia indicate OOM error.
  • 09:19 akosiaris: powercycling mw1114
  • 09:02 volans: Apply grant for tendril on labservices1002 T106303
  • 09:02 akosiaris: upload php5_5.3.10-1ubuntu3.23+wmf1 on apt.wikimedia.org/precise-wikimedia
  • 08:39 jynus: perform second schema change on x1 wikis for T135699
  • 08:27 moritzm: restarting apache on uranium
  • 08:08 moritzm: installing php5 security updates on trusty systems
  • 07:41 jynus: rm runJobs.log-20160[1-3]*.gz on fluorine archive log
  • 02:00 mutante: reboot unresponse mw1140
  • 01:48 logmsgbot: dereckson@tin scap sync-l10n completed (1.28.0-wmf.2) (duration: 15m 24s)
  • 00:22 logmsgbot: dereckson@tin Synchronized /srv/mediawiki-staging/php-1.28.0-wmf.3/extensions/Flow/maintenance/FlowRemoveOldTopics.php: Don't assume workflows/revisions are inserted in chronological order (T119509) (duration: 00m 23s)
  • 00:16 logmsgbot: dereckson@tin Synchronized /srv/mediawiki-staging/php-1.28.0-wmf.2/extensions/Flow/maintenance/FlowRemoveOldTopics.php: Don't assume workflows/revisions are inserted in chronological order (T119509) (duration: 00m 28s)

2016-05-24

  • 23:58 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.2/extensions/MobileFrontend/tests/phpunit/MobileFormatterTest.php: "Pi" article on mobile en.wp throws a 503 fatal (no-op, Gerrit:290587) (duration: 00m 23s)
  • 23:58 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.2/extensions/MobileFrontend/includes/MobileFormatter.php: "Pi" article on mobile en.wp throws a 503 fatal (T135923, Gerrit:290587 + Gerrit:290598) (duration: 00m 24s)
  • 23:46 Dereckson: scap pull on mw1140 (duration: 02m 42s)
  • 23:20 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Set Tamil projects to use uca-ta collation (T75453) (duration: 02m 18s)
  • 23:07 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Adding WMF Support and Safety user groups to meta (T136046) (duration: 00m 26s)
  • 22:29 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.28.0-wmf.3
  • 22:25 logmsgbot: twentyafterfour@tin scap sync-l10n completed (1.28.0-wmf.3) (duration: 00m 37s)
  • 22:21 logmsgbot: twentyafterfour@tin Finished scap: sync testwiki to wmf/1.28.0-wmf.3 (duration: 41m 48s)
  • 21:40 logmsgbot: twentyafterfour@tin Started scap: sync testwiki to wmf/1.28.0-wmf.3
  • 21:24 volans: set innodb_flush_log_at_trx_commit=0 on db1056 that is lagging behind as a temporary measure
  • 21:13 Krinkle: mwscript deleteEqualMessages.php --wiki mrwiki (T45917)
  • 21:08 mutante: mw1137,mw1146 restarted hhvm service
  • 20:34 mutante: ocg1003 - reinstalled, replaced puppet cert, salt key..re-added
  • 20:07 Krinkle: mwscript deleteEqualMessages.php --wiki diqwiki (T45917)
  • 19:55 Krinkle: mwscript deleteEqualMessages.php --wiki cywiktionary (T45917)
  • 19:50 mutante: mw1133 - Could not allocate memory, restarted hhvm service, ran puppet
  • 19:48 mutante: mw1142 - Could allocate memory on puppet run, restarted hhvm service, that fixed it
  • 19:44 Krinkle: mwscript deleteEqualMessages.php --wiki warwiki (T45917)
  • 19:14 mutante: ocg1003, powercycle for reinstall, scheduled downtime
  • 19:06 urandom: Performing cleanup on restbase2001-a.codfw.wmnet
  • 19:05 urandom: Performing cleanup on restbase2003-a.codfw.wmnet
  • 18:59 urandom: Performing cleanup on restbase2008-a.codfw.wmnet
  • 18:55 urandom: Performing cleanups on restbase2004.codfw.wmnet
  • 18:28 urandom: Starting bootstrap of restbase1010-c.eqiad.wmnet : T134016
  • 18:08 urandom: Starting bootstrap of restbase2006-b.codfw.wmnet : T95253
  • 17:38 elukey: soft rebooted mw1134 due to unresponsiveness (no root login, no ssh login, memory exhaustion from server-board)
  • 17:22 yurik: deployed & restarted kartotherian & tilerator. https://gerrit.wikimedia.org/r/#/c/290494/ https://gerrit.wikimedia.org/r/#/c/290497/
  • 16:11 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Compact Language Links as default in Beta PART II (duration: 00m 25s)
  • 16:10 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Enable Compact Language Links as default in Beta PART I (duration: 00m 23s)
  • 16:05 Krinkle: mwscript deleteEqualMessages.php --wiki fiwikinews (T45917)
  • 15:57 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Enable Compact Language Links as default in Beta PART II (duration: 00m 25s)
  • 15:57 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Compact Language Links as default in Beta PART I (duration: 00m 23s)
  • 15:26 _joe_: shutting down mw2001-2060
  • 15:11 logmsgbot: thcipriani@tin Synchronized wmf-config/Wikibase.php: SWAT: Make entityNamespaces setting available to Wikibase Client wikis (duration: 00m 29s)
  • 15:01 _joe_: revoking certs, puppet facts, salt keys for mw2001-60
  • 14:01 jynus: dropping labswiki.bounce_records on db1-BETA
  • 13:29 elukey: restarted hhvm on mw1145 (ran hhvm-dump-debug, output available in hhvm.14155.bt)
  • 13:23 moritzm: reverted net.netfilter.nf_conntrack_tcp_timeout_time_wait on kafka1013 back to 65 (as it should have been set by sysctl.d)
  • 12:49 elukey: stopping kafka on kafka1013 and rebooting the host for kernel upgrade
  • 12:33 jynus: deploying GTID to all codfw db production hosts
  • 12:10 _joe_: cleaning puppet facts, salt keys for mw2001-2060
  • 11:53 mobrovac: restbase resting nodes to pick up https://gerrit.wikimedia.org/r/#/c/290264/
  • 11:22 jynus: enabling GTID on s1 codfw db servers
  • 11:21 moritzm: rolling restart of elasticsearch in logstash cluster to pick up openjdk security update
  • 10:50 _joe_: disabling puppet on mw2001-60 (minus 2017) for decommissioning
  • 10:32 jynus: creating backup of beta database just in case T119567
  • 10:05 Nemo_bis: Phabricator search by keyword (aka field "contains words") has been down about 20 min, looks ok now.
  • 09:51 godog: reenable puppet on graphite1001 T135385
  • 09:51 jynus: applying schema change to x1 hosts T135699
  • 09:18 godog: reboot restbase2003 for multi-instance conversion T113714
  • 08:45 godog: reboot restbase2006 for multi-instance conversion T113714
  • 08:12 moritzm: enabled base::firewall on potassium (pool counter)
  • 07:56 mobrovac: restbase deploy end of a5d00d1
  • 07:33 mobrovac: restbase deploy start of a5d00d1
  • 07:17 mobrovac: change-prop deploying 20eda89
  • 02:25 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.2) (duration: 11m 38s)

2016-05-23

  • 23:36 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Enable Flow beta feature on frwikivoyage (T135702) (duration: 00m 25s)
  • 23:26 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.2/extensions/Flow/maintenance/FlowRemoveOldTopics.php: More reliable post sorting (T119509, 2/2) (duration: 00m 34s)
  • 23:24 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.2/extensions/Flow/includes/Data/Index/BoardHistoryIndex.php: More reliable post sorting (T119509, 1/2) (duration: 00m 26s)
  • 21:36 andrewbogott: reimaging holmium to labservices1002.
  • 21:14 ejegg: updated payments from 76b5c559e08b340fb3d66254b27f624a9c0a4b95 to e6807395d7687d521070b83d159b77b242e5c04f
  • 21:11 ejegg: rolled back payments to 76b5c559e08b340fb3d66254b27f624a9c0a4b95
  • 21:09 ejegg: updated payments from 76b5c559e08b340fb3d66254b27f624a9c0a4b95 to e6807395d7687d521070b83d159b77b242e5c04f
  • 20:52 awight: Update SmashPig from 5cadcf3abcfcda4552b068c783337d82b743e2e5 to f0bf4385afac65a27f99c5f657c3d0931c991fa8
  • 20:36 bearND: mobileapps deployed cd76f5a
  • 20:18 bearND: starting mobileapps deploy
  • 19:55 gehel: putting wdqs1001 out of maintenance
  • 19:54 gehel: deployed latest WDQS version
  • 19:35 hashar: killed all mysqld process on Trusty CI slaves
  • 19:32 gehel: putting wdqs1001 in maintenance to fix deployment issues
  • 19:11 legoktm: manually kicking stuck global renames (T135656)
  • 19:01 awight: rollback SmashPig from f0bf4385afac65a27f99c5f657c3d0931c991fa8 to 5cadcf3abcfcda4552b068c783337d82b743e2e5
  • 19:01 awight: update SmashPig from 5cadcf3abcfcda4552b068c783337d82b743e2e5 to f0bf4385afac65a27f99c5f657c3d0931c991fa8
  • 18:50 awight: Rollback SmashPig from aa1614afa845358669208c2f6c4cd62e83a98f4c to 5cadcf3abcfcda4552b068c783337d82b743e2e5
  • 18:49 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/290290 (duration: 00m 38s)
  • 18:39 awight: Disable Adyen SmashPig runner and just stop spamming the server admin log.
  • 18:39 awight: Reenable Adyen SmashPig job runner
  • 18:38 awight: Paused Adyen SmashPig job runner
  • 18:34 awight: update SmashPig from 5cadcf3abcfcda4552b068c783337d82b743e2e5 to aa1614afa845358669208c2f6c4cd62e83a98f4c
  • 18:25 andrewbogott: temporarily turning off pdns and recursor on holmium (https://phabricator.wikimedia.org/T106303)
  • 18:10 gehel: removing maintenance from wdqs1001
  • 17:35 gehel: putting wdqs1001 in maintenance to fix deployment issues
  • 17:13 andrewbogott: rebooting labvirt1003
  • 17:10 elukey: restarting Yarn Resource manager (master node) on analytics1001 to apply a new Spark configuration. The service will automatically failover to analytics1002
  • 16:45 bd808: Stashbot back online. Will continue to monitor for a while to see if ES cluster is happier.
  • 16:32 bd808: Stashbot down due to backing elasticsearch cluster instability. Investigating.
  • 16:28 urandom: Bouncing RESTBase on restbase-test200[1-3].codfw.wmnet
  • 16:12 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.2/extensions/Wikidata/extensions/Wikibase/client/includes/Hooks/DataUpdateHookHandlers.php: Update Wikidata - fix file deletion issue on commons (duration: 00m 29s)
  • 16:02 elukey: restarting yarn on analytics10* hosts to pick up the new Spark shuffler process
  • 16:02 volans: testing thread_pool_max_threads=2000 on db1072 (s1) [instead of db1076 (s2)] T133333
  • 15:47 volans: testing thread_pool_max_threads=2000 on db1076 (s2) T133333
  • 15:42 logmsgbot: thcipriani@tin Synchronized wmf-config: SWAT: Final Commons configuration for $wgUploadDialog (duration: 00m 28s)
  • 15:32 logmsgbot: thcipriani@tin Synchronized wmf-config: SWAT: revert Final Commons configuration for $wgUploadDialog (duration: 00m 28s)
  • 15:29 logmsgbot: thcipriani@tin Synchronized wmf-config: SWAT: Final Commons configuration for $wgUploadDialog (duration: 00m 30s)
  • 15:21 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Set interwiki sorting order for West Frisian Wikibooks (duration: 00m 25s)
  • 15:13 jynus: performing schema change on s3 T130692
  • 15:11 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Creation of page mover userright on enwiki (duration: 00m 30s)
  • 15:06 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Adjust groups permissions on fa.wikipedia (duration: 00m 41s)
  • 14:44 godog: reboot restbase2005 in single user mode for T113714
  • 14:43 logmsgbot: filippo@palladium conftool action : set/pooled=no; selector: restbase2005.codfw.wmnet
  • 12:57 moritzm: rolling restart of cassandra on maps-test cluster for openjdk security update
  • 12:44 mobrovac: restbase restarting to apply https://gerrit.wikimedia.org/r/#/c/289092/
  • 12:33 jynus: stopping, backing up and reimage db1016 T135973 (it will also affect db2010 lag)
  • 12:15 logmsgbot: filippo@palladium conftool action : set/pooled=yes; selector: restbase2009.codfw.wmnet
  • 12:15 logmsgbot: filippo@palladium conftool action : set/pooled=yes; selector: restbase2008.codfw.wmnet
  • 12:15 logmsgbot: filippo@palladium conftool action : set/pooled=yes; selector: restbase2007.codfw.wmnet
  • 11:42 mobrovac: restbase deploying 75a94ee to restbase2009
  • 11:24 godog: run puppet and roll-restart cassandra-metrics-collector on restbase codfw/eqiad
  • 11:20 godog: deploy new version of cassandra-metrics-collector T135385
  • 10:57 moritzm: restarting hhvm on app servers in codfw for librsvg update
  • 10:32 _joe_: running updateCollations.php --force on ptwiki, T58041
  • 10:24 moritzm: rolling restart of restbase1* for openjdk-8 update
  • 10:00 moritzm: reverted net.netfilter.nf_conntrack_tcp_timeout_time_wait on kafka1013 back to 65 (as set by default by puppet)
  • 08:57 jynus: stopping, backing up and reimage db2010 T135973
  • 08:44 moritzm: rolling restart of restbase2* for openjdk-8 update
  • 08:20 moritzm: install restarting hhvm on canary systems for librsvg update
  • 08:01 jynus: performing schema change on s7 T130692
  • 07:00 _joe_: installed the new hhvm package on mw2017, T86096
  • 06:55 _joe_: uploaded a new hhvm package for trusty linked to libicu52, T86096
  • 06:42 elukey: Removed Kafka temp. override for webrequest_upload retention.ms after freeing some disk space.
  • 06:31 elukey: Set kafka retention.ms=172800000 for the topic webrequest_upload to free some disk space on kafka1022
  • 02:33 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon May 23 02:33:00 UTC 2016 (duration 8m 38s)
  • 02:24 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.2) (duration: 10m 41s)

2016-05-22

  • 17:16 jynus: defragmenting db1028
  • 14:38 jynus: trying to restart kraz and planet2001 (both service and console unresponsive)
  • 14:02 jynus: performing schema change on s6 T130692
  • 02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun May 22 02:31:13 UTC 2016 (duration 8m 45s)
  • 02:22 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.2) (duration: 09m 57s)

2016-05-21

  • 02:32 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat May 21 02:32:13 UTC 2016 (duration 8m 44s)
  • 02:23 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.2) (duration: 10m 22s)
  • 01:24 ejegg: updated payments from da453121d4086d6c1218d603149e4f568db321b5 to 76b5c559e08b340fb3d66254b27f624a9c0a4b95
  • 00:34 ejegg: updated payments-wiki from 0b2d6807f68e7ccfc36a431e90d563b4d5a9e966 to da453121d4086d6c1218d603149e4f568db321b5

2016-05-20

  • 20:36 chasemp: restart rabbitmq on labcontrol1001
  • 17:05 moritzm: uploaded librsvg 2.40.2-1+wm2 for trusty-wikimedia to carbon (backported patches from librsvg DSA to our custom trusty build)
  • 16:54 bd808: Cleaned up /tmp/mw-cache-1.27.0-wmf.2* cache files on tin
  • 16:50 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri May 20 16:50:25 UTC 2016 (duration 9m 38s)
  • 16:40 logmsgbot: bd808@tin scap sync-l10n completed (1.28.0-wmf.2) (duration: 10m 06s)
  • 16:18 bd808: kicking off manual l10nupdate run on tin
  • 16:11 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.23) (duration: 20990m 58s)
  • 15:52 jynus: performing schema change on s5 T130692
  • 15:16 godog: roll-restart cassandra-metrics-collector in eqiad for T135385
  • 15:05 gehel: starting cluster rejoining for cassandra onmaps-test2001
  • 15:03 godog: roll-restart cassandra-metrics-collector in codfw for T135385
  • 15:02 moritzm: uploaded librsvg 2.40.5-1+deb8u2+wmf1 for jessie-wikimedia to carbon (rebase of locally patched package on top of latest security update)
  • 14:50 gehel: shutting down kartotherian on maps-test2001 (accidental data deletion)
  • 14:22 gehel: cassandra downgraded on maps2*.codfw.wmnet
  • 14:04 _joe_: removing libicu48 from trusty archives, kept a copy of the packages in my homedir on carbon
  • 13:45 mark: Enabled cr2-codfw et-0/* interfaces, reenabling OSPF/OSPF3
  • 13:38 mark: Bringing cr2-codfw FPC 0 back up
  • 13:37 moritzm: upgraded java on xenon/praseodymium/cerium and restbase2001 to latest openjdk-8 release (along with restarts of Cassandra)
  • 13:37 mark: Offlining cr2-codfw FPC 0
  • 13:35 jynus: changing dbstore1001 to be a direct slave of db1075
  • 13:31 mark: Disabling cr2-codfw et-0/* interfaces
  • 13:24 mark: Disabling OSPF on all cr2-codfw row subnets to drain FPC0
  • 13:14 mark: Lowering VRRP priority to 50 on all VRRP groups on cr2-codfw to drain FPC0
  • 13:07 mark: Performing acupuncture on cr2-codfw:ae4.2020 (Lowered VRRP priority from 100 to 50, inet/inet6)
  • 13:05 jynus: freeing up space on db1038 by defragmenting its tables
  • 12:31 moritzm: rolling restart of nginx in eqiad to pick up expat update
  • 12:22 moritzm: rolling restart of nginx in codfw to pick up expat update
  • 12:16 elukey: restarting cassandra on aqs100[123] for Java upgrades
  • 12:02 moritzm: rolling restart of nginx in esams to pick up expat update
  • 11:49 moritzm: rolling restart of nginx in ulsfo to pick up expat update
  • 10:40 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Increase db1029 and db1033 weight back to normal after upgrade (duration: 01m 52s)
  • 09:00 jynus: altering db1040 commonswiki.categorylinks
  • 08:50 elukey: upgrading cassandra from 2.1.12 to 2.1.13 on aqs1003.eqiad.mwnet
  • 08:45 awight: Update payments fraud config
  • 08:11 elukey: upgrading cassandra from 2.1.12 to 2.1.13 on aqs1002.eqiad.mwnet
  • 08:08 mobrovac: mathoid deploying 243a530
  • 07:36 jynus: testing medata lock detectiom on db1069
  • 06:11 bblack: restarted gdnsd on eeden.esams (with new config, esams marked down)
  • 05:58 bblack: gdnsd stopped on eeden.esams, puppet disabled
  • 05:56 volans: Killed transaction 3262258 on db1040 (alter table stuck in "Waiting for table metadata lock" blocking the replica) T130692
  • 04:24 moritzm: restarted hhvm on mw1014 (got stuck, output of hhvm-dump-debug available)
  • 00:52 mutante: stashbot test (T122690)
  • 00:07 logmsgbot: dereckson@tin Finished scap: Revert "Convert Special:WhatLinksHere from XML form to OOUI form" (Gerrit:289772, T135773) (duration: 34m 12s)

2016-05-19

  • 23:33 logmsgbot: dereckson@tin Started scap: Revert "Convert Special:WhatLinksHere from XML form to OOUI form" (Gerrit:289772, T135773)
  • 23:29 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Drop already-enabled VisualEditorNewAccountEnableProportion wikis (duration: 00m 27s)
  • 23:25 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Move VisualEditor to secondary status on English Wikipedia (T132806) (duration: 00m 29s)
  • 23:13 logmsgbot: dereckson@tin Synchronized wmf-config/throttle.php: Add Account throttle exception for SF Edit-a-thon (T135777) (duration: 00m 27s)
  • 23:10 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Enable experimental Video.js player on test2wiki (2/2) (duration: 00m 27s)
  • 23:10 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Enable experimental Video.js player on test2wiki (1/2) (duration: 00m 26s)
  • 23:09 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Enable experimental Video.js player on test2wiki (1/2) (duration: 00m 30s)
  • 22:13 logmsgbot: krenair@tin Synchronized php-1.28.0-wmf.2/extensions/VisualEditor/ApiVisualEditor.php: rv for now, have another idea for later (duration: 00m 40s)
  • 21:28 logmsgbot: krenair@tin Synchronized php-1.28.0-wmf.2/extensions/VisualEditor/ApiVisualEditor.php: more debug logs (duration: 00m 25s)
  • 21:02 yurik_: deployed tilerator service update https://gerrit.wikimedia.org/r/#/c/289736/
  • 21:01 logmsgbot: krenair@tin Synchronized php-1.28.0-wmf.2/extensions/VisualEditor/ApiVisualEditor.php: (no message) (duration: 00m 27s)
  • 20:50 logmsgbot: krenair@tin Synchronized php-1.28.0-wmf.2/extensions/VisualEditor/ApiVisualEditor.php: (no message) (duration: 00m 25s)
  • 20:49 logmsgbot: krenair@tin Synchronized php-1.28.0-wmf.2/extensions/VisualEditor/ApiVisualEditor.php: update to the debug logging to try to find other params (duration: 00m 28s)
  • 20:45 urandom: RESTBase staging Cassandra upgrade complete : T126629
  • 20:44 urandom: restbase-test2003.codfw.wmnet upgraded and online : T126629
  • 20:42 urandom: Upgrading Cassandra to 2.2.6 on restbase-test2003.codfw.wmnet : T126629
  • 20:42 urandom: Stopping Cassandra on restbase-test2003.codfw.wmnet : T126629
  • 20:38 urandom: restbase-test2002.codfw.wmnet upgraded and online : T126629
  • 20:36 logmsgbot: krenair@tin Synchronized php-1.28.0-wmf.2/extensions/VisualEditor/ApiVisualEditor.php: slight update to the debug logging to try to find the callers (duration: 00m 31s)
  • 20:36 urandom: Upgrading Cassandra to 2.2.6 on restbase-test2002.codfw.wmnet : T126629
  • 20:35 urandom: Stopping Cassandra on restbase-test2002.codfw.wmnet : T126629
  • 20:35 chasemp: iridium try to block vandal by ip temp so puppet disable and edit of /etc/apache2/phabbanlist.conf
  • 20:31 urandom: restbase-test2001.codfw.wmnet upgraded and online : T126629
  • 20:29 urandom: Upgrading Cassandra to 2.2.6 on restbase-test2001.codfw.wmnet : T126629
  • 20:28 urandom: Stopping Cassandra on restbase-test2001.codfw.wmnet : T126629
  • 20:19 urandom: praseodymium.eqiad.wmnet upgraded and online : T126629
  • 20:16 urandom: Upgrading Cassandra to 2.2.6 on praseodymium.eqiad.wmnet : T126629
  • 20:16 logmsgbot: aaron@tin Synchronized wmf-config/CommonSettings.php: Lowered to 5 (duration: 00m 29s)
  • 20:15 urandom: Stopping Cassandra on praseodymium.eqiad.wmnet : T126629
  • 20:00 urandom: Upgrading Cassandra on cerium.eqiad.wmnet, and forcing puppet run : T126629
  • 19:59 urandom: Stopping Cassanra on cerium.eqiad.wmnet for Cassandra 2.2.6 upgrade : T126629
  • 19:32 urandom: Disabling puppet in RESTBase staging : T126629
  • 19:26 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.2
  • 18:38 cscott: updated Parsoid to version 67816adf (T100681, T130638)
  • 18:27 urandom: Enabling puppet on xenon.eqiad.wmnet and forcing run : T126629
  • 18:26 urandom: Upgrading Cassandra on xenon.eqiad.wmnet to 2.2.6 : T126629
  • 18:26 cscott: synced; restarted Parsoid on wtp1001.eqiad as a canary
  • 18:25 urandom: Stopping Cassandra on xenon.eqiad.wmnet : T126629
  • 18:23 cscott: re-attempting parsoid deploy of 67816adf
  • 18:22 cscott: to cleanup Parsoid repos ori ran: salt 'wtp*' cmd.run "sed -i -e '/106801025/d' /srv/deployment/parsoid/deploy/src/lib/api/routes.js"
  • 18:13 cscott: parsoid deploy reverted to parsoid/deploy-sync-20160504-200410 tag (b0d015fa); 21 repos still dirty
  • 18:09 cscott: starting to revert Parsoid deploy due to unresolved dirty repos
  • 18:05 cscott: git-deploy of Parsoid failed with "21/44 minions completed checkout" due to dirty repos, root had applied patch during the restbase/changeprop/parsoid outage.
  • 17:39 cscott: starting Parsoid deploy (of 67816adf)
  • 17:25 elukey: execute sysctl -w net.netfilter.nf_conntrack_max=512000 on kafka1013 as temporary measure (investigating why conntrack count is higher after leader election) - T135557
  • 17:13 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1033 after maintenance with low weight; increase db1029 weight (duration: 00m 29s)
  • 16:31 volans: Set runtime value for max_allowed_packet, innodb_buffer_pool_dump_at_shutdown, innodb_buffer_pool_load_at_startup to their configured values for s1-s7, es1-es3, x1 T133333
  • 16:31 urandom: Disabling puppet on xenon.eqiad.wmnet in preparation for Cassandra upgrade : T126629
  • 16:29 elukey: upgrading cassandra from 2.1.12 to 2.1.13 on aqs1001.eqiad.mwnet
  • 15:48 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.1/extensions/VisualEditor/ApiVisualEditor.php: SWAT: Debug log strange-looking ETags being sent to RB (duration: 00m 29s)
  • 15:41 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.2/extensions/VisualEditor/ApiVisualEditor.php: SWAT: Debug log strange-looking ETags being sent to RB (duration: 00m 44s)
  • 15:16 hashar: Restarted zuul-merger daemons on both gallium and scandium : file descriptors leaked
  • 14:20 logmsgbot: volans@tin Synchronized wmf-config/db-eqiad.php: Repool db1029 (x1) with low weight - T112079 (duration: 00m 40s)
  • 14:18 akosiaris: enable puppet on maps-test200{2,3,4}.
  • 14:02 akosiaris: enabled and ran puppet on maps-test2001
  • 13:58 akosiaris: disable puppet on maps-test200{1,2,3,4} for enabling cassandra metrics collection selectively
  • 13:31 chasemp: reboot labstore1003 kernel upgrade
  • 12:54 godog: bounce carbon-c-relay on graphite1001, run with debug version
  • 12:45 elukey: restarted oozie on analytics1003 for security upgrades
  • 12:28 elukey: restarted hue on analytics1027 for security upgrades
  • 11:26 moritzm: restarting salt-master on neodymium
  • 11:09 ori: dropped negative values from mc_get_hits_rate ganglia metrics for eqiad memcached hosts by running https://phabricator.wikimedia.org/P3138
  • 10:49 volans: db1029 stop, backup and reimage T112079
  • 10:48 jynus: db1033 stop, backup and reimage T134555
  • 10:41 volans: Disable puppet on db1029 for reimaging T112079
  • 10:03 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1033 (s7 old master) & db1029 (x1-slave) for maintenance (duration: 02m 05s)
  • 09:51 moritzm: restarting apache2 on pallaium (will impose a few temporary puppet failures)
  • 09:46 moritzm: restarting apache2 on strontium (will impose a few temporary puppet failures)
  • 09:39 hashar: Restarting Jenkins
  • 09:23 kart_: updated cxserver to 4aaec58
  • 09:22 moritzm: restarting apache on neon (hosting icinga) for security update
  • 09:08 moritzm: restarting apache on silver (hosting wikitech) for security update
  • 08:35 hashar: gallium: purging old Linux kernel packages (~2.2Gbytes)
  • 08:27 moritzm: restarting apache on ytterbium (hosting gerrit.wikimedia.org) for security update
  • 08:06 moritzm: rolling restart of hhvm on mediawiki in eqiad to pick up expat security update
  • 07:17 jynus: performing schema change on s4 T130692
  • 07:04 moritzm: installed chromium security updates on osmium
  • 06:11 gehel: completed rolling restart of Elasticsearch codfw for Java update (T135499)
  • 03:10 ejegg|away: updated fundraising tools from 220afdeaa36bc3feaaff1f781e7761d7878c4ee8 to b2425aef2154d6b689900f4848cca02880321230
  • 02:47 ejegg|away: updated misc fundraising tools from e2978024e6f6b6881d087ac5d07e4c40f7374709 to 220afdeaa36bc3feaaff1f781e7761d7878c4ee8
  • 02:06 ejegg: enabled banner history queue consumer
  • 02:02 ejegg: updated civicrm from 7952ba43a012cb6a2e8d16af19bb13ed520bd56f to b7b46740d701942507dca0a98a75f3f87b6b31b1
  • 01:22 twentyafterfour: Phabricator upgrade completed and service restored.
  • 01:15 twentyafterfour: Phabricator deployment T134443 starting momentarily. Downtime should be minimal but there will be a short interruption while the service restarts.

2016-05-18

  • 23:28 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.2/extensions/Echo/: SWAT: fix URLs in notification emails (T135625) (duration: 00m 35s)
  • 23:14 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Enable Flow beta feature on outreachwiki (T135582) (duration: 00m 31s)
  • 22:23 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: second attempt: group1 to 1.28.0-wmf.2
  • 22:19 logmsgbot: twentyafterfour@tin Synchronized php-1.28.0-wmf.2/includes/deferred/LinksUpdate.php: deploy https://gerrit.wikimedia.org/r/#/c/289569/ (duration: 00m 29s)
  • 22:00 akosiaris: restart cassandra on maps-test200{2,3,4} to get logstash changing working
  • 21:39 akosiaris: enable puppet on aqs1001 aqs1002 aqs1003 aqs1004 aqs1005 aqs1006 maps-test2001 maps-test2002 maps-test2003 maps-test2004 maps2001 maps2002 maps2003 maps2004
  • 21:32 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: roll back group1 to 1.28.0-wmf.1
  • 21:25 ejegg: updated DjangoBannerStats from 8172c4176ea2b78df0dccbfa052064f2739c64bd to 220f80ece976aecbfbfbe1cb476d79910eb48ff4
  • 20:41 urandom: Renabling puppet on restbase production cluster : T126629
  • 20:27 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.2
  • 20:26 yurik: git deployed tilerator, will restart one by one manually. https://gerrit.wikimedia.org/r/#/c/289542/
  • 20:19 urandom: Rolling restart of restbase-test200[1-3].codfw.wmnet : T126629
  • 20:17 urandom: restarting Cassandra on cerium.eqiad.wmnet : T126629
  • 20:16 urandom: restarting Cassandra on praseodymium.eqiad.wmnet : T126629
  • 20:14 twentyafterfour: Deployment of 1.28.0-wmf.2 [T134450] is no longer blocked Preparing to deploy to group1 wikis.
  • 20:02 urandom: Restarting Cassandra on xenon.eqiad.wmnet post-puppet-run : T126629
  • 19:55 akosiaris: disable puppet on restbase1010 restbase1011 restbase1012 restbase1013 restbase1014 restbase1015 as well
  • 19:51 akosiaris: disabled puppet on aqs1001 aqs1002 aqs1003 aqs1004 aqs1005 aqs1006 maps-test2001 maps-test2002 maps-test2003 maps-test2004 maps2001 maps2002 maps2003 maps2004 restbase1007 restbase1008 restbase1009 restbase2001 restbase2002 restbase2003 restbase2004 restbase2005 restbase2006 restbase2007 restbase2008 restbase2009 for cassandra upgrade
  • 18:51 csteipp: added oathauth-enable right to Staff group on testwiki.
  • 18:40 urandom: Starting bootstrap of restbase2008-b.codfw.wmnet : T95253
  • 18:37 logmsgbot: csteipp@tin Synchronized wmf-config/CommonSettings.php: Enable OATH on test wikis (duration: 00m 28s)
  • 18:36 logmsgbot: csteipp@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 34s)
  • 18:35 urandom: Starting restbase2008-a.codfw.wmnet : T95253
  • 18:32 urandom: Stopping restbase2008-a.codfw.wmnet and downgrading Cassandra to 2.1.13 : T95253
  • 18:31 urandom: Stopping failed bootstrap of restbase2008-b.codfw.wmnet : T95253
  • 18:14 csteipp: created oathauth_users in centralauth db
  • 17:45 logmsgbot: twentyafterfour@tin Synchronized php-1.28.0-wmf.2/extensions/Renameuser/RenameuserSQL.php: (no message) (duration: 00m 38s)
  • 17:33 moritzm: rolling restart of hhvm on mediawiki canaries in eqiad to pick up expat security update
  • 16:01 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.2/includes/DefaultSettings.php: SWAT: Increase BotPasswordSessionProvider default priority (duration: 00m 26s)
  • 15:52 bblack: restarting varnish frontends on cache_misc + cache_maps to clear cached X-Cache entries
  • 15:43 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.2/extensions/CentralAuth/includes/CentralAuthHooks.php: SWAT: Fix central logout (duration: 00m 26s)
  • 15:32 hashar: Deleted Nodepool snapshot images created around 14:30 apparently they havent been provisioned properly. Started nodepool again. Poke T135631
  • 15:29 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wgDisableAuthManager = true (duration: 00m 25s)
  • 15:18 logmsgbot: thcipriani@tin Synchronized portals/prod/wikipedia.org/assets/js/index-1a803501de.js: SWAT: manual js sync for portals (duration: 00m 28s)
  • 15:14 logmsgbot: thcipriani@tin Synchronized portals: (no message) (duration: 00m 31s)
  • 15:14 logmsgbot: thcipriani@tin Synchronized portals/prod/wikipedia.org/assets: (no message) (duration: 00m 30s)
  • 15:12 hashar: Nodepool / labs instance issue filled as https://phabricator.wikimedia.org/T135631
  • 15:11 logmsgbot: thcipriani@tin Synchronized portals: (no message) (duration: 00m 32s)
  • 15:10 logmsgbot: thcipriani@tin Synchronized portals/prod/wikipedia.org/assets: (no message) (duration: 00m 33s)
  • 15:10 thcipriani: SWAT: running sync-portals for Updating footer on www.wikipedia.org portal
  • 15:09 hashar: Stopping Nodepool on labnodepool1001 , it can't spawn instances
  • 15:02 hashar: Zuul/Nodepool is out of instances. Looking
  • 14:15 moritzm: rolling restart of hhvm in codfw to pick up expat security update
  • 13:57 chasemp: downtime for dataset1001 puppet runs as T134896 causes failure (temporary for resize)
  • 13:55 jynus: disabling puppet on all database masters to test replication monitoring change T133337
  • 13:29 chasemp: resize volume for nfs dumps per T134896
  • 12:45 gehel: elasticsearch eqiad - reducing high watermark to rebalance disk space
  • 11:41 gehel: starting rolling restart of Elasticsearch codfw for Java update (T135499)
  • 11:41 gehel: rolling restart of Elasticsearch equiad for Java update completed (T135499)
  • 11:17 moritzm: install expat security updates
  • 10:41 kart_: Updated cxserver to 700dac2
  • 10:39 mobrovac: zotero restarted it on sca100, was on 50% mem
  • 10:38 mobrovac: zotero restarted it on sca1002, was on 50% mem and 100% cpu
  • 10:37 kart_: Updating cxserver to 700dac2
  • 09:35 moritzm: installing jansson security updates
  • 09:01 moritzm: installing libarchive security updates
  • 08:43 moritzm: installing xerces-c security updates
  • 08:36 jynus: performing schema change on s2 T130692
  • 08:10 gehel: starting rolling restart of Elasticsearch equiad fro Java update (T135499)
  • 07:04 mobrovac: restbase deploy end of 75a94ee
  • 06:54 mobrovac: restbase deploy start of 75a94ee
  • 06:38 moritzm: restarted hhvm on mw1207
  • 03:09 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed May 18 03:09:11 UTC 2016 (duration 9m 40s)
  • 02:59 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.2) (duration: 11m 15s)
  • 02:31 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.1) (duration: 11m 05s)
  • 01:51 logmsgbot: dereckson@tin Finished scap: Echo: Bring back Echo email messages (Gerrit:289346, for wmf1) (duration: 26m 30s)
  • 01:25 logmsgbot: dereckson@tin Started scap: Echo: Bring back Echo email messages (Gerrit:289346, for wmf1)
  • 01:03 logmsgbot: dereckson@tin Finished scap: Echo: Bring back Echo email messages (duration: 50m 15s)
  • 00:12 logmsgbot: dereckson@tin Started scap: Echo: Bring back Echo email messages
  • 00:09 Dereckson: mwscript namespaceDupes.php jamwiki --fix: 4 pages to fix, 4 were resolvable. (T135479)
  • 00:07 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Add Puotal: namespace to jam.wikipedia (T135479) (duration: 00m 27s)
  • 00:01 logmsgbot: dereckson@tin Synchronized wmf-config/: Undeploy Gather extension (T128568) (duration: 00m 33s)

2016-05-17

  • 23:52 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Add tasnimnews.com & khamenei.ir to wgCopyUploadsDomains (duration: 00m 36s)
  • 23:24 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.2/extensions/Echo/includes/formatters/EchoHtmlEmailFormatter.php: HTML email footer shows raw HTML (Gerrit:289270) (duration: 00m 31s)
  • 23:14 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Disable $wgCentralAuthCheckSULMigration functionality (T127887) (duration: 00m 25s)
  • 23:10 logmsgbot: dereckson@tin Synchronized portals: Turning off survey banner on www.wikipedia.org (T135235) (duration: 00m 26s)
  • 23:09 logmsgbot: dereckson@tin Synchronized portals/prod/wikipedia.org/assets: Turning off survey banner on www.wikipedia.org (T135235) (duration: 00m 25s)
  • 22:40 logmsgbot: ori@tin Synchronized php-1.28.0-wmf.1/includes/api/ApiStashEdit.php: Id1e0808c: Improve edit stash hit rate for logged-out users (duration: 00m 32s)
  • 21:45 gehel: stopping rolling restart of elasticsearch cluster for the night (T135499)
  • 21:21 logmsgbot: ori@tin Synchronized php-1.28.0-wmf.1/extensions/NavigationTiming: I62e20087c1: Expand coverage of conformance test (duration: 00m 28s)
  • 21:15 twentyafterfour: finished deploying wmf/1.28.0-wmf.2
  • 21:15 logmsgbot: ori@tin Synchronized php-1.28.0-wmf.2/includes/api/ApiStashEdit.php: Id1e0808c: Improve edit stash hit rate for logged-out users (duration: 00m 35s)
  • 20:53 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.28.0-wmf.2
  • 20:52 logmsgbot: twentyafterfour@tin Purged l10n cache for 1.27.0-wmf.22
  • 20:51 logmsgbot: twentyafterfour@tin Purged l10n cache for 1.27.0-wmf.21
  • 20:51 logmsgbot: twentyafterfour@tin Purged l10n cache for 1.27.0-wmf.20
  • 20:41 logmsgbot: twentyafterfour@tin Finished scap: (no message) (duration: 27m 42s)
  • 20:27 bblack: rebooting kafka1013 from racadm
  • 20:14 twentyafterfour: syncing testwiki to wmf/1.28.0-wmf.2
  • 20:13 logmsgbot: twentyafterfour@tin Started scap: (no message)
  • 20:09 bblack: starting upgrade of varnish4 packages on cache_maps
  • 20:09 bblack: finished upgrade of varnish4 packages on cache_misc
  • 20:03 ejegg: enabled banner impression loader
  • 20:02 ejegg: updated DjangoBannerStats from 30396a6df5bb9eadd5d4485a08fc4f6cb4096bd9 to 8172c4176ea2b78df0dccbfa052064f2739c64bd
  • 20:00 ejegg: disabled banner impression loader
  • 19:58 bblack: starting upgrade of varnish4 packages on cache_misc
  • 19:01 twentyafterfour: preparing to deploy wmf/1.28.0-wmf.2
  • 18:37 matt_flaschen: Ran manual SQL write in production to work around T122262; see task for query.
  • 18:36 urandom: Restarting (failed) bootstrap of restbase2008-b.codfw.wmnet : T95253
  • 18:34 urandom: Restart restbase2008-a.codfw.wmnet; Hail Mary pass for failed 2008-b bootstraps : T95253
  • 17:22 ejegg: updated CiviCRM from 6e3581692bb81c2c507612e7cd66dcdf78ef3cc0 to 7952ba43a012cb6a2e8d16af19bb13ed520bd56f
  • 16:23 godog: disable mod_deflate and restart apache2 on graphite1001 T135515
  • 16:01 twentyafterfour: branching wmf/1.28.0-wmf.2
  • 15:12 logmsgbot: thcipriani@tin Synchronized portals: (no message) (duration: 00m 25s)
  • 15:12 logmsgbot: thcipriani@tin Synchronized portals/prod/wikipedia.org/assets: (no message) (duration: 00m 25s)
  • 15:11 thcipriani: running /srv/mediawiki-staging/portals/sync-portals for gerrit:289218
  • 15:06 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor for IP users on the Japanese Wikipedia gerrit:287646 (duration: 00m 35s)
  • 14:59 elukey: disabled puppet on aqs* nodes as prep step to bootstrap the new testing cluster (https://gerrit.wikimedia.org/r/#/c/288373/5)
  • 14:46 gehel: taking elastic1001 down for investigation (T135509)
  • 14:37 mobrovac: change-prop deploying 5b5a07a3
  • 14:16 gehel: restarting elastic1001 - high load (T135509)
  • 13:32 hashar: upgrading Jenkins T133737
  • 13:25 godog: bounce carbon/frontend-relay on graphite1001 to increase queue size T135385
  • 13:22 elukey: memcahced restarted on mc1009 with -o slab_reassign,slab_automove,lru_crawler,lru_maintainer as part of a perf experiment (T129963)
  • 12:21 gehel: starting rolling restart of Elasticsearch equiad fro Java update (T135499)
  • 12:16 mobrovac: mathoid deployed 10c7cb8
  • 11:04 elukey: updated facts on puppet compiler following https://wikitech.wikimedia.org/wiki/Nova_Resource:Puppet3-diffs
  • 10:59 jynus: disabling puppet and starting decom process of db1027
  • 10:30 jynus: running schema change on s1 T130692
  • 09:59 logmsgbot: filippo@palladium conftool action : set/pooled=yes; selector: mw2050.codfw.wmnet
  • 09:14 ema: upgrading ulsfo cache_misc to varnish 4 (T131501, T134989)
  • 09:02 ema: upgrading esams cache_misc to varnish 4 (T126206, T134989)
  • 08:54 jynus: increased pool stall limit to 500 on db1049
  • 08:53 logmsgbot: filippo@palladium conftool action : set/pooled=no; selector: mw2050.codfw.wmnet
  • 08:31 ema: upgrading codfw cache_misc to varnish 4 (T126206, T134989)
  • 08:15 jynus: reducing durability and enabling GTID on db1026 T135100
  • 08:14 jynus: reducing durability and enabling GTID on db1026
  • 08:13 ema: upgrading eqiad cache_misc to varnish 4 (T126206, T134989)
  • 08:11 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Set db1026 back as rc node; move roles around (duration: 00m 31s)
  • 07:08 _joe_: restarted hhvm on mw1255, stuck in a deadlock on HPHP::Treadmill::getAgeOldestRequest
  • 07:02 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Reduce db1026 load (duration: 00m 37s)
  • 03:44 ori: Upgraded Grafana from 3.0.0-pre1 to 3.0.2
  • 02:37 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue May 17 02:37:35 UTC 2016 (duration 8m 46s)
  • 02:28 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.1) (duration: 10m 33s)
  • 01:13 logmsgbot: krenair@tin Synchronized docroot/noc: https://gerrit.wikimedia.org/r/#/c/289118/ (duration: 00m 27s)
  • 00:54 YuviPanda: restarted rabbitmq on labcontrol1001
  • 00:52 YuviPanda: restarted nova-scheduler, let's see if this fixes nodepool
  • 00:52 YuviPanda: restarted nova-conductor a few mins ago, no help for nodepool
  • 00:08 logmsgbot: ebernhardson@tin Finished scap: Full scap to sync out WikimediaMessages update (duration: 25m 31s)
  • 00:04 mutante: created planet2001 ganeti VM on ganeti2001
  • 00:03 mutante: mw1230, restart hhvm

2016-05-16

  • 23:42 logmsgbot: ebernhardson@tin Started scap: Full scap to sync out WikimediaMessages update
  • 23:38 logmsgbot: ebernhardson@tin scap sync-l10n completed (1.28.0-wmf.1) (duration: 00m 34s)
  • 23:37 logmsgbot: ebernhardson@tin Synchronized php-1.28.0-wmf.1/extensions/WikimediaMessages/WikimediaMessages.php: re-add interwiki search result messages (duration: 00m 25s)
  • 23:26 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.1/extensions/Echo/includes/EmailFormatter.php: Fix unsubstituted message in emails (duration: 00m 25s)
  • 23:18 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.1/extensions/WikimediaEvents: SWAT (duration: 00m 26s)
  • 23:17 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.1/extensions/UploadWizard: SWAT (duration: 00m 28s)
  • 23:13 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: SWAT (duration: 00m 25s)
  • 23:13 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: SWAT (duration: 00m 25s)
  • 23:09 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: SWAT (duration: 00m 36s)
  • 23:09 logmsgbot: catrope@tin Synchronized wmf-config/CirrusSearch-common.php: SWAT (duration: 00m 25s)
  • 17:06 ejegg: updated payments from 1af7208b4c7387a701f9ce83e33483d1e19213cb to 0b2d6807f68e7ccfc36a431e90d563b4d5a9e966
  • 17:00 godog: bounce cassandra-b on restbase2008-b to restart bootstrap
  • 16:59 ottomata: restarting kafka broker on kafka1022
  • 16:54 ottomata: restarted kafka broker on kafka1020 (about 5 mins ago)
  • 16:36 ottomata: restarting kafka broker on kafka1018
  • 16:23 ottomata: restarting kafka broker on kafka1014
  • 16:15 ottomata: restarting broker on kafka1013
  • 15:42 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add an alias "ΒΠ" for Project namespace in elwiki gerrit:288965 (duration: 00m 26s)
  • 15:36 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Enable MathML rendering by default on test, wikidata and dewikibooks PART II gerrit:286180 (duration: 00m 24s)
  • 15:35 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable MathML rendering by default on test, wikidata and dewikibooks PART I gerrit:286180 (duration: 00m 26s)
  • 15:31 bblack: starting slow cache_upload frontend restarts (wipes) for cache size upgrades (~10H process)
  • 15:29 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add "deletedtext" right to "eliminator" user group on fawiki gerrit:288956 (duration: 00m 26s)
  • 15:25 logmsgbot: thcipriani@tin Synchronized wmf-config/abusefilter.php: SWAT: Enable blocking feature of AbuseFilter on ptwiktionary gerrit:288810 (duration: 00m 24s)
  • 15:14 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Set meta namespace names for jamwiki gerrit:288628 (duration: 00m 27s)
  • 15:09 bblack: varnish package upgrade on cache_maps done
  • 15:08 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: wgCopyUploadProxy: Vary per datacenter gerrit:287095 (duration: 00m 26s)
  • 15:01 logmsgbot: hoo@tin Synchronized wmf-config/Wikibase.php: Update Wikidata property blacklist (duration: 00m 25s)
  • 14:55 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Remove all mentions to db1027, db2008 and db2009 (duration: 00m 33s)
  • 14:54 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Remove all mentions to db1027, db2008 and db2009 (duration: 00m 34s)
  • 14:45 mobrovac: change-propagation deployed ef5f6ff55
  • 14:45 bblack: starting varnish package upgrade on cache_maps
  • 14:39 bblack: general package updates on cache clusters (4.4.0 image, libtasn6-1, libjansson4, libidn11)
  • 11:20 jynus: testing create table on s7-master T130700
  • 10:08 jynus: applying puppet and restarting sanitarium instances to apply new filter (temporary labs lag)
  • 09:37 volans: Slowly re-enabling Puppet on db*, es*, pc*, labsdb*, labservices*, silver, holmium after merged change 288361 T133780
  • 09:31 volans: restarted mysql on db2034 to test merged change 288361 , T133780
  • 09:00 volans: Temporary disabling Puppet on db*, es*, pc*, labsdb*, labservices*, silver, holmium to merge change 288361 T133780
  • 08:48 volans: !log Temporary disabling Puppet on db*, es*, pc*, labsdb*, labservices*, silver, holmium to merge change 288361 T133780
  • 08:45 elukey: !log memcached on mc101[0123] got restarted because puppet did run gerrit/288880 and gerrit/288886 at the same time (operators fault of course)
  • 08:29 elukey: !log puppet disabled on mc* hosts as prep step for https://gerrit.wikimedia.org/r/#/c/288886 (precaution, not really needed)
  • 08:08 elukey: !log restarted memcached on mc1007 to ensure that https://gerrit.wikimedia.org/r/288880 was applied and working correctly. Will not do the same thing with the other mc hosts.
  • 08:01 elukey: !log puppet disabled on mc* hosts as prep step for https://gerrit.wikimedia.org/r/#/c/288880 (precaution, not really needed)
  • 07:07 _joe_: !log restarted hhvm on mw1161, stuck in HPHP::Treadmill::getAgeOldestRequest
  • 06:57 logmsgbot: !log jynus@tin Synchronized wmf-config/db-eqiad.php: Take db1027 out of production (duration: 00m 26s)
  • 02:30 logmsgbot: !log l10nupdate@tin ResourceLoader cache refresh completed at Mon May 16 02:30:49 UTC 2016 (duration 8m 40s)
  • 02:22 logmsgbot: !log mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.1) (duration: 09m 44s)

2016-05-15

  • 02:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun May 15 02:29:07 UTC 2016 (duration 8m 44s)
  • 02:20 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.1) (duration: 08m 47s)
  • 02:15 YuviPanda: !log console stuck on labnet1002, powercycling

2016-05-14

  • 02:28 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat May 14 02:28:07 UTC 2016 (duration 8m 42s)
  • 02:19 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.1) (duration: 08m 06s)
  • 01:25 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.1/includes/api/ApiStashEdit.php: f33ed7ae239 (duration: 00m 36s)
  • 00:07 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.1/includes/api/ApiMain.php: Fix missing parameter for OutputPageCheckLastModified hook (duration: 00m 32s)

2016-05-13

  • 23:54 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.1/extensions/Echo/includes/api/ApiEchoNotifications.php: Do not reuse CentralAuth tokens (duration: 00m 25s)
  • 22:54 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.1/extensions/Echo/includes/api/ApiEchoNotifications.php: Logging live hack (duration: 00m 31s)
  • 22:50 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.1/extensions/Echo/includes/api/ApiEchoNotifications.php: More logging (duration: 00m 25s)
  • 22:38 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.1/extensions/Flow/includes/Notifications/Formatter.php: Fix fatal for old Flow notifications (duration: 00m 26s)
  • 22:34 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.1/extensions/Echo/: Log failures to fetch cross-wiki notifications (duration: 00m 41s)
  • 21:55 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.1/extensions/Echo/Echo.php: Bump cache version now that cache pollution is hopefully fixed (duration: 00m 25s)
  • 21:52 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.1/extensions/Echo/: Fixes for cross-wiki notifications deployment fallout (duration: 00m 38s)
  • 21:17 bblack: cache_misc varnish3 downgrade complete (except 3007 + 1045 - those remain depooled/downtimed/puppet-disabled/etc, do not touch)
  • 21:06 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Enable $wgEchoCrossWikiNotifications on the right wikis (unused for now) (duration: 00m 28s)
  • 20:09 logmsgbot: mattflaschen@tin Synchronized wmf-config: Disable cross-wiki notifications entirely on non-SUL wikis and hide preference (duration: 00m 34s)
  • 20:05 bblack: starting varnish3 downgrade process for most of cache_misc
  • 17:43 godog: powercycle ms-be2008, sdb failed
  • 15:41 godog: bounce graphite-web on graphite1001, debugging carbon-c-relay stalls while sending metrics
  • 13:57 jynus: installing and testing sqlproxy on dbproxy1005
  • 13:44 hashar: Jenkins zmq is all fine
  • 13:43 hashar: Restarted Jenkins, the zmq plugin daemon did not start
  • 13:13 ema: depooling cp3007 to try reproducing T134989
  • 11:28 ema: repooling cp1061
  • 11:09 moritzm: uploaded firejail 0.38-1+wmf1 for trusty-wikimedia to carbon
  • 10:27 ema: depooling cp1061 trying to reproduce T134989
  • 09:48 godog: nodetool cleanup on restbase2006 T132976
  • 09:11 godog: start backfilling cassandra metrics from graphite1001 to graphite1003
  • 05:27 jynus: reimporting x1 on dbstore2002
  • 02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri May 13 02:30:49 UTC 2016 (duration 8m 50s)
  • 02:22 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.1) (duration: 08m 17s)
  • 01:48 awight|animal: rollback fundraising-tools from 53f6fe635dd8cc451b86788018e53f418b690b00 to e2978024e6f6b6881d087ac5d07e4c40f7374709
  • 00:53 mutante: killing multiple icinga-wm processes on neon, running puppet. manually started and by puppet as well?
  • 00:34 logmsgbot: demon@tin rebuilt wikiversions.php and synchronized wikiversions files: Remaining wikis to wmf.1
  • 00:17 cwd: updated payments from b16c3cb0e12aacb49b91974d693c401dcaa9cca9 to 1af7208b4c7387a701f9ce83e33483d1e19213cb
  • 00:13 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.1/extensions/Echo/includes/api/ApiEchoNotifications.php: attempt to fix notices/fatals in Echo (duration: 00m 29s)
  • 00:11 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.23/extensions/Echo/includes/api/ApiEchoNotifications.php: attempt to fix notices/fatals in Echo (duration: 00m 25s)

2016-05-12

  • 23:58 chasemp: restart slapd instances. we have some seemingly ldap issues in labs but it may not be the servers, could be bad merged changes.
  • 23:54 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.1/extensions/CirrusSearch/includes/CirrusSearch.php: Adjust textcat data collection for AB test (T121542) (duration: 00m 26s)
  • 23:43 logmsgbot: dereckson@tin Synchronized wmf-config/CirrusSearch-common.php: Revert "A/B/C test of control vs textcat vs accept-lang + textcat" (Gerrit:288545, 3/3) (duration: 00m 25s)
  • 23:43 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Revert "A/B/C test of control vs textcat vs accept-lang + textcat" (Gerrit:288545, 2/3) (duration: 00m 25s)
  • 23:42 logmsgbot: dereckson@tin Synchronized tests/cirrusTest.php: Revert "A/B/C test of control vs textcat vs accept-lang + textcat" (Gerrit:288545, 1/3) (duration: 00m 24s)
  • 23:33 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: A/B/C test of control vs textcat vs accept-lang + textcat (Gerrit:268048) (duration: 00m 25s)
  • 23:23 logmsgbot: dereckson@tin Synchronized wmf-config/CirrusSearch-common.php: A/B/C test of control vs textcat vs accept-lang + textcat (Gerrit:268048) (duration: 00m 25s)
  • 23:22 logmsgbot: dereckson@tin Synchronized tests/cirrusTest.php: A/B/C test of control vs textcat vs accept-lang + textcat (no-op) (duration: 00m 25s)
  • 23:16 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Allow import from outreach to meta (T134788) (duration: 00m 27s)
  • 23:10 logmsgbot: dereckson@tin Synchronized wmf-config/throttle.php: UK EU edit-a-thon throttle rule (Gerrit:288377, T134902) (duration: 00m 28s)
  • 23:06 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Enable cross-wiki notifications by default in production (Gerrit:287035, 2/2) (duration: 00m 29s)
  • 23:06 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings-labs.php: Enable cross-wiki notifications by default in production (Gerrit:287035, 1/2) (duration: 00m 42s)
  • 22:17 eileen: Updating civicrm from d32032965f2a45d24f1eaeb352eb6bfd6641753c to 6e3581692bb81c2c507612e7cd66dcdf78ef3cc0
  • 21:00 logmsgbot: aude@tin Synchronized wmf-config/Wikibase.php: Update repoUrl setting for Wikibase (duration: 00m 28s)
  • 21:00 logmsgbot: aude@tin Synchronized wmf-config/Wikibase-labs.php: Update repoUrl setting for Wikibase (duration: 00m 32s)
  • 20:58 logmsgbot: aude@tin Synchronized wmf-config/Wikibase-production.php: Update repoUrl setting for Wikibase (duration: 00m 27s)
  • 20:53 awight: update fundraising-tools from b114b7174c3bd9bf53cd44bf55397049a03b96fb to 53f6fe635dd8cc451b86788018e53f418b690b00
  • 19:57 logmsgbot: demon@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 to 1.28.0-wmf.1
  • 19:23 chasemp: adding static route to labtest instance vlan on cr(1|2)-codfw through labtestnet
  • 19:03 ottomata: restarting kafka broker on kafka1012 to pick up inter.broker.protocol.version change
  • 18:41 kaldari: ran mwscript maintenance/updateCollation.php --wiki=ruwiktionary --force
  • 17:34 jynus: reverting dbstore1001 to regular replication coords
  • 17:13 jynus: enabling GTID on selected db production hosts to change topology of dbstore1001
  • 17:03 godog: upgrade carbon-c-relay from jessie-backports on graphite1003 / graphite2002
  • 16:46 godog: investigating carbon-c-relay crash with buffer overflow on graphite1001
  • 16:45 akosiaris: restart alsafi for 2.5+dfsg-4~bpo8+1 qemu upgrade
  • 16:43 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.1/extensions/UploadWizard/resources/controller/uw.controller.Details.js: SWAT: Fix Uncaught TypeError: this.copyMetadataWidget.remove is not a function gerrit:288423 (duration: 00m 24s)
  • 16:38 logmsgbot: mattflaschen@tin Synchronized wmf-config/CommonSettings-labs.php: Beta Cluster change (duration: 00m 28s)
  • 16:30 logmsgbot: mattflaschen@tin Synchronized wmf-config/CommonSettings-labs.php: Beta Cluster change (duration: 00m 29s)
  • 16:30 jynus: running "megacli -PDOffline -PhysDrv '[32:8]' -aALL" on db1023
  • 16:29 logmsgbot: mattflaschen@tin Synchronized wmf-config/db-labs.php: Beta Cluster change (duration: 00m 27s)
  • 16:24 gehel: removing wdqs1001 from rotation for disk upgrade (T120714)
  • 16:11 gehel: merged and applied https://gerrit.wikimedia.org/r/#/c/288376/ (T134456)
  • 16:10 akosiaris: migrate all ganeti instances except alsafi off ganeti2006
  • 16:10 akosiaris: evacuate ganeti2006.codfw.wmnet from ganeti secondary instances
  • 16:06 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.1/extensions/Wikidata/extensions/ArticlePlaceholder/includes/SearchHookHandler.php: SWAT: Update ArticlePlaceholder - Fix notability checks and props params gerrit:288396 (duration: 00m 24s)
  • 16:03 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.23/maintenance/updateCollation.php: SWAT: updateCollation.php sql changes backport (duration: 00m 26s)
  • 15:52 ema: testing file storage on misc eqiad+esams T134989
  • 15:46 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.23/extensions/Wikidata/extensions/ArticlePlaceholder/includes/SearchHookHandler.php: SWAT: Update ArticlePlaceholder - Fix notability checks and props params gerrit:288396 (duration: 00m 25s)
  • 15:41 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.1/maintenance/updateCollation.php: SWAT: updateCollation.php sql updates gerrit:288386 gerrit:288384 gerrit:288385 (duration: 00m 26s)
  • 15:25 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.23/extensions/ContentTranslation/modules/tools/ext.cx.tools.mt.js: SWAT: MT: Use custom labels instead of provider id gerrit:288342 (duration: 00m 26s)
  • 15:20 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.1/extensions/ContentTranslation/modules/tools/ext.cx.tools.mt.js: SWAT: MT: Use custom labels instead of provider id gerrit:288171 (duration: 00m 25s)
  • 15:13 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Bump CirrusSearchRequestSet avro schema to rev 121456865906 PART II gerrit:287973 (duration: 00m 25s)
  • 15:12 logmsgbot: thcipriani@tin Synchronized wmf-config/event-schemas: SWAT: Bump CirrusSearchRequestSet avro schema to rev 121456865906 PART I gerrit:287973 (duration: 00m 26s)
  • 15:06 logmsgbot: thcipriani@tin Synchronized wmf-config/CirrusSearch-common.php: SWAT: Remove deprecated settings gerrit:281966 (duration: 00m 34s)
  • 14:55 jynus: executing random CHANGE MASTERS and crashes on db200[89] trying to break them
  • 14:50 godog: downtime flapping redis replication on rdb2006 alert for 10d
  • 14:45 jynus: enabling GTID on x1 shard (except db1029)
  • 13:34 ema: upgrading misc to varnish 4.1.2-1wm4 and wiping caches (T134989)
  • 13:30 elukey: restarted memcached on mc1007 with chunk growth factor set to 1.15 - Part of a perf experiment (T129963)
  • 13:14 jynus: deploying GTID replication on all of es3 shard
  • 13:08 volans: disabling puppet and restarting MySQL on db2040 to test change 288361 (in scheduled downtime on icinga) T133780
  • 12:40 godog: kill duplicate ircecho daemon on neon
  • 12:39 akosiaris: upgraded nodejs on etherpad1001 to nodejs 4.3
  • 12:32 akosiaris: etherpad-lite_1.6.0-1 uploaded on apt.wikimedia.org
  • 12:05 moritzm: added jenkins 1.651.2 for precise-wikimedia to carbon
  • 10:44 jynus: testing dual masters and r/w mode on parsercache nodes
  • 10:20 elukey: restarted oozie, hive-* daemons on analytics1003 for java upgrades
  • 10:09 godog: run puppet on graphite2001 to split cassandra metrics
  • 10:06 godog: reload carbon-c-relay on labmon1001, noop
  • 09:51 volans: Apply at runtime thread_pool_stall_limit=10 for all coredb masters (Gerrit 287394, T133333)
  • 09:47 volans: Apply at runtime thread_pool_max_threads=2000 for all coredb masters (Gerrit 287394, T133333)
  • 08:33 ema: wiping misc caches once again (T134989)
  • 07:51 ema: upgrading misc ulsfo to varnish 4.1.2-1wm3 and wiping caches (T134989)
  • 07:48 jynus: testing schema change T73563 on db1040
  • 07:47 ema: upgrading misc codfw to 4.1.2-1wm3 and wiping caches (T134989)
  • 07:05 ema: running varnish 4.1.2-1wm3 in misc esams (T134989)
  • 06:50 ema: repooling cp3007 (ran puppet after varnish upgrade)
  • 06:42 ema: depooling cp3007
  • 04:56 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.23/includes/objectcache/RedisBagOStuff.php: Fix unserialization of negative numbers (duration: 00m 32s)
  • 04:37 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.1/includes/objectcache/RedisBagOStuff.php: Fix unserialization of negative numbers (duration: 00m 31s)
  • 03:54 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.23/extensions/Flow/includes/Notifications/MentionPresentationModel.php: Fix Flow fatal (duration: 00m 26s)
  • 03:05 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu May 12 03:05:39 UTC 2016 (duration 9m 30s)
  • 02:56 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.1) (duration: 07m 23s)
  • 02:30 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.27.0-wmf.23) (duration: 09m 28s)
  • 00:05 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.23/extensions/ZeroBanner: https://gerrit.wikimedia.org/r/#/c/288236/ (duration: 00m 25s)
  • 00:03 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.1/extensions/ZeroBanner: https://gerrit.wikimedia.org/r/#/c/288236/ (duration: 00m 26s)

2016-05-11

  • 23:52 logmsgbot: catrope@tin Finished scap: Updating wmf23 Echo to wmf1 (duration: 26m 45s)
  • 23:26 logmsgbot: catrope@tin Started scap: Updating wmf23 Echo to wmf1
  • 23:13 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.1/extensions/Echo/: SWAT (duration: 00m 33s)
  • 23:08 logmsgbot: catrope@tin Synchronized portals: (no message) (duration: 00m 25s)
  • 23:08 logmsgbot: catrope@tin Synchronized portals/prod/wikipedia.org/assets: (no message) (duration: 00m 31s)
  • 23:06 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Enable Flow beta feature on specieswiki (duration: 00m 34s)
  • 21:19 robh: ulsfo maint complete, will push traffic back shortly.
  • 21:15 logmsgbot: robh@palladium conftool action : set/pooled=yes; selector: dc=ulsfo,cluster=cache_text
  • 21:12 logmsgbot: oblivian@palladium conftool action : set/pooled=yes; selector: dc=ulsfo,cluster=cache_upload
  • 20:47 mutante: re-enabled icinga notifications for ipsec
  • 20:23 bblack: wiping cache misc caches, again....
  • 19:39 mutante: turned off notifications for icinga services that match IPSEC
  • 19:25 mutante: stopped icinga-wm, flood due to ulsfo maintenance
  • 19:08 logmsgbot: hoo@tin Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on eowiki, orwiki and napwiki (duration: 00m 29s)
  • 18:58 robh: disregard ulsfo cp system icinga spam, onsite work for thermal paste per T134831
  • 18:55 logmsgbot: hoo@tin Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on htwiki - T134273 (duration: 00m 46s)
  • 18:47 logmsgbot: hoo@tin Finished scap: Update ArticlePlaceholder to master for initial Wikipedia deployment. (duration: 51m 18s)
  • 17:59 moritzm: uploaded new linux package for jessie-wikimedia to carbon (based on 4.4.9)
  • 17:56 logmsgbot: hoo@tin Started scap: Update ArticlePlaceholder to master for initial Wikipedia deployment.
  • 17:53 logmsgbot: hoo@tin Synchronized php-1.28.0-wmf.1/extensions/Wikidata: Update Wikibase (Property ordering related backports) (duration: 01m 52s)
  • 17:51 logmsgbot: hoo@tin Synchronized php-1.27.0-wmf.23/extensions/Wikidata: Update Wikibase (Property ordering related backports) (duration: 02m 11s)
  • 17:34 bblack: wiping cache_misc caches again...
  • 16:14 akosiaris: upgrade etherpad software to 1.6.0-1 and evaluate stability. packages NOT yet uploaded on apt.w.o
  • 16:07 bblack: wiping cache_misc caches again...
  • 15:22 thcipriani: first SWAT with scap subcommands (3.2.0-1) complete
  • 15:20 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.1/extensions/Translate/tag/PageTranslationHooks.php: SWAT: On documentation unit deletion, dont attempt to re-render translation page gerrit:288165 (duration: 00m 27s)
  • 15:11 logmsgbot: thcipriani@tin Synchronized portals: SWAT: T128546 updating portal stats gerrit:288192 (duration: 00m 33s)
  • 15:10 logmsgbot: thcipriani@tin Synchronized portals/prod/wikipedia.org/assets: SWAT: T128546 updating portal stats gerrit:288192 (duration: 00m 43s)
  • 14:36 bblack: wiping caches on cache_misc...
  • 13:33 ottomata: starting kafka 0.9 upgrade of analytics-eqiad cluster
  • 13:31 elukey: camus + puppet disabled on analytics1027 as prep step for kafka cluster migration to 0.9
  • 12:50 elukey: mw1147 powercycled due to unresponsiveness (not able to login as root)
  • 12:50 ema: upgrading cp1061 to varnish 4 (T131501)
  • 12:47 ema: upgrading cp1058 to varnish 4 (T131501)
  • 12:39 ema: upgrading cp1051 to varnish 4 (T131501)
  • 12:33 ema: upgrading cp1045 to varnish 4 (T131501)
  • 12:18 ema: upgrading cp2025 to varnish 4 (T131501)
  • 12:10 ema: upgrading cp2018 to varnish 4 (T131501)
  • 12:09 moritzm: installing libarchive security updates
  • 12:05 ema: upgrading cp2012 to varnish 4 (T131501)
  • 11:55 ema: upgrading cp2006 to varnish 4 (T131501)
  • 11:34 _joe_: restarting just HHVM on mw1142
  • 11:34 _joe_: restarting mw1142
  • 10:52 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: kafka1002.eqiad.wmnet
  • 10:50 logmsgbot: elukey@palladium conftool action : set/pooled=no; selector: kafka1002.eqiad.wmnet
  • 10:48 elukey: restarting eventbus on kafka100[12] for security upgrades
  • 10:47 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: kafka1001.eqiad.wmnet
  • 10:44 moritzm: gradually disabling unprivileged bpf on Linux 4.4 hosts via sysctl (once completed this will be puppetised, but the sysctl can't be reverted without a reboot so be careful for the initial activation)
  • 10:43 logmsgbot: elukey@palladium conftool action : set/pooled=no; selector: kafka1001.eqiad.wmnet
  • 10:20 ema: upgrading cp4004 to varnish 4 (T131501)
  • 10:12 ema: upgrading cp4003 to varnish 4 (T131501)
  • 10:03 ema: upgrading cp4002 to varnish 4 (T131501)
  • 09:41 ema: upgrading cp4001 to varnish 4 (T131501)
  • 09:37 moritzm: rolling restart of scb in eqiad to pick up openssl update
  • 09:14 ema: upgrading cp3010 to varnish 4 (T131501)
  • 09:06 ema: upgrading cp3009 to varnish 4 (T131501)
  • 08:55 ema: upgrading cp3008 to varnish 4 (T131501)
  • 08:34 mobrovac: restbase deploy end of beaaa71
  • 08:28 godog: bootstrap restbase2008-a T132976
  • 08:24 mobrovac: restbase deploy start of beaaa71
  • 08:18 elukey: memcached on mc1009 restarted with chunk size growth factor 1.15 (was: 1.05)
  • 08:04 moritzm: restarting apache on palladium for openssl update
  • 08:03 elukey: puppet disabled on mc10XX hosts for https://gerrit.wikimedia.org/r/#/c/287913
  • 02:55 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed May 11 02:55:55 UTC 2016 (duration 9m 47s)
  • 02:46 logmsgbot: mwdeploy@tin sync-l10n completed (1.28.0-wmf.1) (duration: 05m 54s)
  • 02:24 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.23) (duration: 09m 39s)

2016-05-10

  • 23:59 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.23/extensions/Kartographer/: Security patch (duration: 00m 25s)
  • 23:58 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.1/extensions/Kartographer/: Security patch (duration: 00m 26s)
  • 23:52 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.1/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.js: Fix 'Uncaught TypeError: this.emit is not a function' (T134794) (duration: 00m 25s)
  • 23:49 logmsgbot: dereckson@tin Synchronized php-1.27.0-wmf.23/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.js: Fix 'Uncaught TypeError: this.emit is not a function' (T134794) (duration: 00m 28s)
  • 23:34 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Centralise feedback for the visual editor at the Hindi Wikipedia (Gerrit:287873, T134789) (duration: 00m 25s)
  • 23:31 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Enable VisualEditor in single edit mode on ja.wiki (Gerrit:285985, 2/2) (duration: 00m 25s)
  • 23:30 logmsgbot: dereckson@tin Synchronized dblists/visualeditor-default.dblist: Enable VisualEditor in single edit mode on ja.wiki (Gerrit:285985, 1/2) (duration: 00m 25s)
  • 23:21 robh: disabling ulsfo via dns for onsite work tomorrow per T134831
  • 23:18 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Configure cross-wiki uploads from test2wiki to testwiki (Gerrit:285708, 2/2) (duration: 00m 27s)
  • 23:17 logmsgbot: dereckson@tin Synchronized wmf-config/filebackend-production.php: Configure cross-wiki uploads from test2wiki to testwiki (Gerrit:285708, 1/2) (duration: 00m 27s)
  • 23:05 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Undeploy UploadWizard from test2wiki (Gerrit:287944, 2/2) (duration: 00m 27s)
  • 23:04 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Undeploy UploadWizard from test2wiki (Gerrit:287944, 1/2) (duration: 00m 30s)
  • 22:28 yurik: graphoid was restarted on all scb servers with the new caching configuration. T134542
  • 22:08 mutante: ocg1003 - papaul fixed the install issue
  • 22:08 mutante: ocg1003 - revoked old puppet cert, signed new cert, re-adding after reinstall
  • 21:07 gehel: merging and applying configuration for new maps servers
  • 20:24 mutante: scheduled icinga downtime for ocg1003 and all services on it, rebooting to PXE (T84723)
  • 20:01 bblack: re-enabling puppet on caches
  • 19:53 logmsgbot: demon@tin Finished scap: group0 wikis to 1.28.0-wmf.1 (duration: 26m 49s)
  • 19:26 logmsgbot: demon@tin Started scap: group0 wikis to 1.28.0-wmf.1
  • 19:20 ottomata: upgraded kafka1022 to confluent kafka 0.9.0.1
  • 19:11 ottomata: reenabling camus and puppet on analytics1027
  • 18:38 mutante: deleted /tmp/make-wmf-branch on tin by request
  • 18:20 yurik: finished graphoid deployment & restart. T134575
  • 18:18 bblack: disabling puppet on caches for nginx change observe/deploy...
  • 18:10 jynus: testing GTID replication on es2019 T133385 T130702
  • 18:01 cscott: OCG: script reported "Cleared 0 (of 363141 total) entries from cache in 56.894 seconds" (T120079)
  • 18:01 ottomata: stopping kafka on kafka1022 to upgrade to 0.9
  • 18:00 cscott: OCG: clearing cache for ocg1003.eqiad.wmnet and ocg1003 (T120079)
  • 17:57 ottomata: stopping camus and puppet on analytics1027 during upgrade of one kafka broker
  • 17:56 yurik: about to deploy graphoid
  • 17:41 cscott: updated OCG to version b0c57a1c6890e9fa1f2c3743fc14cb6a7f244fc3 (T120079)
  • 17:38 cscott: starting OCG deploy
  • 16:40 ema: repooling cp3007 running varnish 4
  • 16:04 godog: nodetool cleanup on restbase2005 T132976
  • 16:03 robh: i fubar'd sinistra's grub, it'll be offline for a bit while longer.
  • 15:52 csteipp: deployed patch for T134863
  • 15:48 godog: collect mysqld metrics with prometheus-metrics-collector 0.8.1 on db2070 for 24h T128185
  • 15:35 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Configure $wgCheckUserCAMultiLock for CentralAuth wikis gerrit:286927 (duration: 00m 26s)
  • 15:27 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Simplify $wgApiFrameOptions configuration (2/2) PART II gerrit:287968 (duration: 00m 27s)
  • 15:26 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Simplify $wgApiFrameOptions configuration (2/2) PART I gerrit:287968 (duration: 00m 26s)
  • 15:23 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Simplify $wgApiFrameOptions configuration (1/2) gerrit:287939 (duration: 00m 26s)
  • 15:22 robh: sinistra powercycling and troubleshooting starting for disk issue
  • 15:15 moritzm: restarting apache on silver (wikitech host) for openssl update
  • 15:15 godog: graphite-web reload on graphite1001 after merging https://gerrit.wikimedia.org/r/281631
  • 15:10 logmsgbot: thcipriani@tin Synchronized portals: (no message) (duration: 00m 33s)
  • 15:10 logmsgbot: thcipriani@tin Synchronized portals/prod/wikipedia.org/assets: (no message) (duration: 00m 27s)
  • 15:09 thcipriani: running sync-portals for portals SWAT
  • 15:08 moritzm: restarted apache on californium for openssl update
  • 15:06 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Remove misleading comment in UploadWizard configuration gerrit:287948 (duration: 00m 47s)
  • 14:57 jynus: dropped pdns db and associated user accounts on m5-master
  • 14:54 bblack: restarting pybal on lvs3002
  • 14:53 moritzm: restarting ntp on nescio for openssl update
  • 14:52 bblack: restarting pybal on lvs3004
  • 14:45 moritzm: restarting ntp on maerlant for openssl update
  • 14:37 moritzm: restarting ntp on acamar for openssl update
  • 14:32 moritzm: restarting ntp on chromium for openssl update
  • 14:27 ema: upgrading cp3007 (misc) to varnish 4
  • 14:24 moritzm: restarting ntp on hydrogen for openssl update
  • 14:15 moritzm: restarting ntp on achernar for openssl update
  • 14:10 moritzm: restarting apache on neon for openssl update
  • 14:08 moritzm: restarting apache on uranium for openssl update
  • 14:01 ema: depooling and rebooting cp1066 to test mdadm boot workaround T131961
  • 13:58 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Add db1023 db weight; reduce weight on db1035 and db1044 (duration: 00m 26s)
  • 13:55 moritzm: restarting nginx on dataset1001, ms1001 and francium for openssl update
  • 13:30 ema: depooling and rebooting cp1065 to test mdadm boot workaround T131961
  • 13:18 logmsgbot: krinkle@tin Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 28s)
  • 13:17 logmsgbot: krinkle@tin Synchronized wmf-config/ProductionServices.php: (no message) (duration: 00m 28s)
  • 12:47 logmsgbot: hashar@tin Synchronized multiversion/MWMultiVersion.php: Typo fix in a couple comment blocks (duration: 00m 28s)
  • 12:46 logmsgbot: hashar@tin Synchronized typos: (no message) (duration: 01m 55s)
  • 12:33 godog: restart ircecho on kraz
  • 12:28 mobrovac: mobileapps deploying b8c396ae
  • 12:07 moritzm: rolling restart of maps cluster for openssl update
  • 12:03 moritzm: "powercycled" kraz (stuck by qemu bug)
  • 11:36 moritzm: apache restart on bohrium/piwik for openssl update
  • 11:26 jynus: stopping es2017 and es2019 for cloning 17 -> 19 + regular conf/upgrades
  • 11:02 logmsgbot: jmm@palladium conftool action : set/pooled=yes; selector: ms-fe2001.codfw.wmnet
  • 11:02 logmsgbot: jmm@palladium conftool action : set/pooled=no; selector: ms-fe2001.codfw.wmnet
  • 11:01 moritzm: rolling restart of swift frontend servers in codfw and eqiad for openssl update
  • 09:44 moritzm: rolling restart of swift backend servers in codfw and eqiad
  • 09:32 moritzm: powercycling ms-be2016, unresponsive and serial console is dead
  • 09:06 moritzm: restarting etherpad-lite on etherpad1001 for openssl update
  • 08:56 moritzm: restarting exim on fermium/lists.wikimedia.org for openssl update
  • 08:52 moritzm: restarting tor on radium for openssl update
  • 08:29 godog: bootstrap restbase2007-b T132976
  • 06:33 mobrovac: restbase deploy end of 1c890c4
  • 06:24 mobrovac: restbase deploy start of 1c890c4
  • 02:33 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue May 10 02:33:17 UTC 2016 (duration 8m 57s)
  • 02:24 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.23) (duration: 09m 23s)

2016-05-09

  • 23:15 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: nowiki flow beta (duration: 00m 26s)
  • 23:12 logmsgbot: demon@tin Synchronized wmf-config/: Obsolete graph settings (duration: 00m 29s)
  • 23:08 logmsgbot: demon@tin Synchronized wmf-config/mobile.php: Enable $wgMFStripResponsiveImages (duration: 00m 27s)
  • 23:04 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: lazy load images in MF for bnwiki (duration: 00m 26s)
  • 22:14 gehel: restarting elasticsearch on elastic1026 (high load)
  • 21:56 logmsgbot: demon@tin rebuilt wikiversions.php and synchronized wikiversions files: pedias back to wmf.23
  • 21:42 mutante: argon - scheduling eternal downtime, shut down | https://en.wiktionary.org/wiki/good_riddance#Etymology
  • 21:37 mutante: argon - revoke puppet cert, stop salt
  • 21:04 chasemp: clean out old snapshots on labstore1001
  • 20:58 mdholloway: mobileapps deployed f206e94
  • 20:36 mdholloway: starting mobileapps deployment
  • 20:29 gehel: restarting logstash server logstash100[26].eqiad.wmnet (T110236)
  • 20:26 gehel: restarting logstash server logstash1001.eqiad.wmnet (T110236)upgrade
  • 19:58 logmsgbot: demon@tin Synchronized php-1.27.0-wmf.22/extensions/NavigationTiming/: backport firstPaintTime fix (duration: 00m 33s)
  • 18:41 mutante: irc.wm.org - before restarting ircd on old, ~ 199 users on new, after: ~ 293 users on new
  • 18:38 mutante: argon - restarting ircd (this is the old server)
  • 17:37 elukey: camus+puppet re-enabled on analytics1027 after maintenance
  • 17:30 elukey: analytics1002 Yarn+HDFS masters failed over to analytics1001 for Java upgrades (restored original state)
  • 17:22 elukey: analytics1001 Yarn+HDFS masters failed over to analytics1002 for Java upgrades
  • 17:07 elukey: executed authdns-update on ns0.w.o to introduce new aqs records
  • 17:05 gehel: cluster restart completed for eqiad / codfw elasticsearch (T110236)$
  • 16:34 gehel: restarting elasticsearch server elastic1031.eqiad.wmnet (T110236), includes JDK upgrade
  • 16:12 gehel: restarting elasticsearch server elastic1030.eqiad.wmnet (T110236), includes JDK upgrade
  • 15:51 elukey: disabled camus+puppet on analytics1027 as prep step for maintenance on the cluster.
  • 15:41 gehel: restarting elasticsearch server elastic1029.eqiad.wmnet (T110236), includes JDK upgrade
  • 15:37 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add museudaimigracao.org.br to wgCopyUploadsDomains gerrit:287636 (duration: 00m 32s)
  • 15:30 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add National Digital Library of Brazil domains to wgCopyUploadsDomains gerrit:287635 (duration: 00m 27s)
  • 15:01 jynus: turning off es2001-es2010
  • 14:51 hashar: Zuul went deadlock. Restarted
  • 14:32 gehel: restarting elasticsearch server elastic1028.eqiad.wmnet (T110236), includes JDK upgrade
  • 14:30 gehel: restarting cassandra.metrics-collector on maps-test200[1234] (T134514) - correction
  • 14:29 gehel: restarting cassandra on maps-test200[1234] (T134514)
  • 14:23 elukey: restarting hadoop java daemons for Java upgrades on analytics104X and analytics105X hosts
  • 14:09 elukey: memcached restarted on mc1009 with only slab_reassign set (T129963)
  • 14:06 gehel: restarting elasticsearch server elastic1027.eqiad.wmnet (T110236), includes JDK upgrade
  • 13:37 gehel: restarting elasticsearch server elastic1026.eqiad.wmnet (T110236), includes JDK upgrade
  • 13:27 moritzm: restarting salt-master on neodymium to pick up openssl update
  • 13:14 gehel: restarting elasticsearch server elastic1025.eqiad.wmnet (T110236), includes JDK upgrade
  • 13:12 elukey: restarting hadoop java daemons for Java upgrades on analytics102X and analytics 103X hosts
  • 13:07 moritzm: restarting nginx on carbon/apt.wikimedia.org to pick up openssl update
  • 12:55 hashar: Enabled a setting in Jenkins for T132895
  • 12:53 gehel: restarting elasticsearch server elastic1024.eqiad.wmnet (T110236), includes JDK upgrade
  • 12:43 hashar: Updating Jenkins job operations-puppet-typos to use extended regular expressions when reading /typos ( T133047 )
  • 12:32 gehel: restarting elasticsearch server elastic1023.eqiad.wmnet (T110236), includes JDK upgrade
  • 12:03 Dereckson: mwscript initSiteStats.php --wiki iawiki --update (T134749)
  • 11:32 moritzm: rolling restart of ocg for openssl update
  • 11:27 hashar: restarting Jenkins
  • 11:17 gehel: restarting elasticsearch server elastic1022.eqiad.wmnet (T110236), includes JDK upgrade
  • 11:12 moritzm: restarting archiva on titanium for java update
  • 11:02 moritzm: restarting gitblit for java update
  • 10:54 gehel: restarting elasticsearch server elastic1021.eqiad.wmnet (T110236), includes JDK upgrade
  • 10:28 jynus: general decommission of db1058 (puppet, salt, etc.)
  • 10:21 elukey: restarted eventlogging on eventlogging1001 for security upgrades
  • 09:56 elukey: memcached restarted on mc1009, now running with slab_reassign,maxconns_fast,hash_algorithm=murmur3,slab_automove,lru_crawler,lru_maintainer (T129963, performance experiment)
  • 09:45 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Increase db1070 weight after repooling (duration: 00m 38s)
  • 09:27 gehel: restarting elasticsearch server elastic1020.eqiad.wmnet (T110236), includes JDK upgrade
  • 09:15 godog: bootstrap restbase2007-a T132976
  • 08:18 gehel: restarting elasticsearch server elastic1019.eqiad.wmnet (T110236)
  • 08:16 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1070 with low weight, weight increases, retire db1058 (duration: 00m 30s)
  • 08:13 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Retire db1058 server (duration: 00m 39s)
  • 07:38 gehel: restarting elasticsearch server elastic1018.eqiad.wmnet (T110236)
  • 06:48 jynus: powercycling pc2006 (crashed)
  • 06:11 gehel: restarting elasticsearch server elastic1017.eqiad.wmnet (T110236)
  • 02:52 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon May 9 02:51:58 UTC 2016 (duration 9m 18s)
  • 02:42 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.23) (duration: 08m 40s)
  • 02:24 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.22) (duration: 09m 53s)

2016-05-08

  • 07:44 gehel: restarting elasticsearch server elastic1016.eqiad.wmnet (T110236)
  • 06:07 gehel: restarting elasticsearch server elastic1015.eqiad.wmnet (T110236)
  • 02:52 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun May 8 02:52:46 UTC 2016 (duration 9m 47s)
  • 02:43 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.23) (duration: 08m 19s)
  • 02:25 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.22) (duration: 09m 52s)

2016-05-07

  • 20:28 SMalyshev: deploying updated Blazegraph version for WDQS to mitigate deadlock issue
  • 20:13 SMalyshev: restarted blazegraph on wdqs1001 and wdqs1002
  • 14:38 jynus: inplace precise upgrade to trusty on db1069 before labs explodes
  • 12:32 hoo: Restarted blazegraph on wdqs1002 (Unresponsive, even locally: java.io.IOException: Too many open files) T134238
  • 11:55 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.22/includes/api/ApiStashEdit.php: If56084466: Make stashEditFromPreview() call setCacheTime() (duration: 00m 33s)
  • 02:54 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat May 7 02:54:32 UTC 2016 (duration 9m 14s)
  • 02:45 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.23) (duration: 05m 38s)
  • 02:24 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.22) (duration: 09m 34s)
  • 01:01 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.22/includes/api/ApiStashEdit.php: If56084466: Bump PRESUME_FRESH_TTL_SEC to improve hit rate and avoid link queries (duration: 00m 34s)

2016-05-06

  • 22:37 logmsgbot: krenair@tin Synchronized wmf-config/LabsServices.php: labs-only change: https://gerrit.wikimedia.org/r/#/c/287294/ (duration: 00m 33s)
  • 22:37 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings-labs.php: labs-only change: https://gerrit.wikimedia.org/r/#/c/287294/ (duration: 00m 45s)
  • 19:56 chasemp: launching some test query traffic against labs DNS to test new settings
  • 14:56 logmsgbot: dcausse@tin Synchronized wmf-config/LabsServices.php: Set analytics kafka broker info for labs deployment-prep (duration: 00m 33s)
  • 14:46 moritzm: restarting exim on MX servers to pick up openssl update
  • 14:46 gehel: restarting elasticsearch server elastic1014.eqiad.wmnet (T110236)
  • 14:35 logmsgbot: demon@tin Synchronized php-1.27.0-wmf.22/extensions/CentralAuth: Backporting T134246 (duration: 00m 38s)
  • 13:54 gehel: restarting elasticsearch server elastic1013.eqiad.wmnet (T110236)
  • 13:26 jynus: reimaging db1070
  • 13:06 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1070 for maintenance (duration: 00m 29s)
  • 12:40 moritzm: restarted hhvm on mw1226 (hhvm-dump-debug output available)
  • 12:28 mobrovac: restbase rolling restart in eqiad for openssl update
  • 11:52 moritzm: rolling restart of restbase in codfw for openssl update
  • 11:47 elukey: restarting aqs on aqs100[123] for security upgrades.
  • 11:15 moritzm: restarting slapd on dubnium to pick up openssl update
  • 11:10 hoo: Reverted the property suggester data to data from the 20160411 dump (done testing T132839)
  • 11:02 hoo: Overwrote property suggester data with data from the 20160215 dump (T132839)
  • 10:59 moritzm: restarting slapd on pollux to pick up openssl update
  • 09:17 moritzm: restarted hhvm on mw1148
  • 08:36 gehel: restarting elasticsearch server elastic1012.eqiad.wmnet (T110236)
  • 08:13 godog: delete blacklisted cassandra metrics for restbase meta tables T134016
  • 08:05 gehel: restarting elasticsearch server elastic1011.eqiad.wmnet (T110236)
  • 07:12 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1065 after maintenance; repool db1023 with low weight (duration: 00m 34s)
  • 05:57 gehel: restarting elasticsearch server elastic1010.eqiad.wmnet (T110236)
  • 05:06 kart_: Update cxserver to 155c2d4
  • 04:35 logmsgbot: ori@tin rebuilt wikiversions.php and synchronized wikiversions files: Wikipedias back to wmf.22 due to page load performance regression

2016-05-05

  • 21:37 gehel: restarting elasticsearch server elastic1009.eqiad.wmnet (T110236)
  • 19:30 csteipp: deployed patch for T132874
  • 19:07 logmsgbot: demon@tin rebuilt wikiversions.php and synchronized wikiversions files: wikipedias to wmf.23
  • 18:47 csteipp: deployed patch for T133507
  • 18:44 robh: mr1-ulsfo power cable reroute complete
  • 18:43 robh: mr1-ulsfo going to flap as i reroute the power cable
  • 18:23 csteipp: deployed patch for T130947
  • 18:17 csteipp: deployed revert/new patches for core & extension for T129506
  • 17:40 _joe_: restarted ocg after packet capture
  • 17:40 ejegg: updated payments-wiki from 8962b558b5eb5a4dba91be7ed7649b532fa4ee35 to b16c3cb0e12aacb49b91974d693c401dcaa9cca9
  • 17:36 _joe_: temporarily stopping ocg on ocg1003 to better debug cleanup cache
  • 17:23 urandom: Rolling restart of cassandra-metrics-collector in RESTBase test cluster : T134016
  • 17:19 urandom: Rolling restart of cassandra-metrics-collector in RESTBase cluster : T134016
  • 17:09 _joe_: clearing ocg cache entries for ocg1003
  • 17:06 logmsgbot: oblivian@palladium conftool action : set/pooled=no; selector: name=ocg1003.*
  • 16:55 godog: restart cassandra-metrics-collector on restbase2001 to test white/black listing
  • 16:31 mdholloway: mobileapps finished no-op test deploy
  • 16:22 mdholloway: mobileapps starting no-op deploy on scb1001
  • 16:01 jynus: restarting db1023 for reimage to jessie
  • 15:11 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.23/extensions/CentralAuth/: SWAT: Use master CentralAuthUser instances when writing gerrit:287079 (duration: 00m 32s)
  • 15:05 logmsgbot: anomie@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Commons: Restrict changetags userright gerrit:286522 (duration: 00m 29s)
  • 15:03 logmsgbot: anomie@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Grant groups from flaggedrevs the patrol rights at test2wiki gerrit:287090 (duration: 00m 47s)
  • 15:02 jynus: stopping db1065 for hardware maintenance
  • 14:42 elukey: restarted memcached on mc1008 as part of a performance test (T129963)
  • 14:39 elukey: memcached on mc1009 is running now with new parameters only available for 1.4.25-2~wmf1 - part of a performance test (T129963)
  • 14:28 elukey: memcached 1.4.25-2~wmf1 manually installed on mc1009 as part of a performance test (T129963)
  • 14:02 andrewbogott: stopping nova-api on labnet1002
  • 13:55 andrewbogott: downtimed labservices1001, holmium, labcontrol1001 for one hour, disabled puppet as per https://phabricator.wikimedia.org/T128737
  • 13:14 akosiaris: set ganeti2006 as drained (excluded from allocation operations)
  • 11:32 gehel: restarting blazegraph on wdqs1001
  • 10:40 akosiaris: rebooted labsdb1004
  • 10:40 akosiaris: fix problems with url_downloader created by PEBKAC
  • 10:23 jynus: stopping mysql & backuping db1023 in preparation for reimage
  • 09:59 jynus: SET GLOBAL thread_pool_stall_limit = 10; on db1057
  • 09:56 jynus: SET GLOBAL thread_pool_max_threads = 2000; on db1057
  • 09:54 elukey: installed memcached 1.4.25-2~wmf1 manually on mc2009 as part of T129963
  • 09:49 mobrovac: restbase deploy end of 2a3972a
  • 09:42 jynus: powercycling mw2027
  • 09:30 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Really depool db1023 for maintenance (duration: 02m 27s)
  • 09:26 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1023 for maintenance (duration: 02m 44s)
  • 09:22 _joe_: reloaded the backend varnsih config on cp1058
  • 09:22 mobrovac: restbase deploy start of 2a3972a on canary restbase1008
  • 08:57 hoo: Restarted blazegraph on wdqs1002 (unresponsive)
  • 08:12 hoo: Restarted blazegraph on wdqs1002 (unresponsive)
  • 05:52 gehel: restarting elasticsearch server elastic1008.eqiad.wmnet (T110236)
  • 04:55 gehel: restarting elasticsearch server elastic1007.eqiad.wmnet (T110236)
  • 03:27 ori: restarted parsoid on all wtp* hosts to kill any requests from changepropagation service that may have been in flight
  • 03:20 ori: ran: sudo salt 'scb*' cmd.run 'puppet agent --disable ; service changeprop stop'
  • 03:15 ori: and restarted parsoid on wtp1*
  • 03:15 ori: injected an early return to v1Wt2html in routes.js when oldid = 106801025
  • 02:49 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.22) (duration: 18m 23s)
  • 00:27 twentyafterfour: phabricator upgrade complete
  • 00:21 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Plumbing for wmgEchoCrossWikiByDefault (unused for now) (duration: 00m 24s)
  • 00:16 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Plumbing for wmgEchoCrossWikiByDefault (unused for now) (duration: 00m 26s)
  • 00:06 logmsgbot: aude@tin rebuilt wikiversions.php and synchronized wikiversions files: Put wikidata on wmf/1.27.0-wmf.23
  • 00:04 twentyafterfour: taking phabricator offline for maintenance

2016-05-04

  • 23:54 logmsgbot: aude@tin Synchronized php-1.27.0-wmf.23/extensions/Wikidata: Fix bug in other languages box: T134432 (duration: 02m 18s)
  • 23:37 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Remove emailuser override for hewiki, no longer needed (duration: 00m 33s)
  • 23:21 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.23/extensions/VisualEditor: SWAT (duration: 00m 29s)
  • 23:18 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.23/extensions/Cite: SWAT (duration: 00m 52s)
  • 21:56 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 28s)
  • 21:43 cscott: restarting OCG after puppet deploy of https://gerrit.wikimedia.org/r/286068
  • 21:37 ejegg: updated payments-wiki from c502ab2f6b6ff914d67503a664d36076fdc32dcf to 8962b558b5eb5a4dba91be7ed7649b532fa4ee35
  • 21:35 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.22/extensions/Echo/Hooks.php: Debug logging for seemingly unattached users (duration: 00m 25s)
  • 21:25 gehel: deploying new icinga check on response time for WDQS
  • 21:23 logmsgbot: legoktm@tin Synchronized wmf-config/: touch the directory this time (duration: 00m 37s)
  • 21:13 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings.php: touch (duration: 00m 27s)
  • 21:11 logmsgbot: legoktm@tin Synchronized wmf-config/CommonSettings.php: Enable UploadsLink on Wikimedia Commons (2/2) - try 2 (duration: 00m 30s)
  • 21:08 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings.php: Enable UploadsLink on Wikimedia Commons (1/2) - try 2 (duration: 00m 28s)
  • 20:47 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.22/extensions/Echo/Hooks.php: Fix fatal (T134428) (duration: 00m 32s)
  • 20:32 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.23/extensions/Echo/Hooks.php: Fix fatal (T134428) (duration: 00m 33s)
  • 20:28 logmsgbot: hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: Wikidata back to 1.27.0-wmf.22 due to T134432. Poke T131557.
  • 20:08 subbu: finished deploying parsoid sha b0d015fa (T134017)
  • 20:05 subbu: synced new code; restarted parsoid on wtp1001 as canary
  • 20:05 hashar: Completed group1 wikis to 1.27.0-wmf.23
  • 20:02 subbu: starting deploy of parsoid sha b0d015fa
  • 19:51 logmsgbot: hashar@tin Synchronized wmf-config/: Bump cache epoch for Wikidata - https://gerrit.wikimedia.org/r/#/c/286940/ (duration: 00m 30s)
  • 19:33 logmsgbot: hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.23
  • 19:30 logmsgbot: hashar@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 25s)
  • 19:21 logmsgbot: hashar@tin Synchronized wmf-config: Reverting https://gerrit.wikimedia.org/r/#/c/286517/ due to wmgUseUploadsLink being undefined. T130018 (duration: 00m 29s)
  • 19:17 mutante: wikimedia.ru looks down - hello Moscow? https://meta.wikimedia.org/wiki/Wikimedia_Russia
  • 19:15 logmsgbot: hashar@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 41s)
  • 19:14 logmsgbot: hashar@tin Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 33s)
  • 18:10 gehel: restarting elasticsearch server elastic1006.eqiad.wmnet (T110236)
  • 17:49 gehel: restarting elasticsearch server elastic1005.eqiad.wmnet (T110236)
  • 17:31 twentyafterfour: livehacked phabricator/src/aphront/response/AphrontFileResponse.php to fix filename with newlines
  • 17:14 gehel: restarting elasticsearch server elastic1004.eqiad.wmnet (T110236)
  • 17:09 logmsgbot: legoktm@tin Synchronized wmf-config/: Enable UploadsLink at Wikimedia Commons - T130018 (duration: 00m 43s)
  • 17:05 logmsgbot: legoktm@tin Finished scap: Build l10n cache for UploadsLink deployment - T130018 (duration: 51m 53s)
  • 16:37 gehel: restarting elasticsearch server elastic1003.eqiad.wmnet (T110236)
  • 16:19 gehel: restarting blazegraph (T134238)
  • 16:13 logmsgbot: legoktm@tin Started scap: Build l10n cache for UploadsLink deployment - T130018
  • 16:03 gehel: restarting elasticsearch server elastic1002.eqiad.wmnet (T110236)
  • 16:02 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable NewUserMessage for SUL accounts too on gu.wikiquote gerrit:286877 (duration: 00m 28s)
  • 16:00 bblack: REALLY (from active LVS) removing old mobile IPs from actual production config (no longer in use) - T124482
  • 15:52 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Namespace configuration for gl.wikisource gerrit:286811 (duration: 00m 27s)
  • 15:50 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.23/extensions/ProofreadPage/ProofreadPage.namespaces.php: SWAT: Localize namespaces Page and Index in Galician gerrit:286820 (duration: 00m 26s)
  • 15:45 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Restore Wikipedia: namespace alias on outreach gerrit:286875 (duration: 00m 27s)
  • 15:43 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: SWAT: Shakespeare in London throttle rule gerrit:286807 (duration: 00m 26s)
  • 15:36 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.23/includes/htmlform/HTMLFormField.php: SWAT: Fix HTMLFormField calling Message::setContext with null gerrit:286855 (duration: 00m 25s)
  • 15:34 bblack: removing old mobile IPs from actual production config (no longer in use) - T124482
  • 15:33 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove redundant NS_PROJECT entries from wgNamespacesAliases gerrit:286812 (duration: 00m 34s)
  • 15:28 gehel: restarting elasticsearch server elastic1001.eqiad.wmnet (T110236)
  • 15:24 bblack: changed catchpoint 'Static Assets' checks from (deprecated) https://bits.wikimedia.org/static-current/resources/assets/poweredby_mediawiki_88x31.png to https://meta.wikimedia.org/w/resources/assets/poweredby_mediawiki_88x31.png - T107430
  • 15:11 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Really depool db1065 for maintenance (duration: 00m 26s)
  • 15:09 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Translate: Use Apertium via cxserver Part II gerrit:286632 (duration: 00m 28s)
  • 15:08 logmsgbot: thcipriani@tin Synchronized wmf-config/ProductionServices.php: SWAT: Translate: Use Apertium via cxserver Part I gerrit:286632 (duration: 00m 29s)
  • 14:59 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1065 for maintenance (duration: 00m 35s)
  • 14:30 mobrovac: change-prop: stopping the service until we scale the http proxy service
  • 14:22 jynus: installing sys schema on dbs with performance_schema enabled
  • 14:12 gehel: restarting elasticsearch server elastic2024.codfw.wmnet (T110236)
  • 13:49 logmsgbot: jmm@palladium conftool action : set/pooled=yes; selector: sca1002.eqiad.wmnet
  • 13:49 logmsgbot: jmm@palladium conftool action : set/pooled=no; selector: sca1002.eqiad.wmnet
  • 13:42 logmsgbot: jmm@palladium conftool action : set/pooled=yes; selector: sca1001.eqiad.wmnet
  • 13:41 logmsgbot: jmm@palladium conftool action : set/pooled=no; selector: sca1001.eqiad.wmnet
  • 13:40 moritzm: rolling restart of apertium in sca1* for openssl update
  • 13:24 gehel: restarting elasticsearch server elastic2023.codfw.wmnet (T110236)
  • 12:50 gehel: restarting elasticsearch server elastic2022.codfw.wmnet (T110236)
  • 12:34 gehel: restarting blazegraph (T134238)
  • 12:23 bblack: cache_text HTTP/2 switch process complete
  • 12:12 bblack: starting cache_text HTTP/2 switch process
  • 12:10 bblack: cache_upload HTTP/2 switch process complete
  • 12:08 gehel: restarting elasticsearch server elastic2021.codfw.wmnet (T110236)
  • 12:00 bblack: starting cache_upload HTTP/2 switch process
  • 11:51 gehel: restarting elasticsearch server elastic2020.codfw.wmnet (T110236)
  • 11:16 elukey: updating Pybal/LVS for codfw eventbus on lvs2003
  • 11:09 moritzm: removed obsolete mediawiki-math-texvc/imagemagick from nobelium
  • 11:03 gehel: restarting elasticsearch server elastic2019.codfw.wmnet (T110236)
  • 10:33 gehel: restarting elasticsearch server elastic2018.codfw.wmnet (T110236)
  • 10:29 moritzm: rolling restart of parsoid in eqiad to pick up openssl update
  • 10:25 elukey: updating pybal/LVS with codfw eventbus config on lvs2006
  • 10:23 jynus: restarting db1058 for reimaging to jessie T125028
  • 10:05 gehel: restarting elasticsearch server elastic2017.codfw.wmnet (T110236)
  • 09:59 godog: root@tin:/# lvresize -r -v --size +30G /dev/mapper/tin--vg-root
  • 09:44 kart_: Updated cxserver to 45596ac
  • 09:15 jynus: restarting enwiki-labs reimports (lag could happen temporarily)
  • 08:42 gehel: restarting elasticsearch server elastic2016.codfw.wmnet (T110236)
  • 08:19 jynus: stopping db1058 mysql for backup in preparation for reimage
  • 08:14 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1058 for reimage (duration: 00m 36s)
  • 06:20 gehel: restarting elasticsearch server elastic2015.codfw.wmnet (T110236)
  • 03:06 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed May 4 03:06:41 UTC 2016 (duration 9m 32s)
  • 02:57 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.23) (duration: 17m 12s)
  • 02:23 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.22) (duration: 09m 19s)
  • 00:23 Dereckson: mwscript namespaceDupes.php aswikisource --merge --fix (T133505)
  • 00:21 kaldari: ran mwscript maintenance/updateCollation.php --wiki=ruwiktionary --force
  • 00:02 logmsgbot: dereckson@tin Synchronized php-1.27.0-wmf.23/extensions/Graph/extension.json: Graph: match modern module loading in core (3/3) (duration: 00m 26s)
  • 00:02 logmsgbot: dereckson@tin Synchronized php-1.27.0-wmf.23/extensions/Graph/lib/topojson-global.js: Graph: match modern module loading in core (2/3) (duration: 00m 26s)
  • 00:01 logmsgbot: dereckson@tin Synchronized php-1.27.0-wmf.23/extensions/Graph/lib/d3-global.js: Graph: match modern module loading in core (1/3) (duration: 00m 26s)

2016-05-03

  • 23:56 logmsgbot: dereckson@tin Synchronized php-1.27.0-wmf.22/extensions/Graph/extension.json: Graph: match modern module loading in core (3/3) (duration: 00m 25s)
  • 23:55 logmsgbot: dereckson@tin Synchronized php-1.27.0-wmf.22/extensions/Graph/lib/topojson-global.js: Graph: match modern module loading in core (2/3) (duration: 00m 25s)
  • 23:54 logmsgbot: dereckson@tin Synchronized php-1.27.0-wmf.22/extensions/Graph/lib/d3-global.js: Graph: match modern module loading in core (1/3) (duration: 00m 26s)
  • 23:40 bblack: slow, depooled, staggered restart of varnish frontends on text and upload clusters commencing
  • 23:23 logmsgbot: dereckson@tin Synchronized wmf-config/CirrusSearch-production.php: Cirrus: only use pooled curl in hhvm / Gerrit:286485 (duration: 00m 34s)
  • 23:09 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings-labs.php: Revert Don't yet allow wikidatasparql graph urls (no op in prod) (duration: 00m 25s)
  • 23:08 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Revert Don't yet allow wikidatasparql graph urls (T126741) (duration: 00m 26s)
  • 22:47 bblack: upgrading varnish3 package on cache_text ...
  • 22:46 andrewbogott: restarting rabbitmq on labcontrol1001 to pick up a new ulimit
  • 21:55 bblack: stopped varnishkafka on all cache_upload, and wiped out the spammy junk it fills the disk with in /var/cache/varnishkafka/
  • 21:45 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.23/includes/search/SearchEngine.php: T134305 Fix invalid namespace handling in wmf.23 (duration: 00m 38s)
  • 21:37 bblack: cache_upload: rolling varnishd (backend only) restarts for package update
  • 21:31 bblack: upgrading varnish3 package on cache_upload
  • 21:11 gehel: restarting wdqs1002 (T134238)
  • 20:02 hashar: Restarting Jenkins
  • 19:11 hashar: group0 to 1.27.0-wmf.23 is complete.
  • 19:01 logmsgbot: hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.27.0-wmf.23 T131557
  • 19:01 gehel: restarting elasticsearch server elastic2013.codfw.wmnet (T110236)
  • 18:26 bblack: cache_misc: rolling varnishd restarts for package update
  • 18:23 bblack: upgrading varnish3 package on cache_misc
  • 18:17 bblack: HTTP/2 enable for cache_misc (nginx upgrade - T96848)
  • 17:59 gehel: restarting elasticsearch server elastic2012.codfw.wmnet (T110236)
  • 17:38 jynus: deployed new grants on new hosts for pdns-labs database
  • 17:18 gehel: restarting blazegraph on wdqs1002 (T134238)
  • 17:13 gehel: restarting elasticsearch server elastic2011.codfw.wmnet (T110236)
  • 17:11 bblack: HTTP/2 enable for cache_maps (nginx upgrade) - T96848
  • 17:03 bblack: nginx on all cp* restarted for openssl-1.0.2h
  • 16:58 godog: bootstrap restbase2009-a T132976
  • 16:43 gehel: restarting elasticsearch server elastic2010.codfw.wmnet (T110236)
  • 16:35 bblack: merging https://gerrit.wikimedia.org/r/282323 for puppetswat
  • 16:32 bblack: merging https://gerrit.wikimedia.org/r/#/c/282322 for puppetswat
  • 16:30 moritzm: uploaded openssl 1.0.2h for jessie-wikimedia to carbon
  • 16:28 bblack: merging https://gerrit.wikimedia.org/r/286598 for puppetswat
  • 16:26 bblack: merging https://gerrit.wikimedia.org/r/286642 for puppetswat
  • 16:25 bblack: merging https://gerrit.wikimedia.org/r/286484 for puppetswat
  • 16:22 bblack: authdns-update for https://gerrit.wikimedia.org/r/#/c/285085/ (yue lang) + workarounds from https://phabricator.wikimedia.org/T97051#1994679
  • 16:17 bblack: merging https://gerrit.wikimedia.org/r/285086 for puppetswat (apache redirects change)
  • 16:11 mobrovac: restabse restarting after https://gerrit.wikimedia.org/r/#/c/286278/
  • 16:04 bblack: merging https://gerrit.wikimedia.org/r/286278 (puppetswat)
  • 15:48 gehel: restarting elasticsearch server elastic2009.codfw.wmnet (T110236)
  • 15:47 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable data access for Wikiversity (duration: 00m 38s)
  • 15:44 logmsgbot: aude@tin Synchronized dblists/wikidataclient.dblist: Remove beta.wikiversity as a client. See T54971 (duration: 00m 25s)
  • 15:41 logmsgbot: aude@tin Synchronized dblists/: Remove arbitraryaccess.dblist (duration: 00m 25s)
  • 15:39 logmsgbot: aude@tin Synchronized wmf-config/CommonSettings.php: Remove arbitraryaccess wikitag (duration: 00m 26s)
  • 15:34 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Remove $wmgWikibaseEnableArbitraryAccess setting (duration: 00m 26s)
  • 15:28 paravoid: "systemctl restart ircecho" on kraz
  • 15:27 logmsgbot: aude@tin Synchronized wmf-config/Wikibase.php: Remove use of $wmgWikibaseEnableArbitraryAccess setting (duration: 00m 28s)
  • 15:15 volans: Change runtime semi-synchronous replication on eqiad x1,es2,es3 to match configured value T131753
  • 15:13 logmsgbot: thcipriani@tin Synchronized portals: (no message) (duration: 00m 30s)
  • 15:13 logmsgbot: thcipriani@tin Synchronized portals/prod/wikipedia.org/assets: (no message) (duration: 00m 37s)
  • 15:08 gehel: restarting elasticsearch server elastic2008.codfw.wmnet (T110236)
  • 15:05 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable NewUserMessage on gu.wikiquote gerrit:286638 (duration: 00m 33s)
  • 14:53 logmsgbot: hashar@tin Synchronized php-1.27.0-wmf.23/extensions/Wikidata: wikidata to .23 https://gerrit.wikimedia.org/r/#/c/286657/ (duration: 02m 17s)
  • 14:36 gehel: restarting elasticsearch server elastic2007.codfw.wmnet (T110236)
  • 14:12 elukey: changed kafkatee's config on oxygen to watch all the webrequest text|upload kafka partitions (analytics doubled them recently)
  • 14:00 jynus: creating sanitarium db filtering (redactatron) for jamwiki
  • 13:51 hashar: Warming up HHVM cache hitting testwiki /w/api.php T131557
  • 13:40 gehel: restarting elasticsearch server elastic2006.codfw.wmnet (T110236)
  • 13:39 hashar: Warming up HHVM cache hitting testwiki T131557
  • 13:38 hashar: Warming up HHVM cache hitting testwiki T1315567
  • 13:37 logmsgbot: hashar@tin Finished scap: testwiki to php-1.27.0-wmf.23 and rebuild l10n cache T131557 (duration: 26m 54s)
  • 13:18 jynus: applying schema change on s3-master db T130692
  • 13:10 logmsgbot: hashar@tin Started scap: testwiki to php-1.27.0-wmf.23 and rebuild l10n cache T131557
  • 13:02 gehel: restarting elasticsearch server elastic2005.codfw.wmnet (T110236)
  • 13:01 gehel: restarting wdqs-updater and keeping it under close scrutiny for the moment
  • 13:00 volans: Change runtime semi-synchronous replication on eqiad core DBs (s1-s7) to match configured value T131753
  • 12:31 gehel: restarting elasticsearch server elastic2004.codfw.wmnet (T110236)
  • 12:28 gehel: stopping wdqs-updater as it leaks pipes
  • 12:07 hashar: Cutting 1.27.0-wmf.23 branches T131557
  • 12:07 hashar: Cutting 1.27.0-wmf.23 branches
  • 11:35 bblack: cp1068: upgraded varnish to 3.0.6plus-wm9
  • 10:42 moritzm: installing poppler security updates on ocg (and other trusty hosts)
  • 10:24 jynus: applying schema change on eqiad-s3 db servers T130692
  • 10:03 jynus: applying schema change on codfw-s3 db servers T130692
  • 09:55 jynus: testing online index creation on db2018
  • 09:11 gehel: restarting elasticsearch server elastic2003.codfw.wmnet (T110236)
  • 09:10 moritzm: powercycled alsafi (stuck in KVM)
  • 09:08 kart_: Update cxserver to 8a4254e
  • 08:18 gehel: restarting elasticsearch server elastic2002.codfw.wmnet (T110236)
  • 08:05 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1040 after maintenance (duration: 00m 25s)
  • 07:07 jynus: recovering torrus database @ netmon following T87815
  • 02:34 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue May 3 02:34:45 UTC 2016 (duration 8m 36s)
  • 02:26 awight: CentralNotice fundraising campaigns reenabled after CiviCRM maintenance
  • 02:26 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.22) (duration: 08m 40s)
  • 02:21 mutante: ssh alsafi
  • 02:13 eileen: Updating civicrm from f5e8f98d07a2280118b7153bc342bf52ee67edd5 to d32032965f2a45d24f1eaeb352eb6bfd6641753c
  • 02:05 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Remove obsolete https->http rewrite for IRC notifications (duration: 00m 28s)
  • 02:04 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Remove obsolete ocwiki hack (duration: 00m 37s)
  • 01:54 RoanKattouw: Killed ssh processes on tin that had been hanging for days
  • 01:53 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.21) (duration: 23019m 56s)
  • 01:53 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.21) (duration: 12941m 41s)
  • 01:53 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.22) (duration: 4303m 19s)
  • 01:40 mutante: irc.wm.org - see T123729 if any questions
  • 01:39 mutante: switching irc.wikimedia.org from old server argon to new server kraz. old server still running untouched as argon.wikimedia.org. no clients are kicked. appservers are sending RC to both.
  • 00:50 logmsgbot: krenair@tin Synchronized wmf-config/interwiki.php: https://gerrit.wikimedia.org/r/286552 - interwiki cache update including jamwiki, horizon, and other things (duration: 00m 26s)
  • 00:41 logmsgbot: krenair@tin Synchronized langlist: create jamwiki - https://gerrit.wikimedia.org/r/286258 (duration: 00m 24s)
  • 00:40 logmsgbot: krenair@tin Synchronized static/images/project-logos: create jamwiki - https://gerrit.wikimedia.org/r/286258 (duration: 00m 24s)
  • 00:39 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: create jamwiki - https://gerrit.wikimedia.org/r/286258 (duration: 00m 24s)
  • 00:38 logmsgbot: krenair@tin rebuilt wikiversions.php and synchronized wikiversions files: create jamwiki - https://gerrit.wikimedia.org/r/286258
  • 00:36 logmsgbot: krenair@tin Synchronized dblists: create jamwiki - https://gerrit.wikimedia.org/r/286258 (duration: 00m 27s)
  • 00:12 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/286534/ (duration: 00m 30s)

2016-05-02

  • 23:58 logmsgbot: krenair@tin Finished scap: for https://gerrit.wikimedia.org/r/#/c/286535/ i18n changes (duration: 25m 57s)
  • 23:37 ebernhardson: restart elastic2007, codfw cluster master, to resolve lingering issues after resolving frozen index race condition
  • 23:32 logmsgbot: krenair@tin Started scap: for https://gerrit.wikimedia.org/r/#/c/286535/ i18n changes
  • 23:30 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.22/extensions/Echo: https://gerrit.wikimedia.org/r/#/c/286535/ and https://gerrit.wikimedia.org/r/#/c/286532/ (duration: 00m 30s)
  • 23:10 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.22/extensions/Flow/modules/flow/ui/widgets/editor/mw.flow.ui.EditorSwitcherWidget.js: https://gerrit.wikimedia.org/r/#/c/286527/ (duration: 00m 26s)
  • 22:56 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.22/extensions/CirrusSearch/: Cirrus: Stop auto-creating frozen index (duration: 00m 31s)
  • 22:56 awight: CentralNotice fundraising campaigns disabled for CiviCRM outage
  • 22:52 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-production.php: Config setting to stop auto-creating frozen index in cirrus (duration: 00m 33s)
  • 22:18 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/286537/ (duration: 00m 42s)
  • 22:13 cwd: updated crm from b386a6821c71310950ccdcdcf2616add727e1af4 to f5e8f98d07a2280118b7153bc342bf52ee67edd5
  • 22:10 papaul: restbase200[7-9]- signing puppet certs, salt-key, initial run
  • 21:26 cscott: updated OCG to version b775e612520f9cd4acaea42226bcf34df07439f7
  • 21:21 cscott: starting OCG deploy (a little late)
  • 20:23 gehel: restarting elasticsearch server elastic2001.codfw.wmnet (T110236)
  • 20:21 gehel: starting rolling restart of elasticsearch codfw cluster to disable multicast (T110236)
  • 20:15 subbu: finished deploying parsoid version 0a26f3a4
  • 20:09 subbu: synced code + restarted parsoid on wtp1001 as canary
  • 20:05 subbu: starting deploy of parsoid version 0a26f3a4
  • 19:27 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.22/includes/filebackend/FileBackendMultiWrite.php: 63b2d7b2eae (duration: 00m 32s)
  • 19:17 mutante: manually removing 2fa from my own wikitech account, adding it back ..
  • 18:24 gehel: deploying latest WDQS version
  • 17:23 robh: restbase2004 offline for next few hours for comparison work for new systems T132976
  • 16:01 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/286286/ (duration: 00m 26s)
  • 15:53 logmsgbot: krenair@tin Synchronized wikiversions-labs.json: https://gerrit.wikimedia.org/r/#/c/283689/ (duration: 00m 25s)
  • 15:53 logmsgbot: krenair@tin Synchronized dblists/all-labs.dblist: https://gerrit.wikimedia.org/r/#/c/283689/ (duration: 00m 26s)
  • 15:44 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/286287/ (duration: 00m 25s)
  • 15:40 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/286285/ (duration: 00m 25s)
  • 15:32 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.22/extensions/Wikidata: https://gerrit.wikimedia.org/r/#/c/286434/2 (duration: 02m 02s)
  • 15:29 bblack: re-pooling esams
  • 15:22 jynus: restarting db1040 for reimage
  • 15:21 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.22/extensions/Math/MathRestbaseInterface.php: https://gerrit.wikimedia.org/r/#/c/286412/ (duration: 00m 26s)
  • 15:07 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/285700/ (duration: 00m 42s)
  • 14:52 moritzm: rolling restart of zookeeper to pick up Java update
  • 14:22 bblack: starting gdnsd on esams (esams is marked down there)
  • 14:20 bblack: stopped gdnsd on eeden
  • 13:13 jynus: stopping db1040 mysql for backup before cloning
  • 12:15 elukey: deployed Varnish change to force HTTP 503 for datasets.wikimedia.org, stats.wikimedia.org, metrics.wikimedia.org as prep-step for OS reimage.
  • 12:13 elukey: deployed Varnish cache::misc change to force HTTP 503 for datasets.wikimedia.org, stats.wikimedia.org, metrics.wikimedia.org as prep-step for OS reimage.
  • 12:12 elukey: Merged Varnish cache::misc change to force HTTP 503 for datasets.wikimedia.org, stats.wikimedia.org, metrics.wikimedia.org as prep-step for OS reimage.
  • 11:21 elukey: deployed the last version of Event Logging from tin. Service also restarted.
  • 11:06 moritzm: rolling restart of hhvm in eqiad for pcre security update
  • 10:42 moritzm: rolling restart of hhvm in codfw for pcre security update
  • 09:58 moritzm: uploaded openldap 2.4.41+wmf1 for jessie-wikimedia to carbon (T130593)
  • 08:14 hashar: Restarted stuck Jenkins (due to IRC plugin)
  • 07:44 moritzm: rebooting hasseleh/hassium for kernel upgrade to 4.4
  • 07:10 moritzm: installing poppler security updates
  • 06:46 _joe_: rebooting serpens from ganeti, unreachable
  • 02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon May 2 02:30:33 UTC 2016 (duration 9m 18s)
  • 02:21 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.22) (duration: 09m 31s)

2016-05-01

  • 19:37 SMalyshev: enabled wdqs1002, put wdqs1001 in maintenance mode for reload
  • 16:20 volans: changing live configuration of db1042 thread_pool_stall_limit to 10 to avoid connection timeout errors
  • 16:18 volans: changing live configuration of db1042 thread_pool_stall_limit back to 100 to test impact on connection timeout
  • 16:08 volans: changing live configuration of db1042 thread_pool_stall_limit to 10 to test impact on connection timout
  • 15:24 jynus: alter table puppet.fact_values to a bigint unsigned for m1 T107753
  • 15:07 logmsgbot: volans@tin Synchronized wmf-config/db-eqiad.php: Depool db1040 for investigation T134114 (duration: 01m 22s)
  • 14:44 volans: truncated puppet.fact_values table to fix puppet (as documented on wikitech)
  • 10:58 godog: reboot furud.codfw.wmnet, ganeti instance with increasing load and 100% iowait, kvm/ganeti idle instance bug likely T134098

2016-04-30

  • 13:41 elukey: disabled puppet on analytics1047 and scheduled downtime for the host, IO errors in the dmesg for /dev/sdd. Stopped also Hadoop daemons to remove it from the cluster temporarily (not sure how to do it properly, will write docs).
  • 10:45 volans: Reset slave on sanitarium:3311 due to corrupted relay log after skipping query for duplicate key T132416
  • 10:19 volans: restarted slave on dbstore1001 skipping missing database T132837
  • 08:28 gehel: restarting elasticsearch server elastic1031.eqiad.wmnet (T110236)
  • 07:15 gehel: restarting elasticsearch server elastic1030.eqiad.wmnet (T110236)
  • 06:32 gehel: restarting elasticsearch server elastic1029.eqiad.wmnet (T110236)
  • 06:16 gehel: restarting elasticsearch server elastic1028.eqiad.wmnet (T110236)
  • 01:15 aude: applied Ibd302e1 to terbium for debugging broken wikidata rdf dumps

2016-04-29

  • 22:57 mutante: DNS - forced authdns-gen-zones etc from https://phabricator.wikimedia.org/T97051#1994679 on ns0/ns1/ns2 to get new language added
  • 20:59 gehel: restarting elasticsearch server elastic1027.eqiad.wmnet (T110236)
  • 19:56 urandom: (Re)starting cleanup on restbase1009-{a,b}.eqiad.wmnet
  • 19:56 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.22/extensions/CentralNotice/: T133971 (duration: 00m 41s)
  • 19:29 gehel: restarting elasticsearch server elastic1026.eqiad.wmnet (T110236)
  • 19:07 gehel: restarting elasticsearch server elastic1025.eqiad.wmnet (T110236)
  • 18:21 logmsgbot: jzerebecki@tin Synchronized php-1.27.0-wmf.22/extensions/Wikidata/extensions/Wikibase/repo/includes/Hooks/OutputPageBeforeHTMLHookHandler.php: wmf.22 fc20c54f7915b94ec0d15ef17e207c116910623d 2 of 2 T132645 (duration: 00m 28s)
  • 18:20 logmsgbot: jzerebecki@tin Synchronized php-1.27.0-wmf.22/extensions/Wikidata/extensions/Wikibase/repo/includes/Dumpers/DumpGenerator.php: wmf.22 fc20c54f7915b94ec0d15ef17e207c116910623d 1 of 2 T133924 (duration: 00m 29s)
  • 18:14 logmsgbot: jzerebecki@tin Synchronized php-1.27.0-wmf.22/extensions/Wikidata/extensions/Wikibase/repo/includes/Hooks/OutputPageBeforeHTMLHookHandler.php: wmf.22 fc20c54f7915b94ec0d15ef17e207c116910623d 2 of 2 T132645 (duration: 00m 34s)
  • 18:14 robh: started all slaves via dbstore2001 this time.
  • 18:12 logmsgbot: jzerebecki@tin Synchronized php-1.27.0-wmf.22/extensions/Wikidata/extensions/Wikibase/repo/includes/Dumpers/DumpGenerator.php: wmf.22 fc20c54f7915b94ec0d15ef17e207c116910623d 1 of 2 T133924 (duration: 00m 44s)
  • 18:07 robh: started all slaves via dbstore2002 per jaime's request
  • 17:45 gehel: restarting elasticsearch server elastic1024.eqiad.wmnet (T110236)
  • 16:56 gehel: restarting elasticsearch server elastic1023.eqiad.wmnet (T110236)
  • 16:22 gehel: restarting elasticsearch server elastic1022.eqiad.wmnet (T110236)
  • 15:29 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Repool db2047 and db2068. Depool db2008, db2009. Pool db2033 as the new x1 node. (duration: 00m 27s)
  • 15:17 gehel: restarting elasticsearch server elastic1021.eqiad.wmnet (T110236)
  • 14:56 logmsgbot: oblivian@palladium conftool action : set/pooled=yes; selector: name=mw1153.eqiad.wmnet
  • 14:54 jynus: moving topology of db2033 to be the new x1 master on codfw
  • 14:40 logmsgbot: oblivian@palladium conftool action : set/pooled=no; selector: name=mw1153.eqiad.wmnet
  • 14:32 gehel: restarting elasticsearch server elastic1020.eqiad.wmnet (T110236)
  • 14:26 hashar: Rebased tin:/srv/mediawiki-staging 31886c7..8e2670a . Bring in 3 changes that are solely for beta cluster.
  • 13:54 jynus: stopping mysql db2008 (cloning to db2033)
  • 13:39 jynus: reimaging db2033
  • 13:09 gehel: restarting elasticsearch server elastic1019.eqiad.wmnet (T110236)
  • 12:30 gehel: restarting elasticsearch server elastic1018.eqiad.wmnet (T110236)
  • 11:39 elukey: soft reboot for mw1119 (not responsive to ssh, root login timed out on the console)
  • 09:43 gehel: restarting elasticsearch server elastic1017.eqiad.wmnet (T110236)
  • 09:42 gehel: restarting elasticsearch server elastic1016.eqiad.wmnet (T110236)
  • 09:01 jynus: changing live configuration of db1049 thread_pool_stall_limit to 10 to test impact on connection timout
  • 08:20 gehel: restarting elasticsearch server elastic1016.eqiad.wmnet (T110236)
  • 07:57 elukey: puppet disabled on new kafka codfw instances due to errors while starting Event Bus (hosts not in service)
  • 07:54 moritzm: enabled base::firewall on stat1002
  • 07:52 gehel: restarting elasticsearch server elastic1015.eqiad.wmnet (T110236)
  • 07:36 godog: stop cleanups on restbase1014-b
  • 06:46 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Reduce normal traffic on s2 API servers (duration: 00m 27s)
  • 06:33 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1038, increase weight of new hardware slaves db107[4-8] (duration: 00m 33s)
  • 05:42 gehel: restarting elasticsearch server elastic1014.eqiad.wmnet (T110236)
  • 05:41 mutante: re: "02:29 Krenair: last deployment was slow because of snapshot1007 being offline" it's back, i don't know why, it was powered down and i just tried switching it on. that helped. the command is literally "power on" on HP
  • 05:39 mutante: snapshot1007 - was powered down, powering it on. (..connect to mgmt.. "damn it's a HP")
  • 05:35 mutante: snapshot1007 - not reachable, duration 10h
  • 04:58 gehel: restarting elasticsearch server elastic1013.eqiad.wmnet (T110236)
  • 02:29 Krenair: last deployment was slow because of snapshot1007 being offline, icinga shows it's been like that for the last 7 hours
  • 02:22 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.22/extensions/EventBus: https://gerrit.wikimedia.org/r/286115 (duration: 02m 27s)
  • 02:20 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.22) (duration: 09m 20s)
  • 00:27 mutante: RT - remove libapache2-mod-php5, restart Apache, Perl apps dont need PHP
  • 00:22 logmsgbot: aaron@tin Synchronized wmf-config/filebackend-production.php: Set "autoResync" on for local-multiwrite (duration: 02m 29s)
  • 00:15 cwd: updated civicrm from 777a91b8f9f6003a3eebdb8f2c73e45cc2bfb4a4 to b386a6821c71310950ccdcdcf2616add727e1af4
  • 00:04 Dereckson: Previous deployment: Gerrit:285553 Enable lazy loaded references in beta (T129693)
  • 00:03 Dereckson: Previous deployment: Gerrit:285927 GoogleNewsSitemap configuration (T39608)
  • 00:03 Dereckson: Previous deployment: Gerrit:252627 Revert "Increase abusefilter emergency disable threshold on MediaWiki.org"
  • 00:03 Dereckson: Previous deployment: Gerrit:280865+Gerrit:285989 Allow wmf-config/throttle.php to be lenient on ip/IP typo, clean rules (no-op)
  • 00:02 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 02m 25s)
  • 00:02 Dereckson: Previous deployment: Gerrit:279142 Document FIXME statement in config (no-op)

2016-04-28

  • 23:59 logmsgbot: maxsem@tin Synchronized wmf-config/throttle.php: (no message) (duration: 02m 24s)
  • 23:56 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 02m 25s)
  • 23:26 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.22/extensions/VisualEditor/: (no message) (duration: 02m 30s)
  • 23:17 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.22/extensions/WikidataPageBanner/: https://gerrit.wikimedia.org/r/286018 (duration: 02m 29s)
  • 23:12 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.22/extensions/VisualEditor/: https://gerrit.wikimedia.org/r/#q,285769,n,z (duration: 02m 34s)
  • 23:07 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.22/extensions/UploadWizard/: https://gerrit.wikimedia.org/r/#q,286016,n,z (duration: 02m 34s)
  • 22:01 chasemp: reboot of holmium
  • 21:41 twentyafterfour: added usleep(200000); to slow down the phabricator import even further.
  • 21:32 twentyafterfour: reduced phabricator taskmaster processes to 1
  • 21:08 gehel: restarting elasticsearch server elastic1012.eqiad.wmnet (T110236)
  • 19:47 gehel: restarting elasticsearch server elastic1011.eqiad.wmnet (T110236)
  • 19:15 jynus: manually rotating db1038's error log
  • 19:10 hashar: 1.27.0-wmf.22 deployed. Uneventful.
  • 19:00 logmsgbot: hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.27.0-wmf.22
  • 18:42 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.22/extensions/Echo/: Fix fatal T133921 (duration: 00m 32s)
  • 18:19 gehel: restarting elasticsearch server elastic1010.eqiad.wmnet (T110236)
  • 18:08 logmsgbot: mattflaschen@tin Synchronized wmf-config/db-labs.php: Beta Cluster change (duration: 00m 37s)
  • 17:40 yurik: deployed and restarted kartotherian & tilerator
  • 16:57 gehel: restarting elasticsearch server elastic1009.eqiad.wmnet (T110236)
  • 16:41 ejegg: updated payments-wiki from 16ed5af8c8544ea1c8d837ae16585eba4cbbfd4e to c502ab2f6b6ff914d67503a664d36076fdc32dcf
  • 16:26 twentyafterfour: further reduced the queue worker count on phabricator, to relieve stress on mysql m3 db1048
  • 16:17 bblack: starting SPDY stats sample on 8x caches for 24H - T96848
  • 16:15 gehel: restarting elasticsearch server elastic1008.eqiad.wmnet (T110236)
  • 15:35 elukey: installed memcached 1.4.25-2 (Debian sid/testing) in mc2009 as part of performance test (T129963)
  • 15:27 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Math: increase the number of concurrent connections to 150 gerrit:283269 (duration: 00m 35s)
  • 15:27 logmsgbot: gehel@palladium conftool action : get/pooled; selector: elastic1001.eqiad.wmnet
  • 15:23 elukey: puppet disabled on mc2009 as preparation step for https://gerrit.wikimedia.org/r/#/c/284907
  • 15:12 gehel: restarting elasticsearch server elastic1007.eqiad.wmnet (T110236)
  • 15:05 jynus: restarting db1038 for reimage to jessie
  • 14:32 gehel: wdqs-updater started on wdqs1002 (T133566)
  • 14:26 bblack: started SPDY stats sample on 8x caches - T96848#2248582
  • 14:25 elukey: deployed new zookeeper nodes in codfw (conf200[123])
  • 13:59 gehel: restarting elasticsearch server elastic1006.eqiad.wmnet (T110236)
  • 13:23 bblack: rebooting cp1008
  • 12:50 gehel: restarting elasticsearch server elastic1005.eqiad.wmnet (T110236)
  • 12:33 moritzm: upgrade/rolling restart of mediawiki canaries for pcre upgrade
  • 12:31 volans: Increase eqiad masters expire_logs_days (according to available space) T133333
  • 12:31 jynus: restarting sanitarium:s3 instance- query stuck again
  • 12:04 gehel: restarting elasticsearch server elastic1004.eqiad.wmnet (T110236)
  • 11:25 moritzm: uploaded varnish 3.0.6plus-wm9 to carbon for jessie-wikimedia
  • 11:19 volans: cleaning up some space on puppet-compiler host
  • 11:14 moritzm: upgraded varnish on cp1008 to 3.0.7 (except one patch)
  • 11:14 gehel: restarting elasticsearch server elastic1003.eqiad.wmnet (T110236)
  • 11:03 jynus: backing up db1038 data to dbstore1002
  • 10:50 jynus: stopping and restarting db1038 for backup and upgrade T125028
  • 10:41 jynus: running update table on eventlogging database on the master (db1046) T108856
  • 10:39 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: aqs1001.eqiad.wmnet
  • 10:32 hoo: Set new email for global user "Sebschlicht" per https://meta.wikimedia.org/w/index.php?oldid=15564713#Sebschlicht2.40global and private communication
  • 10:31 moritzm: installing PHP updates for jessie
  • 09:46 gehel: restarting elasticsearch server elastic1002.eqiad.wmnet (T110236)
  • 09:23 jynus: removing unused mysql-server-5.5 from holmium (keeping database just in case) T128737
  • 09:10 logmsgbot: elukey@palladium conftool action : set/pooled=no; selector: aqs1001.eqiad.wmnet
  • 09:03 moritzm: remove obsolete mysql 5.5 installations from mw1022, mw1023, mw1024, mw1025, mw1114 and mw1163
  • 09:00 gehel: restarting elasticsearch server elastic1001.eqiad.wmnet (T110236)
  • 08:59 gehel: starting rolling restart of elasticsearch cluster in eqiad (T110236)
  • 08:58 logmsgbot: oblivian@palladium conftool action : set/weight=10; selector: name=mw2018.codfw.wmnet
  • 08:57 logmsgbot: oblivian@palladium conftool action : set/weight=12; selector: name=mw2018.codfw.wmnet
  • 08:12 elukey: restarting kafka on kafka{1012,1014,1022,1020,2001,2002} for Java upgrades. Will probably trigger some EventLogging alarms due to a bug (T133779)
  • 07:51 twentyafterfour: applied a hotfix to phabricator repository import job so that autoclose will not apply to unmerged refs/changes
  • 07:50 twentyafterfour: reduced the number of phabricator worker processes to hopefully stop exhausting mysql connections.
  • 05:37 mutante: lvs1012 - puppet fail, tries to upgrade tcpdump package and cannot be authenticated
  • 05:34 mutante: mw1146 - hhvm restart
  • 05:27 mutante: krypton remove RT packages, remnants from testing
  • 03:04 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.22/extensions/Echo: Fix T133817 (originally scheduled for SWAT) (duration: 00m 34s)
  • 03:03 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.21/extensions/Echo: Fix T133817 (originally scheduled for SWAT) (duration: 00m 39s)
  • 02:41 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.22) (duration: 09m 24s)
  • 02:24 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.21) (duration: 10m 38s)
  • 02:12 twentyafterfour: manually edited crontab on iridium and killed multiple instances of public_task_dump.py (the cronjob was defined as * 2 * * * instead of 0 2 * * *)
  • 00:48 twentyafterfour: Phabricator's back online, everything seems to have gone smoothly.
  • 00:29 twentyafterfour: Preparing to take phabricator offline for maintenance.

2016-04-27

  • 22:18 logmsgbot: mattflaschen@tin Synchronized wmf-config/db-labs.php: Beta Cluster change (duration: 00m 29s)
  • 22:04 bblack: banned req.url ~ "^/w/load.php.*choiceData" on cache_text
  • 22:00 bblack: banned req.url ~ "^/load.php.*choiceData" on cache_text
  • 21:22 cwd: updated civicrm from 15a0086eef78f16110eba358a28ef78b51a385e1 to 777a91b8f9f6003a3eebdb8f2c73e45cc2bfb4a4
  • 21:03 bblack: rebooting cp1065
  • 21:01 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Restore codfw to elasticsearch config T133784 (duration: 00m 31s)
  • 21:00 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-production.php: Restore codfw to elasticsearch config T133784 (duration: 00m 37s)
  • 20:48 thcipriani: restarting jenkins after plugin downgrade
  • 20:41 hashar: 1.27.0-wmf.22 to group1 has been completed without incident. Deployment is open !
  • 20:41 ebernhardson: Enabled cirrussearch writes to codfw only on mw1165 w/ live hack
  • 20:32 gehel: switching wdqs1002 to maintenance and reimporting data (T133566)
  • 20:28 cscott: updated OCG to version e39e06570083877d5498da577758cf8d162c1af4
  • 20:20 yurik: deployed kartotherian & tilerator services
  • 20:09 gehel: adding back wdqs1001 to varnish configuration after reinstall (T133566)
  • 19:24 Pchelolo: update restbase to e9fbdfe
  • 19:18 Pchelolo: update restbase to e9fbdfe: canary on restbase1007
  • 19:11 Pchelolo: update restbase to e9fbdfe: staging
  • 19:09 logmsgbot: hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.22
  • 19:00 dcausse: restarting elastic on elastic2007.codfw.wmnet (master)
  • 18:56 mutante: creating VM ununpentium on ganeti/eqiad (T123713)
  • 18:55 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Drop codfw from elasticsearch config T133784 (duration: 00m 36s)
  • 18:55 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-production.php: Drop codfw from elasticsearch config T133784 (duration: 00m 25s)
  • 18:02 jynus: generating new triggers for eventlogging_sync schema T108856
  • 16:58 gehel: increase throttling limit and concurrency on recoveries for elasticsearch codfw cluster (T133784)
  • 16:05 gehel: increasing curl pool size for jobrunners (T133755)
  • 15:46 elukey: restarted kafka1013 for java upgrades
  • 15:30 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable NewUserMessage on hiwikiquote gerrit:285639 (duration: 00m 31s)
  • 15:09 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add Subject namespace to hiwikibooks gerrit:285008 (duration: 02m 41s)
  • 14:52 moritzm: uploaded pcre 8.31-2ubuntu2.3+wm1 to carbon for trusty-wikimedia (rebuild of latest trusty update with our patch to enable JIT)
  • 14:44 elukey: repooled kafka1001 after upgrades, will do the same procedure to kafka1002
  • 14:40 _joe_: upgraded conftool on palladium
  • 14:40 elukey: restarted kafka on kafka1001
  • 14:38 _joe_: upgrading conftool on all cp servers
  • 14:33 elukey: kafka1001.eqiad.wmnet depooled from eventbus for kafka upgrades (via confctl)
  • 14:14 chasemp: restart phd on iridium as it keeps complaining it lost procs (seems ok now)
  • 13:53 elukey: restarted kafka on kafka1018.eqiad.wmnet for Java upgrades
  • 13:24 gehel: hard restart of codfw elasticsearch cluster
  • 13:23 _joe_: uploading new conftool packages, T128199
  • 12:15 dcausse: restarting elastic on elastic2004.codfw.wmnet
  • 11:43 dcausse: restarting elastic on elastic2003.codfw.wmnet
  • 11:05 dcausse: restarting elastic on elastic2002.codfw.wmnet
  • 10:58 logmsgbot: reedy@tin LocalisationUpdate failed: git pull of core failed
  • 10:58 logmsgbot: reedy@tin LocalisationUpdate failed: git pull of core failed
  • 10:49 dcausse: restarting elastic on elastic2001.codfw.wmnet
  • 10:36 logmsgbot: gehel@tin Synchronized wmf-config/CirrusSearch-production.php: (no message) (duration: 02m 47s)
  • 10:30 gehel: switching elasticsearch morelike traffic from codfw to eqiad
  • 10:01 _joe_: shutting down mw10[7-8][0-9] and mw112[1-9]/mw1130 for T126242
  • 09:56 _joe_: clean puppet certs and facts on mw10[7-8][0-9] and mw112[1-9]/mw1130 for T126242
  • 09:23 _joe_: restarted hhvm on mw1144, usual deadlock
  • 09:11 _joe_: repooling cp3038
  • 09:09 _joe_: repooling cp3038
  • 08:59 _joe_: hard rebooting cp3038, console unreachable
  • 08:59 jynus: starting mysql on holmium
  • 08:57 _joe_: depooling cp3038 from all live pools
  • 08:40 _joe_: stopping puppet on mw10[7-8][0-9] and mw112[1-9]/mw1130 for T126242
  • 06:11 mutante: krypton - for some unrelated reason on every puppet run there is some noise about analytics::burrow stuff
  • 06:09 mutante: krypton - re-enabled puppet after LE + rt.wm.o puppetization issues fixed with gerrit 285586 @bblack #LEftw
  • 03:11 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Apr 27 03:11:11 UTC 2016 (duration 9m 46s)
  • 03:01 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.22) (duration: 18m 27s)
  • 02:27 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.21) (duration: 11m 46s)

2016-04-26

  • 23:29 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.22/extensions/Flow: SWAT (duration: 00m 45s)
  • 23:28 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.22/extensions/Echo: SWAT (duration: 00m 31s)
  • 23:26 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Revert lazy-loaded references (duration: 00m 25s)
  • 23:24 logmsgbot: catrope@tin Synchronized wmf-config/Wikibase.php: Actually enable data access in user language for Commons and testwikidatawiki (duration: 00m 26s)
  • 23:21 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.22/extensions/WikimediaEvents: SWAT (duration: 00m 25s)
  • 23:21 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.21/extensions/WikimediaEvents: SWAT (duration: 00m 28s)
  • 23:16 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Enable lazy-loaded references in mobile web beta on enwiki (duration: 00m 26s)
  • 23:11 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Enable arbitrary access in user language on Commons and testwikidatawiki (duration: 00m 33s)
  • 23:09 logmsgbot: catrope@tin Synchronized dblists/arbitraryaccess.dblist: Enable arbitrary access on Commons (duration: 00m 33s)
  • 22:54 ejegg: updated payments-wiki from 3ac3a0d7bb6e2d8443a87fb62bd5412a95f75aa2 to 16ed5af8c8544ea1c8d837ae16585eba4cbbfd4e
  • 21:39 gehel: restarting elasticsearch server elastic2021.codfw.wmnet - activating unicast (T110236)
  • 21:26 logmsgbot: ori@tin Synchronized wmf-config/filebackend-production.php: Ia4434256c3: Set descriptionCacheExpiry for Commons repo (duration: 00m 35s)
  • 21:06 gehel: activating geodns for new varnish maps servers (T131880)
  • 20:47 gehel: restarting elasticsearch server elastic2020.codfw.wmnet - activating unicast (T110236)
  • 20:38 bblack: restarting pybal on lvs3002 to enable new cache configuration for maps (T131880)
  • 20:35 bblack: restarting pbyal on lvs3004 to enable new cache configuration for maps (T131880)
  • 20:32 bblack: restarting pbyal on lvs2002 to enable new cache configuration for maps (T131880)
  • 20:30 bblack: restarting pbyal on lvs2005 to enable new cache configuration for maps (T131880)
  • 20:07 hashar: Killing Zuul entirely due to deadlock T128569
  • 20:02 bblack: restarting pbyal on lvs4002 to enable new cache configuration for maps (T131880)
  • 19:54 gehel: restarting pybal on lvs4004 to enable new cache configuration for maps (T131880)
  • 19:52 gehel: restarting elasticsearch server elastic2019.codfw.wmnet - activating unicast (T110236)
  • 19:14 gehel: restarting elasticsearch server elastic2018.codfw.wmnet - activating unicast (T110236)
  • 19:07 logmsgbot: hashar@tin Purged l10n cache for 1.27.0-wmf.20
  • 19:04 logmsgbot: hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.27.0-wmf.22
  • 18:41 gehel: restarting HHVM on terbium to enable upgrade to 3.12.1 (T132751)
  • 17:55 gehel: restarting elasticsearch server elastic2017.codfw.wmnet - activating unicast (T110236)
  • 17:38 gehel: restarting elasticsearch server elastic2016.codfw.wmnet - activating unicast (T110236)
  • 16:45 gehel: restarting elasticsearch server elastic2015.codfw.wmnet - activating unicast (T110236)
  • 16:43 godog: start restbase1014-[ab] cleanup
  • 16:15 gehel: restarting elasticsearch server elastic2014.codfw.wmnet - activating unicast (T110236)
  • 16:07 ejegg: updated payments from f09297028acace67588c2de845b754e2ace75c97 to 3ac3a0d7bb6e2d8443a87fb62bd5412a95f75aa2
  • 16:07 ejegg: updated payments from
  • 15:34 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Clean InitialiseSettings.php gerrit:285381 (duration: 00m 27s)
  • 15:27 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable action=credits on test and beta gerrit:285374 (duration: 00m 29s)
  • 15:25 papaul: OS installation on graphite2002
  • 15:24 gehel: restarting elasticsearch server elastic2013.codfw.wmnet - activating unicast (T110236)
  • 15:24 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Reformatted comment gerrit:285373 (duration: 00m 28s)
  • 15:22 hashar: 1.27.0-wmf.22 HHVM cache is warmed up | T131556
  • 15:20 logmsgbot: thcipriani@tin Synchronized wmf-config: SWAT: Switch to short syntax array gerrit:285361 (duration: 00m 36s)
  • 15:17 logmsgbot: hashar@tin Synchronized php-1.27.0-wmf.22/includes/DefaultSettings.php: Set = 1.27.0-wmf.22 (duration: 01m 04s)
  • 14:55 logmsgbot: hashar@tin Finished scap: testwiki to php-1.27.0-wmf.22 and rebuild l10n cache (duration: 29m 28s)
  • 14:38 gehel: restarting elasticsearch server elastic2012.codfw.wmnet - activating unicast (T110236)
  • 14:26 logmsgbot: hashar@tin Started scap: testwiki to php-1.27.0-wmf.22 and rebuild l10n cache
  • 13:33 hashar: Checking out MediaWiki 1.27.0-wmf.22 on tin | T131556
  • 13:33 hashar: Checking out MediaWiki 1.27.0-wmf.22 on tin
  • 13:11 gehel: restarting elasticsearch server elastic2011.codfw.wmnet - activating unicast (T110236)
  • 12:27 gehel: restarting elasticsearch server elastic2010.codfw.wmnet - activating unicast (T110236)
  • 12:09 elukey: mw1145 powercycled (root login timeout on the console)
  • 11:43 gehel: restarting elasticsearch server elastic2009.codfw.wmnet - activating unicast (T110236)
  • 10:31 jynus: restarting mw1143 as is no longer accessible
  • 09:51 jynus: restarting db2068 for upgrade before returning from maintenance
  • 09:50 gehel: starting reinstall of wdqs1001 (T133566)
  • 09:47 gehel: restarting elasticsearch server elastic2008.codfw.wmnet - activating unicast (T110236)
  • 09:41 moritzm: uploaded apache 2.4.10-10+deb8u4+wmf1 to carbon
  • 09:12 _joe_: restarted hhvm on mw1133, almost OOM
  • 09:11 hashar: CI is back up!
  • 09:09 mobrovac: citoid deployed 36c2bf02
  • 09:03 jynus: restarting db1069 s1 instance- one query is "stuck", creating lag on labs
  • 08:56 gehel: depooling wdqs1001 from varnish for reinstall
  • 08:52 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1052 (old s1-master) with low weight (duration: 00m 31s)
  • 08:50 YuviPanda: restarted nova-conductor & scheduler on labcontrol1001 for T133654
  • 08:45 hashar: Most of CI is down / deadlocked due to wmflabs being unresponsive T133654
  • 08:44 gehel: restarting elasticsearch server elastic2007.codfw.wmnet - activating unicast (T110236)
  • 08:27 jynus: stopping db2068 for cloning to db2047
  • 08:06 hashar: CI jobs deadlocked due to castor being unavailable | https://phabricator.wikimedia.org/T133652
  • 06:52 jynus: restarting db2047 for reimaging
  • 02:46 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Apr 26 02:46:52 UTC 2016 (duration 9m 1s)
  • 02:37 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.21) (duration: 19m 17s)
  • 01:09 paravoid: reinstalling bast4001 w/ jessie

2016-04-25

  • 23:51 logmsgbot: dereckson@tin Synchronized docroot/noc/conf/: noc.wikimedia.org update (Gerrit:285061, Gerrit:285062, Gerrit:281977) (duration: 00m 26s)
  • 23:48 logmsgbot: dereckson@tin Synchronized docroot/noc/createTxtFileSymlinks.sh: noc: PoolCounterSettings-eqiad.php → PoolCounterSettings.php (Gerrit:285062, no-op) (duration: 00m 30s)
  • 23:21 Krenair: Deployed patch for T132653
  • 23:10 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/284831/ - Disable Echo survey on French wikis
  • 22:30 eileen: Updating civicrm
  • 21:02 gehel: adding cp10(46|47|59|60)\.eqiad\.wmnet to maps caching cluster (T109162)
  • 21:02 cscott: updated OCG to version 58a720508deb368abfb7652e6a8c7225f95402d2
  • 20:46 csteipp: deployed updated patch for T98313 to include very headers
  • 20:35 gehel: restarting elasticsearch server elastic2006.codfw.wmnet - activating unicast (T110236)
  • 20:07 subbu: finished deploying parsoid version d5363193
  • 20:04 subbu: synced new code and restarted parsoid on wtp1001 as a canary
  • 20:01 subbu: deploying parsoid version d5363193
  • 19:49 gehel: restarting elasticsearch server elastic2005.codfw.wmnet - activating unicast (T110236)
  • 19:39 hashar: Nodepool now booting 2 trusty instances for CI needs. T133203
  • 19:27 gehel: disabling puppet on cp1043/cp1044 while reconfiguring Maps caching servers
  • 19:27 gehel: start configuration of new maps caching servers (T109162)
  • 19:08 gehel: restarting elasticsearch server elastic2004.codfw.wmnet - activating unicast (T110236)
  • 18:41 logmsgbot: legoktm@tin Synchronized wmf-config: ⓛⓐⓑⓢ-ⓞⓝⓛⓨ, ⓝⓞ-ⓞⓟ (duration: 00m 40s)
  • 18:38 bblack: switching mobile hostnames to text IPs in DNS (10 min TTL)
  • 18:21 Pchelolo: finished update RESTBase to c1d5193
  • 18:09 Pchelolo: update RESTBase to c1d5193
  • 17:55 Pchelolo: update RESTBase to c1d5193, canary on restbase1007
  • 17:53 jynus: wiping db1052 tendril's monitoring data due to 5.5 -> 5.6 incompatibility
  • 17:48 jynus: db1052 now in jessie
  • 17:47 Pchelolo: update RESTBase to c1d5193, canary on restbase1005
  • 17:26 Pchelolo: update RESTBase to c1d5193, staging
  • 17:22 mutante: Ⓐ
  • 17:20 ori: 😊
  • 17:01 gehel: restarting elasticsearch server elastic2003.codfw.wmnet - activating unicast (T110236)
  • 16:34 logmsgbot: jzerebecki@tin Synchronized wmf-config/CommonSettings.php: config 3bd39b8b9ac9fb252ff8a3e9a93b14c1defd45fc T132820 : Use notify-type-availability due to Echo change (duration: 00m 26s)
  • 16:24 jzerebecki: T132914: @tin:/srv/mediawiki-staging (master %=)$ mwscript extensions/Flow/maintenance/FlowUpdateBetaFeaturePreference.php frwikisource
  • 16:15 logmsgbot: jzerebecki@tin Synchronized wmf-config/InitialiseSettings.php: config 1f8fea8e53b2e95caf0f559e0ad6e2cb1d51c967 T132914 : Enable Flow opt-in beta feature on frwikisource (duration: 00m 28s)
  • 16:10 logmsgbot: jzerebecki@tin Synchronized wmf-config/InitialiseSettings.php: config 1b156b46d42f2482c1941427dc4ec8aeb427f0b3 T133286 : Add *.asc-test.nl to wgCopyUploadsDomains (duration: 00m 27s)
  • 15:59 YuviPanda: restart pdns recursor & pdns on holmium
  • 15:58 logmsgbot: jzerebecki@tin Synchronized wmf-config/InitialiseSettings.php: config 0d0d7acf79891ad36d48cd30887bb2299d43f264 T132792 3 of 3 ; and config 7144b0139c85b565af76a993b38209b4c0989dec T132748 (duration: 00m 25s)
  • 15:56 gehel: restarting elasticsearch server elastic2002.codfw.wmnet - activating unicast (T110236)
  • 15:56 logmsgbot: jzerebecki@tin Synchronized static/images/project-logos/dewiki-2x.png: config 0d0d7acf79891ad36d48cd30887bb2299d43f264 T132792 2 of 3 : Add HD versions of logo for dewiki (duration: 00m 25s)
  • 15:55 logmsgbot: jzerebecki@tin Synchronized static/images/project-logos/dewiki-1.5x.png: config 0d0d7acf79891ad36d48cd30887bb2299d43f264 T132792 1 of 3 : Add HD versions of logo for dewiki (duration: 00m 25s)
  • 15:53 godog: test reboot for mw2212 - T129196
  • 15:50 chasemp: restart pdns to see if that helps a labs issue w/ tools proxy
  • 15:50 logmsgbot: jzerebecki@tin Synchronized wmf-config/InitialiseSettings.php: config 2497480f39ea2dad27bab554b7d6e3a996560f46 T131685 : Disable MoodBar on testwiki (duration: 00m 27s)
  • 15:43 jynus: recovering backuped db1052 data from dbstore1002
  • 15:42 jzerebecki: T132746 another run needed @tin:/srv/mediawiki-staging (master %=)$ mwscript namespaceDupes.php foundationwiki --move-talk --merge --fix >try2-out-fix2.txt
  • 15:40 jzerebecki: T132746 @tin:/srv/mediawiki-staging (master %=)$ mwscript namespaceDupes.php foundationwiki --move-talk --merge --fix >try2-out-fix.txt
  • 15:35 jzerebecki: err T132746 instead of T132868
  • 15:35 jzerebecki: T132868 db error: Query: UPDATE `page` SET page_namespace = '100',page_title = '2008-09_Budget' WHERE page_id = '21741' Function: NamespaceConflictChecker::movePage Error: 1062 Duplicate entry '100-2008-09_Budget' for key 'name_title' (10.64.0.205)
  • 15:31 jzerebecki: T132868 @tin:/srv/mediawiki-staging (master %=)$ mwscript namespaceDupes.php foundationwiki --move-talk --fix >out-fix.txt
  • 15:31 jzerebecki: T132868 3508 links to fix, 3508 were resolvable. Looks good!
  • 15:28 jzerebecki: T132746 @tin:/srv/mediawiki-staging (master=)$ mwscript namespaceDupes.php foundationwiki --move-talk
  • 15:26 logmsgbot: jzerebecki@tin Synchronized wmf-config/InitialiseSettings.php: config 70ee0fd97c26da02ed8ea511b2e21b47f374101b T132746 : Add Resolution: namespace to foundationwiki (duration: 00m 31s)
  • 15:22 jzerebecki: T131605 @tin:/srv/mediawiki-staging (master=)$ echo 'https://en.wikipedia.org/static/images/project-logos/cswiki.png' | mwscript purgeList.php
  • 15:22 logmsgbot: jzerebecki@tin Synchronized wmf-config/InitialiseSettings.php: config 5c153e934a88f7b5e16a2d4b244ed76a9dfc1cdc T132868 : Enable RC patrol on ta.wikiquote (duration: 00m 28s)
  • 15:20 jzerebecki: T131605 @tin:/srv/mediawiki-staging (master=)$ echo 'https://cs.wikipedia.org/static/images/project-logos/cswiki.png' | mwscript purgeList.php
  • 15:14 logmsgbot: jzerebecki@tin Synchronized wmf-config/InitialiseSettings.php: config 6c5214dcb3f148dab12fe4aa6ae0921cff05038a T131605 2 of 2 : Revert "350K articles celebration logo on cs.wikipedia" (duration: 00m 30s)
  • 15:12 logmsgbot: jzerebecki@tin Synchronized static/images/project-logos/cswiki.png: config 6c5214dcb3f148dab12fe4aa6ae0921cff05038a T131605 1 of 2 : Revert "350K articles celebration logo on cs.wikipedia" (duration: 00m 26s)
  • 15:07 logmsgbot: jzerebecki@tin Synchronized php-1.27.0-wmf.21/includes/specials/SpecialRunJobs.php: wmf21 e411ad62516bfae31f8d482a22d378912fe5979f T89169 : SpecialRunJobs: delegate error handling to MWExceptionHandler (duration: 00m 37s)
  • 14:47 gehel: restarting elasticsearch server elastic2001.codfw.wmnet - activating unicast (T110236)
  • 14:46 gehel: restarting elasticsearch server elastic1001.codfw.wmnet - activating unicast (T110236)
  • 14:33 jzerebecki: that file is the same diff as the HEAD commit @tin:/srv/mediawiki-staging/php-1.27.0-wmf.21/extensions/CentralAuth ((398f6d4...) %)$ rm test.patch
  • 13:55 gehel: lowering disk high watermark on elasticsearch eqiad to rebalance the cluster
  • 13:07 jynus: restarting and reimaging db1052 (old enwiki master)
  • 12:08 mobrovac: restbase re-deployed 7f69f86ee9 to restbase1015 after reimaging
  • 10:54 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Depool db2068 for cloning to db2047 (duration: 01m 51s)
  • 10:11 jynus: stopping db1052 for backup to dbstore1002
  • 09:38 jynus: deleting imported logs on pc1004 and pc1005
  • 08:23 moritzm: restarted HHVM on mw1116 (output of hhvm-dump-debug available)
  • 08:19 jynus: dropping old imported logs from pc1006

2016-04-24

  • 20:45 Reedy: ran namespaceDupes.php on aswikisource
  • 20:04 ori: Deployed change Ib7e248ccf to statsv (commit id 5323cece2b3; task T132770)

2016-04-23

  • 19:09 _joe_: rebooted meitnerium
  • 02:22 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.21) (duration: 10m 27s)
  • 00:07 Jamesofur: deactivate phabricator account for ktr101 per global ban

2016-04-22

  • 23:33 mutante: powercycled mw1141
  • 23:26 mutante: reboot bast4001
  • 21:21 mutante: short gaps in codfw ganglia expected due to install2001 being an aggregator
  • 21:10 mutante: install2001 - reinstalled, re-adding to puppet etc
  • 20:36 mutante: reboot install2001 to PXE for OS upgrade\
  • 19:09 Krinkle: mwscript deleteEqualMessages.php --wiki thwikibooks
  • 15:54 bblack: testing https redir on ganglia.wm.o (uranium)
  • 15:47 moritzm: uploaded gerrit 2.12.2 for jessie-wikimedia to carbon
  • 15:27 godog: add new librenms template bound to 'port utilization over threshold' alert
  • 15:24 moritzm: uploaded gerrit 2.12.2 to carbon
  • 15:22 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Depool es2019 (duration: 00m 52s)
  • 14:14 mobrovac: restbase initial deploy of 7f69f86ee9 to restbase1015
  • 13:57 mobrovac: restbase bringing the service back up on restbase1007
  • 13:43 godog: bootstrap restbase1015-a T128107
  • 13:34 mobrovac: restbase stopping restbase on rb1007 to manually inspect why is it flapping
  • 13:27 akosiaris: upload 5.3.10-1ubuntu3.22+wmf1 on apt.wikimedia.org
  • 13:03 godog: depool restbase1007, 400s from restbase self check
  • 09:46 moritzm: installing PHP security updates
  • 09:25 moritzm: installing ircbalance bugfix updates (preventing massive logspam on some systems)
  • 02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Apr 22 02:31:45 UTC 2016 (duration 9m 15s)
  • 02:22 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.21) (duration: 10m 12s)

2016-04-21

  • 20:34 gehel: lowering elasticsearch high watermark to rebalance disk space across cluster
  • 20:07 gehel: reindex elasticsearch updates for the duration of the switch back from codfw, just in case
  • 18:57 mutante: rutherfordium - on 4.4.0-1-amd64 now (T131928)
  • 18:56 mutante: people.wm.org / rutherfordium , very short downtime, reboot
  • 18:43 mutante: rutherfordium (people.wm.org) - installing package upgrades
  • 18:29 mutante: bromine - on 4.4.0-1-amd64 now (T131928)
  • 18:25 mutante: bromine (annualreport,bz-static,transparency,releases) reboot for kernel upgrade
  • 18:15 mutante: bromine - apt-get dist-upgrade
  • 18:12 mutante: planet1001 - on 4.4.0-1-amd64 now (T131928)
  • 18:10 mutante: planet1001 - reboot for upgrade
  • 17:44 mutante: planet1001 - apt-get dist-upgrade (libc6, apache, ..)
  • 16:10 logmsgbot: marktraceur@tin Synchronized php-1.27.0-wmf.21/resources/src/mediawiki/api/upload.js: Unbreak finishing stash uploads in upload dialog (duration: 00m 27s)
  • 16:10 jynus: disabling event scheduler on db1041
  • 15:34 _joe_: services switchover done
  • 15:30 logmsgbot: volans@tin Synchronized wmf-config/db-eqiad.php: Adust weights for shard s7 - T133205 (duration: 00m 32s)
  • 15:28 _joe_: [switchback services] moving traffic for restbase/citoid/cxserver back to eqiad
  • 15:23 ostriches: ytterbium: puppet re-enabled for gerrit host
  • 15:23 logmsgbot: filippo@tin Synchronized wmf-config/filebackend-production.php: [switchover swift #1] async writes to codfw (duration: 00m 28s)
  • 15:15 godog: [switchover swift #2] upload backends from codfw to eqiad
  • 15:09 godog: [switchover swift #3] upload codfw to eqiad
  • 15:05 godog: [switchover swift #4] repool upload eqiad in dns
  • 15:04 gehel: starting reindex of lost elasticsearch updates during activation of SSL (T132762)
  • 15:01 godog: [switchover swift #5] upload esams to eqiad
  • 14:57 godog: [switchover swift #6] upload eqiad to 'direct'
  • 14:48 logmsgbot: gehel@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 29s)
  • 14:45 gehel: reverting elasticsearch traffic back to eqiad
  • 14:39 jynus: enabling replication lag alerts for all dbs
  • 14:28 paravoid: traffic/mediawiki codfw->eqiad switchover is done
  • 14:26 jynus: deployed dns updated records for new eqiad masters
  • 14:25 _joe_: [switchover #8/#3] Running rebuildEntityPerPage.php
  • 14:25 _joe_: [switchover #8/#2] Enabling crons in eqiad
  • 14:24 ori: [switchover #8/#1] Starting the jobqueue in eqiad
  • 14:23 volans: [switchover #8/#3] Re-enable puppet on all eqiad and codfw databases masters
  • 14:21 paravoid: wikis are read-write again
  • 14:21 logmsgbot: ori@tin Synchronized wmf-config/db-eqiad.php: [switchover #7/#1] Iac92c8bc6b: Put eqiad in read-write mode for datacenter switchover to eqiad (duration: 00m 39s)
  • 14:20 volans: [switchover #6/#1] Set database masters RW in eqiad for s1-7, es2-3 and x1
  • 14:19 bblack: [switchover #5/#6] Switch Varnish MW backends to eqiad - DONE (confirmed)
  • 14:18 Krinkle: apache-fast-test warmup finished
  • 14:18 godog: [switchover #5/#7] finished swift-proxy roll restart
  • 14:15 bblack: [switchover #5/#6] Switch Varnish MW backends to eqiad - starting
  • 14:14 mobrovac: [switchover #5/2] restbase Yes Done
  • 14:11 subbu: parsoid deploy and restarts done
  • 14:10 godog: [switchover #5/#7] roll-restart swift-proxy in eqiad and codfw
  • 14:09 subbu: restarting parsoid on all nodes
  • 14:09 akosiaris: [switchover #5/#3] Misc services cluster (for the action API endpoint)
  • 14:09 mobrovac: [switchover #5/2] restbase puppet agent -tv && systemctl restart restbase
  • 14:08 Krinkle: krinkle@tin: bin/apache-fast-test wiki-urls-warmup1000.txt eqiad
  • 14:08 subbu: syncing parsoid code
  • 14:07 volans: [switchover #5/#5] Switch parsercache RO/RW
  • 14:07 _joe_: [switchover #5/1] switching redis replication manually
  • 14:07 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: [switchover #4/#2] I0e85c3d20: Switch wmfMasterDatacenter to eqiad (duration: 00m 26s)
  • 14:06 _joe_: [switchover #4/1] puppet merged
  • 14:03 volans: [switchover #3/#1] Set active site's databases (masters) in read-only mode except parsercache ones.
  • 14:03 _joe_: [swichover 3/2 wipe memcacheds]
  • 14:02 paravoid: wikis now in planned read-only mode, cf. http://blog.wikimedia.org/2016/04/18/wikimedia-server-switch/
  • 14:02 logmsgbot: ori@tin Synchronized wmf-config/db-codfw.php: [switchover #2/#1] Id8b2e7a05: Set codfw databases to read-only mode (duration: 00m 24s)
  • 14:00 jynus: disabled all db lag alerts
  • 13:55 volans: [switchover #1/#6] Switch pt-heartbeat from active site (codfw) to new site (eqiad) masters
  • 13:54 ori: [switchover #1/2] stopping jobrunners in codfw
  • 13:54 _joe_: [switchover #1/3] stopping crons on wasat
  • 13:52 volans: [switchover #1/#5] Set final $master status for databases in advance
  • 13:50 volans: [switchover #1/#4] Disable puppet on all eqiad and codfw databases masters
  • 13:50 paravoid: commencing codfw->eqiad datacenter switchover
  • 13:39 logmsgbot: ori@tin Synchronized wmf-config/InitialiseSettings.php: I2171f6b1: Enable MessageCacheError log channel (duration: 00m 25s)
  • 13:37 bblack: [traffic codfw switch revert #3] - DNS TTL done, bulk of end-user traffic rebalanced, graphs starting to level off at new normals, as done as it gets from our end
  • 13:31 bblack: [traffic codfw switch revert #4] - done & confirmed
  • 13:28 bblack: [traffic codfw switch revert #4] - merge -> start salted puppet
  • 13:27 bblack: [traffic codfw switch revert #2] - done & confirmed
  • 13:25 bblack: [traffic codfw switch revert #3] - merge -> authdns-update
  • 13:24 bblack: [traffic codfw switch revert #2] - merge -> start salted puppet
  • 13:23 bblack: [traffic codfw switch revert #1] - done & confirmed
  • 13:23 bblack: [traffic codfw switch revert #1] - merge -> start salted puppet (@13:20, late log)
  • 13:21 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.21/includes: Ie9799f5ea: Make MessageCache handle lock timeouts better (duration: 01m 18s)
  • 13:12 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Temporarely increase es1* master weight to add connection capacity (duration: 00m 37s)
  • 09:57 elukey: removed apache2 logrotate config manually from argon as temp patch to remove cronspam from root@ (T132896)
  • 08:36 jynus: restarting db1031 to apply new mysql config
  • 02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Apr 21 02:31:04 UTC 2016 (duration 8m 37s)
  • 02:22 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.21) (duration: 09m 48s)
  • 01:49 mutante: git pull on strontium, ops/puppet
  • 01:48 mutante: belated log: restarted slapd on seaborgium
  • 01:29 ori: installed python-progressbar on terbium for warmup script, will be puppetized later

2016-04-20

  • 22:18 mutante: creating ganeti VM install1001 on eqiad cluster
  • 19:03 AaronSchulz: Cleared out 'enqueue' job queues to see if corruption comes back
  • 18:17 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Promote db1031 as the new x1 eqiad local master (duration: 00m 28s)
  • 18:16 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.21/extensions/Translate/messagegroups/WikiPageMessageGroup.php: I331bd93b: Avoid more master queries on page views (duration: 00m 31s)
  • 18:16 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.21/includes/jobqueue/JobQueueGroup.php: Ie9799f5ea: Catch errors in pushLazyJobs() and log them (duration: 00m 36s)
  • 18:00 jynus: changing database topology to set db1031 as the master of x1 on eqiad
  • 17:58 volans: Upgrading db1065 and fixing overheathing problems T132515
  • 17:30 volans: Upgrading db1070 and fixing overheathing problems T132515
  • 17:19 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.21/includes/jobqueue/JobQueueRedis.php: 86d185a4bbf52d (duration: 00m 39s)
  • 17:15 volans: Upgrading db1071 and fixing overheathing problems T132515
  • 17:03 akosiaris: aptitude purge php5-xhprof on uranium
  • 16:54 elukey: replaced "#" with ";" manually in uranium's /etc/php5/cli/conf.d/20-xhprof.ini and /etc/php5/apache2/php.ini to avoid cronspam (didn't find puppet/package trails)
  • 15:43 ebernhardson: delete apifeatureusage-2016.01.20 from codfw elasticsearch cluster. Index should never have existed in this cluster (and is beyond retention).
  • 15:42 ebernhardson: delete apifeatureusage-2016-01-(02,09,10) from eqiad elasticsearch cluster. We only keep 30 days of apifeatureusage logs
  • 15:37 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Tweak DB weights for better latency, avoiding peaks on QPS (duration: 00m 32s)
  • 15:18 ottomata: enabling puppet on analytics1015
  • 15:17 andrewbogott: re-imaging labtestvirt2001 and labtestneutron2001
  • 14:56 logmsgbot: volans@tin Synchronized wmf-config/db-eqiad.php: Change eqiad masters for s1,s3-s7 - T105135 (duration: 00m 28s)
  • 14:55 ottomata: started puppet on analytics1003
  • 14:52 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Repool es2019 (duration: 00m 38s)
  • 14:37 ottomata: stopping puppet on analytics1015 and analytics1003 in prep for migration
  • 13:54 elukey: puppet disabled on analytics1027 to stop Camus
  • 13:50 _joe_: rolling restart of ocg servers
  • 13:21 moritzm: rebooting rdb1002,rdb1003,rdb1004,rdb1006,rdb1007,rdb1008 for upgrade to Linux 4.4
  • 13:17 jynus: [switchover-maintenance] Changing DB slave topology for shard s1 on eqiad T111654
  • 12:59 volans: [switchover-maintenance] Changing DB slave topology for shard s4 on eqiad T111654
  • 12:54 volans: [switchover-maintenance] Changing DB slave topology for shard s5 on eqiad T111654
  • 12:48 jynus: [switchover-maintenance] Changing DB slave topology for shard s3 on eqiad T111654
  • 12:37 volans: [switchover-maintenance] Changing DB slave topology for shard s6 on eqiad T111654
  • 12:17 volans: [switchover-maintenance] Changing DB slave topology for shard s7 on eqiad T111654
  • 10:51 godog: pool restbase1014
  • 10:42 volans: [switchover-maintenance] Restarting db1028 (s7)
  • 10:33 jynus: [switchover-maintenance] Restarting db1018
  • 10:31 volans: [switchover-maintenance] Upgrading TLS for shard s7 on eqiad databases
  • 09:13 jynus: backfilling recentchanges on enwiki API servers
  • 09:12 godog: stop compactions on restbase1014-[ab]
  • 08:54 elukey: deployed the new puppet compiler - version 0.1.4 (hosts sorted in the HTML output, minor change)
  • 02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Apr 20 02:31:20 UTC 2016 (duration 8m 46s)
  • 02:22 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.21) (duration: 09m 37s)

2016-04-19

  • 22:42 ori: killing rc insert query on db1065 and db1066
  • 21:27 ori: running rebuildrecentchanges.php --from=20160419144741 --to=20160419151018 on all wikis
  • 21:12 paravoid: clearing the exim4 retry database on mx2001
  • 20:44 ori: on all wikis, deleting from recentchanges where rc_timestamp > 20160419144741 and rc_timestamp < 20160419151018
  • 20:09 ori: ran `mwscript rebuildrecentchanges.php --wiki=testwiki --from=20160419144741 --to=20160419151018`
  • 19:53 paravoid: staggered varnish bans for 'obj.http.server ~ "^mw2.+"' as a workaround for T133069
  • 19:51 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.21/maintenance/rebuildrecentchanges.php: Ie9799f5ea: rebuildrecentchanges: Allow rebuilding specified time range only (duration: 00m 28s)
  • 19:43 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Revert "Depool one db server from each shard as a backup" (duration: 00m 27s)
  • 19:12 AaronSchulz: Cleared enwiki 'enqueue' queue (T133089)
  • 19:06 legoktm: purging sidebar cache across all wikis (T133069)
  • 18:27 mutante: kraz.codfw, reinstalling as kraz.wikimedia
  • 18:00 _joe_: running rebuildEntityPerPage.php on wikidata, T133048
  • 17:49 jynus: setting binlog_format=ROW on old x1-master at eqiad (db1029) to reenable replication
  • 17:25 logmsgbot: demon@tin Synchronized php-1.27.0-wmf.21/extensions/CentralAuth: forgot something (duration: 00m 42s)
  • 17:17 volans: Deleting pc1003* and pc1006* binlog from pc2006 to make some space
  • 17:12 volans: Deleting pc1005* binlog from pc2005 to make some space
  • 17:08 volans: Deleting pc1002* old binlog from pc2005 to make some space
  • 17:01 ostriches: ytterbium: stopped puppet for a bit, testing host key mess.
  • 16:55 ostriches: restarting gerrit to pick up furud's rsa key
  • 15:49 bblack: [traffic codfw switch #4] - puppet change complete - done
  • 15:49 bblack: [traffic codfw switch #2] - confirmed bulk of traffic moved after ~10min for DNS TTL, rates levelling out on eqiad+codfw front network stats
  • 15:46 bblack: [traffic codfw switch #4] - salting puppet change
  • 15:45 bblack: [traffic codfw switch #4] - puppet merging eqiad text -> codfw
  • 15:41 bblack: [traffic codfw switch #3] - puppet change complete - done
  • 15:39 bblack: [traffic codfw switch #3] - salting puppet change
  • 15:38 bblack: [traffic codfw switch #3] - puppet merging esams text -> codfw
  • 15:35 bblack: [traffic codfw switch #2] - authdns-update complete, user traffic to eqiad frontends should start dropping off now
  • 15:34 bblack: [traffic codfw switch #1] - puppet change complete - done
  • 15:31 bblack: [traffic codfw switch #1] - salting puppet change
  • 15:30 bblack: [traffic codfw switch #1] - puppet merging text caches -> direct
  • 15:25 logmsgbot: ori@tin Synchronized wmf-config/ProductionServices.php: Iee2e08df5: Fix codfw redis hostnames [no-op, already synced as live hack] (duration: 00m 36s)
  • 15:11 andrewbogott: testing the log by logging a test
  • 15:10 logmsgbot: ori@tin Synchronized wmf-config/ProductionServices.php: live-hack fix for rdb2*.eqiad (duration: 00m 34s)
  • 15:02 _joe_: [switchover #13] starting maintenace jobs
  • 15:00 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: db weight teaking to better process the load (duration: 00m 28s)
  • 15:00 jynus: applying database weight changes
  • 14:48 paravoid: sites are read-write again
  • 14:48 logmsgbot: ori@tin Synchronized wmf-config/db-codfw.php: [switchover #12] I5e9635b8f4: Set codfw databases in read-write (duration: 00m 35s)
  • 14:43 jynus: [swithchover #11-2] Set and confirmed codfw master dbs in read-write
  • 14:37 godog: [switchover #10] puppet and swift reload finished
  • 14:34 jynus: [switchover #11] applying $master change for codfw masters
  • 14:32 godog: [switchover #10] running puppet on ms-fe and reload swift
  • 14:25 mobrovac: [switchover #7] restbase now uses MW from codfw
  • 14:25 bblack: [switchover #9] varnish - puppet runs complete - done
  • 14:20 bblack: [switchover #9] varnish - change merged, puppet runs starting
  • 14:19 _joe_: [switchover #7] memcached redises are now masters in codfw, running puppet on eqiad to start replicating
  • 14:19 akosiaris: [switchover #8] restarting parsoid on all wtp nodes
  • 14:19 subbu: manually restarted parsoid on wtp1001 and confirmed html identical before/after switchover on enwiki:Hospet
  • 14:16 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: Idbfb0184d: Switch wmfMasterDatacenter to codfw (duration: 00m 30s)
  • 14:16 akosiaris: [switchover #7] puppet agent -t -v on SCA, SCB cluster
  • 14:15 mobrovac: [switchover #7] puppet agent -tv && restbase restart
  • 14:15 ori: [switchover #7] Switch wmfMasterDatacenter to codfw (https://gerrit.wikimedia.org/r/#/c/282897/)
  • 14:14 _joe_: [switchover #7] running puppet on mc* hosts in codfw
  • 14:14 paravoid: [switchover #7] setging mediawiki master datacenter to codfw in puppet
  • 14:10 _joe_: [switchover #6] wiped memcached
  • 14:10 jynus: [switchover #5] DB Masters on eqiad set as read-only, and confirmed it
  • 14:09 _joe_: [switchover #6] disabled puppet on all redis hosts as a safety measure before inverting replication after the puppet change
  • 14:03 paravoid: sites in planned readonly-mode, cf. http://blog.wikimedia.org/2016/04/18/wikimedia-server-switch/
  • 14:02 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Set mediawiki-eqiad in read-only mode for datacenter switchover to codfw (duration: 00m 35s)
  • 14:01 jynus: [switchover #4] Set mediawiki-eqiad in read-only mode for datacenter switchover to codfw
  • 13:56 _joe_: [switchover #3] disabling cronjobs on terbium
  • 13:55 ori: [switchover #1]: disabling eqiad jobrunners via "salt -C 'G@cluster:jobrunner and G@site:eqiad' cmd.run 'service jobrunner stop; service jobchron stop;'".
  • 13:47 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool one db per shard as a backup (duration: 00m 27s)
  • 11:33 volans: changing binlog_format to STATEMENT for codfw masters for shards s1-s7 T124699
  • 10:07 jynus: updated dns entries about mysql masters
  • 09:16 godog: shutdown restbase100[56]
  • 09:11 godog: stop cassandra and restbase on restbase1006
  • 09:06 godog: stop compactions on restbase1014
  • 03:13 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool pc1006 (duration: 00m 28s)
  • 03:12 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Repool pc2006 (duration: 00m 31s)
  • 02:54 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.21/includes/api/ApiStashEdit.php: Ie9799f5ea: Segment stash edit cache stats by basis for hit/miss (duration: 00m 39s)
  • 02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Apr 19 02:31:46 UTC 2016 (duration 9m 46s)
  • 02:22 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.21) (duration: 09m 47s)
  • 02:06 urandom: systemctl mask cassandra-b on restbase2004.codfw.wmnet (it should not be running)
  • 01:34 mutante: restbase2004 - starting crashed cassandra-b service
  • 01:11 mutante: restbase2004 - unit cassandra-b is failed
  • 00:42 mutante: kraz - signing puppet certs, adding salt keys
  • 00:29 mutante: kraz.codfw.wmnet - initial install, adding to site

2016-04-18

  • 23:33 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I1547834: Remove unused ['bits'] configuration (duration: 00m 27s)
  • 23:22 YuviPanda: running sync-common on mw1145 to catch up on deploys
  • 23:17 YuviPanda: rebooted mw1145 from mgmt
  • 23:14 logmsgbot: ori@tin Synchronized wmf-config: I0ec3c015f: Update parser cache configuration for tag-based hashing (duration: 01m 48s)
  • 23:08 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Depool pc2006 (duration: 01m 01s)
  • 23:06 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.21/includes/objectcache/SqlBagOStuff.php: Ie9799f5ea: Allow tag names for SqlBagOStuff consistent hashing (duration: 04m 08s)
  • 22:52 logmsgbot: krinkle@tin Synchronized docroot/noc/conf: noc: remote bits references (duration: 00m 32s)
  • 22:45 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Repool pc2005 (duration: 00m 33s)
  • 22:44 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool pc1005 (duration: 00m 26s)
  • 22:20 chasemp: resetting labcontrol1001 puppet master with auth.conf which is fixing all the puppet clients in Labs
  • 21:33 gehel: deploying latest wdqs
  • 21:30 papaul: conf200[1-3] - signing puppet certs, salt-key, initial run
  • 20:58 papaul: OS install on conf200[1-3]
  • 20:55 mutante: furud.codfw - initial puppet install, signing certs
  • 18:43 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool pc1005 (duration: 00m 28s)
  • 18:42 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Depool pc2005 (duration: 00m 26s)
  • 18:37 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Repool pc2004 (duration: 00m 25s)
  • 18:27 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool pc1004 (duration: 00m 36s)
  • 17:31 elukey: restarted kafkatee on oxygen after module upgrade (was stuck from this morning due to a kafka broker restart)
  • 14:21 ottomata: upgraded pykafka on hafnium and restarted statsv
  • 14:04 andrewbogott: puppet agent -tv on labvirt1002 which restarted nrpe and resolved icinga alerts
  • 14:02 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool pc1004 (duration: 00m 29s)
  • 14:01 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Depool pc2004 (duration: 00m 27s)
  • 11:30 mobrovac: restbase restarting eqiad after https://gerrit.wikimedia.org/r/#/c/283958/
  • 10:07 _joe_: traffic from eqiad caches switched to codfw for restbase,citoid,cxserver
  • 09:43 moritzm: installing openssh updates on jessie systems
  • 09:20 _joe_: upgrading HHVM on terbium
  • 09:19 godog: depool restbase100[56].eqiad.wmnet, about to get decomissioned
  • 09:17 godog: repool restbase2006 after raid expansion
  • 09:08 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Update master comments in preparation for dc failover (duration: 00m 26s)
  • 09:07 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Update master comments in preparation for dc failover (duration: 01m 41s)
  • 07:02 jynus: changing binlog_format to MIXED on db2018

2016-04-17

  • 21:51 urandom: Decommissioning restbase1005.eqiad.wmnet : T95253
  • 21:50 urandom: `systemctl mask cassandra' on restbase1006.eqiad.wmnet (node is decommissioned) : T95253
  • 00:10 urandom: Decommissioning restbase1006.eqiad.wmnet : T95253

2016-04-16

  • 14:25 andrewbogott: rebooting mw1132 — OOM
  • 14:24 andrewbogott: rebooting mw1134 — OOM
  • 10:34 jynus: recreating _counters table on s3 to solve replication issues
  • 10:08 jynus: recreating hitcounter table on s3 to solve replication issues
  • 09:31 volans: Skipped on dbstore1002 query from replica: 'DELETE FROM `bawiktionary`.`hitcounter`'
  • 09:31 jynus: importing sys schema to db1075
  • 06:44 akosiaris: restart fermium to apply extra vcpus assignment
  • 06:23 volans: Stopped and restarted replica on dbstore2002 for s3 to "unstuck" the replica
  • 02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Apr 16 02:31:33 UTC 2016 (duration 9m 11s)
  • 02:22 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.21) (duration: 09m 03s)
  • 02:16 urandom: Bootstraping restbase1009-a.eqiad.wmnet : T95253

2016-04-15

  • 22:28 mutante: rebooting bast4001 - debugging install issue
  • 21:04 logmsgbot: volans@tin Synchronized wmf-config/db-eqiad.php: Repool new db1075,1077,1078 after TLS upgrade on s3 - T111654 (duration: 00m 36s)
  • 16:08 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: more liberal initialisesettings invalidation (duration: 00m 42s)
  • 15:34 godog: remove cassandra metrics for restbase100[1234]* restbase100[789] restbase2004 - T132771
  • 14:40 gehel: reindexing all wikis after the switch to codfw (T132762)
  • 14:06 urandom: start decommission of restbase1009-a.eqiad.wmnet : T95253
  • 12:15 gehel: increase max request size for elasticsearch
  • 12:14 volans: Re-arrange s3 replica topology: making codfw replicate from db1075 (this time for real) - T111654
  • 12:08 bblack: experimenting on carbon HTTP config (apt/mirrors/ubuntu.wm.o) - watch out for installer / package-update issues!
  • 12:00 gehel: remove maintenance from wdqs1002
  • 10:51 logmsgbot: volans@tin Synchronized wmf-config/db-eqiad.php: Depool new db1075,1077,1078 to upgrade TLS on s3 - T111654 (duration: 00m 41s)
  • 10:37 logmsgbot: volans@tin Synchronized wmf-config/db-eqiad.php: Repool db1042 after TLS upgrade on s4 - T111654 (duration: 00m 30s)
  • 10:21 _joe_: restarted zuul, zuul-merger on gallium
  • 10:10 Tim: on helium: scheduled restore of home_pmtpa to bast4001
  • 09:23 volans: Re-arrange s3 replica topology: making codfw replicate from db1075 - T111654
  • 09:01 gehel: reenabling wdqs1002 in varnish rotation after reinstall (T132387)
  • 08:09 logmsgbot: volans@tin Synchronized wmf-config/db-eqiad.php: Depool db1042 to upgrade TLS on s4 - T111654 (duration: 00m 36s)
  • 08:05 volans: starting TLS upgrade for shard s4 T111654
  • 08:00 gehel: removing empty log archives from Fluorine (T132324)
  • 07:40 moritzm: rebooting oresrdb* to Linux 4.4
  • 07:17 moritzm: rebooting oxygen for kernel update to 4.4
  • 02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Apr 15 02:30:21 UTC 2016 (duration 8m 55s)
  • 02:21 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.21) (duration: 08m 28s)

2016-04-14

  • 23:13 logmsgbot: demon@tin Synchronized portals: (no message) (duration: 00m 29s)
  • 23:12 logmsgbot: demon@tin Synchronized portals/prod/wikipedia.org/assets: (no message) (duration: 00m 29s)
  • 23:11 logmsgbot: demon@tin Synchronized php-1.27.0-wmf.21/includes/resourceloader/ResourceLoaderSpecialCharacterDataModule.php: I3e26d08a (duration: 00m 30s)
  • 23:08 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: Add images.unsplash.com to $wgCopyUploadsDomains (duration: 00m 33s)
  • 23:06 logmsgbot: demon@tin Synchronized portals/: updating to master, removing top-links A/B test (duration: 00m 45s)
  • 21:57 logmsgbot: bd808@tin Synchronized php-1.27.0-wmf.21/extensions/ContentTranslation: Enable europeana2802016 campaign (T125626) (duration: 00m 34s)
  • 21:44 logmsgbot: csteipp@tin Synchronized wmf-config/CommonSettings-labs.php: Syncing labs config change (duration: 00m 27s)
  • 21:43 logmsgbot: csteipp@tin Synchronized wmf-config/InitialiseSettings-labs.php: Syncing labs config change (duration: 00m 34s)
  • 20:31 ostriches: gerrit restarting
  • 19:35 godog: force puppet run on cache_upload in eqiad
  • 19:26 godog: force puppet run for cache_upload in esams
  • 19:11 godog: depool upload/eqiad
  • 19:01 logmsgbot: demon@tin rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.21
  • 18:58 godog: force puppet run on cache_upload in codfw
  • 18:51 godog: forcing puppet run on cache_upload in eqiad and codfw
  • 18:36 logmsgbot: demon@tin Finished scap: sync some symlink removals (duration: 04m 54s)
  • 18:31 logmsgbot: demon@tin Started scap: sync some symlink removals
  • 18:23 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: ldap logging tune (duration: 00m 34s)
  • 18:05 cwd: updated crm from 5877ee71abc21140e60934aef627003cfb2d11cc to 2f40e829195dec1cec71d08dc9c656eb247631ae
  • 17:14 godog: set zone access for all - private wikis in codfw
  • 16:49 godog: run on tin: mwscript extensions/WikimediaMaintenance/filebackend/setZoneAccess.php commonswiki --backend=local-multiwrite
  • 16:22 godog: rollback varnish backends to eqiad for thumbs
  • 16:13 godog: route upload backends to codfw - T129089
  • 16:08 logmsgbot: filippo@tin Synchronized wmf-config/filebackend-production.php: swift codfw sync replication T129089 (duration: 00m 26s)
  • 16:00 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Enable GuidedTour extension on sq.wikipedia (T132412) (duration: 00m 26s)
  • 15:54 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Babel configuration for uz.wikipedia (part 1/2) (T131924) (duration: 00m 26s)
  • 15:51 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Enable Kartographer on pl.wikimedia (T132510) (duration: 00m 26s)
  • 15:48 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Set wgKartographerWikivoyageMode (duration: 00m 26s)
  • 15:42 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: HD logo for lad.wikipedia (T132120, 3/3) (duration: 00m 26s)
  • 15:42 logmsgbot: dereckson@tin Synchronized static/images/project-logos/ladwiki-2x.png: HD logo for lad.wikipedia (T132120, 2/3) (duration: 00m 25s)
  • 15:41 logmsgbot: dereckson@tin Synchronized static/images/project-logos/ladwiki-1.5x.png: HD logo for lad.wikipedia (T132120, 1/3) (duration: 00m 25s)
  • 15:38 logmsgbot: dereckson@tin Synchronized static/images/project-logos/vecwikisource.png: New logo for vec.wikisource (T132157) (duration: 00m 26s)
  • 15:37 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Add task identifier comment to vec.wikisource logo change (no-op, T132157) (duration: 00m 27s)
  • 15:33 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Add museumcommons.wikimedia.nl to wgCopyUploadsDomains (T131841) (duration: 00m 26s)
  • 15:29 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Add bio.acousti.ca to wgCopyUploadsDomains (T132140) (duration: 00m 27s)
  • 15:27 Dereckson: Synchronized wmf-config/InitialiseSettings.php: Change voteWiki to fa language temporarily (T132667) (duration: 00m 34s)
  • 15:14 moritzm: uploaded librsvg 2.40.5-1+deb8u1+wmf1 to carbon (T132584)
  • 14:24 ema: rebooting cp3022 to test workaroud for T131961
  • 14:20 paravoid: cr1-esams: enabling IPv4/IPv6 BGP sessions with TeliaSonera
  • 12:41 Dereckson: Ran initSiteStats.php for 10 wikis to fix negative/off-by-one statistics errors (T131306)
  • 11:53 elukey: deployed new puppet-compiler version - 0.1.3 (adding submodules support)
  • 09:00 moritzm: fixed stray salt minion processes on labsdb1003 (apparently caused by stale pidfile)
  • 03:43 Krinkle: mwscript deleteEqualMessages.php --wiki kawikiquote
  • 03:02 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Apr 14 03:02:03 UTC 2016 (duration 9m 25s)
  • 02:52 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.21) (duration: 12m 26s)
  • 02:31 bd808: backfilled missing SAL entries from 2016-04-13T10:56Z to 2016-04-13T20:20Z to https://tools.wmflabs.org/sal/production
  • 02:25 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.20) (duration: 11m 13s)
  • 01:29 mutante: mw1211 - restart hhvm
  • 01:26 mutante: rebooting unresponsive serpens.wm.org
  • 01:06 mutante: bast4001 back up unchanged - tftp wouldnt work across DC, probably network ACLs
  • 00:43 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.20/includes/libs/IEUrlExtension.php: Ie9799f5ea: Revert Hack IEUrlExtension::haveUndecodedRequestUri() to always return true (duration: 00m 30s)
  • 00:42 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.21/includes/libs/IEUrlExtension.php: Ie9799f5ea: Revert Hack IEUrlExtension::haveUndecodedRequestUri() to always return true (duration: 00m 32s)
  • 00:38 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I08cfeca7c: Force ['SERVER_SOFTWARE'] to be"Apache" (duration: 00m 26s)
  • 00:28 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.20/extensions/TextExtracts/: https://gerrit.wikimedia.org/r/#/c/282640/ (duration: 00m 26s)
  • 00:27 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.21/extensions/TextExtracts/: https://gerrit.wikimedia.org/r/#/c/282640/ (duration: 00m 26s)
  • 00:23 logmsgbot: dereckson@tin Synchronized wmf-config/abusefilter.php: Enable $wgAbuseFilterProfile for commonswiki (Gerrit:282806, Task T132200) (duration: 00m 28s)
  • 00:13 logmsgbot: dereckson@tin Synchronized php-1.27.0-wmf.21/extensions/AbuseFilter/AbuseFilter.class.php: Fixes to filter profiling (Gerrit:283333, 3/3) (duration: 00m 26s)
  • 00:12 logmsgbot: dereckson@tin Synchronized php-1.27.0-wmf.21/extensions/AbuseFilter/Views/AbuseFilterViewList.php: Fixes to filter profiling (Gerrit:283333, 1/3) (duration: 00m 26s)
  • 00:11 logmsgbot: dereckson@tin Synchronized php-1.27.0-wmf.21/extensions/AbuseFilter/Views/AbuseFilterViewEdit.php: Fixes to filter profiling (Gerrit:283333, 1/3) (duration: 00m 26s)
  • 00:07 mutante: rebooting bast4001 to PXE, no active users, no screens
  • 00:02 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Import sources on hi.wiktionary (Task T132417, Gerrit:282843) (duration: 00m 26s)

2016-04-13

  • 23:56 mutante: ssh alsafi
  • 23:46 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Enable Echo survey on French-language wikis (Task T131893, Gerrit:283330) (duration: 00m 26s)
  • 23:38 logmsgbot: dereckson@tin Synchronized php-1.27.0-wmf.20/extensions/Echo/modules/ooui/styles/mw.echo.ui.FooterNoticeWidget.less: Fix for Echo footer (Gerrit:282715) (duration: 00m 27s)
  • 23:35 logmsgbot: dereckson@tin Synchronized php-1.27.0-wmf.20/extensions/Echo/modules/ooui/mw.echo.ui.FooterNoticeWidget.js: Fixes for Echo (Gerrit:282714 + Gerrit:282715) (duration: 00m 26s)
  • 23:25 Dereckson: Ran mwscript updateArticleCount.php --wiki=plwikisource --update
  • 23:23 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Consider all pages as valid content articles on pl.wikisource (Task T131771, Gerrit:283349) (duration: 00m 35s)
  • 23:11 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Add signature edit button for the Comments namespace to ru.wikinews (Task T132241, Gerrit:283347) (duration: 00m 38s)
  • 22:23 logmsgbot: hoo@tin Synchronized php-1.27.0-wmf.21/./extensions/Wikidata/extensions/Wikibase/view/resources/jquery/wikibase/jquery.wikibase.statementview.RankSelector.js: touch (duration: 00m 26s)
  • 22:11 logmsgbot: hoo@tin Finished scap: Update Wikibase to master (wmf21) (duration: 32m 06s)
  • 21:39 logmsgbot: hoo@tin Started scap: Update Wikibase to master (wmf21)
  • 21:09 bd808: https://tools.wmflabs.org/sal missing entries since 2016-04-13T09:21. Needs to be backfilled
  • 20:20 bblack: re-pooling ulsfo traffic T128424
  • 19:56 logmsgbot: demon@tin rebuilt wikiversions.php and synchronized wikiversions files: (no message)
  • 19:24 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.21/includes/libs/IEUrlExtension.php: Live-hack IEUrlExtension::haveUndecodedRequestUri() to always return true (duration: 00m 33s)
  • 19:23 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.20/includes/libs/IEUrlExtension.php: Live-hack IEUrlExtension::haveUndecodedRequestUri() to always return true (duration: 00m 33s)
  • 18:48 gehel: activating cache headers for WDQS
  • 18:46 bblack: lvs4002 - enable->run puppet post-reboot
  • 18:45 bblack: lvs4001 - enable->run puppet post-reboot
  • 18:37 gehel: activated maintenance page for wdqs1002 (data load in progress)
  • 18:16 bblack: shutdown lvs400[12]
  • 18:06 mutante: bast1001 back with jessie
  • 18:03 robh: the cp sysetms in ulsfo will be rebooting into maint mode regularly for the next few hours. I'll be scheduling downtime for each host as I get to them, but not echoing every cp host reboot in SAL
  • 18:03 robh: the cp sysetms in ulsfo will be rebooting into maint mode regularly for the next few hours. I'll be scheduling for each host as I get to them, but not echoing every cp host reboot in SAL
  • 18:00 bblack: disable puppet, stop pybal on lvs400[12] (maint shutdown imminent, depooled from DNS since yesterday)
  • 17:46 robh: lvs4003 rebooted and back online, lvs4004 offlining for maint.
  • 17:41 mutante: tungsten stop and remove rsync package and config
  • 17:33 Krinkle: mwscript deleteEqualMessages.php --wiki cywiki (T45917)
  • 17:27 mutante: bast1001 - revoke puppet cert, delete salt key, reinstall with jessie
  • 17:15 mutante: rebooting bast1001 into PXE
  • 17:14 robh: lvs4003 going offline for maint (icinga has been silenced, i think ;)
  • 16:58 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings-labs.php: I5a0abcdc: Make MathML rendering default in labs (duration: 00m 39s)
  • 16:54 volans: completed TLS upgrade for s1 - T111654
  • 16:35 ottomata: rebuilding raid1 array on aqs1001 after hot swapping sdh
  • 16:32 logmsgbot: volans@tin Synchronized wmf-config/db-eqiad.php: Repool db1057 after TLS upgrade on s1 - T111654 (duration: 00m 26s)
  • 16:18 andrewbogott: rebooting mw1139 — OOM
  • 16:11 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.20/extensions/CentralNotice: SWAT: Update CentralNotice gerrit:283205 (duration: 00m 30s)
  • 16:10 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.21/extensions/CentralNotice: SWAT: Update CentralNotice gerrit:283206 (duration: 00m 33s)
  • 15:56 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.20/extensions/WikidataPageBanner: SWAT: Attempt at fixing table of contents problem gerrit:282994 (duration: 00m 28s)
  • 15:21 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.21/extensions/WikidataPageBanner: SWAT: Attempt at fixing table of contents problem gerrit:282995 (duration: 00m 29s)
  • 15:10 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix typo in newikibooks namespaces gerrit:283183 (duration: 00m 30s)
  • 14:53 volans: start upgrading TLS for cross-dc replica on shards s1 - T111654
  • 14:53 logmsgbot: volans@tin Synchronized wmf-config/db-eqiad.php: Depool db1057 to upgrade TLS on s1 - T111654 (duration: 00m 26s)
  • 14:48 elukey: Yarn nodemanager Xmx size bumped up from 1000m to 2048 on all the analytics* hosts to overcome the OutOfMemory errors.
  • 14:38 ottomata: restarting hadoop-yarn-nodemanager on all hadoop worker nodes one by one to apply increase in heap size
  • 14:35 godog: start cleanup on restbase100[569] - T128107
  • 14:25 volans: completed upgrading TLS for cross-dc replica on shards s2 - T111654
  • 13:52 logmsgbot: jzerebecki@tin Finished scap: php-1.27.0-wmf.21: Update Wikidata to wmf/1.27.0-wmf.21 (duration: 29m 56s)
  • 13:22 logmsgbot: jzerebecki@tin Started scap: php-1.27.0-wmf.21: Update Wikidata to wmf/1.27.0-wmf.21
  • 13:14 volans: start upgrading TLS for cross-dc replica on shards s2 - T111654
  • 11:46 logmsgbot: volans@tin Synchronized wmf-config/db-eqiad.php: Reduce db1050 weight - T111654 (duration: 00m 30s)
  • 11:22 volans: completed upgrading TLS for cross-dc replica on shards s6 and s7 - T111654
  • 10:56 logmsgbot: volans@tin Synchronized wmf-config/db-eqiad.php: Repool db1050 and db1041 after TLS upgrade - T111654 (duration: 00m 42s)
  • 09:22 volans: start upgrading TLS for cross-dc replica on shards s6 and s7 - T111654
  • 07:26 moritzm: temporarily bumped connection tracking table size on mw1163 to 512k (randomly spiking)
  • 02:27 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.20) (duration: 11m 56s)
  • 00:38 Krinkle: mwscript deleteEqualMessages.php --wiki newiki
  • 00:05 Krinkle: mwscript deleteEqualMessages.php --wiki maiwiki

2016-04-12

  • 23:59 Krinkle: mwscript deleteEqualMessages.php --wiki fawiktionary
  • 23:50 logmsgbot: mattflaschen@tin Synchronized php-1.27.0-wmf.21/resources/lib/oojs-ui/: OOjs UI hotfix for search box styling (duration: 00m 27s)
  • 23:32 Krinkle: mwscript deleteEqualMessages.php --wiki elwiki (T45917)
  • 23:29 mutante: mw1080 depooled because read-only fs
  • 23:26 mutante: depooled mw1080
  • 23:18 matt_flaschen: Syncing failing only on mw1080
  • 23:16 logmsgbot: mattflaschen@tin Synchronized dblists: Make flow dblist explicit, rather than computed, retry (duration: 00m 29s)
  • 23:12 logmsgbot: mattflaschen@tin Synchronized dblists: Make flow dblist explicit, rather than computed (duration: 00m 31s)
  • 22:36 mutante: powercycled mw1163
  • 22:17 mobrovac: mathoid deploying ca7680521
  • 20:38 Krinkle: mwscript deleteEqualMessages.php --wiki glwikiquote
  • 20:38 Krinkle: mwscript deleteEqualMessages.php --wiki glwiki
  • 20:38 Krinkle: mwscript deleteEqualMessages.php --wiki glwikibooks
  • 20:38 Krinkle: mwscript deleteEqualMessages.php --wiki glwikisource
  • 19:33 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.21/includes/db/loadbalancer/LBFactory.php: I7bc3b3aa: Revert "Measure commitMasterChanges() run time" (duration: 00m 27s)
  • 19:32 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.20/includes/db/loadbalancer/LBFactory.php: I7bc3b3aa: Revert "Measure commitMasterChanges() run time" (duration: 00m 27s)
  • 19:32 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.21/includes/filerepo/file/LocalFile.php: I6457cb91: Don't report image cache hits / misses (duration: 00m 31s)
  • 19:31 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.20/includes/filerepo/file/LocalFile.php: I6457cb91: Don't report image cache hits / misses (duration: 00m 39s)
  • 19:27 awight: update crm from a20ac4c64e195732a27d0e9cfd33f0c23f4a8d4e to 5877ee71abc21140e60934aef627003cfb2d11cc
  • 19:27 mutante: create ganeti VM kraz.codfw.wmnet
  • 19:01 logmsgbot: demon@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.21
  • 18:37 logmsgbot: demon@tin Finished scap: testwiki to wmf.21 and rebuild l10n (duration: 15m 09s)
  • 18:22 logmsgbot: demon@tin Started scap: testwiki to wmf.21 and rebuild l10n
  • 18:22 logmsgbot: demon@tin scap aborted: testwiki to wmf.21 and rebuild l10n (duration: 19m 23s)
  • 18:19 mutante: tin: rm php-1.27.0-wmf.12/13 from mw-staging
  • 18:03 logmsgbot: demon@tin Started scap: testwiki to wmf.21 and rebuild l10n
  • 17:57 mutante: ssh alsafi fixes ganeti VM timeouts once again
  • 17:02 gehel: reinstall of wdqs1002
  • 16:29 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-labs.php: No-op sync of beta cluster config change for T132408 (duration: 00m 26s)
  • 16:18 _joe_: reenabling puppet everywhere
  • 15:55 _joe_: disabling puppet on all mw hosts for apache change
  • 15:55 mutante: removed all email.donate. from DNS
  • 15:47 bblack: Draining traffic from ulsfo via GeoDNS updates for T128424 maintenance
  • 15:45 gehel: disabling wdqs1002 on the varnish cache::misc cluster via puppet
  • 15:10 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Make use of the local, not master, parsoid cluster gerrit:282894 (duration: 00m 28s)
  • 15:04 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor Single Edit Tab on the English Wikipedia gerrit:274131 (duration: 00m 55s)
  • 12:08 moritzm: uploaded ircd-ratbox 2.2.9-3 for jessie-wikimedia to carbon
  • 11:37 hashar: Restarting Jenkins
  • 11:34 hashar: Jenkins upgrading "Script Security Plugin" from 1.17 to 1.18.1 https://wiki.jenkins-ci.org/display/SECURITY/Jenkins+Security+Advisory+2016-04-11
  • 11:19 tgr: running CentralAuth/checkLocalUsers.php --delete on all wikis for T119736
  • 11:10 godog: start raid expansion on restbase2006 T127951
  • 11:03 godog: repool restbase2005 / depool restbase2006
  • 10:16 ema: restarted varnishstatsd-default.service on cp4017 (T132430)
  • 09:55 hoo: Updated Wikidata's property suggester with data from Monday's json dump
  • 09:21 ema: varnishstatsd-default restarted on cp4010 and cp4018
  • 08:35 moritzm: installing ntp bugfix updates (will trigger some ntpd Icinga warnings until the clocks have resynced)
  • 08:01 elukey: restarted hhvm on mw1111 - multiple SEGV signals, hhvm-dump-debug in /tmp/hhvm.8424.bt.
  • 08:00 volans: Set rpl_semi_sync_master_timeout=100 on db1038 T131753 (filling up erro log)
  • 02:33 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Apr 12 02:33:25 UTC 2016 (duration 8m 34s)
  • 02:24 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.20) (duration: 11m 20s)
  • 00:37 ori: Deleted refreshLinksDynamic jobs for commonswiki and enwiki (T132318)
  • 00:34 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.20/includes/page/WikiPage.php: Ie9799f5ea: Flag triggerOpportunisticLinksUpdate() behind $wgMiserMode (duration: 00m 31s)
  • 00:11 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.20/includes/collation/IcuCollation.php: Ie9799f5ea: Cache first-letter data in APC, if available (duration: 00m 25s)
  • 00:07 logmsgbot: dereckson@tin Synchronized php-1.27.0-wmf.20/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.ArticleTargetLoader.js: Check wpSection before converting textbox contents for use in VE (Gerrit:282827) (duration: 00m 25s)

2016-04-11

  • 23:34 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings-labs.php: Fix wgCopyUploadsDomains on Commons Beta (Task T132285, Gerrit:282495) — 2/2 (duration: 00m 25s)
  • 23:33 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings-labs.php: Fix wgCopyUploadsDomains on Commons Beta (Task T132285, Gerrit:282495) — 1/2 (duration: 00m 26s)
  • 23:30 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Add mergehistory right to eliminator group on ja.wikipedia (Task T131751, Gerrit:282055) (duration: 00m 29s)
  • 23:25 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Set wgSemiprotectedRestrictionLevels for fr.wikipedia (Task T132248, Gerrit:282469) (duration: 00m 29s)
  • 23:20 Krinkle: mwscript deleteEqualMessages.php --wiki idwiki (T45917)
  • 23:19 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Remove $wgApiFrameOptions = SAMEORIGIN override for UploadWizard wikis (Task T131182, Gerrit:282768) (duration: 00m 31s)
  • 23:17 logmsgbot: dereckson@tin Synchronized static/images/project-logos/vecwiki.png: New logo for vec.wikipedia (and end of celebration logo) (Task T132185, Gerrit:282396) (duration: 00m 24s)
  • 23:12 Krinkle: mwscript deleteEqualMessages.php --wiki zh_classicalwiki (T45917)
  • 23:09 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Remove $wgApiFrameOptions = SAMEORIGIN override for UploadWizard wikis (Task T131182, Gerrit:282768) (duration: 00m 27s)
  • 22:11 logmsgbot: volans@tin Synchronized wmf-config/db-eqiad.php: Repool db1049 after deploy of Puppet certs for TLS - T111654 (duration: 00m 30s)
  • 22:07 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.20/includes/page/WikiPage.php: Ie9799f5ea: Increase triggerOpportunisticLinksUpdate() backoff TTL (duration: 00m 34s)
  • 22:03 mutante: creating virtual machine furud.codfw.wmnet
  • 21:29 Krinkle: mwscript deleteEqualMessages.php --wiki bgwikiquote (T45917)
  • 21:29 Krinkle: mwscript deleteEqualMessages.php --wiki frwikibooks (T45917)
  • 21:22 ebernhardson: restore elasticsearch cluster high disk watermark to 90%
  • 21:07 logmsgbot: csteipp@tin Synchronized wmf-config/CommonSettings-labs.php: revert oath in labs (duration: 00m 27s)
  • 21:06 logmsgbot: csteipp@tin Synchronized wmf-config/InitialiseSettings-labs.php: reverting oath (duration: 00m 29s)
  • 21:01 bearND: mobileapps deployed 6ef3054
  • 20:55 logmsgbot: csteipp@tin Synchronized wmf-config/CommonSettings-labs.php: retry oath in labs (duration: 00m 29s)
  • 20:53 logmsgbot: csteipp@tin Synchronized wmf-config/InitialiseSettings-labs.php: retry oath in labs (duration: 00m 32s)
  • 20:43 bearND: starting mobileapps deploy
  • 20:41 Krinkle: mwscript deleteEqualMessages.php --wiki plwiki (T45917)
  • 20:19 arlolra: updated Parsoid to version e3766b79
  • 20:09 arlolra: synced code; restarted Parsoid on wtp1001.eqiad as a canary
  • 20:05 ebernhardson: temporarily changing elasticsearch high watermark to 75% to rebalance cluster
  • 20:03 volans: Deploy and use Puppet certs for TLS on cross-dc replica for shard s5 T111654
  • 20:00 mutante: antimony - shred and delete git.wikimedia.org SSL key
  • 19:57 logmsgbot: volans@tin Synchronized wmf-config/db-eqiad.php: Depool db1049 to deploy Pupper certs for TLS - T111654 (duration: 00m 28s)
  • 19:55 mutante: antimony - remove Apache package and config
  • 19:53 logmsgbot: volans@tin Synchronized wmf-config/db-eqiad.php: Depool db1049 to deploy Pupper certs for TLS - T111654 (duration: 02m 25s)
  • 19:52 mutante: powercycled mw1115
  • 19:41 mutante: antimony (gitblit) - stop Apache
  • 19:15 ejegg: extende globalcollect audit timeout from 90 min to 120 min for duration of NL campaign
  • 19:01 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.20/extensions/WikimediaEvents: I672624e9fc30: Collect impact of proposed ResourceLoader feature-test in statsd (duration: 00m 34s)
  • 18:08 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.20/includes/RevisionList.php: Undo oversight live hack (duration: 00m 25s)
  • 17:49 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.20/includes/RevisionList.php: Attempt to fix oversight timeouts (duration: 00m 28s)
  • 17:44 gehel: rebooting wdqs1002 for kernel upgrade
  • 17:36 gehel: rebooting wdqs1001
  • 17:28 gehel: deploying latest WDQS version
  • 16:16 ejegg: updated payments from f71d3fbc7b8331b0427748b3fd358b5e2fc626fa to f09297028acace67588c2de845b754e2ace75c97
  • 15:59 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Match JsonConfig change gerrit:282447 (duration: 00m 25s)
  • 15:54 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Enable Echo survey on French-language wikis" (duration: 00m 26s)
  • 15:40 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.20/extensions/MobileApp/config/config.json: SWAT: Roll out RESTBase usage to Android production app: 100% gerrit:282434 (duration: 00m 27s)
  • 15:29 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Echo survey on French-language wikis gerrit:282414 (duration: 00m 25s)
  • 15:27 elukey: restarted hhvm on mw1190 (hhvm-dump-debug saved in /tmp/hhvm.19289.bt)
  • 15:24 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: [Cleanup] Remove VisualEditor AutoAccountEnable config now unused gerrit:280870 (duration: 00m 25s)
  • 15:20 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: [Cleanup] Remove VisualEditor experimental config 2 of 2 gerrit:280869 (duration: 00m 28s)
  • 15:20 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: [Cleanup] Remove VisualEditor experimental config 1 of 2 gerrit:280869 (duration: 00m 28s)
  • 15:13 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor on the Project ("Wikipedya") of htwiki gerrit:281263 (duration: 00m 32s)
  • 15:10 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add flood group to ladwiki gerrit:282201 (duration: 00m 29s)
  • 15:05 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add new namespaces and new aliases for newikibooks gerrit:281443 (duration: 00m 45s)
  • 14:09 elukey: mw1140 powercycled (ssh not usable, can't access as root via console too)
  • 11:14 elukey: mw1148 powercycled, not responsive to ssh
  • 10:27 elukey: powercycled mw1144.eqiad.wmnet
  • 08:44 volans: Re-enabling Puppet on cluster mysql and parsercache to deploy change 282385, T111654
  • 08:30 godog: bootstrap restbase1014-b T128107
  • 08:28 godog: start raid expansion on restbase2005 T127951
  • 08:19 godog: repool restbase2002, depool restbase2005
  • 08:19 volans: Disabling Puppet on cluster mysql and parsercache to merge and test change 282385 on db2040, T111654
  • 07:05 moritzm: restarted hhvm on mw1143, mw1201 and mw1213
  • 04:42 Krinkle: mwscript deleteEqualMessages.php --wiki shwiki (T45917)
  • 04:34 MaxSem: ran `mwscript extensions/CentralAuth/maintenance/createLocalAccount.php --wiki=enwiki 'Corvin Victor Paul'` due to CA bug
  • 02:33 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Apr 11 02:33:52 UTC 2016 (duration 8m 32s)
  • 02:25 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.20) (duration: 11m 22s)

2016-04-10

  • 22:56 Krinkle: mwscript deleteEqualMessages.php --wiki srwiki (T45917)
  • 22:16 Krinkle: mwscript deleteEqualMessages.php --wiki yowiki (T45917)
  • 21:55 Krinkle: mwscript deleteEqualMessages.php --wiki trwikimedia (T45917)
  • 02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Apr 10 02:31:13 UTC 2016 (duration 8m 33s)
  • 02:26 Krinkle: mwscript deleteEqualMessages.php --wiki zh_yuewiki
  • 02:22 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.20) (duration: 10m 00s)

2016-04-09

  • 20:44 logmsgbot: krenair@tin Synchronized wmf-config/LabsServices.php: sync labs-only merge of gehel's that was just left unmerged on tin (duration: 00m 41s)
  • 11:12 volans: Disabling tendril on db2047 (needs to be reimaged) to avoid flooding logs of tendril DB - T132011
  • 11:05 volans: Disabling tendril on es2005-es2010 (out of prod hosts) to avoid flooding logs of tendril DB - T129452
  • 02:32 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Apr 9 02:31:58 UTC 2016 (duration 8m 40s)
  • 02:23 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.20) (duration: 10m 11s)

2016-04-08

  • 14:03 godog: correction of the above, restbase1014-a
  • 14:02 godog: start cassandra bootstrap of restbase2004-a
  • 10:15 moritzm: rebooting radium (tor node) for kernel upgrade to 4.4
  • 09:53 elukey: dumps.wikimedia.org is now accepting only https:// (redirecting http:// to https://)
  • 09:49 moritzm: rebooting krypton for kernel upgrade
  • 07:50 moritzm: enable base::firewall on netmon1001
  • 04:01 kart_: Update cxserver to bd4739b
  • 02:33 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Apr 8 02:33:06 UTC 2016 (duration 8m 29s)
  • 02:24 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.20) (duration: 10m 04s)
  • 00:04 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Set wgSemiprotectedRestrictionLevels for en.wikipedia (Task T131976) (duration: 00m 26s)

2016-04-07

  • 23:56 logmsgbot: dereckson@tin Synchronized static/images/project-logos/ladwiki.png: Logo update for lad.wikipedia (Task T118491) (duration: 00m 27s)
  • 23:46 Dereckson: Synchronized wmf-config/InitialiseSettings.php: Raise upload limit to 4 GB (Gerrit:280831) — erratum for 23:13
  • 23:42 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Raise upload-by-URL request timeout (task T118887) (duration: 00m 32s)
  • 23:26 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Disable Echo survey on all wikis except test (gerrit:282230) (duration: 00m 25s)
  • 23:13 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Raise upload-by-URL request timeout (T118887) (duration: 00m 41s)
  • 22:05 Pchelolo: finished update RESTBase to 7f69f86ee9
  • 22:00 Pchelolo: start update RESTBase to 7f69f86ee9
  • 21:32 logmsgbot: mattflaschen@tin Synchronized wmf-config/LabsServices.php: Beta Cluster change (duration: 00m 30s)
  • 21:30 logmsgbot: krinkle@tin Synchronized w/: I69653efe0f1968: rm old symlinks (duration: 00m 46s)
  • 19:55 bblack: stop->disable varnishrls service on non-text clusters (upload, maps, misc) - ( https://gerrit.wikimedia.org/r/#/c/281439/ )
  • 19:34 ottomata: restarting eventlogging so it runs out of the scap deploy in eventlogging/analytics
  • 19:15 logmsgbot: dduvall@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.27.0-wmf.20
  • 19:00 ejegg: updated paymentswiki fraud filter settings
  • 16:48 godog: start raid expansion on restbase2002 T127951
  • 16:37 godog: repool restbase2003, raid expansion finished, depool restbase2002
  • 15:30 logmsgbot: thcipriani@tin Synchronized wmf-config/ProductionServices.php: SWAT: Use local resources in codfw for parsoid, url-downloader and mathoid gerrit:279355 (duration: 00m 25s)
  • 15:22 logmsgbot: thcipriani@tin Synchronized wmf-config: SWAT: Use ProductionServices for the jobqueue configuration 3/3 gerrit:279350 (duration: 00m 31s)
  • 15:22 logmsgbot: thcipriani@tin Synchronized wmf-config/jobqueue.php: SWAT: Use ProductionServices for the jobqueue configuration 2/3 gerrit:279350 (duration: 00m 27s)
  • 15:21 logmsgbot: thcipriani@tin Synchronized wmf-config/ProductionServices.php: SWAT: Use ProductionServices for the jobqueue configuration 1/3 gerrit:279350 (duration: 00m 30s)
  • 15:06 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor Beta Feature on Wikisources, Wiktionaries gerrit:280828 (duration: 00m 30s)
  • 14:47 gehel: syncing config to activate switch of CirrusSearch to codfw
  • 14:35 gehel: switching CirrusSearch to use Elasticsearch cluster in codfw (again), testing on mw1017 first
  • 13:58 chasemp: labstore restart nfs-export that crashed
  • 13:53 logmsgbot: gehel@tin Synchronized wmf-config: switching CirrusSearch to use Elasticsearch cluster in codfw (duration: 00m 35s)
  • 13:48 gehel: switching CirrusSearch to use Elasticsearch cluster in codfw
  • 13:28 _joe_: restarted hhvm on mw1015
  • 13:22 logmsgbot: gehel@tin Synchronized wmf-config: Fix TTMServer elastic config (duration: 00m 32s)
  • 12:50 logmsgbot: hoo@tin Synchronized wmf-config/Wikibase.php: Bump $wgCacheEpoch on Wikidata after Property conversions (duration: 00m 26s)
  • 12:49 _joe_: restarting hhvm on mw1213, deadlock in HPHP::Treadmill::getAgeOldestRequest
  • 12:46 logmsgbot: gehel@tin Synchronized wmf-config: Point the codfw label back to the codfw cluster (duration: 00m 27s)
  • 12:37 logmsgbot: gehel@tin Synchronized wmf-config: Don't use HTTPS+pooling for labswiki (duration: 00m 27s)
  • 12:22 logmsgbot: gehel@tin Synchronized wmf-config: (no message) (duration: 00m 35s)
  • 12:08 gehel: enabling HTTPS + connection pooling for CirrusSerach on all mediawiki nodes (T131839)
  • 12:06 godog: remove restbase2004-b cassandra data directory
  • 11:57 godog: bump raid rebuild to 20MB/s on restbase2003
  • 10:03 _joe_: restarted hhvm on mw1114 (one of the child processes was defunct) and on mw1173 (deadlock in HPHP::Treadmill::getAgeOldestRequest)
  • 09:35 gehel: deploying HHVM HTTP pool sizing on all MW nodes (T131839 / https://gerrit.wikimedia.org/r/#/c/281881), not used yet, no impact expected
  • 09:28 elukey: de-pooling/re-pooling aqs100[23].eqiad.wmnet for nodejs upgrade
  • 09:19 elukey: re-added aqs1001.eqiad.wmnet to LVS pool via confctl
  • 08:59 elukey: removed aqs1001.eqiad.wmnet from LVS pool via confd for nodejs upgrade
  • 08:22 godog: stop cassandra bootstrap of restbase2004-b, not enough disk
  • 08:17 akosiaris: renamed duplicate cxserver-deploy and cxserver repos in phabricator to citoid-deploy and citoid respectively
  • 07:04 logmsgbot: volans@tin Synchronized wmf-config/db-codfw.php: Depool crashed db2047, needs to be reimaged T132011 (duration: 00m 38s)
  • 03:08 bd808: backfilled missing SAL entries from 2016-04-04T17:30Z to 2016-04-05T20:36Z to https://tools.wmflabs.org/sal/production
  • 03:08 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Apr 7 03:08:28 UTC 2016 (duration 9m 52s)
  • 02:58 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.20) (duration: 10m 00s)
  • 02:38 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.19) (duration: 17m 51s)
  • 02:09 chasemp: reboot db2047 (it's totally stuck w/ BUG: soft lockup - CPU#26 stuck for 22s! [migration/26:267] across)

2016-04-06

  • 23:44 logmsgbot: mattflaschen@tin Synchronized php-1.27.0-wmf.19/extensions/Flow/includes/Data/Listener/NotificationListener.php: Fix new topic notifications (duration: 00m 29s)
  • 23:40 logmsgbot: mattflaschen@tin Synchronized php-1.27.0-wmf.20/extensions/Flow/includes/Data/Listener/NotificationListener.php: Fix new topic notifications (duration: 00m 37s)
  • 23:11 logmsgbot: mattflaschen@tin Synchronized docroot/wikipedia.org/apple-app-site-association: Support handoff and credential sharing with the iOS app (duration: 00m 34s)
  • 22:49 bd808: Testing stashbot response in #wikimedia-fundraising
  • 22:38 awight: update payments from 6dbd26ce56f416af07655f9c500023096678450b to f71d3fbc7b8331b0427748b3fd358b5e2fc626fa
  • 22:36 awight: drupal update wmf_fredge_qc:7003
  • 22:34 awight: update fundraising-crm from 4cc17b635eb84204cced107d5de78533cc5ce06c to a20ac4c64e195732a27d0e9cfd33f0c23f4a8d4e
  • 21:57 awight: update fundraising-tools from d8c5cc0b399411efa3b0634fa891a236c3dbaab2 to b114b7174c3bd9bf53cd44bf55397049a03b96fb
  • 20:17 subbu: finished deploying parsoid sha 5f6c0c60
  • 20:10 subbu: synced code; restarted parsoid on wtp1001 as a canary
  • 20:06 subbu: starting parsoid deploy
  • 19:51 logmsgbot: dduvall@tin rebuilt wikiversions.php and synchronized wikiversions files: Promote labswiki to 1.27.0-wmf.20 following temporary rollback and fix
  • 19:49 logmsgbot: dduvall@tin Synchronized php-1.27.0-wmf.20/extensions/SemanticMediaWiki/includes/SMW_ParserExtensions.php: Replace usage of Title::newFromRedirect() (duration: 00m 38s)
  • 19:33 akosiaris: pool maps-test200{1,2,3} for kartotherian.svc.codfw.wmnet
  • 19:23 logmsgbot: dduvall@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.20
  • 19:22 akosiaris: bounce hhvm on mw1135, mw1145
  • 19:16 akosiaris: restart tilerator, tileratorui on maps-test200{1,2,3}
  • 19:15 akosiaris: restart kartotherian on maps-test200{1,2,3}
  • 18:58 YuviPanda: reboot notebook1001
  • 18:50 akosiaris: disable salt-minion on maps-test200{1,2,3} for maps services deployment. nodejs upgrade is in place
  • 17:00 logmsgbot: thcipriani@tin Finished scap: SWAT: Add user_wpzero AbuseFilter variable gerrit:281867 (duration: 27m 39s)
  • 16:32 logmsgbot: thcipriani@tin Started scap: SWAT: Add user_wpzero AbuseFilter variable gerrit:281867
  • 16:30 cwd: updated payments from bbf36d804220b61b8eb7e5bf7a9c427d98ae1aaa to 6dbd26ce56f416af07655f9c500023096678450b
  • 16:06 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove Language Overlay experiment gerrit:277837 (duration: 00m 26s)
  • 15:54 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Translate extension on uawikimedia gerrit:281403 (duration: 00m 28s)
  • 15:33 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: REVERT Enable Translate extension on uawikimedia gerrit:281403 (duration: 00m 25s)
  • 15:31 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Translate extension on uawikimedia gerrit:281403 (duration: 00m 27s)
  • 15:23 urandom: Restoring default stream throughput on restbase200{3,4-a}.codfw.wmnet
  • 15:20 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: REVERT Bump CirrusSearchRequestSet rev to 121456865906 PART II gerrit:280448 (duration: 00m 31s)
  • 15:17 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Bump CirrusSearchRequestSet rev to 121456865906 PART II gerrit:280448 (duration: 00m 30s)
  • 15:17 logmsgbot: thcipriani@tin Synchronized wmf-config/event-schemas: SWAT: Bump CirrusSearchRequestSet rev to 121456865906 PART I gerrit:280448 (duration: 00m 27s)
  • 15:15 ema: Upgrading cp* to jessie 8.4 point release and linux 4.4 (T131746, T131928). Not rebooting yet.
  • 15:07 logmsgbot: thcipriani@tin Synchronized docroot/mediawiki/xml: SWAT: Add Flow dumps schema gerrit:281640 (duration: 00m 28s)
  • 15:03 hashar: rebased php-1.27.0-wmf.19/MobileFrontend and php-1.27.0-wmf.20/MobileFrontend (single commit related to CI)
  • 14:34 logmsgbot: filippo@tin Synchronized wmf-config/ProductionServices.php: move mediawiki traffic back to restbase eqiad (duration: 00m 34s)
  • 14:26 elukey: hhvm restarted on mw1187
  • 14:15 bblack: rebooting baham (ns1) for 4.4 kernel + package updates
  • 14:05 godog: move restbase/citoid/cxserver varnish traffic back to eqiad
  • 13:45 ema: Upgrading cp1052 to jessie 8.4 point release and linux 4.4 (T131746, T131928)
  • 08:28 _joe_: disabling puppet on the mw servers to test hhvm changes
  • 06:07 _joe_: restarting HHVM on mw1134, deadlock in what appears to be HPHP::Treadmill::getAgeOldestRequest
  • 03:06 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Apr 6 03:06:18 UTC 2016 (duration 9m 27s)
  • 02:56 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.20) (duration: 09m 43s)
  • 02:31 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.19) (duration: 11m 44s)
  • 01:02 logmsgbot: krenair@tin Finished scap: https://gerrit.wikimedia.org/r/#/c/281846/ - add messages for the new extendedconfirmed protection (duration: 95m 03s)
  • 00:55 MaxSem: restarted hhvm on mw1119, stuck

2016-04-05

  • 23:30 ejegg: updated payments wiki from d50c1b15f3d09c4724d7ad28e3668368ac8112be to bbf36d804220b61b8eb7e5bf7a9c427d98ae1aaa
  • 23:27 logmsgbot: krenair@tin Started scap: https://gerrit.wikimedia.org/r/#/c/281846/ - add messages for the new extendedconfirmed protection
  • 23:27 MaxSem: restarted hhvm on mw1188, stuck
  • 23:18 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/281807/ (duration: 00m 35s)
  • 23:17 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/281807/ (duration: 00m 35s)
  • 21:43 cscott: restarted parsoid on all nodes to complete deploy of version a5be1cdc
  • 21:37 cscott: restarted parsoid on wtp2001.codfw.wmnet as a (better) canary
  • 21:34 cscott: restarted parsoid on wtp1001.eqiad.wmnet as a canary
  • 21:34 cscott: updated Parsoid to version a5be1cdc
  • 21:34 bd808: https://tools.wmflabs.org/sal missing entries since 2016-04-04T16:36
  • 20:36 logmsgbot: dduvall@tin Purged l10n cache for 1.27.0-wmf.18
  • 20:32 logmsgbot: dduvall@tin rebuilt wikiversions.php and synchronized wikiversions files: Group0 to 1.27.0-wmf.20
  • 20:29 logmsgbot: dduvall@tin Finished scap: testwiki to php-1.27.0-wmf.20 and rebuild l10n cache (duration: 32m 19s)
  • 19:57 logmsgbot: dduvall@tin Started scap: testwiki to php-1.27.0-wmf.20 and rebuild l10n cache
  • 19:31 marxarelli: Applying security patches to wmf/1.27.0-wmf.20 checkout on tin
  • 19:23 marxarelli: Cloning mediawiki for checkout of new 1.27.0-wmf.20 branch on tin
  • 18:11 ejegg: updated payments wiki from cc298682f7ee9f43f5be863c6078e1c2ba70e3e6 to d50c1b15f3d09c4724d7ad28e3668368ac8112be
  • 18:09 marxarelli: Creating new wmf/1.27.0-wmf.20 branch on tin
  • 17:00 ejegg: updated payments wiki from a9659965d8b55b11518680a1170242f311c7f1d2 to cc298682f7ee9f43f5be863c6078e1c2ba70e3e6
  • 15:43 godog: ms-be1019 to weight 3500 - T116842
  • 14:40 logmsgbot: oblivian@tin Synchronized wmf-config/ProductionServices.php: make mediawiki talk to codfw restbase only (duration: 00m 47s)
  • 14:10 _joe_: external traffic for restbase, citoid, cxserver fully switched to codfw
  • 14:05 ema: repooling cp1043 (maps) after varnish upgrade to 4.1.2-1wm2. Bug: T131830
  • 13:58 ema: depooling cp1043 (maps) for varnish upgrade to 4.1.2-1wm2. Bug: T131830
  • 13:52 ema: repooling cp1044 (maps) after varnish upgrade to 4.1.2-1wm2. Bug: T131830
  • 13:49 urandom: Throttling outbound stream throughput to 15Mbps on restbase2004-a.codfw.wmnet and restbase2003.codfw.wmnet : T95253
  • 13:40 ema: depooling cp1044 (maps) for varnish upgrade to 4.1.2-1wm2. Bug: T131830
  • 13:29 cmjohnson1: re-installing snapshot1007 with trusy
  • 13:14 elukey: restarted hhvm on mw1248 (hhvm was segfaulting)
  • 10:16 _joe_: all hhvm extensions uploaded for debian jessie as well
  • 07:14 _joe_: uploading the hhvm 3.12.1 backport package for jessie to reprepro
  • 02:36 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Apr 5 02:36:21 UTC 2016 (duration 9m 19s)
  • 02:27 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.19) (duration: 11m 55s)
  • 01:29 eileen: Updating CiviCRM from bfe563f2451725e734d6ddbddb3be716463a3512 to 4cc17b635eb84204cced107d5de78533cc5ce06c
  • 00:14 Krinkle: krinkle@terbium Running deleteEqualMessages.php over previously cleaned wikis with --lang-code (T45917)
  • 00:12 ori: renamed frontend.navtiming.loading -> frontend.navtiming.loadEventStart and frontend.navtiming.sending -> frontend.navtiming.fetchStart on graphite2001 and graphite1001 ahead of merging https://gerrit.wikimedia.org/r/#/c/281082

2016-04-04

  • 23:30 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/281565/ (duration: 00m 29s)
  • 23:25 logmsgbot: maxsem@tin Synchronized wmf-config/: https://gerrit.wikimedia.org/r/#/c/281070/ (duration: 00m 32s)
  • 23:18 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/280445/ (duration: 00m 27s)
  • 23:17 logmsgbot: maxsem@tin Synchronized static/images/project-logos/astwiki.png: https://gerrit.wikimedia.org/r/#/c/280445/ (duration: 00m 27s)
  • 23:13 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/281314/ (duration: 00m 28s)
  • 23:08 awight: update fundraising tool from 1bc23cb4bfaf2a9d4d215aad79dd67d891b5d973 to d8c5cc0b399411efa3b0634fa891a236c3dbaab2
  • 20:12 subbu: finished deploying parsoid sha 579ec3e6
  • 20:07 subbu: synced code; restarted parsoid on wtp1002 as a canary
  • 20:04 subbu: starting parsoid deploy
  • 19:48 ejegg: rolled payments-wiki back to a9659965d8b55b11518680a1170242f311c7f1d2
  • 19:40 apergos: disabled puppet on snapshot1001,2,4 while new hosts come on line, til probably Apr 5-6
  • 19:38 akosiaris: depool maps-test2004
  • 18:59 ejegg: updated payments-wiki from a9659965d8b55b11518680a1170242f311c7f1d2 to eac5f979099056ef142500570555771dfb1992c1
  • 18:36 tgr: ran MassMessages/sendMessages.php for T128056
  • 18:18 cmjohnson1: stat1002 swapping failed disk slot 11
  • 18:04 cmjohnson1: db1052 swapping failed disk slot 8
  • 17:37 cmjohnson1: shutting down iridium to reapply thermal paste
  • 17:30 greg-g: Phabricator going down in about 10 minutes to hopefully address the overheating issue: T131742
  • 16:36 urandom: Restarting bootstrap of restbase2004.codfw.wmnet : T95253
  • 15:53 greg-g: 15:45 < elukey> !log aqs1001 re-added to the aqs pool (nodejs NOT upgraded due to issues with Cassandra)
  • 15:42 Krenair: ran wikitech-static updates
  • 15:18 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.19/extensions/MobileApp/config/config.json: SWAT: Roll out RESTBase usage to Android production app: 50% gerrit:280957 (duration: 00m 46s)
  • 14:55 urandom: Restarting restbase2004-a.codfw.wmnet (cancelling bootstrap of 2004-b)
  • 14:46 elukey: de-pooled aqs1001.eqiad from the confd pool for nodejs upgrade
  • 13:40 ema: nginx rolling restart for openssl upgrade on cache hosts
  • 10:42 elukey: re-pooled aqs1001.eqiad (no node upgrade, need more info about restbase)
  • 10:23 godog: reduce reserved blocks for /srv on restbase2004
  • 09:59 moritzm: installing pcre3 updates
  • 09:57 godog: start expanding raid0 on restbase2003
  • 09:54 ema: nginx rolling restart for openssl upgrade: cp1046, cp1052, cp1068, cp1071, cp1099
  • 09:53 elukey: de-pooled aqs1001.eqiad.wmnet as pre-step for nodejs upgrade
  • 09:41 godog: depool restbase2003 before raid expansion
  • 09:37 godog: repool restbase2004
  • 09:33 godog: deploy restbase ba39d2bcd2f5 to restbase2004 before repooling
  • 09:26 volans: Re-enabling Puppet on cluster mysql and parsercache to deploy change 279596, T111654
  • 09:10 volans: Disabling Puppet on cluster mysql and parsercache to merge and test change 279596 on db2040, T111654
  • 08:50 moritzm: installing gnupg updates
  • 06:54 moritzm: installing apt bugfix updates
  • 04:18 Tim: iridium came back up, but mcelog reports high CPU temperature prior to the shutdown
  • 04:13 Tim: attempting to turn iridium back on via drac. "getraclog" says it powered itself off after resetting four times
  • 03:15 greg-g: Phabricator is down
  • 02:33 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Apr 4 02:33:39 UTC 2016 (duration 8m 31s)
  • 02:25 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.19) (duration: 11m 45s)
  • 00:49 logmsgbot: krinkle@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 29s)
  • 00:30 logmsgbot: krinkle@tin Synchronized w/static.php: (no message) (duration: 00m 35s)
  • 00:21 Krinkle: mwscript deleteEqualMessages.php --wiki ruwiki (T45917)
  • 00:06 Krinkle: mwscript deleteEqualMessages.php --wiki metawiki --delete

2016-04-03

  • 22:37 Krinkle: mwscript deleteEqualMessages.php --wiki eswiki --lang-code ca (T45917)
  • 22:37 Krinkle: mwscript deleteEqualMessages.php --wiki eswiki (T45917)
  • 22:34 Krinkle: mwscript deleteEqualMessages.php --wiki eswiki
  • 19:16 Krenair: mw1146 began to respond with 503s to all requests, tried restarting apache2/hhvm and shortly afterwards it started working again
  • 13:26 logmsgbot: reedy@tin Synchronized wmf-config/InitialiseSettings-labs.php: labs copy upload thing. noooooop for prod (duration: 00m 33s)
  • 13:02 logmsgbot: reedy@tin Synchronized wmf-config/InitialiseSettings.php: labs copy upload thing. noooooop for prod (duration: 00m 28s)
  • 11:37 logmsgbot: ori@tin Synchronized wmf-config/extension-list-labs: I0d081186: Also add Newsletter to extension-list-labs (duration: 00m 27s)
  • 11:34 logmsgbot: ori@tin Synchronized wmf-config/InitialiseSettings-labs.php: I3ffe65b8: Load the Newsletter extension on the beta cluster (2/2) (duration: 00m 33s)
  • 11:34 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings-labs.php: I3ffe65b8: Load the Newsletter extension on the beta cluster (1/2) (duration: 00m 34s)
  • 03:14 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.19) (duration: 56m 43s)

2016-04-02

  • 20:28 hashar: Restarted Jenkins on gallium
  • 19:16 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: 350K celebration logo for cs.wikipedia (T131605) (duration: 00m 29s)
  • 19:14 logmsgbot: dereckson@tin Synchronized static/images/project-logos/cswiki.png: 350K celebration logo for cs.wikipedia (duration: 00m 33s)
  • 10:45 logmsgbot: reedy@tin rebuilt wikiversions.php and synchronized wikiversions files: wikitech back to .19
  • 10:01 logmsgbot: reedy@tin Synchronized php-1.27.0-wmf.19/extensions/OATHAuth: Fix for 2FA testing (duration: 00m 30s)
  • 07:22 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.19/extensions/NavigationTiming/modules/ext.navigationTiming.js: T131565 (duration: 00m 33s)
  • 02:32 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Apr 2 02:32:42 UTC 2016 (duration 8m 36s)
  • 02:24 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.19) (duration: 10m 48s)
  • 02:03 logmsgbot: l10nupdate@tin LocalisationUpdate failed (1.27.0-wmf.18) at 2016-04-02 02:03:32+00:00
  • 01:04 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.19/includes/specials/SpecialRedirect.php: T131328 (duration: 00m 39s)

2016-04-01

  • 23:43 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.19/extensions/MobileFrontend/includes/MobileFrontend.hooks.php: T131337 (duration: 00m 38s)
  • 22:39 Pchelolo: rolling restart restbase to pick up https://gerrit.wikimedia.org/r/#/c/280951/ config change
  • 22:13 mutante: hooft - shutdown
  • 21:36 mutante: hooft: stopping ganglia aggregators, remove from icinga/storedconfigclean
  • 21:26 mutante: hooft.esams - stop puppet, stop salt
  • 20:57 mutante: bast3001 - sign puppet certs, salt keys
  • 19:20 logmsgbot: reedy@tin Synchronized wmf-config/InitialiseSettings-labs.php: noop change for labs (duration: 00m 29s)
  • 19:10 mutante: booting slauerhoff into PXE, installing as new bast3001
  • 19:10 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: Replace old copyright image config (duration: 00m 32s)
  • 19:09 logmsgbot: reedy@tin Synchronized wmf-config/InitialiseSettings-labs.php: consistency (duration: 00m 28s)
  • 17:45 bearND: mobileapps deployed 66f8dac
  • 17:12 bearND: starting mobileapps deploy (retry of yesterdays which did not complete)
  • 15:37 papaul: installing graphite2002
  • 15:04 YuviPanda: grafana made me and godog admins
  • 15:01 moritzm: rebooting copper for upgrade to 4.4
  • 13:39 gehel: deploying apache config for MW appservers: new cache-control headers for portals (https://gerrit.wikimedia.org/r/#/c/280204/ T126280)
  • 11:06 Dereckson: Imported Jerusalem_Old_City_Walking_to_the_Western_Wall_4K.webm to Commons (T131441)
  • 09:14 gehel: testing new Cache-Control headers on mw1017
  • 09:09 logmsgbot: reedy@tin Synchronized wmf-config/throttle.php: Fix my quote fail (duration: 00m 29s)
  • 08:42 logmsgbot: reedy@tin rebuilt wikiversions.php and synchronized wikiversions files: Revert labswiki to wmf.18 as 2FA seems to be broken
  • 08:38 godog: bootstrap restbase2004-b
  • 08:29 logmsgbot: dereckson@tin Synchronized wmf-config/throttle.php: Fix throttle rules (Gerrit change 280819). (duration: 00m 29s)
  • 08:11 _joe_: progressively reducing weight of the older servers in the api cluster
  • 07:56 _joe_: depooling mw1121-1130 from the api cluster
  • 07:15 logmsgbot: reedy@tin Synchronized wmf-config/throttle.php: T130460 Jerusalem Hackathon (duration: 00m 40s)
  • 07:00 _joe_: depooling mw1070-89 from the appserver cluster. T126242
  • 06:53 _joe_: setting all newer appservers weight to 20 in eqiad
  • 05:28 bearND: mobileapps deployed 66f8dac
  • 05:04 bearND: starting mobileapps deploy
  • 02:38 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Apr 1 02:38:39 UTC 2016 (duration 8m 58s)
  • 02:29 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.19) (duration: 10m 29s)
  • 00:55 logmsgbot: csteipp@tin Synchronized wmf-config/CommonSettings-labs.php: Synching labs revert (duration: 00m 31s)
  • 00:54 logmsgbot: csteipp@tin Synchronized wmf-config/InitialiseSettings-labs.php: Synching labs revert (duration: 00m 27s)
  • 00:37 logmsgbot: csteipp@tin Synchronized wmf-config/CommonSettings-labs.php: Sync labs change to keep repo clean (duration: 00m 31s)
  • 00:36 logmsgbot: csteipp@tin Synchronized wmf-config/InitialiseSettings-labs.php: (no message) (duration: 00m 34s)

2016-03-31

  • 23:48 logmsgbot: awight@tin Finished scap: (no message) (duration: 26m 44s)
  • 23:21 logmsgbot: awight@tin Started scap: (no message)
  • 23:14 logmsgbot: awight@tin Synchronized php-1.27.0-wmf.19/extensions/OATHAuth: SWAT deployment of OATHAuth fixes, take 2 (duration: 00m 32s)
  • 23:13 logmsgbot: awight@tin Synchronized php-1.27.0-wmf.19/extensions/OATHAuth: SWAT deployment of OATHAuth fixes (duration: 00m 46s)
  • 22:51 Pchelolo: rolling restart restbase. Apply https://gerrit.wikimedia.org/r/#/c/280711/ config change
  • 22:22 Pchelolo: finished update restbase to ba39d2bc
  • 22:15 Pchelolo: started update restbase to ba39d2bc
  • 22:06 Pchelolo: update restbase to ba39d2bc canary on restbase1005
  • 22:02 Pchelolo: reenable puppet on cerium in restbase staging
  • 21:26 Pchelolo: disabling puppet on cerium, updating config and deploying restbase to staging. Testing https://gerrit.wikimedia.org/r/#/c/280711/
  • 21:00 mutante: bast2001 has been reinstalled and can be used again. fingerprints at https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/bast2001.wikimedia.org
  • 20:38 mutante: bast2001 - revoking old, signing new puppet certs, salt key..
  • 20:21 mutante: bast2001 - reinstall after disk replacement
  • 20:01 ottomata: stopping eventlogging, uninstalling globally installed eventlogging python code, running puppet, restarting eventlogging from /srv/deployment/eventlogging/eventlogging
  • 19:59 ottomata: temporarily stopped puppet on eventlog1001
  • 19:12 volans: Starting slave for s2 on db1047
  • 19:04 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.27.0-wmf.19
  • 17:46 mutante: rcs1002 - stop ferm
  • 15:56 tgr: running checkLocalUser.php --delete on some wikis for T119736
  • 15:17 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.19/extensions/CirrusSearch/includes/ElasticsearchIntermediary.php: SWAT: Ignore ResultSets that do not return pages gerrit:280669 (duration: 00m 38s)
  • 14:01 volans: manually running puppet on es2018 to double verify merged changes
  • 13:55 mutante: hooft - reboot to pxe, one more time
  • 13:49 mutante: netmon1001 - re-enabled puppet (was for torrus issue earlier)
  • 13:44 elukey: forced puppet agent on netmon1001
  • 13:27 ema: repooling cp1044, upgraded to varnish 4 (T122880)
  • 13:17 moritzm: installing rsync security updates
  • 13:09 ema: depooling cp1044 for varnish 4 upgrade (T122880)
  • 12:02 moritzm: removed rsync 3.10-2ubuntu0.1~wmf1 from carbon. this backport was only needed when the hosts used a mix of precise and trusty. this is no longer the case, so remove the backport and allow to use stock Ubuntu updates on precise again
  • 10:42 elukey: stopping torrus-common on netmon1001 to try https://wikitech.wikimedia.org/wiki/Torrus#Deadlock_problem
  • 10:35 elukey: restarted torrus-common on netmon1001
  • 10:24 moritzm: logstash on logstash100[1-3] is now using systemd
  • 09:50 moritzm: disabled puppet on logstash1002/1003 (to activate the new logstash systemd unit in steps)
  • 09:00 volans: Forced WriteBack cache policy mode on db1047 RAID
  • 06:11 Krinkle: mwscript deleteEqualMessages.php --wiki zh_min_nanwiki (T45917)
  • 05:31 logmsgbot: krinkle@tin Synchronized docroot/wikipedia.org/speed-tests/: (no message) (duration: 00m 33s)
  • 05:13 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.19/extensions/MobileFrontend/: Iaa5ed38c712b19e (duration: 00m 31s)
  • 05:10 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.18/extensions/MobileFrontend/: Iaa5ed38c712b19e (duration: 00m 42s)
  • 02:48 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.19) (duration: 10m 26s)
  • 02:27 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.18) (duration: 11m 52s)
  • 00:30 mutante: argon - removing apache and config
  • 00:04 logmsgbot: krinkle@tin Synchronized errorpages/: (no message) (duration: 00m 33s)

2016-03-30

  • 23:42 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.19/extensions/Kartographer/: (no message) (duration: 00m 27s)
  • 23:35 yuvipanda: cleaned salt keys for analytics1017 and 1021
  • 23:34 yuvipanda: cleaned out puppet cert for analytics 1017 and 1021
  • 23:10 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.18/extensions/Kartographer/: https://gerrit.wikimedia.org/r/#/c/280600/ (duration: 00m 31s)
  • 23:08 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.19/extensions/Kartographer/: https://gerrit.wikimedia.org/r/#/c/280601/ (duration: 00m 34s)
  • 23:06 mutante: torrus - follow instruction for deadlock problem
  • 23:06 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/280449/ (duration: 00m 43s)
  • 22:13 mutante: smokeping/netmon - remove hooft/bast3001, restart service
  • 21:05 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.19/extensions/CirrusSearch/: https://gerrit.wikimedia.org/r/#/c/280549/ (duration: 00m 40s)
  • 20:20 subbu: finished deploying parsoid version a20ef276
  • 20:15 Pchelolo: finished update restbase to 1b52276f
  • 20:11 subbu: synced code; restarted parsoid on wtp1004 as a canary
  • 20:10 Pchelolo: starting update restbase to 1b52276f
  • 20:07 subbu: starting parsoid deploy
  • 20:03 Pchelolo: update restbase to 1b52276f canary deploy to restbase1005
  • 19:48 mutante: hooft revoking puppet cert, salt key, reboot into PXE
  • 19:43 mutante: hooft is going to be reinstalled and renamed, affects esams bastion and ganglia during the install
  • 19:39 mutante: hooft - updating root password
  • 19:08 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.19
  • 18:58 Pchelolo: doing a puppet run in codfw restbase nodes and performing a rolling restart of restbase
  • 18:55 Pchelolo: finished restbase update to 9fe8676d
  • 18:20 Pchelolo: starting restbase update to 9fe8676d
  • 17:46 logmsgbot: hoo@tin Synchronized wmf-config/interwiki.php: Sync changes on meta (duration: 00m 42s)
  • 17:22 akosiaris: restarting grrrit-wm
  • 16:24 ostriches: gerrit: restarting to clean up mismatched jvm versions
  • 15:41 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.18/extensions/Echo/modules/ooui/mw.echo.ui.NotificationBadgeWidget.js: SWAT: Change threshold for survey invitation from 2 unread notifs to 1 gerrit:280369 (duration: 00m 28s)
  • 15:32 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.19/extensions/Translate/resources/js/ext.translate.editor.js: SWAT: Fix regressions in insertables placement gerrit:280414 (duration: 00m 27s)
  • 15:26 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.19/extensions/ContentTranslation/modules/publish/ext.cx.publish.js: SWAT: Try to avoid JS error gerrit:280389 (duration: 00m 29s)
  • 15:20 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.19/extensions/Echo/modules/ooui/mw.echo.ui.NotificationBadgeWidget.js: SWAT: Change threshold for survey invitation from 2 unread notifs to 1 gerrit:280370 (duration: 00m 28s)
  • 14:53 akosiaris: restart zotero on sca2001, sca2002
  • 14:52 akosiaris: restart zotero on sca1001, sca1002
  • 14:34 mobrovac: restbase rolling restart to apply https://gerrit.wikimedia.org/r/#/c/280091/
  • 14:29 mobrovac: restbase deploy 3ea08751a8 on restbase2004 after reimage
  • 14:26 godog: bootstrap cassandra-a on restbase2004
  • 12:40 ema: repooling cp1043 running varnish4 (T122880)
  • 11:37 godog: reimage restbase2004.codfw.wmnet
  • 11:18 kart_: Updated cxserver to 5699a49
  • 10:25 jynus: starting regression/stress testing of codfw mediawiki infrastructure
  • 09:07 ema: depooling cp1043 for varnish4 upgrade (T122880)
  • 08:58 moritzm: repooling maps-test2001
  • 08:45 moritzm: depooling maps-test2001(to apply ferm)
  • 08:22 jynus: restarting db1047 due to data corruption on Aria tables
  • 06:56 moritzm: restarted hhvm on mw1128, mw1139
  • 06:06 Tim: on terbium: making visual diff dump lists with makeDumpList.php
  • 05:52 logmsgbot: tstarling@tin Synchronized php-1.27.0-wmf.18/extensions/WikimediaMaintenance/makeDumpList.php: (no message) (duration: 00m 27s)
  • 05:39 logmsgbot: tstarling@tin Synchronized php-1.27.0-wmf.19/extensions/WikimediaMaintenance/makeDumpList.php: (no message) (duration: 00m 38s)
  • 03:17 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Mar 30 03:17:15 UTC 2016 (duration 9m 29s)
  • 03:07 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.19) (duration: 18m 03s)
  • 02:32 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.18) (duration: 11m 29s)

2016-03-29

  • 23:55 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.19/extensions/WikidataPageBanner/: https://gerrit.wikimedia.org/r/#/c/280327/ (duration: 00m 33s)
  • 23:50 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.19/extensions/UploadWizard/: https://gerrit.wikimedia.org/r/#/c/280339/ (duration: 00m 33s)
  • 23:48 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.18/extensions/WikidataPageBanner/: https://gerrit.wikimedia.org/r/#/c/280328/ (duration: 00m 38s)
  • 23:10 awight: update payments from 5a1996bc21fe694b99556bc1b501e484075dabe2 to a9659965d8b55b11518680a1170242f311c7f1d2
  • 22:47 dapatrick: Deployed patch for T123653 to wmf18 and wmf19
  • 22:23 awight: update payments to experimentally rollback
  • 22:21 csteipp: Deployed patch for T127420 to wmf18 and wmf19
  • 21:47 awight: updating payments from 5a1996bc21fe694b99556bc1b501e484075dabe2 to a9659965d8b55b11518680a1170242f311c7f1d2
  • 21:40 ejegg: enabled thank you mail send job
  • 21:32 ejegg: updated CiviCRM from 18431eec74de251066deb77d5b63607e06f6f135 to bfe563f2451725e734d6ddbddb3be716463a3512
  • 20:53 awight: rollback payments
  • 20:52 awight: update payments from 5a1996bc21fe694b99556bc1b501e484075dabe2 to a9659965d8b55b11518680a1170242f311c7f1d2
  • 20:48 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: Wikipedia course, Prague throttle rule gerrit:280255 (duration: 00m 38s)
  • 20:48 mutante: welcome Dereckson to Mediawiki deployers
  • 20:29 logmsgbot: thcipriani@tin Purged l10n cache for 1.27.0-wmf.15
  • 20:27 awight: rolled back payments to 5a1996bc21fe694b99556bc1b501e484075dabe2
  • 20:26 awight: update payments from 5a1996bc21fe694b99556bc1b501e484075dabe2 to a9659965d8b55b11518680a1170242f311c7f1d2
  • 20:24 ejegg: disabled thank you mail send
  • 20:10 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.27.0-wmf.19
  • 20:02 logmsgbot: thcipriani@tin Finished scap: testwiki to php-1.27.0-wmf.19 and rebuild l10ncache (duration: 56m 27s)
  • 19:47 cmjohnson1: disabling puppet on hosts cp1056/57 and cp1069/70. All are being reclaimed
  • 19:06 logmsgbot: thcipriani@tin Started scap: testwiki to php-1.27.0-wmf.19 and rebuild l10ncache
  • 18:51 jynus: performing schema change on testwiki T130692
  • 18:39 mutante: alsafi deleted service template file remnant
  • 17:36 thcipriani: starting branch cut for wmf.19
  • 17:00 ejegg: updated SmashPig from 9f08f6a1891b0a2bb70eacf460c2f9a8153c3b4e to 5cadcf3abcfcda4552b068c783337d82b743e2e5
  • 16:34 gehel: restarting apache on palladium, strontium and rhodium. Restart should be graceful. In case it is not, puppet errors will happen.
  • 16:20 jynus: upgrading some packages on silver, restarting bacula-fd
  • 16:17 gehel: restarting puppetmaster on palladium
  • 15:59 logmsgbot: thcipriani@tin Synchronized wmf-config: SWAT: Use extension registration for DoubleWiki gerrit:280242 (duration: 00m 28s)
  • 15:52 logmsgbot: thcipriani@tin Synchronized wmf-config: SWAT: Use extension registration for UnicodeConverter gerrit:280227 (duration: 00m 28s)
  • 15:48 logmsgbot: thcipriani@tin Synchronized wmf-config: SWAT: Use extension registration for Poem gerrit:280226 (duration: 00m 28s)
  • 15:36 logmsgbot: thcipriani@tin Synchronized wmf-config: SWAT: Revert Use extension registration for DoubleWiki (duration: 00m 27s)
  • 15:33 logmsgbot: thcipriani@tin Synchronized wmf-config: SWAT: Use extension registration for DoubleWiki gerrit:280225 (duration: 00m 28s)
  • 15:29 logmsgbot: thcipriani@tin Synchronized wmf-config: SWAT: Use extension registration for SyntaxHighlight_GeSHi gerrit:280224 (duration: 00m 28s)
  • 15:26 logmsgbot: thcipriani@tin Synchronized wmf-config: SWAT: Use extension registration for ImageMap gerrit:280223 (duration: 00m 31s)
  • 15:12 logmsgbot: thcipriani@tin Synchronized static/images/project-logos/astwiktionary.png: SWAT: Logo for ast.wiktionary gerrit:280057 (duration: 00m 27s)
  • 15:08 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: HD logo for da.wikipedia gerrit:279897 (duration: 00m 27s)
  • 15:07 logmsgbot: thcipriani@tin Synchronized static/images/project-logos: SWAT: HD logo for da.wikipedia gerrit:279897 (duration: 00m 38s)
  • 15:06 moritzm: repooled maps-test2004
  • 15:02 moritzm: repooled maps-test2003 and depooled maps-test2004 (to apply ferm)
  • 14:54 moritzm: depooled maps-test2003 (to apply ferm)
  • 13:53 moritzm: repooled maps-test2002 (nodejs being put on hold prevented generation of ferm rules provided by service::node, this has been fixed)
  • 13:14 jynus: restoring (but not provisioning) older backup from labswiki sql
  • 13:08 moritzm: unheld nodejs on maps*
  • 11:51 hashar: Jenkins / Zuul lagging out trying to catch up with a huge number of changes I have sent
  • 10:11 logmsgbot: reedy@tin Synchronized wmf-config/throttle.php: Remove old throttle rules (duration: 00m 44s)
  • 09:50 godog: powercycle ms-be2008, got stuck again while diagnosing failed /dev/sdl
  • 09:08 paravoid: hard-resetting alsafi, I/O-stuck (qemu bug?)
  • 09:05 paravoid: hard-resetting mx1001, I/O-stuck (qemu bug?)
  • 08:45 moritzm: upgraded chromium on osmium
  • 03:29 bblack: cr1/2-eqiad: re-activating static routes for ns0/ns1 (ipv4/ipv6) pointing to radon -
  • 02:45 Krinkle: mwscript deleteEqualMessages.php --wiki zhwikisource
  • 02:45 Krinkle: mwscript deleteEqualMessages.php --wiki zhwikinews
  • 02:36 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Mar 29 02:36:39 UTC 2016 (duration 8m 46s)
  • 02:27 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.18) (duration: 11m 34s)
  • 01:07 Pchelolo: restbase finish update to 3ea08751a8
  • 01:00 Pchelolo: restbase start update to 3ea08751a8
  • 00:15 logmsgbot: krinkle@tin Synchronized docroot/wikipedia.org/speed-tests/: (no message) (duration: 00m 32s)
  • 00:12 mutante: mw2090 - reinstalled, re-signed puppet/salt

2016-03-28

  • 23:38 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.18/includes/api/ApiUpload.php: https://gerrit.wikimedia.org/r/#/c/280092/ (duration: 02m 26s)
  • 23:34 mutante: mw2090 - reboot, reinstall as regular appserver
  • 23:27 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/280115/ (duration: 00m 35s)
  • 23:19 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.18/extensions/Echo/: https://gerrit.wikimedia.org/r/#/c/280112/ (duration: 00m 43s)
  • 23:16 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.18/extensions/MobileApp/: https://gerrit.wikimedia.org/r/#q,279980,n,z (duration: 01m 07s)
  • 22:03 mutante: mw2090 - disable puppet/cron, crons now on wasat
  • 21:46 mutante: mw1119 restarted hhvm
  • 21:26 bblack: http/2 enabled on pinkunicorn.wikimedia.org for testing - T96848
  • 21:26 bblack: http/2 enabled on pinkunicorn.wikimedia.org for testing
  • 21:12 Pchelolo: restbase finised deploy of 78e6eab37c
  • 21:04 Pchelolo: restbase start deploy of 78e6eab37c
  • 21:04 mdholloway: mobileapps deployed 90cfdcd
  • 20:37 Pchelolo: restbase canary deploy of 78e6eab37c to restbase1005
  • 20:35 mdholloway: starting mobileapps deployment
  • 20:09 ejegg: restarted thank you email job
  • 19:53 ejegg: rolled civicrm back to 18431eec74de251066deb77d5b63607e06f6f135
  • 19:27 ejegg: updated civiCRM from 18431eec74de251066deb77d5b63607e06f6f135 to d95c93f73a9329d6367403ac3e7d60c97987e61f
  • 19:11 ejegg: disabled thank you mail job
  • 18:30 andrewbogott: labs public DNS will remain in a transitional state until MarkMonitor updates their records and the TTL expires. DNS resolution should proceed as usual in the meantime.
  • 18:03 mutante: mw1140 restart hhvm
  • 17:52 cmjohnson1: shutting down radon to re-apply thermal paste:
  • 17:45 mutante: mw1152,mw2090,terbium apt-get remove python-mysqldb (T84075)
  • 17:44 legoktm: manually fixed T129881 rename
  • 17:20 paravoid: cr1/2-eqiad: deactivating static routes for ns0/ns1 (ipv4/ipv6) pointing to radon
  • 17:18 bblack: rebooting radon again
  • 15:31 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.18/extensions/OpenStackManager/OpenStackManager.php: SWAT: Wikitech: Remove address, domain, proxy special pages gerrit:279569 (duration: 00m 31s)
  • 13:32 bblack: rebooting radon (unresponsive console)
  • 08:44 paravoid: force-rebooting ms-be2008 to fix hundreds of unkillable mkfs.xfs stuck due to a hosed disk
  • 08:21 paravoid: powercycling ms-be2016, down for ~1d, unresponsive on console
  • 02:35 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Mar 28 02:35:00 UTC 2016 (duration 8m 26s)
  • 02:26 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.18) (duration: 11m 32s)

2016-03-27

  • 22:21 logmsgbot: ori@tin Synchronized wmf-config/StartProfiler.php: I1b5c620b85: Better request profiling via XWD header (duration: 00m 33s)
  • 20:11 elukey: restarted hhvm on mw1123
  • 20:04 elukey: Increased nf_conntrack_max to ~528k for the kafka brokers (https://gerrit.wikimedia.org/r/279776)
  • 02:57 jzerebecki: disabling 2fa for Dereckson T130892 [labswiki]> delete from oathauth_users where id=402;
  • 02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Mar 27 02:31:03 UTC 2016 (duration 8m 35s)
  • 02:22 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.18) (duration: 10m 06s)
  • 01:51 jzerebecki: disabling 2fa for Hym411 T130994 [labswiki]> delete from oathauth_users where id=1363;

2016-03-26

  • 21:11 csteipp_ooo: removed 2FA from wikitech accounts that looked to be affected by T130892
  • 16:21 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.18/extensions/SemanticForms: https://gerrit.wikimedia.org/r/#/c/279701/ (duration: 00m 32s)
  • 13:46 urandom: Cassandra on restbase2004.codfw.wmnet shut down, hardware failure; Down for the weekend : T130990
  • 13:26 urandom: Stopping Cassandra on restbase2004.codfw.wmnet
  • 02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Mar 26 02:31:13 UTC 2016 (duration 8m 34s)
  • 02:22 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.18) (duration: 10m 06s)
  • 00:11 mutante|away: scb[12]00[12] - delete changeprop main.log per request

2016-03-25

  • 21:52 ori: Updated Grafana to latest nightly
  • 21:19 logmsgbot: ori@tin Synchronized wmf-config/logging.php: Ibc96f9d3bd: Integrate X-Wikimedia-Debug with Logstash; I28b137f5e035: Logging: convert to short array syntax (duration: 00m 36s)
  • 20:46 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.18/includes/filerepo/LocalRepo.php: Idaa1237638: Request-local caching of image_redirect and I545ce6b160b: Lower pcTTL in checkRedirect() to 30 (duration: 00m 37s)
  • 19:32 tgr: running extensions/CentralAuth/maintenance/checkLocal{Names,User}.php on jawiktionary and dewikiquote
  • 19:08 mutante: scb1001 - Unit changeprop.service entered failed state - it's a new deployment though, acked earlier by mobrovac already
  • 18:27 mobrovac: restbase started mobile-sections dump of enwiki on restbase1009 for articles edited before 2016-03-23 as per T130698
  • 17:19 akosiaris: rolling restart of restbase nodes to apply https://gerrit.wikimedia.org/r/#/c/279597/
  • 16:55 logmsgbot: ori@tin Synchronized wmf-config/logging.php: Ia2cd5daaf3: Update logging config for request IDs (duration: 00m 31s)
  • 16:50 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.18/includes: Iaf90c20c330e: Provide a unique request identifier (duration: 01m 22s)
  • 15:32 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: Remove T44894 FIXME note (duration: 00m 27s)
  • 15:03 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: Remove sampling from ApiAction kafka channel (duration: 00m 33s)
  • 14:58 logmsgbot: demon@tin Synchronized wmf-config/db-codfw.php: docfix (duration: 00m 34s)
  • 14:43 godog: chown _graphite:_graphite frontend.navtiming.loadEventEnd
  • 14:38 godog: depool restbase and drain cassandra from restbase1007
  • 14:20 bblack: all authdns servers running ferm rules now
  • 13:15 bblack: enabling ferm on authdns servers (one at a time, while watching stuff...)
  • 08:38 akosiaris: depool maps-test2002.codfw.wmnet temporarily
  • 03:45 csteipp: redeployed Scribunto patch for T110143
  • 02:38 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Mar 25 02:38:53 UTC 2016 (duration 8m 55s)
  • 02:30 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.18) (duration: 09m 54s)
  • 00:24 logmsgbot: catrope@tin Finished scap: Pushing out i18n changes from SWAT (duration: 26m 15s)

2016-03-24

  • 23:58 logmsgbot: catrope@tin Started scap: Pushing out i18n changes from SWAT
  • 23:52 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.18/extensions/ContentTranslation: Remove registration of missing i18n keys (duration: 00m 28s)
  • 23:51 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.18/extensions/Flow: Remove registration of missing i18n keys (duration: 00m 46s)
  • 23:51 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.18/extensions/Graph: Remove registration of missing i18n keys (duration: 00m 28s)
  • 23:42 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.18/extensions/MobileApp/: Roll out RESTbase usage to mobile app at 2% (duration: 00m 27s)
  • 23:38 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.18/extensions/MobileFrontend/: SWAT (duration: 00m 30s)
  • 23:31 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Enable Echo footer notice on enwiki (duration: 00m 28s)
  • 23:22 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Switch back to legacy language overlay (duration: 00m 27s)
  • 23:16 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.18/extensions/Echo/: Styling fixes for footer link (duration: 00m 31s)
  • 23:15 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.18/extensions/OATHAuth/: SWAT (duration: 00m 28s)
  • 23:13 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.18/extensions/VisualEditor: SWAT (duration: 00m 28s)
  • 23:12 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: extension.json for ParserFunctions (duration: 00m 27s)
  • 23:11 logmsgbot: catrope@tin Synchronized wmf-config/extension-list: extension.json for ParserFunctions (duration: 00m 27s)
  • 23:10 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: massmessage group changes for zhwiki (duration: 00m 27s)
  • 23:07 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: extension.json changes (duration: 00m 32s)
  • 23:06 logmsgbot: catrope@tin Synchronized wmf-config/extension-list: extension.json changes (duration: 00m 27s)
  • 23:03 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Enable subpages in Template namespace on ruwikisource (duration: 00m 31s)
  • 22:24 jzerebecki: deactivate 2fa for my account T130892 mysql:wikiadmin@silver [labswiki]> delete from oathauth_users where id=922;
  • 22:10 csteipp: deployed followup patch for T110143
  • 22:09 logmsgbot: csteipp@tin Synchronized php-1.27.0-wmf.18/extensions/SyntaxHighlight_GeSHi: (no message) (duration: 00m 29s)
  • 21:47 bblack: re-weighted eqiad upload frontend caches (even)
  • 21:34 bblack: restarting cache_misc frontends globally for https://gerrit.wikimedia.org/r/#/c/279460/
  • 21:08 logmsgbot: gehel@tin Synchronized wmf-config/LabsServices.php: T130219 Configure a single elasticsearch server for CirrusSearch beta cluster (duration: 00m 38s)
  • 21:05 gehel: deploying mediawiki-config: single elasticsearch server for CirrusSearch beta cluster (T130219)
  • 19:59 gehel: deploying latest versio of WDQS
  • 19:06 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.27.0-wmf.18
  • 18:29 elukey: camus and puppet re-enabled on analytics1027
  • 18:18 bblack: starting decom/re-role of cp3003-6,cp3011-14 (already depooled)
  • 17:57 elukey: enabled Hadoop Master Node automatic failover on analytics1001/1002 (this time without fireworks).
  • 16:47 urandom: Reactivating restbase2004.codfw.wmnet : T130254]
  • 16:43 logmsgbot: filippo@tin Synchronized wmf-config/filebackend-production.php: move codfw swift back to async replication T129089 (duration: 00m 28s)
  • 16:41 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-labs.php: T130219 CirrusSearch labs configured to use HTTPS connection pool (duration: 00m 30s)
  • 16:31 godog: force puppet run on upload caches in eqiad for https://gerrit.wikimedia.org/r/#/c/279358/
  • 16:25 bblack: depooling cp3003-6,cp3011-14 from esams text varnish-be over the next ~70 mins - prep for last steps of T125485
  • 16:22 logmsgbot: demon@tin Synchronized README: no-op, doing co-master sync (duration: 00m 38s)
  • 16:17 godog: force puppet run on upload caches in codfw/eqiad for https://gerrit.wikimedia.org/r/#/c/279357/
  • 16:15 elukey: disabled camus and puppet on analytics1027
  • 16:10 godog: force puppet run on upload caches in eqiad for https://gerrit.wikimedia.org/r/#/c/279356/
  • 15:46 bblack: starting depool/downtime/re-role process to decom cp3019-22
  • 15:43 bblack: restarted pybal on lvs300x (persistent failure to load new etcd data...)
  • 15:38 moritzm: upgrading xenon, cerium, praseodymium to Linux 4.4
  • 15:09 bblack: downtime/re-role of cp3007-10 starting - T125485
  • 14:21 bblack: restarted ganglia-monitor-aggregator instances on hooft.esams
  • 14:17 ema: puppet re-enabled on all cache nodes T124279
  • 13:29 ema: testing vcl v4 merge, puppet disabled on all cache nodes T124279
  • 13:22 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1027; pool db1078 with low weight (duration: 00m 42s)
  • 13:17 elukey: puppet re-enabled on analytics1027
  • 11:20 hashar: Restarted Jenkins, jobs got blocked notifying to IRC
  • 11:10 gehel: running puppet-merge manually on strontium to correct unmerged changes.
  • 10:18 gehel: activating SSL certificate check on elasticsearch - T130366
  • 10:07 moritzm: installing exim upgrades on non-mail hubs in production
  • 09:56 elukey: puppet disabled on analitycs1027 to disable Camus
  • 09:50 elukey: puppet disabled on analytics1001/1002 (HDFS namenodes) as pre-step before enabling automatic failover
  • 09:30 jynus: stopping slave on db2018
  • 09:18 jynus: stopping db1027 for upgrade and cloning to db1078
  • 08:57 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1044; Pool db1077; Depool db1027 (duration: 00m 31s)
  • 08:56 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Add db1077 and db1078 (duration: 00m 28s)
  • 05:05 Krinkle: mwscript deleteEqualMessages.php --wiki zhwikiquote (T45917 , P1988)
  • 05:05 Krinkle: mwscript deleteEqualMessages.php --wiki zhwiktionary
  • 05:04 Krinkle: mwscript deleteEqualMessages.php --wiki zhwiki
  • 05:03 Krinkle: mwscript deleteEqualMessages.php --wiki zhwikibooks
  • 04:58 logmsgbot: krinkle@tin Synchronized wmf-config/InitialiseSettings.php: 3f2943e (duration: 00m 26s)
  • 04:55 logmsgbot: krinkle@tin Synchronized wmf-config/CommonSettings.php: 3f2943e (duration: 00m 41s)
  • 04:43 logmsgbot: krinkle@tin Synchronized wmf-config/CommonSettings-labs.php: (no message) (duration: 00m 35s)
  • 03:02 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Mar 24 03:02:09 UTC 2016 (duration 8m 52s)
  • 02:53 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.18) (duration: 09m 59s)
  • 02:27 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.17) (duration: 11m 07s)
  • 00:47 twentyafterfour: phab upgrade/maintenance finished
  • 00:47 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.18/includes/media/DjVu.php: Idbd11637a8: Request-local caching of DjVu dimensions (duration: 00m 39s)
  • 00:13 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.18/extensions/Echo/modules/ooui/mw.echo.ui.FooterNoticeWidget.js: touch (duration: 00m 26s)
  • 00:06 logmsgbot: catrope@tin Finished scap: Oops, Echo had i18n changes (duration: 26m 50s)
  • 00:02 moritzm: rebooting iridium (phabricator host) for kernel upgrade

2016-03-23

  • 23:45 yuvipanda: restart nova-compute on labvirt1008
  • 23:39 logmsgbot: catrope@tin Started scap: Oops, Echo had i18n changes
  • 23:33 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Plumbing for Echo footer link settings (duration: 00m 26s)
  • 23:33 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Enable Echo footer link on testwiki (duration: 00m 26s)
  • 23:32 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.18/extensions/UploadWizard/: SWAT (duration: 00m 27s)
  • 23:31 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.18/extensions/Echo/: sWAT (duration: 00m 32s)
  • 23:29 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.18/includes/specials/SpecialEditWatchlist.php: Try to fix watchlist fatals (duration: 00m 25s)
  • 23:28 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.17/extensions/VisualEditor/lib/ve/src/ce/ve.ce.Surface.js: Move table paste logic out of external paste block (duration: 00m 26s)
  • 23:25 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Use extension registration for Cite (duration: 00m 28s)
  • 23:22 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Namespace config for newiktionary (duration: 00m 43s)
  • 23:21 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Import sources for newikibooks (duration: 00m 47s)
  • 23:03 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Add channel for slow diff logs (duration: 00m 28s)
  • 22:17 andrewbogott: rebooting labvirt1010
  • 22:17 logmsgbot: csteipp@tin Synchronized php-1.27.0-wmf.18/extensions/OATHAuth: Deploy HTMLForm update for OATH (duration: 00m 30s)
  • 22:13 andrewbogott: rebooting labvirt1007 <- correction
  • 22:12 andrewbogott: rebooting labvirt1006
  • 22:12 csteipp: ran update_scratch_token_format.php on silver
  • 22:03 logmsgbot: csteipp@tin Synchronized php-1.27.0-wmf.18/extensions/OATHAuth: (no message) (duration: 00m 34s)
  • 21:59 andrewbogott: rebooting labvirt1006
  • 21:43 andrewbogott: rebooting labvirt1005
  • 21:41 bblack: depooling cp3043 - T125485
  • 21:38 bblack: depooling cp3042 - T125485
  • 21:20 chasemp: reboot labvirt1004
  • 20:53 chasemp: reboot labvirt1003
  • 20:40 mdholloway: found an issue with the mobileapps deployment, reverting to 85856f7
  • 20:29 mdholloway: starting mobileapps deployment
  • 20:18 chasemp: reboot labvirt1002
  • 20:14 subbu: finished deploying parsoid version 5538d868
  • 20:10 subbu: synced code. restarted parsoid on wtp1002 (~4 minutes back) as a canary
  • 20:03 subbu: starting parsoid deploy
  • 19:56 andrewbogott: rebooting labvirt1001
  • 19:38 chasemp: rebooting labvirt1011
  • 19:36 andrewbogott: rebooting labvirt1009
  • 19:27 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.18/includes/Revision.php: I77575d6d0ea: Request-local caching of revision text (duration: 00m 28s)
  • 19:25 andrewbogott: rebooting labvirt1008
  • 19:05 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.18
  • 19:01 legoktm: restarting zuul
  • 18:12 urandom: Removing compaction throughput throttling on restbase2004.codfw.wmnet : T130254
  • 18:05 urandom: Incresing compactionthroughput to 200MB/s on restbase2004.codfw.wmnet : T130254
  • 17:54 urandom: Removing old heap dumps on restbase2004.codfw.wmnet : T130254
  • 17:48 urandom: Increasing compactionthroughput to 120MB/s on restbase2004.codfw.wmnet : T130254
  • 17:36 urandom: Increasing compactionthroughput to 100MB/s on restbase2004.codfw.wmnet : T130254
  • 17:24 urandom: Starting scrub of parsoid_html on restbase2004.codfw.wmnet : T130254
  • 17:21 urandom: Disabling gossip and binary transport on restbase2004.codfw.wmnet : T130254
  • 17:18 urandom: Starting Cassandra on restbase2004.codfw.wmnet : T130254
  • 17:16 urandom: Disabling puppet on restbase2004.codfw.wmnet to override compactor concurrency : T130254
  • 17:14 urandom: Cancelling offline scrubs on restbase2004.codfw.wmnet : T130254
  • 17:02 logmsgbot: bd808@tin Synchronized wmf-config/InitialiseSettings.php: touched (duration: 00m 25s)
  • 16:57 logmsgbot: bd808@tin Synchronized wmf-config/InitialiseSettings.php: Logging: add ApiAction kafka logging (34f236c) (T108618) (duration: 00m 28s)
  • 16:56 logmsgbot: bd808@tin Synchronized wmf-config/event-schemas: Logging: add ApiAction kafka logging (34f236c) (duration: 00m 31s)
  • 16:51 logmsgbot: bd808@tin Synchronized php-1.27.0-wmf.17/includes/api/ApiMain.php: Rename ApiRequest to ApiAction (4dc12de) (duration: 00m 47s)
  • 15:49 elukey: updated puppet-compiler to 0.1.2 version (added submodule support)
  • 15:18 urandom: CORRECTION: Starting cleanups on restbase10{08,10,11}-{a,b}.eqiad.wmnet : T125842
  • 15:17 urandom: Starting cleanups on restbase10{08,12,13}-{a,b}.eqiad.wmnet : T125842
  • 14:33 godog: rolling-restart restbase after https://gerrit.wikimedia.org/r/279112
  • 12:29 godog: pool restbase1012 / restbase1013
  • 12:27 godog: halt restbase1003 / restbase1004
  • 12:17 moritzm: installing various security updates on mediawiki eqiad servers (along with HHVM restarts): graphite2, libldap, pixman, sqlite, pygments, gnutls26 (already running fine on canaries since yesterday)
  • 11:54 godog: swift eqiad-prod ms-be1020 / ms-be1021 to weight 3500
  • 10:59 jynus: stopping and restarting db1015 for upgrade and clone to db1077
  • 10:44 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1015, increase weight of db1044 (duration: 00m 25s)
  • 10:38 godog: depool restbase1003 / restbase1004 prior to deprovisioning the hardware
  • 10:29 moritzm: installing various security updates on mediawiki codfw servers (along with HHVM restarts): graphite2, libldap, pixman, sqlite, pygments, gnutls26 (already running fine on canaries since yesterday)
  • 09:21 jynus: start mysql on es2019 at es2018-bin.000044:287914983
  • 08:38 jynus: starting mysql at es2019
  • 08:37 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Add db2008 (x1) depooled, depool es2019 (duration: 00m 26s)
  • 08:30 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1044, pool db1075 (duration: 00m 25s)
  • 08:27 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Add db1075 (duration: 00m 40s)
  • 07:59 jynus: powercycling es2019 - it was down
  • 07:19 _joe_: progressively activating cross-dc replica and encryption between the jobqueue redises
  • 03:34 mutante: tin - re-arm keyholder
  • 03:26 mutante: tin - restart keyholder - re: < twentyafterfour> 1 mismatch. I guess that keyholder-proxy needs to be restarted on tin
  • 03:12 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Mar 23 03:12:07 UTC 2016 (duration 9m 20s)
  • 03:02 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.18) (duration: 17m 31s)
  • 02:28 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.17) (duration: 11m 18s)

2016-03-22

  • 23:47 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.18/extensions/WikimediaEvents/: https://gerrit.wikimedia.org/r/#/c/279066/ (duration: 00m 27s)
  • 23:46 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.17/extensions/WikimediaEvents/: https://gerrit.wikimedia.org/r/#/c/279066/ (duration: 00m 28s)
  • 23:40 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: SWAT x 3 (duration: 00m 26s)
  • 23:35 logmsgbot: maxsem@tin Synchronized portals: (no message) (duration: 00m 25s)
  • 23:35 logmsgbot: maxsem@tin Synchronized portals/prod/wikipedia.org/assets: (no message) (duration: 00m 26s)
  • 23:26 MaxSem: ran echo 'https://test.wikidata.org/static/favicon/testwikidata.ico' | mwscript purgeList.php
  • 23:23 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/278334/ (duration: 00m 27s)
  • 23:23 logmsgbot: maxsem@tin Synchronized static/images/project-logos: https://gerrit.wikimedia.org/r/#/c/278334/ (duration: 00m 26s)
  • 23:20 logmsgbot: maxsem@tin Synchronized static/favicon/testwikidata.ico: https://gerrit.wikimedia.org/r/#/c/278321/ (duration: 00m 31s)
  • 23:15 mutante: mira - restarting and re-arming keyholder
  • 23:14 mutante: tin - restarting and re-arming keyholder
  • 23:04 mutante: re-arm keyholder on tin for new phab key
  • 23:02 mutante: mira re-arm keyholder for new phab key
  • 22:56 logmsgbot: aaron@tin Synchronized wmf-config: Lower "max lag" and $wgAPIMaxLagThreshold to 8/6 (duration: 00m 29s)
  • 22:52 logmsgbot: aaron@tin Synchronized wmf-config/jobqueue-eqiad.php: Bump timeout to 300ms (duration: 00m 29s)
  • 22:50 logmsgbot: aaron@tin Synchronized wmf-config/filebackend-production.php: comment tweak (duration: 00m 35s)
  • 22:50 mutante: replacing phab deploy key with a new one
  • 21:41 hoo: Ran sync-common on mw1139 (which missed two deploys)
  • 21:28 bd808: Restarted logstash process on logstash1002; dead from OOM since 2016-03-18T11:47:12
  • 21:27 logmsgbot: hoo@tin Synchronized php-1.27.0-wmf.18/extensions/Wikidata: Update Wikibase: Fix add qualifier link not getting disabled (duration: 03m 43s)
  • 21:24 ejegg: update payments from 79f5c9389edd089ae5951a7d172e74e68946a93c to 5a1996bc21fe694b99556bc1b501e484075dabe2
  • 21:23 logmsgbot: hoo@tin Synchronized php-1.27.0-wmf.17/extensions/Wikidata: Update Wikibase: Fix add qualifier link not getting disabled (duration: 07m 30s)
  • 21:01 mutante: mira - arm keyholder
  • 20:59 mutante: tin - arm keyholder with deployment keys
  • 20:49 urandom: Stopping restbase2004.codfw.wmnet for offline sstablescrub : T130254
  • 20:46 urandom: decommissioning restbase1004-a.eqiad.wmnet : T125842
  • 20:45 urandom: Rolling restart of RESTBase Cassandra cluster complete : T130393, T128787
  • 20:25 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.27.0-wmf.18
  • 20:22 gehel: deployed latest version of WDQS
  • 20:18 logmsgbot: thcipriani@tin Finished scap: testwiki to php-1.27.0-wmf.18 and rebuild l10n cache (duration: 46m 09s)
  • 19:53 urandom: Rolling restart of RESTBase Cassandra cluster : T130393, T128787
  • 19:32 logmsgbot: thcipriani@tin Started scap: testwiki to php-1.27.0-wmf.18 and rebuild l10n cache
  • 19:00 mutante: mw1125 - powercycle
  • 18:41 urandom: Aborting restart of Cassandra on maps cluster : T130393, T128787
  • 18:39 urandom: Rolling restart of Cassandra on maps cluster : T130393, T128787
  • 18:39 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Increase weight of db1024, db1074 and db1076 (duration: 01m 34s)
  • 17:33 mdholloway: mobileapps deployment complete, deployed 85856f7
  • 17:33 thcipriani: starting wmf.18 branching
  • 17:33 ejegg: rolled payments back to 79f5c9389edd089ae5951a7d172e74e68946a93c
  • 17:27 ejegg: updated payments from from 79f5c9389edd089ae5951a7d172e74e68946a93c to 62365063548836618b93b12a3bcbe65781a62a94
  • 17:15 ejegg: rolled payments back to 79f5c9389edd089ae5951a7d172e74e68946a93c
  • 17:11 ejegg: updated payments from from 79f5c9389edd089ae5951a7d172e74e68946a93c to 62365063548836618b93b12a3bcbe65781a62a94
  • 17:05 mdholloway: starting mobileapps deployment
  • 17:03 urandom: Rolling restart of AQS Cassandra cluster complete : : T130393, T128787
  • 16:54 urandom: Performing rolling restart of AQS Cassandra cluster : T130393, T128787
  • 16:49 jynus: stopping db1044 mysql to clone to db1075
  • 16:37 elukey: re-added rdb1001 (Job Queue master) to the Jobrunners' config. Forcing puppet agent and restarting jobchron on all jobrunners/videoscalers in eqiad.
  • 16:33 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1024; pool db1074 and db1076; depool db1044 (duration: 00m 28s)
  • 16:32 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Add db1074 and db1076 eqiad databases (duration: 00m 31s)
  • 16:21 logmsgbot: elukey@tin Synchronized wmf-config/jobqueue-eqiad.php: REVERT - Remove rdb1001 from the Redis Job Queues for maintenance (duration: 00m 25s)
  • 16:21 urandom: Restarting Cassandra on restbase1007.eqiad.wmnet (canary) : T130393, T128787
  • 15:56 dcausse: elasticsearch: creating wikimania2017wiki indices in codfw
  • 15:38 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable signature button at NS:102 for frwiki gerrit:272479 (duration: 00m 30s)
  • 15:34 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Configuring wgMetaNamespace for an.wiktionary gerrit:278876 (duration: 00m 29s)
  • 15:28 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.17/extensions/ContentTranslation/modules/tools/ext.cx.tools.mt.js: SWAT: Fix JS error in MT tool: MTControlCard.providers undefined gerrit:278843 (duration: 00m 25s)
  • 14:56 bblack: deploying VCL changes for do_stream on all pass traffic
  • 14:37 elukey: puppet disabled on rdb1001/rdb1002 (redis master slave) as part of the rdb1001 re-image. rdb1002 set with SLAVEOF NO ONE as precaution.
  • 14:09 elukey: removed rdb1001 from the JobRunner config (hieradata/eqiad/mediawiki/jobrunner). Forcing also a puppet run and a jobchron restart on all the Job Runners and VideoScalers in eqiad.
  • 14:03 logmsgbot: elukey@tin Synchronized wmf-config/jobqueue-eqiad.php: Remove rdb1001 from the Redis Job Queues for maintenance (duration: 00m 25s)
  • 13:30 moritzm: installing various security updates on mediawiki canary servers (along with HHVM restarts): graphite2, libldap, pixman, sqlite, pygments, gnutls26
  • 12:14 godog: nodetool decommission restbase1003 T125842
  • 10:35 hoo: Updated WikibaseQualityConstraints data on wikidata (wikidatawiki.wbqc_constraints)
  • 10:20 moritzm: rolling reboot of ocg* for kernel upgrades
  • 09:59 jynus: stopping and cloning db1024 to db1074 and db1076
  • 09:49 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1024 for maintenance (duration: 01m 15s)
  • 09:46 moritzm: rolling reboot of sca* for kernel upgrades
  • 09:14 moritzm: rebooting tin for kernel upgrade
  • 08:46 moritzm: installing squid/jessie security updates
  • 07:59 moritzm: installing git security updates
  • 07:56 moritzm: restarted cassandra on restbase2004, it ran out of heap memory
  • 07:51 moritzm: restarting hhvm on mw1119, mw1121, mw1136 and mw1140, all got stuck over night
  • 06:54 _joe_: banning all pages with content-length of 25 from the caches, T130575
  • 06:04 _joe_: restarted hhvm on mw1122, memory leak

2016-03-21

  • 19:58 akosiaris: powercycle mw1142, console available but not ever prompting for the root password, stuck at username
  • 15:44 logmsgbot: hoo@tin Synchronized wmf-config/Wikibase.php: Bump $wgCacheEpoch on Wikidata after Property conversions (duration: 00m 28s)
  • 15:13 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.17/includes/upload/UploadBase.php: SWAT: UploadBase: Set mFileSize, if given, even if mTempPath is unknown gerrit:278724 (duration: 00m 30s)
  • 14:26 ottomata: altering kafka topics webrequest_text and webrequest_upload, increasing each from 12 partitions to 24 partitions
  • 14:23 jynus: restarting labsdb1001 mysql
  • 14:05 mobrovac: restbase deploy end of 26f9e90
  • 13:33 mobrovac: restbase deploy start of 26f9e90 on canary restbase1003
  • 10:52 hashar: Live hacked puppet compiler on compiler02.puppet3-diffs.eqiad.wmflabs to debug it not processing submodules. Reinstalled it from the last tag in the process
  • 10:24 jynus: Altering user_properties engine to InnoDB on db1069:3313
  • 09:26 jynus: Altering change_tag engine to InnoDB on db1069:3313
  • 09:09 elukey: restarted hhvm on mw1116
  • 02:33 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Mar 21 02:33:31 UTC 2016 (duration 8m 41s)
  • 02:24 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.17) (duration: 10m 57s)

2016-03-20

  • 02:32 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Mar 20 02:32:04 UTC 2016 (duration 8m 30s)
  • 02:23 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.17) (duration: 10m 22s)

2016-03-19

  • 22:28 jynus: powercycling oxygen, looks kernel-dead
  • 22:16 urandom: removing 22G of heap dumps from restbase2004.codfw.wmnet
  • 22:16 urandom: removing 22G of heap dumps
  • 22:07 urandom: clearing snapshots on restbase2004.codfw.wmnet
  • 15:43 logmsgbot: reedy@tin Synchronized wmf-config/throttle.php: Throttle rules for event T130447 (duration: 00m 26s)
  • 11:38 godog: restart slapd on seaborgium, oom-killed
  • 02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Mar 19 02:31:46 UTC 2016 (duration 8m 31s)
  • 02:23 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.17) (duration: 10m 07s)
  • 01:54 urandom: bootstrapping restbase1013-b.eqiad.wmnet : T125842

2016-03-18

  • 23:35 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.17/extensions/WikimediaEvents/modules/ext.wikimediaEvents.deprecate.js: (no message) (duration: 00m 35s)
  • 21:11 ostriches: cleaned up stale /srv/mediawiki/php-1.27.0-wmf.{10,11} from the apaches.
  • 21:09 logmsgbot: krinkle@tin Synchronized wmf-config/missing.php: (no message) (duration: 00m 25s)
  • 20:53 ottomata: reenabling puppet on krypton
  • 19:53 ottomata: temporarily disabling puppet on krypton
  • 19:21 ori: rebooting bohrium
  • 19:20 ori: upgraded bohrium VM: vcpus 2 => 8, ram 4 => 8g
  • 19:06 logmsgbot: ori@tin Synchronized wmf-config/logging.php: Iabca8858e: Allow finer-grained control over debug logging via XWD (duration: 00m 32s)
  • 18:56 logmsgbot: demon@tin Synchronized .arclint: no op really, co master sync (duration: 00m 39s)
  • 18:08 gehel: restarting elasticsearch server elastic1031.eqiad.wmnet
  • 17:59 mutante: netmon1001: failed torrus service - recovery steps as outlined on wikitech Torrus
  • 17:55 ori: on bohrium: /etc/apache2/sites-enabled/.links2 ; was causing puppet to refresh apache2 on each run
  • 17:30 gehel: restarting elasticsearch server elastic1030.eqiad.wmnet
  • 17:05 gehel: restarting elasticsearch server elastic1029.eqiad.wmnet
  • 16:53 jynus: starting enwiki import to labs from dbstore1002 (expect lag and consistency problems during the hot import)
  • 16:37 moritzm: restarted hhvm on mw1205
  • 16:30 moritzm: bumped connection tracking table size on mw1161-mw1169 to 524288 to cope with currently elevated connections on those (T130364)
  • 16:19 godog: reboot ms-be2010 to pick up new disk ordering
  • 15:23 logmsgbot: elukey@tin Synchronized wmf-config/jobqueue-eqiad.php: REVERT - Re-enabled persistence between Job Queues and Job Runners. (duration: 00m 19s)
  • 15:03 logmsgbot: elukey@tin Synchronized wmf-config/jobqueue-eqiad.php: Re-enabled persistence between Job Queues and Job Runners. (duration: 00m 30s)
  • 15:02 godog: bootstrap restbase1013-a
  • 14:36 gehel: restarting elasticsearch server elastic1028.eqiad.wmnet
  • 14:02 elukey: restarted eventlog1001.eqiad.wmnet and eventlog2001.codfw.wmnet for kernel upgrade
  • 13:43 gehel: restarting elasticsearch server elastic1027.eqiad.wmnet
  • 13:24 gehel: restarting pybal on lvs2003.codfw.wmnet
  • 13:22 gehel: enabling all nodes for service search.svc.codfw.wmnet:9243 (elastic-https) on codfw
  • 13:22 gehel: restarting pybal on lvs2006.codfw.wmnet
  • 13:06 gehel: restarting elasticsearch server elastic1026.eqiad.wmnet
  • 12:43 gehel: restarting elasticsearch server elastic1025.eqiad.wmnet
  • 12:35 godog: finished ms-fe1* rolling reboot
  • 12:15 godog: finished ms-be1* rolling reboot
  • 12:00 elukey: Forcing puppet agent run on all the Jobrunners and videoscalers since rdb1005 is now back in service. Will also restart jobchron as well.
  • 11:58 elukey: Added rdb1005 back to the jobrunners puppet config after maintenance.
  • 11:57 gehel: restarting elasticsearch server elastic1024.eqiad.wmnet
  • 11:46 gehel: restarting pybal on lvs1003
  • 11:43 logmsgbot: elukey@tin Synchronized wmf-config/jobqueue-eqiad.php: Add rdb1005 back to the Redis Job Queues after maintenance (duration: 01m 22s)
  • 11:23 moritzm: powercycled mw1163, hung on reboot and serial console stuck
  • 11:05 moritzm: rolling reboot of mw1161 to mw1169 for kernel upgrade
  • 11:04 gehel: restarting pybal on lvs1012
  • 11:04 gehel: restarting pybal on lvs1009
  • 10:58 gehel: activating elasticsearch-ssl service on LVS / eqiad
  • 10:51 gehel: restarting pybal on lvs1006
  • 10:48 jynus: dbstore2002 just crashed
  • 10:34 godog: reboot ms-fe1003 for kernel upgrade
  • 10:33 akosiaris: gehel: restarting pybal on lvs1006
  • 10:27 gehel: activating elasticsearch HTTPS on LVS for eqiad - https://gerrit.wikimedia.org/r/#/c/277956/
  • 10:06 moritzm: rolling reboot of swift backend servers in codfw for kernel upgrade
  • 09:46 godog: rolling-reboot ms-be1* for kernel updates
  • 09:37 elukey: forcing puppet agent and restarting jobchron on all the Job Runners and VideoScalers as rdb1005 has been removed from the configs.
  • 09:32 elukey: removed rdb1005 from the Job Runners config for maintenance
  • 09:24 logmsgbot: elukey@tin Synchronized wmf-config/jobqueue-eqiad.php: Remove rdb1005 from the Redis Job Queues for maintenance (duration: 01m 07s)
  • 09:19 moritzm: rolling reboot of swift frontend servers in codfw for kernel upgrade
  • 09:08 godog: Issuing nodetool scrub -s -- local_group_wikipedia_T_parsoid_html data on restbase2004.eqiad.wmnet : T130254
  • 09:01 moritzm: rolling reboot of mw1001 to mw1016 for kernel upgrade
  • 08:22 _joe_: started cassandra on restbase2004
  • 08:11 moritzm: rearmed keyholder on mira
  • 08:06 moritzm: rebooting mira for kernel update
  • 07:42 jynus: restarting dbstore2002 to apply new mysql config
  • 06:29 gehel: restarting elasticsearch server elastic1023.eqiad.wmnet
  • 02:38 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Mar 18 02:38:00 UTC 2016 (duration 8m 40s)
  • 02:29 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.17) (duration: 09m 58s)
  • 01:31 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.17/extensions/CirrusSearch/: Emergency fix https://gerrit.wikimedia.org/r/#/c/278224/ (duration: 00m 36s)
  • 00:38 ebernhardson: reboot elastic1022.eqiad.wmnet for kernel upgrade
  • 00:09 ebernhardson: reboot elastic1021.eqiad.wmnet for kernel upgrade
  • 00:09 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/276934/ (duration: 00m 26s)
  • 00:02 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/276410/ (duration: 00m 26s)
  • 00:01 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/278206/ (duration: 00m 25s)

2016-03-17

  • 23:57 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/276917 (duration: 00m 26s)
  • 23:53 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/278021/ (duration: 00m 26s)
  • 23:46 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/278029/ (duration: 00m 28s)
  • 23:42 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.17/includes/specials/SpecialUploadStash.php: https://gerrit.wikimedia.org/r/#/c/278190/ (duration: 00m 27s)
  • 23:39 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/278198/ (duration: 00m 37s)
  • 23:25 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/278032/ (duration: 00m 32s)
  • 23:21 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/278054/ (duration: 00m 47s)
  • 22:52 logmsgbot: bd808@tin Synchronized php-1.27.0-wmf.17/includes/api/ApiMain.php: Cast API timeSpentBackend to an int (duration: 00m 25s)
  • 22:47 gehel: resetting cluster.routing.allocation.disk.watermark.high to 90% on eqiad elasticsearch cluster - shards have moved round, cluster is mostly balanced
  • 22:42 logmsgbot: bd808@tin Synchronized wmf-config/InitialiseSettings.php: Disable ApiRequest properly (T108618) (duration: 00m 27s)
  • 22:37 gehel: restarting elasticsearch server elastic1020.eqiad.wmnet
  • 22:17 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.17/includes/specials/SpecialUploadStash.php: Revert: Debug code for T130204 (it worked!) (duration: 00m 25s)
  • 22:14 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.17/includes/specials/SpecialUploadStash.php: Debug code for T130204 (duration: 00m 28s)
  • 22:12 logmsgbot: bd808@tin Synchronized wmf-config/InitialiseSettings.php: touch (duration: 00m 28s)
  • 22:10 logmsgbot: bd808@tin Synchronized wmf-config/InitialiseSettings.php: Disable ApiRequest kafka logging (T108618) (duration: 00m 31s)
  • 22:05 logmsgbot: bd808@tin Synchronized wmf-config/InitialiseSettings.php: Add ApiRequest kafka logging (T108618) (duration: 00m 34s)
  • 22:04 logmsgbot: bd808@tin Synchronized wmf-config/event-schemas: Add ApiRequest kafka logging (T108618) (duration: 00m 38s)
  • 21:47 bd808: deleted /var/lib/l10nupdate/caches/cache-1.27.0-wmf.1[345] on tin. Freed ~4G of disk
  • 21:46 gehel: reducing cluster.routing.allocation.disk.watermark.high to 70% on eqiad elasticsearch cluster
  • 21:07 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: all to 1.27.0-wmf.17
  • 20:51 logmsgbot: twentyafterfour@tin Finished scap: sync php-1.27.0-wmf.17 (duration: 26m 57s)
  • 20:40 urandom: Issuing `nodetool scrub -s -- local_group_wikipedia_T_parsoid_html data` on restbase2004.eqiad.wmnet : T130254
  • 20:24 logmsgbot: twentyafterfour@tin Started scap: sync php-1.27.0-wmf.17
  • 20:14 urandom: Starting Cassandra on restbase2004.codfw.wmnet (OOM. again.)
  • 20:13 urandom: Restarting Cassandra on restbase1007-a.eqiad.wmnet (compaction seems stalled)
  • 20:00 gehel: restarting elasticsearch server elastic1019.eqiad.wmnet
  • 19:41 gehel: restarting elasticsearch server elastic1018.eqiad.wmnet
  • 18:37 cscott: updated OCG to version c1a8232594fe846bd2374efd8f7c20d7e97ac449
  • 18:28 moritzm: rebooting labnet1001 for kernel update
  • 18:23 moritzm: rebooting labcontrol1002 for kernel update
  • 18:03 moritzm: rebooting silver for kernel update
  • 18:00 mobrovac: restbase restarted cassandra on restbase2004, OOM
  • 17:57 gehel: restarting elasticsearch server elastic1017.eqiad.wmnet
  • 17:10 urandom: rolling restart of restbase production complete
  • 16:51 urandom: rolling restart of restbase production to apply https://gerrit.wikimedia.org/r/#/c/277112/ and https://gerrit.wikimedia.org/r/#/c/277836/
  • 16:45 urandom: performing restart of restbase1003.eqiad.wmnet (canary) to apply https://gerrit.wikimedia.org/r/#/c/277112/ and https://gerrit.wikimedia.org/r/#/c/277836/
  • 16:41 urandom: rolling restart of restbase staging complete
  • 16:41 gehel: restarting elasticsearch server elastic1016.eqiad.wmnet
  • 16:35 urandom: performing rolling restart of restbase staging to apply https://gerrit.wikimedia.org/r/#/c/277112/ and https://gerrit.wikimedia.org/r/#/c/277836/
  • 16:27 urandom: restarting restbase on xenon.eqiad.wmnet (canary), to apply https://gerrit.wikimedia.org/r/#/c/277112/ and https://gerrit.wikimedia.org/r/#/c/277836/
  • 16:27 akosiaris: do a for i in mathoid citoid graphoid cxserver; do sudo confctl --tags dc=eqiad,cluster=sca,service=$i --action delete sca1002.eqiad.wmnet ; done. Same for sca1001.eqiad.wmnet
  • 16:18 urandom: bootstrapping restbase1012-b.eqiad.wmnet : T125842
  • 16:13 _joe_: load testing done
  • 15:58 _joe_: performing a load test on mw1239
  • 15:44 gehel: restarting elasticsearch server elastic1015.eqiad.wmnet
  • 15:32 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Use wmfLocalServices for wgUploadThumbnailRenderHttpCustomDomain PART II gerrit:277786 (duration: 00m 25s)
  • 15:32 logmsgbot: thcipriani@tin Synchronized wmf-config/ProductionServices.php: SWAT: Use wmfLocalServices for wgUploadThumbnailRenderHttpCustomDomain PART I gerrit:277786 (duration: 00m 25s)
  • 15:31 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Use wmfLocalServices for wgUploadThumbnailRenderHttpCustomDomain PART II gerrit:277786 (duration: 00m 25s)
  • 15:30 logmsgbot: thcipriani@tin Synchronized wmf-config/ProductionServices.php: SWAT: Use wmfLocalServices for wgUploadThumbnailRenderHttpCustomDomain PART I gerrit:277786 (duration: 00m 26s)
  • 15:22 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add Portal namespace to ne.wikipedia gerrit:278009 (duration: 00m 28s)
  • 15:15 logmsgbot: thcipriani@tin Synchronized portals: SWAT: Bumping portals to master gerrit:277964 (duration: 00m 29s)
  • 15:14 logmsgbot: thcipriani@tin Synchronized portals/prod/wikipedia.org/assets: SWAT: Bumping portals to master gerrit:277964 (duration: 00m 28s)
  • 15:09 logmsgbot: thcipriani@tin Synchronized wmf-config/CirrusSearch-common.php: SWAT: Enable ICU Folding on greek wikipedia PART II gerrit:277477 (duration: 00m 31s)
  • 15:08 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ICU Folding on greek wikipedia PART I gerrit:277477 (duration: 00m 30s)
  • 15:07 gehel: restarting elasticsearch server elastic1014.eqiad.wmnet
  • 14:27 gehel: restarting elasticsearch server elastic1013.eqiad.wmnet
  • 14:09 moritzm: rolling reboot of parsoid servers in eqiad for kernel upgrade
  • 13:42 mobrovac: restbase resrting cassandra on restbase2004 - OOM again
  • 13:36 gehel: restarting elasticsearch server elastic1012.eqiad.wmnet
  • 13:02 gehel: restarting elasticsearch server elastic1011.eqiad.wmnet
  • 13:00 moritzm: rolling reboot of parsoid servers in codfw for kernel upgrade
  • 12:38 _joe_: switching all services back to eqiad
  • 12:25 gehel: restarting elasticsearch server elastic1010.eqiad.wmnet
  • 12:19 akosiaris: restart pybal on lvs2001
  • 12:16 akosiaris: restart pybal on lvs2002
  • 12:05 _joe_: switching cxserver to codfw
  • 11:44 moritzm: rebooting acamar for kernel upgrade
  • 11:41 gehel: restarting elasticsearch server elastic1009.eqiad.wmnet
  • 11:33 _joe_: switching citoid to use codfw as well
  • 11:31 moritzm: rebooting install2001 for kernel upgrade
  • 11:11 logmsgbot: oblivian@tin Synchronized wmf-config/ProductionServices.php: switching mediawiki to use restbase in eqiad again (duration: 00m 32s)
  • 11:04 mobrovac: restbase restarted cassandra-a on restbase1007
  • 11:02 gehel: restarting elasticsearch server elastic1008.eqiad.wmnet
  • 11:00 mobrovac: restbase restarted cassandra on restbase2004 - OOM
  • 10:45 moritzm: rebooting francium for kernel upgrade
  • 10:28 logmsgbot: oblivian@tin Synchronized wmf-config/ProductionServices.php: switching mediawiki to use restbase in codfw (duration: 00m 32s)
  • 10:20 moritzm: rebooting osmium for kernel upgrade
  • 10:14 _joe_: external traffic is now flowing through restbase in codfw
  • 10:08 _joe_: running puppet across eqiad varnishes to switch traffic of restbase eqiad => codfw
  • 09:34 volans: Copying data from db2009 to db2008 T130098
  • 09:02 jynus: reimporting missing rows from production to labs (expect some lag during the day)
  • 08:59 gehel: restarting elasticsearch server elastic1007.eqiad.wmnet
  • 08:44 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.16/includes/jobqueue/jobs/RefreshLinksJob.php: Revert: Job queue bankruptcy: force all refreshlinks jobs to be non-recursive (duration: 00m 29s)
  • 08:44 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.17/includes/jobqueue/jobs/RefreshLinksJob.php: Revert: Job queue bankruptcy: force all refreshlinks jobs to be non-recursive (duration: 00m 39s)
  • 08:26 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.17/includes/jobqueue/jobs/RefreshLinksJob.php: Job queue bankruptcy: force all refreshlinks jobs to be non-recursive (duration: 00m 25s)
  • 08:25 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.16/includes/jobqueue/jobs/RefreshLinksJob.php: Job queue bankruptcy: force all refreshlinks jobs to be non-recursive (duration: 00m 25s)
  • 07:48 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.16/includes/jobqueue/jobs/RefreshLinksJob.php: Job queue bankruptcy: force all refreshlinks jobs to be non-recursive (duration: 00m 24s)
  • 07:06 ori: Short-circuiting RefreshLinksJob::run() to bail if the root job timestamp is older than March 1st
  • 06:46 ori: Powercycled elastic1006; unresponsive.
  • 06:11 ebernhardson: rebooting elastic1006.eqiad.wmnet for kernel update
  • 03:23 eileen: Updated CiviCRM from c99abc42188b7f47726202b4740fbaf68d1c06ab to 18431eec74de251066deb77d5b63607e06f6f135
  • 03:22 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Mar 17 03:22:55 UTC 2016 (duration 9m 33s)
  • 03:14 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/277452 (labs only change, just keeping file in sync) (duration: 00m 27s)
  • 03:13 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.17) (duration: 17m 44s)
  • 02:38 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.16) (duration: 17m 36s)
  • 00:58 ebernhardson: rebooting elastic1005.eqiad.wmnet for kernel update
  • 00:05 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Enable Flow by default in all talk namespaces on gomwiki (duration: 00m 28s)

2016-03-16

  • 23:56 ebernhardson: rebooting elastic1004.eqiad.wmnet for kernel update
  • 23:47 logmsgbot: catrope@tin Finished scap: bswiki namespace changes; MobileFrontend SWAT patches (duration: 05m 04s)
  • 23:42 logmsgbot: catrope@tin Started scap: bswiki namespace changes; MobileFrontend SWAT patches
  • 23:15 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Namespace config changes for bswiki (duration: 00m 44s)
  • 22:09 ebernhardson: rebooting elastic1003.eqiad.wmnet for linux kernel upgrade
  • 21:47 csteipp: deployed patch for T123071
  • 21:18 gehel: restarting elasticsearch server elastic1003.eqiad.wmnet
  • 20:06 mutante: mira - git pull in mw-staging
  • 19:58 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.17
  • 19:57 gehel: regenerating puppet SSL certificates for elasticsearch eqiad cluster
  • 19:54 mutante: rcs1001 - starting redis, disabling puppet (T130147)
  • 19:42 mutante: rcs1002 - starting redis-server
  • 19:42 mutante: rcs1001 - starting redis-server
  • 19:32 twentyafterfour: Deploying 1.27.0-wmf.17 to group1 wikis
  • 19:31 godog: restart pybal on lvs1005 T130143
  • 19:21 ejegg: rolled back CiviCRM to c99abc42188b7f47726202b4740fbaf68d1c06ab
  • 19:18 legoktm: created urlshortcodes table on wikishared db for UrlShortener
  • 19:16 logmsgbot: legoktm@tin Synchronized wmf-config/CommonSettings.php: Configure UrlShortener extension in read-only mode (2/2) (duration: 00m 26s)
  • 19:15 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings.php: Configure UrlShortener extension in read-only mode (1/2) (duration: 00m 28s)
  • 19:14 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.17/includes/jobqueue/Job.php: 3da38ce (duration: 00m 37s)
  • 19:12 logmsgbot: legoktm@tin Finished scap: Building l10n cache for UrlShortener - T108557 (try #2) (duration: 63m 05s)
  • 19:11 gehel: regenerating puppet SSL certificates for elasticsearch codfw cluster
  • 19:08 gehel: restarting elasticsearch server elastic1002.eqiad.wmnet
  • 19:02 volans: restarted mysql on dbstore2002 and all replicas except x1, still investigating T130128
  • 19:01 godog: restart pybal on lvs1011 T130143
  • 18:52 ejegg: updated CiviCRM from c99abc42188b7f47726202b4740fbaf68d1c06ab to 9e03bcc4d0e6e80aedefedbbf1eed608b7e9d38d
  • 18:51 ejegg: disabled thank you mail sender
  • 18:20 mutante: repooling rcs1001
  • 18:18 gehel: restarting elasticsearch server elastic1001.eqiad.wmnet
  • 18:15 mutante: rebooting rcs1001
  • 18:10 mutante: depool rcs1001
  • 18:09 logmsgbot: legoktm@tin Started scap: Building l10n cache for UrlShortener - T108557 (try #2)
  • 18:09 legoktm: deleting /var/lib/l10nupdate/caches/cache-1.27.0-wmf.12 on tin to free up some space
  • 18:05 mutante: repooling rcs1002
  • 17:59 mutante: rcs1002 - traffic graph flat in ganglia, reboot
  • 17:57 mutante: rcs1002 - the last message was about 1002
  • 17:56 mutante: rcs1002 - depool from rcstream service
  • 17:55 logmsgbot: legoktm@tin scap aborted: Building l10n cache for UrlShortener - T108557 (duration: 02m 51s)
  • 17:52 logmsgbot: legoktm@tin Started scap: Building l10n cache for UrlShortener - T108557
  • 17:43 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Enable completion suggester as default on ja, zh, pl, ar and nlwiki (duration: 00m 29s)
  • 17:28 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Enable completion suggester as default on ru, fr, pt and itwiki (duration: 00m 28s)
  • 17:10 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Enable completion suggester as default on eswiki (duration: 00m 25s)
  • 17:09 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.17/extensions/Kartographer/extension.json: (no message) (duration: 00m 25s)
  • 16:46 gehel: restarting elasticsearch server elastic2024.codfw.wmnet
  • 16:36 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Enable completion suggester as default on enwiki (duration: 00m 30s)
  • 16:32 logmsgbot: krinkle@tin Synchronized wmf-config/CommonSettings-labs.php: (no message) (duration: 00m 31s)
  • 16:11 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Enable completion suggester as default on dewiki (duration: 00m 44s)
  • 16:11 moritzm: upgraded exim packages on magnesium to the version from USN 2933, without it add_environment can't be set exim failed to start
  • 16:06 moritzm: upgraded remaining exim binary packages on krypton to 4.84-8+deb8u2 (only exim4-daemon-heavy was at the version, resulting in the new add_environment variable being unknown and exim failing to start)
  • 16:02 gehel: restarting elasticsearch server elastic2023.codfw.wmnet
  • 15:25 moritzm: rebooting graphite1001 for kernel upgrade
  • 15:16 moritzm: rebooting graphite2001 for kernel upgrade
  • 15:09 volans: restarting mysql on dbstore2002 Aria engine crashed
  • 15:08 moritzm: rebooting lithium for kernel upgrade
  • 15:07 logmsgbot: jzerebecki@tin Synchronized wmf-config/InitialiseSettings.php: Whitelist feeds on mediawiki.org 622c186b2712c923fbcc48c27f65ebf396176f3e T127176 (duration: 00m 31s)
  • 14:52 volans: Reimporting table bnwikisource.shorturls into dbstore2002 as InnoDB
  • 14:51 gehel: restarting elasticsearch server elastic2022.codfw.wmnet
  • 14:36 volans: REPAIR NO_WRITE_TO_BINLOG TABLE bnwikisource.shorturls replica failed for corrupted table
  • 14:18 papaul: labstore200[3-4}] - signing puppet certs, salt-key, initial run
  • 14:06 papaul: wasat signing puppet certs, salt-key, initial run
  • 14:04 gehel: restarting elasticsearch server elastic2021.codfw.wmnet
  • 14:03 chasemp: cleanup snapshots on labstore1001
  • 14:02 volans: Reimaging db2008 to jessie T130098
  • 13:23 gehel: restarting elasticsearch server elastic2020.codfw.wmnet
  • 13:15 elukey: puppet re-enabled on analytics1027
  • 12:39 gehel: restarting elasticsearch server elastic2019.codfw.wmnet
  • 12:36 godog: bootstrapping restbase1012-a T125842
  • 11:44 elukey: rebooting analytics1001/1002 (Yarn/HDFS master nodes) for kernel upgrade
  • 11:24 elukey: rebooting analytics105* for kernel upgrade
  • 11:02 volans: Revoking puppet key for db2008.codfw.wmnet (puppet wasn't running already) T130098
  • 10:51 elukey: rebooting analytics102[89] and analtics104* for kernel upgrades
  • 10:51 moritzm: rolling reboot of mw* servers in eqiad (except job runners to not interfere with T129517)
  • 10:21 elukey: Rebooting analytics103* for kernel upgrade
  • 09:33 elukey: puppet disabled on analytics1027 as preparation step for the Hadoop nodes reboots
  • 09:18 jynus: setting up pending cross-datacenter master-master database links
  • 09:04 moritzm: rebooting stat1002/stat1003 for kernel upgrade
  • 08:54 _joe_: restarting hhvm on mw1007, stuck in a deadlock, apparently on HPHP::Treadmill::getAgeOldestRequest
  • 08:53 moritzm: rebooting labtest*2001 for kernel upgrade
  • 08:45 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Return s2 slaves to normal weight (duration: 00m 35s)
  • 08:04 moritzm: restarted mw1129/1002 (HHVM hung)
  • 06:47 ebernhardson: rebooting elastic2018.codfw.wmnet for kernel upgrade
  • 06:10 ebernhardson: rebooting elastic2017.codfw.wmnet for kernel upgrade
  • 05:39 ebernhardson: rebooting elastic2016.codfw.wmnet for kernel upgrade
  • 04:42 ebernhardson: rebooting elastic2015.codfw.wmnet for kernel upgrade
  • 03:41 ebernhardson: rebooting elastic2014.codfw.wmnet for kernel upgrade
  • 03:19 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Mar 16 03:19:26 UTC 2016 (duration 9m 41s)
  • 03:09 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.17) (duration: 17m 36s)
  • 02:45 ebernhardson: rebooting elastic2013.codfw.wmnet for kernel upgrade
  • 02:34 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.16) (duration: 13m 44s)
  • 02:17 ebernhardson: rebooting elastic2012.codfw.wmnet for kernel upgrade
  • 01:32 ebernhardson: rebooting elastic2011.codfw.wmnet for kernel upgrade
  • 01:18 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.16/includes/jobqueue/JobQueueGroup.php: Debug (duration: 00m 26s)
  • 00:57 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.16/includes/jobqueue/JobQueueGroup.php: Debug (duration: 00m 24s)
  • 00:49 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.16/includes/jobqueue/utils/BacklinkJobUtils.php: I32ec0 (duration: 00m 30s)
  • 00:46 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.16/includes/jobqueue/JobQueueGroup.php: Debug (duration: 00m 28s)
  • 00:17 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.16/includes/jobqueue/JobQueueGroup.php: Log REQUEST_URI (duration: 00m 28s)
  • 00:15 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.16/extensions/Wikidata/extensions/Wikibase/client/includes/Hooks/DataUpdateHookHandlers.php: Revert briefly disable Wikibase\Client\Hooks\DataUpdateHookHandlers::onParserCacheSaveComplete (duration: 00m 31s)
  • 00:09 eileen: Updating civicrm from 097bc2d298b93424ff07abb18eaf1cfb3f79a0f2 to c99abc42188b7f47726202b4740fbaf68d1c06ab
  • 00:01 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.16/extensions/Wikidata/extensions/Wikibase/client/includes/Hooks/DataUpdateHookHandlers.php: Briefly disable Wikibase\Client\Hooks\DataUpdateHookHandlers::onParserCacheSaveComplete (duration: 00m 26s)

2016-03-15

  • 23:43 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.16/includes/jobqueue/JobQueueGroup.php: (no message) (duration: 00m 24s)
  • 23:40 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.16/includes/jobqueue/JobQueueGroup.php: AdHocDebug for JobQueueGroup->lazyPush() (exclude job runners) (duration: 00m 28s)
  • 23:39 awight: update crm from 090d443c856574d45c80f89c4ae7ccb86c97448f to 097bc2d298b93424ff07abb18eaf1cfb3f79a0f2
  • 23:37 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.16/includes/jobqueue/JobQueueGroup.php: AdHocDebug for JobQueueGroup->lazyPush() (with ip) (duration: 00m 30s)
  • 23:33 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.16/includes/jobqueue/JobQueueGroup.php: AdHocDebug for JobQueueGroup->lazyPush() (duration: 00m 28s)
  • 23:12 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Strip rather than hide HTML in MobileFrontend (duration: 00m 29s)
  • 23:10 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: WikidataPageBanner config changes (duration: 00m 33s)
  • 22:40 logmsgbot: ori@tin Synchronized wmf-config/InitialiseSettings.php: I87c174cf83: Turn on RecursiveLinkPurge log bucket, for I29636c045 (duration: 00m 25s)
  • 22:32 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.16/extensions/VisualEditor/extension.json: T129704 (duration: 00m 24s)
  • 22:01 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.16/includes/api/ApiPurge.php: I29636c04: Add RecursiveLinkPurge log for API requests (duration: 00m 33s)
  • 21:47 logmsgbot: twentyafterfour@tin Purged l10n cache for 1.27.0-wmf.14
  • 21:44 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.27.0-wmf.17
  • 21:24 gehel: restarting elasticsearch server elastic2010.codfw.wmnet
  • 20:48 gehel: restarting elasticsearch server elastic2009.codfw.wmnet
  • 20:45 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.16/extensions/MoodBar/MoodBar.php: T129978 (duration: 00m 53s)
  • 20:36 twentyafterfour: grr now it says .17, pebkac
  • 20:35 twentyafterfour: testwiki still shows 1.27.0-wmf.16 :(
  • 20:35 mutante: terbium - gzip nutcracker.log.1 for disk space
  • 20:34 logmsgbot: twentyafterfour@tin Finished scap: testwiki to 1.27.0-wmf.17 (duration: 52m 15s)
  • 20:05 papaul: labstore200[3-4] OS install on hold: making new partman recipe
  • 19:42 logmsgbot: twentyafterfour@tin Started scap: testwiki to 1.27.0-wmf.17
  • 19:16 gehel: restarting elasticsearch server elastic2008.codfw.wmnet
  • 19:05 akosiaris: enable puppet on neon
  • 19:04 _joe_: rolling restart of restbase in codfw
  • 19:00 mutante: alsafi back up with 4.4 kernel
  • 18:57 mutante: alsafi - url-downloader codfw - reboot
  • 18:44 gehel: restarting elasticsearch server elastic2007.codfw.wmnet
  • 18:43 akosiaris: disable puppet on neon for a few minutes while deploying https://gerrit.wikimedia.org/r/#/c/276199/7
  • 18:27 akosiaris: pool sca1001, sca1002 for apertium.svc.eqiad.wmnet in conftool
  • 18:23 papaul: OS install labstore200[3-4]
  • 17:37 gehel: restarting elasticsearch server elastic2006.codfw.wmnet
  • 17:05 mutante: mw1017 (canary) - test wikimania redirect change, restart apache
  • 16:58 gehel: restarting elasticsearch server elastic2005.codfw.wmnet
  • 16:55 _joe_: repooling mw1107
  • 16:51 _joe_: repool mw1201
  • 16:51 mutante: authdns-update - add labstore2004
  • 16:46 mutante: labstore200[3-4] added to DHCP (T128764) @papaul #codfw
  • 16:46 _joe_: repooling mw1196
  • 16:44 _joe_: repooling mw2020
  • 16:30 mutante: osmium - delete /srv/ruthenium data that has already been copied back
  • 16:27 mutante: osmium - stopping rsyncd, removing remnants from backup job for ruthenium upgrade T122328
  • 16:27 ottomata: reenabled puppet on analytics1027
  • 16:16 gehel: restarting elasticsearch server elastic2004.codfw.wmnet
  • 16:08 akosiaris: restarted pybal on lvs1004, lvs1005, lvs1006 to pickup apertium change
  • 16:07 akosiaris: restarted pybal on lvs2004, lvs2005, lvs2006
  • 15:53 ottomata: disabled puppet on analytics1027 for camus stop and reboot to apply kernel update
  • 15:50 moritzm: rebooting analytics1027 for kernel upgrade
  • 15:38 akosiaris: restarted pybal on lvs1007, lvs1008, lvs1009. Had already restarted pybal on lvs1010, lvs1011, lvs1012 about an hour before
  • 15:26 moritzm: rebooting nobelium for kernel upgrade
  • 15:20 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: guwikiquote namespace and import stuff (duration: 00m 27s)
  • 15:13 moritzm: rebooting californium for kernel upgrade
  • 15:11 logmsgbot: demon@tin Synchronized wmf-config/throttle.php: Taller d'iniciació a la Viquipèdia, Montserrat throttle rule (duration: 00m 27s)
  • 15:09 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: VE for ip users on dewiki (duration: 00m 28s)
  • 15:08 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: VE single edit table plwiki (duration: 00m 29s)
  • 15:05 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: password policy stuff (duration: 00m 32s)
  • 14:57 gehel: restarting elasticsearch server elastic2003.codfw.wmnet
  • 14:23 logmsgbot: hoo@tin Synchronized wmf-config/InitialiseSettings-labs.php: (no message) (duration: 00m 36s)
  • 14:15 gehel: restarting elasticsearch server elastic2002.codfw.wmnet
  • 13:14 gehel: restarting elasticsearch server elastic2001.codfw.wmnet
  • 11:55 logmsgbot: hoo@tin Synchronized wmf-config/: Consistency sync (duration: 02m 29s)
  • 11:24 hoo: Updated Wikidata's property suggester with data from Monday's json dump
  • 11:05 moritzm: rebooting subra/suhail for kernel upgrade
  • 10:55 moritzm: rolling reboot of mw* in codfw for kernel upgrade
  • 09:27 mobrovac: restbase rolling restart for https://gerrit.wikimedia.org/r/277056
  • 08:24 mobrovac: restbase deploy end of c68f5f456
  • 08:15 mobrovac: restbase deploy start of c68f5f456
  • 08:10 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I01c01dcd: Follow-up for Ieeb76087: tolerate missing trailing semicolon (duration: 00m 29s)
  • 07:59 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: Ieeb76087: Allow X-Wikimedia-Debug header to request pages in read-only mode (duration: 00m 25s)
  • 07:57 logmsgbot: ori@tin Synchronized wmf-config/StartProfiler.php: I82ec01a: X-Wikimedia-Debug: profile if "profile" attribute set (duration: 00m 25s)
  • 07:27 ori: log files for RO test in fluorine:/a/mw-log.read-only.1458025363.tar.bz2
  • 07:15 logmsgbot: ori@tin Synchronized wmf-config/db-eqiad.php: I1eb69f16: Revert "Put eqiad in read-only mode for scheduled test" (for real) (duration: 00m 28s)
  • 07:10 _joe_: reenabling puppet, jobrunner, jobchron on jobrunners and videoscalers
  • 07:10 logmsgbot: ori@tin Synchronized wmf-config/db-eqiad.php: I1eb69f16: Revert "Put eqiad in read-only mode for scheduled test" (duration: 00m 28s)
  • 07:02 logmsgbot: ori@tin Synchronized wmf-config/db-eqiad.php: Ie3f798ac: Put eqiad in read-only mode for scheduled test (duration: 00m 55s)
  • 06:59 _joe_: stopping all jobrunners in eqiad
  • 06:56 _joe_: stopping jobrunner and jobchron on the videoscalers in eqiad
  • 06:45 _joe_: stopping puppet on the eqiad jobrunners, in preparation for the read-only test
  • 02:53 logmsgbot: krinkle@tin Synchronized wmf-config/mobile.php: Remove legacy scripts from autoload on mobile (duration: 00m 26s)
  • 02:48 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Mar 15 02:48:16 UTC 2016 (duration 8m 54s)
  • 02:39 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.16) (duration: 17m 31s)
  • 01:46 mutante: alsafi was hanging and in the second in connected to the Ganeti console it was back like nothing happened
  • 00:07 logmsgbot: reedy@tin Synchronized wmf-config/extension-list: Remove OAI (duration: 00m 24s)
  • 00:06 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: Disable OAI (duration: 00m 25s)
  • 00:01 MaxSem: ran mwscript maintenance/updateCollation.php --wiki=ruwikinews --force

2016-03-14

  • 23:55 MaxSem: ran mwscript maintenance/updateCollation.php --wiki=ruwikiquote --force
  • 23:52 MaxSem: ran mwscript maintenance/updateCollation.php --wiki=ruwikiversity --force
  • 23:51 MaxSem: ran mwscript maintenance/updateCollation.php --wiki=ruwikivoyage --force
  • 23:50 MaxSem: mwscript namespaceDupes.php --wiki=knwiki --fix produced 3changes unrelated to new namespaces, --source-pseudo-namespace gave no results
  • 23:48 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/277432 (duration: 00m 26s)
  • 23:46 MaxSem: ran mwscript maintenance/updateCollation.php --wiki=ruwikibooks --force
  • 23:41 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/276743/ (duration: 00m 26s)
  • 23:39 MaxSem: mwscript namespaceDupes.php --wiki=kowiktionary gives no pages to fix
  • 23:37 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/276919/ (duration: 00m 27s)
  • 23:34 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.16/extensions/WikimediaEvents/: https://gerrit.wikimedia.org/r/#/c/277337/ (duration: 00m 26s)
  • 23:33 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.16/extensions/CirrusSearch/: (no message) (duration: 00m 32s)
  • 23:29 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: SWAT (duration: 00m 27s)
  • 23:28 logmsgbot: maxsem@tin Synchronized dblists/commonsuploads.dblist: SWAT (duration: 00m 26s)
  • 23:26 logmsgbot: maxsem@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/276895/ (duration: 00m 26s)
  • 23:23 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/276547/ (duration: 00m 26s)
  • 23:22 logmsgbot: maxsem@tin Synchronized static/images/project-logos/guwiktionary.png: https://gerrit.wikimedia.org/r/#/c/276547/ (duration: 00m 30s)
  • 23:21 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/276485 (duration: 00m 26s)
  • 23:14 logmsgbot: maxsem@tin Synchronized docroot/noc: https://gerrit.wikimedia.org/r/276993 (duration: 00m 35s)
  • 22:14 greg-g: read-only mode, intermittently, on mw.org and other s3 wikis(?)
  • 22:11 hoo: Restarted hhvm on mw1166 (was flooding with undefined variable notices)
  • 22:09 logmsgbot: hoo@tin Synchronized wmf-config/InitialiseSettings.php: Log 'Wikibase\Client\Changes\WikiPageUpdater' (duration: 00m 27s)
  • 21:56 mutante: logstash - nginx -> T129934
  • 20:36 gehel: HTTPS activated on elasticsearch (no client using it yet)
  • 19:17 logmsgbot: krinkle@tin Synchronized docroot/: Update static symlinks (duration: 00m 28s)
  • 19:09 logmsgbot: krinkle@tin Synchronized docroot/mediawiki/: Update static symlink (duration: 00m 29s)
  • 18:56 urandom: Starting Cassandra repairs on restbase1007-a.eqiad.wmnet : T108611
  • 18:44 logmsgbot: krinkle@tin Synchronized w: Replace /w/static with symlink (duration: 00m 30s)
  • 18:43 logmsgbot: krinkle@tin Synchronized static/: Moved /srv/mediawiki/w/static to /srv/mediawiki/static (duration: 00m 29s)
  • 18:27 ejegg: updated SmashPig from ad27a3a18cccd6bbd843e671218a1f5c190bdb8a to 9f08f6a1891b0a2bb70eacf460c2f9a8153c3b4e
  • 18:04 ejegg: increased max jobs per run for Adyen job runner from 10 to 250, time limit from 60 to 120
  • 18:00 chasemp: restart pdns on labservices1001 :)
  • 16:39 csteipp: deployed fix for scribunto issue related to T110143
  • 16:31 logmsgbot: thcipriani@tin Finished scap: Better announce new optional MT services available gerrit:277195 (duration: 26m 17s)
  • 16:04 logmsgbot: thcipriani@tin Started scap: Better announce new optional MT services available gerrit:277195
  • 16:01 _joe_: repooling mw1141
  • 16:00 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.16/extensions/VisualEditor/lib/ve/src/ce/nodes/ve.ce.TableNode.js: SWAT: Update VE core submodule to wmf/1.27.0-wmf.16 HEAD gerrit:277183 (duration: 00m 30s)
  • 15:54 _joe_: repooling mw1128
  • 15:28 moritzm: rebooting nobelium for kernel update
  • 15:16 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.16/extensions/Echo: SWAT: thank-you-edit: canRender for deleted page and extra fix gerrit:276916 (duration: 00m 39s)
  • 14:53 ejegg: updated civicrm from 7e21d5ad1f9ff404ac155a38f771b744bf238ccf to 090d443c856574d45c80f89c4ae7ccb86c97448f
  • 14:37 elukey: re-imaging mw2090.codfw for T126987
  • 14:27 godog: pool restbase1009
  • 14:16 godog: pool restbase101[01]
  • 13:16 elukey: re-added kafka1002 to the eventbus confd pool after maintenance
  • 13:14 elukey: removed kafka1002.eqiad.wmnet from eventbus' pool via confd for maintenance
  • 13:13 elukey: re-added kafka1001 to the eventbus confd pool after maintenance
  • 12:57 elukey: removed kafka1001 from eventbut's pool via confd
  • 11:47 moritzm: installing security updates for openssl, curl, gcrypt, libpng, jasper, expat and libxml2 on remaining mw1* app servers (along with HHVM restarts)
  • 10:56 moritzm: installing rsync security updates
  • 10:38 godog: shut restbase100[12]
  • 10:37 moritzm: installing security updates for openssl, curl, gcrypt, libpng, jasper, expat and libxml2 on mw2* (along with HHVM restarts)
  • 10:27 mobrovac: restbase restarting restbase in prod to apply https://gerrit.wikimedia.org/r/276728
  • 10:13 mobrovac: restbase deployed 3bedb8f on restbase101[01].eqiad.wmnet
  • 10:06 mobrovac: restbase stopping restbase on rb100[12] before full deprovisioning
  • 09:46 godog: depool restbase1001 / restbase1002 before deprovisioning
  • 05:19 ori: Restarting HHVM on mw1025 with known-bad config option 'hhvm.enable_reusable_tc = true' for debugging
  • 02:38 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Mar 14 02:38:54 UTC 2016 (duration 8m 41s)
  • 02:30 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.16) (duration: 13m 23s)

2016-03-13

  • 02:39 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Mar 13 02:39:06 UTC 2016 (duration 8m 35s)
  • 02:30 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.16) (duration: 13m 24s)

2016-03-12

  • 17:12 hoo: Updated operations/dumps/dcat on snapshot1003 from 92ab37d94e to e97408df39
  • 04:44 logmsgbot: legoktm@tin Synchronized php-1.27.0-wmf.16/extensions/CharInsert/: Revert "Remove inline event handler js from charinsert" - https://gerrit.wikimedia.org/r/#/c/276932/ T129524 (duration: 00m 29s)
  • 02:38 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Mar 12 02:38:16 UTC 2016 (duration 8m 45s)
  • 02:29 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.16) (duration: 12m 52s)
  • 00:55 logmsgbot: aaron@tin Synchronized rpc/RunJobs.php: 7a8bd37247b7dfb (duration: 00m 38s)

2016-03-11

  • 23:21 logmsgbot: krinkle@tin Synchronized wmf-config/InitialiseSettings.php: test2wiki favicon (duration: 00m 30s)
  • 22:50 logmsgbot: aaron@tin Synchronized wmf-config/jobqueue-eqiad.php: Disable persistent redis connections and bump timeout a bit (duration: 00m 38s)
  • 19:40 jynus: restarting dbstore1002
  • 18:51 mutante: mw1001 - removed font again, mw1153 confirmed puppet installs it (only) on imagescalers
  • 18:42 mutante: mw1001 install fonts-gujr-extra to confirm gerrit 276501 is fine
  • 17:38 elukey: Increased mediawiki::jobrunner::runners_basic to 30 for mw116[123789]
  • 16:38 logmsgbot: mattflaschen@tin Synchronized wmf-config/InitialiseSettings.php: Comment-only change (duration: 00m 41s)
  • 15:44 chasemp: labtestmetal and labtestvirt I am commondeering for some nfs and storage testing that cannot be done in labs, I will reimage when done
  • 15:41 elukey: forced puppet agent -tv on mw1164
  • 15:39 elukey: increased mediawiki::jobrunner::runners_basic from 20 to 30 for mw116[3-6]
  • 15:26 gehel: deploying https://gerrit.wikimedia.org/r/#/c/274382/ - new defined type not used anywhere, should have less than zero impact.
  • 15:20 moritzm: installing security updates for openssl, curl, gcrypt, libpng, jasper, expat and libxml2 on mw1018-1025 (and restarts of HHVM)
  • 15:04 logmsgbot: demon@tin Synchronized README: no-op, co-master sync (duration: 00m 30s)
  • 15:03 logmsgbot: demon@tin rebuilt wikiversions.php and synchronized wikiversions files: moving large.dblist back to wmf.16, did not help
  • 14:45 logmsgbot: demon@tin rebuilt wikiversions.php and synchronized wikiversions files: moving largest.dblist back to wmf.15 for now
  • 14:33 godog: run swiftrepl thumbs eqiad -> codfw with concurrency 96
  • 14:25 godog: run swiftrepl thumbs eqiad -> codfw with concurrency 128
  • 14:12 moritzm: installed security updates for openssl, curl, gcrypt, libpng, jasper, expat and libxml2 on mw1017 (other canaries will be upgraded later on if all is well, the rest of mw* on Monday)
  • 13:50 apergos: on fluorine truncated archive/redis.log-20160311 and archive/JobQueueFederated.log-20160311 to 5mb each, they were each about 500gb
  • 13:26 mobrovac: restbase deploy end of 3bedb8f5c42
  • 12:56 mobrovac: restbase deploy start of 3bedb8f5c42 on canary restbase1001
  • 12:36 mobrovac: mathoid deploying 7a282a4181a4
  • 12:11 moritzm: uploaded backport of linux-tools 4.4-4 for jessie-wikimedia to carbon (provides kbuild and perf amonst others)
  • 11:48 akosiaris: restarting gerrit on ytterbium
  • 11:27 jynus: running authdns-update to solve a dns typo
  • 11:25 jynus: killed some long-running git-upload-pack tasks
  • 10:26 hashar: Gerrit task management documented on https://wikitech.wikimedia.org/wiki/Gerrit#Tasks_management
  • 10:11 hashar: Gerrit: killed stuck "git-upload-pack '/mediawiki/core.git'" tasks
  • 09:46 elukey: Rebooting rdb200[56].codfw for kernel upgrade
  • 09:13 ema: umounted /sys/kernel/debug/tracing on cp1067
  • 09:12 hashar: Enabling puppet on gallium.wikimedia.org . Been disabled since ~ Mon Mar 7 10:26:01 UTC 2016
  • 09:07 moritzm: uploaded linux-meta 1.9 for jessie-wikimedia to carbon (which now defaults linux-meta to installing Linux 4.4)
  • 08:49 ema: cp1067: apt-get removed linux-image-3.16.0-4-amd64 and linux-image-4.4.0-1-amd64-dbg to free up some disk space
  • 08:20 moritzm: restarted hhvm on mw1246, mw1258
  • 08:18 moritzm: restarted hhvm on mw1220, mw1244
  • 08:17 moritzm: restarted hhvm on mw1174, mw1186, mw1211
  • 07:31 logmsgbot: kartik@tin Synchronized php-1.27.0-wmf.16/extensions/ContentTranslation: Deploying 276700 for ContentTranslation (duration: 00m 43s)
  • 06:44 _joe_: restarted hhvm on mw1166
  • 04:50 ori: Uninstalled HHVM on nobelium; not puppetized.
  • 04:15 papaul: rdb200[5-6] installation complete
  • 03:45 papaul: rdb200[5-6] signing puppet certs, salt-key, initial run
  • 03:00 ejegg: updated payments-wiki from 7248c10613018c1a15a1754ab80242f79d04532f to 79f5c9389edd089ae5951a7d172e74e68946a93c
  • 02:57 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.16/extensions/VisualEditor/modules/ve-mw/ui/pages/ve.ui.MWTemplatePage.js: touch - see comment on https://gerrit.wikimedia.org/r/#/c/274120/ (duration: 00m 31s)
  • 02:54 papaul: installing rdb200[5-6]
  • 02:39 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Mar 11 02:39:01 UTC 2016 (duration 8m 45s)
  • 02:31 mutante: cygnus/technetium: puppet cert clean, salt-key -d (neodymium), puppetstoredconfigclean.rb (rm from Icinga), gnt-instance remove (destroy VMs)
  • 02:30 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.16) (duration: 12m 33s)
  • 02:23 mutante: cygnus - poweroff (for variety)
  • 02:21 mutante: technetium - shutdown -h now
  • 01:56 ejegg: updated SmashPig from 154688db3595060b55498dea3cdf1ee206a854a0 to ad27a3a18cccd6bbd843e671218a1f5c190bdb8a
  • 01:25 logmsgbot: krinkle@tin Synchronized w/static.php: 6da604f and 49c07ac (duration: 00m 39s)
  • 01:06 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Un-revert Flow change on mw.org (duration: 00m 26s)
  • 01:06 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Un-revert Flow change on mw.org (duration: 00m 29s)
  • 01:04 MaxSem: Made mediawiki/php/wikidiff read-only in Gerrit
  • 00:46 logmsgbot: catrope@tin Synchronized portals: (no message) (duration: 00m 26s)
  • 00:45 logmsgbot: catrope@tin Synchronized portals/prod/wikipedia.org/assets: (no message) (duration: 00m 26s)
  • 00:43 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.16/extensions/Echo/Hooks.php: Try fixing the thank-you-edit bug again (duration: 00m 26s)
  • 00:41 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.16/includes/diff/DifferenceEngine.php: Convert timing to milliseconds (duration: 00m 26s)
  • 00:32 RoanKattouw: Running populateContentModel.php on mediawikiwiki so I can un-revert the Flow change
  • 00:29 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Revert Flow change on mw.org (duration: 00m 27s)
  • 00:28 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Revert Flow change on mw.org (duration: 00m 27s)
  • 00:24 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Enable cross-wiki notifications beta feature on all wikis (duration: 00m 27s)
  • 00:22 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Clean up old Flow occupy stuff (duration: 00m 26s)
  • 00:22 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Clean up old Flow occupy stuff (duration: 00m 25s)
  • 00:19 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Make Flow the default in talk namespaces on mediawikiwiki (duration: 00m 38s)
  • 00:08 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Enable completion suggester on all but top 12 wikis (duration: 00m 32s)

2016-03-10

  • 23:53 urandom: Starting Cassandra cleanup op on restbase10{07,10,11}-{a,b}.eqiad.wmnet : T125842
  • 23:53 urandom: Starting Cassandra cleanup op on restbase10{07,10,11}-{a,b}.eqiad.wmnet : T125832
  • 23:27 ejegg: updated civicrm from cbcfafcb2e6d6e1dae12a2b2d554445871992aff to 7e21d5ad1f9ff404ac155a38f771b744bf238ccf
  • 23:06 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.16/languages/LanguageConverter.php: deploying https://gerrit.wikimedia.org/r/#/c/276467/ (duration: 00m 31s)
  • 22:17 ejegg: updated payments wiki from 07dcb0f3962143fd9497ccad19b7b682beb991fe to 7248c10613018c1a15a1754ab80242f79d04532f
  • 22:03 csteipp: deployed patch for T129506 to wmf15 & 16
  • 21:55 mutante: cygnus , killing akumar's processes
  • 21:42 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.16/extensions/Wikidata: Deploy https://gerrit.wikimedia.org/r/#/c/276473/ (duration: 02m 10s)
  • 21:03 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.27.0-wmf.16
  • 20:32 cmjohnson1: replacing failed disk ms1001
  • 20:31 AaronSchulz: Ran <<mwscriptwikiset extensions/WikimediaMaintenance/filebackend/setZoneAccess.php private.dblist --backend=local-multiwrite --private>>
  • 19:23 godog: enable puppet in cache_cluster in eqiad, followed by the rest
  • 19:17 godog: set timeline and math data containers as readable in swift codfw
  • 18:53 logmsgbot: krinkle@tin Synchronized wmf-config/db-codfw.php: I47954e21 (duration: 00m 30s)
  • 18:46 godog: sync wikipedia-commons-gwtoolset-metadata with swiftrepl eqiad -> codfw T129359
  • 18:43 yurik: deployed latest kartotherian
  • 18:19 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.15/includes/GlobalFunctions.php: wfShellExec() debug logging for T129467 (take 2) (duration: 00m 26s)
  • 18:13 jynus: importing zhwiki missing records from production to labs
  • 18:13 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.15/includes/GlobalFunctions.php: wfShellExec() debug logging for T129467 (duration: 00m 28s)
  • 17:44 godog: running puppet in batches on cache_upload in eqiad after https://gerrit.wikimedia.org/r/#/c/276223/
  • 17:38 logmsgbot: filippo@tin Synchronized wmf-config/filebackend-production.php: swift codfw sync replication T129089 (duration: 00m 39s)
  • 17:19 ori: cleared hhbc bytecode repo on mw1082 and mw1245 on suspicion that old translations were reused
  • 16:36 moritzm: mw1122: restarted hhvm
  • 15:12 akosiaris: restart hhvm on mw1107
  • 14:29 moritzm: mw1107: restarted hhvm
  • 14:15 _joe_: restarted multiple times hhvm on mw112; it didn't help, so removed manually the warmup upstart task and the old sqlite cache; also disabling puppet on the machine
  • 14:08 urandom: increasing outbound stream throughput on restbase1002.eqiad.wmnet to 200mbps : T125842
  • 13:32 bblack: restarted mw1122 hhvm
  • 13:26 bblack: mw1210: restarted hhvm
  • 13:25 moritzm: mw1241: restarted hhvm
  • 13:20 bblack: mw1188: restarted hhvm (before alert hit IRC, was already pending in icinga)
  • 13:19 bblack: mw1240: restarted hhvm
  • 13:17 bblack: mw1122: restarted hhvm, killed stale procs from Jan24 looking like: /usr/bin/hhvm --php -c /etc/hhvm/fcgi.ini -r echo ini_get("hhvm.jit_warmup_requests")?:11;
  • 13:15 bblack: restarting hhvm on mw1112
  • 13:12 bblack: restarting hhvm on mw1248, mw1151
  • 13:06 bblack: restarting hhvm on mw1251 ...
  • 13:02 bblack: restarting hhvm on mw1258 ...
  • 13:01 bblack: restarting hhvm on mw1243
  • 12:59 bblack: restarting hhvm on mw1091 mw1249 mw1252 mw1242 ...
  • 12:56 bblack: restarting hhvm on mw1217
  • 12:01 _joe_: restarted hhvm on mw1122, stuck at startup
  • 10:39 mobrovac: restbase deploy end of 26bd4aa28
  • 10:31 jynus: setting up the rest of the cross-datacenter master-master connections pending in wmf databases
  • 10:09 godog: decommissioning restbase1002.eqiad.wmnet : T125842
  • 09:50 logmsgbot: volans@tin Synchronized wmf-config/db-codfw.php: Rebalance external storage servers in codfw T127330 (duration: 00m 34s)
  • 09:50 mobrovac: restbase deploy start of 26bd4aa28 on restbase1001
  • 08:55 ema: apache2 and hhvm restarted on mw1107, mw1122 and mw1119
  • 03:08 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Mar 10 03:08:54 UTC 2016 (duration 9m 38s)
  • 02:59 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.16) (duration: 12m 33s)
  • 02:33 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.15) (duration: 14m 08s)
  • 00:45 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.15/extensions/ZeroBanner: SWAT (duration: 00m 28s)
  • 00:44 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.15/extensions/MobileFrontend: SWAT (duration: 00m 32s)
  • 00:39 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.16/extensions/ZeroBanner: SWAT (duration: 00m 28s)
  • 00:39 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.16/extensions/MobileFrontend: SWAT (duration: 00m 36s)
  • 00:31 ori: Upgrade of HHVM package to 3.12.1+dfsg-1 complete on all eqiad hosts save terbium
  • 00:23 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Set NS_PROJECT on azwiktionary (duration: 00m 29s)
  • 00:13 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Remove svnadmins group (duration: 00m 41s)

2016-03-09

  • 23:54 bblack: ending caches codfw->direct testing
  • 23:15 bblack: restarted puppetmaster on palladium
  • 23:03 bblack: caches: starting test codfw->direct (codfw caches -> eqiad apps)
  • 22:42 bearND: mobileapps deployed 26d4031
  • 22:36 bearND: starting mobileapps deploy, second try (without dead link removal patch)
  • 21:54 bearND: mobileapps deployed 95a2d76
  • 21:28 ori: Depooled mw1107
  • 21:26 bearND: starting mobileapps deploy
  • 21:22 ori: restarted HHVM on mw1107; lock-up
  • 21:06 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.16
  • 20:57 ori: Upgrade HHVM on CODFW app servers to 3.12.1+dfsg-1
  • 20:19 urandom: decommissioning restbase1001.eqiad.wmnet : T125842
  • 20:12 logmsgbot: krinkle@tin Synchronized wmf-config/db-eqiad.php: Clean up - Ibb4bb0b32f5 (duration: 00m 29s)
  • 20:11 logmsgbot: krinkle@tin Synchronized wmf-config/db-codfw.php: Clean up - Ibb4bb0b32f5 (duration: 00m 35s)
  • 19:01 logmsgbot: volans@tin Synchronized wmf-config/db-codfw.php: Change codfw external storage topology T127330 (duration: 00m 27s)
  • 18:52 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.16/includes/session/SessionManager.php: Add backtrace to log for MW_NO_SESSION warning mode gerrit:276232 (duration: 00m 50s)
  • 18:44 volans: Changing topology of local codfw masters for es2 and es3 before merging https://gerrit.wikimedia.org/r/#/c/276229/1 T127330
  • 17:53 csteipp: deployed patch for T110143 to wmf16
  • 17:40 csteipp: deployed patch for T122056
  • 16:09 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: SWAT: Wikipedia while at Women of the World Festival throttle rule gerrit:276179 (duration: 00m 30s)
  • 16:05 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Change LanguageOverlay bucket rates gerrit:276172 (duration: 00m 52s)
  • 14:54 logmsgbot: volans@tin Synchronized wmf-config/db-codfw.php: Rebalance external storage servers in codfw T127330 (duration: 00m 41s)
  • 14:14 moritzm: installing Django security updates
  • 13:19 moritzm: installing beanshell security updates
  • 12:46 paravoid: set disable on cr2-knams:xe-1/2/0 (Init7), issues with at least one large network
  • 12:25 elukey: added rdb1003 back to the Job Runners queue. All the jobchron processes on jobrunners/videoscalers need to be restarted.
  • 12:07 logmsgbot: elukey@tin Synchronized wmf-config/jobqueue-eqiad.php: Add rdb1003 back to the Redis JobQueue pool after maintenance (duration: 00m 34s)
  • 12:00 elukey: re-enabled puppet on rdb1004
  • 11:47 moritzm: uploaded firmware-nonfree 20151018 to jessie-wikimedia on carbon
  • 11:39 godog: rolling restart cassandra in staging after merging https://gerrit.wikimedia.org/r/#/c/275917/
  • 11:27 jynus: setting up master-master cross-datacenter replication for s2 (db1018-db2017)
  • 11:20 ema: hhvm restarted on mw1140
  • 11:04 jynus: deployed manually new code at dbtree (noc.wikimedia.org)
  • 10:39 elukey: puppet disabled on rdb1004 (plus Redis instances) as precautionary step for master reimage - rdb1003
  • 10:35 jynus: setting up master-master replication on tools (labsdb100[45])
  • 09:32 moritzm: rebooting iron for kernel update
  • 09:26 elukey: puppet disabled on rdb1003 for debian re-image. Redis servers going to be stopped too as pre-step for backup.
  • 08:32 logmsgbot: oblivian@tin Synchronized wmf-config/jobqueue-eqiad.php: re-routing writes from rdb1003 to rdb1005 for reimaging (duration: 00m 36s)
  • 08:29 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Repool db2035 (duration: 00m 39s)
  • 07:49 _joe_: uploading hhvm 3.12, extensions to reprepro
  • 06:26 mutante: bast2001 - still installing in snail mode - please feel free to check if it's done, and if so re-add to puppet so users get created.thx
  • 03:16 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Mar 9 03:16:21 UTC 2016 (duration 8m 2s)
  • 03:08 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.16) (duration: 17m 51s)
  • 02:32 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.15) (duration: 13m 28s)
  • 02:13 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.15/includes/diff/DairikiDiff.php: I4d4b8f81c: Dont quote assert expressions in DairikiDiff (duration: 00m 27s)
  • 02:13 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.16/includes/diff/DairikiDiff.php: I4d4b8f81c: Dont quote assert expressions in DairikiDiff (duration: 00m 31s)
  • 01:40 csteipp: redeploy patch for T129120
  • 01:39 logmsgbot: csteipp@tin Synchronized php-1.27.0-wmf.16/includes/api/ApiQueryInfo.php: (no message) (duration: 00m 33s)
  • 00:33 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.15/extensions/Echo/: SWAT (duration: 00m 32s)
  • 00:32 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.15/extensions/MobileFrontend/: SWAT (duration: 00m 32s)
  • 00:30 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.16/extensions/MobileFrontend/: SWAT (duration: 00m 29s)
  • 00:29 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.16/extensions/Echo/: SWAT (duration: 00m 38s)
  • 00:25 logmsgbot: maxsem@tin Synchronized wmf-config/: https://gerrit.wikimedia.org/r/275378 - comment only (duration: 00m 27s)
  • 00:16 logmsgbot: maxsem@tin Synchronized wmf-config/filebackend-production.php: https://gerrit.wikimedia.org/r/272922 (duration: 00m 27s)
  • 00:13 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/275858/ (duration: 00m 28s)
  • 00:09 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/275275 (duration: 00m 29s)

2016-03-08

  • 23:55 ejegg: updated payments wiki from 2f16a0121fb9313a01a9646b8c71750385523569 to 07dcb0f3962143fd9497ccad19b7b682beb991fe
  • 23:20 mutante: bast2001 - install issues - extending downtime, bbiaw
  • 23:20 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.27.0-wmf.16
  • 23:11 mutante: neon: re-enable puppet, start icinga-wm
  • 23:03 logmsgbot: twentyafterfour@tin Synchronized w/static/: (no message) (duration: 00m 32s)
  • 22:53 mutante: bast2001 - powercycle, reinstall
  • 22:19 logmsgbot: twentyafterfour@tin Finished scap: testwiki to php-1.27.0-wmf.16 and rebuild l10n cache (duration: 26m 06s)
  • 22:04 ori: Canary application servers (mw1017-mw1025) and canary API application servers (mw1114-mw1119) upgraded to HHVM 3.12.1
  • 22:04 jynus: recreating slave watchdog events on all servers
  • 21:53 logmsgbot: twentyafterfour@tin Started scap: testwiki to php-1.27.0-wmf.16 and rebuild l10n cache
  • 21:02 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Repool db2040 (duration: 00m 34s)
  • 21:01 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1066 (duration: 00m 33s)
  • 20:49 ori: Updated mw1200 to HHVM 3.12.1; repooling
  • 20:45 ori: Depooled mw1200 for HHVM update
  • 20:40 jynus: killing long running queries on db1066
  • 20:39 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1066 (duration: 01m 49s)
  • 18:55 mutante: temp. stopped icinga-wm
  • 18:54 moritzm: uploaded linux 4.4.2-3+wmf1/jessie-wikimedia (based on Linux 4.4.4) to carbon
  • 17:46 bblack: switching text cache routing: ulsfo->codfw
  • 17:36 papaul: sinistra signing puppet certs, salt-key, initial run
  • 17:34 jynus: applying new grants to all database servers
  • 17:12 godog: upgrade cassandra to 2.1.13 on restbase1010
  • 16:57 godog: upgrade cassandra to 2.1.13 on restbase1009
  • 16:46 godog: upgrade cassandra to 2.1.13 on restbase1008
  • 16:34 godog: upgrade cassandra to 2.1.13 on restbase1007
  • 16:25 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove reader segmentation survey gerrit:275669 (duration: 00m 25s)
  • 16:19 logmsgbot: thcipriani@tin Synchronized wmf-config: SWAT: Update pbkdf2 hash parameters gerrit:274795 (duration: 00m 31s)
  • 16:16 godog: restarting cassandra on restbase1*
  • 16:12 godog: upgrade cassandra to 2.1.13 on restbase1001
  • 16:12 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor for new accounts on the German Wikipedia PART II gerrit:271712 (duration: 00m 29s)
  • 16:11 logmsgbot: thcipriani@tin Synchronized dblists/visualeditor-default.dblist: SWAT: Enable VisualEditor for new accounts on the German Wikipedia PART I gerrit:271712 (duration: 00m 32s)
  • 16:06 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Permissions configuration changes for gl.wikipedia gerrit:275285 and Modify throttle settings for frwiki and cawiki due to Workshop gerrit:275287 (duration: 00m 28s)
  • 15:40 godog: upgrade cassandra to 2.1.13 on restbase2006
  • 15:36 bblack: switching upload cache routing: ulsfo->codfw
  • 15:32 godog: upgrade cassandra to 2.1.13 on restbase2005
  • 15:29 godog: upgrade cassandra to 2.1.13 on restbase2004
  • 15:24 godog: upgrade cassandra to 2.1.13 on restbase2003
  • 15:15 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Repool db2038, db2039; depool db2035 for partitioning (duration: 00m 40s)
  • 15:09 bblack: switching misc-web cache routing: ulsfo->codfw
  • 14:51 godog: upgrade cassandra to 2.1.3 on restbase2002
  • 14:27 godog: upgrade cassandra to 2.1.3 on restbase2001
  • 14:10 godog: update reprepro with cassandra 2.1.13 T126629
  • 11:54 jynus: performing schema change (s7 partitioning) on db2040
  • 11:27 _joe_: restarting pybal on lvs2003,6 to pick up the config change
  • 08:43 elukey: Redis and Puppet stopped on rdb1002 (Job queue slave) as pre-step for Debian re-image
  • 08:00 jynus: starting schema change (s6 partitioning) on db2039
  • 03:05 ori: Upgraded mw1017 to hhvm_3.12.1+dfsg-1
  • 02:33 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.15) (duration: 13m 50s)
  • 02:09 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.15/resources/src/mediawiki.special/mediawiki.special.preferences.js: T122702 (duration: 02m 28s)
  • 01:03 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.15/extensions/Kartographer/: https://gerrit.wikimedia.org/r/#/c/275732/ (duration: 02m 25s)
  • 00:36 ejegg: updated payments-wiki from b85a671bc96edda35f36846d5c669a78b294524a to 2f16a0121fb9313a01a9646b8c71750385523569, updated fraud filters
  • 00:30 logmsgbot: ebernhardson@tin Synchronized wmf-config/: Set VisualEditorSingleEditTabSwitchTime to correct dates (duration: 02m 27s)
  • 00:25 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.15/extensions/Echo/Hooks.php: T128249 Try and avoid race conditions with thank-you-edit notifications (duration: 02m 23s)
  • 00:20 logmsgbot: ebernhardson@tin Synchronized wmf-config: T128774 enable completion suggester by default on test/test2wiki (duration: 02m 27s)
  • 00:14 logmsgbot: ebernhardson@tin Synchronized wmf-config/filebackend-production.php: Enable async swift writes to all wikis except commons (duration: 02m 25s)

2016-03-07

  • 23:45 greg-g: ssh: connect to host mw2212.codfw.wmnet port 22: Connection timed out
  • 23:40 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.15/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: Remember editor preference in WikiEditor too (duration: 02m 21s)
  • 23:34 ori: Updating mw1099 (which is depooled) to HHVM 3.12
  • 23:20 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/275703, now for realz (duration: 02m 22s)
  • 23:13 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/275703 (duration: 02m 23s)
  • 23:06 logmsgbot: ebernhardson@tin Synchronized tests/loggingTest.php: Sync out test only change (duration: 02m 20s)
  • 22:58 logmsgbot: maxsem@tin Finished scap: Enable Kartographer on testwiki (duration: 28m 31s)
  • 22:50 papaul: installing sinistra :new mw log host
  • 22:29 logmsgbot: maxsem@tin Started scap: Enable Kartographer on testwiki
  • 22:24 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.15/extensions/Kartographer: Initial deploy: get files into place (duration: 02m 27s)
  • 22:10 bearND: revert of mobileapps deploy complete
  • 22:06 bearND: starting to revert mobileapps deploy
  • 21:36 bearND: mobileapps deployed 49169e9
  • 21:31 bearND: starting mobileapps deploy
  • 21:14 subbu: finished deploying parsoid sha 5db1d28b
  • 21:07 subbu: synced code; restarted parsoid on wtp1001 as a canary
  • 21:05 subbu: starting parsoid deploy
  • 20:56 hashar: Nodepool successfully upgraded. T118573
  • 20:19 hashar: Nodepool restarting
  • 20:15 hashar: stopping nodepool
  • 20:07 csteipp: deployed patch for T129120
  • 20:02 hashar: Upgrading Nodepool from 0.1.1-wmf3 to 0.1.1-wmf.4 with andrewbogott | T118573
  • 19:17 csteipp: deployed initial patch for T109140
  • 19:03 mobrovac: restbase deploy end of 5add37b16
  • 18:32 mobrovac: restbase deploy start of 5add37b16 on restbase1001
  • 16:56 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable assignment of accountcreator for maiwiki gerrit:270897 (duration: 02m 21s)
  • 16:55 mobrovac: restbase deploy start of 88363c03e0 on restbase1001
  • 16:51 logmsgbot: thcipriani@tin Synchronized wmf-config: SWAT: Update CirrusSearch PoolCounter for cross-dc search gerrit:270897 (duration: 02m 25s)
  • 16:46 logmsgbot: thcipriani@tin Synchronized wmf-config/PoolCounterSettings.php: SWAT: Create pool counter for CirrusSearch completion suggester gerrit:268029 (duration: 02m 22s)
  • 16:38 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.15/extensions/Translate/tag/PageTranslationHooks.php: SWAT: Fix regression in marking page for translation gerrit:275366 (duration: 02m 20s)
  • 16:33 elukey: moved OCG Redis Job Queue from rdb1002 to rdb1007 for maintenance.
  • 16:31 _joe_: powercycling mw2212
  • 16:30 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Ateneo de Manila University workshops throttle rule gerrit:275149 and Namespace configuration on he.wikivoyage gerrit:275154 (duration: 02m 30s)
  • 16:17 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Namespace configuration on wuu.wikipedia gerrit:275155 (duration: 02m 26s)
  • 16:10 _joe_: shutting down mw1033
  • 15:02 jynus: depooling mw1033
  • 14:59 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Depool db2038, db2039, db2040 (duration: 02m 29s)
  • 14:54 chasemp: clean out snapshots from teh weekend on labstore1001 as load is running higher than expected
  • 14:50 moritzm: installing squid security updates
  • 14:47 jynus: sync-common mw1033
  • 14:41 jynus: powercycling mw1033 (unresponsive)
  • 14:40 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Depool db2038, db2039, db2040 (duration: 02m 59s)
  • 13:42 jynus: performing schema change on db2038 (s5: T120513), lag on that server expected
  • 12:52 godog: repool ms-fe1003
  • 12:13 godog: depool ms-fe1003 for trusty upgrade T125024
  • 10:51 moritzm: reimaging iron to jessie
  • 10:24 godog: disable puppet on graphite1001 / graphite2001 / labmon1001 before merging https://gerrit.wikimedia.org/r/#/c/274716
  • 10:04 moritzm: uploaded kernel-wedge 2.93+wmf1 for jessie-wikimedia to carbon (needed to build modern kernels)
  • 08:41 _joe_: disabled puppet on mw1026-69, cleaning up puppet facts and certs, then shutting them down
  • 02:32 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.15) (duration: 14m 27s)

2016-03-06

  • 10:45 _joe_: rebooting alsafi
  • 02:41 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Mar 6 02:41:06 UTC 2016 (duration 8m 2s)
  • 02:33 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.15) (duration: 15m 10s)

2016-03-05

  • 13:29 ema: hhvm restarted on mw1025
  • 11:32 godog: nodetool stop COMPACTION / CLEANUP on restbase1006
  • 11:14 volans: Data trasnfer completed during the night, (re)starting MySQL on es200[124] and es201[123] T127330
  • 02:37 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Mar 5 02:37:44 UTC 2016 (duration 7m 43s)
  • 02:30 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.15) (duration: 13m 51s)
  • 00:01 chasemp: initiate dd replicate from labstore1001 tools snapshot to labstore1002 lv of tools-04032016

2016-03-04

  • 22:41 gwicke: restbase1005: `nodetool stop -- CLEANUP; nodetool stop -- COMPACTION`
  • 22:29 matt_flaschen: Ran P2709 against DB manually to work around T127693
  • 21:58 mutante: bast2001 if your ssh client shows the fingerprint as base64 SHA256, the new default, you can ssh -o FingerprintHash=md5 bast2001.wikimedia.org to compare
  • 21:29 mutante: bast2001 - reinstalled with jessie, fingerprints on https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/bast2001.wikimedia.org
  • 21:17 mutante: bast2001 - revoke and sign new puppet cert / salt keys
  • 21:01 mutante: bast2001 - rebooting into PXE for T128899
  • 20:00 volans: Added logging to post-merge hook on palladium T128895
  • 17:51 cwd: updated paymentswiki from fbd22230ebf14e57f2c69ef1b342d4cdbc47d9a6 to b85a671bc96edda35f36846d5c669a78b294524a
  • 17:28 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-labs.php: prod nop, enables https in beta cluster for elasticsearch connections (duration: 00m 33s)
  • 17:25 jynus: chgrp recursive on tin to wikidev on .git/objects
  • 16:25 jynus: changing in a hot way db1047 replication filters
  • 16:03 ejegg: re-enabled payments queue consumers
  • 16:02 Jeff_Green: taking payments & listener out of maintenance mode
  • 15:58 ottomata: puppet disabled on stat1003 for reportupdater deployment, paused until dan is out of meetings
  • 14:56 cmjohnson1: rebooting iron to fix virtual console problem
  • 14:34 jynus: upgrade and restart dbstore2002 to apply new replication filters
  • 14:28 apergos: all services back in operation from dataset1001
  • 14:20 apergos: web service restored for dumps/download.wikimedia.org
  • 14:16 moritzm: installing perl security updates
  • 14:09 Jeff_Green: putting payments, civi, listener into maintenance mode
  • 14:06 ejegg: disabled fundraising queue consumer jobs
  • 14:04 moritzm: installing postgres security updates on labsdb1004
  • 14:02 bblack: puppet back online for all caches (ipsec changes complete)
  • 13:48 moritzm: installing pillow security updates
  • 13:41 bblack: disabling puppet on esams,ulsfo,codfw caches for ipsec changes, to minimize alertspam...
  • 13:39 urandom: canceling doomed bootstrap on restbase1009-a.eqiad.wmnet
  • 13:31 apergos: dumps/download wikimedia.org service interrupted now while server is being upgraded
  • 13:03 apergos: nfs filesystem from dataset1001 now unavailable as we prep for upgrade
  • 11:19 jynus: deploying new replication check algorithm cross-fleet
  • 10:30 volans: Start copying data from es200[124] to es201[123] (ETA ~16-17h) T127330
  • 10:07 logmsgbot: volans@tin Synchronized wmf-config/db-codfw.php: Update codfw external storage servers topology T127330 (duration: 00m 39s)
  • 09:10 moritzm: re-imaging iron with jessie
  • 08:41 jynus: downtiming all mysql replicas lag for 2 hours to test new alert check
  • 06:24 mutante: gerrit being restarted for config change 274741
  • 03:34 urandom: Starting `nodetool cleanup' on restbase100{1,2,7-a,7-b}.eqiad.wmnet and restbase1010-a : T95253
  • 03:28 urandom: starting decomission of restbase1009.eqiad.wmnet : T95253
  • 02:33 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Mar 4 02:33:49 UTC 2016 (duration 7m 44s)
  • 02:26 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.15) (duration: 11m 22s)
  • 00:50 ejegg: updated payments wiki from 111f92133cbd6b3890f53e29831ac022d3c26f51 to fbd22230ebf14e57f2c69ef1b342d4cdbc47d9a6
  • 00:35 logmsgbot: awight@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: followup Namespace configuration on wuu.wikipedia (duration: 00m 26s)
  • 00:34 RoanKattouw: Running extensions/Echo/maintenance/backfillUnreadWikis.php on all wikis. This will probably take a few days
  • 00:26 ejegg: rolled payments wiki back to 111f92133cbd6b3890f53e29831ac022d3c26f51
  • 00:24 ejegg: updated payments wiki from 111f92133cbd6b3890f53e29831ac022d3c26f51 to b8ac7f9188ba997d7e26882d0911f4eb97276d86
  • 00:23 logmsgbot: awight@tin Synchronized wmf-config: SWAT: Disable useless Echo eventlogging schema; Site name configuration on wuu.wikipedia; Set default completion suggester scoring for beta and prod (take 2) (duration: 00m 32s)
  • 00:12 logmsgbot: awight@tin Synchronized wmf-config: SWAT: Disable useless Echo eventlogging schema; Site name configuration on wuu.wikipedia; Set default completion suggester scoring for beta and prod (duration: 00m 36s)

2016-03-03

  • 22:36 twentyafterfour: applying hotfix for T128751 (link ../../../extensions/Gerrit* to /srv/phab/phabricator/src/extensions/Gerrit*) - will submit puppet patch to make this permanent.
  • 22:01 urandom: rolling restart of restbase production complete : T127387
  • 21:53 urandom: rolling restart of restbase production (config change) : T127387
  • 21:45 urandom: rolling restart of restbase staging complete : T127387
  • 21:40 urandom: rolling restart of restbase staging (config change) : T127387
  • 21:38 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings.php: Enable VisualEditor Single Edit Tab on officewiki https://gerrit.wikimedia.org/r/#/c/274814/ (duration: 00m 33s)
  • 20:26 urandom: rolling restart of restbase production complete : T127387
  • 20:22 logmsgbot: demon@tin rebuilt wikiversions.php and synchronized wikiversions files: (no message)
  • 20:22 logmsgbot: demon@tin Synchronized php-1.27.0-wmf.15/extensions/Echo: Roan made me do it (duration: 00m 53s)
  • 20:20 urandom: rolling restart of restbase production to apply config change : T127387
  • 20:03 urandom: rolling restart of restbase staging complete : T127387
  • 19:58 urandom: rolling restart of restbase in staging to apply config change : T127387
  • 19:44 gehel: elastic1031.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)
  • 19:27 volans: Completed migration of data from es200[68] to es201[579], added es201[579] to tendril. T127330
  • 19:08 logmsgbot: csteipp@tin Synchronized wmf-config/CommonSettings-labs.php: (no message) (duration: 00m 40s)
  • 19:05 chasemp: create temp 3T lv on labstore2001 to store test backup deltas
  • 18:41 gehel: elastic1030.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)
  • 18:14 urandom: lowering outbound stream throughput limit on restbase1010-a.eqiad.wmnet to 25mbps : T128107 T95253
  • 17:58 urandom: increasing stream throughput for restbase1010-b.eqiad.wmnet boostrap by 25mbps (5x5) : T128107 T95253
  • 17:09 gehel: elastic1029.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)
  • 16:25 gehel: elastic1028.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)
  • 16:25 Krinkle: krinkle@terbium mwscript deleteEqualMessages.php --wiki frwiki
  • 16:18 godog: restbase1001 bump stream throughput to 60mbps on restbase1001
  • 16:10 jynus: downtime on all mariadb replication lag checks in preparation to changing its check
  • 16:09 logmsgbot: jzerebecki@tin Synchronized w/static/images/project-logos/wikitech.png: Change the wikitech favicon and logo to the actual wikitech logo a29196d359b9924719b9166dca98a474ad9a6a2b 2 of 2 (duration: 00m 29s)
  • 16:08 logmsgbot: jzerebecki@tin Synchronized w/static/favicon/wikitech.ico: Change the wikitech favicon and logo to the actual wikitech logo a29196d359b9924719b9166dca98a474ad9a6a2b 1 of 2 (duration: 00m 30s)
  • 16:06 logmsgbot: jzerebecki@tin Synchronized wmf-config/CirrusSearch-production.php: CirrusSearch: Enable popqual (quality+pageviews) scoring method for the completion suggester T127943 (duration: 00m 37s)
  • 15:42 gehel: elastic1027.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)
  • 15:12 godog: cassandra throttle 1001, 1002, 1007-a, 1007-b, and 1010-a to 30mbps T95253
  • 15:03 moritzm: uploaded openssl 1.0.2g for jessie-wikimedia
  • 14:34 gehel: elastic1026.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)
  • 13:47 gehel: elastic1025.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)
  • 12:48 gehel: elastic1024.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)
  • 12:22 godog: temporary repool ms-fe1004, apply https://gerrit.wikimedia.org/r/#/c/273431 to test T128081
  • 12:12 logmsgbot: elukey@tin Synchronized wmf-config/jobqueue-eqiad.php: Revert - Remove rdb1003 from the Redis JobQueue pool for maintenance (duration: 00m 28s)
  • 12:07 logmsgbot: elukey@tin Synchronized wmf-config/jobqueue-eqiad.php: Remove rdb1003 from the Redis JobQueue pool for maintenance (duration: 00m 32s)
  • 12:02 gehel: elastic1023.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)
  • 11:55 _joe_: disabled notifications from the redises IPSEC checks while replication is disabled
  • 11:53 volans: Migrating data es2006->es2015 and es2008->es2017->es2019 T127330
  • 11:40 moritzm: upgrading cp1008 to openssl 1.0.2g
  • 11:21 logmsgbot: volans@tin Synchronized wmf-config/db-codfw.php: Depool es2005,es2008 to migrate data to es2015,es2017 T127330 (duration: 00m 53s)
  • 10:54 godog: replicate swift unsharded -deleted containers eqiad -> codfw T128096
  • 10:48 volans: Changing local replica topology for shard es3 in codfw for T127330
  • 10:37 volans: Changing local replica topology for shard es2 in codfw for T127330
  • 10:21 _joe_: rolling restart of strongswan on eqiad failing servers
  • 10:17 _joe_: restarted strongswan on mc1011
  • 10:07 gehel: elastic1022.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)
  • 09:57 volans: Added es2014,es2016,es2018 to tendril [ T127330 ]
  • 09:46 jynus: schema change finished on all hosts (except delayed slaves)
  • 09:21 _joe_: puppet re-enabled everywhere, now troubleshooting ipsec issues
  • 08:59 moritzm: repooled scb1002
  • 08:49 moritzm: repooled scb1001, depooling scb1002 for nodejs upgrade
  • 08:35 _joe_: disabled puppet across the main redises fleet in order to merge https://gerrit.wikimedia.org/r/271261 safely
  • 08:33 moritzm: depooling scb1001 for nodejs upgrade
  • 08:27 jynus: altering heartbeat table on all production servers
  • 06:39 ebernhardson: upgrade elastic1021.eqiad.wmnet to elasticsearch 1.7.5
  • 05:47 ebernhardson: upgrade elastic1020.eqiad.wmnet to elasticsearch 1.7.5
  • 05:02 ebernhardson: upgrade elastic1019.eqiad.wmnet to elasticseach 1.7.5
  • 04:42 eileen: CiviCRM updated from ff0e0c6f7bc8424f8097bc66e529b7836474d416 to cbcfafcb2e6d6e1dae12a2b2d554445871992aff
  • 04:05 bblack: disabling puppet on caches for a bit, JIC
  • 03:51 ebernhardson: upgrade elastic1018.eqiad.wmnet to elasticsearch 1.7.5
  • 03:13 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Mar 3 03:13:23 UTC 2016 (duration 8m 38s)
  • 03:04 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.15) (duration: 18m 46s)
  • 03:03 ebernhardson: upgrade elastic1017.eqiad.wmnet to elasticsearch 1.7.5
  • 02:28 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.14) (duration: 13m 44s)
  • 02:18 ebernhardson: upgrade elastic1016.eqiad.wmnet to elasticserach 1.7.5
  • 02:03 bd808: Events flowing into logstash elasticsearch cluster again after forcing allocation of missing shard replica
  • 01:59 twentyafterfour: puppet ran on iridium, no errors. :)
  • 01:54 bd808: Deleted logstash-2016.02.03 index to free disk space
  • 01:51 bd808: New index not being created due to low disk watermark exceeded on logstash1006
  • 01:49 bd808: Logstash elasticsearch cluster not responsive; investigating
  • 01:48 ebernhardson: upgrade elastic1015.eqiad.wmnet to elasticsearch 1.7.5
  • 01:44 twentyafterfour: phabricator is back online
  • 01:22 twentyafterfour: manually installed scap package on iridium, will fix in puppet immediately after maintenance is finished
  • 01:19 mutante: elastic1013 "dpkg reports broken packages "
  • 01:19 twentyafterfour: puppet says "Provider scap3 is not functional on this host"
  • 01:16 twentyafterfour: testing puppet on iridium
  • 01:07 mutante: iridium - stop apache
  • 01:00 ebernhardson: upgrade elastic1014.eqiad.wmnet to elasticsearch 1.7.5
  • 00:51 twentyafterfour: Phabricator will be going down for maintenance around 01:00 UTC (Approximately 10 minutes from now)
  • 00:37 logmsgbot: hoo@tin Synchronized php-1.27.0-wmf.15/extensions/TemplateData/: Change default format to null instead of 'inline' (duration: 01m 02s)
  • 00:30 bd808: Ran sync-common on mw1025
  • 00:23 hoo: Ran sync-common on mw1025, because it apparently didn't pick up recent changes
  • 00:20 ebernhardson: upgrade elastic1013.eqiad.wmnet to elasticsearch 1.7.5
  • 00:19 hoo: Restarted hhvm on mw1025 because of "Cannot access property on non-object in /srv/mediawiki/php-1.27.0-wmf.14/includes/filerepo/LocalRepo.php"
  • 00:14 mutante: wikitech: delete /a/backup/public/foo and ./bar cruft
  • 00:10 logmsgbot: hoo@tin Synchronized wmf-config/Wikibase-production.php: Set $wgWikimediaBadgesCommonsCategoryProperty to null on commons (T128661) (duration: 01m 09s)
  • 00:00 andrewbogott: restarting pdns on labservices1001

2016-03-02

  • 23:51 chasemp: ran puppet on elastic1012 manually which started a mystery stopped (crashed?) elastic search
  • 22:49 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.15/includes/api/ApiMain.php: Fix PHP Notice (duration: 01m 17s)
  • 22:28 urandom: enabling brotli compression on local_group_wikipedia_T_parsoid_html.data in staging, and forcing rewrite of corresponding tables on xenon : T125906
  • 21:12 urandom: forcing a major compaction on {local_group_wikipedia_T_parsoid_dataW4ULtxs1oMqJ,local_group_wikipedia_T_parsoid_html}.data, xenon.eqiad.wmnet : T125906
  • 20:53 bblack: repooling cp1048, seems unlikely to recrash (rare kernel bug)
  • 20:46 bblack: cp1048: depooled in confd, too
  • 20:45 bblack: cp1048: unresponsive console, powercycled
  • 20:35 gehel: elastic1011.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)
  • 20:11 logmsgbot: demon@tin Finished scap: group1 to wmf.15 (duration: 08m 41s)
  • 20:03 logmsgbot: demon@tin Started scap: group1 to wmf.15
  • 19:21 mobrovac: restbase rolling restart for https://gerrit.wikimedia.org/r/274456 T127387
  • 19:07 volans: Data transfer completed, started MySQL and replica on es2014,es2016,es2018 [ T127330 ]
  • 18:58 gehel: elastic1010.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)
  • 18:44 apergos: rolled back all changes for dataset1001, running with same old precise OS, grrrrr
  • 18:06 apergos: still slugging away at pxe book with these broadcom netxtreme II nics (dataset1001)
  • 17:55 logmsgbot: volans@tin Synchronized wmf-config/db-codfw.php: Repooling external storage DBs in codfw after data was copied: T127330 (duration: 01m 06s)
  • 17:44 godog: bounce statsdlb on graphite1001 to add 3x statsite instances T105679
  • 17:35 jynus: disabling puppet on db1009 (m5-master) to test heartbeat changes
  • 17:20 gehel: elastic1009.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)
  • 16:54 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enabling ShortURL for bnwikisource gerrit:273936 (duration: 01m 04s)
  • 16:39 mobrovac: restbase deploy end of fb66dbf
  • 16:34 gehel: elastic1008.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)
  • 16:31 mobrovac: restbase deploy continue of fb66dbf for the rest of the nodes
  • 16:30 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.15/extensions/ContentTranslation/includes/TranslationStorageManager.php: SWAT: Use correct timestamp for updates gerrit:274363 (duration: 00m 59s)
  • 16:28 urandom: starting post-bootstrap (1009-b) cleanup on restbase100{5,6,9-a}.eqiad.wmnet : T95253
  • 16:25 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.15/extensions/ContentTranslation/modules/widgets/translator/ext.cx.translator.js: SWAT: Translator widget: Fix js error if translator does not have recent contributions gerrit:274340 (duration: 01m 05s)
  • 16:07 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Do not send Referer from private wikis gerrit:274414 (duration: 01m 18s)
  • 15:53 apergos: extending maintenance window for dataset1001 by one hour to 5 pm UTC
  • 15:53 mobrovac: restbase deploy start of fb66dbf on restbase1001
  • 15:44 apergos: may extend the maintenance window for dataset1001 upgrade if headway can be made on PXE boot issues... 15 minutes left to decide
  • 15:16 andrewbogott: rebooting californium just to make sure dist-upgrade didn’t mess up grub
  • 15:15 gehel: elastic1007.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)
  • 15:06 andrewbogott: running apt-get dist upgrade to upgrade californium packages to openstack Liberty
  • 15:02 mobrovac: restbase reverting to fa1207e95, problems spotted in logstash
  • 14:58 mobrovac: restbase deploy start of 5def2f8 on restbase1001
  • 14:32 gehel: elastic1006.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)
  • 14:06 godog: bootstrap restbase1010-a T128107
  • 14:03 apergos: web service for dumps.wikimedia.org and download.wikimedia.org is now unavailable (upgrade of server to jessie)
  • 13:32 apergos: nfs service for dataset1001 disabled (impacts users of stat100{2,3} in prep for jessie upgrade
  • 13:23 gehel: elastic1005.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)
  • 13:13 _joe_: re-enabled puppet on scb1002, repooled scb1001 for mobileapps
  • 13:10 mobrovac: mobileapps re-deploying d384f1ba for T113542
  • 12:33 bblack: restarted logstash on logstash1002
  • 12:32 mobrovac: mobileapps stopping (again) the service on scb1001 for debugging, T113542
  • 12:29 bblack: restarted logstash on logstash1001
  • 12:27 _joe_: puppet disabled on both scb1001/2, depooled scb1001 for moborovac to test and config manually patched on scb1002 so that it runs with the old code correctly
  • 12:25 mobrovac: mobileapps rolling back to 68e38ec7, problems found in the latest deploy for T113542
  • 12:00 mobrovac: mobileapps stopping the service on scb1001 for debug purposes, T113542
  • 11:56 _joe_: stopped puppet on scb1002, depooled scb1001 from mobileapps
  • 11:36 mobrovac: mobileapps deploying d384f1ba
  • 11:09 jynus: profiling db1023 and db1061 for 24 hours- 1/20th of the queries slightly slower
  • 10:42 hashar: Zuul should no more be caught in death loop due to Depends-On on an event-schemas change. Hole filled with https://gerrit.wikimedia.org/r/#/c/274356/ T128569
  • 10:42 moritzm: restarting graphite-web on graphite1001 (for django security update)
  • 10:36 elukey: stopped Redis multi-instance on rdb1006 (Job Queue slave) as pre-step for Debian re-image
  • 10:16 gehel: elastic1004.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)
  • 09:43 volans: Cloning es2005->es2014, es2007->es2016, es2009->es2018, see T127330
  • 09:30 moritzm: installing nodejs updates on restbase*
  • 09:19 elukey: redis multi-instance stopped on rdb1004 (jobqueue slave) as pre-step for Debian re-image
  • 09:16 logmsgbot: volans@tin Synchronized wmf-config/db-codfw.php: Depooling external storage DBs in codfw for migration: T127330 (duration: 01m 24s)
  • 09:13 hashar: Zuul went crazy / caught in a loop of doom. Same has Saturday. It went back magically at 08:32 UTC T128569
  • 08:48 gehel: elastic1003.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)
  • 08:33 moritzm: installing Django security updates
  • 08:17 _joe_: disabling puppet on all memcached hosts in preparation for enabling ipsec
  • 07:36 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings.php: Disable $wgReferrerPolicy on private wikis (duration: 01m 01s)
  • 06:45 _joe_: rebooting serpens
  • 03:04 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Mar 2 03:04:14 UTC 2016 (duration 8m 49s)
  • 02:55 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.15) (duration: 09m 31s)
  • 02:29 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.14) (duration: 12m 32s)
  • 00:45 logmsgbot: krenair@tin Synchronized portals: https://gerrit.wikimedia.org/r/#/c/274316/ - try #2, this time with the submodule update (duration: 01m 17s)
  • 00:44 logmsgbot: krenair@tin Synchronized portals/prod/wikipedia.org/assets: https://gerrit.wikimedia.org/r/#/c/274316/ - try #2, this time with the submodule update (duration: 01m 16s)
  • 00:31 logmsgbot: krenair@tin Synchronized portals: https://gerrit.wikimedia.org/r/#/c/274316/ (duration: 01m 18s)
  • 00:30 logmsgbot: krenair@tin Synchronized portals/prod/wikipedia.org/assets: https://gerrit.wikimedia.org/r/#/c/274316/ (duration: 01m 18s)
  • 00:26 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/272926/ - prepare for VE default switch on dewiki (duration: 01m 17s)
  • 00:12 logmsgbot: krenair@tin Synchronized dblists/visualeditor-default.dblist: https://gerrit.wikimedia.org/r/#/c/274129/ - +testwiki (duration: 01m 20s)
  • 00:10 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/274129/ - VE SET on mediawikiwiki/testwiki (duration: 01m 21s)
  • 00:04 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/271932/ - disable Gather on enwiki (duration: 01m 26s)

2016-03-01

  • 23:57 ebernhardson: upgrade elastic1002.eqiad.wmnet to elasticsearch 1.7.5
  • 23:17 mutante: maps-test2001 - could not find dependency for postgres class is NOT related to my recent change. icinga crit since a long time
  • 22:34 mutante: re-enabled puppet runs on all mw* servers, mediawiki roles now in modules/role/manifests/mediawiki/
  • 22:27 mutante: temp. disabling puppet runs on mw appservers to be extra safe during mediawiki module change
  • 21:29 gehel: elastic1001.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)
  • 20:29 logmsgbot: demon@tin Finished scap: group0 to wmf.15 (duration: 31m 24s)
  • 19:58 logmsgbot: demon@tin Started scap: group0 to wmf.15
  • 19:19 jynus: testing heartbeat in m5 (db1009, db2030)
  • 19:14 logmsgbot: demon@tin scap aborted: testwikis to wmf.15 and rebuild l10n (duration: 01m 19s)
  • 19:14 chasemp: clean out /var/log/atop and /var/log/account on iridium
  • 19:13 logmsgbot: demon@tin Started scap: testwikis to wmf.15 and rebuild l10n
  • 18:53 mutante: iridium - gzip /var/log/atop/atop_20160*
  • 18:51 mutante: iridium: apt-get clean for some more disk space
  • 18:49 subbu: finished deploying parsoid sha 1f7ed5d0
  • 18:44 subbu: synced parsoid code; restarted parsoid on wtp1002 as a canary
  • 18:41 subbu: starting parsoid deploy
  • 17:52 gehel: elastic2024.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)
  • 17:18 mobrovac: restbase rolling-restart restbase for https://gerrit.wikimedia.org/r/#/c/273974/
  • 17:05 logmsgbot: thcipriani@tin Synchronized wmf-config/ProductionServices.php: SWAT: Add kafka1012.eqiad.wmnet back to the media-wiki config gerrit:273488 (duration: 00m 39s)
  • 16:44 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable rollbacker and suppressredirect group at cewiki gerrit:273828 (duration: 00m 41s)
  • 16:40 gehel: elastic2023.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)
  • 16:39 hashar: restarting Jenkins
  • 16:37 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Correct one Domain at $wgCopyUploadsDomains gerrit:273776 (duration: 00m 40s)
  • 16:33 hashar: Bunch of Jenkins job got stall because I have killed threads in Jenkins to unblock integration-slave-trusty-1003 :-( Jenkins / Zuul is catching up.
  • 16:32 logmsgbot: thcipriani@tin Synchronized wmf-config/filebackend-production.php: SWAT: Configure redis LockManager in both DCs, use the master everywhere. PART II gerrit:266514 (duration: 00m 46s)
  • 16:30 logmsgbot: thcipriani@tin Synchronized wmf-config/ProductionServices.php: SWAT: Configure redis LockManager in both DCs, use the master everywhere. PART I gerrit:266514 (duration: 00m 40s)
  • 16:24 logmsgbot: thcipriani@tin Synchronized wmf-config/redis.php: SWAT: Use wmfMasterDatacenter for picking the master redis config gerrit:266513 (duration: 00m 39s)
  • 16:18 logmsgbot: thcipriani@tin Synchronized wmf-config/CirrusSearch-production.php: SWAT: Add references to wmfServices for Cirrusearch gerrit:266512 (duration: 00m 56s)
  • 15:45 gehel: elastic2022.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)
  • 14:55 logmsgbot: elukey@tin Synchronized wmf-config/filebackend-production.php: Add mc1003 to the lock managers pool after maintenance (duration: 00m 40s)
  • 14:47 elukey: mc1003.eqiad added back to the redis/memcached pool after maintenance.
  • 14:34 gehel: elastic2021.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)
  • 13:53 gehel: elastic2020.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)
  • 13:09 logmsgbot: elukey@tin Synchronized wmf-config/filebackend-production.php: Remove mc1003 from the lock managers pool for maintenance (duration: 00m 40s)
  • 12:48 elukey: removed mc1003 from redis/memcached pools for maintenance
  • 12:37 gehel: elastic2019.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)
  • 12:16 moritzm: shutting down curium (decomissioned)
  • 11:53 moritzm: shutting down berkelium (decomissioned)
  • 11:44 jynus: shutting down pc100[123]
  • 11:43 logmsgbot: elukey@tin Synchronized wmf-config/filebackend-production.php: Add mc1002 back to the lock managers pool after maintenance (duration: 01m 01s)
  • 11:40 gehel: elastic2018.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)
  • 11:20 elukey: mc1002.eqiad added back to the memcached/redis pools after maintenance
  • 10:53 jynus: disabling puppet and following steps to decommission pc100[123]
  • 10:42 gehel: elastic2017.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)
  • 10:06 elukey: Amended previous log - Remove mc1002 from the lock managers after maintenance
  • 10:05 logmsgbot: elukey@tin Synchronized wmf-config/filebackend-production.php: Add mc1002 from the lock managers after maintenance (duration: 00m 56s)
  • 09:46 gehel: elastic2016.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)
  • 09:32 elukey: removed mc1002 from the redis/memcached pools for maintenance
  • 07:11 ebernhardson: upgrade elastic2015.codfw.wmnet to elasticsearch 1.7.5
  • 06:10 ebernhardson: upgrade elastic2014.codfw.wmnet to elasticsearch 1.7.5
  • 04:55 ebernhardson: upgrade elastic2013.codfw.wmnet to elasticsearch 1.7.5
  • 04:03 ebernhardson: upgrade elastic2012.codfw.wmnet to elasticsearch 1.7.5
  • 02:32 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Mar 1 02:32:41 UTC 2016 (duration 7m 36s)
  • 02:30 ebernhardson: upgrade elastic2011.codfw.wmnet to elasticsearch 1.7.5
  • 02:25 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.14) (duration: 10m 50s)
  • 01:40 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.14/extensions/CentralAuth: Idc873134: Avoid using "new CentralAuthUser" since it avoids the cache (duration: 00m 51s)
  • 01:36 ebernhardson: upgrade elastic2010.codfw.wmnet to elasticsearch 1.7.5
  • 01:03 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.14/includes/user/User.php: I419f356b: Cache user data in memory (duration: 00m 42s)
  • 00:45 ebernhardson: upgrade elastic2009.codfw.wmnet to elasticsearch 1.7.5
  • 00:39 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.14/extensions/Echo/Hooks.php: Add debug logging for thank-you-edit notifications (duration: 00m 43s)
  • 00:36 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Configure default Echo subscriptions user options on hewiki (take 2, part 2) (duration: 00m 40s)
  • 00:35 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Configure default Echo subscriptions user options on hewiki (take 2, part 1) (duration: 00m 42s)
  • 00:31 logmsgbot: catrope@tin Synchronized wmf-config/StartProfiler.php: XHGui: Use SCRIPT_NAME as the URI (duration: 00m 46s)
  • 00:30 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Revert "Configure default Echo subscriptions user options on hewiki", doesnt work (duration: 00m 46s)
  • 00:26 ori: Cleaned up remnants of XHGui role on hafnium, now that XHGui is on tungsten
  • 00:13 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Change rate for reader segmentation survey (duration: 00m 41s)
  • 00:10 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Configure default Echo subscriptions user options on hewiki (duration: 00m 48s)

2016-02-29

  • 23:56 ebernhardson: upgrade elastic2008.codfw.wmnet to elasticsearch 1.7.5
  • 23:17 eileen: Updating civicrm from 181ef79fd1360ca337990b6768ebc10b1185a431 to ff0e0c6f7bc8424f8097bc66e529b7836474d416
  • 23:05 ebernhardson: upgrade elastic2006.codfw.wmnet to elasticsearch 1.7.5
  • 22:56 logmsgbot: ori@tin Synchronized wmf-config/StartProfiler.php: I6d3ab949a6a: Apply xhgui role on tungsten (duration: 00m 56s)
  • 22:28 ostriches: phabricator: stopped phd daemons for the time being so replication can catch up and number of connections stop yelling ad people
  • 22:25 ebernhardson: upgrade elastic2006.codfw.wmnet to elasticsearch 1.7.5
  • 22:14 jynus: setting innodb_flush_log_at_trx_commit=0 on phabricator db slave
  • 21:37 ebernhardson: upgrade elastic2005.codfw.wmnet to elastic 1.7.5
  • 21:37 arlolra: updated Parsoid to version d809ad7a
  • 21:24 logmsgbot: elukey@tin Synchronized wmf-config/filebackend-production.php: Add mc1001 from the lock managers after maintenance (duration: 00m 54s)
  • 20:56 elukey: Added mc1001.eqiad back into memcached/redis pools after maintenance
  • 20:21 gehel: elastic2004.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)
  • 20:04 ori: Restarted statsdlb on graphite1001 with xxh64() improvement
  • 19:45 mutante: new planets, Albanian and Bulgarian: https://sq.planet.wikimedia.org/ | https://bg.planet.wikimedia.org/
  • 19:25 elukey: removed mc1001.equiad from the redis/memcached pools for maintenance
  • 19:22 gehel: elastic2003.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)
  • 19:13 logmsgbot: ori@tin Synchronized wmf-config/filebackend-production.php: I0ad3e23719c: Follow-up for I8a87a679ed: unlist rdb1 server (duration: 00m 44s)
  • 19:07 logmsgbot: elukey@tin Synchronized wmf-config/filebackend-production.php: Remove mc1001 from the lock managers for maintenance (duration: 00m 41s)
  • 18:40 logmsgbot: mattflaschen@tin Synchronized dblists/nonflow.dblist: Re-enable Flow on ptwikibooks. Accidentally disabled earlier. (duration: 00m 47s)
  • 18:31 ejegg: updated SmashPig from cf71e988c25e8569b84fb4ee52d1b043c7da27a6 to 154688db3595060b55498dea3cdf1ee206a854a0
  • 18:05 logmsgbot: krinkle@tin Synchronized w/static.php: (no message) (duration: 01m 01s)
  • 18:01 gehel: elastic2002.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)
  • 17:48 ejegg: disable Adyen SmashPig job runner
  • 17:46 ejegg: updated smashpig from e32e8fcdb43fedee6d2296e25ca1aff1e5028163 to cf71e988c25e8569b84fb4ee52d1b043c7da27a6
  • 16:43 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Set transwiki import sources for hi.wikiquote gerrit:272992 (duration: 00m 41s)
  • 16:38 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: SWAT: Maintenance on throttle.php gerrit:273580 (duration: 00m 41s)
  • 16:32 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.14/extensions/Flow: SWAT: Fix board move DB issue using new hook TitleMoveStarting gerrit:273536 (duration: 01m 00s)
  • 16:28 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.14/extensions/Wikidata: SWAT: Update Wikibase: Fix over-encoding of expanded URLs gerrit:273917 (duration: 02m 04s)
  • 16:23 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.14/includes/MovePage.php: SWAT: Add TitleMoveStarting, mirroring TitleMoveCompleting gerrit:273535 (duration: 00m 41s)
  • 16:09 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Temporarily disable thank-you-edit notifications gerrit:273836 (duration: 00m 41s)
  • 16:06 logmsgbot: thcipriani@tin Synchronized wmf-config/Wikibase.php: SWAT: Update Wikidata property blacklist gerrit:271971 (duration: 01m 01s)
  • 16:02 _joe_: stopping hhvm on mw1050 (depooled) for testing bug T128380
  • 15:50 urandom: Rolling restart of restbase production complete : T103124
  • 15:43 urandom: Perform rolling restart of restbase in production cluster : T103124
  • 15:42 urandom: Rolling restart of restbase staging complete : T103124
  • 15:34 urandom: Perform rolling restart of restbase in staging cluster : T103124
  • 15:32 urandom: forcing puppet run in restbase clustter ((noop) config deploy) : T103124
  • 15:30 urandom: forcing puppet run in restbase staging ((noop) config deploy) : T103124
  • 14:17 bblack: upgrading nginx on cp* to 1.9.4-1+wmf2 for T126616
  • 14:16 gehel: elastic2001.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)
  • 14:11 gehel: re-enabling icinga notifications for elasticsearch cluster on codfw
  • 14:00 gehel: icinga: adding authorization for Gehel
  • 13:54 bblack: puppet back to normal on caches
  • 13:35 bblack: disabling puppet on cp* for traffic-pool work
  • 12:50 godog: gmetad on uranium restarted following https://gerrit.wikimedia.org/r/273881 it will converge asap
  • 12:16 gehel: elastic2001.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101)
  • 11:57 jynus: reimaging dbproxy1005 with jessie before its put into production (potential alerts due to new role)
  • 10:46 ema: hhvm restarted on mw1119
  • 10:25 jynus: powercycling mw1140, almost 100% unresponsive, OOM probable cause
  • 09:58 godog: bootstrap restbase1009-b T95253
  • 02:32 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Feb 29 02:32:43 UTC 2016 (duration 7m 43s)
  • 02:25 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.14) (duration: 11m 12s)

2016-02-28

  • 22:34 logmsgbot: legoktm@tin Synchronized php-1.27.0-wmf.14/includes/parser/ParserCache.php: Include ParserCache::save() backtrace in MF pollution debug log - T124356 (duration: 00m 50s)
  • 07:49 apergos: restarted logmsbot on neon
  • 02:32 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Feb 28 02:32:00 UTC 2016 (duration 7m 42s)
  • 02:24 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.14) (duration: 10m 24s)
  • 00:59 ori_: Restarted logmsgbot on neon; it lost its IRC connection and did not recover it.

2016-02-27

  • 23:19 legoktm: 23:19:43 Synchronized php-1.27.0-wmf.14/extensions/wikihiero/: Revert "Compress PNGs with zopflipng" - https://gerrit.wikimedia.org/r/273661 (duration: 00m 46s)
  • 22:39 legoktm: legoktm just deployed https://gerrit.wikimedia.org/r/#/c/273655/ (MF cache pollution debug log) for T124356
  • 21:19 urandom: issuing test repair on cerium (restbase staging), wp_parsoid html keyspace, (-w 5) : T108611
  • 20:08 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.14/includes/mime.types: Iec9459c921: Unbreak ULS fonts via wmfstatic (duration: 00m 41s)
  • 19:32 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.14/includes/user/User.php: Revert I43cde3a48: Prevent duplicate memcached lookups for user record (T128246) (duration: 00m 56s)
  • 04:15 ori: On silver: commented out exception from I3eae28a58 to work around T128246

2016-02-26

  • 18:41 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.14/includes/user/User.php: I43cde3a48: Prevent duplicate memcached lookups for user record (duration: 01m 02s)
  • 16:46 chasemp: labstore1001 'mdadm --manage /dev/md126 --add /dev/sdaf'
  • 15:51 jynus: shutting down mariadb on db2030 to clone from db1009
  • 15:24 urandom: forcing puppet run on restbase1009.eqiad.wmnet
  • 15:24 urandom: re-enabling puppet on restbase1009.eqiad.wmnet
  • 15:20 jynus: performing backup of m5-master mysql data
  • 15:17 urandom: blocking CQL native port on restbase1009.eqiad.wmnet : https://phabricator.wikimedia.org/P2677
  • 15:14 urandom: disabling puppet on restbase1009.eqiad to preserve local changes during a quick experiment
  • 15:03 hashar: Switched MediaWiki core npm test to Nodepool instance T119143
  • 13:59 logmsgbot: krinkle@tin Synchronized wmf-config/InitialiseSettings.php: T99096: Enable wmgUseWmfstatic on remaining wikis (duration: 00m 50s)
  • 13:54 moritzm: rebooting lithium for kernel update
  • 13:27 godog: launch swiftrepl continuous replication for unsharded containers on ms-fe1003 T128096
  • 12:31 elukey: added mc1017/mc1018 back to the redis/memcached pools after maintenance
  • 11:42 godog: run swiftrepl eqiad -> codfw for unsharded containers
  • 11:01 elukey: removed mc1018/1017 from the redis memcached pools for maintenance
  • 09:46 elukey: mc1016.eqiad re-added to the memcached/redis pools after maintenance
  • 08:12 elukey: removed mc1016.eqiad from the redis/memcached pools for maintenance
  • 08:01 moritzm: blacklisting aufs kernel module
  • 02:32 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Feb 26 02:32:19 UTC 2016 (duration 7m 42s)
  • 02:24 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.14) (duration: 10m 34s)
  • 01:06 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Lower survey rate again (duration: 01m 05s)
  • 00:33 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.14/extensions/MobileFrontend/: SWAT (duration: 01m 05s)
  • 00:31 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Raise file upload limit to 2047MB (duration: 01m 02s)
  • 00:22 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Add plumbing for wmgUseGraphWithJsonNamespace (duration: 01m 03s)
  • 00:21 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Add wmgUseGraphWithJsonNamespace (duration: 01m 04s)
  • 00:15 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Run reader segmentation survey at 1:500 to test DNT (duration: 01m 21s)

2016-02-25

  • 23:16 bblack: turning puppet back on for cp*, pushing changes through https://gerrit.wikimedia.org/r/#/c/273385/
  • 22:33 bblack: disabling puppet on caches for more scary VCL merges
  • 22:33 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.14/extensions/CentralAuth: I2cfcbf98f3: Reduce memcache traffic for central session storage (duration: 01m 21s)
  • 22:06 bblack: turning puppet back on for cp*, pushing changes through https://gerrit.wikimedia.org/r/273217 to all
  • 22:02 ejegg: set payments wiki to maintenance mode
  • 22:02 ejegg: shut down CiviCRM queue consumer jobs
  • 22:02 ejegg: shut down fundraising campaigns for queue server reboot
  • 21:59 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.14/extensions/CirrusSearch/includes/InterwikiSearcher.php: Fix undefined variable $term in InterwikiSearcher gerrit:273369 (duration: 01m 08s)
  • 21:49 ori: Upgraded Grafana to v3.0.0-pre1.
  • 21:28 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.27.0-wmf.14
  • 21:21 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.14/extensions/CirrusSearch/includes/Searcher.php: Update file that wasnt synced properly (duration: 01m 50s)
  • 20:58 ejegg: updated SmashPig from 5ef60d1fb73d70b7b1501bc97273505c2d625159 to e6d5c292352156d076ebb089c7e55c9d952b1149
  • 20:57 bblack: disabling puppet on caches for scarier VCL merges
  • 20:46 urandom: starting bootstrap of restbase1008-a T119935
  • 20:31 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: rollback wmf.14
  • 20:05 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.27.0-wmf.14
  • 19:13 elukey: added mc1015.eqiad.wmnet back to the redis/memcached pools
  • 18:58 jynus: testing schema changes on db2057:testwiki (s3)
  • 18:07 jynus: testing schema change on db2070 (enwiki)
  • 18:05 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.13/includes/session/SessionBackend.php: I43cde3a48: Revert Revert SessionBackend: skip isUserSessionPrevented check for anons (duration: 01m 38s)
  • 18:02 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.14/includes/session/SessionBackend.php: I43cde3a48: Revert Revert SessionBackend: skip isUserSessionPrevented check for anons (duration: 01m 39s)
  • 17:59 godog: bounce cassandra instances on restbase2001, cql not listening
  • 17:41 elukey: removed mc1015 from the redis/memcached pool for maintenance
  • 17:38 logmsgbot: thcipriani@tin Synchronized README: test sync-file (duration: 01m 42s)
  • 17:34 logmsgbot: thcipriani@tin Synchronized README: test sync-file (duration: 01m 46s)
  • 17:14 moritzm: installing xerces-c security updates
  • 17:06 jynus: applying schema change to flowdb (x1)
  • 17:01 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.13/extensions/QuickSurveys/resources/ext.quicksurveys.init/init.js: SWAT: Do not show a survey if DNT is enabled gerrit:273268 (duration: 01m 31s)
  • 16:58 jynus: applying schema change to officewiki:flow (s3)
  • 16:52 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.14/extensions/QuickSurveys/resources/ext.quicksurveys.init/init.js: SWAT: Do not show a survey if DNT is enabled gerrit:273267 (duration: 01m 35s)
  • 16:25 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: SWAT: wikidatawiki to wmf.14
  • 16:19 elukey: added mc1014 back to the redis/memcached pool after maintenance
  • 16:17 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.14/resources/lib/oojs-ui/oojs-ui-core.js: SWAT: OOjs UI: Fix #gatherPreInfuseState called incorrectly, causing TypeErrors gerrit:273250 (duration: 01m 42s)
  • 15:33 godog: stop cassandra/restbase on restbase2001 to finish raid0 grow
  • 15:23 logmsgbot: demon@tin rebuilt wikiversions.php and synchronized wikiversions files: wikidata back to wmf.13 for now
  • 11:47 hashar: Reverting session manager cherry picks from wmf branches ( https://gerrit.wikimedia.org/r/#/c/273201/ and https://gerrit.wikimedia.org/r/#/c/273202/ ) they have not been deployed after they got merged
  • 11:33 godog: depool ms-fe1004 for trusty dist-upgrade
  • 11:23 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1021 and db1024 (duration: 01m 45s)
  • 10:58 moritzm: powercycling cp2010
  • 10:41 godog: set xff to 0.01 for graphite metrics swift.*.containers (was 0.5)
  • 10:03 hashar: starting Jenkins
  • 09:57 hashar: Stopping Jenkins
  • 09:29 elukey: removed mc1014.eqiad from the redis/memcached pool for maintenance
  • 04:44 urandom: decommissioning Cassandra on restbase1008-a.eqiad.wmnet T119935
  • 04:35 urandom: restarting restbase1008-a to cancel rebuild T108611 T119935
  • 03:19 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Feb 25 03:19:30 UTC 2016 (duration 9m 6s)
  • 03:10 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.14) (duration: 18m 00s)
  • 02:35 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.13) (duration: 19m 03s)
  • 00:57 bd808: Started crashed Logstash process on logstash1002 (systemd doesn't restart authomatically due to T127677)
  • 00:03 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I4cc836f3ca: Fully-qualify EventLoggingBaseUri (duration: 01m 40s)
  • 00:01 logmsgbot: ori@tin Synchronized wmf-config/StartProfiler.php: I016e23d81: xhgui: Sample fewer requests (1:100k instead of 1:10k) (duration: 01m 58s)

2016-02-24

  • 22:49 gehel: reboot logstash1005 for kernel and elasticsearch update
  • 22:29 logmsgbot: demon@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.14 too
  • 22:09 gehel: reboot logstash1006 for kernel and elasticsearch update
  • 22:07 logmsgbot: demon@tin Finished scap: group0 to wmf.14 (duration: 47m 50s)
  • 21:19 logmsgbot: demon@tin Started scap: group0 to wmf.14
  • 21:19 subbu: finished deploying parsoid version 581a43c75
  • 21:08 subbu: synced code; restarted parsoid on wtp1001 as a canary
  • 21:01 subbu: starting parsoid deploy
  • 20:49 moritzm: reboot logstash1004 for kernel/elasticsearch update
  • 20:47 ejegg: updated payments-wiki from fdea9fa9a5951d7ae57bb4d54aa9374f236638d6 to 111f92133cbd6b3890f53e29831ac022d3c26f51
  • 20:39 gehel: reboot logstash1003 for kernel and elasticsearch update
  • 20:28 gehel: reboot logstash1002 for kernel and elasticsearch update
  • 20:15 chasemp: reboot labstore1002 to ensure io scheduler grub options work
  • 20:13 moritzm: reboot logstash1001 for kernel update
  • 19:58 ejegg: rolled back payments-wiki to fdea9fa9a5951d7ae57bb4d54aa9374f236638d6 over a missing image :P
  • 19:56 ejegg: updated payments-wiki from fdea9fa9a5951d7ae57bb4d54aa9374f236638d6 to 8fad21b940e59b30ca5382a4b7b705f90404b337
  • 19:46 chasemp: runonce apply for https://gerrit.wikimedia.org/r/#/c/272891/ for labs vm's (only affects nfs clients)
  • 19:46 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/273032 (duration: 01m 41s)
  • 19:41 cmjohnson1: db1021 replacing disk 8
  • 19:04 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/273017 (duration: 01m 37s)
  • 18:52 papaul: es201[1-9] -signing puppet certs, salt-key. initial run
  • 18:39 mutante: restart gitblit
  • 18:10 bblack: disabling nginx keepalives on remaining clusters (upload, misc, maps)
  • 18:07 ori: hafnium did not have enough disk space for mongo to execute db.repairDatabase(), which is necessary for reclaiming disk space. Since existing profile data can be tossed, ran `db.dropDatabase(); db.repairDatabase();`. Need to think this through better, obviously.
  • 18:02 ori: mongodb on hafnium: ran `db.results.remove( { "meta.SERVER.REQUEST_URI": "/wiki/Special:BlankPage" } ); db.repairDatabase();` to drop profiles of PyBal requests and compact the database.
  • 17:44 logmsgbot: demon@tin Synchronized wmf-config/: poolcounter config simplification (duration: 01m 39s)
  • 17:21 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: Re-apply "Set $wgResourceBasePath to /w for medium wikis" (duration: 01m 42s)
  • 17:16 logmsgbot: demon@tin Synchronized wmf-config/: service entries for initialisesettings + fix (duration: 01m 45s)
  • 16:59 papaul: es201[1-9] disabling /revoking puppet and salt keys for re-image
  • 16:57 papaul: es200[1-9] disabling /revoking puppet and salt keys for re-image
  • 16:53 bd808: https://wmflabs.org/sal/production missing SAL data since 2016-02-21T14:39 due to bot crash; needs to be backfilled from wikitech data
  • 16:41 _joe_: started nutcracker on mw1099
  • 16:39 bblack: +do_gzip done for all cache_text
  • 16:38 logmsgbot: demon@tin Synchronized wmf-config/: Rationalize services definitions for labs too. (duration: 01m 45s)
  • 16:16 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: Don't yet allow wikidatasparql graph urls (duration: 01m 37s)
  • 16:12 logmsgbot: demon@tin Synchronized wmf-config/throttle.php: New throttle settings for Edit-a-thon workshop for orwiki (urgent) (duration: 01m 29s)
  • 16:09 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: Revert "Set $wgResourceBasePath to "/w" for medium wikis" (duration: 01m 30s)
  • 15:20 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.13/extensions/wikihiero: Ia0990f5f (duration: 01m 33s)
  • 15:18 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.14/extensions/wikihiero: Ia0990f5f (duration: 01m 33s)
  • 15:15 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.13/includes/OutputPage.php: Iad94bb2 (duration: 01m 50s)
  • 15:13 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.14/includes/OutputPage.php: Iad94bb2 (duration: 01m 43s)
  • 14:55 godog: nodetool-a repair -pr on restbase1008 T108611
  • 14:46 bblack: cache_text: -do_gzip experiment live on all
  • 14:43 godog: bump max reconstruction speed on restbase2001 to 20000 T127951
  • 14:08 logmsgbot: krinkle@tin Synchronized wmf-config/InitialiseSettings.php: T99096: Enable wmfstatic for medium wikis (duration: 01m 40s)
  • 13:46 godog: bump max reconstruction speed on restbase2001 to 12000 T127951
  • 13:15 godog: restart cassandra on restbase2001, throttle raid rebuild speed to 8MB/s
  • 11:06 godog: reboot restbase2001
  • 11:01 godog: mdadm errors on restbase2001 while growing the raid0, load increasing
  • 10:53 godog: grow restbase2001 raid0 to include a 5th disk
  • 08:42 moritzm: installing libssh2 security updates across the cluster
  • 05:46 logmsgbot: demon@tin Synchronized docroot/: removing skel-1.5 symlinks (duration: 01m 41s)
  • 05:37 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: import meta to wikitechwiki (duration: 01m 45s)
  • 05:28 twentyafterfour: applied https://secure.phabricator.com/rP03d6e7f1b699d89c829e92ba0da2178b41ad1d6a on iridium to fix visibility on pastes
  • 05:11 ori: Restarting HHVM on codfw app servers to make sure they pick a file-scope change to stop profiling PyBal health-checks
  • 05:04 logmsgbot: ori@mira Synchronized wmf-config/StartProfiler.php: I0e7be0b5: Never profile PyBal health-checks (duration: 03m 12s)
  • 03:19 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Feb 24 03:19:41 UTC 2016 (duration 8m 46s)
  • 03:10 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.14) (duration: 18m 05s)
  • 02:40 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Add default to fix notices about wmgUseFlow (duration: 01m 36s)
  • 02:37 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.13/extensions/Echo: SWAT (duration: 01m 40s)
  • 02:35 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.14/extensions/Echo: SWAT (duration: 01m 42s)
  • 02:33 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.13) (duration: 13m 47s)
  • 01:50 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Use Flow dblists for deciding which wikis have Flow (duration: 01m 38s)
  • 01:47 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Recognize new Flow dblist (duration: 01m 35s)
  • 01:38 logmsgbot: catrope@tin Synchronized dblists/: Add new dblists for Flow (duration: 01m 32s)
  • 01:34 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Use project logos for welcome notifications (duration: 01m 34s)
  • 01:32 logmsgbot: catrope@tin Synchronized w/static/images: Add project logos for Echo (duration: 01m 33s)
  • 01:24 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/272889/ (duration: 01m 35s)
  • 01:21 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/272837/ (duration: 01m 35s)
  • 01:13 logmsgbot: aaron@tin Synchronized wmf-config/filebackend-production.php: Enable async secondary swift writes for non-"big" wikis (duration: 01m 31s)
  • 00:40 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.14/extensions/CentralAuth: SessionManager backports (duration: 01m 39s)
  • 00:37 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.14/extensions/WikimediaMessages: SessionManager backports (duration: 01m 37s)
  • 00:35 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.14/includes: SessionManager backports (duration: 02m 17s)
  • 00:00 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.13/includes: Backport MW_NO_SESSION changes (duration: 02m 13s)

2016-02-23

  • 23:57 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.13/autoload.php: (no message) (duration: 01m 42s)
  • 23:22 logmsgbot: krinkle@tin Synchronized w/static.php: (no message) (duration: 01m 35s)
  • 23:08 logmsgbot: aude@tin Synchronized wmf-config/Wikibase.php: Bump cache epoch for Wikidata for Wikiversity sitelinks section (duration: 01m 32s)
  • 22:36 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Add Wikibase settings for Wikiversity (duration: 01m 31s)
  • 22:30 logmsgbot: aude@tin Synchronized wmf-config/Wikibase.php: Add Wikiversity site link section to Wikidata (duration: 02m 23s)
  • 22:23 logmsgbot: aude@tin Finished scap: Add Wikidata i18n messages to WikimediaMessages, update Wikibase on wmf14, and some core backports (duration: 51m 45s)
  • 21:32 chasemp: reboot mw1114 via mgmt as unresponsive
  • 21:31 logmsgbot: aude@tin Started scap: Add Wikidata i18n messages to WikimediaMessages, update Wikibase on wmf14, and some core backports
  • 21:25 ejegg: updated SmashPig from e106af8d440b461779cca60d55efba33dbfa7518 to 5ef60d1fb73d70b7b1501bc97273505c2d625159
  • 21:01 ejegg: updated SmashPig from d2083b28c28bd6afe61d30fc913fa0edd4203f82 to e106af8d440b461779cca60d55efba33dbfa7518
  • 20:50 urandom: initiating `nodetool upgradesstables -a' in staging to rewrite sstables (restored earlier compression settings)
  • 20:26 logmsgbot: aude@tin Synchronized php-1.27.0-wmf.13/includes/session/CookieSessionProvider.php: Fix invalid key warning (duration: 01m 32s)
  • 19:38 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/272799 (duration: 01m 37s)
  • 19:02 yuvipanda: download and decompress enwiki sessions dataset on labsdb1001
  • 18:59 ejegg: updated SmashPig from c34a5fc3cfbb16d6fb019bea9a3d8d73eeb73f7b to d2083b28c28bd6afe61d30fc913fa0edd4203f82
  • 18:57 cmjohnson1: db1021 replacing disk 7
  • 18:34 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: un-rv (duration: 01m 37s)
  • 18:23 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: rv (duration: 01m 32s)
  • 18:15 robh: disabled puppet on dataset1001
  • 18:07 ejegg: updated SmashPig from 97629339994bffe8831a9067f5e9c21fa423586b to c34a5fc3cfbb16d6fb019bea9a3d8d73eeb73f7b
  • 18:05 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/272778/ (duration: 01m 31s)
  • 17:34 logmsgbot: volans@tin Synchronized wmf-config/db-eqiad.php: Repool es1016 T127330 (duration: 01m 39s)
  • 17:29 godog: bootstrap restbase1008-b T119935
  • 17:16 bblack: depool cp1061 (eqiad upload) - T125486
  • 17:02 ottomata: rebooting analytics1027 for kernel upgrae
  • 16:58 jynus: stopping again db2035 replication
  • 16:49 godog: grow raid0 on restbase1009 T119935
  • 16:23 godog: depool restbase1007 and shut for ram upgrade
  • 16:20 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable uploader group at Simple English Wikipedia gerrit:272744 (duration: 01m 32s)
  • 16:16 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: VisualEditor: Switch to Single Edit Tab mode on Hungarian Wikipedia gerrit:270346 (duration: 01m 32s)
  • 16:11 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable the structured language overlay and increase the instrumentation rate gerrit:271264 (duration: 01m 32s)
  • 16:00 godog: shut restbase1009 for cpu/mem upgrade
  • 15:37 jynus: stopping db2035 replication to test no paging
  • 15:22 godog: halt restbase1008 for cpu/mem upgrade
  • 15:20 ottomata: shutting down analytics (hadoop) cluster for CDH 5.5 upgrade
  • 15:10 andrewbogott: restarted pdns on labservices1001
  • 14:46 jynus: testing database paging changes by stopping mysql slave on db1021 (it should page dbas)
  • 14:44 volans: Run sysbench on es1016 (already depooled) T127330
  • 14:43 bblack: depool cp1051 (upload eqiad) - T125486
  • 14:27 moritzm: fixed apt config on es2011 (also affected by T125044)
  • 14:23 logmsgbot: volans@tin Synchronized wmf-config/db-eqiad.php: Depool to compare RAID perfs with new es201* T127330 (duration: 01m 42s)
  • 13:45 gehel: elasticsearch 1.7.5 now available on apt repository
  • 13:40 gehel: updating reprepro configuration on carbon.eqiad.wmnet to include elasticsearch 1.7 repo
  • 12:38 moritzm: installing cpio security updates
  • 11:27 ema: re-enabling puppet agent on mc2016
  • 11:15 ema: re-enabling puppet agent on scandium
  • 11:08 volans: Starting MariaDB on es2011 (not yet in production) [ T127330 ]
  • 10:46 volans: Executing /opt/wmf-mariadb10/install on not-yet-production es2011-es2019 [T127330]
  • 08:30 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Depool es2010 (duration: 01m 34s)
  • 05:25 ori: StartProfiler.php sync was of Ic952fab90f: xhgui: profile 1:10,000 requests
  • 05:20 logmsgbot: ori@tin Synchronized wmf-config/StartProfiler.php: (no message) (duration: 01m 46s)
  • 05:06 ori: Deleted gdash docroot on graphite2001 and krypton
  • 04:05 bblack: parsoid-lb.eqiad.wikimedia.org turned off
  • 03:03 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Feb 23 03:03:06 UTC 2016 (duration 9m 13s)
  • 02:53 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.14) (duration: 11m 44s)
  • 02:29 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.13) (duration: 13m 34s)
  • 02:27 tgr: switching mw1017 to wmf.12 for perf tests
  • 01:28 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.14/extensions/VisualEditor: SWAT (duration: 01m 34s)
  • 01:07 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Fix wgReferrerPolicy (duration: 01m 36s)
  • 01:04 ori: ran `mwscript namespaceDupes.php --wiki=labswiki` on silver
  • 01:00 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Add Tool namespace on wikitech (duration: 01m 37s)
  • 00:52 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Freeze LQT on fiwikimedia (duration: 01m 39s)
  • 00:10 ori: Depooling mw1099 for debugging
  • 00:07 eileen: Updating civicrm from 07f8ea80d3fee19d4977d673a719f5a692ffbded to 181ef79fd1360ca337990b6768ebc10b1185a431

2016-02-22

  • 23:04 logmsgbot: krinkle@tin Synchronized w/static.php: Set MW_NO_SESSION to warn (duration: 01m 34s)
  • 22:42 jynus: powercycling es2010
  • 22:14 logmsgbot: ori@tin Finished scap: I1d4f90533: Third time's the charm: kill live-1.5 (duration: 11m 05s)
  • 22:03 logmsgbot: ori@tin Started scap: I1d4f90533: Third time's the charm: kill live-1.5
  • 22:02 logmsgbot: ori@tin Synchronized w: I1d4f90533: Third time's the charm: kill live-1.5 (duration: 01m 43s)
  • 21:40 logmsgbot: demon@tin Synchronized README: no-op for co-master sync (duration: 01m 31s)
  • 21:34 yurik: deployed graphoid https://gerrit.wikimedia.org/r/#/c/272602/
  • 21:29 ostriches: undid previous scap with dsh, live-1.5 is still used.
  • 21:28 logmsgbot: demon@tin Finished scap: removing live-1.5 symlink (duration: 13m 03s)
  • 21:26 ori: ran sync-common on silver
  • 21:15 logmsgbot: demon@tin Started scap: removing live-1.5 symlink
  • 21:05 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.13/extensions/Graph/: Graph ext https://gerrit.wikimedia.org/r/#/c/272473/ (duration: 01m 43s)
  • 20:59 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.14/extensions/Graph/lib/graph2.compiled.js: https://gerrit.wikimedia.org/r/#/c/272472/ (duration: 01m 40s)
  • 20:50 yurik: deployed and restarted kartotherian - https://gerrit.wikimedia.org/r/#/c/272425/
  • 20:33 ejegg: restarted antifraud queue consumer
  • 20:29 ejegg: update civicrm from 352bd7ce9abc4d3ce954e978aa333e6ec4a59849 to 07f8ea80d3fee19d4977d673a719f5a692ffbded
  • 20:28 ejegg: paused antifraud queue consumer
  • 18:32 logmsgbot: krinkle@tin Synchronized wmf-config/InitialiseSettings.php: T99096: Enable wmfstatic for small wikis (duration: 01m 43s)
  • 18:04 moritzm: corrected dpkg installation status for cassandra on restbase-test200[12]
  • 18:03 logmsgbot: hoo@tin Synchronized php-1.27.0-wmf.13/extensions/Wikidata: Reset entity access counts between parser runs (duration: 03m 01s)
  • 16:52 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-production.php: Cache morelike search queries for 24h (duration: 01m 34s)
  • 16:34 godog: reboot krypton.eqiad.wmnet, no answer to gnt-instance console / no ssh
  • 16:25 logmsgbot: demon@tin Synchronized README: no-op for co-master sync (duration: 01m 29s)
  • 16:20 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: azwiki wants some wikilove too (duration: 01m 29s)
  • 16:16 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: dpl on nowikimedia (duration: 01m 29s)
  • 16:13 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: default referrer policy (duration: 01m 31s)
  • 16:09 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: Translation Notifications on commons (duration: 01m 31s)
  • 16:07 logmsgbot: demon@tin Synchronized wmf-config/throttle.php: throttle exemption for or.wikipedia workshop (duration: 01m 32s)
  • 16:02 logmsgbot: volans@tin Synchronized wmf-config/db-codfw.php: Repool of es2001 (duration: 01m 39s)
  • 16:01 volans: repooled es2001 [ T127330 ]
  • 15:39 ottomata: stopping restbase on aqs1001
  • 15:26 ottomata: stopping puppet on aqs* for scap deployment
  • 13:50 logmsgbot: hoo@tin Synchronized wmf-config/Wikibase.php: Bump $wgCacheEpoch on Wikidata after Property conversions (duration: 01m 39s)
  • 13:31 _joe_: restarting ocg on ocg1001, got stuck after redis restart on rdb1002
  • 12:23 moritzm: restarting apache on neon for glibc update
  • 11:09 moritzm: restarting apache on graphite1001 for glibc update
  • 10:53 moritzm: started salt-minion on scandium (process had died)
  • 04:00 logmsgbot: ori@tin Synchronized wmf-config/StartProfiler.php: Ie4c87619: xhgui: sanitize query string & I219c0901: xhgui: sanitize keys (duration: 01m 45s)
  • 03:03 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Feb 22 03:03:21 UTC 2016 (duration 8m 50s)
  • 02:54 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.14) (duration: 12m 05s)
  • 02:29 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.13) (duration: 13m 28s)
  • 02:13 bd808: Logstash process on logstash1002 died from jvm OOM

2016-02-21

  • 14:39 jynus: restarting db2070 to test configuration change
  • 03:02 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Feb 21 03:02:48 UTC 2016 (duration 8m 47s)
  • 02:54 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.14) (duration: 11m 34s)
  • 02:37 logmsgbot: krinkle@tin Synchronized wmf-config/InitialiseSettings.php: T107395 (duration: 01m 29s)
  • 02:35 logmsgbot: krinkle@tin Synchronized images: clean up (duration: 01m 32s)
  • 02:33 logmsgbot: krinkle@tin Synchronized w/static/images: clean up (duration: 01m 32s)
  • 02:30 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.13) (duration: 13m 20s)

2016-02-20

  • 19:39 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.13/extensions/MobileFrontend/includes/MobileFrontend.hooks.php: Remove live-hacked logging code for T124356 (duration: 01m 42s)
  • 19:37 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.13/extensions/MultimediaViewer/extension.json: (no message) (duration: 01m 36s)
  • 12:56 jynus: disabling ipmi_si kernel module on labsdb* hosts for high cpu usage
  • 08:08 _joe_: gnt-console reboot of alsafi
  • 05:43 logmsgbot: ori@tin Synchronized wmf-config/StartProfiler.php: I14612374: Send xhprof profiles from mw1017 to xhgui (duration: 01m 38s)
  • 03:10 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Feb 20 03:10:55 UTC 2016 (duration 8m 53s)
  • 03:02 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.14) (duration: 10m 48s)
  • 02:33 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.13) (duration: 13m 16s)
  • 00:45 logmsgbot: ori@tin Finished scap: I2a66b40e4c6: add a dependency on xhprof/xhgui (duration: 50m 12s)
  • 00:32 ori: Restarted HHVM on mw1248 (locked up, T89912).

2016-02-19

  • 23:55 logmsgbot: ori@tin Started scap: I2a66b40e4c6: add a dependency on xhprof/xhgui
  • 23:43 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Enable SecurePoll poll creation on officewiki (duration: 02m 21s)
  • 23:11 logmsgbot: ori@mira scap failed: ValueError /srv/mediawiki-staging/multiversion/vendor/slim/slim/tests/templates/test.php has content before opening <?php tag (duration: 00m 17s)
  • 23:11 logmsgbot: ori@mira Started scap: I2a66b40e4c6: add a dependency on xhprof/xhgui
  • 20:23 chasemp: reboot labvirt1002
  • 17:01 andrewbogott: reenabling puppet on labservices1001 and restoring original dns settings, pending merge of https://gerrit.wikimedia.org/r/#/c/271797/
  • 16:55 chasemp: reboot labvirt1011
  • 16:09 elukey: rebooted kafka2001.codfw.wmnet for kernel upgrade
  • 15:59 elukey: rebooted kafka2002.codfw for kernel upgrade
  • 15:59 moritzm: rebooting osmium
  • 15:49 elukey: added kafka1002 back to eventbus pool via confctl
  • 15:48 moritzm: rolling restart of maps cluster for glibc update
  • 15:42 elukey: removed kafka1002.eqiad.wmnet from eventbus pool via confctl
  • 15:38 elukey: re-added kafka1001.eqiad.wmnet back to eventbus' pool via confctl
  • 15:34 elukey: rebooting kafka1001 for kernel upgrade
  • 15:29 elukey: removed kafka1001 from eventbus via confctl
  • 15:14 andrewbogott: restarting pdns on labservices1001 again
  • 15:02 andrewbogott: restarting pdns on labservices1001
  • 15:00 paravoid: setting up (e)BGP sessions between ulsfo-codfw
  • 14:52 apergos: labstore1001 issues were (again) cluebot writing to its error log. I chowned that log to root and left a README file in the directory with an explanation plus a pointer to us here if they have questions/need help.
  • 14:48 moritzm: rolling restart of swift-proxy in eqiad
  • 14:27 volans: Restarting MariaDB on es2001 (still depooled) [T127330]
  • 14:27 paravoid: cr2-knams: re-activating BGP with 1257
  • 14:12 jynus: restarting mysql at dbstore1001
  • 13:59 moritzm: restarting salt-master on neodymium
  • 13:49 elukey: puppet re-enabled on analytics1027
  • 13:37 moritzm: rebooting mira
  • 13:06 moritzm: restarting slapd on dubnium/pollux
  • 12:48 moritzm: restarting apache on uranium
  • 12:44 elukey: puppet stopped on analytics1027 for issues with the cluster
  • 11:40 godog: start a new ^global- swift container replication eqiad -> codfw
  • 11:39 akosiaris: reboot cygnus, stuck in 100% IOwait
  • 11:13 moritzm: rolling restart of aqs cluster for glibc update
  • 11:08 mark: correction: Reduced sync_speed_max to 100000 (half) for md125 on labstore1001
  • 11:08 mark: Reduced sync_speed_max to 100000 (half) for md126 on labstore1001
  • 11:02 moritzm: restarted gerrit on ytterbium (actual restart happened ten minutes earlier than this log entry, though)
  • 10:22 moritzm: rolling restart of cassandra on restbase/eqiad
  • 10:05 jynus: restarting db021 slave, will be testing/depooled for a while
  • 10:00 jynus: purging requested rows on eventlogging Edit table (m4) at db1046, db1047, dbstore1002 and dbstore2002
  • 09:59 moritzm: rolling restart of cassandra on restbase/codfw
  • 09:04 _joe_: idled the raid check for md123 on labstore1001
  • 08:51 _joe_: killed the backup rsync on labstore1001 to alleviate the high io load
  • 03:00 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Feb 19 03:00:47 UTC 2016 (duration 9m 10s)
  • 02:51 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.14) (duration: 11m 18s)
  • 02:43 mutante: powercycle cp2017
  • 02:29 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.13) (duration: 13m 18s)
  • 02:07 logmsgbot: krinkle@tin Synchronized wmf-config/InitialiseSettings.php: Clean up wmgUseWmfstatic setting (duration: 01m 34s)
  • 02:04 logmsgbot: krinkle@tin Synchronized wmf-config/CommonSettings.php: Clean up wmgUseWmfstatic setting (duration: 01m 56s)
  • 01:02 mobrovac: restbase deploy end of fa1207e95
  • 00:55 mobrovac: restbase deploy start of fa1207e95 for T127370
  • 00:55 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Enable cross-wiki notifications beta feature on initial set of wikis (duration: 01m 40s)
  • 00:55 mobrovac: restbase deploy start of fa1207e95
  • 00:49 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.13/extensions/Echo/: SWAT (duration: 01m 41s)
  • 00:44 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.14/extensions/Echo/: SWAT (duration: 01m 37s)
  • 00:41 logmsgbot: catrope@tin Synchronized wmf-config/filebackend-production.php: Enable deferred writes to codfw swift cluster (duration: 01m 33s)
  • 00:37 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Set initial $wgMaxUserDBWriteDuration value (duration: 01m 38s)
  • 00:28 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: VE config var cleanup (2/2) (duration: 01m 38s)
  • 00:26 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: VE config var cleanup (1/2) (duration: 01m 44s)

2016-02-18

  • 23:49 csteipp: deployed patch for T125290
  • 22:22 chasemp: reboot labvirt1010
  • 22:18 logmsgbot: ori@mira Synchronized php-1.27.0-wmf.13/extensions/SpamBlacklist: I0ad5289324: Pre-cache the link list for external link filters (duration: 02m 26s)
  • 22:01 chasemp: reboot labvirt1009
  • 21:59 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.13/includes/rcfeed/IRCColourfulRCFeedFormatter.php: T127360: Ignore RC_CATEGORIZE (duration: 01m 41s)
  • 21:59 volans: Shutting down MySQL on es2001 (depooled) and starting data trasnfer to es2011 [ T127330 ]
  • 21:32 chasemp: reboot labvirt1008
  • 21:15 logmsgbot: volans@tin Synchronized wmf-config/db-codfw.php: depool es2001 (duration: 01m 36s)
  • 20:52 volans: Depool es2001 to copy the data to es2011 (T127330)
  • 20:41 chasemp: reboot labvirt1006
  • 20:39 logmsgbot: krinkle@tin Synchronized w/static.php: Set MW_NO_SESSION (duration: 01m 55s)
  • 20:21 urandom: restarted restbase in staging (5 min. delay)
  • 20:16 chasemp: reboot labvirt1005
  • 20:00 ejegg: updated payments-wiki from 5b909f06acce6444186ac02a494439c2ddd624aa to fdea9fa9a5951d7ae57bb4d54aa9374f236638d6
  • 19:56 ejegg: rolled back payments-wiki to 5b909f06acce6444186ac02a494439c2ddd624aa
  • 19:55 ejegg: updated payments-wiki from 5b909f06acce6444186ac02a494439c2ddd624aa to fdea9fa9a5951d7ae57bb4d54aa9374f236638d6
  • 19:53 urandom: upgrade Cassandra to 2.1.13 on restbase-test200[1-3].codfw.wmnet (restbase staging) : T126629
  • 19:50 chasemp: reboot labvirt1004
  • 19:40 paravoid: cr2-knams: deactivating BGP with 1257
  • 19:21 chasemp: reboot labvirt1003
  • 19:06 moritzm: restarting apache on bohrium
  • 19:04 subbu: finished deploying parsoid dfbafb60
  • 18:58 subbu: synced code; restarted parsoid on wtp1001 as a canary
  • 18:56 logmsgbot: ori@mira Synchronized wmf-config/mobile.php: Ib6fff26be162: Modify $wgRenderHashAppend when disabling responsive images on mobile (duration: 02m 24s)
  • 18:56 subbu: starting parsoid deploy
  • 18:53 twentyafterfour: tagged phabricator hotfixes as release/2016-02-18/2 in the phabricator/phabricator repository. This includes fixes T127290 and T127349
  • 18:50 logmsgbot: ori@mira Synchronized wmf-config/InitialiseSettings.php: I02dbbdb79ea9a: Re-set default vega version to 2 (duration: 03m 24s)
  • 18:49 jynus: stopping, upgrading and reconfiguring dbstore1001
  • 18:24 chasemp: rebooting labvirt1001
  • 18:22 yurik: deployed graphoid https://gerrit.wikimedia.org/r/#/c/271563/
  • 18:17 elukey: re-enabled puppet on analytics1027 after maintenance
  • 18:06 hoo: Updated Wikidata's property suggester with data from Monday's json dump
  • 18:04 mobrovac: restbase deploy end of a42976cc82
  • 18:03 twentyafterfour: applied a hotfix from https://secure.phabricator.com/D15306 on iridium to test a fix for https://phabricator.wikimedia.org/T127290
  • 18:00 godog: reenable puppet on restbase1008
  • 17:49 mobrovac: restbase deploy start of a42976cc82
  • 17:47 elukey: manual failover of hadoop master node (analytics1001) to secondary (analytics1002) for maintenance (plus service restarts)
  • 17:41 urandom: upgrading Cassandra to 2.1.13 on cerium.eqiad.wmnet (restbase staging) T126629
  • 17:28 mobrovac: restbase deploying a42976cc82 to restbase1002
  • 17:27 urandom: Cassandra on xenon.eqiad.wmnet killed by kernel after Cassandra package upgrade (coincidence?): [1482254.046078] Out of memory: Kill process 21854 (java) score 595 or sacrifice child : T126629
  • 17:26 urandom: Cassandra on xenon.eqiad.wmnet killed by kernel after Cassandra package upgrade (coincidence): [1482254.046078] Out of memory: Kill process 21854 (java) score 595 or sacrifice child
  • 17:22 urandom: upgrading Cassandra to 2.1.13 on xenon.eqiad.wmnet (restbase staging) T126629
  • 17:20 elukey: disabled puppet on analytics1027 to avoid any Camus job to run
  • 17:04 dcausse: updating completion suggester indices in eqiad
  • 16:54 elukey: restarting hadoop services on analytics105* nodes for security updates
  • 16:49 gehel: removing cirrus maintenance crons from mw1152 (T127322)
  • 15:52 dcausse: creating adywiki indices in codfw
  • 15:44 elukey: restarting hadoop services on analytics104* nodes for security updates
  • 15:37 elukey: restarting hadoop services on analytics102* nodes for security update
  • 15:33 moritzm: restarting apache on silver/wikitech
  • 15:10 elukey: restarting hadoop services on analytics103* hosts for security upgrades
  • 14:06 bblack: restarting apache on gallium (integration)
  • 13:13 mark: decreased raid md2 sync_speed_max to 6000 on restbase1008
  • 12:55 elukey: rebooted kafka1022.eqiad.wmnet for kernel upgrade
  • 12:52 godog: decrease raid min_speed to 8000 on restbase1008
  • 12:50 logmsgbot: hoo@tin Synchronized wmf-config/Wikibase.php: Bump $wgCacheEpoch for Wikidata (duration: 01m 54s)
  • 12:41 elukey: rebooted kafka1020 for kernel upgrade.
  • 12:40 godog: decrease raid min_speed to 10000 on restbase1008
  • 12:24 godog: increase stripe_cache_size to 32470 on restbase1008
  • 12:21 godog: expand raid0 on restbase1008 to sdd and sde
  • 11:36 paravoid: upgrading mr1-ulsfo to its pre-recovery version and rebooting (T127295)
  • 11:34 hashar: Hard restarting Jenkins T127294
  • 11:32 jynus: logical import of db1021 starting for data consistency check and defragmenting purposes
  • 11:29 paravoid: mr1-ulsfo: "request system snapshot media internal slice alternate" + reboot (T127295)
  • 11:27 hashar: Jenkins web UI busy with 'jenkins.model.RunIdMigrator doMigrate' while it migrate build records. I did a bunch of cleanup yesterday. Jenkins runs jobs in the background just fine though. T127294
  • 11:12 hashar: Jenkins: reloading configuration from disk. Some metadata are corrupted T127294
  • 10:48 elukey: rebooted kafka1018 for maintenance
  • 10:17 elukey: rebooted kafka1014 for maintenance
  • 10:10 moritzm: restarting hhvm on mw1* to put glibc update into effect
  • 09:49 godog: remove old restbase metrics under restbase.* from graphite1001 and graphite2001
  • 03:13 twentyafterfour: running puppet one last time on iridium. Phabricator upgrade successful with just a few minor issues now resolved.
  • 03:01 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Feb 18 03:01:01 UTC 2016 (duration 9m 24s)
  • 02:51 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.14) (duration: 11m 20s)
  • 02:29 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.13) (duration: 13m 55s)
  • 02:18 twentyafterfour: phabricator is back online, sprint extension is broken, I'm investigating
  • 01:57 mutante: powercycled frozen mw1147
  • 01:51 twentyafterfour: phab pre-upgrade: http://pastebin.com/RTmXfDhp
  • 01:49 twentyafterfour: about to bring down phabricator to do the upgrade
  • 01:49 twentyafterfour: ran puppet on iridium for testing
  • 01:08 twentyafterfour: stopped phd and started dumping phabricator's database to /srv/dumps/20160218.phabricator.sql.gz (just in case I need to roll back the update)
  • 00:34 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.13/extensions/Flow: Trying again (duration: 01m 50s)
  • 00:28 RoanKattouw: 00:28:25 64 apaches had sync errors , /usr/bin/sync-common missing
  • 00:28 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.13/extensions/Flow: SWAT (duration: 02m 06s)
  • 00:18 godog: restart cassandra-a on restbase1008 after extending /srv

2016-02-17

  • 23:53 csteipp: redeployed wmf14 patches
  • 23:30 csteipp: deployed all missing security patches from wmf14
  • 23:10 logmsgbot: csteipp@tin Synchronized php-1.27.0-wmf.14/resources/src/mediawiki/page/patrol.ajax.js: add security patches (duration: 01m 28s)
  • 23:08 logmsgbot: csteipp@tin Synchronized php-1.27.0-wmf.14/includes: add security patches (duration: 01m 35s)
  • 23:03 logmsgbot: ori@mira Synchronized php-1.27.0-wmf.13/extensions/MobileFrontend/includes/MobileFrontend.hooks.php: live-hacked debug logging for T124356 (duration: 02m 16s)
  • 21:42 mobrovac: mathoid deploying ed98ffe9d
  • 21:35 mobrovac: restbase restarted restbase1002 on nodejs v4.3.0
  • 20:40 papaul: es201[1-9] - signing puppet certs, salt-key, initial run
  • 20:25 logmsgbot: krinkle@tin Synchronized wmf-config/CommonSettings.php: Re-enable T99096 for mediawiki.org (duration: 01m 29s)
  • 20:23 logmsgbot: catrope@tin Synchronized docroot/: (no message) (duration: 01m 33s)
  • 19:18 yuvipanda: truncate 1.2T php error log file on labstore1003 from cluebot
  • 18:35 jynus: testing now that alerts still work by stopping db1024 replication (depooled)
  • 18:30 logmsgbot: krinkle@tin Synchronized wmf-config/CommonSettings.php: T127194 (duration: 01m 31s)
  • 18:27 jynus: no issues found with new mysql, lag monitoring, renabling puppet again on the pending eqiad servers
  • 17:49 bblack: restarting pybal on eqiad primary LVS ( lvs100[123] )
  • 17:47 bblack: restarting pybal on codfw primary LVS ( lvs200[123])
  • 17:42 bblack: restarting pybal on ulsfo/esams primary LVS ( lvs[34]00[12])
  • 17:40 bblack: restarting pybal on eqiad backup LVS ( lvs100[456] )
  • 17:38 bblack: restarting pybal on eqiad inactive LVS clusters ( lvs1007-12 )
  • 17:38 bblack: restarting pybal on codfw backup LVS ( lvs200[456] )
  • 17:34 bblack: restarting pybal on ulsfo/esams backup LVS ( lvs[34]00[34])
  • 17:13 hoo: Updated the sites and site_identifiers table for on all non-Wikipedias (including Wikidata)
  • 17:02 ema: depooled ulsfo https://phabricator.wikimedia.org/T127094
  • 16:48 ostriches: purged ancient boardvote gpg key from mediawiki fleet. unused since forever.
  • 16:25 logmsgbot: anomie@tin Synchronized wmf-config/: SWAT: Undeploy Extension:ApiSandbox (duration: 01m 30s)
  • 16:20 logmsgbot: anomie@tin Synchronized wmf-config/CommonSettings.php: SWAT: Remove $wgMWOAuthGrantPermissions (duration: 01m 34s)
  • 16:16 urandom: restbase deploy (15a6c50) complete, sans restbase1008.eqiad.wmnet (down for maintenance during deploy)
  • 16:16 anomie: Ran namespaceDupes.php on tawiki
  • 16:14 urandom: restbase deploy (15a6c50) completet
  • 16:14 hoo: Re-populating the sites and site_identifiers table for all Wikipedias and testwikidata
  • 16:10 urandom: restbase deploy restarting at restbase1009
  • 16:09 logmsgbot: anomie@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: New userrights and configuration for cswiki (task T126931) (duration: 01m 31s)
  • 16:09 urandom: restbase deploy stalled at restbase1008 (under maintenance)
  • 16:05 logmsgbot: anomie@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Namespace aliases for tawiki (task T126604) (duration: 01m 31s)
  • 16:00 urandom: continuing production-wide restbase deploy (15a6c50)
  • 15:58 godog: copy restbase.{v1_*,sys_*,ALL,GET,HEAD,POST,OPTIONS,_robots} to restbase.external on graphite1001 and graphite2001
  • 15:55 godog: copy restbase.private to restbase.internal on graphite1001 and graphite2001
  • 15:53 logmsgbot: aude@tin Synchronized php-1.27.0-wmf.13/extensions/Wikidata: Fix caching data types bug: T127095 (duration: 01m 44s)
  • 15:53 urandom: canary deploy of restbase to restbase1001.eqiad.wmnet (15a6c50) complete
  • 15:53 urandom: canary deploy of restbase to restbase1001.eqiad.wmnet (15a6c50)
  • 15:51 bblack: package upgrades commencing on lvs*
  • 15:43 urandom: restbase staging deploy (15a6c50) complete
  • 15:38 urandom: deploying restbase (15a6c50) in staging
  • 15:33 jynus: stopping puppet on all database hosts (db, dbstore, es, etc.) for lag alert testing
  • 14:05 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1022 after maintenance (duration: 01m 34s)
  • 14:02 bblack: package upgrades on cp* commence
  • 13:23 elukey: rebooted kafka1013 for maintenance
  • 13:06 elukey: stopping kafka on kafka1013
  • 12:51 moritzm: rolling restart of ms-fe2*
  • 12:31 moritzm: restarting apache on krypton for glibc update
  • 12:00 godog: bump stripe_cache_size to 10240 for md2 on restbase1008
  • 11:54 jynus: restarting schema change on wikidatawiki (s5) T62539
  • 11:47 moritzm: restarting hhvm on mw2* to put glibc update into effect
  • 11:29 jynus: rolling schema change on wikidatawiki (s5)
  • 10:40 jynus: refreshed dns server config
  • 09:57 godog: stop cassandra-a on restbase1008 for raid expansion
  • 09:55 godog: depool restbase1008 for raid expansion T119935
  • 08:19 akosiaris: gnt-instance reboot alsafi.wikimedia.org
  • 08:19 akosiaris: enabled puppet on sca1001, sca1002
  • 03:15 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Feb 17 03:15:40 UTC 2016 (duration 9m 30s)
  • 03:06 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.14) (duration: 10m 31s)
  • 02:39 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.13) (duration: 18m 59s)
  • 01:36 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.13/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: touch - https://phabricator.wikimedia.org/T125249#2012068 (duration: 01m 31s)
  • 01:33 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.13/extensions/VisualEditor/extension.json: https://gerrit.wikimedia.org/r/#/c/271174/ (duration: 01m 29s)
  • 01:26 mutante: labservices1001 - out of disk again (T126572) - moved designate-mdns.log files to /srv/var/
  • 01:13 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.13/extensions/CentralAuth/includes/session/CentralAuthSessionProvider.php: https://gerrit.wikimedia.org/r/#/c/271153/ (duration: 01m 31s)
  • 00:47 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.13/extensions/Math/MathValidator.php: https://gerrit.wikimedia.org/r/#/c/270981/ (duration: 01m 31s)
  • 00:33 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/271074 (duration: 01m 29s)
  • 00:29 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/266603 (duration: 01m 30s)
  • 00:26 Krenair: ... after showing rsync common finished line
  • 00:25 Krenair: sync-common taking a while to terminate
  • 00:14 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/270985 (duration: 01m 34s)

2016-02-16

  • 23:39 logmsgbot: demon@tin Finished scap: pre-deploying wmf.14 to testwiki to prime l10n cache (duration: 30m 36s)
  • 23:17 hashar: Jenkins accepting slave creations again. Root cause is /var/lib/jenkins/config-history/nodes/ has reached the 32k inode limit.
  • 23:12 mobrovac: restbase deploy end of 6f69a30
  • 23:09 logmsgbot: demon@tin Started scap: pre-deploying wmf.14 to testwiki to prime l10n cache
  • 23:06 eileen: Updating CiviCRM from b2566a9849c2b3a2a0c8f0b524c69bbc2dca5f8f to bccb044c39b1eb9f270a60df69771db59623599f
  • 23:04 logmsgbot: demon@tin Synchronized w/health-check.php: Ensure procinfo exists (duration: 00m 58s)
  • 23:02 hashar: Nodepool can not authenticate with Jenkins anymore. Thus it can not add slaves it spawned.
  • 22:58 mobrovac: restbase deploy start of 6f69a30
  • 22:56 hashar: contint: Nodepool instances pool exhausted
  • 22:02 hashar: tin/mira : git -C /srv/patches/ config core.sharedRepository true
  • 21:55 chasemp: "chgrp -R mwdeploy /srv/mediawiki-staging/.git/objects && chmod -R g+w /srv/mediawiki-staging/.git/objects" on tin
  • 20:50 logmsgbot: demon@tin Finished scap: removing expired wmf.9 branch (duration: 28m 03s)
  • 20:22 logmsgbot: demon@tin Started scap: removing expired wmf.9 branch
  • 20:14 twentyafterfour: restarted phd on iridium to kick-start the outbound email queue
  • 19:38 logmsgbot: hashar@tin Synchronized multiversion/updateBranchPointers: Missing require_once MWWikiversions (duration: 00m 57s)
  • 19:16 logmsgbot: ori@mira Synchronized wmf-config/InitialiseSettings.php: I2926f73b78fa0: Revert "Test HTML stripping in production mobile beta" (duration: 01m 29s)
  • 19:01 hashar: tin: checking out mw 1.27.0-wmf.14
  • 18:16 eileen: Updating CiviCRM from b499564de8a2239b1c20acba0f8d52493c9a9c22 to b2566a9849c2b3a2a0c8f0b524c69bbc2dca5f8f
  • 18:15 logmsgbot: ori@tin Synchronized wmf-config/mobile.php: I5818f0350925: Don't serve HiDPI thumbs on mobile web (duration: 00m 58s)
  • 18:06 logmsgbot: aude@tin Synchronized php-1.27.0-wmf.13/extensions/Wikidata/extensions/Wikibase/repo/resources/dataTypes/wikibase.dataTypeStore.js: Try to fix T127095 (duration: 00m 57s)
  • 18:03 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.13/includes/mime.types: Fix .htc static (T99096) (duration: 00m 58s)
  • 17:48 logmsgbot: krinkle@tin Synchronized wmf-config/CommonSettings.php: Enable wmfstatic (wgResourceBasePath) for group0 wikis (duration: 01m 00s)
  • 17:31 logmsgbot: krinkle@tin Synchronized wmf-config/CommonSettings.php: wgLocalStylePath cleanup (use /static) (duration: 00m 59s)
  • 17:26 volans: Reboot+reimage of db1021 (T126996)
  • 17:07 volans: Disabled puppet on db1021 for reimaging: T126996
  • 17:01 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: gerrit:270556 gerrit:270679 gerrit:270556 (duration: 00m 59s)
  • 16:54 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: CX: Remove ContentTranslationCorpora setting PART II gerrit:267236 (duration: 00m 59s)
  • 16:53 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Remove ContentTranslationCorpora setting PART I gerrit:267236 (duration: 00m 58s)
  • 16:24 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Enable survey at reduced sample rate" (duration: 00m 58s)
  • 16:20 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Enable survey at reduced sample rate" (duration: 01m 01s)
  • 16:18 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable survey at reduced sample rate gerrit:270344 (duration: 01m 00s)
  • 16:12 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.13/extensions/OpenStackManager: SWAT: rebase openstack submodule (duration: 00m 58s)
  • 16:08 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: Termporary lift of IP cap for an Edit-a-thon gerrit:270646, Cleanup: removing expired event gerrit:270648 (duration: 00m 58s)
  • 15:54 urandom: running `nodetool cleanup' on restbase{1,2,7-a}.eqiad.wmnet (bootstrap of 1007-b now complete)
  • 15:34 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Remove other projects sidebar beta feature and per wiki settings (duration: 01m 09s)
  • 15:32 logmsgbot: aude@tin Synchronized wmf-config/Wikibase.php: Enable in other projects sidebar feature by default, out of beta (duration: 00m 58s)
  • 15:24 logmsgbot: aude@tin Synchronized wmf-config/Wikibase-production.php: Enable external identifier data type on Wikidata (duration: 00m 57s)
  • 15:17 logmsgbot: aude@tin Synchronized wmf-config/Wikibase.php: Enable identifiers section on Wikidata (duration: 01m 00s)
  • 15:13 logmsgbot: aude@tin Synchronized wmf-config/Wikibase-production.php: Link commons sidebar link to commons category (duration: 00m 59s)
  • 15:09 akosiaris: upgrades OTRS to 5.0.7
  • 14:58 moritzm: rebooting nescio
  • 12:28 logmsgbot: hoo@tin Synchronized wmf-config/InitialiseSettings.php: touch (duration: 00m 58s)
  • 12:15 hoo: Restarted hhvm on mw1237
  • 12:01 paravoid: rebooting pfw-codfw for upgrade
  • 11:59 logmsgbot: hoo@tin Synchronized wmf-config/: Enable Capiunto on testwiki, test2wiki and testwikidata (T126399) (2nd try) (duration: 00m 58s)
  • 11:54 logmsgbot: hoo@tin Synchronized wmf-config/: rv (duration: 00m 58s)
  • 11:49 logmsgbot: hoo@tin Finished scap: Deploy Capiunto (master) to testwiki, test2wiki and testwikidata - T126399 (duration: 24m 46s)
  • 11:46 paravoid: upgrading pfw-codfw to newer JunOS
  • 11:25 logmsgbot: hoo@tin Started scap: Deploy Capiunto (master) to testwiki, test2wiki and testwikidata - T126399
  • 11:08 godog: repool restbase on restbase1007
  • 11:02 moritzm: installing nettle security updates
  • 10:50 godog: start swiftrepl commons thumbs for top50 popular size T125791
  • 10:14 akosiaris: disable puppet, stop salt-minion on sca100{1,2}
  • 09:52 hashar: will cut the wmf branches this afternoon starting around 14:00 CET
  • 09:31 logmsgbot: hoo@tin Synchronized wmf-config/InitialiseSettings-labs.php: (no message) (duration: 00m 58s)
  • 02:34 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Feb 16 02:34:30 UTC 2016 (duration 8m 34s)
  • 02:25 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.13) (duration: 12m 28s)

2016-02-15

  • 21:26 logmsgbot: krinkle@tin Synchronized w/static.php: (no message) (duration: 00m 58s)
  • 21:25 ejegg: enabled CiviCRM queue consumer jobs
  • 21:25 ejegg: set CiviCRM back online
  • 20:55 twentyafterfour: moved contents of /srv/phab/dumps into /srv/dumps
  • 20:48 twentyafterfour: backing up /srv/phab to /srv/phab.bak in case it needs to be restored in a hurry. Note: the backup excludes /srv/phab/repos which is backed up separately
  • 20:47 twentyafterfour: brought iridium up to date with a current puppet run and checked that all repositories are at the correct tag
  • 20:38 eileen: updating civicrm from c009af16944a6478bd0292422f5bb0151f7a22c1 to b499564de8a2239b1c20acba0f8d52493c9a9c22
  • 20:17 ejegg: set CiviCRM to offline maintenance mode
  • 20:15 ejegg: disabled CiviCRM queue consumer jobs
  • 18:23 volans: Upgrading db1022 (MariaDB and kernel) already depooled and put in scheduled downtime
  • 16:46 yurik: Deployed patch for T126897
  • 16:38 logmsgbot: volans@tin Synchronized wmf-config/db-eqiad.php: depool db1022 (duration: 00m 58s)
  • 16:34 bd808: Ran sync-common on mw1119 after stuck scap update was killed there
  • 16:29 bd808: Killed stuck rsync by mw1119 that was keeping l10nudpate running
  • 16:28 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.13) (duration: 855m 18s)
  • 16:27 moritzm: installing postgres security updates on maps*
  • 15:32 _joe_: restarting the salt minions on all deployment targets
  • 15:22 apergos: disabled puppet again on iridium, had to restore sshd config from filebucket after puppet run
  • 15:18 hoo|busy: Changed email for global account "Frau pomerenke"
  • 14:51 apergos: phab role commented out on iridium and puppet re-enabled, no phab changes will be applied there
  • 14:24 godog: start restbase1007-b cassandra instance, bootstrapping T119935
  • 12:30 _joe_: deployment master is now tin
  • 11:57 _joe_: switching the deployment host back to tin
  • 11:48 _joe_: restarted hhvm on mw1130, memory exhausted
  • 11:28 jynus: offlining disk 32:5 on db1063
  • 10:58 logmsgbot: hoo@mira Synchronized wmf-config/throttle.php: Throttle exception for de:WP:Wikimedia Deutschland/WPFF Berlinale2016 (duration: 01m 28s)
  • 09:38 godog: swift codfw-prod: ms-be2020 / ms-be2021 weight to 3500
  • 09:09 _joe_: powercycled mw1131, OOM'd

2016-02-14

  • 23:18 jynus: repaired table and restarted replication on sanitarium (s3)
  • 23:06 andrewbogott: moved tools dbs off of labsdb1002 via https://gerrit.wikimedia.org/r/#/c/270650/
  • 22:40 _joe_: labsdb lvm partition /srv unavailable because /dev/sdc is apparently broken
  • 16:55 _joe_: powercycling mw1140, stuck in OOM, no ssh no console login
  • 13:57 logmsgbot: jynus@mira Synchronized wmf-config/db-eqiad.php: vslow, dump to db1054; do not send api to db1054; 1036 only rc (duration: 01m 17s)
  • 13:14 logmsgbot: hoo@mira Synchronized wmf-config/db-eqiad.php: Depool db1021 (duration: 01m 26s)
  • 02:40 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Feb 14 02:40:27 UTC 2016 (duration 9m 8s)
  • 02:31 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.13) (duration: 12m 01s)

2016-02-13

  • 06:30 urandom: `nodetool stop -- COMPACTION && nodetool cleanup' on restbase1002.eqiad, an abundance of caution (https://phabricator.wikimedia.org/P2612)
  • 04:37 urandom: (ephemerally) dropping compactor thread count from 10 to 8 on restbase1002.eqiad
  • 01:18 mutante: omg testing this log feature that logs straight to tickets (T108720)
  • 00:42 logmsgbot: krinkle@mira Synchronized wmf-config/CommonSettings-labs.php: (no message) (duration: 01m 14s)
  • 00:28 logmsgbot: krinkle@mira Synchronized wmf-config/CommonSettings-labs.php: (no message) (duration: 01m 17s)
  • 00:10 mutante: ruthenium - restarting parsoid, now works out of /srv/

2016-02-12

  • 23:37 mutante: ruthenium - moving parsoid path, cleaning up old resources
  • 22:33 logmsgbot: ori@mira Synchronized php-1.27.0-wmf.13/extensions/MobileFrontend/extension.json: I315628aef3: Don't use 'qlow' for NetSpeed=B (duration: 01m 16s)
  • 21:30 logmsgbot: tgr@mira Synchronized wmf-config/InitialiseSettings.php: T125455: log session-ip channel to logstash (duration: 01m 17s)
  • 21:22 logmsgbot: ori@mira Synchronized docroot and w: Ifc5b02cba4: Speed trials: add no-srcset variant (duration: 01m 16s)
  • 19:26 logmsgbot: bd808@mira Synchronized php-1.27.0-wmf.13/extensions/Disambiguator/Disambiguator.hooks.php: Check for array index existence (7b5f87f) (T126651) (duration: 01m 15s)
  • 19:16 bd808: Wikis back up thankfully
  • 19:12 logmsgbot: bd808@mira Synchronized php-1.27.0-wmf.13/includes/DefaultSettings.php: Log multiple IPs using the same session or the same user account (4d8b8ca) (duration: 01m 16s)
  • 18:34 logmsgbot: krenair@mira Synchronized docroot/noc: https://gerrit.wikimedia.org/r/#/c/270143/ (duration: 01m 15s)
  • 18:09 logmsgbot: krenair@mira Synchronized php-1.27.0-wmf.13/extensions/WikimediaMaintenance/dumpInterwiki.php: https://gerrit.wikimedia.org/r/#/c/270328/ (duration: 01m 16s)
  • 17:55 logmsgbot: krenair@mira Synchronized wmf-config/interwiki.php: https://gerrit.wikimedia.org/r/#/c/270327/ (duration: 01m 18s)
  • 17:39 mutante: wikibugs broken in operations and other channels
  • 17:13 _joe_: reloading apache on all the appservers
  • 17:06 hashar: CI is processing jobs again. Nodepool instances are spawning
  • 17:03 _joe_: soft-reloading apache on half of appservers
  • 16:39 jynus: purging rows from analytics-slave as requested (eventlogging database)
  • 09:39 godog: restbase1001 nodetool cleanup && nodetool stop compaction
  • 09:36 godog: restbase1002 cleanup nodetool cleanup local_group_wikipedia_T_parsoid_dataW4ULtxs1oMqJ && stop compactions
  • 07:55 moritzm: repooling elastic1023, hw problem has been fixed
  • 05:49 bblack: re-enabling cr2-knams xe-0/0/2 (Tele2)
  • 05:48 gwicke: restbase *staging*: started low-concurrency test dump run from ruthenium against xenon
  • 01:26 mobrovac: restbase deploy end of 6f6311f
  • 01:22 mobrovac: restbase2002 re-enable puppet
  • 00:45 logmsgbot: krenair@mira Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/270145/ (duration: 01m 16s)
  • 00:35 logmsgbot: krenair@mira Synchronized php-1.27.0-wmf.13: https://gerrit.wikimedia.org/r/#/c/270126/ (duration: 02m 26s)
  • 00:34 mobrovac: restbase deploy start of 6f6311f
  • 00:32 logmsgbot: krenair@mira Synchronized php-1.27.0-wmf.13/autoload.php: https://gerrit.wikimedia.org/r/#/c/270126/ (duration: 01m 16s)
  • 00:30 logmsgbot: krenair@mira Synchronized php-1.27.0-wmf.13/includes/session/MetadataMergeException.php: https://gerrit.wikimedia.org/r/#/c/270126/ (duration: 01m 14s)
  • 00:26 logmsgbot: krenair@mira Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/270140/ (duration: 01m 17s)
  • 00:19 logmsgbot: krenair@mira Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/270121/ (duration: 01m 14s)
  • 00:18 ori: Depooled mw2173
  • 00:17 urandom: `nodetool stop -- COMPACTION' on restbase1002.eqiad to free disk space (https://phabricator.wikimedia.org/P2598)
  • 00:00 mutante: add niedzielski to nda LDAP group (T106064)

2016-02-11

  • 23:53 logmsgbot: ori@mira Synchronized php-1.27.0-wmf.13/extensions/MobileFrontend: I1fe8538eb67: Provide low-resolution NetSpeed option in Special:MobileOptions (duration: 01m 22s)
  • 23:28 logmsgbot: krenair@mira Synchronized w/static/images/project-logos/adywiki.png: https://gerrit.wikimedia.org/r/#/c/268004/ (duration: 01m 19s)
  • 23:27 XenoRyet: updated payments-wiki from fad669c99db8240b26a524aa70c85cfebd13a18c to 5b909f06acce6444186ac02a494439c2ddd624aa
  • 23:26 logmsgbot: krenair@mira Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/268004/ (duration: 01m 19s)
  • 23:24 logmsgbot: krenair@mira Synchronized langlist: https://gerrit.wikimedia.org/r/#/c/268004/ (duration: 01m 21s)
  • 23:22 logmsgbot: krenair@mira rebuilt wikiversions.php and synchronized wikiversions files: (no message)
  • 23:22 logmsgbot: krenair@mira Synchronized dblists: https://gerrit.wikimedia.org/r/#/c/268004/ (duration: 01m 23s)
  • 22:21 mobrovac: restbase1009 removed dpkg hold on nodejs
  • 22:17 bblack: disable port xe-0/0/2 on cr2-knams (tele2)
  • 20:08 mutante: mendelevium (OTRS) - delete unused ssl cert files, shred key
  • 20:01 logmsgbot: demon@mira rebuilt wikiversions.php and synchronized wikiversions files: remaining wikis to wmf.13
  • 19:45 godog: restart swift on ms-be1008, sdd offlined
  • 19:40 logmsgbot: demon@mira Synchronized multiversion/MWWikiversions.php: newlines redux (duration: 02m 13s)
  • 19:27 mutante: ruthenium - chmod g+w on git repo so wikidev members can deploy
  • 18:29 bblack: concluding manual experiment on cp1065 (puppet re-enabled)
  • 18:22 yurik: updated kartotherian https://gerrit.wikimedia.org/r/#/c/270016/
  • 18:12 elukey: Add mc1012/mc1013 to redis/memcached pools after maintenance.
  • 18:01 urandom: `nodetool stop -- COMPACTION' on restbase1001.eqiad to free disk space (https://phabricator.wikimedia.org/P2596)
  • 17:06 elukey: disabled puppet, memcached, redis on mc1012/mc1013 for maintenance
  • 17:04 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert enable math extension on Wikitech. Namespace configuration on ne.wikibooks (duration: 02m 15s)
  • 16:57 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Revert "Enable Math extension on Wikitech"" gerrit:269770 (duration: 02m 13s)
  • 16:51 logmsgbot: thcipriani@mira Synchronized php-1.27.0-wmf.13/extensions/SemanticForms/includes/SF_Utils.php: SWAT: Semantic form path fixes gerrit:269869 (duration: 02m 14s)
  • 16:44 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: Change Nepali Wikibooks sitename and logo Part II gerrit:267170, Clean duplicate wgCopyUploadsDomains setting gerrit:268419 (duration: 02m 14s)
  • 16:41 logmsgbot: thcipriani@mira Synchronized w/static/images/project-logos/newikibooks.png: SWAT: Change Nepali Wikibooks sitename and logo Part I gerrit:267170 (duration: 02m 13s)
  • 16:32 logmsgbot: thcipriani@mira Synchronized wmf-config/CommonSettings.php: SWAT: Add $wgGraphAllowedDomains setting for future gerrit:269896 (duration: 02m 13s)
  • 16:27 elukey: Removed mc1012/mc1013 from the redis/memcached pools for maintenance.
  • 16:26 bblack: puppet disabled on cp1065 for some live and careful experimentation...
  • 16:20 logmsgbot: thcipriani@mira Synchronized wmf-config/wikitech.php: SWAT: Add wgOpenStackManagerNovaIdentityV3URI to wikitech configs gerrit:269558 (duration: 02m 11s)
  • 16:16 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: Add new WEF enwiki IP rate limit exception gerrit:269862 (duration: 02m 13s)
  • 16:11 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: wgRCWatchCategoryMembership true on wikisource gerrit:264735 (duration: 02m 14s)
  • 15:43 elukey: Add mc1010/mc1011 back to the redis/memcached pools after maintenance.
  • 15:24 elukey: puppet re-enabled on mc1010/mc1011
  • 14:42 elukey: Stopped puppet, memcached, redis on mc1010/mc1011 for maintenance
  • 14:07 elukey: removed mc1010/mc1011 from the redis/memcached pools for maintenance
  • 14:01 urandom: `nodetool stop -- COMPACTION' on restbase1002.eqiad to free disk space (current state: https://phabricator.wikimedia.org/P2593)
  • 13:47 jynus: backuping, reimaging, restarting and defragmenting db1024 (old s2 master)
  • 13:37 paravoid: downgrading berkelium to Linux 3.19 & rebooting
  • 13:34 apergos: reenabled scap runs (removed flag)
  • 13:21 jynus: migrating dbstore1001 to the new s2-master
  • 13:00 paravoid: upgrading backup LVS servers to Linux 4.4.0 across all sites
  • 12:42 elukey: Add mc1008/mc1009 back into redis/memcached pools after maintenance
  • 12:40 moritzm: uploaded nodejs 4.3.0 for jessie-wikimedia to carbon
  • 12:13 elukey: re-enabled puppet on mc1009
  • 12:07 elukey: enabled puppet on mc1008
  • 11:54 _joe_: stopped scap runs for now (touched sync.flag)
  • 11:52 moritzm: updated xenon, praseodymium and cerium to nodejs 4.3.0
  • 11:46 apergos: removed scap3 deb install from all hosts in prod
  • 11:33 elukey: disabled puppet, memcached, redis on mc1008/mc1009 for maintenance
  • 11:12 jynus: moving pending s2 slaves to the new master
  • 10:52 elukey: Correction about mc1008/mc1009 - puppet not disabled, will be done later on.
  • 10:50 elukey: removed mc1008/mc1009 from redis/memcached pools. Puppet disabled.
  • 10:03 godog: ms-be2018 / ms-be2019 swift weight to 3500
  • 09:52 elukey: re-added mc1007.eqiad back into redis/memcached pools after maintenance
  • 09:39 moritzm: depooled elastic1023 for hardware problems (T126586)
  • 09:31 elukey: re-enabled puppet on mc1007.eqiad
  • 09:04 jynus: restarting HHVM on all running mediawiki job queue processors
  • 08:46 _joe_: enabled puppet on mw1017, was disabled with no reason
  • 08:45 elukey: Disabling puppet, redis, memcached on mc1007 for maintenance
  • 08:33 elukey: removed mc1007 from the redis/memcached pool for Jessie migration
  • 06:56 _joe_: removing content from sd{d,j}1 written on the root partition on ms-be1008
  • 06:39 _joe_: stipped swift on ms-be1008, disk full because object were being written to the root partition
  • 06:16 bd808: Re-enabled puppet on logstash1003 and forced pupept run
  • 06:02 bd808: Raised logstash mem limit to -Xms512m on logstash1003
  • 05:57 bd808: disabled puppet on logstash1003 to debug configuration
  • 05:40 bd808: logstash process on logstash1003 flapping; continuing to investigate
  • 05:25 bd808: Restarted logstash on logstash1003; killed by OOMKiler
  • 02:16 ori: restarted HHVM on mw1130; locked up (T89912)
  • 02:15 bd808: Fetched 1271d36 to tin:/srv/mediawiki-staging; should have been updated by sync-wikiversions
  • 01:58 logmsgbot: tgr@mira rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.13 again
  • 01:47 mutante: labservices1001 - out of disk
  • 01:46 mutante: ms-be1008 - delete/gzip large syslog, out of disk
  • 01:41 logmsgbot: tgr@mira Synchronized wmf-config/mobile.php: fix for T49647 (duration: 02m 15s)
  • 01:05 logmsgbot: krenair@mira Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/269883/1 (duration: 02m 11s)
  • 00:53 logmsgbot: krenair@mira Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/263051/ (duration: 02m 13s)
  • 00:48 logmsgbot: krenair@mira Synchronized w/static/images/project-logos/wuuwiki.png: https://gerrit.wikimedia.org/r/#/c/263051/ (duration: 02m 13s)
  • 00:44 logmsgbot: krenair@mira Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/264937/ (duration: 02m 15s)
  • 00:35 logmsgbot: krenair@mira Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/269827/1 (duration: 02m 13s)
  • 00:32 logmsgbot: krenair@mira Synchronized w/static/images/project-logos/hiwikiquote.png: https://gerrit.wikimedia.org/r/#/c/269827/1 (duration: 02m 13s)
  • 00:12 logmsgbot: krenair@mira Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/259439/ (duration: 02m 20s)

2016-02-10

  • 23:49 tgr: switched mw1017 to wmf.13 (all groups)
  • 23:38 logmsgbot: ori@mira rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis back to php-1.27.0-wmf.12
  • 23:34 ori: Restarted HHVM on mw1017
  • 23:27 urandom: performing rolling restart of Cassandra in restbase staging (experimental gc settings)
  • 23:15 logmsgbot: ori@mira Synchronized wmf-config/mobile.php: I6946eccf9c: Better hack for T49647 (duration: 02m 19s)
  • 23:10 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.12/includes/WebResponse.php: I13fcc3ce4: Allow changing cookie options in WebResponseSetCookie hook (duration: 01m 30s)
  • 23:08 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.13/includes/WebResponse.php: I13fcc3ce4: Allow changing cookie options in WebResponseSetCookie hook (duration: 01m 37s)
  • 22:49 logmsgbot: krinkle@mira Synchronized w/static.php: (no message) (duration: 02m 18s)
  • 22:40 logmsgbot: krenair@mira Synchronized php-1.27.0-wmf.13/extensions/MobileFrontend/resources/mobile.loggingSchemas/SchemaEdit.js: https://gerrit.wikimedia.org/r/#/c/269755/ (duration: 02m 23s)
  • 22:36 urandom: rolling restart of Cassandra staging complete (experimental gc settings)
  • 22:35 logmsgbot: krenair@mira Synchronized php-1.27.0-wmf.12/extensions/MobileFrontend/resources/mobile.loggingSchemas/SchemaEdit.js: https://gerrit.wikimedia.org/r/#/c/269848/ (duration: 02m 18s)
  • 22:32 yurik: deployed and restarted kartotherian services
  • 22:28 urandom: performing rolling restart of Cassandra in restbase staging (experimental gc settings)
  • 22:23 yurik: deployed and restarted tilerator & tileratorui services
  • 22:15 ori: Restarted apache on palladium and strontium
  • 21:21 subbu: finished deploying parsoid version 8976ab93
  • 21:16 hashar: CI dust have settled. Krinkle and I have pooled a lot more Trusty slaves to accommodate for the overload caused by switching to php55 (jobs run on Trusty)
  • 21:14 subbu: synced code + restarted parsoid on wtp1001 as a canary
  • 21:10 subbu: starting parsoid deploy
  • 20:54 ejegg: updated DjangoBannerStats from 71df14d4d8b11f3ca0ef1eeb6c6e2db9be79103a to c143b8dd41ae413dab494ae3e194f40b8cd04bb1
  • 20:24 chasemp: tc per client shaping for labstore1001 test
  • 20:21 logmsgbot: demon@mira Synchronized php-1.27.0-wmf.13/includes/interwiki/Interwiki.php: fix cache stuff (duration: 02m 18s)
  • 20:17 apergos: cleaned up integrations slave trusty 1001,10012,10013, 1016, missed in first round.
  • 20:10 logmsgbot: demon@mira rebuilt wikiversions.php and synchronized wikiversions files: rebuild
  • 20:10 logmsgbot: demon@mira Synchronized multiversion/MWWikiversions.php: rm newline addition (duration: 02m 19s)
  • 20:03 logmsgbot: demon@mira rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.13
  • 19:45 apergos: did cleanup across all integration slaves, some were very close to out of room. results: https://phabricator.wikimedia.org/P2587
  • 19:26 mutante: ms-be1008 - powercycling - the known XFS issue
  • 19:04 mobrovac: restbase disabled puppet in staging, testing brotli compression which requires JAVA_OPTS tuning
  • 18:30 mutante: puppetstoredconfigclean.rb iodine.wikimedia.org, revoke puppet cert, delete salt key on new master
  • 18:15 mutante: iodine - shutdown, decom
  • 18:10 mutante: iodine - schedule downtime, stop puppet, stop salt, ..
  • 17:21 logmsgbot: demon@mira Synchronized multiversion/MWWikiversions.php: newlines in wikiversion.json (duration: 02m 21s)
  • 17:15 logmsgbot: demon@mira Synchronized errorpages/404.html: minor html fix (duration: 02m 17s)
  • 17:12 logmsgbot: thcipriani@mira Finished scap: VE, Cite, and Citoid bumps gerrit:269592 gerrit:269593 gerrit:269575 gerrit:269590 (duration: 29m 53s)
  • 17:02 elukey: readded mc1006 back into redis/memcached pool
  • 16:42 logmsgbot: thcipriani@mira Started scap: VE, Cite, and Citoid bumps gerrit:269592 gerrit:269593 gerrit:269575 gerrit:269590
  • 16:35 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Enable Math extension on Wikitech" (duration: 02m 14s)
  • 16:24 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Math extension on Wikitech gerrit:269694 (duration: 02m 14s)
  • 16:18 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: Namespaces configuration on ru.wikisource gerrit:269702 (duration: 02m 10s)
  • 16:09 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Enable specialcx campaign gerrit:269416 (duration: 02m 22s)
  • 15:51 jynus: recreating labsdb heartbeat views to correctly measure lag from the new s2-master
  • 15:46 elukey: re-enabled puppet on mc1006.eqiad
  • 15:42 ostriches: gerrit: flushed all caches, things will be slow for a bit while they warm
  • 15:22 elukey: disabled puppet/memcached/redis on mc1006.eqiad
  • 15:02 bblack: SPDY disable for cache_text: test starts in a few minutes! - https://phabricator.wikimedia.org/T125979
  • 14:48 elukey: removed mc1006 from the redis/memcached pool for Jessie migration
  • 13:48 godog: restbase1002 nodetool setstreamthroughput 500
  • 13:24 elukey: adding mc1005.eqiad back into service (redis/memcached)
  • 12:51 jynus: restarting hhvm at mw1015 - db errors continue
  • 12:42 Krinkle: Purged MediaWiki/wmfstatic/* metrics in Graphite (spurious test data)
  • 12:35 logmsgbot: krinkle@mira Synchronized w/static.php: (no message) (duration: 02m 11s)
  • 12:33 moritzm: disabling puppet on mw1001-1009, mw1011-1016 to enable ferm in batches
  • 11:51 jynus: restarting mw1015 jobqueue/chron processing again
  • 11:44 godog: restbase1001 nodetool setstreamthroughput 500
  • 11:37 elukey: re-enabled puppet on mc1005.eqiad
  • 11:36 godog: restbase1001 nodetool setstreamthroughput 400
  • 11:16 godog: restbase1007 nodetool-a setstreamthroughput 350
  • 11:02 jynus: enabling semi-sync replication fo s2 slaves
  • 10:58 elukey: disabling puppet, redis and memcached on mc1005 (preparation for Jessie migration)
  • 10:44 _joe_: removing mw1051-mw1070 from the appservers pool (T126242)
  • 10:26 elukey: Removing mc1005.eqiad from the redis/memcached pools
  • 10:20 godog: ms-be2016 / ms-be2017 swift weight to 3500
  • 08:24 godog: removenode restbase1007-a finished, start cassandra-a on restbase1007 for bootstrap
  • 07:54 apergos: cleared out a bunch of clones from /mnt/jenkins-workspace/workspace on integration-slave-trusty-1016, /mnt was full preventing jenkins from completing e.g. https://integration.wikimedia.org/ci/job/operations-puppet-typos/50458/console
  • 07:20 ottomata: puppet disabled on analytics1027 til tomorrow while cron is disabled and CirrusSearchRequestSet backfills into Hadoop from kafka
  • 06:19 mutante: integration-slave-trusty-1014 - out of disk ? jenkins voted things -1 because it had no space left on device
  • 02:01 logmsgbot: jynus@mira Synchronized wmf-config/db-eqiad.php: Reducing new s2 master weight for reads (duration: 02m 15s)
  • 01:49 logmsgbot: jynus@mira Synchronized wmf-config/db-codfw.php: Updating new master on codfw configuration (duration: 02m 15s)
  • 01:41 jynus: starting pt-heartbeat on db1018
  • 01:23 jynus: started db1048 replication. For some reason, replication was stopped. Need further investigation.
  • 01:14 logmsgbot: catrope@mira Synchronized wmf-config/InitialiseSettings.php: Set $wgPageLanguageUseDB = true on testwii (duration: 02m 14s)
  • 01:12 logmsgbot: catrope@mira Synchronized docroot/noc/conf/highlight.php: Remove ob_start() from highlight.php (duration: 02m 13s)
  • 00:46 RoanKattouw: Running updateCollation.php on nlwiki
  • 00:31 logmsgbot: catrope@mira Synchronized wmf-config/CommonSettings.php: BetaFeatures wmg->wg rename, part 2 (duration: 02m 13s)
  • 00:28 logmsgbot: catrope@mira Synchronized wmf-config/InitialiseSettings.php: Set collation to uca-nl on nlwiki; add Recherche: to content namespaces on frwikiversity; BetaFeatures wmg->wg rename (duration: 02m 12s)
  • 00:20 logmsgbot: catrope@mira Synchronized wmf-config/InitialiseSettings.php: Increase completion suggester replicas for busy wikis (duration: 02m 11s)
  • 00:18 logmsgbot: catrope@mira Synchronized wmf-config/logging.php: Reduce Kafka timeouts (duration: 02m 13s)
  • 00:11 logmsgbot: catrope@mira Synchronized wmf-config/InitialiseSettings.php: Test HTML stripping in production mobile beta (duration: 02m 12s)

2016-02-09

  • 23:55 jynus: restarting jobrunner and jobchron
  • 23:38 jynus: setting db1018's binlog_format as STATEMENT
  • 23:30 logmsgbot: jynus@mira Synchronized wmf-config/db-eqiad.php: Disable read only mode for s2 after its master failover (duration: 02m 09s)
  • 23:27 _joe_: disabled puppet on mc1004, added "bind 0.0.0.0" to its redis config, restarted redis (T126395)
  • 23:23 jynus: setting db1018 in read/write mode
  • 23:22 logmsgbot: jynus@mira Synchronized wmf-config/db-eqiad.php: Actual mediawiki master failover (duration: 02m 14s)
  • 23:19 bd808: Changed /src/mediawiki/wikiverisons.php on mw1017 (X-Wikimedia-Debug) to set all wikis to 1.27.0-wmf.13
  • 23:18 jynus: setting db1024 in read only mode
  • 23:16 logmsgbot: jynus@mira Synchronized wmf-config/db-eqiad.php: Enabling read only mode for s2 before its master failover (duration: 02m 14s)
  • 23:09 jynus: setting up circular replication between db1018 and db1024 for potential rollback
  • 23:02 jynus: changing topology of s2 slaves in preparation for master failover
  • 22:04 logmsgbot: bd808@mira Synchronized wmf-config/logging.php: Monolog: reorder Monolog processors (b356eeb) (duration: 02m 15s)
  • 21:33 Krenair: ran package upgrades on wikitech-static
  • 20:37 bblack: restarting nginx for libssl update on cp1049.eqiad.wmnet,cp4008.ulsfo.wmnet,cp3042.esams.wmnet,cp3049.esams.wmnet
  • 20:32 logmsgbot: demon@mira Finished scap: all group0 to wmf.13 (duration: 29m 45s)
  • 20:25 bblack: cache kernel reboots done (all on '3.19.0-2-amd64 #1 SMP Debian 3.19.3-9 (2016-01-04)', except 4x canaries on '4.4.0-1-amd64 #1 SMP Debian 4.4-1~wmf1 (2016-01-26)')
  • 20:11 bblack: cp1067, cp1071 (text, upload in eqiad) -> 4.4 canaries (rebooting over the next ~8 mins or so)
  • 20:02 logmsgbot: demon@mira Started scap: all group0 to wmf.13
  • 19:55 hoo: Updated operations/dumps/dcat on snapshot1003 from 0a71deb232 to 92ab37d94e
  • 19:37 logmsgbot: demon@mira Finished scap: pruning tons of stale branches + sync wmf.13 files for later + testwiki to wmf.13 to build l10n cache (try 2) (duration: 27m 24s)
  • 19:35 bblack: cp3048 (upload esams) rebooting -> kernel 4.4 canary
  • 19:13 mutante: gerrit - add ppchelko to mediawiki-services
  • 19:11 bblack: cp4006 (upload ulsfo) rebooting -> kernel 4.4 canary
  • 19:09 logmsgbot: demon@mira Started scap: pruning tons of stale branches + sync wmf.13 files for later + testwiki to wmf.13 to build l10n cache (try 2)
  • 19:07 logmsgbot: demon@mira scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_2315818744" --threads=10 --lang en --quiet' returned non-zero exit status 255 (duration: 00m 34s)
  • 19:07 logmsgbot: demon@mira Started scap: pruning tons of stale branches + sync wmf.13 files for later + testwiki to wmf.13 to build l10n cache
  • 18:57 yurik: deployed graphoid
  • 18:05 jynus: bringing down db1048's mysql for cloning to db2012
  • 17:54 Krenair: ssh: connect to host mw1037.eqiad.wmnet port 22: Connection timed out
  • 17:53 logmsgbot: krenair@mira Synchronized php-1.27.0-wmf.12/extensions/OpenStackManager: https://gerrit.wikimedia.org/r/#/c/269439/ (duration: 03m 15s)
  • 17:26 elukey: mc1004.eqiad put back into redis/memcached pool
  • 17:23 godog: nodetool-a removenode ec0c5a3d-2648-4933-8434-a8d163b92188 in preparation for restbase1007 bootstrap
  • 17:22 bblack: rebooting cp1008/pinkunicorn for 4.4-rt kernel test
  • 17:19 _joe_: powered down mw1037
  • 17:07 godog: start cassandra-a on restbase1007 with replace_address=10.64.0.230
  • 16:57 logmsgbot: thcipriani@mira Finished scap: SWAT: Clarify and expand messages mentioning loss of session data gerrit:269424 (duration: 27m 36s)
  • 16:53 bblack: rebooting cp1008/pinkunicorn for 4.4 kernel
  • 16:34 jynus: reimage db2012
  • 16:30 logmsgbot: thcipriani@mira Started scap: SWAT: Clarify and expand messages mentioning loss of session data gerrit:269424
  • 16:18 logmsgbot: thcipriani@mira Synchronized wmf-config: SWAT: Enable ArticlePlaceholder on test wikis gerrit:269399 (duration: 01m 19s)
  • 16:15 thcipriani: mw1037.eqiad.wmnet error during SWAT rsync: failed to set times on "/srv/mediawiki/.": Read-only file system (30)
  • 16:09 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable math data type on Wikidata and everywhere gerrit:269398 (duration: 02m 31s)
  • 15:59 elukey: puppet re-enabled on kafka1012
  • 15:56 paravoid: "power"cycling alsafi
  • 15:55 moritzm: uploaded linux 4.4-1~wmf1 (jessie-wikimedia/experimental) to carbon
  • 15:48 _joe_: re-removed the puppet facts for protactinium
  • 15:40 paravoid: echo 1 > /proc/sys/net/ipv4/vs/schedule_icmp on lvs3001
  • 15:36 elukey: disabled puppet on kafka1012, changing temporary kafka retention to purge some extra logs
  • 15:17 cmjohnson1: snapshot1002 mistakenly taken offline -- booting now
  • 15:15 paravoid: upgrading lvs4001/4002 to linux 4.4.0
  • 15:07 godog: stop cassandra on restbase1007, cpu/mem upgrade and reimage
  • 14:59 paravoid: upgrading lvs3001/3002 to linux 4.4.0
  • 14:53 godog: reboot ms-be1004, xfs hosed
  • 14:51 hashar: Cutting branches 1.27.0-wmf.13
  • 14:46 elukey: re-enabled puppet on mc1004.eqiad
  • 14:45 bblack: resuming cpNNNN rolling kernel reboots
  • 14:41 _joe_: setting mw1026-1050 as inactive in the appservers pool (T126242)
  • 13:58 hashar: shutting down jenkins finally, and restarting it
  • 13:51 hashar: Restarting Jenkins. It can not manage to add slaves
  • 13:15 paravoid: upgrading lvs1001/lvs1007/lvs1002/lvs1008/lvs1003/lvs1009 to 4.4.0
  • 13:11 akosiaris: reboot serpens to apply memory increase of 2G
  • 13:07 paravoid: installing linux 4.4.0 on lvs1001
  • 13:01 hashar: Jenkins disabled again :(
  • 12:53 akosiaris: reboot seaborgium to apply memory increase of 2G
  • 12:47 hashar: Updated faulty script that caused 'php' too loop infinitely. Jenkins back up.
  • 12:36 hashar: Jenkins no more accept new jobs until the slaves are fixed :/
  • 12:33 hashar: all CI slaves looping to death because of a php loop
  • 11:43 paravoid: upgrading lvs2001, lvs2002, lvs2003 to kernel 4.4.0
  • 11:36 paravoid: reverting lvs2005 to 3.19 and rebooting, test is over and was successful
  • 11:19 paravoid: stopping pybal on lvs2002
  • 11:05 paravoid: installing linux-image-4.4.0 on lvs2005 and rebooting for testing
  • 10:53 apergos: salt minions on labs instances that respond to labcontrol1001 will be coming back up over the next 1/2 hour as puppet runs (salt master key fixes)
  • 10:45 elukey: disabled puppet, redis and memcached on mc1004 for jessie migration
  • 10:33 _joe_: pybal updated everywhere
  • 10:32 gehel: elasticsearch codfw: cleanup leftover logs /var/log/elasticsearch/*.[2-7]
  • 10:24 gehel: elasticsearch eqiad: cleanup leftover logs /var/log/elasticsearch/*.[2-7]
  • 10:09 _joe_: upgrading pybal on active nodes in esams and eqiad
  • 10:04 _joe_: depooling elastic1021.eqiad.wmnet as RAM has failed
  • 09:56 jynus: running table engine conversion script on db1069 (potential small lag on labs for 1 day)
  • 09:40 moritzm: restarted cassandra-a service on praseodymium
  • 09:21 ema: restarted hhvm on mw1132
  • 08:49 _joe_: installing the new pybal package in esams and eqiad backups
  • 08:23 moritzm: restarted cassandra-a service on praseodymium
  • 07:11 _joe_: manually touched (with -h) the wmf-config/PrivateSettings.php symlink on all mw* hosts
  • 07:02 logmsgbot: tgr@mira Synchronized wmf-config/PrivateSettings.php: Mass logout via $wgAuthenticationTokenVersion - T124440#2010709 (duration: 01m 20s)
  • 01:58 legoktm: added SMalyshev to wikidata-query gerrit group
  • 01:34 logmsgbot: krenair@mira Synchronized php-1.27.0-wmf.12/extensions: https://gerrit.wikimedia.org/r/#/c/269344/ and https://gerrit.wikimedia.org/r/#/c/269293/1 (duration: 01m 51s)
  • 01:27 logmsgbot: krenair@mira Synchronized php-1.27.0-wmf.12/extensions/OAuth/frontend/specialpages/SpecialMWOAuthManageConsumers.php: https://gerrit.wikimedia.org/r/#/c/269333/ (duration: 01m 19s)
  • 01:24 logmsgbot: krenair@mira Synchronized php-1.27.0-wmf.12/resources/src: https://gerrit.wikimedia.org/r/#/c/269140/ (duration: 01m 19s)
  • 00:54 logmsgbot: krenair@mira Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/266509/ (duration: 01m 17s)
  • 00:52 logmsgbot: krenair@mira Synchronized docroot/noc: https://gerrit.wikimedia.org/r/#/c/266509/8 (duration: 01m 17s)
  • 00:50 logmsgbot: krenair@mira Synchronized wmf-config/ProductionServices.php: https://gerrit.wikimedia.org/r/#/c/266509/8 (duration: 01m 18s)
  • 00:42 hashar: killed Zuul scheduler. On gallium edited /usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/trigger/gerrit.py and modified: replication_timeout = 300 -> replication_timeout = 10 . Started Zuul
  • 00:26 logmsgbot: krenair@mira Synchronized portals: https://gerrit.wikimedia.org/r/#/c/268849/ (duration: 01m 18s)
  • 00:25 logmsgbot: krenair@mira Synchronized portals/prod/wikipedia.org/assets: https://gerrit.wikimedia.org/r/#/c/268849/ (duration: 01m 18s)
  • 00:17 logmsgbot: krenair@mira Synchronized portals: https://gerrit.wikimedia.org/r/#/c/268849/ (duration: 01m 17s)
  • 00:15 logmsgbot: krenair@mira Synchronized portals/prod/wikipedia.org/assets: https://gerrit.wikimedia.org/r/#/c/268849/ (duration: 01m 16s)

2016-02-08

  • 23:30 logmsgbot: ori@mira Synchronized php-1.27.0-wmf.12/includes/interwiki/Interwiki.php: ac6e170fa5: Fix-up for I5a979f047031e (duration: 01m 18s)
  • 23:24 logmsgbot: bd808@mira Synchronized wmf-config/logging.php: logging: Collect mw1017 logs for debugging (9d6d0e0) (duration: 01m 18s)
  • 23:20 logmsgbot: bd808@mira Synchronized wmf-config/logging.php: logging: Send all udp2log eligible messages to $wmgDefaultMonologHandler (cd25586) (duration: 01m 17s)
  • 23:16 logmsgbot: bd808@mira Synchronized wmf-config/logging.php: Monolog: Add mwversion to udp2log log events (9b54967) (duration: 01m 18s)
  • 23:05 logmsgbot: bd808@mira Synchronized wmf-config/logging.php: Monolog: normalize messages before PSR3 expansion (e5ee5d8) (duration: 01m 18s)
  • 22:52 logmsgbot: ori@mira Synchronized wmf-config/missing.php: Ib5407c560: Update missing.php for interwiki.php (duration: 01m 18s)
  • 22:51 logmsgbot: ori@mira Synchronized docroot and w: Ifd7fe8c3c: createTxtFileSymlinks.sh: drop interwiki.cdb; add interwiki.php (duration: 01m 21s)
  • 22:43 logmsgbot: bd808@mira Synchronized php-1.27.0-wmf.12/includes/debug/logger/monolog/WikiProcessor.php: Add $wgVersion to MediaWiki\Logger\Monolog\WikiProcessor (3cea726) (duration: 01m 19s)
  • 21:51 logmsgbot: ori@mira Synchronized wmf-config/CommonSettings.php: Ie9bdd77fb: Use interwiki.php on all wikis; delete unused interwiki.json (duration: 01m 19s)
  • 21:31 logmsgbot: ori@mira Synchronized wmf-config/CommonSettings.php: I39c9ecd4b: Enable static PHP interwiki cache on mediawikiwiki and testwiki (duration: 01m 18s)
  • 21:28 ori: Restarting HHVM on mw1017 to wipe APC cache
  • 21:23 logmsgbot: ori@mira Synchronized wmf-config/CommonSettings.php: Ib599f9984a: Add interwiki.php; use it on mw1017 & on labs (2/2) (duration: 01m 16s)
  • 21:22 logmsgbot: ori@mira Synchronized wmf-config/interwiki.php: Ib599f9984a: Add interwiki.php; use it on mw1017 & on labs (1/2) (duration: 01m 20s)
  • 21:19 subbu: finished deploying parsoid version 4d44fcc7
  • 21:10 subbu: synced code; restarted parsoid on wtp1003 as a canary
  • 21:04 subbu: starting parsoid deploy
  • 21:02 bblack: resuming rolling reboots of cpNNNN caches for kernel updates
  • 20:34 mobrovac: restbase deploy end of c929ceb
  • 20:26 mobrovac: restbase deploy start of c929ceb
  • 19:36 logmsgbot: mattflaschen@mira Synchronized wmf-config/InitialiseSettings-labs.php: Beta Cluster-only change (duration: 01m 20s)
  • 19:04 papaul: sarin - signing puppet certs, salt-key, initial run
  • 19:03 urandom: restart restbase on restbase-test2001.codfw (staging)
  • 18:54 mobrovac: mathoid deployed 4bdb2f18c
  • 18:43 urandom: rolling Cassandra restart in restbase staging complete
  • 18:35 MatmaRex: Reopened 54 Phabricator tasks that someone merged into one, hope I haven't made more of a mess than it was before
  • 18:30 jynus: applying ferm on dbstore1001 and dbstore1002
  • 18:29 urandom: performing rolling restart of Cassandra in staging (to pickup /usr/share/cassandra/lib/cassandra-brotli-1.0.0-a64ce47.jar in classpath)
  • 18:25 elukey: re-enabled puppet on mc1004
  • 18:17 logmsgbot: ebernhardson@mira Synchronized wmf-config/CirrusSearch-common.php: dont reindex wgCirrusSearchNamespaceWeights from 0 (duration: 01m 17s)
  • 18:13 chasemp: cleanup snapshots on labstore1001
  • 17:03 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: Set category collation on gd.wikipedia gerrit:267820 (duration: 01m 21s)
  • 16:59 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: Deploy Translate extension on ru.wikimedia gerrit:267822 (duration: 01m 17s)
  • 16:52 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable signature button for the Project namespace in ru.wiki gerrit:267997 (duration: 01m 19s)
  • 16:45 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: Namespaces configuration on mai.wikipedia gerrit:268573 (duration: 01m 17s)
  • 16:39 logmsgbot: thcipriani@mira Synchronized wmf-config/mobile.php: SWAT: Use custom generator for mobile search on Wikidata Part II gerrit:254645 (duration: 01m 19s)
  • 16:37 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: Use custom generator for mobile search on Wikidata Part I gerrit:254645 (duration: 01m 18s)
  • 16:32 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: Do not request pageprops for mobile search/nearby on wikidata gerrit:268208 (duration: 01m 20s)
  • 16:28 logmsgbot: thcipriani@mira Synchronized wmf-config/Wikibase-production.php: SWAT: Add $wgWBRepoSettings[sparqlEndpoint] gerrit:268467 (duration: 01m 18s)
  • 16:27 elukey: restarted nutcracker in G@cluster:appserver and G@site:eqiad due to connect error issues (5 hosts per batch)
  • 16:27 _joe_: reinstalling pybal's new version (reduced) on ulsfo and codfw caches
  • 16:24 jynus: reverting slaves topology back to db1024 master
  • 16:19 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: wgRCWatchCategoryMembership true everywhere except wikisource gerrit:264734 (duration: 01m 26s)
  • 16:01 andrewbogott: restarting pdns on labservices 1001 to test loglevels
  • 15:30 elukey: stopping redis and memcached for mc1004.eqiad.wmnet due to Jessie re-image
  • 15:30 chasemp: restarting pdns and pdns-recursor on labservices1001
  • 15:01 jynus: restarting and upgrading dbstore1001 (db backups agent host)
  • 14:41 bblack: mobile LVS service decom complete (IPs now belong to text service)
  • 14:03 bblack: starting mobile LVS service decom (IPs moving to text) - puppet disabled on text caches and high-traffic1 LVSes
  • 13:56 bblack: cpNNNN rolling reboots paused (3038 still coming up)
  • 13:12 bblack: start up more rolling cache reboots for kernels (cpNNNN)
  • 13:09 elukey: updated hhvm on mw2016.codfw.wmnet, mw2161.codfw.wmnet, mw2199.codfw.wmnet, mw1259.eqiad.wmnet, mw1260.eqiad.wmnet
  • 13:05 _joe_: roll back installation of pybal, issues with upd and ipv6
  • 12:56 elukey: updated hhvm on mw1080, mv1084, mw1241
  • 12:32 elukey: restarting hhvm on mw1052, mw1075, mw1080, mw1081, mw1094, mw1095 to rollout the new version
  • 12:32 _joe_: uploaded a new pybal package; installing on codfw and ulsfo backups
  • 12:05 _joe_: restarted cron on tin, to catch up with the uid change for the l10nupdate user
  • 11:53 bblack: rebooting cp1074, cp3047 (for kernels, also to compare bios/drac settings...)
  • 11:26 jynus: stopping mysql at db2012
  • 11:25 jynus: starting mysql at db2012
  • 11:05 moritzm: rebooting db2012 for kernel update
  • 11:00 moritzm: rebooting terbium for kernel update
  • 10:26 moritzm: rebooting es2006,es2008 for kernel update
  • 10:25 moritzm: upgrading jobrunners/imagescalers in eqiad for hhvm float timeout fix
  • 10:20 jynus: changing s2 replication topology in preparation for master failover
  • 09:45 jynus: starting es2004
  • 09:29 moritzm: rebooting es2005,es2007,es2009,es2010 for kernel update
  • 09:15 elukey: hhvm restarted on mw1044.eqiad.wmnet due to hhvm package update
  • 09:15 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Feb 8 09:15:11 UTC 2016 (duration 8m 10s)
  • 09:12 elukey: hhvm restarted on mw1034.eqiad.wmnet due to hhvm package update
  • 09:07 logmsgbot: oblivian@tin sync-l10n completed (1.27.0-wmf.12) (duration: 11m 55s)
  • 08:42 _joe_: trying a manual run of l10nupdate since it failed last night again
  • 08:25 moritzm: rebooting es2001 to es2004 for kernel update

2016-02-07

  • 04:54 andrewbogott: upgraded python-openstackclient python-glanceclient python-novaclient python-keystoneclient on silver

2016-02-06

  • 05:43 bblack: rebooted cp2006 via racadm after crash - no crash data in logs...

2016-02-05

  • 23:54 chasemp: nfs shaping is really writes :)
  • 23:54 chasemp: tc to shape some nfs read traffic in tools for labs (also logged there) can be cancelled with: /sbin/tc qdisc del dev eth0 root
  • 23:51 YuviPanda: dropped old nfs snapshots from labstore1001
  • 23:30 logmsgbot: maxsem@mira Synchronized portals: (no message) (duration: 01m 18s)
  • 23:29 logmsgbot: maxsem@mira Synchronized portals/prod/wikipedia.org/assets: (no message) (duration: 01m 19s)
  • 22:56 jynus: reimaging db1018
  • 22:48 jynus: restarting slave on m2/codfw (db2011)
  • 22:41 logmsgbot: krenair@mira Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/268818/ (duration: 01m 22s)
  • 22:10 bblack: cache rolling reboots stopped for the weekend, can pick up the other half monday
  • 20:36 bblack: resuming rolling cache reboots
  • 20:07 mutante: cygnus - reboot VM
  • 19:28 bblack: halted rolling cache reboots, we seem to be having problems with a batch of them coming back...
  • 18:23 logmsgbot: demon@mira Synchronized wmf-config/InitialiseSettings.php: comment stuff, gerrit 267994 (duration: 01m 19s)
  • 18:15 jynus: stopping mysql@db1018 and starting to clone it for reimaging
  • 18:10 logmsgbot: jynus@mira Synchronized wmf-config/db-eqiad.php: Depool db1018 for maintenance (duration: 02m 12s)
  • 17:31 cmjohnson1: trouble shooting elastic1021
  • 17:08 bblack: rolling cpNNNN reboots are 27% complete, only two hosts so far failed to reboot on their own (but came up fine after manual racadm powercycle)
  • 16:20 ottomata: reenabling kafka1012 in analytics-eqiad kafka cluster
  • 16:03 jynus: reimaging db2030 to test jessie installer
  • 15:53 logmsgbot: oblivian@tin sync-l10n completed (1.27.0-wmf.12) (duration: 00m 08s)
  • 15:47 urandom: performing rolling restbase restart in staging env
  • 15:35 andrewbogott: rebooting silver for kernel update - wikitech outage will ensue
  • 15:33 urandom: re-restarting restbase on restbase1002.eqiad.wmnet,restbase1005.eqiad.wmnet,restbase1006.eqiad.wmnet,restbase1009.eqiad.wmnet (prior restarts may have happened before puppet run)
  • 15:29 andrewbogott: rebooting holmium for kernel update
  • 15:27 andrewbogott: rebooting labcontrol1002 for kernel update
  • 15:24 bblack: cp3005 didn't come back online during rolling reboot, investigating (remains depooled)
  • 15:22 _joe_: initializing mediawiki repos on tin
  • 15:22 andrewbogott: rebooting labnet1001 for kernel update
  • 15:15 urandom: restbase rolling restart complete
  • 15:08 urandom: performing rolling restbase restart to apply config change (https://gerrit.wikimedia.org/r/#/c/268611/)
  • 14:56 urandom: forcing puppet run and bouncing restbase on restbase1001.eqiad.wmnet (https://gerrit.wikimedia.org/r/#/c/268611/)
  • 14:41 elukey: confctl mw1228.eqiad.wmnet: weight changed 10 => 20
  • 14:24 moritzm: rebooting db2065 to db2070 for kernel update
  • 14:20 jynus: reimporting nlwiktionary revision into labs (expect some temporary lag on labs-s3)
  • 14:06 moritzm: rebooting db2060 to db2064 for kernel update
  • 13:34 bblack: starting rolling reboots of cp* (traffic cache hosts) for kernel updates
  • 12:51 moritzm: rebooting db2055 to db2059 for kernel update
  • 12:38 elukey: repooled mw1228.eqiad.wmnet
  • 12:34 moritzm: rebooting db2050 to db2054 for kernel update
  • 12:15 moritzm: rebooting db2045 to db2049 for kernel update
  • 12:07 jynus: reimporting nlwiktionary pages into labs
  • 12:05 logmsgbot: l10nupdate@tin LocalisationUpdate failed: git pull of core failed
  • 12:05 logmsgbot: l10nupdate@tin LocalisationUpdate failed: git clone of core failed
  • 11:54 moritzm: rebooting db2041 to db2044 for kernel update
  • 11:37 moritzm: rebooting db2038 to db2040 for kernel update
  • 11:36 godog: start swiftrepl replication pass of common thumbs eqiad -> codfw
  • 10:15 moritzm: rolling reboot of ocg* cluster
  • 02:27 mobrovac: restbase deploy end of caae1f7
  • 02:20 mobrovac: restbase deploy start of caae1f7
  • 01:52 Tim: deploying apache log format change following successful test on deployment-prep
  • 01:47 logmsgbot: aude@mira Finished scap: Re-add user rights messages in Echo (duration: 24m 37s)
  • 01:22 logmsgbot: aude@mira Started scap: Re-add user rights messages in Echo
  • 01:20 logmsgbot: aude@mira Synchronized php-1.27.0-wmf.12/extensions/Echo: Re-add user rights messages in Echo (duration: 01m 20s)
  • 01:06 logmsgbot: aude@mira Synchronized wmf-config/: Sync wikidata config changes for beta (duration: 01m 15s)
  • 01:00 mobrovac: restbase deploy end of 2aef1b67a0
  • 00:58 logmsgbot: aude@mira Synchronized wmf-config/InitialiseSettings.php: Set $wgEnotifMinorEdits to true on huwiki (duration: 01m 16s)
  • 00:53 logmsgbot: aude@mira Synchronized wmf-config/InitialiseSettings.php: Add museumvictoria.com.au to $wgCopyUploadsDomains (duration: 01m 17s)
  • 00:46 logmsgbot: aude@mira Synchronized wmf-config/InitialiseSettings.php: Re-enable ShortUrl on maiwiki, bhwiki and orwikisource, after creating db table (duration: 01m 18s)
  • 00:36 logmsgbot: aude@mira Synchronized wmf-config/InitialiseSettings.php: revert shorturl changes (duration: 01m 17s)
  • 00:29 logmsgbot: aude@mira Synchronized wmf-config/InitialiseSettings.php: Enable ShortUrl on maiwiki, bhwiki and orwikisource (duration: 01m 17s)
  • 00:26 akosiaris: restart apache on mendelevium.eqiad.wmnet. seems there's a memory leak, need to investigate tomorrow
  • 00:23 logmsgbot: aude@mira Synchronized wmf-config/InitialiseSettings.php: Re-enable category watch on wikipedia and commons (duration: 01m 18s)
  • 00:19 mobrovac: restbase deploy start of 2aef1b67a0 on rb1001

2016-02-04

  • 23:54 mobrovac: restbase disabled temporarily puppet in prod to test https://gerrit.wikimedia.org/r/#/c/268597/
  • 23:29 mutante: rcs1001 - rebooting
  • 22:45 bblack: rebooting cp1060 to test traffic-pool stuff
  • 22:30 logmsgbot: demon@mira Synchronized wmf-config/: undo my cleanup grumble grumble (duration: 01m 16s)
  • 22:12 logmsgbot: demon@mira Synchronized wmf-config/: touch (duration: 01m 14s)
  • 22:10 logmsgbot: demon@mira Synchronized private/: touch (duration: 01m 15s)
  • 22:01 logmsgbot: demon@mira Synchronized wmf-config/PrivateSettings.php: touch symlink (duration: 01m 15s)
  • 21:57 logmsgbot: demon@mira Synchronized wmf-config/: gerrit 268471, 268454 (duration: 01m 18s)
  • 21:52 mutante: mira chgrp -R wikidev /srv/mediawiki-staging/.git/objects
  • 21:29 mutante: rcs1001 started redis
  • 21:21 paravoid: setting up OSPF/OSPF3/PIM between ulsfo and codfw (cr2-ulsfo/cr1-codfw)
  • 21:19 mutante: rcs1002 - start redis
  • 21:15 mutante: rcs1002 - reboot for kernel
  • 20:45 mutante_: rsc1001 - schedule downtime, reboot
  • 20:30 paravoid: cr1-ulsfo: deactivating BGP peering with GTT
  • 20:26 mutante: eeden service ntp restart
  • 20:26 hashar: All wikis to 1.27.0-wmf.12 No troubles so far congratulations to everyone involved @wikimedia #wikimedia
  • 20:23 mutante: mw1115 service hhvm restart
  • 20:16 mutante: mw1117 - powercycled
  • 20:15 paravoid: cr1-ulsfo: turning up BGP with Zayo
  • 20:15 mutante: scb1001/scb1002 service mathoid restart
  • 20:03 logmsgbot: hashar@mira rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.27.0-wmf.12
  • 20:03 hashar: all wikis to 1.27.0-wmf.12 (yeah really)
  • 20:00 mobrovac: restbase rolling restart after merging https://gerrit.wikimedia.org/r/#/c/268016/
  • 19:54 logmsgbot: demon@mira Synchronized private/PrivateSettings.php: (no message) (duration: 02m 05s)
  • 19:43 ottomata1: rebooting kafka1012
  • 19:32 logmsgbot: demon@mira Synchronized wmf-config/InitialiseSettings.php: T125850 (duration: 02m 11s)
  • 19:00 logmsgbot: demon@mira Synchronized wmf-config/InitialiseSettings-labs.php: prod no op for completeness (duration: 03m 02s)
  • 18:49 logmsgbot: demon@mira Synchronized wmf-config/: removing old mwblocker.log (duration: 02m 07s)
  • 18:45 logmsgbot: demon@mira Synchronized private/mwblocker.log: (no message) (duration: 02m 10s)
  • 18:40 yurik: deployed and reenabled tilerator & tileratorui
  • 17:30 logmsgbot: hoo@mira Synchronized php-1.27.0-wmf.12/extensions/Wikidata: Fix editing terms in languages other than the interface language via the term box (duration: 02m 18s)
  • 17:29 hoo: (Re)started wdqs-updater on wdqs1001, but seems it doesn't work
  • 16:53 twentyafterfour: restarted phd to synchronize settings with phabricator
  • 16:52 twentyafterfour: restarted apache2 on iridium so that phabricator recognizes sprint.phragile-uri
  • 16:45 logmsgbot: thcipriani@mira Synchronized php-1.27.0-wmf.12/includes/media/Bitmap.php: SWAT: BitmapHandler: Implement validateParam() gerrit:268407 (duration: 02m 08s)
  • 16:07 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable confirmed group at nowiki gerrit:267804 (duration: 02m 15s)
  • 15:53 urandom: rolling restbase restart complete
  • 15:47 urandom: restbase cluster puppet run complete; performing rolling restart of restbase (applying https://gerrit.wikimedia.org/r/#/c/266297/)
  • 15:44 moritzm: rebooting db203[678] for kernel update
  • 15:35 urandom: forcing puppet run on restbase cluster (config deploy)
  • 15:34 urandom: reenabling puppet on restbase cluster (continue config deploy)
  • 15:18 urandom: restarting restbase on restbase1001.eqiad.wmnet (config deploy)
  • 15:15 urandom: re-enabling puppet and forcing run on restbase1001.eqiad.wmnet (canary config deploy)
  • 15:11 _joe_: restarting pybal on lvs200{3,6}
  • 15:08 urandom: disabling puppet on restbase cluster in preparation for configuration deploy (https://gerrit.wikimedia.org/r/#/c/266297/)
  • 15:07 moritzm: rebooting oxygen for kernel update
  • 14:58 ottomata: stopping eventlogging to reboot eventlog1001 for kernel update
  • 14:41 moritzm: rebooting db203[45] for kernel update
  • 14:09 hoo: Restarted blazegraph on wdqs1001
  • 13:57 godog: powercycle ms-be2020
  • 13:39 moritzm: continue rolling reboot of maps cluster for kernel update (2002-2004)
  • 12:21 jynus: starting mysql at db2009
  • 12:08 moritzm: rebooting db2001 to db2019 for kernel update
  • 11:44 jynus: dropping echo_* tables from labs
  • 11:19 dcausse: elastic codfw: resuming writes and setting cluster.routing.allocation.balance.threshold back to default (1%)
  • 10:35 dcausse: elastic codfw: freezing writes and setting cluster.routing.allocation.balance.threshold to 100% (fast recovery test)
  • 10:35 logmsgbot: hashar@mira Synchronized php-1.27.0-wmf.12/.gitmodules: Set branch in .gitmodules for extensions/Wikidata https://gerrit.wikimedia.org/r/#/c/268218/ (duration: 02m 08s)
  • 10:16 moritzm: rolling reboot of maps cluster for kernel update
  • 10:14 jynus: testing new replication filters from production's testwiki
  • 10:13 elukey: running smartctl -t long on kafka1012 (kafka not running, host de-pooled from the broker list)
  • 10:11 moritzm: repooling restbase2006
  • 10:01 jynus: applying live on the 7 sanitarium instance the newly puppet-configured labs replication filters
  • 09:57 moritzm: repooling restbase2005, depooling restbase2006 for kernel reboot/Java update
  • 09:46 dcausse: elastic in codfw: reducing the number of replicas from 0-3 to 0-2 for commonswiki_file
  • 09:46 moritzm: repooling restbase2004, depooling restbase2005 for kernel reboot/Java update
  • 09:39 ema: re-enabling puppet on mw1161
  • 09:34 moritzm: depooling restbase2004 for kernel reboot/Java update
  • 09:11 jynus: converting remaining InnoDB tables (s3) to TokuDB on db1069
  • 08:14 chasemp: iridium puppet agent --enable && puppet agent --disable "DO NO ENABLE AS IT WILL BREAK THINGS CONTACT MUKUNDA"
  • 07:51 twentyafterfour: phabricator repositories checked out to these revisions: http://pastebin.com/JxEaYKiW
  • 07:49 chasemp: git checkout tag release/2015-11-18/1 for phab & libphutil on iridiuum
  • 07:35 andrewbogott: disabling puppet on iridium to prevent it from smashing phabricator (as it seems to do now and then)
  • 07:00 andrewbogott: on iridium in /srv/deployment/phabricator/deploy/phabricator, naming the currently detached git branch ‘andrewfounditlikethis'
  • 06:49 robh: phabricator down with errors during repo updates in phd daemon log
  • 02:12 mutante: OTRS - changed motd message in /opt/otrs/Kernel/Output/HTML/Templates/Standard/Motd.tt - admins can turn it on and off
  • 01:04 logmsgbot: krenair@mira Synchronized php-1.27.0-wmf.12/tests: https://gerrit.wikimedia.org/r/#/c/268332/ (duration: 02m 08s)
  • 01:01 logmsgbot: krenair@mira Synchronized php-1.27.0-wmf.12/includes/parser: https://gerrit.wikimedia.org/r/#/c/268332/ (duration: 02m 25s)
  • 01:00 moritzm: rebooting iridium (phabricator host) for kernel update
  • 00:42 YuviPanda: yuvipanda@labstore2001:~$ sudo lvremove backup/maps20160121040005
  • 00:41 YuviPanda: yuvipanda@labstore2001:~$ sudo lvremove backup/tools20160121020007
  • 00:04 logmsgbot: thcipriani@mira rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.12

2016-02-03

  • 23:53 moritzm: repooling restbase2002 , depooling restbase2003 for kernel/Java update
  • 23:39 moritzm: repooling restbase2001 , depooling restbase2002 for kernel/Java update
  • 23:36 logmsgbot: thcipriani@mira rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.27.0-wmf.12
  • 23:29 hashar: passing wmf12 responsibility to thcipriani . Crashing to bed myself.
  • 23:22 moritzm: depooling restbase2001 for kernel/Java update
  • 23:15 moritzm: rebooting wdqs1002 for kernel update
  • 23:08 hashar: Full script of my deployment session is on mira.codfw.wmnet:/home/hashar/wmf12-deploy.script
  • 23:07 logmsgbot: hashar@mira rebuilt wikiversions.php and synchronized wikiversions files: Clarify only testwiki and test2wiki are on php-1.27.0-wmf.12
  • 23:07 moritzm: rebooting wdqs1001 for kernel update
  • 22:51 hashar: test / test2 wikis are incredibly slow . Filled https://phabricator.wikimedia.org/T125727
  • 22:47 subbu: finished deploying parsoid sha 98619f7f
  • 22:43 logmsgbot: hashar@mira rebuilt wikiversions.php and synchronized wikiversions files: test2wiki to php-1.27.0-wmf.12
  • 22:43 hashar: sync-wikiversions "test2wiki to php-1.27.0-wmf.12"
  • 22:41 moritzm: repooling restbase1009
  • 22:38 logmsgbot: hashar@mira Finished scap: to properly sync other master tin due to l10nupdate ui mismatch (duration: 24m 27s)
  • 22:34 moritzm: repooling restbase1006 , depooling restbase1009 for kernel/Java update
  • 22:34 hashar: Still looking at test.wikipedia.org being super "slow" . scap still rebuilding though
  • 22:32 ejegg: updated payments-wiki from 1817327b4b0919ebe26bbd8b9d84fac1bd7ddb03 to fad669c99db8240b26a524aa70c85cfebd13a18c
  • 22:21 moritzm: repooling restbase1005 , depooling restbase1006 for kernel/Java update
  • 22:14 ejegg: rolled payments-wiki back to 1817327b4b0919ebe26bbd8b9d84fac1bd7ddb03
  • 22:14 hashar: https://test.wikipedia.org/ switched to 1.27.0-wmf.12
  • 22:13 logmsgbot: hashar@mira Started scap: to properly sync other master tin due to l10nupdate ui mismatch
  • 22:13 subbu: restarted parsoid on wtp1002 as a canary
  • 22:13 logmsgbot: hashar@mira Finished scap: testwiki to php-1.27.0-wmf.12 and rebuild l10n cache (with proper branches for special_extensions) (duration: 20m 23s)
  • 22:07 moritzm: repooling restbase1004 , depooling restbase1005 for kernel/Java update
  • 22:06 ejegg: updated payments-wiki from 1817327b4b0919ebe26bbd8b9d84fac1bd7ddb03 to 52afbc735ef5d759fd42bef072bed286fe3a5581
  • 22:06 subbu: starting parsoid deploy
  • 22:03 mutante: mira, tin: find /srv/mediawiki-staging/ -uid 1001 -exec chown 10002 {} \;
  • 21:53 hashar: reopened https://phabricator.wikimedia.org/T119165 l10nupdate user uid mismatch between tin and mira
  • 21:52 logmsgbot: hashar@mira Started scap: testwiki to php-1.27.0-wmf.12 and rebuild l10n cache (with proper branches for special_extensions)
  • 21:51 mutante: tin - find / -uid 1001 -exec chown 10002 {} \;
  • 21:49 mutante: tin - fixing UID of l10nupdate user (T119165)
  • 21:45 moritzm: depooling restbase1004 for kernel/Java update
  • 21:45 moritzm: repooling restbase1003
  • 21:35 hashar: mismatching uid for l10nupdate user between mira and tin
  • 21:34 logmsgbot: hashar@mira scap aborted: testwiki to php-1.27.0-wmf.12 and rebuild l10n cache (with proper branches for special_extensions) (duration: 07m 41s)
  • 21:32 moritzm: depooling restbase1003 for kernel/Java update
  • 21:27 moritzm: repooling restbase1008
  • 21:26 logmsgbot: hashar@mira Started scap: testwiki to php-1.27.0-wmf.12 and rebuild l10n cache (with proper branches for special_extensions)
  • 21:25 hashar: mira had to hard reset CentralNotice / SemanticMediaWiki / SemanticResultFormats / Validator after we pointed them from master to their proper branch, submodule attempted a rebase automatically.. That is a no no
  • 21:14 moritzm: depooling restbase1008 for kernel/Java update
  • 21:08 hashar: waiting for the submodule patch https://gerrit.wikimedia.org/r/#/c/268214/ to land and will scap again
  • 20:33 logmsgbot: hashar@mira scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="testwiki" --list-file="/srv/mediawiki-staging/wmf-config/extension-list" --output="/tmp/tmp.bTBpxD6CuI" ' returned non-zero exit status 1 (duration: 01m 13s)
  • 20:32 logmsgbot: hashar@mira Started scap: testwiki to php-1.27.0-wmf.12 and rebuild l10n cache (after RandomRootPage had a dummy entry point added)
  • 20:31 logmsgbot: demon@mira Synchronized php-1.27.0-wmf.12/extensions/RandomRootPage/: unbreak (duration: 01m 19s)
  • 20:23 logmsgbot: hashar@mira scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_2188303825" --threads=10 --lang en --quiet' returned non-zero exit status 255 (duration: 01m 49s)
  • 20:21 logmsgbot: hashar@mira Started scap: testwiki to php-1.27.0-wmf.12 and rebuild l10n cache
  • 20:20 hashar: Hacked wikiversions.json to only have testwiki on .12
  • 19:58 logmsgbot: demon@mira Synchronized wmf-config/InitialiseSettings.php: touch (duration: 01m 19s)
  • 19:49 logmsgbot: demon@mira Synchronized wmf-config/: fix wikibase/mobilefrontend config (duration: 01m 19s)
  • 19:48 robh: halting puppet on carbon for a few minutes to livehack a partition recipe change in netboot.cfg
  • 19:45 hashar: https://phabricator.wikimedia.org/T125672 blocking wmf.12 "Notice: Undefined variable: wgMFQueryPropModules in /srv/mediawiki/wmf-config/Wikibase.php on line 120"
  • 19:39 akosiaris: hot patch OTRS installation with https://github.com/OTRS/otrs/commit/c7ea6d64e02518e166fbac02f42f25dacad54342
  • 19:35 hashar: mira: manually fixed /php and /w/static/current symlinks to point back to .10 (wikiversions migrated them to .11 which we skip)
  • 19:30 moritzm: repooling restbase1002
  • 19:29 hashar: Create patches to update wikiversions.json
  • 19:24 hashar: Applying security patches on mira
  • 19:24 hashar: starting train deployment of 1.27.0-wmf.12
  • 19:09 csteipp: deployed patch for T125684
  • 19:08 moritzm: depooling restbase1002 for kernel/Java update
  • 18:38 logmsgbot: bd808@mira Synchronized wmf-config/InitialiseSettings-labs.php: Experiment one: Labs stripping HTML in beta (360e5af) (duration: 01m 19s)
  • 18:34 moritzm: rebooting californium for kernel update
  • 18:16 bblack: restarting pybal on lvs1001
  • 18:04 jynus: previous announcement was for db2011, not db2010
  • 18:02 jynus: starting slave IO thread on db2010
  • 17:32 logmsgbot: jynus@mira Synchronized wmf-config/db-eqiad.php: Repool db1060 after maintenance (duration: 01m 20s)
  • 17:22 mobrovac: restbase restarting rb1001
  • 17:03 mdholloway: mobileapps deployed 68e38ec
  • 17:02 bblack: restarting pybal on lvs1004 (not 1003!) T125397
  • 17:02 bblack: restarting pybal on lvs1003 T125397
  • 16:57 hashar: mira: updating /srv/mediawiki-staging/php-1.27.0-wmf.12 (prep deployment train)
  • 16:55 logmsgbot: thcipriani@mira Synchronized wmf-config/CirrusSearch-production.php: SWAT: Return more like search queries to codfw gerrit:268097 (duration: 01m 17s)
  • 16:45 logmsgbot: thcipriani@mira Synchronized wmf-config/CommonSettings.php: SWAT: Remove unused/no longer existing item-create oauth grant gerrit:265447 (duration: 01m 18s)
  • 16:39 logmsgbot: thcipriani@mira Synchronized wmf-config: SWAT: Enable math data type on test wikidata + test wikipedias gerrit:268086 (duration: 01m 18s)
  • 16:32 logmsgbot: thcipriani@mira Synchronized wmf-config/mobile.php: SWAT: Remove section collapsing config gerrit:267776 (duration: 01m 18s)
  • 16:28 akosiaris: OTRS migration to 4.0 completed, starting upgrade to 5.0
  • 16:24 logmsgbot: thcipriani@mira Synchronized wmf-config/CommonSettings.php: SWAT: MW parsoid URLs: s/parsoidcache/parsoid/ gerrit:267234 (duration: 01m 18s)
  • 16:18 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: Add 2 sites to $wgCopyUploadsDomains gerrit:262893 (duration: 01m 18s)
  • 16:13 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: Just use the default MobileFrontend specified page actions. Part II gerrit:267807 (duration: 01m 18s)
  • 16:11 logmsgbot: thcipriani@mira Synchronized wmf-config/mobile.php: SWAT: Just use the default MobileFrontend specified page actions. Part I gerrit:267807 (duration: 02m 14s)
  • 15:41 hashar: mira symlink pointing to current version got changed to wmf.11 by the checkoutMediaWiki script. Manually changed to proper wmf.10 https://phabricator.wikimedia.org/T125475#1994078
  • 15:32 jynus: restart and reconfigure mysql in db1060
  • 15:30 hashar: MediaWiki 1.27.0-wmf.12, from 1.27.0-wmf.12, successfully checked out.
  • 15:23 logmsgbot: jynus@mira Synchronized wmf-config/db-eqiad.php: Depool db1060 (duration: 00m 43s)
  • 15:21 hashar: mira: cloning 1.27.0-wmf.12 (no link updates)
  • 15:15 bblack: rebooting cp1060 (depooled/downtimed)
  • 15:11 bblack: depooling cp1060 temporarily from cache_mobile varnish backends
  • 14:56 logmsgbot: jynus@mira Synchronized wmf-config/db-eqiad.php: Repool db1054 with low weight, repool db1067 with original weight (duration: 01m 22s)
  • 14:50 bblack: rebooting cp1008 for kernel
  • 14:28 godog: investigating uwsgi processes for graphite-web not coming up after reboot
  • 14:10 moritzm: rebooting graphite1001 for kernel update
  • 13:41 godog: powercycle ms-be2015
  • 13:39 jynus: restarting and reconfiguring mysql at db1054
  • 13:27 logmsgbot: jynus@mira Synchronized wmf-config/db-eqiad.php: Repool db1067 at low weight; depool db1054 (duration: 01m 16s)
  • 11:45 jynus: restarting and reconfiguring mysql at db1067
  • 11:11 moritzm: repooling restbase1001
  • 11:04 akosiaris: OTRS database upgraded to 3.3, moving on with 4.0
  • 11:00 logmsgbot: jynus@mira Synchronized wmf-config/db-eqiad.php: Repool db1063 at 100% load; depool db1067 for maintenance (duration: 01m 16s)
  • 10:48 moritzm: depooling restbase1001 for kernel/Java update
  • 10:37 _joe_: ending the load test on the eqiad apaches
  • 10:11 moritzm: reboot francium for kernel update
  • 09:53 jynus: m2 backup finished on /srv/backups/2016-02-03_08-51-06, filename 'db1020-bin.000842', position 220103947
  • 09:50 moritzm: restarting neodymium for kernel update
  • 09:49 _joe_: doing some basic load test on appservers in eqiad
  • 08:52 akosiaris: stop otrs-daemon on mendelevium
  • 08:51 jynus: starting mysql backup on db1020 (/srv/backups)
  • 08:44 akosiaris: stop slave on db2011, db1020's (m2-master) slave, for OTRS migration. DO NOT ENABLE
  • 08:40 akosiaris: stop exim4, cron, apache2 on iodine, mendelevium
  • 08:39 akosiaris: disabling puppet on iodine, mendelevium, OTRS migration
  • 08:24 logmsgbot: jynus@mira Synchronized wmf-config/db-eqiad.php: Repool db1063 with low weight (duration: 01m 20s)

2016-02-02

  • 23:13 logmsgbot: demon@mira Finished scap: everything re-sync one more time for good measure (duration: 17m 04s)
  • 22:56 logmsgbot: demon@mira Started scap: everything re-sync one more time for good measure
  • 22:50 bblack: repooling scap proxies: mw10033, mw1070, mw1097, mw1216
  • 22:45 chasemp: restart hhvm & apache2 on mw1235.eqiad.wmnet
  • 22:44 _joe_: restarted hhvm on mw1231, stat_cache again
  • 22:42 logmsgbot: demon@mira Finished scap: resync final batch with master (duration: 06m 48s)
  • 22:35 logmsgbot: demon@mira Started scap: resync final batch with master
  • 22:31 logmsgbot: demon@mira Finished scap: re-sync batch of mw1136-50, mw1190-1220, mw2150-mw2200 with master (duration: 09m 33s)
  • 22:22 logmsgbot: demon@mira Started scap: re-sync batch of mw1136-50, mw1190-1220, mw2150-mw2200 with master
  • 22:20 ori: restarted HHVM on mw1243. Lock-up. Backtrace in /tmp/hhvm.2897.bt
  • 22:20 logmsgbot: demon@mira Finished scap: re-sync batch of mw1101-1135,1240-1260, 2101-2150 with master (duration: 12m 51s)
  • 22:07 logmsgbot: demon@mira Started scap: re-sync batch of mw1101-1135,1240-1260, 2101-2150 with master
  • 22:00 logmsgbot: demon@mira Finished scap: re-sync batch of mw1151-mw1225, mw2174-mw2214 with master (duration: 11m 24s)
  • 21:49 logmsgbot: demon@mira Started scap: re-sync batch of mw1151-mw1225, mw2174-mw2214 with master
  • 21:45 logmsgbot: demon@mira Finished scap: re-sync batch of mw1051-1100, mw2051-2100 with master (duration: 13m 41s)
  • 21:31 logmsgbot: demon@mira Started scap: re-sync batch of mw1051-1100, mw2051-2100 with master
  • 21:28 logmsgbot: demon@mira Finished scap: re-sync batch of mw1025-1050 and mw2007-mw2050 with master (2nd try) (duration: 14m 33s)
  • 21:27 _joe_: depooling eqiad scap-proxies
  • 21:13 logmsgbot: demon@mira Started scap: re-sync batch of mw1025-1050 and mw2007-mw2050 with master (2nd try)
  • 21:04 logmsgbot: demon@mira scap aborted: re-sync batch of mw1025-1050 and mw2007-mw2050 with master (duration: 10m 11s)
  • 20:54 logmsgbot: demon@mira Started scap: re-sync batch of mw1025-1050 and mw2007-mw2050 with master
  • 20:32 hashar: mw1114-mw1119 are canary api appservers Finished syncing
  • 20:28 ori: restarted hhvm on mw1116
  • 20:17 hashar: Running sync-common on mw1114-mw1119 (canary api appservers)
  • 20:16 ostriches: mira: removed untracked wmf-config/x.php testing file
  • 20:11 ori: Running sync-common on canary app servers (mw1017-mw1025)
  • 19:46 hashar: Running sync-common on mw1260 (video scaler)
  • 19:40 ori: Running sync-common on all jobscalers
  • 19:35 ori: Running sync-common on mw1259 (video scaler) and mw1153 (image scaler) too
  • 19:29 ori: Running sync-common on mw100[123]
  • 18:59 _joe_: running sync-common on mw1020
  • 18:54 _joe_: repooled mw1119
  • 17:45 hashar: mira /srv/mediawiki-staging git submodule update --init --recursive
  • 17:43 hashar: mw1119 sync-common
  • 17:37 godog: disable unused swift container-sync for wikibooks-ka-local-thumb wikibooks-hr-local-thumb wikibooks-km-local-thumb wikibooks-sk-local-thumb wikibooks-tr-local-thumb wikipedia-it-local-thumb.fc
  • 17:36 hashar: mw1119:/srv/mediawiki/wmf-config/event-schemas is empty
  • 17:31 _joe_: depooled mw1119, partial sync
  • 16:59 hashar: files were /srv/mediawiki/docroot/wikimedia.org/WikipediaMobileFirefoxOS/.git and /srv/mediawiki/docroot/wikimedia.org/WikipediaMobileFirefoxOS/js/lib/MobileFrontend/.git
  • 16:58 ostriches: mw1017: removed stray .git directory from WikipediaFirefoxMobileOS or w/e. It shouldn't be there anyway. sync-common is happy again on it
  • 16:48 hashar: tin /srv/mediawiki-staging  : running git submodule update --init --recursive
  • 16:47 hashar: tin /srv/mediawiki-staging  : running git submodule update --init
  • 16:40 hashar: mw1017 sync-common --verbose
  • 16:35 _joe_: sync-common on mw2030 and mw1161; re-enable puppet, jobrunner, jobchron on mw1161
  • 16:34 _joe_: restarted puppet and rsync on both tin and mira, removed comments on the l10nupdate job on tin
  • 16:23 logmsgbot: thcipriani@mira rebuilt wikiversions.php and synchronized wikiversions files: rebuild wikiversion.php
  • 14:57 godog: disable swift container-sync for wikipedia-it-local-public.a7
  • 14:43 hashar: tin /srv/mediawiki-staging/multiversion/checkoutMediaWiki 1.27.0-wmf.10 php-1.27.0-wmf.10
  • 14:43 hashar: tin /srv/mediawiki-staging/multiversion/checkoutMediaWiki 1.27.0-wmf.9 php-1.27.0-wmf.9
  • 14:43 hashar: tin /srv/mediawiki-staging/multiversion/checkoutMediaWiki 1.27.0-wmf.8 php-1.27.0-wmf.8
  • 14:21 hashar: starting rebuilding /srv/mediawiki-staging from scratch on tin (not mira)
  • 14:20 hashar: starting rebuilding /srv/mediawiki-staging from scratch on mira
  • 14:04 bblack: nevermind, not looking at eeden
  • 14:04 bblack: looking at eeden
  • 13:58 moritzm: rebooting eeden for kernel update
  • 13:09 moritzm: rolling reboot of scb* (for kernel update)
  • 13:02 akosiaris: reboot dubnium for kernel upgrades
  • 13:01 akosiaris: reboot pollux for kernel upgrades
  • 12:45 moritzm: rebooting baham for kernel update
  • 12:20 _joe_: stopping rsync on mira too, to avoid accidental deploys
  • 12:15 _joe_: stopped puppet on mira, added a big warning in the motd
  • 12:15 _joe_: stopped rsync, puppet, l10nupdate cronjob on tin
  • 12:06 _joe_: stopped rsync on tin to avoid problems
  • 11:38 moritzm: rolling reboot of aqs* (for kernel update)
  • 11:24 hashar_: Restarting Zuul. Stuck in a dependency loop :(
  • 11:12 jynus: restarting and reconfiguring mysql at db1063
  • 10:51 _joe_: stopped jobrunner on mw1161 after failed sync-common
  • 10:44 logmsgbot: jynus@mira Synchronized wmf-config/db-eqiad.php: Depool db1063, repool db1036 (duration: 00m 21s)
  • 10:00 jynus: reconfigure and upgrade db1036
  • 09:51 logmsgbot: jynus@mira Synchronized wmf-config/db-eqiad.php: Testing scap-reduce db1018 weight (duration: 00m 21s)
  • 09:42 logmsgbot: jynus@mira Synchronized wmf-config/db-eqiad.php: Depool db1036, repool db1021 (duration: 00m 22s)
  • 09:38 hashar: Jenkins is fully up and operational
  • 09:36 jynus: armed keyholder on tin
  • 09:34 dcausse: elastic (codfw and eqiad): unfreezing indices
  • 09:33 moritzm: restarting gerrit on ytterbium for java security update
  • 09:33 _joe_: re-syncing tin homes
  • 09:32 hashar: gallium: apt-get upgrade | Restarting Jenkins
  • 09:12 logmsgbot: jynus@mira Synchronized wmf-config/db-eqiad.php: Depool db1036, repool db1021 (duration: 00m 21s)
  • 09:08 dcausse: elastic (codfw and eqiad): freezing indices to stop titlesuggest maint scripts
  • 09:03 godog: repool restbase1007 via confctl
  • 08:13 jynus: restarting and upgrading db1021
  • 08:02 logmsgbot: jynus@mira Synchronized wmf-config/db-eqiad.php: Pool db1018; Depool db1021 (duration: 00m 20s)
  • 07:46 jynus: https://phabricator.wikimedia.org/rOMWC2ea9167221d11eb1880e4d26eae64a85cb9b2697 and https://phabricator.wikimedia.org/rOMWCa55d2bf8cd3a2853fac35d5b8239b8e8c2fe6a0f merged but not deployed
  • 06:58 _joe_: reimaging tin.eqiad.wmnet
  • 01:30 logmsgbot: ebernhardson@mira Finished scap: Add Cookie statement link to footer of all WMF wikis per legal (duration: 19m 42s)
  • 01:11 logmsgbot: ebernhardson@mira Started scap: Add Cookie statement link to footer of all WMF wikis per legal
  • 01:07 logmsgbot: ebernhardson@mira scap failed: CalledProcessError Command '/srv/deployment/scap/scap/bin/refreshCdbJsonFiles --directory="/srv/mediawiki-staging/php-1.27.0-wmf.10/cache/l10n" --threads=10 ' returned non-zero exit status 255 (duration: 03m 31s)
  • 01:03 logmsgbot: ebernhardson@mira Started scap: Add Cookie statement link to footer of all WMF wikis per legal
  • 00:31 logmsgbot: ebernhardson@mira scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="cawikibooks" --outdir="/tmp/scap_l10n_1684485672" --threads=10 --quiet' returned non-zero exit status 255 (duration: 02m 35s)
  • 00:30 mobrovac: restbase deploy end of c3bd864
  • 00:29 logmsgbot: ebernhardson@mira Started scap: Add Cookie statement link to footer of all WMF wikis per legal
  • 00:26 logmsgbot: ebernhardson@mira Synchronized wmf-config/logging.php: Revert "monolog: Ensure that context data added by WebProcessor is utf-8 safe" (duration: 01m 27s)
  • 00:23 logmsgbot: ebernhardson@mira Synchronized wmf-config/CirrusSearch-production.php: Move morelike query load back to eqiad to allow load testing on codfw (duration: 01m 38s)

2016-02-01

  • 23:51 mobrovac: restbase deploy start of c3bd864 on canary rb1001
  • 19:28 logmsgbot: ori@mira Synchronized docroot/wikipedia.org/speed-tests: I5b48a491390: Speed trials: add preconnect (duration: 01m 27s)
  • 18:54 bblack: banned obj.http.Content-Length == 13817 on all cache_text
  • 18:54 mutante: LDAP - added elukey to "ops" group
  • 18:11 mutante: planet1001 - rebooting for upgrade
  • 17:54 hoo: restarted hhvm on mw1253
  • 17:06 logmsgbot: thcipriani@mira Synchronized wmf-config: SWAT: Use extension registration for Graph gerrit:266433 (duration: 01m 29s)
  • 16:59 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable SandboxLink on or.wikipedia.org gerrit:267194 (duration: 01m 31s)
  • 16:54 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable WikidataPageBanner on es.wikivoyage gerrit:267195 (duration: 01m 29s)
  • 16:52 _joe_: restarted pybal on lvs1001
  • 16:47 _joe_: installing the new HHVM package to the api appserver cluster in eqiad
  • 16:38 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: Set WikidataPageBanner namespaces on fr.wikivoyage gerrit:266541 (duration: 01m 26s)
  • 16:32 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: Namespace configuration on cu.wikipedia gerrit:265885 (duration: 01m 26s)
  • 16:26 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: Centralise all VisualEditor feedback pages except for a few wikis gerrit:258206 (duration: 01m 30s)
  • 16:22 logmsgbot: thcipriani@mira Synchronized dblists/visualeditor-default.dblist: SWAT: Enable VisualEditor by default for some other wikis gerrit:264765 (duration: 01m 58s)
  • 16:05 ema: hhvm restarted on mw1072
  • 15:54 logmsgbot: krenair@mira Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 01m 52s)
  • 15:48 bblack: restarted pybal on lvs1004 (lvs1003 above was a bad log message!)
  • 15:42 bblack: restarted pybal on lvs1003
  • 15:13 bblack: cp3042 repooled
  • 15:10 ema: restarting hhvm on mw1057
  • 14:33 chasemp: labstore1002 cfg scheduling
  • 14:04 godog: set ms-be1019 swift weight to 4000
  • 13:33 moritzm: rolling reboot of xenon/cerium/praseodymium for kernel update (and updating to new openjdk-8)
  • 12:40 _joe_: depooling cp3042 from esams uploads
  • 12:15 _joe_: backing up tin homes before reimaging
  • 11:59 moritzm: rolling reboot of ms-be1016 to ms-be1021 for kernel update
  • 11:39 moritzm: uploaded openjdk-8 8u72-b15-1~bpo8+1 for jessie-wikimedia to carbon
  • 11:34 moritzm: uploaded openssl 1.0.2f for jessie-wikimedia to carbon
  • 11:19 godog: repool restbase1007
  • 10:32 godog: reboot ms-be1010, xfs
  • 10:27 jynus: partitioning revision and logging for db2037 and db2044 (s4)
  • 00:04 logmsgbot: tstarling@mira Synchronized php-1.27.0-wmf.11/includes: (no message) (duration: 01m 31s)

2016-01-31

  • 23:58 logmsgbot: krenair@mira Synchronized php-1.27.0-wmf.11/extensions/VisualEditor/extension.json: https://gerrit.wikimedia.org/r/#/c/267617/ (duration: 01m 28s)
  • 22:31 ori: restarted parsoid-rt-client.service
  • 22:14 ori: Updated parsoid on ruthenium and restarted parsoid-rt-client on ruthenium, per subbu's request.
  • 22:03 bd808: backfilled missing data in https://tools.wmflabs.org/sal/production from https://wikitech.wikimedia.org/wiki/Server_Admin_Log
  • 21:37 bd808: https://tools.wmflabs.org/sal/production missing data from 2016-01-30 until now
  • 21:33 logmsgbot: ori@mira Synchronized php-1.27.0-wmf.10/includes/jobqueue/jobs/HTMLCacheUpdateJob.php: Live-hacked wfDebugLog() call for T124418 (duration: 01m 31s)
  • 16:01 tgr: changed wikiversions.php on mw1017 to serve wmf.10 for SessionManager-related debugging
  • 05:35 legoktm: restarted extensions/CentralAuth/maintenance/resetGlobalUserTokens.php
  • 02:32 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Jan 31 02:32:12 UTC 2016 (duration 7m 11s)
  • 02:25 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.10) (duration: 10m 14s)

2016-01-30

  • 23:20 logmsgbot: bd808@mira rebuilt wikiversions.php and synchronized wikiversions files: Revert all wikis to 1.27.0-wmf.10 (again)
  • 23:01 logmsgbot: bd808@mira Synchronized wmf-config/InitialiseSettings.php: Revert Enable debug level session logging to fluorine (17bfb06) (duration: 01m 28s)
  • 22:36 logmsgbot: bd808@mira Synchronized wmf-config/InitialiseSettings.php: Enable debug level session logging to fluorine (5ac9412) (duration: 01m 26s)
  • 18:43 _joe_: updated visualdiff, restarted parsoid-vd
  • 13:00 godog: discard preserved cache on ms-be2003, powercycle
  • 03:40 Krenair: Deleted old /srv/mediawiki/php-1.27.0-wmf.[1-5] directories across the cluster to match the deployment tree, T124567
  • 02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Jan 30 02:31:56 UTC 2016 (duration 7m 2s)
  • 02:24 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.11) (duration: 10m 24s)
  • 00:08 logmsgbot: bd808@mira Synchronized php-1.27.0-wmf.11/includes/session/SessionBackend.php: Remove proposed fix for T125267 (duration: 01m 33s)

2016-01-29

  • 23:53 jynus: restarted db1018 replication (and its codfw slaves) after a (somewhat) failed maintenance
  • 23:41 mutante: ruthenium - restart parsoid-rt-client, parsoid-vd-client
  • 23:37 mutante: ruthenium - git pull origin in /srv/visualdiff/
  • 23:22 logmsgbot: bd808@mira Synchronized php-1.27.0-wmf.11/includes/session/SessionBackend.php: Testing proposed fix for T125267 (duration: 01m 26s)
  • 22:52 jynus: powercycling cp3042 to test it is really the broken one
  • 22:37 jynus: powercycle cp3049, not 42
  • 22:37 jynus: powercycle cp3042
  • 22:27 mutante: cp3042 - md0: unknown partition table
  • 22:23 mutante: powercycled cp1049
  • 22:06 mutante: powercycle cp3049
  • 21:13 mutante: bromine - stop and remove rsync service
  • 20:16 logmsgbot: aaron@mira Synchronized wmf-config/CommonSettings.php: Use the logical redis definition for GettingStarted (duration: 01m 26s)
  • 19:36 jynus: reinstall db1018
  • 18:11 jynus: creating special partitioning for db2037 and db2044 (ETA:5 days, lag)
  • 18:01 jynus: creating special partitioning for db2034 and db2042 (ETA:5 days, lag)
  • 17:51 logmsgbot: bd808@mira Synchronized wmf-config/InitialiseSettings.php: Stop the first survey in fawiki and eswiki (f89621d) (duration: 01m 25s)
  • 17:44 logmsgbot: bd808@mira Synchronized php-1.27.0-wmf.11/includes/api/ApiMain.php: Log user-agents that are using HTTP when HTTPS is preferred (55ac0b7) (duration: 01m 26s)
  • 17:41 logmsgbot: bd808@mira Synchronized wmf-config/CommonSettings.php: Grant autocreateaccount to anons on loginwiki (d916008) (duration: 01m 27s)
  • 17:39 logmsgbot: bd808@mira Synchronized php-1.27.0-wmf.11/extensions/CentralAuth/includes/session/CentralAuthSessionProvider.php: CentralAuth: Take auto-creation into account (f526ef1) (duration: 01m 28s)
  • 17:35 logmsgbot: bd808@mira Synchronized php-1.27.0-wmf.11/includes/session/SessionBackend.php: SessionManager: Save user name to metadata even if the user doesn't exist locally (a39b4ac) (duration: 01m 29s)
  • 17:01 jynus: restarting mysql at db1018
  • 16:50 robh: parsoid-vd restart was due to subbu irc request (i wasnt just randomly restarting things ;)
  • 16:47 robh: restarting parsoid-vd & parsoid-vd-client on ruthenium
  • 16:33 ottomata: uinstalling impala in analytics cluster
  • 15:45 bblack: upgrade packages (incl kernel) on eqiad caches hosts (cp1xxx)
  • 15:37 logmsgbot: jynus@mira Synchronized wmf-config/db-eqiad.php: Depool db1018 for maintenance (duration: 01m 49s)
  • 15:32 akosiaris: remove all networking configuration from asw-b-eqiad switch for nas1001-a, nas1001-b. Leave just descriptions
  • 15:21 bblack: upgrading packages (incl kernel) on esams cache hosts (cp3xxx) (codfw, ulsfo already done)
  • 15:11 akosiaris: powering off nas1001-a.eqiad.wmnet. https://phabricator.wikimedia.org/T124156
  • 15:08 akosiaris: powering off nas1001-b.eqiad.wmnet. https://phabricator.wikimedia.org/T124156
  • 15:01 elukey: re-enabled puppet on analytics1027
  • 14:39 elukey: stopped kafka (service) on kafka1012 (the host that caused the outage)
  • 14:24 moritzm: rebooting bohrium for kernel update
  • 14:04 _joe_: installing the new hhvm package on all the codfw appserver
  • 13:43 _joe_: installing the new HHVM package to the canary appservers (main and api)
  • 12:30 paravoid: force-rebooting pollux
  • 11:43 _joe_: uploaded hhvm_3.6.5+dfsg1-1+wm8 to trusty-wikimedia
  • 11:22 moritzm: rolling restart of swift in codfw
  • 11:14 elukey: disabled puppet on analytics1027 due to issues with Camus and HDFS
  • 10:17 moritzm: rolling restart of swift in esams
  • 02:32 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Jan 29 02:32:56 UTC 2016 (duration 7m 28s)
  • 02:25 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.11) (duration: 10m 40s)
  • 01:31 logmsgbot: ori@mira Synchronized wmf-config: I83da57cf: Enable persistent redis connections for job runners (duration: 01m 11s)
  • 01:03 logmsgbot: krenair@mira Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/267186/ (duration: 01m 09s)
  • 01:01 logmsgbot: krenair@mira Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/265292/ (duration: 01m 14s)
  • 00:57 logmsgbot: krenair@mira Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/267071/ (duration: 01m 11s)
  • 00:53 logmsgbot: krenair@mira Synchronized wmf-config/CirrusSearch-production.php: https://gerrit.wikimedia.org/r/#/c/266995/ (duration: 01m 11s)
  • 00:50 yurik: synced latest graphoid
  • 00:49 logmsgbot: krenair@mira Synchronized php-1.27.0-wmf.11/extensions/MobileFrontend/resources/skins.minerva.editor/init.js: https://gerrit.wikimedia.org/r/#/c/267168/ (duration: 01m 12s)
  • 00:45 logmsgbot: krenair@mira Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/267053/ (duration: 01m 10s)
  • 00:43 logmsgbot: krenair@mira Synchronized wmf-config/CirrusSearch-common.php: https://gerrit.wikimedia.org/r/#/c/267053/ (duration: 01m 10s)
  • 00:42 logmsgbot: krenair@mira Synchronized tests/cirrusTest.php: https://gerrit.wikimedia.org/r/#/c/267053/ (duration: 01m 11s)
  • 00:35 logmsgbot: krenair@mira Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/267025/ (duration: 01m 12s)
  • 00:25 logmsgbot: krenair@mira Synchronized php-1.27.0-wmf.11/extensions/Graph/modules/graph2.js: https://gerrit.wikimedia.org/r/#/c/267065/ (duration: 01m 11s)
  • 00:17 logmsgbot: krenair@mira Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/267060/ (duration: 01m 12s)
  • 00:02 logmsgbot: krenair@mira Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/267189/2 (duration: 01m 11s)

2016-01-28

  • 23:51 mutante: caesium - stop puppet, shutdown server, remove from icinga, clean puppet cert ...
  • 23:46 Tim: on ruthenium installing build dependencies and compiling uprightdiff for test
  • 23:20 logmsgbot: ori@mira Synchronized php-1.27.0-wmf.11/includes/api/ApiStashEdit.php: Ia4196eba9: Add ParserOutputStashForEdit hook for extension cache warming (duration: 01m 10s)
  • 23:17 logmsgbot: tgr@mira Synchronized php-1.27.0-wmf.11/includes/session/SessionManager.php: T125161 (duration: 01m 11s)
  • 22:58 ottomata: restoring MobileWebSectionUsage_14321266 from db1047 to dbstore1002 using mysqlimport
  • 22:23 bblack: starting cache_mobile->cache_text conversion in eqiad - https://phabricator.wikimedia.org/T109286
  • 22:09 bblack: eqiad pybal->etcd conversion done
  • 22:01 logmsgbot: dduvall@mira Synchronized php-1.27.0-wmf.11/extensions/WikimediaEvents/WikimediaEventsHooks.php: deploying fix for T125151 (duration: 01m 15s)
  • 21:59 mutante: releases.wm.org - switched backend to bromine
  • 21:58 bblack: converting active eqiad LVS/pybal to etcd
  • 21:56 mutante: caesium - stopped apache
  • 21:31 logmsgbot: ori@mira Synchronized php-1.27.0-wmf.11/extensions/AbuseFilter: I13fcc3ce4: Updated mediawiki/core Project: mediawiki/extensions/AbuseFilter 19baa3b6e51b8fe6baf6e3ce7e590060e8e6eec9 (duration: 01m 11s)
  • 21:27 bblack: converting backup/inactive eqiad LVS/pybal to etcd
  • 21:16 logmsgbot: dduvall@mira rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.27.0-wmf.11
  • 20:54 mutante: sca1001 - stop mathoid,graphoid,citoid
  • 20:52 mutante: sca1002 - stop mathoid,graphoid,citoid
  • 20:50 logmsgbot: dduvall@mira Synchronized php-1.27.0-wmf.11: syncing 1.27.0-wmf.11 for T125114 and https://gerrit.wikimedia.org/r/#/c/267128/ (duration: 03m 30s)
  • 20:25 bblack: depool -> reboot cp4008 (ulsfo text, trying new kernel with live traffic)
  • 20:00 bblack: depool -> reboot cp4011 (ulsfo mobile, currently unused for traffic - testing local conftool-scripts depool + new kernel)
  • 19:55 logmsgbot: ori@mira Synchronized wmf-config: Iea2573ccfbe: Revert "Autopromotion: remove deprecated onView event, fix INGROUPS" (duration: 02m 13s)
  • 19:43 ori: added tgr and marxarelli to security group on phab
  • 19:26 ottomata: kafka preferred-replica-election to rebalanace analytics-eqiad brokers
  • 18:22 elukey: rebooting analytics1001 for new kernel upgrade
  • 18:21 yurik: deployed graphoid
  • 17:43 elukey: rebooting analytics1002.eqiad.wmnet (Hadoop master's slave) for kernel upgrade
  • 17:39 urandom: finished deploying configuration change (https://gerrit.wikimedia.org/r/266299) to restbase staging
  • 17:38 robh: neglected to log i ifinished icinga/neon updates and its back to normal service (never interrrupted)
  • 17:38 urandom: restarting restbase on restbase200[1-3].codfw.wmnet (restbase staging)
  • 17:34 urandom: forcing puppet run on restbase200[1-3].codfw.wmnet (restbase staging)
  • 17:30 urandom: forcing puppet run on praseodymium.eqiad.wmnet, and restarting restbase (staging env)
  • 17:27 urandom: restarting restbase on xenon.eqiad.wmnet (restbase staging)
  • 17:25 urandom: forcing puppet run on xenon.eqiad.wmnet (restbase staging)
  • 17:21 urandom: restarting restbase on cerium.eqiad.wmnet
  • 17:18 urandom: forcing puppet run on cerium.eqiad.wmnet (restbase staging)
  • 17:18 robh: pushing icinga updates (shouldnt affect service but others shouldnt also try to update neon right now)
  • 17:17 logmsgbot: krenair@mira Synchronized README: testing (duration: 02m 08s)
  • 17:15 urandom: disabling pupplet on restbase staging hosts
  • 17:01 logmsgbot: krenair@mira Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/266957/ (duration: 02m 15s)
  • 16:52 logmsgbot: krenair@mira Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/267040/ (duration: 02m 13s)
  • 16:48 cmjohnson1: mw1172, mw1178,mw1217, mw1257 powering off task# T124642
  • 16:45 logmsgbot: krenair@mira Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/264219/ (duration: 02m 12s)
  • 16:42 logmsgbot: krenair@mira Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/264219/ (duration: 02m 12s)
  • 16:37 Krenair: Downloaded and `chmod +x`'d mira:/srv/mediawiki-staging/.git/hooks/commit-msg
  • 16:29 mdholloway: mobileapps deployed 7583148, reverting in part 869ec35
  • 16:25 logmsgbot: krenair@mira Synchronized wmf-config/InitialiseSettings.php: rv (duration: 02m 10s)
  • 16:25 bblack: upgrading packages (incl kernel) on all codfw caches
  • 16:19 logmsgbot: krenair@mira Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/266955/ (duration: 02m 14s)
  • 16:13 logmsgbot: krenair@mira Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/266564/ (duration: 02m 12s)
  • 16:05 logmsgbot: krenair@mira Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/264733/ (duration: 02m 11s)
  • 15:39 bblack: kafka1012 booted up normally
  • 15:39 mdholloway: mobileapps deployed 869ec35
  • 15:37 bblack: rebooting kafka1012
  • 15:36 bblack: kafka1012: manually edited fstab, s/sdb1/sdb3/, s/sdc3/sdc1/, and now the filesystems mount and data looks right
  • 15:23 bblack: powering up kafka1012
  • 14:09 moritzm: rebooting serpens/seaborgium for kernel update
  • 13:58 logmsgbot: faidon@mira Synchronized wmf-config/InitialiseSettings.php: depool kafka1012 (duration: 02m 10s)
  • 13:31 bblack: citoid and cxserver public hostnames moving to cache_text
  • 12:59 moritzm: rebooting rutherfordium (peopleweb) for kernel update
  • 12:53 elukey: stopping kafka on kafka1012 + host reboot for kernel upgrade
  • 12:23 jynus: generating empty schema for new codfw parsercaches
  • 12:14 logmsgbot: jynus@mira Synchronized wmf-config/db-codfw.php: New parsercache servers for codfw datacenter (duration: 03m 10s)
  • 12:11 logmsgbot: jynus@mira Synchronized wmf-config/db-eqiad.php: New parsercache servers for codfw datacenter (duration: 02m 15s)
  • 12:07 jynus: pooling new parsercaches for codfw datacenter
  • 12:01 moritzm: powercycled mw1163, was unreachable after reboot of the jobrunners (but now up again after powercycle via mgmt)
  • 11:31 elukey: disabled puppet on analytics1027 due to some issues with camus and hdfs
  • 10:42 moritzm: rebooted parsoid systems in codfw for kernel update, rolling reboot for eqiad
  • 10:39 _joe_: rolling reboot of jobrunners in eqiad
  • 02:46 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.11) (duration: 06m 16s)
  • 02:41 logmsgbot: tgr@mira Synchronized php-1.27.0-wmf.11/includes/: deploy SessionManager patch for T124971: gerrit 266944, 266946 (duration: 03m 20s)
  • 02:27 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.10) (duration: 10m 21s)
  • 01:03 logmsgbot: krenair@mira Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/264460/ (duration: 02m 30s)
  • 00:58 logmsgbot: krenair@mira Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/264066/ (duration: 02m 26s)
  • 00:46 logmsgbot: krenair@mira Synchronized php-1.27.0-wmf.11/extensions/Gather/resources: https://gerrit.wikimedia.org/r/#/c/266793/ and https://gerrit.wikimedia.org/r/#/c/266792/ (duration: 02m 23s)
  • 00:41 logmsgbot: krenair@mira Synchronized php-1.27.0-wmf.11/extensions/Flow/: https://gerrit.wikimedia.org/r/#/c/266939/ (duration: 02m 27s)
  • 00:27 logmsgbot: krenair@mira Synchronized php-1.27.0-wmf.10/extensions/Flow/includes: https://gerrit.wikimedia.org/r/#/c/266938/ (duration: 02m 29s)
  • 00:09 logmsgbot: krenair@mira Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/266945/ (duration: 02m 36s)

2016-01-27

  • 22:36 robh: restarting parsoid-rt-client service on ruthenium
  • 22:29 ottomata: starting mysqldump of MobileWebSectionUsage_14321266 from db1047 into m4-master
  • 21:45 yurik: updated graphoid on scb*
  • 21:29 mdholloway: mobileapps deployed 6f35859
  • 21:26 cscott: updated OCG to version 64050af0456a43344b32e3e93561a79207565eaf
  • 21:26 logmsgbot: ori@mira Synchronized docroot and w: (no message) (duration: 02m 26s)
  • 19:48 YuviPanda: started nfs-exports daemon on labstore1001, had been dead for a few days
  • 19:32 mutante: stat1002 - redis.exceptions.ConnectionError: Error connecting to mira.codfw.wmnet:6379. timed out.
  • 19:31 mutante: stat1002 - running puppet, was reported as last run about 4 hours ago but not deactivated
  • 19:14 logmsgbot: dduvall@mira rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.11
  • 19:07 ejegg: set donation queue consumer time limit back to 90 sec
  • 18:49 logmsgbot: jynus@mira Synchronized wmf-config/db-eqiad.php: Repool pc1006 after cloning (duration: 02m 25s)
  • 18:48 bd808: HHVM on mw1019 still dying on a regular basis with "Lost parent, LightProcess exiting"
  • 18:00 csteipp: deploy patch for T103239
  • 17:50 csteipp: deploy patch for T97157
  • 17:47 jynus: migrating ruthenium parsoid-test database to m5-master
  • 17:27 elukey: rebooting analytics105* hosts to upgrade their kernel
  • 17:16 elukey: rebooting analytics1035.eqiad.wmnet for kernel upgrade
  • 16:23 ejegg: updated SmashPig from 072c7ec6ed94e7074ba35b7986d5dde94866fe2f to 97629339994bffe8831a9067f5e9c21fa423586b
  • 16:22 logmsgbot: thcipriani@mira Synchronized php-1.27.0-wmf.11/extensions/CentralAuth/includes/CentralAuthUtils.php: SWAT: Preserve certain keys when updating central session gerrit:266672 (duration: 02m 28s)
  • 16:11 logmsgbot: thcipriani@mira Synchronized php-1.27.0-wmf.11/extensions/CentralAuth/includes/session/CentralAuthSessionProvider.php: SWAT: Avoid forceHTTPS cookie flapping if core and CA are setting the same cookie gerrit:266671 (duration: 02m 26s)
  • 16:03 elukey: rebooting analytics 1043 -> 1050 for kernel upgrade.
  • 15:47 elukey: rebooting analytics 1026, 1040 -> 1042 due to kernel upgrade.
  • 14:58 jynus: cloning persercache contents from pc1003 to pc1006
  • 14:45 elukey: rebooting analytics 1036 to 1039 for kernel upgrade
  • 14:35 elukey: analytics 1035 hasn't been rebooted because it is a Hadoop Journal Node (will be restarted in the end)
  • 14:04 elukey: rebooting analytics 1032 to 1035 for kernel upgrades
  • 14:03 logmsgbot: jynus@mira Synchronized wmf-config/db-eqiad.php: Depool pc1003 for cloning to pc1006 (duration: 02m 30s)
  • 13:59 jynus: about to going new hardware/OS/mariadb-only for parsercache service
  • 13:32 elukey: rebooting analytics1030/1031 for kernel upgrade
  • 13:15 akosiaris: rebooting fermium for kernel upgrades
  • 13:10 elukey: rebooting analytics1029 for kernel upgrade
  • 12:29 moritzm: rebooting analytics1028 for kernel update
  • 10:25 ema: restarting apache2 and hhvm on mw1119
  • 03:19 logmsgbot: ebernhardson@mira Synchronized wmf-config/CirrusSearch-production.php: Correct invalid cirrus shard configuration (duration: 02m 59s)
  • 02:55 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Jan 27 02:55:21 UTC 2016 (duration 7m 13s)
  • 02:48 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.11) (duration: 10m 25s)
  • 02:24 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.10) (duration: 09m 51s)
  • 01:59 logmsgbot: ori@mira Synchronized docroot and w: Icc4f6134b0: Add a speed experiment which inlines the top stylesheet (duration: 02m 28s)
  • 01:29 MaxSem: on terbium: ran mwscript namespaceDupes.php --wiki=wuuwiki --source-pseudo-namespace= --add-suffix=/renamed --fix
  • 01:26 MaxSem: Fail, trying something else...
  • 01:21 MaxSem: running mwscript namespaceDupes.php --wiki=wuuwiki --move-talk --fix
  • 00:52 logmsgbot: krenair@mira Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/266497/ (duration: 02m 26s)
  • 00:48 logmsgbot: krenair@mira Synchronized w/static/images/project-logos/ukwikinews.png: https://gerrit.wikimedia.org/r/#/c/266497/ (duration: 02m 29s)
  • 00:44 logmsgbot: krenair@mira Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/266161/ (duration: 02m 27s)
  • 00:15 logmsgbot: ebernhardson@mira Synchronized php-1.27.0-wmf.11/extensions/CirrusSearch/: Allow pointing morelike queries at a specific datacenter (duration: 03m 04s)
  • 00:10 logmsgbot: ebernhardson@mira Synchronized wmf-config/CirrusSearch-production.php: point morelike queries back at the eqiad cluster (duration: 05m 41s)
  • 00:02 chasemp: enable puppet and codify the 192 thread count for nfsd

2016-01-26

  • 22:25 logmsgbot: dduvall@mira rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.27.0-wmf.11, for real this time
  • 22:17 logmsgbot: dduvall@mira rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.27.0-wmf.11
  • 22:15 logmsgbot: dduvall@mira Synchronized php-1.27.0-wmf.11: syncing wmf.11 backports of session fixes (duration: 03m 55s)
  • 21:55 logmsgbot: ori@mira Synchronized docroot and w: I9b054d847a: New set of speed experiments (duration: 01m 29s)
  • 21:41 marxarelli: filed https://phabricator.wikimedia.org/T124828 for fatal in extensions/Echo
  • 21:22 marxarelli: Fatal error: Cannot redeclare class CallbackFilterIterator in /srv/mediawiki-staging/php-1.27.0-wmf.11/extensions/Echo/includes/iterator/CallbackFilterIterator.php on line 24
  • 21:21 marxarelli: lint error found when running sync-dir 'Errors parsing /srv/mediawiki-staging/php-1.27.0-wmf.11/extensions/Echo/includes/iterator/CallbackFilterIterator.php'
  • 21:11 marxarelli: sync-dir php linting failed
  • 21:02 marxarelli: resuming sync-dir and ignoring error as a known issue
  • 20:59 marxarelli: getting 'Lost parent, LightProcess exiting' when running sync-dir
  • 20:57 chasemp: drop labstore1001 nfs threads down to 192
  • 20:46 chasemp: stopping nfs on labstore1001
  • 20:46 marxarelli: modified wikiversions.php locally on mw1017 to promote all wikis to wmf.11 for initial testing
  • 20:18 marxarelli: locally modified wikiversions.php and wikiversions.json on mw1017 for testing
  • 20:14 marxarelli: running 'sync-common --verbose deployment.eqiad.wmnet' on mw1017 to sync wmf.11 for initial testing
  • 20:02 marxarelli: proceeding with train deploy. wmf.11 to mw1017, then group0
  • 19:46 akosiaris: issuing a varnish ban on all esams mobile frontend varnish for req.http.host .*wikimedia.org
  • 19:45 akosiaris: issuing a varnish ban on all esams mobile backend varnish for req.http.host .*wikimedia.org
  • 19:44 akosiaris: issuing a varnish ban on all ulsfo mobile frontend varnish for req.http.host .*wikimedia.org
  • 19:44 akosiaris: issuing a varnish ban on all ulsfo mobile backend varnish for req.http.host .*wikimedia.org
  • 19:43 akosiaris: issuing a varnish ban on all codfw mobile frontend varnish for req.http.host .*wikimedia.org
  • 19:36 akosiaris: issuing a varnish ban on all codfw mobile backend varnish for req.http.host .*wikimedia.org
  • 19:36 akosiaris: issuing a varnish ban on all eqiad mobile frontend varnish for req.http.host .*wikimedia.org
  • 19:36 akosiaris: issuing a varnish ban on all eqiad mobile backend varnish for req.http.host .*wikimedia.org
  • 19:36 akosiaris: all of the above referred to cache_text
  • 19:29 akosiaris: all of the above already done, back logging
  • 19:29 akosiaris: issuing a varnish ban on all esams frontend varnish for req.http.host .*wikimedia.org
  • 19:29 akosiaris: issuing a varnish ban on all esams backend varnish for req.http.host .*wikimedia.org
  • 19:29 akosiaris: issuing a varnish ban on all ulsfo backend varnish for req.http.host .*wikimedia.org
  • 19:29 akosiaris: issuing a varnish ban on all ulsfo frontend varnish for req.http.host .*wikimedia.org
  • 19:28 akosiaris: issuing a varnish ban on all ulsfo backend varnish for req.http.host .*wikimedia.org
  • 19:28 akosiaris: issuing a varnish ban on all codfw frontend varnish for req.http.host .*wikimedia.org
  • 19:28 akosiaris: issuing a varnish ban on all codfw backend varnish for req.http.host .*wikimedia.org
  • 19:28 akosiaris: issuing a varnish ban on all eqiad frontend varnish for req.http.host .*wikimedia.org
  • 19:14 akosiaris: issuing a varnish ban on all eqiad backend varnish for req.http.host .*wikimedia.org
  • 19:02 marxarelli: backports to wmf.11 ready on mira but delaying train due to wikimedia.org outage
  • 18:44 _joe_: running salt --batch-size=20 -C 'G@luster:appserver and G@site:eqiad' cmd.run 'puppet agent -t --tags mw-apache-config'
  • 18:27 robh: i broke icinga, but then i fixed it, icinga back to normal.
  • 18:21 robh: icinga is broken, it seems it was from a change before mine, but my forced reload broke it
  • 18:18 legoktm: running mwscript updateArticleCount.php --wiki=jawiki --update=1
  • 18:14 cmjohnson1: starting puppet on mw cluster
  • 18:14 robh: i broke icinga, fixing
  • 18:08 logmsgbot: jynus@mira Synchronized wmf-config/db-eqiad.php: Pool new parsercache pc1005 after cloning it from pc1002 (duration: 01m 28s)
  • 17:43 thcipriani: ltwiki collation updated 503623 rows processed
  • 17:35 mutante: mw1258 - restart hhvm
  • 17:20 cmjohnson: disabling puppet on mw cluster
  • 17:02 thcipriani: running updateCollation on ltwiki
  • 17:01 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: Set category collation to uca-lt on lt.wikipedia gerrit:266427 (duration: 01m 33s)
  • 16:55 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: Namespace configuration on ur.wikipedia gerrit:265888 (duration: 07m 10s)
  • 16:36 logmsgbot: thcipriani@mira Synchronized w/static/images/project-logos/etwikiquote.png: SWAT: Update et.wikiquote logo gerrit:265623 (duration: 01m 27s)
  • 16:31 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable SandboxLink on nl.wikiquote gerrit:265666 (duration: 01m 26s)
  • 16:26 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: Namespaces configuration on sk.wikipedia gerrit:265896 (duration: 01m 27s)
  • 16:19 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove Tranwiki namespace on wuu.wikipedia gerrit:265892 and Add Portal namespace on wuu.wikipedia gerrit:265893 (duration: 01m 27s)
  • 16:12 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: Namespace configuration for wuu.wikipedia gerrit:265891 (duration: 01m 29s)
  • 14:57 ema: Finished migration of mobile traffic to text cluster in esams https://phabricator.wikimedia.org/T109286
  • 14:48 chasemp: RPS on eth0 on labstores
  • 14:39 bblack: upgrading packages (incl kernel) on all ulsfo caches (cp4xxx)
  • 14:21 akosiaris: migrating alsafi,mx2001 back to 2004 for testing
  • 14:14 akosiaris: migrate alsafi,mx2001 back from ganeti2004 to fix a network misconfiguration
  • 13:32 moritzm: rebooted nescio/maerlant for kernel update
  • 13:14 logmsgbot: jynus@mira Synchronized wmf-config/db-eqiad.php: Depool pc1002 for maintenance (clone to pc1005) (duration: 01m 39s)
  • 12:39 akosiaris: rolling reboot of ganeti200{1,2,3,4,5,6}.codfw.wmnet for kernel upgrade
  • 12:10 moritzm: rebooting mx2001/mx1001 (with a delay in between) for kernel update
  • 11:50 moritzm: rebooting etherpad1001 for kernel update
  • 11:46 moritzm: rebooting bromine for kernel update
  • 10:50 ema: Starting migration of mobile traffic to text cluster in esams https://phabricator.wikimedia.org/T109286
  • 09:30 hashar: restarting Jenkins to upgrade the gearman plugin with https://review.openstack.org/#/c/271543/
  • 09:28 _joe_: finishing reboots of appservers in eqiad
  • 04:27 legoktm: restarted resetGlobalUserTokens.php after it lost mysql connection again
  • 02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Jan 26 02:30:58 UTC 2016 (duration 7m 0s)
  • 02:24 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.10) (duration: 09m 36s)
  • 01:45 logmsgbot: krenair@mira Synchronized wmf-config/wikitech.php: https://gerrit.wikimedia.org/r/#/c/266453/ (duration: 01m 27s)
  • 00:45 mobrovac: mobileapps deploying c2318b6
  • 00:40 logmsgbot: ebernhardson@mira Synchronized wmf-config/CommonSettings.php: (no message) (duration: 01m 25s)
  • 00:37 logmsgbot: ebernhardson@mira Synchronized wmf-config/InitialiseSettings.php: SWAT bd808 (duration: 01m 34s)
  • 00:32 logmsgbot: ebernhardson@mira Synchronized portals/: SWAT jgirault (duration: 01m 28s)
  • 00:29 logmsgbot: ebernhardson@mira Synchronized wmf-config/InitialiseSettings.php: SWAT ebernhardson (duration: 01m 26s)
  • 00:27 logmsgbot: ebernhardson@mira Synchronized wmf-config/CirrusSearch-common.php: SWAT ebernhardson (duration: 01m 26s)
  • 00:25 logmsgbot: ebernhardson@mira Synchronized wmf-config/CommonSettings.php: SWAT ebernhardson (duration: 01m 27s)
  • 00:15 logmsgbot: ebernhardson@mira Synchronized wmf-config/CommonSettings.php: SWAT AaronSchulz (duration: 01m 26s)
  • 00:13 logmsgbot: ebernhardson@mira Synchronized wmf-config/filebackend-production.php: SWAT AaronSchulz (duration: 01m 26s)
  • 00:10 logmsgbot: ebernhardson@mira Synchronized wmf-config/CommonSettings.php: SWAT James_F (duration: 01m 26s)
  • 00:08 logmsgbot: ebernhardson@mira Synchronized wmf-config/InitialiseSettings.php: SWAT James_F (duration: 01m 35s)

2016-01-25

  • 23:14 logmsgbot: legoktm@mira Synchronized php-1.27.0-wmf.10/includes/parser/: live hacks, now committed (duration: 01m 27s)
  • 23:07 logmsgbot: legoktm@mira Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/266410/ (duration: 01m 35s)
  • 22:52 logmsgbot: ori@mira Synchronized php-1.27.0-wmf.10/includes/parser/ParserOutput.php: Fix-up for ParserOutput.php@263 debug logging (duration: 01m 27s)
  • 22:30 logmsgbot: legoktm@mira Synchronized php-1.27.0-wmf.10/includes/parser/: https://gerrit.wikimedia.org/r/#/c/266401/ + https://gerrit.wikimedia.org/r/#/c/266406/ + live hacks (duration: 01m 28s)
  • 22:28 logmsgbot: legoktm@mira Synchronized php-1.27.0-wmf.10/includes/content/WikitextContent.php: https://gerrit.wikimedia.org/r/#/c/266401/ (duration: 01m 29s)
  • 21:53 logmsgbot: hoo@mira Synchronized wmf-config/Wikibase-production.php: Disable (not yet deployed) commons category sidebar link overwrite in production (duration: 01m 28s)
  • 21:47 mutante: nitrogen - shutdown -h now ....
  • 21:45 mutante: alsafi - was reported down in icinga , is ganeti VM - fixed by just logging in as if it went to hibernate
  • 21:37 mdholloway: mobileapps deployed 9252a22
  • 21:30 mutante: nitrogen - stop puppet, stop salt, remove from stored configs / icinga
  • 20:19 logmsgbot: hoo@mira Synchronized wmf-config/Wikibase-labs.php: (no message) (duration: 01m 28s)
  • 20:14 chasemp: bump labstore nfs threads to 288 from 244
  • 19:32 paravoid: eqiad: removing static routes for 6to4/Teredo to nitrogen (decommissioning our own relays)
  • 19:10 bd808: Live hacking on mw1017 to debug 1.27.0-wmf.11 issues. All wikis there currently set to use 1.27.0-wmf.11.
  • 19:05 chasemp: labstore1001 temp change to CFQ scheduler on 01/22/2015
  • 19:04 chasemp: the nfsd thread change is on labstore1001
  • 19:04 chasemp: nfsd has 224 threads atm and was bumped up over the weekend
  • 18:58 ori: removed unused wikiversions.cdb on mira and tin
  • 18:28 jynus: retroactively logging the depool of mw1217, mw1178 and mw1257 3 hours ago (Jan 25 15:45:26)
  • 16:49 ema: Finished migration of mobile traffic to text cluster in ulsfo https://phabricator.wikimedia.org/T109286
  • 16:38 logmsgbot: jynus@mira Synchronized wmf-config/db-eqiad.php: Preparing ips for new parsercache deployments (third try) (duration: 01m 35s)
  • 16:26 logmsgbot: jynus@mira Synchronized wmf-config/db-eqiad.php: Preparing ips for new parsercache deployments (second try after running puppet) (duration: 03m 23s)
  • 16:25 _joe_: restarting salt-minion on all deployment targets
  • 16:24 _joe_: running salt deploy.fixurl on all deployment targets
  • 16:09 logmsgbot: jynus@mira Synchronized wmf-config/db-eqiad.php: Preparing ips for new parsercache deployments (duration: 03m 32s)
  • 15:51 ejegg: updated DjangoBannerStats from a64fe0e373a978d3df0b7f1dd74ac4cc5c78d34e to 71df14d4d8b11f3ca0ef1eeb6c6e2db9be79103a
  • 15:35 ema: Starting migration of mobile traffic to text cluster in ulsfo https://phabricator.wikimedia.org/T109286
  • 15:14 chasemp: restart of pdns and pdns-recursor on labservices1001
  • 14:56 logmsgbot: jynus@mira Synchronized wmf-config/db-eqiad.php: deploy new parsercache hardware (pc1004) substituting pc1001 (duration: 03m 25s)
  • 13:16 elukey: ran kafka preferred-replica-election on kafka1022 to balance the leaders
  • 13:07 elukey: restarting kafka on kafka1022
  • 12:57 elukey: restarting kafka on kafka1013
  • 12:38 elukey: restarting kafka on kafka1014
  • 12:20 jynus: compressed and truncated iridium's phab daemons.log - it was taking 20% of disk space
  • 12:04 ema: restarting kafka on kafka1018
  • 11:26 jynus: stopping mysql at pc1001 and cloning to pc1004
  • 10:55 logmsgbot: jynus@mira Synchronized wmf-config/db-eqiad.php: Depool pc1001 for maintenance (clone to pc1004) (duration: 01m 41s)
  • 10:11 _joe_: switching the active deployment host to mira
  • 09:56 ema: limiting GCLogFileSize and restarting kafka on kafka1012
  • 09:31 _joe_: rolling reboot of the eqiad appserver cluster
  • 09:27 moritzm: installed fuse security update on labnodepool1001 (the other fuse installations are on Ubuntu, which doesn't ship the udev rule, but uses mountall instead)
  • 07:47 paravoid: stat1002: umount -f /mnt/hdfs
  • 07:34 _joe_: rebooting alsafi, unresponsive to ssh
  • 07:24 _joe_: restarting hhvm on mw1148, stuck in HPHP::Treadmill::startRequest (__lll_lock_wait)
  • 07:23 _joe_: restarting hhvm on mw1143, stuck into HPHP::SynchronizableMulti::waitImpl (__pthread_cond_wait)
  • 03:10 logmsgbot: tstarling@tin Synchronized php-1.27.0-wmf.10/includes/parser/ParserCache.php: (no message) (duration: 00m 25s)
  • 03:03 logmsgbot: tstarling@tin Synchronized php-1.27.0-wmf.10/includes/parser/ParserCache.php: (no message) (duration: 00m 25s)
  • 03:02 logmsgbot: tstarling@tin Synchronized php-1.27.0-wmf.10/includes/parser/ParserOutput.php: (no message) (duration: 00m 27s)
  • 02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Jan 25 02:30:13 UTC 2016 (duration 6m 52s)
  • 02:23 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.10) (duration: 09m 09s)

2016-01-24

  • 02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Jan 24 02:31:21 UTC 2016 (duration 6m 58s)
  • 02:24 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.10) (duration: 09m 11s)

2016-01-23

  • 19:03 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-production.php: config change to repoint morelike search from eqiad to codfw (duration: 00m 26s)
  • 19:02 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.10/extensions/CirrusSearch/: Support code for repointing morelike queries from eqiad to codfw (duration: 00m 30s)
  • 19:00 ebernhardson: repoint most expensive search queries (morelike) at codfw cluster to reduce load. 1/2 of eqiad cluster maxed on cpu
  • 16:47 Krinkle: mwscript deleteEqualMessages.php --wiki wowiki
  • 13:25 jynus: upgrading and restarting db1046
  • 13:13 jynus: db1046 maintenance finished- restarting mysql to apply latest configuration
  • 02:32 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Jan 23 02:32:15 UTC 2016 (duration 7m 3s)
  • 02:25 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.10) (duration: 09m 09s)
  • 01:33 logmsgbot: bd808@tin rebuilt wikiversions.php and synchronized wikiversions files: Back to 1.27.0-wmf10 again after fixking l10n cache problems
  • 01:28 logmsgbot: bd808@tin rebuilt wikiversions.php and synchronized wikiversions files: Temporarily back to 1.27.0-wmf11; need to rebuild l10n cache
  • 01:16 logmsgbot: bd808@tin rebuilt wikiversions.php and synchronized wikiversions files: Revert all wikis to 1.27.0-wmf.10
  • 00:08 logmsgbot: bd808@tin Synchronized php-1.27.0-wmf.11/extensions/CentralAuth/includes/session/CentralAuthSessionProvider.php: https://gerrit.wikimedia.org/r/#/c/265872/ (duration: 00m 25s)
  • 00:07 logmsgbot: bd808@tin Synchronized php-1.27.0-wmf.11/includes/session/CookieSessionProvider.php: https://gerrit.wikimedia.org/r/#/c/265871/ (duration: 00m 25s)

2016-01-22

  • 23:43 logmsgbot: legoktm@tin Synchronized php-1.27.0-wmf.11/extensions/CentralAuth/includes/session/CentralAuthSessionProvider.php: https://gerrit.wikimedia.org/r/#/c/265870/ (duration: 00m 26s)
  • 23:42 logmsgbot: legoktm@tin Synchronized php-1.27.0-wmf.11/includes/session/CookieSessionProvider.php: https://gerrit.wikimedia.org/r/#/c/265869/ (duration: 00m 26s)
  • 23:22 mobrovac: restbase cassandra truncating local_group_wiktionary_T_term_definition.data
  • 22:33 mdholloway: mobileapps deployed 2900faa
  • 22:23 logmsgbot: twentyafterfour@tin Finished scap: deploy https://gerrit.wikimedia.org/r/#/c/263415/ and clean up old branches (duration: 07m 02s)
  • 22:16 logmsgbot: twentyafterfour@tin Started scap: deploy https://gerrit.wikimedia.org/r/#/c/263415/ and clean up old branches
  • 22:06 bblack: upgrading vhtcpd on all caches
  • 22:05 eileen: upgrade Civicrm from b9ebf3d31aeab8120143cfbf6bc2df0f617341cf to c009af16944a6478bd0292422f5bb0151f7a22c1
  • 21:49 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.11/includes/: Fix T124468, for real this time (duration: 00m 36s)
  • 21:48 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.11/includes/: Fix T124468 (duration: 00m 38s)
  • 21:17 legoktm: running migrateAccount.php --attachbroken over list of all unattached users (T74791)
  • 20:04 mutante: ruthenium - rebooting for reinstall
  • 19:42 logmsgbot: aaron@tin Synchronized wmf-config/CommonSettings.php: Revert "Bump $wgJobBackoffThrottling to lower the htmlcacheupdate backlog" (duration: 00m 32s)
  • 18:51 jynus: "repairing" enwiki.oldtable on dbstore1001
  • 18:40 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Aborting pc1001 maintenance (duration: 00m 31s)
  • 18:15 legoktm: running CentralAuth's resetGlobalUserTokens.php to force session resets for all users T124440
  • 18:02 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.11/includes/user/User.php: Fix T124414 (duration: 00m 33s)
  • 17:53 legoktm: manually attaching User:Mower Genetics and User:Themeetingplace because they made edits somehow (T74791)
  • 17:46 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Stop logging the CirrusSearchRequests channel (duration: 00m 32s)
  • 17:44 legoktm: running migrateAccount.php --attachbroken over lists on T74791
  • 17:39 _joe_: removed an archived CirrusSearchRequests.log on fluorine, now we have enough room for the weekend
  • 17:29 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.11/extensions/CentralAuth/includes: Fix T124406 (duration: 00m 35s)
  • 17:05 mobrovac: mobileapps deploying bba45456
  • 17:00 logmsgbot: reedy@tin Synchronized docroot and w: Extra noc symlinks (duration: 00m 32s)
  • 16:58 logmsgbot: jynus@tin Synchronized wmf-config/InitialiseSettings.php: monolog: reduce on-disk logging of DBPerformance to warning (duration: 00m 32s)
  • 16:47 jynus: truncating 100GB DBPerformance.log on fluorine, compressed backup available
  • 16:46 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.11/extensions/CentralAuth/includes/session/CentralAuthSessionProvider.php: Fix T124409, part 2 (duration: 00m 32s)
  • 16:46 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.11/includes/session/SessionBackend.php: Fix T124409, part 1 (duration: 00m 33s)
  • 16:41 cmjohnson1: Troubleshooting mw1228
  • 16:36 _joe_: all api appservers in eqiad have been restarted
  • 16:21 ori: restarted statsv on hafnium
  • 15:53 ema: Finished migrating mobile traffic to text cluster in codfw (Mexico + green US states on this map https://phabricator.wikimedia.org/T114659)
  • 15:39 gwicke: aqs: increased compression block size on per-article table from 128k to 256k; expectation is to further increase compression ratio & reduce seeks on rotating disks
  • 15:22 Reedy: created translate tables on ruwikimedia T121766
  • 14:18 paravoid: cr1-eqord: turning up BGP with Zayo
  • 13:08 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.10/extensions/MobileFrontend: I08cdf37a1: Use TitleSquidURLs hook to purge mobile URLs directly (Bug: T124165) (duration: 00m 33s)
  • 13:05 logmsgbot: ori@tin Synchronized wmf-config/InitialiseSettings.php: If443f3c80: monolog: explicitly declare logstash as debug for sessions (duration: 00m 34s)
  • 12:31 ema: Starting migration of mobile traffic to text cluster https://phabricator.wikimedia.org/T109286
  • 11:35 logmsgbot: oblivian@tin Synchronized wmf-config/InitialiseSettings.php: Re-synching (duration: 00m 31s)
  • 11:25 logmsgbot: oblivian@tin Synchronized wmf-config/InitialiseSettings.php: Stop writing session logs to fluorine (duration: 01m 25s)
  • 11:17 bblack: codfw LVS under etcd/conftool control now, like ulsfo
  • 10:57 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool pc1001 for maintenance (duration: 02m 48s)
  • 10:45 _joe_: rolling restarting the API cluster in eqiad
  • 10:34 _joe_: rolling restart of all api appservers in eqiad
  • 10:07 _joe_: dropping api logs from 2015 on fluorine
  • 09:10 _joe_: rolling restart of imagescalers in eqiad
  • 08:48 _joe_: powercycling ms-be1002, blank console, down
  • 08:46 _joe_: rebooting mw1001 with a new kernel
  • 08:07 _joe_: upgrading kernel on all mw hosts in eqiad
  • 05:07 logmsgbot: tstarling@tin Synchronized php-1.27.0-wmf.11/includes/parser/ParserCache.php: (no message) (duration: 01m 28s)
  • 02:42 logmsgbot: tstarling@tin Synchronized php-1.27.0-wmf.11/includes/parser/ParserCache.php: (no message) (duration: 01m 28s)
  • 02:40 logmsgbot: tstarling@tin Synchronized php-1.27.0-wmf.11/includes/OutputPage.php: (no message) (duration: 01m 32s)
  • 02:30 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.11) (duration: 09m 31s)
  • 01:44 logmsgbot: catrope@tin Finished scap: Deploying OATHAuth and WikimediaMessages i18n changes (duration: 30m 52s)
  • 01:37 gwicke: restbase cassandra: increased compression chunk size from 256 to 512k on wikimedia and wikipedia html and data-parsoid
  • 01:13 logmsgbot: catrope@tin Started scap: Deploying OATHAuth and WikimediaMessages i18n changes
  • 01:08 eileen: Updating CiviCRM from cb5e20c29d7376920c45eb5c343e6ee464217833 to to b9ebf3d31aeab8120143cfbf6bc2df0f617341cf
  • 00:19 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Add ability for OfficeWiki sysops to add and remove flood group rights from themselves. (duration: 01m 27s)
  • 00:14 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: enable EventBus extension on mediawikiwiki (duration: 01m 27s)
  • 00:10 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: enable sandboxlink on ladwiki and dont sent messages to autocreated accounts on metawiki (duration: 01m 27s)
  • 00:08 logmsgbot: ebernhardson@tin Synchronized wmf-config/throttle.php: Santiago Editatón throttle rule (duration: 01m 27s)
  • 00:02 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-production.php: configure cirrus completion suggester recycling (duration: 01m 29s)
  • 00:00 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: configure cirrus completion suggester recycling (duration: 01m 28s)

2016-01-21

  • 22:46 legoktm: started running migratePass0.php (CentralAuth) on group1 wikis
  • 22:24 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.27.0-wmf.11
  • 22:23 legoktm: started running migratePass0.php (CentralAuth) on group0 wikis
  • 21:35 ejegg: re-enabled low-level fundraising banner campaigns
  • 21:30 ejegg: reverted donatewiki maintenance message
  • 21:19 ejegg: updated paymentswiki from a7785baa7b40b442ecf0b60d47572502d0759780 to 1817327b4b0919ebe26bbd8b9d84fac1bd7ddb03
  • 21:13 andrewbogott: all reachable labs instances are now running security-patched kernels.
  • 21:12 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: cswiktionary to 1.27.0-wmf.11
  • 21:12 ejegg: disabled low-level fundraising banner campaigns
  • 21:12 andrewbogott: all labvirt10xx hosts are now running the latest utopic kernel
  • 21:09 ejegg: replaced form on donatewiki with maintenance notice
  • 21:08 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.11/includes/session/SessionManager.php: SessionManager: Notify AuthPlugin when auto-creating accounts gerrit:265578 (duration: 01m 26s)
  • 21:01 andrewbogott: rebooting labvirt1010
  • 20:51 andrewbogott: rebooting labvirt1009
  • 20:33 andrewbogott: rebooting labvirt1007
  • 20:33 logmsgbot: dduvall@tin Synchronized php-1.27.0-wmf.11/includes/user/BotPassword.php: deploy fix for T124335 (duration: 01m 29s)
  • 20:27 mobrovac: restbase deploy end of 79a4d27
  • 20:20 mobrovac: restbase deploy start of 79a4d27
  • 20:16 andrewbogott: rebooting labvirt1006
  • 19:58 mobrovac: mobileapps deploying 68c09e
  • 19:54 logmsgbot: dduvall@tin rebuilt wikiversions.php and synchronized wikiversions files: rollback cswiktionary to 1.27.0-wmf.10
  • 19:54 andrewbogott: rebooting labvirt1005
  • 19:32 andrewbogott: rebooting labvirt1004
  • 19:31 logmsgbot: dduvall@tin Synchronized php-1.27.0-wmf.11/extensions/CentralAuth/includes/session/CentralAuthTokenSessionProvider.php: deploy https://gerrit.wikimedia.org/r/#/c/265545/ for 1.27.0-wmf.11 (duration: 01m 28s)
  • 19:24 mobrovac: restbase rolling-restart after firejail inclusion
  • 19:22 mobrovac: restbase re-enabling puppet in prod
  • 19:14 andrewbogott: rebooting labvirt1003
  • 18:57 logmsgbot: dduvall@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.11
  • 18:53 marxarelli: starting train promotion of group1 to 1.27.0-wmf.11
  • 18:52 marxarelli: sync to mw2020 failed due to failed host key verification, mw2087/mw2039/mw2098 due to connection failed
  • 18:47 marxarelli: 4 apache sync failures during sync-file, appear to be know issues
  • 18:46 andrewbogott: rebooting labvirt1002
  • 18:43 logmsgbot: dduvall@tin Synchronized php-1.27.0-wmf.11/includes/session/PHPSessionHandler.php: deploy follow-up warning fix for T124126 (duration: 01m 28s)
  • 18:43 mobrovac: restbase disabling puppet in prod for testing firejail in staging
  • 18:41 akosiaris: enable puppet and salt-minion on sca100{1,2}.eqiad.wmnet
  • 18:39 akosiaris: depool sca1001, sca1002 for citoid
  • 18:34 akosiaris: pool scb1001, scb1002 for citoid
  • 18:07 andrewbogott: rebooting labvirt1001
  • 17:57 akosiaris: depool sca1001,sca1002 for graphoid pybal config
  • 17:49 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Really enable ContentTranslationCorpora gerrit:265514 (duration: 01m 29s)
  • 17:48 akosiaris: add scb1001, scb1002 in pybal graphoid config
  • 17:30 akosiaris: disabled puppet and salt-minion on sca1001, sca1002 for graphoid upgrade
  • 17:24 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Enable ContentTranslationCorpora Part II gerrit:265459 (duration: 01m 28s)
  • 17:22 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ContentTranslationCorpora Part I gerrit:265459 (duration: 01m 28s)
  • 17:12 _joe_: restarting pybal on the main balancers in ulsfo to consume from etcd
  • 17:02 andrewbogott: rebooting labvirt1008
  • 16:42 jynus: batch-converting m4-master (log) tables from innodb to tokudb
  • 16:42 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.11/extensions/MobileFrontend/MobileFrontend.php: SWAT: Use TitleSquidURLs hook to purge mobile URLs directly Part II gerrit:265486 (duration: 01m 28s)
  • 16:40 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.11/extensions/MobileFrontend/includes/MobileFrontend.hooks.php: SWAT: Use TitleSquidURLs hook to purge mobile URLs directly Part I gerrit:265486 (duration: 01m 28s)
  • 16:35 ottomata: stopped eventlogging mysql consumers for long downtime: https://phabricator.wikimedia.org/T120187
  • 16:28 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.10/extensions/MobileApp/config/config.json: SWAT: Roll out RESTBase usage to Android Beta app: 100% gerrit:265117 (duration: 01m 27s)
  • 16:22 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.11/extensions/MobileApp/config/config.json: SWAT: Roll out RESTBase usage to Android Beta app: 100% gerrit:265118 (duration: 01m 28s)
  • 16:20 ottomata: started eventlogging mysql consumers
  • 16:19 paravoid: deactivating GTT BGP peering on cr2-eqiad
  • 16:05 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: wgRCWatchCategoryMembership true on dewiki gerrit:264732 (duration: 01m 28s)
  • 15:59 ottomata: stopping eventlogging mysql consumers for https://phabricator.wikimedia.org/T123546
  • 14:37 paravoid: upgraded cr2-codfw to JunOS 13.3R8.7
  • 13:20 _joe_: rolling reboot of imagescalers, jobrunners in codfw
  • 12:10 paravoid: upgrading cr1-codfw to JunOS 13.3R8.7
  • 11:27 _joe_: restarting pybal on lvs4003, switching to etcd
  • 11:25 _joe_: restarting pybal on lvs4004, switching to etcd
  • 11:09 jynus: adding new version of mariadb to carbon for jessie (10.0.23-1)
  • 10:19 _joe_: mw2098 doesn't reboot, console unreachable
  • 10:10 jynus: mw2098.codfw.wmnet failed to sync
  • 10:10 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Restore s5 DB configuration (duration: 01m 57s)
  • 09:53 _joe_: rolling reboot of the codfw appserver layer
  • 09:27 _joe_: powercycled mw1162, memory exhaustion
  • 08:01 _joe_: upgrading all codfw appserver layer's kernel to linux-image-3.13.0-76-generic
  • 02:56 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Jan 21 02:56:44 UTC 2016 (duration 7m 9s)
  • 02:49 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.11) (duration: 09m 39s)
  • 02:27 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.10) (duration: 09m 33s)
  • 02:24 mobrovac: citoid deploying 3a1b6c8648
  • 02:16 ori: Restarting jobrunner service on job runners to ensure I180856917 gets picked up
  • 01:47 mutante: nitrogen - install package upgrades
  • 01:15 bd808: Restarted logstash on logstash1003
  • 01:14 bd808: Restarted logstash on logstash1002
  • 01:04 logmsgbot: maxsem@tin Synchronized wmf-config/: https://gerrit.wikimedia.org/r/#/c/265395/ (duration: 00m 32s)
  • 00:56 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.11/extensions/GeoData/: https://gerrit.wikimedia.org/r/#/c/265409/ (duration: 00m 33s)
  • 00:50 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/265142/ (duration: 00m 32s)

2016-01-20

  • 23:56 logmsgbot: reedy@tin Synchronized php-1.27.0-wmf.10/extensions/SemanticForms/: fix wikitech again (duration: 00m 34s)
  • 23:06 bd808: Restarted logstash on logstash1001
  • 23:04 bd808: Logstash1001 went nuts and decided that instead of 2016 it would go back to the start of 2015 after 2015-12-31T23:59
  • 22:54 bd808: no HHVM log events in logstash since 2015-12-31T23:59:44.000Z
  • 22:48 bd808: HHVM log messages not being recorded in Logstash; bd808 to investigate
  • 22:38 logmsgbot: tgr@tin Synchronized php-1.27.0-wmf.11/includes/: T124143,T124126 (duration: 00m 36s)
  • 22:06 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.11/extensions/OAuth: Deploy fix for T124224 (duration: 00m 32s)
  • 22:04 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.2/extensions/OAuth: Deploy fix for T124224 (duration: 00m 34s)
  • 21:51 logmsgbot: reedy@tin Synchronized php-1.27.0-wmf.11/extensions/SemanticResultFormats: Fix wikitech log noise (duration: 00m 31s)
  • 21:50 logmsgbot: reedy@tin Synchronized php-1.27.0-wmf.11/extensions/SemanticMediaWiki: Fix wikitech log noise (duration: 00m 34s)
  • 21:48 subbu: finished deploying parsoid sha f1ddfb88
  • 21:41 subbu: synced new parsoid code; restarted parsoid on wtp1001 as a canary
  • 21:35 subbu: starting parsoid deploy
  • 21:32 thcipriani: reverted group1 wikis to 1.27.0-wmf.10 due to session errors.
  • 21:30 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.10
  • 21:14 andrewbogott: rebooting labvirt1011
  • 21:08 logmsgbot: reedy@tin Synchronized php-1.27.0-wmf.11/extensions/SemanticForms/: Fix fatal on wikitech (duration: 00m 36s)
  • 20:37 akosiaris: s#/dev/md1#/dev/mapper/tank-data# on labvirt1010, reverted by puppet with Notice: /Stage[main]/Role::Labs::Openstack::Nova::Compute/Mount[/var/lib/nova/instances]/device: device changed '/dev/mapper/tank-data' to '/dev/md1'
  • 20:37 akosiaris: s#/dev/md1#/dev/mapper/tank-data#
  • 19:32 logmsgbot: dduvall@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.11
  • 19:14 marxarelli: including labswiki and labtestwiki in group1 promotion after all
  • 19:09 marxarelli: starting promotion of group1, but holding back labswiki and labtestwiki until Jan 21 'all' promotion
  • 18:54 paravoid: manually triggering an ubuntu mirror update ("sudo -u mirror /usr/local/sbin/update-ubuntu-mirror" on carbon)
  • 18:41 jynus: schema change on wikidatawiki (wb_terms) finished- slaves already catching up
  • 18:34 mutante: restart hhvm on mw1206
  • 18:32 godog: bounce stuck hhvm on mw1205
  • 18:06 paravoid: turning up BGP with Zayo in codfw
  • 17:48 jynus: restarting replication on db1026 after schema change
  • 17:09 gwicke: restbase cassandra: set DTCS max_window_size_seconds to 70736000, large enough to accommodate a two-year window
  • 16:56 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Set default graph vega version back to 1 gerrit:265289 (duration: 00m 32s)
  • 16:46 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add davidabian.com to wgCopyUploadsDomains gerrit:265286 (duration: 00m 32s)
  • 16:42 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Change default graph version param. Part II gerrit:265282 (duration: 00m 32s)
  • 16:42 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Change default graph version param. Part I gerrit:265282 (duration: 00m 36s)
  • 16:33 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add davidabian.com to wgCopyUploadsDomains gerrit:259003 (duration: 00m 32s)
  • 16:21 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add *.bodleian.ox.ac.uk to wgCopyUploadsDomains gerrit:265165 (duration: 00m 33s)
  • 16:19 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add *.archives.gov to wgCopyUploadsDomains gerrit:265163 (duration: 00m 32s)
  • 16:13 godog: bounce hhvm on mw1191 and syntaxlight runaway processes
  • 16:05 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable active gadget user stats on enwiki since it takes too long gerrit:265185 (duration: 00m 32s)
  • 14:52 logmsgbot: reedy@tin Synchronized php-1.27.0-wmf.11/vendor/: Fix ?PHP properly from commit (duration: 00m 36s)
  • 14:50 godog: powercycle mw1123, hhvm oom
  • 14:47 ema: Finished reverting migration of mobile traffic to text cluster in codfw https://phabricator.wikimedia.org/T109286
  • 14:24 logmsgbot: hoo@tin Synchronized wmf-config/db-eqiad.php: Set db1045 load to 0 (duration: 00m 32s)
  • 14:23 logmsgbot: reedy@tin Synchronized php-1.27.0-wmf.11/: consistency (duration: 02m 38s)
  • 14:15 logmsgbot: hoo@tin Synchronized wmf-config/db-eqiad.php: Re-Pool lagged db1045 (duration: 00m 35s)
  • 14:14 _joe_: syncronizing /srv/deployment manually between the two deployment servers for the first time
  • 14:11 logmsgbot: hoo@tin Synchronized wmf-config/db-eqiad.php: Has not been synced before (duration: 00m 32s)
  • 14:07 logmsgbot: reedy@tin Synchronized php-1.27.0-wmf.10/: consistency (duration: 02m 38s)
  • 13:58 logmsgbot: reedy@tin Synchronized php-1.27.0-wmf.11/extensions/Validator/: noop for wikitech deploy (duration: 00m 32s)
  • 13:58 logmsgbot: reedy@tin Synchronized php-1.27.0-wmf.11/extensions/SemanticMediaWiki/: noop for wikitech deploy (duration: 00m 34s)
  • 13:57 logmsgbot: reedy@tin Synchronized php-1.27.0-wmf.11/extensions/SemanticResultFormats/: noop for wikitech deploy (duration: 00m 33s)
  • 13:41 ema: Revert migration of mobile traffic to text cluster in codfw https://phabricator.wikimedia.org/T109286
  • 12:55 akosiaris: restart hhvm on mw1130
  • 12:43 jynus: performing alter table on db1026 (ETA: 5 hours)
  • 12:20 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Setting s5 master as recentchanges role (duration: 00m 32s)
  • 12:04 jynus: trying schema change on wikidata (wb_terms)
  • 09:36 akosiaris: gnt-instance modify -H disk_aio=native cygnus.codfw.wmnet
  • 09:18 akosiaris: offline fr_archive volume on nas1001-a
  • 09:15 akosiaris: unexport /vol/fr_archive on nas1001-a
  • 07:56 _joe_: powercycling mw1162, unable to login from console, memory exhaustion
  • 07:24 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.10/extensions/CirrusSearch/includes/DataSender.php: stop checking for frozen indices while codfw elasticsearch recovers (duration: 01m 42s)
  • 06:24 ebernhardson: codfw elasticsearch cluster stopped responding during load test, idling test to see if it recovers
  • 03:44 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Jan 20 03:44:48 UTC 2016 (duration 7m 29s)
  • 03:37 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.11) (duration: 16m 21s)
  • 03:02 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.10) (duration: 10m 06s)
  • 02:35 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 11m 20s)
  • 01:27 logmsgbot: aaron@tin Synchronized wmf-config: Configure $wgCdnReboundPurgeDelay (duration: 00m 32s)
  • 01:01 mobrovac: restbase deploy end of d621b76
  • 00:57 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/264917/ (duration: 00m 32s)
  • 00:56 legoktm: delete from localuser where lu_name ="Αντώνης Μανιός" and lu_wiki ="mediawikiwiki" limit 1 on centralauth db for T119736
  • 00:53 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/264920/ (duration: 00m 33s)
  • 00:49 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.10/extensions/MobileFrontend/includes/api/ApiMobileView.php: https://gerrit.wikimedia.org/r/#/c/264973/ (duration: 00m 32s)
  • 00:49 mobrovac: restbase deploy start of d621b76
  • 00:38 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/264961/ (duration: 00m 31s)
  • 00:37 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/264961/ (duration: 00m 33s)
  • 00:22 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/264260/ (duration: 00m 32s)
  • 00:21 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/264260/ (duration: 00m 32s)
  • 00:17 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.10/extensions/CirrusSearch: https://gerrit.wikimedia.org/r/#/c/265146/ (duration: 00m 33s)
  • 00:10 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.10/extensions/CirrusSearch/includes/ElasticsearchIntermediary.php: https://gerrit.wikimedia.org/r/#/c/264989/ (duration: 00m 32s)

2016-01-19

  • 23:33 logmsgbot: aaron@tin Synchronized wmf-config/CommonSettings.php: Bump $wgJobBackoffThrottling to lower the htmlcacheupdate backlog (duration: 00m 32s)
  • 23:22 logmsgbot: krenair@tin Synchronized wmf-config/wikitech.php: https://gerrit.wikimedia.org/r/265145 (duration: 02m 24s)
  • 23:19 logmsgbot: dduvall@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.27.0-wmf.11
  • 23:13 logmsgbot: dduvall@tin Finished scap: testwiki to php-1.27.0-wmf.11 and rebuild l10n cache (duration: 72m 03s)
  • 22:01 logmsgbot: dduvall@tin Started scap: testwiki to php-1.27.0-wmf.11 and rebuild l10n cache
  • 21:35 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/265135 (duration: 00m 32s)
  • 21:33 logmsgbot: krenair@tin Synchronized dblists/nonglobal.dblist: https://gerrit.wikimedia.org/r/265135 (duration: 03m 21s)
  • 21:33 ema: Finished migrating mobile traffic to text cluster in codfw (Mexico + green US states on this map https://phabricator.wikimedia.org/T114659)
  • 21:15 logmsgbot: dduvall@tin scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="testwiki" --list-file="/srv/mediawiki-staging/wmf-config/extension-list" --output="/tmp/tmp.qyk48j8kem" ' returned non-zero exit status 1 (duration: 16m 11s)
  • 20:59 Krenair: sync-common on labtestweb2001
  • 20:58 logmsgbot: dduvall@tin Started scap: testwiki to php-1.27.0-wmf.11 and rebuild l10n cache
  • 20:48 mutante: tin: deleted unused things from /srv/deployment (T120157)
  • 20:46 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Disable global AbuseFilters on non-global wikis (duration: 02m 04s)
  • 20:25 logmsgbot: dduvall@tin scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="labtestwiki" --list-file="/srv/mediawiki-staging/wmf-config/extension-list" --output="/tmp/tmp.jRNpeW67FO" ' returned non-zero exit status 1 (duration: 01m 31s)
  • 20:23 logmsgbot: dduvall@tin Started scap: testwiki to php-1.27.0-wmf.11 and rebuild l10n cache
  • 20:13 mutante: ruthenium: disable puppet, copy data over to osmium (screen)
  • 20:12 mutante: ruthenium: service mysql stop
  • 19:15 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: EventBus plumbing (duration: 00m 30s)
  • 19:14 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Disable Flow on wikitech; add EventBus plumbing (duration: 00m 31s)
  • 19:13 logmsgbot: catrope@tin Synchronized wmf-config/extension-list: Add EventBus (duration: 00m 31s)
  • 19:00 marxarelli: starting branch cut for 1.27.0-wmf.11
  • 18:42 ema: Starting migration of mobile traffic to text cluster https://phabricator.wikimedia.org/T109286
  • 17:54 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.10/extensions/UploadWizard/UploadWizard.config.php: https://gerrit.wikimedia.org/r/#/c/264969/ (duration: 00m 31s)
  • 16:51 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/264964/ (duration: 00m 31s)
  • 16:47 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.10/extensions/Graph/modules/graph-loader.js: https://gerrit.wikimedia.org/r/#/c/264715/ (duration: 00m 31s)
  • 16:45 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/264469/ (duration: 00m 31s)
  • 16:41 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/264437/ (duration: 00m 32s)
  • 14:58 cmjohnson1: reseating asw-c-eqiad uplink module (xe-1/1/0 and xe-1/1/2)
  • 14:29 jynus: reimporting some fawiki tables from production into labsdb hosts
  • 13:52 godog: powercycle ms-be1001
  • 13:51 paravoid: powercycling alsafi
  • 02:53 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Jan 19 02:53:40 UTC 2016 (duration 7m 0s)
  • 02:46 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.10) (duration: 09m 21s)
  • 02:26 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 10m 40s)

2016-01-18

  • 23:26 logmsgbot: krenair@tin Synchronized multiversion/MWMultiVersion.php: https://gerrit.wikimedia.org/r/264895 (duration: 00m 31s)
  • 23:08 logmsgbot: krenair@tin Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/264786/ (duration: 00m 32s)
  • 22:55 logmsgbot: krenair@tin rebuilt wikiversions.php and synchronized wikiversions files: (no message)
  • 22:55 logmsgbot: krenair@tin Synchronized dblists: (no message) (duration: 00m 31s)
  • 22:53 logmsgbot: krenair@tin Synchronized w/static/images/project-logos/wikitech.png: https://gerrit.wikimedia.org/r/#/c/264786/ (duration: 00m 31s)
  • 17:30 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/264758 - labs-only change (duration: 00m 36s)
  • 14:24 godog: powercycle praseodymium
  • 10:42 godog: powercycle ms-be2016, high load avg
  • 10:16 godog: dist-upgrade ms-be3002 to trusty
  • 02:57 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Jan 18 02:57:41 UTC 2016 (duration 7m 8s)
  • 02:50 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.10) (duration: 08m 39s)
  • 02:49 YuviPanda: updated annualreport for foks
  • 02:30 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 11m 38s)

2016-01-17

  • 04:58 YuviPanda: started restbase on restbase1002
  • 02:53 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Jan 17 02:53:19 UTC 2016 (duration 6m 59s)
  • 02:46 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.10) (duration: 08m 53s)
  • 02:26 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 10m 41s)
  • 01:47 paravoid: restarting HHVM on mw1120, mw1125, mw1127, mw1132, mw1148; OOM

2016-01-16

  • 19:52 andrewbogott: renaming and reimaging labcontrol2001 -> labtestweb2001
  • 15:57 milimetric: piwik is taking events on bohrium but the interface can't complete the queries to load because there's too much data. Mysql is maxing the CPU but it seems ok for now, will check again Monday.
  • 15:22 milimetric: restarted mysql on bohrium because it had stopped working (probably due to piwik performance problems)
  • 03:02 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Jan 16 03:02:21 UTC 2016 (duration 6m 57s)
  • 02:55 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.10) (duration: 08m 35s)
  • 02:35 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 18m 55s)

2016-01-15

  • 22:43 logmsgbot: aaron@tin Synchronized wmf-config/CommonSettings.php: Set $wgCentralAuthUseSlaves for testwiki (duration: 00m 33s)
  • 22:38 mutante: gadolinium - shutdown -h now
  • 22:35 mutante: erbium - killing from puppet/icinga/salt
  • 21:54 mutante: mira - starting salt
  • 21:29 mutante: protactinium - shut down, unused system with outdated software
  • 21:09 mutante: (ganglia for ulsfo will be affected, brb)
  • 21:07 mutante: bast4001 - reinstalling with jessie
  • 18:55 ori: disabled gzip in apache for javascript mime types and did an apache config reload
  • 18:04 logmsgbot: ori@tin Synchronized docroot and w: Ie60638b0: Mirror homepage.js from 15.wikipedia.org (duration: 00m 42s)
  • 16:01 godog: bounce hhvm on mw1129 / mw1204
  • 15:41 godog: reimage ms-be3001 with trusty
  • 14:54 godog: reimage ms-fe3002 with trusty
  • 14:13 mark: Temporarily paused md126 RAID check on labstore1001 (sync_action idle)
  • 14:09 chasemp: phab restart phd (reports as not running in phab itself) seems ok now
  • 14:03 mark: set sync_speed_min to 5000 for md126 on labstore1001
  • 13:28 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: w:he as import source for commonswiki (duration: 00m 49s)
  • 12:17 hashar: restarting Jenkins for plugins updates
  • 11:07 _joe_: re-enabled puppet on mw1013, restarted HHVM to make it pick up our latest changes
  • 10:01 moritzm: installed ganeti security updates
  • 09:18 moritzm: installed git security updates on all jessie systems
  • 03:10 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Jan 15 03:10:09 UTC 2016 (duration 6m 48s)
  • 03:03 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.10) (duration: 16m 02s)
  • 02:30 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.10/includes/api/ApiQueryRecentChanges.php: https://gerrit.wikimedia.org/r/264231 (duration: 00m 42s)
  • 02:29 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 14m 00s)
  • 02:23 YuviPanda: pull annualreport git repo on bromine for Krenair
  • 01:00 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.10/includes/api/ApiQueryWatchlist.php: https://gerrit.wikimedia.org/r/#/c/264224/ (duration: 00m 31s)
  • 00:27 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/263905/ (duration: 00m 32s)
  • 00:24 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: touch (duration: 00m 31s)
  • 00:22 logmsgbot: krenair@tin Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/264091/ (duration: 00m 32s)
  • 00:06 mobrovac: restbase started a dump of enwiki to populate storage with mobileapps renders

2016-01-14

  • 23:56 mobrovac: restbase end deploy of dac31a8c
  • 23:49 mobrovac: restbase start deploy of dac31a8c
  • 22:17 csteipp: deployed patch for T122807
  • 19:55 ottomata: restarted eventlogging_sync script to insert batches of 1000
  • 19:31 logmsgbot: dduvall@tin rebuilt wikiversions.php and synchronized wikiversions files: rollback labswiki to wmf.9
  • 19:02 logmsgbot: dduvall@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.27.0-wmf.10
  • 18:40 bblack: removing old eqiad misc-web IP (DNS switched 50h ago (not 26 like above), TTLs are max 1h)
  • 18:39 bblack: removing old eqiad misc-web IP (DNS switched 26h ago, TTLs are max 1h)
  • 18:01 paravoid: turning up BGP with Zayo in eqiad
  • 16:25 logmsgbot: demon@tin Synchronized wmf-config/throttle.php: (no message) (duration: 00m 49s)
  • 15:48 moritzm: installed DHCP security updates across the fleet
  • 14:44 _joe_: powercycling mw1013, console stuck
  • 11:28 godog: bounce uwsgi on labmon1001
  • 11:18 godog: upgrade graphite-carbon / graphite-web on labmon1001
  • 10:38 _joe_: restarting hhvm on odd-numbered jobrunners
  • 10:29 moritzm: installed DHCP security updates on carbon
  • 04:28 paravoid: powercycling mw1005/mw1011
  • 04:24 paravoid: restart hhvm on odd-numbered appservers
  • 02:30 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 12m 21s)
  • 01:32 Krenair: Wikitech rolled back to wmf.9 due to T123583
  • 01:27 logmsgbot: krenair@tin rebuilt wikiversions.php and synchronized wikiversions files: (no message)
  • 01:06 mutante: mw1009 - restarted hhvm
  • 01:00 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.10/extensions/VisualEditor/extension.json: https://gerrit.wikimedia.org/r/#/c/264031/ (duration: 01m 35s)
  • 00:30 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.10/extensions/CirrusSearch/includes: https://gerrit.wikimedia.org/r/#q,263991,n,z (duration: 06m 08s)
  • 00:11 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/263804/ (duration: 00m 31s)
  • 00:10 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/263804/ (duration: 00m 31s)
  • 00:08 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.10/extensions/Echo/modules/echo.variables.less: https://gerrit.wikimedia.org/r/#/c/263767/ (duration: 00m 45s)

2016-01-13

  • 23:46 tgr: T123451: running mwscript sql.php --wiki=metawiki patch-bot_passwords.sql
  • 23:09 mobrovac: restbase end deploy of 536e15b6
  • 22:58 andrewbogott: /etc/init.d/nfs-kernel-server restart on labstore1001
  • 22:54 mobrovac: restbase start deploy of 536e15b6
  • 22:20 logmsgbot: catrope@tin Synchronized wmf-config/: sync labs-only config changes (duration: 00m 32s)
  • 21:54 mobrovac: restbase end deploy of 559a13a
  • 21:44 mobrovac: restbase start deploy of 559a13a
  • 21:40 mdholloway: mobileapps deployed c9e7e28
  • 21:27 aude: Updated cirrus search mappings for testwikidata and wikidata to add new fields
  • 21:02 ori: Disabling Puppet on mw1013 (eqiad jobrunner) to hack in some debug logging into GWT jobs.
  • 20:01 ottomata: dropped MobileWebSectionUsage_14321266 and MobileWebSectionUsage_15038458 from analytics-store eventlogging slave db
  • 19:55 ostriches: *wikimania2017wiki_content
  • 19:55 ostriches: elasticsearch: wikimania2017_content was reporting as missing in logstash, ran updateSearchIndexConfig. messy aliases? Seems to be working again.
  • 19:27 ottomata: dropping eventlogging tables from MobileWebSectionUsage_14321266 and MobileWebSectionUsage_15038458 m4-master log database. These are too large and have been blacklisted from mysql. No more events will be inserted into mysql for these. We are attempting to help replication catch up on the analytics-store slave.
  • 19:11 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.10
  • 18:33 RobH: restarted zotero/mobileapps on sca1*/scb1* respectively for marko's code deploy
  • 18:33 RobH: restarted zotero/mobileapps on sca1*/scb1* respectively
  • 18:27 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: OfficeIT namespace on wikitech (duration: 00m 31s)
  • 18:03 mobrovac: zotero deploying translators 0476aa0
  • 17:12 gwicke: restarted mathoid on scb1001 and scb1002
  • 17:06 gwicke: restarted mathoid on sca1001 and sca1002
  • 17:00 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.10/extensions/Wikidata: https://gerrit.wikimedia.org/r/#/c/263865/ (duration: 00m 41s)
  • 16:31 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/263625/ (duration: 00m 31s)
  • 16:28 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/263341/ (duration: 00m 31s)
  • 16:22 logmsgbot: krenair@tin Synchronized portals: https://gerrit.wikimedia.org/r/#/c/263796/ (duration: 00m 31s)
  • 16:20 logmsgbot: krenair@tin Synchronized wmf-config/Wikibase-production.php: https://gerrit.wikimedia.org/r/#/c/263838/ (duration: 00m 31s)
  • 16:14 logmsgbot: krenair@tin Synchronized wmf-config/Wikibase.php: https://gerrit.wikimedia.org/r/#/c/263354/ (duration: 00m 31s)
  • 16:03 logmsgbot: krenair@tin Synchronized docroot/noc: https://gerrit.wikimedia.org/r/#/c/263370/3 (duration: 00m 31s)
  • 14:11 godog: bounce hhvm on mw1007
  • 14:03 godog: bounce hhvm on mw1005, powercycle mw1011
  • 13:46 godog: bounce hhvm on mw1009, powercycle mw1003
  • 13:39 godog: bounce hhvm on mw1013
  • 10:31 paravoid: upgrading grafana 2.6.0-beta1 -> 2.6.0
  • 06:45 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.9/extensions/GWToolset: Ib9375b: Make sure XMLReader::close() is always called (T122069) (duration: 00m 32s)
  • 06:43 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.10/extensions/GWToolset: Ib9375b: Make sure XMLReader::close() is always called (T122069) (duration: 01m 07s)
  • 03:15 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Jan 13 03:15:57 UTC 2016 (duration 7m 13s)
  • 03:08 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.10) (duration: 16m 09s)
  • 02:57 Krinkle: Manually killed uwsgi graphite-web child processes on graphite1001. Service recovered itself from there.
  • 02:44 Krinkle: Graphite is down. Consistently returns HTTP 502 Bad Gateway for any/all requests
  • 02:34 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 11m 13s)
  • 01:33 yurik: deployed tilerator maps service
  • 01:19 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.10/extensions/Echo/Resources.php: https://gerrit.wikimedia.org/r/#/c/263645/ (duration: 00m 32s)
  • 01:18 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.10/extensions/Flow/modules/editor/editors/visualeditor/mw.flow.ve.Target.js: https://gerrit.wikimedia.org/r/#/c/263644/ (duration: 00m 31s)
  • 01:03 logmsgbot: krenair@tin Synchronized portals: https://gerrit.wikimedia.org/r/#/c/263770/ - after having done the submodule update this time (duration: 00m 31s)
  • 00:37 logmsgbot: krenair@tin Synchronized portals: https://gerrit.wikimedia.org/r/#/c/263770/ (duration: 00m 33s)
  • 00:31 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/261994/ (duration: 00m 31s)
  • 00:28 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/262895/ (duration: 00m 32s)
  • 00:25 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/262894/ (duration: 00m 30s)
  • 00:17 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/263237/ (duration: 00m 31s)
  • 00:15 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/262999/ (duration: 00m 31s)
  • 00:10 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/263201/ (duration: 00m 30s)
  • 00:08 yurik: switched all maps kartotherian servers to v5, restarted
  • 00:06 logmsgbot: krenair@tin Synchronized images/mobile/wikivoyage.png: https://gerrit.wikimedia.org/r/#/c/263201/ (duration: 00m 31s)
  • 00:06 logmsgbot: krenair@tin Synchronized images/mobile/wikidata.png: https://gerrit.wikimedia.org/r/#/c/263201/ (duration: 00m 32s)

2016-01-12

  • 21:58 ori: Restarting jobchron / jobrunner / HHVM on all job runners for I44990808
  • 21:07 logmsgbot: hoo@tin Synchronized php-1.27.0-wmf.10/extensions/Math/: Introduce a "MathEnableWikibaseDataType" config (duration: 00m 32s)
  • 20:52 logmsgbot: hoo@tin Synchronized wmf-config/: Set $wgMathEnableWikibaseDataType to false (duration: 01m 29s)
  • 20:44 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.27.0-wmf.10
  • 20:34 logmsgbot: thcipriani@tin Finished scap: testwiki to php-1.27.0-wmf.10 and rebuild l10n cache (duration: 54m 42s)
  • 20:14 mobrovac: restbase switching restbase200x to node 4.2
  • 20:13 mobrovac: restbase switch of restbase100[1-4] to node 4.2 completed
  • 20:10 mobrovac: restbase switching restbase100[1-4] to node 4.2
  • 19:39 logmsgbot: thcipriani@tin Started scap: testwiki to php-1.27.0-wmf.10 and rebuild l10n cache
  • 19:31 logmsgbot: dduvall@tin scap failed: CalledProcessError Command 'sudo -u www-data -n -- /bin/mktemp' returned non-zero exit status 1 (duration: 00m 42s)
  • 19:30 logmsgbot: dduvall@tin Started scap: testwiki to php-1.27.0-wmf.10 and rebuild l10n cache
  • 19:26 YuviPanda: import new r-base package into carbon
  • 18:15 marxarelli: cutting MW branch 1.27.0-wmf.10
  • 17:37 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/263632/ (duration: 00m 31s)
  • 16:53 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Import sources on gu.wikipedia gerrit:258441 (duration: 00m 29s)
  • 16:48 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Get rid of old unused $wgAllowed* variables gerrit:256853 (duration: 00m 29s)
  • 16:47 _joe_: restarted salt-minion on tin
  • 16:44 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add portal namespace to ps.wikipedia.org gerrit:255519 (duration: 00m 30s)
  • 16:42 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove proxyunbannable gerrit:254842 (duration: 00m 30s)
  • 16:37 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Allow sysop to grant and revoke transwiki on gu.wikipedia gerrit:258474 (duration: 00m 29s)
  • 16:33 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Namespace configuration on pa.wikipedia gerrit:258436 (duration: 00m 29s)
  • 16:22 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Namespace configuration on my.wikipedia gerrit:258442 (duration: 00m 30s)
  • 15:56 godog: reprovision ms-fe3001 with jessie
  • 14:55 ema: added myself to ops and wmf ldap groups
  • 11:57 _joe_: enabling auth on the production etcd cluster
  • 08:37 paravoid: ms-be1002: echo b > /proc/sysrq-trigger, kernel misbehaving and unrecoverable (out of kernel memory/XFS issues)
  • 07:38 paravoid: cr2-eqiad: reenable BGP peerings with GTT
  • 05:31 paravoid: rm CirrusSearchRequests.log-201510*.gz on fluorine (saving ~200G)
  • 04:07 paravoid: cleaning up elastic1006's /var/log from old logs
  • 03:59 paravoid: reenabling puppet on sca1001/2; no reason was left
  • 02:33 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Jan 12 02:33:00 UTC 2016 (duration 6m 55s)
  • 02:26 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 10m 47s)
  • 00:46 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: rv 443026e3ad18934dd0017a258673d88104cf6b5e (duration: 00m 29s)
  • 00:32 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/258670/ (duration: 00m 30s)
  • 00:29 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/258672/ (duration: 00m 30s)
  • 00:25 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/258453/ (duration: 00m 30s)
  • 00:18 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/258444/ (duration: 00m 30s)
  • 00:14 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/255361/ (duration: 00m 30s)
  • 00:10 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/244140/ (duration: 00m 30s)
  • 00:09 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/244140/ (duration: 00m 30s)
  • 00:06 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/260242/ (duration: 00m 30s)

2016-01-11

  • 22:52 logmsgbot: jzerebecki@tin Synchronized wmf-config/throttle.php: deploying https://gerrit.wikimedia.org/r/#/c/263427/ (duration: 00m 30s)
  • 22:48 YuviPanda: restart eventlogging_synch on dbstore1002
  • 22:47 logmsgbot: jzerebecki@tin Synchronized php-1.27.0-wmf.9/extensions/Wikidata/extensions/Wikibase/repo/maintenance/dispatchChanges.php: restoring truncated Wikidata dispatchChanges.php to let dispatchers run again (duration: 00m 30s)
  • 22:46 mutante: restbase1004, restbase2002, restbase2005 - manually install nodejs
  • 22:45 logmsgbot: jzerebecki@tin Synchronized php-1.27.0-wmf.9/extensions/Wikidata/extensions/Wikibase/repo: deploying https://gerrit.wikimedia.org/r/#/c/253898/ with dispatchChanges.php still truncated (duration: 00m 33s)
  • 22:40 mutante: restbase1001 - apt-get install nodejs
  • 22:40 jzerebecki: dispatchChanges.php killed on terbium
  • 22:38 logmsgbot: jzerebecki@tin Synchronized php-1.27.0-wmf.9/extensions/Wikidata/extensions/Wikibase/repo/maintenance/dispatchChanges.php: truncating Wikidata dispatchChanges.php to stop dispatchers as preparation for https://gerrit.wikimedia.org/r/#/c/253898/ (duration: 00m 31s)
  • 21:19 papaul: pc200[4-6] - signing puppet certs, salt-key, initial run
  • 21:13 subbu: finished deploying parsoid sha 07494cf2
  • 21:06 papaul: installing OS on pc200[4-6]
  • 21:06 subbu: synced new code; restarted parsoid on wtp1003 as a canary
  • 21:02 subbu: starting parsoid deploy
  • 18:52 RobH: rt.w.o cert expired and its replacement will be later today (rt is internal ops only tool)
  • 18:36 RobH: tendril cert updated and neon returned to normal service
  • 18:30 ori: Restarting HHVM on all job runners, to vacate memory now that the cause of the leak appears to have subsided.(T122069)
  • 18:24 RobH: tendril updating ssl cert on neon, https may flap for a second (this is on neon, so icinga https portal may also flap)
  • 17:29 hoo: Updated Wikidata's property suggester with data from today's json dump
  • 17:16 papaul: db2033 - signing puppet certs, salt-key, initial run
  • 16:58 papaul: installing OS on db2033
  • 16:49 logmsgbot: thcipriani@tin Synchronized robots.txt: SWAT: Remove overager unrequested /wiki/User: robots.txt rule gerrit:263360 (duration: 00m 30s)
  • 16:41 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable new user groups on gu.wikipedia.org gerrit:255810 (duration: 00m 30s)
  • 16:34 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: dewikibooks: Set $wgRestrictDisplayTitle to false gerrit:260964 (duration: 00m 30s)
  • 16:30 godog: halt ms-be1013, required to reset idrac
  • 16:27 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable global AubseFilter at French Wikipedia gerrit:257868 (duration: 00m 29s)
  • 16:23 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Changed user group rights at trwikiquote gerrit:261869 (duration: 00m 30s)
  • 16:16 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Added noindex rule for uawikimedia user namespace gerrit:261902 (duration: 00m 30s)
  • 16:09 logmsgbot: thcipriani@tin Synchronized robots.txt: SWAT: Tidy robots.txt gerrit:240065 (duration: 00m 30s)
  • 16:08 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Set wgLocaltimezone for orwiki gerrit:260745 (duration: 00m 29s)
  • 16:03 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add enwiki as transwiki import source for ta.wikipedia gerrit:262352 (duration: 00m 33s)
  • 15:05 godog: repool restbase1004 in pybal, fully bootstrapped and running latest code
  • 11:14 _joe_: upgrading etcd to 2.2.1 in production
  • 10:36 _joe_: updating nodejs on restbase-test2002
  • 07:17 _joe_: restarting HHVM on a few jobrunners
  • 02:32 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Jan 11 02:32:37 UTC 2016 (duration 6m 55s)
  • 02:25 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 10m 39s)
  • 01:11 paravoid: deactivating eqiad<->GTT BGP peering, reported network issues (P2469)

2016-01-10

  • 22:00 gwicke: restbase: 1005-1009 now on node 4.2
  • 19:44 paravoid: powercycling mw1004, mw1008, mw1012
  • 19:38 paravoid: restarting hhvm on jobrunners again
  • 12:40 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 626m 20s)
  • 10:13 ori: disabled categoryMembershipChange on mw1165 too, then restart jobrunner / jobchron / hhvm on mw1165 and mw1164
  • 08:55 ori: mw1166 -- disabled puppet; disabled categoryMembershipChange jobs
  • 08:48 ori: mw1167 -- disabled puppet; disabled deleteLinks and refreshLinks* jobs
  • 08:45 ori: mw1168 -- disabled puppet; disabled restbase jobs
  • 08:41 ori: mw1169 -- disables cirrus jobs.
  • 08:33 ori: Attempting to isolate cause of T122069 by toggling job types on mw1169. Disabling Puppet to prevent it from clobbering config changes.
  • 08:29 paravoid: restarting hhvm on jobrunners again
  • 04:58 paravoid: powercycling mw1005, mw1008, mw1009 -- unresponsive due to OOM
  • 04:56 paravoid: restarting HHVM on eqiad jobrunners, OOM, memleak faster than the 24h restarts

2016-01-09

  • 02:33 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Jan 9 02:33:40 UTC 2016 (duration 6m 57s)
  • 02:26 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 11m 19s)

2016-01-08

  • 23:49 RobH: stalled puppet on carbon for now, messing with partman files
  • 02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Jan 8 02:31:46 UTC 2016 (duration 7m 0s)
  • 02:24 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 10m 15s)

2016-01-07

  • 23:24 akosiaris: repooled scb1002 for mobileapps
  • 23:24 akosiaris: enabled puppet,salt on scb1001
  • 23:23 mobrovac: mobileapps deploying 58b371a on scb1001
  • 23:09 mobrovac: mobileapps deploying 58b371a on scb1002
  • 23:01 akosiaris: apt-mark hold nodejs on scb1001, etherpad1001 and maps-test200{1,2,3,4}
  • 22:58 akosiaris: disable puppet and salt on scb1001 from nodejs 4.2 transition
  • 22:57 akosiaris: depool scb1002 for mobileapps. Transition to nodejs 4.2 ongoing
  • 19:21 YuviPanda: started tools / maps backup on labstore1001
  • 19:13 YuviPanda: remove snapshots others20150815030010, others20150815030010, maps20151216040005 and maps20151028040004 that were all stale and should've been removed anyway (on labstore2001)
  • 19:13 YuviPanda: remove snapshots others20150815030010, others20150815030010, maps20151216040005 and maps20151028040004 that were all stale and should've been removed anyway
  • 19:11 jynus: setting up watchdog process killing long running queries on db1051
  • 19:11 YuviPanda: run sudo lvremove backup/tools20151216020005 on labstore2001 to clean up full snapshot
  • 18:54 _joe_: also resetting the drac
  • 18:53 _joe_: powercycling ms-be1013
  • 02:32 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Jan 7 02:32:04 UTC 2016 (duration 6m 54s)
  • 02:25 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 10m 33s)

2016-01-06

  • 23:03 gwicke: switched restbase1009 to node 4.2 for testing, and restarted restbase; see https://phabricator.wikimedia.org/T107762
  • 02:34 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Jan 6 02:34:38 UTC 2016 (duration 6m 53s)
  • 02:27 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 10m 30s)

2016-01-05

  • 22:38 logmsgbot: aaron@tin Synchronized rpc: 830e1ed8d80295710dc02f18102b4fadae7fca86 (duration: 00m 55s)
  • 18:34 logmsgbot: jzerebecki@tin scap aborted: deploy-log (duration: 00m 04s)
  • 18:34 logmsgbot: jzerebecki@tin Started scap: deploy-log
  • 15:47 ottomata: transitioned analytics1001 to active namenode
  • 03:51 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.9/includes/specials/SpecialJavaScriptTest.php: Idaacf71870 (duration: 00m 30s)
  • 03:50 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.9/resources/src/mediawiki.special/: Idaacf71870 (duration: 00m 30s)
  • 03:49 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.9/resources/Resources.php: Idaacf71870 (duration: 00m 36s)
  • 02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Jan 5 02:31:46 UTC 2016 (duration 6m 54s)
  • 02:24 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 10m 13s)

2016-01-04

  • 20:50 mutante: ms-be1011 - powercycled, was frozen
  • 20:43 mutante: ms-be2007 - System halted!Error: Integrated RAID
  • 20:42 mutante: ms-be2007 - powercycle (was status: on but all frozen) (i assume xfs like be2006 appears in SAL recently)
  • 20:36 mutante: mw2019 - puppet run (icinga claimed it failed but just here)
  • 20:19 mutante: rutherfordium - attempt to restart with gnt-instance
  • 20:12 mutante: rutherfordium (people.wm) was down for days per icinga - then magically fixes itself when i connect to console but before even loggin in (ganeti VM)
  • 20:00 mutante: mw1123 - start HHVM (was 503 and service stopped)
  • 19:28 mutante: elastic1006 - out of disk - gzip eqiad_index_search_slowlog.log files
  • 17:37 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.9/extensions/Graph/: Deployed Graph ext - gerrit 262357 (duration: 00m 33s)
  • 02:32 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Jan 4 02:32:10 UTC 2016 (duration 6m 53s)
  • 02:25 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 10m 05s)

2016-01-03

  • 02:32 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Jan 3 02:31:58 UTC 2016 (duration 6m 52s)
  • 02:25 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 10m 22s)

2016-01-02

  • 03:34 twentyafterfour: deploying https://gerrit.wikimedia.org/r/261725, restarted apache2 on iridium
  • 02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Jan 2 02:31:28 UTC 2016 (duration 6m 58s)
  • 02:24 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 10m 09s)
  • 01:04 YuviPanda: imported vagrant 1.8.1 for jessie per bd808
  • 00:04 ori: (at 23:46 UTC) restarted nova-compute on labvirt1002

2016-01-01

  • 23:50 legoktm: restarted nodepool on labnodepool1001
  • 23:37 ori: restarting nodepool on labnodepool1001.eqiad.wmnet (T122731)
  • 19:41 bd808: Updated scholarships.wikimedia.org with latest translation data from translatewiki
  • 02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Jan 1 02:30:27 UTC 2016 (duration 6m 47s)
  • 02:23 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 09m 58s)


2000s

2010s

2020s