Server Admin Log/Archive 41

2020-07-31

• 23:48 ejegg: updated payments-wiki from c365c136d2 to cd012f37f1
• 22:53 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 22:53 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 22:53 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 22:53 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 22:03 mutante: wtp2019 - parsoid could not start after reimaging - was missing /etc/parsoid/config.yaml which is a symbolic link deep onto /srv/deployment/parsoid/deploy-cache/.. like in some other cases before manually deleted deploy-cache dir and ran puppet again .. T258775
• 21:58 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=wtp2020.codfw.wmnet
• 21:57 dzahn@cumin1001: conftool action : set/pooled=no; selector: name=wtp2019.codfw.wmnet
• 21:50 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=wtp2019.codfw.wmnet
• 21:36 mutante: [wtp2019:~] $ sudo rm -rf /srv/deployment/parsoid/deploy-cache
• 21:13 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=wtp2018.codfw.wmnet
• 20:57 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 20:56 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 20:56 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 20:56 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 20:56 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 20:56 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 20:55 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 20:55 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 20:55 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 20:55 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 20:55 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 20:55 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 20:55 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 20:55 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 20:22 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 20:22 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 20:22 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 20:22 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 20:11 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 20:11 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 20:03 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 20:02 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 19:51 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 19:51 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 19:39 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 19:39 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 19:21 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=wtp2017.codfw.wmnet
• 18:57 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=wtp2016.codfw.wmnet
• 17:45 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 17:45 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 17:45 mutante: rebooting / reinstalling OS on xhgui1001
• 17:25 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 17:25 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 17:21 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 17:21 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 17:13 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 17:13 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 17:12 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 17:11 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 16:24 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 16:22 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 16:22 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 16:20 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 15:32 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:30 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 13:52 elukey: update cr1/cr2-eqiad's analytics filters (ref: https://gerrit.wikimedia.org/r/c/operations/homer/public/+/617649/)
• 13:51 moritzm: installing cups security updates (client-side tools/libs only)
• 13:20 moritzm: installing openjpeg2 security updates
• 13:04 kormat: proudly uploaded version 0.1 of python3-wmfmariadbpy + wmfmariadbpy
• 11:55 moritzm: installing mercurial security updates
• 11:21 jynus: restart dbstore1004
• 11:19 moritzm: installing ffmpeg security updates for jessie (standard version from security.debian.org, not the VP9-enabled component)
• 11:16 moritzm: imported ffmpeg 3.2.15-0+deb9u1+wmf1 to component/vp9 for stretch-wikimedia T259336
• 07:51 moritzm: updating lilypond on mw* servers
• 07:50 moritzm: uploaded lilypond 2.19.81+really-2.18.2-13~bpo9+1+wmf1 to stretch-wikimedia T256877
• 07:07 elukey: stop mysql replication on db1108; update port config for mysql instances and restart them; restart replication on instances
• 06:32 elukey: roll restart of druid brokers on druid100[4-8] to pick up new changes
• 06:26 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 06:26 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 05:59 moritzm: installing qemu updates on stretch
• 04:56 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 04:54 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 04:22 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 04:20 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 03:55 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 03:53 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 03:14 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 03:12 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 02:57 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 02:55 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 00:47 tstarling@deploy1001: Synchronized wmf-config/CommonSettings.php: disable lilypond execution again (duration: 01m 10s)
• 00:07 catrope@deploy1001: Synchronized php-1.36.0-wmf.2/extensions/Echo/modules/mobile/notificationsFilterOverlay.js: T258954 (duration: 01m 06s)
• 00:06 catrope@deploy1001: Synchronized php-1.36.0-wmf.1/extensions/Echo/modules/mobile/notificationsFilterOverlay.js: T258954 (duration: 01m 10s)
• 00:00 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=wtp2015.codfw.wmnet

2020-07-30

• 23:56 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 23:56 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 22:59 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=wtp2014.codfw.wmnet
• 22:34 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 22:34 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 22:27 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 22:27 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 22:12 brennen@deploy1001: rebuilt and synchronized wikiversions files: Revert group2 wikis to 1.36.0-wmf.1
• 21:52 brennen@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.2
• 21:52 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 21:51 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 21:51 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 21:51 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 21:45 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=wtp2013.codfw.wmnet
• 21:41 mutante: revoking and resigning puppet cert for xhgui2001.codfw.wmnet T259206
• 21:40 catrope@deploy1001: Synchronized php-1.36.0-wmf.2/extensions/GrowthExperiments/: T258609 (duration: 01m 06s)
• 21:40 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 21:40 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 21:39 catrope@deploy1001: Synchronized php-1.36.0-wmf.2/skins/MinervaNeue/: T258939 (duration: 01m 08s)
• 21:30 mutante: reinstalling xhgui2001
• 21:27 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 21:27 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 21:27 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 21:27 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 21:10 ryankemper@deploy1001: Finished deploy [wdqs/wdqs@e797cf0]: 0.3.42 (duration: 24m 41s)
• 20:45 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=wtp2012.codfw.wmnet
• 20:45 ryankemper@deploy1001: Started deploy [wdqs/wdqs@e797cf0]: 0.3.42
• 20:23 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 20:23 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 20:09 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 20:09 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 20:08 mutante: [wtp2012:~] $ sudo rm -rf /srv/deployment/parsoid/deploy-cache
• 19:24 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=wtp2011.codfw.wmnet
• 19:13 brennen@deploy1001: rebuilt and synchronized wikiversions files: Revert group2 wikis to 1.36.0-wmf.1
• 19:10 brennen@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.2
• 19:01 mforns@deploy1001: Finished deploy [analytics/refinery@adb0d09]: Regular analytics weekly train [analytics/refinery@adb0d09b6584a7a26143623cf6173ae8983423e3] (duration: 10m 41s)
• 18:59 mutante: imported twig (php-twig) into APT repo
• 18:50 mforns@deploy1001: Started deploy [analytics/refinery@adb0d09]: Regular analytics weekly train [analytics/refinery@adb0d09b6584a7a26143623cf6173ae8983423e3]
• 18:34 Urbanecm: Morning B&C done
• 18:32 urbanecm@deploy1001: Synchronized php-1.36.0-wmf.2/extensions/Kartographer/modules/box/Map.js: aa3dbd5: Disable panning and zooming until ready (T257872) (duration: 01m 06s)
• 18:24 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 617516: Add import sources for yuewiktionary | https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/617516; 617518: Fix definition of yuewiktionary import sources | https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/617518 # T258913 (duration: 01m 06s)
• 18:08 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 18:08 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 18:06 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 14ef2ec: sysop_itwiki: Set favicon to Wikimedia_logo_blue.svg (T259243; 2/2) (duration: 01m 06s)
• 18:04 urbanecm@deploy1001: Synchronized static/favicon/wmf-blue.ico: 14ef2ec: sysop_itwiki: Set favicon to Wikimedia_logo_blue.svg (T259243; 1/2) (duration: 01m 06s)
• 17:04 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@d3ab874]: airflow: refinery_drop_hive_partitions: Fix kerberos token passing (duration: 00m 55s)
• 17:03 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@d3ab874]: airflow: refinery_drop_hive_partitions: Fix kerberos token passing
• 16:53 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 16:53 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 16:53 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 16:53 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 16:44 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 16:44 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 16:44 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 16:44 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 16:41 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 16:41 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 16:41 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 16:40 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 16:40 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 16:38 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
• 14:30 moritzm: installing squid security updates
• 14:04 volans@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:50 volans@cumin1001: START - Cookbook sre.dns.netbox
• 13:47 vgutierrez: upgrade acme-chief to version 0.27 - T255249
• 13:47 vgutierrez: upload acme-chief 0.27 to apt.wm.o (buster) - T255249
• 13:46 moritzm: installing qemu security updates on Buster
• 13:02 jayme: imported chartmuseum_0.12.0-3 to buster-wikimedia
• 12:07 elukey: upgrade of the druid public cluster (serving AQS) from 0.12.3 to 0.19
• 11:53 urbanecm@deploy1001: Synchronized static/favicon/: c08f774: Revert "sysop_itwiki: Set favicon to Wikimedia_logo_blue.svg" (T259243) (duration: 01m 06s)
• 11:48 urbanecm@deploy1001: Synchronized static/favicon/wmf-blue.ico: 399e9c5: sysop_itwiki: Set favicon to Wikimedia_logo_blue.svg (T259243; 1/2) (duration: 01m 06s)
• 11:46 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: fc48441: Add import sources to sysop_itwiki (T259243) (duration: 01m 08s)
• 11:44 urbanecm@deploy1001: Synchronized wmf-config/CommonSettings.php: fc5de15: ClosedWikiProvider: Do not run when $wmgUseCentralAuth is false (T259246) (duration: 01m 07s)
• 11:42 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 7aa0c23: sysop_itwiki: Add several pages to wgWhitelistRead (T259243) (duration: 01m 06s)
• 11:39 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 5ea4bc8: sysop_itwiki: Add WP as an alias for NS_PROJECT (T259243) (duration: 01m 08s)
• 10:49 liw@deploy1001: Synchronized php: group1 wikis to 1.36.0-wmf.2 (duration: 01m 07s)
• 10:48 liw@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.36.0-wmf.2
• 10:38 marostegui: Reload haproxy on dbproxy1013 and dbproxy1015
• 08:43 godog: flip smokeping/librenms from netmon2001 to netmon1002 - T247967
• 07:52 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 07:50 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 07:32 filippo@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
• 07:31 filippo@cumin1001: START - Cookbook sre.hosts.decommission
• 07:22 filippo@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 07:22 filippo@cumin1001: START - Cookbook sre.hosts.downtime
• 07:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1121', diff saved to https://phabricator.wikimedia.org/P12129 and previous config saved to /var/cache/conftool/dbconfig/20200730-071633-marostegui.json
• 06:57 elukey: upload druid_0.19.0-1 packages to buster-wikimedia
• 05:26 marostegui: Deploy MCR schema change on labswiki (wikitech) T238966
• 02:18 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=wtp2010.codfw.wmnet
• 01:53 dpifke@deploy1001: Finished deploy [performance/arc-lamp@ad87f69]: Deploying https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/615302 (duration: 00m 05s)
• 01:53 dpifke@deploy1001: Started deploy [performance/arc-lamp@ad87f69]: Deploying https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/615302
• 01:37 eileen: civicrm revision changed from cc5d17fbaf to 150c3476c4, config revision is b6ece03513
• 01:33 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 01:33 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 01:30 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=wtp2009.codfw.wmnet
• 01:22 mutante: imported in apt.wikimedia.org for buster: php-slim, php-slim-views, php-perftools-xhgui-collector, php-pimple, php-psr-http-server-middleware, php-psr-http-server-handler, xhgui
• 01:07 mholloway-shell@deploy1001: Synchronized php-1.36.0-wmf.2/extensions/JsonConfig: Backport: Implement GetContentModels hook (T259126) (duration: 01m 07s)
• 00:52 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 00:52 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 00:23 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 00:23 dzahn@cumin1001: START - Cookbook sre.hosts.downtime

2020-07-29

• 23:51 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 23:51 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 23:41 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=wtp2008.codfw.wmnet
• 23:28 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 23:28 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 23:09 Urbanecm: Run mwscript namespaceDupes.php --wiki=mswiktionary --fix (T255391)
• 23:07 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 396a395: Add several extra namespaces for mswiktionary (T255391) (duration: 01m 07s)
• 22:46 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=wtp2006.codfw.wmnet
• 22:45 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=wtp2007.codfw.wmnet
• 22:00 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 22:00 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 21:22 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 21:22 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 21:15 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 21:15 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 21:00 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 21:00 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 20:35 crusnov@deploy1001: Finished deploy [netbox/deploy@fde9dfe]: Test deploy of 2.8.8 to netbox-next pt2 (duration: 00m 05s)
• 20:35 crusnov@deploy1001: Started deploy [netbox/deploy@fde9dfe]: Test deploy of 2.8.8 to netbox-next pt2
• 20:35 crusnov@deploy1001: Finished deploy [netbox/deploy@fde9dfe]: Test deploy of 2.8.8 to netbox-next (duration: 01m 12s)
• 20:34 crusnov@deploy1001: Started deploy [netbox/deploy@fde9dfe]: Test deploy of 2.8.8 to netbox-next
• 20:19 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=wtp2004.codfw.wmnet
• 19:45 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 19:44 herron@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 19:43 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 19:41 herron@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 19:41 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 19:41 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 19:29 herron@cumin1001: START - Cookbook sre.ganeti.makevm
• 19:27 herron@cumin1001: START - Cookbook sre.ganeti.makevm
• 19:20 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 19:20 herron@cumin1001: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99)
• 19:19 herron@cumin1001: START - Cookbook sre.ganeti.makevm
• 19:18 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 19:18 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 19:17 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 19:17 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 19:16 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 19:04 qchris: Restarting Gerrit on gerrit2001 (gerrit-replica) to make security fix effective.
• 19:04 qchris@deploy1001: Finished deploy [gerrit/gerrit@9275b30]: Gerrit to v3.2.3-1-g185bdc3a69 on gerrit2001 (duration: 00m 09s)
• 19:03 qchris@deploy1001: Started deploy [gerrit/gerrit@9275b30]: Gerrit to v3.2.3-1-g185bdc3a69 on gerrit2001
• 19:00 qchris: Restarting Gerrit on gerrit1001 to make security fix effective.
• 19:00 qchris@deploy1001: Finished deploy [gerrit/gerrit@9275b30]: Gerrit to v3.2.3-1-g185bdc3a69 on gerrit1001 (duration: 00m 08s)
• 19:00 qchris@deploy1001: Started deploy [gerrit/gerrit@9275b30]: Gerrit to v3.2.3-1-g185bdc3a69 on gerrit1001
• 18:47 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 18:44 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 18:42 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 18:42 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 18:39 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 18:39 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 18:36 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 18:36 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 18:36 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 18:36 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 18:32 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 18:32 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 18:13 Urbanecm: Morning B&C window is done
• 18:13 urbanecm@deploy1001: Synchronized php-1.36.0-wmf.2/extensions/DiscussionTools/: 00ecec8: Revert new reply API for now (T252558) (duration: 01m 06s)
• 18:11 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: d54f041: Enable Translate extension at plwikimedia (T259087) (duration: 01m 08s)
• 18:07 urbanecm@deploy1001: Synchronized dblists/visualeditor-nondefault.dblist: a237f5b: Move VisualEditor from beta to default on enwikiversity (T258992) (duration: 01m 06s)
• 18:05 Urbanecm: Create tables for Translate extension in plwikimedia (T259087)
• 18:05 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 18:03 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 18:03 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 18:01 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 18:01 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 18:01 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 17:50 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=wtp2003.codfw.wmnet
• 17:50 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=wtp2002.codfw.wmnet
• 17:21 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 17:21 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 17:19 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 17:16 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 17:16 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 17:16 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 16:47 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 16:45 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 16:45 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 16:43 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 16:43 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 16:43 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 16:25 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 16:24 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 16:24 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 16:24 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 16:21 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 16:19 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 16:18 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 16:16 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 16:16 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 16:16 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 16:16 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 16:16 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 16:15 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 16:15 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 16:02 volans@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 15:48 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 617167: Revert "Set muswiki to read only" | https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/617167 (T259004) (duration: 01m 06s)
• 15:44 volans@cumin1001: START - Cookbook sre.dns.netbox
• 15:33 liw@deploy1001: rebuilt and synchronized wikiversions files: Revert "group[0|1] wikis to 1.36.0-wmf.1"
• 15:24 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 617152: Set muswiki to read only | https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/617152 (T259004) (duration: 01m 08s)
• 15:10 jayme: imported docker-report_0.0.8-1 to buster-wikimedia
• 14:49 moritzm: installing ruby-json security updates
• 14:34 pt1979@cumin2001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 14:30 jbond42: install curl security update for jessie
• 14:29 moritzm: installing exiv2 security updates
• 14:27 pt1979@cumin2001: START - Cookbook sre.dns.netbox
• 13:55 volans: migrating *all* codfw mgmt DNS records to the autogenerated ones via Netbox - T233183
• 13:50 volans@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:45 volans@cumin1001: START - Cookbook sre.dns.netbox
• 13:29 jayme@cumin1001: conftool action : set/pooled=yes; selector: name=wtp2001.codfw.wmnet
• 13:05 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:05 liw@deploy1001: Synchronized php: group1 wikis to 1.36.0-wmf.2 (duration: 01m 07s)
• 13:04 liw@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.36.0-wmf.2
• 13:03 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 12:58 volans@cumin1001: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
• 12:56 volans@cumin1001: START - Cookbook sre.dns.netbox
• 12:49 cmjohnson@cumin1001: END (ERROR) - Cookbook sre.dns.netbox (exit_code=97)
• 12:48 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
• 12:44 moritzm: imported curl 7.38.0-4+deb8u16+wmf1 to apt.wikimedia.org (jessie-wikimedia) T259102
• 12:30 urbanecm@deploy1001: Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 02m 21s)
• 12:28 urbanecm@deploy1001: Synchronized langlist: Creating avkwiki (T257943) (duration: 01m 05s)
• 12:27 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Creating avkwiki (T257943) (duration: 01m 03s)
• 12:26 urbanecm@deploy1001: Synchronized static/images/project-logos/: Creating avkwiki (T257943) (duration: 01m 06s)
• 12:24 urbanecm@deploy1001: rebuilt and synchronized wikiversions files: Creating avkwiki (T257943)
• 12:15 urbanecm@deploy1001: Synchronized dblists: Creating avkwiki (T257943) (duration: 01m 06s)
• 12:14 urbanecm@deploy1001: Synchronized wmf-config/db-codfw.php: Creating avkwiki (T257943) (duration: 01m 06s)
• 12:12 urbanecm@deploy1001: Synchronized wmf-config/db-eqiad.php: Creating avkwiki (T257943) (duration: 01m 05s)
• 12:09 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 12:07 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 12:07 moritzm: rebooting idp2001 for kernel update
• 11:41 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 252bb6c: Add Wikipedia wordmark for trwiki (T255489; sync 2/2) (duration: 01m 05s)
• 11:39 urbanecm@deploy1001: Synchronized static/images/mobile/copyright/wikipedia-wordmark-tr.svg: 252bb6c: Add Wikipedia wordmark for trwiki (T255489; sync 1/2) (duration: 01m 06s)
• 11:36 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 9f7e032: Fix overindentation (duration: 01m 08s)
• 11:11 Lucas_WMDE: EU B&C window done
• 11:09 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ printf 'https://en.wikipedia.org/static/images/project-logos/%s\n' 'wuuwiki.png' 'wuuwiki-1.5x.png' 'wuuwiki-2x.png' | mwscript purgeList.php # T259005
• 11:08 lucaswerkmeister-wmde@deploy1001: Synchronized static/images/project-logos/: Config: Change the logo for Wu Wikipedia (T259005) (duration: 01m 08s)
• 10:40 vgutierrez: rolling upgrade of ATS to version 8.0.8-1wm2
• 10:21 tstarling@deploy1001: Synchronized php-1.36.0-wmf.1/extensions/Score/includes/Score.php: do not offer .ly downloads (duration: 01m 07s)
• 10:19 tstarling@deploy1001: Synchronized php-1.36.0-wmf.1/extensions/Score/extension.json: do not offer .ly downloads (duration: 01m 20s)
• 10:12 vgutierrez: upgrade ATS to version 8.0.8-1wm2 on cp3064 and cp3065
• 09:44 vgutierrez: upgrade ATS to version 8.0.8-1wm2 on cp5006 and cp5012
• 09:20 jayme@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 09:20 jayme@cumin1001: START - Cookbook sre.hosts.downtime
• 09:16 vgutierrez: upgrade ATS to version 8.0.8-1wm2 on cp4026 and cp4032
• 09:15 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1112', diff saved to https://phabricator.wikimedia.org/P12115 and previous config saved to /var/cache/conftool/dbconfig/20200729-091528-marostegui.json
• 09:15 vgutierrez: upload trafficserver 8.0.8-1wm2 to apt.wm.o (buster)
• 09:13 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1112', diff saved to https://phabricator.wikimedia.org/P12114 and previous config saved to /var/cache/conftool/dbconfig/20200729-091319-marostegui.json
• 09:10 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1112', diff saved to https://phabricator.wikimedia.org/P12113 and previous config saved to /var/cache/conftool/dbconfig/20200729-091006-marostegui.json
• 08:55 marostegui: The above was db1112
• 08:55 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1121', diff saved to https://phabricator.wikimedia.org/P12112 and previous config saved to /var/cache/conftool/dbconfig/20200729-085504-marostegui.json
• 08:42 jayme@cumin1001: conftool action : set/pooled=inactive; selector: name=wtp2001.codfw.wmnet
• 08:26 filippo@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:24 filippo@cumin1001: START - Cookbook sre.hosts.downtime
• 08:05 marostegui: Deploy MCR schema change on db1121 (lag will show up on s4), also remove triggers on db1124:3314
• 08:04 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1121', diff saved to https://phabricator.wikimedia.org/P12111 and previous config saved to /var/cache/conftool/dbconfig/20200729-080442-marostegui.json
• 08:03 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1141', diff saved to https://phabricator.wikimedia.org/P12110 and previous config saved to /var/cache/conftool/dbconfig/20200729-080318-marostegui.json
• 07:55 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1141', diff saved to https://phabricator.wikimedia.org/P12109 and previous config saved to /var/cache/conftool/dbconfig/20200729-075558-marostegui.json
• 07:48 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1141', diff saved to https://phabricator.wikimedia.org/P12108 and previous config saved to /var/cache/conftool/dbconfig/20200729-074828-marostegui.json
• 07:44 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1141', diff saved to https://phabricator.wikimedia.org/P12107 and previous config saved to /var/cache/conftool/dbconfig/20200729-074414-marostegui.json
• 06:50 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 06:48 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 06:26 XioNoX: standardize mr1-eqiad interfaces
• 06:22 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1112', diff saved to https://phabricator.wikimedia.org/P12106 and previous config saved to /var/cache/conftool/dbconfig/20200729-062224-marostegui.json
• 06:20 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1078', diff saved to https://phabricator.wikimedia.org/P12105 and previous config saved to /var/cache/conftool/dbconfig/20200729-062009-marostegui.json
• 06:16 XioNoX: standardize mr1-codfw interfaces
• 06:14 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1078', diff saved to https://phabricator.wikimedia.org/P12104 and previous config saved to /var/cache/conftool/dbconfig/20200729-061450-marostegui.json
• 06:05 XioNoX: standardize mr1-ulsfo interfaces
• 06:01 legoktm: ssh doc1001.eqiad.wmnet sudo -u doc-uploader git -C /srv/docroot pull
• 05:52 XioNoX: standardize mr1-eqsin interfaces
• 05:45 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 05:43 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 05:03 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1078', diff saved to https://phabricator.wikimedia.org/P12103 and previous config saved to /var/cache/conftool/dbconfig/20200729-050346-marostegui.json
• 05:02 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1141', diff saved to https://phabricator.wikimedia.org/P12102 and previous config saved to /var/cache/conftool/dbconfig/20200729-050247-marostegui.json
• 05:02 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1142', diff saved to https://phabricator.wikimedia.org/P12101 and previous config saved to /var/cache/conftool/dbconfig/20200729-050204-marostegui.json
• 04:58 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1142', diff saved to https://phabricator.wikimedia.org/P12100 and previous config saved to /var/cache/conftool/dbconfig/20200729-045859-marostegui.json
• 02:19 tstarling@deploy1001: Synchronized wmf-config/CommonSettings.php: re-enable lilypond in safe mode (duration: 01m 09s)
• 01:47 tstarling@deploy1001: Synchronized php-1.36.0-wmf.2/extensions/Score/includes/Score.php: work around firejail bug (duration: 01m 07s)
• 01:45 tstarling@deploy1001: Synchronized php-1.36.0-wmf.1/extensions/Score/includes/Score.php: work around firejail bug (duration: 01m 08s)
• 01:19 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=wtp1048.eqiad.wmnet
• 01:15 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=wtp1047.eqiad.wmnet
• 00:53 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=wtp1046.eqiad.wmnet
• 00:48 ryankemper: sudo -E cumin -b 10 'A:wdqs-all' 'sudo run-puppet-agent'
• 00:18 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 00:14 dzahn@cumin1001: START - Cookbook sre.hosts.downtime

2020-07-28

• 23:38 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 23:37 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: cirrus: reduce mlr window size on enwiki (duration: 01m 05s)
• 23:36 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 23:34 ebernhardson@deploy1001: Synchronized wmf-config/CirrusSearch-common.php: cirrus: reduce mlr window size on enwiki (duration: 01m 06s)
• 23:31 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 23:29 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 23:04 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Remove unused setting $wgGEHomepageSuggestedEditsNewAccountInitiatedPercentage (no-op) (duration: 01m 06s)
• 22:55 dzahn@cumin1001: conftool action : set/pooled=no; selector: name=wtp1046.eqiad.wmnet
• 22:19 jayme@cumin1001: conftool action : set/pooled=yes; selector: name=wtp1044.eqiad.wmnet
• 21:27 dancy@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'blubberoid' for release 'production' .
• 21:24 dancy@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'blubberoid' for release 'production' .
• 21:17 dancy@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' .
• 20:38 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 20:36 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 20:02 eileen: process-control config revision is b6ece03513
• 19:50 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 19:48 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 19:48 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 19:48 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 19:25 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1)
• 19:25 andrew@cumin1001: START - Cookbook sre.hosts.decommission
• 19:24 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1)
• 19:24 andrew@cumin1001: START - Cookbook sre.hosts.decommission
• 19:24 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1)
• 19:23 andrew@cumin1001: START - Cookbook sre.hosts.decommission
• 19:19 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1142', diff saved to https://phabricator.wikimedia.org/P12097 and previous config saved to /var/cache/conftool/dbconfig/20200728-191926-marostegui.json
• 19:12 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1147', diff saved to https://phabricator.wikimedia.org/P12096 and previous config saved to /var/cache/conftool/dbconfig/20200728-191237-marostegui.json
• 19:11 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@69bbbbb]: airflow: drop_old_data_daily: top_queries table renamed to fulltext_head_queries (duration: 00m 53s)
• 19:11 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@69bbbbb]: airflow: drop_old_data_daily: top_queries table renamed to fulltext_head_queries
• 19:09 jhuneidi@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'blubberoid' for release 'production' .
• 19:09 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1147', diff saved to https://phabricator.wikimedia.org/P12095 and previous config saved to /var/cache/conftool/dbconfig/20200728-190933-marostegui.json
• 19:06 jhuneidi@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'blubberoid' for release 'production' .
• 19:05 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1147', diff saved to https://phabricator.wikimedia.org/P12094 and previous config saved to /var/cache/conftool/dbconfig/20200728-190517-marostegui.json
• 19:03 jhuneidi@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' .
• 19:01 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1147', diff saved to https://phabricator.wikimedia.org/P12093 and previous config saved to /var/cache/conftool/dbconfig/20200728-190137-marostegui.json
• 18:35 cdanis: ✔️ cdanis@lvs1015.eqiad.wmnet ~ 🕝☕ sudo ipvsadm -D -t 10.2.2.51:9283
• 18:29 cdanis: ❌cdanis@lvs1016.eqiad.wmnet ~ 🕝☕ sudo ipvsadm -D -t 10.2.2.51:9283
• 18:29 catrope@deploy1001: Synchronized php-1.36.0-wmf.2/extensions/GrowthExperiments/extension.json: Fix reference to MentorChangeLogFormatter (T259041) (duration: 01m 05s)
• 18:20 catrope@deploy1001: Synchronized wmf-config/CommonSettings.php: No-op sync for wmgUseWikimediaApiPortal and wmgUseWikimediaApiPortalOAuth (2 of 2) (duration: 00m 58s)
• 18:17 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: No-op sync for wmgUseWikimediaApiPortal and wmgUseWikimediaApiPortalOAuth (1 of 2) (duration: 01m 05s)
• 18:16 cdanis: primary pybal restart ✔️ cdanis@lvs1015.eqiad.wmnet ~ 🕑☕ sudo systemctl restart pybal.service
• 18:14 cdanis: backup pybal restart: ✔️ cdanis@lvs1016.eqiad.wmnet ~ 🕑☕ sudo systemctl restart pybal.service
• 18:10 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 18:05 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
• 18:05 catrope@deploy1001: Synchronized php-1.36.0-wmf.2/includes/libs/filebackend/SwiftFileBackend.php: Fix index error in SwiftFileBackend (T259023) (duration: 01m 07s)
• 17:46 volker-e@deploy1001: Finished deploy [design/style-guide@e3fda83]: Deploy design/style-guide: (duration: 00m 05s)
• 17:46 volker-e@deploy1001: Started deploy [design/style-guide@e3fda83]: Deploy design/style-guide:
• 17:41 volans: run apt-get clean on wtp[1046,1048].eqiad.wmnet and wtp2001.codfw.wmnet to free ~`2GB as they were 100% - T258775
• 17:33 XioNoX: standardize mr1-esams interfaces
• 17:30 brennen@deploy1001: sync aborted: (no justification provided) (duration: 28m 53s)
• 17:03 brennen: prior scap sync for https://gerrit.wikimedia.org/r/c/mediawiki/core/+/616842 (T259023)
• 17:02 brennen@deploy1001: Started scap: (no justification provided)
• 16:51 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@0982d4e]: convert_to_esbulk: repair variable ref before assign (duration: 04m 33s)
• 16:49 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 16:47 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 16:47 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@0982d4e]: convert_to_esbulk: repair variable ref before assign
• 16:45 XioNoX: remove mr1-codfw source NAT (not used)
• 16:43 jayme@cumin1001: conftool action : set/pooled=yes; selector: name=wtp1045.eqiad.wmnet
• 16:39 ppchelko@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' .
• 16:36 ppchelko@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' .
• 16:34 ppchelko@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' .
• 16:33 jayme@cumin1001: conftool action : set/pooled=yes; selector: name=wtp1035.eqiad.wmnet
• 16:32 jayme@cumin1001: conftool action : set/pooled=yes; selector: name=wtp1034.eqiad.wmnet
• 16:31 XioNoX: mr1-eqiad# delete security nat source rule-set mgmt-to-untrust (unused, no matching ACL)
• 16:25 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 16:23 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 16:21 hnowlan: imported envoyproxy 1.15.0-1 deb into component/envoy-future for buster-wikimedia
• 16:11 jayme@cumin1001: conftool action : set/pooled=yes; selector: name=wtp1042.eqiad.wmnet
• 16:09 jayme@cumin1001: conftool action : set/pooled=yes; selector: name=wtp1043.eqiad.wmnet
• 15:54 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:52 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:51 jayme@cumin1001: START - Cookbook sre.hosts.downtime
• 15:50 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:48 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 15:48 jayme@cumin1001: START - Cookbook sre.hosts.downtime
• 15:45 jayme@cumin1001: conftool action : set/pooled=no; selector: name=wtp1035.*
• 15:44 jayme@cumin1001: conftool action : set/pooled=no; selector: name=wtp1034.*
• 15:35 ayounsi@deploy1001: Finished deploy [homer/deploy@5e999c8]: once more (duration: 03m 06s)
• 15:32 ayounsi@deploy1001: Started deploy [homer/deploy@5e999c8]: once more
• 15:32 ayounsi@deploy1001: Finished deploy [homer/deploy@5e999c8]: CR613642 (duration: 03m 38s)
• 15:31 jayme@cumin1001: conftool action : set/pooled=inactive; selector: name=wtp1045.eqiad.wmnet
• 15:30 jayme@cumin1001: conftool action : set/pooled=yes; selector: name=wtp1041.eqiad.wmnet
• 15:30 jayme@cumin1001: conftool action : set/pooled=inactive; selector: name=wtp1044.eqiad.wmnet
• 15:29 jayme@cumin1001: conftool action : set/pooled=yes; selector: name=wtp1039.eqiad.wmnet
• 15:28 ayounsi@deploy1001: Started deploy [homer/deploy@5e999c8]: CR613642
• 15:19 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:17 jayme@cumin1001: START - Cookbook sre.hosts.downtime
• 15:16 mholloway-shell@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'proton' for release 'production' .
• 15:15 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:14 mholloway-shell@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'proton' for release 'production' .
• 15:13 jayme@cumin1001: START - Cookbook sre.hosts.downtime
• 15:11 mholloway-shell@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'proton' for release 'production' .
• 15:08 ayounsi@deploy1001: Finished deploy [homer/deploy@fcf4332]: CR613642 (duration: 02m 14s)
• 15:06 ayounsi@deploy1001: Started deploy [homer/deploy@fcf4332]: CR613642
• 15:01 ayounsi@deploy1001: Finished deploy [homer/deploy@fcf4332]: CR613642 (duration: 00m 11s)
• 15:01 ayounsi@deploy1001: Started deploy [homer/deploy@fcf4332]: CR613642
• 14:58 jayme@cumin1001: conftool action : set/pooled=inactive; selector: name=wtp1043.eqiad.wmnet
• 14:58 jayme@cumin1001: conftool action : set/pooled=yes; selector: name=wtp1040.eqiad.wmnet
• 14:57 mholloway-shell@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'proton' for release 'production' .
• 14:55 jayme@cumin1001: conftool action : set/pooled=inactive; selector: name=wtp1042.eqiad.wmnet
• 14:54 jayme@cumin1001: conftool action : set/pooled=yes; selector: name=wtp1038.eqiad.wmnet
• 14:52 mholloway-shell@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'proton' for release 'production' .
• 14:48 mholloway-shell@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'proton' for release 'production' .
• 14:23 herron: bounced centrallog rsyslog services in codfw/eqiad
• 14:19 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:15 jayme@cumin1001: START - Cookbook sre.hosts.downtime
• 14:03 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1147', diff saved to https://phabricator.wikimedia.org/P12087 and previous config saved to /var/cache/conftool/dbconfig/20200728-140313-marostegui.json
• 14:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1148', diff saved to https://phabricator.wikimedia.org/P12086 and previous config saved to /var/cache/conftool/dbconfig/20200728-140249-marostegui.json
• 14:02 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:02 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1148', diff saved to https://phabricator.wikimedia.org/P12085 and previous config saved to /var/cache/conftool/dbconfig/20200728-140220-marostegui.json
• 14:02 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1148', diff saved to https://phabricator.wikimedia.org/P12084 and previous config saved to /var/cache/conftool/dbconfig/20200728-140207-marostegui.json
• 14:00 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:59 jayme@cumin1001: START - Cookbook sre.hosts.downtime
• 13:58 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:58 moritzm: installing perl security updates
• 13:56 jayme@cumin1001: START - Cookbook sre.hosts.downtime
• 13:56 jayme@cumin1001: START - Cookbook sre.hosts.downtime
• 13:55 jayme@cumin1001: conftool action : set/pooled=inactive; selector: name=wtp1041.eqiad.wmnet
• 13:55 jayme@cumin1001: conftool action : set/pooled=yes; selector: name=wtp1037.eqiad.wmnet
• 13:50 godog: remove stale ipvs thanos-query service on port 80
• 13:39 jayme@cumin1001: conftool action : set/pooled=inactive; selector: name=wtp1040.eqiad.wmnet
• 13:38 jayme@cumin1001: conftool action : set/pooled=yes; selector: name=wtp1036.eqiad.wmnet
• 13:38 jayme@cumin1001: conftool action : set/pooled=inactive; selector: name=wtp1039.eqiad.wmnet
• 13:37 jayme@cumin1001: conftool action : set/pooled=yes; selector: name=wtp1035.eqiad.wmnet
• 13:37 jayme@cumin1001: conftool action : set/pooled=inactive; selector: name=wtp1038.eqiad.wmnet
• 13:36 jayme@cumin1001: conftool action : set/pooled=yes; selector: name=wtp1034.eqiad.wmnet
• 13:29 godog: roll-restart pybal on eqiad lvs low-traffic to change port for thanos-query
• 13:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1075', diff saved to https://phabricator.wikimedia.org/P12083 and previous config saved to /var/cache/conftool/dbconfig/20200728-132520-marostegui.json
• 13:20 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1075 with less weight', diff saved to https://phabricator.wikimedia.org/P12082 and previous config saved to /var/cache/conftool/dbconfig/20200728-132023-marostegui.json
• 13:09 godog: roll-restart pybal on lvs low-traffic to apply thanos-query changes
• 13:04 XioNoX: standardize cr3-esams interfaces
• 13:03 liw@deploy1001: rebuilt and synchronized wikiversions files: group0 wikis to 1.36.0-wmf.2
• 12:41 XioNoX: standardize cr2-esams interfaces
• 12:38 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 12:36 jayme@cumin1001: START - Cookbook sre.hosts.downtime
• 12:33 jayme@cumin1001: START - Cookbook sre.hosts.downtime
• 12:32 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1075', diff saved to https://phabricator.wikimedia.org/P12081 and previous config saved to /var/cache/conftool/dbconfig/20200728-123201-marostegui.json
• 12:28 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 12:26 jayme@cumin1001: START - Cookbook sre.hosts.downtime
• 12:26 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 12:24 jayme@cumin1001: START - Cookbook sre.hosts.downtime
• 12:17 jayme@cumin1001: conftool action : set/pooled=inactive; selector: name=wtp1037.eqiad.wmnet
• 12:14 jayme@cumin1001: conftool action : set/pooled=inactive; selector: name=wtp1036.eqiad.wmnet
• 12:08 jayme@cumin1001: conftool action : set/pooled=inactive; selector: name=wtp1035.eqiad.wmnet
• 12:07 jayme@cumin1001: conftool action : set/pooled=yes; selector: name=wtp1032.eqiad.wmnet
• 12:07 jayme@cumin1001: conftool action : set/pooled=yes; selector: name=wtp1033.eqiad.wmnet
• 12:05 jayme@cumin1001: conftool action : set/pooled=yes; selector: name=wtp1031.eqiad.wmnet
• 12:04 jayme@cumin1001: conftool action : set/pooled=inactive; selector: name=wtp1034.eqiad.wmnet
• 12:04 tstarling@deploy1001: Synchronized wmf-config/CommonSettings.php: disabling lilypond rendering in Score again due to error running gs (duration: 01m 05s)
• 11:56 tstarling@deploy1001: Synchronized wmf-config/CommonSettings.php: re-enabling Score in safe mode (duration: 01m 04s)
• 11:50 Urbanecm: EU B&C window done
• 11:49 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 1a56726: Add Turkish powered by MW and Wikimedia project icons (T257732) (duration: 00m 59s)
• 11:46 urbanecm@deploy1001: Synchronized static/images/footer/: 1a56726: Add Turkish powered by MW and Wikimedia project icons (T257732) (duration: 01m 01s)
• 11:43 urbanecm@deploy1001: Synchronized static/images: df9b9ac: Move footer logos to /static/images/footer (T257732) (duration: 01m 02s)
• 11:38 marostegui: Deploy schema change on s3 codfw, this will generate lag on codfw T256682
• 11:38 ema: A:cp-text varnish ban pt.wikiversity.org T256750
• 11:37 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: df9b9ac: Move footer logos to /static/images/footer (T257732) (duration: 00m 58s)
• 11:36 ema: A:cp-text varnish ban fr.wiktionary.org T256750
• 11:35 urbanecm@deploy1001: Synchronized static/images/footer: df9b9ac: Move footer logos to /static/images/footer (T257732) (duration: 01m 05s)
• 11:34 ema: A:cp-text varnish ban eu.wikipedia.org T256750
• 11:32 ema: A:cp-text varnish ban he.wikipedia.org T256750
• 11:30 marostegui: Deploy MCR change on db1143, db1148, db1146:3314
• 11:30 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1148 for MCR schema change', diff saved to https://phabricator.wikimedia.org/P12079 and previous config saved to /var/cache/conftool/dbconfig/20200728-113009-marostegui.json
• 11:30 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 04c7ef9: Undeploy graphoid for phase 2 wikis (T258463) (duration: 01m 00s)
• 11:28 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1143', diff saved to https://phabricator.wikimedia.org/P12078 and previous config saved to /var/cache/conftool/dbconfig/20200728-112850-marostegui.json
• 11:25 ema: A:cp-text varnish ban fa.wikipedia.org T256750
• 11:21 dcausse@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [cirrus] use more neutral config var names (duration: 01m 06s)
• 11:20 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1143', diff saved to https://phabricator.wikimedia.org/P12077 and previous config saved to /var/cache/conftool/dbconfig/20200728-112046-marostegui.json
• 11:15 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1143', diff saved to https://phabricator.wikimedia.org/P12076 and previous config saved to /var/cache/conftool/dbconfig/20200728-111522-marostegui.json
• 11:12 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1143', diff saved to https://phabricator.wikimedia.org/P12075 and previous config saved to /var/cache/conftool/dbconfig/20200728-111226-marostegui.json
• 11:11 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 11:10 jdrewniak@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Config: gerrit:614890 desktop improvements by default for testing group (round 2) (T254227) (duration: 01m 06s)
• 11:09 jayme@cumin1001: START - Cookbook sre.hosts.downtime
• 11:09 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 11:07 jayme@cumin1001: START - Cookbook sre.hosts.downtime
• 10:56 hashar@deploy1001: Finished deploy [integration/docroot@ba85bdf]: Catch up with HEAD and support DOCUMENT_ROOT being a symbolic link for T149924 (duration: 00m 06s)
• 10:56 hashar@deploy1001: Started deploy [integration/docroot@ba85bdf]: Catch up with HEAD and support DOCUMENT_ROOT being a symbolic link for T149924
• 10:55 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 10:53 jayme@cumin1001: START - Cookbook sre.hosts.downtime
• 10:50 jayme@cumin1001: conftool action : set/pooled=inactive; selector: name=wtp1033.eqiad.wmnet
• 10:48 jayme@cumin1001: conftool action : set/pooled=yes; selector: name=wtp1030.eqiad.wmnet
• 10:48 jayme@cumin1001: conftool action : set/pooled=yes; selector: name=wtp1029.eqiad.wmnet
• 10:47 jayme@cumin1001: conftool action : set/pooled=inactive; selector: name=wtp1032.eqiad.wmnet
• 10:47 jayme@cumin1001: conftool action : set/pooled=yes; selector: name=wtp1028.eqiad.wmnet
• 10:33 jayme@cumin1001: conftool action : set/pooled=inactive; selector: name=wtp1031.eqiad.wmnet
• 10:32 jayme@cumin1001: conftool action : set/pooled=yes; selector: name=wtp1027.eqiad.wmnet
• 10:23 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1082', diff saved to https://phabricator.wikimedia.org/P12074 and previous config saved to /var/cache/conftool/dbconfig/20200728-102342-marostegui.json
• 10:04 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1082', diff saved to https://phabricator.wikimedia.org/P12072 and previous config saved to /var/cache/conftool/dbconfig/20200728-100412-marostegui.json
• 09:58 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:57 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 09:55 XioNoX: standardize cr2-esams interfaces
• 09:52 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:50 jayme@cumin1001: START - Cookbook sre.hosts.downtime
• 09:49 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:47 jayme@cumin1001: START - Cookbook sre.hosts.downtime
• 09:45 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:43 jayme@cumin1001: START - Cookbook sre.hosts.downtime
• 09:40 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:38 jayme@cumin1001: START - Cookbook sre.hosts.downtime
• 09:35 moritzm: imported libmysqlclient18 to component/cloudera T258768
• 09:31 jayme@cumin1001: conftool action : set/pooled=inactive; selector: name=wtp1030.eqiad.wmnet
• 09:28 jayme@cumin1001: conftool action : set/pooled=inactive; selector: name=wtp1029.eqiad.wmnet
• 09:26 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1082', diff saved to https://phabricator.wikimedia.org/P12070 and previous config saved to /var/cache/conftool/dbconfig/20200728-092606-marostegui.json
• 09:24 jayme@cumin1001: conftool action : set/pooled=inactive; selector: name=wtp1028.eqiad.wmnet
• 09:19 XioNoX: standardize cr3-eqsin interfaces
• 09:18 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1082', diff saved to https://phabricator.wikimedia.org/P12069 and previous config saved to /var/cache/conftool/dbconfig/20200728-091849-marostegui.json
• 09:18 jayme@cumin1001: conftool action : set/pooled=inactive; selector: name=wtp1027.eqiad.wmnet
• 09:10 jayme@cumin1001: conftool action : set/pooled=yes; selector: name=wtp1026.eqiad.wmnet
• 09:07 ema: cp3050: restart varnishmtail.service, stuck on "Condition(c->offset <= c->vtx->len) not true."
• 08:39 XioNoX: standardize cr2-eqsin interfaces
• 08:38 godog: temporary downgrade prometheus-snmp-exporter on netmon2001
• 08:33 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1143 for MCR schema change', diff saved to https://phabricator.wikimedia.org/P12067 and previous config saved to /var/cache/conftool/dbconfig/20200728-083336-marostegui.json
• 08:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1146:3314', diff saved to https://phabricator.wikimedia.org/P12066 and previous config saved to /var/cache/conftool/dbconfig/20200728-083209-marostegui.json
• 08:20 liw@deploy1001: Finished scap: testwikis wikis to 1.36.0-wmf.2 (duration: 53m 11s)
• 08:09 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:07 jayme@cumin1001: START - Cookbook sre.hosts.downtime
• 08:06 godog: failover librenms/smokeping to netmon2001 - T247967
• 08:04 marostegui: Reduce labsdb1009 weight
• 07:56 ayounsi@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 07:48 jayme: depooled wtp1026.eqiad.wmnet for reimage
• 07:48 moritzm: switched superset to CAS
• 07:47 ayounsi@cumin1001: START - Cookbook sre.dns.netbox
• 07:46 ayounsi@cumin1001: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
• 07:43 ayounsi@cumin1001: START - Cookbook sre.dns.netbox
• 07:34 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 07:32 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 07:31 jayme@cumin1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
• 07:27 liw@deploy1001: Started scap: testwikis wikis to 1.36.0-wmf.2
• 07:03 liw: 1.36.0-wmf.2 was branched at 04e863f for T257970
• 05:19 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1146:3314 for MCR schema change', diff saved to https://phabricator.wikimedia.org/P12065 and previous config saved to /var/cache/conftool/dbconfig/20200728-051928-marostegui.json
• 05:18 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1144:3314 and restore db1146:3314 original weight', diff saved to https://phabricator.wikimedia.org/P12064 and previous config saved to /var/cache/conftool/dbconfig/20200728-051813-marostegui.json
• 02:17 eileen: process-control config revision is 6811ca294a - just delayed silverpop_daily a bit as clashing with dedupe
• 00:18 andrew@cumin1001: conftool action : set/pooled=inactive; selector: name=cloudcephmon1003.eqiad.wmnet
• 00:17 andrew@cumin1001: conftool action : set/pooled=no; selector: name=cloudcephmon1003.eqiad.wmnet

2020-07-27

• 23:49 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@ac8e5d0]: airflow: head queries report, managed variables, refinery-drop-hive-partitions support (duration: 00m 54s)
• 23:48 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@ac8e5d0]: airflow: head queries report, managed variables, refinery-drop-hive-partitions support
• 23:28 mutante: otrs1001 - ran puppet (it was alerting in icinga that puppet failed, but it was neither disabled nor failing and changed nothing when it ran)
• 21:31 sbassett@deploy1001: Synchronized wmf-config/CommonSettings.php: Deployed CentralNotice CSP conifg change for T258459 (duration: 00m 57s)
• 21:10 sbassett: Deployed mitigations for T238075
• 20:41 urbanecm@deploy1001: Synchronized php-1.36.0-wmf.1/extensions/InterwikiSorting/: c5f6c97: Use LanguageLinksHook to sort interwiki links (T257625) (duration: 00m 59s)
• 19:50 jhuneidi@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'blubberoid' for release 'production' .
• 19:44 jhuneidi@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'blubberoid' for release 'production' .
• 19:36 jhuneidi@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' .
• 19:23 jhuneidi@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'sessionstore' for release 'production' .
• 19:19 jhuneidi@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'sessionstore' for release 'production' .
• 19:11 jhuneidi@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'sessionstore' for release 'staging' .
• 19:06 jhuneidi@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'echostore' for release 'production' .
• 19:00 jhuneidi@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'echostore' for release 'production' .
• 18:57 urbanecm@deploy1001: sync-file aborted: 3833b13: Move footer logos to /static/images/footer (T257732) (duration: 00m 04s)
• 18:50 urbanecm@deploy1001: Synchronized wmf-config/CommonSettings.php: c6a9674: Move footer logos to wmg* variables (T257732) (duration: 00m 56s)
• 18:50 urbanecm@deploy1001: Synchronized wmf-config/CommonSettings.php: (no justification provided) (duration: 00m 57s)
• 18:49 urbanecm@deploy1001: sync-file aborted: (no justification provided) (duration: 00m 01s)
• 18:49 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: c6a9674: Move footer logos to wmg* variables (T257732) (duration: 00m 57s)
• 18:29 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable desktop web UI click tracking instrumentation on frwiki, hewiki, fawiki (T258058) (duration: 00m 56s)
• 18:13 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Remove WPBSkinBlacklist (T254675) (duration: 00m 57s)
• 17:42 liw@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.1
• 17:30 liw: promoting train to group2
• 17:14 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99)
• 17:14 andrew@cumin1001: START - Cookbook sre.hosts.decommission
• 17:14 dpifke@deploy1001: Finished deploy [performance/arc-lamp@f14888b]: Deploying arclamp-compress-logs (T235456) (duration: 00m 05s)
• 17:14 dpifke@deploy1001: Started deploy [performance/arc-lamp@f14888b]: Deploying arclamp-compress-logs (T235456)
• 16:59 andrew@cumin1001: conftool action : set/pooled=yes; selector: name=cloudcephmon1002.eqiad.wmnet
• 16:58 andrew@cumin1001: conftool action : set/pooled=no; selector: name=cloudcephmon1002.eqiad.wmnet
• 16:57 andrew@cumin1001: conftool action : set/pooled=inactive; selector: name=cloudcephmon1002.eqiad.wmnet
• 16:50 andrew@cumin1001: conftool action : set/pooled=inactive; selector: name=cloudcephosd1003.eqiad.wmnet
• 16:50 andrew@cumin1001: conftool action : set/pooled=inactive; selector: name=cloudcephosd1002.eqiad.wmnet
• 16:50 andrew@cumin1001: conftool action : set/pooled=inactive; selector: name=cloudcephosd1001.eqiad.wmnet
• 16:50 andrew@cumin1001: conftool action : set/pooled=yes; selector: name=cloudcephmon1003.eqiad.wmnet
• 16:50 andrew@cumin1001: conftool action : set/pooled=yes; selector: name=cloudcephmon1002.eqiad.wmnet
• 16:49 andrew@cumin1001: conftool action : set/pooled=yes; selector: name=cloudcephmon1001.eqiad.wmnet
• 16:48 andrew@cumin1001: conftool action : set/pooled=no; selector: name=cloudcephosd1003.wikimedia.org
• 16:48 andrew@cumin1001: conftool action : set/pooled=no; selector: name=cloudcephosd1002.wikimedia.org
• 16:48 andrew@cumin1001: conftool action : set/pooled=no; selector: name=cloudcephosd1001.wikimedia.org
• 16:48 andrew@cumin1001: conftool action : set/pooled=yes; selector: name=cloudcephosd1001.eqiad.wmnet
• 16:48 andrew@cumin1001: conftool action : set/pooled=yes; selector: name=cloudcephosd1002.eqiad.wmnet
• 16:47 andrew@cumin1001: conftool action : set/pooled=yes; selector: name=cloudcephosd1003.eqiad.wmnet
• 16:44 andrew@cumin1001: conftool action : set/pooled=yes; selector: name=cumin1001.eqiad.wmnet
• 16:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db2087:3316, db2087:3317 after on-site maintenance T258587', diff saved to https://phabricator.wikimedia.org/P12063 and previous config saved to /var/cache/conftool/dbconfig/20200727-163311-marostegui.json
• 16:05 marostegui: Will show up on labsdb hosts for s5
• 16:04 marostegui: Stop MySQL on db1082 for onsite maintenance - T258910
• 15:03 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:03 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 14:57 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:55 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 14:50 marostegui@cumin1001: dbctl commit (dc=all): 'Reduce db1146:3314 weight while db1144:3314 is depooled', diff saved to https://phabricator.wikimedia.org/P12060 and previous config saved to /var/cache/conftool/dbconfig/20200727-145010-marostegui.json
• 14:48 marostegui: Deploy MCR change on db1144:3314
• 14:48 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1144:3314 for MCR schema change', diff saved to https://phabricator.wikimedia.org/P12059 and previous config saved to /var/cache/conftool/dbconfig/20200727-144807-marostegui.json
• 14:40 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1149', diff saved to https://phabricator.wikimedia.org/P12058 and previous config saved to /var/cache/conftool/dbconfig/20200727-144034-marostegui.json
• 14:21 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:19 XioNoX: standardize cr1-codfw interfaces
• 14:19 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 14:07 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:04 jayme@cumin1001: START - Cookbook sre.hosts.downtime
• 13:57 moritzm: upgrading idp2001 to CAS 6.1.7.1
• 13:19 XioNoX: standardize some cr2-esams interfaces
• 13:11 marostegui@cumin1001: dbctl commit (dc=all): 'More weight to db1089 in main traffic', diff saved to https://phabricator.wikimedia.org/P12057 and previous config saved to /var/cache/conftool/dbconfig/20200727-131123-marostegui.json
• 13:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1105:3311 with normal weight and pool db1089 into vslow', diff saved to https://phabricator.wikimedia.org/P12056 and previous config saved to /var/cache/conftool/dbconfig/20200727-130954-marostegui.json
• 13:07 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1105:3311', diff saved to https://phabricator.wikimedia.org/P12055 and previous config saved to /var/cache/conftool/dbconfig/20200727-130713-marostegui.json
• 13:03 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:01 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 12:58 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1105:3311 with less weight', diff saved to https://phabricator.wikimedia.org/P12054 and previous config saved to /var/cache/conftool/dbconfig/20200727-125824-marostegui.json
• 12:53 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1105:3311', diff saved to https://phabricator.wikimedia.org/P12053 and previous config saved to /var/cache/conftool/dbconfig/20200727-125351-marostegui.json
• 12:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1105:3311 with less weight', diff saved to https://phabricator.wikimedia.org/P12052 and previous config saved to /var/cache/conftool/dbconfig/20200727-125207-marostegui.json
• 12:50 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1105:3311', diff saved to https://phabricator.wikimedia.org/P12051 and previous config saved to /var/cache/conftool/dbconfig/20200727-125045-marostegui.json
• 12:41 marostegui: Compress innodb on db1106, this will generate lag on enwiki on labsdb hosts (wiki replicas) T254462
• 12:38 moritzm: disable puppet on idp1001/2001
• 12:38 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1106 and pool db1105:3311 as vslow T254462', diff saved to https://phabricator.wikimedia.org/P12050 and previous config saved to /var/cache/conftool/dbconfig/20200727-123833-marostegui.json
• 12:37 akosiaris@cumin1001: conftool action : set/pooled=no; selector: dc=eqiad,service=mobileapps,name=scb1001.eqiad.wmnet
• 12:37 akosiaris@cumin1001: conftool action : set/weight=0; selector: dc=eqiad,service=mobileapps,name=scb1001.eqiad.wmnet
• 12:37 akosiaris@cumin1001: conftool action : set/weight=10; selector: dc=eqiad,service=mobileapps,name=scb1001.eqiad.wmnet
• 12:37 akosiaris@cumin1001: conftool action : set/pooled=yes; selector: dc=eqiad,service=mobileapps,name=scb1001.eqiad.wmnet
• 12:36 akosiaris@cumin1001: conftool action : set/weight=0; selector: dc=eqiad,service=mobileapps,name=scb1001.eqiad.wmnet
• 12:31 XioNoX: standardize cr2-codfw interfaces
• 12:28 volans@deploy1001: Finished deploy [debmonitor/deploy@25dbd20]: Release v0.2.7 (duration: 00m 27s)
• 12:28 volans@deploy1001: Started deploy [debmonitor/deploy@25dbd20]: Release v0.2.7
• 12:25 jbond42: upload new cas package to buster-wikimedia
• 12:25 jbond42: upload new cas package
• 12:23 ema: A:cp rolling varnish-frontend restart to actually discard old VCL still pointing at varnishcheck/check T255015 T236754
• 12:21 moritzm: installing ruby-json security updates
• 12:16 moritzm: installing batik security updates
• 11:59 marostegui: Deploy MCR schema change on db1149
• 11:58 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1149 for MCR schema change', diff saved to https://phabricator.wikimedia.org/P12049 and previous config saved to /var/cache/conftool/dbconfig/20200727-115818-marostegui.json
• 11:57 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1138', diff saved to https://phabricator.wikimedia.org/P12048 and previous config saved to /var/cache/conftool/dbconfig/20200727-115739-marostegui.json
• 11:52 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1138', diff saved to https://phabricator.wikimedia.org/P12047 and previous config saved to /var/cache/conftool/dbconfig/20200727-115258-marostegui.json
• 11:28 moritzm: installing an-tool1009 T258768
• 10:54 ema: upload atskafka 0.10 to buster-wikimedia, upgrade cp3050 T254317
• 10:46 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (616463) (duration: 01m 05s)
• 10:45 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (616463) (duration: 01m 10s)
• 10:33 XioNoX: make cr*-ulsfo interfaces netbox compliant
• 08:39 XioNoX: push "Add 185.71.138.0/24 to wikimedia4" to all routers
• 07:00 marostegui: Deploy schema change on s5 codfw T256682
• 06:44 elukey: truncate big log file on an-launcher1002 that is filling up the /srv partition
• 06:36 elukey: apt-get clean on netbox1001 to free some space
• 05:11 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1138 for MCR schema change', diff saved to https://phabricator.wikimedia.org/P12043 and previous config saved to /var/cache/conftool/dbconfig/20200727-051156-marostegui.json
• 05:01 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2087:3316, db2087:3317 for on-site maintenance T258587', diff saved to https://phabricator.wikimedia.org/P12042 and previous config saved to /var/cache/conftool/dbconfig/20200727-050058-marostegui.json
• 04:58 marostegui: Stop MySQL on db2087 for on-site maintenance T258587

2020-07-25

• 12:41 marostegui@cumin1001: dbctl commit (dc=all): 'Pool db1096:3315 into s5 api afte db1082 crashed T258336', diff saved to https://phabricator.wikimedia.org/P12041 and previous config saved to /var/cache/conftool/dbconfig/20200725-124104-marostegui.json
• 09:16 oblivian@cumin1001: dbctl commit (dc=all): 'Depool db1082 T258336', diff saved to https://phabricator.wikimedia.org/P12040 and previous config saved to /var/cache/conftool/dbconfig/20200725-091616-oblivian.json
• 01:52 mutante: ganeti - also removing (unmounted) disk 2 (100G) from webperf1002. T257931
• 00:46 mutante: ganeti - removing disk 3 (20G) from webperf1002. the disks are 0-indexed, so the ones actually mounted are 0 (50G) and 1 (300G) (T257931)
• 00:42 dpifke: Manually compressing some more data on webperf1002, using arclamp-compress-logs from https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/615904.

2020-07-24

• 23:00 ryankemper@cumin1001: END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99)
• 20:08 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 20:06 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 19:57 dpifke: Manually gzipping some older ArcLamp data on webperf1002, to free up space and verify new compression support.
• 19:55 dpifke@deploy1001: Finished deploy [performance/arc-lamp@772b4a3]: Deploy CLs 611465 and 613740 to add compression support to ArcLamp (duration: 00m 05s)
• 19:55 dpifke@deploy1001: Started deploy [performance/arc-lamp@772b4a3]: Deploy CLs 611465 and 613740 to add compression support to ArcLamp
• 16:55 Amir1: deployment done
• 16:49 ladsgroup@deploy1001: Synchronized php-1.36.0-wmf.1/extensions/Wikibase/repo/includes/RepoHooks.php: Prevent onTitleGetRestrictionTypes changing ns0 protections, Part II (duration: 01m 07s)
• 16:47 ladsgroup@deploy1001: Synchronized php-1.36.0-wmf.1/extensions/Wikibase/repo/includes/WikibaseRepo.php: Prevent onTitleGetRestrictionTypes changing ns0 protections, Part I (duration: 01m 06s)
• 15:06 reedy@deploy1001: Finished scap: Score backports (duration: 36m 50s)
• 14:30 reedy@deploy1001: Started scap: Score backports
• 13:31 XioNoX: advertise 185.71.138.0/24 from AMS
• 13:17 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'production' .
• 13:00 ladsgroup@deploy1001: Synchronized php-1.36.0-wmf.1/includes/import/ImportableOldRevisionImporter.php: Import: use master DB for loading slots. (T258666) (duration: 01m 07s)
• 12:34 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'mobileapps' for release 'production' .
• 12:04 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'mobileapps' for release 'staging' .
• 11:48 hnowlan: bootstrapped restbase-dev1004-b
• 11:13 hnowlan: started bootstrap of restbase-dev1004-a
• 10:51 hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 10:49 hnowlan@cumin1001: START - Cookbook sre.hosts.downtime
• 10:35 hnowlan: started reimage of restbase-dev1004
• 09:59 jmm@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 09:48 jmm@cumin1001: START - Cookbook sre.ganeti.makevm
• 08:40 kormat: restarting mariadb on all sanitarium hosts T258711
• 08:35 akosiaris: start nagios-nrpe-server on kubernetes2002
• 07:44 elukey: depool wtp1025 - disk full
• 06:30 tstarling@deploy1001: Started scap: for Score
• 02:36 tstarling@deploy1001: Synchronized php-1.36.0-wmf.1/extensions/Score/includes/Score.php: removing superseded local patch for hard-coding lilypond version (duration: 01m 09s)
• 01:19 ejegg: updated payments-wiki from 31a3de1130 to c365c136d2
• 01:04 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 01:02 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 01:02 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 01:02 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 01:02 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 01:02 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 01:02 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 01:02 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 01:02 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 01:02 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 01:02 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 01:02 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 00:46 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1)
• 00:46 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1)
• 00:46 andrew@cumin1001: START - Cookbook sre.hosts.decommission
• 00:46 andrew@cumin1001: START - Cookbook sre.hosts.decommission
• 00:45 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1)
• 00:45 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1)
• 00:44 andrew@cumin1001: START - Cookbook sre.hosts.decommission
• 00:44 andrew@cumin1001: START - Cookbook sre.hosts.decommission
• 00:44 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1)
• 00:44 andrew@cumin1001: START - Cookbook sre.hosts.decommission
• 00:43 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1)
• 00:42 andrew@cumin1001: START - Cookbook sre.hosts.decommission
• 00:32 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 00:30 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 00:15 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1)
• 00:14 andrew@cumin1001: START - Cookbook sre.hosts.decommission

2020-07-23

• 23:32 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 23:30 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 23:30 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 23:30 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 23:30 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 23:30 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 23:30 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 23:30 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 22:55 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 22:53 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 22:52 mutante: stashbot quadruple log test
• 22:51 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 22:51 jhuneidi@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'echostore' for release 'staging' .
• 22:51 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 22:51 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 22:51 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 22:51 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 22:51 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 22:51 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 22:50 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 22:50 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 22:50 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 22:21 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 22:18 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 21:48 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 21:45 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 21:29 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 21:27 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 21:21 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@c99c626]: airflow: centralize installation specific airflow Variables (duration: 00m 34s)
• 21:20 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@c99c626]: airflow: centralize installation specific airflow Variables
• 21:02 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 21:00 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 20:59 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 20:58 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 20:58 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 20:58 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 20:58 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 20:58 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 20:58 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 20:58 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 20:58 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 20:58 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 19:13 mholloway-shell@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'wikifeeds' for release 'production' .
• 19:11 mholloway-shell@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'wikifeeds' for release 'production' .
• 19:09 mholloway-shell@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' .
• 18:51 ryankemper: restarted blazegraph on codfw wdqs2001
• 18:44 ryankemper: Restarted blazegraph on following codfw wdqs nodes: 2007, 2003, and 2002
• 18:39 Amir1: BACC is done
• 18:29 ladsgroup@deploy1001: Synchronized wmf-config/Wikibase.php: Load WikibaseClient from extension.json file instead of php one (T257437 T256228 T88258) (duration: 01m 05s)
• 18:21 mutante: testreduce1001 - rm -rf /srv/testreduce and run puppet to re-clone testreduce to it from the scandium branch (T257906)
• 18:13 ryankemper: restarted blazegraph on 2001
• 17:59 ryankemper: sudo -E cumin -b 10 'A:wdqs-all and not A:wdqs-test and not P{wdqs1003.eqiad.wmnet} and not P{wdqs2001.codfw.wmnet}' 'sudo systemctl restart wdqs-blazegraph.service'
• 17:53 cdanis: ❌cdanis@cumin1001.eqiad.wmnet ~ 🕑☕ sudo cumin -b10 'wdqs*' "run-puppet-agent --unless-version 1a4ae81"
• 17:52 cdanis@cumin1001: conftool action : set/pooled=true; selector: dnsdisc=wdqs.*,name=codfw
• 17:35 cdanis@cumin1001: conftool action : set/pooled=false; selector: dnsdisc=wdqs.*,name=codfw
• 17:22 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-reload
• 16:57 ryankemper@cumin1001: END (ERROR) - Cookbook sre.wdqs.data-reload (exit_code=97)
• 16:56 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-reload
• 15:36 urbanecm@deploy1001: Synchronized private/PrivateSettings.php: Update T250887 mitigations (duration: 01m 05s)
• 13:49 akosiaris@cumin1001: conftool action : set/pooled=true; selector: dnsdisc=helm-charts,name=.*
• 12:29 marostegui: Decrease labsdb1009 weight a bit, as it is lagging again.
• 12:23 XioNoX: remove bogus lo0 IPs from cr3-knams
• 12:21 Urbanecm: Stagging at mwdebug1001 ended, run scap pull to clean changes
• 12:17 Urbanecm: Stagging at mwdebug1001 again
• 12:02 Urbanecm: Stagging at mwdebug1001 ended, run scap pull to clean changes
• 12:00 Urbanecm: Stagging at mwdebug1001
• 11:49 urbanecm@deploy1001: Synchronized wmf-config/CommonSettings.php: 745ff20: Log ClosedWikiProviders start with info level (T258695) (duration: 01m 05s)
• 11:48 marostegui: Deploy MCR schema change on db1145:3314
• 11:36 dcausse: European mid-day backport window done
• 11:31 dcausse@deploy1001: Synchronized php-1.36.0-wmf.1/extensions/Wikibase: T258507: Fix bug that causes wrong prefixes in RDF output (duration: 01m 11s)
• 11:18 akosiaris: depool scb in mobileapps/eqiad. T218733
• 11:17 akosiaris@cumin1001: conftool action : set/pooled=no; selector: dc=eqiad,service=mobileapps,name=scb.*
• 11:13 dcausse@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T258474: [sdoc] fix entity source base URIs (duration: 01m 07s)
• 10:27 akosiaris@cumin1001: conftool action : set/weight=1; selector: dc=eqiad,service=mobileapps,name=scb.*
• 10:27 akosiaris@cumin1001: conftool action : set/weight=1; selector: dc=eqiad,service=mobileapps,name=scb*
• 10:25 akosiaris@cumin1001: conftool action : set/pooled=no; selector: dc=eqiad,service=mobileapps,name=scb1002.*
• 10:24 akosiaris@cumin1001: conftool action : set/pooled=no; selector: dc=eqiad,service=mobileapps,name=scb1001.*
• 10:18 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'zotero' for release 'production' .
• 10:14 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'zotero' for release 'production' .
• 10:11 akosiaris: poole kubernetes in mobileapps/eqiad. T218733
• 10:11 akosiaris@cumin1001: conftool action : set/pooled=yes; selector: dc=eqiad,service=mobileapps,name=kubernetes.*
• 10:06 volans@deploy1001: Finished deploy [debmonitor/deploy@16d0c45]: Release v0.2.6 (duration: 00m 36s)
• 10:06 volans@deploy1001: Started deploy [debmonitor/deploy@16d0c45]: Release v0.2.6
• 10:05 volans@deploy1001: Finished deploy [debmonitor/deploy@44aa1ee]: Release v0.2.6 (duration: 00m 14s)
• 10:05 volans@deploy1001: Started deploy [debmonitor/deploy@44aa1ee]: Release v0.2.6
• 10:04 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'zotero' for release 'staging' .
• 09:51 akosiaris: prepare for pooling kubernetes mobileapps capacity in eqiad. T218733
• 09:51 akosiaris@cumin1001: conftool action : set/weight=10; selector: dc=eqiad,service=mobileapps,name=kubernetes.*
• 09:46 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'wikifeeds' for release 'production' .
• 09:40 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'wikifeeds' for release 'production' .
• 09:38 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' .
• 09:27 akosiaris@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'mobileapps' for release 'production' .
• 09:27 akosiaris@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' .
• 09:25 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'mobileapps' for release 'staging' .
• 09:24 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'termbox' for release 'production' .
• 09:20 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'termbox' for release 'production' .
• 09:19 akosiaris: lower replica count back to 80 for mobileapps. T218733
• 09:19 akosiaris@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'production' .
• 09:19 akosiaris@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' .
• 09:02 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'termbox' for release 'staging' .
• 08:59 marostegui: transfer --type=xtrabackup from db1117:3322 to db1107 T257540
• 08:45 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'proton' for release 'production' .
• 08:42 godog: test librenms poller from netmon2001
• 08:41 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:40 XioNoX: remove pim-rp IPs from last routers - T257573
• 08:40 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'proton' for release 'production' .
• 08:39 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 08:29 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'proton' for release 'production' .
• 08:26 marostegui@cumin1001: dbctl commit (dc=all): 'Remove db1107 from s1 T257540', diff saved to https://phabricator.wikimedia.org/P12025 and previous config saved to /var/cache/conftool/dbconfig/20200723-082647-marostegui.json
• 08:16 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1107 to move it to m2 T257540', diff saved to https://phabricator.wikimedia.org/P12024 and previous config saved to /var/cache/conftool/dbconfig/20200723-081650-marostegui.json
• 05:29 marostegui: Restore labsdb1009's original weight
• 00:24 legoktm@deploy1001: Synchronized php-1.35.0-wmf.41/includes/: T258664: Revert "Add a new type of database to the installer from extension" (2/2) (duration: 01m 08s)
• 00:22 legoktm@deploy1001: Synchronized php-1.35.0-wmf.41/includes/libs/rdbms/database/Database.php: T258664: Revert "Add a new type of database to the installer from extension" (duration: 01m 05s)
• 00:20 legoktm@deploy1001: Scap failed!: 9/9 canaries failed their endpoint checks(https://en.wikipedia.org)
• 00:16 legoktm@deploy1001: Synchronized php-1.36.0-wmf.1/includes/: T258664: Revert "Add a new type of database to the installer from extension" (duration: 01m 09s)
• 00:11 legoktm@deploy1001: scap failed: average error rate on 3/9 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/e474f13ffac6b8c3bf919c4aeafc8c9b for details)

2020-07-22

• 22:07 cdanis: remove downtime on api.svc.codfw.wmnet T258614
• 19:26 jhuneidi@deploy1001: Synchronized php: group1 wikis to 1.36.0-wmf.1 (duration: 01m 03s)
• 19:25 jhuneidi@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.36.0-wmf.1
• 19:15 urbanecm@deploy1001: Finished scap: 9529cf8: b66ec91: OOUI backport; 93755a6: i18n changes for OAuth, removal of spam messages (duration: 42m 26s)
• 19:14 ejegg: updated payments-wiki from bf91f8adff to 31a3de1130
• 19:11 mutante: mw2335 - mw2339 - scap pull
• 18:39 dzahn@cumin1001: conftool action : set/weight=15; selector: name=mw233[5-9].codfw.wmnet
• 18:38 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw233[6-9].codfw.wmnet
• 18:36 dzahn@cumin1001: conftool action : set/pooled=no; selector: name=mw233[6-9].codfw.wmnet
• 18:33 urbanecm@deploy1001: Started scap: 9529cf8: b66ec91: OOUI backport; 93755a6: i18n changes for OAuth, removal of spam messages
• 18:33 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw2335.codfw.wmnet
• 18:28 dzahn@cumin1001: conftool action : set/pooled=inactive; selector: name=mw233[5-9].codfw.wmnet
• 18:16 dzahn@cumin1001: conftool action : set/pooled=no; selector: name=mw2339.codfw.wmnet
• 17:58 dzahn@cumin1001: conftool action : set/pooled=no; selector: name=mw2338.codfw.wmnet
• 17:58 dzahn@cumin1001: conftool action : set/pooled=no; selector: name=mw2337.codfw.wmnet
• 17:58 dzahn@cumin1001: conftool action : set/pooled=no; selector: name=mw2336.codfw.wmnet
• 17:26 dzahn@cumin1001: conftool action : set/pooled=no; selector: name=mw2335.codfw.wmnet
• 15:31 moritzm: updated stretch installer image to Stretch 9.13 release T258407
• 15:27 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventstreams' for release 'production' .
• 15:27 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventstreams' for release 'canary' .
• 14:52 XioNoX: add accept-data and remove bogus v6 IP from ulsfo sandbox vlan
• 14:43 akosiaris@cumin1001: conftool action : set/pooled=no; selector: dc=codfw,service=mobileapps,name=scb.*
• 14:43 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventstreams' for release 'canary' .
• 14:43 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventstreams' for release 'production' .
• 14:35 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventstreams' for release 'canary' .
• 14:35 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventstreams' for release 'production' .
• 14:12 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-main' for release 'production' .
• 14:12 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' .
• 14:06 filippo@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:04 filippo@cumin1001: START - Cookbook sre.hosts.downtime
• 13:54 akosiaris@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' .
• 13:54 akosiaris@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'production' .
• 13:50 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' .
• 13:49 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-main' for release 'production' .
• 13:36 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' .
• 13:36 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-main' for release 'production' .
• 13:34 akosiaris@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' .
• 13:33 akosiaris@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'production' .
• 13:20 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' .
• 13:19 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' .
• 13:18 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' .
• 13:18 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' .
• 13:16 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' .
• 13:16 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' .
• 12:36 akosiaris@cumin1001: conftool action : set/weight=1; selector: dc=codfw,service=mobileapps,name=scb.*
• 12:32 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' .
• 12:28 akosiaris@cumin1001: conftool action : set/weight=10; selector: dc=codfw,service=mobileapps,name=scb.*
• 12:20 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' .
• 12:18 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' .
• 12:17 akosiaris@cumin1001: conftool action : set/weight=0; selector: dc=codfw,service=mobileapps,name=scb.*
• 12:05 ema: A:cp-text varnish ban ptwikiversity T256750
• 12:01 ema: A:cp-text varnish ban frwiktionary T256750
• 11:56 ema: A:cp-text varnish ban euwiki T256750
• 11:54 akosiaris@cumin1001: conftool action : set/pooled=yes; selector: dc=codfw,service=mobileapps,name=scb.*
• 11:54 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' .
• 11:54 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' .
• 11:52 Urbanecm: EU B&C window done
• 11:52 akosiaris@cumin1001: conftool action : set/pooled=no; selector: dc=codfw,service=mobileapps,name=scb.*
• 11:49 ema: A:cp-text force puppet run to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/615446 T256750
• 11:48 urbanecm@deploy1001: Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 02m 15s)
• 11:42 jdrewniak@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Config: Enable desktop improvements by default for testing group (round 1) (T254227) (duration: 01m 05s)
• 11:30 jdrewniak@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Config: Enable instrumentation for wikis in the desktop improvements testing group (T254228) (duration: 01m 04s)
• 11:30 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' .
• 11:30 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' .
• 11:28 jdrewniak@deploy1001: Synchronized wmf-config/config: Config: Enable instrumentation for wikis in the desktop improvements testing group (T254228) (duration: 01m 05s)
• 11:20 jdrewniak@deploy1001: Synchronized multiversion/MWConfigCacheGenerator.php: Config: Enable instrumentation for wikis in the desktop improvements testing group (T254228) (duration: 01m 05s)
• 11:18 jdrewniak@deploy1001: Synchronized dblists/desktop-improvements.dblist: Config: Enable instrumentation for wikis in the desktop improvements testing group (T254228) (duration: 01m 18s)
• 11:13 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' .
• 11:13 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' .
• 10:39 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'cxserver' for release 'production' .
• 10:24 jbond42: upload prometheus-swagger-exporter_0.3-1+deb10u1 to apt1001 buster repo
• 10:24 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'cxserver' for release 'production' .
• 10:22 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'cxserver' for release 'staging' .
• 10:19 akosiaris@deploy2001: helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' .
• 10:19 akosiaris@deploy2001: helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'production' .
• 10:12 akosiaris@cumin1001: conftool action : set/pooled=yes; selector: dc=codfw,service=mobileapps,name=scb.*
• 10:08 akosiaris@cumin1001: conftool action : set/pooled=no; selector: dc=codfw,service=mobileapps,name=scb.*
• 10:04 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'citoid' for release 'production' .
• 10:01 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'citoid' for release 'production' .
• 09:58 marostegui: Deploy MCR schema change on s4 codfw master (lag will appear on codfw) - T238966
• 09:55 akosiaris: bump memory in codfw mobileapps another 20% T218733
• 09:55 akosiaris@deploy2001: helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'production' .
• 09:55 akosiaris@deploy2001: helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' .
• 09:52 godog: centrallog1001 lvextend /srv by 130G
• 09:51 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'citoid' for release 'staging' .
• 09:46 akosiaris: codfw mobileapps kubernetes traffic back to 96% T218733 again. scb pooled again.
• 09:46 akosiaris@cumin1001: conftool action : set/pooled=yes; selector: dc=codfw,service=mobileapps,name=scb.*
• 09:43 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'blubberoid' for release 'production' .
• 09:43 akosiaris@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' .
• 09:43 akosiaris@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'mobileapps' for release 'production' .
• 09:40 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'blubberoid' for release 'production' .
• 09:40 akosiaris: increase codfw mobileapps kubernetes traffic to 100% T218733
• 09:40 akosiaris@cumin1001: conftool action : set/pooled=no; selector: dc=codfw,service=mobileapps,name=scb.*
• 09:34 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'blubberoid' for release 'staging' .
• 09:27 akosiaris@deploy2001: helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'production' .
• 09:27 akosiaris@deploy2001: helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' .
• 09:25 akosiaris: bump memory limits for mobileapps by 25% T218733
• 09:25 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'mobileapps' for release 'staging' .
• 09:10 jayme: updated docker-report to 0.0.7-1 on deneb
• 09:09 jayme: import docker-report 0.0.7-1 to buster-wikimedia
• 09:06 gehel: restarting blazegraph on all wdqs nodes - new vocabulary
• 08:48 dcausse: restarting blazegraph on wdqs1010 (testing new vocab)
• 08:46 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1126', diff saved to https://phabricator.wikimedia.org/P12017 and previous config saved to /var/cache/conftool/dbconfig/20200722-084613-marostegui.json
• 08:42 kormat@cumin1001: dbctl commit (dc=all): 'Increase es1020 to 100% pooled in es4, reduce es1021 to weight 0 T257284', diff saved to https://phabricator.wikimedia.org/P12016 and previous config saved to /var/cache/conftool/dbconfig/20200722-084159-kormat.json
• 08:39 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1126', diff saved to https://phabricator.wikimedia.org/P12015 and previous config saved to /var/cache/conftool/dbconfig/20200722-083926-marostegui.json
• 08:35 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1084 and db1107', diff saved to https://phabricator.wikimedia.org/P12014 and previous config saved to /var/cache/conftool/dbconfig/20200722-083535-marostegui.json
• 08:31 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1126', diff saved to https://phabricator.wikimedia.org/P12013 and previous config saved to /var/cache/conftool/dbconfig/20200722-083140-marostegui.json
• 08:30 kart_: Updated cxserver to 2020-07-20-200559-production (T257674)
• 08:28 kartik@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'cxserver' for release 'production' .
• 08:25 kartik@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'cxserver' for release 'production' .
• 08:25 volans@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 08:23 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1084 and db1107', diff saved to https://phabricator.wikimedia.org/P12012 and previous config saved to /var/cache/conftool/dbconfig/20200722-082309-marostegui.json
• 08:22 kartik@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'cxserver' for release 'staging' .
• 08:20 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1126', diff saved to https://phabricator.wikimedia.org/P12010 and previous config saved to /var/cache/conftool/dbconfig/20200722-082023-marostegui.json
• 08:19 volans@cumin1001: START - Cookbook sre.dns.netbox
• 08:16 akosiaris: increase codfw mobileapps kubernetes traffic to 96% T218733. Take #2. Let's see if I can reproduce the weird increases in p99 latencies and figure out their cause
• 08:15 akosiaris@cumin1001: conftool action : set/weight=1; selector: dc=codfw,service=mobileapps,name=scb.*
• 08:14 kormat@cumin1001: dbctl commit (dc=all): 'Increase es1020 to 75% pooled in es4, reduce es1021 to weight 25 T257284', diff saved to https://phabricator.wikimedia.org/P12009 and previous config saved to /var/cache/conftool/dbconfig/20200722-081457-kormat.json
• 08:13 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1084 and db1107', diff saved to https://phabricator.wikimedia.org/P12008 and previous config saved to /var/cache/conftool/dbconfig/20200722-081330-marostegui.json
• 08:12 moritzm: Turnilo switched to CAS
• 08:05 jayme: updated docker-report to 0.0.6-1 on deneb
• 07:57 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1084 and db1107', diff saved to https://phabricator.wikimedia.org/P12007 and previous config saved to /var/cache/conftool/dbconfig/20200722-075749-marostegui.json
• 07:53 kormat@cumin1001: dbctl commit (dc=all): 'Increase es1020 to 50% pooled in es4 T257284', diff saved to https://phabricator.wikimedia.org/P12006 and previous config saved to /var/cache/conftool/dbconfig/20200722-075312-kormat.json
• 07:50 marostegui@cumin1001: dbctl commit (dc=all): 'Add db1084 to s1, depooled T253217', diff saved to https://phabricator.wikimedia.org/P12005 and previous config saved to /var/cache/conftool/dbconfig/20200722-075040-marostegui.json
• 07:49 jayme: import docker-report 0.0.6-1 to buster-wikimedia
• 07:40 jynus: stop db1145 for hw maintenance T258249
• 06:47 elukey: update analytics-in4/6 filters on cr1/cr2 eqiad (ref https://gerrit.wikimedia.org/r/c/operations/homer/public/+/614702)
• 06:26 marostegui: Stop MySQL on db1107
• 06:11 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 06:09 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 06:04 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1107 to clone db1084', diff saved to https://phabricator.wikimedia.org/P12003 and previous config saved to /var/cache/conftool/dbconfig/20200722-060432-marostegui.json
• 05:16 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1126', diff saved to https://phabricator.wikimedia.org/P12002 and previous config saved to /var/cache/conftool/dbconfig/20200722-051607-marostegui.json

2020-07-21

• 23:37 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Bump cirrus MLR models to latest (duration: 01m 06s)
• 23:13 Urbanecm: Evening backport window done
• 23:12 urbanecm@deploy1001: Synchronized wmf-config/CommonSettings.php: 7a50168: Updating UploadWizard template: PD-old-70-1923->PD-old-70-expired (T258523) (duration: 01m 06s)
• 23:06 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 7acc9d9: Enable $wgWatchlistExpiry on testwiki (T257506) (duration: 01m 08s)
• 19:10 jhuneidi@deploy1001: rebuilt and synchronized wikiversions files: group0 wikis to 1.36.0-wmf.1
• 19:02 catrope@deploy1001: Synchronized php-1.36.0-wmf.1/includes/Storage/PageUpdater.php: Fix handling of null edits (T257766) (duration: 01m 06s)
• 19:01 catrope@deploy1001: Synchronized php-1.35.0-wmf.41/includes/Storage/PageUpdater.php: Fix handling of null edits (T257766) (duration: 01m 11s)
• 18:33 jhuneidi@deploy1001: Finished scap: testwikis wikis to 1.36.0-wmf.1 (duration: 41m 22s)
• 18:27 ejegg: restored new URL for TY page in payments-wiki settings
• 18:22 mforns@deploy1001: Finished deploy [analytics/refinery@0c25de1] (thin): Redeploying to unbreak unique devices per domain monthly THIN [analytics/refinery@0c25de19a3a309276654b4463cca4f574336d8fd] (duration: 00m 07s)
• 18:22 mforns@deploy1001: Started deploy [analytics/refinery@0c25de1] (thin): Redeploying to unbreak unique devices per domain monthly THIN [analytics/refinery@0c25de19a3a309276654b4463cca4f574336d8fd]
• 18:21 mforns@deploy1001: Finished deploy [analytics/refinery@0c25de1]: Redeploying to unbreak unique devices per domain monthly - third try [analytics/refinery@0c25de19a3a309276654b4463cca4f574336d8fd] (duration: 00m 12s)
• 18:21 mforns@deploy1001: Started deploy [analytics/refinery@0c25de1]: Redeploying to unbreak unique devices per domain monthly - third try [analytics/refinery@0c25de19a3a309276654b4463cca4f574336d8fd]
• 18:17 mforns@deploy1001: Finished deploy [analytics/refinery@0c25de1]: Redeploying to unbreak unique devices per domain monthly - second try [analytics/refinery@0c25de19a3a309276654b4463cca4f574336d8fd] (duration: 00m 17s)
• 18:16 mforns@deploy1001: Started deploy [analytics/refinery@0c25de1]: Redeploying to unbreak unique devices per domain monthly - second try [analytics/refinery@0c25de19a3a309276654b4463cca4f574336d8fd]
• 18:13 mforns@deploy1001: Finished deploy [analytics/refinery@0c25de1]: Redeploying to unbreak unique devices per domain monthly [analytics/refinery@0c25de19a3a309276654b4463cca4f574336d8fd] (duration: 05m 32s)
• 18:08 mforns@deploy1001: Started deploy [analytics/refinery@0c25de1]: Redeploying to unbreak unique devices per domain monthly [analytics/refinery@0c25de19a3a309276654b4463cca4f574336d8fd]
• 17:52 jhuneidi@deploy1001: Started scap: testwikis wikis to 1.36.0-wmf.1
• 17:50 volans@cumin1001: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
• 17:45 volans@cumin1001: START - Cookbook sre.dns.netbox
• 17:10 jhuneidi@deploy1001: Pruned MediaWiki: 1.35.0-wmf.39 (duration: 16m 25s)
• 16:32 ppchelko@deploy1001: Finished deploy [restbase/deploy@4f3cb41]: Add new wikis to RESTBase, take 2 (duration: 04m 54s)
• 16:27 ppchelko@deploy1001: Started deploy [restbase/deploy@4f3cb41]: Add new wikis to RESTBase, take 2
• 16:27 ppchelko@deploy1001: Finished deploy [restbase/deploy@4f3cb41]: Add new wikis to RESTBase (duration: 10m 37s)
• 16:21 longma: 1.36.0-wmf.1 was branched at 3a1faac for T257969
• 16:16 ppchelko@deploy1001: Started deploy [restbase/deploy@4f3cb41]: Add new wikis to RESTBase
• 15:16 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 15:12 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 15:10 moritzm: draining restbase1027 for eventual reboot for kernel security update
• 15:09 godog: poweroff ms-be1024 for bbu replacement - T257949
• 15:08 filippo@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:08 filippo@cumin1001: START - Cookbook sre.hosts.downtime
• 15:04 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 15:01 vgutierrez: show a synthetic warning for traffic using ECDHE-RSA-AES128-SHA - T258405
• 15:01 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 15:00 moritzm: draining restbase1026 for eventual reboot for kernel security update
• 14:57 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:57 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 14:56 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 14:52 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:51 moritzm: draining restbase1025 for eventual reboot for kernel security update
• 14:48 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 14:44 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:35 akosiaris@cumin1001: conftool action : set/weight=10; selector: dc=codfw,service=mobileapps,name=scb.*
• 14:35 akosiaris: decrease codfw mobileapps kubernetes traffic to 72% T218733. Weird latency patterns exhibited when 92% was reached. See https://grafana.wikimedia.org/d/5CmeRcnMz/mobileapps?panelId=34&fullscreen&orgId=1&from=1595338489749&to=1595342071227&var-dc=codfw%20prometheus%2Fk8s&var-service=mobileapps&var-container_name=All
• 14:35 moritzm: draining restbase1024 for eventual reboot for kernel security update
• 14:32 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1119', diff saved to https://phabricator.wikimedia.org/P11994 and previous config saved to /var/cache/conftool/dbconfig/20200721-143204-marostegui.json
• 14:26 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1119', diff saved to https://phabricator.wikimedia.org/P11993 and previous config saved to /var/cache/conftool/dbconfig/20200721-142634-marostegui.json
• 14:24 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:24 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 14:23 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 14:19 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:18 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1119', diff saved to https://phabricator.wikimedia.org/P11992 and previous config saved to /var/cache/conftool/dbconfig/20200721-141813-marostegui.json
• 14:16 moritzm: draining restbase1023 for eventual reboot for kernel security update
• 14:10 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 14:06 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:06 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:06 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 14:03 moritzm: draining restbase1022 for eventual reboot for kernel security update
• 14:01 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 13:57 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 13:55 moritzm: draining restbase1021 for eventual reboot for kernel security update
• 13:51 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 13:50 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1119', diff saved to https://phabricator.wikimedia.org/P11991 and previous config saved to /var/cache/conftool/dbconfig/20200721-135028-marostegui.json
• 13:48 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 13:46 moritzm: draining restbase1020 for eventual reboot for kernel security update
• 13:42 akosiaris@cumin1001: conftool action : set/weight=1; selector: dc=codfw,service=mobileapps,name=scb.*
• 13:41 akosiaris: increase codfw mobileapps kubernetes traffic to 96% T218733
• 13:41 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:41 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 13:15 Amir1: end of ladsgroup@mwmaint1002:~$ foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https (T258472 T258473)
• 13:13 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:11 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 13:10 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 13:06 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 13:03 moritzm: draining restbase1019 for eventual reboot for kernel security update
• 13:01 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:01 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 12:55 Amir1: start of ladsgroup@mwmaint1002:~$ foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https (T258472 T258473)
• 12:54 marostegui: Stop haproxy on dbproxy1012 - T255408
• 12:13 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1087', diff saved to https://phabricator.wikimedia.org/P11988 and previous config saved to /var/cache/conftool/dbconfig/20200721-121302-marostegui.json
• 12:05 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 12:01 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 11:53 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 11:49 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 11:45 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 11:41 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 11:25 Urbanecm: EU B&C window done
• 11:24 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 7b96c7e: Enable botpasswords at checkuserwiki and stewardwiki (T258358, T258355) (duration: 00m 57s)
• 11:11 Urbanecm: Create bot_passwords table at checkuserwiki (T258358)
• 11:10 Urbanecm: Create bot_passwords table at stewardwiki (T258355)
• 11:09 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 5d5bb37: Enable Vector opt in preference everywhere (T254228) (duration: 00m 57s)
• 11:08 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1085 T258360', diff saved to https://phabricator.wikimedia.org/P11987 and previous config saved to /var/cache/conftool/dbconfig/20200721-110854-marostegui.json
• 11:00 effie: enable puppet on P:mediawiki::mcrouter_wancache - T247956
• 10:58 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1085 T258360', diff saved to https://phabricator.wikimedia.org/P11986 and previous config saved to /var/cache/conftool/dbconfig/20200721-105852-marostegui.json
• 10:45 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1085 T258360', diff saved to https://phabricator.wikimedia.org/P11985 and previous config saved to /var/cache/conftool/dbconfig/20200721-104546-marostegui.json
• 10:34 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1085', diff saved to https://phabricator.wikimedia.org/P11984 and previous config saved to /var/cache/conftool/dbconfig/20200721-103430-marostegui.json
• 10:20 effie: disable puppet on P:mediawiki::mcrouter_wancache - T247956
• 10:13 effie: enable puppet on on wtp*
• 10:02 marostegui: Analyze revision table on db1119 T258480
• 10:02 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1119 T258480', diff saved to https://phabricator.wikimedia.org/P11983 and previous config saved to /var/cache/conftool/dbconfig/20200721-100159-marostegui.json
• 09:59 akosiaris: move all codfw mobileapps nodes (kubernetes and scb) to weight 10. Traffic level remains at 72.727272% flowing to kubernetes, the rest to scb T218733
• 09:59 akosiaris: move all codfw mobileapps nodes (kubernetes and scb) to weight 10. Traffic level remains at 72.727272% flowing to kubernetes, the rest to scb
• 09:59 effie: disable puppet on wtp* to merge 613307
• 09:58 akosiaris@cumin1001: conftool action : set/weight=10; selector: dc=codfw,service=mobileapps
• 09:58 akosiaris: increase codfw mobileapps kubernetes traffic to 72.727272% T218733
• 09:57 akosiaris@cumin1001: conftool action : set/weight=1; selector: dc=codfw,service=mobileapps,name=scb.*
• 09:44 elukey: add term 'idp' to analytics-in4/6 filters on cr1-eqiad and cr2-eqiad (ref: https://gerrit.wikimedia.org/r/c/operations/homer/public/+/615160)
• 09:21 kormat@cumin1001: dbctl commit (dc=all): 'Re-pool es1020 at 25% in es4 T257284', diff saved to https://phabricator.wikimedia.org/P11982 and previous config saved to /var/cache/conftool/dbconfig/20200721-092126-kormat.json
• 08:37 akosiaris: increase codfw mobileapps kubernetes traffic to 47% T218733
• 08:34 akosiaris@cumin1001: conftool action : set/weight=3; selector: dc=codfw,service=mobileapps,name=scb.*
• 08:28 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:26 kormat@cumin1001: START - Cookbook sre.hosts.downtime
• 08:08 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1119', diff saved to https://phabricator.wikimedia.org/P11980 and previous config saved to /var/cache/conftool/dbconfig/20200721-080842-marostegui.json
• 07:52 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1119', diff saved to https://phabricator.wikimedia.org/P11979 and previous config saved to /var/cache/conftool/dbconfig/20200721-075233-marostegui.json
• 07:49 marostegui: Deploy schema change on db1087, lag will appear on s8 (wikidata) on labsdb hosts T256685
• 07:48 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1087 T256685', diff saved to https://phabricator.wikimedia.org/P11978 and previous config saved to /var/cache/conftool/dbconfig/20200721-074843-marostegui.json
• 07:37 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1119', diff saved to https://phabricator.wikimedia.org/P11977 and previous config saved to /var/cache/conftool/dbconfig/20200721-073757-marostegui.json
• 07:29 kormat@deploy1001: Synchronized wmf-config/db-eqiad.php: Re-enable writes to es4 T257847 (duration: 00m 57s)
• 07:22 kormat@cumin1001: dbctl commit (dc=all): 'Depool es1020 from es4 T257847', diff saved to https://phabricator.wikimedia.org/P11976 and previous config saved to /var/cache/conftool/dbconfig/20200721-072251-kormat.json
• 07:21 kormat@cumin1001: dbctl commit (dc=all): 'Promote es1021 to es4 master T257847', diff saved to https://phabricator.wikimedia.org/P11975 and previous config saved to /var/cache/conftool/dbconfig/20200721-072127-kormat.json
• 07:13 kormat: killing James_F('s script) on mwmaint1002
• 07:06 _joe_: systemctl reset-failed on deneb, the usual known issue with releng image reporting
• 07:03 kormat@deploy1001: Synchronized wmf-config/db-eqiad.php: Disable writes to es4 T257847 (duration: 01m 00s)
• 06:59 kormat: Starting es4 failover from es1020 to es1021 T257847
• 06:54 kormat@cumin1001: dbctl commit (dc=all): 'Set es1021 to weight 50 T257847', diff saved to https://phabricator.wikimedia.org/P11974 and previous config saved to /var/cache/conftool/dbconfig/20200721-065457-kormat.json
• 06:54 marostegui: Pool db1119 into enwiki with MCR schema change done - T238966
• 06:54 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1119', diff saved to https://phabricator.wikimedia.org/P11973 and previous config saved to /var/cache/conftool/dbconfig/20200721-065430-marostegui.json
• 06:27 _joe_: systemctl reset-failed on lists1001, a network interface was failing since 1 month
• 06:26 _joe_: enabling notifications for lists1001
• 06:23 _joe_: systemctl reset-failed on both centrallogs
• 02:43 eileen: civicrm revision changed from 7f1e7d8e38 to cc5d17fbaf, config revision is 23460676f6
• 00:02 ryankemper: Began Elasticsearch reindex job on index `dewiki_content` across [`eqiad`, `codfw`, `cloudelastic`], on `rkemper@mwmaint1002` under tmux session `reindex`. Should complete in <24 hours

2020-07-20

• 23:49 eileen: tools revision changed from b915d8efbd to 22550f38c5
• 23:34 ejegg: updated fundraising CiviCRM from 8b09c87ce2 to 7f1e7d8e38
• 23:12 urbanecm@deploy1001: Synchronized php-1.35.0-wmf.41/extensions/ProofreadPage/ProofreadPage.namespaces.php: 03ed74f: Add ProofreadPage namespace translation for lij (T257672) (duration: 00m 57s)
• 23:06 Urbanecm: run mwscript namespaceDupes.php --wiki=lijwikisource -- fix (T257672)
• 23:05 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 2147774: Add English aliases for WS-specific namespaces to lijwikisource (T257672) (duration: 00m 57s)
• 22:59 ryankemper@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 613669: cirrussearch: Allow 2 dewiki->content shards/node | https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/613669 (duration: 00m 57s)
• 21:53 eileen: tools revision changed from 40d52a0008 to b915d8efbd
• 21:15 sbassett: Revised mitigation deployed for T257687
• 20:07 eileen: tools revision changed from 711d671600 to 40d52a0008
• 19:10 mforns@deploy1001: Finished deploy [analytics/refinery@af86a05] (thin): Regular analytics weekly train THIN [analytics/refinery@af86a05be470ed8283f6585afb5cc231b26944a2] (duration: 00m 07s)
• 19:10 mforns@deploy1001: Started deploy [analytics/refinery@af86a05] (thin): Regular analytics weekly train THIN [analytics/refinery@af86a05be470ed8283f6585afb5cc231b26944a2]
• 19:09 mforns@deploy1001: Finished deploy [analytics/refinery@af86a05]: Regular analytics weekly train [analytics/refinery@af86a05be470ed8283f6585afb5cc231b26944a2] (duration: 05m 46s)
• 19:03 mforns@deploy1001: Started deploy [analytics/refinery@af86a05]: Regular analytics weekly train [analytics/refinery@af86a05be470ed8283f6585afb5cc231b26944a2]
• 18:37 urbanecm@deploy1001: Synchronized wmf-config/CommonSettings.php: df2584f: Switch $wgUrlShortenerDomainsWhitelist --> $wgUrlShortenerAllowedDomains (T255491) (duration: 00m 57s)
• 18:26 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: dfed472: Adding rollbacker group for arzwiki (T258100) (duration: 00m 57s)
• 18:24 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: ee7ac95: Change of rollbacker group settings at jawiki (T258339) (duration: 00m 57s)
• 17:36 ejegg: updated payments-wiki settings to point TY page at new URL
• 16:32 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@10afb4b]: airflow: Turn off catchup on cirrus_namespace_map (duration: 00m 25s)
• 16:31 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@10afb4b]: airflow: Turn off catchup on cirrus_namespace_map
• 16:27 akosiaris: increase codfw mobileapps kubernetes traffic to 25% T218733. Take #2
• 16:27 akosiaris@cumin1001: conftool action : set/weight=8; selector: dc=codfw,service=mobileapps,name=scb.*
• 15:59 elukey: restart airflow-webserver/scheduler to pick up TLS to mysql settings
• 15:21 hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:21 hnowlan@cumin1001: START - Cookbook sre.hosts.downtime
• 15:17 hnowlan: draining and restarting sessionstore2002
• 15:17 hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:17 hnowlan@cumin1001: START - Cookbook sre.hosts.downtime
• 15:16 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 15:13 jynus: dropping and recreating nagios@localhost users on all m1 servers
• 15:12 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 15:09 hnowlan: draining and restarting sessionstore2001
• 15:09 hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:09 hnowlan@cumin1001: START - Cookbook sre.hosts.downtime
• 15:09 hnowlan@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 15:09 hnowlan@cumin1001: START - Cookbook sre.hosts.downtime
• 15:08 moritzm: draining restbase2023 for eventual reboot for kernel security update
• 15:04 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 15:00 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:56 moritzm: draining restbase2022 for eventual reboot for kernel security update
• 14:56 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:56 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 14:54 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 14:52 hnowlan: draining and restarting sessionstore1003
• 14:52 hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:52 hnowlan@cumin1001: START - Cookbook sre.hosts.downtime
• 14:51 mholloway-shell@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' .
• 14:51 mholloway-shell@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'production' .
• 14:49 mholloway-shell@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'mobileapps' for release 'production' .
• 14:49 mholloway-shell@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' .
• 14:49 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:47 mholloway-shell@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'mobileapps' for release 'staging' .
• 14:47 moritzm: draining restbase2021 for eventual reboot for kernel security update
• 14:44 hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:43 hnowlan@cumin1001: START - Cookbook sre.hosts.downtime
• 14:37 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 14:36 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@ff49fdf]: Update mobileapps to 0bf7bafa (duration: 03m 50s)
• 14:34 hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:34 hnowlan@cumin1001: START - Cookbook sre.hosts.downtime
• 14:34 hnowlan: starting drain and restart of sessionstore hosts for new kernel
• 14:33 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:32 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@ff49fdf]: Update mobileapps to 0bf7bafa
• 14:26 moritzm: draining restbase2020 for eventual reboot for kernel security update
• 14:23 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:23 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 14:20 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 14:17 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:14 moritzm: draining restbase2019 for eventual reboot for kernel security update
• 14:08 ema: lvs101[34] (primaries) - restart pybal to apply varnish healthcheck changes https://gerrit.wikimedia.org/r/c/operations/puppet/+/610047 T255015
• 14:07 ema: lvs1016 (secondary) - restart pybal to apply varnish healthcheck changes https://gerrit.wikimedia.org/r/c/operations/puppet/+/610047 T255015
• 14:06 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 14:02 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 13:59 ema: lvs300[56] (primaries) - restart pybal to apply varnish healthcheck changes https://gerrit.wikimedia.org/r/c/operations/puppet/+/610047 T255015
• 13:57 ema: lvs3007 (secondary) - restart pybal to apply varnish healthcheck changes https://gerrit.wikimedia.org/r/c/operations/puppet/+/610047 T255015
• 13:50 ema: lvs500[12] (primaries) - restart pybal to apply varnish healthcheck changes https://gerrit.wikimedia.org/r/c/operations/puppet/+/610047 T255015
• 13:48 moritzm: draining restbase2018 for eventual reboot for kernel security update
• 13:47 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:47 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 13:47 ema: lvs5003 (secondary) - restart pybal to apply varnish healthcheck changes https://gerrit.wikimedia.org/r/c/operations/puppet/+/610047 T255015
• 13:44 ema: lvs200[78] (primaries) - restart pybal to apply varnish healthcheck changes https://gerrit.wikimedia.org/r/c/operations/puppet/+/610047 T255015
• 13:42 ema: lvs2010 (secondary) - restart pybal to apply varnish healthcheck changes https://gerrit.wikimedia.org/r/c/operations/puppet/+/610047 T255015
• 13:34 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 13:31 ema: lvs400[56] (primaries) - restart pybal to apply varnish healthcheck changes https://gerrit.wikimedia.org/r/c/operations/puppet/+/610047 T255015
• 13:31 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 13:27 moritzm: draining restbase2017 for eventual reboot for kernel security update
• 13:24 ema: lvs4007 (secondary) - restart pybal to apply varnish healthcheck changes https://gerrit.wikimedia.org/r/c/operations/puppet/+/610047 T255015
• 13:22 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 13:16 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 13:09 moritzm: draining restbase2016 for eventual reboot for kernel security update
• 13:08 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:08 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 13:07 moritzm: reset broken ifup systemd states on puppetdb* hosts
• 13:05 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 13:01 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 12:59 Urbanecm: creating arywiki (T257674), lijwikisource (T257672), sysop_itwiki (T256545) done
• 12:59 moritzm: draining restbase2015 for eventual reboot for kernel security update
• 12:56 Urbanecm: Create Daimona Eaytoy at sysop_itwiki (T256545)
• 12:55 urbanecm@deploy1001: Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 01m 59s)
• 12:50 urbanecm@deploy1001: Synchronized static/images/project-logos/: Creating sysop_itwiki (T256545) (duration: 00m 57s)
• 12:49 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Creating sysop_itwiki (T256545) (duration: 00m 57s)
• 12:48 urbanecm@deploy1001: rebuilt and synchronized wikiversions files: Creating sysop_itwiki (T256545)
• 12:46 urbanecm@deploy1001: Synchronized dblists: Creating sysop_itwiki (T256545) (duration: 00m 57s)
• 12:46 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 12:43 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 12:40 moritzm: draining restbase2014 for eventual reboot for kernel security update
• 12:38 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 12:38 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 12:35 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 12:34 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Creating lijwikisource (T257672) (duration: 00m 57s)
• 12:32 urbanecm@deploy1001: rebuilt and synchronized wikiversions files: Creating lijwikisource (T257672)
• 12:31 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 12:30 urbanecm@deploy1001: Synchronized dblists: Creating lijwikisource (T257672) (duration: 00m 56s)
• 12:28 urbanecm@deploy1001: Synchronized dblists/rtl.dblist: Add arywiki to rtl.dblist (T257674) (duration: 00m 57s)
• 12:27 moritzm: draining restbase2013 for eventual reboot for kernel security update
• 12:27 urbanecm@deploy1001: sync-file aborted: (no justification provided) (duration: 00m 00s)
• 12:21 urbanecm@deploy1001: Synchronized langlist: Creating arywiki (T257674) (duration: 00m 56s)
• 12:20 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Creating arywiki (T257674) (duration: 00m 56s)
• 12:19 urbanecm@deploy1001: Synchronized static/images/project-logos/: Creating arywiki (T257674) (duration: 00m 57s)
• 12:17 urbanecm@deploy1001: rebuilt and synchronized wikiversions files: Creating arywiki (T257674)
• 12:16 urbanecm@deploy1001: Synchronized dblists: Creating arywiki (T257674) (duration: 00m 57s)
• 12:02 moritzm: installing qemu security updates on buster
• 11:50 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 946bf3d: Update brwikimedia logo and add upscaled versions (config) (T257925) (duration: 00m 57s)
• 11:49 urbanecm@deploy1001: sync-file aborted: (no justification provided) (duration: 00m 00s)
• 11:49 Urbanecm: Purge 'https://en.wikipedia.org/static/images/project-logos/bnwikimedia.png'
• 11:46 urbanecm@deploy1001: Synchronized static/images/project-logos/: f7560b6: Update brwikimedia logo and add upscaled versions (T257925) (duration: 00m 56s)
• 11:44 urbanecm@deploy1001: Synchronized wmf-config/CommonSettings.php: 5b97a06: Set $wgUrlShortenerAllowedDomains for all wikis (T258134) (duration: 00m 57s)
• 11:42 urbanecm@deploy1001: sync-file aborted: (no justification provided) (duration: 00m 00s)
• 11:36 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: c12f1de: Remove wgPopupsPageBlacklist config setting (T254676) (duration: 00m 57s)
• 11:35 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript createAndPromote.php testwikidatawiki --custom-groups=interface-admin --force 'Lucas Werkmeister (WMDE)'
• 11:34 urbanecm@deploy1001: sync-file aborted: (no justification provided) (duration: 00m 01s)
• 11:25 Urbanecm: mwscript namespaceDupes.php --wiki=kowikiquote --fix (T255031)
• 11:24 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 3719668: Add NamespaceAliases for kowikiquote (T255031) (duration: 00m 57s)
• 11:22 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: bc5671a: Add media.farsnews.ir to the wgCopyUploadsDomains allowlist of Wikimedia Commons (T253800) (duration: 00m 57s)
• 11:18 Urbanecm: Run mwscript updateCollation.php --wiki=bswiktionary --previous-collation=uppercase in a tmux session at mwmaint1002 (T258346)
• 11:17 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 0c78478: Set $wgCategoryCollation to uca-bs-u-kn on Bosnian Wiktionary (T258346) (duration: 00m 58s)
• 11:13 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 6830723: Convert ukwikisource ns:250 and ns:251 to have subpages (T255930) (duration: 00m 57s)
• 11:10 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 1c7a621: Create closer group at itwikinews (T257927) (duration: 00m 57s)
• 10:55 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 10:51 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 10:50 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 10:48 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 10:48 moritzm: rebooting releases* hosts for kernel security update
• 10:35 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (614698) (duration: 00m 56s)
• 10:34 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (614698) (duration: 00m 59s)
• 10:30 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1114', diff saved to https://phabricator.wikimedia.org/P11962 and previous config saved to /var/cache/conftool/dbconfig/20200720-103058-marostegui.json
• 09:48 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1114', diff saved to https://phabricator.wikimedia.org/P11961 and previous config saved to /var/cache/conftool/dbconfig/20200720-094609-marostegui.json
• 09:31 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1114', diff saved to https://phabricator.wikimedia.org/P11960 and previous config saved to /var/cache/conftool/dbconfig/20200720-093154-marostegui.json
• 09:25 godog: update compiler facts
• 09:17 jayme: updating envoyproxy to 1.14.4-1 on all eqiad hosts
• 09:11 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1114', diff saved to https://phabricator.wikimedia.org/P11959 and previous config saved to /var/cache/conftool/dbconfig/20200720-091119-marostegui.json
• 09:04 jayme: updating envoyproxy to 1.14.4-1 on all codfw hosts
• 07:54 moritzm: installing libopenmpt security updates
• 07:51 jayme: updating envoyproxy to 1.14.4-1 on all non mw and restbase hosts
• 07:29 marostegui: Move m1-master from dbproxy1012 to dbproxy1014 - T255408
• 07:19 marostegui: Drop non used reviewdb database - T255715
• 06:55 elukey: restart matomo1002's mariadb to pick up new TLS settings
• 06:54 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1114', diff saved to https://phabricator.wikimedia.org/P11958 and previous config saved to /var/cache/conftool/dbconfig/20200720-065438-marostegui.json
• 06:15 tstarling@deploy1001: Synchronized php-1.35.0-wmf.41/extensions/Score/includes/Score.php: reverting Reedy's temporary patch for hardcoding the lilypond version (duration: 00m 57s)
• 06:07 tstarling@deploy1001: Finished scap: fixing missing message from previous sync-dir (duration: 29m 57s)
• 05:56 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1082 after a crash T258336', diff saved to https://phabricator.wikimedia.org/P11957 and previous config saved to /var/cache/conftool/dbconfig/20200720-055614-marostegui.json
• 05:47 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1082 after a crash T258336', diff saved to https://phabricator.wikimedia.org/P11956 and previous config saved to /var/cache/conftool/dbconfig/20200720-054747-marostegui.json
• 05:40 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1082 after a crash T258336', diff saved to https://phabricator.wikimedia.org/P11955 and previous config saved to /var/cache/conftool/dbconfig/20200720-053816-marostegui.json
• 05:37 tstarling@deploy1001: Started scap: fixing missing message from previous sync-dir
• 05:30 tstarling@deploy1001: scap sync-l10n completed (1.35.0-wmf.41) (duration: 02m 44s)
• 05:25 marostegui: Deploy MCR schema change on enwiki on db1119 - T238966
• 05:24 tstarling@deploy1001: Synchronized wmf-config/CommonSettings.php: disable lilypond with better error message (duration: 00m 57s)
• 05:18 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1082 after a crash T258336', diff saved to https://phabricator.wikimedia.org/P11953 and previous config saved to /var/cache/conftool/dbconfig/20200720-051846-marostegui.json
• 05:18 tstarling@deploy1001: Synchronized php-1.35.0-wmf.41/extensions/Score: better error message for disabling of Score (duration: 01m 10s)

2020-07-19

• 19:16 marostegui: Upgrade and reboot db1085 T258360
• 18:57 marostegui: Start mysql on db1082 T258336
• 18:51 marostegui: Upgrade and reboot db1082 T258336
• 18:45 cdanis@cumin1001: dbctl commit (dc=all): 'db1085 also crashed', diff saved to https://phabricator.wikimedia.org/P11952 and previous config saved to /var/cache/conftool/dbconfig/20200719-184511-cdanis.json
• 18:06 Urbanecm: Run mwscript emptyUserGroup.php --wiki=testwiki contestadmin (T256555)

2020-07-18

• 21:41 shdubsh: restart logstash on logstash200[456]
• 21:14 shdubsh: bounce logstash on logstash1007
• 21:10 shdubsh: bounce logstash on logstash1008
• 21:06 shdubsh: bounce logstash on logstash1009
• 20:52 marostegui: Due to db1082 crash there will be replication lag on s5 on labsdb hosts - T258336
• 20:37 cdanis@cumin1001: dbctl commit (dc=all): 'depool db1082, it crashed', diff saved to https://phabricator.wikimedia.org/P11951 and previous config saved to /var/cache/conftool/dbconfig/20200718-203704-cdanis.json
• 00:13 dpifke: Performing one-time expiration of ArcLamp files older than 40 days (normal retention is 45 days), to solve disk space issue until either Ganeti issue is solved or compressed logfile support is merged.

2020-07-17

• 21:16 dpifke: Removing MongoDB packages and data from webperf1002.
• 17:39 dpifke@deploy1001: Finished deploy [performance/arc-lamp@a5d2fd3]: (no justification provided) (duration: 00m 05s)
• 17:38 dpifke@deploy1001: Started deploy [performance/arc-lamp@a5d2fd3]: (no justification provided)
• 13:53 akosiaris: powercycle kubernetes2002
• 12:24 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1104', diff saved to https://phabricator.wikimedia.org/P11944 and previous config saved to /var/cache/conftool/dbconfig/20200717-122400-marostegui.json
• 12:01 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1104', diff saved to https://phabricator.wikimedia.org/P11941 and previous config saved to /var/cache/conftool/dbconfig/20200717-120126-marostegui.json
• 11:51 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1104', diff saved to https://phabricator.wikimedia.org/P11940 and previous config saved to /var/cache/conftool/dbconfig/20200717-115155-marostegui.json
• 11:38 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1104', diff saved to https://phabricator.wikimedia.org/P11939 and previous config saved to /var/cache/conftool/dbconfig/20200717-113800-marostegui.json
• 11:30 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1104', diff saved to https://phabricator.wikimedia.org/P11938 and previous config saved to /var/cache/conftool/dbconfig/20200717-113050-marostegui.json
• 11:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1104', diff saved to https://phabricator.wikimedia.org/P11937 and previous config saved to /var/cache/conftool/dbconfig/20200717-112413-marostegui.json
• 09:15 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1280.eqiad.wmnet
• 09:12 elukey@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=mw1280.eqiad.wmnet
• 08:48 moritzm: imported prometheus-atlas-exporter 1.0+git20191204.ffafab7-2 to buster-wikimedia T247967
• 08:29 elukey@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 08:05 elukey@cumin1001: START - Cookbook sre.ganeti.makevm
• 07:54 elukey@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 07:51 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1104', diff saved to https://phabricator.wikimedia.org/P11936 and previous config saved to /var/cache/conftool/dbconfig/20200717-075124-marostegui.json
• 07:43 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1111', diff saved to https://phabricator.wikimedia.org/P11935 and previous config saved to /var/cache/conftool/dbconfig/20200717-074335-marostegui.json
• 07:34 akosiaris@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' .
• 07:34 akosiaris@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'mobileapps' for release 'production' .
• 07:33 akosiaris@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'production' .
• 07:33 akosiaris@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' .
• 07:32 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'mobileapps' for release 'staging' .
• 07:30 elukey@cumin1001: START - Cookbook sre.ganeti.makevm
• 06:30 XioNoX: rename msw1-codfw interface range
• 06:28 XioNoX: rename msw1-eqiad interface range
• 04:47 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1111', diff saved to https://phabricator.wikimedia.org/P11934 and previous config saved to /var/cache/conftool/dbconfig/20200717-044748-marostegui.json
• 04:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1092', diff saved to https://phabricator.wikimedia.org/P11933 and previous config saved to /var/cache/conftool/dbconfig/20200717-044658-marostegui.json

2020-07-16

• 22:15 mutante: testreduce1001 manually git clone 'scandium' branch of integration/visualdiff into /srv/visualdiff (T257906)
• 21:54 crusnov@deploy1001: Finished deploy [netbox/deploy@39c5cae]: Deploying Netbox 2.8.7 part 3 (duration: 01m 49s)
• 21:52 crusnov@deploy1001: Started deploy [netbox/deploy@39c5cae]: Deploying Netbox 2.8.7 part 3
• 21:42 crusnov@deploy1001: Finished deploy [netbox/deploy@39c5cae]: Deploying Netbox 2.8.7 part 2 (duration: 01m 33s)
• 21:41 crusnov@deploy1001: Started deploy [netbox/deploy@39c5cae]: Deploying Netbox 2.8.7 part 2
• 21:40 crusnov@deploy1001: Finished deploy [netbox/deploy@39c5cae]: Deploying Netbox 2.8.7 (duration: 01m 01s)
• 21:39 crusnov@deploy1001: Started deploy [netbox/deploy@39c5cae]: Deploying Netbox 2.8.7
• 21:08 cstone: payments-wiki revision changed from 91852dbc9b to bf91f8adff
• 20:32 mholloway-shell@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable client error logging on Catalan Wikipedia (T258073) (duration: 00m 57s)
• 19:32 sbassett: Deployed mitigations for T257687
• 19:14 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T248418 TimedMediaHandler: Make videojs the only player on all group0 (duration: 00m 57s)
• 18:54 herron@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 18:53 herron@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 18:50 herron@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 18:49 addshore: deployment windows finished with
• 18:46 addshore@deploy1001: Synchronized wmf-config/extension-list: gerrit:611393 extension-list: Load WikibaseClient via JSON (duration: 00m 56s)
• 18:36 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: gerrit:613226 Wikibase: Always set wgWBRepoSettings idGeneratorSeparateDbConnection PT 2/2 (duration: 00m 56s)
• 18:35 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: gerrit:613226 Wikibase: Always set wgWBRepoSettings idGeneratorSeparateDbConnection PT 1/2 (duration: 00m 56s)
• 18:25 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: gerrit:613165 T138104 Wikibase: stop setting wmgWikibaseTmpSerializeEmptyListsAsObjects (duration: 00m 57s)
• 18:23 addshore@deploy1001: Synchronized wmf-config/config/incubatorwiki.yaml: gerrit:613199 T256957 Move VisualEditor from beta to default on incubatorwiki PT2/2 (duration: 00m 57s)
• 18:22 addshore@deploy1001: Synchronized dblists/visualeditor-nondefault.dblist: gerrit:613199 T256957 Move VisualEditor from beta to default on incubatorwiki PT1/2 (duration: 00m 56s)
• 18:20 addshore@deploy1001: Synchronized wmf-config/config/nlwikimedia.yaml: gerrit:613198 T256142 Move VisualEditor from beta to default on nlwikimedia PT2/2 (duration: 00m 57s)
• 18:18 addshore@deploy1001: Synchronized dblists/visualeditor-nondefault.dblist: gerrit:613198 T256142 Move VisualEditor from beta to default on nlwikimedia PT1/2 (duration: 00m 56s)
• 18:14 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: gerrit:613164 T138104 Wikibase: stop setting wgWBRepoSettings tmpSerializeEmptyListsAsObjects (duration: 00m 57s)
• 18:12 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: gerrit:613192 T246420 Enable limited-width layout for Modern Vector (duration: 00m 56s)
• 18:08 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: gerrit:612870 T246977 Disable affinity quicksurveys for the following wikis (duration: 00m 57s)
• 18:03 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 18:03 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 17:54 herron@cumin1001: START - Cookbook sre.ganeti.makevm
• 17:53 herron@cumin1001: START - Cookbook sre.ganeti.makevm
• 17:50 herron@cumin1001: START - Cookbook sre.ganeti.makevm
• 17:50 herron@cumin1001: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99)
• 17:49 herron@cumin1001: START - Cookbook sre.ganeti.makevm
• 17:17 XioNoX: msw1-eqiad delete unused VC-ports
• 17:05 XioNoX: msw1-codfw - replace member-range with list of individual interfaces
• 16:45 lucaswerkmeister-wmde@deploy1001: Synchronized php-1.35.0-wmf.41/extensions/Wikibase/: Backport: Re add OtherProjectsSidebarGenerator::buildProjectLinkSidebarFromItemId (T258184) (duration: 01m 02s)
• 16:11 effie: reboot rdb1009 - T254990
• 16:06 effie: Reboot rdb1010 - T254990
• 15:51 lucaswerkmeister-wmde@deploy1001: Synchronized php-1.35.0-wmf.41/extensions/Wikibase/: Backport: Revert "Revert "Removes OtherProjectsSidebar hook"" (T258184) (duration: 01m 02s)
• 15:40 lucaswerkmeister-wmde@deploy1001: scap failed: average error rate on 7/9 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/e474f13ffac6b8c3bf919c4aeafc8c9b for details)
• 15:15 akosiaris: lower codfw mobileapps kubernetes traffic to 10% T218733. Will open up task for it
• 15:15 akosiaris@cumin1001: conftool action : set/weight=24; selector: dc=codfw,service=mobileapps,name=scb.*
• 15:07 XioNoX: repool eqsin - T257154
• 15:04 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 15:02 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 15:00 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'mathoid' for release 'production' .
• 14:54 XioNoX: load config on cr3-eqsin - T257154
• 14:54 lucaswerkmeister-wmde@deploy1001: Synchronized php-1.35.0-wmf.41/extensions/Wikibase/: Backport: Avoid trying to register wikibase.Site twice (T258065) (duration: 01m 03s)
• 14:43 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'mathoid' for release 'production' .
• 14:31 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'mathoid' for release 'staging' .
• 14:15 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 14:12 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:12 moritzm: rebooting webperf hosts in eqiad for kernel update
• 14:09 XioNoX: upgrade junos on cr3-eqsin - T257154
• 14:03 jayme: published image docker-registry.discovery.wmnet/envoy:1.14.4-1
• 13:47 XioNoX: remove nonstop-bridging from asw1-eqsin
• 13:36 XioNoX: power-off cr3-eqsin - T257154
• 13:36 akosiaris: increase codfw mobileapps kubernetes traffic to 25% T218733
• 13:35 akosiaris@cumin1001: conftool action : set/weight=8; selector: dc=codfw,service=mobileapps,name=scb.*
• 13:30 XioNoX: deactivate BGP groups IX/Transit/PyBal on cr3-eqsin - T257154
• 13:27 moritzm: installing an-tool1008
• 13:23 XioNoX: depool eqsin for cr3 replacement - T257154
• 13:13 volans@deploy1001: Finished deploy [homer/deploy@fcf4332]: Force deploy of the homer plugin (duration: 01m 27s)
• 13:12 volans@deploy1001: Started deploy [homer/deploy@fcf4332]: Force deploy of the homer plugin
• 13:04 kormat: restarting tendril to pick up new mariadb config T257816
• 13:02 jforrester@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.41
• 13:02 akosiaris: increase codfw mobileapps kubernetes traffic to 10% T218733
• 13:01 akosiaris@cumin1001: conftool action : set/weight=24; selector: dc=codfw,service=mobileapps,name=scb.*
• 12:56 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1092', diff saved to https://phabricator.wikimedia.org/P11926 and previous config saved to /var/cache/conftool/dbconfig/20200716-125643-marostegui.json
• 12:56 ayounsi@deploy1001: Finished deploy [homer/deploy@fcf4332]: CR607011 (duration: 04m 32s)
• 12:52 ayounsi@deploy1001: Started deploy [homer/deploy@fcf4332]: CR607011
• 12:42 ayounsi@deploy1001: Finished deploy [homer/deploy@fcf4332]: CR607011 (duration: 03m 42s)
• 12:38 ayounsi@deploy1001: Started deploy [homer/deploy@fcf4332]: CR607011
• 12:38 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:36 akosiaris@cumin1001: conftool action : set/weight=50; selector: dc=codfw,service=mobileapps,name=scb.*
• 12:35 akosiaris: increase codfw mobileapps kubernetes traffic to 5% T218733
• 12:35 akosiaris: increase codfw mobileapps kubernetes traffic to 5%
• 12:34 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
• 12:22 jmm@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 12:12 jmm@cumin1001: START - Cookbook sre.ganeti.makevm
• 12:12 jmm@cumin1001: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99)
• 12:12 jmm@cumin1001: START - Cookbook sre.ganeti.makevm
• 12:12 jmm@cumin1001: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99)
• 12:12 jmm@cumin1001: START - Cookbook sre.ganeti.makevm
• 12:08 jayme: updated envoyproxy to 1.14.4-1 on mw-canary and restbase-canary
• 11:44 XioNoX: remove BGP to AS396253 in eqdfw (peer left the IX)
• 11:26 jforrester@deploy1001: Synchronized php-1.35.0-wmf.41/extensions/UrlShortener/includes/UrlShortenerUtils.php: T258134 Fix config variables regex concatenation (duration: 01m 05s)
• 11:23 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: T254315 gerrit:612670 Wikibase: remove wmgWikibaseLocalEntitySourceName (duration: 01m 05s)
• 11:18 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T254315 T257266 gerrit:609988 Wikidata client wikis: Define entity sources configuration (take 3) (duration: 01m 08s)
• 10:17 jbond42: upgrade to hiera5
• 10:08 jbond42: disable puppet for hiera5 deployment
• 09:37 jayme: updated envoyproxy to 1.14.4-1 on mw1325.eqiad.wmnet and restbase1026.eqiad.wmnet
• 09:32 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 09:30 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 09:22 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 09:21 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 09:17 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 09:15 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 09:15 jmm@cumin2001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99)
• 09:15 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 09:15 moritzm: rebooting flowspec1001
• 08:52 jayme: updated envoyproxy to 1.14.4-1 on mwdebug1001.eqiad.wmnet
• 08:41 moritzm: installing sqlite3 security updates
• 08:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db2081', diff saved to https://phabricator.wikimedia.org/P11924 and previous config saved to /var/cache/conftool/dbconfig/20200716-083954-marostegui.json
• 08:35 XioNoX: Remove PIM/IGMP related CR stanza (acls) - T257573
• 08:33 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:33 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 08:33 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:33 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 08:32 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:32 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 08:26 moritzm: installing dbus security updates
• 08:25 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:24 XioNoX: remove igmp-snooping from access switches - T257573
• 08:23 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 08:15 moritzm: installing python-urllib3 security updates
• 08:15 XioNoX: remove PIM config from eqord/eqdfw/knams routers - T257573
• 08:14 XioNoX: remove PIM config from eqiad routers - T257573
• 08:11 XioNoX: remove PIM config from esams routers - T257573
• 08:09 XioNoX: remove PIM config from eqsin routers - T257573
• 08:08 jbond42: update mail delivery for phabricator to use phabricator.discovery.wmnet cname
• 08:07 XioNoX: remove PIM config from codfw routers - T257573
• 08:06 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2081', diff saved to https://phabricator.wikimedia.org/P11923 and previous config saved to /var/cache/conftool/dbconfig/20200716-080613-marostegui.json
• 08:03 XioNoX: remove PIM config from ulsfo routers - T257573
• 07:41 jayme: imported envoyproxy_1.14.4-1 to stretch-wikimedia
• 07:31 jayme: imported envoyproxy_1.14.4-1 to buster-wikimedia
• 07:28 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1131', diff saved to https://phabricator.wikimedia.org/P11922 and previous config saved to /var/cache/conftool/dbconfig/20200716-072838-marostegui.json
• 07:25 marostegui: Drop database reviewdb-test T255715
• 07:03 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1131', diff saved to https://phabricator.wikimedia.org/P11921 and previous config saved to /var/cache/conftool/dbconfig/20200716-070331-marostegui.json
• 06:40 XioNoX: remove peering with AS8403 in eqsin (peer left the IX)
• 05:13 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1131', diff saved to https://phabricator.wikimedia.org/P11920 and previous config saved to /var/cache/conftool/dbconfig/20200716-051342-marostegui.json
• 05:11 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1131', diff saved to https://phabricator.wikimedia.org/P11919 and previous config saved to /var/cache/conftool/dbconfig/20200716-051109-marostegui.json

2020-07-15

• 23:54 eileen: tools revision changed from 7b6018a16e to 711d671600
• 23:50 eileen: process-control config revision is 1fc4a9686d
• 23:21 dzahn@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 23:04 bd808: tools.admin Removed valhallasw from maintainers (T255697)
• 23:02 dzahn@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 22:58 dzahn@cumin1001: START - Cookbook sre.ganeti.makevm
• 22:52 dzahn@cumin1001: START - Cookbook sre.ganeti.makevm
• 22:52 dzahn@cumin1001: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99)
• 22:52 dzahn@cumin1001: START - Cookbook sre.ganeti.makevm
• 22:30 akosiaris@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' .
• 22:30 akosiaris@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'mobileapps' for release 'production' .
• 22:29 akosiaris@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' .
• 22:29 akosiaris@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'production' .
• 22:27 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'mobileapps' for release 'staging' .
• 22:21 akosiaris@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' .
• 22:21 akosiaris@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'mobileapps' for release 'production' .
• 22:10 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'mobileapps' for release 'staging' .
• 18:16 brennen: restarting jenkins for upgrade
• 18:00 mutante: DNS - new language 'avk' has been added - This language is called Kotava and is "a proposed international auxiliary language (IAL) that focuses especially on the principle of cultural neutrality". Learn more at https://en.wikipedia.org/wiki/Kotava
• 17:32 mutante: puppetmaster - revoking cert for planet.discovery.wmnet, add planet.wikimedia.org, remove planet.svc records, remove specific and outdated hostnames (T257840)
• 16:11 moritzm: uploaded jenkins 2.235.2 to thirdparty/ci for stretch/buster T257614
• 15:29 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 15:24 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 15:24 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 15:20 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 15:20 moritzm: rebooting webperf* hosts for kernel update
• 14:58 addshore@deploy1001: Synchronized php-1.35.0-wmf.41/extensions/Wikibase/repo: gerrit:612723 Stop checking if WikibaseLib is loaded T258062 (already on mwmaint1002) (duration: 01m 08s)
• 14:51 addshore: pulled https://gerrit.wikimedia.org/r/612723 onto mwmaint 1002 ahead of syncing everywhere (and CI finishing)
• 14:37 ema: A:cp: upgrade purged to 0.17 T257573
• 14:30 ema: upload purged 0.17 to buster-wikimedia T257573
• 14:28 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add exceptional wikitech VE/Parsoid config T241961 (duration: 01m 04s)
• 14:26 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: Add exceptional wikitech VE/Parsoid config T241961 (duration: 01m 05s)
• 14:25 gehel: repooling wdqs1006 - catched up on lag
• 14:12 akosiaris: increase codfw mobileapps kubernetes traffic to 2% T218733
• 14:10 akosiaris@cumin1001: conftool action : set/weight=132; selector: dc=codfw,service=mobileapps,name=scb.*
• 13:58 jforrester@deploy1001: Synchronized php-1.35.0-wmf.41/extensions/UrlShortener/includes/UrlShortenerUtils.php: T258056 Add temporary fix to ensure array is passed to array_map() (duration: 01m 08s)
• 13:54 akosiaris: pool kubernetes nodes for mobileapps in codfw
• 13:53 akosiaris@cumin1001: conftool action : set/pooled=yes; selector: dc=codfw,service=mobileapps,name=kubernetes.*
• 13:53 akosiaris@cumin1001: conftool action : set/weight=264; selector: dc=codfw,service=mobileapps,name=scb.*
• 13:51 akosiaris@cumin1001: conftool action : set/weight=1; selector: dc=codfw,service=mobileapps,name=kubernetes.*
• 13:04 jforrester@deploy1001: Synchronized php: group1 wikis to 1.35.0-wmf.41 (duration: 01m 05s)
• 13:03 jforrester@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.41
• 11:59 addshore: deploy window closed / done :)
• 11:57 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: gerrit:609987 Commons: Define entity sources configuration (take 2) T254315 (duration: 01m 03s)
• 11:36 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: gerrit:612668 Wikibase test: Client local entity sources are always testwikidata T254315 (duration: 01m 05s)
• 11:27 addshore@deploy1001: Synchronized wmf-config: T254315 gerrit:612669 Wikidata test: Split client db lists. PT2/2 (duration: 01m 06s)
• 11:26 addshore@deploy1001: Synchronized dblists/wikidataclient.dblist: T254315 gerrit:612669 Wikidata test: Split client db lists. PT1/2 (duration: 01m 05s)
• 11:16 XioNoX: remove as-path prepending in esams
• 11:11 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: LABS gerrit:612667 Wikibase labs: All client "local" entity sources are wikidata T254315 (duration: 01m 04s)
• 11:08 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: gerrit:612666 Wikibase: Split localEntitySourceName config for repo and client T254315 (duration: 01m 16s)
• 11:05 XioNoX: re-enable ping offload in esams
• 11:05 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 11:01 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 10:56 XioNoX: disable ping offload in esams
• 10:55 XioNoX: re-enable ping offload in codfw
• 10:52 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 10:50 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 10:45 XioNoX: disable ping offload in codfw
• 10:44 XioNoX: re-enable ping offload in eqiad
• 10:43 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 10:41 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 10:31 XioNoX: disable ping offload in eqiad
• 10:31 akosiaris@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' .
• 10:30 akosiaris@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' .
• 10:30 akosiaris@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'mobileapps' for release 'production' .
• 10:30 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'mobileapps' for release 'staging' .
• 10:26 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1120 after reimage', diff saved to https://phabricator.wikimedia.org/P11916 and previous config saved to /var/cache/conftool/dbconfig/20200715-102605-marostegui.json
• 10:20 jayme: updating python3-docker-report to 0.0.5-1 on deneb
• 10:08 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1120 after reimage', diff saved to https://phabricator.wikimedia.org/P11915 and previous config saved to /var/cache/conftool/dbconfig/20200715-100855-marostegui.json
• 10:07 jayme: imported docker-report_0.0.5-1 to buster-wikimedia
• 09:48 marostegui: Deploy schema change on s8 codfw master, lag will appear on codfw T256685
• 09:42 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1120 after reimage', diff saved to https://phabricator.wikimedia.org/P11914 and previous config saved to /var/cache/conftool/dbconfig/20200715-094226-marostegui.json
• 09:22 akosiaris@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'production' .
• 09:21 akosiaris@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'mobileapps' for release 'production' .
• 09:19 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'mobileapps' for release 'staging' .
• 09:19 akosiaris: deploy mobileapps in kubernetes to talk HTTPS to the mw API
• 09:10 akosiaris@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' .
• 09:10 akosiaris@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' .
• 09:07 akosiaris: Correction: deploy eventgate-analytics-external in staging, eqiad, codfw for switching to using discovery records and HTTPS for talking to the API
• 09:06 akosiaris: deploy eventgate-analytics in staging, eqiad, codfw for switching to using discovery records and HTTPS for talking to the API
• 09:06 akosiaris@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' .
• 09:06 akosiaris@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' .
• 09:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1099:3318', diff saved to https://phabricator.wikimedia.org/P11913 and previous config saved to /var/cache/conftool/dbconfig/20200715-090545-marostegui.json
• 09:04 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' .
• 09:04 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' .
• 08:50 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1120 after reimage', diff saved to https://phabricator.wikimedia.org/P11912 and previous config saved to /var/cache/conftool/dbconfig/20200715-085032-marostegui.json
• 08:39 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:36 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 08:19 moritzm: piwik.wikimedia.org switched to CAS authentication
• 08:19 elukey: move piwik.wikimedia.org to CAS (idp.wikimedia.org)
• 07:29 XioNoX: delete deprecated AS3209 AMS-IX router
• 06:59 dcausse: depooling wdqs1006 (high lag)
• 06:09 marostegui: Stop replication on db1120 to avoid having 10.4 -> 10.1 replication for long T254871
• 06:06 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1120 for reimage T254871', diff saved to https://phabricator.wikimedia.org/P11911 and previous config saved to /var/cache/conftool/dbconfig/20200715-060649-marostegui.json
• 06:01 marostegui@cumin1001: dbctl commit (dc=all): 'Promote db1103 to x1 master T254871', diff saved to https://phabricator.wikimedia.org/P11910 and previous config saved to /var/cache/conftool/dbconfig/20200715-060145-marostegui.json
• 06:00 marostegui: Starting x1 failover from db1120 to db1103 - T254871
• 05:29 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1099:3318 ', diff saved to https://phabricator.wikimedia.org/P11909 and previous config saved to /var/cache/conftool/dbconfig/20200715-052939-marostegui.json
• 04:46 marostegui: Start x1 pre failover steps T254871
• 04:44 marostegui@cumin1001: dbctl commit (dc=all): 'Set db1103 weight to 0 before the switchover T254871', diff saved to https://phabricator.wikimedia.org/P11908 and previous config saved to /var/cache/conftool/dbconfig/20200715-044432-marostegui.json
• 04:43 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1135', diff saved to https://phabricator.wikimedia.org/P11907 and previous config saved to /var/cache/conftool/dbconfig/20200715-044332-marostegui.json
• 01:45 eileen: tools revision changed from a9e7dc1559 to 7b6018a16e
• 00:26 ryankemper@deploy1001: Finished deploy [wdqs/wdqs@8f6f660]: 0.3.41 (duration: 15m 10s)
• 00:11 ryankemper@deploy1001: Started deploy [wdqs/wdqs@8f6f660]: 0.3.41

2020-07-14

• 19:52 jforrester@deploy1001: Synchronized php-1.35.0-wmf.41/vendor/wikimedia/parsoid/: T252448 T255190 Bump Parsoid to v0.12.0-a23 (duration: 01m 06s)
• 18:13 ryankemper: all long-running elasticsearch reindex jobs are complete
• 18:09 jforrester@deploy1001: Synchronized dblists/: T32405 T254287 Remove the mobilemainpagelegacy dblist (duration: 01m 04s)
• 18:07 jforrester@deploy1001: Synchronized multiversion/MWConfigCacheGenerator.php: T32405 T254287 Stop loading the mobilemainpagelegacy dblist (duration: 01m 05s)
• 18:05 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T32405 T254287 Stop varying wgMFSpecialCaseMainPage (duration: 01m 05s)
• 15:56 elukey: upgrade spark2 on stat100x to 2.4.4-bin-hadoop2.6-3
• 15:40 hnowlan@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop' for release 'production' .
• 15:25 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 15:23 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 15:17 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 15:13 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 15:12 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 15:10 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 15:04 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 15:02 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:57 jforrester@deploy1001: Synchronized php-1.35.0-wmf.41/skins/Vector/includes/SkinVector.php: T257914 Restore div wrapper around print footer (duration: 01m 03s)
• 14:53 hnowlan@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'changeprop' for release 'production' .
• 14:50 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 14:48 jforrester@deploy1001: Synchronized php-1.35.0-wmf.41/extensions/WikibaseMediaInfo/src/WikibaseMediaInfoHooks.php: Fix case of directory name (duration: 01m 05s)
• 14:48 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:48 moritzm: rebooting apt1001 for kernel update
• 14:42 jynus: stopping db1117:3322 (m2) replication temp. for otrs db cloning T257928
• 14:40 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' .
• 14:33 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 14:31 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:26 oblivian@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'cxserver' for release 'production' .
• 14:23 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 14:21 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:20 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 14:18 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:18 jmm@cumin2001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99)
• 14:18 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:14 oblivian@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'cxserver' for release 'production' .
• 14:13 andrewbogott: upgrading wikitech-static to mw 1.34.2
• 14:11 oblivian@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'cxserver' for release 'staging' .
• 14:05 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 14:01 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 13:42 elukey@cumin1001: END (PASS) - Cookbook sre.hadoop.change-distro (exit_code=0)
• 13:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1112', diff saved to https://phabricator.wikimedia.org/P11900 and previous config saved to /var/cache/conftool/dbconfig/20200714-132823-marostegui.json
• 13:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1130', diff saved to https://phabricator.wikimedia.org/P11899 and previous config saved to /var/cache/conftool/dbconfig/20200714-132742-marostegui.json
• 13:27 jbond@cumin1001: conftool action : set/pooled=yes; selector: name=dns1001.wikimedia.org
• 13:24 jbond42: reboot dns1001
• 13:23 jbond@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:23 jbond@cumin1001: START - Cookbook sre.hosts.downtime
• 13:22 jbond@cumin1001: conftool action : set/pooled=no; selector: name=dns1001.wikimedia.org
• 13:22 jbond@cumin1001: conftool action : set/pooled=yes; selector: name=dns1002.wikimedia.org
• 13:18 jbond42: reboot dns1002
• 13:18 jbond@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:18 jbond@cumin1001: START - Cookbook sre.hosts.downtime
• 13:18 jbond@cumin1001: conftool action : set/pooled=no; selector: name=dns1002.wikimedia.org
• 13:16 jbond@cumin1001: conftool action : set/pooled=yes; selector: name=dns2002.wikimedia.org
• 13:13 jbond42: reboot dns2002
• 13:13 jbond@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:13 jbond@cumin1001: START - Cookbook sre.hosts.downtime
• 13:13 jbond@cumin1001: conftool action : set/pooled=no; selector: name=dns2002.wikimedia.org
• 13:13 jbond@cumin1001: conftool action : set/pooled=yes; selector: name=dns2001.wikimedia.org
• 13:10 jbond42: reboot dns2001
• 13:10 jbond@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:10 jbond@cumin1001: START - Cookbook sre.hosts.downtime
• 13:10 jbond@cumin1001: conftool action : set/pooled=no; selector: name=dns2001.wikimedia.org
• 13:09 elukey@cumin1001: START - Cookbook sre.hadoop.change-distro
• 13:06 elukey@cumin1001: END (PASS) - Cookbook sre.hadoop.stop-cluster (exit_code=0)
• 13:01 jbond42: rebooting dns3002
• 13:01 jbond@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:01 jbond@cumin1001: START - Cookbook sre.hosts.downtime
• 12:58 elukey@cumin1001: START - Cookbook sre.hadoop.stop-cluster
• 12:57 oblivian@deploy1001: Synchronized wmf-config/InitialiseSettings.php: revert forcehttps after fixing T257887 (duration: 01m 02s)
• 12:31 elukey@cumin1001: END (PASS) - Cookbook sre.hadoop.change-distro (exit_code=0)
• 12:24 jbond42: route ns0.wikimedia.org to codfw for reboot
• 12:20 moritzm: installing xen security updates (client-side tools/libs)
• 12:19 jbond42: re-enable puppet fleet
• 12:07 jbond42: disable puppet fleet wide to reboot puppetdb's
• 12:07 jbond42: disable puppet ro reboot puppetdb's
• 12:01 jforrester@deploy1001: rebuilt and synchronized wikiversions files: group0 wikis to 1.35.0-wmf.41
• 11:36 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1130 for query plan checks T238966 ', diff saved to https://phabricator.wikimedia.org/P11898 and previous config saved to /var/cache/conftool/dbconfig/20200714-113612-marostegui.json
• 11:35 _joe_: restart pybal on lvs2009 T257887
• 11:31 _joe_: restart pybal on lvs2010 T257887
• 11:25 _joe_: restart pybal on lvs1015 T257887
• 11:22 _joe_: restart pybal on lvs1016
• 11:15 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'wikifeeds' for release 'production' .
• 11:03 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'wikifeeds' for release 'production' .
• 10:59 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' .
• 10:56 volans@cumin1001: conftool action : set/pooled=inactive; selector: name=wtp2005.codfw.wmnet
• 10:52 volans: powerdown wtp2005, hardware issue - T257903
• 10:47 volans@cumin1001: conftool action : set/pooled=no; selector: name=wtp2005.codfw.wmnet
• 10:45 jiji@cumin1001: conftool action : set/pooled=no; selector: name=wtp2005.codfw.wmnet,service=parsoid-php
• 10:45 jiji@cumin1001: conftool action : set/pooled=no; selector: name=wtp2005.codfw.wmnet,service=parsoid
• 10:45 effie: depool wtp2005
• 10:42 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 10:42 aborrero@cumin1001: START - Cookbook sre.hosts.downtime
• 10:39 elukey@cumin1001: START - Cookbook sre.hadoop.change-distro
• 10:39 elukey@cumin1001: END (PASS) - Cookbook sre.hadoop.stop-cluster (exit_code=0)
• 10:32 elukey@cumin1001: START - Cookbook sre.hadoop.stop-cluster
• 10:18 oblivian@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'termbox' for release 'production' .
• 10:14 James_F: Running AbuseFilter's updateVarDumps for group1 T246539
• 10:13 oblivian@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'termbox' for release 'production' .
• 10:10 oblivian@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'termbox' for release 'staging' .
• 10:10 oblivian@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'termbox' for release 'test' .
• 09:44 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1112', diff saved to https://phabricator.wikimedia.org/P11897 and previous config saved to /var/cache/conftool/dbconfig/20200714-094449-marostegui.json
• 09:43 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1075', diff saved to https://phabricator.wikimedia.org/P11896 and previous config saved to /var/cache/conftool/dbconfig/20200714-094354-marostegui.json
• 09:39 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: ExtensionDistributor: Add REL1_35 as a candidate release (duration: 01m 06s)
• 09:05 jforrester@deploy1001: Finished scap: Re-re-start full scap to push out wmf.41 and switch testwikis to it T256669 (duration: 51m 41s)
• 08:40 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1135 for PDU upgrade T257871', diff saved to https://phabricator.wikimedia.org/P11895 and previous config saved to /var/cache/conftool/dbconfig/20200714-084033-marostegui.json
• 08:30 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:30 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 08:30 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:30 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 08:30 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:30 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 08:13 jforrester@deploy1001: Started scap: Re-re-start full scap to push out wmf.41 and switch testwikis to it T256669
• 08:05 akosiaris: restart pybal on lvs2009
• 08:03 _joe_: restart pybal on lvs1016
• 08:02 akosiaris: restart pybal on lvs2007
• 08:01 akosiaris@cumin1001: conftool action : set/pooled=inactive; selector: name=restbase2009.codfw.wmnet
• 08:00 _joe_: restart pybal on lvs1015
• 08:00 akosiaris: restart pybal on lvs2010 after merging https://gerrit.wikimedia.org/r/612487
• 07:52 jforrester@deploy1001: sync aborted: Re-start full scap to push out wmf.41 and switch testwikis to it T256669 (duration: 02m 14s)
• 07:50 jforrester@deploy1001: Started scap: Re-start full scap to push out wmf.41 and switch testwikis to it T256669
• 07:48 oblivian@deploy1001: Synchronized wmf-config/InitialiseSettings.php: revert forcehttps in an attempt to fix T257887 (duration: 01m 06s)
• 07:32 oblivian@deploy1001: sync-file aborted: revert forcehttps in an attempt to fix T257887 (duration: 00m 20s)
• 07:31 oblivian@deploy1001: Scap failed!: 7/9 canaries failed their endpoint checks(http://en.wikipedia.org)
• 07:27 moritzm: installing libtasn1-6 security updates
• 07:12 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1075', diff saved to https://phabricator.wikimedia.org/P11894 and previous config saved to /var/cache/conftool/dbconfig/20200714-071233-marostegui.json
• 07:04 marostegui: Drop gerrit, gerritro, gerrittest users from m2 databases - T255715
• 06:58 marostegui: Stop mysql on db1131 for HW maintenance
• 06:56 oblivian@deploy2001: helmfile [EQIAD] Ran 'sync' command on namespace 'wikifeeds' for release 'production' .
• 06:54 jforrester@deploy1001: scap failed: RuntimeError Scap failed!: 9/9 canaries failed their endpoint checks(http://en.wikipedia.org) (duration: 24m 59s)
• 06:54 jforrester@deploy1001: Scap failed!: 9/9 canaries failed their endpoint checks(http://en.wikipedia.org)
• 06:53 oblivian@deploy2001: helmfile [EQIAD] Ran 'sync' command on namespace 'wikifeeds' for release 'production' .
• 06:53 marostegui: Deploy MCR schema change on s5 primary master T238966
• 06:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1078', diff saved to https://phabricator.wikimedia.org/P11893 and previous config saved to /var/cache/conftool/dbconfig/20200714-065229-marostegui.json
• 06:29 jforrester@deploy1001: Started scap: testwikis wikis to 1.35.0-wmf.41
• 05:15 marostegui@cumin1001: dbctl commit (dc=all): 'Decrease a bit db1088 load', diff saved to https://phabricator.wikimedia.org/P11891 and previous config saved to /var/cache/conftool/dbconfig/20200714-051551-marostegui.json
• 05:09 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1131 for HW maintenance', diff saved to https://phabricator.wikimedia.org/P11890 and previous config saved to /var/cache/conftool/dbconfig/20200714-050931-marostegui.json
• 05:09 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1093 from api', diff saved to https://phabricator.wikimedia.org/P11889 and previous config saved to /var/cache/conftool/dbconfig/20200714-050912-marostegui.json
• 05:01 marostegui@cumin1001: dbctl commit (dc=all): 'Promote db1093 to s6 master and remove read-only from s6 T257253', diff saved to https://phabricator.wikimedia.org/P11888 and previous config saved to /var/cache/conftool/dbconfig/20200714-050157-marostegui.json
• 05:00 marostegui@cumin1001: dbctl commit (dc=all): 'Set s6 as read-only for maintenance T257253', diff saved to https://phabricator.wikimedia.org/P11887 and previous config saved to /var/cache/conftool/dbconfig/20200714-050039-marostegui.json
• 05:00 marostegui: Starting s6 failover from db1131 to db1093 - T257253
• 04:59 James_F: 1.35.0-wmf.41 branched at 7d04152
• 04:39 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1078', diff saved to https://phabricator.wikimedia.org/P11886 and previous config saved to /var/cache/conftool/dbconfig/20200714-043907-marostegui.json
• 04:15 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1093 in preparation for failover', diff saved to https://phabricator.wikimedia.org/P11885 and previous config saved to /var/cache/conftool/dbconfig/20200714-041548-marostegui.json
• 04:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1130', diff saved to https://phabricator.wikimedia.org/P11884 and previous config saved to /var/cache/conftool/dbconfig/20200714-041440-marostegui.json
• 01:23 ryankemper: Started long-running Elasticsearch reindex of `eqiad`, `codfw`, and `cloudelastic`. tmux session `reindex` under `ryankemper` on `mwmaint1002`
• 01:20 cdanis: ❌cdanis@lvs1015.eqiad.wmnet ~ 🕤🍺 sudo systemctl restart pybal.service
• 01:15 cdanis: ✔️ cdanis@lvs1016.eqiad.wmnet ~ 🕘🍺 sudo systemctl restart pybal.service
• 01:14 cdanis: ✔️ cdanis@lvs2009.codfw.wmnet ~ 🕘🍺 sudo systemctl restart pybal.service
• 01:01 cdanis: ✔️ cdanis@lvs2010.codfw.wmnet ~ 🕘🍺 sudo systemctl restart pybal.service

2020-07-13

• 23:06 mutante: releases* delete /usr/local/sbin/sync-* scripts created by rsync::quickdatacopy and let puppet recreate the ones still needed
• 22:27 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: I80ca62643f5c (duration: 00m 58s)
• 20:12 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@1edde21]: airflow: ship_to_es: Implement multi-index understanding (duration: 00m 29s)
• 20:12 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@1edde21]: airflow: ship_to_es: Implement multi-index understanding
• 20:03 mutante: rsynced reprepro data from releases1001 to releases1002, releases2002
• 19:50 eileen: disable target smart job process-control config revision is b00e7680ca
• 19:48 milimetric@deploy1001: Finished deploy [analytics/refinery@de0a1f1] (thin): Regular analytics weekly train THIN [analytics/refinery@de0a1f1] (duration: 00m 07s)
• 19:47 milimetric@deploy1001: Started deploy [analytics/refinery@de0a1f1] (thin): Regular analytics weekly train THIN [analytics/refinery@de0a1f1]
• 19:47 milimetric@deploy1001: Finished deploy [analytics/refinery@de0a1f1]: Regular analytics weekly train [analytics/refinery@de0a1f1] (duration: 06m 41s)
• 19:41 milimetric@deploy1001: Started deploy [analytics/refinery@de0a1f1]: Regular analytics weekly train [analytics/refinery@de0a1f1]
• 19:39 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
• 19:33 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: I1a1212 (duration: 00m 57s)
• 18:53 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: T248343 Don't use the 'zeroconf' configuration for VisualEditor (duration: 00m 55s)
• 18:43 dcausse: BACON done
• 18:40 dcausse@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T257745: Add rollbacker to elwiki (duration: 00m 56s)
• 18:26 dcausse@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T250810: Set proper language code for some wikis (duration: 00m 56s)
• 18:18 dcausse@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T256928: Scale largest shards to be closer to 30GB (duration: 00m 56s)
• 16:17 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 16:17 aborrero@cumin1001: START - Cookbook sre.hosts.downtime
• 15:56 ladsgroup@deploy1001: Synchronized wmf-config/Wikibase.php: Load WikibaseClient using extension registration in beta (T257435) (duration: 00m 55s)
• 15:52 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1130', diff saved to https://phabricator.wikimedia.org/P11882 and previous config saved to /var/cache/conftool/dbconfig/20200713-155240-marostegui.json
• 15:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1144:3315', diff saved to https://phabricator.wikimedia.org/P11881 and previous config saved to /var/cache/conftool/dbconfig/20200713-154847-marostegui.json
• 15:39 mholloway-shell@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'proton' for release 'production' .
• 15:35 mholloway-shell@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'proton' for release 'production' .
• 15:30 mholloway-shell@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'proton' for release 'production' .
• 14:50 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Stop setting DiscussionToolsEnableVisual, default value (duration: 00m 57s)
• 14:17 moritzm: removing lilypond from production T257066
• 13:36 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1144:3315', diff saved to https://phabricator.wikimedia.org/P11880 and previous config saved to /var/cache/conftool/dbconfig/20200713-133604-marostegui.json
• 13:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1082', diff saved to https://phabricator.wikimedia.org/P11879 and previous config saved to /var/cache/conftool/dbconfig/20200713-133535-marostegui.json
• 13:05 kormat@cumin1001: dbctl commit (dc=all): 'Fully repool es1022, and set es1020 to zero weight T257284', diff saved to https://phabricator.wikimedia.org/P11878 and previous config saved to /var/cache/conftool/dbconfig/20200713-130532-kormat.json
• 12:08 kormat@cumin1001: dbctl commit (dc=all): 'Start repooling es1022 after reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11873 and previous config saved to /var/cache/conftool/dbconfig/20200713-120818-kormat.json
• 11:49 Urbanecm: Password reset for User:Alert5 (T257806)
• 11:44 akosiaris: repool ganeti1007 T244530. Start emptying ganeti1008
• 11:08 Urbanecm: EU B&C done
• 11:06 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 896c042: Enable SandboxLink extension in trwiki (T256782) (duration: 00m 56s)
• 10:44 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (612175) (duration: 00m 56s)
• 10:43 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (612175) (duration: 00m 56s)
• 09:42 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:39 kormat@cumin1001: START - Cookbook sre.hosts.downtime
• 08:58 ema: cp: rolling ats-backend-restart to apply SyslogIdentifier changes -> https://gerrit.wikimedia.org/r/c/operations/puppet/+/611311
• 08:57 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: T248343 Explicitly set visualeditor-enable to 0 when non-default (duration: 00m 57s)
• 08:44 kormat@cumin1001: dbctl commit (dc=all): 'Depool es1022 for reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11871 and previous config saved to /var/cache/conftool/dbconfig/20200713-084449-kormat.json
• 08:39 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1093', diff saved to https://phabricator.wikimedia.org/P11870 and previous config saved to /var/cache/conftool/dbconfig/20200713-083902-marostegui.json
• 08:34 kormat@cumin1001: dbctl commit (dc=all): 'Add weight to es1020, reduce weight on es1022 T257284', diff saved to https://phabricator.wikimedia.org/P11869 and previous config saved to /var/cache/conftool/dbconfig/20200713-083414-kormat.json
• 08:20 kormat: reimaging es1022 T257284
• 06:54 marostegui@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1)
• 06:53 marostegui@cumin1001: START - Cookbook sre.hosts.decommission
• 06:52 marostegui@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1)
• 06:52 marostegui@cumin1001: START - Cookbook sre.hosts.decommission
• 06:51 marostegui@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99)
• 06:50 marostegui@cumin1001: START - Cookbook sre.hosts.decommission
• 06:16 marostegui: Reverse gerrit password on m2 master - T255715
• 06:04 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1093', diff saved to https://phabricator.wikimedia.org/P11868 and previous config saved to /var/cache/conftool/dbconfig/20200713-060410-marostegui.json
• 05:54 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1093', diff saved to https://phabricator.wikimedia.org/P11867 and previous config saved to /var/cache/conftool/dbconfig/20200713-055422-marostegui.json
• 05:48 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1093 for upgrade', diff saved to https://phabricator.wikimedia.org/P11866 and previous config saved to /var/cache/conftool/dbconfig/20200713-054840-marostegui.json
• 05:34 marostegui: Deploy schema change on s3 codfw master, lag will appear on codfw T253276
• 05:30 marostegui: Stop replication on db1082 for schema change and triggers removal T238966
• 05:29 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1082', diff saved to https://phabricator.wikimedia.org/P11865 and previous config saved to /var/cache/conftool/dbconfig/20200713-052928-marostegui.json
• 05:14 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1119 for innodb compression', diff saved to https://phabricator.wikimedia.org/P11864 and previous config saved to /var/cache/conftool/dbconfig/20200713-051428-marostegui.json

2020-07-11

• 19:16 qchris: Restarting Gerrit on gerrit1001 to switch to new gerrit.war and zuul plugin
• 19:16 qchris@deploy1001: Finished deploy [gerrit/gerrit@a71a0df]: Gerrit to v3.2.2-138-g230805407f and zuul plugin to master-12-ge51d7e8 on gerrit1001 (duration: 00m 07s)
• 19:15 qchris@deploy1001: Started deploy [gerrit/gerrit@a71a0df]: Gerrit to v3.2.2-138-g230805407f and zuul plugin to master-12-ge51d7e8 on gerrit1001
• 19:08 qchris: Restarting Gerrit on gerrit2001 to switch to new gerrit.war and zuul plugin
• 18:55 qchris@deploy1001: Finished deploy [gerrit/gerrit@a71a0df]: Gerrit to v3.2.2-138-g230805407f and zuul plugin to master-12-ge51d7e8 on gerrit2001 (duration: 00m 10s)
• 18:55 qchris@deploy1001: Started deploy [gerrit/gerrit@a71a0df]: Gerrit to v3.2.2-138-g230805407f and zuul plugin to master-12-ge51d7e8 on gerrit2001

2020-07-10

• 21:52 ryankemper: Started long-running reindex of Elasticsearch indices in `eqiad`, `codfw`, and `dewiki` on `mwmaint1002` under tmux session `reindex` for user `ryankemper`
• 20:26 jgleeson: updated fundraising-tools from 08ba1f6177 to f8e424fe32
• 19:02 mutante: removing firewall hole for gerrit -> mysql servers on dbproxy servers for misc db's
• 18:44 mutante: kubernetes1004 - started nagios-nrpe-server
• 17:57 ebernhardson: change loginwiki password for Cindy-the-browser-test-bot, no email account was associated to allow for normal reset.
• 17:05 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: I63fcea7737 (duration: 00m 57s)
• 16:16 elukey@cumin1001: END (FAIL) - Cookbook sre.hadoop.change-distro (exit_code=99)
• 15:57 milimetric@deploy1001: Finished deploy [analytics/refinery@4d40145] (thin): Update EventLogging refine whitelist (THIN) (duration: 00m 08s)
• 15:56 milimetric@deploy1001: Started deploy [analytics/refinery@4d40145] (thin): Update EventLogging refine whitelist (THIN)
• 15:44 milimetric@deploy1001: Finished deploy [analytics/refinery@4d40145]: Update EventLogging refine whitelist (duration: 15m 17s)
• 15:30 elukey@cumin1001: START - Cookbook sre.hadoop.change-distro
• 15:29 elukey@cumin1001: END (PASS) - Cookbook sre.hadoop.stop-cluster (exit_code=0)
• 15:29 milimetric@deploy1001: Started deploy [analytics/refinery@4d40145]: Update EventLogging refine whitelist
• 15:19 elukey@cumin1001: START - Cookbook sre.hadoop.stop-cluster
• 15:03 elukey@cumin1001: END (PASS) - Cookbook sre.hadoop.change-distro (exit_code=0)
• 14:39 elukey@cumin1001: START - Cookbook sre.hadoop.change-distro
• 14:37 elukey@cumin1001: END (PASS) - Cookbook sre.hadoop.stop-cluster (exit_code=0)
• 14:30 elukey@cumin1001: START - Cookbook sre.hadoop.stop-cluster
• 13:41 godog: bounce ms-be1037, not quite responsive
• 12:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1110', diff saved to https://phabricator.wikimedia.org/P11860 and previous config saved to /var/cache/conftool/dbconfig/20200710-123604-marostegui.json
• 12:20 reedy@deploy1001: Synchronized php-1.35.0-wmf.40/extensions/Score/: Make Score errors use a specific css class (duration: 00m 58s)
• 10:21 kormat@cumin1001: dbctl commit (dc=all): 'Finish repooling es1021, and remove weight from es1010 T257284', diff saved to https://phabricator.wikimedia.org/P11859 and previous config saved to /var/cache/conftool/dbconfig/20200710-102147-kormat.json
• 09:49 kormat@cumin1001: dbctl commit (dc=all): 'Start repooling es1021 after reimage @ 50% T257284', diff saved to https://phabricator.wikimedia.org/P11858 and previous config saved to /var/cache/conftool/dbconfig/20200710-094954-kormat.json
• 09:04 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:02 kormat@cumin1001: START - Cookbook sre.hosts.downtime
• 08:51 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1110', diff saved to https://phabricator.wikimedia.org/P11857 and previous config saved to /var/cache/conftool/dbconfig/20200710-085157-marostegui.json
• 08:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1106', diff saved to https://phabricator.wikimedia.org/P11856 and previous config saved to /var/cache/conftool/dbconfig/20200710-085112-marostegui.json
• 08:50 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1107', diff saved to https://phabricator.wikimedia.org/P11855 and previous config saved to /var/cache/conftool/dbconfig/20200710-085040-marostegui.json
• 08:23 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1106', diff saved to https://phabricator.wikimedia.org/P11853 and previous config saved to /var/cache/conftool/dbconfig/20200710-082346-marostegui.json
• 08:23 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1107', diff saved to https://phabricator.wikimedia.org/P11852 and previous config saved to /var/cache/conftool/dbconfig/20200710-082329-marostegui.json
• 08:22 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:22 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 08:22 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:22 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 08:09 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1107', diff saved to https://phabricator.wikimedia.org/P11851 and previous config saved to /var/cache/conftool/dbconfig/20200710-080912-marostegui.json
• 08:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1119', diff saved to https://phabricator.wikimedia.org/P11850 and previous config saved to /var/cache/conftool/dbconfig/20200710-080854-marostegui.json
• 08:09 kormat@cumin1001: dbctl commit (dc=all): 'Depool es1021 for reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11849 and previous config saved to /var/cache/conftool/dbconfig/20200710-080843-kormat.json
• 08:01 kormat@cumin1001: dbctl commit (dc=all): 'Reset es2020/es2021 to correct weights after master switch T257284', diff saved to https://phabricator.wikimedia.org/P11848 and previous config saved to /var/cache/conftool/dbconfig/20200710-080133-kormat.json
• 08:00 moritzm: installing cron security updates on jessie (stretch/buster already fixed)
• 07:56 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1119', diff saved to https://phabricator.wikimedia.org/P11847 and previous config saved to /var/cache/conftool/dbconfig/20200710-075608-marostegui.json
• 07:55 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1107', diff saved to https://phabricator.wikimedia.org/P11846 and previous config saved to /var/cache/conftool/dbconfig/20200710-075500-marostegui.json
• 07:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1079', diff saved to https://phabricator.wikimedia.org/P11845 and previous config saved to /var/cache/conftool/dbconfig/20200710-075431-marostegui.json
• 07:44 kormat: reimaging es1021 to buster T257284
• 07:43 kormat@cumin1001: dbctl commit (dc=all): 'Add weight to es1020, reduce weight on es1021 T257284', diff saved to https://phabricator.wikimedia.org/P11844 and previous config saved to /var/cache/conftool/dbconfig/20200710-074326-kormat.json
• 07:41 jbond@deploy1001: Finished deploy [librenms/librenms@0a88d64]: redeplopy to [try and] fix php errors (duration: 00m 05s)
• 07:41 jbond@deploy1001: Started deploy [librenms/librenms@0a88d64]: redeplopy to [try and] fix php errors
• 07:32 moritzm: installing e2fsprogs security updates on jessie (stretch/buster already fixed)
• 07:15 akosiaris@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'proton' for release 'production' .
• 07:14 akosiaris@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'proton' for release 'production' .
• 07:13 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'proton' for release 'production' .
• 06:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1099:3311', diff saved to https://phabricator.wikimedia.org/P11843 and previous config saved to /var/cache/conftool/dbconfig/20200710-065751-marostegui.json
• 06:38 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1099:3311', diff saved to https://phabricator.wikimedia.org/P11841 and previous config saved to /var/cache/conftool/dbconfig/20200710-063818-marostegui.json
• 06:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1134', diff saved to https://phabricator.wikimedia.org/P11840 and previous config saved to /var/cache/conftool/dbconfig/20200710-063746-marostegui.json
• 06:35 marostegui: Compress InnoDB on db1124:3311 (Sanitarium - lag will appear on s1 on labsdb) - T254462
• 04:44 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1134', diff saved to https://phabricator.wikimedia.org/P11839 and previous config saved to /var/cache/conftool/dbconfig/20200710-044428-marostegui.json
• 01:44 mutante: LDAP - adding coka to wmde and nda (T257038)
• 00:47 Reedy: truncated labswiki.interwiki table (outdated and unnecessary)

2020-07-09

• 23:10 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: I2c2dea832 (duration: 00m 56s)
• 21:52 tgr: all sessions have been invalidated due to T256395
• 20:58 eileen: https://phabricator.wikimedia.org/T253152
• 19:16 herron: upgraded eqiad elk7 cluster from 7.4.2 to 7.8.0 T234854
• 19:05 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.40 refs T256668
• 18:51 elukey: update spark2 to 2.4.4-bin-hadoop2.6-3 for buster-wikimedia
• 18:44 mutante: stat1004, stat1006, stat1007 - upgrading git-review package from 1.25 to 1.27 so that it keeps working with new Gerrit 3.2 (T257609)
• 18:10 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 9f2557f: Updating config for Readers Web affinity quicksurvey (T246977) (duration: 01m 06s)
• 17:42 chaomodus: codfw frack management dns automation deployment complete T233183
• 17:37 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 17:36 James_F: Synchronized wmf-config/CommonSettings.php: ExtensionDistribution: Drop REL1_33, EOL'ed T256087
• 17:35 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 17:35 moritzm: rebooting moscovium for kernel update
• 17:33 chaomodus: deploying frack codfw management dns automation
• 17:32 crusnov@cumin2001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 17:31 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 17:28 crusnov@cumin2001: START - Cookbook sre.dns.netbox
• 17:27 moritzm: rebooting planet1002 (planet.wikimedia.org) for kernel update
• 17:27 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 17:10 krinkle@deploy1001: Synchronized wmf-config/: Ia2f5ed (duration: 01m 04s)
• 17:09 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Ia2f5ed (duration: 01m 05s)
• 15:32 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 15:29 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:29 papaul: replacing msw-b1,b2,b3 and b4
• 14:03 moritzm: installing libtirpc security updates
• 13:45 moritzm: installing gnutls28 security updates
• 13:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1089', diff saved to https://phabricator.wikimedia.org/P11831 and previous config saved to /var/cache/conftool/dbconfig/20200709-133134-marostegui.json
• 13:31 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 13:29 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 13:29 moritzm: rebooting puppetboard1001 (puppetboard.wikimedia.org) for kernel update
• 13:15 moritzm: installing ffmpeg security updates
• 13:11 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 13:10 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1089', diff saved to https://phabricator.wikimedia.org/P11830 and previous config saved to /var/cache/conftool/dbconfig/20200709-131039-marostegui.json
• 13:08 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 13:07 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 13:05 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 13:00 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 12:58 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 12:57 akosiaris@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'proton' for release 'production' .
• 12:57 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 12:56 akosiaris@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'proton' for release 'production' .
• 12:56 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'proton' for release 'production' .
• 12:54 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 12:54 moritzm: rebooting install* servers for kernel security update
• 12:43 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 12:40 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 12:40 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 12:38 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 12:38 moritzm: rebooting urldownloader1001/2001 for kernel update (failed over, these are now the inactive ones)
• 12:23 jmm@cumin2001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99)
• 12:22 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 12:22 moritzm: rebooting dbmonitor1001 / tendril.wikimedia.org for kernek update
• 12:11 XioNoX: enable asw2-b-eqiad:ae3 (to cloudsw1-c8) - T251632
• 11:56 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 11:54 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 11:52 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 11:50 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 11:50 moritzm: rebooting debmonitor1001 for kernel update
• 11:42 urbanecm@deploy1001: Synchronized php-1.35.0-wmf.40/extensions/Translate/tag/SpecialPageTranslation.php: 6541d3f: DeprecatablePropertyArray: Use MW_VERSION instead of array_key_exists (T257531) (duration: 01m 05s)
• 11:28 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 3a7c1c3: Rename namespace on kn.wikipedia.org (T255337) (duration: 01m 04s)
• 11:24 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 0a3c1f9: Add *.oireachtas.ie to the wgCopyUploadsDomains whitelist for commonswiki (T256543) (duration: 01m 04s)
• 11:19 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 11:17 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 11:10 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 11:10 aborrero@cumin1001: START - Cookbook sre.hosts.downtime
• 11:09 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: e6f442c: Enable Quicksurveys for Desktop Improvements Project (T246977) (duration: 01m 06s)
• 11:01 vgutierrez: restart ats-tls on cp1085
• 10:55 _joe_: restarting php7.2-fpm on mw1282, workers failing with sigill
• 10:54 _joe_: depool mw1282
• 10:54 mvolz@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'citoid' for release 'production' .
• 10:34 mvolz@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'citoid' for release 'production' .
• 10:23 _joe_: rolling restart the remaining restbases in eqiad, and all of codfw
• 10:22 mvolz@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'citoid' for release 'staging' .
• 10:09 _joe_: restarting restbase on rb1020-22
• 09:53 _joe_: restarting restbase on restbase1024,1023
• 09:36 _joe_: restarting restbase on rb1026,1027 to switch to proton on k8s
• 09:34 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:31 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 09:28 _joe_: restarting restbase on restbase1025 to pick up the switch to k8s of proton
• 09:27 godog: bounce thanos-compact on thanos-fe2001
• 09:07 elukey@cumin1001: END (PASS) - Cookbook sre.hadoop.change-distro (exit_code=0)
• 08:52 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1079', diff saved to https://phabricator.wikimedia.org/P11828 and previous config saved to /var/cache/conftool/dbconfig/20200709-085228-marostegui.json
• 08:44 marostegui: Stop haproxy on dbproxy1017 before upgrading to buster - T255408
• 08:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1136', diff saved to https://phabricator.wikimedia.org/P11827 and previous config saved to /var/cache/conftool/dbconfig/20200709-082355-marostegui.json
• 08:23 moritzm: imported osm2pgsql 0.96.0+ds-1~bpo9+1 to "main" component T256877
• 08:22 elukey@cumin1001: START - Cookbook sre.hadoop.change-distro
• 08:20 elukey@cumin1001: END (PASS) - Cookbook sre.hadoop.stop-cluster (exit_code=0)
• 08:13 elukey@cumin1001: START - Cookbook sre.hadoop.stop-cluster
• 08:11 XioNoX: disable igmp snooping on msw1-codfw
• 07:59 marostegui: Stop db1117:3322 to clone db1084, this will trigger haproxy alerts - T257540
• 07:57 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1136', diff saved to https://phabricator.wikimedia.org/P11825 and previous config saved to /var/cache/conftool/dbconfig/20200709-075749-marostegui.json
• 07:19 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 07:15 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 06:52 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 06:49 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 05:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1101:3317', diff saved to https://phabricator.wikimedia.org/P11824 and previous config saved to /var/cache/conftool/dbconfig/20200709-053905-marostegui.json
• 05:32 marostegui@cumin1001: dbctl commit (dc=all): 'Remove db1084 from dbctl', diff saved to https://phabricator.wikimedia.org/P11823 and previous config saved to /var/cache/conftool/dbconfig/20200709-053206-marostegui.json
• 05:18 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1084', diff saved to https://phabricator.wikimedia.org/P11822 and previous config saved to /var/cache/conftool/dbconfig/20200709-051826-marostegui.json
• 05:13 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1101:3317', diff saved to https://phabricator.wikimedia.org/P11821 and previous config saved to /var/cache/conftool/dbconfig/20200709-051355-marostegui.json
• 05:11 marostegui: Remove revision triggers from db2093:3315 T238966
• 05:10 marostegui: Deploy schema change on s5 codfw, lag will be generated - T238966
• 01:43 tzatziki: reset email for GseSro
• 00:58 cdanis: ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕘🍺 sudo cumin A:cp 'enable-puppet "cdanis deploying I6c1b646e T256395"'
• 00:49 cdanis: ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕘🍺 sudo cumin A:cp 'disable-puppet "cdanis deploying I6c1b646e T256395"'

2020-07-08

• 21:56 mutante: deleting files from releases2001 that are not existing on releases1001 to make them mirrors. rsync with --delete and the command from quickdatacopy class (T247652)
• 21:55 mutante: rsyncing releases files from releases1001 to releases2002 and releases1002. deleting files from releases2002 not existing on releases1002 to make them mirrors ( T247652_
• 20:59 cstone: civicrm revision changed from d73ee2e73f to 8b09c87ce2,
• 20:27 Amir1: end of foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https (T256012)
• 20:08 Amir1_: start of foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https (T256012)
• 19:18 twentyafterfour@deploy1001: Synchronized php: group1 wikis to 1.35.0-wmf.40 refs T256668 (duration: 01m 04s)
• 19:17 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.40 refs T256668
• 18:58 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 091442c: Add *.nga.gov to the wgCopyUploadsDomains allowlist of Wikimedia Commons (T256518) (duration: 01m 04s)
• 18:55 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 2e5943d: Add scan-bugs.org to $wgCopyUploadsDomains (T256569) (duration: 01m 04s)
• 18:46 urbanecm@deploy1001: Synchronized static/images/project-logos/: f42cdf2: Change bnwiki logo (T255328) (duration: 01m 04s)
• 18:27 ppchelko@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Cleanup: remove temporary wmgDisableHTCP variable gerrit:607596 T250781 IS.php (duration: 01m 01s)
• 18:20 ppchelko@deploy1001: Synchronized wmf-config/CommonSettings.php: Disable HTCP purging everywhere gerrit:607593 T250781 CS.php (duration: 01m 03s)
• 18:18 ppchelko@deploy1001: Synchronized wmf-config/wikitech.php: Disable HTCP purging everywhere gerrit:607593 T250781 wikitech.php (duration: 01m 04s)
• 18:17 ppchelko@deploy1001: Synchronized wmf-config/reverse-proxy.php: Disable HTCP purging everywhere gerrit:607593 T250781 reverse-proxy.php (duration: 01m 04s)
• 18:11 ppchelko@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add wgEventServiceDefault to refactor EventBus event stream config gerrit:610160 T229863, IS.php (duration: 01m 03s)
• 18:04 ppchelko@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add wgEventServiceDefault to refactor EventBus event stream config gerrit:610160 T229863 (duration: 01m 04s)
• 17:34 elukey@cumin1001: END (FAIL) - Cookbook sre.hadoop.change-distro (exit_code=99)
• 17:16 elukey@cumin1001: START - Cookbook sre.hadoop.change-distro
• 17:16 elukey@cumin1001: END (PASS) - Cookbook sre.hadoop.stop-cluster (exit_code=0)
• 17:08 elukey@cumin1001: START - Cookbook sre.hadoop.stop-cluster
• 16:57 _joe_: restarting restbase across the fleet to transition to using envoy
• 16:40 _joe_: restarting restbase on restbase2010 to route calls to mediawiki, parsoid via envoy
• 16:40 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 16:37 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
• 16:32 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 16:27 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
• 15:22 jgleeson: updated fundraising-tools from a244e0e85f --> f5b8528214
• 15:16 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 15:12 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 15:12 moritzm: rebooting people1002 (people.wikimedia.org) for kernel security update
• 15:06 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 15:04 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:48 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 14:46 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:46 moritzm: installing isc-dhcp security updates
• 14:31 elukey@cumin1001: END (FAIL) - Cookbook sre.hadoop.change-distro (exit_code=99)
• 14:31 moritzm: installing gdk-pixbuf security updates
• 14:26 _joe_: repooling mw1346
• 14:24 _joe_: php7adm /opcache-free on mw1346
• 14:15 jbond42: switch icinga authentication to CAS SSO
• 14:12 _joe_: depooling mw1346
• 14:12 elukey@cumin1001: START - Cookbook sre.hadoop.change-distro
• 14:11 elukey@cumin1001: END (PASS) - Cookbook sre.hadoop.stop-cluster (exit_code=0)
• 14:06 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 14:04 elukey@cumin1001: START - Cookbook sre.hadoop.stop-cluster
• 14:04 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:04 moritzm: rebooting idp-test1001 for kernel update
• 13:59 elukey@cumin1001: END (ERROR) - Cookbook sre.hadoop.stop-cluster (exit_code=97)
• 13:59 elukey@cumin1001: START - Cookbook sre.hadoop.stop-cluster
• 13:39 elukey@cumin1001: END (PASS) - Cookbook sre.hadoop.change-distro (exit_code=0)
• 13:31 jynus: replacing ssh key for ci_docroot at deploy1001
• 13:31 moritzm: imported git 2.20.1-2+deb10u3~wmf1 for stretch-wikimedia component/git T257308
• 13:10 elukey@cumin1001: START - Cookbook sre.hadoop.change-distro
• 13:07 elukey@cumin1001: END (PASS) - Cookbook sre.hadoop.stop-cluster (exit_code=0)
• 13:00 elukey@cumin1001: START - Cookbook sre.hadoop.stop-cluster
• 12:41 marostegui: Deploy schema change on s7 codfw, lag is expected
• 12:17 xionox-tmp: rollout less frequent option-refresh-rate - T240658
• 12:01 xionox-tmp: renumber eqiad NTT link - T254877
• 11:42 awight: EU BACON complete
• 11:41 awight@deploy1001: Synchronized wmf-config/InitialiseSettings.php: BACON: Undeploy graphoid for phase 1 wikis (T257402) (duration: 01m 03s)
• 11:31 awight@deploy1001: Synchronized wmf-config/InitialiseSettings.php: BACON: Add nature.com to commonswiki wgCopyUploadDomains (T254342) (duration: 01m 03s)
• 11:29 moritzm: installing freetype security updates
• 11:26 awight@deploy1001: Synchronized wmf-config/InitialiseSettings.php: BACON: [hiwikibooks] Translate sitename for hi.wikibooks (T256587) (duration: 01m 03s)
• 11:19 awight@deploy1001: Synchronized wmf-config/InitialiseSettings.php: BACON: [arwiki] Grant 'patrolmarks' to all (T257106) (duration: 01m 04s)
• 11:18 akosiaris@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'proton' for release 'production' .
• 11:18 moritzm: installing libgcrypt20 security updates
• 11:16 akosiaris@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'proton' for release 'production' .
• 11:07 awight@deploy1001: Synchronized wmf-config/InitialiseSettings.php: BACON: Provision WMDE TeWü survey for prototype 1 (T257306), file 2/2 (duration: 01m 03s)
• 11:06 awight@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: BACON: Provision WMDE TeWü survey for prototype 1 (T257306), file 1/2 (duration: 01m 16s)
• 11:05 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'proton' for release 'production' .
• 11:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1121', diff saved to https://phabricator.wikimedia.org/P11818 and previous config saved to /var/cache/conftool/dbconfig/20200708-110546-marostegui.json
• 10:51 akosiaris@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' .
• 10:51 akosiaris@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' .
• 10:50 akosiaris: apply calico egress policies
• 10:50 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' .
• 10:45 moritzm: installing json-c security updates
• 10:25 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1121', diff saved to https://phabricator.wikimedia.org/P11817 and previous config saved to /var/cache/conftool/dbconfig/20200708-102553-marostegui.json
• 10:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1084', diff saved to https://phabricator.wikimedia.org/P11816 and previous config saved to /var/cache/conftool/dbconfig/20200708-102500-marostegui.json
• 10:13 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1084', diff saved to https://phabricator.wikimedia.org/P11815 and previous config saved to /var/cache/conftool/dbconfig/20200708-101313-marostegui.json
• 09:58 kormat@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:57 elukey@cumin1001: END (PASS) - Cookbook sre.hadoop.stop-cluster (exit_code=0)
• 09:56 kormat@cumin2001: START - Cookbook sre.hosts.downtime
• 09:50 elukey@cumin1001: START - Cookbook sre.hadoop.stop-cluster
• 09:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1149', diff saved to https://phabricator.wikimedia.org/P11814 and previous config saved to /var/cache/conftool/dbconfig/20200708-094539-marostegui.json
• 09:26 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1149', diff saved to https://phabricator.wikimedia.org/P11813 and previous config saved to /var/cache/conftool/dbconfig/20200708-092650-marostegui.json
• 09:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1148', diff saved to https://phabricator.wikimedia.org/P11812 and previous config saved to /var/cache/conftool/dbconfig/20200708-092627-marostegui.json
• 09:24 xionox-tmp: renumber eqord NTT link - T254877
• 09:18 xionox-tmp: remove eqord-eqiad tunnel - T254877
• 09:15 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1148', diff saved to https://phabricator.wikimedia.org/P11811 and previous config saved to /var/cache/conftool/dbconfig/20200708-091557-marostegui.json
• 08:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1147', diff saved to https://phabricator.wikimedia.org/P11810 and previous config saved to /var/cache/conftool/dbconfig/20200708-085745-marostegui.json
• 08:55 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
• 08:54 marostegui@cumin1001: START - Cookbook sre.hosts.decommission
• 08:54 marostegui@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99)
• 08:54 marostegui@cumin1001: START - Cookbook sre.hosts.decommission
• 08:50 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1147', diff saved to https://phabricator.wikimedia.org/P11809 and previous config saved to /var/cache/conftool/dbconfig/20200708-085024-marostegui.json
• 08:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1074', diff saved to https://phabricator.wikimedia.org/P11808 and previous config saved to /var/cache/conftool/dbconfig/20200708-084227-marostegui.json
• 08:40 moritzm: upgrading docker on remaining buster hosts
• 08:38 hashar: Upgraded docker.io on contint1001 and contint2001
• 08:28 marostegui: Remove dbproxy1003 grants from misc hosts T231280
• 08:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1146:3314', diff saved to https://phabricator.wikimedia.org/P11807 and previous config saved to /var/cache/conftool/dbconfig/20200708-082624-marostegui.json
• 08:20 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1146:3314', diff saved to https://phabricator.wikimedia.org/P11806 and previous config saved to /var/cache/conftool/dbconfig/20200708-082040-marostegui.json
• 08:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1144:3314', diff saved to https://phabricator.wikimedia.org/P11805 and previous config saved to /var/cache/conftool/dbconfig/20200708-081647-marostegui.json
• 08:15 kormat@cumin1001: dbctl commit (dc=all): 'Depool es2020 for reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11804 and previous config saved to /var/cache/conftool/dbconfig/20200708-081519-kormat.json
• 08:00 marostegui: Failover m1 from db1097 to db1080 - T256717
• 07:57 kormat: reimaging es2020 to buster T257284
• 07:51 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 07:49 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1144:3314', diff saved to https://phabricator.wikimedia.org/P11803 and previous config saved to /var/cache/conftool/dbconfig/20200708-074939-marostegui.json
• 07:49 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 07:48 jynus: stop bacula-director on backup1001 in preparation for m1 switchover T256717
• 07:47 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 07:47 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 07:47 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 07:47 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 07:47 akosiaris@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 07:47 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 07:45 moritzm: installing PHP 7.3 security updates
• 07:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1143', diff saved to https://phabricator.wikimedia.org/P11802 and previous config saved to /var/cache/conftool/dbconfig/20200708-073548-marostegui.json
• 07:30 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1143', diff saved to https://phabricator.wikimedia.org/P11801 and previous config saved to /var/cache/conftool/dbconfig/20200708-073037-marostegui.json
• 07:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1142', diff saved to https://phabricator.wikimedia.org/P11800 and previous config saved to /var/cache/conftool/dbconfig/20200708-073011-marostegui.json
• 07:24 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1142', diff saved to https://phabricator.wikimedia.org/P11799 and previous config saved to /var/cache/conftool/dbconfig/20200708-072431-marostegui.json
• 07:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1141', diff saved to https://phabricator.wikimedia.org/P11798 and previous config saved to /var/cache/conftool/dbconfig/20200708-070921-marostegui.json
• 07:04 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1141', diff saved to https://phabricator.wikimedia.org/P11797 and previous config saved to /var/cache/conftool/dbconfig/20200708-070432-marostegui.json
• 07:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1138', diff saved to https://phabricator.wikimedia.org/P11796 and previous config saved to /var/cache/conftool/dbconfig/20200708-070403-marostegui.json
• 06:47 marostegui: start topology changes on m1 T256717
• 06:43 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1138', diff saved to https://phabricator.wikimedia.org/P11795 and previous config saved to /var/cache/conftool/dbconfig/20200708-064354-marostegui.json
• 06:36 marostegui: Deploy schema change on s2 primary master db1122 T238966
• 06:18 _joe_: rolling restart of restbase to pick up the proton url change
• 03:36 andrew@deploy1001: Finished deploy [horizon/deploy@505819d]: further fixes for proxy editing --bug 610130 (duration: 03m 44s)
• 03:32 andrew@deploy1001: Started deploy [horizon/deploy@505819d]: further fixes for proxy editing --bug 610130

2020-07-07

• 22:41 mutante: new Wikimedia Annual Report 2019 now available on annual.wikimedia.org
• 21:29 andrew@deploy1001: Finished deploy [horizon/deploy@fce8183]: further fixes for proxy editing --bug 610130 (duration: 03m 35s)
• 21:25 andrew@deploy1001: Started deploy [horizon/deploy@fce8183]: further fixes for proxy editing --bug 610130
• 21:10 andrew@deploy1001: Finished deploy [horizon/deploy@abcd051]: further fixes for proxy editing --bug 610130 (duration: 03m 26s)
• 21:07 andrew@deploy1001: Started deploy [horizon/deploy@abcd051]: further fixes for proxy editing --bug 610130
• 20:41 ppchelko@deploy1001: Finished deploy [restbase/deploy@05b8bd5]: Remove restbase2009, take 2 (duration: 09m 15s)
• 20:32 ppchelko@deploy1001: Started deploy [restbase/deploy@05b8bd5]: Remove restbase2009, take 2
• 20:32 ppchelko@deploy1001: Finished deploy [restbase/deploy@05b8bd5]: Remove restbase2009 (duration: 14m 28s)
• 20:24 mutante: kubernetes1003 - starting nagios-nrpe-server
• 20:23 mutante: kubernetes1001 - starting nagios-nrpe-server
• 20:17 ppchelko@deploy1001: Started deploy [restbase/deploy@05b8bd5]: Remove restbase2009
• 19:29 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
• 19:27 mutante: destroying VM gerrit1002 - decom cookbook
• 19:26 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
• 19:18 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group0 wikis to 1.35.0-wmf.40 refs T256668
• 19:04 mutante: contint2001 - move /var/lib/zuul/.ssh/known_hosts to root and run puppet to recreate it
• 18:38 andrew@deploy1001: Finished deploy [horizon/deploy@eaa056e]: fix for proxy editing --bug 610130 (duration: 03m 18s)
• 18:35 andrew@deploy1001: Started deploy [horizon/deploy@eaa056e]: fix for proxy editing --bug 610130
• 18:27 andrew@deploy1001: Finished deploy [horizon/deploy@a39e86c]: update proxy UI to support editing existing proxies (duration: 03m 26s)
• 18:23 andrew@deploy1001: Started deploy [horizon/deploy@a39e86c]: update proxy UI to support editing existing proxies
• 18:10 krinkle@deploy1001: Synchronized w/: remove untracked test cookie file (duration: 01m 04s)
• 18:08 krinkle@deploy1001: Synchronized php-1.35.0-wmf.40/includes/Revision/RevisionStore.php: I8f986daeab4 (duration: 01m 05s)
• 17:59 herron: imported (logstash|kibana|elasticsearch)-oss-7.8.0 into buster-wikimedia thirdparty/elastic78
• 17:54 hnowlan: finished removing restbase2009 from cassandra pool
• 17:06 hnowlan: removed restbase2009-b from cassandra pool, removing restbase2009-c
• 16:40 lucaswerkmeister-wmde@deploy1001: Synchronized php-1.35.0-wmf.40/extensions/Wikibase: Backport: Revert "Don’t load $wgWBClientSettings in WikibaseClient.php" (T257296) (duration: 01m 10s)
• 15:49 hnowlan: running nodetool removenode for restbase2009-a
• 15:38 hnowlan@deploy1001: Started restart [restbase/deploy@05b8bd5]: Restarting restbase after removal of restbase2009
• 15:27 elukey: root-tmux on cumin1001 - cumin 'c:profile::mediawiki::mcrouter_wancache' '/usr/local/sbin/restart-mcrouter' -b 2 -s 5 - roll restart of mw-mcrouter to pick up new settings - T255511
• 15:13 hnowlan@deploy1001: Started restart [restbase/deploy@05b8bd5]: Restarting restbase after removal of restbase2009
• 15:12 otto@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' .
• 15:12 otto@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-main' for release 'production' .
• 15:09 otto@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' .
• 15:09 otto@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' .
• 15:06 otto@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' .
• 15:04 otto@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' .
• 15:04 otto@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' .
• 15:02 otto@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-main' for release 'production' .
• 15:02 otto@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' .
• 15:01 otto@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' .
• 14:58 hashar@deploy1001: Finished deploy [integration/docroot@708d3eb]: Second deployment to ensure everything works fine. Thank you jynus (duration: 00m 04s)
• 14:58 hashar@deploy1001: Started deploy [integration/docroot@708d3eb]: Second deployment to ensure everything works fine. Thank you jynus
• 14:53 _joe_: restarted restbase on restbase2022 after removing restbase2009 from the cassandra seeds
• 14:48 otto@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' .
• 14:47 otto@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' .
• 14:38 otto@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' .
• 14:38 otto@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' .
• 14:31 otto@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-main' for release 'production' .
• 14:31 otto@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' .
• 14:30 papaul: replacing msw-a5,a6,a7 and a8
• 14:30 otto@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' .
• 14:24 otto@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' .
• 14:24 otto@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' .
• 14:20 otto@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' .
• 14:20 otto@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' .
• 14:16 hashar@deploy1001: Finished deploy [integration/docroot@708d3eb]: (no justification provided) (duration: 00m 09s)
• 14:16 hashar@deploy1001: Started deploy [integration/docroot@708d3eb]: (no justification provided)
• 13:38 _joe_: rolling restart of restbase to pick up using envoy
• 13:31 otto@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' .
• 13:31 otto@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' .
• 13:29 XioNoX: cr2-eqiad> request vmhost snapshot routing-engine both - T257153
• 13:24 XioNoX: cr1-eqiad> request vmhost snapshot routing-engine both - T257153
• 13:15 kormat@cumin1001: dbctl commit (dc=all): 'Promote es2021 to es4 master T257284', diff saved to https://phabricator.wikimedia.org/P11789 and previous config saved to /var/cache/conftool/dbconfig/20200707-131524-kormat.json
• 12:59 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 12:57 elukey@cumin1001: START - Cookbook sre.hosts.downtime
• 12:44 kormat: starting (codfw) es5 failover from es2020 to es2021 T257284
• 12:30 kormat@cumin1001: dbctl commit (dc=all): 'Set es2021 to weight 50 T257284', diff saved to https://phabricator.wikimedia.org/P11787 and previous config saved to /var/cache/conftool/dbconfig/20200707-123003-kormat.json
• 12:12 jforrester@deploy1001: Finished scap: Full scap and testwikis to 1.35.0-wmf.40 for T256668 (duration: 33m 09s)
• 12:01 marostegui: Deploy schema change on labswiki (wikitech) master - T253276
• 11:58 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1082', diff saved to https://phabricator.wikimedia.org/P11786 and previous config saved to /var/cache/conftool/dbconfig/20200707-115838-marostegui.json
• 11:39 jforrester@deploy1001: Started scap: Full scap and testwikis to 1.35.0-wmf.40 for T256668
• 11:38 jforrester@deploy1001: scap failed: LockFailedError Failed to acquire lock "/var/lock/scap.operations_mediawiki-config.lock"; owner is "jforrester"; reason is "testwikis wikis to 1.35.0-wmf.40" (duration: 00m 00s)
• 11:33 moritzm: installing PHP 7.0 security updates
• 11:29 marostegui: Deploy schema change on db1082, this will create lag on s5 labs
• 11:29 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1082', diff saved to https://phabricator.wikimedia.org/P11784 and previous config saved to /var/cache/conftool/dbconfig/20200707-112926-marostegui.json
• 11:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1130', diff saved to https://phabricator.wikimedia.org/P11783 and previous config saved to /var/cache/conftool/dbconfig/20200707-112830-marostegui.json
• 11:26 godog: test bumping logstash7 batch size to 256
• 11:17 moritzm: prune PHP 7.0 packages from mwdebug1001/2001/2002
• 11:05 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1130', diff saved to https://phabricator.wikimedia.org/P11782 and previous config saved to /var/cache/conftool/dbconfig/20200707-110506-marostegui.json
• 11:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1110', diff saved to https://phabricator.wikimedia.org/P11781 and previous config saved to /var/cache/conftool/dbconfig/20200707-110412-marostegui.json
• 10:57 moritzm: prune PHP 7.0 packages from mw2190-mw2214
• 10:46 jforrester@deploy1001: Started scap: testwikis wikis to 1.35.0-wmf.40
• 10:44 jforrester@deploy1001: Pruned MediaWiki: 1.35.0-wmf.38 (duration: 17m 23s)
• 10:32 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1110', diff saved to https://phabricator.wikimedia.org/P11780 and previous config saved to /var/cache/conftool/dbconfig/20200707-103255-marostegui.json
• 10:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1113:3315', diff saved to https://phabricator.wikimedia.org/P11779 and previous config saved to /var/cache/conftool/dbconfig/20200707-102757-marostegui.json
• 10:26 moritzm: prune PHP 7.0 packages from mw2135-mw2147
• 10:12 addshore@deploy1001: Synchronized wmf-config/config/testcommonswiki.yaml: gerrit:609985 Make testcommonswiki a testwikidata client T257266 PT2/2 (duration: 00m 55s)
• 10:11 addshore@deploy1001: sync-file aborted: gerrit:609985 Make testcommonswiki a testwikidata client T257266 PT1/2 (duration: 00m 00s)
• 10:10 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1113:3315', diff saved to https://phabricator.wikimedia.org/P11778 and previous config saved to /var/cache/conftool/dbconfig/20200707-101043-marostegui.json
• 10:10 addshore@deploy1001: Synchronized dblists/wikidataclient-test.dblist: gerrit:609985 Make testcommonswiki a testwikidata client T257266 PT1/2 (duration: 00m 56s)
• 10:08 addshore@deploy1001: sync-file aborted: gerrit:609985 Make testcommonswiki a testwikidata client T257266 PT1/2 (duration: 00m 36s)
• 10:06 elukey: decommission archiva1001
• 10:05 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
• 10:03 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1144:3315', diff saved to https://phabricator.wikimedia.org/P11777 and previous config saved to /var/cache/conftool/dbconfig/20200707-100328-marostegui.json
• 10:03 elukey@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 10:03 elukey@cumin1001: START - Cookbook sre.hosts.downtime
• 10:03 elukey@cumin1001: START - Cookbook sre.hosts.decommission
• 09:54 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1144:3315', diff saved to https://phabricator.wikimedia.org/P11776 and previous config saved to /var/cache/conftool/dbconfig/20200707-095443-marostegui.json
• 09:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1096:3315', diff saved to https://phabricator.wikimedia.org/P11775 and previous config saved to /var/cache/conftool/dbconfig/20200707-095428-marostegui.json
• 09:42 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: gerrit:609971 T257266 Enable sitelinks to testcommons from test wikidata sites (duration: 00m 56s)
• 09:40 kormat@cumin1001: dbctl commit (dc=all): 'Repool es2021 after reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11774 and previous config saved to /var/cache/conftool/dbconfig/20200707-094017-kormat.json
• 09:37 addshore@deploy1001: Synchronized wmf-config: gerrit:609986 T257266 T241975 Wikibase: Remove config option wmgUseEntitySourceBasedFederation (take2) (duration: 00m 57s)
• 09:36 _joe_: errata: restbase2010, not 2009
• 09:36 _joe_: applying the new configuration using the service proxy to restbase2009 too
• 09:34 godog: bounce logstash on logstash1023
• 09:33 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: gerrit:609645 T257266 T241975 Wikibase: stop using wmgUseEntitySourceBasedFederation (take2) (duration: 00m 59s)
• 09:33 _joe_: depooling restbase1025 while we fix the troubled relationship between envoy and proton
• 09:33 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1096:3315', diff saved to https://phabricator.wikimedia.org/P11773 and previous config saved to /var/cache/conftool/dbconfig/20200707-093345-marostegui.json
• 09:26 marostegui@cumin1001: dbctl commit (dc=all): 'Remove weight from es1024 as it is the current master T255755', diff saved to https://phabricator.wikimedia.org/P11772 and previous config saved to /var/cache/conftool/dbconfig/20200707-092635-marostegui.json
• 09:24 James_F: 1.35.0-wmf.40 was branched at 88ecd6d for T256668
• 09:23 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool es1023 after reimage T255755', diff saved to https://phabricator.wikimedia.org/P11771 and previous config saved to /var/cache/conftool/dbconfig/20200707-092357-marostegui.json
• 09:10 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool es1023 after reimage T255755', diff saved to https://phabricator.wikimedia.org/P11770 and previous config saved to /var/cache/conftool/dbconfig/20200707-091015-marostegui.json
• 08:33 kormat@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:31 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool es1023 after reimage T255755', diff saved to https://phabricator.wikimedia.org/P11769 and previous config saved to /var/cache/conftool/dbconfig/20200707-083144-marostegui.json
• 08:30 kormat@cumin2001: START - Cookbook sre.hosts.downtime
• 08:26 XioNoX: cr2-codfw> request vmhost snapshot routing-engine both - T257153
• 08:22 XioNoX: cr2-eqsin> request vmhost snapshot - T257153
• 08:19 XioNoX: cr2-eqord> request vmhost snapshot - T257153
• 08:19 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool es1023 after reimage T255755', diff saved to https://phabricator.wikimedia.org/P11768 and previous config saved to /var/cache/conftool/dbconfig/20200707-081909-marostegui.json
• 08:18 elukey@cumin1001: END (ERROR) - Cookbook sre.hadoop.change-distro (exit_code=97)
• 08:17 XioNoX: cr2-eqdfw> request vmhost snapshot - T257153
• 08:15 XioNoX: cr3-knams> request vmhost snapshot - T257153
• 08:15 hashar: upgrading and restart CI Jenkins on contint2001 # T256978
• 08:12 XioNoX: cr4-ulsfo> request vmhost snapshot - T257153
• 08:09 kormat@cumin1001: dbctl commit (dc=all): 'Depool es2021 for reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11767 and previous config saved to /var/cache/conftool/dbconfig/20200707-080914-kormat.json
• 07:50 marostegui: Stop MySQL on db1074 to deploy schema change and remove triggers - T238966
• 07:45 _joe_: restarting restbase again on rb1025
• 07:44 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1074 for schema change', diff saved to https://phabricator.wikimedia.org/P11766 and previous config saved to /var/cache/conftool/dbconfig/20200707-074435-marostegui.json
• 07:41 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 07:39 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1079 and db1136 T257216', diff saved to https://phabricator.wikimedia.org/P11765 and previous config saved to /var/cache/conftool/dbconfig/20200707-073918-marostegui.json
• 07:38 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 07:31 _joe_: restarting restbase on restbase1025, reaching proton via envoy for now
• 07:31 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Config: Revert "Commons: Define entity sources configuration" (T256906, T256907, T256909, T254315, T257266) (forgot to git rebase so the last sync was a no-op) (duration: 00m 56s)
• 07:27 elukey@cumin1001: START - Cookbook sre.hadoop.change-distro
• 07:27 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Config: Revert "Commons: Define entity sources configuration" (T256906, T256907, T256909, T254315, T257266) (duration: 00m 53s)
• 07:27 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1079 and give more main weight to db1136 T257216', diff saved to https://phabricator.wikimedia.org/P11764 and previous config saved to /var/cache/conftool/dbconfig/20200707-072703-marostegui.json
• 07:24 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/: Config: Revert "Wikidata client wikis: Define entity sources configuration (take 2)" (T254315, T257266) (duration: 00m 56s)
• 07:24 elukey@cumin1001: END (PASS) - Cookbook sre.hadoop.stop-cluster (exit_code=0)
• 07:23 lucaswerkmeister-wmde@deploy1001: Synchronized dblists/wikidataclient.dblist: Config: Revert "Wikidata client wikis: Define entity sources configuration (take 2)" (T254315, T257266) (duration: 00m 56s)
• 07:19 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/Wikibase.php: Config: Revert "Wikibase: stop using wmgUseEntitySourceBasedFederation" (T241975, T257266) (duration: 00m 55s)
• 07:16 elukey@cumin1001: START - Cookbook sre.hadoop.stop-cluster
• 07:15 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Config: Revert "Wikibase: Remove config option wmgUseEntitySourceBasedFederation" (T241975, T257266) (duration: 00m 57s)
• 07:10 _joe_: restart restbase on restbase1025 to pick up the switch to https for cxserver
• 06:37 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1079 and give more main weight to db1136 T257216', diff saved to https://phabricator.wikimedia.org/P11762 and previous config saved to /var/cache/conftool/dbconfig/20200707-063737-marostegui.json
• 06:29 marostegui: Reimage es1023 to Buster T255755
• 06:20 marostegui@cumin1001: dbctl commit (dc=all): 'Give db1136 some weight back into main traffic T257216', diff saved to https://phabricator.wikimedia.org/P11761 and previous config saved to /var/cache/conftool/dbconfig/20200707-062008-marostegui.json
• 06:18 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1079 T257216', diff saved to https://phabricator.wikimedia.org/P11760 and previous config saved to /var/cache/conftool/dbconfig/20200707-061849-marostegui.json
• 05:26 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Enable es5 writes T255755 (duration: 00m 56s)
• 05:16 marostegui@cumin1001: dbctl commit (dc=all): 'Depool es1023 entirely T255755', diff saved to https://phabricator.wikimedia.org/P11759 and previous config saved to /var/cache/conftool/dbconfig/20200707-051620-marostegui.json
• 05:12 marostegui@cumin1001: dbctl commit (dc=all): 'Promote es1024 to es5 master T255755', diff saved to https://phabricator.wikimedia.org/P11758 and previous config saved to /var/cache/conftool/dbconfig/20200707-051236-marostegui.json
• 05:02 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Disable es5 writes T255755 (duration: 00m 56s)
• 05:01 marostegui: "Starting es failover from es1023 to es1024 - https://phabricator.wikimedia.org/T255755"
• 01:05 ejegg: turned on debug logging for Adyen SmashPig
• 00:22 cstone: civicrm revision changed from a48caf0f37 to d73ee2e73f

2020-07-06

• 23:32 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable sidebar instrumentation on test wikipedia (duration: 00m 56s)
• 23:32 eileen: process-control config revision is 3fe6753e56
• 23:22 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Change some zh canonical namespaces. Don't index NS_USER on hywiki (duration: 00m 58s)
• 22:59 eileen: tools revision changed from e974147f27 to 73557b8038
• 22:14 ryankemper@deploy1001: Finished deploy [wdqs/wdqs@65502b2]: 0.3.40 (duration: 18m 58s)
• 21:55 ryankemper@deploy1001: Started deploy [wdqs/wdqs@65502b2]: 0.3.40
• 21:52 hashar: Upgraded Jenkins on releases1002 and releases2002 # T256978
• 21:41 mutante: upgrading jenkins on releases1001 and releases2001 (T256980)
• 21:37 mutante: importing jenkins 2.235.1 into APT repo for both stretch and buster T256980
• 20:08 eileen: tools revision is e974147f27
• 19:41 qchris: Enabling puppet on gerrit1002 again to catch up with puppetmaster.
• 18:56 addshore: backport / deploy window done
• 18:55 addshore@deploy1001: Synchronized wmf-config: gerrit:569263 T241975 Wikibase: Remove config option wmgUseEntitySourceBasedFederation (duration: 00m 58s)
• 18:54 addshore@deploy1001: sync-file aborted: gerrit:569263 (duration: 00m 00s)
• 18:51 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: gerrit:608944 T241975 Wikibase: stop using wmgUseEntitySourceBasedFederation (duration: 00m 56s)
• 18:47 addshore@deploy1001: Synchronized dblists/wikidataclient.dblist: T254315 Wikidata client wikis: Define entity sources configuration (take 2) gerrit:608839 (duration: 00m 56s)
• 18:45 addshore@deploy1001: Synchronized wmf-config: T254315 Wikidata client wikis: Define entity sources configuration (take 2) gerrit:608839 (duration: 00m 58s)
• 18:38 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T256906 T256907 T256909 T254315 gerrit:569260 Commons: Define entity sources configuration (duration: 00m 56s)
• 18:30 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: adffbe6: Enable validation of new signatures (T248632) (duration: 00m 57s)
• 18:24 urbanecm@deploy1001: Synchronized wmf-config/abusefilter.php: 8878c60: Add `abusefilter-view` as a default right for the CU log user (T255506) (duration: 00m 55s)
• 18:22 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 1398171: Add arbcom group to plwiki (T256572) (duration: 00m 56s)
• 18:08 andrew@deploy1001: Finished deploy [horizon/deploy@bb176c2]: update proxy UI to support multiple pre-set domains (duration: 03m 39s)
• 18:04 andrew@deploy1001: Started deploy [horizon/deploy@bb176c2]: update proxy UI to support multiple pre-set domains
• 17:54 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Migrate SearchSatisfaction from EventLogging to EventGate on all wikis - T249261 - take 2 (duration: 00m 56s)
• 17:50 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Migrate SearchSatisfaction from EventLogging to EventGate on all wikis - T249261 (duration: 00m 56s)
• 16:09 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Migrate SearchSatisfaction from EventLogging to EventGate on group1 - T249261 (duration: 00m 58s)
• 15:02 jynus: removing old snapshots for x1 on dbprov[12]002
• 14:50 mholloway-shell@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'wikifeeds' for release 'production' .
• 14:46 mholloway-shell@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'wikifeeds' for release 'production' .
• 14:44 mholloway-shell@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' .
• 14:42 moritzm: installing PHP 7.0 security updates
• 14:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1074', diff saved to https://phabricator.wikimedia.org/P11753 and previous config saved to /var/cache/conftool/dbconfig/20200706-143754-marostegui.json
• 14:36 godog: reboot ms-be2025 for hw raid software upgrade - T257214
• 14:28 godog: powercycle ms-be2025, no ssh available - T257214
• 14:14 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 14:09 marostegui: Stop MySQL and poweroff db1079 T257216
• 14:08 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
• 14:02 jynus@cumin1001: dbctl commit (dc=all): 'depool db1136 from main traffic as it is the only s7 api host right now', diff saved to https://phabricator.wikimedia.org/P11752 and previous config saved to /var/cache/conftool/dbconfig/20200706-140217-jynus.json
• 13:56 marostegui: Downtime and reboot db1079 after BBU crash
• 13:54 jynus@cumin1001: dbctl commit (dc=all): 'depool db1079', diff saved to https://phabricator.wikimedia.org/P11751 and previous config saved to /var/cache/conftool/dbconfig/20200706-135430-jynus.json
• 13:30 marostegui: Deploy schema change on s5 codfw master T253276
• 13:26 marostegui@cumin1001: dbctl commit (dc=all): 'Reduce es1024 weight in preparation for tomorrow's switchover T255755', diff saved to https://phabricator.wikimedia.org/P11750 and previous config saved to /var/cache/conftool/dbconfig/20200706-132634-marostegui.json
• 13:03 elukey: force umount/mount of /mnt/hdfs on an-airflow1001 to unblock dpkg checks (fuse misbehaving, all checks hanging)
• 12:53 elukey: kill hanging lsof processes on an-airflow to reduce cpu load
• 12:42 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1074', diff saved to https://phabricator.wikimedia.org/P11748 and previous config saved to /var/cache/conftool/dbconfig/20200706-124237-marostegui.json
• 12:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1129', diff saved to https://phabricator.wikimedia.org/P11747 and previous config saved to /var/cache/conftool/dbconfig/20200706-124105-marostegui.json
• 11:17 Urbanecm: EU B&C window was done
• 11:16 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 5d971dc: GrowthExperiments: Remove overrides to welcome survey privacy policy URL (T252572) (duration: 00m 56s)
• 11:12 marostegui: Deploy schema changes on db1129
• 11:12 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1129', diff saved to https://phabricator.wikimedia.org/P11746 and previous config saved to /var/cache/conftool/dbconfig/20200706-111221-marostegui.json
• 11:09 marostegui: Compress InnoDB on db1107 T254462
• 11:09 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: f4b5001: Add arxiv.org to commonswiki wgCopyUploadsDomains (T257036) (duration: 00m 56s)
• 11:07 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1107 T254462', diff saved to https://phabricator.wikimedia.org/P11745 and previous config saved to /var/cache/conftool/dbconfig/20200706-110723-marostegui.json
• 11:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1076', diff saved to https://phabricator.wikimedia.org/P11744 and previous config saved to /var/cache/conftool/dbconfig/20200706-110544-marostegui.json
• 11:05 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 3bc1b46: Remove "Create a book" link from sidebar on Finnish Wikipedia (T257073) (duration: 00m 56s)
• 10:52 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (609762) (duration: 00m 57s)
• 10:51 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (609762) (duration: 00m 56s)
• 10:31 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 10:28 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 10:28 moritzm: rebooting idp1001 for kernel update
• 09:35 urbanecm@deploy1001: Synchronized private/PrivateSettings.php: Update T250887 mitigations (duration: 00m 58s)
• 08:51 XioNoX: cr1-codfw> request vmhost snapshot routing-engine both - T257153
• 08:44 XioNoX: cr3-ulsfo> request vmhost snapshot - T257153
• 08:24 kormat: restarting all mariadb instances on sanitarium hosts T256545
• 08:09 elukey: roll restart aqs on aqs100[4-9] to pick up new druid settings
• 08:05 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1076', diff saved to https://phabricator.wikimedia.org/P11742 and previous config saved to /var/cache/conftool/dbconfig/20200706-080509-marostegui.json
• 07:58 qchris: Disable puppet on gerrit1002 (gerrit-test) to deploy Gerrit UI updates there to gather more feedback
• 07:51 elukey: enable binlog on matomo's database on matomo1002
• 07:46 XioNoX: repool eqsin - T257154
• 07:11 XioNoX: reboot cr3-eqsin - T257154
• 06:55 XioNoX: depool eqsin for cr3-eqsin reboot/investigation - T257154
• 06:54 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1089', diff saved to https://phabricator.wikimedia.org/P11740 and previous config saved to /var/cache/conftool/dbconfig/20200706-065437-marostegui.json
• 06:54 elukey@cumin1001: END (FAIL) - Cookbook sre.hadoop.change-distro (exit_code=99)
• 06:22 elukey@cumin1001: START - Cookbook sre.hadoop.change-distro
• 06:21 elukey@cumin1001: END (PASS) - Cookbook sre.hadoop.stop-cluster (exit_code=0)
• 06:14 elukey@cumin1001: START - Cookbook sre.hadoop.stop-cluster
• 05:45 kart_: Updated cxserver to 2020-07-01-044435-production (T254143)
• 05:40 kartik@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'cxserver' for release 'production' .
• 05:36 kartik@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'cxserver' for release 'production' .
• 05:32 kartik@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'cxserver' for release 'staging' .
• 05:13 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1089', diff saved to https://phabricator.wikimedia.org/P11739 and previous config saved to /var/cache/conftool/dbconfig/20200706-051333-marostegui.json
• 05:03 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1089', diff saved to https://phabricator.wikimedia.org/P11738 and previous config saved to /var/cache/conftool/dbconfig/20200706-050347-marostegui.json
• 04:49 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1089', diff saved to https://phabricator.wikimedia.org/P11737 and previous config saved to /var/cache/conftool/dbconfig/20200706-044908-marostegui.json

2020-07-05

• 21:50 qchris: Restarting gerrit on gerrit1001 to pick up new war and jars.
• 21:50 qchris@deploy1001: Finished deploy [gerrit/gerrit@fbd0684]: Bump gerrit to 3.2.2-102-g3bbb138e13, zuul plugin to master-0-g7accc67, and gitiles to v3.2.2-1-g00c5ca0-with-0e3b533 on gerrit1001 (duration: 00m 07s)
• 21:50 qchris@deploy1001: Started deploy [gerrit/gerrit@fbd0684]: Bump gerrit to 3.2.2-102-g3bbb138e13, zuul plugin to master-0-g7accc67, and gitiles to v3.2.2-1-g00c5ca0-with-0e3b533 on gerrit1001
• 21:46 qchris: Restarting gerrit on gerrit2001 to pick up new war and jars.
• 21:45 qchris@deploy1001: Finished deploy [gerrit/gerrit@fbd0684]: Bump gerrit to 3.2.2-102-g3bbb138e13, zuul plugin to master-0-g7accc67, and gitiles to v3.2.2-1-g00c5ca0-with-0e3b533 on gerrit2001 (duration: 00m 10s)
• 21:45 qchris@deploy1001: Started deploy [gerrit/gerrit@fbd0684]: Bump gerrit to 3.2.2-102-g3bbb138e13, zuul plugin to master-0-g7accc67, and gitiles to v3.2.2-1-g00c5ca0-with-0e3b533 on gerrit2001
• 21:32 qchris: Restarting gerrit on gerrit1002 to pick up new wars and jars.
• 21:32 qchris@deploy1001: Finished deploy [gerrit/gerrit@fbd0684]: Bump gerrit to 3.2.2-102-g3bbb138e13 and zuul plugin to master-0-g7accc67 (duration: 00m 08s)
• 21:32 qchris@deploy1001: Started deploy [gerrit/gerrit@fbd0684]: Bump gerrit to 3.2.2-102-g3bbb138e13 and zuul plugin to master-0-g7accc67
• 21:20 qchris: Enable puppet on gerrit1002 (gerrit-test) again to let it catch up again
• 16:01 gehel: restart elastic-psi on elastic1052 (high GC rate)
• 15:56 gehel: restart blazegraph + updater on wdqs1007 and depool to allow catching up on lag

2020-07-04

• 19:23 qchris@deploy1001: Finished deploy [gerrit/gerrit@b78914b]: Bump gitiles to v3.2.2-1-g00c5ca0-with-0e3b533 on gerrit1002 (duration: 00m 08s)
• 19:23 qchris@deploy1001: Started deploy [gerrit/gerrit@b78914b]: Bump gitiles to v3.2.2-1-g00c5ca0-with-0e3b533 on gerrit1002
• 14:05 qchris: Disable puppet on gerrit1002 (gerrit-test) to deploy Gerrit UI updates there to gather feedback
• 12:42 reedy@deploy1001: Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 02m 24s)
• 02:28 reedy@deploy1001: Synchronized php-1.35.0-wmf.39/extensions/Score/includes/Score.php: Short circuit lilypond version check to allow usage of cached files T257066 (duration: 00m 55s)

2020-07-03

• 21:49 reedy@deploy1001: Synchronized php-1.35.0-wmf.39/extensions/Score/: Sync maintenance script (duration: 00m 58s)
• 18:47 cdanis: ✔️ cdanis@an-coord1001.eqiad.wmnet ~ 🕒☕ sudo systemctl restart hive-server2.service
• 16:51 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: Ifa929b2ad4 (duration: 00m 57s)
• 16:02 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: Rename wgRestrictionMethod to wgShellRestrictionMethod (duration: 00m 58s)
• 15:46 jayme@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 15:43 jayme@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 15:43 jynus@cumin1001: dbctl commit (dc=all): 'Reduce db1118 weight to spread load mode evenly', diff saved to https://phabricator.wikimedia.org/P11730 and previous config saved to /var/cache/conftool/dbconfig/20200703-154337-jynus.json
• 15:40 jayme@cumin1001: START - Cookbook sre.ganeti.makevm
• 15:38 jayme@cumin1001: START - Cookbook sre.ganeti.makevm
• 15:09 elukey@cumin1001: END (PASS) - Cookbook sre.hadoop.stop-cluster (exit_code=0)
• 15:02 elukey@cumin1001: START - Cookbook sre.hadoop.stop-cluster
• 14:11 elukey@cumin1001: END (FAIL) - Cookbook sre.hadoop.stop-cluster (exit_code=99)
• 14:11 _joe_: restarted php-fpm on wtp1033, stuck in sigill
• 13:59 elukey@cumin1001: START - Cookbook sre.hadoop.stop-cluster
• 12:41 hashar: Restarting Zuul / CI
• 11:39 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 11:36 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 11:32 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 11:29 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 11:29 moritzm: rebooting urldownloader standby hosts for kernel updates (1002/2002)
• 10:59 moritzm: installing json-c security updates on jessie
• 10:51 moritzm: installing ruby-json security updates
• 10:25 moritzm: installing nss security updates on jessie
• 10:18 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 10:15 elukey: notebook1004 renamed to an-scheduler1001
• 10:15 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 10:09 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 10:07 elukey@cumin1001: START - Cookbook sre.hosts.downtime
• 09:06 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 09:04 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 09:00 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 08:58 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 08:58 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 08:56 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 08:55 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 08:51 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 08:47 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 08:43 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 08:43 moritzm: rebooting netflow* hosts for kernel security update
• 08:16 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 08:14 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 08:04 jayme: authdns-update for chartmuseum - T256970
• 08:03 elukey@cumin1001: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
• 07:55 moritzm: installing mutt security updates for jessie (stretch/buster already fixed)
• 07:44 elukey@cumin1001: START - Cookbook sre.dns.netbox
• 07:40 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
• 07:39 elukey@cumin1001: START - Cookbook sre.hosts.decommission
• 07:00 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 07:00 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 06:47 moritzm: installing php5 security updates
• 06:27 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 06:27 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 06:09 moritzm: rebooting mw1390-mw1419 for kernel security updates
• 05:46 XioNoX: remove chassis redundancy failover from fasw-c-eqiad for consistency with all other VCs
• 05:33 XioNoX: remove chassis redundancy failover from fasw-c-codfw for consistency with all other VCs

2020-07-02

• 23:22 jhuneidi@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'blubberoid' for release 'production' .
• 23:16 jhuneidi@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'blubberoid' for release 'production' .
• 22:03 jhuneidi@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'blubberoid' for release 'staging' .
• 21:56 mutante: gerrit1001 (prod gerrit) - restarting gerrit service
• 21:52 maryum: frwikibooks reindex sucessful, continuing on with remainder of french wikis
• 21:32 mutante: gerrit - deleted gerrit db_pass from prod private repo, running puppet
• 21:25 mutante: gerrit2001 - restarted gerrit
• 21:14 mutante: gerrit1002 restarted gerrit
• 20:20 maryum: reindexing frwikibooks to test https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CirrusSearch/+/604221
• 19:52 mutante: gerrit2001 - restarting gerrit after removing db_pass from config
• 16:05 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
• 16:01 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
• 15:40 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 15:37 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 15:23 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
• 15:19 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
• 15:07 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:07 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 14:42 moritzm: rebooting mw1370-mw1389 for kernel security updates
• 14:33 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:33 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 14:03 kormat: stopped mariadb@s8 on dbstore1005 for data restoration T256966
• 12:43 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 12:43 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 12:36 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 12:36 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 12:32 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 12:32 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 12:31 moritzm: rebooting mw1349-mw1369 for kernel security updates
• 12:28 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 12:27 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 12:27 vgutierrez: rolling restart of esams load balancers to catch up on kernel upgrades
• 12:12 XioNoX: pre-configure asw2-b-eqiad<->cloudsw1-c8-eqiad - T251632
• 12:10 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 12:10 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 11:53 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 11:53 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 11:40 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 11:40 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 11:35 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 11:35 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 11:33 vgutierrez: rolling restart of codfw load balancers to catch up on kernel upgrades
• 11:18 akosiaris: preactively restart docker-registry on registry1001, registry1002 to force CA refresh
• 11:16 akosiaris: restart docker-registry on registry2002 for CA refresh
• 11:14 _joe_: restarting docker-registry on registry2001
• 10:34 godog: move "cluster overview" dashboard to Thanos - T256954
• 09:35 XioNoX: advertise codfw prefixes from eqord
• 09:28 jayme: imported chartmuseum_0.12.0-2 to buster-wikimedia - T253843
• 09:07 addshore: addshore@mwmaint1002:~$ mwscript maintenance/createAndPromote.php --wiki testwikidatawiki --force --custom-groups oversight "DCausse_(WMF)" # T256949
• 09:07 addshore: addshore@mwmaint1002:~$ mwscript maintenance/createAndPromote.php --wiki testwikidatawiki --force --custom-groups oversight "Addshore" # T256949
• 08:59 XioNoX: deploy flex flow for MX204s - T248394
• 05:52 _joe_: removing all tags for envoy-tls-local-proxy
• 05:46 _joe_: upload docker-report 0.0.4 on buster-wikimedia T242604
• 04:32 eileen: process-control config revision is b4655897b5
• 03:17 eileen: process-control config revision is 12fe6b5151
• 03:15 eileen: tools revision changed from 4ea8567819 to e974147f27
• 02:32 eileen: tools revision changed from e38f7a83d4 to 4ea8567819
• 00:53 eileen: tools revision changed from 806e2b4412 to e38f7a83d4

2020-07-01

• 23:53 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Set $wgForceUIAsContentMsg for zhwikibooks, zhwikinews, zhwikiquote, zhwikisource, zhwikiversity, zhwiktionary (T256521) (duration: 00m 55s)
• 23:35 ejegg: updated fundraising CiviCRM from 391d0fdf75 to a48caf0f37
• 23:32 catrope@deploy1001: Synchronized static/images/project-logos/: Change Simplified Chinese logo for zhwiki (T256839) (duration: 00m 55s)
• 23:18 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: Ibb42db7fd1ee (duration: 00m 55s)
• 23:00 bstorm: set a short downtime on labstore1006/7 to prevent alert while disabling direct systemd monitoring
• 22:37 krinkle@deploy1001: Synchronized php-1.35.0-wmf.39/includes/Title.php: I8d5bad (duration: 01m 00s)
• 21:00 mholloway-shell@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'production' .
• 20:58 mholloway-shell@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'mobileapps' for release 'production' .
• 20:56 mholloway-shell@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'mobileapps' for release 'staging' .
• 20:56 Krinkle: krinkle@deploy1001 Ran `scap deploy --init` for /srv/deployment/performance/arc-lamp
• 20:55 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@d7476f5]: Update mobileapps to 953fc41a (duration: 04m 08s)
• 20:51 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@d7476f5]: Update mobileapps to 953fc41a
• 20:27 eileen: tools revision changed from 6f38c14fe3 to 806e2b4412 -
• 20:11 eileen: tools revision changed from aab96444df to 6f38c14fe3
• 19:23 twentyafterfour: 1.35.0-wmf.39 is now deployed to group2 wikis, everything appears to be normal. refs T254176
• 19:18 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group2 wikis to 1.35.0-wmf.39 refs T254176
• 18:44 addshore@deploy1001: Synchronized wmf-config: REVERT T254315 Wikidata client wikis: Define entity sources configuration gerrit:569259 (duration: 01m 04s)
• 18:41 addshore@deploy1001: sync-file aborted: T254315 Wikidata client wikis: Define entity sources configuration gerrit:569259 (duration: 00m 38s)
• 18:38 joal@deploy1001: Finished deploy [analytics/refinery@8b7bddf] (thin): Regular analytics weekly train THIN [analytics/refinery@8b7bddf] (duration: 02m 19s)
• 18:36 joal@deploy1001: Started deploy [analytics/refinery@8b7bddf] (thin): Regular analytics weekly train THIN [analytics/refinery@8b7bddf]
• 18:35 joal@deploy1001: Finished deploy [analytics/refinery@8b7bddf]: Regular analytics weekly train [analytics/refinery@8b7bddf] (duration: 08m 09s)
• 18:27 joal@deploy1001: Started deploy [analytics/refinery@8b7bddf]: Regular analytics weekly train [analytics/refinery@8b7bddf]
• 18:25 joal@deploy1001: Finished deploy [analytics/refinery@114bfed]: Regular analytics weekly train [analytics/refinery@114bfed] (duration: 03m 41s)
• 18:21 joal@deploy1001: Started deploy [analytics/refinery@114bfed]: Regular analytics weekly train [analytics/refinery@114bfed]
• 18:18 ppchelko@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: Enable kafka purges on wikitech gerrit:607590 IS-labs.php (duration: 01m 03s)
• 18:07 ppchelko@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Deploy MediaModeration on all production wikis gerrit:608753 (duration: 01m 07s)
• 17:14 XioNoX: set flex-flow-sizing to cr2-eqsin - T248394
• 16:57 XioNoX: restart cr2-eqsin for software upgrade - T243080
• 16:00 XioNoX: updating eqsin LVS BGP neighbors IPs - T255766
• 15:16 XioNoX: re0.cr1-eqsin> request system power-off both-routing-engines - T255766
• 15:15 XioNoX: disable BGP to pybal on cr1-eqsin - T255766
• 15:13 XioNoX: disable cr1-eqsin transit/peering BGP - T255766
• 15:09 XioNoX: bump eqsin-codfw ospf link cost - T255766
• 15:03 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:03 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 15:03 XioNoX: move vrrp master to cr2-eqsin - T255766
• 15:00 XioNoX: depool eqsin for routers work - T255766
• 14:28 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:28 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 14:04 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:04 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 13:37 hashar: contint1001 stopped zuul-merger for a test. started it again
• 13:35 hashar: Restarting zuul-merger on contint2001 # T252310
• 13:30 hashar@deploy1001: Finished deploy [zuul/deploy@00f69b3]: (no justification provided) (duration: 00m 08s)
• 13:30 hashar@deploy1001: Started deploy [zuul/deploy@00f69b3]: (no justification provided)
• 13:29 hashar@deploy1001: Finished deploy [zuul/deploy@00f69b3]: (no justification provided) (duration: 00m 32s)
• 13:28 hashar@deploy1001: Started deploy [zuul/deploy@00f69b3]: (no justification provided)
• 13:16 hashar@deploy1001: Synchronized php: group1 wikis to 1.35.0-wmf.39 (duration: 01m 04s)
• 13:15 hashar@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.39
• 13:10 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:10 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 13:09 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:09 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 13:08 cdanis: ✔️ cdanis@netflow2001.codfw.wmnet ~ 🕘☕ sudo apt remove valgrind libc6-dbg
• 13:03 cdanis: T256790 ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕘☕ sudo cumin 'netflow[3-5]001*' 'systemctl restart nfacctd'
• 12:58 cdanis: T256790 ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕘☕ sudo debdeploy deploy -u 2020-07-01-pmacct.yaml -s netflow
• 12:55 cdanis: T256790 ✔️ cdanis@apt1001.wikimedia.org ~ 🕘☕ sudo -E reprepro -C main include buster-wikimedia pmacct_1.7.2-3+wmf1_amd64.changes
• 12:53 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 12:53 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 11:47 ema: A:cp upgrade librdkafka1 to 0.11.6-1.1wmf1 and restart purged, varnishkafka T256444
• 11:46 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T254315 Wikidata: Define entity sources configuration gerrit:569258 (duration: 01m 06s)
• 11:32 Lucas_WMDE: EU B&C window done
• 11:24 lucaswerkmeister-wmde@deploy1001: Synchronized w/touch.php: Config: Fully set MW_NO_SESSION for browser metadata endpoints, 4/4 (duration: 01m 06s)
• 11:22 lucaswerkmeister-wmde@deploy1001: Synchronized w/robots.php: Config: Fully set MW_NO_SESSION for browser metadata endpoints, 3/4 (duration: 01m 03s)
• 11:21 lucaswerkmeister-wmde@deploy1001: Synchronized w/favicon.php: Config: Fully set MW_NO_SESSION for browser metadata endpoints, 2/4 (duration: 01m 04s)
• 11:19 lucaswerkmeister-wmde@deploy1001: Synchronized w/extract2.php: Config: Fully set MW_NO_SESSION for browser metadata endpoints, 1/4 (duration: 01m 16s)
• 11:07 Amir1: Changing datatype of several properties with mwscript extensions/Wikibase/repo/maintenance/changePropertyDataType.php (T255241)
• 11:07 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'blubberoid' for release 'staging' .
• 11:02 ema: restbase2009 depooled T256863
• 11:02 ema@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2009.codfw.wmnet
• 10:50 ema: power on restbase2009
• 10:45 jayme: draining and docker restart (one at a time) kubernetes[1001-1004].eqiad.wmnet - T256786
• 10:34 ema: power-cycle restbase2009
• 10:17 XioNoX: renumber NTT transit links - T254877
• 10:16 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 10:16 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 10:14 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 10:14 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 10:09 jayme: draining and docker restart (one at a time) kubernetes[2001-2004].codfw.wmnet
• 09:52 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:52 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 09:46 jayme: cordoning kubernetes[2001-2004].codfw.wmnet,kubernetes[1001-1004].eqiad.wmnet - T256786
• 09:42 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:42 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 09:34 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:34 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 09:23 jayme: restarting dockerd on kubestage1002.eqiad.wmnet - T256786
• 09:15 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:15 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 09:08 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:08 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 08:53 jayme: draining kubernetes staging node kubestage1001.eqiad.wmnet - T256786
• 08:52 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:52 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 08:44 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:44 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 08:29 XioNoX: disable BGP to nfacct in eqiad - T256790
• 08:23 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:23 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 08:08 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'mobileapps' for release 'staging' .
• 08:05 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:05 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 08:01 vgutierrez: rolling restart of esams cache nodes to catch up on kernel upgrades
• 07:42 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 07:42 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 07:40 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 07:40 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 07:39 ema: cp2041: restart purged, varnishkafka after librdkafka1 upgrade to 0.11.6-1.1wmf1 T256444
• 05:47 _joe_: restarting nfacctd on netflow1001, it's segfaulting
• 04:01 krinkle@deploy1001: Synchronized php-1.35.0-wmf.39/maintenance/findBadBlobs.php: I47c11190b665 (duration: 01m 08s)
• 00:14 krinkle@deploy1001: Synchronized private/PrivateSettings.php: T254795 - Set $wmgXhguiDBuser and $wmgXhguiDBpasswor (duration: 01m 06s)

2020-06-30

• 21:48 crusnov@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 21:46 crusnov@cumin1001: START - Cookbook sre.hosts.reboot-single
• 21:45 crusnov@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 21:43 crusnov@cumin1001: START - Cookbook sre.hosts.reboot-single
• 21:42 crusnov@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 21:40 crusnov@cumin1001: START - Cookbook sre.hosts.reboot-single
• 21:40 crusnov@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 21:38 crusnov@cumin1001: START - Cookbook sre.hosts.reboot-single
• 21:38 crusnov@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99)
• 21:38 crusnov@cumin1001: START - Cookbook sre.hosts.reboot-single
• 19:19 hashar@deploy1001: rebuilt and synchronized wikiversions files: group 0 wikis to 1.35.0-wmf.39 # T254176
• 18:31 cdanis: T256790 ✔️ cdanis@netflow2001.codfw.wmnet ~ 🕝☕ sudo apt install valgrind
• 18:27 tgr: Morning deploys done
• 18:23 tgr@deploy1001: Synchronized php-1.35.0-wmf.39/extensions/ElectronPdfService/src/ElectronPdfServiceHooks.php: Backport: Hotfix: "Undefined index: print" (T256761) (duration: 01m 05s)
• 18:11 shdubsh: restart varnishmtail,atsmtail,ncredirmtail on ncredir,cp hosts in codfw and eqsin
• 18:05 cdanis: installing libc6-dbg on netflow2001 T256790
• 17:40 mdholloway: mobileapps deployments on k8s failing with timeouts; filed T256786
• 17:37 cdanis: ✔️ cdanis@netflow2001.codfw.wmnet ~ 🕜☕ sudo systemctl restart nfacctd
• 17:33 mholloway-shell@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'mobileapps' for release 'staging' .
• 17:18 mholloway-shell@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'mobileapps' for release 'staging' .
• 17:17 papaul: uplugging msw-c3 power to relocate port on PDU
• 17:09 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@f9df1af]: Update mobileapps to 5c7611b9 (duration: 03m 33s)
• 17:05 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@f9df1af]: Update mobileapps to 5c7611b9
• 16:57 cdanis: T256444 restarted purged on cp2030 and repooling
• 16:48 cdanis: T256444 ✔️ cdanis@cp2030.codfw.wmnet ~ 🕐☕ sudo depool
• 15:54 otto@deploy1001: Finished deploy [analytics/refinery@d63944e]: Deploying new camus wmf10 jar to an-launcher1002 for T256370 - take 3 (duration: 00m 03s)
• 15:54 otto@deploy1001: Started deploy [analytics/refinery@d63944e]: Deploying new camus wmf10 jar to an-launcher1002 for T256370 - take 3
• 15:40 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 15:35 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 15:32 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 15:30 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 15:24 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 15:20 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 15:16 otto@deploy1001: Finished deploy [analytics/refinery@1112749]: roll back to 1112749 on an-launcher1002, git-fat not pulling artifacts (duration: 01m 21s)
• 15:14 otto@deploy1001: Started deploy [analytics/refinery@1112749]: roll back to 1112749 on an-launcher1002, git-fat not pulling artifacts
• 15:14 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 15:10 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 15:10 moritzm: rebooting mwdebug* hosts for kernel security update
• 15:05 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 15:03 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 15:01 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 14:59 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:59 moritzm: rebooting failoid hosts for kernel update
• 14:49 otto@deploy1001: Finished deploy [analytics/refinery@d63944e]: Deploying new camus wmf10 jar to an-launcher1002 for T256370 - take 3 (duration: 00m 03s)
• 14:49 otto@deploy1001: Started deploy [analytics/refinery@d63944e]: Deploying new camus wmf10 jar to an-launcher1002 for T256370 - take 3
• 14:47 otto@deploy1001: Finished deploy [analytics/refinery@d63944e]: Deploying new camus wmf10 jar to an-launcher1002 for T256370 - take 2 (duration: 00m 03s)
• 14:47 otto@deploy1001: Started deploy [analytics/refinery@d63944e]: Deploying new camus wmf10 jar to an-launcher1002 for T256370 - take 2
• 14:44 hashar: Train blocked on Flow being broken: T256761 # T254176
• 14:38 hashar@deploy1001: rebuilt and synchronized wikiversions files: Revert "group0 wikis to 1.35.0-wmf.39" - T256759
• 14:34 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 14:32 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:28 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 14:26 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:25 hashar@deploy1001: rebuilt and synchronized wikiversions files: group0 wikis to 1.35.0-wmf.39
• 14:21 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:21 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 14:21 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 14:18 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:18 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 14:15 moritzm: rebooting miscweb servers for kernel security update
• 14:15 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:13 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 14:11 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:10 otto@deploy1001: Finished deploy [analytics/refinery@d63944e]: Deploying new camus wmf10 jar to an-launcher1002 for T256370 (duration: 01m 56s)
• 14:10 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 14:09 hashar@deploy1001: Finished scap: testwikis wikis to 1.35.0-wmf.39 (duration: 62m 30s)
• 14:08 otto@deploy1001: Started deploy [analytics/refinery@d63944e]: Deploying new camus wmf10 jar to an-launcher1002 for T256370
• 14:08 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 13:59 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:59 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 13:59 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 13:57 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 13:51 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 13:49 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 13:37 moritzm: rebooting LDAP replicas for kernel security update
• 13:32 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:32 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 13:31 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:30 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 13:07 hashar@deploy1001: Started scap: testwikis wikis to 1.35.0-wmf.39
• 12:03 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 12:03 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 11:35 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 11:35 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 11:33 awight: EU BACON cooked
• 11:32 awight@deploy1001: Synchronized wmf-config/InitialiseSettings.php: BACON: Configure TeWü survey on dewiki (take 2) (T253112) (duration: 00m 58s)
• 11:32 jayme: restarted docker-reporter-base-images and docker-reporter-releng-images on deneb - T253396
• 11:31 jayme: pushed a scratch docker image as docker-registry.discovery.wmnet/envoy-tls-local-proxy:dontuseme - T253396
• 11:28 awight@deploy1001: Synchronized php-1.35.0-wmf.38/extensions/QuickSurveys: BACON: Embedded surveys are hidden when no element is available (T256627) (duration: 00m 56s)
• 11:26 awight@deploy1001: Synchronized php-1.35.0-wmf.38/extensions/FileImporter: BACON: Set Status error if permission check returns false. (T256428) (duration: 00m 58s)
• 11:13 ema: deneb: systemctl restart docker-reporter-base-images.service
• 10:59 ema: upload librdkafka 0.11.6-1.1wmf1 to buster-wikimedia https://phabricator.wikimedia.org/P11703 T256444
• 10:59 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 10:59 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 10:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1076', diff saved to https://phabricator.wikimedia.org/P11710 and previous config saved to /var/cache/conftool/dbconfig/20200630-105254-marostegui.json
• 10:45 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 10:45 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 10:41 ema: cp2040: restart purged and varnishkafka to use updated librdkafka1 T256444
• 10:38 ema: cp2040: upgrade librdkafka1 to 0.11.6-1.1wmf1 https://phabricator.wikimedia.org/P11703 T256444
• 10:37 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 10:37 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 10:30 hashar@deploy1001: Synchronized php-1.35.0-wmf.39/includes/specials/SpecialUndelete.php: Remove another use of PageArchive::getRevision - T249982 T254176 (duration: 00m 56s)
• 10:09 marostegui: Deploy schema change on db1076
• 10:09 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1076', diff saved to https://phabricator.wikimedia.org/P11708 and previous config saved to /var/cache/conftool/dbconfig/20200630-100912-marostegui.json
• 10:04 vgutierrez: rolling restart of eqiad cache nodes to catch up on kernel upgrades
• 10:03 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 10:03 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 10:02 volker-e@deploy1001: Finished deploy [design/style-guide@e3fda83]: Deploy design/style-guide: (duration: 00m 07s)
• 10:02 volker-e@deploy1001: Started deploy [design/style-guide@e3fda83]: Deploy design/style-guide:
• 09:47 hashar@deploy1001: Pruned MediaWiki: 1.35.0-wmf.37 (duration: 02m 20s)
• 09:40 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:40 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 09:21 hashar@deploy1001: Pruned MediaWiki: 1.35.0-wmf.36 (duration: 28m 11s)
• 08:54 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:53 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime
• 08:53 hashar@deploy1001: clean aborted: Pruned MediaWiki: 1.35.0-wmf.36 (duration: 00m 00s)
• 08:51 hashar: Applied security patches to wmf/1.35.0-wmf.39 # T254176
• 08:51 vgutierrez: rolling restart of codfw cp nodes after "re-formatting" nvme devices - T256655
• 08:23 vgutierrez: repool cp3053 - T256632
• 08:10 hashar: 1.35.0-wmf.39 was branched at e169e3d T254176
• 08:05 marostegui: Stop MySQL on db1117:3322 to clone db1080 (this will trigger haproxy alerts) - T256717
• 08:05 vgutierrez: powercycle cp3053 (unresponsive after reboot) - T256632
• 08:01 jbond42: disable puppet to restart puppetmasters front ends
• 07:42 vgutierrez: reboot cp3053 - T256632
• 05:51 jhuneidi@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'blubberoid' for release 'staging' .
• 05:13 marostegui: Deploy schema change on s8 codfw - T256680
• 04:58 marostegui: remove pl_from index from db1141, db1121, db1148 - T256684
• 04:57 jhuneidi@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'blubberoid' for release 'staging' .
• 04:56 marostegui: Remove plfrom from db1096:3316 and db1098:3316 - T256684

2020-06-29

• 23:28 eileen: civicrm revision changed from 52a32f2d66 to 391d0fdf75, config revision is f1b4bdb7b7
• 22:00 sbassett: Deployed patch for T256171
• 21:56 sbassett: Deployed patch for T255918
• 20:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1144:3315 T256679', diff saved to https://phabricator.wikimedia.org/P11699 and previous config saved to /var/cache/conftool/dbconfig/20200629-200002-marostegui.json
• 19:43 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1144:3315 T256679', diff saved to https://phabricator.wikimedia.org/P11698 and previous config saved to /var/cache/conftool/dbconfig/20200629-194327-marostegui.json
• 18:55 shdubsh: test mtail rc35+wmf2 on cp5001 - T255776
• 18:15 Urbanecm: Morning B&C done
• 18:15 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: c86fcd4: Add HTTP proxy to MediaModeration (T247943) (duration: 00m 58s)
• 18:10 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: aeb7b52: Setup rollbacker and mover on lijwiki (T256109) (duration: 02m 05s)
• 17:30 sukhe: LDAP - added datn to groups wmde, nda - T254442
• 15:43 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:43 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 15:37 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:37 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 15:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1105:3312', diff saved to https://phabricator.wikimedia.org/P11696 and previous config saved to /var/cache/conftool/dbconfig/20200629-153140-marostegui.json
• 15:20 gehel: repool wdqs1004 - catched up on lag
• 14:50 hnowlan@deploy1001: Finished deploy [restbase/deploy@900bcf6]: Redeploy to fix transient error in gom wiktionary deploy (duration: 00m 06s)
• 14:50 hnowlan@deploy1001: Started deploy [restbase/deploy@900bcf6]: Redeploy to fix transient error in gom wiktionary deploy
• 14:48 hnowlan@deploy1001: Finished deploy [restbase/deploy@900bcf6]: Enable gom wiktionary (duration: 13m 40s)
• 14:34 hnowlan@deploy1001: Started deploy [restbase/deploy@900bcf6]: Enable gom wiktionary
• 14:33 hnowlan@deploy1001: Finished deploy [restbase/deploy@900bcf6]: Enable gom wiktionary (duration: 17m 49s)
• 14:28 ema: A:cp rolling purged upgrade to 0.16 T256479
• 14:22 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Config: Add "E" as an alias of EntitySchema namespace on wikidata (T245529) (duration: 00m 57s)
• 14:20 ema: upload purged 0.16 to apt.wm.org T256479
• 14:16 hnowlan@deploy1001: Started deploy [restbase/deploy@900bcf6]: Enable gom wiktionary
• 14:14 hnowlan@deploy1001: Finished deploy [restbase/deploy@ce5177e]: Enable gom wiktionary (duration: 20m 44s)
• 14:02 jforrester@deploy1001: Synchronized multiversion/MWConfigCacheGenerator.php: Fix 'closed-labs' reading as 'closed' for static config (duration: 00m 56s)
• 13:54 jforrester@deploy1001: Synchronized dblists/: Drop nonbetafeatures dblist, unused (duration: 00m 57s)
• 13:54 hnowlan@deploy1001: Started deploy [restbase/deploy@ce5177e]: Enable gom wiktionary
• 13:50 jforrester@deploy1001: Synchronized multiversion/MWConfigCacheGenerator.php: Drop 'nonbetafeatures' dblist from production reads (duration: 00m 56s)
• 13:49 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Switch uses from nonbetafeatures to lockeddown (duration: 00m 57s)
• 13:47 jforrester@deploy1001: Synchronized multiversion/MWConfigCacheGenerator.php: Add 'lockeddown' dblist to production reads (duration: 00m 57s)
• 13:43 jforrester@deploy1001: Synchronized dblists/lockeddown.dblist: Add lockddown dblist (unused as yet) (duration: 00m 59s)
• 13:35 vgutierrez: depool cp3053 due to nvme hardware issues
• 13:02 XioNoX: test pfw3-codfw uplinks failover
• 13:00 elukey: move archiva.wikimedia.org to archiva1002 (new buster vm); create archiva-old.wikimedia.org to archiva1001
• 12:58 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1105:3312', diff saved to https://phabricator.wikimedia.org/P11693 and previous config saved to /var/cache/conftool/dbconfig/20200629-125824-marostegui.json
• 12:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1085', diff saved to https://phabricator.wikimedia.org/P11692 and previous config saved to /var/cache/conftool/dbconfig/20200629-125630-marostegui.json
• 12:41 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 12:38 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 12:32 jayme: deleted all tags for docker-registry.wikimedia.org/envoy-tls-local-proxy from docker registry - T253396
• 12:20 marostegui: Stop MySQL on db2096 (codfw x1 master) for reimage T254871
• 12:03 cdanis: re-pool eqiad T256512
• 11:59 cdanis: deployed I132075ee on cr1-eqiad T256512
• 11:58 cdanis: deployed I132075ee on cr2-eqiad T256512
• 11:58 cdanis: deployed I132075ee on cr2-eqiad
• 11:41 cdanis: depool eqiad T256512
• 11:15 awight: EU BACON cooked
• 11:08 marostegui: Deploy schema change on db1095:3312 (lag will show up)
• 10:41 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (608284) (duration: 00m 57s)
• 10:41 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (608284) (duration: 00m 58s)
• 10:29 gehel: restart blazegraph on wdqs1004 + depool to catchup on lag
• 09:59 ema: cp2040: upgrade purged to 0.16 T256479
• 09:59 jbond42: switch idp to memcached
• 09:47 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:47 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 09:45 marostegui: Deploy schema change on dbstore1004:3312
• 09:11 jbond42: dploying shellcheck CI https://gerrit.wikimedia.org/r/c/operations/puppet/+/602693
• 08:59 marostegui: Compress InnoDB on db1089 (this will cause lag and will take a few days) - T254462
• 08:58 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1089 for InnoDB compression T254462', diff saved to https://phabricator.wikimedia.org/P11690 and previous config saved to /var/cache/conftool/dbconfig/20200629-085854-marostegui.json
• 08:48 marostegui@cumin1001: dbctl commit (dc=all): 'Fully pool db1135 into s1 T253217', diff saved to https://phabricator.wikimedia.org/P11688 and previous config saved to /var/cache/conftool/dbconfig/20200629-084827-marostegui.json
• 08:40 ema: cp2034: restart purged T256444
• 08:36 ema: cp4025: restart purged T256444
• 08:36 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly pool db1135 into s1 T253217', diff saved to https://phabricator.wikimedia.org/P11687 and previous config saved to /var/cache/conftool/dbconfig/20200629-083631-marostegui.json
• 08:33 ema: cp1087, cp2033, cp2037, cp2039: repool after spending (way) more than 24h depooled T256444
• 08:26 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly pool db1135 into s1 T253217', diff saved to https://phabricator.wikimedia.org/P11686 and previous config saved to /var/cache/conftool/dbconfig/20200629-082635-marostegui.json
• 08:24 marostegui: Deploy schema change on s2 codfw (lag will show up) T253276
• 08:04 XioNoX: add term selected-paths to policy BGP_IXP_in on all routers
• 08:03 godog: prometheus eqiad -- lvextend --resizefs --size +200G vg-ssd/prometheus-ops
• 08:02 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly pool db1135 into s1 T253217', diff saved to https://phabricator.wikimedia.org/P11685 and previous config saved to /var/cache/conftool/dbconfig/20200629-080253-marostegui.json
• 07:46 marostegui@cumin1001: dbctl commit (dc=all): 'Add db1135 (depooled) to s1 T253217', diff saved to https://phabricator.wikimedia.org/P11684 and previous config saved to /var/cache/conftool/dbconfig/20200629-074611-marostegui.json
• 07:16 XioNoX: push new pfw firewall rules - T256170
• 07:13 marostegui: Deploy schema change on db1085 with replication to labs T253276
• 07:12 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1085', diff saved to https://phabricator.wikimedia.org/P11683 and previous config saved to /var/cache/conftool/dbconfig/20200629-071236-marostegui.json
• 06:53 marostegui@cumin1001: dbctl commit (dc=all): 'Remove db1080 from MW', diff saved to https://phabricator.wikimedia.org/P11682 and previous config saved to /var/cache/conftool/dbconfig/20200629-065335-marostegui.json
• 06:50 elukey: execute gnt-instance remove an-launcher1001.eqiad.wmnet on ganeti1011 - T256363
• 06:47 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
• 06:46 elukey@cumin1001: START - Cookbook sre.hosts.decommission
• 06:45 marostegui: Deploy MCR schema change on db1090:3312
• 06:35 elukey: force puppet run on ores* to overcome celery OOMs on some nodes
• 04:57 marostegui: Stop MySQL on db1080 to clone db1135 T253217
• 04:56 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 04:53 marostegui@cumin1001: START - Cookbook sre.hosts.downtime

2020-06-28

• 21:43 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: no-op I56eb4a802 (duration: 00m 58s)
• 21:38 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: beta-only I56eb4a802 (duration: 01m 00s)

2020-06-27

• 20:22 qchris: Gerrit upgrade done.
• 19:49 mutante: removed 2620:0:861:3:208:80:154:136 from /etc/network/interfaces on gerrit1001, rebooting
• 19:27 mutante: rebooting gerrit1001 one more time
• 19:24 mutante: restarted ferm on gerrit1001
• 19:19 mutante: rebooting gerrit1001 one more time
• 19:05 mutante: rebooting gerrit1001
• 18:58 mutante: rebooting gerrit2001
• 18:49 hashar: Enabling beta cluster update job (gerrit maintenance) https://integration.wikimedia.org/ci/view/Beta/job/beta-code-update-eqiad/
• 18:35 qchris@deploy1001: Finished deploy [gerrit/gerrit@da40615]: Gerrit to v3.2.2-98-g98d827eaa3 on gerrit2001 (duration: 00m 10s)
• 18:34 qchris@deploy1001: Started deploy [gerrit/gerrit@da40615]: Gerrit to v3.2.2-98-g98d827eaa3 on gerrit2001
• 18:27 qchris@deploy1001: Finished deploy [gerrit/gerrit@da40615]: Gerrit to v3.2.2-98-g98d827eaa3 on gerrit1001 (duration: 00m 08s)
• 18:27 qchris@deploy1001: Started deploy [gerrit/gerrit@da40615]: Gerrit to v3.2.2-98-g98d827eaa3 on gerrit1001
• 17:25 hashar: Disabled beta cluster update job (gerrit maintenance) https://integration.wikimedia.org/ci/view/Beta/job/beta-code-update-eqiad/
• 17:19 qchris: Stopping gerrit on gerrit1001 for the Gerrit upgrade
• 17:14 qchris: Duplicating reviewdb changes so we get a cheap and quick rollback
• 17:11 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 17:11 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 17:11 qchris: Disabling puppet on gerrit1001 for Gerrit upgrades + data migrations
• 17:11 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 17:11 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 17:07 qchris: Starting Gerrit upgrade to v3.2.2-98-g98d827eaa3
• 15:44 qchris@deploy1001: Finished deploy [gerrit/gerrit@da40615]: Gerrit to v3.2.2-98-g98d827eaa3 on gerrit1002 (gerrit-test) (duration: 00m 08s)
• 15:44 qchris@deploy1001: Started deploy [gerrit/gerrit@da40615]: Gerrit to v3.2.2-98-g98d827eaa3 on gerrit1002 (gerrit-test)
• 13:03 qchris@deploy1001: Finished deploy [gerrit/gerrit@460e439]: Gerrit to v3.2.2-97-gcaf5020db1 on gerrit1002 (gerrit-test) (duration: 00m 08s)
• 13:03 qchris@deploy1001: Started deploy [gerrit/gerrit@460e439]: Gerrit to v3.2.2-97-gcaf5020db1 on gerrit1002 (gerrit-test)

2020-06-26

• 18:42 robh: all ulsfo onsite work completed as of 30 minutes ago
• 17:52 robh: msw2-ulsfo work done, all mgmt items confirmed back online and icinga alerts cleared, moving onto msw1-ulsfo (rack 22) and will lose all mgmt in that rack for next 10-20 minutes T256300
• 17:52 robh: msw2-ulsfo work done, all mgmt items confirmed back online and icinga alerts cleared, moving onto msw1-ulsfo (rack 22) and will lose all mgmt in that rack for next 10-20 minutes
• 17:11 robh: msw work in ulsfo via T256300
• 10:24 ema: pool 5006 T256449
• 10:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1085', diff saved to https://phabricator.wikimedia.org/P11677 and previous config saved to /var/cache/conftool/dbconfig/20200626-102248-marostegui.json
• 10:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1093', diff saved to https://phabricator.wikimedia.org/P11676 and previous config saved to /var/cache/conftool/dbconfig/20200626-102201-marostegui.json
• 10:03 ema: cp2039: restart purged T256444
• 09:57 ema: cp2037: restart purged T256444
• 09:55 ema: cp1087: restart purged T256444
• 09:46 ema: cp2033: restart purged T256444
• 09:38 akosiaris: move the sessionstore eqiad pods back to the dedicated sessionstore nodes
• 09:37 akosiaris@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'sessionstore' for release 'production' .
• 09:35 akosiaris: move the sessionstore codfw pods back to the dedicated sessionstore nodes
• 09:35 akosiaris@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'sessionstore' for release 'production' .
• 09:08 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1093 for schema change', diff saved to https://phabricator.wikimedia.org/P11675 and previous config saved to /var/cache/conftool/dbconfig/20200626-090813-marostegui.json
• 08:58 jynus@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:56 jynus@cumin1001: START - Cookbook sre.hosts.downtime
• 08:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1088', diff saved to https://phabricator.wikimedia.org/P11674 and previous config saved to /var/cache/conftool/dbconfig/20200626-083319-marostegui.json
• 08:25 ayounsi@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 08:22 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1088 for schema change', diff saved to https://phabricator.wikimedia.org/P11673 and previous config saved to /var/cache/conftool/dbconfig/20200626-082242-marostegui.json
• 08:20 ayounsi@cumin1001: START - Cookbook sre.dns.netbox
• 08:20 ayounsi@cumin1001: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
• 08:05 akosiaris@cumin1001: conftool action : set/pooled=yes; selector: name=kubernetes.*.wmnet
• 08:04 akosiaris@cumin1001: conftool action : set/weight=10; selector: name=kubernetes.*.wmnet
• 08:04 akosiaris: pool all new kubernetes nodes in LVS T252185 T256236
• 07:57 ayounsi@cumin1001: START - Cookbook sre.dns.netbox
• 07:44 volans: force rebooted cp5006 that is unresponsive (after having depooled it) - T256449
• 07:42 volans@cumin1001: conftool action : set/pooled=no; selector: name=cp5006.eqsin.wmnet
• 06:40 tstarling@deploy1001: Synchronized wmf-config/InitialiseSettings.php: add cache-cookies log channel (duration: 00m 59s)
• 05:13 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db2088:3312, db2104', diff saved to https://phabricator.wikimedia.org/P11672 and previous config saved to /var/cache/conftool/dbconfig/20200626-051328-marostegui.json
• 05:06 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 05:03 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 04:01 cdanis: re-enable puppet on cps
• 03:54 cdanis: ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕛🍺 sudo cumin A:cp 'disable-puppet "I39e1c68a is broken"'
• 03:54 cdanis: https://gerrit.wikimedia.org/r/c/operations/puppet/+/607917
• 02:52 tstarling@deploy1001: Synchronized private/PrivateSettings.php: updating wgAuthenticationTokenVersion per my wikitech-l post (duration: 00m 57s)
• 02:19 cdanis: three more hosts not processing purges for multiple days ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕥🍺 sudo cumin 'cp2033*,cp2037*,cp2039*' 'depool'
• 02:17 cdanis: depooling cp1087 which has not been processing purges for 11.415 days
• 01:53 cdanis: I6cc5f3e6 has been deployed to all cp text nodes T256395
• 01:41 cdanis: ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕘🍺 sudo cumin A:cp 'enable-puppet "cdanis deploying I6cc5f3e6 T256395"'
• 01:13 cdanis: ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕘🍺 sudo cumin A:cp 'disable-puppet "cdanis deploying I6cc5f3e6 T256395"'
• 00:41 eileen: tools revision changed from c96813eda4 to aab96444df
• 00:38 tstarling@deploy1001: Synchronized w/T256395-cookie-test.php: (no justification provided) (duration: 00m 56s)
• 00:36 tstarling@deploy1001: Synchronized w/T256395-cookie-test.php: (no justification provided) (duration: 00m 58s)

2020-06-25

• 23:37 mutante: puppetmaster - signing certs and initial puppet run for logstash1030/logstash1031 - no prod role yet
• 22:25 mutante: puppetmaster - signing certs and initial run for logstash2030/2031 - no prod role yet
• 20:57 dzahn@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 20:31 dzahn@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 19:30 dcausse: repooling wdqs1007.eqiad.wmnet
• 19:05 brennen@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.38
• 18:58 mutante: LDAP - added qchris to archiva-deployers (T256404)
• 17:37 mutante: mwmaint1002 - restarted apache2 to add server_headers snippet for T255629 - but not working as expected yet
• 16:40 dzahn@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 16:31 sukhe@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 16:31 sukhe@cumin1001: START - Cookbook sre.hosts.downtime
• 16:28 krinkle@deploy1001: Synchronized wmf-config/logging.php: Ia6ef7617d378 (duration: 01m 02s)
• 16:20 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 16:16 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 16:16 dzahn@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 16:16 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 16:15 Krinkle: I've deleted a "saved object" visualisation in logstash called "Production Errors & Deployments" which seemed to be corrupt and redirect random logstash dashboards to a management page. Backed up at https://phabricator.wikimedia.org/P11666 (NDA)
• 16:15 moritzm: installing libxml2 security updates
• 16:12 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 16:09 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 16:06 moritzm: installing 4.9.210-1+deb9u1~deb8u1 on jessie hosts (fixed kernel for recent cacheoutattack CPU leaks)
• 16:04 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 16:03 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 15:59 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 15:55 krinkle@deploy1001: Synchronized wmf-config/logging.php: I4c519f (duration: 01m 05s)
• 15:54 dzahn@cumin1001: START - Cookbook sre.ganeti.makevm
• 15:53 dzahn@cumin1001: START - Cookbook sre.ganeti.makevm
• 15:51 vgutierrez: upgrade ATS in eqiad to version 8.0.8
• 15:42 ppchelko@deploy1001: Finished deploy [restbase/deploy@821e96b]: Only emit vary: accept-language for feeds when it matters T256358, more groups (duration: 05m 09s)
• 15:37 ppchelko@deploy1001: Started deploy [restbase/deploy@821e96b]: Only emit vary: accept-language for feeds when it matters T256358, more groups
• 15:37 ppchelko@deploy1001: Finished deploy [restbase/deploy@821e96b]: Only emit vary: accept-language for feeds when it matters T256358, more groups (duration: 03m 38s)
• 15:33 ppchelko@deploy1001: Started deploy [restbase/deploy@821e96b]: Only emit vary: accept-language for feeds when it matters T256358, more groups
• 15:33 ppchelko@deploy1001: Finished deploy [restbase/deploy@821e96b]: Only emit vary: accept-language for feeds when it matters T256358, more groups (duration: 03m 24s)
• 15:30 vgutierrez: upgrade ATS in codfw to version 8.0.8
• 15:30 dzahn@cumin1001: START - Cookbook sre.ganeti.makevm
• 15:30 ppchelko@deploy1001: Started deploy [restbase/deploy@821e96b]: Only emit vary: accept-language for feeds when it matters T256358, more groups
• 15:29 ppchelko@deploy1001: Finished deploy [restbase/deploy@821e96b]: Only emit vary: accept-language for feeds when it matters T256358, take 2 (duration: 06m 38s)
• 15:29 dzahn@cumin1001: START - Cookbook sre.ganeti.makevm
• 15:25 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: structured logging for xff log, stop logging jobrunner requests (duration: 01m 05s)
• 15:23 ppchelko@deploy1001: Started deploy [restbase/deploy@821e96b]: Only emit vary: accept-language for feeds when it matters T256358, take 2
• 15:20 ppchelko@deploy1001: Finished deploy [restbase/deploy@821e96b]: Only emit vary: accept-language for feeds when it matters T256358 (duration: 01m 37s)
• 15:19 ppchelko@deploy1001: Started deploy [restbase/deploy@821e96b]: Only emit vary: accept-language for feeds when it matters T256358
• 14:51 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:48 elukey@cumin1001: START - Cookbook sre.hosts.downtime
• 14:43 vgutierrez: upgrade ATS in esams to version 8.0.8
• 14:29 papaul: replacing mr1-codfw
• 14:24 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 14:19 vgutierrez: upgrade ATS in eqsin to version 8.0.8
• 14:19 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:17 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 14:12 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:05 marostegui: Stop MySQL on db2104 and db2088:3312
• 14:05 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2104', diff saved to https://phabricator.wikimedia.org/P11664 and previous config saved to /var/cache/conftool/dbconfig/20200625-140519-marostegui.json
• 14:04 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 14:04 marostegui@cumin2001: dbctl commit (dc=all): 'Depool db2088:3312', diff saved to https://phabricator.wikimedia.org/P11663 and previous config saved to /var/cache/conftool/dbconfig/20200625-140421-marostegui.json
• 13:59 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 13:57 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: T254301 Remove OAuthReplaceMessage hook subscriber (duration: 01m 05s)
• 13:56 vgutierrez: upgrade ATS in ulsfo to version 8.0.8
• 13:51 vgutierrez: upload trafficserver 8.0.8 to apt.wm.o (buster)
• 13:51 reedy@deploy1001: Synchronized wmf-config/CommonSettings-labs.php: Replace PasswordNotInLargeBlacklist with PasswordNotInCommonList (duration: 01m 05s)
• 13:49 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: Replace PasswordNotInLargeBlacklist with PasswordNotInCommonList (duration: 01m 06s)
• 13:40 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 13:36 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 13:30 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 13:28 godog: bounce logstash on logstash1007
• 13:26 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 13:19 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 13:13 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 13:02 moritzm: installing 4.9.210-1+deb9u1~deb8u1 on jessie hosts (fixed kernel for recent cacheoutattack CPU leaks)
• 12:55 elukey: rename notebook1003 to an-launcher1002 - T256363
• 12:51 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 12:47 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 12:45 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
• 12:44 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 12:44 elukey@cumin1001: START - Cookbook sre.hosts.decommission
• 12:42 moritzm: installing libmspack security updates
• 12:39 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 12:34 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 12:32 moritzm: installing libssh2 security updates
• 12:30 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 12:30 elukey@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 12:27 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 12:26 moritzm: installing libjpeg-turbo security updates
• 12:25 elukey@cumin1001: START - Cookbook sre.hosts.downtime
• 12:25 elukey@cumin1001: START - Cookbook sre.hosts.downtime
• 12:21 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 12:17 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 12:13 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 12:09 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 11:55 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 11:55 moritzm: installing python3.4 security updates
• 11:55 awight: EU BACON is cooked
• 11:51 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 11:50 awight@deploy1001: Synchronized wmf-config/InitialiseSettings.php: BACON: Enable QuickSurveys on metawiki (T253112) (duration: 01m 05s)
• 11:48 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 11:45 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 11:41 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 11:38 awight@deploy1001: Synchronized wmf-config/InitialiseSettings.php: BACON: Enable WMDE Tech Wishes survey configuration (T253112) (duration: 01m 09s)
• 11:36 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 11:27 moritzm: rolling reboot of ms-be[1044-1059].eqiad.wmnet
• 11:25 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 11:21 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 11:06 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 11:01 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 11:00 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 10:56 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 10:53 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 10:50 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 10:45 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 10:45 moritzm: rolling reboot of ms-be[2044-2056]
• 10:41 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 10:38 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 10:33 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 10:32 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 10:27 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 10:25 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 10:22 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 10:18 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 10:17 akosiaris@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 10:17 akosiaris@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 10:13 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 10:12 akosiaris@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 10:12 akosiaris@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 10:07 akosiaris@cumin1001: START - Cookbook sre.ganeti.makevm
• 10:07 akosiaris@cumin1001: START - Cookbook sre.ganeti.makevm
• 10:07 akosiaris@cumin1001: START - Cookbook sre.ganeti.makevm
• 10:07 akosiaris@cumin1001: START - Cookbook sre.ganeti.makevm
• 10:04 akosiaris: poweroff kubestagetcd1004 and ganeti1005 for T244530
• 10:04 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 10:00 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 10:00 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 10:00 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 10:00 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 09:59 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 09:58 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 09:57 akosiaris@cumin1001: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99)
• 09:57 akosiaris@cumin1001: START - Cookbook sre.ganeti.makevm
• 09:53 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 09:37 volans@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 09:34 volans@cumin1001: START - Cookbook sre.dns.netbox
• 09:28 akosiaris: schedule downtime for eqiad wikifeeds as it's flapping too much without yet knowing why. T256358
• 09:28 godog: extend lv on thanos-fe2001 and restart thanos-compact
• 09:21 vgutierrez: rolling restart of ncredir instances to catch up on kernel updates
• 09:13 joal@deploy1001: Finished deploy [analytics/refinery@4aba370] (thin): Analytics fix over weekly train THIN [analytics/refinery@4aba370] (duration: 00m 10s)
• 09:13 joal@deploy1001: Started deploy [analytics/refinery@4aba370] (thin): Analytics fix over weekly train THIN [analytics/refinery@4aba370]
• 09:13 joal@deploy1001: Finished deploy [analytics/refinery@4aba370]: Analytics fix over weekly train [analytics/refinery@4aba370] (duration: 16m 27s)
• 09:01 vgutierrez: restarting acme-chief instances to catch up on kernel updates
• 08:56 joal@deploy1001: Started deploy [analytics/refinery@4aba370]: Analytics fix over weekly train [analytics/refinery@4aba370]
• 08:42 hashar: releases2002: restarted bacula-fd to take in account the puppet provided configuration # T247652
• 08:14 jynus: restarting bacula-dir on backup1001
• 08:09 akosiaris: restart etherpad-lite on etherpad1002
• 08:03 marostegui: Failover m1 from db1135 to db1097 - T254556
• 07:52 jynus: stop bacula-director on backup1001 for db maintenance T254556
• 07:49 akosiaris@cumin1001: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99)
• 07:49 akosiaris@cumin1001: START - Cookbook sre.ganeti.makevm
• 07:49 akosiaris@cumin1001: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99)
• 07:49 akosiaris@cumin1001: START - Cookbook sre.ganeti.makevm
• 07:49 akosiaris@cumin1001: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99)
• 07:48 akosiaris@cumin1001: START - Cookbook sre.ganeti.makevm
• 07:48 akosiaris@cumin1001: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99)
• 07:47 akosiaris@cumin1001: START - Cookbook sre.ganeti.makevm
• 07:36 elukey: reboot an-launcher1001 for kernel upgrades
• 07:18 elukey: reboot kafkamon* vms for kernel upgrades
• 07:08 marostegui: Start pre switchover steps on m1 T254556
• 06:40 elukey: reboot matomo1002 for kernel upgrades
• 06:35 elukey: reboot archiva1002 (new vm, not yet in service) for kernel upgrades
• 06:34 elukey: reboot archiva for kernel upgrades
• 06:31 elukey: force puppet run on ores1003/1005 to restore celery (killed by the oom)
• 06:24 elukey: reboot an-tool* vms for kernel upgrades
• 06:23 elukey: reboot analytics-tool1004 for kernel upgrades (Superset host)
• 06:22 elukey: reboot analytics-tool1001 for kernel upgrades
• 06:19 elukey: execute ip addr flush ens5 on an-airflow1001 to clear RTNETLINK answers: File exists (error from ifup@ens5.service)
• 06:03 elukey: reboot an-airflow1001 for kernel upgrades
• 04:26 marostegui: Remove triggers from db2095:3312 - T238966
• 04:25 marostegui: Deploy schema change on s2 codfw - T238966
• 00:48 twentyafterfour: restart php-fpm on phab1001 to fix T256343
• 00:12 twentyafterfour: phabricator updated, all seems normal
• 00:11 twentyafterfour: updating phabricator to release/2020-06-25/1, momentary (<1 minute) downtime expected.

2020-06-24

• 23:44 mutante: releases2002 - systemctl stop jenkins, kill 15244 (rogue jenkins process), start jenkins with systemctl start jenkins (T247652)
• 23:43 mutante: releases1002 - kill rogue jenkins process, start jenkins with systemctl start jenkins (T247652)
• 23:02 mutante: releases1002/2002 - disabling puppet, removing failing cron job to pull deployment_charts (because /srv/deployment-charts does not exist yet)
• 21:45 shdubsh: install mtail 3.0.0~rc35+wmf2 on logstash1007 - T255776
• 20:42 brennen@deploy1001: Synchronized php: group1 wikis to 1.35.0-wmf.38 (duration: 01m 06s)
• 20:41 brennen@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.38
• 20:41 brennen: train 1.35.0-wmf.38: attempting to roll forward to group1 after php-fpm restart on mw1287 (T256305, T254175)
• 20:32 cdanis: restarting php-fpm on mw1287 T256305
• 20:32 bsitzmann@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'production' .
• 20:30 bsitzmann@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'mobileapps' for release 'production' .
• 20:28 bsitzmann@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'mobileapps' for release 'staging' .
• 20:14 halfak@deploy1001: Finished deploy [ores/deploy@1b87365]: T254505 (duration: 14m 08s)
• 20:09 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@80c763d]: Update mobileapps to a413db4f (duration: 03m 37s)
• 20:06 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@80c763d]: Update mobileapps to a413db4f
• 20:00 halfak@deploy1001: Started deploy [ores/deploy@1b87365]: T254505
• 19:38 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Revert Migrate SearchSatisfaction from EventLogging to EventGate on group1 - T249261 (duration: 01m 06s)
• 19:17 brennen@deploy1001: rebuilt and synchronized wikiversions files: Revert group1 wikis to 1.35.0-wmf.37
• 19:11 brennen@deploy1001: Synchronized php: group1 wikis to 1.35.0-wmf.38 (duration: 01m 04s)
• 19:10 brennen@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.38
• 19:01 brennen: train 1.35.0-wmf.38: finished triage meeting, clear to proceed to group 1 (T254175)
• 18:53 joal@deploy1001: Finished deploy [analytics/refinery@1112749] (thin): Regular analytics weekly train THIN [analytics/refinery@1112749] (duration: 00m 09s)
• 18:53 joal@deploy1001: Started deploy [analytics/refinery@1112749] (thin): Regular analytics weekly train THIN [analytics/refinery@1112749]
• 18:53 joal@deploy1001: Finished deploy [analytics/refinery@1112749]: Regular analytics weekly train [analytics/refinery@1112749] (duration: 05m 50s)
• 18:49 Urbanecm: Morning B&C deploy window is done
• 18:48 cstone: payments-wiki revision changed from 28ad76dcd7 to 91852dbc9b
• 18:47 Urbanecm: mwscript namespaceDupes.php --wiki=guwiki --fix (T255358)
• 18:47 joal@deploy1001: Started deploy [analytics/refinery@1112749]: Regular analytics weekly train [analytics/refinery@1112749]
• 18:46 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 2a1dfc5: Set namespace aliases for guwiki (T255358) (duration: 01m 05s)
• 18:42 Urbanecm: mwscript namespaceDupes.php --wiki=banwiki --add-prefix=T255941 --fix (T255941)
• 18:41 Urbanecm: Run mwscript namespaceDupes.php --wiki=banwiki --fix (T255941)
• 18:41 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: c6d6c85: Set WP as a NS_PROJECT alias for banwiki (T255941) (duration: 01m 06s)
• 18:38 Urbanecm: Run mwscript namespaceDupes.php dewiktionary --fix (T256242)
• 18:38 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 2b93e0f: Define Rekonstruktion NS for dewiktionary (T256242) (duration: 01m 05s)
• 18:29 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: dea9214: Revert "IS: Cleanup some redundant rows." (T256279) (duration: 01m 05s)
• 18:25 ppchelko@deploy1001: Synchronized wmf-config/InitialiseSettings.php: EventBus: Emit kafka purges for everything gerrit:607298 (duration: 01m 05s)
• 18:19 ppchelko@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable MediaModeration on group0 gerrit:607327 (duration: 01m 04s)
• 18:08 ppchelko@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable click tracking in Vector on beta cluster gerrit:607136 IS.php (duration: 01m 05s)
• 18:06 ppchelko@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: Enable click tracking in Vector on beta cluster gerrit:607136 IS-labs.php (duration: 01m 07s)
• 17:31 elukey: update archiva-ci user's password in Jenkins credentials plugin
• 16:56 elukey: update archiva-deploy user's password in Jenkins credentials plugin
• 16:46 ppchelko@deploy1001: Finished deploy [restbase/deploy@5f08f32]: Release PCS endpoints updates, feeds timed out, redo (duration: 05m 11s)
• 16:41 ppchelko@deploy1001: Started deploy [restbase/deploy@5f08f32]: Release PCS endpoints updates, feeds timed out, redo
• 16:40 ppchelko@deploy1001: Finished deploy [restbase/deploy@5f08f32]: Release PCS endpoints updates, take 2 (duration: 14m 11s)
• 16:34 brennen@deploy1001: Finished scap: (no justification provided) (duration: 60m 22s)
• 16:30 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 16:27 elukey@cumin1001: START - Cookbook sre.hosts.downtime
• 16:26 ppchelko@deploy1001: Started deploy [restbase/deploy@5f08f32]: Release PCS endpoints updates, take 2
• 16:17 elukey: reimage db1108 to debian Buster - T234826
• 15:53 ppchelko@deploy1001: Finished deploy [restbase/deploy@386b736]: Revert (duration: 27m 21s)
• 15:38 brennen: previous scap sync for T256151 - gerrit:607379 and gerrit:607380
• 15:36 kormat@cumin1001: dbctl commit (dc=all): 'Pool db1088 @ 100% into s6 T255927', diff saved to https://phabricator.wikimedia.org/P11652 and previous config saved to /var/cache/conftool/dbconfig/20200624-153604-kormat.json
• 15:34 brennen@deploy1001: Started scap: (no justification provided)
• 15:25 ppchelko@deploy1001: Started deploy [restbase/deploy@386b736]: Revert
• 15:24 ppchelko@deploy1001: deploy aborted: Release updates to PCS endpoints (duration: 05m 04s)
• 15:20 jayme: rolling restart of swift-proxy on thanos-fe[2001-2003].codfw.wmnet,thanos-fe[1001-1003].eqiad.wmnet - T256020
• 15:19 ppchelko@deploy1001: Started deploy [restbase/deploy@9686627]: Release updates to PCS endpoints
• 15:06 brennen: merging backports and running a full scap sync for UBN at T256151
• 15:00 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 14:57 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:57 moritzm: rebooting deneb for kernel update
• 14:57 ema: rmlist teampractices T255525
• 14:42 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Migrate SearchSatisfaction from EventLogging to EventGate on group0 - T249261 (duration: 01m 06s)
• 13:28 nikerabbit@deploy1001: Synchronized wmf-config/CommonSettings.php: [config] 603167 Remove TranslationNotifications user settings 1/2 (2nd attempt, now with correct file) (duration: 01m 06s)
• 13:23 marostegui: Deploy schema change on s6 eqiad primary master - T238966
• 12:59 jbond42: update metamonitoring to use icinga-extmon.wikimedia.org
• 12:23 akosiaris@cumin1001: conftool action : set/pooled=yes; selector: name=kubernetes1005.eqiad.wmnet
• 12:23 akosiaris@cumin1001: conftool action : set/pooled=yes; selector: name=kubernetes1006.eqiad.wmnet
• 12:19 akosiaris@cumin1001: conftool action : set/pooled=no; selector: name=kubernetes1006.eqiad.wmnet
• 12:19 akosiaris@cumin1001: conftool action : set/pooled=no; selector: name=kubernetes1005.eqiad.wmnet
• 12:19 akosiaris@cumin1001: conftool action : set/pooled=yes; selector: name=kubernetes2005.codfw.wmnet
• 12:19 akosiaris@cumin1001: conftool action : set/pooled=yes; selector: name=kubernetes2006.codfw.wmnet
• 12:17 akosiaris: depool/drain/reboot/pool kubernetes1005,6 for CPU capacity increase T256236
• 12:14 akosiaris: reboot kubernetes2005,6 for CPU capacity increase T256236
• 12:11 akosiaris: depool kubernetes2005,kubernetes2006 for CPU capacity increase T256236
• 12:10 akosiaris: depool kubernetes2005,kubernetes2006 for CPU capacity increase
• 12:05 akosiaris@cumin1001: conftool action : set/pooled=no; selector: name=kubernetes2006.codfw.wmnet
• 12:05 akosiaris@cumin1001: conftool action : set/pooled=no; selector: name=kubernetes2005.codfw.wmnet
• 12:04 awight: EU vegan BACON cooked
• 12:03 awight@deploy1001: Synchronized php-1.35.0-wmf.38/extensions/GrowthExperiments: BACON: Help panel home screen menu item fixes (T255254) (duration: 01m 06s)
• 11:40 nikerabbit@deploy1001: Synchronized private/PrivateSettings.php: Remove TranslationNotifications user settings 3/2 (duration: 01m 06s)
• 11:35 nikerabbit@deploy1001: Synchronized private/readme.php: [config] 607414 Remove TranslationNotifications user settings 2/2 (duration: 01m 04s)
• 11:28 nikerabbit@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [config] 603167 Remove TranslationNotifications user settings 1/2 (duration: 01m 03s)
• 11:09 awight@deploy1001: Synchronized wmf-config/CommonSettings.php: BACON: TwoColConflict: Talk page small deployment CommonSettings.php (T254458) (duration: 01m 17s)
• 10:45 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 10:39 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 10:38 marostegui: Stop haproxy on dbproxy1003 T256216
• 10:36 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 10:36 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 10:01 volans: Production management IP allocation must be done from Netbox from now on, see https://wikitech.wikimedia.org/wiki/DNS/Netbox#Cutoff_dates
• 09:55 volans@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 09:53 kormat@cumin1001: dbctl commit (dc=all): 'Pool db1088 @ 75% into s6 T255927', diff saved to https://phabricator.wikimedia.org/P11648 and previous config saved to /var/cache/conftool/dbconfig/20200624-095338-kormat.json
• 09:50 volans@cumin1001: START - Cookbook sre.dns.netbox
• 09:36 kormat@cumin1001: dbctl commit (dc=all): 'Pool db1088 @ 50% into s6 T255927', diff saved to https://phabricator.wikimedia.org/P11647 and previous config saved to /var/cache/conftool/dbconfig/20200624-093624-kormat.json
• 09:13 marostegui@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:10 marostegui@cumin2001: START - Cookbook sre.hosts.downtime
• 08:40 moritzm: prune remaining nginx packages on mw* servers T255565
• 08:31 kormat@cumin1001: dbctl commit (dc=all): 'Pool db1088 @ 20% into s6 T255927', diff saved to https://phabricator.wikimedia.org/P11645 and previous config saved to /var/cache/conftool/dbconfig/20200624-083120-kormat.json
• 08:06 moritzm: re-enable puppet in eqiad
• 08:04 marostegui@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 08:04 marostegui@cumin2001: START - Cookbook sre.hosts.downtime
• 08:00 moritzm: disable puppet in eqiad to unblock puppetdb1002 VM migration
• 07:22 gehel: restarting blazegraph on wdqs1007
• 06:53 moritzm: draining ganeti1009 for eventual reboot
• 06:28 XioNoX: enable peering BGP sessions on AMS-IX - T253970
• 05:59 XioNoX: disable peering BGP sessions on AMS-IX - T253970
• 05:34 marostegui@cumin2001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
• 05:33 marostegui@cumin2001: START - Cookbook sre.hosts.decommission
• 05:14 marostegui: Remove grants from dbproxy1008 - T231280 T255406
• 05:03 marostegui: Remove revision triggers from db1125:·3316
• 05:02 marostegui@cumin2001: dbctl commit (dc=all): 'Depool db1085 for MCR schema change', diff saved to https://phabricator.wikimedia.org/P11643 and previous config saved to /var/cache/conftool/dbconfig/20200624-050235-marostegui.json
• 04:53 marostegui: Reload haproxy on dbproxy1012 and dbproxy1014
• 00:35 ejegg: restarted fundraising jobs on main CiviCRM box
• 00:33 ejegg: updated Fundraising CiviCRM from f01b036128 to 52a32f2d66

2020-06-23

• 23:16 wkandek: releases1002 is back after being moved to row D (T255590)
• 23:11 dzahn@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 22:35 ejegg: disabled fundraising jobs on civi1001 for testing on civi2001
• 22:24 wkandek@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 22:13 AndyRussG: updated payments-wiki from 5fd4eb1519 to 28ad76dcd7
• 22:06 dzahn@cumin1001: START - Cookbook sre.ganeti.makevm
• 21:23 wkandek@cumin1001: START - Cookbook sre.ganeti.makevm
• 21:23 dzahn@cumin1001: END (ERROR) - Cookbook sre.ganeti.makevm (exit_code=97)
• 21:23 dzahn@cumin1001: START - Cookbook sre.ganeti.makevm
• 21:22 wkandek@cumin1001: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99)
• 21:22 wkandek@cumin1001: START - Cookbook sre.ganeti.makevm
• 21:22 dzahn@cumin1001: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99)
• 21:22 dzahn@cumin1001: START - Cookbook sre.ganeti.makevm
• 21:15 wkandek@cumin1001: END (ERROR) - Cookbook sre.hosts.decommission (exit_code=97)
• 21:14 wkandek@cumin1001: START - Cookbook sre.hosts.decommission
• 20:31 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Migrate TemplateWizard from EventLogging to EventGate on all wikis - take 2 - T238230 (duration: 01m 06s)
• 19:16 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Migrate TemplateWizard from EventLogging to EventGate on all wikis - T238230 (duration: 01m 05s)
• 19:06 brennen@deploy1001: rebuilt and synchronized wikiversions files: group0 wikis to 1.35.0-wmf.38
• 18:55 mutante: gerrit1001 (prod) - restarting gerrit service to verify config changes
• 18:53 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Migrate TemplateWizard from EventLogging to EventGate on group0 - T238230 (duration: 01m 06s)
• 18:24 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T254925 T246489 (duration: 01m 06s)
• 18:04 brennen@deploy1001: Finished scap: testwikis wikis to 1.35.0-wmf.38 (duration: 85m 53s)
• 16:39 brennen@deploy1001: Started scap: testwikis wikis to 1.35.0-wmf.38
• 16:01 brennen: 1.35.0-wmf.38 was branched at a35f7318 for https://phabricator.wikimedia.org/T254175
• 15:47 moritzm: prune nginx packages on mwdebug hosts T255565
• 15:37 moritzm: prune nginx packages on mw1380-mw1412 T255565
• 15:28 moritzm: installing libvpx security updates
• 15:27 mutante: removing ganeti VM xhgui1001 from eqiad row_A, will recreate in another row for rebalancing VMs between rows (T180761 T238098)
• 15:26 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
• 15:18 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
• 15:12 mutante: removing ganeti VM releases1002 in eqiad row_A - will recreate in another row to re-balance (T255590)
• 15:12 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
• 15:10 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
• 14:56 moritzm: failover ganeti master in eqiad to ganeti1011
• 14:55 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 14:50 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:48 urbanecm@deploy1001: Synchronized private/PrivateSettings.php: T250887 (duration: 00m 58s)
• 14:08 mholloway-shell@deploy1001: Finished deploy [recommendation-api/deploy@db7fd80]: Update recommendation-api to 7e00177 (duration: 03m 13s)
• 14:05 mholloway-shell@deploy1001: Started deploy [recommendation-api/deploy@db7fd80]: Update recommendation-api to 7e00177
• 13:54 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:54 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 13:54 jmm@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 13:54 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 13:34 moritzm: draining ganeti1012 for eventual reboot
• 13:34 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 13:28 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 12:56 jynus@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 12:54 jynus@cumin1001: START - Cookbook sre.hosts.downtime
• 12:45 moritzm: draining ganeti1011 for eventual reboot
• 12:45 marostegui: Deploy schema change on s6 codfw master (lag will appear on codfw) - T253276
• 12:36 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 12:31 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 12:00 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 11:56 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 11:35 awight: EU BACON cooked
• 11:34 awight@deploy1001: Synchronized php-1.35.0-wmf.37/extensions/TwoColConflict/: BACON: Fix broken copy link in JS mode (T253724) (duration: 00m 57s)
• 11:07 mlitn@deploy1001: Synchronized wmf-config/InitialiseSettings.php: test commons: Use the database name in the Wikibase entity source config (duration: 00m 59s)
• 11:04 moritzm: draining ganeti1008 for eventual reboot
• 10:58 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 10:55 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 10:42 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 10:42 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 10:40 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 10:38 moritzm: temporarily shutdown xhgui1001/releases1002 to reshuffle Ganeti instances for reboots
• 10:38 kormat@cumin1001: START - Cookbook sre.hosts.downtime
• 10:35 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 10:35 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 10:22 kormat: reimaging db1088 to buster T250666
• 10:03 jynus@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 10:01 jynus@cumin2001: START - Cookbook sre.hosts.downtime
• 09:48 jbond42: add new CI check for cloud yaml data https://gerrit.wikimedia.org/r/c/operations/puppet/+/606444/
• 09:46 jynus: stopping and reimaging db2101 into buster T254871
• 09:32 marostegui: Reload haproxy on dbproxy1012 and dbproxy1014 to test db1097 as secondary for 24h T254556
• 08:46 ema: mwmaint1002: add uid=abban,ou=people,dc=wikimedia,dc=org to group 'nda' T255775
• 08:38 XioNoX: re-enable peering BGP sessions on AMS-IX - T253970
• 08:03 moritzm: draining ganeti1007 for eventual reboot
• 07:58 XioNoX: restart scs-a8-eqiad - T256101
• 07:51 marostegui@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 07:49 marostegui@cumin2001: START - Cookbook sre.hosts.downtime
• 07:42 marostegui: Deploy schema change on db1088
• 07:30 marostegui: Reimage db2133 (m2 codfw master) to Buster (this will trigger haproxy IRC alert) T250666
• 07:01 marostegui@cumin2001: dbctl commit (dc=all): 'Fully repool db1118', diff saved to https://phabricator.wikimedia.org/P11637 and previous config saved to /var/cache/conftool/dbconfig/20200623-070120-marostegui.json
• 06:06 XioNoX: disable peering BGP sessions on AMS-IX - T253970
• 05:24 marostegui: Compress InnoDB on db1080 T254462
• 05:23 marostegui@cumin2001: dbctl commit (dc=all): 'Depool db1080 for InnoDB compression', diff saved to https://phabricator.wikimedia.org/P11636 and previous config saved to /var/cache/conftool/dbconfig/20200623-052350-marostegui.json
• 05:22 marostegui@cumin2001: dbctl commit (dc=all): 'Slowly repool db1118', diff saved to https://phabricator.wikimedia.org/P11635 and previous config saved to /var/cache/conftool/dbconfig/20200623-052254-marostegui.json
• 05:12 marostegui@cumin2001: dbctl commit (dc=all): 'Slowly repool db1118', diff saved to https://phabricator.wikimedia.org/P11634 and previous config saved to /var/cache/conftool/dbconfig/20200623-051159-marostegui.json
• 05:03 marostegui@cumin2001: dbctl commit (dc=all): 'Slowly repool db1118', diff saved to https://phabricator.wikimedia.org/P11633 and previous config saved to /var/cache/conftool/dbconfig/20200623-050314-marostegui.json

2020-06-22

• 23:41 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: touch for T247330 (duration: 00m 56s)
• 23:36 catrope@deploy1001: Synchronized dblists/: Close trwikinews (T247330) (duration: 00m 58s)
• 23:28 RoanKattouw: Synchronized wmf-config/InitialiseSettings.php: Create rollbacker group on elwiktionary (T255569) (typoed the task number before)
• 23:26 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Create rollbacker group on elwiktionary (T225569) (duration: 00m 56s)
• 23:21 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add localized sitename for bewikibooks (T253962) (duration: 00m 57s)
• 23:16 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add domains to wgCopyUploadsDomains (T255336, T255363, T255386, T255313) (duration: 01m 01s)
• 22:39 bstorm_: downtimed labstore1005 to prevent an alert during puppet merge T253353
• 22:38 volans@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 22:35 volans@cumin1001: START - Cookbook sre.dns.netbox
• 22:16 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@f2002c8]: bump glent jar to 0.2.2 (duration: 00m 56s)
• 22:15 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@f2002c8]: bump glent jar to 0.2.2
• 22:12 volans: cleanup interfaces and addresses in Netbox for offline servers - T233183
• 21:59 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@6e7f9f7]: bump glent jar to 0.2.2 (duration: 00m 18s)
• 21:58 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@6e7f9f7]: bump glent jar to 0.2.2
• 17:19 mutante: gerrit1002 - let puppet remove [database] secttion from config; restart gerrit another time
• 17:14 mutante: gerrit1002 (gerrit-test): re-enabled puppet, restarted gerrit service
• 16:58 volans@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 16:49 volans@cumin1001: START - Cookbook sre.dns.netbox
• 15:05 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 15:01 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:48 moritzm: installing mutt security updates
• 14:47 Amir1: creating shnwiktionary is done
• 14:44 ladsgroup@deploy1001: Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 02m 58s)
• 14:42 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 14:41 ladsgroup@deploy1001: Synchronized static/images/project-logos/: Creating shnwiktionary (T253029) (duration: 00m 56s)
• 14:40 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Creating shnwiktionary (T253029) (duration: 00m 56s)
• 14:39 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:37 ladsgroup@deploy1001: rebuilt and synchronized wikiversions files: Creating shnwiktionary (T253029)
• 14:36 ladsgroup@deploy1001: Synchronized dblists: Creating shnwiktionary (T253029) (duration: 00m 58s)
• 14:16 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 14:10 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:05 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 13:59 moritzm: re-enabling Puppet in codfw
• 13:55 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 13:51 moritzm: disable Puppet in codfw to reduce puppetdb2002 memory activity, unblocking the migration of the Ganeti instance for a reboot
• 13:19 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Bump eventlogging_Test schema version to 1.1.0 to pick up client_dt and set wgEventLoggingServiceUri for all wikis - T238230 (duration: 00m 58s)
• 13:11 marostegui: Stop MySQL on db2078 instances
• 12:53 vgutierrez: upgrade to trafficserver 8.0.8~rc0-1wm1 on cp5006 and cp5012
• 12:45 moritzm: draining ganeti2007 for eventual reboot
• 12:42 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 12:38 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 12:31 akosiaris: failover logstash2023 from ganeti2007->ganeti2023 for migration_downtime change to apply
• 12:26 volans@deploy1001: Finished deploy [homer/deploy@e9acec8]: Release v0.2.3 on cumin1001 now on buster (duration: 01m 25s)
• 12:24 volans@deploy1001: Started deploy [homer/deploy@e9acec8]: Release v0.2.3 on cumin1001 now on buster
• 12:22 volans@deploy1001: Finished deploy [homer/deploy@e9acec8]: Release v0.2.3 on cumin1001 now on buster (duration: 00m 03s)
• 12:22 volans@deploy1001: Started deploy [homer/deploy@e9acec8]: Release v0.2.3 on cumin1001 now on buster
• 11:53 Urbanecm: EU B&C window done
• 11:50 urbanecm@deploy1001: Synchronized php-1.35.0-wmf.37/extensions/VisualEditor/modules/: Backport: 0a08066: Revert "Allow generic params to be passed to getWikitextFragment" (T255785) (duration: 00m 58s)
• 11:45 marostegui@cumin2001: dbctl commit (dc=all): 'Fully repool db1094', diff saved to https://phabricator.wikimedia.org/P11627 and previous config saved to /var/cache/conftool/dbconfig/20200622-114554-marostegui.json
• 11:40 moritzm: draining ganeti2008 for eventual reboot
• 11:37 volans@deploy1001: Finished deploy [homer/deploy@e9acec8]: Release v0.2.3 on cumin1001 now on buster (duration: 00m 28s)
• 11:37 volans@deploy1001: Started deploy [homer/deploy@e9acec8]: Release v0.2.3 on cumin1001 now on buster
• 11:34 marostegui@cumin2001: dbctl commit (dc=all): 'Slowly repool db1094', diff saved to https://phabricator.wikimedia.org/P11625 and previous config saved to /var/cache/conftool/dbconfig/20200622-113401-marostegui.json
• 11:30 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 74e8295: IS: Cleanup some redundant rows (duration: 00m 56s)
• 11:29 Urbanecm: Run namespaceDupes.php for zh* projects (T165593)
• 11:24 marostegui@cumin2001: dbctl commit (dc=all): 'Slowly repool db1094', diff saved to https://phabricator.wikimedia.org/P11623 and previous config saved to /var/cache/conftool/dbconfig/20200622-112451-marostegui.json
• 11:24 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: db952ba: Add zh-hans and zh-hant translation of Module and Module_talk aliases for all Zh Projects (T165593) (duration: 00m 56s)
• 11:16 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 1301fd4: Add import sources for gomwiktionary (T255098) (duration: 00m 57s)
• 11:08 marostegui@cumin2001: dbctl commit (dc=all): 'Slowly repool db1094', diff saved to https://phabricator.wikimedia.org/P11622 and previous config saved to /var/cache/conftool/dbconfig/20200622-110806-marostegui.json
• 11:07 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: defa81e: Disable NS_USER(_TALK) search engine indexing on trwiki (T255538) (duration: 00m 58s)
• 10:35 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (606985) (duration: 00m 56s)
• 10:34 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (606985) (duration: 01m 12s)
• 09:58 marostegui@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:56 marostegui@cumin2001: START - Cookbook sre.hosts.downtime
• 09:33 marostegui@cumin2001: dbctl commit (dc=all): 'Depool db1094 for reimage', diff saved to https://phabricator.wikimedia.org/P11621 and previous config saved to /var/cache/conftool/dbconfig/20200622-093323-marostegui.json
• 09:31 godog: roll-restart logstash in codfw/eqiad to apply configuration change
• 08:59 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:56 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 08:33 moritzm: reimaging cumin1001 to buster T245114
• 08:13 godog: extend prometheus codfw ops filesystem to 1TB
• 08:02 vgutierrez: upgrade to trafficserver 8.0.8~rc0-1wm1 on cp4026 and cp4032
• 08:02 vgutierrez: upload trafficserver 8.0.8~rc0-1wm1 to apt.wm.o (buster)
• 07:33 marostegui@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 07:30 marostegui@cumin2001: START - Cookbook sre.hosts.downtime
• 07:16 marostegui: Reimage db1117 (irc haproxy alerts will be triggered)
• 06:26 marostegui@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 06:24 marostegui@cumin2001: START - Cookbook sre.hosts.downtime
• 06:06 marostegui: Stop MySQL on dbstore1005 for reimage to Buster - T254870
• 05:58 marostegui: Compress InnoDb on db1118 T254462
• 05:51 marostegui@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 05:49 marostegui@cumin2001: START - Cookbook sre.hosts.downtime
• 05:43 marostegui: Stop haproxy on dbproxy1008 - T255406
• 05:33 marostegui@cumin2001: dbctl commit (dc=all): 'Depool db1118 for reimage and InnoDB compression', diff saved to https://phabricator.wikimedia.org/P11617 and previous config saved to /var/cache/conftool/dbconfig/20200622-053334-marostegui.json
• 05:31 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1134', diff saved to https://phabricator.wikimedia.org/P11616 and previous config saved to /var/cache/conftool/dbconfig/20200622-053104-marostegui.json
• 05:17 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1134', diff saved to https://phabricator.wikimedia.org/P11615 and previous config saved to /var/cache/conftool/dbconfig/20200622-051730-marostegui.json
• 05:17 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1134', diff saved to https://phabricator.wikimedia.org/P11614 and previous config saved to /var/cache/conftool/dbconfig/20200622-051720-marostegui.json
• 05:03 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1134', diff saved to https://phabricator.wikimedia.org/P11613 and previous config saved to /var/cache/conftool/dbconfig/20200622-050259-marostegui.json
• 04:50 marostegui: Deploy schema change on s3 primary master with a big sleep between wikis - T250066
• 04:48 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1134', diff saved to https://phabricator.wikimedia.org/P11612 and previous config saved to /var/cache/conftool/dbconfig/20200622-044853-marostegui.json

2020-06-20

• 22:56 cdanis@cumin2001: dbctl commit (dc=all): 'db1088 seems to have crashed', diff saved to https://phabricator.wikimedia.org/P11611 and previous config saved to /var/cache/conftool/dbconfig/20200620-225624-cdanis.json
• 07:42 elukey: powercycle an-worker1093 - bug soft lock up CPU showed in mgmt console
• 07:36 elukey: powercycle an-worker1091 - bug soft lock up CPU showed in mgmt console

2020-06-19

• 18:10 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Bump eventlogging_Test schema version to 1.1.0 to pick up client_dt - T238230 (duration: 00m 59s)
• 16:07 mutante: ganeti4003 - rebooting install4001 - trying to bootstrap OS install from install2003
• 15:47 dzahn@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 15:28 godog: roll-restart kibana to apply new settings
• 13:01 moritzm: installing cups security updates (client side libs/tools)
• 12:31 qchris: Disabling puppet on gerrit1002 (test instance) to do some more testing
• 12:14 godog: delete march indices from logstash 5 eqiad to free up space
• 12:12 marostegui@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 12:10 marostegui@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 12:08 marostegui@cumin2001: START - Cookbook sre.hosts.downtime
• 12:07 marostegui@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 12:06 marostegui@cumin2001: START - Cookbook sre.hosts.downtime
• 12:05 marostegui@cumin2001: START - Cookbook sre.hosts.downtime
• 11:39 marostegui: Reimage db2116 db2119 db2130
• 10:55 moritzm: installing mesa security updates
• 10:49 godog: close april logstash indices on logstash 5 eqiad
• 10:45 moritzm: installing tomcat8 security updates
• 10:38 jayme: imported chartmuseum_0.12.0-1 to buster-wikimedia
• 10:24 marostegui@cumin2001: dbctl commit (dc=all): 'Repool db1093', diff saved to https://phabricator.wikimedia.org/P11604 and previous config saved to /var/cache/conftool/dbconfig/20200619-102447-marostegui.json
• 10:21 godog: start closing logstash indices for 2020.03 in elastic 5 eqiad
• 09:22 godog: restart elasticsearch on logstash1010
• 09:14 apergos: rsync from dumpsdata1003 as root to labstore1007 of dumps output files to catch up, with --bwlimit=160000 up from 80000
• 08:45 volans: backup netbox and run one-time script to reserve first IPs on all infra prefixes on Netbox - T233183
• 08:45 godog: roll restart elasticsearch_5@production-logstash-eqiad
• 08:26 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 08:21 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 08:18 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 08:15 godog: roll-restart logstash elk5 for "JVM GC Old generation-s runs" alert
• 08:12 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 08:00 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 07:59 marostegui@cumin2001: dbctl commit (dc=all): 'Depool db1093', diff saved to https://phabricator.wikimedia.org/P11601 and previous config saved to /var/cache/conftool/dbconfig/20200619-075907-marostegui.json
• 07:54 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 07:52 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 07:47 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 07:47 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 07:44 marostegui@cumin2001: dbctl commit (dc=all): 'Repool db1098:3316', diff saved to https://phabricator.wikimedia.org/P11600 and previous config saved to /var/cache/conftool/dbconfig/20200619-074420-marostegui.json
• 07:39 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 07:28 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 07:23 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 07:22 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 07:16 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 07:15 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 07:10 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 07:09 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 07:03 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 07:02 moritzm: rebooting ganeti nodes in eqiad for kernel security updates
• 06:57 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 06:51 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 06:47 moritzm: force reinstall of memcached 1.6 deb packages to ensure that the override is used in addition to the unmodified systemd unit from the deb T233933
• 06:39 marostegui@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 06:36 marostegui@cumin2001: START - Cookbook sre.hosts.downtime
• 06:20 marostegui: Stop mysql on db2132 to reimage m1 codfw master - T254556
• 06:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db2075 db2111', diff saved to https://phabricator.wikimedia.org/P11599 and previous config saved to /var/cache/conftool/dbconfig/20200619-061922-marostegui.json
• 06:05 marostegui@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 06:02 marostegui@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 06:01 marostegui@cumin2001: START - Cookbook sre.hosts.downtime
• 06:00 marostegui@cumin2001: START - Cookbook sre.hosts.downtime
• 05:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1112', diff saved to https://phabricator.wikimedia.org/P11598 and previous config saved to /var/cache/conftool/dbconfig/20200619-055430-marostegui.json
• 05:41 marostegui@cumin2001: dbctl commit (dc=all): 'Depool db2075 and db2111 for reimage', diff saved to https://phabricator.wikimedia.org/P11597 and previous config saved to /var/cache/conftool/dbconfig/20200619-054118-marostegui.json
• 05:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db2108', diff saved to https://phabricator.wikimedia.org/P11596 and previous config saved to /var/cache/conftool/dbconfig/20200619-053402-marostegui.json
• 05:25 marostegui@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 05:23 marostegui@cumin2001: START - Cookbook sre.hosts.downtime
• 04:44 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2108 for reimage', diff saved to https://phabricator.wikimedia.org/P11595 and previous config saved to /var/cache/conftool/dbconfig/20200619-044440-marostegui.json
• 04:39 marostegui@cumin2001: dbctl commit (dc=all): 'Depool db1098:3316', diff saved to https://phabricator.wikimedia.org/P11594 and previous config saved to /var/cache/conftool/dbconfig/20200619-043956-marostegui.json
• 04:35 marostegui@cumin2001: dbctl commit (dc=all): 'Depool db1112', diff saved to https://phabricator.wikimedia.org/P11593 and previous config saved to /var/cache/conftool/dbconfig/20200619-043554-marostegui.json

2020-06-18

• 22:30 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: EventLogging to EventGate: - SearchSatisfaction on all wikis - T249261 (duration: 00m 56s)
• 21:14 volans: start check-homer-diff.service on cumin2001 after merging the fix r/606526
• 20:17 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: EventLogging to EventGate: - SearchSatisfaction on all wikis - T249261 (duration: 00m 57s)
• 19:44 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: EventLogging to EventGate: - SearchSatisfaction on group1 wikis - T249261 (duration: 00m 57s)
• 18:53 wkandek@cumin1001: conftool action : set/pooled=yes; selector: name=mw2339.codfw.wmnet
• 18:35 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 18:32 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 17:16 wkandek@cumin1001: conftool action : set/pooled=no; selector: name=mw2339.codfw.wmnet
• 17:14 wkandek@cumin1001: conftool action : set/pooled=yes; selector: name=mw2339.codfw.wmnet
• 17:12 dzahn@cumin1001: conftool action : set/pooled=no; selector: name=mw2339.codfw.wmnet
• 16:51 maryum: reindex suspended until deployment of code
• 16:49 hnowlan: Shut off non-dockerised deployment-prep instance of changeprop
• 16:15 maryum: reindexing French wiki in Elasticsearch
• 15:37 Reedy: creatd bot_passwords tables on officeiwki and otrs_wikiwiki T254925 T246489
• 15:34 moritzm: installing harfbuzz security updates
• 15:23 moritzm: installing Ruby 2.1 security updates
• 15:15 moritzm: installing python-django security updates (packaged buster version)
• 15:04 moritzm: installing bind updates on jessie (client side tools/libs)
• 14:19 marostegui@cumin2001: dbctl commit (dc=all): 'Repool db1078', diff saved to https://phabricator.wikimedia.org/P11591 and previous config saved to /var/cache/conftool/dbconfig/20200618-141941-marostegui.json
• 14:14 moritzm: failover ganeti master in codfw to ganeti2021
• 14:03 marostegui@cumin2001: dbctl commit (dc=all): 'Depool db1078 for schema change', diff saved to https://phabricator.wikimedia.org/P11590 and previous config saved to /var/cache/conftool/dbconfig/20200618-140352-marostegui.json
• 14:02 marostegui@cumin2001: dbctl commit (dc=all): 'Repool db1075', diff saved to https://phabricator.wikimedia.org/P11589 and previous config saved to /var/cache/conftool/dbconfig/20200618-140203-marostegui.json
• 13:53 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:53 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 13:52 akosiaris: restart logstash2005 for applying an increased ganeti migration_downtime of 10k
• 13:47 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 13:43 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 13:40 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 13:34 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 13:11 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 13:05 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 12:52 marostegui@cumin2001: dbctl commit (dc=all): 'Depool db1075 for schema change', diff saved to https://phabricator.wikimedia.org/P11586 and previous config saved to /var/cache/conftool/dbconfig/20200618-125216-marostegui.json
• 12:48 marostegui@cumin1001: dbctl commit (dc=all): 'Remove weight from es5 master as es1024 is fully repooled now', diff saved to https://phabricator.wikimedia.org/P11585 and previous config saved to /var/cache/conftool/dbconfig/20200618-124801-marostegui.json
• 12:23 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 12:20 kormat@cumin1001: START - Cookbook sre.hosts.downtime
• 12:05 kormat: reimaging db1077 for final test T251768
• 11:51 jbond@deploy1001: Synchronized wmf-config/CommonSettings-labs.php: (no justification provided) (duration: 01m 00s)
• 11:06 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 11:00 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 10:34 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 10:34 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 09:54 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 09:48 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 09:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db2076', diff saved to https://phabricator.wikimedia.org/P11583 and previous config saved to /var/cache/conftool/dbconfig/20200618-094001-marostegui.json
• 09:39 akosiaris: update wikifeeds to latest chart version in codfw
• 09:39 akosiaris@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'wikifeeds' for release 'production' .
• 09:38 marostegui@cumin2001: dbctl commit (dc=all): 'Repool es2022', diff saved to https://phabricator.wikimedia.org/P11582 and previous config saved to /var/cache/conftool/dbconfig/20200618-093803-marostegui.json
• 09:38 akosiaris: uncordon kubernetes20{07..14} and kubernetes10{07..14}. Nodes are now fully put in rotation and ready to receive production traffic
• 09:34 marostegui: Deploy schema change on s3 codfw master (this will create lag on codfw) - T250066
• 09:31 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 09:30 godog: temp stop logstash on elk7 to test 8 pipeline workers - T255243
• 09:25 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 09:15 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 09:09 marostegui@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:08 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 09:06 marostegui@cumin2001: START - Cookbook sre.hosts.downtime
• 09:04 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 08:59 marostegui@cumin2001: dbctl commit (dc=all): 'Fully repool es1025', diff saved to https://phabricator.wikimedia.org/P11581 and previous config saved to /var/cache/conftool/dbconfig/20200618-085927-marostegui.json
• 08:59 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 08:50 ayounsi@cumin2001: END (FAIL) - Cookbook sre.network.prepare-upgrade (exit_code=1)
• 08:49 ayounsi@cumin2001: START - Cookbook sre.network.prepare-upgrade
• 08:49 ayounsi@cumin2001: END (FAIL) - Cookbook sre.network.prepare-upgrade (exit_code=99)
• 08:49 ayounsi@cumin2001: START - Cookbook sre.network.prepare-upgrade
• 08:49 marostegui@cumin2001: dbctl commit (dc=all): 'Slowly repool es1025', diff saved to https://phabricator.wikimedia.org/P11580 and previous config saved to /var/cache/conftool/dbconfig/20200618-084929-marostegui.json
• 08:47 marostegui@cumin2001: dbctl commit (dc=all): 'Depool es2022 for reimage', diff saved to https://phabricator.wikimedia.org/P11578 and previous config saved to /var/cache/conftool/dbconfig/20200618-084720-marostegui.json
• 08:47 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 08:40 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 08:37 marostegui@cumin2001: dbctl commit (dc=all): 'Slowly repool es1025', diff saved to https://phabricator.wikimedia.org/P11577 and previous config saved to /var/cache/conftool/dbconfig/20200618-083749-marostegui.json
• 08:25 elukey: change archiva-ci password in archiva
• 08:24 marostegui@cumin2001: dbctl commit (dc=all): 'Slowly repool es1025', diff saved to https://phabricator.wikimedia.org/P11576 and previous config saved to /var/cache/conftool/dbconfig/20200618-082432-marostegui.json
• 08:12 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 08:10 marostegui@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:08 marostegui@cumin2001: START - Cookbook sre.hosts.downtime
• 08:06 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 08:05 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 08:00 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 07:57 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 07:51 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 07:50 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 07:45 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 07:41 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 07:41 marostegui: Reimage es1025
• 07:35 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 07:34 marostegui@cumin2001: dbctl commit (dc=all): 'Repool db1136', diff saved to https://phabricator.wikimedia.org/P11574 and previous config saved to /var/cache/conftool/dbconfig/20200618-073414-marostegui.json
• 07:33 ayounsi@cumin2001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 07:31 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 07:25 ayounsi@cumin2001: START - Cookbook sre.dns.netbox
• 07:24 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 07:22 moritzm: rolling reboot of ganeti servers in codfw
• 07:10 ayounsi@cumin1001: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
• 07:07 ayounsi@cumin1001: START - Cookbook sre.dns.netbox
• 04:50 marostegui@cumin2001: dbctl commit (dc=all): 'Depool db1136', diff saved to https://phabricator.wikimedia.org/P11573 and previous config saved to /var/cache/conftool/dbconfig/20200618-045047-marostegui.json

2020-06-17

• 23:25 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 0e7079d: Install DiscussionTools on all wikis (attempt 2) (T252264; T253943) (duration: 00m 56s)
• 23:23 urbanecm@deploy1001: Synchronized php-1.35.0-wmf.36/extensions/DiscussionTools/includes/Hooks.php: ff01083: Use $wgLocaltimezone global instead of request context (T255704) (duration: 00m 57s)
• 23:21 urbanecm@deploy1001: Synchronized php-1.35.0-wmf.37/extensions/DiscussionTools/includes/Hooks.php: 4551d29: Use $wgLocaltimezone global instead of request context (T252264; T253943; T255704) (duration: 00m 58s)
• 23:01 ryankemper@deploy1001: Finished deploy [wdqs/wdqs@79fb82f]: 0.3.39 (duration: 14m 38s)
• 22:47 ryankemper@deploy1001: Started deploy [wdqs/wdqs@79fb82f]: 0.3.39
• 21:01 ryankemper@cumin2001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
• 20:32 hashar: Fixed up zuul-merger on contint1001 due to some faulty hotfix
• 20:08 hashar: Stopped zuul-merger on contint1001
• 19:21 marostegui: Deploy schema change on s6 codfw master T238966
• 19:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1094', diff saved to https://phabricator.wikimedia.org/P11572 and previous config saved to /var/cache/conftool/dbconfig/20200617-191723-marostegui.json
• 19:11 ryankemper@cumin2001: START - Cookbook sre.wdqs.data-transfer
• 19:08 ryankemper@cumin2001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
• 19:05 ryankemper@cumin2001: START - Cookbook sre.wdqs.data-transfer
• 18:57 milimetric@deploy1001: Finished deploy [analytics/refinery@6640d6f] (thin): Quick fix for data quality bundles (THIN) (duration: 00m 10s)
• 18:57 milimetric@deploy1001: Started deploy [analytics/refinery@6640d6f] (thin): Quick fix for data quality bundles (THIN)
• 18:52 ryankemper@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
• 18:49 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-transfer
• 18:44 milimetric@deploy1001: Finished deploy [analytics/refinery@6640d6f]: Quick fix for data quality bundles (duration: 27m 55s)
• 18:41 Urbanecm: Morning B&C window done
• 18:31 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 96153f9: Add temporary logging for mediamoderation (T247943) (duration: 00m 56s)
• 18:24 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: REVERT: ae76450: Install DiscussionTools on all wikis (T252264; T253943) (duration: 00m 34s)
• 18:22 urbanecm@deploy1001: scap failed: average error rate on 3/9 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/e474f13ffac6b8c3bf919c4aeafc8c9b for details)
• 18:21 urbanecm@deploy1001: scap failed: average error rate on 9/9 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/e474f13ffac6b8c3bf919c4aeafc8c9b for details)
• 18:16 milimetric@deploy1001: Started deploy [analytics/refinery@6640d6f]: Quick fix for data quality bundles
• 18:14 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: c9f6452: Set DiscussionToolsEnableVisual to true by default (T251654) (duration: 00m 56s)
• 18:05 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
• 18:04 elukey@cumin1001: START - Cookbook sre.hosts.decommission
• 16:57 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: EventLogging to EventGate: - SearchSatisfaction on group0 wikis - T249261 (duration: 00m 56s)
• 16:00 marostegui@cumin2001: dbctl commit (dc=all): 'Depool db1094', diff saved to https://phabricator.wikimedia.org/P11571 and previous config saved to /var/cache/conftool/dbconfig/20200617-160013-marostegui.json
• 15:28 godog: temp bump logstash7 workers to 8 and temp stop logstash - T255243
• 15:17 jforrester@deploy1001: Synchronized private/PrivateSettings.php: T247943 Add API key and recipient config for MediaModeration (duration: 00m 55s)
• 15:17 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw2338.codfw.wmnet
• 15:11 dzahn@cumin1001: conftool action : set/weight=15; selector: name=mw233[5-9].codfw.wmnet
• 15:11 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: T247943 Install MediaModeration extension - III: Install where enabled (duration: 00m 56s)
• 15:10 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw2335.codfw.wmnet
• 15:09 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw2336.codfw.wmnet
• 15:09 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw2337.codfw.wmnet
• 15:09 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw2339.codfw.wmnet
• 15:08 dzahn@cumin1001: conftool action : set/pooled=no; selector: name=mw233[5-9].codfw.wmnet
• 14:58 jforrester@deploy1001: Synchronized php-1.35.0-wmf.37/extensions/GrowthExperiments/modules/help/ext.growthExperiments.HelpPanelProcessDialog.js: T255607 Fix help panel sizing logic (duration: 00m 56s)
• 14:54 hnowlan@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' .
• 14:52 hnowlan@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' .
• 14:52 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:50 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 14:49 mdholloway: rolled back recommendation-api deployment due to canary endpoint check failure (T255683)
• 14:44 mholloway-shell@deploy1001: Finished deploy [recommendation-api/deploy@c39d567]: Update recommendation-api to db97742 (duration: 01m 16s)
• 14:43 mholloway-shell@deploy1001: Started deploy [recommendation-api/deploy@c39d567]: Update recommendation-api to db97742
• 14:30 akosiaris: redrain kubernetes1007-14
• 14:27 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:27 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 14:27 mutante: disabling puppet on icinga to avoid alert spam when adding new appservers
• 14:25 akosiaris@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'wikifeeds' for release 'production' .
• 14:22 akosiaris: uncordon kubernetes10{07..14} again
• 14:13 mutante: generating new mcrouter certs for mw2335 - mw2339 (T247021)
• 14:02 mutante: rebooting mw2335 through mw2339 (not in service)
• 13:51 XioNoX: cleanup msw1-codfw interfaces
• 13:44 akosiaris: redrain kubernetes1007-14
• 13:37 akosiaris@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'mathoid' for release 'production' .
• 13:35 akosiaris@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'blubberoid' for release 'production' .
• 13:31 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: EventLogging to EventGate: - SearchSatisfaction on testwiki version 1.1.0 - T249261 (duration: 00m 58s)
• 13:30 moritzm: upgrade remaining parsoid nodes to PHP 7.2.31
• 13:21 jbond42: re-enable puppet on C:memcached nodes
• 13:04 marostegui@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:04 marostegui: The above db1129 depool was meant to be a repool, wrong commit message
• 13:03 liw@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.37
• 13:03 jbond42: disable puppet on C:memcache to deploy a new change
• 13:02 marostegui@cumin2001: dbctl commit (dc=all): 'Depool db1129', diff saved to https://phabricator.wikimedia.org/P11567 and previous config saved to /var/cache/conftool/dbconfig/20200617-130236-marostegui.json
• 13:02 akosiaris@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 13:00 marostegui@cumin2001: START - Cookbook sre.hosts.downtime
• 13:00 akosiaris@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 13:00 akosiaris@cumin2001: START - Cookbook sre.hosts.downtime
• 13:00 akosiaris@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 13:00 akosiaris@cumin2001: START - Cookbook sre.hosts.downtime
• 13:00 akosiaris@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 12:59 akosiaris@cumin2001: START - Cookbook sre.hosts.downtime
• 12:59 akosiaris@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 12:59 akosiaris@cumin2001: START - Cookbook sre.hosts.downtime
• 12:59 akosiaris@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 12:59 akosiaris@cumin2001: START - Cookbook sre.hosts.downtime
• 12:59 akosiaris@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 12:59 akosiaris@cumin2001: START - Cookbook sre.hosts.downtime
• 12:59 akosiaris@cumin2001: START - Cookbook sre.hosts.downtime
• 12:54 hnowlan: upgraded cpjobqueue to newer container image, rolled back
• 12:40 marostegui@cumin2001: dbctl commit (dc=all): 'Add db2091 to s8 T253217', diff saved to https://phabricator.wikimedia.org/P11566 and previous config saved to /var/cache/conftool/dbconfig/20200617-124034-marostegui.json
• 12:32 hnowlan: Removed remaining changeprop systemd components from scb
• 12:06 marostegui@cumin2001: dbctl commit (dc=all): 'Depool db2076 to remove triggers from sanitarium T238966', diff saved to https://phabricator.wikimedia.org/P11565 and previous config saved to /var/cache/conftool/dbconfig/20200617-120622-marostegui.json
• 11:59 Amir1: not today, just EU noon
• 11:59 Amir1: B&C is done for today
• 11:58 ladsgroup@deploy1001: Synchronized wmf-config/config/trwikisource.yaml: Change sidebar upload link destination for tr.wikisource (T253490) (duration: 01m 03s)
• 11:55 ladsgroup@deploy1001: Synchronized dblists/commonsuploads.dblist: Change sidebar upload link destination for tr.wikisource (T253490) (duration: 01m 04s)
• 11:48 hnowlan@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' .
• 11:47 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add extended-confirmed group and restriction level for rowiki (T254471) (duration: 01m 04s)
• 11:30 marostegui@cumin1001: dbctl commit (dc=all): 'Depool es1025 for reimage, give weight to es1023 (es5 master)', diff saved to https://phabricator.wikimedia.org/P11563 and previous config saved to /var/cache/conftool/dbconfig/20200617-113026-marostegui.json
• 11:23 ladsgroup@deploy1001: Synchronized php-1.35.0-wmf.37/extensions/GrowthExperiments/extension.json: Fix NewcomerTask schema (T255597) (duration: 01m 04s)
• 11:18 ladsgroup@deploy1001: Synchronized php-1.35.0-wmf.36/extensions/GrowthExperiments/extension.json: Fix NewcomerTask schema (T255597) (duration: 01m 06s)
• 11:07 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Set hiwiktionary timezone to Asia/Kolkata (T255531) (duration: 01m 05s)
• 10:48 marostegui@cumin2001: dbctl commit (dc=all): 'Remove db2091 from dbctl in s2 and s4', diff saved to https://phabricator.wikimedia.org/P11562 and previous config saved to /var/cache/conftool/dbconfig/20200617-104816-marostegui.json
• 10:40 marostegui@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 10:38 marostegui@cumin2001: START - Cookbook sre.hosts.downtime
• 10:31 liw@deploy1001: Synchronized php: group1 wikis to 1.35.0-wmf.37 (duration: 01m 04s)
• 10:30 liw@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.37
• 09:44 marostegui@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:42 marostegui@cumin2001: START - Cookbook sre.hosts.downtime
• 09:40 hnowlan: killing stale changeprop instances running on scb hosts
• 09:16 jforrester@deploy1001: Synchronized php-1.35.0-wmf.37/extensions/Flow/: T255608 Revert 'Hooks: Use PageMoveComplete instead of TitleMoveCompleting' (duration: 01m 05s)
• 09:15 marostegui@cumin2001: dbctl commit (dc=all): 'Fully repool db1113:3315, db1113:3316', diff saved to https://phabricator.wikimedia.org/P11558 and previous config saved to /var/cache/conftool/dbconfig/20200617-091509-marostegui.json
• 09:11 jforrester@deploy1001: Synchronized php-1.35.0-wmf.37/includes/HookContainer/DeprecatedHooks.php: T255608 Revert 'Hard deprecate the hook' (duration: 01m 05s)
• 09:02 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T247943 Install MediaModeration extension - II: Add flag to IS (duration: 01m 05s)
• 08:56 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:56 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 08:52 marostegui@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:49 marostegui@cumin2001: START - Cookbook sre.hosts.downtime
• 08:47 marostegui@cumin2001: dbctl commit (dc=all): 'Slowly repool db1113:3315, db1113:3316', diff saved to https://phabricator.wikimedia.org/P11557 and previous config saved to /var/cache/conftool/dbconfig/20200617-084751-marostegui.json
• 08:44 marostegui@cumin2001: dbctl commit (dc=all): 'Slowly repool db1113:3315, db1113:3316', diff saved to https://phabricator.wikimedia.org/P11556 and previous config saved to /var/cache/conftool/dbconfig/20200617-084402-marostegui.json
• 08:43 jforrester@deploy1001: Synchronized php-1.35.0-wmf.37/includes/EditPage.php: T255177 T255614 Do not return internal edit status from EditPage (duration: 01m 08s)
• 08:31 marostegui@cumin2001: dbctl commit (dc=all): 'Slowly repool db1113:3315, db1113:3316', diff saved to https://phabricator.wikimedia.org/P11554 and previous config saved to /var/cache/conftool/dbconfig/20200617-083120-marostegui.json
• 08:30 godog: start logstash on logstash7 - T255243
• 08:29 moritzm: prune nginx from remaining mw* servers in codfw T255565
• 08:23 marostegui@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:20 marostegui@cumin2001: START - Cookbook sre.hosts.downtime
• 08:10 godog: stop logstash temporarily on logstash7 hosts to test increased es shards - T255243
• 08:05 marostegui@cumin2001: dbctl commit (dc=all): 'Depool db1113:3315 db1113:3316', diff saved to https://phabricator.wikimedia.org/P11553 and previous config saved to /var/cache/conftool/dbconfig/20200617-080511-marostegui.json
• 07:53 elukey: reboot kafka-jumbo1009 for kernel upgrades
• 06:40 elukey: reboot krb1001 for kernel upgrades
• 06:24 elukey: reboot an-master100[1,2] for kernel upgrades
• 06:23 XioNoX: set lacp active on cr2-esams:ae2 - T253970
• 06:15 tstarling@deploy1001: Synchronized wmf-config/PoolCounterSettings.php: test fast stale mode on testwiki T250248 (duration: 01m 17s)
• 06:03 elukey: reboot an-conf100[1-3] for kernel upgrades
• 05:45 elukey: reboot stat1007/8 for kernel upgrades
• 05:45 elukey: clean up old systemd timer config on an-coord1001 (came up after the last reboot)
• 05:42 volker-e@deploy1001: Finished deploy [design/style-guide@37c67dd]: Deploy design/style-guide: (duration: 00m 05s)
• 05:42 volker-e@deploy1001: Started deploy [design/style-guide@37c67dd]: Deploy design/style-guide:
• 05:34 marostegui@cumin2001: dbctl commit (dc=all): 'Fully repool db1090:3312, db1090:3317', diff saved to https://phabricator.wikimedia.org/P11552 and previous config saved to /var/cache/conftool/dbconfig/20200617-053421-marostegui.json
• 05:29 marostegui: Deploy schema change on s7 codfw (lag will appear) - T250066
• 05:28 marostegui@cumin2001: dbctl commit (dc=all): 'Slowly repool db1090:3312, db1090:3317', diff saved to https://phabricator.wikimedia.org/P11551 and previous config saved to /var/cache/conftool/dbconfig/20200617-052809-marostegui.json
• 05:22 marostegui@cumin2001: dbctl commit (dc=all): 'Slowly repool db1090:3312, db1090:3317', diff saved to https://phabricator.wikimedia.org/P11550 and previous config saved to /var/cache/conftool/dbconfig/20200617-052202-marostegui.json
• 05:19 marostegui@cumin2001: dbctl commit (dc=all): 'Slowly repool db1090:3312, db1090:3317', diff saved to https://phabricator.wikimedia.org/P11549 and previous config saved to /var/cache/conftool/dbconfig/20200617-051916-marostegui.json
• 05:10 marostegui@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 05:08 marostegui@cumin2001: START - Cookbook sre.hosts.downtime
• 04:51 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1090:3312, db1090:3317 for reimage', diff saved to https://phabricator.wikimedia.org/P11548 and previous config saved to /var/cache/conftool/dbconfig/20200617-045105-marostegui.json
• 04:44 marostegui: Reload pt-kill on labsdb analytics host to pick up new config
• 04:38 marostegui@cumin2001: dbctl commit (dc=all): 'Depool db1129', diff saved to https://phabricator.wikimedia.org/P11547 and previous config saved to /var/cache/conftool/dbconfig/20200617-043826-marostegui.json
• 01:43 shdubsh: restart elasticsearch on logstash1011

2020-06-16

• 23:43 crusnov@deploy1001: Finished deploy [netbox/deploy@5251cf1]: Deploying Netbox to netbox-dev T253140 (duration: 00m 05s)
• 23:43 crusnov@deploy1001: Started deploy [netbox/deploy@5251cf1]: Deploying Netbox to netbox-dev T253140
• 23:35 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: cirrus: update ML models for ko and zh, drop ja (duration: 01m 00s)
• 23:34 ebernhardson@deploy1001: sync-file aborted: cirrus: update ML models for ko and zh, drop ja (duration: 00m 04s)
• 22:40 krinkle@deploy1001: Synchronized src/Noc/: (no justification provided) (duration: 01m 04s)
• 22:31 krinkle@deploy1001: Synchronized docroot/noc: (no justification provided) (duration: 01m 05s)
• 21:12 krinkle@deploy1001: Synchronized php-1.35.0-wmf.37/extensions/WikimediaEvents/modules/: I67794c (duration: 01m 04s)
• 20:42 brennen@deploy1001: rebuilt and synchronized wikiversions files: Revert group1 wikis to 1.35.0-wmf.37
• 20:41 foks: reset email and pw for CactusJack
• 20:32 brennen: rolling 1.35.0-wmf.37 back to group0
• 20:29 mutante: signing puppet cert requests for releases1002 and releases2002 - T255590
• 19:24 brennen@deploy1001: Synchronized php: group1 wikis to 1.35.0-wmf.37 (duration: 01m 04s)
• 19:23 brennen@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.37
• 19:18 otto@deploy1001: Started deploy [analytics/refinery@8b8ce6e]: deploying refinery source 0.0.127 for eventlogging -> eventgate migration - T249261
• 19:15 brennen@deploy1001: Synchronized php-1.35.0-wmf.37/skins/Vector/resources/skins.vector.styles/: Restore Watchlist star (duration: 01m 05s)
• 19:03 brennen: CORRECTION: holding _1.35.0-wmf.37_ deploy to group1 for a few minutes while merging & testing fix for T255574
• 19:01 brennen: holding 1.35.0-wmf.27 deploy to group1 for a few minutes while merging & testing fix for T255574
• 18:59 dzahn@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 18:52 qchris: Turning on puppet again on gerrit1002 to avoid having it lag too far behind.
• 18:32 dzahn@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 18:18 mutante: mw2293 - scap pull (because Icinga reports mismatched MW versions)
• 18:01 crusnov@cumin2001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 17:55 dzahn@cumin1001: START - Cookbook sre.ganeti.makevm
• 17:52 crusnov@cumin2001: START - Cookbook sre.ganeti.makevm
• 17:44 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@f4f5d7b]: airflow: adjust glent legal cutoff (duration: 01m 35s)
• 17:42 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@f4f5d7b]: airflow: adjust glent legal cutoff
• 17:32 dzahn@cumin1001: START - Cookbook sre.ganeti.makevm
• 17:03 herron: performing rolling reboots of kafka-main hosts for security updates T254990
• 16:27 hnowlan@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop' for release 'production' .
• 16:26 hnowlan: Updating changeprop to new container version with updated dependencies
• 16:07 hnowlan@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'changeprop' for release 'production' .
• 16:04 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' .
• 16:02 elukey: reboot kafka-jumbo1008 for kernel upgrades
• 15:58 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' .
• 15:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1076', diff saved to https://phabricator.wikimedia.org/P11543 and previous config saved to /var/cache/conftool/dbconfig/20200616-154924-marostegui.json
• 15:45 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@7d4458c]: Reduce glent maximum yarn resource usage to reasonable levels (duration: 00m 41s)
• 15:44 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@7d4458c]: Reduce glent maximum yarn resource usage to reasonable levels
• 15:26 milimetric@deploy1001: Finished deploy [analytics/refinery@c652f62] (thin): Regular analytics weekly THIN train [analytics/refinery@c652f62] (duration: 00m 08s)
• 15:25 milimetric@deploy1001: Started deploy [analytics/refinery@c652f62] (thin): Regular analytics weekly THIN train [analytics/refinery@c652f62]
• 15:23 milimetric@deploy1001: Finished deploy [analytics/refinery@c652f62]: Regular analytics weekly train [analytics/refinery@c652f62] (duration: 07m 56s)
• 15:20 elukey: reboot kafka-jumbo1007 for kernel upgrades
• 15:15 moritzm: upgrading intel-microcode on jessie hosts
• 15:15 milimetric@deploy1001: Started deploy [analytics/refinery@c652f62]: Regular analytics weekly train [analytics/refinery@c652f62]
• 15:06 elukey: reboot an-coord1001 for kernel upgrades
• 14:49 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' .
• 14:49 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 14:45 moritzm: rebooting scandium for kernel security update
• 14:45 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:43 cdanis: repool eqiad T243080
• 14:40 papaul: power off ms-be2018 for BBU replacement
• 14:33 cdanis: eqiad router upgrades completed! 🎉 T243080
• 14:33 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 14:31 elukey: reboot druid100[7,8] for kernel upgrades
• 14:28 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:25 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 14:22 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:15 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1076', diff saved to https://phabricator.wikimedia.org/P11541 and previous config saved to /var/cache/conftool/dbconfig/20200616-141540-marostegui.json
• 14:14 cdanis: T243080 cdanis@re1.cr2-eqiad> request chassis routing-engine master switch
• 14:11 moritzm: removing stray nginx packages from mw canaries (mw1261-mw1265 and mw1276-mw1283) T255565
• 14:06 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
• 14:03 akosiaris@cumin1001: START - Cookbook sre.hosts.decommission
• 14:03 akosiaris@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1)
• 14:03 akosiaris@cumin1001: START - Cookbook sre.hosts.decommission
• 14:03 akosiaris@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99)
• 14:03 akosiaris@cumin1001: START - Cookbook sre.hosts.decommission
• 13:56 cdanis: T243080 cdanis@re0.cr2-eqiad> request chassis routing-engine master switch
• 13:50 cdanis: cr2-eqiad: rebooting RE1 [backup] with new junos version T243080
• 13:39 cdanis: cr2-eqiad: disable transit/peering BGP & bump fr MED T243080
• 13:32 marostegui@cumin2001: dbctl commit (dc=all): 'Repool db2092 T254462', diff saved to https://phabricator.wikimedia.org/P11535 and previous config saved to /var/cache/conftool/dbconfig/20200616-133241-marostegui.json
• 13:17 XioNoX: pfw3-eqiad rollback MED to cr1 to 0 - T243080
• 13:12 XioNoX: add graceful-switchover to cr1-eqiad
• 13:09 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
• 13:08 liw@deploy1001: rebuilt and synchronized wikiversions files: group0 wikis to 1.35.0-wmf.37
• 13:06 akosiaris@cumin1001: START - Cookbook sre.hosts.decommission
• 13:03 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:03 cdanis: T243080 cdanis@re1.cr1-eqiad> request chassis routing-engine master switch
• 13:03 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 13:03 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:03 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 13:01 moritzm: rebooting mw2291-mw2334
• 12:54 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' .
• 12:51 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' .
• 12:47 jbond42: upload new memcache package with TLS to component/memcached16 in buster-wikimedia
• 12:42 XioNoX: pfw3-eqiad set MED to cr1 to 300 - T243080
• 12:38 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' .
• 12:31 cdanis: T243080 cr1-eqiad: request chassis routing-engine master switch
• 12:31 cdanis: cr1-eqiad: request chassis routing-engine master switch
• 12:25 cdanis: cr1-eqiad: rebooting RE1 [backup] with new junos version T243080
• 12:15 cdanis: cdanis@re0.cr1-eqiad# commit confirmed 2 comment "force VRRP failover T243080"
• 12:14 cdanis: disable transit/peering & increase frack MED on cr1-eqiad T243080
• 12:09 hnowlan@cumin2001: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0)
• 11:48 cdanis: depooling eqiad for router upgrade T243080
• 11:42 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 11:42 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 11:42 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 11:42 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 11:42 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 11:42 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 11:42 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 11:41 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 11:41 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 11:41 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 11:41 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 11:41 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 11:41 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 11:41 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 11:41 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 11:41 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 11:41 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 11:41 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 11:41 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 11:41 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 11:40 hnowlan: roll-restarting restbase201[0-2] for cert updates
• 11:40 hnowlan@cumin2001: START - Cookbook sre.cassandra.roll-restart
• 11:39 jmm@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 11:39 jmm@cumin1001: START - Cookbook sre.hosts.downtime
• 11:38 hnowlan@cumin2001: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0)
• 11:35 elukey: reboot an-druid100[1,2] for kernel upgrades
• 11:27 hnowlan: roll-restart restbase2009 for cert update
• 11:26 hnowlan@cumin2001: START - Cookbook sre.cassandra.roll-restart
• 11:21 hnowlan@cumin1001: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0)
• 11:18 jforrester@deploy1001: Synchronized dblists/mobilemainpagelegacy.dblist: T32405 T254731 Drop mobile special casing of main page for simplewiki, itwikisource, vecwikisource (duration: 01m 05s)
• 11:15 moritzm: updating perf on stretch hosts
• 11:14 marostegui: Deploy MCR schema change on db2087:3316
• 11:09 moritzm: updating perf on buster
• 11:02 moritzm: rebooting mw2350-mw2376
• 11:01 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Stop setting wgActorTableSchemaMigrationStage, no longer read in core (duration: 01m 05s)
• 10:52 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Stop setting wgTagStatisticsNewTable, no longer read in core (duration: 01m 04s)
• 10:51 hnowlan: roll-restarting restbase101[6-8].eqiad.wmnet for cert updates
• 10:50 hnowlan@cumin1001: START - Cookbook sre.cassandra.roll-restart
• 10:44 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Stop setting wgChangeTagsSchemaMigrationStage, no longer read in core (duration: 01m 06s)
• 10:26 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Stop setting wgCommentTableSchemaMigrationStage, no longer read in core (duration: 01m 07s)
• 09:54 volans: restarting netbox to pickup modified customscripts
• 09:14 filippo@cumin1001: conftool action : set/pooled=true; selector: dnsdisc=thanos-swift,name=eqiad
• 08:53 godog: roll restart prometheus eqiad ops to enable thanos upload
• 08:48 marostegui: Upgrade db2132
• 08:44 marostegui@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:42 marostegui@cumin2001: START - Cookbook sre.hosts.downtime
• 08:39 liw@deploy1001: Finished scap: testwikis wikis to 1.35.0-wmf.37 (duration: 59m 05s)
• 08:19 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:19 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 08:19 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:19 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 08:19 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:19 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 08:18 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:18 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 08:18 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:18 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 08:18 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:18 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 08:09 volans@deploy1001: Finished deploy [homer/deploy@e9acec8]: Release v0.2.3 on cumin2001 now on buster (take 3bis) (duration: 00m 12s)
• 08:09 volans@deploy1001: Started deploy [homer/deploy@e9acec8]: Release v0.2.3 on cumin2001 now on buster (take 3bis)
• 08:09 volans@deploy1001: Finished deploy [homer/deploy@e9acec8]: Release v0.2.3 on cumin2001 now on buster (take 3) (duration: 01m 37s)
• 08:08 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:08 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 08:08 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:08 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 08:08 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:08 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 08:07 volans@deploy1001: Started deploy [homer/deploy@e9acec8]: Release v0.2.3 on cumin2001 now on buster (take 3)
• 07:59 volans@deploy1001: Finished deploy [homer/deploy@85e92b8]: Release v0.2.3 on cumin2001 now on buster (take 2) (duration: 00m 57s)
• 07:58 volans@deploy1001: Started deploy [homer/deploy@85e92b8]: Release v0.2.3 on cumin2001 now on buster (take 2)
• 07:49 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 07:49 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 07:40 liw@deploy1001: Started scap: testwikis wikis to 1.35.0-wmf.37
• 07:37 liw@deploy1001: Pruned MediaWiki: 1.35.0-wmf.35 (duration: 01m 47s)
• 07:31 liw@deploy1001: Pruned MediaWiki: 1.35.0-wmf.34 (duration: 11m 52s)
• 07:23 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.prepare-upgrade (exit_code=0)
• 07:08 ayounsi@cumin1001: START - Cookbook sre.network.prepare-upgrade
• 07:07 liw: 1.35.0-wmf.37 was branched at f856960 for T254174
• 07:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1148', diff saved to https://phabricator.wikimedia.org/P11526 and previous config saved to /var/cache/conftool/dbconfig/20200616-070651-marostegui.json
• 07:04 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1148', diff saved to https://phabricator.wikimedia.org/P11525 and previous config saved to /var/cache/conftool/dbconfig/20200616-070450-marostegui.json
• 07:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1084', diff saved to https://phabricator.wikimedia.org/P11524 and previous config saved to /var/cache/conftool/dbconfig/20200616-070429-marostegui.json
• 07:02 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1084', diff saved to https://phabricator.wikimedia.org/P11523 and previous config saved to /var/cache/conftool/dbconfig/20200616-070209-marostegui.json
• 06:57 marostegui: Compress InnoDB on db1134 T254462
• 06:56 marostegui@cumin2001: dbctl commit (dc=all): 'Depool db1134 for InnoDB compression T254462', diff saved to https://phabricator.wikimedia.org/P11522 and previous config saved to /var/cache/conftool/dbconfig/20200616-065600-marostegui.json
• 06:55 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.prepare-upgrade (exit_code=0)
• 06:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1093', diff saved to https://phabricator.wikimedia.org/P11521 and previous config saved to /var/cache/conftool/dbconfig/20200616-065412-marostegui.json
• 06:40 ayounsi@cumin1001: START - Cookbook sre.network.prepare-upgrade
• 06:25 elukey: roll restart memcached on mc-gp* (gutter pools) to pick up new slab size distribution setting - T252391
• 06:04 hashar: Restarted Zuul scheduler and merger on contint2001 a couple hotfixes # T252310 T255424
• 05:54 volker-e@deploy1001: Finished deploy [design/style-guide@37c67dd]: Deploy design/style-guide: (duration: 00m 05s)
• 05:54 volker-e@deploy1001: Started deploy [design/style-guide@37c67dd]: Deploy design/style-guide:
• 05:04 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 05:01 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 04:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1146:3314', diff saved to https://phabricator.wikimedia.org/P11520 and previous config saved to /var/cache/conftool/dbconfig/20200616-045958-marostegui.json
• 04:57 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1146:3314', diff saved to https://phabricator.wikimedia.org/P11519 and previous config saved to /var/cache/conftool/dbconfig/20200616-045744-marostegui.json
• 04:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1147', diff saved to https://phabricator.wikimedia.org/P11518 and previous config saved to /var/cache/conftool/dbconfig/20200616-045636-marostegui.json
• 04:55 marostegui: Deploy schema change on db1147
• 04:54 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1147', diff saved to https://phabricator.wikimedia.org/P11517 and previous config saved to /var/cache/conftool/dbconfig/20200616-045451-marostegui.json
• 04:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1149', diff saved to https://phabricator.wikimedia.org/P11516 and previous config saved to /var/cache/conftool/dbconfig/20200616-044612-marostegui.json
• 04:44 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1149', diff saved to https://phabricator.wikimedia.org/P11515 and previous config saved to /var/cache/conftool/dbconfig/20200616-044409-marostegui.json
• 04:43 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1143', diff saved to https://phabricator.wikimedia.org/P11514 and previous config saved to /var/cache/conftool/dbconfig/20200616-044326-marostegui.json
• 04:41 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1143', diff saved to https://phabricator.wikimedia.org/P11513 and previous config saved to /var/cache/conftool/dbconfig/20200616-044126-marostegui.json
• 04:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1138', diff saved to https://phabricator.wikimedia.org/P11512 and previous config saved to /var/cache/conftool/dbconfig/20200616-044036-marostegui.json
• 04:37 marostegui: Deploy schema change on db1138
• 04:37 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1138', diff saved to https://phabricator.wikimedia.org/P11511 and previous config saved to /var/cache/conftool/dbconfig/20200616-043748-marostegui.json
• 00:28 tstarling@deploy1001: Synchronized wmf-config/CommonSettings.php: limit HTTP client timeout T245170 (duration: 00m 56s)
• 00:25 tstarling@deploy1001: Synchronized wmf-config/set-time-limit.php: expose excimer timeout as a global variable T245170 (duration: 00m 56s)
• 00:16 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@17212bb]: airflow: migrate leven-dist to edit-dist (duration: 00m 45s)
• 00:16 volker-e@deploy1001: Finished deploy [design/style-guide@37c67dd]: Deploy design/style-guide: (duration: 00m 04s)
• 00:16 volker-e@deploy1001: Started deploy [design/style-guide@37c67dd]: Deploy design/style-guide:
• 00:16 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@17212bb]: airflow: migrate leven-dist to edit-dist

2020-06-15

• 23:56 tstarling@deploy1001: Synchronized wmf-config/PoolCounterSettings.php: reducing connect timeout per T105378 (duration: 01m 00s)
• 23:31 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@eb0ac12]: Ship templatad table names in HivePartitionRangeSensor (duration: 00m 49s)
• 23:30 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@eb0ac12]: Ship templatad table names in HivePartitionRangeSensor
• 22:58 krinkle@deploy1001: Synchronized wmf-config/PhpAutoPrepend.php: If7e1613cbcf8 (duration: 00m 56s)
• 22:57 krinkle@deploy1001: Synchronized wmf-config/profiler.php: If7e1613cbcf8 (duration: 00m 59s)
• 22:02 bstorm_: downtimed puppet alerts for testing some changes on labstore1004/5
• 20:59 ebernhardson@deploy1001: Finished deploy [search/airflow@62a024b]: Add pydruid to airflow (duration: 00m 50s)
• 20:58 ebernhardson@deploy1001: Started deploy [search/airflow@62a024b]: Add pydruid to airflow
• 20:55 shdubsh: update mtail to 3.0.0~rc35 on the rest of the hosts - eqiad and esams
• 20:44 shdubsh: update mtail to 3.0.0~rc35 on cp nodes in eqiad and esams
• 20:30 shdubsh: update mtail to 3.0.0~rc35 on wtp in eqiad
• 19:35 shdubsh: update mtail to 3.0.0~rc35 on mw in eqiad
• 18:50 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@41186c8]: port glent from oozie to airflow (duration: 00m 39s)
• 18:50 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@41186c8]: port glent from oozie to airflow
• 18:28 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: gerrit:605584 T254315 test wikidata: Use the database name in the Wikibase entity source config (duration: 00m 58s)
• 17:56 krinkle@deploy1001: Synchronized wmf-config: I7721f4 (duration: 00m 58s)
• 17:55 krinkle@deploy1001: Synchronized wmf-config/ProductionServices.php: I7721f4 (duration: 00m 57s)
• 17:52 krinkle@deploy1001: Synchronized lib/: I7721f4 (duration: 00m 58s)
• 15:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1142', diff saved to https://phabricator.wikimedia.org/P11504 and previous config saved to /var/cache/conftool/dbconfig/20200615-153825-marostegui.json
• 15:37 marostegui: Deploy schema change on db1142
• 15:36 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1142', diff saved to https://phabricator.wikimedia.org/P11503 and previous config saved to /var/cache/conftool/dbconfig/20200615-153630-marostegui.json
• 15:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1141', diff saved to https://phabricator.wikimedia.org/P11502 and previous config saved to /var/cache/conftool/dbconfig/20200615-153546-marostegui.json
• 15:33 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1141', diff saved to https://phabricator.wikimedia.org/P11501 and previous config saved to /var/cache/conftool/dbconfig/20200615-153344-marostegui.json
• 15:16 moritzm: upgrading wtp1025-wtp1027 to PHP 7.2.31
• 15:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1121', diff saved to https://phabricator.wikimedia.org/P11499 and previous config saved to /var/cache/conftool/dbconfig/20200615-150908-marostegui.json
• 15:07 marostegui: Deploy schema change on db1121 (and labs)
• 15:06 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1121', diff saved to https://phabricator.wikimedia.org/P11498 and previous config saved to /var/cache/conftool/dbconfig/20200615-150639-marostegui.json
• 15:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1144:3314', diff saved to https://phabricator.wikimedia.org/P11497 and previous config saved to /var/cache/conftool/dbconfig/20200615-150148-marostegui.json
• 15:00 marostegui: Deploy schema change on db1144:3314
• 14:59 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1144:3314', diff saved to https://phabricator.wikimedia.org/P11496 and previous config saved to /var/cache/conftool/dbconfig/20200615-145914-marostegui.json
• 14:55 XioNoX: delete VCP from msw1-codfw
• 14:24 marostegui: Deploy schema change on db2107 (s2 codfw master) - T250066
• 14:16 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 14:11 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 14:09 elukey@cumin2001: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0)
• 13:54 marostegui: Deploy schema change on db1100 (s5 master) - T250066
• 13:54 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 13:49 marostegui: Upgrade db2133
• 13:49 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 13:44 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:44 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 13:41 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 13:40 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 13:38 elukey@cumin2001: START - Cookbook sre.hadoop.roll-restart-workers
• 13:31 volans@deploy1001: Finished deploy [homer/deploy@ac7a4c6]: Release v0.2.3 on cumin2001 now on buster (duration: 01m 15s)
• 13:30 moritzm: rolling reboot on the ganeti cluster in esams (for kernel security updates and to pick up the network changes to provides instances with a public IP)
• 13:30 volans@deploy1001: Started deploy [homer/deploy@ac7a4c6]: Release v0.2.3 on cumin2001 now on buster
• 13:26 hashar: Started zuul-merger on contint1001 with newer virtualenv # T255424
• 13:26 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 13:21 filippo@cumin1001: conftool action : set/pooled=true; selector: dnsdisc=thanos-query,name=eqiad
• 13:20 hashar: Stopping zuul-merger on contint1001 to rebuild the virtualenv # T255424
• 13:19 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 13:04 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 12:59 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2091:3312, db2091:3314 - T253217', diff saved to https://phabricator.wikimedia.org/P11495 and previous config saved to /var/cache/conftool/dbconfig/20200615-125856-marostegui.json
• 12:58 vgutierrez: upgrade acme-chief to version 0.26
• 12:57 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 12:46 vgutierrez: upload acme-chief 0.26 to apt.wm.o (buster) - T255249
• 12:43 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 12:38 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 12:34 moritzm: rolling reboot on the ganeti cluster in eqsin (for security updates and to pick up the network changes to provides instances with a public IP)
• 12:12 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 12:11 marostegui: Upgrade db2134
• 12:09 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 12:01 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 11:59 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 11:57 moritzm: reimaging sretest1002 to validate the reimage script on Buster
• 11:43 marostegui: Reimage dbproxy2003 which points to m3-master.codfw.wmnet (not in use) - T255408
• 11:40 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Config: GrowthExperiments: Switch on guidance feature (T239181) (duration: 00m 57s)
• 11:10 sukhe@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 11:10 sukhe@cumin1001: START - Cookbook sre.hosts.downtime
• 11:07 hnowlan: regenerated certificates for restbase2009, restbase101[678], restbase201[012]. Did not roll-restart yet
• 11:07 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 11:04 jmm@cumin2001: START - Cookbook sre.hosts.reboot-single
• 11:03 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 11:03 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 10:54 moritzm: imported python-phabricator 0.7.0-2~wmf2 to apt.wikimedia.org/buster-wikimedia T245114
• 10:39 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (605553) (duration: 00m 58s)
• 10:38 hnowlan: regenerated restbase2009's cassandra certificates
• 10:38 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (605553) (duration: 00m 58s)
• 10:16 jmm@cumin1001: END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97)
• 10:16 jmm@cumin1001: START - Cookbook sre.hosts.reboot-single
• 10:12 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T254820 [enwikivoyage] Undeploy the Listings extension (duration: 01m 00s)
• 10:08 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 10:05 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 09:53 filippo@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:50 filippo@cumin1001: START - Cookbook sre.hosts.downtime
• 09:46 godog: run logstash benchmark on logstash1023
• 09:42 volans: deploying esams mgmt DNS records automatically generated by Netbox ( operations/dns/+/604136/ ) - T233183
• 09:41 volans@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 09:35 volans@cumin1001: START - Cookbook sre.dns.netbox
• 09:29 elukey: update analytics-in4/6 filters on cr1-cr2 eqiad to update the Druid term (new nodes added)
• 09:21 jbond42: offlining puppetmaster1003 and 2003 for reboot
• 09:17 XioNoX: reduce ae device-count from 10 to 3 on asw2-a/b/c-eqiad
• 09:14 jmm@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:11 jmm@cumin1001: START - Cookbook sre.hosts.downtime
• 08:55 marostegui: Deploy schema change on db2123 (s5 codfw master) - T250066
• 08:50 kart_: Updated cxserver to 2020-06-10-044445-production (T246319, T254959)
• 08:46 kartik@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'cxserver' for release 'production' .
• 08:42 kartik@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'cxserver' for release 'production' .
• 08:39 kartik@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'cxserver' for release 'staging' .
• 08:34 moritzm: reimaging cumin2001 T245114
• 08:22 marostegui: Switchover m3-master from dbproxy1008 to dbproxy1016 - T202367
• 08:17 marostegui: Deploy schema change on db1131 (s6 master) - T250066
• 08:09 moritzm: installing libexif security updates
• 07:51 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 07:49 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 07:46 XioNoX: standardize ae device-count on all routers
• 07:36 XioNoX: push new pfw firewall policies - T255185
• 07:28 marostegui: Deploy schema change on db1093
• 07:28 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1093 for schema change', diff saved to https://phabricator.wikimedia.org/P11492 and previous config saved to /var/cache/conftool/dbconfig/20200615-072835-marostegui.json
• 07:27 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2092', diff saved to https://phabricator.wikimedia.org/P11491 and previous config saved to /var/cache/conftool/dbconfig/20200615-072742-marostegui.json
• 06:21 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 06:19 marostegui@cumin1001: START - Cookbook sre.hosts.downtime

2020-06-14

• 13:51 qchris: Disabling puppet on gerrit1002 (test instance) to do some more upgrade testing

2020-06-13

• 21:12 qchris: Enabling puppet on gerrit1002 (test instance). Done with testing for today.
• 12:51 herron: restarted logstash service on logstash1007, logstash1009
• 12:34 qchris: Disabling puppet on gerrit1002 (test instance) to do some more upgrade testing
• 12:33 godog: bounce logstash on logstash1008, GC death

2020-06-12

• 17:44 herron: restarting logstash1011 elasticsearch instance
• 16:49 elukey: restart php-fpm and pool mw1384 - T255282
• 16:33 elukey: (correct) depool again mw1384 - investigation will follow up in a task
• 16:32 elukey: depool again mw1348 - investigation will follow up in a task
• 15:49 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:44 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 15:40 hnowlan@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'changeprop' for release 'production' .
• 15:40 pt1979@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:37 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
• 15:36 hnowlan@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop' for release 'production' .
• 15:27 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:25 hnowlan@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop' for release 'production' .
• 15:24 akosiaris@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 15:24 akosiaris@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 15:24 akosiaris@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 15:24 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:22 akosiaris@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 15:22 akosiaris@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 15:22 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 15:22 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 15:22 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 15:22 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 15:22 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 15:22 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 15:22 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 14:51 elukey: repool mw1384 as test
• 14:31 akosiaris@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'changeprop' for release 'production' .
• 14:30 akosiaris: bump cpu limits for changeprop another 50%
• 14:30 akosiaris@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop' for release 'production' .
• 13:36 akosiaris@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'changeprop' for release 'production' .
• 13:34 akosiaris: update changeprop in eqiad+codfw for higher CPU limits
• 13:34 akosiaris@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop' for release 'production' .
• 13:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1088 after schema change', diff saved to https://phabricator.wikimedia.org/P11483 and previous config saved to /var/cache/conftool/dbconfig/20200612-131205-marostegui.json
• 12:40 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1088 for schema change', diff saved to https://phabricator.wikimedia.org/P11482 and previous config saved to /var/cache/conftool/dbconfig/20200612-124015-marostegui.json
• 12:18 filippo@cumin1001: END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0)
• 11:52 filippo@cumin1001: START - Cookbook sre.hosts.upgrade-and-reboot
• 11:23 jmm@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 11:19 jmm@cumin1001: START - Cookbook sre.hosts.reboot-single
• 11:15 moritzm: failover ganeti master in ulsfo to ganeti4003
• 11:14 marostegui@cumin1001: dbctl commit (dc=all): 'Pool db2080 and db2084 into s8 T253217', diff saved to https://phabricator.wikimedia.org/P11481 and previous config saved to /var/cache/conftool/dbconfig/20200612-111422-marostegui.json
• 11:11 jmm@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 11:07 jmm@cumin1001: START - Cookbook sre.hosts.reboot-single
• 11:02 jmm@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 10:58 jmm@cumin1001: START - Cookbook sre.hosts.reboot-single
• 10:39 jmm@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 10:36 jmm@cumin1001: START - Cookbook sre.hosts.reboot-single
• 10:33 moritzm: rolling restart of the ulsfo ganeti cluster
• 10:21 filippo@cumin1001: END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0)
• 10:02 filippo@cumin1001: START - Cookbook sre.hosts.upgrade-and-reboot
• 10:01 jmm@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 10:01 filippo@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
• 10:01 filippo@cumin1001: START - Cookbook sre.hosts.reboot-single
• 10:01 jmm@cumin1001: START - Cookbook sre.hosts.downtime
• 09:58 marostegui@cumin1001: dbctl commit (dc=all): 'Include db2084 in dbctl, depooled', diff saved to https://phabricator.wikimedia.org/P11480 and previous config saved to /var/cache/conftool/dbconfig/20200612-095855-marostegui.json
• 09:58 godog: roll-restart thanos-fe / thanos-be for microcode updates
• 08:51 elukey: restart gerrit on gerrit1001
• 08:48 elukey: update cr1/cr2 analyitics filters for T252767 and T252675
• 08:44 marostegui: Compress InnoDB on db2092 - T254462
• 08:36 marostegui: Clone db2084 from db2080
• 08:32 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2080 to clone db2084', diff saved to https://phabricator.wikimedia.org/P11478 and previous config saved to /var/cache/conftool/dbconfig/20200612-083231-marostegui.json
• 08:24 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:22 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 08:14 marostegui@cumin1001: dbctl commit (dc=all): 'Remove db2084 from s4 and s5', diff saved to https://phabricator.wikimedia.org/P11477 and previous config saved to /var/cache/conftool/dbconfig/20200612-081455-marostegui.json
• 07:56 elukey: depool mw1384
• 07:52 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2084 from s4 and s5', diff saved to https://phabricator.wikimedia.org/P11476 and previous config saved to /var/cache/conftool/dbconfig/20200612-075202-marostegui.json
• 07:26 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 07:24 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 07:08 marostegui: Reimage db2086
• 07:07 elukey: depool/scap pull/pool mw1384
• 07:05 moritzm: installing intel-microcode security updates (regressions have been sorted out)
• 05:42 moritzm: installing stretch kernel security updates (no reboots yet)
• 05:40 moritzm: installing buster kernel security updates (no reboots yet)
• 04:54 marostegui: Deploy schema change on s6 codfw - T250066
• 01:02 ejegg: updated payments-wiki from aceddff8b5 to 5fd4eb1519
• 00:10 Amir1: BACON is done

2020-06-11

• 23:54 ladsgroup@deploy1001: Synchronized php-1.35.0-wmf.36/extensions/Wikibase: Fix entity id lookup for interwiki special page links (T255078) (duration: 00m 38s)
• 23:51 ladsgroup@deploy1001: scap failed: average error rate on 3/9 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/e474f13ffac6b8c3bf919c4aeafc8c9b for details)
• 23:43 ladsgroup@deploy1001: Synchronized wmf-config/extension-list: Remove ContributionTracking extension (T255216), Part III (duration: 00m 57s)
• 23:42 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Remove ContributionTracking extension (T255216), Part II (duration: 00m 58s)
• 23:38 ladsgroup@deploy1001: Synchronized wmf-config/CommonSettings.php: Remove ContributionTracking extension (T255216), Part I (duration: 00m 59s)
• 23:37 Reedy: create cn_notice_regions on metawiki and testwiki T252596
• 20:34 pt1979@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 20:31 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
• 20:15 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 20:13 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
• 20:00 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 19:59 jhuneidi@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.36
• 19:58 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
• 19:33 akosiaris: apply emergency sessionstore fixes in codfw as well
• 19:32 akosiaris@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'sessionstore' for release 'production' .
• 19:25 gilles@deploy1001: Finished deploy [performance/asoranking@0a096c4]: T252424 (duration: 00m 47s)
• 19:19 gilles@deploy1001: Started deploy [performance/asoranking@0a096c4]: T252424
• 19:12 akosiaris: repool eqiad for sessionstore
• 19:12 akosiaris@cumin1001: conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=sessionstore
• 19:10 akosiaris: remove the podaffinity restrictions for sessionstore in eqiad
• 19:10 akosiaris@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'sessionstore' for release 'production' .
• 19:07 akosiaris@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'sessionstore' for release 'production' .
• 18:08 ppchelko@deploy1001: Synchronized wmf-config/reverse-proxy-staging.php: Beta: Switch from HTCP purging to kafka purging gerrit:603530, reverse-proxy-staging.php (duration: 01m 06s)
• 18:06 ppchelko@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: Beta: Switch from HTCP purging to kafka purging gerrit:603530, IS-labs.php (duration: 01m 06s)
• 17:29 mbsantos@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'proton' for release 'production' .
• 17:26 mbsantos@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'production' .
• 17:22 mbsantos@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'proton' for release 'production' .
• 17:19 mbsantos@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'mobileapps' for release 'production' .
• 17:12 bstorm_: reboot for stretch upgrade on labstore1004 T224582
• 16:49 bstorm_: doing stretch upgrade for labstore1004 T224582
• 16:36 bstorm_: rebooting labstore1004 for upgrades T224582
• 16:12 bstorm_: downtimed labstore1005 for upgrades on 1004 since that will alert as well T224582
• 16:10 bstorm_: downtimed labstore1004 for upgrades T224582
• 15:50 cstone: SmashPig revision changed from b9de3c7aac to 2246685626
• 15:34 jmm@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
• 15:31 jmm@cumin1001: START - Cookbook sre.hosts.reboot-single
• 15:25 moritzm: installing buster kernel security updates (no reboots yet)
• 15:04 jmm@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99)
• 15:04 mforns@deploy1001: Finished deploy [analytics/refinery@c969b56]: Regular analytics weekly train [analytics/refinery@c969b56afae1b2532e07f0ff699c2ce161360966] (duration: 01m 39s)
• 15:04 root@cumin1001: END (FAIL) - Cookbook sre.network.prepare-upgrade (exit_code=99)
• 15:04 root@cumin1001: START - Cookbook sre.network.prepare-upgrade
• 15:02 mforns@deploy1001: Started deploy [analytics/refinery@c969b56]: Regular analytics weekly train [analytics/refinery@c969b56afae1b2532e07f0ff699c2ce161360966]
• 15:02 jmm@cumin1001: START - Cookbook sre.hosts.reboot-single
• 14:56 herron: bounced elasticsearch on logstash1012
• 14:41 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
• 14:40 akosiaris@cumin1001: START - Cookbook sre.hosts.decommission
• 14:37 herron: enabled VO incident resolution notification in global settings
• 14:34 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
• 14:31 akosiaris@cumin1001: START - Cookbook sre.hosts.decommission
• 14:30 godog: bounce logstash on logstash1009, apparent GC death spiral
• 14:03 jmm@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99)
• 14:03 jmm@cumin1001: START - Cookbook sre.hosts.reboot-single
• 14:03 jmm@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
• 14:03 jmm@cumin1001: START - Cookbook sre.hosts.reboot-single
• 13:35 filippo@cumin1001: conftool action : set/pooled=false; selector: dnsdisc=thanos-query,name=eqiad
• 13:35 filippo@cumin1001: conftool action : set/pooled=false; selector: dnsdisc=thanos-swift,name=eqiad
• 12:39 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.prepare-upgrade (exit_code=0)
• 12:36 elukey: updated pcc facts
• 12:28 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' .
• 12:28 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-main' for release 'production' .
• 12:28 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 12:25 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 12:15 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' .
• 12:15 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' .
• 12:04 jforrester@deploy1001: Synchronized php-1.35.0-wmf.36/includes/title/NamespaceInfo.php: T253098 NamespaceInfo::makeValidNamespace: Don't throw for -1 or -2 (duration: 01m 06s)
• 12:03 marostegui: Reimage es2023 (es5 codfw master)
• 11:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db2075 T254139', diff saved to https://phabricator.wikimedia.org/P11469 and previous config saved to /var/cache/conftool/dbconfig/20200611-115430-marostegui.json
• 11:46 marostegui: Deploy schema change on s6 codfw - T250066
• 11:44 volans@deploy1001: Finished deploy [homer/deploy@df83901]: Release v0.2.3 (duration: 00m 25s)
• 11:44 volans@deploy1001: Started deploy [homer/deploy@df83901]: Release v0.2.3
• 11:36 ayounsi@cumin1001: START - Cookbook sre.network.prepare-upgrade
• 11:36 matthiasmullie: EU BACON done
• 11:35 mlitn@deploy1001: Synchronized php-1.35.0-wmf.36/extensions/GrowthExperiments: Help panel: Update guidance behavior rules (duration: 01m 06s)
• 11:34 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' .
• 11:34 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' .
• 11:28 kartik@deploy1001: Synchronized php-1.35.0-wmf.36/extensions/ContentTranslation/modules/tools/mw.cx.tools.IssueTrackingTool.js: Backport: 604587|IssueTrackingTool: Fix js error in getCurrentNodeId method (T254965) (duration: 01m 07s)
• 11:08 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' .
• 11:04 mlitn@deploy1001: Synchronized php-1.35.0-wmf.36/extensions/MachineVision: $aliases should be an array of strings, not AliasGroup objects (duration: 01m 07s)
• 10:47 moritzm: repooling mw1318,mw2139,mw2145,mw2147,mw2221,mw2219,mw2250,mw2350 (these were depooled, but seem all fine in Icinga and were probably just forgotten)
• 10:41 filippo@cumin1001: conftool action : set/pooled=yes; selector: cluster=thanos,service=thanos-swift
• 10:40 filippo@cumin1001: conftool action : set/pooled=yes; selector: cluster=thanos,service=thanos-query
• 10:37 moritzm: installing buster kernel security updates (no reboots yet, on hold for regression-free microcode update)
• 10:32 godog: roll-restart pybal in eqiad lvs low-traffic
• 10:21 mutante: restarting gerrit on gerrit-replica (gerrit2001) - java.lang.OutOfMemoryError: Java heap space
• 10:21 Urbanecm: Run scap pull at mwdebug1001 to revert temporary changes
• 10:14 Urbanecm: Applying temporary changes on mwdebug1001
• 09:58 moritzm: upgrading netmon* to PHP 7.2.31
• 09:55 marostegui: Upgrade es2025
• 09:54 moritzm: upgrading mwmaint* to PHP 7.2.31
• 09:46 moritzm: upgrading labweb* PHP 7.2.31
• 09:36 elukey: switch piwik.wikimedia.org from matomo1001 to matomo1002 (new buster node)
• 09:02 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:00 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 08:48 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-main' for release 'production' .
• 08:48 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' .
• 08:42 moritzm: imported memcached 1.6.6-1~wmf10u1
• 08:39 marostegui: Reimage es2024 to buster
• 08:30 filippo@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:30 filippo@cumin1001: START - Cookbook sre.hosts.downtime
• 08:25 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:25 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 08:25 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:25 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 08:24 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:24 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 08:24 akosiaris@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 08:24 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 08:23 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' .
• 08:23 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' .
• 08:22 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:20 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 08:18 filippo@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:18 filippo@cumin1001: START - Cookbook sre.hosts.downtime
• 08:01 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' .
• 08:01 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' .
• 07:59 moritzm: upgrading remaining job runners in eqiad to PHP 7.2.31
• 07:59 hashar: Restarted Zuul on contint2001 for config change # T253263
• 07:43 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' .
• 07:34 moritzm: upgrading remaining app servers in eqiad to PHP 7.2.31
• 07:26 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 07:24 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 07:07 marostegui: Stop MySQL on dbstore1003 for reimage - T254870
• 06:38 XioNoX: make asw2-esams interfaces Homer like - T250429
• 05:55 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1127 T253217', diff saved to https://phabricator.wikimedia.org/P11467 and previous config saved to /var/cache/conftool/dbconfig/20200611-055536-marostegui.json
• 05:25 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1127 T253217', diff saved to https://phabricator.wikimedia.org/P11466 and previous config saved to /var/cache/conftool/dbconfig/20200611-052535-marostegui.json
• 05:04 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1127 T253217', diff saved to https://phabricator.wikimedia.org/P11465 and previous config saved to /var/cache/conftool/dbconfig/20200611-050446-marostegui.json
• 05:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1078', diff saved to https://phabricator.wikimedia.org/P11464 and previous config saved to /var/cache/conftool/dbconfig/20200611-050200-marostegui.json
• 04:54 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1078', diff saved to https://phabricator.wikimedia.org/P11463 and previous config saved to /var/cache/conftool/dbconfig/20200611-045426-marostegui.json
• 04:50 marostegui: Deploy schema change on testwiki - T254371
• 04:47 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1084 and slowly repool db1127 T253217', diff saved to https://phabricator.wikimedia.org/P11462 and previous config saved to /var/cache/conftool/dbconfig/20200611-044725-marostegui.json
• 03:13 shdubsh: removing WDQS-Streaming-Updater-POC metrics on graphite1004 - T255044
• 02:43 tstarling@deploy1001: Synchronized php-1.35.0-wmf.36/extensions/Wikibase/lib/includes/Store/EntityLinkTargetEntityIdLookup.php: investigate UBN T255078 (duration: 01m 07s)

2020-06-10

• 23:55 catrope@deploy1001: Synchronized php-1.35.0-wmf.36/includes/skins/SkinTemplate.php: T255073 (duration: 01m 07s)
• 22:14 eileen: civicrm revision changed from 80a0d22350 to f01b036128, config revision is a26d023633
• 21:23 akosiaris: increase memory/cpu limits for proton
• 21:23 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'proton' for release 'production' .
• 21:11 mbsantos@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'proton' for release 'production' .
• 21:08 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'mobileapps' for release 'staging' .
• 21:06 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'mobileapps' for release 'staging' .
• 20:45 mbsantos@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'proton' for release 'production' .
• 20:33 jhuneidi@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'mobileapps' for release 'staging' .
• 20:15 mbsantos@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'mobileapps' for release 'staging' .
• 20:04 mbsantos@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'mobileapps' for release 'staging' .
• 19:46 herron: bouncing elasticsearch on logstash1011
• 19:01 ppchelko@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Use EventRelayerNull for wikitech, gerrit:604469 (duration: 01m 05s)
• 18:54 urbanecm@deploy1001: Synchronized php-1.35.0-wmf.36/extensions/VisualEditor/: 8958860: Make VisualEditorDisableForAnons only hide the tabs, not disable the editor (T253941) (duration: 01m 07s)
• 18:32 urbanecm@deploy1001: Synchronized php-1.35.0-wmf.35/extensions/VisualEditor/: 5f4c609: Make VisualEditorDisableForAnons only hide the tabs, not disable the editor (T253941) (duration: 01m 14s)
• 16:40 godog: EDIT: in esams
• 16:39 godog: restart prometheus@ops in eqiad
• 16:31 ppchelko@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Disable HTCP purges everywhere, gerrit:603655 (duration: 01m 05s)
• 16:27 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' .
• 16:27 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-main' for release 'production' .
• 16:18 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' .
• 16:18 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' .
• 16:13 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 16:13 ema: correction: restart purged on all *cache_upload* hosts to apply https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/604430/ T250781 T133821
• 16:12 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' .
• 16:12 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' .
• 16:12 ema: restart purged on all cache hosts to apply https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/604430/ T250781 T133821
• 16:11 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
• 16:06 ema: cp3051: restart purged to apply https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/604430/ T250781 T133821
• 16:02 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 16:00 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
• 15:49 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 15:45 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
• 15:38 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' .
• 15:37 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 15:36 ppchelko@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Send kafka purges everywhere, gerrit:603654 (duration: 01m 05s)
• 15:35 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
• 15:32 ema: remaining-cp (non-ulsfo): rolling ats-tls-restart to apply https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/604305/ T255015
• 15:29 ppchelko@deploy1001: Synchronized wmf-config/CommonSettings.php: Make kafka purges config more robust, gerrit:603649, CS.php (duration: 01m 05s)
• 15:27 ppchelko@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Make kafka purges config more robust, gerrit:603649, IS.php (duration: 01m 08s)
• 15:21 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 15:19 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
• 15:08 godog: roll-restart prometheus k8s to enable thanos upload
• 15:02 ema: A:cp-ulsfo: rolling ats-tls-restart to apply https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/604305/ T255015
• 14:43 ema: A:cp rolling systemctl restart trafficserver
• 14:28 ema: systemctl restart trafficserver for instances critical in icinga
• 14:21 ema: cp3056: ats-backend-restart
• 14:09 ema: A:cp rolling ats-be/ats-tls restarts to apply https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/604305/ T255015
• 14:08 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:06 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 14:02 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:59 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 13:57 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1094 into s7', diff saved to https://phabricator.wikimedia.org/P11458 and previous config saved to /var/cache/conftool/dbconfig/20200610-135753-marostegui.json
• 13:50 ema: cp3050: ats-tls-restart to apply https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/604305/ T255015
• 13:50 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1094 into s7', diff saved to https://phabricator.wikimedia.org/P11457 and previous config saved to /var/cache/conftool/dbconfig/20200610-135039-marostegui.json
• 13:40 ema: cp3050: ats-backend-restart to apply https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/604305/ T255015
• 13:36 ayounsi@cumin1001: END (FAIL) - Cookbook sre.network.prepare-upgrade (exit_code=99)
• 13:06 liw@deploy1001: Synchronized php: group1 wikis to 1.35.0-wmf.36 (duration: 01m 04s)
• 13:05 liw@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.36
• 12:33 ayounsi@cumin1001: START - Cookbook sre.network.prepare-upgrade
• 12:32 ayounsi@cumin1001: END (FAIL) - Cookbook sre.network.prepare-upgrade (exit_code=99)
• 12:32 ayounsi@cumin1001: START - Cookbook sre.network.prepare-upgrade
• 12:13 akosiaris: pool thumbor2002, thumbor2001. T251570
• 12:12 akosiaris@cumin1001: conftool action : set/pooled=yes; selector: name=thumbor2002.codfw.wmnet
• 12:12 akosiaris@cumin1001: conftool action : set/pooled=yes; selector: name=thumbor2001.codfw.wmnet
• 11:50 marostegui: Deploy schema change on commonswiki codfw T255003
• 11:41 moritzm: upgrading remaining app servers in codfw to PHP 7.2.31
• 11:38 marostegui: Deploy schema change on testcommonswiki T255003
• 11:37 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 52091b8: Grant cswiki accountcreators tboverride-account and override-antispoof (T254927) (duration: 01m 06s)
• 11:13 moritzm: upgrading remaining job runners in codfw to PHP 7.2.31
• 11:02 marostegui: Stop MySQL on db1094 to clone db1127
• 11:02 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1094 moving to clone db1127 T253217', diff saved to https://phabricator.wikimedia.org/P11453 and previous config saved to /var/cache/conftool/dbconfig/20200610-110204-marostegui.json
• 10:57 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 10:54 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 10:37 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1127 moving it to s7 T253217', diff saved to https://phabricator.wikimedia.org/P11452 and previous config saved to /var/cache/conftool/dbconfig/20200610-103742-marostegui.json
• 10:28 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1103,db1137 into x1', diff saved to https://phabricator.wikimedia.org/P11451 and previous config saved to /var/cache/conftool/dbconfig/20200610-102805-marostegui.json
• 10:24 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T254036 Undeploy CollaborationKit: IV – Drop flag to load (duration: 01m 05s)
• 10:23 jayme: T254581 re-enabled puppet on all mw, api and jobrunner servers
• 10:20 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: T254036 Undeploy CollaborationKit: III – Drop ability to load (duration: 01m 05s)
• 10:16 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T254036 Undeploy CollaborationKit: II – Disable on Test Wikipedia (duration: 01m 37s)
• 10:14 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1103,db1137 into x1', diff saved to https://phabricator.wikimedia.org/P11450 and previous config saved to /var/cache/conftool/dbconfig/20200610-101407-marostegui.json
• 10:12 moritzm: upgrading remaining API servers in codfw to PHP 7.2.31
• 10:08 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1103,db1137 into x1', diff saved to https://phabricator.wikimedia.org/P11449 and previous config saved to /var/cache/conftool/dbconfig/20200610-100834-marostegui.json
• 10:03 jynus: cloning reviewdb into reviewdb-test at db1132 with replication enabled T254516
• 10:03 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1103 into x1', diff saved to https://phabricator.wikimedia.org/P11448 and previous config saved to /var/cache/conftool/dbconfig/20200610-100306-marostegui.json
• 10:00 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1137 into x1', diff saved to https://phabricator.wikimedia.org/P11447 and previous config saved to /var/cache/conftool/dbconfig/20200610-100037-marostegui.json
• 09:35 volans: imported 0.0.38-1+deb10u1 into buster-wikimedia APT - T245114
• 09:35 marostegui: Stop mysql on db1127 to clone db1103
• 09:34 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1137 for cloning db1103 - T253217', diff saved to https://phabricator.wikimedia.org/P11443 and previous config saved to /var/cache/conftool/dbconfig/20200610-093440-marostegui.json
• 09:31 elukey@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 09:31 godog: configure thanos-be1* HDDs as raid0 - T252186
• 09:26 marostegui@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 09:26 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 09:26 marostegui@cumin1001: dbctl commit (dc=all): 'Add db1103 to dbctl, depooled T253217', diff saved to https://phabricator.wikimedia.org/P11442 and previous config saved to /var/cache/conftool/dbconfig/20200610-092603-marostegui.json
• 09:24 marostegui@cumin1001: dbctl commit (dc=all): 'Remove db1103:3312 and db1103:3314', diff saved to https://phabricator.wikimedia.org/P11441 and previous config saved to /var/cache/conftool/dbconfig/20200610-092406-marostegui.json
• 09:14 jayme: T254581 disabling puppet on all mw, api and jobrunner servers to move termbox envoy config to TLS
• 09:08 kormat@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 09:08 kormat@cumin1001: START - Cookbook sre.hosts.downtime
• 08:50 XioNoX: make asw1-eqsin interfaces Homer like - T250429
• 08:45 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventstreams' for release 'production' .
• 08:45 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventstreams' for release 'canary' .
• 08:45 elukey@cumin1001: START - Cookbook sre.ganeti.makevm
• 08:17 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventstreams' for release 'production' .
• 08:15 kormat@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 08:14 kormat@cumin1001: START - Cookbook sre.hosts.downtime
• 08:13 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventstreams' for release 'canary' .
• 07:53 kormat: reimaging db1077 T252027
• 07:36 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventstreams' for release 'production' .
• 07:36 XioNoX: make asw2-ulsfo interfaces Homer like - T250429
• 07:33 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventstreams' for release 'canary' .
• 07:31 moritzm: upgrade mw1298-mw1309 (job runners) to PHP 7.2.31
• 07:26 XioNoX: trunk public vlan to esams ganeti hosts - T254157
• 07:16 XioNoX: trunk public vlan to eqsin ganeti hosts - T254157
• 07:15 moritzm: upgrade remaining API servers in eqiad to PHP 7.2.31
• 07:08 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1103 for reimage - T253217', diff saved to https://phabricator.wikimedia.org/P11439 and previous config saved to /var/cache/conftool/dbconfig/20200610-070822-marostegui.json
• 07:05 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db2113 after on-site maintenance T251570', diff saved to https://phabricator.wikimedia.org/P11438 and previous config saved to /var/cache/conftool/dbconfig/20200610-070508-marostegui.json
• 06:53 XioNoX: trunk public vlan to ulsfo ganeti hosts - T254157
• 05:10 marostegui: Deploy schema change on s3 master with 2 minutes sleep between wikis - T206103

2020-06-09

• 23:18 Reedy: run namespaceDupes.php --fix for hiwikibooks T254012
• 23:10 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T254706 T254012 T241893 (duration: 01m 06s)
• 23:03 Reedy: created wikilove_log on slwiki T254706
• 20:00 jhuneidi@deploy1001: Pruned MediaWiki: 1.35.0-wmf.32 (duration: 05m 11s)
• 19:51 jhuneidi@deploy1001: rebuilt and synchronized wikiversions files: group0 wikis to 1.35.0-wmf.36
• 19:42 jhuneidi@deploy1001: Finished scap: testwikis wikis to 1.35.0-wmf.36 (duration: 57m 47s)
• 19:29 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 19:26 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 19:26 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 19:23 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 19:10 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 19:07 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 19:07 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 19:05 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 18:45 jhuneidi@deploy1001: Started scap: testwikis wikis to 1.35.0-wmf.36
• 18:41 jforrester@deploy1001: Synchronized php-1.35.0-wmf.36/extensions/TimedMediaHandler/includes/TimedMediaHandler.php: T254824 Avoid undefined index error (duration: 00m 57s)
• 18:36 volans: migrated mgmt DNS records in eqsin to the Netbox-generated records - T233183
• 18:13 jforrester@deploy1001: Synchronized php-1.35.0-wmf.36/extensions/CheckUser/: T234921 T254912 Use UserGroupManagerFactory with correct domain to fetch groups (duration: 02m 26s)
• 18:12 volans: uploaded cumin_4.0.0rc1-1_amd64.deb to apt.wikimedia.org buster-wikimedia
• 16:43 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 16:40 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 16:06 longma: cutting the branch for 1.35.0-wmf.36 T254173
• 15:26 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:26 aborrero@cumin1001: START - Cookbook sre.hosts.downtime
• 15:25 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:25 aborrero@cumin1001: START - Cookbook sre.hosts.downtime
• 15:06 volans: forcing a debmonitor GC to verify the fix of T254865
• 14:59 mutante: gerrit2001 - delete gerrit logfiles older than 30 days, crons are now enabled to keep doing it in the future
• 14:55 volans@deploy1001: Finished deploy [debmonitor/deploy@44aa1ee]: Release v0.2.5 (duration: 00m 43s)
• 14:54 volans@deploy1001: Started deploy [debmonitor/deploy@44aa1ee]: Release v0.2.5
• 14:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db2131 after reimage', diff saved to https://phabricator.wikimedia.org/P11436 and previous config saved to /var/cache/conftool/dbconfig/20200609-144929-marostegui.json
• 14:45 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:42 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:40 kormat@cumin1001: START - Cookbook sre.hosts.downtime
• 14:40 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 14:34 moritzm: rebooting auth1002
• 14:33 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:32 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:32 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 14:30 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 14:12 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:09 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 14:00 elukey: update release repository's settings on Archiva - T254849
• 14:00 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:00 aborrero@cumin1001: START - Cookbook sre.hosts.downtime
• 14:00 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:00 aborrero@cumin1001: START - Cookbook sre.hosts.downtime
• 13:56 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:54 kormat@cumin1001: START - Cookbook sre.hosts.downtime
• 12:38 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2131 for reimage', diff saved to https://phabricator.wikimedia.org/P11434 and previous config saved to /var/cache/conftool/dbconfig/20200609-123817-marostegui.json
• 12:22 kormat: reimaging sretest1002 T252027
• 12:18 kartik@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'cxserver' for release 'production' .
• 12:16 kartik@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'cxserver' for release 'production' .
• 12:14 kartik@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'cxserver' for release 'staging' .
• 12:00 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1141 into s4 T252512', diff saved to https://phabricator.wikimedia.org/P11433 and previous config saved to /var/cache/conftool/dbconfig/20200609-120009-marostegui.json
• 11:50 kartik@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'cxserver' for release 'production' .
• 11:50 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1141 into s4 T252512', diff saved to https://phabricator.wikimedia.org/P11432 and previous config saved to /var/cache/conftool/dbconfig/20200609-115016-marostegui.json
• 11:46 kartik@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'cxserver' for release 'production' .
• 11:46 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1148 into s4 T252512', diff saved to https://phabricator.wikimedia.org/P11431 and previous config saved to /var/cache/conftool/dbconfig/20200609-114615-marostegui.json
• 11:44 kartik@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'cxserver' for release 'staging' .
• 11:38 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1141 into s4 T252512', diff saved to https://phabricator.wikimedia.org/P11430 and previous config saved to /var/cache/conftool/dbconfig/20200609-113818-marostegui.json
• 11:37 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1148 into s4 T252512', diff saved to https://phabricator.wikimedia.org/P11429 and previous config saved to /var/cache/conftool/dbconfig/20200609-113702-marostegui.json
• 11:30 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1141 into s4 T252512', diff saved to https://phabricator.wikimedia.org/P11428 and previous config saved to /var/cache/conftool/dbconfig/20200609-113056-marostegui.json
• 11:27 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1148 into s4 T252512', diff saved to https://phabricator.wikimedia.org/P11427 and previous config saved to /var/cache/conftool/dbconfig/20200609-112701-marostegui.json
• 11:15 ladsgroup@deploy1001: Synchronized langlist: Add be-tarask to langlist (T111853) (duration: 00m 57s)
• 11:14 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1148 into s4 T252512', diff saved to https://phabricator.wikimedia.org/P11426 and previous config saved to /var/cache/conftool/dbconfig/20200609-111443-marostegui.json
• 10:49 elukey: update pcc facts
• 10:48 moritzm: imported tqdm 4.23.4-1+wmf1 to buster-wikimedia/component/spicerack
• 10:35 volans: installed spicerack 0.0.38 on cumin[12]001
• 10:32 marostegui@cumin1001: dbctl commit (dc=all): 'Add db1141 depooled to s4 T252512', diff saved to https://phabricator.wikimedia.org/P11425 and previous config saved to /var/cache/conftool/dbconfig/20200609-103252-marostegui.json
• 10:27 volans: uploaded spicerack_0.0.38-1_amd64.deb to apt.wikimedia.org stretch-wikimedia
• 10:14 jayme: restarting pybal on lvs1015 and lvs2009 for T254581
• 10:12 XioNoX: "Re-order some BGP transit neighbors terms"
• 10:07 marostegui: Deploy schema change on s7 T206103
• 10:00 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 10:00 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 10:00 akosiaris@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 10:00 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 09:57 jayme: restarting pybal on lvs1016 and lvs2010 for T254581
• 09:57 akosiaris: correction: depool and set as inactive thumbor200{1,2} for T251570
• 09:57 akosiaris: depool and set as inactive thumber200{1,2} for T251750
• 09:56 vgutierrez: disable parent proxies on ats-tls
• 09:55 akosiaris@cumin1001: conftool action : set/pooled=inactive; selector: name=thumbor2001.codfw.wmnet
• 09:55 akosiaris@cumin1001: conftool action : set/pooled=inactive; selector: name=thumbor2002.codfw.wmnet
• 09:41 marostegui: Compress InnoDB on db2072 T254462
• 09:34 marostegui: Stop MySQL on db1148 to clone db1141 - T252512
• 09:30 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:29 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1148 to clone db1141 - T252512', diff saved to https://phabricator.wikimedia.org/P11423 and previous config saved to /var/cache/conftool/dbconfig/20200609-092915-marostegui.json
• 09:28 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 09:01 moritzm: rolling restart of cassandra on maps* to pick up Java security updates
• 08:39 moritzm: upgrading snapshot servers to PHP 7.2.31
• 08:28 moritzm: upgrading deployment servers to PHP 7.2.31
• 08:01 marostegui: stop m1 on db1117 to clone db1097 (this will trigger an haproxy irc alert) - T254556
• 07:36 marostegui@cumin1001: dbctl commit (dc=all): 'Remove db1097 from config', diff saved to https://phabricator.wikimedia.org/P11421 and previous config saved to /var/cache/conftool/dbconfig/20200609-073635-marostegui.json
• 07:36 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 07:33 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 07:30 moritzm: upgrading mw1390-mw1413 to PHP 7.2.31
• 07:11 ema: deployment-cache-text06: stop vhtcpd, start purged T254844
• 07:09 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1097:3314, db1097:3315 T253217', diff saved to https://phabricator.wikimedia.org/P11420 and previous config saved to /var/cache/conftool/dbconfig/20200609-070917-marostegui.json
• 06:53 marostegui: Stop MySQL on db2113 for maintenance - T251570
• 06:51 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2113 for on-site maintenance T251570', diff saved to https://phabricator.wikimedia.org/P11419 and previous config saved to /var/cache/conftool/dbconfig/20200609-065125-marostegui.json
• 06:48 marostegui@cumin1001: dbctl commit (dc=all): 'Fully pool db1091 into s1 T253217', diff saved to https://phabricator.wikimedia.org/P11418 and previous config saved to /var/cache/conftool/dbconfig/20200609-064829-marostegui.json
• 06:40 marostegui: Deploy schema change on s2 T206103
• 06:33 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly pool db1091 into s1 T253217', diff saved to https://phabricator.wikimedia.org/P11417 and previous config saved to /var/cache/conftool/dbconfig/20200609-063344-marostegui.json
• 06:19 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly pool db1091 into s1 T253217', diff saved to https://phabricator.wikimedia.org/P11416 and previous config saved to /var/cache/conftool/dbconfig/20200609-061916-marostegui.json
• 05:51 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly pool db1091 into s1 T253217', diff saved to https://phabricator.wikimedia.org/P11415 and previous config saved to /var/cache/conftool/dbconfig/20200609-055128-marostegui.json
• 05:32 marostegui: Switch dbproxy1018 from "master" service to "replicas" - T249188
• 01:02 eileen: civicrm revision changed from 4a19db672f to 80a0d22350, config revision is 386b9bc457
• 00:39 ejegg: updated payments-wiki from c1d14a5db7 to aceddff8b5
• 00:30 shdubsh: restart elasticsearch on logstash1010
• 00:24 eileen: civicrm revision changed from be4c5a4951 to 4a19db672f, config revision is 386b9bc457

2020-06-08

• 23:49 krinkle@deploy1001: Synchronized wmf-config/logging.php: If99192 (duration: 00m 57s)
• 23:35 krinkle@deploy1001: Synchronized wmf-config/logging.php: I8c22a1a8fc402 (duration: 00m 58s)
• 23:32 foks: removing one file for legal compliance
• 23:25 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 23:23 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 23:02 ryankemper@cumin2001: END (FAIL) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=99)
• 22:58 ryankemper@cumin2001: START - Cookbook sre.elasticsearch.rolling-upgrade
• 22:53 ryankemper@cumin2001: END (FAIL) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=99)
• 22:53 shdubsh: update mtail to 3.0.0~rc35 on mw and wtp hosts codfw
• 22:49 eileen: civicrm revision changed from 11b0e7c7e5 to be4c5a4951, config revision is 386b9bc457
• 22:49 ryankemper@cumin2001: START - Cookbook sre.elasticsearch.rolling-upgrade
• 20:52 Amir1: applying the sql alter table on ipblocks on labswiki (T251188)
• 20:27 RoanKattouw: Running initUserPreference.php -s growthexperiments-homepage-enable -t growthexperiments-help-panel-tog-help-panel on wikis that have GrowthExperiments installed (T240920)
• 18:56 Urbanecm: Morning SWATconfig/backport window done
• 18:56 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: 1630a10: Set wgProofreadPagePageJoiner to __PAGEJOIN__ for zhwikisource (T205826) (duration: 00m 58s)
• 18:55 urbanecm@deploy1001: sync-file aborted: SWAT: 1630a10: Set wgProofreadPagePageJoiner to __PAGEJOIN__ for zhwikisource (duration: 00m 00s)
• 18:51 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: 0e85203: Enable subpages in Page namespace on napwikisource (T252755) (duration: 00m 58s)
• 18:44 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 18:42 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 18:28 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: End GrowthExperiments homepage A/B test (T254413) (duration: 00m 57s)
• 18:23 catrope@deploy1001: Synchronized wmf-config/CommonSettings.php: Disable HTCP purges for testwiki (T250781) (part 2) (duration: 00m 56s)
• 18:20 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Disable HTCP purges for testwiki (T250781) (part 1) (duration: 00m 59s)
• 17:50 elukey: restart prometheus burrow exporter for kafka main on kafkamon1001 - T254498
• 17:43 ladsgroup@deploy1001: Synchronized php-1.35.0-wmf.35/resources/src/mediawiki.misc-authed-curate/rollback.js: Fix: Diff pages show rollback confirmation prompt if there is the "Mark as patrolled" link (T254538) (duration: 00m 59s)
• 17:14 elukey@cumin1001: END (PASS) - Cookbook sre.kafka.roll-restart-mirror-maker (exit_code=0)
• 16:55 elukey@cumin1001: START - Cookbook sre.kafka.roll-restart-mirror-maker
• 16:54 elukey@cumin1001: END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0)
• 16:44 liw: testing upcoming Scap release on beta
• 15:29 hnowlan: Migrated all cpjobqueue jobs from scb to Kubernetes
• 15:29 hnowlan@deploy1001: Finished deploy [cpjobqueue/deploy@07d8c32]: Disabling jobs migrated to k8s (duration: 04m 34s)
• 15:28 jynus@cumin2001: dbctl commit (dc=all): 'depool db2075 for mw maintenance T254139', diff saved to https://phabricator.wikimedia.org/P11411 and previous config saved to /var/cache/conftool/dbconfig/20200608-152811-jynus.json
• 15:24 hnowlan@deploy1001: Started deploy [cpjobqueue/deploy@07d8c32]: Disabling jobs migrated to k8s
• 15:12 ladsgroup@deploy1001: Synchronized php-1.35.0-wmf.35/extensions/Wikibase/client/includes/Store/Sql/DirectSqlStore.php: Wrap WAN-cached PropertyInfoLookup with an APCu cache, Part III out of III (T254536) (duration: 00m 57s)
• 15:10 ladsgroup@deploy1001: Synchronized php-1.35.0-wmf.35/extensions/Wikibase/repo/includes/Store/Sql/SqlStore.php: Wrap WAN-cached PropertyInfoLookup with an APCu cache, Part II out of III (T254536) (duration: 00m 57s)
• 15:09 ladsgroup@deploy1001: Synchronized php-1.35.0-wmf.35/extensions/Wikibase/lib/includes/Store/CachingPropertyInfoLookup.php: Wrap WAN-cached PropertyInfoLookup with an APCu cache, Part I out of III (T254536) (duration: 00m 59s)
• 15:05 hnowlan@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' .
• 14:53 cdanis: ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕚☕ sudo cumin A:mw-canary 'enable-puppet "cdanis deploying I25ab44c1 T252605"'
• 14:52 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' .
• 14:48 papaul: powering down ms-be2016 for BBU replacement
• 14:47 cdanis: ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕚☕ sudo cumin A:mw-canary 'disable-puppet "cdanis deploying I25ab44c1 T252605"'
• 14:41 moritzm: upgrading mw API servers in codfw to PHP 7.2.31
• 14:00 jbond42: updating puppet-merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/602738/4
• 13:58 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:58 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 13:50 urbanecm@deploy1001: Synchronized private/PrivateSettings.php: Update mitigations for T250887 (duration: 00m 57s)
• 13:41 elukey@cumin1001: START - Cookbook sre.kafka.roll-restart-brokers
• 12:23 XioNoX: repool codfw - T243080
• 12:18 XioNoX: rollback cr2-codfw vrrp/ospf/bgp changes - T243080
• 12:18 marostegui: Compress InnoDB on db2094:3311 T254462
• 12:09 XioNoX: cr2-codfw> request chassis routing-engine master switch - T243080
• 12:05 XioNoX: reboot cr2-codfw:re0 (backup) - T243080
• 11:53 XioNoX: cr2-codfw> request chassis routing-engine master switch - T243080
• 11:53 moritzm: restarting dnsdist on malmok
• 11:53 marostegui: Deploy schema change on s3 - T251188
• 11:49 XioNoX: reboot cr2-codfw:re1 (backup) - T243080
• 11:45 moritzm: restarting slapd on ldap-corp* for Gnu TLS security update
• 11:43 moritzm: rolling restart of Apache on Kibana/7 host to pick up Gnu TLS security update
• 11:41 XioNoX: de-pref cr2-codfw OSPF - T243080
• 11:39 XioNoX: deactivate cr2-codfw transit/peering - T243080
• 11:38 XioNoX: fail vrrp master from cr2 to cr1 - T243080
• 11:32 XioNoX: cr1-codfw set OSPF metrics back to normal - T243080
• 11:30 XioNoX: cr1-codfw re-enable transit/peering - T243080
• 11:29 XioNoX: cr1-codfw add graceful-restart - T243080
• 11:28 XioNoX: cr1-codfw add graceful-switchover - T243080
• 11:18 Lucas_WMDE: EU SWAT done
• 11:16 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove Wikibase idBlacklist setting (T254686), part 2 (duration: 00m 56s)
• 11:15 XioNoX: cr1-codfw> request chassis routing-engine master switch - T243080
• 11:15 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/Wikibase.php: SWAT: Remove Wikibase idBlacklist setting (T254686), part 1 (duration: 00m 56s)
• 11:11 XioNoX: reboot cr1-codfw:re0 (backup) - T243080
• 11:09 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable GrowthExperiments guidance everywhere behind feature flag (T253794) (duration: 00m 57s)
• 11:05 marostegui: Install events on es1 T254689
• 11:05 XioNoX: install Junos on cr1-codfw:re0 (backup) - T243080
• 10:56 XioNoX: do cr1-codfw RE mastership switch - T243080
• 10:53 XioNoX: reboot cr1-codfw:re1 (backup) - T243080
• 10:46 XioNoX: install Junos on cr1-codfw:re1 (backup) - T243080
• 10:43 XioNoX: deactivate cr1-codfw transit/peering - T243080
• 10:41 XioNoX: bump all cr1-codfw OSPF metrics - T243080
• 10:41 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (603408) (duration: 00m 57s)
• 10:40 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (603408) (duration: 01m 09s)
• 10:39 XioNoX: depool codfw - T243080
• 09:46 moritzm: installing gnutls28 security updates on buster (older releases not affected)
• 09:32 qchris: Turning on puppet on gerrit1002 again to avoid starting to lag too far behind
• 08:17 XioNoX: push T250136 to eqsin - T250136
• 08:09 XioNoX: push T250136 to eqiad - T250136
• 08:07 moritzm: upgrading mw1349-mw1383 to PHP 7.2.31
• 08:07 mutante: stat1006 moved broken jupyter-dedcode-singleuser.service out of /run/systemd/transient. systemctl reset-failed
• 08:02 XioNoX: push T250136 to codfw - T250136
• 07:58 XioNoX: push T250136 to eqord/eqdfw - T250136
• 07:58 mutante: stat1006 bash[40607]: /bin/bash: line 0: exec: jupyterhub-singleuser: not found
• 07:57 mutante: ran puppet on all stat* hosts for an access request (dcipoletti was added) - stat1006 systemd state broke right after, jupyter-dedcode-singleuser.service failed
• 07:46 XioNoX: push T250136 to esams/knams - T250136
• 07:42 XioNoX: cr4-ulsfo protocols bgp group Transit4 family inet any -> unicast - T250136
• 07:39 XioNoX: cr3-ulsfo protocols bgp group Transit4 family inet any -> unicast - T250136
• 07:37 moritzm: installing nodejs security updates
• 07:05 marostegui: Stop MySQL on labsdb1012 to clone labsdb1011 T249188
• 05:22 marostegui: Upgrade db1077 to 10.4.13 to test events memory leak
• 04:45 _joe_: de-firewalling mc1029

• 04:27 _joe_: firewallingf off memcached on mc1029

2020-06-05

• 16:45 elukey@deploy1001: Finished deploy [analytics/turnilo/deploy@f7e4f78]: Upgrade to 1.24.0 (duration: 00m 11s)
• 16:45 elukey@deploy1001: Started deploy [analytics/turnilo/deploy@f7e4f78]: Upgrade to 1.24.0
• 16:29 bd808: Testing stashbot following hard restart of service. It was having LDAP connection failure problems.
• 16:00 AndyRussG: Turned off Fundraising job recurring_smashpig_charge
• 15:54 cdanis: enabling & rerunning puppet on netflow* T254574
• 15:39 cdanis: disabling puppet on netflow* and trying I6598d8f8 on netflow3001 first T254574
• 15:39 cdanis: disabling puppet on netflow* and trying I6598d8f8 on netflow3001 first
• 13:33 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'mathoid' for release 'staging' .
• 13:19 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'cxserver' for release 'staging' .
• 13:19 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:19 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'citoid' for release 'staging' .
• 13:18 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' .
• 13:15 elukey@cumin1001: START - Cookbook sre.hosts.downtime
• 12:55 ladsgroup@deploy1001: Synchronized wmf-config/interwiki.php: Hotfix for be-tarask interwiki link being broken (T111853) (duration: 01m 00s)
• 12:41 mutante: rebooting gerrit1002 to add more vCPUs, after [ganeti1009:~] $ sudo gnt-instance modify -B vcpus=8 gerrit1002.wikimedia.org T239151
• 12:20 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'zotero' for release 'staging' .
• 12:19 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' .
• 12:19 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'cxserver' for release 'staging' .
• 12:19 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'citoid' for release 'staging' .
• 12:19 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' .
• 12:17 akosiaris: update blubberoid changeprop changeprop-jobqueue citoid cxserver wikifeeds zotero in staging to latest charts
• 12:17 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' .
• 12:17 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'blubberoid' for release 'staging' .
• 12:17 akosiaris: fix typo in ganeti2016 /etc/network/interfaces and reboot
• 11:28 akosiaris: master-failover from ganeti2001 to ganeti2019 for ganeti01.svc.codfw.wmnet
• 11:25 akosiaris@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' .
• 11:25 akosiaris@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' .
• 11:25 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' .
• 11:14 mutante: running puppet on all ganeti nodes
• 11:05 ladsgroup@deploy1001: Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 02m 14s)
• 10:32 elukey@cumin1001: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0)
• 10:11 elukey@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 10:02 dzahn@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 09:49 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'mathoid' for release 'staging' .
• 09:46 elukey@cumin1001: START - Cookbook sre.ganeti.makevm
• 09:25 elukey@cumin1001: START - Cookbook sre.cassandra.roll-restart
• 09:03 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:00 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 08:44 akosiaris: reimage ganeti2016 for stretch
• 08:42 akosiaris: migrate mx2001.wikimedia.org to new ganeti nodes
• 08:40 akosiaris: migrate acrab to new ganeti nodes
• 08:38 akosiaris: failover master IP from ganeti1003 to ganeti1009
• 08:37 akosiaris: empty ganeti100{1,2,3,4}. Move all VMs to new ganeti nodes
• 08:28 akosiaris: migrate seaborgium.wikimedia.org to new ganeti nodes
• 08:27 akosiaris: migrate etherpad1002 to new ganeti nodes
• 08:11 marostegui: Upgrade db2075 to 10.1.45
• 07:52 vgutierrez: rolling restart of ats-tls - T249335
• 07:20 dzahn@cumin1001: START - Cookbook sre.ganeti.makevm
• 06:20 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 06:17 elukey@cumin1001: START - Cookbook sre.hosts.downtime

2020-06-04

• 23:45 catrope@deploy1001: Synchronized wmf-config/mc.php: Set coalesceKeys=non-global for WANCache on enwiki (duration: 00m 59s)
• 23:29 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable Minerva site notices on Wikivoyage wiis (T254391) (duration: 00m 58s)
• 23:19 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Set guwiki timezone to Asia/Kolkata (T253827) (duration: 00m 57s)
• 23:17 catrope@deploy1001: Synchronized static/images/: Change logo for zhwiki (T254467) (duration: 01m 00s)
• 22:56 ryankemper: re-enabled puppet on `cloudelastic1006`. All `cloudelastic` instances now have puppet enabled and are in sync
• 20:56 ryankemper: enabled puppet on `cloudelastic1005` in order to kick off a puppet run and verify that this new node joins the ES cluster properly
• 20:39 ryankemper: disabled puppet on `cloudelastic100[5,6]` which are two racked nodes that we are now bringing into service. Will re-enable after successful puppet-merge / elasticsearch cluster join
• 20:38 ryankemper: disabled puppet on `cloudelastic100[5,6]` which are two racked nodes that we are now bringing into service. Will re-enable after successful puppet-merge / elasticsearch cluster join
• 19:04 jforrester@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.35
• 15:12 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=druid1004.eqiad.wmnet
• 15:11 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:10 hnowlan@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' .
• 15:08 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:07 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 15:06 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 14:37 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' .
• 14:36 moritzm: installing libexif security updates on jessie
• 14:08 moritzm: installing clamav security updates on mendelevium (ticket.wikimedia.org)
• 14:00 qchris: Stopping puppet on gerrit1002 (gerrit-test) to run tests for Gerrit upgrade
• 13:41 moritzm: bounced ferm on ms-be1023
• 13:35 moritzm: installing exim security updates on jessie (stretch/buster already done)
• 12:54 urbanecm@deploy1001 Synchronized wmf-config/InitialiseSettings.php: c06e720: Revert "wgNamespaceRobotPolicies: thwiki: Add 100 NS to noindex" (T253574) (duration: 01m 06s)
• 12:18 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'cxserver' for release 'production' .
• 12:14 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'citoid' for release 'production' .
• 12:02 moritzm: upgrading mw1276 to PHP 7.2.31
• 11:59 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1107', diff saved to https://phabricator.wikimedia.org/P11396 and previous config saved to /var/cache/conftool/dbconfig/20200604-115933-marostegui.json
• 11:53 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: ec07467: wgNamespaceRobotPolicies: thwiki: Add 100 NS to noindex (T253574) (duration: 01m 15s)
• 11:49 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 11:49 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: 338cb90: 1ade16f: Change $wgNamespaceRobotPolicies on Thai wikis (T253578; T253577; T253576; T253575; T253574) (duration: 01m 07s)
• 11:46 elukey@cumin1001: START - Cookbook sre.hosts.downtime
• 11:41 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1107', diff saved to https://phabricator.wikimedia.org/P11395 and previous config saved to /var/cache/conftool/dbconfig/20200604-114149-marostegui.json
• 11:29 marostegui: Compress InnoDB on db1091 before pooling it as new slave on s1 - T254462
• 11:21 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [metawiki] Add `centralauth-rename` to WMF OIT staff - T254372 (duration: 01m 08s)
• 11:14 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 11:12 elukey@cumin1001: START - Cookbook sre.hosts.downtime
• 11:04 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'wikifeeds' for release 'production' .
• 10:59 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'cxserver' for release 'production' .
• 10:53 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'citoid' for release 'production' .
• 10:53 elukey@puppetmaster1001: conftool action : set/pooled=no; selector: name=druid1004.eqiad.wmnet
• 10:46 marostegui: Deploy schema change on s3 (only testwiki) eqiad - T238966
• 10:42 marostegui: Deploy schema change on s3 (only testwiki) codfw - T238966
• 10:41 jbond42: deployed new version of puppet-merge revert is https://gerrit.wikimedia.org/r/c/operations/puppet/+/602329
• 09:57 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' .
• 09:56 akosiaris@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 09:56 akosiaris@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 09:55 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:55 akosiaris@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 09:55 akosiaris@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 09:54 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'cxserver' for release 'staging' .
• 09:53 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:51 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 09:51 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 09:50 akosiaris@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 09:50 akosiaris@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 09:50 akosiaris@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 09:50 akosiaris@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 09:50 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:50 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 09:50 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 09:50 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 09:50 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 09:50 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 09:50 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 09:50 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 09:48 akosiaris@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 09:48 akosiaris@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 09:48 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 09:48 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 09:48 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 09:48 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 09:46 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'citoid' for release 'staging' .
• 09:42 jmm@cumin2001: END (FAIL) - Cookbook sre.cassandra.roll-restart (exit_code=99)
• 09:42 jmm@cumin2001: START - Cookbook sre.cassandra.roll-restart
• 09:41 jmm@cumin2001: END (FAIL) - Cookbook sre.cassandra.roll-restart (exit_code=99)
• 09:41 jmm@cumin2001: START - Cookbook sre.cassandra.roll-restart
• 09:26 moritzm: rolling restart of cassandra on maps* to pick up Java security updates
• 09:09 akosiaris@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 09:08 akosiaris@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 09:08 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:05 akosiaris@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 09:05 akosiaris@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 09:05 akosiaris@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 09:05 akosiaris@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 09:05 akosiaris@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 09:05 akosiaris@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 09:05 akosiaris@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 09:04 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 09:04 akosiaris@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 09:03 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 09:03 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 09:03 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 09:03 moritzm: deploying Java security updates on elastic search nodes
• 09:03 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:00 akosiaris@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 09:00 akosiaris@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:00 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 09:00 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 09:00 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 09:00 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 09:00 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 09:00 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 08:59 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 08:59 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 08:59 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 08:58 akosiaris@cumin1001: START - Cookbook sre.hosts.downtime
• 08:50 marostegui: Repool labsdb1009 after running maintain-views T252219
• 08:42 moritzm: restarting archiva to pick up Java security updates
• 08:15 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1107 to clone db1091 on s1 T253217', diff saved to https://phabricator.wikimedia.org/P11392 and previous config saved to /var/cache/conftool/dbconfig/20200604-081545-marostegui.json
• 08:14 marostegui: Run sudo /usr/local/sbin/maintain-views --all-databases --replace-all on labsdb1009 - T252219
• 07:49 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 07:45 marostegui: Depool labsdb1009 - T252219
• 07:45 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 07:33 oblivian@puppetmaster1001: conftool action : set/weight=10; selector: dc=eqiad,cluster=labweb,service=labweb-ssl
• 07:32 oblivian@puppetmaster1001: conftool action : set/pooled=yes:weight=10; selector: dc=eqiad,cluster=cloudceph,service=cloudceph
• 06:52 mutante: mwmaint1002 started mediawiki_job_cirrus_build_completion_indices_eqiad.service
• 06:06 oblivian@puppetmaster1001: conftool action : set/weight=10; selector: name=logstash200.*
• 06:05 oblivian@puppetmaster1001: conftool action : set/weight=10; selector: name=logstash100.*
• 06:04 oblivian@puppetmaster1001: conftool action : set/weight=10; selector: cluster=eventschemas,service=eventschemas
• 06:02 oblivian@puppetmaster1001: conftool action : set/weight=10; selector: dc=codfw,cluster=elasticsearch,service=elasticsearch.*
• 06:01 oblivian@puppetmaster1001: conftool action : set/weight=10; selector: dc=codfw,cluster=elasticsearch,service=elasticsearch
• 06:00 _joe_: fixing weights of cp2040 T245594
• 05:31 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 05:28 elukey@cumin1001: START - Cookbook sre.hosts.downtime
• 00:36 reedy@deploy1001: Synchronized php-1.35.0-wmf.35/includes/specials/SpecialUserrights.php: T254417 T251534 (duration: 01m 06s)

2020-06-03

• 23:08 reedy@deploy1001: Synchronized wmf-config/CommonSettings-labs.php: T249834 (duration: 01m 06s)
• 23:06 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: T249834 (duration: 01m 06s)
• 22:22 ryankemper@cumin2001: END (PASS) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=0)
• 21:54 jforrester@deploy1001: rebuilt and synchronized wikiversions files: Re-rolling group1 to 1.35.0-wmf.35 for T253023
• 21:49 jforrester@deploy1001: Synchronized php-1.35.0-wmf.35/extensions/EventStreamConfig/includes/ApiStreamConfigs.php: T254390 ApiStreamConfigs: If the 'constraints' parameter is unset, don't explode (duration: 01m 06s)
• 21:43 cstone: civicrm revision changed from 63508b01b9 to 11b0e7c7e5
• 21:16 ryankemper@cumin2001: START - Cookbook sre.elasticsearch.rolling-upgrade
• 21:15 ryankemper: The previously ran `_cluster/reroute?retry_failed=true` command worked as intended, the two shards in question have recovered and we're back to green cluster status. We're now in a known state and ready to proceed with the eqiad rolling upgrade
• 21:13 ryankemper: Ran `curl -X POST "https://localhost:9243/_cluster/reroute?pretty&retry_failed=true&explain=true" -H 'Content-Type: application/json' -d '{}' --insecure` via the ssh tunnel `ssh bast4002.wikimedia.org -L 9243:search.svc.eqiad.wmnet:9243 -L 9443:search.svc.eqiad.wmnet:9443 -L 9643:search.svc.eqiad.wmnet:9643`, two unassigned shards are now initializing
• 21:05 ryankemper: Elasticsearch Eqiad was in yellow cluster status before starting the above cookbook run (therefore the run was a no-op until I ctlr+C'd), going to try unsticking the two unassigned shards via `/_cluster/reroute?retry_failed=true`
• 21:03 ryankemper@cumin2001: END (ERROR) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=97)
• 20:58 ryankemper@cumin2001: START - Cookbook sre.elasticsearch.rolling-upgrade
• 20:52 ryankemper@cumin2001: END (PASS) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=0)
• 20:49 eileen: civicrm revision changed from eb156dffa4 to 63508b01b9, config revision is 95dcdb0a8a
• 20:47 ryankemper@cumin2001: START - Cookbook sre.elasticsearch.rolling-upgrade
• 20:19 gehel: elasticsearch cluster restart stopped
• 20:18 ryankemper@cumin2001: END (ERROR) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=97)
• 19:35 ppchelko@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' .
• 19:35 ppchelko@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-main' for release 'production' .
• 19:33 ppchelko@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-main' for release 'production' .
• 19:32 ppchelko@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' .
• 19:30 ryankemper@cumin2001: START - Cookbook sre.elasticsearch.rolling-upgrade
• 19:29 ppchelko@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-main' for release 'production' .
• 19:29 ppchelko@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' .
• 19:20 jforrester@deploy1001: rebuilt and synchronized wikiversions files: Revert group1 wikis to wmf.34 T253023
• 19:16 hnowlan@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' .
• 19:15 hnowlan@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' .
• 19:14 jforrester@deploy1001: Synchronized php: group1 wikis to 1.35.0-wmf.35 (duration: 01m 05s)
• 19:13 jforrester@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.35
• 19:05 jforrester@deploy1001: Synchronized dblists/mobilemainpagelegacy.dblist: T32405 Stop special casing the main page on another 47 projects (duration: 01m 08s)
• 19:01 ppchelko@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: gerrit 601843 Enable talk pages on Swedish Minerva (duration: 01m 08s)
• 18:59 hnowlan@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' .
• 18:56 hnowlan@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' .
• 18:55 ppchelko@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: gerrit 601842 - Disable growth survey (duration: 01m 06s)
• 18:49 ppchelko@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: gerrit 596277 Use AddFooterLink hook for code of conduct and contact links (duration: 01m 05s)
• 18:34 ppchelko@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: gerrit 599150 - enable kafka purges for group0 (duration: 01m 06s)
• 18:19 ppchelko@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: gerrit 570396 - enable kask-session everywhere. CS.php (duration: 01m 05s)
• 18:14 ppchelko@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: gerrit 570396 - enable kask-session everywhere. IS.php (duration: 01m 06s)
• 17:15 ejegg: updated payments-wiki from e46114d8b1 to c1d14a5db7
• 17:08 elukey: ganeti: gnd-instance reboot an-launcher1001 to get new memory settings - T254125
• 15:21 hnowlan@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' .
• 15:19 hnowlan@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' .
• 15:12 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' .
• 14:50 kormat@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool pc1009 in pc3 after reimaging T252182 (duration: 01m 06s)
• 14:47 moritzm: updated grafana on cloudmetrics* to 6.7.4
• 14:26 kormat: stopping replication on pc1010
• 14:20 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:17 kormat@cumin1001: START - Cookbook sre.hosts.downtime
• 14:16 gehel: cleaning commonsrdf-dumps cron entry manually on snapshot1008
• 14:00 hashar: Restarted CI Jenkins for plugin update
• 13:59 kormat@deploy1001: Synchronized wmf-config/db-eqiad.php: Replace pc1009 with pc1010 reimaging T252182 (duration: 01m 06s)
• 13:47 kormat: reimaging *pc1009 (promise) to buster T252182
• 13:44 kormat: reimaging pc1007 to buster, wish me luck T252182
• 13:20 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' .
• 13:13 kormat@deploy1001: Synchronized wmf-config/db-codfw.php: Put pc2009 back into pc3 after reimaging T252182 (duration: 01m 05s)
• 13:01 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' .
• 12:01 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1080', diff saved to https://phabricator.wikimedia.org/P11385 and previous config saved to /var/cache/conftool/dbconfig/20200603-120136-marostegui.json
• 11:57 moritzm: updating linux-libc-dev on stretch and buster hosts
• 11:56 XioNoX: configure management-instance on cr1/2-eqiad - T247073
• 11:51 XioNoX: configure management-instance on cr2-codfw - T247073
• 11:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db2124 after MCR schema change', diff saved to https://phabricator.wikimedia.org/P11384 and previous config saved to /var/cache/conftool/dbconfig/20200603-114409-marostegui.json
• 11:43 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1080', diff saved to https://phabricator.wikimedia.org/P11383 and previous config saved to /var/cache/conftool/dbconfig/20200603-114351-marostegui.json
• 11:31 Lucas_WMDE: EU SWAT done
• 11:30 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes.php eswiki --fix | tee T254077.fix
• 11:29 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes.php eswiki | tee T254077.dry-run
• 11:27 moritzm: installing rubygems-integration updates for Buster
• 11:26 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [eswiki] Normalize talk namespaces for Anexo, Portal and Wikiproyecto (T254077) (duration: 01m 03s)
• 11:25 moritzm: install brltty updates on Buster
• 11:23 XioNoX: configure management-instance on cr1-codfw - T247073
• 11:19 XioNoX: configure management-instance on cr1-eqsin - T247073
• 11:15 moritzm: installing python-oslo.utils security updates
• 11:12 XioNoX: remove unused logical-systems from all MX204 routers - T247073
• 11:10 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2124', diff saved to https://phabricator.wikimedia.org/P11382 and previous config saved to /var/cache/conftool/dbconfig/20200603-111055-marostegui.json
• 11:08 marostegui: Add rev_id to revision table on db2124 - T238966
• 11:05 moritzm: installing pango updates for buster
• 10:42 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1091 - will be reimaged and moved to s1 T252512', diff saved to https://phabricator.wikimedia.org/P11381 and previous config saved to /var/cache/conftool/dbconfig/20200603-104251-marostegui.json
• 10:14 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1080', diff saved to https://phabricator.wikimedia.org/P11380 and previous config saved to /var/cache/conftool/dbconfig/20200603-101426-marostegui.json
• 09:38 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1080', diff saved to https://phabricator.wikimedia.org/P11378 and previous config saved to /var/cache/conftool/dbconfig/20200603-093810-marostegui.json
• 09:28 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:25 kormat@cumin1001: START - Cookbook sre.hosts.downtime
• 09:21 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:18 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 09:10 moritzm: upgrading mw1262-1265 to PHP 7.2.31
• 09:08 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:05 elukey@cumin1001: START - Cookbook sre.hosts.downtime
• 09:04 marostegui: Reimage db1080
• 09:01 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1080 for reimage', diff saved to https://phabricator.wikimedia.org/P11376 and previous config saved to /var/cache/conftool/dbconfig/20200603-090143-marostegui.json
• 08:42 kormat@deploy1001: Synchronized wmf-config/db-codfw.php: Replace pc2009 with pc2010 while reimaging (duration: 01m 16s)
• 08:19 moritzm: upgrading mw1261 to PHP 7.2.31
• 08:17 XioNoX: re-add ae2 physical interfaces to external group - T253970
• 08:09 moritzm: upgrading remaining mwdebug* servers to PHP 7.2.31
• 08:08 kormat: reimaging pc2009 to buster T252182
• 08:08 XioNoX: remove ae2 physical interfaces from external group - T253970
• 07:52 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
• 07:51 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
• 07:48 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
• 07:45 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
• 07:44 dzahn@cumin1001: conftool action : set/pooled=inactive; selector: name=mw218[0-6].codfw.wmnet
• 07:36 mutante: depooling mw2180 - mw2186
• 07:36 dzahn@cumin1001: conftool action : set/pooled=no; selector: name=mw218[0-6].codfw.wmnet
• 07:35 moritzm: imported PHP 7.2.31 to apt.wikimedia.org/component/php72
• 07:33 ema: cp: upgrade purged to 0.15
• 07:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db2071 after cloning it from db2130 to restore all the schema changes applied', diff saved to https://phabricator.wikimedia.org/P11375 and previous config saved to /var/cache/conftool/dbconfig/20200603-072841-marostegui.json
• 07:15 XioNoX: repool esams - T254021
• 07:09 XioNoX: re-activate peering/transit BGP on cr2-esams - T254021
• 07:00 XioNoX: re0.cr2-esams> request system reboot both-routing-engines - T254021
• 06:56 XioNoX: deactivate peering/transit BGP cr2-esams - T244497
• 06:54 XioNoX: failover vrrp to cr3-esams - T244497
• 06:37 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1138 T253808', diff saved to https://phabricator.wikimedia.org/P11374 and previous config saved to /var/cache/conftool/dbconfig/20200603-063752-marostegui.json
• 06:18 XioNoX: cr3-esams> request chassis routing-engine master switch - T244497
• 06:11 XioNoX: cr3-esams> request vmhost reboot re1 (backup re) - T244497
• 06:08 XioNoX: re-activate transit BGP to cr3-knams - T254021
• 06:01 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1138 T253808', diff saved to https://phabricator.wikimedia.org/P11373 and previous config saved to /var/cache/conftool/dbconfig/20200603-060124-marostegui.json
• 05:58 XioNoX: reboot cr3-knams - T254021
• 05:51 XioNoX: deactivate transit BGP ton cr3-knams - T254021
• 05:48 XioNoX: depool esams - T254021
• 05:41 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2130 to clone db2071', diff saved to https://phabricator.wikimedia.org/P11371 and previous config saved to /var/cache/conftool/dbconfig/20200603-054117-marostegui.json
• 05:40 marostegui: Stop MySQL on db2130 to clone db2071
• 05:38 XioNoX: deactivate graceful-switchover on cr3-esams - T254021
• 05:37 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1138 T253808', diff saved to https://phabricator.wikimedia.org/P11370 and previous config saved to /var/cache/conftool/dbconfig/20200603-053748-marostegui.json
• 05:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 05:32 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 05:14 XioNoX: turn cr1-codfw:fpc0 online - T254110
• 05:09 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1138 T253808', diff saved to https://phabricator.wikimedia.org/P11369 and previous config saved to /var/cache/conftool/dbconfig/20200603-050911-marostegui.json
• 01:00 reedy@deploy1001: Synchronized wmf-config/CommonSettings-labs.php: wgLocalVirtualHosts (duration: 01m 06s)
• 00:59 krinkle@deploy1001: Synchronized wmf-config/mc.php: Ic27b60 (duration: 01m 11s)

2020-06-02

• 23:58 ejegg: updated fundraising CiviCRM from 657c4b9455 to eb156dffa4
• 23:55 ejegg: updated payments-wiki from 1942a537ef to e46114d8b1
• 22:48 cstone: civicrm revision changed from d1cd99166f to 657c4b9455
• 21:48 reedy@deploy1001: Synchronized wmf-config/interwiki-labs.php: laaaaabs (duration: 01m 05s)
• 21:23 reedy@deploy1001: Synchronized multiversion/MWMultiVersion.php: beta apiportalwiki T254185 (duration: 01m 06s)
• 21:21 reedy@deploy1001: Synchronized wmf-config/config/apiportalwiki.yaml: beta apiportalwiki T254185 (duration: 01m 05s)
• 21:20 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: beta apiportalwiki T254185 (duration: 01m 05s)
• 21:19 reedy@deploy1001: Synchronized wikiversions-labs.json: beta apiportalwiki T254185 (duration: 01m 05s)
• 21:17 reedy@deploy1001: Synchronized dblists/all-labs.dblist: beta apiportalwiki T254185 (duration: 01m 06s)
• 21:12 cdanis: repooled wtp1032 T254258
• 20:24 reedy@deploy1001: Synchronized composer.lock: Update (duration: 01m 06s)
• 20:02 jforrester@deploy1001: rebuilt and synchronized wikiversions files: group0 wikis to 1.35.0-wmf.35
• 19:59 jforrester@deploy1001: Finished scap: testwikis wikis to 1.35.0-wmf.35 (duration: 93m 52s)
• 18:25 jforrester@deploy1001: Started scap: testwikis wikis to 1.35.0-wmf.35
• 18:22 jforrester@deploy1001: Pruned MediaWiki: 1.35.0-wmf.31 (duration: 19m 59s)
• 18:05 cdanis: fixing g+w permissions of deploy1001 /srv/mediawiki-staging/php-*/.git/objects/*
• 17:20 James_F: 1.35.0-wmf.35 was branched at 8d70150 for T253023
• 16:55 cstone: SmashPig revision changed from 44690f761c to b9de3c7aac
• 15:57 ejegg: updated payments-wiki from d11efeb1cf to 1942a537ef
• 15:50 cdanis: thumbor1003 and thumbor1004 blipped, no obvious explanation, logs gathered at P11365 P11366 P11367
• 15:49 XioNoX: push frack fw rules - T254260
• 15:48 mutante: contint1001 - rm -rf /mnt/docker (T224591)
• 15:45 mutante: contint1001 - restarting docker afer changed data-root path (T224591)
• 15:37 cdanis@cumin1001: conftool action : set/pooled=no; selector: name=wtp1032.*
• 15:35 cdanis: power cycling wtp1032 which is bootlooping? https://phabricator.wikimedia.org/P11364
• 15:31 hnowlan@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' .
• 15:24 rzl@cumin1001: conftool action : set/pooled=yes; selector: name=thumbor100[34].*
• 15:23 XioNoX: repool codfw - T254216
• 15:19 XioNoX: rollback ospf changes - T254216
• 15:09 hnowlan@deploy1001: Finished deploy [cpjobqueue/deploy@8a53ff1]: (no justification provided) (duration: 02m 33s)
• 15:07 XioNoX: reboot cr1-codfw:fpc5 - T254216
• 15:06 hnowlan@deploy1001: Started deploy [cpjobqueue/deploy@8a53ff1]: (no justification provided)
• 15:05 hnowlan: shifting all high traffic cpjobqueue rules to k8s
• 14:57 hnowlan@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' .
• 14:57 XioNoX: depref ulsfo-codfw link - T254216
• 14:51 hnowlan@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' .
• 14:50 jynus@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 14:49 jynus@cumin1001: START - Cookbook sre.hosts.downtime
• 14:49 XioNoX: prefer eqsin-ulsfo tunnel - T254216
• 14:47 cdanis@cumin1001: conftool action : set/pooled=no; selector: name=thumbor100[34].*
• 14:38 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' .
• 14:31 XioNoX: depool codfw - T254216
• 14:09 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' .
• 13:45 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
• 13:42 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
• 13:42 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
• 13:38 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
• 13:37 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
• 13:28 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
• 13:19 dzahn@cumin1001: END (ERROR) - Cookbook sre.hosts.decommission (exit_code=97)
• 13:18 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
• 13:18 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1)
• 13:18 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
• 13:05 cdanis@deploy1001: Synchronized php-1.35.0-wmf.31/includes/specials/pagers/ContribsPager.php: revert contribs limit to 5000 T234450 (duration: 00m 57s)
• 13:04 cdanis@deploy1001: Synchronized php-1.35.0-wmf.32/includes/specials/pagers/ContribsPager.php: revert contribs limit to 5000 T234450 (duration: 00m 57s)
• 13:03 cdanis@deploy1001: Synchronized php-1.35.0-wmf.34/includes/specials/pagers/ContribsPager.php: revert contribs limit to 5000 T234450 (duration: 00m 58s)
• 12:59 marostegui@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 12:59 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 12:56 cdanis@deploy1001: Synchronized wmf-config/PoolCounterSettings.php: 5debc3223 limit per-user Special:Contributions concurrency to 2 T234450 (duration: 00m 58s)
• 12:50 kormat@cumin1001: dbctl commit (dc=all): 'Pool db2140 into s4 T252985', diff saved to https://phabricator.wikimedia.org/P11363 and previous config saved to /var/cache/conftool/dbconfig/20200602-125012-kormat.json
• 12:39 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' .
• 12:31 dzahn@cumin1001: conftool action : set/pooled=inactive; selector: name=mw217[3-9].codfw.wmnet
• 12:30 kormat@cumin1001: dbctl commit (dc=all): 'Repool db2110, copy to db2140 complete T252985', diff saved to https://phabricator.wikimedia.org/P11362 and previous config saved to /var/cache/conftool/dbconfig/20200602-123020-kormat.json
• 12:28 dzahn@cumin1001: conftool action : set/pooled=no; selector: name=mw217[3-9].codfw.wmnet
• 11:10 kart_: Finished EU Mid-day SWAT.
• 11:08 mutante: contint1001 - common issue after reinstalls again - a2dismod mpm_event ; systemctl restart apache2 ; puppet agent -tv ( T196968) https://gerrit.wikimedia.org/r/c/operations/puppet/+/451206
• 11:07 kartik@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: 601174|Create URL campaign for African languages for COVID-19 translation project (T253305) (duration: 01m 00s)
• 11:01 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' .
• 10:48 mutante: LDAP - added uid=lulu to group nda (T254121)
• 10:29 akosiaris: switch over ores1XXX hosts to redis::misc from oresrdb hosts. T254226
• 10:12 jynus: disable non-global root login to gerrit2001 T254162
• 10:12 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 10:11 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1121, db1148 T252512', diff saved to https://phabricator.wikimedia.org/P11361 and previous config saved to /var/cache/conftool/dbconfig/20200602-101150-marostegui.json
• 10:09 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 10:09 akosiaris: switch over ores2XXX hosts to redis::misc from oresrdb hosts. T254226
• 10:02 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1121, db1148 T252512', diff saved to https://phabricator.wikimedia.org/P11360 and previous config saved to /var/cache/conftool/dbconfig/20200602-100246-marostegui.json
• 09:53 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1121, db1148 T252512', diff saved to https://phabricator.wikimedia.org/P11359 and previous config saved to /var/cache/conftool/dbconfig/20200602-095321-marostegui.json
• 09:49 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1138', diff saved to https://phabricator.wikimedia.org/P11358 and previous config saved to /var/cache/conftool/dbconfig/20200602-094914-marostegui.json
• 09:44 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1121, db1148 T252512', diff saved to https://phabricator.wikimedia.org/P11357 and previous config saved to /var/cache/conftool/dbconfig/20200602-094441-marostegui.json
• 09:38 marostegui@cumin1001: dbctl commit (dc=all): 'Add db1148 to dbctl depooled T252512', diff saved to https://phabricator.wikimedia.org/P11356 and previous config saved to /var/cache/conftool/dbconfig/20200602-093841-marostegui.json
• 08:59 ema: upload purged 0.15 to buster-wikimedia
• 08:09 mutante: re-imaging contint1001 with buster
• 07:43 marostegui: Stop MySQL on db1121
• 07:40 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1121 to clone db1148', diff saved to https://phabricator.wikimedia.org/P11353 and previous config saved to /var/cache/conftool/dbconfig/20200602-074027-marostegui.json
• 07:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1079 after data check', diff saved to https://phabricator.wikimedia.org/P11351 and previous config saved to /var/cache/conftool/dbconfig/20200602-073245-marostegui.json
• 07:22 marostegui: Stop slave on db1079 for data check
• 07:22 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1079 for data check', diff saved to https://phabricator.wikimedia.org/P11350 and previous config saved to /var/cache/conftool/dbconfig/20200602-072214-marostegui.json
• 07:06 marostegui: Stop MySQL and poweroff on db1138 for on-site maintenance - T253808
• 05:01 marostegui: Stop mysql on db1141 to save a binary backup - T249188
• 01:03 krinkle@deploy1001: Synchronized wmf-config/mc.php: I06897bcc92c5 (duration: 00m 59s)

2020-06-01

• 20:14 shdubsh: downgrade mtail to rc5 in ulsfo -- T254192
• 20:12 XioNoX: enable IX4/6 on cr4-ulsfo - T237575
• 19:57 XioNoX: disable IX4/6 on cr4-ulsfo - T237575
• 19:55 XioNoX: fail vrrp over cr3-ulsfo - T237575
• 19:44 shdubsh: restart atsmtail in eqsin
• 18:21 ppchelko@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable kask-transition for all wikis (duration: 01m 00s)
• 17:59 XioNoX: offline cr1-codfw:fpc0 - T254110
• 17:47 XioNoX: turn online cr1-codfw:fpc0 - T254110
• 17:46 shdubsh: update mtail in ulsfo caching hosts. restarting atsmtail and varnishmtail
• 17:31 mutante: backup1001 - queued job 42 - gerrit backup after renaming of the file set and addition of LFS data (T254155, T254162) it is incremental, the full one already ran
• 16:49 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: EventLogging - fix searchsatisfaction schema URI - testwiki only - T249261 (duration: 00m 59s)
• 16:48 otto@deploy1001: sync-file aborted: EventLogging - fix searchsatisfaction schema URI - testwiki only - T249261 (duration: 00m 02s)
• 16:39 bstorm_: running view updates on db1141 T252219
• 14:53 elukey: ganeti: increase memory available for an-launcher1001 from 8g to 12g - T254125
• 14:44 volans: deploying ulsfo mgmt DNS records automatically generated by Netbox ( operations/dns/+/585545/ ) - T233183
• 12:00 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1142, db1147 T252512', diff saved to https://phabricator.wikimedia.org/P11345 and previous config saved to /var/cache/conftool/dbconfig/20200601-120000-marostegui.json
• 11:44 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1142, db1147 T252512', diff saved to https://phabricator.wikimedia.org/P11344 and previous config saved to /var/cache/conftool/dbconfig/20200601-114440-marostegui.json
• 11:30 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1142, db1147 T252512', diff saved to https://phabricator.wikimedia.org/P11343 and previous config saved to /var/cache/conftool/dbconfig/20200601-113032-marostegui.json
• 10:49 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (601328) (duration: 00m 59s)
• 10:48 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (601328) (duration: 01m 03s)
• 09:37 volans@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 09:30 volans@cumin1001: START - Cookbook sre.dns.netbox
• 09:26 jynus: reenabling puppet on all db/es/pc hosts after deploy of gerrit:599596
• 09:22 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1142, db1147 T252512', diff saved to https://phabricator.wikimedia.org/P11342 and previous config saved to /var/cache/conftool/dbconfig/20200601-092220-marostegui.json
• 09:18 marostegui@cumin1001: dbctl commit (dc=all): 'Add db1147 to dbctl, depooled T252512', diff saved to https://phabricator.wikimedia.org/P11341 and previous config saved to /var/cache/conftool/dbconfig/20200601-091809-marostegui.json
• 09:06 filippo@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
• 09:05 filippo@cumin1001: START - Cookbook sre.hosts.decommission
• 09:05 filippo@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
• 09:05 XioNoX: offline cr1-codfw:fpc0 - T254110
• 09:05 filippo@cumin1001: START - Cookbook sre.hosts.decommission
• 09:04 filippo@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
• 09:03 filippo@cumin1001: START - Cookbook sre.hosts.decommission
• 08:58 godog: prometheus eqiad lvextend --resizefs --size +100G vg-ssd/prometheus-ops
• 08:43 mutante: deneb - apt-get remove --purge apt-listchanges (packages was in status "rc" causing DPKG alert, should be removed but config was not purged)
• 08:41 mutante: deneb - apt-get remove python3-debconf (package was in status "ri" causing DPKG icinga alert. ri means it should be removed but is not)
• 08:33 XioNoX: restart cr1-codfw:fpc0 - T254110
• 08:22 mutante: mw1331 re-enabled puppet (SAL told me about an experiment a little while ago)
• 08:19 jynus: disabling puppet on all db/es/pc hosts for deploy of gerrit:599596
• 07:05 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1142 to clone db1147 T252512', diff saved to https://phabricator.wikimedia.org/P11339 and previous config saved to /var/cache/conftool/dbconfig/20200601-070519-marostegui.json
• 05:03 marostegui@cumin1001: dbctl commit (dc=all): 'Depool enwiki db2071 slave to test new index - T238966', diff saved to https://phabricator.wikimedia.org/P11338 and previous config saved to /var/cache/conftool/dbconfig/20200601-050354-marostegui.json
• 04:54 marostegui: Drop testreduce_0715 from m5 master T245408
• 04:44 marostegui: Depool db1141 from Analytics role - T249188

2020-05-31

• 09:56 Urbanecm: mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=commonswiki --logwiki=metawiki 'Vox Golf' 'Colonel Chicken' (T254068)

2020-05-29

• 22:32 bstorm_: updated views on labsdb1010 T252219
• 20:55 bstorm_: updating views on labsdb1011 T252219
• 19:27 ryankemper: Successfully finished a rolling restart of the `cloudelastic` clusters (chi, psi, omega) as part of elasticsearch plugins upgrade. Host and service checks re-enabled.
• 17:28 bstorm_: updating views on labsdb1009 T252219
• 16:50 ryankemper: Performing a rolling restart of the `cloudelastic` clusters (chi, psi, omega) as part of elasticsearch plugins upgrade. Host and service checks disabled.
• 16:00 bstorm_: Updating views on labsdb1012 T252219
• 15:59 ryankemper: Concluded rolling restart of the `relforge` clusters as part of elasticsearch plugins upgrade. Both hosts `relforge1001` and `relforge1002` are back up. Downtime lifted.
• 15:29 ryankemper: Performing a rolling restart of the `relforge` clusters as part of elasticsearch plugins upgrade
• 14:59 cdanis: disabling puppet on netflow* to deploy Ic71e96f0 T253128
• 14:47 akosiaris@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' .
• 14:47 akosiaris@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'coredns' .
• 14:41 akosiaris@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' .
• 14:41 akosiaris@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'kube-system' for release 'coredns' .
• 14:35 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' .
• 14:35 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'coredns' .
• 14:27 hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:24 hnowlan@cumin1001: START - Cookbook sre.hosts.downtime
• 14:15 mdholloway: ran extensions/MachineVision/maintenance/removeBlacklistedSuggestions.php on commonswiki (T253821)
• 12:49 hnowlan: reimaging restbase2009 after disk replacement
• 12:37 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 12:35 elukey@cumin1001: START - Cookbook sre.hosts.downtime
• 12:15 godog: roll-restart to upgrade thanos to 0.13.0rc0 - T252186 T233956
• 11:32 moritzm: installing cups security updates (client-side libs/tools)
• 11:01 ema: upload prometheus-rdkafka-exporter 0.2 to buster-wikimedia T253551
• 10:53 moritzm: updating mwdebug2002 to 7.2.31
• 10:02 marostegui: Compress InnoDB on db1138 T232446
• 08:30 godog: update swift uid/gid on thanos hosts - T123918
• 08:04 mutante: phabricator - restarted apache2 - back for me now
• 08:03 XioNoX: add new AMS-IX link to LACP bundle
• 08:01 mutante: phabricator - broken due to "PhabricatorRepositoryMirrorEngine::pushToGitRepository" starting git process that uses 100% CPU, stopped phd service
• 07:56 mutante: phabricator - killed pid 25070 (git) which used 100% of CPU, restarted phd service
• 07:25 moritzm: updating perf on buster systems to new version from 10.4 point release
• 07:15 moritzm: installing el-api update from latest Buster point release
• 07:12 moritzm: installing xdg-utils update from latest Buster point release
• 07:11 mutante: mw1293 (canary jobrunner ) replace apache2.conf with version from mwdebug1001, restart apache, to debug for T190111
• 07:00 moritzm: installing rake security updates
• 06:36 mutante: deneb - systemctl start docker-reporter-releng-images
• 05:20 marostegui: Deploy schema change on db1138 (no longer s4 master) - T250055
• 05:02 marostegui@cumin1001: dbctl commit (dc=all): 'Promote db1081 to s4 master and remove read-only from s4 T253808', diff saved to https://phabricator.wikimedia.org/P11334 and previous config saved to /var/cache/conftool/dbconfig/20200529-050224-marostegui.json
• 05:01 marostegui@cumin1001: dbctl commit (dc=all): 'Set s4 as read-only for maintenance T253808', diff saved to https://phabricator.wikimedia.org/P11333 and previous config saved to /var/cache/conftool/dbconfig/20200529-050153-marostegui.json
• 05:00 marostegui: Starting s4 failover from db1138 to db1081 -T253808
• 04:25 marostegui: Start topology changes in s4 - T253808

2020-05-28

• 23:48 jforrester@deploy1001: Synchronized php-1.35.0-wmf.34/skins/Vector/resources/skins.vector.styles/Menu.less: T253912 Hotfix: Cannot rename emptyPortlet to empty-portlet yet (duration: 00m 59s)
• 22:41 jforrester@deploy1001: Synchronized php-1.35.0-wmf.34/extensions/WikibaseMediaInfo/src/Services/FilePageLookup.php: T253792 Follow-up 1827c7a: Ensure inNamespace() is called only on Title object (duration: 00m 58s)
• 22:24 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T253821 Update MachineVision block list for 2020-05-27 (duration: 00m 57s)
• 22:09 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: Move one CheckUser right change next to the other (duration: 00m 57s)
• 22:06 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: Remove version wrapper around wgOverrideUcfirstCharacters; always true (duration: 00m 59s)
• 21:48 jforrester@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.34
• 21:26 jforrester@deploy1001: Synchronized php-1.35.0-wmf.34/includes/filerepo/FileRepo.php: T253922 Mark two FileRepo functions public (duration: 01m 07s)
• 21:12 jforrester@deploy1001: Synchronized php-1.35.0-wmf.34/includes/specials/SpecialUserrights.php: T253909 Restore visibility (previously implicitely public) (duration: 01m 06s)
• 20:38 jforrester@deploy1001: Synchronized php-1.35.0-wmf.32/skins/Vector/resources/skins.vector.styles: T253905 HOTFIX: Do not apply p-personal absolute positioning to all menus (duration: 01m 07s)
• 20:22 shdubsh: restart varnishmtail and atsmtail eqsin
• 20:11 shdubsh: restart ncredirmtail on ncredir5001
• 19:20 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: roll back the train due to T253905
• 19:20 twentyafterfour: group2 back to wmf.32 due to T253905
• 19:20 milimetric@deploy1001: Finished deploy [analytics/refinery@f6d73c8] (thin): Hotfix #2 today (thin): forgot jars [analytics/refinery@f6d73c8] (duration: 00m 09s)
• 19:20 milimetric@deploy1001: Started deploy [analytics/refinery@f6d73c8] (thin): Hotfix #2 today (thin): forgot jars [analytics/refinery@f6d73c8]
• 19:17 milimetric@deploy1001: Finished deploy [analytics/refinery@f6d73c8]: Hotfix #2 today: forgot jars [analytics/refinery@f6d73c8] (duration: 16m 54s)
• 19:14 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.34 refs T253022
• 19:01 shdubsh: restart varnishmtail and atsmtail on cp5001.eqsin.wmnet
• 19:00 milimetric@deploy1001: Started deploy [analytics/refinery@f6d73c8]: Hotfix #2 today: forgot jars [analytics/refinery@f6d73c8]
• 17:03 twentyafterfour@deploy1001: Synchronized php: group1 wikis to 1.35.0-wmf.34 refs T253022 (duration: 01m 06s)
• 17:02 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.34 refs T253022
• 16:32 jforrester@deploy1001: Synchronized php-1.35.0-wmf.34/extensions/Wikibase: T253804 Use ThrowingEntityTermStoreWriter when writers shouldn't be called (duration: 01m 15s)
• 15:37 milimetric@deploy1001: Finished deploy [analytics/refinery@203d182] (thin): Three hotfixes (THIN) [analytics/refinery@203d182] (duration: 00m 10s)
• 15:37 milimetric@deploy1001: Started deploy [analytics/refinery@203d182] (thin): Three hotfixes (THIN) [analytics/refinery@203d182]
• 15:05 milimetric@deploy1001: Finished deploy [analytics/refinery@203d182]: Three hotfixes [analytics/refinery@203d182] (duration: 25m 59s)
• 15:02 moritzm: installing exim4 security updates on jessie (stretch/buster already fixed)
• 14:39 milimetric@deploy1001: Started deploy [analytics/refinery@203d182]: Three hotfixes [analytics/refinery@203d182]
• 14:33 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:30 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 14:01 ema: atskafka 0.8 uploaded to buster-wikimedia T253551
• 13:49 godog: roll-restart prometheus k8s-staging to enable thanos upload - T252186
• 13:36 hashar: Restarting CI Jenkins for plugin rollback
• 11:49 moritzm: installing unbound security updates
• 11:03 kormat@cumin1001: dbctl commit (dc=all): 'Add db2138 to s2+s4 T252985', diff saved to https://phabricator.wikimedia.org/P11330 and previous config saved to /var/cache/conftool/dbconfig/20200528-110333-kormat.json
• 10:36 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'blubberoid' for release 'production' .
• 10:34 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'blubberoid' for release 'production' .
• 10:30 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'blubberoid' for release 'staging' .
• 10:02 mutante: gerrit1002 (test server) - chown -R gerrit2:gerrit2 /var/lib/gerrit/review_site ; restarted gerrit service, now the service is not in restart loop anymore, gerrit-ssh is listening too, just not accepting publickey (T239151)
• 09:51 XioNoX: failover VRRP in ulsfo
• 09:41 XioNoX: re-activate peering/transit on cr2-eqdfw - T243080
• 09:35 mutante: restarting gerrit on gerrit1002 after fixing db_pass to the readonly one (T243800)
• 09:33 XioNoX: restart cr2-eqdfw for upgrade - T243080
• 09:30 XioNoX: deactivate peering/transit on cr2-eqdfw - T243080
• 09:25 _joe_: updating ACLs on all etcd servers
• 09:22 XioNoX: install new Junos on cr2-eqdfw - T243080
• 09:16 XioNoX: rollback cr2-eqord ospf/bgp - T243080
• 09:07 XioNoX: restart cr2-eqord for upgrade - T243080
• 09:05 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'blubberoid' for release 'staging' .
• 08:50 _joe_: upgrading etcd ACLs (adding new users) to conf1004
• 08:50 XioNoX: install new Junos on cr2-eqord - T243080
• 08:46 XioNoX: deactivate peering/transit on cr2-eqord - T243080
• 08:45 XioNoX: de-pref all OSPF links to cr2-eqord - T243080
• 08:13 marostegui: Pool db1141 into labsdb analytics role - T249188
• 07:33 gilles@deploy1001: Synchronized static/images: T252108 Deploying optimised static PNGs (duration: 01m 39s)
• 07:31 gilles@deploy1001: Synchronized static/apple-touch: T252108 Deploying optimised static PNGs (duration: 01m 12s)
• 06:30 marostegui@cumin1001: dbctl commit (dc=all): 'Remove db1081 from API and set its weight to 0 on main traffic - preparation for tomorrow's failover T253808', diff saved to https://phabricator.wikimedia.org/P11329 and previous config saved to /var/cache/conftool/dbconfig/20200528-063037-marostegui.json
• 04:44 marostegui: Run check_private data on db1141 - T249188
• 04:22 marostegui: Stop MySQL on db1141 - T249188

2020-05-27

• 23:20 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add autoreviewrestore right to rollbacker group on hiwiki (T252986) (duration: 01m 05s)
• 23:16 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add thwiki Draft namespace to wmgExemptFromUserRobotsControlExtra and enable VE there (T252959) (duration: 01m 06s)
• 22:58 gehel@cumin1001: END (PASS) - Cookbook sre.postgresql.postgres-init (exit_code=0)
• 22:02 crusnov@deploy1001: Finished deploy [netbox/deploy@5251cf1]: Netbox Upgrade to 2.8.4 (part4) (duration: 00m 10s)
• 22:02 crusnov@deploy1001: Started deploy [netbox/deploy@5251cf1]: Netbox Upgrade to 2.8.4 (part4)
• 22:01 crusnov@deploy1001: Finished deploy [netbox/deploy@5251cf1]: Netbox Upgrade to 2.8.4 (part3) (duration: 01m 29s)
• 22:00 crusnov@deploy1001: Started deploy [netbox/deploy@5251cf1]: Netbox Upgrade to 2.8.4 (part3)
• 22:00 crusnov@deploy1001: deploy aborted: Netbox Upgrade to 2.8.4 (part2) (duration: 01m 31s)
• 21:58 crusnov@deploy1001: Started deploy [netbox/deploy@5251cf1]: Netbox Upgrade to 2.8.4 (part2)
• 21:58 crusnov@deploy1001: Finished deploy [netbox/deploy@5251cf1]: Netbox Upgrade to 2.8.1 (part1) (duration: 01m 01s)
• 21:57 crusnov@deploy1001: Started deploy [netbox/deploy@5251cf1]: Netbox Upgrade to 2.8.1 (part1)
• 20:43 gehel@cumin1001: START - Cookbook sre.postgresql.postgres-init
• 20:28 marostegui: Decrease innodb poolsize on s4 master and restart mysql
• 20:11 mbsantos@deploy1001: Finished deploy [mobileapps/deploy@9dc827f]: Update mobileapps to b3b9214c (T253648) (duration: 03m 31s)
• 20:08 mbsantos@deploy1001: Started deploy [mobileapps/deploy@9dc827f]: Update mobileapps to b3b9214c (T253648)
• 20:04 twentyafterfour@deploy1001: Synchronized php: group1 wikis to 1.35.0-wmf.32 refs T253022 (duration: 01m 04s)
• 20:03 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.32 refs T253022
• 20:00 gehel@cumin1001: END (PASS) - Cookbook sre.postgresql.postgres-init (exit_code=0)
• 19:56 twentyafterfour@deploy1001: scap failed: average error rate on 4/9 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/e474f13ffac6b8c3bf919c4aeafc8c9b for details)
• 19:46 jforrester@deploy1001: Synchronized php-1.35.0-wmf.34/includes/parser/CoreParserFunctions.php: T253725 Partially revert 'Fix impedance mismatch with Parser::getRevisionRecordObject()' (duration: 01m 05s)
• 19:12 joal@deploy1001: Finished deploy [analytics/refinery@8a3dcb3]: Analytics regular weekly train (an-launcher1001 only) [8a3dcb3] (duration: 06m 07s)
• 19:09 jforrester@deploy1001: Synchronized dblists/mobilemainpagelegacy.dblist: T32405 Stop special casing the main page on mobile for twelve wikis (duration: 01m 05s)
• 19:06 joal@deploy1001: Started deploy [analytics/refinery@8a3dcb3]: Analytics regular weekly train (an-launcher1001 only) [8a3dcb3]
• 19:03 joal@deploy1001: Finished deploy [analytics/refinery@8a3dcb3] (thin): Analytics regular weekly train THIN [8a3dcb3] (duration: 00m 08s)
• 19:03 joal@deploy1001: Started deploy [analytics/refinery@8a3dcb3] (thin): Analytics regular weekly train THIN [8a3dcb3]
• 19:03 joal@deploy1001: Finished deploy [analytics/refinery@8a3dcb3]: Analytics regular weekly train [8a3dcb3] (duration: 21m 20s)
• 18:41 joal@deploy1001: Started deploy [analytics/refinery@8a3dcb3]: Analytics regular weekly train [8a3dcb3]
• 18:06 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable DiscussionTools as beta on mediawiki.org, part II T251208 (duration: 01m 05s)
• 17:56 jayme: updated tiller to 2.16.7-wmf1 for all services in kubernetes cluster: eqiad
• 17:53 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable DiscussionTools as beta on mediawiki.org T251208 (duration: 01m 05s)
• 17:42 gehel@cumin1001: START - Cookbook sre.postgresql.postgres-init
• 17:40 gehel: repool maps2003
• 17:32 twentyafterfour@deploy1001: Synchronized php-1.35.0-wmf.34/extensions/Translate/: Deploy https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Translate/+/599027/ to wmf.34 refs T253748 and T253022 (duration: 01m 07s)
• 16:55 hnowlan@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' .
• 16:53 hnowlan@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' .
• 16:52 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' .
• 16:26 hnowlan@deploy1001: Finished deploy [cpjobqueue/deploy@c8c653e]: Disabling ThumbnailRender as a test of k8s cpjobqueue (duration: 01m 57s)
• 16:24 hnowlan@deploy1001: Started deploy [cpjobqueue/deploy@c8c653e]: Disabling ThumbnailRender as a test of k8s cpjobqueue
• 16:10 hnowlan@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' .
• 16:09 hnowlan@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' .
• 16:06 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' .
• 15:52 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Stop defining wmgUsePerformanceInspector, unread T253689 (duration: 01m 04s)
• 15:52 gehel@cumin1001: END (PASS) - Cookbook sre.postgresql.postgres-init (exit_code=0)
• 15:52 jayme: updated tiller to 2.16.7-wmf1 for all services in kubernetes cluster: codfw
• 15:49 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: Stop loading PerformanceInspector on any wiki T253689 (duration: 01m 06s)
• 15:18 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' .
• 15:02 godog: eqiad-prod: decom ms-be101[678] - T252008
• 14:58 jayme: updated tiller to 2.16.7-wmf1 for all services in cluster: staging
• 14:58 volans@cumin1001: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
• 14:56 otto@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-main' for release 'production' .
• 14:56 otto@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' .
• 14:54 volans@cumin1001: START - Cookbook sre.dns.netbox
• 14:52 otto@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' .
• 14:51 cdanis: cumin1001: upgrading python3-conftool and python3-conftool-dbctl
• 14:50 otto@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' .
• 14:50 otto@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' .
• 14:46 cdanis: cumin2001: upgrading python3-conftool and python3-conftool-dbctl
• 14:43 otto@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' .
• 14:43 otto@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' .
• 14:40 cdanis: reprepro: upload conftool_1.3.1-1{,+deb10u1} to {stretch,buster}-wikimedia
• 14:36 otto@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-main' for release 'production' .
• 14:36 otto@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' .
• 14:32 otto@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' .
• 14:32 otto@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' .
• 14:30 otto@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' .
• 14:16 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1146:3312, db1146:3314 and db1103:3312, db1103:3314', diff saved to https://phabricator.wikimedia.org/P11318 and previous config saved to /var/cache/conftool/dbconfig/20200527-141635-marostegui.json
• 14:13 otto@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' .
• 14:13 otto@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' .
• 14:07 otto@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-main' for release 'production' .
• 14:07 otto@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' .
• 14:04 otto@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' .
• 14:04 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1146:3312, db1146:3314 and db1103:3312, db1103:3314', diff saved to https://phabricator.wikimedia.org/P11317 and previous config saved to /var/cache/conftool/dbconfig/20200527-140442-marostegui.json
• 14:03 otto@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' .
• 14:03 otto@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' .
• 13:58 dzahn@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 13:51 otto@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' .
• 13:51 otto@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' .
• 13:48 dzahn@cumin1001: START - Cookbook sre.ganeti.makevm
• 13:47 dzahn@cumin1001: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99)
• 13:47 dzahn@cumin1001: START - Cookbook sre.ganeti.makevm
• 13:47 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1146:3312, db1146:3314 and db1103:3312, db1103:3314', diff saved to https://phabricator.wikimedia.org/P11316 and previous config saved to /var/cache/conftool/dbconfig/20200527-134704-marostegui.json
• 13:45 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1)
• 13:42 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
• 13:36 gehel@cumin1001: START - Cookbook sre.postgresql.postgres-init
• 13:34 gehel@cumin1001: END (FAIL) - Cookbook sre.postgresql.postgres-init (exit_code=99)
• 13:34 gehel@cumin1001: START - Cookbook sre.postgresql.postgres-init
• 13:21 gehel: repool maps2004 / depool maps2003
• 13:21 ema: cp: upgrade purged to 0.14
• 13:19 marostegui: Kill /usr/local/bin/mwscriptwikiset updateSpecialPages.php s8.dblist --override --only=Fewestrevisions T238199
• 13:16 dzahn@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 13:15 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1146:3312, db1146:3314 and db1103:3312, db1103:3314', diff saved to https://phabricator.wikimedia.org/P11313 and previous config saved to /var/cache/conftool/dbconfig/20200527-131515-marostegui.json
• 13:08 marostegui@cumin1001: dbctl commit (dc=all): 'Add db1146:3312 and db1146:3314 to dbctl T252512', diff saved to https://phabricator.wikimedia.org/P11312 and previous config saved to /var/cache/conftool/dbconfig/20200527-130820-marostegui.json
• 13:06 dzahn@cumin1001: START - Cookbook sre.ganeti.makevm
• 11:34 Urbanecm: EU SWAT done
• 11:29 urbanecm@deploy1001: Synchronized php-1.35.0-wmf.34/extensions/GrowthExperiments/: SWAT: 983eda5: Mentorship dialog: Swap panel to ask-help on open (T253692) (duration: 01m 06s)
• 11:18 ema: cp2027: upgrade purged to 0.14
• 11:17 ema: purged 0.14 uploaded to buster-wikimedia
• 11:09 kartik@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: 598678|Enable ContentTranslation in Galician Wikipedia as a default tool (T250355) (duration: 01m 18s)
• 10:15 hashar: contint2001: starting zuul
• 10:15 hashar: contint2001: started jenkins
• 10:03 mutante: contint2001 - find /var/lib/jenkins -user statsite -exec chown -h jenkins:jenkins {} \;
• 10:02 mutante: repeated rsync of /var/lib/jenkins with -p ; find /var/lib/jenkins -group bacula -user statsite -exec chown -h jenkins:jenkins {} \;
• 09:55 hashar: contint2001: starting jenkins
• 09:54 hashar: contint1001 / contint2001 : deleted obsolete files /var/lib/jenkins/.git and /var/lib/jenkins/jobs/_shared/
• 09:52 mutante: contint2001 - find /var/lib/jenkins -user statsite -exec chown -h jenkins:jenkins {} \;
• 09:51 godog: roll restart prometheus on the fleet to apply I0e2fe8af
• 09:49 mutante: contint2001 - find /var/lib/jenkins -group bacula -user statsite -exec chown jenkins:jenkins {} \;
• 09:48 hashar: contint2001: unmasked jenkins and started it
• 09:42 filippo@cumin1001: conftool action : set/pooled=yes; selector: name=prometheus2003.codfw.wmnet
• 09:42 mutante: switching CI backend from contint1001 to contint2001
• 09:40 mutante: repeated rsync -avp --delete /var/lib/zuul/ rsync://contint2001.wikimedia.org/ci--var-lib-zuul-
• 09:40 hashar: contint1001: masked jenkins and zuul
• 09:39 mutante: repeated rsync -avp --delete /var/lib/jenkins/ rsync://contint2001.wikimedia.org/ci--var-lib-jenkins-
• 09:39 hashar: Stopping Zuul and Jenkins CI for scheduled maintenance # T224591
• 09:35 filippo@cumin1001: conftool action : set/pooled=no; selector: name=prometheus2003.codfw.wmnet
• 08:52 hashar: contint1001: find /srv/jenkins/builds/operations-puppet-wmf-style-guide -type f -name '*.tmp' -delete # T253729
• 08:48 marostegui: Stop MySQL on db1103
• 08:47 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1103:3312 db1103:3314 to clone db1146 T252512', diff saved to https://phabricator.wikimedia.org/P11308 and previous config saved to /var/cache/conftool/dbconfig/20200527-084713-marostegui.json
• 08:46 arturo: removing more old packages in labstore1006 (all packages in 'rc' state)
• 08:43 arturo: running apt-get autoremove on labstore1006
• 08:42 jynus: starting again db2097 db instances T252492
• 08:11 jayme: updated admin tiller (namespace: kube-system) to 2.16.7-wmf1 in clusters: staging, codfw, eqiad
• 08:08 hashar: contint1001 / contint2001 : deleted unused /var/lib/zuul/git (the real one is /srv/zuul/git )
• 08:02 mutante: contint2001 - chown root:root /var/lib/zuul/git
• 07:54 XioNoX: test new bird conf on dns4001 - T253666
• 07:45 hashar: contint2001 also fixing symlink permissions: sudo find /var/lib/jenkins -not -user jenkins -exec chown -h jenkins:jenkins {} +
• 07:35 mutante: contint2001 - find /var/lib/jenkins -group bacula -user jenkins -exec chown jenkins:jenkins {} \;
• 07:30 mutante: contint2001 - find /var/lib/jenkins -user statsite -exec chown jenkins {} \;
• 07:26 mutante: contint2001 - chown -R zuul:zuul /var/lib/zuul/
• 07:26 mutante: contint1001:~# rsync -avpz --delete /srv/jenkins/ rsync://contint2001.wikimedia.org/ci--srv-/jenkins/
• 07:25 mutante: contint1001:~# rsync -avp --delete /var/lib/jenkins/ rsync://contint2001.wikimedia.org/ci--var-lib-jenkins-
• 07:25 mutante: contint1001:~# rsync -avp --delete /var/lib/zuul/ rsync://contint2001.wikimedia.org/ci--var-lib-zuul-
• 07:18 moritzm: installing bind security updates (only client-side tools/libraries in use)
• 07:04 elukey: matomo upgraded to 3.13.5 on matomo1001 - T252741
• 06:57 elukey: update matomo on stretch-wikimedia to 3.13.5
• 06:10 elukey@deploy1001: Finished deploy [analytics/superset/deploy@369a2dd]: Upgrade Superset to 0.36 - second attempt (duration: 00m 57s)
• 06:09 elukey@deploy1001: Started deploy [analytics/superset/deploy@369a2dd]: Upgrade Superset to 0.36 - second attempt
• 05:17 marostegui: Remove tmp_3 key from enwiki.recentchanges on db1099:3311 - T206103
• 04:41 _joe_: cassandra cannot start on restbase2009, one of the disk is failed.
• 04:39 _joe_: restarting cassandra instances on restbase2009, has a broken disk
• 04:20 marostegui: Depool labsdb1011 - T249188

2020-05-26

• 21:34 krinkle@deploy1001: Synchronized wmf-config/mc.php: I0fb124b3593 (duration: 01m 05s)
• 21:30 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: I2714e2ae26404 (duration: 01m 06s)
• 21:18 krinkle@deploy1001: Synchronized wmf-config/profiler.php: Ib0bf8d97b10b, T253674 (duration: 01m 06s)
• 20:29 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group0 wikis to 1.35.0-wmf.34 refs T253022
• 20:08 twentyafterfour@deploy1001: Finished scap: testwikis wikis to 1.35.0-wmf.34 refs T253022 (duration: 70m 02s)
• 18:58 twentyafterfour@deploy1001: Started scap: testwikis wikis to 1.35.0-wmf.34 refs T253022
• 18:07 jforrester@deploy1001: Pruned MediaWiki: 1.35.0-wmf.30 (duration: 20m 45s)
• 18:02 bblack: cr[12]-eqiad: re-route ns0.wikimedia.org to authdns1001 - T241770
• 18:02 ejegg: restarted fundraising jobs: recurring charge, audit processing, deduplication
• 17:57 moritzm: installing bind security updates for stretch (only client-side tools/libraries in use)
• 17:47 cdanis: netflow3001: disabling puppet and testing some pmacct/librdkafka config tweaks T253128
• 17:16 James_F: 1.35.0-wmf.34 was branched at b5012a1 for T253022
• 16:45 moritzm: installing jsp-api bugfix update from Buster point release
• 15:22 akosiaris: sync kubernetes eqiad namespaces configuration with helmfile
• 15:15 akosiaris: sync kubernetes codfw namespaces configuration with helmfile
• 15:08 arturo: delete/re-import docker/containerd.io packages in the right version in buster-wikimedia/thirdparty/kubeadm-k8s-1-{15,16} (T250866)
• 15:08 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: Add lazy-loading to Wikimedia Foundation powered-by icon T239377 (duration: 00m 57s)
• 15:01 jforrester@deploy1001: Synchronized dblists/mobilemainpagelegacy.dblist: Drop enwiki mobile mainpage special casing T32405 (duration: 00m 59s)
• 14:58 mholloway-shell@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'wikifeeds' for release 'production' .
• 14:57 akosiaris: sync staging namespaces configuration
• 14:57 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' .
• 14:57 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'coredns' .
• 14:57 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'rbac-deploy-clusterrole' .
• 14:56 jforrester@deploy1001: Synchronized docroot/noc/: Clear out symlink to mobile.php, now removed (duration: 00m 55s)
• 14:56 mholloway-shell@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'wikifeeds' for release 'production' .
• 14:54 mholloway-shell@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' .
• 14:53 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: Move mobile.php into CommonSettings.php (duration: 00m 57s)
• 14:44 arturo: upgrade packages in buster-wikimedia/thirdpardy/kubeadm-k8s-1-16 (T246122)
• 14:44 jforrester@deploy1001: Synchronized docroot/noc/: Clear out symlink to mobile-labs.php, now removed (duration: 00m 58s)
• 14:43 moritzm: installing rails security updates
• 14:41 jforrester@deploy1001: Synchronized wmf-config/mobile.php: Don't try to load mobile-labs.php from mobile.php (duration: 00m 57s)
• 14:38 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: CommonSettings.php: Move uncondition/no-sideeffect includes up (duration: 00m 57s)
• 14:35 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: Clean up MWMultiVersion check in CommonSettings.php (duration: 00m 59s)
• 14:33 XioNoX: test bgp med on dns4002
• 14:31 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SpecialVersionVersionUrl: Don't use confusing local variable name (duration: 00m 58s)
• 14:30 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: ExtensionDistributor: Remove EOL REL1_32 (duration: 00m 58s)
• 13:54 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'zotero' for release 'staging' .
• 13:04 hashar@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.32
• 12:43 godog: swift eqiad-prod: decom ms-be101[678] - T252008
• 12:21 XioNoX: repool ulsfo - T243080
• 12:11 XioNoX: cr4-ulsfo re-activate transit/ix/4/6 - T243080
• 12:03 XioNoX: cr4-ulsfo> request vmhost reboot - T243080
• 12:01 XioNoX: cr4-ulsfo deactivate transit/ix/4/6 - T243080
• 11:49 XioNoX: cr3-ulsfo> request vmhost reboot - T243080
• 11:42 XioNoX: cr4-ulsfo> request vmhost software add ... - T243080
• 11:28 XioNoX: cr3-ulsfo> request vmhost software add ... - T243080
• 11:27 awight: nnwiki updateCollation.php script has finished.
• 11:26 XioNoX: depool ulsfo for routers upgrade - T243080
• 11:16 awight: EU SWAT done (pending a maintenance script to updateCollation)
• 11:14 awight@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add 'deletedtext' permission to researcher group (T253420) (duration: 01m 06s)
• 11:06 awight@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [nnwiki] Change category collation to (T253559) (duration: 01m 10s)
• 10:46 marostegui: Stop tendril's event scheduler
• 10:18 jynus: stop db2097 for hw maintenance T252492
• 09:48 vgutierrez: rolling upgrade to ats 8.0.7-1wm11
• 09:41 _joe_: all jobrunners converted to use envoy for TLS termination
• 09:38 oblivian@cumin1001: conftool action : set/weight=10; selector: name=mw131[0-1].eqiad.wmnet
• 09:38 oblivian@cumin1001: conftool action : set/weight=10; selector: name=mw133[4-8].eqiad.wmnet
• 09:37 oblivian@cumin1001: conftool action : set/weight=10; selector: name=mw130[0-9].eqiad.wmnet
• 09:37 oblivian@cumin1001: conftool action : set/weight=10; selector: name=mw130[0-3].eqiad.wmnet
• 09:36 oblivian@cumin1001: conftool action : set/weight=10:pooled=yes; selector: name=mw129[3-9].eqiad.wmnet
• 09:31 oblivian@cumin1001: conftool action : set/weight=1:pooled=yes; selector: name=mw130[0-3].eqiad.wmnet
• 09:27 oblivian@cumin1001: conftool action : set/weight=1:pooled=yes; selector: name=mw130[4-7].eqiad.wmnet
• 09:22 gehel: repool wdqs1007, catched up on lag
• 09:09 oblivian@cumin1001: conftool action : set/weight=1:pooled=yes; selector: name=mw13(0[89]|1[01]).eqiad.wmnet
• 09:02 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
• 09:02 mutante: decom'ing people1001 - replaced by people1002
• 09:01 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
• 09:01 oblivian@cumin1001: conftool action : set/weight=1:pooled=yes; selector: name=mw13(1|3)8.eqiad.wmnet
• 08:57 oblivian@cumin1001: conftool action : set/weight=1:pooled=yes; selector: name=mw133[4-7].eqiad.wmnet
• 08:55 _joe_: progressively converting jobrunners to envoy
• 08:41 oblivian@cumin1001: conftool action : set/weight=1:pooled=yes; selector: name=mw1337.eqiad.wmnet
• 07:20 moritzm: installing libssh security updates
• 07:03 vgutierrez: upgrade to ats 8.0.7-1wm11 on cp3064 and cp3065
• 06:49 marostegui: Deploy schema change on s3 directly on the master with 1 minute sleep in between wikis T253342
• 06:47 marostegui: Deploy schema change on s1 directly on the master T253342
• 06:44 marostegui: Deploy schema change on s4 directly on the master T253342
• 06:35 XioNoX: reboot scs-ulsfo - T253609
• 06:29 marostegui: Deploy schema change on s7 directly on the master T253342
• 06:24 marostegui: Deploy schema change on s8 directly on the master T253342
• 06:01 marostegui: Deploy schema change on s2 directly on the master T253342
• 04:35 marostegui: Repool labsdb1011 - T249188
• 04:14 marostegui: Stop slaves and stop mysql on labsdb1011 T249188
• 03:55 tstarling@deploy1001: Synchronized php-1.35.0-wmf.31/includes/export/XmlDumpWriter.php: T253468 (duration: 01m 06s)
• 03:53 tstarling@deploy1001: Synchronized php-1.35.0-wmf.32/includes/export/XmlDumpWriter.php: T253468 (duration: 01m 07s)
• 03:20 tstarling@deploy1001: Synchronized php-1.35.0-wmf.32/includes/specials/SpecialChangeContentModel.php: for UBN T252963 (duration: 01m 07s)
• 03:18 tstarling@deploy1001: sync-file aborted: (no justification provided) (duration: 00m 32s)

2020-05-25

• 23:34 ejegg: re-enabled fundraising queue consumers and job runners, except audits, dedupe, and recurring
• 21:38 eileen: civicrm revision changed from 5428c5c449 to d1cd99166f, config revision is 6b05d6bb25
• 21:18 eileen: civicrm revision is 7380e0e8ce, config revision is 6b05d6bb25
• 21:01 ejegg: updated fundraising CiviCRM from 737d88a5ee to 7380e0e8ce
• 17:17 ejegg: updated fundraising CiviCRM from 6b1d5902dd to 737d88a5ee
• 17:09 ejegg: enabled contribution tracking queue on payments-wiki
• 16:24 ejegg: updated standalone SmashPig from 2702b04329 to 44690f761c
• 16:17 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'mathoid' for release 'staging' .
• 16:16 XioNoX: enable IX4/6 BGP group on cr4-ulsfo - T237575
• 16:00 hnowlan@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' .
• 15:55 XioNoX: disable IX4/6 BGP group on cr4-ulsfo - T237575
• 15:17 hnowlan@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' .
• 15:15 ejegg: updated payments-wiki from 3c465cb11c to d11efeb1cf, put it into maintenance mode
• 15:15 hnowlan@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' .
• 14:53 hnowlan@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' .
• 14:39 hnowlan@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' .
• 14:06 hnowlan@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' .
• 14:00 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'zotero' for release 'production' .
• 13:46 _joe_: uploaded doxygen 1.8.17-1 to wikimedia-buster component/ci
• 13:43 filippo@cumin1001: conftool action : set/pooled=true; selector: dnsdisc=thanos-swift
• 13:40 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'zotero' for release 'production' .
• 13:18 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'zotero' for release 'staging' .
• 13:10 hnowlan@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' .
• 13:09 vgutierrez: upgrade ATS to version 8.0.7-1wm11 on cp4026 and cp4032
• 12:52 godog: roll-restart pybal in low-traffic codfw
• 12:44 ema: upload atskafka 0.7 to buster-wikimedia, upgrade cp3050 T253551
• 12:37 hnowlan@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' .
• 12:30 marostegui: Deploy schema change on s5 directly on the master T253342
• 12:14 hnowlan@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' .
• 12:09 hnowlan@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' .
• 12:01 _joe_: converting the remaining appservers to use envoy for TLS termination
• 11:57 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' .
• 11:54 marostegui: Install a new tendril_purge_global_status_log event on db1115 (tendril) T252331
• 11:52 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' .
• 11:51 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' .
• 11:48 marostegui: Stop event scheduler on db1115 (tendril) - T252331
• 11:46 moritzm: uploaded CAS 6.1.5-1 to apt.wikimedia.org T233947
• 11:36 _joe_: switch mw[1349-1355,1364-1373].eqiad.wmnet to envoy
• 11:27 marostegui: Extend /srv 1100G on db213[6-9] T252985
• 11:23 marostegui: Extend /srv 1100G on db114[1-9] T252512
• 11:21 marostegui: Extend db1141's (temporary labsdb test host) /srv 1TB extra - T249188
• 11:09 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' .
• 11:09 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' .
• 11:01 ema: upload prometheus-rdkafka-exporter to buster-wikimedia T253197
• 10:34 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (598439) (duration: 01m 05s)
• 10:33 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (598439) (duration: 01m 06s)
• 10:20 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'zotero' for release 'staging' .
• 09:56 _joe_: transition done
• 09:49 _joe_: depooled mw1337, it was getting all traffic supposed to go to the jobrunners
• 09:45 vgutierrez: upload trafficserver 8.0.7-1wm10 to apt.wm.o (buster)
• 09:42 _joe_: converting mw1319-1333 to use envoy for TLS termination
• 09:17 _joe_: migrated mw1337 to use envoy for TLS termination T247389
• 09:10 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'zotero' for release 'staging' .
• 09:04 godog: turn on sni by default for check_http --ssl icinga invocations - T253292
• 08:52 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'termbox' for release 'production' .
• 08:39 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'termbox' for release 'production' .
• 08:21 filippo@cumin1001: conftool action : set/pooled=yes:weight=100; selector: service=thanos-swift
• 08:05 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'termbox' for release 'staging' .
• 07:36 moritzm: installed linux-image-amd64 on labstore1005 (current meta package for kernels following the Stretch update) T224582
• 07:36 moritzm: installed linux-imageamd64 on labstore (current meta package for kernels following the Stretch update) T224582
• 07:02 marostegui: Stop event scheduler on tendril T252331
• 05:11 marostegui: Deploy schema change on s6, directly on the master - T253342
• 04:54 marostegui: Depool labsdb1011 - T249188
• 04:11 kart_: Updated cxserver to 2020-05-22-083137-production (T246317, T252871)
• 04:07 kartik@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'cxserver' for release 'production' .
• 04:04 kartik@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'cxserver' for release 'production' .
• 04:02 kartik@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'cxserver' for release 'staging' .

2020-05-24

• 17:36 gehel: restarting elasticsearch psi on elastic1052
• 16:44 gehel: depool wdqs1007 to catch on lag
• 16:43 gehel: restart blazegraph on wdqs1007

2020-05-23

• 19:04 krinkle@deploy1001: Synchronized php-1.35.0-wmf.31/includes/filerepo/file/LocalFile.php: I0f7e885997d60 (duration: 01m 06s)
• 18:58 krinkle@deploy1001: Synchronized php-1.35.0-wmf.32/includes/filerepo/file/LocalFile.php: I0f7e885997d60 (duration: 01m 08s)
• 18:06 krinkle@deploy1001: Synchronized php-1.35.0-wmf.32/includes/filerepo/: I31a9bb6672 (duration: 01m 06s)
• 18:05 krinkle@deploy1001: Synchronized php-1.35.0-wmf.31/includes/filerepo/: I31a9bb6672 (duration: 01m 10s)
• 15:44 krinkle@deploy1001: Synchronized wmf-config/mc.php: I5ad8fe - Disable coalesceKeys on commonswiki (duration: 01m 09s)
• 14:58 Krinkle: scap-pull to reset state on mwdebug1002
• 14:50 Krinkle: Testing mc.php changes on mwdebug1002
• 08:04 elukey: powercycle an-presto1004 - unresponsive, racadm getsel shows CPU overheating alerts

2020-05-22

• 22:42 krinkle@deploy1001: Synchronized php-1.35.0-wmf.31/includes/filerepo/: Ie19613ef7643a (duration: 01m 06s)
• 22:40 krinkle@deploy1001: Synchronized php-1.35.0-wmf.32/includes/filerepo/: Ie19613ef7643a (duration: 01m 08s)
• 15:58 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' .
• 15:58 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' .
• 15:57 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' .
• 15:53 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' .
• 15:47 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' .
• 15:45 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' .
• 15:45 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' .
• 15:30 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' .
• 15:25 cdanis: fixing prometheus-nic-firmware-textfile.service wherever it is broken T253374
• 15:25 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' .
• 15:24 hnowlan@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' .
• 15:06 marostegui: Decrease tendril_purge_global_status_log_5m storing rows time from 2 days to 1 day T252331
• 15:01 kormat@cumin1001: dbctl commit (dc=all): 'Pool db2137 into s4+s5 T252985', diff saved to https://phabricator.wikimedia.org/P11292 and previous config saved to /var/cache/conftool/dbconfig/20200522-150120-kormat.json
• 14:53 reedy@deploy1001: Synchronized php-1.35.0-wmf.31/maintenance/blockUsers.php: (no justification provided) (duration: 01m 08s)
• 14:51 reedy@deploy1001: Synchronized php-1.35.0-wmf.32/maintenance/blockUsers.php: (no justification provided) (duration: 01m 09s)
• 14:35 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1144:331[45] and db1097:331[45]', diff saved to https://phabricator.wikimedia.org/P11290 and previous config saved to /var/cache/conftool/dbconfig/20200522-143541-marostegui.json
• 14:24 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
• 14:22 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
• 14:15 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1144:331[45] and db1097:331[45]', diff saved to https://phabricator.wikimedia.org/P11289 and previous config saved to /var/cache/conftool/dbconfig/20200522-141513-marostegui.json
• 14:13 sukhe: upload dnsdist_1.4.0-1~deb10u1 to apt.wm.o (buster) - T252132
• 14:08 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1144:331[45] and db1097:331[45]', diff saved to https://phabricator.wikimedia.org/P11288 and previous config saved to /var/cache/conftool/dbconfig/20200522-140847-marostegui.json
• 13:14 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1144:331[45] and db1097:331[45]', diff saved to https://phabricator.wikimedia.org/P11286 and previous config saved to /var/cache/conftool/dbconfig/20200522-131452-marostegui.json
• 13:10 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'termbox' for release 'production' .
• 13:10 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'termbox' for release 'production' .
• 13:09 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'termbox' for release 'staging' .
• 13:08 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'termbox' for release 'test' .
• 13:07 marostegui@cumin1001: dbctl commit (dc=all): 'Add db1144:3314 and db1144:3315 to the list of hosts', diff saved to https://phabricator.wikimedia.org/P11284 and previous config saved to /var/cache/conftool/dbconfig/20200522-130707-marostegui.json
• 12:56 vgutierrez: depool cp4032 for some ats tests
• 12:13 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
• 12:12 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
• 12:06 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
• 12:04 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
• 12:03 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1)
• 12:03 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
• 10:48 marostegui: Stop MySQL on db1097:3314, db1097:3315 to clone db1144 - T252512
• 10:44 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1097:3314, db1097:3315 - T252512', diff saved to https://phabricator.wikimedia.org/P11281 and previous config saved to /var/cache/conftool/dbconfig/20200522-104437-marostegui.json
• 10:37 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
• 10:35 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
• 10:34 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
• 10:32 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
• 10:10 marostegui: Stop event_scheduler on db1115 - T252331
• 10:09 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
• 10:05 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
• 10:05 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1)
• 10:05 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
• 10:00 jbond42: update pdns-recursor on dns recursors
• 09:43 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
• 09:41 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
• 09:22 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'mathoid' for release 'production' .
• 09:09 elukey@deploy1001: Finished deploy [analytics/superset/deploy@be203c8]: Rollback superset to 0.35.2 (duration: 00m 43s)
• 09:09 elukey@deploy1001: Started deploy [analytics/superset/deploy@be203c8]: Rollback superset to 0.35.2
• 08:41 vgutierrez: reverting hugepages experiment on cp2041
• 08:27 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1149 and db1081', diff saved to https://phabricator.wikimedia.org/P11278 and previous config saved to /var/cache/conftool/dbconfig/20200522-082700-marostegui.json
• 08:18 elukey@deploy1001: Finished deploy [analytics/superset/deploy@59ba01d]: Upgrade Superset to 0.36 (duration: 01m 01s)
• 08:17 elukey@deploy1001: Started deploy [analytics/superset/deploy@59ba01d]: Upgrade Superset to 0.36
• 08:13 vgutierrez: test hugepages allocator on ATS in cp2041
• 08:06 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1149 and db1081', diff saved to https://phabricator.wikimedia.org/P11277 and previous config saved to /var/cache/conftool/dbconfig/20200522-080629-marostegui.json
• 07:48 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1149 and db1081', diff saved to https://phabricator.wikimedia.org/P11276 and previous config saved to /var/cache/conftool/dbconfig/20200522-074853-marostegui.json
• 07:20 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1149 and db1081', diff saved to https://phabricator.wikimedia.org/P11275 and previous config saved to /var/cache/conftool/dbconfig/20200522-072000-marostegui.json
• 07:07 elukey@puppetmaster1001: conftool action : set/pooled=yes:weight=10; selector: name=druid1008.eqiad.wmnet
• 07:04 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=druid1007.eqiad.wmnet
• 07:04 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=druid1007.eqiad.wmnet
• 04:34 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1081 - T252512', diff saved to https://phabricator.wikimedia.org/P11272 and previous config saved to /var/cache/conftool/dbconfig/20200522-043418-marostegui.json

2020-05-21

• 23:58 ejegg: updated civicrm from b658fd8233 to 6b1d5902dd
• 23:54 krinkle@deploy1001: Synchronized php-1.35.0-wmf.32/includes/content/ContentHandlerFactory.php: If578893f5689 (duration: 01m 06s)
• 23:47 krinkle@deploy1001: Synchronized php-1.35.0-wmf.32/extensions/LiquidThreads/classes/Thread.php: If3418cba06e (duration: 01m 07s)
• 23:41 krinkle@deploy1001: Synchronized wmf-config/mc.php: I222457729a5b (duration: 01m 08s)
• 21:46 eileen: civicrm revision changed from ed4c9522ac to b658fd8233, config revision is 9babae3954
• 21:10 foks: removing two files for legal compliance
• 20:44 bstorm_: labstore1005 is now running stretch and drbd devices are resyncing after several reboots and some significant effort T224582
• 18:24 twentyafterfour: restarting phabricator on phab1001 to deploy https://phabricator.wikimedia.org/rPHEX2687d08786a9dadcbaa96709de991f471f239830
• 17:24 bblack: anycast experiment done, all back to normal
• 17:20 bblack: anycast experimentation commencing in ulsfo (test route withdrawal)...
• 17:04 bstorm_: starting labstore1005 upgrades T224582
• 16:14 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 16:12 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 16:04 sbassett@deploy1001: Synchronized private/PrivateSettings.php: Update mitigations for T250887 (duration: 01m 08s)
• 15:48 andrewbogott: rebuilding cloudnet1003.eqiad.wmnet with Debian Buster for T253124
• 15:22 XioNoX: Add BGP between cr1/2-eqiad and authdns1001 - T253196
• 15:09 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:09 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 15:08 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 15:08 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 15:07 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 15:07 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 14:59 dzahn@cumin1001: conftool action : set/pooled=inactive; selector: name=mw217[0-2].codfw.wmnet
• 14:59 dzahn@cumin1001: conftool action : set/pooled=inactive; selector: name=mw216[0-9].codfw.wmnet
• 14:58 dzahn@cumin1001: conftool action : set/pooled=inactive; selector: name=mw215[8-9].codfw.wmnet
• 14:50 bblack@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:47 bblack@cumin1001: START - Cookbook sre.hosts.downtime
• 14:44 akosiaris@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'mathoid' for release 'canary' .
• 14:33 akosiaris: upload helmfile 0.109.0 to apt.wikimedia.org/buster-wikimedia and stretch-wikimedia, component main
• 13:51 vgutierrez: depool cp4032 for some ats tests
• 13:22 mutante: cloudnet1004 - reboot to test PXE boot
• 12:44 andrewbogott: reimaging cloudnet1004.eqiad.wmnet for T253124
• 12:29 elukey: roll restart druid-public cluster (druid100[4-6], backend for the AQS API) to apply new settings + openjdk upgrade - T252771
• 12:13 mutante: depooled mw2158 through mw2172 to make room again in C3 as planned (T247018)
• 12:12 marostegui: Repool labsdb1011 into the analytics role 🤞- T249188
• 12:12 dzahn@cumin1001: conftool action : set/pooled=no; selector: name=mw217[0-2].codfw.wmnet
• 12:10 dzahn@cumin1001: conftool action : set/pooled=no; selector: name=mw216[0-9].codfw.wmnet
• 12:05 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1143 and db1091', diff saved to https://phabricator.wikimedia.org/P11270 and previous config saved to /var/cache/conftool/dbconfig/20200521-120555-marostegui.json
• 12:05 dzahn@cumin1001: conftool action : set/pooled=no; selector: name=mw215[8-9].codfw.wmnet
• 11:18 hnowlan: Removed changeprop from scb hosts
• 11:04 vgutierrez: rolling restart of ncredir servers for kernel update
• 10:17 vgutierrez: restart of acme-chief servers for kernel update
• 10:13 jbond42: deploy CI for pupet privcate repo
• 10:11 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1143 and db1091', diff saved to https://phabricator.wikimedia.org/P11268 and previous config saved to /var/cache/conftool/dbconfig/20200521-101100-marostegui.json
• 10:07 mutante: replaced backend of people.wikimedia.org - people1001 will be inaccessible, replaced with people1002 on buster. all home dirs have been synced over, there should be no difference except you have to use people1002 now for uploads (T247649)
• 10:06 godog: test adding --sni to check_http -S on icinga2001 - T253292
• 09:51 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1143 and db1091', diff saved to https://phabricator.wikimedia.org/P11267 and previous config saved to /var/cache/conftool/dbconfig/20200521-095100-marostegui.json
• 09:28 mutante: deneb - sudo systemctl reset-failed to clear Icinga alerts about systemd degraded state
• 09:12 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1143 and db1091', diff saved to https://phabricator.wikimedia.org/P11266 and previous config saved to /var/cache/conftool/dbconfig/20200521-091245-marostegui.json
• 09:01 mutante: LDAP - added lmata to wmf group (T253277)
• 08:55 XioNoX: Advertise Anycast 198.35.27.0/24 from esams - T253196
• 08:52 XioNoX: Advertise Anycast 198.35.27.0/24 from eqsin - T253196
• 08:49 marostegui@cumin1001: dbctl commit (dc=all): 'Pool db1143 with minimal weight for the first time T252512', diff saved to https://phabricator.wikimedia.org/P11265 and previous config saved to /var/cache/conftool/dbconfig/20200521-084933-marostegui.json
• 08:47 XioNoX: Advertise Anycast 198.35.27.0/24 from eqiad/eqord - T253196
• 08:42 marostegui@cumin1001: dbctl commit (dc=all): 'Add db1143 to the list of s4 hosts, depooled - T252512', diff saved to https://phabricator.wikimedia.org/P11264 and previous config saved to /var/cache/conftool/dbconfig/20200521-084226-marostegui.json
• 08:34 XioNoX: Advertise Anycast 198.35.27.0/24 from dfw - T253196
• 08:27 XioNoX: Advertise Anycast 198.35.27.0/24 from ulsfo - T253196
• 08:20 XioNoX: Delete ARIN route object for 198.35.26.0/23 - T253196
• 08:13 XioNoX: Delete ROA for 198.35.26.0/23 - T253196
• 08:10 XioNoX: repool ulsfo - T253196
• 08:03 XioNoX: Shrink ulsfo's 198.35.26.0/23 to 198.35.26.0/24 - T253196
• 07:29 XioNoX: depool ulsfo - T253196
• 07:22 marostegui: Purge events from tendril.global_status_log older than 24h - T252331
• 07:03 jynus@cumin1001: dbctl commit (dc=all): 'Repool es1019 fully', diff saved to https://phabricator.wikimedia.org/P11263 and previous config saved to /var/cache/conftool/dbconfig/20200521-070335-jynus.json
• 06:59 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1091 - T252512', diff saved to https://phabricator.wikimedia.org/P11261 and previous config saved to /var/cache/conftool/dbconfig/20200521-065858-marostegui.json
• 06:28 jynus@cumin1001: dbctl commit (dc=all): 'Repool es1019 with 50% weight', diff saved to https://phabricator.wikimedia.org/P11260 and previous config saved to /var/cache/conftool/dbconfig/20200521-062823-jynus.json
• 06:04 vgutierrez: pool cp5012 - T251219
• 05:42 jynus@cumin1001: dbctl commit (dc=all): 'Repool es1019 with low weight', diff saved to https://phabricator.wikimedia.org/P11259 and previous config saved to /var/cache/conftool/dbconfig/20200521-054231-jynus.json
• 05:03 marostegui@cumin1001: dbctl commit (dc=all): 'Set enwiki as read-only=off after maintenance T251982', diff saved to https://phabricator.wikimedia.org/P11258 and previous config saved to /var/cache/conftool/dbconfig/20200521-050328-marostegui.json
• 05:00 marostegui@cumin1001: dbctl commit (dc=all): 'Set enwiki as read-only for maintenance T251982', diff saved to https://phabricator.wikimedia.org/P11257 and previous config saved to /var/cache/conftool/dbconfig/20200521-050029-marostegui.json
• 01:03 krinkle@deploy1001: Synchronized wmf-config/mc.php: Ic9efa98312b (duration: 01m 08s)

2020-05-20

• 20:16 herron: logstash1011:~# kafka-preferred-replica-election --zookeeper conf1004.eqiad.wmnet,conf1005.eqiad.wmnet,conf1006.eqiad.wmnet/kafka/logging-eqiad
• 19:27 robh: cp5012 still offline for mem tests, "fast" testing complete without errors and extended testing in progress. system firmware was updated before testing. T251219
• 18:10 XioNoX: accept 198.35.27.0/24 from Anycast peers on all routers - T253196
• 18:01 XioNoX: add BGP between authdns2001 and cr1-codfw - T253196
• 17:57 XioNoX: accept 198.35.27.0/24 from Anycast peers on cr3-ulsfo - T253196
• 17:44 robh: cp5012 rebooting for troubleshooting
• 17:02 bblack: dns* + authdns* - disabling puppet to test https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/597311/
• 16:53 bblack: kraz.wikimedia.org ( https://wikitech.wikimedia.org/wiki/IRCD ) - stopping ircecho then ircd, then restarting them in reverse order - T239993
• 16:01 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'mathoid' for release 'production' .
• 16:01 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'mathoid' for release 'canary' .
• 15:42 elukey: update puppet compiler's facts
• 15:21 moritzm: installing libssh security updates
• 15:15 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'mathoid' for release 'staging' .
• 15:00 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T253096 [itwikivoyage] Undeploy Insider and Listings extensions (duration: 01m 08s)
• 14:43 marostegui: Replace tendril_purge_global_status_log_5m event with the new one (purging every 2d of data and with a higher limit of rows) - T252331
• 14:34 hnowlan@deploy1001: Finished deploy [restbase/deploy@6d2f88c]: Add awa.wikipedia.org to wikipedia list (duration: 19m 49s)
• 14:15 hnowlan@deploy1001: Started deploy [restbase/deploy@6d2f88c]: Add awa.wikipedia.org to wikipedia list
• 14:06 XioNoX: special-ranges6, remove 4000::/2 and 8000::/1
• 14:03 bblack: authdns1001 - poweroff for T241770
• 14:00 bblack: cr2-eqiad - re-routing ns[01] public IPs from authdns1001 (going offline for hw work) to dns1002 - T241770 (redo from earlier, commit didn't take for whatever reason)
• 13:52 bblack: cr[12]-eqiad - re-routing ns[01] public IPs from authdns1001 (going offline for hw work) to dns1002 - T241770
• 13:51 bblack: authdns1001 - downtimed for physical work - T241770
• 13:24 milimetric@deploy1001: Finished deploy [analytics/refinery@a891999] (thin): Regular analytics weekly train THIN [analytics/refinery@a891999] (duration: 00m 10s)
• 13:23 milimetric@deploy1001: Started deploy [analytics/refinery@a891999] (thin): Regular analytics weekly train THIN [analytics/refinery@a891999]
• 13:23 milimetric@deploy1001: Finished deploy [analytics/refinery@a891999]: Regular analytics weekly train [analytics/refinery@a891999] (duration: 38m 33s)
• 13:23 godog: remove stale tcp service on lvs codfw low-traffic 10.2.1.53:10902
• 13:00 Amir1: creating two wikis are done
• 12:52 ladsgroup@deploy1001: Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 10m 49s)
• 12:45 milimetric@deploy1001: Started deploy [analytics/refinery@a891999]: Regular analytics weekly train [analytics/refinery@a891999]
• 12:41 ladsgroup@deploy1001: Synchronized static/images/project-logos/: Creating Wiktionary Konkani (gomwiktionary) - T249506 (duration: 01m 06s)
• 12:40 dzahn@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 12:38 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Creating Wiktionary Konkani (gomwiktionary) - T249506 (duration: 01m 05s)
• 12:35 ladsgroup@deploy1001: rebuilt and synchronized wikiversions files: Creating Wiktionary Konkani (gomwiktionary) - T249506
• 12:33 ladsgroup@deploy1001: Synchronized dblists: Creating Wiktionary Konkani (gomwiktionary) - T249506 (duration: 01m 06s)
• 12:28 godog: roll-restart pybal on codfw low-traffic - T233956
• 12:26 dzahn@cumin1001: START - Cookbook sre.ganeti.makevm
• 12:22 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1)
• 12:22 ladsgroup@deploy1001: Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 03m 01s)
• 12:21 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
• 12:18 ladsgroup@deploy1001: Synchronized langlist: Create Awadhi Wikipedia (awawiki) - T251371 (duration: 01m 06s)
• 12:16 ladsgroup@deploy1001: Synchronized static/images/project-logos: Create Awadhi Wikipedia (awawiki) - T251371 (duration: 01m 06s)
• 12:14 ladsgroup@deploy1001: Synchronized multiversion/MWMultiVersion.php: Create Awadhi Wikipedia (awawiki) - T251371 (duration: 01m 06s)
• 12:12 ladsgroup@deploy1001: rebuilt and synchronized wikiversions files: Create Awadhi Wikipedia (awawiki) - T251371
• 12:07 ladsgroup@deploy1001: Synchronized dblists: (no justification provided) (duration: 01m 08s)
• 11:37 mutante: rebooting ganeti1009 and ganeti1011 to hopefully clear icinga alerts about microcode mitigations
• 11:10 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool new host db1142 and db1084', diff saved to https://phabricator.wikimedia.org/P11253 and previous config saved to /var/cache/conftool/dbconfig/20200520-111013-marostegui.json
• 11:07 jynus@cumin1001: dbctl commit (dc=all): 'Repool es1018, es1015 fully', diff saved to https://phabricator.wikimedia.org/P11252 and previous config saved to /var/cache/conftool/dbconfig/20200520-110732-jynus.json
• 11:04 jbond42: roll out update or exim4
• 10:46 moritzm: installing 4.19.118 Linux packages on Buster hosts
• 10:28 vgutierrez: rolling restart of ats-tls in text@esams - T249335
• 10:19 marostegui@cumin1001: dbctl commit (dc=all): 'Increase weight for db1142 and db1084 on s4', diff saved to https://phabricator.wikimedia.org/P11250 and previous config saved to /var/cache/conftool/dbconfig/20200520-101928-marostegui.json
• 10:07 jynus@cumin1001: dbctl commit (dc=all): 'Repool es1018, es1015 at 50% weight', diff saved to https://phabricator.wikimedia.org/P11249 and previous config saved to /var/cache/conftool/dbconfig/20200520-100726-jynus.json
• 09:43 vgutierrez: disable KA for POST/PUT requests on esams - T249335
• 09:36 XioNoX: create ROAs for 198.35.26.0/24 and 198.35.27.0/24 - T253196
• 09:33 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 09:31 marostegui@cumin1001: dbctl commit (dc=all): 'Increase weight for db1142 and db1084 on s4', diff saved to https://phabricator.wikimedia.org/P11247 and previous config saved to /var/cache/conftool/dbconfig/20200520-093141-marostegui.json
• 09:30 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 09:29 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:28 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 09:28 XioNoX: create ARIN inetnum 198.35.27.0/24 and route 198.35.26.0/24 + 198.35.27.0/24 - T253196
• 09:26 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:26 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 09:26 marostegui: Upgrade db1083 (s1 master) to 10.1.43-2 without restarting T251982
• 09:25 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 09:24 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 09:15 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:12 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 09:11 marostegui@cumin1001: dbctl commit (dc=all): 'Increase weight for new host db1142 and start to repool db1084', diff saved to https://phabricator.wikimedia.org/P11246 and previous config saved to /var/cache/conftool/dbconfig/20200520-091153-marostegui.json
• 09:08 dzahn@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 09:08 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:05 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 09:02 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:01 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'mathoid' for release 'staging' .
• 09:00 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:00 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 08:57 marostegui@cumin1001: dbctl commit (dc=all): 'Pool db1142 with minimum weight for the first time T252512', diff saved to https://phabricator.wikimedia.org/P11245 and previous config saved to /var/cache/conftool/dbconfig/20200520-085757-marostegui.json
• 08:57 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 08:55 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:52 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 08:52 dzahn@cumin1001: START - Cookbook sre.ganeti.makevm
• 08:49 _joe_: converting mw1266-1275 to use envoy T247389
• 08:46 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:44 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 08:42 XioNoX: Remove bogons4 for policy options on all routers - gerrit 597272
• 08:36 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:34 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:33 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 08:33 _joe_: disabling puppet on mw1266-1275 for migration to envoy
• 08:31 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 08:21 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 08:19 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 07:41 marostegui: alter table categorylinks engine=Innodb ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=8,force on all labsdb1011 wikis - T249188
• 07:24 moritzm: install systemd security updates
• 07:10 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1084 to clone db1142 T252512', diff saved to https://phabricator.wikimedia.org/P11241 and previous config saved to /var/cache/conftool/dbconfig/20200520-071010-marostegui.json
• 00:05 RoanKattouw: Ran namespaceDupes.php on tiwiki and tiwiktionary for T251287
• 00:03 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Set sitename and meta namespace localizations for tiwiki and tiwiktionary (T251287) (duration: 01m 06s)

2020-05-19

• 23:59 RoanKattouw: Ran namespaceDupes.php on jvwiki and jvwiktionary for T252754
• 23:57 jforrester@deploy1001: Synchronized php-1.35.0-wmf.32/extensions/Insider/includes/InsiderHooks.php: T252846 Use SidebarBeforeOutput hook with correct format (duration: 01m 06s)
• 23:55 catrope@deploy1001: Finished scap: i18n scap for namespace localizations (T251287, T252754) (duration: 62m 26s)
• 22:53 catrope@deploy1001: Started scap: i18n scap for namespace localizations (T251287, T252754)
• 18:46 herron: performing rolling restarts of codfw/eqiad ELK clusters for java updates
• 18:41 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Grant template editors editcontentmodel on enwiki (T253081) (duration: 01m 06s)
• 18:35 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable GrowthExperiments features on frwiki (T252420) (duration: 01m 08s)
• 17:09 arturo: added tesseract suite to stretch-wikimedia component/tesseract-410-bpo (T247422)
• 16:24 godog: power cycle thanos-fe* / thanos-be*
• 15:23 kormat@cumin1001: dbctl commit (dc=all): 'Repool db2073 into s4 T252985', diff saved to https://phabricator.wikimedia.org/P11236 and previous config saved to /var/cache/conftool/dbconfig/20200519-152340-kormat.json
• 15:20 sukhe@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:20 sukhe@cumin1001: START - Cookbook sre.hosts.downtime
• 15:16 cdanis: canary on ~150 hosts looks great, re-enabling puppet on all physical hosts ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕥☕ sudo cumin 'F:virtual = physical' 'enable-puppet "cdanis deploying I68c97d5"'
• 15:04 sukhe@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:04 sukhe@cumin1001: START - Cookbook sre.hosts.downtime
• 14:59 moritzm: installing fuse update from Buster point release
• 14:47 cdanis: disabling puppet on all physical hosts ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕥☕ sudo cumin 'F:virtual = physical' 'disable-puppet "cdanis deploying I68c97d5"'
• 14:38 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'blubberoid' for release 'staging' .
• 14:26 XioNoX: Set minimum-links 2 to AMS-IX LACP - T253122
• 13:53 XioNoX: configure new AMS-IX port as quarantine - T251121
• 13:18 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'mathoid' for release 'staging' .
• 13:09 elukey@cumin1001: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99)
• 13:09 jayme: updated helm: 2.16.7-1 -> 2.16.7-2 on deploy[1,2]001 and contint[1,2]001
• 13:09 elukey@cumin1001: START - Cookbook sre.ganeti.makevm
• 13:03 kormat@cumin1001: dbctl commit (dc=all): 'Pool db2136 into s4 T252985', diff saved to https://phabricator.wikimedia.org/P11233 and previous config saved to /var/cache/conftool/dbconfig/20200519-130313-kormat.json
• 12:40 ariel@deploy1001: Finished deploy [dumps/dumps@a329605]: make page content fixup script move inprog files into place if good (duration: 00m 04s)
• 12:40 ariel@deploy1001: Started deploy [dumps/dumps@a329605]: make page content fixup script move inprog files into place if good
• 12:37 jayme: imported helm 2.16.7-2 to main for buster-wikimedia, stretch-wikimedia, jessie-wikimedia
• 12:17 hnowlan@cumin1001: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0)
• 11:51 jynus: starting backups of es1, es2, es3 on eqiad into backup1002
• 11:41 jynus@cumin1001: dbctl commit (dc=all): 'Depool es1018, es1015, es1019', diff saved to https://phabricator.wikimedia.org/P11232 and previous config saved to /var/cache/conftool/dbconfig/20200519-114148-jynus.json
• 11:12 marostegui: Deploy schema change on db2124 (frwiki, jawiki, ruwiki) T238966
• 10:34 mutante: releases2001 - restarted failed jenkins
• 10:33 mutante: releases2001 - Failed to restart jenkins.service: The name org.freedesktop.PolicyKit1 was not provided by any .service files
• 10:32 volans: flushed all Netbox caches (manage.py invalidate all) - T253091
• 10:29 volans: start Netbox restore - T253091
• 10:18 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'mathoid' for release 'staging' .
• 10:13 akosiaris: upgrade etherpad-lite to 1.8.4 on etherpad1002
• 09:58 hnowlan: roll-restart of eqiad restbase hosts for java security updates
• 09:58 hnowlan@cumin1001: START - Cookbook sre.cassandra.roll-restart
• 09:55 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'mathoid' for release 'production' .
• 09:55 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'mathoid' for release 'canary' .
• 09:55 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'mathoid' for release 'production' .
• 09:54 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'mathoid' for release 'staging' .
• 09:10 godog: eqiad-prod: decom ms-be101[678] - T252008
• 08:07 XioNoX: Push 596597: BGP: standardize fixed part of IX4/IX6 groups - eqsin
• 08:04 XioNoX: Push 596597: BGP: standardize fixed part of IX4/IX6 groups - esams
• 08:01 XioNoX: Push 596597: BGP: standardize fixed part of IX4/IX6 groups - eqiad
• 07:55 volker-e@deploy1001: Finished deploy [design/style-guide@37c67dd]: Deploy design/style-guide: (duration: 00m 06s)
• 07:54 volker-e@deploy1001: Started deploy [design/style-guide@37c67dd]: Deploy design/style-guide:
• 07:52 XioNoX: Push 596597: BGP: standardize fixed part of IX4/IX6 groups - *dfw
• 07:49 XioNoX: Push 596597: BGP: standardize fixed part of IX4/IX6 groups - ulsfo
• 07:45 vgutierrez: rolling upgrade to trafficserver 8.0.7-1wm10 with puppet disabled on cp hosts
• 07:09 jynus: starting es4 & es5 eqiad backups with low concurrency
• 06:35 elukey@cumin1001: END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0)
• 06:29 elukey@cumin1001: START - Cookbook sre.zookeeper.roll-restart-zookeeper
• 06:24 elukey@cumin1001: END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0)
• 06:17 elukey@cumin1001: START - Cookbook sre.zookeeper.roll-restart-zookeeper
• 05:57 volker-e@deploy1001: Finished deploy [design/style-guide@7bfbd2a]: Deploy design/style-guide: (duration: 00m 06s)
• 05:57 volker-e@deploy1001: Started deploy [design/style-guide@7bfbd2a]: Deploy design/style-guide:
• 05:03 marostegui@cumin1001: dbctl commit (dc=all): 'Set s2 and s8 as read-only=off for maintenance T251981', diff saved to https://phabricator.wikimedia.org/P11227 and previous config saved to /var/cache/conftool/dbconfig/20200519-050346-marostegui.json
• 05:00 marostegui@cumin1001: dbctl commit (dc=all): 'Set s2 and s8 as read-only for maintenance T251981', diff saved to https://phabricator.wikimedia.org/P11226 and previous config saved to /var/cache/conftool/dbconfig/20200519-050043-marostegui.json
• 04:27 marostegui: Repool labsdb1011 T249188
• 03:29 volker-e@deploy1001: Finished deploy [design/style-guide@4b4bc51]: Deploy design/style-guide: (duration: 00m 07s)
• 03:28 volker-e@deploy1001: Started deploy [design/style-guide@4b4bc51]: Deploy design/style-guide:

2020-05-18

• 23:50 pt1979@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 23:47 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
• 23:25 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 23:23 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
• 23:12 ryankemper: Restarted `wdqs-updater` across all wdqs nodes and restarted `wdqs-categories` across all nodes except 1010 (test wdqs server) and 1009 (automated deployment server)
• 22:55 Krinkle: Clear module_deps on dewiki (group2, old mw version, s5) to monitor regeneration
• 22:48 Krinkle: Clear module_deps on group0 (mostly s3) to monitor regeneration
• 22:35 Krinkle: Clear module_deps on commonswiki (group1, s4) to monitor regeneration
• 22:33 ryankemper@deploy1001: Finished deploy [wdqs/wdqs@4886dc3]: 0.3.32 (duration: 17m 12s)
• 22:19 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 22:18 Krinkle: Clear module_deps on s2 wikis to monitor regeneration
• 22:16 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
• 22:15 ryankemper@deploy1001: Started deploy [wdqs/wdqs@4886dc3]: 0.3.32
• 22:02 Krinkle: Clear module_deps on hewiki (group1, s7) to monitor regeneration, ref T247028
• 21:40 pt1979@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 21:37 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
• 21:23 krinkle@deploy1001: Synchronized php-1.35.0-wmf.32/includes/resourceloader/dependencystore/: I015fa5885, I972a93806006 (duration: 01m 07s)
• 21:14 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 21:11 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
• 20:27 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@12efc14]: Update mobileapps to c960b349 (duration: 03m 31s)
• 20:24 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@12efc14]: Update mobileapps to c960b349
• 19:07 herron: performing rolling maintenance on kafka-main to pick up java security updates
• 19:00 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Ic005093778d (duration: 01m 08s)
• 18:58 krinkle@deploy1001: Synchronized wmf-config/CommonSettings-labs.php: Ic005093778d (duration: 01m 06s)
• 18:49 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 18:46 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
• 18:38 volans: upgraded spicerack to 0.0.37-1 on cumin[12]001
• 18:24 hnowlan@cumin1001: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0)
• 18:13 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Fix English Wikipedia wordmark dimensions (T252143) (duration: 01m 06s)
• 17:14 XioNoX: update domain object for 56.15.185.in-addr.arpa - T247972
• 17:06 bblack: dns1001 - removing downtimes, back in service - T241770
• 16:45 bstorm_: updated views on labsdb1011 for the wb_terms changes T251598
• 16:32 bblack@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 16:30 bblack@cumin1001: START - Cookbook sre.hosts.downtime
• 16:17 bblack: dns1001 - reimaging for new NIC - T241770
• 16:10 volans: uploaded spicerack_0.0.37-1_amd64.deb to apt.wikimedia.org stretch-wikimedia
• 15:52 hnowlan: rolling codfw cassandra for java security updates
• 15:51 hnowlan@cumin1001: START - Cookbook sre.cassandra.roll-restart
• 15:30 hnowlan@cumin1001: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0)
• 15:22 elukey@cumin1001: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0)
• 15:11 Krinkle: krinkle@mc1021 Pruning the old `echo:seen:` Redis keys that didn't have a ttl yet
• 14:57 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'termbox' for release 'production' .
• 14:56 hnowlan: roll-restart of sessionstore cassandra hosts for java security update
• 14:55 hnowlan@cumin1001: START - Cookbook sre.cassandra.roll-restart
• 14:53 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'termbox' for release 'production' .
• 14:50 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'termbox' for release 'staging' .
• 14:50 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'termbox' for release 'test' .
• 14:35 hnowlan@deploy1001: Finished deploy [changeprop/deploy@16bf19f]: Stop consuming purges topic, purged is now doing this (duration: 01m 22s)
• 14:34 hnowlan@deploy1001: Started deploy [changeprop/deploy@16bf19f]: Stop consuming purges topic, purged is now doing this
• 14:33 _joe_: start consuming $dc.resource-purge kafka topic from purged in all of esams T133821
• 14:29 _joe_: start consuming $dc.resource-purge kafka topic from purged in all of eqiad T133821
• 14:23 _joe_: start consuming $dc.resource-purge kafka topic from purged in all of eqsin, ulsfo T133821
• 14:19 _joe_: start consuming $dc.resource-purge kafka topic from purged in all of codfw T133821
• 14:15 kormat@cumin1001: dbctl commit (dc=all): 'Depool db2073 while replacing it T252985', diff saved to https://phabricator.wikimedia.org/P11216 and previous config saved to /var/cache/conftool/dbconfig/20200518-141505-kormat.json
• 14:12 bblack: dns1001 - shutting down for T241770
• 14:09 volans: uploaded spicerack_0.0.36-1_amd64.deb to apt.wikimedia.org stretch-wikimedia
• 14:07 bblack: authdns - ns[01] static routes on cr[12]-eqiad switching back to authdns1001 (oops, that's not the server we're taking offline today!)
• 14:06 vgutierrez: upload trafficserver 8.0.7-1wm9 to apt.wm.o (buster)
• 14:02 elukey@cumin1001: START - Cookbook sre.hadoop.roll-restart-workers
• 14:00 elukey@cumin1001: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0)
• 13:57 bblack: authdns - ns[01] static routes on cr[12]-eqiad switching from authdns1001 to dns1002 for T241770
• 13:29 elukey@cumin1001: START - Cookbook sre.hadoop.roll-restart-workers
• 13:00 hashar@deploy1001: Synchronized php-1.35.0-wmf.32/skins/Vector/includes/VectorTemplate.php: VectorTemplate: SkinTemplateToolboxEnd hook isn't deprecated - T252906 (duration: 01m 07s)
• 11:52 marostegui: Install 10.1.43-2 on db1122 and db1109 - T251981
• 11:27 Lucas_WMDE: EU SWAT done
• 11:25 lucaswerkmeister-wmde@deploy1001: Synchronized php-1.35.0-wmf.32/extensions/Wikibase/: SWAT: Fix core's TitleFactory not being used correctly (T252803) (duration: 01m 12s)
• 11:20 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Update GrowthExperiments mentor list page for viwiki (duration: 01m 06s)
• 11:10 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Make the threshold for Chinese WP to prevent publishing 5% more strict (T252786) (duration: 01m 06s)
• 10:38 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (597033) (duration: 01m 06s)
• 10:37 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (597033) (duration: 01m 32s)
• 10:37 elukey: copy prometheus-druid-exporter 0.8-1 from stretch to buster wikimedia
• 10:20 _joe_: upgrading purged in the remaining datacenters
• 10:07 elukey: upload druid 0.12.3-1.1 to stretch|buster-wikimedia
• 10:02 vgutierrez: upload trafficserver 8.0.7-1wm8 to apt.wm.o (buster)
• 09:53 _joe_: upgrading purged in codfw, ulsfo
• 09:46 mutante: contint2001 - apt-get remove --purge openjdk-11-* - T224591
• 09:43 _joe_: upload purged 0.13 to buster-wikimedia
• 08:44 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'termbox' for release 'production' .
• 08:43 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'termbox' for release 'production' .
• 08:25 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'termbox' for release 'staging' .
• 08:25 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'termbox' for release 'test' .
• 08:13 godog: set weight to 0 for all but objects in ms-be10[678] - T252008
• 07:57 mutante: replacing apache module with httpd module on deployment servers
• 07:47 moritzm: installing apt security updates on jessie systems
• 07:36 marostegui: Remove and add pc2007 from tendril as the Act is frozen after reimage - T250666
• 07:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db2088 after upgrade', diff saved to https://phabricator.wikimedia.org/P11214 and previous config saved to /var/cache/conftool/dbconfig/20200518-072234-marostegui.json
• 07:20 marostegui: Upload MariaDB 10.4.13 to the buster repo - T250666
• 07:00 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 06:58 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 06:41 marostegui: Stop MySQL on db2088
• 06:24 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2088 for upgrade', diff saved to https://phabricator.wikimedia.org/P11213 and previous config saved to /var/cache/conftool/dbconfig/20200518-062452-marostegui.json
• 05:55 _joe_: installing purged 0.12 on cp2027
• 05:54 _joe_: uploaded purged 0.12 to apt.w.o
• 05:00 marostegui: Stop MySQL on labsdb1011 to copy its content to backup1001 T249188

2020-05-16

• 22:04 Krinkle: krinkle@mc1022 Pruning the old `echo:seen:` Redis keys that didn't have a ttl yet
• 21:56 Krinkle: krinkle@mc1019 Pruning the old `echo:seen:` Redis keys that didn't have a ttl yet
• 20:23 Krinkle: krinkle@mc1034,mc1035,mc1036 Pruning the old `echo:seen:` Redis keys that didn't have a ttl yet
• 20:04 Krinkle: krinkle@mc1033 Pruning the old `echo:seen:` Redis keys that didn't have a ttl yet
• 19:57 Krinkle: krinkle@mc1032 Pruning the old `echo:seen:` Redis keys that didn't have a ttl yet
• 19:51 Krinkle: krinkle@mc1031 Pruning the old `echo:seen:` Redis keys that didn't have a ttl yet
• 19:42 Krinkle: krinkle@mc1030 Pruning the old `echo:seen:` Redis keys that didn't have a ttl yet
• 19:25 Krinkle: krinkle@mc1029 Pruning the old `echo:seen:` Redis keys that didn't have a ttl yet
• 19:10 Krinkle: krinkle@mc1028 Pruning the old `echo:seen:` Redis keys that didn't have a ttl yet
• 18:58 Krinkle: krinkle@mc1027 Pruning the old `echo:seen:` Redis keys that didn't have a ttl yet, ref T252945
• 18:54 Krinkle: krinkle@mc1026 Pruning the old `echo:seen:` Redis keys that didn't have a ttl yet, ref T252945
• 18:30 Krinkle: krinkle@mc1024 Pruning the old `echo:seen:` Redis keys that didn't have a ttl yet, ref T252945
• 18:24 Krinkle: krinkle@mc1025 Pruning the old `echo:seen:` Redis keys that didn't have a ttl yet, ref T252945
• 17:56 Krinkle: krinkle@mc1023 Pruning old echo:seen: Redis keys that didn't use a ttl yet, ref T252945
• 17:49 Krinkle: krinkle@mwmaint1002: Running cleanupRemovedModules.php to prune old module_deps rows T113916
• 17:24 Krinkle: krinkle@mc1020 Prune old echo:seen: keys that have ttl:-1 from Redis main stash, ref T252945
• 15:16 Krinkle: krinkle@mc1020 Looking at why there are still over 2M echo:seen keys in redis main stash
• 00:55 krinkle@deploy1001: Synchronized wmf-config/logging.php: I046868190b472 (duration: 01m 13s)
• 00:24 ryankemper@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
• 00:21 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-transfer
• 00:21 ryankemper@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
• 00:18 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-transfer
• 00:18 ryankemper@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
• 00:16 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-transfer
• 00:16 ryankemper@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
• 00:13 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-transfer
• 00:13 ryankemper@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
• 00:10 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-transfer
• 00:10 ryankemper@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
• 00:08 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-transfer
• 00:06 ryankemper@cumin2001: END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99)
• 00:06 ryankemper@cumin2001: START - Cookbook sre.wdqs.data-transfer
• 00:05 ryankemper@cumin2001: END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99)
• 00:05 ryankemper@cumin2001: START - Cookbook sre.wdqs.data-transfer

2020-05-15

• 23:50 ryankemper@cumin2001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
• 23:47 ryankemper@cumin2001: START - Cookbook sre.wdqs.data-transfer
• 23:46 ryankemper@cumin2001: END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99)
• 23:46 ryankemper@cumin2001: START - Cookbook sre.wdqs.data-transfer
• 23:46 ryankemper@cumin2001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
• 23:43 ryankemper@cumin2001: START - Cookbook sre.wdqs.data-transfer
• 23:43 ryankemper@cumin2001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
• 23:37 ryankemper@cumin2001: START - Cookbook sre.wdqs.data-transfer
• 23:35 ryankemper: Pooled wdqs2007 following successful query tests (all data transfers are done now)
• 22:53 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: I1b1578a57ef5 (duration: 01m 07s)
• 22:51 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Iaa240eb8cf9 (duration: 01m 06s)
• 21:41 ryankemper: depooled wdqs2007 while it catches up on lag
• 21:40 ryankemper@cumin2001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
• 20:36 ryankemper@cumin2001: START - Cookbook sre.wdqs.data-transfer
• 20:33 ryankemper: pooled wdqs2003 and wdqs1007 following successful query tests
• 19:46 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: If0fd1b51 (duration: 01m 08s)
• 18:53 ryankemper@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
• 18:34 ryankemper: depooled wdqs2003 while lag catches up
• 18:32 ryankemper@cumin2001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
• 17:55 vgutierrez: upload acme-chief 0.25 to apt.wm.o (buster) - T252881
• 17:27 XioNoX: renumber cr2-eqord:xe-0/1/1 to xe-0/1/3 - T221259
• 17:02 ryankemper@cumin2001: START - Cookbook sre.wdqs.data-transfer
• 17:01 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-transfer
• 17:00 ryankemper: depooled wqds1007 in preparation for impending wdqs data xfer
• 16:58 ryankemper@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
• 16:53 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-transfer
• 16:52 ryankemper@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
• 16:49 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-transfer
• 16:02 gehel@cumin2001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
• 15:57 gehel@cumin2001: START - Cookbook sre.wdqs.data-transfer
• 15:56 gehel@cumin2001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
• 15:52 gehel@cumin2001: START - Cookbook sre.wdqs.data-transfer
• 15:49 gehel@cumin2001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
• 15:45 gehel@cumin2001: START - Cookbook sre.wdqs.data-transfer
• 15:44 gehel@cumin2001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
• 15:40 gehel@cumin2001: START - Cookbook sre.wdqs.data-transfer
• 15:36 gehel@cumin2001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
• 15:32 gehel@cumin2001: START - Cookbook sre.wdqs.data-transfer
• 15:31 gehel@cumin2001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
• 15:27 gehel@cumin2001: START - Cookbook sre.wdqs.data-transfer
• 14:19 cdanis: reverting sysctl net.ipv4.udp_mem to original on netflow3001
• 14:18 cdanis: re-enable puppet on netflow*
• 14:14 cdanis: disable puppet on netflow*
• 14:04 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 14:01 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
• 13:47 ema: cp2029, cp3050: varnish-fe-restart to clear 'child restarted' alerts
• 13:47 vgutierrez: downgrade ats to version 8.0.7-1wm7 on cp4032
• 13:42 vgutierrez: upgrade ats to version 8.0.7-1wm8 on cp4032
• 13:37 mutante: rsyncing gerrit git data from gerrit1001 to gerrit1002 (T200739)
• 13:13 cdanis: increase samplicator recvbuf on netflow3001 & restart samplicator
• 13:01 cdanis: increasing sysctl net.ipv4.udp_mem on netflow3001
• 09:57 vgutierrez: upload trafficserver 8.0.7-1wm7 to apt.wm.o (buster)
• 09:21 ema: cp2029: attempt forced discard of stuck VCL T236754
• 09:09 elukey: restart druid brokers on druid100[4-6] - locked up due to datasources dropped - T226035
• 08:51 ema: cp2029: try out varnish 5.1.3-1wm15 T236754
• 07:36 XioNoX: bumps prefix limit for AS16735 in eqiad
• 05:35 jynus: stop replication on pc2009, pc2010 for benchmarking T252761
• 04:53 volker-e@deploy1001: Finished deploy [design/style-guide@dc956a3]: Deploy design/style-guide: (duration: 00m 10s)
• 04:52 volker-e@deploy1001: Started deploy [design/style-guide@dc956a3]: Deploy design/style-guide:
• 04:42 vgutierrez: repool cp5006
• 04:28 vgutierrez: depool and reboot cp5006

2020-05-14

• 23:24 catrope@deploy1001: Synchronized static/images/project-logos/: Revert temporary 20k logo for vecwiki (T252770) (duration: 01m 06s)
• 23:23 RoanKattouw: Ran namespaceDupes.php for T252343
• 23:20 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Create Gapura (Portal) namespace on jvwiki (T252343) (duration: 01m 06s)
• 23:09 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add *.ub.uni-heidelberg.de and hq.eso.org to $wgCopyUploadDomains (T252600, T252726) (duration: 01m 07s)
• 21:43 ryankemper: depooled wdqs2006 while lag recovers
• 21:42 ryankemper@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
• 21:08 ryankemper@cumin2001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
• 20:16 volans: moved codereview.tar.gz and with_r.tar.gz from miscweb1002 to cumin1001 to free space
• 20:15 hashar@deploy1001: Synchronized php-1.35.0-wmf.32/skins/Vector/includes/VectorTemplate.php: Allow plain text labels in side bar - T252727 (duration: 01m 06s)
• 19:51 ryankemper@cumin2001: START - Cookbook sre.wdqs.data-transfer
• 19:50 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-transfer
• 19:49 ryankemper: Depooled wqds1006 in preparation for impending wdqs data xfer
• 18:36 Urbanecm: Morning SWAT done
• 18:35 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: 15adbbc: [thwikisource] Set ProofReadPage separator to an empty string (T252610) (duration: 01m 06s)
• 18:26 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: 4b8399c: Undeploy graphoid from mediawikiwiki (T242855) (duration: 01m 05s)
• 18:16 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: f03a45c: Adding import to test wikis from mediawikiwiki (T242855) (duration: 01m 07s)
• 17:03 XioNoX: asw2-d-eqiad> request virtual-chassis vc-port delete pic-slot 1 port 1 member 1 - T252797
• 16:55 XioNoX: asw2-d-eqiad> request virtual-chassis vc-port delete pic-slot 1 port 3 member 1 - T252797
• 16:51 XioNoX: asw2-d-eqiad> request virtual-chassis vc-port set pic-slot 0 port 48 member 2 - T252797
• 16:50 XioNoX: request virtual-chassis vc-port set pic-slot 1 port 2 member 1 - T252797
• 16:42 XioNoX: request virtual-chassis vc-port delete pic-slot 1 port 2 member 1 - T252797
• 16:36 XioNoX: asw2-d-eqiad> request virtual-chassis vc-port delete pic-slot 0 port 48 member 2 - T252797
• 15:59 hnowlan@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'changeprop' for release 'production' .
• 15:57 hnowlan@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop' for release 'production' .
• 15:56 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' .
• 15:25 XioNoX: disable asw2-d1-eqiad:et-1/1/0 - T251663
• 14:39 mutante: kuai kuai is https://twitter.com/Arlieth/status/1257714333133357056 | https://en.wikipedia.org/wiki/Kuai_Kuai_culture
• 13:31 _joe_: updating purged to 0.11 in eqiad,eqsin,esams
• 12:47 vgutierrez: rolling upgrade ats to version 8.0.7-1wm7
• 12:46 elukey@cumin1001: END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0)
• 12:43 elukey@cumin1001: START - Cookbook sre.aqs.roll-restart
• 12:22 kormat: reverted iosched on pc1010 to `mq-deadline` T252761
• 11:47 kormat: changed iosched on pc1010 to `none` as a test T252761
• 11:07 matthiasmullie: EU swat done
• 11:05 mlitn@deploy1001: Synchronized php-1.35.0-wmf.32/extensions/WikibaseMediaInfo/: [MediaInfo] Enable media search for all users by default (duration: 01m 12s)
• 11:04 vgutierrez: upgrade ats to version 8.0.7-1wm7 on cp3064
• 10:31 fdans@deploy1001: Finished deploy [analytics/refinery@6f13979]: Regular analytics weekly train (duration: 17m 14s)
• 10:14 fdans@deploy1001: Started deploy [analytics/refinery@6f13979]: Regular analytics weekly train
• 09:58 elukey: remove matomo 3.11 from the main component of stretch-wikimedia
• 09:56 elukey: upgrade matomo on matomo1001 to 3.13.3 (latest upstream) - T252741
• 09:30 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'zotero' for release 'production' .
• 09:29 elukey: upload matomo-3.13.3 to thirdparty/matomo on stretch|buster-wikimedia
• 09:22 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'zotero' for release 'production' .
• 08:57 elukey: imported gpg key 1FD752571FE36FF23F78F91B81E2E78B66FED89E in apt1001 (Matomo public debian repo)
• 08:56 moritzm: installing Java security updates on Presto
• 08:43 jayme: updated helm: 2.12.2-1 -> 2.16.7-1 on deploy[1,2]001 and contint1001. 2.12.2-4 -> 2.16.7-1 on contint2001
• 08:39 jayme: imported helm 2.16.7-1 to main for jessie-wikimedia
• 08:32 moritzm: installing Java security updates on Hadoop/AQS/Druid
• 08:20 jayme@deploy2001: helmfile [STAGING] Ran 'sync' command on namespace 'zotero' for release 'staging' .
• 08:00 vgutierrez: upgrade ats to version 8.0.7-1wm7 on cp5011
• 07:03 moritzm: installing apt security updates
• 06:33 ryankemper: Pooled wdqs2005 following successful test queries
• 04:46 ryankemper@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
• 04:02 ryankemper@cumin2001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
• 02:59 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-transfer
• 02:59 ryankemper: wdqs1005 has been de-pooled pending wdqs data xfer
• 02:57 ryankemper@cumin2001: START - Cookbook sre.wdqs.data-transfer
• 02:57 ryankemper: wdqs1004 was repooled after successful test queries
• 02:55 ryankemper: wdqs2006 was repooled after successful test queries
• 01:32 ryankemper: depooled wdqs2006 while waiting for lag to recover
• 00:54 foks: change password for "Python eggs"
• 00:37 ryankemper@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
• 00:31 ryankemper@cumin2001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
• 00:08 twentyafterfour: phabricator update appears to be stable.
• 00:05 twentyafterfour: updating phabricator. 1 patch + new translations. Expect only brief downtime.

2020-05-13

• 23:46 cstone: SmashPig revision changed from cd1a49da5f to 2702b04329
• 23:43 ejegg: updated payments-wiki from dabba1804c to 3c465cb11c
• 23:36 ejegg: rolled back payments-wiki to dabba1804c
• 23:29 ejegg: updated payment-wiki from dabba1804c to 3c465cb11c
• 22:40 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-transfer
• 22:39 ryankemper@cumin2001: START - Cookbook sre.wdqs.data-transfer
• 22:36 ryankemper: Depooled wdqs1004 for subsequent wdqs data xfer
• 22:29 ryankemper: Pooled wdqs2005 given that lag has returned to normal levels and the instance is responding to queries correctly
• 22:26 ryankemper: Pooled wdqs1008 given that lag has returned to normal levels and the instance is responding to queries correctly
• 21:30 elukey: powercycle analytics1055
• 21:05 eileen: civicrm revision changed from cfb6101e39 to ed4c9522ac, config revision is 2eb75f8dff
• 20:16 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: T242430 Stop loading the ParsoidBatchAPI extension (duration: 01m 08s)
• 19:09 hashar@deploy1001: Synchronized php: group1 wikis to 1.35.0-wmf.32 (duration: 01m 05s)
• 19:08 hashar@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.32
• 18:54 twentyafterfour: restarted php-fpm on phab1001
• 18:53 thcipriani: restarting gerrit
• 18:52 twentyafterfour: restarting apache on phab1001 for lack of a better idea
• 18:50 herron: restarted kafka broker on kafka-main1001 for java security updates
• 18:22 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: 38db3e0: Update production wordmarks (T252143) (duration: 01m 07s)
• 18:17 urbanecm@deploy1001: Synchronized static/images/mobile/copyright/: SWAT: 38db3e0: Update production wordmarks (T252143) (duration: 01m 09s)
• 17:55 hnowlan@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop' for release 'production' .
• 17:53 hnowlan@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'changeprop' for release 'production' .
• 17:52 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'production' .
• 17:51 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' .
• 17:24 ryankemper: Manually depooled wdqs2005 while lag catches up following the data xfer
• 17:21 ryankemper@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
• 17:18 ryankemper@cumin2001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
• 17:12 urandom: restarted cassandra-c, restbase2017
• 17:04 hnowlan@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'changeprop' for release 'production' .
• 16:57 hnowlan@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop' for release 'production' .
• 16:54 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' .
• 16:11 James_F: Running AbuseFilter updateVarDumps on group0 on mwmaint1002 T246539
• 16:00 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 15:58 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
• 15:41 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 15:38 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
• 15:34 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 15:32 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
• 15:32 vgutierrez: upgrade ats to version 8.0.7-1wm7 on cp4032
• 15:30 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-transfer
• 15:30 jayme: imported scap 3.14.0-1 to main for buster-wikimedia
• 15:30 jayme: imported scap 3.14.0-1 to main for jessie-wikimedia
• 15:29 ryankemper: Manually de-pooling `wdqs1008.eqiad.wmnet` in preparation for wdqs data transfer
• 15:29 jayme: imported scap 3.14.0-1 to main for stretch-wikimedia
• 15:26 ryankemper@cumin2001: START - Cookbook sre.wdqs.data-transfer
• 15:23 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' .
• 15:08 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 15:06 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
• 14:55 _joe_: upgrading + restarting purged across ulsfo and codfw T133821
• 14:50 filippo@deploy1001: Finished deploy [librenms/librenms@0a88d64]: Upgrade LibreNMS to 1.63 T251222 (duration: 00m 10s)
• 14:50 filippo@deploy1001: Started deploy [librenms/librenms@0a88d64]: Upgrade LibreNMS to 1.63 T251222
• 14:35 vgutierrez: upload trafficserver 8.0.7-1wm6 to apt.wm.o (buster) - T249335 T251537
• 13:59 mholloway-shell@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'wikifeeds' for release 'production' .
• 13:57 mholloway-shell@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'wikifeeds' for release 'production' .
• 13:55 mholloway-shell@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' .
• 11:39 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add *.deutsche-digitale-bibliothek.de to the wgCopyUploadsDomains (T252296) (duration: 01m 06s)
• 11:17 Amir1: EU SWAT is done
• 11:14 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Disable wgLegacyJavaScriptGlobals on fawiki and wikidatawiki (T72470) (duration: 01m 06s)
• 11:09 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'mathoid' for release 'production' .
• 11:06 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: Anchor RegExp for Data Bridge in Beta (BETA-ONLY) (duration: 01m 06s)
• 11:00 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'mathoid' for release 'production' .
• 11:00 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'mathoid' for release 'canary' .
• 10:55 volans: imported tqdm 4.11.2-1 packages into buster-wikimedia component/spicerack
• 10:34 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'mathoid' for release 'staging' .
• 10:09 kormat@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool pc1007 as pc1 master T252182 (duration: 01m 05s)
• 09:55 jbond42: deployed a fix to ferm-status script. unmanaged ferm rules may get removed
• 09:40 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:38 kormat@cumin1001: START - Cookbook sre.hosts.downtime
• 09:37 marostegui: Upgrade db2102 to the new 10.4.13 - T250666
• 09:32 _joe_: installing purged 0.11 on cp2027 T133821
• 09:21 _joe_: installing purged 0.11 on cp2028 T133821
• 09:11 moritzm: re-enabling puppet
• 09:08 mutante: rsyncing /home dirs from people.wikimedia.org to new backend people1002
• 09:00 moritzm: disabling puppet temporarily
• 08:53 _joe_: uploaded purged 0.11
• 08:52 kormat@deploy1001: Synchronized wmf-config/db-eqiad.php: Pool pc1010 as pc1 master T252182 (duration: 01m 17s)
• 07:42 jayme: imported helm 2.16.7-1 to main for stretch-wikimedia
• 07:41 jayme: imported helm 2.16.7-1 to main for buster-wikimedia
• 07:29 godog: roll-restart logstash in codfw/eqiad for configuration change
• 07:14 elukey: upload spark2_2.4.4-bin-hadoop2.6-2 for buster/stretch on apt1001
• 05:33 ryankemper: wdqs2004 was depooled ~3 hours ago and was re-pooled ~10 mins ago after verifying the wdqs service was healthy
• 05:32 ryankemper: wdqs1003 was depooled ~6 hours ago and was re-pooled ~10 mins ago after verifying the wdqs service was healthy
• 05:27 _joe_: restarting php-fpm on mw1374, children dying with SIGILL
• 05:11 root@cumin1001: END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0)
• 05:11 root@cumin1001: Updating IPMI password on 1 hosts - root@cumin1001
• 05:10 root@cumin1001: START - Cookbook sre.hosts.ipmi-password-reset
• 05:10 root@cumin1001: END (FAIL) - Cookbook sre.hosts.ipmi-password-reset (exit_code=99)
• 05:10 root@cumin1001: START - Cookbook sre.hosts.ipmi-password-reset
• 04:52 kart_: Updated cxserver to 2020-05-11-082207-production (T250004)
• 04:47 kartik@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'cxserver' for release 'production' .
• 04:44 kartik@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'cxserver' for release 'production' .
• 04:42 kartik@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'cxserver' for release 'staging' .
• 02:27 ryankemper@cumin2001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
• 02:10 ryankemper@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
• 00:43 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-transfer
• 00:33 ryankemper@cumin2001: START - Cookbook sre.wdqs.data-transfer

2020-05-12

• 23:09 ryankemper@cumin2001: END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99)
• 23:06 ryankemper@cumin2001: START - Cookbook sre.wdqs.data-transfer
• 20:15 hashar@deploy1001: Synchronized php-1.35.0-wmf.32/includes/revisionlist/RevisionItemBase.php: Fix RevisionItemBase::getId to actually return an int, as intended - T252076 (duration: 01m 06s)
• 19:55 dpifke@deploy1001: Finished deploy [performance/navtiming@48110b9]: Fixes swapped dc/host labels - T238086 (duration: 00m 05s)
• 19:55 dpifke@deploy1001: Started deploy [performance/navtiming@48110b9]: Fixes swapped dc/host labels - T238086
• 19:05 hashar@deploy1001: rebuilt and synchronized wikiversions files: group0 wikis to 1.35.0-wmf.32
• 18:41 legoktm: started codereview-archiver script in screen on mwmaint1002
• 18:23 otto@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' .
• 18:23 otto@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' .
• 18:17 otto@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' .
• 18:17 otto@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' .
• 18:14 otto@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' .
• 18:14 otto@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' .
• 17:49 bblack: 'gdnsdctl replace' on all authdns to load new maxmind data
• 17:43 bblack: updating maxmind database on puppetmasters (usually automated weekly; we're mid-cycle)
• 17:10 James_F: Running AbuseFilter updateVarDumps on testwikis on mwmaint1002 T246539
• 16:55 James_F: Running AbuseFilter updateVarDumps on closed wikis on mwmaint1002 T246539
• 16:55 mstyles@deploy1001: Finished deploy [wdqs/wdqs@f617307]: v0.3.31 (duration: 14m 53s)
• 16:40 mstyles@deploy1001: Started deploy [wdqs/wdqs@f617307]: v0.3.31
• 16:35 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' .
• 15:48 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:48 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 15:48 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:48 andrew@cumin1001: START - Cookbook sre.hosts.downtime
• 15:34 filippo@cumin1001: conftool action : set/pooled=true; selector: dnsdisc=thanos-query
• 15:15 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' .
• 15:15 akosiaris@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' .
• 15:14 akosiaris@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' .
• 15:13 sukhe@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:13 sukhe@cumin1001: START - Cookbook sre.hosts.downtime
• 15:12 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 15:12 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 15:09 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:09 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 15:07 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:05 kormat@cumin1001: START - Cookbook sre.hosts.downtime
• 15:04 moritzm: installing 4.9.118 Linux updates on Buster nodes (reboots happening later)
• 15:02 moritzm: upgrading contint2001 to openjdk-8 u252
• 15:01 godog: bounce pybal on lvs2010 and lvs2009 - T252186
• 14:40 moritzm: imported openjdk-8 u252 forward port for buster-wikimedia component/jdk8
• 14:40 ema: rolling thumbor upgrade to 2.8-1+deb10u1 T252509 T219569 T236240
• 14:39 andrewbogott: rebuilding cloudcontrol1003 and 1004
• 14:38 hashar: 1.35.0-wmf.22 is on test wikis. Will be pushed to group0 later today during the american window (19:00 - 21:00 UTC) # T249964
• 14:34 ema: thumbor2001: repool
• 14:33 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: EventLogging to EventGate: - Test everywhere, SearchSatisfaction on testwiki only - T249261 (duration: 01m 06s)
• 14:33 ema: thumbor2001: upgrade python-thumbor-wikimedia to 2.8-1+deb10u1 T252509 T219569 T236240
• 14:23 moritzm: installing Java security updates on WDQS hosts
• 14:20 hashar@deploy1001: Finished scap: testwikis wikis to 1.35.0-wmf.32 (duration: 72m 04s)
• 14:05 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:05 aborrero@cumin1001: START - Cookbook sre.hosts.downtime
• 14:05 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:05 aborrero@cumin1001: START - Cookbook sre.hosts.downtime
• 14:02 dzahn@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
• 14:00 ema: thumbor2001: depool due to minor bug in 2.7-1+deb10u1 T252509 T219569 T236240
• 13:54 ema: thumbor2001: pool thumbor 2.7-1+deb10u1 for prod traffic T252509 T219569 T236240
• 13:50 ema: thumbor2001: upgrade python-thumbor-wikimedia to 2.7-1+deb10u1 T252509 T219569 T236240
• 13:42 jbond42: disable puppet on all CP hosts to deploy https://gerrit.wikimedia.org/r/c/operations/puppet/+/583342
• 13:36 kormat: reimaging pc2007 to buster T252182
• 13:36 moritzm: rebooting netflow* hosts for kernel update
• 13:36 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:36 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 13:33 vgutierrez: rolling upgrade of ATS to version 8.0.7-1wm5 - T249335
• 13:31 moritzm: rebooting deneb for kernel update
• 13:30 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:30 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 13:24 dzahn@cumin1001: START - Cookbook sre.ganeti.makevm
• 13:24 dzahn@cumin1001: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99)
• 13:24 dzahn@cumin1001: START - Cookbook sre.ganeti.makevm
• 13:08 hashar@deploy1001: Started scap: testwikis wikis to 1.35.0-wmf.32
• 13:05 hashar@deploy1001: Pruned MediaWiki: 1.35.0-wmf.28 (duration: 23m 47s)
• 12:37 moritzm: installing iputils update from Buster point release
• 12:08 hashar: Cutting branch 1.35.0-wmf.32 # T249964
• 12:08 gehel: restart blazegraph + updater on wdqs2002 - JVM upgrade
• 11:56 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' .
• 11:20 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 11:17 kormat@cumin1001: START - Cookbook sre.hosts.downtime
• 10:55 vgutierrez: upgrade trafficserver to version 8.0.7-1wm5 on cp5011 - T249335
• 10:54 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1)
• 10:53 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
• 10:53 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1)
• 10:52 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
• 10:43 kormat: reimaging pc2010 to buster T252182
• 10:30 vgutierrez: upgrade trafficserver to version 8.0.7-1wm5 on cp4032 - T249335
• 10:30 ema: rolling thumbor upgrade to 2.6-1+deb10u1 T226707
• 10:19 ema: repool thumbor2001 with upgraded python-thumbor-wikimedia
• 10:13 ema: thumbor2001: upgrade python-thumbor-wikimedia to 2.6-1+deb10u1
• 10:04 godog: update compiler facts
• 09:35 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:35 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 09:34 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99)
• 09:34 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
• 09:31 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99)
• 09:31 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
• 09:31 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99)
• 09:31 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
• 09:29 filippo@cumin1001: conftool action : set/pooled=yes:weight=100; selector: cluster=thanos
• 09:07 moritzm: rebooting contint2001 for kernel update
• 09:07 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 09:06 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 07:46 godog: reboot thanos hosts for kernel upgrade
• 07:41 filippo@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 07:41 filippo@cumin1001: START - Cookbook sre.hosts.downtime
• 07:29 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 07:29 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 07:12 moritzm: rebooting the IDP hosts, SSO sessions will need to be renewed
• 07:04 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 07:04 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 06:56 vgutierrez: upload trafficserver 8.0.7-1wm4 to apt.wm.o (buster) - T242767 T249335
• 05:29 marostegui: Restart docker-report-releng on deneb
• 05:03 marostegui@cumin1001: dbctl commit (dc=all): 'Set s4 as read-only=off for maintenance T251502', diff saved to https://phabricator.wikimedia.org/P11180 and previous config saved to /var/cache/conftool/dbconfig/20200512-050339-marostegui.json
• 05:00 marostegui@cumin1001: dbctl commit (dc=all): 'Set s4 as read-only for maintenance T251502', diff saved to https://phabricator.wikimedia.org/P11179 and previous config saved to /var/cache/conftool/dbconfig/20200512-050054-marostegui.json
• 04:46 marostegui: Stop mysql on labsdb1011 to transfer its content - T249188
• 02:14 pt1979@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 02:12 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
• 01:45 pt1979@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 01:43 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
• 01:16 pt1979@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 01:14 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
• 00:37 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 00:34 pt1979@cumin2001: START - Cookbook sre.hosts.downtime

2020-05-11

• 21:00 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.cf (exit_code=0)
• 21:00 ayounsi@cumin1001: START - Cookbook sre.network.cf
• 20:19 cdanis@cumin1001: END (PASS) - Cookbook sre.network.cf (exit_code=0)
• 20:19 cdanis@cumin1001: START - Cookbook sre.network.cf
• 19:08 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 19:05 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 19:03 Zoranzoki21: T235414 is wrong task number, T235415 is correct
• 19:02 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add *.bollywoodhungama.in and *.britishmuseum.org to $wgCopyUploadDomains (T235414, T251882) (duration: 00m 57s)
• 18:51 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Remove "Create a book" link on enwiki (T241683) (duration: 00m 57s)
• 18:44 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable modern Vector on officewiki, reveal preference on testwiki (T251285) (duration: 00m 58s)
• 18:42 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 18:40 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add tw-photometa.de to $wgCopyUploadsDomains (T252141) (duration: 00m 58s)
• 18:39 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 18:38 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 18:37 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 18:32 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 18:30 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 18:28 catrope@deploy1001: Synchronized dblists/mobilemainpagelegacy.dblist: Drop mainpage special casing for scowiki and itwiki (T252048, T252065) (duration: 00m 58s)
• 18:27 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 18:25 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 18:20 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 18:19 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 18:16 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 18:15 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 18:15 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 18:14 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 18:11 jforrester@deploy1001: Synchronized php-1.35.0-wmf.31/includes/Revision/RevisionStore.php: T252156 T212428 RevisionStore: fall back to master db if main slot is missing (duration: 00m 58s)
• 18:11 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 18:08 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 17:51 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 17:49 elukey@cumin1001: START - Cookbook sre.hosts.downtime
• 17:30 jforrester@deploy1001: Synchronized php-1.35.0-wmf.31/extensions/AbuseFilter/maintenance/updateVarDumps.php: updateVarDumps: wait for replication after each batch (duration: 00m 58s)
• 17:27 jforrester@deploy1001: Synchronized php-1.35.0-wmf.30/skins/Vector/includes/VectorTemplate.php: T251521 Correctly populate the language variants drop-down rather than breaking early (duration: 00m 59s)
• 17:24 jforrester@deploy1001: Synchronized php-1.35.0-wmf.31/skins/Vector/includes/VectorTemplate.php: T251521 Correctly populate the language variants drop-down rather than breaking early (duration: 00m 59s)
• 17:16 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 17:14 elukey@cumin1001: START - Cookbook sre.hosts.downtime
• 17:04 brennen@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.31
• 16:47 brennen@deploy1001: Synchronized php: group1 wikis to 1.35.0-wmf.31 (duration: 04m 43s)
• 16:42 mholloway-shell@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'wikifeeds' for release 'production' .
• 16:42 brennen@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.31
• 16:40 mholloway-shell@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'wikifeeds' for release 'production' .
• 16:34 brennen@deploy1001: rebuilt and synchronized wikiversions files: group0 wikis to 1.35.0-wmf.31
• 16:17 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' .
• 16:13 brennen@deploy1001: rebuilt and synchronized wikiversions files: mediawikiwiki to 1.35.0-wmf.31 (T249963) for testing T252179
• 16:10 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' .
• 16:06 brennen@deploy1001: Synchronized php-1.35.0-wmf.31/extensions/WikimediaMaintenance: Revert "Remove use of WikiPage::doEditContent" (duration: 01m 06s)
• 16:05 brennen@deploy1001: Synchronized php-1.35.0-wmf.31/extensions/UploadWizard: Revert "Remove use of WikiPage::doEditContent" (duration: 01m 06s)
• 16:04 hnowlan@deploy1001: Finished deploy [changeprop/deploy@82276cb]: Enabling consumption of purges topic (duration: 01m 58s)
• 16:04 brennen@deploy1001: Synchronized php-1.35.0-wmf.31/extensions/Babel: Revert "Remove use of WikiPage::doEditContent" (duration: 01m 07s)
• 16:03 brennen@deploy1001: Synchronized php-1.35.0-wmf.31/extensions/Translate: Revert "Remove uses of WikiPage::doEditContent" (duration: 01m 08s)
• 16:02 hnowlan@deploy1001: Started deploy [changeprop/deploy@82276cb]: Enabling consumption of purges topic
• 15:56 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:56 jmm@cumin2001: START - Cookbook sre.hosts.downtime
• 15:54 hnowlan@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop' for release 'production' .
• 15:52 hnowlan@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'changeprop' for release 'production' .
• 15:52 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' .
• 15:49 cdanis@cumin1001: conftool action : set/ttl=300; selector: dnsdisc=eventgate-analytics.*
• 15:45 hnowlan@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'changeprop' for release 'production' .
• 15:42 brennen: syncing backports to 1.35.0-wmf.31 (T249963) for T252179
• 15:42 hnowlan@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'changeprop' for release 'production' .
• 15:01 moritzm: installing puma security updates
• 14:29 mholloway-shell@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' .
• 13:44 vgutierrez: upgrade ATS to 8.0.7-1wm4 in cp4032 - T249335
• 13:36 hashar: Rolling back CI system switch to previous known state # T224591
• 13:20 marostegui: Upgrade mysql package on s4 master in preparation for tomorrow's maintenance T251502
• 12:50 hashar: Pointing CI Jenkins to contint2001 Gearman server T224591
• 12:46 mutante: contint2001 - chown -R jenkins-slave:jenkins-slave /srv/.git
• 12:45 mutante: contint1001 - rsync -avz --delete /srv/.git/ rsync://contint2001.wikimedia.org/ci--srv/.git/
• 12:43 mutante: contint1001 - rsync -avz --delete /srv/.git/ rsync://contint2001.wikimedia.org/ci--srv-/org/.git/
• 12:40 mutante: contint1001 - rsync -avz --delete /srv/org/wikimedia/integration/ rsync://contint2001.wikimedia.org/ci--srv-/org/wikimedia/integration/
• 12:24 mutante: contint2001 - find /var/lib/jenkins/ -group bacula -exec chown jenkins:jenkins {} \;
• 12:21 mutante: contint2001 - find /var/lib/jenkins/ -user statsite -exec chown jenkins {} \;
• 12:19 mutante: contint2001 - chown -R jenkins:jenkins /srv/jenkins/*
• 12:19 mutante: contint1001 - rsync -avz --delete /srv/jenkins/ rsync://contint2001.wikimedia.org/ci--srv-/jenkins/
• 12:17 mutante: contint1001 - rsync -avz --delete /var/lib/jenkins/ rsync://contint2001.wikimedia.org/ci--var-lib-jenkins-
• 12:14 hashar: shutting down Zuul and Jenkins for system switch # T224591
• 12:02 jynus@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 11:59 jynus@cumin1001: START - Cookbook sre.hosts.downtime
• 11:47 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 11:45 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 11:45 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 11:43 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 11:43 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 11:42 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 11:32 Lucas_WMDE: EU SWAT done
• 11:30 lucaswerkmeister-wmde@deploy1001: Synchronized php-1.35.0-wmf.30/extensions/WikimediaEvents/: SWAT: Update Banner Interaction Schema (T250791, wmf.30) (duration: 01m 08s)
• 11:23 lucaswerkmeister-wmde@deploy1001: Synchronized php-1.35.0-wmf.31/extensions/WikimediaEvents/: SWAT: Update Banner Interaction Schema (T250791, wmf.31) (duration: 01m 07s)
• 11:14 kartik@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: 595478|Revert limit adjustment for Chinese translation with ContentTranslation (T252371) (duration: 01m 09s)
• 10:58 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (595498) (duration: 01m 06s)
• 10:56 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (595498) (duration: 01m 07s)
• 10:15 vgutierrez: upload trafficserver 8.0.7-1wm3 to apt.wm.o (buster) - T242767 T249335
• 09:44 mutante: contint2001 - find /var/lib/jenkins -user statsite -exec chown jenkins:jenkins {} \;
• 09:31 hashar: contint2001 started zuul-merger again (had permission issues in /var/lib/zuul )
• 09:07 mutante: contint1001 - rsync -avpz --delete /srv/jenkins/ rsync://contint2001.wikimedia.org/ci--srv-/jenkins/ (T224591)
• 09:05 mutante: contint2001 - mkdir /srv/jenkins
• 08:55 hashar: contint2001 stopping zuul-merger , permission problem
• 08:46 godog: bounce ferm on kubernetes1007 to resolve icinga UNKNOWN
• 08:40 mutante: rsyncing /var/lib/jenkins from contint1001 to contint2001 with --delete
• 08:32 mutante: rsynced data from contint1001 to contint2001 - pathes per T224591#6039192 for the migration later today
• 08:30 ema: cp3050: upgrade atskafka to 0.6 T237993
• 08:30 _joe_: removing the iptables DROP rule on mc1020 T251378
• 07:54 moritzm: installing squid security updates
• 07:21 moritzm: updated buster netboot images to 10.4 (updated to latest point release)
• 07:09 _joe_: dropping requests to mc1020 via a firewall rule T251378
• 06:04 elukey: restart wikimedia-discovery-golden on stat1007 - apparenlty killed by no memory left to allocate on the system

2020-05-10

• 12:18 marostegui: Start event scheduler on db1115 after a massive delete - T252324
• 11:05 marostegui: Stop event scheduler on db1115 to perform a massive delete - T252324
• 10:27 dcausse: restarting blazgraph on wdqs1004: T242453
• 09:56 marostegui: Change scaling_governor from powersave to performance on db1115 - T252324
• 09:25 marostegui: Stop MySQL and restart db1115 - T252324
• 08:50 marostegui: Restart mysql on db1115 to change buffer pool size from 20GB to 40GB T252324 (
• 08:44 elukey: Power cycle analytics1052 after eno1 issue
• 08:01 marostegui: Disable unused events like %_schema T252324 T231185
• 07:11 marostegui: Restart mysql on db1115 T231185
• 07:11 marostegui: Truncate tendril. processlist_query_log T231185

2020-05-08

• 21:45 bstorm_: cleaned up wb_terms_no_longer_updated view for testwikidatawiki and testcommonswiki on labsdb1010 T251598
• 21:45 bstorm_: cleaned up wb_terms_no_longer_updated view on labsdb1012 T251598
• 21:33 bstorm_: cleaning up wb_terms_no_longer_updated view on labsdb1009 T251598
• 21:06 ottomata: running prefered replica election for kafka-jumbo to get preferred leaders back after reboot of broker earlier today - T252203
• 19:16 jhuneidi@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'blubberoid' for release 'production' .
• 19:12 jhuneidi@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'blubberoid' for release 'production' .
• 19:07 jhuneidi@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'blubberoid' for release 'staging' .
• 18:12 andrewbogott: reprepro copy buster-wikimedia stretch-wikimedia prometheus-openstack-exporter for T252121
• 17:59 marostegui: Extend /srv by 500G on labsdb1011 T249188
• 16:55 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 16:53 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
• 16:51 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 16:48 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 16:39 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 16:37 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
• 16:14 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 16:12 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
• 15:43 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 15:41 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
• 15:36 ottomata: starting kafka broker on kafka-jumbo1006, same issue on other brokers when they are leaders of offending partitions - T252203
• 15:31 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 15:28 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
• 15:27 ottomata: stopping kafka broker on kafka-jumbo1006 to investigate camus import failures - T252203
• 14:50 otto@deploy1001: Finished deploy [analytics/refinery@4a2c530]: fix for camus wrapper, deploy to an-launcher1001 only (duration: 00m 03s)
• 14:50 otto@deploy1001: Started deploy [analytics/refinery@4a2c530]: fix for camus wrapper, deploy to an-launcher1001 only
• 14:05 akosiaris: T243106 undo experiment with DROP iptable rules this time around. Use mw1331, mw1348
• 13:22 vgutierrez: rolling restart of ats-tls on eqiad, codfw, ulsfo and eqsin - T249335
• 13:20 akosiaris: T243106 redo experiment with DROP iptable rules this time around. Use mw1331, mw1348
• 13:16 akosiaris: T243106 undo experiment with REJECT, DROP iptable rules now that we have envoy in the middle. Use mw1331, mw1348. Experiment done successfully, no issues to the infrastructure.
• 12:49 akosiaris: T243106 redo experiment with REJECT, DROP iptable rules now that we have envoy in the middle. Use mw1331, mw1348
• 12:49 akosiaris: T243106 redo experiment with REJECT, DROP iptable rules now that we have envoy in the middle
• 11:49 hnowlan: restarting cassandra on restbase2009 for java updates
• 11:28 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 11:25 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 11:08 akosiaris: repool eqiad eventgate-analytics. Test concluded
• 11:08 akosiaris@cumin1001: conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=eventgate-analytics
• 09:54 mutante: disabling puppet on puppetmasters temporarily to switch them carefully to use httpd module and not apache module which we want to get rid of
• 09:52 akosiaris: depool eqiad eventgate-analytics for a test involving reinitializing the eqiad kubernetes cluster
• 09:52 akosiaris@cumin1001: conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=eventgate-analytics
• 09:51 akosiaris@cumin1001: conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=eventgate-analytics
• 09:45 oblivian@puppetmaster1001: conftool action : set/ttl=10; selector: dnsdisc=eventgate-analytics.*
• 08:20 vgutierrez: rolling restart of ats-tls on esams - T249335
• 07:19 vgutierrez: ats-tls restart on cp3050 and cp3052 (max_connections_active_in experiment) - T249335
• 07:07 mutante: phabricator rmdir /var/run/phd/pid - empty and now unused
• 07:01 moritzm: installing php5 security updates
• 05:27 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 05:24 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 05:10 marostegui: Upgrade pc1010
• 00:30 brennen@deploy1001: rebuilt and synchronized wikiversions files: Revert all wikis except test to 1.35.0-wmf.30 for T252179
• 00:19 brennen: rolling 1.35.0-wmf.31 train back to group0 for T252179

2020-05-07

• 22:36 brennen@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.31
• 22:31 brennen@deploy1001: Synchronized php-1.35.0-wmf.31/extensions/Scribunto/includes/engines/LuaCommon/TitleLibrary.php: Handle RevisionAccessException with try-catch (T252156) (duration: 01m 08s)
• 20:40 ryankemper@cumin2001: END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99)
• 20:37 ryankemper@cumin2001: START - Cookbook sre.wdqs.data-transfer
• 20:10 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: wgEventLoggingStreamNames: set initial stream names, as yet unused - T238230 (duration: 01m 07s)
• 19:12 brennen@deploy1001: rebuilt and synchronized wikiversions files: Revert group2 wikis to 1.35.0-wmf.30
• 19:09 brennen: rolling 1.35.0-wmf.31 back to group1
• 19:09 XioNoX: Upgrade Routinator 3000 to 0.7.0 on rpki1001 - T252010
• 19:05 brennen@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.31
• 18:25 ppchelko@deploy1001: Finished deploy [changeprop/deploy@383fba5]: Enable both purging types T252142 (duration: 01m 17s)
• 18:23 ppchelko@deploy1001: Started deploy [changeprop/deploy@383fba5]: Enable both purging types T252142
• 18:15 Urbanecm: Morning SWAT done
• 18:14 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: 899c175: Update project icons to refreshed SVGs (T249047; part 2/2) (duration: 01m 06s)
• 18:13 urbanecm@deploy1001: Synchronized static/images/mobile/copyright/: SWAT: 899c175: Update project icons to refreshed SVGs (T249047; part 1/2) (duration: 01m 08s)
• 18:08 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: 54bd2f1: Add the investigate right to the checkuser group on testwiki (T251932) (duration: 01m 08s)
• 17:50 bsitzmann@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'wikifeeds' for release 'production' .
• 17:46 bsitzmann@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'wikifeeds' for release 'production' .
• 17:44 bsitzmann@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' .
• 17:44 otto@deploy1001: Finished deploy [analytics/refinery@4a2c530]: (no justification provided) (duration: 05m 31s)
• 17:38 otto@deploy1001: Started deploy [analytics/refinery@4a2c530]: (no justification provided)
• 17:18 ejegg: updated payments-wiki from afb84cc391 to dabba1804c
• 16:46 hnowlan@deploy1001: Finished deploy [changeprop/deploy@cd1386e]: Rollback varnish consumption (duration: 01m 05s)
• 16:45 hnowlan@deploy1001: Started deploy [changeprop/deploy@cd1386e]: Rollback varnish consumption
• 16:42 mvolz@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'citoid' for release 'production' .
• 16:36 mvolz@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'citoid' for release 'production' .
• 16:32 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 16:30 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 16:29 mvolz@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'citoid' for release 'staging' .
• 16:27 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 16:27 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 16:26 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 16:26 hnowlan@deploy1001: Finished deploy [changeprop/deploy@cd1386e]: Enabling consumption of purges topic (duration: 01m 45s)
• 16:25 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 16:24 hnowlan@deploy1001: Started deploy [changeprop/deploy@cd1386e]: Enabling consumption of purges topic
• 16:23 hnowlan@deploy1001: Finished deploy [changeprop/deploy@6c65779]: Enabling consumption of purges topic (duration: 00m 24s)
• 16:23 hnowlan@deploy1001: Started deploy [changeprop/deploy@6c65779]: Enabling consumption of purges topic
• 15:59 mvolz@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'citoid' for release 'production' .
• 15:51 mvolz@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'citoid' for release 'production' .
• 15:36 jforrester@deploy1001: Synchronized php-1.35.0-wmf.31/extensions/Collection/includes/Specials/SpecialCollection.php: T251460 Set skin on BaseTemplates if you are using getSkin (duration: 01m 08s)
• 15:28 mvolz@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'citoid' for release 'staging' .
• 15:27 vgutierrez: rolling restart of ats-tls on text@esams - T249335
• 15:26 jayme@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'wikifeeds' for release 'production' .
• 15:19 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:17 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:15 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 15:14 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 15:12 hnowlan@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop' for release 'production' .
• 15:09 hnowlan@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'changeprop' for release 'production' .
• 15:03 jayme@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'wikifeeds' for release 'production' .
• 14:59 moritzm: imported component/facter3 for stretch-wikimedia into "main"
• 14:57 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:55 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 14:54 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:51 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 14:50 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:50 moritzm: imported component/puppet5 for stretch-wikimedia into "main"
• 14:49 jayme@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' .
• 14:47 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 14:42 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 14:41 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:40 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' .
• 14:39 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 14:38 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 14:37 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 14:36 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 14:30 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' .
• 14:17 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' .
• 14:07 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' .
• 14:06 moritzm: imported component/facter3 for jessie-wikimedia into "main"
• 13:33 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:30 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:30 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 13:28 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 13:21 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:19 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 13:12 hashar@deploy1001: rebuilt and synchronized wikiversions files: (no justification provided)
• 13:04 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 13:04 jynus: disabling puppet on all db hosts to control deployment of new paging alert T172489
• 13:02 zpapierski@deploy1001: Finished deploy [wdqs/wdqs@94906d0]: Deploy WDQS 0.3.28 + GUI - new servers (duration: 02m 43s)
• 13:01 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 12:59 zpapierski@deploy1001: Started deploy [wdqs/wdqs@94906d0]: Deploy WDQS 0.3.28 + GUI - new servers
• 12:50 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 12:48 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 12:43 zpapierski@deploy1001: Finished deploy [wdqs/wdqs@94906d0]: Deploy WDQS 0.3.28 + GUI (duration: 16m 20s)
• 12:27 zpapierski@deploy1001: Started deploy [wdqs/wdqs@94906d0]: Deploy WDQS 0.3.28 + GUI
• 12:13 addshore@deploy1001: Synchronized php-1.35.0-wmf.31/extensions/Wikibase: gerrit:594920 T252079 Revert "Move prefetching-term-lookup-callback service wiring" (duration: 01m 12s)
• 12:12 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 12:10 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 11:55 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
• 11:53 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
• 11:33 moritzm: imported component/puppet5 for jessie-wikimedia into "main"
• 11:31 jbond42: enable ferm-status script https://gerrit.wikimedia.org/r/c/operations/puppet/+/576102
• 11:10 matthiasmullie: EU swat done
• 11:07 mlitn@deploy1001: Synchronized php-1.35.0-wmf.31/extensions/WikibaseMediaInfo/: [MediaInfo] Add dummy concept chips without thumbnail (duration: 01m 09s)
• 10:07 moritzm: installing Java security updates on restbase/sessionstore
• 09:11 elukey: roll restart cassandra on aqs1005 to pick up new openjdk upgrades (canary)
• 08:32 moritzm: upgrading restbase-dev to latest OpenJDK security update
• 08:06 jynus: setting pc2007, pc2009 as read-write
• 07:44 godog: further decrease weight for ms-be10[678] - T252008
• 05:49 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 05:45 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
• 05:33 elukey: restart hadoop yarn nodemanager on analytics1071
• 05:22 marostegui: Reimage db2078
• 05:04 marostegui@cumin1001: dbctl commit (dc=all): 'Set s3 and s7 as read-only=off for maintenance T251158', diff saved to https://phabricator.wikimedia.org/P11167 and previous config saved to /var/cache/conftool/dbconfig/20200507-050419-marostegui.json
• 05:00 marostegui@cumin1001: dbctl commit (dc=all): 'Set s3 and s7 as read-only for maintenance T251158', diff saved to https://phabricator.wikimedia.org/P11166 and previous config saved to /var/cache/conftool/dbconfig/20200507-050046-marostegui.json
• 02:56 brennen@deploy1001: rebuilt and synchronized wikiversions files: Revert group1 wikis to 1.35.0-wmf.30 for T252079
• 02:55 brennen: reverting group1 to 1.35.0-wmf.30 for T252079
• 00:12 ryankemper@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)

2020-05-06

• 23:59 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Disable GrowthExperiments guidance on testwiki (duration: 01m 07s)
• 23:18 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable password-reset-update on Wikipedias (T245791) (duration: 01m 07s)
• 22:22 brennen@deploy1001: Synchronized php-1.35.0-wmf.31/includes/revisionlist/RevisionItem.php: RevisionItem: Fix providing timestamp in getRevisionLink (duration: 01m 09s)
• 21:45 andrewbogott: updating puppet compiler facts
• 21:07 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-transfer
• 21:05 ryankemper@cumin1001: END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99)
• 21:04 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-transfer
• 20:35 ejegg: updated Fundraising CiviCRM from b15b2cfbb5 to cfb6101e39
• 19:08 brennen@deploy1001: Synchronized php: group1 wikis to 1.35.0-wmf.31 (duration: 01m 08s)
• 19:07 brennen@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.31
• 19:03 brennen: CORRECTION: 1.35.0-wmf.31 train unblocked (T249963), rolling forward to group1
• 19:03 brennen: 1.35.0-wmf.31 train unblocked (T249963), rolling forward to group0
• 18:58 twentyafterfour@deploy1001: Synchronized php-1.35.0-wmf.31/includes/specials/pagers/DeletedContribsPager.php: deploy https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/594778/ fixes UBN T252052 (duration: 01m 09s)
• 18:54 volans: upgraded spicerack to spicerack_0.0.34-1_amd64.deb on cumin[12]001
• 18:45 volans: uploaded spicerack_0.0.34-1_amd64.deb to apt.wikimedia.org stretch-wikimedia
• 18:44 volans@deploy1001: Finished deploy [homer/deploy@8224f0a]: Release v0.2.2 (duration: 00m 18s)
• 18:44 volans@deploy1001: Started deploy [homer/deploy@8224f0a]: Release v0.2.2
• 18:28 twentyafterfour@deploy1001: Synchronized php-1.35.0-wmf.31/includes/specials/pagers/DeletedContribsPager.php: sync https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/594768/ fixes T252043 (duration: 01m 08s)
• 17:34 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' .
• 17:31 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' .
• 17:12 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' .
• 17:06 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' .
• 17:05 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' .
• 16:21 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' .
• 15:41 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' .
• 15:27 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' .
• 13:36 mutante: puppetmaster - revoking cert for webserver-misc-apps , recreating it with static-codereview.wikimedia.org as addiitonal SAN (T243056)
• 13:32 hashar: Restarting CI Jenkins
• 13:27 mutante: puppetmaster - revoking cert for webserver-misc-static, not used anymore, merged into webserver-misc-apps
• 13:27 moritzm: installing graphicsmagick security updates
• 13:26 XioNoX: Upgrade Routinator 3000 to 0.7.0 on rpki2001 - T252010
• 13:25 XioNoX: add routinator 3000 0.7.0 to buster-wikimedia - T252010
• 13:19 ema: cp: upgrade purged to v0.10
• 13:08 godog: start swift decom ms-be101[678] - T252008
• 11:22 kart_: EU SWAT done.
• 11:13 kartik@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: 594668|Enable ContentTranslation in Armenian WP as a default tool (T249229) (duration: 01m 08s)
• 10:27 ema: cp2027: test purged v0.10
• 10:20 moritzm: restarting apache on dbmonitor/grafana/miscweb/graphite/netmon to pick up openldap update
• 10:00 moritzm: installing remaining openldap security updates (client-side libs, tools)
• 09:52 jbond42: enable rember me feature of CAS
• 09:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1121 and remove db1103:3314 from vslow in s4', diff saved to https://phabricator.wikimedia.org/P11159 and previous config saved to /var/cache/conftool/dbconfig/20200506-093940-marostegui.json
• 09:12 marostegui: Upgrade package on s3 and s7 master (db1123 and db1086) in preparation for tomorrow's restart - T251158
• 08:56 jbond42: restarting ps1-a4-eqiad.mgmt.eqiad.wmnet.
• 08:53 jynus: kill FTWRL on db2101
• 08:43 oblivian@deploy1001: Synchronized wmf-config/CommonSettings.php: Reverting change on mw1407 T99740 (duration: 01m 16s)
• 08:02 _joe_: restarted php-fpm with tweaked parameters on mw1407, now briefly pooling for traffic (T99740)
• 07:38 kormat@cumin1001: dbctl commit (dc=all): 'Set es1023 (es5 master) to 0 weight after reimaging es1024 T250666', diff saved to https://phabricator.wikimedia.org/P11158 and previous config saved to /var/cache/conftool/dbconfig/20200506-073856-kormat.json
• 07:32 vgutierrez: downgrade to ATS 8.0.7-1wm3 on cp4026, cp4031, cp5006 and cp5011
• 06:00 elukey: powercycle analytics1060 - host stuck - T251973
• 05:03 marostegui@cumin1001: dbctl commit (dc=all): 'Pool db1103:3314 in vslow on s4 while db1121 is out T250055', diff saved to https://phabricator.wikimedia.org/P11157 and previous config saved to /var/cache/conftool/dbconfig/20200506-050340-marostegui.json
• 05:02 marostegui: Deploy schema change on db1121

2020-05-05

• 23:44 catrope@deploy1001: Synchronized wmf-config/flaggedrevs.php: Restore the reviewer group on fawiki (T249643) (duration: 01m 06s)
• 23:22 crusnov@deploy1001: Finished deploy [netbox/deploy@03cc2dd]: Netbox upgrade to 2.8.1 (part3) (duration: 00m 11s)
• 23:22 crusnov@deploy1001: Started deploy [netbox/deploy@03cc2dd]: Netbox upgrade to 2.8.1 (part3)
• 23:22 crusnov@deploy1001: Finished deploy [netbox/deploy@03cc2dd]: Netbox upgrade to 2.8.1 (part1) (duration: 01m 14s)
• 23:21 crusnov@deploy1001: Started deploy [netbox/deploy@03cc2dd]: Netbox upgrade to 2.8.1 (part1)
• 23:21 crusnov@deploy1001: Finished deploy [netbox/deploy@03cc2dd]: Netbox upgrade to 2.8.1 (part1) (duration: 01m 20s)
• 23:20 crusnov@deploy1001: Started deploy [netbox/deploy@03cc2dd]: Netbox upgrade to 2.8.1 (part1)
• 22:00 reedy@deploy1001: Synchronized php-1.35.0-wmf.31/includes/parser/CoreParserFunctions.php: T251952 take 2 (duration: 01m 06s)
• 21:57 reedy@deploy1001: Synchronized php-1.35.0-wmf.31/includes/parser/CoreParserFunctions.php: T251952 (duration: 01m 05s)
• 21:55 reedy@deploy1001: Synchronized php-1.35.0-wmf.31/includes/specials/SpecialNewpages.php: T251950 (duration: 01m 06s)
• 20:02 herron: added ryankemper to wmf and ops ldap groups T251572
• 19:38 mforns@deploy1001: Finished deploy [analytics/refinery@6868fc0] (thin): Regular analytics weekly train THIN [analytics/refinery@ebd624a5e4c88ac6983387d4603971f8a326ee7c] (duration: 00m 08s)
• 19:38 mforns@deploy1001: Started deploy [analytics/refinery@6868fc0] (thin): Regular analytics weekly train THIN [analytics/refinery@ebd624a5e4c88ac6983387d4603971f8a326ee7c]
• 19:38 mforns@deploy1001: Finished deploy [analytics/refinery@6868fc0]: Regular analytics weekly train (2nd try) [analytics/refinery@ebd624a5e4c88ac6983387d4603971f8a326ee7c] (duration: 25m 18s)
• 19:19 brennen@deploy1001: rebuilt and synchronized wikiversions files: group0 wikis to 1.35.0-wmf.31
• 19:13 mforns@deploy1001: Started deploy [analytics/refinery@6868fc0]: Regular analytics weekly train (2nd try) [analytics/refinery@ebd624a5e4c88ac6983387d4603971f8a326ee7c]
• 19:12 brennen@deploy1001: Finished scap: testwikis wikis to 1.35.0-wmf.31 (duration: 97m 23s)
• 19:02 brennen: train status: 1.35.0-wmf.31: presently pressing enter through scap-cdb-rebuild; at 8% (T249963, T223287)
• 18:39 cdanis: depool mw2221 for some manual testing
• 18:35 mforns@deploy1001: Finished deploy [analytics/refinery@ebd624a] (thin): Regular analytics weekly train THIN [analytics/refinery@ebd624a5e4c88ac6983387d4603971f8a326ee7c] (duration: 00m 09s)
• 18:35 mforns@deploy1001: Started deploy [analytics/refinery@ebd624a] (thin): Regular analytics weekly train THIN [analytics/refinery@ebd624a5e4c88ac6983387d4603971f8a326ee7c]
• 18:34 mforns@deploy1001: Finished deploy [analytics/refinery@ebd624a]: Regular analytics weekly train [analytics/refinery@ebd624a5e4c88ac6983387d4603971f8a326ee7c] (duration: 18m 54s)
• 18:15 mforns@deploy1001: Started deploy [analytics/refinery@ebd624a]: Regular analytics weekly train [analytics/refinery@ebd624a5e4c88ac6983387d4603971f8a326ee7c]
• 17:35 brennen@deploy1001: Started scap: testwikis wikis to 1.35.0-wmf.31
• 16:48 brennen: 1.35.0-wmf.31 was branched at 4d3fed3 for T249963
• 16:34 brennen: triggering branch cut for 1.35.0-wmf.31 (T249963) via https://releases-jenkins.wikimedia.org/job/MediaWiki%20Train%20Branch%20Cut/build?delay=0sec
• 16:18 brennen: notice: planning branch cut for 1.35.0-wmf.31 (T249963) at 16:30 UTC
• 15:47 cstone: SmashPig revision changed from 8c30ed7fe5 to cd1a49da5f
• 15:38 kormat@cumin1001: dbctl commit (dc=all): 'Repool es1024 to 100% after reimaging T250666', diff saved to https://phabricator.wikimedia.org/P11153 and previous config saved to /var/cache/conftool/dbconfig/20200505-153843-kormat.json
• 15:26 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:24 elukey@cumin1001: START - Cookbook sre.hosts.downtime
• 15:03 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:00 elukey@cumin1001: START - Cookbook sre.hosts.downtime
• 14:58 hnowlan@deploy1001: Finished deploy [changeprop/deploy@6c65779]: Enabling on_transclusion_update on k8s, disabling on scb (duration: 01m 31s)
• 14:56 hnowlan@deploy1001: Started deploy [changeprop/deploy@6c65779]: Enabling on_transclusion_update on k8s, disabling on scb
• 14:45 hnowlan@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop' for release 'production' .
• 14:43 hnowlan@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'changeprop' for release 'production' .
• 14:32 kormat@cumin1001: dbctl commit (dc=all): 'Repool es1024 to 75% after reimaging T250666', diff saved to https://phabricator.wikimedia.org/P11149 and previous config saved to /var/cache/conftool/dbconfig/20200505-143158-kormat.json
• 13:46 akosiaris: deploy cxserver chart 0.0.15 to staging, codfw, eqiad. T219921
• 13:45 akosiaris@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'cxserver' for release 'production' .
• 13:41 akosiaris@deploy1001: helmfile [CODFW] Ran 'sync' command on namespace 'cxserver' for release 'production' .
• 13:41 hashar: Updated Jenkins job https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler to have it defined in JJB # T97513
• 13:36 akosiaris@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'cxserver' for release 'staging' .
• 13:18 vgutierrez: upgrade ATS to version 8.1 () on cp4026, cp4032, cp5006 and cp5011
• 13:15 kormat@cumin1001: dbctl commit (dc=all): 'Repool es1024 to 50% after reimaging T250666', diff saved to https://phabricator.wikimedia.org/P11147 and previous config saved to /var/cache/conftool/dbconfig/20200505-131520-kormat.json
• 12:52 kormat@cumin1001: dbctl commit (dc=all): 'Repool es1024 at 25% after reimaging T250666', diff saved to https://phabricator.wikimedia.org/P11145 and previous config saved to /var/cache/conftool/dbconfig/20200505-125254-kormat.json
• 12:37 XioNoX: push pfw policy - T251769
• 12:07 jbond42: updating cas login page
• 12:07 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 12:05 kormat@cumin1001: START - Cookbook sre.hosts.downtime
• 12:03 moritzm: rolling restart of apache on puppetboard* to pick up OpenLDAP update
• 11:47 moritzm: rolling restart of apache on kibana hosts
• 11:41 mutante: LDAP - added eamedia to wmf group (T251358)
• 11:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1087 T248086', diff saved to https://phabricator.wikimedia.org/P11144 and previous config saved to /var/cache/conftool/dbconfig/20200505-113152-marostegui.json
• 11:31 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1087 T248086', diff saved to https://phabricator.wikimedia.org/P11143 and previous config saved to /var/cache/conftool/dbconfig/20200505-113100-marostegui.json
• 11:30 marostegui: Drop T248086_wb_terms table on labsdb hosts - T248086
• 11:26 moritzm: rolling restart of apache/FPM on mw1261-mw1265
• 11:22 kart_: EU SWAT done.
• 11:09 kartik@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: 592479|Adjust ContentTranslation MT threshold for Chinese WP to 70% (T246383) (duration: 01m 01s)
• 11:01 moritzm: installing remaining openldap security updates (client-side libs, tools)
• 11:00 kormat@cumin1001: dbctl commit (dc=all): 'Depool es1024 for reimaging, add es1023 (master) for reading in the meantime T250666', diff saved to https://phabricator.wikimedia.org/P11141 and previous config saved to /var/cache/conftool/dbconfig/20200505-110031-kormat.json
• 10:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1126 T248086', diff saved to https://phabricator.wikimedia.org/P11140 and previous config saved to /var/cache/conftool/dbconfig/20200505-104540-marostegui.json
• 10:44 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1126 T248086', diff saved to https://phabricator.wikimedia.org/P11139 and previous config saved to /var/cache/conftool/dbconfig/20200505-104441-marostegui.json
• 10:33 hnowlan@deploy1001: helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' .
• 10:23 arturo: copy prometheus-rabbitmq-exporter v0.4 from stretch-wikimedia to buster-wikimedia in apt1001 (T251660)
• 10:18 arturo: copy prometheus-pdns-exporter v0.5.1 from stretch-wikimedia to buster-wikimedia in apt1001 (T251575)
• 10:16 mutante: temp disabling puppet on all ganeti hosts to carefully deploy change related to rapi cert location
• 09:37 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
• 09:36 moritzm: removing boron.eqiad.wmnet
• 09:36 jmm@cumin2001: START - Cookbook sre.hosts.decommission
• 09:03 gehel: restarting wdqs updater on all servers
• 08:53 moritzm: installing Java security updates on releases*
• 08:44 kormat: reimaging es1024 to buster T250666
• 08:27 ema: cp2028 and cp2030 (both upload): varnish-fe restart to clear cache and evaluate 'exp' admission policy T144187 T249809
• 08:26 moritzm: upgrading slapd on serpens/seaborgium
• 08:19 ema: cp2027 and cp2029 (both text): varnish-fe restart to clear cache and evaluate 'exp' admission policy T144187 T249809
• 08:08 moritzm: installing Java security updates on notebook/stat hosts
• 07:54 gehel@deploy1001: Finished deploy [wdqs/wdqs@d37a059]: rollback wdqs to v 0.3.22 (duration: 04m 18s)
• 07:50 gehel@deploy1001: Started deploy [wdqs/wdqs@d37a059]: rollback wdqs to v 0.3.22
• 07:36 zpapierski@deploy1001: Started deploy [wdqs/wdqs@d37a059]: fix for the duplicated jars
• 06:59 addshore: depool wdqs1006 heavy lag
• 05:23 marostegui@cumin1001: dbctl commit (dc=all): 'Set s5 and s6 as read-only=off for maintenance T251154', diff saved to https://phabricator.wikimedia.org/P11133 and previous config saved to /var/cache/conftool/dbconfig/20200505-052334-marostegui.json
• 05:21 marostegui@cumin1001: dbctl commit (dc=all): 'Set s5 and s6 as read-only for maintenance T251154', diff saved to https://phabricator.wikimedia.org/P11132 and previous config saved to /var/cache/conftool/dbconfig/20200505-052058-marostegui.json
• 05:19 marostegui: Start s5 and s6 maintenance - T251154
• 04:39 marostegui: Restart mysql on tendril host: db1115 - T231769

2020-05-04

• 23:38 mstyles@deploy1001: Finished deploy [wdqs/wdqs@6518a8d]: v.0.3.26 (duration: 14m 39s)
• 23:37 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: Use namespaced EventBus classes (duration: 00m 57s)
• 23:35 reedy@deploy1001: Synchronized wmf-config/logging.php: Use namespaced EventBus classes (duration: 00m 56s)
• 23:33 reedy@deploy1001: Synchronized rpc/RunSingleJob.php: Use namespaced EventBus classes (duration: 00m 58s)
• 23:29 reedy@deploy1001: Synchronized wmf-config/logging.php: Replace AuthManagerStatsdHandler with WikimediaEventsAuthManagerStatsdHandler::class (duration: 00m 57s)
• 23:23 mstyles@deploy1001: Started deploy [wdqs/wdqs@6518a8d]: v.0.3.26
• 22:42 sbassett@deploy1001: Synchronized private/PrivateSettings.php: T251835: Restore dc752af (duration: 00m 57s)
• 22:16 eileen: process-control config revision is 2eb75f8dff
• 22:06 sbassett@deploy1001: Synchronized private/PrivateSettings.php: Partial mitigation for T250887 (duration: 00m 57s)
• 21:45 sbassett@deploy1001: Synchronized private/PrivateSettings.php: Revert partial mitigation for T250887 (duration: 00m 57s)
• 21:41 sbassett@deploy1001: Synchronized private/PrivateSettings.php: Deploy partial mitigation for T250887 (duration: 00m 57s)
• 18:20 dpifke@deploy1001: Finished deploy [performance/navtiming@239d359]: Deploy navtiming with new/updated Prometheus metrics - T249822, T238086 (duration: 00m 05s)
• 18:19 dpifke@deploy1001: Started deploy [performance/navtiming@239d359]: Deploy navtiming with new/updated Prometheus metrics - T249822, T238086
• 18:16 Urbanecm: Morning SWAT done
• 18:15 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: c04fbdd: Adding upload_by_url user right to all registered users on Commons (T251474) (duration: 00m 57s)
• 18:11 urbanecm@deploy1001: Synchronized php-1.35.0-wmf.30/extensions/DiscussionTools/includes/DiscussionToolsHooks.php: SWAT: b85fc16: Enable on all ExtraSignaturesNamespaces (T249036) (duration: 01m 00s)
• 18:07 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: 18c1efb: Load DiscussionTools on en.wiki (T249376) (duration: 00m 58s)
• 17:57 XioNoX: configure singtel interface on cr1-eqsin
• 17:36 volans: upgraded spicerack on cumin[12]001 to 0.0.33-1
• 17:02 joal@deploy1001: Finished deploy [analytics/refinery@2252f9a] (thin): Analytics hotfix deploy 2 THIN (sqoop) [2252f9a] (duration: 00m 09s)
• 17:02 joal@deploy1001: Started deploy [analytics/refinery@2252f9a] (thin): Analytics hotfix deploy 2 THIN (sqoop) [2252f9a]
• 17:01 joal@deploy1001: Finished deploy [analytics/refinery@2252f9a]: Analytics hotfix deploy 2 (sqoop) [2252f9a] (duration: 16m 45s)
• 16:44 joal@deploy1001: Started deploy [analytics/refinery@2252f9a]: Analytics hotfix deploy 2 (sqoop) [2252f9a]
• 16:08 liw@deploy1001: rebuilt and synchronized wikiversions files: group2 wikis to 1.35.0-wmf.30
• 15:59 liw@deploy1001: Synchronized php: group1 wikis to 1.35.0-wmf.30 (duration: 01m 05s)
• 15:58 liw@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.30
• 15:53 root@cumin1001: END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0)
• 15:53 root@cumin1001: Updating IPMI password on 1 hosts - root@cumin1001
• 15:53 root@cumin1001: START - Cookbook sre.hosts.ipmi-password-reset
• 15:52 root@cumin1001: END (FAIL) - Cookbook sre.hosts.ipmi-password-reset (exit_code=99)
• 15:52 root@cumin1001: START - Cookbook sre.hosts.ipmi-password-reset
• 15:47 kormat@cumin1001: dbctl commit (dc=all): 'Repool es2025 after reimaging T250666', diff saved to https://phabricator.wikimedia.org/P11128 and previous config saved to /var/cache/conftool/dbconfig/20200504-154747-kormat.json
• 15:45 jforrester@deploy1001: Synchronized php-1.35.0-wmf.30/includes/libs/rdbms/database/DatabaseMysqlBase.php: T251457 rdbms: don't treat lock() as a write operation (duration: 01m 04s)
• 15:43 jforrester@deploy1001: Synchronized php-1.35.0-wmf.30/resources/src/mediawiki.diff.styles/diff.less: T250393 Follow-up I07dd6f7: Fix font size in diff (duration: 01m 05s)
• 15:34 volans: uploaded spicerack_0.0.33-1_amd64.deb to apt.wikimedia.org stretch-wikimedia
• 15:26 volans: deploy1001: deleted old .hhvm.hhbc files (/home/*/.hhvm.hhbc) https://phabricator.wikimedia.org/P11127
• 15:23 volans: deploy1001: deleted old .hhvm.hhbc files moved from tin (/home/*/home-tin/.hhvm.hhbc) https://phabricator.wikimedia.org/P11126
• 15:12 kormat@cumin1001: dbctl commit (dc=all): 'Repool db1101:3318 fully after reimaging T250666', diff saved to https://phabricator.wikimedia.org/P11125 and previous config saved to /var/cache/conftool/dbconfig/20200504-151243-kormat.json
• 15:11 ppchelko@deploy1001: Finished deploy [restbase/deploy@74db57e]: Enable greek community wiki, fix analytics endpoints (duration: 14m 36s)
• 15:05 joal@deploy1001: Finished deploy [analytics/refinery@3396279] (thin): Analytics hotfix deploy (sqoop) THIN [3396279] (duration: 00m 10s)
• 15:05 joal@deploy1001: Started deploy [analytics/refinery@3396279] (thin): Analytics hotfix deploy (sqoop) THIN [3396279]
• 15:05 joal@deploy1001: Finished deploy [analytics/refinery@3396279]: Analytics hotfix deploy (sqoop) [3396279] (duration: 15m 07s)
• 15:05 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 15:02 kormat@cumin1001: START - Cookbook sre.hosts.downtime
• 14:57 ppchelko@deploy1001: Started deploy [restbase/deploy@74db57e]: Enable greek community wiki, fix analytics endpoints
• 14:50 joal@deploy1001: Started deploy [analytics/refinery@3396279]: Analytics hotfix deploy (sqoop) [3396279]
• 14:19 kormat@cumin1001: dbctl commit (dc=all): 'Repool db1101:3317 fully and db1101:3318 to 75% after reimaging T250666', diff saved to https://phabricator.wikimedia.org/P11123 and previous config saved to /var/cache/conftool/dbconfig/20200504-141919-kormat.json
• 14:15 XioNoX: add static nat for fran1001 - T251763
• 13:50 kormat@cumin1001: dbctl commit (dc=all): 'Depool es2025 for reimaging T250666', diff saved to https://phabricator.wikimedia.org/P11122 and previous config saved to /var/cache/conftool/dbconfig/20200504-135039-kormat.json
• 13:34 kormat: reimaging es2025 to buster T250666
• 13:27 kormat@cumin1001: dbctl commit (dc=all): 'Repool db1101:3317 and db1101:3318 some more after reimaging T250666', diff saved to https://phabricator.wikimedia.org/P11121 and previous config saved to /var/cache/conftool/dbconfig/20200504-132744-kormat.json
• 13:02 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T248664 Stop setting legacy wmgWikibase(Repo/Client)Repositories for TEST wikis (duration: 01m 06s)
• 12:47 kormat@cumin1001: dbctl commit (dc=all): 'Repool db1101:3317 and db1101:3318 after reimaging T250666', diff saved to https://phabricator.wikimedia.org/P11120 and previous config saved to /var/cache/conftool/dbconfig/20200504-124659-kormat.json
• 12:10 marostegui: Temporary enable slow query log on db1099:3311 - T206103
• 12:09 Amir1: EU SWAT is done
• 11:53 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Increase wmgMemoryLimit from 660MB to 666MB (duration: 01m 06s)
• 11:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1099:3311 T206103 after removing tmp_2 index', diff saved to https://phabricator.wikimedia.org/P11119 and previous config saved to /var/cache/conftool/dbconfig/20200504-114727-marostegui.json
• 11:46 tgr@deploy1001: Synchronized php-1.35.0-wmf.30/extensions/GrowthExperiments/modules/helppanel/ext.growthExperiments.HelpPanel.cta.js: SWAT: Help panel: Check if guidance feature flag is set before loading mobile peek (T251589) (duration: 01m 06s)
• 11:46 marostegui: Remove index tmp_2 from recentchanges on db1099:3311 T206103
• 11:45 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1099:3311 T206103 to remove tmp_2 index', diff saved to https://phabricator.wikimedia.org/P11118 and previous config saved to /var/cache/conftool/dbconfig/20200504-114539-marostegui.json
• 11:43 tgr@deploy1001: Synchronized php-1.35.0-wmf.28/extensions/GrowthExperiments/modules/helppanel/ext.growthExperiments.HelpPanel.cta.js: SWAT: Help panel: Check if guidance feature flag is set before loading mobile peek (T251589) (duration: 01m 10s)
• 11:38 jbond42: rebooting ps1-a7-codfw.mgmt.eqiad.wmnet.
• 11:30 jbond42: rebooting ps1-a7-codfw.mgmt.eqiad.wmnet.
• 11:30 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: 4d00236: Enable cross-project search on frwikibooks (T251683) (duration: 01m 05s)
• 11:25 Urbanecm: Purge https://en.wikipedia.org/static/images/project-logos/elwikiversity*.png (T251050)
• 11:24 urbanecm@deploy1001: Synchronized static/images/project-logos/: SWAT: 64556ba: Correct typo in Greek Wikiversity logo (T248391) (duration: 01m 06s)
• 11:20 Urbanecm: Purge https://en.wikipedia.org/static/images/project-logos/jvwiki*.png (T251050)
• 11:20 urbanecm@deploy1001: Synchronized static/images/project-logos/: SWAT: 3b8c618: Update jvwiki logos (T251050) (duration: 01m 05s)
• 11:18 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: cc94ea7: Enable VisualEditor for more namespaces on vecwiki (T250419) (duration: 01m 07s)
• 10:49 arturo: update packages in buster-wikimedia | thirdparty/kubead-k8s-1-15 and thirdparty/kubeadm-k8s-1-16 (T250866)
• 10:44 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (563985) (duration: 01m 05s)
• 10:43 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (563985) (duration: 01m 29s)
• 10:39 vgutierrez: rolling upgrade of ATS to version 8.0.7-1wm3
• 10:36 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 10:33 kormat@cumin1001: START - Cookbook sre.hosts.downtime
• 10:30 arturo: running `aborrero@apt1001:~ $ sudo -i reprepro --delete clearvanished` to cleanup buster-wikimedia|thirdparty/kubeadm-k8s (T250866)
• 09:46 vgutierrez: upload trafficserver 8.0.7-1wm2 to apt.wm.o (buster)
• 09:22 kormat: reimaging db1101 to buster T250666
• 08:50 XioNoX: configure BGP peering with AS132203
• 08:20 godog: add 50G to prometheus-ops on prometheus100[34]
• 08:17 marostegui: Deploy schema change on s5 codfw - T251188
• 07:51 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1101:3317 and db1101:3318 for reimage', diff saved to https://phabricator.wikimedia.org/P11113 and previous config saved to /var/cache/conftool/dbconfig/20200504-075148-marostegui.json
• 07:31 marostegui: Drop unused flagged* tables from mediawikiwiki - T248298
• 07:26 moritzm: removed jmorgan from cn=wmf
• 07:24 marostegui: Install 10.1.43-2 on s5 (db110) and s6 (db1131) masters in preparations for tomorrow's restart - T251154
• 07:24 moritzm: removed Kerberos principal for lexnasser and jmorgan
• 07:23 moritzm: removed lexnasser from cn=nda
• 07:07 elukey: execute ifdown eno1; ifup eno1 on analytics1052 - interface neg speed flapping
• 06:41 elukey: upload prometheus-druid-exporter 0.8-1 to stretch-wikimedia

2020-05-03

• 22:52 Krinkle: scap pull mwmaint1002 and mw2001 for noc.wm.o. – https://gerrit.wikimedia.org/r/593929
• 22:42 Krinkle: scap pull mwmaint1002 and mw2001 for noc.wm.o. – https://gerrit.wikimedia.org/r/591459
• 21:37 bmansurov@deploy1001: Finished deploy [recommendation-api/deploy@0c68d62]: Update the recommendation API service (duration: 04m 22s)
• 21:32 bmansurov@deploy1001: Started deploy [recommendation-api/deploy@0c68d62]: Update the recommendation API service

2020-05-02

• 07:49 oblivian@cumin1001: conftool action : set/pooled=yes; selector: name=mw13(49|5[0-9]|6[0-2])\.eqiad\.wmnet
• 07:08 XioNoX: asw2-d-eqiad> request virtual-chassis vc-port delete pic-slot 1 port 0 member 1
• 02:36 volker-e@deploy1001: Finished deploy [design/style-guide@f0d467b]: Deploy design/style-guide: (duration: 00m 07s)
• 02:36 volker-e@deploy1001: Started deploy [design/style-guide@f0d467b]: Deploy design/style-guide:

2020-05-01

• 19:56 rzl@cumin1001: conftool action : set/pooled=no; selector: name=mw13(5[6-9]|6[0-2]).eqiad.wmnet
• 18:57 gehel: restart blazegraph on wdqs1006 - T242453
• 14:23 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1104 - T232446', diff saved to https://phabricator.wikimedia.org/P11110 and previous config saved to /var/cache/conftool/dbconfig/20200501-142354-marostegui.json
• 14:18 hknust: holger@mwmaint1002 finished renameInvalidUsernames.php (fail) as part of T219279
• 14:06 marostegui@cumin1001: dbctl commit (dc=all): 'More traffic to db1104 - T232446', diff saved to https://phabricator.wikimedia.org/P11109 and previous config saved to /var/cache/conftool/dbconfig/20200501-140603-marostegui.json
• 13:47 marostegui@cumin1001: dbctl commit (dc=all): 'More traffic to db1104 - T232446', diff saved to https://phabricator.wikimedia.org/P11108 and previous config saved to /var/cache/conftool/dbconfig/20200501-134707-marostegui.json
• 13:28 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly warm up db1104 - T232446', diff saved to https://phabricator.wikimedia.org/P11107 and previous config saved to /var/cache/conftool/dbconfig/20200501-132804-marostegui.json
• 13:06 hknust: holger@mwmaint1002 Starting renameInvalidUsernames.php as part of T219279
• 13:01 vgutierrez: rolling restart of ats-tls in text@esams - T249335
• 12:24 mutante: mw230* - rolling restart of php-fpm - icinga warnings about opcache health in codfw
• 12:20 mutante: mw2376 - restarting php-fpm - icinga warnings about opcache health in codfw
• 12:07 mutante: notebook1004 - puppet was failed due to removal of jmorgan while one of his processes was still running. "change to absent failed.. user jmorgan currently used by process 29038". killing 29038, running puppet T251560
• 12:05 mutante: notebook1003 - puppet was failed due to removal of jmorgan while one of his processeswas still running. "change to absent failed.. user jmorgan currently used by porcess 3288". killing 3288, running puppet T251560
• 11:52 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
• 11:51 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
• 11:50 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
• 11:50 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
• 11:31 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 11:31 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 11:31 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
• 11:31 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
• 08:54 _joe_: depooled all servers in the app pool in rack D1
• 08:54 oblivian@cumin1001: conftool action : set/pooled=no:weight=30; selector: name=mw13(49|5[0-5])\.eqiad\.wmnet
• 08:50 oblivian@cumin1001: conftool action : set/weight=10; selector: name=mw13(49|5[0-5])\.eqiad\.wmnet
• 08:48 _joe_: repooling mw1407 with LCStoreStaticArray, increased opcache, puppet disabled
• 08:45 _joe_: repooling mw1409
• 08:39 _joe_: repool mw1352
• 08:37 _joe_: depooling mw1352
• 07:44 marostegui: Copy wikireplica dump from labsdb1009 to labsdb1011 - T249188
• 01:36 bmansurov@deploy1001: Finished deploy [recommendation-api/deploy@5f47cd7]: Update the recommendation API service (duration: 04m 33s)
• 01:32 bmansurov@deploy1001: Started deploy [recommendation-api/deploy@5f47cd7]: Update the recommendation API service

2000s

• Archive 1: 2004 Jun - 2004 Sep
• Archive 2: 2004 Oct - 2004 Nov
• Archive 3: 2004 Dec - 2005 Mar
• Archive 4: 2005 Apr - 2005 Jul
• Archive 5: 2005 Aug - 2005 Oct, with revision history 2004-06-23 to 2005-11-25
• Archive 6: 2005 Nov - 2006 Feb
• Archive 7: 2006 Mar - 2006 Jun
• Archive 8: 2006 Jul - 2006 Sep
• Archive 9: 2006 Oct - 2007 Jan, with revision history 2005-11-25 to 2007-02-21
• Archive 10: 2007 Feb - 2007 Jun
• Archive 11: 2007 Jul - 2007 Dec
• Archive 12: 2008 Jan - 2008 Jul
• Archive 12a: 2008 Aug
• Archive 12b: 2008 Sept
• Archive 13: 2008 Oct - 2009 Jun
• Archive 14: 2009 Jun - 2009 Dec

2010s

• Archive 15: 2010 Jan - 2010 Jun
• Archive 16: 2010 Jul - 2010 Oct
• Archive 17: 2010 Nov - 2010 Dec
• Archive 18: 2011 Jan - 2011 Jun
• Archive 19: 2011 Jul - 2011 Dec
• Archive 20: 2011 Dec - 2012 Jun, with revision history 2007-02-21 to 2012-03-27
• Archive 21: 2012 Jul - 2013 Jan
• Archive 22: 2013 Jan - 2013 Jul
• Archive 23: 2013 Aug - 2013 Dec
• Archive 24: 2014 Jan - 2014 Mar
• Archive 25: 2014 April - 2014 September
• Archive 26: 2014 October - 2014 December
• Archive 27: 2015 January - 2015 July
• Archive 28: 2015 August - 2015 December
• Archive 29: 2016 January - 2016 May
• Archive 30: 2016 June - 2016 August
• Archive 31: 2016 September - 2016 December
• Archive 32: 2017 January - 2017 July
• Archive 33: 2017 August - 2017 December
• Archive 34: 2018 January - 2018 April
• Archive 35: 2018 May - 2018 August
• Archive 36: 2018 September - 2018 December
• Archive 37: 2019 January - 2019 April
• Archive 38: 2019 May - 2019 August
• Archive 39: 2019 September - 2019 December

2020s

• Archive 40: 2020 January - 2020 April
• Archive 41: 2020 May - 2020 July
• Archive 42: 2020 August - 2020 November
• Archive 43: 2020 December
• Archive 44: 2021 January - 2021 April
• Archive 45: 2021 May - 2021 July
• Archive 46: 2021 August - 2021 October
• Archive 47: 2021 November - 2021 December
• Archive 48: 2022 January
• Archive 49: 2022 February
• Archive 50: 2022 March
• Archive 51: 2022 April 1-15
• Archive 52: 2022 April 16-30
• Archive 53: 2022 May
• Archive 54: 2022 June
• Archive 55: 2022 July
• Archive 56: 2022 August
• Archive 57: 2022 September
• Archive 58: 2022 October
• Archive 59: 2022 November 1-15
• Archive 60: 2022 November 16-30
• Archive 61: 2022 December
• Archive 62: 2023 January
• Archive 63: 2023 February
• Archive 64: 2023 March
• Archive 65: 2023 April
• Archive 66: 2023 May
• Archive 67: 2023 June
• Archive 68: 2023 July
• Archive 69: 2023 August 1-15
• Archive 70: 2023 August 16-31
• Archive 71: 2023 September
• Archive 72: 2023 October
• Archive 73: 2023 November
• Archive 74: 2023 December
• Archive 75: 2024 January
• Archive 76: 2024 February
• Archive 77: 2024 March