Server Admin Log/Archive 36

2018-12-31

16:27 ejegg: disabled SmashPig recurring donation charge job for investigation

2018-12-30

07:17 elukey: restart pdfrender on scb1002 (alarms flapping)
02:07 bawolff@deploy1001: Synchronized private/PrivateSettings.php: fine-tune antispam measure T212667 (duration: 00m 47s)

2018-12-29

14:53 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T212667 218371fd35 - Adjust mw.org abusefilter emergency shutoff threshold down to 0.3 (duration: 00m 46s)
14:30 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T212667 fe72284c Adjust account throttle limits (duration: 00m 46s)
13:30 bawolff@deploy1001: Synchronized private/PrivateSettings.php: T212667 - adjust spam block (duration: 00m 44s)
12:34 bawolff@deploy1001: Synchronized private/PrivateSettings.php: T212667 - make spam mitigation global (duration: 00m 49s)
off: manually recreated /dev/log symlink on kubernetes1001, restarting systemd-journald.socket didn't worked (this should fix cron-spam emails from the host every hour)
off: restarted pdfrender on scb1004
09:21 elukey: restart pdfrender on scb1004

2018-12-28

18:14 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 97446843a27 T212667 - Temp increase abusefilter emergency cutoff on mw.org to deal with spam attack (duration: 00m 46s)
18:02 bawolff@deploy1001: Synchronized private/PrivateSettings.php: T212667 - adjust account creation (duration: 00m 47s)
15:28 bawolff@deploy1001: Synchronized private/PrivateSettings.php: Attempt to adjust captcha settings for T212667 (duration: 00m 46s)

2018-12-27

19:31 mobrovac@deploy1001: Finished deploy [restbase/deploy@a67f38e]: Fix rate-limiter crash (with increased deploy delays), take #3 (duration: 25m 49s)
19:05 mobrovac@deploy1001: Started deploy [restbase/deploy@a67f38e]: Fix rate-limiter crash (with increased deploy delays), take #3
18:15 mobrovac@deploy1001: Started deploy [restbase/deploy@a67f38e]: Fix rate-limiter crash (with increased deploy delays), take #2
18:15 mobrovac@deploy1001: Finished deploy [restbase/deploy@a67f38e]: Fix rate-limiter crash (with increased deploy delays) - T212631 (duration: 11m 20s)
18:04 mobrovac@deploy1001: Started deploy [restbase/deploy@a67f38e]: Fix rate-limiter crash (with increased deploy delays) - T212631
18:04 mobrovac@deploy1001: deploy aborted: Fix rate-limiter crash (with increased deploy delays) - T212631 (duration: 00m 09s)
18:04 mobrovac@deploy1001: Started deploy [restbase/deploy@a67f38e]: Fix rate-limiter crash (with increased deploy delays) - T212631
18:03 mobrovac@deploy1001: deploy aborted: Fix rate-limiter crash - T212631 (duration: 13m 09s)
17:50 mobrovac@deploy1001: Started deploy [restbase/deploy@70c4752]: Fix rate-limiter crash - T212631
17:20 mobrovac@deploy1001: Finished deploy [restbase/deploy@ae7a537]: Fix rate-limiter crash - T212631 - deploy only on canary restbase1007 (duration: 04m 24s)
17:15 mobrovac@deploy1001: Started deploy [restbase/deploy@ae7a537]: Fix rate-limiter crash - T212631 - deploy only on canary restbase1007
16:00 cmjohnson1: powercycling frdb1001 for troubleshooting
15:21 bawolff: Updated interwiki cache https://biblio.wiki to https://wikilivres.org -> T212650
15:19 bawolff@deploy1001: Synchronized wmf-config/interwiki.php: Updating interwiki cache (duration: 03m 34s)

2018-12-26

16:37 godog: clear logstash persistent queue /var/lib/logstash on logstash100[789]
15:47 godog: roll-restart logstash on logstash100[789]
14:26 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=wtp1028.eqiad.wmnet
14:18 godog: powercycle wtp1028 - nothing on console
09:13 godog: swift eqiad-prod: more weight for ms-be10[44-50].eqiad.wmnet - T209618
07:20 marostegui: Fix replication db1124:3318

2018-12-24

05:38 _joe_: powercycling lvs3001

2018-12-23

11:53 marostegui: Fix replication on db1124:3318
10:25 ariel@deploy1001: Finished deploy [dumps/dumps@af74350]: python3 fixup for show runtimes (duration: 00m 05s)
10:25 ariel@deploy1001: Started deploy [dumps/dumps@af74350]: python3 fixup for show runtimes
08:53 apergos: restarted pdfrender on scb1003

2018-12-22

18:45 elukey: manually clean up of old log files on an-coord1001 (disk space issues)
15:58 godog: reboot ms-be2018, stuck on sd 0:1:0:1: rejecting I/O to offline device
10:21 cwd: re-enabled process-control
08:36 cwd: took down sidebar fr campaign
08:36 cwd: disabled process-control
01:00 AaronSchulz: Deployed b47e9fcfece99 to navtiming
00:59 aaron@deploy1001: Finished deploy [performance/navtiming@b47e9fc]: (no justification provided) (duration: 00m 05s)
00:59 aaron@deploy1001: Started deploy [performance/navtiming@b47e9fc]: (no justification provided)

2018-12-21

21:29 mutante: phab1002 - temp hack to unbreak phd / systemd alert, real fix will be phab deployment to new server
21:28 mutante: phab1002 - mkdir -p /srv/phab/libext/ava/src ; touch __phutil_library_init__.php
21:05 mutante: phab1002 - apt autoremove
21:04 mutante: phab1002 - removing all php related packages and letting puppet reinstall them
20:38 mutante: phab1002 - restart php-fpm, restart phd for testing. phd fails
18:50 mutante: [scb1001:~] $ sudo systemctl restart pdfrender
14:20 dcausse: elastic@eqiad deleting unused index enwiki_general_1537906513
13:24 moritzm: installing subversion updates from stretch point release
12:34 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: T211494 (duration: 00m 44s)
12:31 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T211494 (duration: 00m 45s)
12:10 moritzm: rebooting url downloaders to pick up SSBD-enabled QEMU
11:38 moritzm: rebooting debug proxies to pick up SSBD-enabled QEMU
09:58 ema@puppetmaster1001: conftool action : set/pooled=yes; selector: name=ms-fe2006.codfw.wmnet
09:57 ema: repool ms-fe2006 with old certs, test successful T212215#4839960
09:23 ema@puppetmaster1001: conftool action : set/pooled=no; selector: name=ms-fe2006.codfw.wmnet
09:22 ema: depool ms-fe2006 to test new TLS certs T212215
08:45 moritzm: upgrading nginx on sodium
08:41 moritzm: upgrading nginx on debug proxies
06:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Remove old comments T211973 (duration: 00m 46s)
00:52 twentyafterfour: SWAT Finished. See you all next year!
00:50 twentyafterfour@deploy1001: Synchronized php-1.33.0-wmf.9/extensions/MobileFrontend/: SWAT: sync https://gerrit.wikimedia.org/r/c/mediawiki/extensions/MobileFrontend/+/481026 (duration: 00m 48s)

2018-12-20

22:46 mutante: phab1001 / phabricator: upgraded nodejs package
22:39 mutante: phab1001 / phabricator: installing php5 package upgrades
19:29 kaldari@deploy1001: Synchronized wmf-config/Wikibase.php: syncing Wikibase for SWAT deployment (duration: 00m 45s)
19:27 kaldari@deploy1001: Synchronized wmf-config/InitialiseSettings.php: syncing InitialiseSettings for SWAT deployment (duration: 00m 46s)
19:03 elukey: restart hdfs namenode on an-master1002 with new heap settings (currently standby, 8->12G)
18:30 elukey: remove hdfs journalnode config+packages from analytics10(28|35) - not used anymore - T209929
18:29 elukey: restart hdfs namenode on an-master1001 with new heap settings (currently standby, 8->12G)
18:06 mutante: doc1001 - meged gerrit:480881 and then manually moved the entire /srv/org/wikimedia/doc/ structure into /srv/docroot/srv/org/wikimedia/ and deleted the old dirs T137890
17:53 arturo: updating puppet compiler facts: `PUPPET_COMPILER=compiler1002.puppet-diffs.eqiad.wmflabs modules/puppet_compiler/files/compiler-update-facts`
17:49 arturo: updating puppet compiler facts: `PUPPET_COMPILER=compiler1001.puppet-diffs.eqiad.wmflabs modules/puppet_compiler/files/compiler-update-facts`
17:05 XioNoX: add 208.80.155.88/29 to cloud-in4 term icmp - T207663
17:02 XioNoX: configure additional 208.80.155.88/29 IPs on cloud-instance-transport1-b-eqiad - T207663
16:59 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.9
16:51 zfilipin@deploy1001: Synchronized php-1.33.0-wmf.9/extensions/Wikibase: SWAT: Revert "Fail hard if an entity namespace is not configured." (T212427) (duration: 01m 17s)
16:31 elukey: remove two journal nodes from the Analytics hadoop cluster - T209929
14:53 moritzm: installing nodejs security updates on maps* (was tested via T211419)
14:41 moritzm: restarted etherpad for nodejs security updates
14:39 elukey: add two journal nodes to the Analytics Hadoop cluster - T209929
14:30 moritzm: installing libdap updates from stretch point release
14:26 moritzm: rearmed keyholder on netmon1002 after reboot
14:22 moritzm: rebooting netmon1002 for kernel security update
13:09 moritzm: installing xapian-core updates from stretch point release
12:53 arturo: T209616 installing cloudvirt1030, icinga downtime for 1 day
12:48 kartik@deploy1001: Finished deploy [cxserver/deploy@16f65cb]: Update cxserver to 803baa4 (T210581, T211889, T144467, T209473) (duration: 04m 42s)
12:47 moritzm: installing libxcursor security updates
12:43 kartik@deploy1001: Started deploy [cxserver/deploy@16f65cb]: Update cxserver to 803baa4 (T210581, T211889, T144467, T209473)
12:31 moritzm: installing fuse updates from stretch point release
12:09 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546, T202497) (duration: 00m 52s)
12:08 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546, T202497) (duration: 00m 53s)
12:02 moritzm: draining restbase1016 for eventual reboot for kernel security update
11:46 moritzm: draining restbase1015 for eventual reboot for kernel security update
11:34 moritzm: powercycling restbase1014, similar EFI ASSERT error to T212305
11:34 moritzm: powercycling restbase1014, similar EFI ASSEER error to T212305
11:21 moritzm: draining restbase1014 for eventual reboot for kernel security update
10:59 banyek: executing schema change on db1068 (s4 master) - T85757
10:58 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1121 after schema change - T85757 (duration: 00m 52s)
10:53 banyek: repooling db1121 after schema change T85757
10:45 moritzm: draining restbase1013 for eventual reboot for kernel security update
10:44 banyek: stopping replication on db1121 - T85757
10:40 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1121 for schema change - T85757 (duration: 00m 52s)
10:35 banyek: depooling db1121 for schema change T85757
10:33 banyek: executing schema change on dbstore1002 - T85757
10:22 banyek: executing schema change on db1102 - T85757
10:20 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1081 after schema change - T85757 (duration: 00m 51s)
10:19 moritzm: draining restbase1012 for eventual reboot for kernel security update
10:15 banyek: repooling db1081 after schema change T85757
10:10 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1081 for schema change - T85757 (duration: 00m 51s)
10:05 banyek: depooling db1081 for schema change T85757
10:00 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1103:3314 after schema change - T85757 (duration: 00m 52s)
09:56 banyek: repooling db1103:3314 after schema change T85757
09:49 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1103:3314 for schema change - T85757 (duration: 00m 51s)
09:46 banyek: depooling db1103:3314 for schema change T85757
09:45 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1097:3314 after schema change - T85757 (duration: 00m 51s)
09:41 legoktm@deploy1001: Synchronized php-1.33.0-wmf.8/includes/: T199540 (duration: 01m 06s)
09:40 legoktm@deploy1001: Synchronized php-1.33.0-wmf.9/includes/: T199540 (duration: 01m 14s)
09:39 banyek: repooling db1097:3314 after schema change T85757
09:35 gilles@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T187299 Increase ruwiki navtiming rate + frwiki survey rate (duration: 00m 52s)
09:29 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1097:3314 for schema change - T85757 (duration: 00m 52s)
09:23 banyek: depooling db1097:3314 for schema change T85757
08:06 elukey: roll restart of druid middlemanagers on druid* to pick up new port settings
07:17 marostegui: Re-start codfw s4 backup as the previous one failed
07:11 elukey: restart pdfrender on scb1002
07:10 elukey: restart rsyslog on lithium - in:imtcp stuck in recvfrom ms-be2047.codfw.wmnet - T199406
06:35 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2057 T212277 (duration: 00m 57s)
01:41 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@c7977a7]: Update mobileapps to 42c011e (duration: 04m 08s)
01:37 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@c7977a7]: Update mobileapps to 42c011e
01:09 eileen_: civicrm revision changed from 9d727e4708 to b33dcd3c94, config revision is 0f94a475b7
00:09 maxsem@deploy1001: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/479911/ (duration: 00m 53s)

2018-12-19

23:24 mutante: syncing facts from puppetmaster1001 to compiler1001/compiler1002
23:21 eileen_: civicrm revision changed from d5c3d5fd17 to 9d727e4708, config revision is 0f94a475b7
23:05 krinkle@deploy1001: Finished deploy [performance/navtiming@64e3f63]: (no justification provided) (duration: 00m 05s)
23:05 krinkle@deploy1001: Started deploy [performance/navtiming@64e3f63]: (no justification provided)
22:15 mutante: scb1003: systemctl restart pdfrender
20:44 hashar: 1.33.0-wmf.9 on group1 looks fine.
20:03 hashar@deploy1001: Synchronized php: group1 wikis to 1.33.0-wmf.9 (duration: 00m 51s)
20:02 hashar@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.9
19:33 XioNoX: Revert "Redirect eqsin/ulsfo caches to eqiad" - T210467
19:32 XioNoX: repool codfw - T210467
19:28 XioNoX: reactive BGP sessions to telia on cr1-codfw - T211715
19:07 chasemp: cp of files to ext drive on labstore1007
18:41 marostegui: Stop MySQL on db2057
18:28 XioNoX: replace `interface-range vlan-private1-b-eqiad member ge-6/0/*` with individual interfaces on asw2-b-eqiad
18:09 ejegg: updated fundraising CiviCRM from 8e18485697 to d5c3d5fd17
17:32 addshore: SWAT done
17:22 addshore@deploy1001: Synchronized wmf-config: Wikibase: wikidatawiki upsert idGenerator, T194299 (duration: 00m 52s)
17:13 addshore@deploy1001: Synchronized wmf-config: Wikibase: testwikidatawiki upsert idGenerator, T194299 (duration: 00m 52s)
17:05 XioNoX: remove 2nd port to AS8220 (cf. email to peering@)
17:04 addshore@deploy1001: Synchronized wmf-config: Wikibase: prepare to set $wgWBRepoSettings idGenerator, T194299 (duration: 00m 53s)
16:59 XioNoX: deactive BGP sessions to telia on cr1-codfw - T211715
16:58 fsero: DNS: updating wmnet to include new registries T212212
16:57 hashar@deploy1001: Synchronized php-1.33.0-wmf.9/extensions/ExtensionDistributor/includes/specials/SpecialBaseDistributor.php: Follow-up f686d348: No need for an <img> tag any more - T212217 (duration: 00m 52s)
16:52 XioNoX: codfw row D maintenance finished without issues - T210467
16:39 cmjohnson1: swapping disk in slot 2 on db1072
16:33 XioNoX: shutdown asw-d4-codfw - T210467
16:21 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting comment migration to new on group 2 (T166733) (duration: 00m 52s)
16:04 XioNoX: Redirect eqsin/ulsfo caches to eqiad - T210467
16:01 moritzm: installing php5 security updates on jessie
15:48 XioNoX: depool codfw - T210467
15:39 chasemp: various cp jobs on labstore1007 to ext media
15:33 chasemp: labstore1007 mount /dev/sde /mnt/T211327
15:13 moritzm: draining restbase1011 for eventual reboot for kernel security update
15:07 marostegui: Drop image_comment_temp from labswiki and labtestwiki - T209591
14:58 moritzm: draining restbase1010 for eventual reboot for kernel security update
14:42 moritzm: draining restbase1009 for eventual reboot for kernel security update
14:16 moritzm: draining restbase1008 for eventual reboot for kernel security update
14:03 moritzm: draining restbase1007 for eventual reboot for kernel security update
13:57 moritzm: installing nodejs updates on wtp*
13:36 marostegui: Correction from the previous !log: Rename table valid_tag on db1089 (s1) - T212254
13:35 marostegui: Rename table valid_tag on db1081 (s1) - T212254
13:31 marostegui: Drop image_comment_temp on s4 - T209591
12:46 marostegui: Drop image_comment_temp on s3 - T209591
12:24 zeljkof: EU SWAT finished
12:23 zfilipin@deploy1001: Synchronized php-1.33.0-wmf.9/extensions/Kartographer/: SWAT: Fix using at-ease functions in namespaced class (T212218) (duration: 00m 53s)
12:06 moritzm: rearmed keyholder after netmon2001 reboot
11:58 moritzm: rebooting netmon2001 for kernel security update
11:50 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1091 after schema change - T85757 (duration: 00m 52s)
11:49 moritzm: rebooting matomo1001 to pick up SSBD-enabled qemu
11:46 banyek: repooling db1091 after schema change - T85757
11:41 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1091 for schema change - T85757 (duration: 00m 52s)
11:37 banyek: depooling db1091 for schema change - T85757
11:34 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1084 after schema change - T85757 (duration: 00m 51s)
11:30 banyek: repooling db1084 after schema change - T85757
11:29 moritzm: upgrading nodejs on restbase2013-2018
11:26 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2050 after recloning db2057 T212275 (duration: 00m 52s)
11:25 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1084 for schema change - T85757 (duration: 00m 52s)
11:15 banyek: depooling db1084 for schema change - T85757
11:13 marostegui: Stop MySQL and power off db2057 for firmware upgrade - T212277
11:11 marostegui: Drop image_comment_temp from s5 - T209591
10:37 moritzm: draining restbase2012 for eventual reboot for kernel security update
10:28 banyek: executing schema change in db2051 (s4 codfw master) with replication enabled - T85757
10:25 banyek: stopping replication on db2073 as executing schema change on codfw master - T85757
10:14 moritzm: draining restbase2011 for eventual reboot for kernel security update
09:53 banyek: dropping tables 'flagged%' on db1066 ptwiki with replication enabled - T211544
09:41 moritzm: draining restbase2010 for eventual reboot for kernel security update
09:37 banyek: dropping tables with 'T211544' prefix on db1122 - T211544
09:24 moritzm: draining restbase2009 for eventual reboot for kernel security update
09:14 moritzm: rebooting restbase2008 for kernel security update
08:53 elukey: roll restart of cassandra on aqs1005-1009 for opendjdk upgrades
08:50 marostegui: Drop image_comment_temp from s7 - T209591
08:46 marostegui: Drop image_comment_temp from s6 - T209591
08:43 akosiaris: rebalance row_A ganeti01.svc.codfw.wmnet nodegroup after recabling T210447
08:40 marostegui: Drop image_comment_temp from s8 - T209591
08:37 moritzm: draining restbase2007 for eventual reboot for kernel security update
08:28 marostegui: Drop image_comment_temp from s2 - T209591
08:27 marostegui: Drop image_comment_temp from s1 - T209591
08:16 godog: swift eqiad-prod: more weight for ms-be10[44-50].eqiad.wmnet - T209618
07:37 marostegui: Drop nodepooldb on m5 master - T212230
07:22 marostegui: Stop MySQL on db2050 to clone db2057 - T212275
07:15 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2050 to clone db2057 T212275 (duration: 00m 52s)
07:00 marostegui: Enable GTID on s8 codfw master (db2045) - T211973
06:58 marostegui: Enable GTID on s1 codfw master (db2048) - T211973
06:44 marostegui: Remove nodepool@10.64.16.155 user from m5 master - T212230
06:34 marostegui: Hard reboot db2057 - T212275
06:27 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2057 - storage crashed T212275 (duration: 01m 08s)
01:27 tstarling@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/AbuseFilter/maintenance/normalizeThrottleParameters.php: g 480681 make maintenance script dry run more useful (duration: 00m 52s)
01:25 tstarling@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/AbuseFilter/includes/AbuseFilter.php: g 480680 fix exception in maintenance script (duration: 00m 54s)
00:11 mutante: contint1001 - rsyncing /srv/org/wikimedia/docs to rsync://docs1001.eqiad.wmnet/docs T211974

2018-12-18

23:49 XioNoX: remove BGP session to AS50629 from cr2-esams (not in AMS-IX anymore)
20:33 otto@deploy1001: Finished deploy [eventlogging/analytics@104adb5]: Send JSON string of event for validation errors in EventError (duration: 00m 04s)
20:33 otto@deploy1001: Started deploy [eventlogging/analytics@104adb5]: Send JSON string of event for validation errors in EventError
19:43 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@963d704]: Parse summaries from lead objects only (T202642) (duration: 05m 26s)
19:38 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@963d704]: Parse summaries from lead objects only (T202642)
19:30 robh: migration of ulsfo pdus complete
19:19 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Production configuration for GrowthExperiments Help Panel (duration: 00m 52s)
19:06 XioNoX: redirect ns1 back to authdns2001 - T210447
18:51 moritzm: installing libx11 security updates
18:48 XioNoX: depool ulsfo - T209101
18:42 XioNoX: repool codfw - T210447
18:41 XioNoX: Revert "Redirect eqsin/ulsfo caches to eqiad" - T210447
18:05 chasemp: stat1004:~# umount /mnt/T211327
17:58 XioNoX: shutdown fpc4 for replacement - T210447
17:51 bblack: deploying php7 cache-splitter patch to cache_text - https://gerrit.wikimedia.org/r/c/operations/puppet/+/478680 - T206339
17:49 godog: bounce rsyslog on lithium, tls listener timeout
17:34 volans: triggered restart of ircecho on icinga1001 while applying https://gerrit.wikimedia.org/r/480509
16:37 jijiki: librsvg* 2.40.20-3+wmf1+stretch1 uploaded to components/thumor to stretch-wikimedia - T209886
16:22 bblack: powercycle cp1075 from console (crashed, apparently)
16:07 XioNoX: starting codfw row A recabling - T210447
16:05 ejegg: updated fundraising python tools from af5dbee8eb to 5f44d9dd43
15:53 XioNoX: redirect ns1 to authdns1001 for T210447
15:47 XioNoX: redirect eqsin/ulsfo caches to eqiad for T210447
15:44 XioNoX: depool codfw for T210447
15:30 akosiaris: empty ganeti2005, ganeti2006 for T210447
15:16 Amir1: mwscript extensions/WikibaseQualityConstraints/maintenance/ImportConstraintEntities.php --wiki=testwikidatawiki --config-format=wgConf | tee WikibaseQualityConstraints-config.php
14:55 akosiaris: restart pybal on lvs1006, lvs2003 for blubberoid LVS deployment. T205919
14:40 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.33.0-wmf.9
14:26 akosiaris: restart ircecho, seems to have croaked with Dec 17 19:39:52 icinga1001 ircecho[861]: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 174: ordinal not in range(128)
14:25 zfilipin@deploy1001: Finished scap: testwiki to php-1.33.0-wmf.9 and rebuild l10n cache (duration: 34m 11s)
14:15 akosiaris: restart pybal on lvs1016, lvs2006 for blubberoid LVS deployment. T205919
13:51 zfilipin@deploy1001: Started scap: testwiki to php-1.33.0-wmf.9 and rebuild l10n cache
13:12 addshore@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/Wikibase/lib/includes/Formatters/ControlledFallbackEntityIdFormatter.php: T201930 ControlledFallbackEntityIdFormatter, track unique value formats (duration: 00m 46s)
12:52 Amir1: deployed a patch on wmf.8 for T207814
12:41 filippo@deploy1001: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 45s)
12:17 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Perform even more PHP constraint checks before falling back (T209504) (duration: 00m 46s)
12:16 moritzm: installing fuse updates from stretch point release
11:41 moritzm: installing remaining libgd2 security updates
09:38 godog: swift eqiad-prod: initial weights for ms-be10[44-50].eqiad.wmnet - T209618
08:44 marostegui: Enable GTID on s7 codfw master (db2040) - T211973
08:26 marostegui: Enable GTID on es3 - T211973
08:23 marostegui: Enable GTID on es2 - T211973
{{safesubst:SAL entry|1=07:57 elukey: restart cassandra-{a,b} on aqs1004 for openjdk upgrades}}
07:49 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1078 T86338 T202167 (duration: 00m 45s)
07:49 marostegui: Deploy schema change on db1075 (s3 master) T86338 T202167
06:07 marostegui: Deploy schema change on db1078 T86338 T202167
06:07 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1078 T86338 T202167 (duration: 00m 45s)
06:03 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2038 after mysql and kernel upgrade (duration: 00m 47s)
00:13 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable MediaViewer thumbnail URL guessing for private wikis (T212099) (duration: 00m 45s)

2018-12-17

23:21 tgr: deployed security patch per T207750#4829275
23:17 chasemp: launch gpg jobs for files on stat1004
23:17 chasemp: launch gpg jobs for files on labstore1007
22:00 mutante: puppetmaster - signed cert for doc1001 (ganeti VM), initial puppet run
21:58 chasemp: stat1004:/var/log# mkfs.exfat /dev/sde && mkdir /mnt/T211327 && mount /dev/sde /mnt/T211327/
21:55 arlolra: Updated Parsoid to 4eba44e (T204622, T211941)
21:42 arlolra@deploy1001: Finished deploy [parsoid/deploy@1bf4dab]: Updating Parsoid to 4eba44e (duration: 08m 55s)
21:33 arlolra@deploy1001: Started deploy [parsoid/deploy@1bf4dab]: Updating Parsoid to 4eba44e
21:20 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@92fdf43]: Update mobileapps to d244439 (duration: 04m 31s)
21:15 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@92fdf43]: Update mobileapps to d244439
20:25 ppchelko@deploy1001: Finished deploy [recommendation-api/deploy@c1b6b32]: Rollback to c1b6b32 until the checks are fixed (duration: 01m 56s)
20:23 ppchelko@deploy1001: Started deploy [recommendation-api/deploy@c1b6b32]: Rollback to c1b6b32 until the checks are fixed
20:17 mutante: creating new ganeti VM doc1001.eqiad.wmnet for doc.wikimedia.org - specs as requested by hashar on T211974
20:05 XioNoX: remove sandbox-out4 from all routers - T212155
20:02 XioNoX: remove sandbox-out4 from ulsfo - T212155
19:47 ppchelko@deploy1001: Finished deploy [recommendation-api/deploy@f183af7]: Update to 657a515. All hosts (duration: 10m 45s)
19:36 ppchelko@deploy1001: Started deploy [recommendation-api/deploy@f183af7]: Update to 657a515. All hosts
19:33 ppchelko@deploy1001: Finished deploy [recommendation-api/deploy@f183af7]: Update to 657a515. Canary on scb2001 (duration: 00m 18s)
19:32 ppchelko@deploy1001: Started deploy [recommendation-api/deploy@f183af7]: Update to 657a515. Canary on scb2001
19:29 raynor: Morning SWAT finished
19:27 pmiazga@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Page issues treatment enabled on all wikis except enwiki(T210553) (duration: 00m 45s)
18:58 ppchelko@deploy1001: Finished deploy [recommendation-api/deploy@8036f9b]: Update to 2991db1. Canary on scb2001 (duration: 00m 51s)
18:57 ppchelko@deploy1001: Started deploy [recommendation-api/deploy@8036f9b]: Update to 2991db1. Canary on scb2001
18:42 herron: manually restarting pybal on lvs2003 to add kibana service T205850
18:34 herron@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=logstash,service=kibana,name=logstash2006.codfw.wmnet
18:34 herron@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=logstash,service=kibana,name=logstash2005.codfw.wmnet
18:34 mobrovac@deploy1001: Finished deploy [proton/deploy@ff7c8a2]: Config: Use extdomain and add printable=yes to the req query - T210793 (duration: 01m 32s)
18:32 herron: performing manual puppet run and manually restarting pybal on lvs2006 to add kibana service T205850
18:32 mobrovac@deploy1001: Started deploy [proton/deploy@ff7c8a2]: Config: Use extdomain and add printable=yes to the req query - T210793
18:29 gtirloni: prometheus-node-exporter: Ignore Docker/Kubelet mount points T211810
18:26 akosiaris@deploy1001: scap-helm blubberoid finished
18:26 akosiaris@deploy1001: scap-helm blubberoid cluster codfw completed
18:26 akosiaris@deploy1001: scap-helm blubberoid install --name production --set docker.registry=docker-registry.discovery.wmnet --set main_app.version=2018-12-13-183249-production --set service.deployment=production --set service.externalIP=10.2.2.31 --set service.port=8748 stable/blubberoid [namespace: blubberoid, clusters: codfw]
18:25 akosiaris@deploy1001: scap-helm blubberoid finished
18:25 akosiaris@deploy1001: scap-helm blubberoid cluster eqiad completed
18:25 akosiaris@deploy1001: scap-helm blubberoid install --name production --set docker.registry=docker-registry.discovery.wmnet --set main_app.version=2018-12-13-183249-production --set service.deployment=production --set service.externalIP=10.2.1.31 --set service.port=8748 stable/blubberoid [namespace: blubberoid, clusters: eqiad]
18:24 akosiaris@deploy1001: scap-helm blubberoid install --name production --set docker.registry=docker-registry.discovery.wmnet --set main_app.version=2018-12-13-183249-production --set service.deployment=production --set service.externalIP=10.2.1.31 --set service.port=8748 stable/blubberoid [namespace: blubberoid, clusters: eqiad]
18:23 akosiaris@deploy1001: scap-helm blubberoid install --name production --set docker.registry=docker-registry.discovery.wmnet --set main_app.version=2018-12-13-183249-production --set service.deployment=production --set service.port=8748 stable/blubberoid [namespace: blubberoid, clusters: eqiad,codfw]
18:21 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@69c000a]: DelayQueue fix and mitigation of stale index for updater (duration: 11m 52s)
18:21 moritzm: installing libgd2 security updates
18:10 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@69c000a]: DelayQueue fix and mitigation of stale index for updater
18:07 akosiaris@deploy1001: scap-helm blubberoid finished
18:07 akosiaris@deploy1001: scap-helm blubberoid cluster staging completed
18:07 akosiaris@deploy1001: scap-helm blubberoid install --name staging --set docker.registry=docker-registry.discovery.wmnet --set main_app.version=2018-12-13-183249-production stable/blubberoid [namespace: blubberoid, clusters: staging]
18:04 herron@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=logstash,service=kibana,name=logstash2004.codfw.wmnet
16:41 moritzm: installing libarchive security updates on jessie
16:33 moritzm: installing php5 security updates on jessie
16:13 anomie: Aborted migrateActors.php run, queries were too slow.
16:07 anomie@mwmaint1002: Running migrateActors.php on test wikis and mediawikiwiki for T188327
16:07 mobrovac@deploy1001: Started restart [electron-render/deploy@94d27d7]: Electron stuck
14:33 marostegui: Upgrade mysql and kernel on db2093 (tendril sby host on codfw)
14:10 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: wmgWikibaseMaxItemIdForNewItemIdHtmlFormatter 200million everywhere T201837 (duration: 00m 44s)
14:09 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: wmgWikibaseMaxItemIdForNewItemIdHtmlFormatter 200million everywhere T201837 (duration: 00m 44s)
14:06 marostegui: Stop MySQL on db2038 for mysql and kernel upgrade
14:06 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2038 for mysql and kernel upgrade (duration: 00m 44s)
13:58 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: wmgWikibaseMaxItemIdForNewItemIdHtmlFormatter 10million for wikidatawiki T201837 (duration: 00m 45s)
13:49 marostegui: Enable GTID on s3 codfw master db2043 - T211973
13:49 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: wmgWikibaseMaxItemIdForNewItemIdHtmlFormatter 1million for wikidatawiki T201837 (duration: 00m 45s)
13:46 fsero: installing new version of php-excimer on mwdebug* - T205059
13:41 fsero: installing new version of php-excimer on mwdebug2001 - T205059
13:31 Amir1: ladsgroup@mwmaint1002:~$ mwscript extensions/Wikibase/lib/maintenance/populateSitesTable.php --wiki incubatorwiki --force-protocol https
13:29 Amir1: deleting everything from site_identifiers@incubatorwiki
13:15 ariel@deploy1001: Finished deploy [dumps/dumps@0393ca7]: switch dumps to python3 (duration: 00m 03s)
13:15 ariel@deploy1001: Started deploy [dumps/dumps@0393ca7]: switch dumps to python3
13:04 Amir1: ladsgroup@mwmaint1002:~$ foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https (T209820)
13:04 Amir1: ladsgroup@mwmaint1002:~$ foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https
13:02 Amir1: ladsgroup@mwmaint1002:~$ mwscript extensions/Wikibase/lib/maintenance/populateSitesTable.php --wiki=fawiki --force-protocol https
12:52 ema: lvs1016: bounce pybal for new service kartotherian-ssl T211970
12:50 ladsgroup@deploy1001: Synchronized wmf-config/InterwikiSortOrders.php: Update InterwikiSortOrders.php (T209820) (duration: 00m 45s)
12:49 ladsgroup@deploy1001: sync-file aborted: Use the right index for change_tag (T211896) (duration: 00m 02s)
12:38 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add be.wikisource new 1.5x and 2x logos to wgLogoHD (T150618) (duration: 00m 44s)
12:38 ema: lvs2003: bounce pybal for new service kartotherian-ssl T211970
12:29 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: Update be.wikisource logo (T211795) (duration: 00m 45s)
12:29 ema: lvs1006: bounce pybal to pick up new service kartotherian-ssl T211970
12:24 ema: lvs2006: bounce pybal to pick up new service kartotherian-ssl
12:22 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable extendedmover user group at ur.wiki (T211978) (duration: 00m 45s)
12:15 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Use new minerva logos for cswiki in IS.php (T210979) (duration: 00m 45s)
12:08 zfilipin@deploy1001: Synchronized static/images/mobile/copyright/: SWAT: Upload custom minerva logo for cswiki (T210979) (duration: 00m 44s)
12:02 ladsgroup@deploy1001: Finished deploy [ores/deploy@18d3657]: T206333 T211267 (duration: 14m 14s)
11:48 ladsgroup@deploy1001: Started deploy [ores/deploy@18d3657]: T206333 T211267
10:52 paravoid: reprepro include python 3.4, 3.6, 3.7 to component/pyall (use with care)
10:28 moritzm: installing openssl security updates on stretch
10:18 marostegui: Repool labsdb1011 - T86338
09:55 moritzm: remove debmonitor entries for restbase2001-restbase2006
09:44 marostegui: Enable GTID on s4 codfw master - T211973
09:08 marostegui: Depool labsdb1011 - T86338
09:04 marostegui: Repool labsdb1010 - T86338
09:01 elukey: stop kafkatee on oxygen and rsync /srv/log data to weblog1001
08:47 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1077 T86338 T202167 (duration: 00m 45s)
08:21 marostegui: Deploy schema change on db1077 with replication (lag will be generated on labsdb:s3) T86338 T202167
08:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1077 T86338 T202167 (duration: 00m 45s)
08:16 marostegui: Stop replication on labsdb1011
08:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1123 T86338 T202167 (duration: 00m 45s)
07:58 marostegui: Enable GTID on db2052 (s5 master) - T211973
06:48 marostegui: Enable GTID on db2035 (s2 master) - T211973
06:36 marostegui: Enable GTID on db2034 (x1 master) - T211973
06:36 marostegui: Enable GTID on db2034 (x1 master)
06:29 marostegui: Enable replication consistency options on codfw masters - T211973
06:28 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1123 T86338 T202167 (duration: 00m 46s)
06:28 marostegui: Depool labsdb1010 - T86338
04:22 tstarling@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/AbuseFilter/maintenance/normalizeThrottleParameters.php: gerrit 479998 for testing of normalizeThrottleParameters.php (duration: 00m 44s)
04:20 tstarling@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/AbuseFilter/includes/AbuseFilter.php: gerrit 479998 for testing of normalizeThrottleParameters.php (duration: 00m 59s)

2018-12-16

09:52 elukey: mask + reset-failed kafkatee default instance on sulfur (kafkatee-webrequest works fine)
07:30 marostegui: Reboot db1115 after OOM
07:28 marostegui: Stop MySQL on db1115 so tendril can get back to work

2018-12-15

off: restarted pdfrender on scb1004
09:22 elukey: mask + reset-failed kafkatee default instance on weblog1001

2018-12-14

23:59 mutante: LDAP: added aezell to wmf group (T211945) for grafana access
22:00 XioNoX: increase accepted-prefix-limit for HE to 200000
20:38 mutante: sulfur systemctl restart nagios-nrpe-server
20:32 andrew@deploy1001: Finished deploy [horizon/deploy@1a830b9]: Rolling out fix for T177855 (duration: 03m 17s)
20:29 andrew@deploy1001: Started deploy [horizon/deploy@1a830b9]: Rolling out fix for T177855
20:13 otto@deploy1001: Finished deploy [analytics/refinery@ef1f7c6]: deploying refinery-source 0.0.82 with fix for T211833 (duration: 06m 04s)
20:07 otto@deploy1001: Started deploy [analytics/refinery@ef1f7c6]: deploying refinery-source 0.0.82 with fix for T211833
20:07 otto@deploy1001: deploy aborted: (no justification provided) (duration: 00m 00s)
20:07 otto@deploy1001: Started deploy [analytics/refinery@ef1f7c6]: (no justification provided)
17:22 cmjohnson1: mw1272 down for h/w troubleshooting
14:38 marostegui: Enable GTID on db2039 (s6 codfw master) - T211973
14:03 marostegui: Compare ruwiki.revision between db2039 (s6 master) and db1085 - T211973
13:59 marostegui: Enable notifications for db1095 (s3 lag check)- T211973
13:57 marostegui: Enable notifications for db2068 (s7 lag check)- T211973
13:47 dcausse: elastic@codfw copying index data from the main cluster to psi & omega (test disk usage & import speed)
13:42 marostegui: Enable GTID on db1124:3318 - T211973
11:11 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2083 db2068 db2067 after mysql and kernel upgrade (duration: 00m 44s)
10:51 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2083 db2068 db2067 for mysql and kernel upgrade (duration: 00m 45s)
10:50 marostegui: Stop MySQL on db2083, db2068 and db2067 for mysql and kernel upgrade
10:19 filippo@deploy1001: Synchronized wmf-config/logging.php: wmf-config/InitialiseSettings-labs.php (duration: 00m 44s)
10:18 filippo@deploy1001: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 45s)
09:33 marostegui: Deploy schema change on db1095:3313 T86338 T202167
09:28 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2049, db2059, db2070 after mysql and kernel upgrade (duration: 00m 45s)
09:21 banyek: global user rename is in progress - T209488
08:58 marostegui: Stop MySQL on db2049, db2059 and db2070 for mysql and kernel upgrade
08:56 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2049, db2059, db2070 for mysql and kernel upgrade (duration: 00m 43s)
08:50 elukey: swap oxygen with weblog1001
08:48 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2084 and db2088 after mysql and kernel upgrade (duration: 00m 44s)
08:47 elukey: disabled kafkatee-webrequest logstash output on oxygen (prep step before weblog1001)
07:59 marostegui: Deploy schema change on dbstore1002:s3 T86338 T202167
07:49 marostegui: Upgrade mysql and kernel on db2084 and db2088
07:49 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2084 and db2088 for mysql and kernel upgrade (duration: 00m 45s)
06:54 marostegui: Deploy schema change on db2043 (s3 codfw master) - this will generate lag on s3 codfw T86338 T202167
06:53 marostegui: Deploy schema change on db2043 (s3 codfw master) - this will generate lag on s3 codfw T86338 T20216
06:14 marostegui: Deploy schema change on db1062 (s7 primary master) T86338 T202167
06:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1094 T86338 T202167 (duration: 00m 44s)
06:10 marostegui: Deployed schema change on db1094 T86338 T202167
05:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1094 T86338 T202167 (duration: 00m 47s)
01:39 mutante: install2002 deleted /srv/ contents,then mounted /mnt/vdb on /srv so same content but now / is used only 7% and /srv 57% (T211850)
00:43 mutante: install2002 (T211850) restarted instance, created ext4 filesystem on new /dev/vdb, mounted on /mnt/vdb, rsyncing /srv/ to /mnt/vdb/
00:33 mutante: rebooting install2002 via ganeti2003, to add new virtual disk
00:31 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: Enforce a 10-byte password for +staff users, I4ecac70e (duration: 00m 44s)
00:27 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT I7867277d Make wmgBabelMainCategory consistent for sr* wikis (duration: 00m 45s)

2018-12-13

23:57 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: T208246: Increase default minimum new password length to 10 for privileged groups (duration: 00m 44s)
23:17 andrew@deploy1001: Finished deploy [horizon/deploy@18c4ca6]: Rolling out fix for T131367 (duration: 03m 25s)
23:13 andrew@deploy1001: Started deploy [horizon/deploy@18c4ca6]: Rolling out fix for T131367
22:31 reedy@deploy1001: Synchronized php-1.33.0-wmf.8/includes/http/HttpRequestFactory.php: T211886 (duration: 00m 44s)
22:29 reedy@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/MobileFrontend: T211903 (duration: 00m 48s)
21:55 mutante: Ganeti - creating new 120G virtual disk on install2002 (T211850)
21:00 otto@deploy1001: Finished deploy [analytics/superset/deploy@UNKNOWN]: revert to version 0.26.3 (duration: 00m 32s)
20:59 otto@deploy1001: Started deploy [analytics/superset/deploy@UNKNOWN]: revert to version 0.26.3
20:33 dcausse: creating 300+ wikis indices in elastic-psi @eqiad and @codfw
20:21 volans: imported elasticsearch-curator_5.2.0-1~deb9u1 into apt.w.o stretch-wikimedia component/spicerack - T205884
20:10 mutante: LDAP - added mneisler to wmf (T211742) - existing shell user, so no gerrit change needed
19:53 hoo: Ran scap pull on snapshot1005 to undo live changes done for dump performance testing
19:49 dcausse: creating 300 wiki indices in elastic-omega@eqiad
19:46 dcausse: SF Morning SWAT done
19:44 dcausse@deploy1001: Synchronized wmf-config/CirrusSearch-production.php: T210381: [cirrus] fix cluster settings for temp clusters psi&omega (duration: 00m 44s)
19:40 moritzm: removed labvirt1014 from debmonitor DB, has been renamed to cloudvirt1014
19:39 volans: imported python-elasticsearch_5.4.0-1~deb9u1 into apt.w.o stretch-wikimedia component/spicerack - T205884
19:35 dcausse@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/TwoColConflict/: T210501: Add missing code to not loose edits on the other side (duration: 00m 45s)
19:29 bblack: authdns2001: upgrading gdnsd to 2.99.9944-beta
19:27 bblack: multatuli: upgrading gdnsd to 2.99.9944-beta
19:22 dcausse@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T211234: Enable Block notice stats on top blocking wikis (duration: 00m 45s)
19:19 bblack: authdns1001: upgrading gdnsd to 2.99.9944-beta
18:52 arlolra: Updated Parsoid to 4242ad0 (T204622, T211738)
18:40 arlolra@deploy1001: Finished deploy [parsoid/deploy@e27574c]: Updating Parsoid to 4242ad0 (duration: 09m 17s)
18:31 arlolra@deploy1001: Started deploy [parsoid/deploy@e27574c]: Updating Parsoid to 4242ad0
18:30 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@93443fe]: Refine MW API queries (duration: 03m 41s)
18:26 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@93443fe]: Refine MW API queries
18:24 jforrester@deploy1001: Synchronized wmf-config/Wikibase.php: T204748 [Beta only] Use newly-fixed config for Wikibase->Commons federation (duration: 00m 44s)
18:22 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T204748 Rename repo-only Wikibase config for clarity [no-op] (duration: 00m 45s)
18:11 bblack: restart pybal on lvs1006 for config updates
18:07 ladsgroup@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/FlaggedRevs/frontend/specialpages/reports/ProblemChanges_body.php: Use the right index for change_tag (T211896) (duration: 00m 46s)
17:45 ladsgroup@deploy1001: Finished deploy [ores/deploy@1a3de73]: T211267 (duration: 13m 53s)
17:41 akosiaris: reapply the zotero calico policy to allow LVS endpoints
17:35 dcausse: elastic@eqiad created cirrus metastore on psi&omega
17:32 ladsgroup@deploy1001: Started deploy [ores/deploy@1a3de73]: T211267
17:31 arturo: T168967 added shiny-server .deb to stretch-wikimedia
17:20 godog: run puppet and bounce pybal on lvs in eqiad to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/479184
17:19 akosiaris: T205919 create namespace for blubberoid on eqiad/codfw/staging clusters
17:00 anomie: Set comment migration to new on group 1 (T166733)
16:33 mutante: icinga1001 - started service again, enabeld puppet
16:27 mutante: icinga1001 - disable puppet, stopped icinga, for cable replacement
16:18 anomie: Deployed fix for T210937: API: Use parenthesized join in ApiQueryBase::showHiddenUsersAddBlockInfo
15:44 moritzm: installing openssl 1.1 security updates on Hadoop workers
15:33 moritzm: rebooting pybal-test hosts to pick up SSBD-enabled qemu
15:18 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting actor migration to write-both/read-old on all wikis (T188327) (duration: 00m 45s)
15:12 anomie@deploy1001: Synchronized php-1.33.0-wmf.8/includes/api/ApiPageSet.php: Backport fix for T211804: ApiPageSet::initFromPageIds: Default $filterIds to true (duration: 00m 46s)
15:03 akosiaris: ores2* deploy https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/479206/
14:55 moritzm: installing openssl 1.1 security updates on mw canaries (along with nginx restart/upgrade)
14:46 cdanis: updating grafana/stretch-wikimedia to 5.4.2: reprepro --restrict grafana update stretch-wikimedia
14:40 akosiaris: disable puppet on ores1* and ores2* machines to deploy https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/479206/
14:36 ladsgroup@deploy1001: Finished deploy [ores/deploy@a9d5e95]: noop (duration: 14m 59s)
14:23 ladsgroup@deploy1001: Started deploy [ores/deploy@a9d5e95]: noop
13:22 dcausse: creating 300+ wiki indices on elastic-omega@codfw
13:15 godog: stop restbase and cassandra on restbase200[1-6] - T211070
12:59 elukey: superset on analytics-tool1003 upgraded to 0.28.1
12:41 arturo: icinga downtime (30 mins) cloudcontrol1003, cloudnet1003 and cloudnet1004 for package upgrades
12:38 zeljkof: EU SWAT finished
12:35 zfilipin@deploy1001: Synchronized static/images/project-logos: SWAT: Update maiwiki logo (T211845) (duration: 00m 52s)
12:35 oblivian@deploy1001: scap-helm zotero finished
12:35 oblivian@deploy1001: scap-helm zotero cluster codfw completed
12:35 oblivian@deploy1001: scap-helm zotero cluster eqiad completed
12:35 oblivian@deploy1001: scap-helm zotero upgrade production -f ../zotero-values-codfw.yaml stable/zotero [namespace: zotero, clusters: eqiad,codfw]
12:26 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable flood user group at ne.wiki (T211181) (duration: 00m 51s)
12:14 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Namespace configuration on shnwiki (T210699) (duration: 00m 53s)
12:12 mobrovac@deploy1001: Finished deploy [restbase/deploy@55fcd4b]: Remove restbase200[1-6], ensure body.tfa exists for feed responses and disable Citoid check - T211070 T211871 T211411 (duration: 18m 59s)
12:10 oblivian@deploy1001: scap-helm -h finished
12:10 oblivian@deploy1001: scap-helm -h cluster codfw completed
12:10 oblivian@deploy1001: scap-helm -h cluster eqiad completed
12:10 oblivian@deploy1001: scap-helm -h [namespace: -h, clusters: eqiad,codfw]
12:00 moritzm: rebooting ununpentium to pick up SSBD-enabled qemu
11:53 mobrovac@deploy1001: Started deploy [restbase/deploy@55fcd4b]: Remove restbase200[1-6], ensure body.tfa exists for feed responses and disable Citoid check - T211070 T211871 T211411
11:51 elukey@deploy1001: Finished deploy [analytics/superset/deploy@35841a7]: (no justification provided) (duration: 00m 38s)
11:51 elukey@deploy1001: Started deploy [analytics/superset/deploy@35841a7]: (no justification provided)
11:45 mobrovac@deploy1001: Finished deploy [restbase/deploy@29a0902]: Remove restbase200[1-6] and ensure body.tfa exists for feed responses - T211070 T211871 (duration: 06m 08s)
11:43 moritzm: rebooting vega/bromine to pick up SSBD-enabled qemu
11:39 mobrovac@deploy1001: Started deploy [restbase/deploy@29a0902]: Remove restbase200[1-6] and ensure body.tfa exists for feed responses - T211070 T211871
11:39 mobrovac@deploy1001: Finished deploy [restbase/deploy@29a0902]: Remove restbase200[1-6] and ensure body.tfa exists for feed responses - T211070 T211871 (duration: 07m 08s)
11:38 fsero@deploy1001: scap-helm zotero finished
11:38 fsero@deploy1001: scap-helm zotero cluster codfw completed
11:38 fsero@deploy1001: scap-helm zotero upgrade production -f ../zotero-values-codfw.yaml stable/zotero [namespace: zotero, clusters: codfw]
11:38 moritzm: rebooting krypton to pick up SSBD-enabled qemu
11:31 mobrovac@deploy1001: Started deploy [restbase/deploy@29a0902]: Remove restbase200[1-6] and ensure body.tfa exists for feed responses - T211070 T211871
11:31 fsero@deploy1001: scap-helm zotero finished
11:31 fsero@deploy1001: scap-helm zotero cluster eqiad completed
11:31 fsero@deploy1001: scap-helm zotero upgrade production -f ../zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
11:14 moritzm: rebooting webperf hosts to pick up SSBD-enabled qemu
10:51 moritzm: rebooting dbmonitor hosts to pick up SSBD-enabled qemu
10:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1101:3317 T86338 T202167 (duration: 00m 51s)
10:00 elukey: upgrade nodejs on aqs100[5-9]
09:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101:3317 T86338 T202167 (duration: 00m 51s)
09:55 moritzm: removed openssl 1.1.0f-3+deb9u2+wmf1 from stretch-wikimedia/component/node10 (superseded by openssl update in DSA 4348 for stretch)
09:35 moritzm: rebooting etcd/kubernetes hosts in codfw to pick up SSBD-enabled qemu
09:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3317 T86338 T202167 (duration: 00m 51s)
09:06 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2006.codfw.wmnet
09:06 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2005.codfw.wmnet
09:06 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2004.codfw.wmnet
09:06 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2003.codfw.wmnet
09:05 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2002.codfw.wmnet
09:05 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2001.codfw.wmnet
08:50 godog: stress-test ms-be10[44-50] - T209618
08:45 marostegui: Deploy schema change on db1090:3317 T86338 T202167
08:45 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3317 T86338 T202167 (duration: 00m 53s)
08:45 moritzm: installing openssl security updates on stretch
08:26 vgutierrez: Use certcentral managed TLS certificates in mx[12]001.wikimedia.org - T207050
08:15 marostegui: Drop unused flaggedrevs tables from srwikinews - T209761
08:11 moritzm: rolling reboot of scb in eqiad for kernel security update (combined with nodejs update)
08:09 marostegui: Repool labsdb1011 T86338
08:08 moritzm: installing nodejs updates on restbase1007
07:28 marostegui: Depool labsdb1011 T86338
07:25 marostegui: Repool labsdb1010 T86338
07:22 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1079 T86338 T202167 (duration: 00m 52s)
06:46 marostegui: Deploy schema change on db1079 with replication, lag will be generated on labsdb:s7 T86338 T202167
06:45 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1079 T86338 T202167 (duration: 00m 53s)
06:43 marostegui: Depool labsdb1010 T86338
06:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1086 T86338 T202167 (duration: 00m 51s)
06:17 marostegui: Deploy schema change on db1086 T86338 T202167
06:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1086 T86338 T202167 (duration: 00m 51s)
06:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1098:3316 db1098:3317 after kernel and mysql upgrade (duration: 00m 54s)
00:52 reedy@deploy1001: Synchronized php-1.33.0-wmf.8/includes/http/GuzzleHttpRequest.php: T211806 (duration: 00m 51s)
00:41 mutante: einsteinium - rm /lib/systemd/system/update-etcd-mw-config-lastindex.service ; systemctl reset-failed
00:36 tzatziki: changing two passwords for compromised accounts
00:32 dcausse: elastic@codfw created cirrus metastore on psi&omega clusters
00:22 dcausse@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/MobileFrontend/extension.json: T210390: Reset default mobilefrontend provider (duration: 00m 53s)
00:13 dcausse@deploy1001: Synchronized wmf-config/CirrusSearch-production.php: T210381: [cirrus] fix temp clusters for codfw (duration: 00m 52s)

2018-12-12

22:47 tzatziki: change email for User:Denrique
21:48 reedy@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/EventBus: T211805 (duration: 00m 53s)
21:26 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@ced6fab]: Update mobileapps to 55981a8. Summary: Get modified date with regexes to avoid unneeded Document parse (duration: 04m 03s)
21:22 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@ced6fab]: Update mobileapps to 55981a8. Summary: Get modified date with regexes to avoid unneeded Document parse
19:45 hashar: contint1001: sudo chown -R zuul:zuul /etc/zuul/wikimedia/.git
19:28 XioNoX: repool codfw - T210456
19:28 XioNoX: revert redirecting eqsin/ulsfo caches to eqiad - T210456
19:16 XioNoX: re-enable BGP to telia on cr1-codfw - T211715
19:08 XioNoX: disable BGP to telia on cr1-codfw - T211715
19:06 bblack: uploading gdnsd 2.99.9944-beta-1+wmf1 to stretch-wikimedia
19:04 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: pool db1098 for recentchanges and recentchangeslinked (duration: 00m 50s)
18:53 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: pool db1098 for recentchanges and recentchangeslinked (duration: 02m 58s)
18:46 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: pool db1098 for recentchanges and recentchangeslinked (duration: 03m 00s)
18:41 banyek: pool db1098 for recentchanges and recenlchangeslinked
18:12 catrope@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/GrowthExperiments/extension.json: Temporarily disable help panel / VisualEditor integration (duration: 03m 00s)
17:55 banyek: pooling db1098
17:54 XioNoX: shutting down asw-b4-codfw - T210456
17:02 zfilipin@deploy1001: Synchronized php: group1 wikis to 1.33.0-wmf.8 (duration: 00m 50s)
16:42 moritzm: installing lxml security updates
16:40 moritzm: installing cups updates on trusty (only client libs used)
16:34 jijiki: Merged 477472 "mcrouter: replace codfw proxy before maintenance", eqiad mcrouters are picking up the change - T210467
16:19 ladsgroup@deploy1001: Synchronized php-1.33.0-wmf.8/includes/specials/pagers/ImageListPager.php: T211774 (duration: 00m 52s)
16:17 ladsgroup@deploy1001: Synchronized php-1.33.0-wmf.8/includes/specials/pagers/ImageListPager.php: T211774 (duration: 00m 52s)
16:01 XioNoX: Redirect eqsin/ulsfo caches to eqiad - T210456
15:54 XioNoX: Depool codfw for row B recabling - T210456
15:35 elukey: upload matomo 3.7.0 to stretch-wikimedia, removed 3.5.1 from jessie-wikimedia
15:27 moritzm: installing PHP security updates on matomo1001 (piwik host)
15:24 godog: poweroff ms-be2044 for hardware inspection - T209921
15:22 ladsgroup@deploy1001: Synchronized php-1.33.0-wmf.8/includes/api/ApiBase.php: T211769 (duration: 00m 52s)
15:21 urandom: decommissioning cassandra-c, restbase2006 -- T210843
15:20 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1098:3316 db1098:3317 after kernel and mysql upgrade (duration: 00m 53s)
14:43 banyek: renaming tables on db1122 ptwiki: flagged* -> T211544_flagged* - T211544
14:29 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: Revert "group1 wikis to 1.33.0-wmf.8"
14:18 moritzm: restart uwsgi-netbox on netmon2001
14:16 zfilipin@deploy1001: Synchronized php: group1 wikis to 1.33.0-wmf.8 (duration: 00m 51s)
14:07 mobrovac@deploy1001: Started restart [restbase/deploy@5946231]: Restart RB to pick up the new seeds in codfw - T211416
14:04 marostegui: Stopy MySQL on db1098:3316 and db1098:3317 for kernel and mysql upgrade
13:28 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1098:3316 db1098:3317 for kernel and mysql upgrade (duration: 00m 52s)
13:26 moritzm: installing PHP security updates
12:52 zeljkof: EU SWAT finished
12:49 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add NS_PROJECT localised name for tt.wiktionary (T211312) (duration: 00m 52s)
12:41 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add new namespace aliases for zhwikiversity (T207544) (duration: 00m 52s)
12:35 arturo: T205969 icinga downtime load-avg check for labstore1007 until January (1 month)
12:32 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable extension SandboxLink for nowiki (T210325) (duration: 00m 52s)
12:27 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Use HD logos for cawikimedia in IS.php (T198507) (duration: 00m 52s)
12:17 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: Upload new logos for cawikimedia (T198507) (duration: 00m 52s)
12:03 hoo@deploy1001: Synchronized wmf-config/Wikibase.php: Display Kartographer mapframes for geocoordinate statements (T184933) (duration: 00m 52s)
11:30 volans: re-enabled puppet on icinga[12]001, re-activated crontab to sync files on 2001 and manually run it + run puppet
11:14 volans: restarting icinga with dropped downtimes from last night (start_date > 1544489652)
11:03 volans: restarting Icinga with debug log on icinga1001
10:51 mobrovac@deploy1001: Finished deploy [restbase/deploy@44e0955]: Bring restbase201[3-8] up to date, try #2b - T211416 (duration: 10m 11s)
10:41 mobrovac@deploy1001: Started deploy [restbase/deploy@44e0955]: Bring restbase201[3-8] up to date, try #2b - T211416
10:40 mobrovac@deploy1001: Finished deploy [restbase/deploy@44e0955]: Bring restbase201[3-8] up to date, try #2 - T211416 (duration: 00m 15s)
10:40 mobrovac@deploy1001: Started deploy [restbase/deploy@44e0955]: Bring restbase201[3-8] up to date, try #2 - T211416
10:26 volans: Icinga is having issue restarting properly, investigation ongoing
10:08 banyek: executing schema change on db1070 (s5 master) - T85757
09:40 banyek: repooling labsdb1010 - T210693
09:34 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1082 (duration: 00m 52s)
09:29 banyek: repooling db1082 - T85757
09:25 banyek: restarting replication on db1082 after schema change - T85757
09:25 banyek: fixing triggers on db1124:3315- T85757
09:22 banyek: executing schema change with replication on db1082 - T85757
09:20 banyek: stopping replication on db1082 for schema change - T85757
09:15 moritzm: installing pixman security updates on trusty (Debian already fixed)
08:48 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1082 (duration: 00m 51s)
08:38 banyek: depooling db1082 for schema change - T85757
08:37 marostegui: Remove old backup directory from db1116 - T206743
08:18 godog: decommissioning cassandra-b, restbase2006 -- T210843
08:03 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1088 (duration: 00m 51s)
07:49 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1088 (duration: 00m 52s)
07:38 marostegui: Deploy schema change on db2040 (s7 codfw master), this will generate lag on codfw T86338 T202167
07:35 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1088 (duration: 00m 51s)
07:09 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1088 (duration: 00m 51s)
06:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1088 (duration: 00m 52s)
06:44 marostegui: Stop MySQL on db1088 for mysql and kernel upgrade
06:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1088 for mysql upgrade (duration: 01m 07s)
06:37 marostegui: Deploy schema change on s4 primary master (db1068) T86338
06:00 marostegui: Deploy schema change on s8 primary master (db1071) T86338 T202167
00:47 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add some missing groups to the privileged list (duration: 00m 51s)
00:46 tgr@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: Bring up password change logging to the same standards as login logging Add some missing groups to the privileged list (duration: 00m 53s)

2018-12-11

22:50 chasemp: ssh to tar archive data from logstash1006 /mnt (external) to labstore1007
22:06 XioNoX: push loopback filter term return-tcp to all routers - T207962
22:04 urandom: decommissioning cassandra-a, restbase2006 -- T210843
19:57 XioNoX: apply BGP_IXP_RS_in and avoid HE to cr4-ulsfo - T211079
19:28 jforrester@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/Flow/modules/engine/misc/flow-handlebars.js: T211707 Hot-deploy fix for StructuredDiscussions pages (duration: 00m 52s)
19:22 jforrester@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.Target.js: T209619: Hot-deploy Ibcf3c93e (duration: 00m 52s)
18:59 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@ecd5fb6]: fix site CSS URL (duration: 03m 58s)
18:55 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@ecd5fb6]: fix site CSS URL
18:54 XioNoX: push BGP_IXP_RS_in to all routers (but don't apply it to any peers, needs to be done manually) - T211079
18:33 cmjohnson1: swapping disk slot 0 db1063
18:23 XioNoX: push BGP_IXP_RS_in to cr2-eqord - T211079
18:06 anomie@deploy1001: Synchronized php-1.33.0-wmf.6/includes/user/User.php: Backport fix for T210621, for real this time (duration: 00m 53s)
18:00 jforrester@deploy1001: Synchronized wmf-config/Wikibase.php: T211237 Hot-deploy disabling Wikibase federation, unused except in Beta Cluster (duration: 00m 52s)
17:33 _joe_: restarting pybal on lvs2003
17:23 XioNoX: push changes tested on cr4-ulsfo to all routers - T211079
17:16 _joe_: restarting pybal on lvs2006
17:15 oblivian@puppetmaster1001: conftool action : set/weight=1; selector: service=search-omega-ssl
17:15 oblivian@puppetmaster1001: conftool action : set/weight=1; selector: service=search-psi-ssl
17:15 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: service=search-psi-ssl
17:14 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: service=search-omega-ssl
17:05 XioNoX: remove redundant term classification from BGP_transit_in on cr4-ulsfo - T211079
16:55 banyek: executing schema change on dbstore1002 - T85757
16:50 XioNoX: replace local-preference/default-action by next policy for BGP_IXP_in and BGP_Private_Peer_in on cr4-ulsfo - T211079
16:49 banyek: executing schema change on db1102 - T85757
16:48 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1113:3315 (duration: 00m 51s)
16:44 banyek: repooling db1113:3315 after schema change - T85757
16:38 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1113:3315 (duration: 00m 51s)
16:35 banyek: depooling db1113:3315 for schema change - T85757
16:33 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1110 (duration: 00m 51s)
16:32 banyek: repooling db1110 after schema change - T85757
16:28 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1110 (duration: 00m 50s)
16:25 banyek: depooling db1110 for schema change - T85757
16:23 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1100 (duration: 00m 51s)
16:22 XioNoX: Remove static routes for NS v6 IPs - T211699
16:20 banyek: repooling db1100 after schema change - T85757
16:15 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1100 (duration: 00m 52s)
16:09 banyek: depooling db1100 for schema change - T85757
15:49 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.33.0-wmf.8
15:45 papaul: re-installing OS on ms-be2047.codfw.wmnet
15:31 zfilipin@deploy1001: Finished scap: testwiki to php-1.33.0-wmf.8 and rebuild l10n cache (duration: 36m 08s)
15:06 vgutierrez: Use certcentral managed TLS certificate in mirrors.wikimedia.org - T207050
14:55 zfilipin@deploy1001: Started scap: testwiki to php-1.33.0-wmf.8 and rebuild l10n cache
14:50 zfilipin@deploy1001: Pruned MediaWiki: 1.33.0-wmf.3 (duration: 02m 51s)
14:49 banyek: depooling labsdb1010 - T210693
14:46 zfilipin@deploy1001: Pruned MediaWiki: 1.33.0-wmf.2 (duration: 03m 12s)
14:43 zfilipin@deploy1001: Pruned MediaWiki: 1.33.0-wmf.1 (duration: 11m 40s)
14:42 volans: 'sudo systemctl reload icinga' on icinga1001
14:32 godog: decommissioning cassandra-c, restbase2005 -- T210843
14:31 bblack: removed unused public IPv6 IPs from authdnses manually with "ip -6 addr del ..." - https://gerrit.wikimedia.org/r/c/operations/puppet/+/478939
14:23 filippo@puppetmaster1001: conftool action : set/pooled=yes; selector: name=restbase2013.codfw.wmnet
14:13 mobrovac@deploy1001: Finished deploy [restbase/deploy@44e0955]: Bring restbase201[3-8] up to date (duration: 00m 38s)
14:13 mobrovac@deploy1001: Started deploy [restbase/deploy@44e0955]: Bring restbase201[3-8] up to date
14:10 mobrovac@deploy1001: Finished deploy [restbase/deploy@44e0955]: Bring restbase201[3-8] up to date - T211416 (duration: 01m 53s)
14:08 mobrovac@deploy1001: Started deploy [restbase/deploy@44e0955]: Bring restbase201[3-8] up to date - T211416
13:48 vgutierrez: Use certcentral TLS managed certificate in lists.wikimedia.org - T207050
13:06 zeljkof: EU SWAT finished
13:02 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: HD Logos: Add fr and fy wikibooks and fr wiiknews variants to InitaliseSettings.php (T150618) (duration: 00m 46s)
12:52 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Update settings to include new HD logos (T150618) (duration: 00m 47s)
12:50 hoo@deploy1001: Finished deploy [wdqs/wdqs@f914415]: Fix WDQS UI embeds (T211629) (duration: 10m 31s)
12:39 hoo@deploy1001: Started deploy [wdqs/wdqs@f914415]: Fix WDQS UI embeds (T211629)
12:38 bblack: Authdns CI/config refactoring done, all is well, resume normal DNS ops!
12:37 elukey: updated nodejs nodejs-legacy on aqs1004 (security upgrades)
12:36 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: HD Logos: Add 1.5x and 2x variants of fr and fy wikibooks and fr wikinews (T150618) (duration: 00m 46s)
12:16 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: Add HD logos for 3 projects (T150618) (duration: 00m 47s)
12:12 marostegui: Repool labsdb1011 - T86338
11:59 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@dcde39f]: GUI update (duration: 01m 05s)
11:58 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@dcde39f]: GUI update
11:39 bblack: puppet disabled on authdnses for attempting https://gerrit.wikimedia.org/r/q/topic:%22authdns-ci%22
11:23 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1087 T86338 T202167 (duration: 00m 46s)
11:10 marostegui: Depool labsdb1011 - T86338
11:06 marostegui: Repool labsdb1010 - T86338
11:03 volans: restarted pdfrender on scb1003 [last time, we need an automatic restart]
10:47 fsero: pooling mw1272
10:42 fsero: scap pull mw1272
09:30 ema: mw1272 down for the past 12h. Nothing in console, power-cycling
09:08 marostegui: Deploy schema change on db1087 with replication (this will generate lag on labsdb:s8) T202167 T86338
09:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1087 T86338 T202167 (duration: 00m 46s)
09:04 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1092 T86338 T202167 (duration: 00m 46s)
09:01 marostegui: Depool labsdb1010 - T86338
08:15 marostegui: Deploy schema change on db1092 T202167 T86338
08:15 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1092 T86338 T202167 (duration: 00m 46s)
08:10 godog: decommissioning cassandra-b, restbase2005 -- T210843
07:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1104 T86338 T202167 (duration: 00m 46s)
07:32 oblivian@deploy1001: Synchronized wmf-config/PhpAutoPrepend.php: Hotfix for logging on php7 (2/2) (duration: 02m 51s)
07:29 oblivian@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=mw1272.*
07:28 oblivian@deploy1001: Synchronized wmf-config/php7.php: Hotfix for logging on php7 (1/2) (duration: 02m 50s)
07:06 marostegui: Deploy schema change on db1104 T202167 T86338
07:06 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1104 T86338 T202167 (duration: 02m 51s)
06:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1109 T86338 T202167 (duration: 02m 52s)
06:45 marostegui: Rename flaggedrevs tables on srwikinews on db1078 - T209761
06:13 marostegui: Deploy schema change on db1109 T202167 T86338
06:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1109 T86338 T202167 (duration: 02m 55s)
05:57 marostegui: Deploy schema change on s4 primary master (db1068) T202167 T86338
01:03 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Configure localized logos for nywiki (T211570) (duration: 01m 36s)
01:01 catrope@deploy1001: Synchronized static/images/project-logos/: Add localised logos for nywiki (T211570) (duration: 01m 00s)
00:52 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Use new HD logos for zhwiktionary, zhwikivoyage, zhwikinews (T150618) (duration: 01m 16s)
00:50 RoanKattouw: mw1272 is down (does not respond to ping), but scap still tries to deploy to it
00:50 catrope@deploy1001: Synchronized static/images/project-logos/: Add HD logos for zhwikinews, zhwikivoyage, zhwiktionary (T150618) (duration: 02m 30s)
00:15 mutante: icinga2001 - killed all nagios processes, restarted nsca service, something is different from icinga1001, service failed when trying to restart (T211641)

2018-12-10

23:51 andrewbogott: silencing the kvm process count alert on cloudvirt1023 until I can figure out why it's misfiring
22:13 mutante: Welcome new Mediawiki deployer Christoph 'WMDE-Fisch' Jauera (T211014)
21:29 arlolra@deploy1001: Finished deploy [parsoid/deploy@dc9b3a1]: Updating Parsoid to 19560da (duration: 11m 15s)
21:20 ladsgroup@deploy1001: Finished deploy [ores/deploy@03b9c98]: Add celery4 configs back to the deploy repo (duration: 15m 25s)
21:19 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@9f4b567]: More internal promisification and other performance tweaks (T202642) (duration: 04m 17s)
21:17 arlolra@deploy1001: Started deploy [parsoid/deploy@dc9b3a1]: Updating Parsoid to 19560da
21:14 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@9f4b567]: More internal promisification and other performance tweaks (T202642)
21:05 ladsgroup@deploy1001: Started deploy [ores/deploy@03b9c98]: Add celery4 configs back to the deploy repo
20:35 cdanis: T210416: grafana.wikimedia.org switched to point to grafana1001.eqiad.wmnet (running grafana 5.4.1)
20:32 jforrester@deploy1001: Synchronized wmf-config/extension-list: Uninstall the ParserMigration extension, Part III I332939809 (duration: 00m 46s)
20:30 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Uninstall the ParserMigration extension, Part II I1f7266f55a (duration: 00m 46s)
20:29 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: Uninstall the ParserMigration extension, Part I I338a3d8a87fd (duration: 00m 47s)
20:26 cdanis: T210416: switching grafana.wikimedia.org to point to grafana1001.eqiad.wmnet
20:25 robh: messing with ulsfo power for 103.02.23 tower b, shouldnt disrupt anything T209101
20:20 cdanis: T210416: setting grafana.wikimedia.org (currently served by krypton) to read-only and copying to grafana1001 (serving grafana-beta)
20:13 urandom: decommissioning cassandra-a, restbase2005 -- T210843
19:58 cdanis: T210416: updating grafana to 5.4.1 in stretch-wikimedia: reprepro --restrict grafana update stretch-wikimedia
18:15 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@dcde39f]: GUI Update (duration: 09m 31s)
18:05 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@dcde39f]: GUI Update
17:59 banyek: restarting mysql instance on labsdb1004 to restore replication filters to the original state - T211210
17:58 banyek: restarting mysql instance on labsdb1004 to restore replication filters to the original state
17:29 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: T211527 Hot-deploy Disable ParserMigration now that Raggett has been dropped (duration: 00m 47s)
16:03 moritzm: installing PHP updates on netmon1002
15:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 T86338 T202167 (duration: 00m 46s)
15:36 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1097:3315 (duration: 00m 45s)
15:31 banyek: repooling db1097:3315 after schema change - T85757
15:24 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1097:3315 (duration: 00m 46s)
15:13 banyek: depooling db1097:3315 on a schema change - T85757
15:05 anomie@deploy1001: Synchronized php-1.33.0-wmf.6/includes/user/User.php: Backport fix for T210621 (duration: 00m 46s)
14:55 marostegui: Deploy schema change db1101:3318 T86338 T202167
14:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101:3318 T86338 T202167 (duration: 00m 46s)
14:50 _joe_: uploading php-mongodb 1.5.3 to stretch-wikimedia thirdparty/php72 T206152
14:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099 T86338 T202167 (duration: 00m 47s)
14:00 marostegui: Deploy schema change db1099:3318 T86338 T202167
14:00 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099 T86338 T202167 (duration: 00m 46s)
13:34 ema: trafficserver 8.0.1-1wm1 uploaded to stretch-wikimedia T207048
12:56 gilles@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T187299 T197607 Oversample performance survey on specific ruwiki articles (duration: 00m 46s)
12:49 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 46s)
12:48 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 46s)
12:43 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add http://idb.ub.uni-tuebingen.de/digitue to the wgCopyUploadsDomains (T211466) (duration: 00m 47s)
12:37 zfilipin@deploy1001: Synchronized dblists/flaggedrevs.dblist: SWAT: Remove FlaggedRevs for ptwikipedia (T211433) (duration: 00m 46s)
12:25 moritzm: installing imagemagick security update for jessie
12:21 zfilipin@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: Set FileImporter config help location (T199108) (duration: 00m 47s)
12:20 fsero: running puppet agent on icinga to add fsero
12:13 hoo@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Remove the "wikibase-debug" log channel (T207850) (duration: 00m 47s)
11:04 godog: decommissioning cassandra-c, restbase2004 -- T210843
10:42 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 46s)
10:41 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 52s)
10:39 marostegui: Deploy schema change on db1116:3318 T86338 T202167
10:36 marostegui: Deploy schema change on dbstore1002:s8 T86338 T202167
10:35 marostegui: Repool labsdb1011 T86338
09:07 marostegui: Deploy schema change on s8 codfw master with replication (db2045) - lag will be generated on codfw - T202167 T86338
09:01 marostegui: Depool labsdb1011 - T86338
08:58 moritzm: installing chromium security updates on proton*
08:57 marostegui: Repool labsdb1010 - T86338
08:39 elukey: roll restart of aqs on aqs100* to pick up new Druid backend settings
08:36 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1121 T86338 T202167 (duration: 00m 51s)
08:23 godog: final round of weight addition to new ms-be codfw hosts - T209395
07:18 _joe_: reenabling puppet given my changes were useless
06:52 _joe_: running puppet on the puppetmasters in codfw, twice, then restarting apache to ensure cleanup of any cache
06:50 _joe_: disabled puppet across the fleet for merge of hiera change
06:47 marostegui: Stop slave on s4 on labsdb1011
06:25 marostegui: Deploy schema change on db1121 with replication (this will generate lag on labs) - T86338 T202167
06:24 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1121 T86338 T202167 (duration: 00m 49s)
06:23 marostegui: Reload haproxy on dbproxy1010 to depool labsdb1010 - T86338
02:53 urandom: decommissioning cassandra-b, restbase2004 -- T210843
00:55 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Ic07ff9acfbe17 - T211529, T205546 (duration: 00m 47s)

2018-12-09

20:16 urandom: decommissioning cassandra-a, restbase2004 -- T210843
18:39 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Icb9ad2f554e1 - Fix ruwikinews logo config (duration: 00m 57s)
11:55 godog: decommissioning cassandra-c, restbase2003 -- T210843
03:00 urandom: decommissioning cassandra-b, restbase2003 -- T210843

2018-12-08

23:07 krinkle@deploy1001: Synchronized docroot/wikipedia.org/speed-tests/: T185446 - I6cf29d598a11 (duration: 00m 47s)
22:21 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T209794 (duration: 00m 56s)
20:59 urandom: decommissioning cassandra-a, restbase2003 -- T210843
13:54 godog: decommissioning cassandra-c, restbase2002 -- T210843

2018-12-07

23:54 ejegg: updated payments-wiki from b99cd0816e to b8acb95a2a
23:41 urandom: decommissioning cassandra-b, restbase2002 -- T210843
22:43 ejegg: re-enabled Thank You mail sender
22:31 ejegg: updated fundraising CiviCRM from 3e5d74f17e to 8e18485697
22:29 ejegg: Turned off Thank You mailing job for letter update
17:47 herron: rebooting logstash1006 for security updates
17:17 bstorm_: T207377 rebooted labstore1007 for kernel upgrades
17:15 _joe_: uploading php-tideways (rebuilt with php 7.2 support) to stretch-wikimedia thirdparty/php72 T206152
15:37 moritzm: rebooting sarin/neodymium
14:52 urandom: decommissioning cassandra-a, restbase2002 -- T210843
14:21 godog: more weight to new ms-be codfw hosts - T209395
13:37 moritzm: rolling reboot of scb in codfw (along with nodejs update)
12:18 moritzm: installing nodejs security updates on stat/notebook
12:05 mobrovac@deploy1001: Finished deploy [restbase/deploy@44e0955]: Fix: Encode recommendation api title (duration: 21m 11s)
11:43 mobrovac@deploy1001: Started deploy [restbase/deploy@44e0955]: Fix: Encode recommendation api title
11:42 mobrovac@deploy1001: Finished deploy [restbase/deploy@9e4af13]: Fix: Encode recommendation api title (duration: 00m 21s)
11:42 mobrovac@deploy1001: Started deploy [restbase/deploy@9e4af13]: Fix: Encode recommendation api title
11:41 mobrovac@deploy1001: Finished deploy [restbase/deploy@9e4af13]: Fix: Encode recommendation api title (duration: 18m 58s)
11:22 mobrovac@deploy1001: Started deploy [restbase/deploy@9e4af13]: Fix: Encode recommendation api title
11:20 moritzm: rolling upgrade of nginx on swift frontends
11:19 mobrovac@deploy1001: Finished deploy [restbase/deploy@31c44e8]: Fix: Encode recommendation api title (duration: 03m 49s)
11:15 mobrovac@deploy1001: Started deploy [restbase/deploy@31c44e8]: Fix: Encode recommendation api title
11:09 mobrovac@deploy1001: Finished deploy [citoid/deploy@269c9c7]: Add an explicit check for Zotero (duration: 06m 19s)
11:03 mobrovac@deploy1001: Started deploy [citoid/deploy@269c9c7]: Add an explicit check for Zotero
10:58 mobrovac@deploy1001: Finished deploy [citoid/deploy@6b36331]: Add an explicit check for Zotero (duration: 02m 42s)
10:55 mobrovac@deploy1001: Started deploy [citoid/deploy@6b36331]: Add an explicit check for Zotero
09:40 banyek: importing back linkwatcher_linklog into database s51230__linkwatcher on host labsdb1004.eqiad.wmnet. - T211210
09:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1096:3315, db1096:3316 after kernel and mysql upgrade (duration: 00m 46s)
09:10 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1096:3315, db1096:3316 after kernel and mysql upgrade (duration: 00m 46s)
08:39 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1096:3315, db1096:3316 after kernel and mysql upgrade (duration: 00m 46s)
08:21 marostegui: Stop MySQL on db1096:3315,3316 for kernel and mysql upgrade
08:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3315, db1096:3316 for kernel and mysql upgrade (duration: 00m 46s)
08:15 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1084 (duration: 00m 47s)
07:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1084 (duration: 00m 46s)
07:50 godog: decommissioning cassandra-c, restbase2001 -- T210843
07:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1084 (duration: 00m 46s)
07:23 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1084 (duration: 00m 46s)
07:11 marostegui: Stop MySQL on db1084 for mysql and kernel upgrade
07:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084 (duration: 00m 49s)
00:47 XioNoX: done troubleshoting bird bfd on dns2001/cr1-codfw
00:10 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: turn off wbsearchentities ab test T209402 (duration: 00m 47s)

2018-12-06

23:47 ppchelko@deploy1001: Finished deploy [recommendation-api/deploy@299b268]: Add 'morelike' article recommendations API T201192 (duration: 02m 06s)
23:47 XioNoX: troubleshoot bird bfd on dns2001/cr1-codfw
23:45 ppchelko@deploy1001: Started deploy [recommendation-api/deploy@299b268]: Add 'morelike' article recommendations API T201192
23:21 ppchelko@deploy1001: Finished deploy [restbase/deploy@be8f0c0]: Add 'morelike' recommendation public API specification T201192 (duration: 22m 46s)
22:58 ppchelko@deploy1001: Started deploy [restbase/deploy@be8f0c0]: Add 'morelike' recommendation public API specification T201192
22:12 urandom: decommissioning cassandra-b, restbase2001 -- T210843
21:39 gtirloni: reimaging cloudvirt1019 with jessie T196507
21:33 ppchelko@deploy1001: Finished deploy [changeprop/deploy@f675fcc]: Added performer to the revision-scores event (duration: 01m 15s)
21:32 ppchelko@deploy1001: Started deploy [changeprop/deploy@f675fcc]: Added performer to the revision-scores event
21:07 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@cbe4551]: Install new Updater with INSERT DATA (duration: 09m 18s)
21:00 XioNoX: remove 2 esams avoid path + 4 prefered/selected transits - T194542
20:58 smalyshev@deploy1001: Started deploy [wdqs/wdqs@cbe4551]: Install new Updater with INSERT DATA
20:51 XioNoX: remove 2 eqiad avoid path - T194542
20:48 gtirloni: reimaging cloudvirt1019 with stretch T196507
20:45 XioNoX: remove codfw/eqdfw avoid path - T194542
19:32 gehel: shutting down elasticsearch on elastic2001-2024 (third time is a charm) - T211023
18:21 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@1dba3cd]: Internally promisify page processing steps (T202642) (duration: 03m 54s)
18:17 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@1dba3cd]: Internally promisify page processing steps (T202642)
17:31 moritzm: installing nodejs updates on proton*
17:11 moritzm: uploaded nodejs 6.11~dfsg-1+wmf5 for jessie-wikimedia (the upstream patch for CVE-2018-12122 had a regression, this update fixes it)
16:27 urandom: decommissioning cassandra-a, restbase2001 -- T210843
16:17 gehel: shutting down elasticsearch on elastic2001-2024 (second try) - T211023
15:51 moritzm: upgrading spamassassin on mx1001/fermium
15:45 fsero@deploy1001: scap-helm zotero finished
15:45 fsero@deploy1001: scap-helm zotero cluster codfw completed
15:45 fsero@deploy1001: scap-helm zotero upgrade production -f ../zotero-values-codfw.yaml stable/zotero [namespace: zotero, clusters: codfw]
15:42 fsero: modifying zotero deploy CLUSTER=codfw scap-helm zotero upgrade production -f zotero-values-codfw.yaml stable/zotero - T211322
15:42 _joe_: disabling puppet fleet-wide for a change in the role() function
15:38 moritzm: uploaded nodejs 6.11~dfsg-1+wmf5 for stretch-wikimedia (the upstream patch for CVE-2018-12122 had a regression, this update fixes it)
15:08 gehel: restartign new elasticsearch masters on codfw - T211023
15:05 gehel: upgrade nginx on wdqs servers
14:59 elukey@deploy1001: Finished deploy [analytics/turnilo/deploy@6bd6e2f]: upgrade deps to nodejs 10 (duration: 00m 09s)
14:59 elukey@deploy1001: Started deploy [analytics/turnilo/deploy@6bd6e2f]: upgrade deps to nodejs 10
14:46 moritzm: uploaded nodejs 10.4.0~dfsg-1+wmf2 to apt.wikimedia.org/component/node10 (backports of recent security fixes)
14:16 moritzm: installing nginx security updates on mw in eqiad
13:46 moritzm: upgrading spamassassin on mx2001
12:56 gehel: depooling and shutting down elasticsearch on elastic2001-2024 - T211023
12:55 moritzm: installing nginx updates on mw in codfw
11:59 moritzm: installing nginx updates on mw canaries
10:59 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1096:3315 (duration: 00m 47s)
10:56 volans: disable event handler on Icinga for ms-be2047 MD Raid and MegaRAID checks, it's spamming Phabricator - T209921
10:56 banyek: repooling db1096 for schema change - T85757
10:36 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1096:3315 (duration: 00m 49s)
10:29 banyek: depooling db1096 for schema change - T85757
09:56 dcausse: elastic@codfw cleanup: deleting wikidatawiki_content_1537469318 index (failed reindex probably)
01:03 tstarling@deploy1001: Synchronized w/fatal-error.php: (no justification provided) (duration: 00m 46s)
00:55 tstarling@deploy1001: Synchronized w/fatal-error.php: (no justification provided) (duration: 00m 47s)
00:42 tstarling@deploy1001: Synchronized w/fatal-error.php: (no justification provided) (duration: 00m 46s)
00:40 tstarling@deploy1001: Synchronized private/FatalErrorSettings.php: (no justification provided) (duration: 00m 46s)
00:38 tstarling@deploy1001: Synchronized private/FatalErrorSettings.php: (no justification provided) (duration: 00m 46s)
00:15 mutante: MPM prefork tweaks for high load systems are applied again (apparently they were not since a change in the past that resulted in 2 competing configs in mods-enabled and conf-enabled with the latter one being loaded last and containing the package defaults
00:13 mutante: re-enabling puppet on phabricator, applying change that adds php-fpm support on stretch ..which doesnt affect phab1001 (prod) on jessie.. BUT re-adds tuning config from the past for mpm_prefork.conf (more SpareServers etc) that was not actually applied due to a bug
00:10 urandom: bootstrapping cassandra-c, restbase2018 -- T210843

2018-12-05

23:50 ejegg: updated payments-wiki from 20595cca97 to b99cd0816e
23:40 ejegg: re-enabled fundraising queue consumer jobs
23:33 ejegg: updated fundraising CiviCRM from e757753a46 to 3e5d74f17e
23:32 ejegg: turned off fundraising queue jobs for base queue consumer logic update
22:43 jijiki: restarting pdfreder on scb* hosts in eqiad
21:44 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@243a503]: Update mobileapps to 2f44362 (duration: 02m 47s)
21:41 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@243a503]: Update mobileapps to 2f44362
21:41 mdholloway: mobileapps deployment failed for group default03, rolling back and retrying
21:39 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@243a503]: Update mobileapps to 2f44362 (duration: 18m 46s)
21:39 arlolra: Updated Parsoid to a6058e3 (T210647, T208360, T205333)
21:37 urandom: bootstrapping cassandra-b, restbase2018 -- T210843
21:23 arlolra@deploy1001: Finished deploy [parsoid/deploy@5e9a496]: Updating Parsoid to a6058e3 (duration: 11m 36s)
21:20 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@243a503]: Update mobileapps to 2f44362
21:12 arlolra@deploy1001: Started deploy [parsoid/deploy@5e9a496]: Updating Parsoid to a6058e3
20:08 banyek: repooling labsdb1010 - T210693
19:12 cmjohnson1: cloudvirt1019 for an all inclusive part swap by HPE
18:53 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: Fix formatting of help panel links (duration: 00m 47s)
18:01 jijiki: uploaded thumbor_6.3.2+git20170607-1+deb9u1 to stretch-wikimedia
17:23 hoo@deploy1001: Synchronized wmf-config/Wikibase.php: Enable Kartographer maps on testwikidatawiki (T184933) (duration: 00m 46s)
17:22 hoo@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: (no justification provided) (duration: 00m 46s)
17:20 hoo@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: (no justification provided) (duration: 00m 46s)
17:19 XioNoX: remove private IPs from codfw cloud-instance-transport1-b T207663
17:11 XioNoX: add public IPs to codfw cloud-instance-transport1-b T207663
16:58 XioNoX: re-deactivate ams-ix prefix list entry on cr2-esams
16:53 XioNoX: activate ams-ix prefix list entry on cr2-esams
16:27 jijiki: uploaded python-thumbor-wikimedia_2.2-1+deb9u1 to stretch-wikimedia
16:18 akosiaris@deploy1001: scap-helm zotero finished
16:18 akosiaris@deploy1001: scap-helm zotero cluster codfw completed
16:18 akosiaris@deploy1001: scap-helm zotero upgrade production -f zotero-values-codfw.yaml stable/zotero [namespace: zotero, clusters: codfw]
16:17 akosiaris@deploy1001: scap-helm zotero finished
16:17 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
16:17 akosiaris@deploy1001: scap-helm zotero upgrade production -f zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
16:17 akosiaris@deploy1001: scap-helm zotero finished
16:17 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
16:17 akosiaris@deploy1001: scap-helm zotero upgrade production -f zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
16:15 akosiaris@deploy1001: scap-helm zotero finished
16:15 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
16:15 akosiaris@deploy1001: scap-helm zotero upgrade production -f zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
16:08 fsero: redeploying zotero on eqiad
16:02 akosiaris@deploy1001: scap-helm zotero finished
16:02 akosiaris@deploy1001: scap-helm zotero cluster codfw completed
16:02 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
16:02 akosiaris@deploy1001: scap-helm zotero upgrade production --set resources.replicas=16 stable/zotero [namespace: zotero, clusters: eqiad,codfw]
16:02 akosiaris@deploy1001: scap-helm zotero upgrade production --set resources.replicas=16 [namespace: zotero, clusters: eqiad,codfw]
15:34 thcipriani: restarting ci jenkins for update
15:33 akosiaris: add back pods/portforward right to kubernetes deploy user. T211040
15:07 anomie: Running cleanupUsersWithNoId.php on potentially missed s3 and s7 wikis for T181731
15:03 fsero: repooling citoid mathoid eqiad
15:03 godog: bootstrap cassandra-c, restbase2017 - T210843
14:56 banyek: executing schema change on s5 codfw master replication lag could be expected - T85757
14:54 fsero: upgrading k8s on eqiad to 1.10.11
14:54 elukey: restart HDFS namenode and Yarn resource manager on an-master100[1,2] to update rack topology config - T209929
14:51 fsero: depool mathoid/eqiad: pooled changed True => False
14:51 fsero: depool citoid/eqiad: pooled changed True => False
14:43 anomie: Running cleanupUsersWithNoId.php on metawiki for T181731 / T210985
14:09 hoo@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable arbitrary item/property access for all wiktionaries (T175273) (duration: 00m 47s)
13:53 akosiaris: repool citoid/mathoid codfw
12:59 onimisionipe: banning elastic2001-elastic2024 from codfw production, psi and omega clusters
12:53 jijiki: uploaded python-thumbor-community-core_0.4.0-1+deb9u1 to stretch-wikimedia
12:47 dcausse: EU SWAT done
12:45 oblivian@puppetmaster1001: conftool action : set/pooled=false; selector: dnsdisc=(cit|math)oid,name=codfw
12:44 dcausse@deploy1001: Synchronized wmf-config/CirrusSearch-production.php: T210381: [cirrus] Add temp clusters but still write to the old ones 2/2 (duration: 00m 46s)
12:42 dcausse@deploy1001: Synchronized wmf-config/CommonSettings.php: T210381: [cirrus] Add temp clusters but still write to the old ones 1/2 (duration: 00m 46s)
12:31 fsero: pooling mathoid and citoid again on codfw
12:30 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T202497) (duration: 00m 46s)
12:29 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T202497) (duration: 00m 49s)
12:23 dcausse@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T211188: Increase autoconfirmed count for Meta-Wiki to 5 (duration: 00m 47s)
12:15 fsero: upgrading codfw k8s cluster to 1.10.11
12:15 dcausse: running namespaceDupes & cirrus indexNamespaces on yuewiktionary
12:11 dcausse@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T205546: Define 2 new namespaces for yuewiktionary (duration: 00m 47s)
12:02 fsero: depooling mathoid and citoid servers on codfw for k8s upgrade
11:28 mobrovac@deploy1001: Finished deploy [citoid/deploy@b10e034]: Truncate Zotero-reported time stamp to date - T211127 (duration: 05m 55s)
11:23 mobrovac@deploy1001: Started deploy [citoid/deploy@b10e034]: Truncate Zotero-reported time stamp to date - T211127
11:05 akosiaris: upgrade kubernetes-client and kubernetes-master on staging to 1.10.11
11:04 godog: bootstrap cassandra-b, restbase2017 - T210843
10:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3312 (duration: 00m 45s)
10:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3317 (duration: 00m 45s)
10:51 ema: cache hosts: begin nginx rolling upgrade to 1.13.6-2+wmf2
10:46 marostegui: Reboot db1090 for kernel upgrade
10:44 moritzm: uploaded jenkins 2.138.4 to jessie-wikimedia/thirdparty and stretch-wikimedia/thirdpary/ci
10:42 marostegui: Stop MySQL on db1090:3312 and db1090:3317 for MySQL upgrade
10:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3312 (duration: 00m 46s)
10:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3317 (duration: 00m 46s)
10:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1080 (duration: 00m 46s)
10:21 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=elasticsearch
10:17 arturo: T205969 icinga downtime the load avg check in labstore1007 for 1 week
10:10 banyek: depooling labsdb1010 for testing materialized views - T210693
10:02 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1080 (duration: 00m 46s)
09:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly pool db1080 (duration: 00m 46s)
09:32 gehel: setting up new elasticsearch servers on codfw - elastic2045-2054 - T210265
09:22 marostegui: Stop MySQL on db1080 for mysql and kernel upgrade
09:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1080 for MySQL upgrade (duration: 00m 46s)
09:07 elukey: matomo read only + upgrade to matomo 3.7.0 on matomo1001 - T209808
09:00 _joe_: disabed puppet on mw1261, used for logging tests for T211184
08:48 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T207850 Define a new Wikibase log channel to use (duration: 00m 47s)
08:01 moritzm: installing pdns-recursor security update in esams
07:59 godog: bootstrap cassandra-a, restbase2017 - T210843
07:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1091 T86338 T202167 (duration: 00m 46s)
06:55 marostegui: Deploy schema change on db1091 T86338 T202167
06:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1091 T86338 T202167 (duration: 00m 51s)
04:03 urandom: bootstrapping cassandra-c, restbase2016 -- T210843
03:11 kartik@deploy1001: Finished deploy [cxserver/deploy@a3dd2ca]: Update cxserver to c4240e6 and enable Youdao MT (T208985, T210578) (duration: 04m 26s)
03:06 kartik@deploy1001: Started deploy [cxserver/deploy@a3dd2ca]: Update cxserver to c4240e6 and enable Youdao MT (T208985, T210578)
00:32 dcausse: Evening SWAT done
00:30 dcausse@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [cirrus] prepare multi-instance services (T210381) (duration: 00m 46s)
00:28 dcausse@deploy1001: Synchronized wmf-config/ProductionServices.php: [cirrus] prepare multi-instance services (T210381) (duration: 00m 46s)
00:16 dcausse@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable Block notice stats on itwiki (T210452) (duration: 00m 47s)
00:10 urandom: bootstrapping cassandra-b, restbase2016 -- T210843

2018-12-04

23:46 eileen: civicrm revision changed from a411d6bd64 to e757753a46, config revision is 0e6ccc37fe
23:33 XioNoX: update prefix-list peering4 on cr1-eqsin to match jnt
22:46 XioNoX: remove neodymium/sarin from term labs-in4 on cr1/2-eqiad - T210612
22:40 ejegg: Updated payments-wiki from 7403a196b4 to 20595cca97
21:58 XioNoX: clear ethernet-swtiching table for labvirt1004:eth1's switch port
21:57 XioNoX: clear ethernet-swtiching table for labvirt1009:eth1's switch port
21:49 XioNoX: make cr1/2-codfw conform to jnt
21:44 XioNoX: make cr2-eqord/eqdfw conform to jnt
21:41 XioNoX: make cr3/4-ulsfo conform to jnt
20:04 urandom: bootstrapping cassandra-a, restbase2016 -- T210843
19:35 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@81dac18]: Install new Updater for T210044 investigation (duration: 10m 36s)
19:24 smalyshev@deploy1001: Started deploy [wdqs/wdqs@81dac18]: Install new Updater for T210044 investigation
18:04 joal@deploy1001: Finished deploy [analytics/aqs/deploy@e7d48e9]: Add underestimate and offset to uniques-devices endpoint (duration: 17m 33s)
18:03 akosiaris: bump zotero pod number from 4 to 16 in eqiad/codfw
17:47 joal@deploy1001: Started deploy [analytics/aqs/deploy@e7d48e9]: Add underestimate and offset to uniques-devices endpoint
17:46 ppchelko@deploy1001: Finished deploy [changeprop/deploy@e1aeb27]: Do not initialize scores and errors arrays in advance T210465 (duration: 01m 13s)
17:45 ppchelko@deploy1001: Started deploy [changeprop/deploy@e1aeb27]: Do not initialize scores and errors arrays in advance T210465
17:22 Reedy: created oathauth tables on punjabiwikimedia T211110
17:16 godog: bootstrap cassandra-c on restbase2015 - T210843
16:48 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Configure 'api-warning' log channel (duration: 00m 47s)
15:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3314 T86338 T202167 (duration: 00m 47s)
14:04 elukey: upgrade turnilo on analytics-tools1002 to nodejs-10 - T210705
13:56 addshore: addshore@mwmaint1002:~$ mwscript namespaceDupes.php --wiki=bnwikisource --fix --add-prefix=T210472
13:53 addshore: addshore@mwmaint1002:~$ mwscript namespaceDupes.php --wiki=euwiki --fix
13:52 marostegui: Deploy schema change on db1103:3314 T86338 T202167
13:52 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3314 T86338 T202167 (duration: 00m 47s)
13:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1081 T86338 T202167 (duration: 00m 46s)
13:34 cdanis: T210416: adding grafana 5 to wikimedia-stretch: reprepro --restrict grafana update stretch-wikimedia
13:33 moritzm: installing nodejs security updates on restbase in codfw
13:32 godog: bootstrap cassandra-b on restbase2015 - T210843
13:29 moritzm: installing nodejs security updates on proton*
13:19 marostegui: Deploy schema change on db1081 T86338 T202167
13:19 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1081 T86338 T202167 (duration: 00m 46s)
13:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1084 T86338 T202167 (duration: 00m 46s)
12:39 Lucas_WMDE: EU SWAT done
12:38 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create namespace "Work" on bnwikisource (T210472) (duration: 00m 46s)
12:33 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create List namespace on euwiki (T209834) (duration: 00m 47s)
12:28 lucaswerkmeister-wmde@deploy1001: Synchronized static/images/project-logos/: SWAT: Revert "Milestone logo for atjwiki" (T200713) (duration: 00m 47s)
12:17 moritzm: installing tiff security updates
11:49 mobrovac@deploy1001: Finished deploy [restbase/deploy@8abcbda] (dev-cluster): (no justification provided) (duration: 04m 47s)
11:44 mobrovac@deploy1001: Started deploy [restbase/deploy@8abcbda] (dev-cluster): (no justification provided)
11:41 moritzm: rebooting puppetboard2001 to pick up SSBD-enabled qemu
11:38 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1004 - T197242 (duration: 00m 21s)
11:38 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1004 - T197242
11:37 moritzm: rebooting puppetboard1001 to pick up SSBD-enabled qemu
11:36 akosiaris: enable puppet on scb1004, run puppet T197242
11:36 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1003 - T197242 (duration: 00m 20s)
11:35 marostegui: Deploy schema change on db1084 T86338 T202167
11:35 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1003 - T197242
11:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084 T86338 T202167 (duration: 00m 47s)
11:34 akosiaris: enable puppet on scb1003, run puppet T197242
11:33 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1002 - T197242 (duration: 00m 28s)
11:33 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1002 - T197242
11:31 akosiaris: enable puppet on scb1002, run puppet T197242
11:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3314 T86338 T202167 (duration: 00m 46s)
11:18 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1001 - T197242 (duration: 00m 30s)
11:18 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1001 - T197242
11:17 akosiaris: enable puppet on scb1001, run puppet T197242
11:12 elukey@deploy1001: Finished deploy [analytics/aqs/deploy@e9a63cc]: Expose offset and underestimate numbers on unique devices - T164201 (duration: 09m 06s)
11:04 mobrovac@deploy1001: Finished deploy [restbase/deploy@8abcbda]: Disable Citoid test for switching it to Zotero v2 - T211088 T197242 (duration: 20m 59s)
11:03 elukey@deploy1001: Started deploy [analytics/aqs/deploy@e9a63cc]: Expose offset and underestimate numbers on unique devices - T164201
10:59 fdans@deploy1001: Finished deploy [analytics/aqs/deploy@e9a63cc]: Deploying offset and underestimate numbers for uniques (duration: 00m 37s)
10:58 fdans@deploy1001: Started deploy [analytics/aqs/deploy@e9a63cc]: Deploying offset and underestimate numbers for uniques
10:57 fdans: deploying AQS to expose offset and underestimate numbers on unique devices
10:51 moritzm: rebooting analytics-tool1003 to pick up SSBD-enabled qemu
10:47 moritzm: rebooting analytics-tool1002 to pick up SSBD-enabled qemu
10:43 mobrovac@deploy1001: Started deploy [restbase/deploy@8abcbda]: Disable Citoid test for switching it to Zotero v2 - T211088 T197242
10:41 moritzm: rebooting analytics-tool1001 to pick up SSBD-enabled qemu
10:03 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 in codfw - T197242 (duration: 01m 45s)
10:02 godog: bootstrap cassandra-a on restbase2015 - T210843
10:01 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 in codfw - T197242
09:59 gehel: upgrading nginx on elasticsearch eqiad
09:54 akosiaris: enable puppet on all scb2*, run puppet T197242
09:52 gehel: upgrading nginx on elasticsearch codfw
09:52 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb2001 - T197242 (duration: 00m 30s)
09:51 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb2001 - T197242
09:50 akosiaris: enable puppet on scb2001, run puppet T197242
09:46 akosiaris: disable puppet on scb for citoid migration to zoterov2 T197242
09:46 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=elasticsearch
09:31 gehel: add elastic2039-2044 to cirrus eqiad (new server) - T210265
09:11 gehel: add elastic2038 to cirrus eqiad (new server) - T210265
09:00 addshore: graphite1004 & graphite2003, /var/lib/carbon/whisper/MediaWiki/electronpdf/action # Ran https://phabricator.wikimedia.org/P7882 for T157012
08:54 marostegui: Deploy schema change on db1097:3314 T86338 T202167
08:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3314 T86338 T202167 (duration: 00m 47s)
08:46 addshore: graphite1004 & graphite2003, /var/lib/carbon/whisper/daily/wikidata/api/actions$ sudo -u _graphite find . -type f -name "*-*.wsp" -delete # T120639
08:46 addshore: graphite1004 & graphite2003, /var/lib/carbon/whisper/daily/wikidata/api/actions$ sudo -u _graphite find . -type f -name "*_*.wsp" -delete # T120639
08:43 marostegui: Deploy schema change on db1102:3314 T86338 T202167
08:42 marostegui: Deploy schema change on dbstore1002:s4 T86338 T202167
08:41 addshore: graphite1004 & graphite2003, /var/lib/carbon/whisper/daily/wikidata/datamodel$ sudo -u _graphite rm wikipedia_references.wsp # T121521
08:36 gehel: restarting stuck tilerator on maps* - T204047
08:35 addshore: graphite1004 & graphite2003, /var/lib/carbon/whisper/daily/wikidata/api/wbgetclaims$ sudo -u _graphite find . -type f -name "*.wsp" -delete # T140280
08:11 moritzm: installing perl security updates on jessie/trusty (stretch already updated)
07:49 godog: bootstrap cassandra-c on restbase2014 - T209615
07:26 marostegui: Deploy schema change on s4 codfw master (db2051) with replication T86338 T202167
07:22 marostegui: Deploy schema change on wikitech primary master (db1073) for labswiki and labtestwiki T86338 T202167
06:39 marostegui: Deploy schema change on s5 primary master (db1070) T86338 T202167
06:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1110 T86338 T202167 (duration: 00m 49s)
06:12 marostegui: Deploy schema change on db1110 T86338 T202167
06:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1110 T86338 T202167 (duration: 00m 53s)
01:22 foks: Removing 2FA per request at https://phabricator.wikimedia.org/T210703
01:22 foks: Reset password for user "Orangemike"

2018-12-03

23:54 legoktm@deploy1001: Synchronized php-1.33.0-wmf.6/tests/: for completeness (duration: 00m 58s)
23:53 legoktm@deploy1001: Synchronized php-1.33.0-wmf.6/resources/src/mediawiki.legacy/: Restore gray coloring for autocomments (T165189 part 2) (duration: 00m 47s)
23:51 legoktm@deploy1001: Synchronized php-1.33.0-wmf.6/includes/Linker.php: Restore old HTML structure for history section links (T165189 part 1) (duration: 00m 47s)
22:56 sbassett@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/AbuseFilter/includes/api/ApiQueryAbuseLog.php: Deploy security fix for T210329 (duration: 00m 47s)
21:24 mutante: temp. disabling puppet on logstash1007 and logstash1008 to carefully deploy gerrit:476916
21:17 XioNoX: push firewall change to pfw3-eqiad - T211028
21:10 catrope@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/WikimediaEvents/includes/WikimediaEventsHooks.php: Fix ChangesListFilters validation errors (duration: 00m 49s)
21:00 urandom: bootstrapping restbase2014-a -- T210843
20:20 ppchelko@deploy1001: Finished deploy [changeprop/deploy@7470c85]: Start emitting revision-score events with new schema (duration: 01m 13s)
20:19 ppchelko@deploy1001: Started deploy [changeprop/deploy@7470c85]: Start emitting revision-score events with new schema
20:14 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@0c94e5f]: New GUI and updater build (duration: 09m 31s)
20:05 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@0c94e5f]: New GUI and updater build
20:05 XioNoX: push firewall change to pfw3-eqiad - T211028
19:59 ppchelko@deploy1001: Finished deploy [changeprop/deploy@867c571]: TEMP: stop production of revision-scor events for schema change (duration: 01m 13s)
19:58 ppchelko@deploy1001: Started deploy [changeprop/deploy@867c571]: TEMP: stop production of revision-scor events for schema change
19:39 jforrester@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/MobileFrontend/resources/mobile.toc/TableOfContents.js: SWAT T210869 Fix Table of contents rendering (duration: 00m 47s)
19:30 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT I3b906e8b1 CS part of setting enhanced RC (duration: 00m 46s)
19:27 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT Ic2787309e59e IS part of setting enhanced RC (duration: 00m 47s)
19:25 jforrester@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/Echo/modules/styles/mw.echo.ui.PaginationWidget.less: SWAT T210487 I914b94515 (duration: 00m 47s)
19:15 jforrester@deploy1001: Synchronized wmf-config/ProductionServices.php: SWAT T210381 I73c7596818b Actual config (duration: 00m 46s)
19:09 jforrester@deploy1001: Synchronized wmf-config/CirrusSearch-production.php: SWAT T210381 I2ae162f5 Part II (duration: 00m 47s)
19:08 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT T210381 I2ae162f5 Part I (duration: 00m 46s)
18:42 anomie@deploy1001: Synchronized wmf-config/CommonSettings.php: Updating SkinBuildSidebar hook function for T210528 (duration: 00m 47s)
18:08 gehel: add elastic2037 to cirrus eqiad (new server) - T210265
18:06 godog: bootstrap cassandra-c on restbase2013 - T209615
17:09 bstorm_: T207377 reboot labstore1006 for upgrades
16:28 godog: poweroff ms-be2021 for battery replacement - T208269
15:23 gehel: start configuration of elastic2037-2044 (new servers) - T210265
15:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1100 T86338 T202167 (duration: 00m 46s)
14:54 marostegui: Deploy schema change on db1100 T86338 T202167
14:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1100 T86338 T202167 (duration: 00m 48s)
14:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1113:3315 T86338 T202167 (duration: 00m 47s)
14:17 marostegui: Deploy schema change on db1113:3315 T86338 T202167
14:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1113:3315 T86338 T202167 (duration: 00m 46s)
14:07 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1082 T86338 T202167 (duration: 00m 46s)
13:47 marostegui: Deploy schema change on db1082 (sanitarium master) with replication, lag will be generated on labs (s5) T86338 T202167
13:47 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1082 T86338 T202167 (duration: 00m 47s)
13:01 Lucas_WMDE: EU SWAT done
12:44 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Cleaning of wgLogoHD (T150618) (duration: 00m 46s)
12:39 lucaswerkmeister-wmde@deploy1001: Synchronized static/images/project-logos/: SWAT: Upload HD logos for multiple projects (T150618) (duration: 00m 47s)
12:37 moritzm: installing nodejs security updates on scb1001
12:34 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Close internalwiki (T205584) (duration: 00m 46s)
12:28 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Change sitename of shnwiki (T206777) (duration: 00m 47s)
12:23 godog: bootstrap cassandra-b on restbase2013 - T209615
12:19 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Perform more PHP constraint checks before falling back (T209504) (duration: 00m 48s)
12:14 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Don’t send SPARQL prefixes in WikibaseQualityConstraints (T204317) (duration: 00m 49s)
11:39 godog: more weight to new ms-be codfw hosts - T209395
11:02 moritzm: installing nodejs security updates on stat/notebook hosts
10:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3315 T86338 T202167 (duration: 00m 47s)
10:52 moritzm: rolling upgrade of scb in codfw to nodejs security update
10:39 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 46s)
10:39 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 47s)
10:34 moritzm: installing nodejs security updates on scb2001
10:31 marostegui: Deploy schema change on db1097:3315 T86338 T202167
10:30 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3315 T86338 T202167 (duration: 00m 46s)
10:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3315 T86338 T202167 (duration: 00m 45s)
09:48 marostegui: Deploy schema change on db1096:3315 T86338 T202167
09:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3315 T86338 T202167 (duration: 00m 47s)
09:27 banyek: executing schema change on db1066 (s2 master) - T85757
09:22 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1074 (duration: 00m 47s)
09:16 banyek: repooling db1074 - T85757
09:16 banyek: repooling db1074
08:50 banyek: stopping replication on db1074 - T85757
08:49 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1074 (duration: 00m 48s)
08:44 godog: bootstrap cassandra-a on restbase2013 - T209615
08:43 banyek: depooling db1074 - T85757
08:32 moritzm: restarted keyholder agents/proxies on netmon1002/netmon2001 to pick up removal of netbox key
08:30 marostegui: Deploy schema change on s5 codfw master (db2052) with replication, lag will be generated on codfw T86338 T202167
08:13 moritzm: rearmed keyholders on netmon1002/netmon2001
08:07 marostegui: Deploy schema change on s2 master (db1066) T86338 T202167
07:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1074 T86338 T202167 (duration: 00m 47s)
07:32 marostegui: Deploy schema change db1074 with replication (lag will appear on labs) T86338 T202167
07:32 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1074 T86338 T202167 (duration: 00m 46s)
07:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1122 T86338 T202167 (duration: 00m 47s)
07:09 marostegui: Stop MySQL on pc1004, pc1005 and pc1006 as they will be decommissioned - T210969
06:52 marostegui: Remove pc1004, pc1005 and pc1006 from tendril and zarcillo - T210969
06:38 marostegui: Deploy schema change db1122 T86338 T202167
06:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1122 T86338 T202167 (duration: 00m 48s)
06:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1076 T86338 T202167 (duration: 00m 47s)
06:16 marostegui: Deploy schema change db1076 T86338 T202167
06:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1076 T86338 T202167 (duration: 00m 50s)
00:34 legoktm@deploy1001: Synchronized php-1.33.0-wmf.6/includes/Title.php: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/477182 (duration: 00m 52s)

2018-12-02

22:39 addshore: addshore@mwmaint1002:~$ mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=testwikidatawiki addless # This is my account, and apparently I no longer have the 2fa for it

2018-12-01

16:48 andrewbogott: rebuilding labvirt1014 as cloudvirt1014, T210904

2018-11-30

20:23 ottomata: temporarily disabled puppet on stat1005 to test rsyncd changes
20:02 ssastry@deploy1001: Finished deploy [parsoid/deploy@9981ddf]: Update Parsoid to 310edecd (deploy-20181130 branch) (duration: 11m 38s)
19:50 ssastry@deploy1001: Started deploy [parsoid/deploy@9981ddf]: Update Parsoid to 310edecd (deploy-20181130 branch)
15:29 urandom: bootstrapping restbase2013-a -- T210843
15:22 moritzm: uploaded nodejs 6.11.0~dfsg-1+wmf4+jessie to apt.wikimedia.org/jessie-wikimedia (fixes a dependency compared to the initial jessie update)
12:55 moritzm: uploaded nodejs 6.11.0~dfsg-1+wmf3+jessie to apt.wikimedia.org/jessie-wikimedia (backporting the current security fixes)
11:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1105:3312 T86338 T202167 (duration: 00m 47s)
11:07 banyek: deploy schema change on dbstore1002 - T85757
10:59 marostegui: Deploy schema change on db1105:3312 T86338 T202167
10:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1105:3312 T86338 T202167 (duration: 00m 45s)
09:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3312 T86338 T202167 (duration: 00m 46s)
09:00 marostegui: Deploy schema change on db1090:3312 T86338 T202167
09:00 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3312 T86338 T202167 (duration: 00m 47s)
08:55 moritzm: removed rutherfordium from debmonitor DB (T210036)
08:21 marostegui: Deploy schema change on dbstore1002 T86338 T202167
08:12 moritzm: installing perl security updates
07:28 marostegui: Deploy schema change on db1095:3312 T86338 T202167
07:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3312 T86338 T202167 (duration: 00m 47s)
06:55 marostegui: Purge binary logs on pc1005
06:52 marostegui: Deploy schema change on db1103:3312 T86338 T202167
06:51 marostegui: Deploy schema change on db1103:3312 T86338 T20216
06:51 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3312 T86338 T202167 (duration: 00m 48s)
06:12 marostegui: Deploy schema change on s2 codfw master with replication (db2035), this will generate lag on codfw - T86338 T202167
02:17 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: update wbsearchentities ab test configuration T209402 (duration: 00m 47s)
01:06 thcipriani@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/WikimediaEvents: SWAT: wbsearchentities ab test improvements T209402 (duration: 00m 46s)
01:04 thcipriani@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/MobileFrontend: SWAT: Change config flag for enabling Block Notice stats T201719 (duration: 00m 48s)
00:57 thcipriani@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/VisualEditor: SWAT: Rename configs for tracking block notices on visual editor ve.init.mw.ArticleTarget: Stop when we fail to load metadata T209542 (duration: 00m 48s)

2018-11-29

23:26 mutante: puppetmaster: sudo puppet cert revoke rutherfordium.eqiad.wmnet; sudo puppet node clean rutherfordium.eqiad.wmnet ; sudo puppet node deactivate rutherfordium.eqiad.wmnet ; run puppet on icinga1001.. removed host from monitoring (decom for ganeti VM) (T210036)
22:21 hashar: 1.33.0-wmf.6 is on all wikis and looks stable.
22:04 hashar: hashar@deploy1001 rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.6
21:54 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T210636 - I9ebbc6 (duration: 00m 55s)
21:41 tzatziki: changing email for User:Mathounette
21:34 XioNoX: removed unused vc-port on asw2-c-eqiad:fpc8 - T210788
20:50 mutante: people - rsynced /home one last time, switched DNS people.eqiad CNAME over, varnish change merged (T210036)
20:42 mutante: people.wikimedia.org is switching backends from rutherfordium to people1001, please stand by during a short maintenance period.. data has been copied | https://wikitech.wikimedia.org/wiki/People.wikimedia.org#Backend_upgrade_November_2018 | T210036
20:01 XioNoX: Apply Icinga:check_vcp to all VC switches - T201097
19:42 XioNoX: remove neodymium/sarin from mgmt routers - T210612
19:16 hashar@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/MobileFrontend/: RecordRevision::getUser() returns UserIdentity not int - T210737 (duration: 00m 55s)
19:09 ejegg: updated payments-wiki from 34250e80b5 to 7403a196b4
18:53 XioNoX: Netbox: remove Napalm integration
18:28 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting comment migration to write-new/read-new on group 0 (T166733) (duration: 00m 52s)
17:55 XioNoX: remove test netbox user from cr3-ulsfo - T205898
17:49 mforns@deploy1001: Finished deploy [analytics/refinery@40b1972]: deploying refinery to refinery-source version v0.0.81 (duration: 06m 01s)
17:43 mforns@deploy1001: Started deploy [analytics/refinery@40b1972]: deploying refinery to refinery-source version v0.0.81
17:41 anomie@deploy1001: Synchronized php-1.33.0-wmf.6/includes/revisiondelete/RevisionDeleteUser.php: Fix RevisionDeleteUser rev_actor query for MySQL, for real this time (T210628) (duration: 00m 53s)
17:34 anomie@deploy1001: Synchronized php-1.33.0-wmf.6/includes/revisiondelete/RevisionDeleteUser.php: Fix RevisionDeleteUser rev_actor query for MySQL (T210628) (duration: 00m 53s)
17:02 robh: decom of labvirt101[01] continuing
16:10 anomie@mwmaint1002: Running Wikibase/populateSitesTable.php and cleanupUsersWithNoId.php on more wiktionaries, incubatorwiki, and sourceswiki for T210732
15:54 papaul: shutting down ms-be2047 for maintenance
15:35 gtirloni: T196507 downtimed and powercycled cloudvirt1019
15:32 anomie@mwmaint1002: Running Wikibase/populateSitesTable.php and cleanupUsersWithNoId.php on several other wiktionaries for T210732
15:29 gehel: activating multiple elasticsearch instances on cirrus / eqiad - T207918
15:18 $WHO: Running Wikibase populateSitesTable.php on eswiktionary for T210732
14:31 hashar@deploy1001: rebuilt and synchronized wikiversions files: Revert all wikis to 1.33.0-wmf.6
14:29 hashar@deploy1001: scap failed: average error rate on 11/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details)
14:17 hashar@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.6
14:10 hashar: test stashbot
14:03 moritzm: uploaded nodejs 6.11.0~dfsg-1+wmf3 to apt.wikimedia.org/stretch-wikimedia (backporting the current security fixes)
13:45 hashar@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/Wikibase: feature flag for globe coordinator formatter using kartographer - T184933 T210617 (duration: 01m 18s)
13:40 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Clarify parsercache keys section (duration: 00m 53s)
13:39 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Clarify parsercache keys section (duration: 00m 52s)
13:17 moritzm: rebooting certcentral1001 to pick up SSBD-enabled qemu/kernel update
13:14 moritzm: rebooting certcentral2001 to pick up SSBD-enabled qemu/kernel update
13:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Pool pc1009 in pc3 - T208383 (duration: 00m 53s)
13:05 marostegui: Upgrade pc3 tendril topology - T208383
13:00 mobrovac@deploy1001: Started restart [eventstreams/deploy@07033d4]: Restart ES on scb1004 due to possible memory leak (again)
12:49 dcausse: EU swat done
12:48 dcausse@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/TwoColConflict/: Fix unescaped HTML injected into conflict resolution interface (duration: 00m 53s)
12:30 jynus: run puppet on notebook1004, people1001, rutherfordium to fix failures
12:13 dcausse@deploy1001: Synchronized dblists/cirrussearch-big-indices.dblist: T210381: [cirrus] multi-instance: add cirrussearch-big-indices.dblist (duration: 00m 53s)
12:09 dcausse@deploy1001: Synchronized wmf-config/CirrusSearch-production.php: T210381: [cirrus] Use normal config for labswiki (duration: 00m 55s)
12:01 banyek: repooling labsdb1010 after upgrades - T209517
10:17 elukey: remove zookeeper's crontabs from conf100[1-3] to fix cronspam
10:16 arturo: T209626 icinga downtime labvirt1011 for 1 month to avoid bogus pages
10:01 banyek: depooling labsdb1010 due of maintenance - T209517
09:55 gehel: restarting prometheus-elasticsearch-exporter-9200 on all elastic cirrus nodes
09:11 akosiaris: increase nofile of process to 20k and maxclients to 15k to account for the backlog of ores scorings
08:48 _joe_: restarting uwsgi-ores on ores1003
08:04 vgutierrez: replacing TLS certificates in gerrit - T207050
06:59 marostegui: Stop MySQL on pc1006 to clone pc1009 - T208383
06:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool pc1006 - T208383 (duration: 00m 53s)
06:36 marostegui: Deploy schema change on db1061 (s6 master) - T202167
06:35 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1088 T202167 (duration: 00m 53s)
06:31 marostegui: Deploy schema change on db1088 - T202167
06:31 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1088 T202167 (duration: 00m 56s)
06:15 reedy@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/OATHAuth: revert logging (loldeployingfromaplane) (duration: 00m 59s)
06:11 marostegui: Deploy schema change on s6 primary master - T86338
04:08 ejegg: updated fundraising internal dashboard from 5e9fb9a3ef to 1e4dd2a9ec
03:30 ejegg: updated payments-wiki from 42283c73d0 to 34250e80b5
00:08 catrope@deploy1001: Synchronized wmf-config/throttle.php: T210681 (duration: 01m 04s)

2018-11-28

23:23 tzatziki: changing a few passwords for compromised accounts
21:35 mutante: gnt-instance reboot proton1001.eqiad (stopped working, no SSH)
21:28 arlolra: Updated Parsoid to 18a98af (T209236, T210437, T184755, T187142, T208470, T207286, T206777, T205710, T205546, T204477)
21:26 mutante: rebooting proton1002
21:19 herron: restarted nagios-nrpe-server on proton1002
21:16 arlolra@deploy1001: Finished deploy [parsoid/deploy@9ed8c47]: Updating Parsoid to 18a98af (duration: 09m 35s)
21:06 arlolra@deploy1001: Started deploy [parsoid/deploy@9ed8c47]: Updating Parsoid to 18a98af
20:41 herron: rebooting logstash1005 for security updates
20:22 paravoid: neodymium: mv /srv/jnt{,.old} (use cumin1001 instead!)
19:40 herron: rebooting logstash1004 to pick up security updates
19:31 ebernhardson: start goreplay logging of port 9200 across eqiad elastic cluster to track down T208248
19:05 ejegg: updated payments-wiki from 3e0c97a969 to 6552acdc0f
18:58 ladsgroup@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/ORES/includes/Hooks/ApiHooksHandler.php: Don't try to add scores in API where there is nothing to add (T210610) (duration: 00m 55s)
18:33 thcipriani@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/Wikibase/lib/includes/Formatters/CachingKartographerEmbeddingHandler.php: Never return null in CachingKartographerEmbeddingHandler::getParserOutput T210617 (duration: 00m 53s)
18:26 thcipriani@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/EventBus/includes/EventBus.php: SWAT: Revert "Revert "Revert "Set event datetime with microsecond resolution.""" T210608 (duration: 00m 55s)
17:16 bblack: rebooting lvs1006, console was unresponsive
17:11 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable classic_entity wbsearchentities AB test T209402 T210618 (duration: 00m 55s)
17:01 chasemp: stat1004:~# aptitude install exfat-fuse exfat-utils (elukey fyi)
16:57 mutante: restarting gerrit
16:54 thcipriani@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/WikimediaIncubator/extension.json: Revert "Replace wiki with wikipedia as wmf-config has been updated" T117023 (duration: 00m 54s)
16:53 mutante: gerrit about to restart for logging config change
16:50 gtirloni: T206916 created shnwiki views/index in labsdb replicas
15:55 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1122 (duration: 00m 53s)
15:52 banyek: repooling db1122 after schema change (T85757)
15:51 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1085 T86338 (duration: 00m 52s)
15:49 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: wikidatawiki back to 1.33.0-wmf.4
15:24 vgutierrez: use a certcentral managed TLS certificate in dumps.wm.o - T207050
15:19 banyek: Deploy schema change on db1122 - T85757
15:17 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1122 (duration: 00m 53s)
15:10 banyek: depooling db1122 due schema change (T85757)
15:09 marostegui: Deploy schema change on db1085 with replication - T86338
15:09 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1085 T86338 (duration: 00m 56s)
14:05 hashar@deploy1001: Synchronized php: group1 wikis to 1.33.0-wmf.6 (duration: 00m 52s)
14:04 hashar@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.6
13:59 zeljkof: EU SWAT (finally) done
13:57 zfilipin@deploy1001: Finished scap: SWAT: Add user preference to disable the advanced interface (T210479) (duration: 38m 12s)
13:19 zfilipin@deploy1001: Started scap: SWAT: Add user preference to disable the advanced interface (T210479)
12:54 moritzm: installing serf update from stretch point release
12:43 zfilipin@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/Cite: SWAT: Make backlink highlighting robust for community customized HTML (T205270 T210520) (duration: 00m 55s)
12:34 moritzm: installing ghostscript security updates
12:24 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Make AdvancedSearch default on all wikis (T207639) (duration: 00m 54s)
11:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1085 T202167 (duration: 00m 53s)
11:17 marostegui: Deploy schema change on db1085 (s6) (sanitarium master) with replication - T202167
11:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1085 T202167 (duration: 00m 53s)
11:15 banyek: repooling labsdb1009 after maintenance
10:46 moritzm: rolling reboot of logstash1007-1009 to pick up new SSBD instructions and OpenJDK security updates
10:26 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Pool pc1008 - T208383 (duration: 00m 50s)
10:26 ladsgroup@deploy1001: Finished deploy [ores/deploy@9b9ba06]: T206333 (duration: 14m 48s)
10:11 ladsgroup@deploy1001: Started deploy [ores/deploy@9b9ba06]: T206333
10:08 jynus: disable mailing list mediation-en-l
10:00 banyek: depooling labsdb1009 (T209517)
09:55 marostegui: Update tendril topology for pc1 - T208383
09:51 marostegui: Change pc2008 to replicate from pc1008 instead of pc1005 - T208383
09:22 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1113:3316 T86338 T202167 (duration: 00m 53s)
09:14 vgutierrez: Use a TLS certificate managed by certcentral in icinga.wm.o - T207050
09:00 marostegui: Deploy schema change db1113:3316 T86338 T202167
09:00 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1113:3316 T86338 T202167 (duration: 00m 53s)
08:41 moritzm: installing git security updates on trusty (Debian already fixed)
08:28 moritzm: installing samba security updates (client libs)
08:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3316 T86338 (duration: 00m 53s)
08:12 elukey: apply -R 200 to memcached on mc1022 (cache wipe) - T208844
07:45 marostegui: Deploy schema change db1096:3316 T86338
07:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3316 T86338 (duration: 00m 53s)
07:39 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1098:3316 T86338 T202167 (duration: 00m 54s)
07:17 marostegui: Deploy schema change db1098:3316 T86338 T202167
07:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1098:3316 T86338 T202167 (duration: 00m 53s)
06:49 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1093 T86338 T202167 (duration: 00m 52s)
06:40 marostegui: Deploy schema change db1093 T86338 T202167
06:40 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1093 T86338 T202167 (duration: 00m 53s)
06:14 marostegui: Stop MySQL on pc1005 to clone pc1008 - T208383
06:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool pc1005 - T208383 (duration: 01m 04s)
06:03 marostegui: Deploy schema change on s6 codfw - T86338 T202167
02:17 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/WikimediaEvents/modules/wikibase/ext.wikimediaEvents.completionClicks.js: Fix wbsearchentities collection of non-test bucketed data (duration: 00m 56s)
02:16 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/WikimediaEvents/modules/wikibase/ext.wikimediaEvents.completionClicks.js: Fix wbsearchentities collection of non-test bucketed data (duration: 00m 54s)
01:45 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/WikimediaEvents/: wbsearchentities needed extension.json to be deployed as well. Sync the whole directory (duration: 00m 53s)
01:44 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/WikimediaEvents/: wbsearchentities needed extension.json to be deployed as well. Sync the whole directory (duration: 00m 53s)
01:34 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T209402: Start wbsearchentities ab test at 10% (duration: 00m 54s)
01:29 ebernhardson: ebernhardson@deploy1001 Synchronized php-1.33.0-wmf.6/includes/parser/Parser.php: SWAT: T209236 Protect legacy URL parameter syntax in link and alt options (duration: 00m 51s)
01:22 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/ParsoidBatchAPI/includes/ApiParsoidBatch.php: SWAT: Revert ApiParsoidBatch update (duration: 00m 54s)
01:12 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T209402: Configuration for wbsearchentities AB test (duration: 00m 53s)
01:09 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/WikimediaEvents/modules/wikibase/ext.wikimediaEvents.completionClicks.js: T209402: AB testing support for wbsearchentities (duration: 00m 52s)
01:08 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/WikimediaEvents/modules/wikibase/ext.wikimediaEvents.completionClicks.js: T209402: AB testing support for wbsearchentities (duration: 00m 53s)
01:06 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/Wikibase/view/resources/jquery/wikibase/jquery.wikibase.entityselector.js: Allow AB test to modify entityselector api request (duration: 00m 56s)
01:02 ejegg: updated payments-wiki config to 51abff4b66
00:19 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/ParsoidBatchAPI/includes/ApiParsoidBatch.php: SWAT I4e4373a7 revert Modernize ApiParsoidBatch using ApiResult to generate prettier output (duration: 00m 54s)
00:07 ebernhardson@deploy1001: Synchronized wmf-config/CommonSettings-labs.php: SWAT: no-op labs sync for 476027 (duration: 00m 53s)
00:06 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: no-op labs sync for 476027 (duration: 00m 55s)

2018-11-27

23:49 ejegg: updated payments-wiki config to 04f718e8f1
23:29 ejegg: updated payments-wiki config to 082eab3566
22:35 ejegg: rolled back payments-wiki to 86742dd4fd
22:33 ejegg: updated payments-wiki from 86742dd4fd to 3ba10e49b0
21:26 ejegg: updated payments-wiki config to d497b7e880
19:40 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: no op sync wikiversions to test scap 3.8.10-1
18:03 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: no op sync wikiversions to test scap 3.8.9-1
17:51 godog: upload scap 3.8.9-1 - T210469
17:40 bstorm_: T209517 rebooted labsdb1005 after upgrades
17:30 bstorm_: T209517 icinga downtime labsdb1004
17:30 XioNoX: repool uslfo
17:25 arturo: T209517 icinga downtime labsdb1005
17:23 arturo: T207377 icinga downtime labnet1001
16:17 mutante: einsteinium - apt-get remove --purge icinga nsca; apt-get autoremove ; apt-get remove --purge icinga-doc icinga-common icinga-cgi-bin icinga-cgi; apt-get remove --purge monitoring-plugin* ; rm /etc/rsync.d/frag-icinga* T209738
16:14 hashar@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.33.0-wmf.6
16:09 moritzm: rebooting planet1001 for kernel security update
16:04 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1105 (duration: 00m 53s)
15:53 moritzm: rebooting planet2001 for kernel security update
15:44 moritzm: rebooting kafkamon1001 for kernel security update
15:41 mutante: einsteinium - removed icinga package
15:39 moritzm: rebooting kafkamon2001 for kernel security update
15:35 mutante: einsteinium - stopped icinga, stopped nsca, stopped rsyncd, killall -u icinga, killall -u nagios ... T209738
15:35 hashar@deploy1001: Finished scap: testwiki to php-1.33.0-wmf.6 and rebuild l10n cache | T206660 (duration: 32m 42s)
15:21 akosiaris: create graphoid namespace on kubernetes eqiad, codfw, staging clusters T203091
15:02 hashar@deploy1001: Started scap: testwiki to php-1.33.0-wmf.6 and rebuild l10n cache | T206660
14:52 hashar@deploy1001: Pruned MediaWiki: 1.32.0-wmf.26 (duration: 09m 48s)
14:40 hashar: Applied security patches for 1.33.0-wmf.6 | T206660
14:33 hashar: scap prep 1.33.0-wmf.6 | T206660
14:04 hashar: Cutting 1.33.0-wmf.6 branches | T206660
14:00 banyek: repooling db1105 due a schema change (T85757)
13:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Pool pc1010 in pc1 - T208383 (duration: 00m 46s)
13:34 marostegui: Change pc2007 and pc2010 to replicate from pc1010 instead of from pc1004 - T208383
13:18 banyek: executing schema change on db1105 (T85757)
13:18 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1105 (duration: 00m 46s)
13:17 pmiazga@deploy1001: Finished deploy [proton/deploy@9efc072]: Proton: Rewrite Queue to promise-way flow (T204055) (duration: 02m 43s)
13:15 pmiazga@deploy1001: Started deploy [proton/deploy@9efc072]: Proton: Rewrite Queue to promise-way flow (T204055)
13:14 banyek: depooling db1105 due a schema change (T85757)
13:09 raynor: proton deploying 9efc072
12:39 mobrovac@deploy1001: Synchronized php-1.33.0-wmf.4/includes/page/WikiPage.php: Convert $archivedRevisionCount to integer - T210013 T210451 (duration: 00m 47s)
12:32 gilles@deploy1001: Synchronized docroot/wikipedia.org/speed-tests/http2priorities/upload.wikimedia.org.html: T210141 Add variant of HTTP/2 priorities test pointing to upload (duration: 00m 46s)
12:25 mobrovac@deploy1001: Synchronized static/images/project-logos: Add localised logos for the Minangkabau Wikipedia - T210387 (duration: 00m 47s)
12:19 mobrovac@deploy1001: Synchronized rpc/RunSingleJob.php: RunSingleJob: Check that JobExecutor has been loaded - T208922 (duration: 00m 47s)
11:56 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1103 (duration: 00m 46s)
11:50 banyek: repooling db1103 after schema change (T85757)
11:32 banyek: executing schema change on db1103:3312 (T85757)
11:32 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1103 (duration: 00m 46s)
11:28 banyek: depooling db1103 due a schema change (T85757)
10:59 banyek: executing schema change on db1095 (T85757)
10:55 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1090 (duration: 00m 46s)
10:51 banyek: repooling db1090 after schema change (T85757)
10:51 banyek: repooling db1090 due a schema change (T85757)
10:42 gilles@deploy1001: Synchronized docroot/wikipedia.org/speed-tests/http2priorities: T210141 HTTP/2 prioritie speed test (duration: 00m 47s)
10:36 arturo: T209948 disable puppet in all WMCS hw servers
10:30 arturo: T209948 schedule 2h icinga downtime in all WMCS hw servers
10:26 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1090 (duration: 00m 46s)
10:19 banyek: depooling db1090 due a schema change (T85757)
10:18 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1076 (duration: 00m 45s)
10:13 banyek: repooling db1076 after schema change (T85757)
10:06 banyek: executing schema change on db1076 (T85757)
10:02 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1076 (duration: 00m 46s)
09:57 banyek: depooling db1076 due a schema change (T85757)
09:35 marostegui: Stop MySQL on pc1004 to clone pc1010 - T208383
09:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool pc1004 - T208383 (duration: 00m 46s)
09:00 banyek: executing schema change on db2035 for s2 (T85757)
08:43 elukey: roll restart of all druid daemons on druid100[1-6] for openjdk-8 upgrades
08:41 vgutierrez: Use a TLS certificate managed by certcentral in apt.wm.o - T207050
08:15 godog: more weight to new ms-be hosts in codfw - T209395
07:35 _joe_: depooling mw1261 for benchmarking, T206341
07:23 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3312 after cloning db1095:3312 (duration: 00m 46s)
07:16 marostegui: Start MySQL on db1090:3312 after recloning db1095:3312
07:16 marostegui: Start MySQL on db1095:3312 after recloning it
06:16 marostegui: Stop mysql on db1090:3312 to clone db1095:3312
06:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3312 (duration: 00m 49s)
06:15 marostegui: Stop mysql on db1095:3312 to get it recloned
04:33 ejegg: updated payments-wiki from d2b66c5bab to 86742dd4fd
04:31 ejegg: updated fundraising CiviCRM from 013807a7b9 to a411d6bd64
04:27 ejegg: updated standalone SmashPig deployment from f65daa8550 to fb3268897b
00:38 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Fix weird caching issue maybe for Ian's patch (duration: 00m 46s)
00:29 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: Enable SVGs in page language everywhere T208899 (duration: 00m 45s)
00:27 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable SVGs in page language everywhere T208899 (duration: 00m 46s)
00:17 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Set wgMinervaSchemaMainMenuClickTrackingSampleRate T205008 (duration: 00m 46s)
00:12 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Delete a namespace from ruwikisource T210171 (duration: 00m 46s)
00:05 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: Enable trust survey on labs T209882 (duration: 00m 46s)
00:03 niharika29@deploy1001: sync-file aborted: (no justification provided) (duration: 00m 03s)

2018-11-26

23:50 ottomata: temporarily disabling puppet on stat1007 to copy over eventbus validation logs (not using stat1007 after all)
23:48 ottomata: temporarily disabling puppet on stat1007 to copy over eventbus validation logs
23:24 sbassett@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Remove unblockself right everywhere 2046d8df3 T150826 (duration: 00m 47s)
23:15 XioNoX: depool ulsfo for transport providers maintenance
20:14 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/Wikibase/repo/includes/Search/Elastic/EntitySearchElastic.php: SWAT T209402 Make wbsearchentities tie-breaker configurable (duration: 00m 47s)
19:33 ebernhardson@deploy1001: Synchronized wmf-config/WikibaseSearchSettings.php: SWAT T209402 Search profiles for wbsearchentities AB test (duration: 00m 46s)
19:25 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/WikimediaEvents/includes/: SWAT T210003 T210004 EditorJourney: Adjust DeferredUpdates usage (duration: 00m 46s)
19:24 XioNoX: remove IP from blacklist on Amsterdam routers - T201411
19:24 cmjohnson1: cloudvirt1019 is going down to check something for HPE
19:16 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT T209432 Enable NewUserMessage extension on tcy.wikipedia (duration: 00m 48s)
18:50 mutante: stopping icinga service on einsteinium, is a role(spare) now T209738
18:46 mutante: removed allowed sender addresses from AQL (mail2SMS gateway) portal: @einsteinium @tegmen addresses T208824 T209738
18:45 ppchelko@deploy1001: Finished deploy [changeprop/deploy@c89bff5]: Prepared for ORES error response T197000 (duration: 01m 13s)
18:44 ppchelko@deploy1001: Started deploy [changeprop/deploy@c89bff5]: Prepared for ORES error response T197000
18:41 XioNoX: re-activate BGP to AS41692 on cr2-esams
18:15 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@8bfea2b]: GUI updates and updater with dump and revision logging. (duration: 11m 13s)
18:04 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@8bfea2b]: GUI updates and updater with dump and revision logging.
17:54 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp5001.eqsin.wmnet
17:53 bblack: re-pooling cp5001 - T199675
17:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1122 (duration: 00m 46s)
17:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1122 (duration: 00m 46s)
17:35 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1122 (duration: 00m 46s)
17:26 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1122 (duration: 00m 45s)
17:23 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Reverting: Bumping portals to master (T128546) (duration: 00m 46s)
17:22 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Reverting: Bumping portals to master (T128546) (duration: 00m 46s)
17:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3312 and db1122 (duration: 00m 46s)
16:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3312 to clone db1122 (duration: 00m 46s)
16:00 marostegui: Stop MySQL on db1090:3312 to clone db1122
15:57 marostegui: Stop MySQL on db1122 - it will be recloned
15:50 ppchelko@deploy1001: Finished deploy [changeprop/deploy@77be2c6]: Change schema for revision-score events and start emitting again T197000 (duration: 01m 24s)
15:48 ppchelko@deploy1001: Started deploy [changeprop/deploy@77be2c6]: Change schema for revision-score events and start emitting again T197000
15:32 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1122 - actor table missing (duration: 00m 46s)
15:27 ppchelko@deploy1001: Finished deploy [changeprop/deploy@b97e8eb]: Temporary stop emitting revision-score events for schema change T197000 (duration: 01m 21s)
15:26 ppchelko@deploy1001: Started deploy [changeprop/deploy@b97e8eb]: Temporary stop emitting revision-score events for schema change T197000
15:24 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting actor migration to write-both/read-old on group 1 (T188327) (duration: 00m 46s)
13:47 moritzm: rebooting releases1001
13:36 moritzm: rebooting releases2001
13:33 moritzm: rebooting netmon1003
12:04 _joe_: uploaded php-excimer for component thirdparty/php72 to stretch-wikimedia T205059
12:00 kartik@deploy1001: Finished deploy [cxserver/deploy@da41227]: Disable Youdao MT until service is back (duration: 04m 01s)
11:56 kartik@deploy1001: Started deploy [cxserver/deploy@da41227]: Disable Youdao MT until service is back
11:41 arturo: icinga downtime cloudcontrol1004 for some systemd slice tests
11:01 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 46s)
11:00 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 47s)
10:32 moritzm: rebooting boron
09:58 banyek: performing schema change on s6 master - db1061 for frwiki (T85757)
09:51 banyek: performing schema change on s6 master - db1061 for ruwiki (T85757)
09:44 banyek: performing schema change on s6 master - db1061 for jawiki (T85757)
08:35 moritzm: removed labvirt1015 from debmonitor DB (got renamed to cloudvirt1015)
08:22 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3316 - T202167 (duration: 00m 46s)
08:16 marostegui: Deploy schema change on db1096:3316 - T202167
08:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3316 - T202167 (duration: 00m 47s)
08:10 moritzm: installing pixman security updates
07:37 elukey: restart memcached on mc1021 (cache wipe) to add -R 200 - T208844
07:10 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1088 - T86338 (duration: 00m 46s)
07:00 marostegui: Deploy schema change on db1088 - T86338
07:00 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1088 - T86338 (duration: 00m 46s)
06:46 marostegui: Deploy schema change on db2067 - T86338
06:36 marostegui: Deploy schema change on s3 master (db1075) - T86339
06:28 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1077 - T86339 (duration: 00m 46s)
06:22 marostegui: Reload haproxy on dbproxy1005
06:22 marostegui: Deploy schema change on db1077 (s3 sanitarium master) with replication - T86339
06:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1077 - T86339 (duration: 00m 50s)
06:02 kartik@deploy1001: Finished deploy [cxserver/deploy@d2173ca]: Update cxserver to 218a543 (duration: 04m 21s)
05:58 kartik@deploy1001: Started deploy [cxserver/deploy@d2173ca]: Update cxserver to 218a543

2018-11-25

10:40 _joe_: restarting pdfrender on scb1002, flapping since 3:00 UTC

2018-11-23

21:15 bawolff: deploy patch for T210192
16:53 bblack: cleaned up remnants of globalsign-2017 unified cert (OCSP cache/config, unmanaged cert files, etc) on all cpNNNN - T206804
14:02 gehel: restor wdqs-updater heap to 2G - T210235
13:32 moritzm: installing confuse security updates
12:04 gehel: manually increasing wdqs-updater heap to 4G - T210235
11:37 gehel: restarting updater on all wdqs ndoes
08:41 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=.*,service=zotero,cluster=kubernetes,name=.*
08:10 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1078 - T86339 (duration: 00m 46s)
07:48 moritzm: installing libtirpc security updates
07:14 marostegui: Deploy schema change db1078 - T86339
07:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1078 - T86339 (duration: 00m 45s)
07:00 marostegui: Deploy schema change db1095 - T86339
06:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1123 - T86339 (duration: 00m 49s)
06:32 marostegui: Deploy schema change db1123 - T86339
06:31 marostegui: Deploy schema change dbstore1002:s3 - T86339
06:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1123 - T86339 (duration: 00m 48s)
06:15 marostegui: Deploy schema change on s3 codfw master (db2043) with replication - T86339
06:13 marostegui: Deploy schema change on db1067 (s1) master - T86339

2018-11-22

21:08 godog: disable raid handler for ms-be2021 - T208096
19:01 moritzm: installing uriparser security updates
18:46 moritzm: installing openjpeg2 security updates
18:45 arturo: enable puppet in all CloudVPS HW servers
18:38 arturo: disable puppet in all CloudVPS HW servers to test a patch (T209948)
17:46 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1016 (duration: 00m 46s)
16:49 jynus: upgrading, and restarting es1016 (but not deleting, that was a mistake)
16:49 jynus: upgrading, deleting at and restarting es1016
16:35 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1016 (duration: 00m 47s)
16:06 ema: trafficserver_8.0.0-1wm3 uploaded to stretch-wikimedia
15:23 akosiaris@deploy1001: scap-helm zotero finished
15:23 akosiaris@deploy1001: scap-helm zotero cluster codfw completed
15:23 akosiaris@deploy1001: scap-helm zotero install --name production -f zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: codfw]
15:22 akosiaris@deploy1001: scap-helm zotero finished
15:22 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
15:22 akosiaris@deploy1001: scap-helm zotero install --name production -f zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
14:50 akosiaris@deploy1001: scap-helm zotero install --name production -f zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
13:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1080 - T86339 (duration: 00m 45s)
13:52 marostegui@deploy1001: sync-file aborted: Depool db1080 - T86339 (duration: 00m 00s)
13:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1080 - T86339 (duration: 00m 46s)
13:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1083 - T86339 (duration: 00m 46s)
13:45 marostegui: Deploy schema change on s1 eqiad hosts T86339
13:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1083 - T86339 (duration: 00m 46s)
13:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1089 - T86339 (duration: 00m 45s)
13:36 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1089 - T86339 (duration: 00m 46s)
13:30 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1114 - T86339 (duration: 00m 45s)
13:26 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1114 - T86339 (duration: 00m 43s)
13:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1119 - T86339 (duration: 00m 46s)
13:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1119 - T86339 (duration: 00m 47s)
12:19 jynus: upgrading, deleting at and restarting dbstore2002
10:21 jynus: upgrading and restarting dbstore1001
10:01 godog: bounce rsyslog on lithium, tls listener timeout on icinga
09:54 godog: bounce rsyslog on wezen, tls listener timeout on icinga
09:49 jynus: stop and upgrade dbstore2001
09:24 moritzm: installing ruby-l18n security updates
09:20 moritzm: installing ruby-rack security updates
09:06 moritzm: installing jasper security updates
08:21 marostegui: Deploy schema change on s1 codfw master (db2048) with replication - T86339
08:19 marostegui: Deploy schema change on db1062 (s7 master) - T86339
08:19 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1094 - T86339 (duration: 00m 46s)
08:17 marostegui: Deploy schema change on db1094 - T86339
08:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1094 - T86339 (duration: 00m 49s)
06:58 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1086 - T86339 (duration: 00m 46s)
06:55 marostegui: Deploy schema change on db1086 - T86339
06:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1086 - T86339 (duration: 00m 46s)
06:51 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1079 - T86339 (duration: 00m 46s)
06:48 marostegui: Deploy schema change on db1079 (sanitarium master) with replication - T86339
06:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1079 - T86339 (duration: 00m 51s)
02:25 mutante: stat1005 - started nagios-nrpe-server

2018-11-21

23:34 mutante: rsyncing /home from rutherfordium.eqiad to people1001.eqiad (people.wikimedia.org) T210036
21:16 robh: cp5001 is offline running hardware tests after firmware updates to see if memory error still exists. ref: T199675
20:55 robh: cp5001 reboot for firmware update
19:54 milimetric@deploy1001: Finished deploy [analytics/aqs/deploy@e114d99]: Fixing sorting bug on top endpoints (duration: 05m 34s)
19:49 milimetric@deploy1001: Started deploy [analytics/aqs/deploy@e114d99]: Fixing sorting bug on top endpoints
19:43 ejegg: updated fundraising internal dashboard from b01458b260 to 5e9fb9a3ef
17:21 elukey: manually started systemd-journald.service on scb1001 after OOM
17:20 jynus: stop and upgrade db2081
17:04 jynus: stop and upgrade db2080
16:40 jynus: stop and upgrade db2066
16:37 bawolff: deploy patch T209794
16:25 jynus: stop and upgrade db2063
15:15 jynus: stop and upgrade db2073
15:15 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1085 (duration: 00m 46s)
15:03 banyek: repooling db1085 after schema change (T85757)
15:00 banyek: restarting replication on db1085 (T85757)
14:31 banyek: stopping replication on db1085 (T85757)
14:27 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1085 (duration: 00m 46s)
14:22 banyek: depooling db1085 due schema change (T85757)
13:39 jynus: stop and upgrade db2077
13:05 XioNoX: remove BGP session to 2603 on cr4-ulsfo
12:13 jynus: stop and upgrade db2076
12:08 banyek: running schema change on dbstore1001:3316 (T85757)
12:08 banyek: running schema change on dbstore1001 (T85757)
12:04 jynus: stop and upgrade db2075
11:53 banyek: running schema change on dbstore1002 (T85757)
11:00 akosiaris: disable puppet on ores2* ores1* for gradual rollout of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/474694/1/modules/ores/manifests/web.pp
10:55 jynus: stop and upgrade db2074
10:51 _joe_: uploading prometheus-php-fpm-exporter to stretch-wikimedia main, T209573
10:44 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1096:3316 (duration: 00m 46s)
10:42 banyek: repooling db1096:3316 after schema change (T85757)
10:30 jynus: stop and upgrade db2095
10:26 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1096:3316 (duration: 00m 45s)
10:26 godog: initial weight for new ms-be2* hosts (all but ms-be2047) - T209395
10:23 banyek: depooling db1096:3316 due schema change (T85757)
10:14 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1098:3316 (duration: 00m 46s)
10:13 banyek@deploy1001: sync-file aborted: T85757: depool db1098:3316 (duration: 00m 03s)
10:11 banyek: repooling db1098 after schema change (T85757)
09:57 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1098:3316 (duration: 00m 46s)
09:52 banyek: depooling db1098:3316 due schema change (T85757)
09:49 volans: restarted pdfrender on scb1003
09:29 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1113 (duration: 00m 46s)
09:21 banyek: repooling db1113 after schema change (T85757)
09:09 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1113 (duration: 00m 46s)
09:01 banyek: depooling db1113 due schema change (T85757)
08:48 marostegui: Deploy schema change on s7 codfw master - T86339
08:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1104 - T86339 (duration: 00m 45s)
08:40 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1104 - T86339 (duration: 00m 45s)
08:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1109 - T86339 (duration: 00m 46s)
08:30 marostegui: Deploy schema changes on s8 eqiad hosts - T86339
08:28 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1109 - T86339 (duration: 00m 46s)
07:50 marostegui: Deploy schema change on s8 codfw master (db2045) with replication - T86339
07:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1091 - T86339 (duration: 00m 45s)
07:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1091 - T86339 (duration: 00m 46s)
07:36 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1084 - T86339 (duration: 00m 46s)
07:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084 - T86339 (duration: 00m 46s)
07:32 marostegui: Deploy schema change on s4 eqiad hosts - T86339
07:19 marostegui: Deploy schema change on db2051 (s4 codfw master) with replication - T86339
07:10 marostegui: Drop foundationwiki.petition_data from s3 master (db1075) with replication - T208979
07:06 marostegui: Deploy schema change on db1066 (s2 master) - T86339
07:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1076 - T86339 (duration: 00m 46s)
07:02 marostegui: Deploy schema change on db1076 - T86339
07:02 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1076 - T86339 (duration: 00m 46s)
06:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1074 - T86339 (duration: 00m 46s)
06:48 marostegui: Deploy schema change on db1074 (sanitarium master) - T86339
06:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1074 - T86339 (duration: 00m 46s)
06:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1122 - T86339 (duration: 00m 46s)
06:39 marostegui: Deploy schema change on db1122 - T86339
06:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1122 - T86339 (duration: 00m 51s)
06:30 marostegui: Drop schema change on db1103:3312 and db1105:3312 - T86339
00:48 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: gerrit:474967 Disable FlaggedRevs, enable RC patrol and add rights on srwikinews (duration: 00m 47s)
00:39 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: gerrit:475005 Enable SVGs in page in group1, rest of group0 (duration: 00m 46s)
00:32 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: gerrit:474976 Enable suppressredirect on srwiki (duration: 00m 47s)
00:24 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: gerrit:472744 Enable RCPatrol and add some rights on srwikibooks (duration: 00m 46s)
00:07 sbisson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/GrowthExperiments/includes/Specials/SpecialWelcomeSurvey.php: gerrit:474946 WelcomeSurvey: indicate that the special page does write (duration: 00m 47s)

2018-11-20

23:01 XioNoX: create vol.ans account on switches - T208726
22:54 pmiazga@deploy1001: Synchronized wmf-config: SYNC: noop Doc: add repoConceptBaseUri comment (T209352)noop: Remove utf-8 characters from DOC comment for better readability (T209352)beta: Wikibase: override repoConceptBaseUri (T209352) (duration: 00m 49s)
22:13 XioNoX: create volans account on routers - T208726
21:12 eileen: civicrm revision changed from f4127d5316 to 013807a7b9, config revision is 684ec9b7c0
21:08 jgleeson: civicrm changed from a31dbefc61 to f4127d5316
18:56 ebernhardson: start loading dumps into elastic codfw omega and psi from mwmaint2001
18:19 ladsgroup@deploy1001: Synchronized wmf-config/Wikibase.php: Create Federated Wikibase instance on Beta Commons, part II (T204748) (duration: 00m 47s)
18:17 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Create Federated Wikibase instance on Beta Commons (T204748) (duration: 00m 48s)
18:08 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@7553087]: Deploy 2018 app fundraising announcement config (T204821) (duration: 03m 37s)
18:04 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@7553087]: Deploy 2018 app fundraising announcement config (T204821)
18:03 bstorm_: rebooting labsdb1006 for upgrades T209517
17:25 bstorm_: rebooting labsdb1004 for upgrades T209517
17:19 gehel: reload nginx configuration on elasticsearch codfw
16:59 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2072 (duration: 00m 46s)
16:53 XioNoX: rollback all BFD tests on cr1-codfw
16:10 jynus: stop and upgrade db2072
15:59 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2071, depool db2072 (duration: 00m 47s)
15:51 banyek: repooling labsdb1011 (T209517)
15:41 banyek: uploaded wmf-pt-kill_2.2.20-1+wmf5 packages to stretch-wikimedia (T209517)
15:38 vgutierrez: switching to certcentral managed TLS certificate for librenms.wikimedia.org - T209856
15:36 XioNoX: add test term allow BFD multihop on cr1-codfw loopback4 filter
15:36 ejegg: updated fundraising CiviCRM from e648be0d9e to a31dbefc61
15:20 moritzm: installing libopenmpt security updates
15:12 XioNoX: enable bfd traceoptions on cr1-codfw
15:02 XioNoX: Add BFD multihop support to Bird anycast DNS
14:55 jijiki: libthumbor_1.3.2-0+wmf1+stretch1 uploaded to stretch-wikimedia T209886
14:43 chasemp: puppet temp disable on es2001 for data transfer work
14:32 jynus: stop and upgrade db2033
13:19 jynus: stop and upgrade db2082
13:06 banyek: depooling labsdb1011 (T209517)
13:03 zeljkof: EU SWAT finished
13:03 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: deployment-prep: Update parsoid09 IP (T208101) (duration: 00m 47s)
12:55 zfilipin@deploy1001: Synchronized wmf-config/db-labs.php: SWAT: deployment-prep: Update deployment-db* IPs (T208101) (duration: 00m 47s)
12:55 banyek: setting innodb_flush_log_at_trx_commit to 2 on dbstore2002 (s3 instance only!) (T208320)
12:53 banyek: setting innodb_flush_log_at_trx_commit to 2 on dbstore2002 (T208320)
12:49 zfilipin@deploy1001: scap failed: average error rate on 4/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details)
12:45 zfilipin@deploy1001: Synchronized wmf-config/reverse-proxy-staging.php: SWAT: deployment-prep: Update cache-upload private IP (T208101) (duration: 00m 45s)
12:30 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Use HD logos in InitialiseSettings.php for multiple projects (T150618) (duration: 00m 48s)
12:25 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add tboverride permission to extendedmover group on enwiki (T209753) (duration: 00m 47s)
12:19 jynus: powercycling db2087, stuck on reboot
12:13 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: Upload HD logos for multiple projects (T150618) (duration: 00m 48s)
11:55 moritzm: rolling reboot of proton hosts for kernel security update
11:27 jynus: stop and upgrade db2087
11:16 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: (now really) repool db1093 (duration: 00m 47s)
11:11 banyek: repooling db1093 (T85757)
11:05 banyek: executing schema change on db1093 (T85757)
11:00 jynus: stop and upgrade db2086
10:59 banyek: db1093 was depooled wrong message sent
10:51 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1093 (duration: 00m 47s)
10:48 banyek: depooling db1093 (T85757)
10:48 banyek: depooling db1093
10:47 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Repool es2018 (duration: 00m 46s)
10:17 jynus: upgrade and reboot es2018
10:13 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Repool es2014, depool es2018 (duration: 00m 46s)
09:34 marostegui: Deploy schema change on s2 hosts: dbstore1002, db1090:3312 and db1095:3312 - T86339
09:26 marostegui: Deploy schema change on s2 codfw master (db2035) with replication - T86339
09:25 jynus: upgrade and reboot es2014
09:23 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Repool es2011, depool es2014 (duration: 00m 46s)
09:23 godog: stress-test new ms-be hardware - T209395
09:13 marostegui: Stop MySQL on pc2004, pc2005 and pc2006 for decommission - T209858
09:05 gehel: powercycle elastic2021
09:04 marostegui: Remove pc2004, pc2005 and pc2006 from tendril and zarcillo - T209858
08:53 jynus: upgrade and reboot es2011
08:48 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Depool es2011 (duration: 00m 47s)
06:28 marostegui: Deploy schema change on db1070 (s5 master) - T86339
06:28 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1082 - T86339 (duration: 00m 47s)
06:21 marostegui: Deploy schema change on db1082 - T86339
06:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1082 - T86339 (duration: 00m 52s)
00:55 catrope@deploy1001: Synchronized static/images/project-logos/: Correct logos for Sindhi Wiktionary (duration: 00m 47s)
00:51 mutante: Gerrit: added Jeena Huneidi to wmf-deployers (T209722)
00:26 catrope@deploy1001: Synchronized php-1.33.0-wmf.4/resources/src/mediawiki.rcfilters/ui/mw.rcfilters.ui.FilterTagMultiselectWidget.js: RCFilters bug fix (T209657) (duration: 00m 47s)
00:23 XioNoX: registering librenms IRC bot
00:15 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Increase Schema.org page split test to 100% sampling (T208755) (duration: 00m 48s)

2018-11-19

23:18 ejegg: updated fundraising CiviCRM from 275716f000 to e648be0d9e
22:41 ejegg: updated fundraising CiviCRM from bbc0dddd1e to 275716f000
22:21 ejegg: updated fundraising CiviCRM from 6b279509f8 to bbc0dddd1e
21:58 XioNoX: restart bird on dns2001 to try to establish the BFD sessions
20:27 catrope@deploy1001: Finished scap: Full scap for special alias changes for GrowthExperiments (duration: 21m 03s)
20:06 catrope@deploy1001: Started scap: Full scap for special alias changes for GrowthExperiments
19:50 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable WelcomeSurvey on cswiki and kowiki (T209725) (duration: 00m 46s)
19:44 catrope@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/WikimediaEvents/: EditorJourney fixes (T207307) (duration: 00m 46s)
19:36 catrope@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/GrowthExperiments/: WelcomeSurvey fixes (T206371) (duration: 00m 46s)
19:25 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable WelcomeSurvey on testwiki (T209725) (duration: 00m 49s)
18:29 cmjohnson1: connecting eqiad asw2-b fpc2 and fpc8
18:17 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@a25eb30]: GUI Update, new executor limits and new blazegraph build (duration: 08m 53s)
18:08 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@a25eb30]: GUI Update, new executor limits and new blazegraph build
15:45 ladsgroup@deploy1001: Finished deploy [ores/deploy@e957b24]: T209587 T170950 (duration: 17m 09s)
15:28 ladsgroup@deploy1001: Started deploy [ores/deploy@e957b24]: T209587 T170950
15:23 milimetric@deploy1001: Finished deploy [analytics/aqs/deploy@b399c34]: Removing empty fields from unique result (duration: 05m 17s)
15:18 milimetric@deploy1001: Started deploy [analytics/aqs/deploy@b399c34]: Removing empty fields from unique result
15:08 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting actor migration to write-both/read-old on group 0 (T188327) (duration: 00m 47s)
14:02 joal@deploy1001: Finished deploy [analytics/aqs/deploy@7cde8c8]: Update unique-devices schema adding 2 fields (duration: 20m 57s)
13:55 gtirloni: T207377 reboot cloudcontrol1004
13:43 moritzm: installing chromium security update on proton* (tested new upstream release in deployment-prep)
13:41 joal@deploy1001: Started deploy [analytics/aqs/deploy@7cde8c8]: Update unique-devices schema adding 2 fields
13:39 fdans@deploy1001: Finished deploy [analytics/aqs/deploy@7cde8c8]: Deploying AQS to add two new fields to uniques (duration: 06m 18s)
13:39 akosiaris: cumin -b1 -s 300 'ores2*' 'enable-puppet "merge of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/474158/" ; puppet agent -t ; service uwsgi-ores restart ; service celery-ores-worker restart'
13:36 akosiaris: disable puppet on ores1* and ores2* for slow deployment of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/474158/
13:33 fdans@deploy1001: Started deploy [analytics/aqs/deploy@7cde8c8]: Deploying AQS to add two new fields to uniques
13:21 gtirloni: T207377 icinga downtime and reboot of labcontrol1001 and labservices1001
13:08 arturo: T207377 icinga downtime and reboot of cloudcontrol1003 and cloudservices1003
13:06 raynor: EU SWAT finished
13:03 pmiazga@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:474679]|In SecurePoll use gpg1 to avoid gpg-agent autostart (T209802)]] (duration: 00m 48s)
12:50 raynor: EU SWAT reopened
12:40 raynor: EU SWAT finished
12:38 pmiazga@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:472918]|Enable autopatrol, patrol, rollback rights and RCPatrol on srwiktionary (T209252)]] (duration: 00m 46s)
12:21 pmiazga@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:474124]|Remove wgMetaNamespaceTalk for shnwiki (T206777)]] (duration: 00m 46s)
12:10 pmiazga@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:473225]|Enable Schema.org page split test at 50% sampling (T208755)]] (duration: 00m 46s)
11:33 gtirloni: labsdb1011 upgraded packages on labsdb1011 (pre-work T209517)
11:20 elukey: restart memcached on mc1020 to apply -R 200 settings (shard wiped) - T208844
10:41 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1078 T209754 (duration: 00m 46s)
10:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1078 T209754 (duration: 00m 46s)
10:21 banyek: stopping replication on db2076 (T85757)
10:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1078 T209754 (duration: 00m 46s)
09:57 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1123 and increase weight for db1078 T209754 (duration: 00m 46s)
09:48 marostegui: Rename table foundationwiki.petition_data on db1078 - T208979
09:46 marostegui: Drop empty testwiki.petition_data from db1075 with replication - T208979
09:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1123 and db1078 T209754 (duration: 00m 46s)
09:10 Nikerabbit: Rebuilt message group stats cache for T208521
08:43 banyek: executing schema change on db2095 (T85757)
07:57 marostegui: Stop MySQL on db1123 - T209754
07:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1123 to clone db1078 T209754 (duration: 00m 47s)
06:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Add db1078 line back to config file but depooled T209754 (duration: 00m 51s)
06:20 marostegui@deploy1001: sync-file aborted: Add db1078 line back to config file but depooled T209754 (duration: 00m 02s)

2018-11-18

17:26 andrewbogott: restarting cp1078 from mgmt console
09:00 elukey: cleaned up analytics1039 and restarted Yarn

2018-11-17

off: 'reset modified attributes' on IcingaUI for db1078 (and mgmt) and all its services
06:38 oblivian@deploy1001: Synchronized wmf-config/db-eqiad.php: Depooling db1078 (duration: 00m 59s)
02:54 RoanKattouw: Deployed patches for T208112, T208109, T208110

2018-11-16

23:13 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@ee91c41]: Deploy on test wdqs1010 (duration: 00m 23s)
23:12 smalyshev@deploy1001: Started deploy [wdqs/wdqs@ee91c41]: Deploy on test wdqs1010
16:23 Trey314159: reindexing Chinese wikis on elastic@eqiad and elastic@codfw (T209156)
15:46 moritzm: rebooting debmonitor* instances for kernel security update and to pick up SSBD
15:01 hashar: restarting zuul with 1300 events to process
14:56 marostegui: Create ipblocks_restrictions on labswiki and labtestwiki on db1073 - T209674
14:56 ema: upgrade cp-ats to 8.0.0-1wm2 T204225 T204209
14:39 ema: trafficserver 8.0.0-1wm2 uploaded to stretch-wikimedia T204225 T204209
14:36 hashar: Gracefully stopping zuul (kill -SIGUSR1)
14:29 _joe_: re-depooling mw1261 for php-fpm testing
14:28 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw126[1-6].*,dc=eqiad,cluster=appserver
14:26 _joe_: repooling the mw canaries
13:02 moritzm: installing spamassassin security update on mendelevium
12:10 godog: reboot restbase1014, nothing on console
11:13 kartik@deploy1001: Finished deploy [cxserver/deploy@473b0de]: Update cxserver to b7cdb26 (T208831, T203077, T203160, T206777) (duration: 04m 26s)
11:09 kartik@deploy1001: Started deploy [cxserver/deploy@473b0de]: Update cxserver to b7cdb26 (T208831, T203077, T203160, T206777)
10:39 akosiaris: upgrade OTRS to 5.0.32 T209691
09:31 marostegui: Set back sync_binlog=1 and trx_commit=1 after dbstore2002:3313 has caught up
09:25 moritzm: installing postgres updates on labsdb1006
09:21 moritzm: removed labvirt1016 from debmonitor db, got renamed to cloudvirt1016
08:40 moritzm: installing curl security updates on jessie
07:32 elukey: forced reboot + fsck + removal of /var/lib/hadoop/data/l from fstab on analytics1029
06:36 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Pool pc2009 in pc3 - T208383 (duration: 00m 56s)
06:28 marostegui: Set sync_binlog=0 and trx_commit=2 on dbstore2002:3313 to let it catch up
05:47 vgutierrez: uploaded certcentral 0.7 to apt.wikimedia.org (stretch) - T208967 T209475
00:55 mutante: some users reported missing files in home dirs on mwmaint1002, reversed rsyncd/ferm setup and rsynced /home from mwmaint2001 to /root on mwmaint1002, restored individually where requested, rsync is not fully automatic but puppetized with rsync::quickdatacopy
00:35 sbisson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/WikimediaEvents/includes/PageViews.php: SWAT: Exclude users where getRegistration() returns null (duration: 00m 47s)
00:26 eileen: civicrm revision changed from 71755d021b to 6b279509f8, config revision is 684ec9b7c0 (lybunt report)
00:22 sbisson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/GrowthExperiments: SWAT: gerrit:473843 gerrit:473844 gerrit:473845 (duration: 00m 49s)
00:21 sbisson@deploy1001: sync aborted: php-1.33.0-wmf.4/extensions/GrowthExperiments SWAT: gerrit:473843 gerrit:473844 gerrit:473845 (duration: 06m 16s)
00:15 sbisson@deploy1001: Started scap: php-1.33.0-wmf.4/extensions/GrowthExperiments SWAT: gerrit:473843 gerrit:473844 gerrit:473845

2018-11-15

21:58 mutante: mwmaint1002 - restoring entire /home of mwmaint1001 from Bacula (job queued and to tmp dir, not directly into /home)
21:06 hashar: Deleting Nodepool instances on contintcloud T209361
21:05 hashar: Stopped nodepool on labnodepool1001.eqiad.wmnet . Service is no more used. T209361 T209642
20:22 mholloway-shell@deploy1001: Finished deploy [kartotherian/deploy@UNKNOWN]: Fix: Loosen WDQS content-type header check to unbreak maps (T209471) (duration: 03m 55s)
20:18 mholloway-shell@deploy1001: Started deploy [kartotherian/deploy@UNKNOWN]: Fix: Loosen WDQS content-type header check to unbreak maps (T209471)
20:17 mutante: re-added Chase to pwstore, signed .users file, re-encrypted all pwstore files, git pushed
20:15 mholloway-shell@deploy1001: Finished deploy [kartotherian/deploy@48a1e83]: Fix: Loosen WDQS content-type header check to unbreak maps (T209471) (duration: 04m 26s)
20:11 mholloway-shell@deploy1001: Started deploy [kartotherian/deploy@48a1e83]: Fix: Loosen WDQS content-type header check to unbreak maps (T209471)
20:04 urandom: dropping disused keyspaces -- T208616
20:04 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable data collection for UnderstandingFirstDay on cswiki and kowiki (duration: 00m 53s)
19:51 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Configure sensitive namespaces for EditorJourney schema (T207307) (duration: 00m 53s)
19:49 catrope@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/WikimediaEvents/: cherry-picks for T208773 (duration: 00m 54s)
18:55 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Hot-deploy I26f2dc2e: Don't over-ride default Wikibase string limits (duration: 00m 53s)
18:50 jforrester@deploy1001: Synchronized wmf-config/Wikibase.php: Hot-deploy Ib10de2e3: Don't set Wikibase string limits when null (duration: 00m 55s)
18:31 ladsgroup@deploy1001: Finished deploy [ores/deploy@dba11e9]: Another small update (duration: 13m 42s)
18:18 ladsgroup@deploy1001: Started deploy [ores/deploy@dba11e9]: Another small update
18:17 ladsgroup@deploy1001: Finished deploy [ores/deploy@51cdf6b]: T208623 (duration: 14m 41s)
18:03 ladsgroup@deploy1001: Started deploy [ores/deploy@51cdf6b]: T208623
17:27 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 7 wikis in group 2 for T209373. This may cause lag in codfw.
17:27 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 6 wikis in group 2 for T209373. This may cause lag in codfw.
17:27 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 5 wikis in group 2 for T209373. This may cause lag in codfw.
17:27 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 3 wikis in group 2 for T209373. This may cause lag in codfw.
17:27 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 2 wikis in group 2 for T209373. This may cause lag in codfw.
17:27 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 1 wikis in group 2 for T209373. This may cause lag in codfw.
17:20 gehel: upgrade prometheus-blazegraph-exporter on all wdqs nodes - T206123
16:41 bstorm_: rebooted labsdb1007 for upgrades
16:37 andrewbogott: rebuilding labvirt1015 and cloudvirt1015
14:21 anomie@mwmaint1002: Running refreshExternallinksIndex.php on wikitech for T209373. This may cause lag in codfw.
14:21 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 8 wikis in group 1 for T209373. This may cause lag in codfw.
14:21 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 7 wikis in group 1 for T209373. This may cause lag in codfw.
14:21 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 5 wikis in group 1 for T209373. This may cause lag in codfw.
14:21 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 4 wikis in group 1 for T209373. This may cause lag in codfw.
14:21 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 3 wikis in group 1 for T209373. This may cause lag in codfw.
14:20 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 2 wikis in group 1 for T209373. This may cause lag in codfw.
14:07 gehel: plugin and JVM upgrade on elasticsearch / cirrus / eqiad completed - T209293
13:59 hashar@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.4
13:55 XioNoX: push firewall policies to pfw3-eqiad - T209421
13:50 banyek: Deploy schema change on s6 codfw master, this will generate lag on s6 codfw - T85757
13:48 moritzm: installing qemu security updates (which also backport support for SSBD passthrough) on ganeti clusters
13:48 moritzm: installing qemu security updates (which also backport support for SSBD passthrough)
13:12 mobrovac@deploy1001: Finished deploy [restbase/deploy@22cb0ec]: Add new wikis to RESTBase - T206777 T205710 T205546 T204477 (duration: 19m 56s)
13:00 Lucas_WMDE: EU SWAT finished
12:59 tarrow@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/RevisionSlider/modules/ext.RevisionSlider.SliderView.js: gerrit:473710 Fix (accidentally?) reversed blue and yellow lines SWAT T208238 T162119 again (duration: 00m 55s)
12:57 _joe_: upping pm.maxworkers to 40 on mw1261 on php7.2-fpm, benchmarking T206341
12:56 tarrow@deploy1001: scap failed: average error rate on 3/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details)
12:53 moritzm: installing nginx security updates
12:52 mobrovac@deploy1001: Started deploy [restbase/deploy@22cb0ec]: Add new wikis to RESTBase - T206777 T205710 T205546 T204477
12:47 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Remove wgWBQualityConstraintsCacheCheckConstraintsResults (T207854) (duration: 00m 54s)
12:42 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Prod: increase Schema.org page split test to 25% sampling T208755 (duration: 00m 53s)
12:40 arturo: T207377 downtime and reboot labmon1001
12:38 lucaswerkmeister-wmde@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/RevisionSlider/modules/ext.RevisionSlider.SliderView.js: Fix (accidentally?) reversed blue and yellow lines (T162119, T208238) (duration: 00m 54s)
12:37 lucaswerkmeister-wmde@deploy1001: sync aborted: php-1.33.0-wmf.3/extensions/RevisionSlider/modules/ext.RevisionSlider.SliderView.js Fix (accidentally?) reversed blue and yellow lines (T162119, T208238) (duration: 00m 11s)
12:37 lucaswerkmeister-wmde@deploy1001: Started scap: php-1.33.0-wmf.3/extensions/RevisionSlider/modules/ext.RevisionSlider.SliderView.js Fix (accidentally?) reversed blue and yellow lines (T162119, T208238)
12:30 tarrow@deploy1001: Synchronized wmf-config/Wikibase.php: gerrit:473716 Read WikibaseStringLimit in Wikibase.php T154660 (duration: 00m 53s)
12:30 moritzm: draining ganeti1001 for reboot/kernel security update
12:26 tarrow@deploy1001: sync-file aborted: gerrit:473716 Read WikibaseStringLimit in Wikibase.php (duration: 00m 01s)
12:21 tarrow@deploy1001: Synchronized wmf-config/InitialiseSettings.php: gerrit:473694 Set Wikibase string-limits for wikidata dblist T154660 (duration: 00m 54s)
12:08 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Make AdvancedSearch the default on de-, fa-, ar-, and hu-wiki T207640 (duration: 00m 55s)
11:34 _joe_: depooling mw1261 for early benchmarks of php7.2-fpm
11:04 akosiaris: enable puppet across the fleet. puppetdb1001 reboot done, ganeti migration_downtime setting applied
10:49 akosiaris: disable puppet across the fleet for puppetdb1001 reboot
10:49 moritzm: fail over ganeti master in eqiad to ganeti1003
10:34 akosiaris: set migration_downtime=2000 for puppetdb1001. Should help with migration stalls
10:15 banyek: sanitizing db1124 ( T205714 T207584 T205713 T206916 )
10:07 banyek: sanitizing db2094 ( T205714 T207584 T205713 T206916 )
09:57 moritzm: draining ganeti1002 for reboot/kernel security update
09:40 volans: restarting icinga on icinga1001
09:32 vgutierrez: restarting icinga on icinga1001
09:31 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1110 (duration: 00m 52s)
09:22 moritzm: draining ganeti1003 for reboot/kernel security update
09:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1110 (duration: 00m 53s)
09:16 marostegui: Stop MySQL on db1110 for upgrade
09:08 moritzm: reset failed debmonitor session in ms-be2038
09:08 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1088 (duration: 00m 53s)
09:03 banyek: repooling db1088 (T85757)
08:58 ema: upload fifo-log-demux 0.1 to stretch-wikimedia T204225
08:56 banyek: Deploy schema change on db1088 (T85757)
08:49 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1088 (duration: 00m 53s)
08:45 banyek: depooling db1088 due a schema change (T85757)
08:21 moritzm: draining ganeti1004 for reboot/kernel security update
07:42 marostegui: Drop site_stats.ss_total_views from labswiki - T86339
07:08 elukey: memcached on mc1019 restarted to apply -R 200 - T208844
06:57 marostegui: Stop MySQL on db2071 to upgrade MySQL and kernel
06:57 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2071 (duration: 00m 54s)
06:06 marostegui: Stop MySQL on pc2006 to clone pc2009 - T208383
06:06 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool pc2006 - T208383 (duration: 00m 53s)
06:05 marostegui@deploy1001: sync-file aborted: Dool pc2006 - T208383 (duration: 00m 00s)
05:59 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Pool pc2008 in pc2 - T208383 (duration: 00m 56s)
00:50 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: increase Schema.org page split test to 5% sampling T208755 (duration: 00m 54s)

2018-11-14

22:46 ejegg: updated payments-wiki from 5751286f1c to d2b66c5bab
21:34 thcipriani: restart gerrit to load JavaMelody dependency library
21:28 thcipriani@deploy1001: Finished deploy [gerrit/gerrit@ab2fa18]: deploy javamelody on cobalt (duration: 00m 09s)
21:28 thcipriani@deploy1001: Started deploy [gerrit/gerrit@ab2fa18]: deploy javamelody on cobalt
21:27 thcipriani@deploy1001: Finished deploy [gerrit/gerrit@ab2fa18]: deploy javamelody on gerrit2001 (duration: 00m 11s)
21:26 thcipriani@deploy1001: Started deploy [gerrit/gerrit@ab2fa18]: deploy javamelody on gerrit2001
20:13 hashar@deploy1001: Synchronized php: group1 wikis to 1.33.0-wmf.4 (duration: 00m 52s)
20:12 hashar@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.4
19:14 hashar@deploy1001: Synchronized php-1.33.0-wmf.4/includes/jobqueue/JobQueue.php: Actually return the value from getRootJobCacheKey() - T209429 (duration: 00m 53s)
18:33 hashar@deploy1001: Finished scap: php-1.33.0-wmf.4/includes/libs/objectcache/MemcachedPeclBagOStuff.php Add trace to debug memcached bad key error - T209429 (duration: 34m 07s)
17:58 hashar@deploy1001: Started scap: php-1.33.0-wmf.4/includes/libs/objectcache/MemcachedPeclBagOStuff.php Add trace to debug memcached bad key error - T209429
17:53 arturo: T207377 downtime and reboot cloudnet1003 (cloudnet1004 is the active one already)
17:37 arturo: T207377 downtime and reboot cloudnet1004 (cloudnet1003 is the active one already)
17:31 bawolff_: Running importImage.php for 'Opening ceremony of First accusation protest against presumption of guilt of judicial branch.webm' per request T209495
17:26 sbisson@deploy1001: Synchronized dblists/wikidataclient.dblist: SWAT: Add incubatorwiki to wikidataclient.dblist (duration: 00m 48s)
17:19 sbisson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/MobileFrontend/resources/mobile.editor.common/schemaEditAttemptStep.js: SWAT: schemaEditAttemptStep.js: Use correct config var name for sampling rate (duration: 00m 54s)
17:12 sbisson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/WikimediaEvents/includes/WikimediaEventsHooks.php: SWAT: Fix EditAttemptStepSamplingRate variable export (duration: 00m 54s)
16:33 bblack: [Done] replacement of GlobalSign unified TLS cert at US edges complete - T206804
16:25 moritzm: rebooting ganeti1005 for kernel security update
16:16 moritzm: rebooting restbase-dev1006 for kernel security update and OpenJDK security update
16:10 bblack: disabling puppet as precaution on all caches (cumin A:cp) - T206804
16:09 bblack: starting replacement of GlobalSign unified TLS cert at US edges (affects all public TLS termination for US traffic edges) - T206804
16:08 moritzm: rebooting restbase-dev1005 for kernel security update and OpenJDK security update
15:58 moritzm: rebooting restbase-dev1004 for kernel security update and OpenJDK security update
15:53 jiji: Restarting pdfrender on scb*.eqiad.wmnet
15:50 godog: roll restart swift-proxy in eqiad to apply statsd changes
15:41 banyek@deploy1001: Synchronized wmf-config/db-codfw.php: T85757: repool db2046 (duration: 00m 52s)
15:39 banyek: repooling db2046 (T85757)
15:12 godog: roll-restart swift on ms-be1* to pick up statsd changes
15:07 Amir1: ladsgroup@mwmaint1002:/srv/mediawiki-staging/php-1.33.0-wmf.4$ mwscript sql.php --wiki=incubatorwiki extensions/Wikibase/client/sql/entity_usage.sql (T209207)
14:56 addshore@deploy1001: Synchronized wmf-config: Prod: Enable Schema.org page split test at 1% sampling (again) (duration: 00m 54s)
14:29 godog: roll-restart swift-proxy in codfw to pick up statsd changes
14:14 addshore@deploy1001: Synchronized wmf-config: Revert Prod: Enable Schema.org page split test at 1% sampling (duration: 00m 54s)
14:10 Reedy: Wiki created T205714 T207584 T205713 T206916
14:07 gehel: starting plugin and JVM upgrade on elasticsearch / cirrus / eqiad - T209293
14:07 reedy@deploy1001: Synchronized wmf-config/interwiki.php: Updating interwiki cache (duration: 02m 25s)
14:02 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Fix shnwiki TZ (duration: 00m 54s)
13:57 reedy@deploy1001: rebuilt and synchronized wikiversions files: (no justification provided)
13:55 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Add pc2010 as spare - T208383 (duration: 00m 53s)
13:54 reedy@deploy1001: Synchronized multiversion/MWMultiVersion.php: (no justification provided) (duration: 00m 53s)
13:51 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: new wikis (duration: 00m 53s)
13:51 gehel: restarting tilerator on maps1004 for config change
13:50 reedy@deploy1001: Synchronized static/images/project-logos/: (no justification provided) (duration: 00m 53s)
13:49 reedy@deploy1001: Synchronized dblists/: new wikis! (duration: 00m 53s)
13:48 reedy@deploy1001: Synchronized langlist: shn (duration: 00m 52s)
13:45 gehel: plugin and JVM upgrade completed on elasticsearch / cirrus / codfw - T209293
13:45 reedy@deploy1001: rebuilt and synchronized wikiversions files: (no justification provided)
13:07 moritzm: installing ghostscript security updates on stretch
13:01 reedy@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/WikimediaMaintenance/addWiki.php: Fixing addshores code... (duration: 00m 53s)
13:00 reedy@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/WikimediaMaintenance/addWiki.php: Fixing addshores code... (duration: 00m 55s)
12:56 moritzm: installing gettext "security" updates for trusty
12:48 moritzm: installing python3.4 security updates on trusty (Debian already fixed)
12:40 reedy@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/WikimediaMaintenance/addWiki.php: Unbreak adding wiktionary (duration: 00m 52s)
12:39 reedy@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/WikimediaMaintenance/addWiki.php: Unbreak adding wiktionary (duration: 00m 53s)
12:39 moritzm: installing python security updates on trusty
12:36 Amir1: EU SWAT is done
12:35 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert the language of votewiki to English (en) (T207560) (duration: 00m 55s)
12:30 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Start reading from change_tag_def on wikidatawiki (T208846) (duration: 00m 55s)
12:19 pmiazga@deploy1001: Synchronized wmf-config: SWAT: [[gerrit:473079]|Enable Schema.org page split test at 1% sampling (T208755)]] (duration: 00m 54s)
12:02 reedy@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/WikimediaMaintenance/addWiki.php: Unbreak adding TMH tables (duration: 00m 53s)
12:01 reedy@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/WikimediaMaintenance/addWiki.php: Unbreak adding TMH tables (duration: 00m 55s)
11:11 moritzm: draining ganeti1005 for reboot/kernel security update
11:09 banyek: Deploy schema change on db2046 (T85757)
10:07 banyek@deploy1001: Synchronized wmf-config/db-codfw.php: T85757: depooling db2046 (duration: 00m 55s)
09:59 banyek: depooling db2046 (T85757)
09:22 moritzm: updated stretch netinst image for 9.6 point release
09:17 marostegui: Deploy schema change on db2053 - T86339
08:24 marostegui: Deploy schema change on s5 codfw master, this will generate lag on s5 codfw - T205913
08:22 marostegui: Deploy schema change on s2 codfw master, this will generate lag on s7 codfw - T205913
08:19 marostegui: Deploy schema change on s2 codfw master, this will generate lag on s2 codfw - T205913
08:17 marostegui: Deploy schema change on s6 codfw master, this will generate lag on s6 codfw - T205913
08:14 marostegui: Deploy schema change on s4 codfw master, this will generate lag on s4 codfw - T205913
08:08 marostegui: Deploy schema change on s3 codfw master, this will generate lag on s3 codfw - T205913
08:07 godog: rollout rsyslog_exporter to eqiad
07:42 marostegui: Deploy schema change on s3 codfw master, this will generate lag on s3 codfw - T203709
07:19 marostegui: Deploy schema change on s7 codfw master, this will generate lag on s7 codfw - T203709
07:07 marostegui: Deploy schema change on s2 codfw master, this will generate lag on s2 codfw - T203709
06:52 marostegui: Deploy schema change on s4 codfw master, this will generate lag on s4 codfw - T203709
06:40 marostegui: Deploy schema change on s6 codfw master, this will generate lag on s6 codfw -T203709
06:32 marostegui: Stop MySQL on pc2005 to clone it to pc2008 - T208383
06:27 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool pc2005 - T208383 (duration: 01m 04s)
05:46 _joe_: restarting gerrit
01:02 thcipriani@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/Wikibase/client/includes: SWAT: Update: use wikibase-debug logger instead of "PageRandomLookup" T208796 (duration: 00m 56s)
00:42 mutante: restarted smokeping on netmon1002 and netmon2001

2018-11-13

22:42 XioNoX: restart librenms irc bot
22:24 XioNoX: add term labnet-nova-api to cloud-in4 on cr1/2-eqiad - T209424
20:22 herron: updated labs realm smarthosts (via hiera) to mx-out0[12].wmflabs.org T41785
19:49 otto@deploy1001: Finished deploy [analytics/refinery@62d6f4b]: Deploy hive jars from CDH 5.10.0 to workaround Refine bug: T209407 (duration: 05m 57s)
19:43 otto@deploy1001: Started deploy [analytics/refinery@62d6f4b]: Deploy hive jars from CDH 5.10.0 to workaround Refine bug: T209407
19:31 herron: uploaded librdkafka_0.11.6-1~bpo9+1+wikimedia1 packages to stretch-wikimedia T209300
18:11 mutante: the CUSTOM message from ores.svc.codfw was the (one-time) test of the new Icinga server
18:03 mutante: icinga migration has concluded, we are now on stretch and icinga1001, einsteinium is passive (T202782)
17:27 mutante: re-enabled puppet on icinga1001, einsteinium becoming passive
17:21 mutante: ran puppet on einsteniumr; e-enabling puppet on tegmen and icinga1001
17:13 bstorm_: Added 172.16.0.0/21 to the allowed connections for wikilabels postgresql on labsdb1004
17:04 mutante: disabled puppet on all 3 icinga servers, re-enabling on einsteinium , going through https://wikitech.wikimedia.org/wiki/Icinga#Failover_Icinga_between_the_active_and_passive_servers
17:02 ejegg: updated payments-wiki from 20542c9184 to 5751286f1c
17:01 mutante: starting migration of icinga server - maintenance windows
16:33 thcipriani: restarting gerrit service for upgrade to 2.15.6
16:32 thcipriani@deploy1001: Finished deploy [gerrit/gerrit@d2763c6]: v2.15.6 to cobalt (duration: 00m 10s)
16:32 thcipriani@deploy1001: Started deploy [gerrit/gerrit@d2763c6]: v2.15.6 to cobalt
16:29 thcipriani@deploy1001: Finished deploy [gerrit/gerrit@d2763c6]: v2.15.6 to gerrit2001 (duration: 00m 11s)
16:29 thcipriani@deploy1001: Started deploy [gerrit/gerrit@d2763c6]: v2.15.6 to gerrit2001
16:22 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting actor migration to write-both/read-old on test wikis and mediawikiwiki (T188327) (duration: 00m 54s)
16:07 anomie@mwmaint1002: Running refreshExternallinksIndex.php on labtestwiki for T209373
16:07 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 3 wikis in group 0 for T209373
15:48 _joe_: upgrading extensions on all appservers / jobrunners while upgrading to php 7.2
15:45 gehel: restart tilerator on maps1004
15:21 moritzm: draining ganeti1006 for reboot/kernel security update
15:18 marostegui: Restore replication consistency options on dbstore2002:3313 as it has caught up - T208320
14:59 akosiaris: increase the migration downtime for kafkamon1001. It should make live migration of these VMs easier and without the need for manual fiddling
14:54 hashar@deploy1001: rebuilt and synchronized wikiversions files: group to 1.33.0-wmf.4 | T206658
14:40 hashar@deploy1001: Finished scap: testwiki to php-1.33.0-wmf.4 | T206658 (duration: 19m 34s)
14:27 moritzm: draining ganeti1007 for reboot/kernel security update
14:20 hashar@deploy1001: Started scap: testwiki to php-1.33.0-wmf.4 | T206658
14:20 akosiaris: reboot logstash1007, logstash1008, logstash1009 with 500 secs of sleep between them for the migration_downtime ganeti setting to be applied
14:18 akosiaris: increase the migration downtime for logstash1007, logstash1008, logstash1009. It should make live migration of these VMs easier and without the need for manual fiddling
14:15 hashar@deploy1001: Pruned MediaWiki: 1.32.0-wmf.24 (duration: 08m 55s)
14:03 hashar: Applied security patches to 1.33.0-wmf.4 | T206658
14:03 gehel: start plugin and JVM upgrade on elasticsearch / cirrus / codfw - T209293
14:00 hashar: scap prep 1.33.0-wmf.4 # T206658
13:58 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Pool pc2007 to replace pc2004 (duration: 00m 48s)
13:41 marostegui: Deploy schema change on s8 codfw master (db2045) this will generate lag on s8 codfw - T203709
13:40 hashar: Cutting wmf/1.33.0-wmf.4 branch | T206658
13:30 moritzm: draining ganeti1008 for reboot/kernel security update
12:51 phuedx: European Mid-day SWAT finished
12:50 phuedx@deploy1001: Finished scap: SWAT: Define WikimediaMessages for Wikibase SEO change l18n refresh (duration: 21m 43s)
12:28 phuedx@deploy1001: Started scap: SWAT: Define WikimediaMessages for Wikibase SEO change l18n refresh
12:22 phuedx@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/WikimediaMessages/: SWAT: Define WikimediaMessages for Wikibase SEO change (T208755) (duration: 00m 56s)
10:57 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1092 (duration: 00m 52s)
10:47 marostegui: Deploy schema change on db1116:3318 T203709
10:40 godog: stop sending metrics to old graphite hardware
10:15 gehel: restart elasticsearch on relforge for plugin upgrade - T209293
09:54 moritzm: restarting jenkins on releases1001 to pick up Java security update
09:25 _joe_: uploading new versions of php-msgpack, php-geoip compatible with both php 7.0 and php 7.2 to thirdparty/php72 T208433
09:23 marostegui: Deploy schema change on db1092 T203709
09:23 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1092 (duration: 00m 52s)
09:20 elukey: rollout new prometheus-mcrouter-exporter to mw* - previous rollout didn't work as expected
09:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1104 (duration: 00m 55s)
08:37 moritzm: updating remaining rsyslog on stretch to 8.38.0-1~bpo9+1wmf1
07:21 marostegui: Deploy schema change on db1104 T203709
07:20 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1104 (duration: 00m 53s)
07:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1109 (duration: 00m 54s)
07:05 elukey: powercycle lvs2006 - mgmt/serial console blank, not responsive since hours ago
06:02 marostegui: Add ipb_sitewide column to db1073:labtestwiki
05:43 marostegui: Stop MySQL on pc2004 to transfer its data to pc2007 - T208383
05:42 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool pc2004 - T208383 (duration: 00m 53s)
05:39 marostegui: Deploy schema change on db2048 (s1 codfw master), this will create lag on s1 codfw - T114117
05:34 marostegui: Deploy schema change on db1109 T203709
05:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1109 (duration: 00m 55s)

2018-11-12

19:22 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT T208663 4ff32d1df - Enable moving files for users with patrol and rollbacker rights on srwiki (duration: 00m 54s)
18:29 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@ee91c41]: GUI update, New Thesaurus endpoint, New updater build and blazegraph update (duration: 11m 28s)
18:17 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@ee91c41]: GUI update, New Thesaurus endpoint, New updater build and blazegraph update
18:03 elukey: rolling restart of aqs on aqs* to pick up new druid datasource settings
17:44 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Optimize s2 for throughput (duration: 00m 53s)
17:19 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Pool more resources into s2 api (duration: 00m 54s)
17:15 _joe_: restarting HHVM on the high-cpu api hosts in eqiad, to ease the pressure and latencies
17:10 _joe_: depooling mw1222 for debug
16:41 banyek: disabling puppet on parsercache hosts (T208383)
16:14 elukey: upgrade prometheus-mcrouter-exporter on all the mw* hosts to the new version
16:09 phuedx: phuedx@mwmaint1002 running restPageRandom.php maintenance script for large wikis
16:02 volans: restarted proton on proton1002
15:45 jynus: stop and upgrade db2094
15:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 (duration: 00m 53s)
14:49 banyek: disabling puppet on parsercache hosts - pc[12]00[456] (T208383)
14:17 phuedx: phuedx@mwmaint1002 running restPageRandom.php maintenance script for medium wikis
14:08 moritzm: updating rsyslog on stretch to 8.38.0-1~bpo9+1wmf1
13:59 phuedx: phuedx@mwmaint1002 running restPageRandom.php maintenance script for small wikis (small.dblist)
13:59 marostegui: Deploy schema change on db1101:3318 - T203709
13:55 hashar: Upgrading Jenkins on contint1001 , contint2001, releases1001 and releases2002 | T209264
13:46 moritzm: updating libfastjson on stretch to 0.99.8-1~bpo9+1wmf1
13:41 gehel: starting rolling restart of elasticsearch codfw for JVM upgrade
13:32 phuedx: phuedx@mwmaint1002 running restPageRandom.php maintenance script for mediawikiwiki
13:23 phuedx: phuedx@mwmaint1002 running resetPageRandom.php maintenance script for testwiki
13:17 zeljkof: EU SWAT finished
13:16 phuedx@deploy1001: Synchronized php-1.33.0-wmf.3/maintenance/resetPageRandom.php: SWAT: Provide a script to reset the page_random column (T208909) (duration: 00m 53s)
13:16 moritzm: updating liblognorm on stretch to 2.0.3-1~bpo9+1wmf1
13:14 phuedx@deploy1001: Synchronized php-1.33.0-wmf.3/autoload.php: SWAT: Provide a script to reset the page_random column (T208909) (duration: 00m 55s)
13:12 elukey: upgrade the Hadoop Analytics cluster to CDH 5.15 (downtime required)
12:54 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: Add new throttle rule for Wikipedia event in Ireland on 2018-11-13 (T209037) (duration: 00m 53s)
12:15 jiji: Restarting nutcracker on scb200[1-6] - T206450
12:00 moritzm: uploaded jenkins 2.138.3 to apt.wikimedia.org (jessie and stretch)
11:49 hashar: updating puppet CI job for mtail upgrade https://gerrit.wikimedia.org/r/#/c/integration/config/+/472962/
11:37 hashar: contint1001 : cleaning disk | T209123 ?
11:26 moritzm: installing Java security updates on elastic*
10:51 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101:3318 (duration: 00m 55s)
10:48 godog: upload mtail 3.0.0~rc5-1~bpo9+1wmf1 to stretch-wikimedia
10:45 marostegui: Deploy schema change on db2048 (s1 codfw master), this will generate lag on s1 codfw - T51191
10:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099:3318 (duration: 00m 53s)
10:32 elukey: upload mcrouter exporter 0.0.0+git20181106 to stretch-wikimedia
09:57 elukey: upgraded cdh packages (cdh 5.10 -> 5.15) for thirdparty/cloudera in jessie/stretch-wikimedia
09:12 marostegui: Deploy schema change on db2048 (s1 codfw master) (replication will be stopped) - T67448
08:53 marostegui: Deploy schema change on db1099:3318 - T203709
08:52 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099:3318 (duration: 01m 01s)
08:41 marostegui: Change sync_binlog to 0 and trx_commit to 2 on dbstore2002:3313 to let it catch up
{{safesubst:SAL entry|1=08:26 _joe_: uploading new php-{luasandbox,wikidiff2} to stretch main component, rebuild php-{luasandbox,wikidiff2,geoip,msgpack} for php 7.2, upload to stretch component php72, T208433}}
08:23 godog: temporarily disable puppet in codfw before enabling rsyslog_exporter

2018-11-10

01:10 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@7eeede7]: redeploy runUpdate.sh for better reporting (duration: 00m 39s)
01:10 smalyshev@deploy1001: Started deploy [wdqs/wdqs@7eeede7]: redeploy runUpdate.sh for better reporting
01:07 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@7eeede7]: redeploy runUpdate.sh for better reporting (duration: 00m 04s)
01:07 smalyshev@deploy1001: Started deploy [wdqs/wdqs@7eeede7]: redeploy runUpdate.sh for better reporting

2018-11-09

21:46 SMalyshev: repooled wdqs1004 - looks like other servers feel worse so probably makes sense to share the load equally
21:16 jiji: Reimaging rdb2003, rdb2004 - T206450
20:46 SMalyshev: depooled wdqs1004 to let it catch up
20:40 legoktm@deploy1001: Synchronized docroot/mediawiki/keys/: Add my (20after4) PGP key to mediawiki.org/keys/keys.(txt|html) (duration: 00m 55s)
20:08 andrewbogott: restarted neutron-linuxbridge-agent on cloudvirt1018 and cloudvirt1023
away: repooling labsdb1011 (T189158)
15:24 banyek: depooling labsdb1011 (T189158)
15:23 banyek: depooling labsdb1011
15:08 banyek: repooling labsdb1009 (T189158)
15:06 bblack: cp1008/pinkunicorn: puppet disabled, public-facing testing of new globalsign 2018 certs
15:04 ladsgroup@deploy1001: Finished deploy [ores/deploy@bb39f4b]: T191842 T209060, try II (duration: 14m 43s)
14:50 andrewbogott: rebooting cloudvirt1024 to (I hope) cause a page
14:49 ladsgroup@deploy1001: Started deploy [ores/deploy@bb39f4b]: T191842 T209060, try II
14:48 ladsgroup@deploy1001: deploy aborted: T191842 T209060 (duration: 09m 32s)
14:39 ladsgroup@deploy1001: Started deploy [ores/deploy@0728805]: T191842 T209060
14:18 addshore@deploy1001: Synchronized wmf-config: BETA ONLY: Enable SSR termbox for wikibase on beta - T209143 (duration: 00m 56s)
13:32 moritzm: rebooting acrab for some qemu tests
13:21 godog: upload graphite-web_1.0.2+debian-2.1wmf1 to stretch-wikimedia - T208782
13:10 moritzm: upgrading qemu on ganeti2001 (packages supporting SSBD passthrough)
12:40 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T189158: repool db1106 (duration: 00m 53s)
12:34 banyek: repooling db1106 (T208954)
12:17 kartik@deploy1001: Finished deploy [cxserver/deploy@fc21164]: Update cxserver to 01686f6 (T208831) (duration: 01m 09s)
12:16 kartik@deploy1001: Started deploy [cxserver/deploy@fc21164]: Update cxserver to 01686f6 (T208831)
11:45 banyek: data load finished restarting replication on db1106 (T208954)
11:43 akosiaris: set previous normal wait for scb1001 for apertium service T206439
11:39 akosiaris@puppetmaster1001: conftool action : set/weight=8; selector: dc=eqiad,service=apertium,cluster=scb,name=scb1001.*
11:30 akosiaris: upgrade apertium apertium-cat apertium-fra apertium-fra-cat apertium-lex-tools apertium-separable cg3 libapertium3-3.5-1 libcg3-1 lttoolbox on all scb boxes and restart apertium-apy
11:26 akosiaris: upgrade apertium apertium-cat apertium-fra apertium-fra-cat apertium-lex-tools apertium-separable cg3 libapertium3-3.5-1 libcg3-1 lttoolbox on scb1002
11:22 jiji: switch scb*.eqiad.wmnet nutcracker rdb1003:6382 with rdb1005:6379
10:51 vgutierrez: uploaded certcentral 0.6 to apt.wikimedia.org (stretch) - T208859 T208948 T208967 T208970
09:48 ema: repool cp2018, cp2025 (cache_upload) T208588
09:45 banyek: truncating enwiki.archive on db1124 and labsdb hosts too (T208954)
09:21 banyek: stopping replication on db1106 (T208954)
09:21 banyek: stopping replication on db1106 (T208672)
09:08 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T189158: depool db1106 (duration: 00m 55s)
09:02 banyek: depooling db1106 (T208954)
08:28 moritzm: installing nginx security updates
08:05 ema: repool cp2006, cp2012 (cache_text) T208588
04:33 ejegg: restarted recurring donation charge jobs
04:24 ejegg: updated fundraising CiviCRM from 1154cca3f2 to 71755d021b
03:25 ejegg: updated fundraising CiviCRM from 02cc1f80d4 to 1154cca3f2
00:07 ejegg: updated fundraising CiviCRM from 07183ed7cc to 02cc1f80d4
00:03 ejegg: updated payments-wiki from 983ce3af0f to 20542c9184

2018-11-08

22:48 mutante: gerrit - adding Thomas Arrow to 'wmf-deployment' group for +2 on mw-config for T208491 access request
22:37 mutante: gerrit - adding Lucas Werkmeister (WMDE) to 'wmf-deployment' group for +2 on mw-config for T208518 access request
20:28 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.3
19:35 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Disable wmgUseTwoColConflict everywhere" T205942 T208840 T209012 T209036 (duration: 00m 54s)
18:19 shdubsh: update statsd-proxy to 0.0.9-2 on graphite1004
17:29 banyek: depooling labsdb1009 (T189158)
17:24 banyek: repooling labsdb1010 (T189158)
17:07 godog: upload libfastjson 0.99.8-1~bpo9+1wmf1 version bump only
16:59 akosiaris@deploy1001: scap-helm zotero finished
16:59 akosiaris@deploy1001: scap-helm zotero cluster staging completed
16:59 akosiaris@deploy1001: scap-helm zotero [namespace: zotero, clusters: staging]
16:50 akosiaris@deploy1001: scap-helm zotero install --name alextest --set main_app.version=20181019165254-production --set monitoring.enable=true charts/zotero [namespace: zotero, clusters: staging]
16:50 akosiaris@deploy1001: scap-helm zotero install --name alextest --set main_app.version=20181019165254-production --set monitoring.enable=true charts/zotero [namespace: zotero, clusters: staging]
16:29 XioNoX: enable Zayo transit on cr3-ulsfo
15:42 chasemp: disable /etc/logrotate.d/udp2log-mw for a bit on mwlog1001
15:25 Amir1: rolling restart of celery on ores nodes (T209060)
15:20 akosiaris: 'cd /srv/deployment/ores/deploy/submodules/wheels && sudo -u deploy-service git lfs pull' on all ores1* and ores2* hosts T209060
15:07 XioNoX: zeroize asw-c8-codfw (decom)
14:12 moritzm: rebooting releases2001 for some tests with ssbd for KVM
13:52 moritzm: installing postgres updates on labsdb1006/1007
13:38 jiji: Done reimaging rdb1006 - T206450
13:37 moritzm: draining ganeti2001 for reboot/kernel security update
13:36 moritzm: failing over ganeti master in codfw from ganeti2001 to ganeti2003
13:13 godog: upload rsyslog 8.38.0-1~bpo9+1wmf1 to stretch-wikimedia, version bump only
13:07 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Re-enable wmgUseTwoColConflict on dewiki - T205942 T208840 T209012 T209036 (duration: 00m 53s)
12:56 akosiaris: increase weight of scb1001 for apertium to 99+%
12:56 akosiaris@puppetmaster1001: conftool action : set/weight=3800; selector: dc=eqiad,service=apertium,cluster=scb,name=scb1001.*
12:53 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Re-enable wmgUseTwoColConflict on group0 only T205942 T208840 T209012 T209036 (duration: 00m 54s)
12:52 moritzm: draining ganeti2002 for reboot/kernel security update
12:41 jiji: Shutdown and reimage rdb200[56] - T206450
12:31 moritzm: draining ganeti2003 for reboot/kernel security update
12:30 zeljkof: EU SWAT finished
12:29 zfilipin@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/TwoColConflict: SWAT: Fix harmless edits turning into conflicts (T205942 T208840 T209012 T209036) (duration: 00m 55s)
12:19 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set AdvancedSearch to default on group0 wikis (T207641) (duration: 00m 55s)
12:18 moritzm: draining ganeti2004 for reboot/kernel security update
11:57 moritzm: draining ganeti2005 for reboot/kernel security update
11:51 akosiaris: increase weight of scb1001 for apertium to 50%
11:50 akosiaris@puppetmaster1001: conftool action : set/weight=38; selector: dc=eqiad,service=apertium,cluster=scb,name=scb1001.*
11:41 moritzm: draining ganeti2006 for reboot/kernel security update
11:18 moritzm: draining ganeti2007 for reboot/kernel security update
11:05 moritzm: draining ganeti2008 for reboot/kernel security update
10:52 jiji: Reimaging rdb1006 to stretch - T206450
10:52 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Disable wmgUseTwoColConflict everywhere T209012 T208840 T195724 (duration: 00m 58s)
10:26 elukey: restart memcached on mc2029 (was depooled yesterday for network maintenance)
10:23 jiji: restarting pdfrender on scb1003
10:19 volans: restarting pdfrender on scb1004
10:18 volans: restarting pdfrender on scb1002
10:18 _joe_: restarting pdfrender on scb1001
10:02 moritzm: installing ppp security updates on trusty
09:37 godog: keep 2x not 3x copies of older (>15d) logstash elasticsearch indices
09:29 moritzm: installing curl security updates
09:29 godog: temporarily set elasticsearch logstash watermark to low:0.85 and high:0.9
06:34 bawolff@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/OpenStackManager/special/SpecialNovaSudoer.php: T203885 (duration: 00m 54s)
05:30 bawolff: deployed patch T208881
01:18 mutante: scb1004 - systemctl restart pdfrender (T174916)
00:43 jforrester@deploy1001: Synchronized php-1.33.0-wmf.3/includes/resourceloader/ResourceLoader.php: ResourceLoader: Fail less hard when JSON serialization of config fails I673f59d93 (duration: 00m 53s)
00:33 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T205368 Enable BotPasswords on Governance wiki (duration: 00m 55s)
00:32 ejegg: updated fundraising CiviCRM from 769dcf6456 to 07183ed7cc
00:26 James_F: Created the bot_passwords table for Governance wiki T205368
00:21 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T208449 Disable wgWelcomeSurveyEnabled everywhere in production (duration: 00m 54s)
00:18 jforrester@deploy1001: Synchronized wmf-config/extension-list: T208081 Drop the Petition extension from extension-list (duration: 00m 53s)
00:16 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T208081 Drop the Petition extension from InitialiseSettings (duration: 00m 52s)
00:14 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: T208081 Drop the Petition extension from CommonSettings (duration: 00m 53s)
00:12 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T208899 Enabling wgMediaInTargetLanguage for testwiki (duration: 00m 54s)
00:00 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T208081 Disable the Petition extension in production (duration: 00m 52s)

2018-11-07

23:48 catrope@deploy1001: Synchronized wmf-config/CommonSettings.php: Make GrowthExperiments flag operative in CommonSettings (duration: 00m 53s)
23:44 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add flag for GrowthExperiments to InitialiseSettings (duration: 00m 53s)
23:37 catrope@deploy1001: Finished scap: Full scap to rebuild i18n for the addition of the GrowthExperiments extension (duration: 39m 40s)
23:21 jiji: Disabled nagios checks on rdb1006 and rdb2005 due to rdb1005 reimaging - T206450
22:57 catrope@deploy1001: Started scap: Full scap to rebuild i18n for the addition of the GrowthExperiments extension
22:13 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: Revert "labswiki rollback to 1.33.0-wmf.2"
22:07 thcipriani@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/LdapAuthentication/LdapAuthenticationPlugin.php: Expose methods used by OpenStackManager T208995 (duration: 00m 54s)
22:06 XenoRyet: updated payments-wiki from 34506ce636 to 983ce3af0f
22:02 thcipriani@deploy1001: Synchronized wmf-config/CommonSettings.php: Allow Cloud VPS 172.16.0.0/16 for $wmgAllowLabsAnonEdits wikis T208986 (duration: 00m 54s)
22:02 arlolra: Updated Parsoid to 970751a (T206940)
21:54 arlolra@deploy1001: Finished deploy [parsoid/deploy@4edc771]: Updating Parsoid to 970751a (duration: 09m 34s)
21:45 arlolra@deploy1001: Started deploy [parsoid/deploy@4edc771]: Updating Parsoid to 970751a
21:21 ladsgroup@deploy1001: Finished deploy [ores/deploy@25dfa4f]: T191842 T197096 (duration: 17m 24s)
21:18 krinkle@deploy1001: Synchronized php-1.33.0-wmf.2/extensions/AbuseFilter/includes/AbuseFilter.php: T208144 - I0fdda5 (duration: 00m 53s)
21:16 krinkle@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/VipsScaler: Id9f82afd (duration: 00m 55s)
21:06 krinkle@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/AbuseFilter/includes/AbuseFilter.php: T208144 - I0fdda51010243 (duration: 00m 53s)
21:06 banyek: stopping replication on db2072 (T208954)
21:04 krinkle@deploy1001: Synchronized php-1.33.0-wmf.3/includes/jobqueue/jobs/RefreshLinksJob.php: T208147 -I7f5fafe9439d8a7b4 (duration: 00m 54s)
21:03 ladsgroup@deploy1001: Started deploy [ores/deploy@25dfa4f]: T191842 T197096
20:55 banyek: depool labsdb1010 (T189158)
20:24 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: rollback labswiki to 1.33.0-wmf.2
20:12 thcipriani@deploy1001: Synchronized php: group1 wikis to 1.33.0-wmf.3 (duration: 00m 53s)
20:11 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.3
20:00 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T206173 Adding namespaces to Governance wiki (duration: 00m 55s)
19:50 chasemp: labstore1007:~# mkdir /srv/security/
19:48 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Re-sync for skipped apaches due to maintenance (whoops) (duration: 00m 55s)
19:48 XioNoX: Revert "Redirect eqsin/ulsfo caches to eqiad" - T208272
19:47 XioNoX: repool codfw - T208272
19:45 XioNoX: asw-c-codfw maintenance finished successfuly - T208272
18:51 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T201285: Disable wgRawHTML on Governance wiki (duration: 05m 12s)
18:31 onimisionipe: restarting relforge-eqiad and relforge-eqiad-small-alpha clusters on relforge100[1-2]
18:21 XioNoX: power down asw-c4-codfw - T208272
17:31 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add sourceswiki to wikidata clients (duration: 00m 53s)
17:25 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Inject wikidata rc records on wikidata itself (duration: 00m 53s)
17:21 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable WMEUnderstandingFirstDay on testwiki (duration: 00m 53s)
16:37 XioNoX: remove asw-c-codfw FPC8 from config - T208272
16:35 XioNoX: shutdown asw-c-codfw FPC8 - T208272
16:33 jforrester@deploy1001: Synchronized php-1.33.0-wmf.3/resources/src/mediawiki.rcfilters/styles/mw.rcfilters.ui.less: T208898 Hot-deploy to wmf.3 (duration: 00m 53s)
16:22 jforrester@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/Echo/modules/styles: Hot-deploy T208930 to wmf.3 (duration: 00m 54s)
16:20 XioNoX: Enable all VC ports (except uplinks) on spines - T208272
15:58 XioNoX: Redirect eqsin/ulsfo caches to eqiad - T208272
15:57 XioNoX: depool codfw for row C maintenance - T208272
15:36 moritzm: installing Java security updates on relforge*
15:29 jiji@deploy1001: Synchronized wmf-config/ProductionServices.php: Remove jobqueue_redis references, T198220 (duration: 00m 54s)
15:21 akosiaris: T206439 direct 30% of the apertium.svc.eqiad.wmnet traffic to scb1001. Will increase tomorrow to 50%
15:20 akosiaris@puppetmaster1001: conftool action : set/weight=15; selector: dc=eqiad,service=apertium,cluster=scb,name=scb1001.*
15:16 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,service=apertium,cluster=scb,name=scb1001.*
15:15 akosiaris: T206439 pool upgraded scb1001 to apertium.svc.eqiad.wmnet as a form of canary
15:13 moritzm: uploaded nginx 1.13.6-2+wmf2 to apt.wikimedia.org/stretch-wikimedia
14:55 akosiaris@puppetmaster1001: conftool action : set/pooled=no; selector: dc=eqiad,service=apertium,cluster=scb,name=scb1001.*
14:37 oblivian@deploy1001: Synchronized docroot/wwwportal/w/search-redirect.php: Fixing redirects if no language is specified (duration: 00m 54s)
14:33 moritzm: uploaded nginx 1.13.6-2+wmf2~jessie1 to apt.wikimedia.org/jessie-wikimedia
14:32 akosiaris: T206439 upload apertium-cat_2.6.0-1+wmf1 apertium-fra-cat_1.5.0-1+wmf1 apertium-fra_1.5.0-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
14:26 bblack: rebooting graphite1004
14:16 akosiaris: T206439 upload apertium-separable_0.3.2-1+wmf1 apertium-lex-tools_0.2.1-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
14:07 akosiaris: T206439 upload apertium_3.5.2-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
13:43 bblack: hi
13:21 Amir1: ladsgroup@mwmaint1002:/srv/mediawiki/php-1.33.0-wmf.1$ mwscript sql.php --wiki=sourceswiki extensions/Wikibase/client/sql/entity_usage.sql (T208858)
12:38 zeljkof: EU SWAT finished
12:36 zfilipin@deploy1001: Synchronized wmf-config: SWAT: BC: Enable Schema.org page split test (T208763) (duration: 00m 54s)
12:35 akosiaris: T206439 upload hfst-ospell_0.5.0-1+wmf1to apt.wikimedia.org/jessie-wikimedia/main
12:27 akosiaris: T206439 upload cg3_1.1.7-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
12:16 zfilipin@deploy1001: Synchronized wmf-config/InterwikiSortOrders.php: SWAT: Add dty, gor, inh, kbp and lfn to InterwikiSortOrders (T208217) (duration: 00m 53s)
12:12 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: New throttle rule for Art&Feminism event in Chile (T208866) (duration: 00m 54s)
12:12 akosiaris: T206439 upload hfst_3.15.0-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
12:08 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: Remove expired throttle rules (duration: 01m 05s)
11:49 akosiaris: T206439 upload lttoolbox_3.5.0-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
11:49 akosiaris: T206439 upload lttoolbox_3.5.0-1+wmf1
10:06 hashar: CI: switched operations/puppet job to be based on Stretch ( T208422 ) and to add python3 ( T208873 )
09:25 _joe_: run systemctl reset-failed on ms-be1029, had a failed debmonitor session
08:16 kartik@deploy1001: Finished deploy [cxserver/deploy@6f97d25]: Update cxserver to f9ffd24 (duration: 04m 59s)
08:11 kartik@deploy1001: Started deploy [cxserver/deploy@6f97d25]: Update cxserver to f9ffd24
02:55 ejegg: disabled recurring charge jobs
00:46 mutante: tegmen - shutting down for renaming and reinstall (T208824)
00:11 dereckson@deploy1001: Synchronized wmf-config/interwiki.php: Updating interwiki cache (duration: 02m 44s)

2018-11-06

23:55 mutante: cp1084 - network went down, powercycled, probably T203194
22:49 ejegg: updated fundraising CiviCRM from e0742d2210 to 769dcf6456
21:50 mutante: icinga1001-"MediaWiki EtcdConfig up-to-date" checks were all UNKNOWN because systemd unit update-etcd-mw-config-lastindex was present but service not running. it was turned off in https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/427328/ on purpose. manually ran "systemctl start update-etcd-mw-config-lastindex" and the checks all work (T202782)
21:49 mutante: icinga1001 - the "MediaWiki EtcdConfig up-to-date" checks were all unknown on the new icinga server, this was because systemd unit update-etcd-mw-config-lastindex was present but service not running. that was turned off in https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/427328/ on purpose. manually ran "systemctl start update-etcd-mw-config-lastindex" to start it and the checks
21:40 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: Group0 wikis to 1.33.0-wmf.3
20:19 thcipriani@deploy1001: Finished scap: testwiki to 1.33.0-wmf.3 and rebuild l10n cache (duration: 34m 06s)
19:45 thcipriani@deploy1001: Started scap: testwiki to 1.33.0-wmf.3 and rebuild l10n cache
19:44 thcipriani@deploy1001: Pruned MediaWiki: 1.32.0-wmf.23 (duration: 07m 19s)
18:36 thcipriani: cutting branch for MediaWiki and extensions version 1.33.0-wmf.3
17:37 XioNoX: add vlan-analytics1-a-eqiad interface-range on asw2-a-eqiad
16:47 XioNoX: enable cr4-ulsfo zayo transport to cr1-codfw
16:23 akosiaris@deploy1001: scap-helm zotero install --name alextest --set main_app.version=20181019165254-production --set monitoring.enable=true charts/zotero [namespace: zotero, clusters: staging]
16:23 akosiaris@deploy1001: scap-helm zotero install --set main_app.version=20181019165254-production --set monitoring.enable=true charts/zotero [namespace: zotero, clusters: staging]
16:12 mutante: einsteinium - temp disabling icinga notifications and puppet, reloading icinga (for extra caution while deploying global NRPE change)
16:07 mutante: planet1001 - disabling puppet, editing NRPE config, testing allowed_hosts change
16:07 akosiaris@deploy1001: scap-helm -h finished
16:07 akosiaris@deploy1001: scap-helm -h cluster staging completed
16:07 akosiaris@deploy1001: scap-helm -h [namespace: -h, clusters: staging]
15:59 banyek: updating facts for the puppet compilers
15:37 akosiaris: create zotero namespace in eqiad, codfw, staging cluster T201611
15:20 godog: switch all graphite read traffic to graphite1004
15:16 XioNoX: push `lldp port-id-subtype interface-name` to all compatible switches - T208630
15:14 jiji: scb1001/scb1002 switched nutcracker redis from rdb1001:6382 to rdb1009:6379
15:08 XioNoX: push `lldp port-id-subtype interface-name` to all routers - T208630
14:40 jynus_: reducing consistenct temp. on db2048 to avoid lagging
14:30 moritzm: installing Ruby 2.1 security updates
14:22 godog: add graphite1004 to graphite cluster for reads
14:21 moritzm: installing clamav security updates on mendelevium/ticket.wikimedia.org
13:25 moritzm: restart HHVM on canaries to pick up new curl
13:10 XioNoX: zeroize asw-b-eqiad (decom) - T208788
12:33 moritzm: installing curl security updates
12:17 moritzm: installing chromium security updates on proton* (new upstream release tested in deployment-prep)
12:12 zeljkof: EU SWAT finished
12:10 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Update several Wikidata-related configs (duration: 00m 55s)
11:37 moritzm: installing Ruby 2.3 security updates
11:37 moritzm: installing Ruby 2.3 security updates on trusty
11:30 moritzm: installing Mono security updates
11:23 moritzm: installing Ruby 1.9 security updates on trusty
11:07 banyek: stopping replication on db2077 (T208672)
07:25 _joe_: also restarting on the other eqiad nodes
07:25 _joe_: restarting tilerator on maps1002
04:47 kartik@deploy1001: Finished deploy [cxserver/deploy@ddb0031]: Update cxserver to 17f9a10 (T144467, T198699, T208386) (duration: 05m 26s)
04:42 kartik@deploy1001: Started deploy [cxserver/deploy@ddb0031]: Update cxserver to 17f9a10 (T144467, T198699, T208386)
03:40 eileen: civicrm revision changed from 99895316de to e0742d2210, config revision is e832b5a04a
00:26 maxsem@deploy1001: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/471875/ (duration: 00m 51s)
00:07 maxsem@deploy1001: Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/471874/ (duration: 00m 54s)
00:03 eileen: civicrm revision changed from 042eeaeca9 to 99895316de, config revision is e832b5a04a

2018-11-05

22:41 mutante: sodium - reboot after disk replacement (T202705)
21:51 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@a92fce5]: Increase cirrusSearchLinksUpdate concurrency to 100 (duration: 01m 02s)
21:50 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@a92fce5]: Increase cirrusSearchLinksUpdate concurrency to 100
21:49 arlolra: Updated Parsoid to 8ed698b (T205334, T208360)
21:42 mobrovac@deploy1001: Started restart [zotero/translation-server@50f216a]: Free up some memory
21:41 arlolra@deploy1001: Finished deploy [parsoid/deploy@96d739b]: Updating Parsoid to 8ed698b (duration: 10m 59s)
21:38 thcipriani@deploy1001: Synchronized wmf-config/CommonSettings.php: Removed decomissioned citoid url T133001 (duration: 00m 53s)
21:30 arlolra@deploy1001: Started deploy [parsoid/deploy@96d739b]: Updating Parsoid to 8ed698b
21:23 ppchelko@deploy1001: Finished deploy [restbase/deploy@5b8ad3c]: Update deps, removed sections table, T207904 T206048 T207324 take 2 (duration: 09m 18s)
21:14 ppchelko@deploy1001: Started deploy [restbase/deploy@5b8ad3c]: Update deps, removed sections table, T207904 T206048 T207324 take 2
21:10 ppchelko@deploy1001: Finished deploy [restbase/deploy@5b8ad3c]: Update deps, removed sections table, T207904 T206048 T207324 (duration: 12m 15s)
21:09 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: Ensure all wikis on 1.33.0-wmf.2
20:58 ppchelko@deploy1001: Started deploy [restbase/deploy@5b8ad3c]: Update deps, removed sections table, T207904 T206048 T207324
20:38 ppchelko@deploy1001: Finished deploy [restbase/deploy@5b8ad3c] (dev-cluster): Update deps, removed sections table (duration: 03m 40s)
20:35 ppchelko@deploy1001: Started deploy [restbase/deploy@5b8ad3c] (dev-cluster): Update deps, removed sections table
19:53 akosiaris: do a depool, scap deploy, scap wikiversions-compile, hhvm restart and then a pool in eqiad mediawiki servers
{{safesubst:SAL entry|1=19:50 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:470868|Disable page issues A/B test (duration: 00m 53s)}}
19:44 sbisson@deploy1001: Synchronized php-1.33.0-wmf.2/includes/block/BlockRestriction.php: SWAT: BlockRestriction::update() unnecessarily does a SELECT on the page table. (duration: 01m 00s)
19:19 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable error logging for WikimediaEvents (duration: 00m 52s)
19:12 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable GrowthExperiments logging channel (duration: 00m 53s)
18:57 _joe_: restarting hhvm on mwdebug1002
18:44 godog: pool graphite1004 for reads - T196484
18:43 XioNoX: delete asw2-b - asw-b interface - T183585
18:41 XioNoX: remove asw-b-eqiad from LibreNMS - T183585
18:37 XioNoX: remove vrrp priority 70 on cr2-eqiad:ae2 to failback VIPs to cr2 - T183585
18:26 XioNoX: re-enable ae2 on cr2-eqiad - T183585
18:21 thcipriani: rollback mwdebug1001 group2 wikis
18:13 thcipriani: testing php-1.33.0-wmf.2 on group2 wikis on mwdebug1001
18:05 XioNoX: disable ae2 on cr2-eqiad - T183585
18:02 XioNoX: set vrrp priority 70 on cr2-eqiad:ae2 to failover VIP to cr1 - T183585
16:49 XioNoX: Update LLDP config on cr3-ulsfo - T208630
16:48 vgutierrez: uploaded certcentral 0.5 to apt.wikimedia.org (stretch) - T208572 T208378
16:06 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting MCR to read-new on all wikis (T198308) (duration: 00m 55s)
13:57 jynus_: increase consistency of db2050, dbstore2002 s3 after them catching up replication T208462
12:33 ladsgroup@deploy1001: Finished deploy [ores/deploy@096ffb3]: T208577 T181632 T208608 (duration: 22m 58s)
12:23 zeljkof: EU SWAT finished
12:23 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Increase wikidata dispatchers to 3 (duration: 00m 54s)
12:16 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set wgForeignUploadTargets to [] for zhwiki (T208397) (duration: 00m 54s)
12:10 ladsgroup@deploy1001: Started deploy [ores/deploy@096ffb3]: T208577 T181632 T208608
12:05 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: Revert "Anniversary logo for cswiki" (T207589) (duration: 00m 58s)
10:02 godog: reformat xfs filesystems on ms-be1040 - T199198
09:17 elukey@deploy1001: Finished deploy [analytics/refinery@9d39efa]: fixing stat1004 (duration: 00m 04s)
09:17 elukey@deploy1001: Started deploy [analytics/refinery@9d39efa]: fixing stat1004
09:08 joal@deploy1001: Finished deploy [analytics/refinery@9d39efa]: regular analytics weekly deploy (duration: 05m 21s)
09:02 joal@deploy1001: Started deploy [analytics/refinery@9d39efa]: regular analytics weekly deploy

2018-11-04

23:42 jynus_: deleting the same row on all s8 broken servers
23:39 jynus_: deleting one row on db1104
20:38 krinkle@deploy1001: Synchronized php-1.33.0-wmf.2/extensions/FlaggedRevs/frontend/specialpages/reports/ProblemChanges_body.php: T176232 - Ia43626584e (duration: 01m 17s)
18:32 jynus_: reduce temp. consistency level of s4, s5, and s6 codfw masters to prevent excessive lagging due to ongoing mediawiki core maintenance
08:42 eileen: process-control config revision is e832b5a04a renable running job list (all jobs on again now0
08:38 eileen: process-control config revision is e16b2c1c61 renable jobs
02:00 eileen: I think I got the rest of the jobs off process-control config revision is 4422254128
01:52 eileen: process-control config revision is 6ec67b3d01 - also turn off omnirecipient repair job
01:40 eileen: process-control config revision is 5b72cfe874 - reapply turn off q jobs

2018-11-03

09:35 elukey: run tcpdump on mc1035 to grab memcache traffic (rotating pcaps, ~30G maximum)

2018-11-02

17:04 thcipriani: rollback group2 wikis to 1.33.0-wmf.1 on mwdebug100{1,2}
16:54 thcipriani: deploying 1.33.0-wmf.2 to group2 wikis on mwdebug1002
16:43 _joe_: live-hacking removal of time limit on mwdebug1001
16:32 thcipriani: deploying 1.33.0-wmf.2 to group2 wikis on mwdebug1001
15:12 jynus: restarting replication @ db2074 after db2094:s3 table fix T208565
15:00 jynus: stopping replication on db2074 to fix db2094:s3 T208565
14:01 vgutierrez: reimaging eeden.wikimedia.org as jessie test system - T208583
11:43 jynus: ignoring cawikimedia.archive replication on db2094:s3 until a reimport happens T208565
11:29 jijiki: Rebooting mw2244 (spare system) for maintenance
10:52 ema: restart varnish-be on cp3032 T208574
08:19 jynus: performing alter table on dbstore2002 s3 and reducing consistency to improve recovery time T208462 T204006
08:01 jynus: reducing consitency on db2050 to improve recovery time T208462
07:59 jynus: performing alter table on db2050 T208462 T204006
07:38 godog: reformat ms-be1043 xfs filesystems - T199198
07:38 jynus: reducing consistency temporarily (flush, binlog sync) at db2040 to prevent lagging
07:26 jynus: reducing consistency temporarily (flush, binlog sync) at db2035 to prevent lagging

2018-11-01

23:01 shdubsh: restart hhvm on mw1261
22:29 ejegg: restarted fundraising queue consumer jobs
22:21 ejegg: updated fundraising CiviCRM from 65130ef3dd to 042eeaeca9
22:18 ejegg: turned off fundraising queue jobs for civi update
22:12 _joe_: rolling restart of hhvm on appservers and api in eqiad
22:09 shdubsh: cumin -b 2 -s 30 "O:mediawiki::appserver and *.eqiad.wmnet" "restart-hhvm"
22:05 _joe_: restarting hhvm on mw1238,1240
22:02 _joe_: restart hhvm on mw1244
21:52 shdubsh: restart hhvm on mw1247
21:49 _joe_: depooling mw1238 for debugging
21:09 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: group2 back to 1.33.0-wmf.1
20:55 hoo: Restarted hhvm on mwdebug2002
19:40 hoo: Ran "UPDATE wb_changes_dispatch SET chd_seen = '775203911' WHERE chd_site LIKE '%wikt%' AND chd_seen < '775180000';" on wikidata master (dispatching for wiktionaries)
19:00 hoo@deploy1001: Synchronized php-1.33.0-wmf.1/includes/export/WikiExporter.php: Fix for missing end tag </page> on some exports (T207974) (duration: 01m 01s)
18:38 hoo@deploy1001: Synchronized php-1.33.0-wmf.2/includes/export/WikiExporter.php: Fix for missing end tag </page> on some exports (T207974) (duration: 00m 55s)
18:25 jijiki: Enabling puppet on mw servers (T206923)
18:19 hoo@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Remove now redundant Wikidata config for wiktionary (T208317) (duration: 00m 54s)
18:12 hoo@deploy1001: Synchronized dblists/wikidataclient.dblist: Add all wiktionaries to wikidataclient.dblist, sort list (T208317) (duration: 00m 57s)
18:02 gehel: restart nginx on relforge100*
17:57 jijiki: Disabling puppet on mw servers (T206923)
16:07 anomie@mwmaint1002: Running migrateComments.php on section 4 wikis for T166733
13:46 anomie@mwmaint1002: Running migrateComments.php on remaining section 3 wikis for T166733
13:37 anomie@mwmaint1002: Running migrateComments.php on section 7 wikis for T166733
13:37 anomie@mwmaint1002: Running migrateComments.php on wikitech for T166733
13:37 anomie@mwmaint1002: Running migrateImageCommentTemp.php on wikitech for T188132
13:37 anomie@mwmaint1002: Running migrateComments.php on section 6 wikis for T166733
13:37 anomie@mwmaint1002: Running migrateComments.php on section 8 wikis for T166733
13:37 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 8 wikis for T188132
13:37 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 7 wikis for T188132
13:37 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 6 wikis for T188132
13:36 anomie@mwmaint1002: Running migrateComments.php on section 5 wikis for T166733
13:36 anomie@mwmaint1002: Running migrateComments.php on section 1 wikis for T166733
13:36 anomie@mwmaint1002: Running migrateComments.php on section 2 wikis for T166733
13:36 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 5 wikis for T188132
13:36 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 4 wikis for T188132
13:36 anomie@mwmaint1002: Running migrateImageCommentTemp.php on remaining section 3 wikis for T188132
13:36 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 2 wikis for T188132
13:35 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 1 wikis for T188132
12:50 addshore@deploy1001: Synchronized wmf-config/CommonSettings.php: List wikidataclient-test in CS.php dblists T208488 (duration: 00m 57s)
09:10 elukey: added a tmux session on mw1314m mw1344, mw1316 that checks mcrouter stats every 10s
00:58 onimisionipe: repooling wdqs1004. It has caught up on lag with others
00:22 tgr@deploy1001: Synchronized php-1.33.0-wmf.2/extensions/SyntaxHighlight_GeSHi/extension.json: SWAT: Follow-up I3daca6fb: Fix exception thrown when inserting new code block (invalidate RL cache) (duration: 00m 53s)
00:20 tgr@deploy1001: Synchronized php-1.33.0-wmf.2/extensions/SyntaxHighlight_GeSHi/modules/ve-syntaxhighlight/ve.ui.MWSyntaxHighlightWindow.js: SWAT: Follow-up I3daca6fb: Fix exception thrown when inserting new code block (duration: 00m 54s)
00:13 mobrovac@deploy1001: Finished deploy [restbase/deploy@e8f3a85] (dev-cluster): Add title normalisation and remove Accept-Language header duplicates (duration: 03m 00s)
00:10 mobrovac@deploy1001: Started deploy [restbase/deploy@e8f3a85] (dev-cluster): Add title normalisation and remove Accept-Language header duplicates
00:07 tgr@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: Move auth logging to different channels for easier counting (T150300, T123243) (duration: 00m 53s)
00:05 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Move auth logging to different channels for easier counting (T150300, T123243) (duration: 00m 53s)

2018-10-31

23:53 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Undeploy RelatedSites from pt and ro Wikivoyages (T202761) (duration: 00m 54s)
23:50 tgr@deploy1001: Synchronized dblists/wikidataclient.dblist: SWAT: Add trwiktionary to wikidataclient.dblist (T204419) (duration: 00m 53s)
23:44 tgr@deploy1001: Synchronized static/images/wmf-hor-googpub.png: SWAT: Update: add Wikimedia logo for SEO (T198946, T207790) (duration: 00m 53s)
23:36 tgr@deploy1001: Synchronized php-1.33.0-wmf.2/skins/MinervaNeue/skin.json: SWAT: MinervaNeue: Article width should not be full screen (T193061) (invalidate RL cache) (duration: 00m 53s)
23:34 tgr@deploy1001: Synchronized php-1.33.0-wmf.2/skins/MinervaNeue/resources/skins.minerva.tablet.styles/common.less: SWAT: MinervaNeue: Article width should not be full screen (T193061) (duration: 00m 53s)
23:25 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable the print schema (T208454) (duration: 00m 55s)
23:07 tstarling@deploy1001: Synchronized php-1.33.0-wmf.1: security patch (duration: 03m 45s)
22:58 tstarling@deploy1001: sync-dir aborted: security patch (duration: 00m 13s)
22:56 tstarling@deploy1001: Synchronized php-1.33.0-wmf.2: security patch (duration: 06m 47s)
22:36 catrope@deploy1001: Synchronized php-1.33.0-wmf.2/extensions/MobileFrontend/includes/specials/SpecialMobileContributions.php: Fix PHP warnings in User.php (T208469) (duration: 00m 53s)
22:19 catrope@deploy1001: Synchronized php-1.33.0-wmf.2/includes/user/User.php: Revert debugging code for T208469 (duration: 00m 52s)
22:17 catrope@deploy1001: Synchronized php-1.33.0-wmf.2/includes/user/User.php: Temporary logging to debug T208469 (duration: 00m 53s)
22:03 catrope@deploy1001: Synchronized php-1.33.0-wmf.2/includes/user/User.php: Temporary logging to debug T208469 (duration: 00m 53s)
21:47 catrope@deploy1001: Synchronized php-1.33.0-wmf.2/includes/user/User.php: Temporary logging to debug T208469 (duration: 00m 53s)
21:43 catrope@deploy1001: Synchronized php-1.33.0-wmf.2/includes/user/User.php: Temporary logging to debug T208469 (duration: 00m 57s)
20:51 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T203821 Fix config typo to point at testwikidatawiki properly (duration: 00m 53s)
20:34 thcipriani@deploy1001: Synchronized php: group1 wikis to 1.33.0-wmf.2 (duration: 00m 53s)
20:32 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.2
20:28 ladsgroup@deploy1001: Finished deploy [ores/deploy@70ba14b]: Upgrade to celery4 and flask 0.12.4, logstash fixes: T181546 T181630 T168921 T205256 T169586 T208258 T178441 (duration: 21m 29s)
20:06 ladsgroup@deploy1001: Started deploy [ores/deploy@70ba14b]: Upgrade to celery4 and flask 0.12.4, logstash fixes: T181546 T181630 T168921 T205256 T169586 T208258 T178441
19:38 jijiki: Enabling puppet on mw hosts
19:37 thcipriani@deploy1001: Synchronized php: rollback group1 wikis to 1.33.0-wmf.2 (duration: 00m 52s)
19:36 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: rollback group1 wikis to 1.33.0-wmf.2
19:20 thcipriani@deploy1001: Synchronized php: group1 wikis to 1.33.0-wmf.2 (duration: 00m 53s)
19:19 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.2
19:11 jijiki: Disabling puppet on mw hosts for T206923
19:01 chasemp: install logster on mwlog1001
18:59 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T203821 Enable Partial Blocks on testwiki and testwikidata in case we have to stop the train (duration: 00m 56s)
18:13 bblack: disabling puppet on labstore100[45], ahead of interface::rps change merges
18:08 bblack: disabling puppet on all LVS balancers, ahead of interface::rps change merges
17:41 bblack: reboot cp1085, network interface transmit hang
17:08 onimisionipe: depooling wdqs1004 to catch up on lag
17:04 jforrester@deploy1001: Synchronized php-1.33.0-wmf.2/includes/Block.php: Hot-deploy train blocker T208398 with 3a1f95d4d (duration: 00m 56s)
16:56 onimisionipe: repooling wdqs1005 - It has caught up with others
14:43 cmjohnson1: movd wtp1034 eth0 to new switch...it was left over
14:24 anomie@mwmaint1002: Running migrateComments.php on group0 for T166733
14:22 anomie@mwmaint1002: Running migrateImageCommentTemp.php on group0 for T188132
14:18 anomie: Testing dologmsg on mwmaint1002
14:17 hasharLunch: Adding a Stretch based CI job for operations/puppet (non voting job for now) | T208422
14:11 herron: re-enabling ircecho on einsteinium
13:39 herron: temporarily stopping ircecho on einsteinium
12:35 onimisionipe: depooling wdqs1005 to catch up with others
11:36 dereckson@deploy1001: Synchronized docroot/noc/createTxtFileSymlinks.sh: UNIX-agnostic shebang for createTxtFileSymlinks (Gerrit:470585, no-op in prod) (duration: 00m 54s)
11:08 ladsgroup@deploy1001: Synchronized wmf-config/Wikibase.php: SWAT: Do not load WikibaseQuality (T205064) (duration: 01m 05s)
10:57 hashar: contint1001: upgraded java and restarted Jenkins
10:56 hashar: restarting CI jenkins on contint1001
10:23 volans: restarted pdfrender on scb1003
09:24 elukey: upgraded memkeys to 20181031-1 on all the mc* - T208376
09:16 elukey: upload memkeys 20181031-1 to jessie-wikimedia thirdparty
08:39 godog: start rolling out rsyslog 8.38 to stretch hosts

2018-10-30

23:23 thcipriani@deploy1001: Synchronized wmf-config/throttle.php: SWAT: Throttle lift for Wikidata event at University of Edinburgh T208236 (duration: 00m 54s)
22:53 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: [Beta Cluster] UploadWizard: Enable Structured Data captions when WBMI is enabled (duration: 00m 53s)
22:38 jforrester@deploy1001: Synchronized php-1.33.0-wmf.2/extensions/VisualEditor/: Hot-deploy UBN train blocker VisualEditor bug T208366 (duration: 00m 56s)
20:53 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.33.0-wmf.2
20:45 thcipriani@deploy1001: Synchronized php-1.33.0-wmf.2/extensions/VisualEditor/lib/ve/src/ui/actions/ve.ui.WindowAction.js: ve.ui.WindowAction: Fix exception when opening windows T208347 (duration: 00m 54s)
20:22 twentyafterfour: hotfixing T208254 (restarting apache2 on phab1001)
19:40 thcipriani@deploy1001: Finished scap: testwiki to 1.33.0-wmf.2 and rebuild l10n cache (duration: 19m 45s)
19:20 thcipriani@deploy1001: Started scap: testwiki to 1.33.0-wmf.2 and rebuild l10n cache
18:59 thcipriani@deploy1001: Pruned MediaWiki: 1.32.0-wmf.22 (duration: 03m 10s)
18:52 ebernhardson: restarted mjolnir-kafka-bulk-daemon on all elastic hosts
18:51 thcipriani@deploy1001: Pruned MediaWiki: 1.32.0-wmf.20 (duration: 07m 55s)
18:46 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@14dd09e]: adjust kafka bulk daemon timeouts (duration: 03m 42s)
18:42 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@14dd09e]: adjust kafka bulk daemon timeouts
18:42 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@14dd09e]: adjust kafka bulk daemon timeouts (duration: 00m 48s)
18:41 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@14dd09e]: adjust kafka bulk daemon timeouts
18:34 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@14dd09e]: (no justification provided) (duration: 00m 26s)
18:34 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@14dd09e]: (no justification provided)
18:14 andrew@deploy1001: Finished deploy [horizon/deploy@ce0b9b4]: Rolling out fix for T208099 (duration: 03m 19s)
18:10 andrew@deploy1001: Started deploy [horizon/deploy@ce0b9b4]: Rolling out fix for T208099
17:23 thcipriani: starting branch cut for MediaWiki and extensions 1.33.0-wmf.2
16:55 mforns: Finished deployment of AQS using scap
16:48 mforns@deploy1001: Finished deploy [analytics/aqs/deploy@3a1d937]: (no justification provided) (duration: 02m 00s)
16:46 mforns@deploy1001: Started deploy [analytics/aqs/deploy@3a1d937]: (no justification provided)
16:45 mforns: Starting deployment of AQS using scap
16:31 godog: install carbon-c-relay 3.2-1 on graphite1004
16:27 shdubsh: updated graphite-in cname to graphite1004 - T196484
16:11 hashar: contint1001: rm -fR /srv/zuul-debug-logs # old logs from May 2018
16:07 ema: reboot cp-ats hosts for L1TF kernel/microcode updates T203011
15:17 anomie: Running migrateComments.php on test wikis and mediawikiwiki for T166733
15:16 anomie: Running migrateImageCommentTemp.php on test wikis and mediawikiwiki for T188132
14:56 godog: gradually upgrade rsyslog to 8.38 on jessie hosts - T206633
14:12 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add entry to wmgWikibaseClientEntityNamespaces for wiktionaries T208293 (duration: 00m 47s)
13:56 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add item & property to wmgWikibaseClientRepoNamespaces for wiktionaries T208293 (duration: 00m 48s)
13:54 godog: reenable puppet in eqiad
13:41 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Wikibase IS.php use 120 for property namespace (duration: 00m 47s)
13:31 ema: prometheus-trafficserver-exporter 0.2.0-1 uploaded to stretch-wikimedia T204232
13:27 godog: temporarily disable puppet in eqiad to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/470382
12:17 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: Wikibase.php, check for 2 more wmg vars before using them (duration: 00m 47s)
12:13 elukey: start memkeys on mc1035 to periodically dump the status of the most used keys - memkeys will use a bit of resources, please stop it if needed (root tmux) - T203786
12:10 jynus: removing s3 replication filters on dbstore1002 T184805
12:06 jynus: removing s3 replication filters on labsdb1009/10/11 T184805
12:02 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: commonswiki, Wikibase, clientDbList as empty array (duration: 00m 47s)
12:01 jynus: finishing deleting moved to s5, s3 wikis T184805
11:59 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Changing the language of votewiki to Persian (fa) (T207560) (duration: 00m 48s)
11:52 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: Wikibase.php, wrap $wmgWikibaseClientPropertyOrderUrl use in condition (duration: 00m 46s)
11:41 addshore@deploy1001: Synchronized wmf-config: 2x beta only patches, 66b8d5b 8fd1614 (duration: 00m 53s)
10:40 moritzm: installing ghostscript security updates
10:07 moritzm: installing exiv2 security updates
09:52 moritzm: installing mysql-5.5 security updates on trusty/jessie (only clients as packaged in Debian/Ubuntu)
09:40 moritzm: installing libmspack security updates
09:24 moritzm: installing paramiko security updates
08:24 jynus: starting to delete moved to s5, s3 wikis T184805
07:01 Krinkle: graphite2001: Remove Graphite data from corrupted names under media_* and ve_* (T189530)
06:58 Krinkle: graphite1004: Remove Graphite data from corrupted names under media_* and ve_* (T189530)
06:57 Krinkle: graphite1001: Remove Graphite data from corrupted names under media_* and ve_* (T189530)

2018-10-29

21:44 mutante: contint2001 - installing libtiff upgrade, with the same command from apt/history.log that had shown as failed
20:45 arlolra: Updated Parsoid to b9fa661 (T100841, T186965, T167349, T198618, T206040, T207956)
20:35 arlolra@deploy1001: Finished deploy [parsoid/deploy@e36608c]: Updating Parsoid to b9fa661 (duration: 12m 27s)
20:23 arlolra@deploy1001: Started deploy [parsoid/deploy@e36608c]: Updating Parsoid to b9fa661
20:01 XioNoX: rollback redirect ns1 to authdns1001
19:56 moritzm: rebooting authdns2001 for kernel security update
19:48 XioNoX: redirect ns1 to authdns1001 - try 3
19:47 XioNoX: replace radon IPs with authdns1001 on cr1/2-eqiad
19:40 XioNoX: redirect ns1 to authdns1001 - try 2
19:31 XioNoX: redirect ns1 to authdns1001
19:27 XioNoX: rollback redirect ns0 to authdns2001
19:23 moritzm: rebooting authdns1001 for kernel security update
19:12 XioNoX: redirect ns0 to authdns2001
19:03 twentyafterfour: restart apache on phab1001 to hotfix T208254
18:53 moritzm: installing tiff security updates for jessie
18:33 thcipriani: deploy1001:scap pull (ref: T208196)
18:31 thcipriani: deploy1001:sudo -u l10nupdate scap cdb-refresh-json --directory /srv/mediawiki-staging/php-1.33.0-wmf.1/cache/l10n (ref: T208196)
18:24 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable hewikivoyage HD logo (T208148) (duration: 00m 47s)
18:23 catrope@deploy1001: Synchronized static/images: Update logo for hewikivoyage, add HD logos (T208148) (duration: 00m 48s)
18:22 catrope@deploy1001: sync-file aborted: Update logo for hewikivoyage, add HD logos (duration: 00m 07s)
18:22 herron: moving mail_smarthost (and wikimail_smarthost) to hiera (gerrit 469524)
18:20 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@7eeede7]: Improved time handling for Kafka, GUI Update and caching removal from updater (duration: 10m 42s)
18:16 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable PageTriage/Copyvio on testwiki and enwiki (duration: 00m 47s)
18:13 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable Wikidata data access on trwiktionary (T204419) (duration: 00m 48s)
18:09 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@7eeede7]: Improved time handling for Kafka, GUI Update and caching removal from updater
16:49 ottomata: reassigning eventlogging_ReadingDepth partition 0 from 1002,1004,1006 to 1003,1001,1005 to move preferred leadership from 1002 to 1003
16:42 addshore@deploy1001: Synchronized wmf-config: phpdoc comments only (duration: 00m 48s)
16:37 addshore@deploy1001: Synchronized wmf-config: Wikibase.php, finish the grand cleanup (duration: 00m 48s)
16:36 ebernhardson: create wikimaniawiki_general indices for eqiad and codfw elasticsearch clusters
16:35 addshore@deploy1001: Synchronized wmf-config: Wikibase, move client list config to nice part of Wikibase.php (duration: 00m 47s)
16:32 addshore@deploy1001: Synchronized wmf-config: Wikibase, move namespace config to IS.php PT 2/2 (duration: 00m 47s)
16:31 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Wikibase, move namespace config to IS.php PT 1/2 (duration: 00m 47s)
16:20 chasemp: remove emailbot from acl*sre-team in phab, afaik this is unused now and unmaintained
15:56 addshore@deploy1001: Synchronized wmf-config: BETA ONLY (duration: 00m 48s)
15:48 godog: power off ms-be2021 for controller alarms troubleshooting - T208096
15:44 vgutierrez: uploaded certcentral 0.4 to apt.wikimedia.org (stretch) - T207927
15:43 addshore@deploy1001: Synchronized wmf-config: Create and use wikidataclient-test.dblist PT 2/2 (duration: 00m 48s)
15:42 addshore@deploy1001: Synchronized dblists: Create and use wikidataclient-test.dblist PT 1/2 (duration: 00m 48s)
15:19 cmjohnson1: cloudvirt1019 going down to re-seat the backplane cables
15:17 addshore@deploy1001: Synchronized wmf-config: Remove Wikibase-* config files (duration: 00m 47s)
15:14 addshore@deploy1001: Synchronized wmf-config: Stop loading Wikibase-* files (duration: 00m 47s)
15:06 addshore@deploy1001: Synchronized wmf-config: PT2/2 Totally empty Wikibase-* files (duration: 00m 47s)
15:04 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: PT1/2 Totally empty Wikibase-* files (duration: 00m 46s)
14:58 banyek: shutting down db1117 for hardware maintenance (T208150)
14:52 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: Wikibase.php, check $wmgWBRepoSettingsSparqlEndpoint is set (duration: 00m 46s)
14:37 addshore@deploy1001: Synchronized wmf-config: PT 2/2 Wikibase, move repo definitions to IS.php gerrit:470373 (duration: 00m 47s)
14:35 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: PT 1/2 Wikibase, move repo definitions to IS.php gerrit:470373 (duration: 00m 48s)
14:17 gehel: repooling wdqs1003, catched up on lag
14:16 hashar: Restarting Jenkins
13:25 addshore@deploy1001: Synchronized wmf-config: Wikibase, move 2 usage tracking configs to Wikibase.php PT 2/2 (duration: 00m 47s)
13:24 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: Wikibase, move 2 usage tracking configs to Wikibase.php PT 1/2 (duration: 00m 46s)
13:22 addshore@deploy1001: Synchronized wmf-config: Wikibase, move more misc settings to IS.php PT 2/2 (duration: 00m 47s)
13:21 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Wikibase, move more misc settings to IS.php PT 1/2 (duration: 00m 49s)
13:19 addshore@deploy1001: Synchronized wmf-config: Remove unused wmgArticlePlaceholderSearchEngineIndexed (duration: 00m 48s)
13:16 addshore@deploy1001: Synchronized wmf-config: Wikibase, set wgArticlePlaceholderSearchEngineIndexed in IS.php PT 2/2 (duration: 00m 47s)
13:15 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Wikibase, set wgArticlePlaceholderSearchEngineIndexed in IS.php PT 1/2 (duration: 00m 47s)
13:13 addshore@deploy1001: Synchronized wmf-config: Wikibase, Move quality contraints settings to IS.php PT 2/2 (duration: 00m 47s)
13:12 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Wikibase, Move quality contraints settings to IS.php PT 1/2 (duration: 00m 47s)
13:10 addshore@deploy1001: Synchronized wmf-config: Wikibase, cleanup some duplicated settings PT 2/2 (duration: 00m 47s)
13:09 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: Wikibase, cleanup some duplicated settings PT 1/2 (duration: 00m 46s)
13:06 addshore@deploy1001: Synchronized wmf-config: Wikibase, Move wgArticlePlaceholderImageProperty to IS.php PT 2/2 (duration: 00m 46s)
13:05 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Wikibase, Move wgArticlePlaceholderImageProperty to IS.php PT 1/2 (duration: 00m 46s)
13:03 addshore@deploy1001: Synchronized wmf-config: Wikibase, move some property lists to IS php files PT 2/2 (duration: 00m 47s)
13:02 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Wikibase, move some property lists to IS php files PT 1/2 (duration: 00m 47s)
13:01 addshore@deploy1001: sync-file aborted: Wikibase, move some property lists to IS php files PT 1/2 (duration: 00m 04s)
13:00 addshore@deploy1001: Synchronized wmf-config: BETA FILES only (duration: 00m 47s)
12:57 addshore@deploy1001: Synchronized wmf-config: Wikibase, Move badge related config to IS.php PT 2/2 (duration: 00m 47s)
12:55 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Wikibase, Move badge related config to IS.php PT 1/2 (duration: 00m 45s)
12:52 addshore@deploy1001: Synchronized wmf-config: PT2/2 Wikibase, Move property suggester settings to IS files (duration: 00m 47s)
12:51 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: PT1/2 Wikibase, Move property suggester settings to IS files (duration: 00m 47s)
12:47 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: Wikibase, Move $wgPropertySuggesterMinProbability to IS.php PT 2/2 (duration: 00m 47s)
12:46 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Wikibase, Move $wgPropertySuggesterMinProbability to IS.php PT 1/2 (duration: 00m 47s)
12:43 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: PT2/2 gerrit:470348 Wikibase, move dispatching settings to IS.php (duration: 00m 47s)
12:42 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: PT1/2 gerrit:470348 Wikibase, move dispatching settings to IS.php (duration: 00m 48s)
11:44 moritzm: installing graphicsmagick update for stretch
11:23 zeljkof: EU SWAT finished
11:19 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove global action related permissions (T208035) (duration: 00m 48s)
10:22 gehel: switch wdqs1003 and wdqs1006 completed, wdqs1003 still depooled to catch up on update lag - T207947
10:21 hashar: Restore ti l10n files on deploy1001:/srv/mediawiki-staging/php-1.33.0-wmf.1/cache/l10n/upstream # T208196
10:15 gilles@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T187299 Double sampling of performance perception survey on ruwiki (duration: 00m 47s)
10:12 hashar: Moving l10n_cache-ti.cdb files on deploy1001: sudo -u l10nupdate mv l10n_cache-ti.cdb.json l10n_cache-ti.cdb.json-back-T208196 && sudo -u l10nupdate mv l10n_cache-ti.cdb.MD5 l10n_cache-ti.cdb.MD5-back-T208196 # T208196
10:10 addshore@deploy1001: Synchronized wmf-config: final sync (duration: 00m 47s)
10:09 addshore@deploy1001: Synchronized wmf-config/Wikibase-production.php: PT3/3 gerrit:470161 Wikibase, Create and use wmgWikibaseRepoStatementSections (duration: 00m 47s)
10:08 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: PT2/3 gerrit:470161 Wikibase, Create and use wmgWikibaseRepoStatementSections (duration: 00m 47s)
10:07 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: PT1/3 gerrit:470161 Wikibase, Create and use wmgWikibaseRepoStatementSections (duration: 00m 50s)
10:04 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: gerrit:470160 Wikibase.php, move a bunch of config into clean area NOOP (duration: 00m 47s)
10:03 gehel: starting to switch wdqs1003 and wdqs1006 - T207947
10:02 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: gerrit:470159 PT2/2 Wikibase, add IS.php setting for each possible extension (duration: 00m 47s)
10:01 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: gerrit:470159 PT1/2 Wikibase, add IS.php setting for each possible extension (duration: 00m 47s)
09:57 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: gerrit:470158 Wikibase, Remove unused wmgUseWikibaseQualityExternalValidation (duration: 00m 47s)
09:55 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: PT2/2 gerrit:470156 Wikibase, remove unused wmgWikibaseClientSettings (duration: 00m 47s)
09:54 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: PT1/2 gerrit:470156 Wikibase, remove unused wmgWikibaseClientSettings (duration: 00m 47s)
09:51 addshore@deploy1001: Synchronized wmf-config/Wikibase-production.php: pt3/3 gerrit:470155 Wikibase, create and use wmgWikibaseClientInjectRecentChanges (duration: 00m 46s)
09:50 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: pt2/3 gerrit:470155 Wikibase, create and use wmgWikibaseClientInjectRecentChanges (duration: 00m 47s)
09:49 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: pt1/3 gerrit:470155 Wikibase, create and use wmgWikibaseClientInjectRecentChanges (duration: 00m 47s)
09:45 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: gerrit:470154 PT2/2 Wikibase, put all wgNamespaceAliases in IS.php (duration: 00m 46s)
09:44 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: gerrit:470154 PT1/2 Wikibase, put all wgNamespaceAliases in IS.php (duration: 00m 47s)
09:43 addshore@deploy1001: sync-file aborted: gerrit:470154 PT1/2 (duration: 00m 00s)
09:41 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: gerrit:470153 PT2/2 Wikibase, define $wgExtraNamespaces in IS.php (duration: 00m 47s)
09:40 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: gerrit:470153 PT1/2 Wikibase, define $wgExtraNamespaces in IS.php (duration: 00m 46s)
09:37 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: gerrit:470152 Wikibase, kill $wmgWBSharedSettings (duration: 00m 47s)
09:35 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: PT2/2 gerrit:470151 Wikibase, move wmgWBSiteLinkGroups to IS.php (duration: 00m 46s)
09:34 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: PT1/2 gerrit:470151 Wikibase, move wmgWBSiteLinkGroups to IS.php (duration: 00m 47s)
09:30 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: PT2/2 gerrit:470150 Wikibase, Split specialSiteLinkGroups and manage from IS.php (duration: 00m 46s)
09:30 ema: resume cache hosts rolling reboots for kernel/microcode updates T203011
09:29 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: PT1/2 gerrit:470150 Wikibase, Split specialSiteLinkGroups and manage from IS.php (duration: 00m 47s)
09:19 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: PT2/2 gerrit:470339 Introduce wmgWikibaseMaxSerializedEntitySize (duration: 00m 46s)
09:17 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: PT1/2 gerrit:470339 Introduce wmgWikibaseMaxSerializedEntitySize (duration: 00m 47s)
09:07 addshore@deploy1001: Finished scap: sync with no changes (duration: 14m 39s)
08:56 elukey: restart yarn on an-master100[1,2] to pick up new zookeeper timeout settings (10s -> 20s) - T206943
08:52 addshore@deploy1001: Started scap: sync with no changes
08:50 addshore@deploy1001: sync aborted: (no justification provided) (duration: 00m 03s)
08:50 addshore@deploy1001: Started scap: (no justification provided)
08:49 addshore@deploy1001: sync-l10n aborted: (no justification provided) (duration: 01m 19s)
08:42 gilles@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T208088 Enable performance perception survey shuffling (duration: 00m 47s)
08:38 gilles@deploy1001: Synchronized php-1.33.0-wmf.1/extensions/QuickSurveys: T208088 Add ability to shuffle answers display order (duration: 01m 51s)
08:35 gilles@deploy1001: sync aborted: T208088 Enable performance QuickSurvey shuffling (duration: 00m 00s)
08:35 gilles@deploy1001: Started scap: T208088 Enable performance QuickSurvey shuffling
08:35 gilles@deploy1001: sync aborted: T208088 Enable performance QuickSurvey shuffling (duration: 06m 30s)
08:29 gilles@deploy1001: Started scap: T208088 Enable performance QuickSurvey shuffling
08:07 godog: reformat ms-be1042 xfs filesystems - T199198
08:00 gilles: Deploying time-sensitive backport to QuickSurveys
02:08 onimisionipe: repooling wdqs1003. It has caught up with others

2018-10-28

23:36 krinkle@deploy1001: Synchronized php-1.33.0-wmf.1/extensions/Graph: T184128 - I02da92de33 (duration: 00m 58s)
19:17 onimisionipe: depooling wdqs1003 to catch up on lag
17:30 elukey: restart yarn resource manager on an-master1002 to force failover to an-master1001 - T206943
16:36 onimisionipe: repooling wdqs1003 - it didn't really catch up with others, but lag time on others are beginning to up.
13:38 onimisionipe: depooling wdqs1003 again to catch up with others
02:16 onimisionipe: repooling wdqs1003 - it has caught up with others
00:03 krinkle@deploy1001: Synchronized php-1.33.0-wmf.1/extensions/CirrusSearch: T206967 - Ia23d19cf1e6 (duration: 01m 02s)

2018-10-27

22:22 krinkle@deploy1001: Synchronized php-1.33.0-wmf.1/resources/src: T208093- I25012a2c6f (duration: 00m 58s)
21:24 banyek: resetting power on db1117 as the host is DOWN and the serial console shows nothing
20:56 onimisionipe: depooling wdqs1003 to catch up with others
16:18 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: Wikibase, fix duplicate specialSiteLinkGroups key T208124 (duration: 00m 54s)
15:57 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: Wikibase, make sure specialSiteLinkGroups has wikidata group (duration: 00m 54s)
12:32 Amir1: Deployed patch for T207576
12:29 banyek: resuming replication on s1@dbstore2002 as table compression is finished (T204930)
09:17 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: BETA ONLY (4x patches) (duration: 00m 55s)
09:09 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: Remove wgArticlePlaceholderSearchIntegrationBackend BETA override (duration: 01m 00s)
08:34 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: Wikibase, Set siteLinkGroups settings on all wikis again T208048 T208077 T208074 (duration: 00m 54s)
08:24 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: BETA ONLY T208043 (duration: 01m 06s)
03:41 SMalyshev: depool wdqs1003 again to let it catch up some more
02:35 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@7eeede7]: Re-deploy Updater to deal with performance issues (duration: 00m 38s)
02:34 smalyshev@deploy1001: Started deploy [wdqs/wdqs@7eeede7]: Re-deploy Updater to deal with performance issues
02:34 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@e9392f4]: Re-deploy Updater to deal with performance issues (duration: 00m 05s)
02:33 smalyshev@deploy1001: Started deploy [wdqs/wdqs@e9392f4]: Re-deploy Updater to deal with performance issues
00:00 mutante: icinga1001 - using wmf-auto-reimage to reinstall gets stuck at initial puppet run after reboot - Still waiting for Puppet after 105.0 minutes - aborting on cumin, loggin in directly and manually running puppet (T202782 T208100)

2018-10-26

22:54 mutante: sodium - attempted to replace broken disk for RAID - did not go well
21:38 ejegg: updated fundraising CiviCRM from 97506677e8 to 65130ef3dd
21:34 aaron@deploy1001: Synchronized php-1.33.0-wmf.1/autoload.php: 86c0b56 (duration: 00m 52s)
21:33 aaron@deploy1001: Synchronized php-1.33.0-wmf.1/tests: 86c0b56 (duration: 01m 08s)
20:03 mutante: icinga1001 - disabled puppet, changed: check_result_reaper_frequency=2 ; max_check_result_reaper_time=10 to test if it lowers latency (T208066)
19:40 chasemp: remove 2fa for charlottepotero and cwd users in phab (so they can readd)
19:09 SMalyshev: repooled wdqs1003 - looks like it caught up now
17:18 SMalyshev: depool wdqs1003 again to let it catch up some more
16:10 ejegg: updated payments-wiki to 34506ce636
15:32 elukey: rolling restart of all prometheus-mcrouter-exporters on app/api servers - metrics not reported after the last mcrouter restart
15:20 gehel: repooling wdqs1003, other nodes are starting to lag as well
14:56 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: Prevent sysops from disabling 2FA for other users as part of upcoming feature (duration: 00m 53s)
13:02 gehel: depool wdqs1003 to catch up on updates
07:51 bawolff: adjust patch T207916
07:16 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T207916 13b993ab9f - auth log on in arwiki (duration: 00m 54s)
06:51 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T207916 13b993ab9f - auth log on in group1 (duration: 00m 54s)
06:33 moritzm: uploaded openjdk-8 backport for recent Java 8 security updates to apt.wikimedia.org/jessie-wikimedia
06:24 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: ecf579e9f9 - T207916 - enable auth log group0 (duration: 00m 55s)
06:01 bawolff: adjust patch for T207916
05:07 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: d3b2c346 T207916 (duration: 00m 55s)
04:58 SMalyshev: depooled wdqs1003 again, let's see if it helps it catch up now
04:12 bawolff: deploy patch T207916
03:19 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: bc9b863e - T207900 - enable CSP report only for users w/session everywhere (duration: 00m 55s)
03:10 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 745d0b61 - T207900 - enable CSP report only for users w/session enwiki (duration: 00m 55s)
03:01 bawolff@deploy1001: Synchronized wmf-config/CommonSettings.php: T207900 - Add wikimedia.org (no subdomain) to allow list for math (duration: 00m 53s)
02:54 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 745d0b61 - T207900 - enable CSP report only for users w/session enwiki (duration: 00m 53s)
02:43 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: a8aa9d6aae - T207900 - enable CSP report only for users w/session fawiki, frwiki, svwiki, eswiki, ruwiki, zhwiki, dewiki (duration: 00m 56s)
02:22 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: d743db261 - T207900 - enable CSP report only for users w/session arwiki (duration: 00m 54s)
02:05 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: bd55034d122 - T207900 - enable CSP report only for users w/session on medium wikis (duration: 00m 55s)
01:41 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T207900 (b74911f6201) enable csp users with session all group1 wikis (duration: 00m 55s)
01:28 bawolff@deploy1001: Synchronized wmf-config/CommonSettings.php: Ia518c031 (duration: 00m 55s)
01:26 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@e9392f4]: Re-deploy Updater to deal with performance issues (duration: 31m 28s)
01:21 bawolff@deploy1001: Synchronized wmf-config/CommonSettings.php: T207900 - deploy CSP to people with session on enwikiquote (duration: 00m 54s)
01:19 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T207900 - deploy CSP to people with session on enwikiquote (duration: 00m 55s)
00:56 twentyafterfour: twentyafterfour@deploy1001 rebuilt and synchronized wikiversions files: group2 wikis to 1.33.0-wmf.1 refs T206655
00:55 smalyshev@deploy1001: Started deploy [wdqs/wdqs@e9392f4]: Re-deploy Updater to deal with performance issues
00:39 jforrester@deploy1001: Synchronized wmf-config/Wikibase.php: Post-SWAT: De-register all entities on WBMI installations calling themselves Commons I09e066f2 (duration: 00m 56s)
00:32 ejegg: updated payments-wiki from f5999d963d to 57e8438e9c
00:18 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: Define and specify lexeme NS for wikidatawiki (duration: 00m 55s)

2018-10-25

23:59 twentyafterfour@deploy1001: Synchronized php-1.33.0-wmf.1/includes/parser/Parser.php: deploy https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/469799/ refs T208000 (duration: 00m 56s)
23:57 twentyafterfour: deploying https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/469799/
23:53 jforrester@deploy1001: Synchronized php-1.32.0-wmf.26/extensions/CentralNotice/special/SpecialCentralNotice.php: SWAT Sync versions of SpecialCentralNotice to avoid dirty repo checkout T208004 (duration: 00m 56s)
23:39 maxsem@deploy1001: Synchronized php-1.33.0-wmf.1/extensions/CentralNotice/: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/CentralNotice/+/469794/ (duration: 00m 57s)
23:26 maxsem@deploy1001: Synchronized php-1.33.0-wmf.1/extensions/GlobalPreferences/: https://gerrit.wikimedia.org/r/c/469793/ (duration: 00m 58s)
21:35 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: rolling back group1 refs T206655 T208000
21:29 XioNoX: configure 208.80.153.185/29 on cr1/2-codfw - T207663
21:25 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group2 wikis to 1.33.0-wmf.1 refs T206655
20:48 twentyafterfour: staying at group1, error rate seems to have stabilized
20:43 ejegg: updated fundraising python tools from 5a2d39b41b to af5dbee8eb
20:37 ejegg: updated standalone SmashPig deployment from b638ca02bc to f65daa8550
20:32 twentyafterfour: db error rate increased again. rolling back
20:31 twentyafterfour@deploy1001: Synchronized php: group1 wikis to 1.33.0-wmf.1 refs T206655 (duration: 00m 54s)
20:30 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.1 refs T206655
20:07 jgleeson_: Updated paymentswiki from 5a7a8e7e4f to f5999d963d
20:04 twentyafterfour: still haven't deployed wmf.1 yet error rate increased and icinga is alerting about mediawiki exceptions + wdqs1010 degraded
20:02 twentyafterfour@deploy1001: Finished scap: full sync to be sure that 1.33.0-wmf.1 is fully deployed (duration: 36m 57s)
19:54 mutante: mw1272 - repooled (T207983)
19:51 mutante: mw1272 - rebooting (a stop job is running for HHVM PH/Hack runtime) (T207983)
19:47 mutante: mw1272 - depooled, restarting hhvm (T207983)
19:45 mutante: mw1272 - depooled
19:26 twentyafterfour@deploy1001: Started scap: full sync to be sure that 1.33.0-wmf.1 is fully deployed
19:23 twentyafterfour: beginning mediawiki train. Will start with group1 and then monitor the situation for a few minutes. If everything looks good then we go to group2.
19:20 sbisson@deploy1001: Synchronized php-1.32.0-wmf.26/extensions/ContentTranslation/: SWAT: Add detailed logging for AbuseFilter (duration: 00m 56s)
18:37 sbisson@deploy1001: Synchronized php-1.33.0-wmf.1/extensions/ContentTranslation/: SWAT: Remove the session parameter from AbuseFilter logging (duration: 00m 56s)
17:58 aaron@deploy1001: Synchronized php-1.33.0-wmf.1/extensions/Translate/tag: c5fa239 (duration: 00m 55s)
17:52 aaron@deploy1001: Synchronized php-1.33.0-wmf.1/includes/page/WikiPage.php: f3b5a1d (duration: 00m 54s)
17:52 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@4967dba]: Test deploy new update & scripts (duration: 00m 28s)
17:51 smalyshev@deploy1001: Started deploy [wdqs/wdqs@4967dba]: Test deploy new update & scripts
17:50 aaron@deploy1001: Synchronized php-1.33.0-wmf.1/tests/phpunit/includes/page/WikiPageDbTestBase.php: f3b5a1d (duration: 00m 55s)
17:36 aaron@deploy1001: Synchronized php-1.33.0-wmf.1/includes/changetags/ChangeTags.php: 08f8e6a (duration: 00m 55s)
17:24 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@95452cf]: Update mobileapps to 58cbdff (T206527) (duration: 03m 50s)
17:20 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@95452cf]: Update mobileapps to 58cbdff (T206527)
17:20 mutante: planet - regenerating feeds for 'en' and 'de', others will follow by cron. switching to new theme. replaced bootstrap with bulma. removed jQuery. thanks to paladox
16:34 gehel@puppetmaster1001: conftool action : set/weight=20; selector: dc=eqiad,cluster=wdqs,name=wdqs1005.codfw.wmnet
16:34 gehel@puppetmaster1001: conftool action : set/weight=20; selector: dc=eqiad,cluster=wdqs,name=wdqs1004.codfw.wmnet
16:34 gehel: decreasing relative weight of wdqs1003 in LVS to ease the updater
16:24 shdubsh: installed patched nagios-nrpe-plugin and nagios-nrpe-server on icinga1001 - T207775
15:36 elukey: shutdown aqs1006 to replace one broken disk - T206915
15:31 SMalyshev: depooling wdqs1003 again, it's not catching up like the other hosts
15:16 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: [Beta Cluster] Re-enable WBMI on Beta Commons (duration: 00m 54s)
15:11 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Revert "logging: Disable Wikibase.NewItemIdFormatter channel" (duration: 00m 55s)
15:08 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Explicitly set wgLexemeEnableRepo for wikidatas gerrit:469625 (duration: 00m 55s)
15:02 godog: test rsyslog 8.38 upgrade on lithium - T136312
14:28 elukey: upgrade druid on druid100[4-6] to Druid 0.12.3
14:20 banyek: running dns update (gerrit patch: 467711)
13:48 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting comment table migration stage to write-new/read-both on all wikis (T166733) (duration: 00m 55s)
13:46 godog: reformat ms-be2043 xfs filesystems - T199198
13:29 XioNoX: test successful, rollback add term return-tcp permit on cr2-codfw
13:28 XioNoX: test add term return-tcp permit on cr2-codfw
12:14 volans: rebooting cumin1001 to pick new kernel and clear any potential weird state after OOMs
12:01 zeljkof: EU SWAT finished
11:17 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: New throttle rule for Johannesburg Event on 2018-10-27 (T207742) (duration: 00m 55s)
11:09 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Stop collecting data CitaitonUsage and CitationUsagePageLoad (T191086 T203253) (duration: 00m 57s)
10:57 volans: restart pdfrender on scb1003
10:11 elukey: upgrade druid100[1-3] to druid 0.12.3
09:51 gehel: resetting deployment directory on wdqs1003
09:15 elukey@deploy1001: Finished deploy [analytics/turnilo/deploy@84bf1ad]: Upgrade to 1.8.1 (duration: 00m 10s)
09:15 elukey@deploy1001: Started deploy [analytics/turnilo/deploy@84bf1ad]: Upgrade to 1.8.1
09:10 ema: resume cache hosts rolling reboots for kernel/microcode updates T203011
07:16 vgutierrez: Uploaded certcentral 0.3 to apt.wikimedia.org (stretch) - T207737 T207478
07:11 moritzm: installing requests security updates on trusty
06:17 SMalyshev: depooling wdqs1003 again, it's not catching up like the other hosts
06:06 elukey: upload druid 0.12.3-1 debs to stretch-wikimedia

2018-10-24

23:24 maxsem@deploy1001: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/469495/ (duration: 00m 54s)
23:15 maxsem@deploy1001: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/462040/ (duration: 00m 55s)
23:08 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Deploy csp report-only to small.dblist wikis T207900 (duration: 00m 56s)
22:38 bawolff@deploy1001: Synchronized wmf-config/CommonSettings.php: Deploy csp report-only to outreachwiki T207900 (duration: 00m 54s)
22:36 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Deploy csp report-only to outreachwiki T207900 (duration: 00m 54s)
22:33 bawolff@deploy1001: scap failed: average error rate on 8/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details)
22:27 eileen_: civicrm revision changed from 1c0a1b2406 to 97506677e8, config revision is c0a8be03a1
21:33 banyek: compressing tables in s1@dbstore2002 (T204930)
21:26 banyek: pausing replication on dbstore2002 (T204930)
19:38 twentyafterfour: The train is now blocked by database lock contention of unknown origin
19:31 twentyafterfour: the errors were all coming from wmf.26 but the error rate skyrocketed after deploying 1.33.0-wmf.1 to group1 so there is some query in the new branch which is holding a lock. T207881
19:19 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.1 refs T206655
18:16 XioNoX: enable BGP sessions to transit/peering on cr2-eqord - T204170
17:20 gehel: repooling all elasticsearch servers in eqiad
17:12 cmjohnson1: rebooting cloudvirt1019
17:04 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: [Beta Cluster] Re-disable WBMI on Beta Commons for now T180981 (duration: 00m 54s)
17:03 jforrester@deploy1001: scap failed: average error rate on 4/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details)
16:36 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: [Beta Cluster] Re-disable WBMI on Beta Commons for now T180981 (duration: 00m 54s)
16:31 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: gerrit:469444 Wikibase.php, dont load wikidata repo settings on other repos (take 2) (duration: 00m 54s)
16:04 XioNoX: power-off cr1-eqord - T204170
16:00 twentyafterfour: 15:59:06 Synchronized php-1.33.0-wmf.1/extensions/EventBus/: revert "Set event datetime with microsecond resolution." on 1.33.0-wmf.1 refs T207817 (duration: 00m 56s)
15:59 XioNoX: disable BGP sessions to transit/peering on cr1-eqord - T204170
15:54 twentyafterfour: deploying https://gerrit.wikimedia.org/r/469451
14:23 herron: scheduled icinga downtime and disabling puppet on logstash hosts. deploying role::kafka::logging to logstash elasticserach data hosts
13:35 XioNoX: pre-configure switch ports for labvirt1007/8/9/12:eth1 in cloud-virt-instance-trunk range on asw2-b-eqiad
13:17 ema: begin cache hosts rolling reboots for kernel/microcode updates T203011
12:24 ema: cp-ats: upgrade trafficserver to 8.0.0-1wm1 T204232
12:12 ema: cp1072: upgrade trafficserver to 8.0.0-1wm1 T204232
11:22 ema: cp1071: upgrade trafficserver to 8.0.0-1wm1 T204232
10:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Restore db1092 and db1104 original weight (duration: 00m 52s)
10:35 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1092 and starting to restore db1104 original weight (duration: 00m 54s)
10:28 marostegui: Compare revision table on dewiki cebwiki shwiki srwiki mgwiktionary enwikivoyage on db1100 and db2075 - T184805
09:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1092 (duration: 00m 54s)
09:39 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1092 (duration: 00m 54s)
09:20 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1092 and db1087 (duration: 01m 05s)
08:55 marostegui: Stop MySQL for upgrade and reboot on db1087
08:47 marostegui: Update MySQL on db1092 for upgrade and reboot
08:03 godog: fix aggregation to 'sum' for MediaWiki.RevisionSlider - T205416
07:33 gehel: powercycling wdqs1010 - T207817
07:19 _joe_: powercycling wdqs1009
07:04 elukey: powercycle wdqs1008
06:59 elukey: powercycle wdqs1007
06:55 elukey: powercycle wdqs1006 (depool first)
06:46 elukey: powercycle wdqs1005
06:42 SMalyshev: repooled wdqs1003
06:35 _joe_: powercycling wdqs[2001-2002,2004-2006].codfw.wmnet, one at a time
06:33 elukey: powercycle wdqs1004
05:24 kartik@deploy1001: Finished deploy [cxserver/deploy@80dc518]: Update cxserver to 9ad60d9 (T207445) (duration: 04m 06s)
05:20 kartik@deploy1001: Started deploy [cxserver/deploy@80dc518]: Update cxserver to 9ad60d9 (T207445)
02:34 mutante: powercycled wdqs1009 - by request
02:24 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@d4692ea]: Reverting update on wdqs1003 to fix wdqs-updater issue (duration: 00m 03s)
02:24 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@d4692ea]: Reverting update on wdqs1003 to fix wdqs-updater issue
02:12 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@d4692ea]: Reverting update on wdqs1003 to fix wdqs-updater issue (duration: 00m 23s)
02:12 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@d4692ea]: Reverting update on wdqs1003 to fix wdqs-updater issue
01:56 tstarling@deploy1001: Synchronized php-1.33.0-wmf.1/includes/page/WikiPage.php: T207530 (duration: 00m 53s)
01:46 tstarling@deploy1001: Synchronized php-1.32.0-wmf.26/includes/page/WikiPage.php: fix deletion performance regression T207530 (duration: 00m 55s)
01:37 bawolff: deployed T207750
00:28 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group0 wikis to 1.33.0-wmf.1 refs T206655
00:26 twentyafterfour: finished with mediawiki train for group0 refs T206655
00:08 twentyafterfour@deploy1001: Finished scap: syncing 1.33.0-wmf.1 refs T206655 (duration: 36m 58s)

2018-10-23

23:31 twentyafterfour@deploy1001: Started scap: syncing 1.33.0-wmf.1 refs T206655
23:30 twentyafterfour@deploy1001: Synchronized php-1.32.0-wmf.26/includes/export/WikiExporter.php: sync https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/469319/ refs T207628 (duration: 01m 39s)
22:16 eileen: civicrm revision changed from bde28d4453 to 1c0a1b2406, config revision is c0a8be03a1
22:14 twentyafterfour: scap prep 1.33.0-wmf.1
21:47 mutante: icinga1001 - replacing check_ping with check_fping as the standard host check command, for faster host checks (another tip from Nagios Tuning guide, still manual testing) (T202782)
21:30 mutante: icinga1001 - changing check_result_reaper_frequecy from 10 to 3, trying to lower average check latency. "allow faster check result processing -> requires more CPU" (T202782)
19:31 twentyafterfour@deploy1001: Synchronized php-1.32.0-wmf.26/skins/MinervaNeue/resources/skins.minerva.scripts/pageIssuesLogger.js: sync https://gerrit.wikimedia.org/r/#/c/mediawiki/skins/MinervaNeue/+/469244/ refs T207423 (duration: 00m 48s)
19:27 twentyafterfour: deploying https://gerrit.wikimedia.org/r/#/c/mediawiki/skins/MinervaNeue/+/469244/
19:22 bawolff: deploy patch T207778
18:17 mutante: icinga - performance/latency comparison - https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=4 vs https://icinga-stretch.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=4 (T202782)
18:13 mutante: icinga1001 - manually set max_concurrent_checks to 0 (unlimited), restart icinga, keep puppet disabled, for testing (it ran into the limit of 10000 all the time, causing lots of logging, and the CPU power is actually slightly lower than on einsteinium (T202782) refs: Nagios Tuning, point 7 https://assets.nagios.com/downloads/nagioscore/docs/nagioscore/3/en/tuning.html
17:20 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: BETA: Set wmgWikibaseCachePrefix for commonswiki I0badd355723 (duration: 00m 46s)
17:18 ejegg: updated standalone SmashPig deploy from 2292111bda to b638ca02bc
17:15 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: For WBMI, intentionally rather than implicitly install Wikibase I38574e670 (duration: 00m 47s)
17:13 mutante: icinga1001 rm /var/log/user.log.1 - was 14G and using 25% of the / partition and server out of disk :/
17:06 ejegg: rolled SmashPig back to 2292111bda
17:03 ejegg: updated standalone SmashPig deployment from 2292111bda to 18da9727d8
16:20 volans: restarted pdfrender on scb1004
14:47 herron: added confluent-kafka-2.11 1.1.0-1 package to jessie-wikimedia/thirdparty T206454
14:34 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting comment table migration stage to write-new/read-both on group 1 (T166733) (duration: 00m 46s)
14:22 anomie@deploy1001: Synchronized php-1.32.0-wmf.26/includes/filerepo/file/LocalFile.php: Backport for T207419 (duration: 00m 47s)
14:02 gehel: repooling / banning elastics1031 - T207724
14:01 moritzm: installing spice security updates
14:00 ema: upload trafficserver 8.0.0-1wm1 to stretch-wikimedia/main T204232
13:49 gehel: depooling / banning elastics1031 - T207724
13:43 gehel: depooling / banning elastics1029 - T207724
13:35 gehel: rolling restart of blazegraph for change to blazegraph home dir
13:22 gehel: depooling / banning elastics1018 - T207724
12:29 gehel: depooling / banning elastics1028 and 1030 - T207724
11:23 zeljkof: EU SWAT finished
11:20 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: New throttle rule for Wikipedia in Ort (T207714) (duration: 00m 46s)
11:11 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable RCPatrol for srwikiquote (T207732) (duration: 00m 47s)
10:13 ema: upload libc++ 6.0.1 to stretch-wikimedia/main T204232
09:42 jynus: stopping db1087 to fix db1124
09:31 gehel: depooling / banning elastics1017 and 1022 - T207724
09:13 godog: roll-restart thumbor to send statsd traffic through statsd_exporter - T205870
08:08 godog: update hp firmware to 6.60 on ms-be2017 - T141756
07:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1100 - T184805 (duration: 00m 48s)
06:50 elukey: powercycle ms-be2017 (frozen since ~8hrs ago)
06:42 elukey: restart yarn and hdfs daemon on analytics1068 to pick up correct config (the host was down since before we swapped the Hadoop masters due to hw failure)
06:39 marostegui: Stop replication on db1092 and db1087 for checking T206743
06:02 marostegui: Deploy schema change on s3 - T207359
00:35 SMalyshev: temp depooled wdq1003 to let it catch up
00:17 Amir1: evening SWAT is done

2018-10-22

23:59 ladsgroup@deploy1001: Synchronized php-1.32.0-wmf.26/includes/changetags/ChangeTags.php: SWAT: Fix bad join on ChangeTag subquery (T207313) (duration: 00m 47s)
23:39 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@d4692ea]: Redeploy Updater for T207673 (duration: 10m 12s)
23:29 smalyshev@deploy1001: Started deploy [wdqs/wdqs@d4692ea]: Redeploy Updater for T207673
22:12 pmiazga@deploy1001: Synchronized wmf-config//InitialiseSettings-labs.php: SWAT: beta: Disable page issues A/B test on beta cluster only (T200792) (duration: 00m 46s)
21:44 mutante: adding new prod ServerAlias punjabi.wikimedia.org to Apache cluster (T207583)
21:13 ayounsi@deploy1001: Finished deploy [librenms/librenms@0fd8da6]: Revert LibreNMS upgrade - T207481 (duration: 00m 08s)
21:13 ayounsi@deploy1001: Started deploy [librenms/librenms@0fd8da6]: Revert LibreNMS upgrade - T207481
21:08 andrewbogott: rebooting cloudvirt1023
20:52 ayounsi@deploy1001: Finished deploy [librenms/librenms@737683a]: Upgreade LibreNMS to 1.44 - T207481 (duration: 00m 10s)
20:52 ayounsi@deploy1001: Started deploy [librenms/librenms@737683a]: Upgreade LibreNMS to 1.44 - T207481
20:29 ladsgroup@deploy1001: Finished deploy [ores/deploy@e89e880]: Use redis task tracker (T152012) (duration: 22m 02s)
20:06 ladsgroup@deploy1001: Started deploy [ores/deploy@e89e880]: Use redis task tracker (T152012)
18:54 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT Deploy TemplateWizard everywhere T202545, re-try (duration: 00m 45s)
18:50 jforrester@deploy1001: scap failed: average error rate on 4/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details)
18:48 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: [Beta] Temporarily disable WBMI from Beta Commons whilst Wikibse is fixed T180981 (duration: 00m 46s)
18:38 jforrester@deploy1001: Synchronized php-1.32.0-wmf.26/resources/src/mediawiki.rcfilters/styles/mw.rcfilters.ui.ChangesListWrapperWidget.highlightCircles.seenunseen.less: SWAT RCFIlters: Fix highlight circles for unseen changes T207472 (duration: 00m 46s)
18:36 jforrester@deploy1001: Synchronized php-1.32.0-wmf.26/skins/MinervaNeue/resources/skins.minerva.scripts/pageIssuesLogger.js: SWAT Fix reading depth logging part 2 T207423 (duration: 00m 46s)
18:35 jforrester@deploy1001: Synchronized php-1.32.0-wmf.26/extensions/WikimediaEvents/modules/all/ext.wikimediaEvents.readingDepth.js: SWAT Fix reading depth logging part 1 T207423 (duration: 00m 46s)
18:31 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT Add TemplateWizard to the BF allow list T205290 (duration: 00m 48s)
18:05 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: [Beta Cluster] Load but don't enable MediaInfo on Beta Commons cf. T180981 (duration: 00m 45s)
18:00 jforrester@deploy1001: Synchronized wmf-config/Wikibase.php: For WikibaseMediaInfo wikis, load basic Wikibase repo code cf. T180981 (duration: 00m 46s)
17:57 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Allow enablement of the WikibaseMediaInfo, still off everywhere cf. T180981 (duration: 00m 48s)
17:46 jforrester@deploy1001: Synchronized wmf-config/extension-list: Add WikibaseMediaInfo i18n to cache cf. T180981 (duration: 00m 46s)
17:40 mobrovac@deploy1001: Finished deploy [proton/deploy@b3e254a]: Update Puppeteer to v1.9.0 - T207416 (duration: 01m 34s)
17:38 mobrovac@deploy1001: Started deploy [proton/deploy@b3e254a]: Update Puppeteer to v1.9.0 - T207416
17:34 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@975a67b]: WDQS deployment - GUI update and binaries upgrade (duration: 11m 47s)
17:23 XioNoX: enable cr2:xe-4/0/0 (to asw-a) for optics replacement - T203719
17:22 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@975a67b]: WDQS deployment - GUI update and binaries upgrade
17:20 XioNoX: disable cr2:xe-4/0/0 (to asw-a) for optics replacement - T203719
17:19 elukey@deploy1001: Finished deploy [analytics/refinery@1de5f44]: Deploy new version of Camus and pageview whitelist (duration: 07m 05s)
17:16 cmjohnson1: analytics1068 down for mother board swap
17:12 elukey@deploy1001: Started deploy [analytics/refinery@1de5f44]: Deploy new version of Camus and pageview whitelist
17:11 XioNoX: re-enable puppet fleet-wide for puppetmaster1001 uplink move
17:10 XioNoX: moving puppetmaster1001 uplink to asw2-b
17:08 andrew@deploy1001: Finished deploy [horizon/deploy@431a55d]: Rolling out fix for T207510 (duration: 03m 40s)
17:07 XioNoX: disable puppet fleet-wide for puppetmaster1001 uplink move
17:05 andrew@deploy1001: Started deploy [horizon/deploy@431a55d]: Rolling out fix for T207510
16:24 arturo: T206261 2h icinga downtime cloudnet1003/4 for another patch
15:54 ejegg: updated payments-wiki from 06848600ed to 5a7a8e7e4f
15:51 ejegg: updated fundraising CiviCRM from 1f10dc8a18 to bde28d4453
15:35 XioNoX: push firewall changes to pfw3-eqiad - T207175
15:35 ejegg: updated standalone SmashPig deployment from 581c685326 to 2292111bda
15:03 mforns@deploy1001: Finished deploy [analytics/refinery@bbebc20]: deploying refinery together with refinery-source v0.0.79 (duration: 10m 16s)
14:52 mforns@deploy1001: Started deploy [analytics/refinery@bbebc20]: deploying refinery together with refinery-source v0.0.79
14:49 XioNoX: push firewall changes to pfw3-codfw - T207175
13:19 marostegui: Run myloader for enwikivoyage cebwiki shwiki srwiki mgwiktionary on db2052 (s5 codfw master) - T184805
13:12 kartik@deploy1001: Finished deploy [cxserver/deploy@5f53734]: Update cxserver to 7f996f3 (T207445) (duration: 03m 53s)
13:08 kartik@deploy1001: Started deploy [cxserver/deploy@5f53734]: Update cxserver to 7f996f3 (T207445)
11:51 zeljkof: eu swat finished
11:49 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable rollbacker right on srwikisource (T206935) (duration: 00m 46s)
11:37 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable autopatroller, patroller and rollbacker rights on srwikiquote (T206936) (duration: 00m 49s)
11:28 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable suppressredirect and markbotedit rights to rollbackers on it.wikiversity (T207300) (duration: 00m 46s)
11:21 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable cx2outreach campaign (T207031) (duration: 00m 47s)
11:09 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: Anniversary logo for cswiki (T207589) (duration: 00m 47s)
11:06 zfilipin@deploy1001: sync-file aborted: SWAT: Test if logo specified in wgLogo/wgLogoHD exists (T207053) (duration: 00m 02s)
10:03 arturo: icinga downtime for cloudnet1003/4 for T206261
09:16 marostegui: Remove replication filters from db2052 (s5 codfw master) - T184805
09:04 marostegui: Run mydumper on db1100 for enwikivoyage cebwiki shwiki srwiki mgwiktionary - T184805
08:58 marostegui: Stop replication in sync on db1100 and db2052 (codfw master) to reimport wikis - T184805
08:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1100 - T184805 (duration: 00m 47s)
08:29 moritzm: powercycling ms-be1018, stuck during reboot
08:28 jynus: performing deletes on db1087 to fix wb_terms on labs
08:27 marostegui: Deploy schema change on db2043 (s3 master) without replication - T204006
08:22 marostegui: Disconnect codfw -> eqiad replication on s5 (db1070)
08:19 marostegui: Disconnect codfw -> eqiad replication on s3 (db1075)
08:13 marostegui: Disconnect codfw -> eqiad replication on es3 (es1017)
08:11 marostegui: Disconnect codfw -> eqiad replication on es2 (es1015)
08:08 marostegui: Disconnect codfw -> eqiad replication on x1 (db1069)
08:05 marostegui: Disconnect codfw -> eqiad replication on s8 (db1071)
08:03 marostegui: Disconnect codfw -> eqiad replication on s7 (db1062)
08:01 marostegui: Disconnect codfw -> eqiad replication on s6 (db1061)
07:59 marostegui: Disconnect codfw -> eqiad replication on s4 (db1068)
07:57 marostegui: Disconnect codfw -> eqiad replication on s2 (db1066)
07:52 marostegui: Disconnect codfw -> eqiad replication on s1 (db1067)
07:38 moritzm: rebooting swift-be servers in eqiad for kernel security update
07:24 godog: reformat ms-be2042 - T199198
06:34 marostegui: Deploy schema change on db2036 - T204006
06:11 marostegui: Deploy schema change on db2050 - T204006
06:00 marostegui: Deploy schema change on db2057 - T204006
05:47 marostegui: Deploy schema change on s3 db2074 (and db2094 sanitarium) - T204006
05:31 marostegui: Deploy schema change on dbstore2002:3313 - T204006
05:29 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Clarify db2033 BBU status (duration: 00m 49s)
04:37 kartik@deploy1001: Finished deploy [cxserver/deploy@904151f]: Update cxserver to eee8974 (T207070, T203077, T199529) (duration: 05m 42s)
04:31 kartik@deploy1001: Started deploy [cxserver/deploy@904151f]: Update cxserver to eee8974 (T207070, T203077, T199529)

2018-10-21

22:15 onimisionipe: repooling wdqs1003 as it has caught up on lag
20:42 banyek: resuming replication on s4@dbstore2002 (T204930)
16:15 reedy@deploy1001: Synchronized wmf-config/interwiki.php: Updating interwiki cache (duration: 04m 52s)
15:57 bawolff: adjust patch for T194204
12:39 onimisionipe: depooling wdqs1003 to catchup on lag time

2018-10-20

23:05 reedy@deploy1001: Synchronized php-1.32.0-wmf.26/extensions/CentralAuth/: Update setEmail (duration: 00m 55s)
21:29 gehel: repooling wdqs1003 (still some lag, but 100[45] start to be impacted)
19:54 gehel: depooling wdqs1003 to catch up on lag
13:53 reedy@deploy1001: Synchronized php-1.32.0-wmf.26/includes/auth/AuthManager.php: (no justification provided) (duration: 00m 55s)
12:46 hoo@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add CentralAuth related permissions to stewards at metawiki (T207531) (duration: 01m 09s)
05:38 marostegui: Force writeback on db2033 - T184888

2018-10-19

20:33 twentyafterfour: deployed RCFilters: Fix completely broken highlight circles refs T207472
20:32 twentyafterfour@deploy1001: Synchronized php-1.32.0-wmf.26/resources/src/mediawiki.rcfilters/styles/: sync https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/468636/ (duration: 00m 54s)
20:31 twentyafterfour: deploying https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/468636/ to the full cluster.
20:28 twentyafterfour: deployed https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/468636/ to mwdebug1002
19:20 mutante: ns0 / ns1 - authdns-gen-zones -f /srv/authdns/git/templates /etc/gdnsd/zones && gdnsdctl reload-zones - to add new language shn (T206777)
19:16 mutante: ns2/multatuli - gnddctl reload-zones
19:12 mutante: labweb1001 / wikitech - disabling 2fa for myself, logging in , re-enabling it again
17:49 ejegg: updated fundraising CiviCRM from 83874e75ba to 1f10dc8a18
17:47 mutante: DNS - 'authdns-gen-zones -f /srv/authdns/git/templates /etc/gdnsd/zones && gdnsd checkconf && gdnsd reload-zones' - needed when adding new languages to langs.tmpl - adding "shn" (Shan language) T206777
16:36 XioNoX: deactivate BGP to 15426 in ams-ix (down and no reply to emails) - T207428
14:16 banyek: disconnecting s4 replication on dbstore2002 (T204930)
14:12 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Remove useless comments (duration: 00m 54s)
13:58 vgutierrez: Uploaded certcentral 0.2 to apt.wikimedia.org (stretch) - T207457
11:46 banyek: starting compression of s4 tables @dbstore2002 (T204930)
11:33 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T207313 UBN - Revert back wikidata for change_tag backend (duration: 00m 59s)
10:53 arturo: icinga downtime for 2h for clounet1003/1004 to deploy patch related to T206261
09:37 godog: bump /proc/sys/net/core/rmem_default temporarily to 6MB and bounce statsd-proxy statsite-instances on graphite1004 - T196484
08:53 banyek: adding wmf-pt-kill_2.2.20-1+wmf4 package for stretch (T206521)
08:28 jynus: stopping db1092 and db1087 in sync
07:50 godog: bump /proc/sys/net/core/rmem_default temporarily to 2MB and bounce statsd-proxy statsite-instances on graphite1004 - T196484
07:20 marostegui: Remove mwmaint1001 grants from m5 - https://phabricator.wikimedia.org/T201343 https://phabricator.wikimedia.org/T192457
07:15 godog: powercycle ms-be1021, [19601329.556259] sd 0:1:0:1: rejecting I/O to offline device
07:05 godog: bump /proc/sys/net/core/rmem_default temporarily to 1MB and bounce statsd-proxy statsite-instances on graphite1004 - T196484
06:13 marostegui: Deploy schema change on s7 codfw host by host without replication - T204006
05:58 marostegui: Deploy schema change on s2 codfw host by host without replication - T204006
05:25 marostegui: Deploy schema change on s1 codfw host by host without replication - T204006
01:49 krinkle@deploy1001: Synchronized php-1.32.0-wmf.26/extensions/WikimediaEvents/includes/WikimediaEventsHooks.php: Ic74a9d5601b8c (duration: 00m 55s)

2018-10-18

22:00 mutante: lvs1011,lvs1012 - manually editing nagios NRPE config and restarting service (to make monitoring from icinga1001 work and puppet is disabled)
21:52 mutante: eeden - manually editing nagios NRPE config and restarting service (to make monitoring from icinga1001 work and puppet is disabled)
21:49 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group2 wikis to 1.32.0-wmf.26 refs T191072
21:46 twentyafterfour@deploy1001: Synchronized php-1.32.0-wmf.26/includes/filerepo/file/LocalFile.php: sync Id97e1c refs T207419 (duration: 00m 53s)
21:29 twentyafterfour@deploy1001: Synchronized php-1.32.0-wmf.26/includes/filerepo/file/LocalFile.php: sync https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/468470/ refs T207419 (duration: 00m 54s)
20:49 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group2 wikis to 1.32.0-wmf.24 refs T191072
20:39 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.26
20:21 volans: start ferm on db2042, it failed to start at reboot due to DNS resolution timeout
19:22 ejegg: updated SmashPig standalone deploy from 5f21d3f2db to 581c685326
19:21 ejegg: updated payments-wiki from a3892e4ed3 to 06848600ed
19:17 shdubsh: rebooting graphite1004
19:11 shdubsh: upping ring buffer size on graphite1004 in an attempt to mitigate dropped packets at the interface -- T196484
19:02 sbisson@deploy1001: Synchronized php-1.32.0-wmf.26/extensions/PageTriage/: SWAT: Use Main Object Stash for keeping track of PageTriage last use (duration: 00m 54s)
18:19 awight: Restarting ORES services for T88997
17:33 ladsgroup@deploy1001: Finished deploy [ores/deploy@4ac4c8b]: Logstash support for ores: T181546 T169586 T168921 T181630 T205256 (duration: 23m 48s)
17:19 herron: aborted enabling kafka on logstash elasticsearch cluster due to puppet errors. reverted change T206454
17:09 ladsgroup@deploy1001: Started deploy [ores/deploy@4ac4c8b]: Logstash support for ores: T181546 T169586 T168921 T181630 T205256
17:00 twentyafterfour@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.26 refs T191072 (duration: 00m 53s)
16:59 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.26 refs T191072
16:57 herron: enabling kafka on logstash elasticsearch cluster T206454
16:55 twentyafterfour@deploy1001: Synchronized php-1.32.0-wmf.26/extensions/WikibaseQualityConstraints/src/ServiceWiring.php: sync https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/WikibaseQualityConstraints/+/468352/ refs T207394 (duration: 00m 54s)
16:52 mobrovac@deploy1001: Finished deploy [restbase/deploy@6c879fa]: Have 100% of traffic directed to Proton as well - T186748 (duration: 20m 52s)
16:31 mobrovac@deploy1001: Started deploy [restbase/deploy@6c879fa]: Have 100% of traffic directed to Proton as well - T186748
15:51 XioNoX: trunk cloud-instances2-b-eqiad between asw-b-eqiad and asw2-b-eqiad
15:50 cmjohnson1: disabling checks on cloudvirt1019 for maintenance
15:42 twentyafterfour: twentyafterfour@deploy1001 Synchronized php: group1 wikis to 1.32.0-wmf.24 refs T191072 (duration: 00m 53s)
15:35 twentyafterfour@deploy1001: scap failed: average error rate on 6/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details)
14:46 moritzm: installing tomcat8 security updates
14:34 moritzm: remove labvirt1018 from debmonitor (T207317)
14:28 godog: temporarily bump default socket receive memory to 1MB on graphite1001, restart statsd-proxy and statsite
14:22 godog: begin reformat of ms-be2041 - T199198
14:21 banyek: shutting down mysql and powering down db2042 (T202051)
14:13 godog: corrections to the statements above, graphite1004 not graphite1001
14:11 godog: ditto for statsite instances on graphite1001, temporarily bump receive socket memory to 1MB and bounce the service
14:08 godog: temporarily bump receive socket memory for statsd-proxy on graphite1001 and bounce the service
13:51 moritzm: installing libidn security updates
12:59 moritzm: installing libssh security updates
12:55 godog: bounce statsd-proxy on graphite1001
11:59 addshore: SWAT done
11:59 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Wikidata.org: enable sense data type T203888 (duration: 00m 54s)
11:54 mobrovac@deploy1001: Finished deploy [restbase/deploy@1041a02]: Disable onthisday check - T203588 (duration: 21m 23s)
11:54 zfilipin@deploy1001: Synchronized tests/InitialiseSettingsTest.php: SWAT: Test if logo specified in wgLogo/wgLogoHD exists (T207053) (duration: 00m 53s)
11:49 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix typo in IS.php: use ltwiki instead of ltwikipedia (T207081) (duration: 00m 54s)
11:39 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Use testwikidatawiki instead of testwikidata in IS.php (T207089) (duration: 00m 53s)
11:33 mobrovac@deploy1001: Started deploy [restbase/deploy@1041a02]: Disable onthisday check - T203588
11:29 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Use new wordmarks in uzwiki (T205226) (duration: 00m 53s)
11:10 zfilipin@deploy1001: Synchronized static/images/mobile/copyright/: SWAT: Upload uz specific wordmark (T205226) (duration: 00m 54s)
10:59 addshore: wikidata senses deploy slot done
10:57 addshore: addshore@mwmaint1002:~$ mwscript purgeList.php --wiki wikidatawiki --namespace 146
10:57 mobrovac@deploy1001: Finished deploy [restbase/deploy@88c8f26]: Parallelise onthisday call, take #4 (duration: 03m 52s)
10:55 addshore@deploy1001: Synchronized wmf-config/CommonSettings.php: RejectParserCacheValue Wikidata lexemes before sense deployment T203888 (duration: 00m 54s)
10:54 addshore@deploy1001: sync-file aborted: RejectParserCacheValue Wikidata lexemes before sense deploymentT203888 (duration: 00m 00s)
10:53 mobrovac@deploy1001: Started deploy [restbase/deploy@88c8f26]: Parallelise onthisday call, take #4
10:53 mobrovac@deploy1001: Finished deploy [restbase/deploy@88c8f26]: Parallelise onthisday call, take #3 (duration: 04m 13s)
10:51 addshore@deploy1001: Synchronized php-1.32.0-wmf.24/extensions/WikibaseLexeme: Wikidata: Make statement group IDs on Senses unique (duration: 00m 59s)
10:49 mobrovac@deploy1001: Started deploy [restbase/deploy@88c8f26]: Parallelise onthisday call, take #3
10:49 mobrovac@deploy1001: Finished deploy [restbase/deploy@88c8f26]: Parallelise onthisday call, take #2 (duration: 07m 32s)
10:41 mobrovac@deploy1001: Started deploy [restbase/deploy@88c8f26]: Parallelise onthisday call, take #2
10:41 mobrovac@deploy1001: Finished deploy [restbase/deploy@88c8f26]: Parallelise onthisday call - T203588 (duration: 11m 24s)
10:34 addshore@deploy1001: Synchronized wmf-config/Wikibase-production.php: Combine if blocks in Wikibase-production NOOP (duration: 00m 53s)
10:32 volans@deploy1001: Finished deploy [netbox/deploy@1cd4d43]: Upgrade to upstream v2.4.6 (5) - T205896 (duration: 00m 29s)
10:31 volans@deploy1001: Started deploy [netbox/deploy@1cd4d43]: Upgrade to upstream v2.4.6 (5) - T205896
10:31 volans@deploy1001: Finished deploy [netbox/deploy@1cd4d43]: Upgrade to upstream v2.4.6 (5) - T205896 (duration: 01m 37s)
10:31 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: BETA ONLY Remove wgLexemeEnableSenses from IS-labs (duration: 00m 53s)
10:30 volans@deploy1001: Started deploy [netbox/deploy@1cd4d43]: Upgrade to upstream v2.4.6 (5) - T205896
10:29 mobrovac@deploy1001: Started deploy [restbase/deploy@88c8f26]: Parallelise onthisday call - T203588
10:28 volans@deploy1001: Finished deploy [netbox/deploy@1cd4d43]: Upgrade to upstream v2.4.6 (4) - T205896 (duration: 00m 05s)
10:28 volans@deploy1001: Started deploy [netbox/deploy@1cd4d43]: Upgrade to upstream v2.4.6 (4) - T205896
10:15 addshore: purging wikidata lexemes
10:12 volans@deploy1001: Finished deploy [netbox/deploy@1cd4d43]: Upgrade to upstream v2.4.6 (3) - T205896 (duration: 00m 29s)
10:11 volans@deploy1001: Started deploy [netbox/deploy@1cd4d43]: Upgrade to upstream v2.4.6 (3) - T205896
10:10 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable senses on wikidatawiki T203888 (duration: 00m 53s)
10:09 volans@deploy1001: Finished deploy [netbox/deploy@1cd4d43]: Upgrade to upstream v2.4.6 (2) - T205896 (duration: 02m 01s)
10:07 volans@deploy1001: Started deploy [netbox/deploy@1cd4d43]: Upgrade to upstream v2.4.6 (2) - T205896
10:00 volans@deploy1001: Finished deploy [netbox/deploy@438f1c0]: Upgrade to upstream v2.4.6 - T205896 (duration: 03m 07s)
09:57 volans@deploy1001: Started deploy [netbox/deploy@438f1c0]: Upgrade to upstream v2.4.6 - T205896
09:52 XioNoX: activate bgp group Customer6 on cr4-ulsfo
09:20 banyek: enabling replication monitor check on pc1005 pc1006 pc2005 pc2006 (T206992)
09:18 godog: bounce statsd-proxy on graphite1001
09:08 moritzm: powercycling ms-be2019, stuck during reboot
09:01 banyek: enabling replication monitor check on pc1004 (T206992)
08:56 banyek: enabling replication monitor check on pc2004 (T206992)
08:41 banyek: disabling puppet on parser caches (T206992)
08:40 banyek: adding replication monitoring checks to parsercache hosts (T206992)
08:26 vgutierrez: Uploaded certcentral 0.1-2 to apt.wikimedia.org (stretch)
07:56 moritzm: rebooting swift backend servers in codfw for spectre v3/v4/L1TF security updates
07:43 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: Wikidata dispatch: reduce concurrent dispatchers to 2 (duration: 00m 59s)
05:34 marostegui: Restarting a failed s8 backup from dbstore1001 to db1116:3318
05:05 XioNoX: start office-DC link renumbering - T205985
02:51 ejegg: updated fundraising CiviCRM from 7b8d33bb4e to 83874e75ba
00:32 twentyafterfour: restarting apache on phab1001 to apply b3bfff1

2018-10-17

22:56 awight: Restarting ORES uwsgi service for T88997
22:38 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group0 wikis to 1.32.0-wmf.26 refs T191072
22:36 robh: bast4001 reboot is my fault, power cables were justled when i was decommssioning lvs4002 right above it in the rack
22:31 ejegg: updated fundraising CiviCRM from 5eac0634e6 to 7b8d33bb4e
22:24 ejegg: updated payments-wiki from 0385ad02a7 to a3892e4ed3
22:21 ppchelko@deploy1001: Finished deploy [restbase/deploy@88c8f26] (dev-cluster): Spread requests beetween MCS nodes for onthisday (duration: 02m 54s)
22:18 ppchelko@deploy1001: Started deploy [restbase/deploy@88c8f26] (dev-cluster): Spread requests beetween MCS nodes for onthisday
20:50 arlolra: Updated Parsoid to e6b708b (T204622, T187848, T207093)
20:40 arlolra@deploy1001: Finished deploy [parsoid/deploy@babf1da]: Updating Parsoid to e6b708b (duration: 08m 41s)
20:32 arlolra@deploy1001: Started deploy [parsoid/deploy@babf1da]: Updating Parsoid to e6b708b
20:17 mobrovac@deploy1001: Started restart [proton/deploy@a657059]: (no justification provided)
20:10 ejegg: updated fundraising CiviCRM from 4cc21d61c5 to 5eac0634e6
19:26 shdubsh: restart eventlogging for statsd DNS change - T88997
19:23 twentyafterfour: Mediawiki train is still blocked by T207288
19:19 godog: restart zuul for statsd DNS change - T88997
19:12 mutante: scb1003 - restart pdfrender
19:09 godog: roll-restart eventbus for statsd DNS change - T88997
19:00 krinkle@deploy1001: Synchronized php-1.32.0-wmf.26/includes/cache/: T193271 - I25aa0e27200a0 (duration: 01m 01s)
18:57 awight: Restarting ORES cluster to refresh DNS, T88997
18:48 banyek: repooling labsdb1009 (T181650)
18:48 shdubsh: restart navtiming on webperf nodes
18:39 godog: restart jmxtrans on kafka hosts
18:17 shdubsh: moving statsd cname to graphite1004
18:07 banyek: depooling labsdb1009 (T181650)
17:08 banyek: depooling labsdb1009 (T181650)
16:53 banyek: repooling labsdb1011
15:53 twentyafterfour@deploy1001: Synchronized php-1.32.0-wmf.26/extensions/AbuseFilter/: sync AbuseFilter revision 4e2a6b6 to 1.32.0-wmf.26 refs T207220 (duration: 00m 58s)
15:34 banyek@deploy1001: Synchronized wmf-config/db-codfw.php: T206593: Enabling db2096 for x1 (duration: 00m 56s)
15:31 banyek@deploy1001: Synchronized wmf-config/db-codfw.php: T206593: Enabling db2096 for x1 (duration: 00m 56s)
15:28 banyek: enabling db2096 for cluster x1 (T206593)
14:33 godog: upload prometheus-statsd-exporter 0.7.0+ds1-2 - T205870
14:01 marostegui: Repool labsdb1010, depool labsdb1011 - T181650
13:08 gehel: applying rps NIC config for all wdqs nodes - T206105
13:05 banyek: deplooling labsdb1010 (T181650)
12:56 banyek: enabling notifications on db2096 (T206593)
12:55 banyek: enabling notifications on db2096
11:40 Amir1: EU SWAT is done
11:40 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable reading from new backend of change tag everywhere (T194164) (duration: 00m 57s)
11:32 moritzm: installing graphicsmagick security updates
11:30 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: T206702 Enable client side error counting on Minerva (duration: 00m 57s)
11:26 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: T207196 gerrit:467736 Wikidata: enable JSON-LD data format on test.wikidata.org (duration: 00m 56s)
11:21 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: SWAT: T207196 Wikidata: add setting for setting the enabled entity data forms gerrit:467735 PT 2/2 (duration: 00m 56s)
11:19 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: T207196 Wikidata: add setting for setting the enabled entity data forms gerrit:467735 PT 1/2 (duration: 00m 57s)
11:17 Amir1: ladsgroup@mwmaint1002:~$ mwscript deleteLocalPasswords.php --wiki=enwiki --delete --batch-size 200 (This will cause lag on codfw)
11:15 addshore@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: T205611 T205330 Remove Wikidata RejectParserCacheValue hook gerrit:467913 (duration: 00m 56s)
11:11 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: SWAT: Increase wikidata dispatch randomness to 30 (duration: 00m 56s)
11:08 addshore@deploy1001: Synchronized wmf-config/Wikibase-production.php: SWAT: T207019 gerrit:467343 Enable WBQualityConstraintsSuggestionsBetaFeature on wikidatawiki (duration: 00m 56s)
11:04 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT gerrit:467691 Add constraint-suggestions to wgBetaFeaturesWhitelist (duration: 01m 10s)
11:04 ariel@deploy1001: Finished deploy [dumps/dumps@ed7eed9]: use lbzip2 for recombine steps if configured (duration: 00m 03s)
11:04 ariel@deploy1001: Started deploy [dumps/dumps@ed7eed9]: use lbzip2 for recombine steps if configured
09:34 XioNoX: update interfaces and BGP IPs for office-DC link (DC side, interfaces still disabled) - T205985
09:30 banyek: truncating parsercache tables on pc2006 (T206740)
09:12 _joe_: reenabling puppet (not running it) in codfw
09:12 _joe_: change applied to all appservers serving traffic
09:08 _joe_: running puppet on all apaches (appserver/api) in eqiad to pick up the wikipedia.org vhost refactor
09:05 _joe_: running puppet on mwdebug1001, then testing again wikipedia.org for regressions
09:04 _joe_: puppet disabled on the appservers, now merging the wikipedia.org conversion to mediawiki::web::vhost
08:43 mobrovac@deploy1001: Started restart [proton/deploy@a657059]: (no justification provided)
08:30 kartik@deploy1001: Finished deploy [cxserver/deploy@b30a323]: Update cxserver to 29e01e4 (T206305, T204668) (duration: 03m 54s)
08:27 kartik@deploy1001: Started deploy [cxserver/deploy@b30a323]: Update cxserver to 29e01e4 (T206305, T204668)
08:09 banyek: stopping binlog purgers on the parsercache hosts (the binlogs will be kept for 24hrs) - T206740
08:00 banyek: truncating parsercache tables on pc2005 (T206740)
06:52 jynus: fixing s8 master drifts T206743
02:10 ejegg: updated payments-wiki from 7fb1aae963 to 0385ad02a7
01:24 legoktm@deploy1001: Synchronized wmf-config/CommonSettings.php: Add REL1_32 to ExtensionDistributor (duration: 00m 59s)

2018-10-16

22:11 ppchelko@deploy1001: Finished deploy [restbase/deploy@d9e3a09]: Downgrade major-greater to minor-greater if no-cache is required, take 6 (duration: 01m 18s)
22:09 ppchelko@deploy1001: Started deploy [restbase/deploy@d9e3a09]: Downgrade major-greater to minor-greater if no-cache is required, take 6
22:09 ppchelko@deploy1001: Finished deploy [restbase/deploy@d9e3a09]: Downgrade major-greater to minor-greater if no-cache is required, take 5 (duration: 05m 16s)
22:04 ppchelko@deploy1001: Started deploy [restbase/deploy@d9e3a09]: Downgrade major-greater to minor-greater if no-cache is required, take 5
22:04 ppchelko@deploy1001: Finished deploy [restbase/deploy@d9e3a09]: Downgrade major-greater to minor-greater if no-cache is required, take 4 (duration: 03m 53s)
22:00 ppchelko@deploy1001: Started deploy [restbase/deploy@d9e3a09]: Downgrade major-greater to minor-greater if no-cache is required, take 4
22:00 ppchelko@deploy1001: Finished deploy [restbase/deploy@d9e3a09]: Downgrade major-greater to minor-greater if no-cache is required, take 3 (duration: 04m 15s)
21:58 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group0 wikis to 1.32.0-wmf.24 refs T191072
21:55 ppchelko@deploy1001: Started deploy [restbase/deploy@d9e3a09]: Downgrade major-greater to minor-greater if no-cache is required, take 3
21:55 ppchelko@deploy1001: Finished deploy [restbase/deploy@d9e3a09]: Downgrade major-greater to minor-greater if no-cache is required, take 2 (duration: 09m 11s)
21:46 ppchelko@deploy1001: Started deploy [restbase/deploy@d9e3a09]: Downgrade major-greater to minor-greater if no-cache is required, take 2
21:45 ppchelko@deploy1001: Finished deploy [restbase/deploy@d9e3a09]: Downgrade major-greater to minor-greater if no-cache is required (duration: 03m 53s)
21:42 ppchelko@deploy1001: Started deploy [restbase/deploy@d9e3a09]: Downgrade major-greater to minor-greater if no-cache is required
21:18 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group0 wikis to 1.32.0-wmf.26 refs T191072
20:55 twentyafterfour@deploy1001: Finished scap: Syncing 1.32.0-wmf.26 refs T191072 (duration: 26m 32s)
20:28 twentyafterfour@deploy1001: Started scap: Syncing 1.32.0-wmf.26 refs T191072
20:14 shdubsh: restarted pdfrender on scb1003
18:44 ppchelko@deploy1001: Started restart [proton/deploy@a657059]: Try restarting again for metrics
18:43 ppchelko@deploy1001: Started restart [proton/deploy@a657059]: Try restarting again for metrics
18:42 ppchelko@deploy1001: Finished deploy [proton/deploy@a657059]: Try restarting for metrics (duration: 00m 20s)
18:42 ppchelko@deploy1001: Started deploy [proton/deploy@a657059]: Try restarting for metrics
17:01 _joe_: restarted pdfrender on scb1004
16:33 akosiaris: depool restbase-async from eqiad in order to test traffic going to parsoid codfw
16:15 _joe_: disabled puppet on all appservers, merging wikidata apache change, re-enabling puppet on mwdebug1001 for testing
14:51 mobrovac@deploy1001: Finished deploy [proton/deploy@a657059]: Rollback to puppeteer v1.5.0 - T186748 (duration: 00m 49s)
14:51 mobrovac@deploy1001: Started deploy [proton/deploy@a657059]: Rollback to puppeteer v1.5.0 - T186748
14:28 godog: roll-restart elasticsearch on logstash100[456] to change elasticsearch data dir - T206454
14:06 godog: depool in turn logstash1008 and logstash1009 to change elasticsearch data dir - T206454
13:55 godog: depool logstash1007 to change elasticsearch data dir - T206454
13:54 XioNoX: router back and healthy, enable external BGP sessions on cr2-eqdfw - T203261
13:51 moritzm: rebooting acamar for update to stretch-proposed-updates kernel
13:44 XioNoX: reboot cr2-eqdfw for upgrade - T203261
13:43 XioNoX: disable external BGP sessions on cr2-eqdfw - T203261
13:43 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting comment table migration stage to write-new/read-both on group 0 (T166733) (duration: 00m 50s)
13:34 XioNoX: start install process on cr2-eqdfw (non impacting before reboot) - T203261
13:11 akosiaris: pool codfw for apertium|citoid|cxserver|eventbus|eventstreams|graphoid|mathoid|mobileapps|ores|parsoid|pdfrender|proton|recommendation-api|restbase|restbase-async|wdqs|wdqs-internal|zotero
13:11 akosiaris@puppetmaster1001: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=^apertium|citoid|cxserver|eventbus|eventstreams|graphoid|mathoid|mobileapps|ores|parsoid|pdfrender|proton|recommendation-api|restbase|restbase-async|wdqs|wdqs-internal|zotero$
13:08 elukey: restart memcached on mc1035 with -R 200 (will wipe the object cache shard as consequence) - T203786
12:57 akosiaris: pool mathoid eqiad
12:52 gtirloni: T186571 removed legofan4000 user from project-tools group (leftover from T165624 legofan4000->macfan4000 rename)
12:44 akosiaris@deploy1001: scap-helm mathoid finished
12:43 akosiaris@deploy1001: scap-helm mathoid cluster eqiad completed
12:43 akosiaris@deploy1001: scap-helm mathoid upgrade production stable/mathoid --reset-values -f mathoid.yaml [namespace: mathoid, clusters: eqiad]
12:35 akosiaris: depool eqiad mathoid for helm chart upgrade
12:32 akosiaris: pool codfw mathoid
12:14 akosiaris@deploy1001: scap-helm mathoid finished
12:14 akosiaris@deploy1001: scap-helm mathoid cluster codfw completed
12:14 akosiaris@deploy1001: scap-helm mathoid upgrade production stable/mathoid --reset-values -f mathoid.yaml [namespace: mathoid, clusters: codfw]
12:08 Amir1: EU SWAT is done
12:08 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable reading from new backend of change_tag in s7 (T194164) (duration: 00m 50s)
12:03 akosiaris@deploy1001: scap-helm mathoid finished
12:03 akosiaris@deploy1001: scap-helm mathoid cluster codfw completed
12:03 ladsgroup@deploy1001: Synchronized php-1.32.0-wmf.24/includes/changetags/ChangeTags.php: SWAT: Avoid fatals when the filter tags is empty (T194164) (duration: 00m 50s)
12:03 akosiaris@deploy1001: scap-helm mathoid upgrade production stable/mathoid --set main_app.limits.memory=1G [namespace: mathoid, clusters: codfw]
12:02 akosiaris@deploy1001: scap-helm mathoid upgrade production stable/mathoid --set main_app.limits.memory=1g [namespace: mathoid, clusters: codfw]
11:49 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Re-enable search integration for ArticlePlaceholder (T195751) (duration: 00m 50s)
11:38 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Translate on idwikimedia (T204292) (duration: 00m 49s)
11:32 banyek: the binlog purging stopped on pc2004 (T206740)
11:27 akosiaris: upgrade mathoid chart to version 0.0.12
11:26 akosiaris@deploy1001: scap-helm mathoid finished
11:26 akosiaris@deploy1001: scap-helm mathoid cluster codfw completed
11:26 akosiaris@deploy1001: scap-helm mathoid upgrade production stable/mathoid [namespace: mathoid, clusters: codfw]
11:24 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: Add throttle rule for editathon at University of North Carolina at Charlotte (T207043) (duration: 00m 49s)
11:18 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: Add new throttle rule for WMCL Editathon (T206914) (duration: 00m 49s)
11:09 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: Add throttle rule for "Night of the Digital Language" (T206408) (duration: 00m 49s)
11:05 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: Remove expired throttle rule (T207015) (duration: 00m 50s)
11:02 banyek: truncating tables in parsecache@pc2004 (T206740)
10:52 moritzm: rolling reboot of thumbor in eqiad for kernel security updates
10:50 godog: run puppet on scb to deploy db configuration for recommendation-service
10:37 banyek: stopping pc2005 -> pc1005 replication (T206740)
10:37 banyek: stopping pc2006 -> pc1006 replication (T206740)
10:22 jynus: running database maintenance tasks on cumin1001, expect very high memory usage
09:53 akosiaris: upload blubber_0.6.0-1_amd64 to apt.wikimedia.org/jessie-wikimedia/main and apt.wikimedia.org/stretch-wikimedia/main T206766
09:03 moritzm: rolling reboot of thumbor in codfw for kernel security updates
08:56 banyek: stopping pc2004 -> pc1004 replication (T206740)
08:42 moritzm: removed mwmaint1001 from debmonitor (T192457)
07:46 akosiaris: upgrade apertium-apy throught the fleet T199447
07:46 akosiaris: upgrade apertium-apy throught the fleet
07:22 akosiaris: upload apertium-apy_0.11.4-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main T199447
07:22 akosiaris: upload apertium-apy_0.11.4-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
07:20 akosiaris@deploy1001: scap-helm mathoid finished
07:19 akosiaris@deploy1001: scap-helm mathoid cluster codfw completed
07:19 akosiaris@deploy1001: scap-helm mathoid upgrade production stable/mathoid [namespace: mathoid, clusters: codfw]
07:19 akosiaris@deploy1001: scap-helm mathoid upgrade [namespace: mathoid, clusters: codfw]
07:17 moritzm: installing net-snmp security updates
06:32 legoktm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Revert "Enable reading from new backend of change_tag in s7" (T194164) (duration: 00m 50s)
06:05 jynus: stopping db1092 and db1087 in sync T206743
05:10 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Remove db1092 BBU comments after BBU replacement (duration: 00m 52s)
00:23 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@c81dd9e]: Redeploy Updater for removal of props channel (duration: 10m 21s)
00:13 smalyshev@deploy1001: Started deploy [wdqs/wdqs@c81dd9e]: Redeploy Updater for removal of props channel

2018-10-15

20:52 arlolra: Updated Parsoid to 8f3ff40 (T205642, T206003, T187848, T205455, T205743)
20:37 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@834d00a]: Update mobileapps to c2a4ef9 (T206701 T206467 T168875) (duration: 03m 47s)
20:34 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@834d00a]: Update mobileapps to c2a4ef9 (T206701 T206467 T168875)
20:32 arlolra@deploy1001: Finished deploy [parsoid/deploy@b758124]: Updating Parsoid to 8f3ff40 (duration: 11m 43s)
20:20 arlolra@deploy1001: Started deploy [parsoid/deploy@b758124]: Updating Parsoid to 8f3ff40
19:37 mforns@deploy1001: Finished deploy [analytics/refinery@3f4adf8]: deploy refinery together with source version 0.0.78 without all removed old jars (duration: 05m 18s)
19:33 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@ff3bf90]: Redeploy 1010 (duration: 00m 28s)
19:33 smalyshev@deploy1001: Started deploy [wdqs/wdqs@ff3bf90]: Redeploy 1010
19:32 mforns@deploy1001: Started deploy [analytics/refinery@3f4adf8]: deploy refinery together with source version 0.0.78 without all removed old jars
19:27 mforns@deploy1001: Finished deploy [analytics/refinery@1fc53d9]: deploy refinery together with source version 0.0.78 (duration: 15m 56s)
19:11 mforns@deploy1001: Started deploy [analytics/refinery@1fc53d9]: deploy refinery together with source version 0.0.78
18:59 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable reading from new backend of change_tag in s7 (T194164) (duration: 00m 49s)
18:59 mutante: LDAP - added crusnov to wmf and ops groups
18:51 tgr: pulled gerrit 467315 to mwdeploy1001 (no-op, no scap needed)
18:47 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@ff3bf90]: GUI updates and new Updater build (duration: 13m 57s)
18:44 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: cswikivoyage has HD logo even the project doesnt exist (T207066) (duration: 00m 49s)
18:39 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable AICaptcha data collection (T186244) (duration: 00m 49s)
18:33 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix a typo in wgLogoHD (mapwiki => napwiki) T207056, Remove techcomwikis row in wgLogo, techcomwiki doesnt exist T207056 (duration: 00m 48s)
18:33 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@ff3bf90]: GUI updates and new Updater build
18:30 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: Beta: Show share button on mobile web for beta user (no-op) (duration: 00m 49s)
18:14 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT enable senses on testwikidatawiki T203887 (duration: 00m 49s)
18:10 addshore@deploy1001: Synchronized wmf-config/Wikibase-production.php: SWAT: T207019 Enable WBQualityConstraintsSuggestionsBetaFeature on testwikidatawiki (duration: 00m 49s)
18:01 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@ff3bf90]: Test deployment - GUI update and new Updater build(wdqs1009) (duration: 02m 11s)
17:59 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@ff3bf90]: Test deployment - GUI update and new Updater build(wdqs1009)
17:57 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@ff3bf90]: Test deployment - GUI update and new Updater build(wdqs1009) (duration: 02m 10s)
17:55 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@ff3bf90]: Test deployment - GUI update and new Updater build(wdqs1009)
16:54 marostegui: Start replication on db1087 and db1092 to avoid them lagging behind the whole night (nothing running there at this time)
16:36 cmjohnson1: replacing pem0 on asw2-a7-eqiad T206972
16:18 _joe_: restart prometheus-mcrouter-exporter.service across the fleet
15:39 marostegui: Stop MySQL and poweroff db1092 for BBU replacement - T205514
15:31 andrewbogott: restarting slapd on seaborgium as a test for T205463
15:14 cmjohnson1: replacing optics asw2-b fpc2 -fpc8
15:13 mforns@deploy1001: Finished deploy [analytics/refinery@9b288c5]: deploy refinery together with source version 0.0.77 (duration: 20m 19s)
14:53 mforns@deploy1001: Started deploy [analytics/refinery@9b288c5]: deploy refinery together with source version 0.0.77
14:46 marostegui: Ease consistency replication options on db2048 to mitigate lag
14:29 moritzm: rebooting backup2001 for some tests
13:35 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting MCR migration stage to write-both/read-new on Commons (T198308) (duration: 00m 49s)
13:32 banyek@deploy1001: Synchronized wmf-config/db-codfw.php: T206593: adding db2096 to hosts (and repooling db2069) (duration: 00m 49s)
13:30 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T206593: adding db2096 to hosts (and repooling db2069) (duration: 00m 49s)
13:16 jynus: stopping db1092 and db1087 in sync T206743
13:10 Jeff_Green: auithdns-update to deploy saiph->frpig2001 rename
13:02 godog: upload prometheus-statsd-exporter 0.7.0 - T205870
12:45 banyek: rebooting db2096
12:44 gehel: reseting kafka offsets on wdqs public cluster
12:44 elukey: complete rolling restart of eventbus on kafka[12]00[1-3] for python security upgrades (only codfw was done)
12:41 elukey: upgrade prometheus-memcached-exporter on swift and thumbor
11:57 Amir1: start of mwscript deleteLocalPasswords.php --delete --batch-size 200 on all wikis
11:38 zeljkof: EU SWAT finished
11:29 hoo: Started rebuildItemsPerSite on mwmaint1002 (T44325). Can be killed at any time, if necessary.
11:26 zfilipin@deploy1001: scap failed: average error rate on 4/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details)
11:09 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable reading from ct_tag_id in s7 (T194164) (duration: 00m 49s)
10:57 moritzm: installing ghostscript security updates for jessie
10:47 moritzm: installing tomcat7 security updates
10:43 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 49s)
10:42 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 49s)
09:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1092 for recloning - T206743 (duration: 00m 49s)
09:45 marostegui: Stop MySQL on db1116:3318 to reclone db1092
09:41 banyek: max_binlog_size is set back to 1048576000 on ParseCache hosts (T206740)
09:23 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Restore original weight for db1104 (duration: 00m 49s)
09:00 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1104 (duration: 00m 48s)
08:58 banyek@deploy1001: Synchronized wmf-config/db-codfw.php: T206593: depooling db2069 (duration: 00m 48s)
08:50 elukey: restart hadoop yarn resource managers on an-master* to pick up new jvm settings
08:49 XioNoX: repool eqsin - T206861
08:48 banyek: depooling db2033 (T206593)
08:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1104 - T206743 (duration: 00m 49s)
08:17 moritzm: installing imagemagick security update
07:57 godog: reformat ms-be2040 with crc=1 finobt=0 - T199198
07:32 banyek: reimaging db2096(T206593)
07:31 banyek: reimaging db2096
07:24 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1104 - T206743 (duration: 00m 48s)
07:15 marostegui: Stop MySQL at db1116:3318 to clone db1104
07:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1104 - T206743 (duration: 00m 49s)
07:04 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1109 (duration: 00m 49s)
06:55 XioNoX: add v6 monitoring for mr1-ulsfo OOB - T206778
06:49 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1109 (duration: 00m 49s)
06:40 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1109 (duration: 00m 50s)
05:20 kartik@deploy1001: Finished deploy [cxserver/deploy@fd74c3b]: Update cxserver to b51f363 (T203077, T99934, T203550) (duration: 04m 25s)
05:16 kartik@deploy1001: Started deploy [cxserver/deploy@fd74c3b]: Update cxserver to b51f363 (T203077, T99934, T203550)
05:16 marostegui: Stop MySQL on db1109 for recloning - T206743
05:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1109 (duration: 00m 50s)
05:11 marostegui: Stop MySQL on db1116:3318 to use it to clone db1109
03:18 kartik@deploy1001: Finished deploy [cxserver/deploy@5a70ef1]: Update cxserver to 47a864b (T205420, T203077, T205700, T205616) (duration: 04m 44s)
03:14 kartik@deploy1001: Started deploy [cxserver/deploy@5a70ef1]: Update cxserver to 47a864b (T205420, T203077, T205700, T205616)
00:45 krinkle@deploy1001: Synchronized multiversion/MWRealm.php: I79fb3d194a58: use env.php (duration: 00m 49s)
00:08 krinkle@deploy1001: Synchronized wmf-config/: I79fb3d194a: add env.php file (not yet used) (duration: 00m 50s)

2018-10-14

23:42 krinkle@deploy1001: Synchronized multiversion/getMWVersion: Ice9a74e73481 no-op (duration: 00m 49s)
23:21 krinkle@deploy1001: Synchronized wmf-config/ProductionServices.php: If4d8faa4 (duration: 00m 48s)
21:48 krinkle@deploy1001: Synchronized multiversion/MWMultiVersion.php: I83b2bdd53c13e (duration: 00m 50s)
20:47 krinkle@deploy1001: Synchronized wmf-config/import.php: beta-only (duration: 00m 54s)
16:34 volans: forcing a puppet run on all eqsin hosts with batch 1 to clear most of the alarms - T206861
08:54 elukey: restart Yarn resource manager on an-master1002 to force an-master1001 to take the leadership back - T206943
08:34 elukey: powercycle restbase1015 (frozen, no ssh, no metrics, no root console via serial available)
00:48 krinkle@deploy1001: Synchronized php-1.32.0-wmf.24/extensions/CentralAuth/includes/specials/SpecialGlobalGroupMembership.php: T203767 - If2bfa092b (duration: 00m 50s)

2018-10-13

23:37 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T45086 - I4857e8ac (duration: 00m 51s)
03:07 bblack: eqsin repooled

2018-10-12

18:56 brion: restarted vp9 background transcodes in eqiad, via mwmaint1002
18:37 addshore: modified attachLatest.php script finished running over 9395 pages T206743
18:25 addshore: running modified attachLatest.php script over ~9000 pages on wikidatawiki (with added wait for slaves) T206743
15:50 mutante: repair /dev/sde1 on ms-be2041 - T199198
15:48 mutante: repair /dev/sdh1 on ms-be1043 - T199198
14:23 _joe_: depooling eqsin via geodns due to loss of power redundancy
13:35 gehel: repooling wdqs1003 catched up on lag
12:59 gehel: depooling wdqs1003 to catch up on lag
12:20 bblack: uploading gdnsd 2.99.9942-beta-1+wmf1 to stretch-wikimedia
10:51 _joe_: depooling mw2252 for mcrouter tests T203786
10:27 hoo: Updated the Wikidata property suggester with data from Monday's JSON dump and applied the T132839 workarounds
10:08 addshore@deploy1001: Synchronized php-1.32.0-wmf.24/extensions/WikimediaEvents/extension.json: T205283 gerrit:466843 Update Schema:WMDEBannerEvents rev to 18437830 (duration: 00m 52s)
09:01 elukey: rolling restart of eventbus on kafka[1,2]00[1-3] to pick up python security upgrades
05:54 moritzm: installing git security updates on trusty
02:25 ejegg: updated fundraising tools from 3754f32 to 5a2d39b

2018-10-11

23:33 Reedy: ran mwscript extensions/ShortUrl/populateShortUrlTable.php --wiki=gomwiki T206741
23:32 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable shorturl on gomwiki (duration: 00m 48s)
23:30 Reedy: created shorturl table on gomwiki T206741
23:26 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable FileExporter to Meta-Wiki (duration: 00m 49s)
23:21 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Disable CongressLookup (duration: 00m 49s)
23:05 krinkle@deploy1001: Synchronized php-1.32.0-wmf.24/includes/jobqueue/jobs/ThumbnailRenderJob.php: T203135 - Ib4640e (duration: 00m 49s)
22:56 dzahn@neodymium: conftool action : set/pooled=inactive; selector: name=mwmaint1001.eqiad.wmnet
22:53 mutante: netbox - correction, mwmaint1001 to status "Staged", following new lifecycle docs T192457
22:50 mutante: netbox - renamed mwmaint1001 to mw1279, changed status to inventory, renamed in DNS - T192457
22:45 krinkle@deploy1001: Synchronized php-1.32.0-wmf.24/includes/Revision/RenderedRevision.php: I553dba13486 (duration: 00m 51s)
22:30 mutante: mwmaint1001 - shutting down after final backup of /home, renaming back to mw1297 in DNS and DHCP, and reinstalling (T192457)
21:53 mutante: mwmaint1001 - schduled downtime, is being renamed back to mw1297 and reinstalled
21:47 mutante: mwmaint2001 - rsyncing home dirs from mwmaint1002 to /root/home-mwmaint1002 (which includes home-terbium even!) in case anyone is missing anything from one of mwaint*
21:41 mutante: mwmaint2001 - deleting 60G of unneeded files from home
20:37 XioNoX: add IPv6 to mr1-ulsfo OOB - T206778
18:46 sbisson@deploy1001: Synchronized php-1.32.0-wmf.24/extensions/PageTriage/: SWAT: Handle page that are unnominated for deletion (duration: 00m 50s)
18:34 sbisson@deploy1001: Synchronized php-1.32.0-wmf.24/extensions/PageTriage/modules/ext.pageTriage.views.list/ext.pageTriage.listControlNav.js: SWAT: Default to deleted and others when no type is selected on mode switch (duration: 00m 50s)
18:22 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove config for RCFilters variables being removed from Core (duration: 00m 49s)
18:14 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2083 and db2085:3318 (duration: 00m 48s)
18:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 (duration: 00m 49s)
18:09 sbisson@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: Add copyviobot group management to relevant wikis (duration: 00m 49s)
17:36 gehel: repooling wdqs1003, catched up on lag
away: automated binlog purging started on pc2004, pc2005, pc2006
16:54 gehel: depooling wdqs1003 to let it catch up on lag
15:38 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1087 (duration: 00m 50s)
15:12 marostegui: Stop MySQL on db2085:3318 to reclone db1101:3318 - T206743
15:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101:3318 (duration: 00m 49s)
15:04 akosiaris: Media storage/Swift Swift set to active/passive
15:01 akosiaris: Media storage/Swift Swift set to active/active
14:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099:3318 (duration: 00m 48s)
14:52 jynus: deploying wikidata row fix to db1087 with replication enabled
14:47 END: (PASS) - Cookbook sre.switchdc.services.02-restore-ttl (exit_code=0) (volans@neodymium)
14:47 START: - Cookbook sre.switchdc.services.02-restore-ttl (volans@neodymium)
14:36 END: (PASS) - Cookbook sre.switchdc.services.01-switch-dc (exit_code=0) (volans@neodymium)
14:36 Switching: services parsoid, restbase, restbase-async, mobileapps, apertium, citoid, cxserver, eventstreams, graphoid, mathoid, proton, pdfrender, recommendation-api, zotero, eventbus, ores, wdqs, wdqs-internal: codfw => eqiad (volans@neodymium)
14:36 START: - Cookbook sre.switchdc.services.01-switch-dc (volans@neodymium)
14:35 END: (PASS) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=0) (volans@neodymium)
14:30 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T206743: mariadb: Depool db1087 (duration: 00m 49s)
14:30 START: - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (volans@neodymium)
14:28 banyek: depooling db1087 (T206743)
14:28 banyek: depooling db1087
14:15 elukey: reboot eventlog1002 for kernel upgrades
14:15 jynus: applying row filling to (most) eqiad s8 dbs, including the mater
14:13 moritzm: install libxml2 security updates on jessie servers
13:55 jynus: recovering rows to db1092
13:26 jynus: filling in missing rows on dbstore1002
13:23 marostegui: Stop MySQL on db2083 to reclone db1116:3318 - T206743
13:21 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2083 (duration: 00m 49s)
13:20 marostegui: Stop MySQL on db1116:3318 to reclone it from db2083 - T206743
12:43 elukey: upgrade prometheus-memcached-exporter on mc1*
12:38 elukey: upgrade prometheus-memcached-exporter on mc2*
12:15 elukey: upgrade prometheus-memcached-exporter on mc2035
12:14 elukey: upload prometheus-memcached-exporter_0.4.1+git20181010.2fa99eb-1 to (jessie|stretch)-wikimedia
12:12 Amir1: EU SWAT is done
11:30 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set some small wikis to read new for change tag backend (T194164) (duration: 00m 50s)
11:10 marostegui: Stop MYSQL on db2085:3318 and db1099:3318 T206743
11:09 marostegui: Stop MYSQL on db2088:3318 and db1099:3318 T206743
11:08 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2085:3318 and db1099:3318 (duration: 00m 49s)
11:07 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db2085:3318 and db1099:3318 (duration: 00m 49s)
11:07 banyek: binlog expiration set to 60 days on db2045
08:30 banyek: setting up some automated binlog purge mechanism on pc1004,pc1005,pc1006
08:26 jynus: setting up replication from pc2005 -> pc1005 and from pc2006 -> pc2006
08:20 jynus: setting up replication from pc2004 -> pc1004
08:04 banyek: purging binary logs on pc1006
08:04 banyek: purging binary logs on pc1005
08:04 jynus: running /usr/local/bin/mwscript purgeParserCache.php --wiki=aawiki --age=1900800 --msleep 0
08:04 banyek: purging binary logs on pc1004
07:57 gehel: rolling restart blazegraph on wdqs-internal for config change - T206648
07:43 addshore: deploy https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Wikibase/+/466031 to mwmaint1002 only (increasing tracking of wikidata dispatching) T205865
07:36 elukey: roll restart of aqs on aqs100[4-9] to pick up new Druid settings
06:00 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase db1092 weight (duration: 00m 49s)
05:43 marostegui: Purge binary logs on pc2005 due to disk space issues - T206740
05:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1109 (duration: 00m 48s)
05:24 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1109 (duration: 00m 51s)
02:25 krinkle@deploy1001: Synchronized w/static.php: T127233 - Ic6acb70 (duration: 00m 49s)
02:10 krinkle@deploy1001: Synchronized php-1.32.0-wmf.24/includes/page/WikiPage.php: T203942 - Ib211d98498f (duration: 00m 49s)
02:07 krinkle@deploy1001: Synchronized php-1.32.0-wmf.24/tests/phpunit/includes/page/: Ib211d98498f (duration: 00m 49s)
01:38 krinkle@deploy1001: Synchronized wmf-config/etcd.php: T176370 - I5e7e5d167d517 (duration: 00m 55s)

2018-10-10

23:08 krinkle@deploy1001: Synchronized php-1.32.0-wmf.24/maintenance/resources/foreign-resources.yaml: Ic865e7077d (duration: 00m 49s)
22:59 krinkle@deploy1001: Synchronized php-1.32.0-wmf.24/extensions/MultimediaViewer/: T206099 - I53dbce0a (duration: 00m 49s)
22:43 krinkle@deploy1001: Synchronized php-1.32.0-wmf.24/includes/specials/SpecialDeletedContributions.php: T187619 - Ic6b0d8020553 (duration: 00m 48s)
22:41 krinkle@deploy1001: Synchronized php-1.32.0-wmf.24/extensions/ORES/includes/FetchScoreJob.php: T204753 - Icc28230585bc (duration: 00m 49s)
22:25 mutante: icinga1001 - chmod 2710 /var/lib/icinga/rw
22:16 krinkle@deploy1001: Synchronized wmf-config/arclamp.php: T206092 - If607ad111a (duration: 00m 48s)
21:51 krinkle@deploy1001: Synchronized php-1.32.0-wmf.24/extensions/ContentTranslation/specials/SpecialContentTranslation.php: T205433 - Ib34b28 (duration: 00m 49s)
21:48 krinkle@deploy1001: Synchronized php-1.32.0-wmf.24/extensions/Echo/includes/DiscussionParser.php: T204291 - Ia5323b401b94 (duration: 00m 51s)
21:45 XioNoX: Add icinga1001 to mr* security policies - T206704
20:34 thcipriani: upgrading ci jenkins install on contint1001
20:19 thcipriani: upgrading releases-jenkins jenkins install on releases1001
20:17 thcipriani: upgrading releases-jenkins jenkins install on releases2001
19:58 mutante: icinga - enabled icinga service on icinga1001 (stretch), but all notifications are disabled
19:43 mutante: awight restarted ORES celery workers on ores2003 (~17:00), ores200* (17:05)
19:35 kaldari@deploy1001: Finished scap: (no justification provided) (duration: 22m 05s)
19:13 kaldari@deploy1001: Started scap: (no justification provided)
19:11 kaldari: scap sync to rebuild i18n cache
18:35 XioNoX: disable VC port 1/2 on asw2-c-eqiad:fpc3 (to fpc8)
18:20 otto@deploy1001: Finished deploy [analytics/refinery@28bbee8]: Add accept header to webrequest logs - T170606 (duration: 10m 34s)
18:19 XioNoX: delete sessions to AS6805 on cr2-esams (left AMS-IX)
18:10 otto@deploy1001: Started deploy [analytics/refinery@28bbee8]: Add accept header to webrequest logs - T170606
18:09 otto@deploy1001: Finished deploy [analytics/refinery@4e2d956]: Add accept header to webrequest logs - T170606 (duration: 04m 35s)
18:05 otto@deploy1001: Started deploy [analytics/refinery@4e2d956]: Add accept header to webrequest logs - T170606
17:49 XioNoX: replace 10.195.0.0/25 with 10.195.0.0/24 in prefix-list fundraising-codfw4 on cr1/2-codfw - T206637
16:25 mutante: LDAP - added isaacj to wmf group (for SWAP access, existing shell user since recently) (T206631) (T205840)
16:16 _joe_: restart of now-unused jobqueue redises for stopping the alerts post-switchover
16:09 ejegg: updated CiviCRM from 1165e7ed79 to 4cc21d61c5
15:59 vgutierrez: Uploaded certcentral 0.1 to apt.wikimedia.org (stretch) - T199711
15:55 cmjohnson1: scheduled downtime for host cloudvirt1019 swap raid card T196507
15:35 moritzm: uploaded jenkins 2.138.2 security release to apt.wikimedia.org (jessie/stretch) (T206234)
15:11 _joe_: started again hhvm on mwmaint2001
14:51 ejegg: turned fundraising scheduled jobs back on
14:43 ejegg: turned off fundraising scheduled jobs
14:42 END: (PASS) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=0) (volans@neodymium)
14:42 START: - Cookbook sre.switchdc.mediawiki.08-restore-ttl (volans@neodymium)
14:42 END: (FAIL) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=99) (volans@neodymium)
14:40 START: - Cookbook sre.switchdc.mediawiki.08-start-maintenance (volans@neodymium)
14:39 END: (FAIL) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=99) (volans@neodymium)
14:38 START: - Cookbook sre.switchdc.mediawiki.08-start-maintenance (volans@neodymium)
14:33 oblivian@puppetmaster1001: conftool action : set/weight=15; selector: cluster=api_appserver,service=apache2,dc=eqiad,name=mw123.*
14:31 oblivian@puppetmaster1001: conftool action : set/weight=15; selector: cluster=api_appserver,service=apache2,dc=eqiad,name=mw122.*
14:19 END: (PASS) - Cookbook sre.switchdc.mediawiki.08-update-tendril (exit_code=0) (volans@neodymium)
14:19 START: - Cookbook sre.switchdc.mediawiki.08-update-tendril (volans@neodymium)
14:18 END: (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) (volans@neodymium)
14:18 MediaWiki: read-only period ends at: 2018-10-10 14:18:26.908958 (volans@neodymium)
14:18 START: - Cookbook sre.switchdc.mediawiki.07-set-readwrite (volans@neodymium)
14:18 END: (PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0) (volans@neodymium)
14:18 START: - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (volans@neodymium)
14:17 END: (PASS) - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions (exit_code=0) (volans@neodymium)
14:17 START: - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions (volans@neodymium)
14:17 END: (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-traffic (exit_code=0) (volans@neodymium)
14:15 START: - Cookbook sre.switchdc.mediawiki.04-switch-traffic (volans@neodymium)
14:15 END: (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0) (volans@neodymium)
14:14 START: - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (volans@neodymium)
14:14 END: (PASS) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=0) (volans@neodymium)
14:14 START: - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (volans@neodymium)
14:14 END: (PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0) (volans@neodymium)
14:13 MediaWiki: read-only period starts at: 2018-10-10 14:13:46.068081 (volans@neodymium)
14:13 START: - Cookbook sre.switchdc.mediawiki.02-set-readonly (volans@neodymium)
14:10 END: (PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0) (volans@neodymium)
14:10 START: - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (volans@neodymium)
14:10 END: (PASS) - Cookbook sre.switchdc.mediawiki.00-warmup-caches (exit_code=0) (volans@neodymium)
14:07 START: - Cookbook sre.switchdc.mediawiki.00-warmup-caches (volans@neodymium)
14:07 END: (PASS) - Cookbook sre.switchdc.mediawiki.00-warmup-caches (exit_code=0) (volans@neodymium)
14:05 START: - Cookbook sre.switchdc.mediawiki.00-warmup-caches (volans@neodymium)
14:05 END: (PASS) - Cookbook sre.switchdc.mediawiki.00-warmup-caches (exit_code=0) (volans@neodymium)
14:01 START: - Cookbook sre.switchdc.mediawiki.00-warmup-caches (volans@neodymium)
14:01 END: (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) (volans@neodymium)
14:01 START: - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (volans@neodymium)
14:00 END: (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) (volans@neodymium)
14:00 START: - Cookbook sre.switchdc.mediawiki.00-disable-puppet (volans@neodymium)
12:18 _joe_: decommissioning conf1001-1003: stopping etcd, nginx, and masking both
11:41 jynus: renaming some s3 wiki tables on eqiad master to prevent split brain T184805
11:29 zeljkof: EU SWAT finished
11:26 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Permissions changes on itwikibooks (T206447) (duration: 00m 57s)
10:54 marostegui: Set a replication filter on db1075 (s3 eqiad) to ignore enwikivoyage, cebwiki, shwiki, srwiki & mgwiktionary - T184805
10:49 marostegui@deploy1001: Synchronized dblists/s5.dblist: Update s5.dblist to reflect the wikis moved from s3 - T184805 (duration: 00m 56s)
10:48 marostegui@deploy1001: Synchronized dblists/s3.dblist: Update s3.dblist to reflect the wikis moved to s5 - T184805 (duration: 00m 58s)
09:12 ema: Traffic: move restbase back to eqiad T203777
09:07 ema: Traffic: set services active/active T203777
09:00 ema: Traffic: route esams caches back to eqiad T203777
08:27 moritzm: installing fuse security updates
08:07 ariel@deploy1001: Finished deploy [dumps/dumps@0714a93]: fix adds/changes dumps generation when prev run is missing (duration: 00m 06s)
08:07 ariel@deploy1001: Started deploy [dumps/dumps@0714a93]: fix adds/changes dumps generation when prev run is missing
08:01 moritzm: rolling out debdeploy 0.0.99.6
07:51 elukey: cleaned up some log files from eventlog1002
02:55 ejegg: updated payments-wiki from 1472604b6e to 7fb1aae963
00:19 krinkle@deploy1001: Synchronized php-1.32.0-wmf.24/includes/utils/UIDGenerator.php: T94522 - I2a0c51bea58 (duration: 00m 56s)
00:15 krinkle@deploy1001: sync-file aborted: T205567 - I75f1eb6dc2cb (duration: 00m 01s)
00:14 krinkle@deploy1001: Synchronized php-1.32.0-wmf.24/tests/phpunit/includes/utils/: T94522 - I2a0c51bea58 (duration: 01m 02s)

2018-10-09

22:58 SMalyshev: repooled wdqs2003
22:26 shdubsh: repairing /dev/sdl1 on ms-be2040 - T199198
21:52 bblack: cp1085: varnish backend restart for mbox lag
21:50 mutante: releases1001 - restarted jenkins (it went from 200 -> 503 -> 403) curl localhost:8080 works again after restart, icinga check still getting 403 now
food: updated fundraising CiviCRM from 7a0d14015e to 1165e7ed79
20:08 mutante: repair /dev/sdg1 on ms-be2041 - T199198
19:37 XioNoX: disable igmp-snooping on asw2-c-eqiad - T201039
19:25 XioNoX: disable igmp-snooping on asw2-b-eqiad - T201039
19:20 XioNoX: bounce igmp-snooping on asw2-b-eqiad
18:24 ottomata: adding Accept header to all varnishkafka generated webrequest logs
17:21 SMalyshev: depooled wdq23 again, sigh
13:54 moritzm: rebooting prometheus1004 for kernel security update
13:41 moritzm: rebooting prometheus1003 for kernel security update
13:28 moritzm: rebooting prometheus2004 for kernel security update
13:13 moritzm: rebooting prometheus2003 for kernel security update
12:54 gehel: silencing wdqs-public lag alerts (service still functional, and SLO unclear) - T199228
12:45 moritzm: installing imagemagick security updates
11:47 END: (ERROR) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=2) (volans@neodymium)
11:47 START: - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (volans@neodymium)
11:45 akosiaris: dry-run services switchover from codfw to eqiad in preparation for Thursday
11:37 END: (ERROR) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=2) (volans@neodymium)
11:37 START: - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (volans@neodymium)
11:14 volans: live-test of the inverted switchdc (eqiad->codfw) completed, all good - T203777
11:14 END: (PASS) - Cookbook sre.switchdc.mediawiki.08-update-tendril (exit_code=0) (volans@neodymium)
11:13 START: - Cookbook sre.switchdc.mediawiki.08-update-tendril (volans@neodymium)
11:12 END: (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) (volans@neodymium)
11:11 START: - Cookbook sre.switchdc.mediawiki.08-start-maintenance (volans@neodymium)
11:11 END: (PASS) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=0) (volans@neodymium)
11:11 START: - Cookbook sre.switchdc.mediawiki.08-restore-ttl (volans@neodymium)
11:11 END: (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) (volans@neodymium)
11:11 [DRY-RUN]: MediaWiki read-only period ends at: 2018-10-09 11:11:05.042622 (volans@neodymium)
11:11 START: - Cookbook sre.switchdc.mediawiki.07-set-readwrite (volans@neodymium)
11:08 END: (PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0) (volans@neodymium)
11:08 START: - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (volans@neodymium)
11:07 END: (PASS) - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions (exit_code=0) (volans@neodymium)
11:07 START: - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions (volans@neodymium)
11:06 END: (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-traffic (exit_code=0) (volans@neodymium)
11:04 START: - Cookbook sre.switchdc.mediawiki.04-switch-traffic (volans@neodymium)
11:03 END: (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0) (volans@neodymium)
11:03 START: - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (volans@neodymium)
11:00 END: (PASS) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=0) (volans@neodymium)
10:59 START: - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (volans@neodymium)
10:56 END: (PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0) (volans@neodymium)
10:56 [DRY-RUN]: MediaWiki read-only period starts at: 2018-10-09 10:56:12.213026 (volans@neodymium)
10:56 START: - Cookbook sre.switchdc.mediawiki.02-set-readonly (volans@neodymium)
10:53 END: (PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0) (volans@neodymium)
10:53 START: - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (volans@neodymium)
10:51 END: (PASS) - Cookbook sre.switchdc.mediawiki.00-warmup-caches (exit_code=0) (volans@neodymium)
10:49 onimisionipe: repooling wdqs2001 catched up on lag - T206423
10:48 START: - Cookbook sre.switchdc.mediawiki.00-warmup-caches (volans@neodymium)
10:47 END: (PASS) - Cookbook sre.switchdc.mediawiki.00-warmup-caches (exit_code=0) (volans@neodymium)
10:41 START: - Cookbook sre.switchdc.mediawiki.00-warmup-caches (volans@neodymium)
10:40 END: (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) (volans@neodymium)
10:40 START: - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (volans@neodymium)
10:37 END: (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) (volans@neodymium)
10:36 START: - Cookbook sre.switchdc.mediawiki.00-disable-puppet (volans@neodymium)
10:35 onimisionipe: deploying prometheus-blazegraph-exporter 0.6 on all wdqs clusters - T206123
10:34 volans: about to perform live-test of the inverted switchdc (eqiad->codfw), actions will be real but basically noop due to codfw being already active - T203777
09:25 elukey: swapped Hadoop's hive/oozie from analytics1003 to an-coord1001
09:16 ema: restart pybal on lvs1005 to pick up config changes (conf2001 -> conf1004)
09:00 ema: re-enable puppet/pybal on lvs1002, IPv6 connectivity with phab1001 working again T201039
08:16 elukey: update puppet compiler facts
08:06 onimisionipe: depooling wdqs2001 to catch up on lag -T206423
07:03 akosiaris: restart zuul and zuul-merger on contint1001 for the upgrade of zuul to finish
06:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1122 (duration: 00m 57s)
05:19 marostegui: Stop MySQL on db1122 for binlog format change, mysql and kernel upgrade
05:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1122 (duration: 00m 59s)
02:41 krinkle@deploy1001: Synchronized wmf-config/profiler.php: T176916 / T206092 - Ie86e88777c48 (duration: 00m 56s)
02:21 krinkle@deploy1001: Synchronized wmf-config/arclamp.php: T176916 - Id79baae90: ensure file exists before Ie86e88777c48 (duration: 00m 57s)
00:04 krinkle@deploy1001: Synchronized php-1.32.0-wmf.24/includes/libs/rdbms/database: T201900 - I8ae754a2518 (duration: 00m 59s)

2018-10-08

22:45 XioNoX: increase accepted-prefix-limit for 24115 on cr4-ulsfo
22:41 XioNoX: clear BGP neighbor cr1-eqsin:AS9583 (bgp limit threshold reached)
21:11 ejegg: updated payments-wiki from d623de9494 to 1472604b6e
20:42 gehel: repooling wdqs2003 catched up on lag - T206423
19:41 XioNoX: troubleshooting asw2-b-eqid with JTAC - T201039
19:08 gehel: depooling wdqs2003 to catch up on lag -T206423
19:00 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable MCR read-new mode on some small wikis (T198308) (duration: 00m 56s)
18:55 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@bd698bd]: WDQS deployment - New federation whitelist entries (duration: 10m 07s)
18:45 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@bd698bd]: WDQS deployment - New federation whitelist entries
18:37 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@bd698bd]: WDQS test deployment - New federation whitelist entries(wdqs1009) (duration: 00m 33s)
18:37 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@bd698bd]: WDQS test deployment - New federation whitelist entries(wdqs1009)
18:36 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Extension:File exporter to mrwikipedia (T206437) (duration: 00m 57s)
16:29 XioNoX: push firewall filter counters on asw2-b-eqiad - T201039
16:28 elukey: restart eventlogging on eventlog1002 for python security upgrades
14:05 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T184805: Revert 'mariadb: Depool db1110 for testing s3 imports' (duration: 00m 57s)
14:03 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T184805: Revert 'mariadb: Depool db1110 for testing s3 imports' (duration: 00m 56s)
13:43 elukey: restart confd on esams nodes to pick up new srv settings
13:41 elukey: restart navtiming.service on webperf1001 to pick up the dns change for etcd
13:39 marostegui: Enable gtid on the following slaves: db2068 db1122 db1117:3323
13:37 elukey: restart confd on all the other eqiad nodes to pick up new srv records
13:32 elukey: restart confd on cp1* to pick up new srv records
13:11 _joe_: purging the dnsrec cache for eqiad,esams etcd client SRV records
13:09 ema: depool eqiad front-edge traffic T201039
13:05 banyek: converting cebwiki.templatelinks to TokuDB on host dbstore1002.eqiad.wmnet (T205544)
13:04 banyek: downtime notifications for dbstore1002 repliaction threads (T205544)
12:49 banyek: pt-kill-wmf enabled on the wikireplicas (T203674)
11:59 _joe_: restart pybal in esams, after running puppet, to switch etcd cluster used
11:46 _joe_: restart pybal on lvs1001
11:46 addshore: SWAT done
11:45 addshore@deploy1001: Synchronized wmf-config/throttle.php: Add throttle exception for Netherlands Hackathon October 2018 - Wiki Techstorm T206241, and remove other rules. (duration: 00m 56s)
11:39 addshore: addshore@mwmaint2001:~$ mwscript namespaceDupes.php --wiki fywiktionary --fix --add-prefix=T202769 # T202769
11:35 addshore: addshore@mwmaint2001:~$ mwscript namespaceDupes.php --wiki fywiktionary --fix # Finished, still 111 pages to fix
11:34 addshore: addshore@mwmaint2001:~$ mwscript namespaceDupes.php --wiki fywiktionary --fix # Started
11:33 addshore: addshore@mwmaint2001:~$ mwscript namespaceDupes.php --wiki fywiktionary # (dryrun, 11529 links to fix, 11529 were resolvable.)
11:32 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: gerrit:455249 Use translated MetaNamespace for fy.wiktionary T202769 (duration: 00m 58s)
11:27 addshore@deploy1001: Synchronized wmf-config/flaggedrevs.php: SWAT: gerrit:464890 Remove the "reviewer" group at ruwikisource T205997 (duration: 00m 57s)
10:41 elukey: restart mcrouter on mw2201 with more verbose logging settings as test
09:55 moritzm: installing python3.5/python2.7 security updates
09:51 godog: rebuild sdc sdh sdj sdi on ms-be2041 with crc=1 finobt=0 - T199198
08:20 marostegui: Disable gtid on es2 and es3 eqiad master
08:20 gehel@puppetmaster1001: conftool action : set/weight=15; selector: dc=codfw,cluster=wdqs,name=wdqs2001.codfw.wmnet
08:20 gehel@puppetmaster1001: conftool action : set/weight=15; selector: dc=codfw,cluster=wdqs,name=wdqs2002.codfw.wmnet
07:50 marostegui: Enabling replication eqiad -> codfw in preparation for DC failover
07:40 marostegui: Disable GTID on s1,s2,s3,s4,s6,s7,s8 eqiad masters in preparation for enabling replication eqiad -> codfw
07:39 _joe_: disabling puppet, doing etcd tests on lvs1006
07:38 gehel@puppetmaster1001: conftool action : set/weight=15; selector: dc=codfw,cluster=wdqs,name=wdqs2002.eqiad.wmnet
07:38 gehel@puppetmaster1001: conftool action : set/weight=15; selector: dc=codfw,cluster=wdqs,name=wdqs2001.eqiad.wmnet
07:38 gehel: reducing relative weight of wdqs2003 in pybal - T206423
07:27 banyek: enabling first time wmf-pt-kill on labsdb1010
07:20 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1092 with low weight - T205514 (duration: 01m 27s)
07:00 moritzm: installing git security updates

2018-10-07

16:40 dereckson: Reset user email for account "Dominic Mayers" (T206421)
16:35 elukey: run a script in tmux (my username) on mw2201 to poll the status of a mcrouter key/route every 10s using its admin api (very lightweight but kill if needed)
14:52 onimisionipe: repooling wdqs2003. Catched up on Lag and also Lag issues seems to be creeping on wdqs200[1|2]
04:29 SMalyshev: temp depooled wdqs2003
03:12 ejegg: disabled all fundraising scheduled jobs - something that looks like disk issues on civi1001

2018-10-06

21:20 gehel: repooling wdqs2003: catched up on updater lag
20:43 _joe_: restarting apache2 on puppetmaster1001
19:16 onimisionipe: depooling wdqs2003
18:10 elukey: restart Yarn Resource Manager on an-master1002 to force an-master1001 to take the active role back (failed over due to a zk conn issue)
17:07 onimisionipe: restarting wdqs-blazegraph on wdqs2003
13:48 bblack: multatuli: update gdnsd package to 2.99.9930-beta-1+wmf1
13:47 bblack: authdns1001: update gdnsd package to 2.99.9930-beta-1+wmf1 (correction to last msg)
13:46 bblack: authdns1001: update gdnsd package to 2.99.9161-beta-1+wmf1
12:57 bblack: rebooting cp1076
12:49 bblack: depool cp1076, apparently has disk issues

2018-10-05

23:50 bblack: <<<<<<< repooling eqiad edge caches, a few days ahead of intended switchback next Weds, to alleviate some traffic engineering concerns over the weekend >>>>>>
20:48 mutante: T191183 - it's still showing the error page as before but that isn't due to apache issues, it just needs additional ferm rules
20:44 mutante: gerrit - adding gerrit.wmfusercontent.org virtual host for avatars. applied first on gerrit2001, then on cobalt (T191183)
20:03 ejegg: updated fundraising CiviCRM from ebc2e0076c to 7a0d14015e
19:48 banyek: repooling labsdb1009 (T195747)
19:44 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@f8776de]: Redeploy 1009 (duration: 00m 26s)
19:44 smalyshev@deploy1001: Started deploy [wdqs/wdqs@f8776de]: Redeploy 1009
18:37 bblack: authdns2001: upgraded gdnsd to 2.99.9930-beta
18:31 bblack: gdnsd-2.99.9930-beta-1+wmf1 uploaded to stretch-wikimedia
18:26 mutante: icinga - noop on all servers, no change, puppet re-enabled, operations normal
18:08 mutante: disabling puppet on icinga for 5 min for extra safety before a change that should be noop
17:58 banyek: depooling labsdb1009 (T195747)
17:50 banyek: repooling labsdb1011 (T195747)
17:12 elukey: set etcd in codfw as read/write (was readonly) and eqiad as readonly (was read/write)
14:57 banyek: depooling labsdb1011 (T195747)
14:56 banyek: depooling labsdb1011
13:26 banyek: adding wmf-pt-kill_2.2.20-1+wmf3 package for stretch
13:25 moritzm: installing python3.5/2.7 security updates
13:02 volans: upgraded spicerack to version 0.0.9 on sarin/neodymium/cumin* - T199079
12:13 vgutierrez: Creating certcentral1001.eqiad.wmnet in ganeti - T206308
12:12 vgutierrez: Creating certcentral2001.codfw.wmnet in ganeti - T206308
11:59 elukey: deleted bohrium from ganeti via gnt-instance
11:43 moritzm: rebooting wezen for kernel security update
11:29 moritzm: rebooting ruthenium for kernel security update
10:40 jynus: restarting replication on labsdb1010/1 on s3 and s5
10:37 volans: uploaded spicerack_0.0.9-1{,+deb9u1} to apt.wikimedia.org {jessie,stretch}-wikimedia - T199079
10:17 moritzm: rearmed keyholder on netmon2001
10:10 elukey: restart confd on labs-puppetmaster to pick up new etcd settings (eqiad -> codfw)
10:03 _joe_: restarting navtiming.service on webperf1001 to pick up the dns change for etcd
09:37 elukey: restart rsyslog on lithium - broken connection to tegmen - T199406
09:37 banyek: disabling puppet on labsdb1009,labsdb1010,labsdb1011 (T203674)
09:36 banyek: adding wmf-pt-kill_2.2.20-1+wmf2 package for stretch
09:16 volans: rebooting tegmen, console stuck, possible re-occurrence of T199413 (to be confirmed)
09:12 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Move some wikis for s3 to s5 (duration: 00m 56s)
09:06 elukey: stop etcdmirror replication on conf2002
09:05 _joe_: restarting confd on all nodes in eqiad and esams
08:58 _joe_: wiped cached values for the read-only etcd SRV record
08:56 _joe_: read-write connections to etcd only go to codfw now
08:35 _joe_: reenabling notifications for etcdmirror on conf1005
08:02 jynus: start replication on db1069 (x1)
07:54 jynus: starting replicatios on db1075; db1070, db1070:s3 with disabled gtid
07:50 jynus: stopping dbstore1001:x1
07:33 jynus: chaning s3 master for db1070
07:28 jynus: stopping s3 replication on db1070
07:20 jynus: stopping x1 replication on db1069
07:20 godog: temporarily stop prometheus on bast4001 to finalize data transfer - T179050
07:19 jynus: stopping s3 replication on db1075
07:18 jynus: stopping s5 replication on db1070
07:09 moritzm: installing python3.4/2.7 security updates
05:55 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T205599 - Ic28e00c30 (duration: 00m 57s)
05:53 _joe_: upgrading python-etcd on conf1004-6, restarting etcdmirror
05:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Clarify db1092 status - T205514 (duration: 00m 57s)
04:18 krinkle@deploy1001: Synchronized php-1.32.0-wmf.24/includes/libs/filebackend/FileBackendStore.php: T205567 - I75f1eb6dc2cb (duration: 00m 56s)
04:16 krinkle@deploy1001: Synchronized php-1.32.0-wmf.24/extensions/CirrusSearch/includes/DataSender.php: I0769c50c (duration: 01m 01s)
00:31 mutante: LDAP: added user skvjold to group wmf (T204377)

2018-10-04

22:51 ejegg: updated fundraising CiviCRM from 944b954bac to ebc2e0076c
21:27 XioNoX: bounce phab1001 switch port - T201039
20:47 ejegg: updated fundraising CiviCRM from ddf4865650 to 944b954bac
20:23 mforns@deploy1001: Finished deploy [analytics/refinery@3eb9bf2]: deploying refinery together with refinery-source v0.0.76 (duration: 00m 17s)
20:22 mforns@deploy1001: Started deploy [analytics/refinery@3eb9bf2]: deploying refinery together with refinery-source v0.0.76
20:10 mforns@deploy1001: Finished deploy [analytics/refinery@3eb9bf2]: deploying refinery together with refinery-source v0.0.76 (duration: 14m 04s)
19:56 mforns@deploy1001: Started deploy [analytics/refinery@3eb9bf2]: deploying refinery together with refinery-source v0.0.76
19:30 marxarelli: rise in fatals "Fatal error: entire web request took longer than 60 seconds and timed out in /srv/mediawiki/php-1.32.0-wmf.24/includes/Title.php"
19:26 dduvall@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.24
19:15 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@6dc89c0]: Bump cirrusSearchLinksUpdate concurrency to 50 (duration: 00m 53s)
19:14 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@6dc89c0]: Bump cirrusSearchLinksUpdate concurrency to 50
18:49 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:460202|]] (duration: 00m 59s)
18:24 XioNoX: bounce lvs1002:eth1 switch port
18:23 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable PageTriage/ORES on enwiki (T206149) (duration: 01m 01s)
18:21 bblack: lvs1002: puppet disabled, stopping pybal (fail to 1005)
18:07 _joe_: disabled notifications for etcd replication lag on conf1005, not in production
17:47 banyek: repooling labsb1010 (T195747)
17:41 _joe_: uploaded new python-etcd packages for jessie, stretch
17:38 XioNoX: asw2-b-eqiad recabling done - T201039
17:34 elukey: pool kafka1002 (eventbus) after maintenance
17:22 elukey: re-enable ircecho after alarms shower
17:15 andrewbogott: triggering some alerts on labvirt1018 to figure out about alert thresholds
17:06 elukey: stop ircecho on einstenium - alarms shower
17:02 gtirloni: tools - published updated toollabs-* Docker images
16:54 ejegg: updated standalone SmashPig deploy from 82f9d49c23 to 5f21d3f2db
16:52 XioNoX: Step 3) Add missing links - T201039
16:45 shdubsh: etherpad1001 running systemctl reset-failed
16:41 XioNoX: Connect/enable fpc2:0/51-fpc5:1/0 (5m DAC) - T201039
16:39 XioNoX: Enable fpc5-fpc7 - T201039
16:33 twentyafterfour: started phd on phab1001 and re-enabled puppet (I had it disabled to prevent starting phd during read-only)
16:25 twentyafterfour: phabricator is read-write
16:21 jynus: reloading dbproxy1003,8
16:16 marostegui: Stop and reboot db1072 (phabricator master) for maintenance
16:16 twentyafterfour: phabricator is read-only
16:14 XioNoX: Enable all VC ports on FPC2 and FPC7 - T201039
16:13 XioNoX: starting asw2-b-eqiad re-cabling - T201039
16:08 twentyafterfour: logged downtime for phabricator in icinga, stopped phd queue processing in preparation for read-only mode
16:07 jynus: reloading haproxy @ dbproxy1005
16:00 marostegui: Stop MySQL on db1073 for mariadb and kernel upgrade - T201039 T148507
15:58 arturo: icinga downtime every server in the main cloudvps deployment for 2h T201039
15:56 arturo: icinga downtime every server with the cloudXXXX scheme for 2h T201039
15:54 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@55dbb8b]: Proper reconnect on topics change T199444 (duration: 00m 55s)
15:53 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@55dbb8b]: Proper reconnect on topics change T199444
15:52 ppchelko@deploy1001: Finished deploy [changeprop/deploy@5d00448]: Proper reconnect on topics change T199444 (duration: 01m 40s)
15:51 ppchelko@deploy1001: Started deploy [changeprop/deploy@5d00448]: Proper reconnect on topics change T199444
15:41 elukey: depool kafka1002 from eventbus as precautionary step for T201039
14:48 banyek: depooling labsb1010 (T195747)
14:09 marostegui: Sanitize enwikivoyage cebwiki shwiki srwiki mgwiktionary on db1124:3315 T184805
13:46 pmiazga@deploy1001: Finished deploy [proton/deploy@ecb9a0e]: Bugfix:handle undefined response and fix grafana stats (T186748,T201158) (duration: 02m 55s)
13:43 pmiazga@deploy1001: Started deploy [proton/deploy@ecb9a0e]: Bugfix:handle undefined response and fix grafana stats (T186748,T201158)
13:14 banyek: muting alerts on s2replication @dbstore2002 and resuming compression of s2 database tables (T204930)
13:14 banyek: muting alerts on dbstore2002 and resuming compression of s2 database tables (T204930)
12:23 elukey: deploy etcdmirror on conf1005 - T205814
12:06 zeljkof: EU SWAT finished
12:06 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add permission "move-rootuserpages" to usergroup "eliminator" at ptwiki (T205595) (duration: 00m 57s)
12:01 moritzm: rolling reboot of ms-fe hosts in codfw for kernel security update
12:00 zeljkof: one more patch for EU SWAT
11:57 zeljkof: EU SWAT finished
11:57 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add *.nasimonline.ir to wgCopyUploadsDomains whitelist for Commons (T203371) (duration: 00m 56s)
11:52 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: add Radlines.org to $wgCopyUploadsDomains (T203219) (duration: 00m 57s)
11:42 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add .bollywoodhungama.in to wgCopyUploadsDomains (T203363) (duration: 00m 57s)
11:35 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add some namespaces aliases for zhwikiversity (T201675) (duration: 00m 57s)
11:27 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Change acewiki default time zone to Asia/Jakarta (T205693) (duration: 00m 56s)
11:17 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create Photowalk and Photowalk Talk namespaces for bd.wikimedia.org (T205747) (duration: 00m 57s)
10:44 twentyafterfour@deploy1001: Synchronized php-1.32.0-wmf.23/README: noop sync to verify that scap 3.8.7-1 works (at least on a basic level) (duration: 00m 59s)
10:38 godog: upload scap 3.8.7-1 - T204383
10:36 _joe_: uploading etcd-mirror to stretch-wikimedia T205814
10:08 moritzm: rolling reboot of ms-fe hosts in eqiad for kernel security update
09:13 arturo: T203177 schedule 8h icinga downtime for cloudcontrol1003,1004 and labmon1001
08:52 moritzm: installing python2.7/python3.4/python3.5 security updates on jessie/stretch
08:34 moritzm: installing ca-certificates updates for jessie/stretch
08:09 marostegui: Restart icinga T196336
08:00 gehel: re-enabling puppet on maps1004
07:31 elukey: move Piwik/Matomo from bohrium to matomo1001 - T202962
07:25 godog: reformat ms-be1041 with crc=1 finobt=0 - T199198
06:57 jynus: starting multisource replication of s3 from s5 at eqiad master
06:51 jynus: reenabling consistency configuration on s5 replica databases
06:24 jynus: create manual backup of databases on eqiad s6, s7, s8, x1
05:36 marostegui: Deploy schema change on db2048 (s1 master) - T205913
05:35 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2062 (duration: 00m 56s)
05:30 marostegui: Deploy schema change on db2062 - T205913
05:30 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2062 (duration: 00m 57s)
04:04 SMalyshev: repooled wdqs2003
03:22 SMalyshev: depool wdqs2003 to let it catch up
03:21 SMalyshev: repooled wdqs2001
03:16 ejegg: re-enabled PayPal EC orphan rectifier
03:06 ejegg: updated CiviCRM from 80cb98e33e to ddf4865650
02:43 SMalyshev: depooled wdqs2001 to see if it catches up faster
01:54 ejegg: updated payments-wiki from 8b673cfb4f to d623de9494

2018-10-03

23:54 mutante: scheduled downtime for wdqs as it's flapping and already known
23:45 catrope@deploy1001: Synchronized php-1.32.0-wmf.23/extensions/VisualEditor/: Require Parsoid HTML 2.0.0, and handle its <audio> tags (T201081); ext.visualEditor.mwlanguage: Actually load all of the code (T205834) (duration: 00m 57s)
23:41 catrope@deploy1001: Synchronized php-1.32.0-wmf.24/extensions/VisualEditor/: Require Parsoid HTML 2.0.0, and handle its <audio> tags (T201081) (duration: 00m 59s)
23:29 catrope@deploy1001: Synchronized php-1.32.0-wmf.23/extensions/PageTriage/: Hide copyvio AFC filter option behind flag (T205918) (duration: 00m 57s)
23:23 catrope@deploy1001: Synchronized php-1.32.0-wmf.24/includes/utils/UIDGenerator.php: Make UID clock drift error have more details (T94522) (duration: 00m 58s)
23:20 XenoRyet: shut off Paypal orphan rectifier
23:12 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Bump Minerva A/B test rates to 100% on jawiki, ruwiki, fawiki (T200792) (duration: 00m 56s)
22:49 shdubsh: re-enable puppet on einsteinium
22:45 shdubsh: einsteinium: setting enable_notifications=1 and reloading icinga
22:36 herron: herron@neodymium:~$ sudo cumin -b 15 -p 95 '*' 'run-puppet-agent -q --failed-only'
22:20 shdubsh: einsteinium: setting enable_notifications=0 and starting icinga
22:06 herron: herron@neodymium:~$ sudo cumin -b 40 -p 95 'R:file = /etc/nagios/nrpe_local.cfg' run-puppet-agent
22:02 mutante: mw2242 - started nagios-nrpe-server
22:01 shdubsh: icinga stopped manually
21:57 mutante: einstienium - disabling puppet
21:25 bblack: upgraded gdnsd to 2.99.9161 on authdns1001
21:17 dduvall@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.24 (duration: 00m 55s)
21:16 dduvall@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.24
21:12 dduvall@deploy1001: Synchronized php-1.32.0-wmf.24/extensions/WikibaseQualityConstraints/src/ServiceWiring.php: deploying fix to 1.32.0-wmf.24 for T206161 (duration: 00m 57s)
20:28 marxarelli: deployed proposed WikibaseQualityConstraints fix and wikiversions bump for wikidatawiki to mwdebug1001 and mwdebug1002 for verification (T206161)
20:18 robh: optic swap on cr4-ulsfo:et-0/0/1
20:03 bblack: upgraded gdnsd to 2.99.9161 on multatuli
19:40 bblack: upgraded gdnsd to 2.99.9161 on authdns2001
19:35 bblack: uploaded 2.99.9161-beta-1+wmf1 to stretch-wikimedia
19:33 mateusbs17: running initial osm import in maps1004
19:23 dduvall@deploy1001: Synchronized php: rollback group1 to 1.32.0-wmf.23 (duration: 00m 54s)
19:18 dduvall@deploy1001: rebuilt and synchronized wikiversions files: rollback group1 to 1.32.0-wmf.23
19:15 marxarelli: rolling back group1 after rapid rise in fatals
19:14 dduvall@deploy1001: scap failed: average error rate on 6/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details)
18:49 RoanKattouw: Deployed patches for T206130
18:36 papaul: reinstalling OS on lvs2010
18:16 mutante: lvs2010 - schduled downtime for host and services for 12 hours for reinstall
18:09 mutante: lvs2009 - schedule downtime in icinga for 4 hours, reinstall in progress
18:08 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@d5bab41]: Bump cirrusSearchLinksUpdate concurrency to 20 (duration: 00m 57s)
18:07 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@d5bab41]: Bump cirrusSearchLinksUpdate concurrency to 20
18:07 XioNoX: disable ulsfo Zayo transit/transport links
17:42 XioNoX: re-enable cr1-eqiad:ae1 - T201145
17:28 XioNoX: start of recabling asw2-a-eqiad between asw and cr1 - T201145
17:26 XioNoX: disable cr1-eqiad:ae1 - T201145
17:10 papaul: reinstalling OS on lvs2009
16:24 reedy@deploy1001: Synchronized php-1.32.0-wmf.24/extensions/Flow/: fixup flow exporting T203424 (duration: 01m 03s)
15:45 ejegg: updated fundraising CiviCRM from e3e1963915 to 80cb98e33e
14:42 jynus: fixed some prometheus metrics grants on dbstore1001:3306, db1116:3317 and db1116:3318
14:07 banyek: converting wikidatawiki.change_tag to TokuDB on host dbstrore1002 (T205544)
12:54 urandom: DROP unused RESTBase tables - T204752
12:26 stephanebisson: Finished mwscript extensions/ORES/maintenance/BackfillPageTriageQueue.php --wiki enwiki (T203286)
12:12 stephanebisson: Starting mwscript extensions/ORES/maintenance/BackfillPageTriageQueue.php --wiki enwiki (T203286)
11:54 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Don't purge articlequality, draftquality scores (T203286) (duration: 00m 57s)
11:45 banyek: converting enwiki.slots to TokuDB on host dbstrore1002 (T205544)
11:42 pmiazga@deploy1001: Synchronized wmf-config: SWAT: Remove dead config relating to wgRelatedArticlesEnabledBucketSize (T202306) (duration: 00m 57s)
11:38 arturo: downtime cloudcontrol1003,1004 for 2h for T203177
11:30 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create eliminator group at Vietnamese Wikibooks (T202207) (duration: 00m 58s)
11:25 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix a typo in zhwikiversitys importsources definition (T201328) (duration: 00m 57s)
11:20 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: Fix a typo in lift account creation cap for cswiki event (T206119) (duration: 00m 56s)
10:41 jynus: start compressing dbstore1001:x1 tables
09:26 jynus: reducing io overhead temporarilly in exchange for crash safety for s5 replicas T184805
09:23 jynus: fixing replication filters on dbstore1002 (again)
08:34 jynus: fixing replication filters on dbstore1002
08:18 jynus: starting importing of certain s3 wikis into eqiad s5 master T184805
07:51 jynus: deploying replication filtes to s5 at labsdb1009/10/11 and dbstore1002 T184805
07:06 mholloway-shell@deploy1001: Finished deploy [kartotherian/deploy@27062b4] (maps1004): Specify WDQS endpoint at wdqs.discovery.wmnet in the service config (T205607) (duration: 00m 28s)
07:05 mholloway-shell@deploy1001: Started deploy [kartotherian/deploy@27062b4] (maps1004): Specify WDQS endpoint at wdqs.discovery.wmnet in the service config (T205607)
06:42 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2055 (duration: 00m 55s)
06:37 marostegui: Deploy schema change on db2055 - T205913
06:37 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2055 (duration: 00m 56s)
06:03 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2085:3311 (duration: 00m 56s)
05:59 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@e1aab7b]: Request Parsoid HTML version 2.0.0 (0866a07) (duration: 03m 32s)
05:57 marostegui: Deploy schema change on db2085:3311 - T205913
05:56 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@e1aab7b]: Request Parsoid HTML version 2.0.0 (0866a07)
05:55 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2085:3311 (duration: 00m 58s)
05:26 marostegui: Deploy schema change on db1067 (s1 eqiad master), lag will be generated - T205913
05:25 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2070 (duration: 00m 57s)
05:24 krinkle@deploy1001: Synchronized php-1.32.0-wmf.24/languages/Language.php: T206030 - I985dfa3eb17 (duration: 00m 56s)
05:21 marostegui: Deploy schema change on db1075 (s3 eqiad master), lag will be generated - T205913
05:20 marostegui: Deploy schema change on db2070 - T205913
05:20 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2070 (duration: 00m 56s)
04:45 krinkle@deploy1001: Synchronized php-1.32.0-wmf.24/extensions/NavigationTiming: T205580 - I04c52658fbf6d (duration: 01m 03s)
00:42 Amir1: Evening SWAT is done
00:41 ladsgroup@deploy1001: Synchronized php-1.32.0-wmf.23/extensions/GlobalPreferences/resources/ext.GlobalPreferences.global.ooui.js: SWAT: Fail gracefully if we failed to find associated widget (T205991) (duration: 00m 57s)
00:38 mutante: icinga1001 (not prod yet), removing all icinga packages, running puppet to reinstall them, debugging dpkg issue
00:19 ladsgroup@deploy1001: Synchronized php-1.32.0-wmf.24/extensions/GlobalPreferences/resources/ext.GlobalPreferences.global.ooui.js: SWAT: Fail gracefully if we failed to find associated widget (T205991) (duration: 00m 55s)

2018-10-02

23:54 ladsgroup@deploy1001: Synchronized php-1.32.0-wmf.24/extensions/PageTriage/i18n/en.json: SWAT: Align copyvio log terminology (T199359) (duration: 00m 56s)
23:38 ladsgroup@deploy1001: Synchronized php-1.32.0-wmf.24/extensions/PageTriage/modules/ext.pageTriage.views.list/ext.pageTriage.listControlNav.underscore: SWAT: Hide copyvio, none afc filter options behind flag (T205918) (duration: 00m 56s)
23:33 ejegg: updated fundraising CiviCRM from c353eba283 to e3e1963915
23:26 ladsgroup@deploy1001: Synchronized php-1.32.0-wmf.24/extensions/ORES/tests/phpunit/includes/HooksTest.php: SWAT: Disable RCFilters in tests (duration: 00m 54s)
23:16 ladsgroup@deploy1001: Synchronized php-1.32.0-wmf.23/extensions/FlaggedRevs/frontend/specialpages/reports/ProblemChanges_body.php: SWAT: Fix using the old index when new indexes are not there (T205904) (duration: 00m 57s)
22:53 shdubsh: powercycling icinga1001 after removing problematic entry from fstab
22:26 gtirloni: labstore2003 re-started service block_sync
21:39 XioNoX: Fix unused vlans XLink1/2 on asw2-a5
21:15 banyek: enabling puppet on es2001
21:12 banyek: re-enabling and starting backups on host es2001 (TT205257)
21:01 gtirloni: labstore2003 stopped service block_sync
20:15 dduvall@deploy1001: Finished scap: group0 to php-1.32.0-wmf.24 (duration: 33m 00s)
20:04 Jeff_Green: authdns-update to deploy new IP for frbast2001.frack.eqiad.wmnet
19:50 XioNoX: update prefix-list fundraising-codfw-internal4 to /24 on pfw3-codfw - T204271
19:42 dduvall@deploy1001: Started scap: group0 to php-1.32.0-wmf.24
19:36 dduvall@deploy1001: Pruned MediaWiki: 1.32.0-wmf.19 (duration: 07m 25s)
19:21 XioNoX: update fw policies on pfw3-eqiad - T204271
19:19 XioNoX: update fw policies on pfw3-codfw - T204271
18:39 XioNoX: replace 10.195.0.73/29 with 10.195.0.65/28 on pfw3-codfw - T204271
18:26 XioNoX: remove old 10.195.0.65/29 from pfw3-codfw - T204271
18:24 jynus: restarting ferm on dbstore2002 T205257
18:08 arlolra: Updated Parsoid to 65d6f82 (T163438, T205674, T205673)
18:07 ariel@deploy1001: Finished deploy [dumps/dumps@a9570fb]: fix incr dumps multiversion conf setting (duration: 00m 06s)
18:07 ariel@deploy1001: Started deploy [dumps/dumps@a9570fb]: fix incr dumps multiversion conf setting
18:01 arlolra@deploy1001: Finished deploy [parsoid/deploy@19053a3]: Updating Parsoid to 65d6f82 (duration: 10m 44s)
17:51 arlolra@deploy1001: Started deploy [parsoid/deploy@19053a3]: Updating Parsoid to 65d6f82
17:37 XioNoX: update NAT for frbast2001 on pfw3-codfw - T204271
17:25 XioNoX: update fw policies on pfw3-eqiad - T204271
17:22 XioNoX: update fw policies on pfw3-codfw - T204271
17:22 andrewbogott: upgraded wikitech-static to remotes/origin/REL1_31
17:18 andrewbogott: upgrading debian packages and MediaWiki version on wikitech-static
16:53 jynus: setup test s3 replication channel on db1110 (filtered)
16:49 XioNoX: assign 10.195.0.129/29 to pfw3-codfw:reth0.2133 - T204271
16:38 cmjohnson1: swapping failed disk db1067 T205780
16:04 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@093551f]: Increase cirrusSearchLinksUpdate concurrency (duration: 01m 06s)
16:03 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@093551f]: Increase cirrusSearchLinksUpdate concurrency
15:50 marxarelli: cutting 1.32.0-wmf.24 branch
15:33 gehel: cleanup old cronjob (cleanup GC logs) on all elasticsearch servers
15:24 akosiaris: upgrade mathoid chart version to 0.0.11
15:24 akosiaris@deploy1001: scap-helm mathoid finished
15:23 akosiaris@deploy1001: scap-helm mathoid cluster codfw completed
15:23 akosiaris@deploy1001: scap-helm mathoid cluster eqiad completed
15:23 akosiaris@deploy1001: scap-helm mathoid upgrade production stable/mathoid [namespace: mathoid, clusters: eqiad,codfw]
15:21 akosiaris@deploy1001: scap-helm mathoid finished
15:21 akosiaris@deploy1001: scap-helm mathoid cluster codfw completed
15:21 akosiaris@deploy1001: scap-helm mathoid cluster eqiad completed
15:21 akosiaris@deploy1001: scap-helm mathoid upgrade -h [namespace: mathoid, clusters: eqiad,codfw]
14:11 banyek: powering off dbstore2002.codfw.wmnet for BBU change (T205257)
13:47 marostegui: Deploy schema change on s4 eqiad, this will generate lag on eqiad - T205913
13:06 marostegui: Deploy schema change on s7 eqiad, this will generate lag on eqiad - T205913
12:47 banyek: converting enwiki.content to TokuDB on host dbstrore1002 (T205544)
12:47 banyek: converting enwiki.contents to TokuDB on host dbstrore1002 (T205544)
11:58 banyek: converting wikidatawiki.slots to TokuDB on host dbstrore1002 (T205544)
11:41 arturo: downtime labstore1007 load check in icinga for 1d
11:21 zeljkof: EU SWAT finished
11:19 ladsgroup@deploy1001: Synchronized php-1.32.0-wmf.23/extensions/FlaggedRevs/frontend/specialpages/reports/ProblemChanges_body.php: SWAT: Use proper index on change_tag table (T205904) (duration: 00m 57s)
10:58 mobrovac@deploy1001: Synchronized rpc/RunSingleJob.php: RunSingleJob: Delay job execution while in read-only mode - T204154 (duration: 00m 57s)
10:34 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2092 (duration: 00m 56s)
10:24 marostegui: Deploy schema change on db2092 - T203709
10:24 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2092 (duration: 00m 56s)
09:30 marostegui: Deploy schema change on s2 eqiad master, lag will be generated T205913
08:43 banyek: disabling puppet on es2001 and disabling backups too
08:28 marostegui: Deploy schema change on s6 eqiad master, lag will be generated T205913
08:16 jynus: test recover some s3 wiki data onto db1110 (s5)
08:04 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1110 (duration: 00m 56s)
08:04 marostegui: Deploy schema change on s5 eqiad master, lag will be generated T205913
08:01 banyek: converting wikidatawiki.content to TokuDB on host dbstrore1002 (T205544)
07:54 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2071 (duration: 00m 55s)
07:50 marostegui: Deploy schema change on db2071 T205913
07:50 mholloway-shell@deploy1001: Finished deploy [tilerator/deploy@6c80537] (maps1004): Disable event logging requests and remove HTTP proxy (duration: 00m 17s)
07:49 mholloway-shell@deploy1001: Started deploy [tilerator/deploy@6c80537] (maps1004): Disable event logging requests and remove HTTP proxy
07:49 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2071 (duration: 00m 56s)
07:48 mholloway-shell@deploy1001: Finished deploy [kartotherian/deploy@0bf513a] (maps1004): Remove HTTP proxy (duration: 00m 16s)
07:48 mholloway-shell@deploy1001: Started deploy [kartotherian/deploy@0bf513a] (maps1004): Remove HTTP proxy
07:42 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2088:3311 (duration: 00m 56s)
07:36 marostegui: Deploy schema change on db2088:3311 T205913
07:36 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2088:3311 (duration: 00m 55s)
07:32 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2072 (duration: 00m 55s)
07:18 marostegui: Deploy schema change on db2072 T205913
07:17 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2072 (duration: 01m 02s)
05:22 _joe_: stopped tilerator on maps1004, was spamming like crazy
01:18 ejegg: updated CiviCRM from e7a620a00c to c353eba283

2018-10-01

23:44 eileen: update process control revision is b9c7ab286e - define but not enable Redis
23:43 foks: disabling 2FA for two users
23:31 twentyafterfour: finished creating database tables
23:18 twentyafterfour: creating ipblocks_restrictions table (command run on mwmaint2001: foreachwiki sql.php maintenance/archives/patch-ipblocks_restrictions-table.sql)
22:52 ppchelko@deploy1001: Finished deploy [restbase/deploy@babfe80]: Don't log the request for transform failures, take 3, feeds check timeouts (duration: 06m 22s)
22:46 ppchelko@deploy1001: Started deploy [restbase/deploy@babfe80]: Don't log the request for transform failures, take 3, feeds check timeouts
22:45 ppchelko@deploy1001: Finished deploy [restbase/deploy@babfe80]: Don't log the request for transform failures, take 2, feeds check timeouts (duration: 03m 57s)
22:41 ppchelko@deploy1001: Started deploy [restbase/deploy@babfe80]: Don't log the request for transform failures, take 2, feeds check timeouts
22:41 ppchelko@deploy1001: Finished deploy [restbase/deploy@babfe80]: Don't log the request for transform failures (duration: 12m 27s)
22:29 ppchelko@deploy1001: Started deploy [restbase/deploy@babfe80]: Don't log the request for transform failures
21:17 arlolra: Updated Parsoid to 224ecde (T198504, T133673, T202666)
20:45 arlolra@deploy1001: Finished deploy [parsoid/deploy@8ff45db]: Updating Parsoid to 224ecde (duration: 08m 22s)
20:37 arlolra@deploy1001: Started deploy [parsoid/deploy@8ff45db]: Updating Parsoid to 224ecde
20:35 gehel@deploy1001: Finished deploy [wdqs/wdqs@a637583]: New version of WDQS GUI, updater and blazegraph (duration: 14m 00s)
20:21 gehel@deploy1001: Started deploy [wdqs/wdqs@a637583]: New version of WDQS GUI, updater and blazegraph
19:52 gehel@deploy1001: Finished deploy [wdqs/wdqs@a637583]: New version of WDQS GUI, updater and blazegraph (wdqs1009 only) (duration: 00m 30s)
19:51 gehel@deploy1001: Started deploy [wdqs/wdqs@a637583]: New version of WDQS GUI, updater and blazegraph (wdqs1009 only)
19:27 ppchelko@deploy1001: Finished deploy [restbase/deploy@7caf4d8]: Content-negotiation filter going live T128040 (duration: 03m 38s)
19:24 ppchelko@deploy1001: Started deploy [restbase/deploy@7caf4d8]: Content-negotiation filter going live T128040
19:11 thcipriani: restarting ci jenkins for new plugins
18:33 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable page issues A/B test at 20% rate (T200792) (duration: 00m 56s)
18:28 Amir1: ladsgroup@mwmaint2001:~$ mwscript extensions/CentralAuth/maintenance/deleteLocalPasswords.php --wiki=enwiki --prefix (T201009)
18:23 catrope@deploy1001: Synchronized php-1.32.0-wmf.23/maintenance/includes/DeleteLocalPasswords.php: T201009 (duration: 00m 56s)
18:17 catrope@deploy1001: Synchronized php-1.32.0-wmf.23/extensions/PageTriage/: Ensure valid AFC option is selected (T205324, T205168); hide copyvio behind a global var and URL param (duration: 00m 57s)
18:12 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable page issues A/B test at 5% rate (T200792) (duration: 00m 59s)
17:59 XioNoX: push fw change on pfw3-eqiad - T205888
17:57 XioNoX: push fw change on pfw3-codfw - T205888
17:28 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@a637583]: Test deployment for recent updater build and GUI changes. Also blazegraph updates(wdqs1009) (duration: 01m 46s)
17:27 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@a637583]: Test deployment for recent updater build and GUI changes. Also blazegraph updates(wdqs1009)
17:06 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1093, db1064 (duration: 00m 57s)
17:02 jynus: stopping some mariadb instances on dbstore1001 and starting compression T201392
16:26 ppchelko@deploy1001: Started restart [cpjobqueue/deploy@58f9ed3]: Fix KafkaConsumer not connected error
15:16 jynus: stopping db1064 to clone it to dbstore1001
15:00 akosiaris: upgrade etherpad to 1.7.0-2
14:14 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting MCR migration stage to write-both/read-new on mediawikiwiki (T198308) (duration: 00m 56s)
13:51 banyek: Downtimed the slave lag monitoring on dbstore1002 while the tables getting converted (T205544)
12:38 akosiaris: upload hfst_3.13.0~r3461-1+wmf2 to apt.wikimedia.org/jessie-wikimedia/main. T199962
12:26 banyek: converting enwiki.categorylinks to TokuDB on host dbstrore1002 (T205544)
12:19 banyek: stopping replication on s2@dbstore20002: the tables being compressed (T204930)
12:19 banyek: stopping replication on s2@dbstore20002: the tables being compressed
12:15 banyek: enabling puppet on labsdb1009, labsdb1010, labsdb1011 (T183983)
12:13 zeljkof: EU SWAT finished
12:12 zfilipin@deploy1001: Synchronized php-1.32.0-wmf.23/extensions/ContentTranslation/: SWAT: Fix error in CXTransclusionNode#afterRender method (T205521) (duration: 00m 59s)
11:56 jynus: stopping db1093 to clone it to dbstore1001
11:52 arturo: install prometheus-openstack-exporte 0.0.8-3 in reprepro T203177
11:41 zfilipin@deploy1001: Synchronized wmf-config: SWAT: Remove unused default source language config for CX (duration: 00m 57s)
11:16 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2058 (duration: 00m 55s)
11:09 _joe_: killed bash runner.sh by user ladsgroup on mwmaint2001
10:58 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2058 (duration: 00m 57s)
10:52 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1093, db1064 (duration: 00m 57s)
10:42 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 56s)
10:41 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 59s)
10:21 godog: repair /dev/sdf1 /dev/sde1 on ms-be1041 - T199198
10:15 Amir1: ladsgroup@mwmaint2001:~$ mwscript extensions/CentralAuth/maintenance/deleteLocalPasswords.php --prefix on all CentralAuth wikis (T201009)
10:10 Amir1: mwscript extensions/CentralAuth/maintenance/deleteLocalPasswords.php --wiki=fawiki --delete (T201009)
09:33 godog: test formatting sdh and sdi on ms-be2040 with crc=0 - T199198
09:15 volans: Set Racktables in read-only mode - T199083
08:56 _joe_: rolling restart of parsoid in codfw; afterwards, parsoid will connect to the MediaWiki API via HTTPS
08:54 _joe_: rolling restart of parsoid in eqiad
07:54 banyek: disabling puppet on labsdb1009, labsdb1010, labsdb1011 (T183983)
07:54 banyek: disabling puppet on labsdb1009, labsdb1010, labsdb1011
07:00 mholloway-shell@deploy1001: Finished deploy [kartotherian/deploy@ab6cb74] (maps1004): Update kartotherian to latest (T205462) (duration: 00m 16s)
07:00 mholloway-shell@deploy1001: Started deploy [kartotherian/deploy@ab6cb74] (maps1004): Update kartotherian to latest (T205462)
06:39 mholloway-shell@deploy1001: Finished deploy [tilerator/deploy@22f90ee] (maps1004): Update tilerator to latest (T205462) (duration: 00m 19s)
06:39 mholloway-shell@deploy1001: Started deploy [tilerator/deploy@22f90ee] (maps1004): Update tilerator to latest (T205462)
05:35 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3312 (duration: 00m 56s)
05:19 marostegui: Stop replication on dbstore1002 and db1103:3312 in sync
05:19 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3312 (duration: 01m 01s)
05:19 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@07cbfb4]: Update mobileapps to a1fa41b (duration: 03m 18s)
05:15 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@07cbfb4]: Update mobileapps to a1fa41b
05:07 marostegui: Deploy schema change on s1 codfw msater - T203709
03:21 onimisionipe: restarting inplace reindexing of enwiki and viwiki at codfw - T204362

2018-09-30

12:41 ariel@deploy1001: Finished deploy [dumps/dumps@26aaee6]: make location of MWScript.php configurable (duration: 00m 03s)
12:41 ariel@deploy1001: Started deploy [dumps/dumps@26aaee6]: make location of MWScript.php configurable

2018-09-29

13:01 gtirloni: tools-mail cleaned frozen messages in exim queue
01:13 krinkle@deploy1001: Synchronized php-1.32.0-wmf.23/extensions/ORES/includes/ORESService.php: T205651 - I1beaea (duration: 00m 59s)

2018-09-28

20:57 mutante: analytics1003 - unmounted and remounted /mnt/hdfs after Icinga alerts that it was not accessible - commands from https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration#Fixing_HDFS_mount_at_/mnt/hdfs - like it happened before on stat1004 and others (T182342)
19:18 mutante: phab1001 (Phabricator), scheduled downtime, reboot for maintenance
19:03 mutante: phab2001 - scheduled downtime, rebooting for kernel
16:25 XioNoX: add HKBN BGP sessions to esams and eqsin
16:05 jynus: compressing tables at db1116:3317, stopping replication
16:01 Amir1: ladsgroup@mwmaint2001:~$ mwscript extensions/CentralAuth/maintenance/deleteLocalPasswords.php --wiki=fawiki --prefix (T201009)
15:51 Amir1: ladsgroup@mwmaint2001:~$ mwscript extensions/CentralAuth/maintenance/deleteLocalPasswords.php --delete on mediawiki.org and testwiki
15:10 XioNoX: activate Equinix peering sessions on cr4-ulsfo
14:29 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1086, db1104 (duration: 00m 55s)
14:22 banyek: converting dewiki.flaggedimages to TokuDB on host dbstrore1002 (T205544)
13:27 moritzm: rebooting tungsten for kernel security update
13:15 mobrovac@deploy1001: Finished deploy [restbase/deploy@7caf4d8]: Update metrics top endpoints, take #4 (duration: 06m 36s)
13:09 mobrovac@deploy1001: Started deploy [restbase/deploy@7caf4d8]: Update metrics top endpoints, take #4
13:08 mobrovac@deploy1001: Finished deploy [restbase/deploy@7caf4d8]: Update metrics top endpoints, take #3 (duration: 03m 39s)
13:05 mobrovac@deploy1001: Started deploy [restbase/deploy@7caf4d8]: Update metrics top endpoints, take #3
13:05 mobrovac@deploy1001: Finished deploy [restbase/deploy@7caf4d8]: Update metrics top endpoints, take #2 (duration: 04m 57s)
13:00 mobrovac@deploy1001: Started deploy [restbase/deploy@7caf4d8]: Update metrics top endpoints, take #2
13:00 mobrovac@deploy1001: Finished deploy [restbase/deploy@7caf4d8]: Update metrics top endpoints (duration: 10m 55s)
12:49 mobrovac@deploy1001: Started deploy [restbase/deploy@7caf4d8]: Update metrics top endpoints
12:42 arturo: downtime cloudcontrol1003.wikimedia.org for 2H (tests related to T203177)
12:39 banyek: converting wikidatawiki.text to TokuDB on host dbstrore1002 (T205544)
12:28 arturo: downtime cloudcontrol1004.wikimedia.org for 2H (tests related to T203177)
12:01 moritzm: installint php security updates on einsteinium (icinga.wikimedia.org)
11:38 arturo: add prometheus-openstack-exporter 0.0.8-2 to reprepro (T203177)
11:21 moritzm: installing php5 security updates
11:01 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@961aa5a]: Update mobileapps to 38271fa (duration: 03m 05s)
10:58 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@961aa5a]: Update mobileapps to 38271fa
10:55 banyek: converting dewiki.flaggedtemplates to TokuDB on host dbstrore1002 (T205544)
10:09 moritzm: reimaging mw2150 to test router ACLs on cumin1001
09:45 jynus: stop db1086 and db1104 for cloning to db1116
09:41 moritzm: installing ca-certificates updates on trusty/stretch
09:34 banyek: converting whikishared.cx_coprora to TokuDB on host dbstrore1002 (T205544)
09:30 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1086, db1104 (duration: 00m 57s)
08:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1078 (duration: 00m 54s)
08:11 marostegui: Deploy schema change on s3 eqiad, this will generate lag - T203709
07:40 marostegui: Stop replication in sync on dbstore1002 and db1078
07:40 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1078 (duration: 00m 55s)
07:27 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2055 (duration: 00m 56s)
07:23 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@bf09080]: Update mobileapps to 7878ffc (duration: 02m 43s)
07:20 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@bf09080]: Update mobileapps to 7878ffc
07:03 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@bf09080]: Update mobileapps to 7878ffc
06:30 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@bf09080]: Update mobileapps to 7878ffc (duration: 01m 57s)
06:28 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@bf09080]: Update mobileapps to 7878ffc
06:27 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@bf09080]: Update mobileapps to 7878ffc (duration: 06m 40s)
06:20 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@bf09080]: Update mobileapps to 7878ffc
06:20 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@bf09080]: Update mobileapps to 7878ffc (duration: 00m 44s)
06:19 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@bf09080]: Update mobileapps to 7878ffc
06:07 marostegui: Deploy schema change on db2055 - T203709
06:07 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2055 (duration: 00m 55s)
05:54 marostegui: Deploy schema change on s7 eqiad, this will generate lag - T203709
05:47 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1094 (duration: 00m 55s)
05:29 marostegui: Stop replication in sync on db1094 and dbstore1002:s7
05:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1094 (duration: 00m 59s)

2018-09-27

23:29 thcipriani@deploy1001: Synchronized wmf-config/WikibaseSearchSettings.php: SWAT: Enable phrase search config T163642 (duration: 00m 56s)
23:08 stephanebisson: Finished mwscript extensions/ORES/maintenance/BackfillPageTriageQueue.php --wiki enwiki (T203286)
21:04 ejegg: updated fundraising CiviCRM from 43ca105873 to e7a620a00c
20:09 arlolra@deploy1001: Finished deploy [parsoid/deploy@0272096]: Updating Parsoid to ff6ffb5 (duration: 08m 05s)
20:03 XioNoX: update pfw3-codfw/eqiad firewall rules - T205574
20:01 arlolra@deploy1001: Started deploy [parsoid/deploy@0272096]: Updating Parsoid to ff6ffb5
19:31 bawolff: deploy related to T194204
19:29 XioNoX: update all cr1/2-eqiad to include cumin1001 - T205513
19:10 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove "metawiki" from "wgForceUIMsgAsContentMsg" T205633 (duration: 00m 56s)
18:59 XioNoX: update all MR routers to include cumin1001 - T205513
18:49 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.23/extensions/WikimediaEvents/modules/wikibase/ext.wikimediaEvents.completionClicks.js: SWAT: Ignore clicks with empty search string T205301 (duration: 00m 56s)
18:43 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Page-previews on Wikivoyage T203981 (duration: 00m 57s)
18:17 stephanebisson: Starting mwscript extensions/ORES/maintenance/BackfillPageTriageQueue.php --wiki enwiki (T203286)
18:14 arlolra: Updated Parsoid to af3a920 (T198511, T163438, T108776, T205334, T114413)
18:04 arlolra@deploy1001: Finished deploy [parsoid/deploy@6a2c25c]: Updating Parsoid to af3a920 (duration: 10m 43s)
17:54 arlolra@deploy1001: Started deploy [parsoid/deploy@6a2c25c]: Updating Parsoid to af3a920
17:50 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@a0054ba]: Update mobileapps to 0d6c2b7 (duration: 03m 21s)
17:47 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@a0054ba]: Update mobileapps to 0d6c2b7
17:27 XioNoX: reboot asw2-a-eqiad (not in prod) - T201145
17:27 ladsgroup@deploy1001: Finished deploy [ores/deploy@a717199]: Send metrics for non-major responses (duration: 23m 55s)
17:03 ppchelko@deploy1001: Finished deploy [restbase/deploy@0f11d5d]: Full deploy for content negotiations T128040 take 5, feeds.... (duration: 06m 32s)
17:03 ladsgroup@deploy1001: Started deploy [ores/deploy@a717199]: Send metrics for non-major responses
17:03 cmjohnson1: swapping failed disk slot 7 db1069
16:57 ppchelko@deploy1001: Started deploy [restbase/deploy@0f11d5d]: Full deploy for content negotiations T128040 take 5, feeds....
16:57 ppchelko@deploy1001: Finished deploy [restbase/deploy@0f11d5d]: Full deploy for content negotiations T128040 take 4, feeds.... (duration: 04m 46s)
16:52 ppchelko@deploy1001: Started deploy [restbase/deploy@0f11d5d]: Full deploy for content negotiations T128040 take 4, feeds....
16:51 ppchelko@deploy1001: Finished deploy [restbase/deploy@0f11d5d]: Full deploy for content negotiations T128040 take 3, feeds (duration: 03m 59s)
16:47 ppchelko@deploy1001: Started deploy [restbase/deploy@0f11d5d]: Full deploy for content negotiations T128040 take 3, feeds
16:47 ppchelko@deploy1001: Finished deploy [restbase/deploy@0f11d5d]: Full deploy for content negotiations T128040 take 2, feeds (duration: 03m 49s)
16:43 ppchelko@deploy1001: Started deploy [restbase/deploy@0f11d5d]: Full deploy for content negotiations T128040 take 2, feeds
16:43 ppchelko@deploy1001: Finished deploy [restbase/deploy@0f11d5d]: Full deploy for content negotiations T128040 (duration: 03m 06s)
16:39 ppchelko@deploy1001: Started deploy [restbase/deploy@0f11d5d]: Full deploy for content negotiations T128040
16:34 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T204791 (duration: 00m 57s)
16:29 ppchelko@deploy1001: Finished deploy [restbase/deploy@0f11d5d]: Canary on 2001 for content negotiations T128040 (duration: 04m 17s)
16:25 ppchelko@deploy1001: Started deploy [restbase/deploy@0f11d5d]: Canary on 2001 for content negotiations T128040
15:55 reedy@deploy1001: Synchronized php-1.32.0-wmf.23/autoload.php: new mtx script (duration: 00m 56s)
15:31 reedy@deploy1001: Synchronized php-1.32.0-wmf.23/maintenance/: add new mtx script (duration: 00m 58s)
15:19 cmjohnson1: swapping out failed disk slot 3 rdb1004
15:10 reedy@deploy1001: Synchronized wmf-config/extension-list: Bye Bye Education Program, Bye Bye. T125618 (duration: 00m 55s)
15:08 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Bye Bye Education Program, Bye Bye. T125618 (duration: 00m 56s)
15:07 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: Bye Bye Education Program, Bye Bye. T125618 (duration: 00m 58s)
15:07 XioNoX: add peering with Telin in esams
15:04 arturo: T196507 2h downtime cloudvirt1019 in icinga
15:02 moritzm: rebooting labtestvirt2003 for microcode tests
14:48 cmjohnson1: disabling checks on cloudvirt1019 to replace raid controller cable T196507
14:29 banyek: converting srwiki.pagelinks to TokuDB on host dbstore1002 (T205544)
13:49 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T205514: revert: depooling db1104, adding db1109 as temproray api host for s8 (duration: 00m 56s)
13:46 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T205514: revert: depooling db1104, adding db1109 as temproray api host for s8 (duration: 00m 55s)
13:44 moritzm: installing ca-certificates updates for jessie/stretch
13:41 banyek: repooling db1104 (T205514)
13:34 moritzm: installing postgres security updates on labsdb1004
13:17 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.23
13:13 godog: repair /dev/sdh1 on ms-be2042 - T199198
12:24 gehel: reboot of wdqs2004-2006 for kernel upgrade
12:11 moritzm: installing libapache2-mod-perl2 security updates
11:45 zeljkof: EU SWAT finished
11:44 zfilipin@deploy1001: Synchronized php-1.32.0-wmf.23/extensions/ContentTranslation/: SWAT: Use numerical option when setting CX version preference (T205493) (duration: 00m 57s)
10:50 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: NOOP phpdoc comment changes pt2/2 (duration: 00m 56s)
10:48 addshore@deploy1001: Synchronized wmf-config/CommonSettings.php: NOOP phpdoc comment changes pt1/2 (duration: 00m 56s)
10:41 godog: stop icinga-wm temporarily
09:54 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T205514: depooling db1104, adding db1109 as temproray api host for s8 (duration: 00m 56s)
09:35 addshore@deploy1001: Synchronized wmf-config/CommonSettings.php: RejectParserCacheValue Hook to purge wikidatawiki to 2018-09-19 (duration: 00m 56s)
09:15 addshore@deploy1001: Synchronized wmf-config/CommonSettings.php: RejectParserCacheValue Hook to purge wikidatawiki to 2018-09-15 (duration: 00m 57s)
09:04 moritzm: rebooting ores1* for kernel security updates
08:56 banyek: stopping replocication & mariadb on db1104 and db1092 as db1092 is getting recloned from db1104 (T205514)
08:54 godog: test formatting sde on ms-be1040 with crc=0 - T199198
08:48 marostegui: Deploy schema change on labswiki (db1073 master) - T203709
08:39 marostegui: Deploy schema change on labtestwiki - T203709
08:30 banyek: upgrading db1104 (kernel-mariadb) and rebooting it (T205514)
07:58 banyek: rebooting db1108 for kernel & mariadb upgrade (T205288)
07:55 addshore@deploy1001: Synchronized wmf-config/CommonSettings.php: RejectParserCacheValue Hook to purge wikidatawiki to 2018-09-12 (duration: 00m 55s)
07:54 addshore@deploy1001: sync-file aborted: RejectParserCacheValue Hook to purge wikidatawiki to 2018-09-08 (duration: 00m 01s)
07:53 jynus: enabling puppet back on all db hosts
07:45 addshore@deploy1001: Synchronized wmf-config/CommonSettings.php: RejectParserCacheValue Hook to purge wikidatawiki to 2018-09-08 (duration: 00m 55s)
07:30 godog: test formatting sdd on ms-be1040 with crc=0 - T199198
07:29 addshore@deploy1001: Synchronized wmf-config/CommonSettings.php: RejectParserCacheValue Hook to purge wikidatawiki to 2018-09-06 (duration: 00m 57s)
07:27 godog: test formatting sdc on ms-be1040 with crc=0 - T199198
07:26 jynus: enabling puppet on all core eqiad hosts
07:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1089 (duration: 00m 55s)
07:05 jynus: disablingh puppet on all databases
07:00 marostegui: Stop replication in sync on db1089 and dbstore1002:s1
06:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1089 (duration: 00m 56s)
06:17 marostegui: Deploy schema change on dbstore2002:3311
05:41 marostegui: Deploy schema change on s1 eqiad master, lag will be generated - T203709
05:40 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2072 (duration: 00m 57s)
05:26 marostegui: Drop wikiuser on dbstore1002
05:21 marostegui: Deploy schema change on db2072
05:21 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2072 (duration: 01m 00s)
03:04 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Sep 27 03:04:40 UTC 2018 (duration 10m 51s)
02:53 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.23) (duration: 07m 16s)
02:41 onimisionipe: starting inplace reindexing of viwiki and commonswiki - T204362
02:35 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.22) (duration: 13m 25s)
01:14 mutante: repair /dev/sdn1 on ms-be0241 - T199198
01:14 mutante: repair /dev/sde1 on ms-be0240 - T199198
00:22 krinkle@deploy1001: Synchronized php-1.32.0-wmf.23/extensions/FeaturedFeeds: T205573 (duration: 00m 59s)

2018-09-26

23:23 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.23/extensions/WikimediaEvents/modules/all/ext.wikimediaEvents.readingDepth.js: SWAT: Ensure Minerva has initialised before loading and executing ReadingDepth T204144 (duration: 00m 57s)
23:14 eileen: process control -renable omnirepair - process-control config revision is b8549d3344
22:48 eileen: civicrm revision changed from 018ff6325f to 43ca105873, config revision is 3b785f0af4
22:28 XioNoX: enable BGP to transits on cr3-ulsfo
21:20 thcipriani: restarting releases jenkins for updates
21:03 gtirloni: tools-mail deleted frozen messages from exim queue
19:54 mutante: icinga1001 - temp. disabling puppet, remove --purge all icinga packages, rm -rf /etc/nagios and /etc/icinga, let puppet recreate everything now that it should not mess with user/group on stretch (T202782)
19:42 krinkle@deploy1001: Synchronized php-1.32.0-wmf.23/extensions/Gadgets/includes/SpecialGadgets.php: (no justification provided) (duration: 00m 57s)
19:25 shdubsh: manually installing python3-jinja2 on puppetmaster1001 to test naggen2 python3 upgrade
18:01 joal@deploy1001: Finished deploy [analytics/aqs/deploy@39b909e]: Deploy Wikistats2 top-metrics updates (duration: 11m 58s)
17:49 joal@deploy1001: Started deploy [analytics/aqs/deploy@39b909e]: Deploy Wikistats2 top-metrics updates
17:22 jforrester@deploy1001: Synchronized php-1.32.0-wmf.23/extensions/3D/src/ThreeDThumbnailImage.php: Hot-deploy I5bb4b699a fix for T205554 train-blocker (duration: 00m 56s)
16:47 jforrester@deploy1001: Synchronized php-1.32.0-wmf.23/extensions/ORES/includes/SpecialORESModels.php: SWAT T205228 fix Iddb9c1f9e (duration: 00m 56s)
16:36 XioNoX: shutting down cr1/2-ulsfo for DC move
16:36 robh: shutting down all ulsfo servers for relocation
16:35 jforrester@deploy1001: Synchronized php-1.32.0-wmf.22/extensions/CentralAuth/includes/CentralAuthHooks.php: SWAT deploy I65ae2f05e (duration: 00m 56s)
16:34 jforrester@deploy1001: Synchronized php-1.32.0-wmf.23/extensions/CentralAuth/includes/CentralAuthHooks.php: SWAT deploy I65ae2f05e (duration: 00m 57s)
16:18 godog: test formatting sdd on ms-be2040 with crc=0 - T199198
16:10 zfilipin@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.23 (duration: 00m 54s)
16:09 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.23
16:07 bblack: downtimed ipsec alerts on cp[12]xxx for ulsfo outage
15:55 addshore@deploy1001: Synchronized php-1.32.0-wmf.22/extensions/WikibaseLexeme: form statement groups: ids without lexeme part T196226 (duration: 01m 01s)
15:54 XioNoX: ulsfo X-connect hot cut scheduled for in 5min
15:53 addshore@deploy1001: Synchronized php-1.32.0-wmf.23/extensions/WikibaseLexeme: form statement groups: ids without lexeme part T196226 (duration: 01m 06s)
15:29 pmiazga@deploy1001: Synchronized php-1.32.0-wmf.23/skins/MinervaNeue/includes/skins/SkinMinerva.php: SWAT: Create $returntoquery variable properly(T205449) (duration: 00m 56s)
15:02 gehel: rolling restart of wdqs-test and wdqs-internal for JVM + kernel upgrade
14:48 reedy@deploy1001: Synchronized php-1.32.0-wmf.23/includes/specials/SpecialMostlinkedcategories.php: T205469 (duration: 00m 55s)
14:46 reedy@deploy1001: Synchronized php-1.32.0-wmf.23/extensions/CirrusSearch/: T205473 (duration: 01m 06s)
13:45 marostegui: Stop replication on dbstore1002:s4 for maintenance
13:38 elukey: reboot an-master1001 to clear out an issue with systemd@fsck (Hadoop master, failover to an-master1002 included)
13:34 marostegui: Stop replication on s4 eqiad master for maintenance, lag will be generated
13:32 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2071 (duration: 00m 55s)
13:12 marostegui: Deploy schema change on db2071 - T203709
13:12 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2071 (duration: 00m 56s)
13:03 moritzm: installing tomcat8 security updates
12:33 marostegui: Deploy schema change on s4 eqiad, will generate lag - T203709
11:43 zeljkof: EU SWAT finished
11:42 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: Lift account creation IP cap for 2018-09-26 (T205529) (duration: 00m 56s)
11:30 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor in Project namespace on srwiki (T205206) (duration: 00m 57s)
11:11 volans: disabled puppet on einsteinium to test ircecho failure, I'll re-enable it in max ~30m
10:30 volans: started ircecho on einsteinium
10:27 volans: stop ircecho on einsteinium to register the nickname
10:26 moritzm: rebooting mw1270-mw1290 for kernel security updates
10:13 _joe_: reenabling puppet on the MediaWiki hosts
10:08 moritzm: rebooting conf1006 for kernel security update
09:51 moritzm: rebooting conf1005 for kernel security update
09:46 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2092 (duration: 00m 56s)
09:45 moritzm: rebooting conf1004 for kernel security update
09:29 jynus: restarting icinga at einsteinium, unresponsive to commands
09:06 marostegui: Deploy schema change on db2092 - T203709
09:06 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2092 (duration: 00m 56s)
08:58 _joe_: disabling puppet on all hosts with a MediaWiki setup before merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/453093
08:35 onimisionipe: restarting blazegraph and updater on wdqs2003
08:24 hashar: Restarting CI Jenkins on contint1001 [#2]
08:22 elukey: start eventlogging_sync on db1108 and the mysql kafka consumers on eventlog1002 after db1107 maintenance
08:21 moritzm: rebooting mw1250-mw1269 for kernel security updates
08:17 banyek: db1107 upgrade finished (T205288)
08:15 banyek: db1107 upgrade finished
08:14 hashar: Restarting CI Jenkins on contint1001
08:04 elukey: stop eventlogging_sync on db1108 and the mysql kafka consumers on eventlog1002 as prep step for db1107 maintenance
08:04 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1092, server crashed - T205514 (duration: 00m 56s)
08:00 banyek: donwtiming db1107 as upgrade (kernel & mariadb)
07:50 marostegui: Hard reset db1092, server crashed - T205514
07:48 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Set testwikidatawiki CacheEpoch to 2018-09-19 T205330 (duration: 00m 56s)
07:47 elukey: reboot an-master1002 as attempt to clear out some systemd@fsck alarms
07:46 godog: repair /dev/sdd1 on ms-be1043 - T199198
07:41 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2085:3311 (duration: 00m 56s)
07:36 banyek: stopping s2 replication on dbstore2002 (compressing tables)
07:34 godog: powercycle ms-be2030 - no console
07:31 jynus: restart db1115
07:29 godog: repair /dev/sdh1 on ms-be2041 - T199198
07:28 jynus: stoping mariadb at db1115
07:19 volans: restarted ircecho on einsteinium "icinga-wm quit (Reason: Remote host closed the connection)"
05:29 marostegui: Deploy schema change on db2085:3311
05:28 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2085:3311 (duration: 00m 56s)
05:20 marostegui: Deploy schema change on s8 eqiad, this will generate lag - T203709
05:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1088, db1098:3316 and db1098:3317 (duration: 01m 06s)
04:17 tstarling@deploy1001: Synchronized php-1.32.0-wmf.23/includes/specials/SpecialExport.php: enable export logging channel (attempt 2) (duration: 00m 51s)
04:09 tstarling@deploy1001: Synchronized php-1.32.0-wmf.23/includes/specials/SpecialExport.php: enable export logging channel (duration: 00m 55s)
03:48 eileen: process-control config revision is 3b785f0af4 - disable omniail repair job
03:42 krinkle@deploy1001: Synchronized php-1.32.0-wmf.23/includes/page/ImageHistoryPseudoPager.php: T204796 - I17455fef0d8 (duration: 00m 58s)
03:21 tstarling@deploy1001: Synchronized wmf-config/InitialiseSettings.php: add export logging channel for g 461647 (duration: 00m 57s)
03:06 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Sep 26 03:06:46 UTC 2018 (duration 10m 54s)
02:55 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.23) (duration: 05m 10s)
02:32 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.22) (duration: 12m 42s)

2018-09-25

23:59 maxsem@deploy1001: Synchronized php-1.32.0-wmf.22/extensions/GlobalPreferences/: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/GlobalPreferences/+/462822/ (duration: 00m 56s)
23:53 eileen: civicrm revision changed from ca49aed673 to 018ff6325f, config revision is e71b8fdbdf
23:49 maxsem@deploy1001: Synchronized php-1.32.0-wmf.22/extensions/UploadWizard/: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/UploadWizard/+/462781/ (duration: 00m 56s)
23:46 maxsem@deploy1001: Synchronized php-1.32.0-wmf.23/extensions/UploadWizard/: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/UploadWizard/+/462780/ (duration: 00m 56s)
23:39 maxsem@deploy1001: Synchronized php-1.32.0-wmf.22/skins/MinervaNeue/: https://gerrit.wikimedia.org/r/#/c/mediawiki/skins/MinervaNeue/+/462705/ (duration: 00m 57s)
23:29 maxsem@deploy1001: Synchronized wmf-config/WikibaseSearchSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/462347/ (duration: 00m 57s)
22:55 ppchelko@deploy1001: Finished deploy [restbase/deploy@0fa695e] (dev-cluster): Deployinng content negotiation to dev cluster for ab testing (duration: 02m 53s)
22:52 ppchelko@deploy1001: Started deploy [restbase/deploy@0fa695e] (dev-cluster): Deployinng content negotiation to dev cluster for ab testing
19:17 catrope@deploy1001: Synchronized php-1.32.0-wmf.23/extensions/ORES/maintenance/BackfillPageTriageQueue.php: I3f1ae92d8645 (duration: 00m 58s)
17:49 otto@deploy1001: Finished deploy [analytics/refinery@ce8f0b3]: Deploying refinery-source 0.0.75 for ConfigHelper Refine - T203804 (duration: 10m 22s)
17:39 otto@deploy1001: Started deploy [analytics/refinery@ce8f0b3]: Deploying refinery-source 0.0.75 for ConfigHelper Refine - T203804
17:12 XioNoX: depool ulsfo for DC move
16:59 gehel: forcing puppet run on all elastic nodes (including logstash) to recover prometheus metric exporter
15:14 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Turn off EducationProgram T188411 T125618 (duration: 00m 57s)
15:02 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.32.0-wmf.23
14:50 marostegui: Stop MySQL on db1098 s6 and s7 for upgrade
14:49 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1098:3317 (duration: 00m 57s)
14:39 zfilipin@deploy1001: Finished scap: testwiki to php-1.32.0-wmf.23 and rebuild l10n cache (duration: 20m 50s)
14:38 moritzm: rebooting mw1240-mw1258 for kernel security updates
14:18 zfilipin@deploy1001: Started scap: testwiki to php-1.32.0-wmf.23 and rebuild l10n cache
14:15 zfilipin@deploy1001: Pruned MediaWiki: 1.32.0-wmf.19 [keeping static files] (duration: 01m 29s)
14:12 zfilipin@deploy1001: Pruned MediaWiki: 1.32.0-wmf.18 (duration: 03m 02s)
14:10 marostegui: Upgrade db1088
14:05 marostegui: Stop db1088 and db1098:3316 in sync
14:03 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1098:3316 (duration: 00m 50s)
14:02 zfilipin@deploy1001: Pruned MediaWiki: 1.32.0-wmf.16 (duration: 08m 27s)
13:29 marostegui: Deploy schema change on s2 eqiad - this might generate lag on s2 eqiad - T203709
13:13 marostegui: Deploy schema change on s6 eqiad - this might generate lag on s6 eqiad - T203709
13:07 marostegui: Deploy schema change on s5 eqiad - this might generate lag on s5 eqiad - T203709
12:51 moritzm: rebooting mw1221-mw1239 for kernel security updates
12:42 moritzm: installing chromium security updates on proton* (tested the new Chromium version in deployment-prep)
12:36 raynor: EU SWAT finished
12:34 pmiazga@deploy1001: Finished scap: php-1.32.0-wmf.22/skins/MinervaNeue/includes SWAT: Minerva A/B tests are not subject to HTML caching time (T205355) (duration: 16m 51s)
12:34 godog: repair sde on ms-be2042 - T199198
12:17 pmiazga@deploy1001: Started scap: php-1.32.0-wmf.22/skins/MinervaNeue/includes SWAT: Minerva A/B tests are not subject to HTML caching time (T205355)
12:05 pmiazga@deploy1001: Finished scap: php-1.32.0-wmf.22/skins/MinervaNeue/resources/skins.minerva.scripts/pageIssues.js SWAT: It should be possible to opt into new page issues treatment via query string parameter (T204746) (duration: 17m 39s)
12:01 elukey: end of the maintenance to swap Hadoop masters from analytics100[1,2] to an-master100[1,2] - T203635
11:47 pmiazga@deploy1001: Started scap: php-1.32.0-wmf.22/skins/MinervaNeue/resources/skins.minerva.scripts/pageIssues.js SWAT: It should be possible to opt into new page issues treatment via query string parameter (T204746)
11:20 pmiazga@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Increase sampling ratio for ReadingDepth (T205176) (duration: 00m 50s)
10:53 onimisionipe: onimisionipe: starting inplace reindex of eqiad enwiki - T204362
09:48 banyek: upgrading & rebooting dbproxy1006
09:38 Amir1: ladsgroup@mwmaint2001:~$ foreachwikiindblist all populateChangeTagDef.php --set-user-tags-only
09:34 banyek: rebooting again dbproxy1004, 1005, 1007, 1009
09:25 banyek: upgrade and reboot dbproxy1008
09:18 banyek: upgrade and reboot dbproxy1007
08:54 banyek: upgrade & reboot dbproxy1005
08:52 moritzm: installing libtirpc security updates
08:49 mobrovac@deploy1001: Finished deploy [restbase/deploy@40b81a8]: Do not dynamically generate Parsoid content if TID is provided, take #2 (duration: 12m 19s)
08:39 banyek: upgrade & reboot dbproxy1009
08:38 moritzm: installing twitter-bootstrap3 security updates
08:37 mobrovac@deploy1001: Started deploy [restbase/deploy@40b81a8]: Do not dynamically generate Parsoid content if TID is provided, take #2
08:37 mobrovac@deploy1001: Finished deploy [restbase/deploy@40b81a8]: Do not dynamically generate Parsoid content if TID is provided - T204880 (duration: 12m 00s)
08:25 mobrovac@deploy1001: Started deploy [restbase/deploy@40b81a8]: Do not dynamically generate Parsoid content if TID is provided - T204880
08:24 moritzm: installing libapache2-mod-perl2 security updates
08:23 banyek: upgrading & rebooting dbproxy1004
08:21 mobrovac@deploy1001: Finished deploy [restbase/deploy@40b81a8] (dev-cluster): Do not dynamically generate Parsoid content if TID is provided (duration: 02m 52s)
08:18 mobrovac@deploy1001: Started deploy [restbase/deploy@40b81a8] (dev-cluster): Do not dynamically generate Parsoid content if TID is provided
08:03 moritzm: installing reportbug DLA update on jessie hosts
08:03 elukey: start of the maintenance to swap Hadoop masters from analytics100[1,2] to an-master100[1,2] - T203635
07:54 marostegui: Deploy schema change on s4 codfw - T204006
07:20 godog: repair sdm sdi on ms-be2043 - T199198
07:13 marostegui: Deploy schema change on s3 eqiad master (db1075), might generate lag on s3 eqiad - T204006
07:00 marostegui: Deploy schema change on labswiki (wikitech) m5 master db1073 - T204006
06:58 marostegui: Deploy schema change on labtestwiki - T204006
06:52 marostegui: Deploy schema change on s1 eqiad master (db1067), might generate lag on s1 eqiad - T204006
06:49 marostegui: Deploy schema change on s2 eqiad master (db1066), might generate lag on s2 eqiad - T204006
06:46 marostegui: Deploy schema change on s7 eqiad master (db1062), might generate lag on s4 eqiad - T204006
06:42 marostegui: Deploy schema change on s4 eqiad master (db1068), might generate lag on s4 eqiad - T204006
06:11 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2038 (duration: 00m 49s)
06:05 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2038 (duration: 00m 49s)
05:59 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2059 (duration: 00m 49s)
05:51 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2059 (duration: 00m 49s)
05:50 marostegui@deploy1001: sync-file aborted: Repool db2059 (duration: 00m 09s)
05:45 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2066 (duration: 00m 50s)
05:37 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2066 (duration: 00m 50s)
05:32 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2075 (duration: 00m 51s)
03:02 krinkle@deploy1001: Synchronized php-1.32.0-wmf.22/includes/specials/SpecialLog.php: T201411 - Ie1a9a8 (duration: 00m 52s)
02:30 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Sep 25 02:30:02 UTC 2018 (duration 10m 42s)
02:19 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.22) (duration: 07m 45s)
01:54 ejegg: updated fundraising CiviCRM from b0223f56b0 to ca49aed673
00:38 mutante: mwmaint1002 - created /var/run/nutcracker dir and fixed permissions on it, then started nutcracker with systemctl. this fixed icinga alerts T201343

2018-09-24

23:21 maxsem@deploy1001: Synchronized wmf-config/: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/461715/ (duration: 00m 51s)
22:54 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Remove DisableAccount (duration: 00m 52s)
22:52 reedy@deploy1001: Synchronized wmf-config/extension-list: Remove DisableAccount (duration: 00m 49s)
22:43 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: Bye bye DisableAccount T106067 (duration: 00m 51s)
21:23 imarlier@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Disable NavTiming oversampling from Asian countries, used during Singapore data center rollout (T204365) (duration: 00m 50s)
20:57 SMalyshev: Started fulltext reindex for wikidatawiki
20:28 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@4be131b]: Update mobileapps to badb463 (T187098 T195838) (duration: 04m 31s)
20:23 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@4be131b]: Update mobileapps to badb463 (T187098 T195838)
19:36 gehel: resetting failed units on elasticsearch codfw
19:02 gjg@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT - [config] {{gerrit|459365}} Remove mhs.ox.ac.uk from $wgCopyUploadsDomains ({{phab|T203904}}) (duration: 00m 50s)
18:57 gjg@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [config] {{gerrit|462011}} Set wgRestrictDisplayTitle = false for pflwiki ({{phab|T205055}}) (duration: 00m 48s)
18:49 gjg@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [config] {{gerrit|462271}} Search by default on the Reconstruction namespace at frwikt ({{phab|T205198}}) (duration: 00m 50s)
18:44 gjg@deploy1001: Synchronized wmf-config/throttle.php: [config] {{gerrit|462014}} New users on IP for edit-a-thon (October 11) ({{phab|T204829}}) (duration: 00m 49s)
18:37 gjg@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [config] {{gerrit|461240}} Allow wikitech bureaucrats to promote to interface-admin, but uh, only wikitech (duration: 00m 50s)
18:21 gjg@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT [config] {{gerrit|462507}} Increase Schema:CitationUsagePageLoad population size ({{phab|T191086}}) (duration: 00m 50s)
18:14 gjg@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT [config] {{gerrit|462045}} Enable Extension:NewUserMessage on kn.wikisource ({{phab|T204405}}) (duration: 00m 50s)
17:59 gehel: restarting updater on wdqs*
17:50 ejegg: updated standalone SmashPig deployment from a2bc0b9fa5 to 82f9d49c23
17:39 XioNoX: powerdown and move bast4002 (not in prod)
17:18 ejegg: updated SmashPig standalone from 8500e75d9d to a2bc0b9fa5
17:15 gehel@deploy1001: Finished deploy [wdqs/wdqs@195ea0e]: new version of wdqs GUI and updater (duration: 13m 15s)
17:02 gehel@deploy1001: Started deploy [wdqs/wdqs@195ea0e]: new version of wdqs GUI and updater
16:59 gehel@deploy1001: Finished deploy [wdqs/wdqs@195ea0e]: new version of wdqs GUI and updater (wdqs1009 only) (duration: 00m 31s)
16:58 gehel@deploy1001: Started deploy [wdqs/wdqs@195ea0e]: new version of wdqs GUI and updater (wdqs1009 only)
15:55 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1064 (duration: 00m 50s)
15:52 moritzm: rebooting rdb1009/rdb1010 for kernel security update
15:46 marostegui: Deploy schema change on s8 eqiad master with replication - might generate lag - T204006
15:23 jynus: stop and upgrade db1064 (x1)
15:18 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1064 (duration: 00m 50s)
15:04 moritzm: rebooting rdb1008 for kernel security update
14:51 moritzm: rebooting rdb1007 for kernel security update
14:30 bblack: upgrade gdnsd 2.99.42 -> 2.99.1729 on authdns2001
14:28 moritzm: rebooting rdb1006 for kernel security update
14:28 bblack: upgrade gdnsd 2.99.42 -> 2.99.1729 on authdns1001
14:20 moritzm: rebooting rdb1005 for kernel security update
14:18 jynus: stop and upgrade es1017 (es3 eqiad master)- it may create some temporary lag on es3
13:59 jynus: stop and upgrade es1015 (es2 eqiad master)- it may create some temporary lag on es2
13:52 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2089 (duration: 00m 49s)
13:43 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2089 (duration: 00m 50s)
13:34 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Setup to db hosts for api-s5 (duration: 00m 50s)
13:23 moritzm: rebooting rdb1004 for kernel security update
13:12 moritzm: rebooting rdb1003 for kernel security update
13:03 gehel: start isolating maps1004 for reimage to stretch - T195285
11:39 elukey: reboot an-master100[1,2] as part of the pre-checks before the hadoop master daemons swap - T203635
11:21 moritzm: installing texlive-bin security updates
10:44 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 49s)
10:43 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 50s)
09:42 jynus: stop and upgrade db1066 (s2 eqiad master)- it may create some temporary lag on s2
09:30 godog: upgrade / roll restart thumbor in eqiad / codfw - T20871 T198370
09:03 jynus: stop and upgrade db1067 (s1 eqiad master)- it may create some temporary lag on s1
07:55 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Sep 24 07:55:18 UTC 2018 (duration 10m 50s)
07:44 bawolff@deploy1001: scap sync-l10n completed (1.32.0-wmf.22) (duration: 07m 44s)
07:31 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2062 (duration: 00m 46s)
07:25 bawolff_: manually running l10n update (T205238)
07:09 moritzm: installing libarchive-zip-perl security updates
07:05 volans: wiping netbox DB to re-import it cleanly from racktables - T199083
06:58 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2062 (duration: 00m 50s)
06:57 godog: repair sde on ms-be2041 - T199198
06:57 godog: repair sdn on ms-be1041 - T199198
06:54 volans@deploy1001: Finished deploy [netbox/deploy@5e70423]: Cherry pick of custom fields fix (duration: 00m 05s)
06:54 volans@deploy1001: Started deploy [netbox/deploy@5e70423]: Cherry pick of custom fields fix
05:48 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2088:3311 (duration: 00m 50s)
05:28 marostegui: Deploy schema change on db2088:3311 - T203709
05:27 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2088:3311 (duration: 00m 50s)
04:15 kartik@deploy1001: Finished deploy [cxserver/deploy@3e2d668]: Update cxserver to d913793 (T203551, T203780, T202716, T203947) (duration: 04m 20s)
04:11 kartik@deploy1001: Started deploy [cxserver/deploy@3e2d668]: Update cxserver to d913793 (T203551, T203780, T202716, T203947)
02:48 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Sep 24 02:48:16 UTC 2018 (duration 10m 49s)
02:37 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.22) (duration: 16m 02s)

2018-09-23

21:41 krinkle@deploy1001: Synchronized wmf-config/profiler.php: (no justification provided) (duration: 00m 52s)
04:32 krinkle@deploy1001: Synchronized php-1.32.0-wmf.22/includes/user/User.php: T202149 - Ic0c25f66f23f (duration: 00m 53s)

2018-09-22

18:29 krinkle@deploy1001: Synchronized php-1.32.0-wmf.22/extensions/MultimediaViewer/resources/: I0954c4, T205162 (duration: 00m 56s)

2018-09-21

23:47 mutante: mwmaint1002 - rsyncing home dirs over from mwmaint1001 which synced them from terbium. still large old files from terbium here, like a NASA video :) (T201343)
22:40 mutante: mwmaint1002 - scap pull
22:13 mutante: mwmaint1002 - re-enabling puppet, now that it has mcrouter certs, that works (T201343)
22:13 mutante: mwmaint1002 - re-enabling puppet, now that it has mcrouter certs, that works (201343)
22:12 mutante: how to generate mcrouter certs, needed when you add any new mw server: https://wikitech.wikimedia.org/wiki/Mcrouter#Generate_certs_for_a_new_host
19:10 XioNoX: shutdown cr3/4-ulsfo (not in prod)
18:19 bawolff: deploy patch related to T197279 T204825
18:08 bblack: upgrade multatuli to gdnsd-2.99.1729-beta
18:06 bblack: uploaded 2.99.1729-beta-1+wmf1 to stretch-wikimedia
17:59 mutante: *.toolserver.org also moved from eqiad to eqiad-r region in cloud vps, which gave it new IP addresses
17:58 mutante: toolserver.org and subdomains (wiki.toolserver, status.toolserver, stable.toolserver) legacy URLs have been switched to new stretch backend, away from trusty
16:37 XioNoX: route ns2-v6 to multatuli.wikimedia.org on cr1/2-esams
16:33 mutante: puppetmaster1001: cergen --base-path /srv/private/modules/secret/secrets/mcrouter/ --generate /etc/cergen/mcrouter.manifests.d
16:32 mutante: puppetmaster1001: mcrouter_generate_certs --generate
16:29 mutante: puppetmaster: running mcrouter_generate_certs to add an mcrouter cert for mwmaint1002 (T201343) https://wikitech.wikimedia.org/wiki/Mcrouter#Generate_certs_for_a_new_host
15:39 bblack: upgrading gdnsd to 2.99.42 on authdns2001
15:16 XioNoX: route ns2 to multatuli.wikimedia.org on cr1/2-esams
13:35 ladsgroup@deploy1001: Finished deploy [ores/deploy@7b987a7]: Pass the real IP (T205087) (duration: 24m 10s)
13:11 ladsgroup@deploy1001: Started deploy [ores/deploy@7b987a7]: Pass the real IP (T205087)
13:01 volans: wiping netbox DB to re-import it cleanly from racktables - T199083
12:43 banyek: adding wmf-pt-kill_2.2.20-1+wmf1 package for jessie
12:31 banyek: adding wmf-pt-kill_2.2.20-1+wmf1 package for strech
12:24 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool pc1006 (duration: 00m 49s)
12:13 marostegui: Stop MySQL on pc1006 for kernel upgrade
12:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool pc1006 (duration: 00m 50s)
07:17 moritzm: uploaded debdeploy 0.0.99.6 to apt.wikimedia.org
07:00 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool pc1005 (duration: 00m 50s)
06:51 marostegui: Stop MySQL on pc1005 for kernel upgrade
06:51 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool pc1005 (duration: 00m 51s)

2018-09-20

23:49 ejegg: updated fundraising CiviCRM from 7f58670e88 to b0223f56b0
23:06 ejegg: updated payments-wiki from e7d373fa71 to 1af8f8c8c0
21:47 ejegg: updated payments-wiki from 43923fadd9 to e7d373fa71
21:26 mutante: releases1001 - rm /usr/local/sbin/sync-srv-org-wikimedia-releases (bromine remnants)
21:26 mutante: releases1001 - deleting cronjob remnants for releases from bromine
21:17 mutante: releases2001 - manually rsync releases files from releases1001
21:16 hashar: 1.32.0-wmf.22 is fully deployed. A quick summary and thanks words are at https://phabricator.wikimedia.org/T191068#4604040
20:55 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: Comment clean-up no-op (duration: 00m 50s)
20:54 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Comment clean-up no-op (duration: 00m 51s)
20:22 SMalyshev: Started wikidata reindex again, hopefully better luck this time
20:07 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Bye bye tidy config! (duration: 00m 50s)
20:00 gehel: elasticsearch codfw cluster restart for new systemd unit completed
19:57 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Disable DisableAccount T106067 (duration: 00m 51s)
19:50 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Temporarily re-enable DisableAccount on wikis it was previous enabled (duration: 00m 51s)
19:36 reedy@deploy1001: Pruned MediaWiki: 1.32.0-wmf.15 (duration: 03m 16s)
19:32 reedy@deploy1001: Pruned MediaWiki: 1.32.0-wmf.14 (duration: 02m 05s)
19:22 tzatziki: Add email to account Abalg~commonswiki
19:22 reedy@deploy1001: clean aborted: Pruned MediaWiki: 1.32.0-wmf.14 (duration: 00m 30s)
19:21 reedy@deploy1001: Pruned MediaWiki: 1.32.0-wmf.14 [keeping static files] (duration: 02m 03s)
19:19 bblack: reboot multatuli
19:13 reedy@deploy1001: clean aborted: Pruned MediaWiki: 1.32.0-wmf.4 (duration: 01m 03s)
19:12 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.22/extensions/CentralNotice/resources/subscribing/ext.centralNotice.display.js: SWAT: Add performance mark for when banner is inserted T195840 (duration: 00m 51s)
19:06 reedy@deploy1001: Pruned MediaWiki: 1.32.0-wmf.4 [keeping static files] (duration: 01m 58s)
19:02 thcipriani@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: Set $wgSiteMatrixNonGlobalSites global for SiteMatrix List $wgSiteMatrixNonGlobalSites as global (duration: 00m 52s)
18:54 reedy@deploy1001: Synchronized php-1.32.0-wmf.22/extensions/DisableAccount/: maintenance script (duration: 00m 52s)
18:53 thcipriani@deploy1001: Synchronized wmf-config/CommonSettings.php: Revert on canaries: SWAT: Set $wgSiteMatrixNonGlobalSites global for SiteMatrix (duration: 00m 53s)
18:45 thcipriani@deploy1001: scap failed: average error rate on 6/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details)
18:43 SMalyshev: Initiating in-place reindex for wikidatawiki (T147505)
18:31 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.22/extensions/PageTriage/modules: SWAT: Correctly sync the form when afc_state === "all" T204629 Use same api params for list and stats on page load T204629 (duration: 00m 54s)
18:21 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable logging for CitationUsage and CitationUsagePageLoad T191086 (duration: 00m 51s)
17:59 bblack: reinstalling multatuli
17:35 thcipriani@deploy1001: Finished scap: Noop (hopefully) test of php7.0 (duration: 03m 07s)
17:31 thcipriani@deploy1001: Started scap: Noop (hopefully) test of php7.0
17:24 ladsgroup@deploy1001: Finished deploy [ores/deploy@ee2d28b]: Returning 429 instead of 408 in case of too many requests (T204956) (duration: 21m 15s)
17:24 godog: repair sdk on ms-be2043 - T199198
17:24 godog: upload scap 3.8.6-1 - T204383
17:03 ladsgroup@deploy1001: Started deploy [ores/deploy@ee2d28b]: Returning 429 instead of 408 in case of too many requests (T204956)
15:56 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting MCR migration stage to write-both/read-new on testwiki (T198309) (duration: 00m 59s)
15:17 moritzm: installing glib2.0 security updates for trusty
15:15 stephanebisson: Starting mwscript extensions/PageTriage/maintenance/populateDraftQueue.php --wiki enwiki (T203184)
15:14 stephanebisson: Finished mwscript extensions/PageTriage/maintenance/DeleteAfcStates.php --wiki enwiki (T203184)
15:11 stephanebisson: Starting mwscript extensions/PageTriage/maintenance/DeleteAfcStates.php --wiki enwiki (T203184)
15:05 hashar: 1.32.0-wmf.22 on group2 seems fine so far \o/
15:05 moritzm: installing bind9 security updates (client-side tools and libraries)
14:48 godog: repair sdn on ms-be2040 - T199198
14:44 gehel: reduce replication factor to 2 on cassandra maps eqiad - T194966
14:38 hashar@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.22
14:28 hashar@deploy1001: Synchronized typos: Dummy sync to verify list of canaries for T204907 (duration: 00m 59s)
14:27 herron: maps1001:~# tune2fs -m0 /dev/mapper/maps1001--vg-data
14:22 marostegui: Deploy schema change on s5 eqiad master with replication T204006
14:15 moritzm: rebooting rdb1001 for kernel security update
14:15 bblack: upgrade authdns1001 gdnsd 2.99.9 -> 2.99.42
14:12 bblack: upload gdnsd-2.99.42-beta to stretch-wikimedia
14:11 herron: removing unused wiki-mail.wikimedia.org cname (gerrit 143762)
14:05 moritzm: rebooting rdb1002 for kernel security update
13:27 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Tune codfw database weights (duration: 00m 58s)
13:20 gtirloni: T204667 reimage labtestnet2003
12:14 gehel: starting elasticsearch codfw cluster restart for new systemd unit
12:07 zeljkof: EU SWAT finished
12:01 moritzm: rebooting mc1036 for kernel security update
11:48 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create new namespaces in zhwikiversity (T201675) (duration: 00m 57s)
11:46 moritzm: rebooting mc1035 for kernel security update
11:38 moritzm: rebooting mc1034 for kernel security update
11:35 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Introduce engineer user group on Czech Wikipedia (T203000) (duration: 00m 58s)
11:28 moritzm: rebooting mc1033 for kernel security update
11:19 dcausse@deploy1001: Synchronized ./docroot/search.wikimedia.org/index.php: search.wikimedia.org should properly handle multivalue separation char (0x1F) (duration: 00m 58s)
11:15 jynus: rebooting es200X hosts for upgrade
11:07 moritzm: rebooting mc1032 for kernel security update
11:03 gehel: elasticsearch eqiad cluster restart for new systemd unit completed
10:58 moritzm: rebooting mc1031 for kernel security update
10:43 moritzm: rebooting mc1030 for kernel security update
10:30 moritzm: rebooting mc1029 for kernel security update
10:16 moritzm: rebooting mc1028 for kernel security update
10:05 moritzm: rebooting mc1027 for kernel security update
09:50 moritzm: rebooting mc1026 for kernel security update
09:37 moritzm: rebooting mc1025 for kernel security update
09:26 marostegui: Change passwords for wikiuser on dbstore1002 - T200801
09:24 moritzm: rebooting mc1024 for kernel security update
08:57 moritzm: rebooting mc1023 for kernel security update
08:44 moritzm: rebooting mc1022 for kernel security update
08:06 banyek: replication stopped and tables being compressed for s2 on dbstore2002
08:01 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2066 (duration: 00m 57s)
07:53 marostegui: Deploy schema change on db2066
07:53 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2066 (duration: 00m 57s)
07:46 moritzm: upgrading intel-microcode on trusty systems to 3.20180807a
07:31 godog: repair sdl on ms-be2042 - T199198
07:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool pc1004 (duration: 00m 57s)
07:15 marostegui: Stop MySQL on pc1004 for kernel upgrade
07:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool pc1004 (duration: 00m 57s)
06:50 eileen: process-control config revision is 3069f85578
06:42 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2070 (duration: 00m 58s)
06:25 mobrovac@deploy1001: Started restart [cpjobqueue/deploy@58f9ed3]: Reset the Kafka connections
06:20 mobrovac@deploy1001: Synchronized rpc/RunSingleJob.php: Have RunSingleJob send the X-Readonly header - T204154 (duration: 00m 58s)
06:09 marostegui: Deploy schema change on db2070 - T203709
06:07 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2070 (duration: 00m 58s)
05:54 marostegui: Deploy schema change on s3:mediawikiwiki for echo tables codfw - T51593
05:49 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Resume reads on db2034 (duration: 00m 57s)
05:34 marostegui: Deploy schema change on db2034 (x1 master)
05:33 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Stop reads on db2034 (duration: 00m 57s)
05:28 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2069 after alter table (duration: 00m 57s)
05:12 marostegui: Deploy schema change on db2069
05:11 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2069 for alter table (duration: 01m 00s)
03:06 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Sep 20 03:06:52 UTC 2018 (duration 10m 47s)
02:56 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.22) (duration: 13m 54s)
02:23 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.20) (duration: 09m 26s)
01:27 krinkle@deploy1001: Synchronized php-1.32.0-wmf.22/resources/src/startup/: I6c77b25856 (duration: 00m 55s)
01:14 krinkle@deploy1001: Synchronized php-1.32.0-wmf.22/extensions/GlobalPreferences/includes/GlobalPreferencesFactory.php: I52434a523f60e, T204864 (duration: 00m 58s)
00:52 krinkle@deploy1001: Synchronized php-1.32.0-wmf.22/extensions/3D/modules/: I9e718957a497, T204621 (duration: 00m 58s)
00:49 mutante: temp disabled puppet on mw2*, deployed gerrit 461393, confirmed noop, re-enabled puppet on mw2*

2018-09-19

23:21 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Remove obsolete $wgPopupsBetaFeature, part 2 (T203589) (duration: 00m 56s)
23:21 catrope@deploy1001: Synchronized wmf-config/CommonSettings.php: Remove obsolete $wgPopupsBetaFeature, part 1 (T203589) (duration: 00m 56s)
23:17 catrope@deploy1001: Synchronized php-1.32.0-wmf.22/extensions/PageTriage/: PageTriage fixes for T203184 (duration: 00m 58s)
23:14 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable Page issues A/B test on lvwiki (T204609) (duration: 00m 58s)
22:00 XioNoX: merging icinga check_bfd
21:24 jforrester@deploy1001: Synchronized php-1.32.0-wmf.22/extensions/VisualEditor/includes/SpecialCollabPad.php: T204873 Fix special page override hot deploy (duration: 00m 57s)
20:26 ladsgroup@deploy1001: Finished deploy [ores/deploy@76fe25a]: ORES doesn't block hammering IPs (T204862) (duration: 21m 13s)
20:24 XioNoX: repool ulsfo
20:21 ejegg: updated fundraising CiviCRM from 2d9360cfac to 7f58670e88
20:11 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@55d0b34]: Update mobileapps to a224e99 Ia80abe02490 (duration: 03m 29s)
20:07 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@55d0b34]: Update mobileapps to a224e99 Ia80abe02490
20:05 ladsgroup@deploy1001: Started deploy [ores/deploy@76fe25a]: ORES doesn't block hammering IPs (T204862)
19:41 hashar: web request 60 second timeout when deploying is filled as https://phabricator.wikimedia.org/T204871
19:30 hashar: while promoting group1 to 1.32.0-wmf.22 lot of web requests timed out at 60 seconds. Roughly from 19:24 to 19:28. But that is no more occurring | T191068
19:25 hashar@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.22 (duration: 00m 55s)
19:24 hashar@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.22
19:16 hashar@deploy1001: Synchronized php-1.32.0-wmf.22/extensions/Translate/tag/PageTranslationHooks.php: Avoid warnings and errors caused by x-pagetranslation-tag - T204797 (duration: 00m 58s)
18:43 gehel: starting elasticsearch eqiad cluster restart for new systemd unit
18:33 chasemp: add security_team_bot to acl*security_team in phab
17:52 robh: no errors in notebook1003 SEL
17:51 robh: notebook1003 unresponsive to icinga checks and serial console, rebooting
17:31 mutante: notebook1003 - starting failed nagios-nrpe-server
17:20 cmjohnson1: powering off mw1254 to reseat DIMM T204491
17:03 mobrovac@deploy1001: Finished deploy [restbase/deploy@b35727e]: Bug fix: return the stored headers in the key rev value bucket, take #4 (duration: 09m 03s)
16:54 mobrovac@deploy1001: Started deploy [restbase/deploy@b35727e]: Bug fix: return the stored headers in the key rev value bucket, take #4
16:54 mobrovac@deploy1001: Finished deploy [restbase/deploy@b35727e]: Bug fix: return the stored headers in the key rev value bucket, take #3 (duration: 05m 28s)
16:52 gehel: rolling restart of elasticsearch / logstash for new systemd unit completed - new systemd unit is elasticsearch_5@production-logstash-eqiad
16:50 sbisson@deploy1001: Synchronized php-1.32.0-wmf.22/extensions/Echo/includes/api/ApiCrossWiki.php: T204758 (duration: 00m 57s)
16:49 mobrovac@deploy1001: Started deploy [restbase/deploy@b35727e]: Bug fix: return the stored headers in the key rev value bucket, take #3
16:48 mobrovac@deploy1001: Finished deploy [restbase/deploy@b35727e]: Bug fix: return the stored headers in the key rev value bucket (duration: 05m 43s)
16:42 mobrovac@deploy1001: Started deploy [restbase/deploy@b35727e]: Bug fix: return the stored headers in the key rev value bucket
16:42 mobrovac@deploy1001: Finished deploy [restbase/deploy@b35727e]: Bug fix: return the stored headers in the key rev value bucket (duration: 03m 38s)
16:38 mobrovac@deploy1001: Started deploy [restbase/deploy@b35727e]: Bug fix: return the stored headers in the key rev value bucket
16:38 mobrovac@deploy1001: Finished deploy [restbase/deploy@b35727e] (dev-cluster): Bug fix: return the stored headers in the key rev value bucket (duration: 03m 01s)
16:35 mobrovac@deploy1001: Started deploy [restbase/deploy@b35727e] (dev-cluster): Bug fix: return the stored headers in the key rev value bucket
15:39 XioNoX: rebooting cr4-ulsfo for upgrade (not in prod)
15:24 XioNoX: rebooting cr3-ulsfo for upgrade (not in prod)
15:23 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Reduce db2057 load (duration: 00m 57s)
14:57 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2033 after alter table (duration: 00m 57s)
14:46 herron: updating MX bulk_smtp helo_data (gerrit 461193)
14:36 marostegui: Deploy alter table on db2033 (x1) enwiki
14:35 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2033 for alter table (duration: 00m 59s)
13:39 marostegui: Deploy schema change on dbstore2002:x1
13:25 marostegui: Deploy schema change on x1 eqiad master on enwiki (this will generate some lag on x1 eqiad) - T51593
13:03 arturo: T203177 add initial prometheus-openstack-exporter package to reprepro (v0.0.8-1)
12:50 marostegui: Deploy schema change on s3 eqiad master on mediawikiwiki (this will generate some lag on s3 eqiad) - T51593
12:18 gehel: rolling restart of elasticsearch / logstash for new systemd unit
12:09 gehel: rolling restart of relforge for new systemd unit
12:07 gehel: stopping puppet on all elasticsearch servers to deploy new systemd unit - https://gerrit.wikimedia.org/r/c/operations/puppet/+/440498
12:00 moritzm: reimaging mw1298 (spare host) to test reimages from cumin2001
10:26 marostegui: Deploy schema change on x1:testwiki - T51593
10:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1110 (duration: 00m 57s)
10:11 banyek@deploy1001: Synchronized wmf-config/db-codfw.php: Fully pool db2041 back into production (duration: 00m 57s)
10:07 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T203565: Weight adjust for S1 API hosts (duration: 00m 57s)
10:03 jynus: fixing db2033 grants
09:41 marostegui: Upgrade MariaDB and kernel on db1110
09:40 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2084:3315 db2053 db1096:3316 (duration: 00m 58s)
09:39 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db2084:3315 db2053 db1096:3316 (duration: 00m 59s)
09:26 godog: repair sdf on ms-be2043 - T199198
09:14 moritzm: rebooting mc1021 for kernel security update
09:04 moritzm: rebooting mc1020 for kernel security update
09:02 moritzm: updating intel-microcode on Debian jessie/stretch to 3.20180807a.1
08:39 moritzm: rebooting mc1019 for kernel security update
08:35 marostegui: Stop MySQL on db1110 for alert testing - T200509
08:04 moritzm: bounced ferm service on elastic2004
07:53 marostegui: Stop MySQL on db1110 for alert testing - T200509
07:45 marostegui: Stop MySQL on db1096:3316 for alert testing - T200509
07:31 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db2084:3315 db2053 db1110 db1096:3316 db1110 (duration: 00m 57s)
07:30 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2084:3315 db2053 db1110 db1096:3316 db1110 (duration: 00m 57s)
07:26 moritzm: reimaging mw2245 (spare host) to test reimages from cumin2001 (router policies have been updated)
07:25 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1109 (duration: 00m 57s)
07:12 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1109 (duration: 00m 58s)
07:12 marostegui: Disable puppet on databases to test new alerts - T200509 https://phabricator.wikimedia.org/T172489
07:10 jynus: restart db1109 for upgrade
06:56 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2041 with low load (duration: 00m 54s)
06:00 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2036 (duration: 00m 58s)
05:53 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2036 (duration: 00m 57s)
05:49 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2050 (duration: 00m 58s)
05:41 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2050 (duration: 00m 57s)
05:37 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2074 (duration: 00m 57s)
05:31 marostegui: Drop echo tables from s3 db2074 - T153638
05:31 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2074 (duration: 00m 57s)
05:25 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2057 (duration: 00m 58s)
05:19 marostegui: Drop echo tables from s7:kowiki - T153638
05:16 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2057 (duration: 02m 13s)
02:34 twentyafterfour@deploy1001: Finished scap: SWAT: full sync to update l10n for https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/WikimediaMessages/+/461230/ (duration: 61m 20s)
01:33 twentyafterfour@deploy1001: Started scap: SWAT: full sync to update l10n for https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/WikimediaMessages/+/461230/
01:28 twentyafterfour@deploy1001: Synchronized php-1.32.0-wmf.22/resources/src/mediawiki.util.js: SWAT: sync https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/461227/ (duration: 01m 00s)

2018-09-18

23:21 twentyafterfour@deploy1001: Synchronized wmf-config/: SWAT: tested on mwdebug1001, now syncing wmf-config settings to the whole cluster (duration: 00m 59s)
23:15 twentyafterfour: SWAT: deploying 3 patches for ebernhardson: 77deae12f 7009fe473 and 12cc420fc
22:10 bblack: depooled ulsfo, some uknown router issue
21:42 bblack: authdns1001 seems stable/fine running gdnsd-2.99.9-beta so far. If issues crop up later, don't hesitate to (a) downgrade back to stretch-backports gdnsd-2.3.0-1~bpo9+1 or (b) call me!
21:25 bblack: authdns1001: testing gdnsd version update (2.99.9-beta)
20:58 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Set ActorTableSchemaMigrationStage back to OLD on test wikis, mediawikiwiki (T188327, T204669) (duration: 00m 57s)
20:57 catrope@deploy1001: sync-file aborted: Set ActorTableSchemaMigrationStage back to OLD on test wikis, mediawikiwiki (duration: 00m 10s)
20:05 hashar@deploy1001: Synchronized php-1.32.0-wmf.22/extensions/OAuth/frontend/specialpages/SpecialMWOAuthListConsumers.php: Fix escapeForHtml method name - T204757 (duration: 00m 58s)
19:56 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@58f9ed3]: Log errors for HTTP error T203929 (duration: 00m 49s)
19:55 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@58f9ed3]: Log errors for HTTP error T203929
19:43 bblack: uploaded gdnsd-2.99.9-beta1-1+wmf1 to reprepro for stretch-wikimedia
19:20 hashar@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.32.0-wmf.22
19:18 ppchelko@deploy1001: Finished deploy [restbase/deploy@4c3128f] (dev-cluster): Remove restrictions table T203835, Split 25% PDF traffic to Proton T186748, Metrics endpoints improvements, don't monitor PDF endpoints, take 2, feed timing out (duration: 06m 32s)
19:11 ppchelko@deploy1001: Started deploy [restbase/deploy@4c3128f] (dev-cluster): Remove restrictions table T203835, Split 25% PDF traffic to Proton T186748, Metrics endpoints improvements, don't monitor PDF endpoints, take 2, feed timing out
18:49 ppchelko@deploy1001: Finished deploy [restbase/deploy@4c3128f]: Remove restrictions table T203835, Split 25% PDF traffic to Proton T186748, Metrics endpoints improvements, don't monitor PDF endpoints, take 2, feed timing out (duration: 06m 14s)
18:43 ppchelko@deploy1001: Started deploy [restbase/deploy@4c3128f]: Remove restrictions table T203835, Split 25% PDF traffic to Proton T186748, Metrics endpoints improvements, don't monitor PDF endpoints, take 2, feed timing out
18:40 ppchelko@deploy1001: Finished deploy [restbase/deploy@4c3128f]: Remove restrictions table T203835, Split 25% PDF traffic to Proton T186748, Metrics endpoints improvements, don't monitor PDF endpoints, take 2, feed timing out (duration: 07m 58s)
18:32 ppchelko@deploy1001: Started deploy [restbase/deploy@4c3128f]: Remove restrictions table T203835, Split 25% PDF traffic to Proton T186748, Metrics endpoints improvements, don't monitor PDF endpoints, take 2, feed timing out
18:31 ppchelko@deploy1001: Finished deploy [restbase/deploy@4c3128f]: Remove restrictions table T203835, Split 25% PDF traffic to Proton T186748, Metrics endpoints improvements, don't monitor PDF endpoints (duration: 08m 55s)
18:22 ppchelko@deploy1001: Started deploy [restbase/deploy@4c3128f]: Remove restrictions table T203835, Split 25% PDF traffic to Proton T186748, Metrics endpoints improvements, don't monitor PDF endpoints
18:08 ppchelko@deploy1001: Finished deploy [restbase/deploy@55100d4]: Remove restrictions table T203835, Split 25% PDF traffic to Proton T186748, Metrics endpoints improvements (duration: 03m 37s)
18:04 ppchelko@deploy1001: Started deploy [restbase/deploy@55100d4]: Remove restrictions table T203835, Split 25% PDF traffic to Proton T186748, Metrics endpoints improvements
17:29 mutante: scandium - move from role(spare) to role(parsoid_testing), making it equal to ruthenium (T201366)
16:45 addshore@deploy1001: Synchronized php-1.32.0-wmf.22/includes/watcheditem/WatchedItemStore.php: WatchedItemStore::countVisitingWatchersMultiple() fix T204729 (duration: 00m 57s)
16:43 addshore@deploy1001: Synchronized php-1.32.0-wmf.20/includes/watcheditem/WatchedItemStore.php: WatchedItemStore::countVisitingWatchersMultiple() fix T204729 (duration: 00m 59s)
16:34 XioNoX: delete `filter common-infrastructure4` on cr1/2-eqiad, unused/obsolete after T198623
16:33 mutante: mwmaint2001 - nagios-nrpe-server had status 'failed' and caused all NRPE Icinga checks to fail but recovered after simply starting it again
16:32 mutante: radon - re-enabled disabled puppet without reason (decom) T202040
16:32 mutante: mwmaint2001 - starting nagios-nrpe-server
16:22 XioNoX: delete `term cumin` from cr1/2-eqiad analytics filter (already permited by established-tcp term)
16:02 moritzm: installing policykit-1 security updates on jessie
15:55 elukey@deploy1001: Finished deploy [analytics/refinery@1a6235a]: Fix cron scrips from the NYC offsite (duration: 09m 32s)
15:53 XioNoX: add cumin2001 to mr* security policies - T204730
15:47 moritzm: installing spice security updates
15:46 elukey@deploy1001: Started deploy [analytics/refinery@1a6235a]: Fix cron scrips from the NYC offsite
15:26 XioNoX: add cumin2001 to labs-in4 on cr1/2-eqiad
15:23 hashar@deploy1001: Finished scap: Sync again testwiki to php-1.32.0-wmf.20, I might have screwed up l10ncache (duration: 06m 31s)
15:20 hashar: hashar scap for testwiki is actually php-1.32.0-wmf.22
15:19 mepps: updated process-control to c03857f6a6
15:17 hashar@deploy1001: Started scap: Sync again testwiki to php-1.32.0-wmf.20, I might have screwed up l10ncache
15:15 mepps: updated process-control to f9ab984c5d
15:07 hashar@deploy1001: Finished scap: testwiki to php-1.32.0-wmf.20 (duration: 35m 46s)
14:31 hashar@deploy1001: Started scap: testwiki to php-1.32.0-wmf.20
14:19 marostegui: Drop T153638_echo_XXX tables from db1123
14:02 marostegui: Drop T153638_echo_XXX tables from db1095:3313 - T153638
13:58 papaul: upgrading BIOS on ms-be2030
13:57 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T203565: Repool db1114 and db1119 (duration: 00m 49s)
13:56 godog: poweroff ms-be2030 - T204567
13:48 moritzm: updating intel-microcode on Debian jessie/stretch to 3.20180807a.1
13:43 moritzm: installing redis security updates on maps* servers
13:32 godog: repair sdd on ms-be2041 - T199198
13:31 marostegui: Rename echo_XXX tables on db1095 and db1123
13:26 moritzm: reimaging mw2245 (spare host) to test reimages from cumin2001
13:24 marostegui: Drop T153638_echo_XXX tables from db1078 - T153638
13:21 hashar: Cutting branches wmf/1.32.0-wmf.22 | T191068
13:10 marostegui: Drop T153638_echo_XXX tables from db1077 - T153638
12:12 moritzm: rebooting labnodepool1001 for kernel security update
10:36 jynus: stopping dbstore2002:s2 mariadb instance
10:31 jynus: stop db2041 to clone to dbstore2002
10:25 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2041 (duration: 00m 49s)
10:24 marostegui: Rename echo_XXX tables on db1077 and db1078 - T153638
10:12 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Tune the s3 and s5 database loads (duration: 00m 49s)
10:09 marostegui: Drop T153638_echo_XXX tables from dbstore2002:3313 - T153638
10:06 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1105 (duration: 00m 49s)
10:04 hashar: ci: updating Quibble Jenkins jobs to 0.0.26
10:01 marostegui: Rename echo_XXX tables on dbstore2002:3313 - T153638
09:37 marostegui: Drop T153638_echo_XXX tables from db2057 - T153638
09:10 jynus: stop mariadb (both instances) and restart db1105 for upgrade
08:54 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully depool db1105 (duration: 00m 49s)
08:45 banyek: stop replication & stop mysql on db1119 (preparing to clone db1114)
08:02 godog: bounce rsyslog on wezen/lithium, tls listener was timing out in icinga
08:02 jynus: stop and restart db1105 for upgrade
07:53 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1105 (duration: 00m 50s)
06:55 moritzm: installin zsh security updates
06:21 marostegui: Drop tmp_2 and tmp_3 index from wikidatawiki.recentchanges on dbstore2001, db2079, db2082,db2083 - T202764
06:09 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1013 after kernel and mariadb upgrade (duration: 00m 49s)
05:56 marostegui: Stop MySQL on es1013 to upgrade mariadb & kernel
05:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1013 for kernel and mariadb upgrade (duration: 00m 49s)
05:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1012 after kernel and mariadb upgrade (duration: 00m 50s)
05:35 marostegui: Stop MySQL on es1012 to upgrade mariadb & kernel
05:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1012 for kernel and mariadb upgrade (duration: 00m 49s)
05:28 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2081 - T202764 (duration: 00m 49s)
05:22 marostegui: Drop tmp_2 and tmp_3 index from wikidatawiki.recentchanges on db2081 - T202764
05:22 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2081 - T202764 (duration: 00m 49s)
05:14 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2080 - T202764 (duration: 00m 49s)
05:08 marostegui: Drop tmp_2 and tmp_3 index from wikidatawiki.recentchanges on db2080 - T202764
05:08 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2080 - T202764 (duration: 00m 51s)
03:01 ejegg: restarted fundraising CiviCRM jobs
02:46 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Sep 18 02:46:09 UTC 2018 (duration 10m 49s)
02:43 eileen_: civicrm revision changed from b0d7df4e60 to 2d9360cfac
02:37 ejegg: disabled fundraising CiviCRM jobs for 5.6 update
02:35 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.20) (duration: 13m 58s)

2018-09-17

23:32 mutante: ms-be2042 - reparing xfs - (T199198)
21:44 andrew@deploy1001: Finished deploy [horizon/deploy@3124052]: Fighting with scap more (duration: 01m 50s)
21:42 andrew@deploy1001: Started deploy [horizon/deploy@3124052]: Fighting with scap more
21:40 andrew@deploy1001: Finished deploy [horizon/deploy@3124052]: Fighting with scap (duration: 00m 10s)
21:40 andrew@deploy1001: Started deploy [horizon/deploy@3124052]: Fighting with scap
21:32 andrew@deploy1001: Finished deploy [horizon/deploy@3124052]: Cleaning up from some by-hand hacks (duration: 00m 10s)
21:32 andrew@deploy1001: Started deploy [horizon/deploy@3124052]: Cleaning up from some by-hand hacks
20:59 andrew@deploy1001: Finished deploy [horizon/deploy@3124052]: Disable unneeded network panels (duration: 03m 34s)
20:56 andrew@deploy1001: Started deploy [horizon/deploy@3124052]: Disable unneeded network panels
20:49 foks: add email address to User:Lanhiaze
20:44 ladsgroup@deploy1001: Finished deploy [ores/deploy@ae96071]: PoolCounter support: Let's get the party started (T160692) (duration: 28m 19s)
20:24 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@e0b7158]: Update mobileapps to d56e4cf (duration: 04m 16s)
20:19 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@e0b7158]: Update mobileapps to d56e4cf
20:15 ladsgroup@deploy1001: Started deploy [ores/deploy@ae96071]: PoolCounter support: Let's get the party started (T160692)
20:10 reedy@deploy1001: Synchronized wmf-config/interwiki.php: Updating interwiki cache (duration: 03m 26s)
18:56 stephanebisson: Starting mwscript extensions/PageTriage/maintenance/populateDraftQueue.php --wiki enwiki
18:55 stephanebisson: Stopped mwscript extensions/PageTriage/maintenance/populateDraftQueue.php --wiki enwiki
18:46 stephanebisson: Starting mwscript extensions/PageTriage/maintenance/populateDraftQueue.php --wiki enwiki
18:40 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T203184 (duration: 00m 50s)
18:35 sbisson@deploy1001: Synchronized php-1.32.0-wmf.20/extensions/Score/includes/Score.php: T203560 (duration: 00m 50s)
18:14 sbisson@deploy1001: Synchronized static/images/project-logos: SWAT (duration: 00m 50s)
17:22 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@6091de6]: Regular WDQS deployment (duration: 11m 45s)
17:10 smalyshev@deploy1001: Started deploy [wdqs/wdqs@6091de6]: Regular WDQS deployment
16:36 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting wgCommentTableSchemaMigrationStage = WRITE_NEW on test wikis, mw.org (T166733) (duration: 00m 50s)
16:18 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting wgActorTableSchemaMigrationStage = WRITE_BOTH on test wikis, mw.org (T188327) (duration: 00m 50s)
15:22 cmjohnson1: db1069 replacing disk in slot 7
15:20 jynus: stop db1061 (may generate s6 lag) for hw maintenance
15:19 jynus: db1061 (may generate s6 lag) for hw maintenance
14:31 banyek: upgrading (kernel & mariadb) db1119
14:27 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T203565: depooling db1119 (duration: 00m 49s)
13:40 godog: repair sdl on ms-be2041 - T199198
13:39 moritzm: removed intel-microcode 3.20180703 from apt.wikimedia.org/jessie-wikimedia (superseded by new release shipped via security.debian.org)
13:30 moritzm: removed intel-microcode 3.20180703 from apt.wikimedia.org/stretch-wikimedia (superseded by new release shipped via security.debian.org)
12:58 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T203565: depooling db1114 (duration: 00m 49s)
12:53 banyek: depooling db1114
12:08 jynus: stop and reimage db1071 (may generate s8 lag)
12:02 zeljkof: EU SWAT finished
12:01 zfilipin@deploy1001: Synchronized php-1.32.0-wmf.20/extensions/CongressLookup/: SWAT: Senator John McCain deceased (T203611) Add Sen. Jon Kyl (T203611) (duration: 00m 51s)
11:47 moritzm: installing ghostscript security updates on stretch
11:37 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable WikidataPageBanner for glwiki (T199713) (duration: 00m 50s)
11:29 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable FileExport on Korean Wikipedia (T204399) (duration: 00m 50s)
11:18 gilles@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T204478 Disable thumbnail prerendering on private wikis (duration: 00m 49s)
11:17 moritzm: installing curl security updates on trusty (Debian already fixed)
10:40 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 50s)
10:40 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 50s)
10:37 addshore: starting my maint script again
10:31 addshore: stopped my maint script
10:28 addshore: START addshore@mwmaint2001:~$ mwscript refreshLinks.php --wiki wikidatawiki --namespace 146 --e 56031711 54387042 # T198301 T195302
09:50 marostegui: Deploy schema change on db1062 - T203548
09:28 marostegui: Deploy schema change on s7 eqiad master (db1062) - T187089
09:16 jynus: stop and reimage db1068 (may generate s4 lag)
08:43 moritzm: installing postgresql security updates on maps*
08:29 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: New throttle rule for enwiki event (T204243) (duration: 00m 49s)
08:07 jynus: stop and reimage db1062 (may generate s7 lag)
07:46 moritzm: repooled wtp1043 (T196886)
07:31 godog: repair sdj on ms-be2040 - T199198
07:29 godog: force power on for ms-be2030
07:24 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1092 and db1099:3318 - T201011 (duration: 00m 49s)
07:24 marostegui: Deploy schema change on s8 - T201011
07:18 marostegui: Stop replication in sync on db1092 and db1099:3318 - T201011
07:18 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1092 and db1099:3318 - T201011 (duration: 00m 49s)
07:02 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1094 and db1098:3317 - T201011 (duration: 00m 49s)
06:53 marostegui: Stop replication in sync on db1094 and db1098:3317 - T201011
06:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1094 and db1098:3317 - T201011 (duration: 01m 04s)
06:33 marostegui: Deploy schema change on s7 (frwiktionary,metawiki) - T201011
06:31 marostegui: Deploy schema change on s4 - T201011
06:04 marostegui: Deploy schema change on s3 - T201011
05:47 marostegui: Deploy schema change on s3:testwikidatawiki directly on the master (db2043) - T201011
05:36 marostegui: Deploy schema change on s3:testwiki directly on the master (db2043) - T201011
05:28 marostegui: Deploy schema change on s1 eqiad master (db1067) - T67448 T114117 T51191
02:46 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Sep 17 02:46:18 UTC 2018 (duration 10m 52s)
02:35 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.20) (duration: 14m 20s)

2018-09-15

11:57 joal@deploy1001: Finished deploy [analytics/refinery@f4d1f24]: Deploying cluster to prevent failures from missing misc data (duration: 15m 19s)
11:42 joal@deploy1001: Started deploy [analytics/refinery@f4d1f24]: Deploying cluster to prevent failures from missing misc data
04:53 SMalyshev: re-pooled wdqs2003
01:08 SMalyshev: depooled wdqs2003 to let it catch up

2018-09-14

21:19 andrew@deploy1001: Finished deploy [horizon/deploy@56340cd]: Fix proxy creation in neutron regions (duration: 03m 31s)
21:16 andrew@deploy1001: Started deploy [horizon/deploy@56340cd]: Fix proxy creation in neutron regions
21:03 mutante: ACKed memory error alert on wtp2011 - existing ticket but fresh alert popped up 9h ago (T200678)
20:12 ejegg: updated payments-wiki from 44b9409104 to 43923fadd9
15:57 jynus: set thread_pool_size to 64 at db2047
14:42 ema: lvs3002: restart pybal to remove misc-web T164609
14:28 ema: lvs2002: restart pybal to remove misc-web T164609
14:23 ema: lvs1002: restart pybal to remove misc-web T164609
14:11 jynus: shutting down db1062 for dc maintenance T204302
13:58 jynus: stop and restart db1118 for upgrade
12:51 banyek@deploy1001: Synchronized wmf-config/db-codfw.php: T204127: Weight Adjust db2068 (duration: 00m 50s)
09:47 banyek: operation stopping db1070 in preparation for reimage
08:45 marostegui: Deploy schema change on s7 eqiad master (db1062) - T89737
08:34 mobrovac@deploy1001: Started restart [cpjobqueue/deploy@32a81be]: (no justification provided)
08:29 moritzm: rebooting acamar for kernel tests
08:27 elukey: reboot kafka100[1-3] (eventbus eqiad) for kernel upgrades
08:20 jynus: stopping and restarting db1069 for upgrade (x1 eqiad master)
08:15 banyek@deploy1001: Synchronized wmf-config/db-codfw.php: T204127: Weight Adjust db2068 (duration: 00m 50s)
07:49 marostegui: Deploy schema change on s4 eqiad master (db1068) - T67448 T114117 T51191
07:40 moritzm: rebooting mwmaint1001 for kernel security update
07:28 jynus: stopping db1062 mariadb in preparation for reimage
07:11 banyek@deploy1001: Synchronized wmf-config/db-codfw.php: T204127: Weight Adjust db2068 (duration: 00m 50s)
05:34 marostegui: Deploy schema change on s3 eqiad master - T187089
05:29 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2050 - T189101 (duration: 00m 49s)
05:17 marostegui: Stop replication in sync on db1075 and db2050 - T189101
05:16 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2050 - T189101 (duration: 00m 49s)
05:10 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Increase weight for db2054 - T204127 (duration: 00m 50s)
05:06 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Increase weight for db2068 - T204127 (duration: 00m 52s)

2018-09-13

21:55 urandom: running fstrim --all on restbase1013 -- T89584
21:51 urandom: running fstrim --all on restbase1008 -- T89584
21:35 urandom: running fstrim --all on restbase1011 -- T89584
21:33 urandom: running fstrim --all on restbase1007 -- T89584
19:31 herron: shutting down old puppet compiler instances in puppet3-diffs project T191438
19:24 herron: moving puppet-compiler.wmlabs.org proxy from puppet3-diffs to puppet-diffs project T191438
18:40 cmjohnson1: updating f/w cloudvirt1019 disabled icinga checks
17:42 krinkle@deploy1001: Synchronized php-1.32.0-wmf.20/resources/src/mediawiki.widgets/mw.widgets.CheckMatrixWidget.js: I1f92479bf1, T203325 (duration: 00m 50s)
17:41 krinkle@deploy1001: Synchronized php-1.32.0-wmf.20/includes/widget/CheckMatrixWidget.php: I1f92479bf1, T203325 (duration: 00m 49s)
17:40 krinkle@deploy1001: Synchronized php-1.32.0-wmf.20/includes/htmlform/fields/HTMLCheckMatrix.php: I1f92479bf1, T203325 (duration: 00m 51s)
17:38 XioNoX: enable cr2:xe-4/0/0 (to asw-a) - T203719
17:31 XioNoX: disable cr2:xe-4/0/0 (to asw-a) for optics replacement (round 2, 1st one didn't clear the errors, need to do the other side) - T203719
17:29 mutante: radium - removing tor package, clearing systemd failed units, to clear Icinga alerts from this host that is to be decomed
17:29 XioNoX: enable cr2:xe-4/0/0 (to asw-a) for optics replacement - T203719
17:25 XioNoX: disable cr2:xe-4/0/0 (to asw-a) for optics replacement - T203719
16:45 godog: repair sde on ms-be1041 - T199198
15:26 jynus: restarting again db1075 for proper kernel upgrade
15:25 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Increase weight for db2054 - T204127 (duration: 00m 49s)
15:15 herron: repool mx1001 — upgrade to stretch complete T175361
15:02 akosiaris: upload blubber_0.5.0-1 to apt.wikimedia.org/{stretch,jessie}-wikimedia/main T203121
14:47 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Increase weight for db2054 - T204127 (duration: 00m 50s)
14:36 marostegui: Reload haproxy on dbproxy1010 to depool labsdb1011 - https://phabricator.wikimedia.org/T174047
14:35 bstorm_: switch labstore1004 and labstore1005 to using the cfq scheduler on the DRBD volumes
14:32 moritzm: rebooting dns1002 for kernel security update
13:59 herron: depool mx1001 and relay queued messages to mx2001 for upgrade to stretch T175361
13:51 moritzm: rebooting dns1001 for kernel security update
13:28 andrew@deploy1001: Finished deploy [horizon/deploy@c9c7a56]: Improvements for VM creation in eqiad1, T167293 (take two) (duration: 04m 10s)
13:24 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Increase weight for db2054 - T204127 (duration: 00m 49s)
13:23 andrew@deploy1001: Started deploy [horizon/deploy@c9c7a56]: Improvements for VM creation in eqiad1, T167293 (take two)
13:20 andrew@deploy1001: Finished deploy [horizon/deploy@12aa2d3]: Improvements for VM creation in eqiad1, T167293 (duration: 00m 13s)
13:20 andrew@deploy1001: Started deploy [horizon/deploy@12aa2d3]: Improvements for VM creation in eqiad1, T167293
13:18 marostegui: Deploy schema change on s2 eqiad master (db1066) - T89737
13:07 marostegui: Deploy schema change on s8 eqiad master (db1071) - T187089
13:03 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2085:3318 - T189101 (duration: 00m 49s)
12:55 marostegui: Stop db1071 (s8 eqiad master) and db2085:3318 in sync - T189101
12:54 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2085:3318 - T189101 (duration: 00m 49s)
12:45 marostegui: Reload haproxy on dbproxy1011 to repool labsdb1009 - https://phabricator.wikimedia.org/T174047
11:47 moritzm: rebooting dns recursors in codfw for kernel security update
11:38 jynus: stopping eqiad s3 replication and shuttinf down db1075 in preparation for reimage T148507
10:49 banyek: shutting down db2068 and dbstore2001:s7 for cloning
10:46 moritzm: uploaded debdeploy 0.0.99.5-1+deb9u1 to apt.wikimedia.org/stretch-wikimedia
10:33 marostegui: Deploy schema change on s4 eqiad master (db1068) - T187089
10:33 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Slowly repool db2054 - T204127 (duration: 00m 50s)
09:57 marostegui: Deploy schema change on s4 eqiad master (db1068) - T89737
09:54 godog: repair sdd on ms-be2043 - T199198
09:44 marostegui: Enable GTID on eqiad masters - T189107
09:41 filippo@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp2020.codfw.wmnet
09:34 moritzm: rebooting dns recursors in esams for kernel security update
09:24 marostegui: Reload haproxy on dbproxy1011 to depool labsdb1009 - https://phabricator.wikimedia.org/T174047
09:08 marostegui: Deploy schema change on s8 eqiad master (db1071) - T89737
08:59 hashar: Bumped CI jobs based on Debian Stretch to use Chrome 69 and Firefox 60 | T203902
08:51 moritzm: rebooting dns recursors in ulsfo for kernel security update
08:49 marostegui: Deploy schema change on s5 eqiad master (db1070) - T89737
08:41 moritzm: removed labvirt1019/labvirt1020 from debmonitor (T204004)
08:36 marostegui: Deploy schema change on s6 eqiad master (db1061) - T89737
08:29 moritzm: installing curl security updates
08:21 marostegui: Deploy schema change on s6 eqiad master (db1061) - T187089
08:16 marostegui: Stop replication on s4 eqiad master (db1068) and deploy a schema change - this will generate lag on s4 eqiad - T144010
08:08 marostegui: Deploy schema change on s5 eqiad master - T187089
08:01 moritzm: rebooting dns recursors in eqsin for kernel security update
08:01 marostegui: Disconnect replication eqiad -> codfw on s1-s8, x1, es2, es3 - T189107
07:47 elukey: reboot kafka10[12-23] (old analytics cluster) for kernel upgrades
07:46 elukey: execute apt-get autoremove on notebook* to remove old nginx packages (not used anymore)
07:25 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Tune db weights even more (duration: 00m 50s)
06:56 elukey: reboot notebook100[3,4] for kernel upgrades - T203165
06:42 elukey: reboot stat100[4-6] for kernel upgrades - T203165
05:11 marostegui: Stop MySQL on db2054 and dbstore2001:3317 to clone db2054 - T204127
02:46 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Sep 13 02:46:16 UTC 2018 (duration 10m 46s)
02:35 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.20) (duration: 14m 03s)
02:24 ejegg: restarted fundraising CiviCRM jobs
01:48 ejegg: updated CiviCRM from 2a3583cca1 to b0d7df4e60
01:45 ejegg: disabled fundraising CiviCRM jobs
00:00 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@7e5e537]: Deploy Blazegraph & Updater for T202765 and T203646 handling (duration: 23m 45s)

2018-09-12

23:36 smalyshev@deploy1001: Started deploy [wdqs/wdqs@7e5e537]: Deploy Blazegraph & Updater for T202765 and T203646 handling
22:41 Trey314159: reindexing Esperanto wikis on elastic@codfw and elastic@eqiad complete (T203005)
19:50 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Decrease db2061 load (duration: 00m 50s)
19:13 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Decrease db2065 load (duration: 00m 49s)
18:24 Trey314159: reindexing Esperanto wikis on elastic@codfw and elastic@eqiad (T203005)
17:59 otto@deploy1001: Finished deploy [analytics/refinery@407da92]: Deploying refinery-source 0.0.74 jars for T203804 (duration: 06m 11s)
17:53 otto@deploy1001: Started deploy [analytics/refinery@407da92]: Deploying refinery-source 0.0.74 jars for T203804
17:52 otto@deploy1001: Finished deploy [analytics/refinery@407da92]: Deploying refinery-source 0.0.74 jars for T203804 (duration: 01m 41s)
17:50 otto@deploy1001: Started deploy [analytics/refinery@407da92]: Deploying refinery-source 0.0.74 jars for T203804
17:38 akosiaris: restart icinga T196336
16:53 ottomata: restarting eventlogging processors to pick up change to mysql whitelist
16:52 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Decrease weight for db2065 (duration: 00m 49s)
16:48 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=wtp2020.codfw.wmnet
16:39 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: cirrussearch: eqiad -> local (codfw) (duration: 00m 50s)
16:32 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Reduce db2065, db2061 weights (duration: 00m 48s)
16:14 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Decrease weight for db2049 (duration: 00m 49s)
16:06 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Reduce db2056 weight (duration: 00m 49s)
15:56 akosiaris: switch s*-master DNS records to codfw
15:51 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Decrease weight for db2049 (duration: 00m 50s)
15:38 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: 43b98e414 (duration: 00m 50s)
15:32 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Decrease weight for db2070 (duration: 00m 49s)
15:24 END: (PASS) - Cookbook sre.switchdc.mediawiki.08-restart-parsoid (exit_code=0) (volans@sarin)
15:22 oblivian@sarin: conftool action : set/weight=15; selector: dc=codfw,cluster=api_appserver,service=apache2,name=mw21.*
15:19 oblivian@sarin: conftool action : set/weight=20; selector: dc=codfw,cluster=api_appserver,service=apache2,name=mw22[2-9].*
15:17 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Tweak weights in s1 and s2 (duration: 00m 50s)
15:15 _joe_: repool mw2141
15:14 _joe_: depooling mw2141 for investigation
15:08 START: - Cookbook sre.switchdc.mediawiki.08-restart-parsoid (volans@sarin)
15:05 END: (PASS) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=0) (volans@sarin)
15:05 START: - Cookbook sre.switchdc.mediawiki.08-restore-ttl (volans@sarin)
15:05 END: (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) (volans@sarin)
15:03 START: - Cookbook sre.switchdc.mediawiki.08-start-maintenance (volans@sarin)
14:59 END: (PASS) - Cookbook sre.switchdc.mediawiki.08-update-tendril (exit_code=0) (volans@sarin)
14:58 START: - Cookbook sre.switchdc.mediawiki.08-update-tendril (volans@sarin)
14:52 END: (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) (volans@sarin)
14:51 MediaWiki: read-only period ends at: 2018-09-12 14:51:58.936291 (volans@sarin)
14:51 START: - Cookbook sre.switchdc.mediawiki.07-set-readwrite (volans@sarin)
14:51 END: (PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0) (volans@sarin)
14:51 START: - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (volans@sarin)
14:50 END: (PASS) - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions (exit_code=0) (volans@sarin)
14:50 START: - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions (volans@sarin)
14:50 END: (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-traffic (exit_code=0) (volans@sarin)
14:48 START: - Cookbook sre.switchdc.mediawiki.04-switch-traffic (volans@sarin)
14:47 END: (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0) (volans@sarin)
14:47 START: - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (volans@sarin)
14:47 END: (PASS) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=0) (volans@sarin)
14:46 START: - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (volans@sarin)
14:44 END: (FAIL) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=99) (volans@sarin)
14:44 MediaWiki: read-only period starts at: 2018-09-12 14:44:24.536913 (volans@sarin)
14:44 START: - Cookbook sre.switchdc.mediawiki.02-set-readonly (volans@sarin)
14:41 END: (PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0) (volans@sarin)
14:41 START: - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (volans@sarin)
14:40 END: (PASS) - Cookbook sre.switchdc.mediawiki.00-wipe-and-warmup-caches (exit_code=0) (volans@sarin)
14:34 jynus: stopping mariadb at db2054
14:29 START: - Cookbook sre.switchdc.mediawiki.00-wipe-and-warmup-caches (volans@sarin)
14:13 jynus: starting replication on db2068
14:07 END: (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) (volans@sarin)
14:07 START: - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (volans@sarin)
14:06 END: (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) (volans@sarin)
14:06 START: - Cookbook sre.switchdc.mediawiki.00-disable-puppet (volans@sarin)
14:04 volans: starting switchover meny "cookbook sre.switchdc.mediawiki eqiad codfw" on sarin - T203777
14:03 jynus: stopping db2068
13:59 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2054 and db2068 looks like they are stuck (duration: 00m 50s)
13:52 ema: restart varnish-be on cp1081 due to mbox lag and 503s
13:24 marostegui: Reload haproxy on dbproxy1010 to repool labsdb1010 - T174047
13:11 otto@deploy1001: Finished deploy [eventlogging/analytics@5c6fab6]: Support loading plugins in eventlogging-processor - T203596 (duration: 00m 05s)
13:11 otto@deploy1001: Started deploy [eventlogging/analytics@5c6fab6]: Support loading plugins in eventlogging-processor - T203596
13:08 jynus@deploy1001: Synchronized wmf-config/CommonSettings.php: Disabling CentralNotice translations (duration: 00m 50s)
12:37 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1110 with original load (duration: 00m 50s)
12:23 marostegui: Reload haproxy on dbproxy1010 to depool labsdb1010 - https://phabricator.wikimedia.org/T174047
12:17 godog: repair sdm / sdj on ms-be2042 - T199198
12:07 elukey: delete meitnerium.wikimedia.org's ganeti VM (decommissioned) - T203087
11:15 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1110 with low load (duration: 00m 50s)
10:56 moritzm: uploaded poolcounter 1.0.4+deb9u1 to apt.wikimedia.org (T199876)
10:54 jynus: restart db1110 for upgrade
10:32 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1121, depool db1110 (duration: 00m 50s)
09:57 jynus: restart db1121 for upgrade
09:46 marostegui: Drop users: connect, status and fabmigrate from dbstore1002 - T200801
09:07 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1098 (s6 and s7) and depool db1121 (duration: 00m 49s)
08:36 godog: repair sdc on ms-be2041 - T199198
08:05 jynus: stopping db1098 (both db instances) for maintenance
08:00 mobrovac@deploy1001: Started restart [proton/deploy@ecb9a0e]: (no justification provided)
07:53 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1098 (duration: 00m 50s)
07:45 addshore@deploy1001: Synchronized php-1.32.0-wmf.20/extensions/Wikibase/lib/includes/Store/: REVERT: Debug logging for T97368 gerrit:459905 (duration: 00m 51s)
07:38 addshore@deploy1001: Synchronized php-1.32.0-wmf.20/extensions/Wikibase/lib/includes/Store/: Debug logging for T97368 gerrit:459905 (duration: 00m 53s)
06:57 mobrovac@deploy1001: Finished deploy [proton/deploy@ecb9a0e]: Update to Puppeteer v1.7.0 and fix browser connection abort handling - T181623 (duration: 00m 58s)
06:56 mobrovac@deploy1001: Started deploy [proton/deploy@ecb9a0e]: Update to Puppeteer v1.7.0 and fix browser connection abort handling - T181623
06:35 moritzm: installing confuse security updates
02:38 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Sep 12 02:38:53 UTC 2018 (duration 10m 44s)
02:28 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.20) (duration: 08m 30s)

2018-09-11

22:06 legoktm: legoktm@mwmaint1001:~$ mwscript deleteEqualMessages.php --wiki fixcopyrightwiki --delete --lang-code='*'
21:56 legoktm@deploy1001: Finished scap: i18n updates for fixcopyright (duration: 32m 21s)
21:23 legoktm@deploy1001: Started scap: i18n updates for fixcopyright
21:22 legoktm@deploy1001: Synchronized php-1.32.0-wmf.20/skins/EUCopyrightCampaignSkin/: add og:image meta tag - https://gerrit.wikimedia.org/r/459836 (duration: 00m 51s)
21:03 mutante: restarted apache on mwdebug1002, running puppet
19:38 ema: switch all services to codfw only
19:31 ema: switch restbase to active/active
19:20 ema: depool eqiad from edge traffic
19:06 ema: route esams via codfw
17:10 XioNoX: delete BGP sessions with old AS10089 router on cr1-eqsin
16:53 godog: repair sdd on ms-be1043 - T199198
16:27 mutante: added gtirloni to acl*sre-team on Phabricator (T203489)
16:17 godog: correction, sdk1 on ms-be1041 - T199198
16:16 godog: repair sdd1 on ms-be1043 - T199198
15:06 godog: serve switch originals and thumbs from codfw only
15:00 godog: begin switching swift to codfw
14:40 END: (PASS) - Cookbook sre.switchdc.services.02-restore-ttl (exit_code=0) (akosiaris@sarin)
14:40 START: - Cookbook sre.switchdc.services.02-restore-ttl (akosiaris@sarin)
14:38 END: (PASS) - Cookbook sre.switchdc.services.01-switch-dc (exit_code=0) (akosiaris@sarin)
14:38 Switching: services parsoid, restbase, restbase-async, mobileapps, apertium, citoid, cxserver, eventstreams, graphoid, mathoid, proton, pdfrender, recommendation-api, zotero, eventbus, ores, wdqs, wdqs-internal: eqiad => codfw (akosiaris@sarin)
14:38 START: - Cookbook sre.switchdc.services.01-switch-dc (akosiaris@sarin)
14:38 END: (PASS) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=0) (akosiaris@sarin)
14:32 START: - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (akosiaris@sarin)
14:31 END: (FAIL) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=99) (akosiaris@sarin)
14:31 START: - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (akosiaris@sarin)
14:31 END: (FAIL) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=99) (akosiaris@sarin)
14:31 START: - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (akosiaris@sarin)
13:21 END: (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0) (volans@sarin)
13:21 START: - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (volans@sarin)
13:14 END: (PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0) (volans@sarin)
13:14 START: - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (volans@sarin)
13:12 END: (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) (volans@sarin)
13:12 START: - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (volans@sarin)
13:08 volans: performing some additional switchdc live test
13:02 volans: upgraded spicerack to version 0.0.8 on sarin/neodymium - T199079
12:28 gehel: restarting tilerator on maps1* (eqiad) - heap memory exceeded
12:09 moritzm: installing jq security updates on trusty
12:01 dereckson@deploy1001: Synchronized wmf-config/throttle.php: Update Informatika SZŠ Chomutov throttle rule (T203909) (duration: 00m 50s)
12:00 dereckson@deploy1001: sync-file aborted: Update Informatika SZŠ Chomutov throttle rule (duration: 00m 04s)
11:49 volans: uploaded spicerack_0.0.8-1{,+deb9u1} to apt.wikimedia.org {jessie,stretch}-wikimedia - T199079
11:37 moritzm: restarting hhvm on mw1261-mw1265 to pick up curl security updates
11:25 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set category collation to uca-et-u-kn on Estonian-language wikis (T202977) (duration: 00m 50s)
10:37 marostegui: Disable GTID on all codfw masters (sX, x1, esX) (not in db2040 as it is not enabled there) T189107
10:36 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3315, db1100 (duration: 00m 49s)
10:30 tgr@deploy1001: Finished scap: T204018 update i18n on fixcopyrightwiki (duration: 31m 01s)
10:27 marostegui: db1096:3315 and db1100 were test pages - NO MORE TEST PAGES ARE EXPECTED FROM NOW ON - T200509
10:16 marostegui: Stop replication on db2075 to test the paging (should not page)
10:14 marostegui: Stop replication on db1100 to test the paging
10:03 marostegui: Stop replication on db2084:3315 for alert testing
09:59 tgr@deploy1001: Started scap: T204018 update i18n on fixcopyrightwiki
09:54 marostegui: Stop replication on db1096:3315 for paging testing
09:25 moritzm: installing curl security updates
08:39 godog: repair xfs on sdh/sdc on ms-be2040 - T199198
08:27 marostegui: Stop replication on db1100 for new alert testing (this should generate a page) T200509
08:25 jynus: restarting replication on db2034 after testing dc switch replication sync step
08:14 marostegui: Stop replication on db1096:3315 for new alert testing (this should generate a page) T200509
08:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3315, db1100 (duration: 00m 49s)
08:13 jynus: stopping replication on db2034 to test dc switch replication sync step
08:12 marostegui@deploy1001: sync-file aborted: Depool db1096:3315, db1100 (duration: 00m 08s)
08:05 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2084:3315 and db2075 (duration: 00m 49s)
07:49 marostegui: Stop replication on db2075 for alert testing T200509
07:33 marostegui: Stop replication on db2084:3315 for alert testing T200509
07:27 marostegui: Disable puppet on all the DBs for alert testing - https://phabricator.wikimedia.org/T200509
05:57 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2084:3315 and db2075 (duration: 00m 51s)
02:45 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Sep 11 02:45:31 UTC 2018 (duration 10m 44s)
02:44 Krinkle: krinkle@mwmaint1001$ mwscript deleteEqualMessages.php --wiki fixcopyrightwiki --delete --lang-code='*'
02:34 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.20) (duration: 13m 27s)
01:00 Krinkle: krinkle@mwmaint1001$ mwscript deleteEqualMessages.php --wiki fixcopyrightwiki --delete --lang-code='*'

2018-09-10

22:56 andrewbogott: rebooting/reimaging labvirt1019 and 1020 for T204003
22:43 ejegg: updated CiviCRM from 52873b5dcc to 2a3583cca1
22:00 ejegg: updated payments-wiki from 05d796e844 to 44b9409104
21:24 bawolff@deploy1001: Synchronized php-1.32.0-wmf.20/extensions/OpenStackManager/special/SpecialNovaSudoer.php: T203885 (duration: 00m 50s)
20:29 volans: running warmup script against codfw appservers
19:03 brion: stopped requeueTranscodes.php job on mwmaint1001.eqiad pending dc switch
17:42 XioNoX: trunk cloud-instances1-b-eqiad to cloudnet1003/4:eth1 - T202636
17:03 elukey: stop ircecho to avoid excessive spamming
16:37 elukey: rolling restart of aqs on aqs* to pick new druid backend settings
15:24 gehel: reducing elasticsearch low watermark to 75% on cirrus / eqiad cluster
15:07 elukey: reboot analytics100[1,2] for kernel security upgrades
14:32 elukey: reboot analytics1003 for kernel + openjdk-8 upgrades
14:28 anomie@deploy1001: Synchronized php-1.32.0-wmf.20/includes/parser/ParserOutput.php: Backport for T203716 (duration: 00m 50s)
14:26 hashar: Switching CI Jenkins mail server from mx1001 to localhost | T203607
13:45 marostegui: Drop user metrics and wikilytics from dbstore1002
12:23 moritzm: restarting hhvm on mw1261-mw1265 to pick up lcms security update
12:09 moritzm: installing lcms2 security updates
11:57 moritzm: installing chromium on proton* (tested on deployment-prep with the new release)
11:44 volans: completed execution of "cookbook sre.switchdc.mediawiki --live-test codfw eqiad" - T199073
11:27 END: (FAIL) - Cookbook sre.switchdc.mediawiki.08-restart-parsoid (exit_code=99) (volans@sarin)
11:21 START: - Cookbook sre.switchdc.mediawiki.08-restart-parsoid (volans@sarin)
11:19 END: (PASS) - Cookbook sre.switchdc.mediawiki.08-update-tendril (exit_code=0) (volans@sarin)
11:18 START: - Cookbook sre.switchdc.mediawiki.08-update-tendril (volans@sarin)
11:16 ladsgroup@deploy1001: Synchronized php-1.32.0-wmf.20/extensions/CentralAuth/maintenance/deleteLocalPasswords.php: SWAT: Fix typo (T201009) (duration: 00m 50s)
11:14 END: (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) (volans@sarin)
11:13 START: - Cookbook sre.switchdc.mediawiki.08-start-maintenance (volans@sarin)
11:11 END: (PASS) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=0) (volans@sarin)
11:11 START: - Cookbook sre.switchdc.mediawiki.08-restore-ttl (volans@sarin)
11:09 END: (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) (volans@sarin)
11:09 [DRY-RUN]: MediaWiki read-only period ends at: 2018-09-10 11:09:40.587760 (volans@sarin)
11:09 START: - Cookbook sre.switchdc.mediawiki.07-set-readwrite (volans@sarin)
11:08 moritzm: uploaded a co-installable 4.14 kernel to apt.wikimedia.org (to be used for installer tests)
11:06 ladsgroup@deploy1001: Synchronized wmf-config/CommonSettings.php: Add ['null'] (T201009) (duration: 00m 50s)
10:57 END: (PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0) (volans@sarin)
10:57 START: - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (volans@sarin)
10:52 END: (PASS) - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions (exit_code=0) (volans@sarin)
10:52 START: - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions (volans@sarin)
10:50 END: (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-traffic (exit_code=0) (volans@sarin)
10:48 START: - Cookbook sre.switchdc.mediawiki.04-switch-traffic (volans@sarin)
10:41 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 50s)
10:40 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 50s)
10:38 END: (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0) (volans@sarin)
10:38 START: - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (volans@sarin)
10:29 END: (PASS) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=0) (volans@sarin)
10:28 START: - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (volans@sarin)
10:27 moritzm: uploaded linux-meta 1.20+deb9u2 for apt.wikimedia.org/stretch-wikimedia
10:25 END: (PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0) (volans@sarin)
10:25 [DRY-RUN]: MediaWiki read-only period starts at: 2018-09-10 10:25:20.558408 (volans@sarin)
10:25 START: - Cookbook sre.switchdc.mediawiki.02-set-readonly (volans@sarin)
10:12 END: (FAIL) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=99) (volans@sarin)
10:12 START: - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (volans@sarin)
10:02 END: (PASS) - Cookbook sre.switchdc.mediawiki.00-wipe-and-warmup-caches (exit_code=0) (volans@sarin)
09:53 START: - Cookbook sre.switchdc.mediawiki.00-wipe-and-warmup-caches (volans@sarin)
09:36 END: (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) (volans@sarin)
09:36 START: - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (volans@sarin)
09:32 END: (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) (volans@sarin)
09:31 START: - Cookbook sre.switchdc.mediawiki.00-disable-puppet (volans@sarin)
09:30 volans: starting execution of "cookbook sre.switchdc.mediawiki --live-test codfw eqiad" - T199073
08:22 marostegui: Drop users metric and wikilytics from core databases
08:04 marostegui: Drop unused root grants from core servers
07:46 moritzm: installing ghostscript security updates
07:18 volans: restarted pdfrender on scb2004 - T174916
07:04 oblivian@deploy1001: Synchronized wmf-config/throttle.php: Deploy throttle rule for Czech School T203909 (duration: 00m 51s)
02:51 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Sep 10 02:51:00 UTC 2018 (duration 10m 52s)
02:40 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.20) (duration: 13m 48s)
00:46 tstarling@deploy1001: Synchronized wmf-config/set-time-limit.php: (no justification provided) (duration: 00m 49s)
00:12 tstarling@deploy1001: Synchronized w/infinite-loop.php: Testing for T97192 (duration: 00m 48s)
00:07 tstarling@deploy1001: Synchronized wmf-config/PhpAutoPrepend.php: T97192 (duration: 00m 49s)
00:04 tstarling@deploy1001: Synchronized wmf-config/set-time-limit.php: T97192 (duration: 00m 52s)

2018-09-08

09:45 gtirloni: tools restarted cron and truncated /var/log/exim4/paniclog (T196137)
04:22 krinkle@deploy1001: Synchronized multiversion/: Ia27a8f7ed612f (duration: 00m 49s)
04:16 krinkle@deploy1001: Synchronized wmf-config/profiler.php: Ia27a8f7ed612f (duration: 00m 54s)
01:10 mutante: also rsyncing /var/lib/tor-instances/ data for second instance and restarting service (T196701)
00:53 mutante: radium - stopping rsync.service
00:27 mutante: torrelay1001 - reset internal state (sighup) with "arm" and pressing x twice
00:18 mutante: to watch what is happenin on torrelay1001 - sudo -u debian-tor arm - if asked for password it's in passwords::tor in private
00:16 mutante: tor relay switched over from radium to torrelay1001, fixed /var/lib/tor permissions, restarted service, flipped DNS CNAME (5M TTL), traffic can be seen with "arm", monitoring all green (T196701)

2018-09-07

23:26 mutante: ms-be2042 - repairing /dev/sdj1 (T199198)
23:25 mutante: ms-be2041 - repairing /dev/sdh1 (T199198)
23:23 mutante: ms-be1041 - repairing xfs per https://wikitech.wikimedia.org/wiki/Swift/How_To#Repair_xfs_free_blocks_counter_corruption (T199198)
22:17 mutante: gerrit - restarting for config change to move log files to /var/log/gerrit/
22:16 mutante: - cobalt (gerrit) - applying change to move log file location, manually moved logs to /var/log/gerrit, remove old log dir, let puppet re-create it, like on gerrit2001
21:31 mutante: gerrit2001, moving gerrit logfiles to /var/log/gerrit, removing old gerrit logdir, letting puppet re-create it as symlink
18:20 mutante: LDAP: correction, 'monipe' replaced with 'onimisionipe' in wmf group (T202708)
18:12 mutante: LDAP: added user 'monipe' to group 'wmf' (T202708)
18:02 legoktm@deploy1001: Synchronized php-1.32.0-wmf.20/extensions/EUCopyrightCampaign/: Update MEPs - https://gerrit.wikimedia.org/r/458628 (for real this time) (duration: 00m 50s)
17:52 legoktm@deploy1001: Synchronized php-1.32.0-wmf.20/extensions/EUCopyrightCampaign/: Update MEPs - https://gerrit.wikimedia.org/r/458628 (duration: 00m 50s)
17:45 XioNoX: apply firewall changes on pfw3-eqiad - T203793
17:40 XioNoX: apply firewall changes on pfw3-codfw - T203793
16:42 XioNoX: explicitely permit install1002/2002:80 in filter labs-in4 on cr1/2-eqiad - T190424
14:56 moritzm: uploaded linux-meta 1.20+deb9u1 to apt.wikimedia.org/stretch-wikimedia (provides a new meta package for Linux 4.14)
14:29 moritzm: installing PHP security updates on krypton
14:27 moritzm: installing libtirpc security updates on trusty
12:35 elukey: reboot kafka200[2,3] (eventbus codfw) for kernel + openjdk-8 upgrades
10:03 Amir1: ladsgroup@mwmaint1001:~$ mwscript extensions/CentralAuth/maintenance/deleteLocalPasswords.php --wiki=fawiki --user Ladsgroup --prefix (T201009)
09:38 hashar@deploy1001: Synchronized php-1.32.0-wmf.20/extensions/UniversalLanguageSelector: Revert "Simplify by using native JavaScript instead of jQuery" - T203750 (duration: 00m 55s)
08:49 ema: passive checks awol on einsteinium, restarting icinga -- T196336
08:45 jynus: reloading apache with bad config for tendril for testing (small downtime)
08:33 marostegui: Rebooting haproxies to pick up new config after all the tests - T201021
08:16 banyek: genarting false alert about https auth on dbmonitor1001
07:24 moritzm: rebooting mw2270-mw2290 for kernel security updates
06:51 moritzm: rebooting mw2240-mw2269 for kernel security updates
06:49 moritzm: rebooting mw2240-mw2269 for kernel security updates
04:58 marostegui: Disable puppet on dbproxy1006 for logging testing - T201021
00:21 thcipriani@deploy1001: Finished scap: SWAT: Add italian translation T203297 Improve German translation German translation: Replace "Vertreter" by "EU-Abgeordnete" (duration: 34m 38s)

2018-09-06

23:46 thcipriani@deploy1001: Started scap: SWAT: Add italian translation T203297 Improve German translation German translation: Replace "Vertreter" by "EU-Abgeordnete"
23:43 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.20/extensions/Score/includes/Score.php: SWAT: Add checks for length of generated audio files T203560 (duration: 00m 58s)
23:16 ejegg: updated CiviCRM from 7918924e73 to 52873b5dcc
22:16 ebernhardson: restart mjolnir-kafka-bulk-daemon service to pickup earlier deploy.
21:43 RoanKattouw: Deleting all user_properties rows with up_property='pagetriage-lastuse' (T202175)
21:32 krinkle@deploy1001: Synchronized php-1.32.0-wmf.20/includes/MovePage.php: T203661 - I9ebdcbc566b (duration: 00m 57s)
21:19 krinkle@deploy1001: Synchronized php-1.32.0-wmf.20/includes/exception/MWExceptionHandler.php: I3f35a519b50ae (duration: 00m 58s)
21:02 ppchelko@deploy1001: Finished deploy [restbase/deploy@53bc0a6]: Revert bumping Parsoid content-type filter T194190, take 5 (duration: 04m 01s)
20:58 ppchelko@deploy1001: Started deploy [restbase/deploy@53bc0a6]: Revert bumping Parsoid content-type filter T194190, take 5
20:56 ppchelko@deploy1001: Finished deploy [restbase/deploy@53bc0a6]: Revert bumping Parsoid content-type filter T194190, take 4 (duration: 04m 47s)
20:51 ppchelko@deploy1001: Started deploy [restbase/deploy@53bc0a6]: Revert bumping Parsoid content-type filter T194190, take 4
20:51 ppchelko@deploy1001: Finished deploy [restbase/deploy@53bc0a6]: Revert bumping Parsoid content-type filter T194190, take 3 (duration: 07m 16s)
20:44 ppchelko@deploy1001: Started deploy [restbase/deploy@53bc0a6]: Revert bumping Parsoid content-type filter T194190, take 3
20:43 ppchelko@deploy1001: Finished deploy [restbase/deploy@53bc0a6]: Revert bumping Parsoid content-type filter T194190, take 2 (duration: 09m 26s)
20:34 ppchelko@deploy1001: Started deploy [restbase/deploy@53bc0a6]: Revert bumping Parsoid content-type filter T194190, take 2
20:34 ppchelko@deploy1001: Finished deploy [restbase/deploy@53bc0a6]: Revert bumping Parsoid content-type filter T194190 (duration: 03m 56s)
20:30 ppchelko@deploy1001: Started deploy [restbase/deploy@53bc0a6]: Revert bumping Parsoid content-type filter T194190
18:27 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@d3e2c23]: repair msearch daemon cli args (duration: 03m 56s)
18:23 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@d3e2c23]: repair msearch daemon cli args
18:20 krinkle@deploy1001: Synchronized php-1.32.0-wmf.20/extensions/Flow/: Ia0112ae62e6b - T203647 (duration: 01m 02s)
18:12 krinkle@deploy1001: Synchronized php-1.32.0-wmf.20/includes/: I31a97d0168 - T203583 (duration: 01m 13s)
17:56 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: Canary machines back to wmf.20
17:45 thcipriani@deploy1001: scap failed: average error rate on 6/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details)
17:34 volans: uploaded spicerack_0.0.7-1{,+deb9u1} to apt.wikimedia.org {jessie,stretch}-wikimedia - T199079
17:20 jynus_: dropping old logs from dbproxy1003 T201021
17:15 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@735235b]: re-try bump to master (duration: 12m 46s)
17:02 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@735235b]: re-try bump to master
17:00 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@735235b]: new cli flags for msearch daemon, bump kafka-python dep to 1.4.x (duration: 16m 19s)
16:44 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@735235b]: new cli flags for msearch daemon, bump kafka-python dep to 1.4.x
16:44 ebernhardson@deploy1001: deploy aborted: new cli flags for msearch daemon, bump kafka-python dep to 1.4.x (duration: 00m 07s)
16:43 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@735235b]: new cli flags for msearch daemon, bump kafka-python dep to 1.4.x
16:40 volans: upgraded spicerack to version 0.0.6 on sarin/neodymium - T199079
16:38 volans: uploaded spicerack_0.0.6-1{,+deb9u1} to apt.wikimedia.org {jessie,stretch}-wikimedia - T199079
15:41 END: (PASS) - Cookbook sre.switchdc.services.02-restore-ttl (exit_code=0) (switchdc/oblivian@neodymium)
15:41 START: - Cookbook sre.switchdc.services.02-restore-ttl (switchdc/oblivian@neodymium)
15:37 END: (PASS) - Cookbook sre.switchdc.services.01-switch-dc (exit_code=0) (switchdc/oblivian@neodymium)
15:37 Switching: services pdfrender: eqiad => codfw (switchdc/oblivian@neodymium)
15:37 START: - Cookbook sre.switchdc.services.01-switch-dc (switchdc/oblivian@neodymium)
15:36 END: (PASS) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=0) (switchdc/oblivian@neodymium)
15:31 START: - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (switchdc/oblivian@neodymium)
15:30 END: (PASS) - Cookbook sre.switchdc.services.02-restore-ttl (exit_code=0) (switchdc/oblivian@neodymium)
15:30 START: - Cookbook sre.switchdc.services.02-restore-ttl (switchdc/oblivian@neodymium)
15:30 END: (PASS) - Cookbook sre.switchdc.services.01-switch-dc (exit_code=0) (switchdc/oblivian@neodymium)
15:30 Switching: services pdfrender: codfw => eqiad (switchdc/oblivian@neodymium)
15:30 START: - Cookbook sre.switchdc.services.01-switch-dc (switchdc/oblivian@neodymium)
15:04 END: (PASS) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=0) (switchdc/oblivian@neodymium)
14:59 START: - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (switchdc/oblivian@neodymium)
14:52 END: (PASS) - Cookbook sre.switchdc.services.02-restore-ttl (exit_code=0) (switchdc/oblivian@neodymium)
14:52 START: - Cookbook sre.switchdc.services.02-restore-ttl (switchdc/oblivian@neodymium)
14:52 END: (PASS) - Cookbook sre.switchdc.services.01-switch-dc (exit_code=0) (switchdc/oblivian@neodymium)
14:52 Switching: services pdfrender: eqiad => codfw (switchdc/oblivian@neodymium)
14:51 START: - Cookbook sre.switchdc.services.01-switch-dc (switchdc/oblivian@neodymium)
14:51 END: (PASS) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=0) (switchdc/oblivian@neodymium)
14:46 START: - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (switchdc/oblivian@neodymium)
14:38 elukey: reboot kafka2001 (eventbus codfw host) for kernel + openjdk-8 upgrades
13:46 moritzm: reboots of mc hosts in codfw completed
13:45 elukey: reboot kafka100[2-6] for kernel + openjdk-8 upgrades
13:24 elukey: reboot kafka-jumbo1001 for openjdk-8 + kernel security upgrades
13:08 hashar@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.20
13:03 hashar: all wikis to 1.32.0-wmf.20 | T191066
12:51 ema: trafficserver 7.1.3+ds-4wm3 uploaded to stretch-wikimedia T199720
10:22 moritzm: rebooting mc2* hosts for kernel security update
09:33 moritzm: disabling puppet on install1002 for some d-i tests
09:11 oblivian@deploy1001: Synchronized wmf-config/mc.php: Fixing memcached configuration for labstestwiki T203479 (duration: 00m 56s)
08:57 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1120 (duration: 00m 58s)
08:40 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1120 (duration: 00m 57s)
08:32 marostegui: Enable replication codfw -> eqiad on es2,es3 - T189107
08:25 marostegui: Enable replication codfw -> eqiad on s7,s8,x1 - T189107
08:17 marostegui: Enable replication codfw -> eqiad on s1,s3,s4 - T189107
08:09 godog: bounce ircecho on einsteinium, stuck and not on irc
08:09 hashar: rebooting contint2001 for kernel security update
08:06 godog: repair sde1 on ms-be2042 - T199198
08:06 marostegui: Enable replication codfw -> eqiad on s5,s6,s2 - T189107
08:01 moritzm: rebooting contint1001 for kernel security update
07:58 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Fix DB configuration in preparation for dc switchover (duration: 00m 57s)
07:54 hashar: Upgraded packages on contint1001 and contint2001
07:24 _joe_: rolling restart of eqiad HHVM appservers
07:10 moritzm: run decomission_appserver on mw2213 (T203434)
04:38 TimStarling: on mwdebug1001 restarting hhvm after probably breaking it by trying to attach with gdb
04:08 tstarling@deploy1001: Synchronized php-1.32.0-wmf.20/extensions/Quiz/Question.php: T203628 (duration: 00m 56s)
04:06 tstarling@deploy1001: Synchronized php-1.32.0-wmf.19/extensions/Quiz/Question.php: (no justification provided) (duration: 00m 57s)
03:17 legoktm@deploy1001: Synchronized wmf-config/CommonSettings.php: Have canonical Main Page URL be the domain root for fixcopyrightwiki (duration: 00m 57s)
02:47 legoktm@deploy1001: Synchronized php-1.32.0-wmf.20/extensions/UniversalLanguageSelector/: ULS fixes for Accept-Language stuff (duration: 01m 44s)
02:38 legoktm@deploy1001: Finished scap: EUCopyrightCampaign updates (duration: 62m 43s)
01:36 legoktm@deploy1001: Started scap: EUCopyrightCampaign updates

2018-09-05

23:37 jforrester@deploy1001: Synchronized php-1.32.0-wmf.20/extensions/Score: SWAT clean-up following messy deployment and undeployment of patch (duration: 00m 57s)
23:27 jforrester@deploy1001: Synchronized php-1.32.0-wmf.20/extensions/Score/includes/Score.php: SWAT Fix error for T203560 (duration: 00m 54s)
23:25 jforrester@deploy1001: Synchronized php-1.32.0-wmf.20/extensions/EducationProgram/includes/pagers/StudentPager.php: SWAT Revert fix forT203577 (duration: 00m 56s)
23:22 jforrester@deploy1001: Synchronized php-1.32.0-wmf.20/extensions/Score/includes/Score.php: SWAT Fix error on malformed MIDI files T203560 (duration: 01m 04s)
23:14 jforrester@deploy1001: Synchronized php-1.32.0-wmf.20/extensions/EducationProgram/includes/pagers/StudentPager.php: SWAT Fix spammy log errors T203577 (duration: 00m 58s)
22:17 XenoRyet: updated payments-wiki from e749025c1a to 05d796e844
21:54 volans: uploaded spicerack_0.0.5-1{,+deb9u1} to apt.wikimedia.org {jessie,stretch}-wikimedia - T199079
20:59 XioNoX: clear bgp neighbor 80.249.209.209 on cr2-esams (max prefix limit)
20:28 arlolra: Updated Parsoid to 740b3a4 (T198400, T202819)
20:19 arlolra@deploy1001: Finished deploy [parsoid/deploy@f3ef0c8]: Updating Parsoid to 740b3a4 (duration: 09m 57s)
20:17 krinkle@deploy1001: Synchronized php-1.32.0-wmf.19/extensions/Kartographer/: Ie5744d45b - T203427 (duration: 00m 57s)
20:14 krinkle@deploy1001: Synchronized php-1.32.0-wmf.20/extensions/Kartographer/: Ie5744d45b - T203427 (duration: 00m 58s)
20:12 legoktm: legoktm@deploy1001:~$ cat /home/legoktm/fixcopyright | mwscript purgeList.php --wiki=aawiki
20:11 legoktm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Tweak $wgSitename for fixcopyrightwiki (duration: 00m 58s)
20:09 arlolra@deploy1001: Started deploy [parsoid/deploy@f3ef0c8]: Updating Parsoid to 740b3a4
19:49 legoktm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Set $wgSitename and $wgULSLanguageDetection on fixcopyrightwiki (duration: 00m 57s)
19:46 legoktm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable EUCopyrightCampaign extensions and SkinPerPage for fixcopyrightwiki (duration: 00m 56s)
19:42 legoktm@deploy1001: Finished scap: build l10n for EUCopyrightCampaign and SkinPerPage (duration: 64m 26s)
18:39 moritzm: fixed package state on auth2001
18:38 legoktm@deploy1001: Started scap: build l10n for EUCopyrightCampaign and SkinPerPage
18:30 mutante: cp1080 - pooling again after T203194 appears fixed
18:04 XioNoX: upgrade asw2-a-eqiad (not in production)
17:47 ppchelko@deploy1001: Finished deploy [restbase/deploy@b768926]: Bump expected Parsoid version and release new metrics endpoints, take 3 (duration: 06m 40s)
17:42 cmjohnson1: swapping DAC cable cp1080
17:40 ppchelko@deploy1001: Started deploy [restbase/deploy@b768926]: Bump expected Parsoid version and release new metrics endpoints, take 3
17:39 ppchelko@deploy1001: Finished deploy [restbase/deploy@b768926]: Bump expected Parsoid version and release new metrics endpoints, take 2 (duration: 04m 02s)
17:35 ppchelko@deploy1001: Started deploy [restbase/deploy@b768926]: Bump expected Parsoid version and release new metrics endpoints, take 2
17:35 ppchelko@deploy1001: Finished deploy [restbase/deploy@b768926]: Bump expected Parsoid version and release new metrics endpoints (duration: 13m 24s)
17:22 ppchelko@deploy1001: Started deploy [restbase/deploy@b768926]: Bump expected Parsoid version and release new metrics endpoints
17:14 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: I73d2ce30 - T191086 (duration: 00m 57s)
16:45 cmjohnson1: swapping ethernet cable for stat1006
16:42 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: dc=eqiad,cluster=wdqs,name=wdqs1003.eqiad.wmnet
16:42 gehel: shutting down wdqs1003 for new SSD and reimage - T202780
16:38 stephanebisson: Finished extensions/PageTriage/maintenance/FixNominatedForDeletion.php --wiki enwiki
16:38 stephanebisson: Starting extensions/PageTriage/maintenance/FixNominatedForDeletion.php --wiki enwiki
16:35 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.20/extensions/PageTriage/modules/ext.pageTriage.models/ext.pageTriage.article.js: SWAT: Fix CopyPatrol links for drafts T203284 (duration: 00m 57s)
16:31 mutante: LDAP: removed user 'albe' from groups 'wmde' and 'nda' (T203561)
16:30 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.19/extensions/PageTriage/maintenance/FixNominatedForDeletion.php: SWAT: Maintenance: fix ptrp_deleted T202582 (duration: 00m 57s)
16:23 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable logging for Schema:CitationUsage at 100% T191086 (duration: 01m 23s)
14:50 elukey@deploy1001: Finished deploy [analytics/refinery@77a5a83]: small changes to pageview whitelist and scripts (duration: 08m 40s)
14:42 elukey@deploy1001: Started deploy [analytics/refinery@77a5a83]: small changes to pageview whitelist and scripts
14:21 akosiaris: switchover the deployment server back to deploy1001
14:11 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: dc=codfw,cluster=wdqs,name=wdqs2002.codfw.wmnet
14:10 gehel: shutting down wdqs2002 for new SSD and reimage - T202777
14:07 elukey: reboot druid* hosts for kernel + openjdk-8 upgrades
13:44 godog: repair sdn1 on ms-be2041 - T199198
13:40 ottomata: reimaging thorium to debian stretch (this will cause an announced {stats,analytics}.http://wm.org/ downtime!) - T192641
13:39 herron: updating phabricator mail smtp-host to localhost T196916
13:20 moritzm: rebooting tegmen for kernel security update
13:10 moritzm: installing php5 security updates on jessie
13:09 _joe_: depooling /repooling restbase, mathoid in codfw for switchover pre-flight testing
13:08 hashar@deploy2001: Synchronized php: group1 wikis to 1.32.0-wmf.20 (duration: 01m 07s)
13:07 hashar@deploy2001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.20
12:29 stephanebisson: Finished extensions/PageTriage/maintenance/FixNominatedForDeletion.php --wiki testwiki
12:29 stephanebisson: Starting extensions/PageTriage/maintenance/FixNominatedForDeletion.php --wiki testwiki
11:20 moritzm: rebooting mwmaint2001 for kernel security update
11:12 moritzm: rearmed keyholder on deploy1001
11:08 moritzm: rebooting deploy1001 for kernel security update
10:43 akosiaris@deploy2001: Synchronized wmf-config/ProductionServices.php: (no justification provided) (duration: 04m 57s)
10:43 marostegui: Stop MySQL and reboot db2093 for kernel upgrade
10:33 akosiaris@deploy2001: Finished deploy [servermon/servermon@c474a6b]: (no justification provided) (duration: 00m 05s)
10:33 akosiaris@deploy2001: Started deploy [servermon/servermon@c474a6b]: (no justification provided)
10:28 akosiaris@deploy2001: Finished deploy [servermon/servermon@c474a6b]: (no justification provided) (duration: 00m 14s)
10:28 akosiaris@deploy2001: Started deploy [servermon/servermon@c474a6b]: (no justification provided)
10:24 akosiaris@deploy2001: Finished deploy [servermon/servermon@c474a6b]: (no justification provided) (duration: 00m 04s)
10:24 akosiaris@deploy2001: Started deploy [servermon/servermon@c474a6b]: (no justification provided)
10:22 akosiaris@deploy2001: Finished deploy [servermon/servermon@c474a6b]: (no justification provided) (duration: 00m 49s)
10:22 akosiaris@deploy2001: Started deploy [servermon/servermon@c474a6b]: (no justification provided)
10:21 akosiaris@deploy2001: Finished deploy [servermon/servermon@c474a6b]: (no justification provided) (duration: 00m 04s)
10:21 akosiaris@deploy2001: Started deploy [servermon/servermon@c474a6b]: (no justification provided)
10:21 akosiaris@deploy2001: deploy aborted: (no justification provided) (duration: 00m 00s)
10:20 moritzm: installing bind9 security updates (client-side tools and libraries)
10:16 akosiaris@deploy2001: Finished deploy [servermon/servermon@c474a6b]: (no justification provided) (duration: 00m 52s)
10:15 akosiaris@deploy2001: Started deploy [servermon/servermon@c474a6b]: (no justification provided)
10:12 akosiaris@deploy2001: Finished deploy [servermon/servermon@c474a6b]: (no justification provided) (duration: 01m 39s)
10:10 akosiaris@deploy2001: Started deploy [servermon/servermon@c474a6b]: (no justification provided)
10:09 akosiaris@deploy2001: Finished deploy [servermon/servermon@c474a6b]: (no justification provided) (duration: 01m 44s)
10:08 akosiaris@deploy2001: Started deploy [servermon/servermon@c474a6b]: (no justification provided)
10:04 godog: repair sdh on ms-be1043 - T199198
10:04 akosiaris: switchover the deployment server as a test for the switchover next week
10:00 volans: upgraded spicerack to version 0.0.4 on sarin/neodymium - T199079
09:59 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=mw2213.codfw.wmnet
09:52 moritzm: installing java security updates on druid*
09:47 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1114 but with small api load due to ongoing issues (duration: 00m 56s)
09:16 volans: uploaded spicerack_0.0.4-1{,+deb9u1} to apt.wikimedia.org {jessie,stretch}-wikimedia - T199079
09:12 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Set all individual codfw sections in read-write, codfw globally still in ro (duration: 00m 56s)
09:07 elukey: rebooting druid1001 for kernel + openjdk-8 upgrades
08:52 oblivian@puppetmaster1001: conftool action : edit; selector: name=ReadOnly,scope=codfw
08:51 oblivian@puppetmaster1001: conftool action : edit; selector: name=ReadOnly,scope=codfw
08:44 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Set s5 section in read-write, codfw should be still in ro (duration: 00m 58s)
08:37 marostegui: Drop partitions from db2040 (s7 master) for metawiki.pagelinks - T203548
08:22 elukey: reboot aqs100[5-9] for kernel + openjdk-8 upgrades
07:49 ema: upload vhtcpd 0.1.2-1 to stretch-wikimedia T199720
07:48 elukey: re-enable puppet on mc1035 - memcache unit refreshed, mw cache shard wiped - T203429
07:27 kartik@deploy1001: Finished deploy [cxserver/deploy@f341eec]: Update cxserver to 81d1a97 (T202933, T202283, T189438) (duration: 04m 03s)
07:24 moritzm: rebooting video scalers/job runners in codfw for kernel security update
07:23 kartik@deploy1001: Started deploy [cxserver/deploy@f341eec]: Update cxserver to 81d1a97 (T202933, T202283, T189438)
07:02 elukey: restart oozie on analytics1003 to pick up new smtp settings
06:57 moritzm: installing java security updates on aqs*
06:40 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1077 (duration: 00m 57s)
06:40 marostegui: Deploy schema change on s3 master (db1075)
05:13 marostegui: Deploy schema change on db1077 with replication, this will generate lag on labsdb:s3
05:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1077 (duration: 01m 09s)
05:06 marostegui: Deploy schema change on s7 primary master (db1062)
03:42 eileen: civicrm revision changed from 7bde3d0ab2 to 7918924e73, config revision is 0b227269a8
03:18 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Sep 5 03:18:07 UTC 2018 (duration 10m 12s)
03:07 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.20) (duration: 16m 17s)
02:33 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.19) (duration: 13m 15s)
01:42 krinkle@deploy1001: Synchronized php-1.32.0-wmf.20/includes/resourceloader/: I8e8d3a2cd2cc - T201686 (duration: 01m 31s)

2018-09-04

23:44 ejegg: updated civicrm from 1330df3064 to 7bde3d0ab2
23:34 catrope@deploy1001: Synchronized php-1.32.0-wmf.20/extensions/PageTriage/maintenance/: Add new maintenance script to fix deleted flag in PageTriage (T202582) (duration: 00m 58s)
23:30 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable TemplateWizard on testwiki and test2wiki (T202545) (duration: 00m 58s)
23:22 RoanKattouw: Ran namespaceDupes.php --fix on plwiktionary and plwiki
23:21 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Modify gender namespaces on plwiktionary (T202347) (duration: 00m 57s)
22:22 XioNoX: add BGP sessions to AS64096 in eqord
22:19 XioNoX: add BGP sessions to AS8220 in eqiad + eqsin
22:19 Jeff_Green: authdns-update to deploy DNS changes removing thulium
21:28 tzatziki: deleted archived file
21:19 legoktm@deploy1001: Synchronized wmf-config/: Enable EUCopyrightCampaign extensions/skin on beta cluster (3/3) (duration: 00m 58s)
21:18 legoktm@deploy1001: Synchronized wmf-config/CommonSettings.php: Enable EUCopyrightCampaign extensions/skin on beta cluster (2/3) (duration: 00m 57s)
21:16 legoktm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable EUCopyrightCampaign extensions/skin on beta cluster (1/3) (duration: 00m 57s)
21:05 krinkle@deploy1001: Synchronized php-1.32.0-wmf.20/resources/src/: resourceloader: startup and mediawiki.base improvements and fixes (duration: 00m 57s)
21:03 krinkle@deploy1001: Synchronized php-1.32.0-wmf.19/extensions/FlaggedRevs/frontend/FlaggablePageView.php: I6dce0c (duration: 00m 58s)
21:00 krinkle@deploy1001: Synchronized php-1.32.0-wmf.19/resources/src/startup/: mw.loader improvements and fixes (duration: 00m 58s)
20:12 ejegg: rolled CiviCRM back to 1330df3064
20:05 ejegg: updated CiviCRM from 1330df3064 to 7bde3d0ab2
19:26 gehel: rolling restart of elasticsearch / cirrus / eqiad for various updates and data directory migration completed - T198351
18:38 XioNoX: change internal NAT for 208.80.155.12 - T203475
18:02 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1078 (duration: 00m 57s)
16:31 elukey: reboot aqs1004 again for kernel + openjdk-8 upgrades (now available since the root partition is not read only anymore)
16:26 thcipriani@deploy1001: Synchronized README: noop sync file - test scap 3.8.5-1 (duration: 00m 54s)
16:24 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: noop wikiversions sync for T198640
16:21 godog: upload scap 3.8.5-1 - T203271
16:18 fdans@deploy1001: Finished deploy [analytics/refinery@2c4ec7a]: deploying refinery to update pageview def (duration: 09m 57s)
16:08 fdans@deploy1001: Started deploy [analytics/refinery@2c4ec7a]: deploying refinery to update pageview def
15:30 elukey: reboot aqs1004 after running fsck on root (was read-only)
15:13 papaul: shutting down backup2001 to disable 1GB NIC
15:08 marostegui: Deploy schema change on db1078
15:07 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1078 (duration: 00m 56s)
15:02 marostegui: Deploy schema change on db1095:3313
15:02 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1123 (duration: 00m 57s)
14:45 addshore@deploy1001: Synchronized php-1.32.0-wmf.20/extensions/Wikibase: Track new ItemId formatter usages (duration: 01m 17s)
14:14 bstorm_: Removing subtree_check from project nfs on labstore1004/5
14:02 jynus: stopping replication and running partitioning on logging on db1088:3311 T189107
14:00 hashar@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.32.0-wmf.20
13:45 hashar@deploy1001: Finished scap: testwiki to php-1.32.0-wmf.20 and rebuild l10n cache - T191066 (duration: 62m 55s)
13:07 hashar: 1.32.0-wmf.20 is still syncing for testwiki due to l10ncache generation
13:06 elukey: restart memcached on mc1035 with -v option - T203429
12:57 moritzm: reimaging backup2001
12:43 hashar@deploy1001: Started scap: testwiki to php-1.32.0-wmf.20 and rebuild l10n cache - T191066
12:42 hashar@deploy1001: Pruned MediaWiki: 1.32.0-wmf.18 [keeping static files] (duration: 06m 58s)
12:35 hashar: scap clean 1.32.0-wmf.18 | T191066
12:32 hashar: Applied security patches for 1.32.0-wmf.20 | T191066
12:24 hashar: scap prep 1.32.0-wmf.20 # T191066
12:07 marostegui: Deploy schema change on db1123
12:07 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1123 (duration: 00m 48s)
11:58 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3317 (duration: 00m 48s)
11:50 hashar: Cutting branch 1.32.0-wmf.20 | T191066
11:47 hashar: Preparing train deploy 1.32.0-wmf.20 | T191066
11:35 hashar: European SWAT completed
11:34 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Wikidata: Use new item ID formatter for Q1-Q100000 - T201835 (duration: 00m 49s)
11:19 hashar@deploy1001: Synchronized php-1.32.0-wmf.19/extensions/DismissableSiteNotice: Revert "Use session storage instead of cookies for site notices" - T199274 (duration: 00m 50s)
11:11 jynus: stopping replication and running partitioning on logging on db1085:3311 T189107
11:10 hashar@deploy1001: Synchronized static/images/project-logos: Update logos for the Russian Wikisource - T203343 (duration: 00m 49s)
11:06 hashar@deploy1001: Synchronized wmf-config/throttle.php: Two throttle rules for SMEX editathon - T203392 (duration: 00m 51s)
11:05 hashar@deploy1001: sync-file aborted: (no justification provided) (duration: 00m 01s)
10:29 marostegui: Deploy schema change on dbstore1002:s3
10:23 marostegui: Deploy schema change on s3 codfw masters (this will generate lag on s3 codfw)
10:17 elukey: restart ircecho on einstenium to force it re-join #wikimedia-analytics
09:57 _joe_: restarted pdfrender on scb1001
09:40 jynus: deploying latest event schedulers to all core db hosts
09:32 elukey: restart ircecho as attempt to see if icinga-vm re-joins the analytics chan
09:24 jynus: stop, upgrade and running analyze on db1114
09:23 marostegui: Deploy schema change on db1090:3317
09:23 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3317 (duration: 00m 49s)
09:12 akosiaris: depool pdfrender in eqiad, hopefully codfw will be better equipped to handle the load
09:10 volans: restarted pdfrender on scb1003
09:07 volans: restarted pdfrender on scb1002
09:06 volans: restarted pdfrender on scb1004
08:57 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1094, depool db1114 (duration: 00m 50s)
08:28 gehel: starting rolling restart of elasticsearch / cirrus / eqiad for various updates and data directory migration - T198351
08:04 moritzm: upgrading mwmaint* to wikidiff 1.7.3
07:52 gehel: rolling restart of elasticsearch / cirrus / codfw for various updates and data directory migration completed - T198351
07:44 moritzm: rearmed keyholder on deploy2001
07:10 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1094 (duration: 00m 52s)
07:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1079 (duration: 00m 54s)
07:00 moritzm: upgrading remaining appservers/API servers to wikidiff 1.7.3
06:36 moritzm: rebooting deploy2001 for kernel security update
06:32 moritzm: upgrading deploy* to wikidiff 1.7.3
06:20 moritzm: upgrading labweb* to wikidiff 1.7.3
05:14 marostegui: Deploy schema change on db1079 (this will generate lag on labsdb:s7)
05:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1079 (duration: 00m 50s)
05:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1086 (duration: 00m 49s)
03:16 krinkle@deploy1001: Synchronized php-1.32.0-wmf.19/extensions/WikimediaEvents/: Ie5c8f5877b (duration: 00m 50s)
03:14 krinkle@deploy1001: Synchronized php-1.32.0-wmf.19/extensions/Kartographer: I351259c46 (duration: 00m 51s)
02:48 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Sep 4 02:48:17 UTC 2018 (duration 10m 10s)
02:38 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.19) (duration: 14m 46s)

2018-09-03

22:45 ejegg: updated CiviCRM from ac5ac25eba to 1330df3064
21:19 krinkle@deploy1001: Synchronized php-1.32.0-wmf.19/extensions/WikimediaEvents/: I920127efb3c4 (duration: 00m 51s)
19:22 ejegg: updated CiviCRM from ea690851f0 to ac5ac25eba
17:14 gehel@deploy1001: Finished deploy [wdqs/wdqs@68d9cab]: new version of wdqs GUI (duration: 10m 26s)
17:04 gehel@deploy1001: Started deploy [wdqs/wdqs@68d9cab]: new version of wdqs GUI
17:03 gehel@deploy1001: Finished deploy [wdqs/wdqs@68d9cab]: new version of wdqs GUI (wdqs1009 only) (duration: 00m 33s)
17:02 gehel@deploy1001: Started deploy [wdqs/wdqs@68d9cab]: new version of wdqs GUI (wdqs1009 only)
16:41 moritzm: depooling mw1310-mw1311 for reboot to address kernel security updates once pending transcoding jobs have been completed
16:32 moritzm: powercycling mw1300, stuck in reboot
16:08 marostegui: Deploy schema change on db1086
15:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1086 (duration: 00m 48s)
15:57 moritzm: rebooting mw2200-mw2239 for kernel security updates
15:52 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1098:3317 (duration: 00m 50s)
15:51 ejegg: updated civicrm (back) to ea690851f0, but rolled back queue server settings
15:48 ejegg: rolled back civicrm to 863ed3f1a2
15:45 ejegg: updated civicrm from from 863ed3f1a2 to ea690851f0
15:44 ejegg: updated payments-wiki from 763e14b6df to e749025c1a
15:27 reedy@deploy1001: Pruned MediaWiki: 1.32.0-wmf.16 [keeping static files] (duration: 01m 48s)
15:24 reedy@deploy1001: Pruned MediaWiki: 1.32.0-wmf.15 [keeping static files] (duration: 03m 44s)
15:20 reedy@deploy1001: Pruned MediaWiki: 1.32.0-wmf.14 [keeping static files] (duration: 02m 13s)
15:19 moritzm: rebooting mw2190-mw2219 for kernel security updates
15:13 moritzm: depooling mw1300-mw1309 for reboot to address kernel security updates once pending transcoding jobs have been completed
14:48 jynus: running basic mw warmup script on codfw as a test
14:26 ema: lvs1016: re-enable puppet
14:23 godog: switch swift thumbnails active/active - T201858 T199073
13:39 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1098:3317 (duration: 00m 49s)
13:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1101:3317 (duration: 00m 56s)
12:03 akosiaris: repool eqiad mathoid for kubernetes upgrade
11:38 zeljkof: EU SWAT finished
11:38 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add high-density logos for Commons (T203342) (duration: 00m 49s)
11:34 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add high-density logos for the Russian Wikisource (T203343) (duration: 00m 50s)
11:27 zfilipin@deploy1001: Synchronized static/images/project-logos: SWAT: Add high-density logos for Commons (T203342) (duration: 00m 49s)
11:23 zfilipin@deploy1001: Synchronized static/images/project-logos: SWAT: Add high-density logos for the Russian Wikisource (T203343) (duration: 00m 50s)
11:21 akosiaris: depool eqiad mathoid for kubernetes upgrade
11:21 akosiaris: repool codfw mathoid. Kubernetes cluster upgraded to 1.10.6
11:15 moritzm: installing libcgroup security updates
10:57 moritzm: installing libx11 security updates on jessie
10:39 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 49s)
10:39 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 51s)
10:30 moritzm: installing ruby2.1 security updates
10:12 moritzm: installing tomcat8 security updates
10:00 akosiaris: depool codfw mathoid for kubernetes upgrade
09:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1121 (duration: 01m 55s)
09:33 moritzm: rebooting mw2135-mw2147 for kernel security updates
09:12 moritzm: upgrading mw1339-mw1348 to wikidiff 1.7.3
09:08 moritzm: depooling mw1334-mw1338 for reboot to address kernel security updates once pending transcoding jobs have been completed
09:00 gehel: starting rolling restart of elasticsearch / cirrus / codfw for various updates and data directory migration - T198351
08:32 godog: fix xfs on ms-be1041 sde - T199198
08:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101:3317 (duration: 00m 52s)
08:21 marostegui: Deploy schema change on s7 codfw master (this will generate lag on s7 codfw)
08:18 moritzm: rebooting mw2163-mw2189 for kernel security updates
08:07 moritzm: depooling mw1293-mw1296,mw1299,mw1318 for reboot to address L1TF once pending transcoding jobs have been completed
08:01 volans: restarted pdfrender on scb1004 and repooled it
07:56 volans: restarted pdfrender on scb1003
07:55 godog: roll restart thumbor to apply latest config changes - T203135 T20871
07:49 volans: depooled pdfrender on scb1004 to let the service owners debug it
07:41 moritzm: upgrading mw1221-mw1235 to wikidiff 1.7.3
07:39 marostegui: Deploy schema change on s8.wikidatawiki on codfw master - T189101
07:34 marostegui: Deploy schema change on s3.testwikidatawiki on codfw master - T189101
07:20 moritzm: upgrading mw1238-mw1258 to wikidiff 1.7.3
06:57 moritzm: upgrading mwdebug* to wikidiff 1.7.3
06:41 moritzm: uploaded wikidiff 1.7.3 to apt.wikimedia.org
05:20 marostegui: Deploy alter table on db1121 with replication, this will generate lag on labs:s4
05:19 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1121 (duration: 00m 52s)
02:47 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Sep 3 02:47:07 UTC 2018 (duration 10m 15s)
02:36 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.19) (duration: 14m 26s)

2018-09-02

14:57 akosiaris: restart pdfrender across scb100* cluster

2018-09-01

21:47 krinkle@deploy1001: Synchronized php-1.32.0-wmf.19/extensions/WikimediaMaintenance/: I219882 - T203154 (duration: 01m 06s)
02:06 krinkle@deploy1001: Synchronized php-1.32.0-wmf.19/resources/src/startup/startup.js: (no justification provided) (duration: 00m 50s)
01:50 krinkle@deploy1001: Synchronized php-1.32.0-wmf.19/resources/src/mediawiki.user.js: I8feecddf0878 - T203275 (duration: 00m 49s)
01:49 krinkle@deploy1001: Synchronized php-1.32.0-wmf.19/resources/src/startup/: If26851 (duration: 00m 49s)
01:14 krinkle@deploy1001: Synchronized dblists/s3.dblist: I43c79297499 (duration: 00m 49s)
01:12 krinkle@deploy1001: Synchronized tests/: I43c79297499 (duration: 00m 51s)

2000s

Archive 1: 2004 Jun - 2004 Sep
Archive 2: 2004 Oct - 2004 Nov
Archive 3: 2004 Dec - 2005 Mar
Archive 4: 2005 Apr - 2005 Jul
Archive 5: 2005 Aug - 2005 Oct, with revision history 2004-06-23 to 2005-11-25
Archive 6: 2005 Nov - 2006 Feb
Archive 7: 2006 Mar - 2006 Jun
Archive 8: 2006 Jul - 2006 Sep
Archive 9: 2006 Oct - 2007 Jan, with revision history 2005-11-25 to 2007-02-21
Archive 10: 2007 Feb - 2007 Jun
Archive 11: 2007 Jul - 2007 Dec
Archive 12: 2008 Jan - 2008 Jul
Archive 12a: 2008 Aug
Archive 12b: 2008 Sept
Archive 13: 2008 Oct - 2009 Jun
Archive 14: 2009 Jun - 2009 Dec

2010s

2020s