Server Admin Log

From Wikitech
(Redirected from Server admin log)
Jump to: navigation, search

2017-01-24

  • 02:23 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Jan 24 02:23:01 UTC 2017 (duration 4m 23s)
  • 02:18 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.8) (duration: 06m 40s)
  • 01:37 ejegg: updated SmashPig from 03880ce to ab52dbe
  • 01:18 Krinkle: mwscript deleteEqualMessages.php --wiki gotwiki (T45917)
  • 01:16 ejegg: updated payments-wiki from c22353b to dd8a16d
  • 00:26 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1065 with low load after reimage (duration: 00m 45s)

2017-01-23

  • 22:38 Pchelolo: update RESTBase to 598fa56f
  • 22:37 Pchelolo: update RESTBase to 598fa56f: canary on restbase1007
  • 22:22 Pchelolo: update RESTBase to d1663345c
  • 22:21 Pchelolo: update RESTBase to d1663345c: canary on restbase1007
  • 22:18 Pchelolo: update RESTBase to d1663345c: staging
  • 21:54 demon@tin: Synchronized wmf-config: interwiki update, dropping some old ExtensionMessages files (duration: 00m 41s)
  • 21:43 mutante: sca2004 was out of memory but also fixed itself and i could run puppet again a few minutes later
  • 21:41 demon@tin: Synchronized w: Removing wiki.phtml, apache does the rewrites (duration: 00m 48s)
  • 21:39 bsitzmann@tin: Finished deploy [mobileapps/deploy@7615bf9]: Update mobileapps to 66ef3c2 (duration: 03m 16s)
  • 21:36 bsitzmann@tin: Starting deploy [mobileapps/deploy@7615bf9]: Update mobileapps to 66ef3c2
  • 20:29 nuria@tin: Finished deploy [analytics/aqs/deploy@025ef23]: (no message) (duration: 00m 08s)
  • 20:29 nuria@tin: Starting deploy [analytics/aqs/deploy@025ef23]: (no message)
  • 20:28 nuria@tin: Finished deploy [analytics/aqs/deploy@025ef23]: (no message) (duration: 00m 07s)
  • 20:28 nuria@tin: Starting deploy [analytics/aqs/deploy@025ef23]: (no message)
  • 20:18 otto@tin: Finished deploy [analytics/aqs/deploy@025ef23]: (no message) (duration: 00m 01s)
  • 20:18 otto@tin: Starting deploy [analytics/aqs/deploy@025ef23]: (no message)
  • 20:11 gehel@tin: Finished deploy [wdqs/wdqs@fd88fda]: (no message) (duration: 01m 56s)
  • 20:09 otto@tin: Finished deploy [analytics/aqs/deploy@025ef23]: (no message) (duration: 00m 01s)
  • 20:09 otto@tin: Starting deploy [analytics/aqs/deploy@025ef23]: (no message)
  • 20:09 gehel@tin: Starting deploy [wdqs/wdqs@fd88fda]: (no message)
  • 20:06 gehel: deplyoing latest wdqs version (2h behind planned schedule)
  • 20:04 otto@tin: Finished deploy [analytics/aqs/deploy@025ef23]: (no message) (duration: 02m 16s)
  • 20:02 otto@tin: Starting deploy [analytics/aqs/deploy@025ef23]: (no message)
  • 19:26 reedy@tin: Synchronized wmf-config/CommonSettings.php: Make sure CommonSettings-labs is one of the last things loaded so we don't get problems from things being included after (duration: 00m 40s)
  • 18:57 nuria@tin: Finished deploy [analytics/aqs/deploy@025ef23]: (no message) (duration: 00m 35s)
  • 18:56 nuria@tin: Starting deploy [analytics/aqs/deploy@025ef23]: (no message)
  • 18:47 nuria@tin: Finished deploy [analytics/aqs/deploy@56ab863]: (no message) (duration: 00m 01s)
  • 18:47 nuria@tin: Starting deploy [analytics/aqs/deploy@56ab863]: (no message)
  • 18:47 nuria@tin: Finished deploy [analytics/aqs/deploy@56ab863]: (no message) (duration: 00m 01s)
  • 18:47 nuria@tin: Starting deploy [analytics/aqs/deploy@56ab863]: (no message)
  • 18:45 nuria@tin: Finished deploy [analytics/aqs/deploy@56ab863]: (no message) (duration: 01m 08s)
  • 18:44 nuria@tin: Starting deploy [analytics/aqs/deploy@56ab863]: (no message)
  • 18:43 nuria@tin: Finished deploy [analytics/aqs/deploy@56ab863]: (no message) (duration: 00m 01s)
  • 18:43 nuria@tin: Starting deploy [analytics/aqs/deploy@56ab863]: (no message)
  • 18:33 demon@tin: Synchronized wmf-config/CommonSettings-labs.php: no-op, completeness (duration: 00m 40s)
  • 18:32 demon@tin: Synchronized wmf-config/extension-list-labs: no-op, completeness (duration: 00m 40s)
  • 18:30 jynus: reimaging db1065 to jessie
  • 18:18 papaul: shutting down ms-be2010 for maintenance
  • 18:12 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1065 (duration: 00m 39s)
  • 16:31 papaul: shutting down mw2098 for maintenance
  • 15:38 marostegui: Alter tables: flow_topic_list and flow_tree_node on db1031 (x1 master) - T149819
  • 15:19 elukey: whitelisted dbproxy1011 on cr1/cr2 for analytics-in4 input filter
  • 15:18 moritzm: installing mysql 5.5 security updates (as packaged by jessie/trusty, not the internal mariadb packages)
  • 15:17 moritzm: installing pdns-recursor security update on labservices1002
  • 15:17 Dereckson: Fixed namespaces dupes following NS_PROJECT update on sa.wikisource (T101634)
  • 15:15 gehel: reimage elastic2025 - T154251
  • 15:02 Dereckson: EU SWAT done (handled by addshore)
  • 15:00 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Set site name and meta namespace for Sanskrit wikis (T101634) (duration: 00m 40s)
  • 14:34 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T155844 [fix] Add finds.org.uk without wildcard too (duration: 00m 39s)
  • 14:24 addshore@tin: Synchronized wmf-config/Wikibase.php: T150183 Move InterwikiSortOrders to own file PT 2/2 (duration: 00m 39s)
  • 14:23 addshore@tin: Synchronized wmf-config/InterwikiSortOrders.php: T150183 Move InterwikiSortOrders to own file PT 1/2 (duration: 00m 40s)
  • 14:23 gehel: disabling puppet on elastic20(2[5-9]|3[0-6]) prior to reimage - T154251
  • 14:20 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T155916 Amend category collation for de.wikisource to uca-de-u-kn (duration: 00m 39s)
  • 14:20 godog: depool ms-fe200[1234] T152612
  • 14:15 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T155906 Add n, n:es and n:fr as import sources in test2wiki (duration: 00m 39s)
  • 14:14 Dereckson: Fix namespaces dupes on sa.wikisource to prepare T101634 / Gerrit:333640
  • 14:09 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: gerrit:333476 (NOOP) Temporarily set $wgDisableUserGroupExpiry to true on labs (duration: 00m 40s)
  • 14:07 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: gerrit:333294 Add *.finds.org.uk to wgCopyUploadsDomains (duration: 00m 41s)
  • 11:54 elukey: whitelisted dbproxy1010 on cr1/cr2 for analytics-in4 input filter
  • 10:50 moritzm: installing pdns-recursor security updates on trusty systems
  • 10:38 moritzm: installing openjpeg security updates
  • 09:06 marostegui: Compress s2 on dbstore2001 - T151552
  • 07:47 marostegui: Enabling gtid_domain_id on db1047 (eventlogging host) - T149418
  • 07:43 marostegui: Enabling gtid_domain_id on db1046 (eventlogging master) - T149418
  • 07:32 marostegui: Deploy gtid_domain_id db1043 (passive master) - last host pending in m3 - T149418
  • 07:28 marostegui: Compressing cebwiki.templatelinks on db1015 (224G table) - T153739
  • 03:02 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Jan 23 03:02:32 UTC 2017 (duration 4m 44s)
  • 02:57 reedy@tin: scap sync-l10n completed (1.29.0-wmf.8) (duration: 13m 14s)
  • 02:26 Reedy: running l10nupdate manually
  • 02:23 Reedy: cleaned up reCaptcha extension in l10ncache dirs
  • 02:02 l10nupdate@tin: LocalisationUpdate failed: git pull of extensions failed

2017-01-22

  • 23:26 mobrovac: restbase deploying d1663345 - blacklist of a bot log page on enwiki
  • 02:01 l10nupdate@tin: LocalisationUpdate failed: git pull of extensions failed
  • 00:08 krenair@tin: Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/333468 - needed to be handled before next window (duration: 00m 42s)

2017-01-21

  • 21:57 mobrovac@tin: Finished deploy [changeprop/deploy@2b980fa]: (no message) (duration: 00m 54s)
  • 21:56 mobrovac@tin: Starting deploy [changeprop/deploy@2b980fa]: (no message)
  • 20:02 legoktm@tin: Synchronized php-1.29.0-wmf.8/RELEASE-NOTES-1.29: for completeness (duration: 00m 39s)
  • 20:01 legoktm@tin: Synchronized php-1.29.0-wmf.8/resources: Revert "Added reason suggestion in block/delete/protect forms" (1/2) - T34950 (duration: 00m 39s)
  • 20:00 legoktm@tin: Synchronized php-1.29.0-wmf.8/includes: Revert "Added reason suggestion in block/delete/protect forms" (1/2) - T34950 (duration: 01m 31s)
  • 04:03 ema: graphite1003: carbon-cache@c restarted, it's been killed by OOM killer again
  • afk: disabled civicrm dedupe high numbers
  • afk: disabled civicrm dedupe
  • 02:02 l10nupdate@tin: LocalisationUpdate failed: git pull of extensions failed
  • 01:21 mobrovac@tin: Finished deploy [changeprop/deploy@eb27062]: (no message) (duration: 01m 03s)
  • 01:20 mobrovac@tin: Starting deploy [changeprop/deploy@eb27062]: (no message)
  • 00:36 volans: restarted carbon-cache@c on graphite1003 (was killed by oom-killer)
  • 00:20 mobrovac: restbase deploying 7c753fe6

2017-01-20

  • 23:37 mattflaschen@tin: Synchronized docroot: No-op file rename (duration: 00m 46s)
  • 23:36 mattflaschen@tin: Synchronized dblists: No-op file rename (duration: 00m 54s)
  • 19:55 robh: done fixing ulsfo serial in ulsfo
  • 19:45 robh: messing with ulsfo serial connections
  • 19:29 robh: cp4012 donating its redundant power supply to lvs4002 with redundant supplies
  • 19:12 ejegg: re-enabled fundraising Jenkins jobs
  • 19:01 ejegg: disabled fundraising jenkins jobs
  • 17:22 chasemp: shutdown eth1 on labstore1004 for testing
  • 17:17 andrewbogott: graceful'd apache on silver, in hopes that the wikitech instance api will update
  • 16:18 cmjohnson1: swapping cable eth0 labstore1004 (chasemp)
  • 16:03 jynus: restart and upgrade of db2066
  • 14:42 jynus: restart and upgrade of db2067
  • 13:30 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2047 - T153300 (duration: 00m 39s)
  • 12:32 ladsgroup@tin: Synchronized php-1.29.0-wmf.8/extensions/ORES/includes/Hooks.php: ORES database query fix (T155500) (duration: 00m 40s)
  • 12:13 Amir1: deploy wmf.8 in mwdebug1002 (T155500)
  • 10:48 godog: reload swift-proxy on ms-fe100* to pick up https://gerrit.wikimedia.org/r/333222
  • 10:39 elukey: manually forcing a /etc/init.d/apache2 reload on mw1259 (videoscaler) to replicate the effects of a logrotate run and test why alarms go off.
  • 10:15 moritzm: installing exim bugfix updates from latest jessie point release
  • 10:02 godog: reload swift-proxy on ms-fe1001 to pick up https://gerrit.wikimedia.org/r/333222
  • 09:19 jynus: rolling restart and upgrade of labsdb1009/10/11 to mariadb 10.1.21-2
  • 08:41 marostegui: Remove partitions on metawiki.pagelinks db2047 - T153300
  • 08:25 _joe_: restarting pybal on lvs1003/1006 to pick up config changes
  • 07:38 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2047 - T153300 (duration: 00m 48s)
  • 07:09 marostegui: Compress pagelinks tables on db1015 - T153739
  • 02:36 l10nupdate@tin: ResourceLoader cache refresh completed at Fri Jan 20 02:36:44 UTC 2017 (duration 5m 34s)
  • 02:31 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.8) (duration: 11m 23s)
  • 00:52 mutante: tin - keyholder disarm and arm again using new passphrase
  • 00:49 mutante: mira - arming keyholder after setting service/dumps/eventlogging/phabricator key passphrases to the same one (T154943)
  • 00:46 mutante: setting all deployment key passphrases to the one used for mw deploy - update key files in private repo (T154943)
  • 00:40 mobrovac@tin: Finished deploy [parsoid/deploy@465f9c4]: Restarting Parsoid everywhere for Node v6 switch T149331 (duration: 04m 21s)
  • 00:39 thcipriani@tin: Synchronized php-1.29.0-wmf.8/includes/specials/SpecialContributions.php: SWAT: SpecialContributions: Username input is not really required T155780 (duration: 00m 39s)
  • 00:35 mobrovac@tin: Starting deploy [parsoid/deploy@465f9c4]: Restarting Parsoid everywhere for Node v6 switch T149331
  • 00:34 thcipriani@tin: Synchronized php-1.29.0-wmf.8/resources/lib/oojs-ui: SWAT: resources: Update OOjs UI with fixes on top of v0.18.3 T155728 (duration: 00m 41s)
  • 00:24 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Wikidata description taglines shown on English Wikipedia T152743 (duration: 00m 39s)
  • 00:22 volans: apt-upgrading nodejs to v6 on the rest of parsoid hosts (a deploy with restart will follow) T149331

2017-01-19

  • 23:34 mutante: force puppet run on restbase1*
  • 23:32 volans: upgrading node to v6 on wtp1003 T149331
  • 23:31 mutante: force puppet run on restbase2*
  • 23:31 mutante: icinga - replace check command names in puppet_services.cfg for change 333010
  • 23:26 mobrovac@tin: Finished deploy [eventstreams/deploy@0d1d9c6]: Bump preq to 0.5.2 for Node v6 (duration: 02m 11s)
  • 23:24 mobrovac@tin: Starting deploy [eventstreams/deploy@0d1d9c6]: Bump preq to 0.5.2 for Node v6
  • 23:16 mobrovac@tin: Finished deploy [cxserver/deploy@5ae4f8b]: Bump preq to 0.5.2 for Node v6 (duration: 01m 56s)
  • 23:14 mobrovac@tin: Starting deploy [cxserver/deploy@5ae4f8b]: Bump preq to 0.5.2 for Node v6
  • 23:11 mobrovac@tin: Finished deploy [graphoid/deploy@da37386]: Bump preq to 0.5.2 for Node v6 (duration: 02m 21s)
  • 23:10 ejegg: updated SmashPig from f05c9a3 to 03880ce
  • 23:08 mobrovac@tin: Starting deploy [graphoid/deploy@da37386]: Bump preq to 0.5.2 for Node v6
  • 22:59 ppchelko@tin: Finished deploy [eventstreams/deploy@fe77f19]: Deploy for switching to node 6 T149331 (duration: 01m 30s)
  • 22:58 mobrovac@tin: Finished deploy [trending-edits/deploy@0abcf25]: Switching to node 6 T149331 (duration: 01m 59s)
  • 22:58 ppchelko@tin: Starting deploy [eventstreams/deploy@fe77f19]: Deploy for switching to node 6 T149331
  • 22:58 ppchelko@tin: Finished deploy [cxserver/deploy@ff0225e]: Deploy for switching to node 6 T149331 (duration: 02m 12s)
  • 22:56 mobrovac@tin: Finished deploy [electron-render/deploy@f1df2d3]: Switching to node 6 T149331 (duration: 01m 58s)
  • 22:56 mobrovac@tin: Starting deploy [trending-edits/deploy@0abcf25]: Switching to node 6 T149331
  • 22:56 mobrovac@tin: Finished deploy [mobileapps/deploy@cacb3c9]: Switching to node 6 T149331 (duration: 02m 41s)
  • 22:55 ppchelko@tin: Starting deploy [cxserver/deploy@ff0225e]: Deploy for switching to node 6 T149331
  • 22:55 ppchelko@tin: Finished deploy [citoid/deploy@95df861]: Deploy for switching to node 6 T149331 (duration: 02m 18s)
  • 22:54 mobrovac@tin: Starting deploy [electron-render/deploy@f1df2d3]: Switching to node 6 T149331
  • 22:54 mobrovac@tin: Finished deploy [mathoid/deploy@ba3217e]: (no message) (duration: 02m 02s)
  • 22:53 mobrovac@tin: Starting deploy [mobileapps/deploy@cacb3c9]: Switching to node 6 T149331
  • 22:53 mobrovac@tin: Finished deploy [graphoid/deploy@f872f94]: Switching to node 6 T149331 (duration: 01m 39s)
  • 22:53 ppchelko@tin: Starting deploy [citoid/deploy@95df861]: Deploy for switching to node 6 T149331
  • 22:52 ppchelko@tin: Finished deploy [changeprop/deploy@ffd0b8b]: Deploy for switching to node 6 T149331 (duration: 00m 58s)
  • 22:52 mobrovac@tin: Starting deploy [mathoid/deploy@ba3217e]: (no message)
  • 22:51 mobrovac@tin: Starting deploy [graphoid/deploy@f872f94]: Switching to node 6 T149331
  • 22:51 ppchelko@tin: Starting deploy [changeprop/deploy@ffd0b8b]: Deploy for switching to node 6 T149331
  • 22:51 mutante: scb1003,scb1004 - upgrade nodejs
  • 22:51 ppchelko@tin: Finished deploy [changeprop/deploy@ffd0b8b]: Canary deploy for switching to node 6 T149331 (duration: 07m 36s)
  • 22:48 mobrovac@tin: Finished deploy [mathoid/deploy@ba3217e]: (no message) (duration: 03m 24s)
  • 22:47 mobrovac@tin: Finished deploy [graphoid/deploy@f872f94]: Switching to node 6 T149331 (duration: 01m 43s)
  • 22:45 mobrovac@tin: Starting deploy [graphoid/deploy@f872f94]: Switching to node 6 T149331
  • 22:45 mobrovac@tin: Finished deploy [graphoid/deploy@f872f94]: Switching to node 6 T149331 (duration: 01m 10s)
  • 22:45 mobrovac@tin: Starting deploy [mathoid/deploy@ba3217e]: (no message)
  • 22:44 mobrovac@tin: Starting deploy [graphoid/deploy@f872f94]: Switching to node 6 T149331
  • 22:44 ppchelko@tin: Starting deploy [changeprop/deploy@ffd0b8b]: Canary deploy for switching to node 6 T149331
  • 22:41 mutante: scb1001-1004 - upgraded nodejs version
  • 22:38 mutante: scb2003 - repool, scb2001,scb2002 - upgrade nodejs, libuv1 packages
  • 22:35 mutante: scb2003 - depool, upgrade nodejs, libuv1 packages
  • 22:33 mutante: scb2004 - re-pooled
  • 21:35 chasemp: rebooting labstore1004
  • 21:33 chasemp: failover secondary labstore cluster from 1004 to 1004
  • 21:17 chasemp: force non tools on NFS to go ro
  • 21:03 volans: upgrading node to v6 on wtp1002 T149331
  • 20:46 mutante: scb2004 - upgrading nodejs, libuv1
  • 20:45 volans: upgrading node to v6 on wtp2003 T149331
  • 20:44 mutante: depooling scb2004 for nodejs install
  • 20:26 volans: upgrading node to v6 on wtp2002 T149331
  • 20:09 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.8
  • 20:05 dereckson@tin: Synchronized wmf-config/throttle.php: Add throttle rule for BAMU event (T154312) (duration: 00m 39s)
  • 19:44 ejegg: updated SmashPig from 48675c3 to f05c9a3
  • 19:40 mutante: switching dumps.wikimedia.org to Letsencrypt SSL cert
  • 19:20 jynus: restarting db1069:3311 due to query being "stuck" on tokudb table
  • 19:13 demon@tin: Synchronized wmf-config/CommonSettings.php: ContentTranslation: Enable publishing article in testwiki (2/2) (duration: 00m 39s)
  • 19:12 demon@tin: Synchronized wmf-config/InitialiseSettings.php: ContentTranslation: Enable publishing article in testwiki (1/2) (duration: 00m 39s)
  • 19:06 demon@tin: Synchronized wmf-config/CommonSettings.php: Double $wgTranscodeBackgroundTimeLimit to compensate for threading (duration: 00m 47s)
  • 18:54 ema: libvmod-header removed from carbon, varnish-modules provides it
  • 18:51 mutante: dataset1001 - temp disabling puppet, ms1001 - switching to Letsencrypt cert
  • 18:34 jynus: aborting rolling restart on labsdb1010, labsdb1011 due to package bug to be fixed on 10.1.21-2
  • 18:19 mobrovac: restbase updating firejail in production
  • 17:52 jynus: rolling restart and upgrade of labsdb1009/10/11 to mariadb 10.1.21
  • 17:47 Dereckson: Reattach Zlazstadpieroniebomiurwieszkabelodinternetu CentralAuth account (T155184)
  • 17:26 jynus: restarting and upgrading mariadb on labsdb1004 to 10.0.29
  • 14:40 Dereckson: EU SWAT done
  • 14:40 Dereckson: `mwscript namespaceDupes.php sawiki --fix` (T101634)
  • 14:36 ottomata: restarting apache/puppetmaster on labcontrol1001 to try to fix 'invalid byte sequence in US-ASCII' puppet error
  • 14:34 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Fix Portal talk namespace name on Sanskrit Wikipedia (T101634) (duration: 00m 39s)
  • 14:25 dereckson@tin: Synchronized wmf-config: Add noratelimit user right to translation admins on Commons (T155162) (duration: 00m 42s)
  • 13:21 marostegui: Compressing revision,pagelinks and templatelinks tables on db1035 - T110504
  • 11:41 marostegui: Compressing dewiki db1045 - T155399
  • 10:14 marostegui: Compressing templatelinks tables on db1015 - T153739
  • 09:32 moritzm: upgrading firejail on image scalers
  • 08:57 godog: bounce udp2log on fluorine after https://gerrit.wikimedia.org/r/313604
  • 07:45 volans@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=mw2098.codfw.wmnet
  • 07:42 volans@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=mw2098.wmnet
  • 07:28 dereckson@tin: Synchronized wmf-config/throttle.php: Fix throttle rule for KCES IMR edit-a-thon (duration: 02m 42s)
  • 07:18 marostegui: Compressing enwikivoyage.text and shwiki.logging tables on db1044 - T153826
  • 07:14 marostegui: Compressing enwikivoyage.text and shwiki.logging tables on db1038 - T154465
  • 02:28 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.7) (duration: 07m 28s)
  • 01:50 mutante: install - if in private1-c-eqiad, private1-b-codfw, you are using install1001 for both DHCP and TFTP, if in other networks you still use carbon as DHCP but then also install1001 as TFTP
  • 01:47 mutante: install1001/2001 - re-enabled, carbon is still DHCP for some rows
  • 01:23 mutante: install1001 - re-enable puppet - install2001 - same thing, temp disable and live-hack mw2251 to use trusty installer
  • 01:16 mutante: switching mw2251 to trusty-installer for test
  • 01:14 mutante: temp disable puppet on install1001 for papaul debugging
  • 00:24 maxsem@tin: Synchronized php-1.29.0-wmf.8/extensions/Graph: SWAT https://gerrit.wikimedia.org/r/#/c/332916/1 (duration: 00m 40s)

2017-01-18

  • 23:38 demon@tin: Synchronized php-1.29.0-wmf.8/extensions/Collection: Unbreak (duration: 00m 40s)
  • 22:59 demon@tin: Synchronized php-1.29.0-wmf.8/extensions/ProofreadPage/includes/index/ProofreadIndexPage.php: Unbreak, T155682 (duration: 00m 39s)
  • 22:49 demon@tin: Synchronized php-1.29.0-wmf.8/extensions/LiquidThreads/classes/Hooks.php: Unbreak hook mess (duration: 00m 41s)
  • 22:48 demon@tin: Synchronized php-1.29.0-wmf.8/includes/widget/search/FullSearchResultWidget.php: Unbreak hook mess (duration: 00m 45s)
  • 22:46 madhuvishy: Reenabled nfs-exportd and puppet on labstore1004. All of misc being exported as rw now. T154336
  • 21:32 bd808: Updated wikimania-scholarships to 29ba0ec "Add Tulu (tcy) to Communities" (T155666)
  • 21:18 volans: restarted pybal on lvs2001 (active) T134893
  • 21:00 volans: restarted pybal on lvs2004 (passive) T134893
  • 20:48 demon@tin: Finished scap: group1 to wmf.8 (duration: 46m 33s)
  • 20:47 volans: Upgraded nodejs to v6 on wtp1001 T149331
  • 20:31 nuria@tin: Finished deploy [analytics/refinery@666d98d]: (no message) (duration: 02m 19s)
  • 20:28 nuria@tin: Starting deploy [analytics/refinery@666d98d]: (no message)
  • 20:01 demon@tin: Started scap: group1 to wmf.8
  • 19:56 volans: restarted pybal on lvs2003
  • 19:51 volans: restarted pybal on lvs2006
  • 19:26 demon@tin: Synchronized dblists: Remove old compact lang list dblist (duration: 00m 39s)
  • 19:25 volans: Upgrading nodejs to v6 on wtp2001 T149331
  • 19:25 demon@tin: Synchronized docroot/noc/conf: Using new compact lang list dblist (duration: 00m 39s)
  • 19:24 demon@tin: Synchronized tests/cirrusTest.php: Use new compact lang list dblist (duration: 00m 39s)
  • 19:22 papaul: OS installation on mw2251-mw2260
  • 19:22 demon@tin: Synchronized wmf-config: Use new compact lang links dblist (duration: 00m 41s)
  • 19:21 demon@tin: Synchronized dblists/compact-language-links.dblist: New dblist (duration: 00m 39s)
  • 19:12 volans: Upgrading nodejs to v6 on ruthenium T149331
  • 19:06 dereckson@tin: Synchronized wmf-config/throttle.php: Add throttle rule for KCES IMR edit-a-thon (T154312) (duration: 00m 39s)
  • 18:35 demon@tin: Synchronized multiversion/MWMultiVersion.php: Swapping 500 -> 400 when specifying invalid host headers (duration: 00m 39s)
  • 18:28 demon@tin: Synchronized multiversion/MWMultiVersion.php: minor cleanup (duration: 00m 48s)
  • 18:03 madhuvishy: Rolling out https://gerrit.wikimedia.org/r/#/c/332735/ across labs instances T154336
  • 17:54 madhuvishy: Disabled (systemctl disable) nfs-export on labstore1001 and 1004 to prevent auto restart from bringing them back up T154336
  • 17:42 mobrovac: restbase deploying 3027682
  • 17:40 madhuvishy: Starting final sync of latest diff from labstore1001 to labstore-secondary T154336
  • 17:38 madhuvishy: Disabling puppet on labstore1001 and 1004 to make sure nfs exports are not overridden T154336
  • 17:37 madhuvishy: Exporting all misc shares from labstore1004 as RO T154336
  • 17:30 madhuvishy: Stopping nfs-exportd on labstore1004 T154336
  • 17:29 madhuvishy: Exported all misc exports as RO on labstore1001 T154336
  • 17:26 madhuvishy: Stopping nfs-exportd on labstore1001 T154336
  • 17:13 madhuvishy: Disabling puppet across labs instances with NFS (/home and/or /data/project) mounted for T154336
  • 17:12 madhuvishy: Silenced shinken, and icinga on labstore1001 for misc nfs migration T154336
  • 15:45 oblivian@puppetmaster1001: conftool action : set/pooled=inactive; selector: service=nginx,cluster=api_appserver,dc=eqiad,name=mw122[6-9].*
  • 15:45 oblivian@puppetmaster1001: conftool action : set/pooled=inactive; selector: service=nginx,cluster=api_appserver,dc=eqiad,name=mw123[0-5].*
  • 15:41 oblivian@puppetmaster1001: conftool action : set/weight=15; selector: service=apache2,cluster=api_appserver,dc=eqiad,name=mw1(1[8-9]|2[0-1]|22[0-5]).*
  • 15:24 zeljkof: finished EU SWAT
  • 15:23 zfilipin@tin: Synchronized php-1.29.0-wmf.7/maintenance/importImages.php: SWAT: maintenance/importImages: Dont sleep after the last upload (duration: 00m 41s)
  • 15:07 hashar: tin.eqiad.wmnet : committed an uncommitted live hack for php-1.29.0-wmf.7/includes/AutoLoader.php by ostriches
  • 15:06 zeljkof: extending EU SWAT until 332766 is deployed
  • 14:39 zfilipin@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Increase $wgHTTPImportTimeout to 50 seconds (T155209) (duration: 00m 39s)
  • 14:31 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set wgDisableUserGroupExpiry to true on production, false on labs (T155605) (duration: 00m 40s)
  • 14:30 zfilipin@tin: Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: Set wgDisableUserGroupExpiry to true on production, false on labs (T155605) (duration: 00m 40s)
  • 14:03 moritzm: upgrading firejail on aqs cluster
  • 13:12 moritzm: uploaded firejail 0.9.44.6 for jessie-wikimedia to carbon
  • 12:21 marostegui: Enable gtid_domain_id on m3 - T149418
  • 12:06 moritzm: installing libio-socket-ssl-perl bugfix updates from jessie point release
  • 11:42 moritzm: installing sed bugfix updates from jessie point release
  • 11:17 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2064 - T154097 (duration: 00m 39s)
  • 11:09 moritzm: restarting mediawiki canary servers to pick up cairo and libpng updates
  • 10:38 moritzm: installing libxml security updates
  • 10:38 oblivian@puppetmaster1001: conftool action : set/pooled=no; selector: service=nginx,cluster=api_appserver,dc=eqiad,name=mw123[0-5].*
  • 10:11 godog: pool ms-fe200[789] T152612
  • 10:10 marostegui: Restart mysql dbstore2001 to enable gtid_domain_id manually before deploying it on m3 - T149418
  • 10:05 marostegui: Remove partitions from enwiktionary.templatelinks on db2064 - T154097
  • 10:05 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2064 - T154097 (duration: 00m 45s)
  • 09:56 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2063 - T154097 (duration: 00m 48s)
  • 09:40 marostegui: Restart mysql dbstore2002 to enable gtid_domain_id manually before deploying it on m3 - T149418
  • 08:51 marostegui: Remove partitions from enwiktionary.templatelinks on db2063 - T154097
  • 08:50 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2063 - T154097 (duration: 00m 39s)
  • 08:33 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2060 - T154031 (duration: 00m 40s)
  • 08:18 marostegui: Compressing templatelinks tables on db1035 - T154465
  • 08:16 marostegui: Compressing templatelinks tables on db1038 - T154465
  • 08:00 _joe_: restarting pybal on lvs1003
  • 07:56 _joe_: restarting pybal on lvs1003
  • 07:34 oblivian@puppetmaster1001: conftool action : set/pooled=no; selector: service=nginx,cluster=api_appserver,dc=eqiad,name=mw122[6-9].*
  • 07:32 _joe_: depooling mw1226-mw1235 from the https pool in eqiad, T152074
  • 07:30 oblivian@puppetmaster1001: conftool action : set/pooled=inactive; selector: service=nginx,cluster=api_appserver,dc=eqiad,name=mw12[7-9].*
  • 07:25 marostegui: Restart MySQL dbstore2001 to apply InnoDB defaults
  • 03:05 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Jan 18 03:05:47 UTC 2017 (duration 5m 38s)
  • 03:00 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.8) (duration: 13m 22s)
  • 02:29 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.7) (duration: 08m 23s)
  • 02:00 ejegg: updated SmashPig from 03ef6b1 to 48675c3
  • 01:21 mobrovac@tin: Finished deploy [citoid/deploy@9f93a00]: (no message) (duration: 04m 14s)
  • 01:17 mobrovac@tin: Starting deploy [citoid/deploy@9f93a00]: (no message)
  • 00:59 mobrovac@tin: Finished deploy [trending-edits/deploy@1d53b7c]: fixes for T153122 and T145571 (duration: 05m 06s)
  • 00:54 mobrovac@tin: Starting deploy [trending-edits/deploy@1d53b7c]: fixes for T153122 and T145571
  • 00:35 demon@tin: Synchronized w/mobilelanding.php: Last major fix for multiversion (duration: 00m 45s)
  • 00:29 ejegg: updated SmashPig from 3da597f to 03ef6b1
  • 00:00 demon@tin: Synchronized multiversion/MWVersion.php: Swap to using MWMultiVersion and make this a fallback (duration: 00m 39s)

2017-01-17

  • 23:47 demon@tin: Synchronized w: Step 4/∞ of multiversion cleanups (duration: 00m 39s)
  • 23:42 demon@tin: Synchronized multiversion/MWScript.php: Step 3/∞ of multiversion cleanups (duration: 00m 39s)
  • 23:21 demon@tin: Synchronized rpc/RunJobs.php: Step 2/∞ of multiversion cleanups (duration: 00m 39s)
  • 22:30 demon@tin: Synchronized multiversion: Step 1/∞ of multiversion cleanups (duration: 00m 55s)
  • 21:48 demon@tin: Synchronized multiversion/MWVersion.php: Removing old getMediaWikiCli() entry point, unused (duration: 00m 39s)
  • 20:41 demon@tin: Synchronized php-1.29.0-wmf.8/extensions/LiquidThreads/classes/Hooks.php: Fix warning about pass-by-ref (duration: 00m 40s)
  • 20:09 ema: restarting hhvm on mw1227 - hhvm-dump-debug in /tmp/hhvm.25127.bt
  • 20:08 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.8
  • 19:52 demon@tin: Finished scap: testwiki to wmf.8 + rebuild l10n (duration: 46m 21s)
  • 19:36 ejegg: updated payments-wiki from 1f9ea80 to c22353b
  • 19:06 demon@tin: Started scap: testwiki to wmf.8 + rebuild l10n
  • 19:05 demon@tin: Synchronized wmf-config/throttle.php: throttle rule for T155493 (duration: 00m 40s)
  • 19:04 demon@tin: Synchronized wmf-config/InitialiseSettings.php: Enable subpages in NS_MAIN in eswikiversity (duration: 00m 39s)
  • 19:02 demon@tin: Synchronized wmf-config/InitialiseSettings.php: Add *.leventhalmap.org to the copyupload whitelist (duration: 00m 39s)
  • 19:01 demon@tin: Synchronized wmf-config/InitialiseSettings.php: avwiki namespace tweaks, T155321 (duration: 00m 39s)
  • 19:00 demon@tin: Synchronized wmf-config/throttle.php: T155510 throttle rule (duration: 01m 36s)
  • 18:03 mobrovac: restbase deploying a0e542b, switching to Node v6 T149331
  • 17:48 mobrovac: restbase installing node v6.9.1 on the cluster T149331
  • 16:29 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2056 - T154097 (duration: 00m 48s)
  • 16:22 marostegui: Powering off db2060 for maintenance - T154031
  • 16:16 filippo@puppetmaster1001: conftool action : set/pooled=yes; selector: name=ms-fe2005.codfw.wmnet
  • 15:01 moritzm: installing bash security updates
  • 14:10 moritzm: installing bind9 security updates
  • 13:33 moritzm: installing libpng security updates
  • 12:29 marostegui: Remove partitions from enwiktionary.templatelinks on db2056 - T154097
  • 12:27 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2056 - T154097 (duration: 00m 42s)
  • 12:16 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2049 - T154097 (duration: 00m 47s)
  • 11:54 moritzm: installing potrace security updates
  • 11:32 moritzm: installing w3m security updates
  • 11:22 moritzm: installing tre security updates
  • 11:19 moritzm: installing python-werkzeug security updates
  • 11:15 marostegui: Remove partitions from enwiktionary.templatelinks on db2049 - T154097
  • 11:13 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2049 - T154097 (duration: 00m 38s)
  • 10:58 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2041 - T154097 (duration: 00m 38s)
  • 10:43 moritzm: installing file/libmagic security updates
  • 10:35 hashar: CI switched NodeJS from v4 to v6 T155443 T149331
  • 10:22 moritzm: installing jq security updates
  • 10:16 hashar: Updating CI Jessie image for NodeJs 4 -> 6 upgrade. T155443
  • 10:06 moritzm: installing libwmf security updates
  • 10:02 marostegui: Remove partitions from enwiktionary.templatelinks on db2041 - T154097
  • 09:59 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2041 - T154097 (duration: 00m 38s)
  • 09:22 moritzm: installing tomcat security updates
  • 08:42 marostegui: Compressing wikidatawiki on db1026 - https://phabricator.wikimedia.org/T154929
  • 08:33 moritzm: installing tiff security updates
  • 07:50 marostegui: Remove partitions from enwiktionary.templatelinks on dbstore2001 - T154097
  • 07:26 marostegui: Compressing revision tables db1035 (depooled)
  • 06:58 marostegui: Compressing cebwiki/templatelinks (215G) table on db1038 - T154465
  • 02:26 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Jan 17 02:26:07 UTC 2017 (duration 4m 22s)
  • 02:21 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.7) (duration: 07m 20s)

2017-01-16

  • 18:42 moritzm: uploaded nodejs 6.9.1 for jessie-wikimedia to carbon
  • 15:01 elukey: restarting hhvm on mw1167 - hhvm-dump-debug in /tmp/hhvm.20360.bt
  • 14:47 hashar: European SWAT complete
  • 14:44 hashar@tin: Synchronized wmf-config/throttle.php: Add a new throttle rule - T155416 (duration: 00m 38s)
  • 14:26 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Namespace aliases on Bhojpuri Wikipedia (bhwiki) - T155278 (duration: 00m 41s)
  • 14:20 hashar@tin: Synchronized wmf-config/throttle.php: Add one throttle rule + remove obsolete ones T155345 (duration: 00m 38s)
  • 14:18 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: wgBabelMainCategory for cswikiversity to Uživatel %code% T155301 (duration: 00m 39s)
  • 13:12 moritzm: installing pysaml2 security updates
  • 12:26 moritzm: installing pdns-recursor security updates
  • 10:35 marostegui: Compressing templatelinks tables on db1044 (depooled) - T153826
  • 10:30 marostegui: Compressing pagelinks tables on db1038 - T154465
  • 09:13 marostegui: Compressing dewiki on db1026 - T154929
  • 08:56 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2034 - T149553 (duration: 00m 38s)
  • 02:25 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Jan 16 02:25:46 UTC 2017 (duration 4m 21s)
  • 02:21 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.7) (duration: 08m 04s)

2017-01-15

  • 02:25 l10nupdate@tin: ResourceLoader cache refresh completed at Sun Jan 15 02:25:27 UTC 2017 (duration 4m 23s)
  • 02:21 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.7) (duration: 07m 46s)

2017-01-14

  • 02:35 l10nupdate@tin: ResourceLoader cache refresh completed at Sat Jan 14 02:35:07 UTC 2017 (duration 4m 25s)
  • 02:30 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.7) (duration: 12m 03s)

2017-01-13

  • 22:56 godog: delete labs instance data older than 60d from graphite[21]001, low disk space
  • 02:25 l10nupdate@tin: ResourceLoader cache refresh completed at Fri Jan 13 02:25:57 UTC 2017 (duration 5m 16s)
  • 02:20 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.7) (duration: 07m 11s)
  • 01:32 bsitzmann@tin: Finished deploy [trending-edits/deploy@cf388a9]: Update trending-edits to 421fa63 (duration: 01m 53s)
  • 01:30 bsitzmann@tin: Starting deploy [trending-edits/deploy@cf388a9]: Update trending-edits to 421fa63
  • 00:12 demon@tin: Synchronized multiversion: Clean up cli entry point (duration: 00m 54s)

2017-01-12

  • 23:05 demon@tin: Synchronized README: no-op for force co-master sync (duration: 00m 40s)
  • 22:49 demon@tin: Synchronized docroot/foundation: Yay no more powerpoints (duration: 00m 38s)
  • 22:38 demon@tin: Synchronized docroot/foundation/presentations: removing some of these powerpoints (duration: 00m 38s)
  • 22:08 maxsem@tin: Synchronized php-1.29.0-wmf.7/extensions/Graph/includes/ApiGraph.php: Debug for T155057 (duration: 00m 38s)
  • 18:13 demon@tin: Synchronized wmf-config/InitialiseSettings.php: Use HD logos for (nap|os|pl|pt)wiki (duration: 00m 41s)
  • 18:12 demon@tin: Synchronized static/images/project-logos: HD logos for (nap|os|pl|pt)wiki (duration: 00m 39s)
  • 09:16 akosiaris: T155112 upload Vagrant 1.9.1 to apt.wikimedia.org/jessie-wikimedia/thirdparty and apt.wikimedia.org/trusty-wikimedia/thirdparty
  • 08:59 hashar: disabling puppet on contint1001 to live hack apache conf ( T150727 )
  • 02:46 demon@tin: Synchronized wmf-config/interwiki.php: T154225 (duration: 00m 38s)
  • 02:37 demon@tin: Synchronized wmf-config/InitialiseSettings-labs.php: no-op, completeness (duration: 00m 38s)
  • 02:36 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Jan 12 02:36:23 UTC 2017 (duration 5m 15s)
  • 02:31 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.7) (duration: 11m 12s)
  • 01:33 Reedy: running scap pull on tin
  • 00:29 demon@tin: Synchronized wmf-config/InitialiseSettings.php: oathauth group for wikitech (duration: 00m 38s)
  • 00:21 demon@tin: Synchronized docroot/noc/db.php: (no message) (duration: 00m 39s)
  • 00:17 reedy@tin: Synchronized wmf-config: More consistency for various commits (duration: 00m 40s)
  • 00:12 nuria: restarted apache2 and mysql on bohrium to see if mysql no connection errors disappear
  • 00:12 demon@tin: Synchronized wmf-config/InitialiseSettings.php: Wikidata lang config (duration: 00m 38s)

2017-01-11

  • 23:55 demon@tin: Synchronized wmf-config/InitialiseSettings.php: Minerva hacks (duration: 00m 38s)
  • 23:47 reedy@tin: Synchronized wmf-config: consistency (duration: 00m 41s)
  • 23:16 demon@tin: Synchronized wmf-config/CommonSettings.php: video transcode jobqueue stuff for Brion (duration: 00m 38s)
  • 23:10 demon@tin: Synchronized w/static.php: For Timo <3 (duration: 00m 40s)
  • 22:48 demon@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Use new metawiki logo, no-op in prod (duration: 00m 38s)
  • 22:47 demon@tin: Synchronized static/images/project-logos: beta logos (duration: 00m 40s)
  • 22:44 demon@tin: Synchronized wmf-config/CommonSettings.php: commentfix (duration: 00m 38s)
  • 22:39 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: Simplify ores config. Enable sitenotice banners for arwiki (duration: 00m 39s)
  • 22:32 demon@tin: Synchronized wmf-config/abusefilter.php: comment fix (duration: 00m 39s)
  • 22:26 elukey: added mw1239.eqiad.wmnet back to service - T148421
  • 22:20 elukey: restarting hhvm on mw1198 (dump-debug in /tmp/hhvm.9737.bt)
  • 22:13 demon@tin: Synchronized wmf-config/abusefilter.php: Set $wgAbuseFilterNotificationsPrivate = true; for Meta-Wiki (duration: 00m 40s)
  • 22:04 demon@tin: Synchronized wmf-config/flaggedrevs.php: Deprecated variable cleanup (duration: 00m 38s)
  • 21:54 demon@tin: Synchronized scap/plugins/wmf-beta-autoupdate.py: no-op, not yet used (duration: 00m 38s)
  • 21:45 demon@tin: Synchronized wmf-config/CommonSettings-labs.php: no-op (duration: 00m 38s)
  • 21:28 demon@tin: Synchronized wmf-config: massmessage hack cleanup + comments on kartographer wikivoyage mode (duration: 00m 41s)
  • 21:17 demon@tin: Synchronized wmf-config/InitialiseSettings.php: use new HD logos (duration: 00m 38s)
  • 21:16 demon@tin: Synchronized static/images/project-logos/notifications: New HD logos (duration: 00m 38s)
  • 21:10 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: pawiki logos. remove technican from trwikiquote (duration: 00m 38s)
  • 21:09 reedy@tin: Synchronized static/images: pawiki (duration: 00m 42s)
  • 21:03 Reedy: update collation of fiwikivoyage T151570
  • 21:02 reedy@tin: Synchronized php-1.29.0-wmf.7/extensions/CentralAuth/extension.json: fix name (duration: 00m 41s)
  • 21:01 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: Translation namespace for mlwikisource. fiwikivoyage collation (duration: 00m 40s)
  • 20:53 reedy@tin: Synchronized wmf-config/CommonSettings.php: Upgrade Collections license URL to HTTPS (duration: 00m 57s)
  • 20:52 reedy@tin: Synchronized wmf-config/throttle.php: Fix throttle (duration: 00m 42s)
  • 20:44 Dereckson: Reset user e-mail for account for Panam2014
  • 20:43 demon@tin: Synchronized tests/noc-conf/NOCDblistTest.php: No-op (duration: 00m 40s)
  • 20:40 demon@tin: Synchronized wmf-config/InitialiseSettings-labs.php: no-op (duration: 00m 40s)
  • 19:36 demon@tin: Synchronized multiversion: rollback (duration: 00m 56s)
  • 19:34 demon@tin: Synchronized multiversion: MWVersion fallbacks & such (duration: 00m 56s)
  • 19:27 demon@tin: Synchronized php-1.29.0-wmf.7/extensions/FlaggedRevs: Stupid errors (duration: 00m 46s)
  • 18:56 demon@tin: Synchronized multiversion/MWMultiVersion.php: Attempt #2 for Multiversion cleanup (duration: 00m 41s)
  • 18:08 ebernhardson: restart elasticsaerch on relforge100[12] for new test version of ltr plugin
  • 14:12 hashar: European SWAT completed
  • 14:11 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Import source on bd.wikimedia.org T154990 + Turn of patrolling on ruwiki T154285 (duration: 00m 42s)
  • 14:10 hashar@tin: Synchronized composer.json: build: Update PHPUnit from 3.7 to 4.8, add phplint to composer-test - T85947 (duration: 00m 45s)
  • 14:09 hashar@tin: Synchronized composer.lock: build: Update PHPUnit from 3.7 to 4.8, add phplint to composer-test - T85947 (duration: 00m 55s)
  • 14:06 hashar: scap pull on terbium
  • 02:35 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Jan 11 02:35:46 UTC 2017 (duration 4m 31s)
  • 02:31 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.7) (duration: 10m 47s)
  • 00:39 _joe_: restart hhvm on mw1182, stuck on HPHP::Treadmill::getAgeOldestRequest

2017-01-10

  • 23:09 hoo: Ran DELETE FROM wbc_entity_usage WHERE eu_row_id IN(1714177, 1714178, 1714179, 1714180, 1714181, 1714182, 1714183, 1714184, 3914375); on s5 master (T147630)
  • 22:53 demon@tin: Synchronized php-1.29.0-wmf.7/extensions/VisualEditor/ApiVisualEditor.php: T154962 logspam (duration: 00m 41s)
  • 22:46 mutante: gerrit restarting for config change 331553
  • 22:13 reedy@tin: Synchronized php-1.29.0-wmf.7/includes/registration/ExtensionRegistry.php: (no message) (duration: 00m 43s)
  • 22:12 reedy@tin: Synchronized php-1.29.0-wmf.7/extensions/Wikidata: (no message) (duration: 02m 21s)
  • 22:06 demon@tin: Synchronized php-1.29.0-wmf.7/includes/libs/objectcache/WANObjectCache.php: Silence obnoxious replag errors (duration: 00m 42s)
  • 21:33 eileen2: civicrm updated from b26844d to af8d735
  • 21:22 mutante: cp3048 - labservices1001 - ran puppet, in this case it wasn't about gerrit, but recovered too
  • 21:15 mutante: sca2004, labsdb1003 - ran puppet (they wanted to git clone during gerrit restart)
  • 21:04 mutante: gerrit restarting for config change 49993 (T40114)
  • 20:16 dereckson@tin: Synchronized php-1.29.0-wmf.7/extensions/SemanticMediaWiki: Remove deprecated function usages (T147924) (duration: 00m 49s)
  • 20:14 dereckson@tin: Synchronized php-1.29.0-wmf.7/extensions/UploadWizard/resources/transports/mw.FormDataTransport.js: mw.FormDataTransport: Don't remove Unicode characters from temp filename (T155039) (duration: 00m 41s)
  • 19:49 demon@tin: Synchronized scap/plugins/prep.py: another no-op (duration: 00m 41s)
  • 19:39 demon@tin: Synchronized scap/plugins/prep.py: prod no-op, for completeness (duration: 00m 40s)
  • 18:41 legoktm: re-attached User:Fuu5tgsrygr / T154983
  • 04:30 dereckson@tin: Synchronized wmf-config/throttle.php: Fix Dayanand College Solapur event throttle rule (T154312) (duration: 00m 44s)
  • 02:26 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Jan 10 02:26:47 UTC 2017 (duration 4m 22s)
  • 02:22 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.7) (duration: 07m 46s)
  • 01:34 reedy@tin: Synchronized rpc/RunJobs.php: revert (duration: 00m 40s)
  • 01:32 reedy@tin: Synchronized w: revert 0a2a096 (duration: 00m 40s)
  • 01:31 reedy@tin: Synchronized multiversion: revert 0a2a096 (duration: 00m 56s)
  • 01:01 Dereckson: Updated articles count on pl.wikisource: 491 100 (T154711)
  • 00:59 Dereckson: Fixed links with namespaceDupes on pl.wikisource
  • 00:58 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Add Collection namespace to the Polish Wikisource (T154711) (duration: 00m 41s)

2017-01-09

  • 23:38 reedy@tin: Synchronized wmf-config/CommonSettings.php: wfLoadExtension (duration: 00m 40s)
  • 23:34 reedy@tin: Synchronized wmf-config/extension-list: More to extension.json (duration: 00m 40s)
  • 23:29 demon@tin: Synchronized multiversion: Final batch of MWVersion cleanup (in song form) (duration: 00m 56s)
  • 23:28 demon@tin: Synchronized rpc/RunJobs.php: More cleanup songs (duration: 00m 40s)
  • 23:28 mutante: ganglia web - replacing SSL cert with Letsencrypt
  • 23:26 demon@tin: Synchronized w: Cleanup cleanup everybody do your share (duration: 00m 40s)
  • 23:25 demon@tin: Synchronized multiversion/MWMultiVersion.php: Cleanup cleanup everybody everywhere (duration: 00m 40s)
  • 23:09 reedy@tin: Synchronized wmf-config/CommonSettings-labs.php: T154927 (duration: 00m 41s)
  • 23:08 reedy@tin: Synchronized wmf-config/InitialiseSettings-labs.php: T154927 (duration: 00m 42s)
  • 23:07 reedy@tin: Synchronized wmf-config/extension-list-labs: T154927 (duration: 00m 41s)
  • 22:53 robh: updating lists.w.o to use LE cert
  • 21:50 akosiaris: service restart zotero on sca1003, sca1004. Zotero OOMed again as usual
  • 19:49 robh: updating librenms.wikimedia.org cert, netmon1001 only system affected
  • 18:16 paravoid: rebooting and powercycling mira, CPU frequency throttled, suspecting firmware bug
  • 17:51 hoo: Updated the Wikidata property suggester with data from last Monday's JSON dump and applied the T132839 workarounds
  • 15:35 zeljkof: finished EU SWAT!
  • 15:33 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add en.wikinews and es.wikinews as import source in testwiki (T154879) (duration: 02m 38s)
  • 15:26 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable import from cswiki to arbcom_cswiki (T154799) (duration: 02m 38s)
  • 15:10 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add DW alias for NS_PROJECT_TALK in frwiki (T153952) (duration: 02m 36s)
  • 15:04 zeljkof: extending EU SWAT, tree more patches left to deploy
  • 15:00 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: [throttle] Lift for 2017-01-10/12 + minor cleanup (T154312) (duration: 02m 36s)
  • 14:52 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Extension:Babel s category on cswikiversity (T67211) (duration: 02m 36s)
  • 14:39 zfilipin@tin: Synchronized static/images/project-logos: SWAT: Add HD logos for multiple projects (T150618) (duration: 02m 36s)
  • 14:25 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add digitalmedia.fws.gov to the whitelist (T154671) (duration: 02m 38s)
  • 10:38 akosiaris: restart nginx and rcstream on rcs1001.eqiad.wmnet to debug issue with prematurely closed connections and 502 returned to clients. No change witnessed.
  • 07:12 ebernhardson: restart elasticsearch on relforge100[12] to adjust ltr logging settings
  • 02:45 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Jan 9 02:45:57 UTC 2017 (duration 4m 36s)
  • 02:41 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.7) (duration: 27m 03s)

2017-01-08

  • 02:27 l10nupdate@tin: ResourceLoader cache refresh completed at Sun Jan 8 02:27:29 UTC 2017 (duration 4m 40s)
  • 02:22 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.7) (duration: 07m 33s)

2017-01-07

  • 19:07 ejegg: disabled dedupe civicrm contacts
  • afk: re-enabled dedupe civicrm contacts
  • afk: disabled dedupe civicrm contacts
  • 13:31 dcausse: elastic@codfw removing/readding replicas for viwiki_general and zhwiki_content (affected by something similar to https://github.com/elastic/elasticsearch/issues/12661) - T154765
  • 11:35 _joe_: from medelevium
  • 11:35 _joe_: restarted apache/otrs, removed a 8 gb error.log
  • 05:11 Dereckson: Update statistics count on so.wikipedia (T154833)
  • 02:35 l10nupdate@tin: ResourceLoader cache refresh completed at Sat Jan 7 02:35:56 UTC 2017 (duration 5m 20s)
  • 02:30 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.7) (duration: 11m 08s)

2017-01-06

  • 23:08 godog: force puppet run on cache_upload in eqiad to switch thumbs back from codfw
  • 22:38 demon@tin: Synchronized multiversion: updateBranchPointers consolidation (duration: 00m 56s)
  • 20:14 demon@tin: Synchronized w: Dropping old entry point (duration: 00m 41s)
  • 19:35 demon@tin: Synchronized php-1.29.0-wmf.7/extensions/UploadWizard/resources: I32e0b8 (duration: 00m 40s)
  • 19:29 demon@tin: Synchronized php-1.29.0-wmf.7/extensions/UploadWizard/resources/mw.UploadWizard.js: I32e0b8 (duration: 00m 59s)
  • 19:17 ebernhardson: restarting elasticsearch on relforge100[12] to test new search-ltr plugin
  • 19:07 ostriches: gerrit: Started full reindex of all changes, should be background but will be watching
  • 18:59 mutante: gerrit restarting for config change 308753 - will be back in seconds
  • 18:13 mutante: mw1205 - restarted hhvm
  • 18:08 godog: force puppet run on eqiad cache_upload to switch thumbs to codfw
  • 16:17 godog: bounce swift-proxy on ms-fe100[123] leave ms-fe1004 for investigation
  • 16:14 hashar: Restarting Nodepool
  • 16:05 ema: wiping codfw caches T154758
  • 15:29 cmjohnson1: powering off mw1239 to reseat DIMM
  • 15:22 papaul: elastic2025-elastic2036 - signing puppet certs, salt-key, initial run
  • 14:53 mark: papaul powercycled asw-a7-codfw 14:50
  • 14:16 reedy@tin: Finished scap: Rebuild message cache for Echo api messages being missing T154110 (duration: 25m 00s)
  • 13:51 reedy@tin: Started scap: Rebuild message cache for Echo api messages being missing T154110
  • 10:00 ariel@tin: Synchronized wmf-config/throttle.php: test, noop (duration: 02m 45s)
  • 09:24 paravoid: asw-a7-codfw is down, serial console unresponsive
  • 09:12 ariel@tin: Synchronized wmf-config/throttle.php: Adjust throttle rule for Maharashtra 'Edit Wikipedia' workshop (VNGIASS) (duration: 02m 46s)
  • 07:08 moritzm: installing crypto++ security updates on trusty hosts
  • 04:37 matt_flaschen: Finished FlowFixInconsistentBoards.php (production mode) on all wikis
  • 04:27 matt_flaschen: Started FlowFixInconsistentBoards.php (production mode) on all wikis
  • 04:10 mutante: Icinga now using Letsencrypt cert and all good
  • 03:42 mutante: icinga - debugging issue with cert change
  • 03:05 papaul: OS instalaltion on elastic2025-elastic2036
  • 02:34 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.7) (duration: 13m 23s)
  • 01:27 bd808: Restarted logstash on logstash1002 (T154732)
  • 01:06 Krinkle: stream.wikimedia.org problems - nginx responds with HTTP 502 Bad Gateway to most requests
  • 01:01 ostriches: gerrit: back up from upgrade
  • 01:00 ostriches: gerrit: down for upgrade
  • 00:13 mutante: analytics1036, ms-fe1003 - ran puppet to fix Icinga
  • 00:11 mutante: carbon - stopping ganglia-monitor-aggregator for good

2017-01-05

  • 23:49 mutante: rolling out exim4 upgrades (DSA 3747-1) on all remaining eqiad (all-eqiad)
  • 23:46 godog: fix root-owned files on puppetmaster1001:/var/lib/git/operations/private/ causing /srv/private post-commit hook to fail
  • 23:18 mutante: switching eqiad ganglia aggregator - running puppet on install1001 - disabling on carbon, re-enabling puppet across eqiad
  • 23:05 mutante: temp disabling puppet on all eqiad hosts via salt - during ganglia aggregator switch
  • 22:51 demon@tin: Synchronized php-1.29.0-wmf.7/extensions/Echo/includes/api: silence api warnings (duration: 02m 46s)
  • 22:40 demon@tin: Synchronized php-1.29.0-wmf.7/extensions/VisualEditor: silence some api warnings (duration: 02m 48s)
  • 22:22 demon@tin: Synchronized php-1.29.0-wmf.7/extensions/CodeReview/api/ApiQueryCodeComments.php: silence some warnings (duration: 02m 46s)
  • 21:59 mutante: rolling out exim4 upgrades (DSA 3747-1) on all remaning ones in codfw (all-codfw)
  • 21:53 ejegg: updated payments wiki from 21ea9bc to 1f9ea80
  • 21:53 mutante: mx1001 - upgrading exim4 packages, exim4-daemon-heavy, forcing puppet run
  • 21:35 mutante: mx2001 - upgrading exim4 packages, daemon-heavey, forcing puppet run
  • 21:22 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.29.0-wmf.7
  • 20:58 mutante: mendelevium (OTRS) - upgrade exim4 packages, force puppet run
  • 20:45 mutante: iridium (phabricator) - upgrade exim4 packages, force puppet run
  • 20:29 demon@tin: Synchronized docroot/wikipedia.org: removing junk 15 stuff (duration: 04m 50s)
  • 20:18 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.29.0-wmf.7
  • 20:07 demon@tin: Synchronized docroot: Ok last docroot thing for today I promise (duration: 05m 00s)
  • 19:52 demon@tin: Synchronized docroot: Final bit of this round of docroot cleanup (duration: 05m 00s)
  • 19:44 thcipriani@tin: Finished scap: SemanticForms l10n cache rebuild for 1.29.0-wmf.7 (duration: 39m 21s)
  • 19:35 mutante: mira - upgrade exim, python-requests, linux-image
  • 19:29 mutante: rolled out exim4 upgrades (DSA 3747-1) on contint1001/2001, dbmonitor1001/2001, tungsten, seaborgium, hassium, labstore*
  • 19:05 thcipriani@tin: Started scap: SemanticForms l10n cache rebuild for 1.29.0-wmf.7
  • 18:34 mutante: fermium (lists server) - upgrading exim packages, exim4-daemon-heavy, forcing puppet run
  • 18:29 yurik@tin: Finished deploy [graphoid/deploy@d20b00e]: (no message) (duration: 03m 11s)
  • 18:26 yurik@tin: Starting deploy [graphoid/deploy@d20b00e]: (no message)
  • 18:25 arlolra: Updated Parsoid to 974dd5b3 (T143183, T102134, T113044)
  • 18:25 mutante: rolling out exim4 upgrades (DSA 3747-1) on prometheus, mwlog, pollux, labmon, lithium, hassaleh, dubnium, graphite1002, tin, serpens, bromine, dataset1001
  • 18:16 arlolra@tin: Finished deploy [parsoid/deploy@465f9c4]: Updating Parsoid to 974dd5b3 (duration: 10m 46s)
  • 18:15 mutante: rolling out exim4 upgrades (DSA 3747-1) on puppetmaster, yubiauth, oresrdb, (oresrdb1001 - Unknown installation error.. eh.. this is new)
  • 18:05 arlolra@tin: Starting deploy [parsoid/deploy@465f9c4]: Updating Parsoid to 974dd5b3
  • 17:57 robh: shutting down mw2075-2089 for decom per T154621
  • 17:35 robh: diabling puppet on mw2075-2089 to decommission them today.
  • 17:15 mutante: rolling out exim4 upgrades (DSA 3747-1) on ruthenium, einsteinium (icinga), etherpad1001, rhodium, | einsteinium: upgrade python packages, kernel | xenon: apt-get autoremove, upgrade python- arcconf, libs...
  • 17:07 mutante: scandium (zuul merger), upgrade exim, python-requests, kernel version
  • 17:06 mutante: rolling out exim4 upgrades (DSA 3747-1) on kraz, wdqs2003, wezen, zosma, tegmen, rutherfordium. upgrade kernel and python-requests on zosma
  • 16:59 mutante: rolling out exim4 upgrades (DSA 3747-1) on all-db-noncore, all-mw-eqiad, restbase-eqiad, kafka-main
  • 16:52 mutante: rolling out exim4 upgrades (DSA 3747-1) on notebook, lvs-canary, lvs, mw-maintenance, all-mw-codfw
  • 16:33 mutante: cobalt upgrading exim packages
  • 16:28 mutante: rolling out exim4 upgrades (DSA 3747-1) on mw-api, swift-fe, swift-be, sca, scb, misc-analytics
  • 16:21 mutante: rolling out exim4 upgrades (DSA 3747-1) on db-core-eqiad, db-misc-servers, videoscaler
  • 15:59 moritzm: upgrading firejail on thumbor servers
  • 15:44 chasemp: labstore1005 systemctl disable create-dbusers
  • 15:01 mobrovac: restbase restarting for firejail upgrade
  • 14:46 moritzm: upgrading firejail on image scalers
  • 14:19 moritzm: upgrading firejail on restbase production hosts
  • 14:15 hashar@tin: Synchronized php-1.29.0-wmf.7/extensions/ContentTranslation: Workaround to fix restoration for truncated section ids - T154279 (duration: 02m 10s)
  • 14:10 moritzm: upgrading firejail on restbase staging hosts
  • 13:46 moritzm: installing audiofile security updates
  • 13:09 mobrovac@tin: Finished deploy [trending-edits/deploy@c5d239b]: Restart for firejail upgrade (duration: 00m 46s)
  • 13:08 mobrovac@tin: Starting deploy [trending-edits/deploy@c5d239b]: Restart for firejail upgrade
  • 13:07 mobrovac@tin: Finished deploy [electron-render/deploy@b2a820e]: Restart for firejail upgrade (duration: 05m 29s)
  • 13:02 mobrovac@tin: Starting deploy [electron-render/deploy@b2a820e]: Restart for firejail upgrade
  • 13:01 mobrovac@tin: Finished deploy [electron-render/deploy@b2a820e]: Restart for firejail upgrade (duration: 03m 40s)
  • 12:57 mobrovac@tin: Starting deploy [electron-render/deploy@b2a820e]: Restart for firejail upgrade
  • 12:45 mobrovac@tin: Finished deploy [mobileapps/deploy@c39bd1f]: Restart for firejail upgrade (duration: 04m 05s)
  • 12:41 mobrovac@tin: Starting deploy [mobileapps/deploy@c39bd1f]: Restart for firejail upgrade
  • 12:41 mobrovac@tin: Finished deploy [mathoid/deploy@79fdd56]: Restart for firejail upgrade (duration: 00m 41s)
  • 12:40 mobrovac@tin: Starting deploy [mathoid/deploy@79fdd56]: Restart for firejail upgrade
  • 12:40 mobrovac@tin: Finished deploy [graphoid/deploy@151f26c]: Restart for firejail upgrade (duration: 00m 43s)
  • 12:39 mobrovac@tin: Starting deploy [graphoid/deploy@151f26c]: Restart for firejail upgrade
  • 12:38 mobrovac@tin: Finished deploy [cxserver/deploy@0279029]: Restart for firejail upgrade (duration: 00m 39s)
  • 12:37 mobrovac@tin: Starting deploy [cxserver/deploy@0279029]: Restart for firejail upgrade
  • 12:36 mobrovac@tin: Finished deploy [citoid/deploy@da96f4b]: (no message) (duration: 01m 06s)
  • 12:35 mobrovac@tin: Starting deploy [citoid/deploy@da96f4b]: (no message)
  • 12:24 moritzm: installing firejail security updates on scb
  • 11:13 akosiaris: rebooting bast3001, T154603
  • 11:11 moritzm: uploaded firejail 0.9.44+wmf2 for jessie-wikimedia to carbon
  • 07:54 elukey: chown www-data:www-data all the root:adm hhvm log files on mw eqiad hosts (T132324)
  • 07:11 marostegui: Compressing revision tables across all the wikis - db1038 - T154465
  • 07:09 marostegui: Compressing pagelinks tables across all the wikis - db1044 - T153826
  • 07:08 marostegui: Compressing revision tables across all the wikis - db1015 - T153739
  • 06:15 bd808: sudo -u l10nupdate rm /var/lock/scap on tin to clean up lock left by bad l10nupdate locking attempt
  • 05:49 mutante: rolling out exim4 upgrades (DSA 3747-1) on swift-fe-codfw, swift-be-codfw, ALL remaining mw
  • 05:44 mutante: rolling out exim4 upgrades (DSA 3747-1) on db-core-codfw, etcd, graphite, kafka-analytics-canary, kafka-analytics, logstash
  • 05:39 mutante: rolling out exim4 upgrades (DSA 3747-1) on prometheus, aqs, db-es
  • 05:35 mutante: rolling out exim4 upgrades (DSA 3747-1) on cp-eqiad, memcached-eqiad
  • 01:48 mutante: rolling out exim4 upgrades (DSA 3747-1) on redis-codfw (rdb2005 needed manual) and all of dc-ulsfo, dc-esams
  • 01:41 mutante: rolling out exim4 upgrades (DSA 3747-1) on ganeti, cp-ulsfo, wdqs, thumbor, db-es-codfw
  • 01:36 mutante: rolling out exim4 upgrades (DSA 3747-1) on parsoid, maps, cp-esams
  • 01:32 mutante: servermon - after the next update by cron - package data is back
  • 01:13 mattflaschen@tin: Synchronized wmf-config/InitialiseSettings.php: Disable NewUserMessage on gomwiki (duration: 00m 41s)
  • 01:10 robh@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw2090.codfw.wmnet
  • 01:06 demon@tin: Synchronized scap/plugins: (no message) (duration: 00m 40s)
  • 01:04 mattflaschen@tin: Synchronized php-1.29.0-wmf.7/extensions/Flow: Flow script to add more troubleshooting information to a maintenance script (duration: 00m 56s)
  • 00:57 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2090.codfw.wmnet
  • 00:57 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2089.codfw.wmnet
  • 00:57 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2088.codfw.wmnet
  • 00:57 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2087.codfw.wmnet
  • 00:57 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2086.codfw.wmnet
  • 00:56 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2085.codfw.wmnet
  • 00:55 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2084.codfw.wmnet
  • 00:55 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2083.codfw.wmnet
  • 00:55 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2082.codfw.wmnet
  • 00:55 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2081.codfw.wmnet
  • 00:54 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2080.codfw.wmnet
  • 00:54 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2080.codfw.wmnet
  • 00:52 mattflaschen@tin: Synchronized php-1.29.0-wmf.6/extensions/Flow: Two Flow fixes related to production database/content inconsistencies. (duration: 00m 59s)
  • 00:42 mutante: phab2001 - same chmod 751 on exim4 dirs that i manually did on krypton is done by puppet here, fully automatic, not sure why krypton was a one-off
  • 00:41 mutante: planet1001/2001, phab2001 - upgrade exim4, exim4-daemon-heavy
  • 00:37 mutante: servermon - weird behaviour in the "pending package upgrades" list? exim4 package was shown as pending on lots of hosts, after next upgrade it disppears from list, even though about half the servers should still be listed
  • 00:23 mutante: krypton - chmod 751 /var/spool/exim4/ to fix Icinga alerts about unaccesible tmpfs (nagios user could not access), it was 751 on other hosts like ununpentium
  • 00:17 mutante: krypton - stop exim, umount orphaned "scan" tmpfs (there is no clamav here)
  • 00:00 mutante: gerrit slowdown reported around 23:55 UTC, was back to normal after 2 minutes (T148478) - attaching latest jvm_gc log
  • 00:00 aaron@tin: Synchronized wmf-config/logging.php: No-op sync of 7e103f2 (duration: 00m 42s)

2017-01-04

  • 23:55 mutante: krypton - chown Debian-exim:Debian-exim /var/spool/exim4/scan/ to fix Icinga-reported DISK issue - wrong permissions - see puppet/modules/exim4/manifests/init.pp line 57 ff "catch-22 with Puppet vs. package"
  • 23:20 mutante: rolled out exim4 upgrades (DSA 3747-1) on memcached-canary, memcached-codfw, restbase-codfw, cp-codfw
  • 23:11 aaron@tin: Synchronized wmf-config/logging.php: Include DB shard as a logstash column (duration: 00m 41s)
  • 23:09 mutante: rolling out exim4 upgrades (DSA 3747-1) on memcached-canary, memcached-codfw, restbase-codfw
  • 23:06 mutante: rolling out exim4 upgrades (DSA 3747-1) on mw-eqiad
  • 22:52 robh: all my server depools and decoms for the mw range are on T154621
  • 22:51 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2078.codfw.wmnet
  • 22:51 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2077.codfw.wmnet
  • 22:51 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2076.codfw.wmnet
  • 22:50 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2075.codfw.wmnet
  • 22:49 godog: rename / reimage restbase-test1* to restbase-dev1*
  • 22:46 bblack: TLS: unified certificates in esams switching to digicert
  • 22:19 mutante: rolling out exim4 upgrades (DSA 3747-1) on mw-codfw
  • 21:55 Krinkle: mwscript deleteEqualMessages.php --wiki nowikinews (T45917)
  • 21:55 Krinkle: mwscript deleteEqualMessages.php --wiki nowiki (T45917)
  • 21:48 otto@tin: Finished deploy [eventstreams/deploy@a103be2]: (no message) (duration: 01m 32s)
  • 21:46 otto@tin: Starting deploy [eventstreams/deploy@a103be2]: (no message)
  • 21:24 bsitzmann@tin: Finished deploy [mobileapps/deploy@c39bd1f]: Update mobileapps to b43c5d6 (duration: 02m 55s)
  • 21:21 bsitzmann@tin: Starting deploy [mobileapps/deploy@c39bd1f]: Update mobileapps to b43c5d6
  • 21:15 smalyshev@tin: Finished deploy [wdqs/wdqs@3762556]: (no message) (duration: 02m 40s)
  • 21:13 smalyshev@tin: Starting deploy [wdqs/wdqs@3762556]: (no message)
  • 21:01 smalyshev@tin: Finished deploy [wdqs/wdqs@3762556]: (no message) (duration: 00m 59s)
  • 21:00 smalyshev@tin: Starting deploy [wdqs/wdqs@3762556]: (no message)
  • 20:49 madhuvishy: adding temporary IP tables rule on labservices1001 to drop traffic from toolchecker for tests (T152369)
  • 20:39 mutante: rolling out exim4 upgrades (DSA 3747-1) on mw-canary hosts
  • 20:36 mutante: rolling out exim4 upgrades (DSA 3747-1) on stat* and kubernetes hosts
  • 20:05 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.29.0-wmf.7
  • 19:09 thcipriani@tin: Synchronized portals: SWAT: Bumping portal to master T128546 (duration: 00m 42s)
  • 19:08 thcipriani@tin: Synchronized portals/prod/wikipedia.org/assets: SWAT: Bumping portal to master T128546 (duration: 00m 43s)
  • 18:30 mutante: rolling out exim4 upgrades (DSA 3747-1) on misc servers
  • 18:29 ejegg: enabled payment processor audit parser jobs
  • 17:47 matt_flaschen: Ran manual DB update to officewiki for T153320.
  • 15:20 zeljkof: EU SWAT finished
  • 15:19 hashar@tin: Synchronized wmf-config/throttle.php: (no message) (duration: 00m 41s)
  • 15:03 zeljkof: extending eu swat until 330392 is merged
  • 14:56 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: Throttle rules for 2017-01-06/07, tewiki (T154568) (duration: 00m 40s)
  • 14:32 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set valid content language for Norwegian wikis (T126146) (duration: 00m 41s)
  • 14:20 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert $wgMFEEditorOptions[anonymousEditing] = false for kowiki (T119823) (duration: 00m 41s)
  • 11:18 jynus: continuing maintenance on db1035 (mysql replication stopped)
  • 10:12 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2060 - T154031 (duration: 00m 47s)
  • 07:43 marostegui: Compressing tables on db1015 - T153739
  • 07:24 marostegui: Compressing more tables on db1044 - T153826
  • 03:10 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Jan 4 03:10:08 UTC 2017 (duration 5m 33s)
  • 03:05 bd808@tin: Synchronized wmf-config/throttle.php: Add throttle rules for January 2017 events in Maharashtra (T154312) (duration: 00m 42s)
  • 03:04 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.7) (duration: 13m 05s)
  • 02:34 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.6) (duration: 11m 17s)
  • 01:24 maxsem@tin: Synchronized php-1.29.0-wmf.6/extensions/CentralAuth: https://gerrit.wikimedia.org/r/#/c/330345/ (duration: 00m 44s)
  • 01:03 maxsem@tin: Synchronized php-1.29.0-wmf.7/extensions/Flow: https://gerrit.wikimedia.org/r/#/c/330338/ (duration: 00m 58s)
  • 00:42 foks: removed 2fa for account per T154171
  • 00:33 maxsem@tin: Synchronized wmf-config/unitConversionConfig.json: https://gerrit.wikimedia.org/r/#/c/327907/5 (duration: 00m 40s)
  • 00:13 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/330264/3 (duration: 00m 41s)
  • 00:06 eileen: update civicrm from f78c894 to b26844d

2017-01-03

  • 23:57 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.29.0-wmf.7
  • 23:48 thcipriani@tin: Finished scap: testwiki to php-1.29.0-wmf.7 and rebuild l10n cache (duration: 49m 50s)
  • 23:19 ostriches: gerrit: quick restart of services
  • 22:58 thcipriani@tin: Started scap: testwiki to php-1.29.0-wmf.7 and rebuild l10n cache
  • 22:55 chasemp: iptables block of tools-checker-01 to debug DNS SPoF
  • 22:44 maxsem@tin: Synchronized php-1.29.0-wmf.7/extensions/Kartographer: https://gerrit.wikimedia.org/r/#/c/330321/ (duration: 00m 42s)
  • 22:38 maxsem@tin: Synchronized php-1.29.0-wmf.6/extensions/Kartographer: https://gerrit.wikimedia.org/r/#/c/330322/ (duration: 00m 42s)
  • 22:11 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: mapframe on fr: and fi: https://gerrit.wikimedia.org/r/#/c/330311/3 (duration: 00m 41s)
  • 21:00 gehel@tin: Finished deploy [wdqs/wdqs@a25d3aa]: (no message) (duration: 00m 55s)
  • 20:59 gehel@tin: Starting deploy [wdqs/wdqs@a25d3aa]: (no message)
  • 20:57 otto@tin: Finished deploy [eventstreams/deploy@9095b4e]: (no message) (duration: 11m 15s)
  • 20:45 otto@tin: Starting deploy [eventstreams/deploy@9095b4e]: (no message)
  • 20:18 gehel@tin: Finished deploy [wdqs/wdqs@cd7215c]: (no message) (duration: 04m 54s)
  • 20:13 gehel@tin: Starting deploy [wdqs/wdqs@cd7215c]: (no message)
  • 19:56 demon@tin: Synchronized multiversion/updateBranchPointers: Removing unused --dry-run option (duration: 00m 40s)
  • 19:52 thcipriani@tin: Synchronized multiversion: SWAT: Remove checkoutMediaWiki (duration: 00m 58s)
  • 19:13 thcipriani@tin: Synchronized wmf-config/Wikibase-production.php: SWAT: Add badge for "digitaldocument" in Wikibase T153186 (duration: 01m 33s)
  • 18:01 moritzm: installing fontconfig security updates
  • 17:44 mutante: terbium - Notice: /Stage[main]/Mediawiki::Maintenance::Generatecaptcha/Cron[generatecaptcha]/ensure: created(T150029)
  • 17:24 thcipriani: starting branch cut for 1.29.0-wmf.7
  • 17:04 mutante: iridium (phab) - apt-get clean ; find /var/log/account/ -mtime +10 -delete ; find /var/log/atop/ -mtime +10 -delete (T154407)
  • 16:59 thcipriani: enable l10nupdate cron post deployment-freeze
  • 16:49 mutante: iridium (phab) - reduce process accounting from 30 days to 10 days to save disk space used by /var/log/account, run /etc/cron.daily/acct (T154407)
  • 16:37 bd808: Updated scholarships to 1690808 on krypton; needed help from _joe_ to make trebuchet work
  • 15:58 _joe_: rolling restart of restbase on the production cluster
  • 15:49 _joe_: rolling restart of restbase on the test cluster
  • 14:53 akosiaris: reenabling ntpd on the restbase in eqiad
  • 14:43 gehel: upgrade liblogstash-gelf on elastic* - T150408
  • 14:35 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: [2/2] Add HD logos for multiple wikis - T150618 (duration: 00m 40s)
  • 14:33 hashar@tin: Synchronized static/images/project-logos: [1/2] Add HD logos for multiple wikis - T150618 (duration: 00m 40s)
  • 14:29 akosiaris: reenabling ntpd on the rest of the boxes. Leaving restbase only out for last
  • 14:21 hashar@tin: Synchronized wmf-config/throttle.php: New rules + remove obsolete rules T154245 (duration: 00m 40s)
  • 14:19 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable mapframe for nowiki T154021 (duration: 00m 39s)
  • 14:17 akosiaris: reenabling ntpd on aqs boxes
  • 14:17 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Set sortPrepend for gdwiki T153900 (duration: 00m 40s)
  • 14:16 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable SandboxLink on ruwiki T153855 (duration: 00m 40s)
  • 14:10 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable subpages in NS0 for arbcom_cswiki - T154247 (duration: 00m 40s)
  • 14:08 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Add new page protection level on etwiki - T153465 (duration: 00m 53s)
  • 14:08 akosiaris: reenabling ntpd on maps, maps-test boxes
  • 14:07 akosiaris: reenabling ntpd on kafka boxes
  • 13:37 akosiaris: reenabling ntpd on elastic boxes
  • 13:27 akosiaris: reenabling ntpd on rdb boxes
  • 13:22 akosiaris: reenabling ntpd on conf boxes
  • 13:22 akosiaris: reenabling ntpd on es* boxes
  • 13:15 akosiaris: reenabling ntpd on scb* boxes
  • 13:09 moritzm: uploaded Linux 4.4.39 for jessie-wikimedia to carbon
  • 13:09 akosiaris: reenabling ntpd on mc* boxes
  • 13:01 akosiaris: reenabling ntpd on ms-fe boxes
  • 13:00 moritzm: installing libgd security updates
  • 12:55 akosiaris: reenabling ntpd on ms-be boxes
  • 12:52 akosiaris: reenabling ntpd on lvs boxes
  • 12:48 akosiaris: reenabling ntpd on analytics boxes
  • 12:41 moritzm: installing python security updates
  • 12:12 moritzm: installing squid security updates
  • 12:07 akosiaris: reenabling ntpd on pc eqiad & codfw boxes
  • 12:06 akosiaris: reenabling ntpd on ganeti eqiad & codfw boxes
  • 11:54 akosiaris: reenabling ntpd on wtp eqiad boxes
  • 11:52 akosiaris: reenabling ntpd on logstash eqiad boxes
  • 11:51 akosiaris: reenabling ntpd on db* eqiad boxes
  • 11:46 akosiaris: reenabling ntpd on cobalt (gerrit)
  • 11:32 moritzm: installing tar security updates on trusty hosts
  • 11:27 gehel: upgrade liblogstash-gelf on deployment-elastic* - T150408
  • 11:16 gehel: upgrade lilogstash-gelf on relforge - T150408
  • 11:13 akosiaris: reenabling ntpd on db* codfw boxes
  • 11:07 akosiaris: reenabling ntpd on wtp codfw boxes
  • 10:59 akosiaris: reenabling ntpd on mw eqiad boxes
  • 10:53 jynus: stopping mysql replication on db1035 (depooled)
  • 10:50 akosiaris: reenabling ntpd on mw codfw boxes
  • 10:44 akosiaris: reenabling ntpd on eqiad cp boxes
  • 10:39 akosiaris: reenabling ntpd on codfw cp boxes
  • 10:14 akosiaris: start enabling ntpd again across the fleet. Starting with cp boxes on ulsfo and esams
  • 09:23 marostegui: stop MySQL dbstore2002 for maintenance - T151552
  • 09:10 marostegui: stop MySQL dbstore2001 for maintenance - T151552
  • 08:21 marostegui: Run optimize table on db1038 on all the revision,templatelinks and pagelinks tables - T154465
  • 08:00 marostegui: Run optimize table on a few large tables - db1015 - T153739
  • 07:58 elukey: chown www-data:www-data all the root:adm hhvm log files on mw codfw hosts (T132324)
  • 07:54 marostegui: Run optimize table on a few large tables - db1044 - T153826
  • 07:30 marostegui: Stop mysql db2048 and db2034 for maintenance - https://phabricator.wikimedia.org/T149553

2017-01-02

  • 23:09 hoo: Removed 2fa from an account, per T154450
  • 17:20 ema: iridium: removed /var/log/account/pacct.2[0-9].gz to free up more disk space
  • 16:05 ema: removing old kernels and kernel headers from iridium to free up some disk space
  • 13:24 elukey: powercycled mw1280, not pingable and mgmt console frozen

2017-01-01

  • 02:23 chasemp: labservices1001 'racadm serveraction hardreset'
  • 02:23 godog: reboot labservices1001, unresponsive on console and MCE/temperature alerts found on lithium
  • 00:56 filippo@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1286.eqiad.wmnet,service=apache2
  • 00:55 bd808: Restarted logstash on logstash1001 (T154388)
  • 00:46 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw1286.eqiad.wmnet
  • 00:27 godog: dump core file and restart varnish-frontend on cp2026

2016-12-31

  • 14:32 akosiaris: reenable puppet and ircecho on einsteinium. NTP disabling has propagated throughout the fleet
  • 14:08 akosiaris: disable ircecho for a while to avoid NTP alert storm

2016-12-29

  • 23:19 akosiaris: schedule downtime for ferm checks on kubernetes nodes. Some race between kubernetes + ferm, investigating
  • 21:19 otto@tin: Finished deploy [eventstreams/deploy@4098bb4]: (no message) (duration: 01m 59s)
  • 21:17 otto@tin: Starting deploy [eventstreams/deploy@4098bb4]: (no message)
  • 16:05 _joe_: restarted HHVM on mw1279, stuck in HPHP::Treadmill::getAgeOldestRequest
  • 15:56 akosiaris: merging https://gerrit.wikimedia.org/r/329597 for T154278 (IP throttle raise)
  • 15:55 akosiaris@tin: Synchronized wmf-config/throttle.php: (no message) (duration: 00m 42s)
  • 12:20 akosiaris: running sudo apt-get autoremove on labtestnet2001. Removing various older kernels
  • 12:19 akosiaris: running sudo apt-get autoremove on labtestnet2001
  • 05:22 cwd: updated civicrm from 038e166 to f78c894
  • 01:10 cwd: updated payments-wiki from 5b4ced9 to 21ea9bc
  • 00:18 cwd: updated smash-pig from c3eaadb to 3da597f

2016-12-28

  • 23:13 cwd: rolled back smashpig
  • 22:57 cwd: updated smashpig from c3eaadb to 8597bd4
  • 14:12 mobrovac@tin: Finished deploy [trending-edits/deploy@c5d239b]: Add depooling and repooling to Trending Edits T144602 (duration: 01m 21s)
  • 14:11 mobrovac@tin: Starting deploy [trending-edits/deploy@c5d239b]: Add depooling and repooling to Trending Edits T144602
  • 14:07 mobrovac@tin: Finished deploy [mobileapps/deploy@ae22656]: Add depooling and repooling to MCS T144602 (duration: 02m 34s)
  • 14:05 mobrovac@tin: Starting deploy [mobileapps/deploy@ae22656]: Add depooling and repooling to MCS T144602
  • 14:00 mobrovac@tin: Finished deploy [electron-render/deploy@b2a820e]: Add depooling and repooling to PDFRender T144602 (duration: 01m 26s)
  • 13:59 mobrovac@tin: Starting deploy [electron-render/deploy@b2a820e]: Add depooling and repooling to PDFRender T144602
  • 13:51 mobrovac@tin: Finished deploy [graphoid/deploy@151f26c]: Add depooling and repooling to Graphoid T144602 (duration: 01m 37s)
  • 13:49 mobrovac@tin: Starting deploy [graphoid/deploy@151f26c]: Add depooling and repooling to Graphoid T144602
  • 13:44 mobrovac@tin: Finished deploy [cxserver/deploy@0279029]: Add depooling and repooling to CXServer T144602 (duration: 03m 22s)
  • 13:41 mobrovac@tin: Starting deploy [cxserver/deploy@0279029]: Add depooling and repooling to CXServer T144602
  • 13:41 mobrovac@tin: Finished deploy [cxserver/deploy@0279029]: (no message) (duration: 00m 25s)
  • 13:40 mobrovac@tin: Starting deploy [cxserver/deploy@0279029]: (no message)
  • 13:33 mobrovac@tin: Finished deploy [citoid/deploy@da96f4b]: Add depooling and repooling to Citoid T144602 (duration: 01m 59s)
  • 13:31 mobrovac@tin: Starting deploy [citoid/deploy@da96f4b]: Add depooling and repooling to Citoid T144602
  • 13:21 mobrovac@tin: Finished deploy [mathoid/deploy@79fdd56]: Add depooling and repooling to Mathoid T144602 (duration: 02m 31s)
  • 13:19 mobrovac@tin: Starting deploy [mathoid/deploy@79fdd56]: Add depooling and repooling to Mathoid T144602
  • 07:36 legoktm: added reedy to "integration" gerrit group per T154207
  • 04:15 awight: disable Fundraising listeners via extraordinary workaround: from f143378 to 6b86491
  • 03:52 awight: disabled Fundraising audit parsers
  • 03:43 awight: put payments frontends into maintenance mode
  • 02:47 awight: reenabling Fundraisin thank-you job
  • 02:46 awight: update Fundraising CiviCRM from 454679d to 038e166
  • 01:34 awight: disabled thank-you mailer per T154209

2016-12-27

  • 21:32 awight: reenabled WMF Fundraising low-level campaigns
  • 21:16 awight: reenabled all Fundraising jobs except for dedupe
  • 16:23 _joe_: power down prometheus2003
  • 05:44 Amir1: ladsgroup@terbium:~$ mwscript extensions/ORES/maintenance/PopulateDatabase.php --wiki=wikidatawiki (T154168)
  • 05:25 Amir1: finished deploy of ores:228b9b4 in all nodes (T154168)
  • 05:25 ladsgroup@tin: Finished deploy [ores/deploy@228b9b4]: (no message) (duration: 18m 46s)
  • 05:14 Amir1: starting deploy of ores:228b9b4 in all nodes (T154168)
  • 05:06 ladsgroup@tin: Starting deploy [ores/deploy@228b9b4]: (no message)
  • 05:06 Amir1: starting deploy of ores:228b9b4 in canary nodes (T154168)

2016-12-26

  • 21:16 ottomata: disabled active checks of stat1001 services T149438
  • 20:38 otto@tin: Finished deploy [eventstreams/deploy@590ea96]: (no message) (duration: 00m 23s)
  • 20:38 otto@tin: Starting deploy [eventstreams/deploy@590ea96]: (no message)
  • 20:34 otto@tin: Finished deploy [eventstreams/deploy@590ea96]: (no message) (duration: 01m 22s)
  • 20:33 otto@tin: Starting deploy [eventstreams/deploy@590ea96]: (no message)
  • 20:32 otto@tin: Finished deploy [eventstreams/deploy@590ea96]: (no message) (duration: 00m 04s)
  • 20:32 otto@tin: Starting deploy [eventstreams/deploy@590ea96]: (no message)
  • 20:31 otto@tin: Finished deploy [eventstreams/deploy@590ea96]: (no message) (duration: 00m 04s)
  • 20:31 otto@tin: Starting deploy [eventstreams/deploy@590ea96]: (no message)
  • 20:31 otto@tin: Finished deploy [eventstreams/deploy@590ea96]: (no message) (duration: 00m 40s)
  • 20:30 otto@tin: Starting deploy [eventstreams/deploy@590ea96]: (no message)
  • 20:30 otto@tin: Starting deploy [eventstreams/deploy@590ea96]: (no message)
  • 20:30 otto@tin: Finished deploy [eventstreams/deploy@590ea96]: (no message) (duration: 02m 45s)
  • 20:27 otto@tin: Starting deploy [eventstreams/deploy@590ea96]: (no message)
  • 20:27 otto@tin: Finished deploy [eventstreams/deploy@590ea96]: (no message) (duration: 00m 12s)
  • 20:27 otto@tin: Starting deploy [eventstreams/deploy@590ea96]: (no message)
  • 20:24 otto@tin: Finished deploy [eventstreams/deploy@648613a]: (no message) (duration: 00m 15s)
  • 20:24 otto@tin: Starting deploy [eventstreams/deploy@648613a]: (no message)
  • 19:38 otto@tin: Finished deploy [eventstreams/deploy@648613a]: (no message) (duration: 00m 34s)
  • 19:38 otto@tin: Starting deploy [eventstreams/deploy@648613a]: (no message)
  • 19:35 otto@tin: Starting deploy [eventstreams/deploy@ed2e39c]: (no message)
  • 19:32 otto@tin: Starting deploy [eventstreams/deploy@90934c3]: (no message)
  • 19:14 otto@tin: Finished deploy [eventstreams/deploy@90934c3]: (no message) (duration: 00m 31s)
  • 19:13 otto@tin: Starting deploy [eventstreams/deploy@90934c3]: (no message)
  • 19:13 otto@tin: Finished deploy [eventstreams/deploy@581a5a1]: (no message) (duration: 00m 22s)
  • 19:13 otto@tin: Starting deploy [eventstreams/deploy@581a5a1]: (no message)
  • 19:10 otto@tin: Finished deploy [eventstreams/deploy@836b441]: (no message) (duration: 01m 16s)
  • 19:09 otto@tin: Starting deploy [eventstreams/deploy@836b441]: (no message)
  • 19:08 otto@tin: Finished deploy [eventstreams/deploy@e771863]: (no message) (duration: 01m 05s)
  • 19:07 otto@tin: Starting deploy [eventstreams/deploy@e771863]: (no message)
  • 19:07 otto@tin: Finished deploy [eventstreams/deploy@e771863]: log (duration: 00m 03s)
  • 19:07 otto@tin: Starting deploy [eventstreams/deploy@e771863]: log
  • 19:07 otto@tin: Finished deploy [eventstreams/deploy@e771863]: (no message) (duration: 00m 04s)
  • 19:07 otto@tin: Starting deploy [eventstreams/deploy@e771863]: (no message)
  • 19:07 otto@tin: Finished deploy [eventstreams/deploy@e771863]: (no message) (duration: 00m 10s)
  • 19:06 otto@tin: Starting deploy [eventstreams/deploy@e771863]: (no message)

2016-12-25

  • 08:30 jynus: restarting dbstore1002 mariadb, alive but unresponsive to queries do to long running queries saturating the service

2016-12-23

  • 21:29 mutante: restarted apache on mw canary servers
  • 19:55 mutante: cobalt - restarted apache
  • 19:46 mutante: install libxml2 upgrade on bastions and mw-canary
  • 19:38 mutante: install libxml2 upgrade on "misc-others"
  • 16:32 robh: rebooted db2060 due to some kind of hw raid or xfs failure, wouldnt take input
  • 15:13 jynus: testing labsdb-analytics automatic failover
  • 09:23 jynus: stopping slave dbstore2001(s2) and db2034 for more table cloning
  • 07:56 moritzm: installing Python security updates

2016-12-22

  • 23:16 ejegg: updated fundraising dashboard from c2cccc0 to bec0077
  • 22:46 ejegg: updated fundraising dashboard from ea4305b to c2cccc0
  • 22:11 thcipriani: disable l10nupdate cron for deployment freeze
  • 21:25 ebernhardson: restarting elasticsearch (again) on relforge100[12] to test ltr plugin
  • 21:10 RoanKattouw: Got lots of errors like 20:49:56 Unable to find remote tracking branch/tag for /srv/mediawiki-staging/php-1.29.0-wmf.6/extensions/ParserFunctions during this scap
  • 21:03 catrope@tin: Finished scap: Sync Idf4618977f172 in the OAuth extension- (duration: 24m 16s)
  • 20:39 catrope@tin: Started scap: Sync Idf4618977f172 in the OAuth extension-
  • 19:59 ebernhardson: restarting elasticsearch (again) on relforge100[12] to test ltr plugin
  • 19:22 gehel: restart wdqs-blazegraph and wdqs-updater on wdqs1001.eqiad.wmnet (suspicious load)
  • 19:18 ebernhardson: restarting elasticsearch on relforge100[12] to test ltr plugin
  • 19:18 jynus: stopping replication on dbstore2001(s2) and db2035 for enwiktionary.templatelinks reimport
  • 19:04 godog: roll restart swift proxy on ms-fe1* to drain thumbor traffic
  • 15:39 jynus: restart dbstore2001 to change buffer pool size, testing gerrit:328671
  • 14:51 elukey: restarting the yarn node manager java daemons on all the Hadoop worker nodes due to suspect memory leak
  • 14:14 elukey: the previous entry is missing: "on analytics1032"
  • 14:13 elukey: manually starting the yarn nodemanager after OOM
  • 13:41 jynus: stopping db1035 (depooled) replication to perform maintenance to avoid disk alerts in the next 2 weeks
  • 10:02 moritzm: installing c-ares security updates on trusty systems (jessie already fixed for quite a while)
  • 10:02 moritzm: installing c-ares security updates
  • 09:02 moritzm: installing tomcat security updates
  • 08:45 moritzm: installing libav security updates on trusty systems
  • 08:18 moritzm: installing Django security updates
  • 07:26 elukey: created /var/log/squid3/access.log.1.gz on aluminum to fix cronspam - T132324
  • 02:26 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Dec 22 02:26:23 UTC 2016 (duration 4m 49s)
  • 02:21 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.6) (duration: 07m 54s)

2016-12-21

  • 23:48 mutante: europium - jessie reinstall done - powered down until until reclaim (T153918)
  • 23:31 mutante: europium - re-installing with jessie (T82239)
  • 19:15 mutante: public1-b-eqiad and public1-c-eqiad are configured to use install1001 as DHCP, all others still use carbon as DHCP | all subnets now use install1001 as TFTP
  • 19:13 mutante: carbon - re-enabled puppet and DHCP
  • 18:13 mutante: carbon - temp stopping dhcp server
  • 15:22 gehel: truncating /var/log/elasticsearch/relforge-eqiad_feature.log on relforge100[12]
  • 15:04 elukey: removed mongodb* packages from stat1003 after https://gerrit.wikimedia.org/r/328519
  • 14:54 moritzm: installing ghostscript security updates on trusty hosts
  • 14:29 moritzm: installing imagemagick security updates
  • 13:09 moritzm: install hdf5 security updates
  • 13:03 mobrovac@tin: Finished deploy [parsoid/deploy@dab1f27]: Bug fix for mwApiServer T153797 (duration: 05m 32s)
  • 12:57 mobrovac@tin: Starting deploy [parsoid/deploy@dab1f27]: Bug fix for mwApiServer T153797
  • 12:46 moritzm: install openjdk-6 security update on labsdb1006
  • 10:43 jynus: dropping non-wiki databases from labsdb1001
  • 10:01 moritzm: installing libgme security updates
  • 09:58 jynus: extending db1035 /srv partition
  • 08:42 elukey: restarted hhvm/jobrunner (and killed ffmpeg processes) on mw116[89]
  • 08:01 marostegui: Running optimize table on db1044 for the pagelinks tables as we urgently need some space back on that host - T153826
  • 07:20 marostegui: Running optimize table on db1015 for the revision tables as we urgently need some space back on that host - https://phabricator.wikimedia.org/T153739
  • 04:36 Niharika: commtech Added samwilson as project admin
  • 03:03 dzahn@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1169.eqiad.wmnet
  • 02:51 mutante: relforge1001 has huge /var/log/elastichsearch/relforge-eqiad_feature.log that wrote GBs just today but then stopped
  • 02:23 mutante: mw1169 - reinstall done - sign new puppet cert, initial run...
  • 02:20 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.6) (duration: 07m 40s)
  • 02:12 mutante: mw1169 - delete salt key, revoke puppet cert
  • 02:06 mutante: reinstalling mw1169 (carbon DHCP, install1001 TFTP)
  • 02:02 mutante: re-enabling DHCP and puppet
  • 01:49 mutante: carbon - temp stop DHCP service to test install from install1001
  • 01:47 dzahn@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw1169.eqiad.wmnet
  • 01:47 mutante: mw1169 - schedule 2 hours downtime - boot for reinstall shortly
  • 01:11 dzahn@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1168.eqiad.wmnet
  • 01:09 mutante: mw1168 - remove old salt key, accept new salt key, start minion

2016-12-20

  • 23:46 mutante: mw1168 - re-signing new puppet cert after install, initial puppet run
  • 23:18 mutante: rebooting mw1168 into PXE for reinstall
  • 23:09 mutante: install1001/2001 - upgrade python-requests python-urllib3 arcconf
  • 21:48 mutante: depooled mw1168 videoscaler for reinstall (part of T153488)
  • 21:47 dzahn@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw1168.eqiad.wmnet
  • 21:43 demon@tin: Synchronized scap/plugins: no-op scap stuff (duration: 00m 39s)
  • 21:42 demon@tin: Synchronized multiversion/submodules.json: no-op scap stuff (duration: 00m 39s)
  • 21:08 eileen1: disable Name match only job as we are caught up now
  • 21:05 eileen1: disable Major gifts dedupe jobs. The main job has caught up to the end of the queue now so we don't need to target these separately
  • 20:55 mobrovac@tin: Finished deploy [changeprop/deploy@77786a5]: Config change: respect the pages blacklisted by RESTBase (duration: 00m 50s)
  • 20:54 mobrovac@tin: Starting deploy [changeprop/deploy@77786a5]: Config change: respect the pages blacklisted by RESTBase
  • 20:25 demon@tin: Synchronized wmf-config/InitialiseSettings.php: Remove dumb comment (duration: 00m 39s)
  • 20:21 demon@tin: Synchronized scap/plugins: no-op scap stuff (duration: 00m 39s)
  • 20:21 demon@tin: Synchronized tox.ini: no-op scap stuff (duration: 00m 39s)
  • 20:20 demon@tin: Synchronized test-requirements.txt: no-op scap stuff (duration: 00m 39s)
  • 20:19 demon@tin: Synchronized setup.py: no-op scap stuff (duration: 00m 39s)
  • 20:19 demon@tin: Synchronized requirements.txt: no-op scap stuff (duration: 00m 39s)
  • 20:13 demon@tin: Synchronized wmf-config/InitialiseSettings.php: No-op comment-only (duration: 00m 52s)
  • 17:50 volans: rcs1002.eqiad.wmnet: python-gevent downgraded from 1.1b6-1~trusty1 to 1.0-1ubuntu1
  • 17:41 ema: rcs1001.eqiad.wmnet: python-gevent downgraded from 1.1b6-1~trusty1 to 1.0-1ubuntu1
  • 17:24 reedy@tin: Synchronized wmf-config/throttle.php: T153741 (duration: 00m 40s)
  • 16:31 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: cluster=restbase,dc=eqiad,name=restbase101[6-8].*
  • 16:30 bblack: upgrading tons of outdated OS packages on rcs100x
  • 16:29 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: cluster=restbase,dc=codfw,name=restbase201.*
  • 16:26 marostegui: Running optimize table on db1045 for the revision tables as we urgently need some space back on that host - T153739
  • 16:19 mutante: notebook1001/1002 - upgraded php packages
  • 16:17 jynus: drop repl_records_partition events from db1069
  • 16:09 mobrovac: change-prop restarting everywhere
  • 15:59 mobrovac: restarting change-prop on scb1001
  • 15:48 mobrovac@tin: Finished deploy [parsoid/deploy@e8f0865]: Deploy of mwApiServer across the wtp fleet (duration: 05m 59s)
  • 15:42 mobrovac@tin: Starting deploy [parsoid/deploy@e8f0865]: Deploy of mwApiServer across the wtp fleet
  • 15:32 mobrovac@tin: Finished deploy [parsoid/deploy@e8f0865]: Canary deploy of mwApiServer to wtp[12]001, take two (duration: 00m 37s)
  • 15:31 mobrovac@tin: Starting deploy [parsoid/deploy@e8f0865]: Canary deploy of mwApiServer to wtp[12]001, take two
  • 14:33 mobrovac@tin: Finished deploy [parsoid/deploy@1d75b14]: Canary deploy of mwApiServer to wtp[12]001 (duration: 00m 32s)
  • 14:33 mobrovac@tin: Starting deploy [parsoid/deploy@1d75b14]: Canary deploy of mwApiServer to wtp[12]001
  • 12:04 jynus: running redact_sanitarium.sh on db1069
  • 08:27 elukey: renamed some log files ($something.1.gz to $something.1a.gz) on cp1008 and rutherium to unblock logrotation and reduce cronspam - T132324
  • 02:29 mutante: fermium (lists) - upgrade apache2 | uranium (ganglia-web) - upgrade apache2, openssh, openssl
  • 02:26 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Dec 20 02:26:08 UTC 2016 (duration 4m 48s)
  • 02:23 mutante: osmium - upgrade imagemagick, php5 packages | hafnium - upgrade php5
  • 02:21 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.6) (duration: 06m 41s)
  • 02:17 mutante: contint1002/2001, einsteinium/tegmen - upgrade php5 | bromine - upgrade php5, libs
  • 02:11 mutante: auth (yubiauth), phab2001 - upgrade php5 | bohrium (piwik) - upgrade ssh,ssl,php5
  • 02:02 mutante: copper - upgraded php5* | krypton - installed all pending upgrades
  • 01:45 mutante: terbium - upgrade imagemagick | wasat - upgrade php5, imagemagick
  • 01:41 mutante: tin - upgrading php, imagemagick,..
  • 01:35 mutante: mira - upgrading php5 packages

2016-12-19

  • standup: updated fundraising dash from 040557d to ea4305b
  • 21:00 nuria@tin: Finished deploy [analytics/refinery@ead5b8b]: (no message) (duration: 01m 59s)
  • 20:58 nuria@tin: Starting deploy [analytics/refinery@ead5b8b]: (no message)
  • 19:44 otto@tin: Finished deploy [analytics/refinery@711a572]: (no message) (duration: 00m 06s)
  • 19:44 otto@tin: Starting deploy [analytics/refinery@711a572]: (no message)
  • 19:41 nuria@tin: Finished deploy [analytics/refinery@711a572]: (no message) (duration: 00m 04s)
  • 19:41 nuria@tin: Starting deploy [analytics/refinery@711a572]: (no message)
  • 19:36 nuria@tin: Finished deploy [analytics/refinery@711a572]: (no message) (duration: 00m 04s)
  • 19:36 nuria@tin: Starting deploy [analytics/refinery@711a572]: (no message)
  • 19:35 nuria@tin: Finished deploy [analytics/refinery@711a572]: (no message) (duration: 27m 10s)
  • 19:16 mutante: analytics1027 - out of disk, apt-get clean to free about 500M
  • 19:08 nuria@tin: Starting deploy [analytics/refinery@711a572]: (no message)
  • 16:17 marostegui: Run lots os small optimize tables on db1015 as it needs to get some space back urgently
  • 16:04 andrewbogott: upgrading to python-urllib3_1.19 on scb1001
  • 14:02 jynus: deploying new firewall rules to labsdb1009/10/11
  • 13:39 elukey: Manually raise hhvm.server.connection_timeout_seconds on mw1259 to one day
  • 13:15 _joe_: restarted hhvm, apache on mw1260, raised the apache timeout to 1 day, restarted the jobrunner, disabled puppet
  • 11:47 _joe_: disabling puppet, reconfiguring timeout on apache, restarting HHVM on mw1259
  • 10:16 elukey: reimaging mw1168 and mw1169 to Trusty - T153488
  • 09:38 elukey: stopping jobrunner/jobchron daemons on mw116[89] as prep step for repurpose to videoscalers - T153488
  • 09:23 marostegui: Stop mysql db2048 (depooled) for maintenance - T149553
  • 09:20 elukey: killing irc-echo
  • 09:04 ariel@tin: Finished deploy [dumps/dumps@c8fb9a1]: table jobs to yaml config; stop dumping private tables completely (duration: 00m 01s)
  • 09:04 ariel@tin: Starting deploy [dumps/dumps@c8fb9a1]: table jobs to yaml config; stop dumping private tables completely
  • 06:44 marostegui: Deploy innodb compression dbstore2001 on dewiki and wikidatawiki - T151552
  • 02:23 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Dec 19 02:23:18 UTC 2016 (duration 4m 23s)
  • 02:18 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.6) (duration: 06m 39s)
  • 00:33 mobrovac: starting back cassandra on restbase1011

2016-12-18

  • 22:34 ariel@tin: Finished deploy [dumps/dumps@92946f0]: make monitoring more robust (duration: 00m 01s)
  • 22:34 ariel@tin: Starting deploy [dumps/dumps@92946f0]: make monitoring more robust
  • 22:17 ariel@tin: Finished deploy [dumps/dumps@2a35e23]: fix checkpoint prefetch jobs (duration: 00m 02s)
  • 22:17 ariel@tin: Starting deploy [dumps/dumps@2a35e23]: fix checkpoint prefetch jobs