Server Admin Log

From Wikitech
(Redirected from Server admin log)
Jump to: navigation, search

2017-12-18

2017-12-16

  • 17:27 no_justification: gerrit: Back, might see a few transient puppet failures if git pulls happened during the d/t, but should all recover
  • 17:21 no_justification: gerrit: halting service momentarily for account reindexing
  • 15:00 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1100 (duration: 00m 58s)
  • 12:29 ariel@tin: Finished deploy [dumps/dumps@95dbfe6]: revert previous deploy (duration: 00m 02s)
  • 12:29 ariel@tin: Started deploy [dumps/dumps@95dbfe6]: revert previous deploy
  • 12:16 ariel@tin: Finished deploy [dumps/dumps@faf7de8]: use cat to recombine gzipped files (duration: 00m 02s)
  • 12:16 ariel@tin: Started deploy [dumps/dumps@faf7de8]: use cat to recombine gzipped files
  • 02:04 mutante: labweb1002 - manually downgrade to scap 3.7.4-1 (disabled puppet)
  • 01:49 mutante: reimported scap 3.7.4-1 into APT (jessie-wikimedia) after fixing md5/sha sums in .dsc and .changes files to match orig.tar.gz | copied it from jessie-wikimedia to trusty and stretch-wikimedia. all distributions downgraded to 3.7.4-1 (T183046)
  • 01:42 mutante: reimported scap 3.7.4-1 into APT (jessie-wikimedia) after fixing md5/sha sums in .dsc and .changes files to match orig.tar.gz
  • 01:15 mutante: reprepro removing scap 3.7.4-2 package, attempting to reimport 3.7.4-1 package
  • 01:11 mutante: reprepro remove trusty-wikimedia scap
  • 00:44 demon@tin: Synchronized php-1.31.0-wmf.12/extensions/LoginNotify/includes/LoginNotify.php: T182867 (duration: 00m 57s)
  • 00:40 demon@tin: Synchronized README: Testing again, this time with feeling (duration: 00m 56s)
  • 00:28 mutante: re-disabled puppet on labweb1001/labweb1002 (as it was before)
  • 00:26 mutante: re-enabling puppet on scap hosts
  • 00:25 demon@tin: Synchronized README: Testing (duration: 00m 57s)
  • 00:10 mutante: no more scap 3.7.4-2 found across 'R:Package = scap' (T183046)

2017-12-15

  • 23:55 mutante: downgrading scap from 3.7.4-2 to 3.7.4-1 where it is installed - cumin -b 10 -s 5 'R:Package = scap' 'if dpkg -l scap | grep "3.7.4.2" && file /var/cache/apt/archives/scap_3.7.4-1_all.deb; then puppet agent --disable; apt-get remove --yes -q scap ; dpkg -i /var/cache/apt/archives/scap_3.7.4-1_all.deb ; fi' targeting 478 hosts (T183046)
  • 23:37 mutante: aqs1004, analytics1003, downgraded scap to 3.7.4-1
  • 22:55 mutante: tin - apt-get remove scap ; dpkg -i /var/cache/apt/archives/scap_3.7.4-1_all.deb
  • 19:39 chasemp: reboot labtestvirt2003
  • 18:15 jgleeson: rolled back to civicrm to 798e2467 to investigate prometheus bug
  • 17:32 jynus: stop, upgrade and reboot labsdb1009
  • 17:18 jynus: reloading dbproxy1010
  • 16:44 chasemp: labtestvirt2003:~# /sbin/reboot to pickup new kernel
  • 16:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1084 original weight (duration: 00m 57s)
  • 16:10 elukey: re-enable piwik on bohrium after mysql backup restore
  • 15:11 chasemp: reimage labtestvirt2003.codfw.wmnet
  • 13:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1084 and restore original weight for db1081 (duration: 00m 56s)
  • 13:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1084 and db1081 (duration: 00m 56s)
  • 13:20 chasemp: disable puppet across eqiad lab* things to land a bit of code gracefully
  • 13:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1084 (duration: 00m 56s)
  • 12:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1084 with low weight - T180788 (duration: 00m 57s)
  • 10:59 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1080 - T174569 (duration: 00m 56s)
  • 10:59 jynus: reloading dbproxy1011
  • 10:50 marostegui: Stop MySQL on db1084 to clone db1111 - T180788
  • 10:49 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1084 to clone db1111 - T180788 (duration: 00m 56s)
  • 10:31 elukey: rolling restart of yarn nodemanagers on an103* to apply new config - T182276
  • 10:14 moritzm: uploaded prometheus-dns-rec-exporter 0.3 to apt.wikimedia.org
  • 09:50 elukey: restore piwik database on bohrium after mysql corruption - piwik disabled
  • 09:26 marostegui: Deploy schema change on db1080 - T174569
  • 09:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1080 - T174569 (duration: 00m 56s)
  • 09:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1083 - T174569 (duration: 00m 56s)
  • 08:50 moritzm: powercycling ganeti1005
  • 08:38 marostegui: Stop MySQL on db1034 to decommission it - T182556
  • 08:26 marostegui: Remove db1034 from tendril - T182556
  • 06:50 marostegui: Deploy schema change on db1083 - T174569
  • 06:49 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1083 - T174569 (duration: 00m 56s)
  • 06:43 marostegui: Deploy schema change on dbstore1001 (s1) - T174569
  • 06:29 marostegui: Fix dbstore1001 s5 replication
  • 06:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 and db1109 - T161294 (duration: 01m 16s)
  • 03:01 mutante: db2016 thru db2019 - had to manually kill gmond process to decom ganglia, other db codfw hosts: didnt need it | running puppet on db205* and others in codfw to remove all ganglia (T177225)
  • 01:23 catrope@tin: Synchronized php-1.31.0-wmf.12/extensions/ORES: Fix broken join conditions (T182936) (duration: 00m 57s)
  • 01:22 hoo: Updated the Wikidata property suggester with data from Monday's JSON dump and applied the T132839 workarounds
  • 01:19 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Don't discount file searches on commonswiki (duration: 00m 57s)
  • 01:11 catrope@tin: Synchronized php-1.31.0-wmf.11: Pulling in today's cherry-picks into wmf.11 too (duration: 10m 49s)
  • 00:44 catrope@tin: Synchronized php-1.31.0-wmf.12/resources/src/mediawiki.rcfilters/ui/mw.rcfilters.ui.MenuSelectWidget.js: T182711 (duration: 00m 56s)
  • 00:33 catrope@tin: Synchronized php-1.31.0-wmf.12/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.ArticleTargetEvents.js: Track editor mode on save events (T182610) (duration: 00m 56s)
  • 00:25 catrope@tin: Synchronized php-1.31.0-wmf.12/resources/lib/oojs-ui/oojs-ui-core.js: Backport OOjs UI fix for T182359, T182395 (duration: 00m 57s)
  • 00:14 RoanKattouw: updateCollation.php finished on sewiki (T181503)
  • 00:13 RoanKattouw: Running updateCollation.php on sewiki
  • 00:13 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Set category collation for sewiki to uppercase-se (duration: 00m 57s)

2017-12-14

  • 22:58 demon@tin: rebuilt and synchronized wikiversions files: group2 to wmf.12 (#2)
  • 22:47 demon@tin: Synchronized php-1.31.0-wmf.12/extensions/ORES/includes/Hooks/ApiHooksHandler.php: fix undefined property (duration: 01m 08s)
  • 22:17 demon@tin: rebuilt and synchronized wikiversions files: rollback, ORES breaking stuff on enwiki
  • 22:08 demon@tin: rebuilt and synchronized wikiversions files: group2 to wmf.12
  • 21:53 ejegg: updated CiviCRM from 1086291bb3 to 41a23f9fc3
  • 20:33 awight: stress testing ores*
  • 20:30 demon@tin: Pruned MediaWiki: 1.31.0-wmf.10 [keeping static files] (duration: 03m 32s)
  • 20:28 jgleeson: turned back on donations queue consume service and thank you mailer service
  • 20:21 awight@tin: Started restart [ores/deploy@b67bba7]: (non-production) Restart ORES services on ores*
  • 20:19 thcipriani@tin: Finished scap: SWAT: Update en/i18n message for multiple-unclosed-formatting-tags (duration: 30m 55s)
  • 20:02 jgleeson: updated civicrm from 798e24671b to 1086291bb3
  • {{safesubst:SAL entry|1=20:02 urandom: lowering cassandra compaction throughput to 5MB/s, restbase101{2,4}-{a,b,c}}}
  • 19:58 jgleeson: Temporarily disabled donations queue consume service and thank you mailer service
  • 19:48 thcipriani@tin: Started scap: SWAT: Update en/i18n message for multiple-unclosed-formatting-tags
  • 19:43 thcipriani@tin: Synchronized wmf-config/throttle.php: SWAT: Define new throttle rule and cleaning expired rules T182889 (duration: 01m 08s)
  • 19:29 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wgNamespaceRobotPolicies for wikidata T181525 (duration: 01m 04s)
  • 19:20 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove single editor tab for plwiki T181045 (duration: 01m 09s)
  • 19:02 awight@tin: Finished deploy [ores/deploy@b67bba7]: Redeploy ORES to scb1001 (duration: 01m 04s)
  • 19:01 awight@tin: Started deploy [ores/deploy@b67bba7]: Redeploy ORES to scb1001
  • 19:01 bsitzmann@tin: Finished deploy [mobileapps/deploy@bf85a55]: Update mobileapps to ff74bb1 (T182868 T182774) (duration: 12m 26s)
  • 18:54 awight@tin: Started restart [ores/deploy@b67bba7]: Restart ORES services
  • 18:51 arlolra: Updated Parsoid to ca20680 (T182793, T182774)
  • 18:48 bsitzmann@tin: Started deploy [mobileapps/deploy@bf85a55]: Update mobileapps to ff74bb1 (T182868 T182774)
  • 18:41 arlolra@tin: Finished deploy [parsoid/deploy@13b5cb5]: (no justification provided) (duration: 09m 19s)
  • 18:32 arlolra@tin: Started deploy [parsoid/deploy@13b5cb5]: (no justification provided)
  • 18:24 elukey: replace kafka1018 with kafka1023 (Analytics Kafka cluster)
  • 18:11 awight@tin: Started deploy [ores/deploy@b67bba7]: Update ORES service to b67bba77acb
  • 16:34 bd808: Scholarships: Set application start and close dates via web UI (T181072)
  • 16:25 niharika29@tin: Finished deploy [scholarships/scholarships@872381d]: Deploy wikimania scholarships app for 2018 T181072 (duration: 00m 02s)
  • 16:25 niharika29@tin: Started deploy [scholarships/scholarships@872381d]: Deploy wikimania scholarships app for 2018 T181072
  • 16:07 bd808: Scholarships: updated database schema with 20171212-add-scholarship-orgs-field.sql (T181072)
  • 15:48 marostegui: Deploy schema change on dbstore1002 (s1) - T174569
  • 15:25 jynus: stop, upgrade and restart labsdb1011
  • 15:07 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1089 - T174569 (duration: 01m 08s)
  • 14:21 hashar@tin: Synchronized php-1.31.0-wmf.12/resources/src/mediawiki.rcfilters: Swat for RCFilters https://gerrit.wikimedia.org/r/#/c/398242/ https://gerrit.wikimedia.org/r/#/c/398239/ (duration: 01m 09s)
  • 13:47 chasemp: I'm reverting https://gerrit.wikimedia.org/r/#/c/394541/ as it broke the database puppet for labservices (used for powerdns backend)
  • 13:42 godog: upgrade grafana to 4.6.3 - T182294
  • 13:41 elukey: update facts for puppet compiler to pick up new hosts
  • 13:32 mobrovac@tin: Finished deploy [restbase/deploy@187d8ba]: Remove Trending Edits end point and stop storing feed results in Cassandra - T180384 T179412 (duration: 05m 37s)
  • 13:26 mobrovac@tin: Started deploy [restbase/deploy@187d8ba]: Remove Trending Edits end point and stop storing feed results in Cassandra - T180384 T179412
  • 13:10 marostegui: Deploy schema change on db1089 (s1) - T174569
  • 13:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1089 - T174569 (duration: 01m 07s)
  • 12:56 awight: stress testing on ores1*.eqiad.wmnet cluster, T182249
  • 12:42 awight@tin: Finished deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster (duration: 00m 59s)
  • 12:41 awight@tin: Started deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster
  • 12:40 awight@tin: Finished deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster (duration: 00m 45s)
  • 12:39 awight@tin: Started deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster
  • 12:18 gehel: re-initialize cassandra on maps-test2001 - T182583
  • 12:11 jynus: disable puppet on all databases to deploy safely https://gerrit.wikimedia.org/r/398246
  • 12:00 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1091 - T174569 (duration: 01m 08s)
  • 11:27 awight@tin: Finished deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster (duration: 00m 20s)
  • 11:27 awight@tin: Started deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster
  • 11:24 awight@tin: Finished deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster (duration: 00m 05s)
  • 11:23 awight@tin: Started deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster
  • 11:23 awight@tin: Finished deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster (duration: 00m 11s)
  • 11:23 awight@tin: Started deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster
  • 11:16 akosiaris@tin: Finished deploy [ores/deploy@b67bba7]: T181661 (duration: 00m 03s)
  • 11:16 akosiaris@tin: Started deploy [ores/deploy@b67bba7]: T181661
  • 11:14 akosiaris@tin: Finished deploy [ores/deploy@b67bba7]: T181661 (duration: 00m 03s)
  • 11:14 akosiaris@tin: Started deploy [ores/deploy@b67bba7]: T181661
  • 11:13 akosiaris@tin: Finished deploy [ores/deploy@b67bba7]: T181661 (duration: 00m 55s)
  • 11:12 akosiaris@tin: Started deploy [ores/deploy@b67bba7]: T181661
  • 09:27 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1067 (duration: 01m 08s)
  • 08:39 marostegui: Reload dbproxy1011 config
  • 07:18 marostegui: Stop replication and set read-only on labsdb1003 - T142807
  • 07:08 marostegui: Stop replication in sync on db1109 and db1101:3318 - T161294
  • 07:08 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1101:3318 and db1109 - T161294 (duration: 01m 07s)
  • 07:02 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 and db1109 - T161294 (duration: 01m 08s)
  • 06:55 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1101:3318 and db1109 - T161294 (duration: 01m 09s)
  • 06:35 marostegui: Deploy schema change on db1091 (s4) - T174569
  • 06:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1091 - T174569 (duration: 01m 08s)
  • 02:26 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.11) (duration: 08m 05s)
  • 01:46 twentyafterfour: no phabricator deployment tonight, not enough time to prepare and test the update due to a short outage earlier this evening.
  • 00:12 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: T182534 (duration: 01m 08s)

2017-12-13

  • 23:54 twentyafterfour: restarted phd on phab1001 (for good measure)
  • 23:23 reedy@tin: Synchronized php-1.31.0-wmf.12/extensions/Flow/Hooks.php: unbreak CheckUser (duration: 01m 08s)
  • 23:22 foks: deleted 22 illegal images from server
  • 22:51 volans: restarting apache2 on phab1001, phabricator timing out
  • 22:23 ppchelko@tin: Finished deploy [cpjobqueue/deploy@044cd23]: Fix sha1-based deduplication (duration: 00m 34s)
  • 22:23 ppchelko@tin: Started deploy [cpjobqueue/deploy@044cd23]: Fix sha1-based deduplication
  • 22:03 mholloway-shell@tin: Finished deploy [mobileapps/deploy@e62d8e3]: Update mobileapps to ddddebb (duration: 05m 24s)
  • 21:58 mholloway-shell@tin: Started deploy [mobileapps/deploy@e62d8e3]: Update mobileapps to ddddebb
  • 21:54 ejegg: updated civicrm from 85a8526eb8 to 798e24671b
  • 21:37 mholloway-shell@tin: Finished deploy [mobileapps/deploy@b8082da]: Update mobileapps to bf67c97 (duration: 04m 19s)
  • 21:33 mholloway-shell@tin: Started deploy [mobileapps/deploy@b8082da]: Update mobileapps to bf67c97
  • 21:24 ppchelko@tin: Finished deploy [restbase/deploy@a993556]: Do not fallback if the revision is not specified T182770 (duration: 04m 04s)
  • 21:20 ppchelko@tin: Started deploy [restbase/deploy@a993556]: Do not fallback if the revision is not specified T182770
  • 20:27 demon@tin: Synchronized php-1.31.0-wmf.12/extensions/WikimediaEvents/extension.json: James_F made me do it (duration: 01m 08s)
  • 20:14 demon@tin: rebuilt and synchronized wikiversions files: group1 to wmf.12
  • 20:10 demon@tin: Synchronized php: symlink bump for wmf.12 (duration: 01m 07s)
  • 19:37 ebernhardson@tin: Synchronized php-1.31.0-wmf.12/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.trackSubscriber.js: SWAT: VE trackSubscriber: Add timing data for 'loaded' state (duration: 01m 07s)
  • 19:35 ppchelko@tin: Finished deploy [restbase/deploy@3f4bedc]: Remove references to Cassandra 2 from Parsoid storage T179417 (duration: 04m 43s)
  • 19:34 ebernhardson@tin: Synchronized php-1.31.0-wmf.12/resources/src/mediawiki.rcfilters/mw.rcfilters.UriProcessor.js: SWAT: T182734: RCLFilters: support target page with a subpage (duration: 01m 07s)
  • 19:30 ppchelko@tin: Started deploy [restbase/deploy@3f4bedc]: Remove references to Cassandra 2 from Parsoid storage T179417
  • 19:25 ebernhardson@tin: Synchronized php-1.31.0-wmf.12/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.trackSubscriber.js: SWAT: VE trackSubscriber: data isn't required (duration: 01m 08s)
  • 19:14 ebernhardson@tin: Synchronized php-1.31.0-wmf.12/resources/src/mediawiki.rcfilters/: SWAT: T182788: RCFilters: Fix live update (duration: 01m 08s)
  • 19:07 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Turn on cirrus MLR for 4 more wikis (duration: 01m 09s)
  • 19:05 mutante: phab[12]001,mx[12]001,mendelevium,fermium: rm /usr/local/bin/exim-to-gmetric and remove root's crontab lines to follow-up gerrit:382916
  • 18:31 mutante: releases2001: /srv/mediawiki# rm -rf extensions/ skins/ vendor/ | clean up removed repos, let puppet clone, to match releases1001 and fix puppet run
  • 17:33 awight@tin: Finished deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster (duration: 00m 59s)
  • 17:32 awight@tin: Started deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster
  • 17:30 moritzm: installing wireshark security updates
  • 16:30 marostegui: Deploy schema change on s4 on dbstore1001 - T174569
  • 16:19 godog: try again deleting obsolete cassandra metrics from graphite2002 - T181964
  • 15:50 akosiaris@tin: Finished deploy [ores/deploy@b4f2b02]: (no justification provided) (duration: 02m 22s)
  • 15:48 akosiaris@tin: Started deploy [ores/deploy@b4f2b02]: (no justification provided)
  • 15:47 akosiaris@tin: Finished deploy [ores/deploy@b4f2b02]: (no justification provided) (duration: 01m 30s)
  • 15:46 akosiaris@tin: Started deploy [ores/deploy@b4f2b02]: (no justification provided)
  • 15:43 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1034 from config - T182556 (duration: 01m 07s)
  • 15:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1034 from config - T182556 (duration: 01m 09s)
  • 15:39 akosiaris@tin: Finished deploy [ores/deploy@b4f2b02]: (no justification provided) (duration: 04m 46s)
  • 15:34 akosiaris@tin: Started deploy [ores/deploy@b4f2b02]: (no justification provided)
  • 15:21 chasemp: remove and purge vblade-persist and runit from labstore1004 T182781
  • 15:02 jynus: upgrade and restart db1067
  • 15:02 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1067 (duration: 01m 07s)
  • 14:43 zeljkof: EU SWAT finished
  • 14:38 zfilipin@tin: Synchronized portals: SWAT: Bumping portals to master (T128546) (duration: 01m 09s)
  • 14:37 zfilipin@tin: Synchronized portals/prod/wikipedia.org/assets: SWAT: Bumping portals to master (T128546) (duration: 01m 08s)
  • 14:29 dcausse@tin: Synchronized php-1.31.0-wmf.12/extensions/Wikibase: T182293 Extract names of search fields as constants (duration: 02m 05s)
  • 14:22 dcausse@tin: Synchronized wmf-config/InitialiseSettings.php: T182293 [cirrus] tune wikidata similarity configuration 2/2 (duration: 01m 07s)
  • 14:20 dcausse@tin: Synchronized wmf-config/Wikibase.php: T182293 [cirrus] tune wikidata similarity configuration 1/2 (duration: 01m 12s)
  • 14:03 akosiaris: gnt-node remove ganeti1006 T181121
  • 14:01 elukey: restart Yarn nodemanagers on analytics102[8,9] to apply new settings - T182276
  • 13:21 moritzm: uploaded prometheus-pdns-rec-exporter 0.2-1 to apt.wikimedia.org
  • 13:10 marostegui: Deploy schema change on s4 - dbstore1002 - T174569
  • 12:53 marostegui: Deploy alter table on s4 db1064 (sanitarium master) with replication, this will generate lag on labs replicas - T174569
  • 12:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1103:3314 - T174569 (duration: 01m 36s)
  • 12:19 moritzm: uploaded prometheus-ircd-exporter and prometheus-pdns-rec-exporter to apt.wikimedia.org
  • 11:59 elukey: forced remount of /mnt/hdfs after OOM event on stat1005
  • 11:41 akosiaris: empty ganeti1006 for reimage after ltpstress and bios upgrades T181121
  • 11:16 mobrovac@tin: Finished deploy [graphoid/deploy@7979a40]: Update to service-template-node v0.5.4 (duration: 03m 56s)
  • 11:12 mobrovac@tin: Started deploy [graphoid/deploy@7979a40]: Update to service-template-node v0.5.4
  • 10:19 mobrovac@tin: Finished deploy [recommendation-api/deploy@ac66089]: Update to service-template-node v0.5.4 (duration: 02m 20s)
  • 10:17 mobrovac@tin: Started deploy [recommendation-api/deploy@ac66089]: Update to service-template-node v0.5.4
  • 09:41 godog: upload prometheus-elasticsearch-exporter to jessie-wikimedia - T181627
  • 08:38 jynus: migrate away from ganeti1008
  • 06:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1103:3314 - T174569 (duration: 01m 11s)
  • 06:25 marostegui: Deploy schema change on db1103:3314 - T174569
  • 06:12 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1097:3314 - T174569 (duration: 01m 08s)
  • 02:26 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.11) (duration: 07m 02s)
  • 00:49 ebernhardson@tin: Synchronized php-1.31.0-wmf.11/extensions/WikimediaEvents/: SWAT: T182616: turn on second mlr ab test for hewiki (duration: 01m 08s)
  • 00:44 ebernhardson@tin: Synchronized php-1.31.0-wmf.12/extensions/WikimediaEvents/: SWAT: T182616: turn on second mlr ab test for hewiki (duration: 01m 08s)
  • 00:40 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Turn on cirrus MLR for most wikis with >1% of search traffic (duration: 01m 08s)
  • 00:35 mholloway-shell@tin: Finished deploy [mobileapps/deploy@5832a8c]: Update mobileapps to bfc3588 (duration: 04m 48s)
  • 00:32 brion: restarted requeueTranscodes.php on terbium for mp3 audio generation backfill (had dropped DB connection)
  • 00:31 mholloway-shell@tin: Started deploy [mobileapps/deploy@5832a8c]: Update mobileapps to bfc3588
  • 00:05 ebernhardson@tin: Synchronized wmf-config/: SWAT: T182616: Setup MLR AB test for hewiki (duration: 01m 10s)

2017-12-12

  • 22:30 mholloway-shell@tin: Finished deploy [mobileapps/deploy@ea8f05d]: Update mobileapps to 94f267b (duration: 05m 10s)
  • 22:24 mholloway-shell@tin: Started deploy [mobileapps/deploy@ea8f05d]: Update mobileapps to 94f267b
  • 22:02 urandom: setting compaction throughput to 5 MB/s, restbase1010
  • 21:31 mutante: releases1001 - rm mediawiki core repo and let puppet try to recreate it (follow-up issue after gerrit:397891)
  • 21:28 mobrovac@tin: Finished deploy [restbase/deploy@dceab2e]: Switch to Parsoid content v1.6.0 and switch to Cassandra 3 storage - T179417 (duration: 04m 16s)
  • 21:27 demon@tin: Synchronized php-1.31.0-wmf.12/extensions/GlobalBlocking/: (no justification provided) (duration: 01m 07s)
  • 21:26 mholloway-shell@tin: Finished deploy [mobileapps/deploy@28bfda3]: Update mobileapps to d0ee651 (duration: 01m 45s)
  • 21:24 demon@tin: Synchronized php-1.31.0-wmf.11/extensions/GlobalBlocking/: (no justification provided) (duration: 01m 08s)
  • 21:24 mholloway-shell@tin: Started deploy [mobileapps/deploy@28bfda3]: Update mobileapps to d0ee651
  • 21:24 mholloway-shell@tin: Finished deploy [mobileapps/deploy@b2d5b8e]: Update mobileapps to 172abc7 (duration: 00m 30s)
  • 21:24 mholloway-shell@tin: Started deploy [mobileapps/deploy@b2d5b8e]: Update mobileapps to 172abc7
  • 21:23 mobrovac@tin: Started deploy [restbase/deploy@dceab2e]: Switch to Parsoid content v1.6.0 and switch to Cassandra 3 storage - T179417
  • 21:21 ppchelko@tin: Finished deploy [restbase/deploy@dceab2e]: Bump expected parsoid version, but do not switch summaries, take 4 (duration: 02m 45s)
  • 21:18 ppchelko@tin: Started deploy [restbase/deploy@dceab2e]: Bump expected parsoid version, but do not switch summaries, take 4
  • 21:15 ppchelko@tin: Finished deploy [restbase/deploy@dceab2e]: Bump expected parsoid version, but do not switch summaries, take 3 (duration: 02m 04s)
  • 21:13 ppchelko@tin: Started deploy [restbase/deploy@dceab2e]: Bump expected parsoid version, but do not switch summaries, take 3
  • 21:13 ppchelko@tin: Finished deploy [restbase/deploy@dceab2e]: Bump expected parsoid version, but do not switch summaries, take 2 after failed content rerender (duration: 05m 05s)
  • 21:12 mobrovac@tin: Started restart [electron-render/deploy@94d27d7]: Electron hanging - T174916
  • 21:08 ppchelko@tin: Started deploy [restbase/deploy@dceab2e]: Bump expected parsoid version, but do not switch summaries, take 2 after failed content rerender
  • 21:08 ppchelko@tin: Finished deploy [restbase/deploy@dceab2e]: Bump expected parsoid version, but do not switch summaries (duration: 05m 51s)
  • 21:02 ppchelko@tin: Started deploy [restbase/deploy@dceab2e]: Bump expected parsoid version, but do not switch summaries
  • 20:55 ppchelko@tin: Finished deploy [restbase/deploy@d3ca789]: Revert deployment for using MCS for summaries (duration: 01m 36s)
  • 20:53 ppchelko@tin: Started deploy [restbase/deploy@d3ca789]: Revert deployment for using MCS for summaries
  • 20:53 demon@tin: rebuilt and synchronized wikiversions files: group0 to wmf.12
  • 20:51 ppchelko@tin: Finished deploy [restbase/deploy@506047c]: Update expected Parsoid version, switched summary to MCS (duration: 03m 30s)
  • 20:48 ppchelko@tin: Started deploy [restbase/deploy@506047c]: Update expected Parsoid version, switched summary to MCS
  • 20:44 mholloway-shell@tin: Finished deploy [mobileapps/deploy@b2d5b8e]: Update mobileapps to 172abc7 (duration: 04m 48s)
  • 20:39 mholloway-shell@tin: Started deploy [mobileapps/deploy@b2d5b8e]: Update mobileapps to 172abc7
  • 20:35 mholloway-shell@tin: Finished deploy [mobileapps/deploy@0a9d635]: Update mobileapps to 035608d (duration: 02m 32s)
  • 20:32 mholloway-shell@tin: Started deploy [mobileapps/deploy@0a9d635]: Update mobileapps to 035608d
  • 20:31 mholloway-shell@tin: Finished deploy [mobileapps/deploy@2690678]: Update mobileapps to 5b8796d (duration: 40m 04s)
  • 19:57 arlolra: Updated Parsoid to 741fc5d (T114072, T181226, T21910, T152540, T103714, T97093, T118520, T181229, T182338, T182170, T169006)
  • 19:51 mholloway-shell@tin: Started deploy [mobileapps/deploy@2690678]: Update mobileapps to 5b8796d
  • 19:43 arlolra@tin: Finished deploy [parsoid/deploy@98139cb]: (no justification provided) (duration: 14m 57s)
  • 19:35 anomie: Running cleanupUsersWithNoId.php on all wikis (this will take a while), see T181731
  • 19:12 arlolra: Parsoid deploy aborted and rolled back to 01c1fc3 while RESTBase fixes an issue
  • 19:06 arlolra@tin: Finished deploy [parsoid/deploy@98139cb]: Updating Parsoid to 741fc5d (duration: 05m 33s)
  • 19:00 arlolra@tin: Started deploy [parsoid/deploy@98139cb]: Updating Parsoid to 741fc5d
  • 18:48 aaron@tin: Synchronized php-1.31.0-wmf.12/includes/Setup.php: 058c17e702eb0 (duration: 01m 09s)
  • 18:21 moritzm: uploaded prometheus-ircd-exporter to apt.wikimedia.org
  • 17:52 demon@tin: Finished scap: bootstrap wmf.12 (duration: 29m 35s)
  • 17:22 demon@tin: Started scap: bootstrap wmf.12
  • 17:15 demon@tin: Pruned MediaWiki: 1.31.0-wmf.7 (duration: 08m 45s)
  • 17:12 akosiaris@tin: Finished deploy [ores/deploy@b4f2b02]: T181661 (duration: 00m 03s)
  • 17:12 akosiaris@tin: Started deploy [ores/deploy@b4f2b02]: T181661
  • 17:11 akosiaris@tin: Finished deploy [ores/deploy@b4f2b02]: T181661 (duration: 00m 36s)
  • 17:11 akosiaris@tin: Started deploy [ores/deploy@b4f2b02]: T181661
  • 17:03 akosiaris@tin: Started deploy [ores/deploy@b4f2b02]: T181661
  • 16:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1084 and db1081 original weight (duration: 00m 56s)
  • 16:40 jynus: restart and upgrade db1059 (phabricator passive db)
  • 16:35 gehel@tin: Finished deploy [kartotherian/deploy@6e223df]: new kartotherian packaging on maps-test2001 (duration: 00m 19s)
  • 16:35 gehel@tin: Started deploy [kartotherian/deploy@6e223df]: new kartotherian packaging on maps-test2001
  • 16:34 gehel@tin: Finished deploy [tilerator/deploy@29d633e]: new tilerator packaging on maps-test2001 (duration: 00m 20s)
  • 16:34 gehel@tin: Started deploy [tilerator/deploy@29d633e]: new tilerator packaging on maps-test2001
  • 16:30 gehel@tin: Finished deploy [tilerator/deploy@29d633e]: new tilerator packaging on maps-test2002 (duration: 00m 20s)
  • 16:30 gehel@tin: Started deploy [tilerator/deploy@29d633e]: new tilerator packaging on maps-test2002
  • 16:28 gehel@tin: Finished deploy [kartotherian/deploy@6e223df]: new kartotherian packaging on maps-test2002 (duration: 00m 18s)
  • 16:28 gehel@tin: Started deploy [kartotherian/deploy@6e223df]: new kartotherian packaging on maps-test2002
  • 16:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1084 (duration: 00m 53s)
  • 16:24 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2072 (duration: 00m 55s)
  • 16:22 akosiaris@tin: Started deploy [ores/deploy@b4f2b02]: T181661
  • 16:21 akosiaris: failover boron to ganeti1008
  • 16:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1084 (duration: 00m 56s)
  • 15:59 akosiaris@tin: Finished deploy [ores/deploy@b4f2b02]: T181661 (duration: 00m 04s)
  • 15:59 akosiaris@tin: Started deploy [ores/deploy@b4f2b02]: T181661
  • 15:59 akosiaris@tin: Finished deploy [ores/deploy@b4f2b02]: T181661 (duration: 00m 09s)
  • 15:58 akosiaris@tin: Started deploy [ores/deploy@b4f2b02]: T181661
  • 15:46 jynus: stop, upgrade and reboot db2072
  • 15:34 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2072 (duration: 00m 56s)
  • 15:31 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1097:3314 - T174569 (duration: 01m 01s)
  • 15:31 marostegui: Deploy schema change on s4 db1097:3314 - T174569
  • 15:25 gehel: powercycling maps-test2003
  • 15:24 elukey: rename notebook1002 -> kafka1023 - step 3, replace notebook1002 with kafka1023 in the puppet config
  • 15:20 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1084 with low weight (duration: 01m 18s)
  • 15:06 marostegui: Upgrade MySQl on db1084
  • 15:02 elukey: clear recdns records related to notebook1002/kafka1023 (rec_control wipe-cache kafka1023.eqiad.wmnet kafka1023.mgmt.eqiad.wmnet notebook1002.eqiad.wmnet 14.5.64.10.in-addr.arpa 104.3.65.10.in-addr.arpa) - T181518
  • 14:58 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1260.eqiad.wmnet
  • 14:46 elukey: start rename notebook1002 -> kafka1023 - step 2, dns config (host already shutdown) - T181518
  • 14:39 hashar: Rebuild operations-puppet-tests-docker image based on c76d892 | T178620 and /cache being owned by root
  • 14:37 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2071 (duration: 00m 56s)
  • 14:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore original weight for db1086 (duration: 00m 56s)
  • 14:20 zeljkof: EU SWAT finished
  • 14:18 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add NS aliases for zh_wikiquote (T181374) (duration: 00m 56s)
  • 14:09 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: Lift account registration on en.wiki (T182665) (duration: 00m 56s)
  • 14:02 herron: upgrading codfw puppet agents
  • 13:59 jynus: stop, upgrade and reboot db2071
  • 13:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1086 (duration: 00m 56s)
  • 13:56 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2071 (duration: 00m 57s)
  • 13:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1086 (duration: 00m 57s)
  • 12:44 akosiaris@puppetmaster1001: conftool action : set/weight=10; selector: scb1004.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
  • 12:10 akosiaris@puppetmaster1001: conftool action : set/weight=10; selector: scb1004.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
  • 12:10 akosiaris@puppetmaster1001: conftool action : set/weight=10; selector: scb1003.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
  • 12:06 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1092 (duration: 00m 55s)
  • 11:56 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1086 with low weight - T163190 (duration: 00m 55s)
  • 11:27 jynus: stop, upgrade and reboot db1092
  • 11:26 marostegui: Upgrade MySQL and kernel on db1086
  • 11:26 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1092 (duration: 00m 56s)
  • 11:12 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2066 (duration: 00m 57s)
  • 11:11 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1084 - T174569 (duration: 00m 57s)
  • 11:11 marostegui: Deploy schema change on db1084 (s4) - T174569
  • 11:05 marostegui: Deploy schema change on db1056 (s4) already depooled - T174569
  • 11:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1081 - T174569 (duration: 00m 56s)
  • 10:46 gehel@tin: Finished deploy [kartotherian/deploy@6e223df]: new kartotherian packaging on maps-test2003 (duration: 00m 10s)
  • 10:46 gehel@tin: Started deploy [kartotherian/deploy@6e223df]: new kartotherian packaging on maps-test2003
  • 10:26 jynus: stop, upgrade and reboot db2066
  • 10:24 gehel@tin: Finished deploy [kartotherian/deploy@6e223df]: new kartotherian packaging on maps-test2004 (duration: 00m 18s)
  • 10:24 gehel@tin: Started deploy [kartotherian/deploy@6e223df]: new kartotherian packaging on maps-test2004
  • 10:23 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2066 (duration: 00m 56s)
  • 10:14 moritzm: reimaging mw1260 (video scaler) to stretch
  • 10:04 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2059 (duration: 00m 56s)
  • 09:51 moritzm: updated stretch installer netboot image after stretch 9.3 point release
  • 09:51 moritzm: updated stretch installer netboot image after jessie 8.10 point release
  • 09:44 marostegui: Stop db1039 and db1086 in sync - T163190
  • 09:43 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1086 - T163190 (duration: 00m 56s)
  • 09:38 gehel: reduce replication factor for cassandra on maps-test cluster and reset cassandra on maps-test2001 to work around limited disk space - T182583
  • 09:34 marostegui: Stop replication in sync on db1034 and db1039 for data consistency check - T163190
  • 09:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1096:3316 after InnoDB there - T178359 (duration: 00m 56s)
  • 09:00 jynus: stop, upgrade and reboot db2059
  • 08:57 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2059 (duration: 00m 56s)
  • 08:47 moritzm: updated jessie installer netboot image after jessie 8.10 point release
  • 07:15 marostegui: Deploy schema change on db1081 - https://phabricator.wikimedia.org/T174569
  • 07:14 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1081 - T174569 (duration: 00m 56s)
  • 06:56 marostegui: stop MySQL on db1055 - T182653
  • 06:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1055 - T178359 T182653 (duration: 00m 56s)
  • 02:27 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.11) (duration: 06m 21s)
  • 00:53 legoktm@tin: Synchronized wmf-config/CommonSettings.php: Remove manual firejailing of Score binaries (T181535) (duration: 00m 56s)
  • 00:47 legoktm@tin: Synchronized wmf-config/CommonSettings.php: Have ExtensionDistributor treat REL1_30 as stable (duration: 00m 56s)
  • 00:39 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Re-enable ORES on fawiki (T182354) (duration: 00m 56s)
  • 00:25 ejegg: updated CiviCRM from 2db9c76 to 85a8526
  • 00:23 catrope@tin: Synchronized wmf-config/CommonSettings.php: Fix default for rcOresDamagingPref (duration: 00m 56s)
  • 00:22 ejegg: disabled nightly Ingenico WX audit parsing job

2017-12-11

  • 23:37 smalyshev@tin: Finished deploy [wdqs/wdqs@f6b110f]: updater fix and GUI update (duration: 06m 10s)
  • 23:31 smalyshev@tin: Started deploy [wdqs/wdqs@f6b110f]: updater fix and GUI update
  • 23:28 mholloway-shell@tin: Finished deploy [mobileapps/deploy@07293bc]: Update mobileapps to e290b17 (duration: 06m 16s)
  • 23:22 mholloway-shell@tin: Started deploy [mobileapps/deploy@07293bc]: Update mobileapps to e290b17
  • 21:11 mholloway-shell@tin: Finished deploy [mobileapps/deploy@6347d62]: Update mobileapps to 61ca333 (duration: 07m 56s)
  • 21:04 mholloway-shell@tin: Started deploy [mobileapps/deploy@6347d62]: Update mobileapps to 61ca333
  • 19:47 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: T181493: Enable Page Previews EventLogging instrumentation (duration: 00m 56s)
  • 19:47 mobrovac@tin: Finished deploy [restbase/deploy@be7d72f]: Expose the Reading Lists end points, take #3 - T181107 (duration: 06m 19s)
  • 19:46 ppchelko@tin: Finished deploy [cpjobqueue/deploy@b1beaf1]: Revert dedupe based on sha1 as well as on event ID (duration: 00m 29s)
  • 19:46 ppchelko@tin: Started deploy [cpjobqueue/deploy@b1beaf1]: Revert dedupe based on sha1 as well as on event ID
  • {{safesubst:SAL entry|1=19:43 urandom: lower compaction throughput to 2 MB/s, restbase1010-{a,b,c} - T178177}}
  • 19:40 mobrovac@tin: Started deploy [restbase/deploy@be7d72f]: Expose the Reading Lists end points, take #3 - T181107
  • 19:34 mobrovac@tin: Finished deploy [restbase/deploy@bce2885]: Expose the Reading Lists end points, take #2 - T181107 (duration: 05m 33s)
  • 19:28 mobrovac@tin: Started deploy [restbase/deploy@bce2885]: Expose the Reading Lists end points, take #2 - T181107
  • 19:28 mobrovac@tin: Finished deploy [restbase/deploy@bce2885]: Expose the Reading Lists end points - T181107 (duration: 01m 26s)
  • 19:26 mobrovac@tin: Started deploy [restbase/deploy@bce2885]: Expose the Reading Lists end points - T181107
  • 19:24 ebernhardson@tin: Synchronized wmf-config/throttle.php: SWAT: T182613 Update throttle rule for McGill University Library (duration: 00m 56s)
  • 19:21 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: T131132: Switch submit button from save to publish on enwiki (duration: 02m 43s)
  • 19:19 awight@tin: Finished deploy [ores/deploy@1c0ede0]: (non-production) Testing parallel ORES deployment, T181661 (duration: 01m 12s)
  • 19:17 awight@tin: Started deploy [ores/deploy@1c0ede0]: (non-production) Testing parallel ORES deployment, T181661
  • 19:16 ebernhardson@tin: Synchronized php-1.31.0-wmf.11/extensions/CirrusSearch/includes/Search/RescoreBuilders.php: SWAT: Add query string for running Cirrus MLR pre-deploy checks (duration: 00m 57s)
  • 18:56 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: Switch cebwiki, ruwiki and small projects to Kafka for htmlCacheUpdate - T182023 (duration: 00m 57s)
  • 18:55 ppchelko@tin: Finished deploy [cpjobqueue/deploy@e1075af]: Enable htmlCacheUpdate for ceb and ru wiki and small projects T182023 (duration: 00m 34s)
  • 18:54 ppchelko@tin: Started deploy [cpjobqueue/deploy@e1075af]: Enable htmlCacheUpdate for ceb and ru wiki and small projects T182023
  • 18:03 gehel@tin: Finished deploy [wdqs/wdqs@353b3cb]: wdqs: GUI and updater updates (duration: 01m 14s)
  • 18:02 gehel@tin: Started deploy [wdqs/wdqs@353b3cb]: wdqs: GUI and updater updates
  • 16:42 hashar: Restarting Jenkins
  • 15:36 marostegui: Deploy schema change on s2 master (db1054) - T174569
  • 14:51 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Disable the wgTranslateNumerals at hiwikiversity T182584 (duration: 00m 56s)
  • 14:46 niharika29@tin: Synchronized php-1.31.0-wmf.11/includes/specials/SpecialUndelete.php: Revert replacing textarea in Special:Undelete with OOUI T182398 (duration: 00m 57s)
  • 14:34 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Bureaucrats to grant and remove translationadmin rights; sysops to add and remove the same from themselves - T182492 (duration: 00m 56s)
  • 14:32 niharika29@tin: Synchronized wmf-config/interwiki.php: Update the Interwiki map - T182506 (duration: 00m 56s)
  • 14:25 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Create NS_PROJECT and NS_PROJECT_TALK alias for kowikisource (T182487) (duration: 00m 56s)
  • 13:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1060 - T174569 (duration: 04m 42s)
  • 12:39 _joe_: trying to get a full core dump from hhvm on mw1200
  • 11:12 _joe_: depooling mw1200 for investigation instead
  • 11:11 _joe_: restarting hhvm on mw1200, stuck in a kernel task
  • 11:08 jdrewniak@tin: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 45s)
  • 11:07 jdrewniak@tin: Synchronized portals/prod/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 44s)
  • 10:49 ema: cp4021: restart varnish-be due to mbox lag
  • 10:04 godog: upgrade grafana to 4.6.2 on labmon1001 - T182294
  • 10:00 jynus: stopping dbstore2001:s5 and dbstore1002 (s5) mysql replication in sync
  • 09:28 akosiaris: upload scap_3.7.4-1 to apt.wikimedia.org/jessie-wikimedia/main
  • 09:16 gehel: cleaning old cassandra dumps on maps-test2001 servers
  • 09:15 gehel: cleaning up old postgres logs on maps-test2001
  • 09:05 elukey: set notebook1002 as role::spare as prep step to reimage it to kafka1023
  • 09:03 jynus: dropping multiple leftover files from db1102
  • 08:52 marostegui: Stop replication in sync on db1034 and db1039 - T163190
  • 08:12 elukey: powercycle ganeti1008 - all vms stuck, console com2 showed a ton of printks without a clear indicator of the root cause
  • 07:49 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1034 - T182556 (duration: 00m 45s)
  • 07:44 _joe_: restarting hhvm on mw1189,mw1229,mw1235,mw1282,mw1285,mw1315,mw1316, all stuck with a kernel hang
  • 06:59 _joe_: restarted hhvm, nginx on mw1280, hanging kernel operations
  • 06:45 marostegui: Deploy schema change on s2 db1060 with replication enabled, this will generate some lag on s2 on labs - T174569
  • 06:45 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1060 - T174569 (duration: 00m 44s)
  • 06:22 marostegui: Compress s6 on db1096 - T178359
  • 06:21 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1096:3316 to compress InnoDB there - T178359 (duration: 00m 45s)
  • 02:43 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.11) (duration: 09m 21s)

2017-12-10

  • 20:33 elukey: execute restart-hhvm on mw1312 - hhvm stuck multiple times queueing requests
  • 20:01 elukey: ran kafka preferred-replica-election for the kafka analytics cluster (1012->1022) to re-add kafka1012 to the kafka brokers acting as partition leaders (will spread the load in a better way)

2017-12-09

  • 17:00 apergos: restarted hhvm on mw1276, the same old hang with the same old symptoms
  • 16:10 awight@tin: Finished deploy [ores/deploy@1c0ede0]: Reducing ORES Celery log verbosity (take 4\!) (duration: 03m 01s)
  • 16:07 awight@tin: Started deploy [ores/deploy@1c0ede0]: Reducing ORES Celery log verbosity (take 4\!)
  • 16:02 awight@tin: Finished deploy [ores/deploy@1c0ede0]: Reducing ORES Celery log verbosity (duration: 05m 58s)
  • 15:56 awight@tin: Started deploy [ores/deploy@1c0ede0]: Reducing ORES Celery log verbosity
  • 15:55 awight@tin: Finished deploy [ores/deploy@1c0ede0]: Reducing ORES Celery log verbosity (duration: 00m 17s)
  • 15:55 awight@tin: Started deploy [ores/deploy@1c0ede0]: Reducing ORES Celery log verbosity
  • 15:53 awight@tin: Finished deploy [ores/deploy@1c0ede0]: Reducing ORES Celery log verbosity (duration: 00m 31s)
  • 15:53 awight@tin: Started deploy [ores/deploy@1c0ede0]: Reducing ORES Celery log verbosity
  • 15:53 apergos: did same on scb1002,3,4
  • 15:48 awight: Making an emergency deployment to ORES logging config to reduce verbosity.
  • 15:45 apergos: on scb1001 moved daemon.log out of the way, did "service rsyslog rotate", saved the last 5000 entries for use by ores team, removed the log
  • 11:44 apergos: that server list: mw1278, 1277, 1226, 1234, 1230
  • 11:42 apergos: restarted hhvm on api servers after lockup
  • 11:19 legoktm@tin: Synchronized wmf-config/InitialiseSettings.php: Disable ORES in fawiki - T182354 (duration: 00m 45s)
  • 00:11 Jamesofur: removed 2FA from EVinente after verification T182373

2017-12-08

  • 23:23 hashar: force ran puppet on contint2001
  • 22:15 madhuvishy: Kicked off rsync of /data/xmldatadumps/public to labstore1006 & 7
  • 22:05 smalyshev@tin: Finished deploy [wdqs/wdqs@353b3cb]: temporary fix for T182464, better fix coming soon (duration: 05m 55s)
  • 21:59 smalyshev@tin: Started deploy [wdqs/wdqs@353b3cb]: temporary fix for T182464, better fix coming soon
  • 20:22 aaron@tin: Synchronized php-1.31.0-wmf.11/includes/Setup.php: a319c3e7ab61 - disable cpPosTime injection (duration: 00m 45s)
  • 18:00 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: Disable GlobalBlocking on fishbowl wikis (duration: 00m 45s)
  • 16:23 urandom: starting cassandra, restbase1010 - T178177
  • 16:22 urandom: disabling smart path, restbase1010, arrays 'b'...'e' - T178177
  • 16:20 urandom: disabling smart path, restbase1010, array 'a' (canary) - T178177
  • 16:15 urandom: shutting down cassandra, restbase1010 - T178177
  • 15:35 marostegui: Fix dbstore1002 s5 replication
  • 15:28 gehel@tin: Finished deploy [tilerator/deploy@29d633e]: testing new tilerator packaging on maps-test2003 (duration: 00m 03s)
  • 15:28 gehel@tin: Started deploy [tilerator/deploy@29d633e]: testing new tilerator packaging on maps-test2003
  • 15:08 gehel@tin: Finished deploy [tilerator/deploy@29d633e]: testing new tilerator packaging on maps-test2003 (duration: 02m 08s)
  • 15:06 gehel@tin: Started deploy [tilerator/deploy@29d633e]: testing new tilerator packaging on maps-test2003
  • 15:05 gehel@tin: Finished deploy [tilerator/deploy@29d633e]: testing new tilerator packaging on maps-test2003 (duration: 00m 42s)
  • 15:05 gehel@tin: Started deploy [tilerator/deploy@29d633e]: testing new tilerator packaging on maps-test2003
  • 14:39 gehel@tin: Finished deploy [tilerator/deploy@e52ea1d]: testing new tilerator packaging on maps-test2003 (duration: 02m 34s)
  • 14:36 gehel@tin: Started deploy [tilerator/deploy@e52ea1d]: testing new tilerator packaging on maps-test2003
  • 11:45 elukey: updated prometheus-druid-exporter on druid* to 0.6
  • 11:39 elukey: upload prometheus-druid-exporter 0.6 to stretch/jessie wikimedia
  • 06:52 marostegui: Fix labsdb1004 replication broken
  • 06:43 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1099:3311 - T178359 (duration: 00m 55s)
  • 04:56 bd808: scholarships: updated db schema for 2018 cycle (T181072)
  • 00:04 bblack: cp4026 - restart varnish backend, mailbox lag

2017-12-07

  • 23:59 tgr@tin: Synchronized wmf-config/InitialiseSettings.php: T181107 enable ReadingLists on all wikis (duration: 00m 46s)
  • 23:54 tgr: ran mwscript extensions/ReadingLists/maintenance/populateProjectsFromSiteMatrix.php --wiki=testwiki
  • 23:47 tgr@tin: Finished scap: T181107 deploy ReadingLists to testwiki (duration: 24m 44s)
  • 23:22 tgr@tin: Started scap: T181107 deploy ReadingLists to testwiki
  • 23:04 tgr: ran mwscript ../../../home/tgr/sql.php --wiki=mediawikiwiki --cluster extension1 --wikidb wikishared /srv/mediawiki-staging/php-1.31.0-wmf.11/extensions/ReadingLists/sql/readinglists.sql
  • 22:33 tgr@tin: Synchronized php-1.31.0-wmf.11/extensions/ReadingLists/: ReadingLists/wmf.11: catching up with master (duration: 00m 45s)
  • 22:29 tgr@tin: Synchronized php-1.31.0-wmf.10/extensions/ReadingLists/: ReadingLists/wmf.10: catching up with master (duration: 00m 46s)
  • 20:39 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.11
  • 20:35 elukey: restart hhvm on mw1235 - hhvm-dump-debug hanging out, not stacktrace available
  • 20:31 elukey: restart hhvm on mw1281 - hhvm stuck (hhvm-dump-debug timing out)
  • 20:22 joal@tin: Finished deploy [analytics/refinery@53bd630]: Regular analytics deploy - Long time no see, deployment :) - post-patch-2 (hopefully last for tonight) (duration: 05m 21s)
  • 20:18 herron: re-pooling eqiad puppet 4 masters via dns puppet.eqiad.wmnet puppet.wikimedia.org
  • 20:17 joal@tin: Started deploy [analytics/refinery@53bd630]: Regular analytics deploy - Long time no see, deployment :) - post-patch-2 (hopefully last for tonight)
  • 20:14 joal@tin: Finished deploy [analytics/refinery@3e52903]: Regular analytics deploy - Long time no see, deployment :) - post-patch (duration: 01m 20s)
  • 20:12 joal@tin: Started deploy [analytics/refinery@3e52903]: Regular analytics deploy - Long time no see, deployment :) - post-patch
  • 19:52 joal@tin: Finished deploy [analytics/refinery@bd9c6cc]: Regualr analytics deploy - Long time no see, deployment :) (duration: 12m 15s)
  • 19:40 joal@tin: Started deploy [analytics/refinery@bd9c6cc]: Regualr analytics deploy - Long time no see, deployment :)
  • 19:35 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Start description usage tracking for commonswiki T106287 (duration: 00m 48s)
  • 19:29 thcipriani@tin: Synchronized php-1.31.0-wmf.11/includes/specialpage/ChangesListSpecialPage.php: SWAT: WLFilters: Correctly check if RCFilters should be enabled on WL T182318 (duration: 00m 48s)
  • 19:24 thcipriani@tin: Synchronized wmf-config/Wikibase-production.php: SWAT: Remove obsolete WikibaseQualityConstraints settings (duration: 00m 48s)
  • 19:14 ppchelko@tin: Finished deploy [changeprop/deploy@3c4f51d]: Long awaited deploy: generic optimizations, gc metric, delay reporting (duration: 01m 15s)
  • 19:13 ppchelko@tin: Started deploy [changeprop/deploy@3c4f51d]: Long awaited deploy: generic optimizations, gc metric, delay reporting
  • 19:12 thcipriani@tin: Synchronized wmf-config/throttle.php: SWAT: Rm all past throttle overrides in throttle.php (duration: 00m 48s)
  • 18:06 mholloway-shell@tin: Finished deploy [mobileapps/deploy@2fa32ed]: Update mobileapps to 71f581c (duration: 05m 36s)
  • 18:00 mholloway-shell@tin: Started deploy [mobileapps/deploy@2fa32ed]: Update mobileapps to 71f581c
  • 18:00 herron: re-pooling eqiad puppet 4 masters as puppet.ulsfo.wnet
  • 17:33 demon@tin: Synchronized php-1.31.0-wmf.11/extensions/VisualEditor/lib/ve: Ief480487, Deskana made me do it (duration: 00m 49s)
  • 17:25 milimetric@tin: Finished deploy [analytics/aqs/deploy@4ec13b4]: (no justification provided) (duration: 07m 28s)
  • 17:25 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1314.eqiad.wmnet
  • 17:25 herron: puppetmaster1001 upgraded to puppet 4. re-enabling puppet agents across the fleet
  • 17:18 milimetric@tin: Started deploy [analytics/aqs/deploy@4ec13b4]: (no justification provided)
  • 17:13 herron: upgrading puppetmaster1001 to puppet 4
  • 17:08 herron: temporarily disabling all puppet agents during puppetmaster1001 (puppet ca) upgrade to puppet 4
  • 16:14 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1099:3311 weight - T178359 (duration: 00m 48s)
  • 16:13 herron: upgrading puppetmaster1002 to puppet 4
  • 16:03 moritzm: uploaded openssl 1.0.2n for jessie-wikimedia to apt.wikimedia.org
  • 16:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1093 back as main traffic in s6 - T178359 (duration: 00m 48s)
  • 15:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1099:3311 weight - T178359 (duration: 00m 48s)
  • 15:42 elukey: hhvm-dump-debug for mw1314 saved to /tmp/hhvm.17991.bt.
  • 15:30 elukey@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw1314.eqiad.wmnet
  • 15:24 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1099:3311 weight - T178359 (duration: 00m 48s)
  • 15:19 otto@tin: Finished deploy [analytics/superset/deploy@f0f5adf]: initial deployment (duration: 00m 02s)
  • 15:18 otto@tin: Started deploy [analytics/superset/deploy@f0f5adf]: initial deployment
  • 14:55 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1099:3311 with low weight - T178359 (duration: 00m 47s)
  • 14:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully repool db1074 (duration: 00m 47s)
  • 14:09 zeljkof: EU SWAT finished
  • 14:08 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: alswiki: Set wgRestrictDisplayTitle = false (T182154) (duration: 00m 49s)
  • 13:39 moritzm: reimaging mw2246 to stretch
  • 12:54 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw2152.codfw.wmnet
  • 12:32 addshore@tin: Synchronized wmf-config/Wikibase.php: Move Wikibase dispatchingLockManager to InitialiseSettings PT 2/2 (duration: 00m 48s)
  • 12:31 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Move Wikibase dispatchingLockManager to InitialiseSettings PT 1/2 (duration: 00m 48s)
  • 11:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase API traffic for db1074 (duration: 00m 48s)
  • 11:41 moritzm: reimaging mw2152 to stretch
  • 11:35 marostegui: Compress s8 on db1099 - T178359
  • 11:21 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase API traffic for db1074 (duration: 00m 48s)
  • 11:11 mobrovac@tin: Finished deploy [restbase/deploy@097ba7d]: Add CORS headers to erroneous responses as well - T182103 (duration: 05m 24s)
  • 11:05 mobrovac@tin: Started deploy [restbase/deploy@097ba7d]: Add CORS headers to erroneous responses as well - T182103
  • 10:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1074 with low weight (duration: 00m 48s)
  • 10:50 elukey: powercycle analytics1003 - no serial console, ssh stuck in System is booting up. See pam_nologin(8)
  • 10:28 _joe_: depooling mw1283 for further investigation
  • 10:14 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1098:3316 db1098:3317 - T178359 (duration: 00m 51s)
  • 10:12 elukey: reboot analytics1003 for kernel+jvm updates - T179943
  • 10:01 gehel: upgrade of ELK stack on logstash100* completed - Kibana was unavailable for longer than expected - T178412
  • 09:48 hashar: CI: removed Wikidata from configuration, replaced by Wikibase. wmf/* and REL branches are going to be broken though | https://gerrit.wikimedia.org/r/395704 | T181838
  • 09:46 akosiaris: silence ganeti1006 on icinga T181121
  • 09:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1098:3316 db1098:3317 - T178359 (duration: 00m 52s)
  • 09:28 marostegui: Upgrade MySQL and kernel on db1074
  • 09:22 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=logstash1009.eqiad.wmnet
  • 09:19 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=logstash1009.eqiad.wmnet
  • 09:18 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=logstash1008.eqiad.wmnet
  • 09:14 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=logstash1008.eqiad.wmnet
  • 09:13 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=logstash1007.eqiad.wmnet
  • 09:08 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=logstash1007.eqiad.wmnet
  • 09:05 gehel@tin: Finished deploy [logstash/plugins@b13d2fa]: (no justification provided) (duration: 00m 02s)
  • 09:05 gehel@tin: Started deploy [logstash/plugins@b13d2fa]: (no justification provided)
  • 09:00 gehel: upgrading ELK stack on logstash100* - some log messages might be lost during the upgrade - T178412
  • 08:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1098:3316 db1098:3317 - T178359 (duration: 00m 48s)
  • 08:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1098:3316 db1098:3317 - T178359 (duration: 00m 48s)
  • 08:28 elukey: install prometheus-druid-exporter 0.5 on druid*
  • 08:26 elukey: upload prometheus-druid-exporter 0.5-1 to jessie/stretch-wikimedia
  • 07:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Slowly pool db1098:3316 db1098:3317 - T178359 (duration: 00m 47s)
  • 07:18 marostegui@tin: Synchronized wmf-config/db-codfw.php: Slowly pool db1098:3316 db1098:3317 - T178359 (duration: 00m 48s)
  • 06:45 marostegui: Stop replication on db1099:3311 to reimport: change_tag, tag_summary, user and watchlist tables and recompress again - T178359
  • 06:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1074 - T174569 (duration: 00m 48s)
  • 06:40 marostegui: Deploy schema change on db1074 (s2) - T174569
  • 03:00 catrope@tin: Synchronized php-1.31.0-wmf.11/resources/src/mediawiki.rcfilters/: T182268 (duration: 00m 57s)
  • 02:29 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.10) (duration: 08m 57s)

2017-12-06

  • 23:45 eileen: update CiviCRM from 85263f6 to 2db9c76 (improve org merges)
  • 23:12 ppchelko@tin: Started restart [changeprop/deploy@065a06e]: (no justification provided)
  • 23:04 eileen: update process-control to b1cd515 (reenable bounce processing, enable dedupe on orgs)
  • 22:58 ppchelko@tin: Finished deploy [cpjobqueue/deploy@761524f]: Separate delay and totaldelay metrics T182216 (duration: 00m 31s)
  • 22:57 ppchelko@tin: Started deploy [cpjobqueue/deploy@761524f]: Separate delay and totaldelay metrics T182216
  • 22:55 mutante: stat1003 - re-enabled puppet after putting role::spare on it (T175150)
  • 22:48 mutante: stat1003 - this host was kind of invisible (not in site, not in icinga) but still up, re-enabling puppet after re-adding it to site
  • 22:03 demon@tin: Synchronized php-1.31.0-wmf.11/extensions/WikidataPageBanner/includes/WikidataPageBanner.hooks.php: unbreak (duration: 00m 48s)
  • 22:00 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.11, again
  • 21:51 hoo@tin: Synchronized php-1.31.0-wmf.10/extensions/Wikibase/lib/includes/Changes/EntityChange.php: Make EntityChange truly forward compatible with compact diffs (T182243) (duration: 00m 48s)
  • 21:43 arlolra: Updated Parsoid to 01c1fc3 (T178253, T61840, T180930, T179259)
  • 21:34 arlolra@tin: Finished deploy [parsoid/deploy@dfcc622]: Updating Parsoid to 01c1fc3 (duration: 09m 45s)
  • 21:24 arlolra@tin: Started deploy [parsoid/deploy@dfcc622]: Updating Parsoid to 01c1fc3
  • 20:45 marostegui: Add 400G to labsdb1003 /srv partition
  • 20:15 hoo: Ran "scap pull" on snapshot1001 after T177486 related tests
  • 20:10 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: partial rollback -- wikidata errors
  • 20:07 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.11
  • 20:05 demon@tin: Synchronized php: symlink bump (duration: 00m 47s)
  • 19:32 hoo@tin: Synchronized php-1.31.0-wmf.11/extensions/Wikibase/repo/maintenance/dispatchChanges.php: dispatch: track how long client selecting takes (duration: 00m 48s)
  • 19:24 ppchelko@tin: Finished deploy [cpjobqueue/deploy@8c66189]: Deduplicate based on event sha1 as well as on id, take 2 (duration: 00m 38s)
  • 19:24 ppchelko@tin: Started deploy [cpjobqueue/deploy@8c66189]: Deduplicate based on event sha1 as well as on id, take 2
  • 19:23 hoo@tin: Synchronized php-1.31.0-wmf.10/extensions/Wikibase/repo/maintenance/dispatchChanges.php: dispatch: track how long client selecting takes (duration: 00m 48s)
  • 19:22 ppchelko@tin: Started deploy [cpjobqueue/deploy@8c66189]: Deduplicate based on event sha1 as well as on id, take 2
  • 19:22 ppchelko@tin: Finished deploy [cpjobqueue/deploy@8c66189]: Deduplicate based on event sha1 as well as on id (duration: 00m 11s)
  • 19:22 ppchelko@tin: Started deploy [cpjobqueue/deploy@8c66189]: Deduplicate based on event sha1 as well as on id
  • 19:20 ppchelko@tin: Finished deploy [cpjobqueue/deploy@df72b34]: Deduplicate based on event sha1 as well as on id (duration: 00m 09s)
  • 19:19 ppchelko@tin: Started deploy [cpjobqueue/deploy@df72b34]: Deduplicate based on event sha1 as well as on id
  • 18:44 volans: stopped irc echo on tegmen, re-enabled puppet and run it (it was disabled by a run-no-puppet sync_icinga_state)
  • 18:32 mobrovac@tin: Finished deploy [zotero/translators@3044b3a]: Update translators to 092c7bc - T178596 (duration: 00m 07s)
  • 18:32 mobrovac@tin: Started deploy [zotero/translators@3044b3a]: Update translators to 092c7bc - T178596
  • 18:16 herron: upgrading rhodium to puppet 4
  • 18:13 mobrovac@tin: Synchronized wmf-config/jobqueue.php: Switch htmlCacheUpdate jobs for wiktionaries to EventBus, file 2/2 - T182023 (duration: 00m 48s)
  • 18:12 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: Switch htmlCacheUpdate jobs for wiktionaries to EventBus, file 1/2 - T182023 (duration: 00m 48s)
  • 18:05 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add https://studiezaal.nijmegen.nl to $wgCopyUploadsDomains (T181713) (duration: 00m 49s)
  • 17:51 ppchelko@tin: Finished deploy [cpjobqueue/deploy@df72b34]: Switch htmlCacheUpdate for wiktionaries, attempt 2 T182023 (duration: 00m 32s)
  • 17:50 ppchelko@tin: Started deploy [cpjobqueue/deploy@df72b34]: Switch htmlCacheUpdate for wiktionaries, attempt 2 T182023
  • 17:44 ppchelko@tin: Finished deploy [cpjobqueue/deploy@3281df1]: Switch htmlCacheUpdate for wiktionaries T182023 (duration: 02m 57s)
  • 17:42 ppchelko@tin: Started deploy [cpjobqueue/deploy@3281df1]: Switch htmlCacheUpdate for wiktionaries T182023
  • 17:36 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1260.eqiad.wmnet
  • 17:05 hoo: Started Wikidata RDF dumps (sudo -b -u datasets bash -c 'dumpwikidatardf.sh all ttl; dumpwikidatardf.sh truthy nt') on snapshot1007
  • 16:33 anomie@terbium: Finished cleanupUsersWithNoId.php on mediawikiwiki
  • 16:29 anomie@terbium: Running cleanupUsersWithNoId.php for mediawikiwiki, see T181731
  • 16:28 anomie@terbium: Finished cleanupUsersWithNoId.php on testwikidatawiki
  • 16:27 anomie@terbium: Running cleanupUsersWithNoId.php for testwikidatawiki, see T181731
  • 16:26 anomie@terbium: Finished cleanupUsersWithNoId.php on test2wiki
  • 16:14 anomie@terbium: Running cleanupUsersWithNoId.php for test2wiki, see T181731
  • 16:11 anomie@terbium: Finished cleanupUsersWithNoId.php on testwiki
  • 16:03 anomie@terbium: Running cleanupUsersWithNoId.php for testwiki, see T181731
  • 15:13 akosiaris: upload kubernetes_1.7.10-1_amd64 on apt.wikimedia.org/stretch-wikimedia/main T181489
  • 14:59 addshore: swat done
  • 14:57 addshore@tin: Synchronized php-1.31.0-wmf.11/extensions/Wikibase/repo/includes/ChangeDispatcher.php: SWAT Tracking within ChangeDispatcher::getPendingChanges (duration: 00m 49s)
  • 14:53 addshore@tin: Synchronized php-1.31.0-wmf.10/extensions/Wikibase/repo/includes/ChangeDispatcher.php: SWAT Tracking within ChangeDispatcher::getPendingChanges (duration: 00m 49s)
  • 14:33 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add https://studiezaal.nijmegen.nl to $wgCopyUploadsDomains (T181713) (duration: 00m 47s)
  • 14:22 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add Portal namespace for mwl.wikipedia (T180052) (duration: 00m 48s)
  • 14:13 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT T180476 wmgUseNewWikiDiff2Extension true for dewiki (duration: 00m 48s)
  • 14:08 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT T181498 T181329 Enable AdvancedSearch on fawiki and huwiki (duration: 00m 50s)
  • 13:40 moritzm: reimaging mw1260 to stretch
  • 10:55 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw2118.codfw.wmnet
  • 10:55 moritzm: reimaging mw2119 to stretch
  • 10:45 mobrovac@tin: Finished deploy [restbase/deploy@b1d7c82]: Use Cass3 for revisions, deprecate trending-edits, fix CX end point - T179421 T180384 T173801 (duration: 06m 02s)
  • 10:39 mobrovac@tin: Started deploy [restbase/deploy@b1d7c82]: Use Cass3 for revisions, deprecate trending-edits, fix CX end point - T179421 T180384 T173801
  • 09:34 gehel: shuttting down logstash / elasticsearch on logstash100[123] in preparation for decommission -T175830
  • 08:05 moritzm: reimaging mw2118 to stretch (now for real, yesterday's reimage logged to SAL was interrupted)
  • 07:33 bblack: cp4024 - backend restart, mailbox lag
  • 02:39 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.10) (duration: 08m 53s)
  • 01:01 eileen: update process-control to 55529ea - threshold for $500+ dedupe was too low
  • 00:36 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable description usage tracking for all wikis except commons T106287 (duration: 00m 48s)
  • 00:27 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable RemexHTML on wikis with zero high priority linter errors T182042 (duration: 00m 51s)

2017-12-05

  • 23:07 eileen: update civicrm from d2c70d2 to 85263f6 (Major gifts address - choose latest)
  • 22:29 eileen: update process_control to 3522186 renable jobs to fetch recipient data
  • 22:13 mutante: restbase1010 failed at reboot with P6431 , after a cold start (power off, power on) it came back though :) (T178177 T141756)
  • 22:00 mutante: restbase1010 - rebooting for firmware upgrade
  • 21:58 mutante: restbase1010 - upgraded HP firmware (Flashing Smart Array P440ar in Slot 0 [ 3.56 -> 6.06 ]) T141756 T178177
  • 21:56 urandom: draining cassandra instances, restbase1010 - T178177
  • 21:52 mutante: restbase1010 - upgrading firmware - Flashing Smart Array P440ar in Slot 0 [ 3.56 -> 6.06 ]
  • 21:31 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 shenanigans / wmf.11
  • 21:12 demon@tin: Finished scap: bootstrap wmf.11 (duration: 51m 01s)
  • 20:50 twentyafterfour: phabricator: running `sudo bin/garbage collect --collector search.ferret.ngram`
  • 20:29 twentyafterfour: phabricator: running `sudo bin/search ngrams --threshold 0.2`
  • 20:21 demon@tin: Started scap: bootstrap wmf.11
  • 19:50 mutante: forcing puppet ron on ores eqiad to reduce number of celery workers used for stress test
  • 19:37 ppchelko@tin: Finished deploy [eventlogging/eventbus@6ca0372]: Make the kafka async deliver callback thread-safe. T180017 (duration: 01m 29s)
  • 19:36 ppchelko@tin: Started deploy [eventlogging/eventbus@6ca0372]: Make the kafka async deliver callback thread-safe. T180017
  • 19:20 gehel: moving eventlogging collection by logstash from logstash1003 to logstash1007, no messages **should** be lost - T175830
  • 18:24 ppchelko@tin: Finished deploy [eventlogging/eventbus@6ca0372]: Make the kafka async deliver callback thread-safe. Limited to kafka1001. T180017 (duration: 00m 14s)
  • 18:24 ppchelko@tin: Started deploy [eventlogging/eventbus@6ca0372]: Make the kafka async deliver callback thread-safe. Limited to kafka1001. T180017
  • 17:27 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw2118.codfw.wmnet
  • 15:56 awight: beginning stress test on ores* (non-production)
  • 15:54 awight@tin: Finished deploy [ores/deploy@6baed71]: (non-production) Test ORES deployment to ores100[1-2] (duration: 01m 01s)
  • 15:53 awight@tin: Started deploy [ores/deploy@6baed71]: (non-production) Test ORES deployment to ores100[1-2]
  • 15:15 ottomata: restarrting kafka-jumbo brokers, applying SSL (downtime scheduled)
  • 15:03 urandom: bootstrapping cassandra, restbase1014-c - T179422
  • 14:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully repool db1076 (duration: 00m 43s)
  • 14:41 zeljkof: EU SWAT finished
  • 14:40 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Add category collation for sewiki" (T181503) (duration: 00m 44s)
  • 14:15 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable RemexHTML on itwiki and dewiki (T181188 T181190) (duration: 00m 43s)
  • 13:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1076 weight (duration: 00m 44s)
  • 13:14 moritzm: reimaging mw2118/mw2119 (video scalers) to stretch
  • 12:21 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully repool db1034 (duration: 00m 43s)
  • 12:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1076 with low weight (duration: 00m 44s)
  • 11:58 marostegui: Upgrade MariaDB and kernel on db1076
  • 11:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1034 - T178359! (duration: 00m 43s)
  • 10:53 moritzm: rebooting meitnerium/archiva.wikimedia.org for update to 4.9.51
  • 10:46 moritzm: rebooting einsteinium (icinga.wikimedia.org) for update to 4.9,51
  • 10:45 elukey: reboot druid1003 for kernel+jvm updates - T179943
  • 10:33 addshore@tin: Synchronized wmf-config/extension-list: extension-list extension.json entrypoint for ArticlePlaceholder (duration: 00m 43s)
  • 10:33 moritzm: rebooting tegmen for update to 4.9,51
  • 10:07 jynus: stopping dbstore1002 (s5) and dbstore2001 (s5) for maintenance
  • 10:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1034 - T178359! (duration: 00m 43s)
  • 10:00 ema: cp4021: restart varnish-be due to mbox lag/fetch failures
  • 09:49 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1034 with low weight - T178359! (duration: 00m 43s)
  • 09:42 elukey: reboot analytics100[12] for kernel+jvm updates (Hadoop Master nodes) - T179943
  • 09:39 godog: bootstrap restbase1014-b - T179422
  • 09:20 marostegui: Optimize s7 on db1098 - T178359
  • 09:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1076 - T174569 (duration: 00m 43s)
  • 09:01 marostegui: Deploy schema change on db1076 (s2) - T174569
  • 08:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1090 - T174569 (duration: 00m 44s)
  • 08:54 moritzm: enabling test production traffic for mw1259 (stretch-based video scaler)
  • 08:24 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1259.eqiad.wmnet
  • 08:08 moritzm: installing python updates on trusty
  • 07:27 kartik@tin: Finished deploy [cxserver/deploy@4b74f03]: Update cxserver to 1693bcf (duration: 03m 22s)
  • 07:23 kartik@tin: Started deploy [cxserver/deploy@4b74f03]: Update cxserver to 1693bcf
  • 06:58 marostegui: Stop MySQL on db1034 to clone db1098:3317 - T178359
  • 06:55 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1034 - T178359! (duration: 00m 43s)
  • 06:48 marostegui: Fix dbstore1002 replication
  • 06:20 marostegui: Deploy schema change on db1090 (s2) - T174569
  • 06:20 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1090 - T174569 (duration: 00m 43s)
  • 06:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1103:3312 - T174569 (duration: 00m 44s)
  • 02:29 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.10) (duration: 06m 36s)
  • 00:29 mutante: bast4001 - removing ganglia aggregators, package, config...

2017-12-04

  • 23:44 mutante: bast3002 - killall -u ganglia to kill all aggregator procs, apt-get remove --purge ganglia-monitor, rm -rf /etc/ganglia, rm -rf /usr/lib/ganglia, apt-get autoremove
  • 23:28 anomie@tin: Synchronized wmf-config/InitialiseSettings.php: Revert wgCommentTableSchemaMigrationStage change, breaks too much stuff (duration: 00m 44s)
  • 22:08 brion: brion running requeueTranscodes.php on terbium to batch-run .mp3 output on Commons (T181749)
  • 21:27 ppchelko@tin: Finished deploy [cpjobqueue/deploy@c8bea2e]: (no justification provided) (duration: 00m 42s)
  • 21:26 ppchelko@tin: Started deploy [cpjobqueue/deploy@c8bea2e]: (no justification provided)
  • 21:26 awight@tin: Finished deploy [ores/deploy@6baed71]: Update ORES to 6baed71 (duration: 12m 54s)
  • 21:25 eileen: re-enable dedupe job process control commit 461f482
  • 21:13 awight@tin: Started deploy [ores/deploy@6baed71]: Update ORES to 6baed71
  • 21:07 gehel@tin: Finished deploy [kartotherian/deploy@6e223df]: testing new kartotherian packaging on maps-test2003 (duration: 00m 20s)
  • 21:07 gehel@tin: Started deploy [kartotherian/deploy@6e223df]: testing new kartotherian packaging on maps-test2003
  • 20:07 anomie@tin: Synchronized wmf-config/InitialiseSettings.php: Revert wgCommentTableSchemaMigrationStage change, breaks too much stuff (duration: 00m 43s)
  • 19:31 legoktm@tin: Synchronized wmf-config/InitialiseSettings.php: Set wgCommentTableSchemaMigrationStage = MIGRATION_WRITE_BOTH on test wikis (T166733) (duration: 00m 43s)
  • 19:18 gehel@tin: Finished deploy [kartotherian/deploy@e166d87]: testing new kartotherian packaging on maps-test2003 (duration: 00m 03s)
  • 19:18 gehel@tin: Started deploy [kartotherian/deploy@e166d87]: testing new kartotherian packaging on maps-test2003
  • 19:17 legoktm@tin: Synchronized wmf-config/InitialiseSettings.php: Set $wgRestrictionMethod = 'firejail'; everywhere (T173370) (duration: 00m 43s)
  • 19:11 legoktm@tin: Synchronized static/images/mobile/copyright/: Add converted copyright svg images as png files - https://gerrit.wikimedia.org/r/#/c/394820/ (duration: 00m 43s)
  • 19:06 legoktm@tin: Synchronized wmf-config/InitialiseSettings-labs.php: beta: Add ORES filter thresholds for simplewiki (duration: 00m 43s)
  • 18:46 hoo: Started dumpwikidatajson.sh on snapshot1007 (T181385)
  • 18:40 hoo: Ran scap pull on snapshot1001 (T181385)
  • 18:35 hoo@tin: Synchronized php-1.31.0-wmf.10/includes/objectcache/ObjectCache.php: Only send statsd data for WAN cache in non-CLI mode (T181385) (duration: 00m 44s)
  • 18:28 godog: bootstrap restbase1014-a - T179422
  • 18:04 gehel@tin: Finished deploy [wdqs/wdqs@2873745]: wdqs GUI update (duration: 02m 10s)
  • 18:02 gehel@tin: Started deploy [wdqs/wdqs@2873745]: wdqs GUI update
  • 17:41 demon@tin: Synchronized php-1.31.0-wmf.10/extensions/CirrusSearch/maintenance/: fix some deprecated spam (duration: 00m 44s)
  • 17:40 ejegg: disabled CiviCRM bounce processing again
  • 17:12 demon@tin: Pruned MediaWiki: 1.31.0-wmf.6 (duration: 02m 29s)
  • 17:08 demon@tin: Pruned MediaWiki: 1.31.0-wmf.5 (duration: 02m 58s)
  • 16:54 demon@tin: Synchronized docroot/noc/conf/dblists: double checking symlink move (duration: 00m 44s)
  • 16:44 ejegg: re-enabled CiviCRM bounced mail processing
  • 16:41 marostegui: Deploy schema change on db1053 (s2) - T174569
  • 16:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1103:3312 - T174569 (duration: 00m 45s)
  • 16:39 demon@tin: Finished scap: docroot/noc/conf/ Adding new dblist symlink (duration: 02m 36s)
  • 16:39 marostegui: Deploy schema change on db1103:3312 - T174569
  • 16:37 demon@tin: Started scap: docroot/noc/conf/ Adding new dblist symlink
  • 16:37 demon@tin: scap aborted: docroot/noc/conf/dblists (duration: 00m 02s)
  • 16:37 demon@tin: Started scap: docroot/noc/conf/dblists
  • 16:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1105:3312 - T174569 (duration: 00m 45s)
  • 16:32 demon@tin: Finished scap: docroot/noc/conf/ drop some dangling symlinks (duration: 03m 53s)
  • 16:28 demon@tin: Started scap: docroot/noc/conf/ drop some dangling symlinks
  • 16:11 herron: re-enabling puppet agents in eqiad
  • 16:07 demon@tin: Finished scap: Revert "Special:Preferences: Use OOjs UI" and follow-ups (duration: 21m 19s)
  • 16:05 herron: re-enabling puppet agents in codfw
  • 15:46 demon@tin: Started scap: Revert "Special:Preferences: Use OOjs UI" and follow-ups
  • 15:31 herron: temporarily disabling puppet agents in eqiad and codfw while puppetdb catches up with command queue
  • 15:29 herron: disabling ircecho temporarily
  • 15:24 _joe_: restarting puppetdb on nihal, will cause puppet failures
  • 15:23 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1259.eqiad.wmnet
  • 15:18 demon@tin: Synchronized wmf-config/CommonSettings-labs.php: no-op (duration: 00m 45s)
  • 15:01 godog: reimage restbase1014 - T179422
  • 14:51 herron: cutting over all production puppet agents to codfw puppet 4 masters via dns
  • 14:50 Amir1: deployed backward compatibility of entity compact diff transmit T113468
  • 14:49 ladsgroup@tin: Synchronized php-1.31.0-wmf.10/extensions/Wikibase: (no justification provided) (duration: 01m 41s)
  • 14:37 gehel@tin: Finished deploy [kartotherian/deploy@e166d87]: dummy kartotherian deployment to test udp2log config change - T175242 (duration: 00m 03s)
  • 14:37 gehel@tin: Started deploy [kartotherian/deploy@e166d87]: dummy kartotherian deployment to test udp2log config change - T175242
  • 14:30 elukey: reboot druid100[23] for kernel updates
  • 14:28 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Localize sitename and meta NS for wawiktionary (T181782) (duration: 00m 46s)
  • 14:01 elukey: reboot analytics106* (hadoop worker nodes) for kernel+jvm updates - T179943
  • 13:24 paravoid: upgrading bast2001 to stretch
  • 13:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1105:3312 - T174569 (duration: 00m 45s)
  • 13:20 marostegui: Deploy schema change on db1105:3312 (s2) - T174569
  • 13:05 marostegui: Compress s6 on db1098 - T178359
  • 12:44 gehel@tin: Finished deploy [kartotherian/deploy@e166d87]: testing new kartotherian packaging on maps-test2003 (duration: 00m 22s)
  • 12:43 gehel@tin: Started deploy [kartotherian/deploy@e166d87]: testing new kartotherian packaging on maps-test2003
  • 11:06 jdrewniak@tin: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 45s)
  • 11:05 jdrewniak@tin: Synchronized portals/prod/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 45s)
  • 10:37 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2085 (duration: 00m 43s)
  • 10:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1096:3316 - T178359wq! (duration: 00m 45s)
  • 09:55 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1099:3318 from s5 (duration: 00m 44s)
  • 09:51 godog: bootstrap restbase1012-c - T179422
  • 09:32 godog: clear erroneous table metrics from graphite1003 / graphite2002 - T181689
  • 09:24 elukey: reboot analytics105* (hadoop worker nodes) for kernel+jvm updates - T179943
  • 09:19 jynus: rebooting mariadb at labsdb1005
  • 09:12 moritzm: reimaging mw1259 (video scaler) to stretch, will be kept disabled initially (some controlled live tests following)
  • 08:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1096:3315 and 3316 - T178359 (duration: 00m 45s)
  • 08:45 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1096:3316 - T178359 (duration: 00m 45s)
  • 08:44 moritzm: updating tor on radium to 0.3.1.9
  • 08:41 moritzm: updating tor packages to 0.3.1.9
  • 08:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1096:3315 and pool db1096:3316 - T178359 (duration: 00m 45s)
  • 08:12 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db1096:3315 - T178359 (duration: 00m 44s)
  • 08:11 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1096:3315 - T178359 (duration: 00m 45s)
  • 07:53 moritzm: installing curl security updates
  • 07:17 marostegui: Compress s1 on db1099 - T178359
  • 07:08 marostegui: Stop MySQL on db1044 as it will be decommissioned - T181696
  • 07:05 _joe_: playing with puppetdb status for ores2003 (deactivating/reactivating node)
  • 06:40 marostegui: Stop MySQL on db1098 to clone db1096.s6 - T178359
  • 06:39 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1044 from config as it will be decommissioned - T181696 (duration: 00m 45s)
  • 06:38 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1044 from config as it will be decommissioned - T181696 (duration: 00m 45s)
  • 06:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1098 - T178359 (duration: 00m 46s)
  • 06:21 marostegui: Deploy alter table on s3 master (db1075) without replication - T174569
  • 02:32 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.10) (duration: 06m 28s)

2017-12-03

  • 15:33 ejegg: disabled CiviCRM bounce processing job
  • 12:17 akosiaris: empty ganeti1006, it had issues this morning per T181121
  • 12:06 marostegui: Fix dbstore1002 replication
  • 07:44 akosiaris: ran puppet on conf2002, etcdmirror-conftool-eqiad-wmnet got started again
  • 05:11 andrewbogott: deleting files on labsdb1003 /srv/tmp older than 30 days
  • 03:57 no_justification: gerrit2001: icinga is flapping on the gerrit process/systemd check, but this is kind of known (not sure why it's doing this all of a sudden). It's not letting me acknowledge it, but it's fine/harmless. Cf T176532

2017-12-02

  • 17:55 marostegui: Reboot db1096.s5 to pick up the correct innodb_buffer_pool size after finishing compressing s5 - T178359
  • 03:51 hoo: Ran "scap pull" on snapshot1001, after final T181385 tests
  • 00:03 mutante: tried one more time on db2028,db2029, both trusty. on db2028: gmond was running as user ganglia-monitor, failed, had to manually kill the process, run puppet again then ok. on db2029, gmond was running as "499" but puppet just ran and removed it without manual intervention. (T177225)

2017-12-01

  • 23:15 urandom: starting cassandra bootstrap, restbase1012-b - T179422
  • 21:49 mutante: db2029 - removing ganglia-monitor, testing to kill gmond, running puppet to figure out how to cleanly remove it on trusty
  • 21:12 mutante: db2023 killed gmond (ganglia-monitor) process manually which was still running even though ganglia-monitor package was removed and caused puppet breakage (it seems only on trusty). after that puppet run is clean again and ganglia removed. (T177225) (https://gerrit.wikimedia.org/r/#/c/394647/1)
  • 20:18 awight@tin: Started deploy [ores/deploy@9afbf14]: (non-production) Test ORES deployment to ores100*
  • 20:17 awight@tin: Finished deploy [ores/deploy@9afbf14]: (non-production) Test ORES deployment to ores1001 (duration: 02m 31s)
  • 20:15 awight@tin: Started deploy [ores/deploy@9afbf14]: (non-production) Test ORES deployment to ores1001
  • 20:03 aaron@tin: Synchronized php-1.31.0-wmf.10/includes/libs/objectcache/WANObjectCache.php: f096d0b465b75d - temp logging for statsd spam (duration: 00m 45s)
  • 18:59 demon@tin: Synchronized wmf-config/CommonSettings-labs.php: no-op (duration: 00m 46s)
  • 18:22 mutante: Phabricator: restarting Apache for php-curl update
  • 18:21 _joe_: restarting apache2 on the codfw puppetmasters
  • 18:06 marktraceur@tin: Synchronized php-1.31.0-wmf.10/extensions/UploadWizard/resources/controller/uw.controller.Deed.js: (no justification provided) (duration: 00m 46s)
  • 17:49 mutante: phab2001 - restarted apache
  • 17:33 herron: stopped ircecho on einsteinium
  • 17:00 awight@tin: Unlocked for deployment [ores/deploy]: Don't deploy while we're messing with git-lfs (duration: 00m 14s)
  • 17:00 awight@tin: Locking from deployment [ores/deploy]: Don't deploy while we're messing with git-lfs (planned duration: 16666666666m 39s)
  • 17:00 awight@tin: Locking from deployment [ores/deploy]: Don't deploy while we're messing with git-lfs (planned duration: -1m 59s)
  • 16:59 awight@tin: Unlocked for deployment [ores/deploy]: Don't deploy while we're messing with git-lfs (duration: 00m 07s)
  • 16:59 awight@tin: Locking from deployment [ores/deploy]: Don't deploy while we're messing with git-lfs (planned duration: 60m 00s)
  • 16:34 jynus: stopping db2092 to clone s1 to db2085
  • 16:24 urandom: starting cassandra bootstrap, restbase1012-a -- T179422
  • 15:27 godog: bounce uwsgi on labmon1001 - stuck
  • 15:21 moritzm: installing nspr security updates on trusty
  • 15:17 moritzm: installing ffmpeg security updates
  • 15:14 gehel@tin: Finished deploy [kartotherian/deploy@df7ebff]: testing new kartotherian packaging on maps-test2003 (duration: 00m 20s)
  • 15:14 jynus@tin: Synchronized wmf-config/db-codfw.php: Undeploy db2092, use db2085 for s1 (duration: 00m 45s)
  • 15:14 gehel@tin: Started deploy [kartotherian/deploy@df7ebff]: testing new kartotherian packaging on maps-test2003
  • 15:14 moritzm: installing libxcursor security updates on trusty
  • 15:09 jynus@tin: Synchronized wmf-config/db-eqiad.php: Undeploy db2092, use db2085 for s1 (duration: 00m 45s)
  • 14:24 awight@tin: Finished deploy [ores/deploy@532bd0b]: (non-production) Update ORES on new cluster (duration: 02m 06s)
  • 14:22 awight@tin: Started deploy [ores/deploy@532bd0b]: (non-production) Update ORES on new cluster
  • 14:22 akosiaris: upload apertium-crh-tur_0.3.0~r83159-1+wmf1 to apt.wikimedia.org/jessie-wikimedia component main. T181465
  • 14:10 herron: cutting all puppet service records over to codfw puppet 4 masters
  • 12:44 elukey: reboot druid1001 for kernel+jvm updates - T179943
  • 12:11 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2055 (duration: 00m 45s)
  • 11:59 jynus: restarting and upgrading mysql on labsdb1004
  • 11:28 jynus: upgrading and restarting dbstore2001
  • 11:14 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2055 (duration: 00m 46s)
  • 11:08 marostegui: Stop MySQL on db2055 for testing
  • 11:04 marostegui: Update MySQL on db1039 for testing
  • 10:57 elukey: reboot analytics1028 for kernel + jvm updates (Hadoop HDFS journalnode) - T179943
  • 10:56 godog: delete docker diskspace metrics from labs - T181476
  • 10:21 godog: initial purge of old table metrics from graphite2002 - T181689
  • 10:18 moritzm: reimaging mw1259 (video scaler) to stretch, will be kept disabled initially (with some live tests starting next week)
  • 09:55 akosiaris: run memtester 61G on ganeti1008 T181121
  • 09:23 elukey: reboot analytics104* for kernel+jvm updates - T179943
  • 09:23 akosiaris: powercycle ganeti1008 T181121, it's largely unresponsive
  • 08:41 marostegui: Remove db1046 and db1047 from tendril - T156844
  • 08:40 elukey: reboot the remaining analytics103* hadoop workers to pick up kernel+jvm updates - T179943
  • 08:28 akosiaris: empty ganeti1008, move VMs to ganeti1006 T181121
  • 08:20 akosiaris: repool ganeti1005, ganeti1006 to empty ganeti1008 T181121
  • 07:51 akosiaris: upload apertium-tur_0.2.0~r83161-1+wmf1, apertium-crh_0.2.0~r83161-1+wmf1 to apt.wikimedia.org/jessie-wikimedia component main. T181465
  • 07:14 marostegui: Logging retroactively for the record, restarting MySQL on db1039
  • 05:14 legoktm@tin: Synchronized wmf-config/InitialiseSettings.php: touch (duration: 00m 44s)
  • 01:46 hoo: Ran scap pull on mwdebug1001 after T181385 testing
  • 00:13 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: HTML5 sections be upon us! (duration: 00m 45s)
  • 00:04 hoo: Killed all remaining Wikidata JSON/RDF dumpers, due to T181385. This means no dumps this week!

2017-11-30

  • 23:54 demon@tin: Synchronized wmf-config/extension-list-labs: no-op (duration: 00m 45s)
  • 23:39 addshore@tin: Synchronized wmf-config: wdbuild: T173818 T177060 Add wikidata extensions to extension-list (duration: 00m 46s)
  • 23:35 addshore@tin: Synchronized wmf-config: wdbuild: T173818 Remove Wikibase-buildentry.php config file (empty) (duration: 00m 46s)
  • 23:34 addshore@tin: Synchronized wmf-config/Wikibase.php: wdbuild: T173818 Remove Wikibase-buildentry.php config file (empty) (duration: 00m 45s)
  • 23:30 addshore@tin: Synchronized wmf-config: wdbuild: T173818 Remove wmgUseWikidataBuild (duration: 00m 46s)
  • 23:29 addshore@tin: Synchronized wmf-config/Wikibase-buildentry.php: wdbuild: T173818 Remove wmgUseWikidataBuild (duration: 00m 45s)
  • 23:21 addshore@tin: Synchronized wmf-config: wdbuild: T173818 wdbuild: Stop loading from build on ALL WIKIS The build is dead! Mwahahaaa (duration: 00m 47s)
  • 23:18 bsitzmann@tin: Finished deploy [mobileapps/deploy@4305d96]: Update mobileapps to 4317ea5 (T181743) (duration: 04m 50s)
  • 23:15 addshore@tin: Synchronized wmf-config: wdbuild: wdbuild: Stop loading from build on all wikis (except the one i really dont want to break) (duration: 00m 46s)
  • 23:14 bsitzmann@tin: Started deploy [mobileapps/deploy@4305d96]: Update mobileapps to 4317ea5 (T181743)
  • 23:06 addshore@tin: Synchronized wmf-config: wdbuild: wdbuild: Stop loading from build on group1 (duration: 00m 46s)
  • 22:55 addshore@tin: Synchronized wmf-config: wdbuild: wdbuild: Stop loading from build on group0 (duration: 00m 46s)
  • 22:42 addshore: BETA ONLY was a lie on that last one ...
  • 22:41 addshore@tin: Synchronized wmf-config: wdbuild: BETA ONLY wdbuild: Stop loading from build on test and testwikidata (duration: 00m 47s)
  • 22:40 demon@tin: Pruned MediaWiki: 1.31.0-wmf.8 [keeping static files] (duration: 02m 33s)
  • 22:27 addshore@tin: Synchronized wmf-config: wdbuild: BETA ONLY gerrit:394411 (duration: 00m 47s)
  • 21:35 addshore@tin: Synchronized wmf-config: wdbuild: BETA ONLY (duration: 00m 47s)
  • 21:31 jynus: upgrading labsdb1010 and restarting mariadb
  • 21:23 mutante: powercycling kafka1018 (was down in Icinga and saw in SAL: reboot kafka10[12-22] for kernel + jvm updates - T179943)
  • 21:18 jynus: upgrade and restart db2078
  • 21:14 addshore@tin: Synchronized wmf-config: wdbuild: BETA ONLY (duration: 00m 47s)
  • 21:01 addshore@tin: Synchronized wmf-config: wdbuild: T173818: add switch to ease killing (again) (duration: 00m 46s)
  • 20:59 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: wdbuild: T173818: add switch to ease killing (again) (duration: 00m 45s)
  • 20:49 XenoRyet: Updated tools from 6e604fd to 626fe02
  • 20:40 bd808: Sending Toolforge survey final reminder emails from silver for T177126
  • 20:32 addshore@tin: Synchronized wmf-config: REVERT wdbuild: T173818: add switch to ease killing (duration: 00m 47s)
  • 20:30 addshore@tin: Synchronized wmf-config: wdbuild: T173818: add switch to ease killing (duration: 00m 47s)
  • 20:13 ejegg: re-adjusted Civi job timings. QC every odd min for 105 sec, TY every even min for 70 sec after 45 sec delay
  • 20:12 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.10
  • 19:55 ejegg: 105 seconds for each job
  • 19:55 ejegg: adjusted timings of Civi jobs to let TY and QC run concurrently
  • 19:37 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Temporary disable remex html" T178632 (duration: 00m 49s)
  • 19:20 thcipriani@tin: Synchronized php-1.31.0-wmf.10/includes/htmlform/fields/HTMLMultiSelectField.php: SWAT: HTMLMultiSelectField: Allow formatting in section headings in OOUI mode T181698 (duration: 00m 49s)
  • 19:12 thcipriani@tin: Synchronized wmf-config: SWAT: Enable MP3 uploads on Commons T120288 (duration: 00m 51s)
  • 17:57 andrewbogott: upgrading labtestpuppetmaster2001 to puppet 4.8
  • 17:47 moritzm: uploaded prometheus-openldap-exporter 0+git20171128-1 for jessie-wikimedia (T181511)
  • 17:06 herron: beginning cut over of esams to codfw puppet 4 masters
  • 16:44 urandom: drop (erroneous) legacy tables from -ng cassandra cluster - T181689
  • 16:12 elukey: drain and reboot analytics1031->39 to pick up jvm+kernel updates - T179943
  • 15:59 jynus: setup prometheus with unix_socket on new server db1107 and db1108
  • 15:19 gehel: restart blazegraph on wdqs1004
  • 15:16 jynus@tin: Synchronized wmf-config/db-eqiad.php: Increase db1110 load (duration: 00m 48s)
  • 15:11 herron: beginning cut over of ulsfo to codfw puppet 4 masters
  • 14:50 zeljkof: EU SWAT finished
  • 14:46 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable WikiLove Extension on pa.wiki (T178919) (duration: 00m 49s)
  • 14:28 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add import sources to de.wiki (T181695) (duration: 00m 49s)
  • 14:20 zfilipin@tin: Synchronized php-1.31.0-wmf.10/extensions/ContentTranslation/modules/dashboard/ext.cx.dashboard.js: SWAT: Set default languages after fetching valid languages (duration: 00m 49s)
  • 14:19 marostegui: Deploy schema change on s3 - db1078 - T174569
  • 13:23 addshore@tin: Synchronized php-1.31.0-wmf.10/extensions/AdvancedSearch/modules/ext.advancedSearch.init.js: pre-swat: T181644 Force search profile advanced in AdvancedSearch gerrit:394297 (duration: 00m 48s)
  • 13:22 addshore@tin: Synchronized php-1.31.0-wmf.8/extensions/AdvancedSearch/modules/ext.advancedSearch.init.js: pre-swat: T181644 Force search profile advanced in AdvancedSearch gerrit:394299 (duration: 00m 50s)
  • 13:18 herron: cutting codfw puppet agents over to puppetmaster2001.codfw.wmnet
  • 12:07 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Tackle s4 DB weights to make them more equal (duration: 00m 48s)
  • 11:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1081 and reduce traffic for db1091 (duration: 00m 50s)
  • 10:33 jynus: stopping replication on db1044 and db1095 (s3)
  • 09:16 moritzm: installing exim security updates on stretch (jessie/trusty not affected)
  • 09:14 elukey: drain and reboot analytics1029/1030 for jvm+kernel updates (Hadoop worker canaries)
  • 09:05 godog: add 200G of space to graphite2002 carbon lv
  • 08:25 moritzm: rolling restart of mw canaries to pick up curl security update
  • 08:21 marostegui: Enable GTID on s8 eqiad hosts that do not have it enabled (db1109, db1104, db1101, db1092, db1087, db1063) - T177208
  • 07:48 moritzm: installing curl security updates
  • 07:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1056 (duration: 01m 18s)
  • 06:48 marostegui: Deploy schema change on dbstore1001 s3 - T174569
  • 06:26 marostegui: Deploy schema change on s3 db1077 - T174569
  • 02:48 ebernhardson@tin: Finished deploy [search/mjolnir/deploy@7aa39b7]: (no justification provided) (duration: 03m 15s)
  • 02:45 ebernhardson@tin: Started deploy [search/mjolnir/deploy@7aa39b7]: (no justification provided)
  • 02:29 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.8) (duration: 08m 43s)
  • 01:17 ejegg: updated fundraising tools from 6d4b6f3 to 6e604fd
  • 01:06 twentyafterfour: finished phabricator upgrade, service is online and appears to be functioning normally
  • 01:03 twentyafterfour: Starting phabricator upgrade & maintenance. Service will be offline for less than 5 minutes.
  • 01:02 bsitzmann@tin: Finished deploy [mobileapps/deploy@fa2a877]: Update mobileapps to dcea7d3 (T181004) (duration: 06m 08s)
  • 00:56 bsitzmann@tin: Started deploy [mobileapps/deploy@fa2a877]: Update mobileapps to dcea7d3 (T181004)
  • 00:53 maxsem@tin: Finished scap: Message updates for https://gerrit.wikimedia.org/r/#/c/394155/ (duration: 26m 38s)
  • 00:48 ejegg: re-enabled CiviCRM jobs
  • 00:42 ejegg: updated CiviCRM from 0f95f3e to e81228f
  • 00:40 ejegg: disabled CiviCRM jobs
  • 00:31 ejegg: updated fundraising dashboard from 6ee6567 to 1141317
  • 00:27 maxsem@tin: Started scap: Message updates for https://gerrit.wikimedia.org/r/#/c/394155/
  • 00:24 maxsem@tin: Synchronized php-1.31.0-wmf.10/extensions/ContentTranslation/: https://gerrit.wikimedia.org/r/#/c/394206/ (duration: 00m 52s)
  • 00:21 ejegg: added weekly Ingenico audit processing job in makemissing mode

2017-11-29

  • 23:20 eileen: update civicrm from a1022cf to 0f95f3e (Benevity import fix)
  • 22:23 hashar: Nodepool had some troubles spawning new instances from 21:09 to 21:36, and took a while to recover. Issue similar to T170492#3581822
  • 21:51 SMalyshev: starting wikidata reindex (T181426)
  • 21:23 no_justification: docroot sync was for If1afa59a
  • 21:21 demon@tin: Synchronized docroot/: (no justification provided) (duration: 00m 49s)
  • 20:29 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.10
  • 20:26 demon@tin: Synchronized php: symlink bump (duration: 00m 48s)
  • 20:05 urandom: restarting cassandra bootstrap of restbase1012-a (T179422)
  • 19:55 mutante: restarting gerrit to apply config change and set gitBasicAuth to true to unblock T171758 (gerrit:391865)
  • 19:50 herron: beginning rolling cut over of eqiad scb hosts to codfw puppet 4 masters
  • 19:15 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/394059/3 (duration: 00m 49s)
  • 18:57 herron: beginning cutover of codfw lvs2* systems to codfw puppet 4 masters. first standby nodes, then active nodes
  • 18:12 marostegui: Compress s5 on db1096 - T178359
  • 18:10 jynus@tin: Synchronized wmf-config/db-eqiad.php: Increase db1110 load (duration: 00m 53s)
  • 18:05 awight@tin: Finished deploy [ores/deploy@532bd0b]: (non-production) Update ORES on new cluster (duration: 01m 11s)
  • 18:04 awight@tin: Started deploy [ores/deploy@532bd0b]: (non-production) Update ORES on new cluster
  • 18:03 herron: beginning cut over of codfw cp2* servers to codfw puppet 4 masters
  • 17:40 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool db1110 with low load (duration: 00m 48s)
  • 17:40 godog: bootstrapping restbase1012-a - T179422
  • 17:19 herron: beginning cut over of cp200[12] to codfw puppet 4 masters
  • 17:16 ejegg: increased time limit for donation queue consumer from 60 to 70 seconds, reduced thank you job from 55 to 45 seconds
  • 17:03 ejegg: re-enabled Civi jobs
  • 16:58 ejegg: disabled Civi jobs for Deadlock-retry update
  • 16:46 jynus@tin: Synchronized wmf-config/db-codfw.php: Update db1110 ip (duration: 00m 50s)
  • 16:43 jynus@tin: Synchronized wmf-config/db-eqiad.php: Update db1110 ip (duration: 00m 49s)
  • 15:02 chasemp: purge old qcow2 images from under /home on labtestcontrol2001
  • 15:00 awight: Restarting ORES celery workers manually, T181538
  • 14:39 _joe_: reloading apache on puppetmasters to pick up the configuration changes
  • 14:37 awight@tin: Started restart [ores/deploy@e58bfbf]: Restart ORES services (take 2), T181538
  • 14:36 elukey: reboot druid100[456] for jvm+kernel updates - T179943
  • 14:35 awight@tin: Finished deploy [ores/deploy@e58bfbf]: Restart ORES services, T181538 (duration: 00m 16s)
  • 14:35 awight@tin: Started deploy [ores/deploy@e58bfbf]: Restart ORES services, T181538
  • 14:25 zeljkof: EU SWAT finished
  • 14:18 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: Define throttle rule (T181367) (duration: 00m 49s)
  • 14:14 herron: beginning cut over of cache::misc and codfw cache::canary cp servers to codfw puppet4 masters
  • 14:13 moritzm: rebooting neodymium for kernel update to 4.9.51
  • 14:12 godog: reimage restbase1012 - T179422
  • 14:10 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT T180291 T180128 AdvancedSearch for arwiki (duration: 00m 49s)
  • 14:07 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT T180128 AdvancedSearch for dewiki (duration: 00m 50s)
  • 13:34 awight@tin: Finished deploy [ores/deploy@e58bfbf]: (non-production) Update ORES on new cluster (take 3) (duration: 16m 49s)
  • 13:18 elukey: reboot kafka100[23] for jvm+kernel updates - T179943
  • 13:17 awight@tin: Started deploy [ores/deploy@e58bfbf]: (non-production) Update ORES on new cluster (take 3)
  • 12:34 jynus@tin: Synchronized wmf-config/db-eqiad.php: Setup s8 on eqiad, with no wikis (duration: 00m 48s)
  • 12:24 jynus: scap pull on mwdebug1001
  • 12:23 jynus@tin: Synchronized wmf-config/db-codfw.php: Pool db1096:3315 (duration: 00m 48s)
  • 12:01 marostegui: Deploy schema change on db1072 (sanitarium master) on s3 with replication enabled to replicate to labs - T174569
  • 11:30 elukey: reboot kafka1001 for kernel + jvm updates - T179943
  • 10:15 akosiaris: disable puppet on oresrdb* for merging https://gerrit.wikimedia.org/r/#/c/394022/. T181563
  • 09:50 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1110 (duration: 00m 48s)
  • 09:46 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1110 (duration: 00m 49s)
  • 09:39 godog: upload cassandra-tools-wmf 1.0.2-1 - T181438
  • 09:28 mobrovac@tin: Finished deploy [electron-render/deploy@94d27d7]: Update to electron v1.7.9 and start using the Charter font - T181200 (duration: 04m 03s)
  • 09:24 mobrovac@tin: Started deploy [electron-render/deploy@94d27d7]: Update to electron v1.7.9 and start using the Charter font - T181200
  • 09:04 godog: bootstrap restbase1007-c - T179422
  • 08:41 moritzm: uploaded prometheus-openldap-exporter 0+git20171128-1 for jessie-wikimedia (T181511)
  • 08:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1096 - T178359 (duration: 00m 48s)
  • 07:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1099:3318 and db1055 - T178359 (duration: 00m 48s)
  • 07:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1099:3318 traffic - T178359 (duration: 00m 45s)
  • 07:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1099:3318 with low weight - T178359 (duration: 00m 45s)
  • 06:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1055 - T178359 (duration: 00m 48s)
  • 06:42 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1099:3311 and db1099:3318 to the config (depooled) T178359 (duration: 00m 48s)
  • 06:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1099:3311 and db1099:3318 to the config (depooled) T178359 (duration: 00m 49s)
  • 06:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1055 with lower weight - T178359 (duration: 00m 50s)
  • 06:17 ebernhardson@tin: Finished deploy [search/mjolnir/deploy@7aa39b7]: (no justification provided) (duration: 04m 27s)
  • 06:12 ebernhardson@tin: Started deploy [search/mjolnir/deploy@7aa39b7]: (no justification provided)
  • 03:39 ejegg: reduced CiviMail sample rate to 0.35
  • 03:09 ejegg: reduced donation queue consumer time limit from 70 to 60 seconds, increased ty mail batch time limit from 45 to 55 seconds
  • 02:50 ejegg: reduced donation queue consumer time limit from 75 to 70 seconds, increased ty mail batch time limit from 40 to 45 seconds
  • 02:44 ejegg: reduced CiviMail record creation rate from 100% to 50%
  • 02:24 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.8) (duration: 06m 08s)
  • 01:25 mutante: snapshot1001 - closed idle screen session
  • 01:22 mutante: analytics1003 - closed idle screen session
  • 01:21 mutante: mw1276 - run "scap pull" to get in sync after hardware issue, then pooled again (T181397)
  • 01:09 mutante: restarting ircecho - it stopped talking
  • 01:07 mutante: forcing puppet run on all labvirt* machines to clean out Icinga alerts
  • 00:44 awight@tin: Synchronized php-1.31.0-wmf.10/extensions/ORES: Hotfix to mitigate cache stampeding, T181567 (duration: 00m 49s)
  • 00:33 reedy@tin: Synchronized php-1.31.0-wmf.10/includes/logging/LogPager.php: Fix fatal on Special:Log T181565 (duration: 00m 48s)
  • 00:32 awight@tin: Synchronized php-1.31.0-wmf.8/extensions/ORES: Hotfix to mitigate cache stampeding, T181567 (duration: 00m 50s)
  • 00:18 Jamesofur: deleted 6 archived files from servers for legal compliance
  • 00:08 ejegg: updated fundrasing dashboard from df94248 to 6ee6567

2017-11-28

  • 23:58 hoo: Ran scap pull on mwdebug1001 after T181385 related testing
  • 23:07 akosiaris@tin: Synchronized wmf-config/CommonSettings.php: T181538 (duration: 00m 49s)
  • 22:31 akosiaris@tin: Synchronized wmf-config/CommonSettings.php: (no justification provided) (duration: 00m 49s)
  • 22:31 akosiaris: deploy wmf-config/CommonSettings.php for ORES internal discovery URL, https://gerrit.wikimedia.org/r/#/c/393924/ T181538
  • 21:41 akosiaris: disable ORES queue redis persistency by config set appendonly no on oresrdb1001
  • 21:24 hoo: Manually killed all remaining Wikidata TTL (RDF) dumpers on snapshot1007. Some shards failed due to the db1110 depool.
  • 21:23 hoo: Manually killed all remaining Wikidata JSON dumpers on snapshot1007. Some shards failed due to the db1110 depool.
  • 20:56 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: removing aawiki from group0
  • 20:42 gehel: repooling elastic2004 after RAID controller maintenance - T181412
  • 20:42 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.10
  • 20:28 mutante: forcing puppet run on cache misc to revert "failover ORES to codfw"
  • 20:19 demon@tin: Synchronized scap/plugins/prep.py: no-op (duration: 00m 48s)
  • 20:16 demon@tin: Synchronized dblists/group0.dblist: adding some new wikis (duration: 00m 48s)
  • 18:57 demon@tin: Finished scap: bootstrap wmf.10 (duration: 35m 20s)
  • 18:51 demon@tin: Finished deploy [gerrit/gerrit@571cf4c]: deploying 2.15+ polygerrit style changes (duration: 00m 09s)
  • 18:51 demon@tin: Started deploy [gerrit/gerrit@571cf4c]: deploying 2.15+ polygerrit style changes
  • 18:40 akosiaris: revert weight changes for scb1001, scb1002 T181835
  • 18:39 akosiaris@puppetmaster1001: conftool action : set/weight=10; selector: scb1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
  • 18:39 akosiaris@puppetmaster1001: conftool action : set/weight=10; selector: scb1001.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
  • 18:35 awight@tin: Finished deploy [ores/deploy@e58bfbf]: (non-production) Update ORES on new cluster (take 2) (duration: 04m 30s)
  • 18:32 akosiaris: force puppet run on cache::misc boxes T181538
  • 18:30 awight@tin: Started deploy [ores/deploy@e58bfbf]: (non-production) Update ORES on new cluster (take 2)
  • 18:28 urandom: (re)bootstrapping cassandra, restbase1007-b - T179422
  • 18:22 demon@tin: Started scap: bootstrap wmf.10
  • 18:18 awight@tin: Finished deploy [ores/deploy@e58bfbf]: (non-production) Update ORES on new cluster (duration: 01m 41s)
  • 18:17 awight@tin: Started deploy [ores/deploy@e58bfbf]: (non-production) Update ORES on new cluster
  • 18:00 akosiaris: force stop celery-ores-worker on scb1001
  • 17:58 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: scb1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
  • 17:58 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: scb1001.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
  • 17:46 urandom: decommissioning cassandra, restbase1007-b - T179422
  • 17:42 urandom: restart cassandra, restbase1007, to pickup logstash java deps - T179422
  • 16:45 ejegg: updated fundraising dashboard from d8c86e7 to df94248
  • 16:25 bblack: mw1329 boot to PXE (should come up with new .66 IP)
  • 15:39 jynus@tin: Synchronized wmf-config/db-eqiad.php: depool db1110 (duration: 00m 44s)
  • 15:38 bblack: powered off mw1329
  • 15:06 marostegui: Compress s5 on db1099
  • 15:04 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Revert "Comply wikidata with new ores thresholds" (duration: 00m 45s)
  • 14:56 marostegui: Deploy schema change on s3 on dbstore1002 - T174569
  • 14:45 gehel@tin: Finished deploy [kartotherian/deploy@cb9b1ef]: testing new kartotherian packaging on maps-test2003 (duration: 00m 02s)
  • 14:45 gehel@tin: Started deploy [kartotherian/deploy@cb9b1ef]: testing new kartotherian packaging on maps-test2003
  • 14:41 otto@tin: Finished deploy [eventlogging/analytics@c464b8c]: Fixing bug where userAgent set by client producer was not used T178440 (duration: 00m 02s)
  • 14:40 otto@tin: Started deploy [eventlogging/analytics@c464b8c]: Fixing bug where userAgent set by client producer was not used T178440
  • 14:40 otto@tin: Started deploy [eventlogging/analytics@c464b8c]: Fixing bug where userAgent set by client producer was not used
  • 14:39 gehel@tin: Finished deploy [kartotherian/deploy@cb9b1ef]: testing new kartotherian packaging on maps-test2003 (duration: 00m 18s)
  • 14:39 gehel@tin: Started deploy [kartotherian/deploy@cb9b1ef]: testing new kartotherian packaging on maps-test2003
  • 14:37 marostegui: Stop MySQL on db1055 to clone db1099:3311 - T178359
  • 14:34 gehel@tin: Finished deploy [kartotherian/deploy@55c5da4]: (no justification provided) (duration: 00m 11s)
  • 14:34 gehel@tin: Started deploy [kartotherian/deploy@55c5da4]: (no justification provided)
  • 14:21 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1055 - T178359 (duration: 00m 44s)
  • 14:17 elukey: reboot kafka10[12-22] for kernel + jvm updates - T179943
  • 14:03 elukey: reboot kafka200[123] for kernel + jvm updates - T179943
  • 12:25 godog: cleanup wanobjectcache metrics with hashes - T178531
  • 12:14 godog: bounce carbon-frontend-relay after https://gerrit.wikimedia.org/r/393749
  • 11:25 hoo: Manually re-started Wikidata JSON dumps on snapshot1007, got stuck after db1082 went down.
  • 10:55 jynus: stop db1044 replication for db1095 master switchover
  • 10:49 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1082. Depool db1044 (duration: 00m 44s)
  • 10:44 mobrovac@tin: Synchronized wmf-config/jobqueue.php: Process wikibase-addUsagesForPage only via EventBus - T175212 (duration: 00m 44s)
  • 10:08 ppchelko@tin: Finished deploy [cpjobqueue/deploy@c4b9e16]: Enable wikibase-addUsagesForPage with low concurrency (duration: 00m 29s)
  • 10:07 ppchelko@tin: Started deploy [cpjobqueue/deploy@c4b9e16]: Enable wikibase-addUsagesForPage with low concurrency
  • 09:35 jynus: restart db1087 for maintenance
  • 09:22 jynus: restart db1082
  • 09:19 godog: unmask and restart restbase1007-b - T179422
  • 09:15 moritzm: installibg libxml-libxml-perl security updates on trusty (Debian already fixed)
  • 09:11 ppchelko@tin: Finished deploy [cpjobqueue/deploy@b0b1793]: Remove double-processing created for consumer group renaming (duration: 00m 28s)
  • 09:10 ppchelko@tin: Started deploy [cpjobqueue/deploy@b0b1793]: Remove double-processing created for consumer group renaming
  • 09:04 marostegui: Drop database log from dbstore1002 - T156844
  • 08:57 ppchelko@tin: Finished deploy [cpjobqueue/deploy@2212086]: Move enabled jobs config to vars.yaml (duration: 00m 39s)
  • 08:56 ppchelko@tin: Started deploy [cpjobqueue/deploy@2212086]: Move enabled jobs config to vars.yaml
  • 08:38 jynus@tin: Synchronized wmf-config/db-eqiad.php: depool db1082 (duration: 00m 44s)
  • 08:02 mobrovac: bootstrap restbase1007-b - T179422
  • 06:39 marostegui: Stop MySQL on db1099 to copy its content to dbstore1001 and reimage it as multi-instance - T178359
  • 05:23 mutante: labtestpuppetmaster2001 - manually purging ganglia things since puppet is disabled
  • 05:05 mutante: labweb1001/labweb1002: manually purging ganglia package/config/service/unit files because puppet is disabled there (T177225)
  • 04:08 eileen: update process-control to b40eaec, disabling 3 jobs to reduce load for Big english starting: dedupe_civicrm_contacts, omnimail_recipient_load, omnimail_recipient_load_backfill
  • 03:58 demon@tin: Pruned MediaWiki: 1.31.0-wmf.4 (duration: 03m 13s)
  • 03:31 mutante: labservices1001/1002,labtestservices2001 - remove pdns_gmetric cronjobs causing cron spam after ganglia decom from lab*
  • 02:20 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.8) (duration: 05m 23s)

2017-11-27

  • 22:29 mutante: labtestnet2001 - re-enabled puppet to decom ganglia, no errors
  • 22:22 mutante: decom'ing Ganglia from all labtest* hosts - packages removed by puppet etc, might cause some false positives in Icinga but it's ok (T177225)
  • 22:21 otto@tin: Finished deploy [eventlogging/eventbus@e024af3]: deploy revert checked out at 3df06ab (we caught one). T180017 (duration: 00m 10s)
  • 22:21 otto@tin: Started deploy [eventlogging/eventbus@e024af3]: deploy revert checked out at 3df06ab (we caught one). T180017
  • 21:59 awight: clearing ORES threshold caches for ruwiki, frwiki, wikidatawiki.
  • 21:59 awight@tin: Synchronized wmf-config/InitialiseSettings.php: Reenable ORES on frwiki, ruwiki, and wikidata; T181006 (duration: 00m 45s)
  • 21:53 awight@tin: Finished deploy [ores/deploy@e58bfbf]: (codfw) Update ORES for impossible threshold handling and new wikidata editquality model, T179711 T180686 T180450 (duration: 05m 06s)
  • 21:47 awight@tin: Started deploy [ores/deploy@e58bfbf]: (codfw) Update ORES for impossible threshold handling and new wikidata editquality model, T179711 T180686 T180450
  • 21:46 awight@tin: Finished deploy [ores/deploy@e58bfbf]: Update ORES for impossible threshold handling and new wikidata editquality model, T179711 T180686 T180450 (duration: 12m 55s)
  • 21:33 awight@tin: Started deploy [ores/deploy@e58bfbf]: Update ORES for impossible threshold handling and new wikidata editquality model, T179711 T180686 T180450
  • 21:23 awight@tin: Synchronized php-1.31.0-wmf.8/extensions/ORES: ORES error handling for bad thresholds, T181191 (duration: 00m 46s)
  • 21:11 awight@tin: Synchronized wmf-config/InitialiseSettings.php: Temporarily disable ORES on wikidata (duration: 00m 45s)
  • 21:10 awight: Previous “rollback ORES” was from a stale screen session, no actions were taken today.
  • 21:08 awight@tin: Finished deploy [ores/deploy@82a13ae]: Rollback ORES (take 3); 181006 (duration: 9942m 42s)
  • 20:21 demon@tin: i lied, that was for wgStyleVersion removal
  • 20:20 demon@tin: Synchronized wmf-config/CommonSettings.php: cli sapi fixes (duration: 00m 45s)
  • 20:17 ejegg: updated payments-wiki from fea6b37 to f594dfa
  • 20:03 kaldari@tin: Synchronized wmf-config/InitialiseSettings-labs.php: (no justification provided) (duration: 00m 45s)
  • 20:02 kaldari@tin: Synchronized wmf-config/CommonSettings.php: (no justification provided) (duration: 00m 45s)
  • 20:01 kaldari@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 46s)
  • 19:54 bblack: crN-ulsfo: remove lvs400[1-4] from PyBal BGP neighbors list
  • 19:42 catrope@tin: Synchronized php-1.31.0-wmf.8/resources/src/mediawiki.rcfilters/mw.rcfilters.UriProcessor.js: T181100 (duration: 00m 45s)
  • 19:41 demon@tin: Synchronized wmf-config/InitialiseSettings.php: babel officewiki thing (duration: 00m 45s)
  • 19:39 catrope@tin: Synchronized php-1.31.0-wmf.8/includes/specials/SpecialRecentchangeslinked.php: T181100 (duration: 00m 45s)
  • 19:34 otto@tin: Finished deploy [eventlogging/eventbus@3df06ab]: Temp deploy to kafka1001 only to catch bug: T180017 (duration: 00m 12s)
  • 19:34 otto@tin: Started deploy [eventlogging/eventbus@3df06ab]: Temp deploy to kafka1001 only to catch bug: T180017
  • 19:21 catrope@tin: Synchronized wmf-config/Wikibase.php: T176903 (duration: 00m 45s)
  • 19:17 AaronSchulz: Removed more bogus md5 wanobjecache/ metric from graphite[12]001
  • 19:11 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: T179241 (duration: 00m 45s)
  • 19:02 volans: restarted ircecho
  • 18:51 ejegg: updated payments-wiki from 6b3019b to fea6b37
  • 18:50 dcausse: elastic/cirrus: reindexing english group2 wikis: T179945
  • 18:45 volans: stopped ircecho temporary to avoid spam
  • 18:14 XioNoX: pushing v6 gre firewall filter
  • 18:12 gehel@tin: Finished deploy [wdqs/wdqs@7ad20f3]: (no justification provided) (duration: 02m 10s)
  • 18:11 gehel: deploying blazegraph + GUI updates
  • 18:10 gehel@tin: Started deploy [wdqs/wdqs@7ad20f3]: (no justification provided)
  • 18:08 hoo: Killed truthy nt dumpers stuck at 100% CPU (from last week) on snapshot1007 (T181385)
  • 18:06 akosiaris: poweroff ganeti1005, ganeti1006 T181121
  • 18:05 bd808: Sending Toolforge survey 1 week reminder emails from silver for T177126
  • 16:07 bblack: lvs400x - puppet disabled for https://gerrit.wikimedia.org/r/#/c/393610
  • 15:52 herron: beginning canary cutover/deployment of codfw prometheus servers to codfw puppet 4 puppetmasters
  • 15:19 chasemp: disable puppet across cloud things for cleanup
  • 15:11 herron: beginning canary cutover/deployment of codfw elasticsearch servers to codfw puppet 4 puppetmasters
  • 14:22 zeljkof: EU SWAT finished
  • 14:19 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: IP cap lift for Semaine contributive 2017-2018 (T181360) (duration: 00m 45s)
  • 14:10 addshore@tin: Synchronized php-1.31.0-wmf.8/extensions/AdvancedSearch: SWAT AdvancedSearch T181175 T181222, adjust links in beta section & fix usability of mobile search (duration: 00m 46s)
  • 13:31 moritzm: installing nspr security updates on trusty
  • 13:22 elukey: remove eventlogging replication support (log database) from dbstore1002 - T156844
  • 13:17 moritzm: updating nginx on francium
  • 13:01 Pchelolo: stop cpjobqueue in eqiad for backlog accumulation
  • 12:44 Pchelolo: started parsoid linter script to generate load on cpjobqueue (python3 parsoid_reparse.py http://parsoid.discovery.wmnet:8000 --sitematrix --linter-only --skip-closed https://commons.wikimedia.org/w/api.php)
  • 12:44 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: gerrit:393587 BETA wmgMonologChannels FileImporter => debug (duration: 01m 01s)
  • 12:43 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: mw1276.eqiad.wmnet
  • 12:35 moritzm: powercycling mw1276
  • 12:21 addshore@tin: Synchronized wmf-config/CommonSettings-labs.php: gerrit:393582 LABS ONLY Actually call wfLoadExtension for FileExporter & Importer on beta BETA (T181383) (duration: 00m 44s)
  • 12:14 marostegui: Stop replication on db1109 to test table rename for s5/s8 failover
  • 12:09 ppchelko@tin: Finished deploy [cpjobqueue/deploy@47d27dc]: Enable keep-alive T181007 (duration: 01m 03s)
  • 12:08 ppchelko@tin: Started deploy [cpjobqueue/deploy@47d27dc]: Enable keep-alive T181007
  • 11:47 addshore@tin: Synchronized wmf-config/CommonSettings-labs.php: gerrit:393577 LABS ONLY Enable FileImporter & FileExporter on BETA PT2/2 (T181383) (duration: 00m 45s)
  • 11:46 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: gerrit:393577 LABS ONLY Enable FileImporter & FileExporter on BETA PT1/2 (T181383) (duration: 00m 45s)
  • 11:44 addshore@tin: Synchronized wmf-config/extension-list-labs: gerrit:393576 BETA ONLY Add FileExporter & FileImporter to extension-list-labs (duration: 00m 45s)
  • 11:27 hashar: contint1001 enable Icinga "Disks space" notification again. It is no more complaing about Docker partitions | ping mutante | T178454
  • 11:17 godog: bootstrap restbase1007-a - T179422
  • 11:05 jdrewniak@tin: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 45s)
  • 11:05 jdrewniak@tin: Synchronized portals/prod/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 46s)
  • 10:53 moritzm: installing postgresql-common security updates
  • 10:08 ppchelko@tin: Finished deploy [cpjobqueue/deploy@e35aa05]: Revert using keep-alive (duration: 00m 22s)
  • 10:08 ppchelko@tin: Started deploy [cpjobqueue/deploy@e35aa05]: Revert using keep-alive
  • 09:42 ppchelko@tin: Finished deploy [cpjobqueue/deploy@b570d4e]: Make http agent use keep-alive (duration: 00m 48s)
  • 09:41 ppchelko@tin: Started deploy [cpjobqueue/deploy@b570d4e]: Make http agent use keep-alive
  • 09:14 godog: reimage restbase1007 - T179422
  • 08:55 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1097:3314 - T178359 (duration: 00m 43s)
  • 08:40 moritzm: installing openjdk security updates on hadoop, druid and kafka clusters
  • 08:27 marostegui: Deploy schema change on dbstore1002 and dbstore1001 - T174569
  • 08:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1097:3314 weight - T178359 (duration: 00m 45s)
  • 07:20 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1097:3314 weight - T178359 (duration: 00m 45s)
  • 07:10 marostegui: Stop MySQL on db1021 as it will be decommissioned - T181378
  • 06:53 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1021 from the config as it will be decommissioned - T181378 (duration: 00m 44s)
  • 06:52 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1021 from the config as it will be decommissioned - T181378 (duration: 00m 45s)
  • 06:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1097:3314 with low weight - T178359 (duration: 00m 46s)
  • 02:22 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.8) (duration: 05m 44s)

2017-11-25

  • 19:31 marostegui: Set 32:3 disk to offline on db1051
  • 16:09 kartik@tin: Finished deploy [cxserver/deploy@11aecc9]: Update cxserver to 0c242c0, Pin service-runner to 2.4.2 (duration: 03m 29s)
  • 16:05 kartik@tin: Started deploy [cxserver/deploy@11aecc9]: Update cxserver to 0c242c0, Pin service-runner to 2.4.2
  • 16:05 godog: unban statsd traffic from scb on graphite1001 - T181333
  • 15:45 ppchelko@tin: Finished deploy [cpjobqueue/deploy@e35aa05]: Rollback. Disable GC metric reporting T181333 (duration: 00m 31s)
  • 15:45 ppchelko@tin: Started deploy [cpjobqueue/deploy@e35aa05]: Rollback. Disable GC metric reporting T181333
  • 15:37 volans: restarted statsd-proxy on graphite1001 (died during investigation) T181333
  • 14:34 godog: rolling restart of cxserver to alleviate metrics leak - T181333
  • 14:26 godog: restart cxserver on scb100[34] - T181333
  • 14:10 godog: roll-restart cpjobqueue to alleviate metrics leak - T181333
  • 13:40 godog: drop incoming statsd from scb to graphite1001 temporarily - T181333
  • 08:32 ariel@tin: Finished deploy [dumps/dumps@ec21673]: fix abstracts recombine job (duration: 00m 02s)
  • 08:32 ariel@tin: Started deploy [dumps/dumps@ec21673]: fix abstracts recombine job

2017-11-24

  • 13:52 moritzm: removing git packages from jessie-wikimedia/experimental (replaced by component/git)
  • 13:24 moritzm: installing openjpeg2 updates (original security already got installed after initial release, but there was a binNMU for amd64)
  • 13:17 marostegui: Stop replication on db1097 to reimport and recompress commonswiki.watchlist
  • 12:54 jynus: reenabling puppet on db1071
  • 12:50 jynus: resetting replication on es1011 for consistency with other replica sets
  • 12:40 jynus: setting up s8 topology on eqiad
  • 12:38 jynus: disable puppet on db1071 and stop local s5 heartbeat there
  • 12:32 reedy@tin: Synchronized docroot/mediawiki/keys/: Fixup keys (duration: 00m 45s)
  • 12:13 marostegui: Enable GTID on es2018 - T181293
  • 11:57 marostegui: Disable puppet on es2018 - T181293
  • 11:50 jynus@tin: Synchronized wmf-config/db-codfw.php: depool es2018 T181293 (duration: 00m 45s)
  • 11:48 marostegui: Reboot es2018 after full-upgrade - T181293
  • 11:25 marostegui: Restart mysql on es2018
  • 11:25 jynus@tin: Synchronized wmf-config/db-eqiad.php: db2085:3318, db2086:3318 (duration: 00m 43s)
  • 11:24 jynus@tin: Synchronized wmf-config/db-codfw.php: Pool db2038, db2085:3318, db2086:3318 (duration: 00m 45s)
  • 10:55 marostegui: Restart MySQL on db2086 to move s5 to s8
  • 10:37 jynus: cancelling db2085 restart, only doing mysql:s5
  • 10:35 jynus: restarting db2085 (including both s5 and s3 instances)
  • 10:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool all future s8 slaves for a topology change - T177208 (duration: 00m 45s)
  • 10:22 moritzm: installing ca-cerfificates updates on trusty hosts
  • 09:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1097:3315 and db1092 (duration: 00m 45s)
  • 09:50 jynus: restarting db2045
  • 09:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1101:3318 db1097:3315 abd db1092 (duration: 00m 45s)
  • 08:49 marostegui: Stop MySQL on db1092
  • 08:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1101:3318 in s5 to warm it up and depool db1092 - T178359 T177208 (duration: 00m 45s)
  • 08:40 moritzm: installing java security updates on notebook* hosts
  • 08:38 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Slowly pool db1097:3315 - T178359 (duration: 00m 45s)
  • 08:37 marostegui@tin: Synchronized wmf-config/db-codfw.php: Slowly pool db1097:3315 - T178359 (duration: 00m 45s)
  • 08:20 moritzm: installing java security updates on meitnerium
  • 08:15 moritzm: installing java security updates on stat1004
  • 08:14 hashar: restarting jenkins on contint1001 for a java update
  • 08:07 elukey: re-enabling piwik on bohrium (only VM running on ganeti1006 atm) after mysql tables restore completed
  • 06:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Slowly pool db1101:3318 in s5 to warm it up - T178359 (duration: 00m 45s)
  • 06:46 marostegui@tin: Synchronized wmf-config/db-codfw.php: Slowly pool db1101:3318 in s5 to warm it up - T178359 (duration: 00m 49s)
  • 00:12 reedy@tin: Synchronized docroot/mediawiki/keys/: Add Brian Wolff's key (duration: 00m 45s)

2017-11-23

  • 23:42 reedy@tin: Synchronized wmf-config/: Simplify variable assignment (duration: 00m 47s)
  • 23:25 reedy@tin: Synchronized composer.lock: (no justification provided) (duration: 00m 44s)
  • 23:24 reedy@tin: Synchronized composer.json: (no justification provided) (duration: 00m 45s)
  • 23:23 reedy@tin: Synchronized phpcs.xml: (no justification provided) (duration: 00m 45s)
  • 22:39 demon@tin: Synchronized wmf-config/: style fixes, no-op (duration: 00m 47s)
  • 22:19 demon@tin: Synchronized wmf-config/CommonSettings.php: no-op, moving stuff around (duration: 00m 47s)
  • 21:34 ariel@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2251.codfw.wmnet
  • 21:15 apergos: rebooted mw2251 after unresponsive on mgmt console and no ping
  • 20:49 demon@tin: Synchronized static/images/project-logos/nowikimedia.png: logo update (duration: 02m 51s)
  • 17:29 akosiaris: set ganeti1006 as drained. ganeti1005 was already set. That will prevent scheduling VMs on those. T181121
  • 17:01 akosiaris: force powercycle on ganeti1006 T181121
  • 16:30 akosiaris: gnt-node failover ganeti1006
  • 15:50 jynus: disable puppet on db2023, db2038 for deployment whle transfer is ongoing
  • 15:23 moritzm: rebooting mwdebug* servers for update to 4.9.51
  • 14:10 marostegui: Stop Mysql on db1101.s5 to clone db1097 - T178359
  • 14:09 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1053 in s2 as vslow to replace db1021 - T134476 (duration: 00m 45s)
  • 13:13 jynus: sutting down db2023 and db2038 for cloning
  • 12:47 reedy@tin: Synchronized wmf-config/CommonSettings.php: GWToolset migratory config T87928 (duration: 00m 46s)
  • 12:40 moritzm: migrating instances off ganeti2001 / kernel reboot to 4.9.51
  • 12:10 moritzm: migrating instances off ganeti2002 / kernel reboot to 4.9.51
  • 11:52 moritzm: migrating instances off ganeti2003 / kernel reboot to 4.9.51
  • 11:37 kartik@tin: Finished deploy [cxserver/deploy@51b78ce]: T181209 Update cxserver to e8fe3f0 to fix Youdao MT (duration: 03m 01s)
  • 11:35 moritzm: migrating instances off ganeti2004 / kernel reboot to 4.9.51
  • 11:35 moritzm: migrating instances off ganeti200 / kernel reboot to 4.9.51
  • 11:34 kartik@tin: Started deploy [cxserver/deploy@51b78ce]: T181209 Update cxserver to e8fe3f0 to fix Youdao MT
  • 11:23 moritzm: migrating instances off ganeti2005 / kernel reboot to 4.9.51
  • 11:15 moritzm: migrating instances off ganeti2006 / kernel reboot to 4.9.51
  • 11:06 moritzm: migrating instances off ganeti2007 / kernel reboot to 4.9.51
  • 10:57 moritzm: migrating instances off ganeti2008 / kernel reboot to 4.9.51
  • 10:07 moritzm: rebooting sarin for update to 4.9.51
  • 09:16 jynus@tin: Synchronized wmf-config/db-codfw.php: mariadb: Switchover s5 codfw master (duration: 00m 45s)
  • 08:56 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1072 (duration: 00m 44s)
  • 08:44 jynus: stopping and restarting db2052
  • 08:31 moritzm: installing bdb security updates on trusty
  • 08:29 jynus: starting switchover of db2023 to db2052
  • 08:21 marostegui: Stop MySQL on db1072 to MySQL upgrade and kernel upgrade
  • 08:21 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1072 (duration: 00m 45s)
  • 07:28 marostegui: Stop MySQL on db1021 and db1053 to clone db1053
  • 07:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1021 - T134476 (duration: 00m 45s)
  • 07:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Move db1053 to s2 to replace db1021 as vslow, dump slave - T134476 (duration: 00m 45s)
  • 06:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1089 - T180045 (duration: 00m 45s)
  • 06:55 marostegui: Drop index on ores_classification on s1 - T180045
  • 06:53 marostegui: Drop index on ores_classification on s2 - T180045
  • 06:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1089 - T180045 (duration: 00m 45s)
  • 06:44 legoktm: legoktm@tin:~$ echo "https://www.mediawiki.org/keys/keys.html" | mwscript purgeList.php
  • 06:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1051 and db1063 in vslow service for s5 to warm them up for the s8 split - T177208 (duration: 00m 46s)
  • 05:58 kartik@tin: Finished deploy [cxserver/deploy@5b35ed5]: Update cxserver to e8fe3f0 (duration: 04m 13s)
  • 05:54 kartik@tin: Started deploy [cxserver/deploy@5b35ed5]: Update cxserver to e8fe3f0
  • 02:23 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.8) (duration: 05m 27s)
  • 01:36 demon@tin: Synchronized wmf-config/InitialiseSettings.php: dropping education program from cswiki (duration: 00m 45s)
  • 01:29 demon@tin: Synchronized wmf-config/CommonSettings.php: Set $wgCentralAuthGlobalBlockInterwikiPrefix (duration: 00m 45s)
  • 01:26 demon@tin: Synchronized wmf-config/InitialiseSettings-labs.php: no-op (duration: 00m 44s)
  • 01:24 demon@tin: Synchronized wmf-config/CommonSettings.php: abusefilter stuff (duration: 00m 45s)
  • 01:21 demon@tin: Synchronized wmf-config/InitialiseSettings.php: abusefilter stuff (duration: 00m 45s)
  • 01:19 demon@tin: Synchronized wmf-config/CommonSettings.php: abusefilter stuff (duration: 00m 45s)
  • 01:05 demon@tin: Synchronized docroot/mediawiki/keys/: update formatting, docs, etc (duration: 00m 46s)

2017-11-22

  • 21:15 mutante: running puppet on logstash hosts to apply config change 392897 (add log4j filter)
  • 21:11 demon@tin: Synchronized docroot/noc/conf/index.php: favicon fix (duration: 00m 46s)
  • 20:35 demon@tin: Synchronized dblists/: now with more alphabeticalizedness (duration: 00m 45s)
  • 20:25 demon@tin: Synchronized dblists/Makefile: no-op (duration: 00m 45s)
  • 20:23 demon@tin: Synchronized tests/noc-conf/NOCDblistTest.php: no-op (duration: 00m 45s)
  • 20:19 demon@tin: Synchronized wmf-config/LabsServices.php: no-op (duration: 00m 45s)
  • 20:14 demon@tin: Synchronized wmf-config/InitialiseSettings.php: adding national library of Israel to copy domains (duration: 00m 45s)
  • 20:13 otto@tin: Finished deploy [eventlogging/analytics@57234e7]: no-op: removing now unneeded code that might accidentally serialize userAgent to json string: T179625 (duration: 00m 04s)
  • 20:13 otto@tin: Started deploy [eventlogging/analytics@57234e7]: no-op: removing now unneeded code that might accidentally serialize userAgent to json string: T179625
  • 20:09 demon@tin: Synchronized robots.txt: block a nasty bot 💔 (duration: 00m 44s)
  • 20:02 demon@tin: Synchronized multiversion/bin/expanddblist: fix param warning (duration: 00m 45s)
  • 20:00 mepps: updated civicrm from a16e566 to a1022cf
  • 19:58 demon@tin: Synchronized wmf-config/InitialiseSettings.php: comments (duration: 00m 45s)
  • 19:56 demon@tin: Synchronized multiversion/MWScript.php: assume aawiki for purgeUrls (duration: 00m 45s)
  • 19:51 demon@tin: Synchronized docroot/: removing old foundation docroot (duration: 00m 46s)
  • 19:49 urandom: starting cassandra cleanups, restbase-200{1,3,5}-a - T179422
  • 19:41 demon@tin: Synchronized w/extract2.php: removing old portal support (duration: 00m 45s)
  • 18:47 demon@tin: Synchronized dblists/closed.dblist: closed transitionteamwiki (duration: 00m 45s)
  • 18:34 demon@tin: Synchronized wmf-config/InitialiseSettings-labs.php: no-op (duration: 00m 45s)
  • 18:32 demon@tin: Synchronized scap/plugins/updatewikiversions.py: minor fix (duration: 00m 45s)
  • 18:30 demon@tin: Pruned MediaWiki: 1.31.0-wmf.7 [keeping static files] (duration: 01m 46s)
  • 18:12 herron: re-enabling puppet agents after puppetdb postgres security updates
  • 18:09 moritzm: installing postgres security updates on nitrogen/puppetdb
  • 18:05 moritzm: installing postgres security updates on nihal/puppetdb
  • 18:02 herron: disabling puppet agents for puppetdb postgres security update
  • 17:30 demon@tin: Synchronized php-1.31.0-wmf.8/extensions/AdvancedSearch/: fixing layout issues in timeless (duration: 00m 46s)
  • 16:54 mepps: updated payments-wiki from 1ca91b1 to 6b3019b
  • 16:45 chasemp: disable puppet accross labtest things
  • 16:34 marostegui: Compress s4 on db1097 - T178359
  • 16:28 herron: starting canary deploy/cutover of codfw scb hosts to codfw puppet 4 masters
  • 16:16 elukey: restart druid broker,coordinator,historical daemons on druid100[123] to pick up new logging settings
  • 15:41 jynus: starting manually pt-heartbeat for s8 on db1071
  • 15:22 herron: beginning cut over of codfw db servers (^db2.*) to codfw puppet 4 masters
  • 14:49 jynus@tin: Synchronized wmf-config/db-codfw.php: mariadb: Setup s8 replica set on codfw (duration: 00m 45s)
  • 14:27 moritzm: installing libxml-libxml-perl security updates
  • 14:21 jynus: starting database topology changes for s8 on codfw T177208
  • 14:11 urandom: bootstrapping cassandra, restbase2004-c.codfw.wmnet - T179422
  • 13:43 apergos: one more round of labstore1006 <-- ms1001 rsync catchup
  • 13:37 moritzm: installing imagemagick security updates
  • 12:39 marostegui: Stop MySQL on db1053 to clone db1097.s4 - T178359
  • 12:38 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1053 - T178359 (duration: 00m 45s)
  • 12:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Start adapting the config to move db1097 to s4 and s5 as multi-instance rc slave T178359 (duration: 00m 45s)
  • 12:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1101.s7 - T178359 (duration: 00m 45s)
  • 11:51 jynus: starting dropping incorrectly created database on s7 amwikimedia (not to be confused with production wiki s3 amwikimedia)
  • 11:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1101.s7 - T178359 (duration: 00m 45s)
  • 11:19 akosiaris: gnt-node evacuate -s -f ganeti1005. T181121
  • 11:02 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1101.s7 - T178359 (duration: 00m 45s)
  • 10:54 akosiaris: gnt-node migrate -f ganeti1005. T181121
  • 10:51 marostegui: Drop index from ores_classification on s5 - T180045
  • 10:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1051 and db1063 in vslow service for s5 to warm them up for the s8 split - T177208 (duration: 00m 45s)
  • 10:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1101 to s5 and s7 as recentchanges multi-instance slave - T178359 (duration: 00m 45s)
  • 10:04 moritzm: running "scap pull" on mw1191, it's depooled and marked as "inactive", but health checks are triggering db errors
  • 09:35 bblack: cr[12]-ulsfo - switch static fallback LVS routes from lvs400[12] to lvs400[56]
  • 09:27 bblack: lvs@ulsfo - done switching primaries (host MED config) - lvs400[56] now primary for text/upload traffic
  • 09:11 akosiaris@tin: Finished deploy [parsoid/deploy@b150764]: T180211 (duration: 05m 05s)
  • 09:08 bblack: puppet disabled on lvs400[1256] for switching primaries
  • 09:06 akosiaris@tin: Started deploy [parsoid/deploy@b150764]: T180211
  • 09:04 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp2017.codfw.wmnet
  • 09:00 bblack: lvs4005 - reboot to clear experimental stuff
  • 08:16 bblack: backend restart on cp4024 (upload@ulsfo) - mailbox lag
  • 07:56 marostegui: Drop index from ores_classification on s3 - T180045
  • 07:50 marostegui: Drop index from ores_classification on s6 - T180045
  • 07:48 marostegui: Drop index from ores_classification on s7 - T180045
  • 07:29 _joe_: stopping the additional workers for htmlCacheUpdate (commons and ruwiki), adding one additional runner for refreshLinks on ruwiki
  • 06:43 marostegui: Stop MySQL on db1063 and db1051 (which is going to be recloned) - T177208
  • 06:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Move db1051 from s1 to s5 - T177208 (duration: 00m 53s)
  • 06:28 ejegg: updated payments-wiki from f871160 to 1ca91b1
  • 03:36 mutante: powercycled mw2251 which had gone down without further comment
  • 02:35 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.8) (duration: 11m 34s)
  • 01:09 hoo: Cleaned out remaining T180934 related log blow up on snapshot1007 (dumpwikidatajson-wikidata-20171120-all-0.log)
  • 00:43 mutante: gerrit restarting service to apply config change
  • 00:40 mutante: gerrit - re-enabling puppet to apply logstash change on cobalt, gerrit restart incoming (T141324)
  • 00:39 maxsem@tin: Synchronized php-1.31.0-wmf.8/extensions/CentralNotice/: https://gerrit.wikimedia.org/r/#/c/392754/ (duration: 00m 47s)
  • 00:11 maxsem@tin: Synchronized php-1.31.0-wmf.8/extensions/InputBox/: https://gerrit.wikimedia.org/r/#/c/392745/ (duration: 00m 45s)
  • 00:10 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/392576/ (duration: 00m 46s)
  • 00:07 bblack@neodymium: conftool action : set/pooled=yes; selector: name=nescio.wikimedia.org
  • 00:04 bblack@neodymium: conftool action : set/pooled=no; selector: name=nescio.wikimedia.org
  • 00:02 bblack@neodymium: conftool action : set/pooled=yes; selector: name=maerlant.wikimedia.org
  • 00:01 bblack@neodymium: conftool action : set/pooled=no; selector: name=maerlant.wikimedia.org
  • 00:01 bblack@neodymium: conftool action : set/pooled=yes; selector: name=maerlant.wikimedia.org
  • 00:00 bblack@neodymium: conftool action : set/pooled=yes; selector: name=chromium.wikimedia.org

2017-11-21

  • 23:55 bblack@neodymium: conftool action : set/pooled=no; selector: name=chromium.wikimedia.org
  • 23:30 mholloway-shell@tin: Finished deploy [mobileapps/deploy@fc01242]: Update mobileapps to 52d6a83 (duration: 04m 36s)
  • 23:26 mholloway-shell@tin: Started deploy [mobileapps/deploy@fc01242]: Update mobileapps to 52d6a83
  • 22:45 bblack@neodymium: conftool action : set/pooled=yes; selector: name=achernar.wikimedia.org
  • 22:38 bblack@neodymium: conftool action : set/pooled=no; selector: name=achernar.wikimedia.org
  • 22:36 bblack@neodymium: conftool action : set/pooled=yes; selector: name=hydrogen.wikimedia.org
  • 22:29 urandom: Bootstrapping Cassandra, restbase2004-b.codfw.wmnet (T179422)
  • 22:12 mutante: gerrit - temp disable puppet on cobalt (prod gerrit), test switching gerrit logging to logstash on gerrit2001 - gerrit:392079 gerrit:392083 T141324
  • 22:05 bblack@neodymium: conftool action : set/pooled=no; selector: name=hydrogen.wikimedia.org
  • 22:03 bblack@neodymium: conftool action : set/pooled=yes; selector: name=acamar.wikimedia.org
  • 22:02 bblack: repooling acamar for recdns
  • 21:26 ariel@tin: Finished deploy [dumps/dumps@16f92d6]: take 2: gzip namespace and abstract dumps; remove last configfile existence checks (duration: 00m 02s)
  • 21:26 ariel@tin: Started deploy [dumps/dumps@16f92d6]: take 2: gzip namespace and abstract dumps; remove last configfile existence checks
  • 20:45 bblack: recdns: puppet disabled on all, acamar depooled, careful deploys going on for anycast+recdns stuff
  • 20:42 ayounsi@neodymium: conftool action : set/pooled=no; selector: name=acamar.wikimedia.org
  • 20:26 ariel@tin: Finished deploy [dumps/dumps@16f92d6]: gzip namespace and abstract dumps; remove last configfile existence checks (duration: 00m 16s)
  • 20:26 ariel@tin: Started deploy [dumps/dumps@16f92d6]: gzip namespace and abstract dumps; remove last configfile existence checks
  • 20:22 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to 1.31.0-wmf.8
  • 20:00 thcipriani: finishing wmf.8 rollout, starting group2 to wmf.8
  • 19:21 addshore: SWAT done!
  • 19:20 addshore@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Add images.collection.cooperhewitt.org & *.dimu.org to wgCopyUploadsDomains. T180791 T180241 (duration: 00m 48s)
  • 19:11 addshore@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Enable AdvancedSearch on group0 PT2/2 (duration: 00m 49s)
  • 19:10 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable AdvancedSearch on group0 PT1/2 (duration: 00m 50s)
  • 18:40 mholloway-shell@tin: Finished deploy [mobileapps/deploy@dd41387]: Update mobileapps to 9d1602d (duration: 05m 09s)
  • 18:35 mholloway-shell@tin: Started deploy [mobileapps/deploy@dd41387]: Update mobileapps to 9d1602d
  • 18:28 smalyshev@tin: Finished deploy [wdqs/wdqs@c69c739]: Restore categories vocabulary to V003 (duration: 01m 55s)
  • 18:26 smalyshev@tin: Started deploy [wdqs/wdqs@c69c739]: Restore categories vocabulary to V003
  • 18:25 smalyshev@tin: Finished deploy [wdqs/wdqs@7d951d2]: Restore categories vocabulary to V003 (duration: 00m 11s)
  • 18:25 smalyshev@tin: Started deploy [wdqs/wdqs@7d951d2]: Restore categories vocabulary to V003
  • 16:45 marostegui: Compress s3 on db2085 - T178359
  • 16:35 papaul: powering down wtp2017 for disk replacement
  • 16:19 papaul: updating firmware on db2068
  • 15:31 _joe_: rolling restart of pybal on low-traffic in codfw, eqiad for the new depool thresholds for MW
  • 15:00 ppchelko@tin: Finished deploy [cpjobqueue/deploy@3e948de]: Revert: Temporarily disable deduplication (duration: 00m 26s)
  • 15:00 ppchelko@tin: Started deploy [cpjobqueue/deploy@3e948de]: Revert: Temporarily disable deduplication
  • 14:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore original weight for db1096 (duration: 00m 48s)
  • 14:39 ppchelko@tin: Finished deploy [cpjobqueue/deploy@aac3201]: Temporarily disable deduplication (duration: 00m 29s)
  • 14:39 ppchelko@tin: Started deploy [cpjobqueue/deploy@aac3201]: Temporarily disable deduplication
  • 14:29 zeljkof: EU SWAT finished
  • 14:25 bblack: cr[12]-ulsfo: allow lvs400[567] as PyBal neighbors for BGP
  • 14:18 zfilipin@tin: Synchronized php-1.31.0-wmf.8/extensions/TimedMediaHandler/MwEmbedModules/EmbedPlayer/resources/mw.EmbedPlayerOgvJs.js: SWAT: Disable wasm, use asm.js codec modules for Safari/Edge (T181022) (duration: 00m 49s)
  • 14:12 jdrewniak@tin: Synchronized portals: SWAT: Bumping portals to master (T128546) (duration: 00m 49s)
  • 14:11 jdrewniak@tin: Synchronized portals/prod/wikipedia.org/assets: SWAT: Bumping portals to master (T128546) (duration: 00m 50s)
  • 14:11 ppchelko@tin: Started restart [cpjobqueue/deploy@5341d94]: Restart to try increased maxSockets
  • 14:11 Pchelolo: restart cpjobqueue to try increasing maxSockets
  • 14:08 bblack: not-abnormally-quick reboot on lvs4005
  • 14:08 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: electron stuck - T174916
  • 13:45 ppchelko@tin: Finished deploy [cpjobqueue/deploy@5341d94]: Enable GC metrics reporting (duration: 00m 36s)
  • 13:44 ppchelko@tin: Started deploy [cpjobqueue/deploy@5341d94]: Enable GC metrics reporting
  • 13:39 _joe_: starting 2 manual runners for htmlcacheupdate on commons, 1 for htmlcacheupdate and 1 for refreshlinks on ruwiki, on terbium
  • 13:39 bblack: quick reboot on lvs4005
  • 12:02 kartik@tin: Finished deploy [cxserver/deploy@b87a27a]: Update cxserver to 4301987 (duration: 03m 24s)
  • 11:58 kartik@tin: Started deploy [cxserver/deploy@b87a27a]: Update cxserver to 4301987
  • 11:04 akosiaris: reboot ununpentium for serial tty change
  • 11:03 akosiaris: reboot oresrdb2001 for serial tty change
  • 11:02 akosiaris: reboot netmon1003 for serial tty change
  • 11:01 akosiaris: reboot install2002 for serial tty change
  • 10:57 akosiaris: reboot install1002 for serial tty change
  • 10:35 godog: bootstrap cassandra restbase2004-a - T179422
  • 10:11 ppchelko@tin: Finished deploy [cpjobqueue/deploy@e35aa05]: Set consumer_batch_size to 10 T181007 (duration: 00m 31s)
  • 10:10 ppchelko@tin: Started deploy [cpjobqueue/deploy@e35aa05]: Set consumer_batch_size to 10 T181007
  • 09:39 elukey: upload prometheus-druid-exporter 0.4 to jessie/stretch-wikimedia
  • 09:28 marostegui: Shutdown db2068 for maintenance - T180927
  • 09:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1096 (duration: 00m 49s)
  • 09:02 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1096 (duration: 00m 49s)
  • 08:39 marostegui: Drop index on db1089 enwiki.ores_classification - T180045
  • 08:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1096 with low weight - T178359 (duration: 00m 48s)
  • 08:27 marostegui: Reboot db1096 for kernel and MariaDB upgrade
  • 08:19 marostegui: Compress s5 on db1101 - T178359
  • 08:11 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1109 and db1110 - T180700 (duration: 00m 48s)
  • 07:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1109 and db1110 - T180700 (duration: 00m 49s)
  • 07:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1109 and db1110 in s5 with small weight - T180700 (duration: 00m 48s)
  • 06:50 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1087 - T179106 (duration: 00m 48s)
  • 06:47 marostegui: Remove index wb_terms_language from db1087 - https://phabricator.wikimedia.org/T179106
  • 06:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1087 - T179106 (duration: 00m 48s)
  • 06:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1096 - T178359 (duration: 00m 48s)
  • 06:27 marostegui: Stop MySQL on db1096 to clone db1101.s5 - T178359
  • 06:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1082 original weight - T177208 (duration: 00m 49s)
  • 02:58 mutante: phabricator back up
  • 02:55 mutante: phab1001 (phabricator prod) reboot for kernel upgrade
  • 02:48 krinkle@tin: Synchronized docroot/noc: clean up (duration: 00m 49s)
  • 02:23 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.7) (duration: 05m 42s)
  • 02:03 smalyshev@tin: Started deploy [wdqs/wdqs@7d951d2]: Restore categories vocabulary to V003
  • 00:19 catrope@tin: Synchronized php-1.31.0-wmf.8/resources/src/mediawiki.rcfilters/ui/mw.rcfilters.ui.ItemMenuOptionWidget.js: T180863 (duration: 00m 49s)

2017-11-20

  • 23:46 mutante: phab2001 - reboot for kernel upgrade
  • 23:35 legoktm@tin: Synchronized wmf-config/InitialiseSettings.php: emergency disable ORES on frwp/ruwp T181006 (duration: 00m 49s)
  • 23:35 awight: purge cache keys for ORES thresholds on frwiki and ruwiki
  • 23:25 awight@tin: Started deploy [ores/deploy@82a13ae]: Rollback ORES (take 3); 181006
  • 23:19 awight: aborted ORES rollback
  • 23:18 awight@tin: Started deploy [ores/deploy@95cd523]: Rollback ORES (take 2); 181006
  • 23:14 smalyshev@tin: Finished deploy [wdqs/wdqs@7d951d2]: Rollback categories vocabulary version due to a bug (duration: 08m 11s)
  • 23:11 awight: purged memcache key 'ruwiki:ORES:threshold_statistics:goodfaith:1’, T181006
  • 23:06 smalyshev@tin: Started deploy [wdqs/wdqs@7d951d2]: Rollback categories vocabulary version due to a bug
  • 22:55 awight@tin: Finished deploy [ores/deploy@5084251]: Rollback ORES; T179711 (duration: 01m 05s)
  • 22:55 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: no wmf.8 for group2. i hate my life
  • 22:55 awight: rolling back ORES to fix T181006
  • 22:54 eileen: update civicrm from 272580a to a16e566
  • 22:54 awight@tin: Started deploy [ores/deploy@5084251]: Rollback ORES; T179711
  • 22:50 Krinkle: MW HTTP 500 spike tracked as https://phabricator.wikimedia.org/T181006
  • 22:49 Krinkle: Sharp rise in HTTP 500 errors as of 22:05 (45 minutes ago)
  • 22:29 aaron@tin: Synchronized php-1.31.0-wmf.7/includes/libs/objectcache/WANObjectCache.php: fix cache key namespace (duration: 00m 49s)
  • 22:27 awight@tin: Finished deploy [ores/deploy@5084251]: Updating ORES to revscoring 2.0.10, T179711 (duration: 49m 54s)
  • 22:26 aaron@tin: Synchronized php-1.31.0-wmf.7/extensions/Collection: cache key name fix (duration: 00m 49s)
  • 22:11 aaron@tin: Synchronized php-1.31.0-wmf.8/includes/libs/objectcache/WANObjectCache.php: 7e74b49: namespace WAN cache variant keys (duration: 00m 48s)
  • 22:04 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.8
  • 21:52 bblack: ns2 back on eeden
  • 21:52 aaron@tin: Synchronized php-1.31.0-wmf.8/extensions/Collection: 3baebf4a: cache key name fix (duration: 00m 51s)
  • 21:44 bblack: rebooting eeden
  • 21:37 awight@tin: Started deploy [ores/deploy@5084251]: Updating ORES to revscoring 2.0.10, T179711
  • 21:33 bblack: routing ns2.wikimedia.org to radon for eeden reboot
  • 21:19 bblack: routing ns0.wikimedia.org back to radon post-reboot
  • 21:13 bblack: rebooting radon
  • 21:07 bblack: re-routing ns0.wikimedia.org traffic to baham for radon reboot
  • 20:17 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Remove overlapping userrights (T101983) (duration: 19m 41s)
  • 20:16 bblack: ns1 dns traffic back to normal on baham
  • 20:16 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Revert "Enable Translate extension in amwikimedia (T180879)" (duration: 00m 48s)
  • 20:11 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Enable Translate extension in amwikimedia (T180879) (duration: 00m 49s)
  • 20:11 hashar: CI docker jobs were all broken due to a mistake. Should be back now. T177684
  • 20:07 ladsgroup@tin: Synchronized php-1.31.0-wmf.8/resources/src/mediawiki.rcfilters/ui/mw.rcfilters.ui.ItemMenuOptionWidget.js: RCFilters: Only apply excluded label to namespace items (T180863) (duration: 00m 49s)
  • 20:06 bblack: rebooting baham
  • 19:58 bblack: re-routing ns1.wikimedia.org traffic to radon for baham reboot
  • 19:52 chasemp: updating phab mail handler
  • 19:51 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Remove overlapping userrights (T101983) (duration: 00m 49s)
  • 19:47 ottomata: restarted coal with fixes for eventcapsule changes in T179625
  • 19:42 ladsgroup@tin: Synchronized wmf-config/throttle.php: Adjust throttle.php for dewiki workshop (T180046) (duration: 00m 49s)
  • 19:40 ladsgroup@tin: Synchronized wmf-config/throttle.php: Adjust throttle.php for dewiki workshop (T180046) (duration: 00m 48s)
  • 19:36 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Allow admins to remove users from MP3 uploaders user group (T180002) (duration: 00m 49s)
  • 19:31 ladsgroup@tin: Synchronized portals: SWAT: Bumping portals to master (T128546) (duration: 00m 51s)
  • 19:30 ladsgroup@tin: Synchronized portals/prod/wikipedia.org/assets: SWAT: Bumping portals to master (T128546) (duration: 00m 49s)
  • 19:22 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Switch submit button from 'save' to 'publish' on dewiki (duration: 00m 50s)
  • 18:51 chasemp: disable puppet across cloud things to rollout https://gerrit.wikimedia.org/r/#/c/392168/ slowly
  • 18:25 smalyshev@tin: Finished deploy [wdqs/wdqs@2e39b69]: Blazegraph and GUI update (duration: 01m 42s)
  • 18:23 smalyshev@tin: Started deploy [wdqs/wdqs@2e39b69]: Blazegraph and GUI update
  • 17:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1082 with low weight - T177208 (duration: 00m 49s)
  • 17:19 bd808: Sending Toolforge survey emails from silver for T177126
  • 17:00 moritzm: uploaded git 2.11-3+deb9u2+bpo8+wmf1 for component/git to apt.wikimedia.org/jessie-wikimedia
  • 16:43 marostegui: Reboot db1082 for kernel upgrade and MariaDB upgrade to 10.0.33
  • 16:39 ppchelko@tin: Finished deploy [cpjobqueue/deploy@174420f]: Revert: Temporary set consumer_batch_size to 50, forgot -f, checkout prev rev (duration: 03m 33s)
  • 16:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1109 and db1110 to the config depooled - T180700 (duration: 00m 48s)
  • 16:38 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1109 and db1110 to the config depooled - T180700 (duration: 00m 48s)
  • 16:36 ppchelko@tin: Started deploy [cpjobqueue/deploy@174420f]: Revert: Temporary set consumer_batch_size to 50, forgot -f, checkout prev rev
  • 16:32 ppchelko@tin: Finished deploy [cpjobqueue/deploy@f0610d3]: Revert: Temporary set consumer_batch_size to 50, forgot -f (duration: 00m 28s)
  • 16:32 ppchelko@tin: Started deploy [cpjobqueue/deploy@f0610d3]: Revert: Temporary set consumer_batch_size to 50, forgot -f
  • 16:24 marostegui: Add db1109 and db1110 to tendril - T180700
  • 16:06 ppchelko@tin: Finished deploy [cpjobqueue/deploy@f0610d3]: Revert: Temporary set consumer_batch_size to 50 (duration: 00m 17s)
  • 16:06 ppchelko@tin: Started deploy [cpjobqueue/deploy@f0610d3]: Revert: Temporary set consumer_batch_size to 50
  • 15:59 urandom: Use lz4 compression instead of deflate (T180804)
  • 15:45 jynus: shutting down db2068 for maintenance after depool T180927
  • 15:42 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2068 (duration: 00m 49s)
  • 15:34 chasemp: disable puppet for a merge across cloud things
  • 15:31 ppchelko@tin: Finished deploy [cpjobqueue/deploy@f0610d3]: Temporary set consumer_batch_size to 50 (duration: 00m 30s)
  • 15:30 ppchelko@tin: Started deploy [cpjobqueue/deploy@f0610d3]: Temporary set consumer_batch_size to 50
  • 15:03 herron: pointing codfw mw servers at codfw puppet 4 masters via puppetmaster2001
  • 14:39 zeljkof: EU SWAT finished
  • 14:34 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Single edit tab in Catalan Wikipedia (T180660) (duration: 00m 49s)
  • 14:25 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable wgNamespacesWithSubpages for hiwikiversity (T180913) (duration: 00m 48s)
  • 14:22 jynus: enable semi-sync replication on s5
  • 14:14 dcausse@tin: Synchronized wmf-config/CirrusSearch-common.php: T180805: Revert [cirrus] disable token count router (duration: 00m 49s)
  • 14:05 elukey: upload prometheus-druid-exporter 0.3 to jessie-wikimedia
  • 13:53 Pchelolo: disable puppet on scb100x and stop cpjobqueue to accumulate some backlog
  • 13:39 ppchelko@tin: Finished deploy [cpjobqueue/deploy@174420f]: Optimise committed offset calculation (duration: 00m 28s)
  • 13:39 ppchelko@tin: Started deploy [cpjobqueue/deploy@174420f]: Optimise committed offset calculation
  • 13:30 elukey: upload prometheus-druid-exporter 0.3 to stretch-wikimedia
  • 12:40 marostegui: Optimize db1063 wikidatawiki.wb_terms
  • 12:17 moritzm: uploaded hhvm 3.18.5+dfsg-1+wmf1+icu57 to apt.wikimedia.org (jessie-wikimedia/component/icu57) (HHVM build linked against a co-installable backport of icu57)
  • 11:46 moritzm: rebooting tungsten for update to 4.9.51
  • 11:41 moritzm: rebooting etherpad1001 (etherpad.wikimedia.org) for update to 4.9.51
  • 11:36 moritzm: rebooting pybal-test for update to 4.9.51
  • 11:27 moritzm: rebooting restbase-test cluster for update to 4.9.51
  • 11:23 ppchelko@tin: Finished deploy [cpjobqueue/deploy@bdcef23]: Various performance improvements in committing (duration: 00m 30s)
  • 11:22 ppchelko@tin: Started deploy [cpjobqueue/deploy@bdcef23]: Various performance improvements in committing
  • 11:13 moritzm: rebooting ruthenium for update to 4.9.51
  • 10:57 godog: reimage restbase2004 - T179422
  • 10:51 moritzm: rebooting cerium/praseodymium/xenon for update to 4.9.51
  • 10:42 moritzm: rebooting mwlog1001 for update to 4.9.51
  • 10:39 moritzm: rebooting mwlog2001 for update to 4.9.51
  • 10:23 hoo: Manually re-started the Wikidata entity JSON dump on snapshot1007 (T180934)
  • 10:19 dcausse: elastic/cirrus: reindexing english group0 and group1 wikis: T179945
  • 10:08 hashar: contint1001: sudo systemctl start jenkins
  • 10:05 moritzm: rebooting contint1001 for update to 4.9.51
  • 09:22 moritzm: rebooting hafnium for update to 4.9.51
  • 08:46 marostegui: Run mydumper for db1047.staging - T156844
  • 07:58 moritzm: installing procmail security updates
  • 07:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1097 - T178359 (duration: 00m 49s)
  • 06:41 marostegui: Stop MySQL on db1082 to clone db1109 and db1110 - T180700
  • 06:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1082 - T177208 (duration: 00m 48s)
  • 06:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore original weights for db1100 and db1071 - T180917 (duration: 00m 49s)
  • 06:15 marostegui: Reboot db2068 - T180927
  • 02:28 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.7) (duration: 06m 26s)

2017-11-19

  • 23:20 Jamesofur: removed 2FA for Ask21 T180889
  • 20:28 krinkle@tin: Synchronized docroot/noc/index.html: noc: Link to Grafana (duration: 00m 49s)
  • 19:50 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1100 (duration: 00m 49s)
  • 07:09 mobrovac@tin: Started restart [zotero/translators@a0c41c3]: Zotero eating up memory
  • 07:01 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: electron stuck - T174916

2017-11-18

  • 20:06 gehel: upgrade of elasticsearch eqiad complete - T178411
  • 18:03 reedy@tin: Synchronized multiversion/vendor/: bump (duration: 01m 14s)
  • 15:53 marostegui: Compress s5 on db2085 - T178359
  • 14:48 marostegui: Compress s3 on db2092 - T178359
  • 00:30 urandom: Bootstrapping restbase2006-b (T179422)

2017-11-17

  • 20:35 mutante: netmon1002 - rsync smokeping data back from local backup to show measurements made from eqiad as requested on T180812
  • 20:05 mutante: running puppet on all cache misc to switch smokeping web to eqiad
  • 20:02 mutante: T180812 copying smokeping data from 2001 to 1002 - netmon1002: /usr/bin/rsync -avp rsync://netmon2001.wikimedia.org/var-lib-smokeping /var/lib/smokeping/ | switching backend from codfw to eqiad
  • 19:37 mutante: T180812 - @netmon1002:# rsync -avp /var/lib/smokeping/ /root/backup/netmon1002/201711717/var/lib/smokeping/@netmon2001:/# rsync -avp /var/lib/smokeping/ /root/backup/netmon2001/201711717/var/lib/smokeping/
  • 19:36 mutante: @netmon1002:/var/lib/smokeping# rsync -avp /var/lib/smokeping/ /root/backup/netmon1002/201711717/var/lib/smokeping/
  • 18:37 anomie@tin: Synchronized wmf-config/InitialiseSettings.php: Setting wgCommentTableSchemaMigrationStage = MIGRATION_WRITE_BOTH on Beta Cluster, no prod change (duration: 00m 48s)
  • 18:36 anomie@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Setting wgCommentTableSchemaMigrationStage = MIGRATION_WRITE_BOTH on Beta Cluster, no prod change (duration: 00m 49s)
  • 17:43 demon@tin: Synchronized php-1.31.0-wmf.8/extensions/Wikidata/extensions/Wikibase/client/WikibaseClient.php: fix client dependencies (duration: 00m 49s)
  • 17:41 demon@tin: Synchronized php-1.31.0-wmf.8/extensions/Wikibase/client/WikibaseClient.php: fix client dependencies (duration: 00m 50s)
  • 17:21 bblack: enabling new normalization code for all upload@esams (done)
  • 17:19 bblack: enabling new normalization code for all upload@eqiad
  • 17:14 marostegui: Revert schema change on dbstore1001 - T180714
  • 17:12 marostegui: Move dbstore1001 under db1070 - T180714
  • 17:12 bblack: enabling new normalization code for all upload@codfw
  • 16:52 bblack: enabling new normalization code for all upload@ulsfo
  • 16:24 bblack: disabling puppet on all cp* (testing encoding patch)
  • 16:19 moritzm: uploaded boost 1.55.0+dfsg-3+wmf2+icu57 to apt.wikimedia.org for jessie-wikimedia/component/icu57 (needed for HHVM build linked against ICU 57)
  • 16:04 dcausse@tin: Synchronized wmf-config/CirrusSearch-common.php: T180795 [cirrus] disable token count router (duration: 00m 49s)
  • 15:43 chasemp: labservices1001:~# mv /var/zones/tools.eqiad.wmflabs /home/rush T180797
  • 15:26 urandom: Starting restbase2006-c w/ -Dcassandra.replace_address=10.192.48.51 (T179422)
  • 14:06 jynus: deploy master events to db1070
  • 14:04 chasemp: labstore1003 service nfs-kernel-server restart && service rsync start
  • 13:55 moritzm: uploaded boost 1.55.0+dfsg-3+wmf1+icu57 to apt.wikimedia.org for jessie-wikimedia/component/icu57 (needed for HHVM build linked against ICU 57)
  • 12:15 moritzm: installing openssl updates on poolcounters
  • 12:07 moritzm: installing openssl updates on graphite* hosts
  • 11:46 moritzm: installing openssl updates on etcd* hosts
  • 11:43 moritzm: installing openssl updates on dbproxy hosts
  • 11:33 moritzm: installing openssl updates on puppetmasters
  • 11:01 akosiaris: create webperf1001, webperf2001 in ganeti T179036
  • 10:50 akosiaris@tin: Synchronized wmf-config/db-eqiad.php: (no justification provided) (duration: 00m 49s)
  • 10:49 akosiaris: sync wmf-config/db-eqiad.php for T180724
  • 10:48 akosiaris@tin: scap aborted: (no justification provided) (duration: 00m 02s)
  • 10:48 akosiaris@tin: Started scap: (no justification provided)
  • 10:47 akosiaris: pool mw2251 T180724
  • 10:47 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw2251.codfw.wmnet
  • 10:01 moritzm: rebooting darmstadtium (docker registry) for update to 4.9.51
  • 09:59 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore original traffic for the s5 eqiad hosts that were out for hours due to schema change revert after s5 master crash (duration: 00m 48s)
  • 09:54 moritzm: rebooting labnodepool* for update to 4.9.51
  • 09:38 ppchelko@tin: Finished deploy [cpjobqueue/deploy@2141162]: Bump concurrency even more (duration: 00m 29s)
  • 09:38 ppchelko@tin: Started deploy [cpjobqueue/deploy@2141162]: Bump concurrency even more
  • 09:20 godog: start restbase2006-c instead, restbase2006-b failed and -c shows as "down" - T179422
  • 09:13 jynus: changing master of db1071 (63 -> 70)
  • 09:10 gehel: cleanup leftover titlesuggest indices on elasticsearch eqiad (jawiki, frwiki, ptwiki)
  • 09:05 ppchelko@tin: Finished deploy [cpjobqueue/deploy@cbd25d3]: Bump overall concurrency to get rid of RecordLintJob backlog (duration: 00m 35s)
  • 09:04 ppchelko@tin: Started deploy [cpjobqueue/deploy@cbd25d3]: Bump overall concurrency to get rid of RecordLintJob backlog
  • 08:54 godog: bootstrap restbase2006-b - T179422
  • 08:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for the s5 eqiad hosts that were out for hours due to schema change revert after s5 master crash (duration: 00m 50s)
  • 08:04 elukey: reboot stat100[456] for kernel updates
  • 07:51 marostegui: Revert schema change on dbstore1002 - T180714
  • 07:50 marostegui: Move dbstore1002 under db1070 - T180714
  • 07:33 marostegui: Enable GTID on all the eqiad up-to-date hosts, only pending db1071 - T180714
  • 07:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool s5 eqiad hosts after reverting schema change - T180714 (duration: 00m 50s)
  • 06:34 marostegui: Revert schema changes on s5 codfw master with replication enabled, lag will be generated on codfw s5 - T180714
  • 04:43 krinkle@tin: Synchronized wmf-config/CommonSettings-labs.php: no-op for labs (duration: 00m 51s)
  • 01:32 ejegg: re-enabled donation queue consumer
  • 01:28 ejegg: updated CiviCRM from 8454e06 to 272580a
  • 01:07 ejegg: updated CiviCRM from 0b8ceea to 8454e06
  • food: disabled donations queue consumer for thank you subject update
  • 00:41 krinkle@tin: Synchronized wmf-config/PhpAutoPrepend.php: I60cce0 (duration: 00m 48s)
  • 00:40 krinkle@tin: Synchronized wmf-config/StartProfiler.php: I60cce0 (duration: 00m 49s)
  • 00:37 krinkle@tin: Synchronized wmf-config/profiler.php: I60cce0 (duration: 00m 48s)
  • 00:23 urandom: Decommissioning Cassandra, restbase1014-c.eqiad.wmnet (T179422)
  • 00:22 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/390131/3 (duration: 00m 49s)

2017-11-16

  • 23:50 mutante: wezen - systemctl restart rsyslog
  • 21:52 mepps: updated civicrm from 22f6532 to 0b8ceea
  • 21:05 mepps: updated payments-wiki from 210cb37 to f871160
  • 20:51 apergos: one more catchup rsync from ms1001 to labstore1006 kicking off
  • 20:15 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool only 'good' servers - third try (duration: 00m 49s)
  • 20:11 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool only 'good' servers - second try (duration: 00m 49s)
  • 20:10 demon@tin: Unlocked for deployment [operations/mediawiki-config]: No deploys, recovering from downtime (more narrow locking) (duration: 06m 49s)
  • 20:03 demon@tin: Locking from deployment [operations/mediawiki-config]: No deploys, recovering from downtime (more narrow locking) (planned duration: 360m 00s)
  • 20:02 twentyafterfour: deploy a9613b4 to hotfix T180706
  • 19:56 demon@tin: Unlocked for deployment [ALL REPOSITORIES]: No deploys, recovering from downtime (duration: 50m 28s)
  • 19:54 ayounsi@tin: Finished deploy [netbox/deploy@19f4f65]: (no justification provided) (duration: 00m 04s)
  • 19:54 ayounsi@tin: Started deploy [netbox/deploy@19f4f65]: (no justification provided)
  • 19:49 akosiaris: schedule extra downtime for s5 slaves
  • 19:46 urandom: Bootstapping Cassandra restbase2006-a (T179422)
  • 19:13 jynus: reset slave all on db1070 @ db1063-bin.001382:234464548
  • 19:05 demon@tin: Locking from deployment [ALL REPOSITORIES]: No deploys, recovering from downtime (planned duration: 360m 00s)
  • 19:05 demon@tin: Unlocked for deployment [ALL REPOSITORIES]: Dealing with outage, no deploys for now (duration: 01m 22s)
  • 19:03 demon@tin: Locking from deployment [ALL REPOSITORIES]: Dealing with outage, no deploys for now (planned duration: 60m 00s)
  • 18:48 marostegui: dewikipedia and wikidata currently back to writable
  • 18:44 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool only 'good' servers (duration: 00m 48s)
  • 18:41 mutante: dewikipedia and wikidata currently back to readonly-mode while wikidata is being worked on
  • 18:41 marostegui: Set s5 master read_only
  • 18:25 akosiaris: set mw2251 to inactive. T180724
  • 18:24 akosiaris@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=mw2251.codfw.wmnet
  • 17:57 gehel: silencing wdqs
  • 17:55 jynus@tin: Synchronized wmf-config/db-eqiad.php: Failover db1063 to db1070 (duration: 00m 46s)
  • 17:44 XioNoX: merging netbox CR
  • 17:41 bblack: disable port ge-5/0/39 on asw-c-eqiad (db1063)
  • 17:40 urandom: Decommissioning Cassandra, restbase1014-b.eqiad.wmnet (T179422)
  • 17:36 jynus: stopping slave on db1070
  • 17:28 urandom: Converting 'enwiki parsoid' to size-tiered compaction (T179422)
  • 17:19 demon@tin: Finished scap: consistency (duration: 25m 12s)
  • 17:18 bblack: Temporary GeoDNS routing changes (eqsin traffic simulation using ulsfo) - https://gerrit.wikimedia.org/r/#/c/391357/ - expecting ~24h, West Asia latencies will probably increase, spike in cache misses, etc...
  • 17:18 urandom: Converting 'wikipedia parsoid' to size-tiered compaction (T179422)
  • 17:16 godog: reimage restbase2006 - T179422
  • 17:15 urandom: Converting 'commons mobile' to size-tiered compaction (T179422)
  • 17:05 urandom: Converting 'others mobile' to size-tiered compaction (T179422)
  • 16:54 demon@tin: Started scap: consistency
  • 16:54 godog: reimage restbase2006 - T179422
  • 16:48 jynus: stop and restart db1071 for upgrade and reconfiguration
  • 16:48 herron: beginning gradual cutover of codfw mw systems to puppet 4 master puppetmaster2001
  • 16:48 godog: upgrade hpsa firmware to 6.06 on restbase2006 - T141756
  • 16:43 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1034 and db1079 original weight (duration: 00m 48s)
  • 16:29 jynus@tin: Synchronized wmf-config/db-eqiad.php: mariadb: Depool db1071, pool db1104 as api (duration: 00m 49s)
  • 16:21 moritzm: restarting apache on labs puppet masters to pick up openssl updates
  • 16:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1034 weight (duration: 00m 48s)
  • 15:47 moritzm: rebooting ores1* for update to 4.9.51
  • 15:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1034 weight (duration: 00m 49s)
  • 15:33 moritzm: rebooting seaborgium (slapd) for update to 4.9.51
  • 15:22 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2038 - T178359 (duration: 00m 49s)
  • 15:16 milimetric@tin: Finished deploy [analytics/refinery@4ef15d3]: Mainly deploying the interlanguage navigation dataset (duration: 14m 29s)
  • 15:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1034 weight (duration: 00m 48s)
  • 15:10 godog: upgrade prometheus-redis-exporter to 0.13-1 - T148637
  • 15:06 moritzm: installing postgres security updates on labsdb1006/1007
  • 15:02 milimetric@tin: Started deploy [analytics/refinery@4ef15d3]: Mainly deploying the interlanguage navigation dataset
  • 15:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1051 - T177208 (duration: 00m 48s)
  • 15:00 herron: beginning cut over of codfw canary appservers to puppet 4 master puppetmaster2001
  • 14:55 moritzm: rebooting serpens (slapd) for update to 4.9.51
  • 14:50 elukey: updating puppet compiler's facts (following https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource:Puppet3-diffs#FAQ)
  • 14:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1034 weight - T178359 (duration: 00m 49s)
  • 14:26 dcausse: EU swat done
  • 14:23 moritzm: rebooting kubetcd* to 4.9.51
  • 14:18 dcausse@tin: Synchronized wmf-config/CirrusSearch-common.php: T177913: [cirrus] Add overridden iw prefix for svwiki (2/2) (duration: 00m 48s)
  • 14:16 dcausse@tin: Synchronized wmf-config/InitialiseSettings.php: T177913: [cirrus] Add overridden iw prefix for svwiki (1/2) (duration: 00m 50s)
  • 14:02 gehel: starting upgrade of elasticsearch eqiad - T178411
  • 13:46 moritzm: rebooting bohrium (piwik host) for update to 4.9.51
  • 13:33 moritzm: installing openssl updates on es* hosts
  • 13:24 moritzm: rebooting dubnium/pollux (openldap corp mirror) for update to 4.9.51
  • 13:12 moritzm: installing openssl updates on pc* hosts
  • 13:07 elukey: restart aqs on aqs100[5-9] to apply localQuorum (https://gerrit.wikimedia.org/r/391765) - T164348
  • 13:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1034 weight - T178359 (duration: 00m 49s)
  • 12:54 moritzm: rebooting radium (tor relay) for update to 4.9.51
  • 12:34 moritzm: installing openssl updates on memcached/redis clusters
  • 12:27 hoo: Updated operations/dumps/dcat (ea4e75..7734e04) on snapshot1007
  • 12:09 moritzm: rebooting dbmonitor* hosts for update to 4.9.51
  • 12:03 addshore@tin: Synchronized php-1.31.0-wmf.8/extensions/Wikidata/extensions/Constraints: T180665 Manually applied: Fix SparqlHelper::getCacheMaxAge() (duration: 00m 52s)
  • 11:58 addshore@tin: Synchronized php-1.31.0-wmf.8/extensions/WikibaseQualityConstraints: T180665 Fix SparqlHelper::getCacheMaxAge() (duration: 00m 53s)
  • 11:42 moritzm: installing openssl updates on conf* clusters
  • 11:37 addshore@tin: Synchronized wmf-config/Wikibase-production.php: testwikidata only, T180665, WBQC configuration for testwikidatawiki (duration: 00m 49s)
  • 11:32 addshore: addshore@terbium:~$ mwscript extensions/WikibaseQualityConstraints/maintenance/ImportConstraintStatements.php --wiki=testwikidatawiki > importConstraintStatements.log
  • 11:21 godog: upgrade prometheus to 1.8.1+ds+k8s-1 in ulsfo/esams/eqiad - T177395
  • 11:09 moritzm: rebooting debug proxies (hassium/hassaleh) for update to 4.9.51
  • 11:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1034 weight - T178359 (duration: 00m 48s)
  • 11:01 jynus: shutting down labsdb1010 to clone to labsdb1009
  • 10:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1034 with low weight - T178359 (duration: 00m 48s)
  • 10:26 addshore: Kill the build deploy slot done!
  • 10:24 addshore@tin: Synchronized php-1.31.0-wmf.8/extensions/Wikibase: T180634 Bring Wikibase up to date with .8 branch of Wikidata build extension (duration: 01m 46s)
  • 10:15 addshore@tin: Synchronized php-1.31.0-wmf.8/extensions/WikibaseQualityConstraints: T180634 Bring WikibaseQualityConstraints up to date with .8 branch of Wikidata build extension (duration: 00m 53s)
  • 09:59 moritzm: rebooting prometheus servers in eqiad for update to 4.9.51
  • 09:44 elukey: restart aqs on aqs1004 to apply localQuorum (https://gerrit.wikimedia.org/r/391765) - T164348
  • 09:38 moritzm: rebooting prometheus servers in codfw for update to 4.9.51
  • 09:30 moritzm: uploaded icu57.1-6+wmf2 to jessie-wikimedia/component/icu57
  • 09:23 godog: bootstrap restbase2002-c - T179422
  • 09:19 godog: upgrade grafana to 4.6.1 on https://grafana.wikimedia.org/ - T180428
  • 08:19 marostegui: Deploy schema change on db1092 - T174569
  • 08:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1092 - T174569 (duration: 00m 49s)
  • 07:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1106 (duration: 00m 49s)
  • 07:40 marostegui: Stop MySQL on db1034 to copy its content to db1101.s7 - T178359
  • 07:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1034 - T178359 (duration: 00m 49s)
  • 07:17 ppchelko@tin: Finished deploy [eventlogging/eventbus@872cfb3]: Revert gerrit 302372 due to AssertionError T180017 (duration: 00m 14s)
  • 07:17 ppchelko@tin: Started deploy [eventlogging/eventbus@872cfb3]: Revert gerrit 302372 due to AssertionError T180017
  • 06:54 marostegui: Deploy alter table on db1096 - T174569
  • 06:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1096 - T174569 (duration: 00m 48s)
  • 06:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1099 and db1071 - T174569 (duration: 00m 49s)
  • 06:15 smalyshev@tin: Finished deploy [wdqs/wdqs@b44cf27]: data reload/T176593 (duration: 00m 28s)
  • 06:14 smalyshev@tin: Started deploy [wdqs/wdqs@b44cf27]: data reload/T176593
  • 02:26 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.7) (duration: 07m 54s)
  • 02:09 twentyafterfour: phabricator database migrations complete, service is back online
  • 01:28 twentyafterfour: Phabricator will be offline for a couple of minutes while I apply database migrations.
  • 01:22 twentyafterfour: updating phabricator (belatedly)
  • 01:05 demon@tin: Synchronized scap/plugins/prep.py: no-op, co-master sync (duration: 00m 55s)
  • 00:59 tzatziki: Removing 2FA from AuburnPilot (T180654)
  • 00:59 tzatziki: Removing 2FA from Twotwo2019 (T180438)
  • 00:42 eileen: update civicrm from b99a9cf to 22f6532

2017-11-15

  • 22:48 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: wikidatawiki back to wmf.7
  • 22:21 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.8
  • 22:11 urandom: Setting keyspaces erroneously configured for leveled compaction, to use size-tiered (T180568)
  • 22:05 madhuvishy: Kicking off re-enabling puppet and puppet runs across Cloud VPS instances
  • 21:33 demon@tin: Synchronized php: symlink swap (duration: 00m 49s)
  • 21:21 madhuvishy: Disabling puppet across cloud VPS through cumin on labpuppetmaster1001
  • 21:01 ottomata: restarting kafka-jumbo brokers to update to Kafka 0.11.0.1
  • 20:52 urandom: Restarting restbase2002-a.codfw.wmnet (T180568)
  • 20:40 mutante: restarting gerrit to enable 'large file support'-plugin gerrit:391635
  • 20:38 addshore@tin: Finished scap: Update extensions/Wikidata to new wmf/1.31.0-wmf.8 branch (again) T180539 (duration: 21m 48s)
  • 20:17 addshore@tin: Started scap: Update extensions/Wikidata to new wmf/1.31.0-wmf.8 branch (again) T180539
  • 20:11 otto@tin: Finished deploy [eventlogging/eventbus@872cfb3]: deploying kafka-futures change to kafka1001 only, will apply async with https://gerrit.wikimedia.org/r/#/c/391634/ (duration: 00m 20s)
  • 20:11 otto@tin: Started deploy [eventlogging/eventbus@872cfb3]: deploying kafka-futures change to kafka1001 only, will apply async with https://gerrit.wikimedia.org/r/#/c/391634/
  • 19:53 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Add BP and WP aliases for project namespace on mwlwiki (T180052) (duration: 00m 48s)
  • 19:48 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable SandboxLink on mwlwiki (T180052) (duration: 00m 48s)
  • 19:41 demon@tin: Finished deploy [gerrit/gerrit@bce982f]: adding lfs plugin @ 2.13.9 (duration: 00m 08s)
  • 19:41 demon@tin: Started deploy [gerrit/gerrit@bce982f]: adding lfs plugin @ 2.13.9
  • 19:40 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Disable EventLogging for Popups (T178500) (duration: 00m 49s)
  • 19:37 gehel: upgrade of elasticsearch codfw completed, cluster still recovering - T178411
  • 19:34 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable Minerva download icon on all wikis (T179914) (duration: 00m 48s)
  • 19:30 chasemp: labstore1004:~# service nfs-kernel-server restart
  • 19:19 catrope@tin: Synchronized php-1.31.0-wmf.8/resources/src/mediawiki.rcfilters/mw.rcfilters.UriProcessor.js: T180577 (duration: 00m 49s)
  • 19:16 catrope@tin: Synchronized php-1.31.0-wmf.7/skins/MinervaNeue/resources/skins.minerva.scripts/init.js: Limit download button to Google Chrome (T179529, T179914) (duration: 00m 49s)
  • 19:14 catrope@tin: Synchronized wmf-config/flaggedrevs.php: Enable RCFilters on all remaining wikis (T177445) (duration: 00m 49s)
  • 19:01 urandom: Restarting Cassandra instances on restbase2001.codfw.wmnet (T180568)
  • 18:38 Jamesofur: removed 2FA from User:Einsbor after verification for votewiki, stewardwiki and SUL
  • 18:20 moritzm: rebooting auth* servers for update to 4.9.51
  • 18:20 moritzm: rebooting auth
  • 17:57 demon@tin: Synchronized docroot/search.wikimedia.org/index.php: removing support for non-wikipedias (duration: 00m 49s)
  • 16:03 addshore@tin: Finished scap: Revert partial scap: Update extensions/Wikidata to new wmf/1.31.0-wmf.8 branch, second try (T180539) (duration: 23m 48s)
  • 15:56 oblivian@tin: Finished deploy [docker-pkg/deploy@9b319d2]: Adding do extension to jinja2, needed for contint images (duration: 00m 18s)
  • 15:56 oblivian@tin: Started deploy [docker-pkg/deploy@9b319d2]: Adding do extension to jinja2, needed for contint images
  • 15:55 ema: restart pybal on lvs[12]00[36] for config change https://gerrit.wikimedia.org/r/#/c/389964/
  • 15:43 jynus: shutting down labsdb1009 for maintenance T179244
  • 15:40 addshore@tin: Started scap: Revert partial scap: Update extensions/Wikidata to new wmf/1.31.0-wmf.8 branch, second try (T180539)
  • 15:29 moritzm: installing perl security updates on trusty (Debian hosts fixed two months ago)
  • 15:14 ladsgroup@tin: scap aborted: Update extensions/Wikidata to new wmf/1.31.0-wmf.8 branch, second try (T180539) (duration: 03m 48s)
  • 15:13 moritzm: removing unused kernels from prometheus*
  • 15:11 ladsgroup@tin: Started scap: Update extensions/Wikidata to new wmf/1.31.0-wmf.8 branch, second try (T180539)
  • 15:05 moritzm: rebooting kubestagetcd* for update to 4.9.51
  • 14:57 ladsgroup@tin: scap aborted: Update extensions/Wikidata to new wmf/1.31.0-wmf.8 branch (T180539) (duration: 00m 31s)
  • 14:57 ladsgroup@tin: Started scap: Update extensions/Wikidata to new wmf/1.31.0-wmf.8 branch (T180539)
  • 14:42 Amir1: deployed ORES change for wikidata (gerrit:391197, phab:T180450)
  • 14:41 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 49s)
  • 14:37 mobrovac: restbase creating Cassandra 3 revision tables on restbase1009 - T179421
  • 14:30 zfilipin@tin: Synchronized wmf-config/: SWAT: Remove wgContentTranslationEnableSuggestions (duration: 00m 51s)
  • 14:27 ema: cache_upload: upgrade varnish to 4.1.8-1wm2
  • 14:22 zfilipin@tin: Synchronized wmf-config/CommonSettings-labs.php: SWAT: Beta: Explicitly set cookieDomain for ContentTranslationSiteTemplates (T149879) (duration: 00m 49s)
  • 14:18 ema: repool cp4024 T174891
  • 14:17 mutante: wtp2017 - systemctl start ferm (ferm wasnt running due to failed DNS lookup for prometheus2003 sometime in the past)
  • 14:12 zfilipin@tin: Synchronized php-1.31.0-wmf.8/extensions/EventBus/EventBus.php: SWAT: Increase request timeout to match kafka produce timeout (T180017) (duration: 00m 50s)
  • 14:11 godog: upgrade hpsa firmware to 6.06 on restbase2004 - T180562 T141756
  • 14:10 ema: upgrade varnish to 4.1.8-1wm2 on cp4024 (cache_upload, depooled)
  • 14:04 jynus: reload haproxy on dbproxy1010
  • 14:00 moritzm: installing openssl updates on kafka and hadoop clusters
  • 13:34 ema: cache_text: upgrade varnish to 4.1.8-1wm2
  • 13:31 ppchelko@tin: Started restart [changeprop/deploy@065a06e]: Restart to rebalance all rules T179684
  • 13:14 moritzm: rebooting video scalers in eqiad for update to 4.9.51 (and to pick up openssl update)
  • 13:07 ema: upgrade varnish to 4.1.8-1wm2 on cp3030 (cache_text)
  • 12:41 elukey: re-enable eventlogging after maintenance
  • 12:24 jynus: deploying dns change for m4-master
  • 12:19 ema: cache_misc: upgrade varnish to 5.1.3-1wm3
  • 12:09 elukey: executed sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=65 on all jobrunners
  • 12:01 ema: upgrade varnish to 5.1.3-1wm3 on cp3007 (cache_misc)
  • 11:58 ema: varnish 4.1.8-1wm2 uploaded to apt.w.o (main)
  • 11:50 ema: varnish 5.1.3-1wm3 uploaded to apt.w.o (experimental)
  • 11:45 jynus: restart haproxy on dbproxy1009
  • 11:32 marostegui: Stop MySQL on db1046
  • 10:58 moritzm: rebooting job runners in eqiad for update to 4.9.51 (and to pick up openssl update)
  • 10:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1106, optimizing wikidatawiki.wb_terms (duration: 00m 49s)
  • 10:09 ppchelko@tin: Finished deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x attempt 4 T179786 (duration: 04m 51s)
  • 10:04 ppchelko@tin: Started deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x attempt 4 T179786
  • 10:04 ppchelko@tin: Finished deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x attempt 3, force T179786 (duration: 04m 44s)
  • 09:59 ppchelko@tin: Started deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x attempt 3, force T179786
  • 09:59 ppchelko@tin: Finished deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x attempt 2 T179786 (duration: 03m 32s)
  • 09:58 ppchelko@tin: (no justification provided)
  • 09:55 ppchelko@tin: Started deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x attempt 2 T179786
  • 09:55 ppchelko@tin: Finished deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x T179786 (duration: 02m 44s)
  • 09:53 godog: reboot restbase2004 - T180562
  • 09:52 ppchelko@tin: Started deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x T179786
  • 09:52 mobrovac@tin: Started restart [restbase/deploy@c76a665]: Pick up the new seeds definition - T179422
  • 09:52 ppchelko@tin: Started deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x T179786
  • 09:32 godog: restart cassandra on restbase2002-b - T180568
  • 09:30 moritzm: updating openssl on database hosts
  • 09:19 godog: restart cassandra on restbase2001-c - T180568
  • 09:15 marostegui: Stop mysql on db1046 to transfer its content to db1107 - T177405
  • 09:08 elukey: stop eventlogging on eventlog1001, eventlogging replication on db1108/db1047/dbstore1002 as preparation steps to migrate the log db from db1046 to db1107
  • 09:02 jynus: rebooting labsdb1010 for kernel upgrade
  • 08:51 elukey: reboot thorium (hosting all analytics websites) for kernel updates
  • 08:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1101 - going to convert it to mult-instance T178359 (duration: 00m 48s)
  • 07:49 marostegui: Deploy schema change on db1071 and db1099 (s5) - T174569
  • 07:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1099 and db1071 - T174569 (duration: 00m 49s)
  • 07:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1106 and db1100 - T174569 (duration: 00m 49s)
  • 05:33 subbu: subbu@terbium (followup to the linter-reparse.py script log entry) it is safe to kill -9 the script on terbium anytime if there are any problems because of it
  • 05:27 subbu: subbu@terbium running linter-reparse.py script to initialize baseline linter categories for all wikis (12 hours in so far .. expected to run for ~2 weeks). hits parsoid eqiad cluster.
  • 04:06 demon@tin: Synchronized docroot/search.wikimedia.org/robots.txt: go away robots / kill some 404s (duration: 00m 50s)
  • 02:29 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.7) (duration: 05m 57s)
  • 00:59 ejegg: updated payments-wiki from d150287 to 210cb37
  • 00:53 Reedy: going to restart zuul as it's got backed up
  • 00:33 maxsem@tin: Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/391220/ (duration: 00m 49s)
  • 00:18 ejegg: updated SmashPig payments listener from 5262d53 to 45aa626
  • 00:12 ejegg: updated CiviCRM from f571c67 to b99a9cf

2017-11-14

  • 23:29 mutante: releases1001 - chmod g+w /srv/org/wikimedia/releases/mediawiki/1.*
  • 23:01 urandom: Decommissioning Cassandra, restbase1014-a.eqiad.wmnet (T179422)
  • 22:57 demon@tin: Synchronized w/: Removing mobilelanding from global w/, few sites actually need it (duration: 00m 49s)
  • 22:55 demon@tin: Synchronized docroot/m.wikipedia.org/w/mobilelanding.php: symlink -> real file (duration: 00m 50s)
  • 21:42 no_justification: gerrit: restarted services on master/cobalt, things will flap for a second
  • 21:38 mutante: gerrit2001 - restarted gerrit
  • 21:17 ebernhardson@tin: Finished deploy [search/mjolnir/deploy@494e0c6]: redeploy mjolnir submodule bump to master (duration: 02m 11s)
  • 21:15 ebernhardson@tin: Started deploy [search/mjolnir/deploy@494e0c6]: redeploy mjolnir submodule bump to master
  • 21:03 ebernhardson@tin: Finished deploy [search/mjolnir/deploy@77310bc]: update mjolnir to master (duration: 02m 05s)
  • 21:01 ebernhardson@tin: Started deploy [search/mjolnir/deploy@77310bc]: update mjolnir to master
  • 20:37 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.8
  • 20:15 bblack: reboot cp4024
  • 19:58 ebernhardson@tin: Finished deploy [search/mjolnir/deploy@b20f0da]: test mjolnir deployment (duration: 02m 12s)
  • 19:57 urandom: Bootstrapping restbase2002-b.codfw.wmnet (T179422)
  • 19:56 ebernhardson@tin: Started deploy [search/mjolnir/deploy@b20f0da]: test mjolnir deployment
  • 19:47 demon@tin: Finished scap: bootstrap wmf.8 (duration: 44m 37s)
  • 19:32 chasemp: clean up instances in error state in testlabs project
  • 19:10 jynus: restart db2034 for mariadb upgrade
  • 19:09 chasemp: for i in `OS_TENANT_NAME=testlabs openstack server list | grep stress | awk '{print $2}'`; do echo $i; OS_TENANT_NAME=testlabs openstack server delete $i; sleep 30; done T171473
  • 19:02 demon@tin: Started scap: bootstrap wmf.8
  • 18:32 arlolra: Updated Parsoid to e71937d0 (T178253)
  • 18:31 mholloway-shell@tin: Finished deploy [mobileapps/deploy@9b10959]: Redeploying: Update mobileapps to c002862 (duration: 29m 12s)
  • 18:28 moritzm: upgraded nginx on notebook* to 1.13.6
  • 18:26 madhuvishy: Upgraded notebook1001 and 1002 to kernel version 4.9.51-1~bpo8+1
  • 18:22 arlolra@tin: Finished deploy [parsoid/deploy@b150764]: Updating Parsoid to e71937d0 (duration: 09m 13s)
  • 18:21 ebernhardson@tin: Finished deploy [search/mjolnir/deploy@e6905f4]: test mjolnir deployment (duration: 01m 13s)
  • 18:19 ebernhardson@tin: Started deploy [search/mjolnir/deploy@e6905f4]: test mjolnir deployment
  • 18:13 arlolra@tin: Started deploy [parsoid/deploy@b150764]: Updating Parsoid to e71937d0
  • 18:08 ebernhardson@tin: Finished deploy [search/mjolnir/deploy@cd6ddda]: (no justification provided) (duration: 04m 28s)
  • 18:07 moritzm: rebooting notebook* hosts for update to 4.9.51
  • 18:04 ebernhardson@tin: Started deploy [search/mjolnir/deploy@cd6ddda]: (no justification provided)
  • 18:02 mholloway-shell@tin: Started deploy [mobileapps/deploy@9b10959]: Redeploying: Update mobileapps to c002862
  • 17:58 ebernhardson@tin: Finished deploy [search/mjolnir/deploy@ceb5c2f]: (no justification provided) (duration: 00m 20s)
  • 17:57 ebernhardson@tin: Started deploy [search/mjolnir/deploy@ceb5c2f]: (no justification provided)
  • 17:56 ebernhardson@tin: Finished deploy [search/MjoLniR/deploy@607adfb]: (no justification provided) (duration: 00m 14s)
  • 17:56 ebernhardson@tin: Started deploy [search/MjoLniR/deploy@607adfb]: (no justification provided)
  • 17:44 herron: restarted puppetdb on nitrogen and nihal to pick up jre updates
  • 17:42 smalyshev@tin: Finished deploy [wdqs/wdqs@b44cf27]: data reload/T176593 (duration: 00m 16s)
  • 17:42 smalyshev@tin: Started deploy [wdqs/wdqs@b44cf27]: data reload/T176593
  • 17:27 demon@tin: Pruned MediaWiki: 1.31.0-wmf.3 (duration: 07m 58s)
  • 17:12 demon@tin: Synchronized wmf-config/CommonSettings.php: Beta-only, no-op (duration: 00m 43s)
  • 17:11 demon@tin: Synchronized wmf-config/extension-list-labs: No-op (duration: 00m 44s)
  • 17:10 demon@tin: Pruned MediaWiki: 1.31.0-wmf.6 [keeping static files] (duration: 02m 48s)
  • 16:32 twentyafterfour: Restarting apache2 on phab1001 (deploy phabricator hotfix: D876 )
  • 16:23 jynus: stop labsdb1010 mariadb to clone it later to labsdb1009 T179244
  • 15:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1105 in s1 and s2 - T178359 (duration: 00m 45s)
  • 15:00 thcipriani@tin: testing IRC logging
  • 15:00 urandom: Decommissioning Cassandra, restbase1012-c.eqiad.wmnet (T179422)
  • 14:37 moritzm: installing postgres security updates on maps*
  • 14:29 zeljkof: EU SWAT finished
  • 14:07 moritzm: installing postgres security updates on labsdb1004
  • 13:11 moritzm: installing openssl updates
  • 13:10 akosiaris: shutdown install1002 for disk resize
  • 12:57 godog: upgrade grafana to 4.6.1 on https://grafana-labs.wikimedia.org/ - T180428
  • 12:03 ema: restart varnish-be on cp3007, requests failing with 'no backend connection'
  • 11:49 moritzm: rebooting remaining app servers in eqiad for update to Linux 4.9.51 (and to pick up OpenSSL updates)
  • 11:22 moritzm: rebooting remaining API servers in eqiad for update to Linux 4.9.51 (and to pick up OpenSSL updates)
  • 11:04 godog: reimage restbase2002 - T179422
  • 10:50 marostegui: Deploy alter table on s5: db1104 db1100 db1106 - T174569
  • 10:32 elukey: removed old target configs from /srv/prometheus/analytics/targets on prometheus100[34] after https://gerrit.wikimedia.org/r/391179
  • 10:23 moritzm: rebooting image scalers in eqiad for update to Linux 4.9.51 (and to pick up OpenSSL updates)
  • 10:07 godog: upload scap 3.7.2-1 - T127762
  • 09:59 mobrovac@tin: Synchronized wmf-config/jobqueue.php: JobQueue: Migrate RecordLintJob to EventBus - T175212 (duration: 00m 46s)
  • 09:56 marostegui: Deploy alter table on s5 - dbstore1001 - T174569
  • 09:49 foks: Disabled 2FA for Jean-Frédéric
  • 09:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1105 as multi-instance host for s1 and s2 - T178359 (duration: 00m 46s)
  • 09:45 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db1105 as multi-instance host for s1 and s2 - T178359 (duration: 00m 47s)
  • 09:43 godog: upgrade prometheus to 1.8.1 with k8s on prometheus2004 - T177395
  • 09:27 ppchelko@tin: Finished deploy [cpjobqueue/deploy@df5fca9]: Enable RecordLintJob processing (duration: 00m 40s)
  • 09:27 ppchelko@tin: Started deploy [cpjobqueue/deploy@df5fca9]: Enable RecordLintJob processing
  • 09:07 moritzm: rebooting remaining app servers in codfw for update to Linux 4.9.51 (and to pick up OpenSSL updates)
  • 09:05 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw2108.codfw.wmnet
  • 09:05 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: wtp2018.codfw.wmnet
  • 08:31 marostegui: Deploy alter table on s5 - dbstore1002 - T174569
  • 08:22 addshore@tin: Synchronized wmf-config/extension-list-labs: Add AdvancedSearch to extension-list T180147 PT 2/2 LABS ONLY (duration: 00m 46s)
  • 08:21 addshore@tin: Synchronized wmf-config/extension-list: Add AdvancedSearch to extension-list T180147 PT 1/2 (duration: 00m 47s)
  • 08:13 moritzm: rebooting job runners in codfw for update to Linux 4.9.51 (and to pick up OpenSSL updates)
  • 06:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103 on s2 and s4 - T178359 (duration: 00m 47s)
  • 06:18 marostegui: Deploy alter table on s6 primary master (db1061) - T174569
  • 02:30 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Nov 14 02:30:59 UTC 2017 (duration 6m 38s)
  • 02:24 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.7) (duration: 07m 28s)
  • 01:35 mdholloway: mobileapps finished rolling back to 11/8 deployment
  • 01:34 mholloway-shell@tin: Finished deploy [mobileapps/deploy@00e60b2]: (no justification provided) (duration: 02m 27s)
  • 01:31 mholloway-shell@tin: Started deploy [mobileapps/deploy@00e60b2]: (no justification provided)
  • 01:30 mdholloway: mobileapps rolling back today's deployment due to a significant increase in errors.

2017-11-13

  • 22:49 bawolff: deployed patch T119158 (Will affect language converter. -{}- no longer allowed in link urls)
  • 22:42 bawolff: deployed patch T124404
  • 22:02 mholloway-shell@tin: Finished deploy [mobileapps/deploy@9b10959]: Update mobileapps to c002862 (duration: 05m 31s)
  • 21:57 mholloway-shell@tin: Started deploy [mobileapps/deploy@9b10959]: Update mobileapps to c002862
  • 21:40 urandom: Decommissioning Cassandra, restbase1012-b.eqiad.wmnet (T179422)
  • 20:20 ejegg: updated CiviCRM from ddc3881 to f571c67
  • 19:20 smalyshev@tin: Finished deploy [wdqs/wdqs@b44cf27]: data reload/T176593 (duration: 00m 20s)
  • 19:20 smalyshev@tin: Started deploy [wdqs/wdqs@b44cf27]: data reload/T176593
  • 19:18 thcipriani@tin: Synchronized wmf-config/abusefilter.php: SWAT: Enable per-filter profiling on enwiki T179323 (duration: 00m 45s)
  • 19:16 thcipriani@tin: Synchronized docroot/search.wikimedia.org/index.php: search.wikimedia.org: Clean up result returning logic (duration: 00m 47s)
  • 19:15 madhuvishy: Kick of second dumps rsync from ms1001 to labstore1006
  • 18:28 gehel: deploy latest blazegraph + GUI on wdqs200[23] to switch vocabulary - T176593
  • 18:20 elukey: drain + shutdown analytics1029 as prep step to replace the BBU - T178742
  • 18:09 hoo: Ran "scap pull" on mwdebug1001/snapshot1001 after (further) tests re T177486
  • 18:02 urandom: Restarting Cassandra, restbase-dev1004.eqiad.wmnet (testing new `c-foreach-restart`)
  • 18:00 demon@tin: Synchronized docroot/search.wikimedia.org/index.php: minor cleanup (duration: 00m 47s)
  • 17:28 hoo: Ran "scap pull" on mwdebug1001 after tests re T177486
  • 17:12 mobrovac: restbase depooling restbase1007, restbase1012, restbase1014, restbase2002, restbase2004, restbase2006 for T179422
  • 16:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103 on s2 and s4 - T178359 (duration: 00m 47s)
  • 16:45 mobrovac@tin: Finished deploy [mathoid/deploy@63b2ddc]: Update to service-template-node v0.5.3 - T151396 (duration: 03m 45s)
  • 16:42 mobrovac@tin: Started deploy [mathoid/deploy@63b2ddc]: Update to service-template-node v0.5.3 - T151396
  • 16:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1085 - T174569 (duration: 00m 46s)
  • 15:58 akosiaris@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=wtp2017.codfw.wmnet
  • 15:58 akosiaris@puppetmaster1001: conftool action : set/pooled=no; selector: name=wtp2017.codfw.wmnet
  • 15:22 otto@tin: Finished deploy [eventlogging/analytics@e024af3]: T179625 (duration: 00m 02s)
  • 15:22 otto@tin: Started deploy [eventlogging/analytics@e024af3]: T179625
  • 15:09 otto@tin: Finished deploy [eventlogging/analytics@03285e4]: Reverting, got an error: userAgent is a <type unicode>. (duration: 00m 02s)
  • 15:09 otto@tin: Started deploy [eventlogging/analytics@03285e4]: Reverting, got an error: userAgent is a <type unicode>.
  • 15:08 urandom: Decommissioning Cassandra, restbase1012-a.eqiad.wmnet (T179422)
  • 15:06 otto@tin: Finished deploy [eventlogging/analytics@5796c27]: T179625 (duration: 00m 04s)
  • 15:06 otto@tin: Started deploy [eventlogging/analytics@5796c27]: T179625
  • 14:58 herron: upgrading puppetmaster2002 to puppet 4
  • 14:15 zeljkof: EU SWAT finished
  • 14:10 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Whitelist jenkins in test wiki (T167432) (duration: 00m 47s)
  • 14:08 marostegui: Deploy alter table on db1102.s6 (with replication - sanitarium master) - T174569
  • 13:49 marostegui: Stop replication on labsdb1010 to copy cebwiki.geo_tags table
  • 13:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103 on s2 and s4 - T178359 (duration: 00m 46s)
  • 13:36 marostegui@tin: Synchronized wmf-config/db-codfw.php: Increase weight for db1103 on s2 and s4 - T178359 (duration: 00m 46s)
  • 13:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1083 - T174569 (duration: 00m 46s)
  • 13:09 marostegui: Deploy schema change on db1083 - T174569
  • 12:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1088 - T174569 (duration: 00m 47s)
  • 12:14 ema: cache_misc: upgrade varnish to 5.1.3-1wm2
  • 12:08 ema: cp3008: upgrade varnish to 5.1.3-1wm2
  • 12:02 ema: cp4021: restart varnish-be (mbox lag)
  • 11:56 moritzm: installing imagemagick security updates
  • 11:48 moritzm: installing irssi security updates
  • 11:18 elukey: restart of all the druid daemons on druid100[1-6] to apply the new prometheus jmx jvm exporters - T177459
  • 10:57 gehel: upgrade elasticsearch on cirrus / codfw - T178411
  • 10:49 marostegui: Deploy schema change on db1088 - T174569
  • 10:49 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1088 - T174569 (duration: 00m 54s)
  • 10:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1093 - T174569 (duration: 00m 46s)
  • 10:28 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db1103 as multi-instance host for s2 and s4 - T178359 (duration: 00m 46s)
  • 10:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1103 as multi-instance host for s2 and s4 - T178359 (duration: 00m 47s)
  • 10:12 moritzm: rebooting remaining Parsoid hosts in eqiad for update to Linux 4.9.51 (and to pick up OpenSSL update)
  • 09:44 godog: test upgrade of prometheus 1.8.1 with k8s on prometheus2003 - T177395
  • 09:29 moritzm: rebooting mw1221-mw1235 for update to Linux 4.9.51 (and to pick up OpenSSL update)
  • 09:02 elukey: restart of druid brokers on druid100[1-6] to apply https://gerrit.wikimedia.org/r/390419 - T177459
  • 08:41 marostegui: Deploy alter table on db1093 - T174569
  • 08:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1093 - T174569 (duration: 00m 46s)
  • 08:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1098 after alter table - T174569 (duration: 00m 47s)
  • 08:33 moritzm: rebooting mw1238-mw1258 for update to Linux 4.9.51 (and to pick up OpenSSL update)
  • 07:48 moritzm: installing ruby2.3 security updates
  • 07:38 marostegui: Deploy alter table to db1104 - T179106
  • 07:33 marostegui: Optimize wb_terms table on db2052 - T179106
  • 06:44 marostegui: Deploy alter table directly on codfw s5 master (db2023), this will generate lag on codfw - T179793
  • 06:27 marostegui: Deploy alter table on db1098 - T174569
  • 06:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1098 - T174569 (duration: 00m 49s)
  • 06:18 marostegui: Deploy alter table db2086 - T179106
  • 04:01 urandom: Decommissioning Cassandra, restbase1007-c.eqiad.wmnet (T179422)
  • 02:38 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Nov 13 02:38:38 UTC 2017 (duration 6m 42s)
  • 02:31 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.7) (duration: 08m 18s)

2017-11-12

  • 19:39 urandom: Decommissioning Cassandra, restbase1007-b.eqiad.wmnet (T179422)
  • 16:19 bblack: cp4026 - restart backend (mailbox lag)
  • 01:29 urandom: Decommissioning Cassandra, restbase1007-a.eqiad.wmnet (T179422)

2017-11-11

  • 13:52 urandom: Decommissioning Cassandra, restbase2006-c.codfw.wmnet (T179422)
  • 00:48 urandom: Decommissioning Cassandra, restbase2006-b.codfw.wmnet (T179422)

2017-11-10

  • 18:08 smalyshev@tin: Finished deploy [wdqs/wdqs@ccab8ce]: data reload/T176593 (duration: 00m 34s)
  • 18:08 smalyshev@tin: Started deploy [wdqs/wdqs@ccab8ce]: data reload/T176593
  • 17:20 moritzm: uploaded icu 57.1-6+wmf1 for jessie-wikimedia/component/icu57 (co-installable build for ICU migration)
  • 17:11 moritzm: freed some disk space on install1002
  • 16:47 marostegui: Deploy alter table on s6, dbstore1001, dbstore1002 abd db1030 - T174569
  • 15:03 moritzm: rebooting remaing API servers in codfw to 4.9.51 (and to pick up OpenSSL updates)
  • 14:57 urandom: Decommissioning Cassandra, restbase2006-a.codfw.wmnet (T179422)
  • 14:17 moritzm: rebooting image scalers in codfw to 4.9.51 (and to pick up OpenSSL updates)
  • 13:10 marostegui: truncate /var/log/nginx/error.log.1 on install1002 as it is filling up
  • 13:09 moritzm: powercycling mw2118, stuck after reboot
  • 13:05 ema: cp4021: restart varnish-be due to mbox lag
  • 12:48 moritzm: rebooting video scalers in codfw to 4.9.51 (and to pick up OpenSSL updates)
  • 11:46 _joe_: restarted all services and repooled scb1001
  • 11:38 moritzm: rebooting mw2163-2199 to 4.9.51 (and to pick up OpenSSL updates)
  • 11:32 _joe_: stopping mobileapps as well on scb1001
  • 11:27 moritzm: rebooting wtp1025 to 4.9.51
  • 11:22 _joe_: stopping changeprop, celery-ores, cpjobqueue on scb1001
  • 11:21 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Disable AdvancedSearch on deployment.beta BETA ONLY T180201 (duration: 00m 46s)
  • 11:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove old comment about db1080 (duration: 00m 46s)
  • 11:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1055 original weight - T178359 (duration: 00m 46s)
  • 10:59 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: wtp2017.codfw.wmnet
  • 10:55 _joe_: depooling scb1001 from all services while it becomes healthy again
  • 10:52 _joe_: restarting ores on scb1001, causing memory exhaustion
  • 10:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1055 weight - T178359 (duration: 00m 47s)
  • 10:49 moritzm: powercycling wtp2017, stuck after reboot
  • 10:24 marostegui: Deploy schema change on db2089 - T179106
  • 10:11 moritzm: rebooting Parsoid servers in codfw to 4.9.51 (and to pick up OpenSSL updates)
  • 10:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1055 weight - T178359 (duration: 00m 58s)
  • 09:54 marostegui: Compress enwiki on db1105.s1 - T178359
  • 09:45 moritzm: powercycling mw2108, stuck after reboot
  • 09:30 hashar: Upgrading operations-puppet-tests-docker jenkins job to stop passing docker --tty and thus have signals forwarded from 'docker run' - T176747
  • 09:23 moritzm: rebooting mw2097-mw2117 to 4.9.51 (and to pick up OpenSSL updates)
  • 09:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1055 with low weight - T178359 (duration: 00m 47s)
  • 09:13 moritzm: powercycling mw2213, stuck after reboot
  • 08:43 moritzm: rebooting mw2200-mw2223 to 4.9.51 (and to pick up OpenSSL updates)
  • 07:50 smalyshev@tin: Finished deploy [wdqs/wdqs@213f864]: (no justification provided) (duration: 00m 33s)
  • 07:49 smalyshev@tin: Started deploy [wdqs/wdqs@213f864]: (no justification provided)
  • 07:27 _joe_: restarting apache on phab1001
  • 07:20 marostegui: Deploy alter table on s3.codfw master (db2018) with replication, this will generate lag on codfw - T174569
  • 06:50 marostegui: Deploy alter table on s5 eqiad master (db1063) - T172207
  • 06:41 marostegui: Stop MySQL on db1055 to copy its content to db1105 - T178359
  • 06:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1055 - T178359 (duration: 00m 49s)
  • 06:39 marostegui: Force a BBU relearn on db1046 - T166141
  • 01:21 aaron@tin: Synchronized php-1.31.0-wmf.7/includes/db: Use the main stash for LBFactory "memStash" parameter (duration: 00m 47s)
  • 01:18 demon@tin: Synchronized docroot/search.wikimedia.org/index.php: minor cleanups, less 500s (duration: 00m 47s)

2017-11-09

  • 22:28 urandom: Decommissioning Cassandra, restbase2004-c.codfw.wmnet (T179422)
  • 20:28 demon@tin: Synchronized php-1.31.0-wmf.7/includes/libs/objectcache/WANObjectCache.php: less spammy error logs (duration: 00m 47s)
  • 20:09 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.7
  • 19:06 ebernhardson@tin: Synchronized php-1.31.0-wmf.7/extensions/WikimediaEvents/modules/all/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Turn off DBN sizing AB test (duration: 00m 51s)
  • 19:00 bblack: cp3030 - end experimentation, puppetizing back to normal config
  • 18:46 bblack: cp3030 - round 2 of ssl_do_wait_shutdown test
  • 18:41 arlolra: Updated Parsoid to 2887b5ad (T178253, T173643, T176728, T180010, T171381, T179757)
  • 18:28 arlolra@tin: Finished deploy [parsoid/deploy@d1c7386]: Updating Parsoid to 2887b5ad (duration: 12m 20s)
  • 18:22 bblack: cp3030: puppet-disabled + manual nginx ssl_do_wait_shutdown config
  • 18:16 arlolra@tin: Started deploy [parsoid/deploy@d1c7386]: Updating Parsoid to 2887b5ad
  • 18:14 moritzm: not rebooting parsoid hosts due to Services deployment window, instead rolling restart of mw2120-mw2139 for kernel update to 4.9.51
  • 18:04 moritzm: rolling restart of parsoid servers in codfw for 4.9.51 kernel update
  • 17:14 urandom: Restarting Cassandra, restbase2005-b.codfw.wmnet (T179419)
  • 16:33 urandom: Restarting Cassandra, restbase2005-a.codfw.wmnet (T179419)
  • 15:12 urandom: Creating mathoid schema (T179419)
  • 15:05 addshore@tin: Synchronized wmf-config/CommonSettings-labs.php: Enable AdvancedSearch on beta LABS / BETA ONLY PT2/2 (duration: 00m 49s)
  • 15:04 addshore: last sync was actually "Enable AdvancedSearch on beta"
  • 15:04 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Add AdvancedSearch to extension-list-labs LABS / BETA ONLY PT1/2 (duration: 00m 50s)
  • 14:57 addshore@tin: Synchronized wmf-config/extension-list-labs: Add AdvancedSearch to extension-list-labs LABS / BETA ONLY (duration: 00m 50s)
  • 14:42 zeljkof: EU SWAT finished
  • 14:40 zfilipin@tin: Synchronized wmf-config/StartProfiler.php: SWAT: xenon: encode the request method as a virtual stack frame (duration: 00m 50s)
  • 14:35 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Use a threshold that ores in frwiki can stand (T180115) (duration: 00m 50s)
  • 14:30 zfilipin@tin: Synchronized php-1.31.0-wmf.7/extensions/ContentTranslation/modules/: SWAT: Bring back the overlay support for a specific screen region (T179997) (duration: 00m 50s)
  • 14:25 zfilipin@tin: Synchronized php-1.31.0-wmf.7/extensions/EventBus/EventBus.php: SWAT: Logging improvements Rename logged field to fix logstash mapping (duration: 00m 54s)
  • 14:09 urandom: Decommissioning Cassandra, restbase2004-b.codfw.wmnet (T179422)
  • 13:04 moritzm: rebooting mw1209-mw1220 (app servers) to 4.9.51 (also to pick up new OpenSSL)
  • 12:38 moritzm: rebooting mw1189-mw1208 (API servers) to 4.9.5 (also to pick up new OpenSSL)
  • 12:37 Amir1: ladsgroup@terbium:/srv/mediawiki-staging/php-1.31.0-wmf.6$ mwscript extensions/ORES/maintenance/CheckModelVersions.php --wiki=frwiki (T180115)
  • 11:51 moritzm: rebooting mw1180-mw1188 (app servers) to 4.9.5 (also to pick up new OpenSSL)
  • 11:45 _joe_: removed all local hacks from puppetmaster1001, now it uses rhodium again
  • 11:37 _joe_: cleaning up spurious directories /var/lib/puppet/server/ssl/ca from eqiad's puppetmaster backends, generated due to some error on 8/11/2017
  • 11:10 ema: powercycle cp2008, stuck rebooting
  • 11:03 moritzm: rebooting mw1276-mw1279 (API canaries) to 4.9.5 (also to pick up new OpenSSL)
  • 09:50 moritzm: rolling reboot of scb in eqiad for kernel update (also to pick up openssl updates)
  • 07:56 ema: cp1074 failed rebooting, power-cycled
  • 07:46 _joe_: restarting apache on rhodium after setting --profile --trace in the puppet settings
  • 07:04 legoktm@tin: Synchronized php-1.31.0-wmf.7/resources/: Restore jquery.badge and jquery.placeholder modules (duration: 00m 53s)
  • 02:58 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Nov 9 02:58:30 UTC 2017 (duration 6m 59s)
  • 02:51 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.7) (duration: 09m 09s)
  • 02:29 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.6) (duration: 09m 18s)
  • 01:02 twentyafterfour: Not deploying any phabricator updates this week.

2017-11-08

  • 23:45 urandom: Decommissioning Cassandra, restbase2004.codfw.wmnet (T179422)
  • 22:50 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.7 try #2
  • 22:43 ebernhardson@tin: Synchronized php-1.31.0-wmf.6/extensions/CirrusSearch/: Backport cirrus rescore profile refactor to wmf.6 (duration: 01m 02s)
  • 22:21 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 back to wmf.6
  • 22:09 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.7
  • 21:44 bsitzmann@tin: Finished deploy [mobileapps/deploy@00e60b2]: Update mobileapps to 8e82983 (T178706 T178708 T178333 T170692) (duration: 07m 12s)
  • 21:37 bsitzmann@tin: Started deploy [mobileapps/deploy@00e60b2]: Update mobileapps to 8e82983 (T178706 T178708 T178333 T170692)
  • 21:34 ejegg: re-started donations queue consumer with new thank you letter
  • 21:09 ejegg: updated CiviCRM from b11c591 to ddc3881
  • 20:52 ejegg: turned off donations consumer for ty letter update
  • 20:36 ejegg: updated payments-wiki from a539d27 to d150287
  • 20:33 aaron@tin: Synchronized php-1.31.0-wmf.6/extensions/CentralAuth: Use the proper cache key method in loadFromCache() (duration: 00m 54s)
  • 20:22 apergos: rsync from ms1001 to labstore1006 of dumps, 17T so expect it to take several days
  • 20:10 herron: depooled rhodium via puppetmaster1001 apache config
  • 20:09 demon@tin: Synchronized php: symlink (duration: 00m 49s)
  • 19:57 demon@tin: Synchronized php-1.31.0-wmf.7/extensions/CentralAuth/includes/CentralAuthUser.php: (no justification provided) (duration: 00m 51s)
  • 19:55 urandom: Creating mathoid schema (T179419)
  • 19:31 niharika29@tin: Synchronized php-1.31.0-wmf.6/extensions/ORES/: Store stats of accessing ores service for getting thresholds T179862 (duration: 00m 51s)
  • 19:31 urandom: Creating page restrictions schema (T179421)
  • 19:30 niharika29@tin: Synchronized php-1.31.0-wmf.7/extensions/ORES/: Store stats of accessing ores service for getting thresholds T179862 (duration: 00m 51s)
  • 19:24 niharika29@tin: Synchronized php-1.31.0-wmf.7/extensions/CirrusSearch/: Revert Improve handling of 5xx responses to elasticsearch requests (duration: 01m 02s)
  • 19:14 smalyshev@tin: Finished deploy [wdqs/wdqs@b330bc8]: Update service whitelist (duration: 02m 58s)
  • 19:11 smalyshev@tin: Started deploy [wdqs/wdqs@b330bc8]: Update service whitelist
  • 18:27 demon@tin: Synchronized wmf-config/: Dropping old PrivateSettings symlink (ducks and covers) (duration: 00m 52s)
  • 18:19 demon@tin: Synchronized phpcs.xml: no-op (duration: 00m 50s)
  • 17:58 urandom: Restarting Cassandra, restbase2005-[abc]
  • 17:51 urandom: Clearing snapshots in RESTBase legacy Cassandra cluster (T179417)
  • 17:47 marostegui: Deploy alter table on s1 codfw primary master (db2048) with replication, this will generate lag on codfw - T174569
  • 17:43 mobrovac: restbase truncate the default parsoid storage group's tables for T179417
  • 17:41 urandom: Restarting Cassandra, restbase2001-[abc]
  • 17:19 urandom: Restarting Cassandra, restbase2003-[abc]
  • 17:09 herron: regenerated rhodium puppet certificate
  • 17:08 urandom: Restarting Cassandra, restbase1009-[abc]
  • 16:58 urandom: Restarting Cassandra, restbase1008-[abc]
  • 16:49 awight@tin: Finished deploy [ores/deploy@82a13ae]: Roll back scb1002 (duration: 02m 37s)
  • 16:47 awight@tin: Started deploy [ores/deploy@82a13ae]: Roll back scb1002
  • 16:45 awight@tin: Finished deploy [ores/deploy@1b0e59f]: Try to purge specter of revscoring 1 (duration: 05m 45s)
  • 16:40 awight@tin: Started deploy [ores/deploy@1b0e59f]: Try to purge specter of revscoring 1
  • 16:37 urandom: Restarting Cassandra, restbase1010-[abc]
  • 16:37 godog: disregard message about thumbor rolling-restart, upgrade already done and only thumbor1001 rebooted now
  • 16:34 awight@tin: Finished deploy [ores/deploy@82a13ae]: Fix ORES on scb1002 (duration: 00m 03s)
  • 16:34 awight@tin: Started deploy [ores/deploy@82a13ae]: Fix ORES on scb1002
  • 16:29 godog: roll-restart thumbor in eqiad for kernel upgrade
  • 16:11 mobrovac@tin: Finished deploy [restbase/deploy@c5dd1e2]: Switch wiktionary definitions to use the next-gen storage, take 2b - T179420 (duration: 07m 22s)
  • 16:04 mobrovac@tin: Started deploy [restbase/deploy@c5dd1e2]: Switch wiktionary definitions to use the next-gen storage, take 2b - T179420
  • 16:02 mobrovac@tin: Finished deploy [restbase/deploy@c5dd1e2]: Switch wiktionary definitions to use the next-gen storage, take 2 - T179420 (duration: 00m 13s)
  • 16:02 mobrovac@tin: Started deploy [restbase/deploy@c5dd1e2]: Switch wiktionary definitions to use the next-gen storage, take 2 - T179420
  • 16:01 otto@tin: Finished deploy [eventlogging/analytics@03285e4]: Reverting EvenCapsule update and fixes, processes got restarted too early (duration: 00m 02s)
  • 16:01 otto@tin: Started deploy [eventlogging/analytics@03285e4]: Reverting EvenCapsule update and fixes, processes got restarted too early
  • 15:33 urandom: Decommissioning restbase2001-c.codfw.wmnet (T179422)
  • 15:23 _joe_: testing changes on rhodium regarding hostprivkey,hostcert
  • 15:21 ema: eqiad lvs reboots: upgrading kernel to 4.9.51, libssl to 1.0.2m
  • 15:17 otto@tin: Finished deploy [eventlogging/analytics@02c5a6b]: EventCapsule update and fixes, this is no-op as is. T179625 (duration: 00m 04s)
  • 15:17 otto@tin: Started deploy [eventlogging/analytics@02c5a6b]: EventCapsule update and fixes, this is no-op as is. T179625
  • 15:01 otto@tin: Started restart [eventlogging/eventbus@41e3418]: Bumping worker processes to 16 on all targets: T180017
  • 14:55 otto@tin: Started restart [eventlogging/eventbus@41e3418]: Bumping worker processes to 16: T180017
  • 14:54 ema: esams lvs reboots: upgrading kernel to 4.9.51, libssl to 1.0.2m
  • 14:54 otto@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 12s)
  • 14:54 otto@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
  • 14:41 ema: powercycle cp1050 (failed reboot)
  • 14:29 addshore@tin: Synchronized php-1.31.0-wmf.7/docs/uidesign/mediawiki.diff.html: SWAT Add render moved paragraphs marker in diff view PT 2/2 DOCS ONLY (duration: 00m 50s)
  • 14:28 addshore@tin: Synchronized php-1.31.0-wmf.7/resources/src/mediawiki/mediawiki.diff.styles.css: SWAT Add render moved paragraphs marker in diff view PT 1/2 (duration: 00m 51s)
  • 14:16 volans: upgrading cumin to v1.3.0 on prod and WMCS cumin masters
  • 14:10 zfilipin@tin: Synchronized php-1.31.0-wmf.6/extensions/EventBus/EventBus.php: SWAT: Logging improvements. (duration: 00m 52s)
  • 14:08 ema: codfw lvs reboots: upgrading kernel to 4.9.51, libssl to 1.0.2m
  • 13:03 hasharAway: Upgrading jenkins on contint1001/contint2001
  • 12:54 bblack@puppetmaster1001: conftool action : set/pooled=yes; selector: name=phab2001-vcs.codfw.wmnet
  • 12:53 bblack: restart pybal on lvs2002 for git-ssh.codfw deploy
  • 12:51 bblack: restart pybal on lvs2005 for git-ssh.codfw deploy
  • 12:46 mutante: osmium - re-enabling puppet - temp test is over and will be decom'ed
  • 12:41 mutante: krypton (misc PHP apps, scholarships.wm, iegreview.wm, grafana, racktables, burrow) rebooting for kernel upgrade
  • 12:35 mutante: bromine (misc static tistes, annual/transparency/static-bz) - rebooting for kernel upgrade
  • 12:07 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1051 original weight after maintenance - T178359 (duration: 00m 50s)
  • 12:03 mutante: alcyone (url-downloader) rebooting for kernel upgrade
  • 11:53 mutante: netmon1003 (servermon) - rebooting for kernel upgrade
  • 11:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1051 weight after maintenance - T178359 (duration: 00m 50s)
  • 11:48 moritzm: installed openjdk-8/openssl updates and new kernels on restbase*
  • 11:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1051 weight after maintenance - T178359 (duration: 00m 50s)
  • 11:12 moritzm: installing jenkins security update on releases*
  • 11:10 moritzm: imported jenkins 2.73.3 to apt.wikimedia.org
  • 10:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1051 with low weight after maintenance - T178359 (duration: 01m 01s)
  • 10:57 ema: ulsfo lvs reboots: upgrading kernel to 4.9.51, libssl to 1.0.2m
  • 10:42 mutante: actinium - reboot for kernel upgrade (url-downloader)
  • 10:40 ema: varnish 5.1.3-1wm2 built and uploaded to apt.w.o (experimental)
  • 10:39 mutante: aluminium - reboot for kernel upgrade (url-downloader)
  • 10:28 elukey@puppetmaster1001: conftool action : set/pooled=no; selector: name=aqs1005.eqiad.wmnet
  • 10:18 elukey@puppetmaster1001: conftool action : set/pooled=no; selector: name=aqs1004.eqiad.wmnet
  • 10:07 elukey: reboot aqs100[4-9] for jvm and kernel updates
  • 09:37 mutante: planet1001, alsafi (url-downloader) - apt autoremove; reboot for kernel upgrade
  • 09:32 mutante: restarting ircecho (icinga-wm)
  • 09:26 mutante: ununpentium (rt.wikimedia.org): apt-get autoremove; reboot for kernel upgrade
  • 09:23 mutante: rutherforidum (people.wikimedia.org) : apt-get autoremove ; reboot for kernel upgrade
  • 09:22 Pchelolo: restart cassandra-a on restbase1010
  • 09:09 Pchelolo: restart restbase on 1013 and 1015
  • 08:58 mutante: planet2001 - apt autoremove; reboot for kernel upgrade
  • 08:40 ema: resume cache_text/upload rolling reboots: upgrading kernel to 4.9.51, libssl to 1.0.2m and 1.1.0g
  • 07:55 marostegui: Deploy alter table on s1 - on codfw master (db2048) with replication enabled - T172207
  • 07:46 marostegui: Deploy alter table on s2 - on codfw master (db2017) with replication enabled - T172207
  • 07:21 marostegui: Deploy alter table on s3 - on codfw master (db2018) with replication enabled - T172207
  • 07:13 marostegui: Deploy alter table on s7 - on codfw master (db2029) with replication enabled - T172207
  • 07:13 krinkle@tin: Synchronized docroot/noc/conf/: I2e51e783a (duration: 01m 06s)
  • 06:57 marostegui: Deploy alter table on s6 - on codfw master (db2028) with replication enabled - T172207
  • 06:26 marostegui: Add 330G to db2023 partition to make sure the alter over logging table runs fine - T174569
  • 06:16 Krinkle: Restarted uwsgi-graphite-web service on graphite2001
  • 06:16 Krinkle: Restarted uwsgi-graphite-web service on graphite1001
  • 06:15 marostegui: Stop MySQL on db1051 to copy its content to db1105.s1 - T178359
  • 06:14 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1051 - T178359 (duration: 00m 51s)
  • 05:45 Krinkle: rm -rf /var/lib/carbon/whisper/MediaWiki/wanobjectcache/centralauth_user_* on graphite1001 and graphite2001 for T179999
  • 03:17 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Nov 8 03:17:53 UTC 2017 (duration 7m 13s)
  • 03:10 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.7) (duration: 15m 16s)
  • 02:44 aaron@tin: Synchronized php-1.31.0-wmf.7/tests: Deploy 087f2d579a9f which reverts 4432e898be0 due to statsd spam (duration: 01m 14s)
  • 02:42 aaron@tin: Synchronized php-1.31.0-wmf.7/includes: Deploy 087f2d579a9f which reverts 4432e898be0 due to statsd spam (duration: 01m 40s)
  • 02:33 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.6) (duration: 07m 38s)

2017-11-07

  • 23:57 urandom: Decommissioning restbase2001-b.codfw.wmnet (T179422)
  • 22:50 ejegg: re-enabled thank you mailer
  • 22:49 ejegg: updated CiviCRM from 85e89c6 to dc1b279
  • 22:39 hoo: Reset a global account's email, per T179950
  • 22:28 ejegg: updated CiviCRM from 122ba65 to 85e89c6
  • 21:56 ejegg: updated CiviCRM from dc9d054 to 122ba65
  • 21:20 addshore@tin: Synchronized wmf-config/Wikibase-buildentry.php: T176948 Remove Shared Cache settings from Wikibase-buildentry (duration: 00m 50s)
  • 21:16 addshore@tin: Synchronized wmf-config/Wikibase.php: T176948 Stop using wgWikibaseSharedCacheKeyPrefix from Wikidata build (duration: 00m 49s)
  • 21:09 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: T176948 Remove wmgWikibaseUseConfigFromWikidataBuild PT 3/3 (LABS) (duration: 00m 50s)
  • 21:07 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T176948 Remove wmgWikibaseUseConfigFromWikidataBuild PT 2/3 (duration: 00m 52s)
  • 21:06 addshore@tin: Synchronized wmf-config/Wikibase.php: T176948 Remove wmgWikibaseUseConfigFromWikidataBuild PT 1/3 (duration: 00m 50s)
  • 21:02 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T176948 wmgWikibaseUseConfigFromWikidataBuild flase for all of PROD (duration: 00m 49s)
  • 21:01 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T176948 Load wikibase build from mediawiki-config for wikidataclient (duration: 00m 50s)
  • 20:59 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T176948 Load wikibase build from mediawiki-config for group0 & group1 (duration: 00m 50s)
  • 20:54 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T176948 Load wikibase build from mediawiki-config for hewiki (duration: 00m 49s)
  • 20:51 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T176948 Load wikibase build from mediawiki-config for wikidatawiki (duration: 00m 50s)
  • 20:48 herron: puppet issue cleared after reverting 386666. restarting ircecho on einsteinium
  • 20:45 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: T176948 #1 #2 #3 Load wikibase build from mediawiki-config for BETA ONLY (duration: 00m 50s)
  • 20:28 addshore@tin: Synchronized wmf-config/Wikibase.php: Add loading of wikibase extensions from build PT 3/3 (duration: 00m 50s)
  • 20:27 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Add loading of wikibase extensions from build PT 2/3 (duration: 00m 50s)
  • 20:25 addshore@tin: Synchronized wmf-config/Wikibase-buildentry.php: Add loading of wikibase extensions from build PT 1/3 (duration: 00m 49s)
  • 20:20 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Remove unused wmgUseWikibasePropertySuggester PT 2/2 (duration: 00m 50s)
  • 20:19 addshore@tin: Synchronized wmf-config/CommonSettings.php: Remove unused wmgUseWikibasePropertySuggester PT 1/2 (duration: 00m 50s)
  • 20:06 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.7
  • 19:44 volans: restarted apache2 on rhodium (puppet master failing)
  • 19:31 herron: restart apache2 service on rhodium
  • 19:12 demon@tin: Finished scap: wmf.7 bootstrap (duration: 48m 15s)
  • 19:03 awight: begin stress test on ores*
  • 18:58 awight@tin: Finished deploy [ores/deploy@29905e5]: test deployment to repair ores1009 (non-production) (duration: 00m 31s)
  • 18:57 awight@tin: Started deploy [ores/deploy@29905e5]: test deployment to repair ores1009 (non-production)
  • 18:56 awight@tin: Finished deploy [ores/deploy@29905e5]: test deployment to repair ores1008 (non-production) (duration: 00m 32s)
  • 18:56 awight@tin: Started deploy [ores/deploy@29905e5]: test deployment to repair ores1008 (non-production)
  • 18:50 awight@tin: Finished deploy [ores/deploy@29905e5]: test deployment to repair ores1002 (non-production) (duration: 00m 33s)
  • 18:49 awight@tin: Started deploy [ores/deploy@29905e5]: test deployment to repair ores1002 (non-production)
  • 18:39 volans: slowly running puppet on failed hosts with cumin (concurrency=5)
  • 18:38 awight@tin: Finished deploy [ores/deploy@29905e5]: test deployment to ores* (non-production) (duration: 00m 20s)
  • 18:38 awight@tin: Started deploy [ores/deploy@29905e5]: test deployment to ores* (non-production)
  • 18:36 _joe_: restarting apache2 on rhodium
  • 18:24 demon@tin: Started scap: wmf.7 bootstrap
  • 18:16 awight@tin: Finished deploy [ores/deploy@29905e5]: test deployment to ores* (non-production) (duration: 01m 04s)
  • 18:15 awight@tin: Started deploy [ores/deploy@29905e5]: test deployment to ores* (non-production)
  • 18:06 herron: restarting puppetdb service on nitrogen
  • 17:56 urandom: Decommissioning restbase2001-a.codfw.wmnet (T179422)
  • 17:55 elukey: stop ircecho on einstenium (puppet shower from nitrogen)
  • 17:40 ema: stop cache_text/upload rolling reboots, resuming tomorrow
  • {{safesubst:SAL entry|1=17:38 urandom: Restart Cassandra, restbase1010-{a,b,c}.eqiad.wmnet (T178177)}}
  • 17:33 urandom: Clearing Cassandra snapshots (T179422)
  • 17:32 moritzm: rolling reboot of scb in codfw to pick up new kernel (and openssl updates)
  • {{safesubst:SAL entry|1=17:02 urandom: Restarting Cassandra, restbase2001-{a,b,c} to apply OpenJDK upgrade}}
  • 16:47 demon@tin: Synchronized multiversion/submodules.json: no op (duration: 00m 47s)
  • 16:12 ema: start cache_text/upload rolling reboots: upgrading kernel to 4.9.51, libssl to 1.0.2m and 1.1.0g
  • 15:46 gehel: rolling restart of cassandra maps-test for logging change
  • 15:43 godog: roll-restart thumbor in eqiad for kernel upgrade
  • 15:37 urandom: T179420: recreating wiktionary definition schemas
  • 15:29 mobrovac@tin: Finished deploy [restbase/deploy@eab2948]: revert definition switch, wrong schema - T179420 (duration: 06m 46s)
  • 15:22 mobrovac@tin: Started deploy [restbase/deploy@eab2948]: revert definition switch, wrong schema - T179420
  • 15:12 _joe_: added a runner for htmlCacheUpdate on cewiki too
  • 15:10 mobrovac@tin: Finished deploy [restbase/deploy@c5dd1e2]: Switch wiktionary definitions to use the next-gen storage - T179420 (duration: 07m 52s)
  • 15:02 mobrovac@tin: Started deploy [restbase/deploy@c5dd1e2]: Switch wiktionary definitions to use the next-gen storage - T179420
  • 14:52 godog: roll-restart thumbor for kernel upgrades
  • 14:38 mobrovac: restbase creating wiktionary definition schemas for T179420
  • 14:36 godog: reboot wezen for kernel upgrade
  • 14:33 godog: reboot lithium for kernel upgrade
  • 14:26 elukey: rolling restart of kafka on kafka-jumbo* for jvm security updates
  • 14:20 moritzm: rebooting mw canaries to 4.9.51 kernel (also picking up openssl/openssl1.1 updates)
  • 14:19 moritzm: rearming keyholder on naos
  • 14:14 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable ShortUrl on pa.wiki - T178919 (duration: 00m 46s)
  • 14:11 hashar: mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki=pawiki ShortUrl - T178919
  • 14:08 moritzm: rebooting tureis for kernel update
  • 14:07 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: add _ in appendix talk namespace at mywiktionary - T179907 (duration: 00m 45s)
  • 14:02 moritzm: actually holding mw canary reboots until SWAT is over
  • 14:01 moritzm: rebooting mw canaries to 4.9.51 kernel (also picking up openssl/openssl1.1 updates)
  • 13:50 herron: rebooted fermium (lists) for kernel update
  • 13:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add comment about db1105 current status (duration: 00m 47s)
  • 13:43 herron: starting rolling mx reboots for kernel update
  • 13:31 marostegui: Deploy schema change on s5 codfw master (db2023) with replication, this will generate lag on codfw - T174569
  • 13:19 moritzm: rebooting naos for kernel update
  • 13:17 gehel: reboot maps eqiad cluster for upgrades
  • 13:10 moritzm: rebooting wasat for kernel update
  • 12:50 mutante: hafnium, tungsten: groupdel perf-roots to go with gerrit:389663 (T179728)
  • 12:00 mobrovac@tin: Finished deploy [restbase/deploy@eab2948]: Use the new storage for wikidata.org - T179417 (duration: 08m 14s)
  • 11:52 mobrovac@tin: Started deploy [restbase/deploy@eab2948]: Use the new storage for wikidata.org - T179417
  • 11:43 mobrovac: restbase truncating cassandra 2 non-WP tables for T179417
  • 11:42 mobrovac: restbase truncating cassandra 2 non-WP tables for T179420
  • 11:29 moritzm: installing java security updates/restarting cassandra on restbase2001 (cassandra3 node)
  • 11:07 ema: cache_misc rolling reboots: upgrading kernel to 4.9.51, libssl to 1.0.2m and 1.1.0g
  • 10:59 ema: reboot cp3007: upgrading kernel to 4.9.51, libssl to 1.0.2m and 1.1.0g
  • 10:59 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db2092 as recentchanges multi-instance host on s1 and s3 T178359 (duration: 00m 45s)
  • 10:58 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2092 as recentchanges multi-instance host on s1 and s3 T178359 (duration: 00m 45s)
  • 10:41 moritzm: restarting ntpd on dns recursors to pick up openssl update
  • 10:28 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2091 as recentchanges multi-instance host on s2 and s4 T178359 (duration: 00m 45s)
  • 10:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db2091 as recentchanges multi-instance host on s2 and s4 T178359 (duration: 00m 46s)
  • 10:24 elukey: create staging database on db1108 (researchers scratch pad) - T177405
  • 10:19 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2086 as recentchanges multi-instance host on s5 and s7 T178359 (duration: 00m 45s)
  • 10:09 gehel: reboot of maps codfw cluster for upgrades
  • 09:59 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db2086 as recentchanges multi-instance host on s5 and s7 T178359 (duration: 00m 45s)
  • 09:58 paravoid: updating certspotter to 0.5 in apt and tegmen/einsteinium
  • 09:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db2089 as recentchanges multi-instance host on s6 and s5(s8) T178359 (duration: 00m 46s)
  • 09:39 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2089 as recentchanges multi-instance host on s6 and s5(s8) T178359 (duration: 00m 45s)
  • 09:13 marostegui: Stop MySQL on db1038 - host to be decommissioned - T177911
  • 09:11 moritzm: installing java security updates/restarting cassandra on restbase2002
  • 09:02 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1038 from config - T177911 (duration: 00m 45s)
  • 09:02 ppchelko@tin: Finished deploy [cpjobqueue/deploy@3db0cc4]: Do not set Host header for requests to jobrunner (duration: 00m 49s)
  • 09:02 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1038 from config - T177911 (duration: 00m 47s)
  • 09:01 ppchelko@tin: Started deploy [cpjobqueue/deploy@3db0cc4]: Do not set Host header for requests to jobrunner
  • 08:49 marostegui: Run redact_sanitarium for hifwiktionary on db1095 (sanitarium) - T173647
  • 08:48 marostegui: Optimize pagelinks and templatelinks on s7 master - db1062 - T174509
  • 07:18 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: Electron stuck, restarting - T174916
  • 07:11 ema: reboot pinkunicorn for kernel (4.9.51) and openssl (1.0.2m) upgrades
  • 06:19 marostegui: Deploy alter table on s7 codfw master (db2029) with replication, this will cause lag in codfw - T174569
  • 02:29 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Nov 7 02:29:54 UTC 2017 (duration 6m 39s)
  • 02:23 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.6) (duration: 07m 19s)

2017-11-06

  • 22:34 bawolff: Deploy patch for T178451
  • 21:48 arlolra: Updated Parsoid to 6cb77104 (T179579, T175792)
  • 21:42 arlolra@tin: Finished deploy [parsoid/deploy@d1df3c3]: (no justification provided) (duration: 08m 46s)
  • 21:35 awight: Begin stress testing ores* (non-production)
  • 21:33 arlolra@tin: Started deploy [parsoid/deploy@d1df3c3]: (no justification provided)
  • 21:20 XioNoX: re-activating BGP session to telia in ulsfo
  • 21:16 ladsgroup@tin: Finished deploy [ores/deploy@97a1d80]: Deploying early November (T179837) (duration: 10m 47s)
  • 21:05 ladsgroup@tin: Started deploy [ores/deploy@97a1d80]: Deploying early November (T179837)
  • 20:56 reedy@tin: Synchronized wmf-config/interwiki.php: Update IW Map again (duration: 00m 46s)
  • 20:50 marostegui: Deploy alter table on s2 codfw master (db2017) with replication, this will generate lag in codfw - T174569
  • 20:36 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: hifwiktionary T173643 (duration: 00m 45s)
  • 20:34 reedy@tin: rebuilt wikiversions.php and synchronized wikiversions files: hifwiktionary T173643
  • 20:33 reedy@tin: Synchronized dblists/: hifwiktionary T173643 (duration: 00m 47s)
  • 20:04 reedy@tin: Synchronized wmf-config/interwiki.php: Update IW map (duration: 00m 45s)
  • 19:57 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: electcomwiki T174370 (duration: 00m 45s)
  • 19:56 reedy@tin: rebuilt wikiversions.php and synchronized wikiversions files: electcomwiki T174370
  • 19:55 reedy@tin: Synchronized dblists/: electcomwiki T174370 (duration: 00m 44s)
  • 19:52 demon@tin: Pruned MediaWiki: 1.31.0-wmf.2 (duration: 02m 49s)
  • 19:47 demon@tin: Pruned MediaWiki: 1.31.0-wmf.5 [keeping static files] (duration: 01m 38s)
  • 19:22 Amir1: ladsgroup@terbium:/srv/mediawiki-staging/php-1.31.0-wmf.4$ mwscript extensions/ORES/maintenance/CheckModelVersions.php --wiki=enwiki (T179596)
  • 19:21 kaldari@tin: Synchronized wmf-config/InitialiseSettings.php: Updating InitialiseSettings for draftquality data (duration: 00m 48s)
  • 18:15 XioNoX: deactivating BGP session to telia in ulsfo
  • 18:09 gehel@tin: Finished deploy [wdqs/wdqs@fa8bb12]: WDQS GUI and Blazegraph update (duration: 01m 53s)
  • 18:07 gehel@tin: Started deploy [wdqs/wdqs@fa8bb12]: WDQS GUI and Blazegraph update
  • 17:50 awight@tin: Finished deploy [ores/deploy@29905e5]: restart services on ores* (non-production) (duration: 17m 18s)
  • 17:48 urandom: T179083: Restarting Cassandra instances on restbase2001.codfw.wmnet
  • 17:32 awight@tin: Started deploy [ores/deploy@29905e5]: restart services on ores* (non-production)
  • 16:51 marostegui: Optimize pagelinks and templatelinks on s2 master - db1054 - T174509
  • 16:26 bblack: Switching unified TLS certificate in the US
  • 16:25 awight@tin: Finished deploy [ores/deploy@82a13ae]: Redeploy ORES master to scb* (duration: 16m 10s)
  • 16:09 awight@tin: Started deploy [ores/deploy@82a13ae]: Redeploy ORES master to scb*
  • 15:50 akosiaris@tin: Started deploy [ores/deploy@29905e5]: testing deploy
  • 15:49 akosiaris@tin: Started deploy [ores/deploy@29905e5]: testing deploy
  • 15:43 awight@tin: Finished deploy [ores/deploy@29905e5]: restart services on ores* (non-production) (duration: 12m 45s)
  • 15:31 awight@tin: Started deploy [ores/deploy@29905e5]: restart services on ores* (non-production)
  • 15:26 gehel: reboot of maps-test cluster for jvm and kernel upgrades
  • 15:13 ppchelko@tin: Finished deploy [cpjobqueue/deploy@96f55c6]: USe correct regex for job selection (duration: 00m 29s)
  • 15:12 ppchelko@tin: Started deploy [cpjobqueue/deploy@96f55c6]: USe correct regex for job selection
  • 15:03 addshore@tin: Synchronized wmf-config/CommonSettings-labs.php: SWAT Set wgWikiDiff2MovedParagraphDetectionCutoff for group0 PT 3/3 (duration: 00m 46s)
  • 15:01 addshore@tin: Synchronized wmf-config/CommonSettings.php: SWAT Set wgWikiDiff2MovedParagraphDetectionCutoff for group0 PT 2/3 (duration: 00m 46s)
  • 15:00 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT Set wgWikiDiff2MovedParagraphDetectionCutoff for group0 PT 1/3 (duration: 00m 47s)
  • 14:57 gehel: reboot of relforge for kernel + jvm upgrade
  • 14:50 mobrovac@tin: Synchronized wmf-config/jobqueue.php: Switch MessageIndexRebuildJob, flaggedrevs_CacheUpdate and deleteLinks jobs to the EventBus infrastructure - T175210 (duration: 00m 46s)
  • 14:50 ppchelko@tin: Finished deploy [cpjobqueue/deploy@e93feba]: Start processing all 'hearted' jobs T175210 (duration: 00m 44s)
  • 14:49 ppchelko@tin: Started deploy [cpjobqueue/deploy@e93feba]: Start processing all 'hearted' jobs T175210
  • 14:48 ppchelko@tin: Started deploy [cpjobqueue/deploy@e93feba]: (no justification provided)
  • 14:45 awight@tin: Finished deploy [ores/deploy@29905e5]: Force deployment on ores* (non-production) (duration: 05m 43s)
  • 14:45 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs2003.codfw.wmnet
  • 14:45 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs2002.codfw.wmnet
  • 14:40 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs2002.codfw.wmnet
  • 14:40 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs2001.codfw.wmnet
  • 14:39 awight@tin: Started deploy [ores/deploy@29905e5]: Force deployment on ores* (non-production)
  • 14:35 awight@tin: Finished deploy [ores/deploy@29905e5]: restart services on ores* (non-production) (duration: 04m 48s)
  • 14:34 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs2001.codfw.wmnet
  • 14:33 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs1005.eqiad.wmnet
  • 14:30 awight@tin: Started deploy [ores/deploy@29905e5]: restart services on ores* (non-production)
  • 14:27 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs1005.eqiad.wmnet
  • 14:27 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs1004.eqiad.wmnet
  • 14:21 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs1004.eqiad.wmnet
  • 14:21 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs1003.eqiad.wmnet
  • 14:20 awight@tin: Finished deploy [ores/deploy@29905e5]: celery 4 -> ores* (non-production) (duration: 19m 22s)
  • 14:18 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: EventBus: Enable logging - T150106 (duration: 00m 47s)
  • 14:14 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs1003.eqiad.wmnet
  • 14:14 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=wdqs1003.eqiad.wmnet
  • 14:13 mobrovac@tin: Synchronized php-1.31.0-wmf.6/extensions/EventBus/EventBus.php: EventBus - Improve logging for T150106 (duration: 00m 48s)
  • 14:01 gehel: full restart of wdqs eqiad / codfw for multiple upgrades
  • 14:01 awight@tin: Started deploy [ores/deploy@29905e5]: celery 4 -> ores* (non-production)
  • 13:03 elukey: rolling restart of druid historical daemons on druid100[1-6] to apply https://gerrit.wikimedia.org/r/#/c/389429
  • 12:58 moritzm: installing openssl security updates
  • 12:55 moritzm: restarting apache on netmon to pick up openssl update
  • 12:43 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: Electron stuck, restrting - T174916
  • 12:29 moritzm: installing imagemagick security updates
  • 11:45 oblivian@tin: Started deploy [jobrunner/jobrunner@a20d043]: (no justification provided)
  • 11:44 marostegui: Deploy alter table on s6 codfw master (with replication, so this will generate lag on codfw) - T174569
  • 11:19 marostegui: Compress InnoDB on db1103.s2 - T178359
  • 11:16 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2087 as recentchanges multi-instance host on s6 and s7 T178359 (duration: 00m 47s)
  • 11:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db2087 as recentchanges multi-instance host on s6 and s7 T178359 (duration: 00m 48s)
  • 10:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1101 original weight after maintenance - T178359 (duration: 00m 46s)
  • 10:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give db1101 more weight after maintenance - T178359 (duration: 00m 46s)
  • 10:05 moritzm: uploading linux 4.9.51-1~bpo8+1 for jessie-wikimedia to apt.wikimedia.org
  • 09:57 godog: roll-upgrade swift to 2.10.2 in codfw and roll-restart for kernel upgrade - T177739
  • 09:56 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db2088 as recentchanges multi-instance host on s1 and s2 T178359 (duration: 00m 46s)
  • 09:55 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2088 as recentchanges multi-instance host on s1 and s2 T178359 (duration: 00m 46s)
  • 09:37 _joe_: manually running htmlCacheUpdate for commonswiki and ruwiki on terbium, T173710
  • 09:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db2085 as recentchanges multi-instance host on s3 and s5 T178553 T178359 (duration: 00m 46s)
  • 09:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2085 as multi-instance core host on eqiad file T178553 T178359 (duration: 00m 46s)
  • 09:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give db1101 more weight after maintenance - T178359 (duration: 00m 46s)
  • 09:07 moritzm: removed monitoring/RAID packages from stretch-wikimedia/thirdparty, now in thirdparty/hwraid
  • 08:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1101 with low weight after maintenance - T178359 (duration: 00m 47s)
  • 06:46 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2084 as multi-instance core host on eqiad file T178553 T178359 (duration: 00m 46s)
  • 06:38 marostegui: Stop MySQL on db1101 to copy its content to db1103.s4 - T178359
  • 06:31 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1101 - T178359 (duration: 00m 48s)
  • 06:19 marostegui: Optimize pagelinks and templatelinks on s6 master (db1061) - T174509
  • 06:16 marostegui: Deploy alter table on s4 codfw master (with no replication) - T174569
  • 02:44 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Nov 6 02:44:41 UTC 2017 (duration 6m 55s)
  • 02:37 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.6) (duration: 07m 17s)

2017-11-04

  • 13:34 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=scb1002.eqiad.wmnet
  • 13:19 elukey@puppetmaster1001: conftool action : set/pooled=no; selector: name=scb1002.eqiad.wmnet
  • 12:36 mobrovac: trending-edits depool it from scb1002 to investigate
  • 00:18 Krinkle: Finished whisper-mass-resize for frontend.navtiming on graphite2001 (T179622)

2017-11-03

  • 20:28 marostegui: Deploy alter table db2058 - T174569
  • 18:52 Krinkle: Starting whisper-mass-resize for frontend.navtiming on graphite2001 (T179622)
  • 18:51 Krinkle: whisper-mass-resize completed for graphite1001.eqiad.wmnet:/var/lib/carbon/whisper/frontend/navtiming (at 01:34 UTC actually)
  • 17:56 demon@tin: Synchronized php-1.31.0-wmf.6/includes/libs/rdbms/database/DatabaseMysqlBase.php: logging (duration: 00m 47s)
  • 16:17 marostegui: Deploy alter table on db2044 - T174569
  • 16:03 demon@tin: Synchronized php-1.31.0-wmf.6/extensions/ContentTranslation/api/ApiQueryTranslatorStats.php: fix warnings about bad min() calls (duration: 00m 47s)
  • 15:50 cmjohnson1: powering off labvirt1015 to swap CPU (again)
  • 15:49 akosiaris: T177393 enable RBAC for kubernetes in production and staging
  • 15:35 volans: uploaded cumin_1.3.0-1_amd64.deb to apt.wikimedia.org jessie-wikimedia
  • 14:19 moritzm: installing java sec updates / cassandra restarts on xenon/praseodymium (cerium already done earlier)
  • 14:16 hashar: restarting Jenkins on contint1001
  • 14:12 hashar: Upgrading packages on contint1001 / contint2001
  • 13:50 moritzm: installing heimdal security updates on trusty
  • 13:35 ppchelko@tin: Started restart [electron-render/deploy@8dd5f13]: Restarting, service is stuck
  • 13:33 Pchelolo: restart electron render service
  • 12:41 moritzm: uploaded openjdk 8u151-b12 to apt.wikimedia.org/jessie-wikimedia
  • 11:45 moritzm: restarting jenkins on releases* to pick up openjdk security update
  • 11:41 mutante: gerrit - restart Apache - apply gerrit:354078 - only listen on service IP
  • 11:12 moritzm: uploaded openssl 1.0.2m-1 for to apt.wikimedia.org
  • 10:46 marostegui: Compress InnoDB on db1103.s4 - T178359
  • 10:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add comment to db1103 status (duration: 00m 47s)
  • 10:42 mutante: cobalt: (gerrit) temp disable puppet - gerrit2001: restart apache, apply gerrit:354078 stop listening on all interfaces
  • 10:39 oblivian@tin: Synchronized wmf-config/CommonSettings.php: Increase concurrency of htmlCacheUpdate jobs T173710 (duration: 00m 48s)
  • 09:09 mobrovac@tin: Finished deploy [restbase/deploy@a67b4a7]: Use only the new storage for summaries - T179418 (duration: 10m 40s)
  • 08:58 mobrovac@tin: Started deploy [restbase/deploy@a67b4a7]: Use only the new storage for summaries - T179418
  • 08:46 ema: pool cp4021
  • 08:16 elukey: drop CommandInvocation_15243810 and CommandInvocation_15243810_15423246 from analytics dbs (db1046/db1047/db1108/dbstore1002) - data archived on HDFS - T166712
  • 08:12 moritzm: installing openjdk-8 security updates on stretch
  • 07:57 marostegui: Deploy alter table on s4.db2037 - T174569
  • 07:29 elukey: depooled mw1191 and powercycle - host down with 'CPU 1 check errors' in racadm getsel
  • 06:39 marostegui: Stop MySQL on db2089.s5 to copy it to db2085 - T178359
  • 06:25 marostegui: Stop MySQL on db1103 to copy it to dbstore1001 and reimage it - T178359

2017-11-02

  • 23:36 krinkle@tin: Synchronized php-1.31.0-wmf.6/extensions/NavigationTiming/modules/ext.navigationTiming.js: Fix zero values - T178479 (duration: 00m 47s)
  • 23:10 herron: disabled puppet master debug logging on puppetmaster2001 via /usr/share/puppet/rack/puppet-master/config.ru
  • 22:48 volans: cleaned daemon.log from puppet spam logs on puppetmaster2001 (15GB) - cc herron
  • 22:39 halfak@tin: Started deploy [ores/deploy@82a13ae]: T179621
  • 22:02 Krinkle: Mass-resizing Graphite/Whisper files on graphite1001 and graphite2001 for T179622 (frontend.* namespace)
  • 21:53 ayounsi@tin: Finished deploy [librenms/librenms@ad48b4d]: (no justification provided) (duration: 00m 05s)
  • 21:52 ayounsi@tin: Started deploy [librenms/librenms@ad48b4d]: (no justification provided)
  • 21:50 XioNoX: upgrading librenms
  • 21:38 hoo: Updated the Wikidata property suggester with data from Monday's JSON dump and applied the T132839 workarounds
  • 20:56 madhuvishy: Powercycling labvirt1015
  • 20:18 chasemp: simulate load for labvirt1015 'sudo cumin "name:labvirt1015stresstest*" "stress-ng --cpu 1 --io 2 --vm 1 --vm-bytes 1G &"'
  • 19:45 mepps: updated payments-wiki from b88b805 to a539d27
  • 19:28 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.6
  • 18:20 gehel: restarting blazegraph on wdqs1003
  • 18:13 gehel: restarting stuck updater on wdqs1003
  • 17:55 demon@tin: Synchronized php-1.31.0-wmf.6/includes/Message.php: logging fixes (duration: 00m 48s)
  • 17:53 demon@tin: Synchronized php-1.31.0-wmf.6/includes/libs/rdbms: logging improvements (duration: 00m 54s)
  • 17:47 moritzm: uploaded openssl 1.1.0g-1+wmf1 to jessie-wikimedia
  • 17:42 awight@tin: Finished deploy [ores/deploy@0e54a3c]: revscoring 2, T175180 (duration: 33m 50s)
  • 17:08 awight@tin: Started deploy [ores/deploy@0e54a3c]: revscoring 2, T175180
  • 16:14 ema: restart varnish-be on cp4026 due to mbox lag
  • 16:04 mobrovac@tin: Synchronized wmf-config/jobqueue.php: Use only EventBus for processing updateBetaFeatureUserCount - T175210 (duration: 00m 51s)
  • 13:46 chasemp: ugrade dnsmasq-base on labcontrol*
  • 13:36 zeljkof: EU SWAT finished
  • 13:35 zfilipin@tin: Synchronized php-1.31.0-wmf.6/extensions/ContentTranslation/modules/publish/ext.cx.publish.js: SWAT: CX1: Check for template adaptation failures before publishing (T154116) (duration: 00m 50s)
  • 13:11 zfilipin@tin: Synchronized wmf-config/ProductionServices.php: SWAT: use the logstash LVS endpoint (T175242) (duration: 00m 51s)
  • 12:09 moritzm: removed obsolete packages from jessie-wikimedia/experimental
  • 12:06 mobrovac@tin: Finished deploy [restbase/deploy@9314cf6]: Parsoid: Switch all but WPs to use the next-generation storage - T179417 (duration: 08m 43s)
  • 11:57 mobrovac@tin: Started deploy [restbase/deploy@9314cf6]: Parsoid: Switch all but WPs to use the next-generation storage - T179417
  • 11:43 elukey: drop table log.PageContentSaveComplete_5588433 from db1046,db1047,db1108,dbstore1002, archived on hdfs - T177101
  • 11:31 moritzm: remove obsolete packages from stretch-wikimedia/experimental, now empty
  • 11:29 mobrovac@tin: Finished deploy [restbase/deploy@f6c4e2d]: Parsoid module: use the Cassandra 2 tables as fallback when needed - T179417 (duration: 08m 15s)
  • 11:26 elukey: drop log.MediaViewer_10867062_15423246 from db1047,db1108 since already archived in hdfs - T168303
  • 11:21 mobrovac@tin: Started deploy [restbase/deploy@f6c4e2d]: Parsoid module: use the Cassandra 2 tables as fallback when needed - T179417
  • 10:54 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=logstash1003.eqiad.wmnet
  • 10:53 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=logstash1002.eqiad.wmnet
  • 10:53 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=logstash1001.eqiad.wmnet
  • 10:53 gehel: depooling logstash100[123] in preparation for decommission - T175830
  • 10:38 gehel: rolling restart of wdqs nodes for GC tuning - T175919
  • 10:13 marostegui: Deploy alter table on db2019 - T174569
  • 09:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1091 original weight - T161088 (duration: 00m 50s)
  • 09:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1091 weight after maintenance - T161088 (duration: 00m 50s)
  • 08:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1091 weight after maintenance - T161088 (duration: 00m 50s)
  • 08:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1091 with low weight after maintenance - T161088 (duration: 00m 50s)
  • 08:15 moritzm: restarting app server canaries to pick up curl security update
  • 08:04 moritzm: installing curl security updates
  • 07:59 marostegui: Stop mysql event scheduler on labsdb1001 - T179464
  • 06:54 marostegui: Stop s3 instance on db2092 to copy it to db2085 - T178359
  • 06:33 legoktm: done on mwdebug1002
  • 06:31 marostegui: Stop MySQL on db1091 and db1103 to migrate db1091 to file per table
  • 06:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1091 - T161088 (duration: 01m 07s)
  • 06:27 marostegui: Deploy optimize table wikidatawiki.pagelinks on s5 master (db1063) - T174509
  • 06:02 legoktm: live-hacking mwdebug1002
  • 05:43 TimStarling: on puppetmaster2001: disk full due to logs, so I'm truncating /var/log/debug which is apparently copied to daemon.log anyway
  • 03:29 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Nov 2 03:29:35 UTC 2017 (duration 7m 12s)
  • 03:22 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.6) (duration: 15m 37s)
  • 02:43 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.5) (duration: 16m 06s)
  • 00:52 ebernhardson@tin: Synchronized php-1.31.0-wmf.5/extensions/WikimediaEvents/extension.json: SWAT followup: Update SearchSatisfaction eventlogging schema revision id (duration: 00m 50s)
  • 00:51 ebernhardson@tin: Synchronized php-1.31.0-wmf.6/extensions/WikimediaEvents/extension.json: (no justification provided) (duration: 00m 50s)
  • 00:32 bblack: cp4022 - varnish backend restart, mailbox
  • 00:28 bblack: lvs1001-3: re-enable ethernet flowcontrol (short ethernet blip), with pybal stopped to failover to backups during...

2017-11-01

  • 23:37 thcipriani@tin: Synchronized php-1.31.0-wmf.5/extensions/WikimediaEvents: SWAT: Turn on Cirrus AB test for DBN group sizing (duration: 00m 50s)
  • 23:35 thcipriani@tin: Synchronized php-1.31.0-wmf.6/extensions/WikimediaEvents: SWAT: Turn on Cirrus AB test for DBN group sizing (duration: 00m 51s)
  • 23:29 thcipriani@tin: Synchronized php-1.31.0-wmf.6/extensions/ParserMigration/includes/ApiParserMigration.php: SWAT: API: Fix WikiPage namespace (duration: 00m 52s)
  • 23:25 thcipriani@tin: Synchronized php-1.31.0-wmf.6/extensions/Wikibase/repo/Wikibase.hooks.php: SWAT: Allow turning Cirrus usage off from query T179428 (duration: 00m 49s)
  • 23:23 thcipriani@tin: Synchronized php-1.31.0-wmf.6/extensions/Wikidata/extensions/Wikibase/repo/Wikibase.hooks.php: SWAT: Allow turning Cirrus usage off from query T179428 (duration: 00m 51s)
  • 23:09 thcipriani@tin: Synchronized wmf-config/Wikibase-production.php: SWAT: Revert "Revert "Add negative weight to disambig entities"" T148411 (duration: 00m 51s)
  • 22:49 ejegg: disabled fundraising unsubscribe import processing
  • 22:33 Krinkle: group2(all-wikidata) wikis to wmf.5 from 24 hours ago seems to have caused a 60% drop in navigation timing metric report count (100/min => 40/min)
  • 22:19 eileen1: process-control updated to 9c8b4bb
  • 22:17 demon@tin: Synchronized wmf-config/CommonSettings.php: rel1_28 is dead (duration: 00m 50s)
  • 22:06 ejegg: re-enabled fundraising jobs
  • 21:48 ejegg: fundraising jobs disabled
  • 20:10 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.6
  • 20:05 demon@tin: Synchronized php: symlink swap (duration: 00m 49s)
  • 19:57 krinkle@tin: Synchronized php-1.31.0-wmf.6/resources/src/startup.js: T178943 (duration: 00m 51s)
  • 19:53 demon@tin: Synchronized php-1.31.0-wmf.5/extensions/CentralNotice: fix weird git rebasing issue (duration: 00m 53s)
  • 19:41 demon@tin: Pruned MediaWiki: 1.31.0-wmf.4 [keeping static files] (duration: 01m 52s)
  • 19:31 awight@tin: Synchronized php-1.31.0-wmf.5/extensions/ORES: Fix for API=oresscores, T179430 (duration: 00m 50s)
  • 19:28 awight@tin: Synchronized php-1.31.0-wmf.6/extensions/ORES: Fix for API=oresscores, T179430 (duration: 00m 52s)
  • 19:14 bblack: all hosts: manual cumin+sed removal of ethernet autoneg params from /e/n/i to match https://gerrit.wikimedia.org/r/#/c/387849/
  • 19:08 otto@tin: Finished deploy [analytics/refinery@6d11d67]: Deploying refinery-source artifacts for 0.0.54 for JsonRefine job, T162610 (duration: 03m 45s)
  • 19:05 otto@tin: Started deploy [analytics/refinery@6d11d67]: Deploying refinery-source artifacts for 0.0.54 for JsonRefine job, T162610
  • 18:39 demon@tin: Synchronized wmf-config/: Dropping squid.php (hang on to your pants folks, this could be fun) (duration: 00m 51s)
  • 18:36 demon@tin: Synchronized wmf-config/CommonSettings.php: use reverse-proxy.php no more squid.php (duration: 00m 50s)
  • 18:35 demon@tin: Synchronized docroot/: dropping squid.php (duration: 00m 52s)
  • 18:13 demon@tin: Synchronized static/images/project-logos/sewikimedia.png: new logo (duration: 00m 50s)
  • 18:02 moritzm: installing openjpeg2 security updates
  • 17:39 demon@tin: Synchronized php-1.31.0-wmf.6/includes/specials/pagers/ContribsPager.php: fix missing page_is_new error (duration: 00m 51s)
  • 17:10 demon@tin: Synchronized wmf-config/: dropped squid-labs, no-op in prod (duration: 00m 52s)
  • 17:09 demon@tin: Synchronized docroot/noc/: dropped squid-labs.php (duration: 00m 51s)
  • 16:01 bblack: lvs1003 - puppet disabled, testing experimental ethtool ringbuffer change
  • 15:15 moritzm: repo reorg: moved ftpsync from thirdparty to main and docker-engine from thirdparty to thirdparty/k8s
  • 15:12 awight@tin: Finished deploy [ores/deploy@9f361d2]: revscoring 2 -> ores1002 (non-production) (duration: 02m 25s)
  • 15:10 awight@tin: Started deploy [ores/deploy@9f361d2]: revscoring 2 -> ores1002 (non-production)
  • 15:09 awight@tin: Finished deploy [ores/deploy@9f361d2]: revscoring 2 -> ores* (non-production) (duration: 01m 57s)
  • 15:07 awight@tin: Started deploy [ores/deploy@9f361d2]: revscoring 2 -> ores* (non-production)
  • 15:07 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: Update CirrusSearch MLR model on enwiki (duration: 00m 51s)
  • 15:01 mobrovac: restbase: removing wikimedia-lvs-realserver from staging hosts T179494
  • 14:55 bblack: strongswan experiment done, cp* back to puppet-agent-enabled
  • 14:41 moritzm: installing poppler security updates
  • 14:09 bblack: cp*: disabling puppet to test strongswan change...
  • 13:49 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable NewUserMessage on fawikiquote - T179442 (duration: 00m 50s)
  • 13:46 hashar@tin: Synchronized wmf-config/abusefilter.php: Enable blocking feature of abuse filter in fawikiquote - T178227 (duration: 00m 50s)
  • 13:45 moritzm: installing quagga security updates
  • 13:43 hashar@tin: Synchronized wmf-config/CommonSettings.php: Properly check for cluster existence prior setting TTM mirrors - T179270 (duration: 01m 05s)
  • 13:41 ppchelko@tin: Finished deploy [restbase/deploy@2321c4c]: Update hyperswitch dependency. Take 2 (duration: 05m 50s)
  • 13:35 ppchelko@tin: Started deploy [restbase/deploy@2321c4c]: Update hyperswitch dependency. Take 2
  • 13:33 ppchelko@tin: Finished deploy [restbase/deploy@2321c4c]: Update hyperswitch dependency (duration: 03m 50s)
  • 13:29 ppchelko@tin: Started deploy [restbase/deploy@2321c4c]: Update hyperswitch dependency
  • 12:48 moritzm: installing libav security updates
  • 12:44 moritzm: installinng libdatetime-timezone-perl stable updates on Debian
  • 12:10 kartik@tin: Finished deploy [cxserver/deploy@10651e2]: Update cxserver to 0227acb (duration: 03m 07s)
  • 12:07 kartik@tin: Started deploy [cxserver/deploy@10651e2]: Update cxserver to 0227acb
  • 08:35 elukey: forced umount/mount for /mnt/hdfs on stat1005 (not working after repeated oom kill actions)
  • 03:22 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Nov 1 03:22:02 UTC 2017 (duration 7m 17s)
  • 03:14 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.6) (duration: 15m 21s)
  • 02:36 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.5) (duration: 09m 46s)

2017-10-31

  • 23:56 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: Adjust cirrussearch interleave configuration for AB test (duration: 00m 50s)
  • 23:48 demon@tin: Synchronized scap/plugins/prep.py: no-op (duration: 00m 50s)
  • 23:21 ebernhardson@tin: Synchronized wmf-config/CirrusSearch-common.php: Setup CirrusSearch AB test on dbn group sizing (duration: 00m 50s)
  • 23:19 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: Setup CirrusSearch AB test on dbn group sizing (duration: 00m 53s)
  • 21:38 awight@tin: Started deploy [ores/deploy@9f361d2]: revscoring 2 -> ores* (non-production)
  • 21:37 awight@tin: Started deploy [ores/deploy@9f361d2]: revscoring 2 -> ores* (non-production)
  • 21:16 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/386553 https://gerrit.wikimedia.org/r/386710 (duration: 00m 52s)
  • 21:00 XioNoX: removing old AMS-IX IPv6 - T167840
  • 20:57 mepps: rollback payments-wiki from 5303a8e to b88b805
  • 20:54 mepps: updated payments-wiki from b88b805 to 5303a8e
  • 20:45 mepps: rolledback payments-wiki from 5303a8e to b88b805
  • 20:43 mepps: updated payments-wiki from b88b805 to 5303a8e
  • 19:18 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.6
  • 18:53 awight@tin: Finished deploy [ores/deploy@437fcf5]: revscoring 2 -> ores* (non-production) (duration: 10m 29s)
  • 18:42 awight@tin: Started deploy [ores/deploy@437fcf5]: revscoring 2 -> ores* (non-production)
  • 18:38 awight@tin: Finished deploy [ores/deploy@437fcf5]: revscoring 2 -> ores1002 (non-production) (duration: 02m 24s)
  • 18:36 awight@tin: Started deploy [ores/deploy@437fcf5]: revscoring 2 -> ores1002 (non-production)
  • 18:32 demon@tin: Synchronized php-1.31.0-wmf.6/extensions/CentralNotice: deployyyyy (duration: 00m 52s)
  • 18:27 ejegg: updated CiviCRM from a34a9c6 to 415fde1
  • 17:23 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: wikidata back to wmf.5
  • 16:50 demon@tin: Finished scap: bootstrap wmf.6 (duration: 47m 39s)
  • 16:41 ppchelko@tin: Finished deploy [restbase/deploy@7631ea7]: Enable new parsoid storage on test wikipedias (duration: 09m 07s)
  • 16:32 ppchelko@tin: Started deploy [restbase/deploy@7631ea7]: Enable new parsoid storage on test wikipedias
  • 16:09 ariel@tin: Finished deploy [dumps/dumps@fa0583b]: adapt dumpadmin script to use override section in config (duration: 00m 02s)
  • 16:09 ariel@tin: Started deploy [dumps/dumps@fa0583b]: adapt dumpadmin script to use override section in config
  • 16:02 demon@tin: Started scap: bootstrap wmf.6
  • 15:43 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: all but wikidata.org -> wmf.5
  • 15:42 demon@tin: Synchronized php: Symlink bump (duration: 00m 48s)
  • 15:41 demon@tin: Pruned MediaWiki: 1.31.0-wmf.1 (duration: 03m 15s)
  • 14:18 bblack: authdns (baham, radon, eeden): upgrade base packages (curl, git, libc-dev, etc)
  • 14:13 bblack: lvsNNNN: upgrade base packages (curl, git, libc-dev, etc)
  • 14:11 bblack: cpNNNN: upgrade base packages (curl, git, libc-dev, etc)
  • 13:57 bblack: caches@eqiad - upgrade nginx to 1.13.6-2+wmf1~jessie1
  • 13:54 bblack: caches@esams - upgrade nginx to 1.13.6-2+wmf1~jessie1
  • 13:22 bblack: caches@codfw - upgrade nginx to 1.13.6-2+wmf1~jessie1
  • 13:18 zeljkof: EU SWAT finished
  • 13:17 zfilipin@tin: Synchronized php-1.31.0-wmf.4/extensions/EventBus/EventBus.php: SWAT: Dont log full request to service in case of delivery error (T179280) (duration: 00m 51s)
  • 13:14 bblack: caches@ulsfo - upgrade nginx to 1.13.6-2+wmf1~jessie1
  • 13:05 mobrovac@tin: Finished deploy [restbase/deploy@c0f9dc4]: Cassandra 3 Parsoid stash tables (duration: 02m 58s)
  • 13:02 mobrovac@tin: Started deploy [restbase/deploy@c0f9dc4]: Cassandra 3 Parsoid stash tables
  • 13:01 mobrovac@tin: Finished deploy [restbase/deploy@c0f9dc4]: Cassandra 3 Parsoid stash tables (duration: 15m 19s)
  • 12:45 mobrovac@tin: Started deploy [restbase/deploy@c0f9dc4]: Cassandra 3 Parsoid stash tables
  • 12:39 bblack: eqiad primary lvses (1001-3): disable pause [LRO not possible] on all ethernet interfaces (under pybal stopped briefly)
  • 12:36 bblack: codfw primary lvses (2003-6): disable LRO,pause on all ethernet interfaces (under pybal stopped briefly)
  • 12:24 bblack: cp4025: restart varnish-be for mailbox lag
  • 12:23 bblack: cp4023: restart varnish-be for mailbox lag
  • 12:21 bblack: esams primary lvses (3001-2): disable LRO,pause on eth0 (under pybal stopped briefly)
  • 12:19 bblack: ulsfo primary lvses (4001-2): disable LRO,pause on eth0 (under pybal stopped briefly)
  • 12:12 bblack: esams+ulsfo backup lvses (3003-4,4003-4): disable LRO,pause on eth0
  • 12:10 bblack: core DC backup lvses (1004-6,2004-6): disable LRO,pause on all ethernet interfaces
  • 11:05 oblivian@tin: Finished deploy [docker-pkg/deploy@20908e8]: Actually deploy the fix (duration: 00m 04s)
  • 11:05 oblivian@tin: Started deploy [docker-pkg/deploy@20908e8]: Actually deploy the fix
  • 11:02 oblivian@tin: Finished deploy [docker-pkg/deploy@576c80f]: Fix image_tag filter (duration: 00m 02s)
  • 11:02 oblivian@tin: Started deploy [docker-pkg/deploy@576c80f]: Fix image_tag filter
  • 10:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1084 original weight - T161088 (duration: 00m 48s)
  • 10:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1084 weight - T161088 (duration: 00m 50s)
  • 09:12 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2084 as multi-instance core host T178553 T178359 (duration: 00m 49s)
  • 08:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1084 weight - T161088 (duration: 00m 50s)
  • 08:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1084 with low weight - T161088 (duration: 00m 50s)
  • 08:15 marostegui: Stop MySQL on db2086 to copy s5 to db2089 - T178359
  • 07:09 marostegui: Stop MySQL on db2087.s6 to copy its content to db2089 - T178359
  • 06:37 marostegui: Stop MySQL on db1103 and db1084 - T161088
  • 06:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1084 - T161088 (duration: 00m 49s)
  • 06:17 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2039 (duration: 01m 12s)
  • 03:08 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Oct 31 03:08:48 UTC 2017 (duration 7m 18s)
  • 03:01 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.5) (duration: 10m 05s)
  • 02:34 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.4) (duration: 10m 33s)

2017-10-30

  • 23:28 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: Update cirrussearch MLR model for enwiki (duration: 00m 50s)
  • 23:22 ebernhardson@tin: Synchronized php-1.31.0-wmf.4/extensions/CirrusSearch/: Fix highlight tags inside section anchors of search results (duration: 01m 01s)
  • 23:19 ebernhardson@tin: Synchronized php-1.31.0-wmf.5/extensions/CirrusSearch/: Fix highlight tags inside section anchors of search results (duration: 01m 03s)
  • 22:03 eileen1: update process-control to revision is 2392ddf
  • 21:55 awight@tin: Finished deploy [ores/deploy@a0f7d5c]: Fun with deployments (non-production) T179336 (duration: 02m 03s)
  • 21:53 awight@tin: Started deploy [ores/deploy@a0f7d5c]: Fun with deployments (non-production) T179336
  • 21:48 awight@tin: Finished deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production) (duration: 01m 42s)
  • 21:47 chasemp: restart nova-fullstack to force a run
  • 21:46 awight@tin: Started deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production)
  • 21:36 eileen1: update CiviCRM from 25c7a33 to a34a9c6
  • 21:02 urandom: T179083: Creating commons_T_parsoid_stash_wikitextf0PBY8UXqY8UuiDv.{meta,data} (30 sec delay)
  • 20:56 urandom: T179083: Creating commons_T_parsoid_stash_htmlmXxc_uDhgnFQAdM8PPlH.{data,meta} (25 sec delay)
  • 20:46 urandom: T179083: Creating enwiki_T_parsoid_stash_section2ACMDK1DRzacK9nUB3.{data,meta} (15sec delay)
  • 20:36 urandom: T179083: Creating enwiki_T_parsoid_stash_dataWH8IDUS9SGI6LPpsJsLOQ.data
  • 20:35 arlolra: Updated Parsoid to 292633c4 (T133334)
  • 20:34 urandom: T179083: Creating enwiki_T_parsoid_stash_dataWH8IDUS9SGI6LPpsJsLOQ.meta
  • 20:33 urandom: T179083: Creating enwiki_T_parsoid_stash_dataWH8IDUS9SGI6LPpsJsLOQ
  • 20:28 urandom: T179083: Creating table enwiki_T_parsoid_stash_wikitextf0PBY8UXqY8UuiDv1.data
  • 20:27 urandom: T179083: Creating table enwiki_T_parsoid_stash_wikitextf0PBY8UXqY8UuiDv1.meta
  • 20:26 arlolra@tin: Finished deploy [parsoid/deploy@8a2c0ce]: Updating Parsoid to 292633c4 (duration: 07m 33s)
  • 20:25 urandom: T179083: Creating keyspace enwiki_T_parsoid_stash_wikitextf0PBY8UXqY8UuiDv1
  • 20:20 urandom: T179083: Creating table enwiki_T_parsoid_stash_htmlmXxc_uDhgnFQAdM8PPlH5.data
  • 20:18 arlolra@tin: Started deploy [parsoid/deploy@8a2c0ce]: Updating Parsoid to 292633c4
  • 20:14 urandom: T179083: Creating table enwiki_T_parsoid_stash_htmlmXxc_uDhgnFQAdM8PPlH5.meta
  • 20:09 awight@tin: Finished deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production) (duration: 14m 11s)
  • 20:08 urandom: T179083: Creating keyspace enwiki_T_parsoid_stash_htmlmXxc_uDhgnFQAdM8PPlH5
  • 19:55 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create Appendix NS on Burmese Wiktionary T178545 (duration: 00m 55s)
  • 19:55 awight@tin: Started deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production)
  • 19:07 awight@tin: Finished deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production) (duration: 06m 16s)
  • 19:01 awight@tin: Started deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production)
  • 19:01 awight@tin: Finished deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production) (duration: 19m 53s)
  • 18:41 awight@tin: Started deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production)
  • 18:40 awight@tin: Started deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production)
  • 18:22 awight@tin: Started deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production)
  • 18:16 thcipriani@tin: Synchronized portals: SWAT: Bumping portals to master T128546 (duration: 00m 50s)
  • 18:15 thcipriani@tin: Synchronized portals/prod/wikipedia.org/assets: SWAT: Bumping portals to master T128546 (duration: 00m 50s)
  • 18:08 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add Timeless skin to wikitech T154371 (duration: 00m 51s)
  • 17:08 madhuvishy: Revert dns switchover for c1 shards to c3 post labsdb1001 reboot T168584
  • 17:07 gehel@tin: Finished deploy [wdqs/wdqs@fdb9100]: GUI update (duration: 02m 17s)
  • 17:05 gehel@tin: Started deploy [wdqs/wdqs@fdb9100]: GUI update
  • 16:45 oblivian@tin: Finished deploy [docker-pkg/deploy@576c80f]: New version with small fixes (duration: 00m 04s)
  • 16:44 oblivian@tin: Started deploy [docker-pkg/deploy@576c80f]: New version with small fixes
  • 15:20 XioNoX: re-enabling Zayo transit in eqiad
  • 14:36 marostegui: Reboot labsdb1001 - T168584
  • 14:30 marostegui: Stop replication and MySQL on labsdb1001 - T168584
  • 14:09 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Revert "UBN! disbale ores for wikidata" (T179107) (duration: 00m 50s)
  • 14:07 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Revert "UBN! disbale ores for wikidata" (T179107) (duration: 00m 49s)
  • 14:00 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Change the af.wiktionary logo, part II (T178824) (duration: 00m 50s)
  • 13:59 ladsgroup@tin: Synchronized static/images/project-logos: Change the af.wiktionary logo, part I (T178824) (duration: 00m 50s)
  • 13:40 madhuvishy: Switched dns over for c1 shards to c3 in preparation of labsdb1001 reboot (Merged https://gerrit.wikimedia.org/r/#/c/386660/, Ran puppet on labservice1001|2)
  • 13:21 marostegui: Set innodb_max_dirty_pages_pct = 10 on labsdb1001 so it powers off a bit faster - T168584
  • 13:19 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable the Autopatrolled User Rights & Ext:SandboxLink on hiwikiversity (T179251 T179252) (duration: 00m 50s)
  • 12:58 reedy@tin: Synchronized wmf-config/db-labs.php: beta labs stuffs (duration: 00m 50s)
  • 12:49 oblivian@tin: Finished deploy [docker-pkg/deploy@e9b9146]: Proper deploy of first version (duration: 00m 07s)
  • 12:49 oblivian@tin: Started deploy [docker-pkg/deploy@e9b9146]: Proper deploy of first version
  • 12:02 hoo: Manually started dumpwikidatajson on snapshot1007 (cron run failed tonight)
  • 12:02 oblivian@tin: Finished deploy [docker-pkg/deploy@549c157]: (no justification provided) (duration: 00m 11s)
  • 12:01 oblivian@tin: Started deploy [docker-pkg/deploy@549c157]: (no justification provided)
  • 12:00 oblivian@tin: Finished deploy [docker-pkg/deploy@549c157]: (no justification provided) (duration: 00m 19s)
  • 11:59 oblivian@tin: Started deploy [docker-pkg/deploy@549c157]: (no justification provided)
  • 11:57 oblivian@tin: Finished deploy [docker-pkg/deploy@549c157]: (no justification provided) (duration: 00m 11s)
  • 11:57 oblivian@tin: Started deploy [docker-pkg/deploy@549c157]: (no justification provided)
  • 11:55 oblivian@tin: Started deploy [docker-pkg/deploy@549c157]: Test deploy for virtualenv creation
  • 11:48 hoo: Ran rebuildPropertyInfo for wikidatawiki and manually cleared the property cache (mc)
  • 11:44 hoo@tin: Synchronized wmf-config/Wikibase.php: Re-add property for RDF mapping of external identifiers for Wikidata (T179156, T178180) (duration: 00m 49s)
  • 11:33 hoo@tin: Synchronized wmf-config/Wikibase-production.php: Re-enable constraints check with SPARQL (T179156) (duration: 00m 50s)
  • 11:24 marostegui: Compress table cebwiki.externallinks on db2092 - T178359
  • 11:15 hoo@tin: rebuilt wikiversions.php and synchronized wikiversions files: Wikidatawiki back to wmf.5 (T179156)
  • 10:58 mobrovac@tin: Finished deploy [restbase/deploy@2b5889b]: Double-process all summaries and include the parsoid no-op switch, take #gazillion (duration: 08m 33s)
  • 10:52 akosiaris: poweroff copper T176957
  • 10:50 mobrovac@tin: Started deploy [restbase/deploy@2b5889b]: Double-process all summaries and include the parsoid no-op switch, take #gazillion
  • 10:31 marostegui: Stop MySQL on db2039 to copy its data to db2087.s6 - T178359
  • 10:27 mobrovac@tin: Finished deploy [restbase/deploy@2b5889b]: Double-process all summaries and include the parsoid no-op switch (duration: 04m 41s)
  • 10:22 mobrovac@tin: Started deploy [restbase/deploy@2b5889b]: Double-process all summaries and include the parsoid no-op switch
  • 10:03 mobrovac@tin: Finished deploy [restbase/deploy@2b5889b]: Double-process all summaries and include the parsoid no-op switch (duration: 12m 12s)
  • 09:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1097 original weight - T161088 (duration: 00m 52s)
  • 09:50 mobrovac@tin: Started deploy [restbase/deploy@2b5889b]: Double-process all summaries and include the parsoid no-op switch
  • 09:22 marostegui: Stop replication in sync on db2039 and db2046 to reimport tables
  • 09:08 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1097 api traffic - T161088 (duration: 00m 50s)
  • 09:08 marostegui: Drop wb_entity_per_page page from s3 and s5 - T177601
  • 08:51 ema: cp4022: restart varnish-be for mbox lag
  • 08:42 elukey: raised priority of refreshlink and htmlcacheupdate job execution on jobrunners (https://gerrit.wikimedia.org/r/#/c/386636/) - T173710
  • 08:39 marostegui: Stop replication on db2039 to reimport and compress tables
  • 08:38 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1097 with low weight - T161088 (duration: 00m 50s)
  • 08:33 moritzm: installing wget security updates
  • 08:32 gehel: rolling restart of wdqs for config reload - T175919
  • 08:31 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2039 (duration: 00m 50s)
  • 08:28 ariel@tin: Finished deploy [dumps/dumps@c204c72]: fix args initialization issue in getconfigvals script (duration: 00m 02s)
  • 08:28 ariel@tin: Started deploy [dumps/dumps@c204c72]: fix args initialization issue in getconfigvals script
  • 08:23 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2040 - T178359 (duration: 00m 50s)
  • 08:11 marostegui: Stop MySQL on db2086 to transfer s7 to db2087 - T178359
  • 06:51 marostegui: Stop MySQL on db1097 to clone db1103 - T161088
  • 06:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1097 - T161088 (duration: 00m 50s)
  • 06:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Move db1103 from s3 to s4 - T161088 (duration: 00m 49s)
  • 06:40 marostegui: Stop MySQL on db1103 to reclone it - T161088
  • 06:24 marostegui: Optimize dewiki.pagelinks and dewiki.templatelinks on db1063 (s5 master) - T174509
  • 06:12 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2040 - T178359 (duration: 00m 51s)
  • 06:08 marostegui: Stop MySQL on db2040 to populate s7 on db2086 - T178359
  • 05:58 marostegui: Deploy alter table on db1075 (s3 primary master) - T174509
  • 03:11 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Oct 30 03:11:37 UTC 2017 (duration 7m 11s)
  • 03:04 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.5) (duration: 11m 38s)
  • 02:35 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.4) (duration: 11m 18s)

2017-10-29

  • 23:49 ema: powercycle cp4024
  • 22:31 ariel@tin: Finished deploy [dumps/dumps@2aa2275]: fix keep setting to work with overrides (duration: 00m 02s)
  • 22:31 ariel@tin: Started deploy [dumps/dumps@2aa2275]: fix keep setting to work with overrides
  • 17:55 ariel@tin: Finished deploy [dumps/dumps@d8978ce]: add overrides section processing to config file (duration: 00m 04s)
  • 17:55 ariel@tin: Started deploy [dumps/dumps@d8978ce]: add overrides section processing to config file
  • 17:23 ariel@tin: Finished deploy [dumps/dumps@d426cf7]: batch 7z jobs, multistream job fixup (duration: 00m 02s)
  • 17:23 ariel@tin: Started deploy [dumps/dumps@d426cf7]: batch 7z jobs, multistream job fixup
  • 12:54 ema: cp4026: restart varnish-be for mbox lag

2017-10-28

  • 21:03 bblack: cp1067 (current target cache): disabling the relatively-new VCL that sets do_stream=false if !CL on applayer fetches...
  • 19:39 hoo@tin: Synchronized wmf-config/CommonSettings.php: Half the Flow -> Parsoid timeout (100s -> 50s) (T179156) (duration: 00m 51s)
  • 19:39 bblack: backend restart on cp1065
  • 18:39 bblack: restarting varnish backend on cp1053 to move the lag/503 issues to another box and buy more time to debug
  • 18:28 bblack: cp4025 - restart backend for mailbox lag (upload@ulsfo, unrelated to text-cluster issues)
  • 18:21 bblack: cp1053 - manual VCL change, backends appservers+api_appservers, reduce connect/firstbyte/betweenbytes timeoues from 5/180/60 to 3/20/10
  • 16:51 elukey: restart varnish backend on cp1055 - mailbox lag + T179156
  • 12:14 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1313.eqiad.wmnet
  • 12:10 elukey: manually killed (SIGTERM) hhvm on mw1313 - high load, hhvm-dump-debug not responsive
  • 12:01 elukey@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw1313.eqiad.wmnet
  • 11:53 elukey: restart hhvm on mw1285 - hhvm-dump-debug in /tmp/hhvm.17700.bt
  • 11:24 hoo@tin: Synchronized wmf-config/Wikibase-labs.php: Consistency sync (duration: 00m 50s)
  • 10:52 volans: restarted pdfrender on scb1001, was stuck since 2d with AssertionError: display is not set!

2017-10-27

  • 20:54 MaxSem: running migratePreferences.php on group2 wikis
  • 19:09 hoo: Ran scap pull on mwdebug1001
  • 18:20 awight@tin: Finished deploy [ores/deploy@185170f]: Test pip-9 scap trick on ores1002 (non-production) (duration: 02m 17s)
  • 18:18 awight@tin: Started deploy [ores/deploy@185170f]: Test pip-9 scap trick on ores1002 (non-production)
  • 17:54 hoo: Taking mwdebug1001 to do tests regarding T179156
  • 16:38 gehel: re-enabling wdqs-updater
  • 16:16 bblack: cp1054 varnish backend restarted (was 503s / bad-conns target of ongoing issues)
  • 16:16 gehel: wdqs updater is now stopped for real
  • 16:10 XioNoX: deactivating BGP sessions to Zayo in eqiad (flapping)
  • 15:58 gehel: disabling wdqs updater on all nodes
  • 15:50 hoo@tin: Synchronized wmf-config/Wikibase-production.php: Disable constraints check with SPARQL for now (T179156) (duration: 00m 50s)
  • 15:48 marostegui: Compress InnoDB on db2039 (s6) - T178359
  • 15:46 bblack: restart varnish-backend on cp4022 (upload@ulsfo) - mailbox
  • 14:49 bblack: turn on cp4024 port on asw-ulsfo
  • 13:52 bblack: reboot cp4021 to clean up oom messes
  • 13:49 bblack: restarting nginx on cp4021, without NUMA memory constraints
  • 12:10 marostegui: Optimize commonswiki.templatelinks on dbstore1001 - T162789
  • 12:03 elukey: execute systemctl reset-failed kafka-mirror-main-eqiad_to_jumbo-eqiad.service on kafka-jumbo hosts (old unit not deployed anymore)
  • 11:41 mutante: gerrit back - maintenance over
  • 11:39 mutante: gerrit restart to apply gerrit:386793 is imminent
  • 11:36 ema: cp4023: varnish-backend-restart for lag
  • 11:06 mobrovac@tin: Finished deploy [citoid/deploy@ff63420]: Update dependencies (duration: 03m 18s)
  • 11:04 marostegui: Optimize cebwiki.templatelinks on db1103 - T174509
  • 11:03 mobrovac@tin: Started deploy [citoid/deploy@ff63420]: Update dependencies
  • 08:56 marostegui: Optimize table commonswiki.recentchanges on dbstore1001 - T162789
  • 08:11 marostegui: Drop redundant indexes from pagelinks and templatelinks on s3 wikis only for dbstore2001 and dbstore2002 - T174509
  • 05:59 marostegui: Stop MySQL on db2084 to copy s5 to db2086 - T178359
  • 05:55 marostegui: dbstore1001: convert itwiki.pagelinks to TokuDB - T162789
  • 05:53 mutante: uploaded parsoid_0.8.0 to releases.wikimedia.org (for Subbu - T179134)
  • 05:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1060 - T174509 (duration: 00m 50s)
  • 03:02 bblack: cp1067, cp4026 - backend restarts, mailbox lag

2017-10-26

  • 22:47 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: wikidata to wmf.4
  • 22:39 bblack: restarting varnish-backend on cp1053 (mailbox lag from ongoing issues elsewhere?)
  • 21:40 bblack: raising backend max_connections for api.svc.eqiad.wmnet + appservers.svc.eqiad.wmnet from 1K to 10K on cp1053.eqiad.wmnet (current funnel for the bulk of the 503s)
  • 21:32 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Temporary disable remex html (T178632) (duration: 00m 50s)
  • 21:30 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Temporary disable remex html (T178632) (duration: 00m 50s)
  • 21:00 hoo: Fully revert all changes related to T178180
  • 20:59 hoo@tin: Synchronized wmf-config/Wikibase.php: Revert "Add property for RDF mapping of external identifiers for Wikidata" (T178180) (duration: 00m 50s)
  • 20:02 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: UBN! disbale ores for wikidata (T179107) (duration: 00m 50s)
  • 20:00 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: UBN! disbale ores for wikidata (T179107) (duration: 00m 50s)
  • 19:41 awight@tin: Finished deploy [ores/deploy@0adae70]: Increase extractor wikidata API timeout to 15s, T179107 (duration: 07m 25s)
  • 19:36 aaron@tin: Started restart [jobrunner/jobrunner@a20d043]: (no justification provided)
  • 19:34 awight@tin: Started deploy [ores/deploy@0adae70]: Increase extractor wikidata API timeout to 15s, T179107
  • 19:33 awight@tin: Finished deploy [ores/deploy@0adae70]: Increase extractor wikidata API timeout to 15s, T179107 (duration: 00m 10s)
  • 19:33 awight@tin: Started deploy [ores/deploy@0adae70]: Increase extractor wikidata API timeout to 15s, T179107
  • 19:27 demon@tin: Synchronized php-1.31.0-wmf.5/includes/libs/filebackend/SwiftFileBackend.php: Better error grouping (duration: 00m 50s)
  • 19:14 ladsgroup@tin: Synchronized php-1.31.0-wmf.5/extensions/WikibaseQualityConstraints/includes/ConstraintCheck/DelegatingConstraintChecker.php: Fix sorting of NullResults (T179038) (duration: 00m 49s)
  • 19:12 ladsgroup@tin: Synchronized php-1.31.0-wmf.5/extensions/WikibaseQualityConstraints/tests/phpunit/DelegatingConstraintCheckerTest.php: Fix sorting of NullResults (T179038) (duration: 00m 50s)
  • 18:52 ladsgroup@tin: Synchronized php-1.31.0-wmf.5/extensions/Wikidata/extensions/Constraints/tests/phpunit/DelegatingConstraintCheckerTest.php: Fix sorting of NullResults (T179038) (duration: 00m 49s)
  • 18:51 ladsgroup@tin: Synchronized php-1.31.0-wmf.5/extensions/Wikidata/extensions/Constraints/includes/ConstraintCheck/DelegatingConstraintChecker.php: Fix sorting of NullResults (T179038) (duration: 01m 04s)
  • 18:41 bblack: eqiad cp servers: rolling quick depool -> repool around ethtool parameter changes for -lro,-pause
  • 18:30 urandom: Dropping and recreating keyspace enwiki_T_page__summary (restbase-ng cluster)
  • 18:27 bblack: esams cp servers: rolling quick depool -> repool around ethtool parameter changes for -lro,-pause
  • 18:26 urandom: T179105: Altering "enwiki_T_page__summary".data to use LZ4Compressor
  • 17:59 bblack: codfw cp servers: rolling quick depool -> repool around ethtool parameter changes for -lro,-pause
  • 17:57 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4024.ulsfo.wmnet
  • 17:49 bblack: ulsfo cp servers: rolling quick depool -> repool around ethtool parameter changes for -lro,-pause
  • 17:47 awight@tin: Finished deploy [ores/deploy@f5deb7f]: Revscoring 2 on ores1002 (non-production) (duration: 01m 45s)
  • 17:45 awight@tin: Started deploy [ores/deploy@f5deb7f]: Revscoring 2 on ores1002 (non-production)
  • 17:36 arlolra: Updated Parsoid to 8e99708a (T176728)
  • 17:30 arlolra@tin: Finished deploy [parsoid/deploy@4882c59]: Updating Parsoid to 8e99708a (duration: 08m 40s)
  • 17:21 arlolra@tin: Started deploy [parsoid/deploy@4882c59]: Updating Parsoid to 8e99708a
  • 17:17 awight@tin: Finished deploy [ores/deploy@971be22]: Rolling back scb1002, T175180 T178441 (duration: 00m 32s)
  • 17:16 awight@tin: Started deploy [ores/deploy@971be22]: Rolling back scb1002, T175180 T178441
  • 17:14 awight@tin: Finished deploy [ores/deploy@971be22]: ORES w/ revscoring 2 and Celery 4, T175180 T178441 (duration: 03m 20s)
  • 17:11 awight@tin: Started deploy [ores/deploy@971be22]: ORES w/ revscoring 2 and Celery 4, T175180 T178441
  • 17:09 marostegui: Optimize ruwiki.recentchanges on dbstore1001 - T162789
  • 17:08 dcausse: [logstash] reimporting today logs
  • 17:08 dcausse: [logstash] deleting today logs to reapply mapping template
  • 16:58 ottomata: now mirroring main Kafka cluster topics to jumbo Kafka cluster, with MirrorMaker instances running on analytics-eqiad broker nodes. https://phabricator.wikimedia.org/T177216
  • 16:44 awight@tin: Finished deploy [ores/deploy@971be22]: Push ORES w/ Celery 4 support to new cluster (try to rebuild venv), T178441 (duration: 01m 02s)
  • 16:43 awight@tin: Started deploy [ores/deploy@971be22]: Push ORES w/ Celery 4 support to new cluster (try to rebuild venv), T178441
  • 16:33 awight@tin: Finished deploy [ores/deploy@971be22]: Push ORES w/ Celery 4 support to new cluster (take 3), T178441 (duration: 15m 33s)
  • 16:21 urandom: restarted cassandra, restbase2001-b
  • 16:18 awight@tin: Started deploy [ores/deploy@971be22]: Push ORES w/ Celery 4 support to new cluster (take 3), T178441
  • 16:15 urandom: truncating cassandra hints in restbase-ng cluster
  • 16:12 mobrovac@tin: Finished deploy [restbase/deploy@860cbfe]: Revert all summaries, back to all but WP (duration: 07m 57s)
  • 16:04 mobrovac@tin: Started deploy [restbase/deploy@860cbfe]: Revert all summaries, back to all but WP
  • 16:04 mobrovac@tin: Started deploy [restbase/deploy@860cbfe]: Revert all summaries, back to all but WP
  • 15:50 mobrovac@tin: Finished deploy [restbase/deploy@522321a]: Double-process all summaries (duration: 09m 09s)
  • 15:45 urandom: restarting cassandra, restbase2001-b
  • 15:41 andrewbogott: upgrading dnsmasq on labnet1001 and labnet1002
  • 15:41 mobrovac@tin: Started deploy [restbase/deploy@522321a]: Double-process all summaries
  • 15:36 dcausse: reindexing logs in logstash
  • 15:34 dcausse: deleting and reimporting logs in logstash (today data, might lose some logs in the process)
  • 14:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1101 - T178383 (duration: 00m 49s)
  • 14:04 zeljkof: EU SWAT finished
  • 14:03 zfilipin@tin: Synchronized php-1.31.0-wmf.5/extensions/Wikibase: SWAT: Make search for titles be always uppercase (T179045) (duration: 01m 36s)
  • 13:52 zfilipin@tin: Synchronized php-1.31.0-wmf.5/extensions/Wikidata: SWAT: Make search for titles be always uppercase (T179045) (duration: 02m 10s)
  • 13:46 Amir1: ladsgroup@terbium:~$ mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildPropertyInfo.php --wiki=wikidatawiki --rebuild-all --force (T178180)
  • 13:41 godog: delete old CF cassandra metrics - T173436
  • 13:32 godog: force delete old graphite cassandra metrics - T179057
  • 13:28 moritzm: installing icu security update on trusty
  • 13:28 zfilipin@tin: Synchronized wmf-config/Wikibase.php: SWAT: Add property for RDF mapping of external identifiers for Wikidata (T178180) (duration: 00m 50s)
  • 13:21 marostegui: Compress InnoDB on db2040 - T178359
  • 13:21 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable signature button in Projekt namespace on sewikimedia (T175363) (duration: 00m 50s)
  • 13:13 zfilipin@tin: Synchronized wmf-config/InterwikiSortOrders.php: SWAT: Fix interwiki sort order for Northern Sami (T178965) (duration: 00m 50s)
  • 12:57 marostegui@tin: Synchronized wmf-config/db-codfw.php: Reorganize future rc multi-instance hosts - T178359 (duration: 00m 50s)
  • 12:30 bblack: powercycle cp4024 (depooled)
  • 12:29 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4024.ulsfo.wmnet
  • 10:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Clean up some comments in s3 (duration: 00m 50s)
  • 09:01 moritzm: upgrade script runners to wikidiff2 1.5.1
  • 08:58 ema: cache_text/upload varnish-be: set rutime parameter timeout_idle to 120s
  • 08:34 moritzm: uploaded php-wikidiff2/hhvm-wikidiff2 1.5.1 for stretch-wikimedia
  • 08:22 godog: add 100G to graphite1003's /var/lib/carbon
  • 08:14 moritzm: upgrade deployment servers to wikidiff2 1.5.1
  • 07:57 marostegui: Drop databases in s1 and s2 from db1047 and unconfigure replication - T177405
  • 07:43 marostegui: Stop MySQL on db1047 to copy data over db1108 - T177405 T156844
  • 07:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1077 - T164488 (duration: 00m 50s)
  • 07:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1077 - T164488 (duration: 00m 50s)
  • 07:12 marostegui: Stop replication in sync on db1103 and db1077 to fix data drifts - T164488
  • 07:10 moritzm: upgrade mw1319-mw1328 to wikidiff2 1.5.1
  • 06:57 marostegui: Stop MySQL on db2088 and db2084 to copy s2 and s4 to db2091 - T178359
  • 06:57 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2038 (duration: 00m 50s)
  • 06:55 marostegui: Optimize pagelinks and templatelinks on db1095 s1 - T174509
  • 06:51 marostegui: Optimize recentchanges on s4 and s6 on labsdb1010 - T177772
  • 06:15 moritzm: upgrade mw1299-mw1311 to wikidiff2 1.5.1
  • 05:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1060 - T174509 (duration: 00m 50s)
  • 05:14 marostegui: Optimize recentchanges on s4 and s6 on labsdb1009 - T177772
  • 05:12 marostegui: Optimize pagelinks and templatelinks on db1060 - T174509
  • 03:01 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Oct 26 03:01:19 UTC 2017 (duration 6m 48s)
  • 02:54 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.5) (duration: 09m 30s)
  • 02:31 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.4) (duration: 10m 34s)

2017-10-25

  • 23:48 legoktm@tin: Synchronized php-1.31.0-wmf.5/extensions/VisualEditor/modules/ve-mw/ui/styles/contextitems/ve.ui.MWInternalLinkContextItem.css: MWInternalLinkContextItem: increase specificity to override OOUI changes - T178933 (duration: 00m 50s)
  • 23:39 legoktm@tin: Synchronized wmf-config/abusefilter.php: Change threshold for slow AbuseFilter logging to 800ms - T179039 (duration: 00m 50s)
  • 22:42 MaxSem: running migratePreferences.php on group1 wikis
  • 22:01 mobrovac@tin: Finished deploy [restbase/deploy@860cbfe]: Revert double-processing of all summaries to all except WPs, take #2 (duration: 09m 09s)
  • 21:52 mobrovac@tin: Started deploy [restbase/deploy@860cbfe]: Revert double-processing of all summaries to all except WPs, take #2
  • 21:48 mobrovac@tin: Started deploy [restbase/deploy@60ce036]: Revert double-processing of all summaries to all except WPs
  • 21:33 bblack: experimental - disabled LRO on lvs4001+lvs4002
  • 20:34 hasharAway: restarted zuul to flush a long queue in experimental pipeline (no other changes enqueued)
  • 20:21 demon@tin: Synchronized php-1.31.0-wmf.5/extensions/ProofreadPage/: unbreak multiline text input (duration: 00m 53s)
  • 19:28 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.5
  • 18:45 bblack: s/1\.13\.16/1.13.6/
  • 18:45 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable fine grained usage tracking on statement usage wikis T172914 (duration: 00m 50s)
  • 18:44 bblack: reprepro: uploaded nginx-1.13.16-2+wmf1 packages to stretch and jessie
  • 18:36 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Make disabled usage aspect use the new config T172914 (duration: 00m 50s)
  • 18:33 niharika29@tin: Synchronized php-1.31.0-wmf.4/extensions/Wikidata: Allow specifying a replacement usage type in disabledUsageAspects T178153 (duration: 02m 10s)
  • 18:25 niharika29@tin: Synchronized wmf-config/Wikibase.php: Make using CirrusSearch engine default for wbsearchentities T175741 (duration: 00m 48s)
  • 18:23 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Add AbuseFilterSlow channel to monolog T178853 (duration: 00m 50s)
  • 18:20 niharika29@tin: Synchronized php-1.31.0-wmf.5/extensions/ORES/: Update ApiHooks to conform with v3 responses T178962 (duration: 00m 50s)
  • 18:19 niharika29@tin: Synchronized php-1.31.0-wmf.4/extensions/ORES/: Update ApiHooks to conform with v3 responses T178962 (duration: 00m 51s)
  • 18:11 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable Special:EmailUser User Prohibit on All Wikis [mediawiki-config] - https://gerrit.wikimedia.org/r/386231 (duration: 00m 51s)
  • 17:17 moritzm: upgrade mw2224-mw2242 to wikidiff2 1.5.1
  • 16:59 moritzm: upgrade mw2190-mw2199 to wikidiff2 1.5.1
  • 16:38 no_justification: gerrit: purging last remnants of debian package and running puppet. service may briefly restart
  • 16:05 moritzm: upgrade mw2180-mw2189 to wikidiff2 1.5.1
  • 16:03 moritzm: copied arcconf, hp-health, hpacucli, hpssa, hpssacli, hpssaducli, lsiutil, megacli, megaclisas-status to stretch-wikimedia/thirdparty/hwraid (thirdparty will be dropped at a later point)
  • 15:39 marostegui: Optimize pagelinks and templatelinks on labsdb1011 - T174509
  • 15:31 marostegui: Power off db1101 for HW maintenance - T178383
  • 15:21 godog: bounce rsyslog on wezen
  • 15:09 mattflaschen@tin: Synchronized wmf-config/InitialiseSettings.php: T177836: Deploy Compact Language Links on the German Wikipedia (duration: 00m 49s)
  • 14:52 gehel: applying new template to elasticsearch / logstash - T178530
  • 14:51 ebernhardson: update logstash template on logstash elsaticsearch cluster for T178530
  • 14:47 mattflaschen@tin: Synchronized wmf-config/InitialiseSettings.php: T166759: Set $wgAutoloadAttemptLowercase to false (duration: 00m 50s)
  • 14:37 mattflaschen@tin: Synchronized php-1.31.0-wmf.5/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T177956: [cirrus] Turn off recall A/B test on enwiki (duration: 00m 49s)
  • 14:36 mattflaschen@tin: Synchronized php-1.31.0-wmf.4/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T177956: [cirrus] Turn off recall A/B test on enwiki (duration: 00m 49s)
  • 14:25 moritzm: moved jmxtrans and prometheus-jmx-exporter from thirdparty to main as part of repo reorg for stretch-wikimedia
  • 14:11 matt_flaschen: 'Deployed T178933: MWInternalLinkContextItem: increase specificity to override OOUI changes'
  • 14:10 mattflaschen@tin: Synchronized php-1.31.0-wmf.4/extensions/VisualEditor/modules/ve-mw/ui/styles/contextitems/ve.ui.MWInternalLinkContextItem.css: (no justification provided) (duration: 00m 50s)
  • 13:36 mattflaschen@tin: Synchronized wmf-config/InitialiseSettings.php: T168886: Updating wikis with consolidate editing feedback (duration: 00m 51s)
  • 13:30 godog: deploy thumbor 1.7 - T178974
  • 13:30 elukey: restart yarn nodemanager and hdfs datanode on analytics1030 to apply new JVM settings - T178876
  • 13:15 chasemp: merge arturo's key as part of ops
  • 13:04 matt_flaschen: About to run ULSCompactLinksDisablePref.php: Setting user preferences for Compact Language Links on dewiki, before removing Beta Feature and changing to regular preference. T177836
  • 12:36 moritzm: upgrade mw2163-mw2179 to wikidiff2 1.5.1
  • 12:17 mobrovac@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Activate warning logging for JobExecutor (duration: 00m 50s)
  • 12:16 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: Activate warning logging for JobExecutor (duration: 00m 50s)
  • 12:12 moritzm: upgrade mw1266-mw1275 to wikidiff2 1.5.1
  • 11:30 moritzm: upgrade mw1280-mw1290, mw1312-mw1318 to wikidiff2 1.5.1
  • 11:23 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2038 - T178359 (duration: 00m 49s)
  • 10:49 mobrovac@tin: Finished scap: wmf-config/jobqueue-labs.php beta-only change for CP4JQ (duration: 04m 43s)
  • 10:45 mobrovac@tin: Started scap: wmf-config/jobqueue-labs.php beta-only change for CP4JQ
  • 10:41 mobrovac@tin: Finished deploy [restbase/deploy@2ae5b0c]: Remove /title/{title}/ and double-process all summaries - T158100 (duration: 09m 41s)
  • 10:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1077 - T164488 (duration: 00m 50s)
  • 10:32 moritzm: upgrade remaining video scalers in eqiad to wikidiff2 1.5.1
  • 10:31 mobrovac@tin: Started deploy [restbase/deploy@2ae5b0c]: Remove /title/{title}/ and double-process all summaries - T158100
  • 09:31 moritzm: upgrade mw1221-mw1235 to wikidiff2 (API servers)
  • 09:07 gehel: rolling restart of all wdqs nodes for GC config change - T175919
  • 08:12 moritzm: upgrade mw1238-mw1258 to wikidiff2
  • 08:02 _joe_: decommissioning mw1168/69 as videoscalers, stopped jobrunner/jobchron, will stop hhvm/nginx/apache once current processing is done
  • 07:42 gehel@tin: Finished deploy [wdqs/wdqs@0bb2b5c]: wdqs-updater upgrade for jolokia - T167842 (duration: 01m 44s)
  • 07:40 gehel@tin: Started deploy [wdqs/wdqs@0bb2b5c]: wdqs-updater upgrade for jolokia - T167842
  • 07:05 marostegui: Optimize pagelinks and templatelinks on db1021 - T174509
  • 06:15 marostegui: Stop MySQL on db2038 to copy its data to db2084 - T178359
  • 06:08 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2035 - T178359 (duration: 00m 50s)
  • 06:02 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1065 - T174509 (duration: 00m 50s)
  • 05:44 marostegui: Stop replication in sync on db1103 and db1077 to checksum data - T164488
  • 05:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1077 - T164488 (duration: 01m 00s)
  • 03:22 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Oct 25 03:22:51 UTC 2017 (duration 7m 20s)
  • 03:15 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.5) (duration: 16m 46s)
  • 02:35 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.4) (duration: 08m 59s)

2017-10-24

  • 22:54 ejegg: updated civicrm from from 0546446 to 25c7a33
  • 21:24 ejegg: updated civicrm from 18edce1 to 0546446
  • 21:21 mepps: updated payments-wiki from 1011434 to b88b805
  • 20:47 arlolra@tin: Finished deploy [parsoid/deploy@UNKNOWN]: (no justification provided) (duration: 00m 13s)
  • 20:47 arlolra@tin: Started deploy [parsoid/deploy@UNKNOWN]: (no justification provided)
  • 20:29 mutante: restarting gerrit to make gerrit-ssh listen on all IPs again
  • 19:49 arlolra@tin: Finished deploy [parsoid/deploy@UNKNOWN]: Enabling logging for skwiki posts (duration: 00m 35s)
  • 19:48 arlolra@tin: Started deploy [parsoid/deploy@UNKNOWN]: Enabling logging for skwiki posts
  • 19:46 mutante: restarting gerrit on cobalt to apply change 354074
  • 19:13 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.5
  • 18:57 awight@tin: Finished deploy [ores/deploy@fb55ab8]: Rollback ORES to Revscoring 1.0, T175180 (duration: 10m 13s)
  • 18:47 awight@tin: Started deploy [ores/deploy@fb55ab8]: Rollback ORES to Revscoring 1.0, T175180
  • 18:47 ejegg: updated civicrm from b9d2067 to 18edce1
  • 18:26 awight@tin: Finished deploy [ores/deploy@fb55ab8]: Update ORES to revscoring 2.x, T175180 (take 2) (duration: 10m 03s)
  • 18:16 awight@tin: Started deploy [ores/deploy@fb55ab8]: Update ORES to revscoring 2.x, T175180 (take 2)
  • 18:05 arlolra: Updated Parsoid to a3be9cfc (T96923, T177784, T176568, T25467)
  • 17:57 arlolra@tin: Finished deploy [parsoid/deploy@af55c4b]: Updating Parsoid to a3be9cfc (duration: 09m 16s)
  • 17:48 arlolra@tin: Started deploy [parsoid/deploy@af55c4b]: Updating Parsoid to a3be9cfc
  • 17:28 demon@tin: Finished scap: bootstrap wmf.5 (duration: 48m 34s)
  • 17:18 urandom: T178845: Rolling Cassandra restart, eqiad
  • 17:07 awight@tin: Started deploy [ores/deploy@fb55ab8]: Update ORES to revscoring 2.x, T175180
  • 16:50 herron: upgrading puppet packages on codfw puppetmaster T177254
  • 16:39 demon@tin: Started scap: bootstrap wmf.5
  • 16:10 demon@tin: Pruned MediaWiki: 1.31.0-wmf.3 [keeping static files] (duration: 01m 20s)
  • 15:58 elukey: drop MediaViewer_10867062_15423246 and MobileWebUIClickTracking_10742159_15423246 from the log database on db1046 (archived on hadoop) - T168303
  • 15:53 urandom: T178845: Rolling Cassandra restart, codfw, row b
  • 15:48 elukey: drop table Edit_13457736_15423246 from the log database (Eventlogging) on db104[6,7], dbstore1002
  • 15:39 elukey: set net.netfilter.nf_conntrack_tcp_timeout_time_wait=65 to mw[1308-1311] - T136094
  • 15:09 gehel: restarting blazegraph on all wdqs nodes for GC config change - T175919
  • 14:48 _joe_: also stopping HHVM, nginx, apache2 , and disabling all the services
  • 14:46 _joe_: stopping the jobrunner service on mw1161-67
  • 14:39 marostegui: Optimize pagelinks templatelinks and recentchanges on db1030 - T174509 https://phabricator.wikimedia.org/T177772
  • 14:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1065 - T174509 (duration: 00m 45s)
  • 14:33 marostegui: Optimize ores_classification on enwiki db1065 - T159753
  • 14:32 marostegui: Optimize pagelinks and templatelinks on db1065 - T174509
  • 14:24 gehel: restarting blazegraph on wdqs2001 for GC config change - T175919
  • 14:20 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1106 weight (duration: 00m 45s)
  • 13:42 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2035 - T178359 (duration: 00m 45s)
  • 13:41 marostegui: Stop MySQL on db2035 to copy its data to db2088 - T178359
  • 13:39 zeljkof: EU SWAT finished
  • 13:39 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add $wgNamespaceRobotPolicies Config for hewikisource (T178775) (duration: 00m 46s)
  • 10:50 dcausse: cirrus/elasticsearch: reindexing of 167 small wikis done (T177871)
  • 09:22 marostegui: Stop s1 on db2092 to copy its data to db2088 - T178359
  • 08:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1082 original weight - T178460 (duration: 00m 45s)
  • 07:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1082 weight - T178460 (duration: 00m 45s)
  • 07:46 marostegui: Stop replication in sync on db1103 and db2018 to fix data drifts - T164488
  • 07:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1078 - T164488 (duration: 00m 46s)
  • 07:28 marostegui: Stop replication in sync on db1078 and db1103 to fix data drifts - https://phabricator.wikimedia.org/T164488
  • 06:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1078 - T164488 (duration: 00m 45s)
  • 06:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1082 with low weight - T178460 (duration: 00m 47s)
  • 02:32 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Oct 24 02:32:25 UTC 2017 (duration 6m 33s)
  • 02:25 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.4) (duration: 08m 13s)

2017-10-23

  • 23:09 demon@tin: Synchronized wmf-config/throttle.php: Working Class Movement Library (Salford) throttle rule (duration: 00m 46s)
  • 22:59 bblack: cp4024: varnish-backend-restart for lag
  • 21:40 XioNoX: increasing MTU on Zayo transit interfaces
  • 20:34 arlolra: Updated Parsoid to 1cefef12 (T178217)
  • 20:25 arlolra@tin: Finished deploy [parsoid/deploy@0759bd1]: Updating Parsoid to 1cefef12 (duration: 11m 14s)
  • 20:14 arlolra@tin: Started deploy [parsoid/deploy@0759bd1]: Updating Parsoid to 1cefef12
  • 19:37 bsitzmann@tin: Finished deploy [mobileapps/deploy@a654ce1]: Update mobileapps to 3628105 (T178828) (duration: 06m 59s)
  • 19:30 bsitzmann@tin: Started deploy [mobileapps/deploy@a654ce1]: Update mobileapps to 3628105 (T178828)
  • 19:00 joal@tin: Finished deploy [analytics/aqs/deploy@ad22e0c]: Upgrade AQS node modules to restbase-latests - Redeploy after config fix (duration: 02m 34s)
  • 18:58 joal@tin: Started deploy [analytics/aqs/deploy@ad22e0c]: Upgrade AQS node modules to restbase-latests - Redeploy after config fix
  • 18:58 joal@tin: Finished deploy [analytics/aqs/deploy@ad22e0c]: Upgrade AQS node modules to restbase-latests - Redeploy after config fix (duration: 05m 07s)
  • 18:52 joal@tin: Started deploy [analytics/aqs/deploy@ad22e0c]: Upgrade AQS node modules to restbase-latests - Redeploy after config fix
  • 18:52 joal@tin: Finished deploy [analytics/aqs/deploy@ad22e0c]: Upgrade AQS node modules to restbase-latests (duration: 321m 51s)
  • 17:56 gehel@tin: Finished deploy [wdqs/wdqs@b84592f]: WDQS GUI update (duration: 01m 58s)
  • 17:54 gehel@tin: Started deploy [wdqs/wdqs@b84592f]: WDQS GUI update
  • 17:48 XenoRyet: updated smashpig from d056a13 to 5262d53
  • 17:32 demon@tin: Synchronized scap/plugins/prep.py: Fixes and improvements of various sorts (duration: 00m 45s)
  • 17:26 gehel@tin: Finished deploy [wdqs/wdqs@4626acb]: deploying latest WDQS version, blazegraph + GUI updates (duration: 02m 07s)
  • 17:24 gehel@tin: Started deploy [wdqs/wdqs@4626acb]: deploying latest WDQS version, blazegraph + GUI updates
  • 16:53 demon@tin: Pruned MediaWiki: 1.30.0-wmf.19 (duration: 03m 09s)
  • 15:59 elukey: forced BBU learn cycle on db1046 - T166141
  • 15:00 herron: wiped resolver cache of records puppet.cofdw.wmnet and puppet.ulsfo.wmnet (T177254)
  • 14:46 herron: depooling codfw puppetmaster (via dns)
  • 14:31 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1078 - T164488 (duration: 00m 46s)
  • 14:05 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2037 - T178359 (duration: 00m 46s)
  • 13:48 marostegui: Stop mysql and poweroff db1082 for HW maintenance - T178460
  • 13:30 joal@tin: Started deploy [analytics/aqs/deploy@ad22e0c]: Upgrade AQS node modules to restbase-latests
  • 13:19 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Add aleph500.biblacad.ro to $wgCopyUploadsDomains - T178753 (duration: 00m 47s)
  • 13:19 gehel: rolling restart of wdqs for GC tuning - T175919
  • 13:18 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Remove hooserv.net from wgCopyUploadsDomains (duration: 00m 47s)
  • 13:08 zeljkof: EU SWAT finished, no deploys
  • 13:00 bblack@neodymium: conftool action : set/pooled=no; selector: dc=ulsfo,cluster=cache_text,name=cp4018.ulsfo.wmnet
  • 12:59 bblack@neodymium: conftool action : set/pooled=yes; selector: dc=ulsfo,cluster=cache_text,name=cp4032.ulsfo.wmnet
  • 12:53 bblack@neodymium: conftool action : set/pooled=no; selector: dc=ulsfo,cluster=cache_text,name=cp4017.ulsfo.wmnet
  • 12:47 bblack@neodymium: conftool action : set/pooled=yes; selector: dc=ulsfo,cluster=cache_text,name=cp4031.ulsfo.wmnet
  • 12:46 moritzm: upgrading hhvm-wikdiff2 on remaining job runners in codfw
  • 12:39 moritzm: upgrading hhvm-wikdiff2 on remaining video scalers in codfw
  • 12:32 bblack@neodymium: conftool action : set/pooled=no; selector: dc=ulsfo,cluster=cache_text,name=cp4010.ulsfo.wmnet
  • 12:29 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Remove hooserv.net from wgCopyUploadsDomains (duration: 00m 46s)
  • 12:26 moritzm: upgrading hhvm-wikdiff2 on mw2153-mw2162 (job runners)
  • 12:21 moritzm: upgrading hhvm-wikdiff2 on mw2254-mw2258 (app servers)
  • 12:20 bblack@neodymium: conftool action : set/pooled=yes; selector: dc=ulsfo,cluster=cache_text,name=cp4030.ulsfo.wmnet
  • 12:15 bblack@neodymium: conftool action : set/pooled=no; selector: dc=ulsfo,cluster=cache_text,name=cp4009.ulsfo.wmnet
  • 12:12 bblack@neodymium: conftool action : set/pooled=yes; selector: dc=ulsfo,cluster=cache_text,name=cp4029.ulsfo.wmnet
  • 12:03 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable description usage tracking on further wikis (T178515) (duration: 00m 47s)
  • 11:11 moritzm: upgrading hhvm-wikdiff2 on mw2097-mw2114
  • 11:10 marostegui: Compress innodb on db2038 and db2084 - T178359
  • 10:56 moritzm: upgrading hhvm-wikdiff2 on remaining API servers in codfw
  • 10:46 moritzm: upgrading hhvm-wikdiff2 on mw1161-mw1167 (job runners)
  • 10:28 moritzm: upgrading hhvm-wikdiff2 on mw1180-mw1188 (app servers)
  • 10:28 marostegui: Remove /srv from db1015 as it has been stopped for weeks now and will be decommissioned (and it is alerting low on disk space) - T173570
  • 10:09 dcausse: elasticsearch/cirrus reindexing 167 wikis from terbium (T177871)
  • 10:08 gehel: restart tilerator / karthotherian for config change on all maps clusters - T160215
  • 10:01 ema: restart varnish-be on cp4021 (mbox lag)
  • 09:58 moritzm: upgrading hhvm-wikidiff2 on mw1293-mw1298 (scalers)
  • 09:09 moritzm: upgrading hhvm-wikidiff2 on mw1189-mw1208 (API servers)
  • 08:09 marostegui: Rename wb_entity_per_page table on s5 db1092 and s3 db1077 - T177601
  • 08:07 moritzm: upgrading hhvm-wikdiff2 on mw1209-mw1220 (app servers)
  • 07:23 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2037 and db2038 - T178359 (duration: 00m 46s)
  • 06:56 moritzm: installing expat security updates on trusty
  • 06:51 marostegui: Stop replication in sync on db1078 and db1103 to checksum data - T164488
  • 06:50 moritzm: installing poppler security updates on trusty
  • 06:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1078 - T164488 (duration: 00m 47s)
  • 06:30 marostegui: Stop replication in sync on db1103 and db2018 to fix data drifts - T164488
  • 06:09 marostegui: Stop replication on db2092 to re-import and compress linter and watchlist table
  • 02:41 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Oct 23 02:41:04 UTC 2017 (duration 6m 55s)
  • 02:34 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.4) (duration: 08m 45s)

2017-10-21

  • 08:16 hashar: Mass code-review+2 changes made by LibraryUpdater that already had Code-Review:+2 and NOT verified=-1 ( ping legoktm )
  • 08:08 hashar: Stopping Zuul to flush its queue

2017-10-20

  • 23:46 demon@tin: Finished deploy [gerrit/gerrit@21b7332]: pushing deleteproject plugin (duration: 00m 09s)
  • 23:46 demon@tin: Started deploy [gerrit/gerrit@21b7332]: pushing deleteproject plugin
  • 19:19 hasharAway: restarting Jenkins - T178608
  • 18:52 bblack: vhtcpd upgrade + queue delay puppetization deploy ( https://gerrit.wikimedia.org/r/385415 ) done on cp* fleet - T133821
  • 18:28 bblack: upgrading vhtcpd to 0.1.1-1 on the cp* fleet
  • 17:38 mutante: removing Apache and HHVM Ganglia stats from all appservers, part of retiring Ganglia (T177225) - some transient puppet issues while purging package and remnants - use grafana dashboard at https://grafana.wikimedia.org/dashboard/db/prometheus-apache-hhvm-dc-stats
  • 16:39 aaron@tin: Synchronized php-1.31.0-wmf.4/extensions/TwoColConflict: 5c3224980fe (duration: 00m 46s)
  • 16:37 aaron@tin: Synchronized php-1.31.0-wmf.4/extensions/OATHAuth: c16a94e17b9981 (duration: 00m 47s)
  • 16:05 krinkle@tin: Synchronized php-1.31.0-wmf.4/extensions/WikimediaMaintenance/getJobQueueLengths.php: I8737795e6 (duration: 00m 47s)
  • 16:05 XioNoX: removing "policy-options policy-statement LVS_import term secondary" on cr routers
  • 14:39 cmjohnson1: updating power firmware analytics1037
  • 10:14 moritzm: upgrading hhvm-wikidiff2 to 1.5.1 on mw2150/mw2151/mw2244/mw2245
  • 09:17 moritzm: upgrading hhvm-wikidiff2 to 1.5.1 on mw2200-mw2223
  • 08:05 Amir1: mwscript createAndPromote.php --wiki=techconductwiki --sysop --bureaucrat 'Ladsgroup' --force
  • 07:39 jynus: stopping db2033, too
  • 07:36 jynus: stopping dbstore1002 and dbstore2002 x1 replication in sync
  • 07:18 _joe_: uploading updated versions of docker base images (wikimedia-jessie, wikimedia-stretch) to docker-registry.wikimedia.org
  • 06:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1078 - T164488 (duration: 00m 46s)
  • 06:17 marostegui: Stop replication in sync on db1103 and db1078 - T164488
  • 06:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1078 - T164488 (duration: 00m 45s)
  • 06:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1072 - T164488 (duration: 00m 46s)
  • 05:52 marostegui: Stop replication in sync on db1103 and db1072 - T164488
  • 05:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1051 - T174509 (duration: 00m 47s)
  • 02:19 bblack: upgrading vhtcpd to 0.1.1-1 on: cp1008, cp3030, cp3034, cp4029-32
  • 02:17 bblack: uploaded vhtcpd-0.1.1-1 to reprepro

2017-10-19

  • 22:21 addshore: stop running extra change dispatchers on terbium for wikidatawiki
  • 21:58 addshore: running extra change dispatchers on terbium for wikidatawiki for a short while
  • 21:24 andrewbogott: restarting nginx on nitrogen
  • 21:22 andrewbogott: restarting puppetdb on nitrogen
  • 20:00 mforns@tin: Finished deploy [analytics/refinery@0c9ba04]: deploying refinery to fix banner activity false alarms (duration: 00m 14s)
  • 19:59 mforns@tin: Started deploy [analytics/refinery@0c9ba04]: deploying refinery to fix banner activity false alarms
  • 19:53 bblack: reboot cp4029-32 (initial reboot post-puppetization)
  • 19:12 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.4
  • 18:36 thcipriani@tin: Synchronized portals: SWAT: Bumping portals to master T128546 (duration: 00m 51s)
  • 18:35 thcipriani@tin: Synchronized portals/prod/wikipedia.org/assets: SWAT: Bumping portals to master T128546 (duration: 00m 50s)
  • 18:28 thcipriani@tin: Synchronized wmf-config/abusefilter.php: SWAT: Enable $wgAbuseFilterProfile on ptwiki T177641 (duration: 00m 50s)
  • 18:20 thcipriani@tin: Synchronized php-1.31.0-wmf.4/extensions/Echo/modules/styles/mw.echo.ui.PageNotificationsOptionWidget.less: SWAT: mw.echo.ui.PageNotificationsOptionWidget: Fix CSS after changes in OOjs UI T178439 (duration: 00m 51s)
  • 16:50 hashar: contint1001: installed python3 (pending https://gerrit.wikimedia.org/r/385210 )
  • 15:14 marostegui: Stop replication in sync on db1103 and db1072 - https://phabricator.wikimedia.org/T164488
  • 15:02 mforns@tin: Finished deploy [analytics/refinery@0c9ba04]: (no justification provided) (duration: 00m 07s)
  • 15:02 mforns@tin: Started deploy [analytics/refinery@0c9ba04]: (no justification provided)
  • 14:36 mforns@tin: Finished deploy [analytics/refinery@0c9ba04]: (no justification provided) (duration: 03m 29s)
  • 14:33 mforns@tin: Started deploy [analytics/refinery@0c9ba04]: (no justification provided)
  • 13:20 moritzm: upgrading mw2120-mw2147 to wikidiff2 1.5.1
  • 12:35 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2034 and db2036 - T178359 (duration: 00m 50s)
  • 11:40 akosiaris: T177196 upload prometheus-postgres-exporter_0.2.0+ds-2 to apt.wikimedia.org/stretch-wikimedia/main and copied over to apt.wikimedia.org/jessie-wikimedia/main
  • 11:10 moritzm: upgrading mw1276-mw1279 (canary servers) to wikidiff2 1.5.1
  • 10:14 godog: upgrade prometheus-hhvm-exporter to 0.4-1
  • 10:07 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: Electron hanging - T174916
  • 09:54 ppchelko@tin: Finished deploy [restbase/deploy@2001a66]: Enable summary on Cass 3 for all but wikipedia (duration: 07m 32s)
  • 09:50 ariel@tin: Finished deploy [dumps/dumps@2b9326b]: improved job batches for rev content 7z recompression (duration: 00m 02s)
  • 09:50 ariel@tin: Started deploy [dumps/dumps@2b9326b]: improved job batches for rev content 7z recompression
  • 09:48 moritzm: installing pyjwt security updates
  • 09:47 ppchelko@tin: Started deploy [restbase/deploy@2001a66]: Enable summary on Cass 3 for all but wikipedia
  • 08:13 marostegui: Stop replication in sync on db1103 and db1072 - T164488
  • 08:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1072 - T164488 (duration: 00m 50s)
  • 07:31 marostegui: Stop MySQL on db2034 and db2036 to clone db2092
  • 06:36 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2034 and db2036 - T178359 (duration: 01m 08s)
  • 06:02 marostegui: Compress revision table across db1044 databases (s3)
  • 05:46 marostegui: Optimize recentchanges, pagelinks and templatelinks on db1102 for s6 - T174509 T177772
  • 05:45 marostegui: Optimize recentchanges, pagelinks and templatelinks on db1064 - T174509 T177772
  • 05:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1051 - T174509 (duration: 00m 50s)
  • 05:41 marostegui: Optimize pagelinks and templatelinks on db1051 - T174509
  • 05:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1055 - T174509 (duration: 00m 50s)
  • 05:31 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1056 - T174509 (duration: 00m 50s)
  • 05:29 awight@tin: Started deploy [ores/deploy@19b1851]: Push ORES w/ Celery 4 support (take 2), T178441
  • 05:08 awight@tin: Finished deploy [ores/deploy@19b1851]: Push ORES w/ Celery 4 support, T178441 (duration: 00m 18s)
  • 05:08 awight@tin: Started deploy [ores/deploy@19b1851]: Push ORES w/ Celery 4 support, T178441
  • 03:38 bblack: cp3034 - upgraded vhtcpd to 0.1.0 for testing
  • 03:38 bblack: cp3030 - upgraded vhtcpd to 0.1.0 for testing
  • 03:01 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Oct 19 03:01:54 UTC 2017 (duration 7m 2s)
  • 02:54 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.4) (duration: 09m 28s)
  • 02:29 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.3) (duration: 09m 40s)
  • 00:01 maxsem@tin: Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/385106/2 (duration: 00m 50s)

2017-10-18

  • 23:51 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/385094/2 (duration: 00m 50s)
  • 23:49 maxsem@tin: Synchronized static/images/project-logos/: https://gerrit.wikimedia.org/r/#/c/385094/2 (duration: 00m 50s)
  • 23:45 maxsem@tin: Synchronized php-1.31.0-wmf.4/extensions/Collection/: https://gerrit.wikimedia.org/r/#/c/384903/ (duration: 00m 50s)
  • 23:44 maxsem@tin: Synchronized php-1.31.0-wmf.3/extensions/Collection/: https://gerrit.wikimedia.org/r/#/c/385005/ (duration: 00m 51s)
  • 23:34 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/385007/ (duration: 00m 51s)
  • 23:05 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.4
  • 23:04 demon@tin: Synchronized php: symlink swap (duration: 00m 49s)
  • 22:55 tgr@tin: Synchronized private/: Remove the llama (T150554) (duration: 00m 50s)
  • 22:27 tgr@tin: Synchronized wmf-config/: T174651 / gerrit 384908 - deploy ReadingLists to beta - noop for production (duration: 00m 53s)
  • 20:49 ejegg: updated CiviCRM from 7c26791 to b9d2067
  • 20:16 XioNoX: Changing MTUs on interfaces to NTT
  • 19:19 bblack: uploaded vhtcpd-0.1.0-1 to jessie-wikimedia, testing on cp1008 only for now
  • 19:17 andrewbogott: restarting nodepool
  • 19:15 andrewbogott: stopping nodepool temporarily to give rabbit a chance to catch up
  • 19:01 MaxSem: ran LoginNotify/maintenance/migratePreferences.php on test and test2
  • 18:57 andrewbogott: restarting nova-network on labnet1001
  • 18:42 andrewbogott: restarting rabbitmq-server on labcontrol1001; too many timeouts
  • 16:29 elukey: stop ircecho on einstenium for puppet shower
  • 16:27 herron: restarted apache on puppetmasters
  • 14:23 moritzm: uploaded hhvm 3.18.5+dfsg-1+wmf1+deb9u1 to apt.wikimedia.org/stretch-wikimedia
  • 14:03 chasemp: add static ip to cloud mwstake project per T178012
  • 14:03 chasemp: bump up quota on cyberpower cloud project per T178332
  • 14:03 chasemp: create reading-lists cloud project
  • 13:28 zeljkof: EU SWAT finished!
  • 13:22 zfilipin@tin: Synchronized php-1.31.0-wmf.3/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: [cirrus] Turn on recall A/B test on enwiki (T177502) (duration: 00m 50s)
  • 13:21 gehel: repooling wdqs1003 now that it has catched up on updates
  • 13:15 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: pagePreviews: Restart A/B test on enwiki and dewiki (T176469) (duration: 00m 51s)
  • 13:05 moritzm: uploaded wikidiff2 1.5.1 to apt.wikimedia.org
  • 12:11 mobrovac@tin: Finished deploy [restbase/deploy@3c7abf6]: Bug fix: Remove the space in the summary table name (duration: 07m 13s)
  • 12:04 mobrovac@tin: Started deploy [restbase/deploy@3c7abf6]: Bug fix: Remove the space in the summary table name
  • 11:54 mobrovac@tin: Finished deploy [restbase/deploy@99052c1]: Use Cass3 storage for ruwiki summaries (duration: 08m 59s)
  • 11:47 jynus@tin: Synchronized wmf-config/db-eqiad.php: Add db1105 to help db1071 because db1082 has crashed (duration: 00m 50s)
  • 11:45 mobrovac@tin: Started deploy [restbase/deploy@99052c1]: Use Cass3 storage for ruwiki summaries
  • 11:31 moritzm: upgrading mw1262-mw1265 (canaries) to wikidiff2 1.5.1
  • 11:26 marostegui: Optimize pagelinks and templatelinks on db1102 s7 - T174509
  • 11:21 moritzm: upgrading mw1261 to wikidiff2 1.5.1
  • 11:11 mobrovac@tin: Finished deploy [restbase/deploy@15619e0]: Introduce the page/summary proxy (no-op for now), part #2 (duration: 08m 53s)
  • 11:03 mobrovac@tin: Started deploy [restbase/deploy@15619e0]: Introduce the page/summary proxy (no-op for now), part #2
  • 11:02 mobrovac@tin: Finished deploy [restbase/deploy@15619e0]: Introduce the page/summary proxy (no-op for now) (duration: 05m 23s)
  • 10:57 mobrovac@tin: Started deploy [restbase/deploy@15619e0]: Introduce the page/summary proxy (no-op for now)
  • 10:50 moritzm: rebooting multatuli for kernel update
  • 10:21 moritzm: imported linux 4.9.30-2+deb9u5~bpo8+1 for jessie-wikimedia
  • 09:51 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1098, depool db1082 (crashed) (duration: 00m 50s)
  • 09:39 moritzm: installing xserver/xvfb security updates
  • 09:36 jynus: reloading dbproxy1010's haproxy configuration
  • 09:30 elukey: drop MobileWikiAppToCInteraction_10375484_15423246 from the log database on dbstore1002,db1047,db1046 - T177960
  • 08:41 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Re-enable Statement usage tracking on cawiki (T151717) (duration: 00m 50s)
  • 08:25 marostegui: Stop MySQL on db2019 to copy its data to db2084 - T178359
  • 08:23 mobrovac@tin: Finished deploy [changeprop/deploy@065a06e]: Bug fix: Apply ignore_errors on a per-request basis (duration: 01m 17s)
  • 08:22 mobrovac@tin: Started deploy [changeprop/deploy@065a06e]: Bug fix: Apply ignore_errors on a per-request basis
  • 08:09 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable description usage tracking on a few test wikis (T177155) (duration: 00m 50s)
  • 07:49 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1056 - T174509 (duration: 00m 49s)
  • 07:46 marostegui: Optimize pagelinks and templatelinks on db1056 - T174509
  • 07:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1034 - T174509 (duration: 00m 59s)
  • 07:14 gehel: depooling wdqs1003 until it catches up on updates
  • 07:02 marostegui: Stop MySQL on db2079 to copy its data to db2084 - T178359
  • 06:46 _joe_: powercycling wdqs1003
  • 05:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1072 - T164488 (duration: 00m 49s)
  • 05:43 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1055 - T174509 (duration: 00m 50s)
  • 05:40 marostegui: Optimize pagelinks and templatelinks on db1055 - T174509
  • 05:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1098 - T174509 (duration: 00m 49s)
  • 05:36 marostegui: Optimize pagelinks and templatelinks on db1098 - T174509
  • 05:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1067 - T174509 (duration: 00m 50s)
  • 05:25 marostegui: Optimize pagelinks and templatelinks on db1102 for s2 - T174509
  • 03:17 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Oct 18 03:17:01 UTC 2017 (duration 7m 3s)
  • 03:09 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.4) (duration: 14m 58s)
  • 02:33 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.3) (duration: 08m 21s)

2017-10-17

  • 23:31 thcipriani@tin: Synchronized php-1.31.0-wmf.4/extensions/Citoid/modules/ve.ui.CiteFromIdInspector.css: SWAT: ve.ui.CiteFromIdInspector: Fix CSS for context menus after changes in OOjs UI T178324 (duration: 00m 50s)
  • 23:29 thcipriani@tin: Synchronized php-1.31.0-wmf.4/extensions/VisualEditor/modules/ve-mw/ui/styles/widgets/ve.ui.MWMediaInfoFieldWidget.css: SWAT: ve.ui.MWMediaInfoFieldWidget: Fix positioning of icons T178415 (duration: 00m 50s)
  • 23:22 thcipriani@tin: Synchronized php-1.31.0-wmf.3/includes/shell/Command.php: SWAT: Shell\Command: Better walltime fallback T178314 (duration: 00m 51s)
  • 23:15 thcipriani@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Remove setting no longer in MediaWiki (duration: 00m 50s)
  • 23:06 mutante: disabling Icinga notifications for service recommendation_api on scb hosts - please remember to re-enable once ticket is resolved (T178445) (https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=recommendation_api%20endpoints)
  • 22:28 ejegg: updated CiviCRM from fb513cc to 7c26791
  • 21:55 foks: Removed 2FA from User:Amakuru
  • 21:39 robh: cp4026 being returned to service post memory replacement. puppet runs fine and all icinga checks are green. repooled.
  • 21:23 robh: cp4026 coming down for memory swap, no point in maint moding the system since other cp checks will alert anyhow
  • 20:14 mutante: gerrit2001 - re-enabled puppet, attempting restart with --slave after gerrit:384758
  • 19:54 mutante: cobalt - re-enabled puppet, ran puppet, systemd says gerrit is active(running)
  • 19:51 mutante: cobalt - stopping gerrit service, confirming pid file gets removed, move /etc/init.d/gerrit to /root
  • 19:50 mutante: short gerrit downtime for systemd change coming up
  • 19:23 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.4
  • 19:06 mutante: gerrit2001 - rebooting to verify systemd change and service comes back properly
  • 18:53 mutante: gerrit2001: stopping the gerrit service
  • 18:52 mutante: cobalt: temp disable puppet | gerrit2001: stop puppet
  • 18:22 paravoid: rebooting sodium for kernel/distro upgrade
  • 18:14 paravoid: dist-upgrading sodium to stretch
  • 18:12 demon@tin: Finished scap: bootstrap wmf.4 (duration: 48m 41s)
  • 17:23 demon@tin: Started scap: bootstrap wmf.4
  • 16:49 demon@tin: Pruned MediaWiki: 1.31.0-wmf.2 [keeping static files] (duration: 01m 24s)
  • 16:35 gehel: restart tilerator / tileratorui on all maps cluster after deployment of latest fix - T177389
  • 16:34 gehel@tin: Finished deploy [tilerator/deploy@f3b26f3]: fix tileshell not exiting - T177389 (duration: 01m 42s)
  • 16:32 gehel@tin: Started deploy [tilerator/deploy@f3b26f3]: fix tileshell not exiting - T177389
  • 15:46 _joe_: killed a stuck gerrit process on cobalt
  • 14:47 bd808: wikitech: Cleaned up 'importers' user_group entries (T171682)
  • 14:25 Amir1: end of cleaning up ores_classification table in plwiki, start of ruwiki (T159753)
  • 14:09 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1081 - T174509 T177772 (duration: 00m 45s)
  • 14:05 Amir1: start of cleaning up ores_classification table in plwiki (T159753)
  • 14:00 zeljkof: EU SWAT finished
  • 13:59 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: [cirrus] Fix typo in UserTesting config var (take 2) (T177502) (duration: 00m 45s)
  • 13:47 zeljkof: EU SWAT continues
  • 13:31 zeljkof: stopping EU SWAT
  • 13:29 jynus: killing commonswiki script on tin
  • 13:17 zfilipin@tin: Synchronized php-1.31.0-wmf.3/extensions/UniversalLanguageSelector/maintenance/ULSCompactLinksDisablePref.php: SWAT: Remove the 20 edits threshold from ULSCompactLinksDisablePref.php (duration: 00m 46s)
  • 12:53 volans: upgrading cumin masters to cumin 1.2.2-1
  • 12:40 moritzm: upgrading mwdebug* to hhvm-wikidiff 1.5.1
  • 11:47 marostegui: Optimize recentchanges on db1102 for s4 - T174509
  • 11:47 marostegui: Optimize templatelinks and pagelinks on db1102 for s4 - T174509
  • 11:45 Amir1: end of cleaning up ores_classification table in hewiki, start of nlwiki (T159753)
  • 11:39 Amir1: end of cleaning up ores_classification table in fiwiki, start of hewiki (T159753)
  • 11:32 jynus: test-installing proxysql on wasat T175672
  • 11:24 Amir1: end of cleaning up ores_classification table in fawiki, start of fiwiki (T159753)
  • 11:12 gehel: restarting wdqs-updater on wdqs1004 - T175919
  • 10:40 Amir1: end of cleaning up ores_classification table in etwiki, start of fawiki (T159753)
  • 10:36 Amir1: end of cleaning up ores_classification table in cswiki, start of etwiki (T159753)
  • 10:26 Amir1: start of cleaning up ores_classification in cswiki (T159753)
  • 10:16 akosiaris: removing akosiaris's yubikey from network devices
  • 09:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1085 - T174509 T177772 (duration: 00m 46s)
  • 09:23 marostegui: Stop MySQL on db2084 for multi-instance testing
  • 09:16 mobrovac@tin: Finished deploy [restbase/deploy@7f228d4]: Remove /page/revision and add history metrics end points, part #3 (duration: 07m 34s)
  • 09:08 mobrovac@tin: Started deploy [restbase/deploy@7f228d4]: Remove /page/revision and add history metrics end points, part #3
  • 09:08 mobrovac@tin: Finished deploy [restbase/deploy@3bcd1b7]: Remove /page/revision and add history metrics end points, part #2 (duration: 25m 57s)
  • 08:45 gehel: restarting blazegraph on wdqs1004 for GC tuning (adding -XX:+G1PrintRegionLivenessInfo) - T175919
  • 08:42 mobrovac@tin: Started deploy [restbase/deploy@3bcd1b7]: Remove /page/revision and add history metrics end points, part #2
  • 08:40 mobrovac@tin: Finished deploy [restbase/deploy@3bcd1b7]: Remove /page/revision and add history metrics end points - T158100 T175805 (duration: 01m 05s)
  • 08:39 mobrovac@tin: Started deploy [restbase/deploy@3bcd1b7]: Remove /page/revision and add history metrics end points - T158100 T175805
  • 07:54 marostegui: Stop MySQL and reboot db2081 to see if it works fine - T178140
  • 07:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2086 to the config - T170662 (duration: 00m 45s)
  • 07:52 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2086 to the config - T170662 (duration: 00m 45s)
  • 07:51 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2086 to the config - T170662 (duration: 00m 45s)
  • 07:42 marostegui: Stop MySQL on db2079 to clone db2086 - T170662
  • 07:36 moritzm: rebooting boron for kernel update
  • 07:31 ema: upgrade misc_eqiad to varnish 5 T177233
  • 07:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1067 - T174509 (duration: 00m 45s)
  • 07:11 marostegui: Optimize templatelinks and pagelinks on db1067 - T174509
  • 07:10 marostegui: Optimize enwiki.ores_classification on db1067 - T159753
  • 06:55 moritzm: installing libffi security updates on trusty (Debian already fixed)
  • 06:07 marostegui: Stop replication in sync on db1072 and db1103 for data drift fixing - T164488
  • 05:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1034 - T174509 (duration: 00m 45s)
  • 05:54 marostegui: Optimize pagelinks and templatelinks on db1034 - T174509
  • 05:50 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1085 - T174509 T177772 (duration: 00m 45s)
  • 05:46 marostegui: Optimize pagelinks, templatelinks and recentchanges on db1085 T177772 T174509
  • 05:42 marostegui: Optimize pagelinks, templatelinks and ores_classification on db1095 - T174509 T159753
  • 05:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1081 - T174509 (duration: 00m 45s)
  • 05:38 marostegui: Optimize recentchanges on db1081 - T177772
  • 05:37 marostegui: Optimize pagelinks and templatelinks on db1081 - T174509
  • 05:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1090 - T174509 (duration: 00m 45s)
  • 05:32 marostegui: Optimize pagelinks and templatelinks on db1090 - T174509
  • 05:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1073 - T174509 (duration: 00m 46s)
  • 02:32 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Oct 17 02:32:30 UTC 2017 (duration 6m 57s)
  • 02:25 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.3) (duration: 08m 23s)
  • 00:34 ejegg: updated CiviCRM from 824c2d8 to fb513cc

2017-10-16

  • 23:49 thcipriani@tin: Synchronized php-1.31.0-wmf.3/extensions/CheckUser/specials/SpecialCheckUser.php: SWAT: Revert "Revert "Restore checkuser-userlinks when exist"" T170507 (duration: 00m 46s)
  • 23:46 ebernhardson: increase elasticsearch log levels for transport to debug to help track down T167410
  • 23:39 ejegg: updated civicrm from b953064 to 824c2d8
  • 23:25 thcipriani@tin: Synchronized php-1.31.0-wmf.3/extensions/Echo/includes/ContainmentSet.php: SWAT: ContainmentSet: Use strict comparison for array_search() T177825 (duration: 01m 45s)
  • 20:35 chasemp: disable puppet around cloud things for a refactor merge
  • 20:20 demon@tin: Pruned MediaWiki: 1.30.0-wmf.18 (duration: 02m 59s)
  • 19:59 demon@tin: Synchronized wmf-config/CommonSettings.php: Ief0ea01f (duration: 00m 46s)
  • 19:58 jynus: sudo apt install wmf-mariadb101-client on terbium for debugging an issue
  • 19:57 demon@tin: Synchronized wmf-config/InitialiseSettings.php: Ief0ea01f (duration: 00m 46s)
  • 19:56 demon@tin: Synchronized tests/cirrusTest.php: Ief0ea01f (duration: 00m 47s)
  • 18:18 thcipriani@tin: Synchronized wmf-config/Wikibase.php: SWAT: Add configuration for statement indexing for Wikidata T175199 (duration: 00m 47s)
  • 17:35 ariel@tin: Finished deploy [dumps/dumps@2b9326b]: do the minimum for latestlinks job, no updating other files (duration: 00m 03s)
  • 17:35 ariel@tin: Started deploy [dumps/dumps@2b9326b]: do the minimum for latestlinks job, no updating other files
  • 17:08 gehel@tin: Finished deploy [wdqs/wdqs@b42eb0d]: WDQS GUI uipdates (duration: 02m 20s)
  • 17:06 gehel@tin: Started deploy [wdqs/wdqs@b42eb0d]: WDQS GUI uipdates
  • 16:56 ottomata: deploying LVS for druid-public-broker https://phabricator.wikimedia.org/T176223
  • 16:39 joal@tin: Finished deploy [analytics/aqs/deploy@339674c]: Deploy new mediawiki-history endpoints (alpha version) after monitorin fix (duration: 04m 42s)
  • 16:35 joal@tin: Started deploy [analytics/aqs/deploy@339674c]: Deploy new mediawiki-history endpoints (alpha version) after monitorin fix
  • 16:23 joal@tin: Finished deploy [analytics/aqs/deploy@aa7ef0d]: Deploy new mediawiki-history endpoints (alpha version) after node-modules fix (duration: 06m 56s)
  • 16:16 joal@tin: Started deploy [analytics/aqs/deploy@aa7ef0d]: Deploy new mediawiki-history endpoints (alpha version) after node-modules fix
  • 16:09 ema: upgrade misc_codfw to varnish 5 T177233
  • 16:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1084 - T174509 T177772 (duration: 00m 46s)
  • 15:51 gehel: killing stuck tileshell process on maps-test2001 - T177389
  • 15:51 gehel@tin: Finished deploy [tilerator/deploy@f3b26f3]: testing tileshell fix - T177389 (duration: 00m 20s)
  • 15:50 gehel@tin: Started deploy [tilerator/deploy@f3b26f3]: testing tileshell fix - T177389
  • 15:45 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1094 - T174509 (duration: 00m 46s)
  • 15:38 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1088 - T174509 T177772 (duration: 00m 47s)
  • 14:34 twentyafterfour: deploying hotfix for T178068 and T178107
  • 14:30 moritzm: uploaded wikidiff 1.5.1 to apt.wikimedia.org for jessie/experimental
  • 14:08 hashar: European swat is complete
  • 14:08 hashar@tin: Synchronized wmf-config/abusefilter.php: (T177760) Enable AbuseFilter profiler | (T177761) Allow AbuseFilter to block & grant relevant permissions to permit administrators to manage the restricted option. (duration: 00m 47s)
  • 13:50 moritzm: upgrading deployment-mediawiki0[56] to wikidiff2 1.5.1
  • 13:45 hashar@tin: Synchronized wmf-config/: [cirrus] Prepare profiles for the recall A/B test on enwiki - T177502 (duration: 00m 48s)
  • 13:44 moritzm: upgrading deployment-mediawiki04 to wikidiff2 1.5.1
  • 13:42 hashar@tin: Synchronized wmf-config: [cirrus] Disable A/B test of MLR on testing on 18 wikis with > 1% of search traffic - T177490 (duration: 00m 48s)
  • 13:38 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Add additional namespaces to search results for bnwikisource - T178041 (duration: 00m 49s)
  • 12:59 gehel: restart cassandra on maps1001
  • 12:38 ema: upgrading eqiad LVSs to pybal 1.14.2 (T178149, T177815)
  • 11:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1074 - T174509 (duration: 00m 46s)
  • 11:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1070 - T174509 (duration: 00m 46s)
  • 11:10 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: Electron hanging - T174916
  • 10:48 Amir1: cleaning up ores_classification table in wikidatawiki (T159753)
  • 10:03 ema: upgrading codfw LVSs to pybal 1.14.2 (T178149, T177815)
  • 09:54 ema: upgrading esams LVSs to pybal 1.14.2 (T178149, T177815)
  • 09:44 ema: upgrading ulsfo LVSs to pybal 1.14.2 (T178149, T177815)
  • 09:43 ema: pybal 1.14.2 uploaded to apt.w.o
  • 09:40 moritzm: restarting cassandra on restbase1007 to pick up NSS security update
  • 09:39 moritzm: restarting cassandra on cerium/xenon/praseodymium to pick up NSS security update
  • 09:37 gehel: restarting elasticsearch on elastic1017
  • 09:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1088 - T174509 T177772 (duration: 00m 46s)
  • 09:07 marostegui: Optimize recentchanges, pagelinks and templatelinks on db1088 - https://phabricator.wikimedia.org/T174509 https://phabricator.wikimedia.org/T177772
  • 09:02 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1093 - T174509 T177772 (duration: 00m 46s)
  • 08:53 marostegui: Stop MySQL on db1050 as it will be decommissioned - T178162
  • 08:46 marostegui: Remove db1050 from tendril - T178162
  • 08:31 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1050 from config - T178162 (duration: 00m 46s)
  • 08:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1050 from config - T178162 (duration: 00m 46s)
  • 07:56 marostegui: Stop db1103 and db1038 at the same position to fix data drifts - T164488
  • 07:42 moritzm: installing NSS security updates
  • 07:30 marostegui: Stop db1103 and db1072 at the same position to fix data drifts - T164488
  • 07:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1072 - T164488 (duration: 00m 46s)
  • 07:19 marostegui: Optimize table ores_classification on db1073 - T159753
  • 07:14 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2084 to the config - T170662 (duration: 00m 46s)
  • 07:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2084 to the config - T170662 (duration: 00m 47s)
  • 06:25 marostegui: Stop MySQL on db2079 to clone db2084 from it - T170662
  • 06:12 marostegui: Optimize pagelinks and templatelinks on db1094 - T174509
  • 06:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1093 - T174509 T177772 (duration: 00m 46s)
  • 06:04 marostegui: Optimize recentchanges, pagelinks and templatelinks on db1093 - T174509 T177772
  • 06:00 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1070 - T174509 (duration: 00m 46s)
  • 05:59 marostegui: Optimize recentchanges, pagelinks and templatelinks on db1070 - T174509
  • 05:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1084 - T174509 T177772 (duration: 00m 46s)
  • 05:53 marostegui: Optimize recentchanges, pagelinks and templatelinks on db1084 - T174509 T177772
  • 05:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1074 - T174509 (duration: 00m 46s)
  • 05:47 marostegui: Optimize pagelinks and templatelinks on db1074 - T174509
  • 05:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1073 - T174509 (duration: 00m 47s)
  • 05:37 marostegui: Optimize templatelinks and pagelinks on db1073 - T174509
  • 05:36 marostegui: Optimize templatelinks and pagelinks on db1073 - 174509
  • 02:35 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Oct 16 02:35:34 UTC 2017 (duration 6m 36s)
  • 02:28 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.3) (duration: 07m 44s)

2017-10-15

  • 11:15 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: Electron hanging - T174916

2017-10-14

  • 03:52 paravoid: manually cleaning up pdns_metric_cron from all dnsrecursors (hydrogen/chromium/acamar/achernar/maerlant/nescio)

2017-10-13

  • 23:04 mutante: scb1001 - systemctl restart pdfrender (Icinga said: pdfrender Connection refused)
  • 20:36 mutante: gerrit2001 - removing hhvm package (which is now possible without also removing scap, thanks thcipriani), apt-get autoremove to remove more unused packages (T178039)
  • 18:52 joal@tin: Finished deploy [analytics/aqs/deploy@0375c34]: Deploy new mediawiki-history endpoints-alpha version (duration: 15m 02s)
  • 18:40 joal@tin: (no justification provided)
  • 18:37 joal@tin: Started deploy [analytics/aqs/deploy@0375c34]: Deploy new mediawiki-history endpoints-alpha version
  • 16:54 otto@tin: Finished deploy [analytics/refinery@28db253]: (no justification provided) (duration: 05m 50s)
  • 16:48 otto@tin: Started deploy [analytics/refinery@28db253]: (no justification provided)
  • 15:31 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1056 (duration: 00m 46s)
  • 14:59 mutante: added cparle to wmf LDAP group
  • 14:38 jynus: stopping slave on db1056 and optimize recentchanges
  • 14:25 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1100, depool db1056, fix db1098 weight (duration: 00m 46s)
  • 14:06 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1053 (duration: 00m 46s)
  • 13:07 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1053 (duration: 00m 47s)
  • 12:57 godog: upload scap 3.7.1-1 - T127762
  • 11:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2082 to the config - T170662 (duration: 00m 46s)
  • 11:26 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2082 to the config - T170662 (duration: 00m 46s)
  • 10:53 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1098 (duration: 00m 47s)
  • 10:22 godog: roll-restart pybal to apply https://gerrit.wikimedia.org/r/#/c/383997/ - T178149
  • 10:04 jynus: stopping slave on db1098 and optimize recentchanges
  • 09:54 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1098 (duration: 00m 46s)
  • 09:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2082 to the config - T170662 (duration: 00m 46s)
  • 09:22 Amir1: a small clean up of ores_classification table in wikidatawiki (T159753)
  • 09:21 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2082 to the config - T170662 (duration: 00m 47s)
  • 09:04 godog: restart pybal on lvs1003 to test logstash depool - T178078
  • 08:57 marostegui: Stop db1103 and db1038 in sync for more checksumming - T164488
  • 08:34 ema: upgrade pybal on lvs1003 to 1.14.1 T177815
  • 08:18 ema: upgrade pybal on lvs1006 to 1.14.1 T177815
  • 08:15 jynus: restarting commons wiki recentchanges purge of wb entries T177772
  • 08:11 ema@neodymium: conftool action : set/pooled=yes; selector: name=logstash1001.eqiad.wmnet,service=logstash-gelf,dc=eqiad
  • 08:11 ema@neodymium: conftool action : set/pooled=no; selector: name=logstash1001.eqiad.wmnet,service=logstash-gelf,dc=eqiad
  • 08:10 ema: depool/repool logstash1001 (service=logstash-gelf) T178078
  • 07:35 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2081 to the config - T170662 (duration: 00m 46s)
  • 07:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2081 to the config - T170662 (duration: 00m 47s)
  • 06:23 marostegui: Stop MySQL on db2080 to copy its data to db2081 - T170662
  • 00:22 mutante: authdns servers: deleted /etc/ganglia/conf.d/gndsd.pyconf & /usr/lib/ganglia/python_modules_gdnsd.py - removing ganglia stats

2017-10-12

  • 23:58 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Restore thanks limit on pl.wikipedia (T169268) (duration: 00m 47s)
  • 23:50 cwd: re-enabled omnimail jobs
  • 23:35 eileen1: update from 2d6668b to b953064
  • 23:25 dereckson@tin: Synchronized wmf-config/CommonSettings.php: Add logging for email blocks (T175419) (duration: 00m 46s)
  • 23:13 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Switch test wikis to HTML5 fragment mode in links (T152540) (duration: 00m 47s)
  • 22:44 demon@tin: Synchronized php-1.31.0-wmf.3/includes/specialpage/ChangesListSpecialPage.php: silence notices (duration: 00m 47s)
  • 21:36 cwd: disabled omnimail jobs
  • 19:47 ejegg: updated CiviCRM from 47de976 to 2d6668b
  • 19:46 chasemp: disable puppet for lab*services* for merge of 383896
  • 19:18 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.3
  • 18:28 mutante: added Pablo (pgrass) to LDAP group 'wmde' (T177599)
  • 18:15 demon@tin: Synchronized php-1.31.0-wmf.3/extensions/ORES/includes/Hooks.php: Make watchlist queries suck less (duration: 00m 50s)
  • 18:01 demon@tin: Synchronized php-1.31.0-wmf.3/extensions/JsonConfig/includes/JCCache.php: Silence junk warnings (duration: 00m 50s)
  • 15:17 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: Electron hanging - T174916
  • 15:11 mutante: gerrit2001 - rebooting for kernel upgrade
  • 14:57 mutante: netmon1002 - arming keyholder for rancid deploy
  • 14:52 mutante: netmon1002 (librenms, rancid) - rebooting for kernel upgrade
  • 14:47 moritzm: rebooting kubernetes workers to new 4.9.51 kernel
  • 14:41 moritzm: rebooting kubernetes staging servers to new 4.9.51 kernel
  • 14:29 moritzm: rebooting kubernetes masters to new 4.9.51 kernel
  • 13:54 moritzm: updating image scalers to librsvg 2.40.18
  • 13:48 elukey: deployed the new Analytics Public Druid cluster - T176223
  • 13:45 moritzm: updating remaining thumbor servers to librsvg 2.40.18
  • 13:40 ladsgroup@tin: Synchronized wmf-config/Wikibase.php: SWAT: Remove deprecated config variable for Wikibase (T129475) (duration: 01m 02s)
  • 13:33 moritzm: updating thumbor1001 to librsvg 2.40.18
  • 13:13 moritzm: rolling restart of logstash to pick up thumbor logs (T150734)
  • 13:07 zeljkof: EU SWAT finished
  • 12:56 moritzm: uploaded librsvg 2.40.18 for jessie-wikimedia to apt.wikimedia.org
  • 11:46 ema: upgrading cp3010 to Varnish 5 T177233
  • 11:27 Amir1: finished cleanup of ores_classification in wikidatawiki, still needs more work (T159753)
  • 10:03 ema: lvs1005: upgrade pybal to 1.14.1 T177815
  • 10:02 ema: pybal 1.14.1 uploaded to apt.w.o
  • 09:40 ema: upgrading cp3008 to Varnish 5 T177233
  • 09:06 moritzm: installing nettle update from stretch 9.2 point release
  • 09:01 moritzm: installing ncurses security updates
  • 08:25 moritzm: installing gnutls28 update from stretch 9.2 point release
  • 08:02 Amir1: starting cleanup of ores_classification in wikidatawiki (T159753)
  • 07:51 gehel: reinitialize cassandra on maps-test2004 to test vector tiles - T153282
  • 07:40 moritzm: installing at-spi2-core update from stretch 9.2 point release
  • 07:24 moritzm: installing gnupg2 update from stretch 9.2 point release
  • 07:18 legoktm: purging html5-misnesting linter entries from database on terbium (T178040)
  • 03:25 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Oct 12 03:25:17 UTC 2017 (duration 7m 17s)
  • 03:18 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.3) (duration: 15m 57s)
  • 02:41 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 15m 39s)
  • 00:30 twentyafterfour: phabricator update finished (a while ago, I forgot to log it)
  • 00:08 twentyafterfour: upgrading phabricator to #phab-2017-10-11
  • 00:05 mutante: netmon2001 arm keyholder for rancid deploy
  • 00:01 twentyafterfour: Preparing to take phabricator offline for a hopefully momentary upgrade.

2017-10-11

  • 23:55 mutante: netmon2001 - smokeping back up - the reboot, as a side-effect, also resolved the systemd-icinga-alert that was pending in Icinga for a while with comments
  • 23:52 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Vector print logo and print styles on all wikis T169732 PART II (duration: 00m 50s)
  • 23:50 thcipriani@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Enable Vector print logo and print styles on all wikis T169732 PART I (duration: 00m 50s)
  • 23:47 mutante: netmon2001 (smokeping) - kernel upgrade, schedule downtime, reboot
  • 23:40 thcipriani@tin: Synchronized php-1.31.0-wmf.2/skins/Vector/ResourceLoaderLessModule.php: SWAT: Print logo should use an absolute URI T177800 (duration: 00m 49s)
  • 23:38 mutante: netmon1003 - servermon back up - apt-get autoremove to drop unrequired python packages
  • 23:37 thcipriani@tin: Synchronized php-1.31.0-wmf.3/skins/Vector/ResourceLoaderLessModule.php: SWAT: Print logo should use an absolute URI T177800 (duration: 00m 50s)
  • 23:35 mutante: netmon1003 - cant seem to reboot normally, attempting via gnt-instance command
  • 23:31 mutante: netmon1003 (servermon) - upgrading kernel (jessie), schedule icinga downtime, rebooting, short downtime for servermon itself
  • 23:31 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Parser: Switch from Tidy to Remex on nowiki T177989 (duration: 00m 50s)
  • 23:27 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Parser: Switch from Tidy to Remex on fawiki T176150 (duration: 00m 50s)
  • 23:23 thcipriani@tin: Synchronized php-1.31.0-wmf.3/extensions/VisualEditor/modules/ve-mw/init/styles/ve.init.MWVESwitchConfirmDialog.css: SWAT: Fix WikiEditor mode switcher widget (duration: 00m 50s)
  • 23:19 thcipriani@tin: Synchronized php-1.31.0-wmf.2/includes/specials/SpecialUnblock.php: SWAT: Fix unblocking autoblocks T177952 (duration: 00m 50s)
  • 23:18 mutante: releases1001 upgrading kernel, installed linux-image-amd64, took out of cache::misc, upgrade libnss3, scheduled downtime, rebooting ...
  • 23:17 thcipriani@tin: Synchronized php-1.31.0-wmf.3/includes/specials/SpecialUnblock.php: SWAT: Fix unblocking autoblocks T177952 (duration: 00m 51s)
  • 22:01 volans: uploaded cumin_1.2.2-1_amd64.deb to apt.wikimedia.org jessie-wikimedia
  • 21:44 robh: cp4009 shutdown accidental, will boot back up immediately
  • 21:13 mutante: releases2001 - scheduled downtime, rebooting
  • 21:07 mutante: running puppet on cache misc hosts to temp remove codfw backend of releases.wm.org, then rebooting releases2001 for kernel upgrade
  • 20:22 subbu: updated Parsoid to ddf7b293 (T138492, T135667, T177612)
  • 20:18 ssastry@tin: Finished deploy [parsoid/deploy@938e305]: Updating Parsoid to ddf7b293 (duration: 08m 42s)
  • 20:17 hashar@tin: Finished deploy [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044 (duration: 02m 54s)
  • 20:14 hashar@tin: Started deploy [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044
  • 20:13 hashar@tin: Finished deploy [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044 (duration: 00m 02s)
  • 20:13 hashar@tin: Started deploy [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044
  • 20:11 hashar@tin: Started restart [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044
  • 20:10 bsitzmann@tin: Finished deploy [mobileapps/deploy@15a52ba]: Update mobileapps to c045afc (T177301) (duration: 05m 25s)
  • 20:09 ssastry@tin: Started deploy [parsoid/deploy@938e305]: Updating Parsoid to ddf7b293
  • 20:07 hashar@tin: Started restart [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044
  • 20:05 bsitzmann@tin: Started deploy [mobileapps/deploy@15a52ba]: Update mobileapps to c045afc (T177301)
  • 20:04 hashar@tin: Started restart [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044
  • 20:01 chasemp: disable puppet around cloud things to roll refactor out slowly
  • 19:29 mutante: releases1001 - same as 2001, upgraded jenkins to 2.73.2, kept existing config (T177962)
  • 19:27 mutante: releases2001 - upgraded jenkins to 2.73.2, kept existing config (vs overwriting with package config)
  • 19:20 mutante: apt: reprepro copy stretch-wikimedia jessie-wikimedia jenkins
  • 19:16 mutante: releases1001/releases2001 - apt-get autoremove to drop non-required python packages
  • 19:03 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.3
  • 19:02 demon@tin: Synchronized php: symlink swap (duration: 00m 49s)
  • 18:47 gehel: killing stuck osm replication on maps-test2001
  • 18:23 demon@tin: Synchronized wmf-config/Wikibase-labs.php: no-op, labs only (duration: 00m 59s)
  • 18:19 demon@tin: Finished scap: wmf.3 bootstrap (duration: 44m 56s)
  • 17:58 herron: ran /home/faidon/update-netboot-stretch.sh on puppetmaster1001
  • 17:49 moritzm: uploaded jenkins 2.73.2 for jessie-wikimedia to apt.wikimedia.org
  • 17:34 demon@tin: Started scap: wmf.3 bootstrap
  • 17:32 demon@tin: Synchronized php: symlink swap to wmf.2, somehow got missed (duration: 00m 45s)
  • 17:29 demon@tin: Synchronized scap/plugins/prep.py: No-op (duration: 01m 49s)
  • 17:09 demon@tin: Pruned MediaWiki: 1.30.0-wmf.17 (duration: 02m 20s)
  • 17:06 demon@tin: Pruned MediaWiki: 1.30.0-wmf.16 (duration: 02m 40s)
  • 16:54 demon@tin: Pruned MediaWiki: 1.31.0-wmf.1 [keeping static files] (duration: 01m 44s)
  • 16:24 hasharAway: Upgrade jenkins Maven integration plugin to 3.0 - T177962
  • 16:15 godog: roll-restart thumbor to apply https://gerrit.wikimedia.org/r/380942 - T150734
  • 16:12 hasharAway: Upgrading Jenkins CI T177962
  • 16:09 gehel: decommission maps-test2004 from its cassandra cluster (free a node to test vector tiles) - T153282
  • 16:09 XioNoX: resuming PDU upgrade
  • 15:39 elukey: drop tables listed in https://phabricator.wikimedia.org/T171629#3674250 from db1046, db1047, dbstore1002
  • 15:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1086 - T174509 (duration: 00m 47s)
  • 15:21 _joe_: stopped redis on ocg* T177931
  • 15:08 _joe_: ocg*: stop ocg; mv /srv/deployment /srv/stale, update-rc.d ocg disable, rm /etc/init/ocg.conf - T177931
  • 14:55 _joe_: manually removing IPVS entries for ocg on eqiad LBs, T177931
  • 14:47 _joe_: rolling restart of low-traffic pybals in eqiad for T177931
  • 14:36 moritzm: installing dbus update from stretch 9.2 point release
  • 13:40 hashar: European SWAT completed
  • 13:39 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Remove compact language links dblist for simplicity (duration: 00m 46s)
  • 13:38 hashar@tin: Synchronized docroot: Remove compact language links dblist for simplicity (duration: 00m 48s)
  • 13:37 hashar@tin: Synchronized dblists: Remove compact language links dblist for simplicity (duration: 00m 47s)
  • 13:36 hashar@tin: Synchronized wmf-config/CommonSettings.php: Remove compact language links dblist for simplicity (duration: 00m 47s)
  • 13:29 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Configure wmgBabelMainCategory for the Dinka Wikipedia (duration: 00m 47s)
  • 13:27 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable mapframe on eswiki - T177695 (duration: 00m 47s)
  • 13:24 hashar@tin: Synchronized wmf-config/Wikibase-production.php: Enable constraint checks on qualifiers and references - T176863 (duration: 00m 47s)
  • 13:21 moritzm: installing db5.3 security updates
  • 13:03 godog: test graphite-web patch on labmon1001 - T177747
  • 12:54 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Disable Statement usage tracking on cawiki (T151717) (duration: 00m 47s)
  • 12:53 marostegui: Start recentchanges purge on s4 primary master T177772
  • 12:48 marostegui: Kill recentchanges purge on s4 primary master - https://phabricator.wikimedia.org/T177772
  • 12:44 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: (temp) Disable Statement usage tracking on cawiki (T151717) (duration: 00m 48s)
  • 12:38 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable Statement usage tracking on cawiki and cewiki (T151717) (duration: 00m 47s)
  • 12:32 hoo@tin: Synchronized wmf-config/: Move WB client "disabledUsageAspects" setting into $wmgWikibaseDisabledUsageAspects (duration: 00m 48s)
  • 12:30 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Introduce $wmgWikibaseDisabledUsageAspects (duration: 00m 47s)
  • 12:27 hoo@tin: scap aborted: wmf-config/InitialiseSettings.php Introduce $wmgWikibaseDisabledUsageAspects (duration: 02m 34s)
  • 12:25 hoo@tin: Started scap: wmf-config/InitialiseSettings.php Introduce $wmgWikibaseDisabledUsageAspects
  • 11:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1087 - T174509 (duration: 00m 50s)
  • 11:49 ema: upgrading cp3007 to Varnish 5 T177233
  • 11:34 moritzm: installing ruby1.9 security updates on trusty hosts
  • 11:19 akosiaris: re-enable OTRS stats group T176221
  • 11:11 jynus: starting purge of ruwiki.recentchanges T177772
  • 10:55 akosiaris: OTRS upgrade done T176221
  • 10:44 akosiaris: starting OTRS upgrade T176221
  • 10:24 jynus: stopping dbstore2001:s3 for maintenance T177908
  • 10:19 moritzm: installing curl security updates
  • 09:32 moritzm: installing libxfont security updates
  • 09:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1087 - T174509 (duration: 00m 46s)
  • 09:27 marostegui: Optimize pagelinks and templatelinks on db1087 - T174509
  • 09:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1082 - T174509 (duration: 00m 47s)
  • 09:04 moritzm: installing curl security updates on app server canaries (along with HHVM restart)
  • 08:50 moritzm: installing ffmpeg security updates
  • 08:38 elukey: reboot kafka-jumbo hosts for kernel updates
  • 08:37 marostegui: Stop replication in sync on db1103 and db1038 to checksum their data - T164488
  • 08:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1103 - T164488 (duration: 00m 47s)
  • 07:46 jynus: restart commonswiki recentchanges purging T177772
  • 07:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1072 into the vslow and dump service for s3 - T172679 (duration: 00m 46s)
  • 07:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1086 - T174509 (duration: 00m 47s)
  • 07:02 marostegui: Optimize templatelinks and pagelinks on db1086 - T174509
  • 06:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1082 - T174509 (duration: 00m 47s)
  • 06:57 marostegui: Optimize templatelinks and pagelinks on db1082 - T174509
  • 06:14 marostegui: Set db1068 s4 primary master to read-only OFF - T168661
  • 06:14 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Set s4 to writable mode after maintenance - T168661 (duration: 00m 47s)
  • 06:07 marostegui: Deploy alter table on s4 primary master - T168661
  • 06:01 marostegui: Set read-only on db1068 s4 primary master - T168661
  • 06:00 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Set s4 in read-only for maintenance - T168661 (duration: 00m 47s)
  • 05:48 marostegui: Package update on db1068 s4 primary master - T168661
  • 05:35 marostegui: Disable puppet on db1068 (s4 primary master) - T168661
  • 05:35 marostegui: Disable puppet on db1068 (s4 primary master)
  • 05:13 marostegui: Disable alerts on s4 for 2 hours - T168661
  • 05:06 marostegui: Dump buffer pool on s4 primary master db1068 - T168661
  • 05:03 kartik@tin: Finished deploy [cxserver/deploy@a79b38c]: Update cxserver to 273b515 (duration: 02m 57s)
  • 05:00 kartik@tin: Started deploy [cxserver/deploy@a79b38c]: Update cxserver to 273b515
  • 02:32 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Oct 11 02:32:46 UTC 2017 (duration 6m 39s)
  • 02:26 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 08m 52s)

2017-10-10

  • 23:53 thcipriani@tin: Synchronized wmf-config: SWAT: Disable OCG services PART II T177795 (duration: 00m 48s)
  • 23:52 thcipriani@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Disable OCG services PART I T177795 (duration: 00m 47s)
  • 23:41 thcipriani@tin: Synchronized php-1.31.0-wmf.2/extensions/Collection/RenderingAPI.php: SWAT: Do not request render if renderer not configured T177795 (duration: 00m 47s)
  • 23:39 XioNoX: upgrading ps1-a2-eqiad - T175341
  • 23:35 thcipriani@tin: Synchronized wmf-config/CommonSettings.php: SWAT: revert Enable Vector print logo on test wiki (duration: 00m 47s)
  • 23:21 thcipriani@tin: scap failed: average error rate on 3/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details)
  • 23:12 thcipriani@tin: Synchronized wmf-config/CommonSettings.php: SWAT: wikitech: Align "contentadmin" and "sysop" permissions T171208 (duration: 00m 48s)
  • 23:08 eileen1: ran the group clean up script (reduced smart groups)
  • 22:28 eileen1: changed civicrm smart group cache timeout to 5 civicrm/admin/setting/search?reset=1
  • 21:15 bblack: esams repooling - T167840
  • 19:30 krinkle@tin: Synchronized docroot/noc/: Clean up noc base.css (duration: 00m 48s)
  • 19:07 krinkle@tin: Synchronized docroot/noc/: Clean up noc/index CSS (duration: 00m 47s)
  • 19:00 XioNoX: starting the work for T167840
  • 18:41 mutante: logstash*: delete /usr/lib/ganglia/python_modules/elasticsearch_monitoring.py and /etc/ganglia/conf.d/elasticsearch.pyconf via cumin (T177225)
  • 17:32 XioNoX: depooling esams from DNS for T167840
  • 17:26 gehel: shutting down and restarting elasticsearch on relforge1001 for testing - T170378
  • 17:25 krinkle@tin: Synchronized php-1.31.0-wmf.2/resources/lib/jquery.ui/: I717f2580e3aae (duration: 00m 47s)
  • 16:19 milimetric@tin: Finished deploy [analytics/refinery@f4a0a33]: Deploying mostly for the new script in the bin folder (duration: 16m 01s)
  • 16:03 milimetric@tin: Started deploy [analytics/refinery@f4a0a33]: Deploying mostly for the new script in the bin folder
  • 15:48 jynus: starting purge of commonswiki.recentchanges T177772
  • 15:39 thcipriani@tin: Synchronized README: noop deployment to test new logstash-checker.py host (duration: 00m 47s)
  • 15:27 jynus: upgrading neodymium and sarin to mariadb-client 10.1.28
  • 15:23 ema: cp1008 upgraded to varnish 5 T168529
  • 15:00 godog: test stretch 9.2 upgrade on ms-fe2005 - T177739
  • 14:59 cmjohnson1: lvs1007 swapping NIC card
  • 14:53 godog: test stretch 9.2 upgrade on ms-be2013 - T177739
  • 14:45 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1079 - T174509 (duration: 00m 47s)
  • 14:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1080 - T174509 (duration: 00m 47s)
  • 14:31 marostegui: Optimize table ores_classifications on db1080 - T159753
  • 14:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1072 with low weight - T172679 (duration: 00m 47s)
  • 14:24 aude@tin: Synchronized wmf-config/throttle.php: Add throttle rule T177835 (duration: 00m 47s)
  • 14:22 elukey: add druid public cluster's IPs to analytics-in4 on cr1/cr2 - T177511
  • 13:56 aude@tin: Synchronized wmf-config/throttle.php: Add throttle rule T177835 (duration: 02m 16s)
  • 13:37 aude@tin: Synchronized wmf-config/Wikibase-production.php: Remove test wiki echo configs from Wikibase config (duration: 01m 31s)
  • 13:20 aude@tin: Synchronized wmf-config/InitialiseSettings.php: Enable SandboxLink on gawiki (duration: 01m 50s)
  • 13:11 gehel: restarting rsyslog on mw1180 - T177833
  • 11:32 moritzm: downgrading HHVM on deployment-mediawiki04 to HHVM 3.18.2 temporarily (to further narrow down a problem with the new wikidiff package)
  • 11:31 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1076 - T174509 (duration: 05m 02s)
  • 11:31 dcausse: upgrading relforge to elastic 5.5.2
  • 11:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1071 - T174509 (duration: 01m 21s)
  • 10:33 akosiaris: T177511 switch druid100[456] to private1-x-eqiad VLANs
  • 10:26 Amir1: start of cleaning up ores_classification in wikidatawiki (T159753)
  • 08:07 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2080 to the config - T170662 (duration: 00m 46s)
  • 08:06 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2080 to the config - T170662 (duration: 00m 47s)
  • 07:04 marostegui: Stop MySQL on db2079 to clone db2080 - https://phabricator.wikimedia.org/T170662
  • 06:35 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s3 - T153033
  • 06:27 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s1 - T153033
  • 06:22 marostegui: Stop MySQL on db1038 to transfer its data to db1072 - T172679
  • 06:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1038 - T172679 (duration: 00m 47s)
  • 06:05 marostegui: Stop MySQL on db1072 to move it to s3 - T172679
  • 05:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1079 - T174509 (duration: 00m 47s)
  • 05:53 marostegui: Optimize templatelinks and pagelinks on db1079 - T174509
  • 05:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1071 - T174509 (duration: 00m 48s)
  • 05:45 marostegui: Optimize templatelinks and pagelinks on db1071 - T174509
  • 05:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1076 - T174509 (duration: 00m 47s)
  • 05:38 marostegui: Optimize templatelinks and pagelinks on db1076 - T174509
  • 05:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1080 - T174509 (duration: 00m 48s)
  • 05:32 marostegui: Optimize templatelinks and pagelinks on db1080 - T174509
  • 02:35 eileen2: update civicrm from 00da4b0 to 47de976
  • 02:32 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Oct 10 02:32:57 UTC 2017 (duration 6m 41s)
  • 02:26 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 08m 49s)
  • 01:21 eileen2: update civicrm from 276cbfd to 00da4b0

2017-10-09

  • 22:18 hoo: Removed empty left-over xmldatadumps/public/other/wikibase/wikidatawiki/20171007 (https://dumps.wikimedia.org/wikidatawiki/entities/20171007/)
  • 19:53 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable new print styles on Vector in test wiki - T169732 (duration: 00m 47s)
  • 19:51 hashar: morning swat completed
  • 19:50 hashar@tin: Synchronized php-1.31.0-wmf.2/extensions/Echo/maintenance/updatePerUserBlacklist.php: Sync a live hack in Echo updatePerUserBlacklist https://gerrit.wikimedia.org/r/#/c/383182/ - T173475 (duration: 00m 47s)
  • 19:48 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable new print styles on Vector in test wiki - T169732 (duration: 00m 47s)
  • 19:22 hashar@tin: Synchronized wmf-config/CommonSettings.php: REVERT: Enable new print styles on Vector - T169732 (duration: 00m 49s)
  • 19:19 hashar@tin: scap failed: average error rate on 8/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details)
  • 19:17 hashar@tin: Synchronized php-1.31.0-wmf.2/extensions/Echo: Reapply "Use User Ids instead of User Names for Echo Mute"" - T173475 (duration: 00m 52s)
  • 18:57 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable NewUserMessage for SUL accounts too on hi.wikiversity - T177690 (duration: 00m 47s)
  • 18:52 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable NewUserMessage for SUL accounts too on dty.wiki - T177688 (duration: 00m 47s)
  • 18:45 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Add autopatrolled user group to sd.wikipedia - T177141 (duration: 00m 47s)
  • 18:00 ottomata: stopping druid services on druid1004 (not 1006)
  • 17:59 ottomata: stopping druid services on druid1006
  • 17:28 ottomata: stopping druid services on druid1005
  • 16:34 ottomata: stopping druid services on druid1006
  • 16:34 ottomata: beginning decom and reinstall process for druid1004-1006 -- T176223
  • 15:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1091 - T174509 (duration: 00m 47s)
  • 13:53 zeljkof: EU SWAT finished
  • 13:53 zfilipin@tin: Synchronized php-1.31.0-wmf.2/extensions/Translate/: SWAT: Revert "[tech-debt] Remove usage of FuzzyLikeThis in favor of simple fuzzy match" (T177727) (duration: 00m 57s)
  • 13:27 zfilipin@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Disallow most file types from upload to enwikivoyage (T176647) (duration: 00m 47s)
  • 13:26 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Disallow most file types from upload to enwikivoyage (T176647) (duration: 00m 47s)
  • 13:13 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix amwikimedia site name (T176042) (duration: 00m 47s)
  • 13:10 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: New throttle rule (T177737) (duration: 00m 47s)
  • 12:47 ema: restart pybal on eqiad load balancers to pick up bgp-med config change T165584
  • 12:38 ema: restart pybal on codfw load balancers to pick up bgp-med config change T165584
  • 12:31 ema: restart pybal on ulsfo load balancers to pick up bgp-med config change T165584
  • 12:26 ema: restart pybal on esams load balancers to pick up bgp-med config change T165584
  • 11:59 joal@tin: Finished deploy [analytics/refinery@c3812c2]: Analytics regular weekly deploy (duration: 10m 24s)
  • 11:49 joal@tin: Started deploy [analytics/refinery@c3812c2]: Analytics regular weekly deploy
  • 11:42 hashar: Upgrading Jenkins - T168644
  • 11:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1083 - T174509 (duration: 00m 46s)
  • 11:22 moritzm: installing apt updates from stretch 9.2 point release
  • 11:22 marostegui: Optimize ores_classification table on db1083 - T159753
  • 11:19 reedy@tin: Synchronized wmf-config/Wikibase-production.php: Actually disable injecting RC everywhere but commonswiki and ruwiki (duration: 00m 46s)
  • 11:11 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1092 - T174509 (duration: 00m 47s)
  • 11:08 moritzm: installing whois updates from stretch 9.2 point release
  • 11:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1090 - T174509 (duration: 00m 46s)
  • 10:54 moritzm: installing expect updates from stretch 9.2 point release
  • 10:50 ladsgroup@tin: Synchronized wmf-config/Wikibase-production.php: Disable injecing RC records for commonswiki and ruwiki (T171027) (duration: 00m 47s)
  • 10:38 ladsgroup@tin: Synchronized php-1.31.0-wmf.2/extensions/Wikidata/extensions/Wikibase/client: Re-instate $wgWBClientSettings[injectRecentChanges] (T171027) (duration: 00m 56s)
  • 10:35 hoo: Started dumpwikidatajson.sh on snapshot1007 after T169680 disruptions
  • 10:30 volans: rebooting snapshot1007 for NFS stuck - T169680
  • 10:21 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s7 - T153033
  • 10:21 volans: rebooting snapshot1006 for NFS stuck - T169680
  • 10:15 elukey: rebooting snapshot1001 for NFS stuck - T169680
  • 10:00 volans: rebooting snapshot1005 for NFS stuck - T169680
  • 09:51 volans: rebooting dataset1001 for NFS stuck - T169680
  • 09:51 ema: test setting bgp med on lvs3001/3003 T165584
  • 09:50 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s5 - T153033
  • 09:43 hashar: restarting Jenkins. Deadlock in SSHSlave plugin that causes memory to leak quite rapidly - T177749
  • 09:34 moritzm: installing vim security updates
  • 09:02 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2075 - T170662 (duration: 00m 47s)
  • 08:56 volans: disabled puppet on 'stat100[5-6]*,snapshot100[1,5-7]*,dataset1001*' to manually recover from nfsd stuck - T169680
  • 08:52 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s4 - T153033
  • 08:51 marostegui: Optimize pagelinks and templatelinks on db1092 - T174509
  • 08:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1092 - T174509 (duration: 00m 47s)
  • 08:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1096 - T174509 (duration: 00m 47s)
  • 08:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1066 on s1 to and move db1072 to s3 instead - T172679 (duration: 00m 47s)
  • 08:12 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Move db1066 from s1 to s3 - T172679 (duration: 01m 25s)
  • 07:33 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s2 - T153033
  • 07:16 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2079 to the config - T170662 (duration: 00m 47s)
  • 07:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2079 to the config - T170662 (duration: 00m 47s)
  • 07:01 ema: cp4022 - backend restart, mailbox lag, cache_upload
  • 06:56 marostegui: Stop MySQL on db2075 to use it to clone db2079 - T170662
  • 06:56 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2075 - T170662 (duration: 00m 46s)
  • 06:25 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s6 - T153033
  • 06:14 marostegui: Optimize pagelinks and templatelinks on db1039 - T174509
  • 06:11 marostegui: Optimize pagelinks and templatelinks on db1050 - T174509
  • 06:08 marostegui: Optimize pagelinks and templatelinks on db1096 - T174509
  • 06:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1096 - T174509 (duration: 00m 47s)
  • 05:59 marostegui: Optimize pagelinks and templatelinks on db1091 - T174509
  • 05:59 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1091 - T174509 (duration: 00m 47s)
  • 05:53 marostegui: Optimize pagelinks and templatelinks on db1090 - T174509
  • 05:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1090 - T174509 (duration: 00m 47s)
  • 05:44 marostegui: Optimize pagelinks and templatelinks on db1083 - T174509
  • 05:38 marostegui: Stop MySQL on db1083 to upgrade it
  • 05:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1083 - T174509 (duration: 00m 47s)
  • 02:48 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Oct 9 02:48:21 UTC 2017 (duration 6m 54s)
  • 02:41 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 13m 49s)

2017-10-08

  • 21:40 hoo: Killed "dumpwikidatardf.sh truthy nt" (wikidata truthy dump) on snapshot1007, got stuck after T169680.
  • 11:28 volans: ack-ed puppet not running on stat100[5-6],snapshot100[1,5-7] due to NFSD stuck on dataset1001 - T169680
  • 08:19 elukey: restart varnish backend on cp4026 to stop 503s

2017-10-07

  • 22:46 reedy@tin: Synchronized php-1.31.0-wmf.2/resources/: T177705 (duration: 00m 48s)
  • 22:44 reedy@tin: Synchronized php-1.31.0-wmf.2/includes/specials/SpecialBlock.php: T177705 (duration: 00m 49s)

2017-10-06

  • 18:58 ejegg: updated CiviCRM from 197b22d to 276cbfd
  • 18:38 catrope@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/: Unbreak preference changes (T177524) (duration: 00m 47s)
  • 18:23 arlolra@tin: Finished deploy [parsoid/deploy@e437c93]: Updating Parsoid to 772e11bf (duration: 12m 41s)
  • 18:10 arlolra@tin: Started deploy [parsoid/deploy@e437c93]: Updating Parsoid to 772e11bf
  • 15:30 moritzm: repooling mw2145 (API server). it was previously depooled, but there's no tickets or SAL entries indicating hardware maintenance, so it was probably simply forgotten to repool it
  • 15:25 moritzm: repooling mw2203 (API server). it was previously depooled, but there's no tickets or SAL entries indicating hardware maintenance, so it was probably simply forgotten to repool it
  • 12:54 moritzm: repooling mw2248-mw2250 (job runners). they were previosly depooled, but there's no tickets or SAL entries indicating hardware maintenance, so they were probably simply forgotten to repool
  • 11:46 bblack: cp1* (eqiad caches) - upgrade to nginx-1.13.5-1+wmf1~jessie1 to match all the other sites
  • 11:34 moritzm: repooling mw2244/mw2245 (image scalers). they were previosly depooled, but there's no tickets or SAL entries indicating hardware maintenance, so they were probably simply forgotten to repool
  • 11:21 moritzm: installing gdk-pixbuf security updates
  • 11:10 elukey: rolling restart of all the druid daemons on druid100[1-6] to pick up new logging changes
  • 11:03 moritzm: installing git security updates on trusty (Debian already fixed)
  • 10:51 moritzm: upgrading HHVM on script runners to HHVM 3.18.5
  • 10:47 moritzm: upgrade remaining video scalers in codfw to HHVM 3.18.5
  • 10:43 elukey: restart replication (mysql+ eventlogging_sync) on dbstore1002 after mysql restart
  • 10:17 moritzm: upgrade remaining job runners in codfw to HHVM 3.18.5
  • 09:53 elukey: stop replication (eventlogging_sync + mysql) on dbstore1002 as prep step for mysql restart
  • 09:17 hashar: Restarted the CI Jenkins for plugin upgrades
  • 09:01 moritzm: upgrade remaining app servers in codfw to HHVM 3.18.5
  • 08:40 moritzm: upgrade remaining image scalers in codfw to HHVM 3.18.5
  • 08:09 ema: upgrade pybal to 1.14.0 on eqiad primary LVSs
  • 07:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1076 original weight - T174054 (duration: 00m 48s)
  • 07:23 ema: upgrade pybal to 1.14.0 on eqiad secondary LVSs
  • 07:18 moritzm: upgrade remaining API servers in codfw to HHVM 3.18.5
  • 07:11 legoktm: live hacking on mwdebug1002
  • 07:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1076 weight - T174054 (duration: 00m 47s)
  • 06:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1076 weight - T174054 (duration: 00m 47s)
  • 06:35 ema: set max_connections=0 on cp3032 varnish-be
  • 06:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1106 to the config - T172679 (duration: 00m 47s)
  • 06:21 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1106 to the config - T172679 (duration: 00m 47s)
  • 06:00 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1076 with low weight - T174054 (duration: 00m 46s)
  • 05:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1053 - T174509 (duration: 00m 48s)
  • 05:35 marostegui: Stop MySQL on db1076 for an upgrade

2017-10-05

  • 23:40 ejegg: updated CiviCRM from cca1a9d to 197b22d
  • 23:37 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/resources/lib/jquery/jquery.migrate.js: sync https://gerrit.wikimedia.org/r/#/c/382530 and https://gerrit.wikimedia.org/r/#/c/382529 (duration: 00m 47s)
  • 23:34 XioNoX: upgrading ps1-d2-eqiad.mgmt.eqiad.wmnet (unused)
  • 23:23 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch/includes/BaseInterwikiResolver.php: sync https://gerrit.wikimedia.org/r/#/c/382625/ (duration: 00m 47s)
  • 23:21 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT https://gerrit.wikimedia.org/r/#/c/382611/ (duration: 00m 47s)
  • 22:47 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch/includes/: Sync CirrusSearch extension again for good measure (duration: 00m 51s)
  • 22:43 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch: Sync CirrusSearch extension again for good measure (duration: 00m 57s)
  • 22:39 twentyafterfour@tin: Synchronized wmf-config/CirrusSearch-common.php: sync https://gerrit.wikimedia.org/r/#/c/382474/ (duration: 00m 47s)
  • 22:37 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch/: deploy https://gerrit.wikimedia.org/r/#/c/382618/ (duration: 00m 58s)
  • 22:13 awight@tin: Finished deploy [ores/deploy@42c5663]: Cause ORES service restart (duration: 00m 19s)
  • 22:13 awight@tin: Started deploy [ores/deploy@42c5663]: Cause ORES service restart
  • 22:04 bd808@tin: Finished deploy [striker/deploy@fbb9019]: Prevent tools from being named with invalid Kubernetes namespace labels (T176681) (duration: 00m 33s)
  • 22:04 bd808@tin: Started deploy [striker/deploy@fbb9019]: Prevent tools from being named with invalid Kubernetes namespace labels (T176681)
  • 21:35 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.31.0-wmf.2 refs T174358
  • 21:32 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch: sync CirrusSearch to deploy https://gerrit.wikimedia.org/r/#/c/382605/ refs T177535 (duration: 01m 06s)
  • 21:22 ejegg: updated CiviCRM from 59e3e00 to cca1a9d
  • 21:13 twentyafterfour: Getting the train back on track: moving to wmf.2 as soon as https://gerrit.wikimedia.org/r/#/c/382605/1 merges
  • 20:47 twentyafterfour: 1.31.0-wmf.2 is now blocked by T177535
  • 20:28 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.31.0-wmf.1 refs T174358
  • 20:26 twentyafterfour: rolling back to 1.31.0-wmf.1
  • 20:11 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.31.0-wmf.2 refs T174358
  • 20:10 twentyafterfour: Promote all wikis to 1.31.0-wmf.2 refs T174358 (currently there are no blockers and no significant logspam)
  • 19:59 thcipriani@tin: Synchronized php-1.31.0-wmf.1/extensions/Echo: SWAT: revert "Use User Ids instead of User Names for Echo Mute" (duration: 00m 55s)
  • 19:58 andrewbogott: upgrading wikitech-static to REL1_30
  • 19:55 thcipriani@tin: Synchronized php-1.31.0-wmf.2/extensions/Echo: SWAT: revert "Use User Ids instead of User Names for Echo Mute" (duration: 00m 52s)
  • 19:49 thcipriani@tin: Synchronized php-1.31.0-wmf.2/extensions/VisualEditor/extension.json: SWAT: Move 'parentheses' message to MWSaveDialog's modules T177446 (duration: 00m 50s)
  • 19:45 thcipriani@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict: SWAT: Dont register simulate page when not enabled as a BF (duration: 00m 52s)
  • 19:05 thcipriani@tin: Synchronized php-1.31.0-wmf.1/extensions/Echo: SWAT: Use User Ids instead of User Names for Echo Mute T173475 (duration: 00m 55s)
  • 18:58 thcipriani@tin: Synchronized php-1.31.0-wmf.2/extensions/Echo: SWAT: Use User Ids instead of User Names for Echo Mute T173475 (duration: 00m 56s)
  • 18:53 bblack: cp3* upgrade nginx to 1.13.5-1+wmf1~jessie1
  • 18:38 urandom: restbase-ng: lowering compactionthroughput to 4 (current/5 jbod devices)
  • 18:26 thcipriani@tin: Synchronized wmf-config: SWAT: Enable structured change filters by default on all wikis except FlaggedRevs wikis using non-protection modes T177445 (duration: 00m 54s)
  • 18:13 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable jQuery 3 on all wikis T124742 (duration: 00m 52s)
  • 17:27 cmjohnson1: disable puppet db1024
  • 16:53 moritzm: upgrade remaining app servers in eqiad to HHVM 3.18.5
  • 16:16 XioNoX: failure test for pfw3-codfw
  • 15:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1078 - T174054 (duration: 00m 50s)
  • 15:25 marostegui: Pulling out one disk from db1078 - T174054
  • 15:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1078 - T174054 (duration: 00m 51s)
  • 15:16 marostegui: Pulling out one disk from db1076 - T174054
  • 14:53 hashar: All Jenkins infra is now on Java 8 - T162828
  • 14:52 marostegui: Sanitize db1095 for amwikimedia - T176043
  • 14:48 hashar: Restarting Jenkins
  • 14:27 moritzm: upgrade remaining eqiad video scalers to HHVM 3.18.5
  • 13:57 ladsgroup@tin: Synchronized wmf-config/Wikibase-production.php: Enable Statement usage tracking on kowiki and trwiki (T151717) (duration: 00m 50s)
  • 13:51 ladsgroup@tin: Synchronized wmf-config/Wikibase-production.php: (no justification provided) (duration: 00m 50s)
  • 13:49 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Document task reference for amwikimedia logo (T176042) (duration: 00m 54s)
  • 13:43 Amir1: ladsgroup@terbium:~$ mwscript createAndPromote.php --wiki=amwikimedia --createAndPromote=sysop,bureaucrat 'Ladsgroup' "password removed obviously" --force (T176042)
  • 13:35 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Fix amwikimedia logo (T176042) (duration: 00m 50s)
  • 13:26 ladsgroup@tin: Synchronized php-1.31.0-wmf.2/extensions/WikimediaMessages/i18n/wikimediaprojectnames: (no justification provided) (duration: 00m 49s)
  • 13:15 godog: test upgrade node-exporter in thumbor/swift/services/puppet3-diffs/monitoring projects - T166561
  • 13:11 ladsgroup@tin: Synchronized wmf-config/interwiki.php: Create amwikimedia (T176042) (duration: 00m 51s)
  • 13:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1089 - T174509 (duration: 00m 50s)
  • 12:58 ladsgroup@tin: Synchronized multiversion/MWMultiVersion.php: Create amwikimedia (T176042) (duration: 00m 50s)
  • 12:53 ladsgroup@tin: Synchronized static/images/project-logos/: Create amwikimedia (T176042) (duration: 00m 50s)
  • 12:52 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Create amwikimedia (T176042) (duration: 00m 50s)
  • 12:50 ladsgroup@tin: rebuilt wikiversions.php and synchronized wikiversions files: Create amwikimedia (T176042)
  • 12:49 ladsgroup@tin: Synchronized dblists: Create amwikimedia (T176042) (duration: 00m 50s)
  • 12:36 ladsgroup@tin: Synchronized dblists: (no justification provided) (duration: 00m 51s)
  • 12:22 moritzm: installing poppler security updates on trusty
  • 11:57 moritzm: upgrade remaining eqiad video scalers to HHVM 3.18.5
  • 11:29 jynus: rebuilding globalimagelinks table on dbstore1001:s4
  • 10:35 hashar: contint2001: switched java to version 8 as well as all the jre/jdk utilities managed by alternatives - T162828
  • 10:26 moritzm: upgrade remaining eqiad API servers to HHVM 3.18.5
  • 10:23 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/includes/TwoColConflictHooks.php: REVERT: Dont register simulate page when not enabled as a BF PT1/3 (duration: 00m 50s)
  • 10:22 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/extension.json: REVERT: Dont register simulate page when not enabled as a BF PT2/3 (duration: 00m 50s)
  • 10:17 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/extension.json: Dont register simulate page when not enabled as a BF PT2/3 (duration: 00m 50s)
  • 10:16 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/includes/TwoColConflictHooks.php: Dont register simulate page when not enabled as a BF PT1/3 (duration: 00m 51s)
  • 10:14 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/includes/SpecialConflictTestPage/SpecialConflictTestPage.php: Fix SimulateTwoColEditConflict when using a page with ns (duration: 00m 50s)
  • 10:09 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/WikimediaEvents/WikimediaEventsHooks.php: 31-wmf.2 WikimediaEvents: WMDE: update $campaignStartTimestamp (duration: 00m 50s)
  • 10:07 addshore@tin: Synchronized php-1.31.0-wmf.1/extensions/WikimediaEvents/WikimediaEventsHooks.php: 31-wmf.1 WikimediaEvents: WMDE: update $campaignStartTimestamp (duration: 00m 52s)
  • 10:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1076 - T174054 (duration: 00m 50s)
  • 09:52 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1099 - T174509 (duration: 00m 50s)
  • 08:54 marostegui: Stop MySQL on db1104 to clone db1106 - T172679
  • 08:53 gehel: rolling restart of wdqs to pick up new GC options - T175919
  • 08:41 moritzm: upgrade remaining eqiad job runners to HHVM 3.18.5
  • 08:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1105 to the config - T172679 (duration: 00m 50s)
  • 08:39 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1105 to the config - T172679 (duration: 00m 52s)
  • 08:29 marostegui: Optimize pagelinks and templatelinks on s5 - db1105 - T174509
  • 08:21 moritzm: upgrade remaining eqiad image scalers to HHVM 3.18.5
  • 08:12 marostegui: Optimize templatelinks and pagelinks on s1 s2 s4 s5 s6 and s7 on labsdb1009 - T174509
  • 08:03 jynus: putting labsdb1009 under maintenance
  • 07:54 moritzm: upgrade mw1221-mw1235 (API servers) to HHVM 3.18.5
  • 07:26 legoktm: running batched query of "DELETE FROM linter WHERE linter_cat=12" on all wikis to clear out mostly bogus html5-misnesting category
  • 07:11 ema: update pybal to 1.14.0 on esams primary LVSs
  • 07:06 ema: update pybal to 1.14.0 on esams secondary LVSs
  • 07:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1099 - T174509 (duration: 00m 52s)
  • 07:01 marostegui: Optimize templatelinks and pagelinks on db1099 - T174509
  • 06:56 moritzm: upgrade mw1238-mw1258 (app servers) to HHVM 3.18.5
  • 06:40 marostegui: Stop MySQL on db1104 to clone db1105 from it - T172679
  • 06:32 marostegui: Optimize templatelinks and pagelinks on db1069 - T174509
  • 06:31 marostegui: Optimize templatelinks and pagelinks on db1053 - T174509
  • 06:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1053 - T174509 (duration: 00m 50s)
  • 06:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1089 - T174509 (duration: 00m 49s)
  • 06:24 marostegui: Optimize templatelinks and pagelinks on db1089 - T174509
  • 06:17 marostegui: Optimize s2.templatelinks and enwiki.pagelinks on codfw master db2017 (without replication) - T174509
  • 06:13 marostegui: Optimize enwiki.templatelinks and enwiki.pagelinks on codfw master db2048 (without replication) - T174509
  • 06:01 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2035 and db2056 - T174509 (duration: 00m 52s)
  • 05:57 elukey: restart varnish backend on cp3040
  • 05:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1097 - T174509 (duration: 01m 09s)
  • 05:42 elukey: restart varnish backend on cp3030
  • 02:59 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Oct 5 02:59:18 UTC 2017 (duration 7m 23s)
  • 02:51 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 09m 01s)
  • 02:29 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 09m 51s)
  • 00:06 catrope@tin: Synchronized php-1.31.0-wmf.2/resources/src/mediawiki.rcfilters/mw.rcfilters.init.js: T176652 (duration: 00m 49s)
  • 00:04 catrope@tin: Synchronized php-1.31.0-wmf.2/extensions/WikimediaEvents/modules/ext.wikimediaEvents.recentChangesClicks.js: T176652 (duration: 00m 50s)

2017-10-04

  • 23:55 eileen: update process_control to 10161f8
  • 23:53 catrope@tin: Synchronized php-1.31.0-wmf.2/extensions/TimedMediaHandler/MwEmbedModules/EmbedPlayer/resources/jquery.embedPlayer.js: Fix jqmigrate warning (T169385) (duration: 00m 50s)
  • 23:51 catrope@tin: Synchronized php-1.31.0-wmf.2/extensions/ORES/includes/Hooks.php: ORES highlights for RCFilters (T172757) (duration: 00m 50s)
  • 23:49 catrope@tin: Synchronized php-1.31.0-wmf.2/resources/src/: ORES highlights for RCFilters (T172757) (duration: 00m 50s)
  • 23:36 catrope@tin: Synchronized php-1.31.0-wmf.2/includes/changes/ChangesListFilter.php: ORES highlights for RCFilters (T172757) (duration: 00m 50s)
  • 23:32 catrope@tin: Synchronized php-1.31.0-wmf.2/includes/changes/ChangesListFilterGroup.php: Fix PHP fatal when transcluding Special:Recentchanges (T176236) (duration: 00m 50s)
  • 23:32 Krinkle: Upgrading xhgui on tungsten (operations/software/xhgui.git) from 0.5.2 to 0.8.1 (matching version used in operations/mediawiki-config vendor) - new feature: flamegraphs
  • 23:29 catrope@tin: Synchronized wmf-config/throttle.php: Remove expired rules (duration: 00m 50s)
  • 23:27 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable AbuseFilter runtime profile on ptwiki (T177336) (duration: 00m 51s)
  • 23:19 catrope@tin: Synchronized static/images/mobile/copyright/wikipedia-wordmark-en.svg: Remove unnecessary id attributes (T175670) (duration: 00m 50s)
  • 23:18 catrope@tin: Synchronized php-1.31.0-wmf.2/skins/MinervaNeue/: Correct feature phone threshold detection (T176286) (duration: 00m 53s)
  • 22:35 ejegg: changed DonationInterface cURL timeout for all processors to 12 sec
  • 21:09 arlolra: Updated Parsoid to 477942c3 (T177115)
  • 21:07 hoo: Updated the Wikidata property suggester with data from Monday's JSON dump and applied the T132839 workarounds
  • 21:03 arlolra@tin: Finished deploy [parsoid/deploy@617cdb3]: Updating Parsoid to 477942c3 (duration: 10m 01s)
  • 20:53 arlolra@tin: Started deploy [parsoid/deploy@617cdb3]: Updating Parsoid to 477942c3
  • 20:33 arlolra@tin: Finished deploy [parsoid/deploy@5b80254]: Updating Parsoid to 477942c3 (duration: 00m 58s)
  • 20:32 arlolra@tin: Started deploy [parsoid/deploy@5b80254]: Updating Parsoid to 477942c3
  • 20:18 pnorman: Start regenerating map tiles on codfw for z13-z14 - T176252
  • 19:55 twentyafterfour@tin: Synchronized php: group1 wikis to 1.31.0-wmf.2 refs T174358 (duration: 00m 48s)
  • 19:55 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.31.0-wmf.2 refs T174358
  • 19:38 pnorman: Start regenerating map tiles on eqiad for z13-z14 - T176252
  • 19:25 twentyafterfour: Deploying MediaWiki 1.31.0-wmf.2 to Group 1 wikis refs T174358
  • 19:06 bblack: cp2* (all codfw caches): upgrade nginx to 1.13.5-1+wmf1~jessie1
  • 18:39 ottomata: reverting roll out of LVS service druid-analytics.svc.eqiad.wmnet - this wont' work with hosts inside of Analytics VLAN
  • 18:23 niharika29@tin: Synchronized php-1.31.0-wmf.2/skins/Vector/: Do not special-case ULS and Not logged in in RTL in personal bar T48947, T177312 (duration: 00m 50s)
  • 18:22 niharika29@tin: Synchronized php-1.31.0-wmf.1/skins/Vector/: Do not special-case ULS and Not logged in in RTL in personal bar T48947, T177312 (duration: 00m 51s)
  • 18:09 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable jQuery 3 on most group1 wikis (T124742) (duration: 00m 51s)
  • 17:37 elukey: enabled basic ACLs on the Kafka Jumbo cluster - T173493
  • 17:32 bblack: text@ulsfo - upgrade nginx to 1.13.5-1+wmf1~jessie1
  • 17:32 ottomata: deploying new LVS service for druid-analytics-broker
  • 17:16 moritzm: uploaded icu52 to stretch-wikimedia (co-installable forward port of jessie's ICU for stretch to be used for HHVM)
  • 17:06 bblack: cp4022-26 (now all of upload@ulsfo) - upgrade nginx to 1.13.5-1+wmf1~jessie1
  • 16:58 krinkle@tin: Synchronized php-1.31.0-wmf.2/extensions/TimedMediaHandler/resources/mw.MediaWikiPlayer.loader.js: T175506 (duration: 00m 52s)
  • 16:29 bblack: lvs1001-3: base package upgrades, autoremove, +python-requests
  • 16:24 bblack: recdns: upgrade base packages
  • 16:23 volans: deleted 'R:File = /usr/local/bin/wmf-reimage' on matching hosts T176955
  • 16:20 bblack: maelant: upgrade base packages
  • 16:11 bblack: lvs1004-6: base package upgrades, autoremove, +python-requests
  • 16:06 jynus: performing schema change on labsdb1009/10/11
  • 16:01 ejegg: disabled CiviCRM dedupe jobs
  • 15:40 bblack: acamar (recdns@codfw) - base package upgrades
  • 15:39 bblack: lvs1007-12: base package upgrades, autoremove, +python-requests
  • 15:37 bblack: hydrogen (recdns@eqiad) - base package upgrades
  • 15:35 marostegui: Power off db2010 to decommission it - T175685
  • 15:35 bblack: recdns: apt autoremove -> install python-requests
  • 15:32 bblack: caches: apt autoremove -> install python-requests
  • 15:28 marostegui: Disable puppet on db2010 - it will be decommissioned - T175685
  • 15:24 andrewbogott: rebuilding labvirt1016
  • 14:57 ema: cp4025: restart varnish backend
  • 14:47 XioNoX: replacing faulty VC cable on fasw-c-eqiad
  • 14:45 bblack: base package upgrades (+autoremove) on lvs3001-2
  • 14:43 bblack: base package upgrades (+autoremove) on lvs3004
  • 14:41 bblack: base package upgrades (+autoremove) on lvs3003
  • 14:24 bblack: upgrading nginx to 1.13.5-1+wmf1~jessie1 on cp4021 (first live cache, upload@ulsfo)
  • 14:18 bblack: base package upgrades (+autoremove) on lvs2*
  • 14:10 bblack: base package upgrades (+autoremove) on lvs4*
  • 13:47 zeljkof: EU SWAT finished
  • 13:46 zfilipin@tin: Synchronized php-1.31.0-wmf.2/extensions/Translate/tag/PageTranslationHooks.php: SWAT: Revert "Use Language type hint in hooks" (T177352) (duration: 00m 50s)
  • 13:45 zfilipin@tin: Synchronized php-1.31.0-wmf.2/extensions/Translate/TranslateHooks.php: SWAT: Revert "Use Language type hint in hooks" (T177352) (duration: 00m 50s)
  • 13:41 zfilipin@tin: Synchronized php-1.31.0-wmf.2/extensions/Translate/TranslateHooks.php: php-1.31.0-wmf.2/extensions/Translate/tag/PageTranslationHooks.php SWAT: Revert "Use Language type hint in hooks" (T177352) (duration: 00m 52s)
  • 13:26 zeljkof: one more commit for EU SWAT
  • 13:26 marostegui: Optimize table ores_classification on enwiki codfw master db2048 with replication - might generate lag - T159753
  • 13:22 marostegui: Optimize enwiki.ores_classification on db2069
  • 13:06 zeljkof: EU SWAT finished
  • 13:05 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: New throttle rule (T177370) (duration: 00m 53s)
  • 12:59 jynus: upgrade and reboot labsdb1009
  • 12:56 marostegui: Optimize templatelinks and pagelinks tables on s2 and s7 - labsdb1010 - T174509
  • 12:38 moritzm: upgrading app servers in codfw to HHVM 3.18.5
  • 12:21 moritzm: upgrading image scalers mw1293-mw1295 to HHVM 3.18.5
  • 12:12 moritzm: upgrading HHVM on deployment servers to 3.18.5
  • 12:10 elukey: added two new mediawiki videoscalers - mw1307/1318
  • 12:09 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1318.eqiad.wmnet
  • 12:09 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1307.eqiad.wmnet
  • 12:00 gehel: mediawiki now uses the LVS endpoint for logstash - T175242
  • 11:49 moritzm: upgrading job runners mw1161-mw1167 to HHVM 3.18.5
  • 11:34 moritzm: installing ghostscript security updates
  • 11:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1087 original weight (duration: 00m 47s)
  • 10:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1087 weight (duration: 00m 51s)
  • 10:29 jynus: upgrade and reboot labsdb1011
  • 10:22 jynus: draining labsdb1011 connections
  • 10:18 akosiaris: T177374, fully depool wtp1001-wtp1024
  • 10:18 akosiaris@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=wtp10([01][0-9]|2[0-4]).eqiad.wmnet
  • 10:03 moritzm: install libidn security updates
  • 10:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1087 with low weight (duration: 00m 50s)
  • 09:54 _joe_: restarting hhvm on canaries (both appservers and canaries) T107128
  • 09:47 moritzm: upgrading API app servers mw1189-mw1208 to HHVM 3.18.5
  • 09:45 elukey: rolling restart of aqs nodes to pick up the new logstash lvs config
  • 09:39 hoo: Killed remaining dumpRdf runners on snapshot1007 after two crashed due to the db1087 depool. Auto-restart will do the rest.
  • 09:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1104 to the config - T172679 (duration: 00m 50s)
  • 09:32 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1104 to the config - T172679 (duration: 00m 51s)
  • 09:25 moritzm: installing ocaml security updates on trusty
  • 09:16 marostegui: Reboot db1087 to pick up new kernel
  • 09:09 _joe_: restarting hhvm on mwdebug*, T107128
  • 09:07 marostegui: Optimize templatelinks and pagelinks tables on db1097 (s4) - T174509
  • 09:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1097 - T174509 (duration: 00m 50s)
  • 08:41 akosiaris@puppetmaster1001: conftool action : set/pooled=no; selector: name=wtp10([01][0-9]|2[0-4]).eqiad.wmnet
  • 08:40 mobrovac@tin: Finished deploy [restbase/deploy@8eb758a]: Switch the mobile feeds to the new storage schema and Cassandra 3 (duration: 10m 05s)
  • 08:36 moritzm: upgrading app servers mw1180-mw1188, mw1209-mw1220 to HHVM 3.18.5
  • 08:30 mobrovac@tin: Started deploy [restbase/deploy@8eb758a]: Switch the mobile feeds to the new storage schema and Cassandra 3
  • 08:28 marostegui: Stop MySQL on db2010 as it will be decommissioned - T175685
  • 07:30 jynus: stopping s5 replication on dbstore1002 and converting wikidatawiki.wb_terms into TokuDB
  • 07:11 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db2010 from config as it will be decommissioned - T175685 (duration: 00m 48s)
  • 07:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db2010 from config as it will be decommissioned - T175685 (duration: 00m 48s)
  • 06:53 marostegui: Stop MySQL on db1087 to clone db1104 from it - T172679
  • 06:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1087 - T172679 (duration: 00m 50s)
  • 06:30 marostegui: Optimize templatelinks and pagelinks tables on db1066 - T174509
  • 06:27 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2035 and db2056 - T174509 (duration: 00m 50s)
  • 06:21 elukey: restart varnish backend on cp3043
  • 06:19 marostegui: Optimize pagelinks and templatelinks on dbstore2002 s1 - T174509
  • 06:16 elukey: restart varnish backend on cp3041
  • 06:13 marostegui: Optimize pagelinks and templatelinks tables on s7 codfw master (db2029) this will generate lag - T174509
  • 05:58 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2034, db2055 db2062 - T174509 (duration: 00m 51s)
  • 03:09 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Oct 4 03:09:34 UTC 2017 (duration 7m 15s)
  • 03:02 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 15m 20s)
  • 02:25 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 07m 44s)
  • 00:33 ejegg: re-enabled fundraising queue consumers
  • 00:29 ejegg: re-enabled CiviCRM de-dupe jobs
  • 00:24 ejegg: re-enabled civicrm cron and ingenico jobs
  • 00:20 ejegg: re-enabled fundraising stats and export jobs
  • 00:16 ejegg: re-enabled omnimail import jobs
  • 00:10 ejegg: re-enabled fundraising audit jobs
  • 00:07 ejegg: re-enabled pending queue consumer

2017-10-03

  • 23:52 ejegg: stopped all jobs for process-control upgrade
  • 23:46 pnorman: Start regenerating map tiles on eqiad for z0-z12 - T176252
  • 23:20 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T175971 T176088 Enable RemexHTML on wikitech and eswikiversity (duration: 00m 51s)
  • 23:11 ebernhardson@tin: Synchronized wmf-config/CirrusSearch-common.php: Pre-configure cirrussearch for 5.5.x upgrade (duration: 00m 51s)
  • 22:26 bsitzmann@tin: Finished deploy [mobileapps/deploy@82aa7d6]: Update mobileapps to 5dc0c02 (T175762 T177001 T176525 T176517 T176519) (duration: 06m 14s)
  • 22:20 bsitzmann@tin: Started deploy [mobileapps/deploy@82aa7d6]: Update mobileapps to 5dc0c02 (T175762 T177001 T176525 T176517 T176519)
  • 20:29 pnorman: Start regenerating map tiles on codfw for z0-z12 - T176252
  • 20:17 demon@tin: Synchronized php-1.31.0-wmf.2/extensions/GlobalUserPage/includes/Hooks.php: fix broken thing (duration: 00m 51s)
  • 19:58 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.2
  • 19:33 demon@tin: Synchronized scap/plugins/prep.py: no-op (duration: 00m 51s)
  • 19:17 demon@tin: Finished scap: bootstrap wmf.2 (duration: 33m 50s)
  • 19:01 twentyafterfour: Starting MediaWiki train for 1.30.1-wmf.2
  • 18:43 demon@tin: Started scap: bootstrap wmf.2
  • 18:25 chasemp: change to private vlan for labweb1001 on asw-b-eqiad
  • 18:24 chasemp: change to private vlan for labweb1002 on asw2-d-eqiad
  • 18:24 chasemp: auth-dnsupdate on bahan for 382027
  • 16:51 ejegg: updated payments-wiki from c2a3d43 to b9859f6
  • 15:57 bblack: upgrading basic packages on role::cache::text
  • 15:50 bblack: upgrading basic packages on role::cache::upload
  • 15:42 bblack: upgrading basic packages on role::cache::misc
  • 15:29 mobrovac@tin: Finished deploy [restbase/deploy@02acc45]: (no justification provided) (duration: 06m 27s)
  • 15:24 _joe_: uploading blubber to jessie-wikimedia T175296
  • 15:23 mobrovac@tin: Started deploy [restbase/deploy@02acc45]: (no justification provided)
  • 15:04 mobrovac@tin: Finished deploy [restbase/deploy@02acc45]: (no justification provided) (duration: 05m 14s)
  • 14:59 mobrovac@tin: Started deploy [restbase/deploy@02acc45]: (no justification provided)
  • 14:56 mobrovac@tin: Finished deploy [restbase/deploy@02acc45]: (no justification provided) (duration: 04m 25s)
  • 14:56 bblack: upgrading basic packages on cp1065
  • 14:52 mobrovac@tin: Started deploy [restbase/deploy@02acc45]: (no justification provided)
  • 13:56 zeljkof: EU SWAT finished
  • 13:55 zfilipin@tin: Synchronized php-1.31.0-wmf.1/extensions/Translate/webservices/CxserverWebService.php: SWAT: Make CxserverWebService forward compatible (duration: 00m 45s)
  • 13:46 godog: upgrade grafana to 4.5.2 on krypton - T175980
  • 13:41 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Adjust wgNamespacesToBeSearchedDefault for enwikibooks, fawikibooks and hewikisource (T176906 T176908 T176907) (duration: 00m 46s)
  • 13:34 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Limit thanks for new users at pl.wikipedia to 3 per day" (T176174 T169268) (duration: 00m 46s)
  • 13:26 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ShortUrl Extension on hiwikiversity (T177187) (duration: 00m 46s)
  • 13:25 marostegui: Optimize templatelinks and pagelinks tables on s1, s4 and s6 on labsdb1010 - T174509
  • 13:13 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable NewUserMessage Extension on hiwikiversity (T177188) (duration: 00m 46s)
  • 13:04 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2044 - T174764 (duration: 00m 47s)
  • 10:50 mobrovac@tin: Finished deploy [restbase/deploy@1cc530b]: (no justification provided) (duration: 08m 38s)
  • 10:41 mobrovac@tin: Started deploy [restbase/deploy@1cc530b]: (no justification provided)
  • 08:10 Amir1: cleaning up ores_classification tables (T159753)
  • 08:05 marostegui: Deplo alter table on s3 codfw master (db2018) to add PK to the main tables - T163912
  • 08:02 mobrovac@tin: Finished deploy [restbase/deploy@65f519b]: Create new buckets for feed end points (duration: 09m 57s)
  • 07:52 mobrovac@tin: Started deploy [restbase/deploy@65f519b]: Create new buckets for feed end points
  • 07:20 gehel: killing stuck tileshell and cleaning up /srv/osm_expire on maps-test2001 - T175123
  • 07:05 elukey: restart varnish backend on cp3031 (503s)
  • 07:03 marostegui: Optimize pagelinks and templatelinks tables on labsdb1010 for s5 - T174509
  • 07:01 marostegui: Optimize templatelinks and pagelinks tables on s6 codfw master db2028 (this might generate lag) - T174509
  • 06:46 marostegui: Drop now redundant indexes from pagelinks and templatelinks on s7 - T174509
  • 06:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1066 - T172679 (duration: 00m 47s)
  • 05:54 marostegui: Optimize templatelinks and pagelinks tables on s5 master codfw (db2023): this will generate lag - T174509
  • 05:51 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2049 and db2041 - T174509 (duration: 00m 47s)
  • 03:23 demon@tin: Pruned MediaWiki: 1.30.0-wmf.19 [keeping static files] (duration: 01m 30s)
  • 02:30 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Oct 3 02:30:30 UTC 2017 (duration 6m 39s)
  • 02:23 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 07m 22s)

2017-10-02

  • 23:38 catrope@tin: Synchronized php-1.31.0-wmf.1/extensions/VisualEditor/: T177250 (duration: 00m 47s)
  • 23:36 catrope@tin: Synchronized php-1.31.0-wmf.1/resources/src/mediawiki.rcfilters/dm/mw.rcfilters.dm.FiltersViewModel.js: T177107 (duration: 00m 47s)
  • 23:33 mutante: releases2001 - kill and restart jenkins with systemctl
  • 23:19 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Update WMF mailing address (T173684) and enable jQuery 3 on Wiktionaries (T124742) (duration: 00m 48s)
  • 23:00 mutante: releases1001 - remove and reinstall jenkins
  • 22:56 Reedy: created newsletter tables on officewiki T176199
  • 19:36 ejegg: turned on CiviMail record creation
  • 18:48 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Change the Turkish Wiktionary logo (T176008) (duration: 00m 46s)
  • 18:46 niharika29@tin: Synchronized static/images/project-logos/: Change the Turkish Wiktionary logo (T176008) (duration: 00m 48s)
  • 17:46 gehel@tin: Finished deploy [wdqs/wdqs@f20b726]: latest wdqs GUI + minor fixes (duration: 01m 48s)
  • 17:44 gehel@tin: Started deploy [wdqs/wdqs@f20b726]: latest wdqs GUI + minor fixes
  • 14:08 zeljkof: EU SWAT finished
  • 14:07 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Sandbox menu to Javanese Wikipedia (T176308) (duration: 00m 46s)
  • 14:01 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Extension:Newsletter on officewiki (T176199) (duration: 00m 46s)
  • 13:59 zeljkof: extending EU SWAT for 5-10 minutes
  • 13:51 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Change the zh-classicalwiki logo (T177165) (duration: 00m 45s)
  • 13:50 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Change the zh-classicalwiki logo (T177165) (duration: 00m 46s)
  • 13:44 zfilipin@tin: Synchronized static/images/project-logos/zh_classicalwiki-1.5x.png: static/images/project-logos/zh_classicalwiki-2x.png static/images/project-logos/zh_classicalwiki.png wmf-config/InitialiseSettings.php SWAT: Change the zh-classicalwiki logo (T177165) (duration: 00m 46s)
  • 13:42 godog: roll restart thumbor to apply https://gerrit.wikimedia.org/r/#/c/381045/ - T166699
  • 13:38 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable the Extension:SandboxLink on the nlwikinews (T177170) (duration: 00m 46s)
  • 13:32 zfilipin@tin: Synchronized wmf-config/Wikibase-production.php: SWAT: Wikidata: Dont persist description usages (yet) (T177153) (duration: 00m 45s)
  • 13:27 zfilipin@tin: Synchronized wmf-config/Wikibase-production.php: SWAT: Wikidata: Dont persist description usages (yet) (T177153) (duration: 00m 46s)
  • 13:20 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add eliminator as a priviliged account (T176554) (duration: 00m 46s)
  • 13:17 zfilipin@tin: Synchronized wmf-config/Wikibase-labs.php: wmf-config/Wikibase.php wmf-config/Wikibase-production.php SWAT: labs: Use redis lock manager for dispatching changes of Wikibase (T175109) (duration: 00m 46s)
  • 13:09 zfilipin@tin: Synchronized wmf-config/Wikibase-labs.php: wmf-config/Wikibase.php wmf-config/Wikibase-production.php SWAT: labs: Use redis lock manager for dispatching changes of Wikibase (T175109) (duration: 00m 47s)
  • 12:56 elukey: added two new mediawiki jobrunners - mw131[01]
  • 12:50 marostegui: Optimize tables pagelinks and templatelinks on s5: dbstore2001
  • 11:51 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2039 - T174509 (duration: 00m 46s)
  • 10:06 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1311.eqiad.wmnet
  • 10:06 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1310.eqiad.wmnet
  • 10:04 mobrovac: restbase restarting on restbase1007 to start logging error bodies for T176728
  • 09:55 jynus: stopping and upgrading labsdb1010
  • 09:27 Amir1: finished the cleanup of ores_classification table in enwiki (T159753)
  • 09:12 godog: roll-restart thumbor to upgrade to 1.6 - T172556
  • 09:06 elukey: forced remount of /mnt/hdfs on notebook1002
  • 08:47 gehel: deploying latest tilerator on maps-test only, fix for T175123
  • 08:46 gehel@tin: Finished deploy [tilerator/deploy@b02bd9a]: (no justification provided) (duration: 00m 22s)
  • 08:46 gehel@tin: Started deploy [tilerator/deploy@b02bd9a]: (no justification provided)
  • 08:46 marostegui: Poweroff db2044 for HW maintenance - T174764
  • 08:40 marostegui: Stop MySQL on db2044 to get it ready to replace its mainboard - T174764
  • 08:29 Amir1: finished the cleanup of ores_classification table in wikidatawiki and starting the enwiki one (T159753)
  • 08:24 jynus: stopping dbstore1001 for maintenance
  • 08:12 marostegui: Optimize tables pagelinks and templatelinks on s5: db1100 (already depooled) - T174509
  • 07:59 marostegui: Drop now redundant indexes from pagelinks and templatelinks from s5 - T174509
  • 07:57 Amir1: starting a round of cleanup in ores_classification table in wikidatawiki
  • 07:41 marostegui: Stop MySQL on db1036 as it is going to be decommissioned - https://phabricator.wikimedia.org/T176311
  • 07:32 marostegui: Optimize table pagelinks and templatelinks on s6: dbstore2001 - T174509
  • 07:21 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2039 - T174509 (duration: 00m 46s)
  • 07:20 marostegui: Optimize table pagelinks and templatelinks on s6: db2039 - T174509
  • 07:10 marostegui: Optimize table pagelinks and templatelinks on s1: db2034, db2055 db2062 - T174509
  • 07:06 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2034, db2055 db2062 - T174509 (duration: 00m 46s)
  • 06:55 _joe_: restarting a few API appservers with high cpu load
  • 06:19 elukey: restart varnish backend on cp3040
  • 06:10 elukey: restart varnish backend on cp3033
  • 06:08 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1036 from config files as it will be decommissioned - T176311 (duration: 00m 46s)
  • 06:07 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1036 from config files as it will be decommissioned - T176311 (duration: 00m 48s)
  • 02:43 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Oct 2 02:43:33 UTC 2017 (duration 6m 39s)
  • 02:36 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 08m 42s)

2017-10-01

  • 21:30 madhuvishy: powercycling labvirt1015
  • 18:58 hashar: restarted Jenkins. Was stuck somehow
  • 13:50 elukey: restart hhvm on mw1167 (jobrunner) - hhvm stuck, dump-debug in /tmp/hhvm.9624.bt.

2017-09-30

  • 20:54 catrope@tin: Synchronized wmf-config/CommonSettings.php: Bump $wgResourceLoaderStorageVersion (T176884) (duration: 00m 47s)
  • 00:37 ejegg: increased PayPal Express Checkout cURL timeout to 12 seconds

2017-09-29

  • 22:12 ejegg: updated CiviCRM from 784756b to 59e3e00
  • 21:19 mutante: labstore2003/2004: closed idle screen sessions
  • 20:27 mutante: mwlog1001, oxygen - closing unused screen session
  • 20:20 mutante: install1002/install2002 - closing unused screen sessions (except a tail -f)
  • 20:11 mutante: restbase2001 - closing screen sessions that weren't running anything anymore
  • 19:23 ebernhardson: completed running cirrussearch category moves on commonswiki
  • 18:19 catrope@tin: Synchronized php-1.31.0-wmf.1/includes/libs/CSSMin.php: T176884 (duration: 00m 47s)
  • 18:08 chasemp: remove 'ToolLabs webservice - lighttpd on precise' catchpoint check for Catchpoint
  • 18:07 chasemp: remove labsdb1002 checks on catchpoint (Toolforge)
  • 18:06 chasemp: remove Labs puppetmaster eqiad catchpoint check (for Toolforge)
  • 18:06 chasemp: remove grid-start-precise catchpoint check
  • 18:04 chasemp: update catchpoing test for stream.wikimedia.org to watch https://stream.wikimedia.org/?doc
  • 17:00 ebernhardson: kicking off extra job runners to process jobs moving commonswiki Category pages from general to content serach index
  • 16:37 andrewbogott: re-imaging labvirt1017, 1018 in order to get it on the standard 4.4.0-81-generic kernel
  • 15:52 andrewbogott: re-imaging labvirt1015 in order to get it on the standard 4.4.0-81-generic kernel
  • 13:21 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2070, db2069 and db2042 - T174509 (duration: 00m 47s)
  • 13:18 Amir1: starting a round of cleanup in ores_classification table in enwiki (T159753)
  • 13:09 elukey: depool mw1265 (hhvm 3.18.5) - disk filled up
  • 12:43 bblack: removed numa=isolate sys entries manually (runtime + future boots) from cp4021
  • 11:53 bblack: cp4021: reboot for numa mode switch
  • 11:44 elukey: re-enable job runner daemons on mw130[8,9]
  • 11:15 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2053 db2046 - T174509 (duration: 00m 48s)
  • 11:05 elukey: precautionary stop of jobrunner/jobchron on the new mw130[8,9] - job queue size rapid increase investigation
  • 09:55 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1309.eqiad.wmnet
  • 09:33 ema: upgrade pybal to 1.14.0 on codfw primaries
  • 08:20 ema: upgrade pybal to 1.14.0 on codfw secondaries
  • 05:53 ema: cp4024 repooled T174891
  • 05:21 ema: powerup cp4024 T174891
  • 00:26 ejegg: updated fundraising tools from 69a9627 to 6d4b6f3
  • 00:16 ejegg: updated CiviCRM 2138869 to 784756b
  • 00:07 mutante: releases1001 - starting Jenkins setup wizard with generated admin pass
  • 00:05 mutante: releases1001 - stopped puppet, manually fixing --prefix=/ci setting for jenkins process, killing it, removing init.d file, starting with systemd, jenkins now up T164030

2017-09-28

  • 23:46 awight@tin: Finished deploy [ores/deploy@42c5663]: Cause ORES service restart (duration: 00m 20s)
  • 23:45 awight@tin: Started deploy [ores/deploy@42c5663]: Cause ORES service restart
  • 22:18 MaxSem: Deployed fix for T176247
  • 21:30 ejegg: updated CiviCRM from 3f48cb0 to 2138869
  • 21:23 XenoRyet: Updated SmashPig from dc62203 to d056a13
  • 19:36 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.1
  • 19:34 legoktm: mysql:wikiadmin@db1052 [enwiki]> delete from user_groups where ug_user =17629530 limit 1; (T176798)
  • 19:31 bblack: cp4026 - depool->reboot for bios updates, isolcpus fix
  • 19:28 legoktm: mysql:wikiadmin@db1062 [centralauth]> UPDATE localuser SET lu_attached_method="new" WHERE lu_attached_method is NULL AND lu_name="Rua";
  • 19:14 ejegg: updated fundraising tools 5708cf1 Add 'first_donation_date' column to Silverpop export
  • 19:14 robh: cp4024 initial tests done (takes about 2 hours or so) then prompted for more in depth testing (hit yes, no failures so far) T174891
  • 19:00 bblack: cp4025 - depool->reboot for bios updates, isolcpus fix
  • 18:54 thcipriani@tin: Synchronized php-1.31.0-wmf.1/skins/MinervaNeue/includes/skins/SkinMinerva.php: SWAT: Revision::newFromTitle may return null T176882 (duration: 00m 50s)
  • 18:42 bblack: cp4023 - depool->reboot for bios updates, isolcpus fix
  • 18:37 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create rollbacker group at viwiktionary T176979 (duration: 00m 50s)
  • 18:26 thcipriani@tin: Synchronized docroot/noc: SWAT: copy squid.php->reverse-proxy.php, squid-labs->reverse-proxy-staging PART II T104148 (duration: 00m 49s)
  • 18:25 thcipriani@tin: Synchronized wmf-config: SWAT: copy squid.php->reverse-proxy.php, squid-labs->reverse-proxy-staging T104148 (duration: 00m 51s)
  • 17:58 arlolra: Updated Parsoid to 2f4b9a8c (T176363, T176151, T170832)
  • 17:52 mutante: librenms - renewed LE TLS cert ..via puppet, after Icinga alerted and accidentally still had "do_acme: false" in Hiera
  • 17:46 arlolra@tin: Finished deploy [parsoid/deploy@84342fe]: Updating Parsoid to 2f4b9a8c (duration: 15m 25s)
  • 17:43 bblack: cp4028 - depool->reboot for bios updates, isolcpus fix
  • 17:40 moritzm: installing apache updates on cobalt
  • 17:32 bblack: cp4022 - depool->reboot for bios updates, isolcpus fix
  • 17:30 arlolra@tin: Started deploy [parsoid/deploy@84342fe]: Updating Parsoid to 2f4b9a8c
  • 17:25 marostegui: Run fixStuckGlobalRename.php --wiki=enwiktionary --logwiki=metawiki "CodeCat" "Rua" on terbium - T176985
  • 17:20 bblack: cp4027 - depool->reboot for bios updates, isolcpus fix
  • 17:08 bblack: cp4021 - depool->reboot for bios update
  • 17:05 robh: bios firmware uploaded and awaiting reboot for application on cp402[1235678]
  • 16:48 bd808: Created centralauth.{analytics,web}.db.svc.eqiad.wmflabs Designate CNAME (T176978)
  • 16:26 robh: cp4024 has had ilom and bios firmware updaed per epsa error checking, now re-running epsa utility T174891
  • 16:09 robh: cp4024 bios update in progress (has been for last 5 minutes), letting it alone for a solid 30 to let it finish
  • 16:01 marostegui: Global rename of CodeCat → Rua - T176985
  • 15:22 hoo: Ran scap pull on mwdebug1001 after experiments with T176844
  • 14:54 dcausse: restarting elastic on relforge servers to reinstall prod ltr plugin
  • 14:51 marostegui: Stop MySQL on db1035 as server is going to be decommissioned - T176931
  • 14:45 moritzm: installing apache updates on puppet mastes (will trigger a few failing puppet runs, but ircecho disabled temporarily)
  • 14:42 marostegui: Remove db1035 from tendril - T176931
  • 14:31 moritzm: upgrading mw1276 (API canary) to HHVM 3.18.5
  • 14:28 moritzm: upgrading mwdebug* to HHVM 3.18.5
  • 13:56 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1035 from config as it will be decommissioned - T176931 (duration: 00m 49s)
  • 13:55 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1035 from config as it will be decommissioned - T176931 (duration: 00m 50s)
  • 13:40 zeljkof: EU SWAT finished
  • 13:38 zfilipin@tin: Synchronized php-1.31.0-wmf.1/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: SWAT: ve.init.mw.DesktopArticleTarget: Remove hack for reversed tabs in RTL in Vector (T50017) (duration: 00m 50s)
  • 13:37 moritzm: upgrading mw1262 to mw1265 (canary app servers) to HHVM 3.18.5
  • 13:35 moritzm: uploaded HHVM 3.18.5 for jessie-wikimedia to apt.wikimedia.org
  • 13:26 marostegui: Optimize table on s4 codfw master (db2051), with replication (lag might be generated) - T174509
  • 13:13 ema: bounce pybal on ulsfo primaries to pick up bgp-local-ips config change T103882
  • 13:12 marostegui: Drop  redundant indexes from pagelinks and templatelinks on s4 - T174509
  • 13:10 moritzm: upgrading mw1261 to HHVM 3.18.5
  • 13:08 ema: bounce pybal on ulsfo secondaries to pick up bgp-local-ips config change T103882
  • 13:02 marostegui: Optimize template links and pagelinks on db2053 and db2046 - T174509
  • 13:00 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2053 db2046 - T174509 (duration: 00m 49s)
  • 12:48 dcausse: restarting elastic on relforge servers to test a new version of the ltr plugin
  • 12:27 ema: reboot cp402[2-6] to disable numa isolation
  • 10:30 gehel: restart elasticsearch on relforge to validate new logging config - T175242
  • 10:09 elukey: incrementally add mw1312-17 api-appservers to serve live traffic (weights will be raised incrementally)
  • 10:00 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1308.eqiad.wmnet
  • 09:59 elukey: added new mediawiki jobrunner - mw1308
  • 09:52 ema: upgrade pybal to 1.14.0 on lvs1009-10
  • 09:49 aaron@tin: Synchronized wmf-config/CommonSettings.php: Enable logging of post-send DB updates (duration: 00m 55s)
  • 09:42 ema: upgrade pybal to 1.14.0 on lvs400[13]
  • 08:00 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1288.eqiad.wmnet
  • 07:27 moritzm: upgrading app servers in deployment-prep to HHVM 3.18.5
  • 07:14 moritzm: installing apache updates on silver/wikitech.wikimedia.org
  • 06:59 moritzm: installing apache updates on graphite* hosts
  • 06:28 elukey: restart varnish backend on cp3032
  • 05:54 marostegui: Optimize tables pagelinks and templatelinks on dbstore2001 (s2) - T174509
  • 05:53 marostegui: Optimize tables pagelinks and templatelinks on dbstore2002 (s2) - T174509
  • 05:43 marostegui: Run maintain-views on labsdb1001, labsdb1003, labsdb1009, labsdb1010, labsdb1011 for hiwikivoyage - T173027
  • 05:38 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2049 and db2041 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 47s)
  • 05:29 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2064 db2063 - T174509 (duration: 00m 48s)
  • 05:21 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2076 db2067 db2060 - T174509 (duration: 00m 48s)
  • 05:19 tstarling@tin: Finished scap: updating Vector skin for gerrit 381153 (duration: 20m 39s)
  • 04:59 tstarling@tin: Started scap: updating Vector skin for gerrit 381153
  • 03:57 bblack: ulsfo dns-depooled at ~03:51
  • 03:26 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Sep 28 03:26:18 UTC 2017 (duration 6m 10s)
  • 03:20 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 15m 44s)
  • 02:42 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 15m 24s)
  • 00:36 twentyafterfour: upgrading phabricator, expect minimal downtime

2017-09-27

  • 23:39 robh: cp4024 is running hardware testing, leave it alone T174891
  • 23:17 catrope@tin: Synchronized wmf-config/Wikibase.php: Make CirrusSearch default for wbsearchentities on testwikidatawiki T175741 (duration: 00m 48s)
  • 22:34 maxsem@tin: Synchronized php-1.31.0-wmf.1/includes/EditPage.php: https://gerrit.wikimedia.org/r/#/c/381139/1 (duration: 00m 49s)
  • 19:14 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 -> wmf.1
  • 19:13 demon@tin: Synchronized php: symlink bump (duration: 00m 47s)
  • 19:10 demon@tin: Synchronized php-1.31.0-wmf.1/maintenance/backup.inc: unbreak dumps (duration: 00m 49s)
  • 18:46 catrope@tin: Finished scap: SWAT: T176762, T175358, T172023, T174719, and add html5-misnesting to Linter (duration: 22m 55s)
  • 18:39 bblack: re-pooling upload@ulsfo (text@ulsfo remains pooled)
  • 18:23 catrope@tin: Started scap: SWAT: T176762, T175358, T172023, T174719, and add html5-misnesting to Linter
  • 18:21 ejegg: updated payments-wiki from 6104e3d to c2a3d43
  • 17:40 ejegg: updated SmashPig Adyen settings
  • 17:29 bblack: uploaded nginx-1.13.5-1+wmf1 to stretch-wikimedia, and matching nginx-1.13.5-1+wmf1~jessie1 to jessie-wikimedia
  • 16:27 XioNoX: setting local-as to selected transit BGP sessions - T167840
  • 16:00 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4026.ulsfo.wmnet
  • 16:00 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4025.ulsfo.wmnet
  • 15:59 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4023.ulsfo.wmnet
  • 15:58 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4022.ulsfo.wmnet
  • 15:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1103 with low weight - T172679 (duration: 00m 47s)
  • 15:51 ema: lvs4002: upgrade pybal to 1.14.0
  • 15:49 ema: lvs4004: upgrade pybal to 1.14.0
  • 15:28 _joe_: uploaded prometheus-statsd-exporter to stretch-wikimedia T175539
  • 15:24 bblack: rebooting cp402[2356] (bnx2x oops)
  • 15:15 moritzm: revoking salt certs for WMCS
  • 15:12 mlitn@tin: Synchronized wmf-config/CommonSettings-labs.php: Remove 3D beta config (duration: 00m 47s)
  • 15:11 mlitn@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Remove 3D beta config (duration: 00m 48s)
  • 15:10 mlitn@tin: Synchronized wmf-config/CommonSettings.php: Load 3D extension (duration: 00m 49s)
  • 15:09 mlitn@tin: Synchronized wmf-config/InitialiseSettings.php: Config 3D to be loaded on test, test2 (duration: 00m 48s)
  • 15:00 mlitn@tin: Finished scap: Enable 3D extension (duration: 37m 09s)
  • 14:58 ema: lvs1007: upgrade pybal to 1.14.0
  • 14:45 elukey: rolling restart of all the Yarn nodemanager daemons on analytics1028-1068
  • 14:43 bblack: cp1008/pinkunicorn upgraded to test build of nginx-1.13.5-1+wmf1
  • 14:28 volans: uploaded cumin_1.2.1-1_amd64.deb to apt.wikimedia.org jessie-wikimedia
  • 14:24 akosiaris: change wtp1001 to wtp1024 weights to 5 from 15 in preparation for deprecation
  • 14:24 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1024.eqiad.wmnet
  • 14:24 mlitn@tin: Started scap: Enable 3D extension
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1023.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1022.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1021.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1020.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1019.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1018.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1017.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1016.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1015.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1014.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1013.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1012.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1011.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1010.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1009.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1008.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1007.eqiad.wmnet
  • 14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1006.eqiad.wmnet
  • 14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1005.eqiad.wmnet
  • 14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1004.eqiad.wmnet
  • 14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1003.eqiad.wmnet
  • 14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1002.eqiad.wmnet
  • 14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1001.eqiad.wmnet
  • 14:17 akosiaris: T175242 restart eventstreams across the fleet to pick up the change in a rolling restart manner with a batch size of 2
  • 14:16 akosiaris: T175242 restart parsoid across the fleet to pick up the change in a rolling restart manner with a batch size of 5
  • 14:03 akosiaris: T175242 restart tilerator, tileratorui, restbase across the fleet to pick up the change in a rolling restart manner with a batch size of 2
  • 13:43 akosiaris: T175242 re-enable puppet across aqs kafka maps maps-test ores restbase restbase-dev sca scb wtp clusters for merging https://gerrit.wikimedia.org/r/#/c/376500/. Run puppet as well in a batched execution
  • 13:42 anomie@tin: Synchronized php-1.31.0-wmf.1/includes/specials/SpecialWatchlist.php: SWAT: {{gerrit|380968}} Fix watchlist "in the last X hours" display (duration: 00m 48s)
  • 13:42 elukey: raised Hadoop HDFS namenode master daemon max heap size to 6G (prev 4G) on analytics100[12]
  • 13:31 akosiaris: T175242 eventstreams requires manual restart
  • 13:25 akosiaris: T175242 parsoid requires manual restart
  • 13:15 akosiaris: T175242 restbase requires manual restart
  • 13:11 addshore: SWAT done
  • 13:11 addshore@tin: Synchronized php-1.31.0-wmf.1/extensions/TwoColConflict/modules/ext.TwoColConflict.BaseVersionSelector.css: SWAT: Address changes in the label style in OOUI (duration: 00m 46s)
  • 13:09 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add autopatrolled user group to dty.wikipedia (T176709) (duration: 00m 47s)
  • 13:01 akosiaris: T175242 tilerator and tileratorui need manually restart
  • 12:54 akosiaris: T175242 enabled puppet in aqs kafka maps maps-test selected hosts and ran puppet manually.
  • 12:47 akosiaris: T175242 disable puppet across aqs kafka maps maps-test ores restbase restbase-dev sca scb wtp clusters for merging https://gerrit.wikimedia.org/r/#/c/376500/
  • 12:20 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1103 to the array of hosts (duration: 00m 48s)
  • 12:19 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1103 to the array of hosts (duration: 00m 49s)
  • 12:14 moritzm: installing apache updates on einsteinium/tegmen
  • 12:06 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on bnwiki (T176771) (duration: 00m 49s)
  • 12:05 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on bnwiki (T176771) (duration: 01m 00s)
  • 09:45 marostegui: Stop mysql on db1035 to copy its data to db1103 - T172679
  • 09:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1035 to transfer its data to db1103 - T172679 (duration: 00m 48s)
  • 08:55 mobrovac@tin: Finished deploy [changeprop/deploy@7a3dc66]: Use the /page/title/{title}/{revision} end point for revision-visibility-change - T158100 (duration: 01m 26s)
  • 08:53 mobrovac@tin: Started deploy [changeprop/deploy@7a3dc66]: Use the /page/title/{title}/{revision} end point for revision-visibility-change - T158100
  • 08:43 mobrovac@tin: Finished deploy [restbase/deploy@1cc530b]: Introduce the /page/title/{title}/{revision} end point - T158100 (duration: 10m 01s)
  • 08:34 elukey: raise traffic weights to 30 for mw13[19-28] incrementally - T165519
  • 08:33 mobrovac@tin: Started deploy [restbase/deploy@1cc530b]: Introduce the /page/title/{title}/{revision} end point - T158100
  • 08:16 marostegui: Optimize tables pagelinks templatelinks on db2064 and db2063 - T174509
  • 08:16 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2064 db2063 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 48s)
  • 07:55 marostegui: Optimize pagelinks and template links tables on  db2076 db2067 db2060 - T174509
  • 07:52 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2076 db2067 db2060 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 48s)
  • 07:37 marostegui: Optimize pagelinks and template links tables on db2070 db2069 db2042 db2016 - T174509
  • 07:27 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2070, db2069 and db2042 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 48s)
  • 07:17 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2071 and db2072 - T174509 (duration: 00m 47s)
  • 06:57 marostegui: Deploy alter table on s6 codfw - T174509
  • 06:55 marostegui: Deploy alter table on db1052 (enwiki master) - T174509
  • 06:37 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add task number to db2044 line (duration: 00m 48s)
  • 06:37 ema: restart varnish-be on cp3040
  • 06:36 moritzm: installing git security updates (our config is not affected, but still fixing the underlying issue)
  • 06:33 ema: restart varnish-be on cp3041
  • 06:29 marosteg1i: Reboot db2044 after storage failure - T174764
  • 06:15 elukey: restart varnish backend on cp3043
  • 05:41 madhuvishy: service nova-fullstack restart on labnet1001
  • 03:17 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Sep 27 03:17:22 UTC 2017 (duration 6m 22s)
  • 03:11 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 15m 55s)
  • 02:32 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 08m 42s)

2017-09-26

  • 23:25 thcipriani@tin: Synchronized php-1.31.0-wmf.1/extensions/VisualEditor/modules/ve-mw/ui/tools/ve.ui.MWSignatureTool.js: SWAT: Revert MWSignatureTool case (duration: 00m 48s)
  • 23:20 thcipriani@tin: Synchronized php-1.30.0-wmf.19/resources/src/mediawiki.rcfilters/mw.rcfilters.init.js: SWAT: RCFilters: Log performance data T176652 (duration: 00m 48s)
  • 23:19 thcipriani@tin: Synchronized php-1.31.0-wmf.1/resources/src/mediawiki.rcfilters/mw.rcfilters.init.js: SWAT: RCFilters: Log performance data T176652 (duration: 00m 51s)
  • 22:03 no_justification: gerrit master restart, back momentarily
  • 20:32 mutante: phab1001 - restarted apache after upgrade
  • 20:30 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2044 - T174764 (duration: 00m 50s)
  • 20:20 mutante: phab1001 (prod phab) - upgrade Apache package
  • 20:15 mutante: phab2001 - install Apache and MariaDB package upgrades
  • 19:16 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.31.0-wmf.1
  • 18:25 mepps: Updated crm from 89b8007 to 3f48cb0
  • 18:00 demon@tin: Finished scap: 1.31.0-wmf.1 bootstrap + testwiki (duration: 47m 56s)
  • 17:12 demon@tin: Started scap: 1.31.0-wmf.1 bootstrap + testwiki
  • 15:50 volans: powecycled cp2012, no ping or ssh console stuck
  • 15:18 XioNoX: starting eqiad frack switch to new infra - T174218
  • 15:06 gehel: restarting blazegraph on all wdqs nodes for heap resize - T175919
  • 14:47 mobrovac@tin: Finished deploy [restbase/deploy@f989dd9]: Remove the old mobileapps module completely - T169940 (duration: 20m 35s)
  • 14:36 marostegui: Optimize tables pagelinks and templatelinks on db2071 and db2072 from s1 - T174509
  • 14:27 mobrovac@tin: Started deploy [restbase/deploy@f989dd9]: Remove the old mobileapps module completely - T169940
  • 14:26 hoo: mwscript refreshLinks.php --wiki elwiki --namespace 0 on terbium has finished (T151717)
  • 14:12 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2071 and db2072 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 44s)
  • 13:39 marostegui: Deploy alter table on s6 master (with replication enable, so lag will be generated) db2028 - T163979
  • 13:25 hashar@tin: Synchronized wmf-config/filebackend.php: Expose Thumbor swift username - T144479 (duration: 00m 44s)
  • 13:24 hashar@tin: Synchronized wmf-config/CommonSettings.php: Enable asia-specific Navigation Timing metric - T169522 (duration: 00m 47s)
  • 13:22 godog: upgrade grafana to 4.5.2 on labmon1001 - T175980
  • 13:15 hashar@tin: Synchronized wmf-config/CommonSettings.php: Disable RelatedArticles instrumentation on all wikis - T174944 (duration: 00m 45s)
  • 13:02 marostegui: Drop redundant indexes from s6 - T174509
  • 13:00 elukey: add mw132[7,8] to live traffic (new appservers) - weights will be increased incrementally from 5 to 20
  • 12:44 gehel: restarting blazegraph on wdqs1004 / wdqs2001 for heap resive - T175919
  • 12:27 marostegui: Drop redundant indexes from s2 - T174509
  • 12:12 marostegui: Drop redundant indexes from enwiki on eqiad - T174509
  • 12:03 kartik@tin: Finished deploy [cxserver/deploy@aa232ee]: Update cxserver, registry reconstruction (duration: 03m 03s)
  • 12:00 kartik@tin: Started deploy [cxserver/deploy@aa232ee]: Update cxserver, registry reconstruction
  • 11:58 moritzm: uploaded prometheus-apache-exporter 0.3-1+deb9u1 to apt.wikimedia.org/stretch-wikimedia
  • 11:57 marostegui: Drop redundant indexes from enwiki on codfw - T174509
  • 11:49 moritzm: uploaded prometheus-hhvm-exporter 0.3.1+deb9u1 to apt.wikimedia.org/stretch-wikimedia
  • 11:30 kartik@tin: (no justification provided)
  • 11:29 kartik@tin: Finished deploy [cxserver/deploy@f1d4851]: Update cxserver, registry reconstruction (duration: 00m 52s)
  • 11:28 kartik@tin: Started deploy [cxserver/deploy@f1d4851]: Update cxserver, registry reconstruction
  • 10:52 volans: uploaded cumin_1.1.1-1_amd64.deb to apt.wikimedia.org jessie-wikimedia
  • 10:34 joal@tin: Finished deploy [analytics/refinery@5d806c6]: Regular analytics deploy (duration: 08m 00s)
  • 10:26 joal@tin: Started deploy [analytics/refinery@5d806c6]: Regular analytics deploy
  • 10:20 _joe_: rebooting copper (again)
  • 09:12 elukey: add mw132[4,5,6] to live traffic (new appservers) - weights will be increased incrementally from 5 to 20
  • 09:10 _joe_: rebooting copper to verify docker setup is correct
  • 08:46 moritzm: installing apache updates on mw* app servers
  • 08:44 godog: run puppet on ms-fe* to reduce conntrack generated by swift internal clients towards backends - T173731
  • 08:24 godog: roll-restart thumbor to upgrade to 1.5 - T173804
  • 08:09 godog: roll-restart swift-proxy to apply https://gerrit.wikimedia.org/r/#/c/380483/ - T161535
  • 08:08 marostegui: Drop old temporary tar.gz files from dbstore1001 to get back some disk space
  • 08:07 marostegui: Drop table trackbacks from s3 - T175051
  • 07:48 mattflaschen@tin: Synchronized wmf-config/CommonSettings-labs.php: RCFilters: Beta Cluster only (duration: 00m 46s)
  • 07:46 marostegui: Drop trackbacks table from s7 - T175051
  • 07:28 marostegui: Drop trackbacks table from s6 - T175051
  • 07:16 moritzm: installing perl security updates
  • 06:45 _joe_: restarting varnish backend on cp3033, throwing 503s
  • 06:29 mobrovac@tin: Finished deploy [cpjobqueue/deploy@17c3833]: (no justification provided) (duration: 00m 28s)
  • 06:28 mobrovac@tin: Started deploy [cpjobqueue/deploy@17c3833]: (no justification provided)
  • 06:18 marostegui: Drop table trackbacks from s4 - T175051
  • 06:06 marostegui: Drop table trackbacks from s5 (only exists on dewiki) - T175051
  • 05:48 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2047 - T176573 (duration: 00m 44s)
  • 05:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove old db1055 comments (duration: 00m 46s)
  • 02:29 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Sep 26 02:29:31 UTC 2017 (duration 5m 52s)
  • 02:23 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 07m 26s)

2017-09-25

  • 23:58 ejegg: updated payments-wiki from d2ce697 to 6104e3d
  • 23:15 mutante: gerrit2001 reinstalled with stretch, revoked old puppet cert, accepted new puppet cert, initial run that will do base and all the gerrit things at once.. (T168562)
  • 22:53 mutante: gerrit2001 - PXE booting
  • 22:34 mutante: gerrit2001 - schedule downtime for reinstall with strech, reboot imminent
  • 20:57 hoo: Manually pruned the sites and site_identifiers table on hiwikivoyage, than ran populateSitesTable.php. (T173030)
  • 20:37 arlolra: Updated Parsoid to 4230c27b (T176425, T173029)
  • 20:30 arlolra@tin: Finished deploy [parsoid/deploy@376faad]: Updating Parsoid to 4230c27b (duration: 16m 52s)
  • 20:13 arlolra@tin: Started deploy [parsoid/deploy@376faad]: Updating Parsoid to 4230c27b
  • 19:58 XioNoX: renumbering ams BGP communities - T167840
  • 19:25 robh@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp4021.ulsfo.wmnet
  • 19:02 robh: cp4021 coming down for memory swap
  • 19:02 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp4021.ulsfo.wmnet
  • 18:54 MaxSem: started populating ip_changes on group2 wikis
  • 18:38 demon@tin: Synchronized scap/plugins/clean.py: no-op/consistency (duration: 00m 45s)
  • 18:22 ejegg: updated CiviCRM from 45c56cf to 89b8007
  • 17:58 demon@tin: Pruned MediaWiki: 1.30.0-wmf.15 (duration: 02m 46s)
  • 17:07 gehel@tin: Finished deploy [wdqs/wdqs@3eec185]: Deploy new Blazegraph binary & GUI update (duration: 01m 51s)
  • 17:05 gehel@tin: Started deploy [wdqs/wdqs@3eec185]: Deploy new Blazegraph binary & GUI update
  • 17:02 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: Disable the updateBetaFeaturesUserCounts logging channel - T175637 (duration: 00m 46s)
  • 16:37 demon@tin: Synchronized php-1.30.0-wmf.19/maintenance/getConfiguration.php: getConfiguration: Don't bail when a valid variable is set null (duration: 00m 47s)
  • 15:57 urandom: T169940: Converting restbase-ng data tables to size-tiered compaction
  • 14:39 gehel: dropping badly named apifeature index from cirrus elasticsearch codfw/eqiad - T176430
  • 14:31 gehel: rolling restart of logstash for config change - T176430
  • 14:13 gehel: rolling restart of logstash for config change - T176430
  • 13:46 zeljkof: EU SWAT finished
  • 13:45 zfilipin@tin: Synchronized static/images/project-logos/hiwiki.png: SWAT: Make hiwiki logo a little bit bigger (T175721) (duration: 00m 46s)
  • 13:37 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Import sources on hr.wikibooks (T176320) (duration: 00m 46s)
  • 13:30 addshore@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/WikimediaEventsHooks.php: SWAT: gerrit:379749 onBeforeInitializeWMDECampaign, fix early bail condition T174948 (duration: 00m 46s)
  • 13:18 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add several rights to eliminators in fawiki (T176553) (duration: 00m 46s)
  • 12:52 moritzm: removed salt master from neodymium
  • 12:38 moritzm: revoked all salt keys on neodymium
  • 11:58 godog: powercycle ms-be2020 - blk_update_request: I/O error, dev sdb, sector in consol
  • 11:54 moritzm: removing salt-minion/salt-common from production
  • 11:47 Dereckson: Repopulate sites tables for Wikidata clients (T173013)
  • 11:47 Dereckson: Create new database and tables for hi.wikivoyage (T173013)
  • 11:45 dereckson@tin: Synchronized wmf-config/interwiki.php: Interwiki map update (+hiwikivoyage and other changes previous month, T176572) (duration: 00m 46s)
  • 11:39 marostegui: Run redact_sanitarium for hiwikivoyage on db1095 - T173027
  • 11:39 moritzm: upgrading tor to latest stable release (0.3.1.7)
  • 11:37 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Initial configuration for hi.wikivoyage.org (T173013) (duration: 00m 46s)
  • 11:37 dereckson@tin: Synchronized static/images/project-logos/: Logos for hi.wikivoyage.org (T173013) (duration: 00m 45s)
  • 11:34 dereckson@tin: rebuilt wikiversions.php and synchronized wikiversions files: +hiwikivoyage (T173013)
  • 11:32 dereckson@tin: Synchronized dblists: Add hiwikivoyage to the set (T173013) (duration: 00m 46s)
  • 10:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1055 weight (duration: 00m 46s)
  • 10:14 elukey: add mw1323 to live traffic (mw appserver) - traffic weights will go from 5 to 20 incrementally
  • 09:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1055 weight and remove main traffic from special slaves (duration: 00m 46s)
  • 09:07 marostegui: Add 50GB to db1036 /srv partition - T176311
  • 09:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1036 and full repool db1101 - T176311 (duration: 00m 46s)
  • 08:49 moritzm: uploaded php-luasandbox 2.0.14, hhvm-tidy 0.1.3 and php-wikidiff2 1.4.1 for stretch-wikimedia to apt.wikimedia.org
  • 08:36 mobrovac@tin: Finished deploy [restbase/deploy@ab24f70]: Switch mobile-sections to next-gen storage and stop storing the current-day feed in Cassandra - T169940 T176233 (duration: 10m 03s)
  • 08:26 mobrovac@tin: Started deploy [restbase/deploy@ab24f70]: Switch mobile-sections to next-gen storage and stop storing the current-day feed in Cassandra - T169940 T176233
  • 08:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1055 weight (duration: 00m 45s)
  • 07:56 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1101 - T176311 (duration: 00m 46s)
  • 07:42 moritzm: uploaded hhvm 3.18.2+dfsg-1+wmf5+deb9u1 for stretch-wikimedia to apt.wikimedia.org
  • 07:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1055 weight (duration: 00m 47s)
  • 07:37 elukey: add mw132[0,2] to live traffic (mw appservers) - traffic weights will go from 5 to 20 incrementally
  • 06:38 moritzm: installing apache security updates on logstash*
  • 02:36 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Sep 25 02:36:48 UTC 2017 (duration 6m 42s)
  • 02:30 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 07m 17s)

2017-09-24

  • 14:23 jynus: restarting db2047 for kernel upgrade
  • 14:16 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2047 (duration: 00m 48s)
  • off: restarted ircecho on einsteinium

2017-09-23

2017-09-22

  • 23:36 mutante: gerrit2001 - rebooting .. wave a dead chicken http://www.catb.org/jargon/html/W/wave-a-dead-chicken.html
  • 22:56 mutante: gerrit2001 - trying to manually start gerrit again, as opposed to puppet doing it.. debugging gerrit-ssh issue there, cobalt still untouched
  • 22:34 mutante: gerrit2001 - stopping gerrit with gerrit.sh stop, letting puppet start it again
  • 22:29 mutante: gerrit2001 - systemd says gerrit.service is failed. gerrit.sh start says "Already Running!!" :p - cobalt is fine
  • 18:26 demon@tin: Pruned MediaWiki: 1.30.0-wmf.18 [keeping static files] (duration: 01m 29s)
  • 15:21 hashar: Restarted Jenkins. Out of memory)
  • 14:49 jynus@tin: Synchronized wmf-config/db-eqiad.php: repool db1101 with low weight (duration: 00m 47s)
  • 14:43 moritzm: uploaded php-luasandbox build for src:php5.5 (required for CI tests on jessie)
  • 14:42 moritzm: updated tor packages to 0.3.1.7 (new stable series)
  • 13:49 elukey: mw1321 (new appserver) serving traffic (going to increase its weight up to 20)
  • 12:02 hashar: apt-get upgrade on contint1001 / contint2001
  • 09:42 elukey: mw1319 (new appserver) serving traffic (going to increase its weight up to 20)
  • 09:14 jynus: stop mariadb at db1055 for upgrade and maintenance
  • 06:32 moritzm: installing emacs security updates on trusty (Debian already fixed)

2017-09-21

  • 23:56 ejegg: updated payments-wiki from 7ed7243 to d2ce697
  • 23:23 ebernhardson@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/extension.json: T175047: Stop delivering search relevance survey javascript (duration: 00m 46s)
  • 23:18 ebernhardson@tin: Synchronized wmf-config: T175047: Turn off human search relevance survey (duration: 00m 48s)
  • 23:16 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T175047: Turn off human search relevance survey (duration: 00m 46s)
  • 22:59 jynus@tin: Synchronized wmf-config/db-eqiad.php: depool db1055 (duration: 00m 46s)
  • 22:12 volans: uploaded cumin_1.1.0-1_amd64.deb to apt.wikimedia.org jessie-wikimedia
  • 22:10 ejegg: updated payments-wiki from 9da8482 to 7ed7243
  • 21:07 demon@tin: Synchronized wmf-config/CommonSettings.php: extdist settings (duration: 00m 46s)
  • 21:04 ejegg: updated CiviCRM from fc56901 to 45c56cf
  • 19:32 demon@tin: Synchronized wmf-config/InitialiseSettings.php: adjust logging on csp reports (duration: 00m 46s)
  • 19:20 demon@tin: Synchronized php-1.30.0-wmf.19/extensions/VisualEditor/ApiVisualEditor.php: fix user/wgUser mixup (duration: 00m 46s)
  • 19:16 gehel: upgrade to nodejs 6.11 on maps servers (including restart of tilerator / kartotherian) - T171707
  • 19:16 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4026.*
  • 19:04 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.19
  • 18:56 urandom: T171772: Applying locally hacked Prometheus exporter config to RESTBase dev Cassandra instances
  • 18:44 thcipriani@tin: Synchronized php-1.30.0-wmf.19/resources/src/mediawiki.rcfilters: SWAT: WLFilters: Do not hide .watchlistDetails while loading T176300 RCFilters: Make the interface not jump around while loading (duration: 00m 49s)
  • 18:41 urandom: T171772: Restarting Cassandra restbase-dev1004-a to apply locally hacked Prometheus exporter config
  • 18:34 thcipriani@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Turning off Explore Similar AB test T175649 (duration: 00m 49s)
  • 18:31 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Print instrumentation T176341 (duration: 00m 52s)
  • 14:56 moritzm: uploaded php-defaults and php-redis built against src:php5.5 to component/ci for jessie-wikimedia
  • 13:45 gehel: upgrade to nodejs 6.11 on the full maps-test cluster - T171707
  • 13:12 zeljkof: EU SWAT finished
  • 13:09 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Switch elasticsearch active cluster to codfw" (duration: 00m 48s)
  • 13:09 zfilipin@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Revert "Switch elasticsearch active cluster to codfw" (duration: 00m 48s)
  • 13:07 zfilipin@tin: Synchronized tests/cirrusTest.php: SWAT: Revert "Switch elasticsearch active cluster to codfw" (duration: 00m 49s)
  • 12:55 _joe_: stopped long-running runjobs for refreshlinks on commons.
  • 12:50 mobrovac@tin: Finished deploy [restbase/deploy@bc02191]: produce a diff for the mobile proxy (duration: 18m 34s)
  • 12:32 mobrovac@tin: Started deploy [restbase/deploy@bc02191]: produce a diff for the mobile proxy
  • 11:59 moritzm: upgrading apache on deployment servers and script runners
  • 11:40 moritzm: upgrading apache on canary app servers
  • 11:17 moritzm: installing apache security updates (our configurations are not affected, the upload of 2.4.10-10+deb8u11+wmf1 is mostly for the benefit of whatever people are doing in Cloud VPS, but we should still fix the underlying bug in production as well)
  • 11:14 dcausse: reindexing all hebrew wikis to pickup the new hebmorph analyzer
  • 11:09 dcausse: reindexing ilwikimedia in codfw to pickup and test new hebmorph analyzer
  • 10:26 akosiaris: upload docker-ce_17.06.2~ce-0~debian_amd64.deb to apt.wikimedia.org jessie-wikimedia/thirdparty/ci T175293
  • 09:59 dcausse: reindexing group1 wikis T176397
  • 09:54 dcausse: reindexing group0 wikis T176397
  • 08:35 moritzm: unbreak deployment-puppetmaster02 in deployment-prep (broken by unattended-upgrades update of apache T159254)
  • 08:35 akosiaris: pool all of wtp1025 to wtp1048 T165520
  • 08:35 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1044.eqiad.wmnet
  • 08:35 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1043.eqiad.wmnet
  • 08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1042.eqiad.wmnet
  • 08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1041.eqiad.wmnet
  • 08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1040.eqiad.wmnet
  • 08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1039.eqiad.wmnet
  • 08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1038.eqiad.wmnet
  • 08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1037.eqiad.wmnet
  • 08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1036.eqiad.wmnet
  • 08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1035.eqiad.wmnet
  • 08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1034.eqiad.wmnet
  • 08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1033.eqiad.wmnet
  • 08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1032.eqiad.wmnet
  • 08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1031.eqiad.wmnet
  • 08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1030.eqiad.wmnet
  • 08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1029.eqiad.wmnet
  • 08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1028.eqiad.wmnet
  • 08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1027.eqiad.wmnet
  • 08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1026.eqiad.wmnet
  • 08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
  • 08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
  • 08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
  • 08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
  • 08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
  • 08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
  • 08:30 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
  • 08:30 akosiaris@puppetmaster1001: conftool action : set/weight=10; selector: name=wtp1025.eqiad.wmnet
  • 08:25 mobrovac@tin: Finished deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests (duration: 03m 18s)
  • 08:22 mobrovac@tin: Started deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests
  • 08:21 mobrovac@tin: Finished deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests (duration: 00m 58s)
  • 08:20 mobrovac@tin: Started deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests
  • 08:19 mobrovac@tin: Finished deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests (duration: 05m 22s)
  • 08:14 mobrovac@tin: Started deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests
  • 08:08 mobrovac@tin: Finished deploy [restbase/deploy@3e9bd9f]: New storage schema for mobile-sections - T169940 (duration: 10m 28s)
  • 07:57 mobrovac@tin: Started deploy [restbase/deploy@3e9bd9f]: New storage schema for mobile-sections - T169940
  • 07:31 _joe_: restarted aphlict on phab1001
  • 07:27 mobrovac@tin: Finished deploy [restbase/deploy@3e9bd9f]: New storage schema for mobile-sections, canary deploy for schema creation, take 2 - T169940 (duration: 00m 30s)
  • 07:27 mobrovac@tin: Started deploy [restbase/deploy@3e9bd9f]: New storage schema for mobile-sections, canary deploy for schema creation, take 2 - T169940
  • 07:09 ema: bounce pybal on lvs1003 to clear stale alert T176388
  • 06:59 ema: bounce pybal on lvs1006 to clear stale alert
  • 06:55 ema: bounce pybal on lvs1009 to clear stale alert
  • 03:24 bblack: depooled cp4026
  • 03:12 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Sep 21 03:12:41 UTC 2017 (duration 6m 36s)
  • 03:06 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 14m 44s)
  • 02:29 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 08m 50s)
  • 00:41 eileen: update civicrm from da833ac to fc56901
  • 00:23 ebernhardson@tin: Synchronized php-1.30.0-wmf.18/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: Fix schema name for search satisfaction error logging (duration: 00m 53s)
  • 00:18 ebernhardson@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: Fix schema name for search satisfaction error logging (duration: 00m 49s)
  • 00:09 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable jQuery 3 on commons (T124742) (duration: 00m 49s)
  • 00:02 catrope@tin: Synchronized php-1.30.0-wmf.19/resources/src/mediawiki.rcfilters/: Lazy-load the RCFilters menu (T176250) (duration: 00m 48s)

2017-09-20

  • 23:49 catrope@tin: Synchronized php-1.30.0-wmf.18/extensions/WikimediaEvents/: Log JS errors during search satisfaction tests (duration: 00m 48s)
  • 23:47 catrope@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/: Log JS errors during search satisfaction tests (duration: 00m 49s)
  • 23:40 mutante: phabricator: aphlict (notification service) now running on prod server
  • 23:36 mutante: phab1001 - re-enable puppet, run after follow-up fix, adding notification server config
  • 23:31 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Add throttle rule for John Michael Kohler Art Center (T176287) (duration: 00m 48s)
  • 23:29 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable user email blacklist on meta (T174694) (duration: 00m 48s)
  • 23:21 catrope@tin: Synchronized dblists/categories-rdf.dblist: Add mlwiki to categories-rdf.dblist (duration: 00m 48s)
  • 23:14 mutante: phab2001 - deluser aphlict, delgroup aphlict, try to let puppet create it again, still fails, just won't create the group
  • 23:13 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Add patroller group on metawiki (T176079) (duration: 00m 48s)
  • 23:11 mutante: phab2001 - groupadd aphlict,temp adduser aphlict to debug user creation issue
  • 22:22 ebernhardson: elasticsearch eqiad completely recovery, all green. Returning recovery max_bytes_per_sec to 20mb
  • 21:54 mutante: phab: disable puppet on phab1001 for a minute, apply notification server change on phab2001
  • 21:45 eileen: update civicrm from 0fb4676 to da833ac
  • 21:09 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4015.ulsfo.wmnet
  • 20:58 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4007.ulsfo.wmnet
  • 20:40 bsitzmann@tin: Finished deploy [mobileapps/deploy@e23bf66]: Update mobileapps to d46860e (T175759 T176263) (duration: 06m 05s)
  • 20:36 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4014.ulsfo.wmnet
  • lunch: increase max recovery bytes/s to 80mb in eqiad elasitcsearch cluster
  • 20:34 bsitzmann@tin: Started deploy [mobileapps/deploy@e23bf66]: Update mobileapps to d46860e (T175759 T176263)
  • 20:33 demon@tin: Synchronized php-1.30.0-wmf.19/includes/diff/DifferenceEngine.php: fix some warnings (duration: 00m 51s)
  • 19:55 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4016.ulsfo.wmnet
  • 19:54 eevans@tin: Finished deploy [cassandra/metrics-collector@df909a1] (dev-cluster): Deploy 4.1.0 artifact to dev cluster (duration: 00m 10s)
  • 19:54 eevans@tin: Started deploy [cassandra/metrics-collector@df909a1] (dev-cluster): Deploy 4.1.0 artifact to dev cluster
  • 19:53 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4006.ulsfo.wmnet
  • 19:48 eevans@tin: Finished deploy [cassandra/metrics-collector@df909a1]: Pushing out 4.1.0 release (duration: 00m 47s)
  • 19:47 eevans@tin: Started deploy [cassandra/metrics-collector@df909a1]: Pushing out 4.1.0 release
  • 19:39 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4008.ulsfo.wmnet
  • 19:26 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.19
  • 19:24 demon@tin: Synchronized php: symlink swap for wmf.19 (duration: 00m 49s)
  • 19:02 catrope@tin: Finished scap: Patches for T176302, T175962, T173533 (duration: 21m 29s)
  • 18:40 catrope@tin: Started scap: Patches for T176302, T175962, T173533
  • 17:29 hoo: Started mwscript refreshLinks.php --wiki elwiki --namespace 0 in a screen on terbium. Can be KILLed at any time, if needed.
  • 17:29 hoo: Started mwscript refreshLinks.php --wiki elwiki --namespace 0 in a screen on elwiki. Can be KILLed at any time, if needed.
  • 17:13 hoo: Ran mwscript refreshLinks.php --wiki elwiki --namespace 0 -e 30 for testing purposes
  • 17:06 ejegg: removed limit from donation queue consumer
  • 17:05 otto@tin: Finished deploy [eventstreams/deploy@e62ab64]: (no justification provided) (duration: 04m 44s)
  • 17:01 otto@tin: Started deploy [eventstreams/deploy@e62ab64]: (no justification provided)
  • 16:53 madhuvishy: Upgraded and restarted apache2 on labs-puppetmaster.wikimedia.org
  • 16:45 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,cluster=elasticsearch
  • 16:42 gehel: rolling restart of logstash ingesters after cirrus eqiad recovery
  • 16:34 gehel: elasticsearch eqiad is finally recovering, after the restart of elastic1051
  • 16:34 gehel: cleanup jar hell and restart elastic1051
  • 16:19 gehel: starting elasticsearch on all nodes on eqiad
  • 16:10 gehel: restarting elasticsearch masters on eqiad (elastic1030, 36 and 40)
  • 16:04 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: dc=eqiad,cluster=elasticsearch
  • 16:03 gehel: depooling all nodes in elasticsearch eqiad to investigate failed cluster restart (traffic is already directed to codfw)
  • 15:46 ejegg: limited donations queue consumer to 100 per run
  • 15:36 ejegg: updated payments-wiki from 896f16a to 9da8482
  • 15:36 gehel: upgrading elasticsearch plugins on elasticsearch eqiad, including cold restart of the cluster - T173231
  • 15:33 ejegg: limited donations queue consumer to 200 per run
  • 14:03 akosiaris@tin: Synchronized wmf-config/InitialiseSettings.php: Whitelist wtp10[25-48] for Linter (duration: 00m 49s)
  • 13:42 otto@tin: Synchronized /srv/mediawiki-staging/wmf-config/CommonSettings-labs.php: (no justification provided) (duration: 00m 49s)
  • 13:24 ema: begin rolling codfw varnish-be restarts, 20m spacing
  • 13:20 dereckson@tin: Synchronized wmf-config/CommonSettings.php: Switch elasticsearch active cluster to codfw (Gerrit:378850, 3/3) (duration: 00m 48s)
  • 13:18 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Switch elasticsearch active cluster to codfw (Gerrit:378850, 2/3) (duration: 00m 49s)
  • 13:17 dereckson@tin: Synchronized tests/cirrusTest.php: Switch elasticsearch active cluster to codfw (Gerrit:378850, 1/3, no-op in prod) (duration: 00m 49s)
  • 12:51 ema: restarted varnish-be on cp4028
  • 12:28 gehel: upgrading nodejs to 6.11 on maps-test2003 for testing - T171707
  • 12:19 ema: restarted varnish-be on cp4027
  • 12:18 moritzm: uploaded apache 2.4.10-10+deb8u11+wmf1 to jessie-wikimedia (rebuild of latest security update with our local patches on top)
  • 12:08 hoo@tin: Synchronized wmf-config/Wikibase-production.php: Enable statement usage tracking on elwiki (T151717) (duration: 00m 49s)
  • 11:58 mobrovac@tin: Finished deploy [restbase/deploy@dea2b41]: New storage schema for mobile-sections, canary deploy for schema creation - T169940 (duration: 07m 20s)
  • 11:56 ema: restarted varnish-be on cp4018
  • 11:51 mobrovac@tin: Started deploy [restbase/deploy@dea2b41]: New storage schema for mobile-sections, canary deploy for schema creation - T169940
  • 11:49 jynus: stopping replication on db1101 for faster repartition work T176311
  • 11:45 jynus@tin: Synchronized wmf-config/db-eqiad.php: depool db1101 (duration: 00m 58s)
  • 11:34 ema: restarted varnish-be on cp4010
  • 11:25 mobrovac@tin: Finished deploy [restbase/deploy@dea2b41]: (no justification provided) (duration: 00m 02s)
  • 11:25 mobrovac@tin: Started deploy [restbase/deploy@dea2b41]: (no justification provided)
  • 11:03 moritzm: rolling out new debdeploy release across the fleet
  • 11:02 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 19s)
  • 11:02 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided)
  • 11:02 ema: restarted varnish-be on cp4008
  • 11:02 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 04s)
  • 11:01 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided)
  • 11:01 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 03s)
  • 11:01 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided)
  • 11:00 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 04s)
  • 11:00 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided)
  • 09:55 bblack: restarted varnish-be on cp3042
  • 09:29 bblack: restarted varnish-be on cp1052
  • 09:23 ema: restarted varnish-be on cp3043
  • 09:21 ema: restarted varnish-be on cp3041
  • 09:19 ema: restarted varnish-be on cp3032
  • 09:14 ema: restarted varnish-be on cp3040
  • 08:42 bblack: restarted varnish-be on cp1053
  • 08:41 bblack: restarted varnish-be on cp1068
  • 08:40 bblack: restarted varnish-be on cp1054
  • 08:40 bblack: restarted varnish-be on cp1065
  • 07:11 ema: cp3030 varnish-be restart due to 503s
  • 03:16 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Sep 20 03:16:25 UTC 2017 (duration 7m 31s)
  • 03:08 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 15m 57s)
  • 02:31 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 08m 06s)
  • 00:55 bblack: cp1066 - backend restart, mailbox lag, text
  • 00:54 bblack: cp4016 - backend restart, mailbox lag, text

2017-09-19

  • 23:57 thcipriani@tin: Synchronized php-1.30.0-wmf.18/extensions/VisualEditor/ApiVisualEditor.php: SWAT: ApiVisualEditor: Fix checkbox label message handling with Message objects T176249 (duration: 00m 49s)
  • 23:56 thcipriani@tin: Synchronized php-1.30.0-wmf.19/extensions/VisualEditor/ApiVisualEditor.php: SWAT: ApiVisualEditor: Fix checkbox label message handling with Message objects T176249 (duration: 00m 48s)
  • 23:46 thcipriani@tin: Synchronized php-1.30.0-wmf.18/extensions/WikimediaEvents/modules/ext.wikimediaEvents.humanSearchRelevance.js: SWAT: Javascript timestamps are in ms, not s T174106 (duration: 00m 48s)
  • 23:45 thcipriani@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/modules/ext.wikimediaEvents.humanSearchRelevance.js: SWAT: Javascript timestamps are in ms, not s T174106 (duration: 00m 50s)
  • 23:39 thcipriani@tin: Synchronized php-1.30.0-wmf.18/includes/specials/SpecialRecentchangeslinked.php: SWAT: SpecialRecentchangeslinked: Unconditionally join on the page table T176228 (duration: 00m 49s)
  • 23:36 thcipriani@tin: Synchronized php-1.30.0-wmf.19/includes/specials/SpecialRecentchangeslinked.php: SWAT: SpecialRecentchangeslinked: Unconditionally join on the page table T176228 (duration: 00m 49s)
  • 23:29 thcipriani@tin: Synchronized php-1.30.0-wmf.18/includes/page/PageArchive.php: SWAT: Do not attempt to copy to ip_changes in PageArchive class (duration: 00m 49s)
  • 23:27 thcipriani@tin: Synchronized php-1.30.0-wmf.19/includes/page/PageArchive.php: SWAT: Do not attempt to copy to ip_changes in PageArchive class (duration: 00m 50s)
  • 23:17 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable jQuery 3 on meta.wikimedia.org T124742 (duration: 00m 50s)
  • 22:58 eileen: manually running Omnirecipient.load from 2017-09-18 with accidentally low throttle setting
  • 22:22 mutante: gerrit2001 - stopped gerrit with init.d script. moved init.d script to /root, started with systemctl
  • 22:21 eileen: update civicrm from 9677cf7 to 0fb4676
  • 22:15 mutante: gerrit2001 /bin/sh /var/lib/gerrit2/review_site/bin/gerrit.sh start
  • 22:02 mutante: gerrit2001 - systemctl start gerrit
  • 21:51 volans: disabled puppet on labpuppetmaster1001, testing cumin T175712
  • 21:27 MaxSem: on terbium: mwscript populateIpChanges.php --wiki=enwikivoyage --force
  • 21:23 bblack@neodymium: conftool action : set/pooled=yes; selector: cluster=cache_text,name=cp4008.ulsfo.wmnet
  • 21:23 bblack@neodymium: conftool action : set/pooled=yes; selector: cluster=cache_text,name=cp4016.ulsfo.wmnet
  • 21:22 bblack@neodymium: conftool action : set/pooled=yes; selector: cluster=cache_text,dc=eqiad,name=cp1066.*
  • 21:18 MaxSem: re-ran populateIpChanges.php on group0 wikis
  • 21:15 maxsem@tin: Synchronized php-1.30.0-wmf.19/maintenance/populateIpChanges.php: https://gerrit.wikimedia.org/r/#/c/379057/1 (duration: 00m 49s)
  • 21:06 ema: cp4009: varnish-be restart, 503 spike
  • 20:04 bblack@neodymium: conftool action : set/pooled=no; selector: cluster=cache_upload,name=cp4013.ulsfo.wmnet
  • 20:03 bblack@neodymium: conftool action : set/pooled=no; selector: cluster=cache_text,name=cp4016.ulsfo.wmnet
  • 19:50 bblack@neodymium: conftool action : set/pooled=no; selector: cluster=cache_upload,name=cp4005.ulsfo.wmnet
  • 19:49 bblack@neodymium: conftool action : set/pooled=no; selector: cluster=cache_text,name=cp4008.ulsfo.wmnet
  • 19:17 ejegg: updated payments-wiki from 26b16ea to 896f16a
  • 19:08 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.19
  • 18:52 gehel: upgrading nodejs to 6.11 on maps-test2004 for testing - T171707
  • 18:35 urandom: T160570, T169940: Upgrade restbase-ng environment to Cassandra 3.11.0-wmf5
  • 18:26 arlolra: Updated Parsoid to 05a0965 (T122965, T175101, T173384, T173061, T172896, T174977, T159894)
  • 18:17 ladsgroup@tin: Synchronized php-1.30.0-wmf.18/extensions/Wikidata/extensions/Wikibase/repo/tests/phpunit/includes/Content/EntityContentTest.php: Fix undoing merge operations that turned Items into a redirects, part II (T175984) (duration: 00m 48s)
  • 18:15 ladsgroup@tin: Synchronized php-1.30.0-wmf.18/extensions/Wikidata/extensions/Wikibase/repo/includes/Content/EntityContent.php: Fix undoing merge operations that turned Items into a redirects, part I (T175984) (duration: 00m 49s)
  • 18:08 ladsgroup@tin: Synchronized php-1.30.0-wmf.19/extensions/Wikidata/extensions/Wikibase/repo/tests/phpunit/includes/Content/EntityContentTest.php: Fix undoing merge operations that turned Items into a redirects, part II (T175984) (duration: 00m 50s)
  • 18:07 ladsgroup@tin: Synchronized php-1.30.0-wmf.19/extensions/Wikidata/extensions/Wikibase/repo/includes/Content/EntityContent.php: Fix undoing merge operations that turned Items into a redirects, part I (T175984) (duration: 00m 50s)
  • 18:00 arlolra@tin: Finished deploy [parsoid/deploy@a01064d]: Updating Parsoid to 05a0965 (duration: 12m 26s)
  • 17:54 ebernhardson: T173464 start cirrussearch reindex of all wikis starting with zh
  • 17:48 arlolra@tin: Started deploy [parsoid/deploy@a01064d]: Updating Parsoid to 05a0965
  • 17:43 jynus: stopping db2010 and cloning to db2078
  • 17:16 demon@tin: Finished scap: bootstrap wmf.19 (duration: 31m 04s)
  • 16:45 demon@tin: Started scap: bootstrap wmf.19
  • 16:44 demon@tin: scap aborted: bootstrap wmf.19 (duration: 01m 52s)
  • 16:42 demon@tin: Started scap: bootstrap wmf.19
  • 16:38 jynus@tin: Synchronized wmf-config/db-codfw.php: decom db1018 (duration: 00m 51s)
  • 16:37 jynus@tin: Synchronized wmf-config/db-eqiad.php: decom db1018 (duration: 01m 01s)
  • 16:34 demon@tin: Pruned MediaWiki: 1.30.0-wmf.17 [keeping static files] (duration: 01m 18s)
  • 16:31 demon@tin: Pruned MediaWiki: 1.30.0-wmf.14 (duration: 03m 15s)
  • 16:29 jynus: disable puppet on db1009
  • 16:27 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Trying that again, got an error the first time (duration: 00m 52s)
  • 16:26 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable RCFilters by default on cawiki, frwiki and hewiki (T157642) (duration: 00m 59s)
  • 16:19 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable structured filters on watchlist on all wikis (duration: 00m 45s)
  • 16:11 ema: pybal 1.14.0 uploaded to apt.w.o
  • 15:54 catrope@tin: Finished scap: core, GuidedTour and WikimediaMessages patches for T176191, T167262 and T175765 (duration: 20m 32s)
  • 15:33 catrope@tin: Started scap: core, GuidedTour and WikimediaMessages patches for T176191, T167262 and T175765
  • 15:22 jynus: enabling firewall on db1020 (m2-master)
  • 15:01 ema: restart varnish-be on cp1055, 503 spike
  • 14:55 jynus: deploying firewall to m5 hosts
  • 14:39 jynus: disabling puppet on all misc active mysql hosts
  • 13:35 zeljkof: EU SWAT finished
  • 13:32 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Rename Wikisaurus namespace on Wiktionary to "Thesaurus" (T174264) (duration: 00m 45s)
  • 13:26 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Rename Wikisaurus namespace on Wiktionary to "Thesaurus" (T174264) (duration: 02m 55s)
  • 13:19 gehel: upgrading elasticsearch plugins on relforge, including cold restart of the cluster - T173231
  • 13:17 gehel: upgrading elasticsearch plugins on elasticsearch codfw, including cold restart of the cluster - T173231
  • 13:15 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Rename Wikisaurus namespace on Wiktionary to "Thesaurus" (T174264) (duration: 02m 56s)
  • 13:04 gehel: upgrading elasticsearch plugins on elastic2001 - T173231
  • 12:55 moritzm: rebooting kubestage1001
  • 11:25 moritzm: uploaded debdeploy 0.99.1 to apt.wikimedia.org (for trusty, jessie and stretch)
  • 11:23 kartik@tin: Finished deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7 (duration: 02m 42s)
  • 11:20 kartik@tin: Started deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7
  • 10:53 volans: restarted nagios-nrpe-server on stat1005
  • 10:14 jynus: shutting down mysql on db1018
  • 09:42 moritzm: installing libxml2 security updates on trusty (Debian already fixed)
  • 09:40 elukey: powercycle analytics1062 - no ssh, console com2 frozen
  • 09:34 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: Enable the updateBetaFeaturesUserCounts logging channel - T175637 (duration: 00m 50s)
  • 09:30 ema: cp1062 - varnish-backend-restart, mbox lag
  • 08:40 mobrovac@tin: Finished deploy [cxserver/deploy@03c1eb1]: (no justification provided) (duration: 00m 52s)
  • 08:39 mobrovac@tin: Started deploy [cxserver/deploy@03c1eb1]: (no justification provided)
  • 08:09 mobrovac@tin: Finished deploy [cxserver/deploy@03c1eb1]: (no justification provided) (duration: 00m 13s)
  • 08:09 mobrovac@tin: Started deploy [cxserver/deploy@03c1eb1]: (no justification provided)
  • 07:58 kartik@tin: Finished deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7 (duration: 00m 37s)
  • 07:58 kartik@tin: Started deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7
  • 07:50 moritzm: installing bind9 regression update on trusty servers
  • 07:35 kartik@tin: (no justification provided)
  • 07:35 kartik@tin: Finished deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7 (duration: 01m 04s)
  • 07:34 kartik@tin: Started deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7
  • 02:28 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Sep 19 02:28:56 UTC 2017 (duration 7m 6s)
  • 02:21 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 07m 04s)
  • 01:52 demon@tin: Synchronized scap/plugins/updatewikiversions.py: Minor newline nitpick. I promise that's it (duration: 00m 46s)
  • 01:41 demon@tin: Synchronized scap/plugins/updatewikiversions.py: Minor spacing nitpick, that's it for today (duration: 00m 45s)
  • 01:17 demon@tin: Synchronized scap/plugins/updatewikiversions.py: add new hotness updatewikiversions.py (duration: 00m 45s)
  • 01:15 demon@tin: Synchronized multiversion/: rm old and busted updateWikiversions (duration: 01m 07s)
  • 00:12 foks: Remove 2FA from users Alan and Magicknight94 (retroactive log)
  • 00:10 eileen: update civicrm from 187238f to b6f7dc3

2017-09-18

  • 23:28 tstarling@tin: Synchronized php-1.30.0-wmf.18/includes/filerepo/file/LocalFile.php: (no justification provided) (duration: 00m 46s)
  • 23:13 thcipriani@tin: Synchronized php-1.30.0-wmf.18/extensions/ArticleCreationWorkflow: SWAT: Only intercept users who can potentially create articles otherwise T176100 (duration: 00m 46s)
  • 22:32 demon@tin: Synchronized wmf-config/CommonSettings.php: Use PrivateSettings.php directly (duration: 00m 46s)
  • 21:57 bawolff: deployed patch T175900
  • 21:18 krinkle@tin: Synchronized php-1.30.0-wmf.18/extensions/NavigationTiming/modules/ext.navigationTiming.js: Unbreak - T176105 (duration: 00m 48s)
  • 21:16 mutante: repeating import of gerrit_2.13.8+git1-wmf.7 on install1002, followed by reprepro copy stretch-wikimedia jessie-wikimedia gerrit to make it available on stretch
  • 21:06 mutante: uploaded gerrit 2.13.8+git1-wmf.7 for jessie-wikimedia on APT.wm.org
  • 20:42 krinkle@tin: Finished deploy [jobrunner/jobrunner@57f5f47]: No-op sync - first time scap3 - T129148 (duration: 03m 23s)
  • 20:38 krinkle@tin: Started deploy [jobrunner/jobrunner@57f5f47]: No-op sync - first time scap3 - T129148
  • 20:17 arlolra@tin: (no justification provided)
  • 20:15 arlolra@tin: Finished deploy [parsoid/deploy@a01064d]: Updating Parsoid to 05a0965 (duration: 01m 53s)
  • 20:13 arlolra@tin: Started deploy [parsoid/deploy@a01064d]: Updating Parsoid to 05a0965
  • 19:29 XioNoX: restarting pfw3-codfw for software upgrade
  • 19:10 urandom: T160570: Upgrading restbase-dev100[5-6].eqiad.wmnet to Cassandra 3.11.0-wmf5
  • 19:07 smalyshev@tin: Finished deploy [wdqs/wdqs@2523836]: Deploy correct prefixes conf (duration: 01m 20s)
  • 19:05 smalyshev@tin: Started deploy [wdqs/wdqs@2523836]: Deploy correct prefixes conf
  • 19:03 smalyshev@tin: Finished deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS, take 4 (duration: 02m 36s)
  • 19:02 no_justification: gerrit: restarting service, back in a moment
  • 19:00 smalyshev@tin: Started deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS, take 4
  • 18:56 urandom: T160570: Upgrading restbase-dev1004.eqiad.wmnet to Cassandra 3.11.0-wmf5 (canary)
  • 18:46 smalyshev@tin: Finished deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS, take 3 (duration: 02m 25s)
  • 18:44 smalyshev@tin: Started deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS, take 3
  • 18:37 smalyshev@tin: (no justification provided)
  • 18:36 smalyshev@tin: Finished deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS (duration: 02m 26s)
  • 18:34 smalyshev@tin: Started deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS
  • 18:20 maxsem@tin: Synchronized wmf-config/: https://gerrit.wikimedia.org/r/#/c/378357/ https://gerrit.wikimedia.org/r/#/c/376791/ (duration: 00m 48s)
  • 17:42 smalyshev@tin: Finished deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS (duration: 00m 21s)
  • 17:42 smalyshev@tin: Started deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS
  • 17:26 demon@tin: Synchronized wmf-config/ProductionServices.php: no-op/comment fix (duration: 00m 45s)
  • 16:46 jynus: shuting down db1100 T175973
  • 16:40 reedy@tin: Synchronized php-1.30.0-wmf.18/maintenance/populateIpChanges.php: perf improvements T175962 (duration: 00m 46s)
  • 16:05 bblack: cp1063 - backend restart, mailbox lag
  • 15:38 bblack: cp1072 - varnish backend restart, mailbox lag
  • 15:28 bblack: cp1074 - backend restart for mailbox lag
  • 15:25 bblack: cp1050 - backend restart for mailbox lahg
  • 14:27 reedy@tin: Synchronized php-1.30.0-wmf.18/includes/filerepo/file/LocalFile.php: T175444 (duration: 00m 47s)
  • 13:29 ppchelko@tin: Finished deploy [cpjobqueue/deploy@2c422f5]: Annotate the job event with pipeline property (duration: 00m 29s)
  • 13:28 ppchelko@tin: Started deploy [cpjobqueue/deploy@2c422f5]: Annotate the job event with pipeline property
  • 13:25 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Meta(Talk)Namespace configuration for be.wiktionary - T175950 (duration: 00m 46s)
  • 13:18 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: pagePreviews: Stop A/B test on enwiki and dewiki - T176068 (duration: 00m 45s)
  • 13:10 hashar@tin: Synchronized php-1.30.0-wmf.18/extensions/BetaFeatures: Temporary log all the executions of the update job. - T175637 (duration: 00m 46s)
  • 13:09 hashar@tin: Synchronized wmf-config: New 'abusefilter-helper' configuration for en.wikipedia - T175684 (duration: 00m 49s)
  • 12:59 moritzm: installing libxen updates
  • 12:36 moritzm: installing tomcat8 security updates
  • 10:40 ema: cp1099 - backend restart, mailbox lag
  • 06:42 moritzm: installing freexl security updates
  • 06:18 TimStarling: testing EtcdConfig in beta cluster
  • 05:40 tstarling@tin: Synchronized wmf-config: just for consistency, should have no effect (gerrit 375108) (duration: 00m 49s)
  • 02:36 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Sep 18 02:36:50 UTC 2017 (duration 7m 6s)
  • 02:29 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 08m 04s)

2017-09-17

  • 12:49 _joe_: taking a full debug dump of mw1288 after depooling it
  • 12:40 _joe_: restarting hhvm on some api appservers, to ease the cpu overlock
  • 09:46 elukey: restart varnish backend on cp1052 - recurrent mailbox lag
  • 08:11 elukey: restart varnish backend on cp1053 - recurrent mailbox lag

2017-09-16

  • 22:05 godog: compress older otrs directories to reclaim inodes - T171490
  • 16:53 elukey: restart varnish-backend on cp1073 (cache upload) for mailbox lag
  • 15:40 andrewbogott: rebooting labvirt1017
  • 15:34 andrewbogott: rebooting labvirt1015 for T176044

2017-09-15

  • 21:50 mutante: contint2001 - systemctl start zuul
  • 21:47 mutante: contint1001 - tmp disable puppet - contint2001 - test zuul unit file (gerrit 359016)
  • 21:43 demon@tin: Finished deploy [gerrit/gerrit@48ad332]: 2.13.9 -- not being used yet though (duration: 00m 08s)
  • 21:43 demon@tin: Started deploy [gerrit/gerrit@48ad332]: 2.13.9 -- not being used yet though
  • 15:46 bblack: cp1049 - backend restart, mailbox lag
  • 15:40 bblack: cp1099 - backend restart, mailbox lag
  • 15:38 bblack: cp1072 - backend restart, mailbox lag
  • 15:21 _joe_: uploaded td-agent-bit (fluentd-bit) to wikimedia-stretch T175527
  • 13:13 moritzm: pruned hhvm bytecode cache on mw2256, led to log spam for an undefined variable after being out of service for hardware maintenance
  • 13:10 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw2256.codfw.wmnet
  • 13:10 addshore@tin: Synchronized wmf-config/CommonSettings-labs.php: beta: $wgWikiDiff2MovedParagraphDetectionCutoff = 25 BETA ONLY 2/2 (duration: 00m 45s)
  • 13:08 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: beta: $wgWikiDiff2MovedParagraphDetectionCutoff = 25 BETA ONLY 1/2 (duration: 00m 47s)
  • 11:22 jmm@puppetmaster1001: conftool action : set/pooled=no; selector: mw2256.codfw.wmnet
  • 09:10 moritzm: installing tcpdump security updates on trusty hosts (Debian systems already fixed)
  • 08:54 moritzm: installing libbluetooth updates
  • 08:03 jynus: dropping mariadb analytics users on db1031, db1029
  • 07:49 Amir1: running maintenance/updateCollation.php --force on Persian (fa) wikis (T173601)
  • 07:39 gehel: shutting down and masking elasticsearch on elastic1020 - T175951
  • 07:35 gehel: depooling elastic1020 - T175951
  • 07:28 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw2256.codfw.wmnet
  • 00:46 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1100 (duration: 00m 46s)

2017-09-14

  • 23:51 mutante: phabricator - tested gerrit 354247 before merging using apache-fast-test from tin -restarted apache
  • 23:37 catrope@tin: Synchronized wmf-config/: VRS config for Electron (T175868) (duration: 00m 47s)
  • 23:34 catrope@tin: Synchronized php-1.30.0-wmf.18/extensions/VisualEditor/lib/ve/src/ui/styles/ve.ui.TableLineContext.css: T169389 (duration: 00m 45s)
  • 23:33 catrope@tin: Synchronized php-1.30.0-wmf.18/extensions/ContentTranslation/modules/widgets/translator/ext.cx.translator.js: Fix JS error in CX (duration: 00m 46s)
  • 23:29 catrope@tin: Synchronized wmf-config/CommonSettings.php: Give sysops the flow-create-board right on Flow wikis (T175934) (duration: 00m 45s)
  • 23:27 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Give sysops the flow-create-board right on Flow wikis (T175934) (duration: 00m 45s)
  • 23:22 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Use RemexHtml on mediawikiwiki and testwiki (T175095) (duration: 00m 46s)
  • 23:19 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=acamar.wikimedia.org
  • 23:18 mutante: acamar is back up and running - no failed units
  • 23:13 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Allow bureaucrats to remove accountcreator on mediawikiwiki (T175903) (duration: 00m 46s)
  • 23:12 mutante: acamar - staged BIOS upgrade in progress
  • 23:09 mutante: acamar - done with upgrade - rebooting - it's depooled and removed from resolv.conf - T162850
  • 22:56 mutante: depooled acamar (codfw dns recursor) for BIOS upgrade - installing firmware
  • 22:56 dzahn@neodymium: conftool action : set/pooled=no; selector: name=acamar.*
  • 22:55 maxsem@tin: Synchronized wmf-config/: Labs only: https://gerrit.wikimedia.org/r/378184 https://gerrit.wikimedia.org/r/378187 (duration: 00m 47s)
  • 22:15 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: ACTRIAL going live: https://gerrit.wikimedia.org/r/#/c/378104/ (duration: 00m 46s)
  • 21:22 maxsem@tin: Synchronized php-1.30.0-wmf.18/extensions/ArticleCreationWorkflow/: https://gerrit.wikimedia.org/r/#/c/378125/ (duration: 00m 47s)
  • 20:34 demon@tin: Synchronized php-1.30.0-wmf.18/skins/MinervaNeue/resources/skins.minerva.userpage.icons/userpage.svg: I95d76339 (duration: 00m 45s)
  • 20:18 demon@tin: Synchronized php-1.30.0-wmf.18/includes/api/ApiFeedWatchlist.php: Ibea5bd88 (duration: 00m 46s)
  • 20:04 demon@tin: Synchronized php-1.30.0-wmf.18/extensions/OAuth/backend/MWOAuthDAO.php: I785230df (duration: 00m 46s)
  • 20:02 ppchelko@tin: Finished deploy [cpjobqueue/deploy@af9b590]: Emit broker statistics more frequentl (duration: 00m 30s)
  • 20:02 ppchelko@tin: Started deploy [cpjobqueue/deploy@af9b590]: Emit broker statistics more frequentl
  • 19:58 dcausse: banning elastic1020 to see if T175951 is caused by mixed versions of the ltr plugin
  • 19:36 demon@tin: Synchronized php-1.30.0-wmf.18/skins/MinervaNeue/resources/skins.minerva.icons.images.scripts/userNormal.svg: fix svg syntax (duration: 00m 46s)
  • 19:31 demon@tin: Synchronized php-1.30.0-wmf.17/skins/MinervaNeue/resources/skins.minerva.icons.images.scripts/userNormal.svg: fix svg syntax (duration: 00m 45s)
  • 19:07 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.18
  • 18:58 tgr@tin: Synchronized wmf-config/PrivateSettings.php: set Electron secret for T175868 (duration: 00m 48s)
  • 18:57 tgr@tin: Synchronized private/PrivateSettings.php: set Electron secret for T175868 (duration: 00m 49s)
  • 18:29 twentyafterfour: Phabricator: Deploying hotfix for T175942
  • 18:10 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Configure enwiki to use CirrusSearch MLR by default T175772 (duration: 00m 50s)
  • 18:03 Jamesofur: disabled oauth validation on metawiki/SUL for Teles after verification
  • 17:57 bblack: cp2001 backend restart, mailbox lag
  • 17:56 bblack: cp1074 backend restart, mailbox lag
  • 17:18 eevans@tin: Started restart [electron-render/deploy@8dd5f13]: (no justification provided)
  • 17:16 gehel@tin: Finished deploy [wikimedia/discovery/analytics@ab5d5c1]: moving discovery/analytics to scap3 (duration: 00m 04s)
  • 17:16 gehel@tin: Started deploy [wikimedia/discovery/analytics@ab5d5c1]: moving discovery/analytics to scap3
  • 16:59 krinkle@tin: Synchronized php-1.30.0-wmf.17/extensions/NavigationTiming: T104902 (duration: 01m 00s)
  • 15:51 bblack: repool cp1063 (after wiping varnish storage + restarting all 3x main service daemons)
  • 15:51 bblack: cp1064 backend restart, mailbox lag
  • 15:46 bblack: depool cp1063 (upload eqiad) - seems to be having some iowait issues?
  • 14:07 dereckson@tin: Synchronized php-1.30.0-wmf.18/extensions/CheckUser/specials/SpecialCheckUser.php: Fix Special cu for ip ranges (temp) (T175898) (duration: 00m 49s)
  • 13:34 dcausse@tin: Synchronized wmf-config/CirrusSearch-production.php: Setup Cirrus MLR models for top 20 language AB test (duration: 00m 49s)
  • 13:33 dcausse@tin: Synchronized wmf-config/CirrusSearch-common.php: Setup Cirrus MLR models for top 20 language AB test (duration: 00m 48s)
  • 13:31 dcausse@tin: Synchronized wmf-config/InitialiseSettings.php: Setup Cirrus MLR models for top 20 language AB test (duration: 00m 50s)
  • 11:14 jynus@tin: Synchronized wmf-config/db-eqiad.php: Remove all references to db1049, pool db1100 (duration: 00m 49s)
  • 11:12 akosiaris: upload apertium-crh-tur_0.2.0~r81866-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 11:12 jynus@tin: Synchronized wmf-config/db-codfw.php: Remove all references to db1049 (duration: 00m 50s)
  • 09:51 akosiaris@puppetmaster1001: conftool action : set/pooled=no; selector: name=wtp1025.eqiad.wmnet
  • 09:43 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
  • 09:43 akosiaris@puppetmaster1001: conftool action : set/weight=1; selector: name=wtp1025.eqiad.wmnet
  • 09:41 gehel: re-enabling puppet on all cassandra nodes - T171704
  • 09:41 akosiaris@tin: Finished deploy [parsoid/deploy@cec7d17]: test deploy using dsh groups. T165520 (duration: 09m 30s)
  • 09:32 akosiaris@tin: Started deploy [parsoid/deploy@cec7d17]: test deploy using dsh groups. T165520
  • 09:02 akosiaris@tin: Finished deploy [parsoid/deploy@cec7d17]: test deploy using dsh groups. T165520 (duration: 02m 25s)
  • 09:00 akosiaris@tin: Started deploy [parsoid/deploy@cec7d17]: test deploy using dsh groups. T165520
  • 08:13 gehel: merging cassandra refactoring for puppet - T171704
  • 02:55 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Sep 14 02:55:10 UTC 2017 (duration 7m 28s)
  • 02:47 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 07m 58s)
  • 02:27 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 09m 27s)
  • 00:46 MaxSem: restarted populateIpChanges to use updated code
  • 00:26 twentyafterfour: everything appears to be up and running. Phabricator upgrade complete.
  • 00:23 twentyafterfour: running puppet on phab1001 to restore files which get lost by scap deploy
  • 00:21 twentyafterfour@tin: Finished deploy [phabricator/deployment@7e3a0a8]: (no justification provided) (duration: 00m 01s)
  • 00:21 twentyafterfour@tin: Started deploy [phabricator/deployment@7e3a0a8]: (no justification provided)
  • 00:21 twentyafterfour@tin: Finished deploy [phabricator/deployment@7e3a0a8]: (no justification provided) (duration: 00m 25s)
  • 00:20 twentyafterfour@tin: Started deploy [phabricator/deployment@7e3a0a8]: (no justification provided)
  • 00:18 twentyafterfour: Begin phabricator upgrade #phab-2017-09-13

2017-09-13

  • 23:48 dereckson@tin: Synchronized php-1.30.0-wmf.18/maintenance/populateIpChanges.php: Improve ip_changes update script to insert rows in batches (Gerrit:377918) (duration: 00m 49s)
  • 23:29 dereckson@tin: Synchronized php-1.30.0-wmf.18/extensions/Wikidata/extensions/Wikibase/client: Split page set before constructing InjectRCRecordsJob (T175316) (duration: 00m 57s)
  • 23:28 dereckson@tin: Synchronized php-1.30.0-wmf.18/extensions/Echo/modules/api/mw.echo.api.APIHandler.js: (no justification provided) (duration: 00m 48s)
  • 23:26 dereckson@tin: Synchronized wmf-config/CommonSettings.php: Set $wgScoreSafeMode to false (T174413) (duration: 00m 50s)
  • 22:06 maxsem@tin: Synchronized php-1.30.0-wmf.18/extensions/EventBus/: https://gerrit.wikimedia.org/r/#/c/377900/1 (duration: 00m 49s)
  • 22:03 maxsem@tin: Synchronized php-1.30.0-wmf.18/extensions/ArticleCreationWorkflow/: https://gerrit.wikimedia.org/r/#/c/377900/1 (duration: 00m 49s)
  • 21:55 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/377874/ (duration: 00m 47s)
  • 21:23 ejegg: updated payments-wiki from ed2e481 to 26b16ea
  • grrrr: populating ip_changes on group1 wikis
  • 20:43 bblack: re-routed ulsfo->eqiad around possible codfw damage as well - https://gerrit.wikimedia.org/r/#/c/377871/
  • 20:34 bblack: deployed https://gerrit.wikimedia.org/r/#/c/377845/ (no gerrit bot, jenkins lint failing, etc) due to possible codfw routing issues
  • 19:39 andrewbogott: rebooting labtestvirt2001
  • 19:35 ppchelko@tin: Finished deploy [changeprop/deploy@86e8103]: Separate kafka metrics by rule (duration: 01m 18s)
  • 19:34 ppchelko@tin: Started deploy [changeprop/deploy@86e8103]: Separate kafka metrics by rule
  • 19:21 urandom: T172384: Upgrading restbase-dev100[5-6] to Cassandra 3.11.0-wmf4
  • 19:19 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.18
  • 19:13 demon@tin: Synchronized php: symlink thingie (duration: 00m 49s)
  • 19:11 urandom: T172384: Upgrading restbase-dev1004 to Cassandra 3.11.0-wmf4 (canary)
  • 18:34 niharika29@tin: Synchronized static/images/project-logos/: Change logo for huwiktionary T175483 (duration: 00m 49s)
  • 18:27 mutante: gerrit2002 - re-enabled puppet - had forgotten to enable it again after recently tryning to test the gerrit->logstash change
  • 18:26 niharika29@tin: Synchronized php-1.30.0-wmf.18/extensions/VisualEditor/: Make ve.dm.Change part of core module T175828 (duration: 00m 50s)
  • 18:25 niharika29@tin: Synchronized docroot/mediawiki/ontology/: Add setup for https://www.mediawiki.org/ontology T171807 (duration: 00m 49s)
  • 18:12 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable Timeless on frwiki T154371 (duration: 00m 50s)
  • 17:55 gwicke: rolling restart of pdfrender service in equiad after hang T174916 T172815
  • 16:42 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=wdqs1001.eqiad.wmnet
  • 16:42 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=wdqs1002.eqiad.wmnet
  • 16:28 gehel: starting decommissioning of wdqs100[12] - T175595
  • 16:25 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs1002.eqiad.wmnet
  • 16:25 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs1001.eqiad.wmnet
  • 16:22 andrewbogott: rebooting labtestvirt2002
  • 16:08 ottomata: restarting druid-brokers with increase in query cache size
  • 15:23 demon@tin: Synchronized php-1.30.0-wmf.18/includes/filerepo/file/LocalFile.php: I2ecb6030 (duration: 00m 49s)
  • 15:22 demon@tin: Synchronized php-1.30.0-wmf.18/includes/diff/DifferenceEngine.php: I2ecb6030 (duration: 00m 49s)
  • 15:21 demon@tin: Synchronized php-1.30.0-wmf.17/includes/filerepo/file/LocalFile.php: I2ecb6030 (duration: 00m 49s)
  • 15:20 demon@tin: Synchronized php-1.30.0-wmf.17/includes/diff/DifferenceEngine.php: I2ecb6030 (duration: 00m 49s)
  • 15:09 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/AbuseFilter/: backporting I13051038 (duration: 00m 52s)
  • 14:31 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,cluster=logstash
  • 14:31 gehel: pooling new logstash servers (logstash100[7-9]) - T175045
  • 14:23 mobrovac@tin: Finished deploy [cpjobqueue/deploy@60d0a78]: Start using the EventBus infrastructure for the updateBetaFeaturesUserCounts job - T175210 (duration: 00m 33s)
  • 14:22 mobrovac@tin: Started deploy [cpjobqueue/deploy@60d0a78]: Start using the EventBus infrastructure for the updateBetaFeaturesUserCounts job - T175210
  • 13:45 akosiaris: upload apertium-crh_0.1.0~r81872-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 13:30 urandom: T169936: Converting RESTBase dev environment to size-tiered compaction
  • 13:29 elukey: update puppet compiler's facts via ./modules/puppet_compiler/files/compiler-update-facts
  • 13:23 gehel: initial installation of logstash100[7-9] - T175045
  • 13:21 zeljkof: EU SWAT finished
  • 13:13 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: Temporary lift account creation limits for WM United Kingdom workshop (T175700) (duration: 00m 49s)
  • 12:14 moritzm: rebooting mc1015 (spare) for some tests
  • 12:07 jynus: stopping wikidata rebuild maintenance script on terbium
  • 12:07 bblack: cp1049 - backend restart, mailbox lag
  • 11:57 jynus@tin: Synchronized wmf-config/db-eqiad.php: Set db1097 as the main api server for s4, Remove regular load from db1070 (duration: 00m 56s)
  • 11:39 jynus: disabling semi-sync replication on the largest s5 database servers
  • 11:03 ariel@tin: Finished deploy [dumps/dumps@3f119ab]: monitor will cd to dumps repo before each run (duration: 00m 02s)
  • 11:03 ariel@tin: Started deploy [dumps/dumps@3f119ab]: monitor will cd to dumps repo before each run
  • 11:02 joal@tin: Finished deploy [analytics/refinery@b2e8852]: Regular analytics weekly deploy (new jars, oozie patches) (duration: 03m 12s)
  • 10:58 joal@tin: Started deploy [analytics/refinery@b2e8852]: Regular analytics weekly deploy (new jars, oozie patches)
  • 10:52 volans: reboot lvs3001
  • 10:06 akosiaris: upload apertium-cat-srd_0.9.0~r82238-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 10:06 akosiaris: upload apertium-srd-ita_0.9.5~r82237-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 09:36 moritzm: installing bind updates from Debian stable/oldstable SUA update (client tools only)
  • 09:25 akosiaris: upload apertium-srd_0.10.0~r82237-1+wmf1_amd64.changes to apt.wikimedia.org/jessie-wikimedia/main T174988
  • 09:24 akosiaris: upload apertium-tur_0.1.0~r81882-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 09:23 akosiaris: upload apertium-ita_0.10.0~r82237-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 09:23 akosiaris: upload apertium-cat_2.3.0~r82237-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 09:05 akosiaris: upload apertium_3.4.2~r68466-2+wmf2 to apt.wikimedia.org/jessie-wikimedia
  • 08:55 elukey: restart varnish backend on cp1072 (upload) - mailbox expiry lag
  • 08:24 moritzm: installing emacs security updates
  • 07:56 akosiaris: bounce varnish on cp1062, mailbox lag
  • 07:47 moritzm: rolling out hhvm-luasandbox 2.0.14 to the remaining hosts in eqiad (along with HHVM restarts) (T173705)
  • 07:08 moritzm: installing tcpdump security updates (fixing 90 CVE IDs...)
  • 05:18 kartik@tin: Finished deploy [cxserver/deploy@ee0081d]: Update cxserver to 3baaa36 (duration: 02m 56s)
  • 05:15 kartik@tin: Started deploy [cxserver/deploy@ee0081d]: Update cxserver to 3baaa36
  • 03:12 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Sep 13 03:12:46 UTC 2017 (duration 7m 30s)
  • 03:05 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 16m 28s)
  • 03:03 jynus: dropping old users from phabricator database
  • 03:00 eileen: update civicrm from ee7dda3 to 187238f
  • 02:41 demon@tin: Synchronized wmf-config/mobile.php: rm unused config var (duration: 00m 46s)
  • 02:27 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 08m 31s)
  • 01:19 jynus: stopping mariadb @ db1048 to clone it to db1059
  • 00:11 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Special:PageLanguage on mul.wikisource Fix correct wiki name for wgPageLanguageUseDB on www.wikisource.org T175622 (duration: 00m 49s)

2017-09-12

  • 23:53 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Do not deploy Timeless on fr.wiktionary for now" (duration: 00m 49s)
  • 23:35 thcipriani@tin: Synchronized wmf-config: SWAT: Enable Flow on Newsletter_talk on mediawiki.org T175502 (duration: 00m 52s)
  • 22:17 chasemp: labvirt1018:~# /sbin/reboot
  • 22:00 bsitzmann@tin: Finished deploy [mobileapps/deploy@b11b75c]: Update mobileapps to 297b048 (T174707 T175305) (duration: 04m 28s)
  • 21:55 bsitzmann@tin: Started deploy [mobileapps/deploy@b11b75c]: Update mobileapps to 297b048 (T174707 T175305)
  • 21:49 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/377527/2 T175725 (duration: 00m 49s)
  • 21:37 MaxSem: Populating ip_changes in group0 wikis
  • 21:33 ejegg: changed contact de-duplication duty cycle to 25 minutes
  • 21:05 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.18
  • 20:33 demon@tin: Finished scap: bootstrap wmf.18 (for real this time) (duration: 26m 23s)
  • 20:06 demon@tin: Started scap: bootstrap wmf.18 (for real this time)
  • 19:53 demon@tin: Finished scap: bootstrap wmf.18 (duration: 08m 14s)
  • 19:45 demon@tin: Started scap: bootstrap wmf.18
  • 19:05 papaul: maintenance complete on mw2256
  • 17:51 bblack: cp3034 - reset fetch_maxchunksize to 0.25G (default) for fe+be
  • 17:40 cwd: turned on dedupe
  • 16:55 niharika29@tin: Finished deploy [iegreview/iegreview@69c4c3f]: Deploying iegreview with scap3 T129154 (duration: 00m 01s)
  • 16:55 niharika29@tin: Started deploy [iegreview/iegreview@69c4c3f]: Deploying iegreview with scap3 T129154
  • 16:40 ejegg: updated CiviCRM from 63288ae to ee7dda3
  • 16:35 niharika29@tin: Finished deploy [scholarships/scholarships@004635d]: Deploying scholarships with scap3 T129134 (duration: 00m 03s)
  • 16:35 niharika29@tin: Started deploy [scholarships/scholarships@004635d]: Deploying scholarships with scap3 T129134
  • 16:26 volans@sarin: conftool action : set/pooled=yes; selector: name=mw2100.codfw.wmnet
  • 16:18 ejegg: updated SmashPig from dce4f0a to dc62203910
  • 15:43 _joe_: restarted ircecho on einsteinium
  • 15:35 gehel: restart elasticsearch on relforge (last check of new plugin deployment) - T158560
  • 15:22 bblack: manually setting fetch_maxchunksize=16MB via varnishadm on cp3034 fe+be instances
  • 15:06 chasemp: disable puppet for lab* things for designate refactor rolleout (delayed a few minutes for a meeting)
  • 15:04 moritzm: uploaded php5.5 5.5.38 to component/ci of jessie-wikimedia (for use in CI)
  • 14:53 papaul: shutting down mw2256 for maintenance
  • 14:43 elukey: add kafka-jumbo IPs to the kafka term of the analytics-in4 filter on cr1/cr2 eqiad
  • 14:42 ema: cp1074 - backend restart, mailbox lag
  • 14:39 bblack: manually setting fetch_maxchunksize=64MB via varnishadm on cp3034 fe+be instances
  • 14:19 jynus@tin: Synchronized wmf-config/db-eqiad.php: Remove all references to db1059 (duration: 00m 44s)
  • 14:18 jynus@tin: Synchronized wmf-config/db-codfw.php: Remove all references to db1059 (duration: 00m 45s)
  • 14:10 gehel: restarting elasticsearch on elastic1020 to validate new plugin deployment - T158560
  • 14:07 gehel: switching elasticsearch plugin deployment to .deb instead of scap on cirrus/eqiad - T158560
  • 14:06 volans: testing reimage of mw2100 - T166300
  • 14:06 gehel: restarting elasticsearch on elastic2001 to validate new plugin deployment - T158560
  • 14:04 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1059, pool db1097 with low load (duration: 00m 45s)
  • 14:02 gehel: switching elasticsearch plugin deployment to .deb instead of scap on cirrus/codfw - T158560
  • 13:37 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs1005.eqiad.wmnet
  • 13:37 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs1004.eqiad.wmnet
  • 13:37 gehel: pooling wdqs100[45] - T171210
  • 13:31 hashar@tin: Synchronized php-1.30.0-wmf.17/extensions/CentralAuth/includes/api/ApiSetGlobalAccountStatus.php: API: Unbreak setglobalaccountstatus locked=lock - T175462 (duration: 00m 44s)
  • 13:26 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable WikidataPageBanner for Russian Wikimedia chapter wiki - T175356 (duration: 00m 45s)
  • 13:25 hashar@tin: Synchronized php-1.30.0-wmf.17/includes/changes/ChangesListBooleanFilter.php: WLFilters: Respect default values - T174725 (duration: 00m 45s)
  • 13:21 hashar@tin: Synchronized wmf-config/CommonSettings.php: Add Extension:Newsletter permissions to CommonSettings (duration: 00m 45s)
  • 13:18 hashar@tin: Synchronized wmf-config/throttle.php: Lift account creation restrictions for WM Taiwan 10th anniversary - T175534 (duration: 00m 45s)
  • 13:15 hashar@tin: Synchronized php-1.30.0-wmf.17/extensions/Wikidata/extensions/Wikibase/view/resources/jquery/wikibase/themes/default/jquery.wikibase.entityselector.css: Remove red highlighting on all entity selectors - T175525 (duration: 00m 45s)
  • 13:12 hashar@tin: Synchronized wmf-config/Wikibase-production.php: Reduce wikiPageUpdaterDbBatchSize to 20 - T173710 (duration: 00m 45s)
  • 09:33 moritzm: upgrading deployment servers to HHVM-Luasandbox 2.0.14
  • 09:32 moritzm: upgrading mw2* to HHVM-Luasandbox 2.0.14
  • 09:25 moritzm: upgrading mw1293-mw1295 (image scalers) to HHVM-Luasandbox 2.0.14 (along with HHVM restarts)
  • 09:24 moritzm: upgrading mw1293-mw1205 (image scalers) to HHVM-Luasandbox 2.0.14 (along with HHVM restarts)
  • 08:38 gehel: deploying elasticsearch plugins on relforge - T158560 (wrong ticket number before)
  • 08:34 gehel: deploying elasticsearch plugins on relforge - T175159
  • 08:28 moritzm: upgrading mw1189-mw1208 (API servers) to HHVM-Luasandbox 2.0.14 (along with HHVM restarts)
  • 07:35 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool es1019 with full weight (duration: 00m 45s)
  • 07:15 moritzm: upgrading mw1180-mw1188, mw1209-1220 (app servers) to HHVM-Luasandbox 2.0.14 (along with HHVM restarts)
  • 02:29 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Sep 12 02:29:20 UTC 2017 (duration 6m 45s)
  • 02:22 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 07m 31s)
  • 02:10 bblack: cp1050 - backend restart, mailbox lag
  • 02:08 bblack: cp1099 - backend restart, mailbox lagh
  • 00:48 awight@tin: Finished deploy [ores/deploy@42c5663]: Cause ORES service restart (duration: 00m 18s)
  • 00:48 awight@tin: Started deploy [ores/deploy@42c5663]: Cause ORES service restart

2017-09-11

  • 23:06 mutante: jenkins-bot says: java.lang.OutofMemoryError: PermGen space
  • 22:30 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4026.*
  • 22:30 dereckson@tin: Synchronized wmf-config/CommonSettings-labs.php: Fix MediaWiki\Auth\AbstractPreAuthenticationProvider case (Gerrit:377281, no-op in prod) (duration: 00m 45s)
  • 21:45 awight@tin: Finished deploy [ores/deploy@42c5663]: Try ORES filehandle fix on new cluster (duration: 13m 47s)
  • 21:41 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4025.*
  • 21:31 awight@tin: Started deploy [ores/deploy@42c5663]: Try ORES filehandle fix on new cluster
  • 21:03 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4023.*
  • 20:13 demon@tin: Pruned MediaWiki: 1.30.0-wmf.13 (duration: 02m 49s)
  • 19:11 chasemp: adjust nodepool rate to 10
  • 19:08 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool es1019 with low load (duration: 00m 46s)
  • 19:01 mutante: restbase-dev1004:/srv# mv /home/eevans/cassandra-1502141526-pid13410.hprof /srv/eevans/ to free disk space
  • 18:52 mutante: restbase - java[12418]: Error: Could not find or load main class org.wikimedia.cassandra.metrics.service.Service
  • 18:40 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable responsive reference columns on wiktionaries, wikivoyages and enwiki https://gerrit.wikimedia.org/r/#/c/376573/, https://gerrit.wikimedia.org/r/#/c/371630/ (duration: 00m 46s)
  • 18:33 robh: firmware update on scs-a1-codfw and scs-c1-codfw has been complete and tested good T174475
  • 18:29 chasemp: bump nodepool rate to 15 and max-servers to 18 (no restart as it picks up config periodically)
  • 18:29 robh: firmware update on scs-a1-codfw and scs-c1-codfw has been started, awaiting completion T174475
  • 18:24 robh: updating firmware on scs-a1-codfw and scs-c1-codfw
  • 18:10 robh: scs-ulsfo has successfully updated firmware T174475
  • 18:03 robh: upgrading firmware on scs-ulsfo
  • 17:09 gehel@tin: Finished deploy [wdqs/wdqs@177d20a]: (no justification provided) (duration: 02m 39s)
  • 17:07 gehel: wdqs weekly deployment (logging and throttling fixes)
  • 17:06 gehel@tin: Started deploy [wdqs/wdqs@177d20a]: (no justification provided)
  • 15:20 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4022.*
  • 15:20 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4028.*
  • 14:40 godog: roll-restart thumbor to apply https://gerrit.wikimedia.org/r/#/c/377264/ and upgrade to 1.4 - T173580 T174997
  • 14:24 hashar: tin.eqiad.wmnet: reverted patch "WLFilters: Respect default values" that got merged but left undeployed
  • 14:23 hashar: tin.eqiad.wmnet: reverted all four mediawiki-config that got merged but left undeployed and rebased the workspace
  • 14:05 hashar: European SWAT cancelled due to deploy infrastructure issue
  • 13:55 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 27s)
  • 13:54 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 27s)
  • 13:50 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 28s)
  • 13:43 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 28s)
  • 13:40 hashar@tin: Synchronized static/images/project-logos/huwiktionary.png: Change logo for huwiktonary - T175483 (duration: 00m 28s)
  • 11:43 moritzm: installing file/libmagic security updates (stretch-specific, does not affect older distros)
  • 10:30 jynus: restarting mysql @ labsdb1001
  • 09:40 godog: roll-restart thumbor to apply changes for gif max animated area - T173580
  • 09:39 mobrovac@tin: Finished deploy [cpjobqueue/deploy@e73a10f]: Initial deploy of cpjobqueue - T175281 (duration: 00m 20s)
  • 09:39 mobrovac@tin: Started deploy [cpjobqueue/deploy@e73a10f]: Initial deploy of cpjobqueue - T175281
  • 08:44 mobrovac@tin: Finished deploy [zotero/translators@a0c41c3]: Update translators to upstream e03695273 - T174992 (duration: 00m 07s)
  • 08:44 mobrovac@tin: Started deploy [zotero/translators@a0c41c3]: Update translators to upstream e03695273 - T174992
  • 08:16 volans: reimage script tests continue on mc100[1-2] T166300
  • 07:59 moritzm: installing systemd updates from stretch point update
  • 07:41 moritzm: installing remaining libonig security updates
  • 02:41 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Sep 11 02:41:43 UTC 2017 (duration 7m 0s)
  • 02:34 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 08m 10s)

2017-09-10

  • 21:00 bawolff: disabled Jforrester's gerrit account and flush sessions
  • 13:41 elukey: restart Varnish backend on cp1073 (cache::upload) for mailbox expiry lag
  • 13:13 elukey: restart cp1053's varnish backend for mailbox expiry lag and 503s - T175473
  • 00:37 chasemp: labnodepool1001:~# service nodepool restart
  • 00:36 bd808: wmcs: purged rabbitmq ceilometer_notifications.* queues

2017-09-09

  • 23:42 chasemp: labnodepool1001:~# service nodepool start
  • 23:06 chasemp: freeswap && labcontrol1001:/home/rush# service rabbitmq-server restart
  • 22:59 chasemp: restart nova-api and nova-network
  • 22:49 chasemp: labnodepool1001:~# service nodepool stop
  • 22:44 madhuvishy: Running service rabbitmq-server restart on labcontrol1001
  • 20:41 legoktm@tin: Synchronized php-1.30.0-wmf.17/includes/filerepo/file/LocalFile.php: Fix issues related to comments and files in UI and API - T175443 T175444 (duration: 00m 46s)
  • 01:12 ebernhardson@tin: Synchronized php-1.30.0-wmf.17/extensions/WikimediaEvents/modules/ext.wikimediaEvents.humanSearchRelevance.js: T171740: Reduce annoyance of survey by enforcing minimum 2 days between showing survey to same browser (duration: 00m 46s)

2017-09-08

  • 23:46 ebernhardson@tin: Synchronized wmf-config/CirrusSearch-rel-survey.php: T171740: Fix inverted sampling rates for human relevance survey (duration: 00m 47s)
  • 23:38 mutante: netmon1002 - terminated unused screen session
  • 23:09 mutante: phab1001, phab2001 - terminating 3 unused screen processes that were from iridium migration / data sync
  • 21:54 mutante: gerrit2001 - restarting gerrit to test logstash config change (while not touching "live" gerrit)
  • 20:24 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/Flow/includes/: bugs and stuff (duration: 00m 54s)
  • 20:14 mutante: heze - rebooting for firmware upgrade
  • 20:10 mutante: heze (bacula storage) - installing BIOS upgrade (T162850)
  • 20:10 demon@tin: Synchronized scap/scap.cfg: drop git_repo for now, T175041 (duration: 00m 46s)
  • 19:43 mutante: zosma.codfw.wmnet - delete salt key, puppet node clean, puppet node deactivate, remove from Icinga,... (T138650)
  • 19:37 mutante: removing ganeti instance 'zosma' on ganeti2001 (T138650)
  • 18:17 demon@tin: Synchronized wmf-config/FeaturedFeedsWMF.php: no-op (duration: 00m 46s)
  • 16:04 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/AbuseFilter/includes/Views/AbuseFilterViewExamine.php: T175338, followup (duration: 00m 46s)
  • 16:03 demon@tin: Synchronized php-1.30.0-wmf.17/includes/revisiondelete/RevDelLogList.php: typofix (duration: 00m 46s)
  • 15:53 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4027.*
  • 15:47 ejegg: increased cURL timeout for PayPal IPN confirmation to 14 sec
  • 15:07 akosiaris@tin: Finished deploy [librenms/librenms@5554213]: Testing new scap keyholder_key config (duration: 00m 04s)
  • 15:06 akosiaris@tin: Started deploy [librenms/librenms@5554213]: Testing new scap keyholder_key config
  • 15:02 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4021.*
  • 14:58 volans: testing wmf-auto-reimage also on mc1002 T166300 T164341
  • 12:36 _joe_: restarting a check of the md2 devices on conf200* T162013 after reducing sync speed
  • 12:36 _joe_: reduced raid resync max speed of raid devices on conf200* to 20000
  • 10:35 hashar@tin: Synchronized php-1.30.0-wmf.17/extensions/ProofreadPage: Restore page status buttons - T175304 (duration: 00m 50s)
  • 10:12 _joe_: stopped all md devices syncs on conf200* machines
  • 10:09 _joe_: etcd cluster lost consensus T162013
  • 10:07 volans: testing wmf-auto-reimage on mc1001 T166300 T164341
  • 09:58 _joe_: running the same command on conf2002 too
  • 09:48 _joe_: running `/usr/share/mdadm/checkarray --cron --all --idle --quiet` on conf2001 and conf2003 trying to reproduce consensus issues in T162013
  • 09:25 moritzm: restarting apache on tegmen/einsteinium to pick up security update
  • 09:09 moritzm: installing libonig security updates
  • 09:08 mobrovac@tin: Finished deploy [restbase/deploy@0d39acf] (dev-cluster): Use double writing for mobileapps in the dev cluster, take #2 - T169940 (duration: 01m 01s)
  • 09:07 mobrovac@tin: Started deploy [restbase/deploy@0d39acf] (dev-cluster): Use double writing for mobileapps in the dev cluster, take #2 - T169940
  • 09:01 moritzm: installing remaining gnutls updates from jessie point release
  • 08:13 moritzm: installing libgd2 security updates
  • 08:11 mobrovac@tin: Started deploy [restbase/deploy@0d39acf] (dev-cluster): Use double writing for mobileapps in the dev cluster - T169940
  • 08:11 mobrovac: restbase enabled back puppet in the dev cluster - T169940
  • 07:53 moritzm: installing ruby2.3 security updates
  • 07:32 mobrovac@tin: Finished deploy [restbase/deploy@0d39acf] (dev-cluster): (no justification provided) (duration: 07m 02s)
  • 07:25 mobrovac@tin: Started deploy [restbase/deploy@0d39acf] (dev-cluster): (no justification provided)
  • 07:21 mobrovac@tin: Finished deploy [restbase/deploy@e6aeeeb] (dev-cluster): (no justification provided) (duration: 00m 11s)
  • 07:21 mobrovac@tin: Started deploy [restbase/deploy@e6aeeeb] (dev-cluster): (no justification provided)
  • 00:42 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/AbuseFilter/includes/Views/AbuseFilterViewExamine.php: fix comment stuff (duration: 00m 46s)

2017-09-07

  • 23:42 bblack: cp1066 - depooled all traffic services, seems to have some bug causing 503 spikes
  • 23:19 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.17
  • 23:15 XenoRyet: Updated Smashpig from 8eb98c1 to dce4f0a
  • 22:41 niharika29@tin: Synchronized php-1.30.0-wmf.17/extensions/ArticleCreationWorkflow/: Bump extension version (duration: 00m 49s)
  • 22:03 niharika29@tin: Finished scap: Deploying ArticleCreation Workflow on test2 wiki TT175302 (duration: 23m 00s)
  • 21:40 niharika29@tin: Started scap: Deploying ArticleCreation Workflow on test2 wiki TT175302
  • 21:23 demon@tin: Synchronized php-1.30.0-wmf.17/includes/api/ApiQueryRecentChanges.php: T175307 (duration: 00m 49s)
  • 21:01 urandom: T169940: Disabling puppet and restbase service in RESTBase dev
  • 21:00 urandom: T169940: Disabling changeprop in RESTBase dev environment
  • 19:05 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/MobileFrontend/includes/specials/SpecialMobileHistory.php: T175161 (duration: 00m 49s)
  • 18:48 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.17
  • 18:47 demon@tin: Synchronized php: symlink update -> wmf.17 (duration: 00m 48s)
  • 18:39 legoktm: restarted script to reparse all pages in parsoid for Linter (python3 parsoid_reparse.py http://parsoid.discovery.wmnet:8000 --sitematrix --linter-only --skip-closed https://aa.wikipedia.org/w/api.php) - T161556
  • 17:55 ejegg: disabled queue consumers for data cleaning job
  • 17:52 ejegg: updated CiviCRM from c1ece1e to 63288ae
  • 17:35 otto@tin: Finished deploy [analytics/refinery@6b60f2c]: (no justification provided) (duration: 03m 23s)
  • 17:32 otto@tin: Started deploy [analytics/refinery@6b60f2c]: (no justification provided)
  • 17:31 otto@tin: Finished deploy [analytics/refinery@bdf6754]: (no justification provided) (duration: 00m 23s)
  • 17:31 otto@tin: Started deploy [analytics/refinery@bdf6754]: (no justification provided)
  • 17:23 otto@tin: Finished deploy [analytics/refinery@bdf6754]: (no justification provided) (duration: 03m 25s)
  • 17:20 otto@tin: Started deploy [analytics/refinery@bdf6754]: (no justification provided)
  • 16:13 demon@tin: Synchronized wmf-config/InitialiseSettings.php: Remove Extension:RelatedSites from zhwikivoyage (duration: 00m 50s)
  • 15:32 mobrovac@tin: Finished deploy [restbase/deploy@77961d0]: Revert using MCS for the summary end point - T168848 (duration: 06m 18s)
  • 15:25 mobrovac@tin: Started deploy [restbase/deploy@77961d0]: Revert using MCS for the summary end point - T168848
  • 14:45 _joe_: manually running /usr/share/mdadm/checkarray --cron --all --idle --quiet on conf2001, trying to reproduce consensus issues in T162013
  • 14:40 jynus: rebooting and upgrading es1019
  • 13:54 godog: powercycle ms-be2023 - load through the roof and no login possible
  • 13:54 oblivian@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=jobrunner,name=eqiad
  • 13:14 zeljkof: EU SWAT finished
  • 13:13 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add English Wiktionary as a client of Wikidata (T159316) (duration: 00m 49s)
  • 13:12 zfilipin@tin: Synchronized dblists/wikidataclient.dblist: SWAT: Add English Wiktionary as a client of Wikidata (T159316) (duration: 00m 49s)
  • 11:24 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1100 with 0 weight - T172679 (duration: 00m 49s)
  • 09:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1100 depooled to s5 array - T172679 (duration: 00m 49s)
  • 08:59 ariel@tin: Finished deploy [dumps/dumps@da05e9f]: add general info to siteinfo dumps (duration: 00m 02s)
  • 08:58 ariel@tin: Started deploy [dumps/dumps@da05e9f]: add general info to siteinfo dumps
  • 08:44 elukey: restart varnish backend on cp1063 - mailbox expiry lag
  • 07:57 elukey: force re-mount of /mnt/hdfs on stat1005
  • 07:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1100 - T172679 (duration: 00m 48s)
  • 07:53 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1100 - T172679 (duration: 00m 49s)
  • 07:45 mobrovac@tin: Finished deploy [restbase/deploy@b51c58c]: Use MCS for the summary endpoint - T168848 (duration: 06m 36s)
  • 07:39 mobrovac@tin: Started deploy [restbase/deploy@b51c58c]: Use MCS for the summary endpoint - T168848
  • 07:35 mobrovac@tin: Finished deploy [restbase/deploy@b51c58c] (staging): Use MCS for the summary endpoint (duration: 01m 43s)
  • 07:33 mobrovac@tin: Started deploy [restbase/deploy@b51c58c] (staging): Use MCS for the summary endpoint
  • 07:20 oblivian@tin: Synchronized rpc/RunSingleJob.php: Adding the RunSingleJob.php rpc endpoint for jobrunners (duration: 00m 56s)
  • 06:07 eileen: civicrm updated from 73a24b4 to c1ece1e
  • 03:13 eileen: seconds behind=78
  • 03:08 eileen: seconds behind 55
  • 02:52 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Sep 7 02:52:05 UTC 2017 (duration 7m 15s)
  • 02:44 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 06m 15s)
  • 02:43 eileen: restarting groupmember.load after hacking out cache delete calls
  • 02:40 eileen: update civicrm from 764bfe1 to 73a24b4
  • 02:24 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 07m 23s)
  • 02:05 eileen: replag 217 behind
  • 01:56 eileen: replag 197 seconds
  • 01:47 eileen: replag 23 seconds
  • 01:29 eileen: replag 0
  • 01:22 eileen: replag 550 sec behind master
  • 01:13 eileen: replag delay 422
  • 01:12 eileen: job killed
  • 00:49 eileen: 8389 contacts now
  • 00:49 eileen: 7971 contacts so far
  • 00:46 eileen: started Omnigroupmember.load
  • 00:24 twentyafterfour: phabricator update deployed
  • 00:12 twentyafterfour@tin: Finished deploy [phabricator/deployment@c265c1a]: (no justification provided) (duration: 00m 24s)
  • 00:11 twentyafterfour@tin: Started deploy [phabricator/deployment@c265c1a]: (no justification provided)
  • 00:07 twentyafterfour: deploying phabricator update 2017-09-06 https://phabricator.wikimedia.org/project/view/2980/

2017-09-06

  • 23:41 twentyafterfour@tin: Synchronized php-1.30.0-wmf.16/extensions/WikimediaEvents/: Sync Change-Id: I7ae522 to all webservers refs T174387 (duration: 00m 49s)
  • 23:28 twentyafterfour@tin: Synchronized php-1.30.0-wmf.17/extensions/WikimediaEvents/: Sync Change-Id: I7ae522 to wmf.17 refs T174387 (duration: 00m 49s)
  • 23:23 twentyafterfour@tin: Synchronized wmf-config/CirrusSearch-common.php: CirrusSearch-common.php goes last (duration: 00m 48s)
  • 23:21 twentyafterfour@tin: Synchronized wmf-config/InitialiseSettings.php: Now sync initializesettings (duration: 00m 49s)
  • 23:19 twentyafterfour@tin: Synchronized wmf-config/CirrusSearch-rel-survey.php: sync the added file first (duration: 00m 49s)
  • 23:17 twentyafterfour@tin: scap failed: average error rate on 9/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details)
  • 23:12 twentyafterfour@tin: Synchronized wmf-config/: roll-back due to huge error rate spike (duration: 00m 51s)
  • 23:10 twentyafterfour@tin: Synchronized wmf-config/: deploy config for CirrusSearch human relevancy survey. Change-Id: I272c69 Bug: T174106 (duration: 00m 52s)
  • 22:18 no_justification: prior deploy was no-op
  • 22:18 demon@tin: Finished deploy [gerrit/gerrit@d4f9a77]: (no justification provided) (duration: 00m 07s)
  • 22:18 demon@tin: Started deploy [gerrit/gerrit@d4f9a77]: (no justification provided)
  • 22:11 bsitzmann@tin: Finished deploy [mobileapps/deploy@507a479]: Update mobileapps to 2cb6281 (T168848 T169277 T169274 T162179 T164033 T167921 T174698 T168848 T174808) (duration: 04m 53s)
  • 22:06 bsitzmann@tin: Started deploy [mobileapps/deploy@507a479]: Update mobileapps to 2cb6281 (T168848 T169277 T169274 T162179 T164033 T167921 T174698 T168848 T174808)
  • 21:39 niharika29@tin: Synchronized wmf-config/CommonSettings-labs.php: Config for ArticleCreationWorkflow T175054 (duration: 00m 50s)
  • 21:28 ejegg: updated payments-wiki from bf97cd2 to ed2e481
  • 21:24 ejegg: updated CiviCRM from f9c373e to 764bfe1
  • 21:17 ottomata: rebooting kafka-jumbo1004
  • 20:34 arlolra: Updated Parsoid to f9d367ea (T169342)
  • 20:27 arlolra@tin: Finished deploy [parsoid/deploy@f07ac8c]: Updating Parsoid to f9d367ea (duration: 08m 27s)
  • 20:18 arlolra@tin: Started deploy [parsoid/deploy@f07ac8c]: Updating Parsoid to f9d367ea
  • 19:53 ottomata: reimaging kafka-jumbo* with stretch
  • 19:48 chasemp: reboot labvirt1018
  • 18:50 maxsem@tin: Synchronized wmf-config/: just to make sure... (duration: 00m 50s)
  • 18:48 maxsem@tin: Synchronized wmf-config/CommonSettings.php: revert (duration: 00m 49s)
  • 18:33 maxsem@tin: Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/370292/3 (duration: 00m 49s)
  • 18:25 maxsem@tin: Synchronized wmf-config/CommonSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/370291/2 (duration: 00m 49s)
  • 18:19 maxsem@tin: Synchronized wmf-config/Wikibase.php: https://gerrit.wikimedia.org/r/#/c/376317/2 (duration: 00m 48s)
  • 18:18 maxsem@tin: Synchronized wmf-config/Wikibase-production.php: https://gerrit.wikimedia.org/r/#/c/376317/2 (duration: 00m 48s)
  • 18:09 maxsem@tin: Synchronized portals: (no justification provided) (duration: 00m 49s)
  • 18:09 ejegg: rolled payments-wiki back to bf97cd2
  • 18:08 maxsem@tin: Synchronized portals/prod/wikipedia.org/assets: (no justification provided) (duration: 00m 50s)
  • 18:05 ejegg: updated payments-wiki from bf97cd2 to 6911158
  • 17:53 bblack: cp1048 - varnish backend restart for mailbox lag...
  • 17:23 demon@tin: Synchronized wmf-config/FeaturedFeedsWMF.php: code cleanup (duration: 00m 49s)
  • 17:20 mutante: added LDAP user schoenbaechler to WMF group (T175156)
  • 17:16 mutante: modify-ldap-group on terbium is broken
  • 17:02 ejegg: disabled creating CiviMail records for Thank You emails
  • 16:51 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/Flow/includes/: I284b5aa (duration: 01m 01s)
  • 16:41 chasemp: disable puppet for cloud things to test changes
  • 16:40 bblack: cp1072 - varnish backend restart (mailbox lag)
  • 16:40 bblack: cp1062 - varnish backend restart (mailbox lag)
  • 15:55 ejegg: re-enabled IPN notification at PayPal console (48 minutes ago)
  • 15:53 akosiaris: powercycling lvs3001
  • 15:28 papaul: firmware upgrade on mw2256
  • 15:11 bblack: cp1049 - restart varnish backend (mailbox lag)
  • 15:07 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool es1019 (duration: 00m 49s)
  • 15:06 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2040 (duration: 00m 49s)
  • 14:17 gehel: wdqs1005 is coming up after a few reimaging issues, expect some icinga noise...
  • 13:19 jynus: restarting and upgrading db2040
  • 13:13 reedy@tin: Synchronized wmf-config/Wikibase-production.php: Move some wikidata config T174962 (duration: 00m 48s)
  • 13:12 reedy@tin: Synchronized wmf-config/Wikibase-labs.php: Move some wikidata config T174962 (duration: 00m 49s)
  • 13:07 reedy@tin: Synchronized wmf-config/throttle.php: Throttle exception T175113 (duration: 00m 49s)
  • 12:08 moritzm: installing libapache2-mod-perl update from jessie 8.9 update
  • 11:41 moritzm: installing gtk+2.0 update from jessie 8.9 update
  • 11:24 elukey: temporarily raise kafka log4j authorizer verbosity to DEBUG on kafka1012 - T173493
  • 11:21 moritzm: installing gnutls update from jessie 8.9 and stretch 9.1 point updates
  • 11:20 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2040 (duration: 00m 49s)
  • 11:20 godog: reimage restbase1008 with cassandra 3 - T169939
  • 11:15 marostegui: Disable puppet on db1100 for mydumper/myloader
  • 10:26 moritzm: installing perl update from stretch 9.1 point release
  • 10:25 moritzm: installing perl update from jessie 8.9 point release
  • 10:12 godog: reimage restbase1010 with cassandra 3 - T169939
  • 09:39 moritzm: installing libgd security updates
  • 09:28 moritzm: installing libonig security uodates#
  • 09:05 jynus: disabling puppet on most db hosts to merge firewall changes safely
  • 08:53 godog: reimage restbase1009 with cassandra 3 - T169939
  • 08:49 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T174948 Add WMDE log channel (duration: 00m 49s)
  • 08:43 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T110170 Enable Newsletter on mediawikiwiki (duration: 00m 51s)
  • 07:15 marostegui: Stop MySQL on db1049 to copy its content to db1100 - https://phabricator.wikimedia.org/T172679
  • 06:46 moritzm: installing php-luasandbox 2.0.14 on API canaries along with HHVM restart (T173705)
  • 06:37 marostegui: Truncate l10n_cache table across production - T150306
  • 04:08 demon@tin: Synchronized wmf-config/throttle.php: no-op (duration: 00m 49s)
  • 03:14 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Sep 6 03:14:16 UTC 2017 (duration 7m 6s)
  • 03:07 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 14m 49s)
  • 02:31 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 08m 18s)
  • away: disabled IPN messages at the PayPal console
  • 00:14 Reedy: that was T174747 Adjust language icon color
  • 00:14 reedy@tin: Synchronized php-1.30.0-wmf.16/skins/MinervaNeue: (no justification provided) (duration: 00m 49s)
  • 00:10 reedy@tin: Synchronized php-1.30.0-wmf.16/extensions/Scribunto/tests/phpunit/: Fix broken test (duration: 00m 49s)
  • 00:09 reedy@tin: Synchronized php-1.30.0-wmf.17/extensions/Scribunto/tests/phpunit/: Fix broken test (duration: 00m 50s)

2017-09-05

  • 23:58 reedy@tin: Synchronized wmf-config/: Test ArticleCreationWorkflow extension on the Beta Cluster (duration: 00m 51s)
  • 23:39 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: Enable AbuseFilter runtime profile T161059 (duration: 00m 49s)
  • 23:33 foks: 2FA disabled for Omshivaprakash (T175075)
  • 22:17 robh: rebooting mw1163 as its serial console is locked up
  • 22:07 demon@tin: Synchronized php-1.30.0-wmf.16/extensions/FlaggedRevs/frontend/specialpages/reports/: kill some warnings and stuff (duration: 02m 55s)
  • 22:06 demon@tin: Synchronized php-1.30.0-wmf.16/extensions/FlaggedRevs/frontend/specialpages/reports/: (no justification provided) (duration: 03m 13s)
  • 20:48 ejegg: re-enabled donation queue consumer and thank you mailer
  • 20:40 ejegg: updated civicrm from 820bd98 to f9c373e
  • 20:37 ejegg: disabled Donation queue consumer and thank-you mail sender
  • 19:38 gilles@tin: Synchronized private/PrivateSettings.php: Add Thumbor username to Swift configuration (duration: 00m 48s)
  • 19:33 jynus: rebuilding querycache table on tewiktionary.dbstore1002, it had crashed
  • 19:28 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: wmf.17 on group0
  • 19:05 Reedy: deleted querycache rows where qc_type = '' on all wikis T174513
  • 18:11 ejegg: updated fundraising tools from 9554229 to 8d39806
  • 18:11 demon@tin: Finished scap: wmf.17 bootstrap (duration: 47m 00s)
  • 17:38 bblack: cp1074 - restart varnish backend, mailbox lag
  • 17:24 demon@tin: Started scap: wmf.17 bootstrap
  • 17:14 demon@tin: Pruned MediaWiki: 1.30.0-wmf.12 (duration: 02m 32s)
  • 16:59 andrewbogott: reducing the number of concurrent nova-conductor workers. May cause hiccups as services restart
  • 16:58 elukey: ran authdns-update from ns1.w.o after https://gerrit.wikimedia.org/r/374385 to create electcom.wikimedia.org
  • 16:51 jynus: reloading dbproxy1005 to repoing to db1009 again- things seem stable right now
  • 16:45 elukey: add new virtualhost electcom.wikimedia.org to the appservers apache config - https://gerrit.wikimedia.org/r/374389 (implies apache config reload)
  • 16:33 mattflaschen@tin: Synchronized php-1.30.0-wmf.16/: Prepare to enable RCFilters (WLFilters) on Watchlist, but without i18n changes (reverted) (duration: 12m 41s)
  • 16:14 addshore: addshore@terbium:~$ mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki mediawikiwiki --extension newsletter
  • 16:08 akosiaris: reboot all kubernetes boxes T170119
  • 15:43 matt_flaschen: 'Scap sync failed on i18n, so I'll deploy just the non-i18n ones'
  • 15:36 mattflaschen@tin: scap aborted: Prepare to enable RCFilters (WLFilters) on Watchlist (duration: 08m 42s)
  • 15:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1083 - T174509 (duration: 00m 47s)
  • 15:27 mattflaschen@tin: Started scap: Prepare to enable RCFilters (WLFilters) on Watchlist
  • 14:30 reedy@tin: Synchronized wmf-config/: phpcs (duration: 00m 47s)
  • 14:28 reedy@tin: Synchronized tests/: phpcs (duration: 00m 46s)
  • 14:26 reedy@tin: Synchronized errorpages/404.php: phpcs (duration: 00m 45s)
  • 14:25 reedy@tin: Synchronized docroot/noc/conf/highlight.php: Fix links for dblists in highlight.php (duration: 00m 44s)
  • 14:15 zeljkof: EU SWAT finished
  • 14:13 zfilipin@tin: Synchronized php-1.30.0-wmf.16/includes/EditPage.php: SWAT: Re add wpScrolltop id in EditPage (T174723) (duration: 00m 45s)
  • 13:47 zfilipin@tin: Synchronized php-1.30.0-wmf.16/extensions/ContentTranslation/: SWAT: encodeURIComponent title to escape / properly (T174792) (duration: 00m 48s)
  • 13:30 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add abusefilter-view-private to rollbackers in zhwiki (T174978) (duration: 00m 46s)
  • 13:28 godog: reimage restbase2005 - T169939
  • 13:17 marostegui: Drop pr_index tables from where ProofreadPage isn't enabled - T174782
  • 13:14 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Update logo for sr.wikibooks.org (T172284) (duration: 00m 46s)
  • 12:24 hashar: ci: switched mediawiki/core mediawiki/vendor php5.5 jobs from trusty to jessie - T161882
  • 12:15 ladsgroup@tin: Synchronized wmf-config/Wikibase-production.php: Enable sendEchoNotification for enwiki, dewiki, frwiki (T142102) (duration: 00m 46s)
  • 11:54 marostegui: Drop reader_feedback, reader_feedback_history, reader_feedback_pages tables - T174586
  • 11:07 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: (Actually) Enable the ArticlePlaceholder on sqwiki (T174335) (duration: 00m 45s)
  • 11:05 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on sqwiki (T174335) (duration: 00m 46s)
  • 10:38 hashar: Nodepool / CI are fully backup and processing queued jobs
  • 10:25 hashar: Restarting Nodepool. I have managed to spawn instances manually for contintcloud tenant
  • 10:18 _joe_: stopping puppet, nginx on mw2249 for experimentation
  • 10:11 ema: reboot cp4024 T174891
  • 10:08 godog: copy python-logstash from jessie-wikimedia to stretch-wikimedia
  • 09:59 hashar: Stopped Nodepool on labnodepool1001.eqiad.wmnet
  • 09:52 akosiaris: reimage kubernetes[12]00[1-4] T170119
  • 08:49 moritzm: upgrading canary app servers to hhvm-luasandbox 2.0.14 (T173705)
  • 08:47 moritzm: uploaded php-luasandbox/hhvm-luasandbox 2.0.14 to apt.wikimedia.org (T173705)
  • 07:14 moritzm: installing libgd security updates on canary app servers (along with hhvm restart)
  • 07:03 _joe_: launching manually 3 workers for refreshLinks jobs on commons, T173710
  • 06:32 marostegui: Deploy alter table on db1083 - T174509
  • 06:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1083 - T174509 (duration: 00m 46s)
  • 06:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1089 - T174509 (duration: 00m 46s)
  • 06:09 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1089 - T174509 (duration: 00m 55s)
  • 02:30 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Sep 5 02:30:03 UTC 2017 (duration 6m 37s)
  • 02:23 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 07m 28s)

2017-09-04

  • 23:32 Dereckson: Set e-mail to Hexabot bot account to allow account recovery (T174973)
  • 22:07 legoktm: starting script to reparse all pages in parsoid for Linter (python2 parsoid-reparse.py http://parsoid.discovery.wmnet:8000 --sitematrix --linter-only --skip-closed https://aa.wikipedia.org/w/api.php) - T161556
  • 17:50 moritzm: uploaded debdeploy 0.0.99-2 to apt.wikimedia.org
  • 17:19 gehel@tin: Finished deploy [wdqs/wdqs@1caaa30]: weekly WDQS deployment (duration: 02m 46s)
  • 17:17 gehel@tin: Started deploy [wdqs/wdqs@1caaa30]: weekly WDQS deployment
  • 16:29 mobrovac@tin: Finished deploy [restbase/deploy@8c9c436]: (no justification provided) (duration: 06m 23s)
  • 16:23 mobrovac@tin: Started deploy [restbase/deploy@8c9c436]: (no justification provided)
  • 16:10 mobrovac: restbase depooled restbase10(0[89]|10) and restbase2005 for T169939
  • 16:07 _joe_: rolling restart of pybal in codfw/eqiad low-traffic pools for picking up the new jobrunner LVS endpoint
  • 15:42 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,service=nginx
  • 14:51 godog: reimage restbase2003 for use with cassandra 3 / jbod - T169939
  • 14:27 moritzm: enabled LDAP authentication for ganglia.wikimedia.org
  • 13:37 gehel: rolling restart of elasticsearch/logstash to cleanup unused plugins - T174933
  • 13:34 gehel: delete leftover grafana-dashboards index on elasticsearch logstash cluster
  • 13:22 gehel: restart elasticsearch on logstash1006 to test plugin removal - T174933
  • 12:26 moritzm: uploaded debdeploy 0.0.99-1+trusty/trusty-wikimedia to apt.wikimedia.org
  • 12:18 marostegui: Stop MySQL on db1037 as it is going to be decommissioned - T174902
  • 12:10 elukey: rolling restart of zookeeper on conf100[123] for jvm security updates
  • 11:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1037 as it will be decommissioned - T174902 (duration: 00m 46s)
  • 11:52 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1037 as it will be decommissioned - T174902 (duration: 00m 46s)
  • 11:45 godog: restart varnish on cp1099 - mailbox lag
  • 10:22 ariel@tin: Finished deploy [dumps/dumps@0b32eef]: fix abstract dumps variants regression (duration: 00m 02s)
  • 10:22 ariel@tin: Started deploy [dumps/dumps@0b32eef]: fix abstract dumps variants regression
  • 09:51 moritzm: uploaded debdeploy 0.0.99-1+jessie/jessie-wikimedia to apt.wikimedia.org
  • 09:41 mobrovac@tin: Started restart [mathoid/deploy@44ea6d8]: Restart for libxml update
  • 09:36 moritzm: uploaded debdeploy 0.0.99-1/stretch-wikimedia to apt.wikimedia.org
  • 08:52 elukey: restart zookeeper on conf2003 for jvm security updates
  • 08:48 marostegui: Stop MySQL on db1045 as it will be decommissioned - T174806
  • 08:47 moritzm: installing mercurial security updates
  • 08:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1045 as it will be decommissioned - T174806 (duration: 00m 46s)
  • 08:39 moritzm: installing libgcrypt20 security updates
  • 08:39 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1045 as it will be decommissioned - T174806 (duration: 00m 48s)
  • 08:33 godog: upload scap 3.7.0-1 to apt.w.o - T127762
  • 07:39 moritzm: installing gnupg security updates
  • 07:03 marostegui: Deploy alter table on s3 codfw master with replication on kpbwiki - T168661
  • 07:02 _joe_: starting additional runJobs instance for htmlcacheupdate on commons T173710
  • 04:52 marostegui: Deploy alter table on db1052 - T168661
  • 02:44 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Sep 4 02:44:15 UTC 2017 (duration 6m 52s)
  • 02:37 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 08m 17s)

2017-09-03

  • 17:52 elukey: depooled cp4024 (ulsfo upload) due to kernel errors in dmesg
  • 15:57 ema: restart varnish backend on cp1073 (mailbox lag)
  • 09:41 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4024.ulsfo.wmnet,service=varnish-fe
  • 09:41 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4024.ulsfo.wmnet,service=nginx
  • 09:25 ema: cp4024 is down, powercycled
  • 09:13 _joe_: manually fixed value of /conftool/v1/pools/eqiad/api_appserver/apache2/mw1208.eqiad.wmnet in codfw to allow replication to restart
  • 00:59 legoktm: legoktm@terbium:~$ mwscript extensions/WikimediaMaintenance/filebackend/setZoneAccess.php --wiki=hiwikiversity --backend=local-multiwrite # T174859

2017-09-01

  • 16:18 bblack: restart varnish backend on cp1074 (mailbox lag)
  • {{safesubst:SAL entry|1=15:15 urandom: Restarting Cassandra: restbase-dev100[5-6]-{a,b}}}
  • {{safesubst:SAL entry|1=15:06 urandom: Restarting Cassandra: restbase-dev1004-{a,b}}}
  • 15:00 reedy@tin: Synchronized php-1.30.0-wmf.16/extensions/Popups/: T174724 (duration: 00m 46s)
  • 14:28 marostegui: Add 150G to /srv partition on labsdb1001
  • 13:42 marostegui: Rename table pr_index on enwiki on db1089 - T174782
  • 13:10 marostegui: Stop MySQL on db1026 as it will be decommissioned - T174763
  • 12:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1081 original weight - T168661 (duration: 00m 43s)
  • 12:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1026 as it will be decommissioned - T174763 (duration: 00m 43s)
  • 12:17 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1026 as it will be decommissioned - T174763 (duration: 00m 43s)
  • 11:50 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1081 weight - T168661 (duration: 00m 43s)
  • 11:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1081 with low weight - T168661 (duration: 00m 44s)
  • 10:35 elukey: stop puppet on thorium and disable root rsyncs - T174756
  • 10:31 marostegui: Upgrade MariaDB to 10.0.32 on db1081 - T168661
  • 10:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1081 - T168661 (duration: 00m 43s)
  • 09:10 godog: bounce graphite-web on graphite1001, problematic query? using all CPU
  • 09:04 ema: lvs1007 upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882
  • 08:59 godog: depool restbase200[135] before reimage - T169939
  • 07:22 elukey: restart apache2 and hue on thorium, Analytics sites down, investigating
  • 07:06 marostegui: Power reset db2044 as it is unresponsive - T174764
  • 03:54 krinkle@tin: Synchronized wmf-config/InitialiseSettings.php: I4dfc33f66c3 - Enable jQuery 3 on nlwiki sister projects (duration: 00m 43s)
  • 00:57 tstarling@tin: Synchronized php-1.30.0-wmf.16/includes/parser/Parser.php: (no justification provided) (duration: 00m 44s)
  • 00:13 RainbowSprinkles: gerrit: flushed all non-login caches, things might be sluggish for the next ~15mins or so

2017-08-31

  • 20:03 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: (no justification provided)
  • 19:27 mattflaschen@tin: Finished scap: Watchlist filters: Convert edit watchlist button to new UX and fix server-side tag filtering. T172030 (duration: 21m 23s)
  • 19:17 cmjohnson1: powercycled mw1294
  • 19:05 mattflaschen@tin: Started scap: Watchlist filters: Convert edit watchlist button to new UX and fix server-side tag filtering. T172030
  • 18:53 mattflaschen@tin: Synchronized docroot/noc/conf/interwiki-labs.php.txt: Interwiki links on Beta Cluster: T69931 (duration: 00m 47s)
  • 18:51 mattflaschen@tin: Synchronized wmf-config: Interwiki links on Beta Cluster: T69931 (duration: 00m 49s)
  • 17:50 demon@tin: Synchronized php-1.30.0-wmf.16/skins/MinervaNeue/resources/skins.minerva.icons.images.scripts/watch.svg: sorta impromptu swat thing (duration: 00m 47s)
  • 16:35 ejegg: re-enabled donation queue consumer and thank you mail sender
  • 15:25 ejegg: updated civicrm from f00611d to 820bd98
  • 15:20 ejegg: updated SmashPig from 8f4388d to 8eb98c1
  • 15:19 ejegg: turned off donation queue consumer and thank you job
  • 15:18 elukey: restart zookeeper on conf200[2,3] for jvm security updates
  • 13:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1084 original weight - T168661 (duration: 00m 47s)
  • 13:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1084 weight - T168661 (duration: 00m 47s)
  • 13:07 elukey: restart zookeeper on conf2001 for security updates (canary node)
  • 13:00 moritzm: installing augeas security updates
  • 12:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1084 with low weight - T168661 (duration: 00m 47s)
  • 12:14 marostegui: Upgrade MariaDB on db1084 - T168661
  • 12:11 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1084 - T168661 (duration: 00m 47s)
  • 12:05 akosiaris: reimage chlorine and kubernetes1004 T170119
  • 12:04 akosiaris: reimage chlorine and kubernetes1004 T10119
  • 11:58 moritzm: restarting nginx on dataset1001/ms1001 to pick up libxml security update
  • 11:21 elukey: restart apache2 on bohrium for libxml + gnutls security updates
  • 11:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1056 original weight - T168661 (duration: 00m 47s)
  • 11:17 moritzm: restarting apache on dbmonitor* to pick up libxml security update
  • 10:27 moritzm: restarting nginx on prometheus* to pick up libxml security update
  • 10:22 moritzm: restarting apache on einsteinium/tegmen (Icinga hosts) to pick up libxml security update
  • 10:14 moritzm: restarting nginx on meitnerium/archiva.wikimedia.org to pick up libxml security update
  • 10:13 moritzm: restarting nginx on debug proxies to pick up libxml security update
  • 10:10 moritzm: restarting apache on hafnium to pick up libxml security update
  • 09:57 addshore: extensions/Cognate/maintenance/recalculateCognateNormalizedHashes.php run done, 6326 hashes recalculated, T172987
  • 09:50 addshore: addshore@terbium:~$ mwscript extensions/Cognate/maintenance/recalculateCognateNormalizedHashes.php --wiki enwiktionary --batch-size 1000000 # Should upsert 6326 rows T172987
  • 09:44 addshore: addshore@terbium:~$ mwscript extensions/Cognate/maintenance/recalculateCognateNormalizedHashes.php --wiki enwiktionary --batch-size 1000000 --dry-run # T172987
  • 09:43 moritzm: restarting apache on auth* to pick up libxml security update
  • 09:40 moritzm: installing libxml2 security updates
  • 09:10 moritzm: installing experimental hhvm-luasandbox package on mw1261 (canary app server) with backported patch for T171392 / T173705
  • 09:04 hashar: contint1001 upgrading git, restarting git-daemon and zuul-merger - T161086
  • 09:01 godog: test reimage of restbase2001 - T169939
  • 08:57 hashar: contint2001 upgrading git, restarting git-daemon and zuul-merger - T161086
  • 08:45 godog: remove /srv/deployment/grafana on naos and tin, unused - T170881
  • 08:41 gehel: restart postgresql / kartotherian / tilerator / tileratorui on maps@eqiad
  • 08:40 addshore@tin: Synchronized php-1.30.0-wmf.16/extensions/Cognate/maintenance/recalculateCognateNormalizedHashes.php: T172987 Add waitForReplication to RecalculateCognateNormalizedHashes script (duration: 00m 47s)
  • 08:20 gehel: restart postgresql / kartotherian / tilerator / tileratorui on maps@codfw
  • 08:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic on db1056 - T168661 (duration: 00m 47s)
  • 08:01 marostegui: Rename reader_feedback, reader_feedback_history, reader_feedback_pages tables on dewiki on db1092 - T174586
  • 07:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic on db1056 - T168661 (duration: 00m 47s)
  • 06:45 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1056 with low weight - T168661 (duration: 00m 46s)
  • 06:19 marostegui: Upgrade MariaDB to 10.0.32 on db1056 - T168661
  • 06:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1056 - T168661 (duration: 00m 47s)
  • 06:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1056 - T168661 (duration: 00m 47s)
  • 02:55 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Aug 31 02:55:12 UTC 2017 (duration 7m 9s)
  • 02:48 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 06m 40s)
  • 02:30 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 10m 02s)

2017-08-30

  • 23:10 demon@tin: Synchronized php-1.30.0-wmf.16/extensions/Newsletter/includes/specials/SpecialNewsletter.php: T174604 (duration: 00m 47s)
  • 23:09 demon@tin: Synchronized php-1.30.0-wmf.15/extensions/Newsletter/includes/specials/SpecialNewsletter.php: T174604 (duration: 00m 48s)
  • 23:05 demon@tin: Synchronized wmf-config/unitConversionConfig.json: swat (duration: 00m 47s)
  • 21:05 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 back to 1.30.0-wmf.16
  • 21:03 thcipriani@tin: Synchronized php-1.30.0-wmf.16/extensions/ProofreadPage/ProofreadPage.body.php: ProofreadPage: Avoids a stack overflow T173520 (duration: 00m 47s)
  • 20:45 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 back to 1.30.0-wmf.15
  • 20:22 ppchelko@tin: Finished deploy [changeprop/deploy@8d5fa29]: Start rejecting deduplicated messages (duration: 01m 05s)
  • 20:21 ppchelko@tin: Started deploy [changeprop/deploy@8d5fa29]: Start rejecting deduplicated messages
  • 20:07 urandom: T169939: Decommission Cassandra: restbase1009-c.eqiad.wmnet
  • 19:09 ppchelko@tin: Finished deploy [changeprop/deploy@ff93ea8]: Don't deduplicate retry messages (duration: 01m 17s)
  • 19:07 ppchelko@tin: Started deploy [changeprop/deploy@ff93ea8]: Don't deduplicate retry messages
  • 19:06 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.16
  • 19:04 demon@tin: Synchronized php: Symlink swap for group1/wmf.16 (duration: 00m 46s)
  • 18:11 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/374399/4 (duration: 00m 47s)
  • 17:29 urandom: T169939: Decommission Cassandra: restbase1009-b.eqiad.wmnet
  • 17:27 hoo: Updated the Wikidata property suggester with data from Monday's JSON dump and applied the T132839 workarounds
  • 17:12 ejegg: disabled creating CiviMail records for Thank You emails
  • 16:49 XioNoX: rolling back vrrp priority change for the cr2<->asw-d-codfw link - T174366
  • 16:34 XioNoX: start work on the cr2<->asw-d-codfw link - T174366
  • 16:13 XioNoX: lowering vrrp priority for the cr2<->asw-d-codfw link - T174366
  • 15:56 elukey: re-added analytics1055 among the hdfs/yarn worker after maintenance
  • 15:49 bblack: cp1049 - restart varnish backend, mailbox lag
  • 15:43 urandom: T169939: Decommission Cassandra: restbase1009-a.eqiad.wmnet
  • 15:02 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1294.eqiad.wmnet
  • 14:31 moritzm: restarting nginx on sodium to pick up libxml security update
  • 14:07 elukey: restart java daemons on druid100[456] for jvm security updates
  • 14:04 moritzm: restarting squid on install* servers to pick up libxml security update
  • 14:04 moritzm: restarting squid on installurl downloaders to pick up libxml security update
  • 14:02 moritzm: restarting apache on krypton to pick up libxml security update
  • 13:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1053 original weight - T168661 (duration: 00m 46s)
  • 13:55 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: pagePreviews: Scale A/B test bucket sizes by 10 - T172291 (duration: 00m 46s)
  • 13:50 moritzm: restarting squid on url downloaders to pick up libxml security update
  • 13:34 moritzm: installing libxml2 security updates
  • 13:23 moritzm: installing ffmpeg security upadtes
  • 13:21 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1053 weight - T168661 (duration: 00m 46s)
  • 13:19 hashar: European SWAT completed
  • 13:16 gehel: restarting wdqs-blazegraph and wdqs-updater on all wdqs nodes for logback config change - T172710
  • 13:11 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable SandboxLink on cywiki - T173054 (duration: 00m 48s)
  • 12:55 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1053 weight - T168661 (duration: 00m 45s)
  • 12:51 gehel: restart wdqs-updater on wdqs2001 for config change
  • 12:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1053 weight - T168661 (duration: 00m 47s)
  • 12:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1053 with low weight - T168661 (duration: 00m 46s)
  • 12:05 marostegui: Upgrade MariaDB to 10.0.32 on db1053 - T168661
  • 12:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1053 - T168661 (duration: 00m 47s)
  • 11:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1055 original weight - T174265 (duration: 00m 46s)
  • 10:24 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give db1055 more traffic - T174265 (duration: 00m 47s)
  • 09:49 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give db1055 more traffic - T174265 (duration: 00m 47s)
  • 09:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1059 - T168661 (duration: 00m 47s)
  • 09:07 elukey: restart all jvm daemons on druid100[123] for security updates
  • 08:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give db1055 more traffic - T174265 (duration: 00m 48s)
  • 08:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give db1055 more traffic - T174265 (duration: 00m 47s)
  • 07:47 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1228.eqiad.wmnet
  • 07:47 marostegui: Upgrade MariaDB on db1059 to 10.0.32 - T168661
  • 07:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1059 for a MariaDB upgrade - T168661 (duration: 00m 53s)
  • 07:14 moritzm: powering down mw1294 for hardware diagnostics (T1674069
  • 07:12 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: mw1294.eqiad.wmnet
  • 07:12 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Slowly repool db1055 - T174265 (duration: 00m 52s)
  • 03:16 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Aug 30 03:16:09 UTC 2017 (duration 7m 19s)
  • 03:08 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 15m 42s)
  • 02:32 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 08m 28s)
  • 02:17 demon@tin: Finished deploy [gerrit/gerrit@f33c63f]: Testing, no-op (duration: 00m 07s)
  • 02:17 demon@tin: Started deploy [gerrit/gerrit@f33c63f]: Testing, no-op
  • 01:27 demon@tin: Synchronized wmf-config/InitialiseSettings.php: remove education program from legalteamwiki (duration: 00m 47s)
  • 00:53 demon@tin: Synchronized wmf-config/flaggedrevs.php: killing FR on testwiki (duration: 00m 46s)
  • 00:52 demon@tin: Synchronized dblists/flaggedrevs.dblist: killing FR on testwiki (duration: 00m 47s)
  • 00:05 cwd: turned dedupe off

2017-08-29

  • 23:50 cmjohnson1: reinstalling mw1228
  • 23:34 demon@tin: Synchronized php-1.30.0-wmf.15/extensions/Popups/: swat (duration: 00m 47s)
  • 23:33 demon@tin: Synchronized php-1.30.0-wmf.16/extensions/Popups/: swat (duration: 00m 50s)
  • 21:33 urandom: T169939: Decommission Cassandra: restbase1008-c.eqiad.wmnet
  • 21:16 andrewbogott: restarting designate-sink on labservices1001
  • 21:01 ejegg: updated DjangoBannerStats from 5963e7c to ce95eab
  • 20:56 ejegg: updated payments-wiki from 58d33b8 to bf97cd2
  • 20:49 godog: bounce varnish on cp1074 / cp1099 / cp1072 - mailbox lag
  • 20:38 ppchelko@tin: Finished deploy [changeprop/deploy@a57c79d]: Release a redis-based deduplicator in test mode. Attempt 2 (duration: 01m 17s)
  • 20:37 ppchelko@tin: Started deploy [changeprop/deploy@a57c79d]: Release a redis-based deduplicator in test mode. Attempt 2
  • 20:36 ppchelko@tin: Finished deploy [changeprop/deploy@ed0fadc]: Release a redis-based deduplicator in test mode (duration: 04m 28s)
  • 20:31 ppchelko@tin: Started deploy [changeprop/deploy@ed0fadc]: Release a redis-based deduplicator in test mode
  • 19:47 urandom: T169939: Decommission Cassandra: restbase1008-b.eqiad.wmnet
  • 19:34 ottomata: restarting analytics kafka brokers and all mirror maker processes to apply https://gerrit.wikimedia.org/r/#/c/374610/
  • 19:12 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.16
  • 19:08 ppchelko@tin: Finished deploy [restbase/deploy@7f2e55f]: Update CXServer endpoints config (duration: 06m 48s)
  • 19:02 ottomata: restarting main-eqiad -> analytics kafka mirror maker processes on analytics kafka brokers, something is not working...
  • 19:01 XenoRyet: updated Smashpig from 98c5516 to 8f4388d
  • 19:01 ppchelko@tin: Started deploy [restbase/deploy@7f2e55f]: Update CXServer endpoints config
  • 18:19 robh: attempting firmware upgrade on scs-a8-eqiad
  • 18:17 Pchelolo: depooling restbase1008,1009,1010,2003,2005 while cluster reshaping is going on
  • 18:15 demon@tin: Finished scap: bootstrap wmf.16 (duration: 45m 27s)
  • 18:15 urandom: T169939: Decommission Cassandra: restbase1008-a.eqiad.wmnet
  • 17:30 demon@tin: Started scap: bootstrap wmf.16
  • 17:23 ejegg: restarted donation queue consumer and thank you mailer
  • 17:20 Reedy: Disabled oathauth for KartikMistry on wikitech
  • 16:33 ejegg: updated civicrm from 7dfea0d to f00611d
  • 16:32 ejegg: turned off donation queue consumer and thank you mailer
  • 16:04 ejegg: restarted ingenico recurring charge job
  • 15:54 urandom: T169939: Decommission Cassandra: restbase2005-c.codfw.wmnet
  • 14:51 elukey: drop log.MobileWebUIClickTracking_10742159_15423246 from db1047 (archived on HDFS) - T172322
  • 14:33 marostegui: Shutdown db1055 to replace its BBU - T174265
  • 13:12 zeljkof: EU SWAT finished
  • 13:10 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add several HD logos (T150618) (duration: 00m 43s)
  • 13:09 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Add several HD logos (T150618) (duration: 00m 43s)
  • 12:37 elukey: restart kafka daemons on kafka1014 for jvm security updates
  • 12:31 godog: bounce pybal on lvs1003 to pick up config changes
  • 11:49 elukey: restart kafka daemons on kafka1013 for jvm security updates
  • 11:43 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1168.eqiad.wmnet
  • 11:30 elukey: restart java daemons on analytics100[1,2] (Hadoop Master nodes) for jvm updates
  • 11:08 marostegui: Stop MariaDB on db1097 to migrate it to file per table - T161088
  • 10:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1097 - T161088 (duration: 00m 43s)
  • 10:48 moritzm: installing libxml2 security updates
  • 10:31 moritzm: reimaging mw1168 (video scaler) to jessie
  • 10:16 jynus@tin: Synchronized wmf-config/db-eqiad.php: decom db1028 (duration: 00m 42s)
  • 10:15 jynus@tin: Synchronized wmf-config/db-codfw.php: decom db1028 (duration: 02m 48s)
  • 09:59 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1064 - T168661 (duration: 00m 43s)
  • 09:39 marostegui: Update MariaDB on db1064 to 10.0.32 - T168661
  • 09:38 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1064 for a mariadb upgrade - T168661 (duration: 00m 43s)
  • 09:33 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1169.eqiad.wmnet
  • 09:29 elukey: re-installed pmacct/librdkafka1/kafkacat on rhenium with stretch versions - T173489
  • 09:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1091 original weight - T168661 (duration: 00m 43s)
  • 08:53 ema: upgrading cache_text to varnish 4.1.8-1wm1
  • 08:45 akosiaris: reprepro copy calico, calico-cni from jessie-wikimedia to stretch-wikimedia (apt.wikimedia.org) T170119
  • 08:43 hashar: Restarting Jenkins for openjdk update
  • 08:42 elukey: restart yarn/hdfs daemons for openjdk security updates
  • 08:42 akosiaris: upload kubernetes_1.7.4-1 to apt.wikimedia.org/stretch-wikimedia/main T170119
  • 08:42 moritzm: restarting archiva to pick up openjdk security update
  • 08:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1091 weight - T168661 (duration: 00m 43s)
  • 08:35 gehel: restart wdqs-updater on wdqs2001
  • 08:32 hashar: apt-get upgrade on contint1001 and contint2001
  • 08:20 moritzm: reimaging mw1169 (video scaler) to jessie
  • 08:09 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1091 with low weight - T168661 (duration: 00m 43s)
  • 08:06 elukey: drop log.MobileWebUIClickTracking_10742159_15423246 from dbstore1002 to free space (table archived on HDFS) - T172322 T168303
  • 07:41 marostegui: Upgrade MariaDB on db1091 to 10.0.32 - T168661
  • 07:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1097 - T168661 (duration: 00m 42s)
  • 06:56 moritzm: installing ghostscript security updates on trusty (Debian already fixed)
  • 06:09 demon@tin: Synchronized scap/plugins/clean.py: no-op, for consistency (duration: 00m 43s)
  • 06:05 marostegui: Restart MariaDB on db1102 and db1095 to pick up new replication filters - T174385
  • 05:49 demon@tin: Pruned MediaWiki: 1.30.0-wmf.11 (duration: 02m 50s)
  • 02:34 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Aug 29 02:34:09 UTC 2017 (duration 6m 44s)
  • 02:27 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 08m 00s)
  • 00:25 urandom: T169939: Decommissioning restbase1010-a.eqiad.wmnet

2017-08-28

  • 23:37 reedy@tin: Synchronized php-1.30.0-wmf.15/extensions/Score/: consistency (duration: 00m 43s)
  • 23:31 reedy@tin: Synchronized php-1.30.0-wmf.15/extensions/Score/Score.body.php: output (duration: 00m 43s)
  • 23:27 reedy@tin: Synchronized php-1.30.0-wmf.15/extensions/Score/Score.body.php: output (duration: 00m 42s)
  • 23:23 reedy@tin: Synchronized wmf-config/CommonSettings.php: abc2ly in firejail (duration: 00m 43s)
  • 23:22 reedy@tin: Synchronized php-1.30.0-wmf.15/extensions/Score/Score.body.php: Fix escaping (duration: 00m 43s)
  • 23:13 cwd: re-enabled dedupe jobs
  • 23:13 ebernhardson@tin: Synchronized php-1.30.0-wmf.15/extensions/WikimediaEvents/: Turn off two cirrus AB tests T171742 T171214 (duration: 00m 44s)
  • 23:08 ebernhardson@tin: Synchronized docroot/noc/conf/categories-rdf.dblist: T173892: Add list for wikis that would have categories dumped into RDF (duration: 00m 43s)
  • 23:07 ebernhardson@tin: Synchronized dblists/categories-rdf.dblist: T173892: Add list for wikis that would have categories dumped into RDF (duration: 00m 43s)
  • 22:56 ejegg: turned fundraising orphan rectifier back on
  • 22:46 urandom: T169939: Decommissioning restbase1010-b.eqiad.wmnet
  • 21:48 krinkle@tin: Synchronized wmf-config/InitialiseSettings.php: I1c9c9c0e - Enable jQuery 3 on nlwiki, svwiki, plwiki (duration: 00m 44s)
  • 21:37 reedy@tin: Synchronized wmf-config/CommonSettings.php: Disable wgScoreAbc2Ly firejail T172582 (duration: 00m 44s)
  • 21:06 ejegg: switched nightly Ingenico audit from wr1 to wx file parsing
  • 21:06 reedy@tin: Synchronized wmf-config/CommonSettings.php: Run Score binaries in firejail T172582 (duration: 00m 43s)
  • 21:01 reedy@tin: Synchronized wmf-config/CommonSettings.php: Move email blacklist to meta:Email_blacklist (duration: 00m 45s)
  • 20:53 XioNoX: pushing "aggregate defaults discard" to all the cr* routers - T174364
  • 20:52 urandom: T169939: Decommissioning restbase1010-c.eqiad.wmnet
  • 20:16 ayounsi@tin: Finished deploy [librenms/librenms@5fea59c]: (no justification provided) (duration: 00m 05s)
  • 20:16 ayounsi@tin: Started deploy [librenms/librenms@5fea59c]: (no justification provided)
  • 20:16 XioNoX: upgrading librenms to 1.31.01
  • 20:08 bd808@tin: Finished deploy [striker/deploy@47b5e8a]: Deploying 47b5e8a (various bug fixes) (duration: 00m 26s)
  • 20:07 bd808@tin: Started deploy [striker/deploy@47b5e8a]: Deploying 47b5e8a (various bug fixes)
  • 19:24 ejegg: updated CiviCRM from 200f34b to 7dfea0d
  • 19:24 ottomata: restarting eventbus in eqiad to apply https://gerrit.wikimedia.org/r/#/c/372179/
  • 19:23 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Grant arbcomers abusefilter-view-private and abusefilter-log-private at cswiki T174357; Enable popups on en and de wiki for A/B test T172291 (duration: 00m 43s)
  • 19:15 herron: scb1001: restarted pdfrender service
  • 19:14 ottomata: restarting eventbus in codfw to verify https://gerrit.wikimedia.org/r/#/c/372179/
  • 19:09 ottomata: rolling restart and rebalances of main-* kafka clusters for https://gerrit.wikimedia.org/r/#/c/372179/
  • 19:03 niharika29@tin: Synchronized php-1.30.0-wmf.15/extensions/CodeMirror/: Fix exception on some combination of quotes T174060 (duration: 00m 44s)
  • 18:48 gwicke: restarted pdfrender instances in eqiad (T159922)
  • 18:48 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: pagePreviews: remove invalidated popup sampling rate variables https://gerrit.wikimedia.org/r/#/c/373171/7 (duration: 00m 43s)
  • 18:47 ayounsi@tin: Finished deploy [librenms/librenms@8c9da11]: (no justification provided) (duration: 00m 02s)
  • 18:47 ayounsi@tin: Started deploy [librenms/librenms@8c9da11]: (no justification provided)
  • 18:43 aaron@tin: Synchronized php-1.30.0-wmf.15/includes/jobqueue/jobs/HTMLCacheUpdateJob.php: 2d83569397 - Fix old regression in HTMLCacheUpdate de-duplication (duration: 00m 44s)
  • 18:37 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Fix incorrect Special:UserLogin in Popups blacklist https://gerrit.wikimedia.org/r/#/c/373696/ (duration: 00m 43s)
  • 18:33 XioNoX: upgrading librenms to 1.31
  • 18:32 ayounsi@tin: Finished deploy [librenms/librenms@8c9da11]: (no justification provided) (duration: 00m 05s)
  • 18:32 ayounsi@tin: Started deploy [librenms/librenms@8c9da11]: (no justification provided)
  • 18:30 ejegg: updated CiviCRM from c30ec13 to 200f34b
  • 18:21 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable wgEchoPerUserBlacklist at all wikis T173838 (duration: 00m 43s)
  • 18:15 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Upload wikipedia wordmark zh https://gerrit.wikimedia.org/r/#/c/373948 (duration: 00m 45s)
  • 18:14 niharika29@tin: Synchronized static/images/mobile/copyright/: Upload wikipedia wordmark zh https://gerrit.wikimedia.org/r/#/c/373948 (duration: 00m 44s)
  • 17:59 XioNoX: pushing "aggregate defaults discard" to *ams - T174364
  • 17:56 urandom: T169939: Decommission Cassandra: restbase2005-b.codfw.wmnet
  • 17:31 gehel: restarting wdqs-blazegraph and wdqs-updater for config change
  • 17:19 gehel: restarting wdqs-blazegraph and wdqs-updater for config change
  • 17:11 XioNoX: pushing "aggregate defaults discard" to cr2-knams - T174364
  • 17:09 gehel@tin: Finished deploy [wdqs/wdqs@90f4e2d]: (no justification provided) (duration: 02m 27s)
  • 17:07 gehel@tin: Started deploy [wdqs/wdqs@90f4e2d]: (no justification provided)
  • 17:07 gehel@tin: Started deploy [wdqs/wdqs@90f4e2d]: (no justification provided)
  • 16:46 demon@tin: Pruned MediaWiki: 1.30.0-wmf.10 (duration: 02m 41s)
  • 16:39 demon@tin: Pruned MediaWiki: 1.30.0-wmf.14 [keeping static files] (duration: 02m 00s)
  • 14:39 elukey: restart eventlogging_sync on dbstore1002 - issue after drop of old table
  • 14:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1099 with normal weight on s5 - T172679 (duration: 00m 44s)
  • 14:31 elukey: drop PageContentSaveComplete_5588433_15423246 from the log database on db1046 (m4-master) - T170720
  • 14:28 zeljkof: EU SWAT finished
  • 14:28 zfilipin@tin: Synchronized php-1.30.0-wmf.15/includes/jobqueue/jobs/HTMLCacheUpdateJob.php: SWAT: Disable rebound CDN purges for backlinks in HTMLCacheUpdateJob (T173710) (duration: 00m 45s)
  • 14:01 zeljkof: extending EU SWAT for 10 or so minutes to deploy two more patches
  • 13:55 elukey: restart kafka* daemons on kafka1012 for openjdk security updates (canary)
  • 13:52 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Log WikibaseQualityConstraints (T171281) (duration: 00m 44s)
  • 13:42 zfilipin@tin: Synchronized wmf-config/throttle-analyze.php: SWAT: Automatically include commons and wikidata in $wmgThrottlingExceptions (T163872) (duration: 00m 44s)
  • 13:36 zfilipin@tin: Synchronized wmf-config/CommonSettings.php: SWAT: throttle.php: Separate the throttling definitions from the exception values itself (T167040) (duration: 00m 44s)
  • 13:34 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: throttle.php: Separate the throttling definitions from the exception values itself (T167040) (duration: 00m 44s)
  • 13:33 zfilipin@tin: Synchronized wmf-config/throttle-analyze.php: SWAT: throttle.php: Separate the throttling definitions from the exception values itself (T167040) (duration: 00m 44s)
  • 13:27 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Allow sysops to grant/remove transwiki user group in dtywiki (T174226) (duration: 00m 44s)
  • 13:21 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add 4 URLs to $wgCopyUploadsDomain whitelist (T174152) (duration: 00m 45s)
  • 13:09 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Remove non-transparent background from dty.wiki logos (T174098) (duration: 00m 45s)
  • 12:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add more weight to db1099 on s5 - T172679 (duration: 00m 44s)
  • 12:43 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: mw1259.eqiad.wmnet
  • 12:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add more weight to db1099 on s5 - T172679 (duration: 00m 44s)
  • 12:07 gehel: restarting nginx for upgrade on wdqs*
  • 12:05 gehel: restarting nginx for upgrade on elastic* / relforge*
  • 12:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1099 pooled with low weight to s5 - T172679 (duration: 00m 45s)
  • 11:05 moritzm: installing libxml2 security updates
  • 08:41 moritzm: restarting squid3 on install1002
  • 08:16 marostegui: Ugprade MariaDB on s4 codfw master - db2051 to 10.0.32 - T168661
  • 08:10 marostegui: Stop MySQL on db1045 - T172679
  • 08:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1045 to clone db1099 from it - T172679 (duration: 00m 46s)
  • 07:51 moritzm: reimaging mw1259 to jessie (T145742)
  • 07:22 marostegui: Force re-learn cycle on db1055 - https://phabricator.wikimedia.org/T174265
  • 07:12 moritzm: installing openjdk security updates on notebook* hosts
  • 07:05 moritzm: installing openjdk security updates on meitnerium
  • 02:36 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Aug 28 02:36:23 UTC 2017 (duration 6m 54s)
  • 02:29 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 09m 36s)

2017-08-27

  • 20:35 ariel@tin: Finished deploy [dumps/dumps@39f9b52]: write output files to temp location and move into place when complete (duration: 00m 02s)
  • 20:34 ariel@tin: Started deploy [dumps/dumps@39f9b52]: write output files to temp location and move into place when complete
  • 15:46 akosiaris: upload kubernetes_1.4.6-7 to apt.wikimedia.org/jessie-wikimedia/main T170346
  • 05:30 marostegui: Force BBU relearn on db1055 - T174265
  • 00:45 jynus: correction last log s/db1051/db1055/
  • 00:42 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1051, hw issues, may get lag (duration: 00m 44s)

2017-08-26

  • 15:46 godog: bounce varnish on cp1073 - mailbox lag

2017-08-25

  • 19:09 gehel: actually not killing the "stuck discovery report-updater process on stat1005", it is already gone - T174110
  • 19:07 gehel: kill stuck discovery report-updater process on stat1005 - T174110
  • 16:03 moritzm: created cn=ciadmin group in LDAP to be used for privileged Jenkins access, initial members in ticket (T169557)
  • 14:29 godog: bounce varnish on cp1049 / cp1072 - mailbox lag
  • 14:16 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db1097 - T168661 (duration: 00m 44s)
  • 13:53 marostegui: Upgrade MariaDB on db1097 to 10.0.32 - T168661
  • 13:53 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db1097 - T168661 (duration: 00m 44s)
  • 13:40 marostegui: Upgrade MariaDB to 10.0.32 on db2019 - T168661
  • 13:37 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2037 - T168661 (duration: 00m 45s)
  • 13:24 marostegui: Upgrade MariaDB on db2037 to 10.0.32 - T168661
  • 13:02 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2037 - T168661 (duration: 00m 44s)
  • 12:51 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2044 - T168661 (duration: 00m 44s)
  • 12:29 marostegui: Upgrade MariaDB on db2044 - T168661
  • 12:29 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2044 - T168661 (duration: 00m 43s)
  • 12:19 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2058 - T168661 (duration: 00m 48s)
  • 11:48 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1028 (duration: 00m 47s)
  • 09:19 moritzm: restart thumbor to pick up libxml2 security updates
  • 09:12 godog: reimage restbase2001 to test new partman recipe - T169939
  • 08:41 moritzm: restarting apache/hhvm on canary app servers to pick up libxml security update
  • 08:00 volans: upgrading cumin to v1.0.0 on sarin/neodymium
  • 07:55 gehel: reimaging logstash1006 after change of failed disk - T173679
  • 07:49 jynus@tin: Synchronized wmf-config/db-eqiad.php: pool db1069 with more weight (duration: 00m 44s)
  • 07:00 jynus@tin: Synchronized wmf-config/db-eqiad.php: decom db1033, pool db1069 with low weight (duration: 00m 44s)
  • 06:55 jynus@tin: Synchronized wmf-config/db-codfw.php: decom db1033 (duration: 00m 43s)
  • 06:30 jynus@tin: Synchronized wmf-config/db-eqiad.php: formatting fixes (duration: 00m 44s)
  • 06:11 jynus@tin: Synchronized wmf-config/db-codfw.php: repool es2013, formatting fixes (duration: 00m 44s)
  • 06:01 marostegui: Upgrade MariaDB to 10.0.32 on db2058 - T168661
  • 06:00 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2058 - T168661 (duration: 00m 43s)
  • 05:30 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2065 - T168661 (duration: 00m 44s)
  • 05:18 moritzm: re-enable mw1260 (video scaler) for active job processing
  • 05:16 marostegui: Reboot db2065 to pick up new kernel
  • 05:08 marostegui: Upgrade MariaDB on db2065 - T168661
  • 05:03 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2065 for a MariaDB upgrade - T168661 (duration: 00m 44s)
  • 01:12 mattflaschen@tin: Synchronized php-1.30.0-wmf.15/extensions/Flow/modules/engine/components/board/features/flow-board-loadmore.js: T173807: Fix Flow infinite scroll (duration: 00m 45s)

2017-08-24

  • 23:17 thcipriani@tin: Synchronized php-1.30.0-wmf.15/extensions/TimedMediaHandler/TimedMediaHandler.php: SWAT: Disable Ogg Theora video transcodes in default config T172445 (duration: 00m 45s)
  • 22:45 demon@tin: Pruned MediaWiki: 1.30.0-wmf.13 [keeping static files] (duration: 02m 01s)
  • 22:24 ejegg: updated civicrm from fa882f2 to c30ec13 (pt-br and ja thank you letters)
  • 21:59 gwicke: restarted pdfrender service instances in eqiad / T159922
  • 21:54 maxsem@tin: Synchronized php-1.30.0-wmf.15/extensions/LoginNotify/: https://gerrit.wikimedia.org/r/#/c/373701/1 (duration: 00m 45s)
  • 21:13 maxsem@tin: Synchronized php-1.30.0-wmf.15/extensions/LoginNotify/: Logging: https://gerrit.wikimedia.org/r/#/c/373691/ (duration: 00m 44s)
  • 20:07 andrewbogott: stopping, starting, restarting etc. nodepool
  • 19:58 urandom: T169939: Decommission Cassandra: restbase2003-c.codfw.wmnet
  • 19:24 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.30.0-wmf.15
  • 19:14 andrewbogott: restarting rabbitmq-server on labcontrol1001
  • 18:38 ejegg: updated payments-wiki from bd9f730 to 58d33b8
  • 18:38 bblack: varnish backend restart - cp1074 - mailbox lag
  • 18:24 urandom: T169939: Decommission Cassandra: restbase2003-b.codfw.wmnet
  • 17:33 arlolra@tin: Finished deploy [parsoid/deploy@bd12f8a]: Updating Parsoid to 538dad7f (duration: 12m 24s)
  • 17:21 arlolra@tin: Started deploy [parsoid/deploy@bd12f8a]: Updating Parsoid to 538dad7f
  • 17:01 jynus: stopping and cloning db1033 to db1069
  • 16:52 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give normal weight to db1096 - T172679 (duration: 00m 47s)
  • 16:43 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give some more weight to db1096 - T172679 (duration: 00m 47s)
  • 16:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give some more weight to db1096 - T172679 (duration: 00m 47s)
  • 16:34 urandom: T169939: Decommission Cassandra: restbase2003-a.codfw.wmnet
  • 16:24 andrewbogott: apt-get upgrade and updated to MW 1.29.1 on wikitech-static. Rebooting to pick up kernel updates.
  • 15:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give some weight to db1096 - T172679 (duration: 00m 47s)
  • 15:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1096 depooled to s5 - T172679 (duration: 00m 47s)
  • 14:44 ladsgroup@tin: Synchronized php-1.30.0-wmf.15/extensions/Wikidata/extensions/Wikibase/client/includes/Changes/WikiPageUpdater.php: Reduce batch size in WikiPageUpdater (T173710) (duration: 00m 48s)
  • 14:14 phuedx: refreshLinks on bawiki (T173994)
  • 14:13 volans: updated cumin package in our APT to version cumin_1.0.0-1
  • 13:19 phuedx@tin: Synchronized wmf-config/InitialiseSettings-labs.php: T171853: Re-enable Page Previews for enwiki and dewiki on the Beta Cluster (duration: 00m 47s)
  • 13:13 phuedx@tin: Synchronized portals: (no justification provided) (duration: 00m 49s)
  • 13:12 phuedx@tin: Synchronized portals/prod/wikipedia.org/assets: (no justification provided) (duration: 00m 49s)
  • 12:45 gehel: killing discovery report updater on stats1005 (stuck since Aug 15) - T173333
  • 12:30 marostegui: Stop MySQL on db1026 to clone db1096 from it - T172679
  • 12:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1026 to clone db1096 from it T172679 (duration: 00m 47s)
  • 11:36 jynus: disable puppet on db1069 in preparation for reimage
  • 11:08 marostegui: Global rename Papa1234 → Karl-Heinz Jansen - T173859
  • 09:56 jynus@tin: Synchronized wmf-config/db-eqiad.php: repool db1078 with full weight (duration: 00m 46s)
  • 09:33 volans: executing https://wikitech.wikimedia.org/wiki/Cumin#Run_Puppet_only_if_last_run_failed to get rid of failed ones
  • 08:54 jynus: 16 minute test run of rebuildTermSqlIndex.php on terbium
  • 08:52 marostegui: Drop tables article_assessment, article_assessment_pages, article_assessment_ratings tables from testwiki - T173590
  • 07:24 Amir1: starting the run for rebuildTermIndex (T171460)
  • 02:50 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Aug 24 02:50:40 UTC 2017 (duration 7m 4s)
  • 02:43 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 05m 56s)
  • 02:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 09m 50s)
  • 01:46 aaron@tin: Synchronized php-1.30.0-wmf.15/includes/jobqueue: Deploy 752580637 (T173710) (duration: 00m 49s)

2017-08-23

  • 23:49 maxsem@tin: Synchronized php-1.30.0-wmf.14/extensions/LoginNotify/: SWAT fixes (duration: 00m 47s)
  • 23:47 maxsem@tin: Synchronized php-1.30.0-wmf.15/extensions/LoginNotify/: SWAT fixes (duration: 00m 47s)
  • 23:42 maxsem@tin: Synchronized php-1.30.0-wmf.14/extensions/CodeMirror/: https://gerrit.wikimedia.org/r/#/c/373324/ (duration: 00m 47s)
  • 23:41 maxsem@tin: Synchronized php-1.30.0-wmf.15/extensions/CodeMirror/: https://gerrit.wikimedia.org/r/#/c/373324/ (duration: 00m 49s)
  • 23:28 maxsem@tin: Synchronized php-1.30.0-wmf.15/resources/: https://gerrit.wikimedia.org/r/#/c/373391/ (duration: 00m 48s)
  • 23:27 maxsem@tin: Synchronized wmf-config/MetaContactPages.php: https://gerrit.wikimedia.org/r/#/c/373369/ (duration: 00m 48s)
  • 20:59 urandom: T169939: Truncating MCS tables
  • 20:26 thcipriani@tin: Synchronized php: group1 wikis to 1.30.0-wmf.15 (duration: 00m 46s)
  • 20:25 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.30.0-wmf.15
  • 20:13 jynus@tin: Synchronized wmf-config/db-eqiad.php: repool db1078 with low weight (duration: 00m 47s)
  • 20:12 thcipriani@tin: Finished scap: Revert ProofreadPage to db7507246665e69384c1d92af2aedc62263a5116 for wmf.15 T173520 (duration: 04m 11s)
  • 20:08 thcipriani@tin: Started scap: Revert ProofreadPage to db7507246665e69384c1d92af2aedc62263a5116 for wmf.15 T173520
  • 19:49 thcipriani@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Beta-only change: pagePreviews: Enable A/B test (BC-only (duration: 00m 47s)
  • 19:45 joal@tin: Finished deploy [analytics/refinery@f467ce1]: Bug fixing deploy (duration: 03m 16s)
  • 19:44 thcipriani@tin: Synchronized php-1.30.0-wmf.14/extensions/MobileFrontend: Verify the existence of `url` key when parsing lang objects T172316 (duration: 00m 56s)
  • 19:42 joal@tin: Started deploy [analytics/refinery@f467ce1]: Bug fixing deploy
  • 19:36 thcipriani@tin: Synchronized php-1.30.0-wmf.15/extensions/TimedMediaHandler: Enable WebM playback via ogv.js T172444 (duration: 00m 50s)
  • 19:22 kaldari: foreachwiki sql.php /srv/mediawiki/php/maintenance/archives/patch-ip_changes.sql
  • 19:19 kaldari: mwscript sql.php --wiki=testwiki /srv/mediawiki/php/maintenance/archives/patch-ip_changes.sql
  • 19:15 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Remove CitThisPage from blacklist for page previews T173865 (duration: 00m 47s)
  • 19:13 niharika29@tin: Synchronized php-1.30.0-wmf.15/extensions/TimedMediaHandler/: Enable WebM playback via ogv.js https://gerrit.wikimedia.org/r/#/c/373126/ (duration: 00m 49s)
  • 19:12 niharika29@tin: Synchronized php-1.30.0-wmf.14/extensions/TimedMediaHandler/: Enable WebM playback via ogv.js https://gerrit.wikimedia.org/r/#/c/373126/ (duration: 00m 50s)
  • 19:00 niharika29@tin: Synchronized php-1.30.0-wmf.14/extensions/MobileFrontend/: Verify the existence of key when parsing lang objects https://gerrit.wikimedia.org/r/#/c/373292/ (duration: 00m 49s)
  • 18:57 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Log channel for LoginNotify T173888 (duration: 00m 47s)
  • 18:55 ppchelko@tin: Finished deploy [changeprop/deploy@1998c10]: [Config] Disable mobile rerenders for non-wikipedia domains (duration: 01m 33s)
  • 18:54 ppchelko@tin: Started deploy [changeprop/deploy@1998c10]: [Config] Disable mobile rerenders for non-wikipedia domains
  • 18:52 niharika29@tin: Synchronized php-1.30.0-wmf.15/extensions/LoginNotify/: Log everything T173888 (duration: 00m 48s)
  • 18:43 gehel: manually running report updater for discovery golden data on stat1005 - T173333
  • 18:40 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Reinforce LoginNotify settings https://gerrit.wikimedia.org/r/#/c/372555/ (duration: 00m 47s)
  • 18:40 gehel: resetting permissions on stat1005:/srv/published-datasets/discovery - T173333
  • 18:39 niharika29@tin: Synchronized wmf-config/CommonSettings.php: Reinforce LoginNotify settings https://gerrit.wikimedia.org/r/#/c/372555/ (duration: 00m 47s)
  • 18:32 thcipriani: jenkins restarted, zuul should pick-up queue
  • 18:26 thcipriani: jenkins appears to be having some issues, trying to dump threads now and then restart to see if it helps
  • 18:13 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Allow crats to add people to accountcreator group on mw.or https://gerrit.wikimedia.org/r/#/c/373317/ (duration: 00m 49s)
  • 17:40 bblack: reboot cp4021
  • 16:38 cwd: disabled all dedupe
  • 16:26 demon@tin: Pruned MediaWiki: 1.30.0-wmf.11 [keeping static files] (duration: 01m 32s)
  • 16:24 bd808@tin: Finished deploy [striker/deploy@2f2dd7c]: Deploying 2f2dd7c "Tool account creation and more" (T128400, T149458, T159044, T164847, T167931, T168480, T173845) (duration: 00m 39s)
  • 16:23 bd808@tin: Started deploy [striker/deploy@2f2dd7c]: Deploying 2f2dd7c "Tool account creation and more" (T128400, T149458, T159044, T164847, T167931, T168480, T173845)
  • 16:22 demon@tin: Pruned MediaWiki: 1.30.0-wmf.9 (duration: 03m 44s)
  • 15:35 legoktm: added addshore to "integration" gerrit group (T173233)
  • 15:10 gehel: shutdown logstash1006 for disk replacement - T173679
  • 14:49 moritzm: installing texlive security updates on trusty (Debian already fixed)
  • 14:36 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4027.ulsfo.wmnet
  • 14:34 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4027.ulsfo.wmnet
  • 13:31 marostegui: Stop MySQL on db1041 to get it ready for decommission - T173915
  • 13:13 zeljkof: EU SWAT finished
  • 13:10 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Flow as a Beta feature on wawiki and wawikionary (T172947) (duration: 00m 48s)
  • 12:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1041 to decommission it - T173915 (duration: 00m 48s)
  • 12:15 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1041 to decommission it - T173915 (duration: 00m 48s)
  • 10:39 jynus: stopping and deleting s1 and s4 from dbstore2001
  • 10:06 joal@tin: Finished deploy [analytics/refinery@84d6ee4]: Regular deploy (1 month since last) (duration: 03m 30s)
  • 10:03 moritzm: upgrading remaining image scalers to luasandbox 2.0.13
  • 10:02 joal@tin: Started deploy [analytics/refinery@84d6ee4]: Regular deploy (1 month since last)
  • 09:36 godog: kill older and running tasks from gerrit queue, it was stuck
  • 09:10 moritzm: upgrading remaining API servers in to luasandbox 2.0.13
  • 09:08 godog: upload prometheus-blackbox-exporter 0.7.0+ds1-1~wmf1 to jessie-wikimedia, backported - T169860
  • 08:35 marostegui: Stop Upgrade MySQL on db2073 to 10.0.32 - T168661
  • 08:22 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2073 - T168661 (duration: 00m 59s)
  • 08:18 moritzm: upgrading remaining job runners to luasandbox 2.0.13
  • 07:12 marostegui: Global rename: Opdire657 → Sakiv - T173834
  • 06:55 moritzm: upgrading remaining app servers in to luasandbox 2.0.13
  • 03:15 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Aug 23 03:15:36 UTC 2017 (duration 7m 6s)
  • 03:08 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 15m 11s)
  • 02:32 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 08m 03s)

2017-08-22

  • 23:39 thcipriani@tin: Synchronized php-1.30.0-wmf.15/extensions/Linter/includes/LintErrorsPager.php: SWAT: Fix up 11f4a97ba6bcd0c1de (duration: 00m 49s)
  • 23:35 thcipriani@tin: Finished scap: SWAT: Fix typo in the notification message (duration: 21m 48s)
  • 23:13 thcipriani@tin: Started scap: SWAT: Fix typo in the notification message
  • 23:10 thcipriani@tin: Synchronized php-1.30.0-wmf.14/extensions/Popups/includes/PopupsHooks.php: SWAT: Remove aborting of BeforePageDisplay hook T173411 (duration: 00m 49s)
  • 21:18 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Retry CodeMirror deployment T170966 (duration: 00m 49s)
  • 20:41 ejegg: updated CiviCRM from 4aa177b to fa882f2
  • 20:25 mutante: restbase-dev* - puppet runs fail due to E: Version '3.11.0' for 'cassandra' was not found
  • 20:00 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.30.0-wmf.15
  • 19:56 urandom: starting cassandra restbase2004-a and restbase2006-c, OOMs
  • 19:46 thcipriani@tin: Finished scap: testwiki to php-1.30.0-wmf.15 and rebuild l10n cache (duration: 45m 55s)
  • 19:00 thcipriani@tin: Started scap: testwiki to php-1.30.0-wmf.15 and rebuild l10n cache
  • 18:50 robh: labservies1002 update completed (not 1003, typo)
  • 18:50 robh: labservies1003 update completed
  • 18:40 robh: firmware update of labservices1002 in progress
  • 17:54 andrewbogott: removing obsolete apache2 and puppetmaster packages from labcontrol boxes for https://phabricator.wikimedia.org/T171786
  • 17:07 thcipriani: starting branch cut for 1.30.0-wmf.15
  • 13:58 godog: bounce varnish on cp1074 / cp1049 / cp1073 - mailbox problems
  • 13:48 zeljkof: EU SWAT finished
  • 13:45 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: relatedArticles: Tidy up config (T165991) (duration: 00m 44s)
  • 13:44 zfilipin@tin: Synchronized wmf-config/CommonSettings.php: SWAT: relatedArticles: Tidy up config (T165991) (duration: 00m 44s)
  • 13:29 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wmgUseWikimediaShopLink to true for ptwiki (T173768) (duration: 00m 45s)
  • 10:41 moritzm: upgrading hhvm-luasandbox on mw1293-mw1295 (image scalers)
  • 10:21 Amir1: another run of rebuildTermSqlIndex (T171460)
  • 09:54 moritzm: upgrading hhvm-luasandbox on mw1189-mw1208 (API servers)
  • 09:42 marostegui: Renaming user Darwinius → DarwIn - T173159
  • 09:35 moritzm: upgrading hhvm-luasandbox on mw1180-1188 and mw1209-mw1220 (app servers)
  • 09:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1064 - T172996 (duration: 00m 44s)
  • 08:59 akosiaris: upload apertium-bel-rus_0.2.0~r81186-1+wmf to apt.wikimedia.org/jessie-wikimedia/main
  • 08:55 moritzm: upgrading hhvm-luasandbox on mw1161-mw1167 (job runners)
  • 08:43 akosiaris: upload apertium-rus_0.1.0~r81184-1+wmf to apt.wikimedia.org/jessie-wikimedia/main
  • 08:43 akosiaris: upload apertium-bel_0.1.0~r81357-1+wmf to apt.wikimedia.org/jessie-wikimedia/main
  • 08:35 godog: bounce varnish on cp1072 - mailbox problems
  • 08:33 godog: bounce varnish on cp4015 - mailbox problems
  • 08:24 marostegui: Drop s4 from db1095 with NO replication - T172996
  • 08:23 moritzm: upgrading hhvm-luasandbox on deployment servers / script runners
  • 07:34 marostegui: Stop replication on db1064 sanitarium2 and sanitarium3 master to move labsdb1009,10 and 11 s4 from db1095 to db1102 - https://phabricator.wikimedia.org/T172996
  • 07:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1064 - T172996 (duration: 00m 52s)
  • 07:28 moritzm: installing ruby security updates on trusty (Debian already fixed)
  • 07:11 moritzm: installing c-ares security updates on trusty (Debian already fixed)
  • 06:54 moritzm: installing graphite2 (the image library, not the metrics tool) security updates on trusty (Debian already fixed)
  • 02:30 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Aug 22 02:30:47 UTC 2017 (duration 6m 55s)
  • 02:23 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 07m 11s)

2017-08-21

  • 22:50 robh: rebooting tin for firmware update since its idle
  • 22:20 mutante: praseodymium - installing BIOS upgrade, reboot
  • 22:11 mutante: xenon - installing BIOS upgrade
  • 21:58 mutante: cerium (cassandra test) - rebooting for firmware upgrade
  • 21:54 mutante: cerium - installing Dell BIOS upgrade (T162850)
  • 21:26 mutante: bast2001 - running Dell BIOS firmware upgrade (T162850)
  • 21:20 arlolra@tin: Finished deploy [parsoid/deploy@2210a38]: Updating Parsoid to 28a9a22b (duration: 10m 59s)
  • 21:09 arlolra@tin: Started deploy [parsoid/deploy@2210a38]: Updating Parsoid to 28a9a22b
  • 18:40 niharika29@tin: Synchronized dblists/pp_stage1.dblist: Roll page previews out to all wikis except en and de wiki T162672 (duration: 00m 44s)
  • 18:34 niharika29@tin: Synchronized wmf-config/CommonSettings.php: Enable OOjs UI EditPage on all wikis https://gerrit.wikimedia.org/r/#/c/366868/ (duration: 00m 44s)
  • 18:33 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable OOjs UI EditPage on all wikis https://gerrit.wikimedia.org/r/#/c/366868/ (duration: 00m 44s)
  • 18:30 niharika29@tin: Synchronized wmf-config/Wikibase.php: Wikibase on deployment-prep: Exclude non-existent wikis from clientDbList T173571 (duration: 00m 44s)
  • 18:25 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Increase AbuseFilter autodisable thresholds for Meta-Wiki T173633 (duration: 00m 44s)
  • 18:22 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Add CookBook and Cookbook Talk NS on hiwikibooks T173398 (duration: 00m 45s)
  • 18:16 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Administrators to add/remove 'transwiki' at nowiktionary T172365 (duration: 00m 45s)
  • 18:12 niharika29@tin: Synchronized static/images/: Set project logo for wikimania2018wiki T173042 (duration: 00m 44s)
  • 18:11 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Set project logo for wikimania2018wiki T173042 (duration: 00m 44s)
  • 17:22 mobrovac@tin: Finished deploy [cxserver/deploy@f43ef96]: Bring back cxserver on scb2001 to a stable state - T173038 (duration: 00m 14s)
  • 17:22 mobrovac@tin: Started deploy [cxserver/deploy@f43ef96]: Bring back cxserver on scb2001 to a stable state - T173038
  • 17:12 gehel@tin: Finished deploy [wdqs/wdqs@a1c4f1f]: (no justification provided) (duration: 02m 27s)
  • 17:10 gehel@tin: Started deploy [wdqs/wdqs@a1c4f1f]: (no justification provided)
  • 15:40 bblack: cp1099 - varnish backend restart for mailbox lag
  • 14:55 mobrovac: cxserver depool scb2001 to debug failed checks - T173038
  • 14:54 mobrovac@tin: Finished deploy [cxserver/deploy@1065ffe]: Deploy 1065ffe2 to canary scb2001 for debugging - T173038 (duration: 00m 18s)
  • 14:53 mobrovac@tin: Started deploy [cxserver/deploy@1065ffe]: Deploy 1065ffe2 to canary scb2001 for debugging - T173038
  • 14:52 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1079 - T163190 (duration: 00m 44s)
  • 14:04 zeljkof: EU SWAT finished
  • 14:04 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add HD logos for srwikisource, update them too (T172268) (duration: 00m 44s)
  • 14:02 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Add HD logos for srwikisource, update them too (T172268) (duration: 00m 44s)
  • 13:53 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Update logos for srwikinews, add HD version for them (T172255) (duration: 00m 49s)
  • 13:52 jynus: stop dbstore2001 mariadb@s4, start mariadb@s7
  • 13:52 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Update logos for srwikinews, add HD version for them (T172255) (duration: 00m 44s)
  • 13:46 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set X-Frame-Options: SAMEORIGIN if UploadWizard enabled (T173631) (duration: 00m 44s)
  • 13:46 ottomata: adding index on (database, rev_timestamp) on mediawiki_page_create_2 table on db1047: T170990
  • 13:37 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Update logos for srwiktionary, add HD logos for srwiktionary (T172245) (duration: 00m 44s)
  • 13:36 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Update logos for srwiktionary, add HD logos for srwiktionary (T172245) (duration: 00m 45s)
  • 13:27 jynus: stop dbstore2001 mariadb@s7
  • 13:27 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add new logo for the Baskhir Wikibooks (T173471) (duration: 00m 44s)
  • 13:26 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Add new logo for the Baskhir Wikibooks (T173471) (duration: 00m 44s)
  • 13:26 ottomata: adding index on (database, rev_timestamp) on mediawiki_page_create_2 table on dbstore1002: T170990
  • 13:11 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Reopen bawikibooks (T173471) (duration: 00m 44s)
  • 13:10 zfilipin@tin: Synchronized dblists/closed.dblist: SWAT: Reopen bawikibooks (T173471) (duration: 00m 44s)
  • 12:52 marostegui: Stop replication on db1079 and db1041 to compare their data - T163190
  • 12:52 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1079 - T163190 (duration: 00m 45s)
  • 12:35 marostegui: Stop MySQL on db1015 to decommission it - T173570
  • 09:50 jynus: restart dbstore2001 mariadb@s1
  • 09:40 jynus: restart dbstore2001 mariadb@x1
  • 09:29 moritzm: upgrading hhvm-luasandbox on mw1262-1265
  • 09:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1015 - T173570 (duration: 00m 44s)
  • 09:15 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1015 - T173570 (duration: 00m 44s)
  • 09:01 moritzm: upgrading hhvm-luasandbox on mw1261
  • 08:08 marostegui: Rename tables article_assessment, article_assessment_pages, article_assessment_ratings tables from testwiki on db1078 - T173590
  • 07:42 marostegui: Drop cx_drafts table from x1 - T172364
  • 07:13 marostegui: Stop s4 replication thread on db1095 - T172996
  • 07:09 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2047 (duration: 00m 45s)
  • 02:36 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Aug 21 02:36:58 UTC 2017 (duration 7m 3s)
  • 02:29 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 08m 57s)

2017-08-19

  • 13:17 Amir1: another run: (Hotwc3 → HotWC3) (Lamia Bahy → Albedo11) (Monóxido de carbono → Roquetero) (PaulMichaels → PaulBenario) (Rodrigo.dst → RodrigoTavares) (Sadia Tasnim (Moyna) → মুহাম্মদ সুমন মাহমুদ) (Syou 18331322 → Ms3102) (TzvetelinaOOD1 → Tzveti1) (World Para Taekwondo → TKD at World Para Taekwondo) (Yaellerner → Ya1levy777) (平井 俊光 → Toshimit) (T173419)
  • 13:00 Amir1: running the script for ("Clopper228" "CGminded") and ("Gregory.lussier" "StevenSmith83473") (T173419)
  • 12:47 Amir1: ladsgroup@terbium:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=metawiki --logwiki=metawiki "Bolsée" "Kathmandu2017" (T173419)
  • 03:21 krinkle@tin: Synchronized wmf-config/InitialiseSettings.php: Enable jQuery 3 on test.wikidata and mediawiki.org - Id4d42d8c53 (duration: 00m 45s)

2017-08-18

  • 21:05 mutante: gerrit2001 - restarted gerrit again after reverting gerrit:372426, using systemctl commands, not 'service' or init.d
  • 20:55 mutante: restarting gerrit on gerrit2001
  • 20:48 demon@tin: Synchronized scap/plugins/clean.py: Completeness (duration: 00m 44s)
  • 20:43 RainbowSprinkles: prior thing was a no-op, testing
  • 20:42 demon@tin: Pruned MediaWiki: 1.30.0-wmf.9 [keeping static files] (duration: 01m 43s)
  • 19:03 bblack: cp1074 - varnish backend restart (mailbox lag)
  • 19:01 bblack: reboot cp4021-8
  • 18:21 bblack: puppeting cp4021-8 - expect possibility of ipsec alerts, etc...
  • 13:32 Amir1: another run of /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=item --from-id $(tail -100 /tmp/rebuildTermSqlIndex.log | grep -E "Processed up to page (\d+?)" | sed -E "s/Processed up to page //; s/ \(Q.+?//" | tail -1) >>/tmp/rebuildTermSqlIndex.log 2>&1
  • 13:26 Amir1: ladsgroup@terbium:~$ mwscript namespaceDupes.php --wiki=hiwikiversity (T172977)
  • 12:44 Amir1: start of ladsgroup@terbium:~$ time /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=item
  • 11:36 jynus: change topology of dbstore2001:x1 and dbstore2002:x1
  • 11:00 elukey: reboot dbstore1002 for kernel updates
  • 10:34 elukey: restart mysql on dbstore1002 - attempt to reclaim space after big table drop (stop slaves and el_sync, check running queries, stop mysql, check process, start mysql)
  • 09:55 Amir1: one small pass of ladsgroup@terbium:~$ time /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=entity (T171460)
  • 09:46 Amir1: ladsgroup@terbium:~$ time /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=property (T171460)
  • 09:42 moritzm: installing libmspack security updates
  • 09:16 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2077 - T168409 (duration: 00m 44s)
  • 07:28 marostegui: Stop MySQL on db2077 to copy it to dbstore2001 - T168409
  • 07:26 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2077 - T168409 (duration: 00m 44s)
  • 07:20 moritzm: installing openjdk-7 security updates on trusty
  • 06:58 jynus: and nginx
  • 06:55 jynus: systemctl restart squid3.service on install2002, apt seem stuck on some servers
  • 06:24 jynus: stopping and upgrading db2033
  • 05:43 jynus: upgrading and restarting all mariadb instances on dbstore2002
  • afk: updated fundraising dash from 696a3ff to 7dfc969
  • 00:48 ebernhardson@tin: Synchronized php-1.30.0-wmf.14/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T171213: Increase sampling rate of cirrus satisfaction schema (again) to 1k per bucket per day (duration: 00m 44s)
  • 00:18 ebernhardson@tin: Synchronized php-1.30.0-wmf.14/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T171213: Increase sampling rate of cirrus satisfaction schema (duration: 00m 44s)

2017-08-17

  • 23:31 thcipriani@tin: Synchronized php-1.30.0-wmf.14/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Revert "Disable cirrus MLR ab test" (duration: 00m 44s)
  • 23:21 thcipriani@tin: Synchronized php-1.30.0-wmf.14/resources/src/mediawiki.rcfilters/dm/mw.rcfilters.dm.FilterGroup.js: SWAT: RCFilters: Fix validation for single_option groups T173303 (duration: 00m 44s)
  • 23:14 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Reapply "Remove temporary wgStructuredChangeFiltersEnableExperimentalViews setting" (duration: 00m 45s)
  • 22:10 ppchelko@tin: Finished deploy [changeprop/deploy@2c553a6]: Lower the concurrecy for transcludes to decrease cassandra load during cluster reshaping (duration: 01m 14s)
  • 22:09 thcipriani@tin: Synchronized php-1.30.0-wmf.14/includes/specials/SpecialNewpages.php: Restore the newFromId() approach in SpecialNewpages::feedItemDesc T173541 (duration: 00m 46s)
  • 22:09 ppchelko@tin: Started deploy [changeprop/deploy@2c553a6]: Lower the concurrecy for transcludes to decrease cassandra load during cluster reshaping
  • 22:07 gwicke: restarting pdfrender on scb100* nodes
  • 21:38 ejegg: updated dash from bec0077 to 696a3ff
  • 21:28 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable LoginNotify on sites without Echo too. :|https://gerrit.wikimedia.org/r/#/c/372456/ (duration: 00m 44s)
  • 21:19 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable LoginNotify finally! Yippeeeeee https://gerrit.wikimedia.org/r/#/c/372450/ (duration: 00m 45s)
  • 21:17 bblack: cp1063: varnish backend restart (mailbox lag)
  • 20:07 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.30.0-wmf.14
  • 19:28 chasemp: disable puppet for cloud things for some careful refactor merging
  • 19:14 thcipriani@tin: Synchronized php: group1 wikis to wmf.14 (duration: 00m 46s)
  • 19:12 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to wmf.14
  • 19:07 thcipriani@tin: Finished scap: ProofReadPage Revert to db7507246665e69384c1d92af2aedc62263a5116 T173520 (duration: 06m 13s)
  • 19:01 thcipriani@tin: Started scap: ProofReadPage Revert to db7507246665e69384c1d92af2aedc62263a5116 T173520
  • 18:28 ebernhardson: restart logstash on logstash1003 to see why its not reading EventError messages from kafka
  • 18:24 niharika29@tin: Synchronized php-1.30.0-wmf.14/extensions/RevisionSlider/: Revert Reintroduce hover and bar clicking - https://gerrit.wikimedia.org/r/#/c/372384/ (duration: 00m 48s)
  • 18:16 jynus: upgrading and restarting all mariadb instances on dbstore2001
  • 18:12 niharika29@tin: Synchronized php-1.30.0-wmf.14/extensions/LoginNotify/: Log usage statistics https://gerrit.wikimedia.org/r/#/c/372214/ (duration: 00m 51s)
  • 18:03 thcipriani@tin: Synchronized php-1.30.0-wmf.14/extensions/UploadWizard/UploadWizard.config.php: SWAT: Preserve array keys (language keys) when sorting the language dropdown T173522 (duration: 00m 51s)
  • 17:53 jynus: restarting mariadb (dbstore2001:x1) to test new buffer pool configuration
  • 17:20 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis back to wmf.13 now T173520
  • 17:10 RainbowSprinkles: gerrit: cpu spikes, lots of large gc logs. Looking into it. (nb: things might be a little slow, but it is /up/)
  • 16:54 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis back to wmf.14 now for T164173
  • 16:51 thcipriani@tin: Synchronized php-1.30.0-wmf.14/includes/jobqueue/jobs/RefreshLinksJob.php: Avoid lock acquisition errors for multi-title refreshlinks jobs T173462 (duration: 00m 51s)
  • 16:45 ejegg: updated SmashPig from c501f53 to 98c5516
  • 16:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db2077 - T170662 (duration: 00m 49s)
  • 16:24 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2077 - T170662 (duration: 00m 51s)
  • 16:16 urandom: T169939: RESTBase: converting (33) new keyspaces to time-windowed compaction
  • 16:11 urandom: T169939: Lower eqiad compaction throughput from 20MB/s to 15MB/s
  • 15:46 godog: reimage restbase2001 - T169939
  • 15:42 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2001.codfw.wmnet
  • 14:32 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp3036.*
  • 13:30 zeljkof: EU SWAT finished
  • 13:24 zfilipin@tin: Synchronized static/favicon/wikiversity.ico: SWAT: Update wikiversity favicon (T160491) (duration: 00m 50s)
  • 13:15 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Change $wgArticleCountMethod to any for srwikiquote (T172974) (duration: 00m 51s)
  • 13:08 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: Add one throttling exception (T173444) (duration: 00m 51s)
  • 12:38 moritzm: installing libgd/libsoup security updates
  • 12:08 urandom: T169939: Decommissioning Cassandra/restbase2001-c.codfw.wmnet
  • 12:07 urandom: T169939: Decommissioning Cassandra/restbase2001-b.codfw.wmnet
  • 10:29 gilles: Thumbor stress test finished
  • 10:20 gilles: Load testing Thumbor
  • 10:13 godog: roll-restart nginx on thumbor to apply https://gerrit.wikimedia.org/r/372199
  • 08:36 godog: roll-restart thumbor to apply webp support change
  • 08:30 godog: restart varnish on cp1062 to fix mailbox lag
  • 08:05 godog: reboot ms-be2021, unreachable on network
  • 06:24 marostegui: Stop slave on db2047 to fix duplicate keys - T151029
  • 05:55 marostegui: Stop replication on db2077 to fix duplicate entries - T151029
  • 05:48 marostegui: Stop replication in sync on db1078 and db1015 - T164488
  • 03:11 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Aug 17 03:11:13 UTC 2017 (duration 7m 18s)
  • 03:03 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 03m 58s)
  • 02:50 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.13) (duration: 08m 33s)
  • 02:27 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 52s)
  • 00:45 urandom: T169939: Decommissioning Cassandra/restbase2001-b.codfw.wmnet

2017-08-16

  • 22:30 urandom: T169939: Decommissioning Cassandra/restbase2001-a.codfw.wmnet
  • 22:14 urandom: T169939: Cleaning up wikipedia parsoid snapshots
  • 22:13 urandom: T169939: Rolling restart of Cassandra complete
  • 21:38 bawolff: deleting private info from securepoll_votes that the script missed due to ref-integ issues for old elections (T173393)
  • 21:37 thcipriani: train is on hold pending resolution of T173462
  • 21:33 bawolff: deleting private info in enwiki arbcom1_vote table (T173393)
  • 21:30 urandom: T169939: Rolling restart of Cassandra instances, eqiad, rack d
  • 21:22 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: revert group1 wikis to 1.30.0-wmf.14 for T173462
  • 21:21 thcipriani@tin: Synchronized php: revert group1 wikis to 1.30.0-wmf.14 for T173462 (duration: 00m 47s)
  • 20:46 urandom: T169939: Rolling restart of Cassandra instances, eqiad, rack b
  • 20:45 arlolra@tin: Finished deploy [parsoid/deploy@a9dc803]: Updating Parsoid to 1832a78e (duration: 08m 52s)
  • 20:37 arlolra@tin: Started deploy [parsoid/deploy@a9dc803]: Updating Parsoid to 1832a78e
  • 19:57 urandom: T169939: Rolling restart of Cassandra instances, eqiad, rack a
  • 19:52 MaxSem: Manually cleaning up PI on enwiki (T173393)
  • 19:37 thcipriani@tin: Synchronized php: group1 wikis to 1.30.0-wmf.14 (duration: 00m 46s)
  • 19:35 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.30.0-wmf.14
  • 19:06 urandom: T169939: Rolling restart of Cassandra instances, codfw, rack d
  • 18:57 thcipriani@tin: Synchronized wmf-config/CirrusSearch-common.php: SWAT: cirrus Tune ordering of crossproject search results on enwiki T171803 PART II (duration: 00m 50s)
  • 18:56 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: cirrus Tune ordering of crossproject search results on enwiki T171803 PART I (duration: 00m 51s)
  • 18:41 thcipriani@tin: Synchronized wmf-config: SWAT: Apply token count limits to phrase queries on all wikis T172653 (duration: 00m 53s)
  • 18:30 thcipriani@tin: Synchronized php-1.30.0-wmf.13/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Disable cirrus MLR ab test T171214 (duration: 00m 50s)
  • 18:29 thcipriani@tin: Synchronized php-1.30.0-wmf.14/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Disable cirrus MLR ab test T171214 (duration: 00m 51s)
  • 18:20 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: JobQueueEventBus: Enable group1 T163380 (duration: 00m 54s)
  • 18:18 urandom: T169939: Rolling restart of Cassandra instances, codfw, rack c
  • 17:31 urandom: T169939: Rolling restart of Cassandra instances, codfw, rack b
  • 17:07 demon@tin: Synchronized php-1.30.0-wmf.14/extensions/AntiSpoof/SpoofUser.php: T173394 (duration: 00m 51s)
  • 15:54 marostegui: Stop MySQL and shutdown db1078 for HW checks - T173365
  • 15:43 elukey: drop PageContentSaveComplete_5588433_15423246 from db1047 and dbstore1002 (analytics-slaves)
  • 15:41 godog: delete outdated CFs cassandra metrics from graphite2002 and graphite1003
  • 15:35 marostegui: Rename cx_drafts table on db1029 - T172364
  • 15:09 godog: restart varnish on cp1072 to clear mailbox lag
  • 14:56 godog: restart varnish on cp1099 to clear mailbox lag
  • 14:48 godog: restart varnish on cp1074 to clear mailbox lag
  • 14:44 godog: restart varnish on cp1049 to clear mailbox lag
  • 14:38 gilles: Thumbor stress test finished
  • 14:29 gilles: Stress testing Thumbor from single IP
  • 13:57 zeljkof: EU SWAT finished
  • 13:35 zfilipin@tin: Synchronized dblists/pp_stage1.dblist: SWAT: pagePreviews: Deploy to next 100 stage 1 wikis (T162672) (duration: 00m 50s)
  • 13:09 zfilipin@tin: Synchronized portals: (no justification provided) (duration: 00m 52s)
  • 13:08 zfilipin@tin: Synchronized portals/prod/wikipedia.org/assets: (no justification provided) (duration: 00m 52s)
  • 12:25 Amir1: ladsgroup@terbium:~$ time /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki testwikidatawiki --entity-type=property (T172776, T171460)
  • 12:25 moritzm: powercycling cp3036
  • 12:24 marostegui: Compressing InnoDB on db2077 - T168409
  • 12:22 jmm@puppetmaster1001: conftool action : set/pooled=no; selector: cp3036.esams.wmnet
  • 10:46 marostegui: Stop replication in sync on db1015 and db1078 - T164488
  • 10:42 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2076 - T170662 (duration: 00m 51s)
  • 10:03 godog: copy ubuntu-font-family-sources to stretch-wikimedia - T170817
  • 08:37 marostegui: Drop wikigrok tables from s1, s3 and s5 - T172020
  • 08:12 marostegui: Stop MySQL on db2047 to copy its content to db2077 - T170662
  • 08:10 moritzm: bounced pdfrender on scb1004 (T159922)
  • 08:07 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2047 - T170662 (duration: 01m 06s)
  • 07:25 marostegui: Stop MySQL on db2076 to copy its content to dbstore2001 - T168409
  • 07:08 elukey: executed sudo find -type f -mtime +30 -exec rm {} \; in /var/log/carbon to free some space
  • 06:40 marostegui: Run pt-table-checksum on s3 for revision table - T164488
  • 06:01 marostegui: Stop replication on db2076 to fix duplicate entries - T151029
  • 03:35 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Aug 16 03:35:43 UTC 2017 (duration 7m 21s)
  • 03:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 16m 32s)
  • 02:51 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.13) (duration: 08m 08s)
  • 02:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 51s)
  • 00:43 dereckson@tin: Synchronized php-1.30.0-wmf.12/extensions/Wikidata/Wikidata.php: Explicitly load badges extension (Gerrit:372088) (duration: 00m 51s)
  • 00:42 dereckson@tin: Synchronized php-1.30.0-wmf.14/extensions/Wikidata/Wikidata.php: Explicitly load badges extension (Gerrit:372051) (duration: 00m 51s)
  • 00:38 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Enable Timeless on three French wikis (T154371) + Fixes for Wikidata: Remove wbq_evaluation logging, Update Wikidata property blacklist (Gerrit:367913 and Gerrit:370846) (duration: 00m 53s)
  • 00:02 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Enable WikidataPageBanner on test wikis (T173388) (duration: 00m 51s)

2017-08-15

  • 23:58 dereckson@tin: Synchronized php-1.30.0-wmf.14/skins/Timeless/resources/screen-common.less: Fix messed up recent changes/watchlist legends (T173151) (duration: 00m 50s)
  • 23:55 dereckson@tin: Synchronized php-1.30.0-wmf.13/skins/Timeless/resources/screen-common.less: Fix messed up recent changes/watchlist legends (T173151) (duration: 00m 54s)
  • 23:14 Dereckson: Ran updateArticleCount.php on sr.wikiquote (T172974)
  • 23:12 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Create a few of namespace aliases for hiwikiversity (T172977) (duration: 00m 53s)
  • 22:50 ppchelko@tin: Started restart [changeprop/deploy@444223d]: (no justification provided)
  • 22:45 Pchelolo: scb - codfw: enabling puppet and starting changeprop back T169939
  • 21:40 mobrovac: scb - codfw: disabling puppet and stopping changeprop to release load on Cassandra while dropping old tables - T169939
  • 21:26 urandom: Starting Cassandra restbase2005-b
  • 19:24 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.14
  • 19:19 mobrovac@tin: Finished deploy [restbase/deploy@1139d00]: Use only new parsoid tables (before the removal of old ones), part #2 - T169939 (duration: 02m 50s)
  • 19:16 mobrovac@tin: Started deploy [restbase/deploy@1139d00]: Use only new parsoid tables (before the removal of old ones), part #2 - T169939
  • 19:14 demon@tin: Finished scap: bootstrap wmf.14 v2.0 (duration: 30m 56s)
  • 19:04 mobrovac@tin: Started deploy [restbase/deploy@1139d00]: Use only new parsoid tables (before the removal of old ones) - T169939
  • 19:03 mobrovac@tin: Finished deploy [restbase/deploy@1139d00] (staging): Use only new parsoid tables (before the removal of old ones) - T169939 (duration: 04m 39s)
  • 18:58 mobrovac@tin: Started deploy [restbase/deploy@1139d00] (staging): Use only new parsoid tables (before the removal of old ones) - T169939
  • 18:43 demon@tin: Started scap: bootstrap wmf.14 v2.0
  • 18:39 demon@tin: scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_3982832257" --threads=10 --lang en --quiet' returned non-zero exit status 255 (duration: 03m 05s)
  • 18:35 demon@tin: Started scap: bootstrap wmf.14
  • 18:18 demon@tin: Pruned MediaWiki: 1.30.0-wmf.12 [keeping static files] (duration: 02m 47s)
  • 17:59 bsitzmann@tin: Finished deploy [mobileapps/deploy@34a1304]: Update mobileapps to 33b80dd (T172829 T152441 T172021 T103362) (duration: 04m 00s)
  • 17:55 bsitzmann@tin: Started deploy [mobileapps/deploy@34a1304]: Update mobileapps to 33b80dd (T172829 T152441 T172021 T103362)
  • 15:21 mobrovac: restarting pdfrender on scb1001, added some debug messages to help us diagnose T159922
  • 13:42 gehel: cleanup of old leftover logs on deployment-logstash2:/var/log/logstash
  • 13:21 moritzm: bounced pdfrender on scb1001 (T159922)
  • 11:32 moritzm: installing Linux updates on stretch systems (no reboots yet)
  • 10:45 moritzm: installing PHP security updates on trusty
  • 10:09 jynus: rebooting db1078
  • 09:39 jynus@tin: Synchronized wmf-config/db-eqiad.php: depool db1078 (duration: 00m 49s)
  • 07:56 moritzm: installing cvs security updates
  • 07:18 elukey: restart pdfrender on scb1003
  • 02:56 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Aug 15 02:56:14 UTC 2017 (duration 6m 42s)
  • 02:49 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.13) (duration: 07m 53s)
  • 02:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 58s)

2017-08-14

  • 22:16 Dereckson: Previous log entry is related to Gerrit:371960 and Gerrit:371961.
  • 22:15 dereckson@tin: Synchronized php-1.30.0-wmf.11/extensions/EventBus/JobQueueEventBus.php: JobQueueEventBus: Populate the database field + not set properties are accessed (duration: 00m 48s)
  • 20:44 ppchelko@tin: Finished deploy [restbase/deploy@4d6c706]: Temporary fallback to the new storage buckets before truncation (duration: 09m 35s)
  • 20:35 ppchelko@tin: Started deploy [restbase/deploy@4d6c706]: Temporary fallback to the new storage buckets before truncation
  • 17:09 gehel: resetting mode on stat1005:/srv/published-datasets/discovery recursively
  • 16:29 elukey: execute sudo find -type f -mtime +60 -exec rm {} \; in /var/log/carbon on graphite2001 to free some space in /
  • 13:32 elukey: Execute systemctl mask nfacctd on rhenium.wikimedia.org for T172681
  • 13:08 zeljkof: nothing for EU SWAT
  • 12:21 jynus: stopping replication on all instances of dbstore2001 T169516
  • 11:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2076 - T170662 T151029 (duration: 00m 47s)
  • 11:00 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2076 - T170662 T151029 (duration: 01m 02s)
  • 10:34 marostegui: Restart MySQL on db1069 to pick up new replcation filters
  • 10:27 marostegui: Restart MySQL on db1095 to pick up new replication filters
  • 09:16 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: service=thumbor,name=thumbor2001.codfw.wmnet
  • 09:07 _joe_: stopping thumbor on thumbor2001 after depooling it for testing
  • 09:05 oblivian@puppetmaster1001: conftool action : set/pooled=no; selector: service=thumbor,name=thumbor2001.codfw.wmnet
  • 03:15 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Aug 14 03:15:31 UTC 2017 (duration 7m 6s)
  • 03:08 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.13) (duration: 06m 47s)
  • 02:39 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 29s)

2017-08-13

  • 22:45 ebernhardson: restart elsaticsearch on elastic1017 after setting md2 readahead to 256 to match md2 on 1032-152
  • 19:36 godog: upload python-thumbor-wikimedia 1.2 - T161719
  • 19:01 godog: bounce pdfrender on scb1001 and scb1003 - T159922
  • 18:46 godog: bounce carbon and uwsgi on graphite1003
  • 17:53 reedy@tin: Synchronized wmf-config/db-eqiad.php: fix comment (duration: 00m 47s)
  • 17:52 reedy@tin: Synchronized wmf-config/db-codfw.php: fix comment (duration: 00m 47s)
  • 17:51 reedy@tin: Synchronized refresh-dblist: phpcs (duration: 00m 48s)

2017-08-12

  • 20:00 krinkle@tin: Synchronized php-1.30.0-wmf.13/includes/jobqueue/JobQueueGroup.php: T171371 - Log job pushes to bogus wikis (duration: 00m 53s)
  • 16:09 Reedy: Deleted some bogus user languages from commonswiki.user_properties
  • 15:25 elukey: powercycle mw2256 (able to use com2 but not to login as root, regular ssh hanging) - T163346

2017-08-11

  • 21:47 legoktm@tin: Finished scap: (no justification provided) (duration: 07m 09s)
  • 21:41 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: mw2256.codfw.wmnet
  • 21:40 legoktm@tin: Started scap: (no justification provided)
  • 21:40 legoktm@tin: scap failed: LockFailedError Failed to acquire lock "/var/lock/scap.unknown-but-probably-mediawiki.lock"; owner is "legoktm"; reason is "Deploying Timeless (try 2) - T154371" (duration: 00m 00s)
  • 21:02 ebernhardson: unban elastic1017 from elasticsearch cluster
  • 20:52 bblack: varnish backend restart on cp1099
  • 20:46 legoktm@tin: Started scap: Deploying Timeless (try 2) - T154371
  • 20:42 legoktm@tin: scap aborted: Deploying Timeless - T154371 (duration: 05m 10s)
  • 20:36 legoktm@tin: Started scap: Deploying Timeless - T154371
  • 20:19 bblack: varnish backend restart on cp1049 + cp1074 (mailbox lag)
  • 19:59 ebernhardson: ban elastic1017 from eqiad search cluster
  • 18:28 moritzm: installing subversion security updates
  • 17:19 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2075 (duration: 00m 47s)
  • 17:05 twentyafterfour: Deploying phabricator security update
  • 16:21 jynus_: stop db1069:s6 replication and dropping frwiki, jawiki, ruwiki
  • 15:41 jynus_: stopping db2075 to clone it to dbstore2001
  • 15:39 moritzm: installing git security updates on trusty (jessie/stretch already fixed)
  • 15:38 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2075 (duration: 00m 48s)
  • 13:51 elukey: moved the eventbus scap deployment dirs on kafka[12]00[123] to deploy-service:deploy-service to allow scap to depool/pool - T171506
  • 13:49 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 27s)
  • 13:49 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
  • 13:42 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 13s)
  • 13:42 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
  • 13:25 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 34s)
  • 13:24 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
  • 13:16 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 12s)
  • 13:15 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
  • 13:00 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 17s)
  • 13:00 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
  • 12:51 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 04s)
  • 12:51 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
  • 12:44 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 05s)
  • 12:44 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
  • 12:05 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2046 - T151029 (duration: 00m 48s)
  • 11:25 jynus: stopping and upgrading labsdb1010
  • 11:19 marostegui: Stop replication on db2046 to fix duplicate entries - T151029
  • 09:28 jynus: stopping and restarting es2013 for upgrade
  • 09:07 jynus: stopping and restarting db2046 for upgrade
  • 07:35 elukey: restart pdfrender on scb1004

2017-08-10

  • 23:43 maxsem@tin: Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/371196/2 (duration: 00m 47s)
  • 23:26 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/370310/ (duration: 00m 47s)
  • 23:25 maxsem@tin: Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/370310/ (duration: 00m 46s)
  • 22:31 twentyafterfour@tin: Synchronized php-1.30.0-wmf.13/includes/specials/SpecialDoubleRedirects.php: Hopefully fix T173045 (duration: 00m 48s)
  • 22:17 moritzm: installing git security-updates
  • 20:51 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment, take #2.5 - T137371 (duration: 00m 21s)
  • 20:51 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment, take #2.5 - T137371
  • 20:50 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment, take #2 - T137371 (duration: 00m 09s)
  • 20:50 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment, take #2 - T137371
  • 20:35 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment - T137371 (duration: 00m 05s)
  • 20:34 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment - T137371
  • 20:00 mobrovac@tin: Finished deploy [cassandra/metrics-collector@5db1a43] (staging): First Scap3 deployment - T137371 (duration: 18m 17s)
  • 19:42 mobrovac@tin: Started deploy [cassandra/metrics-collector@5db1a43] (staging): First Scap3 deployment - T137371
  • 19:35 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: touch (duration: 00m 47s)
  • 19:33 reedy@tin: Synchronized wmf-config/: Newsletter on testwiki (duration: 00m 49s)
  • 19:31 mobrovac@tin: Finished deploy [cassandra/logstash-logback-encoder@d085ffa] (aqs): first Scap3 deployment - T116340 (duration: 01m 08s)
  • 19:30 mobrovac@tin: Started deploy [cassandra/logstash-logback-encoder@d085ffa] (aqs): first Scap3 deployment - T116340
  • 19:30 mobrovac@tin: Finished deploy [cassandra/logstash-logback-encoder@d085ffa]: first Scap3 deployment - T116340 (duration: 00m 34s)
  • 19:29 mobrovac@tin: Started deploy [cassandra/logstash-logback-encoder@d085ffa]: first Scap3 deployment - T116340
  • 19:28 mobrovac@tin: Finished deploy [cassandra/logstash-logback-encoder@d085ffa] (staging): first Scap3 deployment (rest of the nodes) - T116340 (duration: 00m 12s)
  • 19:28 mobrovac@tin: Started deploy [cassandra/logstash-logback-encoder@d085ffa] (staging): first Scap3 deployment (rest of the nodes) - T116340
  • 19:27 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: All wikis (except wikidata) to 1.30.0-wmf.13 refs T170631
  • 19:16 godog: restart cassandra instances on xenon to test logstash-logback-encoder deploy
  • 19:13 mobrovac@tin: Finished deploy [cassandra/logstash-logback-encoder@d085ffa] (staging): first Scap3 deployment - T116340 (duration: 00m 03s)
  • 19:13 mobrovac@tin: Started deploy [cassandra/logstash-logback-encoder@d085ffa] (staging): first Scap3 deployment - T116340
  • 18:56 Reedy: created newsletter tables on testwiki
  • 18:55 reedy@tin: Synchronized php-1.30.0-wmf.13/extensions/Newsletter/: sql file updates (duration: 00m 52s)
  • 18:53 reedy@tin: Synchronized php-1.30.0-wmf.13/extensions/WikimediaMaintenance/createExtensionTables.php: newsletter (duration: 00m 52s)
  • 18:21 demon@tin: Synchronized wmf-config/InitialiseSettings.php: Fix commons/commonswiki snafu (duration: 00m 52s)
  • 17:53 kartik@tin: Finished deploy [cxserver/deploy@f43ef96]: (no justification provided) (duration: 00m 44s)
  • 17:52 kartik@tin: Started deploy [cxserver/deploy@f43ef96]: (no justification provided)
  • 17:42 kartik@tin: Finished deploy [cxserver/deploy@1065ffe]: Update cxserver to 686f4f3 (duration: 00m 40s)
  • 17:42 kartik@tin: Started deploy [cxserver/deploy@1065ffe]: Update cxserver to 686f4f3
  • 16:23 ema: cp1072: restart varnish backend
  • 15:48 XioNoX: troubleshoting interface errors between pfw3-codfw and fasw-codfw
  • 15:09 marostegui: Stop replication on db2046 to fix duplicate entries - T151029
  • 15:05 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2046 - T151029 (duration: 00m 50s)
  • 15:03 marostegui: Poweroff es2013 for maintenance - T172265
  • 14:45 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2075 - T170662 (duration: 00m 51s)
  • 14:32 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2045 - T151029 (duration: 00m 51s)
  • 14:11 elukey: restart kafka1012 temporary with some logs to TRACE to debug T172681
  • away: updated civicrm from 200abc2 to 4aa177b
  • 13:43 marostegui: Compress cebwiki on db1095 - T153058
  • 13:32 zeljkof: EU SWAT finished
  • 13:26 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wgArticleCountMethod to any on srwikisource (T172974) (duration: 00m 53s)
  • 13:18 zeljkof: EU SWAT, part two
  • 13:17 marostegui: Drop m3 databases from dbstore1002 - T156758
  • 13:16 zeljkof: EU SWAT finished
  • 13:15 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable NewUserMessage on knwiki (T172894) (duration: 00m 52s)
  • 12:56 marostegui: Stop MySQL on db2045 to upgrade socket location - T148507
  • 12:07 elukey: restored varnishakafka on cp3032
  • 11:17 elukey: disabled puppet on cp3032 and restarted varnishkafka with debug logging
  • 09:58 jynus: continuing cloning of dbstore2002 to dbstore2001
  • 09:30 gehel: repooling wdqs2001, long after data reload completed
  • 09:30 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs2001.codfw.wmnet
  • 09:27 gehel@tin: Finished deploy [wdqs/wdqs@c186e3e]: (no justification provided) (duration: 01m 31s)
  • 09:25 gehel@tin: Started deploy [wdqs/wdqs@c186e3e]: (no justification provided)
  • 09:17 jynus: disabling lag notification on all s4 replicas
  • 08:59 elukey: update librdkafka1 to 0.9.4.1 on eventlog1001
  • 08:15 elukey: add 50G to carbon lv on graphite1003 and 100G on graphite2002
  • 07:32 jynus: enabling semisymc master replication on db1068
  • 07:29 jynus: disabling semisync slave replication on all SSD hosts on s4
  • 06:45 elukey: powercycle mw2256 - T163346
  • 06:38 elukey: restart pdfrender on scb1004
  • 06:04 Dereckson: Removed 2FA for GoldRingChip account (T172878)
  • 03:23 bblack: cp1008, restbase-dev100[456] have puppet disabled, manual "rm /etc/ssh/userkeys/dzahn"
  • 03:20 bblack: batched cumin puppet agent run on all hosts (not forced)
  • 02:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 09m 12s)

2017-08-09

  • 22:06 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: Get baaack you broken CodeMirror! (duration: 00m 51s)
  • 21:20 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: Deploy CodeMirror https://gerrit.wikimedia.org/r/#/c/370943/ (duration: 00m 50s)
  • 21:11 maxsem@tin: Synchronized php-1.30.0-wmf.12/extensions/CodeMirror: https://gerrit.wikimedia.org/r/#/c/370904/ (duration: 00m 51s)
  • 21:05 ppchelko@tin: Finished deploy [restbase/deploy@f97beeb]: Rollback on canary (duration: 07m 30s)
  • 21:01 godog: add 100G to carbon lv on graphite1003 and graphite2002
  • 20:58 ppchelko@tin: Started deploy [restbase/deploy@f97beeb]: Rollback on canary
  • 20:54 ppchelko@tin: Finished deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. All tables created, finish deploy (duration: 07m 12s)
  • 20:51 marostegui: Remove m3 replication from dbstore1002 - T156758
  • 20:47 ppchelko@tin: Started deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. All tables created, finish deploy
  • 20:43 ppchelko@tin: Finished deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. Attempt 3 (duration: 08m 05s)
  • 20:37 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2046 - T170662 (duration: 00m 49s)
  • 20:35 ppchelko@