Server admin log/Archive 28

From Wikitech
Jump to: navigation, search

2015-12-31

  • 02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Dec 31 02:31:43 UTC 2015 (duration 6m 52s)
  • 02:24 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 10m 25s)

2015-12-30

  • 15:05 andrewbogott: restarting puppet on seaborgium; openldap will restart but config should not change
  • 14:57 andrewbogott: restarting puppet on serpens; openldap will restart but config should not change
  • 13:14 mark: labstore1001: apt-get install irqbalance
  • 08:01 jynus: setting dbstore1001 to read_only, converting ruwiki.recentchanges back to InnoDB
  • 05:27 ejegg|away: set drupal variable wmf_common_requeue_max back to 10
  • 02:32 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Dec 30 02:32:15 UTC 2015 (duration 6m 55s)
  • 02:26 gwicke: restbase: finished full deploy of 7db8e216 (small bug fix & a security fix) to production cluster
  • 02:25 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 09m 50s)
  • 02:17 gwicke: restbase: starting full deploy of 7db8e216 (small bug fix & a security fix) to production cluster
  • 02:14 gwicke: restbase: canary deploy of 7db8e216 (small bug fix & a security fix) to restbase1001
  • 00:08 andrewbogott: restarting nova-compute on labvirt1002

2015-12-29

  • 23:56 andrewbogott: restarting nodepool on labnodepool1001
  • 23:00 ejegg: re-enabled fundraising banner campaigns
  • 22:50 ejegg: restarted fundraising scheduled jobs
  • 22:14 ejegg: shut down campaigns and scheduled jobs due to dead ActiveMQ
  • 22:13 andrewbogott: restarting slapd on seaborgium and serpens
  • 22:11 andrewbogott: disabling puppet on seaborgium and serpens
  • 16:47 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Turn off AB test for search lang detect via accept-language (duration: 00m 29s)
  • 16:04 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable cross-wiki upload A/B test gerrit:261371 (duration: 00m 31s)
  • 15:06 apergos: labs salt instances salt update in progress. It's slow and tedious and automated. A few hundred instances already done, the rest are going one at a time. Only instances that use the labcontrol salt master will be affected.
  • 13:29 apergos: salt wm2 packages now installed on all production hosts except for: mw1041.eqiad.wmnet, technetium.eqiad.wmnet, mw1228.eqiad.wmnet, ms-be1011.eqiad.wmnet
  • 13:08 apergos: labcontrol*, neodymium and palladium updated to latest salt packages (wm2), rest of prod to follow
  • 12:24 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: Attempt to fix math related fatal (duration: 00m 33s)
  • 09:54 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Setting weight values for s6 to original production values (duration: 00m 35s)
  • 09:37 jynus: changing the mysql master of db2028, from db1030 to db1050
  • 02:41 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Dec 29 02:41:49 UTC 2015 (duration 6m 38s)
  • 02:35 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 15m 38s)
  • 01:02 logmsgbot: awight@tin Synchronized php-1.27.0-wmf.9/extensions/CentralNotice: Update CentralNotice: T122251 (duration: 00m 34s)

2015-12-28

  • 21:45 gwicke: restbase: rolling restart to apply https://gerrit.wikimedia.org/r/261206
  • 21:26 mutante: tin & mira: started salt minions that were in status stop/waiting
  • 21:25 logmsgbot: aaron@tin Synchronized private/PrivateSettings.php: (no message) (duration: 00m 30s)
  • 21:20 logmsgbot: aaron@tin Synchronized wmf-config/PrivateSettings.php: $wmfSwiftConfig convenience variable (duration: 00m 30s)
  • 20:59 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1050 & db1022 after emergency fix (duration: 00m 31s)
  • 20:52 mutante: cygnus - starting salt-minion
  • 20:35 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.9/extensions/Graph/modules/graph2.js: https://gerrit.wikimedia.org/r/#/c/261200/ (duration: 00m 31s)
  • 19:22 ejegg: updated DjangoBannerStats from 8d4a9062aab80e5371faebadd72fbe4f19ac2fdd to a64fe0e373a978d3df0b7f1dd74ac4cc5c78d34e
  • 18:46 jynus: importing wikishared from x1-master into analytics-slave and setting up replication
  • 18:28 jynus: restarting and upgrading db1050, using the fact that it is depooled
  • 17:16 paravoid: disabled varnish TBF and force-ran puppet on all cp* hosts (I12ea52165e125aaf4ed779399f34cff16d5cd140)
  • 16:38 jynus: applying production-side replication filters for wikimania2017wiki on labs
  • 16:24 logmsgbot: krenair@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 30s)
  • 16:14 logmsgbot: krenair@tin Synchronized dblists: (no message) (duration: 00m 29s)
  • 16:13 logmsgbot: krenair@tin rebuilt wikiversions.php and synchronized wikiversions files: (no message)
  • 16:12 logmsgbot: krenair@tin Synchronized w/static/images/project-logos/wikimania2017wiki.png: (no message) (duration: 00m 31s)
  • 16:12 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/260521/ (duration: 00m 30s)
  • 14:55 jynus: cloning db1050's mysql data to db1022
  • 14:15 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Emergency depool of db1050 (duration: 00m 31s)
  • 02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Dec 28 02:30:47 UTC 2015 (duration 6m 58s)
  • 02:23 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 09m 55s)

2015-12-27

  • 03:05 YuviPanda: run nodetool clearsnapshot -- v3 and nodetool clearsnapshot -- v1 on maps-test2001
  • 02:45 YuviPanda: run drop keyspace v3; on csql on maps-test1001 for yurik
  • 02:42 YuviPanda: run drop keyspace v1; on csql on maps-test1001 for yurik
  • 02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Dec 27 02:30:29 UTC 2015 (duration 6m 59s)
  • 02:23 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 09m 35s)

2015-12-26

  • 19:12 paravoid: restarting varnish-frontend on cp3042
  • 19:06 jynus: setting db1030 as the new master of db2028
  • 18:40 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Emergency depool of db1022 (duration: 00m 30s)
  • 18:28 jynus: disabling lag notifications for codfw (s6)
  • 18:01 paravoid: cp3048: cleaned up /run/vmod_tbf/tbf.db/, kept a backup copy under ~faidon
  • 17:55 paravoid: cp3048: service varnish-frontend stop (sending 429 to lots of people, T122453)
  • 05:38 paravoid: rolling restart of hhvm jobrunners (T122069)
  • 02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Dec 26 02:31:39 UTC 2015 (duration 6m 54s)
  • 02:24 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 10m 39s)

2015-12-25

  • 15:50 jynus: testing new mariadb packages on db2070
  • 13:19 jynus: setting db2018's binlog_format as MIXED
  • 10:51 jynus: powercycle cp3010
  • 09:17 jynus: powercycling cp4007 (unresponsive to ssh, ping, serial console)
  • 02:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Dec 25 02:29:49 UTC 2015 (duration 6m 56s)
  • 02:22 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 09m 47s)

2015-12-24

  • 23:28 mutante: powercycled mw1114
  • 23:27 mutante: i just reset dra on mw1114 because it said it was in use and i didnt see a log yet :;p
  • 23:26 robh: mw1114 spammed all icinga errors, system is outputting endless scroll of login prompt, not halting for input (like anohter session or crash cart is sending it, or an error)
  • 17:31 gwicke: aqs: tweaked table properties for local_group_default_T_pageviews_per_article_flat: 2 months max DTCS window size, deflate compression
  • 17:20 jynus: restarting and reconfiguring mysql at db2066
  • 16:45 jynus: restart and reconfigure mysql at db2059
  • 16:11 jynus: restart and mysql reconfguration of db2052
  • 15:02 jynus: restarting and reconfiguring mysql at db2045
  • 14:11 paravoid: powercycling mw1012, OOM'ed/stuck
  • 14:09 paravoid: rolling restart of hhvm jobrunners (T122069)
  • 14:06 jynus: restart and reconfigure mysql at db2038
  • 12:32 jynus: restart and reconfigure mysql at db2065
  • 12:12 jynus: restart and reconfiguring mysql for db2058
  • 11:50 jynus: restarting and reconfiguring mysql at db2051
  • 11:28 jynus: restarting and reconfiguring mysql at db2044
  • 10:33 jynus: restarting 's2' replication on dbstore200[12] after cloning
  • 02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Dec 24 02:30:40 UTC 2015 (duration 6m 52s)
  • 02:23 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 10m 03s)
  • 01:05 awight: update payments from bae4d02afd8cfe1f8b8617c2f74bb36e420d281d to a7785baa7b40b442ecf0b60d47572502d0759780
  • 00:38 gwicke: restbase1003: starting `nodetool cleanup`

2015-12-23

  • 23:31 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.9/extensions/Graph/modules/ve-graph: https://gerrit.wikimedia.org/r/#/c/260868/ (duration: 00m 31s)
  • 19:59 mutante: restbase1004 - puppet stopped and host key changed, what's up?
  • 19:42 mutante: ran puppet on mw2112
  • 19:41 mutante: logstash1002 - started logstash service
  • 19:22 logmsgbot: aaron@tin Synchronized wmf-config/CommonSettings.php: Remove unused $wgMaxSquidPurgeTitles setting (duration: 00m 30s)
  • 18:55 ejegg: updated fundraising tools from ebed29c0eccf38c812b20a957b3487a15bfa9cbc to 1bc23cb4bfaf2a9d4d215aad79dd67d891b5d973
  • 18:51 ejegg: updated fundraising dashboard from 59e51c4ff74c3c584daf6c5de3bb66daa764cd28 to af8a493ab9ac5431e0d294e5019ac4e426ac6e08
  • 18:38 jynus: restart and reconfigure mysql at db2037
  • 18:17 mutante: bohrium - finish install, signing puppet certs
  • 17:11 jynus: cloning s2 databases from dbstore2001 to dbstore2002 (s2 replication disabled on both)
  • 14:05 jynus: restart and reconfigure mysql at db2064
  • 13:26 jynus: restart and reconfigure mysql at db2063
  • 12:41 jynus: reenabling event scheduler on db1046 (eventlogging m4-master)
  • 12:30 jynus: restart and reconfigure mysql at db2056
  • 12:14 godog: upgrade cassandra on aqs1003
  • 11:24 jynus: reloading and reconfiguring mysql on db2049
  • 11:11 godog: roll-upgrade cassandra to 2.1.12 on aqs100[123]
  • 10:55 jynus: rebooting and reconfiguring mysql on db2041
  • 10:11 jynus: restarting and reconfiguring mysql at db2035
  • 09:52 gwicke: rebuilding restbase1004
  • 09:51 gwicke: wiped & started boostrap on restbase1008
  • 09:18 gwicke: nodetool removenode e2813bb9-f1f2-4d21-ac19-95a7a35b4513 in preparation for adding 1004 to the cluster without bootstrap
  • 02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Dec 23 02:30:25 UTC 2015 (duration 7m 1s)
  • 02:23 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 09m 18s)
  • 00:40 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings-labs.php: https://gerrit.wikimedia.org/r/260696 & https://gerrit.wikimedia.org/r/260699 (duration: 05m 28s)
  • 00:37 mutante: mw1133 - powercycle
  • 00:36 legoktm: manually fixed up stuck global rename of "RCJU-ArCJ" -> "Archives cantonales jurassiennes"
  • 00:31 matt_flaschen: Ran UPDATE flow_workflow SET workflow_page_id = 41854369 WHERE workflow_wiki = 'enwiki' AND workflow_namespace = 5 AND workflow_title_text = 'Flow/Developer_test_page' AND workflow_page_id = 48099373; to work around DB inconsistency (T117812)

2015-12-22

  • 21:46 gwicke: restbase1004: tune2fs -m 0 /dev/mapper/restbase1004--vg-srv
  • 21:45 gwicke: restbase1004: restarted bootstrap
  • 21:22 gwicke: restbase1003: restarting cassandra to clear up disk space from old stream
  • 21:11 gwicke: restbase1008: restarting cassandra to clear up disk space from old stream
  • 18:36 robh: silver returned to normal service, wikitech.w.o certificate renewed.
  • 18:26 robh: silver puppet staying stalled during toollabs issue (we dont want to rehup silver web serivce)
  • 18:17 robh: puppet disabled on silver, going to update wikitech.wikimedia.org certificate
  • 18:10 jynus: disabling event scheduling on db1046
  • 18:03 jynus: rolling schema change (ALTER TABLE ENGINE=TokuDB) on m4-master (db1046) log (eventlogging)
  • 16:44 godog: bounce cassandra on restbase1004, restart bootstrap
  • 16:42 mutante: powercycling crashed mw1144
  • 16:41 jynus: converting dbstore2001 (delayed slave) into an actual delayed slave, adding redundancy to dbstore1002
  • 16:40 godog: bounce cassandra on restbase1003
  • 16:15 akosiaris: upgrade cassandra on maps-test2001
  • 16:15 akosiaris: upgrade cassandra on maps-test2002
  • 15:53 mutante: kafka1001,1002 - crit - eventlogging not running (?)
  • 15:52 mutante: restbase1003 - disk space, restbase1008 - disk space, restbase1004 - cassandra cql refused
  • 15:23 akosiaris: upgrade cassandra on maps-test2003
  • 15:06 jynus: restarting and reconfiguring mysql at dbstore2001
  • 15:06 mutante: labtestcontrol2001 - puppet had not been running for a while, a bunch of changes have been applied incl. keys and passwords
  • 15:04 mutante: enabling puppet on labtestcontrol2001
  • 15:04 akosiaris: upgraded cassandra on maps-test2004
  • 11:54 apergos: salt packages with wmf packages precise running on ms-{bf}e* in esams; trusty running on analytics103* in eqiad; jessie running on restbase2* in codfw
  • 11:43 godog: restart cassandra bootstrap on restbase1004
  • 10:09 jynus: online resizing /srv/postgres on labsdb1006 +100GB
  • 10:06 hashar: Restarting Jenkins
  • 09:54 apergos: precise and trusty salt packages with wmf patches deployed manually on dataset1001 and analytics1001, seem to work fine
  • 08:42 jynus: restarting and reconfiguring mysql at db2036
  • 02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Dec 22 02:30:28 UTC 2015 (duration 6m 54s)
  • 02:23 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 09m 47s)
  • 00:29 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.9/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/260492/ (duration: 00m 32s)
  • 00:22 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.9/extensions/SyntaxHighlight_GeSHi/modules/ve-syntaxhighlight/ve.ui.MWSyntaxHighlightDialogTool.js: https://gerrit.wikimedia.org/r/#/c/260429/ (duration: 00m 30s)

2015-12-21

  • 22:41 cwd: updated paymentswiki from a1be1ad134d06464e98de180227554fceddc91d4 to bae4d02afd8cfe1f8b8617c2f74bb36e420d281d
  • 20:49 godog: restbase1004 bootstrap failed, restbase1007-a is down java.lang.RuntimeException: A node required to move the data consistently is down (/10.64.0.230).
  • 19:27 legoktm: running checkLocalUser.php --delete=1 for real this time on terbium
  • 19:22 godog: reimage restbase1004
  • 19:14 paravoid: powercycling mw1011
  • 19:11 paravoid: rolling restart of hhvm on the eqiad jobrunners
  • 18:47 jynus: common-sync: Copying to mw1016.eqiad.wmnet from tin.eqiad.wmnet
  • 18:35 ori: correction: previous log message was for mw1015, not mw1017
  • 18:27 ori: mw1017: enabled jemalloc profiling, restarted hhvm, now running hhvm-collect-heaps
  • 17:48 akosiaris: restarted hhvm on mw1012.eqiad.wmnet
  • 16:57 thcipriani: timeout on sync-file to mw1016.eqiad.wmnet
  • 16:56 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.9/extensions/Popups/Popups.hooks.php: SWAT: Use ExtensionRegistry to determine whether TextExtracts is installed gerrit:260346 (duration: 02m 48s)
  • 16:34 jynus: sync-common to mw1085
  • 16:26 jynus: powercycling mw1085.eqiad.wmnet
  • 16:22 thcipriani: mw1085.eqiad.wmnet times out on SSH connection
  • 16:19 godog: reboot restbase1007, load through the roof
  • 16:18 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.9/extensions/CentralNotice/resources/subscribing/ext.centralNotice.geoIP.js: SWAT: Update CentralNotice gerrit:260316 (duration: 03m 03s)
  • 16:08 godog: depool restbase1007
  • 16:01 apergos: jessie packages for salt with local patches deployed on restbase1001, looks fine but just in case.
  • 15:44 godog: adding new 1TB disk to restbase1007
  • 14:22 andrewbogott: disabling puppet on labnet1002 for dnsmasq tests
  • 14:07 MaxSem: me and yurik are nuking old maps data and reimporting planet
  • 13:46 jynus: extending online s2-master data disk by +100GB
  • 13:15 akosiaris: disabled puppet on maps-test2001 and commented out osmupdater crontab entry until we fix the sync process
  • 11:02 jynus: emergency restart of db1047's mysql
  • 09:54 jynus: reenabling semisync replication on s3
  • 09:07 godog: stop cassandra on restbase1004, decomissioned
  • 02:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Dec 21 02:29:51 UTC 2015 (duration 6m 47s)
  • 02:23 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 09m 45s)
  • 02:20 andrewbogott: disabling puppet on labnet1002 to mess with dnsmasq
  • 01:44 andrewbogott: disabled puppet on holmium and labservices1001 to control roll-out of https://gerrit.wikimedia.org/r/#/c/260037/

2015-12-20

  • 23:24 Reedy: Katie and Jeff paged about bellatrix
  • 18:46 andrewbogott: graceful restart of zuul as per https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Restart
  • 18:31 andrewbogott: restarting stuck Jenkins
  • 17:47 logmsgbot: reedy@tin Purged l10n cache for 1.27.0-wmf.6
  • 17:11 godog: depool mw1228, reported ro fs
  • 15:53 logmsgbot: reedy@tin Synchronized README: noop (duration: 00m 32s)
  • 15:50 Reedy: reedy@tin Purged l10n cache for 1.27.0-wmf.6 (hanging due to mw1228 issue)
  • 15:42 Reedy: mw1228 reporting readonly fs
  • 15:41 logmsgbot: reedy@tin Purged l10n cache for 1.27.0-wmf.7
  • 09:00 godog: powercycle ms-be2019, xfs lockup
  • 02:28 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Dec 20 02:28:49 UTC 2015 (duration 6m 54s)
  • 02:21 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 08m 59s)

2015-12-19

  • 21:55 _joe_: restarted zotero on sca1001, various OOM messages
  • 20:48 gwicke: restbase1004: `systemctl mask cassandra` in preparation for the decommission finishing
  • 19:49 akosiaris: killed gmond on db2036. it was clearly misbehaving and running since Jan 02. db2036 was not listed on the ganglia web interface. killing the orphaned process and restarting seems to have fixed it
  • 18:54 akosiaris: scheduled maintenance of s3 slave lag on db2036, db2043, db2050, db2057 (all of db2018's family that pages) to effectively silence pages while debugging. Check is flapping since 15:00 UTC today
  • 15:14 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/259611/ - noop for prod, other than making icinga stop complaining (duration: 00m 31s)
  • 10:07 hashar: CI jobs for MediaWiki were broken because of cssjanus dependency. Should be fixed once mw/core https://gerrit.wikimedia.org/r/#/c/260169/ lands
  • 02:28 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Dec 19 02:28:56 UTC 2015 (duration 6m 53s)
  • 02:22 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 08m 53s)
  • 01:01 gwicke: entire restbase cluster: removed 5% root reserve from data partition with tune2fs -m 0 /dev/mapper/restbase$NODE--vg-{srv,var}
  • 00:49 gwicke: restbase1008: removed 5% root reserve from data partition with tune2fs -m 0 /dev/mapper/restbase1008--vg-srv

2015-12-18

  • 22:57 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.9/resources/src/mediawiki/mediawiki.searchSuggest.js: allow override of suggestion type reported in event loggin (duration: 00m 29s)
  • 22:56 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.9/extensions/CirrusSearch/resources/ext.cirrus.suggest.js: override suggestion type reported in event logging (duration: 00m 30s)
  • 22:50 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.9/includes/jobqueue/aggregator/JobQueueAggregatorRedis.php: 2c942ba1782c42ee68622278a5e0a77e9027945d (duration: 00m 30s)
  • 22:30 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.9/extensions/CirrusSearch/resources/ext.cirrus.suggest.js: override suggestion type reported in event logging (duration: 00m 30s)
  • 22:20 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.9/includes/jobqueue/aggregator/JobQueueAggregator.php: 2c942ba1782c42ee68622278a5e0a77e9027945d (duration: 00m 31s)
  • 19:26 logmsgbot: aaron@tin Synchronized wmf-config/jobqueue-eqiad.php: Adjust queue "maxPartitionsTry" and timeouts (duration: 00m 30s)
  • 18:49 mutante: disregard that, apache config only is enough
  • 18:47 mutante: gerrit will restart in a moment and be right back
  • 18:44 ori: ditto
  • 18:43 Krinkle: Created account "Krinkle" on collabwiki
  • 16:28 twentyafterfour: restarted apache on iridium to deploy redirect script changes
  • 16:20 jynus: restarting and reconfiguring mysql on db1047
  • 14:57 godog: stop compactions on restbase1008
  • 14:55 jynus: SET GLOBAL query_cache_type = 0; on db1025
  • 14:54 hashar: gallium: restarted apache2 , was deadlocked/unresponsive somehow
  • 14:44 godog: update privatesettings with swift codfw configuration
  • 14:43 godog: set temp-url-key for mw:media account in swift codfw
  • 12:19 paravoid: upgrading tor on radium, rebooting for kernel upgrade
  • 12:18 _joe_: disabled puppet on all lvs hosts for a potentially harmful change (should be a noop)
  • 11:47 _joe_: restarted hhvm on mw1107, stuck at startup
  • 11:40 hashar: logstash: reorganized list of dashboards per sections https://logstash.wikimedia.org/#/dashboard/elasticsearch/default
  • 09:43 akosiaris: rebooting planet1001, memory exhaustion, OOM showed up
  • 09:20 hashar: Killed Zuul entirely, the queues were full / deadlocked. Patches need to be retriggered
  • 06:47 gwicke: restbase1004: nodetool stop -- COMPACTION to avoid running out of disk space
  • 03:07 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.9/includes/api/ApiStashEdit.php: ab32f4e740: Make ApiStashEdit use statsd metrics (duration: 00m 49s)
  • 02:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Dec 18 02:29:10 UTC 2015 (duration 6m 55s)
  • 02:22 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 08m 45s)
  • 01:52 ori: re-enabled puppet on rdb* / mc*
  • 01:25 ori: in preparation for Iaefb2d191e, disabling puppet on mc* and rdb*
  • 01:21 logmsgbot: krinkle@tin Synchronized docroot and w: (no message) (duration: 00m 32s)
  • 00:53 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.9/extensions/Flow: Revert Nuke-Flow integration, doesn't work (duration: 00m 32s)
  • 00:42 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.9/extensions/Flow: SWAT: Nuke support for Flow, part 3 (duration: 00m 32s)
  • 00:34 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Add completion suggester to BetaFeatures whitelist (duration: 00m 30s)
  • 00:26 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: grumble grumble touch InitialiseSettings grumble (duration: 00m 30s)
  • 00:25 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.9/extensions/Flow: SWAT: Nuke support for Flow, part 2 (duration: 00m 32s)
  • 00:23 logmsgbot: catrope@tin Synchronized wmf-config/CirrusSearch-production.php: SWAT: enable completion suggester beta on all wikis except wikidata (duration: 00m 30s)
  • 00:23 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: enable completion suggester beta on all wikis except wikidata (duration: 00m 29s)
  • 00:20 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.9/extensions/Nuke/: SWAT: Nuke support in Flow, part 1 (duration: 00m 30s)
  • 00:18 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.9/resources/src/mediawiki.messagePoster/mediawiki.messagePoster.factory.js: SWAT: fix error in messagePoster (duration: 00m 29s)
  • 00:17 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.9/extensions/MobileFrontend: SWAT: Schema:MobileWebSectionUsage: always log the isTestA field (duration: 00m 31s)
  • 00:08 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: SWAT: cleanup (duration: 00m 30s)
  • 00:00 ori: restarted mathoid on sca1001

2015-12-17

  • 23:42 mobrovac: mathoid deploying 8d2295
  • 22:47 hashar: ssh to tin is back https://gerrit.wikimedia.org/r/#/c/259876/
  • 22:39 hashar: Only tin lost SSH user keys apparently due to https://gerrit.wikimedia.org/r/#/c/253465/ overriding the admin::groups to simply "eventlogging-admins"
  • 22:34 hashar: Ssh User keys are gone on deployment servers ( tin / mira )
  • 22:18 eileen1: update CiviCRM from b307d744def9289a7f86cb02bc6e1a00225e474d to cb5e20c29d7376920c45eb5c343e6ee464217833
  • 22:10 eileen1: Updating civicrm from b307d744def9289a7f86cb02bc6e1a00225e474d to cb5e20c29d7376920c45eb5c343e6ee464217833
  • 21:33 hashar: gallium: upgrading Zuul from 2.1.0-60-g1cc37f7-wmf2precise1 .. 2.1.0-60-g1cc37f7-wmf4precise1 . Should be noop, only change zuul-cloner which is not used there
  • 21:29 mutante: add zuul_2.1.0-60-g1cc37f7-wmf4jessie1 to jessie-wikimedia repo
  • 21:28 mutante: add zuul_2.1.0-60-g1cc37f7-wmf4trusty1 to trusty-wikimedia repo
  • 21:22 mutante: add zuul_2.1.0-60-g1cc37f7-wmf4precise1 to precise-wikimedia APT
  • 19:42 mobrovac: mathoid deploying a2187a6
  • 19:07 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.27.0-wmf.9
  • 19:05 godog: disable puppet on graphite2001, brief testing cluster aggregations
  • 19:01 thcipriani: starting update of all wikis to 1.27.0-wmf.9
  • 18:11 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool x1-slave (db1031), increase db1041 load to 100% (duration: 00m 30s)
  • 18:06 gwicke: running `nodetool cleanup` on restbase1001 and restbase1005
  • 18:04 robh: calcium is supposed to be down, reclaiming to spares, ignore any irc alerts (its in maint mode in icinga)
  • 17:55 gwicke: running `nodetool cleanup` on restbase1002
  • 17:18 jynus: setting mysql db1031 as db2009's master
  • 17:05 jynus: restarting and reconfiguring mysql at db2009
  • 16:49 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.8/extensions/Math: SWAT: Make math usable without RESTbase gerrit:259734 (duration: 00m 30s)
  • 16:41 jynus: problems with corruption on x1-slave for cebwiki. Fixed them. Will leave db1031 depooled for a while to check they are gone.
  • 16:39 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.9/extensions/ContentTranslation/includes/Translation.php: SWAT: Fix Undefined index: targetRevisionId in ContentTranslation gerrit:259649 (duration: 00m 29s)
  • 16:23 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.9/extensions/WikimediaEvents/WikimediaEventsHooks.php: SWAT: Actually define tags for cross-wiki upload A/B test gerrit:259729 (duration: 00m 31s)
  • 16:17 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable cross-wiki upload A/B test in additional languages gerrit:259665 (duration: 00m 30s)
  • 16:11 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.9/extensions/CirrusSearch/includes/CirrusSearch.php: SWAT: Fix array-to-string conversion gerrit:259633 (duration: 00m 30s)
  • 15:57 moritzm: stopping opendj LDAP servers on nembus/neptunium (read-only since about days now due to migration to openldap)
  • 15:56 akosiaris: depool sca1001, playing with cxserver config
  • 15:24 jynus: reinstall, reboot and reconfigure mysql at db1031
  • 15:03 moritzm: installing git security updates
  • 14:23 _joe_: restarting HHVM on the first jobrunners
  • 14:11 jynus: soft-rebooting mw1004, responsive to ping, but not to salt, ssh
  • 14:06 godog: nodetool stop -- CLEANUP on restbase1002
  • 13:59 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1031 (x1-slave) for maintenance (duration: 07m 30s)
  • 13:11 moritzm: starting to restart hhvm on application servers (to effect security updates for libxml2, openssl and others)
  • 13:01 akosiaris: moved old cruft /srv/deployment/cxserver/deploy/src/config.js out of the way
  • 12:51 akosiaris: repooled sca1002 for cxserver
  • 12:51 akosiaris: restarted cxserver on sca1002
  • 12:33 akosiaris: depooling sca1002 for cxserver
  • 12:33 akosiaris: repooling sca1001 for cxserver
  • 12:24 kart_: Updated cxserver on sca1002
  • 12:08 akosiaris: disable puppet, stop salt-minion on sca1002
  • 12:08 akosiaris: depool sca1001 from cxserver service.
  • 11:56 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1041 with low weight (duration: 00m 37s)
  • 11:33 jynus: performing schema change on officewiki
  • 11:31 jynus: performing schema change on x1-master flowdb
  • 11:20 jynus: performing schema change on x1-master wikishared.cx_translations
  • 10:47 jynus: performing schema change on s7-master metawiki.oauth_registered_consumer
  • 10:30 godog: nodetool stop -- CLEANUP restbase1004
  • 02:53 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Dec 17 02:53:11 UTC 2015 (duration 7m 12s)
  • 02:46 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 08m 22s)
  • 02:27 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.8) (duration: 10m 32s)
  • 00:28 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.8/includes/api/ApiStashEdit.php: I552cf6b0420: Upgrade some ApiStashEdit logging calls to info() (duration: 00m 30s)
  • 00:05 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Password policy for sysadmin group (duration: 00m 29s)

2015-12-16

  • 23:37 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.8/includes/api/ApiStashEdit.php: local hack some extra debug logging into ApiStashEdit (take 2) (duration: 00m 30s)
  • 23:32 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.8/includes/api/ApiStashEdit.php: local hack some extra debug logging into ApiStashEdit (duration: 00m 30s)
  • 23:27 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.9/extensions/Graph/: Deployed Graph ext to master - protocol issue (duration: 00m 32s)
  • 23:19 ejegg: updated DjangoBannerStats from 47503fca7b0ebf2d56ca79c5ec92a4accf88ee77 to 8d4a9062aab80e5371faebadd72fbe4f19ac2fdd
  • 23:10 legoktm: running foreachwiki extensions/CentralAuth/maintenance/checkLocalUser.php --verbose=1 --delete=1 (T119736)
  • 23:07 robh: fermium returned to normal service with new/renewed ssl cert for lists.w.o
  • 22:58 robh: disabled puppet on fermium for the lists.w.o cert update
  • 22:58 robh: librenms returned to normal service (puppet renabled on netmon1001)
  • 22:48 robh: puppet disabled on netmon1001 for librenms certificate update
  • 22:17 ejegg: updated DjangoBannerStats from 26b48add5c6f70b395a3195c401a12612c626796 to 47503fca7b0ebf2d56ca79c5ec92a4accf88ee77
  • 21:31 subbu: finished deploying parsoid 64029e12
  • 21:29 bearND: mobileapps deployed sha1 9f91ad5
  • 21:23 subbu: restarted parsoid on wtp1005 as a canary
  • 21:19 subbu: starting parsoid deploy
  • 21:14 bearND: starting mobileapps deploy
  • 20:18 ejegg: updated SmashPig from b3ee85aab38fdeef3f045ba1733f582e9a5006b2 to 072c7ec6ed94e7074ba35b7986d5dde94866fe2f
  • 19:46 ostriches: iridium: repacking MW / OPUP repositories as phd user. This needs to be a cron.
  • 19:07 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.9
  • 19:00 thcipriani: updating group1 wikis to 1.27.0-wmf.9
  • 18:39 jynus: performing schema change on ruwiki.geo_tags on db2046
  • 17:46 jynus: setting mysql db1041 as db2029's master
  • 17:21 jynus: restarting and configuring mysql on db2029 (there will be an increase of errors- from pings, not real traffic)
  • 17:08 jynus: setting mysql db1022 as db2028's master
  • 16:50 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: graph namespace removal/cleanup (duration: 00m 31s)
  • 16:41 gwicke: cleared snapshots on cassandra cluster
  • 16:32 gwicke: restarted `nodetool decommission` on restbase1004
  • 16:31 jynus: restarting and reconfiguring mysql on db1041
  • 15:54 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1022, depool db1041 (duration: 00m 30s)
  • 15:53 godog: cassandra on restbase1004 couldn't finish decomissioning, out of disk space, running nodetool clean
  • 15:28 jynus: restarting and reconfiguring mysql on db2028
  • 15:17 yurik: updated graphoid service
  • 15:16 moritzm: removed ecryptfs-utils across the cluster
  • 15:09 godog: roll-restart swift daemons on ms-be1*
  • 15:04 jynus: setting db2023 master to be now db1049 instead of m5-master
  • 14:56 mobrovac: restbase roll-restarting restbase
  • 14:52 godog: reenable and run puppet on restbase* and aqs*
  • 14:49 mobrovac: restbase restart rb1001
  • 14:46 urandom: starting `nodetool cleanup' on restbase1009-a.eqiad (https://phabricator.wikimedia.org/T121535)
  • 14:41 urandom: starting `nodetool cleanup' on restbase1006.eqiad (https://phabricator.wikimedia.org/T121535)
  • 14:17 mobrovac: restbase reenabling and running puppet on canary rb1001
  • 14:08 godog: reenable puppet on restbase test cluster
  • 14:06 akosiaris: delete empty panel in https://grafana-admin.wikimedia.org/dashboard/db/graphoid
  • 14:04 akosiaris: repool scb1001 for mobileapps
  • 13:59 akosiaris: depool scb1001 for mobileapps (oid)
  • 13:58 akosiaris: repool sca1002 for citoid, mathoid, graphoid, cxserver
  • 13:55 godog: enable puppet on cerium and bounce restbase
  • 13:55 moritzm: restarting HHVM on canary appservers to effect various security updates (libxml, openssl, gs, freetype)
  • 13:50 godog: enable puppet on restbase-test2001 and bounce restbase
  • 13:47 godog: (retroactive) enable puppet on sca1001
  • 13:46 akosiaris: depooling sca1002, scb1002 for citoid, cxserver, graphoid, mathoid
  • 13:35 godog: disable puppet on deployment_target:restbase/deploy and sca/scb
  • 12:29 jynus: restarting and reconfiguring mysql at db1022
  • 12:26 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1022 for maintenance (duration: 00m 37s)
  • 11:49 jynus: restarting and reconfiguring mysql on db2023
  • 11:34 akosiaris: merged YuviPanda: labstore: Run sync-exports in start-nfs too (d5273d9) and YuviPanda: labstore: Activate volumes before mounting them (dff32de) on palladium
  • 08:54 YuviPanda: start nfs-kernel-server on labstore1001
  • 08:53 YuviPanda: stopped nfs-kernel-server on labstore1001
  • 08:36 mark: Rebooting labstore1001
  • 03:02 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 16m 00s)
  • 02:54 papaul: kafka200[1-2] signing puppet certs, salt key initial run
  • 02:42 papaul: installation complete on kafka200[1-2]
  • 02:30 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.8) (duration: 13m 37s)
  • 02:18 papaul: installing OS on kafka200[1-2]
  • 00:40 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Enable wmgGraphEnableGzip on all wikis (duration: 00m 30s)
  • 00:36 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Enable wmgGraphEnableGzip on mediawikiwiki (duration: 00m 30s)
  • 00:23 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.9/extensions/Graph/: SWAT: update Vega (duration: 00m 30s)
  • 00:12 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Password policy for staff, take 3; fix $wgExtractsRemoveClasses (duration: 00m 31s)

2015-12-15

  • 23:43 eileen: Updating CiviCRM from fa7124e1d8d92b576c7650030fe64d024c822088 to b307d744def9289a7f86cb02bc6e1a00225e474d
  • 23:31 eileen: Updating CiviCRM from fa7124e1d8d92b576c7650030fe64d024c822088 to 49e949c303466db9f96cb38b718190c887f7ed89
  • 23:26 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enabling Wikidata data access for meta-wiki (duration: 00m 30s)
  • 23:25 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings-labs.php: Enabling Wikidata data access for meta-wiki (duration: 00m 29s)
  • 23:24 logmsgbot: aude@tin Synchronized dblists/arbitraryaccess.dblist: Enabling Wikidata data access for meta-wiki (duration: 00m 30s)
  • 23:13 Krinkle: Synchronized php-1.27.0-wmf.9/extensions/ContentTranslation/extension.json 'T121308 - unbreak VE'
  • 22:14 gwicke: restbase: finished full deploy of 3b7ae07e to restbase prod cluster
  • 22:13 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.27.0-wmf.9 mira test
  • 22:07 gwicke: restbase: starting full deploy of 3b7ae07e to restbase prod cluster
  • 22:05 gwicke: restbase: canary deploy of 3b7ae07e to restbase1001
  • 21:53 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.27.0-wmf.9
  • 21:39 logmsgbot: thcipriani@tin Finished scap: testwiki to php-1.27.0-wmf.9 and rebuild l10ncache (duration: 29m 58s)
  • 21:09 logmsgbot: thcipriani@tin Started scap: testwiki to php-1.27.0-wmf.9 and rebuild l10ncache
  • 20:38 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.8/includes/api/ApiStashEdit.php: ad2e2aeedc: Make edit stashing use named DB locks (duration: 01m 46s)
  • 20:21 thcipriani: 1.27.0-wmf.9 branching complete 1h 18m 2s, checking out to tin
  • 20:00 subbu: restarted parsoid on all nodes to kill stuck processes (thanks to T104523)
  • 19:57 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1049 and db1042 (duration: 00m 29s)
  • 19:57 subbu: restarted parsoid on wtp1019 to kill stuck processes
  • 19:42 mutante: mw1146: hhvm restart
  • 19:39 mutante: ms-fe1001 thru ms-fe1003: swift-proxy-server stop/start
  • 19:37 mutante: ms-fe1004, swift-proxy-server stop/start
  • 19:23 mutante: powercycling mw1129
  • 19:08 ottomata: merged change to allow longer programnames in remote rsyslog config.
  • 19:00 thcipriani: starting branch cut for 1.27.0-wmf.9
  • 18:57 bblack: repooling cp1053 (eqiad text cache)
  • 18:37 ori: Depooled and drained mw1161-1169 app servers, now re-purposing as job runners, per T121549
  • 18:19 jynus: changing db2019 master to be db1042 instead of m4-master
  • 17:55 bblack: manually enabled ipsec rules in iptables on kafka10xx - puppet disabled for now until I can fix the puppetization of it...
  • 17:48 urandom: beginning `nodetool cleanup' on restbase1003.eqiad (https://phabricator.wikimedia.org/T121535)
  • 17:48 jynus: restart and reconfigure mysql at db1049
  • 17:39 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1027, Depool db1049 (duration: 00m 30s)
  • 17:33 urandom: beginning `nodetool cleanup' on restbase1005.eqiad (https://phabricator.wikimedia.org/T121535)
  • 17:30 gwicke: aborted large compaction on restbase1004 with `nodetool stop -- COMPACTION` to free disk space
  • 17:29 urandom: beginning `nodetool cleanup' on restbase1002.eqiad (https://phabricator.wikimedia.org/T121535)
  • 17:04 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Update cirrus avro schema to 111448028943 (duration: 00m 29s)
  • 17:00 jynus: restarting and reconfiguring mysql at db2018
  • 16:34 logmsgbot: thcipriani@tin Synchronized portals: SWAT: Bump portals to master (duration: 00m 29s)
  • 16:30 jynus: restarting and reconfiguring mysql on db1042
  • 16:26 godog: decommission restbase1004
  • 16:23 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: VisualEditor: Centralise feedback from test2wiki to MediaWiki.org gerrit:259041 (duration: 00m 30s)
  • 16:21 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: VisualEditor: Enable single edit tab mode on test2wiki gerrit:259039 (duration: 00m 29s)
  • 16:17 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: In VisualEditor on single edit tab wikis, set the default editor appropriately gerrit:258403 (duration: 00m 28s)
  • 16:14 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: BetaFeatures: Update language and dates of "retirement" gerrit:258409 (duration: 00m 29s)
  • 16:14 jynus: switch db2018's master from s3-master to db1027
  • 16:09 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable cross-wiki upload A/B test on English-language wikis gerrit:259258 (duration: 00m 29s)
  • 16:08 logmsgbot: thcipriani@tin Synchronized wmf-config/db-eqiad.php: SWATish: Depool db1042 for maintenance (duration: 00m 29s)
  • 15:50 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1042 for maintenance (duration: 00m 29s)
  • 15:18 jynus: restarting db2018 for upgrade and configuration change
  • 14:41 jynus: restarting mysql on db1027 to apply new configuration
  • 13:53 bblack: disabling puppet on neon to avoid race-condition ipsec alert spam
  • 13:45 hashar: reverted composer upgrade on CI with https://gerrit.wikimedia.org/r/#/c/259241/
  • 13:41 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1018 after maintenance; depool db1027 for maintenance (duration: 00m 29s)
  • 13:37 hashar: bumping composer on CI to 1.0.0-alpha11 https://gerrit.wikimedia.org/r/#/c/258933/
  • 12:48 mobrovac: restbase deploy end of 844a41d
  • 12:39 mobrovac: restbase deploy start of 844a41d
  • 11:46 jynus: restarting and upgrading dbstore2002
  • 11:33 paravoid: force-rebooting stat1002, kernel borked because of fuse
  • 11:25 paravoid: rebooting lvs4003/lvs4004 for kernel upgrade
  • 11:21 paravoid: rebooting lvs3004 for kernel upgrade
  • 11:01 jynus: reloading haproxy on dbproxy1002
  • 10:20 jynus: stopped eventlogging on dbstore1002 and db1047
  • 10:09 jynus: enabling ferm, and restarting mysql at db1046 (m4-master, eventlogging)
  • 10:02 jynus: stopping eventlogging mysql consumers
  • 09:32 moritzm: umounted/remounted hdfs mount on stat1002 (got stuck due to kernel bug, see T121492)
  • 09:32 logmsgbot: hashar@tin Synchronized wmf-config/InitialiseSettings-labs.php: enable EventBus logging channel (currently only in beta) https://phabricator.wikimedia.org/T116786 (duration: 08m 57s)
  • 08:47 hashar: restarted zuul-merger on gallium
  • 08:29 hashar: stopping zuul-merger on gallium for maintenance
  • 07:48 _joe_: rebooting pollux
  • 07:18 _joe_: logged into console on pollux, that made it responsive
  • 03:09 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/259011/ and https://gerrit.wikimedia.org/r/#/c/259010/1 (duration: 00m 30s)
  • 02:32 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Dec 15 02:32:25 UTC 2015 (duration 6m 59s)
  • 02:25 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.8) (duration: 11m 38s)
  • 02:03 ori: deployed I6ebffe559 to job runners
  • 01:32 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Revert cirrus avro schema to 101446746400 due to camus not picking up events from new schema (duration: 00m 30s)
  • 00:34 Reedy: puppet disabled on silver since last Puppet run was at Fri Dec 11 15:27:28 UTC 2015 due to 'testing osm debugging'
  • 00:33 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Graph config updates (duration: 00m 28s)
  • 00:32 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: SWAT: Graph config updates (duration: 00m 29s)
  • 00:19 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 29s)
  • 00:17 logmsgbot: catrope@tin Synchronized wmf-config/: SWAT: Use event-schemas repository for avro schemas (duration: 00m 29s)
  • 00:10 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/MobileFrontend/: SWAT: fix page invalidation in mobile API (duration: 00m 36s)
  • 00:06 logmsgbot: catrope@tin Synchronized wmf-config/: SWAT: Wikidata config changes (duration: 00m 28s)

2015-12-14

  • 23:47 gwicke: restbase: finished deploy of 9f31847ad to restbase prod cluster
  • 23:39 logmsgbot: aaron@tin Synchronized rpc/RunJobs.php: 82f4f9df64e42fd04bd32395 (duration: 00m 29s)
  • 23:34 gwicke: restbase: starting deploy of 9f31847ad to restbase prod cluster
  • 22:50 logmsgbot: mobrovac@tin Synchronized wmf-config/InitialiseSettings.php: Enable logging for the Math extension (duration: 00m 29s)
  • 22:49 gwicke: restbase: canary deploy of 9f31847ad to restbase1001
  • 22:16 robh: puppet enabled on uranium and resumed normal service, cert updated for ganglia
  • 22:10 robh: puppet disabled on uranium for ganglia cert update
  • 22:09 mobrovac: mathoid deploying 5b20fe1
  • 21:34 logmsgbot: ori@tin Synchronized docroot and w: (no message) (duration: 00m 29s)
  • 21:15 subbu: finished deploy of parsoid sha df3171e6
  • 21:11 subbu: restarted parsoid on wtp1004 (~4 mins back) as a canary
  • 21:04 subbu: starting parsoid deploy
  • 21:01 robh: neon returned to normal puppet updates. icinga new cert is live.
  • 20:57 mutante: mw1259 - enabled hyperthreading (logical processor)
  • 20:54 mutante: rebooting mw1259 for BIOS settting
  • 20:50 robh: puppet on neon disabled for certificate update
  • 20:25 yurik: updated graphoid service
  • 20:15 hashar: scandium restarted zuul-merger
  • 20:00 jynus: changing master of db2017 to be now db1018, instead of s2-master
  • 19:20 jynus: rolling restart s2 codfw mysql servers
  • 19:02 mutante: reboot cygnus
  • 18:37 jynus: restarting, upgrading and reconfiguring mysql on db1018
  • 18:22 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1057 after maintenance; depool db1018 for maintenance (duration: 00m 29s)
  • 17:20 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.8/extensions/ContentTranslation/api/ApiContentTranslationToken.php: SWAT: Fix check for JWT gerrit:258776 (duration: 00m 29s)
  • 17:05 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.8/extensions/UploadWizard/resources/mw.UploadWizardDetails.js: SWAT: mw.UploadWizardDetails update gerrit:258767 (duration: 00m 30s)
  • 16:27 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.8/extensions/MobileApp/config/config.json: SWAT: Roll out RESTBase usage to Android Beta app: 55% Part II gerrit:258985 (duration: 00m 29s)
  • 16:14 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.8/extensions/MobileApp/config/config.json: SWAT: Roll out RESTBase usage to Android Beta app: 55% gerrit:258985 (duration: 00m 30s)
  • 16:13 jynus: performing schema change on testwiki.page (s3)
  • 16:05 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: do not index User namespace on ko.wikipedia gerrit:258667 (duration: 00m 29s)
  • 15:01 bblack: "service ganglia-monitor restart" on cp*, because most had no varnish stats flowing into ganglia
  • 14:47 hashar: Stopping zuul-merger daemon on scandium. It lost its disk somehow earlier "DISK CRITICAL - /srv/ssd is not accessible: No such file or directory"
  • 13:31 bblack: rebooting cp4005
  • 13:30 bblack: issues on cp4005, investigating
  • 13:29 mobrovac: restbase cassandra restarting cassandra on rb1004 due tolow disk space caused by compactions
  • 13:22 godog: repool restbase1007
  • 12:04 mobrovac: restbase deploy on rb1007 after re-imaging
  • 11:31 godog: upgrade diamond to 3.5-5 in eqiad
  • 11:13 dcausse: elastic in eqiad: resuming writes
  • 10:50 jynus: performing schema change on wikishared.cx_translations (x1-master)
  • 10:19 dcausse: elastic in eqiad: restarting elastic1016 (to release deleted filedesc)
  • 10:11 jynus: restarting mysql at labsdb1004
  • 10:11 dcausse: elastic in eqiad: freezing writes (to restart elastic1016)
  • 09:46 godog: reimage restbase1007
  • 02:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Dec 14 02:29:39 UTC 2015 (duration 6m 57s)
  • 02:22 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.8) (duration: 09m 00s)

2015-12-13

  • 11:10 dcausse: elastic in eqiad: restarting elastic1012 to release opened log filedesc
  • 10:54 godog: extend fluorine's storage lv (94% util) lvresize --size +300G --resizefs /dev/mapper/vg0-lv0
  • 10:24 dcausse: elastic in eqiad: disabling TRACE indexing slowlog on jawiki_content and zhwiki_content
  • 07:06 ori: elasticsearch1012 ditto.
  • 07:05 ori: elasticsearch1016: Moving /var/log/elasticsearch/* to /var/lib/elasticsearch/logs to free up space.
  • 02:28 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Dec 13 02:28:52 UTC 2015 (duration 6m 49s)
  • 02:22 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.8) (duration: 09m 08s)

2015-12-12

  • 22:17 ejegg: updated fundraising dashboard from 961a24c9cf76966aaa7ba8c60e13b6d1d37fa859 to 59e51c4ff74c3c584daf6c5de3bb66daa764cd28
  • 22:06 ejegg: updated fundraising dashboard from 34dee88d137aa1d0c4487a3a94b87e7ed2f8d0c4 to 961a24c9cf76966aaa7ba8c60e13b6d1d37fa859
  • 21:26 legoktm: running fixDefaultJsonContentPages.php on all wikis (T108663)
  • 21:02 logmsgbot: ori@tin Synchronized wmf-config/jobqueue-eqiad.php: I53f13a159: Add rdb100[5-6] to job queue configuration (duration: 00m 31s)
  • 19:30 hashar: Restarted Zuul
  • 17:01 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/258669/ (duration: 00m 29s)
  • 10:12 godog: bounce nslcd on tools-submit and stop puppet
  • 10:01 dcausse: elastic in eqiad: disabling TRACE indexing slowlog for urwiki_content
  • 09:44 jynus: recreated events on db1057 with sql_bin_log = 0 and restarted replication on db2016
  • 09:15 godog: move old elasticsearch logs on elastic1026 out to /var/lib/elasticsearch/log (/ is full)
  • 09:06 godog: move old elasticsearch logs on elastic1016 out to /var/lib/elasticsearch/log (/ is full)
  • 09:02 godog: move old elasticsearch logs on elastic1012 out to /var/lib/elasticsearch/log (/ is full)
  • 03:02 legoktm: ran fixDefaultJsonContentPages.php --wiki=thwiktionary for T108663
  • 02:28 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Dec 12 02:28:44 UTC 2015 (duration 6m 53s)
  • 02:21 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.8) (duration: 08m 31s)
  • 00:30 paravoid: restarting slapd on seaborgium with manual hack

2015-12-11

  • 23:59 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/CentralNotice: Improve impression diet state machine (duration: 00m 32s)
  • 22:24 subbu: finished deploying parsoid sha ebd62ab5
  • 22:09 subbu: restarted parsoid on wtp1002 as a canary
  • 22:03 subbu: starting parsoid deploy
  • 21:18 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.8/extensions/OpenStackManager/: deploying https://gerrit.wikimedia.org/r/#/c/258516/ (duration: 00m 28s)
  • 20:55 jynus: stopping and reconfiguring replication on db2016 (only for some minutes)
  • 20:48 jynus: setting db1057 as the parent mysql of db2016 for s1 replication
  • 20:29 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/Flow/maintenance/FlowPopulateRefId.php: Fix fatals due to missing wiki condition (T117786) (duration: 00m 29s)
  • 20:06 jynus: restarting and reloading mysql for upgrade and reconfiguration at db1057
  • 19:33 logmsgbot: demon@mira Synchronized README: testing (duration: 01m 35s)
  • 19:26 legoktm: started foreachwiki checkLocalUser.php (CentralAuth) on terbium
  • 19:14 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: CentralAuth logging for Lego (duration: 00m 28s)
  • 19:13 mutante: creating cygnus.codfw.wmnet on ganeti2001
  • 19:06 logmsgbot: demon@tin Synchronized wmf-config/wikitech.php: rm some old ldap stuff. yay hiera! (duration: 00m 28s)
  • 18:54 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: rm some deprecated profiling config (duration: 00m 28s)
  • 18:29 logmsgbot: demon@tin Synchronized docroot/: cleanup wgconf vhost config (the sequel) (duration: 00m 29s)
  • 18:28 logmsgbot: demon@tin Synchronized wmf-config/: cleanup wgconf vhost config (duration: 00m 29s)
  • 18:21 cwd: updated payments from 74143e43eb36f93be7881f626443182d3bb58cef to a1be1ad134d06464e98de180227554fceddc91d4
  • 18:13 logmsgbot: demon@tin Synchronized wmf-config/: Remove ability to turn off Cirrus completely. It's a scary switch that would Break Everything. There's much better options to use if you need to tune its load or behavior. (duration: 00m 29s)
  • 17:54 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1057 for maintenance (duration: 00m 35s)
  • 17:44 jynus: rolling restart of codfw's S1 mysqls finished
  • 16:36 mark: Installed bgp-med branch of operations/debs/pybal on pybal-test2001
  • 16:23 mark: moved quagga install from pybal-test2001 to pybal-test2002
  • 14:54 jynus: restarting and configuring S1:codfw mysqls (db2016,34,42,48,55,62,69,70)
  • 14:40 mark: root@pybal-test2001:~# apt-get install quagga
  • 14:21 andrewbogott: re-imaging promethium
  • 10:57 _joe_: rolling restart of HHVM across the jobrunners to ease memory consumption (this will go on during the day, hhvm restarts will be distantiated by 2 hours each)
  • 10:53 _joe_: restarting hhvm on mw1122 in order to collect heap dumps
  • 10:35 jynus: restarting HHVM on mw1122, mw1145
  • 09:50 jynus: importing several tables from master to dbstore1002 and labs, lag will will be slightly affected
  • 09:26 _joe_: ganeti-rebooting serpens, cannot get into console
  • 07:10 dcausse: deleting some indexing slow logs on elastic1012 and elastic1016
  • 06:46 gwicke: deploy restbase c67a41e9d52: add an expensive title to re-render blacklist, per subbu's request
  • 03:21 gwicke: cassandra: upgraded codfw production nodes to 2.1.12
  • 02:46 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Dec 11 02:46:13 UTC 2015 (duration 6m 46s)
  • 02:39 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.8) (duration: 17m 01s)
  • 02:34 gwicke: restarted nodetool decommission on restbase1007, as I had to restart one of the stream targets
  • 02:15 bblack: unified cert upgrades complete
  • 01:42 bblack: starting unified cert upgrade process
  • 01:13 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.8/includes/jobqueue: Ia44ec5ed4: Add per-partition JobQueueRedis aggregation + Fix bad regex in 6fe2f48df (duration: 00m 31s)
  • 00:59 andrewbogott: rebooting promethium
  • 00:58 logmsgbot: bd808@tin Synchronized wmf-config/InitialiseSettings.php: Add read-more to beta features whitelist (duration: 00m 30s)
  • 00:27 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Search language detection A/B test and RelatedArticles on all Wikipedias in beta (duration: 00m 29s)
  • 00:20 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/CirrusSearch: SWAT (duration: 00m 30s)
  • 00:20 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/RelatedArticles: SWAT (duration: 00m 29s)
  • 00:19 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/ImageMap: SWAT (duration: 00m 29s)
  • 00:19 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/Graph: SWAT (duration: 00m 30s)
  • 00:08 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Staff password policy, take 2 (duration: 00m 28s)
  • 00:06 logmsgbot: catrope@tin Synchronized portals/: SWAT (duration: 00m 30s)

2015-12-10

  • 23:55 gwicke: restbase: deploy 9657c4e
  • 23:38 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/Math: Fix errors (duration: 00m 29s)
  • 23:38 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/Echo: Fix errors (duration: 00m 29s)
  • 23:27 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.8/extensions/WikimediaEvents: Ia44ec5ed4: Updated mediawiki/core Project: mediawiki/extensions/WikimediaEvents 152ecb10311bb04f4f2f91775cf821aff14aa327 (duration: 00m 30s)
  • 23:20 logmsgbot: krenair@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 29s)
  • 22:48 csteipp: Re-deployed a bunch of security patches for wmf8
  • 22:05 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/VisualEditor/: SWAT (duration: 00m 47s)
  • 21:56 gwicke: restbase: finished full deploy of 7398850fe9 to restbase cluster
  • 21:49 gwicke: restbase: starting full deploy of 7398850fe9 to restbase cluster
  • 21:49 logmsgbot: catrope@tin Synchronized wmf-config/: SWAT: VisualEditor config patches (duration: 00m 30s)
  • 21:46 gwicke: restbase: canary deploy of 7398850fe9 to restbase1001
  • 21:44 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/Gadgets/: Fix MediaWiki:MediaWiki: (duration: 00m 29s)
  • 21:35 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: New password policy for staff group (duration: 00m 28s)
  • 21:28 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: SWAT: Echo and Flow config patches (duration: 00m 29s)
  • 21:27 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Echo and Flow config patches (duration: 00m 29s)
  • 21:20 Reedy: Changed email for mbeattie on otrs_wikiwiki per request of mdennis
  • 20:32 subbu: finished parsoid deploy
  • 20:14 subbu: restarted parsoid on wtp1003 as canary
  • 20:10 subbu: starting parsoid deploy
  • 19:11 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.27.0-wmf.8
  • 18:40 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1014 at 100% load and reduce es1019 load (duration: 00m 29s)
  • 18:33 paravoid: reprepro: updating cassandra to latest upstream (2.1.12)
  • 17:25 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1011 at 100% load, repool es1014 with low load (duration: 00m 29s)
  • 17:06 bd808: Updated scholarships.wikimedia.org to a4e3dbf (Ensure that Auth\UserData is constructed with an array)
  • 16:45 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.7/extensions/MobileApp/config/config.json: SWAT: Roll out RESTBase usage to Android Beta app: 30% gerrit:258159 (duration: 00m 29s)
  • 16:42 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.8/extensions/MobileApp/config/config.json: SWAT: Roll out RESTBase usage to Android Beta app: 30% gerrit:258161 (duration: 00m 29s)
  • 16:32 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add three new groups to pawiki, and allow sysops to add or remove users to them gerrit:257881 (duration: 00m 29s)
  • 16:26 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add a rollbacker group at wuuwiki gerrit:253084 (duration: 00m 28s)
  • 16:19 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable filemover group at ukwiki gerrit:255422 (duration: 00m 29s)
  • 16:14 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Nuke and unblockself only for bureaucrats on en.wikiversity gerrit:254029 and Add new group "curator" to enwikiversity gerrit:252012 (duration: 00m 36s)
  • 16:00 jynus: restarting, upgrading and configuring es1014
  • 15:44 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Switchover es3 master es1014 -> es1019 and depool es1014 (duration: 00m 31s)
  • 15:44 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Switchover es3 master es1014 -> es1019 and depool es1014 (duration: 00m 28s)
  • 15:22 jynus: perming es3 master switchover of es1014 -> es1019, no production impact is expected
  • 15:13 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1011 with lower weight after mantenance (duration: 00m 29s)
  • 13:57 jynus: restarting, upgrading and reconfiguring mysql @ es1011
  • 13:45 moritzm: stopping/starting slapd on seaborgium to update LDAP indices
  • 13:39 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Switchover codfw es2 master es1011 -> es1015 (duration: 00m 29s)
  • 13:39 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Switchover es2 master es1011 -> es1015 (duration: 00m 37s)
  • 13:30 paravoid: reenabled puppet on seaborgium & forced a puppet run
  • 13:05 moritzm: restarted slapd on serpens
  • 13:03 moritzm: stopping slapd on serpens to update LDAP indices
  • 12:54 moritzm: re-enabled puppet on serpens / restarted slapd
  • 12:53 jynus: performing switchover of es2 es1011-> es1015 for master maintenance, no production impact is expected
  • 12:13 godog: powercycle mw1119, login on console sluggish and no ssh
  • 12:11 jynus: codfw es2 server restart finished
  • 09:04 jynus: rebooting, restarting and upgrading mysql on es2 (codfw external storage servers)
  • 05:51 ejegg|away: updated fundraising dashdoard from 9f0a4d98586f1281716e39b34d924de524458177 to 34dee88d137aa1d0c4487a3a94b87e7ed2f8d0c4
  • 03:55 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Dec 10 03:55:07 UTC 2015 (duration 54m 36s)
  • 03:00 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.8) (duration: 15m 54s)
  • 02:28 gwicke: updated restbase1008 to cassandra 2.1.12
  • 02:27 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 10m 53s)
  • 01:25 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.7/extensions/CirrusSearch/: Add master timeout parameter to mapping updates so it can be increased when the master is slow (duration: 00m 30s)
  • 01:23 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.8/extensions/CirrusSearch/: Add master timeout parameter to mapping updates so it can be increased when the master is slow (duration: 00m 29s)
  • 01:20 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-production.php: Increase elasticsearch master timeout for maint actions to 2 minutes (duration: 00m 31s)
  • 01:05 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/WikimediaMaintenance/: Fix refreshMessageBlobs script (duration: 00m 29s)
  • 01:01 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/WikimediaMaintenance/: Fix refreshMessageBlobs script (duration: 00m 29s)
  • 00:55 awight: update fundraising CRM from 49e949c303466db9f96cb38b718190c887f7ed89 to fa7124e1d8d92b576c7650030fe64d024c822088
  • 00:54 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/: MessageBlobStore changes (duration: 02m 29s)
  • 00:51 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/skins/Vector/: Make placeholder in logged-out personal bar greyed out (duration: 00m 29s)
  • 00:26 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/Thanks/: Fix bug with topic titles in Flow thank notifications (duration: 00m 28s)
  • 00:18 logmsgbot: catrope@tin Synchronized wmf-config/mobile.php: Enable A/B test to measure impact of collapsing content (duration: 00m 29s)
  • 00:06 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Set $wgExtDistGraphiteRenderApi (duration: 00m 28s)
  • 00:02 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings-labs.php: Shut up unmerged change warning (duration: 00m 30s)

2015-12-09

  • 23:43 papaul: signing salt-key complete
  • 23:28 logmsgbot: hoo@tin Synchronized php-1.27.0-wmf.8/extensions/Wikidata: Item/ Property id values need separate validators each (duration: 00m 39s)
  • 23:26 papaul: auth2001 signing puppet certs
  • 21:52 papaul: OS install complete on auth2001
  • 21:34 subbu: finished deploying parsoid sha a0c626e4
  • 21:30 hashar: Jenkins upgraded to 1.625.3 ( https://phabricator.wikimedia.org/T56599 )
  • 21:22 subbu: restarted parsoid on wtp1003 as a canary
  • 21:20 hashar: Jenkins going down right now.
  • 21:18 subbu: starting parsoid deploy
  • 21:11 hashar: Stopping Jenkins
  • 19:56 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: hewiki and cawiki to php-1.27.0-wmf.8
  • 19:46 logmsgbot: thcipriani@tin Synchronized dblists/group1.dblist: Train: Add hewiki and cawiki to group1. Part II gerrit:257973 (duration: 00m 29s)
  • 19:45 logmsgbot: thcipriani@tin Synchronized dblists/group1-wikipedia.dblist: Train: Add hewiki and cawiki to group1. Part I gerrit:257973 (duration: 00m 33s)
  • 19:10 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.8
  • 19:02 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.7/extensions/CirrusSearch/includes/Job/Job.php: Revert manual adjustment made to CirrusSearch Job.php (duration: 00m 29s)
  • 18:56 mutante: mw1003 - restarted hhvm (and apache a bit earlier too)
  • 18:16 Jamesofur: inserted decryption key into database for enWP Arbcom case
  • 17:31 akosiaris: restarted hhvm on mw1125, memory starvation
  • 17:31 akosiaris: restarted hhvm on mw1125
  • 17:17 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.8/extensions/Wikidata: SWAT: Update Wikibase gerrit:257901 (duration: 00m 49s)
  • 17:05 logmsgbot: krenair@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 28s)
  • 16:44 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add rights to CU+OS groups on en.wikipedia.org gerrit:255129 (duration: 00m 29s)
  • 16:32 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Turn off language detection user test gerrit:254071 (duration: 00m 29s)
  • 16:25 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Set initial titlesuggest shard sizes gerrit:256974 (duration: 00m 29s)
  • 16:18 logmsgbot: thcipriani@tin Synchronized dblists/group1.dblist: SWAT: Move Catalan and Hebrew Wikipedias to group1 gerrit:253371 (duration: 00m 29s)
  • 16:12 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable importupload and some import sources for officewiki gerrit:256881 (duration: 00m 42s)
  • 16:06 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add config for wgVisualEditorUseSingleEditTab and set true for enwiki Labs Part II gerrit:257800 (duration: 00m 28s)
  • 16:05 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Add config for wgVisualEditorUseSingleEditTab and set true for enwiki Labs Part I gerrit:257800 (duration: 00m 29s)
  • 15:48 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: etherpad-lite_1.5.7-2
  • 15:01 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1019 after maintenance (duration: 00m 33s)
  • 14:09 godog: decommission restbase1007
  • 13:06 hashar: restarting Jenkins (upgrading Gearman plugin)
  • 11:51 akosiaris: reissue certificates for labs-ldap.{eqiad,codfw}.wikimedia.org with a 2048bit key size
  • 11:29 kart_: Update cxserver to c88ef57
  • 10:45 akosiaris: sudo gnt-instance modify -H cpu_type=SandyBridge seaborgium to get aesni and sse cpu flags. rebooting seaborgium to apply those
  • 10:25 akosiaris: reboot serpens to add 3 more vCPUs
  • 10:04 moritzm: restarting slapd on serpens/seaborgium to apply updated indices
  • 10:03 ori: running hhvm-collect-heaps in screen on mw1001 with a 600s interval
  • 10:00 akosiaris: stopped slapd on seaborgium, run sudo -u openldap slapindex to add the new indices, started slapd
  • 09:58 akosiaris: stopped slapd on serpens, run sudo -u openldap slapindex to add the new indices, started slapd
  • 08:15 ori: Memory utilization on the job runners began to climb around 2015-12-08T10:00:00+00:00 UTC. To investigate, enabled jemalloc profiling and restarted HHVM on mw1001.
  • 07:40 YuviPanda: restarted create-dbusers on labstore1001
  • 04:44 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Dec 9 04:44:57 UTC 2015 (duration 1h 50m 43s)
  • 02:54 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.8) (duration: 04m 34s)
  • 02:34 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 10m 27s)
  • 02:23 logmsgbot: ori@tin Synchronized wmf-config/wikitech.php: I4ef826af47: Increase LDAP log verbosity (duration: 00m 44s)
  • 02:16 logmsgbot: catrope@tin Finished scap: Graph cherry-picks for wmf.8, including i18n changes (duration: 26m 38s)
  • 01:49 logmsgbot: catrope@tin Started scap: Graph cherry-picks for wmf.8, including i18n changes
  • 01:45 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/Gadgets/: SWAT: bump MediaWikiGadgetsDefinitionRepo cache version (duration: 00m 29s)
  • 01:19 awight: Updating paymentswiki from 96d8a49ecd521ac8ce75521aaf97ed8f0dbefc9a to 74143e43eb36f93be7881f626443182d3bb58cef
  • 00:36 ejegg: updated paymentswiki from ff7a706219e72f7362f4e06386e6732933a40478 to 96d8a49ecd521ac8ce75521aaf97ed8f0dbefc9a

2015-12-08

  • 23:47 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: Train: group0 to 1.27.0-wmf.8
  • 23:43 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.8/extensions/MobileFrontend: I7b86a521: Ensure the parser cache varies on images disabled and 'light' images (duration: 00m 32s)
  • 23:41 logmsgbot: thcipriani@tin Finished scap: testwiki to 1.27.0-wmf.8 and rebuild l10n cache take II (duration: 56m 13s)
  • 22:45 logmsgbot: thcipriani@tin Started scap: testwiki to 1.27.0-wmf.8 and rebuild l10n cache take II
  • 22:38 eileen: Updating civicrm from d385e34c775aec50a5c17357c8a11cb0244123ce to 291816c88ab07bb914d56c70ff3d58eb1aa0124d
  • 22:37 eileen: Updating civicrm from xyz to abc
  • 22:24 logmsgbot: thcipriani@tin scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="cawikibooks" --list-file="/srv/mediawiki-staging/wmf-config/extension-list" --output="/tmp/tmp.d9wYd5Axk0" ' returned non-zero exit status 1 (duration: 01m 01s)
  • 22:23 logmsgbot: thcipriani@tin Started scap: testwiki to 1.27.0-wmf.8 and rebuild l10n cache
  • 22:16 yurik: deployed latest kartotherian & tilerator
  • 21:09 thcipriani: checking out wmf/1.27.0-wmf.8 on tin for train deploy
  • 20:51 moritzm: moved dns aliases ldap-eqiad.wikimedia.org and ldap-codfw.wikimedia.org to seaborgium/serpens (these are for compat reasons, using ldap-labs.[eqiad|codfw].wikimedia.org is preferred
  • 20:50 gwicke: restbase: finished deploy of 49fdc615b to all nodes
  • 20:37 gwicke: restbase: start deploy of 49fdc615b to all nodes
  • 20:35 hashar: Jenkins: changing LDAP config from ldaps://ldap-eqiad.wikimedia.org:636 to ldaps://ldap-labs.eqiad.wikimedia.org:636
  • 20:34 gwicke: restbase: canary deploy of 49fdc615b to restbase1001
  • 20:31 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.7/extensions/MobileFrontend: I8a1fb69724b: Vary HTML output by NetSpeed designation (duration: 00m 28s)
  • 19:52 akosiaris: fixed ldap logging on seaborgium/serpens
  • 19:51 logmsgbot: ori@tin Synchronized wmf-config/InitialiseSettings.php: Enable ldap log channel (duration: 00m 28s)
  • 19:50 cmjohnson1: swapping failed disk db1019
  • 18:53 akosiaris: running puppet on serpens/seaborgium
  • 18:49 paravoid: enable ldaps:/// on seaborgium
  • 18:49 akosiaris: enable ldaps:/// on serpens
  • 18:36 logmsgbot: demon@tin Synchronized wmf-config/wikitech.php: Point to new ldap servers (duration: 00m 27s)
  • 18:25 andrewbogott: running puppet on labcontrol1002
  • 18:23 andrewbogott: running puppet on labcontrol1001
  • 17:35 legoktm: deleted centralauth.localuser row for User:ThanduxoloKennethTwani5092/enwiki because the account doesn't exist (T120655)
  • 17:13 moritzm: stopping slapd on seaborgium/serpens to refresh LDAP databases with latest export from nembus
  • 17:00 andrewbogott: changing opendj writability-mode to ‘internal-only’ on nembus and neptunium
  • 16:31 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.7/extensions/CentralNotice: SWAT: Update CentralNotice gerrit:257615 (duration: 00m 28s)
  • 16:19 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix typo in namespaces configuration gerrit:257487 (duration: 00m 28s)
  • 16:13 logmsgbot: thcipriani@tin Synchronized wmf-config/Wikibase.php: SWAT: Wikidata: set maxSerializedEntitySize to 2500 gerrit:257318 (duration: 00m 27s)
  • 16:07 godog: repool restbase1008
  • 16:06 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: CX: Use ContentTranslationRESTBase gerrit:255102 (duration: 00m 30s)
  • 16:01 mobrovac: restbase end deploy of 28bf071
  • 15:22 mobrovac: restbase enable puppet in prod
  • 15:17 mobrovac: restbase start deploy of 28bf071
  • 15:10 mobrovac: restbase canary deploy of 28bf071 onto restbase1001
  • 14:19 bblack: upgrading openssl package on caches
  • 14:16 mobrovac: restbase canary deploy of f47405a onto restbase1001
  • 14:09 mobrovac: restbase running puppet on rb1001
  • 11:02 godog: upgrade diamond to 3.5-5 in codfw
  • 10:04 godog: start cassandra on restbase1008, bootstrapping
  • 09:56 hashar: gallium: upgraded python-diamond and java. Restarting Jenkins
  • 09:48 hashar: gallium: removing bunch of :i386 packages which were used for Android build but got removed ( https://gerrit.wikimedia.org/r/#/c/183790/3/modules/androidsdk/manifests/dependencies.pp,unified )
  • 09:15 godog: reimage restbase1008
  • 05:45 gwicke: restbase1008: disable cassandra with `systemctl mask cassandra`
  • 04:00 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Dec 8 04:00:02 UTC 2015 (duration 1h 34m 44s)
  • 03:12 gwicke: Keeping puppet on restbase production cluster disabled until Marko & Filippo can deploy tomorrow morning. PLEASE DO NOT ENABLE!
  • 02:25 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 09m 56s)
  • 01:16 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Also blacklist Minerva for WPB (duration: 00m 28s)
  • 00:59 gwicke: disabling puppet in production restbase cluster in preparation for testing https://gerrit.wikimedia.org/r/#/c/257408/ in staging
  • 00:39 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/extensions/MobileFrontend/: SWAT (duration: 00m 28s)
  • 00:38 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/extensions/MwEmbedSupport/: SWAT (duration: 00m 28s)
  • 00:37 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/resources/src/oojs-ui-local.js: SWAT: connect OOjs UI to MW's l10n system (duration: 00m 28s)
  • 00:37 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/resources/ResourcesOOUI.php: SWAT: connect OOjs UI to MW's l10n system (duration: 00m 27s)
  • 00:34 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: enable banners on mobile web beta on enwiki; remove MobileFrontend Browse config (duration: 00m 27s)
  • 00:31 logmsgbot: catrope@tin Synchronized wmf-config/squid.php: SWAT: stop using separate address for upload in wgHTCPRouting (duration: 00m 29s)
  • 00:07 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Enable Flow opt-in beta feature on bswiki and urwiki (duration: 00m 28s)

2015-12-07

  • 22:29 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.7/extensions/MobileFrontend: I8c94425693: Improve disableImages cookie code (T120151) (duration: 00m 30s)
  • 22:22 logmsgbot: ebernhardson@tin Synchronized portals/prod: Deploy updated portals (duration: 00m 27s)
  • 22:01 mdholloway: mobileapps deployed 984f14b
  • 21:31 logmsgbot: hoo@tin Synchronized php-1.27.0-wmf.7/extensions/Wikidata/: (no message) (duration: 00m 37s)
  • 21:15 subbu: finished deploying parsoid sha 4a7df427 (cf0b9ef + cherry-pick of d65debd)
  • 21:09 subbu: restart parsoid on wtp1002 as a canary
  • 21:05 subbu: starting parsoid deploy
  • 20:33 andrewbogott: enabling puppet on labcontrol1002, testing over
  • 20:07 andrewbogott: disabling puppet on labcontrol1002 for an ldap/pdns test
  • 19:52 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1015 at 100% load (duration: 00m 27s)
  • 19:52 andrewbogott: restarting nova-compute on all labvirt1X nodes
  • 19:20 andrewbogott: restarted pdns on labcontrol1001 and 1002 to catch up with recent opendj restarts
  • 19:11 andrewbogott: restarting ldap on neptunium and nembus due to replication warnings in the logs
  • 18:36 jynus: es1019 and its management interface became unresponsive
  • 18:16 mutante: deleting pmtpa host groups from nagios/icinga
  • 17:04 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.7/extensions/MobileApp/config/config.json: SWAT: Roll out RESTBase usage to Android Beta app: 10% gerrit:254045 (duration: 00m 29s)
  • 17:02 thcipriani: reset uncommited change to includes/jobqueue/JobRunner.php in /srv/mediawiki-staging/php-1.27.0-wmf.7
  • 17:00 hashar: Restarted Jenkins to start the ZeroMQ publisher https://phabricator.wikimedia.org/T120668
  • 16:56 _joe_: upgrading pybal on primaries in esams, ulsfo, codfw
  • 16:44 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Namespace configuration for en.wiktionary gerrit:254476 and Enable subpages on custom aliases from 112 to 119 gerrit:254475 (duration: 00m 27s)
  • 16:43 _joe_: installing pybal on eqiad backups
  • 16:43 jynus: restarting mysql, upgrading and rebooting mysql on es1019
  • 16:36 _joe_: installing the new pybal version on the backup lvss in esams and ulsfo
  • 16:31 _joe_: upgrading pybal on all the backup LVSs in codfw
  • 16:28 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Rights configuration on fa.wikipedia gerrit:254486 (duration: 00m 29s)
  • 16:28 _joe_: upgrading pybal on lvs1007-12
  • 16:19 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Namespace configuration for ur.wikipedia gerrit:254665 (duration: 00m 30s)
  • 16:18 _joe_: uploaded pybal 1.13.2 to reprepro
  • 16:14 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Translate project namesapce for jbowiki gerrit:255954 (duration: 00m 28s)
  • 15:39 moritzm: uploaded openssl 1.0.2e-1~wmf1 to carbon
  • 15:37 yurik: deployed kartotherian
  • 15:25 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1019; es1017 at 100% load; pool es1015 with low weight (duration: 00m 28s)
  • 14:56 yurik: deployed latest tilerator
  • 14:46 _joe_: also restarted pybal on lvs3003
  • 14:42 _joe_: restarted pybal on lvs1006
  • 14:39 hashar: restarting Jenkins
  • 14:32 hashar: Jenkins lost a bunch of executors :/
  • 14:30 hashar: CI / Zuul stalled somehow
  • 12:56 jynus: rolling restart, configuration upgrade of es1015
  • 12:20 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1015; es1013 at 100% load; pool es1017 with low weight (duration: 00m 28s)
  • 11:08 YuviPanda: restarting pdns on holmium
  • 10:52 jynus: database and system maintenance to es1017
  • 10:43 hashar: CI / zuul / nodepool recovered. Root cause was some malfunction in openstack wmflabs
  • 10:20 YuviPanda: restarted nova-conductor and scheduler on labcontrol1001
  • 10:07 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1013 (lower weight for now) and depool es1017 (duration: 00m 41s)
  • 10:05 hashar: stopped Nodepool. Can not create instances anymore on wmflabs ( https://phabricator.wikimedia.org/T120586 )
  • 09:46 hashar: restarting Nodepool on labnodepool1001.eqiad.wment
  • 09:40 hashar: CI / Zuul stalled. Nodepool can no more spawn instances :-/
  • 09:27 godog: nodetool decommission restbase1008
  • 09:13 jynus: es1013 maintenance (mysql restart, upgrade, possible reboot)
  • 08:27 _joe_: uploaded etcd 2.2 package from stretch to jessie-wikimedia
  • 03:56 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Dec 7 03:56:49 UTC 2015 (duration 1h 32m 22s)
  • 02:24 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 09m 59s)

2015-12-06

  • 21:48 ori: krypton unresponsive, nothing on console. shutting down, increasing instance ram from 2 to 4g, and rebooting.
  • 18:49 legoktm: reset auth token for User:QuimGil
  • 05:50 mutante: silver gzip /var/log/nutcracker.log.1
  • 05:40 mutante: silver: apt-get clean for disk space
  • 03:57 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Dec 6 03:57:02 UTC 2015 (duration 1h 31m 41s)
  • 02:25 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 10m 04s)

2015-12-05

  • 18:30 gwicke: started nodetool decommission on restbase1008
  • 11:35 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: Disable common password password policy to come in wmf.8 (duration: 00m 28s)
  • 11:23 logmsgbot: reedy@tin Purged l10n cache for 1.27.0-wmf.5
  • 11:22 logmsgbot: reedy@tin Synchronized php-1.27.0-wmf.7/extensions/WikimediaMaintenance/refreshMessageBlobs.php: Less waiting for slaves (duration: 00m 28s)
  • 11:13 logmsgbot: reedy@tin Synchronized docroot and w: Add jobqueue-labs to noc (duration: 00m 28s)
  • 08:59 bblack: offlined db1019 megacli disk 32:11
  • 06:09 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Dec 5 06:09:07 UTC 2015 (duration 3h 44m 18s)
  • 02:24 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 09m 59s)

2015-12-04

  • 22:20 ejegg: updated dash from bed9895563fc0f5d3b96791fae6ed104b7778f0d to 9f0a4d98586f1281716e39b34d924de524458177
  • 21:44 andrewbogott: disabling puppet on labcontrol1002 for ldap testing
  • 21:36 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.7/includes/Hooks.php: Iba0138a: Don't install a custom error handler for hooks (T117553) (duration: 00m 28s)
  • 21:26 awight: Update fundraising CRM from 8a3deedc04e4d3d1c1f78cb5b7ace7d8670f307a to f0ef0dab1fa7a85426a000c35750093d541fe4f5
  • 20:28 logmsgbot: ori@tin Synchronized wmf-config/jobqueue-eqiad.php: Idee6a1980: job queue: use instances on port 6378 as aggregators (duration: 00m 30s)
  • 19:21 ori: krypton: updated Grafana to 2.6.0-beta1 for bug fix for issue 3422
  • 15:52 Jeff_Green: add mx record for donate.wikimedia.org
  • 15:33 godog: ms-be2019 rebooted by itself, ilo event log shows "Uncorrectable Machine Check Exception (Board 0, Processor 2, APIC ID 0x00000038, Bank 0x00000003, Status 0xFE000040'00020135, Address 0x00000000'FEB82F63, Misc 0x00000000'00002285)"
  • 08:52 godog: reimage restbase1009
  • 05:59 gwicke: ran systemctl mask cassandra on restbase1009; it is important that this node does not start up.
  • 05:53 gwicke: moved /var/lib/cassandra out of the way in an attempt to stop puppet restarting cassandra on decommissioned restbase1009
  • 05:49 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Dec 4 05:49:46 UTC 2015 (duration 3h 21m 36s)
  • 02:28 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 10m 19s)
  • 02:15 ori: CirrusSearch-common.php sync was for I826d000ca: Turn off backoff throttling of CirrusSearch jobs
  • 02:15 logmsgbot: ori@tin Synchronized wmf-config/CirrusSearch-common.php: (no message) (duration: 00m 29s)
  • 01:33 bd808: Updated scholarships.wikimedia.org to af73bf6
  • 00:35 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/extensions/CentralNotice: SWAT (duration: 00m 32s)
  • 00:01 ejegg: pointed SmashPig IPN listener to live Adyen account

2015-12-03

  • 23:09 bblack: restarting pybal (w/ BGP enabled) on lvs100[123] (newly-installed w/ jessie)
  • 22:59 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.7/includes/jobqueue/JobRunner.php: temporarily disable job throttling (duration: 00m 29s)
  • 22:35 cwd: updated paymentswiki configuration [redux]
  • 22:15 cwd: updated paymentswiki configuration
  • 22:09 bd808: Removed zirconium.wikimedia.org from Trebuchet minions list for scholarships/scholarships
  • 22:04 bd808: Updated scholarships.wikimedia.org to cb94319 plus local i18n filtering
  • 21:48 Reedy: finished removing bogus msg_resource rows
  • 21:28 logmsgbot: oblivian@tin Synchronized wmf-config/CommonSettings.php: re-sync (re-merged the change) (duration: 00m 29s)
  • 21:27 bd808: Applied database migrations and purged last year's data from Wikimania Scholarships db
  • 21:21 ottomata: restarted eventlogging with 4 mysql consumer processes running in parallel
  • 21:21 bblack: rebooting lvs100[123] for reinstall to jessie
  • 21:18 Reedy: Cleaning up msg_resource rows with bogus language codes
  • 21:15 gwicke: stopped cassandra on 1009 as it's decommissioned & will be reimaged
  • 21:13 logmsgbot: oblivian@tin Synchronized wmf-config/CommonSettings.php: Re-fix the jobqueue on wikitech after redis cleanup (duration: 00m 26s)
  • 20:55 logmsgbot: oblivian@tin Synchronized wmf-config/CommonSettings.php: Fix the jobqueue on wikitech (duration: 00m 47s)
  • 20:45 _joe_: opening connection from mw1001 to silver, mysql
  • 20:29 ori: on palladium: salt -G 'cluster:jobrunner' cmd.run 'service jobrunner status | grep running && service jobrunner restart'  ; salt -G 'cluster:jobrunner' cmd.run 'service jobchron status | grep running && service jobchron restart'
  • 20:28 ori: ran srem jobqueue:aggregator:s-wikis:v2 labswiki on rdb1001 aggr
  • 19:41 bblack: disabling pybal on lvs100[123] over the next few minutes (for reinstall to jessie later after confirmation everything is still ok on [456])
  • 19:10 jynus: restarting eventlogging_sync on db1047 and dbstore1002
  • 19:04 jynus: starting m4 slave again on dbstore2002
  • 18:45 andrewbogott: disabling puppet on labcontrol1002 to test openldap with pdns
  • 18:33 mutante: neon - remove icinga user from "dialout" group
  • 18:27 jynus: disabling eventlogging_sync process on dbstore1002 and db1047 and replication on the other m4 slaves
  • 18:18 jynus: disabling event scheduler on db1046 (m4-master)
  • 17:26 ejegg: updated fundraising dashboard from 8412a2f5f91bb3a9f7873d6ac9fd5d7884eb988c to bed9895563fc0f5d3b96791fae6ed104b7778f0d
  • 17:03 logmsgbot: kartik@tin Finished scap: Update ContentTranslation (duration: 05m 52s)
  • 16:57 logmsgbot: kartik@tin Started scap: Update ContentTranslation
  • 16:50 logmsgbot: oblivian@tin Synchronized wmf-config/CommonSettings.php: Fix the jobqueue on wikitech (duration: 00m 28s)
  • 15:23 andrewbogott: stopping pdns on labcontrol2001
  • 15:11 moritzm: restarting cassandra on restbase100[56] (subsequently) to effect openjdk security update
  • 14:57 mobrovac: restbase end of deployment of 262da91a
  • 14:48 mobrovac: restbase start deployment of 262da91a
  • 14:06 moritzm: installed dpkg updates across the cluster
  • 11:35 moritzm: restarting cassandra on aqs cluster (subsequently) to effect openjdk security update
  • 10:51 jynus: restarting, upgrading and general maintenance for es1013 (depooled)
  • 10:36 _joe_: imported dh-python into precise/universe from the ubuntu cloud archive
  • 10:26 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1013 for maintenance (duration: 00m 30s)
  • 05:50 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Dec 3 05:50:08 UTC 2015 (duration 50m 7s)
  • 02:32 ejegg: updated CiviCRM from d5afa10383cfd41adbe798c3ed2aa97839be8bc2 to 8a3deedc04e4d3d1c1f78cb5b7ace7d8670f307a
  • 02:25 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 09m 54s)
  • 00:18 awight: Deploy paymentswiki config for T112181

2015-12-02

  • 23:47 ejegg: updated fundraising dashboard from 01a82b5e4909075e3190496e7fcda2b60f8d5d3c to 8412a2f5f91bb3a9f7873d6ac9fd5d7884eb988c
  • 23:07 cwd: updated payments from 02a3771cae6e4e74d9d754a8b075de20aa171adc to ff7a706219e72f7362f4e06386e6732933a40478
  • 22:09 jynus: unscheduled restart of dbstore1002 (analytics-slave)
  • 21:44 jynus: disabling all alert notifications for dbstore1002
  • 21:30 bblack: rebooting lvs1007 for interface config test (not active, no BGP)
  • 20:37 Jeff_Green: enable mail queue monitoring for fundraising
  • 19:13 jynus: restarting mysql on db2067 to test a configuration change
  • 18:07 mutante: mw1136 - hhvm restart
  • 14:45 Coren: deploying cleanup of labs PAM configuration - this should be a functional noop but may cause some puppet noise
  • 12:41 akosiaris: restart cassandra on maps-test200{1,2,3,4}.codfw.wmnet
  • 11:23 moritzm: restarting cassandra on restbase100[78] (subsequently) (to effect openjdk security updates plus related libs)
  • 11:06 moritzm: restarting cassandra on restbase100[2-4] (subsequently) (to effect openjdk security updates plus related libs)
  • 11:04 moritzm: restart cassandra on restbase1001 (to effect openjdk security updates plus related libs)
  • 10:21 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enabling data access for wikinews, wikispecies and mediawiki.org (duration: 00m 27s)
  • 10:20 logmsgbot: aude@tin Synchronized dblists/arbitraryaccess.dblist: Enabling data access for wikinews, wikispecies and mediawiki.org (duration: 00m 30s)
  • 10:14 _joe_: clearing the job cache for ocg1003
  • 09.43 +
  • 09:22 _joe_: stopped ocg service on ocg1003
  • 08:38 _joe_: planet1001 recovered as soon as I did get into console via gnt-instance
  • 06:25 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.7/extensions/NavigationTiming: Idb675cdce: Add isHiDPI and isHttp2 properties; drop isHttps (T119014) (duration: 00m 50s)
  • 05:53 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Dec 2 05:53:21 UTC 2015 (duration 53m 20s)
  • 02:28 mutante|away: labcontrol2001 - disable puppet, kill from puppet stored configs
  • 02:26 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 10m 47s)
  • 00:29 mutante: puppetstoredconfigclean.rb labcontrol2001.wikimedia.org fixes icinga config

2015-12-01

  • 22:41 mutante: stop/start ntp service on lvs2002
  • 21:04 ejegg: updated fundraising from 2ee15180150c8b685b14abdbf7c6769736fb49a0 to 01a82b5e4909075e3190496e7fcda2b60f8d5d3c
  • 20:54 mutante: re-enabled puppet on cache::misc, replaced planet cert
  • 20:19 mutante: temp. disabling puppet on cache::misc
  • 18:14 twentyafterfour: restarted apache on iridium to deploy https://phabricator.wikimedia.org/rPHEXd724c51a4144f7de548546875097da57ee1b38d7
  • 17:05 moritzm: restarting cassandra instances on restbase200[3-6] (to effect openjdk security updates and a few depending libs)
  • 17:00 moritzm: restarting cassandra instances on restbase2002 (to effect openjdk security updates and a few depending libs)
  • 16:55 moritzm: restarting cassandra instances on restbase2001 (to effect openjdk security updates and a few depending libs)
  • 16:22 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor for all accounts on eswiki. Part II gerrit:250474 (duration: 00m 28s)
  • 16:21 logmsgbot: thcipriani@tin Synchronized dblists/visualeditor-default.dblist: SWAT: Enable VisualEditor for all accounts on eswiki. Part I gerrit:250474 (duration: 01m 29s)
  • 15:30 ejegg: changed donation qc time limit from 90 to 100 sec
  • 12:59 godog: reimage mw2208
  • 10:50 moritzm: restarting on cassandra instances on restbase1008 (to effect openjdk security updates and a few depending libs)
  • 06:52 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Dec 1 06:52:40 UTC 2015 (duration 52m 39s)
  • 02:26 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 10m 14s)
  • 02:18 mutante: restarted hhvm on mw1147
  • 01:47 logmsgbot: ori@tin Synchronized docroot and w: (no message) (duration: 00m 30s)
  • 00:51 ejegg: updated fundraising dash from bc0ec4bb2427ebeb2d4cd60eda65e3d82d006937 to 2ee15180150c8b685b14abdbf7c6769736fb49a0
  • 00:47 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/extensions/VisualEditor: SWAT (duration: 00m 29s)
  • 00:47 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/resources/src/mediawiki/mediawiki.ForeignStructuredUpload.js: SWAT (duration: 00m 30s)
  • 00:45 ejegg: updated Adyen credentials in SmashPig IPN listener
  • 00:39 ejegg: updated Adyen credentials on payments-wiki
  • 00:35 ejegg: updated payments-wikifrom 7fb9d6599c3833f0a55500ab77abd2b7ffe74cfe to 02a3771cae6e4e74d9d754a8b075de20aa171adc

2015-11-30

  • 22:53 ejegg: updated fundraising dash from 5a6b2dda71e6ce76d7bbba853acae8dc9416052c to 5afc7b02afbd9385db80222271d3d8cc64c1ea13
  • 22:37 ejegg: updated fundraising dashboard from 5a6b2dda71e6ce76d7bbba853acae8dc9416052c to 5afc7b02afbd9385db80222271d3d8cc64c1ea13
  • 22:16 awight: Configuring paymentswiki to not exlicitly include DefaultSettings.php
  • 21:55 mutante: re-wrote l10nupdate cron; restarted cron service on tin
  • 21:53 awight: Tweak paymentswiki-staging config: remove line that exlicitly includes DefaultSettings.php
  • 20:05 apergos: re-enabled puppet on neodymium, minion testing concluded for now
  • 19:47 gwicke: running `nodetool decommission` on restbase1009 in preparation for the conversion to the multi-instance setup, per https://phabricator.wikimedia.org/T95253#
  • 19:31 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: rm deprecated/unused rate limit log config (duration: 00m 28s)
  • 17:27 logmsgbot: demon@tin Synchronized php-1.27.0-wmf.7/extensions/WikimediaMaintenance/: need maint script errywhere (duration: 00m 28s)
  • 16:51 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.7/extensions/ContentTranslation/modules/draft/ext.cx.draft.js: SWAT: Add some extra information to save failure logging gerrit:255956 (duration: 00m 28s)
  • 16:38 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable QuickSurveys reader segmentation survey gerrit:255448 (duration: 00m 28s)
  • 16:30 paravoid: mw1002 service hhvm restart
  • 16:17 paravoid: rolling back to kernel 3.19 on lvs2001/2/3
  • 15:29 paravoid: stopping pybal on lvs2001/2/3
  • 15:21 paravoid: switching lvs2004/5/6 traffic back to lvs2001/2/3
  • 15:13 paravoid: switching lvs2001/2/3 traffic to lvs2004/5/6 and upgrading kernels
  • 15:12 _joe_: restarting HHVM on mw1147 too, same reason as mw1114
  • 15:10 _joe_: restarting hhvm on mw1114, stuck in __pthread_cond_wait () [folly::EventBase::runInEventBaseThreadAndWait ()], apparently blocked in writing to stdout
  • 15:02 paravoid: switching traffic from lvs4002 to lvs4004; upgrading lvs4002's kernel
  • 15:02 paravoid: switching traffic back to lvs4001
  • 14:57 paravoid: switching traffic from lvs4001 to lvs4003; upgrading lvs4001's kernel
  • 14:45 paravoid: switching traffic from lvs3001 to lvs3003; upgrading lvs3001's kernel
  • 14:38 paravoid: switching traffic back to lvs3002
  • 14:31 paravoid: switching traffic from lvs3002 to lvs3004; upgrading lvs3002's kernel
  • 14:07 bblack: upgrading varnishkafka package on all caches
  • 13:52 bblack: updating varnishkafka on cp1065
  • 11:03 godog: upgrade python-statsd to 3.0.1 in eqiad
  • 10:59 godog: upgrade python-statsd to 3.0.1 in codfw
  • 10:15 godog: reenable puppet on graphite1001
  • 10:10 paravoid: re-enabling OSPF over cr2-eqiad:xe-5/2/2 <-> cr1-ulsfo:xe-0/0/3.538
  • 10:09 paravoid: re-enabling cr2-eqiad:xe-5/2/0 and xe-5/2/1
  • 10:01 jynus: performing schema change on db1046 (analytics master)
  • 09:32 jynus: removing old snapshots from db1046
  • 06:39 ori: Restarted statsv on hafnium
  • 02:00 logmsgbot: l10nupdate@tin LocalisationUpdate failed: git pull of core failed
  • 01:56 gwicke: started `nodetool cleanup` on restbase1002 to get rid of unnecessary data from earlier 1001 decommission attempt
  • 01:05 logmsgbot: bd808@tin sync-l10n completed (1.27.0-wmf.7) (duration: 01m 19s)
  • 01:04 bd808: testing l10n cache rebuild as l10nupdate user (take 2)
  • 00:57 Krenair: test

2015-11-29

  • 21:25 jynus: importing user.user_touched (s7) from dbstore1002 to sanitarium. s7 lag on labs replicas will be higher for some minutes.
  • 20:51 jynus: importing user.user_touched (s6) from dbstore1002 to sanitarium. s6 lag on labs replicas will be higher for some minutes.
  • 20:28 jynus: importing user.user_touched (s5) from dbstore1002 to sanitarium. s5 lag on labs replicas will be higher for some minutes.
  • 19:51 jynus: importing user.user_touched (s4) from dbstore1002 to sanitarium. s4 lab will be affected for some minutes.
  • 04:50 gwicke: restarted cassandra on restbase1009 to avoid it running out of disk space; had large compaction (~2TB) at 80% and only 64G disk space left
  • 03:01 YuviPanda: run chown -R l10nupdate: /var/lib/l10nupdate/mediawiki for Reedy on tin
  • 02:28 Reedy: l10nupdate failed because some git objects owned by 997:l10nupdate
  • 02:00 logmsgbot: l10nupdate@tin LocalisationUpdate failed: git pull of core failed

2015-11-28

  • 22:48 logmsgbot: bd808@tin Synchronized php-1.27.0-wmf.5/cache/l10n: bd808 testing l10nupdate sync-dir using stale branch (duration: 01m 29s)
  • 20:49 logmsgbot: l10nupdate@tin LocalisationUpdate failed: Failed to sync-dir 'php-1.27.0-wmf.7/cache/l10n'
  • 20:49 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 07m 11s)
  • 20:35 logmsgbot: ori@tin Synchronized wmf-config/InitialiseSettings.php: Ie33ae3b6a: Increase $wgCopyUploadTimeout to 90 seconds (from default 25) (T118887) (duration: 00m 27s)
  • 02:00 logmsgbot: l10nupdate@tin LocalisationUpdate failed: git pull of core failed

2015-11-27

  • 21:23 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/255726/1 (duration: 00m 31s)
  • 17:46 paravoid: re-enabling cr2-eqiad:xe-5/2/2 (link to cr1-ulsfo) and xe-5/2/3 (link to cr2-codfw)
  • 15:08 godog: kill stray cpu hog xelatex on ocg1003, orphan and started on oct22
  • 14:16 Jeff_Green: fundraising database maintenance
  • 12:36 godog: ban some grafana dashboards from graphite
  • 10:56 apergos: restarted gitblit on antimony about 40 minutes ago, it takes about 20 minutes to load up caches of commits, now responding but... it's taking 17-18 seconds to load repos instead of 1-3 as normal. so timeouts
  • 10:47 jynus: importing user.user_touched (s3) from dbstore1002 to sanitarium, it will affect s3 labsdbs lag
  • 10:44 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repooling previouly problematic servers (duration: 00m 28s)
  • 10:10 apergos: removed gitblit.log.1 on antimony, it was 58GB
  • 10:03 mark: Reenabling cr2-eqiad:xe-5/3/0
  • 09:53 mark: Disabled cr2-eqiad:xe-5/2/*
  • 09:53 paravoid: issue has been identified as a a problematic PIC (5/2) on cr2-eqiad and is being worked around
  • 09:51 mark: Set VRRP priority 50 on all cr2-eqiad VRRP sessions, making it backup
  • 09:50 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depooling problematic servers (duration: 00m 27s)
  • 09:48 mark: Disabling cr2-eqiad:xe-5/2/1
  • 09:45 mark: Disabling cr2-eqiad:xe-5/{2,3}/0
  • 09:42 paravoid: network troubles within eqiad still being investigated
  • 09:42 paravoid: network troubles; ulsfo was depooled ~20 minutes ago due to backhauling issues
  • 08:31 _joe_: rolling restart of all redis instances on rdb1001 and 1003 (later 1002/4) to fix the upstart bug
  • 07:55 _joe_: restarting redis rdb1003:6380 to test rdb disabling
  • 06:57 ori: ran eventloggingctl stop / eventloggingctl start on eventlog1001
  • 06:18 ori: coal metrics flat since 01:25:00 AM UTC
  • 02:47 Reedy: manually running dsh -g mediawiki-installation -M -F 40 -- "sudo -u mwdeploy /srv/deployment/scap/scap/bin/scap-rebuild-cdbs"
  • 02:45 logmsgbot: l10nupdate@tin LocalisationUpdate failed: Failed to sync-dir 'php-1.27.0-wmf.7/cache/l10n'
  • 02:45 logmsgbot: reedy@tin Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 09m 10s)
  • 02:14 Reedy: killed pre 1.27 l10n cache files from tin:/var/lib/l10nupdate/caches
  • 02:00 logmsgbot: l10nupdate@tin LocalisationUpdate failed: git pull of core failed

2015-11-26

  • 20:59 logmsgbot: aaron@tin Synchronized wmf-config/jobqueue-eqiad.php: Adjust connection timeout and maxPartitionsTry (duration: 00m 34s)
  • 16:50 jynus: backfilling user.user_touched from dbstore1002 to sanitarium on all wikis
  • 16:30 paravoid: switch lvs3001 traffic to lvs3003dd
  • 16:22 paravoid: rebooting lvs3003 with Linux 4.3-1~wmf1
  • 15:58 Krinkle: Deployed patch for T119707
  • 15:49 godog: reenable puppet on graphite1001
  • 14:30 _joe_: run /usr/local/bin/foreachwiki updateSpecialPages.php manually in a screen on terbium
  • 13:27 moritzm: restarted opendj on nembus for export of LDAP data
  • 13:25 moritzm: stopping opendj on nembus for export of LDAP data
  • 11:54 apergos: disabling puppet on neodymium for salt minon tracing
  • 11:35 apergos: re-enabled puppet on neodymium, config testing done for now
  • 11:32 godog: drop local_group_globaldomain_T_mathoid_data and local_group_globaldomain_T_mathoid_request CF stats from graphite
  • 11:25 jynus: manually executing some updateSpecial pages script to try to debug its problems
  • 11:16 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1053 after maintenance (duration: 00m 28s)
  • 11:13 apergos: disabled puppet on neodymium for salt master testing
  • 09:49 moritzm: restarted cassandra nodes on the restbase staging cluster to pick up openjdk security updates
  • 09:38 jynus_: upgrading/rebooting/restarting mysql on db1053 (depooled)
  • 09:02 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1053 for maintenance (duration: 00m 27s)
  • 07:15 ori: rdb100x migrated to redis::instance, meaning they each have two additional redis instances, on port 6380 and 6381, accordingly.
  • 06:46 logmsgbot: ori@tin Synchronized wmf-config/jobqueue-eqiad.php: I3e6021ed3: Update job queue config to use 2 new rdb1003 instances (duration: 00m 28s)
  • 02:00 logmsgbot: l10nupdate@tin LocalisationUpdate failed: git pull of core failed
  • 01:31 awight: updating paymentswiki-staging from 7fb9d6599c3833f0a55500ab77abd2b7ffe74cfe to 988ba71418c370e0365a102a0b87f823671ea1e3
  • 01:03 awight: from f867beecd3a4fb596e2d08c8d2b0e4173ad0b356 to d5afa10383cfd41adbe798c3ed2aa97839be8bc2
  • 00:14 logmsgbot: ori@tin Synchronized wmf-config/jobqueue-eqiad.php: I1e0d0ed8215f: pool rdb1007:6381 (duration: 00m 30s)

2015-11-25

  • 23:21 logmsgbot: krenair@tin Synchronized private/README_BEFORE_MODIFYING_ANYTHING: 334ca105e92aaf7046e244ff39189f3823d31a7d (duration: 00m 32s)
  • 22:14 logmsgbot: demon@tin Finished scap: new MW release, swapping extdist config + msgs (duration: 23m 59s)
  • 21:50 logmsgbot: demon@tin Started scap: new MW release, swapping extdist config + msgs
  • 20:31 gwicke: running `nodetool cleanup` on restbase1007 to make sure that we don't have extra sstables from the 1001 decommision taking up space
  • 19:13 mobrovac: restbase deploy end of 74662c
  • 19:13 moritzm: installed django security updates on stat* and graphite hosts
  • 18:47 robh: removing cr2-ulsfo:xe-1/2/0, Patch ID 1062 as T118171 cancels that link
  • 18:46 mobrovac: restbase deploy start of 74662c
  • 18:19 logmsgbot: bd808@tin Synchronized README: Testing l10nupdate uid fix for T119165 (duration: 00m 28s)
  • 18:04 andrewbogott: restored pdns-recursor on holmium, again
  • 18:00 mobrovac: restbase canary deploy to restbase1001 of 74662c6
  • 17:44 andrewbogott: killing pdns-recursor on holmium
  • 16:53 andrewbogott: killing pdns-recursor on holmium
  • 16:16 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1044 after maintenance (duration: 00m 47s)
  • 15:53 godog: ban grafana kafka dashboard temporarily from graphite
  • 14:57 godog: bounce uwsgi on graphite1001
  • 14:52 godog: stop puppet on restbase1* / restbase2* before https://gerrit.wikimedia.org/r/#/c/254372/
  • 13:09 logmsgbot: hashar@tin Synchronized php-1.27.0-wmf.7/Rakefile: Added Rakefile https://gerrit.wikimedia.org/r/#/c/254423/ (duration: 00m 28s)
  • 11:19 jynus: applying ferm and p_s to db1044 (depooled)
  • 10:57 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1044 for a brief maintenance (duration: 00m 28s)
  • 08:08 logmsgbot: ori@tin Synchronized wmf-config/jobqueue-eqiad.php: I9bea66df: Update job queue config to use 2 rdb1007 instances (duration: 00m 28s)
  • 08:06 akosiaris: update puppet compilers fact database
  • 03:44 Tim: on palladium ran puppet post-merge hook manually after puppet-merge failed with "error: Ref refs/remotes/origin/production is at 380706c5bd2f47d140ffa183dd22fe920927af7e but expected 897486120d645f1c30dcf68bfb3198e2f39ec639"
  • 03:35 logmsgbot: krinkle@tin Synchronized portals/prod/wikipedia.org/index.html: Fix Uncaught TypeError: innerHTML() is not a function (duration: 00m 28s)
  • 03:12 logmsgbot: krinkle@tin Synchronized wmf-config/CommonSettings.php: I4e21eda0f3 (duration: 01m 06s)
  • 02:40 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 06m 50s)
  • 02:00 logmsgbot: l10nupdate@tin LocalisationUpdate failed: git pull of core failed
  • 01:31 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.7/StartProfiler.php: create symlink (duration: 01m 10s)
  • 01:03 mutante: re-armed keyholders on mira
  • 00:46 mutante: starting redis on rdb1007 (it died and was status: dead)
  • 00:40 mutante: tin: fixing l10nupdate UID (997->10002), file ownership
  • 00:37 ori: Migrating rdb1007 from redis::legacy to redis::instance; will involve a service restart.
  • 00:21 ejegg: updated payments-wiki from 5358960d6857f589939c9b86ea37af168a4f8b0b to 7fb9d6599c3833f0a55500ab77abd2b7ffe74cfe
  • 00:20 mutante: rebooting mira
  • 00:14 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/255276/ (duration: 00m 28s)
  • 00:13 awight: Disabled Amazon on donatewiki
  • 00:09 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/254443/ (duration: 00m 28s)

2015-11-24

  • 23:43 mutante: fixinx l10nupdate UID / file ownership on mira
  • 23:40 csteipp: deployed patch for T119309
  • 23:36 ostriches: phabricator: restarted daemons, were complaining about out of sync config
  • 22:07 ejegg: updated SmashPig from 6243d4f172dbfa5e3820f2e1e3a3165efff58bac to b3ee85aab38fdeef3f045ba1733f582e9a5006b2
  • 21:39 logmsgbot: krinkle@tin Synchronized wmf-config/CommonSettings.php: I80d38773057 (duration: 00m 28s)
  • 20:31 mutante: gnt-instance reboot seaborgium.wikimedia.org
  • 20:01 bblack: disabling puppet on neon to avoid monitoring race-condition spam
  • 19:30 moritzm: restarted hadoop workers on analytics1028 to analytics1055 to pick up openjdk security update (plus updates for libpng, nss, nspr, pixbuf and libxml as used by openjdk)
  • 19:27 bblack: misc-cluster: all 4x nodes on new config and pooled
  • 19:18 bblack: misc-cluster: cp1056+7 pooled (new config), cp1069+70 depooled (old config)
  • 18:52 bblack: repooled cp1056, depooled cp1057 (misc 2layer)
  • 18:38 YuviPanda: restarted apache2 on etherpad
  • 18:30 bblack: depooling cp1056 for misc-cluster 2layer work...
  • 17:59 godog: nodetool cleanup on restbase1007
  • 17:20 godog: stop decommissioning restbase1001, bounce cassandra
  • 16:40 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.7/resources/lib/oojs-ui/oojs-ui.js: SWAT: OOjs UI: Backport 4fbbc737c86b500c11bbb471ec1001c50ab8853c gerrit:255032 (duration: 00m 28s)
  • 16:23 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable es labs replica writes for nlwiki, frwiki and eswiki gerrit:255122 (duration: 00m 27s)
  • 16:16 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Fix article-recommender-1 campaign gerrit:255069 (duration: 00m 27s)
  • 16:10 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Third QuickSurveys external survey gerrit:254908 (duration: 00m 30s)
  • 16:08 paravoid: starting pybal on lvs3001 again, testing is over
  • 16:06 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor for all new accounts on eswiki gerrit:250473 (duration: 00m 46s)
  • 15:38 moritzm: restarted analytics1030 to pick up openjdk security update (plus updates for libpng, nss, nspr, pixbuf and libxml as used by openjdk)
  • 15:33 hashar: enabled puppet on both gallium and scandium
  • 15:27 hashar: disabling puppet on both gallium and scandium
  • 14:24 godog: upgrade diamond to 3.5-4 on precise hosts in eqiad
  • 14:22 godog: upgrade diamond to 3.5-4 on precise hosts in ulsfo
  • 13:56 godog: upgrade diamond to 3.5-4 on precise hosts in esams
  • 12:18 paravoid: stopping pybal on lvs3001, switching its traffic to lvs3003
  • 11:48 paravoid: rebooting lvs3003 with linux 4.2.6-1~bpo8+wmf1
  • 10:28 jynus: dropping user_daily_contribs view from labsdb hosts
  • 10:07 godog: stop ircecho on neon temporarily while https://gerrit.wikimedia.org/r/#/c/255094/ applies
  • 08:24 jynus: dbstore1002 (analytics-slave) mysql is not responding, forcing restart
  • 05:58 _joe_: powercycling ms-be1010, xfs kernel lockups
  • 02:26 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 05m 34s)
  • 02:12 ejegg: updated SmashPig from 55c2dbc8cd36cfb89d7e17b75b840ca2e023653e to 6243d4f172dbfa5e3820f2e1e3a3165efff58bac
  • 01:00 ejegg: updated payments-wiki from ecbc49c63b259a4363c51491ddff273e3579f0e9 to 5358960d6857f589939c9b86ea37af168a4f8b0b
  • 00:36 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.7/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: https://gerrit.wikimedia.org/r/#/c/255053/ (duration: 00m 28s)
  • 00:26 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Enable commonswiki for elasticsearch labs replica writes (duration: 00m 28s)
  • 00:15 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Enable wikidatawiki for elasticsearch labs replica (duration: 00m 27s)

2015-11-23

  • 23:55 logmsgbot: maxsem@tin Synchronized portals: (no message) (duration: 00m 28s)
  • 23:30 awight: campaigns reenabled
  • 23:29 awight: reenabling paymentswiki
  • 23:28 awight: reenabling fundraising-jenkins jobs
  • 23:08 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Enable elasticsearch labs replica for enwiki and dewiki (duration: 00m 28s)
  • 23:04 awight: shutting down Jenkins
  • 22:52 awight: disable paymentswiki in config
  • 21:16 hoo: Updated Wikidata's property suggester with data from today's json dump
  • 21:15 jynus: enforcing limits on running queries on all labsdb databases
  • 20:47 jynus: restarting labsdb1002, hopfully for the last time
  • 17:54 jynus: temporarily stopping labsdb1002 to solve replication issues
  • 17:32 Coren: emergency hotfix on labstore1001 (apt-get remove nfsd-ldap) as it appears to not work in all situations.
  • 17:30 ori: restarting hhvm on mw1122 after lock-up; backtrace saved as /tmp/hhvm.32016.bt.
  • 17:28 logmsgbot: thcipriani@tin Finished scap: SWAT: Update CentralNotice gerrit:254865 (duration: 26m 31s)
  • 17:02 logmsgbot: thcipriani@tin Started scap: SWAT: Update CentralNotice gerrit:254865
  • 16:44 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: noindex user namespace on en.wikipedia.org gerrit:237330 (duration: 00m 28s)
  • 16:43 jynus: restarting mysqld on labsdb1003 again
  • 16:38 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: End first mobile QuickSurveys campaign gerrit:254831 (duration: 00m 28s)
  • 16:32 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.7/includes/specials/SpecialWatchlist.php: SWAT: Special:Watchlist: Add user preference to "Show last" options, fix float comparison gerrit:254880 (duration: 00m 29s)
  • 16:10 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Enable article-recommender-1 campaign as default gerrit:254117 (duration: 00m 52s)
  • 15:56 paravoid: rebooted lvs3003 back with (an updated) 3.19
  • 15:53 _joe_: live-resizing /var/lib/wdqs on wdqs1002
  • 15:15 Coren: NFS server on labstore1001 restarted with no issues.
  • 15:11 Coren: Restarting NFS daemon on labstore1001 w/ the new LDAP shim.
  • 14:53 paravoid: powercycling cp1053, unresponsive, blank console
  • 14:53 bblack: depooling cp1053
  • 14:50 ottomata: removing erbium udp2log varnishncsa instances from all caches
  • 14:38 paravoid: rebooting lvs3003 with new kernel
  • 13:18 jynus: restarting mysql on labsdb1003, unresponsive
  • 11:05 _joe_: stopping apache on mw2051 for testing
  • 08:58 godog: restbase1001 nodetool decommission
  • 08:57 mobrovac: mathoid deploying ff03d56
  • 08:12 moritzm: restarted gitblit on antimony
  • 02:19 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 05m 57s)
  • 01:27 Jamesofur: end enwiki test election from cli to avoid confusion with real election

2015-11-22

  • 21:30 Reedy: restarting stuck Jenkins
  • 02:16 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 05m 22s)

2015-11-21

  • 19:14 ejegg|away: updated fundraising tools from d7253eb1c6c9478ce2cf8760279d2ad1b7f0a132 to ebed29c0eccf38c812b20a957b3487a15bfa9cbc
  • 18:43 ejegg|away: rolled back SmashPig d7534bde8c247a1e877a4bcd768b9963f89666d7 --> 55c2dbc8cd36cfb89d7e17b75b840ca2e023653e
  • 18:35 ejegg|away: updated SmashPig from 55c2dbc8cd36cfb89d7e17b75b840ca2e023653e to d7534bde8c247a1e877a4bcd768b9963f89666d7
  • 03:19 ori: SecurePoll arbcomlist.php run for enwiki (requested by Jamesofur) complete.
  • 02:20 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 05m 18s)
  • 00:50 ori: update-ubuntu-mirror on carbon completed successfully ; going to re-enable icinga-wm
  • 00:46 ori: rampant puppet failures appear to be the result of a Hash Sum mismatch error during apt-get update; attempting to fix by running sudo -u mirror /usr/local/sbin/update-ubuntu-mirror on carbon.
  • 00:37 ori: disabled puppet on neon for one hour and killed icinga-wm to silence a tide of puppet failure alerts
  • 00:24 logmsgbot: demon@tin Synchronized .arcconfig: no-op, for completeness (duration: 00m 31s)

2015-11-20

  • 21:45 logmsgbot: csteipp@tin Synchronized php-1.27.0-wmf.7/includes/User.php: Deploy fix for T119021 (duration: 00m 28s)
  • 18:55 ori: resuming interrupted arbcomlist invocation on terbium
  • 16:51 _joe_: decommissioning mw1041: removed facts and certs, removed from conftool and pybal
  • 16:44 logmsgbot: reedy@tin Synchronized wmf-config/throttle.php: Add throttle config for Viquimarató nit digital/2015 (duration: 00m 28s)
  • 16:17 _joe_: depooled mw1041
  • 15:41 Reedy: running sync-common on mw1041
  • 15:32 _joe_: powercycling mw1041
  • 15:26 logmsgbot: reedy@tin Synchronized wmf-config/throttle.php: Update throttle for event tomorrow (duration: 00m 28s)
  • 14:35 godog: swift eqiad-prod: set ms-be1019 / ms-be1020 / ms-be1021 weight 3000
  • 10:06 ori: Running SecurePoll's arbcomlist.php on terbium with performance fix from https://gerrit.wikimedia.org/r/#/c/254375/ , per request from Jamesofur. Looks to be behaving well, can be killed if it acts up.
  • 07:44 _joe_: rebooted pybal-test2001
  • 07:36 _joe_: powercycled mw1041
  • 04:55 logmsgbot: twentyafterfour@tin Synchronized multiversion/checkoutMediaWiki.php: inconsequential sync to test mirroring (duration: 02m 14s)
  • 04:41 logmsgbot: twentyafterfour@tin Synchronized multiversion/checkoutMediaWiki.php: deploying I2ec91aa33e05c2dc367db8a6f6ba56be5397906b to test scap mirroring (duration: 00m 20s)
  • 04:36 twentyafterfour: deploying scap on tin
  • 02:34 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 10m 47s)
  • 01:31 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/resources/src/mediawiki.widgets/mw.widgets.TitleWidget.js: Fix conflicting configuration name in TitleInputWidget (duration: 00m 19s)
  • 01:31 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/includes/widget/TitleInputWidget.php: Fix conflicting configuration name in TitleInputWidget (duration: 00m 20s)
  • 01:30 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/resources/lib/jquery.i18n/src/jquery.i18n.language.js: Unbreak IE8 support in jquery.i18n (duration: 00m 25s)
  • 01:29 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/extensions/Echo/includes/formatters/BasicFormatter.php: unstub wgLang to fix type hint errors (duration: 00m 20s)
  • 01:11 logmsgbot: bd808@tin Synchronized wmf-config/InitialiseSettings.php: Reduce sampling rate for QuickSurveys survey (28c69d2) (duration: 00m 20s)
  • 00:37 awight: Updating fundraising/crm from e486ba424092866cdd8a15d26aeb17a412cd631a to f867beecd3a4fb596e2d08c8d2b0e4173ad0b356
  • 00:18 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-common.php: SWAT https://gerrit.wikimedia.org/r/253933 and https://gerrit.wikimedia.org/r/254070 (duration: 00m 19s)
  • 00:17 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: SWAT https://gerrit.wikimedia.org/r/253933 and https://gerrit.wikimedia.org/r/254070 (duration: 00m 19s)
  • 00:13 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.7/extensions/CirrusSearch/: SWAT https://gerrit.wikimedia.org/r/254320 and https://gerrit.wikimedia.org/r/254319 (duration: 00m 20s)
  • 00:02 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: SWAT https://gerrit.wikimedia.org/r/#/c/254322/ (duration: 00m 20s)

2015-11-19

  • 23:35 awight: Update fundraising/tools from c6315eddf6f495e45401844d5f61bb240464c1e5 to d7253eb1c6c9478ce2cf8760279d2ad1b7f0a132
  • 22:31 csteipp: deployed followup for T109724
  • 22:21 csteipp: deployed patch for T118682
  • 22:13 csteipp: deployed patch for T118032
  • 22:11 csteipp: deployed patch for T97897
  • 21:31 mutante: magnesium - apt-get upgrade
  • 21:13 mutante: racktables moved to krypton and upgraded to 0.20.6
  • 20:28 mutante: racktables - in migration - brb
  • 20:22 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.7/extensions/QuickSurveys/: Deploy https://gerrit.wikimedia.org/r/#/c/254184/ (duration: 00m 19s)
  • 20:19 awight: push paymentswiki config change, T115746
  • 20:13 bblack: upgrading pybal to 1.13.1 on lvs400[12] and failing back traffic
  • 19:51 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1044 after maintenance (duration: 00m 22s)
  • 19:24 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: wikipedia wikis to 1.27.0-wmf.7
  • 18:53 logmsgbot: aude@tin Synchronized php-1.27.0-wmf.7/extensions/Wikidata: Adjust watchlist filter for core change (duration: 00m 30s)
  • 16:24 godog: repool restbase1002
  • 16:19 cmjohnson1: replacing pem 0 cr1-eqiad
  • 14:55 godog: roll restart restbase in eqiad
  • 14:41 godog: roll-restart restbase on aqs100*
  • 14:40 moritzm: uploaded linux-meta 1.3 to carbon
  • 14:39 moritzm: uploaded linux 3.19.3-9 to carbon
  • 14:14 godog: roll-restart restbase in codfw
  • 12:39 bblack: failing over ulsfo traffic to lvs400[34] (for 1.13.1 testing)
  • 11:23 hashar: restarted zuul-merger on gallium. We now have two instances running (gallium and scandium)
  • 11:21 hashar: stopped zuul-merger on gallium, started the one on scandium
  • 11:00 jynus: modifying filters on all labs db replicas (sanitarium), small amounts of lag could be created
  • 10:42 godog: hhvm-dump-debug --full and bounce hhvm on mw1147
  • 10:40 godog: hhvm-dump-debug --full and bounce hhvm on mw1131
  • 10:13 godog: disable puppet on deployment_target:restbase/deploy
  • 10:08 hashar: going to restart Jenkins a few times
  • 09:53 godog: delete old mobileapps CF stats from graphite1001 / graphite2001
  • 06:19 YuviPanda: remove role::labs::instance puppetClass from all labs instances
  • 02:23 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.6/cache/l10n: l10nupdate for 1.27.0-wmf.6 (duration: 06m 58s)
  • 01:31 awight: updating fundraising/crm from 67b855cbce358d215b8025da6fe7031bf3d221d9 to e486ba424092866cdd8a15d26aeb17a412cd631a
  • 01:30 twentyafterfour: finished phabricator upgrade
  • 00:29 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/extensions/Flow/: SWAT: better permissions errors for opt-in (duration: 00m 21s)
  • 00:19 RoanKattouw: Ran namespaceDupes.php on eswiktionary
  • 00:13 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: T118844, T119006 (duration: 00m 19s)
  • 00:12 logmsgbot: catrope@tin Synchronized portals: SWAT (duration: 00m 19s)

2015-11-18

  • 23:51 yurik: git deploy sync latest graphoid
  • 23:49 bblack: upgrading pybal to 1.13.1 on lvs300[34]
  • 23:45 bblack: upgrading pybal to 1.13.1 on lvs400[34]
  • 22:44 awight: Updating fundraising/crm from 230cebb4a3b46a75fef758d46f4db22de29286cb to 67b855cbce358d215b8025da6fe7031bf3d221d9
  • 22:38 mutante: powercycled analytics1030 - no ssh response, mgmt console login timed out
  • 21:15 subbu: finished deploying parsoid sha e0a4fc91
  • 21:10 subbu: synced code; restarted parsoid on wtp1003 as a canary
  • 21:06 subbu: starting parsoid deploy
  • 20:43 bblack: upgrading pybal to 1.13.1 on lvs200[456]
  • 20:09 logmsgbot: maxsem reset User:人神之间 email (confirmed in person with Matanya)
  • 19:59 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.7
  • 18:18 andrewbogott: repooled mw1160. And that’s all of them!
  • 18:09 andrewbogott: repooled mw1159, depooled mw1160
  • 17:59 andrewbogott: repooling mw1157, depooled mw1159
  • 17:48 andrewbogott: repooled mw1155, depooled 1157 (I did 56 and 58 list night)
  • 17:36 mutante: releases.wikimedia.org now enforcing https protocol
  • 17:30 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.6/includes/db/loadbalancer/LoadBalancer.php: I03e1386b: Make getLaggedSlaveMode() use reuseConnection() as needed (T118162) (duration: 00m 19s)
  • 17:26 andrewbogott: repooled mw1154, depooled mw1155
  • 17:17 andrewbogott: repooled mw1153, depooling mw1154
  • 17:09 andrewbogott: depooled mw1153 for https://phabricator.wikimedia.org/T118888
  • 17:07 thcipriani: updateCollation.php for srwiki 1670608 rows processed
  • 17:05 godog: restart restbase2002-a cassandra instance, bootstrap failed with stream error
  • 16:23 bblack: upgrading pybal on lvs1006 (1.13.1)
  • 16:23 _joe_: upgrading pybal on lvs1006 (1.13.1)
  • 16:21 bblack: upgrading pybal on lvs1005 (1.13.1)
  • 16:21 bblack: upgrading pybal on lvs1004 (1.13.1)
  • 16:16 thcipriani: updateCollation.php for srwiktionary complete 82713 rows processed
  • 16:12 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wgCategoryCollation to "uca-sr" at srwiki and srwiktionary gerrit:253903 (duration: 00m 20s)
  • 16:11 bblack: upgrading pybal on lvs1004 (1.13.1)
  • 16:07 mobrovac: restbase cassandra dropping old mobileapps keyspaces
  • 16:05 mobrovac: restbase end deploy of 6b5a602
  • 15:56 mobrovac: restbase start deploy of 6b5a602
  • 15:50 bblack: upgrading pybal to 1.13 on lvs100[56] (with service clean)
  • 15:37 jynus: warming up srwiki.page and srwiki.categorylinks on s3 slaves
  • 15:29 mobrovac: restbase canary deploy to rb1001 of 6b5a602
  • 15:28 bblack: upgrading pybal to 1.13 on lvs1004, with cleared tables
  • 15:23 _joe_: uploaded pybal 1.13 to jessie-wikimedia
  • 15:14 hashar: stopping zuul-merger on scandium. Lacks ssh private key to reach gerrit
  • 14:58 hashar: Pooled zuul-merger instance on scandium.eqiad.wmnet . In case it screw up one has to stop the zuul-merger service on scandium.
  • 14:57 godog: bounce mobileapps on scb1001/scb1002
  • 14:47 mobrovac: mobileapps deploying 151c312
  • 13:55 bblack: disabling pybal on lvs100[123] - fallback to lvs100[456]
  • 12:00 hashar: reenabled puppet on gallium. Havent pooled zuul-merger yet
  • 11:57 hashar: disabled puppet on gallium
  • 11:56 hashar: pooling new zuul-merger on scandium
  • 11:44 mark: Set OSPF/OSPFv3 metric 320 (instead of 360) on new Zayo link between cr2-eqiad:xe-5/2/3 and cr2-codfw:xe-5/0/1
  • 10:49 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Decreasing weight of db1035 due to connection exhaustion (duration: 00m 19s)
  • 10:31 hashar: gallium: apt-get upgrade
  • 10:30 hashar: upgrading Zuul on gallium (CI interruption for a minute)
  • 09:48 jynus: several db maintenance tasks on db1044 -optimization, upgrade- expect lag (node is depooled)
  • 09:25 godog: reimage restbase2002
  • 09:11 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1027, depool db1044 for maintenance (duration: 01m 08s)
  • 09:09 _joe_: restarted HHVM on mw1095, stuck in treadmill::startrequest() on a contended lock
  • 06:31 _joe_: rsyncing terbium homes from rutherfordium
  • 02:39 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.6/cache/l10n: l10nupdate for 1.27.0-wmf.6 (duration: 10m 10s)
  • 01:46 andrewbogott: repooling mw1156
  • 01:43 yurik_: synced graphoid service
  • 01:40 andrewbogott: updated kernel on mw1156, now rebooting.
  • 01:35 andrewbogott: repooling mw1158, depooling mw1156
  • 01:22 andrewbogott: rebooting mw1158
  • 01:19 andrewbogott: upgrading mw1158 to 3.13.0-62
  • 00:26 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings.php: wgRCWatchCategoryMembership true on group0 wikis (duration: 00m 27s)
  • 00:04 logmsgbot: maxsem@tin Synchronized portals/: SWAT (duration: 00m 32s)

2015-11-17

  • 23:43 mutante: deactivated wekipedia.com
  • 23:40 YuviPanda: remove realm variable from all instances in ldap
  • 23:18 mutante: deactivated wikiartpedia.[biz|co|info|me|mobi|net|org]
  • 23:17 awight: updated paymentswiki from 4baa5a66a4510414d6b43b59f1b1cda2341c17fd to ecbc49c63b259a4363c51491ddff273e3579f0e9
  • 23:13 mutante: deactivated border-wikipedia.de
  • 23:01 mutante: deactivated wikimedia.xyz
  • 22:57 mutante: deactivated visualwikipedia.[com|net]
  • 22:56 mutante: deactivated softwarewikipedia.[com|net|org]
  • 22:54 mutante: deactivated indiawikipedia.com
  • 22:52 mutante: deactivated vikipedi.com.tr, vikipedia.com.tr
  • 22:38 awight: updated fundraising/tools from 182f217ea20a496fe0f9737447d04ed9a3bccde8 to c6315eddf6f495e45401844d5f61bb240464c1e5
  • 22:23 mutante: deactivated wikimaps.com, wikimaps.net, wikimaps.org
  • 22:20 mutante: deactivated wikimedia.biz, webhostingwikipedia.com
  • 22:18 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.7
  • 21:07 logmsgbot: twentyafterfour@tin Finished scap: sync new branch 1.27.0-wmf.7 and enable for testwiki (duration: 29m 40s)
  • 20:39 mutante: deleted sitemap.wikimedia.org (T101486)
  • 20:38 logmsgbot: twentyafterfour@tin Started scap: sync new branch 1.27.0-wmf.7 and enable for testwiki
  • 20:31 logmsgbot: twentyafterfour@tin scap failed: CalledProcessError Command 'cp -r "/tmp/scap_l10n_2994655917"/* "/srv/mediawiki-staging/php-1.27.0-wmf.7/cache/l10n"' returned non-zero exit status 1 (duration: 00m 42s)
  • 20:30 logmsgbot: twentyafterfour@tin Started scap: sync new branch 1.27.0-wmf.7 and enable for testwiki
  • 20:26 logmsgbot: twentyafterfour@tin scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_4014066321" --threads=4 --lang en --quiet' returned non-zero exit status 1 (duration: 01m 30s)
  • 20:25 logmsgbot: twentyafterfour@tin Started scap: sync new branch 1.27.0-wmf.7 and enable for testwiki
  • 18:54 andrewbogott: depooled mw1158 due to kernel errors. https://phabricator.wikimedia.org/T118888
  • 18:10 cmjohnson1: swapping failing disk 32:8 on db1027..will cause icinga alert (jynus)
  • 18:02 bblack: upgrading and re-enabling pybal on lvs300[12].esams.wmnet (1.10 -> 1.12)
  • 17:54 yurik: deployed latest graphoid service
  • 16:45 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wgCategoryCollation for bs.wikipedia.org gerrit:250862 (duration: 00m 26s)
  • 16:42 bblack: upgrading / re-enabling pybal on lvs200[123].codfw.wmnet (1.10 -> 1.12)
  • 16:30 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Set import sources on en.wikiversity gerrit:250709 (duration: 00m 27s)
  • 16:25 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Namespace configuration on ru.wikipedia.org gerrit:252911 (duration: 00m 26s)
  • 16:19 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: SWAT: Editaton contra la violencia hacia las mujeres throttle rule gerrit:253541 (duration: 00m 27s)
  • 16:10 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor for 50% of new accounts on eswiki gerrit:250472 (duration: 00m 26s)
  • 16:02 hashar: Zuul-merger deployment aborted / uncomplte (I hate puppet)
  • 16:02 bblack: re-enabling pybal on lvs400[12] (upgraded to 1.12)
  • 15:36 cmjohnson1: reseating pem 0 on cr1-eqiad
  • 15:30 mobrovac: restbase end deploy of e749f6ff
  • 15:20 mobrovac: restbase start deploy of e749f6ff
  • 14:59 bblack: stopping pybal on lvs300[12], fallback to lvs300[34] (pybal 1.10 vs 1.12)
  • 14:55 godog: swift eqiad-prod: set ms-be1019 / ms-be1020 / ms-be1021 weight 1500
  • 14:51 bblack: stopping pybal on lvs200[123], fallback to lvs200[456] (pybal 1.10 vs 1.12)
  • 14:37 bblack: stopping pybal on lvs400[12], fallback to lvs400[34] (pybal 1.10 vs 1.12)
  • 12:32 akosiaris: depool restbase1002
  • 09:54 godog: nodetool decommission on restbase2002
  • 06:18 gwicke: restbase cassandra: Increased tombstone_threshold from 0.02 to 0.1 for wikipedia and wikimedia html and data-parsoid to reduce single-sstable compaction rate.
  • 02:21 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.6/cache/l10n: l10nupdate for 1.27.0-wmf.6 (duration: 06m 26s)
  • 00:21 logmsgbot: krinkle@tin Synchronized wmf-config/jobqueue-codfw.php: (no message) (duration: 00m 26s)
  • 00:16 logmsgbot: legoktm@tin Synchronized wmf-config/CommonSettings.php: REL1_26 knocks on the door - https://gerrit.wikimedia.org/r/#/c/252399/ (duration: 00m 27s)

2015-11-16

  • 22:47 logmsgbot: hoo@tin Synchronized php-1.27.0-wmf.6/includes/Revision.php: Don't claim model validation failed if the content couldn't be loaded (duration: 00m 26s)
  • 22:45 gwicke: rolling restbase restart to apply https://gerrit.wikimedia.org/r/#/c/253468/
  • 21:52 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: Don't log sysops with short passwords (duration: 00m 27s)
  • 21:47 mobrovac: restbase end deployment of 54bc0517
  • 21:38 mobrovac: restbase start deployment of 54bc0517
  • 21:26 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: Log username of users with short passwords (duration: 00m 26s)
  • 21:23 subbu: finished deploying parsoid 3a6f3b9e
  • 21:21 mobrovac: restbase canary deploy to restbase1001 of 54bc0517
  • 21:07 subbu: synced fresh code; restarting parsoid on wtp1005 as a canary
  • 21:06 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: Log privileged users with short passwords. Set minimum password length of 10 for wikitech. (duration: 00m 26s)
  • 21:03 subbu: starting parsoid deploy
  • 19:52 anomie: Running cleanupBlocks.php on arwiki, eswikibooks, eswikinews, and testwiki for phab:T118625
  • 19:23 RoanKattouw: Deployed patch for T116095
  • 16:14 logmsgbot: thcipriani@tin Synchronized wmf-config/squid.php: SWAT: wgHTCPRouting: use separate address for upload gerrit:249121 (duration: 00m 26s)
  • 16:07 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable first QuickSurveys survey gerrit:253038 (duration: 00m 26s)
  • 14:40 godog: carbon daemon bounce on graphite1001
  • 13:43 bblack: reinstalling lvs100[45]
  • 13:39 jynus: dropping ipblocks_old table from all wikis
  • 13:19 bblack: reinstalling lvs1006
  • 11:21 jynus: defragementing, upgrading, general maintenance to mysql at db1027. Expect lag, etc. (it is depooled)
  • 10:25 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1015, depool db1027 for maintenance (duration: 00m 49s)
  • 10:03 jynus: performing schema change on wikishared.bounce_records (x1)
  • 09:39 godog: bootstrap restbase2001-c cassandra instance
  • 09:34 jynus: adding missing tables to vewikimedia https://phabricator.wikimedia.org/rMW7389d7c69035c968553fbd2eaf1cf47038c4577c
  • 09:19 _joe_: reimaging terbium
  • 09:16 godog: swift eqiad-prod: add ms-be1019 / ms-be1020 / ms-be1021 weight 500
  • 08:56 _joe_: rsyncing files from terbium to rutherfordium
  • 08:10 akosiaris: powered off plutonium, removed it from puppet and salt. context: https://phabricator.wikimedia.org/T118586
  • 02:22 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.6/cache/l10n: l10nupdate for 1.27.0-wmf.6 (duration: 06m 29s)

2015-11-15

  • 18:03 jynus: disabling puppet, restarting mysql on es2008, es2009, es2010 for configuration check/upgrade
  • 16:41 jynus: stopping eventlogging_sync for manual resync on dbstore1002 and db1047
  • 15:33 logmsgbot: reedy@tin Synchronized wmf-config/: noop after chmod (duration: 00m 27s)
  • 14:31 logmsgbot: reedy@tin Synchronized wmf-config/InitialiseSettings.php: revert last, no file created on fluorine (duration: 00m 27s)
  • 14:28 logmsgbot: reedy@tin Synchronized wmf-config/InitialiseSettings.php: Temporarily enable error log. Will revert in a few minutes (duration: 00m 27s)
  • 14:19 logmsgbot: reedy@tin Synchronized php-1.27.0-wmf.6/extensions/BounceHandler/: Pull in a few fixes and improvements for BounceHandler (duration: 01m 12s)
  • 12:17 jynus: restarting mysql at db1015 to finish upgrade/maintenance process
  • 12:10 jynus: uploading upgraded mariadb package for trusty, too
  • 09:02 _joe_: rebooting krypton
  • 02:21 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.6/cache/l10n: l10nupdate for 1.27.0-wmf.6 (duration: 06m 07s)
  • 01:39 logmsgbot: reedy@tin Purged l10n cache for 1.27.0-wmf.1
  • 01:36 ostriches: deploying scap master@54d93b3

2015-11-14

  • 23:31 Reedy: dsh delete php-1.26wmf22 23 and 24
  • 23:06 Reedy: dsh delete php-1.27.0-wmf.4/cache/l10n/*
  • 23:06 Reedy: dsh delete php-1.27.0-wmf.3/cache/l10n/*
  • 23:04 Reedy: dsh delete php-1.27.0-wmf.2/cache/l10n/*
  • 23:04 Reedy: dsh delete php-1.27.0-wmf.1/cache/l10n/*
  • 23:03 Reedy: dsh delete php-1.26wmf24/cache/l10n/*
  • 23:01 Reedy: dsh delete php-1.26wmf23/cache/l10n/*
  • 23:00 Reedy: dsh delete php-1.26wmf22/cache/l10n/*
  • 22:58 Reedy: dsh delete php-1.26wmf21
  • 22:57 Reedy: dsh delete php-1.26wmf20
  • 22:57 Reedy: dsh delete php-1.26wmf19
  • 22:56 Reedy: dsh delete php-1.26wmf18
  • 22:55 Reedy: dsh delete php-1.26wmf17
  • 22:54 Reedy: dsh delete php-1.26wmf16
  • 22:51 logmsgbot: reedy@mira Synchronized phpunit.xml: (no message) (duration: 00m 29s)
  • 22:49 logmsgbot: reedy@tin Synchronized php-1.26wmf24/: purge l10n cache... test if deletes are propogated (duration: 01m 12s)
  • 22:43 Reedy: deleted more old l10ncaches from tin staging 1.27 wmf1-4
  • 22:37 logmsgbot: reedy@tin Synchronized phpunit.xml: (no message) (duration: 00m 30s)
  • 22:26 logmsgbot: reedy@tin Synchronized php-1.26wmf24: remove old localisation cache (duration: 01m 38s)
  • 22:22 logmsgbot: reedy@tin Synchronized php-1.26wmf23: remove old localisation cache (duration: 01m 40s)
  • 22:10 logmsgbot: reedy@tin Synchronized phpunit.xml: noop test (duration: 01m 03s)
  • 20:51 Reedy: Deleted php-1.26wmf11, php-1.26wmf12 and php-1.26wmf13 from mira /srv/mediawiki-staging
  • 20:40 logmsgbot: reedy@tin Synchronized php-1.27.0-wmf.6/extensions/BounceHandler/: Email header logging on unsubscribe (duration: 00m 53s)
  • 13:04 anomie: Ran cleanupBlocks.php on ruwiki for phab:T118625
  • 02:20 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.6/cache/l10n: l10nupdate for 1.27.0-wmf.6 (duration: 05m 58s)
  • 01:58 ori: restarted HHVM on all app servers

2015-11-13

  • 23:30 logmsgbot: ori@tin Synchronized rpc/RunJobs.php: I5e15ec9fb: Disable ChronologyProtector for rpc/RunJobs.php (duration: 00m 30s)
  • 23:27 awight: Reenabled fundraising campaigns outside of France
  • 23:09 awight: Disabled Fundraising campaigns due to PayPal outage
  • 21:23 logmsgbot: maxsem@tin Synchronized portals/: Did scap errors go away? (duration: 00m 30s)
  • 21:17 YuviPanda: bounced etherpad again
  • 20:39 logmsgbot: maxsem@tin Synchronized portals/: Did scap errors go away? (duration: 00m 29s)
  • 20:38 logmsgbot: maxsem@tin Synchronized portals/: Did scap errors go away? (duration: 00m 29s)
  • 20:35 logmsgbot: maxsem@tin Synchronized portals/: Did scap errors go away? (duration: 00m 30s)
  • 20:30 logmsgbot: maxsem@tin Synchronized portals/: Bump to master (duration: 00m 39s)
  • 20:19 robh: cr2-xe-1/3/0 optic install completed (detects fine)
  • 20:18 robh: cr1-ulsfo:xe-0/0/2 optic install completed (detects fine)
  • 20:17 robh: cr1-ulsfo:xe-0/0/2 (spare port) xfp optic detects fine, now installing other spare into cr2-xe-1/3/0
  • 20:14 robh: installing xfp optics into cr1-ulsfo:xe-0/0/2 (spare port)
  • 18:04 bblack: added neodymium to salt term in common-infrastructure4 on cr[12]-eqiad
  • 17:37 csteipp: applied and reverted patch for T118032 (lots of exceptions on deploy)
  • 16:16 jynus: restarting mysql on db2056, db2063 and db2064 for regular upgrade
  • 15:16 godog: bootstrap restbase2001-b cassandra instance
  • 15:03 jynus: restarting db2058, db2065 for regular mysql upgrade
  • 14:43 jynus: restarting db2059, db2066 for regular mysql upgrade
  • 11:42 jynus: restarting db2060 mysql for upgrade
  • 09:20 jynus: performing table optimization on db1015, expect some lag while depooled
  • 08:37 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1035 at 100%, depool db1015 (duration: 01m 31s)
  • 06:58 _joe_: trying to manually syncing the ubuntu mirrors, since they are corrupted at the moment
  • 02:26 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.6/cache/l10n: l10nupdate for 1.27.0-wmf.6 (duration: 05m 21s)
  • 01:27 YuviPanda: bounced mw1045 HHVM for ebernhardson
  • 01:15 logmsgbot: catrope@tin Finished scap: I totally forgot that the SWAT contained i18n changes (duration: 26m 46s)
  • 00:48 logmsgbot: catrope@tin Started scap: I totally forgot that the SWAT contained i18n changes
  • 00:28 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.6/extensions/QuickSurveys: SWAT (duration: 00m 30s)
  • 00:24 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.6/extensions/Flow: SWAT (duration: 00m 32s)
  • 00:23 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.6/extensions/CentralNotice: SWAT (duration: 00m 31s)
  • 00:16 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Remove hardcoded max upload size for UploadWizard (duration: 00m 30s)
  • 00:15 awight: Banner impressions loader service restored
  • 00:14 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Turn off language detection user test for CirrusSearch (duration: 00m 30s)
  • 00:04 awight: update DjangoBannerStats from 08ffecf412a6a9ff3e173dded6814092834c7d1e to 26b48add5c6f70b395a3195c401a12612c626796

2015-11-12

  • 23:41 awight: update DjangoBannerStats from 9cc2acba7f6b46e243ada0fac6c48c9a49460a07 to 08ffecf412a6a9ff3e173dded6814092834c7d1e
  • 23:05 logmsgbot: bd808@tin Synchronized wmf-config/logging.php: Monolog: wrap channel handlers in a WhatFailureGroupHandler (b08aaf4) (duration: 00m 29s)
  • 22:46 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: wikipedia wikis to 1.27.0-wmf.6
  • 22:42 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.6/extensions/TranslationNotifications/TranslationNotificationJob.php: hotfix T116960 (duration: 00m 30s)
  • 21:21 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.6/resources/lib/jquery.i18n/src/jquery.i18n.language.js: T118242 - Unbreak IE8 (duration: 00m 30s)
  • 21:10 chasemp: reboot analytics1040 it locked up
  • 19:37 awight: Update DjangoBannerStats from 12e819a04a40ee6fab5dd55fcaf072661df31106 to 9cc2acba7f6b46e243ada0fac6c48c9a49460a07
  • 19:24 awight: banner impressions job disabled
  • 19:11 awight: Disabled WMDE and WMF Fundraising campaigns to do banner impression maintenance
  • 19:05 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: SWAT: Set throttle exception for University of Haifa wiki event gerrit:252010 (duration: 00m 29s)
  • 18:21 logmsgbot: maxsem@tin Synchronized portals/: The rest of portals (duration: 00m 31s)
  • 17:25 subbu: finished parsoid deploy sha 392e25eb
  • 17:02 subbu: starting parsoid deploy
  • 16:29 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Deploy QuickSurveys to English Wikipedia gerrit:252606 (duration: 00m 29s)
  • 16:28 ostriches: gerrit: reloading replication plugin to pick up new config
  • 16:11 andrewbogott: depooled T118469. https://phabricator.wikimedia.org/T118469
  • 16:08 bd808: cleaned up stale branch checkouts on mw1041: php-1.26wmf16 php-1.26wmf17 php-1.26wmf18 php-1.26wmf19 php-1.26wmf20 php-1.26wmf21
  • 16:02 mobrovac: restbase end deploy of e5e45d11ad3f
  • 15:48 mobrovac: restbase start deploy of e5e45d11ad3f
  • 15:46 mobrovac: restbase canary deploy on rb1001 of e5e45d11ad3f
  • 15:39 jynus: powercycling mw1041
  • 15:30 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1035 after maintenance (duration: 00m 29s)
  • 10:11 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.6/extensions/UniversalLanguageSelector: I9206a79: Only use jQuery.tipsy for undo popover (duration: 00m 31s)
  • 09:53 jynus: rebooting/upgrading/restarting mariadb on db1035 after maintenance finished
  • 09:52 jynus: uploading new mariadb package linked to openssl instead of yassl
  • 09:44 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.5/extensions/UniversalLanguageSelector: I9206a79: Only use jQuery.tipsy for undo popover (duration: 01m 15s)
  • 09:39 moritzm: corrected apt sources on db2057, db2058, db2062, db2065, db2066, db2067, db2068, db2069
  • 08:42 moritzm: restarted salt-master on palladium
  • 07:35 _joe_: restarting pybal on lvs1004
  • 02:23 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 06m 56s)
  • 01:02 gwicke: restbase: finished full deploy of restbase-deploy ce4abe784
  • 00:50 gwicke: restbase: starting full deploy of restbase-deploy ce4abe784
  • 00:37 gwicke: restbase: canary deploy of restbase-deploy ce4abe784 on restbase1001
  • 00:32 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Fix typo in QuickSurveys config (duration: 00m 31s)

2015-11-11

  • 23:48 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.6
  • 23:44 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.6/includes/jobqueue/aggregator/: sync https://gerrit.wikimedia.org/r/#/c/252588/ (duration: 00m 32s)
  • 21:13 subbu: finished deploying parsoid sha 7ca999c1
  • 21:06 subbu: synced new code + restarted parsoid on wtp1001 as canary; monitoring graphs for a little bit
  • 21:02 subbu: starting parsoid deploy
  • 20:52 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.5
  • 19:51 hoo: Started rebuildItemsPerSite for wikidatawiki on mw1152. Feel free to kill, should it cause troubles.
  • 19:49 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.6
  • 19:37 twentyafterfour: deploying 1.27.0-wmf.6 to group1
  • 19:09 YuviPanda: nfs-exports on labstore1001 failed because of http failure of wikitech
  • 19:05 YuviPanda: started nfs-exports service on labstore1001
  • 18:18 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Repool db2067 after maintenance (duration: 00m 29s)
  • 16:46 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.5/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Restore satisfaction schema and fix the performance issue that it had part II gerrit:252349 (duration: 00m 30s)
  • 16:45 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.5/extensions/WikimediaEvents/WikimediaEvents.php: SWAT: Restore satisfaction schema and fix the performance issue that it had part I gerrit:252349 (duration: 00m 30s)
  • 16:34 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.6/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Restore satisfaction schema and fix the performance issue that it had part II gerrit:252347 (duration: 00m 30s)
  • 16:33 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.6/extensions/WikimediaEvents/WikimediaEvents.php: SWAT: Restore satisfaction schema and fix the performance issue that it had part I gerrit:252347 (duration: 00m 40s)
  • 16:17 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: First QuickSurvey for reader segmentation research - external survey gerrit:251133 (duration: 00m 30s)
  • 16:02 mutante: uploaded jenkins 1.625.2 to apt.wm.org, upgrading on gallium
  • 15:47 moritzm: restarted kafka broker on kafka1022
  • 15:34 moritzm: stopping kafka broker on kafka1022 (to enable ferm)
  • 15:31 moritzm: restarted kafka broker on kafka1020
  • 15:25 moritzm: stopping kafka1020 to enable ferm
  • 15:22 moritzm: restarted kafka1018 to enable ferm
  • 15:19 moritzm: stopping kafka1018 to enable ferm
  • 15:04 moritzm: disabled puppet on kafka1018, kafka1020, kafka1022 (for enabling ferm)
  • 14:18 godog: start cassandra instance a on restbase2001
  • 13:54 mobrovac: restbase deployed 0d961a2 (deploy end)
  • 13:42 mobrovac: restbase deploying 0d961a2
  • 11:28 godog: reimage restbase2001.codfw.wmnet - take #2
  • 10:16 jynus: disabling puppet and restarting mysql on db2067
  • 10:16 godog: reimage restbase2001.codfw.wmnet
  • 10:07 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Depool db2067 for maintenance (not db1067) (duration: 00m 30s)
  • 10:05 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1067 for maintenance (duration: 00m 30s)
  • 09:57 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.6/extensions/OAuth/: really fix T118372 this time (duration: 00m 29s)
  • 09:43 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.6/extensions/OAuth/: oauth login broken by php error. This fixes T118372 (duration: 00m 31s)
  • 08:51 jynus: importing wmf-mariadb10_10.0.22-1_amd64.deb wmf-mysql57_5.7.9-1_amd64.deb on carbon
  • 04:19 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: rm some old log channels (duration: 00m 31s)
  • 03:23 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.6/includes/cache/MessageBlobStore.php: Logging for T93800 (duration: 00m 31s)
  • 02:51 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.5/includes/Title.php: Deploy I47517021471 which was merged earlier today but forgotten (duration: 01m 58s)
  • 02:36 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 07m 01s)
  • 01:48 mutante: repooled mw1083 (T116184)
  • 00:58 mutante: added jenkins_1.625.1 to APT repo. precise-wikimedia
  • 00:50 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.6
  • 00:38 logmsgbot: twentyafterfour@tin Finished scap: grr: sync 1.27.0-wmf.6 (duration: 55m 05s)

2015-11-10

  • 23:42 logmsgbot: twentyafterfour@tin Started scap: grr: sync 1.27.0-wmf.6
  • 23:41 logmsgbot: twentyafterfour@tin scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="cawikibooks" --list-file="/srv/mediawiki-staging/wmf-config/extension-list" --output="/tmp/tmp.7rNhS6PIRi" ' returned non-zero exit status 1 (duration: 00m 09s)
  • 23:41 logmsgbot: twentyafterfour@tin Started scap: grr: sync 1.27.0-wmf.6
  • 23:40 logmsgbot: twentyafterfour@tin scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="cawikibooks" --list-file="/srv/mediawiki-staging/wmf-config/extension-list" --output="/tmp/tmp.yIUW9abUwO" ' returned non-zero exit status 1 (duration: 00m 15s)
  • 23:39 logmsgbot: twentyafterfour@tin Started scap: trying again: sync 1.27.0-wmf.6
  • 23:23 ori: ran sudo chgrp -R wikidev /srv/mediawiki-staging/php-1.27.0-wmf.6 on tin
  • 22:50 ejegg: rolled back paymentswiki from 3730cff201e314fc54ad76fbc54f33f5b032def5 to 4baa5a66a4510414d6b43b59f1b1cda2341c17fd
  • 22:43 logmsgbot: twentyafterfour@tin scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="cawikibooks" --list-file="/srv/mediawiki-staging/wmf-config/extension-list" --output="/tmp/tmp.sEpytaqlt1" ' returned non-zero exit status 1 (duration: 01m 42s)
  • 22:41 logmsgbot: twentyafterfour@tin Started scap: testwiki to 1.27.0-wmf.6
  • 21:51 hashar: installed Zuul package on scandium.eqiad.wmnet . zuul-merger not running
  • 21:33 ejegg: updated paymentswiki from 4baa5a66a4510414d6b43b59f1b1cda2341c17fd to 3730cff201e314fc54ad76fbc54f33f5b032def5
  • 21:16 ejegg: enabled banner_history module on CiviCRM
  • 21:15 mutante: scb1001 - mobile endpoint health CRIT
  • 21:15 mutante: mira unmerged changes in mw-staging
  • 21:14 mutante: restbase2001: cassandra CQl refused
  • 21:14 mutante: aqs1001-1003: CRITICAL: Test Get aggregate page views
  • 21:12 mutante: tungsten - signed new puppet certs, re-installed
  • 20:48 moritzm: kafka re-enabled on kafka1013
  • 20:42 moritzm: stopping kafka broker on kafka1013 to activate ferm
  • 20:26 ottomata: starting kafka broker on kafka1012 after applying base ferm
  • 20:14 ottomata: stopping kafka broker on kafka1012 to apply base ferm
  • 20:09 moritzm: disabled puppet on kafk1012/1013 in preparation of ferm activation
  • 20:08 _joe_: restarting jobchron across the cluster
  • 18:41 ottomata: powercycling analytics1031
  • 16:51 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: https://gerrit.wikimedia.org/r/#/c/252231/ (duration: 00m 35s)
  • 16:45 logmsgbot: krenair@tin Synchronized wmf-config/Wikibase-labs.php: (no message) (duration: 00m 34s)
  • 16:44 logmsgbot: krenair@tin Synchronized wmf-config/Wikibase.php: https://gerrit.wikimedia.org/r/#/c/252204/ (duration: 00m 34s)
  • 16:41 logmsgbot: krenair@tin Synchronized wmf-config/Wikibase.php: https://gerrit.wikimedia.org/r/#/c/250000/ (duration: 00m 34s)
  • 16:34 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/250471/ (duration: 00m 34s)
  • 16:28 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/ContentTranslation/api/ApiQueryContentTranslationSuggestions.php: https://gerrit.wikimedia.org/r/#/c/252172/ (duration: 00m 35s)
  • 16:22 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/Wikidata/Wikidata.php: https://gerrit.wikimedia.org/r/#/c/252227/ (duration: 00m 45s)
  • 15:51 mutante: restarted krypton
  • 15:32 godog: upgrade rsync on ms-be1* per T93587
  • 14:02 godog: swift codfw-prod: ms-be2017 / ms-be2019 / ms-be2021 weight 3000
  • 10:00 jynus: performing mysql maintenance of db1035 (depooled from production). Replication delay and reboots are expected.
  • 09:46 godog: swift codfw-prod: ms-be2017 / ms-be2019 / ms-be2021 weight 2000
  • 09:39 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1035 for maintenance (duration: 00m 36s)
  • 06:13 ejegg|away: updated fundraising tools from f2f571903d32ec73aeab73ce58c6bee71717213b to 182f217ea20a496fe0f9737447d04ed9a3bccde8
  • 03:12 ottomata: restarted statsv on hafnium just in case it stopped after kafka1013 broker crash and restart (ping ori :) )
  • 03:10 ottomata: kafka preferred-replica-election to bring kafka1013 back as leader for its partitions
  • 03:00 mutante: started kafka on kafka1013
  • 02:55 mutante: tried to start kafka on kafka1018
  • 02:31 mutante: switched people.wm.org to new backend. please use "rutherfordium.eqiad.wmnet" instead of terbium now. all files have been rsynced already
  • 02:22 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 06m 49s)
  • 00:13 hoo: Attached Richardelainechambers@commonswiki to the global account of the same name
  • 00:06 logmsgbot: bd808@tin Synchronized wmf-config/logging.php: Disable microsecond timestamps on all loggers (bf61bfc) (duration: 00m 34s)

2015-11-09

  • 23:55 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Disable cirrus writes to labsearch, the machine cant take the load and some jobs are timing out (duration: 00m 34s)
  • 23:13 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Enable CirrusSearch writes to labsearch for all but enwiki and dewiki (duration: 00m 35s)
  • 22:49 ottomata: rebooting analytics1032: https://phabricator.wikimedia.org/T118175
  • 21:52 bearND: MobileApps deployed sha1 6c63984
  • 19:40 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Pool db1060 after performance patch (duration: 00m 35s)
  • 18:42 jynus: restarting mysql in db1060 to test new performance configuration
  • 17:55 bblack: upgrading pybal (1.10 -> 1.12) on lvs200[456].codfw.wmnet
  • 16:50 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/250454/ (duration: 00m 34s)
  • 16:44 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/VisualEditor/modules/ve-mw/init: https://gerrit.wikimedia.org/r/#/c/251972/ (duration: 00m 34s)
  • 16:40 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/251674/ (duration: 00m 35s)
  • 16:35 logmsgbot: krenair@tin Synchronized docroot/wikipedia.org/apple-app-site-association: https://gerrit.wikimedia.org/r/#/c/250897/ (duration: 00m 34s)
  • 16:30 matt_flaschen: Ran mwscript extensions/Flow/maintenance/FlowUpdateBetaFeaturePreference.php --wiki=wikidatawiki
  • 16:24 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/251677/ (duration: 00m 35s)
  • 16:14 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/Flow/modules/mw.flow.Initializer.js: https://gerrit.wikimedia.org/r/#/c/251560/ (duration: 00m 44s)
  • 16:10 _joe_: removing old builds from copper
  • 15:45 godog: reimage graphite1002
  • 15:06 ottomata: running kafka preferred-replica-election
  • 15:02 ottomata: starting kafka broker on kafka1014
  • 14:56 ottomata: stopping kafka broker on kafka1014
  • 14:48 godog: swift codfw-prod: ms-be2017 / ms-be2019 / ms-be2021 weight 1000
  • 14:40 ottomata: restarting eventlogging to see if it is ok after enabling firewall rules on kafka1014
  • 14:36 _joe_: reducing /tmp size on copper by shrinking the logical volume
  • 13:16 godog: nodetool decomission restbase2001
  • 13:13 mobrovac: restbase finished deploying ae2a44f
  • 12:59 mobrovac: restbase start deploying ae2a44f
  • 12:56 hashar: restarting Jenkins to refresh the cli-shutdown.groovy script -- https://gerrit.wikimedia.org/r/251935 (https://phabricator.wikimedia.org/T118064)
  • 12:10 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depooling db1060 again (duration: 01m 13s)
  • 11:16 hashar: Jenkins back and apparently happy
  • 11:09 hashar: Upgrading Jenkins from LTS 1.609.3 to LTS 1.625.1 https://phabricator.wikimedia.org/T118157
  • 10:53 dcausse: resuming writes to elasticsearch indices in eqiad (test)
  • 10:02 _joe_: repooled mw1061
  • 09:29 godog: swift codfw-prod: ms-be2016 / ms-be2018 / ms-be2020 weight 3000
  • 09:19 dcausse: err (1007 no 1008): restarting elastic on elastic1007.eqiad.wmnet (test)
  • 09:18 dcausse: restarting elastic on elastic1008.eqiad.wmnet (test)
  • 09:10 dcausse: freezing elasticsearch indices in eqiad (test)
  • 02:21 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 06m 49s)

2015-11-08

  • 20:43 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/ConfirmEdit/SimpleCaptcha/Captcha.php: https://gerrit.wikimedia.org/r/#/c/251673/ (duration: 00m 34s)
  • 20:21 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.5/resources: (no message) (duration: 00m 34s)
  • 20:14 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.5/resources/src: I24d9b16ed: Rework the Preferences to prevent FOUC (T115692) (duration: 00m 34s)
  • 17:10 _joe_: powercycled ms-be1009, unresponsive to ping, console unconneptable
  • 13:32 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1060 (duration: 00m 34s)
  • 13:02 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1060 (duration: 01m 29s)
  • 02:20 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 06m 21s)

2015-11-07

  • 19:50 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: https://gerrit.wikimedia.org/r/#/c/251739/ (duration: 00m 35s)
  • 17:54 ejegg|away: updated SmashPig from 5860ef47790010d65981822f30914679df06ae14 to 55c2dbc8cd36cfb89d7e17b75b840ca2e023653e
  • 17:38 awight: more syntax fixing
  • 17:35 awight: fix SmashPig failmail syntax
  • 17:29 awight: point SmashPig failmail at ejegg|away and awight
  • 17:28 jynus: restarting labsdb1004 mysql to test mariadb upgrade
  • 17:09 jynus: restarting db2070 mysql to test mariadb upgrade
  • 17:01 jynus: restarting es2010 mysql to test mariadb upgrade
  • 15:17 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Return db weights on s2 api back to normal (duration: 00m 34s)
  • 11:24 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1054 (duration: 00m 34s)
  • 11:02 jynus: setting SET GLOBAL use_stat_tables = 0; on db1060
  • 10:45 jynus: restarting db1054 (I may need to do it several times)
  • 10:41 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1054 (duration: 00m 35s)
  • 10:27 jynus: reverting last schema change
  • 10:08 jynus: performing schema change on db1054 (zhwiki)
  • 05:00 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 34s)
  • 04:59 logmsgbot: ori@tin Synchronized wmf-config/mobile.php: (no message) (duration: 01m 09s)
  • 02:27 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 06m 12s)
  • 00:23 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.5/extensions/WikimediaMaintenance/getPageCounts.php: (no message) (duration: 00m 34s)
  • 00:05 ejegg: updated SmashPig from ac9772765287e79178e07ddfcc7e532d2f82b6d0 to 5860ef47790010d65981822f30914679df06ae14
  • 00:04 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.5/extensions/WikimediaMaintenance/getPageCounts.php: (no message) (duration: 00m 34s)

2015-11-06

  • 23:48 ejegg: updated SmashPig listener from afefccff0f8d5e0b37044adff964c6cd913e1d80 to ac9772765287e79178e07ddfcc7e532d2f82b6d0 and set failmail target back to fr-tech
  • 23:34 logmsgbot: reedy@tin Synchronized wmf-config/InitialiseSettings.php: Enable DynamicPageList on officewiki (duration: 01m 03s)
  • 22:42 legoktm: mwscript updateSpecialPages.php --wiki=enwiki --only=GadgetUsage
  • 19:22 andrewbogott: disabled puppet on holmium while testing and staging an update for bug T117610
  • 19:13 logmsgbot: ori@tin Synchronized wmf-config/jobqueue-eqiad.php: I920627cf8b: Add rdb1007 as fallback aggregator (duration: 00m 35s)
  • 18:54 logmsgbot: ori@tin Synchronized wmf-config/jobqueue-eqiad.php: I9e5be66680: Add rdb1007 to jobqueue pool (duration: 00m 35s)
  • 18:19 ori: running sync-common on mw1083
  • 17:45 logmsgbot: demon@tin Synchronized multiversion/checkoutMediaWiki: no-op outside tin; just for completeness (duration: 00m 36s)
  • 15:29 paravoid: draining esams for network maintainance (deployed about 7mins ago)
  • 14:52 moritzm: uploaded debdeploy 0.0.9 to apt.wikimedia.org
  • 14:31 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Pool all available db servers in codfw (duration: 00m 49s)
  • 13:58 godog: bounce ganglia-monitor-aggregator-instance ID=2027 on install2001, debugging missing ms-be2020
  • 13:38 godog: swift codfw-prod: ms-be2016 / ms-be2018 / ms-be2020 weight 2000
  • 13:36 bblack: cp4007 repooled - T117746
  • 13:16 bblack: upgrading pybal 1.10 -> 1.12 on lvs300[34].esams.wmnet
  • 13:11 bblack: re-enabled aws-d-eqiad in librenms
  • 13:09 bblack: upgrading pybal 1.10 -> 1.12 on lvs4003
  • 12:18 mark: Swapped cr2-knams fpc 1 mic 1 for a new 2x 10G line card
  • 12:18 mark: Swapped cr2-knams fpc 1 mic 1 for a new 2x 10
  • 12:18 mark: mark@cr2-knams> request chassis mic fpc-slot 1 mic-slot 1 online
  • 12:10 mark: mark@cr2-knams> request chassis mic fpc-slot 1 mic-slot 1 offline
  • 11:31 jynus: stopping mysql and cloning db2042 -> db2070, db2054 -> db2068, db2053 -> db2067, db2049 -> db2056
  • 11:12 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Depool db2042, db2049, db2053 and db2054 for cloning (duration: 00m 34s)
  • 09:09 jynus: Stopping mysql and cloning db2044 -> db2065, db2048 -> db2069, db2045 -> db2066
  • 08:55 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Repool db20 34,42,35,38,39,40. New: 58,64,60,61,62,59. Depool: 44,45,48 (duration: 01m 10s)
  • 02:28 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 05m 56s)
  • 01:43 logmsgbot: krenair@mira Synchronized README: done (duration: 00m 34s)
  • 01:38 logmsgbot: krenair@mira Synchronized README: test (duration: 00m 34s)
  • 01:33 logmsgbot: bd808@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/251446/ (duration: 00m 35s)
  • 01:30 logmsgbot: bd808@tin Synchronized wmf-config/logging.php: https://gerrit.wikimedia.org/r/#/c/249313/ (forgot to apply after fetching) (duration: 00m 34s)
  • 01:28 logmsgbot: bd808@tin Synchronized wmf-config/InitialiseSettings.php: Touch (duration: 00m 35s)
  • 01:26 logmsgbot: bd808@tin Synchronized wmf-config/logging.php: https://gerrit.wikimedia.org/r/#/c/249313/ (duration: 00m 34s)
  • 01:10 logmsgbot: krenair@tin Synchronized multiversion/defines.php: https://gerrit.wikimedia.org/r/#/c/251447/ (duration: 00m 34s)
  • 01:00 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/resources/src/mediawiki/mediawiki.ForeignStructuredUpload.BookletLayout.js: https://gerrit.wikimedia.org/r/#/c/251307/ (duration: 00m 34s)
  • 00:59 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/resources/src/mediawiki/mediawiki.ForeignStructuredUpload.js: https://gerrit.wikimedia.org/r/#/c/251301/ https://gerrit.wikimedia.org/r/#/c/251306/ and https://gerrit.wikimedia.org/r/#/c/251307/ (duration: 00m 35s)
  • 00:34 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/251317/ (duration: 00m 35s)
  • 00:18 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/Flow: https://gerrit.wikimedia.org/r/#/c/251416/ (duration: 00m 37s)

2015-11-05

  • 23:34 bd808: Decreased replica count of logstash-2015.10.13 and logstash-2015.10.14 to free disk space on cluster
  • 23:30 bd808: Logging volume into ELK cluster down dramatically; investigating
  • 23:16 logmsgbot: ori@tin Synchronized wmf-config/logging.php: Ieb8c602a: monolog: Ensure that context data added by WebProcessor is utf-8 safe (duration: 00m 36s)
  • 22:35 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.5/extensions/WikimediaEvents: Ic99ac31f740956: Log backend response time on edit requests (duration: 00m 35s)
  • 21:54 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: sync 1.27.0-wmf.5 to group2
  • 20:17 logmsgbot: demon@tin Finished scap: no changes, testing permissions on mira co-master (duration: 07m 29s)
  • 20:10 logmsgbot: demon@tin Started scap: no changes, testing permissions on mira co-master
  • 20:07 _joe_: chown mwdeploy:wikidev recursively on mira for /srv/mediawiki-staging
  • 19:37 logmsgbot: twentyafterfour@tin Finished scap: Sync everything just to be sure (duration: 22m 01s)
  • 19:15 logmsgbot: twentyafterfour@tin Started scap: Sync everything just to be sure
  • 18:58 logmsgbot: demon@tin Synchronized README: no-op, testing new scap code (duration: 00m 19s)
  • 18:52 ostriches: scap: deploying master@b44c268
  • 18:05 jynus: schema change on x1 - flowdb
  • 17:47 ostriches: 17:45:13 Synchronized wmf-config/: Remove +superprotect, I579c11a2 (duration: 00m 18s)
  • 17:44 jynus: Deploying schema change on officewiki - flow (s3)
  • 17:11 godog: nodetool decommission on praseodymium
  • 14:12 jynus: reinstalling db2056.codfw.wmnet
  • 13:57 jynus: shuting down mysql and cloning db2042 -> db2062, db2038 -> db2059, db2039 -> db2060, db2040 -> db2061
  • 13:43 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Repool db2051; Depool db2042, 38, 39, 40 for cloning (duration: 00m 18s)
  • 12:33 _joe_: manually disabling mod_cgi on terbium
  • 09:56 godog: run removenode on cerium.eqiad.wmnet -- decomission was missed before reimaging
  • 09:33 jynus: stopping mysql and cloning db2034 -> db2062, db2035 -> db2063, db2051 -> db2058
  • 09:25 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Pool db2048, db2049, db2050,db2055, db2057, db2063. Depool db2034, db2035, db2051 (duration: 00m 17s)
  • 08:05 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Nov 5 08:05:13 UTC 2015 (duration 5m 12s)
  • 03:21 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.5) at 2015-11-05 03:21:45+00:00
  • 03:15 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 10m 17s)
  • 02:47 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-11-05 02:47:20+00:00
  • 02:40 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 10m 31s)
  • 02:17 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I3c397e892e: Only set $wgDisableOutputCompression to 'true' on the scalers (duration: 00m 18s)
  • 02:04 Krenair: Someone has left tin:/srv/mediawiki-staging/php-1.27.0-wmf.5 in a mess, see `git log origin/wmf/1.27.0-wmf.5..HEAD --oneline`. Note there is one commit waiting to be merged on tin (https://gerrit.wikimedia.org/r/#/c/251168/) that hasn't been yet because of this.
  • 00:13 ejegg: updated fundraising tools from e1b60fa2c258fd4ff55905b03a4d8886132278c1 to f2f571903d32ec73aeab73ce58c6bee71717213b

2015-11-04

  • 23:56 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.5/maintenance/findOrphanedFiles.php: ced9a62893c (duration: 00m 17s)
  • 23:55 ejegg: updated CiviCRM from 827faf37b1b8f4ade6f8d03077c376a09d9525c7 to 230cebb4a3b46a75fef758d46f4db22de29286cb
  • 23:09 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.5/maintenance/findOrphanedFiles.php: 33d378bee6c4ac (duration: 00m 17s)
  • 22:33 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.5/maintenance/findOrphanedFiles.php: fb43e95858b9 (duration: 00m 18s)
  • 21:36 ejegg: enabled Adyen payments listener
  • 21:17 subbu: finished deploying parsoid sha 04893a18
  • 21:05 subbu: synced code + restarted parsoid on wtp1002 as a canary
  • 21:01 subbu: starting parsoid deploy
  • 20:45 ejegg: disabled Adyen payment listener
  • 20:36 ejegg: re-enabled Adyen payment listener with updated credentials
  • 20:13 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.5
  • 18:37 mutante: correction. ^ that was mw1159
  • 18:35 mutante: mw1159: power cycled - crashed, no console output
  • 18:23 logmsgbot: bd808@tin Synchronized php-1.27.0-wmf.5/includes/debug/logger/MonologSpi.php: MonologSpi: add support for customizing Monolog\Logger instances (59eed42) (duration: 00m 17s)
  • 18:23 mutante: ms-be2016-2020 - accepted salt keys, hafnium.wm.org - deleted salt key - still unaccepted: capella, haedus, rhodium
  • 17:57 mutante: ms-be2020 - added to puppet
  • 17:54 mutante: ms-be2021 - fixing puppet certs after reinstall
  • 17:51 papaul: ms-be2021 installation complete
  • 17:45 papaul: reinstalling ms-be2021
  • 17:20 ejegg: updated SmashPig from be9bf87c5583bca75616284f6a4d16ff8ebd3679 to afefccff0f8d5e0b37044adff964c6cd913e1d80
  • 17:12 chasemp: rebooting cp4007.mgmt.ulsfo.wmnet as it's uresponsive
  • 17:11 bblack: cp4007 depooled in pybal + confctl
  • 17:10 papaul: ms-be2020 installation complete
  • 16:56 ejegg: disabled adyen payments listener
  • 16:30 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/250971/ (duration: 00m 18s)
  • 16:29 bd808: restarted elasticsearch on logstash1004; it fell out of the cluster during ferm firewall application
  • 16:28 jynus: db2056 clone failed due to wrong partitioning, needs reinstall. Cloning db2049 -> db2063 instead
  • 16:01 godog: reimage cerium for multi-instance
  • 15:22 _joe_: moving email batch, flagged reviews, update article count off of terbium
  • 15:20 jynus: syncing sqldata on db2049 -> db2056, db2050 -> db2057, eta 2-4 hours
  • 15:15 _joe_: moved translationnotification, updatetranslationstats, pagetriage from terbium to mw1152
  • 14:55 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Depool db2049 ans db2050 for maintenance (duration: 00m 17s)
  • 14:30 jynus: cloning sqldata from db2048 to db2055
  • 14:04 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Depool db2048 for maintenance (duration: 00m 18s)
  • 13:03 bblack: upgrading pybal (1.10 -> 1.12) on lvs4004.ulsfo.wmnet (inactive low-traffic LVS @ ulsfo)
  • 11:59 _joe_: moved refreshlink jobs off terbium
  • 11:32 _joe_: migrating wikidata jobs off of terbium
  • 11:07 jynus: testing new mariadb-server version on db2056.codfw.wmnet
  • 09:25 godog: bootstrap cassandra xenon-b.eqiad.wmnet
  • 08:47 jynus: reduced durability on dbstore1002 and db1047 to improve performance
  • 08:27 _joe_: moved parser cache purging off of terbium
  • 08:25 _joe_: moved clean_uploadstash to mw1152 from terbium
  • 06:44 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Nov 4 06:44:23 UTC 2015 (duration 44m 22s)
  • 05:52 logmsgbot: demon@tin Finished scap: using wfLoadExtension() for ContentTranslation, shut up warning spam (duration: 49m 22s)
  • 05:03 logmsgbot: demon@tin Started scap: using wfLoadExtension() for ContentTranslation, shut up warning spam
  • 05:02 logmsgbot: demon@tin scap failed: CalledProcessError Command '['sudo', '-u', 'mwdeploy', '-n', '--', '/usr/bin/rsync', '--archive', '--delete-delay', '--delay-updates', '--compress', '--delete', '--exclude=**/.svn/lock', '--exclude=**/.git/objects', '--exclude=**/.git/**/objects', '--exclude=**/cache/l10n/*.cdb', '--no-perms', 'tin.eqiad.wmnet::common', '/srv/mediawiki']' returned non-zero exit status 24 (duration: 02m 45s)
  • 04:59 logmsgbot: demon@tin Started scap: using wfLoadExtension() for ContentTranslation, shut up warning spam
  • 04:54 ejegg|away: updated smashpig from c43db1742d5663f9d3685fd3e0a254ff72a29fa0 to be9bf87c5583bca75616284f6a4d16ff8ebd3679, changed credentials
  • 03:09 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.5) at 2015-11-04 03:09:44+00:00
  • 03:03 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 10m 08s)
  • 02:36 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-11-04 02:36:48+00:00
  • 02:32 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 07m 34s)
  • 01:36 legoktm: ran delete from globaluser where gu_name="" limit 1 on centralauth. (T96233)
  • 01:10 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: https://gerrit.wikimedia.org/r/#/c/250874/ (duration: 00m 17s)
  • 01:02 hoo: Manually recreated https://dumps.wikimedia.org/wikidatawiki/entities/dcatap.rdf using Zend php (instead of HHVM). See T117534.
  • 00:25 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/250869/ (duration: 00m 19s)
  • 00:09 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/250712/ (duration: 00m 18s)
  • 00:03 logmsgbot: krenair@tin Synchronized portals: https://gerrit.wikimedia.org/r/#/c/250851/ (duration: 00m 17s)

2015-11-03

  • 23:11 ejegg: deployed Adyen credentials change to payments wiki
  • 22:36 ostriches: tin: scap now pointing to Phab repo instead of Gerrit.
  • 21:59 logmsgbot: krenair@tin Synchronized README: rv, was just demonstrating for https://phabricator.wikimedia.org/T117574#1779384 (duration: 00m 17s)
  • 21:42 logmsgbot: krenair@tin Synchronized README: (no message) (duration: 00m 18s)
  • 21:21 mutante: ms-be2018,ms-be2019: sign puppet certs, initial run
  • 20:45 csteipp: deployed patch for T109724 to wmf4/5
  • 20:41 gwicke: restbase cassandra: switched local_group_wikipedia_T_parsoid_html to the Date-Tiered Compaction Strategy (DTCS)
  • 20:38 gwicke: cleared cassandra snapshots on aqs servers
  • 19:59 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.5
  • 19:55 mutante: hafnium - revoke puppet cert, salt key for the .wikimedia.org name
  • 19:54 mutante: hafnium - reboot into PXE for jessie reinstall
  • 19:47 logmsgbot: twentyafterfour@tin Finished scap: testwiki to 1.27.0-wmf.5 (duration: 30m 00s)
  • 19:35 awight: update CRM from 2828fdfbf791c6ca28d1f92725d1ec887b9fe4c7 to 827faf37b1b8f4ade6f8d03077c376a09d9525c7
  • 19:33 awight: update SmashPig from c431c8d77521270236c72532a50806b2e852cf7b to c43db1742d5663f9d3685fd3e0a254ff72a29fa0
  • 19:17 logmsgbot: twentyafterfour@tin Started scap: testwiki to 1.27.0-wmf.5
  • 19:11 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/includes/libs/objectcache/BagOStuff.php: 46dff75b5b: Make makeKeyInternal() limit more conservative (duration: 00m 18s)
  • 19:06 twentyafterfour: deploying 1.27.0-wmf.5 to group0
  • 18:34 ori: Moved statsv script from hafnium to a screen session on eventlog2001 in preparation for hafnium reimage
  • 18:29 ori: Moved navtiming script from hafnium to a screen session on eventlog2001 in preparation for hafnium reimage
  • 17:13 moritzm: installed security updates on the internal ntpds (system-local ones to follow)
  • 16:27 dcausse: restarting elastic on nobelium to check new settings
  • 16:19 moritzm: installed unzip and audiofile security updates across the fleet
  • 16:12 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove sampling of CirrusSearchRequestSet log channel gerrit:250691 (duration: 00m 18s)
  • 16:07 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor for 10% of new accounts on eswiki gerrit:250470 (duration: 00m 18s)
  • 14:19 jynus: disabling puppet on dbstore1002 and db1047 to debug replication issue
  • 14:07 godog: manual invocation of /usr/local/bin/archive-instances on labmon1001 to test
  • 13:33 jynus: migrating eventlogging_sync from terbium to db1047 and dbstore1002 (analytics-store)
  • 12:15 _joe_: updated pybal for jessie to 1.12
  • 11:00 godog: start cassandra-a on xenon
  • 09:56 godog: reimage xenon.eqiad.wmnet
  • 09:45 godog: decomission xenon.eqiad.wmnet from cassandra, pending conversion to multi-instance
  • 05:38 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Nov 3 05:38:33 UTC 2015 (duration 38m 32s)
  • 02:26 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-11-03 02:26:14+00:00
  • 02:22 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 07m 00s)
  • 01:05 awight: enabled CRM module amazon_audit
  • 01:03 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.4/extensions/CentralAuth/includes/CentralAuthUser.php: Ia9673a448da81: User cache key (duration: 00m 17s)
  • 01:03 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.4/includes/: I58836a24b9e239f: User cache key (duration: 00m 21s)
  • 00:37 logmsgbot: ebernhardson@tin Synchronized wmf-config/logging.php: https://gerrit.wikimedia.org/r/#/c/240615/ (duration: 00m 19s)
  • 00:35 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/240615/ (duration: 00m 17s)
  • 00:34 logmsgbot: ebernhardson@tin Synchronized wmf-config/avro/CirrusSearchRequestSet.avsc: https://gerrit.wikimedia.org/r/#/c/240615/ (duration: 00m 17s)
  • 00:28 ebernhardson: sync-common on mw1017 for https://gerrit.wikimedia.org/r/240615
  • 00:26 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/241226/4 (duration: 00m 17s)
  • 00:17 awight: update CRM from 77cd64317404b2fcec9500cfbb1c0b19e5846705 to 2828fdfbf791c6ca28d1f92725d1ec887b9fe4c7
  • 00:13 logmsgbot: ebernhardson@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/250590/ (duration: 00m 18s)
  • 00:09 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: add www.webarchive.co.uk and *.unesco.org to server side upload whitelist (duration: 00m 18s)
  • 00:06 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/246169/ (duration: 00m 18s)

2015-11-02

  • 23:11 RoanKattouw: Running convertAllLqtPages.php on ptwikibooks
  • 22:38 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Re-enable Flow on ptwikibooks with English-only Topic namespace name (duration: 00m 18s)
  • 22:38 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Plumbing for wmgFlowEnglishNamespaceOnly (duration: 00m 18s)
  • 21:56 cwd: updated worldpay request timeout to 30s
  • 21:31 _joe_: restarting parsoids in codfw
  • 21:25 subbu: deployed parsoid f0d77afc
  • 20:49 mutante: ms-be2017 - initial puppet run, fresh install
  • 20:38 logmsgbot: csteipp@tin Synchronized php-1.27.0-wmf.4/includes/: (no message) (duration: 00m 22s)
  • 20:38 robh: lawrencium having puppet stopped and reclaiming into spares per T117477 disregard any alerts
  • 20:06 mutante: ms-be2016 - signing puppet certs, salt-key, initial run
  • 20:05 mutante: started morebots (production-logbot job was gone)
  • 19:46 logmsgbot: hoo@tin Synchronized php-1.27.0-wmf.4/extensions/MobileFrontend/includes/MobileFrontend.skin.hooks.php: Fix changing the license message key via the "MobileLicenseLink" hook (duration: 00m 17s)
  • 19:14 gwicke: restbase cassandra: switching local_group_wikipedia_T_parsoid_dataW4ULtxs1oMqJ to DTCS
  • 19:00 matt_flaschen: Completed FlowFixLinks run on all Flow wikis
  • 18:53 csteipp: deployed patches for T97897 & T115522
  • 18:53 awight: updated CRM from 60c4a8e5fc4efb6ac38a520e652afe187e8c177b to 77cd64317404b2fcec9500cfbb1c0b19e5846705
  • 18:48 andrewbogott: dist-upgrade and rebooting labvirt1005
  • 18:41 Coren: manually starting service replicate-tools on labstore1001
  • 18:10 awight: removed failing SmashPig job runner task
  • 18:01 jynus: performing some fixes on labsdb1004 database. Slightly higher load on labsdb1005.
  • 17:59 dcausse: restarting elastic on nobelium
  • 17:43 awight: disabled Donations queue consume job
  • 17:39 gwicke: restbase cassandra: switched local_group_wikisource_T_parsoid_html and local_group_wikibooks_T_parsoid_html to DTCS
  • 17:09 bd808: Decreased replica count to 1 for logstash-2015.10.04 thru logstash-2015.10.12 to free cluster disk space; see T117438
  • 17:07 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/249762/ (duration: 00m 19s)
  • 17:00 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/247088/ (duration: 00m 17s)
  • 16:48 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/250416/ (duration: 00m 17s)
  • 16:47 logmsgbot: krenair@tin Synchronized dblists/visualeditor-default.dblist: https://gerrit.wikimedia.org/r/#/c/250416/ (duration: 00m 17s)
  • 16:45 logmsgbot: krenair@tin Synchronized dblists/closed.dblist: https://gerrit.wikimedia.org/r/#/c/250416/ - close wikimania2014.wikimedia.org (duration: 00m 18s)
  • 16:36 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/249794/ - should be a noop for prod (duration: 00m 17s)
  • 16:30 bd808: Deleted logstash-2015.10.03 index to free disk space on logstash1004; see T113571
  • 16:28 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.4/extensions/Flow: https://gerrit.wikimedia.org/r/#/c/249940/1 (duration: 00m 20s)
  • 16:26 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/249935/ (duration: 00m 18s)
  • 16:09 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/250401/ (duration: 00m 18s)
  • 14:43 ottomata: shutting down analytics1038 to run fsck on root filesystem. Currently it is read only
  • 14:36 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-br-fr_0.5.0+svn~57870-1
  • 14:35 mobrovac: citoid deployed 70abb90
  • 13:34 paravoid: draining ulsfo, T116928
  • 12:19 mobrovac: mobileapps deploying 59e8c30
  • 10:52 hashar: Restarting Jenkins to upgade Gearman plugin from 0.1.2 to 0.1.3 "Send node labels back on build completion"
  • 10:10 akosiaris: powering off pollux. migrating to a VM
  • 09:39 _joe_: moved tor exit node, update_special_pages off of terbium to mw1152
  • 09:24 mobrovac: citoid restarted zotero on sca100x
  • 09:07 jynus: deleting user_daily_contribs view on labs (it points to a non-existent table)
  • 08:57 mobrovac: restbase deploying 8036232
  • 05:33 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Nov 2 05:33:07 UTC 2015 (duration 33m 6s)
  • 02:25 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-11-02 02:25:42+00:00
  • 02:22 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 06m 44s)

2015-11-01

  • 16:59 gwicke: restarting cassandra on aqs1002; was out of heap space
  • 05:25 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Nov 1 05:25:49 UTC 2015 (duration 25m 48s)
  • 02:24 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-11-01 02:24:46+00:00
  • 02:21 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 06m 36s)

2015-10-31

  • 08:57 dcausse: elastic1008 deleting /var/log/elasticsearch/production-search-eqiad_index_indexing_slowlog.log.[2-7]
  • 05:15 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Oct 31 05:15:43 UTC 2015 (duration 15m 42s)
  • 02:54 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/vendor/monolog/monolog/src/Monolog/Logger.php: Iccfda4768 (duration: 00m 19s)
  • 02:26 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-10-31 02:26:22+00:00
  • 02:23 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 06m 26s)

2015-10-30

  • 23:53 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: Ifd3543d99: Re-enabled sidebar cache per 47eb083a0fe4 (duration: 00m 17s)
  • 23:38 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/includes/cache/MessageCache.php: Id9a27ba2bbd3 (duration: 00m 17s)
  • 23:38 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/includes/skins/Skin.php: Id9a27ba2bbd3 (duration: 00m 17s)
  • 23:37 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/Collection: b89cff0a1d: Updated mediawiki/core Project: mediawiki/extensions/Collection 092204bf333e65b2c749e4bc4a32fd2b0254089b (duration: 00m 18s)
  • 23:37 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/DismissableSiteNotice: a20aa8a544: Updated mediawiki/core Project: mediawiki/extensions/DismissableSiteNotice 5b8f1cfac48704fd740eb61f873aabb600c4c5fb (duration: 00m 17s)
  • 23:03 gwicke: fixed restbase labs install by deploying it; the code had gone missing
  • 23:03 ori: Added MaxSem and gwicke to Gerrit Project Creators group
  • 22:44 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/includes/objectcache/ObjectCache.php: If12aedae7f: objectcache: Use singleton cache in newAccelerator() (duration: 00m 18s)
  • 22:05 logmsgbot: ori@tin Synchronized docroot and w: (no message) (duration: 00m 17s)
  • 22:01 mutante: labvirt1010: salt-minion always in "stop/waiting", doesn't restart
  • 21:54 mutante: labvirt1010 - start salt
  • 21:41 ori: Removing MediaWiki.xhprof.* from graphite{1,2}001
  • 21:41 logmsgbot: ori@tin Synchronized wmf-config/StartProfiler.php: I0bfa21b5: Disable xhprof profiling to relieve storage pressure on graphite (duration: 00m 18s)
  • 19:07 K4-713: antifraud tweak on payments wiki (again)
  • 18:56 K4-713: antifraud tweak on payments wiki
  • 18:56 jynus: disabling puppet and restarting mysql on es2008, es2009 and es2010- downtime planned for 2 hours - no production impact
  • 17:33 robh: bismuth has no console output, appears locked up in OS, rebooting via mgmt
  • 17:22 logmsgbot: ori@tin Synchronized docroot and w: (no message) (duration: 00m 18s)
  • 16:56 cmjohnson: reinstalling mw1083 https://phabricator.wikimedia.org/T116184
  • 15:35 moritzm: updated db205[6-8] to the 3.19 kernel
  • 14:38 chasemp: nobelium in downtime -- this is a temporary test host for discovery and is not ops actionable
  • 13:47 Krenair: revert commit for https://gerrit.wikimedia.org/r/#/c/249971/ on tin
  • 13:47 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.4/extensions/VisualEditor/VisualEditor.hooks.php: https://gerrit.wikimedia.org/r/#/c/249999/ (duration: 00m 17s)
  • 11:57 paravoid: upgrading tor to latest stable on radium
  • 11:31 paravoid: ignoring asw-d-eqiad on librenms
  • 11:25 _joe_: powercycling dataset1001, stuck in some kernel task, unable to login from console
  • 10:42 dcausse: elastic in eqiad setting indexing trace threshold to -1 for commons, ruwiki, frwiki and itwiki
  • 10:30 dcausse: elastic in eqiad wide settings for indexing slow threshold are ineffective, trying to set index settings (T117181)
  • 09:18 dcausse: elastic in eqiad setting index.indexing.slowlog.threshold.index.trace to -1
  • 08:46 _joe_: removed production-search-eqiad_index_indexing_slowlog.log.{7,8} on es1008
  • 06:24 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.4/includes/jobqueue/JobQueueRedis.php: (no message) (duration: 00m 18s)
  • 05:11 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Oct 30 05:11:08 UTC 2015 (duration 11m 7s)
  • 04:19 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I1d0337b58: ScribuntoSlowFunctionThreshold: 0.90 -> 0.99 (duration: 00m 17s)
  • 04:06 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/Scribunto/common/Hooks.php: I60b9eb617: When logging perf stats, include wfWikiId() in metric key (duration: 00m 18s)
  • 04:03 awight: update CRM from e0cad7215192464941d0ec282a70cb608619cf2f to 60c4a8e5fc4efb6ac38a520e652afe187e8c177b
  • 03:42 awight|parentoid: update CRM from 367cd85954697c6ad4311deb2799fe9ba08b5d9d to e0cad7215192464941d0ec282a70cb608619cf2f
  • 02:47 awight|parentoid: donations queue consumer reenabled
  • 02:45 awight|parentoid: updated CRM from 61c8c13efb31f7e75564d9a01fa879db0690fb78 to 367cd85954697c6ad4311deb2799fe9ba08b5d9d
  • 02:30 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-10-30 02:30:50+00:00
  • 02:27 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 06m 13s)
  • 00:48 logmsgbot: tgr@tin Synchronized php-1.27.0-wmf.4/includes/specials/SpecialUserlogin.php: T117027 (duration: 00m 18s)
  • 00:34 mutante: labvirt1010 - started salt-minion (no such process)
  • 00:34 mutante: mw1193 - restarted HHVM (socket timeout)
  • 00:34 mutante: wdqs1001 - restarted NTP (unknown offset)
  • 00:09 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.4/extensions/Flow/: Fix regressions from SWAT (duration: 00m 20s)
  • 00:08 mutante: radium - back in service
  • 00:01 bd808: Restarted logstash on logstash1003 again. The first try apparently didn't take

2015-10-29

  • 23:44 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: Ibac0d60bd: Disable ScribuntoGatherFunctionStats (duration: 00m 17s)
  • 23:32 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-common.php: https://gerrit.wikimedia.org/r/249922 (duration: 00m 17s)
  • 23:32 bd808: Restarted logstash on logstash1003; died with OOM error
  • 23:24 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/Scribunto/engines/LuaSandbox/Engine.php: I69e9218 (duration: 00m 18s)
  • 23:21 mutante: radium: scheduled downtime, reinstalling
  • 23:19 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.4/extensions/CirrusSearch/: Fix unwritable cluster errors (duration: 00m 19s)
  • 23:18 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.4/extensions/Flow: Fix CAPTCHA rendering in RTL languages (duration: 00m 19s)
  • 23:17 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.4/extensions/Scribunto/: Make the percentile threshold for slow function stats configurable (duration: 00m 18s)
  • 23:16 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.4/resources/src/mediawiki.special/mediawiki.special.search.css: SWAT: styling tweaks for inline interwiki search (duration: 00m 18s)
  • 23:09 awight: set dashboard cache interval to 60m
  • 22:33 logmsgbot: tgr@tin Synchronized php-1.27.0-wmf.4/extensions/ZeroBanner/includes/ZeroSpecialPage.php: T116821 (duration: 00m 17s)
  • 22:31 logmsgbot: tgr@tin Synchronized php-1.27.0-wmf.4/includes/MagicWord.php: T117066, fixes some exceptions in production (duration: 00m 17s)
  • 22:30 logmsgbot: tgr@tin Synchronized php-1.27.0-wmf.4/includes/specials/SpecialSearch.php: c249899, fixes some warnings in production (duration: 00m 17s)
  • 22:29 logmsgbot: tgr@tin Synchronized php-1.27.0-wmf.4/includes/Category.php: c249890, fixes some warnings in production (duration: 00m 18s)
  • 22:05 awight: CiviCRM upgrade to 4.6.9 succeeded; new data is backed up
  • 22:02 gwicke: restbase: switched local_group_default_T_parsoid_html to Date-Tiered compaction (DTCS)
  • 21:40 logmsgbot: tgr@tin Synchronized php-1.27.0-wmf.4/includes/MagicWord.php: T117066 (duration: 00m 18s)
  • 21:32 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/includes/libs/Xhprof.php: (no message) (duration: 00m 18s)
  • 21:06 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/Cite/Cite_body.php: Revert "Avoid counting arrays if not needed" (duration: 00m 17s)
  • 20:56 gwicke: restbase: switch local_group_wikiquote_T_title__revisions to Date-Tiered compaction (DTCS)
  • 20:47 logmsgbot: ori@tin rebuilt wikiversions.cdb and synchronized wikiversions files: (no message)
  • 19:50 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I06879b6e6e: Enable ScribuntoGatherFunctionStats (duration: 00m 17s)
  • 19:40 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.27.0-wmf.3
  • 19:39 twentyafterfour: rolling back to 1.27.0-wmf.3 due to increase in log errors
  • 19:31 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.27.0-wmf.4
  • 18:59 cmjohnson1: powering down aqs1001 for h/w maintenance
  • 17:53 AaronSchulz: Touched/synced PrivateSettings.php symlink via touch -h
  • 17:47 logmsgbot: aaron@tin Synchronized wmf-config/PrivateSettings.php: (no message) (duration: 00m 18s)
  • 17:36 AaronSchulz: Did touch of InitialiseSettings.php
  • 17:33 logmsgbot: aaron@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 18s)
  • 17:32 logmsgbot: aaron@tin Synchronized private/PrivateSettings.php: (no message) (duration: 00m 17s)
  • 17:17 logmsgbot: kartik@tin Synchronized wmf-config/PrivateSettings.php: Really sync right file this time (duration: 00m 17s)
  • 17:10 logmsgbot: kartik@tin Synchronized private/PrivateSettings.php: Retry syncing for CX token (duration: 00m 17s)
  • 17:04 awight: restarting CiviCRM schema migration: v4.3.alpha1 -> v4.6.9 -- Estimated completion 21:30 UTC
  • 17:02 awight: update crm from 23d2020448f343c8c1b2f4d779be66f57552f935 to 61c8c13efb31f7e75564d9a01fa879db0690fb78
  • 16:57 YuviPanda: restart etherpad on etherpad1001
  • 16:22 moritzm: removed obsolete mysql 5.5 packages on mw102[2-9], mw1032, mw1053, mw1114, mw1163
  • 16:19 _joe_: moved purge_abusefilter from terbium to mw1152
  • 15:48 _joe_: moving purge_checkuser from terbium to mw1152
  • 15:43 _joe_: moving purge_securepoll from terbium to mw1152
  • 15:15 moritzm: removed openjdk-7 on restbase100[1-9] (it's using openjdk-8 for a while)
  • 13:17 logmsgbot: kartik@tin Synchronized wmf-config/CommonSettings.php: Fix ContentTranslationCXServerAuth for CX (duration: 00m 18s)
  • 13:14 logmsgbot: kartik@tin Synchronized private/PrivateSettings.php: Fix name of JWT token (duration: 00m 18s)
  • 12:33 logmsgbot: kartik@tin Synchronized wmf-config/CommonSettings.php: Set ContentTranslationCXServerAuth for CX (duration: 00m 17s)
  • 11:24 mobrovac: restbase rolling-restart after config change https://gerrit.wikimedia.org/r/249465
  • 10:41 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1065 at 100% load. Reduce db1035 weight. (duration: 00m 17s)
  • 10:34 _joe_: migrated jobqueue_stats_reporter to mw1152
  • 10:21 hashar: restarting Jenkins (java upgrade)
  • 10:20 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1065 after maintenance (duration: 00m 18s)
  • 10:14 hashar: Upgrading java on gallium and restarting Jenkins
  • 10:04 moritzm: removed openjdk-7 from cassandra test hosts (now using openjdk-8)
  • 10:01 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable nearby on wikidata (duration: 00m 18s)
  • 10:00 logmsgbot: aude@tin Synchronized wmf-config/Wikibase.php: Config for fetching labels on mobile wikidata (duration: 00m 18s)
  • 09:51 jynus: shutdown mw1083 to avoid querying the mysql servers with an outdated config/spaming the error logs
  • 09:13 jynus: restarting db1065 for regular maintenace
  • 08:59 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1065 for maintenance (duration: 00m 17s)
  • 07:04 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Oct 29 07:04:04 UTC 2015 (duration 4m 3s)
  • 06:43 YuviPanda: disabled two fr-tech related catchpoint tests per awight's request
  • 06:30 logmsgbot: aaron@tin Synchronized rpc/RunJobs.php: 29ccbd248 (duration: 00m 17s)
  • 05:46 ebernhardson: finished manually running 3M enwiki/enqueue, enwiki/wikibase-addUsagesForPage, and wikidatawiki/cirrusSearchLinksUpdatePrioritized jobs from mw1011 and mw1012
  • 05:38 ebernhardson: restarting elasticsearch on nobelium to attempt to clear up extra log GC pauses in the old generation (50s+)
  • 05:37 ebernhardson: restarting elasticsearch on nobelium
  • 03:55 gwicke: cassandra *staging*: testing DateTieredCompactionStrategy (https://labs.spotify.com/2014/12/18/date-tiered-compaction/) on wikipedia html and data-parsoid tables
  • 03:05 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-10-29 03:05:52+00:00
  • 03:02 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 06m 03s)
  • 02:59 YuviPanda: start pybal on lvs1007
  • 02:46 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-29 02:46:27+00:00
  • 02:39 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 10m 47s)
  • 02:05 AaronSchulz: Restarted stuck hhvm on mw1016; dump at /tmp/hhvm.25097.bt
  • 01:06 gwicke: lowered gc_grace on wikipedia parsoid html and data-parsoid keyspaces to 24 hours

2015-10-28

  • 23:31 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/Translate/tag/PageTranslationHooks.php: I0e5f2d3b2 (duration: 00m 18s)
  • 23:29 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.4/extensions/WikimediaEvents/WikimediaEvents.php: https://gerrit.wikimedia.org/r/249642 (duration: 00m 17s)
  • 23:25 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-production.php: (no message) (duration: 00m 17s)
  • 23:22 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-common.php: (no message) (duration: 00m 17s)
  • 23:19 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/249603 (duration: 00m 17s)
  • 23:19 logmsgbot: ebernhardson@tin Synchronized wmf-config/mobile.php: https://gerrit.wikimedia.org/r/249603 (duration: 00m 18s)
  • 23:18 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/249603 (duration: 00m 17s)
  • 23:15 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.4/extensions/MobileFrontend/: https://gerrit.wikimedia.org/r/249585 (duration: 00m 19s)
  • 23:08 logmsgbot: ebernhardson@tin Synchronized portals: Switch www portals to be deployed from Git, but not being served from anywhere yet (duration: 00m 18s)
  • 23:01 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/WikimediaEvents/WikimediaEventsHooks.php: I0e5f2d3b2 (duration: 00m 18s)
  • 22:57 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/CirrusSearch/includes/Hooks.php: I0e5f2d3b2 (duration: 00m 18s)
  • 22:38 urandom: Cassandra cleanup on restbase-test2001-a
  • 22:31 awight: Fundraising CRM db migration started, estimated completion 10:15 UTC (tomorrow)
  • 22:11 awight: update fundraising CRM to civi-4.6.9-deployment branch, from f2fa7b942625b34ede520e11f20e7e0835ecb17d to 23d2020448f343c8c1b2f4d779be66f57552f935
  • 21:11 logmsgbot: legoktm@tin Synchronized php-1.27.0-wmf.4/includes/changes/EnhancedChangesList.php: Fix diff/history links not showing up for ungrouped enhanced RC - https://gerrit.wikimedia.org/r/#/c/249556/ (duration: 00m 19s)
  • 21:08 YuviPanda: enable puppet on labstore1001
  • 20:55 YuviPanda: reverted changes to nsswitch.conf from puppet run manually on labstore2001
  • 20:29 YuviPanda: disable puppet on labstore1001
  • 20:20 awight: Disabled fundraising CiviCRM
  • 19:52 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.4
  • 18:50 Krinkle: mwscript deleteEqualMessages.php --wiki itwikiversity
  • 18:42 logmsgbot: legoktm@tin Synchronized wmf-config/mc.php: https://gerrit.wikimedia.org/r/#/c/249463/ (duration: 00m 17s)
  • 18:41 ostriches: rolled scap back to master@62a250a, needs puppet changes before new code goes live
  • 18:34 gwicke: restbase deploy done
  • 18:34 yurik: updated kartotherian & tilerator services
  • 18:23 gwicke: rolling deploy of restbase-deploy 3b1f6488f2 to restbase cluster
  • 18:18 gwicke: canary deploy of restbase deploy 3b1f6488f2 to restbase1001
  • 18:13 Krinkle: mwscript deleteEqualMessages.php --wiki hiwiki
  • 18:12 Krinkle: mwscript deleteEqualMessages.php --wiki fawiki
  • 17:59 godog: rolling-restart cassandra-metrics-collector, staggered this time
  • 17:23 ostriches: deployed scap master@f823129 to cluster
  • 17:16 ostriches: deployed scap master@abe1973 to cluster
  • 16:54 godog: unblacklist 'max' cassandra metrics and restart cassandra-metrics-collector
  • 16:54 moritzm: installed openjdk security updates on zookeeper hosts
  • 16:18 logmsgbot: aude@tin Synchronized wmf-config/Wikibase.php: add settings for displaying labels on test.wikidata in mobile (duration: 00m 18s)
  • 16:15 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: enable wikibase descriptions on test.wikidata (duration: 00m 17s)
  • 15:46 andrewbogott: testing the bot after the nfs move
  • 15:39 logmsgbot: legoktm@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/modules/ext.wikimediaEvents.search.js: https://gerrit.wikimedia.org/r/#/c/249405/ (duration: 00m 18s)
  • 14:22 kart_: T112626 Finished running fix-stats.php for CX (from rwwiki to zuwiki)
  • 14:21 moritzm: repooled restbase1006
  • 14:09 moritzm: depooled restbase1006 for kernel update
  • 14:05 moritzm: repooled restbase1005
  • 13:54 moritzm: depooled restbase1005 for kernel update
  • 13:47 moritzm: repooled restbase1004
  • 13:35 moritzm: depooled restbase1004 for kernel update
  • 13:34 moritzm: repooled restbase1003
  • 13:16 moritzm: depooled restbase1003 for kernel update
  • 13:12 moritzm: repooled restbase1002
  • 12:56 moritzm: depooled restbase1002 for kernel update
  • 11:46 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1022 after maintenance (duration: 00m 19s)
  • 10:39 moritzm: updated kernel on restbase1001 to latest 3.19
  • 10:37 _joe_: manually removed crontab from mw1152, erroneously created by puppet
  • 10:11 _joe_: reimaging mw1152
  • 09:15 _joe_: preparing to reimage mw1152, disabling puppet, scheduling downtime.
  • 06:22 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Oct 28 06:22:06 UTC 2015 (duration 22m 5s)
  • 05:15 legoktm: ran mwscript updateSpecialPages.php --wiki=testwiki --only=GadgetUsage on terbium
  • 03:03 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-10-28 03:03:34+00:00
  • 03:00 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 06m 00s)
  • 02:39 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-28 02:39:23+00:00
  • 02:34 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 33s)
  • 00:55 yurik: deployed graphoid - https://gerrit.wikimedia.org/r/#/c/249324/
  • 00:37 logmsgbot: tgr@tin Finished scap: Updating MediaViewer with r246112 (duration: 43m 18s)
  • 00:35 mutante: mw1135 temp. unresponsive - OOM killer killing hhvm
  • 00:27 mutante: powercycling unresponsive mw1127

2015-10-27

  • 23:53 logmsgbot: tgr@tin Started scap: Updating MediaViewer with r246112
  • 23:13 ori: running sync-common on mw2050, mw2128, mw2187 and mw2209 (cf I324134438955c7)
  • 23:12 gwicke: ran `sudo mdadm --readwrite md1` on restbase1007 to resolve `pending` state
  • 23:11 gwicke: rebooted restbase1007 to rule out a funky hardware state causing elevated read latencies
  • 23:06 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/248639/ (duration: 00m 17s)
  • 23:05 logmsgbot: krenair@tin Synchronized wikiversions-labs.json: https://gerrit.wikimedia.org/r/#/c/248639/ (duration: 00m 17s)
  • 23:05 logmsgbot: krenair@tin Synchronized dblists/all-labs.dblist: https://gerrit.wikimedia.org/r/#/c/248639/ (duration: 00m 18s)
  • 23:01 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/249306, no-op (duration: 00m 18s)
  • 22:39 mutante: powercycling unresponsive analytics1039, here's what i saw on mgmt https://phabricator.wikimedia.org/P2248
  • 21:50 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.4/includes/MovePage.php: 7a7c7b27d6c (duration: 00m 17s)
  • 21:50 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/: Fix for cache key bug (duration: 00m 20s)
  • 21:29 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.4
  • 21:29 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/includes/debug/logger/LoggerFactory.php: I437bcb532: LoggerFactory: Only check for Psr\Log\LoggerInterface once (duration: 00m 18s)
  • 19:46 logmsgbot: twentyafterfour@tin Finished scap: sync everything for 1.27.0-wmf.4 and point testwiki to the new branch (duration: 31m 32s)
  • 19:44 bd808: Ran sync-common on mw2187 to rebuild l10n caches
  • 19:38 ejegg: enabled donation queue consumer
  • 19:15 logmsgbot: twentyafterfour@tin Started scap: sync everything for 1.27.0-wmf.4 and point testwiki to the new branch
  • 19:09 awight: updated DjangoBannerStats from 57a0392b3f43b65050b01a0465e120ed609a769e to 12e819a04a40ee6fab5dd55fcaf072661df31106
  • 17:56 paravoid: updating jessie debian-installer to 20150422+deb8u2
  • 17:50 Jeff_Green: add fundraising-banner-logger hosts to icinga/nsca
  • 17:40 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.3/extensions/NavigationTiming/modules/ext.navigationTiming.js: I95db9deefe363a65 (duration: 00m 17s)
  • 17:37 ejegg: disabled donation queue consumer to build up queue for benchmark
  • 17:34 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/resources/src/mediawiki.ui: I54c195541: Get rid of CSS transitions on form elements in mediawiki.ui (duration: 00m 17s)
  • 16:07 Jeff_Green: round of fundraising OS updates, occasional icinga noise is expected
  • 16:02 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.3/resources/src/mediawiki/mediawiki.ForeignStructuredUpload.BookletLayout.js: SWAT: mw.ForeignStructuredUpload: Mark description as being in source wikis content language gerrit:249081 (duration: 00m 17s)
  • 15:41 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor in the "Projet" namespace on the French Wikipedia gerrit:248910 (duration: 00m 17s)
  • 15:15 godog: reenable puppet on graphite1001
  • 15:01 kart_: TT112626 Ran fix-stats.php for CX (from bewiki to ruwiki)
  • 13:00 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/249107/ (duration: 00m 18s)
  • 12:59 akosiaris: disabling puppet and bringing down OTRS service on mendelevium
  • 12:21 jynus: Just dropped msg_resource tables from labs dbs. Filters modified to stop replicationg them. Started replicating the heartbeat tables.
  • 11:56 godog: reimage restbase-test2003
  • 11:23 godog: cassandra OOM'd on restbase1007, restarting
  • 10:57 godog: downtime restbase endpoints health for restbase1* while investigating
  • 10:56 hashar: Jenkins job https://integration.wikimedia.org/ci/job/operations-puppet-doc/ is broken. I am on it :-(
  • 10:50 hashar: stopping Jenkins due to an unclean state
  • 10:11 jynus: disabling puppet and restarting mysql servers at db1069- this will create a small amount of lag on labs
  • 09:55 godog: convert restbase-test2003 to cassandra multi-instance
  • 09:08 logmsgbot: aude@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: Add forceParse UpdaterFlag and option in forceSearchIndex script (duration: 00m 19s)
  • 05:47 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Oct 27 05:47:02 UTC 2015 (duration 47m 1s)
  • 03:42 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.3/languages/Language.php: hotfix for T116693 (duration: 00m 19s)
  • 02:29 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-27 02:29:08+00:00
  • 02:24 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 06s)
  • 01:02 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.3/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: T116693 (duration: 00m 19s)
  • 00:42 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/extensions/AbuseFilter: I2f84cff0: Avoid pointless range scan for 'load-recent-authors' (T116557) (duration: 00m 18s)
  • 00:11 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/includes/Parsoid/Utils.php: https://gerrit.wikimedia.org/r/#/c/249026 (duration: 00m 18s)

2015-10-26

  • 23:29 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/248371/ (duration: 00m 20s)
  • 23:22 logmsgbot: krenair@tin Synchronized wmf-config/extension-list-labs: https://gerrit.wikimedia.org/r/#/c/248632/ (duration: 00m 17s)
  • 23:19 logmsgbot: krenair@tin Synchronized w/static/images/project-logos/vecwiki.png: https://gerrit.wikimedia.org/r/#/c/248633/ (duration: 00m 18s)
  • 23:17 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/248478/ (duration: 00m 17s)
  • 23:16 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/248475/ (duration: 00m 17s)
  • 23:15 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/248475/ (duration: 00m 17s)
  • 23:06 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/248871/3 (duration: 00m 18s)
  • 22:39 ejegg: updated payments from 7fd1e880b6a45d79fc305b39f3fee2f324324136 to 4baa5a66a4510414d6b43b59f1b1cda2341c17fd
  • 21:48 cmjohnson1: powering off wdqs1001 to update idrac settings
  • 21:33 ejegg: update payments from 71d2d927f4efac1a639aaf7627c765f48d1b129c to 7fd1e880b6a45d79fc305b39f3fee2f324324136
  • 20:50 subbu: deployed parsoid version 660c59a9
  • 20:21 ebernhardson: started copy of eqiad elasticsearch indices to noeblium
  • 20:19 YuviPanda: stress test on nobelium complete, CPU temperature didn't go above 65C
  • 20:13 paravoid: deactivating ulsfo<->NTT BGP peering due to upcoming network migration
  • 20:05 YuviPanda: running stress on nobelium
  • 19:36 cmjohnson1: swapped bad disk on db1030
  • 15:44 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/WikimediaEvents.php: Re-deploy WME changes after deploying necessary CirrusSearch change first (duration: 00m 18s)
  • 15:43 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/modules: Re-deploy WME changes after deploying necessary CirrusSearch change first (duration: 00m 17s)
  • 15:31 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: Undeploy eventlogging search schema from CirrusSearch (duration: 00m 18s)
  • 15:21 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents: rollback (duration: 00m 18s)
  • 15:13 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/WikimediaEvents.php: Move search schema from cirrussearch -> wikimediavents (duration: 00m 19s)
  • 15:13 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/modules/: Move search schema from cirrussearch -> wikimediavents (duration: 00m 17s)
  • 15:06 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/WikimediaEvents.php: Update satisfaction schema id due to bad varnish caching of old id (duration: 00m 17s)
  • 15:04 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Remove Predundant Page and Index namespaces from $wgContentNamespaces (duration: 00m 17s)
  • 14:58 bblack: repooling cp1059 varnish mobile frontend (wiped)
  • 14:18 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable GeoData on Wikidata (duration: 00m 17s)
  • 14:09 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable geosearch on test.wikidata (duration: 00m 17s)
  • 13:32 logmsgbot: aude@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: Add justMapping option to updateOneSearchIndexConfig script (updated submodule) (duration: 00m 18s)
  • 13:27 logmsgbot: aude@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: Add justMapping option to updateOneSearchIndexConfig script (duration: 00m 18s)
  • 09:42 dcausse: deleting unused elasticsearch indices in eqiad (T112863)
  • 09:20 _joe_: restarting etcd on conf1001
  • 09:16 jynus: rebooting and installing jessie on db2060-db2070
  • 05:48 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Oct 26 05:48:19 UTC 2015 (duration 48m 18s)
  • 02:31 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-26 02:31:46+00:00
  • 02:27 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 14s)

2015-10-25

  • 05:36 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Oct 25 05:36:33 UTC 2015 (duration 36m 32s)
  • 02:28 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-25 02:28:08+00:00
  • 02:23 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 07m 59s)

2015-10-24

  • 19:34 twentyafterfour: deployed https://gerrit.wikimedia.org/r/#/c/248638/ and restarted apache on iridium
  • 06:00 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Oct 24 06:00:53 UTC 2015 (duration 0m 52s)
  • 05:44 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/includes/Data/Index/FeatureIndex.php: (no message) (duration: 00m 17s)
  • 05:11 hoo: Set an email address for user "Ymnes", after request. Confirmed by several, including.
  • 02:37 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-24 02:37:34+00:00
  • 02:33 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 23s)
  • 00:21 ebernhardson: started second machine (nobelium) performing copy of elasticsearch indices to codfw with 40 threads

2015-10-23

  • 23:31 Krinkle: mwscript deleteEqualMessages.php --wiki hewikisource
  • 23:31 Krinkle: mwscript deleteEqualMessages.php --wiki ukwiki
  • 23:09 hoo: Started a local rename for Stefan4 to Stefan2 on commons, per request (and attributed to) DerHexer
  • 22:23 robh: powercycled nobelium
  • 21:24 mutante: nobelium - powercycled, no console output
  • 20:04 jynus: rename user job had created lag on almost all enwiki dbs, things should be better now
  • 19:08 urandom: starting nodetool cleanup on restbase-test2001-b
  • 19:07 urandom: starting nodetool cleanup on restbase-test2001-a
  • 18:16 godog: bounce grafana-server on krypton
  • 17:58 YuviPanda: moved mounts around on nobelium, mounted bigger disk on /var/lib/elasticsearch
  • 17:57 godog: delete wikimedia-grid grafana dashboard (saved a copy first) heavy graphite queries
  • 17:53 YuviPanda: stopping elasticsearch on nobelium for https://phabricator.wikimedia.org/T114856
  • 17:13 akosiaris: enable_notifications=0 in neon's icinga for a few mins while the storm dies down
  • 17:04 cwd: updated payments from 4d38158f3d3a3a6da85d809a6cdc557e46c45d0c to 71d2d927f4efac1a639aaf7627c765f48d1b129c
  • 16:47 godog: stop puppet on graphite1001 and graphite-index cron, suspected root cause
  • 16:26 godog: uwsgi timing out while serving requests, bounce also carbon daemons
  • 16:20 godog: bounce uwsgi for graphite-web on graphite1001
  • 16:14 urandom: starting rebuild of restbase-test2002-b
  • 15:42 jynus: copied wmf-mariadb10 (10.0.16-2) .deb from trusty to jessie on apt.wikimedia.org
  • 15:42 urandom: bouncing restbase-test2002-a Just To See
  • 15:06 urandom: running nodetool cleanup on restbase-test2003
  • 14:47 godog: remove outdated cassandra metrics from graphite2001
  • 14:27 godog: remove restbase-test2001 restbase-test2002 cassandra metrics
  • 14:12 mobrovac: restbase rolling-restarting after applying aab840f
  • 13:55 jynus: Rebooting and installing jessie on db2055-db2059
  • 13:45 godog: remove 98percentile 999percentile meanRate stddev cassandra metrics after https://gerrit.wikimedia.org/r/#/c/248313/
  • 13:34 godog: remove 15MinuteRate cassandra metrics after https://gerrit.wikimedia.org/r/#/c/248313/
  • 11:51 godog: roll-restart cassandra-metrics-collector on restbase cluster after https://gerrit.wikimedia.org/r/248313
  • 10:27 jynus: retrying schema change on EventLogging (db1046) after failure
  • 10:21 jynus: End of online schema change to geo_tags; all wikis on dblist have been updated
  • 09:34 godog: reimage restbase-test2002.codfw.wmnet
  • 09:23 jynus: restarting schema change on geo_tags for all wikis
  • 09:17 mobrovac: restbase rolling-restart after config changes
  • 06:55 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/extensions/AbuseFilter/AbuseFilterTokenizer.php: I65d4c6064: Track tokenizer cache hits / misses (duration: 00m 18s)
  • 06:55 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/extensions/AbuseFilter/AbuseFilterTokenizer.php: I65d4c6064: Track tokenizer cache hits / misses (duration: 00m 17s)
  • 05:53 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Oct 23 05:53:43 UTC 2015 (duration 53m 42s)
  • 03:15 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.3/includes/profiler/TransactionProfiler.php: 5ef4a91480ea (duration: 00m 18s)
  • 02:41 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-23 02:41:25+00:00
  • 02:36 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 04s)
  • 00:10 logmsgbot: krenair@tin Synchronized w/static/images/project-logos/anwiki.png: https://gerrit.wikimedia.org/r/#/c/247253/ (duration: 00m 16s)

2015-10-22

  • 23:58 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/247850/ (duration: 00m 17s)
  • 23:49 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/VisualEditor/modules/ve-mw/ui/dialogs/ve.ui.MWMediaDialog.js: https://gerrit.wikimedia.org/r/#/c/248273/ (duration: 00m 18s)
  • 23:13 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/container.php: https://gerrit.wikimedia.org/r/#/c/248229/ (duration: 00m 17s)
  • 23:13 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/autoload.php: https://gerrit.wikimedia.org/r/#/c/248229/ (duration: 00m 17s)
  • 23:12 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/includes/SpamFilter/RateLimits.php: https://gerrit.wikimedia.org/r/#/c/248229/ (duration: 00m 17s)
  • 22:55 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/extensions/Cite: extensions/Cite update which fell off the SWAT train yesterday (duration: 00m 19s)
  • 22:02 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaMaintenance/getJobQueueLengths.php: Ie95ec067da9: getJobQueueLengths: add '--report' option for StatsD reporting (duration: 00m 18s)
  • 22:02 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/extensions/WikimediaMaintenance/getJobQueueLengths.php: Ie95ec067da9: getJobQueueLengths: add '--report' option for StatsD reporting (duration: 00m 18s)
  • 21:11 logmsgbot: reedy@tin Synchronized docroot and w: Add more dblist symlinks (duration: 00m 18s)
  • 21:10 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.3/includes/changetags/ChangeTags.php: e7126ed331109 (duration: 00m 17s)
  • 21:05 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.3/includes: a6262272c9666d (duration: 00m 23s)
  • 20:39 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.3/extensions/ZeroBanner/: Deploying ZeroBanner T116309 patch 248239 (duration: 00m 18s)
  • 20:38 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.3/extensions/MobileFrontend: Deploying MobileFrontend T116309 patch 248238 (duration: 00m 36s)
  • 20:26 bd808: Removed "zirconium.wikimedia.org" from Trebuchet's minion list for iegreview/iegreview
  • 20:12 bd808: Updated iegreview.wikimedia.org to bcaf23b (Fix logger usage in Controllers\Account\Recover)
  • 19:53 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.3/extensions/ZeroBanner: Deploying ZeroBanner T116309 patch 248116 (duration: 00m 18s)
  • 19:28 bd808: Forced ELK Elasticsearch to allocate replica of logstash-2015.10.22 shard 0 on logstash1004
  • 18:56 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: all wikis to 1.27.0-wmf.3
  • 16:48 bd808: Restarted Elasticsearch on logstash1004
  • 16:39 logmsgbot: kartik@tin Synchronized private/PrivateSettings.php: T116134: Set CX JWT token (duration: 00m 17s)
  • 16:19 logmsgbot: ori@tin Synchronized wmf-config/InitialiseSettings.php: I8f690589: Increase abusefilter emergency disable threshold on MediaWiki.org (duration: 00m 17s)
  • 15:30 logmsgbot: thcipriani@tin Synchronized docroot/noc/conf/highlight.php: SWAT: Remove urlencode from phabricator links gerrit:248049 (duration: 00m 17s)
  • 15:13 logmsgbot: thcipriani@tin Synchronized docroot/noc/conf: SWAT: noc: change Gitblit links to Diffusion gerrit:248027 (duration: 00m 17s)
  • 15:11 moritzm: uploaded openjdk-8 8u66-b17 for jessie-wikimedia to carbon
  • 14:46 jynus: performing schema change on eventlogging database on db1046
  • 14:41 mutante: mw1083 - Error: Could not run Puppet configuration client: Read-only file system
  • 14:25 jynus: setting thread_pool_size to 32 dynamically on all MariaDB hosts
  • 14:01 jynus: performing schema change on officewiki-flow (s3)
  • 13:19 jynus: performing schema change on x1-master (flowdb)
  • 12:31 jynus: Rolling schema change for GeoData on all wikis (geo_tags)
  • 11:35 mobrovac: restbase deployed 2bc05f40
  • 07:41 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/includes/jobqueue/JobQueueRedis.php: Ie7c544fc8: jobqueue: track real job inserts as inserts_actual & I627e8f6ce: JobQueueRedis::doBatchPush(): report metrics even when failures occur (duration: 00m 17s)
  • 07:41 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/includes/jobqueue/JobQueueRedis.php: Ie7c544fc8: jobqueue: track real job inserts as inserts_actual & I627e8f6ce: JobQueueRedis::doBatchPush(): report metrics even when failures occur (duration: 07m 33s)
  • 06:17 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Oct 22 06:17:39 UTC 2015 (duration 17m 38s)
  • 06:14 AaronSchulz: Restarted hhvm on mw1011, it was stuck doing nothing at 100 cpu
  • 05:10 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.3/includes/deferred/LinksUpdate.php: fe323f9b68bbb (duration: 00m 17s)
  • 03:05 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-22 03:05:21+00:00
  • 03:00 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 07m 52s)
  • 02:37 AaronSchulz: Started running 5 threads of enwiki refreshLinks jobs on tin
  • 02:35 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-22 02:35:41+00:00
  • 02:30 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 44s)
  • 01:10 logmsgbot: krenair@mira Synchronized README: testing sync from mira (duration: 00m 17s)
  • 01:01 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/includes/Hooks.php: I0e5f2d3b2: Make hookErrorHandler() only care about serious signature errors (duration: 00m 17s)
  • 01:00 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/includes/Hooks.php: I0e5f2d3b2: Make hookErrorHandler() only care about serious signature errors (duration: 00m 17s)
  • 00:56 logmsgbot: krenair@mira Synchronized README: (no message) (duration: 00m 16s)
  • 00:55 logmsgbot: krenair@mira Synchronized README: (no message) (duration: 00m 17s)
  • 00:49 logmsgbot: krenair@mira Synchronized README: (no message) (duration: 00m 16s)
  • 00:20 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: Ibf752b832: $wgMathCheckFiles = false (duration: 00m 18s)
  • 00:05 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/extensions/AbuseFilter: Ice1b6da43: AbuseFilter: don't install custom error handler and I0ecdcdd142: Use isset() to check array element exists rather than relying on @ operator (duration: 00m 18s)
  • 00:03 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/extensions/AbuseFilter: Ice1b6da43: AbuseFilter: don't install custom error handler and I0ecdcdd142: Use isset() to check array element exists rather than relying on @ operator (duration: 00m 18s)

2015-10-21

  • 23:33 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/CirrusSearch: Performance tweaks for corss-dc copy process (duration: 00m 19s)
  • 23:32 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: Performance tweaks for corss-dc copy process (duration: 00m 18s)
  • 23:25 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/resources/: mw.ForeignStructuredUpload: Provide category suggestions from the right wiki (duration: 00m 17s)
  • 23:16 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/resources/: mw.ForeignStructuredUpload: Rearrange messages to always display license name (duration: 00m 18s)
  • 23:09 logmsgbot: ebernhardson@tin Synchronized wmf-config/throttle.php: Add throtle exception for eswiki (duration: 00m 18s)
  • 23:03 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/extensions/AbuseFilter/AbuseFilter.parser.php: Ad-hoc debug logging of AbuseFilter exceptions (duration: 00m 17s)
  • 23:00 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/extensions/AbuseFilter/AbuseFilter.parser.php: Ad-hoc debug logging of AbuseFilter exceptions (duration: 00m 17s)
  • 21:10 ebernhardson: starting copy of elasticsearch eqiad indices to codfw
  • 20:46 ebernhardson|lch: cancel copying elasticsearch eqiad to labsearch, looks to be writing to wrong disks and will fill up
  • 20:16 AaronSchulz: Started running 8 threads of commonswiki refreshlinks jobs on terbium
  • 20:14 ebernhardson|lch: starting copy of elasticsearch indices from eqiad cluster to labsearch cluster
  • 18:42 ebernhardson: initializing elasticsearch index mapping for all wikis in the codfw and labsearch ES clusters
  • 18:12 ejegg|mtg: updated crm from 22dc4bd7d041126a1d2a0d4acb9a288bfdc1b435 to f2fa7b942625b34ede520e11f20e7e0835ecb17d
  • 16:53 hoo: Attached WeiaR@enwiki to the global account of the same name. T115699
  • 15:50 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/WikimediaEvents/: Revert SearchSatisfaction schema related changes due to suspected perf impact (duration: 00m 18s)
  • 15:49 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/: Revert SearchSatisfaction schema related changes due to suspected perf impact (duration: 00m 18s)
  • 15:38 akosiaris: depooled mw1083
  • 15:29 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: touch and re-sync InitialiseSettings.php to bust cache (duration: 00m 17s)
  • 15:24 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: CirrusSearch multi-datacenter configuration (duration: 00m 17s)
  • 15:24 logmsgbot: ebernhardson@tin Synchronized wmf-config/CommonSettings.php: CirrusSearch multi-datacenter configuration (duration: 00m 17s)
  • 15:23 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-common.php: CirrusSearch multi-datacenter configuration (duration: 00m 17s)
  • 15:23 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-production.php: CirrusSearch multi-datacenter configuration (duration: 00m 17s)
  • 15:18 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/Cite/Cite_body.php: Do not double-parse error references duplicate key (duration: 00m 17s)
  • 15:16 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/Cite/Cite_body.php: Do not double-parse error references duplicate key (duration: 00m 19s)
  • 15:00 aude: rsync failed on mw1083: "failed to set times on "/srv/mediawiki/wmf-config": Read-only file system"
  • 14:49 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable GeoData on test.wikidata (duration: 00m 18s)
  • 14:34 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings-labs.php: enabled geodata on beta wikidata (duration: 00m 18s)
  • 14:08 jynus: applying schema change to testwikidatawiki (s3) and wikidatawiki(s5)
  • 13:56 jynus: performing schema change on testwiki.geo_tags
  • 13:35 jynus: stopping and fixing replication on labsdb1004 (not in production)
  • 08:40 mobrovac: restbase deployment of 3006b77e Yes check.svg Done
  • 08:32 mobrovac: restbase deploying 3006b77e
  • 07:36 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Oct 21 07:36:04 UTC 2015 (duration 36m 3s)
  • 03:13 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-21 03:13:42+00:00
  • 03:09 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 24s)
  • 02:42 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-21 02:42:44+00:00
  • 02:37 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 05s)
  • 01:12 logmsgbot: hoo@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 17s)
  • 01:03 logmsgbot: hoo@tin Synchronized wmf-config/interwiki.cdb: revert (duration: 00m 18s)
  • 01:03 logmsgbot: hoo@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 17s)
  • 00:29 logmsgbot: legoktm wikidata'd everything
  • 00:22 logmsgbot: hoo@tin Synchronized dblists/: Enable WikibaseClient on mediawikiwiki, metawiki and specieswiki (duration: 00m 17s)
  • 00:19 logmsgbot: hoo@tin Synchronized wmf-config/: Enable WikibaseClient on mediawikiwiki, metawiki and specieswiki (duration: 00m 19s)

2015-10-20

  • 23:54 logmsgbot: hoo@tin Synchronized wmf-config/: Add MediaWiki, Meta-Wiki and Wikispecies to Wikibase special site groups (duration: 00m 18s)
  • 23:47 logmsgbot: hoo@tin Synchronized wmf-config/: Add MediaWiki, Meta-Wiki and Wikispecies to Wikibase special site groups (testwikidata) (duration: 00m 18s)
  • 23:36 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/WikimediaEventsHooks.php: https://gerrit.wikimedia.org/r/#/c/247650/ (duration: 00m 17s)
  • 23:26 logmsgbot: hoo@tin Synchronized wmf-config/: Revert "Temporarily disable "item-merge" right on Wikidata" (duration: 00m 18s)
  • 23:25 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.3/includes/deferred: 2a1e1d7dd88a62aba9 (duration: 00m 17s)
  • 23:22 logmsgbot: hoo@tin Synchronized wmf-config/: Bump the cache epoch for (test)wikidata (duration: 00m 18s)
  • 23:20 logmsgbot: hoo@tin Finished scap: Update Wikibase to wmf3b and add messages for sitelinks to MediaWiki, Meta-Wiki and Wikispecies (duration: 48m 44s)
  • 23:16 mutante: mw1232: restarted hhvm
  • 22:31 logmsgbot: hoo@tin Started scap: Update Wikibase to wmf3b and add messages for sitelinks to MediaWiki, Meta-Wiki and Wikispecies
  • 21:11 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/includes/page/WikiPage.php: I5d0440588d: Make triggerOpportunisticLinksUpdate() directly use RefreshLinks (T116001) (duration: 00m 17s)
  • 21:11 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/includes/page/WikiPage.php: I5d0440588d: Make triggerOpportunisticLinksUpdate() directly use RefreshLinks (T116001) (duration: 00m 18s)
  • 20:23 gwicke: reverted restbase deploy on restbase1001 to a4c55e40
  • 20:15 mutante: starting restbase on restbase1001
  • 20:09 mutante: errors argon' on argon
  • 20:04 ottomata: uninstalling hadoop packages on analytics1017
  • 19:46 ori: previous syncs was of I46200d4edb3: Revert "Revert "Revert "Enable config for all three search clusters, but only write to eqiad"""
  • 19:45 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 18s)
  • 19:45 logmsgbot: ebernhardson@tin Synchronized wmf-config/: (no message) (duration: 00m 18s)
  • 18:25 mutante: re-enabling icinga notifications for icinga (neon) itself that were disabled for some reason though all OK
  • 16:50 mutante: temp. disabled ircecho / neon puppet
  • 16:50 jynus: enabling query profiling on a sample of queries on db1072
  • 16:27 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings.php: Revert Temporarily increase redis logging to debug (duration: 00m 18s)
  • 16:25 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings.php: Temporarily increase redis logging to debug (duration: 00m 17s)
  • 16:21 aude: re-populated sites table on metawiki, mediawikiwiki and specieswiki with https protocol links
  • 16:13 cwdent: updated fundraising crm from 738e8c3f8079765841ed4c5f79ecf066c541c7b9 to 22dc4bd7d041126a1d2a0d4acb9a288bfdc1b435
  • 15:49 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Revert "Enable config for all three search clusters, but only write to eqiad"" gerrit:247478 Part V (duration: 00m 18s)
  • 15:48 godog: bump netdev_max_backlog to 10000 on graphite1001, T101141
  • 15:46 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Revert "Enable config for all three search clusters, but only write to eqiad"" gerrit:247478 Part IV (duration: 00m 17s)
  • 15:46 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Revert "Revert "Enable config for all three search clusters, but only write to eqiad"" gerrit:247478 Part III (duration: 00m 17s)
  • 15:45 logmsgbot: thcipriani@tin Synchronized wmf-config/CirrusSearch-common.php: SWAT: Revert "Revert "Enable config for all three search clusters, but only write to eqiad"" gerrit:247478 Part II (duration: 00m 17s)
  • 15:44 logmsgbot: thcipriani@tin Synchronized wmf-config/CirrusSearch-production.php: SWAT: Revert "Revert "Enable config for all three search clusters, but only write to eqiad"" gerrit:247478 Part I (duration: 00m 17s)
  • 15:27 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Enable ContentTranslation suggestion in all Wikipedia gerrit:247515 (duration: 00m 17s)
  • 15:23 awight: update fundraising CRM from 6ad0ad090c23ee41003138a6676131abf70c72f4 to 738e8c3f8079765841ed4c5f79ecf066c541c7b9
  • 15:21 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Switch graphoid to the local restbase proxy gerrit:247494 (duration: 00m 17s)
  • 15:14 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Move ForeignUploadTargets config to production gerrit:246703 part 2 (duration: 00m 17s)
  • 15:14 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Move ForeignUploadTargets config to production gerrit:246703 part I (duration: 00m 18s)
  • 14:33 cmjohnson1: removing tele2(patchid 2953) from dmarc panel @eqiad
  • 14:12 ottomata: deployed varnishreqstats diamond collector to remaining varnish caches
  • 13:48 jynus: backing up and renaming user_daily_contribs table from all wikis as a previous step for its deletion
  • 12:44 kart_: Update cxserver to 6452b68
  • 12:01 logmsgbot: aude@tin Synchronized wmf-config/Wikibase.php: Temporarily disallow item-merge until T115892 is resolved (duration: 00m 19s)
  • 11:16 mobrovac: mathoid deploying 8e1a3327
  • 07:23 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Oct 20 07:23:56 UTC 2015 (duration 23m 55s)
  • 03:14 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-20 03:14:15+00:00
  • 03:09 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 16s)
  • 02:48 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch/: Bring phase0 and phase1 inline with phase2 (duration: 00m 18s)
  • 02:48 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/Elastica/: Bring phase0 and phase1 inline with phase2 (duration: 00m 21s)
  • 02:44 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-20 02:43:53+00:00
  • 02:39 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 20s)

2015-10-19

  • 23:43 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: testwiki Graphoid to restbase (duration: 00m 17s)
  • 23:13 logmsgbot: ebernhardson@tin Synchronized wmf-config/: resync after touching InitialiseSettings.php to bust caches (duration: 00m 18s)
  • 23:12 logmsgbot: ebernhardson@tin Synchronized wmf-config/: Revert multidc cirrussearch config, seeing unexplained errors on commonswiki (duration: 00m 18s)
  • 23:11 mutante: restarted hhvm on mw1231
  • 23:09 logmsgbot: ebernhardson@tin Synchronized wmf-config/throttle.php: Add throttle exception for dewiki (duration: 00m 17s)
  • 23:08 mutante: restarted apache on mw1231
  • 23:07 logmsgbot: ebernhardson@tin Synchronized wmf-config/: Disable cirrus suggester AB test (duration: 00m 17s)
  • 23:05 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/WikimediaEvents/: Bump sampling rate of common terms test from 1:1000 to 1:200 (duration: 00m 17s)
  • 23:01 logmsgbot: ebernhardson@tin Synchronized tests/: noop sync mediawiki-config test dir (duration: 00m 17s)
  • 22:42 logmsgbot: ebernhardson@tin Synchronized wmf-config/: Enable cirrusearch multi cluster configuration, only write to eqiad (duration: 00m 18s)
  • 22:40 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/CirrusSearch/: Handle ElasticaWrite job failures internally (duration: 00m 18s)
  • 22:29 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/CirrusSearch/: Deploy multi-dc cirrusearch code for CirrusSearch extension (duration: 00m 18s)
  • 22:28 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/Elastica/: Deploy multi-dc cirrusearch code for Elastica extension (duration: 00m 17s)
  • 22:20 ebernhardson: sync-common on mw1017 to pre-test cirrussearch multi-dc deployment
  • 22:17 paravoid: salt-run deluser --delete-home gmetric; delgroup systemusers
  • 20.23 subbu: cherrypick deploy for parsoid completed: b317f33f and 60a82ae0 cherrypicked from parsoid master
  • 15:42 logmsgbot: anomie@tin Started scap: SWAT: Add a change tag to cross-wiki uploads gerrit:246701
  • 15:39 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.2/extensions/Cite: SWAT: Display 'cite_error_references_duplicate_key' next to the affected ref gerrit:247255 (duration: 00m 18s)
  • 15:30 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.3/extensions/Cite: SWAT: Display 'cite_error_references_duplicate_key' next to the affected ref gerrit:247256 (duration: 00m 18s)
  • 15:25 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.2/includes/deferred/LinksDeletionUpdate.php: SWAT: Use specified pageId for LinksDeletionUpdate→DeleteLinksJob gerrit:247268 (duration: 00m 18s)
  • 15:24 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.3/includes/deferred/LinksDeletionUpdate.php: SWAT: Use specified pageId for LinksDeletionUpdate→DeleteLinksJob gerrit:247267 (duration: 00m 17s)
  • 15:12 urandom: deploying a4c55e40 to RESTBase
  • 15:11 joal: Scheduling icinga downtime for CQL checks on aqs while heavily loading data - joal (me) babysites the jobs - 1 day downtime, will reiterate tomorrow if needed
  • 15:09 ottomata: enabling varnish reqstats diamond collector on all upload caches
  • 15:09 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/includes/Search/Connection.php: SWAT: Backport gerrit:246134 because the thing it fixed suddenly started breaking unit tests, preventing other merges (duration: 00m 18s)
  • 15:07 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.2/extensions/Flow/includes/Search/Connection.php: SWAT: Backport gerrit:246134 because the thing it fixed suddenly started breaking unit tests, preventing other merges (duration: 00m 18s)
  • 14:52 akosiaris: disabled puppet on maps-test200{1,2,4}. Debugging cassandra multi-instance setup aftermath. Not to be enabled
  • 14:34 urandom: canary deploy (a4c55e40) to restbase1001.eqiad
  • 12:47 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-isl-eng_0.1.0~r20599-1
  • 12:22 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-isl_0.1.0-1
  • 06:19 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Oct 19 06:19:09 UTC 2015 (duration 19m 8s)
  • 03:04 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-19 03:04:49+00:00
  • 02:59 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 40s)
  • 02:36 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-19 02:36:53+00:00
  • 02:32 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 15s)

2015-10-18

  • 10:04 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-mlt-ara_0.1.0~r57554-1
  • 10:04 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-is-sv_0.1.0~r56030-1
  • 10:04 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-es-ro_0.7.3~r57551-1
  • 10:04 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-es-it_0.1.0~r51165-1
  • 06:11 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Oct 18 06:11:37 UTC 2015 (duration 11m 36s)
  • 02:57 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-18 02:57:48+00:00
  • 02:52 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 31s)
  • 02:29 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-18 02:29:50+00:00
  • 02:25 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 25s)

2015-10-17

  • 20:02 godog: powercycle analytics1034, no console no ssh
  • 14:05 godog: reboot krypton, unable to ssh and no console (VM) iowait through the roof
  • 06:27 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Oct 17 06:27:04 UTC 2015 (duration 27m 3s)
  • 03:03 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-17 03:02:58+00:00
  • 02:58 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 14s)
  • 02:35 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-17 02:35:14+00:00
  • 02:30 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 09s)

2015-10-16

  • 20:52 hashar: Restarting Jenkins to remove potential dead locks before the week-end
  • 20:48 ejegg: updated payments from 33b3bd6bee11b3cc9de1570584a23354d0b6525f to 4d38158f3d3a3a6da85d809a6cdc557e46c45d0c
  • 19:29 ori: deleted /var/lib/carbon/whisper/MediaWiki/MediaWiki on graphite1001 & graphite2001 per tgr's request
  • 16:14 awight: Update paymentswiki logging config per T107918
  • 15:52 awight: Fundraising: remove FR fraud exception
  • 13:18 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I14ecd0ae87: Turn off UserDailyContribs extension (duration: 00m 18s)
  • 06:25 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Oct 16 06:25:56 UTC 2015 (duration 25m 55s)
  • 03:11 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-16 03:11:14+00:00
  • 03:06 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 31s)
  • 02:43 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-16 02:43:42+00:00
  • 02:38 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 38s)
  • 02:27 jynus: repairing table and restarting replication on s7 from labsdb1002
  • 02:23 jynus: repairing table and restarting replication on s5 from dbstore1002 (non-production host)
  • 01:36 Jamesofur: reset email address for User:INeverCry after identify verification

2015-10-15

  • 22:09 greg-g: don't worry, _joe_ was around and we approved Roan's last deploy as an exception :)
  • 22:08 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/: Back out categories in sidebar feature (duration: 00m 20s)
  • 20:30 _joe_: rebooting ms-be1011
  • 19:41 awight: update crm from ddfaa209f5b5af4aa0cf3403da91d39b3c52acc1 to a4e74f6f38e6cff16ecf79d28d6e4de9499a5017
  • 19:08 cwdent: updated payments from bdec3220030a396e2a447763e40b940a332e2ab8 to 33b3bd6bee11b3cc9de1570584a23354d0b6525f
  • 18:41 ejegg: updated refund queue name in IPN listener settings
  • 18:13 ejegg: updated CiviCRM from 17dc351d92a8437c93b4a9fa2385840b3581dad6 to ddfaa209f5b5af4aa0cf3403da91d39b3c52acc1
  • 17:18 cwdent: updated payments from ba8f80ec7a858074cd3856a52f3758cf96571f67 to bdec3220030a396e2a447763e40b940a332e2ab8
  • 09:54 _joe_: restarted gitblit, unresponsive
  • 05:41 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Oct 15 05:41:25 UTC 2015 (duration 41m 24s)
  • 02:51 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-15 02:51:27+00:00
  • 02:49 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 05m 02s)
  • 02:35 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-15 02:35:15+00:00
  • 02:31 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 06m 57s)

2015-10-14

  • 23:46 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: Nevermind on skipping timed text, extension config is weird (duration: 00m 17s)
  • 22:26 akosiaris: restarted zotero on sca1002
  • 21:54 akosiaris: restarted zotero on sca1001. complaining out of memory in logs
  • 20:46 ejegg: updated civicrm from ba39f3181431c8409416e580dcf6e15ac5f96a21 to 17dc351d92a8437c93b4a9fa2385840b3581dad6
  • 20:20 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: Also exclude timedtext ns from purges (duration: 00m 18s)
  • 19:52 robh: reenabled puppet agent on etherpad1001, was disabled for a few hours and no reason specified and no SAL entry
  • 19:50 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Turn off the mw "warning" logging channel (duration: 00m 18s)
  • 19:21 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: Tighten check on empty page purges (duration: 00m 18s)
  • 19:11 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I1d7969a9d: Purge pages with blank content beyond the PP limit report, I38cba9d9b6a: Get strpos() parameters correct (duration: 00m 17s)
  • 19:10 logmsgbot: ori@tin Synchronized wmf-config/InitialiseSettings.php: I3a8a750bb3: Enable T115505 log channel for I1d7969a (duration: 00m 17s)
  • 18:06 logmsgbot: demon@tin Synchronized wmf-config: (no message) (duration: 00m 19s)
  • 17:59 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 17s)
  • 17:58 logmsgbot: demon@tin Synchronized wmf-config/extension-list: (no message) (duration: 00m 17s)
  • 15:31 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: SWAT: Split connection to source and target gerrit:246243 (duration: 00m 18s)
  • 15:19 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.2/extensions/UploadWizard/UploadWizard.config.php: SWAT: Remove default category for UploadWizard files gerrit:246225 (duration: 00m 17s)
  • 15:14 ori: restarted navtiming and statsv services on hafnium
  • 15:11 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.3/extensions/UploadWizard/UploadWizard.config.php: SWAT: Remove default category for UploadWizard files gerrit:246226 (duration: 00m 18s)
  • 15:05 Krinkle: Statsv and eventlogging-navtiming seems to have gone down 7 hours ago
  • 14:24 ori: cleaned up tessera leftovers on graphite1001
  • 14:15 mutante|1way: mw1157 - deleted puppet lock file, fix puppet run. ("already running" but didnt since 18h)
  • 13:30 ottomata: kafka-preferred-replica election after kafka1012's broker restarted last night
  • 13:12 ottomata: restarted diamond, puppet didn't seem to after it removed the TcpConnStates from most hosts
  • 12:55 hashar: ERROR: Could not connect to SMTP host: polonium.wikimedia.org, port: 25 (from labs instances)
  • 12:32 logmsgbot: krenair@tin Synchronized README: testing https://gerrit.wikimedia.org/r/246206 (duration: 00m 17s)
  • 12:07 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/VisualEditor/modules/ve-mw/ui/inspectors/ve.ui.MWLinkAnnotationInspector.js: https://gerrit.wikimedia.org/r/#/c/246205/ (duration: 01m 13s)
  • 03:14 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Oct 14 03:14:22 UTC 2015 (duration 14m 21s)
  • 03:14 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-14 03:14:22+00:00
  • 03:07 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 10m 37s)
  • 02:40 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-14 02:40:43+00:00
  • 02:37 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 07m 10s)
  • 00:19 ejegg: updated civicrm from c7af7634e75eb8702f5e16081a86ab86ce69c7c2 to ba39f3181431c8409416e580dcf6e15ac5f96a21

2015-10-13

  • 23:30 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Set $wgUploadNavigationUrl to use uselang=$lang for commonsuploads wikis by default (duration: 01m 14s)
  • 23:27 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/WikimediaEvents/: Turn on cirrus common terms test (duration: 01m 15s)
  • 23:25 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/CirrusSearch/: Add information for common terms ab test (duration: 01m 15s)
  • 23:21 Krenair: restarted apache on silver
  • 23:19 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/Flow: Make flow board descriptions editable again (duration: 01m 16s)
  • 23:13 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/Flow: Bump flow submodule in 1.27.0-wmf.3 (duration: 01m 15s)
  • 23:10 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/: Update WME for common terms AB test (duration: 01m 14s)
  • 23:08 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch/: Update cirrus for common terms AB tes (duration: 01m 15s)
  • 23:02 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Turn on logging of the "warning" channel (duration: 01m 13s)
  • 23:00 ejegg: updated SmashPig from b5ff2a7d5f17aaaa33a169ca101cbea639769c90 to c431c8d77521270236c72532a50806b2e852cf7b
  • 21:05 ottomata: reenabling puppet on cp1052
  • 20:51 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.3
  • 20:48 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/includes/db: I0e5f2d3b2: Revert Enforce lagged-slave read-only mode on the DB layer (duration: 01m 14s)
  • 20:45 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/autoload.php: (no message) (duration: 01m 14s)
  • 20:40 akosiaris: restart gitblit (once more)
  • 20:27 _joe_: rebooting mw1157, stuck in a kernel soft lockup
  • 20:22 ori: locally hacked testwiki and mediawikiwiki to point to php-1.27.0-wmf.3 on mw1017
  • 20:19 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.2
  • 20:18 logmsgbot: ori@tin rebuilt wikiversions.cdb and synchronized wikiversions files: (no message)
  • 20:00 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.3
  • 19:59 ejegg: updated civicrm f25fbe856b92373104985185db77311ea3a4d841 to c7af7634e75eb8702f5e16081a86ab86ce69c7c2
  • 19:38 JohnFLewis: delete wikidata-l mailing list (archives still accessible)
  • 19:30 logmsgbot: twentyafterfour@tin Finished scap: Full scap sync for 1.27.0-wmf.3 (duration: 31m 26s)
  • 18:58 logmsgbot: twentyafterfour@tin Started scap: Full scap sync for 1.27.0-wmf.3
  • 18:45 ottomata: FYI i have puppet disabled on cp1052 while I try to figure out a diamond+VSL+multiprocessing bug
  • 18:11 mutante|1way: started salt on mw2083
  • 18:09 mutante|1way: restarted gitblit
  • 17:58 ori: testing ldap integration in grafana 2
  • 17:54 urandom: performing deploy of a4c55e4 to restbase staging
  • 17:42 urandom: performing canary deploy of a4c55e4 to restbase staging (xenon)
  • 16:14 ostriches: created education program tables for srwiki, T110619
  • 16:14 mobrovac: restbase deploying a01c62a6
  • 16:08 ottomata: restarting diamond on cp1052 in gdb in attempt to figure out why vanrishreqstats segfaults...
  • 15:25 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable CX suggestions for de, fa, fi, he, nn, pa, pl and te wikipedias gerrit:245862 (duration: 01m 13s)
  • 15:18 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Set $wgUploadNavigationUrl to use uselang=$lang for commonsuploads wikis by default" (duration: 01m 13s)
  • 15:15 ori: grafana-test: Imported Grafana dashboards from ElasticSearch
  • 14:12 bblack: note to self: we should migrate all our service to Java
  • 14:11 bblack: restarted gitblit on antimony
  • 13:16 bblack: repooling cp2017 (codfw upload)
  • 02:32 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Oct 13 02:32:17 UTC 2015 (duration 32m 16s)
  • 02:32 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-13 02:32:17+00:00
  • 02:28 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 07m 01s)
  • 01:53 logmsgbot: faidon@tin Synchronized wmf-config/CommonSettings.php: unbreak BounceHandler (duration: 01m 14s)
  • 00:51 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Reapply: Flow-occupy talk namespaces on sewikimedia (duration: 01m 13s)
  • 00:29 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.2/extensions/Flow/modules/flow-initialize.js: https://gerrit.wikimedia.org/r/#/c/245596/ (duration: 01m 13s)
  • 00:21 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/244724/ (duration: 01m 13s)
  • 00:10 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/245608/ (duration: 01m 13s)
  • 00:04 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: rv (duration: 01m 13s)
  • 00:00 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/245588/ and https://gerrit.wikimedia.org/r/#/c/245589/ (duration: 01m 13s)

2015-10-12

  • 23:53 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/245578/ (duration: 01m 12s)
  • 23:38 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/243920/ (duration: 01m 14s)
  • 23:35 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/243921/ (duration: 01m 13s)
  • 23:26 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/244378/ (duration: 01m 13s)
  • 23:23 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/244896/ (duration: 01m 14s)
  • 23:16 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/245194/ (duration: 01m 13s)
  • 23:11 logmsgbot: krenair@tin Synchronized dblists: https://gerrit.wikimedia.org/r/#/c/243517/ (duration: 01m 13s)
  • 23:06 logmsgbot: krenair@tin Synchronized database lists: (no message) (duration: 01m 13s)
  • 20:21 bearND: MobileApps deployed sha1 95293e5
  • 19:09 Tim: on ruthenium installed iotop for stall investigation
  • 16:13 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.2/extensions/ZeroBanner: SWAT: Defer loading of ZeroOverlay until needed gerrit:244737 (duration: 01m 13s)
  • 16:00 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: SWAT: Throttle rule for Ada Lovelace Day editathon 2015 gerrit:245472 (duration: 01m 13s)
  • 15:55 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Extension:ShortURL on bnwiki gerrit:244953 (duration: 01m 14s)
  • 15:39 logmsgbot: thcipriani@tin Synchronized dblists/commonsuploads.dblist: SWAT: Remove duplicate entries from commsuploads.dblist gerrit:244435 (duration: 01m 12s)
  • 15:33 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Use new page name for wmf release notes gerrit:241079 (duration: 01m 14s)
  • 15:26 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Rename Azerbaijani Wikisource project and namespaces gerrit:242096 (duration: 01m 13s)
  • 15:19 logmsgbot: thcipriani@tin Synchronized wmf-config/Wikibase-production.php: SWAT: Add GeoData and PageImages configuration for Wikibase repo wikis gerrit:244165 (duration: 01m 13s)
  • 15:14 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Explicitly set wmgMFNearby = false for wikidata gerrit:244591 (duration: 01m 14s)
  • 15:07 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Modify timezone for cswiktionary gerrit:244649 (duration: 01m 12s)
  • 14:54 dcausse: closing unused cirrus indices in eqiad (T112863)
  • 14:28 cmjohnson: rebooting mw1154
  • 12:10 yurik: deployed kartotherian & tilerator to maps-test200{1-4}
  • 08:54 hashar: zuul-merger process leaked file descriptors and ended up unable to open any more files. Fixed by restarting the service on gallium. https://phabricator.wikimedia.org/T115243
  • 08:44 hashar: Zuul CI in trouble. zuul-merger can't not apply patches anymore https://phabricator.wikimedia.org/T115243
  • 02:33 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Oct 12 02:33:13 UTC 2015 (duration 33m 12s)
  • 02:33 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-12 02:33:13+00:00
  • 02:30 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 06m 50s)

2015-10-11

  • 04:57 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Oct 11 04:57:40 UTC 2015 (duration 57m 39s)
  • 02:28 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-11 02:28:34+00:00
  • 02:25 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 06m 17s)

2015-10-10

  • 06:40 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Oct 10 06:40:04 UTC 2015 (duration 40m 3s)
  • 02:41 ejegg|afk: enabled fundraising banner campaigns
  • 02:36 ejegg|afk: updated payments-wiki from 24d5be6886d34b3600031290c7f55ee84f3dcee2 to ba8f80ec7a858074cd3856a52f3758cf96571f67
  • 02:22 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-10 02:22:30+00:00
  • 02:19 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 05m 58s)

2015-10-09

  • 22:20 ejegg: took down Fundraising campaigns
  • 19:33 logmsgbot: ori@tin Synchronized multiversion/MWWikiversions.php: I9d4cbd3d67: Provide a smooth migration path of dblist files to dblists/ (duration: 01m 13s)
  • 18:13 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.2/extensions/OpenStackManager/nova/OpenStackNovaProject.php: https://gerrit.wikimedia.org/r/#/c/244707/ (duration: 01m 13s)
  • 16:04 godog: rolling restart cassandra test cluster T95253
  • 15:35 urandom: performing Cassandra cleanup on restbase-test2003.codfw
  • 15:15 urandom: bouncing Cassandra on restbase-test2001-a
  • 15:07 urandom: starting nodetool cleanup on restbase-test2002
  • 14:37 jynus: more configuration testing (with puppet disabled) and several mysql restarts on db1022
  • 13:53 hashar: Restarted Zuul, had a deadlocked job
  • 11:50 godog: bounce mathoid on sca100[12], stray instance found not running firejail
  • 11:22 mobrovac: restbase deploying aaee7c31
  • 11:16 godog: force-run puppet on restbase after merging https://gerrit.wikimedia.org/r/#/c/244656/
  • 11:00 jynus: restarting db1022's mysql (depooled) for configuration testing
  • 09:17 jynus: deployed visual glitch fix to dbtree
  • 09:11 akosiaris: poweroff rhodium, remove salt key, remove puppet storedconfigs in preparation for reinstall reinstall as a VM for temporary puppetmaster testing.
  • 09:11 akosiaris: poweroff sodium, remove salt key, remove puppet storedconfigs in preparation for reinstall reinstall as a VM for temporary puppetmaster testing.
  • 08:20 ori: Purged graphite[12]001:/var/lib/carbon/whisper/servers/*/TcpConnStatesCollector and graphite[12]001:/var/lib/carbon/whisper/servers/*/network/work; cleaning up after https://gerrit.wikimedia.org/r/#/c/244637/
  • 05:13 mutante: @RoanKattouw re: sync-file ssh to mira.codfw.wmnet: fixed! sorry. -> T115075#1714464
  • 02:32 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 06m 11s)
  • 01:24 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.2/extensions/CentralNotice/resources/subscribing/ext.centralNotice.display.js: Add period to try to flush out ResourceLoader issue (duration: 01m 26s)
  • 01:18 RoanKattouw: Getting failures from sync-file / scap because mira.codfw.wmnet doesn't respond to ssh
  • 01:17 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/CentralNotice/resources/subscribing/ext.centralNotice.display.js: Add trailing newline to try to flush out ResourceLoader issue (duration: 02m 15s)

2015-10-08

  • 23:56 logmsgbot: catrope@tin Finished scap: SWAT (duration: 29m 10s)
  • 23:51 yurik: reverted Kartotherian to HEAD^^ - the service wouldn't start
  • 23:27 logmsgbot: catrope@tin Started scap: SWAT
  • 23:06 yurik: deployed kartotherian
  • 18:20 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.27.0-wmf.2
  • 16:45 mobrovac: mathoid deploying 110abaf
  • 15:12 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: Fix labs settings for foreign uploads (syncing out so it doesnt surprise future SWATters) gerrit:244332 (duration: 00m 18s)
  • 15:05 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable CX suggestions in ast, bn, ml, nb, ta and ukwiki gerrit:244142 (duration: 00m 17s)
  • 14:53 paravoid: replacing cr1-codfw<->asw-a-codfw QSFPs
  • 14:48 logmsgbot: reedy@tin Synchronized docroot and w: (no message) (duration: 00m 17s)
  • 14:13 jynus: performing schema change on the m4/analytics/eventlogging databases (db1046, db1047, dbstore2002)
  • 13:59 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Reduce db1035 & db1044 weight, repool at 100% db1051 & db1055 (duration: 00m 17s)
  • 13:11 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Reduce db1035 weight (duration: 00m 16s)
  • 12:09 godog: update facts on puppet compiler from palladium
  • 11:38 _joe_: rebooting mc2002
  • 10:39 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1055 (duration: 05m 10s)
  • 10:37 _joe_: rebooting mw1008, in oom spiral
  • 10:15 paravoid: salt rm /home/*/.ssh/authorized_keys
  • 09:21 jynus: downtime for db1055 for maintenance (kernel update, mysql update, config update)
  • 08:45 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1055 (duration: 00m 17s)
  • 07:44 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Increase load of db1051 and db1055 also for regular traffic (duration: 00m 17s)
  • 07:06 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Matching db1051 and db1055 weight for load balancing (duration: 00m 16s)
  • 06:15 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Oct 8 06:15:16 UTC 2015 (duration 15m 15s)
  • 05:56 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 18s)
  • 03:04 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-08 03:04:52+00:00
  • 03:02 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 05m 33s)
  • 02:47 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-08 02:46:57+00:00
  • 02:40 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 10m 30s)
  • 01:17 twentyafterfour: phabricator is now up and running again
  • 01:09 twentyafterfour: buggy update, stopping apache2 on iridium.
  • 01:02 twentyafterfour: finished phabricator update
  • 00:51 mutante: mw1160 - was oom-killer (convert, mw job 31497)
  • 00:50 twentyafterfour: phabricator maintenance/upgrade. Expect 10 minutes downtime
  • 00:44 mutante: powercycled mw1160 - looks like broken hardware or cable (ata1: lost interrupt)
  • 00:08 logmsgbot: krenair@tin Synchronized w/static/apple-touch/commons.png: https://gerrit.wikimedia.org/r/#/c/243685/ (duration: 00m 17s)

2015-10-07

  • 23:37 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/242842/ (duration: 00m 17s)
  • 23:34 logmsgbot: ori@tin Synchronized wmf-config: I924d8e19e17: Make the redis cache configuration multi-DC-ready (T111575) (duration: 00m 17s)
  • 23:22 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/244177/ (duration: 00m 17s)
  • 23:18 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/244176/ (duration: 00m 17s)
  • 23:08 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/243837/ (duration: 00m 17s)
  • 21:25 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: Revert "Don't use nutcracker on wikitech" (duration: 00m 16s)
  • 20:56 mutante: applied firewalling on IRCd server, rc bot still working fine, all public IRC ports as before
  • 20:25 mutante: argon: installing package upgrades
  • 20:24 mutante: resetting drac on argon
  • 20:14 logmsgbot: ori@tin Synchronized wmf-config/session.php: Ie25c368a: Switch mw1017 to use DC-specific redis cluster names (duration: 00m 17s)
  • 19:50 urandom: RESTBase deploy complete
  • 19:41 urandom: doing full deploy of c20e6336 to RESTBase
  • 19:29 urandom: canary deploy to restbase1001.eqiad complete
  • 19:14 urandom: deploying c20e6336 to canary node restbase1001.eqiad
  • 19:09 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.2
  • 18:56 logmsgbot: twentyafterfour@tin Synchronized wmf-config/CommonSettings.php: fix undefined variable warning that has been spamming logs (duration: 00m 17s)
  • 17:56 awight_: update fundraising crm from 7003cc38797848631d0c4d5f6ff68ab1d6118ad8 to f25fbe856b92373104985185db77311ea3a4d841
  • 16:39 mutante: powercycling analytics1035
  • 16:14 mobrovac: citoid deploying ec149fd5
  • 16:02 logmsgbot: marktraceur@tin Synchronized wmf-config/: Adding new config variable for uploads to Commons (duration: 00m 17s)
  • 15:44 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/MobileFrontend: SWAT (duration: 00m 17s)
  • 15:44 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.2/extensions/Echo: SWAT (duration: 00m 18s)
  • 15:44 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/Echo: SWAT (duration: 00m 17s)
  • 15:13 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1051 after maintenance (duration: 00m 17s)
  • 15:03 logmsgbot: krenair@tin Synchronized visualeditor-default.dblist: https://gerrit.wikimedia.org/r/#/c/242041/ (duration: 00m 17s)
  • 15:02 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/242041/ (duration: 00m 18s)
  • 14:59 mobrovac: AQS restarting restbase on aqs100x
  • 13:00 godog: decomission restbase-test2001 and reimage
  • 12:27 paravoid: salt-rm'ing /var/lib/apt/lists/ubuntu.wikimedia.org_ubuntu_dists_trusty_main_i18n_Translation-en%5fUS
  • 10:55 godog: reenable puppet on restbase / maps-test / aqs
  • 10:10 _joe_: depooling cp1059 from pybal, varnish
  • 08:44 godog: disable puppet on restbase, maps, aqs before merging https://gerrit.wikimedia.org/r/#/c/243127
  • 08:05 moritzm: installed spice security updates on labvirt*
  • 06:01 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Oct 7 06:01:01 UTC 2015 (duration 1m 0s)
  • 05:25 ebernhardson: generating elasticsearch indices in codfw, should run ~3 hours
  • 04:33 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Touch InitiialiSettings.php to force config regeneration (duration: 00m 18s)
  • 04:23 logmsgbot: ebernhardson@tin Synchronized wmf-config/: enable second es cluster in testwiki one more time (duration: 00m 18s)
  • 04:15 ebernhardson: sync-common on mw2187.codfw.wmnet to fix localisation cache errors in exception.log
  • 03:37 logmsgbot: ebernhardson@tin Synchronized wmf-config: redisable second cluster only on testwiki (duration: 00m 16s)
  • 03:35 logmsgbot: ebernhardson@tin Synchronized wmf-config/: Reenable second ES cluster on testwiki only (duration: 00m 18s)
  • 03:10 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-07 03:10:02+00:00
  • 03:03 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 10m 18s)
  • 02:37 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-07 02:37:32+00:00
  • 02:32 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 08m 22s)
  • 01:44 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.2/extensions/Echo: Fix JS error (duration: 00m 18s)
  • 01:44 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/Echo: Fix JS error (duration: 00m 17s)
  • 00:04 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.2/extensions/Echo: SWAT (duration: 00m 17s)
  • 00:04 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/Echo: SWAT (duration: 00m 17s)

2015-10-06

  • 23:20 logmsgbot: ebernhardson@tin Synchronized wmf-config: Revert multicluster config for testwiki (duration: 00m 18s)
  • 23:18 logmsgbot: ebernhardson@tin Synchronized wmf-config/: Enable multicluster ES on testwiki (duration: 00m 17s)
  • 22:44 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.2
  • 22:40 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.2: Deploy https://gerrit.wikimedia.org/r/#/c/244066/ (duration: 01m 40s)
  • 21:36 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.1
  • 21:15 twentyafterfour: restarted phd in response to a phabricator setup issue
  • 20:58 bblack: disabling puppet on caches for VCL testing
  • 20:58 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.2
  • 20:53 logmsgbot: twentyafterfour@tin Finished scap: sync wmf/1.27.0-wmf.1 (duration: 32m 13s)
  • 20:29 mutante: service "ishmael" has been removed (T109777) - removed docroot on neon. tarball exists in /root just in case. code is on https://github.com/asher/ishmael
  • 20:20 logmsgbot: twentyafterfour@tin Started scap: sync wmf/1.27.0-wmf.1
  • 18:48 yuvipanda: fixing https://phabricator.wikimedia.org/T109216 on labstore1002
  • 18:10 godog: upgrade videoscalers to ffmpeg2theora 0.29.0~git+20150813-2
  • 15:04 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Suggestion in af, gl, gu, mk, oc, sh and simplewiki gerrit:243919 (duration: 00m 18s)
  • 11:33 jynus: performing schema change on db1051 enwiki revision
  • 11:10 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1051 for more maintenance (duration: 00m 17s)
  • 10:45 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1051 after maintenance (duration: 00m 17s)
  • 10:41 jynus: potential extra load on mediawiki recent changes and watchlist on enwiki, please report any slowdown
  • 10:22 jynus: dropping temp recovered tables from db1051 to prepare for repool
  • 09:50 _joe_: uploaded a new pybal package for jessie
  • 06:19 paravoid: eqord is back up
  • 04:24 paravoid: all waves to eqord down, probably related to RT#9619
  • 02:34 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-06 02:34:31+00:00
  • 02:29 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 08m 53s)

2015-10-05

  • 23:53 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.1/extensions/ZeroBanner/includes/ZeroSpecialPage.php: https://gerrit.wikimedia.org/r/#/c/243833/ (duration: 00m 17s)
  • 23:39 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: rv (duration: 00m 17s)
  • 23:14 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.1/extensions/VisualEditor/modules/ve-mw/ui: https://gerrit.wikimedia.org/r/#/c/243729/ (duration: 00m 17s)
  • 23:05 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/242942/ (duration: 00m 17s)
  • 21:16 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.1/extensions/ZeroBanner: Take2: Deploying ZeroBanner 242661+243808 (duration: 00m 17s)
  • 21:00 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.1/extensions/ZeroBanner: Rolling back ZeroBanner 242661 (duration: 00m 18s)
  • 20:55 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.1/extensions/ZeroBanner: Deploying ZeroBanner 242661 (duration: 00m 17s)
  • 18:38 mutante: restarted gitblit
  • 17:44 gwicke: rolling restart of eqiad restbase nodes done
  • 17:27 gwicke: rolling restart of restbase cluster to rule out driver issues causing the increased p99 read latency
  • 17:19 robh: analytics1035 pegged out, ssh unresponsive and raid failures, and then fixed itself 5 minutes later
  • 16:32 legoktm: running backPopulateRenameQueueLogs.php (gerrit:237169) on metawiki
  • 16:05 hoo: Updated Wikidata's property suggester with data from today's json dump
  • 15:19 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.1/extensions/Flow: SWAT: Fix exception on board and topic history pages gerrit:243337 (duration: 00m 20s)
  • 14:41 hashar: puppet-lint Jenkins job is now strict and will -1 on errors as well as warnings https://gerrit.wikimedia.org/r/#/c/243185/
  • 10:25 godog: roll-restart cassandra in restbase eqiad
  • 09:58 _joe_: installing the new HHVM package to appservers
  • 09:50 _joe_: rebooted mw1153, soft lockup due to bnx2 failure
  • 09:50 godog: roll-restart cassandra in restbase codfw
  • 09:18 godog: roll-restart cassandra on restbase test cluster
  • 09:15 godog: stop puppet on restbase and maps in preparation for https://gerrit.wikimedia.org/r/#/c/242896/1
  • 08:07 _joe_: upgrading HHVM on all API appservers
  • 05:33 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Oct 5 05:33:18 UTC 2015 (duration 33m 17s)
  • 02:33 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-05 02:33:57+00:00
  • 02:29 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 08m 25s)

2015-10-04

  • 23:36 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/extensions/ContentTranslation/extension.json: 8c80ec1273: Updated mediawiki/core Project: mediawiki/extensions/ContentTranslation (duration: 00m 17s)
  • 05:13 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Oct 4 05:13:32 UTC 2015 (duration 13m 31s)
  • 02:31 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-04 02:31:56+00:00
  • 02:27 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 08m 06s)

2015-10-03

  • 15:33 jynus: stopping temporarily labsdb1004 mariadb to complete clone process
  • 09:38 _joe_: rolling restarting all parsoids in eqiad
  • 09:14 _joe_: restarting parsoid on wtp1021
  • 05:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Oct 3 05:28:54 UTC 2015 (duration 28m 53s)
  • 03:12 ori: done; graphite-web back up; url shortening will now work.
  • 03:11 ori: shutting down graphite-web for brief sqlite database schema update
  • 02:36 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-03 02:36:33+00:00
  • 02:31 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 08m 22s)

2015-10-02

  • 22:46 mutante: holmium: apt-get clean for a little disk space - /var/log/designate/designate-mdns.log is more than half the size of / - needs logrotate
  • 21:33 Krinkle: mwscript cleanupRemovedModules.php --wiki zhwiktionary
  • 21:21 Krinkle: mwscript cleanupRemovedModules.php --wiki testwikidatawiki
  • 21:15 Krinkle: mwscript cleanupRemovedModules.php --wiki dewiki
  • 21:15 Krinkle: mwscript cleanupRemovedModules.php --wiki nlwiki
  • 21:15 Krinkle: mwscript cleanupRemovedModules.php --wiki test2wiki
  • 21:06 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 17s)
  • 18:39 ottomata: kafka preferred-replica-election
  • 17:53 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/includes/resourceloader: I680f3fda66c5: Configure ResourceLoader-specific ObjectCache instance (duration: 00m 17s)
  • 17:53 ottomata: rolling restart of all hadoop-yarn-nodemanagers to pick up python3 + spark fix
  • 17:12 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I8318fe892: Configure MultiWriteBagOStuff for ResourceLoader (duration: 00m 17s)
  • 16:29 ori: deployed grafana 8e92884bae (with backport of upstream fb9f9548829f2d4cecf35cda933700e5c2fa1bd6)
  • 15:13 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/resources/src/mediawiki/mediawiki.js: I8029208: Don't clobber existing styles when adding more in IE9 (duration: 00m 17s)
  • 14:37 jynus: providing extra grants to wiki db users for heartbeat monitoring
  • 14:32 Coren: Change of NFS mount options in labs pushed - puppet may report failures to refresh the mounts (once) on instances; expected and harmless.
  • 11:03 moritzm: installed rpcbind security updates on all Ubuntu servers which runs it (jessie was already updated, since the DSA was released earlier)
  • 10:53 jynus: restarting HHVM on mw1130
  • 08:45 hashar: restarting Nodepool to take in account changes made to the logging configuration https://gerrit.wikimedia.org/r/#/c/240986/
  • 06:07 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Oct 2 06:07:05 UTC 2015 (duration 7m 4s)
  • 02:42 logmsgbot: tstarling@tin Synchronized php-1.27.0-wmf.1/extensions/ParsoidBatchAPI: stats (duration: 00m 17s)
  • 02:41 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-02 02:41:33+00:00
  • 02:36 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 08m 04s)
  • 00:44 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/includes/resourceloader: I21bb3f08e7f and follow-ups (duration: 00m 18s)
  • 00:43 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/vendor: Add wikimedia/relpath 1.0.3 (duration: 00m 21s)

2015-10-01

  • 22:04 RoanKattouw: Running FlowFixLinks.php on all wikis
  • 20:33 subbu: deployed parsoid version 62971510b
  • 19:33 twentyafterfour: re-deployed robots.txt patch and restarted apache on iridium (to expand the phabricator robots.txt)
  • 18:22 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/includes/resourceloader: Ic1d802ee2: ResourceLoader: cache minified user and site modules (duration: 00m 17s)
  • 18:04 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.27.0-wmf.1
  • 16:40 ottomata: brought analytics1049 back into hadoop after missing a disk since sept. 25th
  • 16:34 andrewbogott: test log
  • 16:26 moritzm: installed PHP security updates on all precise/trusty systems (the respective DSA for jessie is already deployed, it was released three weeks ago)
  • 16:26 valhallasw`cloud: testing, testing
  • 15:56 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/240049 (duration: 00m 17s)
  • 15:56 cmjohnson1: db1051 replacing failed disk slot 6
  • 15:51 logmsgbot: mattflaschen@tin Synchronized php-1.27.0-wmf.1/extensions/ContentTranslation/modules/dashboard/styles/ext.cx.dashboard.less: Fix: Clicking on down arrow in language selector should trigger ULS (duration: 00m 17s)
  • 15:48 logmsgbot: mattflaschen@tin Synchronized wmf-config/InitialiseSettings.php: Enable CX suggestions in ar, eo, hi, nl, vi and dawiki (duration: 00m 17s)
  • 15:47 cmjohnson1: analytics1049 replacing failed disk /dev/sdi at slot 7
  • 15:46 logmsgbot: mattflaschen@tin Synchronized php-1.27.0-wmf.1/tests/phpunit/includes/objectcache/BagOStuffTest.php: Memcached key decode fix (duration: 00m 18s)
  • 15:45 logmsgbot: mattflaschen@tin Synchronized php-1.27.0-wmf.1/includes/objectcache/MemcachedBagOStuff.php: Memcached key decode fix (duration: 00m 17s)
  • 15:44 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf24/tests/phpunit/includes/objectcache/BagOStuffTest.php: Memcached key decode fix (duration: 00m 17s)
  • 15:43 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf24/includes/objectcache/MemcachedBagOStuff.php: Memcached key decode fix (duration: 00m 18s)
  • 15:33 cmjohnson1: replacing failed disk ms-be1012 /dev/sdf slot 5
  • 15:09 matt_flaschen: Did final run of convertAllLqtPages.php on sewikimedia immediately before freezing LQT
  • 15:08 logmsgbot: mattflaschen@tin Synchronized wmf-config/InitialiseSettings.php: Freeze LQT on se.wikimedia (duration: 00m 18s)
  • 14:58 moritzm: installed PHP security updates on all precise/trusty systems (the respective DSA for jessie is already deployed, it was released three weeks ago)
  • 14:19 andrewbogott: log test
  • 14:17 andrewbogott: testing the log
  • 14:07 _joe_: installing the new hhvm package to all the canaries
  • 13:41 hashar: Am I logging?
  • 13:32 _joe_: uploaded hhvm_3.6.5+dfsg1-1+wm7
  • 13:03 bblack: repooling cp1046 (eqiad mobile) with caches wiped clean just before
  • 12:24 akosiaris: disable SessionRemoteIPcheck on mendelevium's OTRS installation for checking
  • 12:18 logmsgbot: aude@tin Finished scap: Put Wikidata extension back on wmf/1.27.0-wmf.1 (duration: 30m 46s)
  • 12:09 moritzm: installed PHP security updates on all precise/trusty systems (the respective DSA for jessie is already deployed, it was released three weeks ago)
  • 11:47 logmsgbot: aude@tin Started scap: Put Wikidata extension back on wmf/1.27.0-wmf.1
  • 10:35 logmsgbot: hoo@tin Synchronized php-1.27.0-wmf.1/extensions/: Use 1.27.0-wmf.1 for Wikidata again after fixing T114290 (duration: 01m 07s)
  • 10:20 akosiaris: uploaded php5_5.3.10-1ubuntu3.20+wmf1 on apt.wikimedia.org precise-wikimedia
  • 09:51 godog: reboot praseodymium to test cassandra systemd unit
  • 09:47 godog: stop puppet on restbase* in preparation for https://gerrit.wikimedia.org/r/242548
  • 08:59 akosiaris: disabling puppet on maps-test200* in expectance of https://gerrit.wikimedia.org/r/242548
  • 06:56 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Oct 1 06:56:51 UTC 2015 (duration 56m 50s)
  • 03:21 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-01 03:21:39+00:00
  • 03:14 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 10m 38s)
  • 02:47 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf24) at 2015-10-01 02:47:31+00:00
  • 02:40 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 10m 54s)
  • 01:30 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/includes/resourceloader/ResourceLoader.php: 1cfe27030e: Change load.php to minify per-module instead of per-request (duration: 00m 17s)
  • 01:29 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/resourceloader/ResourceLoader.php: 1cfe27030e: Change load.php to minify per-module instead of per-request (duration: 00m 17s)
  • 00:44 mutante: applying puppet fix on fermium (illegal byte sequence in utf-8) as in T114289#1691038 but for other languages

2015-09-30

  • 23:00 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.1
  • 22:55 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.1/: deploying several fixes to the branch (duration: 01m 52s)
  • 22:44 ottomata: perf testing eventlogging in production by hammering https://bits.wikimedia.org/beacon/event.gif
  • 22:13 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/includes/User.php: 0ed7cc8526: Made User::loadFromId() skip cache with READ_LATEST (duration: 00m 17s)
  • 22:12 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/User.php: 0ed7cc8526: Made User::loadFromId() skip cache with READ_LATEST (duration: 00m 17s)
  • 21:36 cwdent: updated payments from bc4bcc44d2337d7a69c5a39f11ff45efdf0c8e11 to 24d5be6886d34b3600031290c7f55ee84f3dcee2
  • 21:27 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/resourceloader/ResourceLoaderFileModule.php: a1e1619461: Fix LESS file dependency tracking in ResourceLoader (duration: 00m 17s)
  • 20:43 cscott: updated Parsoid to version 39c60c67
  • 20:40 mutante: fixing puppet run on fermium, needs manual fix because puppet cant replace existing illegal character in some templates
  • 19:18 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.1/resources/Resources.php: T114288 (duration: 00m 17s)
  • 18:51 jynus: stopping replication on s2, s3 and s7 for dbstore1001
  • 18:50 ottomata: restarted eventlogging with blacklist=^Analytics$
  • 18:40 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf24
  • 18:14 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.1
  • 17:39 ottomata: restarting eventlogging with 12 client side processors
  • 16:28 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/resourceloader: 89595ba49a: Cherry-pick I173a9820b and I7c7546ec (duration: 00m 18s)
  • 16:20 logmsgbot: krenair@tin Finished scap: swat (duration: 53m 33s)
  • 15:27 logmsgbot: krenair@tin Started scap: swat
  • 13:53 moritzm: rebooted stat1001/stat1002/stat1003 for kernel updates (already happened between 13:00 UTC and 13:10 UTC, but forgot to log earlier)
  • 12:34 moritzm: added debdeploy 0.0.8 to carbon
  • 05:43 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Sep 30 05:43:19 UTC 2015 (duration 43m 18s)
  • 03:10 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-09-30 03:10:39+00:00
  • 03:03 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 10m 45s)
  • 02:36 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf24) at 2015-09-30 02:36:37+00:00
  • 02:32 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 07m 21s)
  • 01:02 logmsgbot: ori@tin Synchronized php-1.26wmf24/vendor: 940124a7db: Updated mediawiki/core Project: mediawiki/vendor ff5e254f7eddf811f6f66b4a4063b1a8cc70f265 (duration: 00m 21s)

2015-09-29

  • 23:59 ottomata: restarting eventlogging so that processors use etcd to pick up shared token with which to consistently hash IPs
  • 23:46 logmsgbot: catrope@tin Synchronized php-1.26wmf24/extensions/Echo/: SWAT (duration: 00m 17s)
  • 23:37 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Enable Flow opt-in on mediawikiwiki (duration: 00m 16s)
  • 23:33 logmsgbot: catrope@tin Synchronized php-1.26wmf24/extensions/GuidedTour/GuidedTourHooks.php: T114144 Fix back button logging (duration: 00m 18s)
  • 23:32 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/Echo/: SWAT (duration: 00m 17s)
  • 23:30 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/GuidedTour/GuidedTourHooks.php: T114144 Fix back button logging (duration: 00m 17s)
  • 22:50 ejegg: updated payments-wiki from 3b0915a51a0fd567bdf22f3d4e17548a83e735d8 to bc4bcc44d2337d7a69c5a39f11ff45efdf0c8e11
  • 22:47 ejegg: updated SmashPig from bf302444eae8236734fd43883b06c7b2512b1532 to 513ec01123e6dbb97b00888a3610a7c5ec24a63b
  • 22:38 twentyafterfour: finished deployment of 1.27.0-wmf.1 to group0
  • 22:36 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.1
  • 22:34 logmsgbot: twentyafterfour@tin Finished scap: php-1.27.0-wmf.1/extensions/ (duration: 04m 29s)
  • 22:30 twentyafterfour: 22:27:17 sync-dir failed: <ValueError> /srv/mediawiki-staging/php-1.27.0-wmf.1/vendor/oyejorge/less.php/lib/Less/Version.php has content before opening <?php tag
  • 22:30 chasemp: es-tool unban-node elastic1031
  • 22:29 logmsgbot: twentyafterfour@tin Started scap: php-1.27.0-wmf.1/extensions/
  • 22:23 chasemp: unbanning elastic1006 for shard population
  • 22:05 ori: 22:03:51 Synchronized php-1.26wmf24/extensions/CentralNotice: 30bdfcb386: Updated mediawiki/core Project: mediawiki/extensions/CentralNotice 6bd658e155a02edb4cc506bc3494a3f4699d3e94 (duration: 00m 17s)
  • 19:34 logmsgbot: twentyafterfour@tin Finished scap: sync php-1.27.0-wmf.1 for validation on testwiki (duration: 31m 49s)
  • 19:30 bd808: Updated iegreview.wikimedia.org to c3ac5e6 (Update to Twig 1.20.0) and applied latest schema changes
  • 19:02 logmsgbot: twentyafterfour@tin Started scap: sync php-1.27.0-wmf.1 for validation on testwiki
  • 18:11 cmjohnson1: shutting down elastic1031 to relocate rack/row
  • 18:09 cmjohnson1: shutting down elastic1006 to relocate row/rack
  • 17:46 mutante: size of conntrack table on iron might be increased due to test scans
  • 17:32 ejegg: updated appeal template setting on payments-wiki
  • 16:30 cmjohnson1: swapping failed disk db1050
  • 16:14 subbu: reverted configuration hotfix from yesterday's Parsoid deploy (re-enabled use of Parsoid batching API)
  • 15:58 bblack: ending VCL tests on cp1053
  • 15:52 bblack: ending VCL tests on cp1065, starting on cp1053 instead
  • 15:39 bblack: cp1065: live-testing some VCL patches, puppet disabled, etc...
  • 15:19 chasemp: ban elastic1031 for T112559
  • 15:05 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: VisualEditor: Set TransitionDefault true for the English Wikipedia gerrit:242040 (duration: 00m 17s)
  • 14:08 paravoid: repooling eqiad; 24h codfw test window is over
  • 10:49 godog: bounce pybal on lvs2003 / lvs2006
  • 09:13 paravoid: powercycling pybal-test2003
  • 08:51 jynus: starting cloning of labsdb1005 (Tools DB), minimal disruption is expected
  • 07:03 twentyafterfour: restarted apache2 and phd or iridium to get phabricator back into the correct state
  • 06:58 twentyafterfour: checked out correct phabricator release tag on iridium. Something, somewhere, had reverted everything to an old deployment.
  • 05:01 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Sep 29 05:01:30 UTC 2015 (duration 1m 29s)
  • 02:34 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf24) at 2015-09-29 02:34:55+00:00
  • 02:30 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 06m 57s)
  • 02:25 bblack: ipv6 flap experiment: raise ipv6/route/max_size from 4096 to 131072 manually on cp*, actually
  • 02:24 bblack: ipv6 flap experiment: raise ipv6/route/max_size from 4096 to 131072 manually on cp20*
  • 01:55 robh: upload-lb.codfw.wikimedia.org_ipv6 page flap
  • 01:43 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Plumbing for wmgVisualEditorTransitionDefault (duration: 00m 17s)
  • 01:43 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Add wmgVisualEditorTransitionDefault (false everywhere) (duration: 00m 17s)
  • 01:13 robh: didnt powercycle analytics1035 yet, it recovered on its own.
  • 01:13 robh: powercycling analytics1035, seems oom, cannot login via ssh or serial console
  • 00:56 logmsgbot: tstarling@tin Synchronized php-1.26wmf24/extensions/ParsoidBatchAPI: Fix fatal error I77fd7e8 (duration: 00m 17s)

2015-09-28

  • 23:47 logmsgbot: ebernhardson@tin Synchronized wmf-config/CommonSettings.php: Update ttmserver configuration to match elasticsearch security profile (duration: 00m 17s)
  • 23:43 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I56db35b: Removed ignore_user_abort( true ) line (duration: 00m 18s)
  • 23:13 logmsgbot: catrope@tin Synchronized php-1.26wmf24/extensions/Echo: SWAT (duration: 00m 18s)
  • 23:13 logmsgbot: catrope@tin Synchronized php-1.26wmf24/extensions/Flow: SWAT (duration: 00m 19s)
  • 23:12 logmsgbot: catrope@tin Synchronized php-1.26wmf24/extensions/CentralNotice: SWAT (duration: 00m 19s)
  • 20:51 bearND: MobileApps deployed sha1 9df72ec
  • 20:42 subbu: deployed parsoid version b9e5244e + hotfix on tin to turn off batching api use since canary restart of wtp1002 showed some batching api errors
  • 20:26 yuvipanda: depooled cp2017 from pybal config too
  • 20:22 yuvipanda: depooled cp2017 since it's down
  • 19:02 chasemp: powercycle iridium via console as it's unresponsive
  • 18:01 awight: update crm from 190f689ff7aec7fecefdf5af501293685c55e041 to 7003cc38797848631d0c4d5f6ff68ab1d6118ad8
  • 15:50 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/241579/ (duration: 00m 17s)
  • 15:28 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/241598/ (duration: 00m 17s)
  • 15:11 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/241649/ (duration: 00m 17s)
  • 14:00 paravoid: running failover eqiad->codfw test for all frontend traffic
  • 06:11 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I179be4bd3: Rely on timeouts specified in php.ini rather than calling ini_set() (duration: 00m 17s)
  • 05:52 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Sep 28 05:52:34 UTC 2015 (duration 52m 33s)
  • 02:27 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf24) at 2015-09-28 02:27:16+00:00
  • 02:20 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 06m 54s)

2015-09-27

  • 23:17 logmsgbot: ori@tin Synchronized php-1.26wmf24/./resources/src/mediawiki.toolbar/toolbar.less: I94ced06178: mediawiki.toolbar: temporary workaround for T113868 (duration: 00m 17s)
  • 16:51 Krenair: ran sync-common on snapshot1001 to bring it up to date
  • 16:43 logmsgbot: krenair@tin Purged l10n cache for 1.26wmf21
  • 16:42 logmsgbot: krenair@tin Purged l10n cache for 1.26wmf20
  • 16:42 logmsgbot: krenair@tin Purged l10n cache for 1.26wmf19
  • 15:50 logmsgbot: krenair@tin Synchronized wmf-config/abusefilter.php: https://gerrit.wikimedia.org/r/#/c/241354/ - fix AbuseFilter block durations (duration: 00m 18s)
  • 10:05 jynus: leaving innodb compression tests on es2005 running (could affect lag on that host)
  • 02:23 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 07m 05s)

2015-09-26

  • 19:04 hashar: restarting Jenkins. Just in case :-D
  • 02:23 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 06m 48s)

2015-09-25

  • 23:54 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/GlobalFunctions.php: 37c6972f94: Made wfIsBadImage() use APC (duration: 00m 17s)
  • 22:16 mutante: sodium - shutting down
  • 21:53 ori: Armed new servicedeploy_rsa on mira
  • 21:51 ori: Armed new servicedeploy_rsa on tin
  • 20:45 urandom: starting a Cassandra repair on xeon (nodetool repair -pr)
  • 19:30 mutante: restarted apache on gallium
  • 19:07 bblack: configuring and enabling lvs-cross-row ports on asw-b-eqiad for lvs1007,8,10,11
  • 18:04 jynus: performing schema change on m5-master "nova"
  • 17:26 jynus: restarting and upgrading db1051 mysql (depooled)
  • 17:20 godog: reboot ms-be1005, xfs
  • 16:22 cwdent: updated payments from dc78ff5157b59a8f475dc86194a1059c2d6b2fad to 3b0915a51a0fd567bdf22f3d4e17548a83e735d8
  • 16:17 akosiaris: duplicating database otrs to otrsupgradetest for testing the upgrade procedure
  • 16:16 cmjohnson1: cp1046 stopped icinga checks for hardware troubleshooting
  • 14:43 moritzm: installed rpcbind and apport security updates on various servers
  • 14:36 logmsgbot: krenair@tin Synchronized php-1.26wmf24/includes/specials/SpecialMovepage.php: https://gerrit.wikimedia.org/r/#/c/241045/ (duration: 00m 17s)
  • 14:22 chasemp: 'sudo -u hdfs hdfs haadmin -transitionToActive analytics1001-eqiad-wmnet' per otto on analytics1001
  • 14:03 urandom: starting a Cassandra repair on restbase1006 (nodetool repair -pr -dc eqiad)
  • 13:58 hashar: nodepool back in operations
  • 13:44 moritzm: restarted saltmaster on palladium
  • 13:12 mobrovac: restbase deploying e42bf0fc
  • 13:12 moritzm: added debdeploy 0.0.7 to carbon
  • 13:10 godog: enable puppet on restbase in production
  • 12:54 andrewbogott: restarted rabbitmq-server on labcontrol1001
  • 12:52 godog: stop puppet on restbase in production -- config deployment
  • 12:46 hashar: stopping nodepool to clear out left over mysql connections
  • 12:21 hashar: Nodepool is dead, or at least not adding new slaves anymore
  • 12:16 hashar: Seems Zuul/Jenkins is in trouble somehow :-/
  • 11:42 andrewbogott: upgrading linux-image-generic on labnet1002 to get us away from the recently-crashed 3.13.0-59
  • 04:58 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I252970886: Set "async" for SQL parser cache everywhere else (duration: 00m 18s)
  • 02:57 Krinkle: sync-common failed on mw1010.eqiad.wmnet
  • 02:53 logmsgbot: krinkle@tin Synchronized php-1.26wmf24/extensions/DonationInterface: 381faf5 (duration: 00m 21s)
  • 02:33 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 11m 45s)
  • 01:44 Krinkle: mwscript deleteEqualMessages.php --wiki roa_tarawiki
  • 01:42 Krinkle: mwscript deleteEqualMessages.php --wiki itwikiquote
  • 01:42 Krinkle: mwscript deleteEqualMessages.php --wiki alswiki
  • 01:28 logmsgbot: krinkle@tin Synchronized php-1.26wmf24/extensions/WikimediaEvents: I5608f8ffd1c - Fix trailing comma (duration: 00m 17s)
  • 00:10 logmsgbot: krinkle@tin Synchronized php-1.26wmf24/extensions/EventLogging/modules/ext.eventLogging.core.js: Increase maxUrlSize to 2000 (duration: 00m 17s)

2015-09-24

  • 23:02 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/240880/ (duration: 00m 17s)
  • 21:18 ejegg: updated SmashPig from d1baa32267eaad7d69b47c657f4853eb306fad6b to bf302444eae8236734fd43883b06c7b2512b1532
  • 21:15 ejegg: updated payments-wiki from 8428499feb8760d63faf681d53995697a2ba0fa7 to dc78ff5157b59a8f475dc86194a1059c2d6b2fad
  • 19:57 logmsgbot: ori@tin Synchronized php-1.26wmf24/extensions/ContentTranslation: d079d5dd71: Updated mediawiki/core Project: mediawiki/extensions/ContentTranslation 8559ee614975f25b71a732ca0fb1bb6d489c9d33 (duration: 00m 18s)
  • 19:35 bblack: depooled cp1046 from confd, committed pybal depool for LVS as well
  • 19:34 chasemp: changing labs route on cr1 and cr2 from 10.68.16.0/22 to 10.68.16.0/21 which matches references, fw setting and manifests/network.pp
  • 18:54 logmsgbot: catrope@tin Synchronized php-1.26wmf24/extensions/Flow/: Debugging for FlowFixLinks.php (duration: 00m 20s)
  • 18:21 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.26wmf24
  • 18:20 legoktm: moved oauthadmin group from User:Yuvipanda@metawiki to User:YuviPanda@metawiki
  • 18:19 godog: restart restbase on restbase1005
  • 18:18 godog: restart restbase on restbase1004
  • 18:17 ejegg: reenabling CRM jenkins jobs
  • 18:16 godog: restart restbase on restbase1003
  • 18:08 ejegg: updated civicrm from 9fa38d06a75363a8009bce7ced190e39c75b68bc to 190f689ff7aec7fecefdf5af501293685c55e041
  • 18:06 paravoid: depooling cp1046, stability issues
  • 18:05 ejegg|afk: disabled CRM jenkins jobs
  • 18:00 logmsgbot: demon@tin Synchronized multiversion/MWRealm.php: (no message) (duration: 00m 17s)
  • 17:59 ori: Merged Apache config change Ia095457fb. It will refresh the Apache service as it rolls out, causing elevated 503s for the next 20 minutes.
  • 17:53 godog: rolling restart restbase in eqiad
  • 17:35 chasemp: powercycling cp1046 at mgmt as I can't ssh in and it seems like it should be up
  • 17:26 godog: bounce restbase on restbase1002, apply new datacenter config
  • 17:10 _joe_: cleaning up /tmp on mw1152
  • 17:09 cmjohnson1: powering down for the last time es1001 - es1010
  • 16:17 logmsgbot: thcipriani@tin Synchronized php-1.26wmf23/extensions/Wikidata: SWAT: Do not filter affected pages by namespace gerrit:240727 (duration: 00m 26s)
  • 16:01 robh: nothing on puppet swat window, easiest swat ever.
  • 15:46 logmsgbot: thcipriani@tin Synchronized php-1.26wmf24/extensions/Wikidata: SWAT: Do not filter affected pages by namespace gerrit:240711 (duration: 00m 26s)
  • 15:23 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable suggestions in ca, en, es, fr, it, ja, tr, ru, zh gerrit:240638 (duration: 00m 17s)
  • 14:37 paravoid: repooling codfw
  • 12:54 bblack: restarting varnish daemons on second half of maps, parsoid, misc clusters (package upgrade, shm_reclen change)
  • 12:50 bblack: restarting varnishd instances on text, mobile, upload clusters for package upgrade (slow salt, no parallelism, ~5m spacing - FE cache loss, BE cache stays, should take ~9h)
  • 12:05 moritzm: installed rpcbind security updates on eeden, baham, radon, maerlant, rhenium
  • 11:56 bblack: restarting varnish daemons on half of maps, parsoid, misc clusters (package upgrade, shm_reclen change)
  • 11:36 bblack: reinstall lvs300[12] to jessie - T96375
  • 11:21 akosiaris: killed tail -f varnishncsa.log on cp1065 and ran apt-get clean to reclaim some disk space
  • 11:14 bblack: stopping pybal on lvs300[12]; lvs300[34] taking over
  • 11:07 bblack: upgrading varnishes to 3.0.6plus-wm8 (non-restarting, just pkg update on-disk)
  • 09:40 jynus: performing latest (software) steps to decom es1001-es1010 (puppet disabling, etc.)
  • 08:39 jynus: restarted HHVM @ mw1056, 1104, 1122
  • 05:33 yuvipanda: deleted logstash indexes for 08/27 and 28 too
  • 05:31 yuvipanda: deleted indexes for 08/14, 15, 25, 26 on logstash
  • 03:59 yuvipanda: restarting elasticsearch in logstash1001-3
  • 03:53 yuvipanda: restarted es on logstash1004-6
  • 03:02 yuvipanda: jstack dumped logstash output onto /home/yuvipanda/stack on logstash1001 since strace seems useles
  • 02:51 yuvipanda: restarted logstash on logstash1002
  • 02:41 yuvipanda: gmond at 100% again, killing it and stopping puppet again
  • 02:40 yuvipanda: re-enabling and running puppet on hafnium to see what it's bringing up
  • 02:38 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 30s)
  • 02:23 yuvipanda: kill gmond on hafnium and disable puppet to prevent it from taking it back up. Was taking 100% CPU
  • 02:16 Krinkle: Kibana/Logstash outage. Zero events received after 2015-09-23T23:59:59.999Z.
  • 02:14 Krinkle: Partial EventLogging outage (client-side events via hafnium abruptly stopped 2015-09-23 11:36 UTC - 15 hours ago)
  • 01:53 mutante: started logstash on logstash1002 again
  • 01:35 mutante: bast1001: unmounting /srv/home_pmtpa (backup on bacula)
  • 01:34 mutante: removing subversion packages from bast1001
  • 01:15 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes: Ifa0d4cfe8e3: Backport I1ff61153d and I8e4c3d5a5 (duration: 00m 23s)
  • 00:19 jynus: restarted replication on db1051
  • 00:17 ori: restarting tcpircbot on neon
  • 00:16 mutante: started logstash on logstash1002
  • 00:08 bblack: varnish package on carbon for jessie updated to 3.0.6plus-wm8

2015-09-23

  • 23:17 logmsgbot: krenair@tin Synchronized php-1.26wmf24/includes/specials/SpecialSearch.php: https://gerrit.wikimedia.org/r/#/c/240596/ (duration: 00m 18s)
  • 23:09 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/240575/ (duration: 00m 17s)
  • 22:53 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/resourceloader/ResourceLoaderFileModule.php: 14f46330d9: Backport fix from PS12 of I1ff6115 (duration: 00m 17s)
  • 22:39 paravoid: cr1-codfw RE switchover(s)
  • 22:35 logmsgbot: yurik@tin Synchronized php-1.26wmf24/extensions/ZeroBanner: Deploying ZeroBanner interstitial handling (duration: 00m 18s)
  • 21:13 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/resourceloader/ResourceLoaderFileModule.php: 58bfb6f85b: Backport fix from PS9 of I1ff6115 (duration: 00m 17s)
  • 20:46 subbu: deployed parsoid 6619409e
  • 19:58 urandom: starting Cassandra on restbase2006
  • 19:56 urandom: enabling puppet, and forcing a run on restbase2006
  • 19:52 urandom: starting Cassandra on restbase2005
  • 19:50 urandom: enabling puppet, and forcing a run on restbase2005
  • 19:48 urandom: starting Cassandra on restbase2004
  • 19:45 urandom: enabling puppet, and forcing a run on restbase2004
  • 18:55 twentyafterfour: snapshot1001.eqiad.wmnet returned [12]: rsync: write failed on "/srv/mediawiki/wikiversions.cdb": No space left on device
  • 18:54 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf24
  • 18:01 mobrovac: restbase deploying f65313ed
  • 17:49 urandom: starting Cassandra on restbase2003
  • 17:45 logmsgbot: demon@tin Synchronized php-1.26wmf23/extensions/Wikidata: (no message) (duration: 00m 25s)
  • 17:35 urandom: enabling, and forcing puppet run on restbase2003
  • 17:27 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 13s)
  • 17:20 logmsgbot: ori@tin scap failed: OSError [Errno 2] No such file or directory: '/var/lock/scap' (duration: 44m 24s)
  • 17:11 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/page/Article.php: (no message) (duration: 00m 13s)
  • 17:11 logmsgbot: ori@tin Synchronized php-1.26wmf23/includes/page/Article.php: (no message) (duration: 00m 17s)
  • 16:35 logmsgbot: ori@tin Started scap: (no message)
  • 16:35 logmsgbot: ori@tin Synchronized php-1.26wmf23/includes/page: I952068d2d: Reduced the DOS potential of 404 page floods (duration: 00m 12s)
  • 16:22 godog: start cassandra on restbase2002
  • 16:05 godog: deploy restbase daacf4d on restbase2*
  • 15:34 logmsgbot: demon@tin Synchronized php-1.26wmf24/extensions/Wikidata: (no message) (duration: 00m 20s)
  • 15:34 logmsgbot: demon@tin Synchronized php-1.26wmf23/extensions/Wikidata: (no message) (duration: 00m 21s)
  • 15:34 chasemp: unbanning elastic1005 for T112559
  • 15:32 logmsgbot: demon@tin scap aborted: (no message) (duration: 02m 07s)
  • 15:30 logmsgbot: demon@tin Started scap: (no message)
  • 15:15 logmsgbot: demon@tin Synchronized php-1.26wmf22/extensions/Wikidata: (no message) (duration: 00m 20s)
  • 14:13 logmsgbot: demon@tin Synchronized wmf-config/abusefilter.php: (no message) (duration: 00m 12s)
  • 14:08 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 12s)
  • 14:05 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 12s)
  • 14:04 logmsgbot: demon@tin Synchronized langlist-labs: (no message) (duration: 00m 11s)
  • 14:04 logmsgbot: demon@tin Synchronized docroot/noc/conf/: (no message) (duration: 00m 13s)
  • 14:01 logmsgbot: demon@tin Synchronized multiversion/MWRealm.php: (no message) (duration: 00m 11s)
  • 13:56 godog: force puppet run on restbase2001 to deploy new cassandra config
  • 13:28 godog: reboot ms-be1012, xfs hosed
  • 13:22 paravoid: upgrading cr1-codfw with newer junos
  • 07:43 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: semver the special:version hook (duration: 00m 12s)
  • 03:05 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 10m 12s)
  • 02:38 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf23) at 2015-09-23 02:38:44+00:00
  • 02:35 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 21s)
  • 00:56 logmsgbot: legoktm@tin Synchronized php-1.26wmf24/extensions/Echo/Hooks.php: Remove duplicate 'MediaWiki' prefix from echo.unseen stats (duration: 00m 12s)
  • 00:03 RoanKattouw: Running FlowFixLinks.php on testwiki

2015-09-22

  • 23:40 mutante: renaming search mailing lists to discovery mailing lists
  • 23:35 logmsgbot: krenair@tin Synchronized php-1.26wmf24/extensions/Echo: https://gerrit.wikimedia.org/r/#/c/240283/ and https://gerrit.wikimedia.org/r/#/c/240281/ (duration: 00m 13s)
  • 23:18 logmsgbot: krenair@tin Synchronized wmf-config/abusefilter.php: https://gerrit.wikimedia.org/r/#/c/240278/ (duration: 00m 12s)
  • 23:16 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/240259/ (duration: 00m 12s)
  • 23:15 logmsgbot: krenair@tin Synchronized w/static/images/sul/wikimania.png: https://gerrit.wikimedia.org/r/#/c/239308/ (duration: 00m 11s)
  • 23:14 logmsgbot: krenair@tin Synchronized w/static/images/sul/commons.png: https://gerrit.wikimedia.org/r/#/c/239308/ (duration: 00m 12s)
  • 22:44 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Set $wgFlowMigrateReferenceWiki to false in production (duration: 00m 12s)
  • 22:38 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Enable Flow opt-in on testwiki for testing (duration: 00m 12s)
  • 21:56 cwdent: updated payments from 7b08867d9c5e87f5babb4b5b9cf1f5bec5e243b3 to 8428499feb8760d63faf681d53995697a2ba0fa7
  • 21:49 chasemp: unban elastic1030 from T112559
  • 21:33 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.26wmf24
  • 21:13 logmsgbot: twentyafterfour@tin Finished scap: Test 1.26wmf24 (duration: 50m 34s)
  • 20:44 mutante: cancel backup job of bast1001 on helium because running low on disk
  • 20:22 logmsgbot: twentyafterfour@tin Started scap: Test 1.26wmf24
  • 16:50 robh: all mw servers returned to puppet enabled, puppet swat window over
  • 16:40 awight: updated paymentswiki 153418195a45cab820bc2aacf9a4f7dbc9dde768 to 7b08867d9c5e87f5babb4b5b9cf1f5bec5e243b3
  • 16:23 robh: re-enabled puppet on mw hosts, as both redirection changes are good
  • 16:14 robh: re-enabling puppet on mw hosts, as the new patchset 239278 deployed and tested fine on a single host, deploying to rest
  • 16:04 robh: disabling puppet across mw hosts for new configuration deployment
  • 15:46 godog: running puppet on restbase2001
  • 15:41 godog: stop puppet on restbase2* pending codfw expansion
  • 15:31 cwdent: updated payments from 153418195a45cab820bc2aacf9a4f7dbc9dde768 to 7b08867d9c5e87f5babb4b5b9cf1f5bec5e243b3
  • 14:42 cmjohnson1: shutting down elastic1005 and elastic1030 to move around within the data center
  • 14:19 bblack: starting slow restart of varnish + varnish-frontend daemon processes on global text, upload, and mobile clusters for shm_reclen (all randomly blended, no parallelism, ~5 minute spacing, will take ~9 hours - FEs will lose cache data, BEs will not)
  • 14:14 chasemp: depool elastic nodes for T112559
  • 11:18 logmsgbot: aude@tin Synchronized php-1.26wmf23/extensions/Wikidata: Fix autocomment and change handling bugs (duration: 00m 21s)
  • 10:42 logmsgbot: aude@tin Synchronized arbitraryaccess.dblist: Enable arbitrary access for Wikibooks (duration: 00m 12s)
  • 10:42 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable data access for Wikibooks - try again for snapshot hosts (duration: 00m 12s)
  • 10:35 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable data access for Wikibooks (duration: 01m 12s)
  • 10:17 moritzm: enabled ferm on mw1152 (videoscaler)
  • 10:03 godog: finished stressdisk on restbase200[123] no errors reported
  • 10:03 moritzm: enabled ferm on mw1259 (videoscaler)
  • 04:35 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Sep 22 04:35:46 UTC 2015 (duration 35m 45s)
  • 02:22 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf23) at 2015-09-22 02:22:56+00:00
  • 02:19 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 00s)
  • 01:53 mutante: sodium - deleted salt key, revoked puppet cert, rm from icinga ..
  • 00:32 ori: Disabled Puppet for 24h on hafnium and stopped ganglia-monitor. gmond was saturating CPU.

2015-09-21

  • 23:29 logmsgbot: krinkle@tin Synchronized php-1.26wmf23/extensions/NavigationTiming/modules/ext.navigationTiming.js: T112593 (duration: 00m 14s)
  • 23:21 logmsgbot: krenair@tin Synchronized php-1.26wmf22/extensions/Wikidata: https://gerrit.wikimedia.org/r/#/c/239828/ (duration: 00m 21s)
  • 23:07 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/238357/ (duration: 00m 13s)
  • 22:26 mutante: restarting gerrit for ssh config change
  • 21:15 awight: legacy PayPal listener updated from 1c9ac2e66d11bbf768ea873d6e1a2522ca9841c1 to 55aeef63f6508381e3a8b7fcabddf9a3c3b73b8e
  • 20:43 cwdent: updated worldpay config on payments
  • 20:41 mdholloway: MobileApps deployed sha1 013044e
  • 20:39 chasemp: banning elastic1005 for T112559
  • 20:29 subbu: deployed parsoid version 9984d221
  • 19:41 urandom: temporarily stopping codfw restbase cassandra nodes to test quorum auth
  • 19:15 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: Ieccb23f: Enable async secondary writes for mysql-multiwrite cache (on testwiki) (duration: 00m 13s)
  • 18:36 ejegg: re-enabled paypal audit parser
  • 18:16 cmjohnson1: disabling puppet on mw1031
  • 17:58 chasemp: banning 1030 from eqiad elastic cluster for T112559#1660068
  • 17:57 ejegg: disabled paypal audit parser
  • 16:08 ejegg: updated payments-wiki from 4d9d165c40070e036176dba8987243f6dbc7415e to 153418195a45cab820bc2aacf9a4f7dbc9dde768
  • 15:22 logmsgbot: thcipriani@tin Synchronized php-1.26wmf23/extensions/ContentTranslation/modules/entrypoint/ext.cx.interlanguagelink.js: SWAT: Revert "Do not call cxserver to display gray interwiki link" gerrit:239819 (duration: 00m 11s)
  • 15:10 logmsgbot: thcipriani@tin Synchronized robots.txt: SWAT: Remove redundant entries from robots.txt gerrit:239403 (duration: 00m 12s)
  • 14:33 ottomata: restart eventlogging with mysql consumer replace=True (AKA INSERT IGNORE)
  • 14:09 godog: rolling restart restbase in production after cassandra credentials change
  • 12:53 godog: rolling restart cassandra after enabling dc encryption, no nodes in codfw yet
  • 12:01 moritzm: repooled mw1160 (for T104969)
  • 11:54 moritzm: depooled mw1160 (for T104969)
  • 11:51 moritzm: repooled mw1158, mw1159 (for T104969)
  • 11:39 moritzm: depooled mw1158, mw1159 (for T104969)
  • 11:37 moritzm: depooled and repooled mw1156, mw1157 (for T104969)
  • 11:26 moritzm: repooled mw1154, mw1155 (for T104969)
  • 11:21 moritzm: depooled mw1154, mw1155 (for T104969)
  • 10:39 moritzm: repooled mw1026-mw1029 and mw1110-mw1113 (for T104968)
  • 10:24 moritzm: depooled mw1026-mw1029 and mw1110-mw1113 (for T104968)
  • 10:17 moritzm: repooled mw1100-mw1109 (for T104968)
  • 10:17 godog: create restbase user on cassandra cluster
  • 10:06 moritzm: depooled mw1100-mw1109 (for T104968)
  • 09:56 moritzm: repooled mw1140 and mw1142-mw1148 (for T104968)
  • 09:41 moritzm: depooled mw1140 and mw1142-mw1148 (for T104968)
  • 09:36 moritzm: repooled mw1130-mw1139 (for T104968)
  • 09:22 moritzm: depooled mw1130-mw1139 (for T104968)
  • 09:14 moritzm: repooled mw1120-mw1129 (for T104968)
  • 09:02 moritzm: depooled mw1120-mw1129 (for T104968)
  • 08:48 moritzm: repooled mw1189 and mw1200-mw1208 (for T104968)
  • 08:33 moritzm: depooled mw1189 and mw1200-mw1208 (for T104968)
  • 08:29 godog: switch to 'restbase' cassandra user on restbase test cluster
  • 08:29 moritzm: repooled mw1190-mw1195 and mw1197-mw1199 (for T104968)
  • 08:21 _joe_: restarted the logstash agent on logstash1003, OOM'd
  • 08:18 moritzm: depooled mw1190-mw1195 and mw1197-mw1199 (for T104968)
  • 08:07 _joe_: installing the new HHVM package on the api canaries
  • 08:04 moritzm: repooled mw1221-mw1229 (for T104968)
  • 07:53 moritzm: depooled mw1221-mw1229 (for T104968)
  • 07:49 moritzm: repooled mw1230-mw1235 (for T104968)
  • 07:43 _joe_: installing the new hhvm package on the canary appservers
  • 07:08 moritzm: depooled mw1230-mw1235 (for T104968)
  • 04:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Sep 21 04:31:03 UTC 2015 (duration 31m 2s)
  • 02:23 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf23) at 2015-09-21 02:23:12+00:00
  • 02:20 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 25s)
  • 02:06 MaxSem: Maps: created indexes on admin. <3 Postgres :(
  • 01:56 bblack: downtimed eqiad ipv6 text/upload alerts as well, as with mobile above ( 1 301 TLS Redirect - 505 bytes in 1.008 second response time
  • 01:46 bblack: downtimed the "LVS HTTP IPv6 on mobile-lb.eqiad.wikimedia.org_ipv6" alert for now ( https://phabricator.wikimedia.org/T113154 )

2015-09-20

  • 22:34 yuvipanda: reloda pybal on lvs1012
  • 17:01 bblack: repooling cp1046 varnish-be + varnish-be-rand in confctl, fresh storage, purge queue caught up - T113184
  • 16:44 bblack: depooling cp1046 varnish-be + varnish-be-rand in confctl, wiping storage, re-pooling - T113184
  • 07:24 paravoid: temporarily disabling puppet on fermium and applying antispam countermeasures
  • 04:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Sep 20 04:29:16 UTC 2015 (duration 29m 15s)
  • 02:22 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf23) at 2015-09-20 02:22:53+00:00
  • 02:19 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 12s)

2015-09-19

  • 23:12 urandom: begining Cassandra repair on restbase1005 (nodetool repair -pr)
  • 23:08 urandom: begining Cassandra repair on restbase1004 (nodetool repair -pr)
  • 19:56 jynus: restarting once more giblit, last chance
  • 19:04 paravoid: salt rm /etc/systemd/system/txstatsd.service from all cp*, leftover because of ::txstatsd::decommission (removed with 4a1d4e) missing it
  • 19:00 ejegg|away: updated SmashPig from d5895428d1d8ebc5a6e172e8cdec6dbec0b10d85 to d1baa32267eaad7d69b47c657f4853eb306fad6b
  • 18:45 _joe_: restarted gitblit. I will now substitute myself with a clever perl one-liner.
  • 18:38 paravoid: pooling back cp1046 to pybal eqiad/mobile, has stayed stable
  • 18:34 paravoid: reactivating ΒGP with GTT @ eqiad
  • 08:42 _joe_: cp1046 dead on console again, powercycling to inspect it
  • 05:49 logmsgbot: aaron@tin Synchronized php-1.26wmf23/extensions/TitleBlacklist: 80d3a21a51f9c54ed2d94 (duration: 00m 12s)
  • 05:22 paravoid: pybal-depooling cp1046 from eqiad/mobile until further investigation
  • 05:21 paravoid: powercycling cp1046, dead on console
  • 05:01 awight: deploy SmashPig config to limit weekend spam
  • 04:40 awight: update crm from 15ea14f61338ca9f34e9ccb9f56eae14a161380a to 9fa38d06a75363a8009bce7ced190e39c75b68bc
  • 04:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Sep 19 04:28:59 UTC 2015 (duration 28m 57s)
  • 02:23 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf23) at 2015-09-19 02:23:29+00:00
  • 02:20 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 05s)

2015-09-18

  • 23:22 awight: update fundraising-tools from 3e0e3ae799a507b378d0ece3e71631b10b361329 to e1b60fa2c258fd4ff55905b03a4d8886132278c1
  • 20:52 ebernhardson: restart es on elastic1025 to disable dynamic scripting
  • 20:34 gwicke: dropped by_ns indexes on restbase title_revisions tables
  • 19:54 gwicke: finished deploy of restbase daacf4daa
  • 19:45 gwicke: re-enabled puppet on restbase100*
  • 19:35 gwicke: canary deploy of restbase daacf4daa on restbase1001; moving forward so that we can re-enable puppet over the weekend.
  • 18:38 cwdent: updated payments from 1bdd287b083032ff418434ad6bb6920735af918a to 4d9d165c40070e036176dba8987243f6dbc7415e
  • 17:54 logmsgbot: ebernhardson@tin Synchronized wmf-config/CommonSettings.php: Replace insecure es usage with usage of a plugin (duration: 00m 12s)
  • 16:41 mutante: mailman now on 2.1.18 and jessie
  • 16:14 dcausse: elastic in eqiad plugin updates: restarting elastic1021
  • 16:07 paravoid: deactivating ΒGP with GTT @ eqiad
  • 15:20 godog: create restbase user on cassandra test cluster
  • 14:55 dcausse: elastic in eqiad plugin updates: restarting elastic1020
  • 14:22 bblack: committing lvs1007-1012 port/vlan changes for asw-d-eqiad (but leaving all 6 LVS ports in "disabled" state - T112781 )
  • 14:14 bblack: committing lvs1007-12 port/vlan changes for asw-b-eqiad, round 3...
  • 14:11 mutante: sodium - stopped exim - rsyncing lists to fermium
  • 14:10 dcausse: elastic in eqiad plugin updates: restarting elastic1019
  • 14:07 mutante: stopped mailman on sodium
  • 14:01 bblack: rollback on asw-b-eqiad changes above
  • 13:56 bblack: committing eqiad lvs1007-1012 port/vlan changes for asw-b-eqiad
  • 13:20 bblack: committing eqiad lvs1007-12 port/vlan changes for asw-c-eqiad
  • 13:16 bblack: commiting eqiad lvs1007-12 port/vlan changes for asw2-a5-eqiad
  • 13:12 dcausse: elastic in eqiad plugin updates: restarting elastic1018
  • 12:21 godog: restart logstash on logstash1001, OOM in logs
  • 11:55 dcausse: elastic in eqiad plugin updates: restarting elastic1017
  • 11:06 dcausse: elastic in eqiad plugin updates: restarting elastic1016
  • 10:28 moritzm: restarted salt-master on palladium
  • 09:46 moritzm: installed openldap security updates on plutonium
  • 09:37 moritzm: installed openldap security updates on pollux
  • 09:33 dcausse: elastic in eqiad plugin updates: restarting elastic1015
  • 08:22 dcausse: elastic in eqiad plugin updates: restarting elastic1014
  • 07:21 dcausse: elastic in eqiad plugin updates: restarting elastic1013
  • 06:15 dcausse: elastic in eqiad plugin updates: restarting elastic1012
  • 04:37 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Sep 18 04:37:42 UTC 2015 (duration 37m 41s)
  • 02:31 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf23) at 2015-09-18 02:31:49+00:00
  • 02:28 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 08s)
  • 02:21 logmsgbot: krenair@tin Synchronized wmf-config/abusefilter.php: https://gerrit.wikimedia.org/r/#/c/218353/ (duration: 00m 12s)
  • 02:21 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/218353/ (duration: 00m 11s)
  • 02:13 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/237149/ (duration: 00m 12s)
  • 02:07 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234544/ (duration: 00m 12s)
  • 01:58 logmsgbot: ori@tin Synchronized php-1.26wmf22/includes/resourceloader/ResourceLoaderModule.php: I952068d2d: ResourceLoaderModule: cache file content hash (duration: 00m 12s)
  • 01:58 logmsgbot: ori@tin Synchronized php-1.26wmf23/includes/resourceloader/ResourceLoaderModule.php: I952068d2d: ResourceLoaderModule: cache file content hash (duration: 00m 11s)
  • 01:57 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://phabricator.wikimedia.org/T106264 (duration: 00m 12s)
  • 01:36 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/237331/ (duration: 00m 12s)
  • 00:14 ori: restarted logstash on logstash1001

2015-09-17

  • 23:39 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/238978/ (duration: 00m 12s)
  • 23:05 logmsgbot: mattflaschen@tin Synchronized wmf-config/CommonSettings-labs.php: Beta-only change (duration: 00m 12s)
  • 23:04 logmsgbot: mattflaschen@tin Synchronized wmf-config/CommonSettings-labs.php: Beta-only change (duration: 00m 12s)
  • 22:53 gwicke: puppet on restbase cluster disabled since about 21:30 UTC for gradual deploy; ran into minor issue in staging, which is now being addressed, after which deploy will continue
  • 21:22 logmsgbot: ori@tin Synchronized php-1.26wmf22/includes/resourceloader/ResourceLoaderModule.php: I952068d2d: Use MD4 to compute file hash rather than SHA1 (duration: 00m 13s)
  • 21:22 logmsgbot: ori@tin Synchronized php-1.26wmf23/includes/resourceloader/ResourceLoaderModule.php: I952068d2d: Use MD4 to compute file hash rather than SHA1 (duration: 00m 12s)
  • 20:44 logmsgbot: krenair@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 12s)
  • 20:22 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/239206/ (duration: 00m 12s)
  • 19:46 logmsgbot: krenair@tin Synchronized multiversion/MWMultiVersion.php: (no message) (duration: 00m 12s)
  • 19:41 logmsgbot: krenair@tin Synchronized multiversion/MWMultiVersion.php: https://gerrit.wikimedia.org/r/#/c/239181/ (duration: 00m 14s)
  • 19:12 mutante: powercycling unresponse mw1005
  • 18:14 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.26wmf23
  • 17:38 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/includes/registration/ExtensionRegistry.php: registration: Fix merging of array_plus (duration: 00m 13s)
  • 17:35 logmsgbot: legoktm@tin Synchronized php-1.26wmf23/includes/registration/ExtensionRegistry.php: registration: Fix merging of array_plus (duration: 00m 11s)
  • 16:43 chasemp: restart elasticsearch on 1005
  • 16:16 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/235900/ (duration: 00m 12s)
  • 15:15 dcausse: elastic in eqiad plugin updates: restarting elastic1004 (take 2)
  • 15:06 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Enable Suggestions in ptwiki gerrit:238097 (duration: 00m 13s)
  • 14:22 mutante: analytics1029 - Failed to start Hadoop datanode
  • 14:20 mutante: starting hadoop datanode on analytics1029
  • 14:14 _joe_: reimaging tmh1001 to mw1259
  • 14:11 jynus: stopping replication and applying schema change to db1051
  • 14:05 dcausse: elastic in eqiad plugin updates: can't restart elastic1004 (2 timeouts when disabling replication, too much load?), waiting for more shards to rebalance...
  • 13:58 dcausse: elastic in eqiad plugin updates: restarting elastic1004
  • 13:50 moritzm: repooled mw1236-mw1239 (T104968)
  • 13:34 moritzm: depooled mw1236-mw1239 (T104968)
  • 13:26 moritzm: repooled mw1090-mw1099 (T104968)
  • 13:16 moritzm: depooled mw1090-mw1099 (T104968)
  • 13:13 moritzm: repooled mw1080-mw1089 (T104968)
  • 13:05 moritzm: depooled mw1080-mw1089 (T104968)
  • 13:01 moritzm: repooled mw1070-mw1079 (T104968)
  • 12:49 moritzm: depooled mw1070-mw1079 (T104968)
  • 12:35 moritzm: repooled mw1060 and mw1062-mw1069 (T104968)
  • 12:24 moritzm: depooled mw1060 and mw1062-mw1069 (T104968) (not repooled)
  • 12:24 moritzm: repooled mw1060 and mw1062-mw1069 (T104968)
  • 12:16 moritzm: repooled mw1050-mw1059
  • 12:04 moritzm: depooled mw1050-mw1059
  • 11:39 moritzm: repooled mw1040 and mw1042-mw1049 (T104968)
  • 11:36 dcausse: elastic in eqiad plugin updates: restarting elastic1003
  • 11:26 moritzm: typoed earlier entry: "mw1032-mw1039" instead of "mw1032-mw1239"
  • 11:26 moritzm: depooled mw1040 and mw1042-mw1049 (T104968)
  • 11:18 moritzm: repooled mw1030 and mw1032-mw1239 (T104968)
  • 11:03 moritzm: depooled mw1030 and mw1032-mw1239 (T104968)
  • 10:35 moritzm: repooled mw1250-mw1258 (T104968)
  • 10:27 moritzm: depooled mw1250-mw1258 (T104968)
  • 10:25 _joe_: killing temporarily subra
  • 10:24 moritzm: repooled mw1240-mw1249 (T104968)
  • 10:19 _joe_: experimenting with poolcounter issues on subra
  • 10:18 logmsgbot: oblivian@tin Synchronized wmf-config/PoolCounterSettings-codfw.php: Use codfw poolcounters in codfw (duration: 00m 12s)
  • 10:12 moritzm: depooled mw1240-mw1249 (T104968)
  • 10:12 dcausse: elastic in eqiad plugin updates: restarting elastic1002
  • 10:05 logmsgbot: hoo@tin Synchronized wmf-config/: Set 'repoConceptBaseUri' for all Wikibase clients (duration: 00m 13s)
  • 10:00 dcausse: elastic in eqiad plugin updates: unfreezing indices
  • 09:48 dcausse: elastic in eqiad plugin updates: no more groovy in warmers, waiting for few more shards to move in elastic1001 and will unfreeze indices to test warmers
  • 09:39 dcausse: elastic in eqiad plugin updates: deleting warmers manually for old unused indices (eswikisource_content_1415240352, ruwiki_content_1415302164, thwiki_content_1415318677). We will have to remove these indices.
  • 09:39 paravoid: repooling ulsfo US-West traffic back to ulsfo for the first time since May :)
  • 09:01 dcausse: elastic in eqiad plugin updates: updating warmers on all wikis
  • 08:58 paravoid: penalizing ulsfo-eqiad direct MPLS links to higher OSPF weights
  • 08:57 paravoid: adjusting OSPF weights to be latency-based across the US network
  • 08:53 _joe_: removed iptables rules for dropping traffic to helium on mw1017
  • 08:52 dcausse: elastic in eqiad plugin updates: index warmer queries are outdated with inline groovy script, updating warmers on warwiki first to test
  • 08:05 paravoid: eqiad-codfw -> eqiad-eqord-codfw migration
  • 07:49 moritzm: repooled mw1180-mw1188 (T104968)
  • 07:42 dcausse: elastic in eqiad plugin updates: restarting elastic1001
  • 07:42 moritzm: depooled mw1180-mw1188 (T104968)
  • 07:37 moritzm: repooled mw1170-mw1179 (T104968)
  • 07:36 dcausse: elastic in eqiad plugin updates: freezing indices
  • 07:27 moritzm: depooled mw1170-mw1179 (T104968)
  • 07:14 _joe_: uploading new HHVM package
  • 07:07 moritzm: repooled mw1161-1168 (T104968)
  • 06:57 moritzm: depooled mw1161-1168 (T104968)
  • 06:45 moritzm: repooled mw1209-mw1220 with ferm enabled
  • 06:33 moritzm: depooling mw1209-mw1220 (in two steps)
  • 05:47 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Sep 17 05:47:47 UTC 2015 (duration 47m 46s)
  • 03:06 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf23) at 2015-09-17 03:06:33+00:00
  • 03:03 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 30s)
  • 02:45 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-17 02:45:48+00:00
  • 02:39 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 11m 11s)
  • 00:35 cwdent: updated payments from 155cdeb737c01baf62551292764fd2f5a93a9a63 to 1bdd287b083032ff418434ad6bb6920735af918a

2015-09-16

  • 23:27 bblack: updating eqiad switch configs for lvs1007-1012 vlan/trunk settings
  • 23:19 logmsgbot: krenair@tin Synchronized php-1.26wmf23/extensions/MobileFrontend/resources/mobile.overlays/Overlay.less: https://gerrit.wikimedia.org/r/#/c/238865/ (duration: 00m 11s)
  • 23:13 gwicke: started `nodetool rebuild -- eqiad` on restbase-test200{1,2
  • 23:03 cwdent: updated payments from 9fc8ab40b7f70c7b588c2b9e7b5c94b1f893faa1 to 155cdeb737c01baf62551292764fd2f5a93a9a63
  • 22:26 ejegg: updated SmashPig from fdb053efa617162ac9f695e493c390987a069140 to d5895428d1d8ebc5a6e172e8cdec6dbec0b10d85
  • 22:08 urandom: disabling puppet in RESTBase eqiad staging cluster to test new code and config
  • 22:08 ottomata: powercycling analytcis1029, it is down?
  • 20:47 cscott: updated OCG to version 4032a596ce6eb442b02cc6ee9b79263b1eb23275
  • 19:42 ejegg: updated crm from abc34b87ee9d1dbb1176f1929a3d748e1ee5ac7b to 15ea14f61338ca9f34e9ccb9f56eae14a161380a
  • 19:38 ori: Deployed statsv 0bfd9f06f / change I050a12d3b
  • 18:47 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf23
  • 18:38 logmsgbot: twentyafterfour@tin Synchronized php-1.26wmf23: syncing wmf23 ahead of deployment to group1 (duration: 01m 35s)
  • 17:34 paravoid: asw-d-eqiad: toggling RE mastership again
  • 17:26 godog: stop puppet on restbase* to apply https://gerrit.wikimedia.org/r/#/c/238738/ / merge / reenable puppet
  • 16:54 _joe_: turned on the hhvm tmh, stopping the zend ones for testing
  • 16:44 logmsgbot: oblivian@tin Synchronized wmf-config/CommonSettings.php: use ffmpeg whereever possible (duration: 00m 12s)
  • 16:16 bblack: upgrading pybal on lvs400[12]
  • 16:12 bblack: upgrading pybal on lvs400[34], lvs300[34]
  • 16:08 bblack: upgrading pybal on lvs200[123]
  • 16:05 bblack: upgrading pybal on lvs200[456]
  • 15:44 _joe_: uploading pybal 1.10 to reprepro, installing to the test cluster
  • 15:24 moritzm: uploaded debdeploy 0.0.6 to apt.wikimedia.org
  • 15:10 hashar: Started using Nodepool spawned instances. Moved integration-jjb-config-diff Jenkins job to Nodepool with https://gerrit.wikimedia.org/r/#/c/238752/ . See also: https://phabricator.wikimedia.org/T112750
  • 15:05 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: Add m.wikidata.org to wgCrossSiteAJAXdomains (duration: 00m 12s)
  • 14:51 _joe_: experimenting on testwiki for poolcounter failure scenarios
  • 14:45 moritzm: enabled ferm on mw1010 (jobrunner) in eqiad
  • 14:27 paravoid: asw-d-eqiad: toggling RE mastership
  • 14:18 paravoid: disabling/ignoring asw-d-eqiad @ librenms
  • 14:09 jynus: upgrading and restarting db1051
  • 13:57 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1051 for maintenance (duration: 00m 12s)
  • 13:40 urandom: initiating Cassandra repair on restbase1007 (nodetool repair -pr)
  • 13:40 logmsgbot: catrope@tin Synchronized php-1.26wmf23: (no message) (duration: 01m 37s)
  • 13:35 moritzm: repooled mw1149-mw1151 (with ferm enabled)
  • 13:24 moritzm: depooled mw1149-mw1151 (for enabling ferm)
  • 13:19 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Reverting depool of es1055 (duration: 00m 12s)
  • 13:15 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1055 for maintenance (duration: 00m 12s)
  • 13:03 paravoid: disabling asw-d-eqiad xe-8/0/23, xe-8/0/24, xe-8/0/25, xe-8/0/26, xe-8/0/27, xe-8/0/28; servers reboot-looping -> asw-d's SNMP unhappy -> librenms unhappy -> faidon's mailbox unhappy
  • 12:48 moritzm: repooled mw1115-mw1117, mw1119 (with ferm enabled)
  • 12:42 moritzm: depooling mw1115-mw1117, mw1119 (mw1118 was already depooled) to enable ferm
  • 11:32 moritzm: repooled mw1019-mw1025 with ferm enabled
  • 11:24 jynus: making db1069 a sibling of db1055 (s1)
  • 11:13 godog: create restbase user on cassandra test cluster
  • 11:07 moritzm: depooled mw1019-mw1025 (to enable ferm)
  • 10:52 logmsgbot: catrope@tin Synchronized php-1.26wmf23: (no message) (duration: 02m 04s)
  • 10:49 logmsgbot: catrope@tin Synchronized php-1.26wmf22: (no message) (duration: 02m 12s)
  • 10:48 jynus: reenabling semisync on db1072 and db1073
  • 10:47 logmsgbot: catrope@tin scap aborted: (no message) (duration: 00m 21s)
  • 10:47 logmsgbot: catrope@tin Started scap: (no message)
  • 10:24 logmsgbot: catrope@tin Synchronized php-1.26wmf23/includes/changes/EnhancedChangesList.php: T112738 (duration: 00m 12s)
  • 10:09 logmsgbot: aude@tin Synchronized arbitraryaccess.dblist: (no message) (duration: 00m 11s)
  • 09:37 awight: ruthlessly disabled PayPal IPN listener
  • 08:12 moritzm: repooled mw1153 with ferm enabled
  • 07:57 jynus: truncated some tables from ContentTranslation extension on x1
  • 07:57 moritzm: depooled mw1153 (it's an image scaler, of course) to enable ferm
  • 07:56 moritzm: depooled mw1153 (videoscaler) to enable ferm
  • 06:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Sep 16 06:31:58 UTC 2015 (duration 31m 57s)
  • 03:28 logmsgbot: ori@tin Synchronized php-1.26wmf22/vendor/monolog/monolog/src/Monolog/Logger.php: Iccfda47689: monolog: Don't waste milliseconds counting microseconds (duration: 00m 12s)
  • 03:27 logmsgbot: ori@tin Synchronized php-1.26wmf23/vendor/monolog/monolog/src/Monolog/Logger.php: Iccfda47689: monolog: Dont waste milliseconds counting microseconds ; sync-file php-1.26wmf22/vendor/monolog/monolog/src/Monolog/Logger.php Iccfda47689: monolog: Dont waste milliseconds counting microseconds (duration: 00m 12s)
  • 03:12 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf23) at 2015-09-16 03:12:08+00:00
  • 03:05 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 10m 30s)
  • 02:38 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-16 02:38:48+00:00
  • 02:35 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 07m 02s)
  • 01:03 logmsgbot: krinkle@tin Synchronized php-1.26wmf23/resources/src/mediawiki/mediawiki.js: hotfix Ia2fcd13f4 (duration: 00m 12s)
  • 00:29 logmsgbot: krinkle@tin Synchronized php-1.26wmf22/resources/src/mediawiki/mediawiki.js: hotfix Ia2fcd13f4 (duration: 00m 11s)
  • 00:15 logmsgbot: legoktm@tin Synchronized php-1.26wmf23/extensions/CentralAuth/includes/: Use set() for tokens with unique keys (duration: 00m 12s)
  • 00:14 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/extensions/CentralAuth/includes/: Use set() for tokens with unique keys (duration: 00m 12s)
  • 00:11 bblack: reinstalling lvs400[12] to jessie (traffic on 400[34], already jessie)
  • 00:08 logmsgbot: krenair@tin Synchronized php-1.26wmf23/extensions/VisualEditor/modules/ve-mw/ui/styles/dialogs: https://gerrit.wikimedia.org/r/#/c/238646/ (duration: 00m 12s)

2015-09-15

  • 23:51 logmsgbot: krenair@tin Synchronized php-1.26wmf22/extensions/WikimediaEvents/modules/ext.wikimediaEvents.geoFeatures.js: https://gerrit.wikimedia.org/r/#/c/238617/ (duration: 00m 12s)
  • 23:48 logmsgbot: krenair@tin Synchronized php-1.26wmf23/extensions/WikimediaEvents/modules/ext.wikimediaEvents.geoFeatures.js: https://gerrit.wikimedia.org/r/#/c/238618/ (duration: 00m 12s)
  • 23:42 logmsgbot: krenair@tin Synchronized wmf-config/mobile.php: https://gerrit.wikimedia.org/r/#/c/238543/ (duration: 00m 14s)
  • 23:42 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/238543/ (duration: 00m 12s)
  • 23:40 logmsgbot: krenair@tin Synchronized php-1.26wmf22/includes/resourceloader/ResourceLoader.php: https://gerrit.wikimedia.org/r/#/c/238544/ (duration: 00m 11s)
  • 23:38 logmsgbot: krenair@tin Synchronized php-1.26wmf23/includes/resourceloader/ResourceLoader.php: https://gerrit.wikimedia.org/r/#/c/238545/ (duration: 00m 11s)
  • 23:24 yurik: deployed kartotherian & tilerator
  • 23:22 logmsgbot: krenair@tin Synchronized php-1.26wmf22/extensions/EventLogging/modules/ext.eventLogging.core.js: https://gerrit.wikimedia.org/r/#/c/238512/ (duration: 00m 12s)
  • 23:16 logmsgbot: krenair@tin Synchronized php-1.26wmf23/extensions/EventLogging/modules/ext.eventLogging.core.js: https://gerrit.wikimedia.org/r/#/c/238513/ (duration: 00m 12s)
  • 21:15 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf22/extensions/WikimediaEvents/: touch files edited in I0cb6fe37e and re-sync to cluster (duration: 00m 13s)
  • 21:13 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.26wmf23
  • 21:10 logmsgbot: twentyafterfour@tin Finished scap: sync 1.26wmf23 to testwiki, once more because mw1010 overloaded (duration: 03m 52s)
  • 21:07 logmsgbot: twentyafterfour@tin Started scap: sync 1.26wmf23 to testwiki, once more because mw1010 overloaded
  • 21:05 logmsgbot: twentyafterfour@tin Finished scap: sync 1.26wmf23 to testwiki, again (duration: 47m 49s)
  • 20:47 mutante: mw1010 - extremely slow,finally got on and attempted to restart hhvm. load going down
  • 20:17 logmsgbot: twentyafterfour@tin Started scap: sync 1.26wmf23 to testwiki, again
  • 20:17 logmsgbot: twentyafterfour@tin scap aborted: sync 1.26wmf23 to testwiki (duration: 82m 58s)
  • 20:05 ottomata: restarted mysql (and oozie) on analytics1027 to start mysql binlogging
  • 18:54 logmsgbot: twentyafterfour@tin Started scap: sync 1.26wmf23 to testwiki
  • 16:55 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1003, es1004, es1007 and es1010 for decommision (duration: 00m 12s)
  • 16:40 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Revert depool db1055 for maintenance (duration: 00m 11s)
  • 16:39 ottomata: reinstalling analytics1015
  • 16:32 RoanKattouw: Putting wmf22 versions of Echo and MobileFrontend on mw1017 for testing
  • 16:30 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf22/extensions/WikimediaEvents/WikimediaEvents.php: touch file that is serving old version in prod (duration: 00m 12s)
  • 16:29 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf22/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSuggest.js: Touch file that is serving old version in prod (duration: 00m 12s)
  • 16:27 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1055 for maintenance (duration: 00m 11s)
  • 16:11 bblack: traffic DNS depooled out of codfw for now T112639
  • 15:38 logmsgbot: thcipriani@tin Synchronized wmf-config: SWAT: CX: Enable suggestion for testwiki (part 2) gerrit:237327 (duration: 00m 13s)
  • 15:37 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Enable suggestion for testwiki (part 1) gerrit:237327 (duration: 00m 12s)
  • 15:31 logmsgbot: thcipriani@tin Synchronized php-1.26wmf22/extensions/UploadWizard/resources/jquery/jquery.mwCoolCats.js: SWAT: Do not fail horribly when invalid categories are passed gerrit:238421 (duration: 00m 12s)
  • 15:14 logmsgbot: thcipriani@tin Synchronized wmf-config/PoolCounterSettings-eqiad.php: SWAT: poolcounter: enable connect_timeout for testwiki gerrit:238109 (duration: 00m 19s)
  • 15:09 logmsgbot: thcipriani@tin Synchronized wmf-config/PoolCounterSettings-codfw.php: SWAT: poolcounter: add connect_timeout in codfw gerrit:238108 (duration: 00m 12s)
  • 15:06 logmsgbot: thcipriani@tin Synchronized wmf-config/Wikibase.php: SWAT: Exclude Flow topic boards and Draft NS from Special:UnconnectedPages gerrit:229197 (duration: 00m 11s)
  • 14:51 godog: bounce cassandra on test cluster to deploy https://gerrit.wikimedia.org/r/236391
  • 14:22 cmjohnson1: swapped disk on db1043
  • 13:12 moritzm: repool mw1114 (with ferm enabled)
  • 13:11 bblack: failing over LVS service in ulsfo to secondariess (400[12] pybal stopped, traffic on jessie-based 400[34])
  • 12:53 moritzm: depooled mw1114 (for enabling ferm)
  • 11:42 moritzm: repool mw1018 (with ferm enabled)
  • 11:23 moritzm: depooled mw1018 (for enabling ferm)
  • 08:53 _joe_: created a 100 G partition on a LV on copper, for /tmp
  • 08:24 godog: bounce ms-be2006, xfs
  • 08:22 moritzm: bumped default size of iptables connection tracking table to 256k
  • 06:10 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Sep 15 06:10:52 UTC 2015 (duration 10m 51s)
  • 02:46 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-15 02:46:50+00:00
  • 02:40 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 10m 53s)
  • 02:18 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/extensions/MobileFrontend: Revert Echo to 1.26wmf21 state (duration: 00m 11s)
  • 02:18 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/extensions/Echo: Revert Echo to 1.26wmf21 state (duration: 00m 12s)
  • 01:30 logmsgbot: krinkle@tin Synchronized php-1.26wmf22/resources/src: T112287 (duration: 00m 11s)
  • 00:49 bblack: reinstalling lvs300[34] to jessie
  • 00:43 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-labs.php: noop sync of labs config change (duration: 00m 11s)
  • 00:03 logmsgbot: tstarling@tin Synchronized php-1.26wmf22/extensions/ParsoidBatchAPI: for I56d28e9a for RT testing, not live yet (duration: 00m 13s)

2015-09-14

  • 23:27 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf22/extensions/WikimediaEvents/: Change bucket selection methods in CompletionSuggestions AB test (duration: 00m 12s)
  • 23:23 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf22/extensions/UploadWizard/: Swat out badtoken fix to UploadWizard in 1.26wmf22 (duration: 00m 12s)
  • 22:37 yurik: deployed tilerator
  • 21:15 logmsgbot: ori@tin Synchronized php-1.26wmf22/extensions/TitleBlacklist: Ie44fcb500: Avoid checking blacklists in isBlacklisted() for existing titles (duration: 00m 12s)
  • 21:15 mutante: labnodepool1001 - re-enable puppet and nodepool
  • 20:59 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/extensions/Echo/: Hack around OOUI's icon pack being too large by creating our own (duration: 00m 12s)
  • 20:53 cscott: updated OCG to version 5811056e28f2bc6408b6da96095352ab381bb11f
  • 20:21 andrewbogott: graceful’d apache2 on labcontrol1001
  • 20:15 subbu: deployed parsoid sha 3d5f4359
  • 19:25 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/extensions/Echo/: Only load nojs Special:Notifications styles on the special page (duration: 00m 12s)
  • 18:05 urandom: rebuilding restbase-test2001.codfw (nodetool rebuild -- eqiad)
  • 16:12 logmsgbot: catrope@tin Synchronized php-1.26wmf22/extensions/Echo/: For real this time (duration: 00m 11s)
  • 16:06 ottomata: stopping hdfs journalnode on analytics1011 to copy journal edits to new journalnodes on analytics1035 and analytics1052
  • 15:46 godog: switch to openjdk-8 and bounce cassandra on restbase-test200*
  • 15:39 bblack: reinstalling lvs4003, lvs4004 (jessie upgrade: T96375) (typo earlier)
  • 15:39 bblack: reinstalling lvs4003, lvs4003 (jessie upgrade: T96375)
  • 15:34 logmsgbot: catrope@tin Synchronized php-1.26wmf22/extensions/Echo/: SWAT (duration: 00m 13s)
  • 15:05 logmsgbot: krenair@tin Synchronized .gitignore: https://gerrit.wikimedia.org/r/#/c/237529/ (duration: 00m 13s)
  • 15:05 logmsgbot: krenair@tin Synchronized docroot/noc: https://gerrit.wikimedia.org/r/#/c/237529/ (duration: 00m 12s)
  • 15:04 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/237529/ (duration: 00m 11s)
  • 15:02 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234980/ (duration: 00m 12s)
  • 13:38 godog: stop puppet on restbase-test2001 and turn up cassandra
  • 12:56 bblack: rebooting lvs2006 to test eth hw params stuff...
  • 12:55 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/238125/ (duration: 00m 13s)
  • 12:50 urandom: starting Cassandra repair on restbase1003 (nodetool repair -pr)
  • 12:32 godog: enable dc encryption on cassandra test cluster and rolling restart
  • 11:33 mobrovac: citoid deploying d569951
  • 10:35 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1002, es1005, es1008 (duration: 00m 12s)
  • 10:04 jynus: db1029 (x1-master) temporarily saturated by connections- flow was unresponsive for 10 minutes; migration partially aborted
  • 09:08 jynus: applying schema change to flowdb
  • 08:52 godog: rename cassandra test cluster and restart
  • 08:44 godog: silence mendelevium for today, status unclear T111532
  • 08:30 jynus: endinf profiling and executing pt-query-digest on db1043 [ETA:4h]
  • 07:52 godog: reboot ms-be1010 to pick up disk ordering change
  • 04:48 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Sep 14 04:47:58 UTC 2015 (duration 47m 57s)
  • 02:29 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-14 02:29:48+00:00
  • 02:26 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 06m 59s)
  • 01:31 Krinkle: mwscript deleteEqualMessages.php --wiki sqwiki

2015-09-13

  • 06:02 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Sep 13 06:02:52 UTC 2015 (duration 2m 51s)
  • 02:40 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-13 02:40:43+00:00
  • 02:34 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 10m 13s)

2015-09-12

  • 20:15 ori: Rolling back Echo to 1.26wmf21 branch on mw1017 (testwiki) to measure increase in render-blocking CSS size
  • 19:21 urandom: performing Cassandra repair on restbase1002 (nodetool repair -pr)
  • 14:50 jynus: phab.wmfusercontent.org has been temporarily switched to phab.wikivoyage.org due to cert issues
  • 04:52 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Sep 12 04:52:01 UTC 2015 (duration 52m 0s)
  • 02:35 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-12 02:35:36+00:00
  • 02:32 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 06m 54s)

2015-09-11

  • 21:21 hashar: shutdown nodepool on labnodepool1001.eqiad.wmnet until monday
  • 18:01 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/extensions/Echo/: Echo regression fixes #2 (duration: 00m 12s)
  • 16:43 logmsgbot: krinkle@tin Synchronized php-1.26wmf22/resources/src/mediawiki/mediawiki.js: T112232 (duration: 00m 12s)
  • 16:37 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/extensions/Echo/: Echo regression backports (duration: 00m 12s)
  • 16:35 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/resources/src/mediawiki/mediawiki.js: resourceloader: Document internal mw.loader#jobs property (again) (duration: 00m 13s)
  • 16:33 legoktm: ssh: connect to host mw1156.eqiad.wmnet port 22: Connection timed out
  • 16:32 paravoid: powercycling mw1156, multiple kernel backtraces in console output
  • 16:32 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/resources/src/mediawiki/mediawiki.js: resourceloader: Document internal mw.loader#jobs property (duration: 01m 07s)
  • 16:15 cmjohnson1: mw1031 rebooting for f/w update
  • 16:07 bblack: enabled LRO+GRO on lvs200[123], starting pybal there again ([456] testing looks good so far)
  • 15:45 bblack: enabled LRO+GRO on lvs200[456] (backups). Stopping pybal on lvs200[123] to test...
  • 15:11 cmjohnson1: swapping pem2 cr2-eqiad
  • 10:03 jynus: starting nodepool in labnodepool1001
  • 09:21 jynus: starting profiling of phabricator db (db1043). Very low overhead.
  • 06:03 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Sep 11 06:03:00 UTC 2015 (duration 2m 59s)
  • 02:41 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-11 02:41:24+00:00
  • 02:34 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 11m 18s)
  • 01:16 logmsgbot: ori@tin Synchronized php-1.26wmf22/extensions/TitleBlacklist: 9bf13dbe0b, 3203b045f7 (duration: 00m 12s)

2015-09-10

  • 23:52 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/237064/ (duration: 00m 11s)
  • 23:47 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/237056/ (duration: 00m 11s)
  • 23:13 logmsgbot: krenair@tin Synchronized wmf-config/wikitech.php: https://gerrit.wikimedia.org/r/221825 (duration: 00m 13s)
  • 23:04 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/224771 (duration: 00m 12s)
  • 21:13 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/extensions/Echo/modules: Align popup footer buttons to take 50% width each (duration: 00m 15s)
  • 20:50 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: depool es1001; increase weight of es1015 and es1019 (duration: 00m 19s)
  • 20:47 ottomata: restarting eventlogging with 12 client side processors on eventlog1001
  • 20:31 ottomata: turning off varnishncsa eventlogging eventlistener instances on frontend caches, it is now superseded by varnishkafka
  • 20:28 mutante: killed/restarted ganglia aggregator process for mobile-cache, upload cache, misc esams ...
  • 20:22 jynus: last SCAP failed on 266/466 hosts
  • 20:21 mutante: killed/restarted ganglia aggregator process for text-caches esams on hooft
  • 20:17 yurik: deployed kartotherian
  • 20:08 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1001; increase weight of es1015 and es1019 (duration: 00m 11s)
  • 19:11 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.26wmf22
  • 19:09 logmsgbot: twentyafterfour@tin Synchronized php-1.26wmf22/extensions/CentralNotice: deploy https://gerrit.wikimedia.org/r/#/c/237458/ (duration: 00m 12s)
  • 18:57 twentyafterfour: restarted phd on iridium
  • 18:51 logmsgbot: twentyafterfour@tin Synchronized php-1.26wmf22/extensions/Wikidata: Deploy wikidata patch: https://gerrit.wikimedia.org/r/#/c/237449/ (duration: 00m 19s)
  • 18:23 logmsgbot: twentyafterfour@tin Synchronized php-1.26wmf22: deploy https://gerrit.wikimedia.org/r/#/c/237440/ (duration: 01m 42s)
  • 18:09 cmjohnson1: reseating pem2 cr2-eqiad
  • 16:52 akosiaris: puppetswat done
  • 16:50 mobrovac: restbase rolling restart of rb100x
  • 16:49 mobrovac: restbase enabled puppet on rb100x
  • 16:13 akosiaris: started puppetSWAT
  • 16:10 logmsgbot: marktraceur@tin Finished scap: Make sure codfw got the last few patches sync'd to it (duration: 07m 36s)
  • 16:03 logmsgbot: marktraceur@tin Started scap: Make sure codfw got the last few patches sync'd to it
  • 16:02 logmsgbot: marktraceur@tin Synchronized php-1.26wmf22/: [SWAT] [wmf22] Revert opera redirect loop fix that caused redirect loops in Firefox (duration: 02m 30s)
  • 15:55 mobrovac: restbase disabled puppet on rb100x
  • 15:45 logmsgbot: marktraceur@tin Synchronized php-1.26wmf22/extensions/UploadWizard/resources/transports/mw.FormDataTransport.js: [SWAT] [wmf22] Always set 'offset' with chunked uploads, even for first chunk (offset == 0) (duration: 02m 21s)
  • 15:26 ottomata: started hadoop decomission of analytics1016
  • 15:21 logmsgbot: marktraceur@tin Synchronized wmf-config/: [SWAT] Attempting another sync to mw2187 hoping it's up now (duration: 02m 22s)
  • 15:05 logmsgbot: marktraceur@tin Synchronized wmf-config/: [SWAT] [config] Beta: Enable Content Translation suggestions (duration: 02m 22s)
  • 13:35 moritzm: enabled ferm on mediawiki app servers in codfw
  • 13:30 jynus: performing schema change and maintenance on officewiki and public all wikis with flow enabled
  • 12:51 moritzm: enabled ferm on mediawiki API servers in codfw
  • 12:36 moritzm: enabled ferm on mediawiki video scalers, image scalers and job runners in codfw
  • 09:20 mobrovac: restbase deploying 0182962
  • 06:13 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Sep 10 06:13:14 UTC 2015 (duration 13m 13s)
  • 03:02 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-10 03:02:45+00:00
  • 02:59 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 06m 10s)
  • 02:51 logmsgbot: krenair@tin Synchronized php-1.26wmf22/extensions/WikimediaMaintenance/dumpInterwiki.php: https://gerrit.wikimedia.org/r/237304 (duration: 00m 11s)
  • 02:50 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/WikimediaMaintenance/dumpInterwiki.php: https://gerrit.wikimedia.org/r/237303 (duration: 00m 10s)
  • 02:43 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-10 02:43:20+00:00
  • 02:36 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 10m 45s)
  • 02:24 logmsgbot: krinkle@tin Synchronized php-1.26wmf21/resources/src/mediawiki/mediawiki.js: Ic0b1fb64ee7 backport (duration: 00m 12s)
  • 01:04 logmsgbot: ori@tin Synchronized php-1.26wmf21/extensions/NavigationTiming: I2605c746b: Ensure timings are reported after the page has loaded (duration: 00m 13s)
  • 01:03 logmsgbot: ori@tin Synchronized php-1.26wmf22/extensions/NavigationTiming: I2605c746b: Ensure timings are reported after the page has loaded (duration: 00m 12s)
  • 00:54 mutante: powercycling unresponsive mw1154

2015-09-09

  • 23:34 logmsgbot: krenair@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 12s)
  • 23:31 logmsgbot: krenair@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 12s)
  • 23:29 logmsgbot: krenair@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 12s)
  • 23:23 MaxSem: deployed Kartotherian config updates
  • 23:23 logmsgbot: catrope@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 11s)
  • 23:22 RoanKattouw: Running updateinterwikicache
  • 23:13 logmsgbot: catrope@tin Synchronized php-1.26wmf22/extensions/WikimediaMaintenance: SWAT (duration: 00m 13s)
  • 23:13 logmsgbot: catrope@tin Synchronized php-1.26wmf22/extensions/Flow: SWAT (duration: 00m 32s)
  • 23:12 logmsgbot: catrope@tin Synchronized php-1.26wmf21/extensions/WikimediaMaintenance: SWAT (duration: 00m 14s)
  • 23:12 logmsgbot: catrope@tin Synchronized php-1.26wmf21/extensions/Flow: SWAT (duration: 00m 29s)
  • 20:17 subbu: deployed parsoid version ffd0b444
  • 18:15 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf22
  • 16:47 andrewbogott: systemctl stop nodepool on labnodepool1001
  • 16:06 logmsgbot: aude@tin Synchronized database lists: Remove unused usagetracking.dblist (duration: 00m 12s)
  • 16:01 logmsgbot: krenair@tin Synchronized robots.txt: https://gerrit.wikimedia.org/r/#/c/236200/ (duration: 00m 12s)
  • 15:57 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/236701/ - noop (duration: 00m 12s)
  • 15:56 ejegg: updated payments from from 4c5e30288370db926cbbf7a7528edb9c41c65716 to 9fc8ab40b7f70c7b588c2b9e7b5c94b1f893faa1
  • 15:50 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/237104/ (duration: 00m 12s)
  • 15:46 logmsgbot: krenair@tin Synchronized wmf-config/Wikibase.php: https://gerrit.wikimedia.org/r/#/c/237097/ (duration: 00m 12s)
  • 15:46 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/237097/ (duration: 00m 12s)
  • 15:43 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf21/resources/src/mediawiki/mediawiki.searchSuggest.js: Enable completion suggester AB experiment (duration: 00m 12s)
  • 15:43 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf21/extensions/WikimediaEvents/: Enable suggester AB experiement (duration: 00m 11s)
  • 15:38 logmsgbot: krenair@tin Synchronized php-1.26wmf22/extensions/Wikidata: https://gerrit.wikimedia.org/r/#/c/237091/ (duration: 00m 21s)
  • 15:26 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234425/ (duration: 00m 12s)
  • 15:21 logmsgbot: krenair@tin Synchronized wmf-config/logging.php: https://gerrit.wikimedia.org/r/#/c/236994/ (duration: 00m 12s)
  • 15:15 bd808: Running sync-common manually on mw2187.codfw.wmnet. Host is missing l10n cache files
  • 15:12 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/236025/ (duration: 00m 11s)
  • 15:10 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/236042/ (duration: 00m 13s)
  • 14:03 mutante: beginning mailman migration - expect lists to be down
  • 13:14 moritzm: enabled ferm on test.wikipedia.org (mw1017)
  • 13:05 urandom: issuing Cassandra repair on restbase1001 (nodetool repair -pr)
  • 13:02 moritzm: enabled ferm on various initial mediawiki hosts in codfw: videoscaler (mw2007), appserver (mw200[89]), jobrunner (mw2081), api (mw2050), imagescaler (mw2086)
  • 10:33 logmsgbot: aude@tin Synchronized wmf-config/CommonSettings.php: Remove unused usagetracking tag (duration: 00m 11s)
  • 10:30 logmsgbot: aude@tin Synchronized wmf-config/Wikibase.php: (no message) (duration: 00m 12s)
  • 10:26 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: rv usage tracking (duration: 00m 12s)
  • 10:23 logmsgbot: aude@tin Synchronized usagetracking.dblist: Enable usage tracking on commons and test2wiki (duration: 00m 11s)
  • 10:21 logmsgbot: aude@tin Synchronized wikidataclient.dblist: Sorted dblist (duration: 00m 12s)
  • 09:41 logmsgbot: aude@tin Synchronized usagetracking.dblist: Enable usage tracking on Wikinews (duration: 00m 12s)
  • 08:35 moritzm: installed spice security updates on labvirt*, ganeti* and labnodepool1001
  • 05:11 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Sep 9 05:11:28 UTC 2015 (duration 11m 27s)
  • 02:55 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-09 02:55:24+00:00
  • 02:52 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 05m 34s)
  • 02:31 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-09 02:31:50+00:00
  • 02:28 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 06m 44s)
  • 00:00 logmsgbot: catrope@tin Finished scap: Need to update i18n for a new Echo message (duration: 23m 08s)

2015-09-08

  • 23:36 logmsgbot: catrope@tin Started scap: Need to update i18n for a new Echo message
  • 23:36 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings-labs.php: SWAT (duration: 00m 10s)
  • 23:36 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings-labs.php: SWAT (duration: 00m 13s)
  • 23:34 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: SWAT (duration: 00m 12s)
  • 23:33 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: SWAT (duration: 00m 12s)
  • 23:20 logmsgbot: catrope@tin Synchronized php-1.26wmf22/extensions/WikimediaEvents/: SWAT (duration: 00m 11s)
  • 23:20 logmsgbot: catrope@tin Synchronized php-1.26wmf22/extensions/Echo/: SWAT (duration: 00m 14s)
  • 23:14 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings-labs.php: (no message) (duration: 00m 11s)
  • 22:13 logmsgbot: krinkle@tin Synchronized php-1.26wmf21/extensions/NavigationTiming: re-apply patch 1/2 (jscs) (duration: 00m 12s)
  • 21:36 logmsgbot: krinkle@tin Synchronized php-1.26wmf21/extensions/NavigationTiming: temporarily revert T109756 (duration: 00m 11s)
  • 21:02 csteipp: deployed patches for T108616 T91850 T91205 to wmf21 & 22
  • 20:45 bblack: upgrading nginx to 1.9.4 on cp*
  • 20:38 logmsgbot: ori@tin Synchronized multiversion: wikimedia/cdb 1.2.0 → 1.3.0 (duration: 00m 12s)
  • 20:38 logmsgbot: ori@tin Synchronized php-1.26wmf22/vendor: wikimedia/cdb 1.2.0 → 1.3.0 (duration: 00m 15s)
  • 20:37 logmsgbot: ori@tin Synchronized php-1.26wmf21/vendor: wikimedia/cdb 1.2.0 → 1.3.0 (duration: 00m 14s)
  • 20:07 logmsgbot: aude@tin Finished scap: Update group0 to new Wikidata branch (duration: 24m 27s)
  • 19:42 logmsgbot: aude@tin Started scap: Update group0 to new Wikidata branch
  • 19:14 logmsgbot: twentyafterfour@tin Synchronized php-1.26wmf21/: sync php-1.26wmf21 as well (duration: 02m 31s)
  • 19:10 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.26wmf22
  • 18:55 ejegg: updated payments from 6ac552f280fb839069d117386c4ecbe9e52f90a8 to 4c5e30288370db926cbbf7a7528edb9c41c65716
  • 18:50 logmsgbot: twentyafterfour@tin Finished scap: testwiki to 1.26wmf22 (duration: 29m 29s)
  • 18:20 logmsgbot: twentyafterfour@tin Started scap: testwiki to 1.26wmf22
  • 18:01 ejegg: rolled back payments to 6ac552f280fb839069d117386c4ecbe9e52f90a8
  • 17:59 ejegg: updated payments from 6ac552f280fb839069d117386c4ecbe9e52f90a8 to 4c5e30288370db926cbbf7a7528edb9c41c65716
  • 17:43 moritzm: enabled ferm on remaining hadoop workers (analytics1040-analytics1057)
  • 17:09 logmsgbot: krinkle@tin Synchronized php-1.26wmf21/extensions/NavigationTiming: T109756 (duration: 00m 11s)
  • 16:56 logmsgbot: krinkle@tin Synchronized php-1.26wmf21/extensions/CentralAuth: T108253 sul2 token store (duration: 00m 12s)
  • 16:16 logmsgbot: ori@tin Synchronized php-1.26wmf21/vendor: I5af46eb3: wikimedia/cdb 1.0.1 → 1.2.0 (duration: 00m 14s)
  • 15:43 logmsgbot: ori@tin Synchronized multiversion: wikimedia/cdb 1.0.1 → 1.2.0 (duration: 00m 12s)
  • 15:21 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/236785/ (duration: 00m 12s)
  • 15:17 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234910/ (duration: 00m 12s)
  • 14:41 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Pool es1015 and es1019 (duration: 00m 11s)
  • 14:30 moritzm: enabled ferm on hadoop workers up to analytics1039
  • 12:41 godog: change whisper aggregation for 'sum.wsp' files T111170
  • 10:48 moritzm: restarted salt master on palladium
  • 10:32 logmsgbot: aude@tin Synchronized usagetracking.dblist: Enable usage tracking on Wikibooks (duration: 00m 11s)
  • 09:55 moritzm: uploaded debdeploy 0.0.5 to carbon
  • 04:37 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Sep 8 04:37:06 UTC 2015 (duration 37m 5s)
  • 02:23 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-08 02:23:51+00:00
  • 02:20 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 06m 30s)
  • 00:46 Krinkle: mwscript deleteEqualMessages.php --wiki eswiki

2015-09-07

  • 21:45 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/236682/ (duration: 00m 12s)
  • 21:44 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/WikimediaEvents/WikimediaEvents.php: https://gerrit.wikimedia.org/r/#/c/236196/1 (duration: 00m 12s)
  • 21:42 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/WikiEditor: https://gerrit.wikimedia.org/r/#/c/236197/1 and https://gerrit.wikimedia.org/r/#/c/236679/ (duration: 00m 12s)
  • 18:15 andrewbogott: graceful’d apache, restarted keystone on labcontrol1001
  • 15:41 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/MobileFrontend/includes/MobileFrontend.hooks.php: https://gerrit.wikimedia.org/r/#/c/236558/ (duration: 00m 12s)
  • 15:11 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1004, pool es1018 (duration: 00m 10s)
  • 10:04 godog: powercycle ms-be1003, loadavg skyrocketed
  • 08:13 hashar: Jenkins upgraded to latest LTS ( https://phabricator.wikimedia.org/T111326 )
  • 08:05 hashar: Upgrading Jenkins
  • 04:33 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Sep 7 04:33:11 UTC 2015 (duration 33m 10s)
  • 02:29 Krinkle: mwscript deleteEqualMessages.php --wiki pmswiki
  • 02:23 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-07 02:23:27+00:00
  • 02:20 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 06m 22s)

2015-09-06

  • 04:27 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Sep 6 04:27:57 UTC 2015 (duration 27m 56s)
  • 02:23 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-06 02:23:08+00:00
  • 02:19 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 06m 14s)

2015-09-05

  • 23:37 Krinkle: mwscript deleteEqualMessages.php --wiki fywiktionary
  • 04:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Sep 5 04:31:34 UTC 2015 (duration 31m 33s)
  • 02:30 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-05 02:30:06+00:00
  • 02:27 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 05m 53s)

2015-09-04

  • 23:52 logmsgbot: mattflaschen@tin Synchronized wmf-config/InitialiseSettings-labs.php: Beta-only change (duration: 00m 12s)
  • 23:52 logmsgbot: mattflaschen@tin Synchronized wmf-config/CommonSettings-labs.php: Beta-only change (duration: 00m 11s)
  • 22:49 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/Citoid: https://gerrit.wikimedia.org/r/#/c/236218/ and https://gerrit.wikimedia.org/r/#/c/236222/ (duration: 00m 12s)
  • 21:55 urandom: bouncing Cassandra on restbase1001 to restore default GC settings
  • 18:36 logmsgbot: krenair@tin Synchronized w/static/images/project-logos/ukwikivoyage.png: https://gerrit.wikimedia.org/r/#/c/236063/ (duration: 00m 11s)
  • 18:06 logmsgbot: krinkle@tin Synchronized php-1.26wmf21/extensions/WikimediaEvents/modules/ext.wikimediaEvents.statsd.js: Ib98988f67ef (duration: 00m 11s)
  • 17:35 MaxSem: Maps: dropped duplicate index on water_polygons
  • 16:27 jynus: cloning es1 mysql data from es1004 to es1018 [ETA:16h]
  • 16:11 paravoid: updating firewall border ACLs and BGP border filters across all cr
  • 15:42 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1002, es1016; Depool es1004 (duration: 00m 11s)
  • 15:35 godog: python varnishlog collector + gdb running on cp1052 for debugging T83580
  • 12:55 moritzm: restarted salt-master on palladium
  • 12:47 moritzm: uploaded debdeploy 0.0.4 to carbon
  • 10:18 logmsgbot: kartik@tin Synchronized php-1.26wmf21/extensions/ContentTranslation/api/ApiContentTranslationPublish.php: php-1.26wmf21/extensions/ContentTranslation/extension.json T111490:Use the VirtualRESTService to configure CX (duration: 00m 12s)
  • 09:16 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-fr-ca_1.0.3~r61329-1
  • 09:16 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-eo-fr_0.9.0~r28336-1
  • 09:16 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-eo-es_0.9.1~r60655-1
  • 09:16 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-eo-ca_0.9.1~r60655-1
  • 09:16 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-ca-it_0.1.1~r57554-1
  • 07:50 jynus: cloning es3 mysql data from es1008 to es1019
  • 04:19 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Sep 4 04:19:20 UTC 2015 (duration 19m 19s)
  • 02:26 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-04 02:26:04+00:00
  • 02:23 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 05m 21s)
  • 01:56 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: T111439 (duration: 00m 12s)
  • 00:11 logmsgbot: krinkle@tin Synchronized php-1.26wmf21/includes/resourceloader/ResourceLoader.php: I24f68e34a9fa4918 (duration: 00m 12s)
  • 00:06 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/235940/ (duration: 00m 11s)

2015-09-03

  • 23:53 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/235853/ (duration: 00m 12s)
  • 23:51 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/235843/ (duration: 00m 12s)
  • 23:50 logmsgbot: krenair@tin Synchronized multiversion/MWMultiVersion.php: https://gerrit.wikimedia.org/r/#/c/235843/ (duration: 00m 12s)
  • 23:41 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/235850/ (duration: 00m 12s)
  • 23:40 logmsgbot: krenair@tin Synchronized w/static/images/project-logos/ukwikivoyage.png: https://gerrit.wikimedia.org/r/#/c/235850/ (duration: 00m 12s)
  • 23:37 mutante: mw1224 - killed and restarted defunct hhvm, version is different from the one on mw1225
  • 23:37 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/235728 (duration: 00m 13s)
  • 23:36 logmsgbot: krenair@tin Synchronized w/static/images/project-logos/knwikisource.png: https://gerrit.wikimedia.org/r/#/c/235728/ (duration: 00m 12s)
  • 23:32 Krenair: mw1224 has been sending segfault warnings and "Lost parent, LightProcess exiting" to hhvm.log since about 21:17:34
  • 23:29 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/CirrusSearch: https://gerrit.wikimedia.org/r/#/c/235905/ (duration: 00m 13s)
  • 23:28 logmsgbot: krenair@tin Synchronized php-1.26wmf21/package.json: bd2eb6cc1919c7dab056d5f8fe5b4a164236d78f (duration: 00m 13s)
  • 23:02 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/235908/ (duration: 00m 13s)
  • 21:21 ori: rebuilt HHVM with updated diff from facebook/hhvm PR #6071 (T109540), uploaded to apt as 3.6.5+dfsg1-1+wm5
  • 21:18 urandom: bouncing Cassandra on restbase1001 to apply temporary GC settings
  • 19:54 bearND: MobileApps deployed sha1 553c399
  • 19:31 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.26wmf21
  • 18:13 ottomata: rolling restart of hadoop yarn nodemanagers to pick up Yarn AppMaster port range limitation to apply ferm rules.
  • 18:04 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Add plumbing code for Flow beta feature (unused for now) (duration: 00m 12s)
  • 18:03 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Add plumbing code for Flow beta feature (unused for now) (duration: 00m 12s)
  • 17:39 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/OpenStackManager/nova/OpenStackNovaController.php: https://gerrit.wikimedia.org/r/#/c/235769/ (duration: 00m 12s)
  • 17:34 mutante: bromine - deleting policy docroot
  • 17:06 jynus: cloning es1006 mysql data into es1015 [ETA:8h]
  • 16:30 bblack: updating nginx->1.9.4 on cp1071, cp3033 for prod validation before broader rollout
  • 16:30 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: es3 master switchover from es1009 to es1014 (eqiad) (duration: 00m 13s)
  • 16:28 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: es3 master switchover from es1009 to es1014 (codfw) (duration: 00m 13s)
  • 16:26 mutante: imported jenkins 1.609.3 into APT repo
  • 16:23 legoktm: fixed content model of Template:Languages@metawiki
  • 16:21 robh: re-enabling puppet on all mw systems
  • 16:14 robh: disabling puppet on all mw systems for apache config update
  • 16:01 jynus: performing es3 master switchover from es1009 to es1014
  • 15:40 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: depool es1006 (duration: 00m 12s)
  • 15:17 hashar: stopping nodepool on labnodepool1001.eqiad.wmnet not ready yet
  • 15:15 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: es2 master switchover from es1006 to es1011 (eqiad) (duration: 00m 13s)
  • 15:14 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: es2 master switchover from es1006 to es1011 (codfw) (duration: 00m 12s)
  • 15:05 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 13s)
  • 15:04 logmsgbot: demon@tin Synchronized php-1.26wmf21/extensions/Translate/: (no message) (duration: 00m 15s)
  • 14:51 jynus: performing es2 master switchover from es1006 to es1011
  • 14:33 paravoid: rebooting msw1-eqiad
  • 14:28 twentyafterfour: restarted phd (phabricator daemon) to pick up new configuration
  • 14:25 paravoid: changing IPv6 RA interval/lifetime/virtual-router-only @ eqiad
  • 14:21 paravoid: rebooting msw1-codfw
  • 13:17 paravoid: upgrading mr1-esams and mr1-eqiad to newer junos
  • 13:13 godog: bounce carbon daemons on graphite1001
  • 12:42 chasemp: unban elastic1001 and put back in service
  • 12:24 chasemp: move all shards off of elastic1001
  • 12:24 chasemp: disable elastic1001 in lvs as we are gonig to try fw apply round #2
  • 11:02 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1028; increase the load of es1010, es1013 and es1017 (duration: 00m 12s)
  • 10:45 jynus: applying schema change for ContentTranslation on x1-master "wikishared"
  • 10:02 godog: reenable puppet on ms-be1*
  • 09:16 jynus: started profiling mysql queries at phabricator. Only a 1% overhead is expected.
  • 09:12 moritzm: updated rsyncd firewall rules (see https://gerrit.wikimedia.org/r/235425 for details)
  • 09:12 godog: stop puppet on ms-be1* after ferm rsync change
  • 08:23 godog: fixup current graphite retention T96662
  • 07:26 moritzm: enabled ferm on dbstore* servers in codfw
  • 06:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Sep 3 06:29:35 UTC 2015 (duration 29m 34s)
  • 03:09 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-03 03:09:20+00:00
  • 03:06 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 05m 32s)
  • 02:45 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf20) at 2015-09-03 02:45:36+00:00
  • 02:39 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 10m 41s)
  • 01:32 logmsgbot: krenair@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 12s)
  • 00:36 logmsgbot: ori@tin Synchronized php-1.26wmf21/includes/parser/Preprocessor_Hash.php: Idd1acd903: Decline to cache preprocessor items larger than 1 Mb (duration: 00m 11s)
  • 00:36 logmsgbot: ori@tin Synchronized php-1.26wmf20/includes/parser/Preprocessor_Hash.php: Idd1acd903: Decline to cache preprocessor items larger than 1 Mb (duration: 00m 13s)
  • 00:27 RoanKattouw: Deployed patch for T111029

2015-09-02

  • 23:58 logmsgbot: andyrussg@tin Synchronized php-1.26wmf20/extensions/CentralNotice/: CentralNotice update (duration: 00m 13s)
  • 23:33 logmsgbot: andyrussg@tin Synchronized php-1.26wmf21/extensions/CentralNotice/: Update CentralNotice (duration: 00m 13s)
  • 23:02 logmsgbot: andyrussg@tin Finished scap: Update CentralNotice to 2.6.0 for wmf21 (duration: 48m 18s)
  • 22:13 logmsgbot: andyrussg@tin Started scap: Update CentralNotice to 2.6.0 for wmf21
  • 20:27 arlolra: updated Parsoid to version 5f2fae6c
  • 20:08 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.26wmf21
  • 20:02 logmsgbot: krinkle@tin Synchronized php-1.26wmf21/resources/src/startup.js: Ie65427caee (duration: 00m 12s)
  • 19:09 mutante: restarted gitblit, stopped counting
  • 19:07 paravoid: upgrading mr1-codfw, mr1-ulsfo to newer junos
  • 19:01 urandom: bouncing Cassandra on restbase1001 to address bogus icinga process failure alert
  • 18:52 legoktm: deployed patch for T110553
  • 18:36 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf21
  • 18:32 cmjohnson1: replacing disk 10 on db1028
  • 18:13 urandom: bouncing Cassandra on restbase1001 to apply temporary GC settings
  • 17:50 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/VisualEditor/modules/ve-mw/ui/inspectors: https://gerrit.wikimedia.org/r/#/c/235511/ (duration: 00m 12s)
  • 17:07 logmsgbot: ori@tin Synchronized php-1.26wmf21/extensions/UniversalLanguageSelector: 78a5908fd9: Updated mediawiki/core Project: mediawiki/extensions/UniversalLanguageSelector (duration: 00m 16s)
  • 17:07 logmsgbot: ori@tin Synchronized php-1.26wmf20/extensions/UniversalLanguageSelector: 2154acc529: Updated mediawiki/core Project: mediawiki/extensions/UniversalLanguageSelector (duration: 00m 13s)
  • 16:25 mutante: restarting NTP on lvs2004
  • 16:12 jynus: setting BBU auto-learn mode to warn only (disabled if not possible) on all database hosts
  • 16:03 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/MultimediaViewer/MultimediaViewer.php: https://gerrit.wikimedia.org/r/#/c/235484/ (duration: 00m 12s)
  • 16:01 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/UploadWizard/resources/mw.UploadWizardUploadInterface.js: https://gerrit.wikimedia.org/r/#/c/235486/ (duration: 00m 12s)
  • 15:58 logmsgbot: krenair@tin Synchronized php-1.26wmf20/extensions/MultimediaViewer/MultimediaViewer.php: https://gerrit.wikimedia.org/r/#/c/235483/ (duration: 00m 13s)
  • 15:56 logmsgbot: krenair@tin Synchronized php-1.26wmf20/extensions/UploadWizard/resources/mw.UploadWizardUploadInterface.js: https://gerrit.wikimedia.org/r/#/c/235485/ (duration: 00m 12s)
  • 15:51 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: T110837 (duration: 00m 13s)
  • 15:42 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/OpenStackManager/nova/OpenStackNovaController.php: https://gerrit.wikimedia.org/r/#/c/235482/ (duration: 00m 12s)
  • 15:34 logmsgbot: krenair@tin Synchronized php-1.26wmf20/extensions/OpenStackManager/nova/OpenStackNovaController.php: https://gerrit.wikimedia.org/r/#/c/235479/ (duration: 00m 13s)
  • 15:19 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/ContentTranslation/modules/tools/ext.cx.tools.template.js: https://gerrit.wikimedia.org/r/#/c/235442/ (duration: 00m 12s)
  • 15:14 logmsgbot: krenair@tin Synchronized php-1.26wmf20/extensions/ContentTranslation/modules/tools/ext.cx.tools.template.js: https://gerrit.wikimedia.org/r/#/c/235441/ (duration: 00m 12s)
  • 15:07 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234942/ and https://gerrit.wikimedia.org/r/#/c/234944/ (duration: 00m 13s)
  • 14:40 Nikerabbit: TTMServer reindex complete
  • 11:59 mark: removed tools LV snapshots on labstore1002
  • 11:47 mark: kill STOP'ed rsync on labstore1002
  • 11:00 jynus: cloning mysql data from es1002 into es1016 [ETA:16h]
  • 10:30 moritzm: installed qemu security updates on labvirt*
  • 09:41 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1002 (duration: 00m 12s)
  • 09:21 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1010, pool es1017 (duration: 00m 13s)
  • 09:19 hashar: Merged in "delete 1.26wmf12" https://gerrit.wikimedia.org/r/235347 which was left unmerged in Gerrit but was present on tin /srv/mediawiki-staging confusing people.
  • 08:03 bblack: restarting ntp on lvs2004
  • 08:01 moritzm: enable ferm on db1069/sanitarium
  • 07:50 moritzm: enable ferm on remaining phabricator db hosts
  • 04:54 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Sep 2 04:54:37 UTC 2015 (duration 54m 36s)
  • 02:52 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-02 02:52:51+00:00
  • 02:50 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 05m 09s)
  • 02:29 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf20) at 2015-09-02 02:29:56+00:00
  • 02:26 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 06m 31s)
  • 00:33 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/235366/ (duration: 00m 13s)

2015-09-01

  • 23:59 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/221731/ (duration: 00m 13s)
  • 23:41 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/235285/ (duration: 00m 14s)
  • 23:08 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/235362/ (duration: 00m 14s)
  • 23:02 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/235361/ (duration: 00m 13s)
  • 22:50 awight: update CRM from 0fc8474338e7a31fdde79287bd667b98cd96a252 to abc34b87ee9d1dbb1176f1929a3d748e1ee5ac7b
  • 22:18 MaxSem: Maps: creating and populating admin table
  • 21:20 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/235177/ (duration: 00m 12s)
  • 20:54 ori: restarted nutcracker on mw1142
  • 20:33 logmsgbot: twentyafterfour@tin Finished scap: sync 1.26wmf21 (duration: 30m 37s)
  • 20:03 dcausse: unfreezing elasticsearch indices
  • 20:03 logmsgbot: twentyafterfour@tin Started scap: sync 1.26wmf21
  • 19:52 YuviPanda: removed tools20150901132642 from labstore vg on labstore1002
  • 19:36 logmsgbot: ori@tin Synchronized php-1.26wmf20/includes/skins/SkinTemplate.php: cc643a0934: Deprecate unconditional loading of mediawiki.ui.button on all pages (duration: 00m 13s)
  • 17:31 urandom: bouncing Cassandra on restbase1001 to apply temporary GC setting
  • 17:28 dcausse: freezing elasticsearch indices before applying ferm fules on master
  • 17:23 logmsgbot: aude@tin Synchronized php-1.26wmf20/extensions/Wikidata: Fix for change dispatcher (duration: 00m 20s)
  • 16:45 jynus: performing schema change on testwiki and metawiki
  • 16:12 robh: policy.wikimedia.org dns change happening now
  • 16:00 chasemp: ferm for elastic1003/2/1(master)
  • 15:57 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/235168/ (duration: 00m 13s)
  • 15:51 YuviPanda: stopped replicate-tools on labstore1002, and cleaned out lockdir
  • 15:47 logmsgbot: reedy@tin Synchronized php-1.26wmf20/extensions/SecurePoll/: Stop cronspam (duration: 00m 13s)
  • 15:47 mark: labstore1002: echo 10000 > /sys/block/md123/md/sync_speed_min
  • 15:44 mark: labstore1002: update-initramfs -k all -u
  • 15:38 mark: labstore1002: mdadm /dev/md/slice51 --add /dev/sd{bh,bg,bf,be,bd,bc}
  • 15:36 moritzm: disabled ferm in analytic1028, needs some more work on possibly dynamic mapreduce ports
  • 15:16 mark: labstore1002: mdadm /dev/md/slice15 --re-add /dev/sd{bb,ba,az}
  • 15:14 mark: labstore1002: mdadm /dev/md/slice15 --re-add /dev/sdaw
  • 15:07 mark: labstore1002: mdadm --zero-superblock /dev/sd{aw,bh,bg,bf,be,bd,bc,bb,ba,az}1
  • 15:04 moritzm: enabled ferm in analytic1028 (initial hadoop worker)
  • 15:04 mark: labstore1002: mdadm --zero-superblock /dev/sdax1 && mdadm /dev/md/slice15 --re-add /dev/sdax
  • 15:03 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/231465/ - VE for all new enwiki accounts (duration: 00m 13s)
  • 14:58 mark: labstore1002: mdadm /dev/md/slice15 --re-add /dev/sday
  • 14:58 mark: labstore1002: mdadm --zero-superblock /dev/sday1
  • 14:53 mark: labstore1002: mdadm --stop /dev/md3
  • 14:37 ebernhardson: reset elasticsearch cluster.routing.allocation.disk.high back to 90%
  • 13:38 logmsgbot: krinkle@tin Synchronized w/: Remove rl-test.php (duration: 00m 13s)
  • 13:17 moritzm: enabled ferm on db1048
  • 13:09 moritzm: enabled ferm on labsdb100[467]
  • 12:01 YuviPanda: disable puppet on labsdb1006
  • 08:58 moritzm: enabled ferm on labsdb1001
  • 08:58 godog: fixup current graphite retention for metrics under "servers" hierarchy T96662
  • 08:51 moritzm: enabled ferm on labsdb1002
  • 08:31 moritzm: enabled ferm on labsdb1003
  • 08:29 godog: repool mw1125 mw1142 after nutcracker failures
  • 07:45 jynus: cloning mysql data from es1010 to es1017 [ETA: 6h]
  • 07:23 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1010 (duration: 00m 12s)
  • 07:13 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1007, pool es1013 (duration: 00m 13s)
  • 06:36 mutante: uploaded survey2012 to dumps/dataset1001; ownership as it is for survey2011; - T110746 in time for midnight PST
  • 05:18 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Sep 1 05:18:09 UTC 2015 (duration 18m 8s)
  • 02:28 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf20) at 2015-09-01 02:28:30+00:00
  • 02:25 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 06m 00s)

2015-08-31

  • 23:56 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/233665/ (duration: 00m 11s)
  • 23:49 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: reenable config changes for cirrus experimental completion api (duration: 00m 12s)
  • 23:40 logmsgbot: ori@tin Synchronized php-1.26wmf20/extensions/EducationProgram: 97ab82eab2: Updated mediawiki/core Project: mediawiki/extensions/EducationProgram 85a7d3932c1a4ad28f1a8dd05704f4e524152349 (duration: 00m 14s)
  • 23:27 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf20/extensions/CirrusSearch/: (no message) (duration: 00m 12s)
  • 23:25 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: revert update for cirrussearch experimental suggestions api (duration: 00m 12s)
  • 23:21 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: update config of cirrussearch experimental suggestions api (duration: 00m 12s)
  • 22:45 chasemp: disabled puppet on elastic hosts temporarily to safely roll out fw change. elastic seems to have not taken it well and I'm holding for green cluster state.
  • 21:20 mutante: installing package upgrades on argon
  • 20:58 ori: imported pybal_1.08_amd64.changes to jessie-wikimedia
  • 20:44 chasemp: ferm for elastic100[4-7] and adjust ferm to include wikitech source
  • 20:21 subbu: deployed parsoid version c3e4df5e
  • 16:22 godog: depool mw1125 + mw1142 from api, nutcracker client connections exceeded
  • 16:06 logmsgbot: thcipriani@tin Finished scap: SWAT: Ask the user to log in if the session is lost gerrit:234228 (duration: 27m 07s)
  • 15:59 jynus: restarting hhvm on mw2187
  • 15:39 logmsgbot: thcipriani@tin Started scap: SWAT: Ask the user to log in if the session is lost gerrit:234228
  • 15:33 mutante: terbium - Could not find dependent Service[nscd] for File[/etc/ldap/ldap.conf]
  • 15:28 logmsgbot: thcipriani@tin Synchronized closed-labs.dblist: SWAT: Creating closed-labs.dblist and closing es.wikipedia.beta.wmflabs.org gerrit:234594 (duration: 00m 13s)
  • 15:25 logmsgbot: thcipriani@tin Synchronized wmf-config/CirrusSearch-common.php: SWAT: Remove files from Commons from search results on wikimediafoundation.org gerrit:234040 (duration: 00m 11s)
  • 15:25 ottomata: starting varnishkafka instances on frontend caches to produce eventlogging client side events to kafka
  • 15:21 logmsgbot: thcipriani@tin Synchronized php-1.26wmf20/extensions/Wikidata: SWAT: Update Wikidata - Fix formatting of client edit summaries gerrit:234991 (duration: 00m 21s)
  • 15:16 logmsgbot: thcipriani@tin Synchronized php-1.26wmf20/extensions/UploadWizard/resources/controller/uw.controller.Step.js: SWAT: Keep the uploads sorted in the order they were created in initially gerrit:234553 (duration: 00m 12s)
  • 14:43 ebernhardson: elasticsearch cluster.routing.allocation.disk.watermark.high set to 75% to force elastic1022 to reduce its disk usage
  • 14:41 urandom: bouncing Cassandra on restbase1001 to apply temporary GC setting
  • 14:06 akosiaris: rebooted krypton. was reporting 100% cpu steal time
  • 13:40 paravoid: running puppet on newly-installed mc2001
  • 13:40 paravoid: restarting hhvm on mw1065
  • 11:10 moritzm: restart salt-master on palladium
  • 10:45 paravoid: reenabling asw2-a5-eqiad:xe-0/0/36 (T107635)
  • 10:36 godog: repool ms-fe1004
  • 10:32 godog: repool ms-fe1003 and depool ms-fe1004 for firewall changes
  • 10:19 godog: update graphite retention policy on files with previous retention and older than 30d T96662
  • 10:18 godog: repool ms-fe1002 and depool ms-fe1003 for firewall changes
  • 10:05 godog: depool ms-fe1002 to apply firewall changes
  • 09:55 jynus: cloning es1007 mysql data into es1013 (ETA: 5h30m)
  • 09:51 godog: repool ms-fe1001
  • 09:35 godog: depool ms-fe1001 in preparation for ferm changes
  • 09:27 godog: update graphite retention policy on files with previous retention and older than 60d T96662
  • 09:25 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1007 for maintenance (duration: 00m 13s)
  • 08:33 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1028, return ES servers back from maintenance (duration: 00m 12s)
  • 04:34 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Aug 31 04:34:14 UTC 2015 (duration 34m 13s)
  • 04:05 bblack: disabled ipv6 autoconf on neon, flushed old dynamic addr
  • 02:32 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf20) at 2015-08-31 02:32:25+00:00
  • 02:29 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 06m 42s)

2015-08-30

  • 12:58 godog: lvchange -ay labstore/others on labstore1002
  • 12:52 godog: start-nfs on labstore1002
  • 12:31 godog: lvchange -ay labstore/tools on labstore1002
  • 12:30 godog: also disabled puppet on labstore1002 while investigating
  • 12:15 godog: trying to manually assemble missing raid on labstore1002 with mdadm --assemble /dev/md/slice51 --uuid 0747643d:b89b36ff:57156095:c33694fc --verbose
  • 11:19 YuviPanda: powered labstore1002 back up
  • 11:17 YuviPanda: shut down labstore1002, going to powercycle from mgmt
  • 10:34 YuviPanda: disabled backups on labstore1002 to prevent overwriting of good backups on 2001
  • 10:08 YuviPanda: rebooted labstore1002
  • 04:16 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Aug 30 04:16:17 UTC 2015 (duration 16m 16s)
  • 02:23 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf20) at 2015-08-30 02:23:07+00:00
  • 02:20 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 05m 36s)

2015-08-29

  • 15:26 jynus: killing idle mysql connections from phabricator and setting wait and interactive timeout to 60
  • 09:30 jynus: SCAP failed, cannot depool db1028
  • 09:28 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1028, return ES servers back from maintenance (duration: 00m 03s)
  • 09:28 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1028, return ES servers back from maintenance (duration: 00m 03s)
  • 09:05 jynus: about to depool db1028 due to disk issue
  • 04:17 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Aug 29 04:17:55 UTC 2015 (duration 17m 54s)
  • 02:24 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf20) at 2015-08-29 02:24:01+00:00
  • 02:21 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 05m 48s)

2015-08-28

  • 23:45 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234679/ (duration: 06m 56s)
  • 22:51 logmsgbot: bd808@tin Synchronized wmf-config/CommonSettings-labs.php: Use ffmpeg instead of avconv on labs beta (I250fe33) (duration: 06m 05s)
  • 22:05 ori: disabling puppet on tin for a few minutes to test an ssh-agent-proxy change
  • 20:04 logmsgbot: catrope@tin Synchronized php-1.26wmf20/resources/src/mediawiki.legacy/shared.css: T110716 (duration: 00m 12s)
  • 18:09 robh: updating ldap-codfw cert
  • 17:10 logmsgbot: catrope@tin Synchronized php-1.26wmf20/extensions/Flow/includes/Parsoid/Utils.php: T110676 (duration: 00m 13s)
  • 17:08 urandom: bouncing Cassandra on restbase1001 to apply default (puppet-managed) settings
  • 16:03 chasemp: ferm for elasticsearch10(0[8-9|1[0-13])
  • 15:31 awight: updated crm from fc0fcc8f5af262b56392d3f4f5998f8ea08c99a8 to 0fc8474338e7a31fdde79287bd667b98cd96a252
  • 15:23 chasemp: ferm for elasticsearch10[14-17]
  • 11:09 logmsgbot: aude@tin Synchronized php-1.26wmf20/extensions/Wikidata/Wikidata.php: Sync entry point - updated to work on Jenkins together with ContentTranslation (duration: 00m 12s)
  • 10:29 godog: reenable puppet on ms-fe1, ferm changes will go out on monday
  • 09:48 jynus: Cloning es1001 database into es1012
  • 09:45 moritzm: enabled ferm for swift on esams
  • 09:28 moritzm: enabled ferm on strontium puppetmaster backend
  • 09:00 moritzm: enabled ferm on rhodium puppetmaster backend
  • 08:29 moritzm: uploaded debdeploy 0.0.3 to carbon
  • 08:23 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1001, increas weight of es1011, pool es1014 for the first time (duration: 00m 13s)
  • 05:59 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Aug 28 05:59:09 UTC 2015 (duration 59m 8s)
  • 04:58 logmsgbot: ori@tin Synchronized php-1.26wmf20/includes/parser/Parser.php: 754b222daf: Add ParserOutput cache and expiry times to NewPP report (duration: 00m 13s)
  • 02:41 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf20) at 2015-08-28 02:41:26+00:00
  • 02:35 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 10m 47s)
  • 01:59 Tim: on ruthenium: started parsoid_vd which was previously killed by oom-killer
  • 01:58 Tim: on ruthenium, reduced parsoid-rt-client concurrency from 16 to 8 since it was OOM and oom-killer was killing random things
  • 01:37 Tim: on ruthenium restarted parsoid-rt-client and parsoid-vd-client
  • 00:24 mutante: powercycled mw2027
  • 00:19 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/234450/ (duration: 01m 14s)
  • 00:06 logmsgbot: krenair@tin Synchronized wmf-config/mobile.php: live hack to make previous commit work (duration: 01m 14s)
  • 00:05 Krenair: Another codfw host broke: mw2027
  • 00:01 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234330/ (duration: 00m 13s)

2015-08-27

  • 23:58 logmsgbot: krenair@tin Synchronized php-1.26wmf20/extensions/MobileFrontend/includes/MobileFormatter.php: https://gerrit.wikimedia.org/r/#/c/234331/1 (duration: 00m 12s)
  • 23:57 logmsgbot: krenair@tin Synchronized php-1.26wmf20/extensions/MobileFrontend/includes/config/Experimental.php: https://gerrit.wikimedia.org/r/#/c/234331/1 (duration: 00m 14s)
  • 23:55 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/233439/ (duration: 00m 12s)
  • 23:30 logmsgbot: krenair@tin Synchronized php-1.26wmf20/extensions/Gadgets/extension.json: touch (duration: 00m 13s)
  • 23:24 logmsgbot: krenair@tin Synchronized php-1.26wmf20/includes/DefaultSettings.php: https://gerrit.wikimedia.org/r/#/c/234328/ (duration: 00m 12s)
  • 23:24 logmsgbot: krenair@tin Synchronized php-1.26wmf20/includes/registration/ExtensionProcessor.php: https://gerrit.wikimedia.org/r/#/c/234328/ (duration: 00m 12s)
  • 23:23 logmsgbot: krenair@tin Synchronized php-1.26wmf20/includes/MWNamespace.php: https://gerrit.wikimedia.org/r/#/c/234328/ (duration: 00m 13s)
  • 23:15 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/234009/ (duration: 00m 13s)
  • 23:04 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/233100/ (duration: 00m 12s)
  • 20:11 chasemp: ferm setup on elasticsearch10(1[8-9|2[0-3])
  • 20:06 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.26wmf20
  • 19:57 logmsgbot: twentyafterfour@tin Synchronized php-1.26wmf20/includes/media/XMP.php: deploy fix for T89532 on 1.26wmf20 (duration: 00m 13s)
  • 18:16 chasemp: setting up ferm on elastic1027-31
  • 17:47 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/234320/ (duration: 00m 13s)
  • 17:43 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234320/2 (duration: 00m 13s)
  • 17:37 urandom: ack'd Cassandra process alert on restbase1001; temporary command args have pushed the class name beyond the limit
  • 17:34 logmsgbot: krenair@tin Synchronized multiversion/MWMultiVersion.php: (no message) (duration: 00m 12s)
  • 17:24 logmsgbot: krenair@tin Synchronized multiversion/MWMultiVersion.php: https://gerrit.wikimedia.org/r/#/c/234320/ (duration: 00m 12s)
  • 17:08 urandom: bouncing Cassandra on restbase1001 to apply temporary GC settings
  • 16:51 moritzm: ferm rules on logstash100[1-3] have been amended to allow grafana from reading dashboard configs
  • 16:39 bd808: new ferm rules on logstash100[1-3] are blocking grafana from reading dashboard configs.
  • 16:22 moritzm: ferm enabled on logstash1003
  • 16:18 moritzm: ferm enabled on logstash1002
  • 16:16 bd808: ferm enabled on logstash1001
  • 16:06 bd808: logstash1001 back up after system reboot; we applied a default drop rule without applying the other iptables changes; will try again
  • 15:58 chasemp: rebooting logstash1001.mgmt.eqiad.wmnet for moritz as it is having issues
  • 15:47 bblack: killed hung ubuntu mirror rsync commands on carbon, from Jul 10
  • 15:45 bd808: logstash1001 not responding over ssh following ferm rules application; moritzm investigating
  • 15:30 bd808: Disabled puppet on logstash100[1-3] prior to trying to enable ferm
  • 15:11 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable newarticle campaign in itwiki gerrit:234223 (duration: 01m 52s)
  • 14:52 bblack: re-imaging lvs200[123]
  • 14:47 godog: reenable puppet on ms-be1*
  • 14:22 godog: disable puppet on ms-fe1 / ms-be1 in prepration for puppet work
  • 14:15 godog: reenable puppet on ms-fe2*
  • 13:47 bblack: re-imaging lvs2004 + lvs2005
  • 13:29 ottomata: doing rolling restart of kafka brokers to apply auto_create_topics change
  • 13:21 godog: enable puppet on ms-be2*
  • 13:21 ottomata: stopping kafka on analytics1021, it is no longer a kafka broker.
  • 13:09 godog: disable puppet on ms-be2* in preparation for firewall changes
  • 13:09 jynus: cloning es1008 into es1014
  • 13:04 ottomata: running leader election now that all topics and partitions are rebalanced across new kafka nodes
  • 12:46 bblack: re-imaging lvs2006
  • 12:45 andrewbogott: re-imaging labnet1001 (I hope)
  • 11:33 _joe_: restarted hhvm on mw1143, locked in __lll_lock_wait for stat_cache deadlock
  • 11:10 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Pool es1011 for the first time, depool es1008 (duration: 00m 12s)
  • 09:27 jynus: installing and configuring servers es1012-es1019
  • 06:39 ostriches: tin: dropped useless "gerrit" remote from /srv/mediawiki-staging (uses ssh, lol), pointed {origin,readonly} at the actual repo instead of a redirect.
  • 06:00 _joe_: powercycling mw2140, not responding to ping, blank console
  • 03:17 awight: deploy config cleanup for paymentswiki
  • 02:38 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf19/cache/l10n: l10nupdate for 1.26wmf19 (duration: 10m 44s)
  • 02:16 awight: push config change to the payments orphan slayer: explitly give stomp port to work around strict notice, clean up unused globals. T109911
  • 01:32 ejegg: updated payments from 8ba4b5299f195cf48e6809b18a21e2d53f6eec1b to 6ac552f280fb839069d117386c4ecbe9e52f90a8
  • 00:31 twentyafterfour: finished phabricator upgrade, everything appears to be working
  • 00:24 logmsgbot: aaron@tin Synchronized php-1.26wmf19/extensions/CentralAuth: 47e181adb2898977b146de7398eaa35aebb870e3 (duration: 01m 13s)
  • 00:22 logmsgbot: aaron@tin Synchronized php-1.26wmf20/extensions/CentralAuth: 47e181adb2898977b146de7398eaa35aebb870e3 (duration: 01m 13s)
  • 00:20 twentyafterfour: taking phabricator offline for scheduled upgrade

2015-08-26

  • 23:59 Krinkle: mwscript deleteEqualMessages.php --wiki rowiki
  • 23:57 yurik: git deployed tilerator - had the 4/5 issue - https://phabricator.wikimedia.org/T110434
  • 23:46 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234072/ (duration: 01m 12s)
  • 23:37 logmsgbot: krenair@tin Synchronized php-1.26wmf20/maintenance/deleteEqualMessages.php: https://gerrit.wikimedia.org/r/#/c/234038/ (duration: 01m 12s)
  • 23:35 logmsgbot: krenair@tin Synchronized php-1.26wmf19/maintenance/deleteEqualMessages.php: https://gerrit.wikimedia.org/r/#/c/234037/1 (duration: 01m 12s)
  • 23:27 yurik: deployed kartotherian
  • 23:21 jynus: cloning es1005 into es1011, ETA 9 hours
  • 22:41 ori: armed keyholder on tin
  • 22:40 ori: Disabled Puppet on mw1017 for 2hrs and applied I059b0c96c9 for testing.
  • 21:55 logmsgbot: krinkle@tin Synchronized php-1.26wmf19/includes/poolcounter/PoolWorkArticleView.php: (no message) (duration: 01m 12s)
  • 21:48 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1005 (duration: 01m 12s)
  • 21:40 logmsgbot: krinkle@tin Synchronized php-1.26wmf20/includes/poolcounter/PoolWorkArticleView.php: (no message) (duration: 01m 12s)
  • 21:32 ori: Disabling Puppet on tin again to test an ssh-agent-proxy change
  • 20:30 logmsgbot: ori@tin Synchronized README: testing ssh-agent-proxy changes (duration: 00m 13s)
  • 20:25 ori: Disabling puppet on tin and hacking some debug logging into ssh-agent-proxy
  • 20:24 ori: armed ssh-agent key on mira
  • 20:21 logmsgbot: krinkle@tin Synchronized php-1.26wmf20/includes/poolcounter/PoolWorkArticleView.php: (no message) (duration: 00m 03s)
  • 20:11 subbu: deployed parsoid version 44d657de
  • 19:52 logmsgbot: krenair@tin Synchronized php-1.26wmf20/extensions/Echo/includes/mapper/EventMapper.php: https://gerrit.wikimedia.org/r/#/c/234082/ (duration: 00m 12s)
  • 19:47 mutante: sodium - deleting shunted messages older than 7 days
  • 19:23 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234042/ (duration: 00m 12s)
  • 19:22 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/234024/ (duration: 00m 12s)
  • 19:20 logmsgbot: krenair@tin Synchronized multiversion/MWWikiversions.php: https://gerrit.wikimedia.org/r/#/c/232672/ (duration: 00m 12s)
  • 18:50 logmsgbot: krinkle@tin Synchronized php-1.26wmf20/maintenance/deleteEqualMessages.php: (no message) (duration: 00m 11s)
  • 18:50 logmsgbot: krinkle@tin Synchronized php-1.26wmf19/maintenance/deleteEqualMessages.php: (no message) (duration: 00m 13s)
  • 18:38 twentyafterfour: ^ stupid typo. That sync was group1 to 1.26wmf20
  • 18:37 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: tig
  • 18:31 logmsgbot: ori@tin Synchronized w/404.php: Ided1facc0: Remove auto-redirection from 404 page. (duration: 00m 13s)
  • 17:51 ejegg: updated SmashPig from 258f2c917b1ae50b01231927bcd6f58ecaa8940b to fdb053efa617162ac9f695e493c390987a069140
  • 17:30 urandom: bouncing Cassandra on restbase1001 to apply temporary GC setting
  • 17:12 andrewbogott: ok, /now/ I’m running a dist-upgrade on labcontrol1001, to sort out weird oslo dependencies
  • 17:09 chasemp: adding firewall to elasticsearch2[4-6] (3 was just done as a pilot)
  • 17:03 andrewbogott: upgraded labnet1002 nova services to Juno
  • 16:34 andrewbogott: stopping keystone, updating db, restarting
  • 16:18 andrewbogott: switching labcontrol1001 hiera to Juno which will add the cloud-archive repo for Juno.
  • 16:11 andrewbogott: backing up labs openstack databases into /home/andrew/openstackdbbackups on db1009
  • 16:11 andrewbogott: starting labs openstack update to Juno
  • 15:53 moritzm: ferm enabled on elastic1023
  • 15:45 godog: repool restbase1009 in pybal
  • 15:28 logmsgbot: thcipriani@tin Synchronized php-1.26wmf20/extensions/Wikidata: SWAT: Update Wikidata - wrap usage tracking batch updates in transaction gerrit:233970 (duration: 00m 23s)
  • 13:47 andrewbogott: rebooting/reimaging labnet1001
  • 13:11 mobrovac: restbase deploying 1dfba85
  • 12:54 yurik: git synced kartotherian
  • 11:02 jynus: dropping optin_survey_old table on all wikis
  • 10:33 godog: reenable puppet on ms-fe/ms-be, base::firewall still not enabled
  • 09:58 godog: test-reboot ms-be2001
  • 08:17 godog: disable puppet on ms-be/ms-fe in preparation for merging firewall changes
  • 07:53 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Aug 26 07:53:31 UTC 2015 (duration 53m 30s)
  • 07:01 jynus: restarting mw1239 HHVM, which is unresponsive
  • 04:47 logmsgbot: ori@tin Synchronized wmf-config: I73721936: Enable ParsoidBatchAPI everywhere (duration: 00m 13s)
  • 03:11 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf20) at 2015-08-26 03:11:29+00:00
  • 03:06 logmsgbot: awight@tin Synchronized wmf-config/InitialiseSettings-labs.php: Push labs config to keep in sync with master (duration: 00m 13s)
  • 03:05 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 10m 45s)
  • 02:37 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf19) at 2015-08-26 02:37:51+00:00
  • 02:34 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf19/cache/l10n: l10nupdate for 1.26wmf19 (duration: 06m 29s)
  • 02:00 ottomata: kafka topic webrequest_upload has finished rebalancing across new brokers. starting move of last topic webrequest_text
  • 01:50 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf19/extensions/Flow/: Sync Flow for reply fix (duration: 00m 15s)
  • 00:28 logmsgbot: ori@tin Synchronized php-1.26wmf20/extensions/Scribunto/engines/LuaCommon/LuaCommon.php: (no message) (duration: 00m 13s)
  • 00:26 logmsgbot: ori@tin Synchronized php-1.26wmf19/extensions/Scribunto/engines/LuaCommon/LuaCommon.php: (no message) (duration: 00m 13s)
  • 00:26 Danny_B: 2586dd1c7c obviously broke many pages
  • 00:19 logmsgbot: ori@tin Synchronized php-1.26wmf19/extensions/Scribunto/engines/LuaCommon/LuaCommon.php: (no message) (duration: 00m 14s)
  • 00:14 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I79ffa78fa: Collection/OCG: Turn on plain text output format in Book Creator (duration: 00m 12s)
  • 00:12 logmsgbot: ori@tin Synchronized php-1.26wmf20/extensions/Scribunto/engines/LuaCommon/LuaCommon.php: 2586dd1c7c: Updated mediawiki/core Project: mediawiki/extensions/Scribunto (duration: 00m 13s)

2015-08-25

  • 23:39 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/233860/ (duration: 00m 12s)
  • 23:16 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/233872/ (duration: 00m 13s)
  • 23:13 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/232963/ (duration: 00m 12s)
  • 23:12 logmsgbot: krenair@tin Synchronized wmf-config/extension-list: https://gerrit.wikimedia.org/r/#/c/232963/ (duration: 00m 12s)
  • 23:10 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/232962/ (duration: 00m 12s)
  • 23:10 logmsgbot: krenair@tin Synchronized wmf-config/extension-list: https://gerrit.wikimedia.org/r/#/c/232962/ (duration: 00m 12s)
  • 23:05 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/233781/ (duration: 00m 12s)
  • 22:20 cscott: updated Parsoid to version c3b037b0
  • 22:10 ejegg: disabled paypal audit downloader and parser due to them warning of incorrect data
  • 21:16 logmsgbot: ori@tin Synchronized php-1.26wmf19/extensions/AbuseFilter: I15f5b5b6 & I9c23b607 (duration: 00m 13s)
  • 21:13 logmsgbot: ori@tin Synchronized php-1.26wmf19/extensions/Cite/modules/ext.cite.styles.css: 7344e02216: Updated mediawiki/core Project: mediawiki/extensions/Cite (duration: 00m 12s)
  • 21:09 logmsgbot: ori@tin Synchronized php-1.26wmf20/extensions/AbuseFilter: I15f5b5b6 & I9c23b607 (duration: 00m 14s)
  • 20:54 tgr: finished OAuth migration
  • 20:34 logmsgbot: tgr@tin Synchronized wmf-config/CommonSettings.php: make OAuth DB writable again T108648 (duration: 00m 12s)
  • 20:32 logmsgbot: tgr@tin Synchronized wmf-config/CommonSettings.php: change wgMWOAuthCentralWiki mediawikiwiki -> metawiki T108648 (duration: 00m 12s)
  • 20:24 logmsgbot: tgr@tin Synchronized wmf-config/CommonSettings.php: set OAuth to readonly for DB migration T108648 (duration: 00m 13s)
  • 20:13 subbu: deployed parsoid version 759916fc
  • 19:24 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.26wmf20
  • 19:21 logmsgbot: twentyafterfour@tin Finished scap: testwiki to 1.26wmf20 (duration: 50m 12s)
  • 18:31 logmsgbot: twentyafterfour@tin Started scap: testwiki to 1.26wmf20
  • 17:11 YuviPanda: run authdns-update on radon (ns0.wikimedia.org)
  • 17:10 urandom: bouncing Cassandra on restbase1001 to apply temporary GC settings
  • 16:58 Krinkle: mwscript deleteEqualMessages.php --wiki kawiki
  • 16:56 andrewbogott: restarting pdns on labcontrol1001 and labcontrol2001 to handle a nembus reboot
  • 16:53 Krinkle: mwscript deleteEqualMessages.php --wiki huwiki
  • 16:31 Krinkle: mwscript deleteEqualMessages.php --wiki frwiki
  • 16:17 Krinkle: mwscript deleteEqualMessages.php --wiki frpwiki
  • 15:50 godog: powercycle ms-be1004, likely xfs
  • 15:44 andrewbogott: dist-upgrade and rebooting nembus in an attempt to resolve this acpi_pad issue
  • 15:36 Krinkle: mwscript deleteEqualMessages.php --wiki euwiki (T45917)
  • 15:29 Krinkle: mwscript deleteEqualMessages.php --wiki eowiki (T45917)
  • 15:07 logmsgbot: krenair@tin Synchronized php-1.26wmf19/extensions/Flow: https://gerrit.wikimedia.org/r/#/c/233718/ (duration: 00m 16s)
  • 13:56 jynus: dropping old tables on s7 - T5493
  • 13:48 jynus: dropping old tables on s6 - T54932
  • 12:53 Jeff_Green: authdns-update to change bismuth's IP
  • 11:16 jynus: dropping old tables on s3 - T54932
  • 10:46 jynus: dropping old tables on s2 - T54932
  • 10:05 YuviPanda: restart puppetmaster on labcontrol1001 for https://gerrit.wikimedia.org/r/#/c/233184/
  • 07:35 _joe_: stopping redis, wiping aof, restarting redis on rdb100{1,2} - snapshot saved on rdb1002:/root
  • 07:12 _joe_: stopping redis on rdb1003,4, wiping AOF, restarting
  • 06:38 jynus: performing schema change on officewiki, mediawikiwiki and metawiki
  • 02:21 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf19/cache/l10n: l10nupdate for 1.26wmf19 (duration: 06m 26s)
  • 01:48 ottomata: starting move of kafka partitions for topic webrequest_upload to new brokers. this will take a while!
  • 01:44 ottomata: restarting kafka on new brokers kafka1013,1014,1020 to apply increase in num.replica.fetchers

2015-08-24

  • 23:46 logmsgbot: mattflaschen@tin Synchronized wmf-config: Remove wgFlowOccupyPages (duration: 00m 12s)
  • 23:38 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/233636/ (duration: 00m 12s)
  • 22:16 logmsgbot: tgr@tin Synchronized wmf-config/CommonSettings-labs.php: change OAuth DB on beta +enable writes (duration: 00m 12s)
  • 21:55 logmsgbot: tgr@tin Synchronized wmf-config/CommonSettings-labs.php: set beta OAuth to readonly (duration: 00m 13s)
  • 21:54 logmsgbot: tgr@tin Synchronized wmf-config/CommonSettings-labs.php: set beta OAuth to readonly (duration: 00m 13s)
  • 21:42 akosiaris: enabled puppet on maps-test200{1,2,3,4}.codfw.wmnet
  • 20:21 arlolra: updated Parsoid to version 0b2fbae7
  • 18:58 bblack: reloading primary LVS pybals for BlankPage change ( https://gerrit.wikimedia.org/r/#/c/233053/ ) + ulimit fixup ( https://gerrit.wikimedia.org/r/#/c/233484/ )
  • 18:31 bblack: reloading backup LVS pybals for BlankPage change ( https://gerrit.wikimedia.org/r/#/c/233053/ )
  • 17:19 urandom: bouncing Cassandra on restbase1001 to apply temporary GC settings
  • 16:23 logmsgbot: bd808@tin Purged l10n cache for 1.26wmf18
  • 16:23 logmsgbot: bd808@tin Purged l10n cache for 1.26wmf17
  • 16:05 andrewbogott: rebooting labnet1001
  • 15:53 _joe_: restarted nutcracker on mw1010, holding a 150 GB deleted logfile
  • 15:47 Krenair: running sync-common on mw1010 to bring it up to date after clearing some space
  • 15:44 logmsgbot: krenair@tin Purged l10n cache for 1.26wmf16
  • 15:41 logmsgbot: krenair@tin Purged l10n cache for 1.26wmf15
  • 15:38 logmsgbot: krenair@tin Synchronized php-1.26wmf19/extensions/Wikidata: https://gerrit.wikimedia.org/r/#/c/233411/1 (duration: 00m 49s)
  • 15:37 hashar: stopped and restarted Zuul
  • 15:31 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/232919/ and https://gerrit.wikimedia.org/r/#/c/232915/ (duration: 01m 34s)
  • 15:29 logmsgbot: krenair@tin Synchronized w/static/images/project-logos/knwikiquote.png: https://gerrit.wikimedia.org/r/#/c/232919/ (duration: 02m 04s)
  • 15:19 Krenair: No space left on mw1010, cannot ping or ssh to mw2180
  • 15:16 logmsgbot: krenair@tin Synchronized docroot/noc/db.php: https://gerrit.wikimedia.org/r/#/c/232920/ (duration: 01m 34s)
  • 15:14 hashar: apt-get upgrade on gallium
  • 14:48 andrewbogott: forcing wikitech logouts in order to flush everyone’s service catalog
  • 14:18 ottomata: starting to move kafka topic-partitions to new brokers (and off of analytics1021)
  • 14:12 yurik: git deploy synced kartotherian
  • 13:55 akosiaris: disable puppet on fermium preparing for reinstallation
  • 13:55 akosiaris: disable puppet on fermium
  • 12:54 akosiaris: stop etcd on etcd1002.eqiad.wmnet. Already removed from the cluster
  • 11:58 _joe_: stopping etcd on etcd1001
  • 11:50 _joe_: restarting etcd on etcd1001
  • 09:00 YuviPanda: starting up replicate for tools on labstore1002
  • 09:00 YuviPanda: cleaning up lockdir on labstore for maps and tools
  • 09:00 YuviPanda: others replication on labstore1002 completed successfuly
  • 08:31 YuviPanda: cleaned up others lockdir for replication on labstore1002 and started it manually
  • 06:43 jynus: reloading dbproxy1003 service
  • 02:21 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf19/cache/l10n: l10nupdate for 1.26wmf19 (duration: 06m 36s)

2015-08-23

  • 16:54 urandom: bouncing Cassandra on restbase1001 to apply temporary GC settings
  • 02:20 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf19/cache/l10n: l10nupdate for 1.26wmf19 (duration: 06m 23s)

2015-08-22

  • 23:08 logmsgbot: krenair@tin Synchronized php-1.26wmf19/extensions/AbuseFilter/maintenance/addMissingLoggingEntries.php: (no message) (duration: 01m 05s)
  • 19:41 YuviPanda: manually remove old snapshots from labstore1002
  • 17:28 chasemp: tweaking apache on iridum T109941
  • 16:45 chasemp: scratch that as we have mpm_prefork enabled :)
  • 16:33 chasemp: raising values in mpm_worker.conf for iridium to to debug and hopefully head off further crashing
  • 14:44 twentyafterfour: restarted apache2 on iridium. Segfault again. This time I at least got one clue in the log: "zend_mm_heap corrupted"
  • 09:18 twentyafterfour: phabricator seems stable now, restarting apache2 on iridium did the trick, unfortunately we didn't learn why
  • 08:36 twentyafterfour: restarted phd on iridium
  • 08:36 twentyafterfour: restarted apache2 on iridium
  • 02:20 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf19/cache/l10n: l10nupdate for 1.26wmf19 (duration: 06m 09s)
  • 00:26 mutante: deleting blog.sh and blog_pageviews crontab from stat1003

2015-08-21

  • 23:34 urandom: restarting Cassandra on restbase1001 to restore baseline settings
  • 23:11 yurik: synced kartotherian
  • 22:35 mutante: deleting held messages on mailman that are older than 1 year
  • 21:56 awight: increasing paymentswiki orphan gc-cc-limbo expiry time to 30 days
  • 21:45 mutante: had to reset list creator password for mailman - ask me if you think you should have it and don't (this is not the master pass)
  • 20:37 logmsgbot: ori@tin Synchronized php-1.26wmf19/includes: I1eb8dfc: Revert Count API and hook calls, with 1:1000 sampling (duration: 01m 09s)
  • 19:43 awight: update paymentswiki from 2b08853c977eee0fd17bf00a673a3bbf2a146554 to 8ba4b5299f195cf48e6809b18a21e2d53f6eec1b
  • 18:58 awight: disabling Amazon gateway
  • 18:52 awight: updated paymentswiki from 049ad15323564fd5cd7f5efcadddb532a3590cef to 2b08853c977eee0fd17bf00a673a3bbf2a146554
  • 16:06 jynus: checksumming dewiki database, higher write rate/dbstore lag expected temporarily
  • 15:10 ottomata: rebooting kafka broker analytics1021 to hopefully reload /dev/sdg with new disk, also will turn on hyperthreading
  • 14:13 ottomata: rebooting analytics1056 after upgrading kernel to linux-image-3.13.0-61-generic
  • 13:58 urandom: restarting restbase1001 to apply temporary GC setting
  • 13:34 ottomata: stopping kafka broker on analytics1021 due to bad disk.
  • 13:30 bblack: wiped ganglia apache access log on uranium, to free up half of the (full) rootfs
  • 10:07 godog: enable puppet on ms-fe1/ms-be1
  • 09:49 godog: disable puppet on ms-fe1/ms-be1 before merging https://gerrit.wikimedia.org/r/#/c/231240/
  • 07:06 _joe_: restarting gitblit, because it will be decommissioned "soon"...
  • 02:34 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf19/cache/l10n: l10nupdate for 1.26wmf19 (duration: 11m 19s)

2015-08-20

  • 23:40 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf19/extensions/CirrusSearch/: Fix some cirrussearch logspam (duration: 00m 13s)
  • 23:30 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 12s)
  • 23:29 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 13s)
  • 23:23 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/232854/ (duration: 00m 13s)
  • 23:22 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/232671/ (duration: 00m 12s)
  • 23:15 logmsgbot: krenair@tin Synchronized php-1.26wmf19/extensions/LiquidThreads/classes/Hooks.php: https://gerrit.wikimedia.org/r/#/c/232783/ (duration: 00m 12s)
  • 23:13 logmsgbot: krinkle@tin Synchronized php-1.26wmf19/includes/resourceloader/ResourceLoaderFileModule.php: T102578 (duration: 00m 13s)
  • 23:08 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/232782/ (duration: 00m 12s)
  • 22:48 logmsgbot: ori@tin Synchronized php-1.26wmf19/includes/libs/CSSMin.php: Icc1c23a2: CSSMin: remove dot segments in relative local URLs (duration: 00m 12s)
  • 21:36 cscott: updated Parsoid to version db6e6404f67a9f971b4fbefe9de239735426c738
  • 21:25 matt_flaschen: Ran FlowUpdateRevContentModelFromOccupyPages.php on all wikis
  • 20:41 twentyafterfour: scap failed to sync to mw2180.codwf.wmnet
  • 20:41 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.26wmf19
  • 20:38 logmsgbot: twentyafterfour@tin Synchronized php-1.26wmf19: Silence the undefined index error in CirrusSearch (duration: 06m 24s)
  • 19:40 chasemp: moving enwiki_content_1432182861 elastic shard from 1022 to 1004 due to space (1022 is at 91%)
  • 20:57 mutante: no log bot
  • 18:56 mutante: labvirt1007 "only" 29G space left - but since we have 2.2T there that means 99% full
  • 17:39 ottomata: stopping kafka on analytics1018 and bringing it down for reinstall as kafka1018 with Jessie
  • 16:38 YuviPanda: puppet swat done
  • 15:44 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: etherpad-lite_1.5.7-1
  • 15:43 logmsgbot: krenair@tin Synchronized php-1.26wmf18/extensions/ContentTranslation/modules/tools/ext.cx.tools.reference.js: https://gerrit.wikimedia.org/r/#/c/232729/ (duration: 00m 12s)
  • 15:42 logmsgbot: krenair@tin Synchronized php-1.26wmf19/extensions/ContentTranslation/modules/tools/ext.cx.tools.reference.js: https://gerrit.wikimedia.org/r/#/c/232730/ (duration: 00m 13s)
  • 15:39 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/206480/ (duration: 00m 13s)
  • 15:38 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/206480/ (duration: 00m 13s)
  • 15:32 logmsgbot: krenair@tin Synchronized php-1.26wmf18/extensions/ContentTranslation/api/ApiContentTranslationPublish.php: https://gerrit.wikimedia.org/r/#/c/232687/ (duration: 00m 13s)
  • 15:31 logmsgbot: krenair@tin Synchronized php-1.26wmf19/extensions/ContentTranslation/api/ApiContentTranslationPublish.php: https://gerrit.wikimedia.org/r/#/c/232688/ (duration: 00m 11s)
  • 15:27 greg-g: on mw2187: rsync: failed to set times on "/srv/mediawiki/wmf-config": Read-only file system (30)
  • 15:25 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/231464/ (duration: 00m 13s)
  • 15:08 urandom: restarting restbase1001 to apply temporary heap size of 12G
  • 15:02 jynus: performing online schema change on wikidata
  • 15:00 andrewbogott: rebooting labvirt1008
  • 12:48 jynus: restarted nutcracker on mw1142
  • 12:08 godog: reenable puppet on ms-fe1/ms-be1
  • 12:04 godog: repool ms-fe1001
  • 11:53 godog: depool ms-fe1001 to test a reboot
  • 11:45 godog: disable puppet on ms-fe/be1 in preparation to apply https://gerrit.wikimedia.org/r/#/c/231237
  • 12:08 kart_: Updated cxserver to e221462
  • 03:00 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf19/cache/l10n: l10nupdate for 1.26wmf19 (duration: 06m 27s)
  • 02:45 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf18) at 2015-08-20 02:45:14+00:00
  • 02:38 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf18/cache/l10n: l10nupdate for 1.26wmf18 (duration: 10m 41s)

2015-08-19

  • 23:27 logmsgbot: rmoen@tin Synchronized wmf-config/InitialiseSettings.php: Remove reference to lost wikitech apple touch icon file (duration: 00m 13s)
  • 23:21 logmsgbot: rmoen@tin Synchronized php-1.26wmf19/extensions/TimedMediaHandler/: Re-disable 2-pass Theora encoding temporarily
  • 23:16 logmsgbot: rmoen@tin Synchronized php-1.26wmf18/extensions/Flow: Add debugging code to detect and workaround type hint failure (duration: 00m 14s)
  • 21:20 robh: livehack reverted sodium back to normal, testing done
  • 21:08 robh: disabled puppet on sodium for livehacking tests for T109609
  • 21:04 andrewbogott: disabling puppeton labnet1001 and labnet1002
  • 20:46 urandom: restarting Cassandra on restbase1001 to enable -XX:+PrintAdaptiveSizePolicy
  • 20:43 urandom: disabling puppet on restbase1001 to temporarily enable additional GC logging
  • 20:17 subbu: deployed parsoid version 8d617c99
  • 20:00 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/232580/ (duration: 00m 12s)
  • 19:18 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/232565/ (duration: 00m 12s)
  • 19:04 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/232543/ (duration: 00m 13s)
  • 18:59 logmsgbot: twentyafterfour@tin Synchronized php-1.26wmf19: deploy hotfix for Wikidata: https://gerrit.wikimedia.org/r/#/c/232556/ (duration: 02m 39s)
  • 18:30 ottomata: starting reinsatll of analytics1022 -> kafka1022 as jessie
  • 18:06 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf19
  • 16:26 ottomata: added analytics105[012456] into hadoop cluster as worker nodes
  • 16:02 logmsgbot: thcipriani@tin Synchronized php-1.26wmf18/extensions/ContentTranslation/api/ApiContentTranslationPublish.php: SWAT: Temp disable notifications for cx gerrit:232504 (duration: 00m 13s)
  • 15:57 logmsgbot: thcipriani@tin Synchronized php-1.26wmf19/extensions/ContentTranslation/api/ApiContentTranslationPublish.php: SWAT: Temporarily disable notifications for cx gerrit:232505 (duration: 00m 12s)
  • 15:51 logmsgbot: thcipriani@tin Synchronized php-1.26wmf18/includes/changetags/ChangeTags.php: SWAT: Avoid full RC table scans in ChangeTags::updateTags() gerrit:232484 (duration: 00m 12s)
  • 15:35 logmsgbot: thcipriani@tin Synchronized php-1.26wmf19/includes/changetags/ChangeTags.php: SWAT: Avoid full RC table scans in ChangeTags::updateTags() gerrit:232485 (duration: 00m 13s)
  • 15:25 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Adjust mediawiki.org RSS whitelist to allow technology blog feeds gerrit:118956 (duration: 00m 13s)
  • 15:00 andrewbogott: rebooting labvirt1007
  • 13:58 godog: stop puppet on ms-fe1* while merging swift refactoring
  • 13:58 godog: stop puppet on ms-be1* while merging swift refactoring
  • 09:51 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1018 for normal traffic only (duration: 00m 13s)
  • 09:49 logmsgbot: reedy@tin Synchronized wmf-config/InitialiseSettings.php: Add *.webarchive.org.uk to wgCopyUploadsDomains whitelist (duration: 00m 12s)
  • 08:28 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Aug 19 08:28:19 UTC 2015 (duration 28m 18s)
  • 07:39 jynus: About to perform a schema change on flowdb
  • 03:44 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf19/includes/rcfeed/RCFeedFormatter.php: Fix Flow RC regression in 1.26wmf19 (duration: 00m 12s)
  • 03:43 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf19/includes/changes/RecentChange.php: Fix Flow RC regression in 1.26wmf19 (duration: 00m 12s)
  • 03:43 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf18/includes/rcfeed/RCFeedFormatter.php: Fix Flow RC regression in 1.26wmf18 (duration: 00m 12s)
  • 03:42 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf18/includes/changes/RecentChange.php: Fix Flow RC regression in 1.26wmf18 (duration: 00m 12s)
  • 03:14 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf19) at 2015-08-19 03:14:06+00:00
  • 03:07 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf19/cache/l10n: l10nupdate for 1.26wmf19 (duration: 10m 48s)
  • 02:39 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf18) at 2015-08-19 02:39:54+00:00
  • 02:36 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf18/cache/l10n: l10nupdate for 1.26wmf18 (duration: 07m 51s)
  • 01:10 yurik: updated kartotherian
  • 00:40 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf19/extensions/Flow/: Sync Flow 1.26wmf19 for RC insert failure. (duration: 00m 15s)
  • 00:39 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf18/extensions/Flow/: Sync Flow 1.26wmf18 for RC insert failure. (duration: 00m 14s)

2015-08-18

  • 23:59 logmsgbot: krenair@tin Synchronized wmf-config/wikitech.php: T59040 (duration: 00m 12s)
  • 23:37 mutante: added papaul (pt1979) to WMF LDAP group
  • 23:08 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf18/extensions/Flow/: Sync Flow 1.26wmf18 for watchlist fix. (duration: 00m 14s)
  • 23:02 logmsgbot: hoo@tin Synchronized wmf-config/: Set $wgPropertySuggesterClassifyingPropertyIds for testwikidata (duration: 00m 14s)
  • 22:28 logmsgbot: krenair@tin Synchronized php-1.26wmf19/extensions/VisualEditor/modules/ve-mw/ui/dialogs/ve.ui.MWSaveDialog.js: https://gerrit.wikimedia.org/r/#/c/232385/ (duration: 00m 12s)
  • 22:17 logmsgbot: ori@tin Synchronized php-1.26wmf19/includes/OutputPage.php: 1a4f1df2fe (duration: 00m 12s)
  • 21:22 awight: updated paymentswiki from 823393264d6795bbaec490ff86f17580f722e598 to fca36026b1e90298abd93562803d3ea7d6893d96
  • 19:26 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.26wmf19
  • 19:21 logmsgbot: twentyafterfour@tin Finished scap: testwiki to 1.26wmf19 (duration: 51m 01s)
  • 18:30 logmsgbot: twentyafterfour@tin Started scap: testwiki to 1.26wmf19
  • 18:11 logmsgbot: ori@tin Synchronized php-1.26wmf18/includes/OutputPage.php: 6ee94ca47c: Load all CSS in the top queue (duration: 00m 13s)
  • 18:07 robh: sodium returned to normal, mailman window over.
  • 17:38 logmsgbot: ori@tin Synchronized php-1.26wmf18/includes: 91ae6a39df, 4cc9622214: Added wfTransactionalTimeLimit() method and applied it; Try to make POSTs as transactional as possible (duration: 00m 16s)
  • 17:21 robh: T108099 complete, mailman restarted for a few minutes while i prepare next task.
  • 17:17 robh: puppet disabled on sodium, no touch.
  • 17:03 robh: mailman maint window starts now, list delivery will remain sporadic until I finish. (It'll work off and on, no messages should be lost)
  • 15:42 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/232241/ (duration: 00m 13s)
  • 15:06 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Remove extra transcode enablings gerrit:232228 (duration: 00m 13s)
  • 15:04 andrewbogott: rebooting labvirt1006
  • 08:18 _joe_: reimaging mw1152
  • 08:14 godog: restart cassandra on restbase100[569] to pick up latest openjdk
  • 08:04 _joe_: depooling mw1152 from the imagescalers pool
  • 08:03 godog: restart cassandra on restbase100[348] to pick up latest openjdk
  • 07:23 legoktm: live hacking on mw1017 for T109236
  • 05:45 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Aug 18 05:45:47 UTC 2015 (duration 45m 46s)
  • 02:25 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf18) at 2015-08-18 02:25:28+00:00
  • 02:21 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf18/cache/l10n: l10nupdate for 1.26wmf18 (duration: 06m 50s)
  • 00:02 ottomata: analytics1041 down, attempting power cycle

2015-08-17

  • 22:19 matt_flaschen: LQT->Flow done on MediaWiki.org.
  • 21:57 logmsgbot: mattflaschen@tin Synchronized wmf-config: LQT->Flow: Make frozen wikis no longer able to create LQT pages (duration: 00m 13s)
  • 21:31 chasemp: remove php5-xdebug from terbium per mattflaschen
  • 21:10 MaxSem: renamed Gadget:Invention, Travel, & Adventure --> Gadget Invention, Travel, & Adventure on enwiki using moveBatch.php to work around a permissions screwup
  • 20:54 bd808: T109369: Restarted logstash on logstash1003; parsoid gelf events not being recorded since 2015-08-15
  • 20:16 subbu: deployed parsoid version 4b656b72
  • 19:19 ottomata: stopping kafka on analytics1012, preparing to reinstall with Jessie and rename to kafka1012
  • 15:44 logmsgbot: krenair@tin Synchronized wmf-config/extension-list: https://gerrit.wikimedia.org/r/#/c/232051/ - remove WikiGrok from extension-list, extension is no longer deployed (duration: 00m 11s)
  • 15:40 logmsgbot: krenair@tin Synchronized multiversion/MWMultiVersion.php: https://gerrit.wikimedia.org/r/#/c/231982/2 - clear up --wiki usage to mwscript (duration: 00m 12s)
  • 15:34 logmsgbot: krenair@tin Synchronized php-1.26wmf18/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.Target.js: https://gerrit.wikimedia.org/r/#/c/232048/ (duration: 00m 11s)
  • 15:07 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/231984/ (duration: 00m 13s)
  • 15:05 andrewbogott: rebooting labvirt1004
  • 15:03 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/232021/ (duration: 00m 12s)
  • 14:30 mobrovac: restbase updated production cluster to ed17952
  • 14:12 mobrovac: restbase deployed ed17952 on restbase1001
  • 13:58 mobrovac: restbase deploying ed17952 on staging
  • 11:42 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Reverting change to settings before schema change (no more lag) (duration: 00m 12s)
  • 11:29 jynus: reloading dbproxy1003 haproxy config- it was a temporal max_connections issue; db1043 should be the canonical server again
  • 10:09 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Solving lag issues while schema change is ongoing (duration: 00m 12s)
  • 09:43 jynus: about to perform schema change on centralauth
  • 09:36 godog: upgrade openjdk on restbase100[127] and restart cassandra
  • 05:54 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Aug 17 05:54:24 UTC 2015 (duration 54m 23s)
  • 05:11 twentyafterfour: restarted phd to pick up new configuration, (and to silence the phabricator 'setup issue' warning
  • 04:47 twentyafterfour: changed phabricator policy for the multimeter application from 'public' to 'all users'
  • 04:44 twentyafterfour: deployed https://gerrit.wikimedia.org/r/#/c/231983/ to iridium and restarted apache
  • 04:17 mutante: free some disk space on iridium. apt-get clean; gzip /var/log/account/pacct.0; some apache logs .;.
  • 02:29 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf18) at 2015-08-17 02:29:19+00:00
  • 02:25 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf18/cache/l10n: l10nupdate for 1.26wmf18 (duration: 11m 08s)

2015-08-16

  • 18:15 logmsgbot: krenair@tin Synchronized wmf-config/extension-list: https://gerrit.wikimedia.org/r/#/c/169612/ - remove Extension:Oversight (duration: 00m 21s)
  • 18:14 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/169612/ - remove Extension:Oversight (duration: 00m 25s)
  • 05:39 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Aug 16 05:39:28 UTC 2015 (duration 39m 27s)
  • 02:29 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf18) at 2015-08-16 02:29:19+00:00
  • 02:25 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf18/cache/l10n: l10nupdate for 1.26wmf18 (duration: 11m 52s)

2015-08-15

  • 22:56 legoktm: removed 13 bounce_records for User:odder from bouncehandler database
  • 16:07 _joe_: removing manually core dumps from last night's outage on all appservers in eqiad, they occpy on average 30 GB/server
  • 16:05 ottomata: starting rolling restart of kafka brokers to apply auto leader rebalance enable = false
  • 14:49 ottomata: stopping kafka broker on analytics1012 to again try to figure out why camus can't consume from it
  • 12:46 bblack: restarted gitblit on antimony, because Java is Awesome
  • 05:41 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Aug 15 05:41:57 UTC 2015 (duration 41m 56s)
  • 04:38 andrewbogott: killing some rsync processes on labstore1002 because iowaits are through the roof
  • 02:30 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf18) at 2015-08-15 02:29:05+00:00
  • 02:25 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf18/cache/l10n: l10nupdate for 1.26wmf18 (duration: 06m 37s)
  • 01:15 ottomata: starting broker on analytics1012, camus wasn't happy about that either. hrm.
  • 00:58 ottomata: stopping kafka broker on analytics1012, it is causing consumption problems with camus, will look into why later.

2015-08-14

  • 23:47 logmsgbot: ori@tin Synchronized php-1.26wmf18/includes/libs/ReplacementArray.php: (no message) (duration: 00m 27s)
  • 21:54 mutante: restarted nutcracker on mw1010
  • 21:26 ori: deployed job runner 808d1ae08d40
  • 21:15 ejegg: updated crm from 4f40ac6de0385982d8e672b1ed30ff1a2a2a2aa1 to fc0fcc8f5af262b56392d3f4f5998f8ea08c99a8
  • 19:29 logmsgbot: ori@tin Synchronized php-1.26wmf18/includes/resourceloader/ResourceLoader.php: f72009a543: ResourceLoader: apply minify-js filter to config scripts (duration: 00m 13s)
  • 18:27 logmsgbot: ori@tin Synchronized php-1.26wmf18/extensions/MultimediaViewer: 9ee0437bc6: Updated mediawiki/core Project: mediawiki/extensions/MultimediaViewer 645b6c9e93fae13e09e5b493547aecc5a2e933ae (duration: 00m 12s)
  • 18:24 ori: Repooling mw1041 now that T108601 is resolved.
  • 17:59 yurik: deployed latest kartotherian
  • 14:59 andrewbogott: rebooting labvirt1003
  • 13:53 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1042 (vslow, dump) (duration: 00m 12s)
  • 13:14 logmsgbot: reedy@tin Synchronized php-1.26wmf18/extensions/CirrusSearch/: Fix ElasticaQuery logspam (duration: 00m 13s)
  • 13:06 logmsgbot: reedy@tin Synchronized php-1.26wmf18/extensions/GeoData: Fix ElasticaQuery logspam (duration: 00m 13s)
  • 13:06 logmsgbot: reedy@tin Synchronized php-1.26wmf18/extensions/Flow: Fix ElasticaQuery logspam (duration: 00m 13s)
  • 12:30 jynus: Restarting db1042 after data import
  • 12:11 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Load-balance db1036 roles (duration: 00m 11s)
  • 11:57 logmsgbot: reedy@tin Synchronized php-1.26wmf18/extensions/Translate: Stop calling deprecated Elastica function (duration: 00m 13s)
  • 08:26 akosiaris: upgraded and restarted apertium on sca100{1,2}
  • 08:11 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-apy_0.1+svn~61425-1
  • 07:10 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Aug 14 07:10:37 UTC 2015 (duration 10m 36s)
  • 06:34 Jamesofur: reset email/password for User:Auréola after multi factor user confirmation.
  • 02:35 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf18) at 2015-08-14 02:35:19+00:00
  • 02:31 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf18/cache/l10n: l10nupdate for 1.26wmf18 (duration: 06m 35s)
  • 01:49 matt_flaschen: Resumed LQT->Flow conversion of mw:Project:Support_desk on mw1041
  • 01:46 logmsgbot: ori@tin Synchronized php-1.26wmf18/includes/OutputPage.php: I5e6c79c70: Optimize the order of styles and scripts in <head> (duration: 00m 12s)

2015-08-13

  • 23:45 awight: rollback paymentswiki from 2e7b449224317779d53ff84527166c0d378a0a40 to 823393264d6795bbaec490ff86f17580f722e598
  • 23:16 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/228477/ and https://gerrit.wikimedia.org/r/#/c/231074/ (duration: 00m 12s)
  • 23:15 awight: update paymentswiki-staging 99e3ce08117d18b15bc8138b447c4c21bd452d28 to 65b05fc11896325ae9749318b296c4396a64f649
  • 23:15 logmsgbot: krenair@tin Synchronized w/static/images/project-logos/mrwikibooks.png: https://gerrit.wikimedia.org/r/#/c/228477/ (duration: 00m 12s)
  • 23:11 logmsgbot: krenair@tin Synchronized php-1.26wmf18/extensions/Flow: https://gerrit.wikimedia.org/r/#/c/231450/1 (duration: 00m 14s)
  • 23:03 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/227330/ (duration: 00m 12s)
  • 22:35 awight: rollback payments-wiki-staging to 99e3ce08117d18b15bc8138b447c4c21bd452d28
  • 22:29 awight: update paymentswiki from 823393264d6795bbaec490ff86f17580f722e598 to 2e7b449224317779d53ff84527166c0d378a0a40
  • 22:23 awight: update payments-wiki-staging from 99e3ce08117d18b15bc8138b447c4c21bd452d28 to 2e7b449224317779d53ff84527166c0d378a0a40
  • 22:14 Krenair: Running refreshLinks --dfn-only in a screen on terbium for T44180
  • 21:45 awight: rollback payments-wiki-staging to 99e3ce08117d18b15bc8138b447c4c21bd452d28
  • 21:45 awight: update payments-wiki-staging from 99e3ce08117d18b15bc8138b447c4c21bd452d28 to 96a369651c1130b0a8e53a6395f83c0b9329b9f8
  • 21:43 logmsgbot: ori@tin Synchronized php-1.26wmf18/includes/OutputPage.php: Roll back: Test impact of I5e6c79c (duration: 00m 12s)
  • 21:37 logmsgbot: ori@tin Synchronized php-1.26wmf18/includes/OutputPage.php: Test impact of I5e6c79c (duration: 00m 12s)
  • 21:16 mutante: torrus broken - doing https://wikitech.wikimedia.org/wiki/Torrus#Deadlock_problem
  • 21:14 mutante: service gitblit restart on antimony (maybe that should be paging :)
  • 21:06 bd808: `sudo /etc/init.d/ganglia-monitor restart` on logstash100[1-6] fixed ganglia data loss
  • 20:46 mutante: killed ganglia aggregator for logstash on carbon
  • 20:40 bd808: ganglia not getting elasticsearch jvm data for logstash cluster since 2015-08-13T12:00 -- https://ganglia.wikimedia.org/latest/?c=Logstash+cluster+eqiad&&m=es_heap_used
  • 19:56 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: all wikis to 1.26wmf18
  • 19:38 logmsgbot: demon@tin Synchronized php-1.26wmf18/extensions/CirrusSearch/: (no message) (duration: 00m 11s)
  • 19:35 logmsgbot: demon@tin Synchronized php-1.26wmf18/extensions/Graph: (no message) (duration: 00m 10s)
  • 19:31 logmsgbot: demon@tin Synchronized php-1.26wmf18/extensions/Graph: (no message) (duration: 00m 11s)
  • 18:32 logmsgbot: ori@tin Synchronized php-1.26wmf18/extensions/SyntaxHighlight_GeSHi/extension.json: If0851400: Fix-up for I2de8a400d: explicitly declare module position (duration: 00m 12s)
  • 17:47 logmsgbot: kaldari@tin Synchronized wmf-config/InitialiseSettings.php: syncing InitialiseSettings for WikidataPageBanner (duration: 00m 12s)
  • 17:46 logmsgbot: kaldari@tin Finished scap: (no message) (duration: 50m 05s)
  • 16:55 logmsgbot: kaldari@tin Started scap: (no message)
  • 16:54 logmsgbot: kaldari@tin Synchronized wmf-config/InitialiseSettings.php: syncing InitialiseSettings for WikidataPageBanner off (duration: 00m 11s)
  • 16:51 logmsgbot: kaldari@tin Synchronized wmf-config/CommonSettings.php: syncing CommonSettings for WikidataPageBanner (duration: 00m 12s)
  • 16:50 logmsgbot: kaldari@tin Synchronized wmf-config/InitialiseSettings.php: syncing InitialiseSettings for WikidataPageBanner (duration: 00m 12s)
  • 16:31 logmsgbot: kaldari@tin Synchronized php-1.26wmf18/.gitmodules: (no message) (duration: 00m 11s)
  • 16:30 logmsgbot: kaldari@tin Synchronized php-1.26wmf17/.gitmodules: (no message) (duration: 00m 13s)
  • 16:29 logmsgbot: kaldari@tin Synchronized php-1.26wmf17/extensions/WikidataPageBanner: (no message) (duration: 00m 12s)
  • 16:29 logmsgbot: kaldari@tin Synchronized php-1.26wmf18/extensions/WikidataPageBanner: (no message) (duration: 00m 12s)
  • 16:24 logmsgbot: demon@tin Synchronized wmf-config/: undeploy wikigrok (duration: 00m 12s)
  • 16:12 yurik: sync deployed tilerator
  • 15:20 logmsgbot: thcipriani@tin Synchronized php-1.26wmf17/extensions/ContentTranslation/modules/tools/ext.cx.tools.images.js: SWAT: Images: validate image id before adapting to prevent js error gerrit:231229 (duration: 00m 11s)
  • 15:10 logmsgbot: thcipriani@tin Synchronized php-1.26wmf18/extensions/ContentTranslation/modules/tools/ext.cx.tools.images.js: SWAT: Images: validate image id before adapting to prevent js error gerrit:231230 (duration: 00m 12s)
  • 15:04 andrewbogott: rebooting labvirt1002
  • 13:36 jynus: kill custom query hiting s6 master from terbium. Use of a slave is required.
  • 13:11 andrewbogott: graceful’d apache2 on labcontrol1001
  • 12:42 andrewbogott: restarted keystone on labcontrol1001
  • 08:16 _joe_: removing all stale aggregator configs from netmon1001
  • 08:15 godog: upgrade cassandra on restbase1009
  • 08:11 godog: upgrade cassandra on restbase1006
  • 08:07 godog: upgrade cassandra on restbase1005
  • 08:00 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Aug 13 08:00:40 UTC 2015 (duration 0m 39s)
  • 07:36 _joe_: killing all gmond instances on netmon1001, trying to fix ganglia-monitor-aggregator
  • 06:48 matt_flaschen: Stopped Support desk LQT->Flow conversion for tonight
  • 05:03 logmsgbot: ori@tin Synchronized php-1.26wmf17/includes/cache/MessageCache.php: 5f1ab59d31: MessageCache: derive the hash from the cache contents (duration: 00m 12s)
  • 05:02 logmsgbot: ori@tin Synchronized php-1.26wmf18/includes/cache/MessageCache.php: 5f1ab59d31: MessageCache: derive the hash from the cache contents (duration: 00m 12s)
  • 03:04 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf18) at 2015-08-13 03:04:52+00:00
  • 03:01 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf18/cache/l10n: l10nupdate for 1.26wmf18 (duration: 06m 14s)
  • 02:44 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf17) at 2015-08-13 02:44:49+00:00
  • 02:38 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf17/cache/l10n: l10nupdate for 1.26wmf17 (duration: 10m 47s)
  • 01:26 matt_flaschen: Restarted conversion of support desk from LQT->Flow using convertLqtPageOnLocalWiki.php, using hhvm on mw1041
  • 01:16 logmsgbot: ori@tin Synchronized wmf-config/StartProfiler.php: I482b120289: Ensure all Xenon records begin with the script base name (duration: 00m 12s)
  • 01:07 ori: Depooled mw1041 so it can be set aside for LQT->Flow conversion script (T108601)

2015-08-12

  • 23:55 ori: fluorine is struggling due to I941660b5; I'm fixing.
  • 23:54 logmsgbot: krenair@tin Synchronized php-1.26wmf17/extensions/WikimediaMaintenance: https://gerrit.wikimedia.org/r/#/c/231194/ (duration: 00m 12s)
  • 23:47 logmsgbot: krenair@tin Synchronized php-1.26wmf18/extensions/WikimediaMaintenance: https://gerrit.wikimedia.org/r/#/c/231193/ (duration: 00m 12s)
  • 23:44 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/231169/ (duration: 00m 12s)
  • 23:16 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/231158/ (duration: 00m 11s)
  • 23:15 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/231158/ (duration: 00m 13s)
  • 23:13 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/230729 (duration: 00m 13s)
  • 23:02 logmsgbot: legoktm@tin Synchronized wmf-config: Isolate wikidata.org cookies and CORS policies (duration: 00m 12s)
  • 22:21 matt_flaschen: Killed support desk conversion again to review XDebug information.
  • 22:17 matt_flaschen: Resumed the support desk conversion.
  • 21:53 awight: updated paymentswiki from 325640bd70680a08ae77fd117433565634a98d88 to 99e3ce08117d18b15bc8138b447c4c21bd452d28
  • 20:42 subbu: deployed parsoid version a271c205
  • 20:23 milimetric: deployed the latest EventLogging master to eventlog1001
  • 14:34 godog: upgrade cassandra on restbase1008
  • 14:30 godog: upgrade cassandra on restbase1004
  • 14:22 godog: upgrade cassandra on restbase1003
  • 13:00 akosiaris: disabled puppet on maps-test200X
  • 12:36 logmsgbot: aude@tin Synchronized arbitraryaccess.dblist: Enable arbitrary access on dewiki, frwiki, jawiki and s3 wikis (duration: 00m 12s)
  • 12:21 logmsgbot: aude@tin Synchronized wmf-config/Wikibase-labs.php: Add Wikisource badge config (duration: 00m 13s)
  • 12:20 logmsgbot: aude@tin Synchronized wmf-config/Wikibase-production.php: Add Wikisource badge config (duration: 00m 11s)
  • 10:27 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: depool db2040 (duration: 00m 11s)
  • 07:47 _joe_: restarted apertium-apy on sca1001 and sca1002, too many open files, probably leaking
  • 06:12 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Aug 12 06:12:31 UTC 2015 (duration 12m 30s)
  • 05:54 matt_flaschen: Killed support desk conversion. Will resume with profiling tomorrow.
  • 04:45 bblack: starting slow "apt-get -y upgrade" on cp* (mostly, nginx -> +wmf2), will execute over ~18-24h
  • 03:34 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/229608 (duration: 00m 12s)
  • 03:06 ori: Installing xdebug on terbium so matt_flaschen can debug memory leak
  • 03:04 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf18) at 2015-08-12 03:04:47+00:00
  • 02:58 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf18/cache/l10n: l10nupdate for 1.26wmf18 (duration: 11m 13s)
  • 02:30 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf17) at 2015-08-12 02:30:14+00:00
  • 02:26 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf17/cache/l10n: l10nupdate for 1.26wmf17 (duration: 06m 52s)
  • 01:22 mutante: zirconium - shut down, i'm sure, mollyguard
  • 00:22 yurik: updated kartotherian
  • 00:09 mutante: restarted gitblit
  • 00:06 logmsgbot: ori@tin Synchronized php-1.26wmf17/extensions/Echo: Updated mediawiki/core Project: mediawiki/extensions/Echo 32e5bcf90c702 (duration: 00m 13s)
  • 00:06 logmsgbot: ori@tin Synchronized php-1.26wmf18/extensions/Echo: Updated mediawiki/core Project: mediawiki/extensions/Echo 3ab0b7e0f4948 (duration: 00m 12s)
  • 00:02 matt_flaschen: Resumed convertLqtPageOnLocalWiki.php run on MediaWiki.org's Project:Support_desk.

2015-08-11

  • 23:19 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf18/extensions/Flow/: Sync Flow 1.26wmf18 for memory leaks (duration: 00m 14s)
  • 23:08 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/230805/ (duration: 00m 12s)
  • 23:05 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/230804/ (duration: 00m 12s)
  • 21:03 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.26wmf18
  • 21:02 twentyafterfour: deployed scap fixes for my dumb mistakes
  • 20:10 logmsgbot: ori@tin Synchronized php-1.26wmf18/includes/objectcache/MultiWriteBagOStuff.php: 0acfe6a5bb: Fix argument handling in MultiWriteBagOStuff::get() (duration: 00m 12s)
  • 19:28 logmsgbot: ori@tin Synchronized php-1.26wmf18/includes/resourceloader/ResourceLoader.php: I2089b21fc: ResourceLoader: make "cacheReport" option false by default (duration: 00m 13s)
  • 19:28 logmsgbot: ori@tin Synchronized php-1.26wmf17/includes/resourceloader/ResourceLoader.php: I2089b21fc: ResourceLoader: make "cacheReport" option false by default (duration: 00m 11s)
  • 19:26 logmsgbot: catrope@tin Synchronized php-1.26wmf18/extensions/Flow/modules/editor/editors/visualeditor/mw.flow.ve.Target.js: Fix missing editor switcher (duration: 00m 12s)
  • 18:58 logmsgbot: twentyafterfour@tin Finished scap: again: sync new branch 1.26wmf18 and update testwiki (duration: 04m 58s)
  • 18:53 logmsgbot: twentyafterfour@tin Started scap: again: sync new branch 1.26wmf18 and update testwiki
  • 18:44 logmsgbot: twentyafterfour@tin scap failed: OSError [Errno 1] Operation not permitted: '/srv/mediawiki-staging/wikiversions.php' (duration: 29m 27s)
  • 18:37 mutante: grafana switched to node krypton (jessie/VM)
  • 18:21 bd808: logstash log event volume back to normal levels following elasticsearch upgrade
  • 18:15 logmsgbot: twentyafterfour@tin Started scap: sync new branch 1.26wmf18 and update testwiki
  • 18:06 bd808: logstash cluster recovered after upgrade of elasticsearch on logstash1006
  • 18:03 bd808: upgraded elasticsearch to 1.7.1 on logstash1006; logstash-2015.08.11 shard recovering
  • 18:02 bd808: upgrading elasticsearch on logstash1006
  • 18:01 bd808: logstash cluster recovered after upgrade of elasticsearch on logstash1005
  • 17:43 bd808: log event volume in logstash dropped dramatically again; seems to correlate with final recovery of logstash-2015.08.11 shard
  • 17:29 bd808: upgraded elasticsearch to 1.7.1 on logstash1005; logstash-2015.08.11 shard recovering
  • 17:28 bd808: upgrading elasticsearch on logstash1005
  • 17:27 bd808: logstash event volume recovered after restarting all 3 logstash services
  • 17:14 bd808: log event volume in logstash dropped dramatically at 16:49; investigating
  • 17:13 bd808: logstash cluster recovered after upgrade of elasticsearch on logstash1004
  • 16:42 bd808: upgraded elasticsearch to 1.7.1 on logstash1004; logstash-2015.08.11 shard recovering
  • 16:42 mutante: restarted Apache on Etherpad
  • 16:38 bd808: upgraded elasticsearch to 1.7.1 on logstash1003
  • 16:37 bd808: upgraded elaasticsearch to 1.7.1 on logstash1002
  • 16:36 bd808: upgraded elaasticsearch to 1.7.1 on logstash1001
  • 16:23 bd808: logstash upgrade on logstash1003 complete
  • 16:20 bd808: logstash upgrade on logstash1002 complete
  • 16:16 bd808: logstash upgrade on logstash1001 complete
  • 15:50 jynus: nuking db1002-db1007 on icinga
  • 15:49 bd808: upgrading logstash on logstash1001
  • 15:47 bd808: Trebuchet deploy of logstash/plugins: Add logstash-filter-prune 0.1.5 (36144b2)
  • 15:36 bd808: Disabled puppet on logstash100[1-3] in preparation for upgrade to 1.5.3
  • 15:31 logmsgbot: bd808@tin Synchronized wmf-config/InitialiseSettings.php: logging: Only send info and higher to logstash by default (4388a84) 2/2 (actually rebased this time) (duration: 00m 11s)
  • 15:30 logmsgbot: bd808@tin Synchronized wmf-config/logging.php: logging: Only send info and higher to logstash by default (4388a84) 1/2 (actually rebased this time) (duration: 00m 11s)
  • 15:17 logmsgbot: bd808@tin Synchronized wmf-config/InitialiseSettings.php: Touched wmf-config/InitialiseSettings.php (duration: 00m 13s)
  • 15:12 logmsgbot: bd808@tin Synchronized wmf-config/InitialiseSettings.php: logging: Only send info and higher to logstash by default (4388a84) 2/2 (duration: 00m 12s)
  • 15:11 logmsgbot: bd808@tin Synchronized wmf-config/logging.php: logging: Only send info and higher to logstash by default (4388a84) 1/2 (duration: 00m 12s)
  • 10:45 jynus: general maintenance on db1042 (restart, upgrade, db reconstruction)
  • 10:38 godog: upgrade cassandra on restbase1007
  • 10:31 godog: upgrade cassandra on restbase1002
  • 10:25 godog: upgrade cassandra on restbase1001
  • 09:56 paravoid: switched routing-system autonomous-system to eqiad's subAS on cr1-eqiad/cr2--eqiad
  • 09:09 godog: reboot ms-be2009, cpu soft lockup
  • 05:27 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Aug 11 05:27:33 UTC 2015 (duration 27m 32s)
  • 02:27 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf17) at 2015-08-11 02:26:58+00:00
  • 02:23 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf17/cache/l10n: l10nupdate for 1.26wmf17 (duration: 06m 48s)
  • 01:20 logmsgbot: ori@tin Synchronized php-1.26wmf17/includes/resourceloader/ResourceLoader.php: I2089b21fc: Revert resourceloader: Add must-revalidate to Cache-Control (duration: 00m 12s)
  • 00:10 mutante: apache restart on krypton

2015-08-10

  • 23:53 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-common.php: Limit the number of states in a cirrussearch query (duration: 00m 11s)
  • 23:44 logmsgbot: krenair@tin Synchronized php-1.26wmf17/extensions/TimedMediaHandler/TimedMediaIframeOutput.php: https://gerrit.wikimedia.org/r/#/c/230656/ (duration: 00m 12s)
  • 23:40 logmsgbot: ori@tin Synchronized multiversion/MWMultiVersion.php: I511999: Convert multiversion scripts to use wikiversions.php (duration: 00m 12s)
  • 23:11 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 12s)
  • 23:07 logmsgbot: krenair@tin Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/228040/ (duration: 00m 14s)
  • 23:04 ori: deployed scap a404a39b32... Build wikiversions.php in addition to wikiversions.cdb
  • 22:22 mutante: restart gitblit
  • 21:53 mutante: krypton unloaded mod proxy_balancer
  • 21:30 awight: updated paymentswiki from af16d371f9c46d4f0b78986080f2a2be3226ace8 to 325640bd70680a08ae77fd117433565634a98d88
  • 20:57 logmsgbot: ori Synchronized php-1.26wmf17/includes/cache/MessageCache.php: I2089b21fc: MessageCache: use APC for local caching, rather than files (duration: 00m 12s)
  • 20:12 subbu: deployed parsoid version 7b554ce2f
  • 19:57 yurik: synced new kartotherian
  • 19:22 logmsgbot: ori Synchronized php-1.26wmf17/includes: I9a1aa76de: Moved ObjectCacheSessionHandler renewal logic to wfSetupSession() (duration: 00m 16s)
  • 17:12 akosiaris: stopped postgres on maps-test200{2,3,4}
  • 16:59 logmsgbot: ori Synchronized php-1.26wmf17/includes/OutputPage.php: I2089b21fc: Load mediawiki.legacy.commonPrint styles with a media type property (2/2) (duration: 00m 11s)
  • 16:58 logmsgbot: ori Synchronized php-1.26wmf17/resources/Resources.php: I2089b21fc: Load mediawiki.legacy.commonPrint styles with a media type property (1/2) (duration: 00m 11s)
  • 16:50 logmsgbot: ori Synchronized php-1.26wmf17/extensions/wikihiero: I2089b21fc: Updated mediawiki/core Project: mediawiki/extensions/wikihiero (duration: 00m 12s)
  • 15:11 logmsgbot: thcipriani Synchronized php-1.26wmf17/extensions/ContentTranslation/api/ApiContentTranslationPublish.php: SWAT: Enable scrubWikitext=1 in HTML to wikitext conversion using parsoid gerrit:230381 (duration: 00m 13s)
  • 14:47 akosiaris: running an rsync from nas1001-a to local disks on helium
  • 14:42 ottomata: restarted all varnishkafka instances to pick up proper confs (puppet should have done this!)
  • 14:29 ottomata: starting upgrade of existing kafka cluster to 0.8.2.1 jessie - https://etherpad.wikimedia.org/p/kafka_0.8.2.1_migration2
  • 12:38 bblack: deployed nginx-1.9.3-1+wmf2 to cp1065, cp1070, cp1071 (1x each text, upload, misc) for validation
  • 11:16 logmsgbot: hoo Synchronized wmf-config/: Revert "Set dispatchBatchChunkFactor to 10 for now" (duration: 00m 12s)
  • 11:16 logmsgbot: hoo Synchronized php-1.26wmf17/extensions/Wikidata/: Revert "Set dispatchBatchChunkFactor to 10 for now" (duration: 00m 20s)
  • 09:35 paravoid: manually firewalled backup4001 TCP on neon to temporarily stop the nsca alert storm
  • 09:12 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool db1042 for maintenance (duration: 00m 12s)
  • 08:43 _joe_: manually running logrotate on iridium
  • 07:44 godog: reboot ms-be2006, xfs hosed
  • 07:27 akosiaris: rebooting backup4001
  • 07:20 jynus: schema change on testwikidatawiki
  • 06:52 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Increase db1035 weight (duration: 00m 13s)
  • 05:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Aug 10 05:31:29 UTC 2015 (duration 31m 28s)
  • 03:13 bblack: restarted apache2 on iridium JIC
  • 03:13 bblack: rm /var/log/apache2/phabricator_access.log.1 on iridium (disk full, fixed for now)
  • 02:23 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf17) at 2015-08-10 02:23:47+00:00
  • 02:20 logmsgbot: l10nupdate Synchronized php-1.26wmf17/cache/l10n: l10nupdate for 1.26wmf17 (duration: 06m 43s)

2015-08-09

  • 17:57 urandom: issuing nodetool cleanup on restbase1006
  • 12:56 logmsgbot: hoo Synchronized php-1.26wmf17/extensions/Wikidata/: Set dispatchBatchChunkFactor to 10 for now (duration: 00m 20s)
  • 06:20 twentyafterfour: restarted phabricator phd (just in case - the full partition may have caused the daemons to be in a broken state)
  • 06:17 twentyafterfour: moved some log files on iridium into /srv/logs to free space on /
  • 05:12 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Aug 9 05:12:23 UTC 2015 (duration 12m 22s)
  • 02:23 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf17) at 2015-08-09 02:23:22+00:00
  • 02:20 logmsgbot: l10nupdate Synchronized php-1.26wmf17/cache/l10n: l10nupdate for 1.26wmf17 (duration: 06m 32s)

2015-08-08

  • 14:58 urandom: issuing nodetool cleanup on restbase1005
  • 14:57 urandom: issuing nodetool cleanup on restbase1007
  • 05:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Aug 8 05:30:22 UTC 2015 (duration 30m 21s)
  • 04:20 urandom: issuing nodetool cleanup on restbase1008
  • 03:00 logmsgbot: hoo Synchronized php-1.26wmf17/extensions/Wikidata/: Hack: Don't write change rows where LENGTH(change_info) > 65500 (duration: 00m 21s)
  • 02:26 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf17) at 2015-08-08 02:26:20+00:00
  • 02:23 logmsgbot: l10nupdate Synchronized php-1.26wmf17/cache/l10n: l10nupdate for 1.26wmf17 (duration: 06m 20s)
  • 01:53 hoo: Deleted change 237365841 as well
  • 01:37 hoo: Deleted changes 237357747 and 237363245 from wikidata's wb_changes
  • 01:32 logmsgbot: ori Synchronized php-1.26wmf17/resources/src/mediawiki.legacy/wikibits.js: I664ba9b0af: Override document.writeln to prevent it from blanking pages (duration: 00m 13s)

2015-08-07

  • 21:00 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/230227/ - should be a noop for prod (duration: 00m 12s)
  • 19:40 logmsgbot: hoo Synchronized wmf-config/: Bump wgCacheEpoch for Wikidata (duration: 00m 13s)
  • 19:29 logmsgbot: hoo Synchronized php-1.26wmf17/extensions/Wikidata/: Update Wikibase: Fix WB spinner, UnresolvedRedirectException handling on client (duration: 00m 21s)
  • 18:08 gwicke: switched restbase1001 to CMS temporarily, to gather metrics; will switch back to G1GC tonight
  • 17:09 yurik: synced kartotherian to maps-test* servers again, restarted the service
  • 16:00 jynus: repool db1035 database with low traffic
  • 15:54 logmsgbot: thcipriani Synchronized php-1.26wmf17/extensions/ContentTranslation/modules/tools/ext.cx.tools.images.js: FIX: Use .attr() to set the resource attribute of image, while adapting gerrit:230101 (duration: 00m 11s)
  • 15:10 yurik: synced kartotherian to maps-test* servers
  • 15:10 gwicke: switched cassandra staging cluster back to G1GC / default puppet
  • 13:40 moritzm: install pcre security updates on elastic*, analytics*, wtp*, db* and es*
  • 13:40 urandom: starting nodetool cleanup on restbase1004 (see: T108083)
  • 13:37 urandom: starting nodetool cleanup on restbase1002 (see: T108083)
  • 12:07 moritzm: restarted HHVM on jobrunners/imagescalers in eqiad/codfw for libtidy/PCRE security updates
  • 09:24 logmsgbot: akosiaris Synchronized wmf-config/PoolCounterSettings-eqiad.php: (no message) (duration: 00m 12s)
  • 09:02 akosiaris: disabled helium as a poolcounter temporarily while applying base::firewall again
  • 09:01 logmsgbot: akosiaris Synchronized wmf-config/PoolCounterSettings-eqiad.php: (no message) (duration: 00m 12s)
  • 08:52 moritzm: restarted HHVM on API apaches in codfw for libtidy/PCRE security updates
  • 08:26 godog: restart cassandra on test cluster
  • 08:11 godog: upgrade cassandra test cluster to openjdk 8u66-b01-1~bpo8+1
  • 07:34 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Aug 7 07:34:34 UTC 2015 (duration 34m 33s)
  • 02:43 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf17) at 2015-08-07 02:43:35+00:00
  • 02:40 logmsgbot: l10nupdate Synchronized php-1.26wmf17/cache/l10n: l10nupdate for 1.26wmf17 (duration: 06m 15s)
  • 02:22 bblack: pooled cp105[67],cp1069,cp1070 into eqiad misc caches
  • 02:19 logmsgbot: krinkle Finished scap: Rebuild l10n for Gadgets after I2089b21fc (duration: 24m 28s)
  • 01:54 logmsgbot: krinkle Started scap: Rebuild l10n for Gadgets after I2089b21fc
  • 01:08 logmsgbot: ori Synchronized php-1.26wmf17/extensions/Gadgets: I2089b21fc: Updated mediawiki/core Project: mediawiki/extensions/Gadgets (duration: 00m 13s)
  • 00:59 logmsgbot: ori Synchronized php-1.26wmf17/includes/OutputPage.php: c0ca5700c6: resourceloader: Restore anticipated loader states for hardcoded module requests (duration: 00m 12s)

2015-08-06

  • 23:24 logmsgbot: ori Synchronized php-1.26wmf17: c5c52ec1d8: resourceloader: Async all the way (duration: 01m 41s)
  • 23:20 logmsgbot: ebernhardson Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 12s)
  • 23:16 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: I053a6e9: Enable authmetrics logging on group0 wikis (duration: 00m 12s)
  • 23:15 logmsgbot: ebernhardson Synchronized wmf-config/: Redeploy cirrussearch ab test start (duration: 00m 14s)
  • 23:09 logmsgbot: ebernhardson Synchronized wmf-config/: Start cirrussearch suggester confidence AB test (duration: 00m 13s)
  • 23:07 mutante: puppet/salt-master: signing certs and adding keys for fermium
  • 23:05 logmsgbot: ebernhardson Synchronized php-1.26wmf17/extensions/CirrusSearch: Bump cirrusearch in 1.26wmf17 for SWAT (duration: 00m 11s)
  • 22:44 logmsgbot: ori Synchronized php-1.26wmf17/tests/phpunit/includes/OutputPageTest.php: (no message) (duration: 00m 13s)
  • 22:32 mutante: starting new instance fermium on ganeti
  • 22:31 ori: Previous two syncs were of I2089b21fc and I3f46fee7c
  • 22:31 logmsgbot: ori Synchronized php-1.26wmf17/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: (no message) (duration: 00m 12s)
  • 22:23 logmsgbot: ori Synchronized php-1.26wmf17/resources/src/startup.js: (no message) (duration: 00m 12s)
  • 22:22 logmsgbot: ori Synchronized php-1.26wmf17/includes/resourceloader/ResourceLoader.php: (no message) (duration: 00m 11s)
  • 22:21 logmsgbot: ori Synchronized php-1.26wmf17/extensions/Flow: 94703bc291: Updated mediawiki/core Project: mediawiki/extensions/Flow (duration: 00m 15s)
  • 22:02 mutante: if up for watching the (auto)-upgrade and restarting: @carbon:/srv/wikimedia/incoming# reprepro -C main include adminbot_1.7.12_amd64.changes
  • 22:00 mutante: built adminbot 1.7.12 and copied to carbon to incoming - but not imported
  • 21:56 logmsgbot: ori Synchronized php-1.26wmf17/extensions/Graph: I2089b21fc: Updated mediawiki/core Project: mediawiki/extensions/Graph (duration: 00m 12s)
  • 21:55 hoo: Running "updateSpecialPages.php --wiki wikidatawiki --only DoubleRedirects" on terbium
  • 21:46 ejegg: updated payments from 5bc32b7d0969878e441394c828620d5a44683c18 to af16d371f9c46d4f0b78986080f2a2be3226ace8
  • 21:25 logmsgbot: krinkle Synchronized php-1.26wmf17/extensions/EducationProgram/EducationProgram.hooks.php: T107980 (duration: 00m 12s)
  • 21:12 gwicke: switched cassandra staging cluster (xenon, cerium, praseodymium) to CMS & started a load test on that
  • 21:07 logmsgbot: ebernhardson Synchronized php-1.26wmf17/extensions/CirrusSearch/includes/Hooks.php: Repush file spewing notices into hhvm.log (duration: 00m 12s)
  • 20:26 chasemp: es-tool restart-fast on elastic1031 to test alerting issues
  • 20:06 logmsgbot: ori Synchronized php-1.26wmf17/extensions/Flow: I2089b21fc: Updated mediawiki/core Project: mediawiki/extensions/Flow (duration: 00m 15s)
  • 19:42 logmsgbot: twentyafterfour Synchronized php-1.26wmf17: actually deploy the hotfix this time (duration: 01m 33s)
  • 19:38 urandom: issuing nodetool cleanup on restbase1003
  • 19:36 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: all wikis to 1.26wmf17
  • 19:35 logmsgbot: twentyafterfour Synchronized php-1.26wmf17: sync hotfixes before deploying 1.26wmf17 to group2 (duration: 02m 18s)
  • 18:57 ejegg: updated payments from bbec5799db42f6f5302920a1a69123de7e4986df to 5bc32b7d0969878e441394c828620d5a44683c18
  • 18:55 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: revert 1.26wmf17
  • 18:47 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: all wikis to 1.26wmf17
  • 18:11 ejegg|mtg: updated payments from a8c0ecbedef6179c78ed833da9f2049cb0f2641b to bbec5799db42f6f5302920a1a69123de7e4986df
  • 16:59 logmsgbot: krinkle Synchronized php-1.26wmf17/resources/src/startup.js: touch (duration: 00m 11s)
  • 16:56 logmsgbot: krinkle Synchronized php-1.26wmf17/resources/Resources.php: T108191 unbreak mobile js (duration: 00m 11s)
  • 16:24 chasemp: upgrading elastic1031 to 1.7.1
  • 15:08 logmsgbot: ebernhardson Synchronized php-1.26wmf17/extensions/CirrusSearch/: bump cirrussearch in 1.26wmf17 for swat (duration: 00m 14s)
  • 14:50 dcausse: es1.7.1: restart elastic1030
  • 14:19 urandom: beginning nodetool cleanup on restbase1001
  • 14:14 moritzm: restarted HHVM on appservers in codfw for libtidy/PCRE security updates
  • 14:06 dcausse: es1.7.1: restart elastic1029
  • 13:27 dcausse: es1.7.1: restart elastic1028
  • 12:59 dcausse: es1.7.1: restart elastic1027
  • 12:55 godog: stop syslog-ng on lithium before switching to rsyslog
  • 11:55 dcausse: es1.7.1: restart elastic1026
  • 10:33 dcausse: es1.7.1: restart elastic1025
  • 09:50 moritzm: uploaded openjdk-8_8u66-b01-1~bpo8+1 to jessie-wikimedia and jessie-backports/debian.org
  • 09:39 jynus: Applying schema change to Commons db master
  • 09:39 moritzm: restarted HHVM on API apaches in eqiad for libtiny/PCRE security updates
  • 09:30 dcausse: es1.7.1: restart elastic1024
  • 09:18 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Repool db1068 (duration: 00m 13s)
  • 08:20 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Aug 6 08:20:24 UTC 2015 (duration 20m 23s)
  • 07:34 dcausse: es1.7.1: restart elastic1023
  • 07:07 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Repool db1056, Depool db1068 (duration: 00m 12s)
  • 06:52 moritzm: restart HHVM on canary API servers (mw1114-mw1119) for libtiny/PCRE security updates
  • 06:16 dcausse: es1.7.1: restart elastic1022
  • 05:57 logmsgbot: krinkle Synchronized php-1.26wmf16/includes/resourceloader/ResourceLoaderModule.php: Ib4371255fe (duration: 00m 12s)
  • 05:55 logmsgbot: krinkle Synchronized php-1.26wmf17/includes/resourceloader/ResourceLoaderModule.php: Ib4371255fe (duration: 00m 13s)
  • 05:39 logmsgbot: krinkle Synchronized php-1.26wmf17/resources/src/mediawiki/mediawiki.js: touch (duration: 00m 12s)
  • 05:32 logmsgbot: krinkle Synchronized php-1.26wmf17/resources/src/mediawiki/mediawiki.js: touch (duration: 00m 13s)
  • 05:02 logmsgbot: krinkle Synchronized php-1.26wmf17/includes/OutputPage.php: I885c36398 (duration: 00m 12s)
  • 04:48 ebernhardson: es1.7.1 upgrade on elastic1021
  • 04:42 logmsgbot: krinkle Synchronized php-1.26wmf17/resources/src/mediawiki/mediawiki.js: I885c36398 (duration: 00m 12s)
  • 04:04 logmsgbot: krinkle Synchronized php-1.26wmf17/resources/src/mediawiki.legacy/wikibits.js: T108139 (duration: 00m 12s)
  • 03:58 logmsgbot: krinkle Synchronized php-1.26wmf17/includes: T108124 (duration: 00m 17s)
  • 03:57 logmsgbot: krinkle Synchronized php-1.26wmf17/resources/src/mediawiki/mediawiki.js: T108124 (duration: 00m 12s)
  • 03:25 ebernhardson: es1.7.1 upgrade on elastic1020
  • 03:18 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf17) at 2015-08-06 03:18:27+00:00
  • 03:12 logmsgbot: l10nupdate Synchronized php-1.26wmf17/cache/l10n: l10nupdate for 1.26wmf17 (duration: 10m 32s)
  • 02:44 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf16) at 2015-08-06 02:44:39+00:00
  • 02:38 logmsgbot: l10nupdate Synchronized php-1.26wmf16/cache/l10n: l10nupdate for 1.26wmf16 (duration: 10m 42s)
  • 02:16 ebernhardson: es1.7.1 upgrade on elastic1019
  • 01:34 ebernhardson: es1.7.1 upgrade on elastic1018
  • 00:49 Jamesofur: reset password for User:Tonval after identify verification
  • 00:42 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 12s)
  • 00:34 twentyafterfour: phabricator upgrade complete
  • 00:33 ebernhardson: es1.7.1 upgrade on elastic1017
  • 00:31 RoanKattouw: <twentyafterfour> ok I'm gonna take phabricator down for upgrade
  • 00:04 gwicke: restarted restbase old-render clean-up scripts on wikipedia html and data-parsoid

2015-08-05

  • 23:56 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Unset $wgDiff (duration: 00m 12s)
  • 23:37 logmsgbot: ori Synchronized php-1.26wmf17/extensions/FlaggedRevs: I2089b21fc (duration: 00m 13s)
  • 23:32 logmsgbot: bd808 Synchronized php-1.26wmf17/extensions/VisualEditor/extension.json: VisualEditor b/c anon IP module name fix (Ia92ecc0) (duration: 00m 12s)
  • 23:09 logmsgbot: bd808 Synchronized wmf-config/CommonSettings.php: beta: Configure and (I7d20abb) (duration: 00m 13s)
  • 23:01 logmsgbot: ori Synchronized php-1.26wmf17/extensions/EducationProgram: I2089b21fc (duration: 00m 13s)
  • 23:00 ebernhardson: es1.7.1 upgrade on elastic1016
  • 22:47 logmsgbot: krinkle Synchronized php-1.26wmf17/includes/resourceloader/ResourceLoaderModule.php: T104950 (duration: 00m 12s)
  • 22:47 logmsgbot: krinkle Synchronized php-1.26wmf16/includes/resourceloader/ResourceLoaderModule.php: T104950 (duration: 00m 13s)
  • 22:29 hoo: Started dumpwikidatajson.sh on snapshot1003 again to create a Wikidata json dump after earlier attempts this week and today failed.
  • 22:27 logmsgbot: hoo Synchronized php-1.26wmf17/extensions/Wikidata/: Update Wikibase: Fix use class in CallbackFactory (duration: 00m 21s)
  • 22:27 logmsgbot: hoo Synchronized php-1.26wmf16/extensions/Wikidata/: Update Wikibase: Fix use class in CallbackFactory (duration: 00m 20s)
  • 22:27 ebernhardson: es1.7.1 upgrade on elastic1015
  • 21:44 subbu: deployed cherry-picked ba49b80bdc3a156604eb3996830af0d5bc45c503 hotfix to the parsoid cluster to deal with crashers from deploy earlier today
  • 21:17 gwicke: finished deploy of restbase 9e177f3 (deploy 7006f9f) on restbase cluster
  • 21:12 hoo: Started dumpwikidatajson.sh on snapshoot1003 to create a Wikidata json dump after earlier attempts this week failed.
  • 21:05 ebernhardson: es1.7.1 upgrade for es1014
  • 20:59 gwicke: restbase 9e177f3 (deploy 7006f9f) canary deploy on restbase1001
  • 20:56 logmsgbot: hoo Synchronized php-1.26wmf17/extensions/Wikidata/: Update Wikibase: Fix the dumpJson and the rebuildItemsPerSite maintenance scripts (duration: 00m 20s)
  • 20:55 logmsgbot: hoo Synchronized php-1.26wmf16/extensions/Wikidata/: Update Wikibase: Fix the dumpJson and the rebuildItemsPerSite maintenance scripts (duration: 00m 20s)
  • 20:25 subbu: deployed parsoid version d5a5722c
  • 20:22 logmsgbot: krinkle Synchronized php-1.26wmf16/includes/resourceloader/ResourceLoaderFileModule.php: T104950 (duration: 00m 12s)
  • 20:21 logmsgbot: krinkle Synchronized php-1.26wmf16/includes/resourceloader/ResourceLoader.php: T104950 (duration: 00m 11s)
  • 20:13 logmsgbot: krinkle Synchronized php-1.26wmf17/includes/resourceloader/ResourceLoaderFileModule.php: T104950 (duration: 00m 12s)
  • 20:12 logmsgbot: krinkle Synchronized php-1.26wmf17/includes/resourceloader/ResourceLoader.php: T104950 (duration: 00m 13s)
  • 20:07 logmsgbot: ori Synchronized php-1.26wmf17/extensions/PageTriage: I2089b21fc: Updated mediawiki/core Project: mediawiki/extensions/PageTriage 22eddf4ad5bf6b3fe7c49af5812ce5fcfa5e1911 (duration: 00m 14s)
  • 19:55 gwicke: re-enabled puppet on restbase staging cluster in preparation for deploy
  • 19:52 gwicke: disabled puppet on restbase hosts in preparation for the deploy
  • 19:36 dcausse: es1.7.1: resume writes to indices
  • 19:31 dcausse: es1.7.1: restart elastic1013
  • 19:19 bblack: all caches depooled for thermal stuff repooled
  • 18:54 bblack: depooled cp1060, cp1064 ( thermal batch 3: https://phabricator.wikimedia.org/T103226 )
  • 18:37 dcausse: es1.7.1: restart elastic1012
  • 18:34 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf17
  • 18:07 bblack: depooled cp1059, cp1062, cp1067 ( thermal batch 2: https://phabricator.wikimedia.org/T103226 )
  • 18:02 moritzm: restarted HHVM on appservers (mw1136-mw1158) for tidy/pcre security updates
  • 17:56 dcausse: es1.7.1: restart elastic1011
  • 17:48 dcausse: es1.7.1: freeze indices (take 2)
  • 17:36 logmsgbot: bblack Synchronized wmf-config/squid-labs.php: (no message) (duration: 00m 12s)
  • 17:15 moritzm: restarted HHVM on appservers (mw1149-mw1151, mw1161-1188, mw1209-1220) for tidy/pcre security updates
  • 17:09 logmsgbot: hoo Finished scap: Rebuild l10n cache for wmf17, got forgotten during the train (duration: 26m 02s)
  • 17:07 bblack: really depooled cp1046, cp1061, cp1066 ( thermal batch 1: https://phabricator.wikimedia.org/T103226 )
  • 17:02 bblack: depooled cp1046, cp1061, cp1066 ( thermal batch 1: https://phabricator.wikimedia.org/T103226 )
  • 16:43 logmsgbot: hoo Started scap: Rebuild l10n cache for wmf17, got forgotten during the train
  • 16:28 bblack: cache puppets disabled for a little while, to make sure do_esi doesn't melt things
  • 15:11 logmsgbot: thcipriani Synchronized php-1.26wmf17/extensions/ContentTranslation/modules/tools/ext.cx.tools.mt.js: SWAT: FIX: Not able to set cursor in previous sections gerrit:229328 (duration: 00m 12s)
  • 15:02 andrewbogott: rebooting labvirt1009
  • 14:51 gwicke: stopped restbase on restbase1009
  • 14:44 moritzm: restarted HHVM on appservers (mw1026-mw1113) for tidy/pcre security updates
  • 14:42 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool db1056 (duration: 00m 12s)
  • 14:29 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool db1059 (duration: 00m 13s)
  • 13:16 hoo: Removed Wikidata JSON dumps from Monday and Tuesday as they were incomplete/ had the wrong serialization format
  • 12:41 moritzm: restarted HHVM on canary appservers for tidy/pcre security updates, remaining app servers following soon
  • 12:32 paravoid: upgrading asw-c-codfw and asw-d-codfw to newer junos
  • 11:17 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool db1056, depool db1059 (duration: 00m 12s)
  • 11:01 godog: depool restbase1009, investigating healthcheck returning 500s
  • 10:52 godog: pool restbase100[789] in pybal
  • 10:43 paravoid: upgrading asw-b-codfw to newer junos
  • 10:36 jynus: applying schema change for s4 on codfw, some lag expected
  • 09:08 dcausse: es1.7.1: upgrade elastic1010
  • 07:46 dcausse: es1.7.1: upgrade elastic1009
  • 07:12 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool db1056 for maintenance, db1064 set to 100% (duration: 00m 12s)
  • 06:29 springle: finish OSC gerrit 228756 s5 wb_items_per_site.ips_site_page
  • 06:27 logmsgbot: @tin ResourceLoader cache refresh completed at Wed Aug 5 06:27:08 UTC 2015 (duration 27m 7s)
  • 06:26 dcausse: es1.7.1: upgrade elastic1008
  • 04:56 ebernhardson: restarted elasticsearch on elastic1007 for 1.7.1 upgrade
  • 03:34 logmsgbot: legoktm Synchronized wmf-config/InitialiseSettings.php: Disable two more wikis due to namespace conflicts - https://gerrit.wikimedia.org/r/229292 (duration: 00m 12s)
  • 03:09 ebernhardson: restarted elasticsearch on elastic1006 for 1.7.1 upgrade
  • 03:04 logmsgbot: @tin LocalisationUpdate completed (1.26wmf17) at 2015-08-05 03:04:08+00:00
  • 02:57 logmsgbot: l10nupdate Synchronized php-1.26wmf17/cache/l10n: (no message) (duration: 10m 30s)
  • 02:31 logmsgbot: @tin LocalisationUpdate completed (1.26wmf16) at 2015-08-05 02:31:44+00:00
  • 02:28 logmsgbot: l10nupdate Synchronized php-1.26wmf16/cache/l10n: (no message) (duration: 06m 56s)
  • 01:44 ebernhardson: restarting elasticsearch of es1005

2015-08-04

  • 23:59 logmsgbot: maxsem Synchronized php-1.26wmf16/extensions/WikimediaEvents/: SWAT (duration: 00m 12s)
  • 23:57 logmsgbot: maxsem Synchronized php-1.26wmf17/extensions/WikimediaEvents/: SWAT (duration: 00m 12s)
  • 23:08 logmsgbot: mattflaschen Synchronized wmf-config/InitialiseSettings.php: Disable Flow on betawikiversity (duration: 00m 13s)
  • 22:07 logmsgbot: twentyafterfour Synchronized php-1.26wmf17: forgot submodule update (duration: 01m 39s)
  • 20:46 logmsgbot: twentyafterfour Finished scap: fixup wikidata submodule version (duration: 23m 26s)
  • 20:22 logmsgbot: twentyafterfour Started scap: fixup wikidata submodule version
  • 19:46 dcausse: es1.7.1: upgrade elastic1003
  • 19:12 ori: Applied Icba6d7a87 on mw1017 for a couple of webpagetest runs
  • 19:08 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.26wmf17
  • 18:51 logmsgbot: twentyafterfour Finished scap: rebuild localization cache, sync 1.26wmf17 (duration: 28m 39s)
  • 18:42 dcausse: es1.7.1: upgrade elastic1002
  • 18:22 logmsgbot: twentyafterfour Started scap: rebuild localization cache, sync 1.26wmf17
  • 18:00 andrewbogott: re-imaging labnodepool1001
  • 17:35 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Increase db1064 traffic (duration: 00m 13s)
  • 17:18 dcausse: es1.7.1: upgrade elastic1001
  • 17:17 hoo: Started dumpwikidatajson.sh on snapshot1003 to create a correct Wikidata json dump
  • 17:14 logmsgbot: hoo Synchronized php-1.26wmf16/extensions/Wikidata/: Fix maintenance/dumpJson.php fatal (duration: 00m 21s)
  • 17:11 chasemp: freezing elasticsearch indexes for 1.7.1
  • 16:23 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Repool db1064 with low traffic after maintenance (duration: 00m 12s)
  • 15:34 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable Flow on ptwikibooks gerrit:229133 (duration: 03m 40s)
  • 15:28 jynus: restarting db1064 for regular maintenance and upgrade given that it was depooled in the first place for a schema change
  • 15:24 logmsgbot: thcipriani Synchronized wmf-config: SWAT: Add configuration for authmetrics logging (part II) gerrit:227630 (duration: 02m 41s)
  • 15:21 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Add configuration for authmetrics logging (part I) gerrit:227630 (duration: 03m 11s)
  • 15:13 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor for 10% of new accounts on enwiki gerrit:227329 (duration: 03m 13s)
  • 14:36 paravoid: cr2-codfw upgrading SCBs
  • 14:23 paravoid: upgrading junos on asw-a-codfw again
  • 13:45 _joe_: repooling mw1159,mw1160
  • 13:21 paravoid: rebooting asw-a-codfw, member 2
  • 13:04 Coren: labstore1001 rebooting (possibly a couple of times) during tests and reinstallation
  • 12:55 hoo: Syncing to mw1160 failed (Host key verification failed.)
  • 12:50 logmsgbot: hoo Synchronized php-1.26wmf16/extensions/Wikidata/: Update Wikibase: Fixes for JSON dump creation (duration: 00m 39s)
  • 12:06 moritzm: updated canary appservers mw1017/mw1018 to updated pcre3 + hhvm restart
  • 12:03 moritzm: added pcre3_8.31-2ubuntu2.1+wm1 to trusty-wikimedi (reroll of security update with our JIT enablement patch)
  • 11:48 _joe_: killed ircecho to prevent furter icinga spam
  • 11:44 jynus: schema update on Commons failed, expect some minor inestabilities until everything is fixed
  • 11:41 _joe_: reimaging mw1159 to HAT
  • 11:01 paravoid: upgrading junos on asw-a-codfw
  • 10:57 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Depool db1064 (duration: 00m 13s)
  • 10:27 godog: bootstrap cassandra on restbase1009
  • 10:21 akosiaris: enabling puppet on tin
  • 09:30 jynus: rolling schema change on image table to all wikis
  • 08:07 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Increasing load for db1027 and db1015 (duration: 00m 12s)
  • 07:38 logmsgbot: @tin ResourceLoader cache refresh completed at Tue Aug 4 07:38:01 UTC 2015 (duration 38m 0s)
  • 06:14 _joe_: depooled mw1061
  • 06:14 logmsgbot: legoktm Synchronized wmf-config/InitialiseSettings.php: Disable Flow on Japanese Wikiversity (duration: 00m 13s)
  • 06:09 logmsgbot: legoktm Synchronized wmf-config/InitialiseSettings.php: Disable Flow on English Wikiversity (duration: 00m 12s)
  • 06:07 legoktm: sync to mw1061 failed
  • 06:07 logmsgbot: legoktm Synchronized wmf-config/InitialiseSettings.php: Disable Flow on English Wikiversity (duration: 00m 12s)
  • 02:32 logmsgbot: @tin LocalisationUpdate completed (1.26wmf16) at 2015-08-04 02:32:18+00:00
  • 02:28 logmsgbot: l10nupdate Synchronized php-1.26wmf16/cache/l10n: (no message) (duration: 09m 16s)
  • 02:18 logmsgbot: twentyafterfour Finished scap: sync https://gerrit.wikimedia.org/r/#/c/229036/1 (duration: 25m 41s)
  • 01:52 logmsgbot: twentyafterfour Started scap: sync https://gerrit.wikimedia.org/r/#/c/229036/1
  • 00:02 awight: updated paymentswiki to a8c0ecbedef6179c78ed833da9f2049cb0f2641b

2015-08-03

  • 23:56 awight: updating paymentswiki to b20559f75e0fc0d863efe027d76b78462555767c
  • 23:45 ottomata: rebuilding kafka cluster
  • 23:21 logmsgbot: ebernhardson Synchronized php-1.26wmf16/extensions/VisualEditor/: Bump visualeditor for swat in 1.26wmf16 (duration: 00m 13s)
  • 23:18 logmsgbot: ebernhardson Synchronized php-1.26wmf16/extensions/WikimediaEvents/: Bump WikimediaEvents in SWAT for 1.26wmf16 (duration: 00m 12s)
  • 23:17 logmsgbot: ebernhardson Synchronized php-1.26wmf16/extensions/Flow: Bump flow submodule in swat for 1.26wmf16 (duration: 00m 14s)
  • 23:05 logmsgbot: ebernhardson Synchronized wmf-config/: (no message) (duration: 00m 13s)
  • 22:46 awight: reverting paymentswiki, to 6dbbb4c784349ace5a0ac616c61ec0c3fffa0eff
  • 22:33 ejegg: updated crm from db417a28a247a3fdf3e3023a700d6266e04f3e9d to 4f40ac6de0385982d8e672b1ed30ff1a2a2a2aa1
  • 22:27 awight: deployed debug hack to payments1004
  • 21:43 awight: deploy paymentswiki-staging configuration: add explicit queue name for payments4 connecting to payments1-3
  • 21:32 awight: deploy paymentswiki-staging configuration
  • 21:25 awight: updating payments1004 to 1daf9d0fe773c022a2ab8de5542fc15ddc261e75
  • 21:04 logmsgbot: bd808 Synchronized wmf-config/logging.php: Remove code duplication from monolog config (Ia960203) (duration: 00m 11s)
  • 20:51 awight: updating paymentswiki from d4bdce1cae168448b116d75e3dcd3303b0f13dd2 to d56dad49ef0da0a8b9c7da410bcac12e48724ae5
  • 20:26 arlolra: updated Parsoid to version 38d0cdb13734a40bc2908e779e1a0cde158048f2
  • 19:49 logmsgbot: aude Synchronized php-1.26wmf16/extensions/Wikidata: Fix T104609 and fix/debug T107711 (duration: 00m 19s)
  • 19:21 logmsgbot: aude Synchronized usagetracking.dblist: Enable usage tracking on enwiki (duration: 00m 12s)
  • 19:20 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Add debug log group for T107711 (duration: 00m 12s)
  • 19:07 ottomata: stopped a couple of kafka brokers. acknowldeging..
  • 19:02 bblack: https://gerrit.wikimedia.org/r/228882 reversion salted + nginx reloaded
  • 18:28 gwicke: switched restbase1002 and restbase1003 to iojs as well
  • 17:36 logmsgbot: aude Synchronized usagetracking.dblist: Enable usage tracking on zhwiki (duration: 00m 12s)
  • 17:21 logmsgbot: legoktm Synchronized php-1.26wmf16/includes/Revision.php: https://gerrit.wikimedia.org/r/228853 (duration: 00m 12s)
  • 17:21 ottomata: starting kafka partition reassignment to balance all partiions over to 3 new kafka brokers and off of analytics1021
  • 17:21 gwicke: switching from node 0.10 to iojs 2.5 on restbase1001 after load testing on xenon went well
  • 17:02 logmsgbot: legoktm Synchronized wmf-config/logging.php: logging: Enable stacktrace printing (duration: 00m 12s)
  • 17:00 hoo: Started dumpwikidatajson.sh on snapshot1003 to re-create today's dump
  • 16:55 logmsgbot: legoktm Synchronized php-1.26wmf16/autoload.php: https://gerrit.wikimedia.org/r/#/c/228850/ (duration: 00m 12s)
  • 16:54 logmsgbot: legoktm Synchronized php-1.26wmf16/includes/debug/logger/: https://gerrit.wikimedia.org/r/#/c/228850/ (duration: 00m 11s)
  • 16:49 hoo: Removed today's Wikidata json dump (wikidata-20150803-all.json.gz) because it was incomplete due to the dataset problems earlier
  • 16:27 paravoid: upgrading junos on cr2-codfw
  • 15:34 bblack: wiping cp3034 disk cache (upload esams) for ipsec reload testing
  • 15:23 logmsgbot: thcipriani Synchronized php-1.26wmf16/extensions/MultimediaViewer: SWAT: Track image load time with statsv (touch and re-sync) gerrit:228218 (duration: 00m 12s)
  • 15:22 ottomata: reinstalling analytics1013,1014 and 1020 with Jessie
  • 15:10 logmsgbot: thcipriani Synchronized php-1.26wmf16/extensions/MultimediaViewer: SWAT: Track image load time with statsv gerrit:228218 (duration: 00m 12s)
  • 14:59 logmsgbot: aude Synchronized usagetracking.dblist: Enable usage tracking on trwiki (duration: 00m 12s)
  • 14:54 logmsgbot: krenair Synchronized php-1.26wmf16/extensions/SemanticResultFormats: https://gerrit.wikimedia.org/r/#/c/228793/ (duration: 00m 13s)
  • 14:42 logmsgbot: aude Synchronized usagetracking.dblist: Enable usage tracking on thwiki (duration: 00m 12s)
  • 14:33 mutante: temp. stop puppet on dataset1001
  • 14:27 paravoid: upgrading junos on cr1-codfw
  • 14:23 moritzm: updated iojs on apt.wikimedia.org to 2.5.0 for jessie-wikimedia
  • 14:21 ottomata: upgrading kernel on analytics1042-1049 from 3.13.0.24.28 to 3.13.0.61.68 because T107698
  • 14:18 logmsgbot: aude Synchronized usagetracking.dblist: Enable usage tracking on svwiki (duration: 00m 12s)
  • 13:50 bblack: re-enabling puppet + ircecho on neon (vast majority of recovery spam is over with)
  • 13:17 bblack: re-enable agent, restarted apache2 on palladium, strontium, rhodium (fact_values truncated in mysql)
  • 13:10 bblack: rhodium too (puppetmaster stop)
  • 13:05 bblack: stopped puppet-agent + apache2 on strontium + palladium (no masters alive, for mysql maintenance)
  • 12:59 bblack: stopped ircecho + puppet-agent on neon (spam from epic puppetmaster fail)
  • 12:52 bblack: stop->wait->restart of apache2 service on palladium (seemed dead to puppet reqs)
  • 12:21 _joe_: bumped ganglia-monitor-aggregator on bast4001, the upstart script needs immediate fixing
  • 11:01 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: avoid db1044 SPOF by repooling db1027 and db1015 (duration: 00m 12s)
  • 10:56 paravoid: switching GeoDNS to GeoIP2
  • 10:45 paravoid: upgrading all AuthDNS servers to gdnsd 2.2.0
  • 09:31 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool db1035 for maintenance (duration: 00m 12s)
  • 05:22 logmsgbot: @tin ResourceLoader cache refresh completed at Mon Aug 3 05:22:15 UTC 2015 (duration 22m 14s)
  • 02:23 logmsgbot: @tin LocalisationUpdate completed (1.26wmf16) at 2015-08-03 02:23:21+00:00
  • 02:20 logmsgbot: l10nupdate Synchronized php-1.26wmf16/cache/l10n: (no message) (duration: 06m 21s)
  • 01:47 springle: starting OSC gerrit 228756 s5 wb_items_per_site.ips_site_page
  • 00:03 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/228198/ (duration: 00m 12s)

2015-08-02

  • 17:52 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: If7fcb6e6: Default wikipedias to enwiki.png (duration: 00m 12s)
  • 13:26 jynus: powercycling analytics1044: same kernel fatal issues as 1043
  • 13:10 jynus: powercycling analytics1043: kernel issues
  • 12:05 bblack: started pybal on lvs3001
  • 04:56 logmsgbot: @tin ResourceLoader cache refresh completed at Sun Aug 2 04:56:29 UTC 2015 (duration 56m 28s)
  • 02:23 logmsgbot: @tin LocalisationUpdate completed (1.26wmf16) at 2015-08-02 02:23:09+00:00
  • 02:20 logmsgbot: l10nupdate Synchronized php-1.26wmf16/cache/l10n: (no message) (duration: 06m 11s)

2015-08-01

  • 06:04 _joe_: removing some old apache access logs from mw1114
  • 05:06 logmsgbot: @tin ResourceLoader cache refresh completed at Sat Aug 1 05:06:46 UTC 2015 (duration 6m 45s)
  • 03:53 andrewbogott: cleared out nova-conductor.log on labcontrol1001, restarted nova-conductor, graceful’d apache
  • 02:23 logmsgbot: @tin LocalisationUpdate completed (1.26wmf16) at 2015-08-01 02:23:15+00:00
  • 02:20 logmsgbot: l10nupdate Synchronized php-1.26wmf16/cache/l10n: (no message) (duration: 06m 11s)
  • 00:12 logmsgbot: ori Synchronized extract2.php: Ie919881a4: Add an API listing template to the allowed templates in extract2.php
  • 00:01 logmsgbot: ori Synchronized php-1.26wmf16/includes: Revert I4afaecd8: "Avoiding writing sessions for no reason", and undo several uncommitted live-hacks for debugging T102199 (duration: 00m 16s)

Archives