Server Admin Log/Archive 28

2015-12-31

02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Dec 31 02:31:43 UTC 2015 (duration 6m 52s)
02:24 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 10m 25s)

2015-12-30

15:05 andrewbogott: restarting puppet on seaborgium; openldap will restart but config should not change
14:57 andrewbogott: restarting puppet on serpens; openldap will restart but config should not change
13:14 mark: labstore1001: apt-get install irqbalance
08:01 jynus: setting dbstore1001 to read_only, converting ruwiki.recentchanges back to InnoDB
05:27 ejegg|away: set drupal variable wmf_common_requeue_max back to 10
02:32 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Dec 30 02:32:15 UTC 2015 (duration 6m 55s)
02:26 gwicke: restbase: finished full deploy of 7db8e216 (small bug fix & a security fix) to production cluster
02:25 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 09m 50s)
02:17 gwicke: restbase: starting full deploy of 7db8e216 (small bug fix & a security fix) to production cluster
02:14 gwicke: restbase: canary deploy of 7db8e216 (small bug fix & a security fix) to restbase1001
00:08 andrewbogott: restarting nova-compute on labvirt1002

2015-12-29

23:56 andrewbogott: restarting nodepool on labnodepool1001
23:00 ejegg: re-enabled fundraising banner campaigns
22:50 ejegg: restarted fundraising scheduled jobs
22:14 ejegg: shut down campaigns and scheduled jobs due to dead ActiveMQ
22:13 andrewbogott: restarting slapd on seaborgium and serpens
22:11 andrewbogott: disabling puppet on seaborgium and serpens
16:47 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Turn off AB test for search lang detect via accept-language (duration: 00m 29s)
16:04 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable cross-wiki upload A/B test gerrit:261371 (duration: 00m 31s)
15:06 apergos: labs salt instances salt update in progress. It's slow and tedious and automated. A few hundred instances already done, the rest are going one at a time. Only instances that use the labcontrol salt master will be affected.
13:29 apergos: salt wm2 packages now installed on all production hosts except for: mw1041.eqiad.wmnet, technetium.eqiad.wmnet, mw1228.eqiad.wmnet, ms-be1011.eqiad.wmnet
13:08 apergos: labcontrol*, neodymium and palladium updated to latest salt packages (wm2), rest of prod to follow
12:24 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: Attempt to fix math related fatal (duration: 00m 33s)
09:54 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Setting weight values for s6 to original production values (duration: 00m 35s)
09:37 jynus: changing the mysql master of db2028, from db1030 to db1050
02:41 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Dec 29 02:41:49 UTC 2015 (duration 6m 38s)
02:35 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 15m 38s)
01:02 logmsgbot: awight@tin Synchronized php-1.27.0-wmf.9/extensions/CentralNotice: Update CentralNotice: T122251 (duration: 00m 34s)

2015-12-28

21:45 gwicke: restbase: rolling restart to apply https://gerrit.wikimedia.org/r/261206
21:26 mutante: tin & mira: started salt minions that were in status stop/waiting
21:25 logmsgbot: aaron@tin Synchronized private/PrivateSettings.php: (no message) (duration: 00m 30s)
21:20 logmsgbot: aaron@tin Synchronized wmf-config/PrivateSettings.php: $wmfSwiftConfig convenience variable (duration: 00m 30s)
20:59 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1050 & db1022 after emergency fix (duration: 00m 31s)
20:52 mutante: cygnus - starting salt-minion
20:35 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.9/extensions/Graph/modules/graph2.js: https://gerrit.wikimedia.org/r/#/c/261200/ (duration: 00m 31s)
19:22 ejegg: updated DjangoBannerStats from 8d4a9062aab80e5371faebadd72fbe4f19ac2fdd to a64fe0e373a978d3df0b7f1dd74ac4cc5c78d34e
18:46 jynus: importing wikishared from x1-master into analytics-slave and setting up replication
18:28 jynus: restarting and upgrading db1050, using the fact that it is depooled
17:16 paravoid: disabled varnish TBF and force-ran puppet on all cp* hosts (I12ea52165e125aaf4ed779399f34cff16d5cd140)
16:38 jynus: applying production-side replication filters for wikimania2017wiki on labs
16:24 logmsgbot: krenair@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 30s)
16:14 logmsgbot: krenair@tin Synchronized dblists: (no message) (duration: 00m 29s)
16:13 logmsgbot: krenair@tin rebuilt wikiversions.php and synchronized wikiversions files: (no message)
16:12 logmsgbot: krenair@tin Synchronized w/static/images/project-logos/wikimania2017wiki.png: (no message) (duration: 00m 31s)
16:12 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/260521/ (duration: 00m 30s)
14:55 jynus: cloning db1050's mysql data to db1022
14:15 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Emergency depool of db1050 (duration: 00m 31s)
02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Dec 28 02:30:47 UTC 2015 (duration 6m 58s)
02:23 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 09m 55s)

2015-12-27

03:05 YuviPanda: run nodetool clearsnapshot -- v3 and nodetool clearsnapshot -- v1 on maps-test2001
02:45 YuviPanda: run drop keyspace v3; on csql on maps-test1001 for yurik
02:42 YuviPanda: run drop keyspace v1; on csql on maps-test1001 for yurik
02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Dec 27 02:30:29 UTC 2015 (duration 6m 59s)
02:23 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 09m 35s)

2015-12-26

19:12 paravoid: restarting varnish-frontend on cp3042
19:06 jynus: setting db1030 as the new master of db2028
18:40 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Emergency depool of db1022 (duration: 00m 30s)
18:28 jynus: disabling lag notifications for codfw (s6)
18:01 paravoid: cp3048: cleaned up /run/vmod_tbf/tbf.db/, kept a backup copy under ~faidon
17:55 paravoid: cp3048: service varnish-frontend stop (sending 429 to lots of people, T122453)
05:38 paravoid: rolling restart of hhvm jobrunners (T122069)
02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Dec 26 02:31:39 UTC 2015 (duration 6m 54s)
02:24 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 10m 39s)

2015-12-25

15:50 jynus: testing new mariadb packages on db2070
13:19 jynus: setting db2018's binlog_format as MIXED
10:51 jynus: powercycle cp3010
09:17 jynus: powercycling cp4007 (unresponsive to ssh, ping, serial console)
02:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Dec 25 02:29:49 UTC 2015 (duration 6m 56s)
02:22 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 09m 47s)

2015-12-24

23:28 mutante: powercycled mw1114
23:27 mutante: i just reset dra on mw1114 because it said it was in use and i didnt see a log yet :;p
23:26 robh: mw1114 spammed all icinga errors, system is outputting endless scroll of login prompt, not halting for input (like anohter session or crash cart is sending it, or an error)
17:31 gwicke: aqs: tweaked table properties for local_group_default_T_pageviews_per_article_flat: 2 months max DTCS window size, deflate compression
17:20 jynus: restarting and reconfiguring mysql at db2066
16:45 jynus: restart and reconfigure mysql at db2059
16:11 jynus: restart and mysql reconfguration of db2052
15:02 jynus: restarting and reconfiguring mysql at db2045
14:11 paravoid: powercycling mw1012, OOM'ed/stuck
14:09 paravoid: rolling restart of hhvm jobrunners (T122069)
14:06 jynus: restart and reconfigure mysql at db2038
12:32 jynus: restart and reconfigure mysql at db2065
12:12 jynus: restart and reconfiguring mysql for db2058
11:50 jynus: restarting and reconfiguring mysql at db2051
11:28 jynus: restarting and reconfiguring mysql at db2044
10:33 jynus: restarting 's2' replication on dbstore200[12] after cloning
02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Dec 24 02:30:40 UTC 2015 (duration 6m 52s)
02:23 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 10m 03s)
01:05 awight: update payments from bae4d02afd8cfe1f8b8617c2f74bb36e420d281d to a7785baa7b40b442ecf0b60d47572502d0759780
00:38 gwicke: restbase1003: starting `nodetool cleanup`

2015-12-23

23:31 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.9/extensions/Graph/modules/ve-graph: https://gerrit.wikimedia.org/r/#/c/260868/ (duration: 00m 31s)
19:59 mutante: restbase1004 - puppet stopped and host key changed, what's up?
19:42 mutante: ran puppet on mw2112
19:41 mutante: logstash1002 - started logstash service
19:22 logmsgbot: aaron@tin Synchronized wmf-config/CommonSettings.php: Remove unused $wgMaxSquidPurgeTitles setting (duration: 00m 30s)
18:55 ejegg: updated fundraising tools from ebed29c0eccf38c812b20a957b3487a15bfa9cbc to 1bc23cb4bfaf2a9d4d215aad79dd67d891b5d973
18:51 ejegg: updated fundraising dashboard from 59e51c4ff74c3c584daf6c5de3bb66daa764cd28 to af8a493ab9ac5431e0d294e5019ac4e426ac6e08
18:38 jynus: restart and reconfigure mysql at db2037
18:17 mutante: bohrium - finish install, signing puppet certs
17:11 jynus: cloning s2 databases from dbstore2001 to dbstore2002 (s2 replication disabled on both)
14:05 jynus: restart and reconfigure mysql at db2064
13:26 jynus: restart and reconfigure mysql at db2063
12:41 jynus: reenabling event scheduler on db1046 (eventlogging m4-master)
12:30 jynus: restart and reconfigure mysql at db2056
12:14 godog: upgrade cassandra on aqs1003
11:24 jynus: reloading and reconfiguring mysql on db2049
11:11 godog: roll-upgrade cassandra to 2.1.12 on aqs100[123]
10:55 jynus: rebooting and reconfiguring mysql on db2041
10:11 jynus: restarting and reconfiguring mysql at db2035
09:52 gwicke: rebuilding restbase1004
09:51 gwicke: wiped & started boostrap on restbase1008
09:18 gwicke: nodetool removenode e2813bb9-f1f2-4d21-ac19-95a7a35b4513 in preparation for adding 1004 to the cluster without bootstrap
02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Dec 23 02:30:25 UTC 2015 (duration 7m 1s)
02:23 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 09m 18s)
00:40 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings-labs.php: https://gerrit.wikimedia.org/r/260696 & https://gerrit.wikimedia.org/r/260699 (duration: 05m 28s)
00:37 mutante: mw1133 - powercycle
00:36 legoktm: manually fixed up stuck global rename of "RCJU-ArCJ" -> "Archives cantonales jurassiennes"
00:31 matt_flaschen: Ran UPDATE flow_workflow SET workflow_page_id = 41854369 WHERE workflow_wiki = 'enwiki' AND workflow_namespace = 5 AND workflow_title_text = 'Flow/Developer_test_page' AND workflow_page_id = 48099373; to work around DB inconsistency (T117812)

2015-12-22

21:46 gwicke: restbase1004: tune2fs -m 0 /dev/mapper/restbase1004--vg-srv
21:45 gwicke: restbase1004: restarted bootstrap
21:22 gwicke: restbase1003: restarting cassandra to clear up disk space from old stream
21:11 gwicke: restbase1008: restarting cassandra to clear up disk space from old stream
18:36 robh: silver returned to normal service, wikitech.w.o certificate renewed.
18:26 robh: silver puppet staying stalled during toollabs issue (we dont want to rehup silver web serivce)
18:17 robh: puppet disabled on silver, going to update wikitech.wikimedia.org certificate
18:10 jynus: disabling event scheduling on db1046
18:03 jynus: rolling schema change (ALTER TABLE ENGINE=TokuDB) on m4-master (db1046) log (eventlogging)
16:44 godog: bounce cassandra on restbase1004, restart bootstrap
16:42 mutante: powercycling crashed mw1144
16:41 jynus: converting dbstore2001 (delayed slave) into an actual delayed slave, adding redundancy to dbstore1002
16:40 godog: bounce cassandra on restbase1003
16:15 akosiaris: upgrade cassandra on maps-test2001
16:15 akosiaris: upgrade cassandra on maps-test2002
15:53 mutante: kafka1001,1002 - crit - eventlogging not running (?)
15:52 mutante: restbase1003 - disk space, restbase1008 - disk space, restbase1004 - cassandra cql refused
15:23 akosiaris: upgrade cassandra on maps-test2003
15:06 jynus: restarting and reconfiguring mysql at dbstore2001
15:06 mutante: labtestcontrol2001 - puppet had not been running for a while, a bunch of changes have been applied incl. keys and passwords
15:04 mutante: enabling puppet on labtestcontrol2001
15:04 akosiaris: upgraded cassandra on maps-test2004
11:54 apergos: salt packages with wmf packages precise running on ms-{bf}e* in esams; trusty running on analytics103* in eqiad; jessie running on restbase2* in codfw
11:43 godog: restart cassandra bootstrap on restbase1004
10:09 jynus: online resizing /srv/postgres on labsdb1006 +100GB
10:06 hashar: Restarting Jenkins
09:54 apergos: precise and trusty salt packages with wmf patches deployed manually on dataset1001 and analytics1001, seem to work fine
08:42 jynus: restarting and reconfiguring mysql at db2036
02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Dec 22 02:30:28 UTC 2015 (duration 6m 54s)
02:23 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 09m 47s)
00:29 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.9/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/260492/ (duration: 00m 32s)
00:22 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.9/extensions/SyntaxHighlight_GeSHi/modules/ve-syntaxhighlight/ve.ui.MWSyntaxHighlightDialogTool.js: https://gerrit.wikimedia.org/r/#/c/260429/ (duration: 00m 30s)

2015-12-21

22:41 cwd: updated paymentswiki from a1be1ad134d06464e98de180227554fceddc91d4 to bae4d02afd8cfe1f8b8617c2f74bb36e420d281d
20:49 godog: restbase1004 bootstrap failed, restbase1007-a is down java.lang.RuntimeException: A node required to move the data consistently is down (/10.64.0.230).
19:27 legoktm: running checkLocalUser.php --delete=1 for real this time on terbium
19:22 godog: reimage restbase1004
19:14 paravoid: powercycling mw1011
19:11 paravoid: rolling restart of hhvm on the eqiad jobrunners
18:47 jynus: common-sync: Copying to mw1016.eqiad.wmnet from tin.eqiad.wmnet
18:35 ori: correction: previous log message was for mw1015, not mw1017
18:27 ori: mw1017: enabled jemalloc profiling, restarted hhvm, now running hhvm-collect-heaps
17:48 akosiaris: restarted hhvm on mw1012.eqiad.wmnet
16:57 thcipriani: timeout on sync-file to mw1016.eqiad.wmnet
16:56 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.9/extensions/Popups/Popups.hooks.php: SWAT: Use ExtensionRegistry to determine whether TextExtracts is installed gerrit:260346 (duration: 02m 48s)
16:34 jynus: sync-common to mw1085
16:26 jynus: powercycling mw1085.eqiad.wmnet
16:22 thcipriani: mw1085.eqiad.wmnet times out on SSH connection
16:19 godog: reboot restbase1007, load through the roof
16:18 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.9/extensions/CentralNotice/resources/subscribing/ext.centralNotice.geoIP.js: SWAT: Update CentralNotice gerrit:260316 (duration: 03m 03s)
16:08 godog: depool restbase1007
16:01 apergos: jessie packages for salt with local patches deployed on restbase1001, looks fine but just in case.
15:44 godog: adding new 1TB disk to restbase1007
14:22 andrewbogott: disabling puppet on labnet1002 for dnsmasq tests
14:07 MaxSem: me and yurik are nuking old maps data and reimporting planet
13:46 jynus: extending online s2-master data disk by +100GB
13:15 akosiaris: disabled puppet on maps-test2001 and commented out osmupdater crontab entry until we fix the sync process
11:02 jynus: emergency restart of db1047's mysql
09:54 jynus: reenabling semisync replication on s3
09:07 godog: stop cassandra on restbase1004, decomissioned
02:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Dec 21 02:29:51 UTC 2015 (duration 6m 47s)
02:23 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 09m 45s)
02:20 andrewbogott: disabling puppet on labnet1002 to mess with dnsmasq
01:44 andrewbogott: disabled puppet on holmium and labservices1001 to control roll-out of https://gerrit.wikimedia.org/r/#/c/260037/

2015-12-20

23:24 Reedy: Katie and Jeff paged about bellatrix
18:46 andrewbogott: graceful restart of zuul as per https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Restart
18:31 andrewbogott: restarting stuck Jenkins
17:47 logmsgbot: reedy@tin Purged l10n cache for 1.27.0-wmf.6
17:11 godog: depool mw1228, reported ro fs
15:53 logmsgbot: reedy@tin Synchronized README: noop (duration: 00m 32s)
15:50 Reedy: reedy@tin Purged l10n cache for 1.27.0-wmf.6 (hanging due to mw1228 issue)
15:42 Reedy: mw1228 reporting readonly fs
15:41 logmsgbot: reedy@tin Purged l10n cache for 1.27.0-wmf.7
09:00 godog: powercycle ms-be2019, xfs lockup
02:28 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Dec 20 02:28:49 UTC 2015 (duration 6m 54s)
02:21 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 08m 59s)

2015-12-19

21:55 _joe_: restarted zotero on sca1001, various OOM messages
20:48 gwicke: restbase1004: `systemctl mask cassandra` in preparation for the decommission finishing
19:49 akosiaris: killed gmond on db2036. it was clearly misbehaving and running since Jan 02. db2036 was not listed on the ganglia web interface. killing the orphaned process and restarting seems to have fixed it
18:54 akosiaris: scheduled maintenance of s3 slave lag on db2036, db2043, db2050, db2057 (all of db2018's family that pages) to effectively silence pages while debugging. Check is flapping since 15:00 UTC today
15:14 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/259611/ - noop for prod, other than making icinga stop complaining (duration: 00m 31s)
10:07 hashar: CI jobs for MediaWiki were broken because of cssjanus dependency. Should be fixed once mw/core https://gerrit.wikimedia.org/r/#/c/260169/ lands
02:28 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Dec 19 02:28:56 UTC 2015 (duration 6m 53s)
02:22 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 08m 53s)
01:01 gwicke: entire restbase cluster: removed 5% root reserve from data partition with tune2fs -m 0 /dev/mapper/restbase$NODE--vg-{srv,var}
00:49 gwicke: restbase1008: removed 5% root reserve from data partition with tune2fs -m 0 /dev/mapper/restbase1008--vg-srv

2015-12-18

22:57 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.9/resources/src/mediawiki/mediawiki.searchSuggest.js: allow override of suggestion type reported in event loggin (duration: 00m 29s)
22:56 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.9/extensions/CirrusSearch/resources/ext.cirrus.suggest.js: override suggestion type reported in event logging (duration: 00m 30s)
22:50 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.9/includes/jobqueue/aggregator/JobQueueAggregatorRedis.php: 2c942ba1782c42ee68622278a5e0a77e9027945d (duration: 00m 30s)
22:30 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.9/extensions/CirrusSearch/resources/ext.cirrus.suggest.js: override suggestion type reported in event logging (duration: 00m 30s)
22:20 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.9/includes/jobqueue/aggregator/JobQueueAggregator.php: 2c942ba1782c42ee68622278a5e0a77e9027945d (duration: 00m 31s)
19:26 logmsgbot: aaron@tin Synchronized wmf-config/jobqueue-eqiad.php: Adjust queue "maxPartitionsTry" and timeouts (duration: 00m 30s)
18:49 mutante: disregard that, apache config only is enough
18:47 mutante: gerrit will restart in a moment and be right back
18:44 ori: ditto
18:43 Krinkle: Created account "Krinkle" on collabwiki
16:28 twentyafterfour: restarted apache on iridium to deploy redirect script changes
16:20 jynus: restarting and reconfiguring mysql on db1047
14:57 godog: stop compactions on restbase1008
14:55 jynus: SET GLOBAL query_cache_type = 0; on db1025
14:54 hashar: gallium: restarted apache2 , was deadlocked/unresponsive somehow
14:44 godog: update privatesettings with swift codfw configuration
14:43 godog: set temp-url-key for mw:media account in swift codfw
12:19 paravoid: upgrading tor on radium, rebooting for kernel upgrade
12:18 _joe_: disabled puppet on all lvs hosts for a potentially harmful change (should be a noop)
11:47 _joe_: restarted hhvm on mw1107, stuck at startup
11:40 hashar: logstash: reorganized list of dashboards per sections https://logstash.wikimedia.org/#/dashboard/elasticsearch/default
09:43 akosiaris: rebooting planet1001, memory exhaustion, OOM showed up
09:20 hashar: Killed Zuul entirely, the queues were full / deadlocked. Patches need to be retriggered
06:47 gwicke: restbase1004: nodetool stop -- COMPACTION to avoid running out of disk space
03:07 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.9/includes/api/ApiStashEdit.php: ab32f4e740: Make ApiStashEdit use statsd metrics (duration: 00m 49s)
02:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Dec 18 02:29:10 UTC 2015 (duration 6m 55s)
02:22 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 08m 45s)
01:52 ori: re-enabled puppet on rdb* / mc*
01:25 ori: in preparation for Iaefb2d191e, disabling puppet on mc* and rdb*
01:21 logmsgbot: krinkle@tin Synchronized docroot and w: (no message) (duration: 00m 32s)
00:53 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.9/extensions/Flow: Revert Nuke-Flow integration, doesn't work (duration: 00m 32s)
00:42 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.9/extensions/Flow: SWAT: Nuke support for Flow, part 3 (duration: 00m 32s)
00:34 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Add completion suggester to BetaFeatures whitelist (duration: 00m 30s)
00:26 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: grumble grumble touch InitialiseSettings grumble (duration: 00m 30s)
00:25 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.9/extensions/Flow: SWAT: Nuke support for Flow, part 2 (duration: 00m 32s)
00:23 logmsgbot: catrope@tin Synchronized wmf-config/CirrusSearch-production.php: SWAT: enable completion suggester beta on all wikis except wikidata (duration: 00m 30s)
00:23 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: enable completion suggester beta on all wikis except wikidata (duration: 00m 29s)
00:20 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.9/extensions/Nuke/: SWAT: Nuke support in Flow, part 1 (duration: 00m 30s)
00:18 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.9/resources/src/mediawiki.messagePoster/mediawiki.messagePoster.factory.js: SWAT: fix error in messagePoster (duration: 00m 29s)
00:17 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.9/extensions/MobileFrontend: SWAT: Schema:MobileWebSectionUsage: always log the isTestA field (duration: 00m 31s)
00:08 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: SWAT: cleanup (duration: 00m 30s)
00:00 ori: restarted mathoid on sca1001

2015-12-17

23:42 mobrovac: mathoid deploying 8d2295
22:47 hashar: ssh to tin is back https://gerrit.wikimedia.org/r/#/c/259876/
22:39 hashar: Only tin lost SSH user keys apparently due to https://gerrit.wikimedia.org/r/#/c/253465/ overriding the admin::groups to simply "eventlogging-admins"
22:34 hashar: Ssh User keys are gone on deployment servers ( tin / mira )
22:18 eileen1: update CiviCRM from b307d744def9289a7f86cb02bc6e1a00225e474d to cb5e20c29d7376920c45eb5c343e6ee464217833
22:10 eileen1: Updating civicrm from b307d744def9289a7f86cb02bc6e1a00225e474d to cb5e20c29d7376920c45eb5c343e6ee464217833
21:33 hashar: gallium: upgrading Zuul from 2.1.0-60-g1cc37f7-wmf2precise1 .. 2.1.0-60-g1cc37f7-wmf4precise1 . Should be noop, only change zuul-cloner which is not used there
21:29 mutante: add zuul_2.1.0-60-g1cc37f7-wmf4jessie1 to jessie-wikimedia repo
21:28 mutante: add zuul_2.1.0-60-g1cc37f7-wmf4trusty1 to trusty-wikimedia repo
21:22 mutante: add zuul_2.1.0-60-g1cc37f7-wmf4precise1 to precise-wikimedia APT
19:42 mobrovac: mathoid deploying a2187a6
19:07 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.27.0-wmf.9
19:05 godog: disable puppet on graphite2001, brief testing cluster aggregations
19:01 thcipriani: starting update of all wikis to 1.27.0-wmf.9
18:11 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool x1-slave (db1031), increase db1041 load to 100% (duration: 00m 30s)
18:06 gwicke: running `nodetool cleanup` on restbase1001 and restbase1005
18:04 robh: calcium is supposed to be down, reclaiming to spares, ignore any irc alerts (its in maint mode in icinga)
17:55 gwicke: running `nodetool cleanup` on restbase1002
17:18 jynus: setting mysql db1031 as db2009's master
17:05 jynus: restarting and reconfiguring mysql at db2009
16:49 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.8/extensions/Math: SWAT: Make math usable without RESTbase gerrit:259734 (duration: 00m 30s)
16:41 jynus: problems with corruption on x1-slave for cebwiki. Fixed them. Will leave db1031 depooled for a while to check they are gone.
16:39 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.9/extensions/ContentTranslation/includes/Translation.php: SWAT: Fix Undefined index: targetRevisionId in ContentTranslation gerrit:259649 (duration: 00m 29s)
16:23 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.9/extensions/WikimediaEvents/WikimediaEventsHooks.php: SWAT: Actually define tags for cross-wiki upload A/B test gerrit:259729 (duration: 00m 31s)
16:17 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable cross-wiki upload A/B test in additional languages gerrit:259665 (duration: 00m 30s)
16:11 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.9/extensions/CirrusSearch/includes/CirrusSearch.php: SWAT: Fix array-to-string conversion gerrit:259633 (duration: 00m 30s)
15:57 moritzm: stopping opendj LDAP servers on nembus/neptunium (read-only since about days now due to migration to openldap)
15:56 akosiaris: depool sca1001, playing with cxserver config
15:24 jynus: reinstall, reboot and reconfigure mysql at db1031
15:03 moritzm: installing git security updates
14:23 _joe_: restarting HHVM on the first jobrunners
14:11 jynus: soft-rebooting mw1004, responsive to ping, but not to salt, ssh
14:06 godog: nodetool stop -- CLEANUP on restbase1002
13:59 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1031 (x1-slave) for maintenance (duration: 07m 30s)
13:11 moritzm: starting to restart hhvm on application servers (to effect security updates for libxml2, openssl and others)
13:01 akosiaris: moved old cruft /srv/deployment/cxserver/deploy/src/config.js out of the way
12:51 akosiaris: repooled sca1002 for cxserver
12:51 akosiaris: restarted cxserver on sca1002
12:33 akosiaris: depooling sca1002 for cxserver
12:33 akosiaris: repooling sca1001 for cxserver
12:24 kart_: Updated cxserver on sca1002
12:08 akosiaris: disable puppet, stop salt-minion on sca1002
12:08 akosiaris: depool sca1001 from cxserver service.
11:56 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1041 with low weight (duration: 00m 37s)
11:33 jynus: performing schema change on officewiki
11:31 jynus: performing schema change on x1-master flowdb
11:20 jynus: performing schema change on x1-master wikishared.cx_translations
10:47 jynus: performing schema change on s7-master metawiki.oauth_registered_consumer
10:30 godog: nodetool stop -- CLEANUP restbase1004
02:53 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Dec 17 02:53:11 UTC 2015 (duration 7m 12s)
02:46 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 08m 22s)
02:27 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.8) (duration: 10m 32s)
00:28 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.8/includes/api/ApiStashEdit.php: I552cf6b0420: Upgrade some ApiStashEdit logging calls to info() (duration: 00m 30s)
00:05 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Password policy for sysadmin group (duration: 00m 29s)

2015-12-16

23:37 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.8/includes/api/ApiStashEdit.php: local hack some extra debug logging into ApiStashEdit (take 2) (duration: 00m 30s)
23:32 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.8/includes/api/ApiStashEdit.php: local hack some extra debug logging into ApiStashEdit (duration: 00m 30s)
23:27 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.9/extensions/Graph/: Deployed Graph ext to master - protocol issue (duration: 00m 32s)
23:19 ejegg: updated DjangoBannerStats from 47503fca7b0ebf2d56ca79c5ec92a4accf88ee77 to 8d4a9062aab80e5371faebadd72fbe4f19ac2fdd
23:10 legoktm: running foreachwiki extensions/CentralAuth/maintenance/checkLocalUser.php --verbose=1 --delete=1 (T119736)
23:07 robh: fermium returned to normal service with new/renewed ssl cert for lists.w.o
22:58 robh: disabled puppet on fermium for the lists.w.o cert update
22:58 robh: librenms returned to normal service (puppet renabled on netmon1001)
22:48 robh: puppet disabled on netmon1001 for librenms certificate update
22:17 ejegg: updated DjangoBannerStats from 26b48add5c6f70b395a3195c401a12612c626796 to 47503fca7b0ebf2d56ca79c5ec92a4accf88ee77
21:31 subbu: finished deploying parsoid 64029e12
21:29 bearND: mobileapps deployed sha1 9f91ad5
21:23 subbu: restarted parsoid on wtp1005 as a canary
21:19 subbu: starting parsoid deploy
21:14 bearND: starting mobileapps deploy
20:18 ejegg: updated SmashPig from b3ee85aab38fdeef3f045ba1733f582e9a5006b2 to 072c7ec6ed94e7074ba35b7986d5dde94866fe2f
19:46 ostriches: iridium: repacking MW / OPUP repositories as phd user. This needs to be a cron.
19:07 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.9
19:00 thcipriani: updating group1 wikis to 1.27.0-wmf.9
18:39 jynus: performing schema change on ruwiki.geo_tags on db2046
17:46 jynus: setting mysql db1041 as db2029's master
17:21 jynus: restarting and configuring mysql on db2029 (there will be an increase of errors- from pings, not real traffic)
17:08 jynus: setting mysql db1022 as db2028's master
16:50 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: graph namespace removal/cleanup (duration: 00m 31s)
16:41 gwicke: cleared snapshots on cassandra cluster
16:32 gwicke: restarted `nodetool decommission` on restbase1004
16:31 jynus: restarting and reconfiguring mysql on db1041
15:54 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1022, depool db1041 (duration: 00m 30s)
15:53 godog: cassandra on restbase1004 couldn't finish decomissioning, out of disk space, running nodetool clean
15:28 jynus: restarting and reconfiguring mysql on db2028
15:17 yurik: updated graphoid service
15:16 moritzm: removed ecryptfs-utils across the cluster
15:09 godog: roll-restart swift daemons on ms-be1*
15:04 jynus: setting db2023 master to be now db1049 instead of m5-master
14:56 mobrovac: restbase roll-restarting restbase
14:52 godog: reenable and run puppet on restbase* and aqs*
14:49 mobrovac: restbase restart rb1001
14:46 urandom: starting `nodetool cleanup' on restbase1009-a.eqiad (https://phabricator.wikimedia.org/T121535)
14:41 urandom: starting `nodetool cleanup' on restbase1006.eqiad (https://phabricator.wikimedia.org/T121535)
14:17 mobrovac: restbase reenabling and running puppet on canary rb1001
14:08 godog: reenable puppet on restbase test cluster
14:06 akosiaris: delete empty panel in https://grafana-admin.wikimedia.org/dashboard/db/graphoid
14:04 akosiaris: repool scb1001 for mobileapps
13:59 akosiaris: depool scb1001 for mobileapps (oid)
13:58 akosiaris: repool sca1002 for citoid, mathoid, graphoid, cxserver
13:55 godog: enable puppet on cerium and bounce restbase
13:55 moritzm: restarting HHVM on canary appservers to effect various security updates (libxml, openssl, gs, freetype)
13:50 godog: enable puppet on restbase-test2001 and bounce restbase
13:47 godog: (retroactive) enable puppet on sca1001
13:46 akosiaris: depooling sca1002, scb1002 for citoid, cxserver, graphoid, mathoid
13:35 godog: disable puppet on deployment_target:restbase/deploy and sca/scb
12:29 jynus: restarting and reconfiguring mysql at db1022
12:26 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1022 for maintenance (duration: 00m 37s)
11:49 jynus: restarting and reconfiguring mysql on db2023
11:34 akosiaris: merged YuviPanda: labstore: Run sync-exports in start-nfs too (d5273d9) and YuviPanda: labstore: Activate volumes before mounting them (dff32de) on palladium
08:54 YuviPanda: start nfs-kernel-server on labstore1001
08:53 YuviPanda: stopped nfs-kernel-server on labstore1001
08:36 mark: Rebooting labstore1001
03:02 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 16m 00s)
02:54 papaul: kafka200[1-2] signing puppet certs, salt key initial run
02:42 papaul: installation complete on kafka200[1-2]
02:30 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.8) (duration: 13m 37s)
02:18 papaul: installing OS on kafka200[1-2]
00:40 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Enable wmgGraphEnableGzip on all wikis (duration: 00m 30s)
00:36 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Enable wmgGraphEnableGzip on mediawikiwiki (duration: 00m 30s)
00:23 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.9/extensions/Graph/: SWAT: update Vega (duration: 00m 30s)
00:12 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Password policy for staff, take 3; fix $wgExtractsRemoveClasses (duration: 00m 31s)

2015-12-15

23:43 eileen: Updating CiviCRM from fa7124e1d8d92b576c7650030fe64d024c822088 to b307d744def9289a7f86cb02bc6e1a00225e474d
23:31 eileen: Updating CiviCRM from fa7124e1d8d92b576c7650030fe64d024c822088 to 49e949c303466db9f96cb38b718190c887f7ed89
23:26 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enabling Wikidata data access for meta-wiki (duration: 00m 30s)
23:25 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings-labs.php: Enabling Wikidata data access for meta-wiki (duration: 00m 29s)
23:24 logmsgbot: aude@tin Synchronized dblists/arbitraryaccess.dblist: Enabling Wikidata data access for meta-wiki (duration: 00m 30s)
23:13 Krinkle: Synchronized php-1.27.0-wmf.9/extensions/ContentTranslation/extension.json 'T121308 - unbreak VE'
22:14 gwicke: restbase: finished full deploy of 3b7ae07e to restbase prod cluster
22:13 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.27.0-wmf.9 mira test
22:07 gwicke: restbase: starting full deploy of 3b7ae07e to restbase prod cluster
22:05 gwicke: restbase: canary deploy of 3b7ae07e to restbase1001
21:53 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.27.0-wmf.9
21:39 logmsgbot: thcipriani@tin Finished scap: testwiki to php-1.27.0-wmf.9 and rebuild l10ncache (duration: 29m 58s)
21:09 logmsgbot: thcipriani@tin Started scap: testwiki to php-1.27.0-wmf.9 and rebuild l10ncache
20:38 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.8/includes/api/ApiStashEdit.php: ad2e2aeedc: Make edit stashing use named DB locks (duration: 01m 46s)
20:21 thcipriani: 1.27.0-wmf.9 branching complete 1h 18m 2s, checking out to tin
20:00 subbu: restarted parsoid on all nodes to kill stuck processes (thanks to T104523)
19:57 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1049 and db1042 (duration: 00m 29s)
19:57 subbu: restarted parsoid on wtp1019 to kill stuck processes
19:42 mutante: mw1146: hhvm restart
19:39 mutante: ms-fe1001 thru ms-fe1003: swift-proxy-server stop/start
19:37 mutante: ms-fe1004, swift-proxy-server stop/start
19:23 mutante: powercycling mw1129
19:08 ottomata: merged change to allow longer programnames in remote rsyslog config.
19:00 thcipriani: starting branch cut for 1.27.0-wmf.9
18:57 bblack: repooling cp1053 (eqiad text cache)
18:37 ori: Depooled and drained mw1161-1169 app servers, now re-purposing as job runners, per T121549
18:19 jynus: changing db2019 master to be db1042 instead of m4-master
17:55 bblack: manually enabled ipsec rules in iptables on kafka10xx - puppet disabled for now until I can fix the puppetization of it...
17:48 urandom: beginning `nodetool cleanup' on restbase1003.eqiad (https://phabricator.wikimedia.org/T121535)
17:48 jynus: restart and reconfigure mysql at db1049
17:39 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1027, Depool db1049 (duration: 00m 30s)
17:33 urandom: beginning `nodetool cleanup' on restbase1005.eqiad (https://phabricator.wikimedia.org/T121535)
17:30 gwicke: aborted large compaction on restbase1004 with `nodetool stop -- COMPACTION` to free disk space
17:29 urandom: beginning `nodetool cleanup' on restbase1002.eqiad (https://phabricator.wikimedia.org/T121535)
17:04 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Update cirrus avro schema to 111448028943 (duration: 00m 29s)
17:00 jynus: restarting and reconfiguring mysql at db2018
16:34 logmsgbot: thcipriani@tin Synchronized portals: SWAT: Bump portals to master (duration: 00m 29s)
16:30 jynus: restarting and reconfiguring mysql on db1042
16:26 godog: decommission restbase1004
16:23 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: VisualEditor: Centralise feedback from test2wiki to MediaWiki.org gerrit:259041 (duration: 00m 30s)
16:21 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: VisualEditor: Enable single edit tab mode on test2wiki gerrit:259039 (duration: 00m 29s)
16:17 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: In VisualEditor on single edit tab wikis, set the default editor appropriately gerrit:258403 (duration: 00m 28s)
16:14 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: BetaFeatures: Update language and dates of "retirement" gerrit:258409 (duration: 00m 29s)
16:14 jynus: switch db2018's master from s3-master to db1027
16:09 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable cross-wiki upload A/B test on English-language wikis gerrit:259258 (duration: 00m 29s)
16:08 logmsgbot: thcipriani@tin Synchronized wmf-config/db-eqiad.php: SWATish: Depool db1042 for maintenance (duration: 00m 29s)
15:50 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1042 for maintenance (duration: 00m 29s)
15:18 jynus: restarting db2018 for upgrade and configuration change
14:41 jynus: restarting mysql on db1027 to apply new configuration
13:53 bblack: disabling puppet on neon to avoid race-condition ipsec alert spam
13:45 hashar: reverted composer upgrade on CI with https://gerrit.wikimedia.org/r/#/c/259241/
13:41 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1018 after maintenance; depool db1027 for maintenance (duration: 00m 29s)
13:37 hashar: bumping composer on CI to 1.0.0-alpha11 https://gerrit.wikimedia.org/r/#/c/258933/
12:48 mobrovac: restbase deploy end of 844a41d
12:39 mobrovac: restbase deploy start of 844a41d
11:46 jynus: restarting and upgrading dbstore2002
11:33 paravoid: force-rebooting stat1002, kernel borked because of fuse
11:25 paravoid: rebooting lvs4003/lvs4004 for kernel upgrade
11:21 paravoid: rebooting lvs3004 for kernel upgrade
11:01 jynus: reloading haproxy on dbproxy1002
10:20 jynus: stopped eventlogging on dbstore1002 and db1047
10:09 jynus: enabling ferm, and restarting mysql at db1046 (m4-master, eventlogging)
10:02 jynus: stopping eventlogging mysql consumers
09:32 moritzm: umounted/remounted hdfs mount on stat1002 (got stuck due to kernel bug, see T121492)
09:32 logmsgbot: hashar@tin Synchronized wmf-config/InitialiseSettings-labs.php: enable EventBus logging channel (currently only in beta) https://phabricator.wikimedia.org/T116786 (duration: 08m 57s)
08:47 hashar: restarted zuul-merger on gallium
08:29 hashar: stopping zuul-merger on gallium for maintenance
07:48 _joe_: rebooting pollux
07:18 _joe_: logged into console on pollux, that made it responsive
03:09 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/259011/ and https://gerrit.wikimedia.org/r/#/c/259010/1 (duration: 00m 30s)
02:32 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Dec 15 02:32:25 UTC 2015 (duration 6m 59s)
02:25 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.8) (duration: 11m 38s)
02:03 ori: deployed I6ebffe559 to job runners
01:32 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Revert cirrus avro schema to 101446746400 due to camus not picking up events from new schema (duration: 00m 30s)
00:34 Reedy: puppet disabled on silver since last Puppet run was at Fri Dec 11 15:27:28 UTC 2015 due to 'testing osm debugging'
00:33 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Graph config updates (duration: 00m 28s)
00:32 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: SWAT: Graph config updates (duration: 00m 29s)
00:19 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 29s)
00:17 logmsgbot: catrope@tin Synchronized wmf-config/: SWAT: Use event-schemas repository for avro schemas (duration: 00m 29s)
00:10 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/MobileFrontend/: SWAT: fix page invalidation in mobile API (duration: 00m 36s)
00:06 logmsgbot: catrope@tin Synchronized wmf-config/: SWAT: Wikidata config changes (duration: 00m 28s)

2015-12-14

23:47 gwicke: restbase: finished deploy of 9f31847ad to restbase prod cluster
23:39 logmsgbot: aaron@tin Synchronized rpc/RunJobs.php: 82f4f9df64e42fd04bd32395 (duration: 00m 29s)
23:34 gwicke: restbase: starting deploy of 9f31847ad to restbase prod cluster
22:50 logmsgbot: mobrovac@tin Synchronized wmf-config/InitialiseSettings.php: Enable logging for the Math extension (duration: 00m 29s)
22:49 gwicke: restbase: canary deploy of 9f31847ad to restbase1001
22:16 robh: puppet enabled on uranium and resumed normal service, cert updated for ganglia
22:10 robh: puppet disabled on uranium for ganglia cert update
22:09 mobrovac: mathoid deploying 5b20fe1
21:34 logmsgbot: ori@tin Synchronized docroot and w: (no message) (duration: 00m 29s)
21:15 subbu: finished deploy of parsoid sha df3171e6
21:11 subbu: restarted parsoid on wtp1004 (~4 mins back) as a canary
21:04 subbu: starting parsoid deploy
21:01 robh: neon returned to normal puppet updates. icinga new cert is live.
20:57 mutante: mw1259 - enabled hyperthreading (logical processor)
20:54 mutante: rebooting mw1259 for BIOS settting
20:50 robh: puppet on neon disabled for certificate update
20:25 yurik: updated graphoid service
20:15 hashar: scandium restarted zuul-merger
20:00 jynus: changing master of db2017 to be now db1018, instead of s2-master
19:20 jynus: rolling restart s2 codfw mysql servers
19:02 mutante: reboot cygnus
18:37 jynus: restarting, upgrading and reconfiguring mysql on db1018
18:22 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1057 after maintenance; depool db1018 for maintenance (duration: 00m 29s)
17:20 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.8/extensions/ContentTranslation/api/ApiContentTranslationToken.php: SWAT: Fix check for JWT gerrit:258776 (duration: 00m 29s)
17:05 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.8/extensions/UploadWizard/resources/mw.UploadWizardDetails.js: SWAT: mw.UploadWizardDetails update gerrit:258767 (duration: 00m 30s)
16:27 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.8/extensions/MobileApp/config/config.json: SWAT: Roll out RESTBase usage to Android Beta app: 55% Part II gerrit:258985 (duration: 00m 29s)
16:14 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.8/extensions/MobileApp/config/config.json: SWAT: Roll out RESTBase usage to Android Beta app: 55% gerrit:258985 (duration: 00m 30s)
16:13 jynus: performing schema change on testwiki.page (s3)
16:05 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: do not index User namespace on ko.wikipedia gerrit:258667 (duration: 00m 29s)
15:01 bblack: "service ganglia-monitor restart" on cp*, because most had no varnish stats flowing into ganglia
14:47 hashar: Stopping zuul-merger daemon on scandium. It lost its disk somehow earlier "DISK CRITICAL - /srv/ssd is not accessible: No such file or directory"
13:31 bblack: rebooting cp4005
13:30 bblack: issues on cp4005, investigating
13:29 mobrovac: restbase cassandra restarting cassandra on rb1004 due tolow disk space caused by compactions
13:22 godog: repool restbase1007
12:04 mobrovac: restbase deploy on rb1007 after re-imaging
11:31 godog: upgrade diamond to 3.5-5 in eqiad
11:13 dcausse: elastic in eqiad: resuming writes
10:50 jynus: performing schema change on wikishared.cx_translations (x1-master)
10:19 dcausse: elastic in eqiad: restarting elastic1016 (to release deleted filedesc)
10:11 jynus: restarting mysql at labsdb1004
10:11 dcausse: elastic in eqiad: freezing writes (to restart elastic1016)
09:46 godog: reimage restbase1007
02:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Dec 14 02:29:39 UTC 2015 (duration 6m 57s)
02:22 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.8) (duration: 09m 00s)

2015-12-13

11:10 dcausse: elastic in eqiad: restarting elastic1012 to release opened log filedesc
10:54 godog: extend fluorine's storage lv (94% util) lvresize --size +300G --resizefs /dev/mapper/vg0-lv0
10:24 dcausse: elastic in eqiad: disabling TRACE indexing slowlog on jawiki_content and zhwiki_content
07:06 ori: elasticsearch1012 ditto.
07:05 ori: elasticsearch1016: Moving /var/log/elasticsearch/* to /var/lib/elasticsearch/logs to free up space.
02:28 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Dec 13 02:28:52 UTC 2015 (duration 6m 49s)
02:22 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.8) (duration: 09m 08s)

2015-12-12

22:17 ejegg: updated fundraising dashboard from 961a24c9cf76966aaa7ba8c60e13b6d1d37fa859 to 59e51c4ff74c3c584daf6c5de3bb66daa764cd28
22:06 ejegg: updated fundraising dashboard from 34dee88d137aa1d0c4487a3a94b87e7ed2f8d0c4 to 961a24c9cf76966aaa7ba8c60e13b6d1d37fa859
21:26 legoktm: running fixDefaultJsonContentPages.php on all wikis (T108663)
21:02 logmsgbot: ori@tin Synchronized wmf-config/jobqueue-eqiad.php: I53f13a159: Add rdb100[5-6] to job queue configuration (duration: 00m 31s)
19:30 hashar: Restarted Zuul
17:01 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/258669/ (duration: 00m 29s)
10:12 godog: bounce nslcd on tools-submit and stop puppet
10:01 dcausse: elastic in eqiad: disabling TRACE indexing slowlog for urwiki_content
09:44 jynus: recreated events on db1057 with sql_bin_log = 0 and restarted replication on db2016
09:15 godog: move old elasticsearch logs on elastic1026 out to /var/lib/elasticsearch/log (/ is full)
09:06 godog: move old elasticsearch logs on elastic1016 out to /var/lib/elasticsearch/log (/ is full)
09:02 godog: move old elasticsearch logs on elastic1012 out to /var/lib/elasticsearch/log (/ is full)
03:02 legoktm: ran fixDefaultJsonContentPages.php --wiki=thwiktionary for T108663
02:28 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Dec 12 02:28:44 UTC 2015 (duration 6m 53s)
02:21 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.8) (duration: 08m 31s)
00:30 paravoid: restarting slapd on seaborgium with manual hack

2015-12-11

23:59 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/CentralNotice: Improve impression diet state machine (duration: 00m 32s)
22:24 subbu: finished deploying parsoid sha ebd62ab5
22:09 subbu: restarted parsoid on wtp1002 as a canary
22:03 subbu: starting parsoid deploy
21:18 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.8/extensions/OpenStackManager/: deploying https://gerrit.wikimedia.org/r/#/c/258516/ (duration: 00m 28s)
20:55 jynus: stopping and reconfiguring replication on db2016 (only for some minutes)
20:48 jynus: setting db1057 as the parent mysql of db2016 for s1 replication
20:29 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/Flow/maintenance/FlowPopulateRefId.php: Fix fatals due to missing wiki condition (T117786) (duration: 00m 29s)
20:06 jynus: restarting and reloading mysql for upgrade and reconfiguration at db1057
19:33 logmsgbot: demon@mira Synchronized README: testing (duration: 01m 35s)
19:26 legoktm: started foreachwiki checkLocalUser.php (CentralAuth) on terbium
19:14 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: CentralAuth logging for Lego (duration: 00m 28s)
19:13 mutante: creating cygnus.codfw.wmnet on ganeti2001
19:06 logmsgbot: demon@tin Synchronized wmf-config/wikitech.php: rm some old ldap stuff. yay hiera! (duration: 00m 28s)
18:54 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: rm some deprecated profiling config (duration: 00m 28s)
18:29 logmsgbot: demon@tin Synchronized docroot/: cleanup wgconf vhost config (the sequel) (duration: 00m 29s)
18:28 logmsgbot: demon@tin Synchronized wmf-config/: cleanup wgconf vhost config (duration: 00m 29s)
18:21 cwd: updated payments from 74143e43eb36f93be7881f626443182d3bb58cef to a1be1ad134d06464e98de180227554fceddc91d4
18:13 logmsgbot: demon@tin Synchronized wmf-config/: Remove ability to turn off Cirrus completely. It's a scary switch that would Break Everything. There's much better options to use if you need to tune its load or behavior. (duration: 00m 29s)
17:54 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1057 for maintenance (duration: 00m 35s)
17:44 jynus: rolling restart of codfw's S1 mysqls finished
16:36 mark: Installed bgp-med branch of operations/debs/pybal on pybal-test2001
16:23 mark: moved quagga install from pybal-test2001 to pybal-test2002
14:54 jynus: restarting and configuring S1:codfw mysqls (db2016,34,42,48,55,62,69,70)
14:40 mark: root@pybal-test2001:~# apt-get install quagga
14:21 andrewbogott: re-imaging promethium
10:57 _joe_: rolling restart of HHVM across the jobrunners to ease memory consumption (this will go on during the day, hhvm restarts will be distantiated by 2 hours each)
10:53 _joe_: restarting hhvm on mw1122 in order to collect heap dumps
10:35 jynus: restarting HHVM on mw1122, mw1145
09:50 jynus: importing several tables from master to dbstore1002 and labs, lag will will be slightly affected
09:26 _joe_: ganeti-rebooting serpens, cannot get into console
07:10 dcausse: deleting some indexing slow logs on elastic1012 and elastic1016
06:46 gwicke: deploy restbase c67a41e9d52: add an expensive title to re-render blacklist, per subbu's request
03:21 gwicke: cassandra: upgraded codfw production nodes to 2.1.12
02:46 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Dec 11 02:46:13 UTC 2015 (duration 6m 46s)
02:39 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.8) (duration: 17m 01s)
02:34 gwicke: restarted nodetool decommission on restbase1007, as I had to restart one of the stream targets
02:15 bblack: unified cert upgrades complete
01:42 bblack: starting unified cert upgrade process
01:13 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.8/includes/jobqueue: Ia44ec5ed4: Add per-partition JobQueueRedis aggregation + Fix bad regex in 6fe2f48df (duration: 00m 31s)
00:59 andrewbogott: rebooting promethium
00:58 logmsgbot: bd808@tin Synchronized wmf-config/InitialiseSettings.php: Add read-more to beta features whitelist (duration: 00m 30s)
00:27 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Search language detection A/B test and RelatedArticles on all Wikipedias in beta (duration: 00m 29s)
00:20 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/CirrusSearch: SWAT (duration: 00m 30s)
00:20 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/RelatedArticles: SWAT (duration: 00m 29s)
00:19 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/ImageMap: SWAT (duration: 00m 29s)
00:19 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/Graph: SWAT (duration: 00m 30s)
00:08 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Staff password policy, take 2 (duration: 00m 28s)
00:06 logmsgbot: catrope@tin Synchronized portals/: SWAT (duration: 00m 30s)

2015-12-10

23:55 gwicke: restbase: deploy 9657c4e
23:38 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/Math: Fix errors (duration: 00m 29s)
23:38 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/Echo: Fix errors (duration: 00m 29s)
23:27 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.8/extensions/WikimediaEvents: Ia44ec5ed4: Updated mediawiki/core Project: mediawiki/extensions/WikimediaEvents 152ecb10311bb04f4f2f91775cf821aff14aa327 (duration: 00m 30s)
23:20 logmsgbot: krenair@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 29s)
22:48 csteipp: Re-deployed a bunch of security patches for wmf8
22:05 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/VisualEditor/: SWAT (duration: 00m 47s)
21:56 gwicke: restbase: finished full deploy of 7398850fe9 to restbase cluster
21:49 gwicke: restbase: starting full deploy of 7398850fe9 to restbase cluster
21:49 logmsgbot: catrope@tin Synchronized wmf-config/: SWAT: VisualEditor config patches (duration: 00m 30s)
21:46 gwicke: restbase: canary deploy of 7398850fe9 to restbase1001
21:44 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/Gadgets/: Fix MediaWiki:MediaWiki: (duration: 00m 29s)
21:35 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: New password policy for staff group (duration: 00m 28s)
21:28 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: SWAT: Echo and Flow config patches (duration: 00m 29s)
21:27 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Echo and Flow config patches (duration: 00m 29s)
21:20 Reedy: Changed email for mbeattie on otrs_wikiwiki per request of mdennis
20:32 subbu: finished parsoid deploy
20:14 subbu: restarted parsoid on wtp1003 as canary
20:10 subbu: starting parsoid deploy
19:11 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.27.0-wmf.8
18:40 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1014 at 100% load and reduce es1019 load (duration: 00m 29s)
18:33 paravoid: reprepro: updating cassandra to latest upstream (2.1.12)
17:25 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1011 at 100% load, repool es1014 with low load (duration: 00m 29s)
17:06 bd808: Updated scholarships.wikimedia.org to a4e3dbf (Ensure that Auth\UserData is constructed with an array)
16:45 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.7/extensions/MobileApp/config/config.json: SWAT: Roll out RESTBase usage to Android Beta app: 30% gerrit:258159 (duration: 00m 29s)
16:42 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.8/extensions/MobileApp/config/config.json: SWAT: Roll out RESTBase usage to Android Beta app: 30% gerrit:258161 (duration: 00m 29s)
16:32 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add three new groups to pawiki, and allow sysops to add or remove users to them gerrit:257881 (duration: 00m 29s)
16:26 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add a rollbacker group at wuuwiki gerrit:253084 (duration: 00m 28s)
16:19 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable filemover group at ukwiki gerrit:255422 (duration: 00m 29s)
16:14 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Nuke and unblockself only for bureaucrats on en.wikiversity gerrit:254029 and Add new group "curator" to enwikiversity gerrit:252012 (duration: 00m 36s)
16:00 jynus: restarting, upgrading and configuring es1014
15:44 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Switchover es3 master es1014 -> es1019 and depool es1014 (duration: 00m 31s)
15:44 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Switchover es3 master es1014 -> es1019 and depool es1014 (duration: 00m 28s)
15:22 jynus: perming es3 master switchover of es1014 -> es1019, no production impact is expected
15:13 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1011 with lower weight after mantenance (duration: 00m 29s)
13:57 jynus: restarting, upgrading and reconfiguring mysql @ es1011
13:45 moritzm: stopping/starting slapd on seaborgium to update LDAP indices
13:39 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Switchover codfw es2 master es1011 -> es1015 (duration: 00m 29s)
13:39 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Switchover es2 master es1011 -> es1015 (duration: 00m 37s)
13:30 paravoid: reenabled puppet on seaborgium & forced a puppet run
13:05 moritzm: restarted slapd on serpens
13:03 moritzm: stopping slapd on serpens to update LDAP indices
12:54 moritzm: re-enabled puppet on serpens / restarted slapd
12:53 jynus: performing switchover of es2 es1011-> es1015 for master maintenance, no production impact is expected
12:13 godog: powercycle mw1119, login on console sluggish and no ssh
12:11 jynus: codfw es2 server restart finished
09:04 jynus: rebooting, restarting and upgrading mysql on es2 (codfw external storage servers)
05:51 ejegg|away: updated fundraising dashdoard from 9f0a4d98586f1281716e39b34d924de524458177 to 34dee88d137aa1d0c4487a3a94b87e7ed2f8d0c4
03:55 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Dec 10 03:55:07 UTC 2015 (duration 54m 36s)
03:00 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.8) (duration: 15m 54s)
02:28 gwicke: updated restbase1008 to cassandra 2.1.12
02:27 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 10m 53s)
01:25 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.7/extensions/CirrusSearch/: Add master timeout parameter to mapping updates so it can be increased when the master is slow (duration: 00m 30s)
01:23 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.8/extensions/CirrusSearch/: Add master timeout parameter to mapping updates so it can be increased when the master is slow (duration: 00m 29s)
01:20 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-production.php: Increase elasticsearch master timeout for maint actions to 2 minutes (duration: 00m 31s)
01:05 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/WikimediaMaintenance/: Fix refreshMessageBlobs script (duration: 00m 29s)
01:01 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/WikimediaMaintenance/: Fix refreshMessageBlobs script (duration: 00m 29s)
00:55 awight: update fundraising CRM from 49e949c303466db9f96cb38b718190c887f7ed89 to fa7124e1d8d92b576c7650030fe64d024c822088
00:54 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/: MessageBlobStore changes (duration: 02m 29s)
00:51 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/skins/Vector/: Make placeholder in logged-out personal bar greyed out (duration: 00m 29s)
00:26 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/Thanks/: Fix bug with topic titles in Flow thank notifications (duration: 00m 28s)
00:18 logmsgbot: catrope@tin Synchronized wmf-config/mobile.php: Enable A/B test to measure impact of collapsing content (duration: 00m 29s)
00:06 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Set $wgExtDistGraphiteRenderApi (duration: 00m 28s)
00:02 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings-labs.php: Shut up unmerged change warning (duration: 00m 30s)

2015-12-09

23:43 papaul: signing salt-key complete
23:28 logmsgbot: hoo@tin Synchronized php-1.27.0-wmf.8/extensions/Wikidata: Item/ Property id values need separate validators each (duration: 00m 39s)
23:26 papaul: auth2001 signing puppet certs
21:52 papaul: OS install complete on auth2001
21:34 subbu: finished deploying parsoid sha a0c626e4
21:30 hashar: Jenkins upgraded to 1.625.3 ( https://phabricator.wikimedia.org/T56599 )
21:22 subbu: restarted parsoid on wtp1003 as a canary
21:20 hashar: Jenkins going down right now.
21:18 subbu: starting parsoid deploy
21:11 hashar: Stopping Jenkins
19:56 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: hewiki and cawiki to php-1.27.0-wmf.8
19:46 logmsgbot: thcipriani@tin Synchronized dblists/group1.dblist: Train: Add hewiki and cawiki to group1. Part II gerrit:257973 (duration: 00m 29s)
19:45 logmsgbot: thcipriani@tin Synchronized dblists/group1-wikipedia.dblist: Train: Add hewiki and cawiki to group1. Part I gerrit:257973 (duration: 00m 33s)
19:10 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.8
19:02 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.7/extensions/CirrusSearch/includes/Job/Job.php: Revert manual adjustment made to CirrusSearch Job.php (duration: 00m 29s)
18:56 mutante: mw1003 - restarted hhvm (and apache a bit earlier too)
18:16 Jamesofur: inserted decryption key into database for enWP Arbcom case
17:31 akosiaris: restarted hhvm on mw1125, memory starvation
17:31 akosiaris: restarted hhvm on mw1125
17:17 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.8/extensions/Wikidata: SWAT: Update Wikibase gerrit:257901 (duration: 00m 49s)
17:05 logmsgbot: krenair@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 28s)
16:44 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add rights to CU+OS groups on en.wikipedia.org gerrit:255129 (duration: 00m 29s)
16:32 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Turn off language detection user test gerrit:254071 (duration: 00m 29s)
16:25 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Set initial titlesuggest shard sizes gerrit:256974 (duration: 00m 29s)
16:18 logmsgbot: thcipriani@tin Synchronized dblists/group1.dblist: SWAT: Move Catalan and Hebrew Wikipedias to group1 gerrit:253371 (duration: 00m 29s)
16:12 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable importupload and some import sources for officewiki gerrit:256881 (duration: 00m 42s)
16:06 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add config for wgVisualEditorUseSingleEditTab and set true for enwiki Labs Part II gerrit:257800 (duration: 00m 28s)
16:05 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Add config for wgVisualEditorUseSingleEditTab and set true for enwiki Labs Part I gerrit:257800 (duration: 00m 29s)
15:48 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: etherpad-lite_1.5.7-2
15:01 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1019 after maintenance (duration: 00m 33s)
14:09 godog: decommission restbase1007
13:06 hashar: restarting Jenkins (upgrading Gearman plugin)
11:51 akosiaris: reissue certificates for labs-ldap.{eqiad,codfw}.wikimedia.org with a 2048bit key size
11:29 kart_: Update cxserver to c88ef57
10:45 akosiaris: sudo gnt-instance modify -H cpu_type=SandyBridge seaborgium to get aesni and sse cpu flags. rebooting seaborgium to apply those
10:25 akosiaris: reboot serpens to add 3 more vCPUs
10:04 moritzm: restarting slapd on serpens/seaborgium to apply updated indices
10:03 ori: running hhvm-collect-heaps in screen on mw1001 with a 600s interval
10:00 akosiaris: stopped slapd on seaborgium, run sudo -u openldap slapindex to add the new indices, started slapd
09:58 akosiaris: stopped slapd on serpens, run sudo -u openldap slapindex to add the new indices, started slapd
08:15 ori: Memory utilization on the job runners began to climb around 2015-12-08T10:00:00+00:00 UTC. To investigate, enabled jemalloc profiling and restarted HHVM on mw1001.
07:40 YuviPanda: restarted create-dbusers on labstore1001
04:44 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Dec 9 04:44:57 UTC 2015 (duration 1h 50m 43s)
02:54 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.8) (duration: 04m 34s)
02:34 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 10m 27s)
02:23 logmsgbot: ori@tin Synchronized wmf-config/wikitech.php: I4ef826af47: Increase LDAP log verbosity (duration: 00m 44s)
02:16 logmsgbot: catrope@tin Finished scap: Graph cherry-picks for wmf.8, including i18n changes (duration: 26m 38s)
01:49 logmsgbot: catrope@tin Started scap: Graph cherry-picks for wmf.8, including i18n changes
01:45 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.8/extensions/Gadgets/: SWAT: bump MediaWikiGadgetsDefinitionRepo cache version (duration: 00m 29s)
01:19 awight: Updating paymentswiki from 96d8a49ecd521ac8ce75521aaf97ed8f0dbefc9a to 74143e43eb36f93be7881f626443182d3bb58cef
00:36 ejegg: updated paymentswiki from ff7a706219e72f7362f4e06386e6732933a40478 to 96d8a49ecd521ac8ce75521aaf97ed8f0dbefc9a

2015-12-08

23:47 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: Train: group0 to 1.27.0-wmf.8
23:43 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.8/extensions/MobileFrontend: I7b86a521: Ensure the parser cache varies on images disabled and 'light' images (duration: 00m 32s)
23:41 logmsgbot: thcipriani@tin Finished scap: testwiki to 1.27.0-wmf.8 and rebuild l10n cache take II (duration: 56m 13s)
22:45 logmsgbot: thcipriani@tin Started scap: testwiki to 1.27.0-wmf.8 and rebuild l10n cache take II
22:38 eileen: Updating civicrm from d385e34c775aec50a5c17357c8a11cb0244123ce to 291816c88ab07bb914d56c70ff3d58eb1aa0124d
22:37 eileen: Updating civicrm from xyz to abc
22:24 logmsgbot: thcipriani@tin scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="cawikibooks" --list-file="/srv/mediawiki-staging/wmf-config/extension-list" --output="/tmp/tmp.d9wYd5Axk0" ' returned non-zero exit status 1 (duration: 01m 01s)
22:23 logmsgbot: thcipriani@tin Started scap: testwiki to 1.27.0-wmf.8 and rebuild l10n cache
22:16 yurik: deployed latest kartotherian & tilerator
21:09 thcipriani: checking out wmf/1.27.0-wmf.8 on tin for train deploy
20:51 moritzm: moved dns aliases ldap-eqiad.wikimedia.org and ldap-codfw.wikimedia.org to seaborgium/serpens (these are for compat reasons, using ldap-labs.[eqiad|codfw].wikimedia.org is preferred
20:50 gwicke: restbase: finished deploy of 49fdc615b to all nodes
20:37 gwicke: restbase: start deploy of 49fdc615b to all nodes
20:35 hashar: Jenkins: changing LDAP config from ldaps://ldap-eqiad.wikimedia.org:636 to ldaps://ldap-labs.eqiad.wikimedia.org:636
20:34 gwicke: restbase: canary deploy of 49fdc615b to restbase1001
20:31 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.7/extensions/MobileFrontend: I8a1fb69724b: Vary HTML output by NetSpeed designation (duration: 00m 28s)
19:52 akosiaris: fixed ldap logging on seaborgium/serpens
19:51 logmsgbot: ori@tin Synchronized wmf-config/InitialiseSettings.php: Enable ldap log channel (duration: 00m 28s)
19:50 cmjohnson1: swapping failed disk db1019
18:53 akosiaris: running puppet on serpens/seaborgium
18:49 paravoid: enable ldaps:/// on seaborgium
18:49 akosiaris: enable ldaps:/// on serpens
18:36 logmsgbot: demon@tin Synchronized wmf-config/wikitech.php: Point to new ldap servers (duration: 00m 27s)
18:25 andrewbogott: running puppet on labcontrol1002
18:23 andrewbogott: running puppet on labcontrol1001
17:35 legoktm: deleted centralauth.localuser row for User:ThanduxoloKennethTwani5092/enwiki because the account doesn't exist (T120655)
17:13 moritzm: stopping slapd on seaborgium/serpens to refresh LDAP databases with latest export from nembus
17:00 andrewbogott: changing opendj writability-mode to ‘internal-only’ on nembus and neptunium
16:31 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.7/extensions/CentralNotice: SWAT: Update CentralNotice gerrit:257615 (duration: 00m 28s)
16:19 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix typo in namespaces configuration gerrit:257487 (duration: 00m 28s)
16:13 logmsgbot: thcipriani@tin Synchronized wmf-config/Wikibase.php: SWAT: Wikidata: set maxSerializedEntitySize to 2500 gerrit:257318 (duration: 00m 27s)
16:07 godog: repool restbase1008
16:06 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: CX: Use ContentTranslationRESTBase gerrit:255102 (duration: 00m 30s)
16:01 mobrovac: restbase end deploy of 28bf071
15:22 mobrovac: restbase enable puppet in prod
15:17 mobrovac: restbase start deploy of 28bf071
15:10 mobrovac: restbase canary deploy of 28bf071 onto restbase1001
14:19 bblack: upgrading openssl package on caches
14:16 mobrovac: restbase canary deploy of f47405a onto restbase1001
14:09 mobrovac: restbase running puppet on rb1001
11:02 godog: upgrade diamond to 3.5-5 in codfw
10:04 godog: start cassandra on restbase1008, bootstrapping
09:56 hashar: gallium: upgraded python-diamond and java. Restarting Jenkins
09:48 hashar: gallium: removing bunch of :i386 packages which were used for Android build but got removed ( https://gerrit.wikimedia.org/r/#/c/183790/3/modules/androidsdk/manifests/dependencies.pp,unified )
09:15 godog: reimage restbase1008
05:45 gwicke: restbase1008: disable cassandra with `systemctl mask cassandra`
04:00 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Dec 8 04:00:02 UTC 2015 (duration 1h 34m 44s)
03:12 gwicke: Keeping puppet on restbase production cluster disabled until Marko & Filippo can deploy tomorrow morning. PLEASE DO NOT ENABLE!
02:25 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 09m 56s)
01:16 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Also blacklist Minerva for WPB (duration: 00m 28s)
00:59 gwicke: disabling puppet in production restbase cluster in preparation for testing https://gerrit.wikimedia.org/r/#/c/257408/ in staging
00:39 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/extensions/MobileFrontend/: SWAT (duration: 00m 28s)
00:38 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/extensions/MwEmbedSupport/: SWAT (duration: 00m 28s)
00:37 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/resources/src/oojs-ui-local.js: SWAT: connect OOjs UI to MW's l10n system (duration: 00m 28s)
00:37 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/resources/ResourcesOOUI.php: SWAT: connect OOjs UI to MW's l10n system (duration: 00m 27s)
00:34 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: enable banners on mobile web beta on enwiki; remove MobileFrontend Browse config (duration: 00m 27s)
00:31 logmsgbot: catrope@tin Synchronized wmf-config/squid.php: SWAT: stop using separate address for upload in wgHTCPRouting (duration: 00m 29s)
00:07 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Enable Flow opt-in beta feature on bswiki and urwiki (duration: 00m 28s)

2015-12-07

22:29 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.7/extensions/MobileFrontend: I8c94425693: Improve disableImages cookie code (T120151) (duration: 00m 30s)
22:22 logmsgbot: ebernhardson@tin Synchronized portals/prod: Deploy updated portals (duration: 00m 27s)
22:01 mdholloway: mobileapps deployed 984f14b
21:31 logmsgbot: hoo@tin Synchronized php-1.27.0-wmf.7/extensions/Wikidata/: (no message) (duration: 00m 37s)
21:15 subbu: finished deploying parsoid sha 4a7df427 (cf0b9ef + cherry-pick of d65debd)
21:09 subbu: restart parsoid on wtp1002 as a canary
21:05 subbu: starting parsoid deploy
20:33 andrewbogott: enabling puppet on labcontrol1002, testing over
20:07 andrewbogott: disabling puppet on labcontrol1002 for an ldap/pdns test
19:52 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1015 at 100% load (duration: 00m 27s)
19:52 andrewbogott: restarting nova-compute on all labvirt1X nodes
19:20 andrewbogott: restarted pdns on labcontrol1001 and 1002 to catch up with recent opendj restarts
19:11 andrewbogott: restarting ldap on neptunium and nembus due to replication warnings in the logs
18:36 jynus: es1019 and its management interface became unresponsive
18:16 mutante: deleting pmtpa host groups from nagios/icinga
17:04 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.7/extensions/MobileApp/config/config.json: SWAT: Roll out RESTBase usage to Android Beta app: 10% gerrit:254045 (duration: 00m 29s)
17:02 thcipriani: reset uncommited change to includes/jobqueue/JobRunner.php in /srv/mediawiki-staging/php-1.27.0-wmf.7
17:00 hashar: Restarted Jenkins to start the ZeroMQ publisher https://phabricator.wikimedia.org/T120668
16:56 _joe_: upgrading pybal on primaries in esams, ulsfo, codfw
16:44 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Namespace configuration for en.wiktionary gerrit:254476 and Enable subpages on custom aliases from 112 to 119 gerrit:254475 (duration: 00m 27s)
16:43 _joe_: installing pybal on eqiad backups
16:43 jynus: restarting mysql, upgrading and rebooting mysql on es1019
16:36 _joe_: installing the new pybal version on the backup lvss in esams and ulsfo
16:31 _joe_: upgrading pybal on all the backup LVSs in codfw
16:28 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Rights configuration on fa.wikipedia gerrit:254486 (duration: 00m 29s)
16:28 _joe_: upgrading pybal on lvs1007-12
16:19 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Namespace configuration for ur.wikipedia gerrit:254665 (duration: 00m 30s)
16:18 _joe_: uploaded pybal 1.13.2 to reprepro
16:14 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Translate project namesapce for jbowiki gerrit:255954 (duration: 00m 28s)
15:39 moritzm: uploaded openssl 1.0.2e-1~wmf1 to carbon
15:37 yurik: deployed kartotherian
15:25 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1019; es1017 at 100% load; pool es1015 with low weight (duration: 00m 28s)
14:56 yurik: deployed latest tilerator
14:46 _joe_: also restarted pybal on lvs3003
14:42 _joe_: restarted pybal on lvs1006
14:39 hashar: restarting Jenkins
14:32 hashar: Jenkins lost a bunch of executors :/
14:30 hashar: CI / Zuul stalled somehow
12:56 jynus: rolling restart, configuration upgrade of es1015
12:20 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1015; es1013 at 100% load; pool es1017 with low weight (duration: 00m 28s)
11:08 YuviPanda: restarting pdns on holmium
10:52 jynus: database and system maintenance to es1017
10:43 hashar: CI / zuul / nodepool recovered. Root cause was some malfunction in openstack wmflabs
10:20 YuviPanda: restarted nova-conductor and scheduler on labcontrol1001
10:07 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1013 (lower weight for now) and depool es1017 (duration: 00m 41s)
10:05 hashar: stopped Nodepool. Can not create instances anymore on wmflabs ( https://phabricator.wikimedia.org/T120586 )
09:46 hashar: restarting Nodepool on labnodepool1001.eqiad.wment
09:40 hashar: CI / Zuul stalled. Nodepool can no more spawn instances :-/
09:27 godog: nodetool decommission restbase1008
09:13 jynus: es1013 maintenance (mysql restart, upgrade, possible reboot)
08:27 _joe_: uploaded etcd 2.2 package from stretch to jessie-wikimedia
03:56 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Dec 7 03:56:49 UTC 2015 (duration 1h 32m 22s)
02:24 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 09m 59s)

2015-12-06

21:48 ori: krypton unresponsive, nothing on console. shutting down, increasing instance ram from 2 to 4g, and rebooting.
18:49 legoktm: reset auth token for User:QuimGil
05:50 mutante: silver gzip /var/log/nutcracker.log.1
05:40 mutante: silver: apt-get clean for disk space
03:57 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Dec 6 03:57:02 UTC 2015 (duration 1h 31m 41s)
02:25 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 10m 04s)

2015-12-05

18:30 gwicke: started nodetool decommission on restbase1008
11:35 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: Disable common password password policy to come in wmf.8 (duration: 00m 28s)
11:23 logmsgbot: reedy@tin Purged l10n cache for 1.27.0-wmf.5
11:22 logmsgbot: reedy@tin Synchronized php-1.27.0-wmf.7/extensions/WikimediaMaintenance/refreshMessageBlobs.php: Less waiting for slaves (duration: 00m 28s)
11:13 logmsgbot: reedy@tin Synchronized docroot and w: Add jobqueue-labs to noc (duration: 00m 28s)
08:59 bblack: offlined db1019 megacli disk 32:11
06:09 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Dec 5 06:09:07 UTC 2015 (duration 3h 44m 18s)
02:24 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 09m 59s)

2015-12-04

22:20 ejegg: updated dash from bed9895563fc0f5d3b96791fae6ed104b7778f0d to 9f0a4d98586f1281716e39b34d924de524458177
21:44 andrewbogott: disabling puppet on labcontrol1002 for ldap testing
21:36 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.7/includes/Hooks.php: Iba0138a: Don't install a custom error handler for hooks (T117553) (duration: 00m 28s)
21:26 awight: Update fundraising CRM from 8a3deedc04e4d3d1c1f78cb5b7ace7d8670f307a to f0ef0dab1fa7a85426a000c35750093d541fe4f5
20:28 logmsgbot: ori@tin Synchronized wmf-config/jobqueue-eqiad.php: Idee6a1980: job queue: use instances on port 6378 as aggregators (duration: 00m 30s)
19:21 ori: krypton: updated Grafana to 2.6.0-beta1 for bug fix for issue 3422
15:52 Jeff_Green: add mx record for donate.wikimedia.org
15:33 godog: ms-be2019 rebooted by itself, ilo event log shows "Uncorrectable Machine Check Exception (Board 0, Processor 2, APIC ID 0x00000038, Bank 0x00000003, Status 0xFE000040'00020135, Address 0x00000000'FEB82F63, Misc 0x00000000'00002285)"
08:52 godog: reimage restbase1009
05:59 gwicke: ran systemctl mask cassandra on restbase1009; it is important that this node does not start up.
05:53 gwicke: moved /var/lib/cassandra out of the way in an attempt to stop puppet restarting cassandra on decommissioned restbase1009
05:49 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Dec 4 05:49:46 UTC 2015 (duration 3h 21m 36s)
02:28 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 10m 19s)
02:15 ori: CirrusSearch-common.php sync was for I826d000ca: Turn off backoff throttling of CirrusSearch jobs
02:15 logmsgbot: ori@tin Synchronized wmf-config/CirrusSearch-common.php: (no message) (duration: 00m 29s)
01:33 bd808: Updated scholarships.wikimedia.org to af73bf6
00:35 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/extensions/CentralNotice: SWAT (duration: 00m 32s)
00:01 ejegg: pointed SmashPig IPN listener to live Adyen account

2015-12-03

23:09 bblack: restarting pybal (w/ BGP enabled) on lvs100[123] (newly-installed w/ jessie)
22:59 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.7/includes/jobqueue/JobRunner.php: temporarily disable job throttling (duration: 00m 29s)
22:35 cwd: updated paymentswiki configuration [redux]
22:15 cwd: updated paymentswiki configuration
22:09 bd808: Removed zirconium.wikimedia.org from Trebuchet minions list for scholarships/scholarships
22:04 bd808: Updated scholarships.wikimedia.org to cb94319 plus local i18n filtering
21:48 Reedy: finished removing bogus msg_resource rows
21:28 logmsgbot: oblivian@tin Synchronized wmf-config/CommonSettings.php: re-sync (re-merged the change) (duration: 00m 29s)
21:27 bd808: Applied database migrations and purged last year's data from Wikimania Scholarships db
21:21 ottomata: restarted eventlogging with 4 mysql consumer processes running in parallel
21:21 bblack: rebooting lvs100[123] for reinstall to jessie
21:18 Reedy: Cleaning up msg_resource rows with bogus language codes
21:15 gwicke: stopped cassandra on 1009 as it's decommissioned & will be reimaged
21:13 logmsgbot: oblivian@tin Synchronized wmf-config/CommonSettings.php: Re-fix the jobqueue on wikitech after redis cleanup (duration: 00m 26s)
20:55 logmsgbot: oblivian@tin Synchronized wmf-config/CommonSettings.php: Fix the jobqueue on wikitech (duration: 00m 47s)
20:45 _joe_: opening connection from mw1001 to silver, mysql
20:29 ori: on palladium: salt -G 'cluster:jobrunner' cmd.run 'service jobrunner status | grep running && service jobrunner restart' ; salt -G 'cluster:jobrunner' cmd.run 'service jobchron status | grep running && service jobchron restart'
20:28 ori: ran srem jobqueue:aggregator:s-wikis:v2 labswiki on rdb1001 aggr
19:41 bblack: disabling pybal on lvs100[123] over the next few minutes (for reinstall to jessie later after confirmation everything is still ok on [456])
19:10 jynus: restarting eventlogging_sync on db1047 and dbstore1002
19:04 jynus: starting m4 slave again on dbstore2002
18:45 andrewbogott: disabling puppet on labcontrol1002 to test openldap with pdns
18:33 mutante: neon - remove icinga user from "dialout" group
18:27 jynus: disabling eventlogging_sync process on dbstore1002 and db1047 and replication on the other m4 slaves
18:18 jynus: disabling event scheduler on db1046 (m4-master)
17:26 ejegg: updated fundraising dashboard from 8412a2f5f91bb3a9f7873d6ac9fd5d7884eb988c to bed9895563fc0f5d3b96791fae6ed104b7778f0d
17:03 logmsgbot: kartik@tin Finished scap: Update ContentTranslation (duration: 05m 52s)
16:57 logmsgbot: kartik@tin Started scap: Update ContentTranslation
16:50 logmsgbot: oblivian@tin Synchronized wmf-config/CommonSettings.php: Fix the jobqueue on wikitech (duration: 00m 28s)
15:23 andrewbogott: stopping pdns on labcontrol2001
15:11 moritzm: restarting cassandra on restbase100[56] (subsequently) to effect openjdk security update
14:57 mobrovac: restbase end of deployment of 262da91a
14:48 mobrovac: restbase start deployment of 262da91a
14:06 moritzm: installed dpkg updates across the cluster
11:35 moritzm: restarting cassandra on aqs cluster (subsequently) to effect openjdk security update
10:51 jynus: restarting, upgrading and general maintenance for es1013 (depooled)
10:36 _joe_: imported dh-python into precise/universe from the ubuntu cloud archive
10:26 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1013 for maintenance (duration: 00m 30s)
05:50 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Dec 3 05:50:08 UTC 2015 (duration 50m 7s)
02:32 ejegg: updated CiviCRM from d5afa10383cfd41adbe798c3ed2aa97839be8bc2 to 8a3deedc04e4d3d1c1f78cb5b7ace7d8670f307a
02:25 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 09m 54s)
00:18 awight: Deploy paymentswiki config for T112181

2015-12-02

23:47 ejegg: updated fundraising dashboard from 01a82b5e4909075e3190496e7fcda2b60f8d5d3c to 8412a2f5f91bb3a9f7873d6ac9fd5d7884eb988c
23:07 cwd: updated payments from 02a3771cae6e4e74d9d754a8b075de20aa171adc to ff7a706219e72f7362f4e06386e6732933a40478
22:09 jynus: unscheduled restart of dbstore1002 (analytics-slave)
21:44 jynus: disabling all alert notifications for dbstore1002
21:30 bblack: rebooting lvs1007 for interface config test (not active, no BGP)
20:37 Jeff_Green: enable mail queue monitoring for fundraising
19:13 jynus: restarting mysql on db2067 to test a configuration change
18:07 mutante: mw1136 - hhvm restart
14:45 Coren: deploying cleanup of labs PAM configuration - this should be a functional noop but may cause some puppet noise
12:41 akosiaris: restart cassandra on maps-test200{1,2,3,4}.codfw.wmnet
11:23 moritzm: restarting cassandra on restbase100[78] (subsequently) (to effect openjdk security updates plus related libs)
11:06 moritzm: restarting cassandra on restbase100[2-4] (subsequently) (to effect openjdk security updates plus related libs)
11:04 moritzm: restart cassandra on restbase1001 (to effect openjdk security updates plus related libs)
10:21 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enabling data access for wikinews, wikispecies and mediawiki.org (duration: 00m 27s)
10:20 logmsgbot: aude@tin Synchronized dblists/arbitraryaccess.dblist: Enabling data access for wikinews, wikispecies and mediawiki.org (duration: 00m 30s)
10:14 _joe_: clearing the job cache for ocg1003
09.43 +
09:22 _joe_: stopped ocg service on ocg1003
08:38 _joe_: planet1001 recovered as soon as I did get into console via gnt-instance
06:25 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.7/extensions/NavigationTiming: Idb675cdce: Add isHiDPI and isHttp2 properties; drop isHttps (T119014) (duration: 00m 50s)
05:53 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Dec 2 05:53:21 UTC 2015 (duration 53m 20s)
02:28 mutante|away: labcontrol2001 - disable puppet, kill from puppet stored configs
02:26 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 10m 47s)
00:29 mutante: puppetstoredconfigclean.rb labcontrol2001.wikimedia.org fixes icinga config

2015-12-01

22:41 mutante: stop/start ntp service on lvs2002
21:04 ejegg: updated fundraising from 2ee15180150c8b685b14abdbf7c6769736fb49a0 to 01a82b5e4909075e3190496e7fcda2b60f8d5d3c
20:54 mutante: re-enabled puppet on cache::misc, replaced planet cert
20:19 mutante: temp. disabling puppet on cache::misc
18:14 twentyafterfour: restarted apache on iridium to deploy https://phabricator.wikimedia.org/rPHEXd724c51a4144f7de548546875097da57ee1b38d7
17:05 moritzm: restarting cassandra instances on restbase200[3-6] (to effect openjdk security updates and a few depending libs)
17:00 moritzm: restarting cassandra instances on restbase2002 (to effect openjdk security updates and a few depending libs)
16:55 moritzm: restarting cassandra instances on restbase2001 (to effect openjdk security updates and a few depending libs)
16:22 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor for all accounts on eswiki. Part II gerrit:250474 (duration: 00m 28s)
16:21 logmsgbot: thcipriani@tin Synchronized dblists/visualeditor-default.dblist: SWAT: Enable VisualEditor for all accounts on eswiki. Part I gerrit:250474 (duration: 01m 29s)
15:30 ejegg: changed donation qc time limit from 90 to 100 sec
12:59 godog: reimage mw2208
10:50 moritzm: restarting on cassandra instances on restbase1008 (to effect openjdk security updates and a few depending libs)
06:52 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Dec 1 06:52:40 UTC 2015 (duration 52m 39s)
02:26 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 10m 14s)
02:18 mutante: restarted hhvm on mw1147
01:47 logmsgbot: ori@tin Synchronized docroot and w: (no message) (duration: 00m 30s)
00:51 ejegg: updated fundraising dash from bc0ec4bb2427ebeb2d4cd60eda65e3d82d006937 to 2ee15180150c8b685b14abdbf7c6769736fb49a0
00:47 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/extensions/VisualEditor: SWAT (duration: 00m 29s)
00:47 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/resources/src/mediawiki/mediawiki.ForeignStructuredUpload.js: SWAT (duration: 00m 30s)
00:45 ejegg: updated Adyen credentials in SmashPig IPN listener
00:39 ejegg: updated Adyen credentials on payments-wiki
00:35 ejegg: updated payments-wikifrom 7fb9d6599c3833f0a55500ab77abd2b7ffe74cfe to 02a3771cae6e4e74d9d754a8b075de20aa171adc

2015-11-30

22:53 ejegg: updated fundraising dash from 5a6b2dda71e6ce76d7bbba853acae8dc9416052c to 5afc7b02afbd9385db80222271d3d8cc64c1ea13
22:37 ejegg: updated fundraising dashboard from 5a6b2dda71e6ce76d7bbba853acae8dc9416052c to 5afc7b02afbd9385db80222271d3d8cc64c1ea13
22:16 awight: Configuring paymentswiki to not exlicitly include DefaultSettings.php
21:55 mutante: re-wrote l10nupdate cron; restarted cron service on tin
21:53 awight: Tweak paymentswiki-staging config: remove line that exlicitly includes DefaultSettings.php
20:05 apergos: re-enabled puppet on neodymium, minion testing concluded for now
19:47 gwicke: running `nodetool decommission` on restbase1009 in preparation for the conversion to the multi-instance setup, per https://phabricator.wikimedia.org/T95253#
19:31 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: rm deprecated/unused rate limit log config (duration: 00m 28s)
17:27 logmsgbot: demon@tin Synchronized php-1.27.0-wmf.7/extensions/WikimediaMaintenance/: need maint script errywhere (duration: 00m 28s)
16:51 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.7/extensions/ContentTranslation/modules/draft/ext.cx.draft.js: SWAT: Add some extra information to save failure logging gerrit:255956 (duration: 00m 28s)
16:38 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable QuickSurveys reader segmentation survey gerrit:255448 (duration: 00m 28s)
16:30 paravoid: mw1002 service hhvm restart
16:17 paravoid: rolling back to kernel 3.19 on lvs2001/2/3
15:29 paravoid: stopping pybal on lvs2001/2/3
15:21 paravoid: switching lvs2004/5/6 traffic back to lvs2001/2/3
15:13 paravoid: switching lvs2001/2/3 traffic to lvs2004/5/6 and upgrading kernels
15:12 _joe_: restarting HHVM on mw1147 too, same reason as mw1114
15:10 _joe_: restarting hhvm on mw1114, stuck in __pthread_cond_wait () [folly::EventBase::runInEventBaseThreadAndWait ()], apparently blocked in writing to stdout
15:02 paravoid: switching traffic from lvs4002 to lvs4004; upgrading lvs4002's kernel
15:02 paravoid: switching traffic back to lvs4001
14:57 paravoid: switching traffic from lvs4001 to lvs4003; upgrading lvs4001's kernel
14:45 paravoid: switching traffic from lvs3001 to lvs3003; upgrading lvs3001's kernel
14:38 paravoid: switching traffic back to lvs3002
14:31 paravoid: switching traffic from lvs3002 to lvs3004; upgrading lvs3002's kernel
14:07 bblack: upgrading varnishkafka package on all caches
13:52 bblack: updating varnishkafka on cp1065
11:03 godog: upgrade python-statsd to 3.0.1 in eqiad
10:59 godog: upgrade python-statsd to 3.0.1 in codfw
10:15 godog: reenable puppet on graphite1001
10:10 paravoid: re-enabling OSPF over cr2-eqiad:xe-5/2/2 <-> cr1-ulsfo:xe-0/0/3.538
10:09 paravoid: re-enabling cr2-eqiad:xe-5/2/0 and xe-5/2/1
10:01 jynus: performing schema change on db1046 (analytics master)
09:32 jynus: removing old snapshots from db1046
06:39 ori: Restarted statsv on hafnium
02:00 logmsgbot: l10nupdate@tin LocalisationUpdate failed: git pull of core failed
01:56 gwicke: started `nodetool cleanup` on restbase1002 to get rid of unnecessary data from earlier 1001 decommission attempt
01:05 logmsgbot: bd808@tin sync-l10n completed (1.27.0-wmf.7) (duration: 01m 19s)
01:04 bd808: testing l10n cache rebuild as l10nupdate user (take 2)
00:57 Krenair: test

2015-11-29

21:25 jynus: importing user.user_touched (s7) from dbstore1002 to sanitarium. s7 lag on labs replicas will be higher for some minutes.
20:51 jynus: importing user.user_touched (s6) from dbstore1002 to sanitarium. s6 lag on labs replicas will be higher for some minutes.
20:28 jynus: importing user.user_touched (s5) from dbstore1002 to sanitarium. s5 lag on labs replicas will be higher for some minutes.
19:51 jynus: importing user.user_touched (s4) from dbstore1002 to sanitarium. s4 lab will be affected for some minutes.
04:50 gwicke: restarted cassandra on restbase1009 to avoid it running out of disk space; had large compaction (~2TB) at 80% and only 64G disk space left
03:01 YuviPanda: run chown -R l10nupdate: /var/lib/l10nupdate/mediawiki for Reedy on tin
02:28 Reedy: l10nupdate failed because some git objects owned by 997:l10nupdate
02:00 logmsgbot: l10nupdate@tin LocalisationUpdate failed: git pull of core failed

2015-11-28

22:48 logmsgbot: bd808@tin Synchronized php-1.27.0-wmf.5/cache/l10n: bd808 testing l10nupdate sync-dir using stale branch (duration: 01m 29s)
20:49 logmsgbot: l10nupdate@tin LocalisationUpdate failed: Failed to sync-dir 'php-1.27.0-wmf.7/cache/l10n'
20:49 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 07m 11s)
20:35 logmsgbot: ori@tin Synchronized wmf-config/InitialiseSettings.php: Ie33ae3b6a: Increase $wgCopyUploadTimeout to 90 seconds (from default 25) (T118887) (duration: 00m 27s)
02:00 logmsgbot: l10nupdate@tin LocalisationUpdate failed: git pull of core failed

2015-11-27

21:23 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/255726/1 (duration: 00m 31s)
17:46 paravoid: re-enabling cr2-eqiad:xe-5/2/2 (link to cr1-ulsfo) and xe-5/2/3 (link to cr2-codfw)
15:08 godog: kill stray cpu hog xelatex on ocg1003, orphan and started on oct22
14:16 Jeff_Green: fundraising database maintenance
12:36 godog: ban some grafana dashboards from graphite
10:56 apergos: restarted gitblit on antimony about 40 minutes ago, it takes about 20 minutes to load up caches of commits, now responding but... it's taking 17-18 seconds to load repos instead of 1-3 as normal. so timeouts
10:47 jynus: importing user.user_touched (s3) from dbstore1002 to sanitarium, it will affect s3 labsdbs lag
10:44 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repooling previouly problematic servers (duration: 00m 28s)
10:10 apergos: removed gitblit.log.1 on antimony, it was 58GB
10:03 mark: Reenabling cr2-eqiad:xe-5/3/0
09:53 mark: Disabled cr2-eqiad:xe-5/2/*
09:53 paravoid: issue has been identified as a a problematic PIC (5/2) on cr2-eqiad and is being worked around
09:51 mark: Set VRRP priority 50 on all cr2-eqiad VRRP sessions, making it backup
09:50 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depooling problematic servers (duration: 00m 27s)
09:48 mark: Disabling cr2-eqiad:xe-5/2/1
09:45 mark: Disabling cr2-eqiad:xe-5/{2,3}/0
09:42 paravoid: network troubles within eqiad still being investigated
09:42 paravoid: network troubles; ulsfo was depooled ~20 minutes ago due to backhauling issues
08:31 _joe_: rolling restart of all redis instances on rdb1001 and 1003 (later 1002/4) to fix the upstart bug
07:55 _joe_: restarting redis rdb1003:6380 to test rdb disabling
06:57 ori: ran eventloggingctl stop / eventloggingctl start on eventlog1001
06:18 ori: coal metrics flat since 01:25:00 AM UTC
02:47 Reedy: manually running dsh -g mediawiki-installation -M -F 40 -- "sudo -u mwdeploy /srv/deployment/scap/scap/bin/scap-rebuild-cdbs"
02:45 logmsgbot: l10nupdate@tin LocalisationUpdate failed: Failed to sync-dir 'php-1.27.0-wmf.7/cache/l10n'
02:45 logmsgbot: reedy@tin Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 09m 10s)
02:14 Reedy: killed pre 1.27 l10n cache files from tin:/var/lib/l10nupdate/caches
02:00 logmsgbot: l10nupdate@tin LocalisationUpdate failed: git pull of core failed

2015-11-26

20:59 logmsgbot: aaron@tin Synchronized wmf-config/jobqueue-eqiad.php: Adjust connection timeout and maxPartitionsTry (duration: 00m 34s)
16:50 jynus: backfilling user.user_touched from dbstore1002 to sanitarium on all wikis
16:30 paravoid: switch lvs3001 traffic to lvs3003dd
16:22 paravoid: rebooting lvs3003 with Linux 4.3-1~wmf1
15:58 Krinkle: Deployed patch for T119707
15:49 godog: reenable puppet on graphite1001
14:30 _joe_: run /usr/local/bin/foreachwiki updateSpecialPages.php manually in a screen on terbium
13:27 moritzm: restarted opendj on nembus for export of LDAP data
13:25 moritzm: stopping opendj on nembus for export of LDAP data
11:54 apergos: disabling puppet on neodymium for salt minon tracing
11:35 apergos: re-enabled puppet on neodymium, config testing done for now
11:32 godog: drop local_group_globaldomain_T_mathoid_data and local_group_globaldomain_T_mathoid_request CF stats from graphite
11:25 jynus: manually executing some updateSpecial pages script to try to debug its problems
11:16 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1053 after maintenance (duration: 00m 28s)
11:13 apergos: disabled puppet on neodymium for salt master testing
09:49 moritzm: restarted cassandra nodes on the restbase staging cluster to pick up openjdk security updates
09:38 jynus_: upgrading/rebooting/restarting mysql on db1053 (depooled)
09:02 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1053 for maintenance (duration: 00m 27s)
07:15 ori: rdb100x migrated to redis::instance, meaning they each have two additional redis instances, on port 6380 and 6381, accordingly.
06:46 logmsgbot: ori@tin Synchronized wmf-config/jobqueue-eqiad.php: I3e6021ed3: Update job queue config to use 2 new rdb1003 instances (duration: 00m 28s)
02:00 logmsgbot: l10nupdate@tin LocalisationUpdate failed: git pull of core failed
01:31 awight: updating paymentswiki-staging from 7fb9d6599c3833f0a55500ab77abd2b7ffe74cfe to 988ba71418c370e0365a102a0b87f823671ea1e3
01:03 awight: from f867beecd3a4fb596e2d08c8d2b0e4173ad0b356 to d5afa10383cfd41adbe798c3ed2aa97839be8bc2
00:14 logmsgbot: ori@tin Synchronized wmf-config/jobqueue-eqiad.php: I1e0d0ed8215f: pool rdb1007:6381 (duration: 00m 30s)

2015-11-25

23:21 logmsgbot: krenair@tin Synchronized private/README_BEFORE_MODIFYING_ANYTHING: 334ca105e92aaf7046e244ff39189f3823d31a7d (duration: 00m 32s)
22:14 logmsgbot: demon@tin Finished scap: new MW release, swapping extdist config + msgs (duration: 23m 59s)
21:50 logmsgbot: demon@tin Started scap: new MW release, swapping extdist config + msgs
20:31 gwicke: running `nodetool cleanup` on restbase1007 to make sure that we don't have extra sstables from the 1001 decommision taking up space
19:13 mobrovac: restbase deploy end of 74662c
19:13 moritzm: installed django security updates on stat* and graphite hosts
18:47 robh: removing cr2-ulsfo:xe-1/2/0, Patch ID 1062 as T118171 cancels that link
18:46 mobrovac: restbase deploy start of 74662c
18:19 logmsgbot: bd808@tin Synchronized README: Testing l10nupdate uid fix for T119165 (duration: 00m 28s)
18:04 andrewbogott: restored pdns-recursor on holmium, again
18:00 mobrovac: restbase canary deploy to restbase1001 of 74662c6
17:44 andrewbogott: killing pdns-recursor on holmium
16:53 andrewbogott: killing pdns-recursor on holmium
16:16 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1044 after maintenance (duration: 00m 47s)
15:53 godog: ban grafana kafka dashboard temporarily from graphite
14:57 godog: bounce uwsgi on graphite1001
14:52 godog: stop puppet on restbase1* / restbase2* before https://gerrit.wikimedia.org/r/#/c/254372/
13:09 logmsgbot: hashar@tin Synchronized php-1.27.0-wmf.7/Rakefile: Added Rakefile https://gerrit.wikimedia.org/r/#/c/254423/ (duration: 00m 28s)
11:19 jynus: applying ferm and p_s to db1044 (depooled)
10:57 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1044 for a brief maintenance (duration: 00m 28s)
08:08 logmsgbot: ori@tin Synchronized wmf-config/jobqueue-eqiad.php: I9bea66df: Update job queue config to use 2 rdb1007 instances (duration: 00m 28s)
08:06 akosiaris: update puppet compilers fact database
03:44 Tim: on palladium ran puppet post-merge hook manually after puppet-merge failed with "error: Ref refs/remotes/origin/production is at 380706c5bd2f47d140ffa183dd22fe920927af7e but expected 897486120d645f1c30dcf68bfb3198e2f39ec639"
03:35 logmsgbot: krinkle@tin Synchronized portals/prod/wikipedia.org/index.html: Fix Uncaught TypeError: innerHTML() is not a function (duration: 00m 28s)
03:12 logmsgbot: krinkle@tin Synchronized wmf-config/CommonSettings.php: I4e21eda0f3 (duration: 01m 06s)
02:40 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 06m 50s)
02:00 logmsgbot: l10nupdate@tin LocalisationUpdate failed: git pull of core failed
01:31 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.7/StartProfiler.php: create symlink (duration: 01m 10s)
01:03 mutante: re-armed keyholders on mira
00:46 mutante: starting redis on rdb1007 (it died and was status: dead)
00:40 mutante: tin: fixing l10nupdate UID (997->10002), file ownership
00:37 ori: Migrating rdb1007 from redis::legacy to redis::instance; will involve a service restart.
00:21 ejegg: updated payments-wiki from 5358960d6857f589939c9b86ea37af168a4f8b0b to 7fb9d6599c3833f0a55500ab77abd2b7ffe74cfe
00:20 mutante: rebooting mira
00:14 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/255276/ (duration: 00m 28s)
00:13 awight: Disabled Amazon on donatewiki
00:09 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/254443/ (duration: 00m 28s)

2015-11-24

23:43 mutante: fixinx l10nupdate UID / file ownership on mira
23:40 csteipp: deployed patch for T119309
23:36 ostriches: phabricator: restarted daemons, were complaining about out of sync config
22:07 ejegg: updated SmashPig from 6243d4f172dbfa5e3820f2e1e3a3165efff58bac to b3ee85aab38fdeef3f045ba1733f582e9a5006b2
21:39 logmsgbot: krinkle@tin Synchronized wmf-config/CommonSettings.php: I80d38773057 (duration: 00m 28s)
20:31 mutante: gnt-instance reboot seaborgium.wikimedia.org
20:01 bblack: disabling puppet on neon to avoid monitoring race-condition spam
19:30 moritzm: restarted hadoop workers on analytics1028 to analytics1055 to pick up openjdk security update (plus updates for libpng, nss, nspr, pixbuf and libxml as used by openjdk)
19:27 bblack: misc-cluster: all 4x nodes on new config and pooled
19:18 bblack: misc-cluster: cp1056+7 pooled (new config), cp1069+70 depooled (old config)
18:52 bblack: repooled cp1056, depooled cp1057 (misc 2layer)
18:38 YuviPanda: restarted apache2 on etherpad
18:30 bblack: depooling cp1056 for misc-cluster 2layer work...
17:59 godog: nodetool cleanup on restbase1007
17:20 godog: stop decommissioning restbase1001, bounce cassandra
16:40 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.7/resources/lib/oojs-ui/oojs-ui.js: SWAT: OOjs UI: Backport 4fbbc737c86b500c11bbb471ec1001c50ab8853c gerrit:255032 (duration: 00m 28s)
16:23 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable es labs replica writes for nlwiki, frwiki and eswiki gerrit:255122 (duration: 00m 27s)
16:16 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Fix article-recommender-1 campaign gerrit:255069 (duration: 00m 27s)
16:10 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Third QuickSurveys external survey gerrit:254908 (duration: 00m 30s)
16:08 paravoid: starting pybal on lvs3001 again, testing is over
16:06 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor for all new accounts on eswiki gerrit:250473 (duration: 00m 46s)
15:38 moritzm: restarted analytics1030 to pick up openjdk security update (plus updates for libpng, nss, nspr, pixbuf and libxml as used by openjdk)
15:33 hashar: enabled puppet on both gallium and scandium
15:27 hashar: disabling puppet on both gallium and scandium
14:24 godog: upgrade diamond to 3.5-4 on precise hosts in eqiad
14:22 godog: upgrade diamond to 3.5-4 on precise hosts in ulsfo
13:56 godog: upgrade diamond to 3.5-4 on precise hosts in esams
12:18 paravoid: stopping pybal on lvs3001, switching its traffic to lvs3003
11:48 paravoid: rebooting lvs3003 with linux 4.2.6-1~bpo8+wmf1
10:28 jynus: dropping user_daily_contribs view from labsdb hosts
10:07 godog: stop ircecho on neon temporarily while https://gerrit.wikimedia.org/r/#/c/255094/ applies
08:24 jynus: dbstore1002 (analytics-slave) mysql is not responding, forcing restart
05:58 _joe_: powercycling ms-be1010, xfs kernel lockups
02:26 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 05m 34s)
02:12 ejegg: updated SmashPig from 55c2dbc8cd36cfb89d7e17b75b840ca2e023653e to 6243d4f172dbfa5e3820f2e1e3a3165efff58bac
01:00 ejegg: updated payments-wiki from ecbc49c63b259a4363c51491ddff273e3579f0e9 to 5358960d6857f589939c9b86ea37af168a4f8b0b
00:36 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.7/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: https://gerrit.wikimedia.org/r/#/c/255053/ (duration: 00m 28s)
00:26 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Enable commonswiki for elasticsearch labs replica writes (duration: 00m 28s)
00:15 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Enable wikidatawiki for elasticsearch labs replica (duration: 00m 27s)

2015-11-23

23:55 logmsgbot: maxsem@tin Synchronized portals: (no message) (duration: 00m 28s)
23:30 awight: campaigns reenabled
23:29 awight: reenabling paymentswiki
23:28 awight: reenabling fundraising-jenkins jobs
23:08 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Enable elasticsearch labs replica for enwiki and dewiki (duration: 00m 28s)
23:04 awight: shutting down Jenkins
22:52 awight: disable paymentswiki in config
21:16 hoo: Updated Wikidata's property suggester with data from today's json dump
21:15 jynus: enforcing limits on running queries on all labsdb databases
20:47 jynus: restarting labsdb1002, hopfully for the last time
17:54 jynus: temporarily stopping labsdb1002 to solve replication issues
17:32 Coren: emergency hotfix on labstore1001 (apt-get remove nfsd-ldap) as it appears to not work in all situations.
17:30 ori: restarting hhvm on mw1122 after lock-up; backtrace saved as /tmp/hhvm.32016.bt.
17:28 logmsgbot: thcipriani@tin Finished scap: SWAT: Update CentralNotice gerrit:254865 (duration: 26m 31s)
17:02 logmsgbot: thcipriani@tin Started scap: SWAT: Update CentralNotice gerrit:254865
16:44 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: noindex user namespace on en.wikipedia.org gerrit:237330 (duration: 00m 28s)
16:43 jynus: restarting mysqld on labsdb1003 again
16:38 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: End first mobile QuickSurveys campaign gerrit:254831 (duration: 00m 28s)
16:32 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.7/includes/specials/SpecialWatchlist.php: SWAT: Special:Watchlist: Add user preference to "Show last" options, fix float comparison gerrit:254880 (duration: 00m 29s)
16:10 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Enable article-recommender-1 campaign as default gerrit:254117 (duration: 00m 52s)
15:56 paravoid: rebooted lvs3003 back with (an updated) 3.19
15:53 _joe_: live-resizing /var/lib/wdqs on wdqs1002
15:15 Coren: NFS server on labstore1001 restarted with no issues.
15:11 Coren: Restarting NFS daemon on labstore1001 w/ the new LDAP shim.
14:53 paravoid: powercycling cp1053, unresponsive, blank console
14:53 bblack: depooling cp1053
14:50 ottomata: removing erbium udp2log varnishncsa instances from all caches
14:38 paravoid: rebooting lvs3003 with new kernel
13:18 jynus: restarting mysql on labsdb1003, unresponsive
11:05 _joe_: stopping apache on mw2051 for testing
08:58 godog: restbase1001 nodetool decommission
08:57 mobrovac: mathoid deploying ff03d56
08:12 moritzm: restarted gitblit on antimony
02:19 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 05m 57s)
01:27 Jamesofur: end enwiki test election from cli to avoid confusion with real election

2015-11-22

21:30 Reedy: restarting stuck Jenkins
02:16 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 05m 22s)

2015-11-21

19:14 ejegg|away: updated fundraising tools from d7253eb1c6c9478ce2cf8760279d2ad1b7f0a132 to ebed29c0eccf38c812b20a957b3487a15bfa9cbc
18:43 ejegg|away: rolled back SmashPig d7534bde8c247a1e877a4bcd768b9963f89666d7 --> 55c2dbc8cd36cfb89d7e17b75b840ca2e023653e
18:35 ejegg|away: updated SmashPig from 55c2dbc8cd36cfb89d7e17b75b840ca2e023653e to d7534bde8c247a1e877a4bcd768b9963f89666d7
03:19 ori: SecurePoll arbcomlist.php run for enwiki (requested by Jamesofur) complete.
02:20 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 05m 18s)
00:50 ori: update-ubuntu-mirror on carbon completed successfully ; going to re-enable icinga-wm
00:46 ori: rampant puppet failures appear to be the result of a Hash Sum mismatch error during apt-get update; attempting to fix by running sudo -u mirror /usr/local/sbin/update-ubuntu-mirror on carbon.
00:37 ori: disabled puppet on neon for one hour and killed icinga-wm to silence a tide of puppet failure alerts
00:24 logmsgbot: demon@tin Synchronized .arcconfig: no-op, for completeness (duration: 00m 31s)

2015-11-20

21:45 logmsgbot: csteipp@tin Synchronized php-1.27.0-wmf.7/includes/User.php: Deploy fix for T119021 (duration: 00m 28s)
18:55 ori: resuming interrupted arbcomlist invocation on terbium
16:51 _joe_: decommissioning mw1041: removed facts and certs, removed from conftool and pybal
16:44 logmsgbot: reedy@tin Synchronized wmf-config/throttle.php: Add throttle config for Viquimarató nit digital/2015 (duration: 00m 28s)
16:17 _joe_: depooled mw1041
15:41 Reedy: running sync-common on mw1041
15:32 _joe_: powercycling mw1041
15:26 logmsgbot: reedy@tin Synchronized wmf-config/throttle.php: Update throttle for event tomorrow (duration: 00m 28s)
14:35 godog: swift eqiad-prod: set ms-be1019 / ms-be1020 / ms-be1021 weight 3000
10:06 ori: Running SecurePoll's arbcomlist.php on terbium with performance fix from https://gerrit.wikimedia.org/r/#/c/254375/ , per request from Jamesofur. Looks to be behaving well, can be killed if it acts up.
07:44 _joe_: rebooted pybal-test2001
07:36 _joe_: powercycled mw1041
04:55 logmsgbot: twentyafterfour@tin Synchronized multiversion/checkoutMediaWiki.php: inconsequential sync to test mirroring (duration: 02m 14s)
04:41 logmsgbot: twentyafterfour@tin Synchronized multiversion/checkoutMediaWiki.php: deploying I2ec91aa33e05c2dc367db8a6f6ba56be5397906b to test scap mirroring (duration: 00m 20s)
04:36 twentyafterfour: deploying scap on tin
02:34 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 10m 47s)
01:31 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/resources/src/mediawiki.widgets/mw.widgets.TitleWidget.js: Fix conflicting configuration name in TitleInputWidget (duration: 00m 19s)
01:31 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/includes/widget/TitleInputWidget.php: Fix conflicting configuration name in TitleInputWidget (duration: 00m 20s)
01:30 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/resources/lib/jquery.i18n/src/jquery.i18n.language.js: Unbreak IE8 support in jquery.i18n (duration: 00m 25s)
01:29 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/extensions/Echo/includes/formatters/BasicFormatter.php: unstub wgLang to fix type hint errors (duration: 00m 20s)
01:11 logmsgbot: bd808@tin Synchronized wmf-config/InitialiseSettings.php: Reduce sampling rate for QuickSurveys survey (28c69d2) (duration: 00m 20s)
00:37 awight: Updating fundraising/crm from e486ba424092866cdd8a15d26aeb17a412cd631a to f867beecd3a4fb596e2d08c8d2b0e4173ad0b356
00:18 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-common.php: SWAT https://gerrit.wikimedia.org/r/253933 and https://gerrit.wikimedia.org/r/254070 (duration: 00m 19s)
00:17 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: SWAT https://gerrit.wikimedia.org/r/253933 and https://gerrit.wikimedia.org/r/254070 (duration: 00m 19s)
00:13 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.7/extensions/CirrusSearch/: SWAT https://gerrit.wikimedia.org/r/254320 and https://gerrit.wikimedia.org/r/254319 (duration: 00m 20s)
00:02 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: SWAT https://gerrit.wikimedia.org/r/#/c/254322/ (duration: 00m 20s)

2015-11-19

23:35 awight: Update fundraising/tools from c6315eddf6f495e45401844d5f61bb240464c1e5 to d7253eb1c6c9478ce2cf8760279d2ad1b7f0a132
22:31 csteipp: deployed followup for T109724
22:21 csteipp: deployed patch for T118682
22:13 csteipp: deployed patch for T118032
22:11 csteipp: deployed patch for T97897
21:31 mutante: magnesium - apt-get upgrade
21:13 mutante: racktables moved to krypton and upgraded to 0.20.6
20:28 mutante: racktables - in migration - brb
20:22 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.7/extensions/QuickSurveys/: Deploy https://gerrit.wikimedia.org/r/#/c/254184/ (duration: 00m 19s)
20:19 awight: push paymentswiki config change, T115746
20:13 bblack: upgrading pybal to 1.13.1 on lvs400[12] and failing back traffic
19:51 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1044 after maintenance (duration: 00m 22s)
19:24 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: wikipedia wikis to 1.27.0-wmf.7
18:53 logmsgbot: aude@tin Synchronized php-1.27.0-wmf.7/extensions/Wikidata: Adjust watchlist filter for core change (duration: 00m 30s)
16:24 godog: repool restbase1002
16:19 cmjohnson1: replacing pem 0 cr1-eqiad
14:55 godog: roll restart restbase in eqiad
14:41 godog: roll-restart restbase on aqs100*
14:40 moritzm: uploaded linux-meta 1.3 to carbon
14:39 moritzm: uploaded linux 3.19.3-9 to carbon
14:14 godog: roll-restart restbase in codfw
12:39 bblack: failing over ulsfo traffic to lvs400[34] (for 1.13.1 testing)
11:23 hashar: restarted zuul-merger on gallium. We now have two instances running (gallium and scandium)
11:21 hashar: stopped zuul-merger on gallium, started the one on scandium
11:00 jynus: modifying filters on all labs db replicas (sanitarium), small amounts of lag could be created
10:42 godog: hhvm-dump-debug --full and bounce hhvm on mw1147
10:40 godog: hhvm-dump-debug --full and bounce hhvm on mw1131
10:13 godog: disable puppet on deployment_target:restbase/deploy
10:08 hashar: going to restart Jenkins a few times
09:53 godog: delete old mobileapps CF stats from graphite1001 / graphite2001
06:19 YuviPanda: remove role::labs::instance puppetClass from all labs instances
02:23 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.6/cache/l10n: l10nupdate for 1.27.0-wmf.6 (duration: 06m 58s)
01:31 awight: updating fundraising/crm from 67b855cbce358d215b8025da6fe7031bf3d221d9 to e486ba424092866cdd8a15d26aeb17a412cd631a
01:30 twentyafterfour: finished phabricator upgrade
00:29 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/extensions/Flow/: SWAT: better permissions errors for opt-in (duration: 00m 21s)
00:19 RoanKattouw: Ran namespaceDupes.php on eswiktionary
00:13 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: T118844, T119006 (duration: 00m 19s)
00:12 logmsgbot: catrope@tin Synchronized portals: SWAT (duration: 00m 19s)

2015-11-18

23:51 yurik: git deploy sync latest graphoid
23:49 bblack: upgrading pybal to 1.13.1 on lvs300[34]
23:45 bblack: upgrading pybal to 1.13.1 on lvs400[34]
22:44 awight: Updating fundraising/crm from 230cebb4a3b46a75fef758d46f4db22de29286cb to 67b855cbce358d215b8025da6fe7031bf3d221d9
22:38 mutante: powercycled analytics1030 - no ssh response, mgmt console login timed out
21:15 subbu: finished deploying parsoid sha e0a4fc91
21:10 subbu: synced code; restarted parsoid on wtp1003 as a canary
21:06 subbu: starting parsoid deploy
20:43 bblack: upgrading pybal to 1.13.1 on lvs200[456]
20:09 logmsgbot: maxsem reset User:人神之间 email (confirmed in person with Matanya)
19:59 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.7
18:18 andrewbogott: repooled mw1160. And that’s all of them!
18:09 andrewbogott: repooled mw1159, depooled mw1160
17:59 andrewbogott: repooling mw1157, depooled mw1159
17:48 andrewbogott: repooled mw1155, depooled 1157 (I did 56 and 58 list night)
17:36 mutante: releases.wikimedia.org now enforcing https protocol
17:30 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.6/includes/db/loadbalancer/LoadBalancer.php: I03e1386b: Make getLaggedSlaveMode() use reuseConnection() as needed (T118162) (duration: 00m 19s)
17:26 andrewbogott: repooled mw1154, depooled mw1155
17:17 andrewbogott: repooled mw1153, depooling mw1154
17:09 andrewbogott: depooled mw1153 for https://phabricator.wikimedia.org/T118888
17:07 thcipriani: updateCollation.php for srwiki 1670608 rows processed
17:05 godog: restart restbase2002-a cassandra instance, bootstrap failed with stream error
16:23 bblack: upgrading pybal on lvs1006 (1.13.1)
16:23 _joe_: upgrading pybal on lvs1006 (1.13.1)
16:21 bblack: upgrading pybal on lvs1005 (1.13.1)
16:21 bblack: upgrading pybal on lvs1004 (1.13.1)
16:16 thcipriani: updateCollation.php for srwiktionary complete 82713 rows processed
16:12 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wgCategoryCollation to "uca-sr" at srwiki and srwiktionary gerrit:253903 (duration: 00m 20s)
16:11 bblack: upgrading pybal on lvs1004 (1.13.1)
16:07 mobrovac: restbase cassandra dropping old mobileapps keyspaces
16:05 mobrovac: restbase end deploy of 6b5a602
15:56 mobrovac: restbase start deploy of 6b5a602
15:50 bblack: upgrading pybal to 1.13 on lvs100[56] (with service clean)
15:37 jynus: warming up srwiki.page and srwiki.categorylinks on s3 slaves
15:29 mobrovac: restbase canary deploy to rb1001 of 6b5a602
15:28 bblack: upgrading pybal to 1.13 on lvs1004, with cleared tables
15:23 _joe_: uploaded pybal 1.13 to jessie-wikimedia
15:14 hashar: stopping zuul-merger on scandium. Lacks ssh private key to reach gerrit
14:58 hashar: Pooled zuul-merger instance on scandium.eqiad.wmnet . In case it screw up one has to stop the zuul-merger service on scandium.
14:57 godog: bounce mobileapps on scb1001/scb1002
14:47 mobrovac: mobileapps deploying 151c312
13:55 bblack: disabling pybal on lvs100[123] - fallback to lvs100[456]
12:00 hashar: reenabled puppet on gallium. Havent pooled zuul-merger yet
11:57 hashar: disabled puppet on gallium
11:56 hashar: pooling new zuul-merger on scandium
11:44 mark: Set OSPF/OSPFv3 metric 320 (instead of 360) on new Zayo link between cr2-eqiad:xe-5/2/3 and cr2-codfw:xe-5/0/1
10:49 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Decreasing weight of db1035 due to connection exhaustion (duration: 00m 19s)
10:31 hashar: gallium: apt-get upgrade
10:30 hashar: upgrading Zuul on gallium (CI interruption for a minute)
09:48 jynus: several db maintenance tasks on db1044 -optimization, upgrade- expect lag (node is depooled)
09:25 godog: reimage restbase2002
09:11 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1027, depool db1044 for maintenance (duration: 01m 08s)
09:09 _joe_: restarted HHVM on mw1095, stuck in treadmill::startrequest() on a contended lock
06:31 _joe_: rsyncing terbium homes from rutherfordium
02:39 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.6/cache/l10n: l10nupdate for 1.27.0-wmf.6 (duration: 10m 10s)
01:46 andrewbogott: repooling mw1156
01:43 yurik_: synced graphoid service
01:40 andrewbogott: updated kernel on mw1156, now rebooting.
01:35 andrewbogott: repooling mw1158, depooling mw1156
01:22 andrewbogott: rebooting mw1158
01:19 andrewbogott: upgrading mw1158 to 3.13.0-62
00:26 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings.php: wgRCWatchCategoryMembership true on group0 wikis (duration: 00m 27s)
00:04 logmsgbot: maxsem@tin Synchronized portals/: SWAT (duration: 00m 32s)

2015-11-17

23:43 mutante: deactivated wekipedia.com
23:40 YuviPanda: remove realm variable from all instances in ldap
23:18 mutante: deactivated wikiartpedia.[biz|co|info|me|mobi|net|org]
23:17 awight: updated paymentswiki from 4baa5a66a4510414d6b43b59f1b1cda2341c17fd to ecbc49c63b259a4363c51491ddff273e3579f0e9
23:13 mutante: deactivated border-wikipedia.de
23:01 mutante: deactivated wikimedia.xyz
22:57 mutante: deactivated visualwikipedia.[com|net]
22:56 mutante: deactivated softwarewikipedia.[com|net|org]
22:54 mutante: deactivated indiawikipedia.com
22:52 mutante: deactivated vikipedi.com.tr, vikipedia.com.tr
22:38 awight: updated fundraising/tools from 182f217ea20a496fe0f9737447d04ed9a3bccde8 to c6315eddf6f495e45401844d5f61bb240464c1e5
22:23 mutante: deactivated wikimaps.com, wikimaps.net, wikimaps.org
22:20 mutante: deactivated wikimedia.biz, webhostingwikipedia.com
22:18 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.7
21:07 logmsgbot: twentyafterfour@tin Finished scap: sync new branch 1.27.0-wmf.7 and enable for testwiki (duration: 29m 40s)
20:39 mutante: deleted sitemap.wikimedia.org (T101486)
20:38 logmsgbot: twentyafterfour@tin Started scap: sync new branch 1.27.0-wmf.7 and enable for testwiki
20:31 logmsgbot: twentyafterfour@tin scap failed: CalledProcessError Command 'cp -r "/tmp/scap_l10n_2994655917"/* "/srv/mediawiki-staging/php-1.27.0-wmf.7/cache/l10n"' returned non-zero exit status 1 (duration: 00m 42s)
20:30 logmsgbot: twentyafterfour@tin Started scap: sync new branch 1.27.0-wmf.7 and enable for testwiki
20:26 logmsgbot: twentyafterfour@tin scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_4014066321" --threads=4 --lang en --quiet' returned non-zero exit status 1 (duration: 01m 30s)
20:25 logmsgbot: twentyafterfour@tin Started scap: sync new branch 1.27.0-wmf.7 and enable for testwiki
18:54 andrewbogott: depooled mw1158 due to kernel errors. https://phabricator.wikimedia.org/T118888
18:10 cmjohnson1: swapping failing disk 32:8 on db1027..will cause icinga alert (jynus)
18:02 bblack: upgrading and re-enabling pybal on lvs300[12].esams.wmnet (1.10 -> 1.12)
17:54 yurik: deployed latest graphoid service
16:45 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wgCategoryCollation for bs.wikipedia.org gerrit:250862 (duration: 00m 26s)
16:42 bblack: upgrading / re-enabling pybal on lvs200[123].codfw.wmnet (1.10 -> 1.12)
16:30 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Set import sources on en.wikiversity gerrit:250709 (duration: 00m 27s)
16:25 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Namespace configuration on ru.wikipedia.org gerrit:252911 (duration: 00m 26s)
16:19 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: SWAT: Editaton contra la violencia hacia las mujeres throttle rule gerrit:253541 (duration: 00m 27s)
16:10 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor for 50% of new accounts on eswiki gerrit:250472 (duration: 00m 26s)
16:02 hashar: Zuul-merger deployment aborted / uncomplte (I hate puppet)
16:02 bblack: re-enabling pybal on lvs400[12] (upgraded to 1.12)
15:36 cmjohnson1: reseating pem 0 on cr1-eqiad
15:30 mobrovac: restbase end deploy of e749f6ff
15:20 mobrovac: restbase start deploy of e749f6ff
14:59 bblack: stopping pybal on lvs300[12], fallback to lvs300[34] (pybal 1.10 vs 1.12)
14:55 godog: swift eqiad-prod: set ms-be1019 / ms-be1020 / ms-be1021 weight 1500
14:51 bblack: stopping pybal on lvs200[123], fallback to lvs200[456] (pybal 1.10 vs 1.12)
14:37 bblack: stopping pybal on lvs400[12], fallback to lvs400[34] (pybal 1.10 vs 1.12)
12:32 akosiaris: depool restbase1002
09:54 godog: nodetool decommission on restbase2002
06:18 gwicke: restbase cassandra: Increased tombstone_threshold from 0.02 to 0.1 for wikipedia and wikimedia html and data-parsoid to reduce single-sstable compaction rate.
02:21 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.6/cache/l10n: l10nupdate for 1.27.0-wmf.6 (duration: 06m 26s)
00:21 logmsgbot: krinkle@tin Synchronized wmf-config/jobqueue-codfw.php: (no message) (duration: 00m 26s)
00:16 logmsgbot: legoktm@tin Synchronized wmf-config/CommonSettings.php: REL1_26 knocks on the door - https://gerrit.wikimedia.org/r/#/c/252399/ (duration: 00m 27s)

2015-11-16

22:47 logmsgbot: hoo@tin Synchronized php-1.27.0-wmf.6/includes/Revision.php: Don't claim model validation failed if the content couldn't be loaded (duration: 00m 26s)
22:45 gwicke: rolling restbase restart to apply https://gerrit.wikimedia.org/r/#/c/253468/
21:52 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: Don't log sysops with short passwords (duration: 00m 27s)
21:47 mobrovac: restbase end deployment of 54bc0517
21:38 mobrovac: restbase start deployment of 54bc0517
21:26 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: Log username of users with short passwords (duration: 00m 26s)
21:23 subbu: finished deploying parsoid 3a6f3b9e
21:21 mobrovac: restbase canary deploy to restbase1001 of 54bc0517
21:07 subbu: synced fresh code; restarting parsoid on wtp1005 as a canary
21:06 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: Log privileged users with short passwords. Set minimum password length of 10 for wikitech. (duration: 00m 26s)
21:03 subbu: starting parsoid deploy
19:52 anomie: Running cleanupBlocks.php on arwiki, eswikibooks, eswikinews, and testwiki for phab:T118625
19:23 RoanKattouw: Deployed patch for T116095
16:14 logmsgbot: thcipriani@tin Synchronized wmf-config/squid.php: SWAT: wgHTCPRouting: use separate address for upload gerrit:249121 (duration: 00m 26s)
16:07 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable first QuickSurveys survey gerrit:253038 (duration: 00m 26s)
14:40 godog: carbon daemon bounce on graphite1001
13:43 bblack: reinstalling lvs100[45]
13:39 jynus: dropping ipblocks_old table from all wikis
13:19 bblack: reinstalling lvs1006
11:21 jynus: defragementing, upgrading, general maintenance to mysql at db1027. Expect lag, etc. (it is depooled)
10:25 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1015, depool db1027 for maintenance (duration: 00m 49s)
10:03 jynus: performing schema change on wikishared.bounce_records (x1)
09:39 godog: bootstrap restbase2001-c cassandra instance
09:34 jynus: adding missing tables to vewikimedia https://phabricator.wikimedia.org/rMW7389d7c69035c968553fbd2eaf1cf47038c4577c
09:19 _joe_: reimaging terbium
09:16 godog: swift eqiad-prod: add ms-be1019 / ms-be1020 / ms-be1021 weight 500
08:56 _joe_: rsyncing files from terbium to rutherfordium
08:10 akosiaris: powered off plutonium, removed it from puppet and salt. context: https://phabricator.wikimedia.org/T118586
02:22 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.6/cache/l10n: l10nupdate for 1.27.0-wmf.6 (duration: 06m 29s)

2015-11-15

18:03 jynus: disabling puppet, restarting mysql on es2008, es2009, es2010 for configuration check/upgrade
16:41 jynus: stopping eventlogging_sync for manual resync on dbstore1002 and db1047
15:33 logmsgbot: reedy@tin Synchronized wmf-config/: noop after chmod (duration: 00m 27s)
14:31 logmsgbot: reedy@tin Synchronized wmf-config/InitialiseSettings.php: revert last, no file created on fluorine (duration: 00m 27s)
14:28 logmsgbot: reedy@tin Synchronized wmf-config/InitialiseSettings.php: Temporarily enable error log. Will revert in a few minutes (duration: 00m 27s)
14:19 logmsgbot: reedy@tin Synchronized php-1.27.0-wmf.6/extensions/BounceHandler/: Pull in a few fixes and improvements for BounceHandler (duration: 01m 12s)
12:17 jynus: restarting mysql at db1015 to finish upgrade/maintenance process
12:10 jynus: uploading upgraded mariadb package for trusty, too
09:02 _joe_: rebooting krypton
02:21 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.6/cache/l10n: l10nupdate for 1.27.0-wmf.6 (duration: 06m 07s)
01:39 logmsgbot: reedy@tin Purged l10n cache for 1.27.0-wmf.1
01:36 ostriches: deploying scap master@54d93b3

2015-11-14

23:31 Reedy: dsh delete php-1.26wmf22 23 and 24
23:06 Reedy: dsh delete php-1.27.0-wmf.4/cache/l10n/*
23:06 Reedy: dsh delete php-1.27.0-wmf.3/cache/l10n/*
23:04 Reedy: dsh delete php-1.27.0-wmf.2/cache/l10n/*
23:04 Reedy: dsh delete php-1.27.0-wmf.1/cache/l10n/*
23:03 Reedy: dsh delete php-1.26wmf24/cache/l10n/*
23:01 Reedy: dsh delete php-1.26wmf23/cache/l10n/*
23:00 Reedy: dsh delete php-1.26wmf22/cache/l10n/*
22:58 Reedy: dsh delete php-1.26wmf21
22:57 Reedy: dsh delete php-1.26wmf20
22:57 Reedy: dsh delete php-1.26wmf19
22:56 Reedy: dsh delete php-1.26wmf18
22:55 Reedy: dsh delete php-1.26wmf17
22:54 Reedy: dsh delete php-1.26wmf16
22:51 logmsgbot: reedy@mira Synchronized phpunit.xml: (no message) (duration: 00m 29s)
22:49 logmsgbot: reedy@tin Synchronized php-1.26wmf24/: purge l10n cache... test if deletes are propogated (duration: 01m 12s)
22:43 Reedy: deleted more old l10ncaches from tin staging 1.27 wmf1-4
22:37 logmsgbot: reedy@tin Synchronized phpunit.xml: (no message) (duration: 00m 30s)
22:26 logmsgbot: reedy@tin Synchronized php-1.26wmf24: remove old localisation cache (duration: 01m 38s)
22:22 logmsgbot: reedy@tin Synchronized php-1.26wmf23: remove old localisation cache (duration: 01m 40s)
22:10 logmsgbot: reedy@tin Synchronized phpunit.xml: noop test (duration: 01m 03s)
20:51 Reedy: Deleted php-1.26wmf11, php-1.26wmf12 and php-1.26wmf13 from mira /srv/mediawiki-staging
20:40 logmsgbot: reedy@tin Synchronized php-1.27.0-wmf.6/extensions/BounceHandler/: Email header logging on unsubscribe (duration: 00m 53s)
13:04 anomie: Ran cleanupBlocks.php on ruwiki for phab:T118625
02:20 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.6/cache/l10n: l10nupdate for 1.27.0-wmf.6 (duration: 05m 58s)
01:58 ori: restarted HHVM on all app servers

2015-11-13

23:30 logmsgbot: ori@tin Synchronized rpc/RunJobs.php: I5e15ec9fb: Disable ChronologyProtector for rpc/RunJobs.php (duration: 00m 30s)
23:27 awight: Reenabled fundraising campaigns outside of France
23:09 awight: Disabled Fundraising campaigns due to PayPal outage
21:23 logmsgbot: maxsem@tin Synchronized portals/: Did scap errors go away? (duration: 00m 30s)
21:17 YuviPanda: bounced etherpad again
20:39 logmsgbot: maxsem@tin Synchronized portals/: Did scap errors go away? (duration: 00m 29s)
20:38 logmsgbot: maxsem@tin Synchronized portals/: Did scap errors go away? (duration: 00m 29s)
20:35 logmsgbot: maxsem@tin Synchronized portals/: Did scap errors go away? (duration: 00m 30s)
20:30 logmsgbot: maxsem@tin Synchronized portals/: Bump to master (duration: 00m 39s)
20:19 robh: cr2-xe-1/3/0 optic install completed (detects fine)
20:18 robh: cr1-ulsfo:xe-0/0/2 optic install completed (detects fine)
20:17 robh: cr1-ulsfo:xe-0/0/2 (spare port) xfp optic detects fine, now installing other spare into cr2-xe-1/3/0
20:14 robh: installing xfp optics into cr1-ulsfo:xe-0/0/2 (spare port)
18:04 bblack: added neodymium to salt term in common-infrastructure4 on cr[12]-eqiad
17:37 csteipp: applied and reverted patch for T118032 (lots of exceptions on deploy)
16:16 jynus: restarting mysql on db2056, db2063 and db2064 for regular upgrade
15:16 godog: bootstrap restbase2001-b cassandra instance
15:03 jynus: restarting db2058, db2065 for regular mysql upgrade
14:43 jynus: restarting db2059, db2066 for regular mysql upgrade
11:42 jynus: restarting db2060 mysql for upgrade
09:20 jynus: performing table optimization on db1015, expect some lag while depooled
08:37 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1035 at 100%, depool db1015 (duration: 01m 31s)
06:58 _joe_: trying to manually syncing the ubuntu mirrors, since they are corrupted at the moment
02:26 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.6/cache/l10n: l10nupdate for 1.27.0-wmf.6 (duration: 05m 21s)
01:27 YuviPanda: bounced mw1045 HHVM for ebernhardson
01:15 logmsgbot: catrope@tin Finished scap: I totally forgot that the SWAT contained i18n changes (duration: 26m 46s)
00:48 logmsgbot: catrope@tin Started scap: I totally forgot that the SWAT contained i18n changes
00:28 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.6/extensions/QuickSurveys: SWAT (duration: 00m 30s)
00:24 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.6/extensions/Flow: SWAT (duration: 00m 32s)
00:23 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.6/extensions/CentralNotice: SWAT (duration: 00m 31s)
00:16 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Remove hardcoded max upload size for UploadWizard (duration: 00m 30s)
00:15 awight: Banner impressions loader service restored
00:14 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Turn off language detection user test for CirrusSearch (duration: 00m 30s)
00:04 awight: update DjangoBannerStats from 08ffecf412a6a9ff3e173dded6814092834c7d1e to 26b48add5c6f70b395a3195c401a12612c626796

2015-11-12

23:41 awight: update DjangoBannerStats from 9cc2acba7f6b46e243ada0fac6c48c9a49460a07 to 08ffecf412a6a9ff3e173dded6814092834c7d1e
23:05 logmsgbot: bd808@tin Synchronized wmf-config/logging.php: Monolog: wrap channel handlers in a WhatFailureGroupHandler (b08aaf4) (duration: 00m 29s)
22:46 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: wikipedia wikis to 1.27.0-wmf.6
22:42 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.6/extensions/TranslationNotifications/TranslationNotificationJob.php: hotfix T116960 (duration: 00m 30s)
21:21 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.6/resources/lib/jquery.i18n/src/jquery.i18n.language.js: T118242 - Unbreak IE8 (duration: 00m 30s)
21:10 chasemp: reboot analytics1040 it locked up
19:37 awight: Update DjangoBannerStats from 12e819a04a40ee6fab5dd55fcaf072661df31106 to 9cc2acba7f6b46e243ada0fac6c48c9a49460a07
19:24 awight: banner impressions job disabled
19:11 awight: Disabled WMDE and WMF Fundraising campaigns to do banner impression maintenance
19:05 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: SWAT: Set throttle exception for University of Haifa wiki event gerrit:252010 (duration: 00m 29s)
18:21 logmsgbot: maxsem@tin Synchronized portals/: The rest of portals (duration: 00m 31s)
17:25 subbu: finished parsoid deploy sha 392e25eb
17:02 subbu: starting parsoid deploy
16:29 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Deploy QuickSurveys to English Wikipedia gerrit:252606 (duration: 00m 29s)
16:28 ostriches: gerrit: reloading replication plugin to pick up new config
16:11 andrewbogott: depooled T118469. https://phabricator.wikimedia.org/T118469
16:08 bd808: cleaned up stale branch checkouts on mw1041: php-1.26wmf16 php-1.26wmf17 php-1.26wmf18 php-1.26wmf19 php-1.26wmf20 php-1.26wmf21
16:02 mobrovac: restbase end deploy of e5e45d11ad3f
15:48 mobrovac: restbase start deploy of e5e45d11ad3f
15:46 mobrovac: restbase canary deploy on rb1001 of e5e45d11ad3f
15:39 jynus: powercycling mw1041
15:30 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1035 after maintenance (duration: 00m 29s)
10:11 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.6/extensions/UniversalLanguageSelector: I9206a79: Only use jQuery.tipsy for undo popover (duration: 00m 31s)
09:53 jynus: rebooting/upgrading/restarting mariadb on db1035 after maintenance finished
09:52 jynus: uploading new mariadb package linked to openssl instead of yassl
09:44 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.5/extensions/UniversalLanguageSelector: I9206a79: Only use jQuery.tipsy for undo popover (duration: 01m 15s)
09:39 moritzm: corrected apt sources on db2057, db2058, db2062, db2065, db2066, db2067, db2068, db2069
08:42 moritzm: restarted salt-master on palladium
07:35 _joe_: restarting pybal on lvs1004
02:23 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 06m 56s)
01:02 gwicke: restbase: finished full deploy of restbase-deploy ce4abe784
00:50 gwicke: restbase: starting full deploy of restbase-deploy ce4abe784
00:37 gwicke: restbase: canary deploy of restbase-deploy ce4abe784 on restbase1001
00:32 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Fix typo in QuickSurveys config (duration: 00m 31s)

2015-11-11

23:48 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.6
23:44 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.6/includes/jobqueue/aggregator/: sync https://gerrit.wikimedia.org/r/#/c/252588/ (duration: 00m 32s)
21:13 subbu: finished deploying parsoid sha 7ca999c1
21:06 subbu: synced new code + restarted parsoid on wtp1001 as canary; monitoring graphs for a little bit
21:02 subbu: starting parsoid deploy
20:52 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.5
19:51 hoo: Started rebuildItemsPerSite for wikidatawiki on mw1152. Feel free to kill, should it cause troubles.
19:49 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.6
19:37 twentyafterfour: deploying 1.27.0-wmf.6 to group1
19:09 YuviPanda: nfs-exports on labstore1001 failed because of http failure of wikitech
19:05 YuviPanda: started nfs-exports service on labstore1001
18:18 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Repool db2067 after maintenance (duration: 00m 29s)
16:46 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.5/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Restore satisfaction schema and fix the performance issue that it had part II gerrit:252349 (duration: 00m 30s)
16:45 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.5/extensions/WikimediaEvents/WikimediaEvents.php: SWAT: Restore satisfaction schema and fix the performance issue that it had part I gerrit:252349 (duration: 00m 30s)
16:34 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.6/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Restore satisfaction schema and fix the performance issue that it had part II gerrit:252347 (duration: 00m 30s)
16:33 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.6/extensions/WikimediaEvents/WikimediaEvents.php: SWAT: Restore satisfaction schema and fix the performance issue that it had part I gerrit:252347 (duration: 00m 40s)
16:17 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: First QuickSurvey for reader segmentation research - external survey gerrit:251133 (duration: 00m 30s)
16:02 mutante: uploaded jenkins 1.625.2 to apt.wm.org, upgrading on gallium
15:47 moritzm: restarted kafka broker on kafka1022
15:34 moritzm: stopping kafka broker on kafka1022 (to enable ferm)
15:31 moritzm: restarted kafka broker on kafka1020
15:25 moritzm: stopping kafka1020 to enable ferm
15:22 moritzm: restarted kafka1018 to enable ferm
15:19 moritzm: stopping kafka1018 to enable ferm
15:04 moritzm: disabled puppet on kafka1018, kafka1020, kafka1022 (for enabling ferm)
14:18 godog: start cassandra instance a on restbase2001
13:54 mobrovac: restbase deployed 0d961a2 (deploy end)
13:42 mobrovac: restbase deploying 0d961a2
11:28 godog: reimage restbase2001.codfw.wmnet - take #2
10:16 jynus: disabling puppet and restarting mysql on db2067
10:16 godog: reimage restbase2001.codfw.wmnet
10:07 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Depool db2067 for maintenance (not db1067) (duration: 00m 30s)
10:05 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1067 for maintenance (duration: 00m 30s)
09:57 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.6/extensions/OAuth/: really fix T118372 this time (duration: 00m 29s)
09:43 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.6/extensions/OAuth/: oauth login broken by php error. This fixes T118372 (duration: 00m 31s)
08:51 jynus: importing wmf-mariadb10_10.0.22-1_amd64.deb wmf-mysql57_5.7.9-1_amd64.deb on carbon
04:19 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: rm some old log channels (duration: 00m 31s)
03:23 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.6/includes/cache/MessageBlobStore.php: Logging for T93800 (duration: 00m 31s)
02:51 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.5/includes/Title.php: Deploy I47517021471 which was merged earlier today but forgotten (duration: 01m 58s)
02:36 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 07m 01s)
01:48 mutante: repooled mw1083 (T116184)
00:58 mutante: added jenkins_1.625.1 to APT repo. precise-wikimedia
00:50 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.6
00:38 logmsgbot: twentyafterfour@tin Finished scap: grr: sync 1.27.0-wmf.6 (duration: 55m 05s)

2015-11-10

23:42 logmsgbot: twentyafterfour@tin Started scap: grr: sync 1.27.0-wmf.6
23:41 logmsgbot: twentyafterfour@tin scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="cawikibooks" --list-file="/srv/mediawiki-staging/wmf-config/extension-list" --output="/tmp/tmp.7rNhS6PIRi" ' returned non-zero exit status 1 (duration: 00m 09s)
23:41 logmsgbot: twentyafterfour@tin Started scap: grr: sync 1.27.0-wmf.6
23:40 logmsgbot: twentyafterfour@tin scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="cawikibooks" --list-file="/srv/mediawiki-staging/wmf-config/extension-list" --output="/tmp/tmp.yIUW9abUwO" ' returned non-zero exit status 1 (duration: 00m 15s)
23:39 logmsgbot: twentyafterfour@tin Started scap: trying again: sync 1.27.0-wmf.6
23:23 ori: ran sudo chgrp -R wikidev /srv/mediawiki-staging/php-1.27.0-wmf.6 on tin
22:50 ejegg: rolled back paymentswiki from 3730cff201e314fc54ad76fbc54f33f5b032def5 to 4baa5a66a4510414d6b43b59f1b1cda2341c17fd
22:43 logmsgbot: twentyafterfour@tin scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="cawikibooks" --list-file="/srv/mediawiki-staging/wmf-config/extension-list" --output="/tmp/tmp.sEpytaqlt1" ' returned non-zero exit status 1 (duration: 01m 42s)
22:41 logmsgbot: twentyafterfour@tin Started scap: testwiki to 1.27.0-wmf.6
21:51 hashar: installed Zuul package on scandium.eqiad.wmnet . zuul-merger not running
21:33 ejegg: updated paymentswiki from 4baa5a66a4510414d6b43b59f1b1cda2341c17fd to 3730cff201e314fc54ad76fbc54f33f5b032def5
21:16 ejegg: enabled banner_history module on CiviCRM
21:15 mutante: scb1001 - mobile endpoint health CRIT
21:15 mutante: mira unmerged changes in mw-staging
21:14 mutante: restbase2001: cassandra CQl refused
21:14 mutante: aqs1001-1003: CRITICAL: Test Get aggregate page views
21:12 mutante: tungsten - signed new puppet certs, re-installed
20:48 moritzm: kafka re-enabled on kafka1013
20:42 moritzm: stopping kafka broker on kafka1013 to activate ferm
20:26 ottomata: starting kafka broker on kafka1012 after applying base ferm
20:14 ottomata: stopping kafka broker on kafka1012 to apply base ferm
20:09 moritzm: disabled puppet on kafk1012/1013 in preparation of ferm activation
20:08 _joe_: restarting jobchron across the cluster
18:41 ottomata: powercycling analytics1031
16:51 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: https://gerrit.wikimedia.org/r/#/c/252231/ (duration: 00m 35s)
16:45 logmsgbot: krenair@tin Synchronized wmf-config/Wikibase-labs.php: (no message) (duration: 00m 34s)
16:44 logmsgbot: krenair@tin Synchronized wmf-config/Wikibase.php: https://gerrit.wikimedia.org/r/#/c/252204/ (duration: 00m 34s)
16:41 logmsgbot: krenair@tin Synchronized wmf-config/Wikibase.php: https://gerrit.wikimedia.org/r/#/c/250000/ (duration: 00m 34s)
16:34 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/250471/ (duration: 00m 34s)
16:28 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/ContentTranslation/api/ApiQueryContentTranslationSuggestions.php: https://gerrit.wikimedia.org/r/#/c/252172/ (duration: 00m 35s)
16:22 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/Wikidata/Wikidata.php: https://gerrit.wikimedia.org/r/#/c/252227/ (duration: 00m 45s)
15:51 mutante: restarted krypton
15:32 godog: upgrade rsync on ms-be1* per T93587
14:02 godog: swift codfw-prod: ms-be2017 / ms-be2019 / ms-be2021 weight 3000
10:00 jynus: performing mysql maintenance of db1035 (depooled from production). Replication delay and reboots are expected.
09:46 godog: swift codfw-prod: ms-be2017 / ms-be2019 / ms-be2021 weight 2000
09:39 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1035 for maintenance (duration: 00m 36s)
06:13 ejegg|away: updated fundraising tools from f2f571903d32ec73aeab73ce58c6bee71717213b to 182f217ea20a496fe0f9737447d04ed9a3bccde8
03:12 ottomata: restarted statsv on hafnium just in case it stopped after kafka1013 broker crash and restart (ping ori :) )
03:10 ottomata: kafka preferred-replica-election to bring kafka1013 back as leader for its partitions
03:00 mutante: started kafka on kafka1013
02:55 mutante: tried to start kafka on kafka1018
02:31 mutante: switched people.wm.org to new backend. please use "rutherfordium.eqiad.wmnet" instead of terbium now. all files have been rsynced already
02:22 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 06m 49s)
00:13 hoo: Attached Richardelainechambers@commonswiki to the global account of the same name
00:06 logmsgbot: bd808@tin Synchronized wmf-config/logging.php: Disable microsecond timestamps on all loggers (bf61bfc) (duration: 00m 34s)

2015-11-09

23:55 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Disable cirrus writes to labsearch, the machine cant take the load and some jobs are timing out (duration: 00m 34s)
23:13 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Enable CirrusSearch writes to labsearch for all but enwiki and dewiki (duration: 00m 35s)
22:49 ottomata: rebooting analytics1032: https://phabricator.wikimedia.org/T118175
21:52 bearND: MobileApps deployed sha1 6c63984
19:40 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Pool db1060 after performance patch (duration: 00m 35s)
18:42 jynus: restarting mysql in db1060 to test new performance configuration
17:55 bblack: upgrading pybal (1.10 -> 1.12) on lvs200[456].codfw.wmnet
16:50 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/250454/ (duration: 00m 34s)
16:44 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/VisualEditor/modules/ve-mw/init: https://gerrit.wikimedia.org/r/#/c/251972/ (duration: 00m 34s)
16:40 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/251674/ (duration: 00m 35s)
16:35 logmsgbot: krenair@tin Synchronized docroot/wikipedia.org/apple-app-site-association: https://gerrit.wikimedia.org/r/#/c/250897/ (duration: 00m 34s)
16:30 matt_flaschen: Ran mwscript extensions/Flow/maintenance/FlowUpdateBetaFeaturePreference.php --wiki=wikidatawiki
16:24 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/251677/ (duration: 00m 35s)
16:14 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/Flow/modules/mw.flow.Initializer.js: https://gerrit.wikimedia.org/r/#/c/251560/ (duration: 00m 44s)
16:10 _joe_: removing old builds from copper
15:45 godog: reimage graphite1002
15:06 ottomata: running kafka preferred-replica-election
15:02 ottomata: starting kafka broker on kafka1014
14:56 ottomata: stopping kafka broker on kafka1014
14:48 godog: swift codfw-prod: ms-be2017 / ms-be2019 / ms-be2021 weight 1000
14:40 ottomata: restarting eventlogging to see if it is ok after enabling firewall rules on kafka1014
14:36 _joe_: reducing /tmp size on copper by shrinking the logical volume
13:16 godog: nodetool decomission restbase2001
13:13 mobrovac: restbase finished deploying ae2a44f
12:59 mobrovac: restbase start deploying ae2a44f
12:56 hashar: restarting Jenkins to refresh the cli-shutdown.groovy script -- https://gerrit.wikimedia.org/r/251935 (https://phabricator.wikimedia.org/T118064)
12:10 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depooling db1060 again (duration: 01m 13s)
11:16 hashar: Jenkins back and apparently happy
11:09 hashar: Upgrading Jenkins from LTS 1.609.3 to LTS 1.625.1 https://phabricator.wikimedia.org/T118157
10:53 dcausse: resuming writes to elasticsearch indices in eqiad (test)
10:02 _joe_: repooled mw1061
09:29 godog: swift codfw-prod: ms-be2016 / ms-be2018 / ms-be2020 weight 3000
09:19 dcausse: err (1007 no 1008): restarting elastic on elastic1007.eqiad.wmnet (test)
09:18 dcausse: restarting elastic on elastic1008.eqiad.wmnet (test)
09:10 dcausse: freezing elasticsearch indices in eqiad (test)
02:21 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 06m 49s)

2015-11-08

20:43 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/ConfirmEdit/SimpleCaptcha/Captcha.php: https://gerrit.wikimedia.org/r/#/c/251673/ (duration: 00m 34s)
20:21 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.5/resources: (no message) (duration: 00m 34s)
20:14 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.5/resources/src: I24d9b16ed: Rework the Preferences to prevent FOUC (T115692) (duration: 00m 34s)
17:10 _joe_: powercycled ms-be1009, unresponsive to ping, console unconneptable
13:32 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1060 (duration: 00m 34s)
13:02 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1060 (duration: 01m 29s)
02:20 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 06m 21s)

2015-11-07

19:50 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: https://gerrit.wikimedia.org/r/#/c/251739/ (duration: 00m 35s)
17:54 ejegg|away: updated SmashPig from 5860ef47790010d65981822f30914679df06ae14 to 55c2dbc8cd36cfb89d7e17b75b840ca2e023653e
17:38 awight: more syntax fixing
17:35 awight: fix SmashPig failmail syntax
17:29 awight: point SmashPig failmail at ejegg|away and awight
17:28 jynus: restarting labsdb1004 mysql to test mariadb upgrade
17:09 jynus: restarting db2070 mysql to test mariadb upgrade
17:01 jynus: restarting es2010 mysql to test mariadb upgrade
15:17 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Return db weights on s2 api back to normal (duration: 00m 34s)
11:24 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1054 (duration: 00m 34s)
11:02 jynus: setting SET GLOBAL use_stat_tables = 0; on db1060
10:45 jynus: restarting db1054 (I may need to do it several times)
10:41 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1054 (duration: 00m 35s)
10:27 jynus: reverting last schema change
10:08 jynus: performing schema change on db1054 (zhwiki)
05:00 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 34s)
04:59 logmsgbot: ori@tin Synchronized wmf-config/mobile.php: (no message) (duration: 01m 09s)
02:27 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 06m 12s)
00:23 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.5/extensions/WikimediaMaintenance/getPageCounts.php: (no message) (duration: 00m 34s)
00:05 ejegg: updated SmashPig from ac9772765287e79178e07ddfcc7e532d2f82b6d0 to 5860ef47790010d65981822f30914679df06ae14
00:04 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.5/extensions/WikimediaMaintenance/getPageCounts.php: (no message) (duration: 00m 34s)

2015-11-06

23:48 ejegg: updated SmashPig listener from afefccff0f8d5e0b37044adff964c6cd913e1d80 to ac9772765287e79178e07ddfcc7e532d2f82b6d0 and set failmail target back to fr-tech
23:34 logmsgbot: reedy@tin Synchronized wmf-config/InitialiseSettings.php: Enable DynamicPageList on officewiki (duration: 01m 03s)
22:42 legoktm: mwscript updateSpecialPages.php --wiki=enwiki --only=GadgetUsage
19:22 andrewbogott: disabled puppet on holmium while testing and staging an update for bug T117610
19:13 logmsgbot: ori@tin Synchronized wmf-config/jobqueue-eqiad.php: I920627cf8b: Add rdb1007 as fallback aggregator (duration: 00m 35s)
18:54 logmsgbot: ori@tin Synchronized wmf-config/jobqueue-eqiad.php: I9e5be66680: Add rdb1007 to jobqueue pool (duration: 00m 35s)
18:19 ori: running sync-common on mw1083
17:45 logmsgbot: demon@tin Synchronized multiversion/checkoutMediaWiki: no-op outside tin; just for completeness (duration: 00m 36s)
15:29 paravoid: draining esams for network maintainance (deployed about 7mins ago)
14:52 moritzm: uploaded debdeploy 0.0.9 to apt.wikimedia.org
14:31 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Pool all available db servers in codfw (duration: 00m 49s)
13:58 godog: bounce ganglia-monitor-aggregator-instance ID=2027 on install2001, debugging missing ms-be2020
13:38 godog: swift codfw-prod: ms-be2016 / ms-be2018 / ms-be2020 weight 2000
13:36 bblack: cp4007 repooled - T117746
13:16 bblack: upgrading pybal 1.10 -> 1.12 on lvs300[34].esams.wmnet
13:11 bblack: re-enabled aws-d-eqiad in librenms
13:09 bblack: upgrading pybal 1.10 -> 1.12 on lvs4003
12:18 mark: Swapped cr2-knams fpc 1 mic 1 for a new 2x 10G line card
12:18 mark: Swapped cr2-knams fpc 1 mic 1 for a new 2x 10
12:18 mark: mark@cr2-knams> request chassis mic fpc-slot 1 mic-slot 1 online
12:10 mark: mark@cr2-knams> request chassis mic fpc-slot 1 mic-slot 1 offline
11:31 jynus: stopping mysql and cloning db2042 -> db2070, db2054 -> db2068, db2053 -> db2067, db2049 -> db2056
11:12 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Depool db2042, db2049, db2053 and db2054 for cloning (duration: 00m 34s)
09:09 jynus: Stopping mysql and cloning db2044 -> db2065, db2048 -> db2069, db2045 -> db2066
08:55 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Repool db20 34,42,35,38,39,40. New: 58,64,60,61,62,59. Depool: 44,45,48 (duration: 01m 10s)
02:28 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 05m 56s)
01:43 logmsgbot: krenair@mira Synchronized README: done (duration: 00m 34s)
01:38 logmsgbot: krenair@mira Synchronized README: test (duration: 00m 34s)
01:33 logmsgbot: bd808@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/251446/ (duration: 00m 35s)
01:30 logmsgbot: bd808@tin Synchronized wmf-config/logging.php: https://gerrit.wikimedia.org/r/#/c/249313/ (forgot to apply after fetching) (duration: 00m 34s)
01:28 logmsgbot: bd808@tin Synchronized wmf-config/InitialiseSettings.php: Touch (duration: 00m 35s)
01:26 logmsgbot: bd808@tin Synchronized wmf-config/logging.php: https://gerrit.wikimedia.org/r/#/c/249313/ (duration: 00m 34s)
01:10 logmsgbot: krenair@tin Synchronized multiversion/defines.php: https://gerrit.wikimedia.org/r/#/c/251447/ (duration: 00m 34s)
01:00 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/resources/src/mediawiki/mediawiki.ForeignStructuredUpload.BookletLayout.js: https://gerrit.wikimedia.org/r/#/c/251307/ (duration: 00m 34s)
00:59 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/resources/src/mediawiki/mediawiki.ForeignStructuredUpload.js: https://gerrit.wikimedia.org/r/#/c/251301/ https://gerrit.wikimedia.org/r/#/c/251306/ and https://gerrit.wikimedia.org/r/#/c/251307/ (duration: 00m 35s)
00:34 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/251317/ (duration: 00m 35s)
00:18 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/Flow: https://gerrit.wikimedia.org/r/#/c/251416/ (duration: 00m 37s)

2015-11-05

23:34 bd808: Decreased replica count of logstash-2015.10.13 and logstash-2015.10.14 to free disk space on cluster
23:30 bd808: Logging volume into ELK cluster down dramatically; investigating
23:16 logmsgbot: ori@tin Synchronized wmf-config/logging.php: Ieb8c602a: monolog: Ensure that context data added by WebProcessor is utf-8 safe (duration: 00m 36s)
22:35 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.5/extensions/WikimediaEvents: Ic99ac31f740956: Log backend response time on edit requests (duration: 00m 35s)
21:54 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: sync 1.27.0-wmf.5 to group2
20:17 logmsgbot: demon@tin Finished scap: no changes, testing permissions on mira co-master (duration: 07m 29s)
20:10 logmsgbot: demon@tin Started scap: no changes, testing permissions on mira co-master
20:07 _joe_: chown mwdeploy:wikidev recursively on mira for /srv/mediawiki-staging
19:37 logmsgbot: twentyafterfour@tin Finished scap: Sync everything just to be sure (duration: 22m 01s)
19:15 logmsgbot: twentyafterfour@tin Started scap: Sync everything just to be sure
18:58 logmsgbot: demon@tin Synchronized README: no-op, testing new scap code (duration: 00m 19s)
18:52 ostriches: scap: deploying master@b44c268
18:05 jynus: schema change on x1 - flowdb
17:47 ostriches: 17:45:13 Synchronized wmf-config/: Remove +superprotect, I579c11a2 (duration: 00m 18s)
17:44 jynus: Deploying schema change on officewiki - flow (s3)
17:11 godog: nodetool decommission on praseodymium
14:12 jynus: reinstalling db2056.codfw.wmnet
13:57 jynus: shuting down mysql and cloning db2042 -> db2062, db2038 -> db2059, db2039 -> db2060, db2040 -> db2061
13:43 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Repool db2051; Depool db2042, 38, 39, 40 for cloning (duration: 00m 18s)
12:33 _joe_: manually disabling mod_cgi on terbium
09:56 godog: run removenode on cerium.eqiad.wmnet -- decomission was missed before reimaging
09:33 jynus: stopping mysql and cloning db2034 -> db2062, db2035 -> db2063, db2051 -> db2058
09:25 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Pool db2048, db2049, db2050,db2055, db2057, db2063. Depool db2034, db2035, db2051 (duration: 00m 17s)
08:05 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Nov 5 08:05:13 UTC 2015 (duration 5m 12s)
03:21 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.5) at 2015-11-05 03:21:45+00:00
03:15 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 10m 17s)
02:47 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-11-05 02:47:20+00:00
02:40 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 10m 31s)
02:17 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I3c397e892e: Only set $wgDisableOutputCompression to 'true' on the scalers (duration: 00m 18s)
02:04 Krenair: Someone has left tin:/srv/mediawiki-staging/php-1.27.0-wmf.5 in a mess, see `git log origin/wmf/1.27.0-wmf.5..HEAD --oneline`. Note there is one commit waiting to be merged on tin (https://gerrit.wikimedia.org/r/#/c/251168/) that hasn't been yet because of this.
00:13 ejegg: updated fundraising tools from e1b60fa2c258fd4ff55905b03a4d8886132278c1 to f2f571903d32ec73aeab73ce58c6bee71717213b

2015-11-04

23:56 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.5/maintenance/findOrphanedFiles.php: ced9a62893c (duration: 00m 17s)
23:55 ejegg: updated CiviCRM from 827faf37b1b8f4ade6f8d03077c376a09d9525c7 to 230cebb4a3b46a75fef758d46f4db22de29286cb
23:09 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.5/maintenance/findOrphanedFiles.php: 33d378bee6c4ac (duration: 00m 17s)
22:33 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.5/maintenance/findOrphanedFiles.php: fb43e95858b9 (duration: 00m 18s)
21:36 ejegg: enabled Adyen payments listener
21:17 subbu: finished deploying parsoid sha 04893a18
21:05 subbu: synced code + restarted parsoid on wtp1002 as a canary
21:01 subbu: starting parsoid deploy
20:45 ejegg: disabled Adyen payment listener
20:36 ejegg: re-enabled Adyen payment listener with updated credentials
20:13 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.5
18:37 mutante: correction. ^ that was mw1159
18:35 mutante: mw1159: power cycled - crashed, no console output
18:23 logmsgbot: bd808@tin Synchronized php-1.27.0-wmf.5/includes/debug/logger/MonologSpi.php: MonologSpi: add support for customizing Monolog\Logger instances (59eed42) (duration: 00m 17s)
18:23 mutante: ms-be2016-2020 - accepted salt keys, hafnium.wm.org - deleted salt key - still unaccepted: capella, haedus, rhodium
17:57 mutante: ms-be2020 - added to puppet
17:54 mutante: ms-be2021 - fixing puppet certs after reinstall
17:51 papaul: ms-be2021 installation complete
17:45 papaul: reinstalling ms-be2021
17:20 ejegg: updated SmashPig from be9bf87c5583bca75616284f6a4d16ff8ebd3679 to afefccff0f8d5e0b37044adff964c6cd913e1d80
17:12 chasemp: rebooting cp4007.mgmt.ulsfo.wmnet as it's uresponsive
17:11 bblack: cp4007 depooled in pybal + confctl
17:10 papaul: ms-be2020 installation complete
16:56 ejegg: disabled adyen payments listener
16:30 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/250971/ (duration: 00m 18s)
16:29 bd808: restarted elasticsearch on logstash1004; it fell out of the cluster during ferm firewall application
16:28 jynus: db2056 clone failed due to wrong partitioning, needs reinstall. Cloning db2049 -> db2063 instead
16:01 godog: reimage cerium for multi-instance
15:22 _joe_: moving email batch, flagged reviews, update article count off of terbium
15:20 jynus: syncing sqldata on db2049 -> db2056, db2050 -> db2057, eta 2-4 hours
15:15 _joe_: moved translationnotification, updatetranslationstats, pagetriage from terbium to mw1152
14:55 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Depool db2049 ans db2050 for maintenance (duration: 00m 17s)
14:30 jynus: cloning sqldata from db2048 to db2055
14:04 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Depool db2048 for maintenance (duration: 00m 18s)
13:03 bblack: upgrading pybal (1.10 -> 1.12) on lvs4004.ulsfo.wmnet (inactive low-traffic LVS @ ulsfo)
11:59 _joe_: moved refreshlink jobs off terbium
11:32 _joe_: migrating wikidata jobs off of terbium
11:07 jynus: testing new mariadb-server version on db2056.codfw.wmnet
09:25 godog: bootstrap cassandra xenon-b.eqiad.wmnet
08:47 jynus: reduced durability on dbstore1002 and db1047 to improve performance
08:27 _joe_: moved parser cache purging off of terbium
08:25 _joe_: moved clean_uploadstash to mw1152 from terbium
06:44 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Nov 4 06:44:23 UTC 2015 (duration 44m 22s)
05:52 logmsgbot: demon@tin Finished scap: using wfLoadExtension() for ContentTranslation, shut up warning spam (duration: 49m 22s)
05:03 logmsgbot: demon@tin Started scap: using wfLoadExtension() for ContentTranslation, shut up warning spam
05:02 logmsgbot: demon@tin scap failed: CalledProcessError Command '['sudo', '-u', 'mwdeploy', '-n', '--', '/usr/bin/rsync', '--archive', '--delete-delay', '--delay-updates', '--compress', '--delete', '--exclude=**/.svn/lock', '--exclude=**/.git/objects', '--exclude=**/.git/**/objects', '--exclude=**/cache/l10n/*.cdb', '--no-perms', 'tin.eqiad.wmnet::common', '/srv/mediawiki']' returned non-zero exit status 24 (duration: 02m 45s)
04:59 logmsgbot: demon@tin Started scap: using wfLoadExtension() for ContentTranslation, shut up warning spam
04:54 ejegg|away: updated smashpig from c43db1742d5663f9d3685fd3e0a254ff72a29fa0 to be9bf87c5583bca75616284f6a4d16ff8ebd3679, changed credentials
03:09 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.5) at 2015-11-04 03:09:44+00:00
03:03 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 10m 08s)
02:36 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-11-04 02:36:48+00:00
02:32 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 07m 34s)
01:36 legoktm: ran delete from globaluser where gu_name="" limit 1 on centralauth. (T96233)
01:10 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: https://gerrit.wikimedia.org/r/#/c/250874/ (duration: 00m 17s)
01:02 hoo: Manually recreated https://dumps.wikimedia.org/wikidatawiki/entities/dcatap.rdf using Zend php (instead of HHVM). See T117534.
00:25 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/250869/ (duration: 00m 19s)
00:09 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/250712/ (duration: 00m 18s)
00:03 logmsgbot: krenair@tin Synchronized portals: https://gerrit.wikimedia.org/r/#/c/250851/ (duration: 00m 17s)

2015-11-03

23:11 ejegg: deployed Adyen credentials change to payments wiki
22:36 ostriches: tin: scap now pointing to Phab repo instead of Gerrit.
21:59 logmsgbot: krenair@tin Synchronized README: rv, was just demonstrating for https://phabricator.wikimedia.org/T117574#1779384 (duration: 00m 17s)
21:42 logmsgbot: krenair@tin Synchronized README: (no message) (duration: 00m 18s)
21:21 mutante: ms-be2018,ms-be2019: sign puppet certs, initial run
20:45 csteipp: deployed patch for T109724 to wmf4/5
20:41 gwicke: restbase cassandra: switched local_group_wikipedia_T_parsoid_html to the Date-Tiered Compaction Strategy (DTCS)
20:38 gwicke: cleared cassandra snapshots on aqs servers
19:59 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.5
19:55 mutante: hafnium - revoke puppet cert, salt key for the .wikimedia.org name
19:54 mutante: hafnium - reboot into PXE for jessie reinstall
19:47 logmsgbot: twentyafterfour@tin Finished scap: testwiki to 1.27.0-wmf.5 (duration: 30m 00s)
19:35 awight: update CRM from 2828fdfbf791c6ca28d1f92725d1ec887b9fe4c7 to 827faf37b1b8f4ade6f8d03077c376a09d9525c7
19:33 awight: update SmashPig from c431c8d77521270236c72532a50806b2e852cf7b to c43db1742d5663f9d3685fd3e0a254ff72a29fa0
19:17 logmsgbot: twentyafterfour@tin Started scap: testwiki to 1.27.0-wmf.5
19:11 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/includes/libs/objectcache/BagOStuff.php: 46dff75b5b: Make makeKeyInternal() limit more conservative (duration: 00m 18s)
19:06 twentyafterfour: deploying 1.27.0-wmf.5 to group0
18:34 ori: Moved statsv script from hafnium to a screen session on eventlog2001 in preparation for hafnium reimage
18:29 ori: Moved navtiming script from hafnium to a screen session on eventlog2001 in preparation for hafnium reimage
17:13 moritzm: installed security updates on the internal ntpds (system-local ones to follow)
16:27 dcausse: restarting elastic on nobelium to check new settings
16:19 moritzm: installed unzip and audiofile security updates across the fleet
16:12 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove sampling of CirrusSearchRequestSet log channel gerrit:250691 (duration: 00m 18s)
16:07 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor for 10% of new accounts on eswiki gerrit:250470 (duration: 00m 18s)
14:19 jynus: disabling puppet on dbstore1002 and db1047 to debug replication issue
14:07 godog: manual invocation of /usr/local/bin/archive-instances on labmon1001 to test
13:33 jynus: migrating eventlogging_sync from terbium to db1047 and dbstore1002 (analytics-store)
12:15 _joe_: updated pybal for jessie to 1.12
11:00 godog: start cassandra-a on xenon
09:56 godog: reimage xenon.eqiad.wmnet
09:45 godog: decomission xenon.eqiad.wmnet from cassandra, pending conversion to multi-instance
05:38 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Nov 3 05:38:33 UTC 2015 (duration 38m 32s)
02:26 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-11-03 02:26:14+00:00
02:22 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 07m 00s)
01:05 awight: enabled CRM module amazon_audit
01:03 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.4/extensions/CentralAuth/includes/CentralAuthUser.php: Ia9673a448da81: User cache key (duration: 00m 17s)
01:03 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.4/includes/: I58836a24b9e239f: User cache key (duration: 00m 21s)
00:37 logmsgbot: ebernhardson@tin Synchronized wmf-config/logging.php: https://gerrit.wikimedia.org/r/#/c/240615/ (duration: 00m 19s)
00:35 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/240615/ (duration: 00m 17s)
00:34 logmsgbot: ebernhardson@tin Synchronized wmf-config/avro/CirrusSearchRequestSet.avsc: https://gerrit.wikimedia.org/r/#/c/240615/ (duration: 00m 17s)
00:28 ebernhardson: sync-common on mw1017 for https://gerrit.wikimedia.org/r/240615
00:26 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/241226/4 (duration: 00m 17s)
00:17 awight: update CRM from 77cd64317404b2fcec9500cfbb1c0b19e5846705 to 2828fdfbf791c6ca28d1f92725d1ec887b9fe4c7
00:13 logmsgbot: ebernhardson@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/250590/ (duration: 00m 18s)
00:09 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: add www.webarchive.co.uk and *.unesco.org to server side upload whitelist (duration: 00m 18s)
00:06 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/246169/ (duration: 00m 18s)

2015-11-02

23:11 RoanKattouw: Running convertAllLqtPages.php on ptwikibooks
22:38 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Re-enable Flow on ptwikibooks with English-only Topic namespace name (duration: 00m 18s)
22:38 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Plumbing for wmgFlowEnglishNamespaceOnly (duration: 00m 18s)
21:56 cwd: updated worldpay request timeout to 30s
21:31 _joe_: restarting parsoids in codfw
21:25 subbu: deployed parsoid f0d77afc
20:49 mutante: ms-be2017 - initial puppet run, fresh install
20:38 logmsgbot: csteipp@tin Synchronized php-1.27.0-wmf.4/includes/: (no message) (duration: 00m 22s)
20:38 robh: lawrencium having puppet stopped and reclaiming into spares per T117477 disregard any alerts
20:06 mutante: ms-be2016 - signing puppet certs, salt-key, initial run
20:05 mutante: started morebots (production-logbot job was gone)
19:46 logmsgbot: hoo@tin Synchronized php-1.27.0-wmf.4/extensions/MobileFrontend/includes/MobileFrontend.skin.hooks.php: Fix changing the license message key via the "MobileLicenseLink" hook (duration: 00m 17s)
19:14 gwicke: restbase cassandra: switching local_group_wikipedia_T_parsoid_dataW4ULtxs1oMqJ to DTCS
19:00 matt_flaschen: Completed FlowFixLinks run on all Flow wikis
18:53 csteipp: deployed patches for T97897 & T115522
18:53 awight: updated CRM from 60c4a8e5fc4efb6ac38a520e652afe187e8c177b to 77cd64317404b2fcec9500cfbb1c0b19e5846705
18:48 andrewbogott: dist-upgrade and rebooting labvirt1005
18:41 Coren: manually starting service replicate-tools on labstore1001
18:10 awight: removed failing SmashPig job runner task
18:01 jynus: performing some fixes on labsdb1004 database. Slightly higher load on labsdb1005.
17:59 dcausse: restarting elastic on nobelium
17:43 awight: disabled Donations queue consume job
17:39 gwicke: restbase cassandra: switched local_group_wikisource_T_parsoid_html and local_group_wikibooks_T_parsoid_html to DTCS
17:09 bd808: Decreased replica count to 1 for logstash-2015.10.04 thru logstash-2015.10.12 to free cluster disk space; see T117438
17:07 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/249762/ (duration: 00m 19s)
17:00 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/247088/ (duration: 00m 17s)
16:48 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/250416/ (duration: 00m 17s)
16:47 logmsgbot: krenair@tin Synchronized dblists/visualeditor-default.dblist: https://gerrit.wikimedia.org/r/#/c/250416/ (duration: 00m 17s)
16:45 logmsgbot: krenair@tin Synchronized dblists/closed.dblist: https://gerrit.wikimedia.org/r/#/c/250416/ - close wikimania2014.wikimedia.org (duration: 00m 18s)
16:36 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/249794/ - should be a noop for prod (duration: 00m 17s)
16:30 bd808: Deleted logstash-2015.10.03 index to free disk space on logstash1004; see T113571
16:28 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.4/extensions/Flow: https://gerrit.wikimedia.org/r/#/c/249940/1 (duration: 00m 20s)
16:26 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/249935/ (duration: 00m 18s)
16:09 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/250401/ (duration: 00m 18s)
14:43 ottomata: shutting down analytics1038 to run fsck on root filesystem. Currently it is read only
14:36 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-br-fr_0.5.0+svn~57870-1
14:35 mobrovac: citoid deployed 70abb90
13:34 paravoid: draining ulsfo, T116928
12:19 mobrovac: mobileapps deploying 59e8c30
10:52 hashar: Restarting Jenkins to upgade Gearman plugin from 0.1.2 to 0.1.3 "Send node labels back on build completion"
10:10 akosiaris: powering off pollux. migrating to a VM
09:39 _joe_: moved tor exit node, update_special_pages off of terbium to mw1152
09:24 mobrovac: citoid restarted zotero on sca100x
09:07 jynus: deleting user_daily_contribs view on labs (it points to a non-existent table)
08:57 mobrovac: restbase deploying 8036232
05:33 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Nov 2 05:33:07 UTC 2015 (duration 33m 6s)
02:25 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-11-02 02:25:42+00:00
02:22 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 06m 44s)

2015-11-01

16:59 gwicke: restarting cassandra on aqs1002; was out of heap space
05:25 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Nov 1 05:25:49 UTC 2015 (duration 25m 48s)
02:24 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-11-01 02:24:46+00:00
02:21 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 06m 36s)

2015-10-31

08:57 dcausse: elastic1008 deleting /var/log/elasticsearch/production-search-eqiad_index_indexing_slowlog.log.[2-7]
05:15 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Oct 31 05:15:43 UTC 2015 (duration 15m 42s)
02:54 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/vendor/monolog/monolog/src/Monolog/Logger.php: Iccfda4768 (duration: 00m 19s)
02:26 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-10-31 02:26:22+00:00
02:23 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 06m 26s)

2015-10-30

23:53 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: Ifd3543d99: Re-enabled sidebar cache per 47eb083a0fe4 (duration: 00m 17s)
23:38 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/includes/cache/MessageCache.php: Id9a27ba2bbd3 (duration: 00m 17s)
23:38 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/includes/skins/Skin.php: Id9a27ba2bbd3 (duration: 00m 17s)
23:37 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/Collection: b89cff0a1d: Updated mediawiki/core Project: mediawiki/extensions/Collection 092204bf333e65b2c749e4bc4a32fd2b0254089b (duration: 00m 18s)
23:37 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/DismissableSiteNotice: a20aa8a544: Updated mediawiki/core Project: mediawiki/extensions/DismissableSiteNotice 5b8f1cfac48704fd740eb61f873aabb600c4c5fb (duration: 00m 17s)
23:03 gwicke: fixed restbase labs install by deploying it; the code had gone missing
23:03 ori: Added MaxSem and gwicke to Gerrit Project Creators group
22:44 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/includes/objectcache/ObjectCache.php: If12aedae7f: objectcache: Use singleton cache in newAccelerator() (duration: 00m 18s)
22:05 logmsgbot: ori@tin Synchronized docroot and w: (no message) (duration: 00m 17s)
22:01 mutante: labvirt1010: salt-minion always in "stop/waiting", doesn't restart
21:54 mutante: labvirt1010 - start salt
21:41 ori: Removing MediaWiki.xhprof.* from graphite{1,2}001
21:41 logmsgbot: ori@tin Synchronized wmf-config/StartProfiler.php: I0bfa21b5: Disable xhprof profiling to relieve storage pressure on graphite (duration: 00m 18s)
19:07 K4-713: antifraud tweak on payments wiki (again)
18:56 K4-713: antifraud tweak on payments wiki
18:56 jynus: disabling puppet and restarting mysql on es2008, es2009 and es2010- downtime planned for 2 hours - no production impact
17:33 robh: bismuth has no console output, appears locked up in OS, rebooting via mgmt
17:22 logmsgbot: ori@tin Synchronized docroot and w: (no message) (duration: 00m 18s)
16:56 cmjohnson: reinstalling mw1083 https://phabricator.wikimedia.org/T116184
15:35 moritzm: updated db205[6-8] to the 3.19 kernel
14:38 chasemp: nobelium in downtime -- this is a temporary test host for discovery and is not ops actionable
13:47 Krenair: revert commit for https://gerrit.wikimedia.org/r/#/c/249971/ on tin
13:47 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.4/extensions/VisualEditor/VisualEditor.hooks.php: https://gerrit.wikimedia.org/r/#/c/249999/ (duration: 00m 17s)
11:57 paravoid: upgrading tor to latest stable on radium
11:31 paravoid: ignoring asw-d-eqiad on librenms
11:25 _joe_: powercycling dataset1001, stuck in some kernel task, unable to login from console
10:42 dcausse: elastic in eqiad setting indexing trace threshold to -1 for commons, ruwiki, frwiki and itwiki
10:30 dcausse: elastic in eqiad wide settings for indexing slow threshold are ineffective, trying to set index settings (T117181)
09:18 dcausse: elastic in eqiad setting index.indexing.slowlog.threshold.index.trace to -1
08:46 _joe_: removed production-search-eqiad_index_indexing_slowlog.log.{7,8} on es1008
06:24 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.4/includes/jobqueue/JobQueueRedis.php: (no message) (duration: 00m 18s)
05:11 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Oct 30 05:11:08 UTC 2015 (duration 11m 7s)
04:19 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I1d0337b58: ScribuntoSlowFunctionThreshold: 0.90 -> 0.99 (duration: 00m 17s)
04:06 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/Scribunto/common/Hooks.php: I60b9eb617: When logging perf stats, include wfWikiId() in metric key (duration: 00m 18s)
04:03 awight: update CRM from e0cad7215192464941d0ec282a70cb608619cf2f to 60c4a8e5fc4efb6ac38a520e652afe187e8c177b
03:42 awight|parentoid: update CRM from 367cd85954697c6ad4311deb2799fe9ba08b5d9d to e0cad7215192464941d0ec282a70cb608619cf2f
02:47 awight|parentoid: donations queue consumer reenabled
02:45 awight|parentoid: updated CRM from 61c8c13efb31f7e75564d9a01fa879db0690fb78 to 367cd85954697c6ad4311deb2799fe9ba08b5d9d
02:30 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-10-30 02:30:50+00:00
02:27 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 06m 13s)
00:48 logmsgbot: tgr@tin Synchronized php-1.27.0-wmf.4/includes/specials/SpecialUserlogin.php: T117027 (duration: 00m 18s)
00:34 mutante: labvirt1010 - started salt-minion (no such process)
00:34 mutante: mw1193 - restarted HHVM (socket timeout)
00:34 mutante: wdqs1001 - restarted NTP (unknown offset)
00:09 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.4/extensions/Flow/: Fix regressions from SWAT (duration: 00m 20s)
00:08 mutante: radium - back in service
00:01 bd808: Restarted logstash on logstash1003 again. The first try apparently didn't take

2015-10-29

23:44 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: Ibac0d60bd: Disable ScribuntoGatherFunctionStats (duration: 00m 17s)
23:32 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-common.php: https://gerrit.wikimedia.org/r/249922 (duration: 00m 17s)
23:32 bd808: Restarted logstash on logstash1003; died with OOM error
23:24 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/Scribunto/engines/LuaSandbox/Engine.php: I69e9218 (duration: 00m 18s)
23:21 mutante: radium: scheduled downtime, reinstalling
23:19 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.4/extensions/CirrusSearch/: Fix unwritable cluster errors (duration: 00m 19s)
23:18 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.4/extensions/Flow: Fix CAPTCHA rendering in RTL languages (duration: 00m 19s)
23:17 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.4/extensions/Scribunto/: Make the percentile threshold for slow function stats configurable (duration: 00m 18s)
23:16 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.4/resources/src/mediawiki.special/mediawiki.special.search.css: SWAT: styling tweaks for inline interwiki search (duration: 00m 18s)
23:09 awight: set dashboard cache interval to 60m
22:33 logmsgbot: tgr@tin Synchronized php-1.27.0-wmf.4/extensions/ZeroBanner/includes/ZeroSpecialPage.php: T116821 (duration: 00m 17s)
22:31 logmsgbot: tgr@tin Synchronized php-1.27.0-wmf.4/includes/MagicWord.php: T117066, fixes some exceptions in production (duration: 00m 17s)
22:30 logmsgbot: tgr@tin Synchronized php-1.27.0-wmf.4/includes/specials/SpecialSearch.php: c249899, fixes some warnings in production (duration: 00m 17s)
22:29 logmsgbot: tgr@tin Synchronized php-1.27.0-wmf.4/includes/Category.php: c249890, fixes some warnings in production (duration: 00m 18s)
22:05 awight: CiviCRM upgrade to 4.6.9 succeeded; new data is backed up
22:02 gwicke: restbase: switched local_group_default_T_parsoid_html to Date-Tiered compaction (DTCS)
21:40 logmsgbot: tgr@tin Synchronized php-1.27.0-wmf.4/includes/MagicWord.php: T117066 (duration: 00m 18s)
21:32 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/includes/libs/Xhprof.php: (no message) (duration: 00m 18s)
21:06 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/Cite/Cite_body.php: Revert "Avoid counting arrays if not needed" (duration: 00m 17s)
20:56 gwicke: restbase: switch local_group_wikiquote_T_title__revisions to Date-Tiered compaction (DTCS)
20:47 logmsgbot: ori@tin rebuilt wikiversions.cdb and synchronized wikiversions files: (no message)
19:50 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I06879b6e6e: Enable ScribuntoGatherFunctionStats (duration: 00m 17s)
19:40 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.27.0-wmf.3
19:39 twentyafterfour: rolling back to 1.27.0-wmf.3 due to increase in log errors
19:31 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.27.0-wmf.4
18:59 cmjohnson1: powering down aqs1001 for h/w maintenance
17:53 AaronSchulz: Touched/synced PrivateSettings.php symlink via touch -h
17:47 logmsgbot: aaron@tin Synchronized wmf-config/PrivateSettings.php: (no message) (duration: 00m 18s)
17:36 AaronSchulz: Did touch of InitialiseSettings.php
17:33 logmsgbot: aaron@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 18s)
17:32 logmsgbot: aaron@tin Synchronized private/PrivateSettings.php: (no message) (duration: 00m 17s)
17:17 logmsgbot: kartik@tin Synchronized wmf-config/PrivateSettings.php: Really sync right file this time (duration: 00m 17s)
17:10 logmsgbot: kartik@tin Synchronized private/PrivateSettings.php: Retry syncing for CX token (duration: 00m 17s)
17:04 awight: restarting CiviCRM schema migration: v4.3.alpha1 -> v4.6.9 -- Estimated completion 21:30 UTC
17:02 awight: update crm from 23d2020448f343c8c1b2f4d779be66f57552f935 to 61c8c13efb31f7e75564d9a01fa879db0690fb78
16:57 YuviPanda: restart etherpad on etherpad1001
16:22 moritzm: removed obsolete mysql 5.5 packages on mw102[2-9], mw1032, mw1053, mw1114, mw1163
16:19 _joe_: moved purge_abusefilter from terbium to mw1152
15:48 _joe_: moving purge_checkuser from terbium to mw1152
15:43 _joe_: moving purge_securepoll from terbium to mw1152
15:15 moritzm: removed openjdk-7 on restbase100[1-9] (it's using openjdk-8 for a while)
13:17 logmsgbot: kartik@tin Synchronized wmf-config/CommonSettings.php: Fix ContentTranslationCXServerAuth for CX (duration: 00m 18s)
13:14 logmsgbot: kartik@tin Synchronized private/PrivateSettings.php: Fix name of JWT token (duration: 00m 18s)
12:33 logmsgbot: kartik@tin Synchronized wmf-config/CommonSettings.php: Set ContentTranslationCXServerAuth for CX (duration: 00m 17s)
11:24 mobrovac: restbase rolling-restart after config change https://gerrit.wikimedia.org/r/249465
10:41 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1065 at 100% load. Reduce db1035 weight. (duration: 00m 17s)
10:34 _joe_: migrated jobqueue_stats_reporter to mw1152
10:21 hashar: restarting Jenkins (java upgrade)
10:20 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1065 after maintenance (duration: 00m 18s)
10:14 hashar: Upgrading java on gallium and restarting Jenkins
10:04 moritzm: removed openjdk-7 from cassandra test hosts (now using openjdk-8)
10:01 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable nearby on wikidata (duration: 00m 18s)
10:00 logmsgbot: aude@tin Synchronized wmf-config/Wikibase.php: Config for fetching labels on mobile wikidata (duration: 00m 18s)
09:51 jynus: shutdown mw1083 to avoid querying the mysql servers with an outdated config/spaming the error logs
09:13 jynus: restarting db1065 for regular maintenace
08:59 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1065 for maintenance (duration: 00m 17s)
07:04 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Oct 29 07:04:04 UTC 2015 (duration 4m 3s)
06:43 YuviPanda: disabled two fr-tech related catchpoint tests per awight's request
06:30 logmsgbot: aaron@tin Synchronized rpc/RunJobs.php: 29ccbd248 (duration: 00m 17s)
05:46 ebernhardson: finished manually running 3M enwiki/enqueue, enwiki/wikibase-addUsagesForPage, and wikidatawiki/cirrusSearchLinksUpdatePrioritized jobs from mw1011 and mw1012
05:38 ebernhardson: restarting elasticsearch on nobelium to attempt to clear up extra log GC pauses in the old generation (50s+)
05:37 ebernhardson: restarting elasticsearch on nobelium
03:55 gwicke: cassandra *staging*: testing DateTieredCompactionStrategy (https://labs.spotify.com/2014/12/18/date-tiered-compaction/) on wikipedia html and data-parsoid tables
03:05 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-10-29 03:05:52+00:00
03:02 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 06m 03s)
02:59 YuviPanda: start pybal on lvs1007
02:46 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-29 02:46:27+00:00
02:39 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 10m 47s)
02:05 AaronSchulz: Restarted stuck hhvm on mw1016; dump at /tmp/hhvm.25097.bt
01:06 gwicke: lowered gc_grace on wikipedia parsoid html and data-parsoid keyspaces to 24 hours

2015-10-28

23:31 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/Translate/tag/PageTranslationHooks.php: I0e5f2d3b2 (duration: 00m 18s)
23:29 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.4/extensions/WikimediaEvents/WikimediaEvents.php: https://gerrit.wikimedia.org/r/249642 (duration: 00m 17s)
23:25 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-production.php: (no message) (duration: 00m 17s)
23:22 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-common.php: (no message) (duration: 00m 17s)
23:19 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/249603 (duration: 00m 17s)
23:19 logmsgbot: ebernhardson@tin Synchronized wmf-config/mobile.php: https://gerrit.wikimedia.org/r/249603 (duration: 00m 18s)
23:18 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/249603 (duration: 00m 17s)
23:15 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.4/extensions/MobileFrontend/: https://gerrit.wikimedia.org/r/249585 (duration: 00m 19s)
23:08 logmsgbot: ebernhardson@tin Synchronized portals: Switch www portals to be deployed from Git, but not being served from anywhere yet (duration: 00m 18s)
23:01 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/WikimediaEvents/WikimediaEventsHooks.php: I0e5f2d3b2 (duration: 00m 18s)
22:57 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/CirrusSearch/includes/Hooks.php: I0e5f2d3b2 (duration: 00m 18s)
22:38 urandom: Cassandra cleanup on restbase-test2001-a
22:31 awight: Fundraising CRM db migration started, estimated completion 10:15 UTC (tomorrow)
22:11 awight: update fundraising CRM to civi-4.6.9-deployment branch, from f2fa7b942625b34ede520e11f20e7e0835ecb17d to 23d2020448f343c8c1b2f4d779be66f57552f935
21:11 logmsgbot: legoktm@tin Synchronized php-1.27.0-wmf.4/includes/changes/EnhancedChangesList.php: Fix diff/history links not showing up for ungrouped enhanced RC - https://gerrit.wikimedia.org/r/#/c/249556/ (duration: 00m 19s)
21:08 YuviPanda: enable puppet on labstore1001
20:55 YuviPanda: reverted changes to nsswitch.conf from puppet run manually on labstore2001
20:29 YuviPanda: disable puppet on labstore1001
20:20 awight: Disabled fundraising CiviCRM
19:52 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.4
18:50 Krinkle: mwscript deleteEqualMessages.php --wiki itwikiversity
18:42 logmsgbot: legoktm@tin Synchronized wmf-config/mc.php: https://gerrit.wikimedia.org/r/#/c/249463/ (duration: 00m 17s)
18:41 ostriches: rolled scap back to master@62a250a, needs puppet changes before new code goes live
18:34 gwicke: restbase deploy done
18:34 yurik: updated kartotherian & tilerator services
18:23 gwicke: rolling deploy of restbase-deploy 3b1f6488f2 to restbase cluster
18:18 gwicke: canary deploy of restbase deploy 3b1f6488f2 to restbase1001
18:13 Krinkle: mwscript deleteEqualMessages.php --wiki hiwiki
18:12 Krinkle: mwscript deleteEqualMessages.php --wiki fawiki
17:59 godog: rolling-restart cassandra-metrics-collector, staggered this time
17:23 ostriches: deployed scap master@f823129 to cluster
17:16 ostriches: deployed scap master@abe1973 to cluster
16:54 godog: unblacklist 'max' cassandra metrics and restart cassandra-metrics-collector
16:54 moritzm: installed openjdk security updates on zookeeper hosts
16:18 logmsgbot: aude@tin Synchronized wmf-config/Wikibase.php: add settings for displaying labels on test.wikidata in mobile (duration: 00m 18s)
16:15 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: enable wikibase descriptions on test.wikidata (duration: 00m 17s)
15:46 andrewbogott: testing the bot after the nfs move
15:39 logmsgbot: legoktm@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/modules/ext.wikimediaEvents.search.js: https://gerrit.wikimedia.org/r/#/c/249405/ (duration: 00m 18s)
14:22 kart_: T112626 Finished running fix-stats.php for CX (from rwwiki to zuwiki)
14:21 moritzm: repooled restbase1006
14:09 moritzm: depooled restbase1006 for kernel update
14:05 moritzm: repooled restbase1005
13:54 moritzm: depooled restbase1005 for kernel update
13:47 moritzm: repooled restbase1004
13:35 moritzm: depooled restbase1004 for kernel update
13:34 moritzm: repooled restbase1003
13:16 moritzm: depooled restbase1003 for kernel update
13:12 moritzm: repooled restbase1002
12:56 moritzm: depooled restbase1002 for kernel update
11:46 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1022 after maintenance (duration: 00m 19s)
10:39 moritzm: updated kernel on restbase1001 to latest 3.19
10:37 _joe_: manually removed crontab from mw1152, erroneously created by puppet
10:11 _joe_: reimaging mw1152
09:15 _joe_: preparing to reimage mw1152, disabling puppet, scheduling downtime.
06:22 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Oct 28 06:22:06 UTC 2015 (duration 22m 5s)
05:15 legoktm: ran mwscript updateSpecialPages.php --wiki=testwiki --only=GadgetUsage on terbium
03:03 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-10-28 03:03:34+00:00
03:00 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 06m 00s)
02:39 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-28 02:39:23+00:00
02:34 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 33s)
00:55 yurik: deployed graphoid - https://gerrit.wikimedia.org/r/#/c/249324/
00:37 logmsgbot: tgr@tin Finished scap: Updating MediaViewer with r246112 (duration: 43m 18s)
00:35 mutante: mw1135 temp. unresponsive - OOM killer killing hhvm
00:27 mutante: powercycling unresponsive mw1127

2015-10-27

23:53 logmsgbot: tgr@tin Started scap: Updating MediaViewer with r246112
23:13 ori: running sync-common on mw2050, mw2128, mw2187 and mw2209 (cf I324134438955c7)
23:12 gwicke: ran `sudo mdadm --readwrite md1` on restbase1007 to resolve `pending` state
23:11 gwicke: rebooted restbase1007 to rule out a funky hardware state causing elevated read latencies
23:06 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/248639/ (duration: 00m 17s)
23:05 logmsgbot: krenair@tin Synchronized wikiversions-labs.json: https://gerrit.wikimedia.org/r/#/c/248639/ (duration: 00m 17s)
23:05 logmsgbot: krenair@tin Synchronized dblists/all-labs.dblist: https://gerrit.wikimedia.org/r/#/c/248639/ (duration: 00m 18s)
23:01 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/249306, no-op (duration: 00m 18s)
22:39 mutante: powercycling unresponsive analytics1039, here's what i saw on mgmt https://phabricator.wikimedia.org/P2248
21:50 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.4/includes/MovePage.php: 7a7c7b27d6c (duration: 00m 17s)
21:50 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/: Fix for cache key bug (duration: 00m 20s)
21:29 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.4
21:29 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/includes/debug/logger/LoggerFactory.php: I437bcb532: LoggerFactory: Only check for Psr\Log\LoggerInterface once (duration: 00m 18s)
19:46 logmsgbot: twentyafterfour@tin Finished scap: sync everything for 1.27.0-wmf.4 and point testwiki to the new branch (duration: 31m 32s)
19:44 bd808: Ran sync-common on mw2187 to rebuild l10n caches
19:38 ejegg: enabled donation queue consumer
19:15 logmsgbot: twentyafterfour@tin Started scap: sync everything for 1.27.0-wmf.4 and point testwiki to the new branch
19:09 awight: updated DjangoBannerStats from 57a0392b3f43b65050b01a0465e120ed609a769e to 12e819a04a40ee6fab5dd55fcaf072661df31106
17:56 paravoid: updating jessie debian-installer to 20150422+deb8u2
17:50 Jeff_Green: add fundraising-banner-logger hosts to icinga/nsca
17:40 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.3/extensions/NavigationTiming/modules/ext.navigationTiming.js: I95db9deefe363a65 (duration: 00m 17s)
17:37 ejegg: disabled donation queue consumer to build up queue for benchmark
17:34 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/resources/src/mediawiki.ui: I54c195541: Get rid of CSS transitions on form elements in mediawiki.ui (duration: 00m 17s)
16:07 Jeff_Green: round of fundraising OS updates, occasional icinga noise is expected
16:02 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.3/resources/src/mediawiki/mediawiki.ForeignStructuredUpload.BookletLayout.js: SWAT: mw.ForeignStructuredUpload: Mark description as being in source wikis content language gerrit:249081 (duration: 00m 17s)
15:41 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor in the "Projet" namespace on the French Wikipedia gerrit:248910 (duration: 00m 17s)
15:15 godog: reenable puppet on graphite1001
15:01 kart_: TT112626 Ran fix-stats.php for CX (from bewiki to ruwiki)
13:00 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/249107/ (duration: 00m 18s)
12:59 akosiaris: disabling puppet and bringing down OTRS service on mendelevium
12:21 jynus: Just dropped msg_resource tables from labs dbs. Filters modified to stop replicationg them. Started replicating the heartbeat tables.
11:56 godog: reimage restbase-test2003
11:23 godog: cassandra OOM'd on restbase1007, restarting
10:57 godog: downtime restbase endpoints health for restbase1* while investigating
10:56 hashar: Jenkins job https://integration.wikimedia.org/ci/job/operations-puppet-doc/ is broken. I am on it :-(
10:50 hashar: stopping Jenkins due to an unclean state
10:11 jynus: disabling puppet and restarting mysql servers at db1069- this will create a small amount of lag on labs
09:55 godog: convert restbase-test2003 to cassandra multi-instance
09:08 logmsgbot: aude@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: Add forceParse UpdaterFlag and option in forceSearchIndex script (duration: 00m 19s)
05:47 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Oct 27 05:47:02 UTC 2015 (duration 47m 1s)
03:42 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.3/languages/Language.php: hotfix for T116693 (duration: 00m 19s)
02:29 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-27 02:29:08+00:00
02:24 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 06s)
01:02 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.3/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: T116693 (duration: 00m 19s)
00:42 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/extensions/AbuseFilter: I2f84cff0: Avoid pointless range scan for 'load-recent-authors' (T116557) (duration: 00m 18s)
00:11 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/includes/Parsoid/Utils.php: https://gerrit.wikimedia.org/r/#/c/249026 (duration: 00m 18s)

2015-10-26

23:29 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/248371/ (duration: 00m 20s)
23:22 logmsgbot: krenair@tin Synchronized wmf-config/extension-list-labs: https://gerrit.wikimedia.org/r/#/c/248632/ (duration: 00m 17s)
23:19 logmsgbot: krenair@tin Synchronized w/static/images/project-logos/vecwiki.png: https://gerrit.wikimedia.org/r/#/c/248633/ (duration: 00m 18s)
23:17 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/248478/ (duration: 00m 17s)
23:16 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/248475/ (duration: 00m 17s)
23:15 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/248475/ (duration: 00m 17s)
23:06 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/248871/3 (duration: 00m 18s)
22:39 ejegg: updated payments from 7fd1e880b6a45d79fc305b39f3fee2f324324136 to 4baa5a66a4510414d6b43b59f1b1cda2341c17fd
21:48 cmjohnson1: powering off wdqs1001 to update idrac settings
21:33 ejegg: update payments from 71d2d927f4efac1a639aaf7627c765f48d1b129c to 7fd1e880b6a45d79fc305b39f3fee2f324324136
20:50 subbu: deployed parsoid version 660c59a9
20:21 ebernhardson: started copy of eqiad elasticsearch indices to noeblium
20:19 YuviPanda: stress test on nobelium complete, CPU temperature didn't go above 65C
20:13 paravoid: deactivating ulsfo<->NTT BGP peering due to upcoming network migration
20:05 YuviPanda: running stress on nobelium
19:36 cmjohnson1: swapped bad disk on db1030
15:44 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/WikimediaEvents.php: Re-deploy WME changes after deploying necessary CirrusSearch change first (duration: 00m 18s)
15:43 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/modules: Re-deploy WME changes after deploying necessary CirrusSearch change first (duration: 00m 17s)
15:31 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: Undeploy eventlogging search schema from CirrusSearch (duration: 00m 18s)
15:21 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents: rollback (duration: 00m 18s)
15:13 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/WikimediaEvents.php: Move search schema from cirrussearch -> wikimediavents (duration: 00m 19s)
15:13 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/modules/: Move search schema from cirrussearch -> wikimediavents (duration: 00m 17s)
15:06 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/WikimediaEvents.php: Update satisfaction schema id due to bad varnish caching of old id (duration: 00m 17s)
15:04 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Remove Predundant Page and Index namespaces from $wgContentNamespaces (duration: 00m 17s)
14:58 bblack: repooling cp1059 varnish mobile frontend (wiped)
14:18 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable GeoData on Wikidata (duration: 00m 17s)
14:09 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable geosearch on test.wikidata (duration: 00m 17s)
13:32 logmsgbot: aude@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: Add justMapping option to updateOneSearchIndexConfig script (updated submodule) (duration: 00m 18s)
13:27 logmsgbot: aude@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: Add justMapping option to updateOneSearchIndexConfig script (duration: 00m 18s)
09:42 dcausse: deleting unused elasticsearch indices in eqiad (T112863)
09:20 _joe_: restarting etcd on conf1001
09:16 jynus: rebooting and installing jessie on db2060-db2070
05:48 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Oct 26 05:48:19 UTC 2015 (duration 48m 18s)
02:31 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-26 02:31:46+00:00
02:27 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 14s)

2015-10-25

05:36 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Oct 25 05:36:33 UTC 2015 (duration 36m 32s)
02:28 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-25 02:28:08+00:00
02:23 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 07m 59s)

2015-10-24

19:34 twentyafterfour: deployed https://gerrit.wikimedia.org/r/#/c/248638/ and restarted apache on iridium
06:00 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Oct 24 06:00:53 UTC 2015 (duration 0m 52s)
05:44 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/includes/Data/Index/FeatureIndex.php: (no message) (duration: 00m 17s)
05:11 hoo: Set an email address for user "Ymnes", after request. Confirmed by several, including.
02:37 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-24 02:37:34+00:00
02:33 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 23s)
00:21 ebernhardson: started second machine (nobelium) performing copy of elasticsearch indices to codfw with 40 threads

2015-10-23

23:31 Krinkle: mwscript deleteEqualMessages.php --wiki hewikisource
23:31 Krinkle: mwscript deleteEqualMessages.php --wiki ukwiki
23:09 hoo: Started a local rename for Stefan4 to Stefan2 on commons, per request (and attributed to) DerHexer
22:23 robh: powercycled nobelium
21:24 mutante: nobelium - powercycled, no console output
20:04 jynus: rename user job had created lag on almost all enwiki dbs, things should be better now
19:08 urandom: starting nodetool cleanup on restbase-test2001-b
19:07 urandom: starting nodetool cleanup on restbase-test2001-a
18:16 godog: bounce grafana-server on krypton
17:58 YuviPanda: moved mounts around on nobelium, mounted bigger disk on /var/lib/elasticsearch
17:57 godog: delete wikimedia-grid grafana dashboard (saved a copy first) heavy graphite queries
17:53 YuviPanda: stopping elasticsearch on nobelium for https://phabricator.wikimedia.org/T114856
17:13 akosiaris: enable_notifications=0 in neon's icinga for a few mins while the storm dies down
17:04 cwd: updated payments from 4d38158f3d3a3a6da85d809a6cdc557e46c45d0c to 71d2d927f4efac1a639aaf7627c765f48d1b129c
16:47 godog: stop puppet on graphite1001 and graphite-index cron, suspected root cause
16:26 godog: uwsgi timing out while serving requests, bounce also carbon daemons
16:20 godog: bounce uwsgi for graphite-web on graphite1001
16:14 urandom: starting rebuild of restbase-test2002-b
15:42 jynus: copied wmf-mariadb10 (10.0.16-2) .deb from trusty to jessie on apt.wikimedia.org
15:42 urandom: bouncing restbase-test2002-a Just To See
15:06 urandom: running nodetool cleanup on restbase-test2003
14:47 godog: remove outdated cassandra metrics from graphite2001
14:27 godog: remove restbase-test2001 restbase-test2002 cassandra metrics
14:12 mobrovac: restbase rolling-restarting after applying aab840f
13:55 jynus: Rebooting and installing jessie on db2055-db2059
13:45 godog: remove 98percentile 999percentile meanRate stddev cassandra metrics after https://gerrit.wikimedia.org/r/#/c/248313/
13:34 godog: remove 15MinuteRate cassandra metrics after https://gerrit.wikimedia.org/r/#/c/248313/
11:51 godog: roll-restart cassandra-metrics-collector on restbase cluster after https://gerrit.wikimedia.org/r/248313
10:27 jynus: retrying schema change on EventLogging (db1046) after failure
10:21 jynus: End of online schema change to geo_tags; all wikis on dblist have been updated
09:34 godog: reimage restbase-test2002.codfw.wmnet
09:23 jynus: restarting schema change on geo_tags for all wikis
09:17 mobrovac: restbase rolling-restart after config changes
06:55 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/extensions/AbuseFilter/AbuseFilterTokenizer.php: I65d4c6064: Track tokenizer cache hits / misses (duration: 00m 18s)
06:55 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/extensions/AbuseFilter/AbuseFilterTokenizer.php: I65d4c6064: Track tokenizer cache hits / misses (duration: 00m 17s)
05:53 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Oct 23 05:53:43 UTC 2015 (duration 53m 42s)
03:15 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.3/includes/profiler/TransactionProfiler.php: 5ef4a91480ea (duration: 00m 18s)
02:41 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-23 02:41:25+00:00
02:36 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 04s)
00:10 logmsgbot: krenair@tin Synchronized w/static/images/project-logos/anwiki.png: https://gerrit.wikimedia.org/r/#/c/247253/ (duration: 00m 16s)

2015-10-22

23:58 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/247850/ (duration: 00m 17s)
23:49 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/VisualEditor/modules/ve-mw/ui/dialogs/ve.ui.MWMediaDialog.js: https://gerrit.wikimedia.org/r/#/c/248273/ (duration: 00m 18s)
23:13 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/container.php: https://gerrit.wikimedia.org/r/#/c/248229/ (duration: 00m 17s)
23:13 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/autoload.php: https://gerrit.wikimedia.org/r/#/c/248229/ (duration: 00m 17s)
23:12 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/includes/SpamFilter/RateLimits.php: https://gerrit.wikimedia.org/r/#/c/248229/ (duration: 00m 17s)
22:55 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/extensions/Cite: extensions/Cite update which fell off the SWAT train yesterday (duration: 00m 19s)
22:02 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaMaintenance/getJobQueueLengths.php: Ie95ec067da9: getJobQueueLengths: add '--report' option for StatsD reporting (duration: 00m 18s)
22:02 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/extensions/WikimediaMaintenance/getJobQueueLengths.php: Ie95ec067da9: getJobQueueLengths: add '--report' option for StatsD reporting (duration: 00m 18s)
21:11 logmsgbot: reedy@tin Synchronized docroot and w: Add more dblist symlinks (duration: 00m 18s)
21:10 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.3/includes/changetags/ChangeTags.php: e7126ed331109 (duration: 00m 17s)
21:05 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.3/includes: a6262272c9666d (duration: 00m 23s)
20:39 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.3/extensions/ZeroBanner/: Deploying ZeroBanner T116309 patch 248239 (duration: 00m 18s)
20:38 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.3/extensions/MobileFrontend: Deploying MobileFrontend T116309 patch 248238 (duration: 00m 36s)
20:26 bd808: Removed "zirconium.wikimedia.org" from Trebuchet's minion list for iegreview/iegreview
20:12 bd808: Updated iegreview.wikimedia.org to bcaf23b (Fix logger usage in Controllers\Account\Recover)
19:53 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.3/extensions/ZeroBanner: Deploying ZeroBanner T116309 patch 248116 (duration: 00m 18s)
19:28 bd808: Forced ELK Elasticsearch to allocate replica of logstash-2015.10.22 shard 0 on logstash1004
18:56 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: all wikis to 1.27.0-wmf.3
16:48 bd808: Restarted Elasticsearch on logstash1004
16:39 logmsgbot: kartik@tin Synchronized private/PrivateSettings.php: T116134: Set CX JWT token (duration: 00m 17s)
16:19 logmsgbot: ori@tin Synchronized wmf-config/InitialiseSettings.php: I8f690589: Increase abusefilter emergency disable threshold on MediaWiki.org (duration: 00m 17s)
15:30 logmsgbot: thcipriani@tin Synchronized docroot/noc/conf/highlight.php: SWAT: Remove urlencode from phabricator links gerrit:248049 (duration: 00m 17s)
15:13 logmsgbot: thcipriani@tin Synchronized docroot/noc/conf: SWAT: noc: change Gitblit links to Diffusion gerrit:248027 (duration: 00m 17s)
15:11 moritzm: uploaded openjdk-8 8u66-b17 for jessie-wikimedia to carbon
14:46 jynus: performing schema change on eventlogging database on db1046
14:41 mutante: mw1083 - Error: Could not run Puppet configuration client: Read-only file system
14:25 jynus: setting thread_pool_size to 32 dynamically on all MariaDB hosts
14:01 jynus: performing schema change on officewiki-flow (s3)
13:19 jynus: performing schema change on x1-master (flowdb)
12:31 jynus: Rolling schema change for GeoData on all wikis (geo_tags)
11:35 mobrovac: restbase deployed 2bc05f40
07:41 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/includes/jobqueue/JobQueueRedis.php: Ie7c544fc8: jobqueue: track real job inserts as inserts_actual & I627e8f6ce: JobQueueRedis::doBatchPush(): report metrics even when failures occur (duration: 00m 17s)
07:41 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/includes/jobqueue/JobQueueRedis.php: Ie7c544fc8: jobqueue: track real job inserts as inserts_actual & I627e8f6ce: JobQueueRedis::doBatchPush(): report metrics even when failures occur (duration: 07m 33s)
06:17 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Oct 22 06:17:39 UTC 2015 (duration 17m 38s)
06:14 AaronSchulz: Restarted hhvm on mw1011, it was stuck doing nothing at 100 cpu
05:10 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.3/includes/deferred/LinksUpdate.php: fe323f9b68bbb (duration: 00m 17s)
03:05 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-22 03:05:21+00:00
03:00 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 07m 52s)
02:37 AaronSchulz: Started running 5 threads of enwiki refreshLinks jobs on tin
02:35 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-22 02:35:41+00:00
02:30 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 44s)
01:10 logmsgbot: krenair@mira Synchronized README: testing sync from mira (duration: 00m 17s)
01:01 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/includes/Hooks.php: I0e5f2d3b2: Make hookErrorHandler() only care about serious signature errors (duration: 00m 17s)
01:00 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/includes/Hooks.php: I0e5f2d3b2: Make hookErrorHandler() only care about serious signature errors (duration: 00m 17s)
00:56 logmsgbot: krenair@mira Synchronized README: (no message) (duration: 00m 16s)
00:55 logmsgbot: krenair@mira Synchronized README: (no message) (duration: 00m 17s)
00:49 logmsgbot: krenair@mira Synchronized README: (no message) (duration: 00m 16s)
00:20 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: Ibf752b832: $wgMathCheckFiles = false (duration: 00m 18s)
00:05 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/extensions/AbuseFilter: Ice1b6da43: AbuseFilter: don't install custom error handler and I0ecdcdd142: Use isset() to check array element exists rather than relying on @ operator (duration: 00m 18s)
00:03 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/extensions/AbuseFilter: Ice1b6da43: AbuseFilter: don't install custom error handler and I0ecdcdd142: Use isset() to check array element exists rather than relying on @ operator (duration: 00m 18s)

2015-10-21

23:33 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/CirrusSearch: Performance tweaks for corss-dc copy process (duration: 00m 19s)
23:32 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: Performance tweaks for corss-dc copy process (duration: 00m 18s)
23:25 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/resources/: mw.ForeignStructuredUpload: Provide category suggestions from the right wiki (duration: 00m 17s)
23:16 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/resources/: mw.ForeignStructuredUpload: Rearrange messages to always display license name (duration: 00m 18s)
23:09 logmsgbot: ebernhardson@tin Synchronized wmf-config/throttle.php: Add throtle exception for eswiki (duration: 00m 18s)
23:03 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/extensions/AbuseFilter/AbuseFilter.parser.php: Ad-hoc debug logging of AbuseFilter exceptions (duration: 00m 17s)
23:00 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/extensions/AbuseFilter/AbuseFilter.parser.php: Ad-hoc debug logging of AbuseFilter exceptions (duration: 00m 17s)
21:10 ebernhardson: starting copy of elasticsearch eqiad indices to codfw
20:46 ebernhardson|lch: cancel copying elasticsearch eqiad to labsearch, looks to be writing to wrong disks and will fill up
20:16 AaronSchulz: Started running 8 threads of commonswiki refreshlinks jobs on terbium
20:14 ebernhardson|lch: starting copy of elasticsearch indices from eqiad cluster to labsearch cluster
18:42 ebernhardson: initializing elasticsearch index mapping for all wikis in the codfw and labsearch ES clusters
18:12 ejegg|mtg: updated crm from 22dc4bd7d041126a1d2a0d4acb9a288bfdc1b435 to f2fa7b942625b34ede520e11f20e7e0835ecb17d
16:53 hoo: Attached WeiaR@enwiki to the global account of the same name. T115699
15:50 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/WikimediaEvents/: Revert SearchSatisfaction schema related changes due to suspected perf impact (duration: 00m 18s)
15:49 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/: Revert SearchSatisfaction schema related changes due to suspected perf impact (duration: 00m 18s)
15:38 akosiaris: depooled mw1083
15:29 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: touch and re-sync InitialiseSettings.php to bust cache (duration: 00m 17s)
15:24 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: CirrusSearch multi-datacenter configuration (duration: 00m 17s)
15:24 logmsgbot: ebernhardson@tin Synchronized wmf-config/CommonSettings.php: CirrusSearch multi-datacenter configuration (duration: 00m 17s)
15:23 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-common.php: CirrusSearch multi-datacenter configuration (duration: 00m 17s)
15:23 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-production.php: CirrusSearch multi-datacenter configuration (duration: 00m 17s)
15:18 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/Cite/Cite_body.php: Do not double-parse error references duplicate key (duration: 00m 17s)
15:16 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/Cite/Cite_body.php: Do not double-parse error references duplicate key (duration: 00m 19s)
15:00 aude: rsync failed on mw1083: "failed to set times on "/srv/mediawiki/wmf-config": Read-only file system"
14:49 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable GeoData on test.wikidata (duration: 00m 18s)
14:34 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings-labs.php: enabled geodata on beta wikidata (duration: 00m 18s)
14:08 jynus: applying schema change to testwikidatawiki (s3) and wikidatawiki(s5)
13:56 jynus: performing schema change on testwiki.geo_tags
13:35 jynus: stopping and fixing replication on labsdb1004 (not in production)
08:40 mobrovac: restbase deployment of 3006b77e Done
08:32 mobrovac: restbase deploying 3006b77e
07:36 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Oct 21 07:36:04 UTC 2015 (duration 36m 3s)
03:13 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-21 03:13:42+00:00
03:09 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 24s)
02:42 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-21 02:42:44+00:00
02:37 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 05s)
01:12 logmsgbot: hoo@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 17s)
01:03 logmsgbot: hoo@tin Synchronized wmf-config/interwiki.cdb: revert (duration: 00m 18s)
01:03 logmsgbot: hoo@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 17s)
00:29 logmsgbot: legoktm wikidata'd everything
00:22 logmsgbot: hoo@tin Synchronized dblists/: Enable WikibaseClient on mediawikiwiki, metawiki and specieswiki (duration: 00m 17s)
00:19 logmsgbot: hoo@tin Synchronized wmf-config/: Enable WikibaseClient on mediawikiwiki, metawiki and specieswiki (duration: 00m 19s)

2015-10-20

23:54 logmsgbot: hoo@tin Synchronized wmf-config/: Add MediaWiki, Meta-Wiki and Wikispecies to Wikibase special site groups (duration: 00m 18s)
23:47 logmsgbot: hoo@tin Synchronized wmf-config/: Add MediaWiki, Meta-Wiki and Wikispecies to Wikibase special site groups (testwikidata) (duration: 00m 18s)
23:36 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/WikimediaEventsHooks.php: https://gerrit.wikimedia.org/r/#/c/247650/ (duration: 00m 17s)
23:26 logmsgbot: hoo@tin Synchronized wmf-config/: Revert "Temporarily disable "item-merge" right on Wikidata" (duration: 00m 18s)
23:25 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.3/includes/deferred: 2a1e1d7dd88a62aba9 (duration: 00m 17s)
23:22 logmsgbot: hoo@tin Synchronized wmf-config/: Bump the cache epoch for (test)wikidata (duration: 00m 18s)
23:20 logmsgbot: hoo@tin Finished scap: Update Wikibase to wmf3b and add messages for sitelinks to MediaWiki, Meta-Wiki and Wikispecies (duration: 48m 44s)
23:16 mutante: mw1232: restarted hhvm
22:31 logmsgbot: hoo@tin Started scap: Update Wikibase to wmf3b and add messages for sitelinks to MediaWiki, Meta-Wiki and Wikispecies
21:11 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/includes/page/WikiPage.php: I5d0440588d: Make triggerOpportunisticLinksUpdate() directly use RefreshLinks (T116001) (duration: 00m 17s)
21:11 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/includes/page/WikiPage.php: I5d0440588d: Make triggerOpportunisticLinksUpdate() directly use RefreshLinks (T116001) (duration: 00m 18s)
20:23 gwicke: reverted restbase deploy on restbase1001 to a4c55e40
20:15 mutante: starting restbase on restbase1001
20:09 mutante: errors argon' on argon
20:04 ottomata: uninstalling hadoop packages on analytics1017
19:46 ori: previous syncs was of I46200d4edb3: Revert "Revert "Revert "Enable config for all three search clusters, but only write to eqiad"""
19:45 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 18s)
19:45 logmsgbot: ebernhardson@tin Synchronized wmf-config/: (no message) (duration: 00m 18s)
18:25 mutante: re-enabling icinga notifications for icinga (neon) itself that were disabled for some reason though all OK
16:50 mutante: temp. disabled ircecho / neon puppet
16:50 jynus: enabling query profiling on a sample of queries on db1072
16:27 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings.php: Revert Temporarily increase redis logging to debug (duration: 00m 18s)
16:25 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings.php: Temporarily increase redis logging to debug (duration: 00m 17s)
16:21 aude: re-populated sites table on metawiki, mediawikiwiki and specieswiki with https protocol links
16:13 cwdent: updated fundraising crm from 738e8c3f8079765841ed4c5f79ecf066c541c7b9 to 22dc4bd7d041126a1d2a0d4acb9a288bfdc1b435
15:49 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Revert "Enable config for all three search clusters, but only write to eqiad"" gerrit:247478 Part V (duration: 00m 18s)
15:48 godog: bump netdev_max_backlog to 10000 on graphite1001, T101141
15:46 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Revert "Enable config for all three search clusters, but only write to eqiad"" gerrit:247478 Part IV (duration: 00m 17s)
15:46 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Revert "Revert "Enable config for all three search clusters, but only write to eqiad"" gerrit:247478 Part III (duration: 00m 17s)
15:45 logmsgbot: thcipriani@tin Synchronized wmf-config/CirrusSearch-common.php: SWAT: Revert "Revert "Enable config for all three search clusters, but only write to eqiad"" gerrit:247478 Part II (duration: 00m 17s)
15:44 logmsgbot: thcipriani@tin Synchronized wmf-config/CirrusSearch-production.php: SWAT: Revert "Revert "Enable config for all three search clusters, but only write to eqiad"" gerrit:247478 Part I (duration: 00m 17s)
15:27 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Enable ContentTranslation suggestion in all Wikipedia gerrit:247515 (duration: 00m 17s)
15:23 awight: update fundraising CRM from 6ad0ad090c23ee41003138a6676131abf70c72f4 to 738e8c3f8079765841ed4c5f79ecf066c541c7b9
15:21 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Switch graphoid to the local restbase proxy gerrit:247494 (duration: 00m 17s)
15:14 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Move ForeignUploadTargets config to production gerrit:246703 part 2 (duration: 00m 17s)
15:14 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Move ForeignUploadTargets config to production gerrit:246703 part I (duration: 00m 18s)
14:33 cmjohnson1: removing tele2(patchid 2953) from dmarc panel @eqiad
14:12 ottomata: deployed varnishreqstats diamond collector to remaining varnish caches
13:48 jynus: backing up and renaming user_daily_contribs table from all wikis as a previous step for its deletion
12:44 kart_: Update cxserver to 6452b68
12:01 logmsgbot: aude@tin Synchronized wmf-config/Wikibase.php: Temporarily disallow item-merge until T115892 is resolved (duration: 00m 19s)
11:16 mobrovac: mathoid deploying 8e1a3327
07:23 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Oct 20 07:23:56 UTC 2015 (duration 23m 55s)
03:14 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-20 03:14:15+00:00
03:09 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 16s)
02:48 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch/: Bring phase0 and phase1 inline with phase2 (duration: 00m 18s)
02:48 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/Elastica/: Bring phase0 and phase1 inline with phase2 (duration: 00m 21s)
02:44 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-20 02:43:53+00:00
02:39 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 20s)

2015-10-19

23:43 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: testwiki Graphoid to restbase (duration: 00m 17s)
23:13 logmsgbot: ebernhardson@tin Synchronized wmf-config/: resync after touching InitialiseSettings.php to bust caches (duration: 00m 18s)
23:12 logmsgbot: ebernhardson@tin Synchronized wmf-config/: Revert multidc cirrussearch config, seeing unexplained errors on commonswiki (duration: 00m 18s)
23:11 mutante: restarted hhvm on mw1231
23:09 logmsgbot: ebernhardson@tin Synchronized wmf-config/throttle.php: Add throttle exception for dewiki (duration: 00m 17s)
23:08 mutante: restarted apache on mw1231
23:07 logmsgbot: ebernhardson@tin Synchronized wmf-config/: Disable cirrus suggester AB test (duration: 00m 17s)
23:05 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/WikimediaEvents/: Bump sampling rate of common terms test from 1:1000 to 1:200 (duration: 00m 17s)
23:01 logmsgbot: ebernhardson@tin Synchronized tests/: noop sync mediawiki-config test dir (duration: 00m 17s)
22:42 logmsgbot: ebernhardson@tin Synchronized wmf-config/: Enable cirrusearch multi cluster configuration, only write to eqiad (duration: 00m 18s)
22:40 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/CirrusSearch/: Handle ElasticaWrite job failures internally (duration: 00m 18s)
22:29 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/CirrusSearch/: Deploy multi-dc cirrusearch code for CirrusSearch extension (duration: 00m 18s)
22:28 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/Elastica/: Deploy multi-dc cirrusearch code for Elastica extension (duration: 00m 17s)
22:20 ebernhardson: sync-common on mw1017 to pre-test cirrussearch multi-dc deployment
22:17 paravoid: salt-run deluser --delete-home gmetric; delgroup systemusers
20.23 subbu: cherrypick deploy for parsoid completed: b317f33f and 60a82ae0 cherrypicked from parsoid master
15:42 logmsgbot: anomie@tin Started scap: SWAT: Add a change tag to cross-wiki uploads gerrit:246701
15:39 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.2/extensions/Cite: SWAT: Display 'cite_error_references_duplicate_key' next to the affected ref gerrit:247255 (duration: 00m 18s)
15:30 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.3/extensions/Cite: SWAT: Display 'cite_error_references_duplicate_key' next to the affected ref gerrit:247256 (duration: 00m 18s)
15:25 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.2/includes/deferred/LinksDeletionUpdate.php: SWAT: Use specified pageId for LinksDeletionUpdate→DeleteLinksJob gerrit:247268 (duration: 00m 18s)
15:24 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.3/includes/deferred/LinksDeletionUpdate.php: SWAT: Use specified pageId for LinksDeletionUpdate→DeleteLinksJob gerrit:247267 (duration: 00m 17s)
15:12 urandom: deploying a4c55e40 to RESTBase
15:11 joal: Scheduling icinga downtime for CQL checks on aqs while heavily loading data - joal (me) babysites the jobs - 1 day downtime, will reiterate tomorrow if needed
15:09 ottomata: enabling varnish reqstats diamond collector on all upload caches
15:09 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/includes/Search/Connection.php: SWAT: Backport gerrit:246134 because the thing it fixed suddenly started breaking unit tests, preventing other merges (duration: 00m 18s)
15:07 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.2/extensions/Flow/includes/Search/Connection.php: SWAT: Backport gerrit:246134 because the thing it fixed suddenly started breaking unit tests, preventing other merges (duration: 00m 18s)
14:52 akosiaris: disabled puppet on maps-test200{1,2,4}. Debugging cassandra multi-instance setup aftermath. Not to be enabled
14:34 urandom: canary deploy (a4c55e40) to restbase1001.eqiad
12:47 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-isl-eng_0.1.0~r20599-1
12:22 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-isl_0.1.0-1
06:19 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Oct 19 06:19:09 UTC 2015 (duration 19m 8s)
03:04 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-19 03:04:49+00:00
02:59 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 40s)
02:36 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-19 02:36:53+00:00
02:32 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 15s)

2015-10-18

10:04 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-mlt-ara_0.1.0~r57554-1
10:04 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-is-sv_0.1.0~r56030-1
10:04 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-es-ro_0.7.3~r57551-1
10:04 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-es-it_0.1.0~r51165-1
06:11 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Oct 18 06:11:37 UTC 2015 (duration 11m 36s)
02:57 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-18 02:57:48+00:00
02:52 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 31s)
02:29 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-18 02:29:50+00:00
02:25 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 25s)

2015-10-17

20:02 godog: powercycle analytics1034, no console no ssh
14:05 godog: reboot krypton, unable to ssh and no console (VM) iowait through the roof
06:27 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Oct 17 06:27:04 UTC 2015 (duration 27m 3s)
03:03 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-17 03:02:58+00:00
02:58 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 14s)
02:35 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-17 02:35:14+00:00
02:30 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 09s)

2015-10-16

20:52 hashar: Restarting Jenkins to remove potential dead locks before the week-end
20:48 ejegg: updated payments from 33b3bd6bee11b3cc9de1570584a23354d0b6525f to 4d38158f3d3a3a6da85d809a6cdc557e46c45d0c
19:29 ori: deleted /var/lib/carbon/whisper/MediaWiki/MediaWiki on graphite1001 & graphite2001 per tgr's request
16:14 awight: Update paymentswiki logging config per T107918
15:52 awight: Fundraising: remove FR fraud exception
13:18 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I14ecd0ae87: Turn off UserDailyContribs extension (duration: 00m 18s)
06:25 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Oct 16 06:25:56 UTC 2015 (duration 25m 55s)
03:11 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-16 03:11:14+00:00
03:06 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 31s)
02:43 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-16 02:43:42+00:00
02:38 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 38s)
02:27 jynus: repairing table and restarting replication on s7 from labsdb1002
02:23 jynus: repairing table and restarting replication on s5 from dbstore1002 (non-production host)
01:36 Jamesofur: reset email address for User:INeverCry after identify verification

2015-10-15

22:09 greg-g: don't worry, _joe_ was around and we approved Roan's last deploy as an exception :)
22:08 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/: Back out categories in sidebar feature (duration: 00m 20s)
20:30 _joe_: rebooting ms-be1011
19:41 awight: update crm from ddfaa209f5b5af4aa0cf3403da91d39b3c52acc1 to a4e74f6f38e6cff16ecf79d28d6e4de9499a5017
19:08 cwdent: updated payments from bdec3220030a396e2a447763e40b940a332e2ab8 to 33b3bd6bee11b3cc9de1570584a23354d0b6525f
18:41 ejegg: updated refund queue name in IPN listener settings
18:13 ejegg: updated CiviCRM from 17dc351d92a8437c93b4a9fa2385840b3581dad6 to ddfaa209f5b5af4aa0cf3403da91d39b3c52acc1
17:18 cwdent: updated payments from ba8f80ec7a858074cd3856a52f3758cf96571f67 to bdec3220030a396e2a447763e40b940a332e2ab8
09:54 _joe_: restarted gitblit, unresponsive
05:41 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Oct 15 05:41:25 UTC 2015 (duration 41m 24s)
02:51 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-15 02:51:27+00:00
02:49 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 05m 02s)
02:35 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-15 02:35:15+00:00
02:31 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 06m 57s)

2015-10-14

23:46 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: Nevermind on skipping timed text, extension config is weird (duration: 00m 17s)
22:26 akosiaris: restarted zotero on sca1002
21:54 akosiaris: restarted zotero on sca1001. complaining out of memory in logs
20:46 ejegg: updated civicrm from ba39f3181431c8409416e580dcf6e15ac5f96a21 to 17dc351d92a8437c93b4a9fa2385840b3581dad6
20:20 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: Also exclude timedtext ns from purges (duration: 00m 18s)
19:52 robh: reenabled puppet agent on etherpad1001, was disabled for a few hours and no reason specified and no SAL entry
19:50 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Turn off the mw "warning" logging channel (duration: 00m 18s)
19:21 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: Tighten check on empty page purges (duration: 00m 18s)
19:11 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I1d7969a9d: Purge pages with blank content beyond the PP limit report, I38cba9d9b6a: Get strpos() parameters correct (duration: 00m 17s)
19:10 logmsgbot: ori@tin Synchronized wmf-config/InitialiseSettings.php: I3a8a750bb3: Enable T115505 log channel for I1d7969a (duration: 00m 17s)
18:06 logmsgbot: demon@tin Synchronized wmf-config: (no message) (duration: 00m 19s)
17:59 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 17s)
17:58 logmsgbot: demon@tin Synchronized wmf-config/extension-list: (no message) (duration: 00m 17s)
15:31 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: SWAT: Split connection to source and target gerrit:246243 (duration: 00m 18s)
15:19 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.2/extensions/UploadWizard/UploadWizard.config.php: SWAT: Remove default category for UploadWizard files gerrit:246225 (duration: 00m 17s)
15:14 ori: restarted navtiming and statsv services on hafnium
15:11 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.3/extensions/UploadWizard/UploadWizard.config.php: SWAT: Remove default category for UploadWizard files gerrit:246226 (duration: 00m 18s)
15:05 Krinkle: Statsv and eventlogging-navtiming seems to have gone down 7 hours ago
14:24 ori: cleaned up tessera leftovers on graphite1001
14:15 mutante|1way: mw1157 - deleted puppet lock file, fix puppet run. ("already running" but didnt since 18h)
13:30 ottomata: kafka-preferred-replica election after kafka1012's broker restarted last night
13:12 ottomata: restarted diamond, puppet didn't seem to after it removed the TcpConnStates from most hosts
12:55 hashar: ERROR: Could not connect to SMTP host: polonium.wikimedia.org, port: 25 (from labs instances)
12:32 logmsgbot: krenair@tin Synchronized README: testing https://gerrit.wikimedia.org/r/246206 (duration: 00m 17s)
12:07 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/VisualEditor/modules/ve-mw/ui/inspectors/ve.ui.MWLinkAnnotationInspector.js: https://gerrit.wikimedia.org/r/#/c/246205/ (duration: 01m 13s)
03:14 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Oct 14 03:14:22 UTC 2015 (duration 14m 21s)
03:14 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-14 03:14:22+00:00
03:07 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 10m 37s)
02:40 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-14 02:40:43+00:00
02:37 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 07m 10s)
00:19 ejegg: updated civicrm from c7af7634e75eb8702f5e16081a86ab86ce69c7c2 to ba39f3181431c8409416e580dcf6e15ac5f96a21

2015-10-13

23:30 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Set $wgUploadNavigationUrl to use uselang=$lang for commonsuploads wikis by default (duration: 01m 14s)
23:27 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/WikimediaEvents/: Turn on cirrus common terms test (duration: 01m 15s)
23:25 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/CirrusSearch/: Add information for common terms ab test (duration: 01m 15s)
23:21 Krenair: restarted apache on silver
23:19 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/Flow: Make flow board descriptions editable again (duration: 01m 16s)
23:13 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/Flow: Bump flow submodule in 1.27.0-wmf.3 (duration: 01m 15s)
23:10 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/: Update WME for common terms AB test (duration: 01m 14s)
23:08 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch/: Update cirrus for common terms AB tes (duration: 01m 15s)
23:02 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Turn on logging of the "warning" channel (duration: 01m 13s)
23:00 ejegg: updated SmashPig from b5ff2a7d5f17aaaa33a169ca101cbea639769c90 to c431c8d77521270236c72532a50806b2e852cf7b
21:05 ottomata: reenabling puppet on cp1052
20:51 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.3
20:48 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/includes/db: I0e5f2d3b2: Revert Enforce lagged-slave read-only mode on the DB layer (duration: 01m 14s)
20:45 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/autoload.php: (no message) (duration: 01m 14s)
20:40 akosiaris: restart gitblit (once more)
20:27 _joe_: rebooting mw1157, stuck in a kernel soft lockup
20:22 ori: locally hacked testwiki and mediawikiwiki to point to php-1.27.0-wmf.3 on mw1017
20:19 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.2
20:18 logmsgbot: ori@tin rebuilt wikiversions.cdb and synchronized wikiversions files: (no message)
20:00 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.3
19:59 ejegg: updated civicrm f25fbe856b92373104985185db77311ea3a4d841 to c7af7634e75eb8702f5e16081a86ab86ce69c7c2
19:38 JohnFLewis: delete wikidata-l mailing list (archives still accessible)
19:30 logmsgbot: twentyafterfour@tin Finished scap: Full scap sync for 1.27.0-wmf.3 (duration: 31m 26s)
18:58 logmsgbot: twentyafterfour@tin Started scap: Full scap sync for 1.27.0-wmf.3
18:45 ottomata: FYI i have puppet disabled on cp1052 while I try to figure out a diamond+VSL+multiprocessing bug
18:11 mutante|1way: started salt on mw2083
18:09 mutante|1way: restarted gitblit
17:58 ori: testing ldap integration in grafana 2
17:54 urandom: performing deploy of a4c55e4 to restbase staging
17:42 urandom: performing canary deploy of a4c55e4 to restbase staging (xenon)
16:14 ostriches: created education program tables for srwiki, T110619
16:14 mobrovac: restbase deploying a01c62a6
16:08 ottomata: restarting diamond on cp1052 in gdb in attempt to figure out why vanrishreqstats segfaults...
15:25 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable CX suggestions for de, fa, fi, he, nn, pa, pl and te wikipedias gerrit:245862 (duration: 01m 13s)
15:18 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Set $wgUploadNavigationUrl to use uselang=$lang for commonsuploads wikis by default" (duration: 01m 13s)
15:15 ori: grafana-test: Imported Grafana dashboards from ElasticSearch
14:12 bblack: note to self: we should migrate all our service to Java
14:11 bblack: restarted gitblit on antimony
13:16 bblack: repooling cp2017 (codfw upload)
02:32 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Oct 13 02:32:17 UTC 2015 (duration 32m 16s)
02:32 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-13 02:32:17+00:00
02:28 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 07m 01s)
01:53 logmsgbot: faidon@tin Synchronized wmf-config/CommonSettings.php: unbreak BounceHandler (duration: 01m 14s)
00:51 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Reapply: Flow-occupy talk namespaces on sewikimedia (duration: 01m 13s)
00:29 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.2/extensions/Flow/modules/flow-initialize.js: https://gerrit.wikimedia.org/r/#/c/245596/ (duration: 01m 13s)
00:21 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/244724/ (duration: 01m 13s)
00:10 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/245608/ (duration: 01m 13s)
00:04 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: rv (duration: 01m 13s)
00:00 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/245588/ and https://gerrit.wikimedia.org/r/#/c/245589/ (duration: 01m 13s)

2015-10-12

23:53 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/245578/ (duration: 01m 12s)
23:38 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/243920/ (duration: 01m 14s)
23:35 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/243921/ (duration: 01m 13s)
23:26 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/244378/ (duration: 01m 13s)
23:23 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/244896/ (duration: 01m 14s)
23:16 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/245194/ (duration: 01m 13s)
23:11 logmsgbot: krenair@tin Synchronized dblists: https://gerrit.wikimedia.org/r/#/c/243517/ (duration: 01m 13s)
23:06 logmsgbot: krenair@tin Synchronized database lists: (no message) (duration: 01m 13s)
20:21 bearND: MobileApps deployed sha1 95293e5
19:09 Tim: on ruthenium installed iotop for stall investigation
16:13 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.2/extensions/ZeroBanner: SWAT: Defer loading of ZeroOverlay until needed gerrit:244737 (duration: 01m 13s)
16:00 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: SWAT: Throttle rule for Ada Lovelace Day editathon 2015 gerrit:245472 (duration: 01m 13s)
15:55 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Extension:ShortURL on bnwiki gerrit:244953 (duration: 01m 14s)
15:39 logmsgbot: thcipriani@tin Synchronized dblists/commonsuploads.dblist: SWAT: Remove duplicate entries from commsuploads.dblist gerrit:244435 (duration: 01m 12s)
15:33 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Use new page name for wmf release notes gerrit:241079 (duration: 01m 14s)
15:26 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Rename Azerbaijani Wikisource project and namespaces gerrit:242096 (duration: 01m 13s)
15:19 logmsgbot: thcipriani@tin Synchronized wmf-config/Wikibase-production.php: SWAT: Add GeoData and PageImages configuration for Wikibase repo wikis gerrit:244165 (duration: 01m 13s)
15:14 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Explicitly set wmgMFNearby = false for wikidata gerrit:244591 (duration: 01m 14s)
15:07 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Modify timezone for cswiktionary gerrit:244649 (duration: 01m 12s)
14:54 dcausse: closing unused cirrus indices in eqiad (T112863)
14:28 cmjohnson: rebooting mw1154
12:10 yurik: deployed kartotherian & tilerator to maps-test200{1-4}
08:54 hashar: zuul-merger process leaked file descriptors and ended up unable to open any more files. Fixed by restarting the service on gallium. https://phabricator.wikimedia.org/T115243
08:44 hashar: Zuul CI in trouble. zuul-merger can't not apply patches anymore https://phabricator.wikimedia.org/T115243
02:33 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Oct 12 02:33:13 UTC 2015 (duration 33m 12s)
02:33 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-12 02:33:13+00:00
02:30 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 06m 50s)

2015-10-11

04:57 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Oct 11 04:57:40 UTC 2015 (duration 57m 39s)
02:28 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-11 02:28:34+00:00
02:25 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 06m 17s)

2015-10-10

06:40 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Oct 10 06:40:04 UTC 2015 (duration 40m 3s)
02:41 ejegg|afk: enabled fundraising banner campaigns
02:36 ejegg|afk: updated payments-wiki from 24d5be6886d34b3600031290c7f55ee84f3dcee2 to ba8f80ec7a858074cd3856a52f3758cf96571f67
02:22 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-10 02:22:30+00:00
02:19 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 05m 58s)

2015-10-09

22:20 ejegg: took down Fundraising campaigns
19:33 logmsgbot: ori@tin Synchronized multiversion/MWWikiversions.php: I9d4cbd3d67: Provide a smooth migration path of dblist files to dblists/ (duration: 01m 13s)
18:13 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.2/extensions/OpenStackManager/nova/OpenStackNovaProject.php: https://gerrit.wikimedia.org/r/#/c/244707/ (duration: 01m 13s)
16:04 godog: rolling restart cassandra test cluster T95253
15:35 urandom: performing Cassandra cleanup on restbase-test2003.codfw
15:15 urandom: bouncing Cassandra on restbase-test2001-a
15:07 urandom: starting nodetool cleanup on restbase-test2002
14:37 jynus: more configuration testing (with puppet disabled) and several mysql restarts on db1022
13:53 hashar: Restarted Zuul, had a deadlocked job
11:50 godog: bounce mathoid on sca100[12], stray instance found not running firejail
11:22 mobrovac: restbase deploying aaee7c31
11:16 godog: force-run puppet on restbase after merging https://gerrit.wikimedia.org/r/#/c/244656/
11:00 jynus: restarting db1022's mysql (depooled) for configuration testing
09:17 jynus: deployed visual glitch fix to dbtree
09:11 akosiaris: poweroff rhodium, remove salt key, remove puppet storedconfigs in preparation for reinstall reinstall as a VM for temporary puppetmaster testing.
09:11 akosiaris: poweroff sodium, remove salt key, remove puppet storedconfigs in preparation for reinstall reinstall as a VM for temporary puppetmaster testing.
08:20 ori: Purged graphite[12]001:/var/lib/carbon/whisper/servers/*/TcpConnStatesCollector and graphite[12]001:/var/lib/carbon/whisper/servers/*/network/work; cleaning up after https://gerrit.wikimedia.org/r/#/c/244637/
05:13 mutante: @RoanKattouw re: sync-file ssh to mira.codfw.wmnet: fixed! sorry. -> T115075#1714464
02:32 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 06m 11s)
01:24 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.2/extensions/CentralNotice/resources/subscribing/ext.centralNotice.display.js: Add period to try to flush out ResourceLoader issue (duration: 01m 26s)
01:18 RoanKattouw: Getting failures from sync-file / scap because mira.codfw.wmnet doesn't respond to ssh
01:17 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/CentralNotice/resources/subscribing/ext.centralNotice.display.js: Add trailing newline to try to flush out ResourceLoader issue (duration: 02m 15s)

2015-10-08

23:56 logmsgbot: catrope@tin Finished scap: SWAT (duration: 29m 10s)
23:51 yurik: reverted Kartotherian to HEAD^^ - the service wouldn't start
23:27 logmsgbot: catrope@tin Started scap: SWAT
23:06 yurik: deployed kartotherian
18:20 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.27.0-wmf.2
16:45 mobrovac: mathoid deploying 110abaf
15:12 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: Fix labs settings for foreign uploads (syncing out so it doesnt surprise future SWATters) gerrit:244332 (duration: 00m 18s)
15:05 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable CX suggestions in ast, bn, ml, nb, ta and ukwiki gerrit:244142 (duration: 00m 17s)
14:53 paravoid: replacing cr1-codfw<->asw-a-codfw QSFPs
14:48 logmsgbot: reedy@tin Synchronized docroot and w: (no message) (duration: 00m 17s)
14:13 jynus: performing schema change on the m4/analytics/eventlogging databases (db1046, db1047, dbstore2002)
13:59 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Reduce db1035 & db1044 weight, repool at 100% db1051 & db1055 (duration: 00m 17s)
13:11 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Reduce db1035 weight (duration: 00m 16s)
12:09 godog: update facts on puppet compiler from palladium
11:38 _joe_: rebooting mc2002
10:39 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1055 (duration: 05m 10s)
10:37 _joe_: rebooting mw1008, in oom spiral
10:15 paravoid: salt rm /home/*/.ssh/authorized_keys
09:21 jynus: downtime for db1055 for maintenance (kernel update, mysql update, config update)
08:45 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1055 (duration: 00m 17s)
07:44 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Increase load of db1051 and db1055 also for regular traffic (duration: 00m 17s)
07:06 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Matching db1051 and db1055 weight for load balancing (duration: 00m 16s)
06:15 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Oct 8 06:15:16 UTC 2015 (duration 15m 15s)
05:56 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 18s)
03:04 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-08 03:04:52+00:00
03:02 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 05m 33s)
02:47 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-08 02:46:57+00:00
02:40 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 10m 30s)
01:17 twentyafterfour: phabricator is now up and running again
01:09 twentyafterfour: buggy update, stopping apache2 on iridium.
01:02 twentyafterfour: finished phabricator update
00:51 mutante: mw1160 - was oom-killer (convert, mw job 31497)
00:50 twentyafterfour: phabricator maintenance/upgrade. Expect 10 minutes downtime
00:44 mutante: powercycled mw1160 - looks like broken hardware or cable (ata1: lost interrupt)
00:08 logmsgbot: krenair@tin Synchronized w/static/apple-touch/commons.png: https://gerrit.wikimedia.org/r/#/c/243685/ (duration: 00m 17s)

2015-10-07

23:37 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/242842/ (duration: 00m 17s)
23:34 logmsgbot: ori@tin Synchronized wmf-config: I924d8e19e17: Make the redis cache configuration multi-DC-ready (T111575) (duration: 00m 17s)
23:22 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/244177/ (duration: 00m 17s)
23:18 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/244176/ (duration: 00m 17s)
23:08 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/243837/ (duration: 00m 17s)
21:25 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: Revert "Don't use nutcracker on wikitech" (duration: 00m 16s)
20:56 mutante: applied firewalling on IRCd server, rc bot still working fine, all public IRC ports as before
20:25 mutante: argon: installing package upgrades
20:24 mutante: resetting drac on argon
20:14 logmsgbot: ori@tin Synchronized wmf-config/session.php: Ie25c368a: Switch mw1017 to use DC-specific redis cluster names (duration: 00m 17s)
19:50 urandom: RESTBase deploy complete
19:41 urandom: doing full deploy of c20e6336 to RESTBase
19:29 urandom: canary deploy to restbase1001.eqiad complete
19:14 urandom: deploying c20e6336 to canary node restbase1001.eqiad
19:09 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.2
18:56 logmsgbot: twentyafterfour@tin Synchronized wmf-config/CommonSettings.php: fix undefined variable warning that has been spamming logs (duration: 00m 17s)
17:56 awight_: update fundraising crm from 7003cc38797848631d0c4d5f6ff68ab1d6118ad8 to f25fbe856b92373104985185db77311ea3a4d841
16:39 mutante: powercycling analytics1035
16:14 mobrovac: citoid deploying ec149fd5
16:02 logmsgbot: marktraceur@tin Synchronized wmf-config/: Adding new config variable for uploads to Commons (duration: 00m 17s)
15:44 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/MobileFrontend: SWAT (duration: 00m 17s)
15:44 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.2/extensions/Echo: SWAT (duration: 00m 18s)
15:44 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/Echo: SWAT (duration: 00m 17s)
15:13 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1051 after maintenance (duration: 00m 17s)
15:03 logmsgbot: krenair@tin Synchronized visualeditor-default.dblist: https://gerrit.wikimedia.org/r/#/c/242041/ (duration: 00m 17s)
15:02 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/242041/ (duration: 00m 18s)
14:59 mobrovac: AQS restarting restbase on aqs100x
13:00 godog: decomission restbase-test2001 and reimage
12:27 paravoid: salt-rm'ing /var/lib/apt/lists/ubuntu.wikimedia.org_ubuntu_dists_trusty_main_i18n_Translation-en%5fUS
10:55 godog: reenable puppet on restbase / maps-test / aqs
10:10 _joe_: depooling cp1059 from pybal, varnish
08:44 godog: disable puppet on restbase, maps, aqs before merging https://gerrit.wikimedia.org/r/#/c/243127
08:05 moritzm: installed spice security updates on labvirt*
06:01 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Oct 7 06:01:01 UTC 2015 (duration 1m 0s)
05:25 ebernhardson: generating elasticsearch indices in codfw, should run ~3 hours
04:33 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Touch InitiialiSettings.php to force config regeneration (duration: 00m 18s)
04:23 logmsgbot: ebernhardson@tin Synchronized wmf-config/: enable second es cluster in testwiki one more time (duration: 00m 18s)
04:15 ebernhardson: sync-common on mw2187.codfw.wmnet to fix localisation cache errors in exception.log
03:37 logmsgbot: ebernhardson@tin Synchronized wmf-config: redisable second cluster only on testwiki (duration: 00m 16s)
03:35 logmsgbot: ebernhardson@tin Synchronized wmf-config/: Reenable second ES cluster on testwiki only (duration: 00m 18s)
03:10 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-07 03:10:02+00:00
03:03 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 10m 18s)
02:37 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-07 02:37:32+00:00
02:32 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 08m 22s)
01:44 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.2/extensions/Echo: Fix JS error (duration: 00m 18s)
01:44 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/Echo: Fix JS error (duration: 00m 17s)
00:04 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.2/extensions/Echo: SWAT (duration: 00m 17s)
00:04 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/Echo: SWAT (duration: 00m 17s)

2015-10-06

23:20 logmsgbot: ebernhardson@tin Synchronized wmf-config: Revert multicluster config for testwiki (duration: 00m 18s)
23:18 logmsgbot: ebernhardson@tin Synchronized wmf-config/: Enable multicluster ES on testwiki (duration: 00m 17s)
22:44 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.2
22:40 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.2: Deploy https://gerrit.wikimedia.org/r/#/c/244066/ (duration: 01m 40s)
21:36 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.1
21:15 twentyafterfour: restarted phd in response to a phabricator setup issue
20:58 bblack: disabling puppet on caches for VCL testing
20:58 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.2
20:53 logmsgbot: twentyafterfour@tin Finished scap: sync wmf/1.27.0-wmf.1 (duration: 32m 13s)
20:29 mutante: service "ishmael" has been removed (T109777) - removed docroot on neon. tarball exists in /root just in case. code is on https://github.com/asher/ishmael
20:20 logmsgbot: twentyafterfour@tin Started scap: sync wmf/1.27.0-wmf.1
18:48 yuvipanda: fixing https://phabricator.wikimedia.org/T109216 on labstore1002
18:10 godog: upgrade videoscalers to ffmpeg2theora 0.29.0~git+20150813-2
15:04 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Suggestion in af, gl, gu, mk, oc, sh and simplewiki gerrit:243919 (duration: 00m 18s)
11:33 jynus: performing schema change on db1051 enwiki revision
11:10 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1051 for more maintenance (duration: 00m 17s)
10:45 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1051 after maintenance (duration: 00m 17s)
10:41 jynus: potential extra load on mediawiki recent changes and watchlist on enwiki, please report any slowdown
10:22 jynus: dropping temp recovered tables from db1051 to prepare for repool
09:50 _joe_: uploaded a new pybal package for jessie
06:19 paravoid: eqord is back up
04:24 paravoid: all waves to eqord down, probably related to RT#9619
02:34 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-06 02:34:31+00:00
02:29 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 08m 53s)

2015-10-05

23:53 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.1/extensions/ZeroBanner/includes/ZeroSpecialPage.php: https://gerrit.wikimedia.org/r/#/c/243833/ (duration: 00m 17s)
23:39 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: rv (duration: 00m 17s)
23:14 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.1/extensions/VisualEditor/modules/ve-mw/ui: https://gerrit.wikimedia.org/r/#/c/243729/ (duration: 00m 17s)
23:05 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/242942/ (duration: 00m 17s)
21:16 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.1/extensions/ZeroBanner: Take2: Deploying ZeroBanner 242661+243808 (duration: 00m 17s)
21:00 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.1/extensions/ZeroBanner: Rolling back ZeroBanner 242661 (duration: 00m 18s)
20:55 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.1/extensions/ZeroBanner: Deploying ZeroBanner 242661 (duration: 00m 17s)
18:38 mutante: restarted gitblit
17:44 gwicke: rolling restart of eqiad restbase nodes done
17:27 gwicke: rolling restart of restbase cluster to rule out driver issues causing the increased p99 read latency
17:19 robh: analytics1035 pegged out, ssh unresponsive and raid failures, and then fixed itself 5 minutes later
16:32 legoktm: running backPopulateRenameQueueLogs.php (gerrit:237169) on metawiki
16:05 hoo: Updated Wikidata's property suggester with data from today's json dump
15:19 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.1/extensions/Flow: SWAT: Fix exception on board and topic history pages gerrit:243337 (duration: 00m 20s)
14:41 hashar: puppet-lint Jenkins job is now strict and will -1 on errors as well as warnings https://gerrit.wikimedia.org/r/#/c/243185/
10:25 godog: roll-restart cassandra in restbase eqiad
09:58 _joe_: installing the new HHVM package to appservers
09:50 _joe_: rebooted mw1153, soft lockup due to bnx2 failure
09:50 godog: roll-restart cassandra in restbase codfw
09:18 godog: roll-restart cassandra on restbase test cluster
09:15 godog: stop puppet on restbase and maps in preparation for https://gerrit.wikimedia.org/r/#/c/242896/1
08:07 _joe_: upgrading HHVM on all API appservers
05:33 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Oct 5 05:33:18 UTC 2015 (duration 33m 17s)
02:33 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-05 02:33:57+00:00
02:29 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 08m 25s)

2015-10-04

23:36 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/extensions/ContentTranslation/extension.json: 8c80ec1273: Updated mediawiki/core Project: mediawiki/extensions/ContentTranslation (duration: 00m 17s)
05:13 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Oct 4 05:13:32 UTC 2015 (duration 13m 31s)
02:31 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-04 02:31:56+00:00
02:27 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 08m 06s)

2015-10-03

15:33 jynus: stopping temporarily labsdb1004 mariadb to complete clone process
09:38 _joe_: rolling restarting all parsoids in eqiad
09:14 _joe_: restarting parsoid on wtp1021
05:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Oct 3 05:28:54 UTC 2015 (duration 28m 53s)
03:12 ori: done; graphite-web back up; url shortening will now work.
03:11 ori: shutting down graphite-web for brief sqlite database schema update
02:36 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-03 02:36:33+00:00
02:31 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 08m 22s)

2015-10-02

22:46 mutante: holmium: apt-get clean for a little disk space - /var/log/designate/designate-mdns.log is more than half the size of / - needs logrotate
21:33 Krinkle: mwscript cleanupRemovedModules.php --wiki zhwiktionary
21:21 Krinkle: mwscript cleanupRemovedModules.php --wiki testwikidatawiki
21:15 Krinkle: mwscript cleanupRemovedModules.php --wiki dewiki
21:15 Krinkle: mwscript cleanupRemovedModules.php --wiki nlwiki
21:15 Krinkle: mwscript cleanupRemovedModules.php --wiki test2wiki
21:06 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 17s)
18:39 ottomata: kafka preferred-replica-election
17:53 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/includes/resourceloader: I680f3fda66c5: Configure ResourceLoader-specific ObjectCache instance (duration: 00m 17s)
17:53 ottomata: rolling restart of all hadoop-yarn-nodemanagers to pick up python3 + spark fix
17:12 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I8318fe892: Configure MultiWriteBagOStuff for ResourceLoader (duration: 00m 17s)
16:29 ori: deployed grafana 8e92884bae (with backport of upstream fb9f9548829f2d4cecf35cda933700e5c2fa1bd6)
15:13 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/resources/src/mediawiki/mediawiki.js: I8029208: Don't clobber existing styles when adding more in IE9 (duration: 00m 17s)
14:37 jynus: providing extra grants to wiki db users for heartbeat monitoring
14:32 Coren: Change of NFS mount options in labs pushed - puppet may report failures to refresh the mounts (once) on instances; expected and harmless.
11:03 moritzm: installed rpcbind security updates on all Ubuntu servers which runs it (jessie was already updated, since the DSA was released earlier)
10:53 jynus: restarting HHVM on mw1130
08:45 hashar: restarting Nodepool to take in account changes made to the logging configuration https://gerrit.wikimedia.org/r/#/c/240986/
06:07 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Oct 2 06:07:05 UTC 2015 (duration 7m 4s)
02:42 logmsgbot: tstarling@tin Synchronized php-1.27.0-wmf.1/extensions/ParsoidBatchAPI: stats (duration: 00m 17s)
02:41 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-02 02:41:33+00:00
02:36 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 08m 04s)
00:44 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/includes/resourceloader: I21bb3f08e7f and follow-ups (duration: 00m 18s)
00:43 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/vendor: Add wikimedia/relpath 1.0.3 (duration: 00m 21s)

2015-10-01

22:04 RoanKattouw: Running FlowFixLinks.php on all wikis
20:33 subbu: deployed parsoid version 62971510b
19:33 twentyafterfour: re-deployed robots.txt patch and restarted apache on iridium (to expand the phabricator robots.txt)
18:22 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/includes/resourceloader: Ic1d802ee2: ResourceLoader: cache minified user and site modules (duration: 00m 17s)
18:04 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.27.0-wmf.1
16:40 ottomata: brought analytics1049 back into hadoop after missing a disk since sept. 25th
16:34 andrewbogott: test log
16:26 moritzm: installed PHP security updates on all precise/trusty systems (the respective DSA for jessie is already deployed, it was released three weeks ago)
16:26 valhallasw`cloud: testing, testing
15:56 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/240049 (duration: 00m 17s)
15:56 cmjohnson1: db1051 replacing failed disk slot 6
15:51 logmsgbot: mattflaschen@tin Synchronized php-1.27.0-wmf.1/extensions/ContentTranslation/modules/dashboard/styles/ext.cx.dashboard.less: Fix: Clicking on down arrow in language selector should trigger ULS (duration: 00m 17s)
15:48 logmsgbot: mattflaschen@tin Synchronized wmf-config/InitialiseSettings.php: Enable CX suggestions in ar, eo, hi, nl, vi and dawiki (duration: 00m 17s)
15:47 cmjohnson1: analytics1049 replacing failed disk /dev/sdi at slot 7
15:46 logmsgbot: mattflaschen@tin Synchronized php-1.27.0-wmf.1/tests/phpunit/includes/objectcache/BagOStuffTest.php: Memcached key decode fix (duration: 00m 18s)
15:45 logmsgbot: mattflaschen@tin Synchronized php-1.27.0-wmf.1/includes/objectcache/MemcachedBagOStuff.php: Memcached key decode fix (duration: 00m 17s)
15:44 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf24/tests/phpunit/includes/objectcache/BagOStuffTest.php: Memcached key decode fix (duration: 00m 17s)
15:43 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf24/includes/objectcache/MemcachedBagOStuff.php: Memcached key decode fix (duration: 00m 18s)
15:33 cmjohnson1: replacing failed disk ms-be1012 /dev/sdf slot 5
15:09 matt_flaschen: Did final run of convertAllLqtPages.php on sewikimedia immediately before freezing LQT
15:08 logmsgbot: mattflaschen@tin Synchronized wmf-config/InitialiseSettings.php: Freeze LQT on se.wikimedia (duration: 00m 18s)
14:58 moritzm: installed PHP security updates on all precise/trusty systems (the respective DSA for jessie is already deployed, it was released three weeks ago)
14:19 andrewbogott: log test
14:17 andrewbogott: testing the log
14:07 _joe_: installing the new hhvm package to all the canaries
13:41 hashar: Am I logging?
13:32 _joe_: uploaded hhvm_3.6.5+dfsg1-1+wm7
13:03 bblack: repooling cp1046 (eqiad mobile) with caches wiped clean just before
12:24 akosiaris: disable SessionRemoteIPcheck on mendelevium's OTRS installation for checking
12:18 logmsgbot: aude@tin Finished scap: Put Wikidata extension back on wmf/1.27.0-wmf.1 (duration: 30m 46s)
12:09 moritzm: installed PHP security updates on all precise/trusty systems (the respective DSA for jessie is already deployed, it was released three weeks ago)
11:47 logmsgbot: aude@tin Started scap: Put Wikidata extension back on wmf/1.27.0-wmf.1
10:35 logmsgbot: hoo@tin Synchronized php-1.27.0-wmf.1/extensions/: Use 1.27.0-wmf.1 for Wikidata again after fixing T114290 (duration: 01m 07s)
10:20 akosiaris: uploaded php5_5.3.10-1ubuntu3.20+wmf1 on apt.wikimedia.org precise-wikimedia
09:51 godog: reboot praseodymium to test cassandra systemd unit
09:47 godog: stop puppet on restbase* in preparation for https://gerrit.wikimedia.org/r/242548
08:59 akosiaris: disabling puppet on maps-test200* in expectance of https://gerrit.wikimedia.org/r/242548
06:56 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Oct 1 06:56:51 UTC 2015 (duration 56m 50s)
03:21 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-01 03:21:39+00:00
03:14 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 10m 38s)
02:47 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf24) at 2015-10-01 02:47:31+00:00
02:40 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 10m 54s)
01:30 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/includes/resourceloader/ResourceLoader.php: 1cfe27030e: Change load.php to minify per-module instead of per-request (duration: 00m 17s)
01:29 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/resourceloader/ResourceLoader.php: 1cfe27030e: Change load.php to minify per-module instead of per-request (duration: 00m 17s)
00:44 mutante: applying puppet fix on fermium (illegal byte sequence in utf-8) as in T114289#1691038 but for other languages

2015-09-30

23:00 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.1
22:55 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.1/: deploying several fixes to the branch (duration: 01m 52s)
22:44 ottomata: perf testing eventlogging in production by hammering https://bits.wikimedia.org/beacon/event.gif
22:13 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/includes/User.php: 0ed7cc8526: Made User::loadFromId() skip cache with READ_LATEST (duration: 00m 17s)
22:12 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/User.php: 0ed7cc8526: Made User::loadFromId() skip cache with READ_LATEST (duration: 00m 17s)
21:36 cwdent: updated payments from bc4bcc44d2337d7a69c5a39f11ff45efdf0c8e11 to 24d5be6886d34b3600031290c7f55ee84f3dcee2
21:27 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/resourceloader/ResourceLoaderFileModule.php: a1e1619461: Fix LESS file dependency tracking in ResourceLoader (duration: 00m 17s)
20:43 cscott: updated Parsoid to version 39c60c67
20:40 mutante: fixing puppet run on fermium, needs manual fix because puppet cant replace existing illegal character in some templates
19:18 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.1/resources/Resources.php: T114288 (duration: 00m 17s)
18:51 jynus: stopping replication on s2, s3 and s7 for dbstore1001
18:50 ottomata: restarted eventlogging with blacklist=^Analytics$
18:40 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf24
18:14 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.1
17:39 ottomata: restarting eventlogging with 12 client side processors
16:28 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/resourceloader: 89595ba49a: Cherry-pick I173a9820b and I7c7546ec (duration: 00m 18s)
16:20 logmsgbot: krenair@tin Finished scap: swat (duration: 53m 33s)
15:27 logmsgbot: krenair@tin Started scap: swat
13:53 moritzm: rebooted stat1001/stat1002/stat1003 for kernel updates (already happened between 13:00 UTC and 13:10 UTC, but forgot to log earlier)
12:34 moritzm: added debdeploy 0.0.8 to carbon
05:43 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Sep 30 05:43:19 UTC 2015 (duration 43m 18s)
03:10 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-09-30 03:10:39+00:00
03:03 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 10m 45s)
02:36 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf24) at 2015-09-30 02:36:37+00:00
02:32 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 07m 21s)
01:02 logmsgbot: ori@tin Synchronized php-1.26wmf24/vendor: 940124a7db: Updated mediawiki/core Project: mediawiki/vendor ff5e254f7eddf811f6f66b4a4063b1a8cc70f265 (duration: 00m 21s)

2015-09-29

23:59 ottomata: restarting eventlogging so that processors use etcd to pick up shared token with which to consistently hash IPs
23:46 logmsgbot: catrope@tin Synchronized php-1.26wmf24/extensions/Echo/: SWAT (duration: 00m 17s)
23:37 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Enable Flow opt-in on mediawikiwiki (duration: 00m 16s)
23:33 logmsgbot: catrope@tin Synchronized php-1.26wmf24/extensions/GuidedTour/GuidedTourHooks.php: T114144 Fix back button logging (duration: 00m 18s)
23:32 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/Echo/: SWAT (duration: 00m 17s)
23:30 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/GuidedTour/GuidedTourHooks.php: T114144 Fix back button logging (duration: 00m 17s)
22:50 ejegg: updated payments-wiki from 3b0915a51a0fd567bdf22f3d4e17548a83e735d8 to bc4bcc44d2337d7a69c5a39f11ff45efdf0c8e11
22:47 ejegg: updated SmashPig from bf302444eae8236734fd43883b06c7b2512b1532 to 513ec01123e6dbb97b00888a3610a7c5ec24a63b
22:38 twentyafterfour: finished deployment of 1.27.0-wmf.1 to group0
22:36 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.1
22:34 logmsgbot: twentyafterfour@tin Finished scap: php-1.27.0-wmf.1/extensions/ (duration: 04m 29s)
22:30 twentyafterfour: 22:27:17 sync-dir failed: <ValueError> /srv/mediawiki-staging/php-1.27.0-wmf.1/vendor/oyejorge/less.php/lib/Less/Version.php has content before opening <?php tag
22:30 chasemp: es-tool unban-node elastic1031
22:29 logmsgbot: twentyafterfour@tin Started scap: php-1.27.0-wmf.1/extensions/
22:23 chasemp: unbanning elastic1006 for shard population
22:05 ori: 22:03:51 Synchronized php-1.26wmf24/extensions/CentralNotice: 30bdfcb386: Updated mediawiki/core Project: mediawiki/extensions/CentralNotice 6bd658e155a02edb4cc506bc3494a3f4699d3e94 (duration: 00m 17s)
19:34 logmsgbot: twentyafterfour@tin Finished scap: sync php-1.27.0-wmf.1 for validation on testwiki (duration: 31m 49s)
19:30 bd808: Updated iegreview.wikimedia.org to c3ac5e6 (Update to Twig 1.20.0) and applied latest schema changes
19:02 logmsgbot: twentyafterfour@tin Started scap: sync php-1.27.0-wmf.1 for validation on testwiki
18:11 cmjohnson1: shutting down elastic1031 to relocate rack/row
18:09 cmjohnson1: shutting down elastic1006 to relocate row/rack
17:46 mutante: size of conntrack table on iron might be increased due to test scans
17:32 ejegg: updated appeal template setting on payments-wiki
16:30 cmjohnson1: swapping failed disk db1050
16:14 subbu: reverted configuration hotfix from yesterday's Parsoid deploy (re-enabled use of Parsoid batching API)
15:58 bblack: ending VCL tests on cp1053
15:52 bblack: ending VCL tests on cp1065, starting on cp1053 instead
15:39 bblack: cp1065: live-testing some VCL patches, puppet disabled, etc...
15:19 chasemp: ban elastic1031 for T112559
15:05 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: VisualEditor: Set TransitionDefault true for the English Wikipedia gerrit:242040 (duration: 00m 17s)
14:08 paravoid: repooling eqiad; 24h codfw test window is over
10:49 godog: bounce pybal on lvs2003 / lvs2006
09:13 paravoid: powercycling pybal-test2003
08:51 jynus: starting cloning of labsdb1005 (Tools DB), minimal disruption is expected
07:03 twentyafterfour: restarted apache2 and phd or iridium to get phabricator back into the correct state
06:58 twentyafterfour: checked out correct phabricator release tag on iridium. Something, somewhere, had reverted everything to an old deployment.
05:01 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Sep 29 05:01:30 UTC 2015 (duration 1m 29s)
02:34 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf24) at 2015-09-29 02:34:55+00:00
02:30 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 06m 57s)
02:25 bblack: ipv6 flap experiment: raise ipv6/route/max_size from 4096 to 131072 manually on cp*, actually
02:24 bblack: ipv6 flap experiment: raise ipv6/route/max_size from 4096 to 131072 manually on cp20*
01:55 robh: upload-lb.codfw.wikimedia.org_ipv6 page flap
01:43 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Plumbing for wmgVisualEditorTransitionDefault (duration: 00m 17s)
01:43 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Add wmgVisualEditorTransitionDefault (false everywhere) (duration: 00m 17s)
01:13 robh: didnt powercycle analytics1035 yet, it recovered on its own.
01:13 robh: powercycling analytics1035, seems oom, cannot login via ssh or serial console
00:56 logmsgbot: tstarling@tin Synchronized php-1.26wmf24/extensions/ParsoidBatchAPI: Fix fatal error I77fd7e8 (duration: 00m 17s)

2015-09-28

23:47 logmsgbot: ebernhardson@tin Synchronized wmf-config/CommonSettings.php: Update ttmserver configuration to match elasticsearch security profile (duration: 00m 17s)
23:43 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I56db35b: Removed ignore_user_abort( true ) line (duration: 00m 18s)
23:13 logmsgbot: catrope@tin Synchronized php-1.26wmf24/extensions/Echo: SWAT (duration: 00m 18s)
23:13 logmsgbot: catrope@tin Synchronized php-1.26wmf24/extensions/Flow: SWAT (duration: 00m 19s)
23:12 logmsgbot: catrope@tin Synchronized php-1.26wmf24/extensions/CentralNotice: SWAT (duration: 00m 19s)
20:51 bearND: MobileApps deployed sha1 9df72ec
20:42 subbu: deployed parsoid version b9e5244e + hotfix on tin to turn off batching api use since canary restart of wtp1002 showed some batching api errors
20:26 yuvipanda: depooled cp2017 from pybal config too
20:22 yuvipanda: depooled cp2017 since it's down
19:02 chasemp: powercycle iridium via console as it's unresponsive
18:01 awight: update crm from 190f689ff7aec7fecefdf5af501293685c55e041 to 7003cc38797848631d0c4d5f6ff68ab1d6118ad8
15:50 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/241579/ (duration: 00m 17s)
15:28 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/241598/ (duration: 00m 17s)
15:11 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/241649/ (duration: 00m 17s)
14:00 paravoid: running failover eqiad->codfw test for all frontend traffic
06:11 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I179be4bd3: Rely on timeouts specified in php.ini rather than calling ini_set() (duration: 00m 17s)
05:52 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Sep 28 05:52:34 UTC 2015 (duration 52m 33s)
02:27 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf24) at 2015-09-28 02:27:16+00:00
02:20 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 06m 54s)

2015-09-27

23:17 logmsgbot: ori@tin Synchronized php-1.26wmf24/./resources/src/mediawiki.toolbar/toolbar.less: I94ced06178: mediawiki.toolbar: temporary workaround for T113868 (duration: 00m 17s)
16:51 Krenair: ran sync-common on snapshot1001 to bring it up to date
16:43 logmsgbot: krenair@tin Purged l10n cache for 1.26wmf21
16:42 logmsgbot: krenair@tin Purged l10n cache for 1.26wmf20
16:42 logmsgbot: krenair@tin Purged l10n cache for 1.26wmf19
15:50 logmsgbot: krenair@tin Synchronized wmf-config/abusefilter.php: https://gerrit.wikimedia.org/r/#/c/241354/ - fix AbuseFilter block durations (duration: 00m 18s)
10:05 jynus: leaving innodb compression tests on es2005 running (could affect lag on that host)
02:23 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 07m 05s)

2015-09-26

19:04 hashar: restarting Jenkins. Just in case :-D
02:23 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 06m 48s)

2015-09-25

23:54 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/GlobalFunctions.php: 37c6972f94: Made wfIsBadImage() use APC (duration: 00m 17s)
22:16 mutante: sodium - shutting down
21:53 ori: Armed new servicedeploy_rsa on mira
21:51 ori: Armed new servicedeploy_rsa on tin
20:45 urandom: starting a Cassandra repair on xeon (nodetool repair -pr)
19:30 mutante: restarted apache on gallium
19:07 bblack: configuring and enabling lvs-cross-row ports on asw-b-eqiad for lvs1007,8,10,11
18:04 jynus: performing schema change on m5-master "nova"
17:26 jynus: restarting and upgrading db1051 mysql (depooled)
17:20 godog: reboot ms-be1005, xfs
16:22 cwdent: updated payments from dc78ff5157b59a8f475dc86194a1059c2d6b2fad to 3b0915a51a0fd567bdf22f3d4e17548a83e735d8
16:17 akosiaris: duplicating database otrs to otrsupgradetest for testing the upgrade procedure
16:16 cmjohnson1: cp1046 stopped icinga checks for hardware troubleshooting
14:43 moritzm: installed rpcbind and apport security updates on various servers
14:36 logmsgbot: krenair@tin Synchronized php-1.26wmf24/includes/specials/SpecialMovepage.php: https://gerrit.wikimedia.org/r/#/c/241045/ (duration: 00m 17s)
14:22 chasemp: 'sudo -u hdfs hdfs haadmin -transitionToActive analytics1001-eqiad-wmnet' per otto on analytics1001
14:03 urandom: starting a Cassandra repair on restbase1006 (nodetool repair -pr -dc eqiad)
13:58 hashar: nodepool back in operations
13:44 moritzm: restarted saltmaster on palladium
13:12 mobrovac: restbase deploying e42bf0fc
13:12 moritzm: added debdeploy 0.0.7 to carbon
13:10 godog: enable puppet on restbase in production
12:54 andrewbogott: restarted rabbitmq-server on labcontrol1001
12:52 godog: stop puppet on restbase in production -- config deployment
12:46 hashar: stopping nodepool to clear out left over mysql connections
12:21 hashar: Nodepool is dead, or at least not adding new slaves anymore
12:16 hashar: Seems Zuul/Jenkins is in trouble somehow :-/
11:42 andrewbogott: upgrading linux-image-generic on labnet1002 to get us away from the recently-crashed 3.13.0-59
04:58 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I252970886: Set "async" for SQL parser cache everywhere else (duration: 00m 18s)
02:57 Krinkle: sync-common failed on mw1010.eqiad.wmnet
02:53 logmsgbot: krinkle@tin Synchronized php-1.26wmf24/extensions/DonationInterface: 381faf5 (duration: 00m 21s)
02:33 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 11m 45s)
01:44 Krinkle: mwscript deleteEqualMessages.php --wiki roa_tarawiki
01:42 Krinkle: mwscript deleteEqualMessages.php --wiki itwikiquote
01:42 Krinkle: mwscript deleteEqualMessages.php --wiki alswiki
01:28 logmsgbot: krinkle@tin Synchronized php-1.26wmf24/extensions/WikimediaEvents: I5608f8ffd1c - Fix trailing comma (duration: 00m 17s)
00:10 logmsgbot: krinkle@tin Synchronized php-1.26wmf24/extensions/EventLogging/modules/ext.eventLogging.core.js: Increase maxUrlSize to 2000 (duration: 00m 17s)

2015-09-24

23:02 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/240880/ (duration: 00m 17s)
21:18 ejegg: updated SmashPig from d1baa32267eaad7d69b47c657f4853eb306fad6b to bf302444eae8236734fd43883b06c7b2512b1532
21:15 ejegg: updated payments-wiki from 8428499feb8760d63faf681d53995697a2ba0fa7 to dc78ff5157b59a8f475dc86194a1059c2d6b2fad
19:57 logmsgbot: ori@tin Synchronized php-1.26wmf24/extensions/ContentTranslation: d079d5dd71: Updated mediawiki/core Project: mediawiki/extensions/ContentTranslation 8559ee614975f25b71a732ca0fb1bb6d489c9d33 (duration: 00m 18s)
19:35 bblack: depooled cp1046 from confd, committed pybal depool for LVS as well
19:34 chasemp: changing labs route on cr1 and cr2 from 10.68.16.0/22 to 10.68.16.0/21 which matches references, fw setting and manifests/network.pp
18:54 logmsgbot: catrope@tin Synchronized php-1.26wmf24/extensions/Flow/: Debugging for FlowFixLinks.php (duration: 00m 20s)
18:21 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.26wmf24
18:20 legoktm: moved oauthadmin group from User:Yuvipanda@metawiki to User:YuviPanda@metawiki
18:19 godog: restart restbase on restbase1005
18:18 godog: restart restbase on restbase1004
18:17 ejegg: reenabling CRM jenkins jobs
18:16 godog: restart restbase on restbase1003
18:08 ejegg: updated civicrm from 9fa38d06a75363a8009bce7ced190e39c75b68bc to 190f689ff7aec7fecefdf5af501293685c55e041
18:06 paravoid: depooling cp1046, stability issues
18:05 ejegg|afk: disabled CRM jenkins jobs
18:00 logmsgbot: demon@tin Synchronized multiversion/MWRealm.php: (no message) (duration: 00m 17s)
17:59 ori: Merged Apache config change Ia095457fb. It will refresh the Apache service as it rolls out, causing elevated 503s for the next 20 minutes.
17:53 godog: rolling restart restbase in eqiad
17:35 chasemp: powercycling cp1046 at mgmt as I can't ssh in and it seems like it should be up
17:26 godog: bounce restbase on restbase1002, apply new datacenter config
17:10 _joe_: cleaning up /tmp on mw1152
17:09 cmjohnson1: powering down for the last time es1001 - es1010
16:17 logmsgbot: thcipriani@tin Synchronized php-1.26wmf23/extensions/Wikidata: SWAT: Do not filter affected pages by namespace gerrit:240727 (duration: 00m 26s)
16:01 robh: nothing on puppet swat window, easiest swat ever.
15:46 logmsgbot: thcipriani@tin Synchronized php-1.26wmf24/extensions/Wikidata: SWAT: Do not filter affected pages by namespace gerrit:240711 (duration: 00m 26s)
15:23 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable suggestions in ca, en, es, fr, it, ja, tr, ru, zh gerrit:240638 (duration: 00m 17s)
14:37 paravoid: repooling codfw
12:54 bblack: restarting varnish daemons on second half of maps, parsoid, misc clusters (package upgrade, shm_reclen change)
12:50 bblack: restarting varnishd instances on text, mobile, upload clusters for package upgrade (slow salt, no parallelism, ~5m spacing - FE cache loss, BE cache stays, should take ~9h)
12:05 moritzm: installed rpcbind security updates on eeden, baham, radon, maerlant, rhenium
11:56 bblack: restarting varnish daemons on half of maps, parsoid, misc clusters (package upgrade, shm_reclen change)
11:36 bblack: reinstall lvs300[12] to jessie - T96375
11:21 akosiaris: killed tail -f varnishncsa.log on cp1065 and ran apt-get clean to reclaim some disk space
11:14 bblack: stopping pybal on lvs300[12]; lvs300[34] taking over
11:07 bblack: upgrading varnishes to 3.0.6plus-wm8 (non-restarting, just pkg update on-disk)
09:40 jynus: performing latest (software) steps to decom es1001-es1010 (puppet disabling, etc.)
08:39 jynus: restarted HHVM @ mw1056, 1104, 1122
05:33 yuvipanda: deleted logstash indexes for 08/27 and 28 too
05:31 yuvipanda: deleted indexes for 08/14, 15, 25, 26 on logstash
03:59 yuvipanda: restarting elasticsearch in logstash1001-3
03:53 yuvipanda: restarted es on logstash1004-6
03:02 yuvipanda: jstack dumped logstash output onto /home/yuvipanda/stack on logstash1001 since strace seems useles
02:51 yuvipanda: restarted logstash on logstash1002
02:41 yuvipanda: gmond at 100% again, killing it and stopping puppet again
02:40 yuvipanda: re-enabling and running puppet on hafnium to see what it's bringing up
02:38 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 30s)
02:23 yuvipanda: kill gmond on hafnium and disable puppet to prevent it from taking it back up. Was taking 100% CPU
02:16 Krinkle: Kibana/Logstash outage. Zero events received after 2015-09-23T23:59:59.999Z.
02:14 Krinkle: Partial EventLogging outage (client-side events via hafnium abruptly stopped 2015-09-23 11:36 UTC - 15 hours ago)
01:53 mutante: started logstash on logstash1002 again
01:35 mutante: bast1001: unmounting /srv/home_pmtpa (backup on bacula)
01:34 mutante: removing subversion packages from bast1001
01:15 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes: Ifa0d4cfe8e3: Backport I1ff61153d and I8e4c3d5a5 (duration: 00m 23s)
00:19 jynus: restarted replication on db1051
00:17 ori: restarting tcpircbot on neon
00:16 mutante: started logstash on logstash1002
00:08 bblack: varnish package on carbon for jessie updated to 3.0.6plus-wm8

2015-09-23

23:17 logmsgbot: krenair@tin Synchronized php-1.26wmf24/includes/specials/SpecialSearch.php: https://gerrit.wikimedia.org/r/#/c/240596/ (duration: 00m 18s)
23:09 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/240575/ (duration: 00m 17s)
22:53 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/resourceloader/ResourceLoaderFileModule.php: 14f46330d9: Backport fix from PS12 of I1ff6115 (duration: 00m 17s)
22:39 paravoid: cr1-codfw RE switchover(s)
22:35 logmsgbot: yurik@tin Synchronized php-1.26wmf24/extensions/ZeroBanner: Deploying ZeroBanner interstitial handling (duration: 00m 18s)
21:13 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/resourceloader/ResourceLoaderFileModule.php: 58bfb6f85b: Backport fix from PS9 of I1ff6115 (duration: 00m 17s)
20:46 subbu: deployed parsoid 6619409e
19:58 urandom: starting Cassandra on restbase2006
19:56 urandom: enabling puppet, and forcing a run on restbase2006
19:52 urandom: starting Cassandra on restbase2005
19:50 urandom: enabling puppet, and forcing a run on restbase2005
19:48 urandom: starting Cassandra on restbase2004
19:45 urandom: enabling puppet, and forcing a run on restbase2004
18:55 twentyafterfour: snapshot1001.eqiad.wmnet returned [12]: rsync: write failed on "/srv/mediawiki/wikiversions.cdb": No space left on device
18:54 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf24
18:01 mobrovac: restbase deploying f65313ed
17:49 urandom: starting Cassandra on restbase2003
17:45 logmsgbot: demon@tin Synchronized php-1.26wmf23/extensions/Wikidata: (no message) (duration: 00m 25s)
17:35 urandom: enabling, and forcing puppet run on restbase2003
17:27 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 13s)
17:20 logmsgbot: ori@tin scap failed: OSError [Errno 2] No such file or directory: '/var/lock/scap' (duration: 44m 24s)
17:11 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/page/Article.php: (no message) (duration: 00m 13s)
17:11 logmsgbot: ori@tin Synchronized php-1.26wmf23/includes/page/Article.php: (no message) (duration: 00m 17s)
16:35 logmsgbot: ori@tin Started scap: (no message)
16:35 logmsgbot: ori@tin Synchronized php-1.26wmf23/includes/page: I952068d2d: Reduced the DOS potential of 404 page floods (duration: 00m 12s)
16:22 godog: start cassandra on restbase2002
16:05 godog: deploy restbase daacf4d on restbase2*
15:34 logmsgbot: demon@tin Synchronized php-1.26wmf24/extensions/Wikidata: (no message) (duration: 00m 20s)
15:34 logmsgbot: demon@tin Synchronized php-1.26wmf23/extensions/Wikidata: (no message) (duration: 00m 21s)
15:34 chasemp: unbanning elastic1005 for T112559
15:32 logmsgbot: demon@tin scap aborted: (no message) (duration: 02m 07s)
15:30 logmsgbot: demon@tin Started scap: (no message)
15:15 logmsgbot: demon@tin Synchronized php-1.26wmf22/extensions/Wikidata: (no message) (duration: 00m 20s)
14:13 logmsgbot: demon@tin Synchronized wmf-config/abusefilter.php: (no message) (duration: 00m 12s)
14:08 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 12s)
14:05 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 12s)
14:04 logmsgbot: demon@tin Synchronized langlist-labs: (no message) (duration: 00m 11s)
14:04 logmsgbot: demon@tin Synchronized docroot/noc/conf/: (no message) (duration: 00m 13s)
14:01 logmsgbot: demon@tin Synchronized multiversion/MWRealm.php: (no message) (duration: 00m 11s)
13:56 godog: force puppet run on restbase2001 to deploy new cassandra config
13:28 godog: reboot ms-be1012, xfs hosed
13:22 paravoid: upgrading cr1-codfw with newer junos
07:43 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: semver the special:version hook (duration: 00m 12s)
03:05 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 10m 12s)
02:38 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf23) at 2015-09-23 02:38:44+00:00
02:35 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 21s)
00:56 logmsgbot: legoktm@tin Synchronized php-1.26wmf24/extensions/Echo/Hooks.php: Remove duplicate 'MediaWiki' prefix from echo.unseen stats (duration: 00m 12s)
00:03 RoanKattouw: Running FlowFixLinks.php on testwiki

2015-09-22

23:40 mutante: renaming search mailing lists to discovery mailing lists
23:35 logmsgbot: krenair@tin Synchronized php-1.26wmf24/extensions/Echo: https://gerrit.wikimedia.org/r/#/c/240283/ and https://gerrit.wikimedia.org/r/#/c/240281/ (duration: 00m 13s)
23:18 logmsgbot: krenair@tin Synchronized wmf-config/abusefilter.php: https://gerrit.wikimedia.org/r/#/c/240278/ (duration: 00m 12s)
23:16 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/240259/ (duration: 00m 12s)
23:15 logmsgbot: krenair@tin Synchronized w/static/images/sul/wikimania.png: https://gerrit.wikimedia.org/r/#/c/239308/ (duration: 00m 11s)
23:14 logmsgbot: krenair@tin Synchronized w/static/images/sul/commons.png: https://gerrit.wikimedia.org/r/#/c/239308/ (duration: 00m 12s)
22:44 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Set $wgFlowMigrateReferenceWiki to false in production (duration: 00m 12s)
22:38 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Enable Flow opt-in on testwiki for testing (duration: 00m 12s)
21:56 cwdent: updated payments from 7b08867d9c5e87f5babb4b5b9cf1f5bec5e243b3 to 8428499feb8760d63faf681d53995697a2ba0fa7
21:49 chasemp: unban elastic1030 from T112559
21:33 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.26wmf24
21:13 logmsgbot: twentyafterfour@tin Finished scap: Test 1.26wmf24 (duration: 50m 34s)
20:44 mutante: cancel backup job of bast1001 on helium because running low on disk
20:22 logmsgbot: twentyafterfour@tin Started scap: Test 1.26wmf24
16:50 robh: all mw servers returned to puppet enabled, puppet swat window over
16:40 awight: updated paymentswiki 153418195a45cab820bc2aacf9a4f7dbc9dde768 to 7b08867d9c5e87f5babb4b5b9cf1f5bec5e243b3
16:23 robh: re-enabled puppet on mw hosts, as both redirection changes are good
16:14 robh: re-enabling puppet on mw hosts, as the new patchset 239278 deployed and tested fine on a single host, deploying to rest
16:04 robh: disabling puppet across mw hosts for new configuration deployment
15:46 godog: running puppet on restbase2001
15:41 godog: stop puppet on restbase2* pending codfw expansion
15:31 cwdent: updated payments from 153418195a45cab820bc2aacf9a4f7dbc9dde768 to 7b08867d9c5e87f5babb4b5b9cf1f5bec5e243b3
14:42 cmjohnson1: shutting down elastic1005 and elastic1030 to move around within the data center
14:19 bblack: starting slow restart of varnish + varnish-frontend daemon processes on global text, upload, and mobile clusters for shm_reclen (all randomly blended, no parallelism, ~5 minute spacing, will take ~9 hours - FEs will lose cache data, BEs will not)
14:14 chasemp: depool elastic nodes for T112559
11:18 logmsgbot: aude@tin Synchronized php-1.26wmf23/extensions/Wikidata: Fix autocomment and change handling bugs (duration: 00m 21s)
10:42 logmsgbot: aude@tin Synchronized arbitraryaccess.dblist: Enable arbitrary access for Wikibooks (duration: 00m 12s)
10:42 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable data access for Wikibooks - try again for snapshot hosts (duration: 00m 12s)
10:35 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable data access for Wikibooks (duration: 01m 12s)
10:17 moritzm: enabled ferm on mw1152 (videoscaler)
10:03 godog: finished stressdisk on restbase200[123] no errors reported
10:03 moritzm: enabled ferm on mw1259 (videoscaler)
04:35 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Sep 22 04:35:46 UTC 2015 (duration 35m 45s)
02:22 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf23) at 2015-09-22 02:22:56+00:00
02:19 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 00s)
01:53 mutante: sodium - deleted salt key, revoked puppet cert, rm from icinga ..
00:32 ori: Disabled Puppet for 24h on hafnium and stopped ganglia-monitor. gmond was saturating CPU.

2015-09-21

23:29 logmsgbot: krinkle@tin Synchronized php-1.26wmf23/extensions/NavigationTiming/modules/ext.navigationTiming.js: T112593 (duration: 00m 14s)
23:21 logmsgbot: krenair@tin Synchronized php-1.26wmf22/extensions/Wikidata: https://gerrit.wikimedia.org/r/#/c/239828/ (duration: 00m 21s)
23:07 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/238357/ (duration: 00m 13s)
22:26 mutante: restarting gerrit for ssh config change
21:15 awight: legacy PayPal listener updated from 1c9ac2e66d11bbf768ea873d6e1a2522ca9841c1 to 55aeef63f6508381e3a8b7fcabddf9a3c3b73b8e
20:43 cwdent: updated worldpay config on payments
20:41 mdholloway: MobileApps deployed sha1 013044e
20:39 chasemp: banning elastic1005 for T112559
20:29 subbu: deployed parsoid version 9984d221
19:41 urandom: temporarily stopping codfw restbase cassandra nodes to test quorum auth
19:15 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: Ieccb23f: Enable async secondary writes for mysql-multiwrite cache (on testwiki) (duration: 00m 13s)
18:36 ejegg: re-enabled paypal audit parser
18:16 cmjohnson1: disabling puppet on mw1031
17:58 chasemp: banning 1030 from eqiad elastic cluster for T112559#1660068
17:57 ejegg: disabled paypal audit parser
16:08 ejegg: updated payments-wiki from 4d9d165c40070e036176dba8987243f6dbc7415e to 153418195a45cab820bc2aacf9a4f7dbc9dde768
15:22 logmsgbot: thcipriani@tin Synchronized php-1.26wmf23/extensions/ContentTranslation/modules/entrypoint/ext.cx.interlanguagelink.js: SWAT: Revert "Do not call cxserver to display gray interwiki link" gerrit:239819 (duration: 00m 11s)
15:10 logmsgbot: thcipriani@tin Synchronized robots.txt: SWAT: Remove redundant entries from robots.txt gerrit:239403 (duration: 00m 12s)
14:33 ottomata: restart eventlogging with mysql consumer replace=True (AKA INSERT IGNORE)
14:09 godog: rolling restart restbase in production after cassandra credentials change
12:53 godog: rolling restart cassandra after enabling dc encryption, no nodes in codfw yet
12:01 moritzm: repooled mw1160 (for T104969)
11:54 moritzm: depooled mw1160 (for T104969)
11:51 moritzm: repooled mw1158, mw1159 (for T104969)
11:39 moritzm: depooled mw1158, mw1159 (for T104969)
11:37 moritzm: depooled and repooled mw1156, mw1157 (for T104969)
11:26 moritzm: repooled mw1154, mw1155 (for T104969)
11:21 moritzm: depooled mw1154, mw1155 (for T104969)
10:39 moritzm: repooled mw1026-mw1029 and mw1110-mw1113 (for T104968)
10:24 moritzm: depooled mw1026-mw1029 and mw1110-mw1113 (for T104968)
10:17 moritzm: repooled mw1100-mw1109 (for T104968)
10:17 godog: create restbase user on cassandra cluster
10:06 moritzm: depooled mw1100-mw1109 (for T104968)
09:56 moritzm: repooled mw1140 and mw1142-mw1148 (for T104968)
09:41 moritzm: depooled mw1140 and mw1142-mw1148 (for T104968)
09:36 moritzm: repooled mw1130-mw1139 (for T104968)
09:22 moritzm: depooled mw1130-mw1139 (for T104968)
09:14 moritzm: repooled mw1120-mw1129 (for T104968)
09:02 moritzm: depooled mw1120-mw1129 (for T104968)
08:48 moritzm: repooled mw1189 and mw1200-mw1208 (for T104968)
08:33 moritzm: depooled mw1189 and mw1200-mw1208 (for T104968)
08:29 godog: switch to 'restbase' cassandra user on restbase test cluster
08:29 moritzm: repooled mw1190-mw1195 and mw1197-mw1199 (for T104968)
08:21 _joe_: restarted the logstash agent on logstash1003, OOM'd
08:18 moritzm: depooled mw1190-mw1195 and mw1197-mw1199 (for T104968)
08:07 _joe_: installing the new HHVM package on the api canaries
08:04 moritzm: repooled mw1221-mw1229 (for T104968)
07:53 moritzm: depooled mw1221-mw1229 (for T104968)
07:49 moritzm: repooled mw1230-mw1235 (for T104968)
07:43 _joe_: installing the new hhvm package on the canary appservers
07:08 moritzm: depooled mw1230-mw1235 (for T104968)
04:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Sep 21 04:31:03 UTC 2015 (duration 31m 2s)
02:23 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf23) at 2015-09-21 02:23:12+00:00
02:20 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 25s)
02:06 MaxSem: Maps: created indexes on admin. <3 Postgres :(
01:56 bblack: downtimed eqiad ipv6 text/upload alerts as well, as with mobile above ( 1 301 TLS Redirect - 505 bytes in 1.008 second response time
01:46 bblack: downtimed the "LVS HTTP IPv6 on mobile-lb.eqiad.wikimedia.org_ipv6" alert for now ( https://phabricator.wikimedia.org/T113154 )

2015-09-20

22:34 yuvipanda: reloda pybal on lvs1012
17:01 bblack: repooling cp1046 varnish-be + varnish-be-rand in confctl, fresh storage, purge queue caught up - T113184
16:44 bblack: depooling cp1046 varnish-be + varnish-be-rand in confctl, wiping storage, re-pooling - T113184
07:24 paravoid: temporarily disabling puppet on fermium and applying antispam countermeasures
04:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Sep 20 04:29:16 UTC 2015 (duration 29m 15s)
02:22 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf23) at 2015-09-20 02:22:53+00:00
02:19 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 12s)

2015-09-19

23:12 urandom: begining Cassandra repair on restbase1005 (nodetool repair -pr)
23:08 urandom: begining Cassandra repair on restbase1004 (nodetool repair -pr)
19:56 jynus: restarting once more giblit, last chance
19:04 paravoid: salt rm /etc/systemd/system/txstatsd.service from all cp*, leftover because of ::txstatsd::decommission (removed with 4a1d4e) missing it
19:00 ejegg|away: updated SmashPig from d5895428d1d8ebc5a6e172e8cdec6dbec0b10d85 to d1baa32267eaad7d69b47c657f4853eb306fad6b
18:45 _joe_: restarted gitblit. I will now substitute myself with a clever perl one-liner.
18:38 paravoid: pooling back cp1046 to pybal eqiad/mobile, has stayed stable
18:34 paravoid: reactivating ΒGP with GTT @ eqiad
08:42 _joe_: cp1046 dead on console again, powercycling to inspect it
05:49 logmsgbot: aaron@tin Synchronized php-1.26wmf23/extensions/TitleBlacklist: 80d3a21a51f9c54ed2d94 (duration: 00m 12s)
05:22 paravoid: pybal-depooling cp1046 from eqiad/mobile until further investigation
05:21 paravoid: powercycling cp1046, dead on console
05:01 awight: deploy SmashPig config to limit weekend spam
04:40 awight: update crm from 15ea14f61338ca9f34e9ccb9f56eae14a161380a to 9fa38d06a75363a8009bce7ced190e39c75b68bc
04:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Sep 19 04:28:59 UTC 2015 (duration 28m 57s)
02:23 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf23) at 2015-09-19 02:23:29+00:00
02:20 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 05s)

2015-09-18

23:22 awight: update fundraising-tools from 3e0e3ae799a507b378d0ece3e71631b10b361329 to e1b60fa2c258fd4ff55905b03a4d8886132278c1
20:52 ebernhardson: restart es on elastic1025 to disable dynamic scripting
20:34 gwicke: dropped by_ns indexes on restbase title_revisions tables
19:54 gwicke: finished deploy of restbase daacf4daa
19:45 gwicke: re-enabled puppet on restbase100*
19:35 gwicke: canary deploy of restbase daacf4daa on restbase1001; moving forward so that we can re-enable puppet over the weekend.
18:38 cwdent: updated payments from 1bdd287b083032ff418434ad6bb6920735af918a to 4d9d165c40070e036176dba8987243f6dbc7415e
17:54 logmsgbot: ebernhardson@tin Synchronized wmf-config/CommonSettings.php: Replace insecure es usage with usage of a plugin (duration: 00m 12s)
16:41 mutante: mailman now on 2.1.18 and jessie
16:14 dcausse: elastic in eqiad plugin updates: restarting elastic1021
16:07 paravoid: deactivating ΒGP with GTT @ eqiad
15:20 godog: create restbase user on cassandra test cluster
14:55 dcausse: elastic in eqiad plugin updates: restarting elastic1020
14:22 bblack: committing lvs1007-1012 port/vlan changes for asw-d-eqiad (but leaving all 6 LVS ports in "disabled" state - T112781 )
14:14 bblack: committing lvs1007-12 port/vlan changes for asw-b-eqiad, round 3...
14:11 mutante: sodium - stopped exim - rsyncing lists to fermium
14:10 dcausse: elastic in eqiad plugin updates: restarting elastic1019
14:07 mutante: stopped mailman on sodium
14:01 bblack: rollback on asw-b-eqiad changes above
13:56 bblack: committing eqiad lvs1007-1012 port/vlan changes for asw-b-eqiad
13:20 bblack: committing eqiad lvs1007-12 port/vlan changes for asw-c-eqiad
13:16 bblack: commiting eqiad lvs1007-12 port/vlan changes for asw2-a5-eqiad
13:12 dcausse: elastic in eqiad plugin updates: restarting elastic1018
12:21 godog: restart logstash on logstash1001, OOM in logs
11:55 dcausse: elastic in eqiad plugin updates: restarting elastic1017
11:06 dcausse: elastic in eqiad plugin updates: restarting elastic1016
10:28 moritzm: restarted salt-master on palladium
09:46 moritzm: installed openldap security updates on plutonium
09:37 moritzm: installed openldap security updates on pollux
09:33 dcausse: elastic in eqiad plugin updates: restarting elastic1015
08:22 dcausse: elastic in eqiad plugin updates: restarting elastic1014
07:21 dcausse: elastic in eqiad plugin updates: restarting elastic1013
06:15 dcausse: elastic in eqiad plugin updates: restarting elastic1012
04:37 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Sep 18 04:37:42 UTC 2015 (duration 37m 41s)
02:31 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf23) at 2015-09-18 02:31:49+00:00
02:28 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 08s)
02:21 logmsgbot: krenair@tin Synchronized wmf-config/abusefilter.php: https://gerrit.wikimedia.org/r/#/c/218353/ (duration: 00m 12s)
02:21 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/218353/ (duration: 00m 11s)
02:13 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/237149/ (duration: 00m 12s)
02:07 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234544/ (duration: 00m 12s)
01:58 logmsgbot: ori@tin Synchronized php-1.26wmf22/includes/resourceloader/ResourceLoaderModule.php: I952068d2d: ResourceLoaderModule: cache file content hash (duration: 00m 12s)
01:58 logmsgbot: ori@tin Synchronized php-1.26wmf23/includes/resourceloader/ResourceLoaderModule.php: I952068d2d: ResourceLoaderModule: cache file content hash (duration: 00m 11s)
01:57 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://phabricator.wikimedia.org/T106264 (duration: 00m 12s)
01:36 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/237331/ (duration: 00m 12s)
00:14 ori: restarted logstash on logstash1001

2015-09-17

23:39 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/238978/ (duration: 00m 12s)
23:05 logmsgbot: mattflaschen@tin Synchronized wmf-config/CommonSettings-labs.php: Beta-only change (duration: 00m 12s)
23:04 logmsgbot: mattflaschen@tin Synchronized wmf-config/CommonSettings-labs.php: Beta-only change (duration: 00m 12s)
22:53 gwicke: puppet on restbase cluster disabled since about 21:30 UTC for gradual deploy; ran into minor issue in staging, which is now being addressed, after which deploy will continue
21:22 logmsgbot: ori@tin Synchronized php-1.26wmf22/includes/resourceloader/ResourceLoaderModule.php: I952068d2d: Use MD4 to compute file hash rather than SHA1 (duration: 00m 13s)
21:22 logmsgbot: ori@tin Synchronized php-1.26wmf23/includes/resourceloader/ResourceLoaderModule.php: I952068d2d: Use MD4 to compute file hash rather than SHA1 (duration: 00m 12s)
20:44 logmsgbot: krenair@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 12s)
20:22 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/239206/ (duration: 00m 12s)
19:46 logmsgbot: krenair@tin Synchronized multiversion/MWMultiVersion.php: (no message) (duration: 00m 12s)
19:41 logmsgbot: krenair@tin Synchronized multiversion/MWMultiVersion.php: https://gerrit.wikimedia.org/r/#/c/239181/ (duration: 00m 14s)
19:12 mutante: powercycling unresponse mw1005
18:14 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.26wmf23
17:38 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/includes/registration/ExtensionRegistry.php: registration: Fix merging of array_plus (duration: 00m 13s)
17:35 logmsgbot: legoktm@tin Synchronized php-1.26wmf23/includes/registration/ExtensionRegistry.php: registration: Fix merging of array_plus (duration: 00m 11s)
16:43 chasemp: restart elasticsearch on 1005
16:16 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/235900/ (duration: 00m 12s)
15:15 dcausse: elastic in eqiad plugin updates: restarting elastic1004 (take 2)
15:06 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Enable Suggestions in ptwiki gerrit:238097 (duration: 00m 13s)
14:22 mutante: analytics1029 - Failed to start Hadoop datanode
14:20 mutante: starting hadoop datanode on analytics1029
14:14 _joe_: reimaging tmh1001 to mw1259
14:11 jynus: stopping replication and applying schema change to db1051
14:05 dcausse: elastic in eqiad plugin updates: can't restart elastic1004 (2 timeouts when disabling replication, too much load?), waiting for more shards to rebalance...
13:58 dcausse: elastic in eqiad plugin updates: restarting elastic1004
13:50 moritzm: repooled mw1236-mw1239 (T104968)
13:34 moritzm: depooled mw1236-mw1239 (T104968)
13:26 moritzm: repooled mw1090-mw1099 (T104968)
13:16 moritzm: depooled mw1090-mw1099 (T104968)
13:13 moritzm: repooled mw1080-mw1089 (T104968)
13:05 moritzm: depooled mw1080-mw1089 (T104968)
13:01 moritzm: repooled mw1070-mw1079 (T104968)
12:49 moritzm: depooled mw1070-mw1079 (T104968)
12:35 moritzm: repooled mw1060 and mw1062-mw1069 (T104968)
12:24 moritzm: depooled mw1060 and mw1062-mw1069 (T104968) (not repooled)
12:24 moritzm: repooled mw1060 and mw1062-mw1069 (T104968)
12:16 moritzm: repooled mw1050-mw1059
12:04 moritzm: depooled mw1050-mw1059
11:39 moritzm: repooled mw1040 and mw1042-mw1049 (T104968)
11:36 dcausse: elastic in eqiad plugin updates: restarting elastic1003
11:26 moritzm: typoed earlier entry: "mw1032-mw1039" instead of "mw1032-mw1239"
11:26 moritzm: depooled mw1040 and mw1042-mw1049 (T104968)
11:18 moritzm: repooled mw1030 and mw1032-mw1239 (T104968)
11:03 moritzm: depooled mw1030 and mw1032-mw1239 (T104968)
10:35 moritzm: repooled mw1250-mw1258 (T104968)
10:27 moritzm: depooled mw1250-mw1258 (T104968)
10:25 _joe_: killing temporarily subra
10:24 moritzm: repooled mw1240-mw1249 (T104968)
10:19 _joe_: experimenting with poolcounter issues on subra
10:18 logmsgbot: oblivian@tin Synchronized wmf-config/PoolCounterSettings-codfw.php: Use codfw poolcounters in codfw (duration: 00m 12s)
10:12 moritzm: depooled mw1240-mw1249 (T104968)
10:12 dcausse: elastic in eqiad plugin updates: restarting elastic1002
10:05 logmsgbot: hoo@tin Synchronized wmf-config/: Set 'repoConceptBaseUri' for all Wikibase clients (duration: 00m 13s)
10:00 dcausse: elastic in eqiad plugin updates: unfreezing indices
09:48 dcausse: elastic in eqiad plugin updates: no more groovy in warmers, waiting for few more shards to move in elastic1001 and will unfreeze indices to test warmers
09:39 dcausse: elastic in eqiad plugin updates: deleting warmers manually for old unused indices (eswikisource_content_1415240352, ruwiki_content_1415302164, thwiki_content_1415318677). We will have to remove these indices.
09:39 paravoid: repooling ulsfo US-West traffic back to ulsfo for the first time since May :)
09:01 dcausse: elastic in eqiad plugin updates: updating warmers on all wikis
08:58 paravoid: penalizing ulsfo-eqiad direct MPLS links to higher OSPF weights
08:57 paravoid: adjusting OSPF weights to be latency-based across the US network
08:53 _joe_: removed iptables rules for dropping traffic to helium on mw1017
08:52 dcausse: elastic in eqiad plugin updates: index warmer queries are outdated with inline groovy script, updating warmers on warwiki first to test
08:05 paravoid: eqiad-codfw -> eqiad-eqord-codfw migration
07:49 moritzm: repooled mw1180-mw1188 (T104968)
07:42 dcausse: elastic in eqiad plugin updates: restarting elastic1001
07:42 moritzm: depooled mw1180-mw1188 (T104968)
07:37 moritzm: repooled mw1170-mw1179 (T104968)
07:36 dcausse: elastic in eqiad plugin updates: freezing indices
07:27 moritzm: depooled mw1170-mw1179 (T104968)
07:14 _joe_: uploading new HHVM package
07:07 moritzm: repooled mw1161-1168 (T104968)
06:57 moritzm: depooled mw1161-1168 (T104968)
06:45 moritzm: repooled mw1209-mw1220 with ferm enabled
06:33 moritzm: depooling mw1209-mw1220 (in two steps)
05:47 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Sep 17 05:47:47 UTC 2015 (duration 47m 46s)
03:06 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf23) at 2015-09-17 03:06:33+00:00
03:03 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 30s)
02:45 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-17 02:45:48+00:00
02:39 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 11m 11s)
00:35 cwdent: updated payments from 155cdeb737c01baf62551292764fd2f5a93a9a63 to 1bdd287b083032ff418434ad6bb6920735af918a

2015-09-16

23:27 bblack: updating eqiad switch configs for lvs1007-1012 vlan/trunk settings
23:19 logmsgbot: krenair@tin Synchronized php-1.26wmf23/extensions/MobileFrontend/resources/mobile.overlays/Overlay.less: https://gerrit.wikimedia.org/r/#/c/238865/ (duration: 00m 11s)
23:13 gwicke: started `nodetool rebuild -- eqiad` on restbase-test200{1,2
23:03 cwdent: updated payments from 9fc8ab40b7f70c7b588c2b9e7b5c94b1f893faa1 to 155cdeb737c01baf62551292764fd2f5a93a9a63
22:26 ejegg: updated SmashPig from fdb053efa617162ac9f695e493c390987a069140 to d5895428d1d8ebc5a6e172e8cdec6dbec0b10d85
22:08 urandom: disabling puppet in RESTBase eqiad staging cluster to test new code and config
22:08 ottomata: powercycling analytcis1029, it is down?
20:47 cscott: updated OCG to version 4032a596ce6eb442b02cc6ee9b79263b1eb23275
19:42 ejegg: updated crm from abc34b87ee9d1dbb1176f1929a3d748e1ee5ac7b to 15ea14f61338ca9f34e9ccb9f56eae14a161380a
19:38 ori: Deployed statsv 0bfd9f06f / change I050a12d3b
18:47 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf23
18:38 logmsgbot: twentyafterfour@tin Synchronized php-1.26wmf23: syncing wmf23 ahead of deployment to group1 (duration: 01m 35s)
17:34 paravoid: asw-d-eqiad: toggling RE mastership again
17:26 godog: stop puppet on restbase* to apply https://gerrit.wikimedia.org/r/#/c/238738/ / merge / reenable puppet
16:54 _joe_: turned on the hhvm tmh, stopping the zend ones for testing
16:44 logmsgbot: oblivian@tin Synchronized wmf-config/CommonSettings.php: use ffmpeg whereever possible (duration: 00m 12s)
16:16 bblack: upgrading pybal on lvs400[12]
16:12 bblack: upgrading pybal on lvs400[34], lvs300[34]
16:08 bblack: upgrading pybal on lvs200[123]
16:05 bblack: upgrading pybal on lvs200[456]
15:44 _joe_: uploading pybal 1.10 to reprepro, installing to the test cluster
15:24 moritzm: uploaded debdeploy 0.0.6 to apt.wikimedia.org
15:10 hashar: Started using Nodepool spawned instances. Moved integration-jjb-config-diff Jenkins job to Nodepool with https://gerrit.wikimedia.org/r/#/c/238752/ . See also: https://phabricator.wikimedia.org/T112750
15:05 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: Add m.wikidata.org to wgCrossSiteAJAXdomains (duration: 00m 12s)
14:51 _joe_: experimenting on testwiki for poolcounter failure scenarios
14:45 moritzm: enabled ferm on mw1010 (jobrunner) in eqiad
14:27 paravoid: asw-d-eqiad: toggling RE mastership
14:18 paravoid: disabling/ignoring asw-d-eqiad @ librenms
14:09 jynus: upgrading and restarting db1051
13:57 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1051 for maintenance (duration: 00m 12s)
13:40 urandom: initiating Cassandra repair on restbase1007 (nodetool repair -pr)
13:40 logmsgbot: catrope@tin Synchronized php-1.26wmf23: (no message) (duration: 01m 37s)
13:35 moritzm: repooled mw1149-mw1151 (with ferm enabled)
13:24 moritzm: depooled mw1149-mw1151 (for enabling ferm)
13:19 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Reverting depool of es1055 (duration: 00m 12s)
13:15 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1055 for maintenance (duration: 00m 12s)
13:03 paravoid: disabling asw-d-eqiad xe-8/0/23, xe-8/0/24, xe-8/0/25, xe-8/0/26, xe-8/0/27, xe-8/0/28; servers reboot-looping -> asw-d's SNMP unhappy -> librenms unhappy -> faidon's mailbox unhappy
12:48 moritzm: repooled mw1115-mw1117, mw1119 (with ferm enabled)
12:42 moritzm: depooling mw1115-mw1117, mw1119 (mw1118 was already depooled) to enable ferm
11:32 moritzm: repooled mw1019-mw1025 with ferm enabled
11:24 jynus: making db1069 a sibling of db1055 (s1)
11:13 godog: create restbase user on cassandra test cluster
11:07 moritzm: depooled mw1019-mw1025 (to enable ferm)
10:52 logmsgbot: catrope@tin Synchronized php-1.26wmf23: (no message) (duration: 02m 04s)
10:49 logmsgbot: catrope@tin Synchronized php-1.26wmf22: (no message) (duration: 02m 12s)
10:48 jynus: reenabling semisync on db1072 and db1073
10:47 logmsgbot: catrope@tin scap aborted: (no message) (duration: 00m 21s)
10:47 logmsgbot: catrope@tin Started scap: (no message)
10:24 logmsgbot: catrope@tin Synchronized php-1.26wmf23/includes/changes/EnhancedChangesList.php: T112738 (duration: 00m 12s)
10:09 logmsgbot: aude@tin Synchronized arbitraryaccess.dblist: (no message) (duration: 00m 11s)
09:37 awight: ruthlessly disabled PayPal IPN listener
08:12 moritzm: repooled mw1153 with ferm enabled
07:57 jynus: truncated some tables from ContentTranslation extension on x1
07:57 moritzm: depooled mw1153 (it's an image scaler, of course) to enable ferm
07:56 moritzm: depooled mw1153 (videoscaler) to enable ferm
06:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Sep 16 06:31:58 UTC 2015 (duration 31m 57s)
03:28 logmsgbot: ori@tin Synchronized php-1.26wmf22/vendor/monolog/monolog/src/Monolog/Logger.php: Iccfda47689: monolog: Don't waste milliseconds counting microseconds (duration: 00m 12s)
03:27 logmsgbot: ori@tin Synchronized php-1.26wmf23/vendor/monolog/monolog/src/Monolog/Logger.php: Iccfda47689: monolog: Dont waste milliseconds counting microseconds ; sync-file php-1.26wmf22/vendor/monolog/monolog/src/Monolog/Logger.php Iccfda47689: monolog: Dont waste milliseconds counting microseconds (duration: 00m 12s)
03:12 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf23) at 2015-09-16 03:12:08+00:00
03:05 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 10m 30s)
02:38 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-16 02:38:48+00:00
02:35 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 07m 02s)
01:03 logmsgbot: krinkle@tin Synchronized php-1.26wmf23/resources/src/mediawiki/mediawiki.js: hotfix Ia2fcd13f4 (duration: 00m 12s)
00:29 logmsgbot: krinkle@tin Synchronized php-1.26wmf22/resources/src/mediawiki/mediawiki.js: hotfix Ia2fcd13f4 (duration: 00m 11s)
00:15 logmsgbot: legoktm@tin Synchronized php-1.26wmf23/extensions/CentralAuth/includes/: Use set() for tokens with unique keys (duration: 00m 12s)
00:14 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/extensions/CentralAuth/includes/: Use set() for tokens with unique keys (duration: 00m 12s)
00:11 bblack: reinstalling lvs400[12] to jessie (traffic on 400[34], already jessie)
00:08 logmsgbot: krenair@tin Synchronized php-1.26wmf23/extensions/VisualEditor/modules/ve-mw/ui/styles/dialogs: https://gerrit.wikimedia.org/r/#/c/238646/ (duration: 00m 12s)

2015-09-15

23:51 logmsgbot: krenair@tin Synchronized php-1.26wmf22/extensions/WikimediaEvents/modules/ext.wikimediaEvents.geoFeatures.js: https://gerrit.wikimedia.org/r/#/c/238617/ (duration: 00m 12s)
23:48 logmsgbot: krenair@tin Synchronized php-1.26wmf23/extensions/WikimediaEvents/modules/ext.wikimediaEvents.geoFeatures.js: https://gerrit.wikimedia.org/r/#/c/238618/ (duration: 00m 12s)
23:42 logmsgbot: krenair@tin Synchronized wmf-config/mobile.php: https://gerrit.wikimedia.org/r/#/c/238543/ (duration: 00m 14s)
23:42 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/238543/ (duration: 00m 12s)
23:40 logmsgbot: krenair@tin Synchronized php-1.26wmf22/includes/resourceloader/ResourceLoader.php: https://gerrit.wikimedia.org/r/#/c/238544/ (duration: 00m 11s)
23:38 logmsgbot: krenair@tin Synchronized php-1.26wmf23/includes/resourceloader/ResourceLoader.php: https://gerrit.wikimedia.org/r/#/c/238545/ (duration: 00m 11s)
23:24 yurik: deployed kartotherian & tilerator
23:22 logmsgbot: krenair@tin Synchronized php-1.26wmf22/extensions/EventLogging/modules/ext.eventLogging.core.js: https://gerrit.wikimedia.org/r/#/c/238512/ (duration: 00m 12s)
23:16 logmsgbot: krenair@tin Synchronized php-1.26wmf23/extensions/EventLogging/modules/ext.eventLogging.core.js: https://gerrit.wikimedia.org/r/#/c/238513/ (duration: 00m 12s)
21:15 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf22/extensions/WikimediaEvents/: touch files edited in I0cb6fe37e and re-sync to cluster (duration: 00m 13s)
21:13 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.26wmf23
21:10 logmsgbot: twentyafterfour@tin Finished scap: sync 1.26wmf23 to testwiki, once more because mw1010 overloaded (duration: 03m 52s)
21:07 logmsgbot: twentyafterfour@tin Started scap: sync 1.26wmf23 to testwiki, once more because mw1010 overloaded
21:05 logmsgbot: twentyafterfour@tin Finished scap: sync 1.26wmf23 to testwiki, again (duration: 47m 49s)
20:47 mutante: mw1010 - extremely slow,finally got on and attempted to restart hhvm. load going down
20:17 logmsgbot: twentyafterfour@tin Started scap: sync 1.26wmf23 to testwiki, again
20:17 logmsgbot: twentyafterfour@tin scap aborted: sync 1.26wmf23 to testwiki (duration: 82m 58s)
20:05 ottomata: restarted mysql (and oozie) on analytics1027 to start mysql binlogging
18:54 logmsgbot: twentyafterfour@tin Started scap: sync 1.26wmf23 to testwiki
16:55 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1003, es1004, es1007 and es1010 for decommision (duration: 00m 12s)
16:40 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Revert depool db1055 for maintenance (duration: 00m 11s)
16:39 ottomata: reinstalling analytics1015
16:32 RoanKattouw: Putting wmf22 versions of Echo and MobileFrontend on mw1017 for testing
16:30 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf22/extensions/WikimediaEvents/WikimediaEvents.php: touch file that is serving old version in prod (duration: 00m 12s)
16:29 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf22/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSuggest.js: Touch file that is serving old version in prod (duration: 00m 12s)
16:27 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1055 for maintenance (duration: 00m 11s)
16:11 bblack: traffic DNS depooled out of codfw for now T112639
15:38 logmsgbot: thcipriani@tin Synchronized wmf-config: SWAT: CX: Enable suggestion for testwiki (part 2) gerrit:237327 (duration: 00m 13s)
15:37 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Enable suggestion for testwiki (part 1) gerrit:237327 (duration: 00m 12s)
15:31 logmsgbot: thcipriani@tin Synchronized php-1.26wmf22/extensions/UploadWizard/resources/jquery/jquery.mwCoolCats.js: SWAT: Do not fail horribly when invalid categories are passed gerrit:238421 (duration: 00m 12s)
15:14 logmsgbot: thcipriani@tin Synchronized wmf-config/PoolCounterSettings-eqiad.php: SWAT: poolcounter: enable connect_timeout for testwiki gerrit:238109 (duration: 00m 19s)
15:09 logmsgbot: thcipriani@tin Synchronized wmf-config/PoolCounterSettings-codfw.php: SWAT: poolcounter: add connect_timeout in codfw gerrit:238108 (duration: 00m 12s)
15:06 logmsgbot: thcipriani@tin Synchronized wmf-config/Wikibase.php: SWAT: Exclude Flow topic boards and Draft NS from Special:UnconnectedPages gerrit:229197 (duration: 00m 11s)
14:51 godog: bounce cassandra on test cluster to deploy https://gerrit.wikimedia.org/r/236391
14:22 cmjohnson1: swapped disk on db1043
13:12 moritzm: repool mw1114 (with ferm enabled)
13:11 bblack: failing over LVS service in ulsfo to secondariess (400[12] pybal stopped, traffic on jessie-based 400[34])
12:53 moritzm: depooled mw1114 (for enabling ferm)
11:42 moritzm: repool mw1018 (with ferm enabled)
11:23 moritzm: depooled mw1018 (for enabling ferm)
08:53 _joe_: created a 100 G partition on a LV on copper, for /tmp
08:24 godog: bounce ms-be2006, xfs
08:22 moritzm: bumped default size of iptables connection tracking table to 256k
06:10 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Sep 15 06:10:52 UTC 2015 (duration 10m 51s)
02:46 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-15 02:46:50+00:00
02:40 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 10m 53s)
02:18 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/extensions/MobileFrontend: Revert Echo to 1.26wmf21 state (duration: 00m 11s)
02:18 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/extensions/Echo: Revert Echo to 1.26wmf21 state (duration: 00m 12s)
01:30 logmsgbot: krinkle@tin Synchronized php-1.26wmf22/resources/src: T112287 (duration: 00m 11s)
00:49 bblack: reinstalling lvs300[34] to jessie
00:43 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-labs.php: noop sync of labs config change (duration: 00m 11s)
00:03 logmsgbot: tstarling@tin Synchronized php-1.26wmf22/extensions/ParsoidBatchAPI: for I56d28e9a for RT testing, not live yet (duration: 00m 13s)

2015-09-14

23:27 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf22/extensions/WikimediaEvents/: Change bucket selection methods in CompletionSuggestions AB test (duration: 00m 12s)
23:23 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf22/extensions/UploadWizard/: Swat out badtoken fix to UploadWizard in 1.26wmf22 (duration: 00m 12s)
22:37 yurik: deployed tilerator
21:15 logmsgbot: ori@tin Synchronized php-1.26wmf22/extensions/TitleBlacklist: Ie44fcb500: Avoid checking blacklists in isBlacklisted() for existing titles (duration: 00m 12s)
21:15 mutante: labnodepool1001 - re-enable puppet and nodepool
20:59 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/extensions/Echo/: Hack around OOUI's icon pack being too large by creating our own (duration: 00m 12s)
20:53 cscott: updated OCG to version 5811056e28f2bc6408b6da96095352ab381bb11f
20:21 andrewbogott: graceful’d apache2 on labcontrol1001
20:15 subbu: deployed parsoid sha 3d5f4359
19:25 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/extensions/Echo/: Only load nojs Special:Notifications styles on the special page (duration: 00m 12s)
18:05 urandom: rebuilding restbase-test2001.codfw (nodetool rebuild -- eqiad)
16:12 logmsgbot: catrope@tin Synchronized php-1.26wmf22/extensions/Echo/: For real this time (duration: 00m 11s)
16:06 ottomata: stopping hdfs journalnode on analytics1011 to copy journal edits to new journalnodes on analytics1035 and analytics1052
15:46 godog: switch to openjdk-8 and bounce cassandra on restbase-test200*
15:39 bblack: reinstalling lvs4003, lvs4004 (jessie upgrade: T96375) (typo earlier)
15:39 bblack: reinstalling lvs4003, lvs4003 (jessie upgrade: T96375)
15:34 logmsgbot: catrope@tin Synchronized php-1.26wmf22/extensions/Echo/: SWAT (duration: 00m 13s)
15:05 logmsgbot: krenair@tin Synchronized .gitignore: https://gerrit.wikimedia.org/r/#/c/237529/ (duration: 00m 13s)
15:05 logmsgbot: krenair@tin Synchronized docroot/noc: https://gerrit.wikimedia.org/r/#/c/237529/ (duration: 00m 12s)
15:04 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/237529/ (duration: 00m 11s)
15:02 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234980/ (duration: 00m 12s)
13:38 godog: stop puppet on restbase-test2001 and turn up cassandra
12:56 bblack: rebooting lvs2006 to test eth hw params stuff...
12:55 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/238125/ (duration: 00m 13s)
12:50 urandom: starting Cassandra repair on restbase1003 (nodetool repair -pr)
12:32 godog: enable dc encryption on cassandra test cluster and rolling restart
11:33 mobrovac: citoid deploying d569951
10:35 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1002, es1005, es1008 (duration: 00m 12s)
10:04 jynus: db1029 (x1-master) temporarily saturated by connections- flow was unresponsive for 10 minutes; migration partially aborted
09:08 jynus: applying schema change to flowdb
08:52 godog: rename cassandra test cluster and restart
08:44 godog: silence mendelevium for today, status unclear T111532
08:30 jynus: endinf profiling and executing pt-query-digest on db1043 [ETA:4h]
07:52 godog: reboot ms-be1010 to pick up disk ordering change
04:48 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Sep 14 04:47:58 UTC 2015 (duration 47m 57s)
02:29 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-14 02:29:48+00:00
02:26 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 06m 59s)
01:31 Krinkle: mwscript deleteEqualMessages.php --wiki sqwiki

2015-09-13

06:02 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Sep 13 06:02:52 UTC 2015 (duration 2m 51s)
02:40 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-13 02:40:43+00:00
02:34 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 10m 13s)

2015-09-12

20:15 ori: Rolling back Echo to 1.26wmf21 branch on mw1017 (testwiki) to measure increase in render-blocking CSS size
19:21 urandom: performing Cassandra repair on restbase1002 (nodetool repair -pr)
14:50 jynus: phab.wmfusercontent.org has been temporarily switched to phab.wikivoyage.org due to cert issues
04:52 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Sep 12 04:52:01 UTC 2015 (duration 52m 0s)
02:35 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-12 02:35:36+00:00
02:32 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 06m 54s)

2015-09-11

21:21 hashar: shutdown nodepool on labnodepool1001.eqiad.wmnet until monday
18:01 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/extensions/Echo/: Echo regression fixes #2 (duration: 00m 12s)
16:43 logmsgbot: krinkle@tin Synchronized php-1.26wmf22/resources/src/mediawiki/mediawiki.js: T112232 (duration: 00m 12s)
16:37 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/extensions/Echo/: Echo regression backports (duration: 00m 12s)
16:35 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/resources/src/mediawiki/mediawiki.js: resourceloader: Document internal mw.loader#jobs property (again) (duration: 00m 13s)
16:33 legoktm: ssh: connect to host mw1156.eqiad.wmnet port 22: Connection timed out
16:32 paravoid: powercycling mw1156, multiple kernel backtraces in console output
16:32 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/resources/src/mediawiki/mediawiki.js: resourceloader: Document internal mw.loader#jobs property (duration: 01m 07s)
16:15 cmjohnson1: mw1031 rebooting for f/w update
16:07 bblack: enabled LRO+GRO on lvs200[123], starting pybal there again ([456] testing looks good so far)
15:45 bblack: enabled LRO+GRO on lvs200[456] (backups). Stopping pybal on lvs200[123] to test...
15:11 cmjohnson1: swapping pem2 cr2-eqiad
10:03 jynus: starting nodepool in labnodepool1001
09:21 jynus: starting profiling of phabricator db (db1043). Very low overhead.
06:03 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Sep 11 06:03:00 UTC 2015 (duration 2m 59s)
02:41 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-11 02:41:24+00:00
02:34 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 11m 18s)
01:16 logmsgbot: ori@tin Synchronized php-1.26wmf22/extensions/TitleBlacklist: 9bf13dbe0b, 3203b045f7 (duration: 00m 12s)

2015-09-10

23:52 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/237064/ (duration: 00m 11s)
23:47 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/237056/ (duration: 00m 11s)
23:13 logmsgbot: krenair@tin Synchronized wmf-config/wikitech.php: https://gerrit.wikimedia.org/r/221825 (duration: 00m 13s)
23:04 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/224771 (duration: 00m 12s)
21:13 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/extensions/Echo/modules: Align popup footer buttons to take 50% width each (duration: 00m 15s)
20:50 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: depool es1001; increase weight of es1015 and es1019 (duration: 00m 19s)
20:47 ottomata: restarting eventlogging with 12 client side processors on eventlog1001
20:31 ottomata: turning off varnishncsa eventlogging eventlistener instances on frontend caches, it is now superseded by varnishkafka
20:28 mutante: killed/restarted ganglia aggregator process for mobile-cache, upload cache, misc esams ...
20:22 jynus: last SCAP failed on 266/466 hosts
20:21 mutante: killed/restarted ganglia aggregator process for text-caches esams on hooft
20:17 yurik: deployed kartotherian
20:08 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1001; increase weight of es1015 and es1019 (duration: 00m 11s)
19:11 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.26wmf22
19:09 logmsgbot: twentyafterfour@tin Synchronized php-1.26wmf22/extensions/CentralNotice: deploy https://gerrit.wikimedia.org/r/#/c/237458/ (duration: 00m 12s)
18:57 twentyafterfour: restarted phd on iridium
18:51 logmsgbot: twentyafterfour@tin Synchronized php-1.26wmf22/extensions/Wikidata: Deploy wikidata patch: https://gerrit.wikimedia.org/r/#/c/237449/ (duration: 00m 19s)
18:23 logmsgbot: twentyafterfour@tin Synchronized php-1.26wmf22: deploy https://gerrit.wikimedia.org/r/#/c/237440/ (duration: 01m 42s)
18:09 cmjohnson1: reseating pem2 cr2-eqiad
16:52 akosiaris: puppetswat done
16:50 mobrovac: restbase rolling restart of rb100x
16:49 mobrovac: restbase enabled puppet on rb100x
16:13 akosiaris: started puppetSWAT
16:10 logmsgbot: marktraceur@tin Finished scap: Make sure codfw got the last few patches sync'd to it (duration: 07m 36s)
16:03 logmsgbot: marktraceur@tin Started scap: Make sure codfw got the last few patches sync'd to it
16:02 logmsgbot: marktraceur@tin Synchronized php-1.26wmf22/: [SWAT] [wmf22] Revert opera redirect loop fix that caused redirect loops in Firefox (duration: 02m 30s)
15:55 mobrovac: restbase disabled puppet on rb100x
15:45 logmsgbot: marktraceur@tin Synchronized php-1.26wmf22/extensions/UploadWizard/resources/transports/mw.FormDataTransport.js: [SWAT] [wmf22] Always set 'offset' with chunked uploads, even for first chunk (offset == 0) (duration: 02m 21s)
15:26 ottomata: started hadoop decomission of analytics1016
15:21 logmsgbot: marktraceur@tin Synchronized wmf-config/: [SWAT] Attempting another sync to mw2187 hoping it's up now (duration: 02m 22s)
15:05 logmsgbot: marktraceur@tin Synchronized wmf-config/: [SWAT] [config] Beta: Enable Content Translation suggestions (duration: 02m 22s)
13:35 moritzm: enabled ferm on mediawiki app servers in codfw
13:30 jynus: performing schema change and maintenance on officewiki and public all wikis with flow enabled
12:51 moritzm: enabled ferm on mediawiki API servers in codfw
12:36 moritzm: enabled ferm on mediawiki video scalers, image scalers and job runners in codfw
09:20 mobrovac: restbase deploying 0182962
06:13 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Sep 10 06:13:14 UTC 2015 (duration 13m 13s)
03:02 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-10 03:02:45+00:00
02:59 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 06m 10s)
02:51 logmsgbot: krenair@tin Synchronized php-1.26wmf22/extensions/WikimediaMaintenance/dumpInterwiki.php: https://gerrit.wikimedia.org/r/237304 (duration: 00m 11s)
02:50 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/WikimediaMaintenance/dumpInterwiki.php: https://gerrit.wikimedia.org/r/237303 (duration: 00m 10s)
02:43 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-10 02:43:20+00:00
02:36 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 10m 45s)
02:24 logmsgbot: krinkle@tin Synchronized php-1.26wmf21/resources/src/mediawiki/mediawiki.js: Ic0b1fb64ee7 backport (duration: 00m 12s)
01:04 logmsgbot: ori@tin Synchronized php-1.26wmf21/extensions/NavigationTiming: I2605c746b: Ensure timings are reported after the page has loaded (duration: 00m 13s)
01:03 logmsgbot: ori@tin Synchronized php-1.26wmf22/extensions/NavigationTiming: I2605c746b: Ensure timings are reported after the page has loaded (duration: 00m 12s)
00:54 mutante: powercycling unresponsive mw1154

2015-09-09

23:34 logmsgbot: krenair@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 12s)
23:31 logmsgbot: krenair@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 12s)
23:29 logmsgbot: krenair@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 12s)
23:23 MaxSem: deployed Kartotherian config updates
23:23 logmsgbot: catrope@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 11s)
23:22 RoanKattouw: Running updateinterwikicache
23:13 logmsgbot: catrope@tin Synchronized php-1.26wmf22/extensions/WikimediaMaintenance: SWAT (duration: 00m 13s)
23:13 logmsgbot: catrope@tin Synchronized php-1.26wmf22/extensions/Flow: SWAT (duration: 00m 32s)
23:12 logmsgbot: catrope@tin Synchronized php-1.26wmf21/extensions/WikimediaMaintenance: SWAT (duration: 00m 14s)
23:12 logmsgbot: catrope@tin Synchronized php-1.26wmf21/extensions/Flow: SWAT (duration: 00m 29s)
20:17 subbu: deployed parsoid version ffd0b444
18:15 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf22
16:47 andrewbogott: systemctl stop nodepool on labnodepool1001
16:06 logmsgbot: aude@tin Synchronized database lists: Remove unused usagetracking.dblist (duration: 00m 12s)
16:01 logmsgbot: krenair@tin Synchronized robots.txt: https://gerrit.wikimedia.org/r/#/c/236200/ (duration: 00m 12s)
15:57 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/236701/ - noop (duration: 00m 12s)
15:56 ejegg: updated payments from from 4c5e30288370db926cbbf7a7528edb9c41c65716 to 9fc8ab40b7f70c7b588c2b9e7b5c94b1f893faa1
15:50 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/237104/ (duration: 00m 12s)
15:46 logmsgbot: krenair@tin Synchronized wmf-config/Wikibase.php: https://gerrit.wikimedia.org/r/#/c/237097/ (duration: 00m 12s)
15:46 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/237097/ (duration: 00m 12s)
15:43 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf21/resources/src/mediawiki/mediawiki.searchSuggest.js: Enable completion suggester AB experiment (duration: 00m 12s)
15:43 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf21/extensions/WikimediaEvents/: Enable suggester AB experiement (duration: 00m 11s)
15:38 logmsgbot: krenair@tin Synchronized php-1.26wmf22/extensions/Wikidata: https://gerrit.wikimedia.org/r/#/c/237091/ (duration: 00m 21s)
15:26 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234425/ (duration: 00m 12s)
15:21 logmsgbot: krenair@tin Synchronized wmf-config/logging.php: https://gerrit.wikimedia.org/r/#/c/236994/ (duration: 00m 12s)
15:15 bd808: Running sync-common manually on mw2187.codfw.wmnet. Host is missing l10n cache files
15:12 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/236025/ (duration: 00m 11s)
15:10 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/236042/ (duration: 00m 13s)
14:03 mutante: beginning mailman migration - expect lists to be down
13:14 moritzm: enabled ferm on test.wikipedia.org (mw1017)
13:05 urandom: issuing Cassandra repair on restbase1001 (nodetool repair -pr)
13:02 moritzm: enabled ferm on various initial mediawiki hosts in codfw: videoscaler (mw2007), appserver (mw200[89]), jobrunner (mw2081), api (mw2050), imagescaler (mw2086)
10:33 logmsgbot: aude@tin Synchronized wmf-config/CommonSettings.php: Remove unused usagetracking tag (duration: 00m 11s)
10:30 logmsgbot: aude@tin Synchronized wmf-config/Wikibase.php: (no message) (duration: 00m 12s)
10:26 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: rv usage tracking (duration: 00m 12s)
10:23 logmsgbot: aude@tin Synchronized usagetracking.dblist: Enable usage tracking on commons and test2wiki (duration: 00m 11s)
10:21 logmsgbot: aude@tin Synchronized wikidataclient.dblist: Sorted dblist (duration: 00m 12s)
09:41 logmsgbot: aude@tin Synchronized usagetracking.dblist: Enable usage tracking on Wikinews (duration: 00m 12s)
08:35 moritzm: installed spice security updates on labvirt*, ganeti* and labnodepool1001
05:11 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Sep 9 05:11:28 UTC 2015 (duration 11m 27s)
02:55 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-09 02:55:24+00:00
02:52 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 05m 34s)
02:31 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-09 02:31:50+00:00
02:28 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 06m 44s)
00:00 logmsgbot: catrope@tin Finished scap: Need to update i18n for a new Echo message (duration: 23m 08s)

2015-09-08

23:36 logmsgbot: catrope@tin Started scap: Need to update i18n for a new Echo message
23:36 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings-labs.php: SWAT (duration: 00m 10s)
23:36 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings-labs.php: SWAT (duration: 00m 13s)
23:34 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: SWAT (duration: 00m 12s)
23:33 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: SWAT (duration: 00m 12s)
23:20 logmsgbot: catrope@tin Synchronized php-1.26wmf22/extensions/WikimediaEvents/: SWAT (duration: 00m 11s)
23:20 logmsgbot: catrope@tin Synchronized php-1.26wmf22/extensions/Echo/: SWAT (duration: 00m 14s)
23:14 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings-labs.php: (no message) (duration: 00m 11s)
22:13 logmsgbot: krinkle@tin Synchronized php-1.26wmf21/extensions/NavigationTiming: re-apply patch 1/2 (jscs) (duration: 00m 12s)
21:36 logmsgbot: krinkle@tin Synchronized php-1.26wmf21/extensions/NavigationTiming: temporarily revert T109756 (duration: 00m 11s)
21:02 csteipp: deployed patches for T108616 T91850 T91205 to wmf21 & 22
20:45 bblack: upgrading nginx to 1.9.4 on cp*
20:38 logmsgbot: ori@tin Synchronized multiversion: wikimedia/cdb 1.2.0 → 1.3.0 (duration: 00m 12s)
20:38 logmsgbot: ori@tin Synchronized php-1.26wmf22/vendor: wikimedia/cdb 1.2.0 → 1.3.0 (duration: 00m 15s)
20:37 logmsgbot: ori@tin Synchronized php-1.26wmf21/vendor: wikimedia/cdb 1.2.0 → 1.3.0 (duration: 00m 14s)
20:07 logmsgbot: aude@tin Finished scap: Update group0 to new Wikidata branch (duration: 24m 27s)
19:42 logmsgbot: aude@tin Started scap: Update group0 to new Wikidata branch
19:14 logmsgbot: twentyafterfour@tin Synchronized php-1.26wmf21/: sync php-1.26wmf21 as well (duration: 02m 31s)
19:10 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.26wmf22
18:55 ejegg: updated payments from 6ac552f280fb839069d117386c4ecbe9e52f90a8 to 4c5e30288370db926cbbf7a7528edb9c41c65716
18:50 logmsgbot: twentyafterfour@tin Finished scap: testwiki to 1.26wmf22 (duration: 29m 29s)
18:20 logmsgbot: twentyafterfour@tin Started scap: testwiki to 1.26wmf22
18:01 ejegg: rolled back payments to 6ac552f280fb839069d117386c4ecbe9e52f90a8
17:59 ejegg: updated payments from 6ac552f280fb839069d117386c4ecbe9e52f90a8 to 4c5e30288370db926cbbf7a7528edb9c41c65716
17:43 moritzm: enabled ferm on remaining hadoop workers (analytics1040-analytics1057)
17:09 logmsgbot: krinkle@tin Synchronized php-1.26wmf21/extensions/NavigationTiming: T109756 (duration: 00m 11s)
16:56 logmsgbot: krinkle@tin Synchronized php-1.26wmf21/extensions/CentralAuth: T108253 sul2 token store (duration: 00m 12s)
16:16 logmsgbot: ori@tin Synchronized php-1.26wmf21/vendor: I5af46eb3: wikimedia/cdb 1.0.1 → 1.2.0 (duration: 00m 14s)
15:43 logmsgbot: ori@tin Synchronized multiversion: wikimedia/cdb 1.0.1 → 1.2.0 (duration: 00m 12s)
15:21 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/236785/ (duration: 00m 12s)
15:17 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234910/ (duration: 00m 12s)
14:41 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Pool es1015 and es1019 (duration: 00m 11s)
14:30 moritzm: enabled ferm on hadoop workers up to analytics1039
12:41 godog: change whisper aggregation for 'sum.wsp' files T111170
10:48 moritzm: restarted salt master on palladium
10:32 logmsgbot: aude@tin Synchronized usagetracking.dblist: Enable usage tracking on Wikibooks (duration: 00m 11s)
09:55 moritzm: uploaded debdeploy 0.0.5 to carbon
04:37 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Sep 8 04:37:06 UTC 2015 (duration 37m 5s)
02:23 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-08 02:23:51+00:00
02:20 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 06m 30s)
00:46 Krinkle: mwscript deleteEqualMessages.php --wiki eswiki

2015-09-07

21:45 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/236682/ (duration: 00m 12s)
21:44 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/WikimediaEvents/WikimediaEvents.php: https://gerrit.wikimedia.org/r/#/c/236196/1 (duration: 00m 12s)
21:42 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/WikiEditor: https://gerrit.wikimedia.org/r/#/c/236197/1 and https://gerrit.wikimedia.org/r/#/c/236679/ (duration: 00m 12s)
18:15 andrewbogott: graceful’d apache, restarted keystone on labcontrol1001
15:41 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/MobileFrontend/includes/MobileFrontend.hooks.php: https://gerrit.wikimedia.org/r/#/c/236558/ (duration: 00m 12s)
15:11 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1004, pool es1018 (duration: 00m 10s)
10:04 godog: powercycle ms-be1003, loadavg skyrocketed
08:13 hashar: Jenkins upgraded to latest LTS ( https://phabricator.wikimedia.org/T111326 )
08:05 hashar: Upgrading Jenkins
04:33 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Sep 7 04:33:11 UTC 2015 (duration 33m 10s)
02:29 Krinkle: mwscript deleteEqualMessages.php --wiki pmswiki
02:23 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-07 02:23:27+00:00
02:20 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 06m 22s)

2015-09-06

04:27 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Sep 6 04:27:57 UTC 2015 (duration 27m 56s)
02:23 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-06 02:23:08+00:00
02:19 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 06m 14s)

2015-09-05

23:37 Krinkle: mwscript deleteEqualMessages.php --wiki fywiktionary
04:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Sep 5 04:31:34 UTC 2015 (duration 31m 33s)
02:30 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-05 02:30:06+00:00
02:27 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 05m 53s)

2015-09-04

23:52 logmsgbot: mattflaschen@tin Synchronized wmf-config/InitialiseSettings-labs.php: Beta-only change (duration: 00m 12s)
23:52 logmsgbot: mattflaschen@tin Synchronized wmf-config/CommonSettings-labs.php: Beta-only change (duration: 00m 11s)
22:49 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/Citoid: https://gerrit.wikimedia.org/r/#/c/236218/ and https://gerrit.wikimedia.org/r/#/c/236222/ (duration: 00m 12s)
21:55 urandom: bouncing Cassandra on restbase1001 to restore default GC settings
18:36 logmsgbot: krenair@tin Synchronized w/static/images/project-logos/ukwikivoyage.png: https://gerrit.wikimedia.org/r/#/c/236063/ (duration: 00m 11s)
18:06 logmsgbot: krinkle@tin Synchronized php-1.26wmf21/extensions/WikimediaEvents/modules/ext.wikimediaEvents.statsd.js: Ib98988f67ef (duration: 00m 11s)
17:35 MaxSem: Maps: dropped duplicate index on water_polygons
16:27 jynus: cloning es1 mysql data from es1004 to es1018 [ETA:16h]
16:11 paravoid: updating firewall border ACLs and BGP border filters across all cr
15:42 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1002, es1016; Depool es1004 (duration: 00m 11s)
15:35 godog: python varnishlog collector + gdb running on cp1052 for debugging T83580
12:55 moritzm: restarted salt-master on palladium
12:47 moritzm: uploaded debdeploy 0.0.4 to carbon
10:18 logmsgbot: kartik@tin Synchronized php-1.26wmf21/extensions/ContentTranslation/api/ApiContentTranslationPublish.php: php-1.26wmf21/extensions/ContentTranslation/extension.json T111490:Use the VirtualRESTService to configure CX (duration: 00m 12s)
09:16 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-fr-ca_1.0.3~r61329-1
09:16 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-eo-fr_0.9.0~r28336-1
09:16 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-eo-es_0.9.1~r60655-1
09:16 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-eo-ca_0.9.1~r60655-1
09:16 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-ca-it_0.1.1~r57554-1
07:50 jynus: cloning es3 mysql data from es1008 to es1019
04:19 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Sep 4 04:19:20 UTC 2015 (duration 19m 19s)
02:26 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-04 02:26:04+00:00
02:23 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 05m 21s)
01:56 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: T111439 (duration: 00m 12s)
00:11 logmsgbot: krinkle@tin Synchronized php-1.26wmf21/includes/resourceloader/ResourceLoader.php: I24f68e34a9fa4918 (duration: 00m 12s)
00:06 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/235940/ (duration: 00m 11s)

2015-09-03

23:53 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/235853/ (duration: 00m 12s)
23:51 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/235843/ (duration: 00m 12s)
23:50 logmsgbot: krenair@tin Synchronized multiversion/MWMultiVersion.php: https://gerrit.wikimedia.org/r/#/c/235843/ (duration: 00m 12s)
23:41 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/235850/ (duration: 00m 12s)
23:40 logmsgbot: krenair@tin Synchronized w/static/images/project-logos/ukwikivoyage.png: https://gerrit.wikimedia.org/r/#/c/235850/ (duration: 00m 12s)
23:37 mutante: mw1224 - killed and restarted defunct hhvm, version is different from the one on mw1225
23:37 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/235728 (duration: 00m 13s)
23:36 logmsgbot: krenair@tin Synchronized w/static/images/project-logos/knwikisource.png: https://gerrit.wikimedia.org/r/#/c/235728/ (duration: 00m 12s)
23:32 Krenair: mw1224 has been sending segfault warnings and "Lost parent, LightProcess exiting" to hhvm.log since about 21:17:34
23:29 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/CirrusSearch: https://gerrit.wikimedia.org/r/#/c/235905/ (duration: 00m 13s)
23:28 logmsgbot: krenair@tin Synchronized php-1.26wmf21/package.json: bd2eb6cc1919c7dab056d5f8fe5b4a164236d78f (duration: 00m 13s)
23:02 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/235908/ (duration: 00m 13s)
21:21 ori: rebuilt HHVM with updated diff from facebook/hhvm PR #6071 (T109540), uploaded to apt as 3.6.5+dfsg1-1+wm5
21:18 urandom: bouncing Cassandra on restbase1001 to apply temporary GC settings
19:54 bearND: MobileApps deployed sha1 553c399
19:31 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.26wmf21
18:13 ottomata: rolling restart of hadoop yarn nodemanagers to pick up Yarn AppMaster port range limitation to apply ferm rules.
18:04 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Add plumbing code for Flow beta feature (unused for now) (duration: 00m 12s)
18:03 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Add plumbing code for Flow beta feature (unused for now) (duration: 00m 12s)
17:39 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/OpenStackManager/nova/OpenStackNovaController.php: https://gerrit.wikimedia.org/r/#/c/235769/ (duration: 00m 12s)
17:34 mutante: bromine - deleting policy docroot
17:06 jynus: cloning es1006 mysql data into es1015 [ETA:8h]
16:30 bblack: updating nginx->1.9.4 on cp1071, cp3033 for prod validation before broader rollout
16:30 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: es3 master switchover from es1009 to es1014 (eqiad) (duration: 00m 13s)
16:28 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: es3 master switchover from es1009 to es1014 (codfw) (duration: 00m 13s)
16:26 mutante: imported jenkins 1.609.3 into APT repo
16:23 legoktm: fixed content model of Template:Languages@metawiki
16:21 robh: re-enabling puppet on all mw systems
16:14 robh: disabling puppet on all mw systems for apache config update
16:01 jynus: performing es3 master switchover from es1009 to es1014
15:40 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: depool es1006 (duration: 00m 12s)
15:17 hashar: stopping nodepool on labnodepool1001.eqiad.wmnet not ready yet
15:15 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: es2 master switchover from es1006 to es1011 (eqiad) (duration: 00m 13s)
15:14 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: es2 master switchover from es1006 to es1011 (codfw) (duration: 00m 12s)
15:05 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 13s)
15:04 logmsgbot: demon@tin Synchronized php-1.26wmf21/extensions/Translate/: (no message) (duration: 00m 15s)
14:51 jynus: performing es2 master switchover from es1006 to es1011
14:33 paravoid: rebooting msw1-eqiad
14:28 twentyafterfour: restarted phd (phabricator daemon) to pick up new configuration
14:25 paravoid: changing IPv6 RA interval/lifetime/virtual-router-only @ eqiad
14:21 paravoid: rebooting msw1-codfw
13:17 paravoid: upgrading mr1-esams and mr1-eqiad to newer junos
13:13 godog: bounce carbon daemons on graphite1001
12:42 chasemp: unban elastic1001 and put back in service
12:24 chasemp: move all shards off of elastic1001
12:24 chasemp: disable elastic1001 in lvs as we are gonig to try fw apply round #2
11:02 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1028; increase the load of es1010, es1013 and es1017 (duration: 00m 12s)
10:45 jynus: applying schema change for ContentTranslation on x1-master "wikishared"
10:02 godog: reenable puppet on ms-be1*
09:16 jynus: started profiling mysql queries at phabricator. Only a 1% overhead is expected.
09:12 moritzm: updated rsyncd firewall rules (see https://gerrit.wikimedia.org/r/235425 for details)
09:12 godog: stop puppet on ms-be1* after ferm rsync change
08:23 godog: fixup current graphite retention T96662
07:26 moritzm: enabled ferm on dbstore* servers in codfw
06:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Sep 3 06:29:35 UTC 2015 (duration 29m 34s)
03:09 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-03 03:09:20+00:00
03:06 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 05m 32s)
02:45 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf20) at 2015-09-03 02:45:36+00:00
02:39 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 10m 41s)
01:32 logmsgbot: krenair@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 12s)
00:36 logmsgbot: ori@tin Synchronized php-1.26wmf21/includes/parser/Preprocessor_Hash.php: Idd1acd903: Decline to cache preprocessor items larger than 1 Mb (duration: 00m 11s)
00:36 logmsgbot: ori@tin Synchronized php-1.26wmf20/includes/parser/Preprocessor_Hash.php: Idd1acd903: Decline to cache preprocessor items larger than 1 Mb (duration: 00m 13s)
00:27 RoanKattouw: Deployed patch for T111029

2015-09-02

23:58 logmsgbot: andyrussg@tin Synchronized php-1.26wmf20/extensions/CentralNotice/: CentralNotice update (duration: 00m 13s)
23:33 logmsgbot: andyrussg@tin Synchronized php-1.26wmf21/extensions/CentralNotice/: Update CentralNotice (duration: 00m 13s)
23:02 logmsgbot: andyrussg@tin Finished scap: Update CentralNotice to 2.6.0 for wmf21 (duration: 48m 18s)
22:13 logmsgbot: andyrussg@tin Started scap: Update CentralNotice to 2.6.0 for wmf21
20:27 arlolra: updated Parsoid to version 5f2fae6c
20:08 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.26wmf21
20:02 logmsgbot: krinkle@tin Synchronized php-1.26wmf21/resources/src/startup.js: Ie65427caee (duration: 00m 12s)
19:09 mutante: restarted gitblit, stopped counting
19:07 paravoid: upgrading mr1-codfw, mr1-ulsfo to newer junos
19:01 urandom: bouncing Cassandra on restbase1001 to address bogus icinga process failure alert
18:52 legoktm: deployed patch for T110553
18:36 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf21
18:32 cmjohnson1: replacing disk 10 on db1028
18:13 urandom: bouncing Cassandra on restbase1001 to apply temporary GC settings
17:50 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/VisualEditor/modules/ve-mw/ui/inspectors: https://gerrit.wikimedia.org/r/#/c/235511/ (duration: 00m 12s)
17:07 logmsgbot: ori@tin Synchronized php-1.26wmf21/extensions/UniversalLanguageSelector: 78a5908fd9: Updated mediawiki/core Project: mediawiki/extensions/UniversalLanguageSelector (duration: 00m 16s)
17:07 logmsgbot: ori@tin Synchronized php-1.26wmf20/extensions/UniversalLanguageSelector: 2154acc529: Updated mediawiki/core Project: mediawiki/extensions/UniversalLanguageSelector (duration: 00m 13s)
16:25 mutante: restarting NTP on lvs2004
16:12 jynus: setting BBU auto-learn mode to warn only (disabled if not possible) on all database hosts
16:03 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/MultimediaViewer/MultimediaViewer.php: https://gerrit.wikimedia.org/r/#/c/235484/ (duration: 00m 12s)
16:01 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/UploadWizard/resources/mw.UploadWizardUploadInterface.js: https://gerrit.wikimedia.org/r/#/c/235486/ (duration: 00m 12s)
15:58 logmsgbot: krenair@tin Synchronized php-1.26wmf20/extensions/MultimediaViewer/MultimediaViewer.php: https://gerrit.wikimedia.org/r/#/c/235483/ (duration: 00m 13s)
15:56 logmsgbot: krenair@tin Synchronized php-1.26wmf20/extensions/UploadWizard/resources/mw.UploadWizardUploadInterface.js: https://gerrit.wikimedia.org/r/#/c/235485/ (duration: 00m 12s)
15:51 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: T110837 (duration: 00m 13s)
15:42 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/OpenStackManager/nova/OpenStackNovaController.php: https://gerrit.wikimedia.org/r/#/c/235482/ (duration: 00m 12s)
15:34 logmsgbot: krenair@tin Synchronized php-1.26wmf20/extensions/OpenStackManager/nova/OpenStackNovaController.php: https://gerrit.wikimedia.org/r/#/c/235479/ (duration: 00m 13s)
15:19 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/ContentTranslation/modules/tools/ext.cx.tools.template.js: https://gerrit.wikimedia.org/r/#/c/235442/ (duration: 00m 12s)
15:14 logmsgbot: krenair@tin Synchronized php-1.26wmf20/extensions/ContentTranslation/modules/tools/ext.cx.tools.template.js: https://gerrit.wikimedia.org/r/#/c/235441/ (duration: 00m 12s)
15:07 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234942/ and https://gerrit.wikimedia.org/r/#/c/234944/ (duration: 00m 13s)
14:40 Nikerabbit: TTMServer reindex complete
11:59 mark: removed tools LV snapshots on labstore1002
11:47 mark: kill STOP'ed rsync on labstore1002
11:00 jynus: cloning mysql data from es1002 into es1016 [ETA:16h]
10:30 moritzm: installed qemu security updates on labvirt*
09:41 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1002 (duration: 00m 12s)
09:21 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1010, pool es1017 (duration: 00m 13s)
09:19 hashar: Merged in "delete 1.26wmf12" https://gerrit.wikimedia.org/r/235347 which was left unmerged in Gerrit but was present on tin /srv/mediawiki-staging confusing people.
08:03 bblack: restarting ntp on lvs2004
08:01 moritzm: enable ferm on db1069/sanitarium
07:50 moritzm: enable ferm on remaining phabricator db hosts
04:54 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Sep 2 04:54:37 UTC 2015 (duration 54m 36s)
02:52 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-02 02:52:51+00:00
02:50 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 05m 09s)
02:29 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf20) at 2015-09-02 02:29:56+00:00
02:26 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 06m 31s)
00:33 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/235366/ (duration: 00m 13s)

2015-09-01

23:59 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/221731/ (duration: 00m 13s)
23:41 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/235285/ (duration: 00m 14s)
23:08 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/235362/ (duration: 00m 14s)
23:02 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/235361/ (duration: 00m 13s)
22:50 awight: update CRM from 0fc8474338e7a31fdde79287bd667b98cd96a252 to abc34b87ee9d1dbb1176f1929a3d748e1ee5ac7b
22:18 MaxSem: Maps: creating and populating admin table
21:20 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/235177/ (duration: 00m 12s)
20:54 ori: restarted nutcracker on mw1142
20:33 logmsgbot: twentyafterfour@tin Finished scap: sync 1.26wmf21 (duration: 30m 37s)
20:03 dcausse: unfreezing elasticsearch indices
20:03 logmsgbot: twentyafterfour@tin Started scap: sync 1.26wmf21
19:52 YuviPanda: removed tools20150901132642 from labstore vg on labstore1002
19:36 logmsgbot: ori@tin Synchronized php-1.26wmf20/includes/skins/SkinTemplate.php: cc643a0934: Deprecate unconditional loading of mediawiki.ui.button on all pages (duration: 00m 13s)
17:31 urandom: bouncing Cassandra on restbase1001 to apply temporary GC setting
17:28 dcausse: freezing elasticsearch indices before applying ferm fules on master
17:23 logmsgbot: aude@tin Synchronized php-1.26wmf20/extensions/Wikidata: Fix for change dispatcher (duration: 00m 20s)
16:45 jynus: performing schema change on testwiki and metawiki
16:12 robh: policy.wikimedia.org dns change happening now
16:00 chasemp: ferm for elastic1003/2/1(master)
15:57 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/235168/ (duration: 00m 13s)
15:51 YuviPanda: stopped replicate-tools on labstore1002, and cleaned out lockdir
15:47 logmsgbot: reedy@tin Synchronized php-1.26wmf20/extensions/SecurePoll/: Stop cronspam (duration: 00m 13s)
15:47 mark: labstore1002: echo 10000 > /sys/block/md123/md/sync_speed_min
15:44 mark: labstore1002: update-initramfs -k all -u
15:38 mark: labstore1002: mdadm /dev/md/slice51 --add /dev/sd{bh,bg,bf,be,bd,bc}
15:36 moritzm: disabled ferm in analytic1028, needs some more work on possibly dynamic mapreduce ports
15:16 mark: labstore1002: mdadm /dev/md/slice15 --re-add /dev/sd{bb,ba,az}
15:14 mark: labstore1002: mdadm /dev/md/slice15 --re-add /dev/sdaw
15:07 mark: labstore1002: mdadm --zero-superblock /dev/sd{aw,bh,bg,bf,be,bd,bc,bb,ba,az}1
15:04 moritzm: enabled ferm in analytic1028 (initial hadoop worker)
15:04 mark: labstore1002: mdadm --zero-superblock /dev/sdax1 && mdadm /dev/md/slice15 --re-add /dev/sdax
15:03 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/231465/ - VE for all new enwiki accounts (duration: 00m 13s)
14:58 mark: labstore1002: mdadm /dev/md/slice15 --re-add /dev/sday
14:58 mark: labstore1002: mdadm --zero-superblock /dev/sday1
14:53 mark: labstore1002: mdadm --stop /dev/md3
14:37 ebernhardson: reset elasticsearch cluster.routing.allocation.disk.high back to 90%
13:38 logmsgbot: krinkle@tin Synchronized w/: Remove rl-test.php (duration: 00m 13s)
13:17 moritzm: enabled ferm on db1048
13:09 moritzm: enabled ferm on labsdb100[467]
12:01 YuviPanda: disable puppet on labsdb1006
08:58 moritzm: enabled ferm on labsdb1001
08:58 godog: fixup current graphite retention for metrics under "servers" hierarchy T96662
08:51 moritzm: enabled ferm on labsdb1002
08:31 moritzm: enabled ferm on labsdb1003
08:29 godog: repool mw1125 mw1142 after nutcracker failures
07:45 jynus: cloning mysql data from es1010 to es1017 [ETA: 6h]
07:23 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1010 (duration: 00m 12s)
07:13 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1007, pool es1013 (duration: 00m 13s)
06:36 mutante: uploaded survey2012 to dumps/dataset1001; ownership as it is for survey2011; - T110746 in time for midnight PST
05:18 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Sep 1 05:18:09 UTC 2015 (duration 18m 8s)
02:28 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf20) at 2015-09-01 02:28:30+00:00
02:25 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 06m 00s)

2015-08-31

23:56 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/233665/ (duration: 00m 11s)
23:49 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: reenable config changes for cirrus experimental completion api (duration: 00m 12s)
23:40 logmsgbot: ori@tin Synchronized php-1.26wmf20/extensions/EducationProgram: 97ab82eab2: Updated mediawiki/core Project: mediawiki/extensions/EducationProgram 85a7d3932c1a4ad28f1a8dd05704f4e524152349 (duration: 00m 14s)
23:27 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf20/extensions/CirrusSearch/: (no message) (duration: 00m 12s)
23:25 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: revert update for cirrussearch experimental suggestions api (duration: 00m 12s)
23:21 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: update config of cirrussearch experimental suggestions api (duration: 00m 12s)
22:45 chasemp: disabled puppet on elastic hosts temporarily to safely roll out fw change. elastic seems to have not taken it well and I'm holding for green cluster state.
21:20 mutante: installing package upgrades on argon
20:58 ori: imported pybal_1.08_amd64.changes to jessie-wikimedia
20:44 chasemp: ferm for elastic100[4-7] and adjust ferm to include wikitech source
20:21 subbu: deployed parsoid version c3e4df5e
16:22 godog: depool mw1125 + mw1142 from api, nutcracker client connections exceeded
16:06 logmsgbot: thcipriani@tin Finished scap: SWAT: Ask the user to log in if the session is lost gerrit:234228 (duration: 27m 07s)
15:59 jynus: restarting hhvm on mw2187
15:39 logmsgbot: thcipriani@tin Started scap: SWAT: Ask the user to log in if the session is lost gerrit:234228
15:33 mutante: terbium - Could not find dependent Service[nscd] for File[/etc/ldap/ldap.conf]
15:28 logmsgbot: thcipriani@tin Synchronized closed-labs.dblist: SWAT: Creating closed-labs.dblist and closing es.wikipedia.beta.wmflabs.org gerrit:234594 (duration: 00m 13s)
15:25 logmsgbot: thcipriani@tin Synchronized wmf-config/CirrusSearch-common.php: SWAT: Remove files from Commons from search results on wikimediafoundation.org gerrit:234040 (duration: 00m 11s)
15:25 ottomata: starting varnishkafka instances on frontend caches to produce eventlogging client side events to kafka
15:21 logmsgbot: thcipriani@tin Synchronized php-1.26wmf20/extensions/Wikidata: SWAT: Update Wikidata - Fix formatting of client edit summaries gerrit:234991 (duration: 00m 21s)
15:16 logmsgbot: thcipriani@tin Synchronized php-1.26wmf20/extensions/UploadWizard/resources/controller/uw.controller.Step.js: SWAT: Keep the uploads sorted in the order they were created in initially gerrit:234553 (duration: 00m 12s)
14:43 ebernhardson: elasticsearch cluster.routing.allocation.disk.watermark.high set to 75% to force elastic1022 to reduce its disk usage
14:41 urandom: bouncing Cassandra on restbase1001 to apply temporary GC setting
14:06 akosiaris: rebooted krypton. was reporting 100% cpu steal time
13:40 paravoid: running puppet on newly-installed mc2001
13:40 paravoid: restarting hhvm on mw1065
11:10 moritzm: restart salt-master on palladium
10:45 paravoid: reenabling asw2-a5-eqiad:xe-0/0/36 (T107635)
10:36 godog: repool ms-fe1004
10:32 godog: repool ms-fe1003 and depool ms-fe1004 for firewall changes
10:19 godog: update graphite retention policy on files with previous retention and older than 30d T96662
10:18 godog: repool ms-fe1002 and depool ms-fe1003 for firewall changes
10:05 godog: depool ms-fe1002 to apply firewall changes
09:55 jynus: cloning es1007 mysql data into es1013 (ETA: 5h30m)
09:51 godog: repool ms-fe1001
09:35 godog: depool ms-fe1001 in preparation for ferm changes
09:27 godog: update graphite retention policy on files with previous retention and older than 60d T96662
09:25 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1007 for maintenance (duration: 00m 13s)
08:33 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1028, return ES servers back from maintenance (duration: 00m 12s)
04:34 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Aug 31 04:34:14 UTC 2015 (duration 34m 13s)
04:05 bblack: disabled ipv6 autoconf on neon, flushed old dynamic addr
02:32 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf20) at 2015-08-31 02:32:25+00:00
02:29 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 06m 42s)

2015-08-30

12:58 godog: lvchange -ay labstore/others on labstore1002
12:52 godog: start-nfs on labstore1002
12:31 godog: lvchange -ay labstore/tools on labstore1002
12:30 godog: also disabled puppet on labstore1002 while investigating
12:15 godog: trying to manually assemble missing raid on labstore1002 with mdadm --assemble /dev/md/slice51 --uuid 0747643d:b89b36ff:57156095:c33694fc --verbose
11:19 YuviPanda: powered labstore1002 back up
11:17 YuviPanda: shut down labstore1002, going to powercycle from mgmt
10:34 YuviPanda: disabled backups on labstore1002 to prevent overwriting of good backups on 2001
10:08 YuviPanda: rebooted labstore1002
04:16 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Aug 30 04:16:17 UTC 2015 (duration 16m 16s)
02:23 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf20) at 2015-08-30 02:23:07+00:00
02:20 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 05m 36s)

2015-08-29

15:26 jynus: killing idle mysql connections from phabricator and setting wait and interactive timeout to 60
09:30 jynus: SCAP failed, cannot depool db1028
09:28 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1028, return ES servers back from maintenance (duration: 00m 03s)
09:28 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1028, return ES servers back from maintenance (duration: 00m 03s)
09:05 jynus: about to depool db1028 due to disk issue
04:17 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Aug 29 04:17:55 UTC 2015 (duration 17m 54s)
02:24 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf20) at 2015-08-29 02:24:01+00:00
02:21 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 05m 48s)

2015-08-28

23:45 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234679/ (duration: 06m 56s)
22:51 logmsgbot: bd808@tin Synchronized wmf-config/CommonSettings-labs.php: Use ffmpeg instead of avconv on labs beta (I250fe33) (duration: 06m 05s)
22:05 ori: disabling puppet on tin for a few minutes to test an ssh-agent-proxy change
20:04 logmsgbot: catrope@tin Synchronized php-1.26wmf20/resources/src/mediawiki.legacy/shared.css: T110716 (duration: 00m 12s)
18:09 robh: updating ldap-codfw cert
17:10 logmsgbot: catrope@tin Synchronized php-1.26wmf20/extensions/Flow/includes/Parsoid/Utils.php: T110676 (duration: 00m 13s)
17:08 urandom: bouncing Cassandra on restbase1001 to apply default (puppet-managed) settings
16:03 chasemp: ferm for elasticsearch10(0[8-9|1[0-13])
15:31 awight: updated crm from fc0fcc8f5af262b56392d3f4f5998f8ea08c99a8 to 0fc8474338e7a31fdde79287bd667b98cd96a252
15:23 chasemp: ferm for elasticsearch10[14-17]
11:09 logmsgbot: aude@tin Synchronized php-1.26wmf20/extensions/Wikidata/Wikidata.php: Sync entry point - updated to work on Jenkins together with ContentTranslation (duration: 00m 12s)
10:29 godog: reenable puppet on ms-fe1, ferm changes will go out on monday
09:48 jynus: Cloning es1001 database into es1012
09:45 moritzm: enabled ferm for swift on esams
09:28 moritzm: enabled ferm on strontium puppetmaster backend
09:00 moritzm: enabled ferm on rhodium puppetmaster backend
08:29 moritzm: uploaded debdeploy 0.0.3 to carbon
08:23 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1001, increas weight of es1011, pool es1014 for the first time (duration: 00m 13s)
05:59 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Aug 28 05:59:09 UTC 2015 (duration 59m 8s)
04:58 logmsgbot: ori@tin Synchronized php-1.26wmf20/includes/parser/Parser.php: 754b222daf: Add ParserOutput cache and expiry times to NewPP report (duration: 00m 13s)
02:41 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf20) at 2015-08-28 02:41:26+00:00
02:35 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 10m 47s)
01:59 Tim: on ruthenium: started parsoid_vd which was previously killed by oom-killer
01:58 Tim: on ruthenium, reduced parsoid-rt-client concurrency from 16 to 8 since it was OOM and oom-killer was killing random things
01:37 Tim: on ruthenium restarted parsoid-rt-client and parsoid-vd-client
00:24 mutante: powercycled mw2027
00:19 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/234450/ (duration: 01m 14s)
00:06 logmsgbot: krenair@tin Synchronized wmf-config/mobile.php: live hack to make previous commit work (duration: 01m 14s)
00:05 Krenair: Another codfw host broke: mw2027
00:01 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234330/ (duration: 00m 13s)

2015-08-27

23:58 logmsgbot: krenair@tin Synchronized php-1.26wmf20/extensions/MobileFrontend/includes/MobileFormatter.php: https://gerrit.wikimedia.org/r/#/c/234331/1 (duration: 00m 12s)
23:57 logmsgbot: krenair@tin Synchronized php-1.26wmf20/extensions/MobileFrontend/includes/config/Experimental.php: https://gerrit.wikimedia.org/r/#/c/234331/1 (duration: 00m 14s)
23:55 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/233439/ (duration: 00m 12s)
23:30 logmsgbot: krenair@tin Synchronized php-1.26wmf20/extensions/Gadgets/extension.json: touch (duration: 00m 13s)
23:24 logmsgbot: krenair@tin Synchronized php-1.26wmf20/includes/DefaultSettings.php: https://gerrit.wikimedia.org/r/#/c/234328/ (duration: 00m 12s)
23:24 logmsgbot: krenair@tin Synchronized php-1.26wmf20/includes/registration/ExtensionProcessor.php: https://gerrit.wikimedia.org/r/#/c/234328/ (duration: 00m 12s)
23:23 logmsgbot: krenair@tin Synchronized php-1.26wmf20/includes/MWNamespace.php: https://gerrit.wikimedia.org/r/#/c/234328/ (duration: 00m 13s)
23:15 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/234009/ (duration: 00m 13s)
23:04 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/233100/ (duration: 00m 12s)
20:11 chasemp: ferm setup on elasticsearch10(1[8-9|2[0-3])
20:06 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.26wmf20
19:57 logmsgbot: twentyafterfour@tin Synchronized php-1.26wmf20/includes/media/XMP.php: deploy fix for T89532 on 1.26wmf20 (duration: 00m 13s)
18:16 chasemp: setting up ferm on elastic1027-31
17:47 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/234320/ (duration: 00m 13s)
17:43 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234320/2 (duration: 00m 13s)
17:37 urandom: ack'd Cassandra process alert on restbase1001; temporary command args have pushed the class name beyond the limit
17:34 logmsgbot: krenair@tin Synchronized multiversion/MWMultiVersion.php: (no message) (duration: 00m 12s)
17:24 logmsgbot: krenair@tin Synchronized multiversion/MWMultiVersion.php: https://gerrit.wikimedia.org/r/#/c/234320/ (duration: 00m 12s)
17:08 urandom: bouncing Cassandra on restbase1001 to apply temporary GC settings
16:51 moritzm: ferm rules on logstash100[1-3] have been amended to allow grafana from reading dashboard configs
16:39 bd808: new ferm rules on logstash100[1-3] are blocking grafana from reading dashboard configs.
16:22 moritzm: ferm enabled on logstash1003
16:18 moritzm: ferm enabled on logstash1002
16:16 bd808: ferm enabled on logstash1001
16:06 bd808: logstash1001 back up after system reboot; we applied a default drop rule without applying the other iptables changes; will try again
15:58 chasemp: rebooting logstash1001.mgmt.eqiad.wmnet for moritz as it is having issues
15:47 bblack: killed hung ubuntu mirror rsync commands on carbon, from Jul 10
15:45 bd808: logstash1001 not responding over ssh following ferm rules application; moritzm investigating
15:30 bd808: Disabled puppet on logstash100[1-3] prior to trying to enable ferm
15:11 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable newarticle campaign in itwiki gerrit:234223 (duration: 01m 52s)
14:52 bblack: re-imaging lvs200[123]
14:47 godog: reenable puppet on ms-be1*
14:22 godog: disable puppet on ms-fe1 / ms-be1 in prepration for puppet work
14:15 godog: reenable puppet on ms-fe2*
13:47 bblack: re-imaging lvs2004 + lvs2005
13:29 ottomata: doing rolling restart of kafka brokers to apply auto_create_topics change
13:21 godog: enable puppet on ms-be2*
13:21 ottomata: stopping kafka on analytics1021, it is no longer a kafka broker.
13:09 godog: disable puppet on ms-be2* in preparation for firewall changes
13:09 jynus: cloning es1008 into es1014
13:04 ottomata: running leader election now that all topics and partitions are rebalanced across new kafka nodes
12:46 bblack: re-imaging lvs2006
12:45 andrewbogott: re-imaging labnet1001 (I hope)
11:33 _joe_: restarted hhvm on mw1143, locked in __lll_lock_wait for stat_cache deadlock
11:10 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Pool es1011 for the first time, depool es1008 (duration: 00m 12s)
09:27 jynus: installing and configuring servers es1012-es1019
06:39 ostriches: tin: dropped useless "gerrit" remote from /srv/mediawiki-staging (uses ssh, lol), pointed {origin,readonly} at the actual repo instead of a redirect.
06:00 _joe_: powercycling mw2140, not responding to ping, blank console
03:17 awight: deploy config cleanup for paymentswiki
02:38 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf19/cache/l10n: l10nupdate for 1.26wmf19 (duration: 10m 44s)
02:16 awight: push config change to the payments orphan slayer: explitly give stomp port to work around strict notice, clean up unused globals. T109911
01:32 ejegg: updated payments from 8ba4b5299f195cf48e6809b18a21e2d53f6eec1b to 6ac552f280fb839069d117386c4ecbe9e52f90a8
00:31 twentyafterfour: finished phabricator upgrade, everything appears to be working
00:24 logmsgbot: aaron@tin Synchronized php-1.26wmf19/extensions/CentralAuth: 47e181adb2898977b146de7398eaa35aebb870e3 (duration: 01m 13s)
00:22 logmsgbot: aaron@tin Synchronized php-1.26wmf20/extensions/CentralAuth: 47e181adb2898977b146de7398eaa35aebb870e3 (duration: 01m 13s)
00:20 twentyafterfour: taking phabricator offline for scheduled upgrade

2015-08-26

23:59 Krinkle: mwscript deleteEqualMessages.php --wiki rowiki
23:57 yurik: git deployed tilerator - had the 4/5 issue - https://phabricator.wikimedia.org/T110434
23:46 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234072/ (duration: 01m 12s)
23:37 logmsgbot: krenair@tin Synchronized php-1.26wmf20/maintenance/deleteEqualMessages.php: https://gerrit.wikimedia.org/r/#/c/234038/ (duration: 01m 12s)
23:35 logmsgbot: krenair@tin Synchronized php-1.26wmf19/maintenance/deleteEqualMessages.php: https://gerrit.wikimedia.org/r/#/c/234037/1 (duration: 01m 12s)
23:27 yurik: deployed kartotherian
23:21 jynus: cloning es1005 into es1011, ETA 9 hours
22:41 ori: armed keyholder on tin
22:40 ori: Disabled Puppet on mw1017 for 2hrs and applied I059b0c96c9 for testing.
21:55 logmsgbot: krinkle@tin Synchronized php-1.26wmf19/includes/poolcounter/PoolWorkArticleView.php: (no message) (duration: 01m 12s)
21:48 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1005 (duration: 01m 12s)
21:40 logmsgbot: krinkle@tin Synchronized php-1.26wmf20/includes/poolcounter/PoolWorkArticleView.php: (no message) (duration: 01m 12s)
21:32 ori: Disabling Puppet on tin again to test an ssh-agent-proxy change
20:30 logmsgbot: ori@tin Synchronized README: testing ssh-agent-proxy changes (duration: 00m 13s)
20:25 ori: Disabling puppet on tin and hacking some debug logging into ssh-agent-proxy
20:24 ori: armed ssh-agent key on mira
20:21 logmsgbot: krinkle@tin Synchronized php-1.26wmf20/includes/poolcounter/PoolWorkArticleView.php: (no message) (duration: 00m 03s)
20:11 subbu: deployed parsoid version 44d657de
19:52 logmsgbot: krenair@tin Synchronized php-1.26wmf20/extensions/Echo/includes/mapper/EventMapper.php: https://gerrit.wikimedia.org/r/#/c/234082/ (duration: 00m 12s)
19:47 mutante: sodium - deleting shunted messages older than 7 days
19:23 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234042/ (duration: 00m 12s)
19:22 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/234024/ (duration: 00m 12s)
19:20 logmsgbot: krenair@tin Synchronized multiversion/MWWikiversions.php: https://gerrit.wikimedia.org/r/#/c/232672/ (duration: 00m 12s)
18:50 logmsgbot: krinkle@tin Synchronized php-1.26wmf20/maintenance/deleteEqualMessages.php: (no message) (duration: 00m 11s)
18:50 logmsgbot: krinkle@tin Synchronized php-1.26wmf19/maintenance/deleteEqualMessages.php: (no message) (duration: 00m 13s)
18:38 twentyafterfour: ^ stupid typo. That sync was group1 to 1.26wmf20
18:37 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: tig
18:31 logmsgbot: ori@tin Synchronized w/404.php: Ided1facc0: Remove auto-redirection from 404 page. (duration: 00m 13s)
17:51 ejegg: updated SmashPig from 258f2c917b1ae50b01231927bcd6f58ecaa8940b to fdb053efa617162ac9f695e493c390987a069140
17:30 urandom: bouncing Cassandra on restbase1001 to apply temporary GC setting
17:12 andrewbogott: ok, /now/ I’m running a dist-upgrade on labcontrol1001, to sort out weird oslo dependencies
17:09 chasemp: adding firewall to elasticsearch2[4-6] (3 was just done as a pilot)
17:03 andrewbogott: upgraded labnet1002 nova services to Juno
16:34 andrewbogott: stopping keystone, updating db, restarting
16:18 andrewbogott: switching labcontrol1001 hiera to Juno which will add the cloud-archive repo for Juno.
16:11 andrewbogott: backing up labs openstack databases into /home/andrew/openstackdbbackups on db1009
16:11 andrewbogott: starting labs openstack update to Juno
15:53 moritzm: ferm enabled on elastic1023
15:45 godog: repool restbase1009 in pybal
15:28 logmsgbot: thcipriani@tin Synchronized php-1.26wmf20/extensions/Wikidata: SWAT: Update Wikidata - wrap usage tracking batch updates in transaction gerrit:233970 (duration: 00m 23s)
13:47 andrewbogott: rebooting/reimaging labnet1001
13:11 mobrovac: restbase deploying 1dfba85
12:54 yurik: git synced kartotherian
11:02 jynus: dropping optin_survey_old table on all wikis
10:33 godog: reenable puppet on ms-fe/ms-be, base::firewall still not enabled
09:58 godog: test-reboot ms-be2001
08:17 godog: disable puppet on ms-be/ms-fe in preparation for merging firewall changes
07:53 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Aug 26 07:53:31 UTC 2015 (duration 53m 30s)
07:01 jynus: restarting mw1239 HHVM, which is unresponsive
04:47 logmsgbot: ori@tin Synchronized wmf-config: I73721936: Enable ParsoidBatchAPI everywhere (duration: 00m 13s)
03:11 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf20) at 2015-08-26 03:11:29+00:00
03:06 logmsgbot: awight@tin Synchronized wmf-config/InitialiseSettings-labs.php: Push labs config to keep in sync with master (duration: 00m 13s)
03:05 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 10m 45s)
02:37 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf19) at 2015-08-26 02:37:51+00:00
02:34 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf19/cache/l10n: l10nupdate for 1.26wmf19 (duration: 06m 29s)
02:00 ottomata: kafka topic webrequest_upload has finished rebalancing across new brokers. starting move of last topic webrequest_text
01:50 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf19/extensions/Flow/: Sync Flow for reply fix (duration: 00m 15s)
00:28 logmsgbot: ori@tin Synchronized php-1.26wmf20/extensions/Scribunto/engines/LuaCommon/LuaCommon.php: (no message) (duration: 00m 13s)
00:26 logmsgbot: ori@tin Synchronized php-1.26wmf19/extensions/Scribunto/engines/LuaCommon/LuaCommon.php: (no message) (duration: 00m 13s)
00:26 Danny_B: 2586dd1c7c obviously broke many pages
00:19 logmsgbot: ori@tin Synchronized php-1.26wmf19/extensions/Scribunto/engines/LuaCommon/LuaCommon.php: (no message) (duration: 00m 14s)
00:14 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I79ffa78fa: Collection/OCG: Turn on plain text output format in Book Creator (duration: 00m 12s)
00:12 logmsgbot: ori@tin Synchronized php-1.26wmf20/extensions/Scribunto/engines/LuaCommon/LuaCommon.php: 2586dd1c7c: Updated mediawiki/core Project: mediawiki/extensions/Scribunto (duration: 00m 13s)

2015-08-25

23:39 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/233860/ (duration: 00m 12s)
23:16 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/233872/ (duration: 00m 13s)
23:13 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/232963/ (duration: 00m 12s)
23:12 logmsgbot: krenair@tin Synchronized wmf-config/extension-list: https://gerrit.wikimedia.org/r/#/c/232963/ (duration: 00m 12s)
23:10 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/232962/ (duration: 00m 12s)
23:10 logmsgbot: krenair@tin Synchronized wmf-config/extension-list: https://gerrit.wikimedia.org/r/#/c/232962/ (duration: 00m 12s)
23:05 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/233781/ (duration: 00m 12s)
22:20 cscott: updated Parsoid to version c3b037b0
22:10 ejegg: disabled paypal audit downloader and parser due to them warning of incorrect data
21:16 logmsgbot: ori@tin Synchronized php-1.26wmf19/extensions/AbuseFilter: I15f5b5b6 & I9c23b607 (duration: 00m 13s)
21:13 logmsgbot: ori@tin Synchronized php-1.26wmf19/extensions/Cite/modules/ext.cite.styles.css: 7344e02216: Updated mediawiki/core Project: mediawiki/extensions/Cite (duration: 00m 12s)
21:09 logmsgbot: ori@tin Synchronized php-1.26wmf20/extensions/AbuseFilter: I15f5b5b6 & I9c23b607 (duration: 00m 14s)
20:54 tgr: finished OAuth migration
20:34 logmsgbot: tgr@tin Synchronized wmf-config/CommonSettings.php: make OAuth DB writable again T108648 (duration: 00m 12s)
20:32 logmsgbot: tgr@tin Synchronized wmf-config/CommonSettings.php: change wgMWOAuthCentralWiki mediawikiwiki -> metawiki T108648 (duration: 00m 12s)
20:24 logmsgbot: tgr@tin Synchronized wmf-config/CommonSettings.php: set OAuth to readonly for DB migration T108648 (duration: 00m 13s)
20:13 subbu: deployed parsoid version 759916fc
19:24 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.26wmf20
19:21 logmsgbot: twentyafterfour@tin Finished scap: testwiki to 1.26wmf20 (duration: 50m 12s)
18:31 logmsgbot: twentyafterfour@tin Started scap: testwiki to 1.26wmf20
17:11 YuviPanda: run authdns-update on radon (ns0.wikimedia.org)
17:10 urandom: bouncing Cassandra on restbase1001 to apply temporary GC settings
16:58 Krinkle: mwscript deleteEqualMessages.php --wiki kawiki
16:56 andrewbogott: restarting pdns on labcontrol1001 and labcontrol2001 to handle a nembus reboot
16:53 Krinkle: mwscript deleteEqualMessages.php --wiki huwiki
16:31 Krinkle: mwscript deleteEqualMessages.php --wiki frwiki
16:17 Krinkle: mwscript deleteEqualMessages.php --wiki frpwiki
15:50 godog: powercycle ms-be1004, likely xfs
15:44 andrewbogott: dist-upgrade and rebooting nembus in an attempt to resolve this acpi_pad issue
15:36 Krinkle: mwscript deleteEqualMessages.php --wiki euwiki (T45917)
15:29 Krinkle: mwscript deleteEqualMessages.php --wiki eowiki (T45917)
15:07 logmsgbot: krenair@tin Synchronized php-1.26wmf19/extensions/Flow: https://gerrit.wikimedia.org/r/#/c/233718/ (duration: 00m 16s)
13:56 jynus: dropping old tables on s7 - T5493
13:48 jynus: dropping old tables on s6 - T54932
12:53 Jeff_Green: authdns-update to change bismuth's IP
11:16 jynus: dropping old tables on s3 - T54932
10:46 jynus: dropping old tables on s2 - T54932
10:05 YuviPanda: restart puppetmaster on labcontrol1001 for https://gerrit.wikimedia.org/r/#/c/233184/
07:35 _joe_: stopping redis, wiping aof, restarting redis on rdb100{1,2} - snapshot saved on rdb1002:/root
07:12 _joe_: stopping redis on rdb1003,4, wiping AOF, restarting
06:38 jynus: performing schema change on officewiki, mediawikiwiki and metawiki
02:21 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf19/cache/l10n: l10nupdate for 1.26wmf19 (duration: 06m 26s)
01:48 ottomata: starting move of kafka partitions for topic webrequest_upload to new brokers. this will take a while!
01:44 ottomata: restarting kafka on new brokers kafka1013,1014,1020 to apply increase in num.replica.fetchers

2015-08-24

23:46 logmsgbot: mattflaschen@tin Synchronized wmf-config: Remove wgFlowOccupyPages (duration: 00m 12s)
23:38 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/233636/ (duration: 00m 12s)
22:16 logmsgbot: tgr@tin Synchronized wmf-config/CommonSettings-labs.php: change OAuth DB on beta +enable writes (duration: 00m 12s)
21:55 logmsgbot: tgr@tin Synchronized wmf-config/CommonSettings-labs.php: set beta OAuth to readonly (duration: 00m 13s)
21:54 logmsgbot: tgr@tin Synchronized wmf-config/CommonSettings-labs.php: set beta OAuth to readonly (duration: 00m 13s)
21:42 akosiaris: enabled puppet on maps-test200{1,2,3,4}.codfw.wmnet
20:21 arlolra: updated Parsoid to version 0b2fbae7
18:58 bblack: reloading primary LVS pybals for BlankPage change ( https://gerrit.wikimedia.org/r/#/c/233053/ ) + ulimit fixup ( https://gerrit.wikimedia.org/r/#/c/233484/ )
18:31 bblack: reloading backup LVS pybals for BlankPage change ( https://gerrit.wikimedia.org/r/#/c/233053/ )
17:19 urandom: bouncing Cassandra on restbase1001 to apply temporary GC settings
16:23 logmsgbot: bd808@tin Purged l10n cache for 1.26wmf18
16:23 logmsgbot: bd808@tin Purged l10n cache for 1.26wmf17
16:05 andrewbogott: rebooting labnet1001
15:53 _joe_: restarted nutcracker on mw1010, holding a 150 GB deleted logfile
15:47 Krenair: running sync-common on mw1010 to bring it up to date after clearing some space
15:44 logmsgbot: krenair@tin Purged l10n cache for 1.26wmf16
15:41 logmsgbot: krenair@tin Purged l10n cache for 1.26wmf15
15:38 logmsgbot: krenair@tin Synchronized php-1.26wmf19/extensions/Wikidata: https://gerrit.wikimedia.org/r/#/c/233411/1 (duration: 00m 49s)
15:37 hashar: stopped and restarted Zuul
15:31 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/232919/ and https://gerrit.wikimedia.org/r/#/c/232915/ (duration: 01m 34s)
15:29 logmsgbot: krenair@tin Synchronized w/static/images/project-logos/knwikiquote.png: https://gerrit.wikimedia.org/r/#/c/232919/ (duration: 02m 04s)
15:19 Krenair: No space left on mw1010, cannot ping or ssh to mw2180
15:16 logmsgbot: krenair@tin Synchronized docroot/noc/db.php: https://gerrit.wikimedia.org/r/#/c/232920/ (duration: 01m 34s)
15:14 hashar: apt-get upgrade on gallium
14:48 andrewbogott: forcing wikitech logouts in order to flush everyone’s service catalog
14:18 ottomata: starting to move kafka topic-partitions to new brokers (and off of analytics1021)
14:12 yurik: git deploy synced kartotherian
13:55 akosiaris: disable puppet on fermium preparing for reinstallation
13:55 akosiaris: disable puppet on fermium
12:54 akosiaris: stop etcd on etcd1002.eqiad.wmnet. Already removed from the cluster
11:58 _joe_: stopping etcd on etcd1001
11:50 _joe_: restarting etcd on etcd1001
09:00 YuviPanda: starting up replicate for tools on labstore1002
09:00 YuviPanda: cleaning up lockdir on labstore for maps and tools
09:00 YuviPanda: others replication on labstore1002 completed successfuly
08:31 YuviPanda: cleaned up others lockdir for replication on labstore1002 and started it manually
06:43 jynus: reloading dbproxy1003 service
02:21 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf19/cache/l10n: l10nupdate for 1.26wmf19 (duration: 06m 36s)

2015-08-23

16:54 urandom: bouncing Cassandra on restbase1001 to apply temporary GC settings
02:20 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf19/cache/l10n: l10nupdate for 1.26wmf19 (duration: 06m 23s)

2015-08-22

23:08 logmsgbot: krenair@tin Synchronized php-1.26wmf19/extensions/AbuseFilter/maintenance/addMissingLoggingEntries.php: (no message) (duration: 01m 05s)
19:41 YuviPanda: manually remove old snapshots from labstore1002
17:28 chasemp: tweaking apache on iridum T109941
16:45 chasemp: scratch that as we have mpm_prefork enabled :)
16:33 chasemp: raising values in mpm_worker.conf for iridium to to debug and hopefully head off further crashing
14:44 twentyafterfour: restarted apache2 on iridium. Segfault again. This time I at least got one clue in the log: "zend_mm_heap corrupted"
09:18 twentyafterfour: phabricator seems stable now, restarting apache2 on iridium did the trick, unfortunately we didn't learn why
08:36 twentyafterfour: restarted phd on iridium
08:36 twentyafterfour: restarted apache2 on iridium
02:20 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf19/cache/l10n: l10nupdate for 1.26wmf19 (duration: 06m 09s)
00:26 mutante: deleting blog.sh and blog_pageviews crontab from stat1003

2015-08-21

23:34 urandom: restarting Cassandra on restbase1001 to restore baseline settings
23:11 yurik: synced kartotherian
22:35 mutante: deleting held messages on mailman that are older than 1 year
21:56 awight: increasing paymentswiki orphan gc-cc-limbo expiry time to 30 days
21:45 mutante: had to reset list creator password for mailman - ask me if you think you should have it and don't (this is not the master pass)
20:37 logmsgbot: ori@tin Synchronized php-1.26wmf19/includes: I1eb8dfc: Revert Count API and hook calls, with 1:1000 sampling (duration: 01m 09s)
19:43 awight: update paymentswiki from 2b08853c977eee0fd17bf00a673a3bbf2a146554 to 8ba4b5299f195cf48e6809b18a21e2d53f6eec1b
18:58 awight: disabling Amazon gateway
18:52 awight: updated paymentswiki from 049ad15323564fd5cd7f5efcadddb532a3590cef to 2b08853c977eee0fd17bf00a673a3bbf2a146554
16:06 jynus: checksumming dewiki database, higher write rate/dbstore lag expected temporarily
15:10 ottomata: rebooting kafka broker analytics1021 to hopefully reload /dev/sdg with new disk, also will turn on hyperthreading
14:13 ottomata: rebooting analytics1056 after upgrading kernel to linux-image-3.13.0-61-generic
13:58 urandom: restarting restbase1001 to apply temporary GC setting
13:34 ottomata: stopping kafka broker on analytics1021 due to bad disk.
13:30 bblack: wiped ganglia apache access log on uranium, to free up half of the (full) rootfs
10:07 godog: enable puppet on ms-fe1/ms-be1
09:49 godog: disable puppet on ms-fe1/ms-be1 before merging https://gerrit.wikimedia.org/r/#/c/231240/
07:06 _joe_: restarting gitblit, because it will be decommissioned "soon"...
02:34 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf19/cache/l10n: l10nupdate for 1.26wmf19 (duration: 11m 19s)

2015-08-20

23:40 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf19/extensions/CirrusSearch/: Fix some cirrussearch logspam (duration: 00m 13s)
23:30 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 12s)
23:29 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 13s)
23:23 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/232854/ (duration: 00m 13s)
23:22 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/232671/ (duration: 00m 12s)
23:15 logmsgbot: krenair@tin Synchronized php-1.26wmf19/extensions/LiquidThreads/classes/Hooks.php: https://gerrit.wikimedia.org/r/#/c/232783/ (duration: 00m 12s)
23:13 logmsgbot: krinkle@tin Synchronized php-1.26wmf19/includes/resourceloader/ResourceLoaderFileModule.php: T102578 (duration: 00m 13s)
23:08 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/232782/ (duration: 00m 12s)
22:48 logmsgbot: ori@tin Synchronized php-1.26wmf19/includes/libs/CSSMin.php: Icc1c23a2: CSSMin: remove dot segments in relative local URLs (duration: 00m 12s)
21:36 cscott: updated Parsoid to version db6e6404f67a9f971b4fbefe9de239735426c738
21:25 matt_flaschen: Ran FlowUpdateRevContentModelFromOccupyPages.php on all wikis
20:41 twentyafterfour: scap failed to sync to mw2180.codwf.wmnet
20:41 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.26wmf19
20:38 logmsgbot: twentyafterfour@tin Synchronized php-1.26wmf19: Silence the undefined index error in CirrusSearch (duration: 06m 24s)
19:40 chasemp: moving enwiki_content_1432182861 elastic shard from 1022 to 1004 due to space (1022 is at 91%)
20:57 mutante: no log bot
18:56 mutante: labvirt1007 "only" 29G space left - but since we have 2.2T there that means 99% full
17:39 ottomata: stopping kafka on analytics1018 and bringing it down for reinstall as kafka1018 with Jessie
16:38 YuviPanda: puppet swat done
15:44 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: etherpad-lite_1.5.7-1
15:43 logmsgbot: krenair@tin Synchronized php-1.26wmf18/extensions/ContentTranslation/modules/tools/ext.cx.tools.reference.js: https://gerrit.wikimedia.org/r/#/c/232729/ (duration: 00m 12s)
15:42 logmsgbot: krenair@tin Synchronized php-1.26wmf19/extensions/ContentTranslation/modules/tools/ext.cx.tools.reference.js: https://gerrit.wikimedia.org/r/#/c/232730/ (duration: 00m 13s)
15:39 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/206480/ (duration: 00m 13s)
15:38 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/206480/ (duration: 00m 13s)
15:32 logmsgbot: krenair@tin Synchronized php-1.26wmf18/extensions/ContentTranslation/api/ApiContentTranslationPublish.php: https://gerrit.wikimedia.org/r/#/c/232687/ (duration: 00m 13s)
15:31 logmsgbot: krenair@tin Synchronized php-1.26wmf19/extensions/ContentTranslation/api/ApiContentTranslationPublish.php: https://gerrit.wikimedia.org/r/#/c/232688/ (duration: 00m 11s)
15:27 greg-g: on mw2187: rsync: failed to set times on "/srv/mediawiki/wmf-config": Read-only file system (30)
15:25 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/231464/ (duration: 00m 13s)
15:08 urandom: restarting restbase1001 to apply temporary heap size of 12G
15:02 jynus: performing online schema change on wikidata
15:00 andrewbogott: rebooting labvirt1008
12:48 jynus: restarted nutcracker on mw1142
12:08 godog: reenable puppet on ms-fe1/ms-be1
12:04 godog: repool ms-fe1001
11:53 godog: depool ms-fe1001 to test a reboot
11:45 godog: disable puppet on ms-fe/be1 in preparation to apply https://gerrit.wikimedia.org/r/#/c/231237
12:08 kart_: Updated cxserver to e221462
03:00 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf19/cache/l10n: l10nupdate for 1.26wmf19 (duration: 06m 27s)
02:45 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf18) at 2015-08-20 02:45:14+00:00
02:38 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf18/cache/l10n: l10nupdate for 1.26wmf18 (duration: 10m 41s)

2015-08-19

23:27 logmsgbot: rmoen@tin Synchronized wmf-config/InitialiseSettings.php: Remove reference to lost wikitech apple touch icon file (duration: 00m 13s)
23:21 logmsgbot: rmoen@tin Synchronized php-1.26wmf19/extensions/TimedMediaHandler/: Re-disable 2-pass Theora encoding temporarily
23:16 logmsgbot: rmoen@tin Synchronized php-1.26wmf18/extensions/Flow: Add debugging code to detect and workaround type hint failure (duration: 00m 14s)
21:20 robh: livehack reverted sodium back to normal, testing done
21:08 robh: disabled puppet on sodium for livehacking tests for T109609
21:04 andrewbogott: disabling puppeton labnet1001 and labnet1002
20:46 urandom: restarting Cassandra on restbase1001 to enable -XX:+PrintAdaptiveSizePolicy
20:43 urandom: disabling puppet on restbase1001 to temporarily enable additional GC logging
20:17 subbu: deployed parsoid version 8d617c99
20:00 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/232580/ (duration: 00m 12s)
19:18 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/232565/ (duration: 00m 12s)
19:04 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/232543/ (duration: 00m 13s)
18:59 logmsgbot: twentyafterfour@tin Synchronized php-1.26wmf19: deploy hotfix for Wikidata: https://gerrit.wikimedia.org/r/#/c/232556/ (duration: 02m 39s)
18:30 ottomata: starting reinsatll of analytics1022 -> kafka1022 as jessie
18:06 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf19
16:26 ottomata: added analytics105[012456] into hadoop cluster as worker nodes
16:02 logmsgbot: thcipriani@tin Synchronized php-1.26wmf18/extensions/ContentTranslation/api/ApiContentTranslationPublish.php: SWAT: Temp disable notifications for cx gerrit:232504 (duration: 00m 13s)
15:57 logmsgbot: thcipriani@tin Synchronized php-1.26wmf19/extensions/ContentTranslation/api/ApiContentTranslationPublish.php: SWAT: Temporarily disable notifications for cx gerrit:232505 (duration: 00m 12s)
15:51 logmsgbot: thcipriani@tin Synchronized php-1.26wmf18/includes/changetags/ChangeTags.php: SWAT: Avoid full RC table scans in ChangeTags::updateTags() gerrit:232484 (duration: 00m 12s)
15:35 logmsgbot: thcipriani@tin Synchronized php-1.26wmf19/includes/changetags/ChangeTags.php: SWAT: Avoid full RC table scans in ChangeTags::updateTags() gerrit:232485 (duration: 00m 13s)
15:25 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Adjust mediawiki.org RSS whitelist to allow technology blog feeds gerrit:118956 (duration: 00m 13s)
15:00 andrewbogott: rebooting labvirt1007
13:58 godog: stop puppet on ms-fe1* while merging swift refactoring
13:58 godog: stop puppet on ms-be1* while merging swift refactoring
09:51 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1018 for normal traffic only (duration: 00m 13s)
09:49 logmsgbot: reedy@tin Synchronized wmf-config/InitialiseSettings.php: Add *.webarchive.org.uk to wgCopyUploadsDomains whitelist (duration: 00m 12s)
08:28 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Aug 19 08:28:19 UTC 2015 (duration 28m 18s)
07:39 jynus: About to perform a schema change on flowdb
03:44 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf19/includes/rcfeed/RCFeedFormatter.php: Fix Flow RC regression in 1.26wmf19 (duration: 00m 12s)
03:43 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf19/includes/changes/RecentChange.php: Fix Flow RC regression in 1.26wmf19 (duration: 00m 12s)
03:43 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf18/includes/rcfeed/RCFeedFormatter.php: Fix Flow RC regression in 1.26wmf18 (duration: 00m 12s)
03:42 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf18/includes/changes/RecentChange.php: Fix Flow RC regression in 1.26wmf18 (duration: 00m 12s)
03:14 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf19) at 2015-08-19 03:14:06+00:00
03:07 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf19/cache/l10n: l10nupdate for 1.26wmf19 (duration: 10m 48s)
02:39 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf18) at 2015-08-19 02:39:54+00:00
02:36 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf18/cache/l10n: l10nupdate for 1.26wmf18 (duration: 07m 51s)
01:10 yurik: updated kartotherian
00:40 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf19/extensions/Flow/: Sync Flow 1.26wmf19 for RC insert failure. (duration: 00m 15s)
00:39 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf18/extensions/Flow/: Sync Flow 1.26wmf18 for RC insert failure. (duration: 00m 14s)

2015-08-18

23:59 logmsgbot: krenair@tin Synchronized wmf-config/wikitech.php: T59040 (duration: 00m 12s)
23:37 mutante: added papaul (pt1979) to WMF LDAP group
23:08 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf18/extensions/Flow/: Sync Flow 1.26wmf18 for watchlist fix. (duration: 00m 14s)
23:02 logmsgbot: hoo@tin Synchronized wmf-config/: Set $wgPropertySuggesterClassifyingPropertyIds for testwikidata (duration: 00m 14s)
22:28 logmsgbot: krenair@tin Synchronized php-1.26wmf19/extensions/VisualEditor/modules/ve-mw/ui/dialogs/ve.ui.MWSaveDialog.js: https://gerrit.wikimedia.org/r/#/c/232385/ (duration: 00m 12s)
22:17 logmsgbot: ori@tin Synchronized php-1.26wmf19/includes/OutputPage.php: 1a4f1df2fe (duration: 00m 12s)
21:22 awight: updated paymentswiki from 823393264d6795bbaec490ff86f17580f722e598 to fca36026b1e90298abd93562803d3ea7d6893d96
19:26 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.26wmf19
19:21 logmsgbot: twentyafterfour@tin Finished scap: testwiki to 1.26wmf19 (duration: 51m 01s)
18:30 logmsgbot: twentyafterfour@tin Started scap: testwiki to 1.26wmf19
18:11 logmsgbot: ori@tin Synchronized php-1.26wmf18/includes/OutputPage.php: 6ee94ca47c: Load all CSS in the top queue (duration: 00m 13s)
18:07 robh: sodium returned to normal, mailman window over.
17:38 logmsgbot: ori@tin Synchronized php-1.26wmf18/includes: 91ae6a39df, 4cc9622214: Added wfTransactionalTimeLimit() method and applied it; Try to make POSTs as transactional as possible (duration: 00m 16s)
17:21 robh: T108099 complete, mailman restarted for a few minutes while i prepare next task.
17:17 robh: puppet disabled on sodium, no touch.
17:03 robh: mailman maint window starts now, list delivery will remain sporadic until I finish. (It'll work off and on, no messages should be lost)
15:42 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/232241/ (duration: 00m 13s)
15:06 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Remove extra transcode enablings gerrit:232228 (duration: 00m 13s)
15:04 andrewbogott: rebooting labvirt1006
08:18 _joe_: reimaging mw1152
08:14 godog: restart cassandra on restbase100[569] to pick up latest openjdk
08:04 _joe_: depooling mw1152 from the imagescalers pool
08:03 godog: restart cassandra on restbase100[348] to pick up latest openjdk
07:23 legoktm: live hacking on mw1017 for T109236
05:45 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Aug 18 05:45:47 UTC 2015 (duration 45m 46s)
02:25 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf18) at 2015-08-18 02:25:28+00:00
02:21 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf18/cache/l10n: l10nupdate for 1.26wmf18 (duration: 06m 50s)
00:02 ottomata: analytics1041 down, attempting power cycle

2015-08-17

22:19 matt_flaschen: LQT->Flow done on MediaWiki.org.
21:57 logmsgbot: mattflaschen@tin Synchronized wmf-config: LQT->Flow: Make frozen wikis no longer able to create LQT pages (duration: 00m 13s)
21:31 chasemp: remove php5-xdebug from terbium per mattflaschen
21:10 MaxSem: renamed Gadget:Invention, Travel, & Adventure --> Gadget Invention, Travel, & Adventure on enwiki using moveBatch.php to work around a permissions screwup
20:54 bd808: T109369: Restarted logstash on logstash1003; parsoid gelf events not being recorded since 2015-08-15
20:16 subbu: deployed parsoid version 4b656b72
19:19 ottomata: stopping kafka on analytics1012, preparing to reinstall with Jessie and rename to kafka1012
15:44 logmsgbot: krenair@tin Synchronized wmf-config/extension-list: https://gerrit.wikimedia.org/r/#/c/232051/ - remove WikiGrok from extension-list, extension is no longer deployed (duration: 00m 11s)
15:40 logmsgbot: krenair@tin Synchronized multiversion/MWMultiVersion.php: https://gerrit.wikimedia.org/r/#/c/231982/2 - clear up --wiki usage to mwscript (duration: 00m 12s)
15:34 logmsgbot: krenair@tin Synchronized php-1.26wmf18/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.Target.js: https://gerrit.wikimedia.org/r/#/c/232048/ (duration: 00m 11s)
15:07 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/231984/ (duration: 00m 13s)
15:05 andrewbogott: rebooting labvirt1004
15:03 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/232021/ (duration: 00m 12s)
14:30 mobrovac: restbase updated production cluster to ed17952
14:12 mobrovac: restbase deployed ed17952 on restbase1001
13:58 mobrovac: restbase deploying ed17952 on staging
11:42 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Reverting change to settings before schema change (no more lag) (duration: 00m 12s)
11:29 jynus: reloading dbproxy1003 haproxy config- it was a temporal max_connections issue; db1043 should be the canonical server again
10:09 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Solving lag issues while schema change is ongoing (duration: 00m 12s)
09:43 jynus: about to perform schema change on centralauth
09:36 godog: upgrade openjdk on restbase100[127] and restart cassandra
05:54 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Aug 17 05:54:24 UTC 2015 (duration 54m 23s)
05:11 twentyafterfour: restarted phd to pick up new configuration, (and to silence the phabricator 'setup issue' warning
04:47 twentyafterfour: changed phabricator policy for the multimeter application from 'public' to 'all users'
04:44 twentyafterfour: deployed https://gerrit.wikimedia.org/r/#/c/231983/ to iridium and restarted apache
04:17 mutante: free some disk space on iridium. apt-get clean; gzip /var/log/account/pacct.0; some apache logs .;.
02:29 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf18) at 2015-08-17 02:29:19+00:00
02:25 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf18/cache/l10n: l10nupdate for 1.26wmf18 (duration: 11m 08s)

2015-08-16

18:15 logmsgbot: krenair@tin Synchronized wmf-config/extension-list: https://gerrit.wikimedia.org/r/#/c/169612/ - remove Extension:Oversight (duration: 00m 21s)
18:14 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/169612/ - remove Extension:Oversight (duration: 00m 25s)
05:39 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Aug 16 05:39:28 UTC 2015 (duration 39m 27s)
02:29 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf18) at 2015-08-16 02:29:19+00:00
02:25 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf18/cache/l10n: l10nupdate for 1.26wmf18 (duration: 11m 52s)

2015-08-15

22:56 legoktm: removed 13 bounce_records for User:odder from bouncehandler database
16:07 _joe_: removing manually core dumps from last night's outage on all appservers in eqiad, they occpy on average 30 GB/server
16:05 ottomata: starting rolling restart of kafka brokers to apply auto leader rebalance enable = false
14:49 ottomata: stopping kafka broker on analytics1012 to again try to figure out why camus can't consume from it
12:46 bblack: restarted gitblit on antimony, because Java is Awesome
05:41 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Aug 15 05:41:57 UTC 2015 (duration 41m 56s)
04:38 andrewbogott: killing some rsync processes on labstore1002 because iowaits are through the roof
02:30 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf18) at 2015-08-15 02:29:05+00:00
02:25 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf18/cache/l10n: l10nupdate for 1.26wmf18 (duration: 06m 37s)
01:15 ottomata: starting broker on analytics1012, camus wasn't happy about that either. hrm.
00:58 ottomata: stopping kafka broker on analytics1012, it is causing consumption problems with camus, will look into why later.

2015-08-14

23:47 logmsgbot: ori@tin Synchronized php-1.26wmf18/includes/libs/ReplacementArray.php: (no message) (duration: 00m 27s)
21:54 mutante: restarted nutcracker on mw1010
21:26 ori: deployed job runner 808d1ae08d40
21:15 ejegg: updated crm from 4f40ac6de0385982d8e672b1ed30ff1a2a2a2aa1 to fc0fcc8f5af262b56392d3f4f5998f8ea08c99a8
19:29 logmsgbot: ori@tin Synchronized php-1.26wmf18/includes/resourceloader/ResourceLoader.php: f72009a543: ResourceLoader: apply minify-js filter to config scripts (duration: 00m 13s)
18:27 logmsgbot: ori@tin Synchronized php-1.26wmf18/extensions/MultimediaViewer: 9ee0437bc6: Updated mediawiki/core Project: mediawiki/extensions/MultimediaViewer 645b6c9e93fae13e09e5b493547aecc5a2e933ae (duration: 00m 12s)
18:24 ori: Repooling mw1041 now that T108601 is resolved.
17:59 yurik: deployed latest kartotherian
14:59 andrewbogott: rebooting labvirt1003
13:53 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1042 (vslow, dump) (duration: 00m 12s)
13:14 logmsgbot: reedy@tin Synchronized php-1.26wmf18/extensions/CirrusSearch/: Fix ElasticaQuery logspam (duration: 00m 13s)
13:06 logmsgbot: reedy@tin Synchronized php-1.26wmf18/extensions/GeoData: Fix ElasticaQuery logspam (duration: 00m 13s)
13:06 logmsgbot: reedy@tin Synchronized php-1.26wmf18/extensions/Flow: Fix ElasticaQuery logspam (duration: 00m 13s)
12:30 jynus: Restarting db1042 after data import
12:11 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Load-balance db1036 roles (duration: 00m 11s)
11:57 logmsgbot: reedy@tin Synchronized php-1.26wmf18/extensions/Translate: Stop calling deprecated Elastica function (duration: 00m 13s)
08:26 akosiaris: upgraded and restarted apertium on sca100{1,2}
08:11 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-apy_0.1+svn~61425-1
07:10 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Aug 14 07:10:37 UTC 2015 (duration 10m 36s)
06:34 Jamesofur: reset email/password for User:Auréola after multi factor user confirmation.
02:35 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf18) at 2015-08-14 02:35:19+00:00
02:31 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf18/cache/l10n: l10nupdate for 1.26wmf18 (duration: 06m 35s)
01:49 matt_flaschen: Resumed LQT->Flow conversion of mw:Project:Support_desk on mw1041
01:46 logmsgbot: ori@tin Synchronized php-1.26wmf18/includes/OutputPage.php: I5e6c79c70: Optimize the order of styles and scripts in <head> (duration: 00m 12s)

2015-08-13

23:45 awight: rollback paymentswiki from 2e7b449224317779d53ff84527166c0d378a0a40 to 823393264d6795bbaec490ff86f17580f722e598
23:16 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/228477/ and https://gerrit.wikimedia.org/r/#/c/231074/ (duration: 00m 12s)
23:15 awight: update paymentswiki-staging 99e3ce08117d18b15bc8138b447c4c21bd452d28 to 65b05fc11896325ae9749318b296c4396a64f649
23:15 logmsgbot: krenair@tin Synchronized w/static/images/project-logos/mrwikibooks.png: https://gerrit.wikimedia.org/r/#/c/228477/ (duration: 00m 12s)
23:11 logmsgbot: krenair@tin Synchronized php-1.26wmf18/extensions/Flow: https://gerrit.wikimedia.org/r/#/c/231450/1 (duration: 00m 14s)
23:03 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/227330/ (duration: 00m 12s)
22:35 awight: rollback payments-wiki-staging to 99e3ce08117d18b15bc8138b447c4c21bd452d28
22:29 awight: update paymentswiki from 823393264d6795bbaec490ff86f17580f722e598 to 2e7b449224317779d53ff84527166c0d378a0a40
22:23 awight: update payments-wiki-staging from 99e3ce08117d18b15bc8138b447c4c21bd452d28 to 2e7b449224317779d53ff84527166c0d378a0a40
22:14 Krenair: Running refreshLinks --dfn-only in a screen on terbium for T44180
21:45 awight: rollback payments-wiki-staging to 99e3ce08117d18b15bc8138b447c4c21bd452d28
21:45 awight: update payments-wiki-staging from 99e3ce08117d18b15bc8138b447c4c21bd452d28 to 96a369651c1130b0a8e53a6395f83c0b9329b9f8
21:43 logmsgbot: ori@tin Synchronized php-1.26wmf18/includes/OutputPage.php: Roll back: Test impact of I5e6c79c (duration: 00m 12s)
21:37 logmsgbot: ori@tin Synchronized php-1.26wmf18/includes/OutputPage.php: Test impact of I5e6c79c (duration: 00m 12s)
21:16 mutante: torrus broken - doing https://wikitech.wikimedia.org/wiki/Torrus#Deadlock_problem
21:14 mutante: service gitblit restart on antimony (maybe that should be paging :)
21:06 bd808: `sudo /etc/init.d/ganglia-monitor restart` on logstash100[1-6] fixed ganglia data loss
20:46 mutante: killed ganglia aggregator for logstash on carbon
20:40 bd808: ganglia not getting elasticsearch jvm data for logstash cluster since 2015-08-13T12:00 -- https://ganglia.wikimedia.org/latest/?c=Logstash+cluster+eqiad&&m=es_heap_used
19:56 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: all wikis to 1.26wmf18
19:38 logmsgbot: demon@tin Synchronized php-1.26wmf18/extensions/CirrusSearch/: (no message) (duration: 00m 11s)
19:35 logmsgbot: demon@tin Synchronized php-1.26wmf18/extensions/Graph: (no message) (duration: 00m 10s)
19:31 logmsgbot: demon@tin Synchronized php-1.26wmf18/extensions/Graph: (no message) (duration: 00m 11s)
18:32 logmsgbot: ori@tin Synchronized php-1.26wmf18/extensions/SyntaxHighlight_GeSHi/extension.json: If0851400: Fix-up for I2de8a400d: explicitly declare module position (duration: 00m 12s)
17:47 logmsgbot: kaldari@tin Synchronized wmf-config/InitialiseSettings.php: syncing InitialiseSettings for WikidataPageBanner (duration: 00m 12s)
17:46 logmsgbot: kaldari@tin Finished scap: (no message) (duration: 50m 05s)
16:55 logmsgbot: kaldari@tin Started scap: (no message)
16:54 logmsgbot: kaldari@tin Synchronized wmf-config/InitialiseSettings.php: syncing InitialiseSettings for WikidataPageBanner off (duration: 00m 11s)
16:51 logmsgbot: kaldari@tin Synchronized wmf-config/CommonSettings.php: syncing CommonSettings for WikidataPageBanner (duration: 00m 12s)
16:50 logmsgbot: kaldari@tin Synchronized wmf-config/InitialiseSettings.php: syncing InitialiseSettings for WikidataPageBanner (duration: 00m 12s)
16:31 logmsgbot: kaldari@tin Synchronized php-1.26wmf18/.gitmodules: (no message) (duration: 00m 11s)
16:30 logmsgbot: kaldari@tin Synchronized php-1.26wmf17/.gitmodules: (no message) (duration: 00m 13s)
16:29 logmsgbot: kaldari@tin Synchronized php-1.26wmf17/extensions/WikidataPageBanner: (no message) (duration: 00m 12s)
16:29 logmsgbot: kaldari@tin Synchronized php-1.26wmf18/extensions/WikidataPageBanner: (no message) (duration: 00m 12s)
16:24 logmsgbot: demon@tin Synchronized wmf-config/: undeploy wikigrok (duration: 00m 12s)
16:12 yurik: sync deployed tilerator
15:20 logmsgbot: thcipriani@tin Synchronized php-1.26wmf17/extensions/ContentTranslation/modules/tools/ext.cx.tools.images.js: SWAT: Images: validate image id before adapting to prevent js error gerrit:231229 (duration: 00m 11s)
15:10 logmsgbot: thcipriani@tin Synchronized php-1.26wmf18/extensions/ContentTranslation/modules/tools/ext.cx.tools.images.js: SWAT: Images: validate image id before adapting to prevent js error gerrit:231230 (duration: 00m 12s)
15:04 andrewbogott: rebooting labvirt1002
13:36 jynus: kill custom query hiting s6 master from terbium. Use of a slave is required.
13:11 andrewbogott: graceful’d apache2 on labcontrol1001
12:42 andrewbogott: restarted keystone on labcontrol1001
08:16 _joe_: removing all stale aggregator configs from netmon1001
08:15 godog: upgrade cassandra on restbase1009
08:11 godog: upgrade cassandra on restbase1006
08:07 godog: upgrade cassandra on restbase1005
08:00 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Aug 13 08:00:40 UTC 2015 (duration 0m 39s)
07:36 _joe_: killing all gmond instances on netmon1001, trying to fix ganglia-monitor-aggregator
06:48 matt_flaschen: Stopped Support desk LQT->Flow conversion for tonight
05:03 logmsgbot: ori@tin Synchronized php-1.26wmf17/includes/cache/MessageCache.php: 5f1ab59d31: MessageCache: derive the hash from the cache contents (duration: 00m 12s)
05:02 logmsgbot: ori@tin Synchronized php-1.26wmf18/includes/cache/MessageCache.php: 5f1ab59d31: MessageCache: derive the hash from the cache contents (duration: 00m 12s)
03:04 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf18) at 2015-08-13 03:04:52+00:00
03:01 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf18/cache/l10n: l10nupdate for 1.26wmf18 (duration: 06m 14s)
02:44 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf17) at 2015-08-13 02:44:49+00:00
02:38 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf17/cache/l10n: l10nupdate for 1.26wmf17 (duration: 10m 47s)
01:26 matt_flaschen: Restarted conversion of support desk from LQT->Flow using convertLqtPageOnLocalWiki.php, using hhvm on mw1041
01:16 logmsgbot: ori@tin Synchronized wmf-config/StartProfiler.php: I482b120289: Ensure all Xenon records begin with the script base name (duration: 00m 12s)
01:07 ori: Depooled mw1041 so it can be set aside for LQT->Flow conversion script (T108601)

2015-08-12

23:55 ori: fluorine is struggling due to I941660b5; I'm fixing.
23:54 logmsgbot: krenair@tin Synchronized php-1.26wmf17/extensions/WikimediaMaintenance: https://gerrit.wikimedia.org/r/#/c/231194/ (duration: 00m 12s)
23:47 logmsgbot: krenair@tin Synchronized php-1.26wmf18/extensions/WikimediaMaintenance: https://gerrit.wikimedia.org/r/#/c/231193/ (duration: 00m 12s)
23:44 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/231169/ (duration: 00m 12s)
23:16 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/231158/ (duration: 00m 11s)
23:15 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/231158/ (duration: 00m 13s)
23:13 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/230729 (duration: 00m 13s)
23:02 logmsgbot: legoktm@tin Synchronized wmf-config: Isolate wikidata.org cookies and CORS policies (duration: 00m 12s)
22:21 matt_flaschen: Killed support desk conversion again to review XDebug information.
22:17 matt_flaschen: Resumed the support desk conversion.
21:53 awight: updated paymentswiki from 325640bd70680a08ae77fd117433565634a98d88 to 99e3ce08117d18b15bc8138b447c4c21bd452d28
20:42 subbu: deployed parsoid version a271c205
20:23 milimetric: deployed the latest EventLogging master to eventlog1001
14:34 godog: upgrade cassandra on restbase1008
14:30 godog: upgrade cassandra on restbase1004
14:22 godog: upgrade cassandra on restbase1003
13:00 akosiaris: disabled puppet on maps-test200X
12:36 logmsgbot: aude@tin Synchronized arbitraryaccess.dblist: Enable arbitrary access on dewiki, frwiki, jawiki and s3 wikis (duration: 00m 12s)
12:21 logmsgbot: aude@tin Synchronized wmf-config/Wikibase-labs.php: Add Wikisource badge config (duration: 00m 13s)
12:20 logmsgbot: aude@tin Synchronized wmf-config/Wikibase-production.php: Add Wikisource badge config (duration: 00m 11s)
10:27 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: depool db2040 (duration: 00m 11s)
07:47 _joe_: restarted apertium-apy on sca1001 and sca1002, too many open files, probably leaking
06:12 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Aug 12 06:12:31 UTC 2015 (duration 12m 30s)
05:54 matt_flaschen: Killed support desk conversion. Will resume with profiling tomorrow.
04:45 bblack: starting slow "apt-get -y upgrade" on cp* (mostly, nginx -> +wmf2), will execute over ~18-24h
03:34 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/229608 (duration: 00m 12s)
03:06 ori: Installing xdebug on terbium so matt_flaschen can debug memory leak
03:04 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf18) at 2015-08-12 03:04:47+00:00
02:58 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf18/cache/l10n: l10nupdate for 1.26wmf18 (duration: 11m 13s)
02:30 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf17) at 2015-08-12 02:30:14+00:00
02:26 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf17/cache/l10n: l10nupdate for 1.26wmf17 (duration: 06m 52s)
01:22 mutante: zirconium - shut down, i'm sure, mollyguard
00:22 yurik: updated kartotherian
00:09 mutante: restarted gitblit
00:06 logmsgbot: ori@tin Synchronized php-1.26wmf17/extensions/Echo: Updated mediawiki/core Project: mediawiki/extensions/Echo 32e5bcf90c702 (duration: 00m 13s)
00:06 logmsgbot: ori@tin Synchronized php-1.26wmf18/extensions/Echo: Updated mediawiki/core Project: mediawiki/extensions/Echo 3ab0b7e0f4948 (duration: 00m 12s)
00:02 matt_flaschen: Resumed convertLqtPageOnLocalWiki.php run on MediaWiki.org's Project:Support_desk.

2015-08-11

23:19 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf18/extensions/Flow/: Sync Flow 1.26wmf18 for memory leaks (duration: 00m 14s)
23:08 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/230805/ (duration: 00m 12s)
23:05 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/230804/ (duration: 00m 12s)
21:03 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.26wmf18
21:02 twentyafterfour: deployed scap fixes for my dumb mistakes
20:10 logmsgbot: ori@tin Synchronized php-1.26wmf18/includes/objectcache/MultiWriteBagOStuff.php: 0acfe6a5bb: Fix argument handling in MultiWriteBagOStuff::get() (duration: 00m 12s)
19:28 logmsgbot: ori@tin Synchronized php-1.26wmf18/includes/resourceloader/ResourceLoader.php: I2089b21fc: ResourceLoader: make "cacheReport" option false by default (duration: 00m 13s)
19:28 logmsgbot: ori@tin Synchronized php-1.26wmf17/includes/resourceloader/ResourceLoader.php: I2089b21fc: ResourceLoader: make "cacheReport" option false by default (duration: 00m 11s)
19:26 logmsgbot: catrope@tin Synchronized php-1.26wmf18/extensions/Flow/modules/editor/editors/visualeditor/mw.flow.ve.Target.js: Fix missing editor switcher (duration: 00m 12s)
18:58 logmsgbot: twentyafterfour@tin Finished scap: again: sync new branch 1.26wmf18 and update testwiki (duration: 04m 58s)
18:53 logmsgbot: twentyafterfour@tin Started scap: again: sync new branch 1.26wmf18 and update testwiki
18:44 logmsgbot: twentyafterfour@tin scap failed: OSError [Errno 1] Operation not permitted: '/srv/mediawiki-staging/wikiversions.php' (duration: 29m 27s)
18:37 mutante: grafana switched to node krypton (jessie/VM)
18:21 bd808: logstash log event volume back to normal levels following elasticsearch upgrade
18:15 logmsgbot: twentyafterfour@tin Started scap: sync new branch 1.26wmf18 and update testwiki
18:06 bd808: logstash cluster recovered after upgrade of elasticsearch on logstash1006
18:03 bd808: upgraded elasticsearch to 1.7.1 on logstash1006; logstash-2015.08.11 shard recovering
18:02 bd808: upgrading elasticsearch on logstash1006
18:01 bd808: logstash cluster recovered after upgrade of elasticsearch on logstash1005
17:43 bd808: log event volume in logstash dropped dramatically again; seems to correlate with final recovery of logstash-2015.08.11 shard
17:29 bd808: upgraded elasticsearch to 1.7.1 on logstash1005; logstash-2015.08.11 shard recovering
17:28 bd808: upgrading elasticsearch on logstash1005
17:27 bd808: logstash event volume recovered after restarting all 3 logstash services
17:14 bd808: log event volume in logstash dropped dramatically at 16:49; investigating
17:13 bd808: logstash cluster recovered after upgrade of elasticsearch on logstash1004
16:42 bd808: upgraded elasticsearch to 1.7.1 on logstash1004; logstash-2015.08.11 shard recovering
16:42 mutante: restarted Apache on Etherpad
16:38 bd808: upgraded elasticsearch to 1.7.1 on logstash1003
16:37 bd808: upgraded elaasticsearch to 1.7.1 on logstash1002
16:36 bd808: upgraded elaasticsearch to 1.7.1 on logstash1001
16:23 bd808: logstash upgrade on logstash1003 complete
16:20 bd808: logstash upgrade on logstash1002 complete
16:16 bd808: logstash upgrade on logstash1001 complete
15:50 jynus: nuking db1002-db1007 on icinga
15:49 bd808: upgrading logstash on logstash1001
15:47 bd808: Trebuchet deploy of logstash/plugins: Add logstash-filter-prune 0.1.5 (36144b2)
15:36 bd808: Disabled puppet on logstash100[1-3] in preparation for upgrade to 1.5.3
15:31 logmsgbot: bd808@tin Synchronized wmf-config/InitialiseSettings.php: logging: Only send info and higher to logstash by default (4388a84) 2/2 (actually rebased this time) (duration: 00m 11s)
15:30 logmsgbot: bd808@tin Synchronized wmf-config/logging.php: logging: Only send info and higher to logstash by default (4388a84) 1/2 (actually rebased this time) (duration: 00m 11s)
15:17 logmsgbot: bd808@tin Synchronized wmf-config/InitialiseSettings.php: Touched wmf-config/InitialiseSettings.php (duration: 00m 13s)
15:12 logmsgbot: bd808@tin Synchronized wmf-config/InitialiseSettings.php: logging: Only send info and higher to logstash by default (4388a84) 2/2 (duration: 00m 12s)
15:11 logmsgbot: bd808@tin Synchronized wmf-config/logging.php: logging: Only send info and higher to logstash by default (4388a84) 1/2 (duration: 00m 12s)
10:45 jynus: general maintenance on db1042 (restart, upgrade, db reconstruction)
10:38 godog: upgrade cassandra on restbase1007
10:31 godog: upgrade cassandra on restbase1002
10:25 godog: upgrade cassandra on restbase1001
09:56 paravoid: switched routing-system autonomous-system to eqiad's subAS on cr1-eqiad/cr2--eqiad
09:09 godog: reboot ms-be2009, cpu soft lockup
05:27 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Aug 11 05:27:33 UTC 2015 (duration 27m 32s)
02:27 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf17) at 2015-08-11 02:26:58+00:00
02:23 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf17/cache/l10n: l10nupdate for 1.26wmf17 (duration: 06m 48s)
01:20 logmsgbot: ori@tin Synchronized php-1.26wmf17/includes/resourceloader/ResourceLoader.php: I2089b21fc: Revert resourceloader: Add must-revalidate to Cache-Control (duration: 00m 12s)
00:10 mutante: apache restart on krypton

2015-08-10

23:53 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-common.php: Limit the number of states in a cirrussearch query (duration: 00m 11s)
23:44 logmsgbot: krenair@tin Synchronized php-1.26wmf17/extensions/TimedMediaHandler/TimedMediaIframeOutput.php: https://gerrit.wikimedia.org/r/#/c/230656/ (duration: 00m 12s)
23:40 logmsgbot: ori@tin Synchronized multiversion/MWMultiVersion.php: I511999: Convert multiversion scripts to use wikiversions.php (duration: 00m 12s)
23:11 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 12s)
23:07 logmsgbot: krenair@tin Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/228040/ (duration: 00m 14s)
23:04 ori: deployed scap a404a39b32... Build wikiversions.php in addition to wikiversions.cdb
22:22 mutante: restart gitblit
21:53 mutante: krypton unloaded mod proxy_balancer
21:30 awight: updated paymentswiki from af16d371f9c46d4f0b78986080f2a2be3226ace8 to 325640bd70680a08ae77fd117433565634a98d88
20:57 logmsgbot: ori Synchronized php-1.26wmf17/includes/cache/MessageCache.php: I2089b21fc: MessageCache: use APC for local caching, rather than files (duration: 00m 12s)
20:12 subbu: deployed parsoid version 7b554ce2f
19:57 yurik: synced new kartotherian
19:22 logmsgbot: ori Synchronized php-1.26wmf17/includes: I9a1aa76de: Moved ObjectCacheSessionHandler renewal logic to wfSetupSession() (duration: 00m 16s)
17:12 akosiaris: stopped postgres on maps-test200{2,3,4}
16:59 logmsgbot: ori Synchronized php-1.26wmf17/includes/OutputPage.php: I2089b21fc: Load mediawiki.legacy.commonPrint styles with a media type property (2/2) (duration: 00m 11s)
16:58 logmsgbot: ori Synchronized php-1.26wmf17/resources/Resources.php: I2089b21fc: Load mediawiki.legacy.commonPrint styles with a media type property (1/2) (duration: 00m 11s)
16:50 logmsgbot: ori Synchronized php-1.26wmf17/extensions/wikihiero: I2089b21fc: Updated mediawiki/core Project: mediawiki/extensions/wikihiero (duration: 00m 12s)
15:11 logmsgbot: thcipriani Synchronized php-1.26wmf17/extensions/ContentTranslation/api/ApiContentTranslationPublish.php: SWAT: Enable scrubWikitext=1 in HTML to wikitext conversion using parsoid gerrit:230381 (duration: 00m 13s)
14:47 akosiaris: running an rsync from nas1001-a to local disks on helium
14:42 ottomata: restarted all varnishkafka instances to pick up proper confs (puppet should have done this!)
14:29 ottomata: starting upgrade of existing kafka cluster to 0.8.2.1 jessie - https://etherpad.wikimedia.org/p/kafka_0.8.2.1_migration2
12:38 bblack: deployed nginx-1.9.3-1+wmf2 to cp1065, cp1070, cp1071 (1x each text, upload, misc) for validation
11:16 logmsgbot: hoo Synchronized wmf-config/: Revert "Set dispatchBatchChunkFactor to 10 for now" (duration: 00m 12s)
11:16 logmsgbot: hoo Synchronized php-1.26wmf17/extensions/Wikidata/: Revert "Set dispatchBatchChunkFactor to 10 for now" (duration: 00m 20s)
09:35 paravoid: manually firewalled backup4001 TCP on neon to temporarily stop the nsca alert storm
09:12 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool db1042 for maintenance (duration: 00m 12s)
08:43 _joe_: manually running logrotate on iridium
07:44 godog: reboot ms-be2006, xfs hosed
07:27 akosiaris: rebooting backup4001
07:20 jynus: schema change on testwikidatawiki
06:52 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Increase db1035 weight (duration: 00m 13s)
05:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Aug 10 05:31:29 UTC 2015 (duration 31m 28s)
03:13 bblack: restarted apache2 on iridium JIC
03:13 bblack: rm /var/log/apache2/phabricator_access.log.1 on iridium (disk full, fixed for now)
02:23 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf17) at 2015-08-10 02:23:47+00:00
02:20 logmsgbot: l10nupdate Synchronized php-1.26wmf17/cache/l10n: l10nupdate for 1.26wmf17 (duration: 06m 43s)

2015-08-09

17:57 urandom: issuing nodetool cleanup on restbase1006
12:56 logmsgbot: hoo Synchronized php-1.26wmf17/extensions/Wikidata/: Set dispatchBatchChunkFactor to 10 for now (duration: 00m 20s)
06:20 twentyafterfour: restarted phabricator phd (just in case - the full partition may have caused the daemons to be in a broken state)
06:17 twentyafterfour: moved some log files on iridium into /srv/logs to free space on /
05:12 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Aug 9 05:12:23 UTC 2015 (duration 12m 22s)
02:23 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf17) at 2015-08-09 02:23:22+00:00
02:20 logmsgbot: l10nupdate Synchronized php-1.26wmf17/cache/l10n: l10nupdate for 1.26wmf17 (duration: 06m 32s)

2015-08-08

14:58 urandom: issuing nodetool cleanup on restbase1005
14:57 urandom: issuing nodetool cleanup on restbase1007
05:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Aug 8 05:30:22 UTC 2015 (duration 30m 21s)
04:20 urandom: issuing nodetool cleanup on restbase1008
03:00 logmsgbot: hoo Synchronized php-1.26wmf17/extensions/Wikidata/: Hack: Don't write change rows where LENGTH(change_info) > 65500 (duration: 00m 21s)
02:26 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf17) at 2015-08-08 02:26:20+00:00
02:23 logmsgbot: l10nupdate Synchronized php-1.26wmf17/cache/l10n: l10nupdate for 1.26wmf17 (duration: 06m 20s)
01:53 hoo: Deleted change 237365841 as well
01:37 hoo: Deleted changes 237357747 and 237363245 from wikidata's wb_changes
01:32 logmsgbot: ori Synchronized php-1.26wmf17/resources/src/mediawiki.legacy/wikibits.js: I664ba9b0af: Override document.writeln to prevent it from blanking pages (duration: 00m 13s)

2015-08-07

21:00 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/230227/ - should be a noop for prod (duration: 00m 12s)
19:40 logmsgbot: hoo Synchronized wmf-config/: Bump wgCacheEpoch for Wikidata (duration: 00m 13s)
19:29 logmsgbot: hoo Synchronized php-1.26wmf17/extensions/Wikidata/: Update Wikibase: Fix WB spinner, UnresolvedRedirectException handling on client (duration: 00m 21s)
18:08 gwicke: switched restbase1001 to CMS temporarily, to gather metrics; will switch back to G1GC tonight
17:09 yurik: synced kartotherian to maps-test* servers again, restarted the service
16:00 jynus: repool db1035 database with low traffic
15:54 logmsgbot: thcipriani Synchronized php-1.26wmf17/extensions/ContentTranslation/modules/tools/ext.cx.tools.images.js: FIX: Use .attr() to set the resource attribute of image, while adapting gerrit:230101 (duration: 00m 11s)
15:10 yurik: synced kartotherian to maps-test* servers
15:10 gwicke: switched cassandra staging cluster back to G1GC / default puppet
13:40 moritzm: install pcre security updates on elastic*, analytics*, wtp*, db* and es*
13:40 urandom: starting nodetool cleanup on restbase1004 (see: T108083)
13:37 urandom: starting nodetool cleanup on restbase1002 (see: T108083)
12:07 moritzm: restarted HHVM on jobrunners/imagescalers in eqiad/codfw for libtidy/PCRE security updates
09:24 logmsgbot: akosiaris Synchronized wmf-config/PoolCounterSettings-eqiad.php: (no message) (duration: 00m 12s)
09:02 akosiaris: disabled helium as a poolcounter temporarily while applying base::firewall again
09:01 logmsgbot: akosiaris Synchronized wmf-config/PoolCounterSettings-eqiad.php: (no message) (duration: 00m 12s)
08:52 moritzm: restarted HHVM on API apaches in codfw for libtidy/PCRE security updates
08:26 godog: restart cassandra on test cluster
08:11 godog: upgrade cassandra test cluster to openjdk 8u66-b01-1~bpo8+1
07:34 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Aug 7 07:34:34 UTC 2015 (duration 34m 33s)
02:43 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf17) at 2015-08-07 02:43:35+00:00
02:40 logmsgbot: l10nupdate Synchronized php-1.26wmf17/cache/l10n: l10nupdate for 1.26wmf17 (duration: 06m 15s)
02:22 bblack: pooled cp105[67],cp1069,cp1070 into eqiad misc caches
02:19 logmsgbot: krinkle Finished scap: Rebuild l10n for Gadgets after I2089b21fc (duration: 24m 28s)
01:54 logmsgbot: krinkle Started scap: Rebuild l10n for Gadgets after I2089b21fc
01:08 logmsgbot: ori Synchronized php-1.26wmf17/extensions/Gadgets: I2089b21fc: Updated mediawiki/core Project: mediawiki/extensions/Gadgets (duration: 00m 13s)
00:59 logmsgbot: ori Synchronized php-1.26wmf17/includes/OutputPage.php: c0ca5700c6: resourceloader: Restore anticipated loader states for hardcoded module requests (duration: 00m 12s)

2015-08-06

23:24 logmsgbot: ori Synchronized php-1.26wmf17: c5c52ec1d8: resourceloader: Async all the way (duration: 01m 41s)
23:20 logmsgbot: ebernhardson Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 12s)
23:16 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: I053a6e9: Enable authmetrics logging on group0 wikis (duration: 00m 12s)
23:15 logmsgbot: ebernhardson Synchronized wmf-config/: Redeploy cirrussearch ab test start (duration: 00m 14s)
23:09 logmsgbot: ebernhardson Synchronized wmf-config/: Start cirrussearch suggester confidence AB test (duration: 00m 13s)
23:07 mutante: puppet/salt-master: signing certs and adding keys for fermium
23:05 logmsgbot: ebernhardson Synchronized php-1.26wmf17/extensions/CirrusSearch: Bump cirrusearch in 1.26wmf17 for SWAT (duration: 00m 11s)
22:44 logmsgbot: ori Synchronized php-1.26wmf17/tests/phpunit/includes/OutputPageTest.php: (no message) (duration: 00m 13s)
22:32 mutante: starting new instance fermium on ganeti
22:31 ori: Previous two syncs were of I2089b21fc and I3f46fee7c
22:31 logmsgbot: ori Synchronized php-1.26wmf17/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: (no message) (duration: 00m 12s)
22:23 logmsgbot: ori Synchronized php-1.26wmf17/resources/src/startup.js: (no message) (duration: 00m 12s)
22:22 logmsgbot: ori Synchronized php-1.26wmf17/includes/resourceloader/ResourceLoader.php: (no message) (duration: 00m 11s)
22:21 logmsgbot: ori Synchronized php-1.26wmf17/extensions/Flow: 94703bc291: Updated mediawiki/core Project: mediawiki/extensions/Flow (duration: 00m 15s)
22:02 mutante: if up for watching the (auto)-upgrade and restarting: @carbon:/srv/wikimedia/incoming# reprepro -C main include adminbot_1.7.12_amd64.changes
22:00 mutante: built adminbot 1.7.12 and copied to carbon to incoming - but not imported
21:56 logmsgbot: ori Synchronized php-1.26wmf17/extensions/Graph: I2089b21fc: Updated mediawiki/core Project: mediawiki/extensions/Graph (duration: 00m 12s)
21:55 hoo: Running "updateSpecialPages.php --wiki wikidatawiki --only DoubleRedirects" on terbium
21:46 ejegg: updated payments from 5bc32b7d0969878e441394c828620d5a44683c18 to af16d371f9c46d4f0b78986080f2a2be3226ace8
21:25 logmsgbot: krinkle Synchronized php-1.26wmf17/extensions/EducationProgram/EducationProgram.hooks.php: T107980 (duration: 00m 12s)
21:12 gwicke: switched cassandra staging cluster (xenon, cerium, praseodymium) to CMS & started a load test on that
21:07 logmsgbot: ebernhardson Synchronized php-1.26wmf17/extensions/CirrusSearch/includes/Hooks.php: Repush file spewing notices into hhvm.log (duration: 00m 12s)
20:26 chasemp: es-tool restart-fast on elastic1031 to test alerting issues
20:06 logmsgbot: ori Synchronized php-1.26wmf17/extensions/Flow: I2089b21fc: Updated mediawiki/core Project: mediawiki/extensions/Flow (duration: 00m 15s)
19:42 logmsgbot: twentyafterfour Synchronized php-1.26wmf17: actually deploy the hotfix this time (duration: 01m 33s)
19:38 urandom: issuing nodetool cleanup on restbase1003
19:36 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: all wikis to 1.26wmf17
19:35 logmsgbot: twentyafterfour Synchronized php-1.26wmf17: sync hotfixes before deploying 1.26wmf17 to group2 (duration: 02m 18s)
18:57 ejegg: updated payments from bbec5799db42f6f5302920a1a69123de7e4986df to 5bc32b7d0969878e441394c828620d5a44683c18
18:55 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: revert 1.26wmf17
18:47 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: all wikis to 1.26wmf17
18:11 ejegg|mtg: updated payments from a8c0ecbedef6179c78ed833da9f2049cb0f2641b to bbec5799db42f6f5302920a1a69123de7e4986df
16:59 logmsgbot: krinkle Synchronized php-1.26wmf17/resources/src/startup.js: touch (duration: 00m 11s)
16:56 logmsgbot: krinkle Synchronized php-1.26wmf17/resources/Resources.php: T108191 unbreak mobile js (duration: 00m 11s)
16:24 chasemp: upgrading elastic1031 to 1.7.1
15:08 logmsgbot: ebernhardson Synchronized php-1.26wmf17/extensions/CirrusSearch/: bump cirrussearch in 1.26wmf17 for swat (duration: 00m 14s)
14:50 dcausse: es1.7.1: restart elastic1030
14:19 urandom: beginning nodetool cleanup on restbase1001
14:14 moritzm: restarted HHVM on appservers in codfw for libtidy/PCRE security updates
14:06 dcausse: es1.7.1: restart elastic1029
13:27 dcausse: es1.7.1: restart elastic1028
12:59 dcausse: es1.7.1: restart elastic1027
12:55 godog: stop syslog-ng on lithium before switching to rsyslog
11:55 dcausse: es1.7.1: restart elastic1026
10:33 dcausse: es1.7.1: restart elastic1025
09:50 moritzm: uploaded openjdk-8_8u66-b01-1~bpo8+1 to jessie-wikimedia and jessie-backports/debian.org
09:39 jynus: Applying schema change to Commons db master
09:39 moritzm: restarted HHVM on API apaches in eqiad for libtiny/PCRE security updates
09:30 dcausse: es1.7.1: restart elastic1024
09:18 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Repool db1068 (duration: 00m 13s)
08:20 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Aug 6 08:20:24 UTC 2015 (duration 20m 23s)
07:34 dcausse: es1.7.1: restart elastic1023
07:07 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Repool db1056, Depool db1068 (duration: 00m 12s)
06:52 moritzm: restart HHVM on canary API servers (mw1114-mw1119) for libtiny/PCRE security updates
06:16 dcausse: es1.7.1: restart elastic1022
05:57 logmsgbot: krinkle Synchronized php-1.26wmf16/includes/resourceloader/ResourceLoaderModule.php: Ib4371255fe (duration: 00m 12s)
05:55 logmsgbot: krinkle Synchronized php-1.26wmf17/includes/resourceloader/ResourceLoaderModule.php: Ib4371255fe (duration: 00m 13s)
05:39 logmsgbot: krinkle Synchronized php-1.26wmf17/resources/src/mediawiki/mediawiki.js: touch (duration: 00m 12s)
05:32 logmsgbot: krinkle Synchronized php-1.26wmf17/resources/src/mediawiki/mediawiki.js: touch (duration: 00m 13s)
05:02 logmsgbot: krinkle Synchronized php-1.26wmf17/includes/OutputPage.php: I885c36398 (duration: 00m 12s)
04:48 ebernhardson: es1.7.1 upgrade on elastic1021
04:42 logmsgbot: krinkle Synchronized php-1.26wmf17/resources/src/mediawiki/mediawiki.js: I885c36398 (duration: 00m 12s)
04:04 logmsgbot: krinkle Synchronized php-1.26wmf17/resources/src/mediawiki.legacy/wikibits.js: T108139 (duration: 00m 12s)
03:58 logmsgbot: krinkle Synchronized php-1.26wmf17/includes: T108124 (duration: 00m 17s)
03:57 logmsgbot: krinkle Synchronized php-1.26wmf17/resources/src/mediawiki/mediawiki.js: T108124 (duration: 00m 12s)
03:25 ebernhardson: es1.7.1 upgrade on elastic1020
03:18 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf17) at 2015-08-06 03:18:27+00:00
03:12 logmsgbot: l10nupdate Synchronized php-1.26wmf17/cache/l10n: l10nupdate for 1.26wmf17 (duration: 10m 32s)
02:44 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf16) at 2015-08-06 02:44:39+00:00
02:38 logmsgbot: l10nupdate Synchronized php-1.26wmf16/cache/l10n: l10nupdate for 1.26wmf16 (duration: 10m 42s)
02:16 ebernhardson: es1.7.1 upgrade on elastic1019
01:34 ebernhardson: es1.7.1 upgrade on elastic1018
00:49 Jamesofur: reset password for User:Tonval after identify verification
00:42 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 12s)
00:34 twentyafterfour: phabricator upgrade complete
00:33 ebernhardson: es1.7.1 upgrade on elastic1017
00:31 RoanKattouw: <twentyafterfour> ok I'm gonna take phabricator down for upgrade
00:04 gwicke: restarted restbase old-render clean-up scripts on wikipedia html and data-parsoid

2015-08-05

23:56 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Unset $wgDiff (duration: 00m 12s)
23:37 logmsgbot: ori Synchronized php-1.26wmf17/extensions/FlaggedRevs: I2089b21fc (duration: 00m 13s)
23:32 logmsgbot: bd808 Synchronized php-1.26wmf17/extensions/VisualEditor/extension.json: VisualEditor b/c anon IP module name fix (Ia92ecc0) (duration: 00m 12s)
23:09 logmsgbot: bd808 Synchronized wmf-config/CommonSettings.php: beta: Configure and (I7d20abb) (duration: 00m 13s)
23:01 logmsgbot: ori Synchronized php-1.26wmf17/extensions/EducationProgram: I2089b21fc (duration: 00m 13s)
23:00 ebernhardson: es1.7.1 upgrade on elastic1016
22:47 logmsgbot: krinkle Synchronized php-1.26wmf17/includes/resourceloader/ResourceLoaderModule.php: T104950 (duration: 00m 12s)
22:47 logmsgbot: krinkle Synchronized php-1.26wmf16/includes/resourceloader/ResourceLoaderModule.php: T104950 (duration: 00m 13s)
22:29 hoo: Started dumpwikidatajson.sh on snapshot1003 again to create a Wikidata json dump after earlier attempts this week and today failed.
22:27 logmsgbot: hoo Synchronized php-1.26wmf17/extensions/Wikidata/: Update Wikibase: Fix use class in CallbackFactory (duration: 00m 21s)
22:27 logmsgbot: hoo Synchronized php-1.26wmf16/extensions/Wikidata/: Update Wikibase: Fix use class in CallbackFactory (duration: 00m 20s)
22:27 ebernhardson: es1.7.1 upgrade on elastic1015
21:44 subbu: deployed cherry-picked ba49b80bdc3a156604eb3996830af0d5bc45c503 hotfix to the parsoid cluster to deal with crashers from deploy earlier today
21:17 gwicke: finished deploy of restbase 9e177f3 (deploy 7006f9f) on restbase cluster
21:12 hoo: Started dumpwikidatajson.sh on snapshoot1003 to create a Wikidata json dump after earlier attempts this week failed.
21:05 ebernhardson: es1.7.1 upgrade for es1014
20:59 gwicke: restbase 9e177f3 (deploy 7006f9f) canary deploy on restbase1001
20:56 logmsgbot: hoo Synchronized php-1.26wmf17/extensions/Wikidata/: Update Wikibase: Fix the dumpJson and the rebuildItemsPerSite maintenance scripts (duration: 00m 20s)
20:55 logmsgbot: hoo Synchronized php-1.26wmf16/extensions/Wikidata/: Update Wikibase: Fix the dumpJson and the rebuildItemsPerSite maintenance scripts (duration: 00m 20s)
20:25 subbu: deployed parsoid version d5a5722c
20:22 logmsgbot: krinkle Synchronized php-1.26wmf16/includes/resourceloader/ResourceLoaderFileModule.php: T104950 (duration: 00m 12s)
20:21 logmsgbot: krinkle Synchronized php-1.26wmf16/includes/resourceloader/ResourceLoader.php: T104950 (duration: 00m 11s)
20:13 logmsgbot: krinkle Synchronized php-1.26wmf17/includes/resourceloader/ResourceLoaderFileModule.php: T104950 (duration: 00m 12s)
20:12 logmsgbot: krinkle Synchronized php-1.26wmf17/includes/resourceloader/ResourceLoader.php: T104950 (duration: 00m 13s)
20:07 logmsgbot: ori Synchronized php-1.26wmf17/extensions/PageTriage: I2089b21fc: Updated mediawiki/core Project: mediawiki/extensions/PageTriage 22eddf4ad5bf6b3fe7c49af5812ce5fcfa5e1911 (duration: 00m 14s)
19:55 gwicke: re-enabled puppet on restbase staging cluster in preparation for deploy
19:52 gwicke: disabled puppet on restbase hosts in preparation for the deploy
19:36 dcausse: es1.7.1: resume writes to indices
19:31 dcausse: es1.7.1: restart elastic1013
19:19 bblack: all caches depooled for thermal stuff repooled
18:54 bblack: depooled cp1060, cp1064 ( thermal batch 3: https://phabricator.wikimedia.org/T103226 )
18:37 dcausse: es1.7.1: restart elastic1012
18:34 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf17
18:07 bblack: depooled cp1059, cp1062, cp1067 ( thermal batch 2: https://phabricator.wikimedia.org/T103226 )
18:02 moritzm: restarted HHVM on appservers (mw1136-mw1158) for tidy/pcre security updates
17:56 dcausse: es1.7.1: restart elastic1011
17:48 dcausse: es1.7.1: freeze indices (take 2)
17:36 logmsgbot: bblack Synchronized wmf-config/squid-labs.php: (no message) (duration: 00m 12s)
17:15 moritzm: restarted HHVM on appservers (mw1149-mw1151, mw1161-1188, mw1209-1220) for tidy/pcre security updates
17:09 logmsgbot: hoo Finished scap: Rebuild l10n cache for wmf17, got forgotten during the train (duration: 26m 02s)
17:07 bblack: really depooled cp1046, cp1061, cp1066 ( thermal batch 1: https://phabricator.wikimedia.org/T103226 )
17:02 bblack: depooled cp1046, cp1061, cp1066 ( thermal batch 1: https://phabricator.wikimedia.org/T103226 )
16:43 logmsgbot: hoo Started scap: Rebuild l10n cache for wmf17, got forgotten during the train
16:28 bblack: cache puppets disabled for a little while, to make sure do_esi doesn't melt things
15:11 logmsgbot: thcipriani Synchronized php-1.26wmf17/extensions/ContentTranslation/modules/tools/ext.cx.tools.mt.js: SWAT: FIX: Not able to set cursor in previous sections gerrit:229328 (duration: 00m 12s)
15:02 andrewbogott: rebooting labvirt1009
14:51 gwicke: stopped restbase on restbase1009
14:44 moritzm: restarted HHVM on appservers (mw1026-mw1113) for tidy/pcre security updates
14:42 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool db1056 (duration: 00m 12s)
14:29 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool db1059 (duration: 00m 13s)
13:16 hoo: Removed Wikidata JSON dumps from Monday and Tuesday as they were incomplete/ had the wrong serialization format
12:41 moritzm: restarted HHVM on canary appservers for tidy/pcre security updates, remaining app servers following soon
12:32 paravoid: upgrading asw-c-codfw and asw-d-codfw to newer junos
11:17 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool db1056, depool db1059 (duration: 00m 12s)
11:01 godog: depool restbase1009, investigating healthcheck returning 500s
10:52 godog: pool restbase100[789] in pybal
10:43 paravoid: upgrading asw-b-codfw to newer junos
10:36 jynus: applying schema change for s4 on codfw, some lag expected
09:08 dcausse: es1.7.1: upgrade elastic1010
07:46 dcausse: es1.7.1: upgrade elastic1009
07:12 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool db1056 for maintenance, db1064 set to 100% (duration: 00m 12s)
06:29 springle: finish OSC gerrit 228756 s5 wb_items_per_site.ips_site_page
06:27 logmsgbot: @tin ResourceLoader cache refresh completed at Wed Aug 5 06:27:08 UTC 2015 (duration 27m 7s)
06:26 dcausse: es1.7.1: upgrade elastic1008
04:56 ebernhardson: restarted elasticsearch on elastic1007 for 1.7.1 upgrade
03:34 logmsgbot: legoktm Synchronized wmf-config/InitialiseSettings.php: Disable two more wikis due to namespace conflicts - https://gerrit.wikimedia.org/r/229292 (duration: 00m 12s)
03:09 ebernhardson: restarted elasticsearch on elastic1006 for 1.7.1 upgrade
03:04 logmsgbot: @tin LocalisationUpdate completed (1.26wmf17) at 2015-08-05 03:04:08+00:00
02:57 logmsgbot: l10nupdate Synchronized php-1.26wmf17/cache/l10n: (no message) (duration: 10m 30s)
02:31 logmsgbot: @tin LocalisationUpdate completed (1.26wmf16) at 2015-08-05 02:31:44+00:00
02:28 logmsgbot: l10nupdate Synchronized php-1.26wmf16/cache/l10n: (no message) (duration: 06m 56s)
01:44 ebernhardson: restarting elasticsearch of es1005

2015-08-04

23:59 logmsgbot: maxsem Synchronized php-1.26wmf16/extensions/WikimediaEvents/: SWAT (duration: 00m 12s)
23:57 logmsgbot: maxsem Synchronized php-1.26wmf17/extensions/WikimediaEvents/: SWAT (duration: 00m 12s)
23:08 logmsgbot: mattflaschen Synchronized wmf-config/InitialiseSettings.php: Disable Flow on betawikiversity (duration: 00m 13s)
22:07 logmsgbot: twentyafterfour Synchronized php-1.26wmf17: forgot submodule update (duration: 01m 39s)
20:46 logmsgbot: twentyafterfour Finished scap: fixup wikidata submodule version (duration: 23m 26s)
20:22 logmsgbot: twentyafterfour Started scap: fixup wikidata submodule version
19:46 dcausse: es1.7.1: upgrade elastic1003
19:12 ori: Applied Icba6d7a87 on mw1017 for a couple of webpagetest runs
19:08 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.26wmf17
18:51 logmsgbot: twentyafterfour Finished scap: rebuild localization cache, sync 1.26wmf17 (duration: 28m 39s)
18:42 dcausse: es1.7.1: upgrade elastic1002
18:22 logmsgbot: twentyafterfour Started scap: rebuild localization cache, sync 1.26wmf17
18:00 andrewbogott: re-imaging labnodepool1001
17:35 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Increase db1064 traffic (duration: 00m 13s)
17:18 dcausse: es1.7.1: upgrade elastic1001
17:17 hoo: Started dumpwikidatajson.sh on snapshot1003 to create a correct Wikidata json dump
17:14 logmsgbot: hoo Synchronized php-1.26wmf16/extensions/Wikidata/: Fix maintenance/dumpJson.php fatal (duration: 00m 21s)
17:11 chasemp: freezing elasticsearch indexes for 1.7.1
16:23 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Repool db1064 with low traffic after maintenance (duration: 00m 12s)
15:34 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable Flow on ptwikibooks gerrit:229133 (duration: 03m 40s)
15:28 jynus: restarting db1064 for regular maintenance and upgrade given that it was depooled in the first place for a schema change
15:24 logmsgbot: thcipriani Synchronized wmf-config: SWAT: Add configuration for authmetrics logging (part II) gerrit:227630 (duration: 02m 41s)
15:21 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Add configuration for authmetrics logging (part I) gerrit:227630 (duration: 03m 11s)
15:13 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor for 10% of new accounts on enwiki gerrit:227329 (duration: 03m 13s)
14:36 paravoid: cr2-codfw upgrading SCBs
14:23 paravoid: upgrading junos on asw-a-codfw again
13:45 _joe_: repooling mw1159,mw1160
13:21 paravoid: rebooting asw-a-codfw, member 2
13:04 Coren: labstore1001 rebooting (possibly a couple of times) during tests and reinstallation
12:55 hoo: Syncing to mw1160 failed (Host key verification failed.)
12:50 logmsgbot: hoo Synchronized php-1.26wmf16/extensions/Wikidata/: Update Wikibase: Fixes for JSON dump creation (duration: 00m 39s)
12:06 moritzm: updated canary appservers mw1017/mw1018 to updated pcre3 + hhvm restart
12:03 moritzm: added pcre3_8.31-2ubuntu2.1+wm1 to trusty-wikimedi (reroll of security update with our JIT enablement patch)
11:48 _joe_: killed ircecho to prevent furter icinga spam
11:44 jynus: schema update on Commons failed, expect some minor inestabilities until everything is fixed
11:41 _joe_: reimaging mw1159 to HAT
11:01 paravoid: upgrading junos on asw-a-codfw
10:57 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Depool db1064 (duration: 00m 13s)
10:27 godog: bootstrap cassandra on restbase1009
10:21 akosiaris: enabling puppet on tin
09:30 jynus: rolling schema change on image table to all wikis
08:07 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Increasing load for db1027 and db1015 (duration: 00m 12s)
07:38 logmsgbot: @tin ResourceLoader cache refresh completed at Tue Aug 4 07:38:01 UTC 2015 (duration 38m 0s)
06:14 _joe_: depooled mw1061
06:14 logmsgbot: legoktm Synchronized wmf-config/InitialiseSettings.php: Disable Flow on Japanese Wikiversity (duration: 00m 13s)
06:09 logmsgbot: legoktm Synchronized wmf-config/InitialiseSettings.php: Disable Flow on English Wikiversity (duration: 00m 12s)
06:07 legoktm: sync to mw1061 failed
06:07 logmsgbot: legoktm Synchronized wmf-config/InitialiseSettings.php: Disable Flow on English Wikiversity (duration: 00m 12s)
02:32 logmsgbot: @tin LocalisationUpdate completed (1.26wmf16) at 2015-08-04 02:32:18+00:00
02:28 logmsgbot: l10nupdate Synchronized php-1.26wmf16/cache/l10n: (no message) (duration: 09m 16s)
02:18 logmsgbot: twentyafterfour Finished scap: sync https://gerrit.wikimedia.org/r/#/c/229036/1 (duration: 25m 41s)
01:52 logmsgbot: twentyafterfour Started scap: sync https://gerrit.wikimedia.org/r/#/c/229036/1
00:02 awight: updated paymentswiki to a8c0ecbedef6179c78ed833da9f2049cb0f2641b

2015-08-03

23:56 awight: updating paymentswiki to b20559f75e0fc0d863efe027d76b78462555767c
23:45 ottomata: rebuilding kafka cluster
23:21 logmsgbot: ebernhardson Synchronized php-1.26wmf16/extensions/VisualEditor/: Bump visualeditor for swat in 1.26wmf16 (duration: 00m 13s)
23:18 logmsgbot: ebernhardson Synchronized php-1.26wmf16/extensions/WikimediaEvents/: Bump WikimediaEvents in SWAT for 1.26wmf16 (duration: 00m 12s)
23:17 logmsgbot: ebernhardson Synchronized php-1.26wmf16/extensions/Flow: Bump flow submodule in swat for 1.26wmf16 (duration: 00m 14s)
23:05 logmsgbot: ebernhardson Synchronized wmf-config/: (no message) (duration: 00m 13s)
22:46 awight: reverting paymentswiki, to 6dbbb4c784349ace5a0ac616c61ec0c3fffa0eff
22:33 ejegg: updated crm from db417a28a247a3fdf3e3023a700d6266e04f3e9d to 4f40ac6de0385982d8e672b1ed30ff1a2a2a2aa1
22:27 awight: deployed debug hack to payments1004
21:43 awight: deploy paymentswiki-staging configuration: add explicit queue name for payments4 connecting to payments1-3
21:32 awight: deploy paymentswiki-staging configuration
21:25 awight: updating payments1004 to 1daf9d0fe773c022a2ab8de5542fc15ddc261e75
21:04 logmsgbot: bd808 Synchronized wmf-config/logging.php: Remove code duplication from monolog config (Ia960203) (duration: 00m 11s)
20:51 awight: updating paymentswiki from d4bdce1cae168448b116d75e3dcd3303b0f13dd2 to d56dad49ef0da0a8b9c7da410bcac12e48724ae5
20:26 arlolra: updated Parsoid to version 38d0cdb13734a40bc2908e779e1a0cde158048f2
19:49 logmsgbot: aude Synchronized php-1.26wmf16/extensions/Wikidata: Fix T104609 and fix/debug T107711 (duration: 00m 19s)
19:21 logmsgbot: aude Synchronized usagetracking.dblist: Enable usage tracking on enwiki (duration: 00m 12s)
19:20 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Add debug log group for T107711 (duration: 00m 12s)
19:07 ottomata: stopped a couple of kafka brokers. acknowldeging..
19:02 bblack: https://gerrit.wikimedia.org/r/228882 reversion salted + nginx reloaded
18:28 gwicke: switched restbase1002 and restbase1003 to iojs as well
17:36 logmsgbot: aude Synchronized usagetracking.dblist: Enable usage tracking on zhwiki (duration: 00m 12s)
17:21 logmsgbot: legoktm Synchronized php-1.26wmf16/includes/Revision.php: https://gerrit.wikimedia.org/r/228853 (duration: 00m 12s)
17:21 ottomata: starting kafka partition reassignment to balance all partiions over to 3 new kafka brokers and off of analytics1021
17:21 gwicke: switching from node 0.10 to iojs 2.5 on restbase1001 after load testing on xenon went well
17:02 logmsgbot: legoktm Synchronized wmf-config/logging.php: logging: Enable stacktrace printing (duration: 00m 12s)
17:00 hoo: Started dumpwikidatajson.sh on snapshot1003 to re-create today's dump
16:55 logmsgbot: legoktm Synchronized php-1.26wmf16/autoload.php: https://gerrit.wikimedia.org/r/#/c/228850/ (duration: 00m 12s)
16:54 logmsgbot: legoktm Synchronized php-1.26wmf16/includes/debug/logger/: https://gerrit.wikimedia.org/r/#/c/228850/ (duration: 00m 11s)
16:49 hoo: Removed today's Wikidata json dump (wikidata-20150803-all.json.gz) because it was incomplete due to the dataset problems earlier
16:27 paravoid: upgrading junos on cr2-codfw
15:34 bblack: wiping cp3034 disk cache (upload esams) for ipsec reload testing
15:23 logmsgbot: thcipriani Synchronized php-1.26wmf16/extensions/MultimediaViewer: SWAT: Track image load time with statsv (touch and re-sync) gerrit:228218 (duration: 00m 12s)
15:22 ottomata: reinstalling analytics1013,1014 and 1020 with Jessie
15:10 logmsgbot: thcipriani Synchronized php-1.26wmf16/extensions/MultimediaViewer: SWAT: Track image load time with statsv gerrit:228218 (duration: 00m 12s)
14:59 logmsgbot: aude Synchronized usagetracking.dblist: Enable usage tracking on trwiki (duration: 00m 12s)
14:54 logmsgbot: krenair Synchronized php-1.26wmf16/extensions/SemanticResultFormats: https://gerrit.wikimedia.org/r/#/c/228793/ (duration: 00m 13s)
14:42 logmsgbot: aude Synchronized usagetracking.dblist: Enable usage tracking on thwiki (duration: 00m 12s)
14:33 mutante: temp. stop puppet on dataset1001
14:27 paravoid: upgrading junos on cr1-codfw
14:23 moritzm: updated iojs on apt.wikimedia.org to 2.5.0 for jessie-wikimedia
14:21 ottomata: upgrading kernel on analytics1042-1049 from 3.13.0.24.28 to 3.13.0.61.68 because T107698
14:18 logmsgbot: aude Synchronized usagetracking.dblist: Enable usage tracking on svwiki (duration: 00m 12s)
13:50 bblack: re-enabling puppet + ircecho on neon (vast majority of recovery spam is over with)
13:17 bblack: re-enable agent, restarted apache2 on palladium, strontium, rhodium (fact_values truncated in mysql)
13:10 bblack: rhodium too (puppetmaster stop)
13:05 bblack: stopped puppet-agent + apache2 on strontium + palladium (no masters alive, for mysql maintenance)
12:59 bblack: stopped ircecho + puppet-agent on neon (spam from epic puppetmaster fail)
12:52 bblack: stop->wait->restart of apache2 service on palladium (seemed dead to puppet reqs)
12:21 _joe_: bumped ganglia-monitor-aggregator on bast4001, the upstart script needs immediate fixing
11:01 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: avoid db1044 SPOF by repooling db1027 and db1015 (duration: 00m 12s)
10:56 paravoid: switching GeoDNS to GeoIP2
10:45 paravoid: upgrading all AuthDNS servers to gdnsd 2.2.0
09:31 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool db1035 for maintenance (duration: 00m 12s)
05:22 logmsgbot: @tin ResourceLoader cache refresh completed at Mon Aug 3 05:22:15 UTC 2015 (duration 22m 14s)
02:23 logmsgbot: @tin LocalisationUpdate completed (1.26wmf16) at 2015-08-03 02:23:21+00:00
02:20 logmsgbot: l10nupdate Synchronized php-1.26wmf16/cache/l10n: (no message) (duration: 06m 21s)
01:47 springle: starting OSC gerrit 228756 s5 wb_items_per_site.ips_site_page
00:03 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/228198/ (duration: 00m 12s)

2015-08-02

17:52 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: If7fcb6e6: Default wikipedias to enwiki.png (duration: 00m 12s)
13:26 jynus: powercycling analytics1044: same kernel fatal issues as 1043
13:10 jynus: powercycling analytics1043: kernel issues
12:05 bblack: started pybal on lvs3001
04:56 logmsgbot: @tin ResourceLoader cache refresh completed at Sun Aug 2 04:56:29 UTC 2015 (duration 56m 28s)
02:23 logmsgbot: @tin LocalisationUpdate completed (1.26wmf16) at 2015-08-02 02:23:09+00:00
02:20 logmsgbot: l10nupdate Synchronized php-1.26wmf16/cache/l10n: (no message) (duration: 06m 11s)

2015-08-01

06:04 _joe_: removing some old apache access logs from mw1114
05:06 logmsgbot: @tin ResourceLoader cache refresh completed at Sat Aug 1 05:06:46 UTC 2015 (duration 6m 45s)
03:53 andrewbogott: cleared out nova-conductor.log on labcontrol1001, restarted nova-conductor, graceful’d apache
02:23 logmsgbot: @tin LocalisationUpdate completed (1.26wmf16) at 2015-08-01 02:23:15+00:00
02:20 logmsgbot: l10nupdate Synchronized php-1.26wmf16/cache/l10n: (no message) (duration: 06m 11s)
00:12 logmsgbot: ori Synchronized extract2.php: Ie919881a4: Add an API listing template to the allowed templates in extract2.php
00:01 logmsgbot: ori Synchronized php-1.26wmf16/includes: Revert I4afaecd8: "Avoiding writing sessions for no reason", and undo several uncommitted live-hacks for debugging T102199 (duration: 00m 16s)

2000s

Archive 1: 2004 Jun - 2004 Sep
Archive 2: 2004 Oct - 2004 Nov
Archive 3: 2004 Dec - 2005 Mar
Archive 4: 2005 Apr - 2005 Jul
Archive 5: 2005 Aug - 2005 Oct, with revision history 2004-06-23 to 2005-11-25
Archive 6: 2005 Nov - 2006 Feb
Archive 7: 2006 Mar - 2006 Jun
Archive 8: 2006 Jul - 2006 Sep
Archive 9: 2006 Oct - 2007 Jan, with revision history 2005-11-25 to 2007-02-21
Archive 10: 2007 Feb - 2007 Jun
Archive 11: 2007 Jul - 2007 Dec
Archive 12: 2008 Jan - 2008 Jul
Archive 12a: 2008 Aug
Archive 12b: 2008 Sept
Archive 13: 2008 Oct - 2009 Jun
Archive 14: 2009 Jun - 2009 Dec

2010s

2020s