Server Admin Log/Archive 23

December 31

22:59 ori: morebots went missing. job was active on toollabs but stuck. nothing useful in logs. restarted.
11:59 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'adding *.raa.se to the wgCopyUploadsDomains array 104683'
07:39 springle: resume schema changes for gerrit 51675 externallinks.el_id
07:26 logmsgbot: aaron finished scap: php-1.23wmf6 Timing test
07:22 logmsgbot: aaron started scap: php-1.23wmf6 Timing test
05:25 logmsgbot: aaron finished scap: php-1.23wmf6 Trying timing test again
05:20 logmsgbot: aaron started scap: php-1.23wmf6 Trying timing test again
05:16 springle: package upgrade sanitarium hosts db1053 db1054 db1057
03:23 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Dec 31 03:23:18 UTC 2013
03:20 logmsgbot: aaron finished scap: php-1.23wmf6 Timing test
03:01 logmsgbot: aaron started scap: php-1.23wmf6 testing scap timing
02:09 awight: listeners updated from 1ebcfa83671b2caade50948f88c2be5cc9798692 to a4748700afaa4a7c5e9b84bfa1f78e3298e04ad1
01:06 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Disable MessageBlobStore::clear() via hook'
01:04 awight: updated tools from 874e7039a24f6fa1c23c9d30435dd4d5e9038906 to 7f91e78c211d3ba016e2fa469f1d7093540e18c1
00:24 Aaron|home: Running a swift eqiad->tampa sync script (no-deletes) to catch any inconsistencies that built up
00:14 awight: tools updated from 64ae848b2518bc9e838b42dfbb894eb1eb845043 to 874e7039a24f6fa1c23c9d30435dd4d5e9038906
00:11 logmsgbot: maxsem synchronized wmf-config/CommonSettings.php 'https://gerrit.wikimedia.org/r/103611'
00:07 logmsgbot: maxsem synchronized php-1.23wmf8/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/104682'

December 30

23:52 awight: tools updated from from 47775021e832a5d1aa80a657086ccb4bc73d4577 to 64ae848b2518bc9e838b42dfbb894eb1eb845043
23:34 awight: updated tools from dc39dfdb381d8279d995948f50cb5fe5b3b6de4c to 47775021e832a5d1aa80a657086ccb4bc73d4577
23:17 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Removed fileJournal config; this has been unused for some time'
23:10 hashar: Jenkins / gallium deleting build history from 2012 for operations* and mediawiki* jobs. That is running in a screen.
23:05 hashar: Jenkins purged build history of mediawiki-core-lint from before July 2013.
22:57 hashar: Jenkins on gallium, /var/lib/jenkins/jobs/mediawiki-core-lint/builds eventually reached the maximum possible number of file entries. Deleting a bunch
22:44 hashar: jenkins job mediawiki-core-lint is broken since 20:35 UTC roughly
22:27 logmsgbot: reedy synchronized wmf-config/
22:18 logmsgbot: reedy synchronized wmf-config/
22:04 logmsgbot: reedy synchronized docroot and w
21:58 logmsgbot: reedy updated /a/common to I028589438: Clean up wgSiteName in InitialiseSettings
21:57 logmsgbot: reedy synchronized wmf-config/
20:58 logmsgbot: aaron finished scap: php-1.23wmf6 Scap timing test
20:50 logmsgbot: aaron started scap: php-1.23wmf6 Scap timing test
20:43 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
20:21 logmsgbot: reedy synchronized php-1.23wmf8/extensions/CentralAuth/
20:07 awight: updated tools from 882edb438b384c98c82974067ad05f844af61480 to dc39dfdb381d8279d995948f50cb5fe5b3b6de4c
20:02 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Configured $wgJobBackoffThrottling'
19:49 logmsgbot: reedy synchronized php-1.23wmf8/extensions/CentralAuth/
19:37 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: non wikipedias to 1.23wmf8
19:35 logmsgbot: reedy updated /a/common to Ib1ce6e204: Actually turn cirrus on for new wikis
19:33 awight: tools updated from da577429836bfb056fa3991d58eda24d3812f2eb to 882edb438b384c98c82974067ad05f844af61480
19:25 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php
19:16 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php
19:13 logmsgbot: manybubbles finished scap: active Update CirrusSearch and Elastica to master
19:02 logmsgbot: manybubbles synchronized php-1.23wmf7/includes/media/MediaHandler.php 'Backport getEntireText so cirrus stop complaining'
18:32 logmsgbot: manybubbles started scap: active Update CirrusSearch and Elastica to master
18:01 cmjohnson1: powering down search1005 to swap /dev/sdb
17:56 manybubbles: aborted scap
17:55 logmsgbot: manybubbles started scap: active Update CirrusSearch and Elastica to master
17:29 ^d: aborted scap
17:29 logmsgbot: demon started scap
17:25 ^d: aborted scap
17:25 logmsgbot: demon started scap: active Cirrus and Elastica to master
12:58 akosiaris: killed etherpad-old.wikimedia.org
02:32 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Dec 30 02:32:44 UTC 2013
02:21 logmsgbot: LocalisationUpdate completed (1.23wmf8) at Mon Dec 30 02:21:00 UTC 2013
02:06 logmsgbot: LocalisationUpdate completed (1.23wmf7) at Mon Dec 30 02:06:31 UTC 2013

December 29

22:44 logmsgbot: aaron synchronized php-1.23wmf8/includes/job/JobQueueFederated.php '6d3abab7cd6d344e903c25d4b24bda9ef239652b'
02:30 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Dec 29 02:30:44 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.23wmf8) at Sun Dec 29 02:15:24 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.23wmf7) at Sun Dec 29 02:08:48 UTC 2013

December 28

02:27 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Dec 28 02:27:24 UTC 2013
02:13 logmsgbot: LocalisationUpdate completed (1.23wmf8) at Sat Dec 28 02:13:10 UTC 2013
02:07 logmsgbot: LocalisationUpdate completed (1.23wmf7) at Sat Dec 28 02:07:43 UTC 2013
00:13 logmsgbot: aaron finished scap
00:02 logmsgbot: aaron started scap
00:00 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files:

December 27

23:59 AaronSchulz: doing some sync script timing
22:28 ori: Stopping uWSGI on Tungsten to debug Graphite issue
21:20 awight: updated crm from 0dbfaf8ef9dcf72ec88f1ff1ad38dc7273d987a6 to 81d06d3d21c26c38fb80cd027f8b66308ca3fac7
17:55 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files:
05:17 awight: updated tools from 9f96cad6739504048a32812fff655f14b3371411 to da577429836bfb056fa3991d58eda24d3812f2eb
04:02 awight: updated tools from 727505d13e5f1b7e0d1ece20832900a9cfdd2870 to 9f96cad6739504048a32812fff655f14b3371411
03:00 awight: tools updated from fb30e4e41d27dd4b5b29c9f002c413e3941b00c0 to 727505d13e5f1b7e0d1ece20832900a9cfdd2870
02:35 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Dec 27 02:35:51 UTC 2013
02:18 logmsgbot: LocalisationUpdate completed (1.23wmf8) at Fri Dec 27 02:18:20 UTC 2013
02:10 logmsgbot: LocalisationUpdate completed (1.23wmf7) at Fri Dec 27 02:10:22 UTC 2013
01:46 awight: updated civicrm from 87dea8afb964d0622f3ee92b6e120915846fde56 to 0dbfaf8ef9dcf72ec88f1ff1ad38dc7273d987a6
01:30 awight: updated tools from 3eaf88c3dda53ec8c521e664ff0fd25b50a4814f to fb30e4e41d27dd4b5b29c9f002c413e3941b00c0
00:14 awight: updated tools to 3eaf88c3dda53ec8c521e664ff0fd25b50a4814f

December 26

23:59 logmsgbot: aaron started scap
21:18 logmsgbot: aaron finished scap
21:07 MaxSem: Recreating GeoData index
21:03 logmsgbot: aaron started scap
20:52 andrewbogott: restarting parsoid cluster via salt
20:49 subbu: updated localsettings.js to unbreak VE on wikis with "-" in their name (ex: nds-nl)
14:29 akosiaris: killed stuck puppet agent on db9 and forced manual puppet run.
02:35 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Dec 26 02:35:43 UTC 2013
02:18 logmsgbot: LocalisationUpdate completed (1.23wmf8) at Thu Dec 26 02:18:37 UTC 2013
02:10 logmsgbot: LocalisationUpdate completed (1.23wmf7) at Thu Dec 26 02:10:16 UTC 2013

December 25

02:40 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Dec 25 02:40:35 UTC 2013
02:14 logmsgbot: LocalisationUpdate completed (1.23wmf8) at Wed Dec 25 02:14:49 UTC 2013
02:14 logmsgbot: LocalisationUpdate completed (1.23wmf7) at Wed Dec 25 02:14:05 UTC 2013
02:00 logmsgbot: aaron finished scap
01:53 logmsgbot: aaron started scap
01:53 logmsgbot: aaron finished scap
01:32 logmsgbot: aaron started scap
01:06 logmsgbot: ori finished scap
01:01 logmsgbot: ori started scap
01:00 logmsgbot: aaron finished scap
00:57 logmsgbot: aaron started scap
00:51 logmsgbot: aaron started scap
00:45 logmsgbot: aaron started scap
00:43 logmsgbot: aaron started scap
00:40 logmsgbot: aaron started scap
00:14 logmsgbot: aaron finished scap
00:07 logmsgbot: aaron started scap

December 24

13:31 logmsgbot: reedy synchronized php-1.23wmf7/extensions/GlobalBlocking 'bug 58934 Icc47a2d6367c0b906e40e068635c9fda07108e0f'
03:08 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's1 db1033 to full steam'
02:28 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Dec 24 02:28:42 UTC 2013
02:14 logmsgbot: LocalisationUpdate completed (1.23wmf8) at Tue Dec 24 02:14:09 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.23wmf7) at Tue Dec 24 02:08:04 UTC 2013

December 23

22:15 awight: updated crm from ce73eaa0ea6019f76c63d0cca097bf828eb933e2 to 87dea8afb964d0622f3ee92b6e120915846fde56
20:06 paravoid: cr1-eqiad/cr1-sdtpa: new BGP policy to reset MED for XO
16:59 cmjohnson1: dns update
15:30 cmjohnson1: labnet1001 relocating racks C4 to B3 eqiad
13:24 springle_: starting xtrabackup clone db1001 to db1016
10:39 akosiaris: uploaded php5_5.3.10-1ubuntu3.9+wmf1 to apt.wikimedia.org
09:54 paravoid: rebooting cp1065 with Linux 3.2
07:43 springle: taking down db1016 for xfs_check, locked up and: XFS (dm-0): xlog_space_left: head behind tail
05:57 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'warm up db1033 in s1'
02:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Dec 23 02:26:35 UTC 2013
02:13 logmsgbot: LocalisationUpdate completed (1.23wmf8) at Mon Dec 23 02:13:09 UTC 2013
02:07 logmsgbot: LocalisationUpdate completed (1.23wmf7) at Mon Dec 23 02:07:23 UTC 2013
00:50 springle: xtrabackup clone db1049 to db1033
00:37 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'rebalance group loads while db1050 out'

December 22

19:35 apergos: db1050 boot: (during fsck) hung at "The disk drive for /a is not ready yet or not present. Continue waiting, S to skip.." etc
19:34 apergos: db1050 boot: "[ 65.625950] device-mapper: table: 252:3: snapshot: Snapshot cow pairing for exception table handover failed"
19:33 logmsgbot: ori synchronized wmf-config/db-eqiad.php 'Depool db1050'
19:32 logmsgbot: ori updated /a/common to If33894260: Depool db1050; unresponsive
19:27 apergos: powrcycling db1050, inaccessible
02:28 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Dec 22 02:28:45 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.23wmf8) at Sun Dec 22 02:15:27 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.23wmf7) at Sun Dec 22 02:08:39 UTC 2013

December 21

02:33 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Dec 21 02:33:00 UTC 2013
02:18 logmsgbot: LocalisationUpdate completed (1.23wmf8) at Sat Dec 21 02:18:39 UTC 2013
02:09 logmsgbot: LocalisationUpdate completed (1.23wmf7) at Sat Dec 21 02:09:34 UTC 2013

December 20

21:37 mutante: cp1065 - currently init 1 sitting at maintenance but not powered down, i will poke later or anyone can
20:43 mutante: cp1065 - was 3.11.0-13-generic, rebooting for 3.2.0-57-generic
20:38 mutante: powercycled cp1065 again, will upgrade kernel this time
19:09 mutante: cp1065 - [19060.904886] BUG: scheduling while atomic: kworker/11:0/27073/0x00000200
19:09 mutante: powercycling cp1065
16:18 andrewbogott: merged a couple of tiny OpenStackManager patches on wikitech
14:34 hashar: Jenkins removed http_proxy env variables from lanthanum.eqiad.wmnet does not play well with git urls :D
14:28 hashar: Jenkins setting http_proxy and https_proxy on lanthanum.eqiad.wmnet to point to carbon.wikimedia.org (eqiad web proxy)
13:52 paravoid: powercycling cp1065
11:53 akosiaris: added akosiaris account on all junipers. Also added faidon account wherever it did not exist and synchronized its uid (only one uid change on csw2-esams happened)
11:16 hashar: updated Jenkins job linting operations/apache-config.git , now detects rewrites.conf not being updated when redirects.dat has been changed.
10:19 apergos: powercycled cp1065 and had to manually start varnish, same as yesterday
02:04 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'vslow query load balancing'
01:12 logmsgbot: spage synchronized php-1.23wmf8/extensions/Flow/includes/Notifications/Controller.php 'Bug 58705 notification fix pt.3'
01:08 logmsgbot: spage synchronized php-1.23wmf8/extensions/Echo/includes/EmailFormatter.php 'Bug 58705 notification fix pt.2'
01:06 logmsgbot: spage synchronized php-1.23wmf8/extensions/Echo/formatters/BasicFormatter.php 'Bug 58705 notification fix'
00:48 logmsgbot: ori synchronized php-1.23wmf8/extensions/WikimediaEvents 'I7cad8fd35fd: Update WikimediaEvents to master'
00:47 logmsgbot: ori synchronized php-1.23wmf7/extensions/WikimediaEvents 'I7cad8fd35fd: Update WikimediaEvents to master'
00:22 mutante: create /a/mw-log/Bug58676.log for yurik, touch, chown udplog ..etc
00:10 logmsgbot: ori synchronized php-1.23wmf7/extensions/WikimediaEvents 'Ibe918eb96: Update WikimediaEvents to master for Ia6f758068 (Log event when new editors reach edit milestone)'
00:09 logmsgbot: ori synchronized php-1.23wmf8/extensions/WikimediaEvents 'Ibe918eb96: Update WikimediaEvents to master for Ia6f758068 (Log event when new editors reach edit milestone)'

December 19

21:47 logmsgbot: csteipp finished scap: bug 58472, 57550, 58553
21:08 logmsgbot: csteipp started scap: bug 58472, 57550, 58553
19:53 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
19:52 logmsgbot: reedy synchronized php-1.23wmf7/includes/Message.php
19:20 logmsgbot: reedy synchronized docroot and w
19:19 logmsgbot: reedy synchronized wmf-config/
19:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: phase1 wikis to 1.23wmf8
19:06 logmsgbot: reedy updated /a/common to I952672334: Rename Persian Wikibooks per community consensus
19:05 logmsgbot: reedy synchronized wmf-config/
19:01 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All wikipedias to 1.23wmf7
19:00 logmsgbot: reedy updated /a/common to I22cfefa3c: Add an alias to NS_PROJECT on Old English Wiktionary
18:33 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.23wmf7 till window
18:32 logmsgbot: reedy finished scap: Build l10n cache and testwiki to 1.23wmf8
17:56 logmsgbot: reedy started scap: Build l10n cache and testwiki to 1.23wmf8
17:48 logmsgbot: reedy synchronized docroot and w
17:47 logmsgbot: reedy updated /a/common to Ib6bc15109: previously missed deleted wikis, bug 58680
17:46 logmsgbot: reedy synchronized php-1.23wmf8 'Staging'
09:50 apergos: powercycled cp1065, unresponsive, no messages on mgmt console
03:49 Krinkle: Reloading Zuul to deploy I5b759c388db5e5a
03:20 springle: starting schema changes for gerrit 101170 page_links_updated
02:58 logmsgbot: springle synchronized database lists files:
02:53 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Dec 19 02:53:46 UTC 2013
02:31 logmsgbot: LocalisationUpdate completed (1.23wmf6) at Thu Dec 19 02:31:31 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.23wmf7) at Thu Dec 19 02:16:45 UTC 2013
00:10 subbu: updated parsoid config to add support for tyv and min wikipedias
00:09 mutante: doing parsoid restart via salt, with the 10% option

December 18

23:44 Krinkle: Zuul pipeline for gate-and-submit fixed (the revert was someone undone after the reload, deployed the same git commit again)
23:36 Krinkle: Reloading Zuul
23:34 Krinkle: Zuul is once again not processing any CR+2 events for the gate-and-submit pipeline
23:03 ottomata: kafka brokers on an21 and an22 are now running with cleaned up topics
23:02 ori-l: zirconium: service apache2 reload for wikimania scholarships app
22:40 Krinkle: Zuul pipeline for gate-and-submit fixed
22:38 Krinkle: Reloading Zuul to deploy Ic8cf602a0aa72
22:32 Krinkle: Zuul seems to be unresponsive to gate-and-submit events for mediawiki/* projects
21:45 ottomata: stopping kafka in order to cleanup old topic 'webrequest-mobile'. Usually we won't have to do cleanups like this, but since this topic is new and no one is using it yes, I'd rather clean up now rather than never.'
20:39 awight: update crm from 7deaa1e1a6354de545fedb0e90b85f117e7df0bd to ce73eaa0ea6019f76c63d0cca097bf828eb933e2
20:24 logmsgbot: reedy updated /a/common to I33c68abb1: Have sysops add and remove users from the 'gwtoolset' group
20:12 logmsgbot: reedy synchronized wmf-config/
19:06 Reedy: Created EducationProgram tables on fawiki
19:05 logmsgbot: reedy synchronized php-1.23wmf7/extensions/GWToolset/
19:03 logmsgbot: reedy synchronized wmf-config/
18:20 logmsgbot: otto synchronized wmf-config 'Syncing wmf-config for Cirrus config updates'
18:19 logmsgbot: otto synchronized cirrus.dblist 'Cirrus config update, adding new wikis'
17:52 logmsgbot: otto synchronized php-1.23wmf6/extensions/CirrusSearch 'updating php-1.23wmf6/extensions/CirrusSearch to master'
17:51 logmsgbot: otto synchronized php-1.23wmf6/extensions/CirrusSearch 'updating php-1.23wmf6/extensions/CirrusSearch to master'
17:46 hashar: Zuul upgrade working as expected as far as I monitored it. Mail sent to wikitech-l and eng list.
17:27 logmsgbot: otto synchronized php-1.23wmf7/extensions/CirrusSearch 'updating php-1.23wmf7/extensions/CirrusSearch to master'
17:23 hashar: Zuul : speedy deploy of a Wikimedia hack to prevent Zuul from updating build description. Seems to slowing down everything. Hack is 102465. Deployed using https://www.mediawiki.org/wiki/Continuous_integration/Zuul#upgrading
16:46 ottomata: updated varnishkafka in apt to 1.0.0-2 and installed on mobile varnishes (this includes fix for logrotate postrotate command)
15:40 paravoid: powercycling cp1065 again, kernel BUG, soft lockup
15:29 paravoid: powercycling cp1065, kmem_alloc deadlock
15:01 hashar: Gerrit back :-)
15:00 hashar: Gerrit dead :(
13:51 paravoid: setting up new transit peering in eqiad
10:00 ori-l: disabled puppet on tungsten to test mwprof changes; re-enabling now
07:37 springle: online schema changes on masters with sql_log_bin=0, indexes only, gerrit 85508
05:32 ori-l: DNS update: scholarships.wikimedia.org CNAME misc-web-lb.eqiad
03:09 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Dec 18 03:09:14 UTC 2013
02:47 logmsgbot: LocalisationUpdate completed (1.23wmf6) at Wed Dec 18 02:47:41 UTC 2013
02:25 logmsgbot: LocalisationUpdate completed (1.23wmf7) at Wed Dec 18 02:25:17 UTC 2013
01:52 logmsgbot: reedy synchronized wmf-config/ 'touch'
01:46 logmsgbot: reedy synchronized wmf-config/ 'Ensuring consistency...'
00:19 logmsgbot: mflaschen synchronized wmf-config 'Deploy Draft namespace to English Wikipedia. Remove old Labs overrides that were used for testing this.'
00:12 springle: ongoing schema changes on slaves, indexing only, logging gerrit 85508
00:05 Krinkle: Reloading zuul to deploy I6df1f4acccd7a9

December 17

23:29 LeslieCarr: starting atftpd on carbon
22:37 logmsgbot: ori synchronized php-1.23wmf6/extensions/NavigationTiming 'I933a1e3a2: Add 'mediaWikiLoadComplete' measurement'
22:30 logmsgbot: ori synchronized php-1.23wmf7/extensions/NavigationTiming 'I933a1e3a2: Add 'mediaWikiLoadComplete' measurement'
22:16 logmsgbot: ori synchronized php-1.23wmf6/resources/startup.js 'I3d5bcf10e: startup.js: log current time as global 'mediaWikiLoadStart
22:15 ori-l: restarted apache on zirconium for scholarships vhost
22:03 logmsgbot: ori synchronized php-1.23wmf7/resources/startup.js 'I3d5bcf10e: startup.js: log current time as global 'mediaWikiLoadStart
21:48 logmsgbot: csteipp synchronized php-1.23wmf7/extensions/TimedMediaHandler 'bug56699'
21:47 logmsgbot: csteipp synchronized php-1.23wmf6/extensions/TimedMediaHandler 'bug56699'
21:37 logmsgbot: csteipp synchronized php-1.23wmf7/extensions/CentralAuth 'bug57081'
21:36 logmsgbot: csteipp synchronized php-1.23wmf6/extensions/CentralAuth 'bug57081'
21:31 logmsgbot: csteipp synchronized php-1.23wmf6/includes 'bug58088'
21:30 logmsgbot: csteipp synchronized php-1.23wmf7/includes 'bug58088'
21:23 logmsgbot: reedy synchronized wmf-config/ 'Enable GWToolset on commonswiki'
21:16 logmsgbot: reedy updated /a/common to I2d1c666e1: Production configuration for GWToolset
21:14 logmsgbot: reedy finished scap: Rebuild l10n cache for GWToolset, remove AFT, no wikis moving onto GWToolset at this point (commented out)
20:45 manybubbles: finished the rolling restart of the elasticsearch cluster. it could have been done more quickly but there was no hurry.
20:45 logmsgbot: reedy started scap: Rebuild l10n cache for GWToolset, remove AFT, no wikis moving onto GWToolset at this point (commented out)
20:05 Ryan_Lane: deploying change 102285 to OpenStackManager on virt0
19:58 logmsgbot: reedy synchronized php-1.23wmf7/extensions/GWToolset 'Staging'
19:54 logmsgbot: reedy synchronized php-1.23wmf7/includes/filerepo/LocalRepo.php 'Fix fatal I12513b40453573124e838d54a72a2f9a2d3de338'
19:32 logmsgbot: reedy synchronized wmf-config/ 'Revert Revert Cross-wiki backlink purging for commons file changes'
19:27 logmsgbot: reedy synchronized wmf-config/ 'Revert Cross-wiki backlink purging for commons file changes'
19:26 logmsgbot: reedy updated /a/common to Id4124ef28: All non wikipedias to 1.23wmf7
19:13 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All non wikipedias to 1.23wmf7
19:11 logmsgbot: reedy synchronized wmf-config/
18:44 logmsgbot: reedy synchronized wmf-config/
18:19 logmsgbot: reedy updated /a/common to I5ae36ae21: EasyTimeline support for private wikis via img_auth
18:11 mutante: deleted php-1.22wmf17 from tin per reedy
18:01 logmsgbot: reedy synchronized php-1.23wmf7/includes/revisiondelete/RevisionDelete.php 'I0bd4a5fe9687c4261ca0f57e30f723e8bf2589ac'
17:59 logmsgbot: reedy synchronized php-1.23wmf6/includes/revisiondelete/RevisionDelete.php 'I0bd4a5fe9687c4261ca0f57e30f723e8bf2589ac'
16:58 hashar: Jenkins is back up
16:50 hashar: Jenkins is busy reloading, should be back around 5:20pm UTC. Don't kill it meanwhile it is busy reading a bunch of files :-(
16:27 hashar: restarting Jenkins (stuck)
16:17 hashar: Jenkins web service threads are all hang busy waiting for a long request to achieve. Caused by myself :/
16:10 hashar: jenkins in a bad mood for some unknown reason :(
15:00 hashar: Jenkins manually configured mediawiki-core-phpunit-misc job to be runnable concurrently
14:10 manybubbles: the first pass of index building for all wikisources is complete, starting the second pass
13:31 manybubbles: performing a rolling restart of the elasticsearch cluster to pick up new settings
10:27 mark: Depooled eqiad Squids in PyBal
10:24 mark: Depooled esams Squids in PyBal
06:43 Krinkle: Increased number of Jenkins executors for 'integration-slave01' from 2 to 3
04:53 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
02:58 Krinkle: Reloading Zuul to deploy I4e18cb2dc2a7f4
02:49 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Dec 17 02:49:29 UTC 2013
02:30 logmsgbot: LocalisationUpdate completed (1.23wmf7) at Tue Dec 17 02:30:08 UTC 2013
02:25 manybubbles: manually setting elastic1008 to be master eligible so we have three master eligible machines in production until the puppet code that would do that properly is merged
02:16 logmsgbot: LocalisationUpdate completed (1.23wmf6) at Tue Dec 17 02:16:11 UTC 2013
01:57 Krinkle: Reloading Zuul to deploy I80dafe3457c65
00:27 apergos: elastic1007 failed to come back up after several attempts, a soft drac reset and some more attempts. leaving it in power off state
00:15 apergos: depooled elastic1007 in pybal
00:07 apergos: ERROR: Timeout while waiting for server to perform requested power action. (from attempt to powercycle elastic1007)
00:04 apergos: powercycled elastic1007, inaccessible via ssh or mgmt console

December 16

22:55 Ryan_Lane: rebooting virt0
22:51 Ryan_Lane: dist-upgrading virt0
22:44 Ryan_Lane: rebooting virt1000
22:34 Ryan_Lane: dist-upgrading virt1000
21:50 gwicke: updated Parsoid config to use the API cluster directly for most wikis
21:19 subbu: deployed parsoid 7684df12
21:12 mutante: creating wikilovesearth mailing list for bug 52705
19:27 ottomata: stopping puppet on cp1048 to test varnishkafka ganglia module
19:05 Krinkle: Reloading Zuul to deploy I01d349bf21b20ce94
18:27 logmsgbot: yurik synchronized php-1.23wmf7/extensions/ZeroRatedMobileAccess/
18:24 logmsgbot: yurik synchronized php-1.23wmf6/extensions/ZeroRatedMobileAccess/
18:05 manybubbles: all search indexes built for newly cirrused wikis (wikisources other than enwikisource and frwikisource and commons). populating them now. commons will take the longest. it'll take 13 hours at the current rate. past experience tells me we won't be able to keep that rate up but that this is the slow time of day to be doing it anyway.
17:41 logmsgbot: manybubbles synchronized wmf-config/ 'update cirrus configuration'
17:31 logmsgbot: manybubbles synchronized php-1.23wmf6/extensions/CirrusSearch/ 'update cirrus to master'
17:28 logmsgbot: manybubbles synchronized php-1.23wmf7/extensions/CirrusSearch/ 'update cirrus to master'
15:08 mark: Depooled all pmtpa Squids in PyBal
14:51 hashar: jenkins enabled linting jobs to be runnable in parallel. Whenever several changes are made on the same repo, Jenkins will trigger a linting job per change. That will dramatically speed up the processing of changes since some jobs are now parallelized instead of serialized.
14:48 hashar: zuul made gate-and-submit pipeline a dependent pipeline. Changes would thus be triggered in parallel whenever a repo has several +2 attempting to land in. That should speed up gating process. See also bug 48419 and 101839
14:06 logmsgbot: springle synchronized wmf-config/db-pmtpa.php
14:06 paravoid: ganglia-monitor restart on srv*/mw*; gmond bug with swapoff
13:42 logmsgbot: springle synchronized wmf-config/db-pmtpa.php
13:26 logmsgbot: springle synchronized wmf-config/db-pmtpa.php
11:28 logmsgbot: springle synchronized wmf-config/db-pmtpa.php
06:42 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool db1026 after schema changes, LB lowered during warm up'
06:27 logmsgbot: tstarling synchronized docroot/secure/404.html
05:35 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'If79a9443a: Add MassMessage to $wgDebugLogGroups'
05:34 logmsgbot: ori updated /a/common to If79a9443a: Add MassMessage to $wgDebugLogGroups
05:34 logmsgbot: ori synchronized php-1.23wmf6/extensions/MassMessage/MassMessageJob.php 'Iec240623a: Add debug logging for bug 57464'
04:32 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1026 during schema changes'
02:34 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Dec 16 02:34:36 UTC 2013
02:19 logmsgbot: LocalisationUpdate completed (1.23wmf7) at Mon Dec 16 02:19:59 UTC 2013
02:13 paravoid: salt swapoff -a; sed -i "/swap/d" /etc/fstab on all srv*, mw*
02:11 logmsgbot: LocalisationUpdate completed (1.23wmf6) at Mon Dec 16 02:11:13 UTC 2013

December 15

20:53 Jamesofur: insert arbcom election decryption key
11:24 logmsgbot: maxsem synchronized php-1.23wmf6/includes/parser/ParserOutput.php 'Debugging done'
10:08 logmsgbot: maxsem synchronized php-1.23wmf6/includes/parser/ParserOutput.php 'Debugging'
02:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Dec 15 02:26:19 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.23wmf7) at Sun Dec 15 02:16:36 UTC 2013
02:09 logmsgbot: LocalisationUpdate completed (1.23wmf6) at Sun Dec 15 02:09:37 UTC 2013

December 14

02:28 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Dec 14 02:28:38 UTC 2013
02:19 logmsgbot: LocalisationUpdate completed (1.23wmf7) at Sat Dec 14 02:19:24 UTC 2013
02:10 logmsgbot: LocalisationUpdate completed (1.23wmf6) at Sat Dec 14 02:10:34 UTC 2013

December 13

19:54 spagewmf: Flow team deployed bug 58455 fix to 1.23wmf7
18:33 akosiaris: install PHP5 5.3.10-1ubuntu3.9+wmf1 on mw1017 (test.wikipedia.org)
15:50 paravoid: reenabling ospfv3 on eqiad-esams
14:11 akosiaris: applied the new installserver role to brewster, carbon, bast4001 and hooft. This implies installation of ferm and usage of base::firewall which means default firewall policy will now be DROP. Punched necessary holes for services to work.
14:06 akosiaris: just ran aptitude remove apache2-mpm-prefork apache2-utils apache2.2-bin apache2.2-common libapache2-mod-php5 on bast1001. Autoremoved texlive-* packages and timidity. All seemed to be installed because of wikimedia-task-appserver (also removed)
13:36 springle: killing queries on es100[1-4]
13:31 springle: reduce max_connections on es100[1-4]. try to survive sudden spike
09:58 akosiaris: disabling puppet on carbon for a short time (debugging the new install-server module)
02:50 springle: ongoing schema changes on slaves, indexing only, logging gerrit 85508, wb_terms gerrit 99660
02:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Dec 13 02:48:41 UTC 2013
02:41 logmsgbot: LocalisationUpdate completed (1.23wmf7) at Fri Dec 13 02:41:23 UTC 2013
02:21 logmsgbot: LocalisationUpdate completed (1.23wmf6) at Fri Dec 13 02:21:05 UTC 2013
00:25 logmsgbot: mholmquist synchronized php-1.23wmf7/extensions/MultimediaViewer/ 'Fix for another stray MultimediaViewer event, take 2.'
00:25 logmsgbot: mholmquist synchronized php-1.23wmf6/extensions/MultimediaViewer/ 'Fix for another stray MultimediaViewer event, take 2.'
00:20 logmsgbot: mholmquist synchronized php-1.23wmf7/extensions/MultimediaViewer/ 'Fix for another stray MultimediaViewer event'
00:18 logmsgbot: mholmquist synchronized php-1.23wmf6/extensions/MultimediaViewer/ 'Fix for another stray MultimediaViewer event'
00:12 logmsgbot: mwalker synchronized php-1.23wmf7/extensions/Collection/ 'Reverting to known good condition for collection extension'
00:11 logmsgbot: mwalker synchronized php-1.23wmf6/extensions/Collection/ 'Reverting to known good condition for collection extension'
00:04 logmsgbot: catrope synchronized php-1.23wmf7/extensions/EducationProgram 'Cherry-pick to fix ContextSource compatibility'
00:03 logmsgbot: catrope updated /a/common/php-1.23wmf7 to If7c3c52ed: Update EducationProgram to wmf7 branch for cherry-picks

December 12

23:55 logmsgbot: catrope finished scap: Scap for VisualEditor i18n problems
23:36 logmsgbot: catrope started scap: Scap for VisualEditor i18n problems
23:31 logmsgbot: catrope synchronized php-1.23wmf7/extensions/VisualEditor 'Update VE for cherry-pick'
23:30 logmsgbot: catrope synchronized php-1.23wmf6/extensions/VisualEditor 'Update VE for cherry-pick'
23:29 logmsgbot: catrope updated /a/common/php-1.23wmf7 to I492fe5762: Update VisualEditor to wmf7 branch for cherry-picks
22:10 logmsgbot: aaron synchronized php-1.23wmf6/includes/job/JobQueueGroup.php 'de525efb4a149680f8bbf7f0d5db18489f59e4e8'
21:59 logmsgbot: mflaschen synchronized multiversion/MWMultiVersion.php 'Sync routing fix that only impacts Beta'
21:31 logmsgbot: reedy synchronized php-1.23wmf7/extensions/Wikibase 'Fix scribunto integration'
21:30 logmsgbot: reedy synchronized php-1.23wmf6/extensions/Wikibase 'Fix scribunto integration'
19:35 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: phase1 wikis to 1.23wmf7
19:27 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.23wmf6
19:27 logmsgbot: reedy updated /a/common to Ib597431a3: Set $wgULSNoWebfontsSelectors
19:26 awight: crm updated from 0e7f22692b19eaa8deb3eae0d34ec077b7ab22d7 to 7deaa1e1a6354de545fedb0e90b85f117e7df0bd
19:26 logmsgbot: reedy synchronized php-1.23wmf7/extensions/
19:24 logmsgbot: reedy synchronized php-1.23wmf6/extensions/
17:44 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.23wmf6 till window
15:46 hashar_: started import to commons of several videos from user 80686. bug 58155
15:45 mark: Restarted varnish backend on cp1055 with no config changes
15:04 mark: Manually disabled storage backends main1b, main2b on cp1068 and restarted Varnish (puppet will overwrite)
14:13 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'touch'
14:12 logmsgbot: reedy finished scap: testwiki to 1.23wmf7, build l10n cache
14:12 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'Set $wgULSNoWebfontsSelectors 100990'
13:44 logmsgbot: reedy started scap: testwiki to 1.23wmf7, build l10n cache
13:30 logmsgbot: reedy synchronized php-1.23wmf7 'staging'
13:14 logmsgbot: reedy synchronized docroot and w
13:13 logmsgbot: reedy updated /a/common to Ib2c3794dc: Fix Flow config setting for beta labs
12:54 logmsgbot: reedy synchronized php-1.23wmf6/extensions/Scribunto/Scribunto.namespaces.php 'Fix ml namespace issue'
12:53 logmsgbot: reedy synchronized php-1.23wmf5/extensions/Scribunto/Scribunto.namespaces.php 'Fix ml namespace issue'
12:49 ori-l: Took down gdash.wm.o for some time to finish Nginx -> Apache change.
11:57 springle: xtrabackup clone s5 db1005 to db1017
11:09 springle: labsdb1003:3307 db instance crash, page corruption, now assertion failures. restoring from upstream db snapshot
09:42 hashar: on tin, started a screened wget to download materials that will be uploaded to commons bug 58155
03:25 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Dec 12 03:25:15 UTC 2013
02:38 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Thu Dec 12 02:38:50 UTC 2013
02:18 logmsgbot: LocalisationUpdate completed (1.23wmf6) at Thu Dec 12 02:18:04 UTC 2013
00:58 logmsgbot: catrope synchronized php-1.23wmf6/extensions/VisualEditor/ 'Cherry-picks'
00:12 logmsgbot: catrope synchronized php-1.23wmf6/extensions/VisualEditor/ 'Cherry-picks'
00:12 logmsgbot: catrope synchronized php-1.23wmf6/includes/libs/CSSMin.php 'Fix for bug 58338 (@import URL mangling)'
00:11 logmsgbot: catrope updated /a/common/php-1.23wmf6 to I70eaddd39: Update VisualEditor to wmf6 branch for cherry-picks
00:04 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I62541406b: Disable module storage on enwiki'
00:04 logmsgbot: ori updated /a/common to I62541406b: Disable module storage on enwiki

December 11

23:54 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I47ad0a80b: Disable module storage on all but enwiki'
23:53 logmsgbot: ori updated /a/common to I47ad0a80b: Disable module storage on all but enwiki
23:24 logmsgbot: bsitu synchronized wmf-config/CommonSettings-labs.php 'job-delay-config'
23:22 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable Flow on a few wiki pages'
23:21 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings-labs.php 'Fix the labs config'
23:19 logmsgbot: bsitu updated /a/common to I6d8a495d0: Repair flow config on labs
22:58 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings-labs.php 'Enable Flow on a few wikis (but no pages)'
22:57 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php 'Enable Flow on a few wikis (but no pages)'
22:57 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable Flow on a few wikis (but no pages)'
22:56 logmsgbot: bsitu updated /a/common to I1fba27a98: Enable Flow on a few wikis (but no pages)
22:23 logmsgbot: bsitu synchronized php-1.23wmf6/maintenance/sql.php 'Added --wikidb param to sql.php (backport to wmf6)'
22:02 subbu: deployed Parsoid abc4f426
20:57 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'Id5b930a6b: Disable module storage on eswiki to assess impact'
20:56 logmsgbot: ori updated /a/common to Id5b930a6b: Disable module storage on eswiki to assess impact
20:56 ottomata: reenabled puppet on cp1046
20:10 Krinkle: Reloading zuul to deploy I9c9399473a84
19:38 K4-713: updated payments cluster to 1bb7171fbf1d3a3
18:55 logmsgbot: yurik synchronized php-1.23wmf6/extensions/ZeroRatedMobileAccess/
18:45 ottomata: disabling puppet on cp1046 to test change to ganglia sample interval for varnishkafka stats
18:45 logmsgbot: yurik synchronized php-1.23wmf5/extensions/ZeroRatedMobileAccess/
17:22 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php 'Deploy Cirrus to new wikis as secondary and let wikidata have it as a beta feature'
17:21 logmsgbot: manybubbles synchronized wmf-config/CirrusSearch-common.php 'Make wmgUseCirrus override wmgUseCirrusAsAlternative for easier config editing'
16:51 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'I55dc9cbdf3c777ce7e9f925bb0e5269c05b23dfa'
16:44 logmsgbot: reedy updated /a/common to I42c4c5bcd: Fix Commons config for Cirrus
16:32 logmsgbot: reedy synchronized php-1.23wmf5/cache/l10n/l10n_cache-ml.cdb 'Forced rebuild'
16:24 logmsgbot: reedy synchronized php-1.23wmf6/cache/l10n/l10n_cache-ml.cdb 'Forced rebuild'
08:45 apergos: manually downgraded nodejs on contint (gallium, lanthanum) from 0.10.22 to 0.8.2 (puppet failed to do so)
06:05 LeslieCarr: silencing global collect paging (GC is having issues) for 1 hour
03:17 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Dec 11 03:17:38 UTC 2013
02:30 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Wed Dec 11 02:30:44 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.23wmf6) at Wed Dec 11 02:16:17 UTC 2013
00:22 logmsgbot: catrope synchronized php-1.23wmf6/extensions/VisualEditor/ 'Cherry-picks'
00:09 logmsgbot: mholmquist synchronized php-1.23wmf6/extensions/MultimediaViewer/tests/qunit/lightboxinterface.test.js 'Make VE and MMV work together again'
00:09 logmsgbot: mholmquist synchronized php-1.23wmf6/extensions/MultimediaViewer/resources/ext.multimediaViewer/ext.multimediaViewer.lightboxinterface.js 'Make VE and MMV work together again'
00:08 logmsgbot: mholmquist synchronized php-1.23wmf6/extensions/MultimediaViewer/resources/ext.multimediaViewer/ext.multimediaViewer.js 'Make VE and MMV work together again'

December 10

23:43 logmsgbot: csteipp synchronized php-1.23wmf5/extensions/CentralAuth 'Update to master'
23:38 logmsgbot: csteipp synchronized php-1.23wmf6/extensions/CentralAuth 'Update to master'
23:10 logmsgbot: demon synchronized wmf-config/CirrusSearch-labs.php 'No-op, fixing labs config'
23:08 K4-713: updated payments cluster to 2c44e6dc531
22:46 logmsgbot: bsitu finished scap: Add Flow extension but not enabled yet
22:32 mwalker: issued `git reset bisect a4f97e4`in extensions collection in 1.23wmf6 to clean up after yesterdays bisection mess
22:22 logmsgbot: bsitu started scap: Add Flow extension but not enabled yet
22:11 K4-713: updated payments cluster to 321df2d84cc9c77292b5d
21:45 logmsgbot: bsitu updated /a/common to Ib34f619e7: Enable AbuseFilter block option on Wikidata
21:17 logmsgbot: reedy synchronized wmf-config/abusefilter.php
20:42 logmsgbot: reedy synchronized php-1.23wmf6/extensions/Wikibase
20:24 LeslieCarr: applied a fitler to inet6 sandbox subnet in eqiad
20:11 logmsgbot: reedy synchronized php-1.23wmf6/includes/SkinTemplate.php 'I8edbdfe615f848963e3bea47dac99d1abd64c7f7'
20:09 LeslieCarr: applying filter on sandbox subnet in eqiad
19:46 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias and wikivoyages to 1.23wmf6
19:44 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Revert that, CreditSource broken
19:42 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.23wmf6
19:39 logmsgbot: reedy updated /a/common to Ibceb9638f: beta: gwtoolset-whitelist
19:08 logmsgbot: reedy synchronized php-1.23wmf6/extensions 'I53db62d469a6944cdf24dc209fa55c972f69b73f'
17:38 gwicke: updated Parsoid back to 31910075 after verifying that this was not the source of the memory leak
17:15 mutante: running gerrit reviewer-counts cron command manually for bug 52329
17:02 gwicke: downgraded Parsoid to 0ac82a28 and rolled back config update to rule out code changes for memory leak
16:48 gwicke: updated Parsoid node_modules for 0.10 to fix what looks like a memory leak
16:38 paravoid: restarting all of parsoid again
16:35 MaxSem: Nuking Rezabot@fawiki's watchlist, requested by operator
14:01 logmsgbot: hashar synchronized php-1.23wmf6/extensions/ProofreadPage
12:39 paravoid: restarting parsoid across the cluster, 100% CPU on all appservers
07:33 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'explicit LB for each updateSpecialPages job'
05:35 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'db1049 to LB=0 except groupLoadsByDB'
04:17 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool db1049 after upgrade'
03:32 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1049 for upgrade'
03:15 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Dec 10 03:15:08 UTC 2013
02:29 logmsgbot: LocalisationUpdate completed (1.23wmf6) at Tue Dec 10 02:29:36 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Tue Dec 10 02:15:52 UTC 2013
00:25 gwicke: reverted Parsoid proxy change as officewiki and some other https-only wikis were broken
00:14 logmsgbot: mflaschen synchronized php-1.23wmf6/extensions/Thanks/ 'Deploy Thanks bugfix to 1.23wmf6'
00:04 logmsgbot: catrope synchronized php-1.23wmf6/extensions/VisualEditor/ 'Update VisualEditor with cherry-picks'

December 9

23:56 logmsgbot: mwalker synchronized php-1.23wmf6/extensions/Collection/ 'Bisecting collection; found bad commit -- restoring to known bad until I figure out the correct way to pin this at the good commits or find the fix'
23:54 mwalker: bad collection commit is 734de04a62a669022b3e7883301630f1e7947e78 -- now the only thing left is to figure out why!
23:53 logmsgbot: mwalker synchronized php-1.23wmf6/extensions/Collection/ 'Bisecting collection (live), last bad; now on 734de04a62a669022b3e7883301630f1e7947e78'
23:51 logmsgbot: mwalker synchronized php-1.23wmf6/extensions/Collection/ 'Bisecting collection (live), last bad; now on 484305e9ffb19d96155d3957240d74ff92992bb9'
23:51 gwicke: updated Parsoid config to use the API service directly
23:50 logmsgbot: mwalker synchronized php-1.23wmf6/extensions/Collection/ 'Bisecting collection (live), last bad; now on 93282cb9721316ee427c226b8f5c98274e79f4f0'
23:47 logmsgbot: mwalker synchronized php-1.23wmf6/extensions/Collection/ 'Bisecting collection (live), last good; now on afec13ba4aef2f3f79cae0195744656833c5a1a7'
23:45 logmsgbot: mwalker synchronized php-1.23wmf6/extensions/Collection/ 'Bisecting collection (live), now on 488d8daad500f4d9e9b400b6568ba170b54b0a70'
23:40 logmsgbot: mwalker synchronized php-1.23wmf6/extensions/Collection/ 'Reverting collection (live) to b8a4d14eaca507022b011b979bfabe58bbc82361 from a4f97e428d32f4e6bf5adf88d628df9e49b4f075 to establish good revision'
23:01 paravoid: rolling upgrade of nodejs to 0.10 on all wtp* boxes
22:58 logmsgbot: csteipp synchronized php-1.23wmf5/extensions/CentralAuth 'Update CentralAuth to master for ops'
22:47 LeslieCarr: moving bits-lb.esams from amslvs1 to amslvs3
22:44 paravoid: including new c-ares, libv8 & nodejs 0.10 in apt
21:06 gwicke: deployed Parsoid 3191007575830, see https://www.mediawiki.org/wiki/Parsoid/Deployments
18:48 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php 'readd cirrus to wikidata'
18:47 paravoid: powercycling cp1065, unresposinve, no output on console
18:30 logmsgbot: manybubbles finished scap: Updating messages after extension deploys missed it by syncing files and directories
18:06 logmsgbot: manybubbles started scap: Updating messages after extension deploys missed it by syncing files and directories
17:57 logmsgbot: manybubbles synchronized wmf-config/CirrusSearch-common.php 'Finally make Cirrus a beta feature'
17:56 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php 'Set mediawiki.org as primary and prepare for cirrus as a beta feature'
17:49 logmsgbot: manybubbles synchronized php-1.23wmf5/extensions/CirrusSearch 'Update Cirrus to master'
17:49 logmsgbot: manybubbles synchronized php-1.23wmf5/extensions/BetaFeatures 'Update BetaFeatures so Cirrus can be a beta feature'
17:44 logmsgbot: manybubbles synchronized php-1.23wmf6/extensions/CirrusSearch/ 'Update Cirrus to master'
17:44 logmsgbot: manybubbles synchronized php-1.23wmf6/extensions/BetaFeatures/ 'Update BetaFeatures to master so Cirrus can be a beta feature'
15:54 apergos: rebooting cp1065 (kmem_alloc)
15:00 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'touch'
15:00 logmsgbot: hashar synchronized wmf-config/abusefilter.php '96967 (bug 57395) Add AbuseFilter rights to sysops on fiwiki'
14:38 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '100372 add protection level to articles in hewiki'
14:30 logmsgbot: hashar synchronized php-1.23wmf5/.git/refs/heads/wmf/1.23wmf5 'bump Special:Version for aude'
14:07 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '99030 nomcom clean up'
14:03 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '97458 Change favicon for angwiktionary to w icon'
14:01 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '86643 skwikisource: Restrict uploads to sysops'
13:23 logmsgbot: hashar synchronized docroot/bits/favicon 'updating favicons ( b0d4645..0812087 )'
13:18 hashar: updating favicons by updating operations/mediawiki-config.git ( b0d4645..0812087 )
09:36 apergos: restart backend varnish on cp4019, out of memory (fork failed and other nice things)
09:25 apergos: restarted opendj on sanger
09:12 apergos: powercycled sq80, i/o errors for every command, fs complaints on various filesystems in dmesg, smells like controller but it booted ok so...
09:01 apergos: powercycled ms-be1006, unresponsive, lots of "BUG: soft lockup - CPU#0 stuck for 22s!" etc from time it stopped responding (16.00 utc Sunday)
04:38 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool es1010 after upgrade, max_connections lowered during warm up'
03:21 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool es1010 for upgrade'
03:12 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool es1007 after upgrade, max_connections lowered during warm up'
02:49 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Dec 9 02:49:20 UTC 2013
02:43 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool es1007 for upgrade'
02:21 logmsgbot: LocalisationUpdate completed (1.23wmf6) at Mon Dec 9 02:21:01 UTC 2013
02:11 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Mon Dec 9 02:11:34 UTC 2013
02:00 springle: resync labsdb dewiki.pagelinks from upstream

December 8

21:28 paravoid: disabling OSPFv3 on eqiad-esams
03:15 K4-713: updated payments cluster to 8dcb6f5f8f6cb6b24
02:19 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Dec 8 02:19:11 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.23wmf6) at Sun Dec 8 02:08:14 UTC 2013
02:04 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Sun Dec 8 02:04:26 UTC 2013

December 7

12:35 hashar: restarting Jenkins on gallium
03:22 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Dec 7 03:22:21 UTC 2013
02:37 logmsgbot: LocalisationUpdate completed (1.23wmf6) at Sat Dec 7 02:37:08 UTC 2013
02:20 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Sat Dec 7 02:20:06 UTC 2013
00:53 logmsgbot: demon synchronized wmf-config/PoolCounterSettings-eqiad.php 'Removing old cirrus config'
00:53 logmsgbot: demon synchronized wmf-config/PoolCounterSettings-pmtpa.php 'Removing old cirrus config'
00:13 logmsgbot: demon synchronized php-1.23wmf5/extensions/Wikibase/ 'Updating for debugging'

December 6

23:12 K4-713: un-updated payments cluster back to 671411e77f18a9b... again
23:10 K4-713: updated payments to fb96b8bfb1695cd725cdd
22:45 mutante: mediawiki 1.22 has been released - http://dumps.wikimedia.org/mediawiki/1.22/
21:44 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Fix undefined variable'
20:59 logmsgbot: reedy synchronized docroot/bits/favicon/office.ico 'Ib5332726fd02874b25213dd7813b59a684197ac3'
20:56 ori-l: Restarting gmetad on nickel
20:24 K4-713: un-updated payments cluster back to 671411e77f18a9b
20:18 K4-713: updated payments cluster to 9312d81ef0186edfc
20:13 ori-l: Restarting gmetad on nickel for Ib111e53b6
20:06 logmsgbot: ori synchronized wmf-config/throttle.php 'Ifbd6417a2: add throttle exception for queens public lib event'
20:05 logmsgbot: ori updated /a/common to Ifbd6417a2: add throttle exception for queens public lib event
19:28 logmsgbot: yurik synchronized wmf-config/flaggedrevs.php
18:55 logmsgbot: mholmquist synchronized php-1.23wmf6/extensions/UploadWizard/resources/mw.UploadWizardDetails.js 'Fix UploadWizard for IE8'
18:54 logmsgbot: mholmquist synchronized php-1.23wmf5/extensions/UploadWizard/resources/mw.UploadWizardDetails.js 'Fix UploadWizard for IE8'
17:37 logmsgbot: reedy synchronized php-1.23wmf6/extensions/Wikibase 'I7cf7af5525f3223dbd044e7676b5c0255b45928c'
17:37 logmsgbot: reedy synchronized php-1.23wmf6/extensions/DataValues 'I7cf7af5525f3223dbd044e7676b5c0255b45928c'
17:34 logmsgbot: reedy synchronized php-1.23wmf5/extensions/Wikibase 'I06994427b780cd0b66dc0b0279045df7699aef1c'
10:28 logmsgbot: hashar synchronized php-1.23wmf6/extensions/ProofreadPage 'ProofreadPage bug fixes 99627'
10:16 logmsgbot: hashar synchronized php-1.23wmf5/extensions/ProofreadPage 'ProofreadPage bug fixes 99522'
10:02 apergos: shot and restarted gmond processes on hooft
04:27 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool es1001 after upgrade, max_connections lowered during warm up'
03:33 logmsgbot: reedy synchronized multiversion/MWMultiVersion.php
02:51 springle: set es1004 max_connections back to default (should have been reset previously, hence dberror log noise)
02:46 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool es1001 for upgrade'
02:29 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'direct vslow query group to LB=0 snapshot slaves'
02:27 logmsgbot: tstarling synchronized wmf-config/trusted-xff.cdb
02:22 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Dec 6 02:22:06 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.23wmf6) at Fri Dec 6 02:15:21 UTC 2013
02:02 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Fri Dec 6 02:02:14 UTC 2013
00:27 mwalker: updating payments from 2afd91f91622abb369e05174d1dd244a2ff3adbd for amazon issues
00:18 logmsgbot: maxsem synchronized php-1.23wmf6/extensions/DataValues 'touch'
00:04 logmsgbot: maxsem synchronized php-1.23wmf6/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/99551'

December 5

23:37 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Dec 5 23:37:18 UTC 2013
23:22 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php 'Tweaked $wgJobQueueAggregator to use redis job servers and have failover'
23:22 logmsgbot: aaron synchronized wmf-config/jobqueue-pmtpa.php 'Tweaked $wgJobQueueAggregator to use redis job servers and have failover'
22:54 logmsgbot: LocalisationUpdate completed (1.23wmf6) at Thu Dec 5 22:54:59 UTC 2013
22:21 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Thu Dec 5 22:21:52 UTC 2013
21:22 logmsgbot: catrope synchronized php-1.23wmf5/extensions/VisualEditor/modules/oojs-ui/oojs-ui.js 'touch'
21:19 ottomata: stopping puppet on cp1046 to troubleshoot some ganglia stuff
20:58 logmsgbot: reedy synchronized docroot and w
20:45 apergos: over the last while, restarted swift-object-server on ms-be100* except for 1008, left that for poking at
20:44 logmsgbot: reedy updated /a/common to Ifae924950: Remofve 1.22wmf15 through 1.22wmf19
20:44 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: phase1 wikis to 1.23wmf6
20:39 logmsgbot: reedy synchronized php-1.23wmf6
20:18 logmsgbot: reedy synchronized docroot and w
20:16 logmsgbot: reedy updated /a/common to Ida4a0d980: Everything else to 1.23wmf5
20:09 ori-l: increase in load coincided with sync of wmf6 to apaches and subsided on roll-back, but wmf6 was not enabled anywhere at the time of syncing
20:08 ori-l: image scalers started overloading at 18:56, cause appears to be a spike of convert jobs exceeding limits & getting killed; swift-backend.log on fluoine has lots of InvalidResponseException; syslog on swift has "swift-object-se: page allocation failure".
20:06 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: everything else to 1.23wmf5
20:04 logmsgbot: reedy updated /a/common to Ifda85f2ce: Add/update symlinks
19:41 Coren: Quick bounce of labstore1001 (kernel tweak)
19:38 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.23wmf5 for now
19:37 logmsgbot: reedy finished scap: testwiki to 1.23wmf6, build l10n cache and rebuild for 1.23wmf5
19:14 logmsgbot: reedy started scap: testwiki to 1.23wmf6, build l10n cache and rebuild for 1.23wmf5
19:08 logmsgbot: reedy updated /a/common to I18d8a12b1: repool es1002 after upgrade
18:56 logmsgbot: reedy synchronized php-1.23wmf6 'Staging'
17:38 logmsgbot: catrope synchronized php-1.23wmf5/resources/
17:38 logmsgbot: catrope synchronized php-1.23wmf5/resources/oojs/oojs.js 'I synced you before, now start existing'
17:30 logmsgbot: catrope synchronized php-1.23wmf5/extensions/MultimediaViewer 'oojs fixes'
17:30 logmsgbot: catrope synchronized php-1.23wmf5/extensions/VisualEditor 'oojs fixes'
17:29 logmsgbot: catrope synchronized php-1.23wmf5/resources/Resources.php 'Add oojs'
17:29 logmsgbot: catrope synchronized php-1.23wmf5/resources/oojs/oojs.js 'Add oojs'
16:54 ottomata: deployed varnishkafka 1.0.0-1 to cp1046,cp3011,cp4011
16:07 ottomata: added varnishkafka 1.0.0-1 to apt
15:43 ottomata: added librdkafka 0.8.1-1~precise1 to apt
14:41 bblack: mobile esams caches restarted again on new packages w/ new jemalloc, puppet's enabled there and should be stable
14:17 Coren: bouncing labstore1001 (kernel upgrade)
13:28 logmsgbot: hashar synchronized php-1.23wmf5/extensions/ProofreadPage 'Update Proofreadpage 99042'
13:25 bblack: cp3012 running test varnish pkg w/ jemalloc 3.4.1
06:30 bblack: cp301[123] puppet freshness is me, please leave them disabled
06:15 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool es1002 after upgrade, max_connections lowered during warm up'
03:34 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool es1002 for upgrade'
03:23 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Dec 5 03:23:22 UTC 2013
03:03 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool es1003 after upgrade, max_connections lowered during warm up'
02:38 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Thu Dec 5 02:38:26 UTC 2013
02:22 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Thu Dec 5 02:22:00 UTC 2013
01:07 awight: updated crm from daff8fa245084ff372490d1d6d85313078943fae to 0e7f22692b19eaa8deb3eae0d34ec077b7ab22d7
00:58 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool es1003 for upgrade'

December 4

23:09 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: no actual version changes, just rebuilding to test code changes
23:06 logmsgbot: demon synchronized multiversion/ 'cdb changes for hhvm support (I73195536)'
22:19 logmsgbot: manybubbles synchronized php-1.23wmf4/extensions/CirrusSearch/ 'Update CirrusSearch to speed up indexing'
22:15 logmsgbot: manybubbles synchronized php-1.23wmf5/extensions/CirrusSearch/ 'Update CirrusSearch to speed up indexing'
21:48 awight: updated deduper from 922b3aa6b797a2338fd409be4ffd90720a8de601 to 20f709349fb6ab987e689c5da64f392fa1e4d488
21:18 gwicke: deployed Parsoid 0ac82a2
19:05 logmsgbot: yurik synchronized flaggedrevs.dblist
19:02 logmsgbot: yurik synchronized wmf-config
18:57 yurik: yurik@tin:/a/common/php-1.23wmf5$ mwscript sql.php --wiki=metawiki extensions/FlaggedRevs/backend/schema/mysql/FlaggedRevs.sql
18:32 logmsgbot: yurik synchronized wmf-config/InitialiseSettings.php
18:22 logmsgbot: yurik synchronized php-1.23wmf5/extensions/ZeroRatedMobileAccess/
18:15 logmsgbot: yurik synchronized php-1.23wmf4/extensions/ZeroRatedMobileAccess/
16:07 ottomata: apt now contains kafka-0.8.0-1 .deb
15:05 apergos: intermittent ganglia reporting from db1015, restarted gmond there, then on db1021 (aggregator)
14:40 MaxSem: Nuking HerculeBot's watchlist on frwiki at operator's request, too large to modify from web
13:59 paravoid: nas1001-b cf giveback
13:18 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'touch'
13:17 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'Allow exporting files as gettext in Translate 99070 bug 40341'
10:30 apergos: running Wikibase/repo/maintenance/dumpJson.php against wikidata in screen as ariel n terbium; if it causes problems feel free to shoot it
09:35 paravoid: short esams outage between 09:30-09:33 UTC due to crappy network vendor issues
05:01 springle: add index on db1048 otrs.ticket_history.article_id to stop daily slave lag
04:55 Tim: running sync-apache to deploy I730ec0bf8c388cada17b46497d87146f1e4ded1a
02:54 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Dec 4 02:54:47 UTC 2013
02:21 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Wed Dec 4 02:21:41 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Wed Dec 4 02:15:31 UTC 2013
01:10 logmsgbot: ori synchronized php-1.23wmf5/includes/resourceloader/ResourceLoaderModule.php 'Ifa9088c11: resourceloader: Make sure hashmtime cache key is different by language'
01:09 logmsgbot: ori synchronized php-1.23wmf4/includes/resourceloader/ResourceLoaderModule.php 'Ifa9088c11: resourceloader: Make sure hashmtime cache key is different by language'
00:44 logmsgbot: maxsem synchronized mobilelanding.php
00:43 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
00:30 logmsgbot: catrope synchronized php-1.23wmf5/extensions/VisualEditor/VisualEditor.php 'Blacklist IE11'
00:30 logmsgbot: catrope synchronized php-1.23wmf4/extensions/VisualEditor/VisualEditor.php 'Blacklist IE11'
00:21 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Add plumbing for wmgVisualEditorInContentNamespaces'
00:19 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Add wmgVisualEditorInContentNamespaces and set it to false on svwiktionary'
00:14 logmsgbot: demon synchronized cirrus.dblist 'Fix test2wiki to have Cirrus as primary'

December 3

22:38 mutante: graceful'ed a couple more mw*/srv* in pmtpa to make them all pickup the same config
22:27 mutante: graceful'ed apaches on mw1078,mw1111,mw1176,mw1194,mw1096,mw1107 (dsh out of sync again? sigghh)
22:09 mutante: sync-apache, graceful-all for zero/m redirect (gerrit 97115)
21:41 Krinkle: Reloading zuul to deploy I889600fc65d8f
21:32 Krinkle: Reloading zuul to deploy Iceb6b4016c
21:04 mwalker: updating payments.wm.o from 7d49c5a8c63dd7c8e282b3d83b8cd9fca5e4b143 for translations and improved error handling
20:49 mwalker: updating fundraising civicrm for thankyou templates (from 9c82766cf4265a9e9831f3eb618c233f4758cc68)
20:47 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'If02d8506c: Enable module storage by default'
20:46 logmsgbot: ori updated /a/common to If02d8506c: Enable module storage by default
20:44 mutante: new mediawiki-1.22rc3 is out
19:59 mutante: built and importing php-luasandbox 1.8 to apt repo
19:59 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisources back to 1.23wmf5
19:53 logmsgbot: reedy synchronized php-1.23wmf5/extensions/ProofreadPage/
19:38 ottomata: updated webstatscollector to 0.2-1 on erbium and gadolinium
19:35 logmsgbot: reedy synchronized php-1.23wmf5/extensions/Wikibase
19:32 ottomata: updated webstatscollector package in apt to 0.2-1
19:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource back to 1.23wmf4 as proofreadpage is broken
19:25 logmsgbot: reedy synchronized php-1.23wmf5/extensions/ProofreadPage/
19:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non Wikipedias to 1.23wmf5
18:45 ori-l: restarting gmetad on nickel to clear dead hosts
15:15 paravoid: depool/restart/pool cycle for nginxes
14:15 bblack: cp3012 re-pooled
14:06 bblack: rebooting cp3012.esams
14:01 bblack: depooling cp3012.esams temporarily to fix various ugly issues
12:49 akosiaris: uploaded new jenkins-debian-glue on apt.wikimedia.org, version 0.7.1-6-gf618f4d
11:37 paravoid: stopping varnish backend on cp3012, crashing every 30s
08:27 logmsgbot: nikerabbit synchronized php-1.23wmf4/extensions/Translate/
08:16 logmsgbot: nikerabbit synchronized php-1.23wmf5/extensions/Translate/
05:32 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool es1004 after upgrade, max_connections lowered during warm up'
04:55 K4-713: updated payments cluster to 7d49c5a8c63dd7c8e2
03:45 springle: stopping es1004 mysqld for precise & mariadb upgrade, plus reboot
03:19 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Dec 3 03:19:58 UTC 2013
02:43 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool es1004 for upgrade'
02:37 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Tue Dec 3 02:37:37 UTC 2013
02:17 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Tue Dec 3 02:17:50 UTC 2013
01:13 cmjohnson1: rebooting sq37
00:05 logmsgbot: mwalker synchronized wmf-config/CommonSettings.php
00:05 logmsgbot: mwalker synchronized wmf-config/InitialiseSettings.php
00:04 logmsgbot: mwalker updated /a/common to I9e2f24923: Changing banner expiration to 10 months

December 2

23:13 logmsgbot: demon synchronized php-1.23wmf4/extensions/Elastica
23:04 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Cirrus wikis get searchupdate (take 2)'
22:33 logmsgbot: catrope synchronized php-1.23wmf5/extensions/VisualEditor 'Update VE for cherry-pick'
22:33 logmsgbot: catrope synchronized php-1.23wmf4/extensions/VisualEditor 'Update VE for cherry-pick'
22:30 mwalker: updating civicrm from 82af6188f5e019ab23636e78719e17c4cb838859 for new thankyou message
22:20 logmsgbot: reedy updated /a/common to Idc406aa68: Created mobile portal m.wikipedia.org and zero.wikipedia.org
22:19 logmsgbot: reedy synchronized wmf-config/interwiki.cdb 'Updating interwiki cache'
22:06 logmsgbot: catrope synchronized visualeditor.dblist 'Enable VisualEditor on svwiktionary and sewikimedia'
22:05 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Only activate VisualEditor in the User namespace on svwiktionary'
22:04 logmsgbot: catrope synchronized visualeditor-default.dblist 'Enable VisualEditor by default on 102 wikis'
21:33 RobH: ignore any alerts for testsearch1XXX, I just decommissioned them, but icinga hasn't updated quite yet
21:02 logmsgbot: yurik synchronized docroot and w
20:46 logmsgbot: maxsem synchronized php-1.23wmf5/extensions/MobileFrontend/ 'https://gerrit.wikimedia.org/r/#/c/95636/'
20:45 logmsgbot: maxsem synchronized php-1.23wmf4/extensions/MobileFrontend/ 'https://gerrit.wikimedia.org/r/#/c/95636/'
20:10 logmsgbot: yurik synchronized w/mobilelanding.php
20:02 logmsgbot: yurik synchronized php-1.23wmf5/extensions/ZeroRatedMobileAccess/
19:53 logmsgbot: yurik synchronized php-1.23wmf4/extensions/ZeroRatedMobileAccess/
19:01 logmsgbot: demon synchronized php-1.23wmf4/extensions/Elastica 'Fix missing code in Elastica on 1.23wmf4 wikis'
18:58 K4-713: updated fraud rules on payments
18:53 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Search updates back off for Cirrus wikis :('
18:35 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Dec 2 18:35:42 UTC 2013
18:34 ottomata: added logster deb, installed on cp3003 for testing, will puppetize shortly
18:30 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Search updates back on for Cirrus'
17:47 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Mon Dec 2 17:47:29 UTC 2013
17:31 bblack: cp301[12].esams - puppet temporarily disabled, custom crash handler vmod in place to try to catch an error in the next couple of hours
17:31 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Search update off for cirrus wikis'
17:31 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Mon Dec 2 17:31:11 UTC 2013
17:30 logmsgbot: demon updated /a/common to I507a72cca: Fix submodule reference change that snuck into If5b3a27a
17:21 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Cirrus on all the wikis (that had it before)'
17:09 logmsgbot: demon synchronized php-1.23wmf5/extensions/CirrusSearch 'Cirrus to master'
17:07 logmsgbot: demon synchronized php-1.23wmf4/extensions/CirrusSearch 'Cirrus to master'
17:06 logmsgbot: demon synchronized wmf-config/CirrusSearch-common.php
16:33 paravoid: rebooting cp1065, usual XFS deadlock
12:35 Jeff_Green: stopped mysql on db1008 to clone a database
07:05 springle: upgrade/reboot db1046 m2 slave
06:53 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool pc1003 after upgrade, max_connections lowered during warm up'
06:28 paravoid: fixed ganglia for misc eqiad (possibly others); see Icc5376505
06:04 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool pc1003 for upgrade'
02:51 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool pc1002 after upgrade, max_connections lowered during warm up'
02:37 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Dec 2 02:37:48 UTC 2013
02:23 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool pc1002 for upgrade'
02:14 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Mon Dec 2 02:14:57 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Mon Dec 2 02:08:15 UTC 2013
01:16 springle: restarting labsdb1002 mysqld processes with 25% smaller buffer pools. kernel OOM killer striking. needs investigation

December 1

06:23 apergos: sq37 hardware errors, probably controller, rt #6418
06:04 apergos: pwercycled sq37, was giving 'i/o error' to all commands
02:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Dec 1 02:36:06 UTC 2013
02:13 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Sun Dec 1 02:13:36 UTC 2013
02:07 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Sun Dec 1 02:07:54 UTC 2013

November 30

05:54 paravoid: rebooting cp1052 with the same kernel (control)
02:47 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Nov 30 02:47:22 UTC 2013
02:18 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Sat Nov 30 02:18:36 UTC 2013
02:10 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Sat Nov 30 02:10:13 UTC 2013

November 29

21:09 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'Id9c7321b8: Add a MassMessage-related user group on Meta'
21:08 logmsgbot: ori updated /a/common to Id9c7321b8: Add a MassMessage-related user group on Meta
20:18 paravoid: rebooting cp1065 with new kernel
19:26 paravoid: "swapoff -a" on all cache_text to deal with strange kernel issue with kswapd dropping the whole page cache on memory pressure
14:49 paravoid: restarted gmond on ms-fe1001/2, both were stuck 6h ago and we lost all swift eqiad's metrics for that period
11:12 Reedy: Created EducationProgram tables on arwiki
05:51 Tim: on cp1052 and cp1053: tweaked /proc/sys/net/core/rmem_default to see if that fixes the observed massive gmond packet loss
02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Nov 29 02:07:55 UTC 2013
02:02 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Fri Nov 29 02:02:25 UTC 2013
02:01 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Fri Nov 29 02:01:44 UTC 2013
01:22 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool pc1001 after upgrade, max_connections lowered during warm up'
00:25 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool pc1001 for package upgrade'

November 28

10:49 apergos: turned off logging for parsoid ( https://gerrit.wikimedia.org/r/#/c/98082/ ), old logs remain in place for folks to examine
10:06 apergos: stack traces filling up parsoid nohup.out logs (sveral gigs in only a few minutes once the parsoid gets into that state), sample on wtp1010 in /var/lib/parsoid/nohup.out.errors
08:34 apergos: and wtp1023
08:29 apergos: /var/lib/parsoid/nohup.out on wtp 1005,11,12 was 6gb or more, causing / on these boxes to fill; moved it, restarted parsoid, removed it
07:16 apergos: powercycled sq80
05:41 logmsgbot: ori synchronized wmf-config/CommonSettings.php 'Icdaa4c1b5: Configure parser cache databases in db-$realm file (3/3)'
05:41 logmsgbot: ori synchronized wmf-config/db-pmtpa.php 'Icdaa4c1b5: Configure parser cache databases in db-$realm file (2/3)'
05:40 logmsgbot: ori synchronized wmf-config/db-eqiad.php 'Icdaa4c1b5: Configure parser cache databases in db-$realm file (1/3)'
05:37 logmsgbot: ori updated /a/common to Icdaa4c1b5: Configure parser cache databases in db-$realm file
03:37 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool slaves after package upgrade, (lvm snapshot boxes only, LB=0)'
03:16 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool slaves for package upgrade'
02:42 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Nov 28 02:42:58 UTC 2013
02:29 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'slaves to full steam after package upgrade'
02:15 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Thu Nov 28 02:15:36 UTC 2013
02:09 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Thu Nov 28 02:09:38 UTC 2013
01:17 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'warm up slaves after package upgrade'
01:02 ori-l: started rsync of graphite data (~400gb) from professor.pmtpa to tungsten.eqiad
00:39 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool slaves for package upgrade'

November 27

19:50 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Fixes for Flow config, no-op in prod'
19:49 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Fixes for Flow config, no-op in prod'
18:12 paravoid: kill -9 gdb on cp3012, attached to varnish frontend
11:28 ori-l: faidon switched gdash.wm.o from professor.pmtpa -> tungsten.eqiad behind misc-varnish & rebooted ssl1 in tampa
11:11 apergos: ssl1 rebooted itself about 15 mins ago, no idea why
10:20 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'db1019 (s3) back to full weight in the pool'
10:19 logmsgbot: ariel updated /a/common to If5ebd6194: db1019 (s3) back to full weight in pool
10:08 apergos: shot some old puppet processes hogging memory on db9 (from march and earlier)
09:48 apergos: there was no mount /srv/pagecounts on labstore4, so rsync to /exp/pagecounts wrote to and filled /; did the mkdir and now things seem ok
08:00 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'warm up db1019 (s3) aftr lvm resize'
07:59 logmsgbot: ariel updated /a/common to I50354e622: warm up db1019 (s3) after lvm resize
07:38 apergos: rebooting db1019 after kernel upgrade, fix for broken xfs_growfs
07:02 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'depool db1019 (s3) temporarily for lvm resize'
07:01 logmsgbot: ariel updated /a/common to I4372bb602: depool db1019 (s3) temporarily for lvm resize
02:40 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Nov 27 02:40:52 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Wed Nov 27 02:15:19 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Wed Nov 27 02:08:28 UTC 2013
00:32 springle: stopping replication on sanitarium db1054:3308 and labsdb1002:3308 while restoring dewiki to labs

November 26

22:58 mutante: deploying gerrit 92925 & 91124 (apache-config), makes /entity/ URLs on wikidata 303 and removes non-existent noncom wiki
21:15 mutante: praseodymium, cerium, xenon: disabled icinga notifications and scheduled 1yr downtime for host and all services per gwicke, they are test hosts and not prod.
21:04 bblack: killed attached gdb on cp3012's varnishd frontend, which restarted it...
20:21 logmsgbot: csteipp synchronized php-1.23wmf4/extensions/TimedMediaHandler 'bug56699'
20:16 logmsgbot: csteipp synchronized php-1.23wmf5/extensions/TimedMediaHandler 'bug56699'
20:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Not wmf5 day today
19:33 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Revert to 1.23wmf4 on wikisources
19:23 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All non wikipedias to 1.23wmf5
19:20 MaxSem: Reindexing GeoData
19:16 logmsgbot: reedy updated /a/common to Idb0e7956e: slaves to full steam
14:04 hashar: jenkins / zuul: cherry picked a change from upstream https://review.openstack.org/#/c/43978/ and deployed it in production. 1e3adfd..1b47026 (tagged wmf-deploy-20131126). That should slightly speed up Zuul when it receives a change, previously it would issue git remote update whenever an event is received which takes a few seconds for mediawiki/core.
08:00 ori-l: re-pooled ssl1003
07:56 ori-l: depooling ssl1003 for quick test of puppet config
06:22 apergos: pwercycled labstore1001, unreachable, nothing on mgmt console
05:59 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'slaves to full steam after package upgrade'
05:59 paravoid: applying workaround for Ganglia XSS https://github.com/ganglia/ganglia-web/issues/218
05:32 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'warm up slaves after package upgrade'
04:51 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool slaves for package upgrade'
04:05 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool slaves for package upgrade'
03:29 Tim: deploying varnish normalize_path (I687f6773)
03:14 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Nov 26 03:14:08 UTC 2013
02:25 Tim: deploying new generated redirects.conf
02:24 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Tue Nov 26 02:24:12 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Tue Nov 26 02:16:11 UTC 2013
02:03 awight: updated crm from 9979fe9a0f456499ff5be261aa41ca601a88dc13 to 82af6188f5e019ab23636e78719e17c4cb838859
01:59 awight: rollback crm from 3e7f0d57023e98e6be0e1d7539b2abd04b72ad99 to 9979fe9a0f456499ff5be261aa41ca601a88dc13
01:58 awight: updated crm from 9979fe9a0f456499ff5be261aa41ca601a88dc13 to 3e7f0d57023e98e6be0e1d7539b2abd04b72ad99
01:57 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'slaves to full steam'
01:22 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'warm up slaves after package upgrade'
00:43 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool slaves for packages upgrade'

November 25

23:30 logmsgbot: mwalker synchronized php-1.23wmf4/extensions/CentralNotice 'Updating CentralNotice to master (mostly for caching reasons'
23:11 logmsgbot: ori synchronized php-1.23wmf4/extensions/TemplateData 'Update TemplateData to master for I0782ea669'
23:11 logmsgbot: ori synchronized php-1.23wmf4/tests/phpunit/includes/ArticleTablesTest.php 'Avoid extra parsing in prepareContentForEdit() I2c34baaf8'
23:11 logmsgbot: ori synchronized php-1.23wmf4/includes/WikiPage.php 'Avoid extra parsing in prepareContentForEdit() I2c34baaf8'
22:22 Jamesofur: upload replacement arbcom election config
21:59 ori-l: finished updating / restarting eqiad ssls
21:37 ori-l: Depooling ssl1004 to restart nginx
21:22 springle: restarting sanitarium mysqld processes, db1053, db1054, db1057
21:06 ori-l: Restarting nginx on ssl1008
20:49 ori-l: Depooling ssl1008 to test 96961
20:43 ori-l: Also disabling puppet on cp400*
20:41 ori-l: Disabling Puppet on ssl* in preparation for merging https://gerrit.wikimedia.org/r/#/c/96961/
20:10 LeslieCarr: text-lb.esams, bits-lb.esams and more (amslvs1/3) are now load shared amongst amslvs1 and amslvs3 instead of just amslvs1
18:48 logmsgbot: mholmquist synchronized php-1.23wmf5/extensions/UploadWizard/resources/ext.uploadWizard.uploadCampaign.list.css
18:46 logmsgbot: mholmquist synchronized php-1.23wmf4/extensions/UploadWizard/resources/ext.uploadWizard.uploadCampaign.list.css
16:20 hashar: jenkins : installed curl on lanthanum.eqiad.wmnet puppet change is 97526
13:39 hashar: jenkins : restarted slave daemon on gallium.wikimedia.org
13:38 hashar: jenkins : restarted slave daemon on lanthanum.eqiad.wmnet
13:38 hashar: Jenkins: upgrading openjdk 6 on gallium and lanthanum
07:43 springle: imported de-debianized mariadb 5.5.34 debs into rerepro on brewster
04:45 logmsgbot: springle synchronized wmf-config/db-eqiad.php
04:10 springle: mariadb 5.5.34 live-fire test on db1045
04:09 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'warm up db1045'
03:58 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1045 for upgrade'
03:05 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Nov 25 03:05:19 UTC 2013
02:26 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Mon Nov 25 02:26:16 UTC 2013
02:14 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Mon Nov 25 02:14:09 UTC 2013
01:41 logmsgbot: springle synchronized wmf-config/db-eqiad.php
01:20 springle: mariadb 5.5.34 live-fire test on db1037
01:15 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'warm up db1037'
00:26 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool db1050. depool db1037'

November 24

10:04 logmsgbot: ori synchronized wmf-config/throttle.php 'Icf844849e: (bug 57345) Raise account creation limit for cawiki GLAM event'
10:03 logmsgbot: ori updated /a/common to Icf844849e: (bug 57345) Raise account creation limit for cawiki GLAM event
09:58 ori-l: (...thereby fixing the global job queue length graph)
09:55 ori-l: hume: ln -s /usr/local/apache/common-local /a/common (previously clobbered by Iaf19f0e8c)
09:21 Nemo_bis: global job queue length graph is still dead https://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Miscellaneous%20pmtpa&h=hume.wikimedia.org&v=823574&m=Global_JobQueue_length&r=hour&z=default&jr=&js=&st=1365625056&z=large
06:44 paravoid: wikitech broken with commit 716887 (Cleanup puppetmaster's apache Listen ports); workarounded manually with /etc/apache2/conf.d/ports-wikitech.conf
06:27 paravoid: force-rebooting cp3012, kernel deadlocked
06:02 ori-l: (cp3012) CPU wait, not wio.
05:46 ori-l: cp3012: kern.log, every 2s: XFS: possible memory allocation deadlock in kmem_alloc
05:43 ori-l: cp3012: any attempt to access /proc/17722 immediately hangs ssh
05:31 ori-l: cp3012 wio through the roof; process table filled with '/bin/ps axwwo stat uid pid ppid vsz rss pcpu etime comm args', uid icinga
02:50 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Nov 24 02:50:44 UTC 2013
02:20 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Sun Nov 24 02:20:50 UTC 2013
02:11 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Sun Nov 24 02:11:21 UTC 2013

November 23

14:04 springle: testing 5.5.34 mariadb.org debs on db60
10:13 apergos: ...on zirconium
10:13 apergos: stopped etherpadlite, removed all but the last 5000 lines of its log file, (root running out of space), restarted
03:12 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Nov 23 03:12:24 UTC 2013
02:26 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Sat Nov 23 02:26:18 UTC 2013
02:14 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Sat Nov 23 02:14:03 UTC 2013
00:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable OAuth on fishbowl wikis'
00:48 logmsgbot: reedy synchronized php-1.23wmf5/extensions/Wikibase/client/ 'Fix InfoAction fatal'
00:48 logmsgbot: reedy synchronized php-1.23wmf5/extensions/MobileFrontend/ 'Fix history fatal'
00:47 logmsgbot: reedy synchronized php-1.23wmf4/extensions/MobileFrontend/ 'Fix history fatal'
00:34 aschulz|laptop: Manually fixed img_metadata for File:Zentralbibliothek_Zürich_-_Heinrich_Bullingers_Westerhemd_-_000012135.tif by forcing /bin/identify usage
00:22 logmsgbot: ori synchronized php-1.23wmf4/extensions/PagedTiffHandler 'Update PagedTiffHandler to master for I52fe2ec25 / bug 57359'
00:18 logmsgbot: ori synchronized php-1.23wmf5/extensions/PagedTiffHandler 'Update PagedTiffHandler to master for I52fe2ec25 / bug 57359'

November 22

21:35 logmsgbot: ori synchronized wmf-config/CommonSettings.php 'I615e2b210: correct schema ID of Schema:Echo'
21:34 logmsgbot: ori updated /a/common to I615e2b210: Schema:Echo has outdated revision id
21:09 mutante: mediawiki 1.22.0rc3 is out http://dumps.wikimedia.org/mediawiki/1.22/
20:04 Jeff_Green: dist-upgrade aluminium and reboot
19:56 mutante: starting apache on alumunium
19:54 mutante: killed all processes by user sahar on aluminium
19:27 logmsgbot: reedy finished scap: Rebuild localisation cache to update ce
19:10 logmsgbot: reedy started scap: Rebuild localisation cache to update ce
17:21 logmsgbot: reedy synchronized php-1.23wmf5/extensions/EducationProgram/includes/Events/EditEventCreator.php
17:19 logmsgbot: reedy synchronized php-1.23wmf4/extensions/EducationProgram/includes/Events/EditEventCreator.php
17:10 logmsgbot: reedy synchronized php-1.23wmf5/languages/messages/MessagesCe.php 'Update ce language special page aliases'
17:09 logmsgbot: reedy synchronized php-1.23wmf4/languages/messages/MessagesCe.php 'Update ce language special page aliases'
12:59 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Ic36cd656233dfc959c92a33bade794cc5c1e1bd2'
11:21 akosiaris: neon booted and seems ok. running updatedb manually
10:53 akosiaris: rebooting neon for fsck
09:19 apergos: snmptt stuck waiting for a submit_check_result that had exited (or died), restarted it to clear the snmptt spool queue
08:14 apergos: rebooting neon again, found four filenames with deleted inode but side effect is back in r/o for /
07:06 apergos: rebooting neon, no underlying errors found except for the initial ext3 'deleted inode reference' which caused / to be remounted r/o
06:53 apergos: neon back to read-only root filesystem, investigating
03:23 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Nov 22 03:23:33 UTC 2013
02:35 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Fri Nov 22 02:35:51 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Fri Nov 22 02:16:36 UTC 2013
01:41 logmsgbot: csteipp synchronized wmf-config/InitialiseSettings.php
01:40 logmsgbot: csteipp synchronized wmf-config/CommonSettings.php
01:37 logmsgbot: csteipp finished scap: Really send the new messages this time..
01:16 logmsgbot: csteipp started scap: Really send the new messages this time..
00:56 logmsgbot: ori synchronized php-1.23wmf4/resources/startup.js 'touch'
00:50 logmsgbot: csteipp finished scap: Updating WikimediaMessages to master for OAuth message
00:50 logmsgbot: ori synchronized php-1.23wmf4/extensions/MobileFrontend/javascripts/loggingSchemas/MobileWebInfobox.js 'Cherry-pick I3efc1fa64'
00:46 logmsgbot: ori synchronized php-1.23wmf4/resources/startup.js 'touch'
00:36 logmsgbot: csteipp started scap: Updating WikimediaMessages to master for OAuth message
00:35 logmsgbot: ori synchronized php-1.23wmf4/extensions/MobileFrontend/javascripts/loggingSchemas/MobileWebInfobox.js 'Cherry-pick I3efc1fa64'
00:09 logmsgbot: demon synchronized wmf-config/PoolCounterSettings-eqiad.php 'New pool counter settings for Cirrus'
00:08 logmsgbot: demon synchronized wmf-config/PoolCounterSettings-pmtpa.php 'New pool counter settings for Cirrus'
00:07 Reedy: Built texvc for 1.23wmf5

November 21

23:33 ori-l: Purging tifs older than one hour on app servers
23:06 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable BetaFeatures everywher'
23:03 logmsgbot: reedy updated /a/common to I3ac8621d9: Removed ZERO namespace protection in META
23:02 logmsgbot: reedy synchronized wmf-config 'last few config updates'
22:32 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: phase1 wikis to 1.23wmf5
22:31 logmsgbot: reedy synchronized php-1.23wmf5/extensions/MobileFrontend/
22:21 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.23wmf4
21:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'rv'
21:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wiki -'
21:26 logmsgbot: reedy updated /a/common to I4360e8a34: Enable BetaFeatures everywhere except Wikipedias
21:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable BetaFeatures on all non wikipedias'
21:16 logmsgbot: reedy synchronized php-1.23wmf5/extensions/Wikibase
20:59 logmsgbot: reedy updated /a/common to I6433f2b0d: testwiki to 1.23wmf5
20:57 cmjohnson1: sq48 powering down for decom
20:47 cmjohnson: sq44 powering down for decom
20:24 Reedy: mw117 is also segfaulting quite a bit
19:56 paravoid: depooling mw1183
19:56 Reedy: mw1183/10.64.32.53 is having masses of segfaults
19:52 Reedy: Created BetaFeatures extension tables on all wikis
19:46 logmsgbot: ori synchronized docroot/noc 'I7ea8302f2: noc.wm.o: add images referenced by stylesheets.'
19:45 logmsgbot: ori updated /a/common to I7ea8302f2: noc.wm.o: add images referenced by stylesheets.
19:43 logmsgbot: reedy Finished syncing Wikimedia installation... : testwiki to 1.23wmf5 and build l10n cache
19:33 logmsgbot: ori synchronized docroot/noc 'Ia451d47bd: noc.wm.o: use local CSS files rather than point to svn.wm.o'
19:32 logmsgbot: ori updated /a/common to Ia451d47bd: noc.wm.o: use local CSS files rather than point to svn.wm.o
19:27 logmsgbot: reedy Started syncing Wikimedia installation... : testwiki to 1.23wmf5 and build l10n cache
19:24 logmsgbot: reedy synchronized docroot and w
19:21 logmsgbot: reedy synchronized php-1.23wmf5 'Staging code'
19:02 paravoid: restored service for svn.wm.org
18:48 cmjohnson: shutting down vanadium to replace failed hard drive
17:19 ^d: reloaded zuul to pick up Ibc354db7
15:34 springle: bounced pc1001 mysqld. massive spike of writes exhausted innodb txn log slots and wouldn't be killed
13:55 hashar: restarted Zuul, had some jobs stuck in its queue (96762
09:16 paravoid: reenable ospfv3 between esams-eqiad
08:44 ori-l: Synced wmf-config/InitialiseSettings.php "I5e8317db6: Enable recursion-guard log bucket to investigate bug 54193"
08:44 ori-l: with neon down, scap / syncs not logged to irc
08:04 springle: rebooted neon, ext3 root fs read only
07:52 akosiaris: updated all remaining references to sockpuppet and stafford as puppetmaster on wikitech
04:58 dr0ptp4kt: Wikipedia Zero BDD script run
03:20 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Nov 21 03:20:28 UTC 2013
02:32 logmsgbot: LocalisationUpdate completed (1.23wmf3) at Thu Nov 21 02:32:33 UTC 2013
02:17 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Thu Nov 21 02:17:19 UTC 2013
01:39 springle: enwiki revision table schema change test run db60
01:24 springle: depool db60, db54, db34, db31, db65, db47, db37 for decom and/or shipping. promote db68, db74, db72, db71, db69 to pmtpa shard masters
01:18 logmsgbot: springle synchronized wmf-config/db-pmtpa.php
01:08 awight: updated crm from 326854db84becacf808add24a75ca27d1358c5bc to 9979fe9a0f456499ff5be261aa41ca601a88dc13
00:29 AaronSchulz: Resynced terbium /usr/local/apache/common-local which was totally empty
00:18 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'If961f9365: add dewiki help page for Echo'
00:17 logmsgbot: ori updated /a/common to If961f9365: add dewiki help page for Echo
00:09 ori-l: terbium: symlinked /a/common to /usr/local/apache/common. We should really get rid of the distinction.

November 20

23:40 logmsgbot: yurik synchronized php-1.23wmf4/extensions/ZeroRatedMobileAccess/
23:37 ori-l: puppet.log on terbium: could not set file on ensure: No such file or directory - /usr/local/apache/common/php/extensions/FlaggedRevs/maintenance/wikimedia-periodic-update.sh.puppettmp_8141 at /etc/puppet/manifests/misc/maintenance.pp:155
23:30 logmsgbot: yurik synchronized php-1.23wmf3/extensions/ZeroRatedMobileAccess/
23:27 ori-l: restarting rsyslog on application servers for I31c76fdde.
22:29 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'touch'
22:28 logmsgbot: bsitu synchronized echowikis.dblist 'Enable Echo and Thanks on dewiki and itwiki'
22:23 logmsgbot: bsitu synchronized echowikis.dblist 'Enable Echo and Thanks on dewiki and itwiki'
22:23 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Update email footer and help page'
22:19 logmsgbot: bsitu updated /a/common to Id9e3b9a03: correct default Echo help page
21:47 logmsgbot: faidon synchronized wmf-config/CommonSettings.php 'revert CentralNotice CrossWiki Hiding'
21:46 logmsgbot: faidon updated /a/common to I866838bc2: Revert "Enable CentralNotice CrossWiki Hiding"
21:34 mutante: ran checksetup.pl on bugzilla, deploy gerrit 96479, replaces internal errors on WeeklyReport with proper error pages
21:12 gwicke: deployed Parsoid 20c6afe
19:08 logmsgbot: yurik synchronized docroot/bits/WikipediaMobileFirefoxOS
18:07 akosiaris: changed default ferm policy to DROP. antimony/gallium are the first two servers to get it.
17:51 ^demon|sick: gerrit up, forcing replication of all repos
17:50 ^demon|sick: restarting gerrit since replication plugin won't reload
17:27 ^demon|sick: gerrit: disabled replication plugin fix, pending firewall updates. everything's timing out and flooding logs with errors.
13:59 hashar: jenkins: added slave integration-slave01 with label hasNpm. That is a slave running in labs ( integration-slave01.pmtpa.wmflabs )
12:31 apergos: end of an era: shut down ms7 webserver and opening ticket for decom
10:14 hashar: Jenkins: disabled gearman plugin, Zuul is no more a Gearman server.
10:12 hashar: restarted again stalled Jenkins
10:05 hashar: restarted Zuul with non gearman version.
10:02 hashar: reverting Zuul to last known version: wmf-deploy-20131023 + 6241272...1e3adfd wmf-deploy-20131023 -> master (forced update)
09:58 hashar: shutting down Zuul and reverting upgrade :-(
09:46 hashar: restarting Jenkins to update gearman plugin
09:38 hashar: Zuul upgraded with Gearman support to trigger build in Jenkins. Will monitor for the rest of the morning.
09:34 hashar: enabling Gearman in Jenkins, making it register with Zuul new version
09:33 hashar: restarted Zuul with Gearman version
09:27 hashar: removing all old versions of zuul: rm -fR /usr/local/lib/python2.7/dist-packages/zuul*
09:20 hashar: gallium temp install of python-pip to be able to ugprade Zuul
09:15 hashar: Zuul: bumped source code in integration/zuul.git to Gearman based version: 1e3adfd...6241272 labs -> master
09:05 hashar: stopping Zuul on gallium for upgrading purposes.
07:51 apergos: rebooting sq37, we had /dev//sdc gone and reappeared as /dev/sdc killing back-end squid
07:40 logmsgbot: ori synchronized php-1.23wmf3/extensions/WikimediaEvents/modules/ext.wikimediaEvents.moduleStorage.js 'Re-syncing I51d2d6495'
07:38 logmsgbot: ori synchronized php-1.23wmf4/extensions/WikimediaEvents/modules/ext.wikimediaEvents.moduleStorage.js 'Re-syncing I51d2d6495'
06:47 apergos: removed on mw* hosts from /tmp all *png/map/err/tif older than an hour, as some tmpfs were full
05:55 ori-l: /tmp on Apaches filling up with math .pngs; moving some of the oldest away as a stopgap
05:04 logmsgbot: springle synchronized wmf-config/db-pmtpa.php
03:17 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Nov 20 03:17:15 UTC 2013
02:31 logmsgbot: LocalisationUpdate completed (1.23wmf3) at Wed Nov 20 02:31:26 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Wed Nov 20 02:15:34 UTC 2013
01:38 paravoid: rebooting ms-be1001, i/o stuck kernel bug
01:37 logmsgbot: ori synchronized php-1.23wmf4/extensions/WikimediaEvents 'Update WikimediaEvents to I5b8cfe592'
01:35 logmsgbot: ori synchronized php-1.23wmf3/resources/startup.js 'touch'
01:33 logmsgbot: ori synchronized php-1.23wmf3/extensions/WikimediaEvents 'Update WikimediaEvents to I5b8cfe592'
00:43 logmsgbot: csteipp synchronized php-1.23wmf4/extensions/OAuth 'update OAuth to master for last blocker fix'
00:33 logmsgbot: ori synchronized php-1.23wmf3/resources/startup.js 'touch'
00:28 logmsgbot: ori synchronized php-1.23wmf3/extensions/MobileFrontend 'Updating MobileFrontend for I3efc1fa64'

November 19

23:58 ori-l: Restarting EventLogging jobs on vanadium
23:54 logmsgbot: ori synchronized php-1.23wmf3/resources/startup.js 'touch'
23:14 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'Iee7332c97: Disable infobox experiment'
23:13 logmsgbot: ori updated /a/common to Iee7332c97: Disable infobox experiment
22:53 logmsgbot: reedy synchronized php-1.23wmf4/extensions/AbuseFilter/ 'bug 57268'
21:59 logmsgbot: mholmquist synchronized php-1.23wmf3/extensions/VectorBeta/ 'Sync VectorBeta code to wikis that will never use it'
21:45 logmsgbot: mholmquist synchronized wmf-config/extension-list 'Add VectorBeta to extension-list'
21:43 logmsgbot: mholmquist updated /a/common to Ib532e5558: Add VectorBeta to extension-list
21:41 logmsgbot: mholmquist Finished syncing Wikimedia installation... : Adding VectorBeta
21:21 logmsgbot: mholmquist synchronized php-1.23wmf4/extensions/VectorBeta/
21:17 logmsgbot: mholmquist Started syncing Wikimedia installation... : Adding VectorBeta
21:10 logmsgbot: mholmquist synchronized wmf-config/InitialiseSettings.php 'Set wmgUseVectorBeta = true for phase0 wikis'
21:09 logmsgbot: mholmquist synchronized wmf-config/CommonSettings.php 'Add wmgUseVectorBeta'
21:08 logmsgbot: mholmquist updated /a/common to Ic3a64b434: Enable VectorBeta on group0 wikis
20:55 logmsgbot: reedy synchronized wmf-config/
20:04 logmsgbot: ori synchronized php-1.23wmf4/extensions/WikimediaEvents 'Update WikimediaEvents to d4f2a43'
20:04 logmsgbot: ori synchronized php-1.23wmf3/extensions/WikimediaEvents 'Update WikimediaEvents to d4f2a43'
20:01 logmsgbot: ori synchronized php-1.23wmf3/extensions/WikimediaEvents 'Update WikimediaEvents to d4f2a43'
20:00 logmsgbot: ori synchronized php-1.23wmf4/extensions/WikimediaEvents 'Update WikimediaEvents to d4f2a43'
19:49 logmsgbot: reedy synchronized php-1.23wmf3/extensions/MassMessage 'Update to master'
19:38 logmsgbot: ori synchronized php-1.23wmf4/extensions/WikimediaEvents 'Update WikimediaEvents to 4aa9c629e5: Controlled experiment to assess performance of module storage'
19:37 logmsgbot: ori synchronized php-1.23wmf3/extensions/WikimediaEvents 'Update WikimediaEvents to 4aa9c629e5: Controlled experiment to assess performance of module storage'
19:22 logmsgbot: reedy synchronized wmf-config/ 'Enable MassMessage Ib29cb042e739e26fc5ef56f41317454ea690d0cb'
19:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: https://gerrit.wikimedia.org/r/96297
19:07 logmsgbot: reedy updated /a/common to I9b48be4c0: try to reduce "no working slave" notice flood in pmtpa dberror.log (attempt 2)
18:53 Nemo_bis: but now ok
18:48 Nemo_bis: kaulen overloaded at 100 % CPU since 5 min ago, bugzilla almost unusable
18:32 logmsgbot: ori synchronized php-1.23wmf4/resources/mediawiki/mediawiki.js 'Id2835eca4: Enable module storage for 0.05% of visitors w/storage-capable browsers'
18:31 logmsgbot: ori synchronized php-1.23wmf3/resources/mediawiki/mediawiki.js 'Id2835eca4: Enable module storage for 0.05% of visitors w/storage-capable browsers'
17:35 LeslieCarr: added palladium as a salt master for common-infrastructure4 firewall
17:34 LeslieCarr: CV71028 flapped again
17:32 LeslieCarr: FPL link CV71028 flapped 20 minutes ago
11:56 apergos: powercycled ms-be1002, couldn't get in (lots of BUG: soft lockup - CPU#18 stuck for 22s etc on console)
11:25 akosiaris: remove puppetmaster and apache packages from sockpuppet. It no longer is a puppetmaster
11:18 akosiaris: remove puppetmaster and apache packages from stafford. It no longer is a puppetmaster
05:00 Tim: updating DNS to I3adffd88
02:50 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Nov 19 02:50:34 UTC 2013
02:19 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Tue Nov 19 02:19:11 UTC 2013
02:12 logmsgbot: springle synchronized wmf-config/db-pmtpa.php
02:10 logmsgbot: LocalisationUpdate completed (1.23wmf3) at Tue Nov 19 02:10:34 UTC 2013
02:06 logmsgbot: springle synchronized wmf-config/db-pmtpa.php

November 18

21:04 gwicke: deployed Parsoid fd3d6dc
20:39 RobH: disabled a bunch of rt users who no longer login and dont work here anymore (so if you do work here and i messed up, opps ;)
19:56 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Disabling Cirrus on all wikis but test2wiki'
19:36 mwalker: updating civi from 326854db84becacf808add24a75ca27d1358c5bc for new thank you templates
19:31 mutante: pybal - eqiad/search: disabling testsearch100[13], adding elastic10[12]
19:23 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'I0921b9e853fdacd554de40f138644d6187ae7efe'
17:36 logmsgbot: demon synchronized wmf-config/CirrusSearch-common.php 'If920e0e9'
17:31 logmsgbot: demon synchronized wmf-config/CirrusSearch-common.php 'I593f62a5'
17:30 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'I593f62a5'
17:25 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'I114677d0'
17:24 logmsgbot: demon synchronized wmf-config/CirrusSearch-common.php 'I114677d0'
17:23 logmsgbot: demon updated /a/common to I3b5eac7fb: depool db74 for move to S6
16:06 hashar: gallium: running puppet to apply ferm firewall configuration from 95162
10:00 springle: xtrabackup db50 to db74
09:46 logmsgbot: springle synchronized wmf-config/db-pmtpa.php 'depool db74 for move to S6'
06:28 apergos: powercycled ms-be9, it was unresponsive; doing xfs_repair on /dev/sde1 after 'corruption detected' messages on reboot
03:20 paravoid: adding mw1045 mw1070 mw1085 mw1165 back to the apache pool; no evidence (RT tickets, pybal comments) as why they are not pooled
02:49 springle: for index cardinality on nullable fields, testing innodb_stats_method=nulls_ignored on db1049, db1002, db1003, db1004, db1026, db1040, db1041 + reanalyzing tag_summary and change_tag
02:38 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Nov 18 02:37:59 UTC 2013
02:36 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'correct s3 groupLoadsBySection key'
02:22 logmsgbot: springle synchronized wmf-config/db-pmtpa.php 'pmtpa replication reconfig for decomissioning'
02:15 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Mon Nov 18 02:15:51 UTC 2013
02:07 logmsgbot: LocalisationUpdate completed (1.23wmf3) at Mon Nov 18 02:07:57 UTC 2013

November 17

20.00: the bits crisis is over thanks to paravoid
19:12 Nemo_bis: 20 % packet loss from Toolserver to bits-lb.esams.wikimedia.org
19:11 Nemo_bis: The 5xx errors mega-spikes every ten minutes are back (since 18 UTC)
02:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Nov 17 02:08:53 UTC 2013
02:03 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Sun Nov 17 02:02:54 UTC 2013
02:02 logmsgbot: LocalisationUpdate completed (1.23wmf3) at Sun Nov 17 02:02:01 UTC 2013
01:35 Reedy: Killed updateSpecialPages and related processes on terbium
01:18 MaxSem: Killed a few long queries on db1007
01:08 MaxSem: db1007 is having tough times due to special page updates

November 16

02:33 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Nov 16 02:33:35 UTC 2013
02:12 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Sat Nov 16 02:12:51 UTC 2013
02:07 logmsgbot: LocalisationUpdate completed (1.23wmf3) at Sat Nov 16 02:07:15 UTC 2013

November 15

23:13 logmsgbot: ori synchronized wmf-config/CommonSettings.php 'Ia9e64253d: Configure tungsten as $wgUDPProfilerHost'
23:11 logmsgbot: ori updated /a/common to Ia9e64253d: Configure tungsten as wgUDPProfilerHost
21:05 mutante: install unzip on tin (needs puppetization)
19:06 awight: crm updated from 8d9240fea273faf434881a16d894ef0ad6c616c0 to 326854db84becacf808add24a75ca27d1358c5bc
18:30 logmsgbot: ori updated /a/common to I132d97598: Beta Labs: enable UDP profiler
17:58 mutante: uploaded mw1.22.0rc2 from hexmode http://dumps.wikimedia.org/mediawiki/1.22/
15:02 akosiaris: update cr{1,2}-eqiad ACLs with new puppetmaster frontend address
13:55 akosiaris: switched *eqiad.wmnet to using new puppet infra. triggering puppet run
12:35 hashar: github: added zeljkofilipin to the new 'qa' team. Granted rights on wikimedia/mediawiki-selenium and wikimedia/qa-browsertests
12:33 akosiaris: disabled puppet on *eqiad.wmnet for change
12:32 akosiaris: updated sockpuppet term on cr1-sdtpa , cr2-tmtpa to include palladium
09:45 hashar: restarting zuul, leaked file descriptors pointing to non existent git pack files :/
09:15 akosiaris: disabled commiting in sockpuppet private for evah (hopefully) through a git hook
09:07 akosiaris2: disabled puppet an all *.pmtpa.wmnet servers to facilitate auditing the change to new puppetmaster infra
07:52 apergos: sq44 won't come back up, hung after "Remote Access Controller detected", rt #6312
06:48 Tim: also fixed mw1141, mw1193
06:42 Tim: fixing broken cgroups on mw1038, mw1044, mw1091
05:15 logmsgbot: ori updated /a/common to I67a6d49d4: Beta Labs: set deployment-fluoride as $wgUDPProfilerHost
03:20 Tim: on mc1001-1016: set timeout=0 in the redis soft state, as for rdb*
03:15 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Nov 15 03:15:35 UTC 2013
03:14 Tim: on rdb1001-1004, set timeout=0 in the soft state, to match I8c9b13c1
02:34 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Fri Nov 15 02:34:41 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.23wmf3) at Fri Nov 15 02:15:18 UTC 2013
00:39 cmjohnson1: dns update
00:29 mutante: schedule 1yr downtime for unused host 'gurvin' and all its services (RT #6135) and shutting it down

November 14

22:05 Nemo_bis: hashar> !log Gerrit: added marktraceur to group https://gerrit.wikimedia.org/r/#/admin/groups/539,members stream-events
21:59 hashar: Gerrit: added marktraceur to group stream-events
21:46 logmsgbot: ori synchronized docroot/bits/static-stable 'Correct 'static-stable' symlinks on bits'
21:45 logmsgbot: ori synchronized docroot/bits/static-current 'Correct 'static-current' symlinks on bits'
21:43 logmsgbot: ori updated /a/common to Id1a2bb2c8: Revert "Fix how 'current' branch is determined in updateBitsBranchPointers"
21:06 hashar: vvvv bug 57064
21:05 hashar: jenkins: migrated some phpcs job to use code sniffer style git-deployed on slaves in /srv/deployment/integration/mediawiki-tools-codesniffer/MediaWiki
21:02 hashar: jenkins: migrated some phpcs job to use code sniffer style git-deployed on slaves in /srv/deployment/integration/mediawiki-tools-codesniffer/MediaWiki
20:49 logmsgbot: reedy synchronized php-1.23wmf4/includes/
20:48 mark: Raised session_max on eqiad bits caches from 200k to 400k
20:45 logmsgbot: ori synchronized php-1.23wmf3/resources/startup.js 'touch'
20:42 logmsgbot: ori synchronized php-1.23wmf3/extensions/UniversalLanguageSelector 'Revert "Update UniversalLanguageSelector to b2f9e4211efc"'
20:26 logmsgbot: ori synchronized php-1.23wmf3/extensions/UniversalLanguageSelector 'Update UniversalLanguageSelector to b2f9e4211efc'
20:19 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Phase 1 wikis to 1.23wmf4
20:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.23wmf3
20:12 logmsgbot: reedy updated /a/common to I3034b1e4e: Add/update symlinks
20:05 logmsgbot: reedy Finished syncing Wikimedia installation... : testwiki to 1.23wmf4 and build l10n cache
19:34 logmsgbot: reedy Started syncing Wikimedia installation... : testwiki to 1.23wmf4 and build l10n cache
19:33 mark: Changed IP addresses of upload-lb.eqiad.wikimedia.org
19:24 RobH: 18:50 not 13:30 durrrrr
19:19 RobH: may have spoken too soon about ending, we shall see
19:18 RobH: logging for record: outage started at approx 13:50 GMT, seems to have ended well, now-ish
19:00 paravoid: pt-kill on s7 to kill LogPager queries
18:40 cmjohnson1: dns update
18:38 logmsgbot: reedy updated /a/common to I076d1eddf: CirrusSearch secondary for itwiki and plwiktionary
18:37 logmsgbot: reedy synchronized docroot and w
18:36 logmsgbot: reedy synchronized php-1.23wmf4/
17:56 logmsgbot: reedy synchronized php-1.23wmf3/extensions/Wikibase
17:22 manybubbles: rebuilding all CirrusSearch indexes in place to get config updates
17:19 hashar: Jenkins: updated code sniffer style from 574f68d to 0bebf0f7b
17:18 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php 'plwiktionary and itwiki get cirrus as secondary'
17:14 logmsgbot: manybubbles synchronized php-1.23wmf3/extensions/CirrusSearch/ 'Update CirrusSearch to master'
17:13 logmsgbot: manybubbles synchronized php-1.23wmf2/extensions/CirrusSearch/ 'Update CirrusSearch to master'
16:45 cmjohnson1: powering down analytics1009, analytics1011-13, analtyics1021, analytics1023 to relocate to rack A2
16:06 hashar: Gerrit/Git resetting mediawiki/extensions/Scribunto REL1_21 branch from fd1fbb4 to previous b5015a2. hexmode cherry-picked a bunch of changes from another branch/master instead of submitting a merge commit.
15:58 apergos: preallocated cache files for amssq61 and 62 with ext4, this requires removing logbuf option in fstab
13:27 logmsgbot: faidon synchronized wmf-config/filebackend.php 'switch multiMaster to swift-eqiad'
13:26 logmsgbot: faidon updated /a/common to I898bfd6f1: filebackend: switch multiMaster to swift-eqiad
13:09 apergos: preallocated cache file on amssq: 51, 52, 54, 56 59, 60 in addition to previously logged ones
13:02 apergos: bad news: Nov 14 11:17:43 amssq54 kernel: [246380.560873] XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250) and this host has the preallocated cache file
09:36 ori-l: Manually replaced jquery.ba-bbq.js on nickel:/srv/org/wikimedia/ganglia-web-latest/js with <https://github.com/cowboy/jquery-bbq/pull/42>. Upstream maintainer unresponsive.
07:46 apergos: powercycled sq80, accessible but responded 'bus error' or 'i/o error' to all commands
07:33 apergos: powercycled mw1210, unreachable via mgmt, no messages on console
02:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Nov 14 02:36:09 UTC 2013
02:13 logmsgbot: LocalisationUpdate completed (1.23wmf2) at Thu Nov 14 02:13:58 UTC 2013
02:07 logmsgbot: LocalisationUpdate completed (1.23wmf3) at Thu Nov 14 02:07:56 UTC 2013
01:27 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'Iaae2da0accd: Enable Bug54847 debug log group and route to '
01:26 logmsgbot: ori updated /a/common to I27cf69dd1: Revert "Rearrange W0 config-pages-supported variables order to be more logical."
00:32 ori-l: restarted stuck profiler-to-carbon on professor

November 13

23:37 mutante: power down db61
23:28 mutante: DNS update - remove db61 (former OTRS box)
22:54 mutante: DNS update - remove osm-web in Tampa, replaced with asset tags for wiping
21:17 gwicke: deployed Parsoid be03c28
20:58 mutante: DNS update - fix warnings for analytics IPv6 addresses
20:48 mutante: DNS update - kill frlabs, merge unmerged config-geo changes
20:46 hashar: Jenkins: fixing up jobs *-phpcs-HEAD which were not using the proper phpcs path 95268
19:14 logmsgbot: reedy synchronized wmf-config/ 'wgImgAuthDetails to true. Remove wmgUseUsabilityInitiative'
19:10 logmsgbot: reedy updated /a/common to I2a05d4a97: Remove usages of $wmgUseUsabilityInitiative
19:07 logmsgbot: reedy updated /a/common to I775790df6: Disable and remove CommunityHiring and CommunityApplications
19:03 logmsgbot: reedy synchronized wmf-config/ 'Disable and remove CommunityHiring and CommunityApplications'
17:20 Reedy: populateRevisionLength.php running in screen on terbium
16:56 apergos: recreating esams varnish backend caches, preallocated with fallocate; the following have been done: amssq49, 50, 53, 55, 57, 58
16:48 hashar: Jenkins: phpcs jobs successfully running on lanthanum. !!!
16:31 hashar: Jenkins: applying label 'hasPhpcs' on lanthanum slave, letting php code sniffer jobs to roam there.
16:29 hashar: Jenkins switched PHP Codesniffer (phpcs) to use the version from integration/phpcs.git instead of the pear one 95172
15:56 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Set $wgImgAuthDetails to true'
14:50 logmsgbot: reedy synchronized php-1.23wmf3/ 'Revert I49e34bdf13e8e8c6cd169c362c283fe1034bdc6d'
14:49 logmsgbot: reedy synchronized php-1.23wmf2/ 'Revert I49e34bdf13e8e8c6cd169c362c283fe1034bdc6d'
14:00 paravoid: adjusting new api appservers weight from 30 -> 20 to balance the load a little better
13:24 hashar: deploying phpcs on Jenkins slave using "the nameless wikimedia deployment system"
11:43 mark: Starting amssq* persistent storage cache fragmentation rework, temporarily disabling puppet in the process
10:53 mark: Manually set vm.dirty_ratio to 10 on amssq57
08:38 apergos: rebooting amssq48, deadlock in kmalloc (started about 10 hours ago)
06:31 apergos: rebooting amssq52, deadlock in kmem_alloc
04:38 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I5043210cb: Enable 'exception-json' debug log group; route to '
04:37 logmsgbot: ori updated /a/common to I5043210cb: Enable 'exception-json' debug log group; route to $wmfUdp2logDest
03:17 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Nov 13 03:17:40 UTC 2013
02:56 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'QueryPage::recache traffic'
02:42 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'move S3 Query::recache traffic to snapshot slave'
02:31 logmsgbot: LocalisationUpdate completed (1.23wmf2) at Wed Nov 13 02:31:21 UTC 2013
02:17 logmsgbot: LocalisationUpdate completed (1.23wmf3) at Wed Nov 13 02:17:07 UTC 2013
02:13 springle: restarting externallinks.el_id schema change jobs
01:30 logmsgbot: ori synchronized php-1.23wmf2/includes/specials/SpecialAllpages.php 'I189ba71de: In Special:AllPages, limit the size of hierarchical lists (Bug: 56840)'
01:29 logmsgbot: ori synchronized php-1.23wmf3/includes/specials/SpecialAllpages.php 'I189ba71de: In Special:AllPages, limit the size of hierarchical lists (Bug: 56840)'
01:26 logmsgbot: ori synchronized php-1.23wmf3/includes/specials/SpecialAllpages.php 'I189ba71de: In Special:AllPages, limit the size of hierarchical lists (Bug: 56840)'
00:24 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'touch'
00:19 logmsgbot: catrope synchronized php-1.23wmf3/resources/startup.js 'touch'
00:16 logmsgbot: catrope synchronized visualeditor.dblist 'Enable VisualEditor on office, board, collab and wikimaniateam wikis'
00:15 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Use visualeditor-default.dblist; enable VisualEditorParsoidForwardCookies on private wikis'
00:14 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Add plumbing for visualeditor-default.dblist, wmgVisualEditorParsoidForwardCookies and wmgVisualEditorNamespaces'
00:11 logmsgbot: catrope synchronized visualeditor-default.dblist 'New dblist for wikis with VisualEditor enabled by default'

November 12

23:36 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/94250'
23:33 logmsgbot: maxsem Finished syncing Wikimedia installation... : Demo deployment, no actual changes
23:31 Ryan_Lane: upgrading wikitech-static to 1.23wmf3
23:20 ^d: gitblit process had hung on antimony, restarted
23:17 Ryan_Lane: upgrading wikitech to 1.23wmf3
23:10 logmsgbot: maxsem Started syncing Wikimedia installation... : Demo deployment, no actual changes
22:59 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/94404'
22:55 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/93988'
22:50 logmsgbot: maxsem synchronized php-1.23wmf3/extensions/MobileFrontend/ 'Fix fatal'
22:47 logmsgbot: maxsem synchronized php-1.23wmf2/extensions/MobileFrontend/ 'Fix fatal'
21:57 cmjohnson1: dns update
21:34 logmsgbot: aaron cleared profiling data
21:21 logmsgbot: aaron synchronized php-1.23wmf3/maintenance/cleanupUploadStash.php '6461a94'
21:04 Reedy: Profiling data clearing seems to work, but echo "-truncate" | nc -q0 -u professor.pmtpa.wmnet 3811 doesn't return...
21:02 logmsgbot: reedy synchronized wmf-config/
20:28 andrewbogott: deployed new AbortNewAccount hook on wikitech
20:14 logmsgbot: reedy cleared profiling data
20:03 logmsgbot: reedy cleared profiling data
19:55 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Iab4777'
19:48 logmsgbot: reedy updated /a/common to I1bb030fce: Enable mwsearch logs
19:44 logmsgbot: reedy synchronized php-1.23wmf2 'Support CIDR ranges in $wgSquidServersNoPurge'
19:40 logmsgbot: reedy synchronized php-1.23wmf3 'Support CIDR ranges in $wgSquidServersNoPurge'
19:18 Reedy: Created EducationProgram tables on elwiki bug 56771
18:01 mark: Rebooting amssq60, stuck in xfs kmem_alloc deadlock
17:55 logmsgbot: reedy synchronized php-1.23wmf3/extensions/MWSearch/
17:54 logmsgbot: reedy synchronized php-1.23wmf2/extensions/MWSearch/
17:38 logmsgbot: reedy updated /a/common to I4baa719b9: CirrusSearch as secondary for nlwiki
17:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable mwsearch logs'
17:09 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php 'Enable Cirrus as secondary on nlwiki'
16:30 cmjohnson1: swapping ps1-c7-eqiad...one side at a time...notifications pending
14:17 springle: paused externallinks OSC jobs after replication glitch on dewiki. original table and data remain untouched
11:33 mark: Manually set dirty_background_ratio to 5 (from 10) on amssq58
11:30 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'thottle rule for WikiCon 94860'
11:29 logmsgbot: hashar synchronized wmf-config/throttle.php 'thottle rule for WikiCon 94860'
09:45 mark: rebooting amssq58 with sysrq-trigger
07:02 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'recache jobs on S2 to db1018'
03:34 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'slave balancing'
03:12 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Nov 12 03:12:05 UTC 2013
02:20 logmsgbot: LocalisationUpdate completed (1.23wmf2) at Tue Nov 12 02:20:54 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.23wmf3) at Tue Nov 12 02:15:03 UTC 2013
01:27 springle: restart db1050 mariadb after outage, let repl catch up. new lvm snaps mount ok. leave out of pool for now

November 11

21:32 paravoid: rebooting ms-be1003, kernel bug, system CPU & I/O wait through the roof
20:08 Nemo_bis: network level of bits application servers eqiad is back to the pre-deploy 18+8 MB/s
19:56 logmsgbot: reedy synchronized php-1.23wmf3/extensions/Wikibase '94790'
19:41 Reedy: Switching 588 wikis to 1.23wmf3 in one go seems to have upset the bits appserver pool
19:26 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.23wmf3
19:22 logmsgbot: reedy updated /a/common to Ibab846000: Bump wmgMemoryLimit to 210MB
17:40 logmsgbot: reedy synchronized php-1.23wmf3/extensions/FlaggedRevs '94774'
16:22 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bump wmgMemoryLimit to 210MB'
16:21 pp-pdf1: restarted all services
16:20 pp-pdf1: - upgrade mwlib to 0.15.12
16:13 paravoid: upgrading image/video scalers
15:32 mark: Moved eqiad wikipedia traffic onto Varnish
15:06 mark: Moved eqiad https/ipv6 text traffic from squid to varnish
14:31 mark: rebooting amssq54, XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
08:50 pp-pdf1: restarted all services
08:50 pp-pdf1: - update mwlib.rl to 0.14.4.
06:43 ori-l: scap: "sudo: no tty present and no askpass program specified" for snapshot1 & snapshot4
06:39 apergos: probably gratuitous powercycle of sq80, it seems fine now in any case
06:33 logmsgbot: ori Finished syncing Wikimedia installation... :
06:32 apergos: sq48 repeat of these errors and hung again, so rt #6274 opened
06:28 apergos: powercycled hung sq48, took two tries to come up, "NMI received for unknown reason 31 on CPU 0" and "mptbase: ioc0: ERROR - Failed to come READY after reset"
06:27 logmsgbot: ori Started syncing Wikimedia installation... :
02:40 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Nov 11 02:40:33 UTC 2013
02:29 ori-l: Apache logs filled with "SearchPhaseExecutionException[Failed to execute phase [dfs], all shards failed"
02:15 ori-l: Continuing inspection of logs on fluorine. memcached-serious.log is flooded with 'Memcached error for key [...]' errors, problem started in May or June judging by log sizes.
02:14 logmsgbot: LocalisationUpdate completed (1.23wmf3) at Mon Nov 11 02:14:09 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.23wmf2) at Mon Nov 11 02:07:59 UTC 2013
02:04 ori-l: Earlier issue identified by Ryan and Leslie as intermittent packet loss between eqiad and esams, due to capacity issue with provider.

November 10

23:42 ori-l: CPU overload in text caches esams
23:42 ori-l: Per Ryan: packet loss from esams to eqiad on xe-4-2-2.cr1-eqiad.wikimedia.org
23:26 ori-l: redis.log: flooded with 'Used automatic re-authentication for Lua script [...]' (68,955 such messages)
23:24 ori-l: fatal log: Fatal error: Allowed memory size of 201326592 bytes exhausted (tried to allocate 72 bytes) at /usr/local/apache/common-local/php-1.23wmf2/extensions/WikibaseDataModel/DataModel/Entity/Entity.php on line 130 (47 such fatals)
23:20 ori-l: poolcounter.log: Pool counter is full (multiple wikis)
23:17 ori-l: exception.log: Exception from line 110 of /usr/local/apache/common-local/php-1.23wmf2/includes/WikiPage.php: Invalid or virtual namespace -1 given. (2 such errors)
23:14 ori-l: exception.log: Exception from line 114 of /usr/local/apache/common-local/php-1.23wmf2/includes/upload/UploadStash.php: UploadStash::getFile No user is logged in, files must belong to users (8 such errors)
23:14 ori-l: exception.log: Exception from line 61 of /usr/local/apache/common-local/php-1.23wmf2/includes/media/ImageHandler.php: No width specified to ImageHandler::makeParamString (194 such errors)
23:13 ori-l: Investigating possible site issue and logging everything that I come across.
02:52 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Nov 10 02:52:22 UTC 2013
02:20 logmsgbot: LocalisationUpdate completed (1.23wmf3) at Sun Nov 10 02:20:46 UTC 2013
02:11 logmsgbot: LocalisationUpdate completed (1.23wmf2) at Sun Nov 10 02:11:33 UTC 2013

November 9

13:46 springle: starting pt-kill daemon on S1 slaves for SpecialAllpages::showToplevel queries > 30s
10:08 paravoid: upgraded ganglia-web to 3.5.10
05:37 logmsgbot: aaron synchronized php-1.23wmf3/includes/filebackend/SwiftFileBackend.php '519883b'
05:35 logmsgbot: aaron synchronized php-1.23wmf2/includes/filebackend/SwiftFileBackend.php '519883b'
02:10 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Nov 9 02:10:54 UTC 2013
02:04 logmsgbot: LocalisationUpdate completed (1.23wmf3) at Sat Nov 9 02:04:55 UTC 2013
02:03 logmsgbot: LocalisationUpdate completed (1.23wmf2) at Sat Nov 9 02:03:45 UTC 2013
01:23 logmsgbot: gwicke synchronized php-1.23wmf3/extensions/Parsoid/php
01:20 logmsgbot: aaron synchronized php-1.23wmf2/maintenance 'c674d4c'
01:19 logmsgbot: aaron synchronized php-1.23wmf3/maintenance 'c674d4c'
01:16 logmsgbot: gwicke synchronized php-1.23wmf2/extensions/Parsoid/php deploy fixed job runner
01:11 logmsgbot: aaron synchronized php-1.23wmf3/includes/filebackend '2afdc06'
01:11 logmsgbot: aaron synchronized php-1.23wmf2/includes/filebackend '2afdc06'
00:46 logmsgbot: aaron synchronized php-1.23wmf3/maintenance
00:32 logmsgbot: aaron synchronized php-1.23wmf2/maintenance/cleanupUploadStash.php '01da29d'

November 8

21:38 logmsgbot: ori updated /a/common to I30f4e5975: logmsg-git-hook: fix commit determination logic & run on new MW branch creation
21:22 logmsgbot: marc synchronized wmf-config/InitialiseSettings.php 'Bug: 56203 - Various changes to wikidatawiki's user rights configuration (for real this time)'
21:00 logmsgbot: marc synchronized wmf-config/InitialiseSettings.php 'Bug: 56203 - Various changes to wikidatawiki's user rights configuration'
19:23 ^d: jobqueue: dropped Cirrus-related jobs from wikidatawiki's queue
19:21 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'No more Cirrus for wikidatawiki'
19:19 mwalker: no longer use cmd=_xclick for donors in China
19:02 cmjohnson: B side on ps1-c7-eqiad will be unplugged to be tested
18:50 mutante: db1050 back up after skipping mount of failed /a
18:45 logmsgbot: faidon synchronized wmf-config/db-eqiad.php 'depool db1050'
18:45 logmsgbot: faidon updated /a/common to Ic66d0e783: Depool db1050, down, disk failed
18:32 mutante: powercycling frozen db1050
17:28 cmjohnson1: removing power to B Side on ps1-c7-eqiad
16:33 cmjohnson1: restarting ps1-c7-eqiad
15:52 Jeff_Green: authdns update to add A/PTR for lutetium public ip
07:09 logmsgbot: ori updated /a/common to I027abe363: Fix typo in logmsg-git-hook
06:54 logmsgbot: ori synchronized logmsg-git-hook 'logmsg-git-hook is meant only for tin, but syncing it for consistency'
06:52 logmsgbot: ori updated /a/common to I3691bbf3a: Remove '--first-parent' arg from rev-list invocation in logmsg-git-hook
06:45 logmsgbot: ori updated /a/common to I70534e64e: nostalgiawiki gets Cirrus
06:24 apergos: analytics1012 down, power mgmt firmware initialization error, opened ticket #6238
04:40 logmsgbot: demon synchronized cirrus.dblist
03:17 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Nov 8 03:17:20 UTC 2013
02:29 logmsgbot: LocalisationUpdate completed (1.23wmf3) at Fri Nov 8 02:28:59 UTC 2013
02:23 gwicke: deployed Parsoid 67fca5bdc7
02:09 logmsgbot: LocalisationUpdate completed (1.23wmf2) at Fri Nov 8 02:09:47 UTC 2013
01:53 ^d: tin: set git config branch.*.rebase true for all deployed branches
01:33 logmsgbot: catrope synchronized docroot/bits/static-current/ 'update static-current symlinks to wmf3'
01:33 springle: adding temporary index to S5 wikidatawiki.wb_terms for slow queries
01:33 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'fix logo path for wikimania2014wiki'
00:21 Krinkle: Reloading zuul to enable Iaceb016cf7df20
00:21 logmsgbot: csteipp synchronized php-1.23wmf3/includes 'bug 55332'
00:08 logmsgbot: catrope synchronized php-1.23wmf3/extensions/VisualEditor 'add more EventLogging events (https://gerrit.wikimedia.org/r/#/c/94092/ )'
00:04 logmsgbot: krinkle synchronized php-1.23wmf3/resources 'I8704a6620ece44d'

November 7

23:54 LeslieCarr: draining ulsfo of all traffic
23:26 logmsgbot: ori synchronized docroot/bits/static-current 'Update static-current symlinks to 1.23wmf3'
22:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:20 RoanKattouw: Reloading zuul config for new oojs-core and oojs-ui pipelines
21:16 logmsgbot: anomie synchronized php-1.23wmf3/resources/jquery/jquery.spinner.js 'Backport gerrit change 94161 to fix regression since 1.23wmf1'
21:16 logmsgbot: anomie synchronized php-1.23wmf3/maintenance/jsduck/external.js 'Backport gerrit change 94161 to fix regression since 1.23wmf1'
21:16 logmsgbot: anomie synchronized php-1.23wmf3/maintenance/jsduck/config.json 'Backport gerrit change 94161 to fix regression since 1.23wmf1'
21:15 logmsgbot: anomie synchronized php-1.23wmf3/resources/Resources.php 'Backport gerrit change 94161 to fix regression since 1.23wmf1'
21:15 logmsgbot: anomie synchronized php-1.23wmf3/skins/common/protect.js 'Backport gerrit change 94161 to fix regression since 1.23wmf1'
21:15 logmsgbot: anomie synchronized php-1.23wmf3/skins/common/upload.js 'Backport gerrit change 94161 to fix regression since 1.23wmf1'
21:14 logmsgbot: anomie synchronized php-1.23wmf2/resources/jquery/jquery.spinner.js 'Backport gerrit change 94161 to fix regression since 1.23wmf1'
21:14 logmsgbot: anomie synchronized php-1.23wmf2/maintenance/jsduck/external.js 'Backport gerrit change 94161 to fix regression since 1.23wmf1'
21:14 logmsgbot: anomie synchronized php-1.23wmf2/maintenance/jsduck/config.json 'Backport gerrit change 94161 to fix regression since 1.23wmf1'
21:14 logmsgbot: anomie synchronized php-1.23wmf2/resources/Resources.php 'Backport gerrit change 94161 to fix regression since 1.23wmf1'
21:13 logmsgbot: anomie synchronized php-1.23wmf2/skins/common/protect.js 'Backport gerrit change 94161 to fix regression since 1.23wmf1'
21:13 logmsgbot: anomie synchronized php-1.23wmf2/skins/common/upload.js 'Backport gerrit change 94161 to fix regression since 1.23wmf1'
21:00 Reedy: Created betafeatures database table on metawiki and commonswiki
20:53 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
20:35 ^d: elastic: in-place reindexing for all wikis running cirrus as primary
20:34 logmsgbot: reedy Finished syncing Wikimedia installation... : Ensure l10n cache is up to date
20:26 logmsgbot: reedy Started syncing Wikimedia installation... : Ensure l10n cache is up to date
20:14 logmsgbot: demon synchronized php-1.23wmf2/extensions/CirrusSearch 'Cirrus to master'
20:06 LeslieCarr: put sampling configuration on cr1-sdtpa - possible problems = high RE cpu utilization, which could cause network instability
19:49 logmsgbot: reedy synchronized wmf-config/
19:26 logmsgbot: reedy synchronized w/robots.php
19:18 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: phase1 wikis to 1.23wmf3
19:12 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: all wikipedias to 1.23wmf2
19:09 logmsgbot: reedy synchronized php-1.23wmf3/extensions/Wikibase
19:08 logmsgbot: reedy synchronized docroot and w
19:07 logmsgbot: reedy synchronized wmf-config/
18:33 ^d: for gerrit
18:33 ^d: reloading replication plugin
18:33 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.23wmf2
18:27 logmsgbot: reedy Finished syncing Wikimedia installation... : testwiki to 1.23wmf3 and build l10n cache
18:14 logmsgbot: reedy Started syncing Wikimedia installation... : testwiki to 1.23wmf3 and build l10n cache
18:09 logmsgbot: reedy synchronized docroot and w
18:07 logmsgbot: reedy synchronized php-1.23wmf3 'Staging'
17:10 gwicke: deployed Parsoid 986c1e78708
17:10 ori-l: Ran 'varnishadm ban.url .' on cp1045 & cp1058
17:01 gwicke: roll back Parsoid deploy as we are not able to purge the cache
16:22 gwicke: deployed Parsoid 986c1e787088
16:14 jeremyb: morebots running on tools-login again. will troubleshoot more later
16:13 hashar: jenkins : reducing number of executor on gallium from 8 to 5, we have lanthanum now.
16:11 jeremyb: hi hashar!
16:07 hashar: hello jeremyb
16:06 jeremyb: hello world
15:52 cmjohnson1: dns udpate
15:48 jeremyb: moved morebots back to the grid
11:27 hasharDojo: gallium : killed git process
11:18 hasharDojo: gallium on swap death thanks to git eating all memory :/
10:42 hashar: gallium / jenkins : running git gc --agressive on Zuul git repositories under /srv/ssd/zuul/git
10:01 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'touch'
10:00 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'Use correct bits for uls fonts on labs 94112'
09:37 akosiaris: uploaded snappy version 1.0.4-1build1wmf1 to apt.wikimedia.org
03:12 logmsgbot: aaron synchronized php-1.23wmf2/includes 'eaa917a'
03:07 logmsgbot: aaron synchronized php-1.23wmf1/includes 'eaa917a'
03:00 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Nov 7 03:00:37 UTC 2013
02:23 logmsgbot: LocalisationUpdate completed (1.23wmf1) at Thu Nov 7 02:23:35 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.23wmf2) at Thu Nov 7 02:15:27 UTC 2013
01:42 logmsgbot: ori synchronized php-1.23wmf2/resources/startup.js 'touch'
01:38 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I064c1b202: enable ResourceLoader module storage on mediawikiwiki'

November 6

23:01 awight: update payments from 133e8a7 to 3e5a284
22:03 logmsgbot: aaron synchronized php-1.23wmf2/maintenance '2d08379'
22:02 logmsgbot: aaron synchronized php-1.23wmf1/maintenance '2d08379'
22:00 mutante: cp1059, mobile varnish cache, was OOM but recovered by itself
21:59 logmsgbot: mholmquist Finished syncing Wikimedia installation... : Actually deploy TimedMediaHandler fix to fix the black-box issue for embedded videos
21:55 logmsgbot: mholmquist Started syncing Wikimedia installation... : Actually deploy TimedMediaHandler fix to fix the black-box issue for embedded videos
21:46 logmsgbot: mholmquist Finished syncing Wikimedia installation... : Update TimedMediaHandler to master to fix embedplayer
21:37 logmsgbot: marc synchronized wmf-config/CommonSettings.php 'Make templateeditor available for global groups'
21:29 logmsgbot: mholmquist Started syncing Wikimedia installation... : Update TimedMediaHandler to master to fix embedplayer
20:25 logmsgbot: reedy synchronized docroot/wwwportal/w/404.php 'Add 404 symlink'
19:58 mutante: shred'ding star.wm and *.wm SSL keys from kaulen. backing up old non-SAN wildcard cert and new bz certs on tridge
19:41 logmsgbot: mholmquist Finished syncing Wikimedia installation... : Updates for Multimedia extensions
19:25 logmsgbot: mholmquist Started syncing Wikimedia installation... : Updates for Multimedia extensions
17:32 logmsgbot: mark synchronized wmf-config/squid.php 'Add amssq47-62 IPv6 addresses'
16:54 logmsgbot: reedy synchronized php-1.23wmf2/extensions/WikimediaMaintenance/
16:42 mark: Moved esams ssl -> cache traffic from squid to varnish (manually)
14:33 mark: dist-upgrade && reboot on amssq47..62
12:18 paravoid: started pybal on amslvs1 again; seems to work again, root cause unknown
12:07 paravoid: stopping pybal on amslvs1, wikidata-lb:80 issues, pending further investigation
10:42 akosiaris1: upgraded etherpad-lite to 1.3.0 on zirconium
10:42 akosiaris1: rebooted zirconium for kernel upgrade
08:35 apergos: rebooting neon, ext3 errors on root filesystem
05:09 Tim: on tin: running bug 53687 cleanup script (WikimediaMaintenance/bug-53687/fixOrphans.php) on 37 wikis
03:24 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Nov 6 03:23:59 UTC 2013
02:35 logmsgbot: LocalisationUpdate completed (1.23wmf1) at Wed Nov 6 02:35:33 UTC 2013
02:21 logmsgbot: LocalisationUpdate completed (1.23wmf2) at Wed Nov 6 02:21:52 UTC 2013
00:30 logmsgbot: aaron synchronized php-1.23wmf2/includes/clientpool 'f1df92c'
00:29 logmsgbot: aaron synchronized php-1.23wmf1/includes/clientpool '313457c'

November 5

23:27 superm401: Growth team deployment complete
23:26 logmsgbot: mflaschen synchronized php-1.23wmf2/extensions/GuidedTour/ 'Deploy GuidedTour to 1.23wmf2'
23:26 logmsgbot: mflaschen synchronized php-1.23wmf2/extensions/GettingStarted/ 'Deploy GettingStarted to 1.23wmf2'
23:26 logmsgbot: mflaschen synchronized php-1.23wmf1/extensions/GuidedTour/ 'Deploy GuidedTour to 1.23wmf1'
23:25 logmsgbot: mflaschen synchronized php-1.23wmf1/extensions/GettingStarted/ 'Deploy GettingStarted to 1.23wmf1'
22:41 hashar: sleeping.
21:15 logmsgbot: aaron synchronized php-1.23wmf1/includes/clientpool/RedisConnectionPool.php '9e0a5d8'
21:09 logmsgbot: hashar synchronized wmf-config/logging-labs.php
21:08 logmsgbot: otto synchronized wmf-config/InitialiseSettings.php 'Configuring CirrusSearch for 5 new wikis'
21:08 logmsgbot: hashar synchronized wmf-config/jobqueue-eqiad.php 'removing ParsoidCacheUpdateJob from default queue. Drained a few weeks ago (gwicke)'
20:40 ottomata: re-enabling puppet on analytics1021
20:39 RobH: formey puppet agent disabled
20:26 cmjohnson1: dns update
20:17 cmjohnson1: shutting down search13-20, decommissoned
20:16 mutante: backing up /root on formey to tridge
19:55 mutante: DNS update - switching SVN over to antimony (eqiad)
19:49 RobH: marmontel decommissioned for shipment to eqiad
19:48 RobH: traces of neodyium, promethium, & argon removed and reclaimed to eqiad spare
19:28 logmsgbot: aaron synchronized php-1.23wmf2/includes/clientpool/RedisConnectionPool.php '9e0a5d8'
19:09 logmsgbot: ori synchronized php-1.23wmf1/extensions/Wikibase 'I9d52070a4: Re-introduce siteGroup setting for performance reasons'
19:09 logmsgbot: ori synchronized php-1.23wmf2/extensions/Wikibase 'I9d52070a4: Re-introduce siteGroup setting for performance reasons'
19:07 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I4d6862c3a: add wmgWikibaseSiteGroup'
19:06 logmsgbot: ori synchronized wmf-config/CommonSettings.php 'I4d6862c3a: add wmgWikibaseSiteGroup'
17:56 mutante: more cleanup on kaulen, move old stuff like "1.16wmf4 pilot" :) ,"codereview proxy", old bugzillas etc to backup
17:50 mutante: deleting empty /srv/org/mediawiki on kaulen
17:47 RobH: enabled spam filtering on ops list (logging in case it holds a ton of them)
17:21 cmjohnson1: updating pmtpa pybal searchpool_4 and 5...setting all to false
17:21 mutante: DNS update - killing codereview-proxy
17:16 logmsgbot: ori synchronized php-1.23wmf2/extensions/WikimediaEvents/modules/ext.wikimediaEvents.ve.js 'I4085ee9cb: Update VisualEditor performance event names'
17:15 logmsgbot: ori synchronized php-1.23wmf1/extensions/WikimediaEvents/modules/ext.wikimediaEvents.ve.js 'I4085ee9cb: Update VisualEditor performance event names'
17:12 cmjohnson1: disabling icinga checks on search13-20 for decommissioning rt6194
17:11 RobH: reclaimed marmontel for shipment to eqiad, confirmed decommission steps up to powering down.
17:10 ottomata: stopping puppet on analytics1021 to test jmxtrans kafka stuff
16:40 hashar: beta: extensions update has been blocked since Oct 30th due to some bad registration in mediawiki/extensions.git . Fixed!
16:39 cmjohnson1: dns update
15:48 paravoid: authdns: upgrading gdnsd to 1.10.1 & switching to the proprietary GeoIP database
15:22 logmsgbot: reedy synchronized php-1.23wmf2/includes/media/FormatMetadata.php 'I5f65333cc94b66fd80fcf5abcfad7e10e4669310'
13:29 mark: Changed upload-lb.esams IP addresses
07:09 paravoid: video/imagescaler upgrade
06:33 logmsgbot: ori synchronized wmf-config/CommonSettings.php 'Id33ec9366: Set enwiki's languageLinkSiteGroup to 'wikipedia' (Bug: 56602)'
04:45 springle: begin OSC for el_id on wikis with externallinks >1M rows (smaller ones already done)
03:48 logmsgbot: catrope synchronized php-1.23wmf2/extensions/VisualEditor 'Update VE for cherry-pick'
03:48 logmsgbot: catrope synchronized php-1.23wmf1/extensions/VisualEditor 'Update VE for cherry-pick'
03:45 gwicke: rolled back Parsoid deployment to cdbfdbb28ba2
03:14 gwicke: update Parsoid to 05aa83f to work around Vary: Accept-Encoding issue in production
03:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Nov 5 03:08:30 UTC 2013
02:30 logmsgbot: LocalisationUpdate completed (1.23wmf1) at Tue Nov 5 02:30:26 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.23wmf2) at Tue Nov 5 02:16:42 UTC 2013
02:00 Jeff_Green: stopped mysql on db1008 to clone another
01:57 gwicke: deployed Parsoid f416b82
01:04 RobH: virt0 port saturated at 23:42 (recordkeeping for outage investigation)
01:02 mutante: replaced SSL certs on Bugzilla with new ones that are not star.wm
00:46 springle: running master/slave pt-table-sync on archive tables with 1M+ rows
00:20 Reedy: Test

November 4

23:23 logmsgbot: ori synchronized php-1.23wmf2/resources/mediawiki/mediawiki.inspect.js 'Ib2252003f2: mediawiki.inspect#dumpTable: fix broken FF workaround'
23:12 logmsgbot: csteipp synchronized php-1.23wmf1/includes 'bug 55332'
23:11 logmsgbot: csteipp synchronized php-1.23wmf2/includes 'bug 55332'
22:39 logmsgbot: mwalker synchronized php-1.23wmf2/skins/vector/ 'Deploying fix for bug 56366 : https://gerrit.wikimedia.org/r/#/c/93408/'
22:39 logmsgbot: mwalker synchronized php-1.23wmf1/skins/vector/ 'Deploying fix for bug 56366 : https://gerrit.wikimedia.org/r/#/c/93408/'
22:22 logmsgbot: mwalker synchronized wmf-config/CommonSettings.php 'Enabling cross wiki banner hiding for CentralNotice'
22:16 logmsgbot: mwalker synchronized php-1.23wmf2/resources/mediawiki/mediawiki.inspect.js 'Pushing https://gerrit.wikimedia.org/r/#/c/93587/'
22:15 logmsgbot: mwalker synchronized php-1.23wmf2/extensions/CentralNotice/ 'Updating CentralNotice to master - mobile redirects and cross wiki hiding'
22:15 logmsgbot: mwalker synchronized php-1.23wmf1/extensions/CentralNotice/ 'Updating CentralNotice to master - mobile redirects and cross wiki hiding'
22:07 MaxSem: Reindexing GeoData
21:44 cmjohnson1: dns update
21:43 hashar: on mediawiki/core.git deleted and retagged 1.22.0rc0 (same commit: c00622b )
21:29 mutante: schedule icinga downtime/disable notifications for db3[29] db4[2-6] db5[1235689]
21:00 cmjohnson1: dns update
20:51 logmsgbot: reedy synchronized php-1.23wmf2/extensions/Echo 'bug 56521'
20:47 gwicke: reverted Parsoid deployment due to this charset corruption issue
20:34 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.23wmf2
20:29 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable BetaFeatures and friends on mediawikiwiki and testwikidatawiki'
20:22 Reedy: Created BetaFeatures tables on mediawikiwiki and testwikidatawiki
20:19 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki and testwikidatawiki to 1.23wmf2
19:58 mutante: restarted gitblit on antimony
19:48 gwicke: updated Parsoid to d7b556f25353
18:47 awight: updated crm from 272966e to 8d9240f
18:30 mark: Configured cr1-esams and cr2-knams for new LVS service IP range 91.198.174.192/27
18:25 logmsgbot: aaron synchronized php-1.23wmf2/maintenance/cleanupUploadStash.php 'f3332a3'
17:58 awight: rollback crm to 272966e
17:57 awight: updated crm from 272966e to 63de2d6
17:49 mark: Deactivated ae1.101 on cr1-esams and cr2-knams
16:42 cmjohnson1: dns update
16:31 mark: Disabled multicast routing between eqiad and pmtpa, setup udpmcast between chromium and dobson instead
14:44 mark: Disabled multicast traffic reduction on csw1-sdtpa
14:41 akosiaris: removed LACP configuration from asw-b-eqiad for palladium and set it to standard access-port and private1-b-eqiad vlan
14:40 akosiaris: removed LACP configuration from asw-a-eqiad for strontium and set it to standard access-port and private1-a-eqiad vlan
14:24 mark: Killed udpmcast on chromium
14:13 mark: Configured OSPF/OSPF3 on cr1-eqiad:xe-4/2/2 <--> cr2-knams:xe-1/1/0
13:01 hashar: Jenkins upgrading gearman (0.0.4 -> 0.0.5) plugin on gallium from http://repo.jenkins-ci.org/repo/org/jenkins-ci/plugins/gearman-plugin/0.0.5/
11:22 mark: multicast routing to pmtpa restored for now
10:52 mark: Deactivated BFD on flapping link ae0 between cr1-eqiad and cr2-eqiad
10:47 mark: Deactivated anycast PIM RP on cr2-eqiad
10:21 mark: Reenabled 10G wave between cr2-eqiad and cr1-sdtpa, set high OSPF metrics instead
09:56 akosiaris: added python-apscheduler to apt.wikimedia.org
09:48 hashar: Jenkins: upgrading PHP_CodeSniffer from 1.4.6 to 1.4.7
08:32 logmsgbot: reedy synchronized php-1.23wmf2/ 'Ibcf77ed7f04c14a477d7cfd0e244929c552c3394'
08:00 paravoid: cr2-eqiad set xe-5/2/1 disable; 10g wave to cr1-sdtpa, flapping since yesterday, causing packet loss and outages
06:24 jeremyb: left morebots running on tools-login because it doesn't work on all grid hosts. see https://gerrit.wikimedia.org/r/93426
05:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Nov 4 05:08:07 UTC 2013
04:40 logmsgbot: LocalisationUpdate completed (1.23wmf2) at Mon Nov 4 04:40:40 UTC 2013
04:30 logmsgbot: LocalisationUpdate completed (1.23wmf1) at Mon Nov 4 04:30:48 UTC 2013
04:19 ori-l: l10nupdate failures are due to "No submodule mapping found in .gitmodules for path 'WikibaseDatabase'"; repo appears to have been deleted; fixed with git rm --cached WikibaseDatabase as l10nupdate on tin.
04:15 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
03:12 Tim: moved udpmcast to chromium since it actually has an external IP address
03:05 Tim: moved udpmcast.py from dobson to tungsten to work around failure of multicast routing eqiad -> pmtpa
02:01 logmsgbot: LocalisationUpdate failed: git pull of extensions failed

November 3

12:50 mark: Activated BFD between cr2-eqiad and cr1-sdtpa link
10:07 mark: Send traffic from Japan and Korea to ulsfo, https://gerrit.wikimedia.org/r/#/c/93309/
07:26 ori-l: kaulen's apache was down (killed by oom killer) & refused to start. copied over old config per #5011, restarted apache, & disabled puppet for now.
07:07 p858snake|l: Bugzilla isn't loading
02:02 logmsgbot: LocalisationUpdate failed: git pull of extensions failed

November 2

19:50 cmjohnson1: dns update
16:13 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
12:43 springle: start xtrabackup db52->db69 and db39->db71
09:56 apergos: restarted twemproxy on ~10 of the api servers in eqiad in the lower memory/core group, was seeing a lot of entries in memcached-serious log and some processes churning on that
07:37 apergos: restarted a bunch of the apaches on the 12gb memory apis that were maxed out at their 100 client limit
07:09 apergos: and increased again to 30, after looking at memory on both groups of boxes.
06:58 apergos: increase weight of mw1189-1208 api servers from 20 to 25, they handle the load better (hopefully)
06:03 logmsgbot: springle synchronized wmf-config/db-pmtpa.php 'depool db69 and db71 for reassignment during pmtpa decomm'
02:00 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
00:39 awight: updated crm from da3448df1404311620adc50b4d4ca28abfd2f3cb to 272966e353244e668a64c54ca4fa6f344079f4bf

November 1

23:55 awight: updated crm from 4a4c31cf60b783a5269c3e18b4baaf3bde5568ab to da3448df1404311620adc50b4d4ca28abfd2f3cb
23:45 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'Ib19944b16: $wgResourceLoaderStorageEnabled = true on test & test2'
22:57 awight: crm updated from 28093721d29226e9bd17d8be43db8a0deb51fe22 to 4a4c31cf60b783a5269c3e18b4baaf3bde5568ab
22:45 awight: crm updated from 21cb97b5083f4a32e67b5b6dac4b58d733b8d7ca to 28093721d29226e9bd17d8be43db8a0deb51fe22
22:23 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Nov 1 22:23:20 UTC 2013
22:16 logmsgbot: LocalisationUpdate completed (1.23wmf2) at Fri Nov 1 22:16:51 UTC 2013
21:57 logmsgbot: LocalisationUpdate completed (1.23wmf1) at Fri Nov 1 21:57:34 UTC 2013
21:49 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild l10n cache for MultimediaViewer update
21:39 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild l10n cache for MultimediaViewer update
21:25 logmsgbot: reedy synchronized wmf-config 'touch'
21:24 logmsgbot: reedy synchronized php-1.23wmf2/
21:22 awight: updated crm from dca2c8cc0eaa503e6d0c9921952cb85855e3b110 to 21cb97b5083f4a32e67b5b6dac4b58d733b8d7ca
19:59 logmsgbot: reedy synchronized php-1.23wmf1/extensions/Wikibase
19:49 hashar: jenkins: refreshing all *pep8 jobs, pep8 wrapper for puppet pointing to a wrong path
19:48 Reedy: Added rc_source column to testwikidatawiki.recentchanges
19:47 Reedy: Added rc_source column to mediawikiwiki.recentchanges
19:46 Reedy: Added rc_source column to test2wiki.recentchanges
19:43 Reedy: Added rc_source column to testwiki.recentchanges
19:11 Reedy: Created betafeatures_user_counts on testwiki and test2wiki
19:06 logmsgbot: reedy synchronized wmf-config/ 'Enable magic beta extensions'
18:58 logmsgbot: reedy Finished syncing Wikimedia installation... : scap take 2 to build l10ncache with magical beta things
18:49 logmsgbot: reedy Started syncing Wikimedia installation... : scap take 2 to build l10ncache with magical beta things
18:31 logmsgbot: reedy Finished syncing Wikimedia installation... : testwiki and test2wiki to 1.23wmf2
18:17 logmsgbot: reedy Started syncing Wikimedia installation... : testwiki and test2wiki to 1.23wmf2
18:14 paravoid: reenabling mw1114-mw1148
18:12 logmsgbot: reedy synchronized wmf-config/
18:08 paravoid: swapoff on all api appservers
17:53 logmsgbot: reedy synchronized wmf-config/
17:26 paravoid: depooling mw1114-mw1148; balancing is unfair, boxes overloaded, mw1189-mw1208 capable of handling the load
16:01 apergos: shot one more forceSearchIndex on arsenic because we were back in swap, only one left...
13:51 hasharEat: gallium : Jenkins console log compression completed, saved 80G out of 500G total disk space.
10:57 apergos: shot one more forceSearchIndex on arsenic, only two left... (using about 12gb between them)
10:51 logmsgbot: springle synchronized wmf-config/db-pmtpa.php 'depool first batch of pmtpa boxes to be decommissioned/shipped'
09:08 hashar: Jenkins: fixing up a race condition in MobileFrontend qunit tests. Both variant were using the same path and the first job to complete would break the other one by deleting the mediawiki install. 93030
08:18 apergos: shot two of the forceSearchIndex on arsenic, they were using 7gb between them and arsenic was in swapdeath
03:01 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Nov 1 03:01:11 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.23wmf1) at Fri Nov 1 02:15:15 UTC 2013
01:25 LeslieCarr: reverted the ulsfo traffic check, due to editors appearing to come from 127.0.0.1
01:21 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Clean up 127.0.0.1 logging code'
01:11 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Debugging 127.0.0.1 issue: add XFP logging'
01:04 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Debugging 127.0.0.1 issue'

October 31

23:55 cmjohnson1: dns update
23:41 mutante: delete dsh groups 'nagios' and 'misc-servers' from tin/fenari. they are gone from puppet and unused (just not actively deleted)
23:33 cmjohnson1: dns update
23:32 logmsgbot: reedy synchronized php-1.23wmf1/extensions/MobileFrontend 'touch'
23:31 logmsgbot: reedy synchronized php-1.23wmf1/resources 'touch'
23:30 logmsgbot: reedy synchronized wmf-config/ 'touch'
22:31 RobH: restarted parsoid on wtp1011
22:17 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Bye bye MW 1.22
21:38 logmsgbot: demon Finished syncing Wikimedia installation... : Unbreak cluster. Like most things, the sequel wasn't as good as the original
21:33 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Fixing for the last time'
21:33 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Fixing for the last time'
21:21 logmsgbot: demon Started syncing Wikimedia installation... : Unbreak cluster. Like most things, the sequel wasn't as good as the original
21:02 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files:
20:55 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.23wmf2
18:31 cmjohnson1: removing decommissioned servers alsted, amaranth, durant, spence, williams from dsh group misc-servers on tin
18:29 logmsgbot: demon synchronized wmf-config/
18:24 logmsgbot: demon synchronized wmf-config/ 'Proper poolcounter config for Cirrus'
18:20 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'touch'
18:16 logmsgbot: hashar synchronized php-1.22wmf22/extensions/PoolCounter/ 'making sure all apaches got all the files.'
18:16 logmsgbot: hashar synchronized php-1.23wmf1/extensions/PoolCounter/ 'making sure all apaches got all the files.'
18:11 cmjohnson1: dns update
18:01 logmsgbot: demon synchronized w/ 'Cluster to known good state'
17:59 logmsgbot: demon synchronized wmf-config/ 'Cluster to known good state'
16:52 logmsgbot: reedy synchronized wmf-config/
16:30 logmsgbot: reedy synchronized docroot and w
16:29 logmsgbot: reedy synchronized php-1.23wmf2 'Staging php-1.23wmf2'
14:52 mark: Sending text (all wikis) traffic from OC to ulsfo
13:51 logmsgbot: mark synchronized wmf-config/squid.php 'Update cache list with ulsfo text caches'
13:39 cmjohnson: dns update
12:59 akosiaris: added python-pbr backported from saucy to apt.wikimedia.org
11:23 akosiaris: added python-gear package by hashar to apt.wikimedia.org
11:13 hashar: jenkins compressing all console logs
10:29 hashar: jenkins / gallium : got rid of the old /var/lib/jenkins/tmpfs 512MB mount ( 92853 + manual amount )
03:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Oct 31 03:26:30 UTC 2013
02:47 logmsgbot: LocalisationUpdate completed (1.22wmf22) at Thu Oct 31 02:47:38 UTC 2013
02:25 logmsgbot: LocalisationUpdate completed (1.23wmf1) at Thu Oct 31 02:25:43 UTC 2013
01:07 awight: crm updated from 06348e7eea3c914e9444af588e0d7ffeb640366c to dca2c8cc0eaa503e6d0c9921952cb85855e3b110
00:46 ori-l: running MONITOR for 60 seconds on redis instance on mc1002

October 30

23:51 logmsgbot: maxsem synchronized wmf-config/mobile.php 'https://gerrit.wikimedia.org/r/92811'
23:37 logmsgbot: catrope synchronized php-1.23wmf1/extensions/VisualEditor/ 'VE cherry-pick'
23:30 logmsgbot: maxsem synchronized php-1.23wmf1/extensions/MobileFrontend/ 'https://gerrit.wikimedia.org/r/#/c/92676/'
23:29 logmsgbot: demon synchronized wmf-config/PoolCounterSettings-pmtpa.php 'Upping search maxqueue to 600'
23:29 logmsgbot: demon synchronized wmf-config/PoolCounterSettings-eqiad.php 'Upping search maxqueue to 600'
23:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable Zero on wikidataw'
23:17 logmsgbot: maxsem synchronized php-1.23wmf1/extensions/MobileFrontend/ 'Shit hit fan'
23:08 logmsgbot: maxsem synchronized php-1.23wmf1/extensions/MobileFrontend/ 'https://gerrit.wikimedia.org/r/#/c/92676/'
23:07 logmsgbot: maxsem synchronized php-1.22wmf22/extensions/MobileFrontend/ 'https://gerrit.wikimedia.org/r/#/c/92676/'
23:02 logmsgbot: yurik synchronized php-1.23wmf1/extensions/ZeroRatedMobileAccess/
22:48 logmsgbot: yurik synchronized php-1.22wmf22/extensions/ZeroRatedMobileAccess/
22:00 K4-713: updated civicrm to 06348e7eea3
19:56 logmsgbot: demon Finished syncing Wikimedia installation... : No changes, making sure arsenic is synced
19:46 logmsgbot: demon Started syncing Wikimedia installation... : No changes, making sure arsenic is synced
19:45 logmsgbot: demon synchronized multiversion 'Making sure multiversion is up to date on arsenic'
19:44 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: Making sure wikiversions is up to date on arsenic
19:27 cmjohnson: removing decom'd srv* from /etc/dsh/groups nrpe-ext-stores, nagios
19:22 cmjohnson: dns update
18:39 logmsgbot: demon synchronized robots.txt 'New robots.txt changes for mobileview'
17:47 awight: dedupe updated from 5b9f19ba719dc6 to 922b3aa6b797a
17:38 logmsgbot: demon Finished syncing Wikimedia installation... : No changes, making sure arsenic is synced
17:36 awight: updated crm from 6f370cccad7f9 to 117025ce5c993d
17:20 logmsgbot: demon Started syncing Wikimedia installation... : No changes, making sure arsenic is synced
17:17 ^d: aborted sync
17:17 logmsgbot: demon Started syncing Wikimedia installation... : No changes, making sure arsenic is synced
15:40 hashar: Broke jenkins for a couple hours while doing some path migration. All jobs got updated in the process and failed jobs got retriggered.
10:40 apergos: restarted morebots (failed to rejoin after netsplit)

October 29

21:14 logmsgbot: maxsem synchronized php-1.23wmf1/extensions/MobileFrontend/ 'https://gerrit.wikimedia.org/r/#/c/92474/'
21:12 logmsgbot: maxsem synchronized php-1.22wmf22/extensions/MobileFrontend/ 'https://gerrit.wikimedia.org/r/#/c/92474/'
21:05 LeslieCarr: one of the sdtpa <-> eqiad links is down
20:50 ^demon|lunch: elasticsearch: stopped wikidatawiki indexing processes on terbium, causing too much load. working with RobH on getting an alternative place to run these scripts.
20:08 mwalker: fundraising cluster back to normal
20:06 cmjohnson1: dns update
19:14 mutante: create mw/1.22 dir on dumps.wm.org, publish mw 1.22.0rc0 - http://dumps.wikimedia.org/mediawiki/1.22/
19:06 cmjohnson1: dns update
18:49 cmjohnson1: dns update
17:51 mwalker: frack cluster back in maintenance mode while I figure out why persistent STOMP messages fail to be stored
17:45 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Switch all small wikis to use extension1 db'
17:30 mwalker: taking the payments cluster out of maintenance mode for testing
16:49 logmsgbot: reedy synchronized php-1.23wmf1/extensions/Wikibase
16:35 mwalker: placing fundraising cluster into maintenance mode for ActiveMQ and database maintenance
16:23 mwalker: updating payments from e475603
16:13 ^d: elasticsearch: created wikidatawiki index, running indexing from 8 processes on terbium.
16:08 logmsgbot: reedy synchronized php-1.23wmf1/includes/upload/UploadStash.php
16:07 logmsgbot: reedy synchronized php-1.22wmf22/includes/upload/UploadStash.php
16:04 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Cirrus on wikidatawiki as secondary index'
14:49 cmjohnson1: dns update
13:15 paravoid: reprepro: include backported graphite-carbon, graphite-web, python-whisper (plus deps python-django-tagging, flot) from saucy; replace custom-built python-whisper
12:09 hashar: jenkins refreshing jobs to let them recurse in git submodules [[bugzilla:55614|bug 55614]] 92511
11:56 hashar: lanthanum.eqiad.wmnet running apt-get upgrade
11:36 hashar: gerrit: creating integration/phpunit to hold PHPUnit files for deployment
07:13 paravoid: "ln -s /apache /a" on terbium; foreachwiki and friends was broken (for a while) due to /usr/local/lib/mw-deployment-vars.sh pointing to non-existent /a
03:01 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Oct 29 03:01:47 UTC 2013
02:29 logmsgbot: LocalisationUpdate completed (1.22wmf22) at Tue Oct 29 02:29:03 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.23wmf1) at Tue Oct 29 02:15:32 UTC 2013
00:25 logmsgbot: ori synchronized wmf-config/CommonSettings-labs.php 'I6ca3517dc: Enable localStorage module caching (If2ad2d80d) on beta cluster'

October 28

22:31 cmjohnson1: dns update 2nd set of changes
22:29 RobH: updated dns
22:10 cmjohnson1: powering down search 21 -36
21:25 logmsgbot: reedy synchronized php-1.23wmf1/includes/db/Database.php 'bug 56124'
21:18 logmsgbot: awight synchronized php-1.23wmf1/extensions/CentralNotice
21:17 logmsgbot: awight synchronized php-1.22wmf22/extensions/CentralNotice
20:41 manybubbles: built search indexes for a bunch of small wikis that didn't seem to have them. we should be done getting index doesn't exist errors.
19:20 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: non wikipedia to 1.23wmf1
18:45 awight: payments updated from 84bb204 to e475603
18:26 logmsgbot: reedy synchronized php-1.23wmf1/extensions/Wikibase
18:13 logmsgbot: reedy synchronized php-1.23wmf1/includes/specials/SpecialContributions.php
17:13 logmsgbot: aaron synchronized wmf-config/ 'Switched to JobQueueFederated'
15:02 cmjohnson1: removing search21-36 from pybal search_pools
13:26 manybubbles: restarted elasticsearch nodes to pick up new config
08:42 apergos: restarted etherpadlite on zirconium, see ticket 6093, it was not saving edits
08:22 Nemo_bis: etherpad.wikimedia.org seems read-only
07:17 springle: delaying one tampa slave per shard during OSC
06:41 apergos: powercycled ms-be1001, unaccessible via ssh or mgmt
02:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Oct 28 02:36:16 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.23wmf1) at Mon Oct 28 02:15:57 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.22wmf22) at Mon Oct 28 02:08:37 UTC 2013

October 27

17:19 Reedy: Creating echo_notification, echo_event and echo_email_batch IF NOT EXISTS for all echo related wikis on 10.64.16.18
17:07 Reedy: Created echo related tables for ndswiktionary on 10.64.16.18
17:06 Reedy: Created echo related tables for kawiktionary on 10.64.16.18
02:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Oct 27 02:26:02 UTC 2013
02:12 logmsgbot: LocalisationUpdate completed (1.23wmf1) at Sun Oct 27 02:12:09 UTC 2013
02:07 logmsgbot: LocalisationUpdate completed (1.22wmf22) at Sun Oct 27 02:06:57 UTC 2013

October 26

02:32 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Oct 26 02:32:40 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.23wmf1) at Sat Oct 26 02:16:30 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.22wmf22) at Sat Oct 26 02:08:20 UTC 2013

October 25

23:44 logmsgbot: aaron synchronized php-1.22wmf22/extensions/SwiftCloudFiles '2661b2d'
22:00 cmjohnson1: removing search21-36 from pybal
21:44 cmjohnson1: powering off to decom search21-36
21:15 cmjohnson1: dns update
21:13 cmjohnson1: removing mw110 from pybal
19:42 K4-7131: updated payments cluster to 84bb20494793b4
16:59 mutante: DNS update, fix mw110 reverse entries
16:22 cmjohnson1: dns update
13:59 apergos: shot swiftrepl on copper, almost out of space, tossed the log... restart as you like
13:02 mark: Rebooting cp1046, xfs mem alloc issues
06:38 Tim: set email address for user MahdiBot to be the same as Mahdiz
03:21 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Oct 25 03:21:39 UTC 2013
02:48 logmsgbot: LocalisationUpdate completed (1.23wmf1) at Fri Oct 25 02:48:10 UTC 2013
02:23 springle: starting recentchanges OSC bug 55844
02:16 logmsgbot: LocalisationUpdate completed (1.22wmf22) at Fri Oct 25 02:16:50 UTC 2013
01:04 logmsgbot: kaldari synchronized php-1.22wmf22/extensions/MobileFrontend/ 'Updating MobileFrontend for cherrypick, one more time with feeling'
01:01 logmsgbot: kaldari synchronized php-1.22wmf22/extensions/MobileFrontend/ 'Updating MobileFrontend for cherrypick'
00:49 logmsgbot: kaldari synchronized php-1.22wmf22/extensions/MobileFrontend/ 'Updating MobileFrontend for cherrypick'
00:13 logmsgbot: kaldari synchronized php-1.22wmf22/extensions/MobileFrontend/ 'Updating MobileFrontend for cherrypick'
00:06 logmsgbot: kaldari synchronized php-1.22wmf22/extensions/MobileFrontend/ 'Updating MobileFrontend for cherrypick'
00:03 logmsgbot: kaldari synchronized php-1.23wmf1/extensions/MobileFrontend/ 'Updating MobileFrontend for cherrypick'

October 24

23:40 logmsgbot: reedy synchronized docroot/noc/conf/
23:25 logmsgbot: reedy synchronized docroot/noc/
23:23 logmsgbot: reedy synchronized multiversion
23:05 logmsgbot: reedy synchronized multiversion/
22:59 logmsgbot: reedy synchronized docroot and w
22:08 logmsgbot: awight synchronized php-1.23wmf1/extensions/CentralNotice
22:07 logmsgbot: awight synchronized php-1.22wmf22/extensions/CentralNotice
20:33 mwalker: updating listeners from 15b4723
20:19 logmsgbot: reedy synchronized echowikis.dblist 're-enable echo on test2wiki'
20:18 Reedy: Created echo tables for test2wiki on 10.64.16.18
20:16 logmsgbot: reedy synchronized echowikis.dblist 'Disable Echo on test2wiki due to a lack of database tables in extension1'
19:53 logmsgbot: reedy synchronized wmf-config/
19:45 logmsgbot: reedy synchronized database lists files:
19:12 Reedy: Created WikiLove extension tables on kowiki
18:53 mutante: DNS update - remove nomcom wiki
18:27 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.22wmf22
18:25 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: phase1 wikis to 1.23wmf1
18:22 logmsgbot: reedy Finished syncing Wikimedia installation... : testwiki to 1.23wmf1 and build l10n cache
18:09 logmsgbot: reedy Started syncing Wikimedia installation... : testwiki to 1.23wmf1 and build l10n cache
18:01 logmsgbot: reedy synchronized docroot and w
18:00 logmsgbot: reedy synchronized php-1.23wmf1 'make sure'
18:00 awight: updated tools from bc75134 to 5b9f19b
17:58 paravoid: shutting down ms-be1006, to be used by cmjohnson1 for testing firmware changes
17:58 logmsgbot: reedy synchronized php-1.23wmf1
17:38 mutante: added gwicke to wmf ldap group
12:38 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool db1026'
11:19 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/90759'
11:18 logmsgbot: maxsem synchronized wmf-config/CommonSettings.php 'https://gerrit.wikimedia.org/r/#/c/91058/'
10:47 logmsgbot: tstarling Finished syncing Wikimedia installation... : fix linkprefix regexes in 1.22wmf22 (Iaa7eaa44)
10:41 logmsgbot: tstarling Started syncing Wikimedia installation... : fix linkprefix regexes in 1.22wmf22 (Iaa7eaa44)
10:24 logmsgbot: tstarling Finished syncing Wikimedia installation... : fix linkprefix regexes (Iaa7eaa44)
10:15 logmsgbot: tstarling Started syncing Wikimedia installation... : fix linkprefix regexes (Iaa7eaa44)
10:06 springle: set global userstat=1 on mariadb instances for audit
07:13 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1026'
03:11 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Oct 24 03:11:12 UTC 2013
02:42 logmsgbot: LocalisationUpdate completed (1.22wmf21) at Thu Oct 24 02:42:50 UTC 2013
02:22 logmsgbot: LocalisationUpdate completed (1.22wmf22) at Thu Oct 24 02:22:25 UTC 2013
01:36 logmsgbot: kaldari synchronized wmf-config/mobile.php 'updating for MobileFrontend schema changes'
01:32 logmsgbot: kaldari synchronized php-1.22wmf21/extensions/MobileFrontend/ 'updating MobileFrontend for cherrypick'
01:29 logmsgbot: kaldari synchronized php-1.22wmf22/extensions/MobileFrontend/ 'updating MobileFrontend for cherrypick'
01:16 logmsgbot: kaldari synchronized php-1.22wmf22/extensions/MobileFrontend/ 'updating MobileFrontend for cherrypick'
00:43 logmsgbot: kaldari synchronized wmf-config/mobile.php 'readding mobile schema config for now'
00:22 logmsgbot: kaldari synchronized php-1.22wmf21/extensions/MobileFrontend/ 'syncing MobileFrontend 1.22wmf21'
00:21 logmsgbot: kaldari synchronized php-1.22wmf22/extensions/MobileFrontend/ 'syncing MobileFrontend 1.22wmf22'
00:17 logmsgbot: kaldari synchronized wmf-config/mobile.php 'MobileFrontend schema configs moved to MobileFrontend extension'

October 23

23:55 awight_: payments updated from 300fa32 to f69c37d
21:16 Ryan_Lane: upgrading salt to 0.17.1
20:14 logmsgbot: yurik synchronized php-1.22wmf22/extensions/ZeroRatedMobileAccess/
19:53 hashar: all mw Jenkins jobs now have $wgShowExceptionDetails enabled \O/ [[bugzilla:55595|bug 55595]]
19:35 hashar: Zuul: restarting to apply upgrade.
19:32 hashar: upgrading Zuul on gallium to add a debug statement (tag: wmf-deploy-20131023 commit: 1e3adfd) See https://www.mediawiki.org/wiki/Zuul#upgrading for upgrade procedure used.
19:26 Ryan_Lane: deploying change 91319 to git deployment system
18:54 cmjohnson1: db1033 powering down to reseat DIMM removed from pool according to RT5830
18:35 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php
18:01 logmsgbot: yurik synchronized php-1.22wmf21/extensions/ZeroRatedMobileAccess/
17:47 cmjohnson1: dns update
16:59 cmjohnson1: ms-be1006 replacing disk at slot 2
15:52 logmsgbot: mark synchronized wmf-config/squid.php 'Update squid list for ulsfo'
15:40 mark: Sending OC mobile traffic to ulsfo
15:05 hashar: jenkins updating mediawiki-core-phpunit-databaseless to no more rely on ant
14:55 hashar: jenkins: updating job mediawiki-core-phpunit-api to no more rely on ant
14:21 ^d: elastic: rebuilding all indexes from screen in terbium
12:49 paravoid: restarting apaches, "stray config": mw1117 mw1126 mw1145 mw1201 mw1206
07:43 apergos: mw125: stopped apache shortly after adding back to dns, (needed to wait for an hour for the update to reach the pmtpa recursors so puppet could run), at first successful puppet run mw-sync completed so this host should be good to go now
05:54 ori-l: sync-file: 'Could not resolve hostname mw125: Name or service not known'
05:53 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I11c90ed5a: Added some purge/thumbnail rate limits'
05:30 apergos: powercycled srv291, unresponsive to ping, no login at mgmt console
02:47 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Oct 23 02:47:16 UTC 2013
02:24 logmsgbot: LocalisationUpdate completed (1.22wmf21) at Wed Oct 23 02:24:39 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.22wmf22) at Wed Oct 23 02:16:06 UTC 2013
00:30 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'warmup db1034'

October 22

23:52 mwalker: actually updating payments listeners... 243ec9c to 15b4723
23:28 logmsgbot: catrope synchronized php-1.22wmf22/extensions/VisualEditor 'Update VisualEditor for cherry-picks'
23:28 logmsgbot: catrope synchronized php-1.22wmf21/extensions/VisualEditor 'Update VisualEditor for cherry-picks'
23:20 mwalker: updating listeners from 243ec9c for smashpig recovery
23:05 paravoid: powercycling ms-be1011, locked up, no messages in console
23:05 logmsgbot: mwalker synchronized php-1.22wmf22/extensions/CentralNotice 'Updating CentralNotice to masterish'
22:51 K4-713: updated payments cluster to 300fa32
22:24 logmsgbot: ori synchronized php-1.22wmf22/resources/mediawiki/mediawiki.inspect.js 'I0e7a47b5a: mediawiki.inspect: add CSS report (4/4)'
22:23 logmsgbot: ori synchronized php-1.22wmf22/resources/mediawiki/mediawiki.js 'I0e7a47b5a: mediawiki.inspect: add CSS report (3/4)'
22:23 logmsgbot: ori synchronized php-1.22wmf21/resources/mediawiki/mediawiki.inspect.js 'I0e7a47b5a: mediawiki.inspect: add CSS report (2/4)'
22:23 logmsgbot: ori synchronized php-1.22wmf21/resources/mediawiki/mediawiki.js 'I0e7a47b5a: mediawiki.inspect: add CSS report (1/4)'
22:08 logmsgbot: ori synchronized wmf-config/CommonSettings.php 'I6f6e0ffa3: Update beta to use loginwiki for SUL (bug 55760)'
22:07 ori-l: during previous sync-file: could not resolve mw125; time out connecting to srv291
22:07 logmsgbot: ori synchronized wmf-config/Bug54847.php 'Ide07296c5: Fix error message for recycled passwords (bug 56002)'
21:38 logmsgbot: reedy synchronized php-1.22wmf22/extensions/Wikibase
21:37 logmsgbot: reedy synchronized php-1.22wmf22/extensions/RelatedSites
21:08 Ryan_Lane: pushed git deployment changes 91151 and 91155 to production
20:30 bblack: varnish: banned 'obj.http.content-length == 33518' on text varnishes everywhere (extract2.php leakage)
19:54 Ryan_Lane: updating git deployment system with change 90277 (0424d6f7)
19:23 cmjohnson1: dns update
18:42 logmsgbot: bsitu synchronized php-1.22wmf21/extensions/Echo/modules/overlay 'touch'
18:35 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'touch'
18:08 logmsgbot: bsitu synchronized echowikis.dblist 'Enable Echo and Thanks on all wikis except dewiki and itwiki'
18:08 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable Echo and Thanks on all wikis except dewiki and itwiki'
18:01 mutante: restarting a few Apaches that didnt get restarted by graceful-all (hrmmpf)
17:51 awight: civicrm updated 41b621a to 6f370cc
17:23 mark: Banned Content-Type == "application/x-httpd-php" in text Varnish (eqiad)
17:19 mutante: apache graceful-all
17:06 mutante: sync-apache, graceful to unify www portal docroots
16:31 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Touch'
16:26 logmsgbot: demon synchronized cirrus.dblist 'foundationwiki gets cirrus'
16:19 logmsgbot: reedy synchronized docroot/wwwportal
10:09 apergos: powercycled sq80, hung on mgmt console
09:42 hashar: Zuul: no more voting on code-review label [[bugzilla:55757|bug 55757]] 91157. Will make the database happier I guess
09:30 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool db1018'
03:46 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool db1007, warm up db1041'
03:06 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Oct 22 03:06:46 UTC 2013
03:00 springle: upgrading db1017 to precise
02:39 logmsgbot: LocalisationUpdate completed (1.22wmf21) at Tue Oct 22 02:39:32 UTC 2013
02:25 logmsgbot: LocalisationUpdate completed (1.22wmf22) at Tue Oct 22 02:25:12 UTC 2013
01:40 springle: start xtrabackup clone db1007 to db1041
01:26 springle: start xtrabackup clone db1018 to db1034
01:06 springle: upgrading db1034 & db1041 to precise
00:53 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1007 while cloning'
00:46 awight: crm schema change: wmf_civicrm 6008->6009
00:40 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1018 while cloning'
00:36 awight: manual schema change on crm: update system set schema_version=6008 where name='wmf_civicrm';
00:24 awight: fundraising-crm updated from 9247de6 to 41b621a

October 21

23:22 logmsgbot: mwalker synchronized php-1.22wmf21/extensions/CentralNotice/ 'Reverting earlier change'
23:21 logmsgbot: mwalker synchronized php-1.22wmf22/extensions/CentralNotice/ 'Reverting earlier change'
23:16 mwalker: CentralNotice deploy had an issue where we are now pushing ALL traffic to the mobile site for CN -- ganglia reports a spike and I'm now reverting the change
23:09 ^demon|lunch: elastic: index created for bnwiki, running force indexing in 4 processes on terbium
23:07 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'bnwiki gets cirrus as secondary'
23:05 logmsgbot: mwalker synchronized php-1.22wmf22/extensions/CentralNotice/ 'Updating CentralNotice to master'
23:05 logmsgbot: mwalker synchronized php-1.22wmf21/extensions/CentralNotice/ 'Updating CentralNotice to master'
22:49 paravoid: powercycling ms-be1004, locked up(?)
22:13 logmsgbot: maxsem synchronized php-1.22wmf22/extensions/MobileFrontend/ 'https://gerrit.wikimedia.org/r/91065'
22:04 AaronSchulz: Running syncFilebackend on all wikis (should reduce sync errors; a few a popping up in the logs)
21:34 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Added eqiad swift to multiwrite backends'
21:06 cmjohnson1: morebots are you working?
21:04 cmjohnson1: dns update
20:05 cmjohnson1: dns update
19:38 cmjohnson1: dns update
18:40 logmsgbot: reedy synchronized wmf-config/interwiki.cdb 'Updating interwiki cache'
18:34 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.22wmf22
18:29 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
17:44 cmjohnson1: reclaiming cp1021-36, cp1041-42 per RT5981
17:24 Reedy: The @Wikimedia Operations team is seeking proposals for a new datacenter in the midwestern/western US: https://blog.wikimedia.org/2013/10/21/rfp-new-datacenter-continental-us/
16:30 ori-l: ran sync-apache for I1113b9594 & Ie73ce6213; verified on mw60; gracefuling.
16:26 logmsgbot: demon synchronized wmf-config/CirrusSearch-production.php 'LVS for cirrus on all wikis'
16:11 logmsgbot: demon synchronized wmf-config/CirrusSearch-production.php 'LVS for cirrus on test2wiki'
16:07 RobH: updated akismet blog plugin
16:03 RobH: tweaked permissions of the blog's w3 caching plugin to actually make it work again
11:47 mark: Shutdown csw2-esams:xe-2/1/1 (1 DF leg)
11:31 hashar: Gerrit replication have been broken since Oct 19th roughly 20:50 UTC. [[bugzilla:55948|bug 55948]]
10:43 hashar: Gerrit replication is broken, side effects: git.wm.o shows outdated trees and Jenkins might be missing some commits
09:16 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
09:16 logmsgbot: reedy synchronized docroot/bits/apple-touch/
09:03 hashar: restarting Zuul
09:02 hashar: Jenkins restarted / upgraded
08:59 hashar: rerestarting Jenkins.
08:51 akosiaris: forced verified +2 on gerrit 90874 since jenkins is being upgraded
08:46 hashar: jenkins: upgrading plugins
08:43 hashar: stopping Zuul again. Need to upgrade Jenkins plugins
08:41 hashar: restarted Zuul
08:00 hashar: stopping Zuul / Jenkins
07:59 hashar: upgrading Jenkins for scheduled maintenance
06:15 apergos: moved older swift replication logs from copper:/root to iron:/root/swift-repl/ (now gzipped), copper was full
03:21 Tim: on mw1109: stopped apache to test cgconfig
02:39 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Oct 21 02:39:00 UTC 2013
02:19 logmsgbot: LocalisationUpdate completed (1.22wmf22) at Mon Oct 21 02:19:25 UTC 2013
02:10 logmsgbot: LocalisationUpdate completed (1.22wmf21) at Mon Oct 21 02:10:47 UTC 2013
00:38 ^d: bringing gerrit down to troubleshoot replication

October 20

02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Oct 20 02:07:08 UTC 2013
02:02 logmsgbot: LocalisationUpdate completed (1.22wmf22) at Sun Oct 20 02:02:20 UTC 2013
02:01 logmsgbot: LocalisationUpdate completed (1.22wmf21) at Sun Oct 20 02:01:39 UTC 2013

October 19

20:54 ^d: gerrit: installed 2.7-rc2-507-g1e7090b, service back up
13:14 hashar: Jenkins: updating Zuul config 90695
07:13 ori-l: reports of geoiplookup timing out in AU at enwiki VPT: https://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28technical%29#geoiplookup.wikimedia.org
02:30 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Oct 19 02:30:42 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.22wmf22) at Sat Oct 19 02:15:49 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.22wmf21) at Sat Oct 19 02:08:11 UTC 2013

October 18

23:39 logmsgbot: reedy synchronized docroot
23:17 logmsgbot: reedy synchronized docroot and w
23:14 cmjohnson1: dns update
22:27 logmsgbot: aaron synchronized php-1.22wmf22/includes/job '5ac886a'
22:26 logmsgbot: aaron synchronized php-1.22wmf22/includes/HashRing.php '5ac886a'
22:21 mutante: deleting deprecated postrewrites.conf from Apaches, syncing, gracefulling for cleanup changes
20:44 mutante: sync-apache, graceful'ing ..
20:09 cmjohnson1: dns update
19:23 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'db1035 (s3) back into the pool'
19:11 logmsgbot: marc synchronized wmf-config/InitialiseSettings-labs.php 'Enable zero extension in labs (beta)'
18:52 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'db1019 (s3) to normal weight in pool'
17:56 cmjohnson1: disconnecting ps1-b6-eqiad from network
16:59 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'All wikis in cirrus group get cirrus'
16:58 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Use new cirrus.dblist'
16:57 logmsgbot: demon synchronized cirrus.dblist 'New cirrus.dblist'
15:46 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'db1019 (s3) in pool warming up after upgrade/conversion to mariadb'
13:45 logmsgbot: mark synchronized wmf-config/squid.php 'Update cache list'
09:53 hashar: Jenkins finally producing code coverage reports: https://integration.wikimedia.org/cover/mediawiki-core/master/php/
09:06 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'db1035 (s3) depooled to use as clone source'
06:30 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'remove Haproxy after S7 master switch'
06:25 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1039 for master switch'
06:21 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'insert temporary HAproxy for S7 master rotation'
05:28 logmsgbot: springle synchronized wmf-config/db-pmtpa.php 'sync new S6 master setting to pmtpa'
05:27 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'remove HAproxy after S2 master switch'
05:09 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool new master db1036 from slaves ready for HAproxy write-traffic switch'
05:00 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'insert HAproxy for S2 master rotation'
03:10 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Oct 18 03:10:02 UTC 2013
02:44 logmsgbot: springle synchronized wmf-config/db-pmtpa.php 's3 master rotation done'
02:43 logmsgbot: LocalisationUpdate completed (1.22wmf22) at Fri Oct 18 02:43:45 UTC 2013
02:43 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's3 master rotation done'
02:37 logmsgbot: springle synchronized wmf-config/db-pmtpa.php 's3 master rotation for upgrade'
02:33 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's3 master rotation for upgrade'
02:15 logmsgbot: LocalisationUpdate completed (1.22wmf21) at Fri Oct 18 02:15:02 UTC 2013
01:19 logmsgbot: awight synchronized php-1.22wmf21/extensions/CentralNotice
01:02 logmsgbot: awight synchronized php-1.22wmf21/extensions/CentralNotice
01:01 logmsgbot: awight synchronized php-1.22wmf21/extensions/CentralNotice
00:34 logmsgbot: awight synchronized php-1.22wmf21/extensions/CentralNotice
00:31 Tim: fixing MW cgroup on mw1109
00:24 logmsgbot: awight synchronized php-1.22wmf21/extensions/CentralNotice
00:04 mutante: upgrading bugzilla to 4.2.7

October 17

23:13 logmsgbot: reedy synchronized wmf-config 'touch'
22:58 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wtf
22:38 logmsgbot: reedy synchronized php-1.22wmf21/extensions/Translate
22:35 logmsgbot: reedy synchronized php-1.22wmf22/extensions/Translate
22:17 awight: applied schema change CentralNotice/patches/patch-campaign_throttle.sql
20:23 logmsgbot: mflaschen synchronized wmf-config/InitialiseSettings.php 'Growth team deploy: End GettingStarted A/B test'
19:57 ^d: mw.org elastic index finished rebuilding
19:49 cmjohnson1: authdns update
19:41 mutante: installing package upgrades on kaulen and zinc
19:01 ^d: rebuilt elastic indexes for test/test2/testwikidata/loginwiki, mw.org reindex in progress
18:39 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'loginwiki gets cirrus, for real this time'
18:38 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'loginwiki gets cirrus'
18:13 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki, loginwiki, testwiki, test2wiki and testwikidatawiki to 1.22wmf22
18:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.22wmf21
18:03 logmsgbot: reedy synchronized wmf-config/
17:58 logmsgbot: reedy synchronized php-1.22wmf22/extensions/UniversalLanguageSelector
17:58 Jeff_Green: dist-upgrade and reboot most fundraising hosts
17:55 cmjohnson1: ms-be1008 removing/replacing hdd slot 2 /dev/sdc
17:54 cmjohnson1: ms-be1003 removing/replacing hdd slot 3 /dev/sdd
17:46 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.22wmf21 till window
17:35 logmsgbot: reedy Finished syncing Wikimedia installation... : testwiki to 1.22wmf22 and build l10n cache
17:08 logmsgbot: reedy Started syncing Wikimedia installation... : testwiki to 1.22wmf22 and build l10n cache
16:50 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php '26ee33e'
16:50 logmsgbot: reedy synchronized php-1.22wmf22 'staging'
16:32 MaxSem: Recreating GeoData index
16:32 logmsgbot: reedy synchronized docroot and w
16:20 logmsgbot: reedy synchronized docroot and w
16:01 logmsgbot: reedy synchronized multiversion/
16:00 logmsgbot: reedy synchronized multiversion/
15:58 logmsgbot: reedy synchronized multiversion/
15:57 logmsgbot: reedy synchronized multiversion/
14:30 Jeff_Green: dist-upgrade & reboot payments*
09:53 logmsgbot: maxsem synchronized php-1.22wmf20/extensions/MobileFrontend/
09:52 logmsgbot: maxsem synchronized php-1.22wmf21/extensions/MobileFrontend/
03:58 springle: testing updateCollation on S6 with adaptive hash off
02:50 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Oct 17 02:50:36 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.22wmf20) at Thu Oct 17 02:28:15 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.22wmf21) at Thu Oct 17 02:15:04 UTC 2013
01:51 logmsgbot: marc synchronized wmf-config/InitialiseSettings.php 'bug 55432: add templateeditor restriction level and user right to enwiki'

October 16

23:38 logmsgbot: maxsem synchronized php-1.22wmf20/extensions/MobileFrontend/
23:36 logmsgbot: maxsem synchronized php-1.22wmf21/extensions/MobileFrontend/
23:35 mutante: re-enabling puppet on zinc
23:08 mutante: ddsh -g apaches -F20 "rm /etc/apache2/wmf/en2.conf"
21:27 paravoid: staggered reboot of lvs4003/lvs4001
20:57 apergos: removing also from /etc/dsh/group on tin: yaseo.text yaseo.upload srv284 (note, none of these files are managed by puppet)
20:53 apergos: deleting from /etc/dsh/group on tin these 0 byte files not in puppet: ams.puppettmp_7604 apache-eqiad.puppettmp_2863 apaches-eqiad cp esams mail.ini.list
20:34 paravoid: powercycling lvs4003, locked up(?)
20:25 logmsgbot: olivneh synchronized php-1.22wmf21/extensions/WikimediaEvents 'Updating WikimediaEvents for VE logging (2/2)'
20:25 logmsgbot: olivneh synchronized php-1.22wmf20/extensions/WikimediaEvents 'Updating WikimediaEvents for VE logging (1/2)'
20:17 logmsgbot: olivneh synchronized php-1.22wmf21/extensions/VisualEditor 'Updating VE for cherry-picks I54602394e & I3b58ce0f4'
20:17 logmsgbot: olivneh synchronized php-1.22wmf21/extensions/WikimediaEvents 'Updating WikimediaEvents for VE logging (2/2)'
20:17 logmsgbot: olivneh synchronized php-1.22wmf20/extensions/WikimediaEvents 'Updating WikimediaEvents for VE logging (1/2)'
20:14 mutante: removing unused apache config files via dsh, they have been merged "D" but sync-apache doesn't delete things
18:58 gwicke: updated Parsoid to 9b52271
18:25 mutante: DNS update - remove williams
18:02 mutante: shutting down williams (former OTRS), stab
17:57 mutante: stopping spamassassin on williams
17:57 logmsgbot: olivneh synchronized php-1.22wmf21/extensions/Wikibase 'Update Wikibase to 8903c2234d'
17:44 logmsgbot: olivneh synchronized php-1.22wmf21/resources/startup.js 'Touch startup.js'
17:43 logmsgbot: olivneh synchronized php-1.22wmf21/skins/vector/screen.less 'I412575f37fix for bug 55779'
17:23 Aaron|home: Creating/setting ACLs on eqiad swift containers for all wikis
17:21 logmsgbot: demon synchronized php-1.22wmf21/includes/LinksUpdate.php 'I115f8cf9'
17:21 logmsgbot: demon synchronized php-1.22wmf21/includes/DefaultSettings.php 'I115f8cf9'
17:12 Nemo_bis: fr.wikipedia was briefly down: Frakir> [19:01:27] <rc-pmtpa> - no editions - [19:05:23] <rc-pmtpa>
16:38 logmsgbot: demon synchronized php-1.22wmf20/includes/LinksUpdate.php 'I115f8cf9'
16:37 logmsgbot: demon synchronized php-1.22wmf20/includes/DefaultSettings.php 'I115f8cf9'
16:01 cmjohnson1: authdns update
14:11 hashar: upgrading php on lanthanum.eqiad.wmnet ( jenkins CI slave )
14:11 hashar: jenkins : upgrading php on gallium.
11:58 hashar: Jenkins: migrate extensions jobs to install mediawiki with sqlite with a shell script instead of inlined shell in the job configuration 90104
02:29 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Oct 16 02:29:45 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.22wmf20) at Wed Oct 16 02:16:35 UTC 2013
02:09 logmsgbot: LocalisationUpdate completed (1.22wmf21) at Wed Oct 16 02:09:25 UTC 2013
00:02 logmsgbot: reedy synchronized docroot and w
00:00 logmsgbot: reedy synchronized wmf-config/

October 15

23:21 logmsgbot: maxsem synchronized php-1.22wmf20/extensions/MobileFrontend/
23:19 logmsgbot: maxsem synchronized php-1.22wmf21/extensions/MobileFrontend/
23:12 logmsgbot: catrope synchronized php-1.22wmf21/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.ViewPageTarget.js 'cherry-pick of https://gerrit.wikimedia.org/r/#/c/89773/'
22:55 logmsgbot: maxsem synchronized php-1.22wmf20/extensions/MobileFrontend/
22:53 logmsgbot: maxsem synchronized php-1.22wmf21/extensions/MobileFrontend/
22:42 mutante: DNS update - removing decom'ed search servers1-12
22:22 logmsgbot: aaron synchronized wmf-config/filebackend.php '013c272'
21:56 awight: deploying donor_review module and schema
21:55 awight: deployed civicrm from b4392c7 to 9247de6 (occurred at 1857 UTC)
20:35 mutante: DNS update, add uk.planet
19:30 cmjohnson1: adding mw1125 back, enabling pybal
18:03 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files:
17:54 manybubbles: rebuilt search index for enwikisource's non-content documents because it had snuck out of date and was causing trouble
17:49 mutante: enable icinga notifications for db1006 services
17:42 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'db1022 (s6) back in the pool'
16:42 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'enwikisource getting cirrus as primary'
16:36 logmsgbot: bd808 synchronized php-1.22wmf21/maintenance/purgeChangedPages.php 'Updating purgeChangedPages.php to prepare for HTCP purge'
16:31 logmsgbot: bd808 synchronized php-1.22wmf20/maintenance/purgeChangedPages.php
16:28 logmsgbot: bd808 synchronized php-1.22wmf20/maintenance/purgeChangedPages.php
16:23 logmsgbot: bd808 synchronized php-1.22wmf20/maintenance/purgeChangedPages.php
15:04 logmsgbot: reedy synchronized php-1.22wmf21/extensions/Wikibase
14:35 logmsgbot: reedy synchronized multiversion/ 'Changing away from docroot based site deduction'
13:31 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'db1006 (s6) back to full weight in pool'
11:37 hashar: jenkins refreshing all mediawiki extensions jobs so they log Junit results under /logs/
10:29 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'db1006 (s6) repooled, warming up after upgrade/conversion'
06:56 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'db1022 (s6) depooled to use as clone source'
05:18 apergos: powercycled cp1052, it was unresponsive and unreachable via mgmt
02:37 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Oct 15 02:37:39 UTC 2013
02:20 logmsgbot: LocalisationUpdate completed (1.22wmf21) at Tue Oct 15 02:20:18 UTC 2013
02:14 logmsgbot: LocalisationUpdate completed (1.22wmf20) at Tue Oct 15 02:14:53 UTC 2013
01:53 Tim: rebooting mw1035 for init testing
00:41 Tim: manually created MW cgroups on all apaches since apparently the init script is totally broken
00:29 Tim: on mw1035,mw1145,mw1078,mw1152,mw1150: running "start cgconfig" to fix cgroups
00:21 Tim: some apaches are missing /sys/fs/cgroup/* so shell execution from MW fails. Stopped apache on mw1035 during investigation

October 14

23:46 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
23:42 logmsgbot: tstarling synchronized php-1.22wmf20/extensions/Math
23:41 Tim: pushed out temporary checkout of extension/Math 679ce7fe2fe, will update submodule soon
23:40 logmsgbot: tstarling synchronized php-1.22wmf20/extensions/Math/MathTexvc.php
23:14 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Disable Math debug log group (found to be not pertinent & too verbose)'
22:21 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Enable Math debug log group'
16:11 mark: Setup LVS service IP range aggregates on the core routers
06:45 apergos: resized /var/lib/puppet on stafford, it ran out of room last night (but recovered after cron job reports cleanup)
05:18 logmsgbot: springle synchronized wmf-config/db-pmtpa.php 'update s6 master'
05:17 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool db1007'
04:34 logmsgbot: springle synchronized wmf-config/db-pmtpa.php 'update s6 master'
04:24 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1006 to investigate connection errors. promote db1027'
03:33 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1027 before promotion'
02:25 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Oct 14 02:25:24 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.22wmf21) at Mon Oct 14 02:15:08 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.22wmf20) at Mon Oct 14 02:08:37 UTC 2013

October 13

19:31 ori-l: restarting gmetad on nickel to clear old metrics (again)
18:19 ori-l: restarting gmetad on nickel to clear old metrics
02:27 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Oct 13 02:27:58 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.22wmf21) at Sun Oct 13 02:16:34 UTC 2013
02:09 logmsgbot: LocalisationUpdate completed (1.22wmf20) at Sun Oct 13 02:09:00 UTC 2013

October 12

21:11 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php
02:29 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Oct 12 02:29:58 UTC 2013
02:17 logmsgbot: LocalisationUpdate completed (1.22wmf21) at Sat Oct 12 02:17:30 UTC 2013
02:09 logmsgbot: LocalisationUpdate completed (1.22wmf20) at Sat Oct 12 02:09:44 UTC 2013
00:32 mwalker: trying again to upgrade payments donatoin interface from 73fe0ca (this time without error pages)

October 11

23:46 mwalker: updating payments donationinterface from 73fe0ca -- removing adyen from random selection and deploying error pages
21:51 Ryan_Lane: updated git-deploy (change 88934)
20:45 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php
18:27 Reedy: Updating collation on frwiki as reedy on tin in screen session. Gonna take a while
18:26 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:26 Reedy: Updated collation on viwikivoyage
18:24 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:22 logmsgbot: reedy synchronized database lists files:
18:20 logmsgbot: reedy synchronized wmf-config/
18:19 logmsgbot: reedy synchronized php-fatal-error.html
17:59 Reedy: Created EducationProgram tables on eswiki
17:56 Reedy: Created FlaggedRevs tables on ptwiki
17:22 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
17:21 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
15:36 akosiaris: removed libicu42 from apt.wikimedia.org
15:26 logmsgbot: reedy synchronized php-1.22wmf20/maintenance/purgeChangedPages.php
14:13 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'db1024 (s7) back to full weight in pool'
12:02 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'db1024 (s7) warming up after upgrade/conversion to mariadb'
10:31 mutante: DNS update, add pk.wikimedia and id.planet.wikimedia
10:26 mutante: sync-apache and graceful for pk.wm redirect
10:14 mutante: mass unsubscribed users, then deleted wikimedia-commits list, has been closed down, no archives
02:18 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Oct 11 02:18:46 UTC 2013
02:13 logmsgbot: LocalisationUpdate completed (1.22wmf21) at Fri Oct 11 02:13:55 UTC 2013
02:01 logmsgbot: LocalisationUpdate completed (1.22wmf20) at Fri Oct 11 02:01:46 UTC 2013
00:45 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1021 while cloning to db1045'

October 10

23:29 paravoid: reprepro include php5 5.3.10-1ubuntu3.8+wmf2 (wmf1 rebuilt in a clean environment, i.e. with precise's libicu)
22:24 logmsgbot: olivneh synchronized php-1.22wmf20/resources/Resources.php 'Updating 1.22wmf20 for mw.loader.inspect (3/3)'
22:23 logmsgbot: olivneh synchronized php-1.22wmf20/resources/mediawiki/mediawiki.js 'Updating 1.22wmf20 for mw.loader.inspect (2/3)'
22:23 logmsgbot: olivneh synchronized php-1.22wmf20/resources/mediawiki/mediawiki.inspect.js 'Updating 1.22wmf20 for mw.loader.inspect (1/3)'
22:23 logmsgbot: olivneh synchronized php-1.22wmf21/extensions/WikimediaEvents 'Updating WikimediaEvents to log DOM retrieval timing for VE'
21:28 ori-l: stopping StatsD service on tungsten for a couple of minutes to debug
20:18 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Oct 10 20:18:49 UTC 2013
20:07 logmsgbot: LocalisationUpdate completed (1.22wmf21) at Thu Oct 10 20:07:11 UTC 2013
19:35 logmsgbot: LocalisationUpdate completed (1.22wmf20) at Thu Oct 10 19:35:14 UTC 2013
18:58 logmsgbot: reedy synchronized wmf-config/
18:52 logmsgbot: reedy synchronized php-1.22wmf21/extensions/MobileFrontend/MobileFrontend.php
18:50 logmsgbot: reedy synchronized wmf-config/
18:47 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: phase1 wikis back to 1.22wmf21
18:47 logmsgbot: reedy synchronized php-1.22wmf20/extensions/MobileFrontend
18:41 logmsgbot: reedy synchronized php-1.22wmf21/extensions/
18:32 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: phase1 wikis back to 1.22wmf20
18:30 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: phase1 wikis to 1.22wmf21
18:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.22wmf20
17:37 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.22wmf20
17:34 logmsgbot: reedy Finished syncing Wikimedia installation... : testwiki to 1.22wmf21 and build and sync l10ncache
17:16 logmsgbot: reedy Started syncing Wikimedia installation... : testwiki to 1.22wmf21 and build and sync l10ncache
17:06 logmsgbot: reedy synchronized docroot and w
17:03 logmsgbot: reedy synchronized php-1.22wmf21 'staging'
15:46 mark: Imported firewall ACLs on cr2-ulsfo, activated on lo0.0
15:46 mark: Configured cr2-ulsfo:ae1 with all sub interfaces and VRRP
15:12 hashar: jenkins : triggering operations-puppet-rspec (non voting) on operations/puppet.git 89000
15:04 andrewbogott: removed mysql-server-5.5 package from stat1 to ensure that it doesn't get used now that it's no longer maintained by puppet.
15:01 mark: Corrected wrong interface addresses on cr1-ulsfo:ae0 sub-units
14:38 mark: Setup iBGP between cr1-ulsfo and cr2-ulsfo
14:38 mark: Configured AS65003 ulsfo BGP confederation on cr1-ulsfo and cr2-ulsfo
14:25 mark: Setup OSPF and OSPF3 on cr1-ulsfo:ae0.2 <--> cr2-ulsfo:ae0.2
14:24 mark: Setup cr1-ulsfo:ae0 <--> cr2-ulsfo:ae0
12:04 mark: Rebooting scs-ulsfo, pmshell is segfaulting
11:52 akosiaris: upgraded php5 packages from php5_5.3.10-1ubuntu3.6+wmf1 to php5_5.3.10-1ubuntu3.8+wmf1 on apt.wikimedia.org
08:27 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'depool db1024 (s7) for conversion to mariadb'
08:16 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'db1039 (s7) to normal weight in pool'
06:50 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'warm up db1039 in s7'
05:16 springle: start xtrabackup clone db1039 to db1007
04:19 springle: upgrading db1007 to precise + mariadb
02:28 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Oct 10 02:28:22 UTC 2013
02:18 logmsgbot: LocalisationUpdate completed (1.22wmf19) at Thu Oct 10 02:18:06 UTC 2013

October 9

16:40 RobH: puppetmaster was overloaded and returning 500s, tried killing open processes and restarting puppetmaster on stafford
13:19 springle: start dump db1007 to db1039 for S7 file per table
10:57 mark: Non-Wikipedia projects are now on Varnish for eqiad non-ipv6, non-https traffic
09:34 hashar: jenkins back up
09:10 hashar: gallium : zuul backup, waiting for jenkins to complete start up
08:24 ori-l: Stopping EventLogging, then rebooting vanadium for kernel upgrade
08:02 hashar: rebooting gallium
07:18 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool db1022 in S6. move db1039 to assist with upgrades in S7. depool db1007 for upgrade'
06:12 logmsgbot: aaron synchronized php-1.22wmf19/includes 'c6d64f5488c124636ab46712e1d104e5c7076325'
02:45 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Oct 9 02:44:59 UTC 2013
02:30 logmsgbot: LocalisationUpdate completed (1.22wmf19) at Wed Oct 9 02:30:06 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.22wmf20) at Wed Oct 9 02:16:08 UTC 2013
00:52 logmsgbot: krinkle synchronized php-1.22wmf19/extensions/VisualEditor 'I03d68280ddd9506'
00:49 logmsgbot: krinkle synchronized php-1.22wmf10/extensions/VisualEditor 'I85bce4d40e430318'

October 8

23:29 logmsgbot: mflaschen synchronized php-1.22wmf20/extensions/GettingStarted/ 'Deploy GettingStarted to wmf20 for A/B test'
23:29 logmsgbot: mflaschen synchronized php-1.22wmf19/extensions/GettingStarted/ 'Deploy GettingStarted to wmf19 for A/B test'
23:28 logmsgbot: mflaschen synchronized wmf-config/InitialiseSettings.php 'A/B test for GettingStarted'
23:27 logmsgbot: mflaschen synchronized wmf-config/CommonSettings.php 'A/B test for GettingStarted'
22:26 cmjohnson1: reinstalling amssq48-62
22:25 LeslieCarr: moved fxp0 on cr1-esams into MGMT logical system
21:44 logmsgbot: aaron synchronized php-1.22wmf20/includes/LinksUpdate.php '337272176758b06c0ed9fd8859d358730152c155'
21:40 logmsgbot: aaron synchronized php-1.22wmf20/includes/WikiPage.php '337272176758b06c0ed9fd8859d358730152c155'
20:27 logmsgbot: maxsem synchronized php-1.22wmf20/extensions/MobileFrontend/javascripts/loggingSchemas/MobileWebClickTracking.js
20:05 logmsgbot: maxsem synchronized php-1.22wmf19/extensions/MobileFrontend/javascripts/loggingSchemas/MobileWebClickTracking.js
19:16 logmsgbot: bsitu synchronized php-1.22wmf19/extensions/Echo 'Touch Js file'
19:11 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable Echo and Thanks on various wikis'
19:06 logmsgbot: bsitu synchronized echowikis.dblist 'Enable Echo and Thanks on various wikis'
19:06 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable Echo and Thanks on various wikis'
18:55 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo and Thanks to Master
18:36 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo and Thanks to Master
18:29 logmsgbot: reedy synchronized php-1.22wmf20/includes/Wiki.php
17:25 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php '2 shards per replica in elastic'
17:23 logmsgbot: olivneh synchronized wmf-config/extension-list 'CoreEvents -> WikimediaEvents'
17:22 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'CoreEvents -> WikimediaEvents'
17:18 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'CoreEvents -> WikimediaEvents'
17:15 logmsgbot: olivneh synchronized php-1.22wmf19/extensions/WikimediaEvents 'CoreEvents -> WikimediaEvents'
17:12 logmsgbot: olivneh synchronized php-1.22wmf20/extensions/WikimediaEvents 'CoreEvents -> WikimediaEvents'
16:58 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'db1046 (s6) back to normal weight in pool'
16:48 logmsgbot: reedy synchronized wmf-config/
15:54 apergos: fixed up salt clients on analytics 1005-6, 1009-27
14:34 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'warm up db1040 (s6) after upgrade'
13:38 mutante: installing package upgrades on hooper
13:28 mutante: installing package upgrades on helium
12:49 mutante: installing package upgrades on formey
12:37 hashar: restarted Jenkins by mistake :-(
12:35 mutante: install package upgrades on gallium (jenkins)
12:35 akosiaris: apt-get purge defoma on image_scalers dsh group
12:17 mutante: installing package upgrades on ekrem (IRC)
09:43 mutante: installing package upgrades on antimony (gitblit, gerrit repl.)
09:39 mutante: puppetstoredconfigclean.rb .. killing williams.wikimedia.org .. done
09:11 mutante: installing package upgrades on calcium (cameras)
07:52 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'warm up db1039 in s6'
07:20 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'depool db1040 for upgrade/conversion to mariadb'
06:37 apergos: fixed up salt on mw1046, mw1072, mw1173
03:30 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's6 db1015 to full steam'
03:23 springle: xtrabackup clone s6 db1039 to db1022
03:20 springle: rmmod nf_conntrack on db1002 causing mass mysql connect failure
02:35 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Oct 8 02:35:56 UTC 2013
02:31 springle: start online optimize logging indexes s4 & s5
02:25 logmsgbot: LocalisationUpdate completed (1.22wmf19) at Tue Oct 8 02:25:27 UTC 2013
02:13 logmsgbot: LocalisationUpdate completed (1.22wmf20) at Tue Oct 8 02:13:54 UTC 2013
00:11 LeslieCarr: all ams-ix traffic is now on cr2-esams

October 7

23:36 LeslieCarr: turning up new ams-ix port on cr2-esams
23:24 LeslieCarr: traffic draining from ams-ix on cr2-knams
23:22 LeslieCarr: moving european traffic around
20:46 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Oct 7 20:46:05 UTC 2013
20:33 logmsgbot: LocalisationUpdate completed (1.22wmf19) at Mon Oct 7 20:33:53 UTC 2013
20:23 logmsgbot: LocalisationUpdate completed (1.22wmf20) at Mon Oct 7 20:23:45 UTC 2013
20:05 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
19:35 logmsgbot: reedy synchronized wmf-config/
18:56 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commonswiki back to 1.22wmf20, thumb.php fixed
18:55 logmsgbot: reedy synchronized php-1.22wmf20/thumb.php
18:39 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commonswiki back to 1.22wmf19
18:38 ottomata: backporting python-docopt 0.6.1 for precise and including in our apt repo
18:24 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything non 'pedia to 1.22wmf20
18:17 logmsgbot: reedy synchronized php-1.22wmf20/extensions/Wikibase
17:43 paravoid: changed ms-fe1/2's mgmt IPs, ip clash with nas1-a/b's e0M
17:28 mwalker: payments update from I3a09af2cbd8566bbeb431437c49e72c832b87304 to I61bc80f257c590c1abeb716145c23921b005e5a8
16:05 logmsgbot: krinkle synchronized wmf-config/InitialiseSettings.php 'I59e2547002bac5'
14:50 RoanKattouw: Restarting Parsoid on wtp10[01-24] on request; load avg reaching 90%
14:28 Coren: Labstore4 Configuring; will bounce up and down over the next two days.
12:57 akosiaris: brought icinga back up, after manually running puppet on neon
12:39 hashar: gallium : restarting Zuul.
12:37 hashar: gallium / Zuul : cherry picked a change from OpenStack related to statsd metrics. Our change: 88063
11:30 mutante: new public mailing list for the Wikimania Steering Commitee - wikimania-com
11:25 hashar: gallium : zuul restarted and apparently starting statsd metrics.
11:03 hashar: gallium : killed a bunch of stalled jenkins/java threads
11:02 hashar: gallium : restarting Zuul to enable statsd reporting 86744
10:41 paravoid: rebooting labstore3, NFS lockupo
10:18 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1021'
10:10 apergos: powercycling srv291
09:44 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'warm up db1015'
03:19 springle: xtrabackup clone db43 to db1022
02:04 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
00:59 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1022 for upgrade'

October 6

02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Oct 6 02:07:25 UTC 2013
02:02 logmsgbot: LocalisationUpdate completed (1.22wmf20) at Sun Oct 6 02:02:50 UTC 2013
02:02 logmsgbot: LocalisationUpdate completed (1.22wmf19) at Sun Oct 6 02:02:03 UTC 2013

October 5

02:25 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Oct 5 02:25:46 UTC 2013
02:19 logmsgbot: LocalisationUpdate completed (1.22wmf20) at Sat Oct 5 02:19:16 UTC 2013
02:06 logmsgbot: LocalisationUpdate completed (1.22wmf19) at Sat Oct 5 02:06:38 UTC 2013

October 4

18:44 paravoid: reinstalling erzurumi
16:30 logmsgbot: maxsem Finished syncing Wikimedia installation... : https://gerrit.wikimedia.org/r/87584
16:29 logmsgbot: aaron synchronized php-1.22wmf20/includes/filerepo/file/LocalFile.php 'b48403885d5ac993cb3b3ce7ac7580002ab12e1f'
16:15 logmsgbot: maxsem Started syncing Wikimedia installation... : https://gerrit.wikimedia.org/r/87584
15:35 mark: Added another 10G link to cr1-esams:ae1 <--> csw2-esams:ae1, now 40Gbps total
15:03 mark: Completely disconnected csw1-esams from the network
12:12 mark: Inserted 2x 10G MIC into cr1-esams FPC 0, brought online
08:00 springle: upgrading db1039 to precise + mariadb
07:54 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool db1042'
07:11 apergos: powercycling maerlant, load > 100, couldn't log in via mgmt console
07:09 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool db1035 after upgrade, db1003 to full steam'
06:23 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool db1003 after upgrade'
06:14 Tim: on fenari: running an analysis script for bug 53687 against tampa slave DB servers
03:32 springle: manual sync-common on mw1072
03:19 springle: started xtrabackup clone from db1035 to db1003
02:42 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Oct 4 02:41:40 UTC 2013
02:35 logmsgbot: LocalisationUpdate completed (1.22wmf20) at Fri Oct 4 02:35:00 UTC 2013
02:31 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'refresh mw1072 db-eqiad.php'
02:16 logmsgbot: LocalisationUpdate completed (1.22wmf19) at Fri Oct 4 02:16:10 UTC 2013
02:15 springle: upgrading db1003 to precise + mariadb
02:08 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1003'
01:32 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'db1038 to s3'
01:28 logmsgbot: catrope synchronized php-1.22wmf20/extensions/VisualEditor/ 'Fix SyntaxHighlight icon'

October 3

23:24 logmsgbot: catrope synchronized php-1.22wmf20/extensions/VisualEditor 'Update VisualEditor for cherry-picks'
23:24 logmsgbot: catrope synchronized php-1.22wmf19/extensions/VisualEditor 'Update VisualEditor for cherry-picks'
22:50 paravoid: starting swiftrepl process for pmtpa->eqiad (originals), running on copper
22:50 logmsgbot: mflaschen Finished syncing Wikimedia installation... : Deploy GettingStarted for growth team
22:39 logmsgbot: mflaschen Started syncing Wikimedia installation... : Deploy GettingStarted for growth team
22:38 awight_: updated crm from cf90889b3728e1bfb2175574640a18511d60e2b9 to b4392c76d21287f835076f761436fc7386845a19
21:59 LeslieCarr: restarting pybal on lvs1003 (not 1006)
21:59 LeslieCarr: restarting pybal on lvs1006
21:59 LeslieCarr: restarting pybal on lvs1006
21:45 awight: updated crm from 7690ee9f80c6e53d946195e453cf17e197f3138a to cf90889b3728e1bfb2175574640a18511d60e2b9
21:25 awight: updated civicrm 6affe8a9cee57f76602a25d60d96edc7eaaae4b2 to 7690ee9f80c6e53d946195e453cf17e197f3138a
20:42 logmsgbot: krinkle synchronized php-1.22wmf19/extensions/VisualEditor/modules/ve/ve.Element.js 'touch for bug 54935'
20:27 logmsgbot: krinkle synchronized php-1.22wmf19/resources/startup.js 'Attempt to fix bug 54935'
19:32 logmsgbot: reedy synchronized php-1.22wmf20/extensions/MassMessage/
19:12 ori-l: CentralAuth: UPDATE `bug_54847_password_resets` SET `r_logged_out` = 1 WHERE `r_reset` IS NOT NULL;
18:39 logmsgbot: reedy synchronized php-1.22wmf20/includes/DefaultSettings.php
18:34 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki, test2wiki, mediawikiwiki, loginwiki and testwikidatawiki to 1.22wmf20
18:28 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.22wmf19
17:45 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php
17:19 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.22wmf19 till deploy time
17:13 logmsgbot: reedy synchronized php-1.22wmf20/extensions/VisualEditor
17:09 logmsgbot: reedy Finished syncing Wikimedia installation... : testwiki to 1.22wmf20 and build l10n cache
16:48 logmsgbot: reedy Started syncing Wikimedia installation... : testwiki to 1.22wmf20 and build l10n cache
16:41 logmsgbot: reedy synchronized docroot and w
16:40 logmsgbot: reedy synchronized php-1.22wmf20 'Staging'
16:14 bblack: updated varnish to -wm17 on active mobile caches (cp1046, cp1047, cp1059, cp1060, cp3011, cp3012), fixes 0.0.0.0 assertfail in netmapper
13:47 Jeff_Green: c state disabled on barium, ref. RT 5555
12:14 manybubbles: enwikisource's search index finished building late last night. all english indecies have been updated to include kstem.
08:02 ori-l: Bug 54847: e-mail script completed successfully
06:14 logmsgbot: olivneh synchronized wmf-config/Bug54847.php
05:49 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'if ( $wmgBug54847 && $wmgUseCentralAuth )'
05:47 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'wmgBug54847: default => true, private => false'
05:41 logmsgbot: csteipp Finished syncing Wikimedia installation... :
05:40 logmsgbot: olivneh synchronized wmf-config/Bug54847.php
05:36 Tim: on searchidx2: switched java procs to "idle" ionice class, to improve scap time
04:57 logmsgbot: csteipp Started syncing Wikimedia installation... :
02:59 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Oct 3 02:59:41 UTC 2013
02:35 logmsgbot: LocalisationUpdate completed (1.22wmf18) at Thu Oct 3 02:35:56 UTC 2013
02:20 logmsgbot: LocalisationUpdate completed (1.22wmf19) at Thu Oct 3 02:20:48 UTC 2013
02:11 Tim: on mw1125: server very slow due to disk issues, stopped apache, will shut down
00:34 manybubbles: all english indecies rebuilt except enwikisource. that one will take a while.
00:11 manybubbles1: rebuilding search search indecies on english wikis after CirrusSearch deploy that updated English configuration

October 2

23:16 logmsgbot: demon synchronized php-1.22wmf19/extensions/CirrusSearch 'Cirrus to master'
23:15 logmsgbot: demon synchronized php-1.22wmf18/extensions/CirrusSearch 'Cirrus to master'
22:56 logmsgbot: csteipp synchronized php-1.22wmf18/includes/specials/SpecialUserlogin.php
22:44 logmsgbot: csteipp synchronized php-1.22wmf19/includes/specials/SpecialUserlogin.php 'AbortLogin message fix'
21:28 gwicke: updated Parsoid to 01ebbfd
21:20 gwicke: updated parsoid dependencies
14:25 mutante: sync-apache, apache-graceful-all for wikimania2014 redirect
10:30 MaxSem: Resynched MW on mw1125, looked like slightly out of sync
05:15 superm401: Commons has intermittent search failures: "An error has occurred while searching: HTTP request timed out."
02:56 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Oct 2 02:56:05 UTC 2013
02:30 logmsgbot: LocalisationUpdate completed (1.22wmf18) at Wed Oct 2 02:30:23 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.22wmf19) at Wed Oct 2 02:16:19 UTC 2013

October 1

23:24 logmsgbot: catrope synchronized php-1.22wmf19/extensions/VisualEditor 'Update VisualEditor for cherry-pick of I7bddcfa9'
23:23 logmsgbot: catrope synchronized php-1.22wmf18/extensions/VisualEditor 'Update VisualEditor for cherry-pick of I7bddcfa9'
22:55 LeslieCarr: added search to pybal eqiad backends
21:54 logmsgbot: maxsem synchronized php-1.22wmf19/extensions/MobileFrontend/
21:52 logmsgbot: maxsem synchronized php-1.22wmf18/extensions/MobileFrontend/
21:26 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
21:12 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c/87008/'
21:03 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
20:55 mwalker: last update for SSL seemed to go ok; now updating payments for payment methods from 1cce4e1bcd45171289594ddcf0165a049c3264a1 to fdc9c2b3e3898ac3e7dc738dc6f4ed140b3ddb41
20:48 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
20:46 mwalker: updating payments from b952e9e9656d62ee8b3a70e96ae5eb45918068ce to 1cce4e1bcd45171289594ddcf0165a049c3264a1
18:21 ottomata: backported jq_1.2-8 to precise and imported into apt
15:22 Jeff_Green: dist-upgrade and reboot loudon
15:16 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Enable CirrusSearch extra splitting for mw.org.'
13:12 springle: start xtrabackup s3 db1035 to db1038
09:58 logmsgbot: maxsem synchronized docroot and w
09:50 logmsgbot: yurik synchronized php-1.22wmf19/extensions/ZeroRatedMobileAccess/
09:48 logmsgbot: yurik synchronized php-1.22wmf18/extensions/ZeroRatedMobileAccess/
09:23 mutante: mw1173 had wrong nrpe init script, synced the one from mw1172 and restarted (as nagios), fixes dpkg check
09:11 mutante: killing nrpe on cp1045 and restarting, was running as icinga instead of nagios, fixes "NRPE: Unable to read output" Icinga Unknown's
08:31 hashar: Jenkins: so pyflakes are now actually doing something useful instead of testing nothing. That causes a bunch of jobs to fail
08:28 hashar: jenkins updating pyflakes jobs to run 'pyflakes .' instead of simply 'pyflakes' 86727
02:50 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Oct 1 02:50:54 UTC 2013
02:26 logmsgbot: LocalisationUpdate completed (1.22wmf18) at Tue Oct 1 02:25:59 UTC 2013
02:13 logmsgbot: LocalisationUpdate completed (1.22wmf19) at Tue Oct 1 02:13:45 UTC 2013
00:32 logmsgbot: mwalker synchronized php-1.22wmf19/extensions/CentralNotice/ 're pushing centralnotice to master on the CORRECT branch... grumble'
00:16 mwalker: added china to list of paypal xclick countries -- no longer using donation workflow -- for ChinaUnionPay

September 30

23:52 logmsgbot: mwalker synchronized php-1.22wmf18/extensions/CentralNotice/ 'Reverting previous centralnotice change since it appears to have made the bug worse...'
23:44 logmsgbot: mwalker synchronized php-1.22wmf18/extensions/CentralNotice/ 'Updating CentralNotice to master'
23:07 mwalker: updating payments from 78cbf0d4d2024cff02d4f57ff21c1ffb478f16ee to b952e9e9656d62ee8b3a70e96ae5eb45918068ce for adyen, paypal xclick, and globalcollect cvv
22:41 logmsgbot: reedy Finished syncing Wikimedia installation... : MassMessage
22:36 LeslieCarr: changing local pref for 54058 to downpref tinet directly
22:20 logmsgbot: reedy Started syncing Wikimedia installation... : MassMessage
21:59 logmsgbot: reedy synchronized php-1.22wmf19/extensions/MassMessage 'Staging code'
20:41 RobH: db1014 is coming offline to become tungsten, please disregard any alerts
20:28 Jeff_Green: samarium dist-upgrade and reboot
18:23 mwalker: updating payments listeners (smashpig) from f73e931e8a30c8dbc30899918f4344b34e777d03 to 243ec9ca483993386aa99a10f872c8afa097a548
18:20 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.22wmf19
17:55 awight: civicrm updated from c086eb74eca0ebe513d564abbe2b4b62c0608fe9 to 6affe8a9cee57f76602a25d60d96edc7eaaae4b2
17:46 awight: civicrm updated from 1f659b73fdd to c086eb74ec
16:16 logmsgbot: catrope synchronized php-1.22wmf19/extensions/UniversalLanguageSelector 'bug fixes'
16:16 logmsgbot: catrope synchronized php-1.22wmf19/extensions/VisualEditor 'bug fixes'
16:16 logmsgbot: catrope synchronized php-1.22wmf18/extensions/VisualEditor 'bug fixes'
16:01 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable VE by default for logged-in users on bg, ca, ceb, cs, da, el, et, eu, fi, gl, hr, hu, id, lv, ms, nap, nn, no, simple, sk, sl, tr, uk, vec, vo, war Wikipedias'
15:30 mutante: sync-apache mw1041 and restart apache, was missing in apaches dsh group
13:57 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'REDOING (forgot to rebase): wiktionary logos form 135px to 128px so they fit in the 160px tall element. Ping Nemo_bis and Dereckson.'
12:43 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'wiktionary logos form 135px to 128px so they fit in the 160px tall element. Ping Nemo_bis and Dereckson.'
11:51 logmsgbot: springle synchronized wmf-config/AdminSettings.php 'switch admin account back with new password'
10:14 logmsgbot: springle synchronized wmf-config/AdminSettings.php 'switch admin account, attempt #2'
10:06 logmsgbot: springle synchronized wmf-config/AdminSettings.php 'revert admin account. errors'
10:03 logmsgbot: springle synchronized wmf-config/AdminSettings.php 'switch to temp admin db account'
09:41 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'Point ptwiktionary logo to commons 85516 bug 53685'
02:45 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Sep 30 02:45:10 UTC 2013
02:27 logmsgbot: LocalisationUpdate completed (1.22wmf19) at Mon Sep 30 02:27:03 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.22wmf18) at Mon Sep 30 02:15:09 UTC 2013
01:48 springle: upgrading db1038 to precise + mariadb
01:04 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool db1049'

September 29

23:56 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1049'
23:20 springle: powercycling db1049, soft lockup, unresponsive
20:51 paravoid: swift@pmtpa: pushing rings to set 10.0.6.204/sdm1 10.0.6.204/sdn1 10.0.6.208/sde1 weight from 33 to 66
02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Sep 29 02:07:37 UTC 2013
02:02 logmsgbot: LocalisationUpdate completed (1.22wmf19) at Sun Sep 29 02:02:22 UTC 2013
02:01 logmsgbot: LocalisationUpdate completed (1.22wmf18) at Sun Sep 29 02:01:40 UTC 2013

September 28

02:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Sep 28 02:08:41 UTC 2013
02:02 logmsgbot: LocalisationUpdate completed (1.22wmf19) at Sat Sep 28 02:02:32 UTC 2013
02:01 logmsgbot: LocalisationUpdate completed (1.22wmf18) at Sat Sep 28 02:01:52 UTC 2013

September 27

23:55 ori-l: Manually synchronized changes 86329, 86330, 86331 to hafnium & professor & temporarily disabled puppet to prevent it from clobbering the changes. I'll re-enable once they're merged.
22:35 logmsgbot: olivneh synchronized php-1.22wmf19/extensions/NavigationTiming 'Updating NavigationTiming: schema update 5336845 -> 5832704 (2/2)'
22:35 logmsgbot: olivneh synchronized php-1.22wmf18/extensions/NavigationTiming 'Updating NavigationTiming: schema update 5336845 -> 5832704 (1/2)'
21:25 logmsgbot: reedy synchronized php-1.22wmf19/extensions/WikimediaMessages/
20:57 logmsgbot: ariel synchronized wmf-config/InitialiseSettings.php 'fix up ns refs for ukwikimedia'
20:30 logmsgbot: ariel synchronized wmf-config/InitialiseSettings.php 'ukwikimedia to read-only except for user talk'
19:18 ori-l: restarting apache on professor (graphite)
19:13 logmsgbot: reedy synchronized php-1.22wmf19/extensions/TorBlock/
18:18 logmsgbot: reedy synchronized php-1.22wmf19/includes/GlobalFunctions.php 'Revert'
18:04 logmsgbot: reedy synchronized php-1.22wmf19/includes/GlobalFunctions.php 'all of the debugging'
17:53 logmsgbot: reedy synchronized php-1.22wmf19/includes/GlobalFunctions.php 'all of the debugging'
17:42 logmsgbot: reedy synchronized php-1.22wmf19/includes/GlobalFunctions.php 'all of the debugging'
17:39 logmsgbot: reedy synchronized php-1.22wmf19/includes/GlobalFunctions.php 'all of the debugging'
17:31 logmsgbot: reedy synchronized php-1.22wmf19/includes/GlobalFunctions.php 'all of the debugging'
17:27 cmjohnson1: powering down analytics1021 replacing disk1 (sdb)
16:38 cmjohnson1: mw1072 replacing hard drive
13:02 paravoid: dismantling & repurposing ceph cluster
02:55 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Sep 27 02:55:41 UTC 2013
02:33 logmsgbot: LocalisationUpdate completed (1.22wmf19) at Fri Sep 27 02:33:19 UTC 2013
02:14 logmsgbot: LocalisationUpdate completed (1.22wmf18) at Fri Sep 27 02:14:34 UTC 2013

September 26

23:27 ^d: elastic: reindex all articles that erroneously had <audio> tags indexed
23:11 Ryan_Lane: deploying DNS change to OpenStackManager on virt0, also doing DIT change for all public/private DNS entries in labs
23:01 mwalker: updating fundraising crm from bf4c7090099a866389105a3a38be4f950dd78b89 to 1f659b73fddd26939081d21c3ff4098a77589e61
22:56 logmsgbot: yurik synchronized php-1.22wmf19/extensions/ZeroRatedMobileAccess/
22:46 logmsgbot: yurik synchronized php-1.22wmf18/extensions/ZeroRatedMobileAccess/
22:25 ori-l: installed package python-argparse on professor (lucid) to get git-deploy's 'grain-ensure' to run.
21:49 awight: updated crm from 8180a18a7cb9a4bf64f5a8a7e192ca6f8f910a0a to bf4c7090099a866389105a3a38be4f950dd78b89
21:41 awight: update crm from 1c10b4b181925d882 to 8180a18a7cb9a4bf64f5
21:29 logmsgbot: mflaschen synchronized php-1.22wmf19/extensions/GuidedTour/modules/mediawiki.libs.guiders/mediawiki.libs.guiders.css 'Sync GuidedTour file for Growth team deployment'
21:28 logmsgbot: mflaschen synchronized php-1.22wmf18/extensions/GuidedTour/modules/mediawiki.libs.guiders/mediawiki.libs.guiders.css 'Sync GuidedTour file for Growth team deployment'
19:48 logmsgbot: reedy synchronized php-1.22wmf19/extensions/GlobalBlocking/
19:39 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: loginwiki back to 1.22wmf19
19:34 logmsgbot: reedy Finished syncing Wikimedia installation... : revisionsize
19:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: put loginwiki back to 1.22wmf18
19:23 logmsgbot: reedy Started syncing Wikimedia installation... : revisionsize
19:17 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki, test2wiki, testwikidatawiki, loginwiki and mediawikiwiki to 1.22wmf19
19:09 logmsgbot: reedy synchronized php-1.22wmf19/includes/
18:53 logmsgbot: reedy synchronized php-1.22wmf19/extensions/TorBlock
18:52 logmsgbot: reedy synchronized php-1.22wmf19/extensions/GlobalBlocking
18:49 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: revert 1.22wmf19 wikis
18:40 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki, test2wiki, testwikidatawiki, loginwiki and mediawikiwiki to 1.22wmf19
18:15 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.22wmf18
18:14 mwalker: updating fr crm from aa79ff40f3ce1534d899c21c81dfb914caad3fae to 1c10b4b181925d882ada0024011e28f75b781390
18:14 cmjohnson1: db1006 replacing disk 1
18:08 cmjohnson1: db1004 removing/replacing disk 2
17:39 logmsgbot: reedy Finished syncing Wikimedia installation... : testwiki to 1.22wmf19 and build l10n cache
17:25 logmsgbot: reedy Started syncing Wikimedia installation... : testwiki to 1.22wmf19 and build l10n cache
17:17 logmsgbot: reedy synchronized docroot and w
17:15 logmsgbot: reedy synchronized php-1.22wmf19 'Initial sync v2'
17:13 logmsgbot: reedy synchronized php-1.22wmf19 'Initial sync'
16:10 hashar: Jenkins: deleted /tmp/mw-UIDGenerator-UID-* files on gallium.
15:32 mark: Migrated OSPF between cr1-ulsfo and cr2-eqiad to p2p interface-type, removed high metric, TEMP annotation
15:28 mark: Setup OSPFv3 between cr1-ulsfo and cr2-eqiad
02:45 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Sep 26 02:45:41 UTC 2013
02:25 logmsgbot: LocalisationUpdate completed (1.22wmf17) at Thu Sep 26 02:25:14 UTC 2013
02:13 logmsgbot: LocalisationUpdate completed (1.22wmf18) at Thu Sep 26 02:13:22 UTC 2013
02:09 springle: testing MHA on pmtpa db37, db56, db58, db68. disabled their notifications'

September 25

23:32 logmsgbot: krinkle Finished syncing Wikimedia installation... :
23:21 logmsgbot: krinkle Started syncing Wikimedia installation... :
23:17 logmsgbot: krinkle synchronized php-1.22wmf17/extensions/VisualEditor 'I3653efbe7b6841c70f352da526f4946fa0a4c76f'
23:12 logmsgbot: krinkle synchronized php-1.22wmf18/extensions/VisualEditor 'I3653efbe7b6841c70f352da526f4946fa0a4c76f'
23:10 logmsgbot: krinkle synchronized php-1.22wmf18/extensions/TemplateData 'I3653efbe7b6841c70f352da526f4946fa0a4c76f'
20:31 logmsgbot: csteipp synchronized php-1.22wmf17/extensions/CentralAuth 'update to master for SUL fix'
20:28 logmsgbot: csteipp synchronized php-1.22wmf18/extensions/CentralAuth 'update to master for SUL fix'
19:13 cmjohnson: authdns update
16:52 ottomata: taking down analytics100[34] for reinstall with public IPs
16:51 mutante: install a few package upgrades on iron
16:24 mutante: added a favicon.ico on stats.wikimedia.org for hashar
16:11 ottomata: authdns-update to give analytics100[34] public IPs, about to reinstall them
15:21 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
13:43 mutante: apt-get install php5-cli on mw1046 fixes 'ensure: change from 5.3.10-1ubuntu3.8 to 5.3.10-1ubuntu3.6+wmf1 failed' issue after reinstall
13:22 mutante: re-enabling disabled puppet on (fixed) mw1046
11:04 mutante: running checksetup.pl on bugzilla, installed InlineHistory extension
09:35 6JTAABPF7: Put all *.wikimedia.org, wikidata, wikivoyage eqiad traffic on Varnish
05:12 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1035 for upgrade'
05:05 logmsgbot: tstarling cleared profiling data
03:47 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool db1018'
02:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Sep 25 02:48:19 UTC 2013
02:25 logmsgbot: LocalisationUpdate completed (1.22wmf17) at Wed Sep 25 02:25:30 UTC 2013
02:13 logmsgbot: LocalisationUpdate completed (1.22wmf18) at Wed Sep 25 02:13:00 UTC 2013
01:40 Tim: restarted wikibugs
00:55 springle: started xtrabackup clone db1002 to db1018

September 24

23:45 springle: upgrading db1018 to precise + mariadb
23:23 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1018 for upgrade, pool db1002 and warm up'
22:41 logmsgbot: maxsem synchronized php-1.22wmf18/extensions/MobileFrontend/
22:39 logmsgbot: maxsem synchronized php-1.22wmf17/extensions/MobileFrontend/
22:09 logmsgbot: maxsem synchronized php-1.22wmf18/extensions/MobileFrontend/
22:07 logmsgbot: maxsem synchronized php-1.22wmf17/extensions/MobileFrontend/
21:59 logmsgbot: maxsem synchronized php-1.22wmf18/extensions/MobileFrontend/
21:57 logmsgbot: maxsem synchronized php-1.22wmf17/extensions/MobileFrontend/
21:48 LeslieCarr: routing changes in ulsfo, commit confirmed 5 for safety
21:45 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
21:34 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
19:35 logmsgbot: reedy synchronized wmf-config 'touch'
19:19 logmsgbot: bsitu synchronized echowikis.dblist 'Enable Echo and Thanks on various wikis'
19:07 logmsgbot: bsitu synchronized echowikis.dblist 'Enable Echo and Thanks on various wikis'
19:06 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable Echo and Thanks on various wikis'
18:55 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo and Thanks to master
18:54 cmjohnson: apache-graceful on mw1046
18:53 bblack: varnish updated to -wm16 on all eqiad mobile caches now, yell at me if something starts looking "off" there...
18:53 cmjohnson: sync'ing mw1046
18:42 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo and Thanks to master
18:09 bblack: upgraded varnish pkg to -wm16 on cp1046
17:59 RobH: bast4001 reinstall
17:58 manybubbles: finished building search indexes on closed wikis
17:49 LeslieCarr: Hurricane electric peering moved to 10GE port. the spice packets must flow...
17:14 MaxSem: Recreating GeoData index
16:38 manybubbles: rebuilt search indecies in place for all other wikis runnig cirrus except cawiki and enwikisource. skipped them because they are already up to date.
16:27 cmjohnson: adding mw1046 back to pybal
16:24 manybubbles: rebuilding search indecies in place for all other wikis running cirrus
16:19 mutante: DNS update, add wikipediastories domains
16:18 mutante: sync-apache, graceful for wikipediastories
16:16 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable CleanChanges on meta'
16:10 manybubbles: building search indecies for all closed wikis
16:06 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php 'Cirrus on closed wikis'
15:25 logmsgbot: demon synchronized php-1.22wmf18/extensions/CirrusSearch
15:24 mutante: freshly activated http://wikimediastories.org/ (and .com,.net)
15:21 mutante: sync-apache and graceful-all for wikimediastories and other cleanup
15:21 logmsgbot: yurik synchronized php-1.22wmf18/extensions/ZeroRatedMobileAccess/
15:13 mutante: during sync-apache: mw1072 - read-only file system
15:07 cmjohnson: db1053 removing/replacing disk 3
15:05 logmsgbot: yurik synchronized php-1.22wmf17/extensions/ZeroRatedMobileAccess/
14:59 cmjohnson: db1001 removing/replacing disk 7
14:48 cmjohnson: db1031 removing/replacing disk 7
14:38 cmjohnson: replacing mw1046 hard drive --requires reinstall
14:21 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Sep 24 14:21:10 UTC 2013
14:15 logmsgbot: LocalisationUpdate completed (1.22wmf17) at Tue Sep 24 14:15:56 UTC 2013
14:15 logmsgbot: LocalisationUpdate completed (1.22wmf18) at Tue Sep 24 14:15:16 UTC 2013
14:13 logmsgbot: demon synchronized php-1.22wmf18/extensions/Elastica 'Sync message fixes'
12:26 mutante: shutting down sq45
11:04 mutante: powercycled sq45
10:08 p858snake|l: [20:01AEST] <mark> upped to weight 5
10:08 p858snake|l: [19:51AEST] <mark> !log Pooled Varnish Text caches in eqiad with weight 1 (affecting wikidata, wikivoyage, *.wikimedia.org)
10:07 p858snake|l: [19:36AEST] <akosiaris> !log rebooted labstore4, controller configuration lost ?
10:07 p858snake|l: [19:36AEST] <akosiaris> !log rebooted labstore3 (thought the 14days NFS issue was resolved???)
10:04 akosiaris: rebooted labstore4, controller configuration lost ?
10:04 akosiaris: rebooted labstore3 (thought the 14days NFS issue was resolved???)
08:30 hashar: restarting Jenkins again for one last plugin upgrade
08:30 springle: labstore4 disk issues, /a failed mount in console, ssh key bouncing. awaiting input from ryan or coren
08:30 hashar: 8:20am INFO: Jenkins is fully up and running
08:20 hashar: restarting Jenkins
08:12 hashar: jenkins: upgrading plugins
08:12 hashar: 8:12am INFO: Jenkins is fully up and running
08:06 springle: labstore4 locked up, power cycled
07:43 hashar: upgrading Jenkins on gallium from 1.509.2 to 1.509.3
06:36 springle: upgrading db1002 to precise and mariadb
06:13 ^demon|zzz: ES: enwikisource & cawiki done indexing. Really need to revisit batch indexing, way too slow.
03:42 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1002 for upgrade, db1036 take up slack'
03:17 logmsgbot: springle synchronized wmf-config/db-eqiad.php
02:40 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Sep 24 02:40:38 UTC 2013
02:21 logmsgbot: LocalisationUpdate completed (1.22wmf17) at Tue Sep 24 02:21:27 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.22wmf18) at Tue Sep 24 02:15:01 UTC 2013

September 23

23:36 RoanKattouw: mw1072 is unhappy, sync-file throws warnings about read-only filesystem
23:35 logmsgbot: catrope synchronized php-1.22wmf18/extensions/VisualEditor 'VisualEditor cherry-picks'
23:35 logmsgbot: catrope synchronized php-1.22wmf17/extensions/VisualEditor 'VisualEditor cherry-picks'
23:20 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Set VisualEditor to opt-in on enwiki'
20:57 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php
20:48 logmsgbot: reedy synchronized wmf-config/ 'touch'
20:46 logmsgbot: reedy synchronized php-1.22wmf18/extensions/UploadWizard
20:09 Reedy: mw1072 has a r/o filesystem
20:05 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Cirrus as primary for itwikt, secondary on cawiki and enwikisource'
19:46 logmsgbot: reedy synchronized wmf-config 'touch'
19:36 logmsgbot: reedy synchronized database lists files:
19:35 logmsgbot: reedy synchronized wmf-config/
18:50 Ryan_Lane: enabling cirrussearch for wikitech
18:43 Reedy: Created sites and site_identifier tables on commonswiki
18:41 Ryan_Lane: upgrading wikitech-static to 1.22wmf18
18:41 Ryan_Lane: upgraded wikitech to 1.22wmf18

September 22

02:27 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Sep 22 02:27:32 UTC 2013
02:14 logmsgbot: LocalisationUpdate completed (1.22wmf18) at Sun Sep 22 02:14:05 UTC 2013
02:07 logmsgbot: LocalisationUpdate completed (1.22wmf17) at Sun Sep 22 02:07:46 UTC 2013

September 21

18:42 hashar: Jenkins: upgrading pep8 from 1.4.5 to 1.4.6 on gallium and lanthanum
03:21 Reedy: Scalers fixed
03:16 logmsgbot: reedy synchronized docroot and w
03:07 hashar: imagescalers in eqiad shows a drop of CPU/network at around 1:40am UTC. Thumbnails are no more rendered :-(
02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Sep 21 02:07:36 UTC 2013
02:02 logmsgbot: LocalisationUpdate completed (1.22wmf18) at Sat Sep 21 02:02:26 UTC 2013
02:01 logmsgbot: LocalisationUpdate completed (1.22wmf17) at Sat Sep 21 02:01:44 UTC 2013
01:38 logmsgbot: reedy synchronized docroot and w
00:53 logmsgbot: reedy synchronized wmf-config/throttle.php
00:45 logmsgbot: reedy synchronized docroot and w
00:43 logmsgbot: reedy synchronized docroot and w
00:41 logmsgbot: reedy synchronized docroot and w
00:40 logmsgbot: reedy synchronized docroot and w
00:38 logmsgbot: reedy synchronized docroot and w
00:23 logmsgbot: reedy synchronized w
00:13 logmsgbot: reedy synchronized w
00:09 logmsgbot: reedy synchronized w

September 20

23:53 logmsgbot: reedy synchronized w/404.php
23:03 springle: upgrading db1044 to precise
22:20 springle: parallel dump db1033 to db1036 with innodb_file_per_table=1
22:10 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Sep 20 22:10:50 UTC 2013
22:06 logmsgbot: LocalisationUpdate completed (1.22wmf18) at Fri Sep 20 22:06:08 UTC 2013
22:05 logmsgbot: LocalisationUpdate completed (1.22wmf17) at Fri Sep 20 22:05:21 UTC 2013
22:02 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Sep 20 22:02:35 UTC 2013
21:47 logmsgbot: LocalisationUpdate completed (1.22wmf18) at Fri Sep 20 21:47:58 UTC 2013
21:26 logmsgbot: LocalisationUpdate completed (1.22wmf17) at Fri Sep 20 21:26:27 UTC 2013
21:08 logmsgbot: demon synchronized php-1.22wmf18/extensions/Elastica
21:06 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files:
21:05 logmsgbot: demon synchronized wmf-config
21:05 logmsgbot: demon synchronized php-1.22wmf18/extensions/CirrusSearch
21:04 logmsgbot: demon synchronized php-1.22wmf18/extensions/Elastica
20:46 logmsgbot: reedy synchronized php-1.22wmf18/extensions/Wikibase
20:45 logmsgbot: reedy synchronized php-1.22wmf18/extensions/WikimediaMessages
20:23 logmsgbot: reedy synchronized php-1.22wmf18/extensions/WikimediaMessages
20:23 logmsgbot: reedy synchronized php-1.22wmf18/extensions/Wikibase
19:20 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php
19:14 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Cirrus on officewiki'
18:52 logmsgbot: reedy synchronized multiversion/
18:51 logmsgbot: reedy synchronized docroot and w
18:49 logmsgbot: reedy synchronized wmf-config/
17:41 logmsgbot: reedy synchronized wmf-config/ 'touch'
17:39 logmsgbot: reedy synchronized database lists files:
17:37 Reedy: Created Echo tables on officewiki
15:58 hashar: gerrit: deleting reference refs/for/master from pywikibot/core.git , confuses some git clients
13:39 hashar: Jenkins: PHPUnit jobs for MediaWiki extensions had an issue for the last couple days. They were being fetched in an incorrect place :/ Fix is https://gerrit.wikimedia.org/r/85202 and got deployed
06:31 logmsgbot: reedy synchronized docroot/bits/
02:44 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Sep 20 02:44:51 UTC 2013
02:24 logmsgbot: LocalisationUpdate completed (1.22wmf18) at Fri Sep 20 02:24:44 UTC 2013
02:12 logmsgbot: LocalisationUpdate completed (1.22wmf17) at Fri Sep 20 02:12:46 UTC 2013
00:46 cmjohnson: rebooting sq45
00:17 springle: db1018 set log_warnings=2 to investigate mw-log/dberror.log frequent "Can't connect to MySQL server on '10.64.16.7'"
00:06 logmsgbot: reedy synchronized images/sul/wikivoyage.png

September 19

22:40 logmsgbot: mflaschen Finished syncing Wikimedia installation... : Growth team deployment: GuidedTour and remove outdated config from CommonSettings.php and InitialiseSettings.php
22:22 logmsgbot: mflaschen Started syncing Wikimedia installation... : Growth team deployment: GuidedTour and remove outdated config from CommonSettings.php and InitialiseSettings.php
22:01 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Sep 19 22:01:13 UTC 2013
21:46 logmsgbot: LocalisationUpdate completed (1.22wmf18) at Thu Sep 19 21:46:29 UTC 2013
21:15 logmsgbot: LocalisationUpdate completed (1.22wmf17) at Thu Sep 19 21:15:17 UTC 2013
20:53 hashar: Jenkins: slave integration-debian-builder.pmtpa.wmflabs now uses /mnt/jenkins-workspace as filesystem root, to avoid relying on GlusterFS.
20:49 manybubbles: rebuilt all search indecies. \o/
20:44 hashar: Jenkins: upgrading jenkins-debian-glue packages on integration-debian-builder.pmtpa.wmflabs
19:57 manybubbles: rebuilding search indecies after push of CirrusSearch
19:03 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki, mediawikiwiki, testwikidatawiki and loginwiki to 1.22wmf18
19:03 logmsgbot: demon synchronized php-1.22wmf18/extensions/CirrusSearch
19:02 logmsgbot: demon synchronized php-1.22wmf17/extensions/CirrusSearch
19:00 logmsgbot: reedy synchronized php-1.22wmf18/extensions/TemplateData
18:46 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki, mediawikiwiki, testwikidatawiki and loginwiki to 1.22wmf17
18:44 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki, mediawikiwiki, testwikidatawiki and loginwiki to 1.22wmf18
18:42 logmsgbot: reedy synchronized php-1.22wmf17/extensions/
18:41 logmsgbot: reedy synchronized php-1.22wmf18/extensions/
18:34 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.22wmf17
18:32 logmsgbot: reedy synchronized php-1.22wmf18/extensions/UniversalLanguageSelector
18:28 logmsgbot: reedy Finished syncing Wikimedia installation... : testwiki to 1.22wmf18 and build l10n cache
18:15 hashar: Jenkins: applying label 'hasContintPackages' to lanthanum, that should make some pep8/pyflakes jobs to be executed on that host
18:12 logmsgbot: reedy Started syncing Wikimedia installation... : testwiki to 1.22wmf18 and build l10n cache
18:10 logmsgbot: reedy synchronized docroot and w
18:07 logmsgbot: reedy synchronized docroot and w
18:05 logmsgbot: reedy synchronized php-1.22wmf18
17:03 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Cirrus as primary'
16:44 ^d: ES: indexes created for testwikidatawiki
16:44 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'testwikidatawiki gets Cirrus'
03:35 hashar: Zuul: updated ssh fingerprint of ytterbium for jenkins user on gallium. That is needed for zuul to clone new repositories.
03:05 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Sep 19 03:05:39 UTC 2013
02:42 logmsgbot: LocalisationUpdate completed (1.22wmf16) at Thu Sep 19 02:42:09 UTC 2013
02:22 logmsgbot: LocalisationUpdate completed (1.22wmf17) at Thu Sep 19 02:22:05 UTC 2013
01:34 hashar: jenkins: migrating test extensions jobs to fetch Zuul ref over git protocol. Tie them to label 'hasPhpUnit' as well since it is only installed on gallium ..
01:32 hashar: jenkins: updating qunit jobs to fetch Zuul ref over git protocol
00:56 RobH: iron coming offline
00:39 hashar: Jenkins: migrating phpcs-HEAD mediawiki extension jobs to use the git protocol when fetching Zuul refs.

September 18

23:51 hashar: Jenkins: migrated mediawiki extension 'lint' jobs to fetch Zuul reference over git:// protocol. Migrating jslint ones.
23:44 hashar: Jenkins applied labels "hasPhpcs" and "hasPhpUnit" to gallium slave
23:09 logmsgbot: catrope synchronized php-1.22wmf17/extensions/VisualEditor/ 'Deploy cherry-picked bug fixes'
22:13 logmsgbot: yurik synchronized php-1.22wmf17/extensions/ZeroRatedMobileAccess/
22:10 logmsgbot: yurik synchronized php-1.22wmf16/extensions/ZeroRatedMobileAccess/
21:15 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable EducationProgram on enwikinews'
21:14 Reedy: Created EducationProgram tables on cswiki
21:10 manybubbles: greg-g showed me how to log stuff
21:10 manybubbles: Upgraded elasticsearch to 0.90.4.
21:09 greg-g: manybubbles told his mom he loved her
21:06 Reedy: Created EducationProgram tables on enwikinews
20:54 logmsgbot: reedy synchronized wmf-config/
20:38 springle: cloning from sanitarium to labsdb1003 instances
20:15 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'ULS EventLogging'
20:10 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'ULS EventLogging'
20:04 hashar: Jenkins: migrated sartoris jobs to fetch Zuul reference over the git protocol made available with [[bugzilla:53683|bug 53683]]. Test change is https://gerrit.wikimedia.org/r/#/c/46450/
19:51 springle: stopping mysqld instances on labsdb1003 for troubleshooting after xfs crash/corruption
19:13 logmsgbot: mwalker synchronized php-1.22wmf17/extensions/CentralNotice/ 'Updating CentralNotice to master'
19:12 logmsgbot: mwalker synchronized php-1.22wmf16/extensions/CentralNotice/ 'Updating CentralNotice to master'
18:44 logmsgbot: csteipp synchronized php-1.22wmf16/extensions/CentralAuth 'update centralauth to master for SUL fixes'
18:44 logmsgbot: csteipp synchronized php-1.22wmf17/extensions/CentralAuth 'update centralauth to master for SUL fixes'
18:14 logmsgbot: reedy synchronized php-1.22wmf17/extensions/TimedMediaHandler
05:10 logmsgbot: reedy synchronized docroot and w
04:05 logmsgbot: reedy synchronized docroot and w
03:54 logmsgbot: reedy synchronized docroot and w
03:00 logmsgbot: reedy synchronized docroot and w
02:51 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Sep 18 02:51:31 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.22wmf16) at Wed Sep 18 02:28:37 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.22wmf17) at Wed Sep 18 02:15:07 UTC 2013
02:15 logmsgbot: reedy synchronized wmf-config/ 'Switch Echo to using dblist for enablement criteria'
02:14 logmsgbot: reedy synchronized database lists files:
02:13 logmsgbot: reedy synchronized docroot/noc/
00:47 mutante: sync-apache, repool mw1044
00:37 logmsgbot: reedy synchronized multiversion/MWMultiVersion.php
00:33 logmsgbot: mwalker synchronized php-1.22wmf17/extensions/CentralNotice 'Updating CentralNotice to master'
00:32 logmsgbot: mwalker synchronized php-1.22wmf16/extensions/CentralNotice 'Updating CentralNotice to master'
00:32 cmjohnson1: dns update
00:30 springle: upgrade, reboot, mariadb db1033
00:29 springle: returned db1050 to the pool
00:21 logmsgbot: springle synchronized wmf-config/db-eqiad.php

September 17

23:49 LeslieCarr: upgrading cp1062 to fix broken packages
23:41 logmsgbot: reedy synchronized multiversion/MWMultiVersion.php
23:26 logmsgbot: lcarr synchronized wmf-config/throttle.php
23:18 RobH: updated blogs, still online \o/
23:13 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Re-enable experimental VisualEditor features on mediawiki.org'
22:20 logmsgbot: mwalker synchronized php-1.22wmf17/extensions/CentralNotice 'Updating CentralNotice to master'
22:19 logmsgbot: mwalker synchronized php-1.22wmf16/extensions/CentralNotice 'Updating CentralNotice to master'
21:51 logmsgbot: kaldari synchronized php-1.22wmf16/extensions/MobileFrontend/javascripts/common 'syncing JS files for mobilefrontend'
21:36 logmsgbot: kaldari Finished syncing Wikimedia installation... :
21:19 ottomata: varnishkafka on cp1048 producing snappy compressed json as of Sept 17 20:42
20:59 logmsgbot: kaldari Started syncing Wikimedia installation... :
18:48 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'enable echo and thanks on bs, nl, et, fa, he, ro, es, simple wiki and enwikivoyage'
18:48 logmsgbot: bsitu synchronized echowikis.dblist 'add bs, nl, et, fa, he, ro, es, simple wiki and enwikivoyage to echodblist'
18:38 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo and Thanks to master
18:30 mutante: depooling mw1044
18:30 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo and Thanks to master
18:21 LeslieCarr: depreffing more peers in order to save the ams-ix port
18:00 springle: db1036 upgrade, reboot, mariadb
17:25 logmsgbot: reedy synchronized php-1.22wmf17/extensions/CirrusSearch/
17:14 springle: depooled db1050 while cloning db1045 delayed slave
17:13 logmsgbot: springle synchronized wmf-config/db-eqiad.php
16:59 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild l10ncache to include Math stuffs
16:36 paravoid: rebooting neon, ext3 errors (corruption?) remounted ro
16:31 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild l10ncache to include Math stuffs
16:03 LeslieCarr: deactivated a few peers on ams-ix in order to reduce traffic on congested port
08:43 logmsgbot: yurik synchronized php-1.22wmf17/extensions/ZeroRatedMobileAccess/
08:41 logmsgbot: yurik synchronized php-1.22wmf16/extensions/ZeroRatedMobileAccess/
05:37 logmsgbot: springle synchronized wmf-config/db-eqiad.php
05:13 springle: deployed db1037 as s1 slave
05:13 logmsgbot: springle synchronized wmf-config/db-pmtpa.php
05:13 logmsgbot: springle synchronized wmf-config/db-eqiad.php
02:50 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Sep 17 02:50:53 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.22wmf16) at Tue Sep 17 02:28:14 UTC 2013
02:25 springle: starting xtrabackup clone for s1 from db1017 to db1037
02:14 logmsgbot: LocalisationUpdate completed (1.22wmf17) at Tue Sep 17 02:14:55 UTC 2013
01:21 Reedy: Changed password for global user "Article Feedback v5" and logged into enwiki to create an account
00:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wmgArticleFeedbackv5AutoArchiveEnabled to false for enwiki'

September 16

23:46 mutante: change timeout for watchmouse "wikimedia foundation script"-check from 8 sec to 15 sec
23:03 logmsgbot: yurik synchronized php-1.22wmf17/extensions/ZeroRatedMobileAccess/
23:00 logmsgbot: yurik synchronized php-1.22wmf16/extensions/ZeroRatedMobileAccess/
22:48 ^d: ES: indexing itwiktionary from terbium
22:43 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php
21:09 logmsgbot: demon synchronized php-1.22wmf17/extensions/CirrusSearch
21:08 logmsgbot: demon synchronized php-1.22wmf17/extensions/CirrusSearch
20:23 logmsgbot: demon synchronized php-1.22wmf17/extensions/CirrusSearch
20:12 mutante: re-enable allow_attachment_display on bugzilla
19:38 mark: Disabled csw1-esams:e8/1
19:05 logmsgbot: reedy synchronized php-1.22wmf17/extensions/VisualEditor
18:38 logmsgbot: reedy synchronized php-1.22wmf16/extensions/VisualEditor
18:31 logmsgbot: reedy synchronized php-1.22wmf17/extensions/DataValues/
18:30 logmsgbot: reedy synchronized php-1.22wmf17/extensions/Wikibase
18:27 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything non 'pedia to 1.22wmf17
18:17 mutante: custom logo for el.wikivoyage
18:17 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
18:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikimedia, special, wikinews and wikivoyage to 1.22wmf16
18:16 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
18:07 ori-l: Starting import of some old EL data into the 'log' database on db1047.
17:10 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php
17:07 logmsgbot: demon synchronized php-1.22wmf17/extensions/CirrusSearch
17:02 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php
17:00 logmsgbot: demon synchronized php-1.22wmf17/extensions/CirrusSearch/CirrusSearchSearcher.php 'bug 54167'
16:13 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php
07:40 springle: rebooted neon. icinga pegged at max concurrent connections for 48h. root ext3 fs read-only as of 2013-09-14 06:45 detected aborted journal
02:23 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Sep 16 02:23:57 UTC 2013
02:11 logmsgbot: LocalisationUpdate completed (1.22wmf17) at Mon Sep 16 02:11:46 UTC 2013
02:06 logmsgbot: LocalisationUpdate completed (1.22wmf16) at Mon Sep 16 02:06:34 UTC 2013

September 15

02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Sep 15 02:07:26 UTC 2013
02:02 logmsgbot: LocalisationUpdate completed (1.22wmf17) at Sun Sep 15 02:02:16 UTC 2013
02:01 logmsgbot: LocalisationUpdate completed (1.22wmf16) at Sun Sep 15 02:01:36 UTC 2013

September 14

23:34 springle: throttling back db1045 pt-table-sync jobs to avoid blocking source-slave sql threads
23:25 springle_away: restarted db1049 slave sql thread after lock wait timeout. investigating cause
02:21 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Sep 14 02:21:42 UTC 2013
02:13 logmsgbot: LocalisationUpdate completed (1.22wmf17) at Sat Sep 14 02:13:11 UTC 2013
02:05 logmsgbot: LocalisationUpdate completed (1.22wmf16) at Sat Sep 14 02:05:31 UTC 2013

September 13

17:48 apergos: testing incr dump code on snapshot1003 in screen; if it falls over it won't break anything
15:38 greg-g: ugh, it's intermittent, 503's now
15:33 greg-g: etherpad.wikimedia seems better to me
15:28 Nemo_bis: etherpad currently serving error 503 Service Temporarily Unavailable
02:54 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Sep 13 02:54:31 UTC 2013
02:33 logmsgbot: LocalisationUpdate completed (1.22wmf17) at Fri Sep 13 02:33:49 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.22wmf16) at Fri Sep 13 02:15:07 UTC 2013

September 12

21:50 LeslieCarr: upgrading to 11.4r9 on cr1-ulsfo/cr2-ulsfo
21:29 logmsgbot: reedy synchronized wmf-config/
21:29 logmsgbot: reedy Finished syncing Wikimedia installation... : Update l10n cache for CleanChanges
21:28 logmsgbot: reedy synchronized wmf-config/
21:19 logmsgbot: reedy Started syncing Wikimedia installation... : Update l10n cache for CleanChanges
21:15 logmsgbot: reedy synchronized wmf-config/extension-list 'Enable CleanChanges'
20:49 logmsgbot: reedy synchronized php-1.22wmf16/extensions/MoodBar/
19:35 mark: Rebalanced eqiad apaches in PyBal weight, according to machine specs
18:59 LeslieCarr: restarting pybal on lvs's
18:57 LeslieCarr: runnign ntpdate on lvs servers
18:33 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: loginwiki, mediawikiwiki, test2wiki and testwikidatawiki to 1.22wmf17
18:28 logmsgbot: reedy Finished syncing Wikimedia installation... : testwiki to 1.22wmf17 and build localisation cache
18:11 logmsgbot: reedy Started syncing Wikimedia installation... : testwiki to 1.22wmf17 and build localisation cache
18:02 logmsgbot: reedy synchronized php-1.22wmf16/extensions/Wikibase
17:56 logmsgbot: reedy synchronized docroot and w
17:55 logmsgbot: reedy synchronized php-1.22wmf17 'Initial deploy of code files'
17:19 hashar: gallium : killed some long lasting java Jenkins thread
17:18 logmsgbot: reedy synchronized php-1.22wmf16/extensions/Wikibase
17:16 logmsgbot: reedy synchronized php-1.22wmf16/extensions/DataValues/
04:18 logmsgbot: reedy synchronized php-1.22wmf16/extensions/TemplateData
04:00 logmsgbot: catrope synchronized php-1.22wmf16/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.Target.js 'touch'
03:49 logmsgbot: reedy synchronized php-1.22wmf16/extensions/TemplateData/TemplateData.php
02:51 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Sep 12 02:51:54 UTC 2013
02:43 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.22wmf16
02:32 logmsgbot: LocalisationUpdate completed (1.22wmf15) at Thu Sep 12 02:32:02 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.22wmf16) at Thu Sep 12 02:16:32 UTC 2013

September 11

23:22 paravoid: powercycling neon, swapdeath
23:10 logmsgbot: mwalker synchronized wmf-config/CommonSettings.php 'Do not run Translate message group job synchronously for CentralNotice'
22:50 logmsgbot: mwalker synchronized php-1.22wmf15/extensions/CentralNotice 'Updating CentralNotice to master'
22:50 logmsgbot: mwalker synchronized php-1.22wmf16/extensions/CentralNotice 'Updating CentralNotice to master'
22:48 springle: upgraded packages on db1020 to get ganglia-monitor mysql metrics working
22:44 logmsgbot: mwalker synchronized php-1.22wmf16/extensions/Translate 'Updating Translate to master'
22:43 logmsgbot: mwalker synchronized php-1.22wmf15/extensions/Translate 'Updating Translate to master'
22:35 logmsgbot: reedy synchronized wmf-config/
21:42 logmsgbot: yurik synchronized php-1.22wmf15/extensions/ZeroRatedMobileAccess/
21:39 logmsgbot: yurik synchronized php-1.22wmf16/extensions/ZeroRatedMobileAccess/
21:18 logmsgbot: springle synchronized wmf-config/db-eqiad.php
21:04 LeslieCarr: rolled back
21:01 LeslieCarr: network maintenance in tampa
20:58 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
20:14 pp-pdf1: restarted all services
20:14 pp-pdf1: - update mwlib to 0.15.11
20:14 pp-pdf1: - update mwlib.epub to 0.14.3
20:00 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
19:33 logmsgbot: reedy synchronized php-1.22wmf16/extensions/CleanChanges 'Stage'
19:30 logmsgbot: reedy synchronized wmf-config/
19:10 LeslieCarr: more network changes, network disruptions unlikely but possible for tampa
18:25 LeslieCarr: moving all external traffic to come in via eqiad
18:10 LeslieCarr: doing some network maintenance on cr2-pmtpa
17:57 RobH: forgot to log testsearch1003 is back
17:49 RobH: testsearch1002 back up, 1003 down
17:39 RobH: testsearch1001 back in pool, 1002 rebooting for fix
17:21 RobH: rebooting testsearch1001 to troubleshoot
17:10 logmsgbot: aaron synchronized wmf-config/filebackend.php
17:02 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Removed ceph backend config for decommissioning'
16:39 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Cirrus as default on mw.org, Lucene as secondary'
10:18 apergos: set php memory limit to 256 on nickel, ori reducd the number of metrics for statsd (see https://gerrit.wikimedia.org/r/#/c/83777/), restarted all of apache, gmond, gmetad and still see the occasional php Allowed memory size exhausted error
02:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Sep 11 02:08:28 UTC 2013
02:02 logmsgbot: LocalisationUpdate completed (1.22wmf15) at Wed Sep 11 02:02:32 UTC 2013
02:01 logmsgbot: LocalisationUpdate completed (1.22wmf16) at Wed Sep 11 02:01:46 UTC 2013

September 10

23:46 LeslieCarr: rebooted neon, had crashed
23:33 logmsgbot: dzahn synchronized ./wmf-config/throttle.php
23:32 logmsgbot: dzahn synchronized ./wmf-config/throttle.php
23:28 springle: starting mariadb 10 multi-source replication test from all clusters on db1045
22:24 logmsgbot: reedy synchronized wmf-config/
22:15 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikivoyage and wikidatawiki to 1.22wmf16
22:12 Jeff_Green: dist-upgrade and reboot boron
22:09 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Sep 10 22:08:58 UTC 2013
22:07 logmsgbot: reedy synchronized wmf-config/
22:04 logmsgbot: reedy synchronized php-1.22wmf16/extensions/Wikibase
21:50 logmsgbot: LocalisationUpdate completed (1.22wmf15) at Tue Sep 10 21:50:32 UTC 2013
21:50 logmsgbot: reedy synchronized php-1.22wmf16/includes/search/SearchEngine.php
21:33 mutante: temp. hacked a fork limit into apache-graceful-all so my local ssh agent doesn't die when it's too fast, bleh:p
21:26 logmsgbot: LocalisationUpdate completed (1.22wmf16) at Tue Sep 10 21:26:17 UTC 2013
21:03 logmsgbot: csteipp synchronized php-1.22wmf15/extensions/CentralAuth
20:56 logmsgbot: demon synchronized php-1.22wmf16/extensions/CirrusSearch 'Logging for annoying parser output missing bug'
20:55 LeslieCarr: setting AMSIX Local pref to 110 due to overloading. high risk
20:42 logmsgbot: csteipp synchronized php-1.22wmf15/extensions/CentralAuth
20:42 logmsgbot: csteipp synchronized php-1.22wmf16/extensions/CentralAuth
18:43 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable ContributionReporting'
18:11 logmsgbot: reedy synchronized php-1.22wmf15/extensions/Wikibase
18:10 logmsgbot: reedy synchronized php-1.22wmf15/extensions/WikibaseDataModel
18:09 logmsgbot: reedy synchronized php-1.22wmf14/extensions/Wikibase
18:09 logmsgbot: reedy synchronized php-1.22wmf14/extensions/WikibaseDataModel
17:52 cmjohnson1: removing sq42 from pybal
17:20 cmjohnson1: rebooting sq42
17:05 apergos: ganglia was failign to serve some urls, complaining that memory size of 134217728 bytes exhausted, I upped to 192mb temporarily in /etc/php5/apache2/php.ini, needs followup
16:45 logmsgbot: reedy synchronized wmf-config/interwiki.cdb 'Updating interwiki cache'
16:24 apergos: on labstore3, the various mount lines for /dev/mapper/store-xx are commented out in fstab from aug 15, so naturally there's o nfs etc, not sure if it's ok to just put 'em back in, need someone to take a look-see
16:12 apergos: rebooting labstore3 after restart of nfsd did not do the trick
12:23 Nemo_bis: labstore3 overloaded, beta & tools &co. down
12:21 apergos: rebooted neon, it was out to lunch
01:07 awight: crm updated from 75d31d3443e4 to aa79ff4
00:40 mutante: restarting gmetad on nickel
00:27 LeslieCarr: restarted gmetad on nickel
00:21 logmsgbot: csteipp synchronized php-1.22wmf15/extensions/CentralAuth 'Syncing correct submodule checkout of centralauth'

September 9

23:54 logmsgbot: csteipp synchronized wmf-config/CommonSettings.php
23:52 logmsgbot: mwalker Finished syncing Wikimedia installation... : Scapping to update CentralNotice i18n strings
23:50 mutante: restarted statsd on hafnium
23:47 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Sep 9 23:47:10 UTC 2013
23:38 logmsgbot: mwalker Started syncing Wikimedia installation... : Scapping to update CentralNotice i18n strings
23:28 logmsgbot: LocalisationUpdate completed (1.22wmf15) at Mon Sep 9 23:28:18 UTC 2013
23:09 logmsgbot: LocalisationUpdate completed (1.22wmf16) at Mon Sep 9 23:09:05 UTC 2013
23:03 mutante: deployed localized logo for he.wikivoyage
23:02 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
22:37 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
22:36 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
22:10 RobH: blog updated to wp 3.6 and no one noticed, thats good upgradin folks
22:08 mutante: running update-special-pages on hume
21:54 LeslieCarr: upgrading mw121*
21:46 LeslieCarr: upgrading mw120*
21:44 logmsgbot: reedy synchronized php-1.22wmf16/extensions/EducationProgram/
21:38 LeslieCarr: upgrading mw119*
21:21 hashar: lanthanum.eqiad.wmnet upgrading packages
21:19 hashar: gallium upgrading PHP packages.
21:18 logmsgbot: nikerabbit synchronized wmf-config/CommonSettings.php 'ULS eventlogging'
21:17 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'ULS eventlogging'
20:52 LeslieCarr: upgrading mw118*
20:45 LeslieCarr: upgrading mw117*
20:43 LeslieCarr: upgrading asw-ulsfo
20:40 LeslieCarr: upgrading mw116*
20:33 LeslieCarr: upgrading mw115*
20:29 logmsgbot: mwalker synchronized php-1.22wmf16/extensions/CentralNotice 'Updating CentralNotice for custom banner classes'
20:28 logmsgbot: mwalker synchronized php-1.22wmf15/extensions/CentralNotice 'Updating CentralNotice for custom banner classes'
20:25 LeslieCarr: upgrading mw114*
20:21 paravoid: upgrading gdnsd to 1.10
20:19 LeslieCarr: upgrading mw113* servers
20:18 bblack: bumped vhtcpd to 0.0.10 on brewster (conn close bugfix)
20:08 LeslieCarr: upgrading mw112*
20:04 springle: fixed broken wikiadmin grants on es100[2-4]
20:01 LeslieCarr: upgrading mw111*
18:57 logmsgbot: reedy synchronized php-1.22wmf16
18:56 logmsgbot: reedy synchronized php-1.22wmf15
18:51 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else that isn't wikipedia, wikivoyage or wikidatawiki to 1.22wmf16
18:39 Ryan_Lane: repooling ssl3003
18:30 Ryan_Lane: repooling ssl3001/3002 and depooling ssl3003
18:26 LeslieCarr: upgrading mw1000-mw1009
18:25 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity, wikibooks and wikisource to 1.22wmf16
18:22 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: various misc wikis to 1.22wmf16
18:19 Ryan_Lane: depooling ssl3001-2
18:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: various misc wikis to 1.22wmf16
18:11 Ryan_Lane: repooling ssl1006-9
18:04 Ryan_Lane: repooling ssl1002-5 and depooling ssl1006-9
17:54 Ryan_Lane: pooling ssl1001 and depooling ssl1002-5
17:44 Ryan_Lane: depooling ssl1001
17:43 Ryan_Lane: adding GCM cipher and removing DES from SSL config
02:01 logmsgbot: LocalisationUpdate failed: git pull of extensions failed

September 8

02:01 logmsgbot: LocalisationUpdate failed: git pull of extensions failed

September 7

05:23 apergos: gitblit was returning errors for some urls with java.lang.NullPointerException, stopped and restarted it, seems ok now
04:01 ori-l: git.wikimedia.org: HTTP 500; Varnish XID 196670102
02:01 logmsgbot: LocalisationUpdate failed: git pull of extensions failed

September 6

22:29 mutante: sync-common-file InitSettings, deploy change 83086
22:29 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
20:06 Ryan_Lane: rebooting virt11
18:34 gwicke: updated Parsoid to ef1776732
18:12 ^d: puppet & gerrit service stopped on manganese so nobody hits them anymore
02:33 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
02:02 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
00:16 mutante: restarting irc bot on ekrem

September 5

23:31 mutante: installing package upgrades on ekrem
23:16 logmsgbot: bsitu synchronized php-1.22wmf15/extensions/PageTriage 'Update PageTriage to master'
23:15 logmsgbot: bsitu synchronized php-1.22wmf16/extensions/PageTriage 'Update PageTriage to master'
22:45 logmsgbot: csteipp synchronized php-1.22wmf16/extensions/CentralNotice 'bug53032'
22:44 logmsgbot: csteipp synchronized php-1.22wmf16/includes/specials
22:43 logmsgbot: csteipp synchronized php-1.22wmf16/includes/actions
22:16 logmsgbot: spage synchronized wmf-config/InitialiseSettings.php 'E3 deploy GuidedTour to 7 more wikis'
22:11 logmsgbot: spage Finished syncing Wikimedia installation... : Deploying latest GuidedTour to wmf15 and wmf16
21:59 logmsgbot: spage Started syncing Wikimedia installation... : Deploying latest GuidedTour to wmf15 and wmf16
21:57 mutante: rolled back SSL cert on blog for now to discuss techblog.wm issue which would not be covered by cert but is just a redirect
21:40 mutante: replaced SSL cert on blog.wikimedia.org, don't use star cert, have separate new one
21:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
20:51 logmsgbot: reedy synchronized php-1.22wmf15/extensions/ArticleFeedbackv5/
20:34 logmsgbot: reedy synchronized php-1.22wmf16/extensions/ArticleFeedbackv5/
19:10 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: all wikipedias to 1.22wmf15
19:01 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki, mediawikiwiki, test2wiki, loginwiki and testwikidatawiki to 1.22wmf16
18:58 ^d: gerrit back up
18:47 ^d: gerrit coming down for a short bit. moving data around.
18:42 logmsgbot: reedy synchronized php-1.22wmf16/extensions/UploadWizard/
18:27 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Change I34d1f29d2: clean-up of InitialiseSettings.php'
18:26 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Increase rate of NavigationTiming events'
18:03 Ryan_Lane: adding virt12 to the virt node pool
17:22 akosiaris: upload kafka 0.8-20130903-1 at apt.wikimedia.org, component universe
16:41 cmjohnson1: powering down sq36 to swap disk
16:26 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.22wmf15 till deployment window
16:23 logmsgbot: reedy synchronized php-1.22wmf16/extensions/ 'Minor wikibase updates'
16:10 logmsgbot: reedy Finished syncing Wikimedia installation... : testwiki to 1.22wmf16 and build l10n cache
15:51 logmsgbot: reedy Started syncing Wikimedia installation... : testwiki to 1.22wmf16 and build l10n cache
15:33 Nemo_bis: etherpad.wikimedia.org down (error 503)
15:28 cmjohnson1: replacing disk slot 0 ms-be1011
15:18 logmsgbot: reedy synchronized docroot and w
15:17 logmsgbot: reedy synchronized php-1.22wmf16 'Initial file sync'
14:12 paravoid: recreating mediawiki cgroup lost after cgroup-bin upgrade; thumbnail issues fixed
13:14 paravoid: adjusting depool-threshold, restarting pybal on lvs1003
12:13 paravoid: pushing new swift object/account/container rings: setting ms-be9's sde1 to 33, swapping ms-be5's sda/b <-> sdm/n, also 33
10:21 apergos: lanthanum bios processor settings set to 'custom' and c states disabled (attempt to fix power limit notification issue)
09:57 springle: leaving sync_binlog=0 on db36 and db38 so replication keeps up. to be reviewed
09:05 hashar_: jenkins: killed two requesthandler threads that were eating CPU power.
02:47 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Sep 5 02:47:15 UTC 2013
02:30 logmsgbot: LocalisationUpdate completed (1.22wmf14) at Thu Sep 5 02:30:03 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.22wmf15) at Thu Sep 5 02:15:59 UTC 2013
01:42 ^d: gallium: updated git remotes in /srv/ssd/zuul/git/* to not point to old manganese box and point to gerrit.wm.o instead

September 4

23:17 ^d: restarting gerrit one last time
23:04 ^d: apache stopped on manganese
23:01 ^d: stopped puppet on manganese
21:23 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
21:10 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
19:41 Ryan_Lane: upgraded wiktech-static to 1.22wmf15
19:16 andrewbogott: turning on pluginsync, which will (temporarily) make puppet even slower
19:10 andrewbogott: Moved morebots from wikitech-static to toollabs
06:28 apergos: truncated fact_values puppet table and reset auto increment to start at 1, puppet was broken on all hosts, see http://projects.puppetlabs.com/issues/9225
02:23 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Sep 4 02:21:58 UTC 2013
02:13 logmsgbot: LocalisationUpdate completed (1.22wmf14) at Wed Sep 4 02:11:48 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.22wmf15) at Wed Sep 4 02:06:47 UTC 2013
00:51 logmsgbot: olivneh synchronized php-1.22wmf15/includes/RecentChange.php 'Fix for bug 53720'

September 3

22:47 Ryan_Lane: upgrading wikitech to 1.22wmf15
19:47 logmsgbot: reedy synchronized wmf-config/ 'touch'
19:38 logmsgbot: reedy synchronized php-1.22wmf15/extensions/UploadWizard/resources 'touch'
19:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything non 'pedia to 1.22wmf15
18:54 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special, wikimedia, closed and private wikis to 1.22wmf15
18:48 awight: civicrm updated 5c28984d7e4 to 75d31d3443e45c
18:39 mwalker: optimizing civicrm.civicrm_email on db1025 (db1008)
18:18 mwalker: altering db1025 (db1008) civicrm.civicrm_email --> "ALTER TABLE civicrm_email MODIFY COLUMN email varchar(255);"
18:01 gwicke: updated Parsoid to 859a701
15:23 hashar: gallium running mw phpunit code coverage under gdb in a tmux.
14:29 hashar: gallium: dpkg -i libapache2-mod-php5_5.3.10-1ubuntu3.7+wmf1_amd64.deb php5-cli_5.3.10-1ubuntu3.7+wmf1_amd64.deb php5-common_5.3.10-1ubuntu3.7+wmf1_amd64.deb php5-curl_5.3.10-1ubuntu3.7+wmf1_amd64.deb php5-dbg_5.3.10-1ubuntu3.7+wmf1_amd64.deb php5-dev_5.3.10-1ubuntu3.7+wmf1_amd64.deb php5-gd_5.3.10-1ubuntu3.7+wmf1_amd64.deb php5-intl_5.3.10-1ubuntu3.7+wmf1_amd64.deb php5-mysql_5.3.10-1ubuntu3.7+wmf1_amd64.deb
14:28 hashar: upgrading PHP5 on gallium (contint server)
02:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Sep 3 02:25:20 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.22wmf15) at Tue Sep 3 02:14:15 UTC 2013
02:09 logmsgbot: LocalisationUpdate completed (1.22wmf14) at Tue Sep 3 02:08:03 UTC 2013

September 2

16:21 hashar: restarted Zuul (paranoia)
16:15 Reedy: Running cleanupTitles.php in screen on terbium
16:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
14:36 hashar: Jenkins: restoring mediawiki-core jobs to previous state
14:32 hashar: broken Jenkins whenever a huge repository is being used (i.e. mediawiki/core or operations/puppet )
13:43 akosiaris: aptitude hold puppet puppet-common puppetmaster puppetmaster-common puppetmaster-passenger on stafford till we solve the problem with CPU spiking present in puppet 2.7.11-1ubuntu2.4
13:36 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'reutrn db1027 to the pool'
06:22 springle: granted process/show/usage to watchdog on db masters
04:34 springle: granted process/show/usage to watchdog on db masters
02:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Sep 2 02:25:02 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.22wmf15) at Mon Sep 2 02:13:50 UTC 2013
02:09 logmsgbot: LocalisationUpdate completed (1.22wmf14) at Mon Sep 2 02:07:46 UTC 2013
00:50 logmsgbot: faidon synchronized wmf-config/InitialiseSettings.php 'new wikivoyage logo'
00:50 logmsgbot: faidon synchronized docroot/bits/favicon/wikivoyage.ico 'new wikivoyage favicon'

September 1

23:59 springle: running pt-table-sync over db1027 wikis
21:30 springle: restarted db1027 mysqld. leaving port 3306 blocked while replication recovers
20:53 paravoid: powercycled db1027, BUG: soft lockup - CPU#14 stuck for 22s! [xfsaild/dm-0:933]
20:31 paravoid: restarting gerrit
17:58 paravoid: killed snapshot1004 workers, flooding dberror with wikiadmin auth failures + WikiExporter::dumpFrom slow queries (10' ago)
17:54 logmsgbot: faidon cleared profiling data
17:06 logmsgbot: maxsem synchronized /php-1.22wmf14/includes/api/ApiQueryImageInfo.php 'Not the cause, restoring back'
16:55 logmsgbot: maxsem synchronized /php-1.22wmf14/includes/api/ApiQueryImageInfo.php 'Trying to disable this...'
02:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Sep 1 02:07:18 UTC 2013
02:04 logmsgbot: LocalisationUpdate completed (1.22wmf15) at Sun Sep 1 02:02:30 UTC 2013
02:03 logmsgbot: LocalisationUpdate completed (1.22wmf14) at Sun Sep 1 02:01:46 UTC 2013

August 31

23:49 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depooling db1027'
02:27 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Aug 31 02:26:20 UTC 2013
02:18 logmsgbot: LocalisationUpdate completed (1.22wmf15) at Sat Aug 31 02:16:41 UTC 2013
02:09 logmsgbot: LocalisationUpdate completed (1.22wmf14) at Sat Aug 31 02:08:01 UTC 2013

August 30

22:21 mutante: after re-pooling search1015 a while ago, now temp. disable search1016 while indices are rebuilt for new wikis
21:49 mutante: on sodium: Class[Backup::Host] is already defined ...
21:46 mutante: temp. disabling search1015 in pybal
21:29 mutante: dumping/importing viwikivoyage and tyvwiki lucene search indices
21:24 mutante: restarting lucene search indexers on searchidx2 and searchidx1001
20:24 logmsgbot: csteipp synchronized php-1.22wmf15/extensions/LiquidThreads 'bug53320'
18:01 mutante: powercycling sq36
15:27 hashar: Jenkins: removing labels hasSlaveScripts hasContintPackages productionSlaves from lanthanum slaves, it caught some jobs
14:39 hashar: jenkins: lanthanum slave now has 8 executors and labels hasSlaveScripts hasContintPackages productionSlaves
10:48 mark: reverted
10:36 mark: Moving esams text squid backend traffic from pmtpa to eqiad
08:47 hashar: restarting Jenkins for plugins upgrade
05:05 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Make Flickr API URL protocol-relative for compatibility with HTTPS'
02:55 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Aug 30 02:53:46 UTC 2013
02:39 logmsgbot: LocalisationUpdate completed (1.22wmf15) at Fri Aug 30 02:38:00 UTC 2013
02:20 logmsgbot: LocalisationUpdate completed (1.22wmf14) at Fri Aug 30 02:18:27 UTC 2013
00:19 Ryan_Lane: rebooting sq36
00:18 Ryan_Lane: switching esams from using pmtpa to using eqiad for carp

August 29

23:52 gwicke: updated Parsoid to b26a506
23:37 Ryan_Lane: removing sq33 from squid configuration, it was decommissioned a while ago
21:56 mutante: upgrading apparmor packages on iron
21:32 mutante: powercycled sq36 which was frozen but probably all unrelated to API outage
20:59 logmsgbot: olivneh synchronized php-1.22wmf15/extensions/NavigationTiming 'Updating NavigationTiming to master (2/2)'
20:58 logmsgbot: olivneh synchronized php-1.22wmf14/extensions/NavigationTiming 'Updating NavigationTiming to master (1/2)'
20:57 logmsgbot: mflaschen synchronized php-1.22wmf14/extensions/GuidedTour/ 'Deploy GuidedTour to 1.22wmf14'
20:55 logmsgbot: olivneh synchronized php-1.22wmf15/includes/RecentChange.php 'I4570473ae'
20:52 logmsgbot: reedy synchronized php-1.22wmf14/extensions/Wikibase
20:52 logmsgbot: reedy synchronized php-1.22wmf15/extensions/Wikibase
20:33 logmsgbot: olivneh synchronized php-1.22wmf15/extensions/NavigationTiming 'Updating NavigationTiming to master (2/2)'
20:32 logmsgbot: olivneh synchronized php-1.22wmf14/extensions/NavigationTiming 'Updating NavigationTiming to master (1/2)'
19:54 Aaron|home: Running PurgeDeletedFiles everywhere to bring ceph/swift more in sync with each other and the DB; copy scripts already finished
19:53 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php
19:40 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'NavigationTiming sample factor to 5000'
19:17 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
19:08 logmsgbot: reedy synchronized php-1.22wmf14/extensions/NavigationTiming
19:07 logmsgbot: reedy synchronized php-1.22wmf14/extensions/GettingStarted
19:06 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
19:00 logmsgbot: maxsem synchronized php-1.22wmf15/extensions/MobileFrontend/includes/formatters/ExtractFormatter.php 'https://gerrit.wikimedia.org/r/#/c/81416/'
18:57 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
18:53 logmsgbot: reedy Finished syncing Wikimedia installation... : Oh scap!
18:38 logmsgbot: reedy Started syncing Wikimedia installation... : Oh scap!
18:33 logmsgbot: reedy synchronized wmf-config/
18:22 logmsgbot: reedy synchronized wmf-config/
18:11 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.22wmf14
18:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki, mediawikiwiki, testwikidatawiki and loginwiki to 1.22wmf15
18:03 logmsgbot: reedy Finished syncing Wikimedia installation... : testwiki to 1.22wmf15 and build l10n cache
17:40 logmsgbot: reedy Started syncing Wikimedia installation... : testwiki to 1.22wmf15 and build l10n cache
17:15 logmsgbot: csteipp synchronized php-1.22wmf14/includes/resourceloader/ 'bug46332'
17:13 logmsgbot: csteipp synchronized php-1.22wmf14/includes/api 'bug49090'
16:58 logmsgbot: reedy synchronized docroot and w
16:57 logmsgbot: reedy synchronized php-1.22wmf15/
15:45 MaxSem: Recreating Solr index
14:48 ottomata: restarting gmetad on nickel to see if I can get it to pick up reinstalled analytics boxes
09:40 springle: killing long-running research transaction on db1047, and setting innodb_max_purge_lag to reduce history list length
09:33 mark: Powercycled lanthanum
08:54 hashar: rebooting lanthanum.eqiad.wmnet (high sys cpu)
08:34 hashar: apt-get upgrade on gallium.wikimedia.org
08:31 hashar: apt-get upgrade on lanthanum.eqiad.wmnet
07:29 springle: relaxed slave delay warnings to warn 150 crit 300
02:42 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Aug 29 02:40:47 UTC 2013
02:27 logmsgbot: LocalisationUpdate completed (1.22wmf13) at Thu Aug 29 02:26:16 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.22wmf14) at Thu Aug 29 02:14:03 UTC 2013
01:41 mwalker: fundraising delete query on db1025 ran 1:10minutes -- serious lag on db1008 and db43 now expected for same
00:31 mwalker: clearing ~14M rows from db1025.pgehres.bannerimpressions and bannerimpression_raw

August 28

23:29 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Disable the job queue that processes echo notification'
23:18 logmsgbot: rmoen synchronized php-1.22wmf14/extensions/VisualEditor/ 'Update VisualEditor to master'
22:49 logmsgbot: demon synchronized php-1.22wmf14/extensions/CentralAuth
22:49 logmsgbot: demon synchronized php-1.22wmf13/extensions/CentralAuth
20:42 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'All wikis to secure login'
20:38 logmsgbot: demon Finished syncing Wikimedia installation... : Secure login on alllllll the wikis
20:24 logmsgbot: demon Started syncing Wikimedia installation... : Secure login on alllllll the wikis
19:30 logmsgbot: awight synchronized php-1.22wmf14/extensions/CentralNotice 'Updating CentralNotice to wmf_deploy'
19:30 logmsgbot: awight synchronized php-1.22wmf13/extensions/CentralNotice 'Updating CentralNotice to wmf_deploy'
18:37 logmsgbot: csteipp synchronized php-1.22wmf13/extensions/Wikibase
18:37 logmsgbot: csteipp synchronized php-1.22wmf13/extensions/CheckUser
18:36 logmsgbot: csteipp synchronized php-1.22wmf14/extensions/CheckUser
18:35 logmsgbot: csteipp synchronized php-1.22wmf14/extensions/Wikibase
18:34 logmsgbot: csteipp synchronized php-1.22wmf14/extensions/OAuth
17:50 ottomata: restarting some datanodes
17:48 logmsgbot: reedy synchronized php-1.22wmf14/extensions/CirrusSearch/ 'Fix bug 53487'
16:49 logmsgbot: demon synchronized php-1.22wmf14/extensions/CirrusSearch/forceSearchIndex.php 'Fix for reindexer'
16:47 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php
16:26 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Move mw.org to primary backend as Cirrus to test my theory'
16:21 Reedy: Created EducationProgram tables on ptwiki
16:09 ^d: ES: indexing mediawikiwiki in screen on terbium
16:08 ^d: ES: created mediawikiwiki index
16:07 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'CirrusSearch on mediawiki.org \o/'
15:47 gwicke: updated Parsoid to ca3f45f
15:40 logmsgbot: demon synchronized php-1.22wmf14/extensions/CirrusSearch 'CirrusSearch to master'
15:04 springle: restarting slave threads on db55, db58, db43. will lag for a while.
14:35 paravoid: authdns-update: switching Middle-East to esams
14:16 paravoid: authdns-update: switching Africa to esams
14:00 hashar: jenkins: renamed OAuth jobs from Oauth to OAuth
13:57 hashar: jenkins: renamed Editcount jobs from EditCount to Editcount
13:45 springle: same purge situation for db58 and db43, and remedy
13:37 springle: stopped db55 slave threads while purge catches up; killed long running research user transaction apparently doing nothing but holding locks
13:21 springle: temporarily changing innodb purge settings on db55 (massive growing history list)
12:34 hashar: jenkins: updated various jobs to use $ZUUL_PROJECT when determining the git repository to fetch from [[bugzilla:53470|bug 53470]]
02:47 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Aug 28 02:45:49 UTC 2013
02:32 logmsgbot: LocalisationUpdate completed (1.22wmf13) at Wed Aug 28 02:30:40 UTC 2013
02:17 logmsgbot: LocalisationUpdate completed (1.22wmf14) at Wed Aug 28 02:16:15 UTC 2013

August 27

23:54 logmsgbot: bsitu synchronized php-1.22wmf13/extensions/Echo 'Update Echo to master'
23:53 logmsgbot: bsitu synchronized php-1.22wmf14/extensions/Echo 'Update Echo to master'
23:19 logmsgbot: reedy synchronized php-1.22wmf14/extensions/CentralAuth
23:19 logmsgbot: reedy synchronized php-1.22wmf13/extensions/CentralAuth
23:12 logmsgbot: kaldari synchronized php-1.22wmf14/extensions/MobileFrontend/ 'updating MobileFrontend to master for wmf14'
23:00 logmsgbot: kaldari synchronized php-1.22wmf13/extensions/MobileFrontend/ 'updating MobileFrontend to master for wmf13'
22:50 logmsgbot: reedy synchronized php-1.22wmf14/extensions/CentralAuth/maintenance/
22:33 Ryan_Lane: purging login.wikimedia.org from text varnish
22:21 logmsgbot: kaldari synchronized php-1.22wmf13/extensions/MobileFrontend/includes/formatters/ExtractFormatter.php 'syncing live-hack to try to address fatal error'
22:07 logmsgbot: kaldari synchronized php-1.22wmf13/extensions/MobileFrontend 'syncing MobileFrontend to try to address fatal error'
21:58 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'syncing InitialiseSettings.php to try to fix fatal error from MobileFrontend'
21:51 RobH: disregard netmon1001 alerts, its my project server
21:45 Reedy: Running extensions/Wikibase/repo/maintenance/rebuildPropertyInfo.php against wikidatawiki in screen on terbium
21:33 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
21:13 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
21:04 Ryan_Lane: deployed change 81375 to remove the http to https redirect from loginwiki
20:44 logmsgbot: demon synchronized php-1.22wmf14/extensions/CentralAuth 'CentralAuth to master'
20:43 logmsgbot: demon synchronized php-1.22wmf13/extensions/CentralAuth 'CentralAuth to master'
20:06 logmsgbot: maxsem synchronized w/robots.php 'https://gerrit.wikimedia.org/r/69420'
19:27 cmjohnson1: search1002 going down for bios update
19:19 logmsgbot: mlitn Finished syncing Wikimedia installation... : Updating ArticleFeedbackv5, Echo, PageTriage and Thanks
19:06 logmsgbot: mlitn Started syncing Wikimedia installation... : Updating ArticleFeedbackv5, Echo, PageTriage and Thanks
18:55 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable job queue to process notifs for all wikis'
18:06 gwicke: updated Parsoid to 84fac157
18:05 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php 'Enable job queue to process web and email notifs on testwiki/test2wiki'
18:04 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable job queue to process web and email notifs on testwiki/test2wiki'
16:36 ^d: reloaded gerrit replication plugin to pick up changes
14:56 logmsgbot: hashar synchronized php-1.22wmf14/extensions/Wikibase 'https://gerrit.wikimedia.org/r/#/c/81233/'
14:55 logmsgbot: hashar synchronized php-1.22wmf14/extensions/WikibaseDataModel 'https://gerrit.wikimedia.org/r/#/c/81233/'
14:53 logmsgbot: hashar synchronized php-1.22wmf13/extensions/Wikibase 'https://gerrit.wikimedia.org/r/#/c/81237/'
11:31 logmsgbot: hashar synchronized php-1.22wmf14/extensions/DataValues 'Update DataValues 81198, requested by aude'
09:56 hashar: jenkins restarted :-]
09:30 hashar: jenkins: Failed to exec '/usr/lib/jvm/java-6-openjdk-amd64/jre/bin/java (deleted)' No such file or directory holy crap
09:28 hashar: restarting Jenkins for plugins upgrade
07:35 hashar: jenkins: upgrading pep8 on gallium [[bugzilla:53352|bug 53352]]
02:22 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Aug 27 02:21:16 UTC 2013
02:14 logmsgbot: LocalisationUpdate completed (1.22wmf13) at Tue Aug 27 02:12:42 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.22wmf14) at Tue Aug 27 02:07:20 UTC 2013
01:08 mutante: chown'ing /srv/ssd/gerrit to gerritslave for replication on lanthanum

August 26

22:49 ^d: restarting gerrit
22:47 binasher: fixed labsdb cpufreq governor setting (and got a 2x speedup on a large groupby query, woops!)
21:13 logmsgbot: reedy synchronized php-1.22wmf14/extensions/Wikibase
20:19 logmsgbot: reedy synchronized wmf-config/
20:03 ottomata: taking down analytics1011-analytics1020 for reinstall
19:58 ottomata: taking analytics1010 down in preparation for repaving of hadoop cluster
19:49 logmsgbot: reedy synchronized php-1.22wmf14/extensions/Wikibase
19:48 logmsgbot: reedy synchronized php-1.22wmf14/extensions/DataValues/
19:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote to 1.22wmf14
19:20 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource, wikibooks and wikinews to 1.22wmf14
19:19 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
19:10 logmsgbot: reedy synchronized database lists files: wikivoyage, wikiversity and wiktioanry to 1.22wmf14
19:09 RobH: for operations/dns.git
19:09 RobH: i accidentally (yea right) pushed .gitreview to master rather than head, so no gerrit log for my change, opps!
19:00 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
18:51 logmsgbot: reedy synchronized wmf-config/
18:42 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special, closed and wikimedia to 1.22wmf14
18:29 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Fix wmgHTTPSBlacklistCountries'
17:07 logmsgbot: csteipp synchronized php-1.22wmf13/extensions/LiquidThreads 'Fix bug53320'
17:07 logmsgbot: csteipp synchronized php-1.22wmf14/extensions/LiquidThreads 'Fix bug53320'
16:57 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'GeoIP configuration for secure login'
16:56 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'GeoIP configuration for secure login'
16:54 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'mediawikiwiki to secure login'
16:38 ottomata: taking down analytics1006, analytics1008 and analytics1009 for reinstall
16:30 ottomata: bringing analytics1005 down for reinstall
02:25 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Aug 26 02:24:25 UTC 2013
02:17 logmsgbot: LocalisationUpdate completed (1.22wmf14) at Mon Aug 26 02:16:07 UTC 2013
02:10 logmsgbot: LocalisationUpdate completed (1.22wmf13) at Mon Aug 26 02:09:13 UTC 2013

August 25

16:51 RoanKattouw: Restarted Parsoid on a couple of boxes to clear some disk space. It appears wtp1001-1004 have been set up with larger, LVMed root partitions but I can't find any logs on when this was done, by whom, or why
16:46 RoanKattouw: Restarted Parsoid on wtp1012, was unresponsive
16:41 RoanKattouw: Strike that, my account on fenari is just being weird apparently
16:40 RoanKattouw: Investigating low disk space warnings on wtp1001-1024, but it looks like something weird happened to them, I get host key verification warnings or failures for all of them
02:38 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Aug 25 02:37:03 UTC 2013
02:26 logmsgbot: LocalisationUpdate completed (1.22wmf14) at Sun Aug 25 02:25:16 UTC 2013
02:14 logmsgbot: LocalisationUpdate completed (1.22wmf13) at Sun Aug 25 02:13:19 UTC 2013
01:05 logmsgbot: demon Finished syncing Wikimedia installation... :
00:53 logmsgbot: demon Started syncing Wikimedia installation... :

August 24

19:27 gwicke: updated Parsoid to 8afe5ef
09:57 springle: optimizing tables on db55. expect lag
02:43 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Aug 24 02:42:22 UTC 2013
02:33 logmsgbot: LocalisationUpdate completed (1.22wmf14) at Sat Aug 24 02:31:29 UTC 2013
02:18 logmsgbot: LocalisationUpdate completed (1.22wmf13) at Sat Aug 24 02:16:50 UTC 2013

August 23

23:58 mutante: removing mailing list Wikipedia-ensino, creating list educacao-br
20:59 logmsgbot: marc synchronized docroot
20:53 ori-l: Re-starting eventlogging/consumer NAME=mysql-db1047 on vanadium
20:45 logmsgbot: root synchronized docroot
20:39 ori-l: Stopped eventlogging/consumer NAME=mysql-db1047 on vanadium
20:34 binasher: taking mysqld down on db1047 (s1 analytics + eventlog data) for unplanned maintenance
19:54 RobH: ns2.wikimedia.org glue record updated from 91.198.174.4 to 91.198.174.239
18:20 awight: civicrm updated from e4f1228 to 5c28984
17:20 ottomata: disabling puppet on analytics 1006, 1008, 1009, 1021 and 1022 in preparation for reinstallation and repaving next week
17:19 awight: updated civicrm from 257686e81cf6 to e4f1228
16:28 logmsgbot: reedy synchronized php-1.22wmf14/extensions/WikiEditor/
16:27 logmsgbot: reedy synchronized php-1.22wmf13/extensions/WikiEditor/
16:20 logmsgbot: reedy synchronized wmf-config/
15:45 ottomata: disabling puppet on analytics1024 and analytics1025 in preperation for reinstall of zookeeper nodes
15:08 ottomata: taking analytics1023 offline for reinstall
10:17 paravoid: upgrading ceph to v0.67.2
07:28 springle: enabled dns reverse-lookups on db1016 mysql while replication catches up, to avoid slave sql thread stopping on SET PASSWORD / GRANT statements using hostnames
06:10 springle: ran mysql_upgrade on db1001 (due to missing/incorrect system tables errors in log)
05:25 springle: db35 manual mysql.* check, restart, mysql_upgrade, resolve replication sql error
05:19 springle: stopping db35 mysqld for mysql.* checks and mysql_upgrade
05:13 springle: resolved db1016 replication sql error
03:00 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Aug 23 02:59:10 UTC 2013
02:49 logmsgbot: LocalisationUpdate completed (1.22wmf14) at Fri Aug 23 02:47:36 UTC 2013
02:17 logmsgbot: LocalisationUpdate completed (1.22wmf13) at Fri Aug 23 02:15:37 UTC 2013
01:21 logmsgbot: reedy synchronized php-1.22wmf14/includes/specials/SpecialUserlogin.php

August 22

23:13 logmsgbot: reedy synchronized php-1.22wmf14/
23:05 logmsgbot: catrope synchronized php-1.22wmf14/extensions/VisualEditor 'Update VE for cherry-pick'
23:05 logmsgbot: catrope synchronized php-1.22wmf13/extensions/VisualEditor 'Update VE for cherry-pick'
22:14 logmsgbot: aaron synchronized php-1.22wmf13/includes/job/jobs/RefreshLinksJob.php '228e592'
20:59 Jeff_Green: fixed exim/spamassassin config on iodine so messages are spam-tagged for otrs filters
20:54 Krinkle: Reloading Zuul to deploy I3c754cce8828f
20:54 awight: civicrm updated from 5f735ed to 257686e
20:19 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:38 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: phase 3 wikis to 1.22wmf13
18:36 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: phase 1 wikis to 1.22wmf14
17:47 logmsgbot: reedy synchronized php-1.22wmf13/includes/api/ApiEditPage.php
16:58 logmsgbot: mwalker synchronized php-1.22wmf14/extensions/CentralNotice/special/ 'Updating wmf14 to include CentralNotice bugfix 53032'
16:56 gwicke: updated Parsoid to 437374dc
16:56 logmsgbot: mwalker Finished syncing Wikimedia installation... : Updating CentralNotice to master
16:41 logmsgbot: mwalker Started syncing Wikimedia installation... : Updating CentralNotice to master
15:13 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.22wmf13 for now
15:09 logmsgbot: reedy Finished syncing Wikimedia installation... : testwiki to 1.22wmf14, build l10n cache
14:56 logmsgbot: reedy Started syncing Wikimedia installation... : testwiki to 1.22wmf14, build l10n cache
14:38 logmsgbot: reedy synchronized docroot and w
14:36 logmsgbot: maxsem synchronized php-1.22wmf13/extensions/MobileFrontend/includes/specials/SpecialMobileOptions.php 'debug over'
14:35 logmsgbot: reedy synchronized php-1.22wmf14/extensions/OAuth/
14:34 logmsgbot: reedy synchronized php-1.22wmf14/ 'Initial file sync'
14:32 logmsgbot: maxsem synchronized php-1.22wmf13/extensions/MobileFrontend/includes/specials/SpecialMobileOptions.php 'debug'
13:06 paravoid: changed watchmouse alert and re-enabled
11:37 paravoid: switching off alerts for watchmouse's DNS alert while investigating
11:04 paravoid: disabling puppet on eeden for the next couple of days
02:39 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Aug 22 02:38:17 UTC 2013
02:30 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Thu Aug 22 02:28:28 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.22wmf13) at Thu Aug 22 02:15:07 UTC 2013
00:06 logmsgbot: catrope synchronized php-1.22wmf13/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.ViewPageTarget.js 'touch'
00:06 logmsgbot: catrope synchronized php-1.22wmf12/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.ViewPageTarget.js 'touch'
00:03 logmsgbot: catrope synchronized php-1.22wmf13/extensions/VisualEditor 'Update VE to master'
00:03 logmsgbot: catrope synchronized php-1.22wmf12/extensions/VisualEditor 'Update VE to master'

August 21

22:32 binasher: opened up wikiadmin grants on es1 shard to accommodate snapshot1004
21:01 mutante: cp1061,cp1062 have dpkg issue upgrading varnish packages
20:38 mutante: installing package upgrades on iron
20:36 mutante: re-enabled puppet runs on zirconium which was "admin down"
20:21 mutante: dist-upgrading zirconium
18:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable Collection on testwikidatawiki. It's mucho brokenp'
18:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable Collection on testwikidatawiki'
18:31 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Enabled $wgMWOAuthSecureTokenTransfer'
16:37 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
16:28 paravoid: DNS migration complete
16:22 paravoid: switching ns2 to eeden
16:06 logmsgbot: reedy synchronized docroot and w
15:53 paravoid: switching ns0 to rubidium
15:48 logmsgbot: reedy synchronized wmf-config/
15:47 logmsgbot: reedy synchronized wmf-config/
15:37 logmsgbot: faidon synchronized wmf-config/filebackend.php 'revert ceph promotion to master'
14:00 paravoid: stopping ceph-osd 88 & 95 (ms-be1008), evidence of unexplainable high cpu usage
13:39 paravoid: switching ns1 traffic to mexia (new authdns)
12:09 Reedy: Running extensions/CentralAuth/maintenance/populateHomeDB.php in a screen on tin
11:03 logmsgbot: faidon synchronized wmf-config/filebackend.php 'promote ceph to master'
08:02 logmsgbot: nikerabbit synchronized php-1.22wmf13/extensions/UniversalLanguageSelector/ 'ULS to master'
02:48 mutante: RT - create new system dashboard 'wikimedia default' that lists open ops-requests, quick ticket creation, reminders and bookmarked tickets
02:47 mutante: RT - grant global right to see system dashboards to privileged users
02:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Aug 21 02:35:19 UTC 2013
02:30 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Wed Aug 21 02:28:33 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.22wmf13) at Wed Aug 21 02:15:12 UTC 2013
01:04 logmsgbot: mwalker synchronized php-1.22wmf12/extensions/CentralNotice/special 'Applying CentralNotice fix for bug 53032'
01:03 logmsgbot: mwalker synchronized php-1.22wmf13/extensions/CentralNotice/special 'Applying CentralNotice fix for bug 53032'
00:59 mutante: allow engineering and ops group members in RT to create new saved searches
00:12 logmsgbot: catrope synchronized php-1.22wmf13/extensions/VisualEditor/modules/ve/ui/tools/buttons/ve.ui.UnderlineButtonTool.js 'touch'
00:05 logmsgbot: catrope Finished syncing Wikimedia installation... : Updating VisualEditor to master

August 20

23:58 logmsgbot: catrope Started syncing Wikimedia installation... : Updating VisualEditor to master
23:48 mutante: powercycling frozen mw1173
23:46 mutante: powercycling frozen mw1089
23:43 mutante: powercycling srv281
23:40 logmsgbot: catrope synchronized php-1.22wmf13/extensions/VisualEditor 'Deploy new VE code to wmf13 first for testing'
23:40 RoanKattouw: Repeating previously aborted sync-dir of extensions/VisualEditor now that the dsh node list is fixed
23:38 mutante: manually removed mw1046 from dsh groups mediawiki-installation,apaches,apaches-eqiad on tin
23:34 RoanKattouw: Forcing puppet on tin to update dsh lists for mw1046 removal
23:01 ori-l: During sync-dir, SSH timeouts from srv281, mw1089, mw1173; rsync errors on mw1046 (RO fs; previously reported)
23:00 logmsgbot: olivneh synchronized php-1.22wmf13/extensions/CoreEvents 'Updating CoreEvents to master (2/2)'
23:00 logmsgbot: olivneh synchronized php-1.22wmf12/extensions/CoreEvents 'Updating CoreEvents to master (1/2)'
21:51 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
21:27 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
20:29 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'test2wiki to secure login'
20:26 logmsgbot: demon Finished syncing Wikimedia installation... :
20:04 binasher: granted "lock tables" to eventlog user on db1047
20:03 logmsgbot: demon Started syncing Wikimedia installation... :
19:41 logmsgbot: demon Started syncing Wikimedia installation... :
19:29 logmsgbot: anomie synchronized wmf-config/CommonSettings.php 'Fix OAuth rights assignments'
19:21 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings-labs.php 'Turn HTTPs on by default for beta'
19:20 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable Echo and Thanks on fr, hu, pt, pl and sv wiki'
19:18 logmsgbot: bsitu synchronized echowikis.dblist 'Add fr, hu, pt, pl and sv to Echo dblist'
19:08 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo to master
18:36 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo to master
18:27 gwicke: updated Parsoid to f359548f04e739
14:29 paravoid: starting multiple parallel swift->ceph copy jobs on terbium
10:18 apergos: reboot lvs1002 after dist-upgrade
07:39 apergos: reboot lvs1004 after dist-upgrade
07:11 apergos: rebooting lvs1001 to fix eth2.1003 ip addr, after misguided attempt to simply ifdown/ifup
07:00 paravoid: killing pybal on lvs1001
03:59 paravoid: authdns-update: Google DKIM selector
03:09 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Aug 20 03:07:38 UTC 2013
02:46 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Tue Aug 20 02:45:04 UTC 2013
02:33 logmsgbot: LocalisationUpdate completed (1.22wmf13) at Tue Aug 20 02:31:40 UTC 2013
00:30 logmsgbot: olivneh synchronized php-1.22wmf13/extensions/CoreEvents 'Updating CoreEvents to master for Ide8469db2 (2/2)'
00:29 logmsgbot: olivneh synchronized php-1.22wmf12/extensions/CoreEvents 'Updating CoreEvents to master for Ide8469db2 (1/2)'

August 19

22:48 ^d: ES: reindexing test2wiki
22:44 logmsgbot: demon synchronized php-1.22wmf13/extensions/CirrusSearch 'Cirrus to master'
22:31 logmsgbot: demon synchronized php-1.22wmf13/extensions/CirrusSearch 'Cirrus to master'
22:13 ^d: created oauth tables for mediawikiwiki
22:11 Coren: chown -R gitpuppet /var/lib/git/operations/puppet on sockpuppet: some of those files were (erroneously) owned by root
20:46 logmsgbot: reedy synchronized php-1.22wmf13/extensions/CirrusSearch/
20:03 logmsgbot: csteipp synchronized php-1.22wmf13 'sync again in case mw1046 errors messed up scap'
19:46 logmsgbot: csteipp synchronized php-1.22wmf12
19:40 logmsgbot: reedy synchronized php-1.22wmf13/includes/specials/SpecialUpload.php
19:36 akosiaris: depooled mw1046
19:18 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary and wikiquote to 1.22wmf13
19:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource, wikiversity and wikibooks to 1.22wmf13
19:11 Reedy: mw1046 reporting readonly filesystem
19:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special, closed, wikimedia, wikinews, wikivoyage to 1.22wmf13
18:06 logmsgbot: csteipp Started syncing Wikimedia installation... :
17:57 paravoid: rebooting rubidium, mexia, eeden -- kernel upgrade
17:52 paravoid: apt: new geoip-database package version, from 2 years later
17:43 logmsgbot: csteipp synchronized php-1.22wmf12/extensions/CentralAuth 'revert ca update'
17:37 logmsgbot: csteipp synchronized php-1.22wmf12/extensions/CentralAuth 'Updating wmf12 CentralAuth to master for OAuth'
17:14 cmjohnson1: removing sq41 from dsh node groups and pybal
16:34 logmsgbot: csteipp synchronized php-1.22wmf13/extensions/CentralAuth 'Update CentralAuth to master for OAuth hooks'
14:46 paravoid: authdns-update: remove trailing dot from ulsfo reverse zone names
13:18 springle: temporarily running pt-kill on s5 slaves for slow wikidata queries
12:51 paravoid: hopefully final swiftrepl (swift->ceph thumbs) run on ms-fe1002
12:19 logmsgbot: tstarling synchronized php-1.22wmf12/includes/GlobalFunctions.php 'revert temp hack, superseded'
12:18 logmsgbot: faidon synchronized wmf-config/filebackend.php 'ceph as a filebackend'
11:59 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'changing session cookie name due to bug 53032'
07:56 logmsgbot: tstarling synchronized php-1.22wmf12/includes/GlobalFunctions.php 'hack for session bug'
07:22 paravoid: switching text-varnish pybal group back to squids
07:22 Tim: destroyed leaked session with ID 796c2b...
05:50 paravoid: authdns-update: add ADSP (dkim=unknown), DKIM policy record for lists (o=~)
04:20 paravoid: authdns-update: DKIM & DMARC (both no-op, for now)
03:01 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Aug 19 03:00:26 UTC 2013
02:42 logmsgbot: LocalisationUpdate completed (1.22wmf13) at Mon Aug 19 02:41:19 UTC 2013
02:17 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Mon Aug 19 02:15:45 UTC 2013

August 18

18:55 Elsie: rotated server admin log wikitech page; testing logging
12:27 paravoid: authdns-update: add AAAA for sodium
11:44 mark: Moved eqiad text-varnish traffic back to Squid temporarily
10:15 springle: resuming slave db52 after bug 52853 investigation
07:20 MaxSem: Killed a few hung queries on db1026 that were causing massive lag
02:25 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Aug 18 02:23:36 UTC 2013
02:14 logmsgbot: LocalisationUpdate completed (1.22wmf13) at Sun Aug 18 02:13:27 UTC 2013
02:07 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Sun Aug 18 02:05:50 UTC 2013

August 17

15:03 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'returning db1009 to the pool after sync'
02:41 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Aug 17 02:40:08 UTC 2013
02:21 logmsgbot: LocalisationUpdate completed (1.22wmf13) at Sat Aug 17 02:19:58 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Sat Aug 17 02:14:54 UTC 2013

August 16

21:58 logmsgbot: demon synchronized php-1.22wmf12/extensions/CirrusSearch 'No actual changes, was just a change + revert'
21:48 ori-l: Timeouts on last two sync-dirs: srv281, mw1089, mw1173
21:47 logmsgbot: olivneh synchronized php-1.22wmf13/extensions/CoreEvents 'Updating CoreEvents to master for I08f66861b (2/2)'
21:47 logmsgbot: olivneh synchronized php-1.22wmf12/extensions/CoreEvents 'Updating CoreEvents to master for I08f66861b (1/2)'
21:07 logmsgbot: reedy synchronized docroot and w
20:47 logmsgbot: reedy synchronized docroot and w
19:38 cmjohnson1: shutting down brewster yet again to replace the disk
17:31 logmsgbot: demon synchronized php-1.22wmf13/extensions/CirrusSearch/CirrusSearch.body.php
16:52 cmjohnson1: brewster going down again
16:22 cmjohnson1: shutting down brewster to replace bad disk
16:22 bblack: merged vhtcpd ganglia stuff and the VCL for PURGE hash_ignore_busy into puppet
15:10 logmsgbot: reedy synchronized wmf-config/ 'touch'
15:07 logmsgbot: reedy synchronized database lists files: Ad tyvwiki to visualeditor.dblist
09:38 paravoid: rolling reboot of ceph ms-be nodes
08:11 apergos: sq41 hardware issues, see rt 5615
05:46 paravoid: downgraded puppetmaster to previous release, new minor update had significantly increased CPU requirements and DoSed stafford
02:59 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Aug 16 02:57:56 UTC 2013
02:36 logmsgbot: LocalisationUpdate completed (1.22wmf13) at Fri Aug 16 02:35:14 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Fri Aug 16 02:15:11 UTC 2013
00:00 logmsgbot: demon synchronized php-1.22wmf13/includes/specials/SpecialSearch.php 'Renamed backend to srbackend'

August 15

23:38 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Debug log settings for Cirrus'
23:33 ^d: finished indexing all of test2wiki
23:15 ^d: running ES indexing for test2wiki in screen on terbium
23:10 ^d: built test2wiki index in ES
23:08 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'test2wiki to CirrusSearch'
21:52 logmsgbot: olivneh Finished syncing Wikimedia installation... : E3 deploy: GuidedTour & GettingStarted
21:40 logmsgbot: olivneh Started syncing Wikimedia installation... : E3 deploy: GuidedTour & GettingStarted
21:25 apergos: live hacked /etc/nagios/puppet_servicegroups.cfg and /etc/nagios/puppet_hostgroups.cfg on neon to add host and service group 'cache_misc_eqiad', icinga wouldn't start without it
21:17 ori-l: previous two syncs: timeouts on srv281, mw1089, mw1173
21:16 logmsgbot: olivneh synchronized php-1.22wmf12/extensions/GuidedTour 'E3 deploy: GuidedTour to master'
21:15 logmsgbot: olivneh synchronized php-1.22wmf12/extensions/GettingStarted 'E3 deploy: GettingStarted to master'
19:49 paravoid: ceph: adding osd.93
19:42 Reedy: Added wb_property_info table to wikidatawiki and testwikidatawiki
18:49 paravoid: rebooting stafford & sockpuppet for kernel upgrades
18:21 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: 1.22wmf13 phase1
18:06 cmjohnson1: powering down ms-be1008 to update hard drive firmware
17:59 logmsgbot: reedy synchronized php-1.22wmf13/includes/specials/SpecialNewimages.php
16:36 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: revert test2wiki to 1.22wmf12 till window
16:29 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.22wmf13 and build l10n cache
16:16 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.22wmf13 and build l10n cache
16:09 logmsgbot: reedy synchronized php-1.22wmf13/
15:40 logmsgbot: reedy synchronized docroot and w
15:38 logmsgbot: reedy synchronized php-1.22wmf13/
08:15 springle: starting pt-table-checksum on db1034 bug 52853
07:03 springle: delaying slave db52 during bug 52853 investigation
06:49 logmsgbot: tstarling synchronized wmf-config/db-eqiad.php 'depooling db1009 at springle's request'
06:47 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
06:46 logmsgbot: tstarling synchronized wmf-config/CirrusSearch-production.php
06:45 logmsgbot: tstarling synchronized wmf-config/CirrusSearch-common.php
06:44 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
06:44 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings-labs.php
06:43 Tim: deploying CirrusSearch configuration changes (fa29915)
02:56 paravoid: rebooting iron, kernel upgrade
02:44 paravoid: rebooting bast1001, kernel upgrade
02:32 paravoid: rebooting fenari, kernel upgrade
02:22 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Aug 15 02:20:39 UTC 2013
02:09 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Thu Aug 15 02:08:17 UTC 2013

August 14

17:43 logmsgbot: csteipp synchronized php-1.22wmf12/includes
12:24 mark: Moved non-https, non-IPv6 eqiad traffic for *.wikimedia.org (wikimedialb) to text-varnish, not caching login.wikimedia.org
11:14 paravoid: swift->ceph thumb sync (swiftrepl @ ms-fe1002)
11:00 akosiaris: stoping etherpad-lite for maintainance (character set hell)
10:49 paravoid: upgrading ceph to 0.67
10:02 mark: Moved non-https, non-IPv6 eqiad traffic for *.wikimedia.org (wikimedialb) back to Squid
09:41 mark: Moved non-https, non-IPv6 eqiad traffic for *.wikimedia.org (wikimedialb) to text-varnish
02:19 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Aug 14 02:18:04 UTC 2013

August 13

22:56 cmjohnson1: authdns update
18:52 RobH: wikipedia.in is now working, huzzah.
18:42 RobH: updated apache redirects, but dsh stuff wont workfor me so hacking an apache gracefull all via salt
17:49 RobH: rob is testing his new ssh settings to see if he can dsh graceful all apaches without errors, no actual changes are being made
08:52 akosiaris1: taking down epl.wikimedia.org for maintanance
07:30 akosiaris: enabled https redir for epl.wikimedia.org.
06:00 apergos: ran puppet on antimony and restarted apache to pick up new robots.txt (see r/78919 )
05:58 apergos: restarted apache on stafford, puppet was dead
02:17 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Aug 13 02:15:58 UTC 2013
02:07 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Tue Aug 13 02:05:45 UTC 2013

August 12

18:21 cmjohnson1: powering down testsearch1001 and 1002 to compare settings
17:23 ^d: gerrit restarting
11:06 LeslieCarr: removed wpasupplicant from pdf3
10:38 LeslieCarr: removing wpasupplicant from pdf1
10:38 LeslieCarr: removing wirelesstools from pdf1
10:37 LeslieCarr: removing wirelesstools from pdf2
10:35 LeslieCarr: uninstalling wpasupplicant from pdf2
10:34 LeslieCarr: dist-upgrading pdf2
10:26 LeslieCarr: rebooting pdf2
09:54 akosiaris: stopping etherpad-lite to load old etherpad data
08:02 Krinkle: Ran updateiwcache to fix issue with viwikivoyage interwiki links resolving to $lang.wikipedia.org instead of wikivoyage
07:59 logmsgbot: krinkle synchronized wmf-config/interwiki.cdb 'Updating interwiki cache'
05:12 logmsgbot: reedy synchronized php-1.22wmf12/extensions/WikimediaMaintenance/addToSites.php
03:55 LeslieCarr: restarting apache2 on mw1153
02:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Aug 12 02:24:49 UTC 2013
02:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Revert back to normal wikidata logo'
02:12 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Mon Aug 12 02:10:38 UTC 2013

August 11

14:00 Tim: disabled offending query source and killed queries again
13:59 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'disabling Special:SupportedLanguages due to apache/mysql overload'
13:59 paravoid: powercycling db1009, down for 20h, kernel prints "BUG: soft lockup"
13:46 Tim: killing SpecialSupportedLanguages on db1004 and other s4 slaves
07:18 logmsgbot: reedy synchronized php-1.22wmf12/extensions/CentralAuth 'rv back'
07:15 logmsgbot: reedy synchronized php-1.22wmf12/extensions/CentralAuth/maintenance/populateHomeDB.php
07:09 logmsgbot: reedy synchronized php-1.22wmf12/extensions/CentralAuth/CentralAuthUser.php
06:15 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable upload on viwikivoyage'
06:09 logmsgbot: reedy synchronized wmf-config/ 'touch'
06:07 logmsgbot: reedy synchronized database lists files:
05:58 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
05:57 logmsgbot: reedy synchronized database lists files:
05:06 logmsgbot: reedy synchronized wmf-config/interwiki.cdb 'Updating interwiki cache'
04:45 logmsgbot: reedy synchronized wmf-config 'touch'
04:43 logmsgbot: reedy synchronized database lists files: Add tyvwiki to wikidataclient
04:36 logmsgbot: reedy synchronized php-1.22wmf12/extensions/WikimediaMaintenance/addToSites.php
04:27 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
04:27 logmsgbot: reedy synchronized database lists files:
04:22 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
03:05 logmsgbot: reedy synchronized wmf-config/
02:50 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Aug 11 02:49:14 UTC 2013
02:17 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Sun Aug 11 02:15:52 UTC 2013
01:32 logmsgbot: reedy synchronized php-1.22wmf12/extensions/CodeReview

August 10

23:56 LeslieCarr: restarted gitblit on antimony
12:33 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
11:03 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
07:17 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'rv'
07:13 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Disable CR proxy'
05:59 mutante: deleting codereview-proxy (pre-gerrit,svn) from bugzilla server
05:59 mutante: deleting broken bzapi files from bugzilla server
05:53 mutante: apt-get upgrading kaulen (bugzilla)
02:28 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Aug 10 02:27:23 UTC 2013
02:13 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Sat Aug 10 02:12:01 UTC 2013

August 9

18:49 paravoid: running ceph->swift thumb sync on ms-fe1002
18:40 paravoid: running originals swift->ceph sync (syncFileBackend.php on terbium)
09:31 mark: Assigned IPv4 mapped IPv6 address to mchenry
08:13 mutante: apt-get upgrading zirconium
04:09 paravoid: authdns-update: added "tyv" to langlist
04:03 logmsgbot: reedy synchronized langlist 'attempt 2'
04:02 logmsgbot: reedy synchronized langlist
03:19 logmsgbot: reedy synchronized docroot/wikidata/WikidataHK.png 'Compress'
02:16 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Aug 9 02:15:21 UTC 2013
02:06 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Fri Aug 9 02:05:28 UTC 2013
01:36 logmsgbot: reedy synchronized wmf-config 'Sync-ing for consistency (no changes)'

August 8

20:25 cmjohnson1: working on carbon
18:26 mark: Removed /dev/sda from sw raid arrays on brewster
08:54 logmsgbot: reedy synchronized wmf-config/
08:42 mutante: on overloaded antimony: stopped gitblit, ran puppet, blocked Googlebot, restarted apache and gitblit
08:22 mutante: git.wikimedia.org back
08:21 mutante: attempted to restart gitblit on antimony
08:00 logmsgbot: reedy synchronized wmf-config/
07:58 logmsgbot: reedy synchronized docroot and w
02:18 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Aug 8 02:16:52 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Thu Aug 8 02:06:51 UTC 2013
00:36 mwalker: updated fundraising.wm.o wiki from 0e08f1e to b104c72

August 7

23:48 LeslieCarr: rebooting cr2-knams
23:44 LeslieCarr: deactivating external bgp neighbors on cr2-knams
22:47 ^d: reloaded gerrit replication plugin to pick up new config
21:48 apergos: stopping and restarting gitblit for google testing
21:26 paravoid: authdns-update: adding lists AAAA & (neutral) SPF
19:11 paravoid: authdns-update: removing lists' SPF
17:48 RobH: authdns-update (just incase i break it all)
17:25 RobH: authdnsiupdate for lists.w.o spf records
12:10 paravoid: removing 91.198.174.7/32 from ssl3001 & maerlant, old/deprecated ipv6 testing IP
07:17 apergos: restarted gitblit, (btw the init script did not stop it properly)
02:23 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Aug 7 02:22:30 UTC 2013
02:10 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Wed Aug 7 02:08:47 UTC 2013

August 6

16:02 Jeff_Green: authdns-update to flip ticket.wikimedia.org to williams
10:37 Jeff_Green: stopped icinga notifications for db1048, db1046, db48, db49
10:22 Jeff_Green: db1046 mysql replication stopped
10:17 Jeff_Green: stop exim on williams.wikimedia.org
10:17 Jeff_Green: stop apache on williams.wikimedia.org
02:17 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Aug 6 02:16:18 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Tue Aug 6 02:06:34 UTC 2013

August 5

17:50 akosiaris: added d2to1, pep8, python-setuptools-git packages in apt.wikimedia.org. Packages were created by hashar using backports from ubuntu. I added debian/changelog entry to get an upgrade friendly version and run lintian on each one of them with no errors.
15:56 akosiaris: cleared old cruft snapmirror replication from nas1-a to nas1001-a. Direction has been reverted long ago
14:01 mark: Put non-European wikivoyage traffic on the eqiad text Varnish cluster
09:27 paravoid: powercycling neon, OOM'ed, "login: failure forking: Cannot allocate memory" etc.; icinga down since ~01:00 UTC
02:16 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Aug 5 02:14:37 UTC 2013
02:07 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Mon Aug 5 02:05:54 UTC 2013

August 4

17:18 Nemo_bis: antimony (gitblit) overloaded again as of yesterday around 19 UTC
02:14 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Aug 4 02:13:09 UTC 2013
02:06 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Sun Aug 4 02:04:59 UTC 2013

August 3

19:47 Ryan_Lane: repooling ssl1007-9
14:42 logmsgbot: reedy synchronized wmf-config/squid.php 'ssl1007-ssl1009'
08:00 Ryan_Lane: depooling ssl1007-9 for now. They need to be added to the MW config.
02:38 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Aug 3 02:36:54 UTC 2013
02:18 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Sat Aug 3 02:16:47 UTC 2013
00:15 marktraceur: Deployed new version of parsoid-regressions via Jenkins Job Builder
00:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php

August 2

22:55 Ryan_Lane: repooling ssl1001-1007
22:52 Ryan_Lane: deppoling ssl1007 for more testing
22:43 Ryan_Lane: depooling ssl1005-6 to further test load
22:35 Ryan_Lane: depooling ssl1001-4 to measure load on new hardware
22:11 Ryan_Lane: pooling ssl1007-9
21:13 cmjohnson1: dns update
19:37 cmjohnson1: dns update
19:22 paravoid: installing mexia
14:32 LeslieCarr: pdf3 unresponsive on console , rebooting
13:13 paravoid: rolling reboot ceph nodes for kernel upgrades
13:06 paravoid: upgrading ceph to 0.67-rc3
08:19 springle: restarting slave threads db32 db52 db39 db51 db45 db43 db56 after OSC bug 49194
05:34 logmsgbot: catrope synchronized php-1.22wmf12/resources/startup.js 'touch'
05:34 logmsgbot: catrope synchronized php-1.22wmf12/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.ViewPageTarget.init.js 'touch'
05:32 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Add plumbing for new VE config vars'
05:31 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Add VE config to set Edit source / Edit beta on enwiki'
05:30 logmsgbot: catrope Finished syncing Wikimedia installation... : Update VE to master
05:23 logmsgbot: catrope Started syncing Wikimedia installation... : Update VE to master
02:35 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Aug 2 02:34:08 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Fri Aug 2 02:14:20 UTC 2013

August 1

22:19 cmjohnson1: authdns update
21:24 logmsgbot: mflaschen synchronized php-1.22wmf12/extensions/GuidedTour/ 'Sync GuidedTour for E3 deployment'
21:11 LeslieCarr: restarting gerrit on manganese
21:09 cmjohnson1: dns update
21:06 Krinkle: Gerrit has been slow (~ 10 second per request) for about 30 minutes. Seems down entirely now.
20:19 RobH: authdnsupdate
19:40 cmjohnson1: dns update
19:36 hashar: Slightly described the Zuul pipelines at https://integration.wikimedia.org/zuul/ . Descriptions are in the zuul-config layout.yaml file.
19:23 cmjohnson1: morebots are you working?
19:21 cmjohnson1: authdns update
18:54 akosiaris: created aggregates backups on nas1001-a and nas1-a. Created flexible volumes baculasd1 on each and setup a snapmirror relationship between them (from eqiad to pmtpa)
18:42 cmjohnson1: authdns update
17:29 logmsgbot: csteipp synchronized wmf-config/CommonSettings.php 'EventLogging for CentralAuth'
17:28 logmsgbot: csteipp synchronized wmf-config/InitialiseSettings.php 'EventLogging for CentralAuth'
16:49 akosiaris: set options disk.auto_assign off on nas1001-{a,b}. Then performed disk remove_ownership for 20 disks on nas1001-b and disk assign all on nas1001-a
16:45 logmsgbot: bsitu synchronized php-1.22wmf12/extensions/Echo/Echo.php 'Update Echo to master'
16:05 akosiaris: added my ssh public key to root user of nas1-{a,b}, nas1001-{a,b}
15:32 mark: Prepending AS 14907 once to AS 30217
15:12 Jeff_Green: accidentally powercycled iron, trying to do iodine. fail.
06:12 springle: delaying db32 during OSC bug 49194
06:06 springle: delaying db52 during OSC bug 49194
05:55 springle: delaying db39 during OSC bug 49194
05:44 springle: delaying db51 during OSC bug 49194
05:41 springle: delaying db45 during OSC bug 49194
05:37 springle: delaying db43 during OSC bug 49194
05:29 springle: delaying db56 during OSC bug 49194
05:16 Tim: deleted half-created account melissajan0@enwiki, broken by bug 52368
04:42 ori-l: Not synced to srv281, mw1089, mw1173 (connection timed out).
04:41 logmsgbot: olivneh synchronized php-1.22wmf12/extensions/VisualEditor/VisualEditor.php 'Fix for bug 52368: VisualEditor.php attempts to register a hook handler that was removed'
02:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Aug 1 02:34:57 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Thu Aug 1 02:15:31 UTC 2013
01:55 logmsgbot: catrope synchronized php-1.22wmf12/extensions/VisualEditor 'Update VE to master'

2000s

Archive 1: 2004 Jun - 2004 Sep
Archive 2: 2004 Oct - 2004 Nov
Archive 3: 2004 Dec - 2005 Mar
Archive 4: 2005 Apr - 2005 Jul
Archive 5: 2005 Aug - 2005 Oct, with revision history 2004-06-23 to 2005-11-25
Archive 6: 2005 Nov - 2006 Feb
Archive 7: 2006 Mar - 2006 Jun
Archive 8: 2006 Jul - 2006 Sep
Archive 9: 2006 Oct - 2007 Jan, with revision history 2005-11-25 to 2007-02-21
Archive 10: 2007 Feb - 2007 Jun
Archive 11: 2007 Jul - 2007 Dec
Archive 12: 2008 Jan - 2008 Jul
Archive 12a: 2008 Aug
Archive 12b: 2008 Sept
Archive 13: 2008 Oct - 2009 Jun
Archive 14: 2009 Jun - 2009 Dec

2010s

2020s