Server admin log/Archive 23

From Wikitech
Jump to: navigation, search

December 31

  • 22:59 ori: morebots went missing. job was active on toollabs but stuck. nothing useful in logs. restarted.
  • 11:59 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'adding *.raa.se to the wgCopyUploadsDomains array 104683'
  • 07:39 springle: resume schema changes for gerrit 51675 externallinks.el_id
  • 07:26 logmsgbot: aaron finished scap: php-1.23wmf6 Timing test
  • 07:22 logmsgbot: aaron started scap: php-1.23wmf6 Timing test
  • 05:25 logmsgbot: aaron finished scap: php-1.23wmf6 Trying timing test again
  • 05:20 logmsgbot: aaron started scap: php-1.23wmf6 Trying timing test again
  • 05:16 springle: package upgrade sanitarium hosts db1053 db1054 db1057
  • 03:23 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Dec 31 03:23:18 UTC 2013
  • 03:20 logmsgbot: aaron finished scap: php-1.23wmf6 Timing test
  • 03:01 logmsgbot: aaron started scap: php-1.23wmf6 testing scap timing
  • 02:09 awight: listeners updated from 1ebcfa83671b2caade50948f88c2be5cc9798692 to a4748700afaa4a7c5e9b84bfa1f78e3298e04ad1
  • 01:06 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Disable MessageBlobStore::clear() via hook'
  • 01:04 awight: updated tools from 874e7039a24f6fa1c23c9d30435dd4d5e9038906 to 7f91e78c211d3ba016e2fa469f1d7093540e18c1
  • 00:24 Aaron|home: Running a swift eqiad->tampa sync script (no-deletes) to catch any inconsistencies that built up
  • 00:14 awight: tools updated from 64ae848b2518bc9e838b42dfbb894eb1eb845043 to 874e7039a24f6fa1c23c9d30435dd4d5e9038906
  • 00:11 logmsgbot: maxsem synchronized wmf-config/CommonSettings.php 'https://gerrit.wikimedia.org/r/103611'
  • 00:07 logmsgbot: maxsem synchronized php-1.23wmf8/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/104682'

December 30

  • 23:52 awight: tools updated from from 47775021e832a5d1aa80a657086ccb4bc73d4577 to 64ae848b2518bc9e838b42dfbb894eb1eb845043
  • 23:34 awight: updated tools from dc39dfdb381d8279d995948f50cb5fe5b3b6de4c to 47775021e832a5d1aa80a657086ccb4bc73d4577
  • 23:17 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Removed fileJournal config; this has been unused for some time'
  • 23:10 hashar: Jenkins / gallium deleting build history from 2012 for operations* and mediawiki* jobs. That is running in a screen.
  • 23:05 hashar: Jenkins purged build history of mediawiki-core-lint from before July 2013.
  • 22:57 hashar: Jenkins on gallium, /var/lib/jenkins/jobs/mediawiki-core-lint/builds eventually reached the maximum possible number of file entries. Deleting a bunch
  • 22:44 hashar: jenkins job mediawiki-core-lint is broken since 20:35 UTC roughly
  • 22:27 logmsgbot: reedy synchronized wmf-config/
  • 22:18 logmsgbot: reedy synchronized wmf-config/
  • 22:04 logmsgbot: reedy synchronized docroot and w
  • 21:58 logmsgbot: reedy updated /a/common to I028589438: Clean up wgSiteName in InitialiseSettings
  • 21:57 logmsgbot: reedy synchronized wmf-config/
  • 20:58 logmsgbot: aaron finished scap: php-1.23wmf6 Scap timing test
  • 20:50 logmsgbot: aaron started scap: php-1.23wmf6 Scap timing test
  • 20:43 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 20:21 logmsgbot: reedy synchronized php-1.23wmf8/extensions/CentralAuth/
  • 20:07 awight: updated tools from 882edb438b384c98c82974067ad05f844af61480 to dc39dfdb381d8279d995948f50cb5fe5b3b6de4c
  • 20:02 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Configured $wgJobBackoffThrottling'
  • 19:49 logmsgbot: reedy synchronized php-1.23wmf8/extensions/CentralAuth/
  • 19:37 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: non wikipedias to 1.23wmf8
  • 19:35 logmsgbot: reedy updated /a/common to Ib1ce6e204: Actually turn cirrus on for new wikis
  • 19:33 awight: tools updated from da577429836bfb056fa3991d58eda24d3812f2eb to 882edb438b384c98c82974067ad05f844af61480
  • 19:25 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php
  • 19:16 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php
  • 19:13 logmsgbot: manybubbles finished scap: active Update CirrusSearch and Elastica to master
  • 19:02 logmsgbot: manybubbles synchronized php-1.23wmf7/includes/media/MediaHandler.php 'Backport getEntireText so cirrus stop complaining'
  • 18:32 logmsgbot: manybubbles started scap: active Update CirrusSearch and Elastica to master
  • 18:01 cmjohnson1: powering down search1005 to swap /dev/sdb
  • 17:56 manybubbles: aborted scap
  • 17:55 logmsgbot: manybubbles started scap: active Update CirrusSearch and Elastica to master
  • 17:29 ^d: aborted scap
  • 17:29 logmsgbot: demon started scap
  • 17:25 ^d: aborted scap
  • 17:25 logmsgbot: demon started scap: active Cirrus and Elastica to master
  • 12:58 akosiaris: killed etherpad-old.wikimedia.org
  • 02:32 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Dec 30 02:32:44 UTC 2013
  • 02:21 logmsgbot: LocalisationUpdate completed (1.23wmf8) at Mon Dec 30 02:21:00 UTC 2013
  • 02:06 logmsgbot: LocalisationUpdate completed (1.23wmf7) at Mon Dec 30 02:06:31 UTC 2013

December 29

  • 22:44 logmsgbot: aaron synchronized php-1.23wmf8/includes/job/JobQueueFederated.php '6d3abab7cd6d344e903c25d4b24bda9ef239652b'
  • 02:30 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Dec 29 02:30:44 UTC 2013
  • 02:15 logmsgbot: LocalisationUpdate completed (1.23wmf8) at Sun Dec 29 02:15:24 UTC 2013
  • 02:08 logmsgbot: LocalisationUpdate completed (1.23wmf7) at Sun Dec 29 02:08:48 UTC 2013

December 28

  • 02:27 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Dec 28 02:27:24 UTC 2013
  • 02:13 logmsgbot: LocalisationUpdate completed (1.23wmf8) at Sat Dec 28 02:13:10 UTC 2013
  • 02:07 logmsgbot: LocalisationUpdate completed (1.23wmf7) at Sat Dec 28 02:07:43 UTC 2013
  • 00:13 logmsgbot: aaron finished scap
  • 00:02 logmsgbot: aaron started scap
  • 00:00 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files:

December 27

  • 23:59 AaronSchulz: doing some sync script timing
  • 22:28 ori: Stopping uWSGI on Tungsten to debug Graphite issue
  • 21:20 awight: updated crm from 0dbfaf8ef9dcf72ec88f1ff1ad38dc7273d987a6 to 81d06d3d21c26c38fb80cd027f8b66308ca3fac7
  • 17:55 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 05:17 awight: updated tools from 9f96cad6739504048a32812fff655f14b3371411 to da577429836bfb056fa3991d58eda24d3812f2eb
  • 04:02 awight: updated tools from 727505d13e5f1b7e0d1ece20832900a9cfdd2870 to 9f96cad6739504048a32812fff655f14b3371411
  • 03:00 awight: tools updated from fb30e4e41d27dd4b5b29c9f002c413e3941b00c0 to 727505d13e5f1b7e0d1ece20832900a9cfdd2870
  • 02:35 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Dec 27 02:35:51 UTC 2013
  • 02:18 logmsgbot: LocalisationUpdate completed (1.23wmf8) at Fri Dec 27 02:18:20 UTC 2013
  • 02:10 logmsgbot: LocalisationUpdate completed (1.23wmf7) at Fri Dec 27 02:10:22 UTC 2013
  • 01:46 awight: updated civicrm from 87dea8afb964d0622f3ee92b6e120915846fde56 to 0dbfaf8ef9dcf72ec88f1ff1ad38dc7273d987a6
  • 01:30 awight: updated tools from 3eaf88c3dda53ec8c521e664ff0fd25b50a4814f to fb30e4e41d27dd4b5b29c9f002c413e3941b00c0
  • 00:14 awight: updated tools to 3eaf88c3dda53ec8c521e664ff0fd25b50a4814f

December 26

  • 23:59 logmsgbot: aaron started scap
  • 21:18 logmsgbot: aaron finished scap
  • 21:07 MaxSem: Recreating GeoData index
  • 21:03 logmsgbot: aaron started scap
  • 20:52 andrewbogott: restarting parsoid cluster via salt
  • 20:49 subbu: updated localsettings.js to unbreak VE on wikis with "-" in their name (ex: nds-nl)
  • 14:29 akosiaris: killed stuck puppet agent on db9 and forced manual puppet run.
  • 02:35 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Dec 26 02:35:43 UTC 2013
  • 02:18 logmsgbot: LocalisationUpdate completed (1.23wmf8) at Thu Dec 26 02:18:37 UTC 2013
  • 02:10 logmsgbot: LocalisationUpdate completed (1.23wmf7) at Thu Dec 26 02:10:16 UTC 2013

December 25

  • 02:40 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Dec 25 02:40:35 UTC 2013
  • 02:14 logmsgbot: LocalisationUpdate completed (1.23wmf8) at Wed Dec 25 02:14:49 UTC 2013
  • 02:14 logmsgbot: LocalisationUpdate completed (1.23wmf7) at Wed Dec 25 02:14:05 UTC 2013
  • 02:00 logmsgbot: aaron finished scap
  • 01:53 logmsgbot: aaron started scap
  • 01:53 logmsgbot: aaron finished scap
  • 01:32 logmsgbot: aaron started scap
  • 01:06 logmsgbot: ori finished scap
  • 01:01 logmsgbot: ori started scap
  • 01:00 logmsgbot: aaron finished scap
  • 00:57 logmsgbot: aaron started scap
  • 00:51 logmsgbot: aaron started scap
  • 00:45 logmsgbot: aaron started scap
  • 00:43 logmsgbot: aaron started scap
  • 00:40 logmsgbot: aaron started scap
  • 00:14 logmsgbot: aaron finished scap
  • 00:07 logmsgbot: aaron started scap

December 24

  • 13:31 logmsgbot: reedy synchronized php-1.23wmf7/extensions/GlobalBlocking 'bug 58934 Icc47a2d6367c0b906e40e068635c9fda07108e0f'
  • 03:08 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's1 db1033 to full steam'
  • 02:28 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Dec 24 02:28:42 UTC 2013
  • 02:14 logmsgbot: LocalisationUpdate completed (1.23wmf8) at Tue Dec 24 02:14:09 UTC 2013
  • 02:08 logmsgbot: LocalisationUpdate completed (1.23wmf7) at Tue Dec 24 02:08:04 UTC 2013

December 23

  • 22:15 awight: updated crm from ce73eaa0ea6019f76c63d0cca097bf828eb933e2 to 87dea8afb964d0622f3ee92b6e120915846fde56
  • 20:06 paravoid: cr1-eqiad/cr1-sdtpa: new BGP policy to reset MED for XO
  • 16:59 cmjohnson1: dns update
  • 15:30 cmjohnson1: labnet1001 relocating racks C4 to B3 eqiad
  • 13:24 springle_: starting xtrabackup clone db1001 to db1016
  • 10:39 akosiaris: uploaded php5_5.3.10-1ubuntu3.9+wmf1 to apt.wikimedia.org
  • 09:54 paravoid: rebooting cp1065 with Linux 3.2
  • 07:43 springle: taking down db1016 for xfs_check, locked up and: XFS (dm-0): xlog_space_left: head behind tail
  • 05:57 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'warm up db1033 in s1'
  • 02:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Dec 23 02:26:35 UTC 2013
  • 02:13 logmsgbot: LocalisationUpdate completed (1.23wmf8) at Mon Dec 23 02:13:09 UTC 2013
  • 02:07 logmsgbot: LocalisationUpdate completed (1.23wmf7) at Mon Dec 23 02:07:23 UTC 2013
  • 00:50 springle: xtrabackup clone db1049 to db1033
  • 00:37 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'rebalance group loads while db1050 out'

December 22

  • 19:35 apergos: db1050 boot: (during fsck) hung at "The disk drive for /a is not ready yet or not present. Continue waiting, S to skip.." etc
  • 19:34 apergos: db1050 boot: "[ 65.625950] device-mapper: table: 252:3: snapshot: Snapshot cow pairing for exception table handover failed"
  • 19:33 logmsgbot: ori synchronized wmf-config/db-eqiad.php 'Depool db1050'
  • 19:32 logmsgbot: ori updated /a/common to If33894260: Depool db1050; unresponsive
  • 19:27 apergos: powrcycling db1050, inaccessible
  • 02:28 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Dec 22 02:28:45 UTC 2013
  • 02:15 logmsgbot: LocalisationUpdate completed (1.23wmf8) at Sun Dec 22 02:15:27 UTC 2013
  • 02:08 logmsgbot: LocalisationUpdate completed (1.23wmf7) at Sun Dec 22 02:08:39 UTC 2013

December 21

  • 02:33 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Dec 21 02:33:00 UTC 2013
  • 02:18 logmsgbot: LocalisationUpdate completed (1.23wmf8) at Sat Dec 21 02:18:39 UTC 2013
  • 02:09 logmsgbot: LocalisationUpdate completed (1.23wmf7) at Sat Dec 21 02:09:34 UTC 2013

December 20

  • 21:37 mutante: cp1065 - currently init 1 sitting at maintenance but not powered down, i will poke later or anyone can
  • 20:43 mutante: cp1065 - was 3.11.0-13-generic, rebooting for 3.2.0-57-generic
  • 20:38 mutante: powercycled cp1065 again, will upgrade kernel this time
  • 19:09 mutante: cp1065 - [19060.904886] BUG: scheduling while atomic: kworker/11:0/27073/0x00000200
  • 19:09 mutante: powercycling cp1065
  • 16:18 andrewbogott: merged a couple of tiny OpenStackManager patches on wikitech
  • 14:34 hashar: Jenkins removed http_proxy env variables from lanthanum.eqiad.wmnet does not play well with git urls :D
  • 14:28 hashar: Jenkins setting http_proxy and https_proxy on lanthanum.eqiad.wmnet to point to carbon.wikimedia.org (eqiad web proxy)
  • 13:52 paravoid: powercycling cp1065
  • 11:53 akosiaris: added akosiaris account on all junipers. Also added faidon account wherever it did not exist and synchronized its uid (only one uid change on csw2-esams happened)
  • 11:16 hashar: updated Jenkins job linting operations/apache-config.git , now detects rewrites.conf not being updated when redirects.dat has been changed.
  • 10:19 apergos: powercycled cp1065 and had to manually start varnish, same as yesterday
  • 02:04 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'vslow query load balancing'
  • 01:12 logmsgbot: spage synchronized php-1.23wmf8/extensions/Flow/includes/Notifications/Controller.php 'Bug 58705 notification fix pt.3'
  • 01:08 logmsgbot: spage synchronized php-1.23wmf8/extensions/Echo/includes/EmailFormatter.php 'Bug 58705 notification fix pt.2'
  • 01:06 logmsgbot: spage synchronized php-1.23wmf8/extensions/Echo/formatters/BasicFormatter.php 'Bug 58705 notification fix'
  • 00:48 logmsgbot: ori synchronized php-1.23wmf8/extensions/WikimediaEvents 'I7cad8fd35fd: Update WikimediaEvents to master'
  • 00:47 logmsgbot: ori synchronized php-1.23wmf7/extensions/WikimediaEvents 'I7cad8fd35fd: Update WikimediaEvents to master'
  • 00:22 mutante: create /a/mw-log/Bug58676.log for yurik, touch, chown udplog ..etc
  • 00:10 logmsgbot: ori synchronized php-1.23wmf7/extensions/WikimediaEvents 'Ibe918eb96: Update WikimediaEvents to master for Ia6f758068 (Log event when new editors reach edit milestone)'
  • 00:09 logmsgbot: ori synchronized php-1.23wmf8/extensions/WikimediaEvents 'Ibe918eb96: Update WikimediaEvents to master for Ia6f758068 (Log event when new editors reach edit milestone)'

December 19

  • 21:47 logmsgbot: csteipp finished scap: bug 58472, 57550, 58553
  • 21:08 logmsgbot: csteipp started scap: bug 58472, 57550, 58553
  • 19:53 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 19:52 logmsgbot: reedy synchronized php-1.23wmf7/includes/Message.php
  • 19:20 logmsgbot: reedy synchronized docroot and w
  • 19:19 logmsgbot: reedy synchronized wmf-config/
  • 19:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: phase1 wikis to 1.23wmf8
  • 19:06 logmsgbot: reedy updated /a/common to I952672334: Rename Persian Wikibooks per community consensus
  • 19:05 logmsgbot: reedy synchronized wmf-config/
  • 19:01 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All wikipedias to 1.23wmf7
  • 19:00 logmsgbot: reedy updated /a/common to I22cfefa3c: Add an alias to NS_PROJECT on Old English Wiktionary
  • 18:33 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.23wmf7 till window
  • 18:32 logmsgbot: reedy finished scap: Build l10n cache and testwiki to 1.23wmf8
  • 17:56 logmsgbot: reedy started scap: Build l10n cache and testwiki to 1.23wmf8
  • 17:48 logmsgbot: reedy synchronized docroot and w
  • 17:47 logmsgbot: reedy updated /a/common to Ib6bc15109: previously missed deleted wikis, bug 58680
  • 17:46 logmsgbot: reedy synchronized php-1.23wmf8 'Staging'
  • 09:50 apergos: powercycled cp1065, unresponsive, no messages on mgmt console
  • 03:49 Krinkle: Reloading Zuul to deploy I5b759c388db5e5a
  • 03:20 springle: starting schema changes for gerrit 101170 page_links_updated
  • 02:58 logmsgbot: springle synchronized database lists files:
  • 02:53 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Dec 19 02:53:46 UTC 2013
  • 02:31 logmsgbot: LocalisationUpdate completed (1.23wmf6) at Thu Dec 19 02:31:31 UTC 2013
  • 02:16 logmsgbot: LocalisationUpdate completed (1.23wmf7) at Thu Dec 19 02:16:45 UTC 2013
  • 00:10 subbu: updated parsoid config to add support for tyv and min wikipedias
  • 00:09 mutante: doing parsoid restart via salt, with the 10% option

December 18

  • 23:44 Krinkle: Zuul pipeline for gate-and-submit fixed (the revert was someone undone after the reload, deployed the same git commit again)
  • 23:36 Krinkle: Reloading Zuul
  • 23:34 Krinkle: Zuul is once again not processing any CR+2 events for the gate-and-submit pipeline
  • 23:03 ottomata: kafka brokers on an21 and an22 are now running with cleaned up topics
  • 23:02 ori-l: zirconium: service apache2 reload for wikimania scholarships app
  • 22:40 Krinkle: Zuul pipeline for gate-and-submit fixed
  • 22:38 Krinkle: Reloading Zuul to deploy Ic8cf602a0aa72
  • 22:32 Krinkle: Zuul seems to be unresponsive to gate-and-submit events for mediawiki/* projects
  • 21:45 ottomata: stopping kafka in order to cleanup old topic 'webrequest-mobile'. Usually we won't have to do cleanups like this, but since this topic is new and no one is using it yes, I'd rather clean up now rather than never.'
  • 20:39 awight: update crm from 7deaa1e1a6354de545fedb0e90b85f117e7df0bd to ce73eaa0ea6019f76c63d0cca097bf828eb933e2
  • 20:24 logmsgbot: reedy updated /a/common to I33c68abb1: Have sysops add and remove users from the 'gwtoolset' group
  • 20:12 logmsgbot: reedy synchronized wmf-config/
  • 19:06 Reedy: Created EducationProgram tables on fawiki
  • 19:05 logmsgbot: reedy synchronized php-1.23wmf7/extensions/GWToolset/
  • 19:03 logmsgbot: reedy synchronized wmf-config/
  • 18:20 logmsgbot: otto synchronized wmf-config 'Syncing wmf-config for Cirrus config updates'
  • 18:19 logmsgbot: otto synchronized cirrus.dblist 'Cirrus config update, adding new wikis'
  • 17:52 logmsgbot: otto synchronized php-1.23wmf6/extensions/CirrusSearch 'updating php-1.23wmf6/extensions/CirrusSearch to master'
  • 17:51 logmsgbot: otto synchronized php-1.23wmf6/extensions/CirrusSearch 'updating php-1.23wmf6/extensions/CirrusSearch to master'
  • 17:46 hashar: Zuul upgrade working as expected as far as I monitored it. Mail sent to wikitech-l and eng list.
  • 17:27 logmsgbot: otto synchronized php-1.23wmf7/extensions/CirrusSearch 'updating php-1.23wmf7/extensions/CirrusSearch to master'
  • 17:23 hashar: Zuul : speedy deploy of a Wikimedia hack to prevent Zuul from updating build description. Seems to slowing down everything. Hack is 102465. Deployed using https://www.mediawiki.org/wiki/Continuous_integration/Zuul#upgrading
  • 16:46 ottomata: updated varnishkafka in apt to 1.0.0-2 and installed on mobile varnishes (this includes fix for logrotate postrotate command)
  • 15:40 paravoid: powercycling cp1065 again, kernel BUG, soft lockup
  • 15:29 paravoid: powercycling cp1065, kmem_alloc deadlock
  • 15:01 hashar: Gerrit back :-)
  • 15:00 hashar: Gerrit dead :(
  • 13:51 paravoid: setting up new transit peering in eqiad
  • 10:00 ori-l: disabled puppet on tungsten to test mwprof changes; re-enabling now
  • 07:37 springle: online schema changes on masters with sql_log_bin=0, indexes only, gerrit 85508
  • 05:32 ori-l: DNS update: scholarships.wikimedia.org CNAME misc-web-lb.eqiad
  • 03:09 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Dec 18 03:09:14 UTC 2013
  • 02:47 logmsgbot: LocalisationUpdate completed (1.23wmf6) at Wed Dec 18 02:47:41 UTC 2013
  • 02:25 logmsgbot: LocalisationUpdate completed (1.23wmf7) at Wed Dec 18 02:25:17 UTC 2013
  • 01:52 logmsgbot: reedy synchronized wmf-config/ 'touch'
  • 01:46 logmsgbot: reedy synchronized wmf-config/ 'Ensuring consistency...'
  • 00:19 logmsgbot: mflaschen synchronized wmf-config 'Deploy Draft namespace to English Wikipedia. Remove old Labs overrides that were used for testing this.'
  • 00:12 springle: ongoing schema changes on slaves, indexing only, logging gerrit 85508
  • 00:05 Krinkle: Reloading zuul to deploy I6df1f4acccd7a9

December 17

  • 23:29 LeslieCarr: starting atftpd on carbon
  • 22:37 logmsgbot: ori synchronized php-1.23wmf6/extensions/NavigationTiming 'I933a1e3a2: Add 'mediaWikiLoadComplete' measurement'
  • 22:30 logmsgbot: ori synchronized php-1.23wmf7/extensions/NavigationTiming 'I933a1e3a2: Add 'mediaWikiLoadComplete' measurement'
  • 22:16 logmsgbot: ori synchronized php-1.23wmf6/resources/startup.js 'I3d5bcf10e: startup.js: log current time as global 'mediaWikiLoadStart
  • 22:15 ori-l: restarted apache on zirconium for scholarships vhost
  • 22:03 logmsgbot: ori synchronized php-1.23wmf7/resources/startup.js 'I3d5bcf10e: startup.js: log current time as global 'mediaWikiLoadStart
  • 21:48 logmsgbot: csteipp synchronized php-1.23wmf7/extensions/TimedMediaHandler 'bug56699'
  • 21:47 logmsgbot: csteipp synchronized php-1.23wmf6/extensions/TimedMediaHandler 'bug56699'
  • 21:37 logmsgbot: csteipp synchronized php-1.23wmf7/extensions/CentralAuth 'bug57081'
  • 21:36 logmsgbot: csteipp synchronized php-1.23wmf6/extensions/CentralAuth 'bug57081'
  • 21:31 logmsgbot: csteipp synchronized php-1.23wmf6/includes 'bug58088'
  • 21:30 logmsgbot: csteipp synchronized php-1.23wmf7/includes 'bug58088'
  • 21:23 logmsgbot: reedy synchronized wmf-config/ 'Enable GWToolset on commonswiki'
  • 21:16 logmsgbot: reedy updated /a/common to I2d1c666e1: Production configuration for GWToolset
  • 21:14 logmsgbot: reedy finished scap: Rebuild l10n cache for GWToolset, remove AFT, no wikis moving onto GWToolset at this point (commented out)
  • 20:45 manybubbles: finished the rolling restart of the elasticsearch cluster. it could have been done more quickly but there was no hurry.
  • 20:45 logmsgbot: reedy started scap: Rebuild l10n cache for GWToolset, remove AFT, no wikis moving onto GWToolset at this point (commented out)
  • 20:05 Ryan_Lane: deploying change 102285 to OpenStackManager on virt0
  • 19:58 logmsgbot: reedy synchronized php-1.23wmf7/extensions/GWToolset 'Staging'
  • 19:54 logmsgbot: reedy synchronized php-1.23wmf7/includes/filerepo/LocalRepo.php 'Fix fatal I12513b40453573124e838d54a72a2f9a2d3de338'
  • 19:32 logmsgbot: reedy synchronized wmf-config/ 'Revert Revert Cross-wiki backlink purging for commons file changes'
  • 19:27 logmsgbot: reedy synchronized wmf-config/ 'Revert Cross-wiki backlink purging for commons file changes'
  • 19:26 logmsgbot: reedy updated /a/common to Id4124ef28: All non wikipedias to 1.23wmf7
  • 19:13 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All non wikipedias to 1.23wmf7
  • 19:11 logmsgbot: reedy synchronized wmf-config/
  • 18:44 logmsgbot: reedy synchronized wmf-config/
  • 18:19 logmsgbot: reedy updated /a/common to I5ae36ae21: EasyTimeline support for private wikis via img_auth
  • 18:11 mutante: deleted php-1.22wmf17 from tin per reedy
  • 18:01 logmsgbot: reedy synchronized php-1.23wmf7/includes/revisiondelete/RevisionDelete.php 'I0bd4a5fe9687c4261ca0f57e30f723e8bf2589ac'
  • 17:59 logmsgbot: reedy synchronized php-1.23wmf6/includes/revisiondelete/RevisionDelete.php 'I0bd4a5fe9687c4261ca0f57e30f723e8bf2589ac'
  • 16:58 hashar: Jenkins is back up
  • 16:50 hashar: Jenkins is busy reloading, should be back around 5:20pm UTC. Don't kill it meanwhile it is busy reading a bunch of files :-(
  • 16:27 hashar: restarting Jenkins (stuck)
  • 16:17 hashar: Jenkins web service threads are all hang busy waiting for a long request to achieve. Caused by myself :/
  • 16:10 hashar: jenkins in a bad mood for some unknown reason :(
  • 15:00 hashar: Jenkins manually configured mediawiki-core-phpunit-misc job to be runnable concurrently
  • 14:10 manybubbles: the first pass of index building for all wikisources is complete, starting the second pass
  • 13:31 manybubbles: performing a rolling restart of the elasticsearch cluster to pick up new settings
  • 10:27 mark: Depooled eqiad Squids in PyBal
  • 10:24 mark: Depooled esams Squids in PyBal
  • 06:43 Krinkle: Increased number of Jenkins executors for 'integration-slave01' from 2 to 3
  • 04:53 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
  • 02:58 Krinkle: Reloading Zuul to deploy I4e18cb2dc2a7f4
  • 02:49 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Dec 17 02:49:29 UTC 2013
  • 02:30 logmsgbot: LocalisationUpdate completed (1.23wmf7) at Tue Dec 17 02:30:08 UTC 2013
  • 02:25 manybubbles: manually setting elastic1008 to be master eligible so we have three master eligible machines in production until the puppet code that would do that properly is merged
  • 02:16 logmsgbot: LocalisationUpdate completed (1.23wmf6) at Tue Dec 17 02:16:11 UTC 2013
  • 01:57 Krinkle: Reloading Zuul to deploy I80dafe3457c65
  • 00:27 apergos: elastic1007 failed to come back up after several attempts, a soft drac reset and some more attempts. leaving it in power off state
  • 00:15 apergos: depooled elastic1007 in pybal
  • 00:07 apergos: ERROR: Timeout while waiting for server to perform requested power action. (from attempt to powercycle elastic1007)
  • 00:04 apergos: powercycled elastic1007, inaccessible via ssh or mgmt console

December 16

  • 22:55 Ryan_Lane: rebooting virt0
  • 22:51 Ryan_Lane: dist-upgrading virt0
  • 22:44 Ryan_Lane: rebooting virt1000
  • 22:34 Ryan_Lane: dist-upgrading virt1000
  • 21:50 gwicke: updated Parsoid config to use the API cluster directly for most wikis
  • 21:19 subbu: deployed parsoid 7684df12
  • 21:12 mutante: creating wikilovesearth mailing list for bug 52705
  • 19:27 ottomata: stopping puppet on cp1048 to test varnishkafka ganglia module
  • 19:05 Krinkle: Reloading Zuul to deploy I01d349bf21b20ce94
  • 18:27 logmsgbot: yurik synchronized php-1.23wmf7/extensions/ZeroRatedMobileAccess/
  • 18:24 logmsgbot: yurik synchronized php-1.23wmf6/extensions/ZeroRatedMobileAccess/
  • 18:05 manybubbles: all search indexes built for newly cirrused wikis (wikisources other than enwikisource and frwikisource and commons). populating them now. commons will take the longest. it'll take 13 hours at the current rate. past experience tells me we won't be able to keep that rate up but that this is the slow time of day to be doing it anyway.
  • 17:41 logmsgbot: manybubbles synchronized wmf-config/ 'update cirrus configuration'
  • 17:31 logmsgbot: manybubbles synchronized php-1.23wmf6/extensions/CirrusSearch/ 'update cirrus to master'
  • 17:28 logmsgbot: manybubbles synchronized php-1.23wmf7/extensions/CirrusSearch/ 'update cirrus to master'
  • 15:08 mark: Depooled all pmtpa Squids in PyBal
  • 14:51 hashar: jenkins enabled linting jobs to be runnable in parallel. Whenever several changes are made on the same repo, Jenkins will trigger a linting job per change. That will dramatically speed up the processing of changes since some jobs are now parallelized instead of serialized.
  • 14:48 hashar: zuul made gate-and-submit pipeline a dependent pipeline. Changes would thus be triggered in parallel whenever a repo has several +2 attempting to land in. That should speed up gating process. See also bug 48419 and 101839
  • 14:06 logmsgbot: springle synchronized wmf-config/db-pmtpa.php
  • 14:06 paravoid: ganglia-monitor restart on srv*/mw*; gmond bug with swapoff
  • 13:42 logmsgbot: springle synchronized wmf-config/db-pmtpa.php
  • 13:26 logmsgbot: springle synchronized wmf-config/db-pmtpa.php
  • 11:28 logmsgbot: springle synchronized wmf-config/db-pmtpa.php
  • 06:42 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool db1026 after schema changes, LB lowered during warm up'
  • 06:27 logmsgbot: tstarling synchronized docroot/secure/404.html
  • 05:35 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'If79a9443a: Add MassMessage to $wgDebugLogGroups'
  • 05:34 logmsgbot: ori updated /a/common to If79a9443a: Add MassMessage to $wgDebugLogGroups
  • 05:34 logmsgbot: ori synchronized php-1.23wmf6/extensions/MassMessage/MassMessageJob.php 'Iec240623a: Add debug logging for bug 57464'
  • 04:32 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1026 during schema changes'
  • 02:34 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Dec 16 02:34:36 UTC 2013
  • 02:19 logmsgbot: LocalisationUpdate completed (1.23wmf7) at Mon Dec 16 02:19:59 UTC 2013
  • 02:13 paravoid: salt swapoff -a; sed -i "/swap/d" /etc/fstab on all srv*, mw*
  • 02:11 logmsgbot: LocalisationUpdate completed (1.23wmf6) at Mon Dec 16 02:11:13 UTC 2013

December 15

  • 20:53 Jamesofur: insert arbcom election decryption key
  • 11:24 logmsgbot: maxsem synchronized php-1.23wmf6/includes/parser/ParserOutput.php 'Debugging done'
  • 10:08 logmsgbot: maxsem synchronized php-1.23wmf6/includes/parser/ParserOutput.php 'Debugging'
  • 02:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Dec 15 02:26:19 UTC 2013
  • 02:16 logmsgbot: LocalisationUpdate completed (1.23wmf7) at Sun Dec 15 02:16:36 UTC 2013
  • 02:09 logmsgbot: LocalisationUpdate completed (1.23wmf6) at Sun Dec 15 02:09:37 UTC 2013

December 14

  • 02:28 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Dec 14 02:28:38 UTC 2013
  • 02:19 logmsgbot: LocalisationUpdate completed (1.23wmf7) at Sat Dec 14 02:19:24 UTC 2013
  • 02:10 logmsgbot: LocalisationUpdate completed (1.23wmf6) at Sat Dec 14 02:10:34 UTC 2013

December 13

  • 19:54 spagewmf: Flow team deployed bug 58455 fix to 1.23wmf7
  • 18:33 akosiaris: install PHP5 5.3.10-1ubuntu3.9+wmf1 on mw1017 (test.wikipedia.org)
  • 15:50 paravoid: reenabling ospfv3 on eqiad-esams
  • 14:11 akosiaris: applied the new installserver role to brewster, carbon, bast4001 and hooft. This implies installation of ferm and usage of base::firewall which means default firewall policy will now be DROP. Punched necessary holes for services to work.
  • 14:06 akosiaris: just ran aptitude remove apache2-mpm-prefork apache2-utils apache2.2-bin apache2.2-common libapache2-mod-php5 on bast1001. Autoremoved texlive-* packages and timidity. All seemed to be installed because of wikimedia-task-appserver (also removed)
  • 13:36 springle: killing queries on es100[1-4]
  • 13:31 springle: reduce max_connections on es100[1-4]. try to survive sudden spike
  • 09:58 akosiaris: disabling puppet on carbon for a short time (debugging the new install-server module)
  • 02:50 springle: ongoing schema changes on slaves, indexing only, logging gerrit 85508, wb_terms gerrit 99660
  • 02:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Dec 13 02:48:41 UTC 2013
  • 02:41 logmsgbot: LocalisationUpdate completed (1.23wmf7) at Fri Dec 13 02:41:23 UTC 2013
  • 02:21 logmsgbot: LocalisationUpdate completed (1.23wmf6) at Fri Dec 13 02:21:05 UTC 2013
  • 00:25 logmsgbot: mholmquist synchronized php-1.23wmf7/extensions/MultimediaViewer/ 'Fix for another stray MultimediaViewer event, take 2.'
  • 00:25 logmsgbot: mholmquist synchronized php-1.23wmf6/extensions/MultimediaViewer/ 'Fix for another stray MultimediaViewer event, take 2.'
  • 00:20 logmsgbot: mholmquist synchronized php-1.23wmf7/extensions/MultimediaViewer/ 'Fix for another stray MultimediaViewer event'
  • 00:18 logmsgbot: mholmquist synchronized php-1.23wmf6/extensions/MultimediaViewer/ 'Fix for another stray MultimediaViewer event'
  • 00:12 logmsgbot: mwalker synchronized php-1.23wmf7/extensions/Collection/ 'Reverting to known good condition for collection extension'
  • 00:11 logmsgbot: mwalker synchronized php-1.23wmf6/extensions/Collection/ 'Reverting to known good condition for collection extension'
  • 00:04 logmsgbot: catrope synchronized php-1.23wmf7/extensions/EducationProgram 'Cherry-pick to fix ContextSource compatibility'
  • 00:03 logmsgbot: catrope updated /a/common/php-1.23wmf7 to If7c3c52ed: Update EducationProgram to wmf7 branch for cherry-picks

December 12

  • 23:55 logmsgbot: catrope finished scap: Scap for VisualEditor i18n problems
  • 23:36 logmsgbot: catrope started scap: Scap for VisualEditor i18n problems
  • 23:31 logmsgbot: catrope synchronized php-1.23wmf7/extensions/VisualEditor 'Update VE for cherry-pick'
  • 23:30 logmsgbot: catrope synchronized php-1.23wmf6/extensions/VisualEditor 'Update VE for cherry-pick'
  • 23:29 logmsgbot: catrope updated /a/common/php-1.23wmf7 to I492fe5762: Update VisualEditor to wmf7 branch for cherry-picks
  • 22:10 logmsgbot: aaron synchronized php-1.23wmf6/includes/job/JobQueueGroup.php 'de525efb4a149680f8bbf7f0d5db18489f59e4e8'
  • 21:59 logmsgbot: mflaschen synchronized multiversion/MWMultiVersion.php 'Sync routing fix that only impacts Beta'
  • 21:31 logmsgbot: reedy synchronized php-1.23wmf7/extensions/Wikibase 'Fix scribunto integration'
  • 21:30 logmsgbot: reedy synchronized php-1.23wmf6/extensions/Wikibase 'Fix scribunto integration'
  • 19:35 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: phase1 wikis to 1.23wmf7
  • 19:27 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.23wmf6
  • 19:27 logmsgbot: reedy updated /a/common to Ib597431a3: Set $wgULSNoWebfontsSelectors
  • 19:26 awight: crm updated from 0e7f22692b19eaa8deb3eae0d34ec077b7ab22d7 to 7deaa1e1a6354de545fedb0e90b85f117e7df0bd
  • 19:26 logmsgbot: reedy synchronized php-1.23wmf7/extensions/
  • 19:24 logmsgbot: reedy synchronized php-1.23wmf6/extensions/
  • 17:44 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.23wmf6 till window
  • 15:46 hashar_: started import to commons of several videos from user 80686. bug 58155
  • 15:45 mark: Restarted varnish backend on cp1055 with no config changes
  • 15:04 mark: Manually disabled storage backends main1b, main2b on cp1068 and restarted Varnish (puppet will overwrite)
  • 14:13 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'touch'
  • 14:12 logmsgbot: reedy finished scap: testwiki to 1.23wmf7, build l10n cache
  • 14:12 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'Set $wgULSNoWebfontsSelectors 100990'
  • 13:44 logmsgbot: reedy started scap: testwiki to 1.23wmf7, build l10n cache
  • 13:30 logmsgbot: reedy synchronized php-1.23wmf7 'staging'
  • 13:14 logmsgbot: reedy synchronized docroot and w
  • 13:13 logmsgbot: reedy updated /a/common to Ib2c3794dc: Fix Flow config setting for beta labs
  • 12:54 logmsgbot: reedy synchronized php-1.23wmf6/extensions/Scribunto/Scribunto.namespaces.php 'Fix ml namespace issue'
  • 12:53 logmsgbot: reedy synchronized php-1.23wmf5/extensions/Scribunto/Scribunto.namespaces.php 'Fix ml namespace issue'
  • 12:49 ori-l: Took down gdash.wm.o for some time to finish Nginx -> Apache change.
  • 11:57 springle: xtrabackup clone s5 db1005 to db1017
  • 11:09 springle: labsdb1003:3307 db instance crash, page corruption, now assertion failures. restoring from upstream db snapshot
  • 09:42 hashar: on tin, started a screened wget to download materials that will be uploaded to commons bug 58155
  • 03:25 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Dec 12 03:25:15 UTC 2013
  • 02:38 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Thu Dec 12 02:38:50 UTC 2013
  • 02:18 logmsgbot: LocalisationUpdate completed (1.23wmf6) at Thu Dec 12 02:18:04 UTC 2013
  • 00:58 logmsgbot: catrope synchronized php-1.23wmf6/extensions/VisualEditor/ 'Cherry-picks'
  • 00:12 logmsgbot: catrope synchronized php-1.23wmf6/extensions/VisualEditor/ 'Cherry-picks'
  • 00:12 logmsgbot: catrope synchronized php-1.23wmf6/includes/libs/CSSMin.php 'Fix for bug 58338 (@import URL mangling)'
  • 00:11 logmsgbot: catrope updated /a/common/php-1.23wmf6 to I70eaddd39: Update VisualEditor to wmf6 branch for cherry-picks
  • 00:04 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I62541406b: Disable module storage on enwiki'
  • 00:04 logmsgbot: ori updated /a/common to I62541406b: Disable module storage on enwiki

December 11

  • 23:54 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I47ad0a80b: Disable module storage on all but enwiki'
  • 23:53 logmsgbot: ori updated /a/common to I47ad0a80b: Disable module storage on all but enwiki
  • 23:24 logmsgbot: bsitu synchronized wmf-config/CommonSettings-labs.php 'job-delay-config'
  • 23:22 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable Flow on a few wiki pages'
  • 23:21 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings-labs.php 'Fix the labs config'
  • 23:19 logmsgbot: bsitu updated /a/common to I6d8a495d0: Repair flow config on labs
  • 22:58 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings-labs.php 'Enable Flow on a few wikis (but no pages)'
  • 22:57 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php 'Enable Flow on a few wikis (but no pages)'
  • 22:57 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable Flow on a few wikis (but no pages)'
  • 22:56 logmsgbot: bsitu updated /a/common to I1fba27a98: Enable Flow on a few wikis (but no pages)
  • 22:23 logmsgbot: bsitu synchronized php-1.23wmf6/maintenance/sql.php 'Added --wikidb param to sql.php (backport to wmf6)'
  • 22:02 subbu: deployed Parsoid abc4f426
  • 20:57 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'Id5b930a6b: Disable module storage on eswiki to assess impact'
  • 20:56 logmsgbot: ori updated /a/common to Id5b930a6b: Disable module storage on eswiki to assess impact
  • 20:56 ottomata: reenabled puppet on cp1046
  • 20:10 Krinkle: Reloading zuul to deploy I9c9399473a84
  • 19:38 K4-713: updated payments cluster to 1bb7171fbf1d3a3
  • 18:55 logmsgbot: yurik synchronized php-1.23wmf6/extensions/ZeroRatedMobileAccess/
  • 18:45 ottomata: disabling puppet on cp1046 to test change to ganglia sample interval for varnishkafka stats
  • 18:45 logmsgbot: yurik synchronized php-1.23wmf5/extensions/ZeroRatedMobileAccess/
  • 17:22 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php 'Deploy Cirrus to new wikis as secondary and let wikidata have it as a beta feature'
  • 17:21 logmsgbot: manybubbles synchronized wmf-config/CirrusSearch-common.php 'Make wmgUseCirrus override wmgUseCirrusAsAlternative for easier config editing'
  • 16:51 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'I55dc9cbdf3c777ce7e9f925bb0e5269c05b23dfa'
  • 16:44 logmsgbot: reedy updated /a/common to I42c4c5bcd: Fix Commons config for Cirrus
  • 16:32 logmsgbot: reedy synchronized php-1.23wmf5/cache/l10n/l10n_cache-ml.cdb 'Forced rebuild'
  • 16:24 logmsgbot: reedy synchronized php-1.23wmf6/cache/l10n/l10n_cache-ml.cdb 'Forced rebuild'
  • 08:45 apergos: manually downgraded nodejs on contint (gallium, lanthanum) from 0.10.22 to 0.8.2 (puppet failed to do so)
  • 06:05 LeslieCarr: silencing global collect paging (GC is having issues) for 1 hour
  • 03:17 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Dec 11 03:17:38 UTC 2013
  • 02:30 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Wed Dec 11 02:30:44 UTC 2013
  • 02:16 logmsgbot: LocalisationUpdate completed (1.23wmf6) at Wed Dec 11 02:16:17 UTC 2013
  • 00:22 logmsgbot: catrope synchronized php-1.23wmf6/extensions/VisualEditor/ 'Cherry-picks'
  • 00:09 logmsgbot: mholmquist synchronized php-1.23wmf6/extensions/MultimediaViewer/tests/qunit/lightboxinterface.test.js 'Make VE and MMV work together again'
  • 00:09 logmsgbot: mholmquist synchronized php-1.23wmf6/extensions/MultimediaViewer/resources/ext.multimediaViewer/ext.multimediaViewer.lightboxinterface.js 'Make VE and MMV work together again'
  • 00:08 logmsgbot: mholmquist synchronized php-1.23wmf6/extensions/MultimediaViewer/resources/ext.multimediaViewer/ext.multimediaViewer.js 'Make VE and MMV work together again'

December 10

  • 23:43 logmsgbot: csteipp synchronized php-1.23wmf5/extensions/CentralAuth 'Update to master'
  • 23:38 logmsgbot: csteipp synchronized php-1.23wmf6/extensions/CentralAuth 'Update to master'
  • 23:10 logmsgbot: demon synchronized wmf-config/CirrusSearch-labs.php 'No-op, fixing labs config'
  • 23:08 K4-713: updated payments cluster to 2c44e6dc531
  • 22:46 logmsgbot: bsitu finished scap: Add Flow extension but not enabled yet
  • 22:32 mwalker: issued `git reset bisect a4f97e4`in extensions collection in 1.23wmf6 to clean up after yesterdays bisection mess
  • 22:22 logmsgbot: bsitu started scap: Add Flow extension but not enabled yet
  • 22:11 K4-713: updated payments cluster to 321df2d84cc9c77292b5d
  • 21:45 logmsgbot: bsitu updated /a/common to Ib34f619e7: Enable AbuseFilter block option on Wikidata
  • 21:17 logmsgbot: reedy synchronized wmf-config/abusefilter.php
  • 20:42 logmsgbot: reedy synchronized php-1.23wmf6/extensions/Wikibase
  • 20:24 LeslieCarr: applied a fitler to inet6 sandbox subnet in eqiad
  • 20:11 logmsgbot: reedy synchronized php-1.23wmf6/includes/SkinTemplate.php 'I8edbdfe615f848963e3bea47dac99d1abd64c7f7'
  • 20:09 LeslieCarr: applying filter on sandbox subnet in eqiad
  • 19:46 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias and wikivoyages to 1.23wmf6
  • 19:44 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Revert that, CreditSource broken
  • 19:42 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.23wmf6
  • 19:39 logmsgbot: reedy updated /a/common to Ibceb9638f: beta: gwtoolset-whitelist
  • 19:08 logmsgbot: reedy synchronized php-1.23wmf6/extensions 'I53db62d469a6944cdf24dc209fa55c972f69b73f'
  • 17:38 gwicke: updated Parsoid back to 31910075 after verifying that this was not the source of the memory leak
  • 17:15 mutante: running gerrit reviewer-counts cron command manually for bug 52329
  • 17:02 gwicke: downgraded Parsoid to 0ac82a28 and rolled back config update to rule out code changes for memory leak
  • 16:48 gwicke: updated Parsoid node_modules for 0.10 to fix what looks like a memory leak
  • 16:38 paravoid: restarting all of parsoid again
  • 16:35 MaxSem: Nuking Rezabot@fawiki's watchlist, requested by operator
  • 14:01 logmsgbot: hashar synchronized php-1.23wmf6/extensions/ProofreadPage
  • 12:39 paravoid: restarting parsoid across the cluster, 100% CPU on all appservers
  • 07:33 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'explicit LB for each updateSpecialPages job'
  • 05:35 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'db1049 to LB=0 except groupLoadsByDB'
  • 04:17 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool db1049 after upgrade'
  • 03:32 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1049 for upgrade'
  • 03:15 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Dec 10 03:15:08 UTC 2013
  • 02:29 logmsgbot: LocalisationUpdate completed (1.23wmf6) at Tue Dec 10 02:29:36 UTC 2013
  • 02:15 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Tue Dec 10 02:15:52 UTC 2013
  • 00:25 gwicke: reverted Parsoid proxy change as officewiki and some other https-only wikis were broken
  • 00:14 logmsgbot: mflaschen synchronized php-1.23wmf6/extensions/Thanks/ 'Deploy Thanks bugfix to 1.23wmf6'
  • 00:04 logmsgbot: catrope synchronized php-1.23wmf6/extensions/VisualEditor/ 'Update VisualEditor with cherry-picks'

December 9

  • 23:56 logmsgbot: mwalker synchronized php-1.23wmf6/extensions/Collection/ 'Bisecting collection; found bad commit -- restoring to known bad until I figure out the correct way to pin this at the good commits or find the fix'
  • 23:54 mwalker: bad collection commit is 734de04a62a669022b3e7883301630f1e7947e78 -- now the only thing left is to figure out why!
  • 23:53 logmsgbot: mwalker synchronized php-1.23wmf6/extensions/Collection/ 'Bisecting collection (live), last bad; now on 734de04a62a669022b3e7883301630f1e7947e78'
  • 23:51 logmsgbot: mwalker synchronized php-1.23wmf6/extensions/Collection/ 'Bisecting collection (live), last bad; now on 484305e9ffb19d96155d3957240d74ff92992bb9'
  • 23:51 gwicke: updated Parsoid config to use the API service directly
  • 23:50 logmsgbot: mwalker synchronized php-1.23wmf6/extensions/Collection/ 'Bisecting collection (live), last bad; now on 93282cb9721316ee427c226b8f5c98274e79f4f0'
  • 23:47 logmsgbot: mwalker synchronized php-1.23wmf6/extensions/Collection/ 'Bisecting collection (live), last good; now on afec13ba4aef2f3f79cae0195744656833c5a1a7'
  • 23:45 logmsgbot: mwalker synchronized php-1.23wmf6/extensions/Collection/ 'Bisecting collection (live), now on 488d8daad500f4d9e9b400b6568ba170b54b0a70'
  • 23:40 logmsgbot: mwalker synchronized php-1.23wmf6/extensions/Collection/ 'Reverting collection (live) to b8a4d14eaca507022b011b979bfabe58bbc82361 from a4f97e428d32f4e6bf5adf88d628df9e49b4f075 to establish good revision'
  • 23:01 paravoid: rolling upgrade of nodejs to 0.10 on all wtp* boxes
  • 22:58 logmsgbot: csteipp synchronized php-1.23wmf5/extensions/CentralAuth 'Update CentralAuth to master for ops'
  • 22:47 LeslieCarr: moving bits-lb.esams from amslvs1 to amslvs3
  • 22:44 paravoid: including new c-ares, libv8 & nodejs 0.10 in apt
  • 21:06 gwicke: deployed Parsoid 3191007575830, see https://www.mediawiki.org/wiki/Parsoid/Deployments
  • 18:48 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php 'readd cirrus to wikidata'
  • 18:47 paravoid: powercycling cp1065, unresposinve, no output on console
  • 18:30 logmsgbot: manybubbles finished scap: Updating messages after extension deploys missed it by syncing files and directories
  • 18:06 logmsgbot: manybubbles started scap: Updating messages after extension deploys missed it by syncing files and directories
  • 17:57 logmsgbot: manybubbles synchronized wmf-config/CirrusSearch-common.php 'Finally make Cirrus a beta feature'
  • 17:56 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php 'Set mediawiki.org as primary and prepare for cirrus as a beta feature'
  • 17:49 logmsgbot: manybubbles synchronized php-1.23wmf5/extensions/CirrusSearch 'Update Cirrus to master'
  • 17:49 logmsgbot: manybubbles synchronized php-1.23wmf5/extensions/BetaFeatures 'Update BetaFeatures so Cirrus can be a beta feature'
  • 17:44 logmsgbot: manybubbles synchronized php-1.23wmf6/extensions/CirrusSearch/ 'Update Cirrus to master'
  • 17:44 logmsgbot: manybubbles synchronized php-1.23wmf6/extensions/BetaFeatures/ 'Update BetaFeatures to master so Cirrus can be a beta feature'
  • 15:54 apergos: rebooting cp1065 (kmem_alloc)
  • 15:00 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'touch'
  • 15:00 logmsgbot: hashar synchronized wmf-config/abusefilter.php '96967 (bug 57395) Add AbuseFilter rights to sysops on fiwiki'
  • 14:38 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '100372 add protection level to articles in hewiki'
  • 14:30 logmsgbot: hashar synchronized php-1.23wmf5/.git/refs/heads/wmf/1.23wmf5 'bump Special:Version for aude'
  • 14:07 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '99030 nomcom clean up'
  • 14:03 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '97458 Change favicon for angwiktionary to w icon'
  • 14:01 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '86643 skwikisource: Restrict uploads to sysops'
  • 13:23 logmsgbot: hashar synchronized docroot/bits/favicon 'updating favicons ( b0d4645..0812087 )'
  • 13:18 hashar: updating favicons by updating operations/mediawiki-config.git ( b0d4645..0812087 )
  • 09:36 apergos: restart backend varnish on cp4019, out of memory (fork failed and other nice things)
  • 09:25 apergos: restarted opendj on sanger
  • 09:12 apergos: powercycled sq80, i/o errors for every command, fs complaints on various filesystems in dmesg, smells like controller but it booted ok so...
  • 09:01 apergos: powercycled ms-be1006, unresponsive, lots of "BUG: soft lockup - CPU#0 stuck for 22s!" etc from time it stopped responding (16.00 utc Sunday)
  • 04:38 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool es1010 after upgrade, max_connections lowered during warm up'
  • 03:21 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool es1010 for upgrade'
  • 03:12 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool es1007 after upgrade, max_connections lowered during warm up'
  • 02:49 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Dec 9 02:49:20 UTC 2013
  • 02:43 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool es1007 for upgrade'
  • 02:21 logmsgbot: LocalisationUpdate completed (1.23wmf6) at Mon Dec 9 02:21:01 UTC 2013
  • 02:11 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Mon Dec 9 02:11:34 UTC 2013
  • 02:00 springle: resync labsdb dewiki.pagelinks from upstream

December 8

  • 21:28 paravoid: disabling OSPFv3 on eqiad-esams
  • 03:15 K4-713: updated payments cluster to 8dcb6f5f8f6cb6b24
  • 02:19 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Dec 8 02:19:11 UTC 2013
  • 02:08 logmsgbot: LocalisationUpdate completed (1.23wmf6) at Sun Dec 8 02:08:14 UTC 2013
  • 02:04 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Sun Dec 8 02:04:26 UTC 2013

December 7

  • 12:35 hashar: restarting Jenkins on gallium
  • 03:22 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Dec 7 03:22:21 UTC 2013
  • 02:37 logmsgbot: LocalisationUpdate completed (1.23wmf6) at Sat Dec 7 02:37:08 UTC 2013
  • 02:20 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Sat Dec 7 02:20:06 UTC 2013
  • 00:53 logmsgbot: demon synchronized wmf-config/PoolCounterSettings-eqiad.php 'Removing old cirrus config'
  • 00:53 logmsgbot: demon synchronized wmf-config/PoolCounterSettings-pmtpa.php 'Removing old cirrus config'
  • 00:13 logmsgbot: demon synchronized php-1.23wmf5/extensions/Wikibase/ 'Updating for debugging'

December 6

  • 23:12 K4-713: un-updated payments cluster back to 671411e77f18a9b... again
  • 23:10 K4-713: updated payments to fb96b8bfb1695cd725cdd
  • 22:45 mutante: mediawiki 1.22 has been released - http://dumps.wikimedia.org/mediawiki/1.22/
  • 21:44 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Fix undefined variable'
  • 20:59 logmsgbot: reedy synchronized docroot/bits/favicon/office.ico 'Ib5332726fd02874b25213dd7813b59a684197ac3'
  • 20:56 ori-l: Restarting gmetad on nickel
  • 20:24 K4-713: un-updated payments cluster back to 671411e77f18a9b
  • 20:18 K4-713: updated payments cluster to 9312d81ef0186edfc
  • 20:13 ori-l: Restarting gmetad on nickel for Ib111e53b6
  • 20:06 logmsgbot: ori synchronized wmf-config/throttle.php 'Ifbd6417a2: add throttle exception for queens public lib event'
  • 20:05 logmsgbot: ori updated /a/common to Ifbd6417a2: add throttle exception for queens public lib event
  • 19:28 logmsgbot: yurik synchronized wmf-config/flaggedrevs.php
  • 18:55 logmsgbot: mholmquist synchronized php-1.23wmf6/extensions/UploadWizard/resources/mw.UploadWizardDetails.js 'Fix UploadWizard for IE8'
  • 18:54 logmsgbot: mholmquist synchronized php-1.23wmf5/extensions/UploadWizard/resources/mw.UploadWizardDetails.js 'Fix UploadWizard for IE8'
  • 17:37 logmsgbot: reedy synchronized php-1.23wmf6/extensions/Wikibase 'I7cf7af5525f3223dbd044e7676b5c0255b45928c'
  • 17:37 logmsgbot: reedy synchronized php-1.23wmf6/extensions/DataValues 'I7cf7af5525f3223dbd044e7676b5c0255b45928c'
  • 17:34 logmsgbot: reedy synchronized php-1.23wmf5/extensions/Wikibase 'I06994427b780cd0b66dc0b0279045df7699aef1c'
  • 10:28 logmsgbot: hashar synchronized php-1.23wmf6/extensions/ProofreadPage 'ProofreadPage bug fixes 99627'
  • 10:16 logmsgbot: hashar synchronized php-1.23wmf5/extensions/ProofreadPage 'ProofreadPage bug fixes 99522'
  • 10:02 apergos: shot and restarted gmond processes on hooft
  • 04:27 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool es1001 after upgrade, max_connections lowered during warm up'
  • 03:33 logmsgbot: reedy synchronized multiversion/MWMultiVersion.php
  • 02:51 springle: set es1004 max_connections back to default (should have been reset previously, hence dberror log noise)
  • 02:46 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool es1001 for upgrade'
  • 02:29 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'direct vslow query group to LB=0 snapshot slaves'
  • 02:27 logmsgbot: tstarling synchronized wmf-config/trusted-xff.cdb
  • 02:22 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Dec 6 02:22:06 UTC 2013
  • 02:15 logmsgbot: LocalisationUpdate completed (1.23wmf6) at Fri Dec 6 02:15:21 UTC 2013
  • 02:02 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Fri Dec 6 02:02:14 UTC 2013
  • 00:27 mwalker: updating payments from 2afd91f91622abb369e05174d1dd244a2ff3adbd for amazon issues
  • 00:18 logmsgbot: maxsem synchronized php-1.23wmf6/extensions/DataValues 'touch'
  • 00:04 logmsgbot: maxsem synchronized php-1.23wmf6/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/99551'

December 5

  • 23:37 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Dec 5 23:37:18 UTC 2013
  • 23:22 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php 'Tweaked $wgJobQueueAggregator to use redis job servers and have failover'
  • 23:22 logmsgbot: aaron synchronized wmf-config/jobqueue-pmtpa.php 'Tweaked $wgJobQueueAggregator to use redis job servers and have failover'
  • 22:54 logmsgbot: LocalisationUpdate completed (1.23wmf6) at Thu Dec 5 22:54:59 UTC 2013
  • 22:21 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Thu Dec 5 22:21:52 UTC 2013
  • 21:22 logmsgbot: catrope synchronized php-1.23wmf5/extensions/VisualEditor/modules/oojs-ui/oojs-ui.js 'touch'
  • 21:19 ottomata: stopping puppet on cp1046 to troubleshoot some ganglia stuff
  • 20:58 logmsgbot: reedy synchronized docroot and w
  • 20:45 apergos: over the last while, restarted swift-object-server on ms-be100* except for 1008, left that for poking at
  • 20:44 logmsgbot: reedy updated /a/common to Ifae924950: Remofve 1.22wmf15 through 1.22wmf19
  • 20:44 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: phase1 wikis to 1.23wmf6
  • 20:39 logmsgbot: reedy synchronized php-1.23wmf6
  • 20:18 logmsgbot: reedy synchronized docroot and w
  • 20:16 logmsgbot: reedy updated /a/common to Ida4a0d980: Everything else to 1.23wmf5
  • 20:09 ori-l: increase in load coincided with sync of wmf6 to apaches and subsided on roll-back, but wmf6 was not enabled anywhere at the time of syncing
  • 20:08 ori-l: image scalers started overloading at 18:56, cause appears to be a spike of convert jobs exceeding limits & getting killed; swift-backend.log on fluoine has lots of InvalidResponseException; syslog on swift has "swift-object-se: page allocation failure".
  • 20:06 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: everything else to 1.23wmf5
  • 20:04 logmsgbot: reedy updated /a/common to Ifda85f2ce: Add/update symlinks
  • 19:41 Coren: Quick bounce of labstore1001 (kernel tweak)
  • 19:38 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.23wmf5 for now
  • 19:37 logmsgbot: reedy finished scap: testwiki to 1.23wmf6, build l10n cache and rebuild for 1.23wmf5
  • 19:14 logmsgbot: reedy started scap: testwiki to 1.23wmf6, build l10n cache and rebuild for 1.23wmf5
  • 19:08 logmsgbot: reedy updated /a/common to I18d8a12b1: repool es1002 after upgrade
  • 18:56 logmsgbot: reedy synchronized php-1.23wmf6 'Staging'
  • 17:38 logmsgbot: catrope synchronized php-1.23wmf5/resources/
  • 17:38 logmsgbot: catrope synchronized php-1.23wmf5/resources/oojs/oojs.js 'I synced you before, now start existing'
  • 17:30 logmsgbot: catrope synchronized php-1.23wmf5/extensions/MultimediaViewer 'oojs fixes'
  • 17:30 logmsgbot: catrope synchronized php-1.23wmf5/extensions/VisualEditor 'oojs fixes'
  • 17:29 logmsgbot: catrope synchronized php-1.23wmf5/resources/Resources.php 'Add oojs'
  • 17:29 logmsgbot: catrope synchronized php-1.23wmf5/resources/oojs/oojs.js 'Add oojs'
  • 16:54 ottomata: deployed varnishkafka 1.0.0-1 to cp1046,cp3011,cp4011
  • 16:07 ottomata: added varnishkafka 1.0.0-1 to apt
  • 15:43 ottomata: added librdkafka 0.8.1-1~precise1 to apt
  • 14:41 bblack: mobile esams caches restarted again on new packages w/ new jemalloc, puppet's enabled there and should be stable
  • 14:17 Coren: bouncing labstore1001 (kernel upgrade)
  • 13:28 logmsgbot: hashar synchronized php-1.23wmf5/extensions/ProofreadPage 'Update Proofreadpage 99042'
  • 13:25 bblack: cp3012 running test varnish pkg w/ jemalloc 3.4.1
  • 06:30 bblack: cp301[123] puppet freshness is me, please leave them disabled
  • 06:15 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool es1002 after upgrade, max_connections lowered during warm up'
  • 03:34 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool es1002 for upgrade'
  • 03:23 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Dec 5 03:23:22 UTC 2013
  • 03:03 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool es1003 after upgrade, max_connections lowered during warm up'
  • 02:38 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Thu Dec 5 02:38:26 UTC 2013
  • 02:22 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Thu Dec 5 02:22:00 UTC 2013
  • 01:07 awight: updated crm from daff8fa245084ff372490d1d6d85313078943fae to 0e7f22692b19eaa8deb3eae0d34ec077b7ab22d7
  • 00:58 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool es1003 for upgrade'

December 4

  • 23:09 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: no actual version changes, just rebuilding to test code changes
  • 23:06 logmsgbot: demon synchronized multiversion/ 'cdb changes for hhvm support (I73195536)'
  • 22:19 logmsgbot: manybubbles synchronized php-1.23wmf4/extensions/CirrusSearch/ 'Update CirrusSearch to speed up indexing'
  • 22:15 logmsgbot: manybubbles synchronized php-1.23wmf5/extensions/CirrusSearch/ 'Update CirrusSearch to speed up indexing'
  • 21:48 awight: updated deduper from 922b3aa6b797a2338fd409be4ffd90720a8de601 to 20f709349fb6ab987e689c5da64f392fa1e4d488
  • 21:18 gwicke: deployed Parsoid 0ac82a2
  • 19:05 logmsgbot: yurik synchronized flaggedrevs.dblist
  • 19:02 logmsgbot: yurik synchronized wmf-config
  • 18:57 yurik: yurik@tin:/a/common/php-1.23wmf5$ mwscript sql.php --wiki=metawiki extensions/FlaggedRevs/backend/schema/mysql/FlaggedRevs.sql
  • 18:32 logmsgbot: yurik synchronized wmf-config/InitialiseSettings.php
  • 18:22 logmsgbot: yurik synchronized php-1.23wmf5/extensions/ZeroRatedMobileAccess/
  • 18:15 logmsgbot: yurik synchronized php-1.23wmf4/extensions/ZeroRatedMobileAccess/
  • 16:07 ottomata: apt now contains kafka-0.8.0-1 .deb
  • 15:05 apergos: intermittent ganglia reporting from db1015, restarted gmond there, then on db1021 (aggregator)
  • 14:40 MaxSem: Nuking HerculeBot's watchlist on frwiki at operator's request, too large to modify from web
  • 13:59 paravoid: nas1001-b cf giveback
  • 13:18 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'touch'
  • 13:17 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'Allow exporting files as gettext in Translate 99070 bug 40341'
  • 10:30 apergos: running Wikibase/repo/maintenance/dumpJson.php against wikidata in screen as ariel n terbium; if it causes problems feel free to shoot it
  • 09:35 paravoid: short esams outage between 09:30-09:33 UTC due to crappy network vendor issues
  • 05:01 springle: add index on db1048 otrs.ticket_history.article_id to stop daily slave lag
  • 04:55 Tim: running sync-apache to deploy I730ec0bf8c388cada17b46497d87146f1e4ded1a
  • 02:54 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Dec 4 02:54:47 UTC 2013
  • 02:21 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Wed Dec 4 02:21:41 UTC 2013
  • 02:15 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Wed Dec 4 02:15:31 UTC 2013
  • 01:10 logmsgbot: ori synchronized php-1.23wmf5/includes/resourceloader/ResourceLoaderModule.php 'Ifa9088c11: resourceloader: Make sure hashmtime cache key is different by language'
  • 01:09 logmsgbot: ori synchronized php-1.23wmf4/includes/resourceloader/ResourceLoaderModule.php 'Ifa9088c11: resourceloader: Make sure hashmtime cache key is different by language'
  • 00:44 logmsgbot: maxsem synchronized mobilelanding.php
  • 00:43 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
  • 00:30 logmsgbot: catrope synchronized php-1.23wmf5/extensions/VisualEditor/VisualEditor.php 'Blacklist IE11'
  • 00:30 logmsgbot: catrope synchronized php-1.23wmf4/extensions/VisualEditor/VisualEditor.php 'Blacklist IE11'
  • 00:21 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Add plumbing for wmgVisualEditorInContentNamespaces'
  • 00:19 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Add wmgVisualEditorInContentNamespaces and set it to false on svwiktionary'
  • 00:14 logmsgbot: demon synchronized cirrus.dblist 'Fix test2wiki to have Cirrus as primary'

December 3

  • 22:38 mutante: graceful'ed a couple more mw*/srv* in pmtpa to make them all pickup the same config
  • 22:27 mutante: graceful'ed apaches on mw1078,mw1111,mw1176,mw1194,mw1096,mw1107 (dsh out of sync again? sigghh)
  • 22:09 mutante: sync-apache, graceful-all for zero/m redirect (gerrit 97115)
  • 21:41 Krinkle: Reloading zuul to deploy I889600fc65d8f
  • 21:32 Krinkle: Reloading zuul to deploy Iceb6b4016c
  • 21:04 mwalker: updating payments.wm.o from 7d49c5a8c63dd7c8e282b3d83b8cd9fca5e4b143 for translations and improved error handling
  • 20:49 mwalker: updating fundraising civicrm for thankyou templates (from 9c82766cf4265a9e9831f3eb618c233f4758cc68)
  • 20:47 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'If02d8506c: Enable module storage by default'
  • 20:46 logmsgbot: ori updated /a/common to If02d8506c: Enable module storage by default
  • 20:44 mutante: new mediawiki-1.22rc3 is out
  • 19:59 mutante: built and importing php-luasandbox 1.8 to apt repo
  • 19:59 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisources back to 1.23wmf5
  • 19:53 logmsgbot: reedy synchronized php-1.23wmf5/extensions/ProofreadPage/
  • 19:38 ottomata: updated webstatscollector to 0.2-1 on erbium and gadolinium
  • 19:35 logmsgbot: reedy synchronized php-1.23wmf5/extensions/Wikibase
  • 19:32 ottomata: updated webstatscollector package in apt to 0.2-1
  • 19:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource back to 1.23wmf4 as proofreadpage is broken
  • 19:25 logmsgbot: reedy synchronized php-1.23wmf5/extensions/ProofreadPage/
  • 19:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non Wikipedias to 1.23wmf5
  • 18:45 ori-l: restarting gmetad on nickel to clear dead hosts
  • 15:15 paravoid: depool/restart/pool cycle for nginxes
  • 14:15 bblack: cp3012 re-pooled
  • 14:06 bblack: rebooting cp3012.esams
  • 14:01 bblack: depooling cp3012.esams temporarily to fix various ugly issues
  • 12:49 akosiaris: uploaded new jenkins-debian-glue on apt.wikimedia.org, version 0.7.1-6-gf618f4d
  • 11:37 paravoid: stopping varnish backend on cp3012, crashing every 30s
  • 08:27 logmsgbot: nikerabbit synchronized php-1.23wmf4/extensions/Translate/
  • 08:16 logmsgbot: nikerabbit synchronized php-1.23wmf5/extensions/Translate/
  • 05:32 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool es1004 after upgrade, max_connections lowered during warm up'
  • 04:55 K4-713: updated payments cluster to 7d49c5a8c63dd7c8e2
  • 03:45 springle: stopping es1004 mysqld for precise & mariadb upgrade, plus reboot
  • 03:19 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Dec 3 03:19:58 UTC 2013
  • 02:43 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool es1004 for upgrade'
  • 02:37 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Tue Dec 3 02:37:37 UTC 2013
  • 02:17 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Tue Dec 3 02:17:50 UTC 2013
  • 01:13 cmjohnson1: rebooting sq37
  • 00:05 logmsgbot: mwalker synchronized wmf-config/CommonSettings.php
  • 00:05 logmsgbot: mwalker synchronized wmf-config/InitialiseSettings.php
  • 00:04 logmsgbot: mwalker updated /a/common to I9e2f24923: Changing banner expiration to 10 months

December 2

  • 23:13 logmsgbot: demon synchronized php-1.23wmf4/extensions/Elastica
  • 23:04 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Cirrus wikis get searchupdate (take 2)'
  • 22:33 logmsgbot: catrope synchronized php-1.23wmf5/extensions/VisualEditor 'Update VE for cherry-pick'
  • 22:33 logmsgbot: catrope synchronized php-1.23wmf4/extensions/VisualEditor 'Update VE for cherry-pick'
  • 22:30 mwalker: updating civicrm from 82af6188f5e019ab23636e78719e17c4cb838859 for new thankyou message
  • 22:20 logmsgbot: reedy updated /a/common to Idc406aa68: Created mobile portal m.wikipedia.org and zero.wikipedia.org
  • 22:19 logmsgbot: reedy synchronized wmf-config/interwiki.cdb 'Updating interwiki cache'
  • 22:06 logmsgbot: catrope synchronized visualeditor.dblist 'Enable VisualEditor on svwiktionary and sewikimedia'
  • 22:05 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Only activate VisualEditor in the User namespace on svwiktionary'
  • 22:04 logmsgbot: catrope synchronized visualeditor-default.dblist 'Enable VisualEditor by default on 102 wikis'
  • 21:33 RobH: ignore any alerts for testsearch1XXX, I just decommissioned them, but icinga hasn't updated quite yet
  • 21:02 logmsgbot: yurik synchronized docroot and w
  • 20:46 logmsgbot: maxsem synchronized php-1.23wmf5/extensions/MobileFrontend/ 'https://gerrit.wikimedia.org/r/#/c/95636/'
  • 20:45 logmsgbot: maxsem synchronized php-1.23wmf4/extensions/MobileFrontend/ 'https://gerrit.wikimedia.org/r/#/c/95636/'
  • 20:10 logmsgbot: yurik synchronized w/mobilelanding.php
  • 20:02 logmsgbot: yurik synchronized php-1.23wmf5/extensions/ZeroRatedMobileAccess/
  • 19:53 logmsgbot: yurik synchronized php-1.23wmf4/extensions/ZeroRatedMobileAccess/
  • 19:01 logmsgbot: demon synchronized php-1.23wmf4/extensions/Elastica 'Fix missing code in Elastica on 1.23wmf4 wikis'
  • 18:58 K4-713: updated fraud rules on payments
  • 18:53 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Search updates back off for Cirrus wikis :('
  • 18:35 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Dec 2 18:35:42 UTC 2013
  • 18:34 ottomata: added logster deb, installed on cp3003 for testing, will puppetize shortly
  • 18:30 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Search updates back on for Cirrus'
  • 17:47 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Mon Dec 2 17:47:29 UTC 2013
  • 17:31 bblack: cp301[12].esams - puppet temporarily disabled, custom crash handler vmod in place to try to catch an error in the next couple of hours
  • 17:31 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Search update off for cirrus wikis'
  • 17:31 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Mon Dec 2 17:31:11 UTC 2013
  • 17:30 logmsgbot: demon updated /a/common to I507a72cca: Fix submodule reference change that snuck into If5b3a27a
  • 17:21 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Cirrus on all the wikis (that had it before)'
  • 17:09 logmsgbot: demon synchronized php-1.23wmf5/extensions/CirrusSearch 'Cirrus to master'
  • 17:07 logmsgbot: demon synchronized php-1.23wmf4/extensions/CirrusSearch 'Cirrus to master'
  • 17:06 logmsgbot: demon synchronized wmf-config/CirrusSearch-common.php
  • 16:33 paravoid: rebooting cp1065, usual XFS deadlock
  • 12:35 Jeff_Green: stopped mysql on db1008 to clone a database
  • 07:05 springle: upgrade/reboot db1046 m2 slave
  • 06:53 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool pc1003 after upgrade, max_connections lowered during warm up'
  • 06:28 paravoid: fixed ganglia for misc eqiad (possibly others); see Icc5376505
  • 06:04 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool pc1003 for upgrade'
  • 02:51 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool pc1002 after upgrade, max_connections lowered during warm up'
  • 02:37 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Dec 2 02:37:48 UTC 2013
  • 02:23 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool pc1002 for upgrade'
  • 02:14 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Mon Dec 2 02:14:57 UTC 2013
  • 02:08 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Mon Dec 2 02:08:15 UTC 2013
  • 01:16 springle: restarting labsdb1002 mysqld processes with 25% smaller buffer pools. kernel OOM killer striking. needs investigation

December 1

  • 06:23 apergos: sq37 hardware errors, probably controller, rt #6418
  • 06:04 apergos: pwercycled sq37, was giving 'i/o error' to all commands
  • 02:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Dec 1 02:36:06 UTC 2013
  • 02:13 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Sun Dec 1 02:13:36 UTC 2013
  • 02:07 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Sun Dec 1 02:07:54 UTC 2013

November 30

  • 05:54 paravoid: rebooting cp1052 with the same kernel (control)
  • 02:47 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Nov 30 02:47:22 UTC 2013
  • 02:18 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Sat Nov 30 02:18:36 UTC 2013
  • 02:10 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Sat Nov 30 02:10:13 UTC 2013

November 29

  • 21:09 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'Id9c7321b8: Add a MassMessage-related user group on Meta'
  • 21:08 logmsgbot: ori updated /a/common to Id9c7321b8: Add a MassMessage-related user group on Meta
  • 20:18 paravoid: rebooting cp1065 with new kernel
  • 19:26 paravoid: "swapoff -a" on all cache_text to deal with strange kernel issue with kswapd dropping the whole page cache on memory pressure
  • 14:49 paravoid: restarted gmond on ms-fe1001/2, both were stuck 6h ago and we lost all swift eqiad's metrics for that period
  • 11:12 Reedy: Created EducationProgram tables on arwiki
  • 05:51 Tim: on cp1052 and cp1053: tweaked /proc/sys/net/core/rmem_default to see if that fixes the observed massive gmond packet loss
  • 02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Nov 29 02:07:55 UTC 2013
  • 02:02 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Fri Nov 29 02:02:25 UTC 2013
  • 02:01 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Fri Nov 29 02:01:44 UTC 2013
  • 01:22 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool pc1001 after upgrade, max_connections lowered during warm up'
  • 00:25 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool pc1001 for package upgrade'

November 28

  • 10:49 apergos: turned off logging for parsoid ( https://gerrit.wikimedia.org/r/#/c/98082/ ), old logs remain in place for folks to examine
  • 10:06 apergos: stack traces filling up parsoid nohup.out logs (sveral gigs in only a few minutes once the parsoid gets into that state), sample on wtp1010 in /var/lib/parsoid/nohup.out.errors
  • 08:34 apergos: and wtp1023
  • 08:29 apergos: /var/lib/parsoid/nohup.out on wtp 1005,11,12 was 6gb or more, causing / on these boxes to fill; moved it, restarted parsoid, removed it
  • 07:16 apergos: powercycled sq80
  • 05:41 logmsgbot: ori synchronized wmf-config/CommonSettings.php 'Icdaa4c1b5: Configure parser cache databases in db-$realm file (3/3)'
  • 05:41 logmsgbot: ori synchronized wmf-config/db-pmtpa.php 'Icdaa4c1b5: Configure parser cache databases in db-$realm file (2/3)'
  • 05:40 logmsgbot: ori synchronized wmf-config/db-eqiad.php 'Icdaa4c1b5: Configure parser cache databases in db-$realm file (1/3)'
  • 05:37 logmsgbot: ori updated /a/common to Icdaa4c1b5: Configure parser cache databases in db-$realm file
  • 03:37 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool slaves after package upgrade, (lvm snapshot boxes only, LB=0)'
  • 03:16 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool slaves for package upgrade'
  • 02:42 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Nov 28 02:42:58 UTC 2013
  • 02:29 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'slaves to full steam after package upgrade'
  • 02:15 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Thu Nov 28 02:15:36 UTC 2013
  • 02:09 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Thu Nov 28 02:09:38 UTC 2013
  • 01:17 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'warm up slaves after package upgrade'
  • 01:02 ori-l: started rsync of graphite data (~400gb) from professor.pmtpa to tungsten.eqiad
  • 00:39 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool slaves for package upgrade'

November 27

  • 19:50 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Fixes for Flow config, no-op in prod'
  • 19:49 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Fixes for Flow config, no-op in prod'
  • 18:12 paravoid: kill -9 gdb on cp3012, attached to varnish frontend
  • 11:28 ori-l: faidon switched gdash.wm.o from professor.pmtpa -> tungsten.eqiad behind misc-varnish & rebooted ssl1 in tampa
  • 11:11 apergos: ssl1 rebooted itself about 15 mins ago, no idea why
  • 10:20 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'db1019 (s3) back to full weight in the pool'
  • 10:19 logmsgbot: ariel updated /a/common to If5ebd6194: db1019 (s3) back to full weight in pool
  • 10:08 apergos: shot some old puppet processes hogging memory on db9 (from march and earlier)
  • 09:48 apergos: there was no mount /srv/pagecounts on labstore4, so rsync to /exp/pagecounts wrote to and filled /; did the mkdir and now things seem ok
  • 08:00 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'warm up db1019 (s3) aftr lvm resize'
  • 07:59 logmsgbot: ariel updated /a/common to I50354e622: warm up db1019 (s3) after lvm resize
  • 07:38 apergos: rebooting db1019 after kernel upgrade, fix for broken xfs_growfs
  • 07:02 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'depool db1019 (s3) temporarily for lvm resize'
  • 07:01 logmsgbot: ariel updated /a/common to I4372bb602: depool db1019 (s3) temporarily for lvm resize
  • 02:40 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Nov 27 02:40:52 UTC 2013
  • 02:15 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Wed Nov 27 02:15:19 UTC 2013
  • 02:08 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Wed Nov 27 02:08:28 UTC 2013
  • 00:32 springle: stopping replication on sanitarium db1054:3308 and labsdb1002:3308 while restoring dewiki to labs

November 26

  • 22:58 mutante: deploying gerrit 92925 & 91124 (apache-config), makes /entity/ URLs on wikidata 303 and removes non-existent noncom wiki
  • 21:15 mutante: praseodymium, cerium, xenon: disabled icinga notifications and scheduled 1yr downtime for host and all services per gwicke, they are test hosts and not prod.
  • 21:04 bblack: killed attached gdb on cp3012's varnishd frontend, which restarted it...
  • 20:21 logmsgbot: csteipp synchronized php-1.23wmf4/extensions/TimedMediaHandler 'bug56699'
  • 20:16 logmsgbot: csteipp synchronized php-1.23wmf5/extensions/TimedMediaHandler 'bug56699'
  • 20:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Not wmf5 day today
  • 19:33 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Revert to 1.23wmf4 on wikisources
  • 19:23 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All non wikipedias to 1.23wmf5
  • 19:20 MaxSem: Reindexing GeoData
  • 19:16 logmsgbot: reedy updated /a/common to Idb0e7956e: slaves to full steam
  • 14:04 hashar: jenkins / zuul: cherry picked a change from upstream https://review.openstack.org/#/c/43978/ and deployed it in production. 1e3adfd..1b47026 (tagged wmf-deploy-20131126). That should slightly speed up Zuul when it receives a change, previously it would issue git remote update whenever an event is received which takes a few seconds for mediawiki/core.
  • 08:00 ori-l: re-pooled ssl1003
  • 07:56 ori-l: depooling ssl1003 for quick test of puppet config
  • 06:22 apergos: pwercycled labstore1001, unreachable, nothing on mgmt console
  • 05:59 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'slaves to full steam after package upgrade'
  • 05:59 paravoid: applying workaround for Ganglia XSS https://github.com/ganglia/ganglia-web/issues/218
  • 05:32 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'warm up slaves after package upgrade'
  • 04:51 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool slaves for package upgrade'
  • 04:05 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool slaves for package upgrade'
  • 03:29 Tim: deploying varnish normalize_path (I687f6773)
  • 03:14 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Nov 26 03:14:08 UTC 2013
  • 02:25 Tim: deploying new generated redirects.conf
  • 02:24 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Tue Nov 26 02:24:12 UTC 2013
  • 02:16 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Tue Nov 26 02:16:11 UTC 2013
  • 02:03 awight: updated crm from 9979fe9a0f456499ff5be261aa41ca601a88dc13 to 82af6188f5e019ab23636e78719e17c4cb838859
  • 01:59 awight: rollback crm from 3e7f0d57023e98e6be0e1d7539b2abd04b72ad99 to 9979fe9a0f456499ff5be261aa41ca601a88dc13
  • 01:58 awight: updated crm from 9979fe9a0f456499ff5be261aa41ca601a88dc13 to 3e7f0d57023e98e6be0e1d7539b2abd04b72ad99
  • 01:57 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'slaves to full steam'
  • 01:22 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'warm up slaves after package upgrade'
  • 00:43 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool slaves for packages upgrade'

November 25

  • 23:30 logmsgbot: mwalker synchronized php-1.23wmf4/extensions/CentralNotice 'Updating CentralNotice to master (mostly for caching reasons'
  • 23:11 logmsgbot: ori synchronized php-1.23wmf4/extensions/TemplateData 'Update TemplateData to master for I0782ea669'
  • 23:11 logmsgbot: ori synchronized php-1.23wmf4/tests/phpunit/includes/ArticleTablesTest.php 'Avoid extra parsing in prepareContentForEdit() I2c34baaf8'
  • 23:11 logmsgbot: ori synchronized php-1.23wmf4/includes/WikiPage.php 'Avoid extra parsing in prepareContentForEdit() I2c34baaf8'
  • 22:22 Jamesofur: upload replacement arbcom election config
  • 21:59 ori-l: finished updating / restarting eqiad ssls
  • 21:37 ori-l: Depooling ssl1004 to restart nginx
  • 21:22 springle: restarting sanitarium mysqld processes, db1053, db1054, db1057
  • 21:06 ori-l: Restarting nginx on ssl1008
  • 20:49 ori-l: Depooling ssl1008 to test 96961
  • 20:43 ori-l: Also disabling puppet on cp400*
  • 20:41 ori-l: Disabling Puppet on ssl* in preparation for merging https://gerrit.wikimedia.org/r/#/c/96961/
  • 20:10 LeslieCarr: text-lb.esams, bits-lb.esams and more (amslvs1/3) are now load shared amongst amslvs1 and amslvs3 instead of just amslvs1
  • 18:48 logmsgbot: mholmquist synchronized php-1.23wmf5/extensions/UploadWizard/resources/ext.uploadWizard.uploadCampaign.list.css
  • 18:46 logmsgbot: mholmquist synchronized php-1.23wmf4/extensions/UploadWizard/resources/ext.uploadWizard.uploadCampaign.list.css
  • 16:20 hashar: jenkins : installed curl on lanthanum.eqiad.wmnet puppet change is 97526
  • 13:39 hashar: jenkins : restarted slave daemon on gallium.wikimedia.org
  • 13:38 hashar: jenkins : restarted slave daemon on lanthanum.eqiad.wmnet
  • 13:38 hashar: Jenkins: upgrading openjdk 6 on gallium and lanthanum
  • 07:43 springle: imported de-debianized mariadb 5.5.34 debs into rerepro on brewster
  • 04:45 logmsgbot: springle synchronized wmf-config/db-eqiad.php
  • 04:10 springle: mariadb 5.5.34 live-fire test on db1045
  • 04:09 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'warm up db1045'
  • 03:58 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1045 for upgrade'
  • 03:05 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Nov 25 03:05:19 UTC 2013
  • 02:26 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Mon Nov 25 02:26:16 UTC 2013
  • 02:14 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Mon Nov 25 02:14:09 UTC 2013
  • 01:41 logmsgbot: springle synchronized wmf-config/db-eqiad.php
  • 01:20 springle: mariadb 5.5.34 live-fire test on db1037
  • 01:15 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'warm up db1037'
  • 00:26 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool db1050. depool db1037'

November 24

  • 10:04 logmsgbot: ori synchronized wmf-config/throttle.php 'Icf844849e: (bug 57345) Raise account creation limit for cawiki GLAM event'
  • 10:03 logmsgbot: ori updated /a/common to Icf844849e: (bug 57345) Raise account creation limit for cawiki GLAM event
  • 09:58 ori-l: (...thereby fixing the global job queue length graph)
  • 09:55 ori-l: hume: ln -s /usr/local/apache/common-local /a/common (previously clobbered by Iaf19f0e8c)
  • 09:21 Nemo_bis: global job queue length graph is still dead https://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Miscellaneous%20pmtpa&h=hume.wikimedia.org&v=823574&m=Global_JobQueue_length&r=hour&z=default&jr=&js=&st=1365625056&z=large
  • 06:44 paravoid: wikitech broken with commit 716887 (Cleanup puppetmaster's apache Listen ports); workarounded manually with /etc/apache2/conf.d/ports-wikitech.conf
  • 06:27 paravoid: force-rebooting cp3012, kernel deadlocked
  • 06:02 ori-l: (cp3012) CPU wait, not wio.
  • 05:46 ori-l: cp3012: kern.log, every 2s: XFS: possible memory allocation deadlock in kmem_alloc
  • 05:43 ori-l: cp3012: any attempt to access /proc/17722 immediately hangs ssh
  • 05:31 ori-l: cp3012 wio through the roof; process table filled with '/bin/ps axwwo stat uid pid ppid vsz rss pcpu etime comm args', uid icinga
  • 02:50 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Nov 24 02:50:44 UTC 2013
  • 02:20 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Sun Nov 24 02:20:50 UTC 2013
  • 02:11 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Sun Nov 24 02:11:21 UTC 2013

November 23

  • 14:04 springle: testing 5.5.34 mariadb.org debs on db60
  • 10:13 apergos: ...on zirconium
  • 10:13 apergos: stopped etherpadlite, removed all but the last 5000 lines of its log file, (root running out of space), restarted
  • 03:12 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Nov 23 03:12:24 UTC 2013
  • 02:26 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Sat Nov 23 02:26:18 UTC 2013
  • 02:14 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Sat Nov 23 02:14:03 UTC 2013
  • 00:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable OAuth on fishbowl wikis'
  • 00:48 logmsgbot: reedy synchronized php-1.23wmf5/extensions/Wikibase/client/ 'Fix InfoAction fatal'
  • 00:48 logmsgbot: reedy synchronized php-1.23wmf5/extensions/MobileFrontend/ 'Fix history fatal'
  • 00:47 logmsgbot: reedy synchronized php-1.23wmf4/extensions/MobileFrontend/ 'Fix history fatal'
  • 00:34 aschulz|laptop: Manually fixed img_metadata for File:Zentralbibliothek_Zürich_-_Heinrich_Bullingers_Westerhemd_-_000012135.tif by forcing /bin/identify usage
  • 00:22 logmsgbot: ori synchronized php-1.23wmf4/extensions/PagedTiffHandler 'Update PagedTiffHandler to master for I52fe2ec25 / bug 57359'
  • 00:18 logmsgbot: ori synchronized php-1.23wmf5/extensions/PagedTiffHandler 'Update PagedTiffHandler to master for I52fe2ec25 / bug 57359'

November 22

  • 21:35 logmsgbot: ori synchronized wmf-config/CommonSettings.php 'I615e2b210: correct schema ID of Schema:Echo'
  • 21:34 logmsgbot: ori updated /a/common to I615e2b210: Schema:Echo has outdated revision id
  • 21:09 mutante: mediawiki 1.22.0rc3 is out http://dumps.wikimedia.org/mediawiki/1.22/
  • 20:04 Jeff_Green: dist-upgrade aluminium and reboot
  • 19:56 mutante: starting apache on alumunium
  • 19:54 mutante: killed all processes by user sahar on aluminium
  • 19:27 logmsgbot: reedy finished scap: Rebuild localisation cache to update ce
  • 19:10 logmsgbot: reedy started scap: Rebuild localisation cache to update ce
  • 17:21 logmsgbot: reedy synchronized php-1.23wmf5/extensions/EducationProgram/includes/Events/EditEventCreator.php
  • 17:19 logmsgbot: reedy synchronized php-1.23wmf4/extensions/EducationProgram/includes/Events/EditEventCreator.php
  • 17:10 logmsgbot: reedy synchronized php-1.23wmf5/languages/messages/MessagesCe.php 'Update ce language special page aliases'
  • 17:09 logmsgbot: reedy synchronized php-1.23wmf4/languages/messages/MessagesCe.php 'Update ce language special page aliases'
  • 12:59 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Ic36cd656233dfc959c92a33bade794cc5c1e1bd2'
  • 11:21 akosiaris: neon booted and seems ok. running updatedb manually
  • 10:53 akosiaris: rebooting neon for fsck
  • 09:19 apergos: snmptt stuck waiting for a submit_check_result that had exited (or died), restarted it to clear the snmptt spool queue
  • 08:14 apergos: rebooting neon again, found four filenames with deleted inode but side effect is back in r/o for /
  • 07:06 apergos: rebooting neon, no underlying errors found except for the initial ext3 'deleted inode reference' which caused / to be remounted r/o
  • 06:53 apergos: neon back to read-only root filesystem, investigating
  • 03:23 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Nov 22 03:23:33 UTC 2013
  • 02:35 logmsgbot: LocalisationUpdate completed (1.23wmf5) at Fri Nov 22 02:35:51 UTC 2013
  • 02:16 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Fri Nov 22 02:16:36 UTC 2013
  • 01:41 logmsgbot: csteipp synchronized wmf-config/InitialiseSettings.php
  • 01:40 logmsgbot: csteipp synchronized wmf-config/CommonSettings.php
  • 01:37 logmsgbot: csteipp finished scap: Really send the new messages this time..
  • 01:16 logmsgbot: csteipp started scap: Really send the new messages this time..
  • 00:56 logmsgbot: ori synchronized php-1.23wmf4/resources/startup.js 'touch'
  • 00:50 logmsgbot: csteipp finished scap: Updating WikimediaMessages to master for OAuth message
  • 00:50 logmsgbot: ori synchronized php-1.23wmf4/extensions/MobileFrontend/javascripts/loggingSchemas/MobileWebInfobox.js 'Cherry-pick I3efc1fa64'
  • 00:46 logmsgbot: ori synchronized php-1.23wmf4/resources/startup.js 'touch'
  • 00:36 logmsgbot: csteipp started scap: Updating WikimediaMessages to master for OAuth message
  • 00:35 logmsgbot: ori synchronized php-1.23wmf4/extensions/MobileFrontend/javascripts/loggingSchemas/MobileWebInfobox.js 'Cherry-pick I3efc1fa64'
  • 00:09 logmsgbot: demon synchronized wmf-config/PoolCounterSettings-eqiad.php 'New pool counter settings for Cirrus'
  • 00:08 logmsgbot: demon synchronized wmf-config/PoolCounterSettings-pmtpa.php 'New pool counter settings for Cirrus'
  • 00:07 Reedy: Built texvc for 1.23wmf5

November 21

  • 23:33 ori-l: Purging tifs older than one hour on app servers
  • 23:06 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable BetaFeatures everywher'
  • 23:03 logmsgbot: reedy updated /a/common to I3ac8621d9: Removed ZERO namespace protection in META
  • 23:02 logmsgbot: reedy synchronized wmf-config 'last few config updates'
  • 22:32 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: phase1 wikis to 1.23wmf5
  • 22:31 logmsgbot: reedy synchronized php-1.23wmf5/extensions/MobileFrontend/
  • 22:21 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.23wmf4
  • 21:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'rv'
  • 21:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wiki -'
  • 21:26 logmsgbot: reedy updated /a/common to I4360e8a34: Enable BetaFeatures everywhere except Wikipedias
  • 21:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable BetaFeatures on all non wikipedias'
  • 21:16 logmsgbot: reedy synchronized php-1.23wmf5/extensions/Wikibase
  • 20:59 logmsgbot: reedy updated /a/common to I6433f2b0d: testwiki to 1.23wmf5
  • 20:57 cmjohnson1: sq48 powering down for decom
  • 20:47 cmjohnson: sq44 powering down for decom
  • 20:24 Reedy: mw117 is also segfaulting quite a bit
  • 19:56 paravoid: depooling mw1183
  • 19:56 Reedy: mw1183/10.64.32.53 is having masses of segfaults
  • 19:52 Reedy: Created BetaFeatures extension tables on all wikis
  • 19:46 logmsgbot: ori synchronized docroot/noc 'I7ea8302f2: noc.wm.o: add images referenced by stylesheets.'
  • 19:45 logmsgbot: ori updated /a/common to I7ea8302f2: noc.wm.o: add images referenced by stylesheets.
  • 19:43 logmsgbot: reedy Finished syncing Wikimedia installation... : testwiki to 1.23wmf5 and build l10n cache
  • 19:33 logmsgbot: ori synchronized docroot/noc 'Ia451d47bd: noc.wm.o: use local CSS files rather than point to svn.wm.o'
  • 19:32 logmsgbot: ori updated /a/common to Ia451d47bd: noc.wm.o: use local CSS files rather than point to svn.wm.o
  • 19:27 logmsgbot: reedy Started syncing Wikimedia installation... : testwiki to 1.23wmf5 and build l10n cache
  • 19:24 logmsgbot: reedy synchronized docroot and w
  • 19:21 logmsgbot: reedy synchronized php-1.23wmf5 'Staging code'
  • 19:02 paravoid: restored service for svn.wm.org
  • 18:48 cmjohnson: shutting down vanadium to replace failed hard drive
  • 17:19 ^d: reloaded zuul to pick up Ibc354db7
  • 15:34 springle: bounced pc1001 mysqld. massive spike of writes exhausted innodb txn log slots and wouldn't be killed
  • 13:55 hashar: restarted Zuul, had some jobs stuck in its queue (96762
  • 09:16 paravoid: reenable ospfv3 between esams-eqiad
  • 08:44 ori-l: Synced wmf-config/InitialiseSettings.php "I5e8317db6: Enable recursion-guard log bucket to investigate bug 54193"
  • 08:44 ori-l: with neon down, scap / syncs not logged to irc
  • 08:04 springle: rebooted neon, ext3 root fs read only
  • 07:52 akosiaris: updated all remaining references to sockpuppet and stafford as puppetmaster on wikitech
  • 04:58 dr0ptp4kt: Wikipedia Zero BDD script run
  • 03:20 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Nov 21 03:20:28 UTC 2013
  • 02:32 logmsgbot: LocalisationUpdate completed (1.23wmf3) at Thu Nov 21 02:32:33 UTC 2013
  • 02:17 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Thu Nov 21 02:17:19 UTC 2013
  • 01:39 springle: enwiki revision table schema change test run db60
  • 01:24 springle: depool db60, db54, db34, db31, db65, db47, db37 for decom and/or shipping. promote db68, db74, db72, db71, db69 to pmtpa shard masters
  • 01:18 logmsgbot: springle synchronized wmf-config/db-pmtpa.php
  • 01:08 awight: updated crm from 326854db84becacf808add24a75ca27d1358c5bc to 9979fe9a0f456499ff5be261aa41ca601a88dc13
  • 00:29 AaronSchulz: Resynced terbium /usr/local/apache/common-local which was totally empty
  • 00:18 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'If961f9365: add dewiki help page for Echo'
  • 00:17 logmsgbot: ori updated /a/common to If961f9365: add dewiki help page for Echo
  • 00:09 ori-l: terbium: symlinked /a/common to /usr/local/apache/common. We should really get rid of the distinction.

November 20

  • 23:40 logmsgbot: yurik synchronized php-1.23wmf4/extensions/ZeroRatedMobileAccess/
  • 23:37 ori-l: puppet.log on terbium: could not set file on ensure: No such file or directory - /usr/local/apache/common/php/extensions/FlaggedRevs/maintenance/wikimedia-periodic-update.sh.puppettmp_8141 at /etc/puppet/manifests/misc/maintenance.pp:155
  • 23:30 logmsgbot: yurik synchronized php-1.23wmf3/extensions/ZeroRatedMobileAccess/
  • 23:27 ori-l: restarting rsyslog on application servers for I31c76fdde.
  • 22:29 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'touch'
  • 22:28 logmsgbot: bsitu synchronized echowikis.dblist 'Enable Echo and Thanks on dewiki and itwiki'
  • 22:23 logmsgbot: bsitu synchronized echowikis.dblist 'Enable Echo and Thanks on dewiki and itwiki'
  • 22:23 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Update email footer and help page'
  • 22:19 logmsgbot: bsitu updated /a/common to Id9e3b9a03: correct default Echo help page
  • 21:47 logmsgbot: faidon synchronized wmf-config/CommonSettings.php 'revert CentralNotice CrossWiki Hiding'
  • 21:46 logmsgbot: faidon updated /a/common to I866838bc2: Revert "Enable CentralNotice CrossWiki Hiding"
  • 21:34 mutante: ran checksetup.pl on bugzilla, deploy gerrit 96479, replaces internal errors on WeeklyReport with proper error pages
  • 21:12 gwicke: deployed Parsoid 20c6afe
  • 19:08 logmsgbot: yurik synchronized docroot/bits/WikipediaMobileFirefoxOS
  • 18:07 akosiaris: changed default ferm policy to DROP. antimony/gallium are the first two servers to get it.
  • 17:51 ^demon|sick: gerrit up, forcing replication of all repos
  • 17:50 ^demon|sick: restarting gerrit since replication plugin won't reload
  • 17:27 ^demon|sick: gerrit: disabled replication plugin fix, pending firewall updates. everything's timing out and flooding logs with errors.
  • 13:59 hashar: jenkins: added slave integration-slave01 with label hasNpm. That is a slave running in labs ( integration-slave01.pmtpa.wmflabs )
  • 12:31 apergos: end of an era: shut down ms7 webserver and opening ticket for decom
  • 10:14 hashar: Jenkins: disabled gearman plugin, Zuul is no more a Gearman server.
  • 10:12 hashar: restarted again stalled Jenkins
  • 10:05 hashar: restarted Zuul with non gearman version.
  • 10:02 hashar: reverting Zuul to last known version: wmf-deploy-20131023 + 6241272...1e3adfd wmf-deploy-20131023 -> master (forced update)
  • 09:58 hashar: shutting down Zuul and reverting upgrade :-(
  • 09:46 hashar: restarting Jenkins to update gearman plugin
  • 09:38 hashar: Zuul upgraded with Gearman support to trigger build in Jenkins. Will monitor for the rest of the morning.
  • 09:34 hashar: enabling Gearman in Jenkins, making it register with Zuul new version
  • 09:33 hashar: restarted Zuul with Gearman version
  • 09:27 hashar: removing all old versions of zuul: rm -fR /usr/local/lib/python2.7/dist-packages/zuul*
  • 09:20 hashar: gallium temp install of python-pip to be able to ugprade Zuul
  • 09:15 hashar: Zuul: bumped source code in integration/zuul.git to Gearman based version: 1e3adfd...6241272 labs -> master
  • 09:05 hashar: stopping Zuul on gallium for upgrading purposes.
  • 07:51 apergos: rebooting sq37, we had /dev//sdc gone and reappeared as /dev/sdc killing back-end squid
  • 07:40 logmsgbot: ori synchronized php-1.23wmf3/extensions/WikimediaEvents/modules/ext.wikimediaEvents.moduleStorage.js 'Re-syncing I51d2d6495'
  • 07:38 logmsgbot: ori synchronized php-1.23wmf4/extensions/WikimediaEvents/modules/ext.wikimediaEvents.moduleStorage.js 'Re-syncing I51d2d6495'
  • 06:47 apergos: removed on mw* hosts from /tmp all *png/map/err/tif older than an hour, as some tmpfs were full
  • 05:55 ori-l: /tmp on Apaches filling up with math .pngs; moving some of the oldest away as a stopgap
  • 05:04 logmsgbot: springle synchronized wmf-config/db-pmtpa.php
  • 03:17 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Nov 20 03:17:15 UTC 2013
  • 02:31 logmsgbot: LocalisationUpdate completed (1.23wmf3) at Wed Nov 20 02:31:26 UTC 2013
  • 02:15 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Wed Nov 20 02:15:34 UTC 2013
  • 01:38 paravoid: rebooting ms-be1001, i/o stuck kernel bug
  • 01:37 logmsgbot: ori synchronized php-1.23wmf4/extensions/WikimediaEvents 'Update WikimediaEvents to I5b8cfe592'
  • 01:35 logmsgbot: ori synchronized php-1.23wmf3/resources/startup.js 'touch'
  • 01:33 logmsgbot: ori synchronized php-1.23wmf3/extensions/WikimediaEvents 'Update WikimediaEvents to I5b8cfe592'
  • 00:43 logmsgbot: csteipp synchronized php-1.23wmf4/extensions/OAuth 'update OAuth to master for last blocker fix'
  • 00:33 logmsgbot: ori synchronized php-1.23wmf3/resources/startup.js 'touch'
  • 00:28 logmsgbot: ori synchronized php-1.23wmf3/extensions/MobileFrontend 'Updating MobileFrontend for I3efc1fa64'

November 19

  • 23:58 ori-l: Restarting EventLogging jobs on vanadium
  • 23:54 logmsgbot: ori synchronized php-1.23wmf3/resources/startup.js 'touch'
  • 23:14 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'Iee7332c97: Disable infobox experiment'
  • 23:13 logmsgbot: ori updated /a/common to Iee7332c97: Disable infobox experiment
  • 22:53 logmsgbot: reedy synchronized php-1.23wmf4/extensions/AbuseFilter/ 'bug 57268'
  • 21:59 logmsgbot: mholmquist synchronized php-1.23wmf3/extensions/VectorBeta/ 'Sync VectorBeta code to wikis that will never use it'
  • 21:45 logmsgbot: mholmquist synchronized wmf-config/extension-list 'Add VectorBeta to extension-list'
  • 21:43 logmsgbot: mholmquist updated /a/common to Ib532e5558: Add VectorBeta to extension-list
  • 21:41 logmsgbot: mholmquist Finished syncing Wikimedia installation... : Adding VectorBeta
  • 21:21 logmsgbot: mholmquist synchronized php-1.23wmf4/extensions/VectorBeta/
  • 21:17 logmsgbot: mholmquist Started syncing Wikimedia installation... : Adding VectorBeta
  • 21:10 logmsgbot: mholmquist synchronized wmf-config/InitialiseSettings.php 'Set wmgUseVectorBeta = true for phase0 wikis'
  • 21:09 logmsgbot: mholmquist synchronized wmf-config/CommonSettings.php 'Add wmgUseVectorBeta'
  • 21:08 logmsgbot: mholmquist updated /a/common to Ic3a64b434: Enable VectorBeta on group0 wikis
  • 20:55 logmsgbot: reedy synchronized wmf-config/
  • 20:04 logmsgbot: ori synchronized php-1.23wmf4/extensions/WikimediaEvents 'Update WikimediaEvents to d4f2a43'
  • 20:04 logmsgbot: ori synchronized php-1.23wmf3/extensions/WikimediaEvents 'Update WikimediaEvents to d4f2a43'
  • 20:01 logmsgbot: ori synchronized php-1.23wmf3/extensions/WikimediaEvents 'Update WikimediaEvents to d4f2a43'
  • 20:00 logmsgbot: ori synchronized php-1.23wmf4/extensions/WikimediaEvents 'Update WikimediaEvents to d4f2a43'
  • 19:49 logmsgbot: reedy synchronized php-1.23wmf3/extensions/MassMessage 'Update to master'
  • 19:38 logmsgbot: ori synchronized php-1.23wmf4/extensions/WikimediaEvents 'Update WikimediaEvents to 4aa9c629e5: Controlled experiment to assess performance of module storage'
  • 19:37 logmsgbot: ori synchronized php-1.23wmf3/extensions/WikimediaEvents 'Update WikimediaEvents to 4aa9c629e5: Controlled experiment to assess performance of module storage'
  • 19:22 logmsgbot: reedy synchronized wmf-config/ 'Enable MassMessage Ib29cb042e739e26fc5ef56f41317454ea690d0cb'
  • 19:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: https://gerrit.wikimedia.org/r/96297
  • 19:07 logmsgbot: reedy updated /a/common to I9b48be4c0: try to reduce "no working slave" notice flood in pmtpa dberror.log (attempt 2)
  • 18:53 Nemo_bis: but now ok
  • 18:48 Nemo_bis: kaulen overloaded at 100 % CPU since 5 min ago, bugzilla almost unusable
  • 18:32 logmsgbot: ori synchronized php-1.23wmf4/resources/mediawiki/mediawiki.js 'Id2835eca4: Enable module storage for 0.05% of visitors w/storage-capable browsers'
  • 18:31 logmsgbot: ori synchronized php-1.23wmf3/resources/mediawiki/mediawiki.js 'Id2835eca4: Enable module storage for 0.05% of visitors w/storage-capable browsers'
  • 17:35 LeslieCarr: added palladium as a salt master for common-infrastructure4 firewall
  • 17:34 LeslieCarr: CV71028 flapped again
  • 17:32 LeslieCarr: FPL link CV71028 flapped 20 minutes ago
  • 11:56 apergos: powercycled ms-be1002, couldn't get in (lots of BUG: soft lockup - CPU#18 stuck for 22s etc on console)
  • 11:25 akosiaris: remove puppetmaster and apache packages from sockpuppet. It no longer is a puppetmaster
  • 11:18 akosiaris: remove puppetmaster and apache packages from stafford. It no longer is a puppetmaster
  • 05:00 Tim: updating DNS to I3adffd88
  • 02:50 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Nov 19 02:50:34 UTC 2013
  • 02:19 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Tue Nov 19 02:19:11 UTC 2013
  • 02:12 logmsgbot: springle synchronized wmf-config/db-pmtpa.php
  • 02:10 logmsgbot: LocalisationUpdate completed (1.23wmf3) at Tue Nov 19 02:10:34 UTC 2013
  • 02:06 logmsgbot: springle synchronized wmf-config/db-pmtpa.php

November 18

  • 21:04 gwicke: deployed Parsoid fd3d6dc
  • 20:39 RobH: disabled a bunch of rt users who no longer login and dont work here anymore (so if you do work here and i messed up, opps ;)
  • 19:56 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Disabling Cirrus on all wikis but test2wiki'
  • 19:36 mwalker: updating civi from 326854db84becacf808add24a75ca27d1358c5bc for new thank you templates
  • 19:31 mutante: pybal - eqiad/search: disabling testsearch100[13], adding elastic10[12]
  • 19:23 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'I0921b9e853fdacd554de40f138644d6187ae7efe'
  • 17:36 logmsgbot: demon synchronized wmf-config/CirrusSearch-common.php 'If920e0e9'
  • 17:31 logmsgbot: demon synchronized wmf-config/CirrusSearch-common.php 'I593f62a5'
  • 17:30 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'I593f62a5'
  • 17:25 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'I114677d0'
  • 17:24 logmsgbot: demon synchronized wmf-config/CirrusSearch-common.php 'I114677d0'
  • 17:23 logmsgbot: demon updated /a/common to I3b5eac7fb: depool db74 for move to S6
  • 16:06 hashar: gallium: running puppet to apply ferm firewall configuration from 95162
  • 10:00 springle: xtrabackup db50 to db74
  • 09:46 logmsgbot: springle synchronized wmf-config/db-pmtpa.php 'depool db74 for move to S6'
  • 06:28 apergos: powercycled ms-be9, it was unresponsive; doing xfs_repair on /dev/sde1 after 'corruption detected' messages on reboot
  • 03:20 paravoid: adding mw1045 mw1070 mw1085 mw1165 back to the apache pool; no evidence (RT tickets, pybal comments) as why they are not pooled
  • 02:49 springle: for index cardinality on nullable fields, testing innodb_stats_method=nulls_ignored on db1049, db1002, db1003, db1004, db1026, db1040, db1041 + reanalyzing tag_summary and change_tag
  • 02:38 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Nov 18 02:37:59 UTC 2013
  • 02:36 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'correct s3 groupLoadsBySection key'
  • 02:22 logmsgbot: springle synchronized wmf-config/db-pmtpa.php 'pmtpa replication reconfig for decomissioning'
  • 02:15 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Mon Nov 18 02:15:51 UTC 2013
  • 02:07 logmsgbot: LocalisationUpdate completed (1.23wmf3) at Mon Nov 18 02:07:57 UTC 2013

November 17

  • 20.00: the bits crisis is over thanks to paravoid
  • 19:12 Nemo_bis: 20 % packet loss from Toolserver to bits-lb.esams.wikimedia.org
  • 19:11 Nemo_bis: The 5xx errors mega-spikes every ten minutes are back (since 18 UTC)
  • 02:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Nov 17 02:08:53 UTC 2013
  • 02:03 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Sun Nov 17 02:02:54 UTC 2013
  • 02:02 logmsgbot: LocalisationUpdate completed (1.23wmf3) at Sun Nov 17 02:02:01 UTC 2013
  • 01:35 Reedy: Killed updateSpecialPages and related processes on terbium
  • 01:18 MaxSem: Killed a few long queries on db1007
  • 01:08 MaxSem: db1007 is having tough times due to special page updates

November 16

  • 02:33 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Nov 16 02:33:35 UTC 2013
  • 02:12 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Sat Nov 16 02:12:51 UTC 2013
  • 02:07 logmsgbot: LocalisationUpdate completed (1.23wmf3) at Sat Nov 16 02:07:15 UTC 2013

November 15

  • 23:13 logmsgbot: ori synchronized wmf-config/CommonSettings.php 'Ia9e64253d: Configure tungsten as $wgUDPProfilerHost'
  • 23:11 logmsgbot: ori updated /a/common to Ia9e64253d: Configure tungsten as wgUDPProfilerHost
  • 21:05 mutante: install unzip on tin (needs puppetization)
  • 19:06 awight: crm updated from 8d9240fea273faf434881a16d894ef0ad6c616c0 to 326854db84becacf808add24a75ca27d1358c5bc
  • 18:30 logmsgbot: ori updated /a/common to I132d97598: Beta Labs: enable UDP profiler
  • 17:58 mutante: uploaded mw1.22.0rc2 from hexmode http://dumps.wikimedia.org/mediawiki/1.22/
  • 15:02 akosiaris: update cr{1,2}-eqiad ACLs with new puppetmaster frontend address
  • 13:55 akosiaris: switched *eqiad.wmnet to using new puppet infra. triggering puppet run
  • 12:35 hashar: github: added zeljkofilipin to the new 'qa' team. Granted rights on wikimedia/mediawiki-selenium and wikimedia/qa-browsertests
  • 12:33 akosiaris: disabled puppet on *eqiad.wmnet for change
  • 12:32 akosiaris: updated sockpuppet term on cr1-sdtpa , cr2-tmtpa to include palladium
  • 09:45 hashar: restarting zuul, leaked file descriptors pointing to non existent git pack files :/
  • 09:15 akosiaris: disabled commiting in sockpuppet private for evah (hopefully) through a git hook
  • 09:07 akosiaris2: disabled puppet an all *.pmtpa.wmnet servers to facilitate auditing the change to new puppetmaster infra
  • 07:52 apergos: sq44 won't come back up, hung after "Remote Access Controller detected", rt #6312
  • 06:48 Tim: also fixed mw1141, mw1193
  • 06:42 Tim: fixing broken cgroups on mw1038, mw1044, mw1091
  • 05:15 logmsgbot: ori updated /a/common to I67a6d49d4: Beta Labs: set deployment-fluoride as $wgUDPProfilerHost
  • 03:20 Tim: on mc1001-1016: set timeout=0 in the redis soft state, as for rdb*
  • 03:15 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Nov 15 03:15:35 UTC 2013
  • 03:14 Tim: on rdb1001-1004, set timeout=0 in the soft state, to match I8c9b13c1
  • 02:34 logmsgbot: LocalisationUpdate completed (1.23wmf4) at Fri Nov 15 02:34:41 UTC 2013
  • 02:15 logmsgbot: LocalisationUpdate completed (1.23wmf3) at Fri Nov 15 02:15:18 UTC 2013
  • 00:39 cmjohnson1: dns update
  • 00:29 mutante: schedule 1yr downtime for unused host 'gurvin' and all its services (RT #6135) and shutting it down

November 14

  • 22:05 Nemo_bis: hashar> !log Gerrit: added marktraceur to group https://gerrit.wikimedia.org/r/#/admin/groups/539,members stream-events
  • 21:59 hashar: Gerrit: added marktraceur to group stream-events
  • 21:46 logmsgbot: ori synchronized docroot/bits/static-stable 'Correct 'static-stable' symlinks on bits'
  • 21:45 logmsgbot: ori synchronized docroot/bits/static-current 'Correct 'static-current' symlinks on bits'
  • 21:43 logmsgbot: ori updated /a/common to Id1a2bb2c8: Revert "Fix how 'current' branch is determined in updateBitsBranchPointers"
  • 21:06 hashar: vvvv bug 57064
  • 21:05 hashar: jenkins: migrated some phpcs job to use code sniffer style git-deployed on slaves in /srv/deployment/integration/mediawiki-tools-codesniffer/MediaWiki
  • 21:02 hashar: jenkins: migrated some phpcs job to use code sniffer style git-deployed on slaves in /srv/deployment/integration/mediawiki-tools-codesniffer/MediaWiki
  • 20:49 logmsgbot: reedy synchronized php-1.23wmf4/includes/
  • 20:48 mark: Raised session_max on eqiad bits caches from 200k to 400k
  • 20:45 logmsgbot: ori synchronized php-1.23wmf3/resources/startup.js 'touch'
  • 20:42 logmsgbot: ori synchronized php-1.23wmf3/extensions/UniversalLanguageSelector 'Revert "Update UniversalLanguageSelector to b2f9e4211efc"'
  • 20:26 logmsgbot: ori synchronized php-1.23wmf3/extensions/UniversalLanguageSelector 'Update UniversalLanguageSelector to b2f9e4211efc'
  • 20:19 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Phase 1 wikis to 1.23wmf4
  • 20:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.23wmf3
  • 20:12 logmsgbot: reedy updated /a/common to I3034b1e4e: Add/update symlinks
  • 20:05 logmsgbot: reedy Finished syncing Wikimedia installation... : testwiki to 1.23wmf4 and build l10n cache
  • 19:34 logmsgbot: reedy Started syncing Wikimedia installation... : testwiki to 1.23wmf4 and build l10n cache
  • 19:33 mark: Changed IP addresses of upload-lb.eqiad.wikimedia.org
  • 19:24 RobH: 18:50 not 13:30 durrrrr
  • 19:19 RobH: may have spoken too soon about ending, we shall see
  • 19:18 RobH: logging for record: outage started at approx 13:50 GMT, seems to have ended well, now-ish
  • 19:00 paravoid: pt-kill on s7 to kill LogPager queries
  • 18:40 cmjohnson1: dns update
  • 18:38 logmsgbot: reedy updated /a/common to I076d1eddf: CirrusSearch secondary for itwiki and plwiktionary
  • 18:37 logmsgbot: reedy synchronized docroot and w
  • 18:36 logmsgbot: reedy synchronized php-1.23wmf4/
  • 17:56 logmsgbot: reedy synchronized php-1.23wmf3/extensions/Wikibase
  • 17:22 manybubbles: rebuilding all CirrusSearch indexes in place to get config updates
  • 17:19 hashar: Jenkins: updated code sniffer style from 574f68d to 0bebf0f7b
  • 17:18 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php 'plwiktionary and itwiki get cirrus as secondary'
  • 17:14 logmsgbot: manybubbles synchronized php-1.23wmf3/extensions/CirrusSearch/ 'Update CirrusSearch to master'
  • 17:13 logmsgbot: manybubbles synchronized php-1.23wmf2/extensions/CirrusSearch/ 'Update CirrusSearch to master'
  • 16:45 cmjohnson1: powering down analytics1009, analytics1011-13, analtyics1021, analytics1023 to relocate to rack A2
  • 16:06 hashar: Gerrit/Git resetting mediawiki/extensions/Scribunto REL1_21 branch from fd1fbb4 to previous b5015a2. hexmode cherry-picked a bunch of changes from another branch/master instead of submitting a merge commit.
  • 15:58 apergos: preallocated cache files for amssq61 and 62 with ext4, this requires removing logbuf option in fstab
  • 13:27 logmsgbot: faidon synchronized wmf-config/filebackend.php 'switch multiMaster to swift-eqiad'
  • 13:26 logmsgbot: faidon updated /a/common to I898bfd6f1: filebackend: switch multiMaster to swift-eqiad
  • 13:09 apergos: preallocated cache file on amssq: 51, 52, 54, 56 59, 60 in addition to previously logged ones
  • 13:02 apergos: bad news: Nov 14 11:17:43 amssq54 kernel: [246380.560873] XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250) and this host has the preallocated cache file
  • 09:36 ori-l: Manually replaced jquery.ba-bbq.js on nickel:/srv/org/wikimedia/ganglia-web-latest/js with <https://github.com/cowboy/jquery-bbq/pull/42>. Upstream maintainer unresponsive.
  • 07:46 apergos: powercycled sq80, accessible but responded 'bus error' or 'i/o error' to all commands
  • 07:33 apergos: powercycled mw1210, unreachable via mgmt, no messages on console
  • 02:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Nov 14 02:36:09 UTC 2013
  • 02:13 logmsgbot: LocalisationUpdate completed (1.23wmf2) at Thu Nov 14 02:13:58 UTC 2013
  • 02:07 logmsgbot: LocalisationUpdate completed (1.23wmf3) at Thu Nov 14 02:07:56 UTC 2013
  • 01:27 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'Iaae2da0accd: Enable Bug54847 debug log group and route to '
  • 01:26 logmsgbot: ori updated /a/common to I27cf69dd1: Revert "Rearrange W0 config-pages-supported variables order to be more logical."
  • 00:32 ori-l: restarted stuck profiler-to-carbon on professor

November 13

  • 23:37 mutante: power down db61
  • 23:28 mutante: DNS update - remove db61 (former OTRS box)
  • 22:54 mutante: DNS update - remove osm-web in Tampa, replaced with asset tags for wiping
  • 21:17 gwicke: deployed Parsoid be03c28
  • 20:58 mutante: DNS update - fix warnings for analytics IPv6 addresses
  • 20:48 mutante: DNS update - kill frlabs, merge unmerged config-geo changes
  • 20:46 hashar: Jenkins: fixing up jobs *-phpcs-HEAD which were not using the proper phpcs path 95268
  • 19:14 logmsgbot: reedy synchronized wmf-config/ 'wgImgAuthDetails to true. Remove wmgUseUsabilityInitiative'
  • 19:10 logmsgbot: reedy updated /a/common to I2a05d4a97: Remove usages of $wmgUseUsabilityInitiative
  • 19:07 logmsgbot: reedy updated /a/common to I775790df6: Disable and remove CommunityHiring and CommunityApplications
  • 19:03 logmsgbot: reedy synchronized wmf-config/ 'Disable and remove CommunityHiring and CommunityApplications'
  • 17:20 Reedy: populateRevisionLength.php running in screen on terbium
  • 16:56 apergos: recreating esams varnish backend caches, preallocated with fallocate; the following have been done: amssq49, 50, 53, 55, 57, 58
  • 16:48 hashar: Jenkins: phpcs jobs successfully running on lanthanum. !!!
  • 16:31 hashar: Jenkins: applying label 'hasPhpcs' on lanthanum slave, letting php code sniffer jobs to roam there.
  • 16:29 hashar: Jenkins switched PHP Codesniffer (phpcs) to use the version from integration/phpcs.git instead of the pear one 95172
  • 15:56 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Set $wgImgAuthDetails to true'
  • 14:50 logmsgbot: reedy synchronized php-1.23wmf3/ 'Revert I49e34bdf13e8e8c6cd169c362c283fe1034bdc6d'
  • 14:49 logmsgbot: reedy synchronized php-1.23wmf2/ 'Revert I49e34bdf13e8e8c6cd169c362c283fe1034bdc6d'
  • 14:00 paravoid: adjusting new api appservers weight from 30 -> 20 to balance the load a little better
  • 13:24 hashar: deploying phpcs on Jenkins slave using "the nameless wikimedia deployment system"
  • 11:43 mark: Starting amssq* persistent storage cache fragmentation rework, temporarily disabling puppet in the process
  • 10:53 mark: Manually set vm.dirty_ratio to 10 on amssq57
  • 08:38 apergos: rebooting amssq48, deadlock in kmalloc (started about 10 hours ago)
  • 06:31 apergos: rebooting amssq52, deadlock in kmem_alloc
  • 04:38 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I5043210cb: Enable 'exception-json' debug log group; route to '
  • 04:37 logmsgbot: ori updated /a/common to I5043210cb: Enable 'exception-json' debug log group; route to $wmfUdp2logDest
  • 03:17 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Nov 13 03:17:40 UTC 2013
  • 02:56 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'QueryPage::recache traffic'
  • 02:42 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'move S3 Query::recache traffic to snapshot slave'
  • 02:31 logmsgbot: LocalisationUpdate completed (1.23wmf2) at Wed Nov 13 02:31:21 UTC 2013
  • 02:17 logmsgbot: LocalisationUpdate completed (1.23wmf3) at Wed Nov 13 02:17:07 UTC 2013
  • 02:13 springle: restarting externallinks.el_id schema change jobs
  • 01:30 logmsgbot: ori synchronized php-1.23wmf2/includes/specials/SpecialAllpages.php 'I189ba71de: In Special:AllPages, limit the size of hierarchical lists (Bug: 56840)'
  • 01:29 logmsgbot: ori synchronized php-1.23wmf3/includes/specials/SpecialAllpages.php 'I189ba71de: In Special:AllPages, limit the size of hierarchical lists (Bug: 56840)'
  • 01:26 logmsgbot: ori synchronized php-1.23wmf3/includes/specials/SpecialAllpages.php 'I189ba71de: In Special:AllPages, limit the size of hierarchical lists (Bug: 56840)'
  • 00:24 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'touch'
  • 00:19 logmsgbot: catrope synchronized php-1.23wmf3/resources/startup.js 'touch'
  • 00:16 logmsgbot: catrope synchronized visualeditor.dblist 'Enable VisualEditor on office, board, collab and wikimaniateam wikis'
  • 00:15 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Use visualeditor-default.dblist; enable VisualEditorParsoidForwardCookies on private wikis'
  • 00:14 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Add plumbing for visualeditor-default.dblist, wmgVisualEditorParsoidForwardCookies and wmgVisualEditorNamespaces'
  • 00:11 logmsgbot: catrope synchronized visualeditor-default.dblist 'New dblist for wikis with VisualEditor enabled by default'

November 12

  • 23:36 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/94250'
  • 23:33 logmsgbot: maxsem Finished syncing Wikimedia installation... : Demo deployment, no actual changes
  • 23:31 Ryan_Lane: upgrading wikitech-static to 1.23wmf3
  • 23:20 ^d: gitblit process had hung on antimony, restarted
  • 23:17 Ryan_Lane: upgrading wikitech to 1.23wmf3
  • 23:10 logmsgbot: maxsem Started syncing Wikimedia installation... : Demo deployment, no actual changes
  • 22:59 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/94404'
  • 22:55 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/93988'
  • 22:50 logmsgbot: maxsem synchronized php-1.23wmf3/extensions/MobileFrontend/ 'Fix fatal'
  • 22:47 logmsgbot: maxsem synchronized php-1.23wmf2/extensions/MobileFrontend/ 'Fix fatal'
  • 21:57 cmjohnson1: dns update
  • 21:34 logmsgbot: aaron cleared profiling data
  • 21:21 logmsgbot: aaron synchronized php-1.23wmf3/maintenance/cleanupUploadStash.php '6461a94'
  • 21:04 Reedy: Profiling data clearing seems to work, but echo "-truncate" | nc -q0 -u professor.pmtpa.wmnet 3811 doesn't return...
  • 21:02 logmsgbot: reedy synchronized wmf-config/
  • 20:28 andrewbogott: deployed new AbortNewAccount hook on wikitech
  • 20:14 logmsgbot: reedy cleared profiling data
  • 20:03 logmsgbot: reedy cleared profiling data
  • 19:55 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Iab4777'
  • 19:48 logmsgbot: reedy updated /a/common to I1bb030fce: Enable mwsearch logs
  • 19:44 logmsgbot: reedy synchronized php-1.23wmf2 'Support CIDR ranges in $wgSquidServersNoPurge'
  • 19:40 logmsgbot: reedy synchronized php-1.23wmf3 'Support CIDR ranges in $wgSquidServersNoPurge'
  • 19:18 Reedy: Created EducationProgram tables on elwiki bug 56771
  • 18:01 mark: Rebooting amssq60, stuck in xfs kmem_alloc deadlock
  • 17:55 logmsgbot: reedy synchronized php-1.23wmf3/extensions/MWSearch/
  • 17:54 logmsgbot: reedy synchronized php-1.23wmf2/extensions/MWSearch/
  • 17:38 logmsgbot: reedy updated /a/common to I4baa719b9: CirrusSearch as secondary for nlwiki
  • 17:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable mwsearch logs'
  • 17:09 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php 'Enable Cirrus as secondary on nlwiki'
  • 16:30 cmjohnson1: swapping ps1-c7-eqiad...one side at a time...notifications pending
  • 14:17 springle: paused externallinks OSC jobs after replication glitch on dewiki. original table and data remain untouched
  • 11:33 mark: Manually set dirty_background_ratio to 5 (from 10) on amssq58
  • 11:30 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'thottle rule for WikiCon 94860'
  • 11:29 logmsgbot: hashar synchronized wmf-config/throttle.php 'thottle rule for WikiCon 94860'
  • 09:45 mark: rebooting amssq58 with sysrq-trigger
  • 07:02 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'recache jobs on S2 to db1018'
  • 03:34 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'slave balancing'
  • 03:12 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Nov 12 03:12:05 UTC 2013
  • 02:20 logmsgbot: LocalisationUpdate completed (1.23wmf2) at Tue Nov 12 02:20:54 UTC 2013
  • 02:15 logmsgbot: LocalisationUpdate completed (1.23wmf3) at Tue Nov 12 02:15:03 UTC 2013
  • 01:27 springle: restart db1050 mariadb after outage, let repl catch up. new lvm snaps mount ok. leave out of pool for now

November 11

  • 21:32 paravoid: rebooting ms-be1003, kernel bug, system CPU & I/O wait through the roof
  • 20:08 Nemo_bis: network level of bits application servers eqiad is back to the pre-deploy 18+8 MB/s
  • 19:56 logmsgbot: reedy synchronized php-1.23wmf3/extensions/Wikibase '94790'
  • 19:41 Reedy: Switching 588 wikis to 1.23wmf3 in one go seems to have upset the bits appserver pool
  • 19:26 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.23wmf3
  • 19:22 logmsgbot: reedy updated /a/common to Ibab846000: Bump wmgMemoryLimit to 210MB
  • 17:40 logmsgbot: reedy synchronized php-1.23wmf3/extensions/FlaggedRevs '94774'
  • 16:22 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bump wmgMemoryLimit to 210MB'
  • 16:21 pp-pdf1: restarted all services
  • 16:20 pp-pdf1: - upgrade mwlib to 0.15.12
  • 16:13 paravoid: upgrading image/video scalers
  • 15:32 mark: Moved eqiad wikipedia traffic onto Varnish
  • 15:06 mark: Moved eqiad https/ipv6 text traffic from squid to varnish
  • 14:31 mark: rebooting amssq54, XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
  • 08:50 pp-pdf1: restarted all services
  • 08:50 pp-pdf1: - update mwlib.rl to 0.14.4.
  • 06:43 ori-l: scap: "sudo: no tty present and no askpass program specified" for snapshot1 & snapshot4
  • 06:39 apergos: probably gratuitous powercycle of sq80, it seems fine now in any case
  • 06:33 logmsgbot: ori Finished syncing Wikimedia installation... :
  • 06:32 apergos: sq48 repeat of these errors and hung again, so rt #6274 opened
  • 06:28 apergos: powercycled hung sq48, took two tries to come up, "NMI received for unknown reason 31 on CPU 0" and "mptbase: ioc0: ERROR - Failed to come READY after reset"
  • 06:27 logmsgbot: ori Started syncing Wikimedia installation... :
  • 02:40 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Nov 11 02:40:33 UTC 2013
  • 02:29 ori-l: Apache logs filled with "SearchPhaseExecutionException[Failed to execute phase [dfs], all shards failed"
  • 02:15 ori-l: Continuing inspection of logs on fluorine. memcached-serious.log is flooded with 'Memcached error for key [...]' errors, problem started in May or June judging by log sizes.
  • 02:14 logmsgbot: LocalisationUpdate completed (1.23wmf3) at Mon Nov 11 02:14:09 UTC 2013
  • 02:08 logmsgbot: LocalisationUpdate completed (1.23wmf2) at Mon Nov 11 02:07:59 UTC 2013
  • 02:04 ori-l: Earlier issue identified by Ryan and Leslie as intermittent packet loss between eqiad and esams, due to capacity issue with provider.

November 10

  • 23:42 ori-l: CPU overload in text caches esams
  • 23:42 ori-l: Per Ryan: packet loss from esams to eqiad on xe-4-2-2.cr1-eqiad.wikimedia.org
  • 23:26 ori-l: redis.log: flooded with 'Used automatic re-authentication for Lua script [...]' (68,955 such messages)
  • 23:24 ori-l: fatal log: Fatal error: Allowed memory size of 201326592 bytes exhausted (tried to allocate 72 bytes) at /usr/local/apache/common-local/php-1.23wmf2/extensions/WikibaseDataModel/DataModel/Entity/Entity.php on line 130 (47 such fatals)
  • 23:20 ori-l: poolcounter.log: Pool counter is full (multiple wikis)
  • 23:17 ori-l: exception.log: Exception from line 110 of /usr/local/apache/common-local/php-1.23wmf2/includes/WikiPage.php: Invalid or virtual namespace -1 given. (2 such errors)
  • 23:14 ori-l: exception.log: Exception from line 114 of /usr/local/apache/common-local/php-1.23wmf2/includes/upload/UploadStash.php: UploadStash::getFile No user is logged in, files must belong to users (8 such errors)
  • 23:14 ori-l: exception.log: Exception from line 61 of /usr/local/apache/common-local/php-1.23wmf2/includes/media/ImageHandler.php: No width specified to ImageHandler::makeParamString (194 such errors)
  • 23:13 ori-l: Investigating possible site issue and logging everything that I come across.
  • 02:52 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Nov 10 02:52:22 UTC 2013
  • 02:20 logmsgbot: LocalisationUpdate completed (1.23wmf3) at Sun Nov 10 02:20:46 UTC 2013
  • 02:11 logmsgbot: LocalisationUpdate completed (1.23wmf2) at Sun Nov 10 02:11:33 UTC 2013

November 9

  • 13:46 springle: starting pt-kill daemon on S1 slaves for SpecialAllpages::showToplevel queries > 30s
  • 10:08 paravoid: upgraded ganglia-web to 3.5.10
  • 05:37 logmsgbot: aaron synchronized php-1.23wmf3/includes/filebackend/SwiftFileBackend.php '519883b'
  • 05:35 logmsgbot: aaron synchronized php-1.23wmf2/includes/filebackend/SwiftFileBackend.php '519883b'
  • 02:10 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Nov 9 02:10:54 UTC 2013
  • 02:04 logmsgbot: LocalisationUpdate completed (1.23wmf3) at Sat Nov 9 02:04:55 UTC 2013
  • 02:03 logmsgbot: LocalisationUpdate completed (1.23wmf2) at Sat Nov 9 02:03:45 UTC 2013
  • 01:23 logmsgbot: gwicke synchronized php-1.23wmf3/extensions/Parsoid/php
  • 01:20 logmsgbot: aaron synchronized php-1.23wmf2/maintenance 'c674d4c'
  • 01:19 logmsgbot: aaron synchronized php-1.23wmf3/maintenance 'c674d4c'
  • 01:16 logmsgbot: gwicke synchronized php-1.23wmf2/extensions/Parsoid/php deploy fixed job runner
  • 01:11 logmsgbot: aaron synchronized php-1.23wmf3/includes/filebackend '2afdc06'
  • 01:11 logmsgbot: aaron synchronized php-1.23wmf2/includes/filebackend '2afdc06'
  • 00:46 logmsgbot: aaron synchronized php-1.23wmf3/maintenance
  • 00:32 logmsgbot: aaron synchronized php-1.23wmf2/maintenance/cleanupUploadStash.php '01da29d'

November 8

  • 21:38 logmsgbot: ori updated /a/common to I30f4e5975: logmsg-git-hook: fix commit determination logic & run on new MW branch creation
  • 21:22 logmsgbot: marc synchronized wmf-config/InitialiseSettings.php 'Bug: 56203 - Various changes to wikidatawiki's user rights configuration (for real this time)'
  • 21:00 logmsgbot: marc synchronized wmf-config/InitialiseSettings.php 'Bug: 56203 - Various changes to wikidatawiki's user rights configuration'
  • 19:23 ^d: jobqueue: dropped Cirrus-related jobs from wikidatawiki's queue
  • 19:21 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'No more Cirrus for wikidatawiki'
  • 19:19 mwalker: no longer use cmd=_xclick for donors in China
  • 19:02 cmjohnson: B side on ps1-c7-eqiad will be unplugged to be tested
  • 18:50 mutante: db1050 back up after skipping mount of failed /a
  • 18:45 logmsgbot: faidon synchronized wmf-config/db-eqiad.php 'depool db1050'
  • 18:45 logmsgbot: faidon updated /a/common to Ic66d0e783: Depool db1050, down, disk failed
  • 18:32 mutante: powercycling frozen db1050
  • 17:28 cmjohnson1: removing power to B Side on ps1-c7-eqiad
  • 16:33 cmjohnson1: restarting ps1-c7-eqiad
  • 15:52 Jeff_Green: authdns update to add A/PTR for lutetium public ip
  • 07:09 logmsgbot: ori updated /a/common to I027abe363: Fix typo in logmsg-git-hook
  • 06:54 logmsgbot: ori synchronized logmsg-git-hook 'logmsg-git-hook is meant only for tin, but syncing it for consistency'
  • 06:52 logmsgbot: ori updated /a/common to I3691bbf3a: Remove '--first-parent' arg from rev-list invocation in logmsg-git-hook
  • 06:45 logmsgbot: ori updated /a/common to I70534e64e: nostalgiawiki gets Cirrus
  • 06:24 apergos: analytics1012 down, power mgmt firmware initialization error, opened ticket #6238
  • 04:40 logmsgbot: demon synchronized cirrus.dblist
  • 03:17 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Nov 8 03:17:20 UTC 2013
  • 02:29 logmsgbot: LocalisationUpdate completed (1.23wmf3) at Fri Nov 8 02:28:59 UTC 2013
  • 02:23 gwicke: deployed Parsoid 67fca5bdc7
  • 02:09 logmsgbot: LocalisationUpdate completed (1.23wmf2) at Fri Nov 8 02:09:47 UTC 2013
  • 01:53 ^d: tin: set git config branch.*.rebase true for all deployed branches
  • 01:33 logmsgbot: catrope synchronized docroot/bits/static-current/ 'update static-current symlinks to wmf3'
  • 01:33 springle: adding temporary index to S5 wikidatawiki.wb_terms for slow queries
  • 01:33 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'fix logo path for wikimania2014wiki'
  • 00:21 Krinkle: Reloading zuul to enable Iaceb016cf7df20
  • 00:21 logmsgbot: csteipp synchronized php-1.23wmf3/includes 'bug 55332'
  • 00:08 logmsgbot: catrope synchronized php-1.23wmf3/extensions/VisualEditor 'add more EventLogging events (https://gerrit.wikimedia.org/r/#/c/94092/ )'
  • 00:04 logmsgbot: krinkle synchronized php-1.23wmf3/resources 'I8704a6620ece44d'

November 7

  • 23:54 LeslieCarr: draining ulsfo of all traffic
  • 23:26 logmsgbot: ori synchronized docroot/bits/static-current 'Update static-current symlinks to 1.23wmf3'
  • 22:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 22:20 RoanKattouw: Reloading zuul config for new oojs-core and oojs-ui pipelines
  • 21:16 logmsgbot: anomie synchronized php-1.23wmf3/resources/jquery/jquery.spinner.js 'Backport gerrit change 94161 to fix regression since 1.23wmf1'
  • 21:16 logmsgbot: anomie synchronized php-1.23wmf3/maintenance/jsduck/external.js 'Backport gerrit change 94161 to fix regression since 1.23wmf1'
  • 21:16 logmsgbot: anomie synchronized php-1.23wmf3/maintenance/jsduck/config.json 'Backport gerrit change 94161 to fix regression since 1.23wmf1'
  • 21:15 logmsgbot: anomie synchronized php-1.23wmf3/resources/Resources.php 'Backport gerrit change 94161 to fix regression since 1.23wmf1'
  • 21:15 logmsgbot: anomie synchronized php-1.23wmf3/skins/common/protect.js 'Backport gerrit change 94161 to fix regression since 1.23wmf1'
  • 21:15 logmsgbot: anomie synchronized php-1.23wmf3/skins/common/upload.js 'Backport gerrit change 94161 to fix regression since 1.23wmf1'
  • 21:14 logmsgbot: anomie synchronized php-1.23wmf2/resources/jquery/jquery.spinner.js 'Backport gerrit change 94161 to fix regression since 1.23wmf1'
  • 21:14 logmsgbot: anomie synchronized php-1.23wmf2/maintenance/jsduck/external.js 'Backport gerrit change 94161 to fix regression since 1.23wmf1'
  • 21:14 logmsgbot: anomie synchronized php-1.23wmf2/maintenance/jsduck/config.json 'Backport gerrit change 94161 to fix regression since 1.23wmf1'
  • 21:14 logmsgbot: anomie synchronized php-1.23wmf2/resources/Resources.php 'Backport gerrit change 94161 to fix regression since 1.23wmf1'
  • 21:13 logmsgbot: anomie synchronized php-1.23wmf2/skins/common/protect.js 'Backport gerrit change 94161 to fix regression since 1.23wmf1'
  • 21:13 logmsgbot: anomie synchronized php-1.23wmf2/skins/common/upload.js 'Backport gerrit change 94161 to fix regression since 1.23wmf1'
  • 21:00 Reedy: Created betafeatures database table on metawiki and commonswiki
  • 20:53 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 20:35 ^d: elastic: in-place reindexing for all wikis running cirrus as primary
  • 20:34 logmsgbot: reedy Finished syncing Wikimedia installation... : Ensure l10n cache is up to date
  • 20:26 logmsgbot: reedy Started syncing Wikimedia installation... : Ensure l10n cache is up to date
  • 20:14 logmsgbot: demon synchronized php-1.23wmf2/extensions/CirrusSearch 'Cirrus to master'
  • 20:06 LeslieCarr: put sampling configuration on cr1-sdtpa - possible problems = high RE cpu utilization, which could cause network instability
  • 19:49 logmsgbot: reedy synchronized wmf-config/
  • 19:26 logmsgbot: reedy synchronized w/robots.php
  • 19:18 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: phase1 wikis to 1.23wmf3
  • 19:12 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: all wikipedias to 1.23wmf2
  • 19:09 logmsgbot: reedy synchronized php-1.23wmf3/extensions/Wikibase
  • 19:08 logmsgbot: reedy synchronized docroot and w
  • 19:07 logmsgbot: reedy synchronized wmf-config/
  • 18:33 ^d: for gerrit
  • 18:33 ^d: reloading replication plugin
  • 18:33 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.23wmf2
  • 18:27 logmsgbot: reedy Finished syncing Wikimedia installation... : testwiki to 1.23wmf3 and build l10n cache
  • 18:14 logmsgbot: reedy Started syncing Wikimedia installation... : testwiki to 1.23wmf3 and build l10n cache
  • 18:09 logmsgbot: reedy synchronized docroot and w
  • 18:07 logmsgbot: reedy synchronized php-1.23wmf3 'Staging'
  • 17:10 gwicke: deployed Parsoid 986c1e78708
  • 17:10 ori-l: Ran 'varnishadm ban.url .' on cp1045 & cp1058
  • 17:01 gwicke: roll back Parsoid deploy as we are not able to purge the cache
  • 16:22 gwicke: deployed Parsoid 986c1e787088
  • 16:14 jeremyb: morebots running on tools-login again. will troubleshoot more later
  • 16:13 hashar: jenkins : reducing number of executor on gallium from 8 to 5, we have lanthanum now.
  • 16:11 jeremyb: hi hashar!
  • 16:07 hashar: hello jeremyb
  • 16:06 jeremyb: hello world
  • 15:52 cmjohnson1: dns udpate
  • 15:48 jeremyb: moved morebots back to the grid
  • 11:27 hasharDojo: gallium : killed git process
  • 11:18 hasharDojo: gallium on swap death thanks to git eating all memory :/
  • 10:42 hashar: gallium / jenkins : running git gc --agressive on Zuul git repositories under /srv/ssd/zuul/git
  • 10:01 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'touch'
  • 10:00 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'Use correct bits for uls fonts on labs 94112'
  • 09:37 akosiaris: uploaded snappy version 1.0.4-1build1wmf1 to apt.wikimedia.org
  • 03:12 logmsgbot: aaron synchronized php-1.23wmf2/includes 'eaa917a'
  • 03:07 logmsgbot: aaron synchronized php-1.23wmf1/includes 'eaa917a'
  • 03:00 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Nov 7 03:00:37 UTC 2013
  • 02:23 logmsgbot: LocalisationUpdate completed (1.23wmf1) at Thu Nov 7 02:23:35 UTC 2013
  • 02:15 logmsgbot: LocalisationUpdate completed (1.23wmf2) at Thu Nov 7 02:15:27 UTC 2013
  • 01:42 logmsgbot: ori synchronized php-1.23wmf2/resources/startup.js 'touch'
  • 01:38 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I064c1b202: enable ResourceLoader module storage on mediawikiwiki'

November 6

  • 23:01 awight: update payments from 133e8a7 to 3e5a284
  • 22:03 logmsgbot: aaron synchronized php-1.23wmf2/maintenance '2d08379'
  • 22:02 logmsgbot: aaron synchronized php-1.23wmf1/maintenance '2d08379'
  • 22:00 mutante: cp1059, mobile varnish cache, was OOM but recovered by itself
  • 21:59 logmsgbot: mholmquist Finished syncing Wikimedia installation... : Actually deploy TimedMediaHandler fix to fix the black-box issue for embedded videos
  • 21:55 logmsgbot: mholmquist Started syncing Wikimedia installation... : Actually deploy TimedMediaHandler fix to fix the black-box issue for embedded videos
  • 21:46 logmsgbot: mholmquist Finished syncing Wikimedia installation... : Update TimedMediaHandler to master to fix embedplayer
  • 21:37 logmsgbot: marc synchronized wmf-config/CommonSettings.php 'Make templateeditor available for global groups'
  • 21:29 logmsgbot: mholmquist Started syncing Wikimedia installation... : Update TimedMediaHandler to master to fix embedplayer
  • 20:25 logmsgbot: reedy synchronized docroot/wwwportal/w/404.php 'Add 404 symlink'
  • 19:58 mutante: shred'ding star.wm and *.wm SSL keys from kaulen. backing up old non-SAN wildcard cert and new bz certs on tridge
  • 19:41 logmsgbot: mholmquist Finished syncing Wikimedia installation... : Updates for Multimedia extensions
  • 19:25 logmsgbot: mholmquist Started syncing Wikimedia installation... : Updates for Multimedia extensions
  • 17:32 logmsgbot: mark synchronized wmf-config/squid.php 'Add amssq47-62 IPv6 addresses'
  • 16:54 logmsgbot: reedy synchronized php-1.23wmf2/extensions/WikimediaMaintenance/
  • 16:42 mark: Moved esams ssl -> cache traffic from squid to varnish (manually)
  • 14:33 mark: dist-upgrade && reboot on amssq47..62
  • 12:18 paravoid: started pybal on amslvs1 again; seems to work again, root cause unknown
  • 12:07 paravoid: stopping pybal on amslvs1, wikidata-lb:80 issues, pending further investigation
  • 10:42 akosiaris1: upgraded etherpad-lite to 1.3.0 on zirconium
  • 10:42 akosiaris1: rebooted zirconium for kernel upgrade
  • 08:35 apergos: rebooting neon, ext3 errors on root filesystem
  • 05:09 Tim: on tin: running bug 53687 cleanup script (WikimediaMaintenance/bug-53687/fixOrphans.php) on 37 wikis
  • 03:24 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Nov 6 03:23:59 UTC 2013
  • 02:35 logmsgbot: LocalisationUpdate completed (1.23wmf1) at Wed Nov 6 02:35:33 UTC 2013
  • 02:21 logmsgbot: LocalisationUpdate completed (1.23wmf2) at Wed Nov 6 02:21:52 UTC 2013
  • 00:30 logmsgbot: aaron synchronized php-1.23wmf2/includes/clientpool 'f1df92c'
  • 00:29 logmsgbot: aaron synchronized php-1.23wmf1/includes/clientpool '313457c'

November 5

  • 23:27 superm401: Growth team deployment complete
  • 23:26 logmsgbot: mflaschen synchronized php-1.23wmf2/extensions/GuidedTour/ 'Deploy GuidedTour to 1.23wmf2'
  • 23:26 logmsgbot: mflaschen synchronized php-1.23wmf2/extensions/GettingStarted/ 'Deploy GettingStarted to 1.23wmf2'
  • 23:26 logmsgbot: mflaschen synchronized php-1.23wmf1/extensions/GuidedTour/ 'Deploy GuidedTour to 1.23wmf1'
  • 23:25 logmsgbot: mflaschen synchronized php-1.23wmf1/extensions/GettingStarted/ 'Deploy GettingStarted to 1.23wmf1'
  • 22:41 hashar: sleeping.
  • 21:15 logmsgbot: aaron synchronized php-1.23wmf1/includes/clientpool/RedisConnectionPool.php '9e0a5d8'
  • 21:09 logmsgbot: hashar synchronized wmf-config/logging-labs.php
  • 21:08 logmsgbot: otto synchronized wmf-config/InitialiseSettings.php 'Configuring CirrusSearch for 5 new wikis'
  • 21:08 logmsgbot: hashar synchronized wmf-config/jobqueue-eqiad.php 'removing ParsoidCacheUpdateJob from default queue. Drained a few weeks ago (gwicke)'
  • 20:40 ottomata: re-enabling puppet on analytics1021
  • 20:39 RobH: formey puppet agent disabled
  • 20:26 cmjohnson1: dns update
  • 20:17 cmjohnson1: shutting down search13-20, decommissoned
  • 20:16 mutante: backing up /root on formey to tridge
  • 19:55 mutante: DNS update - switching SVN over to antimony (eqiad)
  • 19:49 RobH: marmontel decommissioned for shipment to eqiad
  • 19:48 RobH: traces of neodyium, promethium, & argon removed and reclaimed to eqiad spare
  • 19:28 logmsgbot: aaron synchronized php-1.23wmf2/includes/clientpool/RedisConnectionPool.php '9e0a5d8'
  • 19:09 logmsgbot: ori synchronized php-1.23wmf1/extensions/Wikibase 'I9d52070a4: Re-introduce siteGroup setting for performance reasons'
  • 19:09 logmsgbot: ori synchronized php-1.23wmf2/extensions/Wikibase 'I9d52070a4: Re-introduce siteGroup setting for performance reasons'
  • 19:07 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I4d6862c3a: add wmgWikibaseSiteGroup'
  • 19:06 logmsgbot: ori synchronized wmf-config/CommonSettings.php 'I4d6862c3a: add wmgWikibaseSiteGroup'
  • 17:56 mutante: more cleanup on kaulen, move old stuff like "1.16wmf4 pilot" :) ,"codereview proxy", old bugzillas etc to backup
  • 17:50 mutante: deleting empty /srv/org/mediawiki on kaulen
  • 17:47 RobH: enabled spam filtering on ops list (logging in case it holds a ton of them)
  • 17:21 cmjohnson1: updating pmtpa pybal searchpool_4 and 5...setting all to false
  • 17:21 mutante: DNS update - killing codereview-proxy
  • 17:16 logmsgbot: ori synchronized php-1.23wmf2/extensions/WikimediaEvents/modules/ext.wikimediaEvents.ve.js 'I4085ee9cb: Update VisualEditor performance event names'
  • 17:15 logmsgbot: ori synchronized php-1.23wmf1/extensions/WikimediaEvents/modules/ext.wikimediaEvents.ve.js 'I4085ee9cb: Update VisualEditor performance event names'
  • 17:12 cmjohnson1: disabling icinga checks on search13-20 for decommissioning rt6194
  • 17:11 RobH: reclaimed marmontel for shipment to eqiad, confirmed decommission steps up to powering down.
  • 17:10 ottomata: stopping puppet on analytics1021 to test jmxtrans kafka stuff
  • 16:40 hashar: beta: extensions update has been blocked since Oct 30th due to some bad registration in mediawiki/extensions.git . Fixed!
  • 16:39 cmjohnson1: dns update
  • 15:48 paravoid: authdns: upgrading gdnsd to 1.10.1 & switching to the proprietary GeoIP database
  • 15:22 logmsgbot: reedy synchronized php-1.23wmf2/includes/media/FormatMetadata.php 'I5f65333cc94b66fd80fcf5abcfad7e10e4669310'
  • 13:29 mark: Changed upload-lb.esams IP addresses
  • 07:09 paravoid: video/imagescaler upgrade
  • 06:33 logmsgbot: ori synchronized wmf-config/CommonSettings.php 'Id33ec9366: Set enwiki's languageLinkSiteGroup to 'wikipedia' (Bug: 56602)'
  • 04:45 springle: begin OSC for el_id on wikis with externallinks >1M rows (smaller ones already done)
  • 03:48 logmsgbot: catrope synchronized php-1.23wmf2/extensions/VisualEditor 'Update VE for cherry-pick'
  • 03:48 logmsgbot: catrope synchronized php-1.23wmf1/extensions/VisualEditor 'Update VE for cherry-pick'
  • 03:45 gwicke: rolled back Parsoid deployment to cdbfdbb28ba2
  • 03:14 gwicke: update Parsoid to 05aa83f to work around Vary: Accept-Encoding issue in production
  • 03:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Nov 5 03:08:30 UTC 2013
  • 02:30 logmsgbot: LocalisationUpdate completed (1.23wmf1) at Tue Nov 5 02:30:26 UTC 2013
  • 02:16 logmsgbot: LocalisationUpdate completed (1.23wmf2) at Tue Nov 5 02:16:42 UTC 2013
  • 02:00 Jeff_Green: stopped mysql on db1008 to clone another
  • 01:57 gwicke: deployed Parsoid f416b82
  • 01:04 RobH: virt0 port saturated at 23:42 (recordkeeping for outage investigation)
  • 01:02 mutante: replaced SSL certs on Bugzilla with new ones that are not star.wm
  • 00:46 springle: running master/slave pt-table-sync on archive tables with 1M+ rows
  • 00:20 Reedy: Test

November 4

  • 23:23 logmsgbot: ori synchronized php-1.23wmf2/resources/mediawiki/mediawiki.inspect.js 'Ib2252003f2: mediawiki.inspect#dumpTable: fix broken FF workaround'
  • 23:12 logmsgbot: csteipp synchronized php-1.23wmf1/includes 'bug 55332'
  • 23:11 logmsgbot: csteipp synchronized php-1.23wmf2/includes 'bug 55332'
  • 22:39 logmsgbot: mwalker synchronized php-1.23wmf2/skins/vector/ 'Deploying fix for bug 56366 : https://gerrit.wikimedia.org/r/#/c/93408/'
  • 22:39 logmsgbot: mwalker synchronized php-1.23wmf1/skins/vector/ 'Deploying fix for bug 56366 : https://gerrit.wikimedia.org/r/#/c/93408/'
  • 22:22 logmsgbot: mwalker synchronized wmf-config/CommonSettings.php 'Enabling cross wiki banner hiding for CentralNotice'
  • 22:16 logmsgbot: mwalker synchronized php-1.23wmf2/resources/mediawiki/mediawiki.inspect.js 'Pushing https://gerrit.wikimedia.org/r/#/c/93587/'
  • 22:15 logmsgbot: mwalker synchronized php-1.23wmf2/extensions/CentralNotice/ 'Updating CentralNotice to master - mobile redirects and cross wiki hiding'
  • 22:15 logmsgbot: mwalker synchronized php-1.23wmf1/extensions/CentralNotice/ 'Updating CentralNotice to master - mobile redirects and cross wiki hiding'
  • 22:07 MaxSem: Reindexing GeoData
  • 21:44 cmjohnson1: dns update
  • 21:43 hashar: on mediawiki/core.git deleted and retagged 1.22.0rc0 (same commit: c00622b )
  • 21:29 mutante: schedule icinga downtime/disable notifications for db3[29] db4[2-6] db5[1235689]
  • 21:00 cmjohnson1: dns update
  • 20:51 logmsgbot: reedy synchronized php-1.23wmf2/extensions/Echo 'bug 56521'
  • 20:47 gwicke: reverted Parsoid deployment due to this charset corruption issue
  • 20:34 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.23wmf2
  • 20:29 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable BetaFeatures and friends on mediawikiwiki and testwikidatawiki'
  • 20:22 Reedy: Created BetaFeatures tables on mediawikiwiki and testwikidatawiki
  • 20:19 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki and testwikidatawiki to 1.23wmf2
  • 19:58 mutante: restarted gitblit on antimony
  • 19:48 gwicke: updated Parsoid to d7b556f25353
  • 18:47 awight: updated crm from 272966e to 8d9240f
  • 18:30 mark: Configured cr1-esams and cr2-knams for new LVS service IP range 91.198.174.192/27
  • 18:25 logmsgbot: aaron synchronized php-1.23wmf2/maintenance/cleanupUploadStash.php 'f3332a3'
  • 17:58 awight: rollback crm to 272966e
  • 17:57 awight: updated crm from 272966e to 63de2d6
  • 17:49 mark: Deactivated ae1.101 on cr1-esams and cr2-knams
  • 16:42 cmjohnson1: dns update
  • 16:31 mark: Disabled multicast routing between eqiad and pmtpa, setup udpmcast between chromium and dobson instead
  • 14:44 mark: Disabled multicast traffic reduction on csw1-sdtpa
  • 14:41 akosiaris: removed LACP configuration from asw-b-eqiad for palladium and set it to standard access-port and private1-b-eqiad vlan
  • 14:40 akosiaris: removed LACP configuration from asw-a-eqiad for strontium and set it to standard access-port and private1-a-eqiad vlan
  • 14:24 mark: Killed udpmcast on chromium
  • 14:13 mark: Configured OSPF/OSPF3 on cr1-eqiad:xe-4/2/2 <--> cr2-knams:xe-1/1/0
  • 13:01 hashar: Jenkins upgrading gearman (0.0.4 -> 0.0.5) plugin on gallium from http://repo.jenkins-ci.org/repo/org/jenkins-ci/plugins/gearman-plugin/0.0.5/
  • 11:22 mark: multicast routing to pmtpa restored for now
  • 10:52 mark: Deactivated BFD on flapping link ae0 between cr1-eqiad and cr2-eqiad
  • 10:47 mark: Deactivated anycast PIM RP on cr2-eqiad
  • 10:21 mark: Reenabled 10G wave between cr2-eqiad and cr1-sdtpa, set high OSPF metrics instead
  • 09:56 akosiaris: added python-apscheduler to apt.wikimedia.org
  • 09:48 hashar: Jenkins: upgrading PHP_CodeSniffer from 1.4.6 to 1.4.7
  • 08:32 logmsgbot: reedy synchronized php-1.23wmf2/ 'Ibcf77ed7f04c14a477d7cfd0e244929c552c3394'
  • 08:00 paravoid: cr2-eqiad set xe-5/2/1 disable; 10g wave to cr1-sdtpa, flapping since yesterday, causing packet loss and outages
  • 06:24 jeremyb: left morebots running on tools-login because it doesn't work on all grid hosts. see https://gerrit.wikimedia.org/r/93426
  • 05:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Nov 4 05:08:07 UTC 2013
  • 04:40 logmsgbot: LocalisationUpdate completed (1.23wmf2) at Mon Nov 4 04:40:40 UTC 2013
  • 04:30 logmsgbot: LocalisationUpdate completed (1.23wmf1) at Mon Nov 4 04:30:48 UTC 2013
  • 04:19 ori-l: l10nupdate failures are due to "No submodule mapping found in .gitmodules for path 'WikibaseDatabase'"; repo appears to have been deleted; fixed with git rm --cached WikibaseDatabase as l10nupdate on tin.
  • 04:15 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
  • 03:12 Tim: moved udpmcast to chromium since it actually has an external IP address
  • 03:05 Tim: moved udpmcast.py from dobson to tungsten to work around failure of multicast routing eqiad -> pmtpa
  • 02:01 logmsgbot: LocalisationUpdate failed: git pull of extensions failed

November 3

  • 12:50 mark: Activated BFD between cr2-eqiad and cr1-sdtpa link
  • 10:07 mark: Send traffic from Japan and Korea to ulsfo, https://gerrit.wikimedia.org/r/#/c/93309/
  • 07:26 ori-l: kaulen's apache was down (killed by oom killer) & refused to start. copied over old config per #5011, restarted apache, & disabled puppet for now.
  • 07:07 p858snake|l: Bugzilla isn't loading
  • 02:02 logmsgbot: LocalisationUpdate failed: git pull of extensions failed

November 2

  • 19:50 cmjohnson1: dns update
  • 16:13 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 12:43 springle: start xtrabackup db52->db69 and db39->db71
  • 09:56 apergos: restarted twemproxy on ~10 of the api servers in eqiad in the lower memory/core group, was seeing a lot of entries in memcached-serious log and some processes churning on that
  • 07:37 apergos: restarted a bunch of the apaches on the 12gb memory apis that were maxed out at their 100 client limit
  • 07:09 apergos: and increased again to 30, after looking at memory on both groups of boxes.
  • 06:58 apergos: increase weight of mw1189-1208 api servers from 20 to 25, they handle the load better (hopefully)
  • 06:03 logmsgbot: springle synchronized wmf-config/db-pmtpa.php 'depool db69 and db71 for reassignment during pmtpa decomm'
  • 02:00 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
  • 00:39 awight: updated crm from da3448df1404311620adc50b4d4ca28abfd2f3cb to 272966e353244e668a64c54ca4fa6f344079f4bf

November 1

  • 23:55 awight: updated crm from 4a4c31cf60b783a5269c3e18b4baaf3bde5568ab to da3448df1404311620adc50b4d4ca28abfd2f3cb
  • 23:45 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'Ib19944b16: $wgResourceLoaderStorageEnabled = true on test & test2'
  • 22:57 awight: crm updated from 28093721d29226e9bd17d8be43db8a0deb51fe22 to 4a4c31cf60b783a5269c3e18b4baaf3bde5568ab
  • 22:45 awight: crm updated from 21cb97b5083f4a32e67b5b6dac4b58d733b8d7ca to 28093721d29226e9bd17d8be43db8a0deb51fe22
  • 22:23 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Nov 1 22:23:20 UTC 2013
  • 22:16 logmsgbot: LocalisationUpdate completed (1.23wmf2) at Fri Nov 1 22:16:51 UTC 2013
  • 21:57 logmsgbot: LocalisationUpdate completed (1.23wmf1) at Fri Nov 1 21:57:34 UTC 2013
  • 21:49 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild l10n cache for MultimediaViewer update
  • 21:39 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild l10n cache for MultimediaViewer update
  • 21:25 logmsgbot: reedy synchronized wmf-config 'touch'
  • 21:24 logmsgbot: reedy synchronized php-1.23wmf2/
  • 21:22 awight: updated crm from dca2c8cc0eaa503e6d0c9921952cb85855e3b110 to 21cb97b5083f4a32e67b5b6dac4b58d733b8d7ca
  • 19:59 logmsgbot: reedy synchronized php-1.23wmf1/extensions/Wikibase
  • 19:49 hashar: jenkins: refreshing all *pep8 jobs, pep8 wrapper for puppet pointing to a wrong path
  • 19:48 Reedy: Added rc_source column to testwikidatawiki.recentchanges
  • 19:47 Reedy: Added rc_source column to mediawikiwiki.recentchanges
  • 19:46 Reedy: Added rc_source column to test2wiki.recentchanges
  • 19:43 Reedy: Added rc_source column to testwiki.recentchanges
  • 19:11 Reedy: Created betafeatures_user_counts on testwiki and test2wiki
  • 19:06 logmsgbot: reedy synchronized wmf-config/ 'Enable magic beta extensions'
  • 18:58 logmsgbot: reedy Finished syncing Wikimedia installation... : scap take 2 to build l10ncache with magical beta things
  • 18:49 logmsgbot: reedy Started syncing Wikimedia installation... : scap take 2 to build l10ncache with magical beta things
  • 18:31 logmsgbot: reedy Finished syncing Wikimedia installation... : testwiki and test2wiki to 1.23wmf2
  • 18:17 logmsgbot: reedy Started syncing Wikimedia installation... : testwiki and test2wiki to 1.23wmf2
  • 18:14 paravoid: reenabling mw1114-mw1148
  • 18:12 logmsgbot: reedy synchronized wmf-config/
  • 18:08 paravoid: swapoff on all api appservers
  • 17:53 logmsgbot: reedy synchronized wmf-config/
  • 17:26 paravoid: depooling mw1114-mw1148; balancing is unfair, boxes overloaded, mw1189-mw1208 capable of handling the load
  • 16:01 apergos: shot one more forceSearchIndex on arsenic because we were back in swap, only one left...
  • 13:51 hasharEat: gallium : Jenkins console log compression completed, saved 80G out of 500G total disk space.
  • 10:57 apergos: shot one more forceSearchIndex on arsenic, only two left... (using about 12gb between them)
  • 10:51 logmsgbot: springle synchronized wmf-config/db-pmtpa.php 'depool first batch of pmtpa boxes to be decommissioned/shipped'
  • 09:08 hashar: Jenkins: fixing up a race condition in MobileFrontend qunit tests. Both variant were using the same path and the first job to complete would break the other one by deleting the mediawiki install. 93030
  • 08:18 apergos: shot two of the forceSearchIndex on arsenic, they were using 7gb between them and arsenic was in swapdeath
  • 03:01 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Nov 1 03:01:11 UTC 2013
  • 02:15 logmsgbot: LocalisationUpdate completed (1.23wmf1) at Fri Nov 1 02:15:15 UTC 2013
  • 01:25 LeslieCarr: reverted the ulsfo traffic check, due to editors appearing to come from 127.0.0.1
  • 01:21 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Clean up 127.0.0.1 logging code'
  • 01:11 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Debugging 127.0.0.1 issue: add XFP logging'
  • 01:04 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Debugging 127.0.0.1 issue'

October 31

  • 23:55 cmjohnson1: dns update
  • 23:41 mutante: delete dsh groups 'nagios' and 'misc-servers' from tin/fenari. they are gone from puppet and unused (just not actively deleted)
  • 23:33 cmjohnson1: dns update
  • 23:32 logmsgbot: reedy synchronized php-1.23wmf1/extensions/MobileFrontend 'touch'
  • 23:31 logmsgbot: reedy synchronized php-1.23wmf1/resources 'touch'
  • 23:30 logmsgbot: reedy synchronized wmf-config/ 'touch'
  • 22:31 RobH: restarted parsoid on wtp1011
  • 22:17 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Bye bye MW 1.22
  • 21:38 logmsgbot: demon Finished syncing Wikimedia installation... : Unbreak cluster. Like most things, the sequel wasn't as good as the original
  • 21:33 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Fixing for the last time'
  • 21:33 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Fixing for the last time'
  • 21:21 logmsgbot: demon Started syncing Wikimedia installation... : Unbreak cluster. Like most things, the sequel wasn't as good as the original
  • 21:02 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 20:55 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.23wmf2
  • 18:31 cmjohnson1: removing decommissioned servers alsted, amaranth, durant, spence, williams from dsh group misc-servers on tin
  • 18:29 logmsgbot: demon synchronized wmf-config/
  • 18:24 logmsgbot: demon synchronized wmf-config/ 'Proper poolcounter config for Cirrus'
  • 18:20 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'touch'
  • 18:16 logmsgbot: hashar synchronized php-1.22wmf22/extensions/PoolCounter/ 'making sure all apaches got all the files.'
  • 18:16 logmsgbot: hashar synchronized php-1.23wmf1/extensions/PoolCounter/ 'making sure all apaches got all the files.'
  • 18:11 cmjohnson1: dns update
  • 18:01 logmsgbot: demon synchronized w/ 'Cluster to known good state'
  • 17:59 logmsgbot: demon synchronized wmf-config/ 'Cluster to known good state'
  • 16:52 logmsgbot: reedy synchronized wmf-config/
  • 16:30 logmsgbot: reedy synchronized docroot and w
  • 16:29 logmsgbot: reedy synchronized php-1.23wmf2 'Staging php-1.23wmf2'
  • 14:52 mark: Sending text (all wikis) traffic from OC to ulsfo
  • 13:51 logmsgbot: mark synchronized wmf-config/squid.php 'Update cache list with ulsfo text caches'
  • 13:39 cmjohnson: dns update
  • 12:59 akosiaris: added python-pbr backported from saucy to apt.wikimedia.org
  • 11:23 akosiaris: added python-gear package by hashar to apt.wikimedia.org
  • 11:13 hashar: jenkins compressing all console logs
  • 10:29 hashar: jenkins / gallium : got rid of the old /var/lib/jenkins/tmpfs 512MB mount ( 92853 + manual amount )
  • 03:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Oct 31 03:26:30 UTC 2013
  • 02:47 logmsgbot: LocalisationUpdate completed (1.22wmf22) at Thu Oct 31 02:47:38 UTC 2013
  • 02:25 logmsgbot: LocalisationUpdate completed (1.23wmf1) at Thu Oct 31 02:25:43 UTC 2013
  • 01:07 awight: crm updated from 06348e7eea3c914e9444af588e0d7ffeb640366c to dca2c8cc0eaa503e6d0c9921952cb85855e3b110
  • 00:46 ori-l: running MONITOR for 60 seconds on redis instance on mc1002

October 30

  • 23:51 logmsgbot: maxsem synchronized wmf-config/mobile.php 'https://gerrit.wikimedia.org/r/92811'
  • 23:37 logmsgbot: catrope synchronized php-1.23wmf1/extensions/VisualEditor/ 'VE cherry-pick'
  • 23:30 logmsgbot: maxsem synchronized php-1.23wmf1/extensions/MobileFrontend/ 'https://gerrit.wikimedia.org/r/#/c/92676/'
  • 23:29 logmsgbot: demon synchronized wmf-config/PoolCounterSettings-pmtpa.php 'Upping search maxqueue to 600'
  • 23:29 logmsgbot: demon synchronized wmf-config/PoolCounterSettings-eqiad.php 'Upping search maxqueue to 600'
  • 23:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable Zero on wikidataw'
  • 23:17 logmsgbot: maxsem synchronized php-1.23wmf1/extensions/MobileFrontend/ 'Shit hit fan'
  • 23:08 logmsgbot: maxsem synchronized php-1.23wmf1/extensions/MobileFrontend/ 'https://gerrit.wikimedia.org/r/#/c/92676/'
  • 23:07 logmsgbot: maxsem synchronized php-1.22wmf22/extensions/MobileFrontend/ 'https://gerrit.wikimedia.org/r/#/c/92676/'
  • 23:02 logmsgbot: yurik synchronized php-1.23wmf1/extensions/ZeroRatedMobileAccess/
  • 22:48 logmsgbot: yurik synchronized php-1.22wmf22/extensions/ZeroRatedMobileAccess/
  • 22:00 K4-713: updated civicrm to 06348e7eea3
  • 19:56 logmsgbot: demon Finished syncing Wikimedia installation... : No changes, making sure arsenic is synced
  • 19:46 logmsgbot: demon Started syncing Wikimedia installation... : No changes, making sure arsenic is synced
  • 19:45 logmsgbot: demon synchronized multiversion 'Making sure multiversion is up to date on arsenic'
  • 19:44 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: Making sure wikiversions is up to date on arsenic
  • 19:27 cmjohnson: removing decom'd srv* from /etc/dsh/groups nrpe-ext-stores, nagios
  • 19:22 cmjohnson: dns update
  • 18:39 logmsgbot: demon synchronized robots.txt 'New robots.txt changes for mobileview'
  • 17:47 awight: dedupe updated from 5b9f19ba719dc6 to 922b3aa6b797a
  • 17:38 logmsgbot: demon Finished syncing Wikimedia installation... : No changes, making sure arsenic is synced
  • 17:36 awight: updated crm from 6f370cccad7f9 to 117025ce5c993d
  • 17:20 logmsgbot: demon Started syncing Wikimedia installation... : No changes, making sure arsenic is synced
  • 17:17 ^d: aborted sync
  • 17:17 logmsgbot: demon Started syncing Wikimedia installation... : No changes, making sure arsenic is synced
  • 15:40 hashar: Broke jenkins for a couple hours while doing some path migration. All jobs got updated in the process and failed jobs got retriggered.
  • 10:40 apergos: restarted morebots (failed to rejoin after netsplit)

October 29

  • 21:14 logmsgbot: maxsem synchronized php-1.23wmf1/extensions/MobileFrontend/ 'https://gerrit.wikimedia.org/r/#/c/92474/'
  • 21:12 logmsgbot: maxsem synchronized php-1.22wmf22/extensions/MobileFrontend/ 'https://gerrit.wikimedia.org/r/#/c/92474/'
  • 21:05 LeslieCarr: one of the sdtpa <-> eqiad links is down
  • 20:50 ^demon|lunch: elasticsearch: stopped wikidatawiki indexing processes on terbium, causing too much load. working with RobH on getting an alternative place to run these scripts.
  • 20:08 mwalker: fundraising cluster back to normal
  • 20:06 cmjohnson1: dns update
  • 19:14 mutante: create mw/1.22 dir on dumps.wm.org, publish mw 1.22.0rc0 - http://dumps.wikimedia.org/mediawiki/1.22/
  • 19:06 cmjohnson1: dns update
  • 18:49 cmjohnson1: dns update
  • 17:51 mwalker: frack cluster back in maintenance mode while I figure out why persistent STOMP messages fail to be stored
  • 17:45 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Switch all small wikis to use extension1 db'
  • 17:30 mwalker: taking the payments cluster out of maintenance mode for testing
  • 16:49 logmsgbot: reedy synchronized php-1.23wmf1/extensions/Wikibase
  • 16:35 mwalker: placing fundraising cluster into maintenance mode for ActiveMQ and database maintenance
  • 16:23 mwalker: updating payments from e475603
  • 16:13 ^d: elasticsearch: created wikidatawiki index, running indexing from 8 processes on terbium.
  • 16:08 logmsgbot: reedy synchronized php-1.23wmf1/includes/upload/UploadStash.php
  • 16:07 logmsgbot: reedy synchronized php-1.22wmf22/includes/upload/UploadStash.php
  • 16:04 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Cirrus on wikidatawiki as secondary index'
  • 14:49 cmjohnson1: dns update
  • 13:15 paravoid: reprepro: include backported graphite-carbon, graphite-web, python-whisper (plus deps python-django-tagging, flot) from saucy; replace custom-built python-whisper
  • 12:09 hashar: jenkins refreshing jobs to let them recurse in git submodules [[bugzilla:55614|bug 55614]] 92511
  • 11:56 hashar: lanthanum.eqiad.wmnet running apt-get upgrade
  • 11:36 hashar: gerrit: creating integration/phpunit to hold PHPUnit files for deployment
  • 07:13 paravoid: "ln -s /apache /a" on terbium; foreachwiki and friends was broken (for a while) due to /usr/local/lib/mw-deployment-vars.sh pointing to non-existent /a
  • 03:01 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Oct 29 03:01:47 UTC 2013
  • 02:29 logmsgbot: LocalisationUpdate completed (1.22wmf22) at Tue Oct 29 02:29:03 UTC 2013
  • 02:15 logmsgbot: LocalisationUpdate completed (1.23wmf1) at Tue Oct 29 02:15:32 UTC 2013
  • 00:25 logmsgbot: ori synchronized wmf-config/CommonSettings-labs.php 'I6ca3517dc: Enable localStorage module caching (If2ad2d80d) on beta cluster'

October 28

  • 22:31 cmjohnson1: dns update 2nd set of changes
  • 22:29 RobH: updated dns
  • 22:10 cmjohnson1: powering down search 21 -36
  • 21:25 logmsgbot: reedy synchronized php-1.23wmf1/includes/db/Database.php 'bug 56124'
  • 21:18 logmsgbot: awight synchronized php-1.23wmf1/extensions/CentralNotice
  • 21:17 logmsgbot: awight synchronized php-1.22wmf22/extensions/CentralNotice
  • 20:41 manybubbles: built search indexes for a bunch of small wikis that didn't seem to have them. we should be done getting index doesn't exist errors.
  • 19:20 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: non wikipedia to 1.23wmf1
  • 18:45 awight: payments updated from 84bb204 to e475603
  • 18:26 logmsgbot: reedy synchronized php-1.23wmf1/extensions/Wikibase
  • 18:13 logmsgbot: reedy synchronized php-1.23wmf1/includes/specials/SpecialContributions.php
  • 17:13 logmsgbot: aaron synchronized wmf-config/ 'Switched to JobQueueFederated'
  • 15:02 cmjohnson1: removing search21-36 from pybal search_pools
  • 13:26 manybubbles: restarted elasticsearch nodes to pick up new config
  • 08:42 apergos: restarted etherpadlite on zirconium, see ticket 6093, it was not saving edits
  • 08:22 Nemo_bis: etherpad.wikimedia.org seems read-only
  • 07:17 springle: delaying one tampa slave per shard during OSC
  • 06:41 apergos: powercycled ms-be1001, unaccessible via ssh or mgmt
  • 02:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Oct 28 02:36:16 UTC 2013
  • 02:15 logmsgbot: LocalisationUpdate completed (1.23wmf1) at Mon Oct 28 02:15:57 UTC 2013
  • 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf22) at Mon Oct 28 02:08:37 UTC 2013

October 27

  • 17:19 Reedy: Creating echo_notification, echo_event and echo_email_batch IF NOT EXISTS for all echo related wikis on 10.64.16.18
  • 17:07 Reedy: Created echo related tables for ndswiktionary on 10.64.16.18
  • 17:06 Reedy: Created echo related tables for kawiktionary on 10.64.16.18
  • 02:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Oct 27 02:26:02 UTC 2013
  • 02:12 logmsgbot: LocalisationUpdate completed (1.23wmf1) at Sun Oct 27 02:12:09 UTC 2013
  • 02:07 logmsgbot: LocalisationUpdate completed (1.22wmf22) at Sun Oct 27 02:06:57 UTC 2013

October 26

  • 02:32 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Oct 26 02:32:40 UTC 2013
  • 02:16 logmsgbot: LocalisationUpdate completed (1.23wmf1) at Sat Oct 26 02:16:30 UTC 2013
  • 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf22) at Sat Oct 26 02:08:20 UTC 2013

October 25

  • 23:44 logmsgbot: aaron synchronized php-1.22wmf22/extensions/SwiftCloudFiles '2661b2d'
  • 22:00 cmjohnson1: removing search21-36 from pybal
  • 21:44 cmjohnson1: powering off to decom search21-36
  • 21:15 cmjohnson1: dns update
  • 21:13 cmjohnson1: removing mw110 from pybal
  • 19:42 K4-7131: updated payments cluster to 84bb20494793b4
  • 16:59 mutante: DNS update, fix mw110 reverse entries
  • 16:22 cmjohnson1: dns update
  • 13:59 apergos: shot swiftrepl on copper, almost out of space, tossed the log... restart as you like
  • 13:02 mark: Rebooting cp1046, xfs mem alloc issues
  • 06:38 Tim: set email address for user MahdiBot to be the same as Mahdiz
  • 03:21 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Oct 25 03:21:39 UTC 2013
  • 02:48 logmsgbot: LocalisationUpdate completed (1.23wmf1) at Fri Oct 25 02:48:10 UTC 2013
  • 02:23 springle: starting recentchanges OSC bug 55844
  • 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf22) at Fri Oct 25 02:16:50 UTC 2013
  • 01:04 logmsgbot: kaldari synchronized php-1.22wmf22/extensions/MobileFrontend/ 'Updating MobileFrontend for cherrypick, one more time with feeling'
  • 01:01 logmsgbot: kaldari synchronized php-1.22wmf22/extensions/MobileFrontend/ 'Updating MobileFrontend for cherrypick'
  • 00:49 logmsgbot: kaldari synchronized php-1.22wmf22/extensions/MobileFrontend/ 'Updating MobileFrontend for cherrypick'
  • 00:13 logmsgbot: kaldari synchronized php-1.22wmf22/extensions/MobileFrontend/ 'Updating MobileFrontend for cherrypick'
  • 00:06 logmsgbot: kaldari synchronized php-1.22wmf22/extensions/MobileFrontend/ 'Updating MobileFrontend for cherrypick'
  • 00:03 logmsgbot: kaldari synchronized php-1.23wmf1/extensions/MobileFrontend/ 'Updating MobileFrontend for cherrypick'

October 24

  • 23:40 logmsgbot: reedy synchronized docroot/noc/conf/
  • 23:25 logmsgbot: reedy synchronized docroot/noc/
  • 23:23 logmsgbot: reedy synchronized multiversion
  • 23:05 logmsgbot: reedy synchronized multiversion/
  • 22:59 logmsgbot: reedy synchronized docroot and w
  • 22:08 logmsgbot: awight synchronized php-1.23wmf1/extensions/CentralNotice
  • 22:07 logmsgbot: awight synchronized php-1.22wmf22/extensions/CentralNotice
  • 20:33 mwalker: updating listeners from 15b4723
  • 20:19 logmsgbot: reedy synchronized echowikis.dblist 're-enable echo on test2wiki'
  • 20:18 Reedy: Created echo tables for test2wiki on 10.64.16.18
  • 20:16 logmsgbot: reedy synchronized echowikis.dblist 'Disable Echo on test2wiki due to a lack of database tables in extension1'
  • 19:53 logmsgbot: reedy synchronized wmf-config/
  • 19:45 logmsgbot: reedy synchronized database lists files:
  • 19:12 Reedy: Created WikiLove extension tables on kowiki
  • 18:53 mutante: DNS update - remove nomcom wiki
  • 18:27 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.22wmf22
  • 18:25 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: phase1 wikis to 1.23wmf1
  • 18:22 logmsgbot: reedy Finished syncing Wikimedia installation... : testwiki to 1.23wmf1 and build l10n cache
  • 18:09 logmsgbot: reedy Started syncing Wikimedia installation... : testwiki to 1.23wmf1 and build l10n cache
  • 18:01 logmsgbot: reedy synchronized docroot and w
  • 18:00 logmsgbot: reedy synchronized php-1.23wmf1 'make sure'
  • 18:00 awight: updated tools from bc75134 to 5b9f19b
  • 17:58 paravoid: shutting down ms-be1006, to be used by cmjohnson1 for testing firmware changes
  • 17:58 logmsgbot: reedy synchronized php-1.23wmf1
  • 17:38 mutante: added gwicke to wmf ldap group
  • 12:38 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool db1026'
  • 11:19 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/90759'
  • 11:18 logmsgbot: maxsem synchronized wmf-config/CommonSettings.php 'https://gerrit.wikimedia.org/r/#/c/91058/'
  • 10:47 logmsgbot: tstarling Finished syncing Wikimedia installation... : fix linkprefix regexes in 1.22wmf22 (Iaa7eaa44)
  • 10:41 logmsgbot: tstarling Started syncing Wikimedia installation... : fix linkprefix regexes in 1.22wmf22 (Iaa7eaa44)
  • 10:24 logmsgbot: tstarling Finished syncing Wikimedia installation... : fix linkprefix regexes (Iaa7eaa44)
  • 10:15 logmsgbot: tstarling Started syncing Wikimedia installation... : fix linkprefix regexes (Iaa7eaa44)
  • 10:06 springle: set global userstat=1 on mariadb instances for audit
  • 07:13 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1026'
  • 03:11 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Oct 24 03:11:12 UTC 2013
  • 02:42 logmsgbot: LocalisationUpdate completed (1.22wmf21) at Thu Oct 24 02:42:50 UTC 2013
  • 02:22 logmsgbot: LocalisationUpdate completed (1.22wmf22) at Thu Oct 24 02:22:25 UTC 2013
  • 01:36 logmsgbot: kaldari synchronized wmf-config/mobile.php 'updating for MobileFrontend schema changes'
  • 01:32 logmsgbot: kaldari synchronized php-1.22wmf21/extensions/MobileFrontend/ 'updating MobileFrontend for cherrypick'
  • 01:29 logmsgbot: kaldari synchronized php-1.22wmf22/extensions/MobileFrontend/ 'updating MobileFrontend for cherrypick'
  • 01:16 logmsgbot: kaldari synchronized php-1.22wmf22/extensions/MobileFrontend/ 'updating MobileFrontend for cherrypick'
  • 00:43 logmsgbot: kaldari synchronized wmf-config/mobile.php 'readding mobile schema config for now'
  • 00:22 logmsgbot: kaldari synchronized php-1.22wmf21/extensions/MobileFrontend/ 'syncing MobileFrontend 1.22wmf21'
  • 00:21 logmsgbot: kaldari synchronized php-1.22wmf22/extensions/MobileFrontend/ 'syncing MobileFrontend 1.22wmf22'
  • 00:17 logmsgbot: kaldari synchronized wmf-config/mobile.php 'MobileFrontend schema configs moved to MobileFrontend extension'

October 23

  • 23:55 awight_: payments updated from 300fa32 to f69c37d
  • 21:16 Ryan_Lane: upgrading salt to 0.17.1
  • 20:14 logmsgbot: yurik synchronized php-1.22wmf22/extensions/ZeroRatedMobileAccess/
  • 19:53 hashar: all mw Jenkins jobs now have $wgShowExceptionDetails enabled \O/ [[bugzilla:55595|bug 55595]]
  • 19:35 hashar: Zuul: restarting to apply upgrade.
  • 19:32 hashar: upgrading Zuul on gallium to add a debug statement (tag: wmf-deploy-20131023 commit: 1e3adfd) See https://www.mediawiki.org/wiki/Zuul#upgrading for upgrade procedure used.
  • 19:26 Ryan_Lane: deploying change 91319 to git deployment system
  • 18:54 cmjohnson1: db1033 powering down to reseat DIMM removed from pool according to RT5830
  • 18:35 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php
  • 18:01 logmsgbot: yurik synchronized php-1.22wmf21/extensions/ZeroRatedMobileAccess/
  • 17:47 cmjohnson1: dns update
  • 16:59 cmjohnson1: ms-be1006 replacing disk at slot 2
  • 15:52 logmsgbot: mark synchronized wmf-config/squid.php 'Update squid list for ulsfo'
  • 15:40 mark: Sending OC mobile traffic to ulsfo
  • 15:05 hashar: jenkins updating mediawiki-core-phpunit-databaseless to no more rely on ant
  • 14:55 hashar: jenkins: updating job mediawiki-core-phpunit-api to no more rely on ant
  • 14:21 ^d: elastic: rebuilding all indexes from screen in terbium
  • 12:49 paravoid: restarting apaches, "stray config": mw1117 mw1126 mw1145 mw1201 mw1206
  • 07:43 apergos: mw125: stopped apache shortly after adding back to dns, (needed to wait for an hour for the update to reach the pmtpa recursors so puppet could run), at first successful puppet run mw-sync completed so this host should be good to go now
  • 05:54 ori-l: sync-file: 'Could not resolve hostname mw125: Name or service not known'
  • 05:53 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I11c90ed5a: Added some purge/thumbnail rate limits'
  • 05:30 apergos: powercycled srv291, unresponsive to ping, no login at mgmt console
  • 02:47 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Oct 23 02:47:16 UTC 2013
  • 02:24 logmsgbot: LocalisationUpdate completed (1.22wmf21) at Wed Oct 23 02:24:39 UTC 2013
  • 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf22) at Wed Oct 23 02:16:06 UTC 2013
  • 00:30 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'warmup db1034'

October 22

  • 23:52 mwalker: actually updating payments listeners... 243ec9c to 15b4723
  • 23:28 logmsgbot: catrope synchronized php-1.22wmf22/extensions/VisualEditor 'Update VisualEditor for cherry-picks'
  • 23:28 logmsgbot: catrope synchronized php-1.22wmf21/extensions/VisualEditor 'Update VisualEditor for cherry-picks'
  • 23:20 mwalker: updating listeners from 243ec9c for smashpig recovery
  • 23:05 paravoid: powercycling ms-be1011, locked up, no messages in console
  • 23:05 logmsgbot: mwalker synchronized php-1.22wmf22/extensions/CentralNotice 'Updating CentralNotice to masterish'
  • 22:51 K4-713: updated payments cluster to 300fa32
  • 22:24 logmsgbot: ori synchronized php-1.22wmf22/resources/mediawiki/mediawiki.inspect.js 'I0e7a47b5a: mediawiki.inspect: add CSS report (4/4)'
  • 22:23 logmsgbot: ori synchronized php-1.22wmf22/resources/mediawiki/mediawiki.js 'I0e7a47b5a: mediawiki.inspect: add CSS report (3/4)'
  • 22:23 logmsgbot: ori synchronized php-1.22wmf21/resources/mediawiki/mediawiki.inspect.js 'I0e7a47b5a: mediawiki.inspect: add CSS report (2/4)'
  • 22:23 logmsgbot: ori synchronized php-1.22wmf21/resources/mediawiki/mediawiki.js 'I0e7a47b5a: mediawiki.inspect: add CSS report (1/4)'
  • 22:08 logmsgbot: ori synchronized wmf-config/CommonSettings.php 'I6f6e0ffa3: Update beta to use loginwiki for SUL (bug 55760)'
  • 22:07 ori-l: during previous sync-file: could not resolve mw125; time out connecting to srv291
  • 22:07 logmsgbot: ori synchronized wmf-config/Bug54847.php 'Ide07296c5: Fix error message for recycled passwords (bug 56002)'
  • 21:38 logmsgbot: reedy synchronized php-1.22wmf22/extensions/Wikibase
  • 21:37 logmsgbot: reedy synchronized php-1.22wmf22/extensions/RelatedSites
  • 21:08 Ryan_Lane: pushed git deployment changes 91151 and 91155 to production
  • 20:30 bblack: varnish: banned 'obj.http.content-length == 33518' on text varnishes everywhere (extract2.php leakage)
  • 19:54 Ryan_Lane: updating git deployment system with change 90277 (0424d6f7)
  • 19:23 cmjohnson1: dns update
  • 18:42 logmsgbot: bsitu synchronized php-1.22wmf21/extensions/Echo/modules/overlay 'touch'
  • 18:35 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'touch'
  • 18:08 logmsgbot: bsitu synchronized echowikis.dblist 'Enable Echo and Thanks on all wikis except dewiki and itwiki'
  • 18:08 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable Echo and Thanks on all wikis except dewiki and itwiki'
  • 18:01 mutante: restarting a few Apaches that didnt get restarted by graceful-all (hrmmpf)
  • 17:51 awight: civicrm updated 41b621a to 6f370cc
  • 17:23 mark: Banned Content-Type == "application/x-httpd-php" in text Varnish (eqiad)
  • 17:19 mutante: apache graceful-all
  • 17:06 mutante: sync-apache, graceful to unify www portal docroots
  • 16:31 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Touch'
  • 16:26 logmsgbot: demon synchronized cirrus.dblist 'foundationwiki gets cirrus'
  • 16:19 logmsgbot: reedy synchronized docroot/wwwportal
  • 10:09 apergos: powercycled sq80, hung on mgmt console
  • 09:42 hashar: Zuul: no more voting on code-review label [[bugzilla:55757|bug 55757]] 91157. Will make the database happier I guess
  • 09:30 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool db1018'
  • 03:46 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool db1007, warm up db1041'
  • 03:06 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Oct 22 03:06:46 UTC 2013
  • 03:00 springle: upgrading db1017 to precise
  • 02:39 logmsgbot: LocalisationUpdate completed (1.22wmf21) at Tue Oct 22 02:39:32 UTC 2013
  • 02:25 logmsgbot: LocalisationUpdate completed (1.22wmf22) at Tue Oct 22 02:25:12 UTC 2013
  • 01:40 springle: start xtrabackup clone db1007 to db1041
  • 01:26 springle: start xtrabackup clone db1018 to db1034
  • 01:06 springle: upgrading db1034 & db1041 to precise
  • 00:53 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1007 while cloning'
  • 00:46 awight: crm schema change: wmf_civicrm 6008->6009
  • 00:40 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1018 while cloning'
  • 00:36 awight: manual schema change on crm: update system set schema_version=6008 where name='wmf_civicrm';
  • 00:24 awight: fundraising-crm updated from 9247de6 to 41b621a

October 21

  • 23:22 logmsgbot: mwalker synchronized php-1.22wmf21/extensions/CentralNotice/ 'Reverting earlier change'
  • 23:21 logmsgbot: mwalker synchronized php-1.22wmf22/extensions/CentralNotice/ 'Reverting earlier change'
  • 23:16 mwalker: CentralNotice deploy had an issue where we are now pushing ALL traffic to the mobile site for CN -- ganglia reports a spike and I'm now reverting the change
  • 23:09 ^demon|lunch: elastic: index created for bnwiki, running force indexing in 4 processes on terbium
  • 23:07 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'bnwiki gets cirrus as secondary'
  • 23:05 logmsgbot: mwalker synchronized php-1.22wmf22/extensions/CentralNotice/ 'Updating CentralNotice to master'
  • 23:05 logmsgbot: mwalker synchronized php-1.22wmf21/extensions/CentralNotice/ 'Updating CentralNotice to master'
  • 22:49 paravoid: powercycling ms-be1004, locked up(?)
  • 22:13 logmsgbot: maxsem synchronized php-1.22wmf22/extensions/MobileFrontend/ 'https://gerrit.wikimedia.org/r/91065'
  • 22:04 AaronSchulz: Running syncFilebackend on all wikis (should reduce sync errors; a few a popping up in the logs)
  • 21:34 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Added eqiad swift to multiwrite backends'
  • 21:06 cmjohnson1: morebots are you working?
  • 21:04 cmjohnson1: dns update
  • 20:05 cmjohnson1: dns update
  • 19:38 cmjohnson1: dns update
  • 18:40 logmsgbot: reedy synchronized wmf-config/interwiki.cdb 'Updating interwiki cache'
  • 18:34 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.22wmf22
  • 18:29 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 17:44 cmjohnson1: reclaiming cp1021-36, cp1041-42 per RT5981
  • 17:24 Reedy: The @Wikimedia Operations team is seeking proposals for a new datacenter in the midwestern/western US: https://blog.wikimedia.org/2013/10/21/rfp-new-datacenter-continental-us/
  • 16:30 ori-l: ran sync-apache for I1113b9594 & Ie73ce6213; verified on mw60; gracefuling.
  • 16:26 logmsgbot: demon synchronized wmf-config/CirrusSearch-production.php 'LVS for cirrus on all wikis'
  • 16:11 logmsgbot: demon synchronized wmf-config/CirrusSearch-production.php 'LVS for cirrus on test2wiki'
  • 16:07 RobH: updated akismet blog plugin
  • 16:03 RobH: tweaked permissions of the blog's w3 caching plugin to actually make it work again
  • 11:47 mark: Shutdown csw2-esams:xe-2/1/1 (1 DF leg)
  • 11:31 hashar: Gerrit replication have been broken since Oct 19th roughly 20:50 UTC. [[bugzilla:55948|bug 55948]]
  • 10:43 hashar: Gerrit replication is broken, side effects: git.wm.o shows outdated trees and Jenkins might be missing some commits
  • 09:16 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 09:16 logmsgbot: reedy synchronized docroot/bits/apple-touch/
  • 09:03 hashar: restarting Zuul
  • 09:02 hashar: Jenkins restarted / upgraded
  • 08:59 hashar: rerestarting Jenkins.
  • 08:51 akosiaris: forced verified +2 on gerrit 90874 since jenkins is being upgraded
  • 08:46 hashar: jenkins: upgrading plugins
  • 08:43 hashar: stopping Zuul again. Need to upgrade Jenkins plugins
  • 08:41 hashar: restarted Zuul
  • 08:00 hashar: stopping Zuul / Jenkins
  • 07:59 hashar: upgrading Jenkins for scheduled maintenance
  • 06:15 apergos: moved older swift replication logs from copper:/root to iron:/root/swift-repl/ (now gzipped), copper was full
  • 03:21 Tim: on mw1109: stopped apache to test cgconfig
  • 02:39 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Oct 21 02:39:00 UTC 2013
  • 02:19 logmsgbot: LocalisationUpdate completed (1.22wmf22) at Mon Oct 21 02:19:25 UTC 2013
  • 02:10 logmsgbot: LocalisationUpdate completed (1.22wmf21) at Mon Oct 21 02:10:47 UTC 2013
  • 00:38 ^d: bringing gerrit down to troubleshoot replication

October 20

  • 02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Oct 20 02:07:08 UTC 2013
  • 02:02 logmsgbot: LocalisationUpdate completed (1.22wmf22) at Sun Oct 20 02:02:20 UTC 2013
  • 02:01 logmsgbot: LocalisationUpdate completed (1.22wmf21) at Sun Oct 20 02:01:39 UTC 2013

October 19

  • 20:54 ^d: gerrit: installed 2.7-rc2-507-g1e7090b, service back up
  • 13:14 hashar: Jenkins: updating Zuul config 90695
  • 07:13 ori-l: reports of geoiplookup timing out in AU at enwiki VPT: https://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28technical%29#geoiplookup.wikimedia.org
  • 02:30 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Oct 19 02:30:42 UTC 2013
  • 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf22) at Sat Oct 19 02:15:49 UTC 2013
  • 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf21) at Sat Oct 19 02:08:11 UTC 2013

October 18

  • 23:39 logmsgbot: reedy synchronized docroot
  • 23:17 logmsgbot: reedy synchronized docroot and w
  • 23:14 cmjohnson1: dns update
  • 22:27 logmsgbot: aaron synchronized php-1.22wmf22/includes/job '5ac886a'
  • 22:26 logmsgbot: aaron synchronized php-1.22wmf22/includes/HashRing.php '5ac886a'
  • 22:21 mutante: deleting deprecated postrewrites.conf from Apaches, syncing, gracefulling for cleanup changes
  • 20:44 mutante: sync-apache, graceful'ing ..
  • 20:09 cmjohnson1: dns update
  • 19:23 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'db1035 (s3) back into the pool'
  • 19:11 logmsgbot: marc synchronized wmf-config/InitialiseSettings-labs.php 'Enable zero extension in labs (beta)'
  • 18:52 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'db1019 (s3) to normal weight in pool'
  • 17:56 cmjohnson1: disconnecting ps1-b6-eqiad from network
  • 16:59 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'All wikis in cirrus group get cirrus'
  • 16:58 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Use new cirrus.dblist'
  • 16:57 logmsgbot: demon synchronized cirrus.dblist 'New cirrus.dblist'
  • 15:46 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'db1019 (s3) in pool warming up after upgrade/conversion to mariadb'
  • 13:45 logmsgbot: mark synchronized wmf-config/squid.php 'Update cache list'
  • 09:53 hashar: Jenkins finally producing code coverage reports: https://integration.wikimedia.org/cover/mediawiki-core/master/php/
  • 09:06 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'db1035 (s3) depooled to use as clone source'
  • 06:30 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'remove Haproxy after S7 master switch'
  • 06:25 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1039 for master switch'
  • 06:21 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'insert temporary HAproxy for S7 master rotation'
  • 05:28 logmsgbot: springle synchronized wmf-config/db-pmtpa.php 'sync new S6 master setting to pmtpa'
  • 05:27 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'remove HAproxy after S2 master switch'
  • 05:09 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool new master db1036 from slaves ready for HAproxy write-traffic switch'
  • 05:00 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'insert HAproxy for S2 master rotation'
  • 03:10 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Oct 18 03:10:02 UTC 2013
  • 02:44 logmsgbot: springle synchronized wmf-config/db-pmtpa.php 's3 master rotation done'
  • 02:43 logmsgbot: LocalisationUpdate completed (1.22wmf22) at Fri Oct 18 02:43:45 UTC 2013
  • 02:43 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's3 master rotation done'
  • 02:37 logmsgbot: springle synchronized wmf-config/db-pmtpa.php 's3 master rotation for upgrade'
  • 02:33 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's3 master rotation for upgrade'
  • 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf21) at Fri Oct 18 02:15:02 UTC 2013
  • 01:19 logmsgbot: awight synchronized php-1.22wmf21/extensions/CentralNotice
  • 01:02 logmsgbot: awight synchronized php-1.22wmf21/extensions/CentralNotice
  • 01:01 logmsgbot: awight synchronized php-1.22wmf21/extensions/CentralNotice
  • 00:34 logmsgbot: awight synchronized php-1.22wmf21/extensions/CentralNotice
  • 00:31 Tim: fixing MW cgroup on mw1109
  • 00:24 logmsgbot: awight synchronized php-1.22wmf21/extensions/CentralNotice
  • 00:04 mutante: upgrading bugzilla to 4.2.7

October 17

  • 23:13 logmsgbot: reedy synchronized wmf-config 'touch'
  • 22:58 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wtf
  • 22:38 logmsgbot: reedy synchronized php-1.22wmf21/extensions/Translate
  • 22:35 logmsgbot: reedy synchronized php-1.22wmf22/extensions/Translate
  • 22:17 awight: applied schema change CentralNotice/patches/patch-campaign_throttle.sql
  • 20:23 logmsgbot: mflaschen synchronized wmf-config/InitialiseSettings.php 'Growth team deploy: End GettingStarted A/B test'
  • 19:57 ^d: mw.org elastic index finished rebuilding
  • 19:49 cmjohnson1: authdns update
  • 19:41 mutante: installing package upgrades on kaulen and zinc
  • 19:01 ^d: rebuilt elastic indexes for test/test2/testwikidata/loginwiki, mw.org reindex in progress
  • 18:39 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'loginwiki gets cirrus, for real this time'
  • 18:38 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'loginwiki gets cirrus'
  • 18:13 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki, loginwiki, testwiki, test2wiki and testwikidatawiki to 1.22wmf22
  • 18:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.22wmf21
  • 18:03 logmsgbot: reedy synchronized wmf-config/
  • 17:58 logmsgbot: reedy synchronized php-1.22wmf22/extensions/UniversalLanguageSelector
  • 17:58 Jeff_Green: dist-upgrade and reboot most fundraising hosts
  • 17:55 cmjohnson1: ms-be1008 removing/replacing hdd slot 2 /dev/sdc
  • 17:54 cmjohnson1: ms-be1003 removing/replacing hdd slot 3 /dev/sdd
  • 17:46 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.22wmf21 till window
  • 17:35 logmsgbot: reedy Finished syncing Wikimedia installation... : testwiki to 1.22wmf22 and build l10n cache
  • 17:08 logmsgbot: reedy Started syncing Wikimedia installation... : testwiki to 1.22wmf22 and build l10n cache
  • 16:50 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php '26ee33e'
  • 16:50 logmsgbot: reedy synchronized php-1.22wmf22 'staging'
  • 16:32 MaxSem: Recreating GeoData index
  • 16:32 logmsgbot: reedy synchronized docroot and w
  • 16:20 logmsgbot: reedy synchronized docroot and w
  • 16:01 logmsgbot: reedy synchronized multiversion/
  • 16:00 logmsgbot: reedy synchronized multiversion/
  • 15:58 logmsgbot: reedy synchronized multiversion/
  • 15:57 logmsgbot: reedy synchronized multiversion/
  • 14:30 Jeff_Green: dist-upgrade & reboot payments*
  • 09:53 logmsgbot: maxsem synchronized php-1.22wmf20/extensions/MobileFrontend/
  • 09:52 logmsgbot: maxsem synchronized php-1.22wmf21/extensions/MobileFrontend/
  • 03:58 springle: testing updateCollation on S6 with adaptive hash off
  • 02:50 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Oct 17 02:50:36 UTC 2013
  • 02:28 logmsgbot: LocalisationUpdate completed (1.22wmf20) at Thu Oct 17 02:28:15 UTC 2013
  • 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf21) at Thu Oct 17 02:15:04 UTC 2013
  • 01:51 logmsgbot: marc synchronized wmf-config/InitialiseSettings.php 'bug 55432: add templateeditor restriction level and user right to enwiki'

October 16

  • 23:38 logmsgbot: maxsem synchronized php-1.22wmf20/extensions/MobileFrontend/
  • 23:36 logmsgbot: maxsem synchronized php-1.22wmf21/extensions/MobileFrontend/
  • 23:35 mutante: re-enabling puppet on zinc
  • 23:08 mutante: ddsh -g apaches -F20 "rm /etc/apache2/wmf/en2.conf"
  • 21:27 paravoid: staggered reboot of lvs4003/lvs4001
  • 20:57 apergos: removing also from /etc/dsh/group on tin: yaseo.text yaseo.upload srv284 (note, none of these files are managed by puppet)
  • 20:53 apergos: deleting from /etc/dsh/group on tin these 0 byte files not in puppet: ams.puppettmp_7604 apache-eqiad.puppettmp_2863 apaches-eqiad cp esams mail.ini.list
  • 20:34 paravoid: powercycling lvs4003, locked up(?)
  • 20:25 logmsgbot: olivneh synchronized php-1.22wmf21/extensions/WikimediaEvents 'Updating WikimediaEvents for VE logging (2/2)'
  • 20:25 logmsgbot: olivneh synchronized php-1.22wmf20/extensions/WikimediaEvents 'Updating WikimediaEvents for VE logging (1/2)'
  • 20:17 logmsgbot: olivneh synchronized php-1.22wmf21/extensions/VisualEditor 'Updating VE for cherry-picks I54602394e & I3b58ce0f4'
  • 20:17 logmsgbot: olivneh synchronized php-1.22wmf21/extensions/WikimediaEvents 'Updating WikimediaEvents for VE logging (2/2)'
  • 20:17 logmsgbot: olivneh synchronized php-1.22wmf20/extensions/WikimediaEvents 'Updating WikimediaEvents for VE logging (1/2)'
  • 20:14 mutante: removing unused apache config files via dsh, they have been merged "D" but sync-apache doesn't delete things
  • 18:58 gwicke: updated Parsoid to 9b52271
  • 18:25 mutante: DNS update - remove williams
  • 18:02 mutante: shutting down williams (former OTRS), stab
  • 17:57 mutante: stopping spamassassin on williams
  • 17:57 logmsgbot: olivneh synchronized php-1.22wmf21/extensions/Wikibase 'Update Wikibase to 8903c2234d'
  • 17:44 logmsgbot: olivneh synchronized php-1.22wmf21/resources/startup.js 'Touch startup.js'
  • 17:43 logmsgbot: olivneh synchronized php-1.22wmf21/skins/vector/screen.less 'I412575f37fix for bug 55779'
  • 17:23 Aaron|home: Creating/setting ACLs on eqiad swift containers for all wikis
  • 17:21 logmsgbot: demon synchronized php-1.22wmf21/includes/LinksUpdate.php 'I115f8cf9'
  • 17:21 logmsgbot: demon synchronized php-1.22wmf21/includes/DefaultSettings.php 'I115f8cf9'
  • 17:12 Nemo_bis: fr.wikipedia was briefly down: Frakir> [19:01:27] <rc-pmtpa> - no editions - [19:05:23] <rc-pmtpa>
  • 16:38 logmsgbot: demon synchronized php-1.22wmf20/includes/LinksUpdate.php 'I115f8cf9'
  • 16:37 logmsgbot: demon synchronized php-1.22wmf20/includes/DefaultSettings.php 'I115f8cf9'
  • 16:01 cmjohnson1: authdns update
  • 14:11 hashar: upgrading php on lanthanum.eqiad.wmnet ( jenkins CI slave )
  • 14:11 hashar: jenkins : upgrading php on gallium.
  • 11:58 hashar: Jenkins: migrate extensions jobs to install mediawiki with sqlite with a shell script instead of inlined shell in the job configuration 90104
  • 02:29 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Oct 16 02:29:45 UTC 2013
  • 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf20) at Wed Oct 16 02:16:35 UTC 2013
  • 02:09 logmsgbot: LocalisationUpdate completed (1.22wmf21) at Wed Oct 16 02:09:25 UTC 2013
  • 00:02 logmsgbot: reedy synchronized docroot and w
  • 00:00 logmsgbot: reedy synchronized wmf-config/

October 15

  • 23:21 logmsgbot: maxsem synchronized php-1.22wmf20/extensions/MobileFrontend/
  • 23:19 logmsgbot: maxsem synchronized php-1.22wmf21/extensions/MobileFrontend/
  • 23:12 logmsgbot: catrope synchronized php-1.22wmf21/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.ViewPageTarget.js 'cherry-pick of https://gerrit.wikimedia.org/r/#/c/89773/'
  • 22:55 logmsgbot: maxsem synchronized php-1.22wmf20/extensions/MobileFrontend/
  • 22:53 logmsgbot: maxsem synchronized php-1.22wmf21/extensions/MobileFrontend/
  • 22:42 mutante: DNS update - removing decom'ed search servers1-12
  • 22:22 logmsgbot: aaron synchronized wmf-config/filebackend.php '013c272'
  • 21:56 awight: deploying donor_review module and schema
  • 21:55 awight: deployed civicrm from b4392c7 to 9247de6 (occurred at 1857 UTC)
  • 20:35 mutante: DNS update, add uk.planet
  • 19:30 cmjohnson1: adding mw1125 back, enabling pybal
  • 18:03 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 17:54 manybubbles: rebuilt search index for enwikisource's non-content documents because it had snuck out of date and was causing trouble
  • 17:49 mutante: enable icinga notifications for db1006 services
  • 17:42 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'db1022 (s6) back in the pool'
  • 16:42 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'enwikisource getting cirrus as primary'
  • 16:36 logmsgbot: bd808 synchronized php-1.22wmf21/maintenance/purgeChangedPages.php 'Updating purgeChangedPages.php to prepare for HTCP purge'
  • 16:31 logmsgbot: bd808 synchronized php-1.22wmf20/maintenance/purgeChangedPages.php
  • 16:28 logmsgbot: bd808 synchronized php-1.22wmf20/maintenance/purgeChangedPages.php
  • 16:23 logmsgbot: bd808 synchronized php-1.22wmf20/maintenance/purgeChangedPages.php
  • 15:04 logmsgbot: reedy synchronized php-1.22wmf21/extensions/Wikibase
  • 14:35 logmsgbot: reedy synchronized multiversion/ 'Changing away from docroot based site deduction'
  • 13:31 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'db1006 (s6) back to full weight in pool'
  • 11:37 hashar: jenkins refreshing all mediawiki extensions jobs so they log Junit results under /logs/
  • 10:29 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'db1006 (s6) repooled, warming up after upgrade/conversion'
  • 06:56 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'db1022 (s6) depooled to use as clone source'
  • 05:18 apergos: powercycled cp1052, it was unresponsive and unreachable via mgmt
  • 02:37 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Oct 15 02:37:39 UTC 2013
  • 02:20 logmsgbot: LocalisationUpdate completed (1.22wmf21) at Tue Oct 15 02:20:18 UTC 2013
  • 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf20) at Tue Oct 15 02:14:53 UTC 2013
  • 01:53 Tim: rebooting mw1035 for init testing
  • 00:41 Tim: manually created MW cgroups on all apaches since apparently the init script is totally broken
  • 00:29 Tim: on mw1035,mw1145,mw1078,mw1152,mw1150: running "start cgconfig" to fix cgroups
  • 00:21 Tim: some apaches are missing /sys/fs/cgroup/* so shell execution from MW fails. Stopped apache on mw1035 during investigation

October 14

  • 23:46 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
  • 23:42 logmsgbot: tstarling synchronized php-1.22wmf20/extensions/Math
  • 23:41 Tim: pushed out temporary checkout of extension/Math 679ce7fe2fe, will update submodule soon
  • 23:40 logmsgbot: tstarling synchronized php-1.22wmf20/extensions/Math/MathTexvc.php
  • 23:14 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Disable Math debug log group (found to be not pertinent & too verbose)'
  • 22:21 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Enable Math debug log group'
  • 16:11 mark: Setup LVS service IP range aggregates on the core routers
  • 06:45 apergos: resized /var/lib/puppet on stafford, it ran out of room last night (but recovered after cron job reports cleanup)
  • 05:18 logmsgbot: springle synchronized wmf-config/db-pmtpa.php 'update s6 master'
  • 05:17 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool db1007'
  • 04:34 logmsgbot: springle synchronized wmf-config/db-pmtpa.php 'update s6 master'
  • 04:24 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1006 to investigate connection errors. promote db1027'
  • 03:33 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1027 before promotion'
  • 02:25 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Oct 14 02:25:24 UTC 2013
  • 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf21) at Mon Oct 14 02:15:08 UTC 2013
  • 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf20) at Mon Oct 14 02:08:37 UTC 2013

October 13

  • 19:31 ori-l: restarting gmetad on nickel to clear old metrics (again)
  • 18:19 ori-l: restarting gmetad on nickel to clear old metrics
  • 02:27 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Oct 13 02:27:58 UTC 2013
  • 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf21) at Sun Oct 13 02:16:34 UTC 2013
  • 02:09 logmsgbot: LocalisationUpdate completed (1.22wmf20) at Sun Oct 13 02:09:00 UTC 2013

October 12

  • 21:11 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php
  • 02:29 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Oct 12 02:29:58 UTC 2013
  • 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf21) at Sat Oct 12 02:17:30 UTC 2013
  • 02:09 logmsgbot: LocalisationUpdate completed (1.22wmf20) at Sat Oct 12 02:09:44 UTC 2013
  • 00:32 mwalker: trying again to upgrade payments donatoin interface from 73fe0ca (this time without error pages)

October 11

  • 23:46 mwalker: updating payments donationinterface from 73fe0ca -- removing adyen from random selection and deploying error pages
  • 21:51 Ryan_Lane: updated git-deploy (change 88934)
  • 20:45 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php
  • 18:27 Reedy: Updating collation on frwiki as reedy on tin in screen session. Gonna take a while
  • 18:26 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 18:26 Reedy: Updated collation on viwikivoyage
  • 18:24 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 18:22 logmsgbot: reedy synchronized database lists files:
  • 18:20 logmsgbot: reedy synchronized wmf-config/
  • 18:19 logmsgbot: reedy synchronized php-fatal-error.html
  • 17:59 Reedy: Created EducationProgram tables on eswiki
  • 17:56 Reedy: Created FlaggedRevs tables on ptwiki
  • 17:22 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 17:21 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 15:36 akosiaris: removed libicu42 from apt.wikimedia.org
  • 15:26 logmsgbot: reedy synchronized php-1.22wmf20/maintenance/purgeChangedPages.php
  • 14:13 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'db1024 (s7) back to full weight in pool'
  • 12:02 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'db1024 (s7) warming up after upgrade/conversion to mariadb'
  • 10:31 mutante: DNS update, add pk.wikimedia and id.planet.wikimedia
  • 10:26 mutante: sync-apache and graceful for pk.wm redirect
  • 10:14 mutante: mass unsubscribed users, then deleted wikimedia-commits list, has been closed down, no archives
  • 02:18 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Oct 11 02:18:46 UTC 2013
  • 02:13 logmsgbot: LocalisationUpdate completed (1.22wmf21) at Fri Oct 11 02:13:55 UTC 2013
  • 02:01 logmsgbot: LocalisationUpdate completed (1.22wmf20) at Fri Oct 11 02:01:46 UTC 2013
  • 00:45 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1021 while cloning to db1045'

October 10

  • 23:29 paravoid: reprepro include php5 5.3.10-1ubuntu3.8+wmf2 (wmf1 rebuilt in a clean environment, i.e. with precise's libicu)
  • 22:24 logmsgbot: olivneh synchronized php-1.22wmf20/resources/Resources.php 'Updating 1.22wmf20 for mw.loader.inspect (3/3)'
  • 22:23 logmsgbot: olivneh synchronized php-1.22wmf20/resources/mediawiki/mediawiki.js 'Updating 1.22wmf20 for mw.loader.inspect (2/3)'
  • 22:23 logmsgbot: olivneh synchronized php-1.22wmf20/resources/mediawiki/mediawiki.inspect.js 'Updating 1.22wmf20 for mw.loader.inspect (1/3)'
  • 22:23 logmsgbot: olivneh synchronized php-1.22wmf21/extensions/WikimediaEvents 'Updating WikimediaEvents to log DOM retrieval timing for VE'
  • 21:28 ori-l: stopping StatsD service on tungsten for a couple of minutes to debug
  • 20:18 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Oct 10 20:18:49 UTC 2013
  • 20:07 logmsgbot: LocalisationUpdate completed (1.22wmf21) at Thu Oct 10 20:07:11 UTC 2013
  • 19:35 logmsgbot: LocalisationUpdate completed (1.22wmf20) at Thu Oct 10 19:35:14 UTC 2013
  • 18:58 logmsgbot: reedy synchronized wmf-config/
  • 18:52 logmsgbot: reedy synchronized php-1.22wmf21/extensions/MobileFrontend/MobileFrontend.php
  • 18:50 logmsgbot: reedy synchronized wmf-config/
  • 18:47 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: phase1 wikis back to 1.22wmf21
  • 18:47 logmsgbot: reedy synchronized php-1.22wmf20/extensions/MobileFrontend
  • 18:41 logmsgbot: reedy synchronized php-1.22wmf21/extensions/
  • 18:32 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: phase1 wikis back to 1.22wmf20
  • 18:30 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: phase1 wikis to 1.22wmf21
  • 18:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.22wmf20
  • 17:37 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.22wmf20
  • 17:34 logmsgbot: reedy Finished syncing Wikimedia installation... : testwiki to 1.22wmf21 and build and sync l10ncache
  • 17:16 logmsgbot: reedy Started syncing Wikimedia installation... : testwiki to 1.22wmf21 and build and sync l10ncache
  • 17:06 logmsgbot: reedy synchronized docroot and w
  • 17:03 logmsgbot: reedy synchronized php-1.22wmf21 'staging'
  • 15:46 mark: Imported firewall ACLs on cr2-ulsfo, activated on lo0.0
  • 15:46 mark: Configured cr2-ulsfo:ae1 with all sub interfaces and VRRP
  • 15:12 hashar: jenkins : triggering operations-puppet-rspec (non voting) on operations/puppet.git 89000
  • 15:04 andrewbogott: removed mysql-server-5.5 package from stat1 to ensure that it doesn't get used now that it's no longer maintained by puppet.
  • 15:01 mark: Corrected wrong interface addresses on cr1-ulsfo:ae0 sub-units
  • 14:38 mark: Setup iBGP between cr1-ulsfo and cr2-ulsfo
  • 14:38 mark: Configured AS65003 ulsfo BGP confederation on cr1-ulsfo and cr2-ulsfo
  • 14:25 mark: Setup OSPF and OSPF3 on cr1-ulsfo:ae0.2 <--> cr2-ulsfo:ae0.2
  • 14:24 mark: Setup cr1-ulsfo:ae0 <--> cr2-ulsfo:ae0
  • 12:04 mark: Rebooting scs-ulsfo, pmshell is segfaulting
  • 11:52 akosiaris: upgraded php5 packages from php5_5.3.10-1ubuntu3.6+wmf1 to php5_5.3.10-1ubuntu3.8+wmf1 on apt.wikimedia.org
  • 08:27 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'depool db1024 (s7) for conversion to mariadb'
  • 08:16 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'db1039 (s7) to normal weight in pool'
  • 06:50 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'warm up db1039 in s7'
  • 05:16 springle: start xtrabackup clone db1039 to db1007
  • 04:19 springle: upgrading db1007 to precise + mariadb
  • 02:28 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Oct 10 02:28:22 UTC 2013
  • 02:18 logmsgbot: LocalisationUpdate completed (1.22wmf19) at Thu Oct 10 02:18:06 UTC 2013

October 9

  • 16:40 RobH: puppetmaster was overloaded and returning 500s, tried killing open processes and restarting puppetmaster on stafford
  • 13:19 springle: start dump db1007 to db1039 for S7 file per table
  • 10:57 mark: Non-Wikipedia projects are now on Varnish for eqiad non-ipv6, non-https traffic
  • 09:34 hashar: jenkins back up
  • 09:10 hashar: gallium : zuul backup, waiting for jenkins to complete start up
  • 08:24 ori-l: Stopping EventLogging, then rebooting vanadium for kernel upgrade
  • 08:02 hashar: rebooting gallium
  • 07:18 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool db1022 in S6. move db1039 to assist with upgrades in S7. depool db1007 for upgrade'
  • 06:12 logmsgbot: aaron synchronized php-1.22wmf19/includes 'c6d64f5488c124636ab46712e1d104e5c7076325'
  • 02:45 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Oct 9 02:44:59 UTC 2013
  • 02:30 logmsgbot: LocalisationUpdate completed (1.22wmf19) at Wed Oct 9 02:30:06 UTC 2013
  • 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf20) at Wed Oct 9 02:16:08 UTC 2013
  • 00:52 logmsgbot: krinkle synchronized php-1.22wmf19/extensions/VisualEditor 'I03d68280ddd9506'
  • 00:49 logmsgbot: krinkle synchronized php-1.22wmf10/extensions/VisualEditor 'I85bce4d40e430318'

October 8

  • 23:29 logmsgbot: mflaschen synchronized php-1.22wmf20/extensions/GettingStarted/ 'Deploy GettingStarted to wmf20 for A/B test'
  • 23:29 logmsgbot: mflaschen synchronized php-1.22wmf19/extensions/GettingStarted/ 'Deploy GettingStarted to wmf19 for A/B test'
  • 23:28 logmsgbot: mflaschen synchronized wmf-config/InitialiseSettings.php 'A/B test for GettingStarted'
  • 23:27 logmsgbot: mflaschen synchronized wmf-config/CommonSettings.php 'A/B test for GettingStarted'
  • 22:26 cmjohnson1: reinstalling amssq48-62
  • 22:25 LeslieCarr: moved fxp0 on cr1-esams into MGMT logical system
  • 21:44 logmsgbot: aaron synchronized php-1.22wmf20/includes/LinksUpdate.php '337272176758b06c0ed9fd8859d358730152c155'
  • 21:40 logmsgbot: aaron synchronized php-1.22wmf20/includes/WikiPage.php '337272176758b06c0ed9fd8859d358730152c155'
  • 20:27 logmsgbot: maxsem synchronized php-1.22wmf20/extensions/MobileFrontend/javascripts/loggingSchemas/MobileWebClickTracking.js
  • 20:05 logmsgbot: maxsem synchronized php-1.22wmf19/extensions/MobileFrontend/javascripts/loggingSchemas/MobileWebClickTracking.js
  • 19:16 logmsgbot: bsitu synchronized php-1.22wmf19/extensions/Echo 'Touch Js file'
  • 19:11 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable Echo and Thanks on various wikis'
  • 19:06 logmsgbot: bsitu synchronized echowikis.dblist 'Enable Echo and Thanks on various wikis'
  • 19:06 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable Echo and Thanks on various wikis'
  • 18:55 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo and Thanks to Master
  • 18:36 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo and Thanks to Master
  • 18:29 logmsgbot: reedy synchronized php-1.22wmf20/includes/Wiki.php
  • 17:25 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php '2 shards per replica in elastic'
  • 17:23 logmsgbot: olivneh synchronized wmf-config/extension-list 'CoreEvents -> WikimediaEvents'
  • 17:22 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'CoreEvents -> WikimediaEvents'
  • 17:18 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'CoreEvents -> WikimediaEvents'
  • 17:15 logmsgbot: olivneh synchronized php-1.22wmf19/extensions/WikimediaEvents 'CoreEvents -> WikimediaEvents'
  • 17:12 logmsgbot: olivneh synchronized php-1.22wmf20/extensions/WikimediaEvents 'CoreEvents -> WikimediaEvents'
  • 16:58 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'db1046 (s6) back to normal weight in pool'
  • 16:48 logmsgbot: reedy synchronized wmf-config/
  • 15:54 apergos: fixed up salt clients on analytics 1005-6, 1009-27
  • 14:34 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'warm up db1040 (s6) after upgrade'
  • 13:38 mutante: installing package upgrades on hooper
  • 13:28 mutante: installing package upgrades on helium
  • 12:49 mutante: installing package upgrades on formey
  • 12:37 hashar: restarted Jenkins by mistake :-(
  • 12:35 mutante: install package upgrades on gallium (jenkins)
  • 12:35 akosiaris: apt-get purge defoma on image_scalers dsh group
  • 12:17 mutante: installing package upgrades on ekrem (IRC)
  • 09:43 mutante: installing package upgrades on antimony (gitblit, gerrit repl.)
  • 09:39 mutante: puppetstoredconfigclean.rb .. killing williams.wikimedia.org .. done
  • 09:11 mutante: installing package upgrades on calcium (cameras)
  • 07:52 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'warm up db1039 in s6'
  • 07:20 logmsgbot: ariel synchronized wmf-config/db-eqiad.php 'depool db1040 for upgrade/conversion to mariadb'
  • 06:37 apergos: fixed up salt on mw1046, mw1072, mw1173
  • 03:30 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's6 db1015 to full steam'
  • 03:23 springle: xtrabackup clone s6 db1039 to db1022
  • 03:20 springle: rmmod nf_conntrack on db1002 causing mass mysql connect failure
  • 02:35 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Oct 8 02:35:56 UTC 2013
  • 02:31 springle: start online optimize logging indexes s4 & s5
  • 02:25 logmsgbot: LocalisationUpdate completed (1.22wmf19) at Tue Oct 8 02:25:27 UTC 2013
  • 02:13 logmsgbot: LocalisationUpdate completed (1.22wmf20) at Tue Oct 8 02:13:54 UTC 2013
  • 00:11 LeslieCarr: all ams-ix traffic is now on cr2-esams

October 7

  • 23:36 LeslieCarr: turning up new ams-ix port on cr2-esams
  • 23:24 LeslieCarr: traffic draining from ams-ix on cr2-knams
  • 23:22 LeslieCarr: moving european traffic around
  • 20:46 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Oct 7 20:46:05 UTC 2013
  • 20:33 logmsgbot: LocalisationUpdate completed (1.22wmf19) at Mon Oct 7 20:33:53 UTC 2013
  • 20:23 logmsgbot: LocalisationUpdate completed (1.22wmf20) at Mon Oct 7 20:23:45 UTC 2013
  • 20:05 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
  • 19:35 logmsgbot: reedy synchronized wmf-config/
  • 18:56 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commonswiki back to 1.22wmf20, thumb.php fixed
  • 18:55 logmsgbot: reedy synchronized php-1.22wmf20/thumb.php
  • 18:39 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commonswiki back to 1.22wmf19
  • 18:38 ottomata: backporting python-docopt 0.6.1 for precise and including in our apt repo
  • 18:24 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything non 'pedia to 1.22wmf20
  • 18:17 logmsgbot: reedy synchronized php-1.22wmf20/extensions/Wikibase
  • 17:43 paravoid: changed ms-fe1/2's mgmt IPs, ip clash with nas1-a/b's e0M
  • 17:28 mwalker: payments update from I3a09af2cbd8566bbeb431437c49e72c832b87304 to I61bc80f257c590c1abeb716145c23921b005e5a8
  • 16:05 logmsgbot: krinkle synchronized wmf-config/InitialiseSettings.php 'I59e2547002bac5'
  • 14:50 RoanKattouw: Restarting Parsoid on wtp10[01-24] on request; load avg reaching 90%
  • 14:28 Coren: Labstore4 Configuring; will bounce up and down over the next two days.
  • 12:57 akosiaris: brought icinga back up, after manually running puppet on neon
  • 12:39 hashar: gallium : restarting Zuul.
  • 12:37 hashar: gallium / Zuul : cherry picked a change from OpenStack related to statsd metrics. Our change: 88063
  • 11:30 mutante: new public mailing list for the Wikimania Steering Commitee - wikimania-com
  • 11:25 hashar: gallium : zuul restarted and apparently starting statsd metrics.
  • 11:03 hashar: gallium : killed a bunch of stalled jenkins/java threads
  • 11:02 hashar: gallium : restarting Zuul to enable statsd reporting 86744
  • 10:41 paravoid: rebooting labstore3, NFS lockupo
  • 10:18 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1021'
  • 10:10 apergos: powercycling srv291
  • 09:44 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'warm up db1015'
  • 03:19 springle: xtrabackup clone db43 to db1022
  • 02:04 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
  • 00:59 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1022 for upgrade'

October 6

  • 02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Oct 6 02:07:25 UTC 2013
  • 02:02 logmsgbot: LocalisationUpdate completed (1.22wmf20) at Sun Oct 6 02:02:50 UTC 2013
  • 02:02 logmsgbot: LocalisationUpdate completed (1.22wmf19) at Sun Oct 6 02:02:03 UTC 2013

October 5

  • 02:25 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Oct 5 02:25:46 UTC 2013
  • 02:19 logmsgbot: LocalisationUpdate completed (1.22wmf20) at Sat Oct 5 02:19:16 UTC 2013
  • 02:06 logmsgbot: LocalisationUpdate completed (1.22wmf19) at Sat Oct 5 02:06:38 UTC 2013

October 4

  • 18:44 paravoid: reinstalling erzurumi
  • 16:30 logmsgbot: maxsem Finished syncing Wikimedia installation... : https://gerrit.wikimedia.org/r/87584
  • 16:29 logmsgbot: aaron synchronized php-1.22wmf20/includes/filerepo/file/LocalFile.php 'b48403885d5ac993cb3b3ce7ac7580002ab12e1f'
  • 16:15 logmsgbot: maxsem Started syncing Wikimedia installation... : https://gerrit.wikimedia.org/r/87584
  • 15:35 mark: Added another 10G link to cr1-esams:ae1 <--> csw2-esams:ae1, now 40Gbps total
  • 15:03 mark: Completely disconnected csw1-esams from the network
  • 12:12 mark: Inserted 2x 10G MIC into cr1-esams FPC 0, brought online
  • 08:00 springle: upgrading db1039 to precise + mariadb
  • 07:54 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool db1042'
  • 07:11 apergos: powercycling maerlant, load > 100, couldn't log in via mgmt console
  • 07:09 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool db1035 after upgrade, db1003 to full steam'
  • 06:23 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool db1003 after upgrade'
  • 06:14 Tim: on fenari: running an analysis script for bug 53687 against tampa slave DB servers
  • 03:32 springle: manual sync-common on mw1072
  • 03:19 springle: started xtrabackup clone from db1035 to db1003
  • 02:42 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Oct 4 02:41:40 UTC 2013
  • 02:35 logmsgbot: LocalisationUpdate completed (1.22wmf20) at Fri Oct 4 02:35:00 UTC 2013
  • 02:31 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'refresh mw1072 db-eqiad.php'
  • 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf19) at Fri Oct 4 02:16:10 UTC 2013
  • 02:15 springle: upgrading db1003 to precise + mariadb
  • 02:08 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1003'
  • 01:32 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'db1038 to s3'
  • 01:28 logmsgbot: catrope synchronized php-1.22wmf20/extensions/VisualEditor/ 'Fix SyntaxHighlight icon'

October 3

  • 23:24 logmsgbot: catrope synchronized php-1.22wmf20/extensions/VisualEditor 'Update VisualEditor for cherry-picks'
  • 23:24 logmsgbot: catrope synchronized php-1.22wmf19/extensions/VisualEditor 'Update VisualEditor for cherry-picks'
  • 22:50 paravoid: starting swiftrepl process for pmtpa->eqiad (originals), running on copper
  • 22:50 logmsgbot: mflaschen Finished syncing Wikimedia installation... : Deploy GettingStarted for growth team
  • 22:39 logmsgbot: mflaschen Started syncing Wikimedia installation... : Deploy GettingStarted for growth team
  • 22:38 awight_: updated crm from cf90889b3728e1bfb2175574640a18511d60e2b9 to b4392c76d21287f835076f761436fc7386845a19
  • 21:59 LeslieCarr: restarting pybal on lvs1003 (not 1006)
  • 21:59 LeslieCarr: restarting pybal on lvs1006
  • 21:59 LeslieCarr: restarting pybal on lvs1006
  • 21:45 awight: updated crm from 7690ee9f80c6e53d946195e453cf17e197f3138a to cf90889b3728e1bfb2175574640a18511d60e2b9
  • 21:25 awight: updated civicrm 6affe8a9cee57f76602a25d60d96edc7eaaae4b2 to 7690ee9f80c6e53d946195e453cf17e197f3138a
  • 20:42 logmsgbot: krinkle synchronized php-1.22wmf19/extensions/VisualEditor/modules/ve/ve.Element.js 'touch for bug 54935'
  • 20:27 logmsgbot: krinkle synchronized php-1.22wmf19/resources/startup.js 'Attempt to fix bug 54935'
  • 19:32 logmsgbot: reedy synchronized php-1.22wmf20/extensions/MassMessage/
  • 19:12 ori-l: CentralAuth: UPDATE `bug_54847_password_resets` SET `r_logged_out` = 1 WHERE `r_reset` IS NOT NULL;
  • 18:39 logmsgbot: reedy synchronized php-1.22wmf20/includes/DefaultSettings.php
  • 18:34 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki, test2wiki, mediawikiwiki, loginwiki and testwikidatawiki to 1.22wmf20
  • 18:28 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.22wmf19
  • 17:45 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php
  • 17:19 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.22wmf19 till deploy time
  • 17:13 logmsgbot: reedy synchronized php-1.22wmf20/extensions/VisualEditor
  • 17:09 logmsgbot: reedy Finished syncing Wikimedia installation... : testwiki to 1.22wmf20 and build l10n cache
  • 16:48 logmsgbot: reedy Started syncing Wikimedia installation... : testwiki to 1.22wmf20 and build l10n cache
  • 16:41 logmsgbot: reedy synchronized docroot and w
  • 16:40 logmsgbot: reedy synchronized php-1.22wmf20 'Staging'
  • 16:14 bblack: updated varnish to -wm17 on active mobile caches (cp1046, cp1047, cp1059, cp1060, cp3011, cp3012), fixes 0.0.0.0 assertfail in netmapper
  • 13:47 Jeff_Green: c state disabled on barium, ref. RT 5555
  • 12:14 manybubbles: enwikisource's search index finished building late last night. all english indecies have been updated to include kstem.
  • 08:02 ori-l: Bug 54847: e-mail script completed successfully
  • 06:14 logmsgbot: olivneh synchronized wmf-config/Bug54847.php
  • 05:49 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'if ( $wmgBug54847 && $wmgUseCentralAuth )'
  • 05:47 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'wmgBug54847: default => true, private => false'
  • 05:41 logmsgbot: csteipp Finished syncing Wikimedia installation... :
  • 05:40 logmsgbot: olivneh synchronized wmf-config/Bug54847.php
  • 05:36 Tim: on searchidx2: switched java procs to "idle" ionice class, to improve scap time
  • 04:57 logmsgbot: csteipp Started syncing Wikimedia installation... :
  • 02:59 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Oct 3 02:59:41 UTC 2013
  • 02:35 logmsgbot: LocalisationUpdate completed (1.22wmf18) at Thu Oct 3 02:35:56 UTC 2013
  • 02:20 logmsgbot: LocalisationUpdate completed (1.22wmf19) at Thu Oct 3 02:20:48 UTC 2013
  • 02:11 Tim: on mw1125: server very slow due to disk issues, stopped apache, will shut down
  • 00:34 manybubbles: all english indecies rebuilt except enwikisource. that one will take a while.
  • 00:11 manybubbles1: rebuilding search search indecies on english wikis after CirrusSearch deploy that updated English configuration

October 2

  • 23:16 logmsgbot: demon synchronized php-1.22wmf19/extensions/CirrusSearch 'Cirrus to master'
  • 23:15 logmsgbot: demon synchronized php-1.22wmf18/extensions/CirrusSearch 'Cirrus to master'
  • 22:56 logmsgbot: csteipp synchronized php-1.22wmf18/includes/specials/SpecialUserlogin.php
  • 22:44 logmsgbot: csteipp synchronized php-1.22wmf19/includes/specials/SpecialUserlogin.php 'AbortLogin message fix'
  • 21:28 gwicke: updated Parsoid to 01ebbfd
  • 21:20 gwicke: updated parsoid dependencies
  • 14:25 mutante: sync-apache, apache-graceful-all for wikimania2014 redirect
  • 10:30 MaxSem: Resynched MW on mw1125, looked like slightly out of sync
  • 05:15 superm401: Commons has intermittent search failures: "An error has occurred while searching: HTTP request timed out."
  • 02:56 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Oct 2 02:56:05 UTC 2013
  • 02:30 logmsgbot: LocalisationUpdate completed (1.22wmf18) at Wed Oct 2 02:30:23 UTC 2013
  • 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf19) at Wed Oct 2 02:16:19 UTC 2013

October 1

  • 23:24 logmsgbot: catrope synchronized php-1.22wmf19/extensions/VisualEditor 'Update VisualEditor for cherry-pick of I7bddcfa9'
  • 23:23 logmsgbot: catrope synchronized php-1.22wmf18/extensions/VisualEditor 'Update VisualEditor for cherry-pick of I7bddcfa9'
  • 22:55 LeslieCarr: added search to pybal eqiad backends
  • 21:54 logmsgbot: maxsem synchronized php-1.22wmf19/extensions/MobileFrontend/
  • 21:52 logmsgbot: maxsem synchronized php-1.22wmf18/extensions/MobileFrontend/
  • 21:26 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
  • 21:12 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c/87008/'
  • 21:03 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
  • 20:55 mwalker: last update for SSL seemed to go ok; now updating payments for payment methods from 1cce4e1bcd45171289594ddcf0165a049c3264a1 to fdc9c2b3e3898ac3e7dc738dc6f4ed140b3ddb41
  • 20:48 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
  • 20:46 mwalker: updating payments from b952e9e9656d62ee8b3a70e96ae5eb45918068ce to 1cce4e1bcd45171289594ddcf0165a049c3264a1
  • 18:21 ottomata: backported jq_1.2-8 to precise and imported into apt
  • 15:22 Jeff_Green: dist-upgrade and reboot loudon
  • 15:16 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Enable CirrusSearch extra splitting for mw.org.'
  • 13:12 springle: start xtrabackup s3 db1035 to db1038
  • 09:58 logmsgbot: maxsem synchronized docroot and w
  • 09:50 logmsgbot: yurik synchronized php-1.22wmf19/extensions/ZeroRatedMobileAccess/
  • 09:48 logmsgbot: yurik synchronized php-1.22wmf18/extensions/ZeroRatedMobileAccess/
  • 09:23 mutante: mw1173 had wrong nrpe init script, synced the one from mw1172 and restarted (as nagios), fixes dpkg check
  • 09:11 mutante: killing nrpe on cp1045 and restarting, was running as icinga instead of nagios, fixes "NRPE: Unable to read output" Icinga Unknown's
  • 08:31 hashar: Jenkins: so pyflakes are now actually doing something useful instead of testing nothing. That causes a bunch of jobs to fail
  • 08:28 hashar: jenkins updating pyflakes jobs to run 'pyflakes .' instead of simply 'pyflakes' 86727
  • 02:50 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Oct 1 02:50:54 UTC 2013
  • 02:26 logmsgbot: LocalisationUpdate completed (1.22wmf18) at Tue Oct 1 02:25:59 UTC 2013
  • 02:13 logmsgbot: LocalisationUpdate completed (1.22wmf19) at Tue Oct 1 02:13:45 UTC 2013
  • 00:32 logmsgbot: mwalker synchronized php-1.22wmf19/extensions/CentralNotice/ 're pushing centralnotice to master on the CORRECT branch... grumble'
  • 00:16 mwalker: added china to list of paypal xclick countries -- no longer using donation workflow -- for ChinaUnionPay

September 30

  • 23:52 logmsgbot: mwalker synchronized php-1.22wmf18/extensions/CentralNotice/ 'Reverting previous centralnotice change since it appears to have made the bug worse...'
  • 23:44 logmsgbot: mwalker synchronized php-1.22wmf18/extensions/CentralNotice/ 'Updating CentralNotice to master'
  • 23:07 mwalker: updating payments from 78cbf0d4d2024cff02d4f57ff21c1ffb478f16ee to b952e9e9656d62ee8b3a70e96ae5eb45918068ce for adyen, paypal xclick, and globalcollect cvv
  • 22:41 logmsgbot: reedy Finished syncing Wikimedia installation... : MassMessage
  • 22:36 LeslieCarr: changing local pref for 54058 to downpref tinet directly
  • 22:20 logmsgbot: reedy Started syncing Wikimedia installation... : MassMessage
  • 21:59 logmsgbot: reedy synchronized php-1.22wmf19/extensions/MassMessage 'Staging code'
  • 20:41 RobH: db1014 is coming offline to become tungsten, please disregard any alerts
  • 20:28 Jeff_Green: samarium dist-upgrade and reboot
  • 18:23 mwalker: updating payments listeners (smashpig) from f73e931e8a30c8dbc30899918f4344b34e777d03 to 243ec9ca483993386aa99a10f872c8afa097a548
  • 18:20 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.22wmf19
  • 17:55 awight: civicrm updated from c086eb74eca0ebe513d564abbe2b4b62c0608fe9 to 6affe8a9cee57f76602a25d60d96edc7eaaae4b2
  • 17:46 awight: civicrm updated from 1f659b73fdd to c086eb74ec
  • 16:16 logmsgbot: catrope synchronized php-1.22wmf19/extensions/UniversalLanguageSelector 'bug fixes'
  • 16:16 logmsgbot: catrope synchronized php-1.22wmf19/extensions/VisualEditor 'bug fixes'
  • 16:16 logmsgbot: catrope synchronized php-1.22wmf18/extensions/VisualEditor 'bug fixes'
  • 16:01 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable VE by default for logged-in users on bg, ca, ceb, cs, da, el, et, eu, fi, gl, hr, hu, id, lv, ms, nap, nn, no, simple, sk, sl, tr, uk, vec, vo, war Wikipedias'
  • 15:30 mutante: sync-apache mw1041 and restart apache, was missing in apaches dsh group
  • 13:57 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'REDOING (forgot to rebase): wiktionary logos form 135px to 128px so they fit in the 160px tall element. Ping Nemo_bis and Dereckson.'
  • 12:43 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'wiktionary logos form 135px to 128px so they fit in the 160px tall element. Ping Nemo_bis and Dereckson.'
  • 11:51 logmsgbot: springle synchronized wmf-config/AdminSettings.php 'switch admin account back with new password'
  • 10:14 logmsgbot: springle synchronized wmf-config/AdminSettings.php 'switch admin account, attempt #2'
  • 10:06 logmsgbot: springle synchronized wmf-config/AdminSettings.php 'revert admin account. errors'
  • 10:03 logmsgbot: springle synchronized wmf-config/AdminSettings.php 'switch to temp admin db account'
  • 09:41 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'Point ptwiktionary logo to commons 85516 bug 53685'
  • 02:45 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Sep 30 02:45:10 UTC 2013
  • 02:27 logmsgbot: LocalisationUpdate completed (1.22wmf19) at Mon Sep 30 02:27:03 UTC 2013
  • 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf18) at Mon Sep 30 02:15:09 UTC 2013
  • 01:48 springle: upgrading db1038 to precise + mariadb
  • 01:04 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool db1049'

September 29

  • 23:56 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1049'
  • 23:20 springle: powercycling db1049, soft lockup, unresponsive
  • 20:51 paravoid: swift@pmtpa: pushing rings to set 10.0.6.204/sdm1 10.0.6.204/sdn1 10.0.6.208/sde1 weight from 33 to 66
  • 02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Sep 29 02:07:37 UTC 2013
  • 02:02 logmsgbot: LocalisationUpdate completed (1.22wmf19) at Sun Sep 29 02:02:22 UTC 2013
  • 02:01 logmsgbot: LocalisationUpdate completed (1.22wmf18) at Sun Sep 29 02:01:40 UTC 2013

September 28

  • 02:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Sep 28 02:08:41 UTC 2013
  • 02:02 logmsgbot: LocalisationUpdate completed (1.22wmf19) at Sat Sep 28 02:02:32 UTC 2013
  • 02:01 logmsgbot: LocalisationUpdate completed (1.22wmf18) at Sat Sep 28 02:01:52 UTC 2013

September 27

  • 23:55 ori-l: Manually synchronized changes 86329, 86330, 86331 to hafnium & professor & temporarily disabled puppet to prevent it from clobbering the changes. I'll re-enable once they're merged.
  • 22:35 logmsgbot: olivneh synchronized php-1.22wmf19/extensions/NavigationTiming 'Updating NavigationTiming: schema update 5336845 -> 5832704 (2/2)'
  • 22:35 logmsgbot: olivneh synchronized php-1.22wmf18/extensions/NavigationTiming 'Updating NavigationTiming: schema update 5336845 -> 5832704 (1/2)'
  • 21:25 logmsgbot: reedy synchronized php-1.22wmf19/extensions/WikimediaMessages/
  • 20:57 logmsgbot: ariel synchronized wmf-config/InitialiseSettings.php 'fix up ns refs for ukwikimedia'
  • 20:30 logmsgbot: ariel synchronized wmf-config/InitialiseSettings.php 'ukwikimedia to read-only except for user talk'
  • 19:18 ori-l: restarting apache on professor (graphite)
  • 19:13 logmsgbot: reedy synchronized php-1.22wmf19/extensions/TorBlock/
  • 18:18 logmsgbot: reedy synchronized php-1.22wmf19/includes/GlobalFunctions.php 'Revert'
  • 18:04 logmsgbot: reedy synchronized php-1.22wmf19/includes/GlobalFunctions.php 'all of the debugging'
  • 17:53 logmsgbot: reedy synchronized php-1.22wmf19/includes/GlobalFunctions.php 'all of the debugging'
  • 17:42 logmsgbot: reedy synchronized php-1.22wmf19/includes/GlobalFunctions.php 'all of the debugging'
  • 17:39 logmsgbot: reedy synchronized php-1.22wmf19/includes/GlobalFunctions.php 'all of the debugging'
  • 17:31 logmsgbot: reedy synchronized php-1.22wmf19/includes/GlobalFunctions.php 'all of the debugging'
  • 17:27 cmjohnson1: powering down analytics1021 replacing disk1 (sdb)
  • 16:38 cmjohnson1: mw1072 replacing hard drive
  • 13:02 paravoid: dismantling & repurposing ceph cluster
  • 02:55 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Sep 27 02:55:41 UTC 2013
  • 02:33 logmsgbot: LocalisationUpdate completed (1.22wmf19) at Fri Sep 27 02:33:19 UTC 2013
  • 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf18) at Fri Sep 27 02:14:34 UTC 2013

September 26

  • 23:27 ^d: elastic: reindex all articles that erroneously had <audio> tags indexed
  • 23:11 Ryan_Lane: deploying DNS change to OpenStackManager on virt0, also doing DIT change for all public/private DNS entries in labs
  • 23:01 mwalker: updating fundraising crm from bf4c7090099a866389105a3a38be4f950dd78b89 to 1f659b73fddd26939081d21c3ff4098a77589e61
  • 22:56 logmsgbot: yurik synchronized php-1.22wmf19/extensions/ZeroRatedMobileAccess/
  • 22:46 logmsgbot: yurik synchronized php-1.22wmf18/extensions/ZeroRatedMobileAccess/
  • 22:25 ori-l: installed package python-argparse on professor (lucid) to get git-deploy's 'grain-ensure' to run.
  • 21:49 awight: updated crm from 8180a18a7cb9a4bf64f5a8a7e192ca6f8f910a0a to bf4c7090099a866389105a3a38be4f950dd78b89
  • 21:41 awight: update crm from 1c10b4b181925d882 to 8180a18a7cb9a4bf64f5
  • 21:29 logmsgbot: mflaschen synchronized php-1.22wmf19/extensions/GuidedTour/modules/mediawiki.libs.guiders/mediawiki.libs.guiders.css 'Sync GuidedTour file for Growth team deployment'
  • 21:28 logmsgbot: mflaschen synchronized php-1.22wmf18/extensions/GuidedTour/modules/mediawiki.libs.guiders/mediawiki.libs.guiders.css 'Sync GuidedTour file for Growth team deployment'
  • 19:48 logmsgbot: reedy synchronized php-1.22wmf19/extensions/GlobalBlocking/
  • 19:39 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: loginwiki back to 1.22wmf19
  • 19:34 logmsgbot: reedy Finished syncing Wikimedia installation... : revisionsize
  • 19:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: put loginwiki back to 1.22wmf18
  • 19:23 logmsgbot: reedy Started syncing Wikimedia installation... : revisionsize
  • 19:17 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki, test2wiki, testwikidatawiki, loginwiki and mediawikiwiki to 1.22wmf19
  • 19:09 logmsgbot: reedy synchronized php-1.22wmf19/includes/
  • 18:53 logmsgbot: reedy synchronized php-1.22wmf19/extensions/TorBlock
  • 18:52 logmsgbot: reedy synchronized php-1.22wmf19/extensions/GlobalBlocking
  • 18:49 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: revert 1.22wmf19 wikis
  • 18:40 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki, test2wiki, testwikidatawiki, loginwiki and mediawikiwiki to 1.22wmf19
  • 18:15 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.22wmf18
  • 18:14 mwalker: updating fr crm from aa79ff40f3ce1534d899c21c81dfb914caad3fae to 1c10b4b181925d882ada0024011e28f75b781390
  • 18:14 cmjohnson1: db1006 replacing disk 1
  • 18:08 cmjohnson1: db1004 removing/replacing disk 2
  • 17:39 logmsgbot: reedy Finished syncing Wikimedia installation... : testwiki to 1.22wmf19 and build l10n cache
  • 17:25 logmsgbot: reedy Started syncing Wikimedia installation... : testwiki to 1.22wmf19 and build l10n cache
  • 17:17 logmsgbot: reedy synchronized docroot and w
  • 17:15 logmsgbot: reedy synchronized php-1.22wmf19 'Initial sync v2'
  • 17:13 logmsgbot: reedy synchronized php-1.22wmf19 'Initial sync'
  • 16:10 hashar: Jenkins: deleted /tmp/mw-UIDGenerator-UID-* files on gallium.
  • 15:32 mark: Migrated OSPF between cr1-ulsfo and cr2-eqiad to p2p interface-type, removed high metric, TEMP annotation
  • 15:28 mark: Setup OSPFv3 between cr1-ulsfo and cr2-eqiad
  • 02:45 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Sep 26 02:45:41 UTC 2013
  • 02:25 logmsgbot: LocalisationUpdate completed (1.22wmf17) at Thu Sep 26 02:25:14 UTC 2013
  • 02:13 logmsgbot: LocalisationUpdate completed (1.22wmf18) at Thu Sep 26 02:13:22 UTC 2013
  • 02:09 springle: testing MHA on pmtpa db37, db56, db58, db68. disabled their notifications'

September 25

  • 23:32 logmsgbot: krinkle Finished syncing Wikimedia installation... :
  • 23:21 logmsgbot: krinkle Started syncing Wikimedia installation... :
  • 23:17 logmsgbot: krinkle synchronized php-1.22wmf17/extensions/VisualEditor 'I3653efbe7b6841c70f352da526f4946fa0a4c76f'
  • 23:12 logmsgbot: krinkle synchronized php-1.22wmf18/extensions/VisualEditor 'I3653efbe7b6841c70f352da526f4946fa0a4c76f'
  • 23:10 logmsgbot: krinkle synchronized php-1.22wmf18/extensions/TemplateData 'I3653efbe7b6841c70f352da526f4946fa0a4c76f'
  • 20:31 logmsgbot: csteipp synchronized php-1.22wmf17/extensions/CentralAuth 'update to master for SUL fix'
  • 20:28 logmsgbot: csteipp synchronized php-1.22wmf18/extensions/CentralAuth 'update to master for SUL fix'
  • 19:13 cmjohnson: authdns update
  • 16:52 ottomata: taking down analytics100[34] for reinstall with public IPs
  • 16:51 mutante: install a few package upgrades on iron
  • 16:24 mutante: added a favicon.ico on stats.wikimedia.org for hashar
  • 16:11 ottomata: authdns-update to give analytics100[34] public IPs, about to reinstall them
  • 15:21 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 13:43 mutante: apt-get install php5-cli on mw1046 fixes 'ensure: change from 5.3.10-1ubuntu3.8 to 5.3.10-1ubuntu3.6+wmf1 failed' issue after reinstall
  • 13:22 mutante: re-enabling disabled puppet on (fixed) mw1046
  • 11:04 mutante: running checksetup.pl on bugzilla, installed InlineHistory extension
  • 09:35 6JTAABPF7: Put all *.wikimedia.org, wikidata, wikivoyage eqiad traffic on Varnish
  • 05:12 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1035 for upgrade'
  • 05:05 logmsgbot: tstarling cleared profiling data
  • 03:47 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool db1018'
  • 02:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Sep 25 02:48:19 UTC 2013
  • 02:25 logmsgbot: LocalisationUpdate completed (1.22wmf17) at Wed Sep 25 02:25:30 UTC 2013
  • 02:13 logmsgbot: LocalisationUpdate completed (1.22wmf18) at Wed Sep 25 02:13:00 UTC 2013
  • 01:40 Tim: restarted wikibugs
  • 00:55 springle: started xtrabackup clone db1002 to db1018

September 24

  • 23:45 springle: upgrading db1018 to precise + mariadb
  • 23:23 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1018 for upgrade, pool db1002 and warm up'
  • 22:41 logmsgbot: maxsem synchronized php-1.22wmf18/extensions/MobileFrontend/
  • 22:39 logmsgbot: maxsem synchronized php-1.22wmf17/extensions/MobileFrontend/
  • 22:09 logmsgbot: maxsem synchronized php-1.22wmf18/extensions/MobileFrontend/
  • 22:07 logmsgbot: maxsem synchronized php-1.22wmf17/extensions/MobileFrontend/
  • 21:59 logmsgbot: maxsem synchronized php-1.22wmf18/extensions/MobileFrontend/
  • 21:57 logmsgbot: maxsem synchronized php-1.22wmf17/extensions/MobileFrontend/
  • 21:48 LeslieCarr: routing changes in ulsfo, commit confirmed 5 for safety
  • 21:45 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
  • 21:34 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
  • 19:35 logmsgbot: reedy synchronized wmf-config 'touch'
  • 19:19 logmsgbot: bsitu synchronized echowikis.dblist 'Enable Echo and Thanks on various wikis'
  • 19:07 logmsgbot: bsitu synchronized echowikis.dblist 'Enable Echo and Thanks on various wikis'
  • 19:06 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable Echo and Thanks on various wikis'
  • 18:55 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo and Thanks to master
  • 18:54 cmjohnson: apache-graceful on mw1046
  • 18:53 bblack: varnish updated to -wm16 on all eqiad mobile caches now, yell at me if something starts looking "off" there...
  • 18:53 cmjohnson: sync'ing mw1046
  • 18:42 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo and Thanks to master
  • 18:09 bblack: upgraded varnish pkg to -wm16 on cp1046
  • 17:59 RobH: bast4001 reinstall
  • 17:58 manybubbles: finished building search indexes on closed wikis
  • 17:49 LeslieCarr: Hurricane electric peering moved to 10GE port. the spice packets must flow...
  • 17:14 MaxSem: Recreating GeoData index
  • 16:38 manybubbles: rebuilt search indecies in place for all other wikis runnig cirrus except cawiki and enwikisource. skipped them because they are already up to date.
  • 16:27 cmjohnson: adding mw1046 back to pybal
  • 16:24 manybubbles: rebuilding search indecies in place for all other wikis running cirrus
  • 16:19 mutante: DNS update, add wikipediastories domains
  • 16:18 mutante: sync-apache, graceful for wikipediastories
  • 16:16 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable CleanChanges on meta'
  • 16:10 manybubbles: building search indecies for all closed wikis
  • 16:06 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php 'Cirrus on closed wikis'
  • 15:25 logmsgbot: demon synchronized php-1.22wmf18/extensions/CirrusSearch
  • 15:24 mutante: freshly activated http://wikimediastories.org/ (and .com,.net)
  • 15:21 mutante: sync-apache and graceful-all for wikimediastories and other cleanup
  • 15:21 logmsgbot: yurik synchronized php-1.22wmf18/extensions/ZeroRatedMobileAccess/
  • 15:13 mutante: during sync-apache: mw1072 - read-only file system
  • 15:07 cmjohnson: db1053 removing/replacing disk 3
  • 15:05 logmsgbot: yurik synchronized php-1.22wmf17/extensions/ZeroRatedMobileAccess/
  • 14:59 cmjohnson: db1001 removing/replacing disk 7
  • 14:48 cmjohnson: db1031 removing/replacing disk 7
  • 14:38 cmjohnson: replacing mw1046 hard drive --requires reinstall
  • 14:21 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Sep 24 14:21:10 UTC 2013
  • 14:15 logmsgbot: LocalisationUpdate completed (1.22wmf17) at Tue Sep 24 14:15:56 UTC 2013
  • 14:15 logmsgbot: LocalisationUpdate completed (1.22wmf18) at Tue Sep 24 14:15:16 UTC 2013
  • 14:13 logmsgbot: demon synchronized php-1.22wmf18/extensions/Elastica 'Sync message fixes'
  • 12:26 mutante: shutting down sq45
  • 11:04 mutante: powercycled sq45
  • 10:08 p858snake|l: [20:01AEST] <mark> upped to weight 5
  • 10:08 p858snake|l: [19:51AEST] <mark> !log Pooled Varnish Text caches in eqiad with weight 1 (affecting wikidata, wikivoyage, *.wikimedia.org)
  • 10:07 p858snake|l: [19:36AEST] <akosiaris> !log rebooted labstore4, controller configuration lost ?
  • 10:07 p858snake|l: [19:36AEST] <akosiaris> !log rebooted labstore3 (thought the 14days NFS issue was resolved???)
  • 10:04 akosiaris: rebooted labstore4, controller configuration lost ?
  • 10:04 akosiaris: rebooted labstore3 (thought the 14days NFS issue was resolved???)
  • 08:30 hashar: restarting Jenkins again for one last plugin upgrade
  • 08:30 springle: labstore4 disk issues, /a failed mount in console, ssh key bouncing. awaiting input from ryan or coren
  • 08:30 hashar: 8:20am INFO: Jenkins is fully up and running
  • 08:20 hashar: restarting Jenkins
  • 08:12 hashar: jenkins: upgrading plugins
  • 08:12 hashar: 8:12am INFO: Jenkins is fully up and running
  • 08:06 springle: labstore4 locked up, power cycled
  • 07:43 hashar: upgrading Jenkins on gallium from 1.509.2 to 1.509.3
  • 06:36 springle: upgrading db1002 to precise and mariadb
  • 06:13 ^demon|zzz: ES: enwikisource & cawiki done indexing. Really need to revisit batch indexing, way too slow.
  • 03:42 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1002 for upgrade, db1036 take up slack'
  • 03:17 logmsgbot: springle synchronized wmf-config/db-eqiad.php
  • 02:40 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Sep 24 02:40:38 UTC 2013
  • 02:21 logmsgbot: LocalisationUpdate completed (1.22wmf17) at Tue Sep 24 02:21:27 UTC 2013
  • 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf18) at Tue Sep 24 02:15:01 UTC 2013

September 23

  • 23:36 RoanKattouw: mw1072 is unhappy, sync-file throws warnings about read-only filesystem
  • 23:35 logmsgbot: catrope synchronized php-1.22wmf18/extensions/VisualEditor 'VisualEditor cherry-picks'
  • 23:35 logmsgbot: catrope synchronized php-1.22wmf17/extensions/VisualEditor 'VisualEditor cherry-picks'
  • 23:20 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Set VisualEditor to opt-in on enwiki'
  • 20:57 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php
  • 20:48 logmsgbot: reedy synchronized wmf-config/ 'touch'
  • 20:46 logmsgbot: reedy synchronized php-1.22wmf18/extensions/UploadWizard
  • 20:09 Reedy: mw1072 has a r/o filesystem
  • 20:05 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Cirrus as primary for itwikt, secondary on cawiki and enwikisource'
  • 19:46 logmsgbot: reedy synchronized wmf-config 'touch'
  • 19:36 logmsgbot: reedy synchronized database lists files:
  • 19:35 logmsgbot: reedy synchronized wmf-config/
  • 18:50 Ryan_Lane: enabling cirrussearch for wikitech
  • 18:43 Reedy: Created sites and site_identifier tables on commonswiki
  • 18:41 Ryan_Lane: upgrading wikitech-static to 1.22wmf18
  • 18:41 Ryan_Lane: upgraded wikitech to 1.22wmf18

September 22

  • 02:27 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Sep 22 02:27:32 UTC 2013
  • 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf18) at Sun Sep 22 02:14:05 UTC 2013
  • 02:07 logmsgbot: LocalisationUpdate completed (1.22wmf17) at Sun Sep 22 02:07:46 UTC 2013

September 21

  • 18:42 hashar: Jenkins: upgrading pep8 from 1.4.5 to 1.4.6 on gallium and lanthanum
  • 03:21 Reedy: Scalers fixed
  • 03:16 logmsgbot: reedy synchronized docroot and w
  • 03:07 hashar: imagescalers in eqiad shows a drop of CPU/network at around 1:40am UTC. Thumbnails are no more rendered :-(
  • 02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Sep 21 02:07:36 UTC 2013
  • 02:02 logmsgbot: LocalisationUpdate completed (1.22wmf18) at Sat Sep 21 02:02:26 UTC 2013
  • 02:01 logmsgbot: LocalisationUpdate completed (1.22wmf17) at Sat Sep 21 02:01:44 UTC 2013
  • 01:38 logmsgbot: reedy synchronized docroot and w
  • 00:53 logmsgbot: reedy synchronized wmf-config/throttle.php
  • 00:45 logmsgbot: reedy synchronized docroot and w
  • 00:43 logmsgbot: reedy synchronized docroot and w
  • 00:41 logmsgbot: reedy synchronized docroot and w
  • 00:40 logmsgbot: reedy synchronized docroot and w
  • 00:38 logmsgbot: reedy synchronized docroot and w
  • 00:23 logmsgbot: reedy synchronized w
  • 00:13 logmsgbot: reedy synchronized w
  • 00:09 logmsgbot: reedy synchronized w

September 20

  • 23:53 logmsgbot: reedy synchronized w/404.php
  • 23:03 springle: upgrading db1044 to precise
  • 22:20 springle: parallel dump db1033 to db1036 with innodb_file_per_table=1
  • 22:10 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Sep 20 22:10:50 UTC 2013
  • 22:06 logmsgbot: LocalisationUpdate completed (1.22wmf18) at Fri Sep 20 22:06:08 UTC 2013
  • 22:05 logmsgbot: LocalisationUpdate completed (1.22wmf17) at Fri Sep 20 22:05:21 UTC 2013
  • 22:02 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Sep 20 22:02:35 UTC 2013
  • 21:47 logmsgbot: LocalisationUpdate completed (1.22wmf18) at Fri Sep 20 21:47:58 UTC 2013
  • 21:26 logmsgbot: LocalisationUpdate completed (1.22wmf17) at Fri Sep 20 21:26:27 UTC 2013
  • 21:08 logmsgbot: demon synchronized php-1.22wmf18/extensions/Elastica
  • 21:06 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 21:05 logmsgbot: demon synchronized wmf-config
  • 21:05 logmsgbot: demon synchronized php-1.22wmf18/extensions/CirrusSearch
  • 21:04 logmsgbot: demon synchronized php-1.22wmf18/extensions/Elastica
  • 20:46 logmsgbot: reedy synchronized php-1.22wmf18/extensions/Wikibase
  • 20:45 logmsgbot: reedy synchronized php-1.22wmf18/extensions/WikimediaMessages
  • 20:23 logmsgbot: reedy synchronized php-1.22wmf18/extensions/WikimediaMessages
  • 20:23 logmsgbot: reedy synchronized php-1.22wmf18/extensions/Wikibase
  • 19:20 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php
  • 19:14 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Cirrus on officewiki'
  • 18:52 logmsgbot: reedy synchronized multiversion/
  • 18:51 logmsgbot: reedy synchronized docroot and w
  • 18:49 logmsgbot: reedy synchronized wmf-config/
  • 17:41 logmsgbot: reedy synchronized wmf-config/ 'touch'
  • 17:39 logmsgbot: reedy synchronized database lists files:
  • 17:37 Reedy: Created Echo tables on officewiki
  • 15:58 hashar: gerrit: deleting reference refs/for/master from pywikibot/core.git , confuses some git clients
  • 13:39 hashar: Jenkins: PHPUnit jobs for MediaWiki extensions had an issue for the last couple days. They were being fetched in an incorrect place :/ Fix is https://gerrit.wikimedia.org/r/85202 and got deployed
  • 06:31 logmsgbot: reedy synchronized docroot/bits/
  • 02:44 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Sep 20 02:44:51 UTC 2013
  • 02:24 logmsgbot: LocalisationUpdate completed (1.22wmf18) at Fri Sep 20 02:24:44 UTC 2013
  • 02:12 logmsgbot: LocalisationUpdate completed (1.22wmf17) at Fri Sep 20 02:12:46 UTC 2013
  • 00:46 cmjohnson: rebooting sq45
  • 00:17 springle: db1018 set log_warnings=2 to investigate mw-log/dberror.log frequent "Can't connect to MySQL server on '10.64.16.7'"
  • 00:06 logmsgbot: reedy synchronized images/sul/wikivoyage.png

September 19

  • 22:40 logmsgbot: mflaschen Finished syncing Wikimedia installation... : Growth team deployment: GuidedTour and remove outdated config from CommonSettings.php and InitialiseSettings.php
  • 22:22 logmsgbot: mflaschen Started syncing Wikimedia installation... : Growth team deployment: GuidedTour and remove outdated config from CommonSettings.php and InitialiseSettings.php
  • 22:01 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Sep 19 22:01:13 UTC 2013
  • 21:46 logmsgbot: LocalisationUpdate completed (1.22wmf18) at Thu Sep 19 21:46:29 UTC 2013
  • 21:15 logmsgbot: LocalisationUpdate completed (1.22wmf17) at Thu Sep 19 21:15:17 UTC 2013
  • 20:53 hashar: Jenkins: slave integration-debian-builder.pmtpa.wmflabs now uses /mnt/jenkins-workspace as filesystem root, to avoid relying on GlusterFS.
  • 20:49 manybubbles: rebuilt all search indecies. \o/
  • 20:44 hashar: Jenkins: upgrading jenkins-debian-glue packages on integration-debian-builder.pmtpa.wmflabs
  • 19:57 manybubbles: rebuilding search indecies after push of CirrusSearch
  • 19:03 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki, mediawikiwiki, testwikidatawiki and loginwiki to 1.22wmf18
  • 19:03 logmsgbot: demon synchronized php-1.22wmf18/extensions/CirrusSearch
  • 19:02 logmsgbot: demon synchronized php-1.22wmf17/extensions/CirrusSearch
  • 19:00 logmsgbot: reedy synchronized php-1.22wmf18/extensions/TemplateData
  • 18:46 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki, mediawikiwiki, testwikidatawiki and loginwiki to 1.22wmf17
  • 18:44 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki, mediawikiwiki, testwikidatawiki and loginwiki to 1.22wmf18
  • 18:42 logmsgbot: reedy synchronized php-1.22wmf17/extensions/
  • 18:41 logmsgbot: reedy synchronized php-1.22wmf18/extensions/
  • 18:34 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.22wmf17
  • 18:32 logmsgbot: reedy synchronized php-1.22wmf18/extensions/UniversalLanguageSelector
  • 18:28 logmsgbot: reedy Finished syncing Wikimedia installation... : testwiki to 1.22wmf18 and build l10n cache
  • 18:15 hashar: Jenkins: applying label 'hasContintPackages' to lanthanum, that should make some pep8/pyflakes jobs to be executed on that host
  • 18:12 logmsgbot: reedy Started syncing Wikimedia installation... : testwiki to 1.22wmf18 and build l10n cache
  • 18:10 logmsgbot: reedy synchronized docroot and w
  • 18:07 logmsgbot: reedy synchronized docroot and w
  • 18:05 logmsgbot: reedy synchronized php-1.22wmf18
  • 17:03 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Cirrus as primary'
  • 16:44 ^d: ES: indexes created for testwikidatawiki
  • 16:44 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'testwikidatawiki gets Cirrus'
  • 03:35 hashar: Zuul: updated ssh fingerprint of ytterbium for jenkins user on gallium. That is needed for zuul to clone new repositories.
  • 03:05 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Sep 19 03:05:39 UTC 2013
  • 02:42 logmsgbot: LocalisationUpdate completed (1.22wmf16) at Thu Sep 19 02:42:09 UTC 2013
  • 02:22 logmsgbot: LocalisationUpdate completed (1.22wmf17) at Thu Sep 19 02:22:05 UTC 2013
  • 01:34 hashar: jenkins: migrating test extensions jobs to fetch Zuul ref over git protocol. Tie them to label 'hasPhpUnit' as well since it is only installed on gallium ..
  • 01:32 hashar: jenkins: updating qunit jobs to fetch Zuul ref over git protocol
  • 00:56 RobH: iron coming offline
  • 00:39 hashar: Jenkins: migrating phpcs-HEAD mediawiki extension jobs to use the git protocol when fetching Zuul refs.

September 18

  • 23:51 hashar: Jenkins: migrated mediawiki extension 'lint' jobs to fetch Zuul reference over git:// protocol. Migrating jslint ones.
  • 23:44 hashar: Jenkins applied labels "hasPhpcs" and "hasPhpUnit" to gallium slave
  • 23:09 logmsgbot: catrope synchronized php-1.22wmf17/extensions/VisualEditor/ 'Deploy cherry-picked bug fixes'
  • 22:13 logmsgbot: yurik synchronized php-1.22wmf17/extensions/ZeroRatedMobileAccess/
  • 22:10 logmsgbot: yurik synchronized php-1.22wmf16/extensions/ZeroRatedMobileAccess/
  • 21:15 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable EducationProgram on enwikinews'
  • 21:14 Reedy: Created EducationProgram tables on cswiki
  • 21:10 manybubbles: greg-g showed me how to log stuff
  • 21:10 manybubbles: Upgraded elasticsearch to 0.90.4.
  • 21:09 greg-g: manybubbles told his mom he loved her
  • 21:06 Reedy: Created EducationProgram tables on enwikinews
  • 20:54 logmsgbot: reedy synchronized wmf-config/
  • 20:38 springle: cloning from sanitarium to labsdb1003 instances
  • 20:15 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'ULS EventLogging'
  • 20:10 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'ULS EventLogging'
  • 20:04 hashar: Jenkins: migrated sartoris jobs to fetch Zuul reference over the git protocol made available with [[bugzilla:53683|bug 53683]]. Test change is https://gerrit.wikimedia.org/r/#/c/46450/
  • 19:51 springle: stopping mysqld instances on labsdb1003 for troubleshooting after xfs crash/corruption
  • 19:13 logmsgbot: mwalker synchronized php-1.22wmf17/extensions/CentralNotice/ 'Updating CentralNotice to master'
  • 19:12 logmsgbot: mwalker synchronized php-1.22wmf16/extensions/CentralNotice/ 'Updating CentralNotice to master'
  • 18:44 logmsgbot: csteipp synchronized php-1.22wmf16/extensions/CentralAuth 'update centralauth to master for SUL fixes'
  • 18:44 logmsgbot: csteipp synchronized php-1.22wmf17/extensions/CentralAuth 'update centralauth to master for SUL fixes'
  • 18:14 logmsgbot: reedy synchronized php-1.22wmf17/extensions/TimedMediaHandler
  • 05:10 logmsgbot: reedy synchronized docroot and w
  • 04:05 logmsgbot: reedy synchronized docroot and w
  • 03:54 logmsgbot: reedy synchronized docroot and w
  • 03:00 logmsgbot: reedy synchronized docroot and w
  • 02:51 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Sep 18 02:51:31 UTC 2013
  • 02:28 logmsgbot: LocalisationUpdate completed (1.22wmf16) at Wed Sep 18 02:28:37 UTC 2013
  • 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf17) at Wed Sep 18 02:15:07 UTC 2013
  • 02:15 logmsgbot: reedy synchronized wmf-config/ 'Switch Echo to using dblist for enablement criteria'
  • 02:14 logmsgbot: reedy synchronized database lists files:
  • 02:13 logmsgbot: reedy synchronized docroot/noc/
  • 00:47 mutante: sync-apache, repool mw1044
  • 00:37 logmsgbot: reedy synchronized multiversion/MWMultiVersion.php
  • 00:33 logmsgbot: mwalker synchronized php-1.22wmf17/extensions/CentralNotice 'Updating CentralNotice to master'
  • 00:32 logmsgbot: mwalker synchronized php-1.22wmf16/extensions/CentralNotice 'Updating CentralNotice to master'
  • 00:32 cmjohnson1: dns update
  • 00:30 springle: upgrade, reboot, mariadb db1033
  • 00:29 springle: returned db1050 to the pool
  • 00:21 logmsgbot: springle synchronized wmf-config/db-eqiad.php

September 17

  • 23:49 LeslieCarr: upgrading cp1062 to fix broken packages
  • 23:41 logmsgbot: reedy synchronized multiversion/MWMultiVersion.php
  • 23:26 logmsgbot: lcarr synchronized wmf-config/throttle.php
  • 23:18 RobH: updated blogs, still online \o/
  • 23:13 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Re-enable experimental VisualEditor features on mediawiki.org'
  • 22:20 logmsgbot: mwalker synchronized php-1.22wmf17/extensions/CentralNotice 'Updating CentralNotice to master'
  • 22:19 logmsgbot: mwalker synchronized php-1.22wmf16/extensions/CentralNotice 'Updating CentralNotice to master'
  • 21:51 logmsgbot: kaldari synchronized php-1.22wmf16/extensions/MobileFrontend/javascripts/common 'syncing JS files for mobilefrontend'
  • 21:36 logmsgbot: kaldari Finished syncing Wikimedia installation... :
  • 21:19 ottomata: varnishkafka on cp1048 producing snappy compressed json as of Sept 17 20:42
  • 20:59 logmsgbot: kaldari Started syncing Wikimedia installation... :
  • 18:48 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'enable echo and thanks on bs, nl, et, fa, he, ro, es, simple wiki and enwikivoyage'
  • 18:48 logmsgbot: bsitu synchronized echowikis.dblist 'add bs, nl, et, fa, he, ro, es, simple wiki and enwikivoyage to echodblist'
  • 18:38 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo and Thanks to master
  • 18:30 mutante: depooling mw1044
  • 18:30 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo and Thanks to master
  • 18:21 LeslieCarr: depreffing more peers in order to save the ams-ix port
  • 18:00 springle: db1036 upgrade, reboot, mariadb
  • 17:25 logmsgbot: reedy synchronized php-1.22wmf17/extensions/CirrusSearch/
  • 17:14 springle: depooled db1050 while cloning db1045 delayed slave
  • 17:13 logmsgbot: springle synchronized wmf-config/db-eqiad.php
  • 16:59 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild l10ncache to include Math stuffs
  • 16:36 paravoid: rebooting neon, ext3 errors (corruption?) remounted ro
  • 16:31 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild l10ncache to include Math stuffs
  • 16:03 LeslieCarr: deactivated a few peers on ams-ix in order to reduce traffic on congested port
  • 08:43 logmsgbot: yurik synchronized php-1.22wmf17/extensions/ZeroRatedMobileAccess/
  • 08:41 logmsgbot: yurik synchronized php-1.22wmf16/extensions/ZeroRatedMobileAccess/
  • 05:37 logmsgbot: springle synchronized wmf-config/db-eqiad.php
  • 05:13 springle: deployed db1037 as s1 slave
  • 05:13 logmsgbot: springle synchronized wmf-config/db-pmtpa.php
  • 05:13 logmsgbot: springle synchronized wmf-config/db-eqiad.php
  • 02:50 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Sep 17 02:50:53 UTC 2013
  • 02:28 logmsgbot: LocalisationUpdate completed (1.22wmf16) at Tue Sep 17 02:28:14 UTC 2013
  • 02:25 springle: starting xtrabackup clone for s1 from db1017 to db1037
  • 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf17) at Tue Sep 17 02:14:55 UTC 2013
  • 01:21 Reedy: Changed password for global user "Article Feedback v5" and logged into enwiki to create an account
  • 00:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wmgArticleFeedbackv5AutoArchiveEnabled to false for enwiki'

September 16

  • 23:46 mutante: change timeout for watchmouse "wikimedia foundation script"-check from 8 sec to 15 sec
  • 23:03 logmsgbot: yurik synchronized php-1.22wmf17/extensions/ZeroRatedMobileAccess/
  • 23:00 logmsgbot: yurik synchronized php-1.22wmf16/extensions/ZeroRatedMobileAccess/
  • 22:48 ^d: ES: indexing itwiktionary from terbium
  • 22:43 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php
  • 21:09 logmsgbot: demon synchronized php-1.22wmf17/extensions/CirrusSearch
  • 21:08 logmsgbot: demon synchronized php-1.22wmf17/extensions/CirrusSearch
  • 20:23 logmsgbot: demon synchronized php-1.22wmf17/extensions/CirrusSearch
  • 20:12 mutante: re-enable allow_attachment_display on bugzilla
  • 19:38 mark: Disabled csw1-esams:e8/1
  • 19:05 logmsgbot: reedy synchronized php-1.22wmf17/extensions/VisualEditor
  • 18:38 logmsgbot: reedy synchronized php-1.22wmf16/extensions/VisualEditor
  • 18:31 logmsgbot: reedy synchronized php-1.22wmf17/extensions/DataValues/
  • 18:30 logmsgbot: reedy synchronized php-1.22wmf17/extensions/Wikibase
  • 18:27 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything non 'pedia to 1.22wmf17
  • 18:17 mutante: custom logo for el.wikivoyage
  • 18:17 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
  • 18:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikimedia, special, wikinews and wikivoyage to 1.22wmf16
  • 18:16 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
  • 18:07 ori-l: Starting import of some old EL data into the 'log' database on db1047.
  • 17:10 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php
  • 17:07 logmsgbot: demon synchronized php-1.22wmf17/extensions/CirrusSearch
  • 17:02 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php
  • 17:00 logmsgbot: demon synchronized php-1.22wmf17/extensions/CirrusSearch/CirrusSearchSearcher.php 'bug 54167'
  • 16:13 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php
  • 07:40 springle: rebooted neon. icinga pegged at max concurrent connections for 48h. root ext3 fs read-only as of 2013-09-14 06:45 detected aborted journal
  • 02:23 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Sep 16 02:23:57 UTC 2013
  • 02:11 logmsgbot: LocalisationUpdate completed (1.22wmf17) at Mon Sep 16 02:11:46 UTC 2013
  • 02:06 logmsgbot: LocalisationUpdate completed (1.22wmf16) at Mon Sep 16 02:06:34 UTC 2013

September 15

  • 02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Sep 15 02:07:26 UTC 2013
  • 02:02 logmsgbot: LocalisationUpdate completed (1.22wmf17) at Sun Sep 15 02:02:16 UTC 2013
  • 02:01 logmsgbot: LocalisationUpdate completed (1.22wmf16) at Sun Sep 15 02:01:36 UTC 2013

September 14

  • 23:34 springle: throttling back db1045 pt-table-sync jobs to avoid blocking source-slave sql threads
  • 23:25 springle_away: restarted db1049 slave sql thread after lock wait timeout. investigating cause
  • 02:21 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Sep 14 02:21:42 UTC 2013
  • 02:13 logmsgbot: LocalisationUpdate completed (1.22wmf17) at Sat Sep 14 02:13:11 UTC 2013
  • 02:05 logmsgbot: LocalisationUpdate completed (1.22wmf16) at Sat Sep 14 02:05:31 UTC 2013

September 13

  • 17:48 apergos: testing incr dump code on snapshot1003 in screen; if it falls over it won't break anything
  • 15:38 greg-g: ugh, it's intermittent, 503's now
  • 15:33 greg-g: etherpad.wikimedia seems better to me
  • 15:28 Nemo_bis: etherpad currently serving error 503 Service Temporarily Unavailable
  • 02:54 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Sep 13 02:54:31 UTC 2013
  • 02:33 logmsgbot: LocalisationUpdate completed (1.22wmf17) at Fri Sep 13 02:33:49 UTC 2013
  • 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf16) at Fri Sep 13 02:15:07 UTC 2013

September 12

  • 21:50 LeslieCarr: upgrading to 11.4r9 on cr1-ulsfo/cr2-ulsfo
  • 21:29 logmsgbot: reedy synchronized wmf-config/
  • 21:29 logmsgbot: reedy Finished syncing Wikimedia installation... : Update l10n cache for CleanChanges
  • 21:28 logmsgbot: reedy synchronized wmf-config/
  • 21:19 logmsgbot: reedy Started syncing Wikimedia installation... : Update l10n cache for CleanChanges
  • 21:15 logmsgbot: reedy synchronized wmf-config/extension-list 'Enable CleanChanges'
  • 20:49 logmsgbot: reedy synchronized php-1.22wmf16/extensions/MoodBar/
  • 19:35 mark: Rebalanced eqiad apaches in PyBal weight, according to machine specs
  • 18:59 LeslieCarr: restarting pybal on lvs's
  • 18:57 LeslieCarr: runnign ntpdate on lvs servers
  • 18:33 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: loginwiki, mediawikiwiki, test2wiki and testwikidatawiki to 1.22wmf17
  • 18:28 logmsgbot: reedy Finished syncing Wikimedia installation... : testwiki to 1.22wmf17 and build localisation cache
  • 18:11 logmsgbot: reedy Started syncing Wikimedia installation... : testwiki to 1.22wmf17 and build localisation cache
  • 18:02 logmsgbot: reedy synchronized php-1.22wmf16/extensions/Wikibase
  • 17:56 logmsgbot: reedy synchronized docroot and w
  • 17:55 logmsgbot: reedy synchronized php-1.22wmf17 'Initial deploy of code files'
  • 17:19 hashar: gallium : killed some long lasting java Jenkins thread
  • 17:18 logmsgbot: reedy synchronized php-1.22wmf16/extensions/Wikibase
  • 17:16 logmsgbot: reedy synchronized php-1.22wmf16/extensions/DataValues/
  • 04:18 logmsgbot: reedy synchronized php-1.22wmf16/extensions/TemplateData
  • 04:00 logmsgbot: catrope synchronized php-1.22wmf16/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.Target.js 'touch'
  • 03:49 logmsgbot: reedy synchronized php-1.22wmf16/extensions/TemplateData/TemplateData.php
  • 02:51 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Sep 12 02:51:54 UTC 2013
  • 02:43 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.22wmf16
  • 02:32 logmsgbot: LocalisationUpdate completed (1.22wmf15) at Thu Sep 12 02:32:02 UTC 2013
  • 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf16) at Thu Sep 12 02:16:32 UTC 2013

September 11

  • 23:22 paravoid: powercycling neon, swapdeath
  • 23:10 logmsgbot: mwalker synchronized wmf-config/CommonSettings.php 'Do not run Translate message group job synchronously for CentralNotice'
  • 22:50 logmsgbot: mwalker synchronized php-1.22wmf15/extensions/CentralNotice 'Updating CentralNotice to master'
  • 22:50 logmsgbot: mwalker synchronized php-1.22wmf16/extensions/CentralNotice 'Updating CentralNotice to master'
  • 22:48 springle: upgraded packages on db1020 to get ganglia-monitor mysql metrics working
  • 22:44 logmsgbot: mwalker synchronized php-1.22wmf16/extensions/Translate 'Updating Translate to master'
  • 22:43 logmsgbot: mwalker synchronized php-1.22wmf15/extensions/Translate 'Updating Translate to master'
  • 22:35 logmsgbot: reedy synchronized wmf-config/
  • 21:42 logmsgbot: yurik synchronized php-1.22wmf15/extensions/ZeroRatedMobileAccess/
  • 21:39 logmsgbot: yurik synchronized php-1.22wmf16/extensions/ZeroRatedMobileAccess/
  • 21:18 logmsgbot: springle synchronized wmf-config/db-eqiad.php
  • 21:04 LeslieCarr: rolled back
  • 21:01 LeslieCarr: network maintenance in tampa
  • 20:58 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 20:14 pp-pdf1: restarted all services
  • 20:14 pp-pdf1: - update mwlib to 0.15.11
  • 20:14 pp-pdf1: - update mwlib.epub to 0.14.3
  • 20:00 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 19:33 logmsgbot: reedy synchronized php-1.22wmf16/extensions/CleanChanges 'Stage'
  • 19:30 logmsgbot: reedy synchronized wmf-config/
  • 19:10 LeslieCarr: more network changes, network disruptions unlikely but possible for tampa
  • 18:25 LeslieCarr: moving all external traffic to come in via eqiad
  • 18:10 LeslieCarr: doing some network maintenance on cr2-pmtpa
  • 17:57 RobH: forgot to log testsearch1003 is back
  • 17:49 RobH: testsearch1002 back up, 1003 down
  • 17:39 RobH: testsearch1001 back in pool, 1002 rebooting for fix
  • 17:21 RobH: rebooting testsearch1001 to troubleshoot
  • 17:10 logmsgbot: aaron synchronized wmf-config/filebackend.php
  • 17:02 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Removed ceph backend config for decommissioning'
  • 16:39 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Cirrus as default on mw.org, Lucene as secondary'
  • 10:18 apergos: set php memory limit to 256 on nickel, ori reducd the number of metrics for statsd (see https://gerrit.wikimedia.org/r/#/c/83777/), restarted all of apache, gmond, gmetad and still see the occasional php Allowed memory size exhausted error
  • 02:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Sep 11 02:08:28 UTC 2013
  • 02:02 logmsgbot: LocalisationUpdate completed (1.22wmf15) at Wed Sep 11 02:02:32 UTC 2013
  • 02:01 logmsgbot: LocalisationUpdate completed (1.22wmf16) at Wed Sep 11 02:01:46 UTC 2013

September 10

  • 23:46 LeslieCarr: rebooted neon, had crashed
  • 23:33 logmsgbot: dzahn synchronized ./wmf-config/throttle.php
  • 23:32 logmsgbot: dzahn synchronized ./wmf-config/throttle.php
  • 23:28 springle: starting mariadb 10 multi-source replication test from all clusters on db1045
  • 22:24 logmsgbot: reedy synchronized wmf-config/
  • 22:15 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikivoyage and wikidatawiki to 1.22wmf16
  • 22:12 Jeff_Green: dist-upgrade and reboot boron
  • 22:09 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Sep 10 22:08:58 UTC 2013
  • 22:07 logmsgbot: reedy synchronized wmf-config/
  • 22:04 logmsgbot: reedy synchronized php-1.22wmf16/extensions/Wikibase
  • 21:50 logmsgbot: LocalisationUpdate completed (1.22wmf15) at Tue Sep 10 21:50:32 UTC 2013
  • 21:50 logmsgbot: reedy synchronized php-1.22wmf16/includes/search/SearchEngine.php
  • 21:33 mutante: temp. hacked a fork limit into apache-graceful-all so my local ssh agent doesn't die when it's too fast, bleh:p
  • 21:26 logmsgbot: LocalisationUpdate completed (1.22wmf16) at Tue Sep 10 21:26:17 UTC 2013
  • 21:03 logmsgbot: csteipp synchronized php-1.22wmf15/extensions/CentralAuth
  • 20:56 logmsgbot: demon synchronized php-1.22wmf16/extensions/CirrusSearch 'Logging for annoying parser output missing bug'
  • 20:55 LeslieCarr: setting AMSIX Local pref to 110 due to overloading. high risk
  • 20:42 logmsgbot: csteipp synchronized php-1.22wmf15/extensions/CentralAuth
  • 20:42 logmsgbot: csteipp synchronized php-1.22wmf16/extensions/CentralAuth
  • 18:43 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable ContributionReporting'
  • 18:11 logmsgbot: reedy synchronized php-1.22wmf15/extensions/Wikibase
  • 18:10 logmsgbot: reedy synchronized php-1.22wmf15/extensions/WikibaseDataModel
  • 18:09 logmsgbot: reedy synchronized php-1.22wmf14/extensions/Wikibase
  • 18:09 logmsgbot: reedy synchronized php-1.22wmf14/extensions/WikibaseDataModel
  • 17:52 cmjohnson1: removing sq42 from pybal
  • 17:20 cmjohnson1: rebooting sq42
  • 17:05 apergos: ganglia was failign to serve some urls, complaining that memory size of 134217728 bytes exhausted, I upped to 192mb temporarily in /etc/php5/apache2/php.ini, needs followup
  • 16:45 logmsgbot: reedy synchronized wmf-config/interwiki.cdb 'Updating interwiki cache'
  • 16:24 apergos: on labstore3, the various mount lines for /dev/mapper/store-xx are commented out in fstab from aug 15, so naturally there's o nfs etc, not sure if it's ok to just put 'em back in, need someone to take a look-see
  • 16:12 apergos: rebooting labstore3 after restart of nfsd did not do the trick
  • 12:23 Nemo_bis: labstore3 overloaded, beta & tools &co. down
  • 12:21 apergos: rebooted neon, it was out to lunch
  • 01:07 awight: crm updated from 75d31d3443e4 to aa79ff4
  • 00:40 mutante: restarting gmetad on nickel
  • 00:27 LeslieCarr: restarted gmetad on nickel
  • 00:21 logmsgbot: csteipp synchronized php-1.22wmf15/extensions/CentralAuth 'Syncing correct submodule checkout of centralauth'

September 9

  • 23:54 logmsgbot: csteipp synchronized wmf-config/CommonSettings.php
  • 23:52 logmsgbot: mwalker Finished syncing Wikimedia installation... : Scapping to update CentralNotice i18n strings
  • 23:50 mutante: restarted statsd on hafnium
  • 23:47 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Sep 9 23:47:10 UTC 2013
  • 23:38 logmsgbot: mwalker Started syncing Wikimedia installation... : Scapping to update CentralNotice i18n strings
  • 23:28 logmsgbot: LocalisationUpdate completed (1.22wmf15) at Mon Sep 9 23:28:18 UTC 2013
  • 23:09 logmsgbot: LocalisationUpdate completed (1.22wmf16) at Mon Sep 9 23:09:05 UTC 2013
  • 23:03 mutante: deployed localized logo for he.wikivoyage
  • 23:02 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
  • 22:37 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
  • 22:36 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
  • 22:10 RobH: blog updated to wp 3.6 and no one noticed, thats good upgradin folks
  • 22:08 mutante: running update-special-pages on hume
  • 21:54 LeslieCarr: upgrading mw121*
  • 21:46 LeslieCarr: upgrading mw120*
  • 21:44 logmsgbot: reedy synchronized php-1.22wmf16/extensions/EducationProgram/
  • 21:38 LeslieCarr: upgrading mw119*
  • 21:21 hashar: lanthanum.eqiad.wmnet upgrading packages
  • 21:19 hashar: gallium upgrading PHP packages.
  • 21:18 logmsgbot: nikerabbit synchronized wmf-config/CommonSettings.php 'ULS eventlogging'
  • 21:17 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'ULS eventlogging'
  • 20:52 LeslieCarr: upgrading mw118*
  • 20:45 LeslieCarr: upgrading mw117*
  • 20:43 LeslieCarr: upgrading asw-ulsfo
  • 20:40 LeslieCarr: upgrading mw116*
  • 20:33 LeslieCarr: upgrading mw115*
  • 20:29 logmsgbot: mwalker synchronized php-1.22wmf16/extensions/CentralNotice 'Updating CentralNotice for custom banner classes'
  • 20:28 logmsgbot: mwalker synchronized php-1.22wmf15/extensions/CentralNotice 'Updating CentralNotice for custom banner classes'
  • 20:25 LeslieCarr: upgrading mw114*
  • 20:21 paravoid: upgrading gdnsd to 1.10
  • 20:19 LeslieCarr: upgrading mw113* servers
  • 20:18 bblack: bumped vhtcpd to 0.0.10 on brewster (conn close bugfix)
  • 20:08 LeslieCarr: upgrading mw112*
  • 20:04 springle: fixed broken wikiadmin grants on es100[2-4]
  • 20:01 LeslieCarr: upgrading mw111*
  • 18:57 logmsgbot: reedy synchronized php-1.22wmf16
  • 18:56 logmsgbot: reedy synchronized php-1.22wmf15
  • 18:51 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else that isn't wikipedia, wikivoyage or wikidatawiki to 1.22wmf16
  • 18:39 Ryan_Lane: repooling ssl3003
  • 18:30 Ryan_Lane: repooling ssl3001/3002 and depooling ssl3003
  • 18:26 LeslieCarr: upgrading mw1000-mw1009
  • 18:25 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity, wikibooks and wikisource to 1.22wmf16
  • 18:22 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: various misc wikis to 1.22wmf16
  • 18:19 Ryan_Lane: depooling ssl3001-2
  • 18:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: various misc wikis to 1.22wmf16
  • 18:11 Ryan_Lane: repooling ssl1006-9
  • 18:04 Ryan_Lane: repooling ssl1002-5 and depooling ssl1006-9
  • 17:54 Ryan_Lane: pooling ssl1001 and depooling ssl1002-5
  • 17:44 Ryan_Lane: depooling ssl1001
  • 17:43 Ryan_Lane: adding GCM cipher and removing DES from SSL config
  • 02:01 logmsgbot: LocalisationUpdate failed: git pull of extensions failed

September 8

  • 02:01 logmsgbot: LocalisationUpdate failed: git pull of extensions failed

September 7

  • 05:23 apergos: gitblit was returning errors for some urls with java.lang.NullPointerException, stopped and restarted it, seems ok now
  • 04:01 ori-l: git.wikimedia.org: HTTP 500; Varnish XID 196670102
  • 02:01 logmsgbot: LocalisationUpdate failed: git pull of extensions failed

September 6

  • 22:29 mutante: sync-common-file InitSettings, deploy change 83086
  • 22:29 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
  • 20:06 Ryan_Lane: rebooting virt11
  • 18:34 gwicke: updated Parsoid to ef1776732
  • 18:12 ^d: puppet & gerrit service stopped on manganese so nobody hits them anymore
  • 02:33 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
  • 02:02 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
  • 00:16 mutante: restarting irc bot on ekrem

September 5

  • 23:31 mutante: installing package upgrades on ekrem
  • 23:16 logmsgbot: bsitu synchronized php-1.22wmf15/extensions/PageTriage 'Update PageTriage to master'
  • 23:15 logmsgbot: bsitu synchronized php-1.22wmf16/extensions/PageTriage 'Update PageTriage to master'
  • 22:45 logmsgbot: csteipp synchronized php-1.22wmf16/extensions/CentralNotice 'bug53032'
  • 22:44 logmsgbot: csteipp synchronized php-1.22wmf16/includes/specials
  • 22:43 logmsgbot: csteipp synchronized php-1.22wmf16/includes/actions
  • 22:16 logmsgbot: spage synchronized wmf-config/InitialiseSettings.php 'E3 deploy GuidedTour to 7 more wikis'
  • 22:11 logmsgbot: spage Finished syncing Wikimedia installation... : Deploying latest GuidedTour to wmf15 and wmf16
  • 21:59 logmsgbot: spage Started syncing Wikimedia installation... : Deploying latest GuidedTour to wmf15 and wmf16
  • 21:57 mutante: rolled back SSL cert on blog for now to discuss techblog.wm issue which would not be covered by cert but is just a redirect
  • 21:40 mutante: replaced SSL cert on blog.wikimedia.org, don't use star cert, have separate new one
  • 21:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 20:51 logmsgbot: reedy synchronized php-1.22wmf15/extensions/ArticleFeedbackv5/
  • 20:34 logmsgbot: reedy synchronized php-1.22wmf16/extensions/ArticleFeedbackv5/
  • 19:10 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: all wikipedias to 1.22wmf15
  • 19:01 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki, mediawikiwiki, test2wiki, loginwiki and testwikidatawiki to 1.22wmf16
  • 18:58 ^d: gerrit back up
  • 18:47 ^d: gerrit coming down for a short bit. moving data around.
  • 18:42 logmsgbot: reedy synchronized php-1.22wmf16/extensions/UploadWizard/
  • 18:27 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Change I34d1f29d2: clean-up of InitialiseSettings.php'
  • 18:26 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Increase rate of NavigationTiming events'
  • 18:03 Ryan_Lane: adding virt12 to the virt node pool
  • 17:22 akosiaris: upload kafka 0.8-20130903-1 at apt.wikimedia.org, component universe
  • 16:41 cmjohnson1: powering down sq36 to swap disk
  • 16:26 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.22wmf15 till deployment window
  • 16:23 logmsgbot: reedy synchronized php-1.22wmf16/extensions/ 'Minor wikibase updates'
  • 16:10 logmsgbot: reedy Finished syncing Wikimedia installation... : testwiki to 1.22wmf16 and build l10n cache
  • 15:51 logmsgbot: reedy Started syncing Wikimedia installation... : testwiki to 1.22wmf16 and build l10n cache
  • 15:33 Nemo_bis: etherpad.wikimedia.org down (error 503)
  • 15:28 cmjohnson1: replacing disk slot 0 ms-be1011
  • 15:18 logmsgbot: reedy synchronized docroot and w
  • 15:17 logmsgbot: reedy synchronized php-1.22wmf16 'Initial file sync'
  • 14:12 paravoid: recreating mediawiki cgroup lost after cgroup-bin upgrade; thumbnail issues fixed
  • 13:14 paravoid: adjusting depool-threshold, restarting pybal on lvs1003
  • 12:13 paravoid: pushing new swift object/account/container rings: setting ms-be9's sde1 to 33, swapping ms-be5's sda/b <-> sdm/n, also 33
  • 10:21 apergos: lanthanum bios processor settings set to 'custom' and c states disabled (attempt to fix power limit notification issue)
  • 09:57 springle: leaving sync_binlog=0 on db36 and db38 so replication keeps up. to be reviewed
  • 09:05 hashar_: jenkins: killed two requesthandler threads that were eating CPU power.
  • 02:47 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Sep 5 02:47:15 UTC 2013
  • 02:30 logmsgbot: LocalisationUpdate completed (1.22wmf14) at Thu Sep 5 02:30:03 UTC 2013
  • 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf15) at Thu Sep 5 02:15:59 UTC 2013
  • 01:42 ^d: gallium: updated git remotes in /srv/ssd/zuul/git/* to not point to old manganese box and point to gerrit.wm.o instead

September 4

  • 23:17 ^d: restarting gerrit one last time
  • 23:04 ^d: apache stopped on manganese
  • 23:01 ^d: stopped puppet on manganese
  • 21:23 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
  • 21:10 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
  • 19:41 Ryan_Lane: upgraded wiktech-static to 1.22wmf15
  • 19:16 andrewbogott: turning on pluginsync, which will (temporarily) make puppet even slower
  • 19:10 andrewbogott: Moved morebots from wikitech-static to toollabs
  • 06:28 apergos: truncated fact_values puppet table and reset auto increment to start at 1, puppet was broken on all hosts, see http://projects.puppetlabs.com/issues/9225
  • 02:23 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Sep 4 02:21:58 UTC 2013
  • 02:13 logmsgbot: LocalisationUpdate completed (1.22wmf14) at Wed Sep 4 02:11:48 UTC 2013
  • 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf15) at Wed Sep 4 02:06:47 UTC 2013
  • 00:51 logmsgbot: olivneh synchronized php-1.22wmf15/includes/RecentChange.php 'Fix for bug 53720'

September 3

  • 22:47 Ryan_Lane: upgrading wikitech to 1.22wmf15
  • 19:47 logmsgbot: reedy synchronized wmf-config/ 'touch'
  • 19:38 logmsgbot: reedy synchronized php-1.22wmf15/extensions/UploadWizard/resources 'touch'
  • 19:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything non 'pedia to 1.22wmf15
  • 18:54 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special, wikimedia, closed and private wikis to 1.22wmf15
  • 18:48 awight: civicrm updated 5c28984d7e4 to 75d31d3443e45c
  • 18:39 mwalker: optimizing civicrm.civicrm_email on db1025 (db1008)
  • 18:18 mwalker: altering db1025 (db1008) civicrm.civicrm_email --> "ALTER TABLE civicrm_email MODIFY COLUMN email varchar(255);"
  • 18:01 gwicke: updated Parsoid to 859a701
  • 15:23 hashar: gallium running mw phpunit code coverage under gdb in a tmux.
  • 14:29 hashar: gallium: dpkg -i libapache2-mod-php5_5.3.10-1ubuntu3.7+wmf1_amd64.deb php5-cli_5.3.10-1ubuntu3.7+wmf1_amd64.deb php5-common_5.3.10-1ubuntu3.7+wmf1_amd64.deb php5-curl_5.3.10-1ubuntu3.7+wmf1_amd64.deb php5-dbg_5.3.10-1ubuntu3.7+wmf1_amd64.deb php5-dev_5.3.10-1ubuntu3.7+wmf1_amd64.deb php5-gd_5.3.10-1ubuntu3.7+wmf1_amd64.deb php5-intl_5.3.10-1ubuntu3.7+wmf1_amd64.deb php5-mysql_5.3.10-1ubuntu3.7+wmf1_amd64.deb
  • 14:28 hashar: upgrading PHP5 on gallium (contint server)
  • 02:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Sep 3 02:25:20 UTC 2013
  • 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf15) at Tue Sep 3 02:14:15 UTC 2013
  • 02:09 logmsgbot: LocalisationUpdate completed (1.22wmf14) at Tue Sep 3 02:08:03 UTC 2013

September 2

  • 16:21 hashar: restarted Zuul (paranoia)
  • 16:15 Reedy: Running cleanupTitles.php in screen on terbium
  • 16:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 14:36 hashar: Jenkins: restoring mediawiki-core jobs to previous state
  • 14:32 hashar: broken Jenkins whenever a huge repository is being used (i.e. mediawiki/core or operations/puppet )
  • 13:43 akosiaris: aptitude hold puppet puppet-common puppetmaster puppetmaster-common puppetmaster-passenger on stafford till we solve the problem with CPU spiking present in puppet 2.7.11-1ubuntu2.4
  • 13:36 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'reutrn db1027 to the pool'
  • 06:22 springle: granted process/show/usage to watchdog on db masters
  • 04:34 springle: granted process/show/usage to watchdog on db masters
  • 02:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Sep 2 02:25:02 UTC 2013
  • 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf15) at Mon Sep 2 02:13:50 UTC 2013
  • 02:09 logmsgbot: LocalisationUpdate completed (1.22wmf14) at Mon Sep 2 02:07:46 UTC 2013
  • 00:50 logmsgbot: faidon synchronized wmf-config/InitialiseSettings.php 'new wikivoyage logo'
  • 00:50 logmsgbot: faidon synchronized docroot/bits/favicon/wikivoyage.ico 'new wikivoyage favicon'

September 1

  • 23:59 springle: running pt-table-sync over db1027 wikis
  • 21:30 springle: restarted db1027 mysqld. leaving port 3306 blocked while replication recovers
  • 20:53 paravoid: powercycled db1027, BUG: soft lockup - CPU#14 stuck for 22s! [xfsaild/dm-0:933]
  • 20:31 paravoid: restarting gerrit
  • 17:58 paravoid: killed snapshot1004 workers, flooding dberror with wikiadmin auth failures + WikiExporter::dumpFrom slow queries (10' ago)
  • 17:54 logmsgbot: faidon cleared profiling data
  • 17:06 logmsgbot: maxsem synchronized /php-1.22wmf14/includes/api/ApiQueryImageInfo.php 'Not the cause, restoring back'
  • 16:55 logmsgbot: maxsem synchronized /php-1.22wmf14/includes/api/ApiQueryImageInfo.php 'Trying to disable this...'
  • 02:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Sep 1 02:07:18 UTC 2013
  • 02:04 logmsgbot: LocalisationUpdate completed (1.22wmf15) at Sun Sep 1 02:02:30 UTC 2013
  • 02:03 logmsgbot: LocalisationUpdate completed (1.22wmf14) at Sun Sep 1 02:01:46 UTC 2013

August 31

  • 23:49 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depooling db1027'
  • 02:27 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Aug 31 02:26:20 UTC 2013
  • 02:18 logmsgbot: LocalisationUpdate completed (1.22wmf15) at Sat Aug 31 02:16:41 UTC 2013
  • 02:09 logmsgbot: LocalisationUpdate completed (1.22wmf14) at Sat Aug 31 02:08:01 UTC 2013

August 30

  • 22:21 mutante: after re-pooling search1015 a while ago, now temp. disable search1016 while indices are rebuilt for new wikis
  • 21:49 mutante: on sodium: Class[Backup::Host] is already defined ...
  • 21:46 mutante: temp. disabling search1015 in pybal
  • 21:29 mutante: dumping/importing viwikivoyage and tyvwiki lucene search indices
  • 21:24 mutante: restarting lucene search indexers on searchidx2 and searchidx1001
  • 20:24 logmsgbot: csteipp synchronized php-1.22wmf15/extensions/LiquidThreads 'bug53320'
  • 18:01 mutante: powercycling sq36
  • 15:27 hashar: Jenkins: removing labels hasSlaveScripts hasContintPackages productionSlaves from lanthanum slaves, it caught some jobs
  • 14:39 hashar: jenkins: lanthanum slave now has 8 executors and labels hasSlaveScripts hasContintPackages productionSlaves
  • 10:48 mark: reverted
  • 10:36 mark: Moving esams text squid backend traffic from pmtpa to eqiad
  • 08:47 hashar: restarting Jenkins for plugins upgrade
  • 05:05 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Make Flickr API URL protocol-relative for compatibility with HTTPS'
  • 02:55 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Aug 30 02:53:46 UTC 2013
  • 02:39 logmsgbot: LocalisationUpdate completed (1.22wmf15) at Fri Aug 30 02:38:00 UTC 2013
  • 02:20 logmsgbot: LocalisationUpdate completed (1.22wmf14) at Fri Aug 30 02:18:27 UTC 2013
  • 00:19 Ryan_Lane: rebooting sq36
  • 00:18 Ryan_Lane: switching esams from using pmtpa to using eqiad for carp

August 29

  • 23:52 gwicke: updated Parsoid to b26a506
  • 23:37 Ryan_Lane: removing sq33 from squid configuration, it was decommissioned a while ago
  • 21:56 mutante: upgrading apparmor packages on iron
  • 21:32 mutante: powercycled sq36 which was frozen but probably all unrelated to API outage
  • 20:59 logmsgbot: olivneh synchronized php-1.22wmf15/extensions/NavigationTiming 'Updating NavigationTiming to master (2/2)'
  • 20:58 logmsgbot: olivneh synchronized php-1.22wmf14/extensions/NavigationTiming 'Updating NavigationTiming to master (1/2)'
  • 20:57 logmsgbot: mflaschen synchronized php-1.22wmf14/extensions/GuidedTour/ 'Deploy GuidedTour to 1.22wmf14'
  • 20:55 logmsgbot: olivneh synchronized php-1.22wmf15/includes/RecentChange.php 'I4570473ae'
  • 20:52 logmsgbot: reedy synchronized php-1.22wmf14/extensions/Wikibase
  • 20:52 logmsgbot: reedy synchronized php-1.22wmf15/extensions/Wikibase
  • 20:33 logmsgbot: olivneh synchronized php-1.22wmf15/extensions/NavigationTiming 'Updating NavigationTiming to master (2/2)'
  • 20:32 logmsgbot: olivneh synchronized php-1.22wmf14/extensions/NavigationTiming 'Updating NavigationTiming to master (1/2)'
  • 19:54 Aaron|home: Running PurgeDeletedFiles everywhere to bring ceph/swift more in sync with each other and the DB; copy scripts already finished
  • 19:53 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php
  • 19:40 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'NavigationTiming sample factor to 5000'
  • 19:17 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 19:08 logmsgbot: reedy synchronized php-1.22wmf14/extensions/NavigationTiming
  • 19:07 logmsgbot: reedy synchronized php-1.22wmf14/extensions/GettingStarted
  • 19:06 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 19:00 logmsgbot: maxsem synchronized php-1.22wmf15/extensions/MobileFrontend/includes/formatters/ExtractFormatter.php 'https://gerrit.wikimedia.org/r/#/c/81416/'
  • 18:57 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 18:53 logmsgbot: reedy Finished syncing Wikimedia installation... : Oh scap!
  • 18:38 logmsgbot: reedy Started syncing Wikimedia installation... : Oh scap!
  • 18:33 logmsgbot: reedy synchronized wmf-config/
  • 18:22 logmsgbot: reedy synchronized wmf-config/
  • 18:11 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.22wmf14
  • 18:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki, mediawikiwiki, testwikidatawiki and loginwiki to 1.22wmf15
  • 18:03 logmsgbot: reedy Finished syncing Wikimedia installation... : testwiki to 1.22wmf15 and build l10n cache
  • 17:40 logmsgbot: reedy Started syncing Wikimedia installation... : testwiki to 1.22wmf15 and build l10n cache
  • 17:15 logmsgbot: csteipp synchronized php-1.22wmf14/includes/resourceloader/ 'bug46332'
  • 17:13 logmsgbot: csteipp synchronized php-1.22wmf14/includes/api 'bug49090'
  • 16:58 logmsgbot: reedy synchronized docroot and w
  • 16:57 logmsgbot: reedy synchronized php-1.22wmf15/
  • 15:45 MaxSem: Recreating Solr index
  • 14:48 ottomata: restarting gmetad on nickel to see if I can get it to pick up reinstalled analytics boxes
  • 09:40 springle: killing long-running research transaction on db1047, and setting innodb_max_purge_lag to reduce history list length
  • 09:33 mark: Powercycled lanthanum
  • 08:54 hashar: rebooting lanthanum.eqiad.wmnet (high sys cpu)
  • 08:34 hashar: apt-get upgrade on gallium.wikimedia.org
  • 08:31 hashar: apt-get upgrade on lanthanum.eqiad.wmnet
  • 07:29 springle: relaxed slave delay warnings to warn 150 crit 300
  • 02:42 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Aug 29 02:40:47 UTC 2013
  • 02:27 logmsgbot: LocalisationUpdate completed (1.22wmf13) at Thu Aug 29 02:26:16 UTC 2013
  • 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf14) at Thu Aug 29 02:14:03 UTC 2013
  • 01:41 mwalker: fundraising delete query on db1025 ran 1:10minutes -- serious lag on db1008 and db43 now expected for same
  • 00:31 mwalker: clearing ~14M rows from db1025.pgehres.bannerimpressions and bannerimpression_raw

August 28

  • 23:29 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Disable the job queue that processes echo notification'
  • 23:18 logmsgbot: rmoen synchronized php-1.22wmf14/extensions/VisualEditor/ 'Update VisualEditor to master'
  • 22:49 logmsgbot: demon synchronized php-1.22wmf14/extensions/CentralAuth
  • 22:49 logmsgbot: demon synchronized php-1.22wmf13/extensions/CentralAuth
  • 20:42 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'All wikis to secure login'
  • 20:38 logmsgbot: demon Finished syncing Wikimedia installation... : Secure login on alllllll the wikis
  • 20:24 logmsgbot: demon Started syncing Wikimedia installation... : Secure login on alllllll the wikis
  • 19:30 logmsgbot: awight synchronized php-1.22wmf14/extensions/CentralNotice 'Updating CentralNotice to wmf_deploy'
  • 19:30 logmsgbot: awight synchronized php-1.22wmf13/extensions/CentralNotice 'Updating CentralNotice to wmf_deploy'
  • 18:37 logmsgbot: csteipp synchronized php-1.22wmf13/extensions/Wikibase
  • 18:37 logmsgbot: csteipp synchronized php-1.22wmf13/extensions/CheckUser
  • 18:36 logmsgbot: csteipp synchronized php-1.22wmf14/extensions/CheckUser
  • 18:35 logmsgbot: csteipp synchronized php-1.22wmf14/extensions/Wikibase
  • 18:34 logmsgbot: csteipp synchronized php-1.22wmf14/extensions/OAuth
  • 17:50 ottomata: restarting some datanodes
  • 17:48 logmsgbot: reedy synchronized php-1.22wmf14/extensions/CirrusSearch/ 'Fix bug 53487'
  • 16:49 logmsgbot: demon synchronized php-1.22wmf14/extensions/CirrusSearch/forceSearchIndex.php 'Fix for reindexer'
  • 16:47 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php
  • 16:26 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Move mw.org to primary backend as Cirrus to test my theory'
  • 16:21 Reedy: Created EducationProgram tables on ptwiki
  • 16:09 ^d: ES: indexing mediawikiwiki in screen on terbium
  • 16:08 ^d: ES: created mediawikiwiki index
  • 16:07 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'CirrusSearch on mediawiki.org \o/'
  • 15:47 gwicke: updated Parsoid to ca3f45f
  • 15:40 logmsgbot: demon synchronized php-1.22wmf14/extensions/CirrusSearch 'CirrusSearch to master'
  • 15:04 springle: restarting slave threads on db55, db58, db43. will lag for a while.
  • 14:35 paravoid: authdns-update: switching Middle-East to esams
  • 14:16 paravoid: authdns-update: switching Africa to esams
  • 14:00 hashar: jenkins: renamed OAuth jobs from Oauth to OAuth
  • 13:57 hashar: jenkins: renamed Editcount jobs from EditCount to Editcount
  • 13:45 springle: same purge situation for db58 and db43, and remedy
  • 13:37 springle: stopped db55 slave threads while purge catches up; killed long running research user transaction apparently doing nothing but holding locks
  • 13:21 springle: temporarily changing innodb purge settings on db55 (massive growing history list)
  • 12:34 hashar: jenkins: updated various jobs to use $ZUUL_PROJECT when determining the git repository to fetch from [[bugzilla:53470|bug 53470]]
  • 02:47 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Aug 28 02:45:49 UTC 2013
  • 02:32 logmsgbot: LocalisationUpdate completed (1.22wmf13) at Wed Aug 28 02:30:40 UTC 2013
  • 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf14) at Wed Aug 28 02:16:15 UTC 2013

August 27

  • 23:54 logmsgbot: bsitu synchronized php-1.22wmf13/extensions/Echo 'Update Echo to master'
  • 23:53 logmsgbot: bsitu synchronized php-1.22wmf14/extensions/Echo 'Update Echo to master'
  • 23:19 logmsgbot: reedy synchronized php-1.22wmf14/extensions/CentralAuth
  • 23:19 logmsgbot: reedy synchronized php-1.22wmf13/extensions/CentralAuth
  • 23:12 logmsgbot: kaldari synchronized php-1.22wmf14/extensions/MobileFrontend/ 'updating MobileFrontend to master for wmf14'
  • 23:00 logmsgbot: kaldari synchronized php-1.22wmf13/extensions/MobileFrontend/ 'updating MobileFrontend to master for wmf13'
  • 22:50 logmsgbot: reedy synchronized php-1.22wmf14/extensions/CentralAuth/maintenance/
  • 22:33 Ryan_Lane: purging login.wikimedia.org from text varnish
  • 22:21 logmsgbot: kaldari synchronized php-1.22wmf13/extensions/MobileFrontend/includes/formatters/ExtractFormatter.php 'syncing live-hack to try to address fatal error'
  • 22:07 logmsgbot: kaldari synchronized php-1.22wmf13/extensions/MobileFrontend 'syncing MobileFrontend to try to address fatal error'
  • 21:58 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'syncing InitialiseSettings.php to try to fix fatal error from MobileFrontend'
  • 21:51 RobH: disregard netmon1001 alerts, its my project server
  • 21:45 Reedy: Running extensions/Wikibase/repo/maintenance/rebuildPropertyInfo.php against wikidatawiki in screen on terbium
  • 21:33 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
  • 21:13 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
  • 21:04 Ryan_Lane: deployed change 81375 to remove the http to https redirect from loginwiki
  • 20:44 logmsgbot: demon synchronized php-1.22wmf14/extensions/CentralAuth 'CentralAuth to master'
  • 20:43 logmsgbot: demon synchronized php-1.22wmf13/extensions/CentralAuth 'CentralAuth to master'
  • 20:06 logmsgbot: maxsem synchronized w/robots.php 'https://gerrit.wikimedia.org/r/69420'
  • 19:27 cmjohnson1: search1002 going down for bios update
  • 19:19 logmsgbot: mlitn Finished syncing Wikimedia installation... : Updating ArticleFeedbackv5, Echo, PageTriage and Thanks
  • 19:06 logmsgbot: mlitn Started syncing Wikimedia installation... : Updating ArticleFeedbackv5, Echo, PageTriage and Thanks
  • 18:55 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable job queue to process notifs for all wikis'
  • 18:06 gwicke: updated Parsoid to 84fac157
  • 18:05 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php 'Enable job queue to process web and email notifs on testwiki/test2wiki'
  • 18:04 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable job queue to process web and email notifs on testwiki/test2wiki'
  • 16:36 ^d: reloaded gerrit replication plugin to pick up changes
  • 14:56 logmsgbot: hashar synchronized php-1.22wmf14/extensions/Wikibase 'https://gerrit.wikimedia.org/r/#/c/81233/'
  • 14:55 logmsgbot: hashar synchronized php-1.22wmf14/extensions/WikibaseDataModel 'https://gerrit.wikimedia.org/r/#/c/81233/'
  • 14:53 logmsgbot: hashar synchronized php-1.22wmf13/extensions/Wikibase 'https://gerrit.wikimedia.org/r/#/c/81237/'
  • 11:31 logmsgbot: hashar synchronized php-1.22wmf14/extensions/DataValues 'Update DataValues 81198, requested by aude'
  • 09:56 hashar: jenkins restarted :-]
  • 09:30 hashar: jenkins: Failed to exec '/usr/lib/jvm/java-6-openjdk-amd64/jre/bin/java (deleted)' No such file or directory holy crap
  • 09:28 hashar: restarting Jenkins for plugins upgrade
  • 07:35 hashar: jenkins: upgrading pep8 on gallium [[bugzilla:53352|bug 53352]]
  • 02:22 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Aug 27 02:21:16 UTC 2013
  • 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf13) at Tue Aug 27 02:12:42 UTC 2013
  • 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf14) at Tue Aug 27 02:07:20 UTC 2013
  • 01:08 mutante: chown'ing /srv/ssd/gerrit to gerritslave for replication on lanthanum

August 26

  • 22:49 ^d: restarting gerrit
  • 22:47 binasher: fixed labsdb cpufreq governor setting (and got a 2x speedup on a large groupby query, woops!)
  • 21:13 logmsgbot: reedy synchronized php-1.22wmf14/extensions/Wikibase
  • 20:19 logmsgbot: reedy synchronized wmf-config/
  • 20:03 ottomata: taking down analytics1011-analytics1020 for reinstall
  • 19:58 ottomata: taking analytics1010 down in preparation for repaving of hadoop cluster
  • 19:49 logmsgbot: reedy synchronized php-1.22wmf14/extensions/Wikibase
  • 19:48 logmsgbot: reedy synchronized php-1.22wmf14/extensions/DataValues/
  • 19:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote to 1.22wmf14
  • 19:20 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource, wikibooks and wikinews to 1.22wmf14
  • 19:19 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 19:10 logmsgbot: reedy synchronized database lists files: wikivoyage, wikiversity and wiktioanry to 1.22wmf14
  • 19:09 RobH: for operations/dns.git
  • 19:09 RobH: i accidentally (yea right) pushed .gitreview to master rather than head, so no gerrit log for my change, opps!
  • 19:00 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 18:51 logmsgbot: reedy synchronized wmf-config/
  • 18:42 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special, closed and wikimedia to 1.22wmf14
  • 18:29 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Fix wmgHTTPSBlacklistCountries'
  • 17:07 logmsgbot: csteipp synchronized php-1.22wmf13/extensions/LiquidThreads 'Fix bug53320'
  • 17:07 logmsgbot: csteipp synchronized php-1.22wmf14/extensions/LiquidThreads 'Fix bug53320'
  • 16:57 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'GeoIP configuration for secure login'
  • 16:56 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'GeoIP configuration for secure login'
  • 16:54 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'mediawikiwiki to secure login'
  • 16:38 ottomata: taking down analytics1006, analytics1008 and analytics1009 for reinstall
  • 16:30 ottomata: bringing analytics1005 down for reinstall
  • 02:25 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Aug 26 02:24:25 UTC 2013
  • 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf14) at Mon Aug 26 02:16:07 UTC 2013
  • 02:10 logmsgbot: LocalisationUpdate completed (1.22wmf13) at Mon Aug 26 02:09:13 UTC 2013

August 25

  • 16:51 RoanKattouw: Restarted Parsoid on a couple of boxes to clear some disk space. It appears wtp1001-1004 have been set up with larger, LVMed root partitions but I can't find any logs on when this was done, by whom, or why
  • 16:46 RoanKattouw: Restarted Parsoid on wtp1012, was unresponsive
  • 16:41 RoanKattouw: Strike that, my account on fenari is just being weird apparently
  • 16:40 RoanKattouw: Investigating low disk space warnings on wtp1001-1024, but it looks like something weird happened to them, I get host key verification warnings or failures for all of them
  • 02:38 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Aug 25 02:37:03 UTC 2013
  • 02:26 logmsgbot: LocalisationUpdate completed (1.22wmf14) at Sun Aug 25 02:25:16 UTC 2013
  • 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf13) at Sun Aug 25 02:13:19 UTC 2013
  • 01:05 logmsgbot: demon Finished syncing Wikimedia installation... :
  • 00:53 logmsgbot: demon Started syncing Wikimedia installation... :

August 24

  • 19:27 gwicke: updated Parsoid to 8afe5ef
  • 09:57 springle: optimizing tables on db55. expect lag
  • 02:43 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Aug 24 02:42:22 UTC 2013
  • 02:33 logmsgbot: LocalisationUpdate completed (1.22wmf14) at Sat Aug 24 02:31:29 UTC 2013
  • 02:18 logmsgbot: LocalisationUpdate completed (1.22wmf13) at Sat Aug 24 02:16:50 UTC 2013

August 23

  • 23:58 mutante: removing mailing list Wikipedia-ensino, creating list educacao-br
  • 20:59 logmsgbot: marc synchronized docroot
  • 20:53 ori-l: Re-starting eventlogging/consumer NAME=mysql-db1047 on vanadium
  • 20:45 logmsgbot: root synchronized docroot
  • 20:39 ori-l: Stopped eventlogging/consumer NAME=mysql-db1047 on vanadium
  • 20:34 binasher: taking mysqld down on db1047 (s1 analytics + eventlog data) for unplanned maintenance
  • 19:54 RobH: ns2.wikimedia.org glue record updated from 91.198.174.4 to 91.198.174.239
  • 18:20 awight: civicrm updated from e4f1228 to 5c28984
  • 17:20 ottomata: disabling puppet on analytics 1006, 1008, 1009, 1021 and 1022 in preparation for reinstallation and repaving next week
  • 17:19 awight: updated civicrm from 257686e81cf6 to e4f1228
  • 16:28 logmsgbot: reedy synchronized php-1.22wmf14/extensions/WikiEditor/
  • 16:27 logmsgbot: reedy synchronized php-1.22wmf13/extensions/WikiEditor/
  • 16:20 logmsgbot: reedy synchronized wmf-config/
  • 15:45 ottomata: disabling puppet on analytics1024 and analytics1025 in preperation for reinstall of zookeeper nodes
  • 15:08 ottomata: taking analytics1023 offline for reinstall
  • 10:17 paravoid: upgrading ceph to v0.67.2
  • 07:28 springle: enabled dns reverse-lookups on db1016 mysql while replication catches up, to avoid slave sql thread stopping on SET PASSWORD / GRANT statements using hostnames
  • 06:10 springle: ran mysql_upgrade on db1001 (due to missing/incorrect system tables errors in log)
  • 05:25 springle: db35 manual mysql.* check, restart, mysql_upgrade, resolve replication sql error
  • 05:19 springle: stopping db35 mysqld for mysql.* checks and mysql_upgrade
  • 05:13 springle: resolved db1016 replication sql error
  • 03:00 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Aug 23 02:59:10 UTC 2013
  • 02:49 logmsgbot: LocalisationUpdate completed (1.22wmf14) at Fri Aug 23 02:47:36 UTC 2013
  • 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf13) at Fri Aug 23 02:15:37 UTC 2013
  • 01:21 logmsgbot: reedy synchronized php-1.22wmf14/includes/specials/SpecialUserlogin.php

August 22

  • 23:13 logmsgbot: reedy synchronized php-1.22wmf14/
  • 23:05 logmsgbot: catrope synchronized php-1.22wmf14/extensions/VisualEditor 'Update VE for cherry-pick'
  • 23:05 logmsgbot: catrope synchronized php-1.22wmf13/extensions/VisualEditor 'Update VE for cherry-pick'
  • 22:14 logmsgbot: aaron synchronized php-1.22wmf13/includes/job/jobs/RefreshLinksJob.php '228e592'
  • 20:59 Jeff_Green: fixed exim/spamassassin config on iodine so messages are spam-tagged for otrs filters
  • 20:54 Krinkle: Reloading Zuul to deploy I3c754cce8828f
  • 20:54 awight: civicrm updated from 5f735ed to 257686e
  • 20:19 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 18:38 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: phase 3 wikis to 1.22wmf13
  • 18:36 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: phase 1 wikis to 1.22wmf14
  • 17:47 logmsgbot: reedy synchronized php-1.22wmf13/includes/api/ApiEditPage.php
  • 16:58 logmsgbot: mwalker synchronized php-1.22wmf14/extensions/CentralNotice/special/ 'Updating wmf14 to include CentralNotice bugfix 53032'
  • 16:56 gwicke: updated Parsoid to 437374dc
  • 16:56 logmsgbot: mwalker Finished syncing Wikimedia installation... : Updating CentralNotice to master
  • 16:41 logmsgbot: mwalker Started syncing Wikimedia installation... : Updating CentralNotice to master
  • 15:13 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.22wmf13 for now
  • 15:09 logmsgbot: reedy Finished syncing Wikimedia installation... : testwiki to 1.22wmf14, build l10n cache
  • 14:56 logmsgbot: reedy Started syncing Wikimedia installation... : testwiki to 1.22wmf14, build l10n cache
  • 14:38 logmsgbot: reedy synchronized docroot and w
  • 14:36 logmsgbot: maxsem synchronized php-1.22wmf13/extensions/MobileFrontend/includes/specials/SpecialMobileOptions.php 'debug over'
  • 14:35 logmsgbot: reedy synchronized php-1.22wmf14/extensions/OAuth/
  • 14:34 logmsgbot: reedy synchronized php-1.22wmf14/ 'Initial file sync'
  • 14:32 logmsgbot: maxsem synchronized php-1.22wmf13/extensions/MobileFrontend/includes/specials/SpecialMobileOptions.php 'debug'
  • 13:06 paravoid: changed watchmouse alert and re-enabled
  • 11:37 paravoid: switching off alerts for watchmouse's DNS alert while investigating
  • 11:04 paravoid: disabling puppet on eeden for the next couple of days
  • 02:39 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Aug 22 02:38:17 UTC 2013
  • 02:30 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Thu Aug 22 02:28:28 UTC 2013
  • 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf13) at Thu Aug 22 02:15:07 UTC 2013
  • 00:06 logmsgbot: catrope synchronized php-1.22wmf13/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.ViewPageTarget.js 'touch'
  • 00:06 logmsgbot: catrope synchronized php-1.22wmf12/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.ViewPageTarget.js 'touch'
  • 00:03 logmsgbot: catrope synchronized php-1.22wmf13/extensions/VisualEditor 'Update VE to master'
  • 00:03 logmsgbot: catrope synchronized php-1.22wmf12/extensions/VisualEditor 'Update VE to master'

August 21

  • 22:32 binasher: opened up wikiadmin grants on es1 shard to accommodate snapshot1004
  • 21:01 mutante: cp1061,cp1062 have dpkg issue upgrading varnish packages
  • 20:38 mutante: installing package upgrades on iron
  • 20:36 mutante: re-enabled puppet runs on zirconium which was "admin down"
  • 20:21 mutante: dist-upgrading zirconium
  • 18:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable Collection on testwikidatawiki. It's mucho brokenp'
  • 18:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable Collection on testwikidatawiki'
  • 18:31 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Enabled $wgMWOAuthSecureTokenTransfer'
  • 16:37 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 16:28 paravoid: DNS migration complete
  • 16:22 paravoid: switching ns2 to eeden
  • 16:06 logmsgbot: reedy synchronized docroot and w
  • 15:53 paravoid: switching ns0 to rubidium
  • 15:48 logmsgbot: reedy synchronized wmf-config/
  • 15:47 logmsgbot: reedy synchronized wmf-config/
  • 15:37 logmsgbot: faidon synchronized wmf-config/filebackend.php 'revert ceph promotion to master'
  • 14:00 paravoid: stopping ceph-osd 88 & 95 (ms-be1008), evidence of unexplainable high cpu usage
  • 13:39 paravoid: switching ns1 traffic to mexia (new authdns)
  • 12:09 Reedy: Running extensions/CentralAuth/maintenance/populateHomeDB.php in a screen on tin
  • 11:03 logmsgbot: faidon synchronized wmf-config/filebackend.php 'promote ceph to master'
  • 08:02 logmsgbot: nikerabbit synchronized php-1.22wmf13/extensions/UniversalLanguageSelector/ 'ULS to master'
  • 02:48 mutante: RT - create new system dashboard 'wikimedia default' that lists open ops-requests, quick ticket creation, reminders and bookmarked tickets
  • 02:47 mutante: RT - grant global right to see system dashboards to privileged users
  • 02:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Aug 21 02:35:19 UTC 2013
  • 02:30 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Wed Aug 21 02:28:33 UTC 2013
  • 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf13) at Wed Aug 21 02:15:12 UTC 2013
  • 01:04 logmsgbot: mwalker synchronized php-1.22wmf12/extensions/CentralNotice/special 'Applying CentralNotice fix for bug 53032'
  • 01:03 logmsgbot: mwalker synchronized php-1.22wmf13/extensions/CentralNotice/special 'Applying CentralNotice fix for bug 53032'
  • 00:59 mutante: allow engineering and ops group members in RT to create new saved searches
  • 00:12 logmsgbot: catrope synchronized php-1.22wmf13/extensions/VisualEditor/modules/ve/ui/tools/buttons/ve.ui.UnderlineButtonTool.js 'touch'
  • 00:05 logmsgbot: catrope Finished syncing Wikimedia installation... : Updating VisualEditor to master

August 20

  • 23:58 logmsgbot: catrope Started syncing Wikimedia installation... : Updating VisualEditor to master
  • 23:48 mutante: powercycling frozen mw1173
  • 23:46 mutante: powercycling frozen mw1089
  • 23:43 mutante: powercycling srv281
  • 23:40 logmsgbot: catrope synchronized php-1.22wmf13/extensions/VisualEditor 'Deploy new VE code to wmf13 first for testing'
  • 23:40 RoanKattouw: Repeating previously aborted sync-dir of extensions/VisualEditor now that the dsh node list is fixed
  • 23:38 mutante: manually removed mw1046 from dsh groups mediawiki-installation,apaches,apaches-eqiad on tin
  • 23:34 RoanKattouw: Forcing puppet on tin to update dsh lists for mw1046 removal
  • 23:01 ori-l: During sync-dir, SSH timeouts from srv281, mw1089, mw1173; rsync errors on mw1046 (RO fs; previously reported)
  • 23:00 logmsgbot: olivneh synchronized php-1.22wmf13/extensions/CoreEvents 'Updating CoreEvents to master (2/2)'
  • 23:00 logmsgbot: olivneh synchronized php-1.22wmf12/extensions/CoreEvents 'Updating CoreEvents to master (1/2)'
  • 21:51 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
  • 21:27 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
  • 20:29 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'test2wiki to secure login'
  • 20:26 logmsgbot: demon Finished syncing Wikimedia installation... :
  • 20:04 binasher: granted "lock tables" to eventlog user on db1047
  • 20:03 logmsgbot: demon Started syncing Wikimedia installation... :
  • 19:41 logmsgbot: demon Started syncing Wikimedia installation... :
  • 19:29 logmsgbot: anomie synchronized wmf-config/CommonSettings.php 'Fix OAuth rights assignments'
  • 19:21 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings-labs.php 'Turn HTTPs on by default for beta'
  • 19:20 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable Echo and Thanks on fr, hu, pt, pl and sv wiki'
  • 19:18 logmsgbot: bsitu synchronized echowikis.dblist 'Add fr, hu, pt, pl and sv to Echo dblist'
  • 19:08 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo to master
  • 18:36 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo to master
  • 18:27 gwicke: updated Parsoid to f359548f04e739
  • 14:29 paravoid: starting multiple parallel swift->ceph copy jobs on terbium
  • 10:18 apergos: reboot lvs1002 after dist-upgrade
  • 07:39 apergos: reboot lvs1004 after dist-upgrade
  • 07:11 apergos: rebooting lvs1001 to fix eth2.1003 ip addr, after misguided attempt to simply ifdown/ifup
  • 07:00 paravoid: killing pybal on lvs1001
  • 03:59 paravoid: authdns-update: Google DKIM selector
  • 03:09 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Aug 20 03:07:38 UTC 2013
  • 02:46 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Tue Aug 20 02:45:04 UTC 2013
  • 02:33 logmsgbot: LocalisationUpdate completed (1.22wmf13) at Tue Aug 20 02:31:40 UTC 2013
  • 00:30 logmsgbot: olivneh synchronized php-1.22wmf13/extensions/CoreEvents 'Updating CoreEvents to master for Ide8469db2 (2/2)'
  • 00:29 logmsgbot: olivneh synchronized php-1.22wmf12/extensions/CoreEvents 'Updating CoreEvents to master for Ide8469db2 (1/2)'

August 19

  • 22:48 ^d: ES: reindexing test2wiki
  • 22:44 logmsgbot: demon synchronized php-1.22wmf13/extensions/CirrusSearch 'Cirrus to master'
  • 22:31 logmsgbot: demon synchronized php-1.22wmf13/extensions/CirrusSearch 'Cirrus to master'
  • 22:13 ^d: created oauth tables for mediawikiwiki
  • 22:11 Coren: chown -R gitpuppet /var/lib/git/operations/puppet on sockpuppet: some of those files were (erroneously) owned by root
  • 20:46 logmsgbot: reedy synchronized php-1.22wmf13/extensions/CirrusSearch/
  • 20:03 logmsgbot: csteipp synchronized php-1.22wmf13 'sync again in case mw1046 errors messed up scap'
  • 19:46 logmsgbot: csteipp synchronized php-1.22wmf12
  • 19:40 logmsgbot: reedy synchronized php-1.22wmf13/includes/specials/SpecialUpload.php
  • 19:36 akosiaris: depooled mw1046
  • 19:18 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary and wikiquote to 1.22wmf13
  • 19:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource, wikiversity and wikibooks to 1.22wmf13
  • 19:11 Reedy: mw1046 reporting readonly filesystem
  • 19:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special, closed, wikimedia, wikinews, wikivoyage to 1.22wmf13
  • 18:06 logmsgbot: csteipp Started syncing Wikimedia installation... :
  • 17:57 paravoid: rebooting rubidium, mexia, eeden -- kernel upgrade
  • 17:52 paravoid: apt: new geoip-database package version, from 2 years later
  • 17:43 logmsgbot: csteipp synchronized php-1.22wmf12/extensions/CentralAuth 'revert ca update'
  • 17:37 logmsgbot: csteipp synchronized php-1.22wmf12/extensions/CentralAuth 'Updating wmf12 CentralAuth to master for OAuth'
  • 17:14 cmjohnson1: removing sq41 from dsh node groups and pybal
  • 16:34 logmsgbot: csteipp synchronized php-1.22wmf13/extensions/CentralAuth 'Update CentralAuth to master for OAuth hooks'
  • 14:46 paravoid: authdns-update: remove trailing dot from ulsfo reverse zone names
  • 13:18 springle: temporarily running pt-kill on s5 slaves for slow wikidata queries
  • 12:51 paravoid: hopefully final swiftrepl (swift->ceph thumbs) run on ms-fe1002
  • 12:19 logmsgbot: tstarling synchronized php-1.22wmf12/includes/GlobalFunctions.php 'revert temp hack, superseded'
  • 12:18 logmsgbot: faidon synchronized wmf-config/filebackend.php 'ceph as a filebackend'
  • 11:59 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'changing session cookie name due to bug 53032'
  • 07:56 logmsgbot: tstarling synchronized php-1.22wmf12/includes/GlobalFunctions.php 'hack for session bug'
  • 07:22 paravoid: switching text-varnish pybal group back to squids
  • 07:22 Tim: destroyed leaked session with ID 796c2b...
  • 05:50 paravoid: authdns-update: add ADSP (dkim=unknown), DKIM policy record for lists (o=~)
  • 04:20 paravoid: authdns-update: DKIM & DMARC (both no-op, for now)
  • 03:01 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Aug 19 03:00:26 UTC 2013
  • 02:42 logmsgbot: LocalisationUpdate completed (1.22wmf13) at Mon Aug 19 02:41:19 UTC 2013
  • 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Mon Aug 19 02:15:45 UTC 2013

August 18

  • 18:55 Elsie: rotated server admin log wikitech page; testing logging
  • 12:27 paravoid: authdns-update: add AAAA for sodium
  • 11:44 mark: Moved eqiad text-varnish traffic back to Squid temporarily
  • 10:15 springle: resuming slave db52 after bug 52853 investigation
  • 07:20 MaxSem: Killed a few hung queries on db1026 that were causing massive lag
  • 02:25 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Aug 18 02:23:36 UTC 2013
  • 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf13) at Sun Aug 18 02:13:27 UTC 2013
  • 02:07 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Sun Aug 18 02:05:50 UTC 2013

August 17

  • 15:03 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'returning db1009 to the pool after sync'
  • 02:41 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Aug 17 02:40:08 UTC 2013
  • 02:21 logmsgbot: LocalisationUpdate completed (1.22wmf13) at Sat Aug 17 02:19:58 UTC 2013
  • 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Sat Aug 17 02:14:54 UTC 2013

August 16

  • 21:58 logmsgbot: demon synchronized php-1.22wmf12/extensions/CirrusSearch 'No actual changes, was just a change + revert'
  • 21:48 ori-l: Timeouts on last two sync-dirs: srv281, mw1089, mw1173
  • 21:47 logmsgbot: olivneh synchronized php-1.22wmf13/extensions/CoreEvents 'Updating CoreEvents to master for I08f66861b (2/2)'
  • 21:47 logmsgbot: olivneh synchronized php-1.22wmf12/extensions/CoreEvents 'Updating CoreEvents to master for I08f66861b (1/2)'
  • 21:07 logmsgbot: reedy synchronized docroot and w
  • 20:47 logmsgbot: reedy synchronized docroot and w
  • 19:38 cmjohnson1: shutting down brewster yet again to replace the disk
  • 17:31 logmsgbot: demon synchronized php-1.22wmf13/extensions/CirrusSearch/CirrusSearch.body.php
  • 16:52 cmjohnson1: brewster going down again
  • 16:22 cmjohnson1: shutting down brewster to replace bad disk
  • 16:22 bblack: merged vhtcpd ganglia stuff and the VCL for PURGE hash_ignore_busy into puppet
  • 15:10 logmsgbot: reedy synchronized wmf-config/ 'touch'
  • 15:07 logmsgbot: reedy synchronized database lists files: Ad tyvwiki to visualeditor.dblist
  • 09:38 paravoid: rolling reboot of ceph ms-be nodes
  • 08:11 apergos: sq41 hardware issues, see rt 5615
  • 05:46 paravoid: downgraded puppetmaster to previous release, new minor update had significantly increased CPU requirements and DoSed stafford
  • 02:59 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Aug 16 02:57:56 UTC 2013
  • 02:36 logmsgbot: LocalisationUpdate completed (1.22wmf13) at Fri Aug 16 02:35:14 UTC 2013
  • 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Fri Aug 16 02:15:11 UTC 2013
  • 00:00 logmsgbot: demon synchronized php-1.22wmf13/includes/specials/SpecialSearch.php 'Renamed backend to srbackend'

August 15

  • 23:38 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Debug log settings for Cirrus'
  • 23:33 ^d: finished indexing all of test2wiki
  • 23:15 ^d: running ES indexing for test2wiki in screen on terbium
  • 23:10 ^d: built test2wiki index in ES
  • 23:08 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'test2wiki to CirrusSearch'
  • 21:52 logmsgbot: olivneh Finished syncing Wikimedia installation... : E3 deploy: GuidedTour & GettingStarted
  • 21:40 logmsgbot: olivneh Started syncing Wikimedia installation... : E3 deploy: GuidedTour & GettingStarted
  • 21:25 apergos: live hacked /etc/nagios/puppet_servicegroups.cfg and /etc/nagios/puppet_hostgroups.cfg on neon to add host and service group 'cache_misc_eqiad', icinga wouldn't start without it
  • 21:17 ori-l: previous two syncs: timeouts on srv281, mw1089, mw1173
  • 21:16 logmsgbot: olivneh synchronized php-1.22wmf12/extensions/GuidedTour 'E3 deploy: GuidedTour to master'
  • 21:15 logmsgbot: olivneh synchronized php-1.22wmf12/extensions/GettingStarted 'E3 deploy: GettingStarted to master'
  • 19:49 paravoid: ceph: adding osd.93
  • 19:42 Reedy: Added wb_property_info table to wikidatawiki and testwikidatawiki
  • 18:49 paravoid: rebooting stafford & sockpuppet for kernel upgrades
  • 18:21 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: 1.22wmf13 phase1
  • 18:06 cmjohnson1: powering down ms-be1008 to update hard drive firmware
  • 17:59 logmsgbot: reedy synchronized php-1.22wmf13/includes/specials/SpecialNewimages.php
  • 16:36 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: revert test2wiki to 1.22wmf12 till window
  • 16:29 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.22wmf13 and build l10n cache
  • 16:16 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.22wmf13 and build l10n cache
  • 16:09 logmsgbot: reedy synchronized php-1.22wmf13/
  • 15:40 logmsgbot: reedy synchronized docroot and w
  • 15:38 logmsgbot: reedy synchronized php-1.22wmf13/
  • 08:15 springle: starting pt-table-checksum on db1034 bug 52853
  • 07:03 springle: delaying slave db52 during bug 52853 investigation
  • 06:49 logmsgbot: tstarling synchronized wmf-config/db-eqiad.php 'depooling db1009 at springle's request'
  • 06:47 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
  • 06:46 logmsgbot: tstarling synchronized wmf-config/CirrusSearch-production.php
  • 06:45 logmsgbot: tstarling synchronized wmf-config/CirrusSearch-common.php
  • 06:44 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
  • 06:44 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings-labs.php
  • 06:43 Tim: deploying CirrusSearch configuration changes (fa29915)
  • 02:56 paravoid: rebooting iron, kernel upgrade
  • 02:44 paravoid: rebooting bast1001, kernel upgrade
  • 02:32 paravoid: rebooting fenari, kernel upgrade
  • 02:22 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Aug 15 02:20:39 UTC 2013
  • 02:09 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Thu Aug 15 02:08:17 UTC 2013

August 14

  • 17:43 logmsgbot: csteipp synchronized php-1.22wmf12/includes
  • 12:24 mark: Moved non-https, non-IPv6 eqiad traffic for *.wikimedia.org (wikimedialb) to text-varnish, not caching login.wikimedia.org
  • 11:14 paravoid: swift->ceph thumb sync (swiftrepl @ ms-fe1002)
  • 11:00 akosiaris: stoping etherpad-lite for maintainance (character set hell)
  • 10:49 paravoid: upgrading ceph to 0.67
  • 10:02 mark: Moved non-https, non-IPv6 eqiad traffic for *.wikimedia.org (wikimedialb) back to Squid
  • 09:41 mark: Moved non-https, non-IPv6 eqiad traffic for *.wikimedia.org (wikimedialb) to text-varnish
  • 02:19 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Aug 14 02:18:04 UTC 2013

August 13

  • 22:56 cmjohnson1: authdns update
  • 18:52 RobH: wikipedia.in is now working, huzzah.
  • 18:42 RobH: updated apache redirects, but dsh stuff wont workfor me so hacking an apache gracefull all via salt
  • 17:49 RobH: rob is testing his new ssh settings to see if he can dsh graceful all apaches without errors, no actual changes are being made
  • 08:52 akosiaris1: taking down epl.wikimedia.org for maintanance
  • 07:30 akosiaris: enabled https redir for epl.wikimedia.org.
  • 06:00 apergos: ran puppet on antimony and restarted apache to pick up new robots.txt (see r/78919 )
  • 05:58 apergos: restarted apache on stafford, puppet was dead
  • 02:17 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Aug 13 02:15:58 UTC 2013
  • 02:07 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Tue Aug 13 02:05:45 UTC 2013

August 12

  • 18:21 cmjohnson1: powering down testsearch1001 and 1002 to compare settings
  • 17:23 ^d: gerrit restarting
  • 11:06 LeslieCarr: removed wpasupplicant from pdf3
  • 10:38 LeslieCarr: removing wpasupplicant from pdf1
  • 10:38 LeslieCarr: removing wirelesstools from pdf1
  • 10:37 LeslieCarr: removing wirelesstools from pdf2
  • 10:35 LeslieCarr: uninstalling wpasupplicant from pdf2
  • 10:34 LeslieCarr: dist-upgrading pdf2
  • 10:26 LeslieCarr: rebooting pdf2
  • 09:54 akosiaris: stopping etherpad-lite to load old etherpad data
  • 08:02 Krinkle: Ran updateiwcache to fix issue with viwikivoyage interwiki links resolving to $lang.wikipedia.org instead of wikivoyage
  • 07:59 logmsgbot: krinkle synchronized wmf-config/interwiki.cdb 'Updating interwiki cache'
  • 05:12 logmsgbot: reedy synchronized php-1.22wmf12/extensions/WikimediaMaintenance/addToSites.php
  • 03:55 LeslieCarr: restarting apache2 on mw1153
  • 02:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Aug 12 02:24:49 UTC 2013
  • 02:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Revert back to normal wikidata logo'
  • 02:12 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Mon Aug 12 02:10:38 UTC 2013

August 11

  • 14:00 Tim: disabled offending query source and killed queries again
  • 13:59 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'disabling Special:SupportedLanguages due to apache/mysql overload'
  • 13:59 paravoid: powercycling db1009, down for 20h, kernel prints "BUG: soft lockup"
  • 13:46 Tim: killing SpecialSupportedLanguages on db1004 and other s4 slaves
  • 07:18 logmsgbot: reedy synchronized php-1.22wmf12/extensions/CentralAuth 'rv back'
  • 07:15 logmsgbot: reedy synchronized php-1.22wmf12/extensions/CentralAuth/maintenance/populateHomeDB.php
  • 07:09 logmsgbot: reedy synchronized php-1.22wmf12/extensions/CentralAuth/CentralAuthUser.php
  • 06:15 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable upload on viwikivoyage'
  • 06:09 logmsgbot: reedy synchronized wmf-config/ 'touch'
  • 06:07 logmsgbot: reedy synchronized database lists files:
  • 05:58 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 05:57 logmsgbot: reedy synchronized database lists files:
  • 05:06 logmsgbot: reedy synchronized wmf-config/interwiki.cdb 'Updating interwiki cache'
  • 04:45 logmsgbot: reedy synchronized wmf-config 'touch'
  • 04:43 logmsgbot: reedy synchronized database lists files: Add tyvwiki to wikidataclient
  • 04:36 logmsgbot: reedy synchronized php-1.22wmf12/extensions/WikimediaMaintenance/addToSites.php
  • 04:27 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 04:27 logmsgbot: reedy synchronized database lists files:
  • 04:22 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 03:05 logmsgbot: reedy synchronized wmf-config/
  • 02:50 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Aug 11 02:49:14 UTC 2013
  • 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Sun Aug 11 02:15:52 UTC 2013
  • 01:32 logmsgbot: reedy synchronized php-1.22wmf12/extensions/CodeReview

August 10

  • 23:56 LeslieCarr: restarted gitblit on antimony
  • 12:33 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 11:03 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 07:17 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'rv'
  • 07:13 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Disable CR proxy'
  • 05:59 mutante: deleting codereview-proxy (pre-gerrit,svn) from bugzilla server
  • 05:59 mutante: deleting broken bzapi files from bugzilla server
  • 05:53 mutante: apt-get upgrading kaulen (bugzilla)
  • 02:28 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Aug 10 02:27:23 UTC 2013
  • 02:13 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Sat Aug 10 02:12:01 UTC 2013

August 9

  • 18:49 paravoid: running ceph->swift thumb sync on ms-fe1002
  • 18:40 paravoid: running originals swift->ceph sync (syncFileBackend.php on terbium)
  • 09:31 mark: Assigned IPv4 mapped IPv6 address to mchenry
  • 08:13 mutante: apt-get upgrading zirconium
  • 04:09 paravoid: authdns-update: added "tyv" to langlist
  • 04:03 logmsgbot: reedy synchronized langlist 'attempt 2'
  • 04:02 logmsgbot: reedy synchronized langlist
  • 03:19 logmsgbot: reedy synchronized docroot/wikidata/WikidataHK.png 'Compress'
  • 02:16 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Aug 9 02:15:21 UTC 2013
  • 02:06 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Fri Aug 9 02:05:28 UTC 2013
  • 01:36 logmsgbot: reedy synchronized wmf-config 'Sync-ing for consistency (no changes)'

August 8

  • 20:25 cmjohnson1: working on carbon
  • 18:26 mark: Removed /dev/sda from sw raid arrays on brewster
  • 08:54 logmsgbot: reedy synchronized wmf-config/
  • 08:42 mutante: on overloaded antimony: stopped gitblit, ran puppet, blocked Googlebot, restarted apache and gitblit
  • 08:22 mutante: git.wikimedia.org back
  • 08:21 mutante: attempted to restart gitblit on antimony
  • 08:00 logmsgbot: reedy synchronized wmf-config/
  • 07:58 logmsgbot: reedy synchronized docroot and w
  • 02:18 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Aug 8 02:16:52 UTC 2013
  • 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Thu Aug 8 02:06:51 UTC 2013
  • 00:36 mwalker: updated fundraising.wm.o wiki from 0e08f1e to b104c72

August 7

  • 23:48 LeslieCarr: rebooting cr2-knams
  • 23:44 LeslieCarr: deactivating external bgp neighbors on cr2-knams
  • 22:47 ^d: reloaded gerrit replication plugin to pick up new config
  • 21:48 apergos: stopping and restarting gitblit for google testing
  • 21:26 paravoid: authdns-update: adding lists AAAA & (neutral) SPF
  • 19:11 paravoid: authdns-update: removing lists' SPF
  • 17:48 RobH: authdns-update (just incase i break it all)
  • 17:25 RobH: authdnsiupdate for lists.w.o spf records
  • 12:10 paravoid: removing 91.198.174.7/32 from ssl3001 & maerlant, old/deprecated ipv6 testing IP
  • 07:17 apergos: restarted gitblit, (btw the init script did not stop it properly)
  • 02:23 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Aug 7 02:22:30 UTC 2013
  • 02:10 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Wed Aug 7 02:08:47 UTC 2013

August 6

  • 16:02 Jeff_Green: authdns-update to flip ticket.wikimedia.org to williams
  • 10:37 Jeff_Green: stopped icinga notifications for db1048, db1046, db48, db49
  • 10:22 Jeff_Green: db1046 mysql replication stopped
  • 10:17 Jeff_Green: stop exim on williams.wikimedia.org
  • 10:17 Jeff_Green: stop apache on williams.wikimedia.org
  • 02:17 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Aug 6 02:16:18 UTC 2013
  • 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Tue Aug 6 02:06:34 UTC 2013

August 5

  • 17:50 akosiaris: added d2to1, pep8, python-setuptools-git packages in apt.wikimedia.org. Packages were created by hashar using backports from ubuntu. I added debian/changelog entry to get an upgrade friendly version and run lintian on each one of them with no errors.
  • 15:56 akosiaris: cleared old cruft snapmirror replication from nas1-a to nas1001-a. Direction has been reverted long ago
  • 14:01 mark: Put non-European wikivoyage traffic on the eqiad text Varnish cluster
  • 09:27 paravoid: powercycling neon, OOM'ed, "login: failure forking: Cannot allocate memory" etc.; icinga down since ~01:00 UTC
  • 02:16 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Aug 5 02:14:37 UTC 2013
  • 02:07 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Mon Aug 5 02:05:54 UTC 2013

August 4

  • 17:18 Nemo_bis: antimony (gitblit) overloaded again as of yesterday around 19 UTC
  • 02:14 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Aug 4 02:13:09 UTC 2013
  • 02:06 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Sun Aug 4 02:04:59 UTC 2013

August 3

  • 19:47 Ryan_Lane: repooling ssl1007-9
  • 14:42 logmsgbot: reedy synchronized wmf-config/squid.php 'ssl1007-ssl1009'
  • 08:00 Ryan_Lane: depooling ssl1007-9 for now. They need to be added to the MW config.
  • 02:38 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Aug 3 02:36:54 UTC 2013
  • 02:18 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Sat Aug 3 02:16:47 UTC 2013
  • 00:15 marktraceur: Deployed new version of parsoid-regressions via Jenkins Job Builder
  • 00:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php

August 2

  • 22:55 Ryan_Lane: repooling ssl1001-1007
  • 22:52 Ryan_Lane: deppoling ssl1007 for more testing
  • 22:43 Ryan_Lane: depooling ssl1005-6 to further test load
  • 22:35 Ryan_Lane: depooling ssl1001-4 to measure load on new hardware
  • 22:11 Ryan_Lane: pooling ssl1007-9
  • 21:13 cmjohnson1: dns update
  • 19:37 cmjohnson1: dns update
  • 19:22 paravoid: installing mexia
  • 14:32 LeslieCarr: pdf3 unresponsive on console , rebooting
  • 13:13 paravoid: rolling reboot ceph nodes for kernel upgrades
  • 13:06 paravoid: upgrading ceph to 0.67-rc3
  • 08:19 springle: restarting slave threads db32 db52 db39 db51 db45 db43 db56 after OSC bug 49194
  • 05:34 logmsgbot: catrope synchronized php-1.22wmf12/resources/startup.js 'touch'
  • 05:34 logmsgbot: catrope synchronized php-1.22wmf12/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.ViewPageTarget.init.js 'touch'
  • 05:32 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Add plumbing for new VE config vars'
  • 05:31 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Add VE config to set Edit source / Edit beta on enwiki'
  • 05:30 logmsgbot: catrope Finished syncing Wikimedia installation... : Update VE to master
  • 05:23 logmsgbot: catrope Started syncing Wikimedia installation... : Update VE to master
  • 02:35 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Aug 2 02:34:08 UTC 2013
  • 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Fri Aug 2 02:14:20 UTC 2013

August 1

  • 22:19 cmjohnson1: authdns update
  • 21:24 logmsgbot: mflaschen synchronized php-1.22wmf12/extensions/GuidedTour/ 'Sync GuidedTour for E3 deployment'
  • 21:11 LeslieCarr: restarting gerrit on manganese
  • 21:09 cmjohnson1: dns update
  • 21:06 Krinkle: Gerrit has been slow (~ 10 second per request) for about 30 minutes. Seems down entirely now.
  • 20:19 RobH: authdnsupdate
  • 19:40 cmjohnson1: dns update
  • 19:36 hashar: Slightly described the Zuul pipelines at https://integration.wikimedia.org/zuul/ . Descriptions are in the zuul-config layout.yaml file.
  • 19:23 cmjohnson1: morebots are you working?
  • 19:21 cmjohnson1: authdns update
  • 18:54 akosiaris: created aggregates backups on nas1001-a and nas1-a. Created flexible volumes baculasd1 on each and setup a snapmirror relationship between them (from eqiad to pmtpa)
  • 18:42 cmjohnson1: authdns update
  • 17:29 logmsgbot: csteipp synchronized wmf-config/CommonSettings.php 'EventLogging for CentralAuth'
  • 17:28 logmsgbot: csteipp synchronized wmf-config/InitialiseSettings.php 'EventLogging for CentralAuth'
  • 16:49 akosiaris: set options disk.auto_assign off on nas1001-{a,b}. Then performed disk remove_ownership for 20 disks on nas1001-b and disk assign all on nas1001-a
  • 16:45 logmsgbot: bsitu synchronized php-1.22wmf12/extensions/Echo/Echo.php 'Update Echo to master'
  • 16:05 akosiaris: added my ssh public key to root user of nas1-{a,b}, nas1001-{a,b}
  • 15:32 mark: Prepending AS 14907 once to AS 30217
  • 15:12 Jeff_Green: accidentally powercycled iron, trying to do iodine. fail.
  • 06:12 springle: delaying db32 during OSC bug 49194
  • 06:06 springle: delaying db52 during OSC bug 49194
  • 05:55 springle: delaying db39 during OSC bug 49194
  • 05:44 springle: delaying db51 during OSC bug 49194
  • 05:41 springle: delaying db45 during OSC bug 49194
  • 05:37 springle: delaying db43 during OSC bug 49194
  • 05:29 springle: delaying db56 during OSC bug 49194
  • 05:16 Tim: deleted half-created account melissajan0@enwiki, broken by bug 52368
  • 04:42 ori-l: Not synced to srv281, mw1089, mw1173 (connection timed out).
  • 04:41 logmsgbot: olivneh synchronized php-1.22wmf12/extensions/VisualEditor/VisualEditor.php 'Fix for bug 52368: VisualEditor.php attempts to register a hook handler that was removed'
  • 02:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Aug 1 02:34:57 UTC 2013
  • 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Thu Aug 1 02:15:31 UTC 2013
  • 01:55 logmsgbot: catrope synchronized php-1.22wmf12/extensions/VisualEditor 'Update VE to master'

Archives