Server Admin Log/Archive 20

June 30

• 16:12 mark: Temporarily added path 6939+ 14907+ to AVOID-PATHs on cr2-knams
• 02:53 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Sat Jun 30 02:53:46 UTC 2012
• 02:28 maplebed: corrected LVS pdns_recursor config error causing DNS queries to fail on LVS servers in gerrit r13554 and r13555.
• 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Sat Jun 30 02:27:08 UTC 2012

June 29

• 19:49 hashar: restarting Jenkins to fix an issue with "parameterized builds" plugin. Updated git plugin as well.
• 19:35 RobH: dns update via authdns-update for vanadium ip
• 18:05 Jeff_Green: sync-apache and apache-graceful-all for http://donate.wikimedia.org-->https redirect
• 16:02 RobH: ms-be1001 and ms-be1002 powering down for ssd installation
• 15:59 RobH: authdns-update run
• 15:47 RobH: updating dns
• 15:16 mutante: dist-upgrading srv280,srv270,srv264
• 15:11 Jeff_Green: apache-graceful-all for redirect conf change
• 15:10 Jeff_Green: sync-apache to push out new foundation.conf
• 14:50 mark: Reinstalled chromium with precise
• 13:46 hashar: fixed interwiki on http://wikisource.org/ main page by hacking a script in production and refreshing cache
• 13:46 logmsgbot: hashar synchronized php-1.20wmf6/cache/interwiki.cdb 'Updating interwiki cache for 1.20wmf6'
• 13:32 logmsgbot: hashar synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
• 12:38 logmsgbot: dzahn synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
• 12:38 mutante: dumping interwiki and updating interwiki cache (to fix broken interwiki links, like wikisource.org -> wikipedia.org)
• 09:31 hashar: Jenkins: deployed gitsqlhaschanged patch ( d04f779 0f069c3 integration/jenkins.git )
• 07:56 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'send header from CS.php only for non CLI scripts 13435'
• 07:08 mutante: upgrading apt packages on brewster
• 02:51 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Fri Jun 29 02:51:13 UTC 2012
• 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Fri Jun 29 02:26:25 UTC 2012

June 28

• 23:30 logmsgbot: reedy synchronized php-1.20wmf6/includes/resourceloader/ResourceLoader.php
• 23:15 binasher: completed aft offload_large_feedback migration on enwiki
• 23:03 logmsgbot: asher synchronized wmf-config/db.php 'returning db36'
• 21:50 logmsgbot: kaldari Finished syncing Wikimedia installation... :
• 21:39 logmsgbot: kaldari Started syncing Wikimedia installation... :
• 21:13 logmsgbot: asher synchronized wmf-config/db.php 'temp pulling db36'
• 20:38 binasher: ran aftv5 offload_large_feedback migrations on testwiki and en_labswikimedia
• 20:14 RobH: dns update for pc1-pc3
• 19:54 logmsgbot: kaldari Finished syncing Wikimedia installation... :
• 19:08 logmsgbot: kaldari Started syncing Wikimedia installation... :
• 18:47 hashar: the internal change to CommonSettings.php caused a lack of stylesheet for less than a minute on most wikis. I did test on test.wikipedia.org and beta project, but there must be a logic error somewhere that mess with the prod projects. Revert changes have been sent out in gerrit and merged in master.
• 18:35 hashar: so the nicely reviewed changes broke the enwiki stylesheets :/ reverted change :-(((
• 18:34 logmsgbot: hashar synchronized wmf-config/CommonSettings.php
• 18:33 hashar: srv190 and srv281 got ssh timeout
• 18:31 logmsgbot: hashar synchronized wmf-config/CommonSettings.php
• 18:30 hashar: did various tests using eval.php. Most important is $realm -> production. $cluster -> pmtpa. Syncing
• 18:25 hashar: updating mediawiki-config to grab a12545d edceb4c & eee97ad
• 18:23 RobHalsell: swapped bad psu out of ms1001-array3, redundant so no downtime
• 15:40 RobHalsell: pulling the following servers, relocating to payments rack: payments1001-1004, boron, beryllium, lithium
• 15:34 RobHalsell: dns updated
• 15:31 RobHalsell: boron appears to be unallocated, pulling IP allocation, rack allocation, moving to payments per 1227
• 14:40 mutante: svn server is rebooting.brb
• 14:38 mutante: dist-upgrading formey (svn/gerrit), rebooting soon
• 14:38 Jeff_Green: manganese rebooted for kernel update
• 14:37 RobH: allocating yttrium to payments rack per rt 1227
• 14:24 Jeff_Green: manganese dist-upgrade
• 14:15 Ryan_Lane: restarting apache on manganese
• 14:15 Ryan_Lane: restarting gerrit
• 05:08 Tim: srv266 was flooding the fatal error log, complaining about a missing file. Killed apache and ran sync-common.
• 05:03 Tim: fixed fatal.log on fenari, socat was writing to a deleted file
• 02:58 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Thu Jun 28 02:58:42 UTC 2012
• 02:30 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Thu Jun 28 02:30:03 UTC 2012

June 27

• 23:50 K4-7131: sync'd payments cluster to 592e0a5ba195
• 23:43 logmsgbot: preilly synchronized wmf-config/InitialiseSettings.php 'add rule for mediawiki'
• 22:37 K4-7131: sync'd payments cluster to 7e9072c2d571c
• 22:23 binasher: temporarily pulling srv211 from pybal
• 21:56 RobH: mw1102 has no nic0, rather than troubleshoot it for a long time, reinstall! (rt 3058)
• 21:01 logmsgbot: preilly synchronized php-1.20wmf6/extensions/MobileFrontend 'fix CSRF'
• 21:00 logmsgbot: preilly synchronized php-1.20wmf5/extensions/MobileFrontend 'fix CSRF'
• 20:55 RobH: db1003 back online, replaced mgmt cable and mgmt is working now as well
• 20:51 LeslieCarr: rebooting srv266 as it is unresponsive
• 20:44 RobH: db1003 mgmt issue due to bad cable, system booting back up, replacing mgmt cable
• 20:35 RobH: clean mysql shutdown, db1003 now offline
• 20:33 RobH: db1003 mgmt is not responsible, I need to remove power and reboot. confirmed iwth asher this is an s3 slave and can do a short downtime without issues
• 20:31 logmsgbot: maxsem synchronized php-1.20wmf5/extensions/MobileFrontend/ 'MF fixes and logging'
• 20:29 logmsgbot: maxsem synchronized php-1.20wmf6/extensions/MobileFrontend/ 'MF fixes and logging'
• 19:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'disable LastModified and LastModified/E3Experiment'
• 19:54 logmsgbot: reedy synchronized php-1.20wmf6/maintenance/runJobs.php
• 19:53 logmsgbot: reedy synchronized php-1.20wmf5/maintenance/runJobs.php
• 19:32 logmsgbot: reedy synchronized php-1.20wmf6/extensions/Translate/specials/SpecialLanguageStats.php
• 19:19 logmsgbot: reedy synchronized php-1.20wmf6/extensions/Translate/
• 19:06 logmsgbot: reedy synchronized php-1.20wmf6/extensions/Translate/
• 19:03 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: meta back to wmf6, not cause of translate issues
• 18:53 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: closed to 1.20wmf6
• 18:51 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikimedia wikis to 1.20wmf6
• 18:48 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary and wikiversity to 1.20wmf6
• 18:47 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource and wikiquote to 1.20wmf6
• 18:47 Jeff_Green: added several mobile hostnames to DNS for RT #2996
• 18:46 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews and wikibooks to 1.20wmf6
• 18:44 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved metawiki back to 1.20wmf5
• 18:41 K4-713: synchronized payments cluster to fundraising/1.20 de0256084a
• 18:33 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved special wikis to php-1.20wmf6
• 18:29 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved en(wikibooks|wikinews|wikiquote|wikisource|wikiversity|wiktionary) to 1.20wmf6
• 18:05 RobH: cp1017 memory replaced
• 17:52 RobH: cp1017 is offline due to memory error. replacement memory on site, pulling system for swap
• 17:46 logmsgbot: preilly synchronized php-1.20wmf6/extensions/ZeroRatedMobileAccess 'update for landing page'
• 17:46 logmsgbot: asher synchronized wmf-config/mc.php 'disabling wgMemCachedPersistent; lowering wgMemCachedTimeout to 2x client default from 30x default'
• 17:45 logmsgbot: preilly synchronized php-1.20wmf5/extensions/ZeroRatedMobileAccess 'update for landing page'
• 17:44 logmsgbot: preilly synchronized php-1.20wmf4/extensions/ZeroRatedMobileAccess 'update for landing page'
• 17:19 LeslieCarr: restarting apache2 on srv258
• 17:11 maplebed: powercycled srv270
• 17:11 mutante: powercycling srv277 (had to, frozen console)
• 17:06 LeslieCarr: rebooting srv287
• 17:05 Ryan_Lane: rebooting srv280
• 17:03 mutante: powercycling srv280
• 17:01 paravoid: rebooting srv264, swapdeath
• 16:52 mark: Rebooting srv279, swapdeath
• 16:52 paravoid: rebooting srv275, swapdeath
• 16:50 paravoid: rebooting srv258, swapdeath
• 16:43 logmsgbot: reedy synchronized php-1.20wmf5/extensions/ArticleFeedbackv5/
• 15:08 Reedy: ExtensionDistributor now works from git on mediawiki.org
• 15:08 logmsgbot: reedy synchronized php-1.20wmf6/extensions/ExtensionDistributor/
• 15:06 logmsgbot: reedy synchronized php-1.20wmf5/extensions/ExtensionDistributor/
• 15:01 logmsgbot: reedy synchronized php-1.20wmf6/extensions/ExtensionDistributor/
• 14:51 logmsgbot: reedy synchronized php-1.20wmf6/extensions/ExtensionDistributor/ 'ED to trunk'
• 14:46 mark: Rebooting lvs1005 (after dist-upgrade)
• 13:28 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild message cache for WikimediaShopLink
• 13:24 mark: Added IPv6 LVS service IPs to the LVS_import policy on cr2-eqiad, for testing with lvs1005
• 13:22 Tim: installing apache2.2-bin-dbgsym on mw1
• 13:04 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild message cache for WikimediaShopLink
• 13:04 mark: Started PyBal 1.02 snapshot build on lvs1005
• 12:39 Reedy: WikimediaShopLink is deployed to testwiki/test2wiki
• 11:59 apergos: kicked morebots
• 13.56.44 (CEST) <logmsgbot> !log reedy synchronized wmf-config/ 'WikimediaShopLink'
• 13.54.28 (CEST) <logmsgbot> !log reedy synchronized php-1.20wmf6/extensions/WikimediaShopLink/
• 13.44.37 (CEST) <logmsgbot> !log reedy synchronized php-1.20wmf6/extensions/WikimediaShopLink/
• 11.36.23 (CEST) <mutante> !log starting swift-container-auditor on ms-be3
• 10.29.51 (CEST) <mutante> !log apt-get upgrade on gallium, installs newer jenkins
• 10.26.41 (CEST) <mutante> !log importing jenkins_1.472_all.deb into lucid-wikimedia using reprepro
• 08.04.26 (CEST) <logmsgbot> !log tstarling synchronized wmf-config/CommonSettings.php
• 08.00.45 (CEST) <logmsgbot> !log tstarling synchronized wmf-config/CommonSettings.php
• 04.48.48 (CEST) <logmsgbot> !log LocalisationUpdate completed (1.20wmf6) at Wed Jun 27 02:48:51 UTC 2012
• 02:25 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Wed Jun 27 02:24:59 UTC 2012
• 01:23 Tim: on manganese: restarting gerrit
• 01:02 Tim: on manganese: killing all gitweb.cgi processes

June 26

• 23:49 Tim: on fenari: doing git and 1.19 checkouts for ExtensionDistributor
• 22:17 JeLuF: Slowly starting to import 100,000 images from the Deutsche Fotothek into Commons using importImages.php on fenari as user jeluf.
• 14:07 mutante: shutting down unused cp1037-cp1040 per RT-3189
• 10:48 mark: Moving all API traffic back to API apaches
• 10:41 mark: Restarted gmetad on nickel
• 10:36 apergos: powercycling srv261
• 02:47 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Tue Jun 26 02:47:56 UTC 2012
• 02:25 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Tue Jun 26 02:25:02 UTC 2012
• 01:05 logmsgbot: preilly synchronized php-1.20wmf6/extensions/MobileFrontend 'fix css'
• 01:04 logmsgbot: preilly synchronized php-1.20wmf5/extensions/MobileFrontend 'fix css'
• 01:03 logmsgbot: preilly synchronized php-1.20wmf4/extensions/MobileFrontend 'fix css'
• 00:59 LeslieCarr: ignoring cp1037 to cp1040 alarms for now as they are unused
• 00:45 LeslieCarr: rebooting cp1040
• 00:45 LeslieCarr: rebooting cp1039
• 00:43 LeslieCarr: rebooting cp1037

June 25

• 22:49 logmsgbot: maxsem synchronized php-1.20wmf5/extensions/MobileFrontend/ 'Weekly MF deployment'
• 22:40 logmsgbot: maxsem synchronized php-1.20wmf6/extensions/MobileFrontend/ 'Weekly MF deployment'
• 22:36 maplebed: powercycling ms1002 - it's unresponsive to ssh and on the console though it does respond to a ping.
• 21:31 Jeff_Green: manual apache restart on srv265, srv277
• 21:21 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 37345 - Request: Enable Ext:Collection on mk.wiki'
• 21:07 LeslieCarr: rebooting neon
• 21:01 hashar: Triggered several jobs on Jenkins to run tests on change that did not received their blame stick token
• 21:00 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 37507 - Babel configuration for tl.wikipedia'
• 20:58 Jeff_Green: pushing out new redirects.conf adjusted for RT #3138
• 20:52 logmsgbot: reedy synchronized wmf-config/ 'Various site config bugs'
• 20:13 logmsgbot: reedy synchronized php-1.20wmf6/extensions/Math/ 'Updating math to master'
• 20:12 logmsgbot: reedy synchronized php-1.20wmf5/extensions/Math/ 'Updating math to master'
• 20:07 logmsgbot: reedy synchronized php-1.20wmf6/extensions/WikiEditor/
• 20:06 logmsgbot: reedy synchronized php-1.20wmf5/extensions/WikiEditor/
• 19:17 LeslieCarr: rebooting unresponsive gallium
• 18:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.20wmf6
• 17:27 logmsgbot: reedy Finished syncing Wikimedia installation... : Take 2
• 16:43 logmsgbot: reedy Started syncing Wikimedia installation... : Take 2
• 16:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable EducationProgram on enwiki per request'
• 16:30 logmsgbot: reedy synchronized php-1.20wmf6/extensions/EducationProgram/ 'sync education programf iles'
• 16:28 logmsgbot: reedy Finished syncing Wikimedia installation... : Scapping to rebuild message cache for 1.20wmf6
• 16:08 logmsgbot: reedy Started syncing Wikimedia installation... : Scapping to rebuild message cache for 1.20wmf6
• 16:03 logmsgbot: reedy synchronized php-1.20wmf6 'Syncing php-1.20wmf6'
• 15:43 Reedy: Copying php-1.20wmf6 from /tmp to NFS /home on fenari
• 13:52 Reedy: Killed php-1.20wmf4/cache/l10n from mediawiki-installation hosts
• 12:26 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 37699) Chage logo on uzwiki'
• 09:38 mutante: so the several redirects for education->outreach requested to work by today look good now. RT-3138
• 09:30 mutante: apache-graceful-all to push out needed redirects for education
• 09:23 mutante: looking good. running sync-apache
• 09:20 mutante: creating dsh group "testwikipedia" with just srv193, creating sync-apache-test to just sync there...testing sync
• 02:49 Tim: configured mediawiki-commits to discard mails from gerrit pending resolution of the "implicit destination" issue
• 02:24 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Mon Jun 25 02:24:46 UTC 2012

June 24

• 18:21 logmsgbot: reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
• 02:25 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Sun Jun 24 02:25:23 UTC 2012

June 23

• 13:28 apergos: powrcycling srv288, swap death etc, some message to mgmt console but only the timestamp so couldn't see the issue, also couldn't get past the login prompt
• 02:24 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Sat Jun 23 02:24:11 UTC 2012

June 22

• 23:11 LeslieCarr: restarted apache on srv278
• 22:23 binasher: stopping mysql on es3, reseeding slave via innodb hotbackup of es1004
• 18:57 logmsgbot: preilly synchronized docroot/bits
• 18:36 LeslieCarr: removing 28790 bounce messages from exim queue on mchenry
• 16:50 Ryan_Lane: added a database account on db9/10 for read-only access to the gerrit database
• 12:06 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 37700) update stewardwiki logo & favicon'
• 11:44 mutante: installing upgrades and kernel on pdf1, can reboot? (also needs puppetizing and precise reinstall)
• 10:46 mutante: installing security upgrades and kernel on bast1001 (still needs reboot, but dont break user sessions)
• 10:42 mutante: fenari upgrade - this included replace wikimedia-lvs-realserver 0.04 (using .../wikimedia-lvs-realserver_0.08
• 10:41 mutante: installing security upgrades on fenari
• 10:31 mutante: installing security upgrades on formey (gerrit)
• 08:49 logmsgbot: hashar synchronized wmf-config/CommonSettings.php '12569 Load transcode conf on -e /etc/wikimedia-transcoding (wmflabs change)'
• 08:40 logmsgbot: hashar synchronized wmf-config/CommonSettings.php '12568 Disable wgNoticeInfrastructure on beta cluster'
• 08:30 logmsgbot: hashar synchronized wmf-config/CommonSettings.php '12566 labs use the same wgCentralDBname on all wiki'
• 07:59 apergos: powercycled lvs1001, not pingable, nothing good from mgmt console, etc.
• 04:11 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php
• 02:24 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Fri Jun 22 02:24:15 UTC 2012
• 01:00 logmsgbot: catrope synchronized php-1.20wmf5/extensions/VisualEditor 'VisualEditor updates'
• 00:37 LeslieCarr: restarting exim4 on mchenry with split_spool_directory = true
• 00:30 logmsgbot: aaron synchronized wmf-config/PrivateSettings.php 'Updated swift user config.'

June 21

• 23:11 logmsgbot: catrope synchronized php-1.20wmf5/extensions/VisualEditor/modules/ve/ce/nodes/ve.ce.TextNode.js
• 23:07 logmsgbot: catrope synchronized php-1.20wmf5/extensions/VisualEditor/VisualEditor.php
• 22:49 logmsgbot: catrope synchronized php-1.20wmf5/extensions/VisualEditor/modules/ve/ce/nodes/ve.ce.TextNode.js
• 22:26 logmsgbot: catrope synchronized php-1.20wmf5/extensions/VisualEditor/modules/ve/ce/nodes/ve.ce.TextNode.js
• 21:28 notpeter: restarting all lucene instances to direct logs to oxygen
• 21:16 binasher: deploying new mobile redirector to esams text squids
• 21:00 logmsgbot: catrope synchronized php-1.20wmf5/extensions/VisualEditor 'VisualEditor bugfixes'
• 20:34 logmsgbot: reedy synchronized wmf-config/
• 20:29 binasher: deployed new squid mobile redirector, now covers additional projects
• 20:27 logmsgbot: reedy synchronized wmf-config/
• 20:10 logmsgbot: hashar: on gallium, cloning mediawiki/extensions.git to /var/lib/jenkins/jobs/MediaWiki-Extensions-Fetching/workspace
• 19:25 mark: Restarted 3 queue runners as exim -qff &
• 19:15 logmsgbot: catrope Finished syncing Wikimedia installation... : VisualEditor updates
• 18:57 logmsgbot: catrope Started syncing Wikimedia installation... : VisualEditor updates
• 18:50 mark: Started 5 exim queue runners on mchenry with exim -qqff &
• 18:49 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Point $wgVisualEditorParsoidURL to cadmium'
• 18:36 logmsgbot: catrope synchronized php-1.20wmf5/includes/OutputPage.php 'Core patches for VisualEditor deploy'
• 18:36 logmsgbot: catrope synchronized php-1.20wmf5/resources/Resources.php 'Core patches for VisualEditor deploy'
• 18:36 logmsgbot: catrope synchronized php-1.20wmf5/resources/mediawiki.page/mediawiki.page.watch.ajax.js 'Core patches for VisualEditor deploy'
• 17:51 notpeter: restarting puppet on brewster
• 15:25 notpeter: stopping puppet on brewster
• 14:11 paravoid: powercycling srv272, unreachable due to load spike
• 05:30 binasher: clearing mobile varnish cache - my friend can't expand some article categories on his iphone after rebooting and clearing cache
• 04:41 logmsgbot: catrope synchronized php-1.20wmf4/extensions/LastModified/modules/lastmodified.js
• 04:40 logmsgbot: catrope synchronized php-1.20wmf5/extensions/LastModified/modules/lastmodified.js
• 02:25 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Thu Jun 21 02:25:40 UTC 2012
• 00:10 binasher: stopped puppet on cp1020 until tomorrow - testing new build of the squid mobile redirector on one server until tomorrow

June 20

• 23:29 logmsgbot: kaldari synchronized php-1.20wmf5/extensions/LastModified/E3Experiments/js/ext.E3Experiments.Timestamp.js 'updating clicktracking for LastModified and E3Experiments exts'
• 21:52 Reedy: pointed /usr/local/apache/common/php at /usr/local/apache/common/php-1.20wmf5 on mediawiki-installation
• 21:49 LeslieCarr: see RT3170 for more details on above change and mchenry pain
• 21:48 LeslieCarr: freezing many bounce messages on mchenry (all older than 2400 minutes)
• 21:04 LeslieCarr: replaced srv268 with srv245 in memcached list
• 21:04 logmsgbot: lcarr synchronized wmf-config/mc.php 'removed broken srv268'
• 21:03 paravoid: powercycling srv268; unreachable due to load spike
• 18:31 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files:
• 18:23 LeslieCarr: reloading mr1-pmtpa for sw upgrade (fixing a cpu bug)
• 18:18 logmsgbot: reedy synchronized wikiversions.dat
• 18:17 logmsgbot: reedy synchronized wikiversions.cdb
• 18:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Rest of pedias to 1.20wmf5
• 18:03 logmsgbot: preilly synchronized php-1.20wmf4/extensions/MobileFrontend 'test with disable caching on'
• 18:02 logmsgbot: preilly synchronized php-1.20wmf5/extensions/MobileFrontend 'test with disable caching on'
• 17:59 logmsgbot: preilly synchronized php-1.20wmf4/extensions/MobileFrontend 'testing with disable caching off'
• 17:58 logmsgbot: preilly synchronized php-1.20wmf5/extensions/MobileFrontend 'testing with disable caching off'
• 17:42 logmsgbot: preilly synchronized docroot/bits
• 17:11 logmsgbot: preilly synchronized wmf-config/mobile.php 'add Grameenphone Bangladesh'
• 17:08 logmsgbot: preilly synchronized wmf-config/mobile.php 'add telenor'
• 14:00 logmsgbot: reedy synchronized php-1.20wmf5/extensions/EducationProgram/
• 13:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable EducationProgram on enwiki *gulp*'
• 13:41 logmsgbot: reedy synchronized php-1.20wmf5/extensions/EducationProgram/ 'Push out master EP'
• 13:37 logmsgbot: reedy synchronized php-1.20wmf5/extensions/EducationProgram/ 'Push out master EP'
• 13:19 Reedy: Created EducationProgram database tables on enwiki
• 12:40 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'touching InitialiseSettings.php to refresh cache'
• 12:39 logmsgbot: hashar synchronized wmf-config/throttle.php '(bug 37740) raise account throttle for an edit marathon'
• 09:43 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 37457) viwikibooks can import from fr/it wikibooks'
• 09:40 logmsgbot: hashar synchronized wmf-config/mobile.php '$wgMobileResourceVersion does not exist anymore'
• 09:25 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 37327) Configure chr.wikipedia site logo'
• 04:35 Tim: on nickel: there were data sources for both "Apaches 8 CPU" and "Application servers", these were getting the same cluster name from the remote gmonds, and so different threads in gmetad were trying to write to the same summary files. Fixed temporarily, will fix in puppet shortly
• 04:26 Tim: on nickel: ran gmetad with -d3, it spews errors when trying to write to the faulty summary info files
• 04:20 Tim: on nickel: restarting gmetad
• 04:19 Tim: on srv258: started gmond
• 04:12 Tim: experimentally stopping gmond on srv258 to check for effects on oscillating appserver stats
• 03:23 Tim: on fenari, queueing refreshLinks jobs for some 2.8M commons image description pages that use location templates
• 02:48 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Wed Jun 20 02:48:41 UTC 2012
• 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Wed Jun 20 02:26:26 UTC 2012
• 02:24 Tim: started socat for /var/log/mw/fatal.log on fenari
• 01:23 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php

June 19

• 23:37 paravoid: temporarily adding wikimedia.org, wikipedia.org etc. to sodium's /etc/exim4/defer_domains
• 23:09 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'bug 37611, plan B'
• 22:55 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'bug 37611'
• 21:07 Jeff_Green: deployed a hacked up exim conf on sodium to block a mail ddos, puppet disabled there too
• 20:37 logmsgbot: mlitn synchronized php-1.20wmf5/extensions/ArticleFeedbackv5 'desc'
• 19:45 logmsgbot: mlitn Finished syncing Wikimedia installation... : Update ArticleFeedbackv5 to master
• 19:28 logmsgbot: mlitn Started syncing Wikimedia installation... : Update ArticleFeedbackv5 to master
• 19:13 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php 'Enable AFTv4 on testwiki'
• 18:06 maplebed: failed out ms-be5 after failed ssd test
• 14:50 RobH: updating dns
• 14:40 RobH: db14 is out of rotation, shutting down to make room for new es servers in rack
• 14:03 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'Syncing misc changes
• 11:03 Ryan_Lane: adding IPs for virt6-8
• 10:11 logmsgbot: nikerabbit Finished syncing Wikimedia installation... : Updating TranslationNotifications
• 10:05 hashar: TranslationNotifications extension updated by Nikerabbit!
• 09:56 logmsgbot: nikerabbit Started syncing Wikimedia installation... : Updating TranslationNotifications
• 09:41 hashar: updating TranslationNotifications extension with NikeRabbit
• 09:04 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Merge I8572d5f4 to fix workflowstates in Translate'
• 07:27 apergos: reboot snapshot1, package and kernel updates
• 07:14 apergos: reboot snapshot2, package and kernel updates
• 02:47 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Tue Jun 19 02:47:42 UTC 2012
• 02:24 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Tue Jun 19 02:24:46 UTC 2012
• 01:34 logmsgbot: tstarling Finished syncing Wikimedia installation... :
• 00:53 logmsgbot: tstarling Started syncing Wikimedia installation... :
• 00:43 Tim: put DolphinBrowser files in docroot/bits (from preilly's Ie3fefec6) and now running scap

June 18

• 22:31 K4-713: updated production civicrm to r1814
• 22:26 logmsgbot: aaron synchronized php-1.20wmf5/extensions/FlaggedRevs 'deployed e53310f548cf3f3e4f1ddfa10f5efd0eff06eeec'
• 22:17 maplebed: rebooting es1002 to look at the raid setup
• 20:43 logmsgbot: preilly synchronized php-1.20wmf4/extensions/MobileFrontend 'update to remove bad code'
• 20:43 logmsgbot: preilly synchronized php-1.20wmf5/extensions/MobileFrontend 'update to remove bad code'
• 19:11 hashar: updating several Jenkins plugins
• 19:06 RobH: updating dns
• 18:52 logmsgbot: maxsem synchronized php-1.20wmf5/extensions/MobileFrontend/
• 18:51 logmsgbot: maxsem synchronized php-1.20wmf4/extensions/MobileFrontend/
• 18:23 logmsgbot: reedy synchronized wikiversions.dat
• 18:15 logmsgbot: reedy synchronized wikiversions.cdb 'sync using sync-file'
• 18:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf5
• 16:30 mutante: installing package upgrades on sodium
• 16:29 mutante: restarting lighttpd on sodium - redirecting mediawiki-cvs list page
• 16:05 binasher: rebooting es1001
• 15:59 mutante: there have been no archives, so that should be it. there may be another issue in BZ 37690 but should be unchanged by renaming
• 15:58 mutante: copied full config/users/passes from mediawiki-cvs to mediawiki-commits, merged redirects, added old list name to acceptable_aliases in recipient filters
• 15:52 mutante: making the mailing list switch. mediawiki-cvs -> mediawiki-commits
• 15:49 Ryan_Lane: assigned service IPs for labs-ns0/labs-ns1
• 15:25 binasher: rebooting es1002 and es1003
• 15:11 Ryan_Lane: added virt1000 as a secondary ldap server for labsconsole
• 15:08 Ryan_Lane: testing gerrit config with multiple ldap servers
• 14:52 hashar: hume is out of disk space again. Probably the wmf branches taking toooo much space
• 14:52 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 37662) change wgUploadNavigationUrl @ dawiki'
• 14:21 mutante: creating new list MediaWiki-commits, not in use yet, but will replace outdated -cvs list soon
• 14:18 apergos: reboot snapshot3, package and kerne updates
• 14:13 apergos: rebooting snapshot4, kernel and other updates
• 09:31 Ryan_Lane: added virt1000 to dns, using titanium misc server
• 08:17 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php ' (bug 37672) Use odf on collection for ml projects '
• 06:03 apergos: powercycling db1047
• 02:47 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Mon Jun 18 02:47:49 UTC 2012
• 02:25 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Mon Jun 18 02:25:09 UTC 2012

June 17

• 02:45 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Sun Jun 17 02:45:15 UTC 2012
• 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Sun Jun 17 02:23:01 UTC 2012

June 16

• 02:49 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Sat Jun 16 02:49:55 UTC 2012
• 02:24 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Sat Jun 16 02:24:56 UTC 2012
• 00:34 paravoid: esams SSL should be back up
• 00:25 paravoid: SSL ipv6 access logs disabled; force running puppet and rm'ing access.logs on esams
• 00:08 paravoid: esams SSL outage, working on it

June 15

• 20:56 LeslieCarr: attaching asw-c1-pmtpa to asw-d-pmtpa ring
• 20:41 RobHalsell: updating dns for mc1-mc16 mgmt
• 20:07 RobH: mobing asw and msw-d3-sdtpa from single to dual power again, got sidetracked
• 19:59 RobH: mobing asw and msw-d3-sdtpa from single to dual power
• 19:24 RobH: updating dns for ms-be12 mgmt
• 19:18 logmsgbot: aaron synchronized php-1.20wmf5 'deployed 2755f255e45b53a083207d69c3e2d9fca62a3a1c'
• 19:15 paravoid: virt0: modify pdns.conf to listen on the old IP; temporarily disable puppet
• 19:15 paravoid: adding pre-renumbering virt0's IP back on eth1; doing policy routing to work out multihoming
• 18:54 RobH: updating dns for educacao redirect
• 18:53 LeslieCarr: deactivated rpf-filter on cr1-sdtpa and cr2-pmtpa temporarily for virt0
• 18:53 Ryan_Lane: doing a git pull for OpenStackManager on virt0
• 18:33 RobH: morebots, dont leave me again!
• 15:50 RobH: updating dns for new cisco machines
• 14:34 hashar: hume: 5.0G 5.0G 68K 100% /usr/local/apache
• 14:33 hashar: hume is out of disk space
• 14:32 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 34866) Change wgLanguageCode of several wikis to be renamed'
• 14:21 logmsgbot: hashar synchronized phpunit.xml
• 14:09 logmsgbot: hashar synchronized tests
• 13:39 Ryan_Lane: adding labs-ns0 and labs-ns1 dns entries
• 11:46 mark: csw1-esams.wikimedia.org line card 2 in trouble, power cycled it
• 02:48 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Fri Jun 15 02:48:46 UTC 2012
• 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Fri Jun 15 02:26:17 UTC 2012
• 00:00 binasher: rebooting / upgrading kernel on es1003 first

June 14

• 23:58 binasher: stopping mysql on es1003 and disabled notifications. going to convert to innodb via hotbackup of es1004 for testing
• 23:23 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'turning LastModified on for en.wiki'
• 23:10 logmsgbot: kaldari Finished syncing Wikimedia installation... :
• 23:00 logmsgbot: kaldari Started syncing Wikimedia installation... :
• 22:36 logmsgbot: kaldari synchronized php-1.20wmf5/extensions/LastModified/E3Experiments/js/ext.E3Experiments.Timestamp.js 'sycing E3Experiment js for wmf5'
• 22:34 logmsgbot: kaldari Started syncing Wikimedia installation... :
• 22:07 logmsgbot: kaldari Finished syncing Wikimedia installation... :
• 21:57 logmsgbot: kaldari Started syncing Wikimedia installation... :
• 21:16 RobH: updating dns for new mgmt ips and move of scs
• 20:37 logmsgbot: kaldari Finished syncing Wikimedia installation... :
• 20:33 logmsgbot: kaldari Started syncing Wikimedia installation... :
• 20:29 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update to MoodBar
• 20:07 logmsgbot: bsitu Started syncing Wikimedia installation... : Update to MoodBar
• 19:51 logmsgbot: py synchronized wmf-config/db.php 're-add db43 to s6 pool after kern upgrade'
• 19:28 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update to PageTriage
• 19:10 notpeter: rebooting db43 for kernel upgrading
• 19:08 logmsgbot: asher synchronized wmf-config/db.php 'returning db22'
• 18:58 logmsgbot: bsitu Started syncing Wikimedia installation... : Update to PageTriage
• 18:52 RobH: unracking and decommissioning db21 and db23
• 18:46 RobH: db22 relocated, powering up
• 18:43 RobH: db22 relocating
• 18:41 logmsgbot: asher synchronized wmf-config/db.php 'temp pulling db22 for hw move'
• 18:39 pgehres: re-enabled donation queue consumption after all updates
• 18:36 notpeter: pushing new dns zone file (minor change)
• 18:33 logmsgbot: preilly synchronized php-1.20wmf5/extensions/ZeroRatedMobileAccess 'update for landing page'
• 18:33 logmsgbot: preilly synchronized php-1.20wmf4/extensions/ZeroRatedMobileAccess 'update for landing page'
• 18:32 notpeter: new master log and pos for s6 MASTER_LOG_FILE='db47-bin.000230', MASTER_LOG_POS=876357616
• 18:27 logmsgbot: py synchronized wmf-config/db.php 'completed master switch for s6'
• 18:24 logmsgbot: preilly synchronized php-1.20wmf5/extensions/ZeroRatedMobileAccess 'update for landing page'
• 18:23 logmsgbot: preilly synchronized php-1.20wmf4/extensions/ZeroRatedMobileAccess 'update for landing page'
• 18:23 logmsgbot: py synchronized wmf-config/db.php 'switching master for s6 to db50'
• 18:11 Jeff_Green: erzurumi dist-upgrade & reboot [up 633 days]
• 18:02 logmsgbot: preilly synchronized php-1.20wmf4/extensions/MobileFrontend 'update for ie7'
• 18:01 logmsgbot: preilly synchronized php-1.20wmf5/extensions/MobileFrontend 'update for ie7'
• 18:00 Jeff_Green: aluminium/db1008 dist-upgrade & reboot
• 17:57 pgehres: disabled queue consumption on aluminum for dist-upgrade
• 17:03 mark: Copied udp-filter package from lucid-wikimedia to precise-wikimedia (but do as I say and rebuild, not as I do...)
• 16:40 binasher: running pagetriage_page schemea changes on enwiki and testwiki via osc (https://gerrit.wikimedia.org/r/#/c/11014/1/sql/PageTriagePagePatch.sql)
• 16:22 Jeff_Green: hume dist-upgrade & reboot
• 16:08 Jeff_Green: loudon dist-upgrade & reboot
• 16:04 mark: Manually disabled GRO on amslvs3/4 eth0
• 16:03 logmsgbot: reedy synchronized wmf-config/ 'Enable EP on test2wiki'
• 16:01 Ryan_Lane: restarting nova-compute on virt5
• 15:58 logmsgbot: reedy synchronized php-1.20wmf5/extensions/EducationProgram/
• 15:54 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild localisationcache for EP
• 15:49 Jeff_Green: grosley dist-upgrade & reboot
• 15:40 Jeff_Green: silicon dist-upgrade and reboot
• 15:29 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild localisationcache for EP
• 15:28 mark: Starting dist-upgrade of manutius to Precise
• 15:27 Ryan_Lane: to satisfy mark's pedantry that's lucid-wikimedia
• 15:27 mark: Ryan needs coffee
• 15:26 Ryan_Lane: specifically wikimedia-lucid repo
• 15:26 Ryan_Lane: added adminbot 1.2 to repo
• 15:00 logmsgbot: reedy synchronized php-1.20wmf5/extensions/EducationProgram/
• 14:30 mark: Reinstalled stat1 with Ubuntu Precise
• 11:21 pp-pdf1: updated mwlib.rl to 0.12.12
• 11:21 pp-pdf2: updated mwlib.rl to 0.12.12
• 11:21 pp-pdf3: updated mwlib.rl to 0.12.12
• 02:46 logmsgbot_: LocalisationUpdate completed (1.20wmf4) at Thu Jun 14 02:46:00 UTC 2012
• 02:23 logmsgbot_: LocalisationUpdate completed (1.20wmf5) at Thu Jun 14 02:23:26 UTC 2012
• 00:01 logmsgbot_: tstarling synchronized php-1.20wmf4/includes/DefaultSettings.php

June 13

• 23:39 logmsgbot_: maxsem synchronized php-1.20wmf5/extensions/MobileFrontend/ 'Deploying MF fix'
• 23:37 logmsgbot_: maxsem synchronized php-1.20wmf4/extensions/MobileFrontend/ 'Deploying MF fix'
• 21:49 logmsgbot_: reedy synchronized wmf-config/ 'More changes from gerrit'
• 21:35 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Bringing in numerous changes merged via gerrit'
• 21:32 logmsgbot_: lcarr synchronized wmf-config/mc.php 'replacing broken srv203 with working srv250'
• 21:31 LeslieCarr: replacing srv203 with srv250 in memcache rotation since srv203 is broken
• 21:20 logmsgbot_: aaron synchronized php-1.20wmf5/includes/WikiPage.php 'deployed 82742bccf3b5f2da0d5df05630eb31978afbbce1'
• 19:57 logmsgbot_: aaron synchronized php-1.20wmf4/includes/WikiPage.php
• 19:51 Jeff_Green: payments cluster dist-upgrades & reboots
• 19:43 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Added debug log.'
• 19:37 logmsgbot_: aaron synchronized php-1.20wmf4/includes/WikiPage.php
• 19:29 binasher: drop table enwiki.trackbacks
• 19:28 binasher: converted enwiki.interwiki to innodb
• 19:27 logmsgbot_: aaron synchronized php-1.20wmf4/includes/WikiPage.php 'temporary logging code.'
• 19:26 binasher: drop table enwiki.exlogging
• 18:35 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything non wikipedia to 1.20wmf5
• 18:33 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote to 1.20wmf5
• 18:31 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary to 1.20wmf5
• 18:30 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity to 1.20wmf5
• 18:22 logmsgbot_: reedy synchronized php-1.20wmf5/extensions/Vector/
• 18:18 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special.dblist wikis to 1.20wmf5
• 18:14 logmsgbot_: aaron synchronized php-1.20wmf5/extensions/FlaggedRevs 'deployed 537bb248bb93948844f195014227512f169a439b'
• 18:06 Jeff_Green: db1025 dist-upgrade & reboot
• 17:37 logmsgbot_: preilly synchronized php-1.20wmf5/extensions/ZeroRatedMobileAccess 'changes for zero needed for carrier testing'
• 17:36 logmsgbot_: preilly synchronized php-1.20wmf4/extensions/ZeroRatedMobileAccess 'changes for zero needed for carrier testing'
• 17:35 Ryan_Lane: restarting opendj on virt0 again...
• 17:29 Ryan_Lane: restarting opendj again on virt0
• 17:24 Ryan_Lane: restarting gerrit
• 17:23 LeslieCarr: rebooting stat1 for wipe and reinstall into precise
• 16:59 Ryan_Lane: restarting opendj on virt0
• 16:49 Ryan_Lane: restarting mysql on virt0 with correct bind address
• 16:46 LeslieCarr: changing virt0's ip address and vlan
• 16:38 cmjohnson1: shutting down search32 to replace main board
• 16:24 cmjohnson1: sq48 powercycled
• 16:19 mutante: shut down sq33
• 16:18 cmjohnson1: performing hard reset on sq33
• 16:12 mutante: adding gerrit@wikimedia.org to accepted nonmembers of mediawiki-cvs list
• 15:29 mark: Unstuck torrus
• 14:46 Ryan_Lane: lowering ttl for virt0
• 14:17 Jeff_Green: storage3 dist-upgrade and reboot
• 12:31 mutante: backing up wikitech dir locally on linode instance
• 09:24 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php '(bug 37482) Adding Proofread Page ext. namespaces on nl.wikisource'
• 09:08 hashar: finished deploying my wmflabs related change. mediawiki-config is now at commit c0baf3e
• 09:07 logmsgbot_: hashar synchronized wmf-config
• 09:06 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings-wmflabs.php
• 09:05 logmsgbot_: hashar synchronized wmf-config/throttle.php
• 09:05 logmsgbot_: hashar synchronized wmf-config/mobile-wmflabs.php
• 09:05 logmsgbot_: hashar synchronized wmf-config/mobile.php
• 08:47 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'bug 37006 - fawiki: add Book namespace + aliases'
• 08:45 hashar: reverted '(bug 37482) Adding Proofread Page ext. namespaces on nl.wikisource' --> used the wrong configuration setting.
• 08:37 mutante: installing samba-common-bin, smbclient package upgrades on tridge
• 08:33 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php '(bug 37482) Adding Proofread Page ext. namespaces on nl.wikisource'
• 08:28 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php '(bug 37482) Adding Proofread Page ext. namespaces on nl.wikisource'
• 08:24 hashar: deploying several changes made to mediawiki-config gerrit changes 11034 11035 9131 11036 11037 9132 9136 and 9237
• 02:52 logmsgbot_: LocalisationUpdate completed (1.20wmf5) at Wed Jun 13 02:52:38 UTC 2012
• 02:27 logmsgbot_: LocalisationUpdate completed (1.20wmf4) at Wed Jun 13 02:27:37 UTC 2012

June 12

• 23:58 maplebed: started swift container listing loop to compare purge timing when listings are fresh
• 23:16 logmsgbot_: preilly synchronized php-1.20wmf5/extensions/MobileFrontend/ 'try again'
• 23:15 logmsgbot_: preilly synchronized php-1.20wmf4/extensions/MobileFrontend/ 'try again'
• 22:58 logmsgbot_: maxsem synchronized php-1.20wmf4/extensions/MobileFrontend/ 'Updating MobileFrontend'
• 22:53 logmsgbot_: maxsem synchronized php-1.20wmf4/extensions/ZeroRatedMobileAccess/ 'Updating ZeroRatedMobileAccess'
• 22:52 logmsgbot_: maxsem synchronized php-1.20wmf4/extensions/MobileFrontend/ 'Updating MobileFrontend'
• 22:50 logmsgbot_: maxsem synchronized php-1.20wmf5/extensions/ZeroRatedMobileAccess/ 'Updating ZeroRatedMobileAccess'
• 22:49 logmsgbot_: maxsem synchronized php-1.20wmf5/extensions/MobileFrontend/ 'Updating MobileFrontend'
• 20:28 RoanKattouw: Correction: the /usr/local/apache filesystem is full on hume, the root fs is not
• 20:27 RoanKattouw: hume has a full disk
• 20:23 RoanKattouw: Fixed ownership of php-1.20wmf{4,5}/cache/l10n , should be l10nupdate:wikidev . The wmf4 copy had wrong ownership causing rebuildLocalisationCache.php to fail for shell users (e.g. from scap)
• 19:40 logmsgbot_: mlitn Finished syncing Wikimedia installation... :
• 19:13 logmsgbot_: mlitn Started syncing Wikimedia installation... :
• 18:09 logmsgbot_: py synchronized wmf-config/db.php 're-adding db25 to pool after kern upgares'
• 18:02 notpeter: halting owa3 for repairs
• 16:44 logmsgbot_: py synchronized wmf-config/db.php 'removing db25 from pools for kern upgares'
• 16:27 logmsgbot_: py synchronized wmf-config/db.php 're-adding dbs 33, 34, 36, 50, 55, 56 to pools after kern upgares'
• 15:55 RobH: virt1001 and virt1002 rebooting, disregard
• 15:48 cmjohnson1_: shutting down search32 to run a diagnostic test
• 15:45 binasher: migrating enwiki.bv2009_edits (?) to innodb
• 15:41 binasher: migrating enwiki.moodbar_feedback to innodb
• 15:39 binasher: migrating enwiki.aft_article_filter_count to innodb
• 15:33 logmsgbot_: py synchronized wmf-config/db.php 'removing dbs 33, 34, 36, 50, 55, 56 from pools for kern upgares'
• 15:31 notpeter: doing another round of DB kernel upgrades
• 15:25 logmsgbot_: py synchronized wmf-config/db.php
• 15:14 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'reenabling mysql pcache'
• 15:00 binasher: rebooting db40
• 14:52 binasher: set innodb_max_dirty_pages_pct = 0 on db40 in prep for shutdown
• 14:48 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'disabling mysql parsercache (db40) in order to perform maintenance'
• 14:45 notpeter: putting kern-upgraded DBs back into pools
• 14:44 binasher: resumed replication on es3, es1002 after cluster23 sync completed
• 13:41 Jeff_Green: added awight to fr-tech@wikimedia.org email alias
• 12:06 mutante: gerrit create-project --name=mediawiki/extensions/UniversalLanguageSelector --parent=mediawiki/extensions
• 11:28 mutante: powercycling downed srv232 (also cause for check_all_memcached crit)
• 11:08 mutante: powercycled mw1042 to check for hardware issues and fscked. appears to be just unused (though down since ~3d like mw1071 per nagios)
• 10:37 mutante: test to show linking from !log via SAL to RT: RT:3100 (before/without template)
• 10:31 hashar: incubatorwiki.translate_messageindex on db39 uses MyISAM engine. See RT #3100
• 10:25 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'bug 37391 , take 2 - Install Translate extension on be.wikimedia.org'
• 10:24 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'bug 37391 , take 2 - Install Translate extension on be.wikimedia.org'
• 10:23 hashar: Compared translate% tables schema on bewikimedia with incubatorwiki. diff prove they are the same so the schema changes made early are successful.
• 10:17 hashar: bewikimedia (db39) : dropped tables translate_tmf , translate_tms and translate_tmt I have incorrectly added
• 09:59 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'revert translate extension on be.wikimedia.org, need DB update'
• 09:58 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'bug 37391 - Install Translate extension on be.wikimedia.org'
• 06:24 apergos: db1047 looks like the aft_article_filter_count is missing a few rows compared to the master (after replication caught up), presumably this is a side effect of the repair, have pinged binasher for help, leaving everything running and hope it's tolerable error for a day
• 02:34 logmsgbot_: LocalisationUpdate completed (1.20wmf5) at Tue Jun 12 02:34:49 UTC 2012
• 02:25 logmsgbot_: LocalisationUpdate completed (1.20wmf4) at Tue Jun 12 02:25:21 UTC 2012
• 01:35 binasher: passes the dba mantel to notpeter
• 01:22 notpeter: removing one slave from each db shard to upgrade/restart
• 00:24 binasher: shutdown mysql on es3. stopped slaving on es1002, rsyncing cluster23 tables to es3
• 00:09 binasher: pointed es3 to MASTER_LOG_FILE='es1-bin.000788', MASTER_LOG_POS=453509865
• 00:05 binasher: es3:~# rm -rf /usr/local/mysql*

June 11

• 23:54 logmsgbot_: asher synchronized wmf-config/db.php 'fully commenting out es3'
• 23:52 logmsgbot_: asher synchronized wmf-config/db.php 'making es1 the master for blobs cluster 23'
• 23:51 binasher: es1 is the new master, now switching mw conf
• 23:48 binasher: preparing to switch es master to es1
• 23:31 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 27706 - enable RSS extension on uawikimedia'
• 23:12 logmsgbot_: reedy Finished syncing Wikimedia installation... :
• 21:49 Reedy: Applied PageTriage schema updates to testwiki and enwiki
• 20:54 logmsgbot_: reedy Started syncing Wikimedia installation... :
• 20:07 logmsgbot_: reedy synchronized php-1.20wmf5/
• 19:20 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki to 1.20wmf5
• 18:54 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki to 1.20wmf5 also
• 18:52 logmsgbot_: reedy synchronized php-1.20wmf5/ 'Now we have some more space...'
• 18:50 logmsgbot_: reedy synchronized php-1.20wmf3/cache/l10n/ 'Kill l10ncache for php-1.20wmf3 as its not needed'
• 18:45 logmsgbot_: reedy synchronized php-1.20wmf2/cache/l10n/ 'Kill l10ncache for php-1.20wmf2 as its not needed'
• 18:44 logmsgbot_: reedy synchronized php-1.20wmf5/ 'Scap is taking an age, just ensure deployment files are in sync'
• 18:42 binasher: resuming coversion of es1004 to innodb, using compact row format after testing dynamic and compressed
• 18:03 logmsgbot_: reedy Started syncing Wikimedia installation... : Consistency
• 17:56 Ryan_Lane: enabling TitleBlacklist on labsconsole
• 17:28 notpeter: moving /usr/local/apache to /a/apche with symbolic link on searchidx1001 as a temp measure until it can be reimaged
• 16:58 logmsgbot_: reedy Started syncing Wikimedia installation... : Running scap to ensure consistency
• 16:53 logmsgbot_: reedy synchronized php-1.20wmf5/cache/l10n/
• 16:28 mutante: installing security upgrades on sodium
• 16:19 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuild messagecache for 1.20wmf5
• 16:17 logmsgbot_: reedy synchronized wmf-config/ExtensionMessages-1.20wmf5.php
• 16:02 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuild messagecache for 1.20wmf5
• 15:55 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuild messagecache for 1.20wmf5
• 15:54 logmsgbot_: reedy Finished syncing Wikimedia installation... : Rebuild messagecache for 1.20wmf5
• 15:46 mutante: hume /usr/local/apache is out of disk (just 5GB but more branches now). (LVM vg "tank" lv "tank-apache" ) but no free extents. could take from /archive but unsure about shrinking the xfs.
• 15:35 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuild messagecache for 1.20wmf5
• 15:30 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.20wmf5
• 15:28 logmsgbot_: reedy synchronized php-1.20wmf5/
• 15:20 Reedy: running sync-dir php-1.20wmf5
• 15:13 Reedy: Copying checkout of 1.20wmf5 onto NFS
• 14:53 mutante: running puppet on stat1. installs plotting packages
• 09:56 apergos: shut down mysqld on db1047, reparing tables
• 06:54 pp-pdf2: upgraded mwlib to 0.13.8
• 06:54 pp-pdf3: upgraded mwlib to 0.13.8
• 06:54 pp-pdf1: upgraded mwlib to 0.13.8
• 02:24 logmsgbot_: LocalisationUpdate completed (1.20wmf4) at Mon Jun 11 02:24:06 UTC 2012

June 10

• 02:22 logmsgbot_: LocalisationUpdate completed (1.20wmf4) at Sun Jun 10 02:22:33 UTC 2012

June 9

• 07:45 logmsgbot_: LocalisationUpdate completed (1.20wmf4) at Sat Jun 9 07:45:17 UTC 2012
• 02:35 logmsgbot_: LocalisationUpdate completed (1.20wmf4) at Sat Jun 9 02:35:29 UTC 2012
• 02:25 Reedy: Running LU manually
• 02:14 Reedy: Cleared a bit of space on fenari by deleting checkouts from /tmp
• 02:00 logmsgbot_: LocalisationUpdate failed: git pull of core failed
• 01:51 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
• 01:50 logmsgbot_: reedy synchronized wikimedia.dblist
• 01:49 Reedy: fenari has a full /
• 01:49 logmsgbot_: reedy synchronized all.dblist

June 8

• 20:32 Reedy: Updated php to point to php-1.20wmf4 rather than php-1.20wmf3
• 20:09 logmsgbot_: reedy synchronized wikimedia.dblist
• 20:08 logmsgbot_: reedy synchronized all.dblist
• 20:04 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: 2 wikimedia wikis to 1.20wmf4
• 19:57 logmsgbot_: reedy Finished syncing Wikimedia installation... : Rebuilding localisation cache for message updates
• 19:39 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuilding localisation cache for message updates
• 18:47 logmsgbot_: reedy synchronized php-1.20wmf4/languages/messages/ 'Pushing out updated files upon siebrands request'
• 16:30 cmjohnson1: shutting down search32 to swap DIMM around
• 10:19 notpeter: ganglia down, restarting apache on nickel.
• 09:32 notpeter: stopping indexing on searchidx1001 to re-copy to searchidx2
• 08:45 notpeter: reimaging searchidx2 with correct partitioning
• 06:50 Tim: on cp1001: disabled HTCP plugin in gmond for testing, seems to work so I will disable it properly
• 03:55 Tim: disabled LastModified extension due to overload on cp1005
• 03:51 logmsgbot_: tstarling synchronized wmf-config/InitialiseSettings.php
• 03:18 Tim: restarting squid on cp1005, maybe out of FDs or something, cachemgr shows exactly 1000 open connections to 10.2.1.1
• 03:08 Tim: stopped gmond on cp1001 with kill -STOP for memory leak debugging
• 03:02 Tim: on cp1002: killed gmond again, it was leaking memory again, already up to 27GB in the few minutes since I restarted it
• 03:01 Tim: on fenari: copied *.text and *.upload from /home/wikipedia/conf/squid/generated/clusters to /etc/dsh/group
• 02:56 Tim: cp1001: same as on cp1002, restarted gmond
• 02:53 Tim: on cp1002: killed gmond, which was using 100% CPU and 23GB RSS. Restarting squid which had died
• 02:23 logmsgbot_: LocalisationUpdate completed (1.20wmf3) at Fri Jun 8 02:23:09 UTC 2012
• 02:14 logmsgbot_: LocalisationUpdate completed (1.20wmf4) at Fri Jun 8 02:14:55 UTC 2012

June 7

• 23:32 Tim: deploying varnish configuration change https://gerrit.wikimedia.org/r/#/c/10672/ on cp1041, cp1042, cp1043, cp1044
• 20:46 logmsgbot_: kaldari synchronized php-1.20wmf4/extensions/LastModified 'syncing LastModified extension'
• 20:32 logmsgbot_: kaldari synchronized wmf-config/InitialiseSettings.php 'turning on LastModified and E3Experiemnts for en.wiki'
• 19:54 logmsgbot_: kaldari synchronized php-1.20wmf4/extensions/LastModified/E3Experiments/js/ext.E3Experiments.Timestamp.js 'syncing js file for E3Experiments'
• 18:58 cmjohnson1_: shutting down search32 for testing
• 16:52 Ryan_Lane: added ldap automount entries for /public/datasets and /public/keys
• 02:19 logmsgbot_: LocalisationUpdate completed (1.20wmf3) at Thu Jun 7 02:19:52 UTC 2012
• 02:11 logmsgbot_: LocalisationUpdate completed (1.20wmf4) at Thu Jun 7 02:11:49 UTC 2012

June 6

• 22:29 logmsgbot_: aaron synchronized wmf-config/swift.php 'deployed 7dc77e431310580da0dbd368b8b290a293e3ee21'
• 19:55 cmjohnson1: shutting down search32 to reseat DIMM B2
• 19:18 cmjohnson1: shutting down storage3
• 18:05 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved remaining wikis to 1.20wmf4
• 17:02 Jeff_Green: mailman 'site' password changed per RT 3039
• 11:12 mark: Added AAAA record to mobile
• 10:38 mark: Wikipedia is IPv6-enabled.
• 10:37 mark: Added AAAA records to all non-mobile wiki projects
• 10:19 mark: Added AAAA record to bits.wikimedia.org
• 10:02 Ryan_Lane: repooling ssl3001
• 10:00 mark: Added AAAA record to upload.wikimedia.org
• 09:51 Ryan_Lane: depooling ssl3001
• 08:41 mark: Converted bits.wikimedia.org into a direct geodns record, removed the old bits -> bits-geo CNAME
• 07:53 mark: Converted geoiplookup.wikimedia.org into a separate, IPv4-only geodns record
• 07:37 pp-pdf1: installed tmpreaper cronjob for /home/pp/mathcache directory
• 07:37 pp-pdf2: installed tmpreaper cronjob for /home/pp/mathcache directory
• 07:37 pp-pdf3: installed tmpreaper cronjob for /home/pp/mathcache directory
• 02:35 logmsgbot_: LocalisationUpdate completed (1.20wmf3) at Wed Jun 6 02:35:47 UTC 2012
• 02:11 logmsgbot_: LocalisationUpdate completed (1.20wmf4) at Wed Jun 6 02:10:58 UTC 2012

June 5

• 22:37 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php 'Undo temporary woff whitelisting'
• 22:34 logmsgbot_: aaron synchronized php-1.20wmf4/includes/filerepo/file/LocalFile.php 'deployed 4791e3d25aebe9643a7cea91f2eb49e6b54593c5'
• 22:33 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php 'Temporarily allow uploading woff files on slwikisource'
• 22:13 logmsgbot_: kaldari synchronized php-1.20wmf4/extensions/PageTriage/modules/ext.pageTriage.models/ext.pageTriage.article.js 'updating default PageTriage filters'
• 20:56 logmsgbot_: mmullie Finished syncing Wikimedia installation... :
• 20:16 logmsgbot_: mmullie Started syncing Wikimedia installation... :
• 19:49 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php 'Fix notice in MobileFrontend config'
• 19:45 pp-pdf2: updated mwlib to 0.13.7-1-g827780b
• 19:45 pp-pdf1: updated mwlib to 0.13.7-1-g827780b
• 19:45 pp-pdf3: updated mwlib to 0.13.7-1-g827780b
• 19:45 pp-pdf2: updated mwlib to 0.13.7-1-g827780b
• 18:45 notpeter: starting innobackupex dump from blondel to bellin
• 18:44 notpeter: starting indexing on new searchidx2
• 18:39 mark: Added static routes 2002::/16 and 2001::/32 for 6to4 and teredo on the Tampa routers; these are redistributed in OSPF to eqiad
• 18:29 notpeter: restart indexing on searchidex1001
• 18:20 mark: Replaced static LVS IPv6 routes with correct next-hops on cr1-eqiad and cr2-eqiad
• 18:09 mark: Redistributing static routes in OSPF on cr1-eqiad and cr2-eqiad
• 18:04 paravoid: rebooting capella to make sure things work after a reboot
• 17:53 mark: Redistributed statics in OSPF3 on csw1-esams
• 17:14 logmsgbot_: aaron synchronized php-1.20wmf4/includes/logging/LogEventsList.php 'deployed d9f146ac42f2884e76390d6bc979eb10032adf7f'
• 17:09 mark: Added uRPF exception for 6to4 traffic on all routers
• 17:05 jeremyb: (UTC) 23:42:14 <binasher> !log re-enabled es4 monitoring. its currently our only es server without any tables marked as crashed / needing recovery, myisam recovery has been absent for all systems since the ms servers were migrated off of in nov 2011. (Sum of human knowledge * Renyi entropy = ES)
• 16:52 mark: Pooled ssl1001
• 15:44 logmsgbot_: asher synchronized wmf-config/db.php 'returning es2 to service'
• 15:25 paravoid: rebooting lvs1004 and reinstalling with precise
• 15:17 binasher: rebooting es2 for kernel + mysql upgrade
• 15:16 logmsgbot_: asher synchronized wmf-config/db.php 'pulling es2 for kernel+mysql upgrades'
• 14:56 paravoid: rebooting amslvs3 & amslvs4 to reinstall with precise
• 14:20 paravoid: rebooting lvs1006 to reinstall with precise
• 13:51 cmjohnson1: shutting down bellin to replace main board
• 13:49 notpeter: reimaging db1042
• 13:40 paravoid: rebooting lvs1005 to reinstall with precise
• 12:59 paravoid: rebooting lvs2 to reinstall with precise
• 12:47 Ryan_Lane: changing capella's subnet in DNS
• 12:10 Ryan_Lane: rebuilding capella as precise
• 10:01 logmsgbot_: asher synchronized wmf-config/db.php 'putting es1 in production'
• 09:53 notpeter: cancel that, it's mid-cron. will do later
• 09:52 notpeter: stopping indexing on searchidx1001 to rsync to searchidx2
• 09:35 binasher: rebooting es1 for kernel+mysql upgrade. dont need to pull from db.php because it was never correctly added or queried?
• 09:14 mark: Built PyBal 1.01 for precise, and included it in the precise-wikimedia APT repository
• 08:45 binasher: restarted mysql on es1004 with default innodb file format as barracuda
• 08:31 notpeter: reimaging searchidx2
• 02:36 logmsgbot_: LocalisationUpdate completed (1.20wmf3) at Tue Jun 5 02:36:44 UTC 2012
• 02:14 logmsgbot_: LocalisationUpdate completed (1.20wmf4) at Tue Jun 5 02:14:17 UTC 2012

June 4

• 23:23 logmsgbot_: asher synchronized wmf-config/db.php 'returning es4'
• 22:49 binasher: started an experiment on es1004 - altering all es tables from myisam to innodb one at a time with file_per_table enabled
• 22:39 binasher: stopping mysql on es4. all tables marked as having repair fails are in cluster22, resyncing just those from es1002
• 21:52 logmsgbot_: asher synchronized wmf-config/db.php 'pulling es4 again'
• 21:43 logmsgbot_: asher synchronized wmf-config/db.php 'returning es4 to service'
• 21:16 Ryan_Lane: restarting nginx on all ssl boxes again
• 21:07 Ryan_Lane: force running puppet on all ssl hosts again
• 21:04 Ryan_Lane: repooling ssl1, ssl1001, ssl3001
• 21:03 Ryan_Lane: restarting nginx on all ssl hosts
• 20:23 binasher: rebooted es4
• 20:18 logmsgbot_: asher synchronized wmf-config/db.php 'pulling es4 for post-crash upgrade'
• 19:55 logmsgbot_: kaldari synchronized php-1.20wmf4/extensions/PageTriage/PageTriage.hooks.php 'syncing PageTriage.hooks.php'
• 19:32 Ryan_Lane: force running puppet on ssl servers
• 18:22 Reedy: Nuked php-1.20wmf4 on mw64 then ran sync-common. Seems to have dealt with the permission errors
• 18:11 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf4
• 16:48 binasher: upgraded kernel on db1047 / analytics
• 16:09 Ryan_Lane: restarting ircecho on manganese
• 15:21 paravoid: reinstalling lvs1 with precise
• 15:13 mark: Added new IPv6 LVS prefixes to all routers for uRPF filters; BGP import filters still need adjusting for dual-family sessions
• 15:08 cmjohnson1: physically power cycling lvs1
• 15:02 Ryan_Lane: depooling ssl1001 and ssl3001
• 14:55 Ryan_Lane: disabling puppet on all ssl hosts
• 13:27 mark: Changed upload.esams.wikimedia.org CNAME to upload-lb.esams, effectively disabling the IPv6 selective answer script
• 12:23 mark: Upgrading wikimedia-lvs-realserver to version 0.08 across the cluster (by Puppet)
• 12:18 Ryan_Lane: depooling ssl1
• 11:32 mark: Copied wikimedia-lvs-realserver 0.08 from APT distribution precise-wikimedia to lucid-wikimedia
• 02:38 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Mon Jun 4 02:38:15 UTC 2012
• 02:14 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Mon Jun 4 02:14:45 UTC 2012

June 3

• 15:45 paravoid: aborting lvs1 install, partition map is not ready; putting it back to production as-is
• 15:31 paravoid: reinstalling lvs1 with precise
• 15:10 RobH: torrus failed to refresh via puppet (failed refresh takes too long) so manually running the refresh/rebuild command as puppet copied the updates to the system
• 14:55 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 37237 - Change Wikisource namespace for Tamil wikisource'
• 14:49 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bug 37211 - Set $wgUseCombinedLoginLink = false'
• 14:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 37294 - Add English Wikibooks as import source at Vietnamese Wikibooks'
• 14:07 logmsgbot: midom synchronized wmf-config/db.php
• 13:10 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '59753a9 (allow bureaucrats on frwiki to add+remove accountcreator group'
• 13:09 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'Commits: 6bef518 (wgHTCPMulticast only used on production cluster) and 882dd69 (wgLoadScript only used on production) -- was not correctly deployed earlier'
• 13:02 logmsgbot: midom synchronized wmf-config/db.php
• 10:18 hashar: mw64: rsync: write failed on "/apache/common-local/wmf-config/CommonSettings.php": No space left on device (28)
• 10:17 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'Commits: 6bef518 (wgHTCPMulticast only used on production cluster) and 882dd69 (wgLoadScript only used on production)'
• 09:16 notpeter: pushing new zone files. only minor changes
• 02:35 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Sun Jun 3 02:35:31 UTC 2012
• 02:13 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Sun Jun 3 02:13:39 UTC 2012

June 2

• 15:37 hashar: We ran out of beer, see bug 37307
• 15:06 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Temporarily enable $wmgReduceStartupExpiry on testwiki for Berlin tutorial'
• 14:12 logmsgbot: hashar synchronized wmf-config/CommonSettings.php
• 14:10 logmsgbot: hashar synchronized wmf-config/ext-wmflabs.php
• 14:10 hashar: deploying some nasty configuration changes in wmf-config
• 14:10 logmsgbot: hashar synchronized wmf-config/ext-pmtpa.php
• 14:04 logmsgbot: reedy synchronized wmf-config/proofreadpage.php 'Default proofreadpage-showheaders to 1 on enwikisource and svnwikisource'
• 13:46 mutante: rebuilding archives for fd-advisorygroup mailing list
• 13:21 RobHalsell: updated quotas on labstore1 for publicdata-proect
• 08:57 logmsgbot: reedy synchronized wmf-config/ 'https://gerrit.wikimedia.org/r/#/c/9717/'
• 02:36 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Sat Jun 2 02:36:20 UTC 2012
• 02:14 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Sat Jun 2 02:14:25 UTC 2012

June 1

• 20:54 logmsgbot: reedy synchronized php-1.20wmf4/includes/specials/SpecialRecentchanges.php
• 20:53 logmsgbot: reedy synchronized php-1.20wmf4/includes/SpecialPage.php
• 20:28 logmsgbot: reedy synchronized php-1.20wmf4/extensions/ExtensionDistributor/
• 20:27 logmsgbot: reedy synchronized php-1.20wmf3/extensions/ExtensionDistributor/
• 19:37 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36805 - Enable NewUserMessage extension on mrwiki and mrwikisource'
• 19:34 logmsgbot: reedy synchronized wmf-config/ 'Fix some typos'
• 19:31 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36965 - Please setup Collection extension on Telugu Wikipedia'
• 19:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 37027 - Install Collection extension in Hebrew Wiktionary'
• 19:15 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Enabling collection on tawikis'
• 19:15 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enabling collection on tawikis'
• 17:56 logmsgbot: reedy synchronized wmf-config/
• 16:50 logmsgbot: reedy Finished syncing Wikimedia installation... : Testing
• 16:23 logmsgbot: reedy Started syncing Wikimedia installation... : Testing
• 15:33 Reedy: pointing php to php-1.20wmf3
• 13:22 logmsgbot: maxsem synchronized php-1.20wmf3/extensions/MobileFrontend/ 'Deploying MF fixes'
• 13:21 logmsgbot: maxsem synchronized php-1.20wmf4/extensions/MobileFrontend/ 'Deploying MF fixes'
• 12:19 Reedy: Purged www.mediawiki.org/xml/export-0.7.xsd
• 12:17 logmsgbot: reedy synchronized docroot/mediawiki/xml/export-0.7.xsd 'Push out updated version of export-0.7.xsd'
• 06:45 apergos: reboot dataset2, kernel update and security updates
• 02:33 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Fri Jun 1 02:33:36 UTC 2012
• 02:11 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Fri Jun 1 02:11:08 UTC 2012

May 31

• 23:40 logmsgbot: kaldari Finished syncing Wikimedia installation... : scapping for new LastModified and E3Experiments extensions
• 23:14 logmsgbot: kaldari Started syncing Wikimedia installation... : scapping for new LastModified and E3Experiments extensions
• 23:09 logmsgbot: aaron synchronized multiversion/ 'Updating multiversion code to head.'
• 22:25 logmsgbot: kaldari Started syncing Wikimedia installation... :
• 22:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Re-enable randomrootpage again'
• 22:04 logmsgbot: aaron synchronized wmf-config/swift.php 'Purge from squid all thumbs in Swift on purge.'
• 21:50 logmsgbot: reedy synchronized php-1.20wmf4/includes/specials/SpecialLog.php
• 21:50 logmsgbot: reedy synchronized php-1.20wmf4/includes/logging/LogEventsList.php
• 20:46 pp-pdf2: changed tmpreaper params in crontab, delete files after 16 hours instead of 48
• 20:46 pp-pdf3: changed tmpreaper params in crontab, delete files after 16 hours instead of 48
• 20:46 pp-pdf1: changed tmpreaper params in crontab, delete files after 16 hours instead of 48
• 20:38 logmsgbot: lcarr synchronized wmf-config/mc.php
• 20:00 LeslieCarr: powering down mw32 for maintenance
• 19:59 LeslieCarr: powering down mw30 for maintenance
• 19:57 LeslieCarr: powering off mw31
• 17:32 LeslieCarr: rebooting mw1135 for kernel upgrade
• 17:24 cmjohnson1: running memtet on mw64
• 17:14 LeslieCarr: rebooting unresponsive mw1143
• 17:13 LeslieCarr: rebooting unresponsive mw1135
• 17:10 LeslieCarr: rebooted mw1091 for kernel upgrade
• 17:09 LeslieCarr: rebooted ms1004 for kernel upgrade
• 17:08 LeslieCarr: rebooted mw1102 because it thinks it has no eth0
• 17:05 LeslieCarr: rebooted mw1091 due to being unresponsive
• 17:01 LeslieCarr: rebooted ms1004 due to it being unresponsive
• 17:01 LeslieCarr: rebooted ms1004
• 14:28 binasher: pulling srv199 from lvs again for further experimentation
• 14:03 binasher: returning srv199 to lvs (dialed back to slow / no longer running php 5.4)
• 13:40 maplebed: upgrading and rebooting eqiad es hosts due to 210 day kernel bug thingy.
• 13:04 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bring in a few shell requests'
• 12:46 logmsgbot: reedy synchronized php-1.20wmf4/languages/Language.php '(bug 36839) Use mb_check_encoding() if available'
• 12:45 logmsgbot: reedy synchronized php-1.20wmf3/languages/Language.php '(bug 36839) Use mb_check_encoding() if available'
• 10:47 binasher: temporarily pulled srv199 from lvs for php testing
• 10:44 Reedy: reedy synchronized php-1.20wmf4/extensions/UploadWizard/ 'Push trunk UW to cluster'
• 10:43 Reedy: reedy synchronized php-1.20wmf3/extensions/UploadWizard/ 'Push trunk UW to cluster'
• 02:32 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Thu May 31 02:32:48 UTC 2012
• 02:17 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'syncing InitialiseSettings to disable PageTriage on test2'
• 02:10 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Thu May 31 02:10:43 UTC 2012
• 00:26 logmsgbot: awjrichards synchronizing Wikimedia installation... :

May 30

• 23:01 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php
• 22:22 logmsgbot: awjrichards synchronizing Wikimedia installation... : Weekly MobileFrontend deployment and picking up ZeroRatedMobileAccess i18n changes
• 21:50 logmsgbot: kaldari synchronized php-1.20wmf4/extensions/CentralNotice 'deploying 98db6a177df977a699576da9688588c77bf81b04'
• 21:30 K4-713: Synchronized payments cluster to DonationInterface 43a457e56d
• 21:16 LeslieCarr: rebooted db1044 (unresponsive server)
• 21:10 LeslieCarr: rebooted db1031 (unresponsive server)
• 21:08 LeslieCarr: rebooted db1029 (unresponsive server)
• 21:07 LeslieCarr: restarted db1026 (unresponsive server)
• 21:03 LeslieCarr: restarted db1012 (unresponsive server)
• 20:57 LeslieCarr: restarted networking on cp1036
• 19:34 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable randomrootpage on wikibooks and wikisources'
• 19:19 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: 27 more of the misc wikis to 1.20wmf4
• 19:15 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Some more of the misc wikis to 1.20wmf4
• 19:11 LeslieCarr: rebooting neon
• 18:55 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: All special wikis to wmf4
• 18:47 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: All wikisource, wikiversity, wiktionary to wmf4
• 18:45 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: All wikibooks, wikinews, wikiquote to wmf4
• 18:29 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Commonswiki to wmf4
• 18:16 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved EN, non-wikipedia, non-special, sites to wmf4
• 18:02 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Fix capitalisation'
• 18:00 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable random root page on testwiki'
• 17:58 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Add enabling code for randomrootpage'
• 17:53 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Add setting for random root page'
• 17:47 LeslieCarr: cleared mobile varnish cache
• 17:45 ssmollett: ganglia uploaded backported ganglia 3.3.5 deb package to precise-wikimedia repository
• 17:45 ssmollett: ganglia uploaded backported ganglia 3.3.5 deb package to precise-wikimedia repo
• 17:45 ssmollett: ganglia uploaded backported ganglia 3.3.5 deb package to precise-wikimedia repo
• 17:40 logmsgbot: aaron synchronized php-1.20wmf4/extensions/PageTriage 'Switched to wmf4 extension branch to get 0be1787634613a36439b760d6d5f0639724f8a7b'
• 16:06 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'subpages for frwikibooks'
• 12:00 mutante: restarting pdns on ns2
• 11:41 mutante: running authdns-update to push analytics1011 to 1022 entries
• 06:05 logmsgbot: hashar synchronized docroot/mediawiki/xml 'bug 37111 deploying export-0.7.xsd'
• 02:37 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Wed May 30 02:37:00 UTC 2012
• 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Wed May 30 02:23:47 UTC 2012

May 29

• 21:09 logmsgbot: aaron synchronized wmf-config/swift.php 'Enabled new thumb purge hook on remaining wikis'
• 20:49 LeslieCarr: renaming analytics1001.eqiad.wmnet to analytics1001.wikimedia.org
• 20:25 pp-pdf2: restarted services
• 20:25 pp-pdf3: restarted services
• 20:25 pp-pdf1: restarted services
• 20:25 pp-pdf2: cleaned /tmp and sandbox/cache/
• 20:25 pp-pdf1: cleaned /tmp and sandbox/cache/
• 20:25 pp-pdf3: cleaned /tmp and sandbox/cache/
• 20:15 Thehelpfulone: "Site requests" was renamed to "Site configuration" under the Wikimedia product in Bugzilla, don't know who did it though
• 20:12 LeslieCarr: reloading analytics1001
• 19:50 pp-pdf2: restarted all services
• 19:50 pp-pdf3: restarted all services
• 19:50 pp-pdf1: restarted all services
• 19:50 pp-pdf3: add libtidy.so
• 19:50 pp-pdf1: add libtidy.so
• 19:50 pp-pdf2: add libtidy.so
• 19:49 pp-pdf3: install mwlib.epub
• 19:49 pp-pdf2: install mwlib.epub
• 19:49 pp-pdf1: install mwlib.epub
• 19:49 pp-pdf1: update simplejson to 2.5.2
• 19:49 pp-pdf3: update simplejson to 2.5.2
• 19:49 pp-pdf2: update simplejson to 2.5.2
• 19:49 pp-pdf1: update mwlib.rl to 0.12.11
• 19:49 pp-pdf2: update mwlib.rl to 0.12.11
• 19:49 pp-pdf3: update mwlib.rl to 0.12.11
• 19:48 pp-pdf1: update pip to 1.1
• 19:48 pp-pdf3: update pip to 1.1
• 19:48 pp-pdf2: update pip to 1.1
• 18:59 LeslieCarr: flushed mobile varnish caches after push
• 16:54 maplebed: kicking pdns on dobson to try and make it happy again.
• 16:40 notpeter: decom of all srv lower than 190
• 16:27 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'https://gerrit.wikimedia.org/r/#/c/9204/ - use protocol-relative url for nostalgiawiki wgSiteNotice'
• 16:21 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'cleanup wgNoticeBanner_Harvard2011 https://gerrit.wikimedia.org/r/#/c/9205/'
• 16:17 hashar: /usr/local/apache/common-local is 4G where as / is 7G on srv187. Looks like deploying wmf2 + wmf3 + wmf4 will require partitions to be resized.
• 16:10 hashar: srv187 and srv188 are out of disk space
• 16:10 logmsgbot: hashar synchronized search-redirect.php 'https://gerrit.wikimedia.org/r/9206 - cleanup search-redirect.php'
• 14:37 cmjohnson1: removing disk4 on virt1 for replacement
• 12:18 mutante: killing / restarting morebots
• 02:37 logmsgbot: LocalisationUpdate completed (1.20wmf4) at Tue May 29 02:37:44 UTC 2012
• 02:24 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Tue May 29 02:24:21 UTC 2012

May 28

• 23:15 logmsgbot: reedy synchronized php-1.20wmf4/extensions/UploadWizard/ 'Master UW per Kaldari'
• 18:25 logmsgbot: reedy synchronized php-1.20wmf4/extensions/Translate/tag/PageTranslationHooks.php 'Fix Catchable fatal error'
• 18:08 logmsgbot: reedy synchronized php-1.20wmf4/LocalSettings.php
• 18:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.20wmf4
• 16:13 logmsgbot: reedy synchronizing Wikimedia installation... : Rebuild l10n for php-1.20wmf4
• 15:44 logmsgbot: reedy synchronizing Wikimedia installation... : test2wiki to 1.20wmf4 to build localisation cache
• 14:39 logmsgbot: reedy synchronizing Wikimedia installation... : Does running scap on its own with no wikis on that version build l10n for it? I suspect not...
• 14:22 logmsgbot: reedy synchronized php-1.20wmf4/ 'Staging php-1.20wmf4'
• 14:15 Reedy: sync-dir'ing php-1.20wmf4
• 14:12 Reedy: Copying php-1.20wmf4 from /tmp to /h/w/c on Fenari
• 13:12 apergos: doing security updates for a batch of mws in eqiad
• 11:22 apergos: updated kernel etc on mw1133, reboot
• 11:08 apergos: powercycled mw1133
• 09:05 apergos: rebooted snapshot4, 3 for security updates
• 08:43 apergos: rebooted snapshot1002, security updates (will do the same for 1003, 1004 shortly)
• 08:37 apergos: rebooted snapshot1001, security updates
• 02:24 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Mon May 28 02:24:05 UTC 2012

May 27

• 22:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 37134 - s:cs: site settings'
• 14:45 logmsgbot: asher synchronized wmf-config/db.php 'returning db51 to prod as an s4 slave'
• 14:34 maplebed: dns and puppet changes for s4 master rotation done.
• 14:18 binasher: rebooted db51, reslaved
• 14:09 maplebed: new s4 master position post-rotation is master_log_file="db31-bin.000334", master_log_pos=583315125
• 14:02 logmsgbot: ben synchronized wmf-config/db.php 's4 master switch complete; db31 is the new master. turning off read only on s4'
• 13:25 Tim: on db31 set global read_only=1
• 13:17 logmsgbot: tstarling synchronized wmf-config/db.php 's4 read-only and taking out db51'
• 04:27 binasher: kaulen - temporarily disabled swap and set oom_adj score to 15 for apache
• 02:24 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Sun May 27 02:24:13 UTC 2012

May 26

• 19:54 apergos: restarting apache2 on kaulen
• 19:34 Nemo_bis: bugzilla down again
• 14:19 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Settting wgBlacklistSettings'
• 11:05 apergos: stopping and restarting apache on kaulen blah blah blah
• 09:12 Reedy: Bugzilla is down, Kaulen looks to be in swap death again
• 06:46 apergos: powercycling kaulen
• 05:53 hashar: kaulen dead :-[
• 05:47 hashar: Bugzilla on Kaulen being super slow again
• 02:22 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Sat May 26 02:22:37 UTC 2012
• 00:40 K4-713: synchronized payments cluster to DonationInterface 04809f1cf0d

May 25

• 22:57 maplebed: powercycled kaulen on the mgmt interface
• 19:09 maplebed: disabled the outdated /etc/init.d/gmond on spence. use ganglia-monitor instead.
• 18:57 RobH: bugzilla appears back online
• 18:57 RobH: kaulen is rebooted, it may have had a runaway process or a memory leak, not sure yet, but it was locked up from access
• 18:55 RobH: kaulen serial console unresponsive, rebooting
• 17:42 paravoid: rebooting gurvin & yvon with new kernel
• 17:25 paravoid: resetting gurvin, load spiking at 370+, SSH unreachable, 214 days of uptime
• 15:34 mark: Power cycled kaulen
• 15:23 hashar: kaulen (bugzilla) unreacheable :-(
• 13:56 RobH: palladium disk replaced
• 13:52 cmjohnson1: replacing ps2 on mw1017
• 13:50 RobH: palladium has a bad disk, goign to replace it
• 13:37 RobH: updating drac on search18, shouldnt cause system reboot.
• 10:42 apergos: restarted apache on kaulen, was seeing page.cgi segfaults in dmesg and he logs, huge cpu wait spikes (why?)

May 24

• 23:34 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php
• 23:32 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Updating technical feedback email address for mobile feedback'
• 23:17 logmsgbot: awjrichards synchronizing Wikimedia installation... : Picking up changes to hide feedback form to prevent spamming of mobile feedback page - f6ed8ba
• 23:12 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Enabling 'technical feedback' link on mobile feedback form to disable feedback form'
• 23:00 binasher: stopped replication on es1002 in order to rsync cluster23 to es1003
• 21:53 logmsgbot: aaron synchronized php-1.20wmf3/extensions/UploadWizard 'deployed 144b58854e38d910210ccd23402225e5b1d2d62d'
• 21:52 RoanKattouw: Restarted morebots

May 23

• 21:46 binasher: shutting down mysql on db12 in able to restart with binlogging disabled
• 21:45 logmsgbot: asher synchronized wmf-config/db.php 'pulling db12 from enwiki, moving watchlist / special queries to db60'
• 21:34 ottomata1: upgraded udp-filter to 0.2.4 on oxygen, emery, and locke (with maplebed's help)
• 18:11 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Last 286 wikis over to 1.20wmf3
• 17:34 maplebed: deployed change to varnish configs for preilly; adding more carriers
• 17:13 logmsgbot: ben synchronized wmf-config/CommonSettings.php 'changing the URL for the mobile feedback page to the Project namespace'
• 17:12 logmsgbot: preilly synchronized php-1.20wmf2/extensions/MobileFrontend/ 'RL hack'
• 17:00 logmsgbot: reedy synchronized wmf-config/ 'Tidying unpushed changes'
• 15:35 RobH: ns1 died on update, restarting pdns
• 15:34 RobH: updated dns for analytics mgmt
• 02:45 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Wed May 23 02:45:44 UTC 2012
• 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Wed May 23 02:23:33 UTC 2012
• 01:10 K4-713: synchronized payments cluster to DonationInterface 4be175e43f
• 00:16 Ryan_Lane: flushed the varnish cache for mobile again
• 00:13 logmsgbot: preilly synchronized php-1.20wmf3/extensions/MobileFrontend/ 'try again'
• 00:11 logmsgbot: preilly synchronized php-1.20wmf3/extensions/MobileFrontend/ 'try again'
• 00:06 logmsgbot: preilly synchronized php-1.20wmf2/extensions/MobileFrontend/ 'try again'
• 00:00 logmsgbot: preilly synchronized php-1.20wmf3/extensions/MobileFrontend/ 'try again'

May 22

• 23:49 Ryan_Lane: flushed the varnish cache for mobile again
• 23:31 Ryan_Lane: flushed the varnish cache for mobile
• 23:21 logmsgbot: preilly synchronizing Wikimedia installation... : MobileFrontend Weekly Deployment
• 21:34 RobH: updating dns for mgmt of new servers in eqiad
• 20:52 logmsgbot: reedy synchronized php-1.20wmf3/extensions/MoodBar/ 'updating to master'
• 20:24 notpeter: powering up db1003
• 20:05 notpeter: starting xtrabackup dump from db1004 to db1020 for new eqiad s4 slave
• 19:55 notpeter: starting xtrabackup dump from db1033 to db1001 for new eqiad s1 slave
• 19:31 notpeter: reimaging db1001 and db1020
• 19:18 RobH: dns update for new servers mgmt ips
• 19:12 logmsgbot: reedy synchronizing Wikimedia installation... : Rebuilding message files for interwiki extension
• 18:44 logmsgbot: aaron synchronized php-1.20wmf3/includes/filerepo/file/LocalFile.php 'deployed 826f82eaccdf2a017a8ddb27829156f7c474db84'
• 18:44 logmsgbot: aaron synchronized php-1.20wmf3/includes/filerepo/backend/SwiftFileBackend.php
• 18:16 logmsgbot: aaron synchronized php-1.20wmf3/includes/filerepo/backend/SwiftFileBackend.php 'deployed 178a8597e32122feeb593219452f26864639d9ad'
• 17:56 maplebed: done with deploy to swift to make mediawiki write thumbnails for all wikis
• 17:47 logmsgbot: aaron synchronized wmf-config/swift.php 'Switched all wikis to new Swift thumb copy hook.'
• 17:44 maplebed: starting deploy to make mediawiki write thumbnails to swift for all wikis
• 17:35 logmsgbot: reedy synchronized php-1.20wmf3/extensions/TranslationNotifications/ 'Pushing new version of translationnotification out'
• 17:32 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'pushing interwiki loading code'
• 17:32 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'pushing interwiki variables out'
• 17:30 logmsgbot: reedy synchronized php-1.20wmf3/extensions/Interwiki/ 'pushing interwiki code to cluster'
• 03:57 hashar: GlusterFS receiving 30Mbytes/sec of input traffic. Killing labs again :-D
• 02:50 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Tue May 22 02:50:09 UTC 2012
• 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Tue May 22 02:26:50 UTC 2012
• 01:40 K4-713: updated payments cluster to Donation Interface 67b40c9307b
• 00:17 logmsgbot: aaron synchronized php-1.20wmf3/includes/filerepo/file/LocalFile.php 'deployed dfa7120f1bcd2c172096caf0ca65a06119e592c3'

May 21

• 22:31 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Remove header fail logging'
• 22:30 logmsgbot: reedy synchronized php-1.20wmf3/includes/HttpFunctions.php
• 22:20 logmsgbot: preilly synchronized wmf-config/PrivateSettings.php
• 22:19 logmsgbot: preilly synchronized wmf-config/CommonSettings.php
• 21:52 logmsgbot: reedy synchronized php-1.20wmf3/includes/HttpFunctions.php 'Don't do content length checking if it's a head request'
• 21:47 logmsgbot: reedy synchronized php-1.20wmf3/includes/HttpFunctions.php 'Don't do content length checking if it's a head request'
• 20:50 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Re-enable PageTriage on enwiki'
• 20:47 logmsgbot: catrope synchronized php-1.20wmf3/extensions/PageTriage 'Deploying raw RL module change, AFTv5 bugfix and PageTriage bugfix'
• 20:47 logmsgbot: catrope synchronized php-1.20wmf3/extensions/ArticleFeedbackv5 'Deploying raw RL module change, AFTv5 bugfix and PageTriage bugfix'
• 20:46 logmsgbot: catrope synchronized php-1.20wmf3/includes/resourceloader 'Deploying raw RL module change, AFTv5 bugfix and PageTriage bugfix'
• 20:46 logmsgbot: catrope synchronized php-1.20wmf2/extensions/PageTriage 'Deploying raw RL module change, AFTv5 bugfix and PageTriage bugfix'
• 20:46 logmsgbot: catrope synchronized php-1.20wmf2/extensions/ArticleFeedbackv5 'Deploying raw RL module change, AFTv5 bugfix and PageTriage bugfix'
• 20:45 logmsgbot: catrope synchronized php-1.20wmf2/includes/resourceloader 'Deploying raw RL module change, AFTv5 bugfix and PageTriage bugfix'
• 18:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf3
• 18:04 Jeff_Green: dist-upgrade and reboot loudon
• 17:48 andrewbogott_: ran authdns-update on dobson to pick up virt1002-1008 changes
• 17:13 logmsgbot: aaron synchronized php-1.20wmf3/includes/filerepo/LocalRepo.php 'deployed dfa7120f1bcd2c172096caf0ca65a06119e592c3'
• 16:22 mutante: analytics1001 to 1010 installed and up in puppet
• 12:45 mark: Started ircecho on manganese
• 02:46 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Mon May 21 02:46:37 UTC 2012
• 02:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'There's no helping metawiki now!'
• 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Mon May 21 02:23:51 UTC 2012

May 20

• 19:29 logmsgbot: aaron synchronized wmf-config/swift.php 'more profiling'
• 17:03 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Syncing I6b0e91cd/bug 36931: tweaking account creation whitelist for ptwiki event'
• 02:43 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Sun May 20 02:43:08 UTC 2012
• 02:21 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Sun May 20 02:21:20 UTC 2012

May 19

• 21:07 cmjohnson1: shutting down storage3 to replace RAID controller card
• 18:37 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Disabling PageTriage extension on enwiki per request from Kaldari, due to bug 36968'
• 02:43 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Sat May 19 02:43:56 UTC 2012
• 02:22 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Sat May 19 02:22:21 UTC 2012

May 18

• 23:27 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Set $wgSiteStatsAsyncFactor=1 on commonswiki.'
• 20:18 logmsgbot: reedy synchronized php-1.20wmf3/includes/HttpFunctions.php 'Tighten debugging'
• 20:12 logmsgbot: reedy synchronized php-1.20wmf3/includes/HttpFunctions.php 'Widen debugging'
• 20:06 logmsgbot: aaron synchronized php-1.20wmf3/includes/filerepo/backend/FileBackendStore.php 'deployed 0624af8f2e9666fbe0820c0caca6d7ea3c6eeb7b'
• 20:02 logmsgbot: reedy synchronized php-1.20wmf3/includes/HttpFunctions.php 're-enable debugging (fatal disabled)'
• 19:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Put wikibooks back on 1.20wmf3
• 19:27 logmsgbot: reedy synchronized php-1.20wmf3/includes/HttpFunctions.php 'Disable debugging and content length checking for now'
• 19:26 logmsgbot: reedy synchronized php-1.20wmf3/includes/HttpFunctions.php 'better debugging'
• 19:24 logmsgbot: reedy synchronized php-1.20wmf3/includes/HttpFunctions.php 'better debugging'
• 19:21 logmsgbot: reedy synchronized php-1.20wmf3/includes/HttpFunctions.php 'RE-enable header fail stuff with debug logs'
• 19:21 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Add debug log group for headerfail'
• 19:00 logmsgbot: reedy synchronized php-1.20wmf3/includes/HttpFunctions.php 'Trying older version'
• 18:36 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Push wikibooks back to 1.20wmf2 due to collection being broken
• 18:31 logmsgbot: reedy synchronized php-1.20wmf2/extensions/Collection
• 18:28 logmsgbot: reedy synchronized php-1.20wmf3/extensions/Collection '1.20wmf2 collection for testing'
• 18:12 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.20wmf3
• 18:11 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.20wmf2
• 18:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable collection on test2wiki'
• 18:02 logmsgbot: reedy synchronized php-1.20wmf3/extensions/Collection/Collection.session.php
• 18:00 logmsgbot: reedy synchronized php-1.20wmf3/extensions/Collection/Collection.templates.php 'Testing partial revert'
• 15:45 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'Applying various changes made this afternoon: 718fb59..811dbd8'
• 14:25 hashar: setup a Jenkins job to lint PHP files in operations/mediawiki-config.git:/wmf-config/
• 14:00 mutante: authdns-update - pushing fix for reverse lookup in eqiad subnets
• 12:04 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'Syncing https://gerrit.wikimedia.org/r/7931 & https://gerrit.wikimedia.org/r/7934 : minor pmtpa/wmflabs switches'
• 10:39 logmsgbot: hashar synchronized wmf-config/wgConf.php 'https://gerrit.wikimedia.org/r/#/c/7933/ change cluster name "beta" to "wmflabs"'
• 08:59 logmsgbot: nikerabbit synchronized php-1.20wmf3/extensions/Translate/specials/SpecialAggregateGroups.php 'Temp fix for bug 36944'
• 03:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable UploadWizard on donatewiki and foundationwiki'
• 02:43 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Fri May 18 02:43:38 UTC 2012
• 02:22 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Fri May 18 02:22:11 UTC 2012
• 01:42 Tim: on cp1004: set net.ipv4.tcp_tw_recycle=0 and net.ipv4.tcp_tw_reuse=1
• 01:39 binasher: filejournal migration complete
• 00:50 binasher: migrating fliejournal to innodb on all wikis
• 00:22 Tim: on cp1005: set tcp_tw_recycle=0

May 17

• 22:18 binasher: migrated centralauth.wikiset to innodb
• 22:01 binasher: migrating centralauth.spoofuser to innodb via osc (13.5mil rows)
• 22:00 binasher: migrated centralauth.global_group to innodb
• 21:53 maplebed: reverted mobile change from this morning - testing completed.
• 21:42 binasher: es1004 is replicating again
• 21:39 binasher: resumed replication to es1002
• 21:21 logmsgbot: catrope synchronized php-1.20wmf3/extensions/ArticleFeedbackv5/ArticleFeedbackv5.php 'Deploy 24ddcdf507e615b1942147654ccde1bdc4ea4bfa'
• 21:20 logmsgbot: catrope synchronized php-1.20wmf2/extensions/ArticleFeedbackv5/ArticleFeedbackv5.php 'Deploy 24ddcdf507e615b1942147654ccde1bdc4ea4bfa'
• 21:09 logmsgbot: aaron synchronized wmf-config/swift.php '+commonswiki'
• 20:58 logmsgbot: aaron synchronized wmf-config/swift.php 'revert change to itwiki'
• 20:57 Jeff_Green: several package updates on payments* and silicon
• 20:56 logmsgbot: aaron synchronized wmf-config/swift.php '+itwiki'
• 20:40 logmsgbot: aaron synchronized wmf-config/swift.php
• 20:34 maplebed: deployed change to swift and mediawiki for MW to write thumbnails to swift instead of rewrite.py with aaron
• 20:32 maplebed: deployed parallel thumbnail purging for test, test2, and mediawiki with aaron
• 20:25 logmsgbot: aaron synchronized wmf-config/swift.php 'Enabled thumb copy hook for testwikis and mw.org'
• 20:14 binasher: completed securepoll_votes.vote_ip and all ipv6 schema migration
• 20:10 logmsgbot: aaron synchronized wmf-config/CommonSettings.php
• 20:09 logmsgbot: aaron synchronized wmf-config/swift.php
• 20:03 logmsgbot: aaron synchronized wmf-config/swift.php 'Enabling new purge hook on testwikis again.'
• 20:03 binasher: running securepoll_votes.vote_ip schema migration on s1
• 20:01 binasher: running securepoll_votes.vote_ip schema migration on all s2 dbs
• 19:19 binasher: running securepoll_votes.vote_ip schema migration on all s4 + s3 dbs
• 19:17 binasher: running securepoll_votes.vote_ip schema migration on all s5 dbs
• 19:16 binasher: running securepoll_votes.vote_ip schema migration on all s6 dbs
• 19:02 binasher: running securepoll_votes.vote_ip schema migration on all s7 dbs
• 18:49 binasher: syncing cluster23 tables from es1002 to es1004
• 18:46 binasher: stopped replication on es1002
• 18:44 notpeter: restarting puppet on brewster
• 18:39 logmsgbot: catrope synchronizing Wikimedia installation... : ArticleFeedbackv5 updates
• 18:13 logmsgbot: aaron synchronized wmf-config/swift.php
• 18:08 logmsgbot: aaron synchronized php-1.20wmf3/includes/filerepo 'deployed 103efda39dd57bc22898bd0e69932982c1cfd588'
• 18:00 Jeff_Green: shutting down grosley for disk and RAM upgrades
• 17:42 notpeter: temporarily turning off puppet on brewster for preseed hackz
• 17:20 maplebed: flushing the mobile cache post-deploy
• 17:17 maplebed: deploying config change to mobile - more zero IP addresses. gerrit r7867
• 15:31 logmsgbot: dzahn synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
• 15:30 mutante: sync-common-file interwiki.cdb
• 15:30 mutante: creating fresh interwiki.cdb from dumpInterwiki.php
• 15:30 Jeff_Green: adding DNS records to wikimedia.org for RT #2960
• 14:22 mutante: adding gerrit project analytics/udplog parent analytics
• 13:44 cmjohnson1: shutting down bellin for troubleshooting
• 09:04 hashar: Site outage was due to our custom wfLogXFF() which uses wfErrorLog(). $wmfUdp2logDest not being global there, caused exception to be shown.
• 08:59 hashar: Broken the cluster by having an invalid global set
• 08:58 logmsgbot: hashar synchronized wmf-config/CommonSettings.php
• 08:47 logmsgbot: hashar synchronizing Wikimedia installation... :
• 08:44 hashar: running scap to apply https://gerrit.wikimedia.org/r/7702
• 08:41 hashar: Deploying https://gerrit.wikimedia.org/r/7702 which abstract out the udp2log destination
• 08:15 hashar: WMFLabs seems to have recovered now
• 06:50 hashar: WMFLabs dieing out, I/O latency raised constantly over the last 2 hours and eventually lead to situation where system (via ssh) is not usable anymore
• 03:41 logmsgbot: asher synchronized wmf-config/db.php 'returning db12 and db46'
• 02:48 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Thu May 17 02:48:02 UTC 2012
• 02:22 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Thu May 17 02:22:02 UTC 2012
• 02:18 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Enable SpecialCite everywhere'
• 01:40 Tim: on cp1004: reverted after TIME_WAIT client connections reached 38k with no sign of a plateau
• 01:37 Tim: on cp1004: trying tcp_tw_reuse=1 instead of tcp_tw_recycle
• 01:00 Tim: reverted after client-side TIME_WAIT connections rose rapidly from 367 to 9000
• 00:59 Tim: experimentally setting net.ipv4.tcp_tw_recycle=0 on cp1004

May 16

• 23:50 logmsgbot: aaron synchronized php-1.20wmf3/includes/upload/UploadBase.php 'deployed 4b0a61227fce37202da2b62b7dc2474bd227873f'
• 22:47 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Made mediawikiwiki use $wgSiteStatsAsyncFactor=1.'
• 22:32 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Set $wgSiteStatsAsyncFactor=1 on testwikis.'
• 21:45 maplebed: reverted this morning's mobile push - tests completed
• 21:12 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36913 - Enable Collection on kkwiki'
• 21:09 logmsgbot: reedy synchronized php-1.20wmf3/cache/interwiki.cdb 'Updating interwiki cache'
• 21:09 logmsgbot: reedy synchronized php-1.20wmf2/cache/interwiki.cdb 'Updating interwiki cache'
• 20:05 binasher: ran ipv6 migrations on globalblocks
• 20:05 binasher: converted centralauth.globalblocks from myisam to innodb
• 19:57 binasher: rebooting db12 for kernel upgrade
• 19:51 binasher: stopping mysql on db12
• 19:50 binasher: recentchanges.rc_ip migration completed
• 19:47 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Last bits of tidying up
• 19:44 binasher: rebooted db46
• 19:38 binasher: shutting down mysql on db46, preparing to reboot for kernel upgrade
• 19:32 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'touch'
• 19:28 logmsgbot: asher synchronized wmf-config/db.php 'pulling db12 from enwiki, moving watchlist / special queries to db59'
• 19:25 binasher: running recentchanges.rc_ip (ipv6) schema migration on enwiki master (5.2mil rows) via os������c - batten down the hatches!
• 19:17 binasher: running recentchanges.rc_ip (ipv6) schema migration on s2 dbs via os������c
• 19:15 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'touch'
• 19:14 Reedy: manually ran ddsh -cM -g mediawiki-installation -o -oSetupTimeout=30 -F30 "sudo -u mwdeploy rsync -a 10.0.5.8::common/*.dblist /usr/local/apache/common-local" because sync-dblist is woefully out of date..
• 19:13 notpeter: restarting ganglia on nickel
• 19:09 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: 12 more misc/wikimedia wikis to 1.20wmf3
• 18:59 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All closed wikis to 1.20wmf3
• 18:55 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All special wikis to 1.20wmf3
• 18:54 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All wikimedia wikis to 1.20wmf3
• 18:52 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All wikisource to 1.20wmf3
• 18:50 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All wikiquote to 1.20wmf3
• 18:48 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All wikiversity to 1.20wmf3
• 18:45 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All wiktibooks to 1.20wmf3
• 18:42 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All wiktionaries to 1.20wmf3
• 18:40 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All wikinews to 1.20wmf3
• 18:31 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All non wikipedia en projects to 1.20wmf3
• 18:27 binasher: running recentchanges.rc_ip (ipv6) schema migration on s3 dbs via os������c (s4 already completed during prior testing)
• 18:25 mutante: synced wikiversions.* files from NFS to spence local to prevent death of check_job_queue monitoring
• 18:21 binasher: running recentchanges.rc_ip (ipv6) schema migration on s5 dbs via os������c
• 18:19 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commonswiki to 1.20wmf3, again
• 18:17 logmsgbot: aaron synchronized php-1.20wmf3/includes/ImagePage.php 'deployed 86e2372772e618c5d1238ae480d9f632789bbe50'
• 18:13 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commonswiki back to 1.20wmf2
• 18:10 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commonswiki to 1.20wmf3
• 18:10 binasher: running recentchanges.rc_ip (ipv6) schema migration on all s6 dbs via os������c
• 18:03 binasher: running recentchanges.rc_ip (ipv6) schema migration on all s7 dbs via os������c
• 17:43 binasher: ipblocks migration completed for all wikis
• 17:38 binasher: running ipblocks schema migration on all s2 dbs via os������c
• 17:35 logmsgbot: awjrichards synchronized php-1.20wmf2/extensions/MobileFrontend 'Picking up fix for fatal in api in MobileFrontend at 9936e7a'
• 17:34 logmsgbot: awjrichards synchronized php-1.20wmf3/extensions/MobileFrontend 'Picking up fix for fatal in api in MobileFrontend at 9936e7a'
• 17:16 maplebed: deploying change to swift to make which containers write thumbs configurable
• 17:11 logmsgbot: preilly synchronized php-1.20wmf3/extensions/MobileFrontend/ 'zero and mobile changes'
• 17:10 logmsgbot: preilly synchronized php-1.20wmf2/extensions/MobileFrontend/ 'zero and mobile changes'
• 17:08 RobH: aluminum back online
• 17:00 binasher: running ipblocks schema migration on all s3 (819) dbs via os������c
• 16:59 binasher: running ipblocks schema migration on all s4 dbs via os������c
• 16:58 binasher: running ipblocks schema migration on s5/dewiki via osc
• 16:57 RobH: aluminum shut down for hard disk additions
• 16:56 binasher: running ipblocks schema migration on all s6 dbs via osc
• 16:51 RobH: udpating dns for osm web servers
• 16:50 binasher: running ipblocks schema migration on all s7 dbs via osc
• 16:49 logmsgbot: preilly synchronized php-1.20wmf2/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing'
• 16:41 logmsgbot: aaron synchronized php-1.20wmf3/includes/filerepo/backend/SwiftFileBackend.php 'deployed 634c3be2bba6a46e28aa997d7ab388ebf90b36a6'
• 16:31 maplebed: clearing the mobile varnish cache
• 16:29 maplebed: deploying gerrit change 7798 to the mobile varnish servers
• 07:41 logmsgbot: raindrift synchronized php-1.20wmf3/extensions/PageTriage/api/ApiPageTriageTemplate.php 'fixing exception bug that makes lots of logspam'
• 07:41 logmsgbot: raindrift synchronized php-1.20wmf2/extensions/PageTriage/api/ApiPageTriageTemplate.php 'fixing exception bug that makes lots of logspam'
• 06:20 Ryan_Lane: restarted lucene on search1015
• 05:58 Tim: setting net.ipv4.tcp_tw_recycle=1 on cp1005 seems to have fixed it, doing it on cp1004 as well now
• 05:52 Tim: on cp1005 setting tcp_tw_recycle=1
• 05:29 Tim: experimentally started squid on cp1004
• 04:05 hashar: updating a few plugins on Jenkins (host: gallium )
• 03:34 Ryan_Lane: stopped the squid process on cp1004 and stopped puppet to avoid it being restarted. it's having issues and I can't debug it right now.
• 03:22 Ryan_Lane: repooling squid frontend on cp1004
• 03:14 Ryan_Lane: depooling cp1004 and stopping the squid backend service to let some connections close
• 02:43 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Wed May 16 02:43:51 UTC 2012
• 01:10 logmsgbot: reedy synchronized php-1.20wmf3/extensions/RandomRootPage 'dark deploy randomrootpage extension (I'll enable it later)'

May 15

• 23:53 logmsgbot: aaron synchronized php-1.20wmf3/includes/Block.php 'deployed 7694faf68f975ea9c4888d575b33dabb84e90083'
• 23:42 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php
• 23:27 ssmollett: upgraded ganglia-monitor and gmetad from 3.1.2-2.1 to 3.3.5-2
• 23:26 logmsgbot: awjrichards synchronizing Wikimedia installation... :
• 23:24 K4-713: upgraded minfraud version on the payments account
• 23:22 K4-713: updated and synchronized the payments cluster to DonationInterface d997e7ea1c
• 23:06 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping MobileFrontend resource version'
• 22:50 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Disable CentralAuth logging to file'
• 22:50 logmsgbot: awjrichards synchronizing Wikimedia installation... : Updating MobileFrontend to 0880467
• 22:34 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Enable CentralAuth logging to file'
• 22:13 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php
• 21:09 logmsgbot: raindrift synchronizing Wikimedia installation... : PageTriage update
• 21:05 logmsgbot: raindrift synchronizing Wikimedia installation... : PageTriage update
• 19:40 logmsgbot: aaron synchronized wmf-config/swift.php 'disabled new hook for now.'
• 19:35 logmsgbot: reedy synchronized php-1.20wmf3/extensions/Translate/tag/PageTranslationHooks.php 'Add wfDebugLog call'
• 19:31 logmsgbot: reedy synchronized php-1.20wmf3/extensions/Translate/tag/PageTranslationHooks.php 'Add wfDebugLog call'
• 19:30 logmsgbot: reedy synchronized wmf-config/CommonSettings.php '$wgDebugLogGroups[updateTranstagOnNullRevisions] = udp://10.0.5.8:8420/updateTranstagOnNullRevisions'
• 19:26 logmsgbot: reedy synchronized php-1.20wmf3/extensions/Translate/tag/PageTranslationHooks.php 'Only conditionally disable updateTranstagOnNullRevisions hook. Debugging to come'
• 17:45 logmsgbot: aaron synchronized php-1.20wmf3/includes/filerepo/backend/SwiftFileBackend.php
• 17:41 logmsgbot: aaron synchronized php-1.20wmf3/includes/filerepo/backend/SwiftFileBackend.php
• 17:35 logmsgbot: aaron synchronized wmf-config/swift.php 'Use new thumb purge hook for testwikis'
• 16:54 RobHalsell: updated apache config for wiki-pedia.org, seems the bot doesnt spam that anymore =[
• 16:36 mutante: srv app servers max. uptime with older kernel down to ~120 days after another bunch of upgrades
• 16:34 RobHalsell: updating dns for wiki-pedia.org
• 12:20 hashar: deployment-prep replaced most occurrences of /mnt/upload to /mnt/upload6
• 10:37 apergos: on db39 dropped triggers pt_osc_elwiki_recentchanges ins, del, upd, they were preventing all elwiki edits except bot edits with the complaint Table 'elwiki._recentchanges_new' doesn't exist ... binasher, doublecheck me please?
• 09:24 mutante: srv278 - still has issues as in reopnened RT #24 - upgrading kernel anyways
• 03:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'update wgUploadNavigationUrl on all cs wikis'
• 02:35 logmsgbot: LocalisationUpdate completed (1.20wmf3) at Tue May 15 02:35:53 UTC 2012
• 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Tue May 15 02:23:47 UTC 2012
• 01:09 logmsgbot: asher synchronized wmf-config/db.php 'returning db31 as an s4 slave'
• 01:05 logmsgbot: aaron synchronized php-1.20wmf3/extensions/SwiftCloudFiles/php-cloudfiles-wmf/cloudfiles.php 'deployed f20e752630575f8384083f0ad0401e250c8babf5'
• 01:00 binasher: shutting down mysql on db31, then rebooting
• 00:59 logmsgbot: asher synchronized wmf-config/db.php 'pulling db31 from s4 for kernel upgrade'
• 00:58 binasher: new s4 master position - MASTER_LOG_FILE='db51-bin.000114', MASTER_LOG_POS=1772578
• 00:57 logmsgbot: asher synchronized wmf-config/db.php 'new s4 master'
• 00:55 logmsgbot: asher synchronized wmf-config/db.php 'setting s4 to read-only, switching master to db51'
• 00:54 binasher: preparing to rotate s4 master from db31 to db51
• 00:48 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
• 00:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bring in numerous shell requests from gerrit'
• 00:48 binasher: rebooting db51 for kernel upgrade, prior to promoting to s4 master
• 00:47 logmsgbot: asher synchronized wmf-config/db.php 'pulling db51 from s4 for kernel upgrade'
• 00:01 binasher: just completed an online schema change for commonswiki.recentchanges in prod. woo!

May 14

• 22:02 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping MobileFrontend resource version'
• 21:08 logmsgbot: reedy synchronized php-1.20wmf3/extensions/Translate/tag/PageTranslationHooks.php 'Live hack out updateTranstagOnNullRevisions'
• 20:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki to 1.20wmf3
• 19:38 logmsgbot: reedy synchronizing Wikimedia installation... : Running scap to make sure everything is ok...
• 19:36 logmsgbot: reedy synchronized php-1.20wmf3/cache/l10n/ 'Resync localisation cache'
• 19:26 logmsgbot: reedy synchronized live-1.5/ 'Push live-1.5 new symlinks'
• 19:24 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki to 1.20wmf3
• 19:16 logmsgbot: reedy synchronized php-1.20wmf3/cache/trusted-xff.cdb
• 19:14 logmsgbot: reedy synchronized wmf-config/ExtensionMessages-1.20wmf3.php
• 19:13 logmsgbot: reedy synchronized php-1.20wmf3/extensions/ 'Push extensions out properly'
• 19:11 binasher: resyncing cluster22 from es1002 to es1004
• 19:02 logmsgbot: reedy synchronized php-1.20wmf3/LocalSettings.php 'Use newer version'
• 19:01 logmsgbot: reedy synchronized php-1.20wmf2/LocalSettings.php 'Use newer version'
• 18:58 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.20wmf3
• 18:57 logmsgbot: reedy synchronized php-1.20wmf3/cache/l10n/ 'Syncing localisation cache files'
• 18:49 Ryan_Lane: added OATHAuth to components list for MediaWiki Extensions product in bugzilla
• 18:43 Ryan_Lane: switching sessions back to memcached for labsconsole
• 18:42 Ryan_Lane: adding OATHAuth to labsconsole
• 18:40 Ryan_Lane: completed upgrade to 1.20wmf2 on labsconsole
• 18:30 Ryan_Lane: upgrading labsconsole to 1.20wmf2
• 18:26 logmsgbot: reedy synchronized php-1.20wmf3 'Initial pushing of php-1.20wmf3 files to apaches'
• 18:12 Reedy: Killing old php-1.20wmf1 directories from apaches to save full disks
• 13:48 mutante: copying outdated wikiversions.dat/.cdb files from /home to /usr/local on spence, which fixes check_job_queue (thanks jeremyb)
• 13:07 mutante: opening a bz bug for check_job_queue issue related to CommonSettings.php BZ:36835
• 07:43 mutante: still upgrading/rebooting a couple srv (API) application servers with long uptime
• 06:22 apergos: restarted lucene search on search1016 it had stopped doing anything useful (see ganglia graphs, also nothinig wtitten to logs)
• 02:22 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Mon May 14 02:22:09 UTC 2012

May 13

• 02:24 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Sun May 13 02:24:51 UTC 2012

May 12

• 02:22 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Sat May 12 02:22:18 UTC 2012

May 11

• 22:10 logmsgbot: preilly synchronized wmf-config 'add wikimedia to zero image disable list fix header'
• 21:52 logmsgbot: preilly synchronized wmf-config 'add wikimedia to zero image disable list'
• 19:49 Reedy: ran apache-graceful-all
• 19:42 RobH: apache restarted by puppet run on srv286
• 19:31 RobH: shutting down srv286 and srv286 for power rebalancing
• 19:23 RobH: srv260 and srv261 back in business
• 19:10 RobH: srv261 & srv261 shutting down for power rebalancing within the rack
• 18:33 notpeter: shutting down search 13-20 for hd upgrades
• 18:05 maplebed: swift: deleting the unsharded version of all sharded containers
• 18:03 logmsgbot: catrope synchronized php-1.20wmf2/extensions/UploadWizard/ 'Deploy 4b5df1a1151ac80e309d396102e5e2a8d0c27ccb'
• 17:46 maplebed: deleted wikipedia-de-local-thumb container from swift. the sharded version is currently being used.
• 15:33 mutante: adding DNS entries for analytics hosts in new vlan 1121 (10.64.21.0/24), hosts starting at .101 to match names analytics1001 = .101 and ++
• 15:03 mutante: mw62 -unless somebody was on that right now it died. mgmt also just Create Instance Error
• 14:06 mutante: kernel upgrading / rebooting srv servers where uptime > 200 d order by uptime desc limit 1
• 13:12 mutante: installing package upgrades on pdf1-3 (and installed requested indic fonts via new puppet role class)
• 11:39 mutante: starting ms-be swift-container-auditors every once in a while
• 11:35 mutante: stat1 - installed new kernel, but waiting to reboot. schedule with aotto
• 11:24 mutante: upgrading packages/kernel on hooper, rebooting (Blog,Etherpad,Racktables)
• 09:21 mutante: ekrem was close running out of disk again. logrotated apache logs, changed config to: size 512M,rotate 3
• 08:58 mutante: package upgrades on ekrem (IRC server, WAP, Apple dict...)
• 08:51 mutante: rebooting marmontel (blog)
• 08:48 mutante: upgrading apache/mysql/kernel on marmontel (blog)
• 02:20 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Fri May 11 02:20:39 UTC 2012
• 02:00 RoanKattouw: Started Apache back up on srv200, done debugging
• 01:58 logmsgbot: catrope synchronized php-1.20wmf2/extensions/UserDailyContribs/UserDailyContribs.hooks.php 'Deploy 3c45831ffe1817f3dc18f06644db46b1b74173e7'
• 01:17 RoanKattouw: Stopping Apache on srv200 so I can use it as my guinea pig for segfault debugging
• 00:56 logmsgbot: tstarling synchronized php-1.20wmf2/includes/User.php 'header log'
• 00:49 logmsgbot: aaron synchronized php-1.20wmf2/includes/User.php
• 00:48 logmsgbot: aaron synchronized php-1.20wmf2/includes/User.php
• 00:40 logmsgbot: aaron synchronized php-1.20wmf2/includes/User.php
• 00:30 Tim: restarted socat on fenari so that fatal.log is reopened
• 00:29 logmsgbot: aaron synchronized php-1.20wmf2/includes/User.php 'removed logging hack tweaks.'
• 00:29 logmsgbot: aaron synchronized php-1.20wmf2/includes/User.php 'logging hack tweaks.'
• 00:27 logmsgbot: aaron synchronized php-1.20wmf2/includes/User.php 'removed some temp logging'
• 00:27 logmsgbot: aaron synchronized php-1.20wmf2/includes/User.php
• 00:16 logmsgbot: aaron synchronized php-1.20wmf2/includes/User.php
• 00:00 binasher: pulling cp1044 from lvs for testing

May 10

• 23:38 logmsgbot: reedy synchronized php-1.20wmf2/extensions/LiquidThreads/classes/Hooks.php 'Updating to master'
• 22:38 logmsgbot: catrope synchronized php-1.20wmf1/.git 'Make Special:Version show the correct commit now that I have fixed the weird repo state'
• 22:37 logmsgbot: catrope synchronized php-1.20wmf2/.git 'Make Special:Version show the correct commit now that I have fixed the weird repo state'
• 22:36 RoanKattouw: Cleaned up weird git repo states on fenari in php-1.20wmf1 and php-1.20wmf2
• 22:04 maplebed: swift: deleting the unsharded wikipedia-de thumb container contents (the sharded version is currently serving traffic)
• 19:51 notpeter: rebooting db29 for do a test install of precise
• 19:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36420 - Wikipedia namespace alias for sr.wp'
• 19:02 logmsgbot: catrope synchronizing Wikimedia installation... : ArticleFeedbackv5 updates
• 18:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36694 - Set wgSitename on srwikisource'
• 18:43 LeslieCarr: restarting mobile varnish
• 18:33 LeslieCarr: reloaded and purged cache of mobile varnish
• 18:03 notpeter: starting innobackupex from db10 to blondel
• 17:39 notpeter: pushing out new zone files. only minor changes
• 16:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'showupdatemarker on enwiki tooooo'
• 03:04 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable show update markers on dewiki'
• 02:14 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Thu May 10 02:14:03 UTC 2012
• 01:07 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c/7133/'
• 00:11 logmsgbot: catrope synchronized php-1.20wmf2/extensions/UploadWizard 'Deploy b45437b6e09018dacfc78c8e4fa822a917858b2d / 62631485ba36f973c0d4a850ef494a8f84c4c86b'
• 00:11 logmsgbot: catrope synchronized php-1.20wmf1/extensions/UploadWizard 'Deploy b45437b6e09018dacfc78c8e4fa822a917858b2d / 62631485ba36f973c0d4a850ef494a8f84c4c86b'
• 00:06 logmsgbot: preilly synchronized wmf-config 'remove MF passwords'
• 00:01 logmsgbot: preilly synchronized wmf-config 'remove MF passwords'

May 9

• 23:33 notpeter: taking down search20 to do precise test-install
• 23:26 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable Translate on outreachwiki'
• 23:25 Reedy: Created Translate tables on outreachwiki
• 22:49 Reedy: ExtensionDistributor fixed
• 22:32 Reedy: Debugging ExtensionDistributor being broken. Likely to show more debug output on mw.org if you attempt to use it (though, it wouldn't give you what you wanted anyway)
• 22:15 logmsgbot: reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
• 21:53 logmsgbot: aaron synchronized php-1.20wmf2/includes/filerepo/file/LocalFile.php 'deployed b9ac85cbf304a65d900cda00fafe53bf82d7a227'
• 21:53 logmsgbot: aaron synchronized php-1.20wmf2/includes/SiteStats.php 'deployed b9ac85cbf304a65d900cda00fafe53bf82d7a227'
• 20:52 LeslieCarr: done
• 20:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bump memory limit to 128MB'
• 19:39 Ryan_Lane: updating OpenStackManager on virt0 to master again
• 19:16 Ryan_Lane: updating OpenStackManager on virt0 to master
• 18:54 logmsgbot: aaron synchronized php-1.20wmf2/includes/filerepo/file/LocalFile.php 'deployed fa1a8d5119e1174f7458eb9516287f4867c46484'
• 18:50 RobH: dns update for db61 and db62
• 18:25 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: 295 other wikipedias over to 1.20wmf2
• 18:20 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: jawiki to 1.20wmf2
• 18:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: ruwiki to 1.20wmf2
• 18:12 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: frwiki to 1.20wmf2
• 18:11 notpeter: turning db30 back on
• 18:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: dewiki to 1.20wmf2
• 17:51 cmjohnson1: to shutting down storage3
• 16:58 LeslieCarr: restarted mobile varnish instances
• 16:58 LeslieCarr: flushed mobile varnish cache
• 16:54 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Make sure Swift backend will have journaling too.'
• 16:31 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Removed backend config conditional now that everything was switched over.'
• 14:06 mutante: started container-auditor on ms-be1
• 09:24 mutante: started container-auditor on ms-be3 and 4
• 02:37 logmsgbot: LocalisationUpdate completed (1.20wmf1) at Wed May 9 02:37:02 UTC 2012
• 02:19 Reedy: Running cleanupUploadStash.php over all wikis
• 02:13 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Wed May 9 02:13:10 UTC 2012
• 01:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36506 - Site logo for Tsonga Wikipedia -- ts.wikipedia.org'
• 01:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36522 - Upload link should lead to UploadWizard instead of commons:Special:Upload'
• 01:36 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36663 - Please allow bureaucrats to add and remove autoreviewer status on pt.wiki'
• 01:30 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wgShowUpdatedMarker enabled on anything that isn't enwiki or dewiki'
• 01:27 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36533 - Set sitename to Telugu Wiktionary'
• 01:25 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36595 - Please enable Extention:NewUserMessage on ml.wikipedia'
• 01:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36595 - Please enable Extention:NewUserMessage on ml.wikipedia'
• 01:21 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36571 - Please lock wikimania2011 wiki'
• 01:21 logmsgbot: reedy synchronized closed.dblist 'Closing wikimania2011wiki'
• 00:11 maplebed: started process to delete objects that don't exist in the container listings on all swift backends

May 8

• 23:44 K4-713: synchronized payments cluster to r115155, DonationInterface ccfbb304
• 23:34 LeslieCarr: purged varnish mobile cache
• 23:25 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping mobilefrontend resource version again'
• 23:25 logmsgbot: awjrichards synchronized php-1.20wmf1/extensions/MobileFrontend/ 'd43f5f19ff3599f16200d247b6838cfb04ef1473'
• 23:25 logmsgbot: awjrichards synchronized php-1.20wmf2/extensions/MobileFrontend/ 'd43f5f19ff3599f16200d247b6838cfb04ef1473'
• 23:22 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping mobilefrontend resource version'
• 23:11 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping MobileFrontend resource version'
• 23:11 logmsgbot: awjrichards synchronized php-1.20wmf2/extensions/MobileFrontend/ '2b1e8573fdbcab0feb3a2481167b68fb96abf663'
• 23:10 logmsgbot: awjrichards synchronized php-1.20wmf1/extensions/MobileFrontend/ '2b1e8573fdbcab0feb3a2481167b68fb96abf663'
• 22:53 RoanKattouw: Actually fixed it now with chmod -R g+w /h/w/conf/httpd
• 22:47 RoanKattouw: Fixed permissions in /h/w/conf/httpd by running find -group wikidev -not -perm 020 -exec chmod g+w \{\} \;
• 22:38 logmsgbot: awjrichards synchronized php-1.20wmf2/extensions/MobileFrontend/stylesheets/sections.css 'Live hack to live test broken interface on ICS devices on very large articles'
• 22:34 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Enable mobile url transformation on testwiki'
• 22:13 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'bumping MobileFrontend resource version number'
• 22:13 logmsgbot: awjrichards synchronized php-1.20wmf2/extensions/MobileFrontend/ 'd828a8196d8bc877afdbd1559e8e6d639b51cef7'
• 22:12 logmsgbot: awjrichards synchronized php-1.20wmf1/extensions/MobileFrontend/ 'd828a8196d8bc877afdbd1559e8e6d639b51cef7'
• 21:53 binasher: rebooting db1018 one more time
• 21:47 logmsgbot: aaron synchronized php-1.20wmf2/includes/filerepo/file/LocalFile.php 'deployed 43aa35016b03935b27d439afe9a6b3f1aad1aa8b'
• 21:45 Ryan_Lane: adding adminbot to the repo
• 21:32 binasher: rebooting eqiad core db slaves for kernel upgrade
• 21:29 logmsgbot: aaron synchronized wmf-config/swift.php 'Added new thumbnail purge/import hooks handlers that use the swift backend class; unused atm.'
• 21:23 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Added swift backend config; unused atm.'
• 21:15 logmsgbot: asher synchronized wmf-config/db.php 'returning db45 to service'
• 21:14 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Matched wikimania2013wiki configuration to that of wikimania2012wiki'
• 21:13 maplebed: delpoyed container sharding for thumbnails to swift for 'dewiki', 'fiwiki', 'frwiki', 'hewiki', 'huwiki', 'idwiki', 'itwiki', 'jawiki', 'rowiki', 'ruwiki', 'thwiki', 'trwiki', 'ukwiki', 'zhwiki' (in addition to existing sharding for commons and enwiki)
• 21:13 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Move wikimania2013wiki to php-1.20wmf2
• 21:12 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Matched wikimania2013wiki configuration to that of wikimania2012wiki'
• 21:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Matched wikimania2013wiki configuration to that of wikimania2012wiki'
• 21:10 binasher: shutting down mysql across all eqiad core db slaves
• 20:59 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'logo for wikimania2013wiki'
• 20:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'remove w'
• 20:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable translate on wikimania2013wiki'
• 20:56 logmsgbot: aaron synchronized wmf-config/swift.php 'Switching purge hook to use new sharding scheme.'
• 20:54 Reedy: Created translate related tables for wikimania2013wiki
• 20:31 logmsgbot: reedy synchronized php-1.20wmf1/extensions/AntiBot/
• 20:30 logmsgbot: reedy synchronized php-1.20wmf2/extensions/AntiBot/
• 20:14 maplebed: creating sharded containers for swift for 'dewiki','fiwiki', 'frwiki', 'hewiki', 'huwiki', 'idwiki', 'itwiki', 'jawiki', 'rowiki', 'ruwiki', 'thwiki', 'trwiki', 'ukwiki', 'zhwiki'
• 19:54 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Moved remaining wikis over to new backend config'
• 19:34 logmsgbot: awjrichards synchronized php-1.20wmf2/extensions/MobileFrontend/
• 19:12 LeslieCarr: flushed mobile varnish cache
• 19:11 logmsgbot: awjrichards synchronized php-1.20wmf2/extensions/MobileFrontend/
• 19:10 logmsgbot: awjrichards synchronized php-1.20wmf1/extensions/MobileFrontend/
• 18:37 LeslieCarr: reenabled services on fpc5 of cr1-eqiad
• 18:16 cmjohnson1: updating md1000 controller card firmware on storage3
• 18:14 LeslieCarr: turned off fpc5 on cr1-eqiad to swap
• 18:05 LeslieCarr: powering on fpc 5 on cr1-eqiad
• 18:03 LeslieCarr: powering off fpc5 on cr1-eqiad in order for RobH to physically reseat the card
• 17:48 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Enable TranslationNotifications on meta, incubator and wikimania2012'
• 17:44 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Enable TranslationNotifications on mediawikiwiki'
• 17:42 LeslieCarr: switching all masterships over to cr2-eqiad in preparation to reseat cr1 linecard
• 17:25 LeslieCarr: flushed the mobile cache
• 17:24 logmsgbot: reedy synchronizing Wikimedia installation... : Rebuilding localisation cache files before TranslationNotification deploy
• 17:18 logmsgbot: reedy synchronized php-1.20wmf2/extensions/MobileFrontend/ 'Pushing out head'
• 17:16 logmsgbot: reedy synchronized php-1.20wmf1/extensions/MobileFrontend/ 'Pushing out head'
• 17:14 RobH: asw-c1-eqiad connected to both cr1 and cr2
• 15:16 cmjohnson1: shutting down storage3 to replace raid card
• 12:40 pp-pdf1: updated mwlib to 0.13.7
• 12:39 pp-pdf2: updated mwlib to 0.13.7
• 12:36 pp-pdf3: updated mwlib to 0.13.7
• 11:59 mutante: merging CSS fix for broken mobile site table layout
• 02:18 RoanKattouw: Removed and recloned /var/lib/l10nupdate/mediawiki/extensions , it was in a weird state because magic extension submodules work now but my hacky workaround for them not working was still in place
• 02:00 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
• 01:12 logmsgbot: tstarling synchronized php-1.20wmf2/includes/api/ApiMain.php
• 01:10 logmsgbot: tstarling synchronized php-1.20wmf1/includes/api/ApiMain.php
• 00:44 binasher: rebooted db1034
• 00:42 logmsgbot: tstarling synchronized php-1.20wmf2/includes/Exception.php
• 00:42 logmsgbot: tstarling synchronized php-1.20wmf2/includes/DefaultSettings.php
• 00:37 logmsgbot: tstarling synchronized php-1.20wmf1/includes/Exception.php
• 00:36 logmsgbot: tstarling synchronized php-1.20wmf1/includes/DefaultSettings.php
• 00:20 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Switched enwiki to new backend config.'

May 7

• 23:52 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'bumping mobilefrontend resource version #'
• 23:44 logmsgbot: reedy synchronized php-1.20wmf2/extensions/PageTriage/
• 23:43 logmsgbot: reedy synchronized php-1.20wmf1/extensions/PageTriage/
• 23:35 logmsgbot: reedy synchronized php-1.20wmf2/extensions/PageTriage/includes/PageTriageUtil.php
• 23:31 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wgShowExceptionDetails to true for testwiki and test2wiki'
• 23:25 logmsgbot: awjrichards synchronizing Wikimedia installation... : Sync'ing MobileFrontend changes per http://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-05-07, take 3
• 23:15 logmsgbot: reedy synchronized php-1.20wmf1/extensions/PageTriage/includes/PageTriageUtil.php
• 23:15 logmsgbot: reedy synchronized php-1.20wmf2/extensions/PageTriage/includes/PageTriageUtil.php
• 23:00 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'wgShowExceptionDetails = false'
• 22:57 Ryan_Lane: restarting glusterd processes on virt1-5
• 22:56 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping mobile resource version'
• 22:54 Ryan_Lane: upgrading glusterfs on virt1-5
• 22:49 Ryan_Lane: upgrading glusterfs on labstore1-4
• 22:48 binasher: running an osc against plwiktionary.recentchanges on master
• 22:40 paravoid: deleting 14k tmp files from spence's /home/nagios
• 22:35 logmsgbot: awjrichards synchronizing Wikimedia installation... : Sync'ing MobileFrontend changes per http://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-05-07
• 22:34 logmsgbot: awjrichards synchronizing Wikimedia installation... : Sync'ing MobileFrontend changes per http://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-05-07
• 22:24 RoanKattouw: chmod 775 /usr/local/apache/common-local/php-1.20wmf2/extensions/PageTriage with dsh as root
• 22:19 logmsgbot: raindrift synchronized php-1.20wmf1/resources/startup.js 'touch'
• 22:18 binasher: rebooting nfs2 to new kernel
• 22:16 logmsgbot: raindrift synchronized wmf-config/InitialiseSettings.php 'enabling PageTriage on enwp'
• 22:14 logmsgbot: raindrift synchronized php-1.20wmf2/extensions/PageTriage 'Syncing PageTriage to enwp, a la carte'
• 22:14 logmsgbot: raindrift synchronized php-1.20wmf1/extensions/PageTriage 'Syncing PageTriage to enwp, a la carte'
• 21:59 mutante: was still upgrading/rebooting amssq* and knsq* hosts on the side (slow,b/c upload squids). expect temp. nagios squid reports tomorrow as well. out for now.
• 21:44 binasher: moved default resolution for upload from eqiad to pmtpa
• 21:29 cmjohnson1: shutting down storage3 for troubleshooting
• 20:37 binasher: attempting a live online schema change for zuwikitionary.recentchanges on the prod master
• 20:22 LeslieCarr: (above) restarted nagios-wm on spence
• 20:20 LeslieCarr: restarted irc bot
• 20:15 binasher: rebooting db45
• 20:11 binasher: rebooting db1019
• 18:46 logmsgbot: reedy synchronized php-1.20wmf1/extensions/Collection/Collection.session.php 'head'
• 18:45 logmsgbot: reedy synchronized php-1.20wmf2/extensions/Collection/Collection.session.php 'head'
• 18:25 logmsgbot: reedy synchronized php-1.20wmf2/extensions/GlobalBlocking/GlobalBlocking.class.php
• 18:24 logmsgbot: reedy synchronized php-1.20wmf1/extensions/GlobalBlocking/GlobalBlocking.class.php
• 18:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf2
• 16:16 cmjohnson1: shutting down storage3 to reseat RAID card
• 15:58 cmjohnson1: Going to power cycling storage3 several times to troubleshoot hardware issue
• 15:15 RobH: updating firmware on storgae3
• 14:20 Jeff_Green: stopped cron jobs on storage3 because of RAID failure
• 12:49 mutante: pushing out virtual host for wikimania2013 wiki. sync / apache-graceful/all
• 11:18 mutante: continuing with upgrades/reboots in amssq* on the side during the day
• 11:09 mutante: squids - sq* done. all latest kernel and 0 pending upgrades.
• 09:27 mutante: rebooting bits varnish sq68-70 one by one..
• 08:00 mutante: upgrading/rebooting the last couple sq* servers
• 07:20 binasher: power cycled db45 (crashed dewiki slave)
• 07:05 logmsgbot: asher synchronized wmf-config/db.php 'db45 is down'
• 02:25 Tim: on locke: introduced 1/100 sampling for banner impressions, changed filename to bannerImpressions-sampled100.log
• 02:12 Tim: on locke: moved fundraising logs back where they were
• 02:00 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
• 01:38 Tim: on locke: compressing bannerImpressions.log
• 01:35 Tim: on locke: moved bannerImpressions.log to archive and restarted udp2log
• 01:26 Tim: on locke: moved fundraising logs from /a/squid/fundraising/logs to /a/squid so that they will be processed by logrotate

May 6

• 07:03 apergos: manually rotates udplogs on locke, copying destined_for_storage3 off to hume:/archive/emergencyfromlocke/ (jeff, this note's for you in particular)
• 06:36 apergos: bringing up storage3 with neither /a nor /archive mounted, saw "The disk drive for /archive is not ready yet or not present" etc on boot, waited a long time, finally skipped them
• 06:12 apergos: and powercycling the box instead. grrrr
• 06:05 apergos: rebooting storage3: we have messages like May 6 05:45:12 storage3 kernel: [465081.410025] Filesystem "dm-0": xfs_log_force: error 5 returned. in the log, and the raid is unaccessible, megacli doesn't run either
• 02:00 logmsgbot: LocalisationUpdate failed: git pull of extensions failed

May 5

• 09:37 mutante: squids - upgrading in the sq5x range (upload)
• 08:53 apergos: disabling modcompress temporarily for lightty on dataset2 (live hack), let's see what that does as far as it dying. could be issue similar to http://redmine.lighttpd.net/issues/2391
• 06:45 mutante: squids - upgrading sq44,48 (upload)
• 05:23 mutante: squids - finishing a couple reboots in the sq7x range
• 03:04 binasher: rebooting db1006 as well
• 03:04 binasher: rebooting db1038, kernel uptime scheduler chaos
• 02:00 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
• 00:21 logmsgbot: reedy synchronized php-1.20wmf2/extensions/GlobalBlocking/GlobalBlocking.class.php

May 4

• 23:46 logmsgbot: reedy synchronized php-1.20wmf2/extensions/GlobalBlocking/GlobalBlocking.class.php
• 23:45 logmsgbot: reedy synchronized php-1.20wmf1/extensions/GlobalBlocking/GlobalBlocking.class.php
• 22:35 logmsgbot: aaron synchronized php-1.20wmf2/includes/filerepo/backend/FSFileBackend.php 'deployed a807624'
• 22:34 LeslieCarr: clearing varnish cache and reloading varnish on mobile
• 21:14 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
• 21:13 logmsgbot: reedy ran sync-common-all
• 20:18 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Fix typo (cswikquote vs cswikiquote)'
• 20:06 logmsgbot: asher synchronized wmf-config/db.php 'setting s2 writable'
• 20:05 binasher: performing mysql replication steps for s2 master switch to db52
• 20:04 logmsgbot: asher synchronized wmf-config/db.php 'setting s2 read-only, db52 (still ro) as master, db13 removed'
• 19:49 logmsgbot: asher synchronized wmf-config/db.php 'setting db52 weight to 0 in prep for making new s2 master'
• 19:32 binasher: powering off db24
• 18:08 LeslieCarr: reloaded mobile varnish caches and purged them
• 18:02 Ryan_Lane: gerrit upgrade is done
• 17:55 Ryan_Lane: starting gerrit
• 17:32 Ryan_Lane: installing gerrit package on manganese
• 17:28 Ryan_Lane: adding gerrit 2.3 package to the repo
• 17:25 Ryan_Lane: shutting down gerrit so that everything can be backed up
• 16:45 apergos: lighty on dataset2 is running under gdb in screen session as root, if it dies please leave that alone (or look at it if you want to investigate)
• 16:26 notpeter: turning off db30 (former s2 db, still on hardy, will ask asher what to do with it) to test noise in DC
• 15:50 mutante: rebooting sq67 (bits)
• 15:42 mutante: going through sq7x servers (text), full upgrades
• 15:32 notpeter: removing srv281 from rending pool until we figure out what's going on with it
• 15:23 notpeter: putting srv224 back into pybal pool
• 15:09 notpeter: removing srv224 from pybal pool for repartitioning
• 14:56 notpeter: putting srv223 back into pybal pool
• 14:50 mutante: going through sq6x (text), full upgrades
• 14:08 notpeter: removing srv223 from pybal pool for repartitioning
• 14:02 notpeter: putting srv222 back into pybal pool
• 13:50 notpeter: removing srv222 from pybal pool for repartitioning
• 13:43 notpeter: putting srv221 back into pybal pool
• 13:30 notpeter: removing srv221 from pybal pool for repartitioning
• 13:16 mutante: going through sq80 to sq86 (upload), full upgrade & reboot
• 12:56 mutante: maximum uptime in the sq* group down to 171 days, so we have like a month now for the rest. stopping upgrades for the moment being.
• 12:54 notpeter: starting script to move /usr/local/apache to /a partition on all remaing non-imagescaler apaches
• 12:47 mutante: (just) new kernels & reboot - sq45,sq49 (upload)
• 12:30 mark: Sending ALL non-european upload traffic to eqiad
• 12:23 mutante: (just) new kernels & reboot - sq63 to sq66 (209 days up)
• 12:06 mutante: dist-upgrade & kernel & reboot - sq42,sq43 - rebooting upload squids one by one
• 11:48 mutante: powercycling srv266 one more time, but now creating RT for it, once already showed CPU issue before it was reinstalled recently
• 11:13 apergos: restarted lighty on dataset2 ... about ... half an hour ago. stupid case sensitivity
• 10:02 apergos: tossed knsq1 through 7 from squid_knams dsh nodegroups file, prolly lots more cleanup where that came from
• 09:34 mutante: dist-upgrade/kernel/reboot: sq37, sq41. rebooting upload squid sq41
• 08:49 mutante: dist-upgrade & new kernel & reboot: sq33, sq36
• 07:47 mutante: preemptive rebooting of sq* servers identified as having > 200 days of uptime
• 02:22 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Fri May 4 02:22:42 UTC 2012
• 02:13 logmsgbot: LocalisationUpdate completed (1.20wmf1) at Fri May 4 02:13:58 UTC 2012
• 00:20 logmsgbot: raindrift synchronizing Wikimedia installation... :
• 00:18 logmsgbot: raindrift synchronizing Wikimedia installation... : Syncing the PageTriage extension, but only enabling on testwiki
• 00:08 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Adding 'fr' to language codes for mobile feedback'
• 00:06 maplebed: moved ms1-3 from the production cluster to the test cluster

May 3

• 23:29 LeslieCarr: restarting networking on sq55
• 23:29 logmsgbot: tstarling synchronizing Wikimedia installation... :
• 23:27 LeslieCarr: restarting networking on sq54
• 23:24 LeslieCarr: restarting networking on sq53
• 23:21 LeslieCarr: restarting networking on sq52
• 23:16 LeslieCarr: restarting networking on sq51
• 21:30 notpeter: removing srv220 from pybal pool for repartitioning
• 21:29 LeslieCarr: switching asw-a4-sdtpa from single uplink to lag
• 21:19 notpeter: putting srv219 back into pybal pool
• 21:14 logmsgbot: asher synchronized wmf-config/db.php 'setting wgDefaultExternalStore to cluster23'
• 21:09 logmsgbot: asher synchronized wmf-config/db.php 'reverting cluster23 change'
• 21:05 logmsgbot: asher synchronized wmf-config/db.php 'setting wgDefaultExternalStore to cluster23'
• 21:02 binasher: about to move ES writes to cluster23
• 20:47 notpeter: removing srv219 from pybal pool for repartitioning
• 20:37 logmsgbot: reedy synchronized php-1.20wmf2/extensions/Collection/Collection.templates.php
• 20:37 logmsgbot: reedy synchronized php-1.20wmf1/extensions/Collection/Collection.templates.php
• 19:50 binasher: restarted profiling collector post parser.php livehack and stats.db removal
• 19:45 notpeter: starting script to move /usr/local/apache to /a partition on all non-imagescaler, non-jobrunner apaches
• 19:42 logmsgbot: aaron synchronized php-1.20wmf2/includes/parser/Parser.php 'live-hack out template profiling...again.'
• 19:40 logmsgbot: aaron synchronized php-1.20wmf1/includes/parser/Parser.php 'live-hack out template profiling...again.'
• 19:31 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Revert $wgDefaultUserOptions[enotifwatchlistpages] = 1'
• 19:20 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bug 36316 - Set Add pages I edit to my watchlist to true by default for new users'
• 19:15 logmsgbot: reedy synchronized wmf-config/CommonSettings.php '$wgDefaultUserOptions[enotifwatchlistpages] = 1'
• 19:09 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable show update markers for some more of the larger wikis'
• 19:00 paravoid: powercycling all of sq51-sq62, hanged due to 209 days uptime
• 18:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36092 - Activation of flood flag on vec.wikipedia.org'
• 18:43 paravoid: powercycling sq59; inaccessible via either SSH or serial due to load
• 18:41 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36183 - Fix namespace alias on Hindi Wikipedia'
• 18:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36171 - Imports from Wikibooks'
• 18:34 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36344 - Remove file upload facility on Gujarati wikipedia'
• 18:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36344 - Remove file upload facility on Gujarati wikipedia'
• 18:31 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36344 - Remove file upload facility on Gujarati wikipedia'
• 18:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36386 - cswikiquote user group changes'
• 18:15 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36480 - Create namespace Comments: in Greek Wikinews'
• 17:44 RobH: db1029 ssd test items removed, can go back to normal service via asher
• 17:43 notpeter: returning mw58 to pool
• 17:34 RobH: shutting down db1029 for ssd card testing removal per rt 2766
• 17:26 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36320 - Set $wgShowUpdatedMarker back to true on ptwiki'
• 17:18 notpeter: removing mw58 from pool for more testin'
• 17:16 LeslieCarr: reloaded and purged varnish cache for mobile in eqiad
• 17:03 notpeter: mwm59 out of apache pool. using it for some testing
• 16:16 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36359 - Add namespace 102 to $wgContentNamespaces on ptwiki Bug 36360 - Add namespace 102 to $wgNamespacesToBeSearchedDefault on ptwiki'
• 16:08 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bug 36460 - Enable chunked uploads as opt-in user preference'
• 16:06 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bug 31406 - Set $wgUseMathJax = true on Wikimedia wikis'
• 15:12 notpeter: chris is taking down search1-12 to replace with new search nodes
• 15:05 mutante: powercycling srv266
• 13:49 mark: Built new wikimedia-base 1.00 package, stripped of most stuff now handled by Puppet, and inserted it into the lucid-wikimedia and precise-wikimedia APT repositories
• 10:33 mutante: starting container-auditor on ms-be3
• 08:42 logmsgbot: ariel synchronized php-1.20wmf2/LocalSettings.php 'job runners don't have /home mounted'
• 08:16 Nemo_bis: siebrand: job queue stuck, on en.wiki jumped from o to 37k in the last ~36h
• 04:52 jeremyb: fixed complaints of beta simplewiki appearing in #cvn-simplewikis on freenode on the labs side. details
• 04:00 logmsgbot: tstarling synchronizing Wikimedia installation... :
• 02:47 logmsgbot: tstarling synchronizing Wikimedia installation... :
• 02:38 Tim: fixed scap, was failing on the remote side due to mwversionsinuse exiting with status 1 due to /home/wikipedia/common not existing on apaches
• 02:21 logmsgbot: tstarling synchronizing Wikimedia installation... :
• 02:21 Tim: aborted scap and re-ran with fanout=5 instead of 30, since nfs1 CPU was maxed out
• 02:14 logmsgbot: tstarling synchronizing Wikimedia installation... :
• 02:04 logmsgbot: aaron synchronized multiversion/activeMWVersions 'deployed r115116'
• 02:00 logmsgbot: LocalisationUpdate failed (php-1.20wmf2) at Thu May 3 02:00:13 UTC 2012
• 02:00 logmsgbot: LocalisationUpdate failed (php-1.20wmf1) at Thu May 3 02:00:12 UTC 2012

May 2

• 23:56 logmsgbot: aaron synchronized multiversion/ 'deployed svn HEAD'
• 23:41 maplebed: started swift old-object-deleter on ms-be3
• 23:28 maplebed: update - roan takes the blame
• 23:28 logmsgbot: raindrift synchronized wmf-config/InitialiseSettings.php 'Aborting todays PageTriage deployment'
• 23:22 maplebed: swift is recovered; ~20 minutes of impaired service. cause unknown, but the swiftcleaner looks likely.
• 23:18 RoanKattouw_away: Scap tried to push two new source trees to php-1.20wmf1-* and php-1.20wmf2-* , causing full disks. Cleaning up now
• 23:13 LeslieCarr: restarting nagios bot
• 22:59 logmsgbot: raindrift synchronizing Wikimedia installation... :
• 22:49 logmsgbot: preilly synchronized php-1.20wmf2/extensions/MobileFrontend/ 'contact us change'
• 22:48 logmsgbot: preilly synchronized php-1.20wmf1/extensions/MobileFrontend/ 'contact us change'
• 21:43 logmsgbot: asher synchronized wmf-config/db.php 's2: pulling db30, raising weights on new hosts'
• 21:02 ^demon: finished database maintenance on db9.reviewdb
• 20:24 hashar: hashar: updated TestSwarm to distribute tests to Firefox 12 users.
• 20:12 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Re-pushing for srv219 and srv220
• 20:07 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files:
• 20:04 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved special, wikimedia, wikiquote, wikiversity, and wiktionary wikis to 1.20wmf2
• 19:59 logmsgbot: asher synchronized wmf-config/db.php 'adding dbs 52,53,57 to s2 at lower weights'
• 19:55 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files:
• 19:47 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: metawiki to 1.20wmf2
• 19:40 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: commonswiki to 1.20wmf2
• 19:36 preilly: fix for PHP Warning: in_array() expects parameter 2 to be array, string given in /usr/local/apache/common-local/php-1.20wmf1/extensions/MobileFrontend/skins/SkinMobile.php on line 156
• 19:36 logmsgbot: preilly synchronized php-1.20wmf2/extensions/MobileFrontend/skins/SkinMobile.php 'fix php notice for in_array'
• 19:35 logmsgbot: preilly synchronized php-1.20wmf1/extensions/MobileFrontend/skins/SkinMobile.php 'fix php notice for in_array'
• 19:34 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved all wikibooks to 1.20wmf2
• 19:25 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved enwikibooks to 1.20wmf2
• 19:21 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved sourceswiki to 1.20wmf2
• 19:20 logmsgbot: preilly synchronized php-1.20wmf1/extensions/ZeroRatedMobileAccess/ 'zero weekly carrier test'
• 19:11 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved all wikisource sites to 1.20wmf2
• 19:03 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved all wikinews sites to 1.20wmf2
• 19:00 logmsgbot: preilly synchronized wmf-config/CommonSettings.php 'only remove images for DIGI'
• 18:51 logmsgbot: asher synchronized wmf-config/db.php 'added ES cluster23 to templateOverridesByCluster but not activating'
• 18:48 binasher: creating a blobs_cluster23 ES shard table for all active projects
• 18:31 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwikisource to 1.20wmf2
• 18:25 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwikinews to 1.20wmf2
• 18:24 RobH: updating dns
• 18:20 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwikinews to 1.20wmf2
• 18:09 logmsgbot: preilly synchronized wmf-config/CommonSettings.php 'only remove images for DIGI'
• 17:57 logmsgbot: preilly synchronized php-1.20wmf2/extensions/ZeroRatedMobileAccess/ 'zero weekly carrier test'
• 17:56 logmsgbot: preilly synchronized php-1.20wmf1/extensions/ZeroRatedMobileAccess/ 'zero weekly carrier test'
• 17:46 K4-713: updated production civicrm to r1726
• 17:36 logmsgbot: aaron synchronized php-1.20wmf2/includes/specials/SpecialContributions.php 'Deployed 799998c3a160ef6dd3b926b7d6fec223682b788c'
• 17:30 logmsgbot: preilly synchronized php-1.20wmf2/extensions/ZeroRatedMobileAccess/ 'zero weekly carrier test'
• 17:28 logmsgbot: preilly synchronized php-1.20wmf1/extensions/ZeroRatedMobileAccess/ 'zero weekly carrier test'
• 17:14 logmsgbot: catrope synchronized php-1.20wmf2/skins/vector/ 'Deploying 7260cc5fe4071e03241378ba1a48bc0b6f188948'
• 16:51 RoanKattouw: Changing docroot/bits/skins-1.19 and other 1.19 symlinks to point to the 1.20wmf1 tree instead. This is needed because we're still getting requests for magnify-clip.png at the 1.19 URL from cached HTML
• 16:16 notpeter: starting innobackupex from db1040 to db1022 for new s6 snapshot slave
• 15:31 notpeter: no nagios bot, kicking nagios on spence
• 15:04 RobH: shutting down mw64 for hw test per rt 1890
• 15:03 RobH: bellin crashed, unresponsive to ssh or serial console
• 14:43 mark: Built varnish for precise as 3.0.2-2wm5 and imported it into APT repository precise-wikimedia
• 11:52 mark: Started distribution upgrade of server stafford from Lucid to Precise
• 10:41 mutante: refreshLinks.php - started it once again in a screen on hume, just for s1. last cron failed with "mwscript command not found"?? well now it is there again and running
• 10:09 mark___: Started distribution upgrade of server sockpuppet from Lucid to Precise
• 09:20 mutante: upgrading bugzilla to 4.0.6
• 08:43 mutante: kaulen: installing various upgrades (apache,mysql,cron,php-wikidiff2,...)
• 08:40 logmsgbot: hashar synchronized php-1.20wmf2/includes/GitInfo.php 'Fix Special:Version for 1.20wmf2 (commit ae12df0 , bug 36361 )'
• 08:20 hashar: cherry-picked ae12df0 commit to 1.20wmf2 since there are mobilefrontend commits pending.
• 02:35 logmsgbot: LocalisationUpdate completed (1.20wmf2) at Wed May 2 02:35:51 UTC 2012
• 02:32 K4-713: updated production civicrm to r1723
• 02:13 logmsgbot: LocalisationUpdate completed (1.20wmf1) at Wed May 2 02:13:30 UTC 2012
• 01:01 notpeter: starting innobackupex from db57 to db53 for new s2 slave for the one zillionth time

May 1

• 22:28 logmsgbot_: asher synchronized wmf-config/db.php 'returning db45'
• 22:23 logmsgbot_: asher synchronized wmf-config/db.php 'pulling db45, last coredb on prior fb mysql build'
• 22:17 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Enable doublepage on test2wiki'
• 22:11 binasher: upgraded percona-toolkit on coredbs to 2.1.1 - now with the potential to run online schema changes on tables without single column unique keys!!
• 21:39 binasher: created an ops db on all core mysql shards
• 21:00 notpeter: reinstalling db53. this time with correct raid!
• 20:40 logmsgbot_: awjrichards synchronized wmf-config/CommonSettings.php 'Fixing mailto links on mobilefrontend feedback form to properly populate subject lines'
• 19:32 LeslieCarr: reverting vrrp mastership of row a to cr2-eqiad
• 19:29 LeslieCarr: switching vrrp mastership of row a to cr1-eqiad
• 18:32 logmsgbot_: awjrichards synchronized wmf-config/InitialiseSettings.php 'Make testwiki use mobile domain for URLs'
• 18:28 LeslieCarr: making routing change, higher risk
• 17:51 Ryan_Lane: make that virt0
• 17:51 Ryan_Lane: switching the session cache back to filesystem on virt1, since it isn't working properly with memcache
• 17:29 maplebed: kicking nagios to check a change to fix the mobile LVS alert
• 17:25 logmsgbot_: nikerabbit synchronized php-1.20wmf2/extensions/TranslationNotifications/ 'Deploying TranslationNotifications code'
• 17:08 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki to 1.20wmf2
• 16:27 notpeter: starting innobackupex from db1034 to db53 for new s2 slave
• 16:27 notpeter: starting innobackupex from db57 to db52 for new s2 slave
• 16:03 notpeter: rebuilding db52 and db53 as s2 slaves
• 15:47 logmsgbot_: asher synchronized wmf-config/db.php 's1: raising db59,60 weights, pulling db52/53 for reuse'
• 09:23 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'hewiki account creation high throttle limits'
• 04:04 Tim: on all apaches, running "chmod -R a+rX /usr/local/apache/common-local/" to clean up after killed rsyncs which left files unreadable
• 02:23 logmsgbot_: LocalisationUpdate completed (1.20wmf2) at Tue May 1 02:23:29 UTC 2012
• 02:21 logmsgbot_: preilly synchronized php-1.20wmf1/extensions/MobileFrontend/specials/SpecialMobileFeedback.php
• 02:14 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Tue May 1 02:14:06 UTC 2012
• 02:06 logmsgbot_: preilly synchronized php-1.20wmf1/extensions/MobileFrontend/specials/SpecialMobileOptions.php
• 01:51 Ryan_Lane: bringing up all labs instances with a 60 second lag
• 01:40 Ryan_Lane: rebooting virt0
• 01:35 Ryan_Lane: rebooting virt3
• 01:33 logmsgbot_: preilly synchronized php-1.20wmf1/extensions/MobileFrontend/HtmlFormatter.php
• 01:26 Ryan_Lane: rebooting virt5
• 01:18 Ryan_Lane: rebooting virt4
• 01:03 Ryan_Lane: rebooting virt2
• 00:51 LeslieCarr: restarted swift-container-auditor on ms-be5
• 00:38 logmsgbot_: tstarling synchronizing Wikimedia installation... :
• 00:26 Tim: removed large syslogs from mw60 and ran sync-common
• 00:18 Tim: on mw60 there was an actual directory at /usr/local/apache/common/php where a symlink should have been. fixed

April 30

• 23:58 logmsgbot_: aaron synchronized php
• 23:44 RoanKattouw: Started Apache back up on mw60
• 23:39 RoanKattouw: Running scap-1 on the Apaches with dsh
• 23:38 RoanKattouw: Moved /home/catrope/php-1.19 to /home/wikipedia/lazy-backups/php-1.19
• 23:38 Reedy: mediawiki.org to 1.20wmf2
• 23:37 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mw.org to 1.20wmf2
• 23:35 RoanKattouw: Strike that, instead moving /home/w/common/php-1.19 to /home/catrope/php-1.19
• 23:34 RoanKattouw: Removing /home/w/common/php-1.19 , NFS might freak out a bit
• 23:31 RoanKattouw: Removed php-1.19 from mw60 , synced it, and restarted Apache
• 23:28 RoanKattouw: Synced docroot and purged varnish for static-1.20wmf2, bits seems to be working for 1.20wmf2 now
• 23:27 RoanKattouw: mw60 has full disk, stopping Apache for now
• 22:50 Ryan_Lane: rebooting virt5
• 22:42 Ryan_Lane: rebooting virt3
• 22:35 Ryan_Lane: rebooting virt4
• 22:28 Ryan_Lane: rebooting virt1
• 22:23 Ryan_Lane: bringing down all instances (yay gluster)
• 21:12 pgehres: re-enabled Jenkins jobs on Aluminium after db1008 reboot
• 21:11 pgehres: CiviCRM back to normal after db1008 reboot
• 21:07 Jeff_Green: db1008 gets kernel update and reboot
• 21:00 pgehres: put CiviCRM on Aluminium in maintenance mode for db1008 reboot
• 20:59 logmsgbot_: reedy synchronized php-1.20wmf2/resources/startup.js 'touch'
• 20:57 pgehres: disabled all Jenkins jobs on Aluminium in prep for db1008 reboot
• 20:50 Jeff_Green: db1025 and storage3 get new kernels and reboot
• 20:28 notpeter: restarting, once again, innobackupex from db1034 to db57 for new s2 slave after fenari crash killed my screen
• 20:24 Reedy: Running ddsh -F30 -cM -g mediawiki-installation -o -oSetupTimeout=10 '/usr/bin/scap-1' in the hope it syncs all the files that would be nice to be on the app servers
• 20:18 logmsgbot_: reedy synchronized php-1.20wmf2/cache/ 'Synching whole cache directory'
• 19:59 notpeter: restarting nagios to get rid of some old checks
• 19:57 Jeff_Green: payments cluster gets kernel updates and reboots
• 19:55 logmsgbot_: reedy synchronizing Wikimedia installation... : Rebuiild l10n for 1.20wmf2
• 19:49 logmsgbot_: reedy synchronized wmf-config/ExtensionMessages-1.20wmf2.php 'Syncing file'
• 19:49 logmsgbot_: reedy synchronized php-1.20wmf2/LocalSettings.php 'Pushing LocalSettings.php'
• 19:48 paravoid: upgraded & rebooted ssl3001, ssl3002, ssl3003
• 19:45 logmsgbot_: reedy synchronizing Wikimedia installation... : Pushing out new symlinks etc, moving test2wiki to 1.20wmf2
• 19:30 logmsgbot_: reedy synchronized php-1.20wmf2 'Syncing 1.20wmf2 live hack revisions'
• 19:28 logmsgbot_: reedy synchronized php-1.20wmf2 'Syncing 1.20wmf1 live hack revisions'
• 19:26 logmsgbot_: reedy synchronized php-1.20wmf2 'Syncing 1.20wmf2 for deployment'
• 19:18 Reedy: Syncing php-1.20wmf2 files from NFS to apaches. Likely to upset NFS (or the uplink for the switch nfs is on) for a little while...
• 19:14 paravoid: rebooting ssl1004
• 19:06 paravoid: rebooting ssl1003
• 19:00 paravoid: rebooting ssl1002
• 18:59 notpeter: starting innobackupex from db1034 to db57 for new s2 slave
• 18:50 paravoid: rebooting ssl1001
• 18:42 Jeff_Green: grosley gets new kernel + reboot
• 18:35 Jeff_Green: aluminium gets kernel update, yayyyyyyy!
• 18:34 paravoid: pooled back ssl1; depooling ssl3 and rebooting
• 18:29 binasher: rebooting mw45 for kernel upgrade
• 18:27 Jeff_Green: power cycling aluminium which faceplanted
• 18:22 binasher: rebooting mw45
• 18:21 notpeter: rebuilding db57 again, this time with more correct raid level!
• 18:19 logmsgbot_: asher synchronized wmf-config/db.php 'adding db59,60 to s1 with low weights'
• 18:16 paravoid: depooled & rebooting ssl1
• 18:09 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Sanity run after script changes.
• 18:00 logmsgbot_: aaron synchronized multiversion
• 17:58 logmsgbot_: reedy synchronized php-1.20wmf1/includes/MagicWord.php 'https://gerrit.wikimedia.org/r/6135'
• 17:44 logmsgbot_: aaron synchronized wikiversions.cdb
• 17:43 AaronSchulz: updating multiversion code
• 08:34 mutante: reinstalling srv266
• 08:08 mutante: upgraded mw1,mw2,mw35
• 07:59 mutante: reinstalling srv206
• 07:50 mutante: upgrading mw36
• 07:37 apergos: powercycling srv266, had this message on mgmt console: Severity: Non Recoverable, SEL:CPU Machine Chk: Processor sensor, transition to non-recoverable was asserted
• 07:22 mutante: installing upgrades on srv212
• 07:19 apergos: reinstalled srv284, seems to be up now
• 07:17 mutante: powercycled mw8
• 02:14 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Mon Apr 30 02:13:59 UTC 2012

April 29

• 20:13 apergos: srv206 won't run puppet, see syslog, clearing out the yaml file didn't help, since it's not urgent I'm leaving it for tomorrow
• 19:51 Ryan_Lane: depooling ssl3004
• 19:51 Ryan_Lane: removed the ipv6 addresses from maerlant and added them to ssl3001, then restarted nginx
• 19:50 Ryan_Lane: repooling ssl3001
• 19:46 apergos: powercycled mw60, same reason as the rest
• 19:12 apergos: power cycled mw48 and mw52 (hung just like the others)
• 18:05 apergos: sll3002 and 3003 were rebooted and are the entire ssl esams pool right now
• 18:02 apergos: ok the ssl300x situation: ssl3001 is now disabled in the pybal conf file on fenari; it is picking up the ipv6and4labs tmplate and I don't know if that's right, anyways nginx doesn't want to bind to one of those addresses. ssl3004 isn't reachable or pingable even via mgmt but at leasy lvs sees it's gone
• 16:34 apergos: powercycling the ssl300x.esams hosts. 212 days of uptime... (and 3001 had gone out to lunch)
• 12:34 mutante: and finally mw1, so just leaving mw1102 and mw60 for having other issues for a while (->Nagios)
• 12:22 mutante: check_all_memcached recovered, but still same treatment for mw10 and 11 (8 and 15h ago)
• 12:15 mutante: powercycling mw32,mw33,mw44,mw46 one by one, they were all frozen and went down between like 17 and 24 hours ago approx.
• 12:07 mutante: powercycling mw30
• 02:56 paravoid: rebooting ssl2 (has 214 days uptime)
• 02:47 paravoid: powercycled ssl3
• 02:13 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Sun Apr 29 02:13:58 UTC 2012

April 28

• 22:53 Reedy: Job queue logs on gdash seem to have stopped on the 26th...
• 22:29 logmsgbot_: reedy synchronized php-1.20wmf1/includes/EditPage.php 'https://gerrit.wikimedia.org/r/6088'
• 21:52 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
• 21:51 logmsgbot_: reedy synchronized php-1.20wmf1/extensions/cldr/LanguageNames.body.php
• 21:12 logmsgbot_: reedy synchronized php-1.20wmf1/extensions/cldr/LanguageNames.body.php
• 21:10 logmsgbot_: reedy synchronized php-1.20wmf1/extensions/cldr/LanguageNames.body.php
• 21:09 logmsgbot_: reedy synchronized common/php-1.20wmf1/extensions/cldr/LanguageNames.body.php 'more debugging'
• 20:51 logmsgbot_: reedy synchronized php-1.20wmf1/extensions/cldr/LanguageNames.body.php 'Add debugging'
• 20:49 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Add debuglog group for language code not being a string'
• 19:04 logmsgbot_: reedy synchronized php-1.20wmf1/includes/ExternalEdit.php 'https://gerrit.wikimedia.org/r/6077'
• 19:03 logmsgbot_: reedy synchronized php-1.20wmf1/includes/api/ApiParse.php 'https://gerrit.wikimedia.org/r/6076'
• 02:24 Ryan_Lane: rebooting all mediawiki boxes that have uptimes affected by the bug are being rebooted at 8 minute intervals
• 02:14 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Sat Apr 28 02:14:14 UTC 2012
• 01:33 paravoid: powecycled mw29
• 01:21 paravoid: powercycled mw38
• 00:17 notpeter: db12 is sooooo sloooooow, starting innobackupex from db1017 to db60 for new s1 slave

April 27

• 22:15 paravoid: upgraded ssl4 to nginx 0.7.65-5wmf1 and added it back to the pool
• 21:45 paravoid: rebooting ssl4 after upgrading (incl. a kernel update)
• 20:00 notpeter: starting innobackupex from db1040 to db1022 for new eqiad s6 snapshot slave, again
• 19:59 notpeter: starting innobackupex from db12 to db60 for new s1 slave, again
• 19:58 notpeter: starting innobackupex from db1017 to db59 for new s1 slave, again
• 19:49 paravoid: de-pooling ssl4
• 19:30 mutante: test - added new gerrit interwiki prefix for SAL/wikitech - gerrit:6002
• 19:14 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php 'Fix rights for afttest and afttest-hide groups'
• 18:25 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Cleanup enotif related settings'
• 18:24 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Set wgEnotifWatchlist to true for all wikis. Leaving wgShowUpdatedMarker set to false for all the big wikis'
• 16:50 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Simplify enotif code'
• 16:45 notpeter: starting innobackupex from db1040 to db1022 for new eqiad s6 snapshot slave
• 16:45 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'wgEnotifWatchlist defaulting to true. Big wikis explicitly set to false'
• 12:25 mutante: fixing integration.mw testswarm and applying fixed erb template by hashar
• 04:35 Tim: added an account for myself on observium
• 04:22 logmsgbot_: tstarling synchronized wmf-config/mc.php 'increased wgMemCachedTimeout from 500ms to 3000ms for bug 35900'
• 02:13 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Fri Apr 27 02:13:51 UTC 2012
• 00:12 Ryan_Lane: upgrading gluster on all instances
• 00:09 Ryan_Lane: upgrading gluster on labstore1-4

April 26

• 23:46 logmsgbot_: asher synchronized wmf-config/db.php 'raising db58 weight'
• 23:09 Reedy: Recreated resources directory symlinks in bits docroot
• 21:21 LeslieCarr: started deletion script on ms-be4
• 19:20 notpeter: restarting puppet on db59
• 19:18 Ryan_Lane: made LiquidThreads disabled by default on labsconsole, now users must add the special string to a page to enable it there.
• 19:18 Ryan_Lane: enabled NewUserMessage on labsconsole
• 19:06 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php 'Add group permissions settings for AFTv5'
• 18:33 logmsgbot_: catrope synchronizing Wikimedia installation... : Deploy AFTv5 updates
• 17:17 LeslieCarr: reloaded varnish on mobile caches
• 14:19 notpeter: cleaned log space on search1017 and search1018 and started lucene
• 14:04 notpeter: stopping lucene on search1017 and 1018 to take that out of the equation
• 13:57 mutante: installing some (security) upgrades on fenari (apt,cron,samba,...)
• 13:54 notpeter: restartin lucene on search1017 and search1018
• 13:27 logmsgbot_: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Narayamon tewiki bug 33480'
• 13:23 logmsgbot_: nikerabbit synchronized php-1.20wmf1/extensions/Narayam/ 'Updating Narayam'
• 13:03 notpeter: (re)starting innobackupex from db1017 to db59 for new s1 slave
• 12:56 mark: Created precise-wikimedia APT distribution
• 08:27 mark: Power cycled mw40
• 06:57 binasher: restart pybal on amlvs1 with bgp disabled
• 06:57 binasher: restarted pybal on amlvs2 with bgp enabled
• 06:47 binasher: restarting pybal on amslvs2
• 06:26 binasher: shifting all traffic out of esams
• 02:14 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Thu Apr 26 02:14:03 UTC 2012
• 01:42 Ryan_Lane: starting mysql on db46
• 01:40 Tim: on professor: restarted udpprofile collector
• 01:37 Ryan_Lane: powercycling db46
• 01:33 logmsgbot_: asher synchronized wmf-config/db.php 'pulling db46, host down'
• 00:44 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php

April 25

• 22:14 LeslieCarr: restarted swift-container-auditor on ms-be3
• 21:55 RobH: pushing dns update for scs-c1-eqiad and ps1-c#-eqiad
• 21:22 LeslieCarr: reloading varnish on mobile caches cp1041 cp1042 cp1043 cp1044
• 21:21 LeslieCarr: clearing mobile varnish cache
• 19:38 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'Attempted fatal fix'
• 19:33 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/Math/ 'Deploying 4c9e7dbe761c798ce15d7e2acef829a1582c058b'
• 19:14 notpeter: starting innobackupex from db12 to db59 for new s1 slave, per mr. feldman's directions
• 18:56 notpeter: starting innobackupex from db1017 to db60 for new s1 slave
• 18:49 logmsgbot_: aaron synchronized php-1.20wmf1/extensions/FeaturedFeeds/SpecialFeedItem.php 'Deployed 4fb14a7b2ca9be715b820a9847d999f21c7d2cfc'
• 18:36 logmsgbot_: aaron synchronized php-1.20wmf1/img_auth.php 'Deployed f7e49bd71bd8356751242c5ce1cbae076a27cf7a'
• 18:10 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moving all remaining wikis to php-1.20wmf1
• 17:07 LeslieCarr: reloaded mobile varnish configs
• 17:06 LeslieCarr: purging mobile cache
• 16:40 LeslieCarr: starting delete script on ms-be3
• 16:14 RobH: done moving mgmt connections and serial connections in s8-eqiad for now
• 16:05 RobH: reshuffling cables in eqiad for serial and mgmt connections in a8, this may affect all eqiad mgmt and serial connections for the next 5 minutes
• 15:29 hashar: hashar: gallium: MySQL had issues most probably because of the mysql configuration snippets. https://gerrit.wikimedia.org/r/5796 might solve that.
• 14:03 mutante: gallium - don't start puppet unless the erb template fix for mysql has been merged
• 13:52 mutante: gallium stopped puppet, moved log_slow_queries config, re-setting up mysql again
• 13:41 mutante: gallium/testswarm - back up after mysql upgrade and issue starting the service
• 13:36 mutante: gallium - dpkg-reconfigure mysql-server-5.1, mysql does not start right
• 13:27 mutante: running apt-get upgrade on gallium
• 12:29 mark: Sending US, Brazil, Indian traffic to upload.eqiad
• 11:39 mutante: running authdns-update to add analytics100x and labsdb100x mgmt names
• 05:35 paravoid: powercycled lvs6, was dead and not responding to serial
• 03:43 logmsgbot_: asher synchronized wmf-config/db.php 'adding db58 to s7 as a new slave with a low weight'
• 03:24 logmsgbot_: asher synchronized wmf-config/db.php 'pulling db58'
• 03:23 logmsgbot_: asher synchronized wmf-config/db.php 'adding db58 to s7 as a new slave with a low weight'
• 02:28 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Wed Apr 25 02:28:47 UTC 2012
• 02:14 logmsgbot_: LocalisationUpdate completed (1.19) at Wed Apr 25 02:14:46 UTC 2012
• 00:02 binasher: profiling collector was pegged at 100% cpu and graphs were turned to swiss cheese due to a bad stats call in 1.20, now fixed

April 24

• 23:59 binasher: powering off db16
• 23:55 binasher: streaming hot backup of db1041 to db58 (building a new s7 slave)
• 23:48 logmsgbot_: aaron synchronized php-1.19/includes/Setup.php 'Hacked out session request stats.'
• 23:46 logmsgbot_: aaron synchronized php-1.20wmf1/includes/Setup.php 'Deployed 42fcd43299246ecd1b265fcfcdd01a60319cf378'
• 23:19 AaronSchulz: Running 'mwscriptwikiset maintenance/populateRevisionSha1.php all.dblist' on hume
• 22:43 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Enabled file change journal on wikis using the new backend config.'
• 22:20 AaronSchulz: Tables added
• 22:18 binasher: rebooting db16 with updated kernel. it's probably still hopeless (dimm errors)
• 22:18 AaronSchulz: Creating the filejournal table on all wikis
• 21:59 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Switched commonswiki to the new backend config format.'
• 21:48 logmsgbot_: asher synchronized wmf-config/db.php 'pulling db16, memory errors'
• 20:13 apergos: re-enabled replication via cron on ms7, it should catch up within an hour or so
• 20:10 binasher: reimaged db58 with fixed raid setup, imaging db59
• 19:51 notpeter: starting innobackupex from db1034 to db57 for new s2 slave
• 19:50 Ryan_Lane: repooling ssl3001
• 19:28 Ryan_Lane: depooling ssl3001
• 18:18 LeslieCarr: deploying to frontend
• 17:48 notpeter: deploying new squid conf to cp1001 frontend. is just a udp2log port change.
• 17:19 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Using newer backend for shared repos for testwiki, test2wiki, and mediawikiwiki.'
• 16:55 logmsgbot_: nikerabbit synchronized wmf-config/CommonSettings.php 'Translate extension configuration changes'
• 11:54 apergos: after much cursing and kicking zfs, a manual snapshot replication is running in screen as root on ms7 to ms8, expect it to take at least a day
• 11:44 mark: Sending all non-european upload traffic back to pmtpa to prepare for eqiad varnish storage rework
• 08:56 mutante: updated blog theme per guillaume (April commits)
• 08:05 apergos: temporarily disabled automatic zfs replication from ms7 -> ms8, cleared out space on ms8, catching up by hand
• 04:00 Ryan_Lane: powercycling ssl1
• 02:47 logmsgbot_: aaron synchronized php-1.19/extensions/MobileFrontend/MobileFrontend.body.php 'Fixed notice spam.'
• 02:45 logmsgbot_: aaron synchronized php-1.20wmf1/extensions/MobileFrontend/MobileFrontend.body.php 'Fixed notice spam.'
• 02:37 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Restructed filerepo a config a bit; nothing changed yet.'
• 02:28 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Tue Apr 24 02:28:47 UTC 2012
• 02:15 logmsgbot_: LocalisationUpdate completed (1.19) at Tue Apr 24 02:15:00 UTC 2012
• 00:15 logmsgbot_: awjrichards synchronized php-1.20wmf1/extensions/MobileFrontend/stylesheets/common.css '0be2dc1288361c51f91533f1f77e78d9279b86e0'
• 00:13 logmsgbot_: awjrichards synchronized php/extensions/MobileFrontend/stylesheets/common.css 'r115019'

April 23

• 23:35 logmsgbot_: awjrichards synchronized wmf-config/CommonSettings.php 'Bumpging MobileFrontend resource version'
• 23:07 logmsgbot_: awjrichards synchronizing Wikimedia installation... :
• 23:02 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php 'Add code for new URL scheme based on version_compare() logic'
• 22:51 logmsgbot_: awjrichards synchronizing Wikimedia installation... : MobileFrontend updates per http://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments#23_April.2C_2012
• 22:33 logmsgbot_: awjrichards synchronized wmf-config/CommonSettings.php 'bumping mobile frontend resource version'
• 21:49 logmsgbot_: catrope synchronized php-1.20wmf1/resources/mediawiki/mediawiki.js 'Deploy 6e55a770b26b17b8fc9b5b4fe943dcc2867df4f3'
• 21:27 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/ArticleFeedbackv5/modules/jquery.articleFeedbackv5/jquery.articleFeedbackv5.js 'Deploy 93d470b'
• 20:41 mutante: neon - upgraded libssl, started icinga after adding monitor group
• 20:32 logmsgbot_: aaron synchronized php-1.20wmf1/includes/filerepo/FileRepo.php 'Disabled write checks in the cleanDir() function.'
• 20:31 logmsgbot_: aaron synchronized php-1.20wmf1/includes/filerepo/FileRepo.php 'Disabled write checks in the quickImport/quickPurge functions.'
• 19:43 logmsgbot_: catrope synchronized php-1.20wmf1/includes/specials/SpecialListgrouprights.php 'Deploy 047543b6805a268c8d689a7a1ce12ec545ef79a9'
• 18:43 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'touch'
• 18:43 logmsgbot_: reedy synchronized flaggedrevs.dblist 'Seems I never added ukwiki to the dblist... Oh well'
• 18:32 logmsgbot_: aaron synchronized wikiversions.dat
• 18:32 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved enwiki to 1.20wmf1
• 18:28 logmsgbot_: aaron synchronized php-1.20wmf1/includes/specials/SpecialContributions.php 'Deployed 72969cf8c9a403430c8c93fc20ab3118328c4d9c'
• 17:06 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Made mediawikiwiki use the newer backend config.'
• 14:33 notpeter: stopping puppet on cp1041 as well
• 14:17 notpeter: temp stopping puppet on cp1042-1044
• 13:09 mutante: powercycling frozen mw25, looks like mw21 above but no console output to paste here
• 13:07 mutante: fix puppet run on spence by removing searchidx1 resources from db9 (was in weird state being in site but also decommissioned)
• 11:23 mutante: mw21 powercycling mw21 - it died with this http://etherpad.wikimedia.org/mw21
• 10:55 mutante: force-reload ircecho on manganese to make gerrit-wm rejoin #mediawiki
• 10:48 hashar: banned CIA bots from #mediawiki IRC channel. It started spamming us with notifications from KDE and mandriva projects. See http://permalink.gmane.org/gmane.science.linguistics.wikipedia.technical/60905
• 10:30 mutante: searchidx1 was in site.pp and decom.pp at the same time. breaks puppet runs on spence. cannot override local resource. removing from site
• 10:27 mutante: killed a couple morebots processes on wikitech and it came back by itself :p

April 21

• 02:29 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Sat Apr 21 02:29:40 UTC 2012
• 02:15 logmsgbot_: LocalisationUpdate completed (1.19) at Sat Apr 21 02:15:20 UTC 2012

April 20

• 22:03 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Switched test2wiki to use the new LocalRepo config style.'
• 22:01 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Switched testwiki to use the new LocalRepo config style.'
• 21:52 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Added NFS backends for local/shared repos; they are not used yet.'
• 21:12 LeslieCarr: starting swift delete script on ms-be2
• 20:02 logmsgbot_: aaron synchronized php-1.20wmf1/includes/filerepo/file/LocalFile.php 'deployed c77fbd394cda701758ad4523113f567bff7ede66'
• 19:45 apergos: powercycled mw4, it was unresponsive to pings and via mgmt
• 18:48 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/ArticleFeedbackv5 'Apply https://gerrit.wikimedia.org/r/5449'
• 18:48 logmsgbot_: catrope synchronized php-1.19/extensions/ArticleFeedbackv5 'Apply https://gerrit.wikimedia.org/r/5449'
• 18:07 notpeter: restarting nginx on ssl1002 and ssl1004 as they are not back up
• 18:01 logmsgbot_: awjrichards synchronizing Wikimedia installation... :
• 17:31 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php 'Remoev wgArticleFeedbackv5OversightEmails override that was messing things up'
• 17:15 notpeter: stopping puppet on locke and emery. just to be safe...
• 17:11 RoanKattouw: Fixed ownership of /h/w/common/php-1.20wmf1/cache/l10n , should be owned by l10nupdate but was owned by reedy
• 17:01 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36124 - Deploy ProofreadPage extension on test2'
• 17:00 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Giving test2wiki moar namespaces'
• 16:11 mutante: add missing memcached servicegroup to nagios, restarted
• 15:10 mutante: apache error log on stafford has ruby exceptions re: phusion_passenger
• 15:01 mark: Converted OSPF directly connected redistributed routes from type 2 to type 1
• 14:51 mutante: starting swift-container-auditor on ms-be1
• 14:30 mark: Disabled down-pref of Tampa AS2828 routes
• 13:14 logmsgbot_: demon synchronized php-1.20wmf1/maintenance/backupTextPass.inc 'Pushing out Idb58ce27 for Ariel/Chris for dumps'
• 13:10 mark: Sending India upload traffic to upload-lb.eqiad
• 12:40 mark: Disabled iptables firewalls on internal prod swift cluster servers as it's dropping packets
• 12:22 mutante: restarted pdns on ns2
• 11:19 mark: Sending US upload traffic to eqiad as well
• 10:27 mark: Sending Brazil upload traffic to eqiad
• 08:39 hashar: Gave up running l10nupdate script it has some file permissions issues. Opened bug 36119 and bug 36120
• 08:36 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Fri Apr 20 08:36:53 UTC 2012
• 08:27 logmsgbot_: LocalisationUpdate completed (1.19) at Fri Apr 20 08:27:36 UTC 2012
• 08:13 hashar: rerunning l10nupdate for bug 34938
• 08:02 hashar: running l10nupdate for bug 34938
• 06:27 pgehres: re-eanabled PayPal on donatewiki and wmfwiki and resumed queue consumer on Aluminium
• 05:32 LeslieCarr: flushing mobile varnish cache
• 04:56 pgehres: disabled paypal on donatewiki and disabled queue consumer for duration of PayPal outage
• 02:33 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Fri Apr 20 02:33:02 UTC 2012
• 02:23 logmsgbot_: LocalisationUpdate completed (1.19) at Fri Apr 20 02:23:57 UTC 2012
• 01:47 logmsgbot_: awjrichards synchronizing Wikimedia installation... : r114983 on wikis still running 1.19

April 19

• 23:33 binasher: powercycled es1004
• 21:08 Jeff_Green: changed nagios contactgroup fundraising from tfinc/awrichards --> jgreen
• 21:03 RoanKattouw: Scap is broken in some weird way, it just stops running after the scap1-skins step. Doesn't run scap-1 (which does the actual sync), doesn't log "sync done", doesn't update graphite
• 21:01 logmsgbot_: catrope synchronizing Wikimedia installation... : Running scap again, AFTv5 is acting up
• 19:34 logmsgbot_: catrope synchronizing Wikimedia installation... : ArticleFeedbackv5 updates
• 19:29 RoanKattouw: Running scap to deploy AFTv5 updates, and running AFTv5 schema changes on enwiki at the same time
• 18:50 logmsgbot_: catrope synchronized wmf-config/InitialiseSettings.php 'Set wmgArticleFeedbackv5OversightEmails for enwiki'
• 18:25 notpeter: nothing obvious in logs on db1005, starting mysql
• 18:15 notpeter: rebooting db1005. it's dead, jim.
• 17:52 RoanKattouw: Running schema changes for AFTv5 on testwiki
• 17:51 Jeff_Green: discovered nfs1 had ~1K redundant iptables rules, removed extras and reloaded
• 17:42 Jeff_Green: discovered sanger had ~7K redundant iptables rules, removed extras and reloaded
• 13:56 mutante: adding refreshLinks cron jobs to hume per RT-2355 (via puppet). if there should be any performance issues, schedule can be changed like <cluster>@<hour> in mediawiki.pp (and/or remove mediawiki::refreshlinks from hume and clear out the jobs of user mwdeploy)
• 08:35 mutante: emery - "udp2log_age" says some squid logfiles have not been written to in 6 hours, but from the filenames looks like this isnt a reason to worry, right
• 07:49 mutante: stat1 - this also needs udp2log stuff fixed. currently Could not find class misc::udp2log::udp-filter
• 07:47 mutante: gilman - what's up with it? closes SSH, does not like mgmt pass, was running jenkins but broken
• 07:43 mutante: owa[1-3] They dont have real puppet freshness issues, it's rather firewalling and the snmp traps
• 02:30 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Thu Apr 19 02:30:33 UTC 2012
• 02:21 logmsgbot_: LocalisationUpdate completed (1.19) at Thu Apr 19 02:21:31 UTC 2012

April 18

• 22:55 LeslieCarr: updating exim4.conf on mchenry to not allow old ranges
• 21:03 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files:
• 20:47 logmsgbot_: catrope synchronized php-1.20wmf1/resources/startup.js 'touch'
• 20:46 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/SyntaxHighlight_GeSHi/ 'Deploying GeSHi fix https://gerrit.wikimedia.org/r/#change,4949'
• 20:04 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: specieswiki and foundationwiki to 1.20wmf1
• 19:56 logmsgbot_: aaron synchronized php-1.20wmf1/extensions/LiquidThreads/classes/Hooks.php 'Avoid fatals on invalid title in API'
• 19:51 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: All *wiki wikis to 1.20wmf1
• 19:25 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: All wikiquote and wikiversity projects to 1.20wmf1
• 19:22 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: All wikibooks to 1.20wmf1
• 19:18 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: All wikinewses to 1.20wmf1
• 19:07 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: All wikisources to 1.20wmf1
• 19:07 logmsgbot_: catrope synchronized wmf-config/mc.php 'Swap out 10.0.2.251 (down) with 10.0.11.24 (spare). This is the last spare, there are now NO SPARES LEFT in mc.php'
• 19:00 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: All wiktionaries to 1.20wmf1
• 18:57 logmsgbot_: aaron synchronized php-1.20wmf1/extensions/LiquidThreads/classes/Dispatch.php 'Added type hint for better fatals'
• 18:44 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwikiversity to 1.20wmf1
• 18:43 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwikiquote to 1.20wmf1
• 18:41 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwikibooks to 1.20wmf1
• 18:40 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwikinews to 1.20wmf1
• 18:39 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwiktionary to 1.20wmf1
• 18:32 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwikisource to 1.20wmf1
• 17:20 logmsgbot_: catrope synchronized docroot/bits/ 'Remove static-1.00 again'
• 16:57 logmsgbot_: catrope synchronized docroot/bits 'Add docroot/bits/static-1.00 for testing'
• 16:41 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Set wmfUseRevSha1Columns to true for enwiki'
• 13:30 mutante: applied a patch to etherpad that allows admins to delete pads
• 12:53 mutante: restarting/fixing etherpad issue
• 11:08 mark: Sending European bits traffic back to esams
• 02:30 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Wed Apr 18 02:30:50 UTC 2012
• 02:21 logmsgbot_: LocalisationUpdate completed (1.19) at Wed Apr 18 02:21:49 UTC 2012
• 02:13 logmsgbot_: catrope synchronized php-1.20wmf1/README 'Dummy sync to capture which hosts time out on sync-file'
• 00:52 K4-713: updated production civi to r1631
• 00:41 Ryan_Lane: adding interface for per-project sudo on OpenStackManager

April 17

• 23:36 K4-713: updated production civi to r1628
• 23:12 logmsgbot_: catrope synchronized wmf-config/InitialiseSettings.php 'Fixes for cswiktionary changes per Danny B'
• 22:49 RoanKattouw: That was bug 34885 of course
• 22:43 logmsgbot_: catrope synchronized php-1.19/extensions/WikiEditor/ 'Deploy fix for bug 348885'
• 22:38 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/WikiEditor/ 'Deploy fix for bug 348885'
• 22:05 K4-713: updated prod civi to r1625
• 21:51 logmsgbot_: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.i18n.php 'changes for zero needed for carrier testing'
• 21:42 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'use $wmgUseMathJax'
• 21:41 logmsgbot_: aaron synchronized wmf-config/InitialiseSettings.php 'use $wmgUseMathJax'
• 21:38 K4-713: queue consumer re-enabled
• 21:35 K4-713: updated prod civi to r1623
• 21:32 logmsgbot_: aaron synchronized wmf-config/InitialiseSettings.php
• 21:29 logmsgbot_: awjrichards synchronized php-1.20wmf1/extensions/MobileFrontend/templates/ApplicationTemplate.php 'ec7c5cc'
• 21:28 logmsgbot_: awjrichards synchronized php/extensions/MobileFrontend/templates/ApplicationTemplate.php 'r114947'
• 21:24 logmsgbot_: aaron synchronized wmf-config/InitialiseSettings.php 'Enabled $wgUseMathJax on mediawikiwiki'
• 20:33 logmsgbot_: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/ArticleFeedbackv5.flagging.php
• 20:26 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/VisualEditor/ 'Deploy VisualEditor beta warning'
• 19:52 logmsgbot_: awjrichards synchronized wmf-config/CommonSettings.php 'Bump mobile resource version'
• 19:52 logmsgbot_: awjrichards synchronized php-1.20wmf1/extensions/MobileFrontend/
• 19:51 logmsgbot_: awjrichards synchronized php/extensions/MobileFrontend/
• 19:50 logmsgbot_: awjrichards synchronized wmf-config/CommonSettings.php 'bumping mobile frontend resource version'
• 19:01 logmsgbot_: awjrichards synchronized wmf-config/CommonSettings.php
• 18:55 logmsgbot_: reedy synchronized php-1.19/includes/api/ApiQueryBlocks.php 'r114941'
• 18:53 logmsgbot_: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping MobileFrontend resource version'
• 18:47 binasher: returning sq68
• 18:36 binasher: pulling sq68 from pybal for a bit
• 18:29 RoanKattouw: Did a graceful restart of all job runners using dsh about 15 mins ago
• 18:29 RoanKattouw: Restarted morebots
• 07:44 apergos: morebots test
• 07:44 apergos: restarted varnish service manually a bit a go on sq67 and sq70, the cron job didn't seem to have gone off. restarted morebots too while I was at it
• 03:37 Jeff_Green: dist-upgrade arsenic
• 03:29 LeslieCarr: restarting varnish on arsenic again
• 03:12 maplebed: started a script to delete old objects on ms-be1 for swift truncated object cleaning
• 02:53 Jeff_Green: dist-upgrade on strontium
• 02:43 LeslieCarr: restarted varnish on arsenic
• 02:26 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Tue Apr 17 02:26:40 UTC 2012
• 02:17 logmsgbot_: LocalisationUpdate completed (1.19) at Tue Apr 17 02:17:24 UTC 2012
• 01:44 LeslieCarr: restarting varnish on niobium
• 00:52 LeslieCarr: reloading amslvs4
• 00:27 logmsgbot_: aaron synchronized php-1.20wmf1/includes/filerepo 'deployed 552ff0f482f3e65e9795fe304dd810e9ae1b03fb'

April 16

• 23:31 logmsgbot_: catrope synchronizing Wikimedia installation... : Now with a touch of the specific WikiEditor.i18n.php file
• 23:11 logmsgbot_: catrope synchronizing Wikimedia installation... : Hopefully fixing the WikiEditor messages this time, now with MessagesEn.php touch
• 23:07 logmsgbot_: catrope synchronizing Wikimedia installation... : Hopefully fixing the WikiEditor messages this time
• 22:58 logmsgbot_: awjrichards synchronizing Wikimedia installation... : Updating MobileFrontend to r114934
• 22:49 logmsgbot_: catrope synchronizing Wikimedia installation... : Need to run scap for this WikiEditor change, contains i18n changes
• 22:39 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/WikiEditor/ 'Deploy WikiEditor revert'
• 20:53 logmsgbot_: catrope synchronized php-1.20wmf1/extensions/WikiEditor/ 'Actually deploy the recent WikiEditor fixes'
• 18:58 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Commons Wiki to 1.20wmf1
• 18:47 logmsgbot_: reedy synchronized php-1.20wmf1/resources/mediawiki/mediawiki.js
• 18:46 logmsgbot_: reedy synchronized php-1.20wmf1/extensions/WikiEditor
• 18:37 mutante: manually added iptables nat rules on nfs2
• 18:13 notpeter: upgrade of udp2log on nfs1/2 complete. should be operating normally now.
• 17:41 mutante: LDAP on nfs2 warnings - opendj was _just_ started there when puppet was fixed with an unrelated issue
• 17:38 mutante: restarting opendj on nfs2 because it refused connections
• 17:08 logmsgbot_: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ 'zero and mobile changes'
• 16:07 notpeter: upgrading and restarting udp2log on nfs1/2
• 15:04 mutante: puppet fresh on nfs[12] after removing nonexistent misc::mediawiki-logger class
• 14:46 mark: Shutdown db24 for memory testing by Chris
• 13:27 mark: Sending European bits traffic back to pmtpa
• 12:24 mark: Sending European bits traffic back to esams
• 12:06 mark: Testing sess_leak_fix2 patch with a snapshot varnish build on cp3001
• 11:56 Reedy: Ran ddsh -cM -g mediawiki-installation -o -oSetupTimeout=30 -- "cd /usr/local/apache/common && sudo -u mwdeploy ln -s php php-1.18" to create symlink for php-1.18
• 11:51 Reedy: Killing php-1.18 again
• 11:48 mutante: sq34 - System halted! Error: Internal Storage Slot, powered down, -> RT
• 11:45 logmsgbot_: reedy synchronized php-1.18/ 'Symlink php-1.18 back to php (our current main running version) as lots of requests on bits are for 1.18 resources'
• 11:44 mutante: sq34 was broken and died when connecting to mgmt, powercycling
• 11:37 mutante: nfs1 - Could not find class misc::mediawiki-logger for nfs1
• 10:57 Krinkle: bits.wikimedia.org back up, mark fixed it.
• 10:33 Krinkle: bits.wikimedia.org serving Error 503 Service Unavailable on all load.php requests for mediawiki.org and nl.wikipedia.org, maybe more
• 09:45 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Set wgEnableJavaScriptTest to true for test2wiki'
• 02:26 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Mon Apr 16 02:26:58 UTC 2012
• 02:17 logmsgbot_: LocalisationUpdate completed (1.19) at Mon Apr 16 02:17:57 UTC 2012

April 15

• 17:35 logmsgbot_: reedy synchronized php-1.20wmf1/includes/api '/me whistles'
• 17:20 logmsgbot_: reedy synchronized php-1.20wmf1/includes/api
• 02:25 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Sun Apr 15 02:25:58 UTC 2012
• 02:17 logmsgbot_: LocalisationUpdate completed (1.19) at Sun Apr 15 02:17:19 UTC 2012

April 14

• 18:14 mark: Shifting european bits traffic back from esams to pmtpa, session leak is still there
• 17:08 mark: Shifting european bits traffic back from pmtpa to esams
• 15:31 mark: Reverted varnish to 3.0.2-2wm4 on cp3001; the race condition patch did not fix the problem
• 14:56 mark: Sending European bits traffic to pmtpa for testing
• 13:52 mark: Backported varnish bug #897 patch to varnish 3.0.2, testing a snapshot build on cp3001
• 11:37 mark: Raised session_max to 300000 (runtime) on cp3001/cp3002
• 05:58 K4-713: re-enabled the queue consumer on aluminium
• 02:26 logmsgbot_: LocalisationUpdate completed (1.20wmf1) at Sat Apr 14 02:26:55 UTC 2012
• 02:17 logmsgbot_: LocalisationUpdate completed (1.19) at Sat Apr 14 02:17:34 UTC 2012
• 02:16 K4-713: updated prod civi to r1616
• 01:36 K4-713: turned off queue consumption on prod civicrm
• 01:36 K4-713: updated production civicrm to r1614

April 13

• 20:53 mark: Rebooting cp3002
• 20:37 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php 'r114889'
• 17:54 Jeff_Green: created new repo operations/debs/wikimedia-search-qa to stay within package naming conventions
• 17:31 notpeter: upgrading udplog on locke to 1.8-2 and restarting, etc
• 17:27 Jeff_Green: created new operations/debs/search-qa repo for packaging search qa scripts
• 17:17 notpeter: restarting udp2log on emery
• 12:53 notpeter: restopping puppet on locke/emery
• 12:09 mark: Deploying varnish 3.0.2-2wm4 and enabling persistent storage on all even numbered eqiad upload varnish hosts
• 11:46 mark: Imported varnish 3.0.2-2wm4 into the Wikimedia APT repository
• 02:48 logmsgbot: tstarling synchronizing Wikimedia installation... :
• 02:39 logmsgbot: LocalisationUpdate completed (1.20wmf1) at Fri Apr 13 02:39:01 UTC 2012
• 02:20 logmsgbot: LocalisationUpdate completed (1.19) at Fri Apr 13 02:20:35 UTC 2012
• 01:40 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/templates/ApplicationTemplate.php 'fix robots file'
• 01:18 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/ 'zero and mobile changes'
• 01:06 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/MobileFrontend.body.php 'fix html formatter'
• 00:56 logmsgbot: tstarling synchronizing Wikimedia installation... :
• 00:08 Ryan_Lane: rebooting ssl1004

April 12

• 23:39 logmsgbot: tstarling synchronizing Wikimedia installation... :
• 23:08 logmsgbot: preilly synchronizing Wikimedia installation... : zero rated mobile access changes and mobile frontend updates
• 21:27 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34923 - namespace required for PORTAL'
• 19:46 notpeter: stopping puppet on locke and emery
• 18:41 logmsgbot: catrope synchronizing Wikimedia installation... : Deploying ArticleFeedbackv5 updates
• 18:22 Reedy: Ran namespaceDupes against bewiki
• 18:17 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34024 - Install ArticleFeedback on es.wikinews'
• 18:15 logmsgbot: reedy synchronized php-1.19/resources/startup.js 'touch'
• 18:14 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34024 - Install ArticleFeedback on es.wikinews'
• 18:11 Reedy: Created AFT tables on eswikinews
• 17:54 RoanKattouw: Running schema updates for ArticleFeedbackv5 on enwiki
• 17:46 RoanKattouw: Deploying ArticleFeedbackv5 updates to testwiki and rebuilding localization cache
• 16:54 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Allow bnwiki crats to grant/remove import'
• 16:49 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35258 - Allow bureaucrats to remove sysop rights on fr.wikipedia'
• 16:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Fix imports for wm2012'
• 16:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35917 - allow transwiki imports on wikimania2012'
• 16:36 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35666 - Renaming Namespace Wikisource:Author in gu.wikisource'
• 16:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35694 - Add enotif on page changes in watchlist (guwiki and source)'
• 16:27 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35818 - Change of Armenian Wikipedia namespace'
• 16:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35905 - Change namespaces configuration - pl.wikipedia'
• 16:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35261 - Add block permissions in rollback on Lusophone Wikipedia'
• 16:13 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35823 - Wikijunior and cookbook namespaces for the Vietnamese Wikibooks'
• 16:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35659 - Set logo for sl.wikiversity'
• 16:00 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35853 - Set a non-empty default value for wmgArticleFeedbackBlacklistCategories on WMF wikis'
• 15:58 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35878 - Enable e-mail notifications for watchlist (EnotifWatchlist) on tawiki'
• 15:52 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35852 - Add a category to $wgArticleFeedbackBlacklistCategories for Portuguese Wikipedia to remove AFT from disambiguation pages'
• 15:10 mutante: gallium - after files have been deleted/moved, puppet back to normal operation (and new clone directory in Apache)
• 13:23 mutante: killed puppets on gallium
• 12:33 mark: repooled ssl1002
• 12:27 mutante: powercycling frozen ssl1002
• 12:22 mark: Manually depooled down ssl1002 in pybal
• 02:24 logmsgbot: LocalisationUpdate completed (1.20wmf1) at Thu Apr 12 02:24:29 UTC 2012
• 02:15 logmsgbot: LocalisationUpdate completed (1.19) at Thu Apr 12 02:15:54 UTC 2012

April 11

• 22:37 maplebed: deployed more log filters to emery: gerrit/r4758
• 21:35 LeslieCarr: restarted nrpe on db10
• 21:33 LeslieCarr: db1004 puppet is fubar
• 21:33 LeslieCarr: restarted puppet on db30
• 21:33 LeslieCarr: restarted puppet on mw1110
• 19:41 notpeter: reimaging bellin and blondel
• 19:28 logmsgbot: reedy synchronized live-1.5/ 'fix resources symlinks'
• 19:23 logmsgbot: reedy synchronized live-1.5/ 'fix resources symlinks'
• 16:54 notpeter: enabling notifications for eqiad lucene vips
• 16:31 mark: Sending Canadian upload traffic to the eqiad varnish upload cluster
• 15:59 logmsgbot: py synchronized wmf-config/lucene.php 'pushing search pool 4 to eqiad. for realz this time!'
• 15:45 logmsgbot: py synchronized wmf-config/lucene.php 'pushing search pool 1 and prefix pool to eqiad. for realz this time!'
• 15:31 logmsgbot: py synchronized wmf-config/lucene.php 'pushing search pool 2 to eqiad. for realz this time!'
• 15:15 logmsgbot: py synchronized wmf-config/lucene.php 'pushing search pool 3 to eqiad. for realz this time!'
• 14:40 notpeter: restarting indexer on searchidx2
• 13:48 logmsgbot: reedy synchronized php-1.20wmf1/extensions/AbuseFilter/special/SpecialAbuseLog.php
• 13:35 mutante: applied patch-RT-2804.diff to bugzilla per RT:2804 re: XMLRPC content-type verification
• 12:07 mutante: moved another list: museum-l -> glam (http://lists.wikimedia.org/pipermail/glam/2012-April/000000.html)
• 11:58 mark: Setup cp1036 with the persistent storage backend
• 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf1) at Wed Apr 11 02:26:28 UTC 2012
• 02:17 logmsgbot: LocalisationUpdate completed (1.19) at Wed Apr 11 02:17:55 UTC 2012
• 00:11 LeslieCarr: nagios down

April 10

• 23:50 RoanKattouw: Removed srv187-189 from /etc/dsh/group/job-runners , their jobrunner class has been commented out in puppet since October
• 23:31 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'bug 35869 - Add strategywiki as an import source on testwiki'
• 22:53 RoanKattouw: Trying a graceful restart of the job runner on mw1 by sending SIGHUP to the jobs-loop.sh process
• 22:53 logmsgbot: catrope synchronized php-1.19/extensions/WikimediaMaintenance/jobs-loop.sh 'r114834'
• 22:24 logmsgbot: reedy synchronized php-1.20wmf1/extensions/CentralAuth/ 'g4102'
• 22:23 logmsgbot: reedy synchronized php-1.20wmf1/extensions/AntiSpoof/ 'g4103'
• 21:20 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Disabling mobile URL template for mediawiki.org (using "mediawikiwiki" this time)'
• 21:18 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Disabling mobile URL template for mediawiki.org'
• 21:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki to 1.20wmf1
• 21:04 logmsgbot: reedy synchronized php-1.20wmf1/extensions/MobileFrontend/javascripts 'minified JS'
• 20:55 logmsgbot: reedy synchronized docroot/ 'Fix symlinks'
• 20:45 logmsgbot: reedy synchronized docroot/
• 20:35 logmsgbot: reedy synchronized docroot/
• 20:31 logmsgbot: reedy synchronized live-1.5/
• 20:24 logmsgbot: reedy synchronized php-1.20wmf1/ 'Resyncing for apaches with no space'
• 20:23 logmsgbot: reedy synchronized live-1.5 'Fix symlinks'
• 20:18 Reedy: Deleting php-1.18 from all apaches due to lack of space
• 20:14 logmsgbot: reedy synchronized php-1.20wmf1/extensions/PrefSwitch/ 'PrefSwitch is needed by SimpleSurvey'
• 19:35 logmsgbot: reedy synchronizing Wikimedia installation... : Rebuilding localisation cache for test2/1.20wmf1
• 19:24 logmsgbot: reedy synchronized wmf-config/ExtensionMessages-1.20wmf1.php 'Sync ExtensionMessages'
• 19:23 logmsgbot: reedy synchronized php-1.20wmf1/extensions/ 'Would you like some extensions to go with that, sir?'
• 19:21 LeslieCarr: restarting gmond on db1004 after removing it's 5gig log
• 19:07 logmsgbot: reedy synchronized php-1.20wmf1/LocalSettings.php 'Push LocalSettings out'
• 19:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.20wmf1
• 19:00 logmsgbot: reedy synchronized php-1.20wmf1/ 'Pushing files for 1.20wmf1'
• 18:03 logmsgbot: aaron synchronized wmf-config/swift.php 'Catch e bogus empty file names from listings'
• 14:17 robh: search in eqiad is being reinstalled, no need to be alarmed (thats a pun!)
• 14:13 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wgLanguageConverterCacheType for git deployment later'
• 11:50 mutante: pxe boot / reinstall cp1029 - cp1036
• 11:24 mark: Imported varnish 3.0.2-2wm3 into the Wikimedia APT repository
• 09:30 apergos: restarted slaving on es1003, it will be a bit before it catches up. patience, young nagios
• 02:16 logmsgbot: LocalisationUpdate completed (1.19) at Tue Apr 10 02:16:58 UTC 2012
• 01:33 Tim: on sodium: enabling mod_auth on lists.wikimedia.org by running puppet

April 9

• 23:14 mutante: migrated foundation-l to wikimedia-l (users/passwords/archive urls/settings stay, old mail address & siteinfo redirect)
• 22:32 logmsgbot: asher synchronized wmf-config/db.php 'returning db12 as enwiki recentchange/watchlist db'
• 21:39 LeslieCarr: restarted mysql on es1004 and cleared out its disk space
• 17:49 LeslieCarr: moving es monitoring to nrpe and variables, may cause false pages if i did it wrong :)
• 17:36 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Bug 35426 - WebFonts on mr.wikisource.org'
• 14:54 RobH: i killed eqiad search nodes, woooo
• 02:17 logmsgbot: LocalisationUpdate completed (1.19) at Mon Apr 9 02:17:22 UTC 2012

April 8

• 08:45 Nemo_bis: Servers have been very slow, almost unresponsive, and network had a drop of ~0.3 Gb/s, at ~8.35-40.
• 02:16 logmsgbot: LocalisationUpdate completed (1.19) at Sun Apr 8 02:16:58 UTC 2012

April 7

• 17:55 logmsgbot: reedy synchronized wmf-config/codereview.php 'Remove deferred paths'
• 02:16 logmsgbot: LocalisationUpdate completed (1.19) at Sat Apr 7 02:16:54 UTC 2012

April 6

• 22:23 LeslieCarr: deploying new squid config to all squids
• 22:14 LeslieCarr: added neon into tiertwo of squid allowed hosts
• 22:13 LeslieCarr: deploying new squid config to amssq35
• 21:55 LeslieCarr: restarted puppet on spence
• 21:35 LeslieCarr: moved jenkins_1.458_all.deb to /srv/wikimedia/incoming/ on brewster
• 21:32 LeslieCarr: restarted squid on brewster
• 18:27 Ryan_Lane: updating OpenStackManager to r114758 on virt0
• 17:33 mark: Sending Japanese upload traffic to varnish in eqiad
• 17:15 mark: Power cycled down host lvs5
• 16:43 mutante: changed master and started slave on es1004
• 15:55 mutante: used gerrit create-project to create operations/debs/wikistats.git
• 14:13 mutante: manganese (gerrit) now sends SSL CA certificate on https, (curl -vvv says verify ok), should resolve RT:2777 and BZ:35709
• 11:51 mutante: es1004 - rsync was finished, deleted all binlogs from old host, mysqld_safe& , but did not "change master.." and "start slave" (see mail)
• 11:39 notpeter: restarting lsearchd on search3... again...
• 02:17 logmsgbot: LocalisationUpdate completed (1.19) at Fri Apr 6 02:17:37 UTC 2012
• 01:21 Ryan_Lane: updating OpenStackManager to r114757 on virt0
• 00:18 Ryan_Lane: updating OpenStackManager to r114754 on virt0

April 5

• 23:49 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Change guwikisource logo to point to the unscaled file instead'
• 21:46 notpeter: halting db15 for it to await decom
• 21:39 binasher: started enwiki.revision sha1 migration on db12
• 21:32 notpeter: restarting lsearchd on search18
• 21:22 logmsgbot: asher synchronized wmf-config/db.php 'pulling db12, moving enwiki watchlist,recentchange,etc to db53'
• 21:19 logmsgbot: asher synchronized wmf-config/db.php 'returning db53'
• 21:17 logmsgbot: py synchronized wmf-config/lucene.php 'pushing all search traffic back to pmtpa'
• 18:34 Ryan_Lane: updating OpenStackManager to r114746 on virt0
• 18:19 Ryan_Lane: updating OpenStackManager to r114744 on virt0
• 16:49 RobH: brewster puppet running again, cisco installs wont work again until i finish puppetizing the files later today
• 15:41 logmsgbot: py synchronized wmf-config/lucene.php 'pushing search pool4 to eqiad. this is the smaller wikis shard'
• 15:40 notpeter: pointing search pool4 to eqiad (this is the "smaller languages" shard)
• 15:14 Rob_H: puppet daemon being halted on brewster, i need to make local test changes to dhcp
• 14:52 logmsgbot: py synchronized wmf-config/lucene.php 'pushing search prefix pool live in eqiad'
• 14:51 notpeter: pushing search prefix pool live in eqiad
• 14:51 mutante: gallium - disabled incompatible GitTool plugin on jenkins and restarted it
• 14:34 mutante: importing jenkins_1.458_all.deb to wikipedia apt repo and upgrading it on gallium
• 14:08 apergos: started rsync in screen session as root on es1003 copying snapshot from es1001 to /a/
• 14:04 andrewbogott: created labs account for cneubauer
• 14:02 logmsgbot: py synchronized wmf-config/lucene.php 'pointing enwiki search and enwiki.prefix at eqiad'
• 14:00 notpeter: pointing enwiki and enwiki.prefix at eqiad search cluster
• 13:48 mutante: gallium - upgraded all pear packages
• 13:45 mutante: gallium - upgraded phpunit and php_codesniffer via pear (have been installed via pear before, distro outdated)
• 13:43 mutante: gallium - upgrading pear
• 13:33 mutante: installing package upgrades on gallium. apache,apt,postgres,php5-*,ruby,...various libs
• 13:24 logmsgbot: py synchronized wmf-config/lucene.php 'pointing de, fr, ja, es, ru, nl, pl, pt, zh, and sv search at eqiad'
• 13:21 notpeter: pointing de, fr, ja, es, ru, nl, pl, pt, zh, and sv search at eqiad
• 12:27 notpeter: search1 and search4 seem to be dead. restarting lsearchd
• 02:16 logmsgbot: LocalisationUpdate completed (1.19) at Thu Apr 5 02:16:52 UTC 2012
• 00:33 Ryan_Lane: updating OpenStackManager to r114730 on virt0
• 00:24 Ryan_Lane: updating OpenStackManager to r114729 on virt0
• 00:19 Ryan_Lane: updating OpenStackManager to r114728 on virt0
• 00:12 Ryan_Lane: updating OpenStackManager to r114726 on virt0
• 00:00 Ryan_Lane: updating OpenStackManager to r114724 on virt0

April 4

• 22:16 maplebed: deployed (3rd time's the charm!) udp-filter changes to emery for diederik
• 22:14 logmsgbot: py synchronized wmf-config/lucene.php 'pointing all search back to pmtpa'
• 22:13 notpeter: flipping all search back to pmtpa (until tomorrow...)
• 22:00 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedback 'r114717'
• 21:24 cmjohnson1: replacing power cable to psu1 (bottom) es1
• 21:22 cmjohnson1: replacing power cable to psu1 (top) es1
• 21:14 logmsgbot: py synchronized wmf-config/lucene.php 'pointing de, fr, and ja search at lvs pool in eqiad for live testing'
• 21:12 notpeter: moving de, fr, and ja search to eqiad
• 21:04 cmjohnson1: replacing power cable on labstore2 array psu2 (right side)
• 21:00 cmjohnson1: replacing power cable on labstore1 array psu1 (left side)
• 20:57 cmjohnson1: removing power from bottom power supply labstore 2
• 20:54 cmjohnson1: removing power from top power supply on labstore2
• 19:44 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable wmgArticleFeedbackv5AbuseFiltering on enwiki'
• 19:40 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Disable wmgArticleFeedbackv5AbuseFiltering on enwiki'
• 19:14 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php 'r114716'
• 19:12 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable wmgArticleFeedbackv5AbuseFiltering on enwiki'
• 19:04 RobH: dns update for zhen mgmt
• 18:54 logmsgbot: catrope synchronizing Wikimedia installation... : Deploying AFTv5 update
• 18:52 logmsgbot: py synchronized wmf-config/lucene.php 'pointing ru, nl, pl, pt, zh, and sv search at lvs pool in eqiad for live testing'
• 18:51 notpeter: moving ru, nl, pl, pt, zh, and sv search to eqiad
• 18:27 mutante: nuked /a contents on es1004, started rsync from es1001
• 18:16 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Add code for wmgArticleFeedbackv5AbuseFiltering'
• 18:16 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Add wmgArticleFeedbackv5AbuseFiltering, enabled on testwiki only'
• 17:55 RoanKattouw: Running AFTv5 schema changes on enwiki
• 17:47 RobH: i didnt crash the site, weeee
• 17:46 RobH: gracefully restarting apaches
• 17:46 RobH: pushing out redirects change to apaches for wikipedia.org/com.il redirect to he.wikipedia.org
• 17:41 binasher: started enwiki.revision sha1 migration on db53
• 17:38 logmsgbot: asher synchronized wmf-config/db.php 'returning db52, pulling db53'
• 17:32 RobH: update done, all nameservers still online
• 17:31 RobH: dns update for wikipedia.org/com.il being resolved
• 17:08 RoanKattouw: Applying AFTv5 schema change on testwik
• 15:30 logmsgbot: py synchronized wmf-config/lucene.php 'pointing eswiki search at lvs pool in eqiad for live testing'
• 15:28 notpeter: pointing eswiki search at eqiad
• 12:51 mutante: db1007 - add mysql startup via 'update-rc.d mysql defaults'
• 12:42 apergos: started mysqld on db1007 via /etc/init.d/mysql (this doesn't seem to point to a special fb build, and can't seem to find one on this host, what's up with that?)
• 12:31 apergos: rebooted bd1007, it was dead in the water (also no helpful messages on console, bah)
• 11:16 mutante: enabled Renameuser extension on wikitech, renamed tchay per RT request, disabled extension again (it was installed but disabled)
• 02:19 logmsgbot: LocalisationUpdate completed (1.19) at Wed Apr 4 02:19:03 UTC 2012
• 01:50 logmsgbot: aaron synchronized php-1.19/includes/filerepo/backend/FileOp.php 'deployed r114697'
• 01:39 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header of landing page only for mswiki'

April 3

• 23:17 LeslieCarr: updating bgp policies on cr1.sdtpa
• 22:44 LeslieCarr: reinstalling neon
• 22:04 maplebed: rolled back changes to emery in udp-filter due to the new binary crashing.
• 21:50 maplebed: ran /etc/init.d/udp2log reload on emery to enact the puppetted changes
• 21:41 maplebed: deploying new udp-filter and teahouse filters to emery for diederik
• 20:13 notpeter: restarting lsearchd on search7. was taosted
• 18:37 logmsgbot: root synchronized wmf-config/mc.php
• 18:37 RobH: syncing new mc.php, forgot to check for all three of the servers i took down, opps.
• 18:28 RobH: shutting down mw28, mw49, & mw58 for rack relocation due to power overload in d2-pmtpa, relocation to d1-sdtpa per rt 2692
• 17:59 K4-713: Synchronized payments cluster to r114642
• 17:52 logmsgbot: reedy synchronized php-1.19/extensions/MobileFrontend/
• 17:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Fix commas in guwikisource namespaces'
• 17:36 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Fix commas in guwikisource namespaces'
• 16:38 RobH: bringing down srv237 for phase balancing
• 16:37 RobH: srv230 back in rotation
• 16:26 RobH: shutting down srv230 for power phase move per rt 2759
• 16:10 RobH: updating brewster to use new dhcp files for cisco, no more local hackin.
• 15:53 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35482 - Add Patroller & Autopatroller groups on ml.wikisource'
• 15:51 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35482 - Add Patroller & Autopatroller groups on ml.wikisource'
• 15:41 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35545 - Grant the abusefilter-log-detail right to patrollers on Commons'
• 15:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35545 - Grant the abusefilter-log-detail right to patrollers on Commons'
• 15:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35624 - Subject namespace for the Vietnamese Wikibooks'
• 15:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35603 - Enable Transwiki import on KN:WP'
• 15:16 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35581 - Closure of nz.wikimedia.org'
• 15:15 logmsgbot: reedy synchronized closed.dblist 'Bug 35581 - Closure of nz.wikimedia.org'
• 13:35 Tim: manually reloaded rsyslogd on all apaches
• 06:16 Tim: deploying limited/split apache syslog (https://gerrit.wikimedia.org/r/#change,4149)
• 02:16 logmsgbot: LocalisationUpdate completed (1.19) at Tue Apr 3 02:16:32 UTC 2012
• 00:37 logmsgbot: aaron synchronized php-1.19/includes/Block.php 'deployed r114672'

April 2

• 23:54 Tim: restarting all apaches with apache-restart-all-hard
• 23:51 logmsgbot: tstarling synchronized php-1.19/extensions/ConfirmEdit/FancyCaptcha.class.php
• 23:37 logmsgbot: tstarling synchronizing Wikimedia installation... :
• 23:36 maplebed: cleared the varnish cache for preilly
• 23:34 Tim: on all apaches: running logrotate -f and deleting the resulting backup syslog files, to free up disk space
• 23:32 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/templates/ApplicationTemplate.php 'r114673'
• 23:21 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping MobileFrontend resource version number'
• 23:05 logmsgbot: awjrichards synchronizing Wikimedia installation... : Deploying MobileFrontend changes at r114671 per http://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments#2_April.2C_2012
• 21:43 maplebed: reverted changes to emery's logging due to a broken package in the deploy.
• 21:30 LeslieCarr: turned down ms7's secondary ethernet port to prevent the flapping (stupid sun boxes)
• 19:51 maplebed: deploying new udp-filter to emery rt-2501 gerrit/r4120
• 19:51 notpeter: running authdns-update on dobson
• 18:30 RobH: brewster puppet daemon stopped, doing local hacks
• 18:17 RobH: removed old bin files on db1004 and prolly borked it by removing the wrong files
• 17:54 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php '35436 - Enable Narayam at Hindi Wikipedia'
• 17:47 logmsgbot: preilly synchronized wmf-config/InitialiseSettings.php 'add disable images option for default on zero domain'
• 17:45 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Bug 35328 - Enable WebFonts for fr.wikisource.org'
• 17:40 logmsgbot: nikerabbit synchronized php-1.19/languages/Names.php 'I18ndeploy r114656'
• 17:15 preilly: carrier testing push for DIGI
• 17:15 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header of landing page only for mswiki'
• 16:46 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header of landing page only for mswiki'
• 02:16 logmsgbot: LocalisationUpdate completed (1.19) at Mon Apr 2 02:16:47 UTC 2012

April 1

• 02:17 logmsgbot: LocalisationUpdate completed (1.19) at Sun Apr 1 02:17:22 UTC 2012

March 31

• 10:22 mutante: srv222,225 were also upgraded but stopping there for now in favor of reinstalls
• 09:58 mutante: nuked /usr/shared/doc on a couple srv's, hey at least 700MB or something, and yes we really should reinstall with a decent partitioning scheme as M ark said
• 02:18 logmsgbot: LocalisationUpdate completed (1.19) at Sat Mar 31 02:18:10 UTC 2012

March 30

• 19:37 hashar: configured jenkins on gallium to use smtp.pmtpa.wmnet as outgoing SMTP server
• 19:28 RobH: puppet daemon restarted on brewster
• 18:13 RobH: killing puppet daemon on brewster, i need to hack at local configuration for cisco server stuff
• 12:56 mutante: db1047 - added system startup for /etc/init.d/mysql
• 12:47 mutante: powercycling db1047
• 12:28 mutante: deleted old kernel sources on upgraded srvs for that little extra space during peaks, suggesting to nuke /usr/share/doc if there should be more disk space warnings
• 10:41 mutante: same for srv223
• 09:18 mutante: srv224,srv219,srv220, upgrade apache, dist-upgrading w/ kernel, disabling ureadahead, rebooting one by one
• 08:06 mutante: storage3 - gmond unable to find the metric information for any mysql_* .."module has not been loaded", starting mysql, running puppet ...
• 07:57 mutante: powercycling storage3
• 07:03 Tim: running bug 35578 cleanup script in screen on fenari
• 06:41 logmsgbot: tstarling synchronized php-1.19/includes/specials/SpecialBlock.php
• 06:40 logmsgbot: tstarling synchronized php-1.19/includes/specials/SpecialBlock.php
• 06:39 logmsgbot: tstarling synchronized php-1.19/includes/specials/SpecialBlock.php
• 06:15 Tim: killed vi on fenari owned by awjrichards, locking CommonSettings.php for two days
• 02:17 logmsgbot: LocalisationUpdate completed (1.19) at Fri Mar 30 02:17:56 UTC 2012
• 01:13 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Remove more crap'
• 01:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove some dupe code'
• 01:00 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove wmgUsabilityPrefSwitch'
• 00:59 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Remove wmgUsabilityPrefSwitch'
• 00:58 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove unused wmgUseUsabilityInitiativeAlpha'

March 29

• 23:49 logmsgbot: aaron synchronized php-1.19/includes/revisiondelete/RevisionDeleteUser.php 'deployed r114619'
• 21:20 LeslieCarr: rebooting db47
• 20:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Swap wgUseCommaCount to wgArticleCountMethod'
• 20:07 notpeter: restarting lsearchd on search2 to del the logfile to end all logfiles
• 20:05 RoanKattouw: Stopping and starting Gerrit on manganese to apply Chad's change of the -1 text in the DB
• 20:02 notpeter: restarting lsearchd on search7 to del the logfile to end all logfiles
• 18:11 logmsgbot: catrope synchronized php-1.19/extensions/ClickTracking/ClickTracking.hooks.php
• 17:59 RobH: search1021 coming back up, done with tests
• 17:53 RobH: search1021 coming down for ssd fit test
• 17:07 notpeter: disabling notifications for search lvs nagios checks for 24 hours to test fix
• 15:42 notpeter: finished clearning up all pmtpa search hosts. hey look! they all have lots of space now!
• 15:15 notpeter: restarting lsearchd on search3
• 15:02 RobH: brewster puppet re-enabled
• 15:02 RobH: virt1001 pxe boots via dhcp and fails tftp download, i have to hold off on further troubleshooting until i have a network admin
• 14:47 RobH: did virt1001 wrong, reupdating dns
• 14:39 RobH: all nameservers still online after udpate
• 14:37 RobH: updating dns for virt1001 testing
• 14:29 RobH: stopping puppet runs on brewster so my hacking at the dhcpd.conf file won't get overwritten until I have it working right
• 14:01 Jeff_Green: restarted varnish on on cp3002 because it was thrashing futiley
• 13:45 notpeter: rebooting (mostly) down cp3001
• 13:17 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Add participation namespace to metawiki per request'
• 13:11 notpeter: trimming logs and such on search1-20
• 09:59 mutante: srv221, disabling ureadahead, installing package upgrades and new kernel, rebooting
• 09:40 mutante: kill and start lsearchd on search7
• 09:36 mutante: restarted defunct lsearchd on search6
• 09:10 mutante: gallium - added demon,hashar,reedy to group jenkins as it's a problem using puppet when users and groups already exist
• 06:25 mutante: powercycling sq40
• 06:21 mutante: installed more package upgrades on sodium
• 05:58 mutante: installed security upgrades on brewster, cadmium, capella (apache,mysql,ruby,apt..)
• 05:49 mutante: db42 - mysql did not autostart after boot, added using update-rc.d
• 05:42 mutante: db42 - reboot worked despite the grub warning about unreliable blocklists
• 05:37 mutante: rebooting db42 to finish upgrades
• 02:17 logmsgbot: LocalisationUpdate completed (1.19) at Thu Mar 29 02:17:53 UTC 2012

March 28

• 23:27 Tim: running apt-get upgrade on mw22,mw66,srv193,srv250,srv253,srv236
• 23:25 Tim: cleaned up stuck apt-get process on srv236
• 23:22 Tim: cleaned up stuck apt-get processes on mw22,mw66,srv193,srv250,srv253
• 21:44 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping mobile frontend resrouce version'
• 21:43 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/javascripts/application.min.js 'r114576'
• 21:43 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/javascripts/banner.min.js 'r114576'
• 21:43 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php 'r114576'
• 21:43 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/templates/ApplicationTemplate.php 'r114576'
• 21:42 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/javascripts/application.js 'r114576'
• 21:42 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/javascripts/banner.js 'r114576'
• 20:43 logmsgbot: LocalisationUpdate completed (1.19) at Wed Mar 28 20:43:20 UTC 2012
• 20:29 notpeter: restarted search1020. nothing conspicuous in logs
• 19:56 RoanKattouw: Running a patched version of l10nupdate that rebuilds the localization cache
• 18:49 logmsgbot: catrope synchronizing Wikimedia installation... : Bugfixes for ArticleFeedbackv5, ArticleFeedback and ClickTracking
• 16:47 cmjohnson1: msw1-d1-pmtpa replacement complete
• 16:34 cmjohnson1: replacing msw-d1-pmtpa per rt2639
• 15:36 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bump account creation limit per request on -tech'
• 15:34 Reedy: srv221 is full
• 15:34 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bump account creation limit per request on -tech'
• 14:39 RobH: restarted morebots in screen on wikitech, no longer as catrope, as roan has root on that box
• 14:36 RobH: got virt1001 to pxe, but dhcp doesnt know how to handle, need subnet details.
• 14:34 notpeter: lucene hosed on search9 and search15. restarting, then will look after cause
• 13:14 Jeff_Green: restarting puppet/puppetmaster on stafford to experiment with report settings
• 02:10 logmsgbot: LocalisationUpdate completed (1.19) at Wed Mar 28 02:10:34 UTC 2012

March 27

• 23:12 logmsgbot: tstarling synchronized php-1.19/cache/trusted-xff.cdb
• 20:19 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33789 - Enable botadmin usergroup on ml.wikipedia'
• 19:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Fix lezwiki namespace'
• 19:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove ruwiki arbcom talk from namespaceprotection'
• 19:00 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35138 - Create Gujarati Wikisource'
• 18:26 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35138 - Create Gujarati Wikisource'
• 18:22 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35138 - Create Gujarati Wikisource'
• 18:16 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34351 - Create Wikisource in Belarusian'
• 18:14 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34351 - Create Wikisource in Belarusian'
• 18:10 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34351 - Create Wikisource in Belarusian'
• 18:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35290 - Create Slovenian Wikiversity'
• 18:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35290 - Create Slovenian Wikiversity'
• 17:00 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Config for lezwiki'
• 16:58 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Config for lezwiki'
• 16:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Config for lezwiki'
• 16:48 logmsgbot: reedy ran sync-common-all
• 16:32 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'prep work for new wikis'
• 16:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34527 - Create a Arbcom namespace on Russian Wikipedia'
• 16:06 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34527 - Create a Arbcom namespace on Russian Wikipedia'
• 15:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35161 - Incubator configuration updates'
• 15:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32825 - Favicon for siwiki'
• 14:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35516 - Add Skin: namespace to MW.org'
• 08:15 apergos: test you silly morebot

• 07:59:56 hashar: archived old server admin logs since the old page was too long for my connection to download :-/
• 06:59:02 apergos: !log powercycled emery, it was unresponsive via the mgmt console and not pingable
• 02:17:52 logmsgbot: LocalisationUpdate completed (1.19) at Tue Mar 27 02:17:52 UTC 2012
• 00:56:51 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php 'r114507'
• 00:55:36 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/templates/LeaveFeedbackTemplate.php 'r114508'
• 00:42:50 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bmping resource version for MobileFrontend'
• 00:41:58 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/stylesheets/beta_common.css 'r114509'
• 00:37:30 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping MobileFrontend resource version #'
• 00:36:36 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/templates/MobileFrontendTemplate.php 'r114507'
• 00:36:09 logmsgbot: awjrichards[00:36:36] synchronized php/extensions/MobileFrontend/templates/LeaveFeedbackTemplate.php 'r114508'
• 00:35:50 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/stylesheets/beta_common.css 'r114508'
• 00:08:55 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php 'r114506'

March 26

• 23:18:17 logmsgbot: awjrichards synchronizing Wikimedia installation... : Syncing MobileFrontend to r114504 changes per http://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments#26_March.2C_2012
• 22:44:53 RobH: !log also rolling firmware to ps1-d[1|2|3]-pmtpa
• 22:28:10 RobH: !log pushing firmware updates to servertechs in sequence: ps1-[a2|a3|a4|a5|b2|b3|b4|b5|c1|c2|c3|d1|d2|d3]-sdtpa, disregard any errors from rebooting alerts
• 19:55:09 notpeter: !log stopping puppet on search6 and search15 for 24 hours to test new log rotation script
• 19:19:35 RobH: !log cp1019 memory replaced per rt 2651
• 19:07:14 apergos: rebooting ms1001 (new kernel)
• 17:53:34 RobH: cp1019 coming down for memory replacement per rt 2651
• 17:51:39 RobH: fluorine disk upgrade done, os install pending, details on rt 2350
• 17:43:48 logmsgbot: nikerabbit synchronized php-1.19/extensions/WebFonts/ 'i18ndeploy r114492'
• 17:36:51 RobH: fluorine coming down for new disks
• 17:14 notpeter: backingup plwiki.nspart1 index on search7, deleting working copy, and restarting lsearchd. (note: this will probably cause some downtime on some languages while the proc restarts...)
• 15:18 RobH: db59 has errors, but as it was a fusion io testbed server, it is more than likely tweaked for such, it is not in any rotation
• 14:54 RobH: db59 shutting down for io card removal per rt 2589
• 13:37 mutante: while on it, installing a whole bunch of package updates on db42
• 13:25 mutante: db42 was out of disk , caused by ~5G citations.csv in /tmp, gzipped the file
• 09:59 mutante: ..and on ms-be-3. running puppet on db59
• 09:43 mutante: another corrupted .yaml file on ssl2
• 09:33 mutante: brewster - delete puppet lock file, restart lighttpd, puppet ...
• 09:05 mutante: brewster was out of disk - deleted lighttpd access.log.1, gzipped access.log
• 08:24 mutante: on several mw* boxes puppet did not run because .yaml files on the puppetmaster became corrupted. need to delete the $hostname files in /var/lib/puppet/yaml/node on stafford and re-run. puppet bug similar to http://projects.puppetlabs.com/issues/7836
• 02:18 logmsgbot: LocalisationUpdate completed (1.19) at Mon Mar 26 02:18:03 UTC 2012

March 25

• 22:26 RobH: row b servertech firmware in eqiad all updated, should clear alarms as they come back online
• 22:18 RobH: firmware updates on servertechs in row b eqiad, disregard alarms
• 20:14 RobH: to fellow ops, you can disregard those observium errors, as I caused them
• 20:13 RobH: firmware updated on all power strips in row a eqiad.
• 16:22 RobH: ps1-a1-sdtpa firmware update complete
• 16:15 RobH: updating firmware on ps1-a1-sdtpa
• 16:14 RobH: ps1-b1-sdtpa firmware updated successfully
• 16:14 RobH: ps1-a1-eqiad firmware updated successfully
• 16:09 RobH: updating firmware on ps1-s1-eqiad and ps1-b1-sdtpa
• 16:07 RobH: updated firmware successfully on ps1-a8-eqiad, if it has observium alarms now then there are bigger issues.
• 02:17 logmsgbot: LocalisationUpdate completed (1.19) at Sun Mar 25 02:17:21 UTC 2012
• 00:59 LeslieCarr: admin down asw-a-eqiad xe-1/1/2 and cr2-eqiad xe-5/0/0 due to framing errors causing packet loss and lacp sporadic timeouts. source of the issue

March 24

• 19:46 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/MobileFrontend.body.php 'Following a performance regression reported on wikitech-l, added merciless profiling to ExtMobileFrontend::DOMParse()'
• 19:46 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/MobileFrontend.body.php 'Following a performance regression reported on wikitech-l, added merciless profiling to ExtMobileFrontend::DOMParse()'
• 17:35 mark: Migration from br1-knams to cr2-knams completed.
• 17:09 mark: Migrated second knams-esams dark fiber link from br1-knams to cr2-knams
• 16:36 mark: Corrected MTU setting on cr2-knams's AMS-IX interface
• 16:20 Reedy: Some european users reporting oruting issues
• 16:01 mark: Cleared OSPF session between csw1-esams and csw2-esams which magically made some internal routes reappear
• 15:40 mark: Brought up AMS-IX ipv4 BGP sessions
• 15:30 mark: Brought up AMS-IX ipv6 BGP sessions
• 15:25 mark: Moved AMS-IX connection to cr2-knams:xe-1/1/0
• 15:22 mark: Shutdown all AMS-IX BGP sessions
• 15:06 mark: Disabled BFD on OSPF3 between cr2-knams and csw1-esams
• 14:49 mark: Moved AS6908 and AS1257 PIs to cr2-knams
• 14:18 mark: Brought up AS13030 and AS1299 BGP sessions on cr2-knams
• 13:57 mark: Shutdown AS1299 BGP session on br1-knams
• 13:14 mark: Established full iBGP mesh with added router cr2-knams. cr2-knams now has full Internet connectivity.
• 12:48 mark: Moved fiber from br1-knams:e1/2 to cr2-knams:xe-0/0/0
• 12:44 mark: Disabled br1-knams:e1/2 (DF leg 1 to esams)
• 12:43 mark: Rack mounted and powered up cr2-knams
• 02:17 logmsgbot: LocalisationUpdate completed (1.19) at Sat Mar 24 02:17:02 UTC 2012

March 23

• 23:49 logmsgbot: reedy synchronized php-1.19/extensions/MoodBar/ 'r114466'
• 23:15 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34005 - Change uploader flag configuration on Russian Wikipedia'
• 23:13 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34005 - Change uploader flag configuration on Russian Wikipedia'
• 23:09 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wmgAutopromoteOnce'
• 23:08 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'wgAutopromoteOnce'
• 23:07 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'wgAutopromoteOnce'
• 23:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wmgAutopromoteOnce empty arrays'
• 22:24 RobH: scs-a1-eqiad back online
• 21:58 RobH: scs-a8-eqiad coming down for re-grounding
• 19:51 RobH: all power strips in eqiad are now properly grounded
• 18:12 maplebed: removed ms1 and most of ms2 from the production swift rings. no effect expected.
• 18:04 logmsgbot: asher synchronized wmf-config/db.php 'returning db32, pulling db52 for migration'
• 16:44 RobH: cp1019 in middle of firmware update, please dont touch
• 16:44 RobH: cp1017 memory error seems ot have cleared post firmware update, will keep an eye on it for the rest of the day
• 16:09 RobH: raid rebuilding on magnesium, however swift stuff is kind of black box mystery right now to me, need Ben to review magnesium later for that
• 15:53 RobH: magnesium coming back online
• 15:44 RobH: shutting down magnesium for disk swap
• 15:37 RobH: firmware updating on cp1017, no one touch it please
• 15:30 RobH: db1020 can go back into whatever rotation Asher wants it in
• 15:29 RobH: db20 memory error on raid controller resolved with firmware updarte
• 06:39 logmsgbot: tstarling synchronized php-1.19/includes/filerepo/file/LocalFile.php 'r114442'
• 02:18 logmsgbot: LocalisationUpdate completed (1.19) at Fri Mar 23 02:18:35 UTC 2012
• 01:55 mutante: deleting puppet report files older than 60hours on stafford to free disk space

March 22

• 23:30 logmsgbot: reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
• 23:18 RobH: db1020 firmware still updating, will check on it later tonight. offline until then
• 22:19 notpeter: all 3 dns servers are responding to digs after reload
• 22:10 notpeter: pushing a new zone file to add 2 more search-related vips for eqiad
• 20:52 notpeter: stopping puppet on brewster temporarily
• 20:25 notpeter: rebuilding search1015 and 1016 for disk shuffles
• 20:01 RobH: magnesium goign down and up again, troubleshooting the disks
• 19:47 apergos: rebooting ms1002, had stuck rsyncs, and kswapds at 100% cpu, weirdness like "ls /export/upload/wikipedia/am/0/00" hanging.
• 18:08 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header of landing page only for mswiki'
• 15:45 RobH: search 1015 and search1016 back up with added disks
• 15:08 RobH: shutting down search1015 & search1016 for hdd additions
• 14:45 RobH: db1020 still offline, requires firmware update on raid controller per rt 2621, will perform later today
• 14:33 logmsgbot: reedy synchronizing Wikimedia installation... :
• 02:17 logmsgbot: LocalisationUpdate completed (1.19) at Thu Mar 22 02:17:47 UTC 2012
• 01:14 K4-713: Re-enabled the donations queue consumer in Jenkins
• 00:28 binasher: started enwiki.revision alter on db32
• 00:26 binasher: disabled lvm snapshots and puppet on db32 for revision sha1 alter
• 00:24 logmsgbot: asher synchronized wmf-config/db.php 'pullin db32 for revision alter'

March 21

• 22:27 ^demon|away: wmf-deployed extensions now r/o in SVN
• 21:52 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php
• 21:27 Ryan_Lane: bringing up all instances on virt3
• 21:08 cmjohnson1: swapped 2 DIMMS in virt3 (b2 and b5)
• 21:01 Ryan_Lane: shutting down virt3 to replace dimms
• 20:47 ^demon: /trunk/phase3 is now r/o in SVN
• 20:21 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable prefswitch'
• 20:10 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Set $wgArticleFeedbackv5OversightEmails on enwiki'
• 18:59 maplebed: rebooted ms-be3 after it crashed.
• 18:51 binasher: brought db24 back up after hang, and reslaving, but leaving out of db.php. just replicating until a replacement s2 snapshot host is built
• 18:51 logmsgbot: catrope synchronizing Wikimedia installation... : Deploying ArticleFeedbackv5 update
• 18:46 logmsgbot: asher synchronized wmf-config/db.php 'returning db36'
• 18:35 logmsgbot: asher synchronized wmf-config/db.php 'pulling db24, failing hw'
• 18:03 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php
• 18:01 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Temporarily disable ShortUrl on testwiki because we think it might conflict with ArticleFeedbackv5'
• 17:59 K4-713: updated and synchronized payments cluster to r114382
• 17:47 logmsgbot: catrope synchronized php-1.19/resources/startup.js 'touch'
• 12:25 notpeter: disabling notifications for search-pool1
• 08:58 mutante: rebooting ms-be4
• 08:37 mutante: stopped/started lsearchd on search9
• 08:05 mutante: ms-be4 down but cant powercycle it yet..Unable to establish LAN session / ipmitool /ipmi_mgmt
• 07:58 mutante: restarted lsearchd on search3 and 9
• 05:23 logmsgbot: tstarling synchronized php-1.19/includes/parser/CoreParserFunctions.php
• 05:23 logmsgbot: tstarling synchronized php-1.19/includes/parser/Parser.php
• 05:23 logmsgbot: tstarling synchronized php-1.19/includes/parser/StripState.php
• 05:22 logmsgbot: tstarling synchronized php-1.19/tests/parser/parserTests.txt
• 03:51 mutante: added "lez" to langlist and running authdns-update, for lez.wikipedia per RT-2665
• 03:29 mutante: magnesium - shutting down, has existing RT-2669 to replace disk
• 03:18 mutante: magnesium - "..drive on port B of the Srial ATA controller is operating outsde of normal specifications.. Strike F1 key to continue"..
• 03:16 mutante: powercycling magnesium - down and just "init: tty4 main" on mgmt, frozen
• 03:10 mutante: running puppet on aluminium
• 02:18 logmsgbot: LocalisationUpdate completed (1.19) at Wed Mar 21 02:18:10 UTC 2012
• 01:06 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php 'r114342'
• 00:25 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/MobileFrontend.body.php 'fix beta logo'
• 00:03 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/MobileFrontend.body.php 'fix beta logo'

March 20

• 23:19 Ryan_Lane: fixing the zero redirect
• 22:46 logmsgbot: reedy synchronized wikipedia.dblist 'test'
• 22:36 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/api/ApiQueryExtracts.php 'r114319'
• 22:09 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumping resrouce version # for MobileFrontend'
• 21:54 logmsgbot: awjrichards synchronizing Wikimedia installation... : Pushing MobileFrontend changes per http://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments#20_March.2C_2012
• 21:46 binasher: stopped eqiad bits servers from udplogging to emery, packet loss is back to zero
• 20:59 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header of landing page only for mswiki'
• 20:17 binasher: killed enwiki.revision sha1 migrator (upgrade-1.19wmf1-2.php). after db36 completes, will run the rest by hand
• 19:52 Ryan_Lane: pushing change for zero.wikipedia.org to redirect to the english message
• 19:41 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header of landing page'
• 19:16 cmjohnson1: pulling disk 5 on virt1 for reseating
• 18:34 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header of landing page'
• 18:02 pgehres: flipped Template:CC-status on wmfwiki since credit cards are still disabled on payments.wikimedia.org
• 17:52 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35193 - Enable sub page feature in Telugu Wikisource'
• 17:49 notpeter: restarting lsearchd on search10
• 17:30 logmsgbot: aaron synchronized php-1.19/includes/filerepo/file/LocalFile.php 'deployed r114285'
• 17:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Revert that then'
• 17:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Test something for sewikimedia'
• 16:42 logmsgbot: reedy synchronized wmf-config/abusefilter.php 'Bug 35355 - Reset of permissions for Hindi Wikipedia'
• 16:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove hiwiki botadmin from whGRoupsRemoveFromSelf'
• 15:52 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35296 - Namespace names changing on Komi Wikipedia'
• 15:50 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35296 - Namespace names changing on Komi Wikipedia'
• 15:49 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35296 - Namespace names changing on Komi Wikipedia'
• 15:30 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35161 - Incubator configuration updates'
• 15:25 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31209 - Enable the WikiLove extension for incubator'
• 14:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove more group dupes'
• 14:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35355 - Reset of permissions for Hindi Wikipedia (hiwiki)'
• 14:14 logmsgbot: reedy synchronizing Wikimedia installation... : sscapping for r114268
• 14:08 logmsgbot: reedy synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.i18n.php 'r114268'
• 09:12 mutante: new URL pointing to Wikipedia Education Program - http://education.wikimedia.org
• 08:59 mutante: several srv's said they were unable to contact NTP server
• 08:57 mutante: apache-graceful-all to deploy changed redirects.conf
• 08:53 logmsgbot: tfinc synchronized wmf-deployment/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'Fixes file pages showing data charge warnings'
• 07:42 mutante: running authdns-update after adding education.wm for redirect RT:2634
• 06:21 logmsgbot: tstarling synchronized php-1.19/includes/User.php
• 05:35 logmsgbot: asher synchronized wmf-config/db.php 'pulling db36 durring db migration'
• 02:17 logmsgbot: LocalisationUpdate completed (1.19) at Tue Mar 20 02:17:55 UTC 2012
• 00:25 logmsgbot: awjrichards synchronizing Wikimedia installation... : Reverting MobileFrontend to r113973
• 00:15 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFrontend.body.php 'r114221'
• 00:07 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Enabling zero rated mobile access everywhere'
• 00:05 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Bumpging version number for MobileFrontend resources'

March 19

• 23:54 logmsgbot: awjrichards synchronizing Wikimedia installation... : Redoing accidentally aborted scap, Pushing changes to MobileFrontend per https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments#19_March.2C_2012
• 23:51 logmsgbot: awjrichards synchronizing Wikimedia installation... : Pushing changes to MobileFrontend per https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments#19_March.2C_2012
• 23:35 AaronSchulz: fixed a few files, on commons and other wikis, with empty oi_archive_name values even though the file was on NFS
• 23:20 Ryan_Lane: restarting all nginx servers
• 23:20 Ryan_Lane: added a new proxy to the ssl configuration to temporarily proxy access to wikimania videos being transcoded
• 21:38 binasher: creating "ops" db and related grants on prod db clusters 2-7 to prep rollout of ishmael / pt-digest beyond s1
• 21:17 binasher: started enwiki.revision sha1 alter on production side
• 20:57 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFormatter.php 'Removing debugging code from MobileFormatter'
• 20:54 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFormatter.php 'Addin more debugging code to MobileFormatter'
• 20:43 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFormatter.php 'Addin more debugging code to MobileFormatter'
• 20:36 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFormatter.php 'Addin more debugging code to MobileFormatter'
• 20:31 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/MobileFormatter.php 'Adding debugging code to MobileFormatter'
• 20:07 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedback/modules/jquery.articleFeedback/jquery.articleFeedback.js 'r114176'
• 19:41 Ryan_Lane: bringing virt3 instances back up
• 19:33 binasher: deploying new frontend squid conf to add support for mf_useformat cookie [rt 2645]
• 19:18 K4-713: CiviCRM 4.1.1 update script finished executing on prod.
• 19:12 Ryan_Lane: shutting down virt3 for memory reseating
• 19:09 K4-713: Started the CiviCRM 4.1.1 update script on prod.
• 19:08 mark: Rebuilding RAID arrays on brewster
• 18:58 K4-713: Put production civicrm / drupal instance in offline mode for upgrade
• 18:54 K4-713: Disabled all production CiviCRM Jenkins jobs, for CiviCRM upgrade.
• 18:54 cmjohnson1: brewster HDD replacement complete
• 18:42 mark: Shutting down brewster for HDD replacement
• 18:26 Jeff_Green: killed kill-slow-queries on db1008 for the duration of the civicrm upgrade
• 18:19 logmsgbot: nikerabbit synchronized php-1.19/includes/Linker.php 'i18ndeploy r114160'
• 18:19 logmsgbot: nikerabbit synchronized php-1.19/extensions/WebFonts/resources/ext.webfonts.fontlist.js 'i18ndeploy r114160'
• 18:14 mark: Running smartctl -t long /dev/sdb on brewster
• 12:58 logmsgbot: hashar synchronized php-1.19/includes/SiteStats.php 'Reenable SiteStatsInit::articles() for bug 35169. SiteStatsInit::doAllAndCommit() still disabled since it breaks the site'
• 10:28 logmsgbot: tstarling synchronized wmf-config/PoolCounterSettings.php 'increased max queue from 50 to 100 on reports that the limit was reached on the enwiki main page in normal operation'
• 09:11 mutante: nomcom and langcom wikis look kind of broken , redirecting to pages on incubator with "Error: This page is unprefixed! "
• 08:49 mutante: making (almost) all private wikis https-only per RT-2565, vi remnant.conf,sync,graceful...
• 07:30 mutante: running sync-apache after making a change to remnant.conf to make grants.wm https-only
• 05:09 Ryan_Lane: bringing up most instances on virt3, doing so by project priority
• 04:42 Ryan_Lane: bringing up all instances on virt4, waiting 30 seconds between instances
• 04:25 Ryan_Lane: bringing up all instances on virt2, waiting 30 seconds between instances
• 04:09 Ryan_Lane: bringing up all instances on virt1, waiting 30 seconds between instances
• 04:00 Ryan_Lane: attempting to bring some instances up
• 02:17 logmsgbot: LocalisationUpdate completed (1.19) at Mon Mar 19 02:17:17 UTC 2012
• 01:15 mutante: killed, updated, restarted wikibugs bot per request in RT:2656, should have fixed bugzilla:18831

March 18

• 23:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35308 - Install mw:Extension:DynamicPageList (Wikimedia) on Portuguese Wikipedia (ptwiki)'
• 19:20 Ryan_Lane: stopping all labs instances, manually recovering gluster volume
• 15:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35295 - Missing a in abusefilter-hide-log permission for oversighters'
• 10:49 Ryan_Lane: rebooting virt4 thanks to defunct libvirt process
• 03:43 Ryan_Lane: bringing all labs instances up
• 02:18 logmsgbot: LocalisationUpdate completed (1.19) at Sun Mar 18 02:18:51 UTC 2012
• 01:09 Ryan_Lane: rebooting all of the virt hosts, gluster is having major issues
• 00:43 Ryan_Lane: rebooting virt2
• 00:40 Ryan_Lane: restarting glusterfs on virt2
• 00:11 Ryan_Lane: rebooting virt3 libirt is non-responsive
• 00:00 Ryan_Lane: bringing up instances that were downed on virt3

March 17

• 23:50 Ryan_Lane: virt3 crashed, powercycling it
• 23:34 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Remove old comments'
• 23:24 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove old comments'
• 23:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Simplify config somewhat'
• 23:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Simplify config somewhat'
• 23:02 logmsgbot: catrope synchronizing Wikimedia installation... : Have to scap for that AFTv5 change to propagate i18n change
• 22:52 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/ 'r114087'
• 21:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35289 - Add wikisource logo to mobile wikisource gateway'
• 02:21 logmsgbot: LocalisationUpdate completed (1.19) at Sat Mar 17 02:21:03 UTC 2012
• 01:23 AaronSchulz: FindFilesMissingDBRows.php done, list under aaron/output/missingFileDBRows
• 00:11 AaronSchulz: Running FindFilesMissingDBRows.php on all wikis

March 16

• 21:21 binasher: running enwiki.revision sha1 schema migrations on eqiad side
• 20:12 logmsgbot: reedy synchronizing Wikimedia installation... : Rebuild moodbar messages
• 20:03 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Re-enable moodbar on enwiki'
• 19:53 logmsgbot: reedy synchronized php-1.19/extensions/MoodBar/ 'r114030'
• 19:15 Reedy: Ran namespaceDupes on stewardwiki
• 17:11 RobH: hdd in search1017/1018 replaced per rt 2583
• 16:54 RobH: search1017 and search1018 coming down for hdd swap
• 16:53 RobH: cp1017 back in service pool
• 16:43 RobH: cp1019 back in full service
• 16:22 logmsgbot: reedy synchronized php-1.19/extensions/CentralAuth/specials/SpecialCentralAuth.php 'r114021'
• 16:22 RobH: cp1017 memory error, coming down for troubleshooting.
• 16:18 RobH: cp1019 memory error cleared after reseating, notes on rt 2651
• 16:09 mark: Migrated all varnish3 packages to newer varnish packages from git
• 16:08 RobH: cp1019 coming down for memory error troubleshooting
• 15:58 RobH: cp1040 repaired per rt 2611
• 15:48 RobH: cp1040 down for memory replacement
• 15:09 logmsgbot: reedy synchronized stylize.php 'Test for hume'
• 15:04 logmsgbot: root synchronized ufg.sql 'test sync to see if hume is fixed'
• 14:55 logmsgbot: reedy synchronized php-1.19/extensions/WikimediaMaintenance/cleanupBug31576.php
• 14:04 apergos: restarted swift-container-auditor on ms-be3, it had died for some reason
• 08:07 mutante: i reverted that (star cert for wikitech), no worries i "shred"ded the files
• 07:51 mutante: replaced self-signed cert on wikitech with the star cert
• 04:19 mutante: on stafford, deleting spence's puppet report files to free some disk space (they are like the largest report files of all)
• 03:09 mutante: stafford - - /var/lib/puppet/reports is getting quite large (18G), and we got the first disk space warning, do we want to keep those?
• 02:45 mutante: killing nrpe on several hosts where it was running as the wrong user again (somehow through the use of dsh)
• 02:21 logmsgbot: LocalisationUpdate completed (1.19) at Fri Mar 16 02:21:35 UTC 2012
• 01:12 mutante: stopping nagios-wm temp. while changing nrpe config (will watch it manually until it's back)
• 00:54 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Add NS to stewardwiki at request of Philippe'
• 00:51 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Add NS to stewardwiki at request of Philippe'

March 15

• 23:17 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/ArticleFeedbackv5.hooks.php 'r113974'
• 23:12 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/templates/DisableTemplate.php 'r113973, fixes bug 35249'
• 23:10 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/modules/ext.articleFeedbackv5/ext.articleFeedbackv5.js 'r113972'
• 22:59 logmsgbot: catrope synchronized php-1.19/resources/startup.js 'touch'
• 22:59 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Bump AFTv4 event logging percentage from 25% to 100%'
• 22:57 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedback/modules/jquery.articleFeedback/jquery.articleFeedback.js
• 22:48 mutante: purging Lucene monitoring on indexer from db9, remove duplicate service definitions manually anyways (still tons left), run purge script, reload Nagios..
• 22:24 logmsgbot: catrope synchronized php-1.19/resources/startup.js 'touch'
• 22:23 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Raise AFTv4 event logging percentage from 5% to 25%'
• 22:21 mutante: getting rid of Swift HTTP checks on non production machines manually (come on spence _purge_ ;P)
• 22:07 logmsgbot: catrope synchronized php-1.19/resources/startup.js 'touch'
• 22:04 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Raise AFTv4 event logging percentage from 1% to 5%'
• 21:44 logmsgbot: catrope synchronized php-1.19/resources/startup.js 'touch'
• 21:28 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/ArticleFeedbackv5.hooks.php 'r113961'
• 21:25 pgehres: K4-713 synchronized payments cluster to r113956
• 21:25 pgehres: disabled credit cards on donate.wikimedia.org
• 21:21 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/ArticleFeedbackv5.hooks.php 'fix fatal'
• 21:20 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Bump AFTv4 event logging percentage from 0.27% to 1%'
• 21:19 Ryan_Lane: rebalancing instances gluster volume
• 21:18 RoanKattouw: That was r113959
• 21:18 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.js
• 21:11 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.js 'r113958'
• 21:09 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/ 'r113957'
• 20:46 mark: bits.pmtpa cluster back online
• 20:44 RobH: dns update for silver and zhen servers
• 20:37 logmsgbot: reedy synchronized php-1.19/extensions/WikimediaMaintenance/cleanupBug31576.php
• 19:54 RobH: sq67-sq70 have been reinstalled, but not signed in puppet, not sure if they are ready for that or if there are other items mark needs to change first
• 19:11 RobH: working on sq67-sq70 reinstalls, disregard alerts
• 19:00 RobH: db1022 resetup and redeployed per rt 2537 and assigned back to asher
• 18:51 logmsgbot: reedy synchronizing Wikimedia installation... : Running scap to deal with message changes earlier
• 18:19 RobH: db1022 coming down for reinstall and resetup of raid per rt 2537
• 17:55 logmsgbot: reedy synchronized php-1.19/extensions/CentralAuth/ 'r113940'
• 17:54 logmsgbot: reedy synchronized php-1.19/extensions/CheckUser/ 'r113940'
• 17:53 logmsgbot: reedy synchronized php-1.19/extensions/wikihiero/modules/ext.wikihiero.css 'r113940'
• 17:52 logmsgbot: reedy synchronized php-1.19/extensions/NewUserMessage/NewUserMessage.class.php 'r113940'
• 17:41 logmsgbot: reedy synchronized php-1.19/includes/RecentChange.php 'r113938'
• 17:38 logmsgbot: reedy synchronized php-1.19/resources/mediawiki/mediawiki.util.js 'r113936'
• 17:37 logmsgbot: reedy synchronized php-1.19/includes/specials/SpecialUndelete.php 'r113936'
• 17:32 logmsgbot: reedy synchronized php-1.19/languages/messages/ 'r113935'
• 17:31 logmsgbot: reedy synchronized php-1.19/resources/ 'r113935'
• 17:31 logmsgbot: reedy synchronized php-1.19/includes/ 'r113935'
• 17:16 logmsgbot: reedy synchronized php-1.19/includes/SkinTemplate.php 'r113932'
• 16:13 logmsgbot: reedy synchronized php-1.19/extensions/WikimediaMaintenance/cleanupBug31576.php 'r113929'
• 15:15 mark: Created git repo operations/debs/varnish in gerrit
• 14:06 apergos: disabled moodbar temporarily on en wikii, see bug 35245
• 14:02 logmsgbot: ariel synchronized wmf-config/InitialiseSettings.php 'emergency disable of feedback dashboard (right config var this time?)'
• 13:51 logmsgbot: ariel synchronized wmf-config/InitialiseSettings.php 'emergency disable of feedback dashboard'
• 13:11 apergos: on screen as root on dataset1001, copying to gluster volume; if this causes problems feel free to shoot it. ( cp -a 20120211 /mnt/glusterpublicdata/public/enwiki/ )
• 09:08 mutante: ran puppet on mw1020
• 08:12 mutante: installing apache,apt,cron,mysql-client upgrades on spence
• 07:51 mutante: messed with /var/lib/dpkg/status on hume to fix broken packages/remove "marked for purging" on libmysql-php5 without removing a ton of other packages, rather hackish but seems fine anyways, like not broken anymore on simulated dist-upgrade etc
• 07:01 mutante: uprading apache and apt on hume
• 02:17 logmsgbot: LocalisationUpdate completed (1.19) at Thu Mar 15 02:17:35 UTC 2012
• 01:26 Ryan_Lane: labsconsole was missing libapache2-mod-php5. puppet must have tried to upgrade a package unsuccessfully
• 01:22 mutante: planet back up (installed libapache2-mod-php5 which installed apache2-mpm-prefork and removed apache2-mpm-worker)
• 01:19 mutante: planet down - apache on singer, syntax error in site config "Invalid command 'php_admin_flag'"
• 01:03 mutante: fixing nrpe "unable to read output" raid check on srv197,207,243,,244,253.. (nrpe running as wrong user)

March 14

• 23:16 maplebed: installed the swiftcleaner to run daily from iron. see root's crontab for more info.
• 20:41 binasher: disabled log_queries_not_using_indexes on all core dbs
• 20:33 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header for image support debugging'
• 20:30 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header for image support debugging'
• 19:29 maplebed: rebooting ms-be1 to enable hyperthreading (and make it the same as all the other ms-be hosts)
• 19:06 preilly: pushing x-images header for vary support
• 19:06 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header for image support debugging'
• 19:05 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/MobileFrontend.body.php 'zero needs to add x-images to vary header'
• 18:58 maplebed: ms-be5 is back in rotatino
• 18:31 preilly: push zero change for carrier testing
• 18:31 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing'
• 16:19 RobH: updating dns for new domain wikimediacommons.pt (nameservers not yet pointed at us)
• 16:04 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'add vcs for extdist updates'
• 13:03 RobH: cp1029-cp1035 all installed and ready for varnish deployment, puppet has been run
• 08:24 mutante: running "apt-get -f install" on snapshot3 to fix dpkg, which installed mysql-client- and client-core-5.1
• 08:02 mutante: stop/start memcached on srv254,srv255,srv257
• 07:51 mutante: restarting mecached on marmontel
• 07:51 mutante: fixing owa[1-3] Swift HTTP commands manually
• 03:44 mutante: ekrem - user agent "AppleDictionaryService" requests cause temp. WAP outage ..it seems
• 03:38 mutante: free some disk space on spence - deleted user.log.1 on spence, compressing messages.1, apt-get clean,...
• 02:52 RobH: cp1032-cp1035 reinstall issue wiped mbr causing issues, will reinstall in my AM
• 02:49 RobH: revoked, cp1032 is some reason in grub error, and its too late at night for me to work on it, will troubleshoot tomorrow
• 02:48 RobH: realized i forgot to log hours ago that cp1029-cp1036 are installed with puppet run, ready for varnish deployment tomorrow
• 02:17 logmsgbot: LocalisationUpdate completed (1.19) at Wed Mar 14 02:17:13 UTC 2012

March 13

• 23:51 mutante: upgrading bugzilla to 4.0.5
• 23:42 logmsgbot: reedy synchronized php-1.19/resources/jquery/jquery.textSelection.js 'r113786'
• 23:14 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing'
• 22:47 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/templates/ApplicationTemplate.php 'r113779'
• 22:44 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/stylesheets/beta_common.css 'r113774'
• 22:44 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/stylesheets/common.css 'r113774'
• 22:43 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/templates/ApplicationTemplate.php 'r113771'
• 22:42 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/api/ApiQueryExcerpts.php 'r113774'
• 22:27 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Removing moile URL template for tewtwiki'
• 21:44 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero'
• 21:44 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.i18n.php 'changes for zero'
• 21:31 logmsgbot: asher synchronized wmf-config/db.php 'replacing db18 with new s7 slave db56'
• 21:19 binasher: started slaving db56 from db37
• 20:30 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.i18n.php 'changes for zero'
• 19:27 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing'
• 19:17 RobH: iron updated to use ipmi_mgmt script
• 19:08 preilly: pushing changes for zero to mswiki
• 19:08 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.i18n.php 'changes for zero'
• 19:08 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero'
• 19:05 binasher: streaming hotbackup of db1041 to db56 (new s7 slave replacing db18)
• 18:10 maplebed: failover successful, restarted pybal on lvs4, failback successful.
• 18:09 binasher: power cycling db1020, which also froze this morning
• 18:08 maplebed: stopping pybal on lvs4 - should fail over to lvs3
• 17:47 maplebed: pybal restarted on lvs3
• 17:47 binasher: power cycling db1040, crashed again
• 17:30 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bug 35183 - p include extensions/Renameuser/Renameuser.php instead of extensions/Renameuser/SpecialRenameuser.php'
• 17:12 mark: Sending all normally-pmtpa upload traffic to upload-lb.eqiad
• 17:05 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero'
• 16:59 preilly: add disable images support to mswiki under zero domain
• 16:59 logmsgbot: preilly synchronized wmf-config/CommonSettings.php 'add disable images option for mswiki on zero domain'
• 16:58 logmsgbot: preilly synchronized wmf-config/InitialiseSettings.php 'add disable images option for mswiki on zero domain'
• 16:46 logmsgbot: preilly synchronized wmf-config/InitialiseSettings.php 'add ZeroRatedMobileAccess extension to mswiki remove from mywiki'
• 16:44 mark: Sending traffic from Japan, India, Mexico to upload-lb.eqiad
• 16:37 LeslieCarr: reinstalling neon
• 16:23 apergos: stole some free space from the phys volume on ms1002 to give us more time for the rsync to keep going til after the move to swift etc
• 15:28 mark: Sending traffic from the USA to upload-lb.eqiad
• 15:27 mark: Rebooting lvs1005 with upgraded kernel/packages
• 15:12 LeslieCarr: manually deleted cp1025 info from nagios config file - nagios restored for now
• 14:51 mark: Sending traffic from Canada to upload-lb.eqiad
• 14:32 mark: Sending traffic from Brazil to upload-lb.eqiad
• 13:58 mark: Sending traffic from Argentina to upload-lb.eqiad
• 12:58 mark: Seeding the eqiad upload caches from live upload requests
• 11:59 mark: Setup squid logging to oxygen, with oxygen relaying to multicast 233.58.59.1
• 11:02 mark: Rebooting lvs1002 with kernel updates
• 10:17 mark: Rebooting manutius with newer 2.6.36 kernel to attempt avoiding i/o kernel bug with torrus
• 02:18 logmsgbot: LocalisationUpdate completed (1.19) at Tue Mar 13 02:18:03 UTC 2012

March 12

• 22:55 K4-713: synchronized payments cluster to r113679, and tweaked the anti-fraud rules
• 21:51 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/modules/jquery.articleFeedbackv5/jquery.articleFeedbackv5.js 'r113671'
• 21:51 logmsgbot: catrope synchronized php-1.19/extensions/ArticleFeedbackv5/ArticleFeedbackv5.hooks.php 'r113671'
• 21:44 Reedy: Running foreachwiki extensions/WikimediaMaintenance/cleanupBug31576.php in screen as me on hume
• 21:39 RobH: search1014 repaired per rt 2483
• 20:26 RobH: cp1040 coming down for hardware stuffs
• 18:19 Nikerabbit: Assuming scap has finished
• 17:48 logmsgbot: nikerabbit synchronizing Wikimedia installation... : Deploying updated Translate
• 17:46 notpeter: restarting indexer on searchidx2
• 17:24 logmsgbot: nikerabbit synchronized php-1.19/includes/Title.php 'r113635'
• 17:22 logmsgbot: nikerabbit synchronized php-1.19/languages/ 'r113635'
• 17:14 logmsgbot: nikerabbit synchronized php-1.19/extensions/Narayam/ 'Updating Narayam'
• 17:13 mark: PXE booting cp1025-cp1028
• 17:11 logmsgbot: nikerabbit synchronized php-1.19/extensions/WebFonts/ 'Updating WebFonts'
• 15:16 mark: Rebooted manutius, stuck in a similar state as streber always did
• 06:10 mutante: turning off debug mode in nagios-nrpe, again had to kill it , restart fails
• 05:53 mutante: dunno, copper was stuck (no mgmt output after reboot) but powercycling it and back
• 05:43 mutante: rebooting copper to make sure grub update didnt break it and asked for restart anyways
• 05:37 mutante: copper - installing (security) updates (apt,grub,openssl,ruby,libc6..)
• 04:19 mutante: wanted to restart nagios-nrpe-server on spence with debug=1 to investigate permission issue. arr! "Address already in use" "cant write to pidfile", killed the one started on Feb18, and reordered allowed_hosts, spence talks to itself again now :p
• 03:40 mutante: same (and nscd) on fenari
• 03:35 mutante: upgrading libc6 and related packages on spence
• 02:17 logmsgbot: LocalisationUpdate completed (1.19) at Mon Mar 12 02:17:28 UTC 2012

March 11

• 08:14 apergos: restarted lighttp on dataset2
• 07:49 apergos: removed current htcp log file, restarted purger, it seems to be logging normallynow
• 07:35 apergos: current ls shows 17416851456 2012-03-11 07:34 HTCPpurger.log while current du -sh shows 175M for /var/log. Sparse file that gets rotated badly? lots of leading nulls (many gb worth), why?
• 07:33 apergos: on ms1004 the HTCPpurger.log file after rotation was 17 gb, filling the disk. Removed it.
• 02:17 logmsgbot: LocalisationUpdate completed (1.19) at Sun Mar 11 02:17:35 UTC 2012

March 10

• 22:09 Reedy: Make that wikimania2012, not wikimediawiki
• 22:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable anon page creation for wikimediawiki'
• 19:28 binasher: set sync_binlog = 1 on all current masters and eqiad dbs
• 19:22 binasher: reslaved db1033
• 07:03 mutante: ran puppet on db1022, another one that works fine manually but somehow did not by itself
• 05:11 mutante: doing more (cp*, db*, msbe-* ,mw*) by hand / for loop
• 05:01 mutante: starting nagios-nrpe-server on all via dsh (fail to restart on config change issue)
• 02:16 logmsgbot: LocalisationUpdate completed (1.19) at Sat Mar 10 02:16:57 UTC 2012
• 01:07 maplebed: started swiftcleaner on owa1 looking for (and purging) bad objects
• 01:06 maplebed: rebalanced the swift rings to finish decreasing traffic sent to ms1 and ms2
• 00:18 Ryan_Lane: powercycling ssl1003
• 00:18 Ryan_Lane: powercycling ssl1001

March 9

• 20:34 notpeter: stopping search indexer on searchidx2 for fresh rsync to searchidx1001
• 19:58 preilly: pushed change to remove description from landing page
• 19:57 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero'
• 18:59 Ryan_Lane: sending test.m.wikipedia.org to the same place as test.wikipedia.org via squid
• 18:58 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Fixing wgMobileUrlTemplate settings for domains that do not have .m. domains configured'
• 18:48 logmsgbot: reedy synchronized php-1.19/extensions/WikiLove/modules/ext.wikiLove/ext.wikiLove.css 'r113497'
• 18:40 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Changing the way in which wgMobileUrlTemplate is configurable by InitialiseSettings.php'
• 18:39 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Disabling wgMobileUrlTemplate for testwiki - hopefully for real this time'
• 18:34 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Making wgMobileUrlTemplate configurable by InitialiseSettings.php'
• 18:34 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Disabling wgMobileUrlTemplate for testwiki'
• 17:40 logmsgbot: awjrichards synchronized php/extensions/MobileFrontend/templates/ApplicationTemplate.php 'r113489'
• 17:32 maplebed: set swift storage device weight on ms2 to 0 and pushed out rings
• 15:52 apergos: cleared up a little bit of space on root partition of snapshot2, but that's about it. I hope we never have 3 versions of mw in test at the same time, the tmp caches will kill us
• 15:52 mark: Turned off vcc_err_unref on all varnish servers, so varnish doesn't complain when ACLs/probes/backends are unused
• 15:44 Jeff_Green: hume apt upgrades, puppetd --test, switch to mysql 5.1.53-fb3753-wm1
• 06:38 Ryan_Lane: reloading autofs on all labs instances
• 06:13 Tim: running svn cleanup on extdist trunk
• 04:18 Tim: switched php and wmf-deployment symlinks over to php-1.19 instead of php-1.18
• 04:18 Tim: restarted morebots
• 00:57 pp-pdf2: updated pyfribidi to 0.11.0 fixing https://github.com/pediapress/pyfribidi/issues/2
• 00:57 pp-pdf3: updated pyfribidi to 0.11.0 fixing https://github.com/pediapress/pyfribidi/issues/2
• 00:57 pp-pdf1: updated pyfribidi to 0.11.0 fixing https://github.com/pediapress/pyfribidi/issues/2
• 00:38 logmsgbot: preilly synchronized wmf-config/InitialiseSettings.php 'add ZeroRatedMobileAccess extension to mywiki'
• 00:32 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/javascripts/beta_opensearch.js 'fixes to code push'
• 00:32 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/javascripts/beta_opensearch.min.js 'fixes to code push'
• 00:27 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/templates/ApplicationTemplate.php 'fixes to code push'
• 00:27 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/javascripts/opensearch.js 'fixes to code push'
• 00:27 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/javascripts/opensearch.min.js 'fixes to code push'
• 00:01 RobH: oxygen install done, booting successfully after multiple tests, now running puppet for initial config
• 00:01 K4-713: updated the paypal IPN listener on aluminium to r1450

March 8

• 23:57 logmsgbot: awjrichards synchronized php-1.19/extensions/MobileFrontend/templates/ApplicationTemplate.php 'r113428'
• 23:56 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'per-wiki memory limit configuration, with extra memory for zh* for converter tables'
• 23:55 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'per-wiki memory limit configuration, with extra memory for zh* for converter tables'
• 23:42 mutante: rebooting ms-be5
• 23:37 logmsgbot: awjrichards synchronizing Wikimedia installation... : Updating MobileFrontend per https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments
• 23:24 binasher: streaming hotbacking of db1017 to db1033 - no snapshots of enwiki in eqiad til db1033 is back
• 23:19 Tim: started changing the php symlink to 1.19 instead of 1.18, but then changed my mind and changed it back.
• 23:16 logmsgbot: tstarling synchronizing Wikimedia installation... :
• 23:07 logmsgbot: tstarling synchronized php-1.19/extensions/ExtensionDistributor/svn-invoker.conf
• 23:01 logmsgbot: asher synchronized wmf-config/db.php 'returning db24 to service'
• 22:58 maplebed: powercycled ms-be3 - it crashed 2.5 hours ag.
• 22:52 logmsgbot: asher synchronized wmf-config/db.php 'pulling db18'
• 22:40 logmsgbot: aaron synchronized php-1.19/includes/filerepo/backend 'deployed r113413, r113414'
• 22:39 LeslieCarr: poked hole to allow labs machines to reach gluster machines in tampa
• 22:13 logmsgbot: catrope synchronized php-1.19/includes/MagicWord.php 'r113411'
• 22:13 logmsgbot: catrope synchronized php-1.19/includes/Cdb.php 'r113411'
• 22:13 logmsgbot: catrope synchronized php-1.19/includes/WebRequest.php 'r113411'
• 22:11 RobH: udpating dns for oxygen
• 22:03 RobH: oxygen coming down for reinstall
• 20:42 cmjohnson1: power to msw-c1-sdtpa restore
• 20:40 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero'
• 20:40 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.php 'changes for zero'
• 20:39 cmjohnson1: removing and relocating power to msw-c1-sdtpa
• 19:38 logmsgbot: catrope synchronizing Wikimedia installation... : ArticleFeedbackv5 updates
• 19:34 RoanKattouw: Running scap for ArticleFeedbackv5 updates
• 19:30 RoanKattouw: Running AFTv5 schema changes on enwiki
• 19:29 logmsgbot: catrope synchronized wmf-config/CommonSettings.php '$wgArticleFeedbackv5OversightEmails'
• 19:29 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php '$wgArticleFeedbackv5OversightEmails'
• 19:26 RoanKattouw: Applying AFTv5 schema changes to en_labswikimedia
• 19:09 preilly: push zero rated changes
• 19:09 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.i18n.php 'changes for zero'
• 19:09 logmsgbot: preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero'
• 19:04 RoanKattouw: Clearing message blobs
• 18:53 RoanKattouw: Running rebuildLocalisationCache.php
• 18:49 binasher: power cycling cp1044
• 18:46 binasher: purging entire mobile varnish cache - the main mobile template included robots no-follow
• 18:43 preilly: needed to fix a google issue with robots
• 18:43 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/ApplicationTemplate.php 'remove ROBOTS metatag'
• 18:40 logmsgbot: preilly synchronized php-1.19/extensions/MobileFrontend/ApplicationTemplate.php 'remove ROBOTS metatag'
• 18:40 binasher: deploying new squid frontend.conf to fix epic fail - all googlebot traffic was being redirected to mobile. now just if it's mobilegooglebot.
• 18:29 RoanKattouw: Applying AFTv5 schema changes on testwiki
• 18:27 RoanKattouw: Pushing new AFTv5 code to testwiki, do not sync to the live site just yet
• 17:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'ptwikipedia to ptwiki'
• 17:14 cmjohnson1: shutting down db18 for memory testing
• 16:57 RobH: search1014 still down per rt2483
• 16:47 maplebed: took ms-be5 out of rotation in the swift cluster - it's crashed 3 times now.
• 16:36 logmsgbot: reedy synchronized php-1.19/extensions/ExtensionDistributor/ExtensionDistributor_body.php 'r113368'
• 16:31 logmsgbot: reedy synchronized php-1.19/extensions/ExtensionDistributor/ExtensionDistributor_body.php 'Revert live hack because it works, will come in properly'
• 16:30 logmsgbot: reedy synchronized php-1.19/extensions/ExtensionDistributor/ExtensionDistributor_body.php 'Test for bug 27246'
• 16:16 RobH: search1008 repaired
• 15:52 RobH: mw1103 finally repaired and ready for os and such
• 14:48 pp-pdf1: installed python faulthandler 2.1
• 14:47 pp-pdf3: installed python faulthandler 2.1
• 14:47 pp-pdf2: installed python faulthandler 2.1
• 14:24 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 35012 - Namespace aliases for wikipedia and wikipedia-talk namespaces on Sanskrit wiki'
• 09:17 mutante: running puppet on mw1010 - finished quickly without problems - uh, wonder why Nagios reported puppet freshness then
• 08:22 mutante: cp1019 - Hitting F1 to continue reboot ( "Alert! System fatal error during previous boot")
• 08:21 mutante: cp1019 went down, then rebooted by itself (i think) after showing "idrac-8W82BP1 Severity: Non Recoverable, SEL:CPU Machine Chk: Processor sensor, transition to non-recoverable was asserted"
• 07:54 mutante: cadmium fixed by adding groups::wikidev
• 07:41 mutante: puppet on cadmium broken due to dependency Group[500] for User[catrope]
• 07:20 mutante: ms1004 ran out of disk - caused by 17G HTCPurger.log.1, trying to gzip it now
• 06:52 logmsgbot: tstarling synchronized multiversion/MWMultiVersion.php
• 06:51 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files:
• 03:04 Guest32353: powercycled ms-be5; it has been unresponsive for 2 hours.
• 02:18 logmsgbot: LocalisationUpdate completed (1.19) at Thu Mar 8 02:18:02 UTC 2012
• 01:32 AaronSchulz: fixBug34995.php done
• 01:26 AaronSchulz: running fixBug34995 on all wikis
• 00:17 Ryan_Lane: adding zero cnames
• 00:16 Ryan_Lane: installing newer wikimedia-task-dns-auth on all dns servers
• 00:15 Ryan_Lane: added wikimedia-task-dns-auth_0.18 to the repo, to add support for zero

March 7

• 23:05 logmsgbot: aaron synchronized php-1.19/includes/filerepo/file/LocalFile.php 'deployed r113319'
• 22:39 maplebed: set swift weight for ms1 to 0 initiating the process to move data off the host in preparation for decomissioning it.
• 21:17 Jeff_Green: running apt upgrades and puppetd --test on srv194, srv197, srv203, srv212, srv213, srv230, srv244, srv245, srv252, srv282 and manually restarting nrpe because they're reporting funky in nagios
• 20:20 logmsgbot: catrope synchronized php-1.19/resources/startup.js 'touch'
• 20:17 Jeff_Green: yet another redirects.conf change, per RT#2498 redirect wikimedia.com-->wikimedia.org
• 20:05 binasher: reverted no-pagecache rsync on search nodes - without corresponding index warmup in lsearchd, it just pushes back the pain a bit and does more harm than good
• 20:04 binasher: deployed support for zero.wikipedia.org and carrier tagging to mobile varnish servers
• 19:38 logmsgbot: catrope synchronized php-1.19/includes/resourceloader/ 'r113278'
• 19:27 Jeff_Green: manual apt-upgrade, puppetd --refresh, and repeat on srv265 because it was running on outdated apache config
• 18:44 RobH: correction sq39
• 18:36 RobH: pulled sq39 from text pybal config, pulled sq46 from upload pybal config
• 18:36 logmsgbot: catrope synchronized php-1.19/resources/startup.js 'touch'
• 18:36 logmsgbot: catrope synchronized php-1.19/extensions/CustomUserSignup/modules/AccountCreationUserBucket.js 'touch'
• 18:12 RobH: shutting down sq38 and sq46 per rt 2581 for testing
• 16:02 cmjohnson1: replacing hdd for disk 10 on db22
• 16:00 cmjohnson1: pulling disk 10 from db22
• 13:28 mark: Removed torrus from streber
• 13:00 pp-pdf2: updated mwlib to 0.13.6
• 13:00 pp-pdf3: updated mwlib to 0.13.6
• 13:00 pp-pdf1: updated mwlib to 0.13.6
• 11:29 logmsgbot: hashar synchronizing Wikimedia installation... : trigger a rebuild of l10n cache
• 04:53 mutante: added ms-be5 drives to swift cluster
• 02:18 logmsgbot: LocalisationUpdate completed (1.19) at Wed Mar 7 02:18:01 UTC 2012
• 02:11 logmsgbot: catrope synchronized php-1.19/includes/api/ApiBase.php 'r113212'
• 01:58 logmsgbot: aaron synchronized php-1.19/includes/filerepo/backend/FileBackend.php 'bumped max file size to 4GiB'
• 00:27 maplebed: put ms-be4 into rotation as a new production swift backend storage node
• 00:21 maplebed: put ms-be3 into rotation as a new production swift backend storage node
• 00:05 maplebed: put ms-be2 into rotation as a new production swift backend storage node

March 6

• 23:54 logmsgbot: catrope synchronized php-1.19/extensions/CustomUserSignup/ 'Belated sync of r113056'
• 23:52 binasher: deploying new frontend squid config to include googlebot in mobile redirects
• 23:36 logmsgbot: reedy synchronized php-1.19/extensions/SyntaxHighlight_GeSHi/SyntaxHighlight_GeSHi.class.php 'r113200 reverting r113198'
• 23:25 Tim: patched 5xx-filter.c live on locke and reloaded udp2log to stop the segfaults
• 23:20 logmsgbot: reedy synchronized php-1.19/extensions/SyntaxHighlight_GeSHi/SyntaxHighlight_GeSHi.class.php 'r113198'
• 21:46 logmsgbot: catrope synchronized php-1.19/extensions/CentralAuth/specials/SpecialCentralAuth.php 'r113183'
• 21:41 notpeter: restarting puppet on brewster
• 21:03 Jeff_Green: pushing another change to redirects.conf and doing a graceful apache restart
• 20:32 logmsgbot: reedy synchronizing Wikimedia installation... : Rebuild message cache stuffs for r113129
• 20:31 Jeff_Green: disabled Global Connect nagios test (check_gcsip) on payments cluster because GC is down and nagios is spammy
• 20:25 notpeter: reimaging search1001-1020 with new partman recipe :/
• 20:22 notpeter: temp stopping puppet on brewster
• 20:21 logmsgbot: reedy synchronized php-1.19/resources/mediawiki.action/mediawiki.action.edit.js 'r113175'
• 20:20 logmsgbot: reedy synchronized php-1.19/maintenance/populateRevisionSha1.php 'r113175'
• 20:19 logmsgbot: reedy synchronized php-1.19/includes/specials/SpecialContributions.php 'r113175'
• 20:18 logmsgbot: reedy synchronized php-1.19/includes/specials/SpecialUserlogin.php 'r113176'
• 20:00 pp-pdf1: installed log-wikimedia-operations (which can be used for automated logging to #wikimedia-operations)
• 19:53 Ryan_Lane: restarting labs mysql to allow for more connections
• 19:26 Ryan_Lane: installing nova-api on virt0
• 19:09 Ryan_Lane: upping FLAGS.sql_max_pool_size for nova-api
• 18:47 logmsgbot: catrope synchronized php-1.19/resources/startup.js 'touch'
• 18:46 Ryan_Lane: rebooting all instances
• 18:34 Ryan_Lane: restarting nova-network on virt2
• 18:19 Ryan_Lane: rebooting virt1
• 18:15 Ryan_Lane: rebooting virt2
• 18:11 Ryan_Lane: rebooting virt3
• 18:07 Ryan_Lane: rebooting virt4
• 17:57 Ryan_Lane: taking the opportunity to apply security updates to virt0-4
• 16:25 logmsgbot: catrope synchronized docroot/foundation/FrameResize.html 'Put Jobvite frame resize file in foundationwiki docroot per Erik'
• 11:40 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: switching sr* to 1.19
• 11:15 logmsgbot: hashar synchronized php-1.19/languages/messages/MessagesSa.php 'r1113039 for bug 34938 : title is sometime empty on Sanskrit wikis'
• 11:13 logmsgbot: tstarling synchronized php-1.19/includes/OutputPage.php 'r113128'
• 10:41 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: switching zh* from 1.18 to 1.19
• 08:36 mutante: on hooper: puppet broken due to dependency Package[libapache2-mod-php5] for Service[apache2]
• 03:33 mutante: rebooting bast1001 for kernel upgrade
• 03:32 mutante: upgrading apache2 packages, base-files, kernel, several libs on bast1001
• 03:27 mutante: installing a couple upgrades on fenari (apache2-utils, update-manager-core, cvs, ruby, libxml*, libopenssl-ruby*...)
• 02:37 logmsgbot: LocalisationUpdate completed (1.18) at Tue Mar 6 02:37:06 UTC 2012
• 02:36 logmsgbot: tstarling synchronizing Wikimedia installation... : updating to r113119
• 02:18 logmsgbot: LocalisationUpdate completed (1.19) at Tue Mar 6 02:18:13 UTC 2012
• 01:27 Jeff_Green: manually updated packages and restarted apache on srv198, srv229, srv262, srv268, mw40 because their apache redirect configs failed to update after sync-apache and restart
• 01:07 Jeff_Green: another adjustment to redirects.conf and apache-graceful-all for RT#2488

March 5

• 22:24 Jeff_Green: modified redirects.conf per RT #2488
• 21:21 Reedy: Ran foreachwiki cleanupUploadStash.php
• 20:36 maplebed: enabled swift for 100% of thumbnails in production
• 18:18 logmsgbot: nikerabbit synchronized php-1.19/extensions/WebFonts/ 'i18ndeploy r113058'
• 18:11 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'WebFonts: bugwiki bug 34550; sawikisource bug 34159; amwiktionary amwikiquote bug 34700'
• 18:01 mark: Raised MTU between cr1-sdtpa - (csw1-sdtpa) - cr2-pmtpa to 9192
• 17:35 Jeff_Green: removed 3GB db30:/tmp/gmond.log and force-restarted gmond b/c the init script failed to restart it
• 17:16 Jeff_Green: adjusted LVS partitions on hume, moved /usr/local/apache to a new 5GB mount
• 15:18 mark: Fixed DNS resolving on the core routers by allowing DNS replies in the loopback filter
• 14:44 logmsgbot: reedy synchronized php-1.19/includes/Title.php 'r113036'
• 14:43 logmsgbot: reedy synchronized php-1.19/includes/AjaxResponse.php 'r113036'
• 14:35 logmsgbot: reedy synchronized php-1.19/extensions/CentralAuth/ 'r113035'
• 14:34 logmsgbot: reedy synchronized php-1.19/extensions/CategoryTree/ 'r113035'
• 13:50 mark: Set increased OSPF/OSPFv3 metric 30 on both directions of the link cr1-eqiad:xe-5/2/1 <--> cr1-sdtpa:xe-0/0/1, to combat higher than normal jitter and packet loss on the link
• 12:53 mark: Upgraded observium to latest version
• 09:41 mutante: restarting memcached on marmontel
• 09:40 mutante: restarting squid backend on knsq25
• 06:52 Ryan_Lane: all of the instances are accessing the file descriptors of files inside of the _base directory, and fuse has an issue with this. gluster can't recreate the base directory because of the processes holding open the old one.
• 06:50 Ryan_Lane: I've corrupted the _base directory on the instance's glusterfs share. I'm recovering the files from file descriptors using lsof. Not totally sure how I'm going to get the _base directory back, yet.
• 02:33 logmsgbot: LocalisationUpdate completed (1.18) at Mon Mar 5 02:33:04 UTC 2012
• 02:16 logmsgbot: LocalisationUpdate completed (1.19) at Mon Mar 5 02:16:39 UTC 2012

March 4

• 21:48 logmsgbot: reedy synchronized wmf-config/ 'Bug 32726 - Set =true for Commons'
• 21:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'fix .'
• 21:41 logmsgbot: reedy synchronized wmf-config/ 'Bug 32726 - Set =true for Commons'
• 21:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34897 - Enable Special:Import on Catalan wikisource'
• 20:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34567 - New logo for Arabic Wiktionary'
• 20:31 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34715 - Please modify the import sources for the Spanish Wikiversity'
• 20:29 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34694 - Install the Quiz extension on de.wikibooks'
• 20:25 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'wmgMoodBarCutoffTime'
• 20:25 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Create wmgMoodBarCutoffTime'
• 20:14 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Variablise moodbarconfig infoUrl'
• 20:12 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Variablise moodbarconfig infoUrl'
• 20:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34618 - Install MoodBar on fr.wikisource'
• 20:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34766 - Logo of Sanskrit Wikisource'
• 19:53 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34867 - Switch Sango wiktionary logo'
• 19:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34931 - Add namespaces aliases on as.wikipedia.org'
• 19:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34690 - Changing the name in the title bar to Assamese'
• 02:35 logmsgbot: LocalisationUpdate completed (1.18) at Sun Mar 4 02:35:16 UTC 2012
• 02:17 logmsgbot: LocalisationUpdate completed (1.19) at Sun Mar 4 02:17:34 UTC 2012

March 3

• 18:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34728 - Categories added to user pages by Babel in pt.wiktionary'
• 13:04 logmsgbot: aaron synchronized php-1.19/includes/Revision.php 'deployed r112949'
• 02:35 logmsgbot: LocalisationUpdate completed (1.18) at Sat Mar 3 02:35:08 UTC 2012
• 02:18 logmsgbot: LocalisationUpdate completed (1.19) at Sat Mar 3 02:18:04 UTC 2012

March 2

• 21:41 logmsgbot: aaron synchronized php-1.19/extensions/CategoryTree/CategoryTreeFunctions.php 'disabled logging hack'
• 20:47 Jeff_Green: added redirect/301 from http://static.wikimedia.org --> http://dumps.wikimedia.org now that archival static html dumps are located there
• 19:53 mark: Decommissioned csw5-pmtpa from AS14907 service. rest in pieces ;)
• 19:10 mark: Did a hot cut to remove csw5-pmtpa out of the path of cr1-sdtpa -> csw1-sdtpa -> csw5-pmtpa -> cr2-pmtpa
• 17:46 cmjohnson1: powering down msw1-pmtpa for relcocation to d1-pmtpa
• 17:40 cmjohnson1: disconnecting management fiber from msw1-pmtpa
• 16:59 logmsgbot: reedy synchronized php-1.19/extensions/CentralAuth/CentralAuth.php 'r112904'
• 16:55 RobH: ms-be4 boot order fixed, fixing ms-be5 & ms-be2
• 16:49 RobH: fixed boot order on ms-be3, fixing ms-be4
• 16:33 RobH: poking at bios on ms-be3
• 16:05 RobH: wikitech outage resolved
• 15:20 RobH: shutdown frdev offsite vm per email to engineering last week
• 15:18 RobH: backing up wikitech in hopes of upgrading some of its software
• 08:36 apergos: on ms1004, low on space, HTCPpurger.log.1 had about 16 gb of nulls before any real content, I tailed off the real stuff and tossed the original. The current log file has the same problem, why?
• 02:34 logmsgbot: LocalisationUpdate completed (1.18) at Fri Mar 2 02:34:34 UTC 2012
• 02:17 logmsgbot: LocalisationUpdate completed (1.19) at Fri Mar 2 02:17:51 UTC 2012
• 01:36 logmsgbot: aaron synchronized php-1.19/includes/filerepo/backend/lockmanager/LockManager.php 'deployed r112867'
• 00:41 logmsgbot: aaron synchronized php-1.19/extensions/CategoryTree 'deployed r112862'

March 1

• 23:33 logmsgbot: aaron synchronized php-1.19/extensions/CategoryTree/CategoryTreeFunctions.php 'log agent'
• 23:29 logmsgbot: reedy synchronizing Wikimedia installation... : Push message updates from r112848
• 23:22 logmsgbot: aaron synchronized php-1.19/extensions/CategoryTree/CategoryTreeFunctions.php 'logging fix'
• 23:22 logmsgbot: aaron synchronized php-1.19/extensions/CategoryTree/CategoryTreeFunctions.php
• 23:20 logmsgbot: aaron synchronized php-1.19/extensions/CategoryTree/CategoryTreeFunctions.php
• 23:17 logmsgbot: reedy synchronized php-1.19/includes/filerepo/backend/FSFileBackend.php 'r112850'
• 23:16 logmsgbot: reedy synchronized php-1.19/includes/Article.php 'r112850'
• 23:11 logmsgbot: aaron synchronized php-1.19/extensions/CategoryTree/CategoryTreeFunctions.php
• 23:06 logmsgbot: reedy synchronized php-1.19/extensions/MoodBar/ApiFeedbackDashboardResponse.php 'r112848'
• 23:05 logmsgbot: reedy synchronized php-1.19/extensions/Collection/Collection.body.php 'r112848'
• 22:12 logmsgbot: aaron synchronized php-1.19/includes/specials/SpecialContributions.php 'deployed r112844'
• 22:06 logmsgbot: aaron synchronized php-1.19/includes/filerepo/backend 'deployed r112841'
• 21:04 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'enabled FileBackend debug log'
• 19:57 cmjohnson1: replaced disk 3 labstore1 chassis
• 19:54 cmjohnson1: removing disk 3 from labstore1 chassis
• 19:47 Ryan_Lane: restarted memcached on virt0
• 19:15 logmsgbot: reedy synchronized php-1.19/cache/interwiki.cdb 'Updating interwiki cache'
• 17:39 Jeff_Green: Removed >5GB /tmp/gmond.log on db25, db32, db33, db37
• 17:36 logmsgbot: hashar synchronized php-1.19/includes/EditPage.php 'r112819 - Bug 34849 diff during editing an old version compares to the old version instead of the current one'
• 17:36 Jeff_Green: Removed >5GB /tmp/gmond.log on db13
• 17:35 Jeff_Green: Removed >5GB /tmp/gmond.log on db11
• 17:25 Jeff_Green: Removed 5.3GB /tmp/gmond.log on db1018
• 17:24 Jeff_Green: Removed 5.3GB /tmp/gmond.log on db1017
• 17:13 Jeff_Green: Removed 4.8GB /tmp/gmond.log on db1008. Tried to resist urge to make snarky comment about ganglia but failed.
• 14:54 RobH: strontium server rebooting to set HT to enabled
• 14:26 mark: Moving bits traffic back from pmtpa to eqiad
• 14:24 mark: Cleared dnsmasq cache on virt2
• 14:16 mark: csw5-pmtpa: Mar 1 14:01:42:A:Power Supply 2 , 2nd from left, bad
• 14:14 mark: mr1-pmtpa rebooted/lost power for some reason
• 14:07 mark: pmtpa/sdtpa management network went down
• 13:54 mark: Pooled new eqiad bits servers strontium and palladium
• 12:45 logmsgbot: hashar synchronized php-1.19/includes/specials/SpecialWatchlist.php 'r111882 for Bug 34835 - watchlist shows times in UTC'
• 10:53 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: reverting sr* wikis back to 1.18 per Siebrand's recommendation due to bug 34832
• 06:26 logmsgbot: tstarling synchronized php-1.19/extensions/SpamBlacklist/SpamBlacklist.php 'r112781'
• 05:46 maplebed: started swift deletion run on owa1, 2, and 3.
• 02:33 logmsgbot: LocalisationUpdate completed (1.18) at Thu Mar 1 02:33:53 UTC 2012
• 02:16 logmsgbot: LocalisationUpdate completed (1.19) at Thu Mar 1 02:16:52 UTC 2012
• 02:15 Ryan_Lane: vlan tagged virt5's eth0 and eth1 ports on csw1-sdtpa
• 02:12 logmsgbot: aaron synchronized php-1.19/extensions/CategoryTree/CategoryTreeFunctions.php 'debug logging'
• 02:02 logmsgbot: reedy synchronized php-1.19/resources/mediawiki.action/mediawiki.action.history.diff.css 'r112750'
• 01:59 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: all zh wikis back to 1.18
• 01:50 logmsgbot: aaron synchronized php-1.19/extensions/WikiLove 'deployed r112758'
• 01:37 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Last 265 wikipedias over to 1.19wmf1
• 01:28 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: s7 to 1.19wmf1
• 01:23 logmsgbot: reedy synchronized php-1.19/extensions/CategoryTree/CategoryTreeFunctions.php 'r112754'
• 01:09 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: s2 to 1.19wmf1
• 00:58 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Meanwhile, on wikipedia.... Hello ruwiki!
• 00:48 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: jawiki to 1.19wmf1
• 00:31 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: frwiki to 1.19wmf1
• 00:21 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: dewiki to 1.19wmf1
• 00:05 logmsgbot: tstarling synchronized php-1.19/extensions/Collection/Collection.body.php 'r112745'

February 29

• 23:42 logmsgbot: reedy synchronized php-1.19/extensions/ArticleFeedbackv5/api/ApiArticleFeedbackv5Utils.php 'r112743'
• 23:38 logmsgbot: catrope synchronized php-1.19/extensions/LiquidThreads/lqt.css 'r112742'
• 23:35 maplebed: trying a run of swiftcleaner against the commons a2 shard on swift.
• 23:34 logmsgbot: reedy synchronized php-1.19/extensions/Collection/Collection.body.php 'r112741'
• 23:28 logmsgbot: reedy synchronized php-1.19/extensions/ArticleFeedbackv5/api/ApiViewRatingsArticleFeedbackv5.php 'r112737'
• 23:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki funtime
• 22:30 Ryan_Lane: restarting gerrit on manganese to enable replication
• 22:18 Ryan_Lane: stopped ircecho on formey and started it on manganese
• 22:18 Ryan_Lane: installed python-paramiko on manganese. needs to be puppetized
• 22:12 Ryan_Lane: reversing gerrit replication from formey -> manganese to manganese -> formey
• 22:12 Ryan_Lane: gerrit moved to manganese.
• 22:09 Ryan_Lane: replacing ssh_host_key on manganese for gerrit with the same one on formey
• 22:09 Ryan_Lane: stopping gerrit on manganese
• 22:02 Ryan_Lane: stopped gerrit service on formey, moving to manganese
• 21:07 logmsgbot: reedy synchronized php-1.19/extensions/OggHandler/ 'r112725'
• 21:00 logmsgbot: hashar synchronized php-1.19/extensions/ApiSandbox/ext.apiSandbox.js '(bug 34790) Pressing "Make Request" should not make two requests to api.php'
• 20:58 Ryan_Lane: restarting nova-compute on virt4
• 19:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Turn wmgReduceStartupExpiry on for wikipedia projects, off for nl/pl wiki ahead of tonights deploy'
• 19:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Turn wmgReduceStartupExpiry off by default. Needs to go to wikipedia only later on'
• 19:23 logmsgbot: aaron synchronized wmf-config/PrivateSettings.php 'updating swift auth'
• 19:08 RobH: dns update for virt5 mgmt
• 17:52 logmsgbot: reedy synchronized php-1.19/includes/api/ApiQueryLogEvents.php 'r112701'
• 17:50 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'Give editors patrolmarks right on plwikisource & plwiktionary'
• 17:42 logmsgbot: reedy synchronized php-1.19/includes/api/ApiQueryLogEvents.php 'Push back to 1.19wmf1 head'
• 17:38 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'Give editors patrolmarks right on plwiki'
• 17:36 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'Give editors patrolmarks right on plwiki'
• 17:33 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'A few tab w/s tweaks'
• 17:08 logmsgbot: reedy synchronized php-1.19/includes/api/ApiQueryLogEvents.php 'Test reverting r112532, merge of r112374'
• 15:14 Reedy: Running a long slow sql query against db1020 in screen on fenari to pull globalusage titles with spaces in them
• 14:36 logmsgbot: reedy synchronized php-1.18/extensions/GlobalUsage/GlobalUsage_body.php 'r112689'
• 14:26 logmsgbot: reedy synchronized php-1.19/extensions/GlobalUsage/GlobalUsage_body.php 'r112688'
• 13:55 mark: Reinstalled strontium and palladium with hw raid1 and fully automatic lvm based partman recipe
• 12:51 schmir: upgraded mwlib to 0.13.5 on pdf cluster
• 11:35 logmsgbot: hashar synchronized wmf-config/codereview.php 'CodeReview: autodefers /trunk/extensions/ParserFun[/$] so ParserFunctions is not deferred'
• 11:32 logmsgbot: hashar synchronized wmf-config/codereview.php 'CodeReview: autodefers /trunk/extensions/ParserFun'
• 10:21 logmsgbot: hashar synchronized php-1.19/extensions/ApiSandbox 'ApiSandBox: r112114: show request time'
• 03:25 maplebed: took swift out of rotation - thumbnails now served by ms5
• 02:34 logmsgbot: LocalisationUpdate completed (1.19) at Wed Feb 29 02:34:48 UTC 2012
• 02:18 logmsgbot: LocalisationUpdate completed (1.18) at Wed Feb 29 02:18:05 UTC 2012
• 02:02 binasher: manually set large rmem_max and rmem_default on locke and restarted udp2log to stem packet loss, opened an rt ticket to fix the (lost) fix
• 00:55 Ryan_Lane: restarting gerrit again
• 00:23 Ryan_Lane: restarting gerrit on formey
• 00:20 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'mysql parser cache'
• 00:20 Tim: reimported schema files on db40 and re-enabled mysql parser cache

February 28

• 23:32 logmsgbot: midom synchronized wmf-config/db.php 'putting db34 back ha ha thanks for reminding'
• 23:31 logmsgbot: asher synchronized wmf-config/db.php 'pulling db24'
• 22:48 Ryan_Lane: rebuilding manganese to act as new gerrit server
• 22:41 logmsgbot: aaron synchronized php-1.19/extensions/FlaggedRevs/backend/FlaggedRevs.hooks.php 'rc_patrolled bug fix'
• 22:34 logmsgbot: aaron synchronized php-1.19/extensions/FlaggedRevs/business/RevisionReviewForm.php 'debug logging'
• 22:33 Tim: on db40: deleting mysqld data dir and recreating from schema files in /a/dump
• 21:57 mark: Setup servers strontium and palladium as additional (internal) bits servers in eqiad. awaiting connection of eth1-3 before deployment
• 20:50 logmsgbot: reedy synchronized php-1.19/extensions/Collection/Collection.body.php 'r112634'
• 20:49 logmsgbot: reedy synchronized php-1.19/extensions/CentralAuth/specials/SpecialCentralAuth.php 'r112634'
• 20:48 logmsgbot: reedy synchronized php-1.19/resources/mediawiki/mediawiki.util.js 'r112632'
• 20:43 binasher: streaming a hotbackup of db1038 to db1004
• 20:40 binasher: streaming hot backup of db1006 to db1040
• 19:10 logmsgbot: reedy synchronizing Wikimedia installation... : Updating message stuffs
• 18:45 RoanKattouw: Installing Apache on cadmium
• 17:33 logmsgbot: aaron synchronized php-1.19/extensions/PagedTiffHandler/PagedTiffHandler_body.php
• 17:28 logmsgbot: aaron synchronized php-1.19/extensions/PagedTiffHandler/PagedTiffHandler_body.php
• 17:20 logmsgbot: aaron synchronized php-1.19/extensions/PagedTiffHandler/PagedTiffHandler_body.php 'deployed r112614'
• 08:49 logmsgbot: nikerabbit synchronized php-1.19/extensions/Narayam/Narayam.php 'Narayam gu mapping out of beta'
• 08:48 logmsgbot: nikerabbit synchronized php-1.18/extensions/Narayam/Narayam.php 'Narayam gu mapping out of beta'
• 04:48 Tim: on manutius: rebuilt torrus config DB, moved old one out to /var/lib/torrus/db.broken. Restarted.
• 04:30 Tim: torrus down, reporting "PANIC: fatal region error detected; run recovery" about its DB files, will stop apache to investigate
• 03:54 logmsgbot: tstarling synchronized deleted.dblist 'removed chwikimedia'
• 03:54 logmsgbot: tstarling synchronized all.dblist 'removed chwikimedia'
• 03:07 logmsgbot: tstarling synchronized php-1.19/includes/specials/SpecialMovepage.php 'r112572'
• 02:59 logmsgbot: catrope synchronized php-1.19/resources 'r112570'
• 02:59 logmsgbot: catrope synchronized php-1.19/extensions/CheckUser 'r112570'
• 02:58 logmsgbot: catrope synchronized php-1.19/includes/logging 'r112570'
• 02:44 logmsgbot: aaron synchronized php-1.19/includes/Block.php 'deployed r112564'
• 02:43 logmsgbot: aaron synchronized php-1.19/includes/resourceloader/ResourceLoaderContext.php 'deployed r112564'
• 02:43 logmsgbot: aaron synchronized php-1.19/resources/mediawiki/mediawiki.js 'deployed r112564'
• 02:33 logmsgbot: LocalisationUpdate completed (1.19) at Tue Feb 28 02:33:30 UTC 2012
• 02:17 logmsgbot: LocalisationUpdate completed (1.18) at Tue Feb 28 02:17:09 UTC 2012
• 02:13 binasher: moved db1006 to new s6 master
• 01:55 Tim: reduced disk space usage on srv191 by running logrotate manually
• 00:43 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: plwiki to 1.19
• 00:35 logmsgbot: aaron synchronized php-1.19/includes/StubObject.php 'trigger errors for debugging bad callbacks'
• 00:28 Tim: removed 2GB of syslogs on srv192
• 00:25 logmsgbot: tstarling synchronized php-1.18/includes/resourceloader/ResourceLoaderFileModule.php
• 00:21 logmsgbot: tstarling synchronized php-1.19/includes/api/ApiQueryLogEvents.php 'revert live hack'
• 00:20 logmsgbot: tstarling synchronized php-1.19/includes/api/ApiQueryRecentChanges.php

February 27

• 23:46 logmsgbot: catrope synchronized php-1.19/includes/api/ApiQueryLogEvents.php 'Attempted bugfix'
• 23:17 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: nlwiki to 1.19
• 22:54 logmsgbot: reedy synchronized php-1.19/includes/MessageBlobStore.php 'r112536'
• 22:45 RobH: dns update for manganese server
• 22:33 logmsgbot: reedy synchronized php-1.19/includes/ 'r112532'
• 22:31 binasher: new s4 master pos - MASTER_LOG_FILE='db31-bin.000253', MASTER_LOG_POS=457980068
• 22:30 logmsgbot: asher synchronized wmf-config/db.php 'done s4 switch'
• 22:29 binasher: switching s4 master to db31
• 22:29 logmsgbot: asher synchronized wmf-config/db.php 'switching s4 master to db31, setting read-only'
• 22:26 logmsgbot: reedy synchronized php-1.19/extensions/UploadWizard/UploadWizard.i18n.php 'r112531'
• 22:16 RobH: cadmium locked up, rebooting
• 22:14 binasher: running 1.19 schema migration script to get former s5, s6, s1 masters (db45, db47, db36)
• 22:09 binasher: new s1 (enwiki) master pos - MASTER_LOG_FILE='db38-bin.000129', MASTER_LOG_POS=255719721
• 21:56 logmsgbot: asher synchronized wmf-config/db.php 'done s1 switch'
• 21:55 logmsgbot: asher synchronized wmf-config/db.php 'swapping s1 enwiki master to db38, setting read-only'
• 21:53 binasher: preparing to swap enwiki master, it will be read only for a couple minutes
• 21:52 logmsgbot: catrope synchronized php-1.19/extensions/ProofreadPage/proofread.js 'r112522'
• 21:49 logmsgbot: catrope synchronized php-1.19/extensions/ProofreadPage/proofread.js 'r112522'
• 21:15 logmsgbot: nikerabbit synchronized php-1.18/resources/startup.js 'touch'
• 21:02 binasher: db1006 (s6-secondary) is still slaving from db47 - it's very behind post hw failure. need to manually swap to db43 once caught up
• 21:02 binasher: new s6 master pos - MASTER_LOG_FILE='db43-bin.000027', MASTER_LOG_POS=577074024
• 21:01 logmsgbot: asher synchronized wmf-config/db.php 'done s6 master swap'
• 20:59 logmsgbot: asher synchronized wmf-config/db.php 'swapping s6 master to db43, setting read-only'
• 20:59 binasher: preparing to switch s6 master
• 20:42 binasher: powercycled db1006 after finding nothing on the serial console. booted without issue, then started mysql.
• 20:41 logmsgbot: hashar synchronized php-1.19/extensions/CodeReview/backend/DiffHighlighter.php 'Special:CodeReview merge r112513 fix bug 27375'
• 20:31 binasher: new s5 master pos - MASTER_LOG_FILE='db35-bin.000011', MASTER_LOG_POS=374074061
• 20:27 logmsgbot: asher synchronized wmf-config/db.php 'done s5 master swap'
• 20:25 logmsgbot: asher synchronized wmf-config/db.php 'swapping s5 master to db35, setting read-only'
• 20:22 logmsgbot: hashar synchronized php-1.19/extensions/CodeReview/backend/DiffHighlighter.php 'Special:CodeReview fix HTML entities showing up in diff output r112459 r112464'
• 20:18 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Put oldwikisource/sourceswiki on 1.19wmf1
• 19:42 RobH: adjusted threshholds for ps1-b4-sdtpa.mgmt.pmtpa.wmnet again, bottom sensor set to high
• 18:32 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'I18ndeploy config changes: bug 33423 bug 34591'
• 18:26 logmsgbot: nikerabbit synchronized php-1.19/extensions/WebFonts/ 'i18ndeploy r112498'
• 18:26 logmsgbot: nikerabbit synchronized php-1.19/extensions/Narayam/ 'i18ndeploy r112498'
• 18:25 logmsgbot: nikerabbit synchronized php-1.19/languages/messages/ 'i18ndeploy r112498'
• 18:18 logmsgbot: nikerabbit synchronized php-1.18/extensions/WebFonts/ 'i18ndeploy r112497'
• 18:18 logmsgbot: nikerabbit synchronized php-1.18/extensions/Narayam/ 'i18ndeploy r112497'
• 18:17 logmsgbot: nikerabbit synchronized php-1.18/languages/messages/ 'i18ndeploy r112497'
• 18:01 logmsgbot: midom synchronized wmf-config/db.php
• 17:26 mark: Denying POST / requests on frontend squids
• 17:10 RobH: blog plugins updated, blog puppet config updated to support unzip package
• 16:55 RobH: blog updated to newest version
• 10:23 logmsgbot: hashar synchronized php-1.19/includes/Pager.php '(bug 34736) empty limit on special pages causes navigation issues'
• 03:02 Ryan_Lane: restarting varnish on arsenic
• 02:53 Ryan_Lane: moving bits traffic to pmtpa
• 02:44 Ryan_Lane: restarted varnish on niobium and arsenic
• 02:34 logmsgbot: LocalisationUpdate completed (1.19) at Mon Feb 27 02:34:02 UTC 2012
• 02:17 logmsgbot: LocalisationUpdate completed (1.18) at Mon Feb 27 02:17:32 UTC 2012

February 26

• 02:34 logmsgbot: LocalisationUpdate completed (1.19) at Sun Feb 26 02:34:55 UTC 2012
• 02:17 logmsgbot: LocalisationUpdate completed (1.18) at Sun Feb 26 02:17:55 UTC 2012

February 25

• 02:33 logmsgbot: LocalisationUpdate completed (1.19) at Sat Feb 25 02:33:27 UTC 2012
• 02:17 logmsgbot: LocalisationUpdate completed (1.18) at Sat Feb 25 02:17:41 UTC 2012
• 01:53 RoanKattouw: Started transcode jobs on cadmium, 16 parallel jobs running in screen
• 01:45 logmsgbot: aaron synchronized php-1.19/includes/filerepo/backend/FileBackend.php 'deployed r112382'
• 01:44 logmsgbot: aaron synchronized php-1.19/includes/StreamFile.php 'deployed r112382'
• 01:34 RoanKattouw: Installing screen on cadmium
• 01:26 RoanKattouw: Installing ffmpeg2theora on cadmium
• 01:25 RoanKattouw: Creating a 'wikimaniatranscode' user locally on cadmium because I don't really want to run ffmpeg as root
• 01:25 LeslieCarr: reloading cp1043 again
• 01:22 LeslieCarr: cp1043 is missing /var/lib/varnish/frontend
• 01:17 LeslieCarr: rebooted cp1043
• 01:09 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Remove outdated itwiki lockdown bypass code'
• 01:01 logmsgbot: aaron synchronized php-1.19/includes/StreamFile.php 'deployed r112379'
• 00:44 logmsgbot: aaron synchronized php-1.19/includes/filerepo/backend/FileBackend.php 'deployed r112377'
• 00:27 notpeter: starting indexer on searchidx2
• 00:18 notpeter: stopping indexer on searchidx2 again :/

February 24

• 23:12 LeslieCarr: restarted apache on singer again
• 23:10 notpeter: restarting indexer and lucene on searchidx2
• 23:05 LeslieCarr: Server should be SSL-aware but has no certificate configured [Hint: SSLCertificateFile] ((null):0)
• 23:03 LeslieCarr: reloading apache2 on singer
• 23:03 LeslieCarr: pushing new apache conf file to singer for secure.wikimedia.org - may impact performance of secure site
• 22:42 logmsgbot: aaron synchronized php-1.19/includes/specials/SpecialContributions.php 'deployed r112366'
• 22:39 LeslieCarr: reloaded apache2 on stafford (puppetmaster)
• 22:05 logmsgbot: aaron synchronized php-1.19/extensions/FlaggedRevs/frontend/modules/ext.flaggedRevs.review.js 'deployed r112361'
• 18:24 logmsgbot: aaron synchronized multiversion/MWVersion.php
• 18:18 logmsgbot: aaron synchronized multiversion/MWMultiVersion.php 'deployed r112335'
• 18:04 logmsgbot: aaron synchronized multiversion 'deployed all changes through HEAD'
• 17:58 logmsgbot: catrope synchronized php-1.19/LocalSettings.php 'Guard against /home/wikipedia not existing'
• 17:51 RoanKattouw: And of course I can't commit this because the code in /h/w/common/multiversion hasn't been updated this calendar year and there are undeployed commits from January *grumble*
• 17:47 notpeter: stopping indexing on searchidx2 to rsync over a clean copy of index to searchidx1001
• 17:45 logmsgbot: catrope synchronized multiversion/MWVersion.php 'Add file_exists check for /home before trying to access /home'
• 13:40 mark: Fixed directory permissions of /srv/swift-storage/{sda4,sdb4} on ms-be1
• 13:35 mark: Copied swift ring builder files from ms-fe1 to all swift hosts
• 12:02 logmsgbot: tstarling synchronized php-1.19/includes/PathRouter.php 'r112316'
• 12:02 logmsgbot: tstarling synchronized php-1.19/includes/AutoLoader.php 'r112316'
• 11:51 mark: Rebalanced swift rings account, container, object after adding ms-be5
• 11:49 mark: Added all devices on ms-be5 into the swift rings, new zone 5, weight 100
• 11:27 mark: Manually preparing swift filesystems sda4 and sdb4 on ms-be5
• 11:00 mark: Reinstalling ms-be5 to correct partitioning of sda and sdb
• 10:15 mark: Doing first puppet run on ms-be5
• 08:48 mark: Restarted apache on stafford to fix puppetmaster
• 05:32 K4-713: synchronized payments cluster to r112287
• 05:18 logmsgbot: tstarling synchronized php-1.19/extensions/SecurePoll/includes/pages/Page.php 'r112301'
• 05:18 logmsgbot: tstarling synchronized php-1.19/extensions/LandingCheck/SpecialLandingCheck.php 'r112301'
• 05:17 logmsgbot: tstarling synchronized php-1.19/extensions/DonationInterface/globalcollect_gateway/globalcollect.adapter.php 'r112301'
• 04:32 logmsgbot: tstarling synchronized php-1.19/includes/api/ApiFeedContributions.php
• 04:00 Tim: cleaning up /tmp on all apaches
• 03:57 logmsgbot: tstarling synchronized php-1.19/extensions/skins/Donate/Donate.class.php
• 03:56 logmsgbot: tstarling synchronized php-1.19/extensions/skins/Tomas/Tomas.class.php
• 03:56 logmsgbot: tstarling synchronized php-1.19/extensions/skins/Schulenburg/Schulenburg.class.php
• 03:49 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Move wikisource over to 1.19
• 03:39 logmsgbot: tstarling synchronized php-1.19/extensions/ContributionTracking/ContributionTracking_body.php 'r112294'
• 03:36 Tim: cleaned up /tmp on mw35
• 03:36 logmsgbot: reedy synchronized php-1.19/extensions/DoubleWiki/DoubleWiki_body.php 'r112292'
• 03:15 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
• 03:13 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'fix kowiki namespace aliases'
• 03:09 logmsgbot: tstarling synchronized php-1.19/includes/db/DatabaseMysql.php 'debugging patch with trigger_error'
• 03:07 logmsgbot: tstarling synchronized php-1.19/includes/db/DatabaseMysql.php 'debugging patch for array parameter warning'
• 03:06 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Move specials over to 1.19
• 02:58 Tim: re-enabling wmerrors
• 02:39 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Move wikitionarys over to 1.19
• 02:34 logmsgbot: LocalisationUpdate completed (1.19) at Fri Feb 24 02:34:38 UTC 2012
• 02:32 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Move wikibooks over to 1.19
• 02:24 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Move wikiquotes over to 1.19
• 02:18 logmsgbot: LocalisationUpdate completed (1.18) at Fri Feb 24 02:18:10 UTC 2012
• 02:13 logmsgbot: reedy synchronized php-1.19/includes/api/ApiParamInfo.php 'r112291'
• 02:11 RoanKattouw: Installing nfs-kernel-server on cadmium
• 02:04 maplebed: deployed updated rewrite.py to swift to pass through error codes and error messages it gets from the back end during 404 handling.
• 02:03 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikimedia wikis to 1.19
• 01:48 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: switching all wikinewses to 1.19
• 01:44 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: and hewikisource
• 01:44 RoanKattouw: Installing ffmpeg on cadmium
• 01:44 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: reverted wikisources back to 1.18 except frwikisource
• 01:17 logmsgbot: aaron synchronized php-1.19/extensions/FlaggedRevs 'deployed r112284'
• 01:03 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: switching wikisource to 1.19 except for wikis with FlaggedRevs enabled
• 00:47 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Put all wikiversity on 1.19wmf1
• 00:24 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews back to 1.18
• 00:23 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews back to 1.18
• 00:19 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'reduce startup module expiry time for all projects except wikipedia'
• 00:19 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'reduce startup module expiry time for all projects except wikipedia'
• 00:05 K4-713: synchronized payments cluster to r112275

February 23

• 23:54 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Put all wikinews' on 1.19wmf1
• 23:35 logmsgbot: reedy synchronizing Wikimedia installation... :
• 23:13 logmsgbot: catrope synchronized php-1.19/extensions/ClickTracking 'r112264'
• 23:13 logmsgbot: catrope synchronized php-1.19/extensions/WikiEditor 'r112264'
• 23:12 logmsgbot: aaron synchronized php-1.19/extensions/OggHandler 'deployed r112264'
• 23:12 logmsgbot: catrope synchronized php-1.18/extensions/ClickTracking 'r112264'
• 23:12 logmsgbot: catrope synchronized php-1.18/extensions/WikiEditor 'r112264'
• 22:35 logmsgbot: reedy synchronized php-1.19/resources/startup.js 'touch'
• 22:35 logmsgbot: reedy synchronized php-1.19/extensions/UploadWizard/ 'r112256'
• 22:14 logmsgbot: reedy synchronized php-1.19/includes/resourceloader/ResourceLoaderStartUpModule.php 'r112249'
• 22:13 logmsgbot: reedy synchronized php-1.19/resources/mediawiki/mediawiki.feedback.css 'r112249'
• 22:07 Ryan_Lane: enabling LiquidThreads on labsconsole
• 22:02 maplebed: fixing permissions in /var/spool on brewster
• 21:46 RobH: rebooting cadmium for pxe test, not reinstalling it.
• 21:22 mark: Moved /var/spool/squid to its own LV on brewster
• 21:02 logmsgbot: aaron synchronized php-1.18/StartProfiler.php 'require() wmf-config profiler'
• 21:01 logmsgbot: aaron synchronized wmf-config/StartProfiler.php 'added empty arrays to constructors as needed'
• 21:00 logmsgbot: aaron synchronized php-1.19/StartProfiler.php
• 20:55 logmsgbot: aaron synchronized php-1.19/StartProfiler.php
• 20:54 logmsgbot: aaron synchronized php-1.19/StartProfiler.php 'require() wmf-config profiler'
• 20:50 logmsgbot: aaron synchronized wmf-config/StartProfiler.php 'Sample thumb.php traffic properly. Removed 'bigpage' and 'incubatorslowness' hacks. Split thumbnail group by MW version and added some more code comments.'
• 20:42 logmsgbot: aaron synchronized wmf-config/StartProfiler.php 'updated profiling class code paths to post-1.17 locations'
• 20:34 logmsgbot: aaron synchronized wmf-config/StartProfiler.php 'Sample thumb.php traffic properly. Removed 'bigpage' and 'incubatorslowness' hacks. Split thumbnail group by MW version and added some more code comments.'
• 20:01 logmsgbot: reedy synchronized php-1.19/extensions/UploadWizard/includes/specials/SpecialUploadWizard.php 'r112235'
• 19:55 logmsgbot: aaron synchronized wmf-config/StartProfiler.php 'removed commented out code and w/s cleanups'
• 18:34 logmsgbot: aaron synchronized php-1.19/includes/parser/Parser.php 'disabled per-template profiling'
• 17:26 RobH: db15 shutting down for memtest
• 15:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34401 - Change sitename and a namespace for Inuktitut Wikipedia'
• 15:43 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34401 - Change sitename and a namespace for Inuktitut Wikipedia'
• 15:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34401 - Change sitename and a namespace for Inuktitut Wikipedia'
• 14:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34407 - New aliases for ko.wikipedia'
• 14:52 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34566 - Modify the URL to upload files in eswiki'
• 14:50 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bump AF lottery odds for ptwikipedia'
• 14:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34476 - Set $wgLogo value for fawikisource'
• 14:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34570 - Set wgBabelMainCategory to false for dewiki'
• 14:41 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34615 - Disable Moodbar on Tamil Wikipedia'
• 14:40 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php 'Bug 33273 - Enable FlaggedRevs on Ukrainian Wikipedia'
• 11:26 logmsgbot: tstarling synchronizing Wikimedia installation... : scap speed test
• 09:36 logmsgbot: tstarling synchronized php-1.18/includes/LocalisationCache.php 'revert live hack'
• 02:37 logmsgbot: LocalisationUpdate completed (1.19) at Thu Feb 23 02:37:12 UTC 2012
• 02:18 logmsgbot: LocalisationUpdate completed (1.18) at Thu Feb 23 02:18:24 UTC 2012
• 01:37 K4-713: Updated and synchronized fraud prevention settings on the payments cluster.
• 01:04 logmsgbot: reedy synchronized php-1.19/resources/ 'r112174'
• 00:34 LeslieCarr: reinstalling neon

February 22

• 23:54 logmsgbot: reedy synchronized php-1.19/extensions/UploadWizard/resources/mw.UploadWizard.js 'r112167'
• 23:42 Ryan_Lane: restarting opendj on virt0
• 23:37 Tim: fixed ownership on /mnt/upload6/wikimedia/rs
• 23:37 logmsgbot: reedy synchronized php-1.19/extensions/WikiEditor/ 'r112164'
• 23:35 logmsgbot: reedy synchronized php-1.19/extensions/Vector/Vector.php 'r112164'
• 23:35 logmsgbot: reedy synchronized php-1.19/extensions/UploadWizard/resources/mw.fileApi.js 'r112164'
• 23:34 logmsgbot: reedy synchronized php-1.19/maintenance/language/ 'r112162'
• 23:32 logmsgbot: reedy synchronized php-1.19/includes/SkinTemplate.php 'r112162'
• 22:21 logmsgbot: aaron synchronized wmf-config/swift.php 'avoid notices'
• 22:19 logmsgbot: aaron synchronized wmf-config/swift.php
• 22:06 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: commonswiki -> 1.19
• 21:52 RobH: dataset1001 eth1 connected
• 21:18 logmsgbot: catrope synchronized php-1.19/thumb.php 'Cleanup: rename fake repo and add comments'
• 21:08 K4-713: updated the payments cluster to r112145
• 20:55 logmsgbot: catrope synchronized php-1.19/thumb.php 'And let's try that again'
• 20:54 logmsgbot: reedy synchronized php-1.19/resources/startup.js 'touch'
• 20:49 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'UW config for test2wiki'
• 20:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable UW on test2wiki'
• 20:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable UW on test2wiki'
• 20:43 Reedy: Created uploadwizard campaign related tables on test2wiki
• 20:35 LeslieCarr: flushed iptables on stafford - all puppet runs shoudl now work
• 19:49 logmsgbot: catrope synchronized php-1.19/thumb.php 'pass in a Title object to UnregisteredLocalFile'
• 19:46 logmsgbot: catrope synchronized php-1.19/thumb.php 'more logging'
• 19:41 logmsgbot: catrope synchronized php-1.19/thumb.php 'more logging'
• 19:39 logmsgbot: catrope synchronized php-1.19/thumb.php 'Add logging for no path supplied error'
• 19:39 LeslieCarr: blocking all new puppet connections on all hosts except neon
• 19:35 logmsgbot: catrope synchronized php-1.19/thumb.php 'use temp path correctly'
• 19:31 logmsgbot: catrope synchronized php-1.19/thumb.php 'readd debugging for 404s'
• 19:30 logmsgbot: catrope synchronized php-1.19/thumb.php 'disable debugging'
• 19:27 logmsgbot: catrope synchronized php-1.19/thumb.php 'attempt at debugging'
• 19:24 LeslieCarr: removing old wap (mobile) site from ekrem as it hasn't been accessed in a day
• 19:18 logmsgbot: catrope synchronized php-1.19/thumb.php 'missing name key'
• 19:16 logmsgbot: catrope synchronized php-1.19/thumb.php 'missing global'
• 19:13 logmsgbot: aaron synchronized php-1.19/includes/logging/LogFormatter.php 'deployed r112136'
• 19:05 RoanKattouw_away: running sync-common on srv256 because Aaron gets key errors for that box
• 18:48 logmsgbot: aaron synchronizing Wikimedia installation... : deploying r112128
• 18:36 Ryan_Lane: shutting down labstore1
• 17:04 notpeter: increasing mem for java to 3300 on pmtpa search hosts
• 16:09 Jeff_Green: restarted lsearchd on search3 and search9, was running but nonresponsive
• 16:09 Jeff_Green: restarted lsearchd on search15, was not running
• 15:34 notpeter: extending database wikiadmin user grants to 10.64.0.0/255.255.252.0
• 10:09 logmsgbot: hashar synchronized php-1.19/extensions/CodeReview/backend/DiffHighlighter.php 'r112098 - (bug 34554) diff chunk fail to parse file add/rm'
• 09:45 Tim: on fenair: stopped apache again due to overload. Restarted it with reduced MaxClients
• 09:34 logmsgbot: tstarling synchronizing Wikimedia installation... :
• 09:23 Tim: on fenari: started apache
• 09:09 logmsgbot: tstarling synchronizing Wikimedia installation... :
• 09:05 logmsgbot: tstarling synchronizing Wikimedia installation... :
• 09:02 logmsgbot: tstarling synchronizing Wikimedia installation... :
• 08:07 logmsgbot: ariel rebuilt wikiversions.cdb and synchronized wikiversions files: fix dewikiversity typo in wikiversions file
• 06:49 logmsgbot: tstarling synchronized php-1.19/languages/messages/MessagesEn.php 'test change for manualRecache'
• 06:33 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'enabling manual recache'
• 06:26 Tim: installed new scap manually since puppet is broken
• 05:51 logmsgbot: tstarling synchronizing Wikimedia installation... :
• 05:18 logmsgbot: catrope synchronized php-1.19/thumb.php 'Experimental fix for UploadStash thumbs in 1.19'
• 05:14 Tim: started xinetd
• 04:34 logmsgbot: catrope synchronized php-1.19/thumb.php 'Experimental fix for 1.19 UploadWizard thumb issue'
• 04:17 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: rolling back commons to 1.18
• 03:48 logmsgbot: catrope synchronized php-1.19/skins/common/shared.css 'r112081'
• 03:25 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files:
• 03:25 Tim: switching commons back to 1.19
• 03:18 logmsgbot: tstarling synchronized php-1.19/includes/LocalisationCache.php
• 03:18 logmsgbot: tstarling synchronized php-1.19/includes/LocalisationCache.php
• 03:10 Tim: on fenari: NFS overload, killed apache and xinetd
• 03:03 RoanKattouw: Manually started ircecho on fenari ; why doesn't this happen upon boot? Why didn't puppet start it?
• 02:57 RoanKattouw: Synced php-1.19/includes/MessageBlobStore.php to disable ::clear() ; where's the logging bot?
• 02:53 LeslieCarr: manually lowering nagios max checks to 300
• 02:48 Tim: rebooted fenari, nonresponsive
• 02:46 LeslieCarr: reset the drac console for spence
• 02:21 logmsgbot: tstarling synchronizing Wikimedia installation... :
• 02:18 logmsgbot: tstarling synchronizing Wikimedia installation... :
• 02:18 logmsgbot: LocalisationUpdate completed (1.18) at Wed Feb 22 02:18:08 UTC 2012
• 02:15 Tim: testing new scap script ~tstarling/bin/scap-new
• 01:57 logmsgbot: reedy synchronized php-1.19/extensions/CentralAuth/specials/ 'r112075'
• 01:18 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files:
• 01:18 Tim: reverting to 1.18 on commons due to DB overload
• 01:09 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Commonswiki to 1.19wmf1
• 01:00 logmsgbot: reedy synchronized php-1.19/includes/ 'r112073'
• 00:59 logmsgbot: reedy synchronized php-1.19/languages/messages/MessagesEn.php 'r112073'
• 00:51 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'reducing cache expiry for unversioned resources on commons'
• 00:51 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'reducing cache expiry for unversioned resources on commons'
• 00:41 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'reducing cache expiry for unversioned resources on commons'

February 21

• 22:44 logmsgbot: catrope synchronized php-1.19/includes/resourceloader/ 'r112055'
• 22:20 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Comment out hack that enabled $wgResourceLoaderExperimentalAsyncLoading for logged-in users'
• 21:36 Ryan_Lane: force-running puppet on every labs instance
• 17:35 logmsgbot: reedy synchronized php-1.18/extensions/FeaturedFeeds/FeaturedFeeds.body.php 'r112029'
• 17:16 notpeter: reimaging searchidx1001 :(
• 17:10 logmsgbot: reedy synchronized php-1.19/includes/ 'r112024'
• 17:08 logmsgbot: reedy synchronized php-1.19/extensions/CentralAuth/CentralAuthHooks.php 'r112023'
• 17:08 logmsgbot: reedy synchronized php-1.19/extensions/FeaturedFeeds/ 'r112023'
• 16:50 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34560 - Moodbar on ta.wikipedia'
• 15:39 logmsgbot: reedy synchronized php-1.19/extensions/CentralAuth/CentralAuth.php 'Enable AntiSpooof for CentralAuth on all 1.19 wikis again, doesn't break signup with a mass of fail'
• 15:37 logmsgbot: reedy synchronized php-1.19/extensions/CentralAuth/CentralAuth.php 'Enable AntiSpoof for CentralAuth on testwiki only'
• 04:42 Tim: on db40: setting innodb-use-purge-thread=4 to test multithreaded purge
• 04:12 notpeter: disabling search lvs1 check because it's going to false-positive in 4 hours...
• 02:34 logmsgbot: LocalisationUpdate completed (1.19) at Tue Feb 21 02:34:36 UTC 2012
• 02:17 logmsgbot: LocalisationUpdate completed (1.18) at Tue Feb 21 02:17:36 UTC 2012

February 20

• 22:14 logmsgbot: hashar synchronized php-1.19/includes/logging/PatrolLog.php 'r111969 - bug 34495 patrol log credit the user patrolled, not the user patrolling'
• 21:10 rainman-sr: shut down lucene on search15, comes up with some strange errors "Connection refused to host: 10.0.3.15"
• 18:46 logmsgbot: aaron synchronized php-1.19/includes/RevisionList.php 'deployed r111952'
• 18:26 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Narayam on knwiki; bug 34516'
• 18:19 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Narayam on mrwiki, mrwikisource; bug 32669, bug 34454'
• 18:17 logmsgbot: nikerabbit synchronized php-1.19/extensions/Narayam/ 'i18ndeploy r111946'
• 18:16 logmsgbot: nikerabbit synchronized php-1.18/extensions/Narayam/ 'i18ndeploy r111945'
• 17:22 notpeter: stopping puppet on brewster
• 15:56 notpeter: initial test-spinup of searchidx1001 and search1001-1006 (en cluster)
• 14:56 logmsgbot: hashar synchronized php-1.19/skins/simple/main.css 'r111580 for Bug 34397: align footer so that it does not overlap with sidebar in Simple skin'
• 13:36 logmsgbot: hashar synchronized php-1.19/includes/UserMailer.php 'r111925 for bug 34421 duplicate Subject / wrong To: headers in mail'
• 03:06 notpeter: re-enabling notifications for search-pool1 and search-pool3, search-pool2 still flapping very badly
• 02:36 logmsgbot: LocalisationUpdate completed (1.19) at Mon Feb 20 02:36:36 UTC 2012
• 02:18 logmsgbot: LocalisationUpdate completed (1.18) at Mon Feb 20 02:18:57 UTC 2012

February 19

• 13:47 notpeter: disabling notifications for search lvs... if anyone still has their phone on
• 02:34 logmsgbot: LocalisationUpdate completed (1.19) at Sun Feb 19 02:34:11 UTC 2012
• 02:17 logmsgbot: LocalisationUpdate completed (1.18) at Sun Feb 19 02:17:17 UTC 2012

February 18

• 15:18 logmsgbot: reedy synchronized php-1.19/extensions/CentralAuth
• 15:17 logmsgbot: reedy synchronized php-1.19/extensions/AntiSpoof/
• 14:31 logmsgbot: reedy synchronized php-1.19/includes/actions/HistoryAction.php 'r111828'
• 10:41 logmsgbot: catrope synchronized php-1.19/resources/startup.js 'touch'
• 10:41 logmsgbot: catrope synchronized php-1.19/resources/mediawiki/mediawiki.user.js 'touch'
• 10:40 logmsgbot: catrope synchronized php-1.19/resources/mediawiki/mediawiki.js 'touch'
• 08:56 apergos: restarted all the searchpool1 lsearchds
• 02:35 logmsgbot: LocalisationUpdate completed (1.19) at Sat Feb 18 02:35:12 UTC 2012
• 02:18 logmsgbot: LocalisationUpdate completed (1.18) at Sat Feb 18 02:18:33 UTC 2012
• 01:59 logmsgbot: aaron synchronized php-1.19/extensions/CentralAuth/CentralAuth.php 'disabled AntiSpoof hooks which broken account creation with DB errors'
• 01:00 logmsgbot: andrew synchronizing Wikimedia installation... :
• 00:58 Andrew: Running scap to ensure a consistent environment
• 00:51 logmsgbot: andrew synchronized php-1.19/resources/mediawiki/mediawiki.user.js 'Attempt to repush r111695'
• 00:44 logmsgbot: andrew synchronized php-1.19/resources/Resources.php 'Deploy r111809'
• 00:43 logmsgbot: aaron synchronized php-1.19/resources/mediawiki.special/mediawiki.special.preferences.js 'deployed r111808'
• 00:21 logmsgbot: andrew synchronized php-1.19/extensions/Vector/modules/ext.vector.collapsibleNav.js 'Deploy r111806'
• 00:16 logmsgbot: andrew synchronized php-1.19/extensions/Vector/modules/ext.vector.collapsibleNav.js 'Deploy r111804'

February 17

• 22:51 logmsgbot: andrew synchronized php-1.19/includes/actions/HistoryAction.php 'Deploy r111800'
• 22:22 logmsgbot: andrew synchronized php-1.19/includes/Linker.php 'deploy r111798'
• 21:30 notpeter: remounted nfs mounts on searchidx2. to protect our house of cards
• 21:03 notpeter: unmounting ms nfs mounts from searchidx2
• 20:44 notpeter: restarting puppet on brewster
• 20:33 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Bug 34235 - Enable WebFonts on am.wikipedia'
• 20:13 maplebed: sending thumbnail traffic back to swift; head bug is fixed.
• 20:10 RobH: reinstalling search1001 and searchidx1001
• 19:53 RobH: search1001 down for reinstall
• 19:37 RobH: ran dsh command to remove all the /tmp/mw-cache-1.17
• 19:27 LeslieCarr: manually cleaned up tmp on mw41
• 19:27 RobH: manually cleaned up tmp on mw21
• 19:01 LeslieCarr: changed sockpuppet's post-merge hook so that you need to have ssh keys forwarded (though you really would anyways due to brokenness)
• 18:51 logmsgbot: aaron synchronized php-1.18/includes/api/ApiQueryAllUsers.php
• 18:50 logmsgbot: reedy synchronized wmf-config/ 'Sync ExtensionMessages'
• 18:44 logmsgbot: reedy synchronized php-1.19/extensions/WikimediaMessages/
• 18:43 logmsgbot: reedy synchronized php-1.18/extensions/WikimediaMessages/
• 18:22 logmsgbot: aaron synchronized php-1.18/includes/api/ApiQueryAllUsers.php
• 18:19 logmsgbot: aaron synchronized php-1.18/includes/api/ApiQueryAllUsers.php 'force index'
• 18:06 maplebed: sending thumbnail traffic back to ms5, taking swift out of production
• 18:00 notpeter: temporarily stopping puppet on brewster. please let me know if you need to turn it back on
• 17:52 maplebed: changing squids to send 100% of thumbnail traffic to swift
• 16:58 maplebed: turned swift live for 50% of all thumbnail requests
• 15:02 logmsgbot: hashar synchronized php-1.19/LocalSettings.php 'make jobrunners for testwiki to use the apache CommonSettings file instead of the non existant /home/wikipedia one'
• 14:59 logmsgbot: reedy synchronized php-1.19/includes/specials/SpecialDeletedContributions.php 'r111752'
• 14:58 logmsgbot: hashar synchronized php-1.19/LocalSettings.php
• 14:40 logmsgbot: hashar synchronized php-1.19/LocalSettings.php 'debug statement with argv join'
• 14:37 logmsgbot: hashar synchronized php-1.19/LocalSettings.php 'syslog debug statement to investigate TESTWIKI issue'
• 13:40 ^demon: srv233: removed /tmp/mw-cache-1.17 to give it a little more space for now
• 09:58 mark: Shutdown ragweed for decommissioning
• 07:42 binasher: upgraded mysql on db40 to 5.1.53-facebook-r3753, enabled innodb_use_purge_thread
• 05:39 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
• 05:38 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
• 05:35 Tim: on db40: reduced to 10M, should be causing massive delays, but the site's not down and the purge rate is lower if anything. Going to disable the mysql parser cache entirely.
• 05:25 Tim: on db40: purge lag is still increasing at 108 per second, so reducing innodb_max_purge_lag to 50M
• 05:21 Tim: on db40: giving the innodb manual the benefit of the doubt and following its advice, setting innodb_max_purge_lag to 100M, which should give a delay of 4.5ms
• 05:13 Tim: killing purgeParserCache.php since it is probably doing more harm than good
• 02:43 maplebed: deployed updated thumb_handler.php to ms5 to include Content-Length in generated images
• 02:34 logmsgbot: LocalisationUpdate completed (1.19) at Fri Feb 17 02:34:32 UTC 2012
• 02:29 Ryan_Lane: installed labstore1-4
• 02:17 logmsgbot: LocalisationUpdate completed (1.18) at Fri Feb 17 02:17:35 UTC 2012
• 02:16 Tim: on db40: truncating pc008 - 15
• 02:01 binasher: db1035 is replicating again
• 01:12 Ryan_Lane: re-enabled the mobile plugin for the blogs, seems w3 total cache supports varying
• 01:03 Ryan_Lane: disabling mobile skin for the blogs - we need to fix varnish support first
• 00:38 Ryan_Lane: fixed singer by adding in ssl configuration to the planet configuration

February 16

• 23:53 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Tweak live hack so async loading is enabled for all logged-in users on all 1.19 wikis'
• 23:10 binasher: truncated 4 tables on db40
• 23:10 logmsgbot: catrope synchronized php-1.19/resources/startup.js 'touch'
• 23:09 logmsgbot: catrope synchronized php-1.19/resources/mediawiki/mediawiki.js 'r111699, r111700'
• 23:09 logmsgbot: catrope synchronized php-1.19/includes/resourceloader/ 'r111699'
• 23:04 binasher: db1035 is fubar after crashing during schema migrations, running a hotbackup from db1019
• 22:59 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Also give User:Cmcmahon experimental async loading on meta'
• 22:37 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Add live hack to enable $wgResourceLoaderExperimentalAsyncLoading on meta only for me (User:Catrope)'
• 22:35 binasher: db1035 died 2 days ago, attempting to power cycle
• 22:26 binasher: adding search15 to search-pool2 lvs vip
• 22:23 Ryan_Lane: restarting pdns ns2
• 22:17 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable wgResourceLoaderExperimentalAsyncLoading on test2wiki'
• 21:45 apergos: singer certificate issues, looks like
• 21:40 logmsgbot: reedy synchronized php-1.19/extensions/Vector/modules/ext.vector.collapsibleNav.js 'r111687'
• 21:18 apergos: the most recent apache update (thanks puppet) must have broke things on singer. the url.wm.o config wants /srv/org/wikimedia/url/ but I have no idea what that service ever did or what is supposed to be in there. will someone who knows this undocumented information please check it? thanks.
• 21:16 notpeter: stopping mysql and apache on searchidx2... not sure why they are there. also, going to clean up some packages... like the ubuntu version of mediawiki
• 20:29 logmsgbot: reedy synchronized php-1.19/includes/api/ApiQueryAllUsers.php 'r111675'
• 19:45 logmsgbot: aaron synchronized php-1.18/includes/api/ApiQueryAllUsers.php 'pushing comment changes :)'
• 19:40 LeslieCarr: running /etc/network/if-up.d/initcwnd on the apaches
• 19:36 logmsgbot: aaron synchronized php-1.18/includes/api/ApiQueryAllUsers.php
• 19:30 logmsgbot: aaron synchronized php-1.18/includes/api/ApiQueryAllUsers.php
• 19:29 AaronSchulz: doing some debugging for bug 34451
• 19:29 logmsgbot: aaron synchronized php-1.18/includes/api/ApiQueryAllUsers.php
• 19:26 logmsgbot: aaron synchronized php-1.19/includes/api/ApiQueryAllUsers.php
• 19:25 logmsgbot: aaron synchronized php-1.19/includes/api/ApiQueryAllUsers.php
• 19:04 apergos: restarted lighty on dataset2, silly thing
• 19:02 LeslieCarr: restarted pdns on ns0
• 18:37 LeslieCarr: reverting $lang.wap.wikipedia.org dns changes
• 18:33 LeslieCarr: updating $lang.wap.wikipedia.org dns to point to mobile-lb.eqiad.wikimedia.org
• 17:31 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: betawikiversity to 1.19wmf1
• 17:23 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: hewikisource to 1.19wmf1
• 17:21 logmsgbot: hashar rebuilt wikiversions.cdb and synchronized wikiversions files: moving eowiki to 1.19
• 16:58 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: frwikisource to 1.19wmf1
• 16:55 logmsgbot: reedy synchronized php-1.18/extensions/FeaturedFeeds/ 'r111650'
• 16:53 RobH: rebooting sq35 & sq38, serial console blank
• 16:30 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwikibooks and enwikiquote to 1.19wmf1
• 13:25 Reedy: Running cleanupUploadStash.php across all wikis
• 12:14 mark: Shutdown lily for decommissioning
• 11:47 mark: Moved udpmcast unicast-to-multicast HTCP relay from lily to hooft
• 09:49 Tim: on db40: truncated pc004, pc005, pc006, pc007
• 09:46 logmsgbot: hashar synchronized wmf-config/swift.php 'Add a wfDebugLog call for bug 34440: swift list_objects giving InvalidResponseException'
• 05:21 Tim: on hume: running mwscript purgeParserCache.php --wiki=enwiki --age=7776000
• 05:19 Tim: truncated pc000, pc001, pc002, pc003
• 05:11 Tim: on db40: truncating a few shards to free up space for the OS
• 04:29 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
• 04:16 logmsgbot: tstarling synchronized php-1.18/StartProfiler.php
• 03:43 logmsgbot: tstarling synchronized wmf-config/db.php 'restored db12 in query groups now that the schema changes have finished'
• 02:46 logmsgbot: aaron synchronized php-1.19/extensions/Translate/TranslateHooks.php 'live hack to deal with 500s on log/RC views'
• 02:34 logmsgbot: LocalisationUpdate completed (1.19) at Thu Feb 16 02:34:30 UTC 2012
• 02:32 logmsgbot: asher synchronized wmf-config/StartProfiler.php 'setting 1.18 wiki profiling id to all'
• 02:25 logmsgbot: asher synchronized wmf-config/db.php 'moving watchlist etc from db12 to db53'
• 02:21 logmsgbot: tstarling synchronized php-1.18/StartProfiler.php
• 02:18 logmsgbot: tstarling synchronized php-1.18/StartProfiler.php
• 02:18 logmsgbot: LocalisationUpdate completed (1.18) at Thu Feb 16 02:18:23 UTC 2012
• 02:18 logmsgbot: tstarling synchronized php-1.19/StartProfiler.php
• 02:15 logmsgbot: tstarling synchronized php-1.19/StartProfiler.php
• 02:14 logmsgbot: tstarling synchronized php-1.19/StartProfiler.php
• 02:11 logmsgbot: tstarling synchronized wmf-config/StartProfiler.php 'split out 1.19'
• 02:06 logmsgbot: reedy synchronized php-1.19/includes/filerepo/backend/FileBackend.php 'Add wfProfileOut( __METHOD__ )'
• 01:59 logmsgbot: aaron synchronized php-1.19/includes/parser/CoreParserFunctions.php 'fixed profiling calls'
• 01:56 binasher: 1.19 schema migraitons now running on enwiki slaves
• 01:35 Ryan_Lane: rebooting formey
• 01:33 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
• 01:31 Ryan_Lane: distupgrading formey
• 01:29 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 're-enabled WikimediaLicenseTexts'
• 01:28 logmsgbot: tstarling synchronized php-1.19/extensions/WikimediaMessages/WikimediaLicenseTexts.i18n.php
• 01:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Just disable wmgUseWikimediaLicenseTexts for testwiki'
• 01:08 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Scrap that'
• 01:04 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Disable WikimediaLicenseTexts for the time being'
• 00:55 Ryan_Lane: rebooting prototype.wikimedia.org
• 00:50 Ryan_Lane: dist-upgrading prototype.wikimedia.org
• 00:49 logmsgbot: aaron synchronized extract2.php 'fixed access to protected Page field'
• 00:47 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: meta to 1.19wmf1
• 00:44 maplebed: starting to delete broken thumbnails from swift and squid. job running in a screen session on ms-fe1
• 00:38 logmsgbot: aaron synchronized live-1.5/robots.php 'fixed access to protected Page field'
• 00:35 binasher: changing search15 to run regular search-pool2 indexes instead of highlights
• 00:35 logmsgbot: tstarling synchronized php-1.19/includes/HistoryBlob.php
• 00:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: strategywiki, usabilitywiki, simplewiki and simplewiktionary to 1.19wmf1
• 00:24 logmsgbot: aaron synchronized php-1.19/includes/Article.php
• 00:16 logmsgbot: tstarling synchronized php-1.19/includes/HistoryBlob.php 'temp fix for checksum bug'

February 15

• 23:57 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mw.org to 1.19wmf1
• 23:53 logmsgbot: reedy synchronized php-1.19/includes/OutputPage.php 'r111599'
• 23:52 logmsgbot: reedy synchronized php-1.19/resources/mediawiki/mediawiki.js 'r111599'
• 23:47 pgehres: re-enabled fundraising queue consumption after exorcism and rain dance
• 23:46 binasher: running a slow staggered restart of lsearchd
• 23:06 binasher: updated /etc/lsearch.conf:Rsync.path to "/usr/local/bin/rsync-no-pagecache" on all search nodes
• 23:06 binasher: installed pagecache-management on all search nodes
• 23:01 pgehres: disabling fundraising queue consumption on Aluminium due to jenkins build failure
• 20:56 logmsgbot: reedy synchronizing Wikimedia installation... : Rebuilt message lists and ExtensionMessages
• 20:50 logmsgbot: reedy synchronizing Wikimedia installation... : Pushing r111581 and making sure everything on the cluster is upto date
• 19:57 RoanKattouw: I meant l10n_cache* of course
• 19:54 RoanKattouw: Deleting /tmp/mw-cache-*/l10nupdate-* on the image scalers
• 19:49 logmsgbot: catrope synchronized php-1.18/includes/Cdb_PHP.php 'Live hack for logging filenames in CDB errors'
• 19:44 RoanKattouw: Syncing l10nupdate files for 1.19
• 19:27 RoanKattouw: Manually running the LU script for 1.19
• 19:25 RoanKattouw: Fixed perms for /h/w/c/php-1.19/cache/l10n on fenari
• 19:21 logmsgbot: LocalisationUpdate failed (1.19) at Wed Feb 15 19:21:54 UTC 2012
• 19:21 logmsgbot: LocalisationUpdate completed (1.18) at Wed Feb 15 19:21:53 UTC 2012
• 19:17 RoanKattouw: Let's try that again: copying /wd/Wikimania\ Edited to /a on cadmium
• 19:14 RoanKattouw: Aborted copy operation on cadmium, data won't fit
• 19:13 RoanKattouw: Copying all Wikimania files from the removable HD to cadmium's HD
• 19:05 RoanKattouw: Rerunning l10nupdate by hand to hopefully fix CDB problems
• 19:02 RoanKattouw: Fixing permissions for php-1.18/cache/l10n on all apaches as root
• 18:56 RoanKattouw: Running sync-l10nupdate by hand to see what kind of perms errors I get, first for 1.18 then for 1.19
• 17:01 RobH: cadmium setup for wikimania video transcoding
• 16:37 RobH: forgot to log, carbon resumed service normally
• 16:23 RobH: carbon halted, allows login and freezes on password entry, rebooting
• 15:53 RobH: os install on cadmium
• 15:49 RobH: updating dns for cadmium
• 15:06 logmsgbot: reedy synchronized php-1.19/extensions/SpamBlacklist/SpamBlacklistHooks.php 'r111543'
• 13:12 logmsgbot: hashar synchronized wmf-config/codereview.php
• 11:32 mutante: sync-apache / graceful not logged anymore by logmsgbot ?
• 10:06 mutante: office.wm now forces https (in a less broken way;) (remnant.conf)
• 09:42 mutante: made a new change to remnant.conf and synced apaches in a fresh attempt to fix office.wm redirect
• 08:59 Nirvanchik: test
• 08:13 mutante: all search boxes had /home: Stale NFS file handle.. remounting
• 08:03 mutante: remounted /home on search6, started lsearchd
• 02:37 Tim: on kaulen: re-enabled jsonrpc.cgi and reduced MaxClients from 500 to 100
• 02:18 logmsgbot: LocalisationUpdate failed (1.19) at Wed Feb 15 02:18:04 UTC 2012
• 02:18 logmsgbot: LocalisationUpdate completed (1.18) at Wed Feb 15 02:18:04 UTC 2012
• 01:18 logmsgbot: reedy synchronized php-1.19/extensions/Gadgets/Gadgets_body.php
• 01:09 logmsgbot: reedy synchronized php-1.19/extensions/Gadgets/Gadgets_body.php
• 00:38 maplebed: deployed new thumb_handler.php with ETag header added in to ms5
• 00:11 logmsgbot: reedy synchronized php-1.19/extensions/Contest/Contest.php 'revert live hack'

February 14

• 23:52 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php 'Enable FR like dewiki on test2wiki'
• 23:43 LeslieCarr: allowed ipv6 pim on edge routers in the US
• 23:25 logmsgbot: reedy synchronized php-1.18/extensions/ArticleFeedbackv5/modules/jquery.articleFeedbackv5/jquery.articleFeedbackv5.js 'r111506'
• 23:21 LeslieCarr: modifying "martian" blocks on cr2-eqiad to allow newly allocated ip ranges
• 23:19 logmsgbot: reedy synchronized php-1.18/extensions/ArticleFeedbackv5/modules/jquery.articleFeedbackv5/jquery.articleFeedbackv5.js 'r111506'
• 23:18 logmsgbot: reedy synchronized php-1.19/includes 'r111486'
• 22:53 logmsgbot: reedy synchronized php-1.19/includes 'r111486'
• 22:46 logmsgbot: reedy ran sync-common-all
• 22:45 maplebed: deployed squid config to upload to send all thumbnail traffic to ms5 instead of swift
• 22:33 Reedy: running ddsh -F5 -cM -g mediawiki-installation 'sudo -u mwdeploy rm -rf /usr/local/apache/common-local/php-1.17'
• 22:26 Reedy: Removing php-1.17 from fenari
• 22:10 binasher: restarted the 1.19 schema migration script - it's going to hit the just rotated s3 (db34), s2 (db30), s7 (db16), and s4 (db31) ex-masters before resuming s5 (db55) and all s6/s1 slaves
• 22:06 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Setting wmfUseRevSha1Columns'
• 21:55 Ryan_Lane: restarting ircecho on spence
• 21:48 binasher: new s4 master pos - MASTER_LOG_FILE='db22-bin.000030', MASTER_LOG_POS=964208442
• 21:48 logmsgbot: asher synchronized wmf-config/db.php 'setting s4 to writeable, new master is db22, db31 still out'
• 21:47 logmsgbot: asher synchronized wmf-config/db.php 'setting s4 to read-only, switching master to db22'
• 21:39 logmsgbot: asher synchronized wmf-config/db.php 'returning db30 to s3, going to wait til after schema migrations to upgrade to lucid/new-mysql'
• 21:36 binasher: new s2-master pos - MASTER_LOG_FILE='db13-bin.000278', MASTER_LOG_POS=599752853
• 21:36 logmsgbot: asher synchronized wmf-config/db.php 'setting s2 to writeable, db13 is new master'
• 21:34 logmsgbot: asher synchronized wmf-config/db.php 'setting s2 to read-only, switching master to db13'
• 21:29 logmsgbot: asher synchronized wmf-config/db.php 'db34 upgrading, returning to s3'
• 21:15 binasher: new s3 master position - MASTER_LOG_FILE='db39-bin.000550', MASTER_LOG_POS=63238699
• 21:15 logmsgbot: asher synchronized wmf-config/db.php 'returning s3 to writeable, db39 is the new master'
• 21:13 logmsgbot: asher synchronized wmf-config/db.php 'setting s3 to read-only, switching master to db39'
• 21:02 logmsgbot: asher synchronized wmf-config/db.php 'returning db16 after upgrading mysql'
• 20:58 binasher: new s7 repl position - MASTER_LOG_FILE='db37-bin.000285', MASTER_LOG_POS=865712092
• 20:52 logmsgbot: asher synchronized wmf-config/db.php 'setting s7 to read-only for master swap, db37 to be new master, db16 still out'
• 20:50 logmsgbot: asher synchronized wmf-config/db.php 'setting s7 to read-only for master swap, db37 to be new master'
• 20:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Fix hewiki namespace talk typo'
• 20:41 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'fix some mrwikisource aliases'
• 20:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'fix some mrwikisource aliases'
• 20:33 logmsgbot: hashar synchronized php-1.19/maintenance/purgeList.php 'r111480 : enable purge of HTTPS URLs'
• 20:29 mutante: purged https://office link, using modified purgeList.php that accepts https urls, thanks hashar
• 20:27 logmsgbot: hashar synchronized php-1.18/maintenance/purgeList.php 'r111480 : enable purge of HTTPS URLs'
• 20:13 Ryan_Lane: built two raid6 arrays per labstore host. raid sets are initializing.
• 19:53 logmsgbot: reedy synchronized wmf-config/db.php 'Bring db46 back in'
• 19:21 RoanKattouw: Ran Varnish purge for 'office
• 19:19 mutante: used purgeList.php on office.wm URLs, but it appears to be in varnish cache (broken redirect)
• 19:06 AaronSchulz: gracefulled apaches to deal with APC corruption
• 18:57 mutante: reverting the (circular) office redirect, syncing..
• 18:49 mutante: running sync-apache to fix office redirect
• 18:39 logmsgbot: reedy synchronized wmf-config/db.php 'Comment out db46 from s6 due to really high lag from db schema updates'
• 16:02 mutante: spence lost /home, mount was "Stale NFS file handle", causing outage of stats.wikimedia.org, fixed by remounting
• 15:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34378 - Rename namespaces on mr.wikisource.org'
• 15:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34342 - Create a new books namespace on he.wiki'
• 14:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki to 1.19
• 09:29 logmsgbot: tstarling synchronized php-1.18/extensions/cldr/LanguageNames.body.php 'r111453'
• 07:19 apergos: symlinkd wikidiff2.so to php_wikidiff2.so on searchidx2
• 03:17 logmsgbot: reedy synchronized wmf-config/ExtensionMessages-1.19.php 'Fix fr message file locations'
• 02:48 logmsgbot: reedy synchronizing Wikimedia installation... : Rebuild messages
• 02:48 logmsgbot: tstarling synchronized wmf-config/AdminSettings.php 'remove $wgUseRootUser and $wgUseNormalUser, broken since 1.17 and 1.16 respectively'
• 02:41 logmsgbot: reedy synchronized php-1.19/extensions/Contest/Contest.php 'Comment out stupid die for the moment'
• 02:38 logmsgbot: reedy synchronizing Wikimedia installation... : Rebuild messages
• 02:38 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 're-enabling the collection extension'
• 02:35 logmsgbot: reedy synchronizing Wikimedia installation... : Rebuild messages
• 02:26 logmsgbot: reedy synchronized php-1.19/extensions/VisualEditor
• 02:25 logmsgbot: reedy synchronized php-1.19/extensions/FundraiserLandingPage/
• 02:25 logmsgbot: LocalisationUpdate failed (1.19) at Tue Feb 14 02:25:33 UTC 2012
• 02:25 logmsgbot: LocalisationUpdate completed (1.18) at Tue Feb 14 02:25:32 UTC 2012
• 02:19 logmsgbot: reedy synchronized wmf-config/ExtensionMessages-1.19.php 'Remove variablepage'
• 01:54 Reedy: Make that ddsh -F5
• 01:53 Ryan_Lane: when rebooting hume I also applied security updates
• 01:52 Tim: started indexer on searchidx2 with /home/rainman/scripts/search-restart-indexer per docs
• 01:52 Reedy: running ddsh -F30 -cM -g mediawiki-installation /usr/bin/sync-common
• 01:47 Tim: rebooting srv193
• 01:45 Tim: on searchidx2: doing apt-get upgrade and rebooting
• 01:44 Ryan_Lane: rebooting hume
• 01:28 binasher: resuming 1.19 schema migrations after fenari reboot (on first s4 commons slave, db22)
• 01:19 Tim: rebooting fenari for kernel upgrades
• 01:14 Tim: doing apt-get upgrade on fenari
• 01:12 Tim: rebooted fenari to fix stale NFS file handle
• 00:57 LeslieCarr: rebooted nfs1 as it was unresponsive on console and via IP
• 00:37 Reedy: killed /usr/local/apache/common/php-1.19 from apaches

February 13

• 23:29 logmsgbot: reedy ran sync-common-all
• 23:13 logmsgbot: reedy synchronized wmf-config/abusefilter.php 'Hard code wgAbuseFilterStyleVersion as it went away in 1.19'
• 23:05 logmsgbot: reedy synchronizing Wikimedia installation... : For good measure
• 23:01 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Switch test2wiki to 1.19wmf1
• 22:57 Tim: increased concurrency on the image scalers from 10 to 15
• 22:33 logmsgbot: reedy synchronized php-1.19/includes/api/ApiWatch.php 'r111422'
• 22:29 Tim: on pdf1: killed a convert process that had been running since Jan 6
• 22:20 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'disabling the collection extension due to image scaler overload'
• 21:28 LeslieCarr: reloading brewster
• 21:17 LeslieCarr: copied a resolv.conf to brewster, apt-get upgrade on brewster and restarted lighttpd and squid on brewster
• 20:48 Ryan_Lane: rebooting brewster
• 17:37 logmsgbot: reedy synchronizing Wikimedia installation... : Rebuilt trusted-xff.cdb
• 17:12 mutante: mailman: deleting test-list
• 16:31 logmsgbot: reedy synchronized php-1.19/extensions/OggHandler/ 'r111385'
• 16:31 logmsgbot: reedy synchronized php-1.19/extensions/PagedTiffHandler/ 'r111385'
• 16:20 logmsgbot: reedy synchronized php-1.19/includes/ 'r111382'
• 16:19 logmsgbot: reedy synchronized php-1.19/extensions/CategoryTree/ 'r111382'
• 15:11 logmsgbot: reedy synchronized php-1.18//includes/ 'Bringing across 1.18wmf1 livehacks'
• 15:03 logmsgbot: reedy synchronizing Wikimedia installation... : Reverting roans live hacks for bug 31576
• 14:29 logmsgbot: reedy synchronized wikimedia.dblist 'Fix double bewikimedia'
• 14:28 logmsgbot: reedy synchronized s3.dblist 'Fix double bewikimedia'
• 14:28 logmsgbot: reedy synchronized pmtpa.dblist 'Fix double bewikimedia'
• 14:28 logmsgbot: reedy synchronized all.dblist 'Fix double bewikimedia'
• 14:27 logmsgbot: reedy synchronized 1.17.dblist '1.17-phase1.dblist 1.17-phase2.dblist all.dblist big.dblist closed.dblist deleted.dblist fishbowl.dblist flaggedrevs.dblist news.dblist new_wiktionaries.dblist pmtpa.dblist pmtpa-dump1.dblist pmtpa-dump2.dblist pmtpa-dump3.dblist private.dblist readonly.dblist s1.dblist s2.dblist s2-fixed.dblist s3.dblist s3-fixed.dblist s4.dblist s5.dblist s6.dblist s7.dblist small.dblist special.dblist switchover-ju
• 13:44 logmsgbot: reedy synchronized php-1.19/extensions/MobileFrontend
• 02:32 Tim: on kaulen: increased MaxClients to 500 to better deal with the connection flood
• 02:23 Tim: bugzilla is mostly working now, although it's very slow. The DDoS requests are blocked after connection setup using <Location>
• 02:21 Tim: on kaulen: restored MaxClients
• 02:17 logmsgbot: LocalisationUpdate completed (1.18) at Mon Feb 13 02:17:50 UTC 2012
• 01:46 Tim: temporarily moved bugzilla to port 444 until the connection flood (~1k req/s) subsides
• 01:15 Tim: started apache with MaxClients=30
• 00:59 Tim: after kaulen came back up, it was immediately overloaded with jsonrpc.cgi. Stopped apache.
• 00:54 Tim: kaulen is not responding on ssh, web down, rebooting

February 12

• 12:09 mark: Killed lsearchd processes on search8, restarted
• 12:07 mark: Rebalanced mw API app servers from load 120 to 150 in pybal list
• 10:08 mark: Increased MaxClients to 100 on API apaches in Puppet
• 09:45 mark: Restricted only opensearch API requests to the API squids
• 09:43 mark: Restricted only opensearch API requests to the API backend apaches, other API requests now hit the main mediawiki cluster
• 08:44 mark: maximum_forwards change deployed to all squids
• 08:42 mark: Set maximum_forwards 2 in squid.conf, deployed to the API squids only so far, rest is pending
• 07:52 binasher: restarted lsearchd on search{3,4,9}
• 02:19 logmsgbot: LocalisationUpdate completed (1.18) at Sun Feb 12 02:19:17 UTC 2012

February 11

• 20:31 apergos: restarted lightty on dataset2
• 17:28 RobH: manual test of each affected service complete, db9 fully online.
• 17:26 RobH: db9 moved, all systems online
• 17:08 RobH: db9 shutting down to move racks, offline during this includes: blogs, bugzilla, racktables, rt, survey, etherpad, observium
• 02:18 logmsgbot: LocalisationUpdate completed (1.18) at Sat Feb 11 02:18:36 UTC 2012
• 00:17 logmsgbot: reedy synchronizing Wikimedia installation... :

February 10

• 22:17 LeslieCarr: fixing the labs apache2 puppet groups
• 21:48 RobH: memory in cp1017 wasnt properly seated as far as i can tell, if it doesnt mess up again it should be ok.
• 21:41 RobH: cp1017 being tested for bad memory
• 21:36 RobH: powercycling msw-a2-eqiad resolves all mgmt issues in rack
• 21:34 RobH: powercycling msw-a1-eqiad.
• 21:29 RobH: db1001 rebooting, locked up
• 20:53 RobH: updating dns for new db hosts
• 19:59 Reedy: Checking out 1.19wmf1 to /tmp on fenari
• 19:12 RobH: oxygen setup and installed per rt2343, still needs puppet runs and full deployment per rt 2430
• 17:58 RobH: updating dns for oxygen internal ip
• 17:21 mutante: labs logging is broken
• 17:14 RobH: oxygen offline for hard disk upgrade to replace locke
• 16:50 mutante: running sync-apache, trying to redirect office.wm to https
• 16:07 mark: Rebalanced appserver load balancing by giving the new mw* pmtpa app servers weight 150 in the pybal server list
• 15:17 mark: Turned on KeepAlive on apaches for better miss service times from eqiad
• 13:42 mark: Configured cp1001 and cp1020 to contact backend servers directly instead of via pmtpa squids
• 12:02 mark: Decommissioning sq38, sq46 and sq47 in squid configurator
• 11:50 mark: Making cp1001-1005 API squids
• 05:08 maplebed: deployed squid config to uploads to send 100% of thumbnail traffic to swift
• 02:49 maplebed: deploying fix for & bug with swift (files with an & in the name wouldn't load properly)
• 02:18 logmsgbot: LocalisationUpdate completed (1.18) at Fri Feb 10 02:18:37 UTC 2012
• 00:22 LeslieCarr: increased nagios max concurrent checks on spence and lowered the interval between processing them
• 00:20 maplebed: deployed squid config to upload squids rolling thumbnails back to 75% handled by swift to test the & bug

February 9

• 22:22 Jeff_Green: adding community-analytics.wikimedia.org to DNS
• 21:36 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Increase wgCodeReviewMaxDiffPaths'
• 20:04 binasher: started 1.19 schema migrations on seconadry db's (mwscript upgrade-1.19wmf1-1.php --secondary)
• 19:32 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Increase wgCodeReviewMaxDiffPaths'
• 17:31 maplebed: deployed squid config to upload squids sending 100% of all thumbnail traffic to swift
• 16:31 mark: Sending ALL non-european wikipedia traffic to eqiad text squids
• 16:17 mark: Added Brazil traffic to eqiad text squids
• 15:49 mark: Sending some Asian wikipedia traffic to wikipedia-lb.eqiad
• 15:13 mark: Sending Canadian wikipedia traffic to wikipedia-lb.eqiad
• 14:27 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34183 - add alias redirection in ml wikisource'
• 14:23 mark: Redirecting {wikiquote,wikisource,wikiversity,wiktionary}-lb.pmtpa traffic to .eqiad (geodns)
• 14:17 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34253 - Change site logo for lb.wiktionary'
• 14:11 mark: Redirecting {foundation,wikibooks,wikinews}-lb.pmtpa traffic to .eqiad (geodns)
• 14:10 logmsgbot: hashar synchronized docroot/mediawiki/xml 'update html pages for 0.1 and 0.2 schemes'
• 14:06 logmsgbot: hashar synchronized docroot/mediawiki/xml/index.html 'Give a home page to our XML namespaces homepage'
• 13:52 logmsgbot: hashar synchronized docroot/mediawiki/xml
• 13:45 mark: Redirecting wikimedia-lb.pmtpa traffic to wikimedia-lb.eqiad (geodns)
• 13:28 mark: Redirecting mediawiki-lb.pmtpa traffic to mediawiki-lb.eqiad (geodns)
• 12:49 apergos: around 11:05 UTC, increased /proc/sys/vm/min_free_kbytes from 16259 to 262144 to check impact on network alloc issues, on dataset1001
• 11:56 mark_: Power cycled cp1017
• 11:50 mark_: Changed topology of eqiad text squids to request from pmtpa (similar to esams)
• 09:29 mark_: Reactivated term selected-paths in policy-statement BGP_transit_in on cr2-eqiad, making path 14907 3257 1299 43821 active again
• 02:26 logmsgbot: LocalisationUpdate completed (1.18) at Thu Feb 9 02:26:49 UTC 2012
• 00:35 logmsgbot: asher synchronized wmf-config/db.php 'returning db38 to s1 post schema migration testing'

February 8

• 23:52 logmsgbot: asher synchronized wmf-config/db.php 'returning db11'
• 23:49 logmsgbot: asher synchronized wmf-config/db.php 'pulling db11'
• 22:01 maplebed: deployed updated to upload squid.conf to send 75% of all thumbnail traffic to swift
• 17:37 maplebed: deployed new squid config to upload to direct 50% of thumbnail traffic to swift
• 17:21 apergos: reboot of dataset1001 for testing.
• 14:21 mutante: srv189 - shut down again, PCI Express Error, -> RT 2413 created
• 14:15 mutante: powercycling srv189
• 13:09 RoanKattouw: Batch-deleting 9,760 redirects on cswiktionary, requested by Danny B
• 02:26 logmsgbot: LocalisationUpdate completed (1.18) at Wed Feb 8 02:26:55 UTC 2012

February 7

• 22:11 maplebed: increasing traffic to swift from 12.5% to 25%
• 21:19 apergos: reboot dataset1001, new kernel 2.6.38, should deal with kswap bug we just ran into under heavy i/o load, need to test
• 20:24 mark: Uncommented sq63 in squid text-settings.php, please DO NOT COMMENT DOWN SQUIDS UNLESS DECOMMISSIONING THEM PERMANENTLY
• 20:19 RobH: dns update for db59/60 mgmt
• 20:07 mark: Added manutius's IP to the stats ACL in squid.conf
• 19:47 Jeff_Green: email aliases adjusted for merchandise@, wikishop@
• 19:09 mark: Deployed torrus on server manutius, migrated DNS over
• 18:48 maplebed: deployed squid config to send 1/8th of all thumbnail traffic to swift
• 18:32 K4-713: Updated the payments cluster to r110857
• 18:27 RobH: updating dns for mgmt on db59-60 and labstore1-4
• 17:16 cmjohnson1: power cable swap cr2-pmtpa complete
• 17:12 cmjohnson1: swapping power cables for cr2-pmtpa
• 15:36 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Extra namespaces for mwikisource bug 33907'
• 15:35 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Extra namespaces for mwikisource bug 33907'
• 15:30 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'set project namespace for mrwikisource'
• 13:25 logmsgbot: reedy synchronized php-1.18/extensions/ShortUrl/ 'r110845'
• 12:43 logmsgbot: reedy synchronized php-1.18/extensions/ShortUrl/ 'r110841'
• 02:58 logmsgbot: asher synchronized wmf-config/db.php 'adding db35 back'
• 02:25 logmsgbot: LocalisationUpdate completed (1.18) at Tue Feb 7 02:25:17 UTC 2012
• 00:04 binasher: streaming hotbackup of db1021 to db35
• 00:02 binasher: shutting down mysql on db35, rebuild fail
• 00:00 logmsgbot: asher synchronized wmf-config/db.php 'pulling db35'

February 6

• 22:10 binasher: pulled db38 from enwiki, running normal "alter table revision add rev_sha1" and on db1043, the pt-online-schema-change equiv (with --chunk-size=1000, --sleep=0.1) to compare timing
• 22:03 logmsgbot: asher synchronized wmf-config/db.php 'pulling db38 from enwiki for revision alter timing'
• 21:38 maplebed: initial deploy of swift to serve thumbnails is complete
• 21:27 maplebed: deployed new squid.conf to enable swift for all thumbs with /a/a2 in the URL
• 20:04 binasher: running mk-slave-prefetch on db1018 which was down for 5 days to see if it can catch up
• 19:28 maplebed: swift deploy aborted due to squid config issues
• 18:42 maplebed: deploying squid config change to put swift in service for all thumbnails with /a/a2 in the URL
• 16:49 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34043 - Change logo at Indonesian Wikibooks'
• 16:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34009 - Translation of project namespace and site-name for ta.wikiquote'
• 16:22 Reedy: Updated user_former_groups.ufg_group to 32 characters
• 16:07 logmsgbot: nikerabbit synchronized wmf-config/CommonSettings.php 'Do not add translation reviewers group'
• 15:58 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'add r'
• 15:57 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'unset( $wgGroupPermissions[translate-proof] )'
• 15:55 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34124 - Create a Project namespace on Russian Wikipedia'
• 15:19 Nikerabbit: that was Bug 34213 - Enabling of Extension:Translate on the Wikimedia Incubator
• 15:18 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php
• 15:14 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34219 - Disable Babel autocreation of categories at the Portuguese Wiktionary'
• 14:59 Nikerabbit: creating Translate db tables for incubator
• 11:32 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Fixing vepwiki logo - bug 34222'
• 11:28 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Fixing vepwiki logo - bug 34222'
• 11:04 logmsgbot: nikerabbit synchronized php-1.18/extensions/Translate/ 'i18ndeploy r110738 Translate fixes'
• 06:29 binasher: rotated all eqiad + anayltic enwiki slaves to replicate from db1017 after db1001 hardware failure
• 02:32 logmsgbot: LocalisationUpdate completed (1.18) at Mon Feb 6 02:32:41 UTC 2012

February 5

• 18:29 logmsgbot: reedy synchronized php-1.18/includes/api
• 16:58 maplebed: restarted lsearchd on search6
• 16:38 binasher: running alter table add column on db1017 enwiki.revision to benchmark
• 16:34 logmsgbot: asher synchronized wmf-config/db.php 'returning db35 to service'
• 14:23 logmsgbot: reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
• 13:10 mutante: stopped and started lsearchd once again on search6
• 12:51 mutante: restarted lsearchd on search6
• 02:25 logmsgbot: LocalisationUpdate completed (1.18) at Sun Feb 5 02:25:51 UTC 2012
• 02:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'touch'

February 4

• 21:30 pgehres: switched cc-status back to enabled on donate.wikimedia.org per nagios and GC email
• 21:25 pgehres: switched cc-status on donate.wikimedia.org to disable per GlobalCollect outage notice
• 02:25 logmsgbot: LocalisationUpdate completed (1.18) at Sat Feb 4 02:25:24 UTC 2012
• 00:02 LeslieCarr: applying loopback filter on cr2-pmtpa

February 3

• 23:59 LeslieCarr: applying loopback filter on cr1-sdtpa
• 23:21 binasher: timing the same operation as a normal alter on db1005. expect db lag to get backed up by hours
• 23:16 binasher: testing and timing pt-online-schema change to dewiki.revision on db1021 (not in rotation)
• 23:07 LeslieCarr: applying loopback filter on cr2-eqiad
• 23:06 binasher: upgrading percona-toolkit to 2.02 on all coredbs
• 22:43 logmsgbot: asher synchronized wmf-config/db.php 'returning db33, 39, 46 to prod'
• 22:39 binasher: db35 had an iblogfile size inconsistent with other s5 hosts. streaming a hotbackup of db1034 to db35
• 22:23 binasher: rebooted db35, db39
• 21:55 logmsgbot: asher synchronized wmf-config/db.php 'pulling db35, 39, 46 for upgrades'
• 20:52 K4-713: updated production civicrm to r1295
• 20:45 logmsgbot: asher synchronized wmf-config/db.php 'adding back dbs 13,18,25'
• 20:32 binasher: upgraded mysql on dbs 13,18,25,33
• 20:17 logmsgbot: reedy synchronized php-1.18/extensions/SpamBlacklist/SpamBlacklist_body.php 'r110682'
• 19:23 logmsgbot: asher synchronized wmf-config/db.php 'pulling dbs 13,18,25,26 for upgrades'
• 19:11 RobH: manutius installed and ready for use
• 17:26 RobH: updated dns for manutius.mgmt
• 17:15 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'touch'
• 17:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'touch'
• 16:08 RobH: db41 being reinstalled, appears down but logging to be safe
• 15:20 mark: Around 14:50 UTC, removed the 3 remaining esams upload squids in the knsq8-15 range from the config. This made ms5 unhappy.
• 15:13 logmsgbot: reedy synchronized wmf-config/db.php 'Add comment that db40 is parsercache'
• 13:53 mutante: resetting stats on new wikis per bz 34184: updateArticleCount.php vepwiki --update; updateArticleCount.php pnbwiktionary --update
• 13:42 mark: Disabled knsq1-15 in PyBal, preparing for decommissioning
• 03:53 maplebed: moved all the individual puppet files out of place, stopped nagios, and re-ran puppet (at now minus 1.5hrs)
• 02:24 logmsgbot: LocalisationUpdate completed (1.18) at Fri Feb 3 02:24:54 UTC 2012
• 00:57 K4-713: re-enabled the donations queue consumer via Jenkins
• 00:42 K4-713: updated production civicrm to r1293
• 00:23 logmsgbot: asher synchronized wmf-config/db.php 'moving watchlist/recentchanges back to db12, returning db24 to s2'
• 00:09 K4-713: Disabled donations queue consumption on aluminium

February 2

• 23:51 K4-713: updated production civicrm to r1291
• 23:44 binasher: db12 back up with lucid + current mysql
• 23:32 binasher: rebooting db12
• 23:08 logmsgbot: asher synchronized wmf-config/db.php 'pulling db12 from enwiki, temporarily moving watchlist/recentchanges to db54'
• 23:02 pgehres: K4-713 synchronized production CiviCRM to r1288 on Aluminium
• 22:59 binasher: db24 upgraded to lucid and current mysql build
• 22:52 binasher: rebooted db24
• 22:44 logmsgbot: reedy synchronized wmf-config/ 'Disable VariablePage completely'
• 22:26 binasher: pulled db24 from s2, preparing to upgrade to lucid
• 22:19 logmsgbot: asher synchronized wmf-config/db.php 'pulling db24 from s2 for upgrade'
• 21:37 apergos: started rsync from dataset2 to dataset1001 in screen session as root on dataset1001
• 21:07 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Drop FundraiserPortal config'
• 21:07 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Drop FundraiserPortal config'
• 21:06 RobH: dataset1001 is alive, mostly
• 19:15 logmsgbot: asher synchronized wmf-config/db.php 'raising db55 weight'
• 19:08 logmsgbot: asher synchronized wmf-config/db.php 'add db55 - new s5 slave'
• 18:06 notpeter: doing initial run of puppet on cp1001-1020
• 17:33 notpeter: reimaging cp1002 and imaging cp1001 and cp1003-1020
• 16:06 cmjohnson1: disk 15 swap complete on db11
• 16:05 cmjohnson1: replacing disk 15 on db11
• 15:55 mark: Running apt-get update && apt-get dist-upgrade && reboot on lvs1
• 15:40 mark: Running apt-get update && apt-get dist-upgrade && reboot on lvs2
• 15:10 logmsgbot: reedy synchronized php-1.18/extensions/CodeReview/api/ 'r110574'
• 14:21 logmsgbot: hashar synchronized php-1.18/includes/UserMailer.php 'work around bug 34158'
• 14:19 logmsgbot: catrope synchronized php-1.18/extensions/LocalisationUpdate/LocalisationUpdate.class.php 'r110570'
• 14:10 RoanKattouw: Finally fixed ownership of cache/l10n on scalers , sync-l10nupdate only throws the expected errors, no more perms errors on the scalers
• 14:09 RoanKattouw: Scalers now have disk space available because php-1.17-test is gone
• 13:59 logmsgbot: catrope synchronizing Wikimedia installation... : Deleted php-1.17-test on fenari, running scap to delete it on the Apaches as well
• 13:49 RoanKattouw: Deleting /home/wikipedia/common/php-1.17-test , has been unused for a long time
• 13:45 RoanKattouw: Deleting /tmp/mw-cache-1.17 on srv219 and srv223
• 13:44 RoanKattouw: srv219-224 have a full disk according to rsync
• 13:38 RoanKattouw: Fixing ownership of /usr/local/apache/common-local/php-1.18/cache/l10n on srv191, srv199, srv219-224
• 13:35 RoanKattouw: Running sync-l10nupdate again to investigate rsync errosr
• 13:34 logmsgbot: LocalisationUpdate completed (1.18) at Thu Feb 2 13:34:53 UTC 2012
• 13:12 RoanKattouw: Running l10nupdate by hand to hopefully fix bug 33768
• 13:11 logmsgbot: catrope synchronized php-1.18/extensions/LocalisationUpdate/LocalisationUpdate.class.php 'Deploy live-hacked version that will hopefully fix bug 33768'
• 10:00 Tim: reinserted the deleted site_stats row for plwiki
• 09:35 Tim: killing statistics queries on all s2 slave servers
• 09:34 logmsgbot: tstarling synchronized php-1.18/includes/SiteStats.php 'disabling even more'
• 09:32 Tim: on db54, killed all SiteStats queries
• 09:28 Tim: restarting all apaches
• 09:26 Tim: disabling SiteStatsInit::articles
• 09:26 logmsgbot: tstarling synchronized php-1.18/includes/SiteStats.php
• 02:06 logmsgbot: LocalisationUpdate completed (1.18) at Thu Feb 2 02:06:48 UTC 2012
• 01:22 AaronSchulz: running "mwscriptwikiset purgeDeletedFiles.php all.dblist --starttime=20120126000000" in a screen on fenari

February 1

• 23:20 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Enabled swift thumbnail purge code'
• 23:12 logmsgbot: aaron synchronized wmf-config/swift.php 'actually register the hook handler'
• 22:43 logmsgbot: aaron synchronized php-1.18/includes/filerepo/LocalFile.php
• 22:26 binasher: streaming a hotbackup of db35 to db55 (new s5 slave)
• 22:21 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'fix timezone typo for mr'
• 22:19 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Configure rights assignments for AFTv5'
• 22:03 AaronSchulz: Enabled SwiftCloudFiles extension on all wikis, doesn't do anything yet
• 21:51 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Asia/Kolata isn't valid'
• 21:48 Jeff_Green: disabling deprecated apache, lighttpd, haproxy, squid, mysql services on loudon
• 21:38 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Add additional articles category to $wgArticleFeedbackv5DashboardCategory'
• 21:36 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'timezone config for new sties'
• 21:30 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34120 - Enable translate extension on Wikimania 2012 wiki'
• 21:29 Reedy: Created Translate tables on wikimania2012wiki
• 21:28 Jeff_Green: dist-upgrading and rebooting loudon
• 21:20 logmsgbot: reedy ran sync-common-all
• 21:09 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'fix timezone'
• 21:07 logmsgbot: reedy ran sync-common-all
• 20:55 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'vepwiki config'
• 20:53 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Undo temp eot change'
• 20:53 logmsgbot: reedy ran sync-common-all
• 20:48 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'wrong file'
• 20:46 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Temporarily enable eot uploads on amwiki'
• 20:35 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bewikimedia site config'
• 20:31 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Set $wgArticleFeedbackv5SelectedCTA = 1'
• 20:29 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'set bewikimedia to en'
• 20:21 logmsgbot: catrope synchronizing Wikimedia installation... : Deploying ArticleFeedbackv5 updates
• 20:15 logmsgbot: reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
• 19:58 RobH: labs switch ports connected per rt 1882
• 19:57 logmsgbot: reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
• 19:52 RobH: strontium.mgmt repaired per rt2352
• 19:50 logmsgbot: catrope synchronized php-1.18/cache/interwiki.cdb 'rebuilt interwiki cache'
• 19:47 logmsgbot: catrope synchronized php-1.18/cache/interwiki-pr.cdb 'rebuilt interwiki cache'
• 19:47 logmsgbot: catrope synchronized php-1.18/cache/interwiki.cdb 'rebuilt interwiki cache'
• 19:43 RobH: lab-ex4200-1 back in rack
• 19:29 logmsgbot: reedy ran sync-common-all
• 19:20 RobH: pushing apache changes for reedy
• 18:27 RobH: ganglia1002 back online ready for install
• 18:26 RobH: ganglia1002 mgmt offline per rt 2247, system was unplugged... no idea why
• 18:24 cmjohnson1: pulled drive 2 db47
• 18:15 RobH: cp1019 memory error repaired, now it is ready for OS install
• 18:14 RobH: cp1017 memory error repaired
• 17:54 RobH: updated dns for payments boxen renames in eqiad
• 17:37 RobH: cp1014 memory was improperly installed (from factory?), installed in supported configuration and system is now ready for OS install per RT2351
• 17:08 RobH: investigating errors on cp1014
• 17:04 RobH: cp1019 console redirection fixed per rt2353, ready for OS install
• 16:57 logmsgbot: reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
• 16:54 logmsgbot: reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
• 16:48 RobH: dataset1001 controller replaced
• 16:41 cmjohnson1: reseating drive2 in db47
• 16:37 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wikilove default on fawikis'
• 16:22 RobH: dataset1001 down for controller replacement
• 16:07 mark: Removed now obsolete package wikimedia-task-squid from the karmic-wikimedia and lucid-wikimedia APT repositories, and deleted in svn.wikimedia.org
• 15:45 logmsgbot: reedy synchronized php/cache/interwiki.cdb 'Try a quietened dumpInterwiki script'
• 14:45 mutante: running authdns-update to remove oldusability
• 14:30 mutante: shutting down "oldusability" linode instance
• 13:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Point wgInterwikiCache at interwiki.cdb'
• 13:41 logmsgbot: reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache copying protocol relative over interwiki.cdb'
• 13:36 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Kill wmgHTTPSExperiment'
• 13:35 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Tidy up cache epoch code'
• 13:30 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Remove some of the wmgHTTPSExperiment related conditionals'
• 13:27 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Only use the protocol relative interwiki cdb'
• 02:06 logmsgbot: LocalisationUpdate completed (1.18) at Wed Feb 1 02:06:20 UTC 2012
• 00:51 binasher: applied articlefeedback v5 schema changes to enwiki, testwiki, en_labswikimedia
• 00:33 logmsgbot: reedy synchronized php/cache/interwiki-pr.cdb 'Updating interwiki cache'
• 00:07 logmsgbot: reedy synchronized php/cache/interwiki-pr.cdb 'Updating interwiki cache'
• 00:05 logmsgbot: reedy synchronized php/cache/interwiki-pr.cdb 'Updating interwiki cache'

January 31

• 21:44 logmsgbot: awjrichards synchronized php/extensions/DonationInterface/gateway_common/us-states.i18n.php 'r110433'
• 21:44 logmsgbot: awjrichards synchronized php/extensions/DonationInterface/gateway_common/interface.i18n.php 'r110433'
• 21:43 logmsgbot: awjrichards synchronized php/extensions/DonationInterface/gateway_common/countries.i18n.php 'r110433'
• 21:41 logmsgbot: awjrichards synchronized php/extensions/DonationInterface/globalcollect_gateway/globalcollect_gateway.i18n.php 'r110433'
• 21:41 logmsgbot: awjrichards synchronized php/extensions/DonationInterface/globalcollect_gateway/globalcollect_gateway.alias.php 'r110433'
• 21:40 logmsgbot: awjrichards synchronized php/extensions/DonationInterface/payflowpro_gateway/payflowpro_gateway.i18n.php 'r110433'
• 21:39 logmsgbot: awjrichards synchronized php/extensions/DonationInterface/payflowpro_gateway/payflowpro_gateway.alias.php
• 21:23 notpeter: restarting puppet on emery
• 20:57 notpeter: temp stopping puppet on emery for testing
• 19:58 notpeter: on stafford, that is
• 19:57 notpeter: restarting puppetmaster proc as it's serving up 500s to all clients (well, 3 randomly selected ones...)
• 18:55 Ryan_Lane: restarted squid and lighttpd on brewster
• 18:36 RoanKattouw: IRC breakage postmortem: MediaWiki was configured to send UDP packets to .179 (ekrem-old) instead of .178 (ekrem)
• 18:32 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Change wgRC2UDPAddress to the new ekrem IP'
• 16:56 mutante: restarted ircd on ekrem once again because we still cant join channels .. problem remains
• 16:35 mutante: restarted IRC bot on ekrem (needs dependency to start after ircd)
• 16:30 mutante: ekrem - gets Error 500 on SERVER when running puppet
• 16:30 logmsgbot: reedy synchronized php-1.18/extensions/SpamBlacklist/SpamBlacklist_body.php 'r110401'
• 16:28 mutante: ekrem - su -c /usr/local/ircd-ratbox/bin/ircd irc
• 16:21 mutante: powercycling ekrem - mgmt just showed "Stopping web" and was frozen completely
• 16:17 RoanKattouw: ekrem suddenly died around 16:03 UTC, breaking the RC IRC feed
• 15:06 mutante: changed nameservers for wikimedia.pl per RT:2277/bugzilla:33509
• 09:28 logmsgbot: catrope synchronized php-1.18/includes/Wiki.php 'r110368'
• 09:27 logmsgbot: catrope synchronized php-1.18/includes/Exception.php 'r110368'
• 06:34 Tim: added myself to the gerrit "administrators" group
• 05:23 Tim: the segfaults didn't stop, so I'm disabling wmerrors entirely for now
• 05:13 Tim: since puppet is broken, disabled wmerrors backtrace logging by adding a separate configuration file in /etc/php5/conf.d and reloading apache
• 02:06 logmsgbot: LocalisationUpdate completed (1.18) at Tue Jan 31 02:06:09 UTC 2012

January 30

• 23:56 awjr: synchronized i18n files for DonationInterface on payments cluster to r110342
• 23:46 Ryan_Lane: moving instances from virt2 to virt1 to rebalance compute cluster
• 19:48 logmsgbot: awjrichards synchronizing Wikimedia installation... : Syncing CentralNotice to r110026 of trunk, includes important fix for 1.19 compatibility
• 19:28 logmsgbot: asher synchronized wmf-config/db.php 'raising db54 weight'
• 18:57 logmsgbot: asher synchronized wmf-config/db.php 'adding db54 to s2'
• 18:39 mutante: running authdns-update to activate be.wikimedia.org
• 18:19 logmsgbot: nikerabbit synchronized php-1.18/extensions/Narayam/resources/ext.narayam.rules.as.js 'I18ndeploy r110311 - bug 33924'
• 18:17 logmsgbot: nikerabbit synchronized php-1.18/extensions/WebFonts/resources/ext.webfonts.fontlist.js 'I18ndeploy r110311 - bug 33599'
• 18:16 logmsgbot: nikerabbit synchronized php-1.18/extensions/Translate/ 'I18ndeploy r110310 - Translate help links'
• 02:06 logmsgbot: LocalisationUpdate completed (1.18) at Mon Jan 30 02:06:42 UTC 2012

January 29

• 18:55 logmsgbot: reedy synchronized php-1.18/extensions/SiteMatrix/SiteMatrix_body.php
• 08:42 mutante: restarted lsearchd on search6
• 02:06 logmsgbot: LocalisationUpdate completed (1.18) at Sun Jan 29 02:06:09 UTC 2012

January 28

• 02:06 logmsgbot: LocalisationUpdate completed (1.18) at Sat Jan 28 02:06:56 UTC 2012
• 00:00 binasher: db54 is now replicating from db30

January 27

• 21:57 pgehres: updates complete, re-enabling queue consumption on jenkins on aluminium
• 21:42 pgehres: pausing payments queue consumption in jenkins to backup and then run some db updates
• 21:34 LeslieCarr: applying loopback filter on cr1-eqiad
• 21:28 RobH: dns update
• 20:53 ^demon: gallium: clearing /tmp yet again. Aaron claims he's fixing it now
• 19:37 RobH: reinstalling sq31
• 18:32 RobH: dns update for fluorine host
• 17:15 binasher: s2 dbs are a sad lot. streaming hotback of db1034 to db54 to build a new slave
• 14:37 Jeff_Green: dist-upgrading storage3
• 13:36 ^demon: gallium: cleaning up /tmp again, tests really need to clean up after themselves.
• 11:34 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'th wikilogos'
• 11:29 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'th wikilogos'
• 11:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33862 - Request for logo change in Tamil Wikiquote'
• 11:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33960 - Import sources for etwiki, etwikisource and etwiktionary'
• 11:07 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33864 - Flood flag on sr.wiki'
• 02:05 logmsgbot: LocalisationUpdate completed (1.18) at Fri Jan 27 02:05:24 UTC 2012
• 00:17 logmsgbot: py synchronized wmf-config/CommonSettings.php 'changing eqiad cp1001-cp1020 IPs to their new, private IPs'

January 26

• 23:58 logmsgbot: catrope synchronized php-1.18/extensions/MoodBar/ 'Update MoodBar'
• 23:56 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Adding $wgMoodbarConfig["feedbackDashboardUrl"]'
• 23:54 logmsgbot: reedy synchronized php-1.18/includes/specials/SpecialBlockList.php 'r110095'
• 23:28 mark: Deployed squid configs to all squids
• 23:26 mark: Deploying modified squid configs of modified squid config generator to text.knams
• 22:14 RobH: poking at puppet change breaking things on sockpuppet puppet runs
• 21:33 ^demon: gallium: cleared a bunch of junk from /tmp
• 21:12 Jeff_Green: upgraded storage3 mysqld from 5.1.47 to mysql-at-facebook-r3753
• 20:17 logmsgbot: asher synchronized wmf-config/db.php 'db37 back in s7'
• 20:10 Reedy: Created "spoofuser" AntiSpoof table in the central auth database
• 19:34 logmsgbot: asher synchronized wmf-config/db.php 'pulling db37 from s7 for upgrades'
• 19:24 RobH: disregard any flapping by mw1001, its my script testbed
• 18:04 RobH: forcing puppet run on srv199
• 17:45 RobH: shutting down srv199 for bios tinkering by chris
• 02:37 RoanKattouw: Started the udp2log process for the AFT logger manually on emery
• 02:06 logmsgbot: LocalisationUpdate completed (1.18) at Thu Jan 26 02:06:22 UTC 2012
• 01:01 logmsgbot: asher synchronized wmf-config/db.php 'adding db32 to s1 at low weight, new enwiki snapshot host'
• 00:33 logmsgbot: asher synchronized wmf-config/db.php 'returning db18, now replicating heartbeat db'
• 00:28 logmsgbot: asher synchronized wmf-config/db.php 'temporarily pulling db18'
• 00:11 pgehres: re-enabling recurring donation module and processing in CiviCRM
• 00:04 logmsgbot: asher synchronized wmf-config/db.php 'adding db26 to s7'
• 00:03 RobH: shutting down srv151-srv186 per RT 2318 (confirmed not in pybal pools for apache or api)

January 25

• 23:59 binasher: removed old external store apaches from pybal config
• 23:50 logmsgbot: asher synchronized wmf-config/db.php 'pulling db26'
• 23:49 logmsgbot: asher synchronized wmf-config/db.php 'adding db26 to s7'
• 23:35 pgehres: re-enabled queue consumption for payments through Jenkins
• 23:35 pgehres: awjr synchronized CiviCRM on aluminium to r1211
• 23:05 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Checking if wmgDisplayFeedsInSidebar === false rather than true, since it defaults to true in the install file'
• 22:50 logmsgbot: awjrichards synchronizing Wikimedia installation... : Enabling FeaturedFeeds everywhere
• 22:41 RobH: updating dns for bellin/blondel db9/10 replacements
• 22:32 logmsgbot: awjrichards synchronizing Wikimedia installation... : Dark-deploying FeaturedFeeds
• 22:07 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'fixed spelling mistake fore FeaturedFeeds configuration'
• 22:05 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Setting up FeaturedFeeds config; disabled by default'
• 22:02 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Setting up FeaturedFeeds config; disabled by default'
• 21:54 LeslieCarr: deactivated selected-paths policy-statement on cr1-eqiad and cr2-eqiad
• 21:23 Tim: on srv197: compiled and installed a local version of wmerrors for segfault investigation
• 21:22 binasher: bits caches: running varnish param.set thread_pool_min, thread_pool_max, where min = 15000 / cores / 4 and max = 15000 / cores
• 20:57 binasher: running "varnishadm param.set thread_pool_max 1875" on mobile varnish servers
• 20:05 Tim: on srv197: temporarily disabled puppet and enabled core dumps in apache2.conf for segfault flood investigation
• 19:57 Jeff_Green: running dist-upgrade on payments* and silicon
• 19:44 Tim: updating TrustedXFF host list using fenari
• 19:29 RobH: dns update go!
• 18:50 LeslieCarr: restarted varnish on niobium
• 18:49 RoanKattouw: Restarted morebots
• 13:33 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Change IP address for bnwiki account creation throttle per bug 33900'
• 11:43 apergos: formey oom (I guess), unresponsive from mgmt console, powercycling.
• 06:07 pgehres: disabled queue consumption of payments in jenkins until stuck message can be removed from queue
• 05:49 pgehres: Disabling the processing of recurring payments in CiviCRM until we can add the appropriate payment_method to the queue msgs
• 02:06 logmsgbot: LocalisationUpdate completed (1.18) at Wed Jan 25 02:06:17 UTC 2012
• 00:57 binasher: streaming hotbackup of db53 to db32
• 00:52 binasher: shutting down mysql on db32, going to reconfigure with lvm and reslave
• 00:45 logmsgbot: asher synchronized wmf-config/db.php 'pulling db32 - this will be the new enwiki pmtpa snapshot host'
• 00:30 binasher: streaming hotbackup of db37 to db26, preparing to reprovision db26 in s7

January 24

• 23:30 pgehres: testing
• 23:28 awjr: Syncing prod CiviCRM on aluminium to r1209
• 22:39 mark: Disabled quota support on sanger's IMAP server to make Dovecot work again
• 22:23 mark: Sanger is upgraded to lucid
• 21:49 RobH: ms-be1 is online! MAN WE ARE AWESOME
• 21:20 mark: Starting dist-upgrade of sanger
• 20:50 binasher: pulled db26, rebooting and re-imaging with lucid
• 20:48 logmsgbot: asher synchronized wmf-config/db.php 'pulling db26 from s1 to reimage'
• 20:31 cmjohnson1: restarting ms4 for memory testing
• 20:19 notpeter: spinning up db54-58 for asher
• 20:09 logmsgbot: asher synchronized wmf-config/db.php 're-weighting s6 dbs'
• 20:06 logmsgbot: asher synchronized wmf-config/db.php 'adding db43 back to s6 at a low weight'
• 20:01 logmsgbot: asher synchronized wmf-config/db.php 'raising db53 weight to 400'
• 19:37 logmsgbot: asher synchronized wmf-config/db.php 'raising db53 weight to 200'
• 19:33 logmsgbot: asher synchronized wmf-config/db.php 'adding db53 as an enwiki slave at 1/4 normal weight'
• 19:18 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'switched to Preprocessor_Hash on ocwiki only'
• 18:52 LeslieCarr: moved cp1001-1040 to private vlan
• 18:52 LeslieCarr: restarted morebots
• 02:05 logmsgbot: LocalisationUpdate completed (1.18) at Tue Jan 24 02:05:48 UTC 2012
• 01:55 Ryan_Lane: fixed reverse dns for labs instances
• 01:32 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Add account creation throttle increase for bug 33900'
• 01:07 LeslieCarr: restarting dhcp3-server on brewster
• 00:54 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'new rsvg command line option'
• 00:54 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
• 00:50 Tim: upgraded rsvg on all mediawiki-installation servers, for some reason it is installed on all of them
• 00:23 binasher: streaming a hotbackup of db1006 to db43
• 00:19 Tim: running apt-get upgrade on image scalers
• 00:18 Tim: uploaded new rsvg to apt.wikimedia.org, deploying to image scalers

January 23

• 23:31 binasher: started slaving db53 from db36 (enwiki)
• 23:16 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'update to Mobile Frontend for custom logo support'
• 23:14 RobH: dns update for a bunch of things
• 23:12 preilly: push config change for custom logos
• 23:11 logmsgbot: preilly synchronized wmf-config/InitialiseSettings.php 'add MobileFrontend custom logo support'
• 23:10 logmsgbot: preilly synchronized wmf-config/CommonSettings.php 'add MobileFrontend custom logo support'
• 23:10 RobH: srv187, srv188, srv189 set to false in pybal for api lvs, old servers that will be decommed soon.
• 21:26 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Wrapping more long lines'
• 21:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33864 - Flood flag on sr.wiki'
• 21:15 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33864 - Flood flag on sr.wiki'
• 20:55 binasher: rebooting db1029 with proprietary binary only huawei kernel module installed, for short term ssd evaluation
• 20:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33899 - Request for Narayam in outreach.wikimedia.org'
• 20:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33841 - Re-point $wgLogo to on zhwiki'
• 20:34 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33166 - Creation of a new namespace for Malagasy Wiktionary'
• 19:59 RobH: added dns info for ms-be1 but not pushing change until leslie pushes her
• 19:19 RobH: locke seems ok
• 19:07 RobH: locke down
• 19:06 RobH: going to shutdown locke now for the move
• 18:05 logmsgbot: nikerabbit synchronized php-1.18/extensions/WebFonts/ 'i18ndeploy r109836'
• 18:04 logmsgbot: nikerabbit synchronized php-1.18/extensions/CodeReview/ui/CodeRevisionView.php 'i18ndeploy r109836'
• 17:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Push changes for aswiki by Santhosh'
• 13:29 rainman-sr: restarted search1, search3, search4 - not sure why they were dead
• 02:22 Ryan_Lane: installing new version of nginx in eqiad
• 02:22 Ryan_Lane: restarted nginx in pmtpa and esams
• 02:21 Ryan_Lane: installed new version of nginx in pmtpa and esams
• 02:06 logmsgbot: LocalisationUpdate completed (1.18) at Mon Jan 23 02:06:35 UTC 2012

January 22

• 23:46 Ryan_Lane: changing nginx config to use the escaped useragent
• 23:45 Ryan_Lane: repooling ssl4
• 23:39 Ryan_Lane: restarting nginx servers
• 23:31 Tim: pushed out unstripped version of wmerrors
• 23:26 Ryan_Lane: testing new nginx package on ssl4
• 22:41 mark: cp1042 stuck on disk i/o, rebooting
• 22:28 mark: Restarted varnish backend on cp1041 and cp1042
• 21:38 preilly: remove SOPA banner
• 21:38 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'update to mobile frontend to remove sopa banner'
• 18:42 Tim: running apt-get upgrade on searchidx2
• 18:39 Tim: running apt-get upgrade on snapshot2 and snapshot4
• 18:32 Tim: running apt-get upgrade on snapshot1 to get wikimedia version of php-wikidiff2
• 06:36 Ryan_Lane: repooling ssl1004, depooling ssl4
• 05:44 Ryan_Lane: depooling ssl1004
• 02:05 logmsgbot: LocalisationUpdate completed (1.18) at Sun Jan 22 02:05:22 UTC 2012
• 00:51 Reedy: Resent mediawiki-cvs commit emails from r109549 through r109704

January 21

• 21:45 logmsgbot: reedy synchronized php-1.18/includes/api/ApiParse.php 'r109695'
• 19:19 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php 'More for bug 29742'
• 19:14 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33215 - Enabling transwiki import on sa.wiktionary+'
• 18:55 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Setting wgNamespaceRobotPolicies for th projects'
• 18:35 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Cleanup wgEnableDnsBlacklist, enable for th projects'
• 18:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Setting timezone for th projects'
• 02:05 logmsgbot: LocalisationUpdate completed (1.18) at Sat Jan 21 02:05:53 UTC 2012
• 00:27 logmsgbot: asher synchronized wmf-config/db.php 'raising db52 load to 400'
• 00:22 logmsgbot: asher synchronized wmf-config/db.php 'raising db52 load to 200'
• 00:17 logmsgbot: asher synchronized wmf-config/db.php 'adding db52 to enwiki, load 100'

January 20

• 23:45 LeslieCarr: ms6 sdc is undergoing fsck due to wrong fs type, bad option, bad superblock, or other on /dev/sdc1,
• 23:44 binasher: moving north america bits back to eqiad
• 23:34 binasher: moved bits eqiad to pmtpa (via scenarios/normal/bits-geo.wikimedia.org)
• 23:32 LeslieCarr: killed carnish on niobium , cpu load seems to be going down
• 23:30 LeslieCarr: reloading arsenic
• 23:10 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Wrap some stupidly long lines'
• 22:52 LeslieCarr: rebooting cp3001
• 22:30 LeslieCarr: reloading cp3001
• 22:24 LeslieCarr: restarting networking on cp3001
• 22:03 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33789 - Enable botadmin usergroup on ml.wikipedia'
• 22:00 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33789 - Enable botadmin usergroup on ml.wikipedia'
• 21:58 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33789 - Enable botadmin usergroup on ml.wikipedia'
• 20:40 RobH: dns servers all still online after update
• 20:40 RobH: dns update for dataset1001
• 18:33 LeslieCarr: knsq9 has recovered post-reboot
• 18:21 LeslieCarr: knsq9 will be rebooted as it is dead, dead, dead
• 17:48 LeslieCarr: knsq9 is dead/overloaded
• 15:11 mutante: knsq30 still has bad disk, powering down again
• 15:07 mutante: powercycling knsq30 after replacing cable
• 15:04 ^demon: fixed post-commit hook on formey email notifs to point to correct smtp server
• 14:41 logmsgbot: reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
• 14:27 logmsgbot: reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
• 13:03 mutante: reinstalling srv199
• 02:05 logmsgbot: LocalisationUpdate completed (1.18) at Fri Jan 20 02:05:21 UTC 2012
• 01:17 awjr: updated representative/zipcode mapping and some contact info for a handful of reps/senators for CongressLookup r109598
• 01:12 binasher: started another hotbackup of db38 to db52
• 01:08 logmsgbot: asher synchronized wmf-config/db.php 'pulling db52'
• 00:45 logmsgbot: asher synchronized wmf-config/db.php 'doubling db52 weight'
• 00:38 logmsgbot: asher synchronized wmf-config/db.php 'lowering db52 weight'
• 00:32 binasher: deployed new enwiki slave, db52
• 00:32 logmsgbot: asher synchronized wmf-config/db.php 'setting db52 to full weight'
• 00:19 logmsgbot: asher synchronized wmf-config/db.php 'adding new enwiki slave db52, with a low weight'
• 00:08 preilly: push weekly mobile frontend update
• 00:08 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'weekly update to Mobile Frontend'

January 19

• 23:56 Ryan_Lane: changed global roles netadmins and sysadmins to be virtual static groups in ldap that autopopulate with any user that has objectclass=novauser
• 23:15 Tim: rebuilt wikidiff2 with package name php-wikidiff2, removed lucid package php5-wikidiff2 from apt using "reprepro remove"
• 22:52 Tim: recompiled wikidiff2 and put the new version up on apt.wikimedia.org
• 21:51 Jeff_Green: starting conversion of fundraisingdb 'faulkner' tables from myisam to innodb, expect replication delays
• 21:12 binasher: starting slaving db52 from db36, running hotbackup of db32 to db53
• 20:36 RobH: dataset1001 shut down for later use
• 20:27 RobH: dataset1001 mgmt online
• 20:15 RobH: dataset1001.mgmt even
• 20:15 RobH: updating dns for dataset1mgmt
• 20:03 LeslieCarr: testing
• 20:00 LeslieCarr: testing
• 05:16 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/ 'new sopa banner'
• 05:14 logmsgbot: laner synchronized wmf-config/InitialiseSettings.php 'Enabling anon editing for enwiki'
• 05:12 logmsgbot: laner synchronized wmf-config/InitialiseSettings.php 'Enabling page creation for users'
• 05:08 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/ 'new sopa banner'
• 05:06 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/ 'new sopa banner'
• 05:00 logmsgbot: laner synchronized wmf-config/InitialiseSettings.php 'Removing all SOPA changes, excluding editing for anons, and page creation'
• 04:57 binasher: flushing mobile varnish caches
• 04:53 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/ 'new sopa banner'
• 04:47 Ryan_Lane: Preparing InitialiseSettings for renabling Wikipedia. DO NOT SCAP, DO NOT PUSH InitializeSettings
• 04:32 logmsgbot: awjrichards synchronizing Wikimedia installation... : Deploying CongressLookup changes for the lifting of the blackout
• 03:11 Ryan_Lane: bringing virt1 back up
• 03:01 Ryan_Lane: rebooting virt1 to ensure hardware virtualization is enabled in the bios
• 02:30 logmsgbot: awjrichards synchronized php/extensions/CongressLookup/SpecialCongressLookup.php 'r109477'
• 02:29 logmsgbot: awjrichards synchronized php/extensions/CongressLookup/CongressLookup.i18n.php 'r109477'
• 02:06 Ryan_Lane: rebalance of gluster volume completed
• 02:05 logmsgbot: LocalisationUpdate completed (1.18) at Thu Jan 19 02:05:55 UTC 2012
• 02:05 Ryan_Lane: rebalancing instance gluster volume. network may get saturated for a while.
• 01:55 Ryan_Lane: added virt1 and virt4 to instance volume for gluster
• 01:17 Reedy: Leaving cleanupUploadStash.php running against commonswiki in a screen session as me on hume
• 01:16 binasher: removing extra mobile varnish capacity - it wasn't needed
• 01:13 awjr: updated zip code/representative data on enwiki to r109465
• 01:01 Ryan_Lane: installed python-argparse on stat1
• 00:54 binasher: running a hot backup of db32, streaming to db52
• 00:22 Ryan_Lane: removing virt1 cname
• 00:21 Ryan_Lane: rebuilding virt1 as a nova compute node
• 00:20 LeslieCarr: changed vlan for virt1 eth0
• 00:18 Ryan_Lane: cleared lighttpd logs on brewster and restarted squid and lighttpd
• 00:05 logmsgbot: asher synchronized wmf-config/db.php 'returning db32 to normal weight'

January 18

• 23:59 logmsgbot: asher synchronized wmf-config/db.php 'returning db32 at a low weight'
• 23:50 binasher: rebooting db32 for mysql/kernel upgrades
• 23:49 logmsgbot: asher synchronized wmf-config/db.php 'pulling db32 from s1 for mysql/kernel upgrades'
• 23:44 logmsgbot: awjrichards synchronized php/extensions/CongressLookup/SpecialCongressLookup.php 'r109457'
• 23:02 maplebed: increased the size of db11's logical volume for /a from 500G to 800G.
• 22:27 binasher: enwiki master changed to db36 - MASTER_LOG_FILE='db36-bin.000599', MASTER_LOG_POS=15773827
• 22:26 logmsgbot: asher synchronized wmf-config/db.php 'done swapping s1 master to db36'
• 22:25 binasher: swapping s1 master to db36
• 22:24 logmsgbot: asher synchronized wmf-config/db.php 'starting swap of s1 master to db36, s1 in read-only'
• 22:13 logmsgbot: asher synchronized wmf-config/db.php 'returning db36 to normal weight'
• 22:07 logmsgbot: asher synchronized wmf-config/db.php 'returning db36 at a low weight'
• 21:59 logmsgbot: awjrichards synchronized php/extensions/CongressLookup/SpecialCongressLookup.php 'r109440'
• 21:58 binasher: rebooting db36, upgrading kernel + mysql
• 21:56 logmsgbot: asher synchronized wmf-config/db.php 'pulling db36 from s1 for mysql/kernel upgrades'
• 21:54 Ryan_Lane: installing python-wurfl on stat1
• 21:35 Ryan_Lane: installing geoip-bin geoip-database libgeoip1 python-geoip on stat1
• 21:13 logmsgbot: asher synchronized wmf-config/db.php 'returning db38 at prior weight'
• 21:05 Reedy: Run patch-ug_group-length-increase.sql on all wikis
• 21:04 Reedy: Run patch-uploadstash_chunk.sql on all wikis
• 21:03 Reedy: Run patch-jobs-add-timestamp.sql on all wikis
• 20:55 awjr: update cl_zip5 table for CongressLookup to data in r 109408
• 20:43 Reedy: Manually running cleanupUploadStash.php against commonswiki
• 20:42 Reedy: Manually ran cleanupUploadStash.php against enwiki
• 20:31 binasher: db38 in service at a low weight with new lucid kernel and current mysql build
• 20:30 RobH: shutting down db17, confirmed not in db rotation and has no mysql instance active
• 20:30 logmsgbot: asher synchronized wmf-config/db.php 'returning db38 at a lower weight'
• 20:28 logmsgbot: asher synchronized wmf-config/db.php 'pulling db38 again'
• 20:26 logmsgbot: asher synchronized wmf-config/db.php 'returning db38 to service'
• 20:17 LeslieCarr: rebooting spence as it's once again gone crazy
• 20:11 binasher: pulled db38, rebooting for kernel and mysql upgrades
• 20:11 logmsgbot: asher synchronized wmf-config/db.php 'pulling db38 from s1 for upgrade'
• 20:04 RobH: mw1102 coming down for mainboard replacement
• 20:03 LeslieCarr: killing puppet processes on spence
• 19:28 Reedy: Run patch-jobs-add-timestamp.sql on enwiki (jobs table is empty!)
• 19:01 mutante: mw1108 - OS installed, added to puppet, finished catalog run, free for use
• 18:37 mutante: pxe booting mw1108, OS install
• 18:36 mutante: fixed DHCP config for mw1108 on brewster, had the string "Failed to connect to 10.65.1.108." where the MAC address should have been.
• 18:27 RobH: searchidx1001 memory replaced per rt 2208
• 18:20 mutante: tried to PXE boot mw1108 but no DHCP offers received
• 18:15 RobH: searchidx1001 memory being replaced
• 18:14 LeslieCarr: re-preffing tele2 routes
• 18:11 RobH: db1004 hard disk replaced per rt#2140, rebuilding
• 17:40 LeslieCarr: Draining HE to perform maintenance on the physical port
• 16:57 logmsgbot: reedy synchronized php-1.18/extensions/CongressLookup/SpecialCongressLookup.php 'r109395'
• 14:45 mark: Changed service IP addresses of lists.wikimedia.org in DNS to US prefixes
• 14:40 mark: Disabled hold_domains on sodium and lily
• 14:28 mark: Setup lily to route lists.wikimedia.org mails to sodium
• 14:21 mark: rsync complete. Running dpkg-reconfigure mailman on sodium
• 13:43 logmsgbot: demon synchronized php-1.18/extensions/CongressLookup/SpecialCongressLookup.php 'r109362'
• 13:38 mark: Started rsync of selected mailman directories under /var/lib/mailman from lily to sodium
• 13:37 mark: Removed all test messages on the exim4 queue on sodium
• 13:37 mark: Created /var LVM snapshot on lily
• 13:35 mark: Stopped lighttpd on lily
• 13:34 mark: Stopped mailman on lily and sodium
• 13:30 mark: Set hold_domains = lists.wikimedia.org on lily, to hold new lists mails on the queue
• 13:30 mark: Starting mailman migration
• 13:03 mutante: restarted pdns on ns0
• 08:49 logmsgbot: neilk synchronizing Wikimedia installation... :
• 07:45 logmsgbot: awjrichards synchronized php/extensions/CongressLookup/SpecialCongressLookup.php 'r109336'
• 06:28 logmsgbot: awjrichards synchronized php/extensions/CongressLookup/SpecialCongressLookup.php 'r109324'
• 06:24 logmsgbot: awjrichards synchronized php/extensions/CongressLookup/SpecialCongressLookup.php 'r109322'
• 06:08 logmsgbot: awjrichards synchronized php/extensions/CongressLookup/SpecialCongressLookup.php 'r109319'
• 05:32 logmsgbot: laner synchronized wmf-config/InitialiseSettings.php 'Disabling CentralNotice for simplewiki'
• 05:25 logmsgbot: laner synchronized wmf-config/InitialiseSettings.php 'Disabling moodbar on enwiki'
• 04:59 logmsgbot: laner synchronized wmf-config/InitialiseSettings.php 'Disabling all editing for enwiki for SOPA blackout'
• 04:50 Ryan_Lane: queuing up changes for totally disabling edits. DO NOT SCAP! DO NOT SYNC InitialiseSettings!
• 04:47 Ryan_Lane: Editing enwiki's MediaWiki:Robots.txt to disallow BannerController for SOPA blackout
• 04:45 logmsgbot: laner synchronized wmf-config/InitialiseSettings.php
• 04:44 Ryan_Lane: Disabling anon editing and page creation by users on enwiki for SOPA blackout
• 04:33 logmsgbot: neilk synchronized wmf-config/InitialiseSettings.php 'enable CongressLookup on enwiki'
• 04:01 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/ApplicationTemplate.php 'update version'
• 03:56 logmsgbot: neilk synchronizing Wikimedia installation... : deploying CongressLookup (for i18n reasons, not deploying to enwiki)
• 03:28 logmsgbot: laner synchronized wmf-config/InitialiseSettings.php 'Removing restriction of display title for SOPA landing pages'
• 02:35 binasher: cp1039-40 are now in service for mobile wikipedia
• 02:04 logmsgbot: LocalisationUpdate completed (1.18) at Wed Jan 18 02:04:57 UTC 2012
• 01:35 RobH: cp1040 and cp1036 ready for use
• 01:33 RobH: cp1037, cp1038, cp1039 os installed, varnish partitions mounted, and puppet run

January 17

• 22:47 binasher: ram only varnish instance now running on marmontel in front of apache/wordpress
• 22:07 Ryan_Lane: installing memcache on marmontel
• 20:54 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33769 - Allow bureaucrats to remove sysop rights at Bashkir Wikipedia'
• 20:13 mutante: en.planet updates were stuck. reason was corrupted cache causing "bsddb.db.DBPageNotFoundError" which broke update script. solution was to kill stuck updates, delete files in cache dir and run update manually
• 19:59 logmsgbot: reedy synchronized php-1.18/includes/Feed.php 'r109197'
• 19:36 Ryan_Lane: added Cite extension to labscosnole
• 19:29 logmsgbot: reedy synchronized php-1.18/extensions/ArticleFeedbackv5/modules/jquery.articleFeedbackv5/jquery.articleFeedbackv5.js 'r109186'
• 18:05 RobH: blog is instantly faster
• 18:05 RobH: theme updated on blog along with settting limit back to 20 comments per page
• 17:46 RobH: aware of blog slowdowns, work is being done
• 17:35 mutante: also upgraded drac firmware on mw1081 & mw1099 (fixes mgmt console problem)
• 16:45 mutante: upgrading drac firmware on mw1108
• 15:54 RobH: db43 rebooting
• 15:27 RobH: db7 shutting down for decom, not listed in db for any clusters, load .01
• 11:05 logmsgbot: neilk synchronized wmf-config/ExtensionMessages-1.18.php 'added CongressLookup to ExtensionMessages-1.18 for i18n'
• 11:04 logmsgbot: neilk synchronized wmf-config/extension-list 'added CongressLookup to extension-list for i18n'
• 10:30 logmsgbot: neilk synchronizing Wikimedia installation... : deploying CongressLookup. We are not deploying to any live wiki, just test, but this is to make i18n work
• 10:28 logmsgbot: neilk synchronized wmf-config/InitialiseSettings.php 'added CongressLookup to InitialiseSettings'
• 10:25 logmsgbot: neilk synchronized wmf-config/CommonSettings.php 'added CongressLookup require'
• 05:47 maplebed: marmontel has now replaced hooper as blog.wikimedia.org
• 05:26 maplebed: installing the mysql client on marmontel to test connectivity to the DB
• 05:16 Ryan_Lane: installing php-apc on marmontel
• 04:52 RobH: another dns update for servermgmt
• 04:18 Ryan_Lane: installing varnish on hooper
• 02:29 Ryan_Lane1: that last message was in regards to hooper
• 02:29 Ryan_Lane1: temporarily disabled puppet, since the apache configuration was manually modified
• 02:06 logmsgbot: LocalisationUpdate completed (1.18) at Tue Jan 17 02:06:00 UTC 2012
• 01:52 Ryan_Lane: installed w3 total cache in wordpress on hooper
• 01:51 Ryan_Lane: installing tidy on hooper
• 01:51 Ryan_Lane: installing php-apc on hooper
• 01:31 Ryan_Lane: powercycling hooper
• 00:44 neilk_: neilk just added config change to set caching for banners on testwiki to 0. Should have no effect anywhere else.
• 00:39 logmsgbot: neilk synchronized wmf-config/CommonSettings.php

January 16

• 23:59 logmsgbot: tstarling synchronized docroot/bits/robots.txt 'removed rule intended for the itwiki protest, left in accidentally'
• 22:06 logmsgbot: laner synchronized wmf-config/InitialiseSettings.php 'Switching permissions back to normal on testwiki'
• 22:02 logmsgbot: laner synchronized wmf-config/InitialiseSettings.php 'Testing disabling edits on testwiki'
• 22:02 Ryan_Lane: testing disabling edits for testwiki
• 21:26 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 33763 DoubleWiki on frwikiversity'
• 20:33 Ryan_Lane: pushing floating address changes to virt0
• 19:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32182 - enable articlefeedback extension on spanish wikipedia'
• 19:14 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33759 - Change sitename for lb.wiktionary'
• 18:48 logmsgbot: nikerabbit synchronized php-1.18/extensions/Translate/ 'i18ndeploy'
• 18:47 logmsgbot: nikerabbit synchronized php-1.18/extensions/WebFonts/ 'i18ndeploy'
• 18:45 logmsgbot: nikerabbit synchronized php-1.18/extensions/Narayam/resources/ext.narayam.rules.as.js 'i19ndeploy'
• 18:44 logmsgbot: nikerabbit synchronized php-1.18/skins/common/shared.css 'i18ndeploy'
• 17:17 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33508 - Enable Rollback group on id.wiki'
• 17:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable autoconfirmed reupload on incubatorwiki'
• 17:00 Reedy: srv221 and srv222 are out of space on /
• 17:00 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable autoconfirmed reupload on incubatorwiki'
• 16:41 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php 'Kill labs setting stuff'
• 16:40 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php 'Kill labs setting stuff'
• 16:37 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33662 - Change project namespace for lb.wiktionary'
• 16:35 Reedy: Ran namespaceDupes on fawikisource
• 16:34 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33662 - Change project namespace for lb.wiktionary'
• 16:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33708 - Add alias to fa wikisource'
• 16:26 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33708 - Add alias to fa wikisource'
• 16:06 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33758 - Arabic numerals on Arabic Wiktionary'
• 15:55 cmjohnson1: shutting down srv199 for main board replacement
• 15:09 mutante: torrus was broken (RT:2279) and did not start due to corrupted berkeleydb, used db4.8_recover, service started again
• 12:38 mark: Running puppet on freshly installed sodium
• 09:15 apergos: cleaned up /tmp on srv223... seems like cleanup once an hour by cron isn't often enough any more, scalers are doing too much work
• 02:05 logmsgbot: LocalisationUpdate completed (1.18) at Mon Jan 16 02:05:05 UTC 2012

January 15

• 02:45 logmsgbot: reedy synchronized php-1.18/extensions/CentralNotice/CentralNotice.db.php 'r108949'
• 02:05 logmsgbot: LocalisationUpdate completed (1.18) at Sun Jan 15 02:05:26 UTC 2012

January 14

• 23:37 Ryan_Lane: shutting down virt1
• 20:29 Ryan_Lane: stopping opendj on virt1
• 20:07 Ryan_Lane: stopping pdns on virt1
• 02:15 Reedy: That was me testing something
• 02:14 logmsgbot: LocalisationUpdate failed: SVN update of extensions failed
• 02:04 logmsgbot: LocalisationUpdate completed (1.18) at Sat Jan 14 02:04:46 UTC 2012
• 01:00 Ryan_Lane: brought pdns on virt1 back up
• 00:53 Ryan_Lane: stopping pdns on virt1 again to test dns
• 00:19 Ryan_Lane: powercycling formey

January 13

• 22:40 Ryan_Lane: force running puppet on all instances
• 22:40 Ryan_Lane: re-generated certificates for all instances
• 22:40 Ryan_Lane: deleted all puppet certificates on all instances
• 22:31 LeslieCarr: restarting ganglia1001
• 21:40 Ryan_Lane: changing virt1 to be a cname of virt0
• 21:17 Ryan_Lane: killed pdns on virt1
• 20:40 Ryan_Lane: stopped pdns on virt1
• 20:32 logmsgbot: reedy synchronized php-1.18/extensions/Contest/specials/ 'r108843'
• 20:28 Ryan_Lane: changed NS records for wmflabs.org and wmflabs to point to virt0
• 20:28 Ryan_Lane: changed recursor to point wmflabs domain to virt0
• 20:16 K4-713: synchronized payments cluster to r108833
• 19:30 Ryan_Lane: shutting down virt1 to ensure migration was completed
• 19:12 LeslieCarr: restarting gmond on cp1043
• 18:55 LeslieCarr: hard powercycling ms1002
• 18:48 LeslieCarr: rebooting ms1002 due to kswapd 100% cpu bug https://bugs.launchpad.net/ubuntu/+bug/721896
• 16:44 mutante: added alswiktionary & alswikibooks to closed.dblist
• 16:43 logmsgbot: dzahn synchronized closed.dblist
• 16:33 mutante: syncing InitialiseSettings.php after changing as wiki namespace per bug 33507
• 16:33 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
• 16:07 mutante: updated blog theme and installed a plugin per RT:2271
• 14:34 mutante: srv191 - has now fresh OS, re-issued puppet certs, ran puppet, restart memcached, etc. - all back in monitoring
• 12:59 mutante: PXE booting srv191, installing OS
• 07:45 Ryan_Lane: disassociated and reassociated some floating IP addresses, to fix NAT issues. Some NAT rules went missing.
• 07:43 Ryan_Lane: added a grant for mediawiki in the database to fix labsconsole mediawiki outage
• 07:42 Ryan_Lane: fixed memcached port in mediawiki configuration on labsconsole to fix slowness issue
• 03:29 Ryan_Lane: switching labsconsole.wikimedia.org address to point to virt0
• 02:58 Ryan_Lane: dns server is up on virt0
• 02:57 Ryan_Lane: switched active ldap server in labs to virt0, for nova itself. instances still need to be re-pointed
• 02:56 Ryan_Lane: switched rabbitmq server in labs to virt0
• 02:56 Ryan_Lane: switched mysql masters for labs to virt0
• 02:05 logmsgbot: LocalisationUpdate completed (1.18) at Fri Jan 13 02:05:29 UTC 2012
• 01:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 33469 - Enable rollback function for editor group kawiki'
• 01:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 33507 for aswiki'
• 01:09 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 33556 - ArticleFeedback settings on Chinese wikipedia'
• 01:02 logmsgbot: reedy synchronized closed.dblist 'Closing en_labswikimedia, de_labswikimedia, liquidthreads_labswikimedia (resync)'
• 01:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 33468 - Email notifications for eswikibooks'
• 00:33 Reedy: That was only touch
• 00:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
• 00:32 Reedy: srv223 is also out of diskspace
• 00:29 Reedy: srv219 is out of diskspace
• 00:28 logmsgbot: reedy synchronized closed.dblist 'Closing en_labswikimedia, de_labswikimedia, liquidthreads_labswikimedia'
• 00:17 Ryan_Lane: stopping puppet on all virt nodes

January 12

• 21:54 Ryan_Lane: relabeled port at virt0
• 21:54 Ryan_Lane: moved new virt0 from squid vlan to public-services2
• 21:43 Ryan_Lane: rebuilding mobile2 as virt0
• 21:43 Ryan_Lane: Adding back mgmt info for mobile1, changing mobile2 to virt0
• 21:11 Ryan_Lane: rebuilding mobile1 as virt0
• 21:08 Ryan_Lane: renaming mobile1 to virt0
• 20:54 binasher: installing percona-toolkit on few remaining hardy dbs
• 20:26 cmjohnson1: shutting down srv178-189 for decommissioning
• 20:14 binasher: granted the "process" priv to nagios@localhost on all production db clusters
• 20:07 logmsgbot: reedy synchronized php-1.18/includes/specials/SpecialSearch.php 'r108751'
• 20:07 LeslieCarr: reassigning ports on asw-b-sdtpa
• 17:00 notpeter: stop sodium to do manual reinstall
• 16:33 RobH: adjusting all power strip humidity sensor 2 (floor level) to 12% humidity, as the center rack has the proper levels, floor levels always are low in humidity.
• 16:17 mutante: after a config change to nrpe_local.cfg and puppet applying the change, the service was not resrted but for some reason all nagios-nrpe-server caught SIGTERM. manually applying the same config change does not cause problems. that caused a Nagios outage until nrpe servers were started again (via dsh)
• 16:04 mutante: starting nagios-nrpe-server on ALL via dsh to speed up nagios recovery
• 15:33 mutante: starting nagios-nrpe-server on srv's via dsh
• 02:04 logmsgbot: LocalisationUpdate completed (1.18) at Thu Jan 12 02:04:31 UTC 2012
• 00:48 preilly: pushing quick fix for special random
• 00:48 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'update to mobile frontend to fix random link'
• 00:41 LeslieCarr: added ganglia1002 and ganglia1001 to dns

January 11

• 23:18 RobH: searchidx1001 offline and powered down until replacement memory arrives (2012-01-13) rt 2208
• 22:56 RobH: poking searchidx1001 for memory error
• 22:45 RobH: mw1108 online and ready for install per rt2253
• 22:42 RobH: mw1099 repaired, ready for os install per rt2252
• 22:39 RobH: mw1081 ready for install rt2251
• 22:32 RobH: no its not ;]
• 22:16 Reedy: lists.wikimedia.org is down
• 21:53 logmsgbot: reedy synchronized php-1.18/includes/api/ 'r108683'
• 21:36 RobH: psw1-eqiad mgmt connected
• 21:24 RobH: leslie is handling the ganglia not starting back up issue even though i caused it to die, yay me
• 21:22 RobH: updated dns for neon/cobalt to ganglia1001/1002
• 21:17 RobH: ganglia offline for a moment, sorry folks
• 21:17 logmsgbot: catrope synchronizing Wikimedia installation... : Deploying MoodBar changes
• 21:16 RobH: i just took nickel offline by mistake
• 20:58 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Change shorturl preefix default'
• 20:57 logmsgbot: reedy synchronized php-1.18/extensions/ShortUrl/ 'r108680'
• 20:50 RoanKattouw: Applying MoodBar schema changes (index addition and column addition) on all wikis
• 20:44 logmsgbot: catrope synchronized php-1.18/extensions/ArticleFeedbackv5/ 'Updating AFTv5 to trunk staet'
• 20:14 Jeff_Green: adjusted firewall rules on payments* to restore ganglia reporting since we switched to nickel
• 20:11 RoanKattouw: Created AFTv5 tables on testwiki
• 20:09 logmsgbot: catrope synchronized php-1.18/extensions/ArticleFeedbackv5/modules/jquery.articleFeedbackv5/jquery.articleFeedbackv5.js 'r108666'
• 19:53 cmjohnson1: shutting down srv191 for new install
• 19:52 cmjohnson1: replaced HDD srv191
• 19:47 logmsgbot: catrope synchronized php-1.18/resources/startup.js 'touch'
• 19:46 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable AFTv5 on testwiki'
• 19:40 RobH: mw1103 hardware issues, disregard nagios flapping
• 19:37 RobH: mw1102 offline due to bad mainboard until replacement arrives tomorrow or next
• 19:30 RobH: working on mw1102, disregard flapping
• 18:40 notpeter: running authdns-update on dobson to pick up new dns temps
• 18:28 RobH: lvs1003 repaired, now needs install and setup. rt1549 and rt 2241
• 18:26 notpeter: gracefulling apache on spence to deactivate nmis.w.o (abandoned install of nedi)
• 16:23 mark: Started rsync of lily:/var/lib/mailman/archives to sodium (in a screen on sodium)
• 15:49 mark: Started rsync of lily:/var/lib/mailman/data to sodium (in a screen on sodium)
• 15:39 logmsgbot: reedy synchronized php-1.18/includes/ 'r108626'
• 15:34 logmsgbot: reedy synchronized php-1.18/includes/ 'revert r108625'
• 15:32 logmsgbot: reedy synchronized php-1.18/includes/ 'r108625'
• 15:23 logmsgbot: reedy synchronized php-1.18/extensions/CodeReview/ 'r108623'
• 15:22 logmsgbot: reedy synchronized php-1.18/extensions/ArticleFeedbackv5/ 'r108623'
• 15:22 logmsgbot: reedy synchronized php-1.18/extensions/ApiSandbox/ 'r108623'
• 15:20 logmsgbot: reedy synchronized php-1.18/resources/mediawiki.action/ 'r108622'
• 15:19 logmsgbot: reedy synchronized php-1.18/includes/ 'r108622'
• 14:11 mutante: nagios https now serves real SSL cert
• 14:09 mutante: fixed Apache VirtualHost warnings on spence, NameVirtualHost *:443 in ports.conf, <VirtualHost *:443> in sites-available,..
• 02:04 logmsgbot: LocalisationUpdate completed (1.18) at Wed Jan 11 02:04:39 UTC 2012
• 01:42 preilly: pushing updates to Zero Rated Mobile Access extension
• 01:41 logmsgbot: preilly synchronized php-1.18/extensions/ZeroRatedMobileAccess/ 'push updates to ZeroRatedMobileAccess extension'
• 01:18 preilly: only activate Zero Rated Mobile Access Extension for test wiki
• 00:38 logmsgbot: preilly synchronized wmf-config/InitialiseSettings.php 'add ZeroRatedMobileAccess extension only on test'
• 00:34 preilly: pushing ZeroRatedMobileAccess extension to production
• 00:34 logmsgbot: preilly synchronized wmf-config/InitialiseSettings.php 'add ZeroRatedMobileAccess extension'
• 00:34 logmsgbot: preilly synchronized wmf-config/CommonSettings.php 'add ZeroRatedMobileAccess extension'
• 00:33 logmsgbot: preilly synchronized wmf-config/extension-list 'add ZeroRatedMobileAccess extension'
• 00:30 preilly: push ZeroRatedMobileAccess extension
• 00:30 logmsgbot: preilly synchronized php-1.18/extensions/ZeroRatedMobileAccess/ 'initial push of ZeroRatedMobileAccess extension'
• 00:23 LeslieCarr: applying new loopback filter to cr1-eqiad - higher risk of issues

January 10

• 23:58 preilly: weekly mobile frontend push
• 23:58 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/ 'weekly update to mobile frontend'
• 22:10 lcarr: broke ganglia redirect on nickel, fixing with next push
• 19:45 LeslieCarr: stopping gmetad on spence and unmounting the tmpfs drive
• 18:32 LeslieCarr: restarting gmetad on nickel
• 11:39 logmsgbot: nikerabbit synchronized php-1.18/extensions/Translate/MessageGroups.php 'Translate bugfix r108500'
• 07:20 logmsgbot: nikerabbit synchronized php-1.18/extensions/Translate/TranslateEditAddons.php 'r108497'
• 02:05 logmsgbot: LocalisationUpdate completed (1.18) at Tue Jan 10 02:05:15 UTC 2012
• 01:31 binasher: all varnish servers have been upgraded to 3.0.2
• 00:38 RobH: db1004 pd8 set to offline per rt 2140, will place call to dell for replacement
• 00:23 RobH: correction for typo, mw1102, not mw1002
• 00:23 binasher: repooled cp3002
• 00:23 RobH: mw1002 coming down for hw testing rt 1656
• 00:19 binasher: depooling cp3002, upgrading varnish

January 9

• 23:57 binasher: testing varnish 3.0.2 upgrade on cp3001 (bits)
• 23:42 binasher: adding two new mobile cache servers running varnish 3.0.2 (cp104[12]) to the m.wiki eqiad vip
• 22:26 LeslieCarr: ganglia moved to new nickel server
• 21:36 LeslieCarr: changing gmond source for ganglia3-tip
• 21:19 RobH: snapshot1001-1004 mgmt online
• 21:05 RobH: updating dns with snapshot1001-1004 primary ip info
• 20:56 RobH: updating dns with snapshot1001-1004 mgmt
• 20:30 logmsgbot: catrope synchronized php-1.18/extensions/ArticleFeedbackv5/modules/jquery.articleFeedbackv5/jquery.articleFeedbackv5.js 'r108470'
• 20:29 logmsgbot: catrope synchronized php-1.18/extensions/ArticleFeedbackv5/api/ApiArticleFeedbackv5.php 'r108470'
• 20:25 LeslieCarr: replacing the ops@ alias with the new ops list on mchenry as people keep forgetting to email the new list
• 19:57 logmsgbot: nikerabbit synchronized php-1.18/extensions/Translate/Translate.php 'Deploy r108469 - bugfix for Translate'
• 19:43 RobH: torrus dead, kicking
• 19:32 logmsgbot: nikerabbit synchronized wmf-config/CommonSettings.php 'Updating Translate config 2/2'
• 19:32 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Updating Translate config 1/2'
• 19:16 logmsgbot: nikerabbit synchronized php-1.18/includes/parser/Parser.php 'Deploying r108461'
• 19:07 logmsgbot: catrope synchronized php-1.18/extensions/UploadWizard/resources/mw.UploadWizardLicenseInput.js 'r108459'
• 18:55 logmsgbot: nikerabbit synchronized php-1.18/extensions/Translate/ 'Deploying translate r108451'
• 18:54 logmsgbot: nikerabbit synchronized php-1.18/languages/messages/MessagesEn.php 'Updating messagesEn'
• 18:42 logmsgbot: nikerabbit synchronized php-1.18/extensions/ParserFunctions/ParserFunctions.i18n.magic.php 'Deploying r108449'
• 18:34 logmsgbot: nikerabbit synchronized php-1.18/extensions/WikimediaMessages/WikimediaGrammarForms.php 'Deploying r108433'
• 18:30 logmsgbot: nikerabbit synchronized p/extensions/WebFonts/ 'Updating WebFonts r108447'
• 18:21 logmsgbot: nikerabbit synchronized php-1.18/extensions/Narayam/ 'Syncing Narayam'
• 18:18 Nikerabbit: running Narayam preference migration script
• 17:30 jeremyb: the time is now 17:30:30 UTC
• 17:20 RoanKattouw: Installing (!) NTP on wikitech
• 17:16 jeremyb: the time is now 17:19:30 UTC
• 02:01 logmsgbot: LocalisationUpdate completed (1.18) at Mon Jan 9 02:04:47 UTC 2012

January 8

• 23:20 Reedy: For some reason cp1001-1042 weren't listed in CommonSettings.php XFF, but (at least) 1042 was in service, meaning edits were attributed to it
• 23:18 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Add cp1001-cp1041'
• 23:10 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Add cp1042 to XFF'
• 21:47 rainman-sr: killed broken search indexer thread on searchidx1 (please note searchidx1 is no longer in use!), and restarted incremental indexing on searchidx2 which was somehow broken
• 21:43 rainman-sr: someone started incremental updating on searchidx1 ??!!
• 14:54 apergos: removed old puppet lockfile on brewster, ran by hand
• 14:47 apergos: cleared out some very large squid logs on brewster, (basically all of them) plus lighty logs, disk was full. restarted squid manually
• 02:01 logmsgbot: LocalisationUpdate completed (1.18) at Sun Jan 8 02:05:11 UTC 2012
• 00:43 tfinc: killing long running show_bug.cgi procs on kaulen

January 7

• 22:30 Reedy: Users reporting slowness while editing. dberror.log shows a few mysql errors for enwiki master and slaves. Few errors on other wikis, mainly enwiki
• 02:01 logmsgbot: LocalisationUpdate completed (1.18) at Sat Jan 7 02:05:09 UTC 2012

January 6

• 23:22 RobH: working rt1549 lvs1003 may flap, it is presently not in service due to possible hdd failure
• 22:55 binasher: db22 is back in s4
• 22:55 logmsgbot: asher synchronized wmf-config/db.php 'adding db22 back to s4'
• 21:41 RobH: db1029 powering back up with ssd testing hardware installed
• 21:35 RobH: db1029 coming down for ssd testing
• 21:26 RobH: cp1014 and cp1019 hdd controller cables replaced (removed for testing controllers), both can be used normally
• 21:19 binasher: restoring db22 from a live hotbackup of db1038
• 21:18 RobH: es1002 back ready for service use per #2220: replace original RAID card in es1002
• 21:05 binasher: putting db51 into production as an s4 slave
• 21:05 logmsgbot: asher synchronized wmf-config/db.php 'adding db51 as an s4 slave'
• 20:57 binasher: started slaving db51 off of db31
• 20:21 RobH: rt2226 - redeploy db22 for asher
• 20:19 RobH: db22 reinstalled and booting into OS. No puppet runs yet, now its Asher's problem ;]
• 20:04 RobH: db22 reinstalling
• 19:24 binasher: started innodb hot backup of db1038 to db51
• 18:43 maplebed: s4 database rotation complete. outage duration 36 minutes.
• 18:37 maplebed: pushed out new db.php setting s4 to read-write
• 18:37 logmsgbot: ben synchronized wmf-config/db.php
• 18:35 maplebed: db31 made read-write as the new master for s4
• 18:31 maplebed: old master for s4 log file db22-bin.000106 log pos 631618956
• 18:30 maplebed: new master for s4: db31, log file db31-bin.000213 log pos is 205612709
• 18:24 logmsgbot: asher synchronized wmf-config/db.php 'setting s4 to read only, preparing to make db31 master'
• 18:21 Reedy: Commons having db issues, db22 (s4 master) has a disk issue
• 16:02 apergos: restarted lilghty on dataset2
• 16:01 Reedy: HTTP server (lighttpd?) seems to be down on dataset2
• 15:46 RoanKattouw: Removing gs_* files in /tmp on srv220 that are >30 min old
• 15:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33556 - ArticleFeedback settings on Chinese wikipedia'
• 15:43 RoanKattouw: Removed /tmp/mw-cache-1.17 and /tmp/mw-cache-1.17-test on srv220
• 15:41 Reedy: srv220 / is at 100% usage
• 15:41 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33556 - ArticleFeedback settings on Chinese wikipedia'
• 14:34 mutante: saw the log about cp1043/44 being deliberately left broken, but requirement in varnish.pp also broke others, fixed on sq67,68,69 (gerrit change 1802)
• 02:01 logmsgbot: LocalisationUpdate completed (1.18) at Fri Jan 6 02:05:01 UTC 2012
• 01:25 binasher: puppet is being deliberately left broken on cp1043 and 1044 until tomorrow
• 01:23 binasher: backend varnish instance on cp1042 running 3.0.2 is in production for 1/3 of mobile requests

January 5

• 22:15 preilly: small fix for iPhone vary support
• 22:15 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php
• 21:39 Ryan_Lane: rebooting virt1
• 21:01 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'wmgShortUrlPrefix'
• 21:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wmgShortUrlPrefix'
• 20:08 Reedy: Created ShortUrl tables on testwiki
• 20:07 logmsgbot: reedy synchronizing Wikimedia installation... : Update extensionmessages
• 20:05 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'wmgUseShortUrl'
• 20:04 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wmgUseShortUrl'
• 20:02 logmsgbot: reedy synchronized php-1.18/extensions/ShortUrl 'Pushing ShortUrl files out'
• 19:08 notpeter: restarting dhcpd on brewster
• 18:45 preilly: pushing fix for js error on production
• 18:45 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/ApplicationTemplate.php
• 18:45 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/javascripts/application.js
• 18:00 mutante: tarin - added "#includedir /etc/sudoers.d" to sudo config, needs to read /etc/sudoers.d/nrpe for Nagios RAID check
• 17:49 logmsgbot_: hashar: gallium: cleaned /tmp . Our test suites leak a large amount of files :D
• 17:49 ^demon: removed chuck norris plugin from jenkins, restarted
• 16:48 mutante: payments4 - 25 running nginx procs cause a warning - but normal and just raise limit?
• 16:15 mutante: people claim it was "completely resolved with "2.6.38-10 backport from PPA." (add-apt-repository ppa:kernel-ppa/ppa ...). wanna try that? (or just reboot ms1002 pls)
• 15:49 mutante: quotes on kswapd problem (that also appeared on other servers): "has nothing to do with swap space or memory".."the kernel process which swaps tasks".."means the kernel is spending more time context switching tasks than it is actually executing the tasks".."you're chasing a ghost if you're trying to tune your swap/memory environment"
• 15:45 mutante: ms1002 - kswapd 100% CPU - but no swap used and free memory left - this looks like https://bugs.launchpad.net/ubuntu/+bug/721896 again
• 15:39 mutante: Nagios check_ntp does stuff like: overall average offset: 0 -> NTP OK: Offset unknown| -> NTP CRITICAL: Offset unknown (even though this bug was supposed to be fixed in a version before the one we use)..sigh
• 15:34 mutante: dataset1 - date was off by ~ 27 hours. known issues RT 216 & 1345 with hardware clock, additionally though Nagios NTP check is still buggy (possibly due to leap seconds ;P) -> http://tech.akom.net/archives/27-Nagios-check_ntp-quits-working-in-2009-with-Offset-unknown.html)
• 15:14 mutante: lvs1004 - puppet didnt run since 12 hours, looked stuck, "already in progress" on every run. rm /var/lib/puppet/state/puppetdlock, restart puppet agent, finished fine in a few seconds. maybe puppet bug 2888,5246 or related
• 14:57 mutante: magnesium - memcached runs on default port 11211, but we run all the others on 11000, this causes Nagios CRIT. Is it supposed to run here? (was also on -l 127.0.0.1 only, but init script starts it on all)
• 14:55 Jeff_Green: searchidx1 /a reached 100%, did the "space issues" maintenance procedure from wikitech search documentation
• 14:39 mutante: same on srv193
• 14:35 mutante: srv290 - before restart memcached was running with -m 64 and -l 127.0.0.1 for some reason, causing Nagios CRIT, now it looks like others and recovered
• 14:32 mutante: restarting memcached on srv290
• 02:01 logmsgbot: LocalisationUpdate completed (1.18) at Thu Jan 5 02:05:03 UTC 2012

January 4

• 23:27 logmsgbot: catrope synchronizing Wikimedia installation... : Deploying MoodBar and MarkAsHelpful changes
• 22:39 Tim: taking srv280 for action=purge slowness investigation
• 21:20 Ryan_Lane: deploying LdapAuthentication 2.0a and OpenStackmanager 1.3 to virt1
• 21:13 RoanKattouw: Applying schema changes to moodbar_feedback_response on all wikis (drop index, create index, add column)
• 19:36 notpeter: restarting dhcpd on brewster
• 19:13 RobH: dns update successful and none of them fell over
• 19:12 Reedy: r108070 even
• 19:12 logmsgbot: reedy synchronized php-1.18/extensions/CentralAuth/specials/ 'r107070'
• 19:11 RobH: updating dns for mgmt of ms-fe1/2 and other new servers in tampa, as well as search boxen in eqiad
• 19:04 mutante: srv199 boots but without eth0, NIC1 is Enabled in BIOS but MAC Address "Not Present" - creating hardware ticket
• 18:55 logmsgbot: catrope synchronized php-1.18/extensions/ArticleFeedbackv5/modules/jquery.articleFeedbackv5/jquery.articleFeedbackv5.js 'r108064'
• 18:43 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Disable AFTv5 bucketing tracking again'
• 18:38 mutante: powercycling srv199
• 18:33 logmsgbot: catrope synchronized php-1.18/resources/startup.js 'touch'
• 18:30 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Actually bump version number'
• 18:28 logmsgbot: catrope synchronized php-1.18/resources/mediawiki/mediawiki.user.js 'Revert live hack'
• 18:24 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'and bump the version number too'
• 18:22 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Enable tracking for AFTv5 bucketing'
• 18:06 mutante: duplicate nagios-wm instances on spence (/home/wikipedia/bin/ircecho vs. /usr/ircecho/bin/ircecho) killed them both, restarted with init.d/ircecho
• 18:00 logmsgbot: catrope synchronized php-1.18/resources/mediawiki/mediawiki.user.js 'Live hack for tracking a percentage of bucketing events'
• 17:52 mutante: knsq11 is broken. boots into installer, then "Dazed and confused" at hardware detection (NMI received for unknown reason 21 on CPU 0). -> RT 2206
• 17:38 mutante: powercycling knsq11
• 11:31 logmsgbot: catrope synchronized php-1.18/extensions/ClickTracking/ClickTracking.hooks.php 'r108017'
• 08:44 logmsgbot: nikerabbit synchronized php-1.18/includes/specials/SpecialAllmessages.php 'r107998'
• 07:40 Tim: fixed puppet by re-running the post-merge hook with key forwarding enabled, and then started puppet on ms6
• 07:32 Tim: on ms6.esams: fixed proxy IP address and stopped puppet while I figure out how to fix it
• 03:25 Tim: experimentally raised max_concurrent_checks to 128
• 03:17 Tim: on spence in nagios.cfg, reduced service_reaper_frequency from 10 to 1, to avoid having a massive process count spike every 10 seconds as checks are started. Locally only as a test.
• 02:27 Ryan_Lane: I should clarify that I removed 10.2.1.13 from /etc/network/interfaces, it's still properly bound to lo
• 02:24 Tim: on spence: setting up logrotate for nagios.log and removing nagios-bloated-log.log
• 02:22 Ryan_Lane: removing manually added 10.2.1.13 address from lvs4
• 02:01 logmsgbot: LocalisationUpdate completed (1.18) at Wed Jan 4 02:04:57 UTC 2012
• 01:43 Nemo_bis: Last week slowness: job queue backlog now cleared on !Wikimedia Commons and (almost) English !Wikipedia http://ur1.ca/77q9b
• 01:02 logmsgbot: reedy synchronized php-1.18/includes/ 'r107978'
• 00:45 logmsgbot: reedy synchronized php-1.18/extensions 'r107977, r107976'
• 00:39 Tim: running purgeParserCache.php on hume, deleting objects older than 3 months
• 00:38 logmsgbot: reedy synchronized php-1.18/includes/specials/ 'r107975'
• 00:29 logmsgbot: tstarling synchronizing Wikimedia installation... :
• 00:27 logmsgbot: reedy synchronized php-1.18/extensions/Nuke/ 'r107974'
• 00:25 logmsgbot: reedy synchronized php-1.18/extensions/ 'r107970'

January 3

• 23:00 Tim: on spence: restarting gmetad
• 22:58 logmsgbot: reedy synchronizing Wikimedia installation... : Pushing r107953, r107955, r107956, r107957
• 22:47 LeslieCarr: stopping and then starting apache2 on spence to try and lower load
• 22:29 RobH: added in the lo addres to lvs4, now its working and generating thumbnails
• 22:09 logmsgbot: reedy synchronizing Wikimedia installation... : Push r107938 r107948
• 21:45 RobH: ganglia graphs will have missing data for past 30 to 40 minutes
• 21:45 RobH: spence back online, ganglia and nagios confirmed operational
• 21:38 RobH: resetting spence and dropping to serial to try to fix it
• 21:25 RobH: nagios and ganglia down due to spence reboot, system still coming back online
• 21:21 RobH: spence is unresponsive to ssh and serial console, rebooting
• 21:14 LeslieCarr: resetting DRAC 5 on spence for management connectivity
• 21:05 binasher: that fixed it. but how did that happen?
• 21:05 binasher: ran ip addr add 10.2.1.22/32 label "lo:LVS" dev lo on lvs4
• 19:36 logmsgbot: reedy synchronized php-1.18/skins/common/images/ 'r107930'
• 17:36 mutante: killing more runJobs.php / nextJobDB.php processes on a bunch of servers (/home/catrope/badjobrunners)
• 17:26 RoanKattouw: Stopping job runners on the following DECOMMISSIONED servers: srv151 srv152 srv153 srv158 srv160 srv164 srv165 srv166 srv167 srv168 srv170 srv176 srv177 srv178 srv181 srv184 srv185
• 15:55 RobH: torrus back, took forever to recompile
• 15:53 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33485 - Enable WikiLove in si.wikipedia'
• 15:52 Reedy: Created wikilove tables on siwiki
• 15:46 RobH: torrus deadlocked, kicking
• 14:00 RoanKattouw: Restarting job runners on srv242 and mw25, those are the last ones that are stuck
• 13:57 RoanKattouw: Restarting all job runners that are stuck
• 13:48 RoanKattouw: Restarting job runner on srv236, seems to be stuck
• 02:02 logmsgbot: LocalisationUpdate completed (1.18) at Tue Jan 3 02:05:21 UTC 2012

January 2

• 23:36 Reedy: Seems to potentially be an issue with job runners, enwiki backed up to over 90k over the last week or so. Needs investigating
• 23:18 logmsgbot: tstarling synchronized php-1.18/includes/parser/Parser.php 'r107856'
• 22:58 logmsgbot: tstarling synchronizing Wikimedia installation... :
• 18:08 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Bug 33368: WebFonts on bpywiki'
• 18:05 logmsgbot: nikerabbit synchronized php-1.18/languages/messages/ 'i18ndeploy r107843'
• 18:04 logmsgbot: nikerabbit synchronized php-1.18/extensions/WebFonts/WebFonts.i18n.php 'i18ndeploy r107843'
• 16:58 mutante: installed SiteMap extension on Bugzilla - soon bugs should be googleable (per BZ:33406)
• 16:33 mutante: upgraded Bugzilla from 4.0.2 to 4.0.3 (http://www.bugzilla.org/releases/4.0.3/release-notes.html#v40_point) (RT #2194)
• 14:47 mutante: cleaned out gammu spool to stop sms bomb - sorry. deamon runs again now though..
• 14:36 mutante: fixed gammu-smsd on spence per wikitech "Nagios#Fixing_the_USB_dongle" (sending out queued SMS now )
• 14:30 mutante: puppet ran on spence, ganglia also seems ok despite the errors i logged before. gammu-smsd cant find device again though
• 14:03 mutante: spence / gmetad - RRD_update .. illegal attempt to update using time .. last update time is .. (minimum one second step)
• 13:57 mutante: gmond complains about missing kernel modules on spence when trying to start on boot
• 13:54 mutante: spence down, no ssh, no mgmt output, powercycling it ..
• 02:01 logmsgbot: LocalisationUpdate completed (1.18) at Mon Jan 2 02:04:47 UTC 2012
• 00:08 logmsgbot: tstarling synchronized php-1.18/includes/media/SVGMetadataExtractor.php 'r107792'

January 1

• 21:28 Ryan_Lane: restarted pdns-recursor on dobson
• 21:26 Ryan_Lane: restarted pdns on ns2 about an hour ago
• 09:46 apergos: restarted lucene search on srch 10, 11, then later on 3,4,9,1
• 09:35 apergos: removed log.1 from /a/search/logs on search6, it was 35gb
• 03:55 Tim: fixed broken package on search7 and search11
• 02:01 logmsgbot: LocalisationUpdate completed (1.18) at Sun Jan 1 02:04:30 UTC 2012
• 01:36 Tim: adjusted FD limit in /etc/init.d/lsearchd on all search servers with sed
• 01:34 Tim: increased FD limit on search6 and restarted lsearchd
• 00:46 Tim: removed some logs on search6 to fix /a disk space exhaustion

2000s

• Archive 1: 2004 Jun - 2004 Sep
• Archive 2: 2004 Oct - 2004 Nov
• Archive 3: 2004 Dec - 2005 Mar
• Archive 4: 2005 Apr - 2005 Jul
• Archive 5: 2005 Aug - 2005 Oct, with revision history 2004-06-23 to 2005-11-25
• Archive 6: 2005 Nov - 2006 Feb
• Archive 7: 2006 Mar - 2006 Jun
• Archive 8: 2006 Jul - 2006 Sep
• Archive 9: 2006 Oct - 2007 Jan, with revision history 2005-11-25 to 2007-02-21
• Archive 10: 2007 Feb - 2007 Jun
• Archive 11: 2007 Jul - 2007 Dec
• Archive 12: 2008 Jan - 2008 Jul
• Archive 12a: 2008 Aug
• Archive 12b: 2008 Sept
• Archive 13: 2008 Oct - 2009 Jun
• Archive 14: 2009 Jun - 2009 Dec

2010s

• Archive 15: 2010 Jan - 2010 Jun
• Archive 16: 2010 Jul - 2010 Oct
• Archive 17: 2010 Nov - 2010 Dec
• Archive 18: 2011 Jan - 2011 Jun
• Archive 19: 2011 Jul - 2011 Dec
• Archive 20: 2011 Dec - 2012 Jun, with revision history 2007-02-21 to 2012-03-27
• Archive 21: 2012 Jul - 2013 Jan
• Archive 22: 2013 Jan - 2013 Jul
• Archive 23: 2013 Aug - 2013 Dec
• Archive 24: 2014 Jan - 2014 Mar
• Archive 25: 2014 April - 2014 September
• Archive 26: 2014 October - 2014 December
• Archive 27: 2015 January - 2015 July
• Archive 28: 2015 August - 2015 December
• Archive 29: 2016 January - 2016 May
• Archive 30: 2016 June - 2016 August
• Archive 31: 2016 September - 2016 December
• Archive 32: 2017 January - 2017 July
• Archive 33: 2017 August - 2017 December
• Archive 34: 2018 January - 2018 April
• Archive 35: 2018 May - 2018 August
• Archive 36: 2018 September - 2018 December
• Archive 37: 2019 January - 2019 April
• Archive 38: 2019 May - 2019 August
• Archive 39: 2019 September - 2019 December

2020s

• Archive 40: 2020 January - 2020 April
• Archive 41: 2020 May - 2020 July
• Archive 42: 2020 August - 2020 November
• Archive 43: 2020 December
• Archive 44: 2021 January - 2021 April
• Archive 45: 2021 May - 2021 July
• Archive 46: 2021 August - 2021 October
• Archive 47: 2021 November - 2021 December
• Archive 48: 2022 January
• Archive 49: 2022 February
• Archive 50: 2022 March
• Archive 51: 2022 April 1-15
• Archive 52: 2022 April 16-30
• Archive 53: 2022 May
• Archive 54: 2022 June
• Archive 55: 2022 July
• Archive 56: 2022 August
• Archive 57: 2022 September
• Archive 58: 2022 October
• Archive 59: 2022 November 1-15
• Archive 60: 2022 November 16-30
• Archive 61: 2022 December
• Archive 62: 2023 January
• Archive 63: 2023 February
• Archive 64: 2023 March
• Archive 65: 2023 April
• Archive 66: 2023 May
• Archive 67: 2023 June
• Archive 68: 2023 July
• Archive 69: 2023 August 1-15
• Archive 70: 2023 August 16-31
• Archive 71: 2023 September
• Archive 72: 2023 October
• Archive 73: 2023 November
• Archive 74: 2023 December
• Archive 75: 2024 January
• Archive 76: 2024 February
• Archive 77: 2024 March
• Archive 78: 2024 April
• Archive 79: 2024 May 1-15
• Archive 80: 2024 May 16-31
• Archive 81: 2024 June 1-15
• Archive 82: 2024 June 16-30
• Archive 83: 2024 July
• Archive 84: 2024 August
• Archive 85: 2024 September
• Archive 86: 2024 October