Server Admin Log/Archive 21

January 29

22:27 RobH: torrus is all happy again, huzzah
22:25 logmsgbot: maxsem Finished syncing Wikimedia installation... : https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2013-01-29
22:11 logmsgbot: maxsem Started syncing Wikimedia installation... : https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2013-01-29
22:02 Ryan_Lane: rebooting labstore2
22:01 RobH: updating torrus files via puppet, sorry if torrus crashes out, its bitchy
22:00 notpeter: temp stopping puppet on brewster
21:58 Ryan_Lane: rebooting labstore1
21:57 Ryan_Lane: stopping all gluster services
21:38 cmjohnson1: authdns update adding mgmt ip's to zone files for hpm misc servers in eqiad
21:33 Ryan_Lane: stopping gluster services on labstore1, this will affect nfs mounts (like ssh keys and public data exports)
20:43 RobH: updated rt config, pmtpa comments now come from pmtpa-comment, not rt-comment to match the rest of the queues configuration
19:51 cmjohnson1: ms-be1004 going down for h/w replacement
19:47 hashar: jenkins: updating all jobs
19:31 cmjohnson1: ms-be1003 going down to replace h/w
19:29 Ryan_Lane: upgrading glusterfs to 3.3.1 from 3.3.0 and replacing old glusterfs package with new packages
19:09 cmjohnson1: ms-be1002 going down to replace h/w
18:37 cmjohnson1: ms-be1001 going down to replace h/w
17:57 andrewbogott: restarted apache and memcached on virt0 (apparently fixing main page errors)
17:10 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php
14:14 mark: powercycled sodium
08:20 logmsgbot: pgehres synchronized php-1.21wmf8/extensions/CentralNotice/ 'Updating CentralNotice to master'
04:58 logmsgbot: tstarling synchronized php-1.21wmf6/cache/trusted-xff.cdb
04:57 logmsgbot: tstarling synchronized php-1.21wmf7/cache/trusted-xff.cdb
04:53 logmsgbot: tstarling synchronized php-1.21wmf8/cache/trusted-xff.cdb
04:51 logmsgbot: tstarling synchronized php-1.21wmf8/cache/trusted-xff.cdb
04:47 logmsgbot: tstarling synchronized php-1.21wmf8/cache/trusted-xff.cdb
04:36 logmsgbot: tstarling synchronized php-1.21wmf8/cache/trusted-xff.cdb 'fixing corrupted CDB file'
04:36 logmsgbot: tstarling synchronized php-1.21wmf7/cache/trusted-xff.cdb 'fixing corrupted CDB file'
02:55 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Tue Jan 29 02:55:44 UTC 2013
02:30 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Tue Jan 29 02:30:42 UTC 2013
02:17 Ryan_Lane: deleting sysadmin and netadmin roles from ldap
01:55 Ryan_Lane: adding ldap entries for combined sysadmin/netadmin role, projectadmin
01:29 K4-713: updated payments cluster to ec4d3f24862559
00:44 RobH: typo, knsq18
00:44 RobH: squid backend on knsq48 rebuilding
00:36 RobH: crazy memory and cpu spike on knsq18 for no apparent reason, restarting process (memory leak?)

January 28

23:27 K4-713: synchronized payments cluster to 1948266b33e905b
22:40 logmsgbot: pgehres synchronized wmf-config/CommonSettings.php 'Enabling CN:Translate on testwiki'
22:37 hashar: jenkins: applied pep8 and pyflakes jobs to sartoris repository.
22:35 logmsgbot: pgehres Finished syncing Wikimedia installation... : Dark deploying Translate integration with CentralNotice
22:20 logmsgbot: pgehres Started syncing Wikimedia installation... : Dark deploying Translate integration with CentralNotice
21:26 logmsgbot: pgehres synchronized php-1.21wmf8/extensions/LandingCheck 'Updating LandingCheck to master'
20:42 LeslieCarr: cleared a lot of 0 size badly named logs on fluorine:/a/mw-log
20:41 sbernardin1: taking down osm-web1 for h/w installation
19:25 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
19:09 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf8
19:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:31 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:08 paravoid: upgrading ffmpeg/libav for USN-1705-1/USN-1706-1 on all image/video scalers in eqiad/pmtpa
18:00 sbernardin1: shutting down ms-be11 for h/w replacement
15:29 paravoid: upgraded & rebooting emery, 208 days bug
15:06 mark: Pooled cp3009 and cp3010 in the esams upload pool
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Mon Jan 28 02:52:53 UTC 2013
02:30 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Mon Jan 28 02:29:55 UTC 2013

January 27

23:50 logmsgbot: reedy synchronized php-1.21wmf7/extensions/CentralAuth/
23:24 logmsgbot: reedy synchronized php-1.21wmf8/extensions/CentralAuth/
23:11 drdee: enabled sampling on Thank_You_Main and a country filter on Emery
07:16 logmsgbot: aaron synchronized php-1.21wmf8/maintenance/nextJobDB.php 'deployed f0f869dfabc1cedcedd5df972fbe6c279789a498 '
04:48 logmsgbot: aaron synchronized php-1.21wmf8/maintenance/nextJobDB.php 'removed live hack'
04:47 logmsgbot: aaron synchronized php-1.21wmf8/maintenance/runJobs.php
02:55 logmsgbot: aaron synchronized php-1.21wmf8/maintenance/runJobs.php
02:51 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Sun Jan 27 02:51:05 UTC 2013
02:48 logmsgbot: aaron synchronized php-1.21wmf8/maintenance/nextJobDB.php 'quick debugging'
02:27 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Sun Jan 27 02:26:57 UTC 2013
01:30 logmsgbot: aaron synchronized php-1.21wmf8/maintenance/nextJobDB.php 'deployed 93869b8d37b6b7999190b378cb6813f6317b0b10'
01:02 LeslieCarr: rpd core caused bgp to restart (even though it wasn't supposed to) - causing routing churn
00:55 LeslieCarr: core dumping cr1-sdtpa - 1% chance of minor network issue

January 26

20:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
17:37 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
02:54 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Sat Jan 26 02:54:55 UTC 2013
02:38 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: aawiki to 1.21wmf8
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Sat Jan 26 02:29:39 UTC 2013
02:28 logmsgbot: aaron synchronized php-1.21wmf8/maintenance/nextJobDB.php 'logging'
02:21 logmsgbot: aaron synchronized php-1.21wmf8/includes/job/JobQueueDB.php
01:21 LeslieCarr: htcp purging across datacenters now "works". dobson is now receiving purge requests on multicast group 239.128.0.112 port 4827 and transmitting them via udpmcast.py (started by rc.local) to hooft in esams
01:14 LeslieCarr: deactivating multicast for 1 minute in order to try and flush the multicast forwarding table

January 25

23:57 mutante: installing mysql-client/common package upgrades on kaulen (bugzilla)
23:07 LeslieCarr: on cr1 and cr2 on eqiad allowing for unicast-reverse-path feasible-paths
22:29 LeslieCarr: clearing multicast joins in order to troubleshoot multicast pruning
20:47 logmsgbot: reedy synchronized live-1.5/
20:47 logmsgbot: reedy synchronized docroot
19:13 Nemo_bis: Job queue not being run somewhat: no FuzzyBot on mw:Communication for a hour, en.wiki and Commons unhappy.
18:54 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:49 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:47 Reedy: Created translate tables on brwikimedia
18:09 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
17:31 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
16:51 ^demon: formey: mediawiki svn repository is now read-only. long live git.
02:53 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Fri Jan 25 02:53:00 UTC 2013
02:27 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Fri Jan 25 02:27:20 UTC 2013
01:15 binasher: moved s[2-7] analytics slave cnames to pmtpa dbs
01:07 mutante: planet is now an IPv6-enabled service
00:38 mutante: DNS update - set non-autoconfig IPv6 and reverse for zirconium
00:11 mutante: DNS update - push reverse IPv6 for zirconium

January 24

23:49 mutante: installing package upgrades on zirconium
23:49 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/GettingStarted
23:48 logmsgbot: olivneh synchronized php-1.21wmf7/extensions/GettingStarted
22:55 mutante: updating all planets
22:55 andrewbogott: changed all lab sudoer policies to !authenticate
22:55 andrewbogott: changed all labs sudoers from "sudoUser: ALL" to "sudoUser: project-<projname>"
22:28 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php
22:04 logmsgbot: olivneh synchronized php-1.21wmf7/extensions/E3Experiments
22:03 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/E3Experiments
22:02 logmsgbot: olivneh synchronized php-1.21wmf7/extensions/EventLogging
21:59 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/EventLogging
21:39 paravoid: deploying squid config for upload's /monitoring/ switch from url_regex to urlpath_regex
21:28 binasher: fixed dbtree to support our current situation of db-pmpta containing eqiad write masters
21:22 sbernardin: taking srv278 down for decommissioning per paravoid
21:03 K4|foodish: Synchronized payments cluster to fe035f6b5a55946b729f
20:56 paravoid: gradual restart of ms-fe[1-4] proxies (switching scaling to eqiad)
19:24 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend/includes/skins/SkinMobile.php 'https://gerrit.wikimedia.org/r/#/c/45562/'
19:23 logmsgbot: maxsem synchronized php-1.21wmf7/extensions/MobileFrontend/includes/skins/SkinMobile.php 'https://gerrit.wikimedia.org/r/#/c/45562/'
19:19 logmsgbot: aaron synchronized php-1.21wmf8/thumb.php 'deployed c6d478f73f33f051c1060ffe4b9c5855e07d358f'
18:42 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Removed unused NFS file backends'
18:42 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php
18:16 logmsgbot: awjrichards synchronized php-1.21wmf7/extensions/MobileFrontend 'touch files'
18:16 logmsgbot: awjrichards synchronized php-1.21wmf8/extensions/MobileFrontend 'touch files'
18:15 logmsgbot: awjrichards Finished syncing Wikimedia installation... : Redploy rolled back MobileFrontend changes from Jan 17
18:00 cmjohnson1: authdns update "fixing typo on db1060.mgmt.eqiad"
17:56 logmsgbot: awjrichards Started syncing Wikimedia installation... : Redploy rolled back MobileFrontend changes from Jan 17
11:06 logmsgbot: mlitn synchronized wmf-config/CommonSettings.php 'Reinstate AFTv5 test groups'
10:25 logmsgbot: mlitn synchronized wmf-config/CommonSettings.php 'Reinstate AFTv5 test groups'
04:08 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
04:08 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
03:28 Ryan_Lane: powercycling formey
03:11 mutante: DNS update - switching planet over to zirconium / new planet
02:53 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Thu Jan 24 02:52:58 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Thu Jan 24 02:28:21 UTC 2013

January 23

23:57 logmsgbot: dzahn gracefulled all apaches
22:53 RobH: calcium is my testbed server, ignore its bitching
22:28 mutante: starting NTP on mw1008 (Tick tock, on the clock But the party don't stop, no)
22:20 paravoid: netapp: nas1001-b cf giveback
22:10 RobH: authdns-update correcting camera fqdn
22:00 Ryan_Lane: restarting pdns on ns0
21:56 cmjohnson1: authdns update adding mgmt ip's to zone files for 10 misc servers in eqiad
21:30 Ryan_Lane: restarting opendj on virt0
21:12 mutante: restarting pdns on ns1
21:03 RobH: authdns-update to push db61 into sandbox
20:56 RobH: swapped port 3/39 on csw1-sdtpa from vlan 2 to 102
20:24 mutante: removing mw1072 (hardware fail, RT-4381) from "apaches" dsh group
20:22 logmsgbot: dzahn gracefulled all apaches
19:43 mutante: puppet signing mw1085 (was sitting there like a fresh install)
19:41 RobH: stopping puppet on brewster cuz i wanna locally hack and reboot 1 time on db61 without partman auto kicking in
19:28 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Switched all non-wikipedias to 1.21-wmf8
19:07 RobH: db61 coming offline, and going into otrs sandbox service
19:05 RobH: authdns-update
18:03 binasher: amended wikiadmin grants on s7 to allow new jobrunners access to centralauth
17:44 paravoid: running authdns-update - a bunch of cleanups, r3861:3867
15:09 mark: Added missing pim interface ae0.400 statement on cr2-pmtpa which prevented eqiad purges from reaching pmtpa
04:11 logmsgbot: tstarling synchronized wmf-config/db-eqiad.php 'cleanup, added pmtpa host/IP mappings'
04:09 logmsgbot: tstarling synchronized wmf-config/db-pmtpa.php 'back to read/write mode (with eqiad master)'
02:49 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Wed Jan 23 02:49:23 UTC 2013
02:26 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Wed Jan 23 02:26:04 UTC 2013
02:23 Tim: reloading all apaches
02:19 Tim: running sync-apache to fix stale eqiad apache configuration
02:14 Tim: added eqiad apaches to /etc/dsh/group/apaches
01:23 logmsgbot: aaron synchronized php-1.21wmf8/includes/api/ApiUpload.php 'deployed 2a4ad3e32cf4f6814554ebcf09ac0250546a2549'
01:22 logmsgbot: aaron synchronized php-1.21wmf7/includes/api/ApiUpload.php 'deployed 2a4ad3e32cf4f6814554ebcf09ac0250546a2549'
01:07 Tim: ran scap-1 on tmh1001 and tmh1002
01:05 Tim: added tmh1001 and tmh1002 to mediawiki-installation

January 22

23:58 Tim: on db1006: fixing permissions for root@fenari
23:54 Tim: wikiadmin permissions for eqiad were broken, attempting to use CIDR notation, fixing them on all masters
23:04 Ryan_Lane: Alex Monk's echo changes to OpenStackManager deployed on labsconsole
23:04 Ryan_Lane: echo enabled on labsconsole
23:03 Ryan_Lane: labsconsole upgraded to 1.21wmf8
22:38 Tim: on neon: cleaned up /var/lib/puppet/clientbucket/* since it was out of disk space
22:29 logmsgbot: catrope synchronized php-1.21wmf8/resources/jquery/jquery.client.js '8a06466e1697f58717e92532437be642a796d5a1'
22:29 logmsgbot: catrope synchronized php-1.21wmf7/resources/jquery/jquery.client.js '8a06466e1697f58717e92532437be642a796d5a1'
21:55 cmjohnson1: msw-a7-eqiad power was accidently removed while switch was being moved
21:45 ^demon: gerrit restarting. don't panic.
20:26 logmsgbot: catrope synchronized php-1.21wmf8/includes/api/ApiQueryImageInfo.php 'Fix API error in imageinfo'
20:26 logmsgbot: catrope synchronized php-1.21wmf7/includes/api/ApiQueryImageInfo.php 'Fix API error in imageinfo'
20:12 AaronSchulz: Recompiled textvc on all hosts
19:37 Ryan_Lane: upgrading labsconsole to 1.21wmf8
19:03 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Disabling ReaderFeedback'
19:03 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Disabling ReaderFeedback'
18:59 mutante: restarted pdns on ns1, failed during update
18:58 mutante: DNS update - adding ca./ja. planet to zirconium for testing new planet
18:51 mutante: DNS update - deleting wikimania2006/2007
18:14 logmsgbot: root synchronized wmf-config/db-eqiad.php 'eqiad read-write!'
17:59 notpeter: moving jobrunning to eqiad
17:54 logmsgbot: aaron synchronized php-1.21wmf7/includes/Block.php 'deployed b0d5ff9bfb09f15ccbab6849564511d372ecd8d4'
17:46 notpeter: authdns-update
17:42 Jeff_Green: db1013 wiped, reimaged, powered off, and un-monitored
17:40 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'setting pmtpa to readonly'
17:14 notpeter: dns svn repo on sockpuppet has changes staged and checked in, not yet deployed on dobson
17:05 notpeter: staging cname switch for pmtpa/eqiad dbs on sockpuppet
15:22 mark: Changed AS14907->AS43821 routing
12:26 ^demon: restarted ircecho on manganese
10:30 Tim: also replaced /etc/sudoers on fenari and hume. Hume needs a puppet change which will follow shortly
10:18 hashar: gallium : restarted puppet
10:14 hashar: jenkins updated all plugins and restarting
10:09 Tim: on searchidx2: replaced /etc/sudoers with the distro default so that /etc/sudoers.d/* from puppet can take effect
06:48 Tim: on eqiad upload varnishes: purged /wikipedia/it/b/bc/Wiki.png at user request. varnishhtcpd appears to be totally broken.
02:48 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Tue Jan 22 02:48:37 UTC 2013
02:26 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Tue Jan 22 02:26:14 UTC 2013

January 21

23:11 Tim: on sockpuppet: fixed puppet checkout, switching branch from mikepatch1 to production, and then did fetch&&rebase for good measure
22:52 Ryan_Lane: adding virt9-11 entries in dns
22:49 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
20:22 mark: Rerouted AS43821->AS14907 traffic
19:41 mark: Restarting knsq* upload frontends manually
19:34 paravoid: deploying squid config for upload's /monitoring/
19:14 mark: Restarting amssq* upload frontends in a slow loop
19:11 mark: Restarted oversized frontend on amssq50
18:55 paravoid: starting knsq19 backend squid
18:50 mark: Restarted knsq16 backend minus two disks
18:40 mark: Started amssq56 squid instances
18:38 mark: Started knsq16 minus one disk
18:37 mark: Started knsq18 minus one disk
18:32 mark: power cycled amssq56
18:28 mark: Rebooting knsq24
18:05 mark: Rerouted AS43821->AS14907 traffic
17:53 paravoid: stopping knsq19 backend squid
17:46 paravoid: restarting knsq19 backend squid
17:35 paravoid: restarting pybal on lvs1002
17:35 cmjohnson1: db1038 swapping bad disk (slot 2) with new disk
17:27 mark: Rerouted AS14907->AS43821 traffic
16:36 paravoid: depooling, restarting and repooling ms-fe2/3/4 one by one
16:30 paravoid: repooling ms-fe1
16:29 mark: Rerouted AS43821->AS14907 traffic
15:48 logmsgbot: reedy synchronized php-1.21wmf8/includes/EditPage.php
15:45 logmsgbot: reedy synchronized php-1.21wmf7/includes/EditPage.php
15:32 paravoid: depooling ms-fe1 for testing
14:17 hashar: gallium: manually installed pyflakes 44974
11:10 apergos: nagios was dead over the weekend (config broken), fixed in puppet and on spence, now back in action
09:57 hashar: relaying Ryan: he restarted ldap on virt0 (was hung after server restart). nscld was properly falling back to virt1000 but ldap was stuck there too. DNS got restarted.
09:52 hashar: jenkins: jobs refresh completed.
08:14 hashar: jenkins: updating all Jenkins jobs based on d31c92e of integration/jenkins-job-builder-config.git
05:51 Tim: on marmontel: removed MW-specific packages php5-wmerrors, php-luasandbox, php-wikidiff2
02:50 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Mon Jan 21 02:50:20 UTC 2013
02:27 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Mon Jan 21 02:26:59 UTC 2013

January 20

23:56 logmsgbot: tstarling Finished syncing Wikimedia installation... :
23:45 logmsgbot: tstarling Started syncing Wikimedia installation... :
23:40 Ryan_Lane: rebooting virt0 to determine which dimm is bad
19:57 binasher: authdns scenario back to normal
19:40 LeslieCarr: deactivating 2828 as a preferred intermediate route between 14907 and 43821
19:30 binasher: seeing major packetloss between esams and pmtpa, failing out of esams
02:48 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Sun Jan 20 02:48:05 UTC 2013
02:25 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Sun Jan 20 02:25:48 UTC 2013

January 19

22:24 MaxSem: Created ProofredPage table on aswikisource
19:22 logmsgbot: pgehres synchronized wmf-config/reporting-setup.php 'Switching S:FundraiserStatistics read slave back to db1025'
02:46 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Sat Jan 19 02:46:06 UTC 2013
02:24 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Sat Jan 19 02:24:18 UTC 2013
01:54 binasher: testing completed, no longer sending live traffic to eqiad
01:53 binasher: i was sending >40 reqs/sec from europe and asia to eqiad apaches for the last hour for real world non-english testing
00:27 mutante: DNS update - add wikivoyager.org/.de
00:08 RobH: blog is fixed, in that its the software equivalent of spinning plates
00:04 RoanKattouw: Running git fsck on mediawiki/core.git as nobody on manganese

January 18

23:42 RobH: blog is partly offline (unreliable loads) investigating
23:38 RobH: stupid w3 caching on blog won't let me use new wp-victor directory, has to match old .git name or caching errors result
23:34 RobH: blog is being bitchy about new theme versus old caching, reverting to stock and clearing local cache on marmontel
23:26 RobH: blog software updated a few hours ago, forgot to log. just updated theme to git update from gerrit repo.
23:21 logmsgbot: asher synchronized wmf-config/CommonSettings.php 'dc specific poolcounter conf'
23:17 notpeter: stopping all jobrunners including tmh boxes per asher's request while troubleshooting
23:16 mutante: DNS update to add wikimania.asia
23:11 logmsgbot: asher synchronized wmf-config 'eqiad/pmtpa config variances'
22:37 logmsgbot: asher synchronized wmf-config 'revert'
22:36 logmsgbot: asher synchronized wmf-config 'revert'
22:34 logmsgbot: asher synchronized wmf-config 'eqiad/pmtpa config variances'
20:19 logmsgbot: root Finished syncing Wikimedia installation... :
19:49 RobH: authdns-update colby back into sandbox
19:08 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Added ceph file backend configuration'
19:08 logmsgbot: root Started syncing Wikimedia installation... :
19:00 mark: rm -rf /usr/local/apache/common-local on eqiad apaches
18:54 RobH: restarted pdns on linne, it borked on authdns-update
18:48 RobH: authdns-update
18:17 logmsgbot: root Finished syncing Wikimedia installation... :
18:04 logmsgbot: root Started syncing Wikimedia installation... :
18:02 logmsgbot: root Finished syncing Wikimedia installation... :
18:00 mark: Removed mw1085 from mediawiki_installation (acting up)
17:54 logmsgbot: root Started syncing Wikimedia installation... :
17:26 logmsgbot: root Started syncing Wikimedia installation... :
16:11 hashar: gallium jenkins: updated tools/fetch-mw-ext script to wipe the extension destination directory 44648 updated: 09bbb44..e41843b
15:59 hashar: gallium jenkins: updated the tools/fetch-mw-ext script which was not properly fetching the extension. {{gerrit|44644}] updated: cfa82ff..09bbb44
15:18 Reedy: recentchanges.rc_params is now a blob on all wikis
15:14 mark: Deployed MediaWiki on eqiad image scalers
15:11 logmsgbot: root Finished syncing Wikimedia installation... :
15:05 logmsgbot: root Started syncing Wikimedia installation... :
15:00 logmsgbot: root Finished syncing Wikimedia installation... :
14:57 paravoid: running /usr/local/bin/foreachwiki maintenance/cleanupUploadStash.php on hume
14:57 logmsgbot: root Started syncing Wikimedia installation... :
14:49 logmsgbot: reedy synchronized php-1.21wmf8/maintenance/cleanupUploadStash.php
14:49 logmsgbot: reedy synchronized php-1.21wmf7/maintenance/cleanupUploadStash.php
14:31 mark: Corrected srv191:/tmp permissions
14:21 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
13:56 mark: Set rt_cache_rebuild_count to -1 on lvs2 and lvs6
13:50 mark: Powercycled lvs6
13:26 logmsgbot: demon synchronized php-1.21wmf7/includes/resourceloader/ResourceLoaderFileModule.php 'Deploying Idc83a0fe'
13:25 logmsgbot: demon synchronized php-1.21wmf7/includes/resourceloader/ResourceLoader.php 'Deploying Idc83a0fe'
13:25 logmsgbot: demon synchronized php-1.21wmf7/includes/MessageBlobStore.php 'Deploying Idc83a0fe'
12:59 logmsgbot: root Started syncing Wikimedia installation... :
12:57 Tim: added eqiad apaches to mediawiki-installation and started scap, it will take a while
12:31 logmsgbot: tstarling Finished syncing Wikimedia installation... :
12:29 logmsgbot: tstarling Started syncing Wikimedia installation... :
12:19 logmsgbot: root Finished syncing Wikimedia installation... :
12:10 logmsgbot: root Started syncing Wikimedia installation... :
12:03 logmsgbot: root Started syncing Wikimedia installation... :
11:58 mark: Mounted nfs1:/home noatime
11:58 logmsgbot: root Started syncing Wikimedia installation... :
10:31 RoanKattouw: Purged all URLs containing 'wikidata' from Varnish
10:22 logmsgbot: tstarling Started syncing Wikimedia installation... :
05:28 Tim: running sync-apache and gracefulling apaches to hopefully fix reports of occasional wikidata redirect issues
05:04 logmsgbot: tstarling synchronized live-1.5/load.php 're-enabling bits for MF'
03:39 logmsgbot: awjrichards synchronized php-1.21wmf8/extensions/MobileFrontend 'Roll back today's MobileFrontend deployment to 1ded0894cdbc089370a58ba41aeb4db53e5c7345'
03:38 logmsgbot: awjrichards synchronized php-1.21wmf7/extensions/MobileFrontend 'Roll back today's MobileFrontend deployment to 1ded0894cdbc089370a58ba41aeb4db53e5c7345'
03:30 logmsgbot: tstarling synchronized live-1.5/load.php
03:24 Tim: attempting to block MF in bits apache configuration
02:50 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Fri Jan 18 02:50:35 UTC 2013
02:48 Tim: restarted bits apaches due to overload, were totally down
02:27 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Fri Jan 18 02:26:57 UTC 2013
01:44 LeslieCarr: purged mobile varnish cache for awjr
01:43 logmsgbot: awjrichards synchronized php-1.21wmf8/extensions/MobileFrontend 'touch files'
01:42 logmsgbot: awjrichards Finished syncing Wikimedia installation... : Updating MobileFrontend per https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2013-01-17
01:42 logmsgbot: awjrichards synchronized php-1.21wmf7/extensions/MobileFrontend/ 'touch files'
01:35 logmsgbot: dzahn gracefulled all apaches
01:30 logmsgbot: reedy synchronized wmf-config/
01:25 cmjohnson1: authdns update adding db1051-1060 to eqiad mgmt zone file
01:02 logmsgbot: aaron synchronized wmf-config/PrivateSettings.php
00:43 logmsgbot: dzahn gracefulled all apaches
00:42 logmsgbot: dzahn synchronized ./wmf-config/CommonSettings.php
00:42 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
00:40 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
00:20 logmsgbot: dzahn gracefulled all apaches
00:10 logmsgbot: aaron synchronized php-1.21wmf8/extensions/UploadWizard/resources/mw.UploadWizardDetails.js
00:08 logmsgbot: dzahn gracefulled all apaches
00:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'fix whitespace'
00:01 logmsgbot: dzahn gracefulled all apaches

January 17

23:50 logmsgbot: awjrichards Started syncing Wikimedia installation... : Updating MobileFrontend per https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2013-01-17
23:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:33 mutante: rolled wikibugs back to last working version from SVN ..Revision 93776
23:21 RobH: authdns-update
23:12 Tim: on srv191: wiped /tmp and started apache
23:08 Tim: fixing broken /tmp partition on srv191. Stopping apache
23:04 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/EventLogging
22:58 logmsgbot: olivneh synchronized php-1.21wmf7/extensions/EventLogging
22:53 logmsgbot: aaron synchronized php-1.21wmf8/includes/Block.php 'deployed d69578ee88a2542cbaec3bd7904026a4edba86b1'
22:18 logmsgbot: reedy synchronized php-1.21wmf8/cache/interwiki.cdb 'Updating 1.21wmf8 interwiki cache'
22:18 logmsgbot: reedy synchronized php-1.21wmf7/cache/interwiki.cdb 'Updating 1.21wmf7 interwiki cache'
21:59 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
21:41 logmsgbot: dzahn gracefulled all apaches
21:41 mutante: dropping www from wikidata in mw and apache configs as requested
21:39 logmsgbot: dzahn synchronized ./wmf-config/CommonSettings.php
21:39 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
21:10 mutante: manually deployed changes to wikibugs bot, which is in git meanwhile
20:03 logmsgbot: bsitu synchronized php-1.21wmf8/extensions/Echo 'Update Echo to master'
19:47 logmsgbot: py synchronized wmf-config/lucene.php 'moving all search traffic to eqiad'
19:46 binasher: stopped all replication on es1 (es1-4,1001-1004) and ensured all nodes are read-only. replication was enabled there briefly for mha testing.
19:42 logmsgbot: py synchronized wmf-config/lucene.php 'moving enwiki search traffic to eqiad'
19:38 binasher: fixed eqiad root + wikiuser grants on es shards
19:16 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Disabled multiwrite to nas-1'
18:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
17:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
17:41 binasher: moved dewiki db master to eqiad with mha and back to pmtpa in < 6 seconds, only one http request wanting to write hit the read-only window
17:38 Reedy: Created moodbar tables for itwikivoyage
17:32 logmsgbot: asher synchronized wmf-config/db.php 'setting s5/dewiki to writeable'
17:31 logmsgbot: asher synchronized wmf-config/db.php 'setting s5/dewiki to read-only for several minutes'
17:06 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'micro design for itwikivoyage'
16:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
15:55 logmsgbot: reedy synchronized wmf-config/
15:51 Reedy: Created wikilove tables on iswiki
15:17 logmsgbot: reedy synchronized extract2.php
15:16 logmsgbot: reedy synchronized live-1.5/
15:13 logmsgbot: reedy synchronized docroot
15:03 logmsgbot: demon synchronized php-1.21wmf8/extensions/Wikibase/repo/resources/wikibase.ui.entityViewInit.js 'Deploying Ie22a7213'
15:03 logmsgbot: demon synchronized php-1.21wmf8/extensions/Wikibase/repo/resources/Resources.php 'Deploying Ie22a7213'
12:56 hashar: Gerrit: created repository operations/debs/python-voluptuous to package http://pypi.python.org/pypi/voluptuous/
11:43 paravoid: "mwscriptwikiset cleanupUploadStash.php all.dblist" on hume (kill it if swift overloads)
11:37 Tim: installed rsyncd on 6 apache servers for network-aware scap testing
03:56 binasher: resumed replication on db67
03:49 binasher: stopped replication on db67.pmtpa (enwiki research slave), re-replicating off the pmtpa master instead of cross datacenter. original config isn't well supported by mha.
02:45 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Thu Jan 17 02:45:26 UTC 2013
02:27 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Thu Jan 17 02:26:58 UTC 2013
01:10 mutante: DNS update to remove some useless stuff from esams zone (RT-4331)
00:44 mutante: installing package upgrades on ms1002
00:38 mutante: powercycling ms1002
00:37 LeslieCarr: reloading unresponsive ms1002
00:00 logmsgbot: reedy synchronized php-1.21wmf7/maintenance/mcc.php

January 16

23:59 logmsgbot: reedy synchronized php-1.21wmf7/maintenance/mctest.php
23:59 logmsgbot: reedy synchronized php-1.21wmf8/maintenance/mctest.php
23:49 logmsgbot: laner synchronized wmf-config/lucene.php 'Switching search prefix pool to pmtpa'
22:25 LeslieCarr: removing ganglios requirements to try to make it happily run in precise with icinga
22:22 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:21 logmsgbot: reedy synchronized php-1.21wmf8/resources/startup.js
21:58 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikidatawiki to 1.21wmf8
21:58 logmsgbot: reedy synchronized php-1.21wmf8/extensions/
21:52 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki to 1.21wmf8
21:48 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki to wmf8
21:46 logmsgbot: reedy synchronized php-1.21wmf8/extensions/Wikibase
21:43 LeslieCarr: restarted nginx on ssl3001 (was depooled)
21:42 logmsgbot: reedy synchronized php-1.21wmf8/cache/trusted-xff.cdb
21:41 logmsgbot: reedy synchronized php-1.21wmf8/cache/interwiki.cdb
21:39 logmsgbot: reedy synchronized php-1.21wmf8/cache/l10n/
21:17 binasher: https://twitter.com/ashbash/status/291655249075240960
21:16 RobH: authdns-update
21:13 logmsgbot: reedy synchronized php-1.21wmf7/extensions/Wikibase
21:13 binasher: set global read_only=0; on enwiki master
21:13 logmsgbot: asher synchronized wmf-config/db.php 'taking s1 out of read-only'
21:10 binasher: set global read_only=1; on enwiki master for "we have no unit tests, so lets test read-only in production"� testing
21:09 logmsgbot: asher synchronized wmf-config/db.php 'setting s1 to read-only for several minutes'
21:01 cmjohnson1: reinstalling sq48 w/lucid
20:53 mutante: "bak" directory from unclean SVN copied to /root on sockpuppet as dns_bak.tar.gz just in case
20:52 mutante: removing uncommitted stuff from DNS SVN, creating RT for wikimedia.jp.net and softwarewikipedia.com
20:42 logmsgbot: reedy synchronized wmf-config/
20:30 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild message cache for 1.21wmf8
20:17 logmsgbot: reedy synchronized live-1.5/
20:16 logmsgbot: reedy synchronized docroot
20:15 logmsgbot: reedy synchronized php-1.21wmf8
20:15 logmsgbot: reedy synchronized php-1.21wmf8
20:13 logmsgbot: reedy synchronized php-1.21wmf8
18:45 LeslieCarr: turning up xe-1/1/2 on asw-c-eqiad (was disabled due ot faulty equipment)
18:44 LeslieCarr: turning up xe-1/1/0 on asw-c-eqiad (was disabled due to faulty equipment)
18:41 mutante: putting logging-only iptables rule on sanger to check out what rules we actually need
18:37 mark: restarted knsq19 earlier
18:24 mark: Rerouting 43821->14907 traffic away from AS1257
18:24 mutante: updating pt.planet
18:21 RobH: praseodymium being odd, its coming down
18:08 mark: Rerouting 14907->43821 traffic away from AS1257
17:40 Reedy: Swapped slot1 from 1.21wmf6 to 1.21wmf8
17:34 cmjohnson1: powering down titanium to verify console redirection status
15:37 ottomata: deployed new webstatscollector filter to collect stats on wikivoyage domains and restarted udp2log on locke
15:30 cmjohnson1: ms-be1005 & ms-be1006 going down
15:16 hashar: restarted Jenkins again, somehow got stuck :/
14:58 hashar: restarted Jenkins on gallium to apply plugin updates.
13:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
13:23 hashar: gallium: reapplied Tim's patch to PHPUnit install in /usr/share/php/PHPUnit/Util/Test.php to prevent PHP segfaulting while running tests. bug 43972
11:34 mark: Added eqiad apaches and image scalers to the LVS pools
11:28 mark: Added eqiad API apaches to the LVS pool
11:12 mark: Restarted PyBal on lvs1003 and lvs1006
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Wed Jan 16 02:28:11 UTC 2013
00:46 Ryan_Lane: initializing all deployment destinations in pmtpa
00:16 Ryan_Lane: testing deployment destination initialization on hume

January 15

23:59 LeslieCarr: copied ganglios from lucid-wikimedia to precise-wikimedia
23:59 LeslieCarr: restarted morebots
23:35 Ryan_Lane: testing deployment destination initialization on fenari
23:28 RobH: ignore msfe1001/msfe1002 errors (the names without the dash) as they are turned off and renamed
23:21 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 44015 - Add en.wikivoyage autopatrolled group'
22:20 mutante: powercycling msfe1002
22:19 RobH: authdns-update
22:14 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wgUseRCPatrol on for wikivoyages'
21:32 mutante: setting knsq17 to enabled: False in pybal esams/uploads-squids as it is broken per RT-4321
21:17 RobH: cerium & titanium being installed as parsoid caching servers, ignore any alerts for now.
20:20 paravoid: stopping squid3 on sq48
20:00 Nemo_bis: nikerabbit rebuilt message index on all Translate-enabled wikis (half a hour ago)
19:54 RobH: authdns-update
19:15 RobH: authdns-update
18:36 mutante: stopping Apache on sanger
16:01 hashar: Made a mistake in Zuul configuration. A slight outage while correcting it :/
12:55 logmsgbot: nikerabbit synchronized php-1.21wmf7/extensions/Translate/ 'Translate to master again'
12:09 logmsgbot: nikerabbit synchronized wmf-config/CommonSettings.php 'Translate config fix'
10:26 logmsgbot: nikerabbit synchronized php-1.21wmf7/extensions/Translate 'Translate to master'
10:18 logmsgbot: nikerabbit synchronized php-1.21wmf7/includes/HTMLForm.php
10:18 logmsgbot: nikerabbit synchronized php-1.21wmf7/includes/Preferences.php
10:17 logmsgbot: nikerabbit synchronized php-1.21wmf7/includes/AutoLoader.php
10:04 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Enable ULS and disable WebFonts/Narayam on Translate wikis'
10:03 logmsgbot: nikerabbit synchronized wmf-config/CommonSettings.php 'Updated ULS configuration'
09:42 logmsgbot: nikerabbit synchronized php-1.21wmf7/extensions/UniversalLanguageSelector/ 'ULS to master'
08:33 hashar: gallium : installing liblua5.1-0-dev package.
04:50 binasher: set pc[1-3] to replicate from pc100[1-3], full master/master all the way across the sky
02:26 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Tue Jan 15 02:26:37 UTC 2013
02:16 LeslieCarr: rebooting neon
02:02 LeslieCarr: created a new lvm for /var/log on neon, copied files over, and restarted processes. some file permission issues and incorrect file owners may remain
01:39 LeslieCarr: restarting neon
01:23 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'Set sqlbagostuff log'
01:11 mutante: neon - changing logrotate config for atop to rotate 7 instead of 14
01:09 mutante: deleting atop.log.1* in /var/log on neon to free disk space
00:47 binasher: the production persistent parsercache is now on pc[1-3], replicated to pc100[1-3]
00:43 logmsgbot: asher synchronized wmf-config/CommonSettings.php 'deploying db parsercache, sharded and replicated'

January 14

23:18 Tim: changed CNAME for search.wikimedia.org to point to wikimedia-lb instead of ekrem
22:46 mutante: installing apache,mysql-common,perl,sudo package updates on sanger
22:45 mutante: killed java process on sanger, restarted opendj
22:10 hashar: pullen 43849 on fenari (simply changes some comments)
22:04 Reedy: Modified rc_params on huwiki to make it a blob
21:30 Jeff_Green: tearing down community-analytics vhost and cname
21:26 binasher: rebooting pc1
20:36 logmsgbot: reedy cleared profiling data
20:33 RobH: xenon & caesium added to parsoid pybal config
20:27 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable WikibaseClient on huwiki'
20:19 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
20:07 logmsgbot: reedy synchronized wmf-config/
19:24 logmsgbot: reedy synchronized php-1.21wmf7/extensions/Wikibase
19:09 cmjohnson1: provision sq48
19:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Rest of the 'pedias to 1.21wmf7
18:29 Ryan_Lane: adding glusterfs-client glusterfs-common glusterfs-server and glusterfs-dbg for lucid and precise to our repo
17:20 logmsgbot: reedy synchronized php-1.21wmf7/includes/ChangesList.php
16:55 ottomata: packet loss on oxygen after adding new wikipedia zero filters. Stopping puppet to temporarily comment them out and make sure they are the problem.
16:53 cmjohnson1: powercycling db1037 to verify raid cfg
16:48 Reedy: Populated huwiki sites and site_identifier tables from testwiki tables
13:34 hashar: upgrade PEAR packages on gallium bug 43957
13:33 hashar: upgrade PEAR packages on gallium
13:20 hashar: jenkins: gating jobs now always report a message, even if tests complete after the change has been merged in 43407
08:29 hashar: gallium: removed /mnt/jenkins-tmp tmpfs, replaced by /var/lib/jenkins/tmpfs bug 43825
04:05 logmsgbot: tstarling synchronized docroot
03:53 logmsgbot: tstarling synchronized docroot/search.wikimedia.org/index.php
02:50 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Mon Jan 14 02:50:08 UTC 2013
02:25 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Mon Jan 14 02:25:38 UTC 2013
01:25 logmsgbot: tstarling synchronized php-1.21wmf7/includes/GlobalFunctions.php
01:24 logmsgbot: tstarling synchronized php-1.21wmf7/includes/DefaultSettings.php
01:24 logmsgbot: tstarling synchronized php-1.21wmf7/bin/ulimit5.sh
01:22 logmsgbot: tstarling synchronized php-1.21wmf6/includes/GlobalFunctions.php
01:21 logmsgbot: tstarling synchronized php-1.21wmf6/includes/DefaultSettings.php
01:21 logmsgbot: tstarling synchronized php-1.21wmf6/bin/ulimit5.sh
00:43 Tim: patched @requires handling out of phpunit on gallium so that it stops segfaulting and accepts Id9f2fea7

January 13

22:45 Tim: killed stuck convert processes on the image scalers
22:35 mark: restarted apache on srv222
22:35 LeslieCarr: restarted ircecho on spence
20:30 cmjohnson1: provisioning sq48
17:11 apergos: restarted morebots
16:43 paravoid: stopping squid on knsq17 (disk broken RT #4321)
16:44 paravoid: restarting backend varnish: cp1021, cp1023, cp1025, cp1026, cp1028, cp1030, cp1031, cp1033, cp1034, cp1036 (varnish bug)
08:28 Ori: synchronized wmf-config/throttle.php to extend throttle exemption for Pune summit per bug 43856

January 12

09:29 logmsgbot: asher synchronized wmf-config/CommonSettings.php 'pulling parsercache'
02:53 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Sat Jan 12 02:53:39 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Sat Jan 12 02:28:21 UTC 2013
01:20 binasher: added mysql grants for eqiad to all shards

January 11

20:08 logmsgbot: asher synchronized wmf-config/db.php 'repooling db59'
19:48 logmsgbot: anomie synchronized php-1.21wmf7/maintenance/rebuildLocalisationCache.php 'Backported for git-deploy l10n work'
19:47 logmsgbot: anomie synchronized php-1.21wmf6/maintenance/rebuildLocalisationCache.php 'Backported for git-deploy l10n work'
18:55 RobH: xenon & caesium installing as parsoid
17:01 hashar: creating Jenkins jobs for Semantic* extensions
16:35 hashar: creating Jenkins jobs for GettingStarted and PHPExcel extensions.
15:41 logmsgbot: maxsem synchronized wmf-config/throttle.php 'https://gerrit.wikimedia.org/r/#/c/43432/'
14:08 hashar: updating Zuul 23ec1ba..ff79197 . Bring up better support for LOST builds.
13:40 logmsgbot: reedy synchronized php-1.21wmf7/extensions/FlaggedRevs
06:09 logmsgbot: tstarling synchronized wmf-config/db.php 'depool db59'
05:44 logmsgbot: aaron synchronized php-1.21wmf7/includes/job/jobs/HTMLCacheUpdateJob.php 'deployed f0f528e254346f04d4afdb85b439cd20c0df93fe'
05:38 logmsgbot: aaron synchronized php-1.21wmf7/includes/cache/HTMLCacheUpdate.php 'deployed 9709a7d7e423c21e38a0436faa233ac0dcf04226 '
05:38 logmsgbot: aaron synchronized php-1.21wmf7/includes/DefaultSettings.php 'deployed 9709a7d7e423c21e38a0436faa233ac0dcf04226 '
04:38 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Set $wgMaxBacklinksInvalidate.'
04:16 logmsgbot: spage synchronized php-1.21wmf6/extensions/EventLogging 'EventLogging meant to be on 1.21wmf6 wikis as well'
02:52 Tim: killed convert processes on all image scalers
02:49 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Fri Jan 11 02:49:45 UTC 2013
02:33 Tim: killed convert processes on srv222
02:26 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Fri Jan 11 02:26:00 UTC 2013
01:35 Reedy: Ran namespaceDupes.php on cswikisource
01:34 logmsgbot: reedy synchronized php-1.21wmf7/extensions/ProofreadPage
01:33 logmsgbot: spage synchronized php-1.21wmf6/extensions/EventLogging/EventLogging.hooks.php 'Log field fix for mobile'
01:21 logmsgbot: spage synchronized php-1.21wmf7/extensions/EventLogging/EventLogging.hooks.php 'Log field fix for mobile'
00:39 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Force https for mobile login and account creation everywhere'
00:38 logmsgbot: asher synchronized wmf-config/db.php 'setting db59 weight to zero'
00:34 logmsgbot: awjrichards synchronized php-1.21wmf6/extensions/MobileFrontend/
00:33 logmsgbot: awjrichards synchronized php-1.21wmf7/extensions/MobileFrontend/

January 10

23:44 logmsgbot: awjrichards synchronized php-1.21wmf6/extensions/ZeroRatedMobileAccess/ 'Updating ZeroRatedMobileAccess to master on php-1.21wmf6'
23:35 logmsgbot: awjrichards synchronized php-1.21wmf7/extensions/ZeroRatedMobileAccess/ 'Updating ZeroRatedMobileAccess to head of master'
23:12 logmsgbot: anomie synchronized php-1.21wmf7/includes/api/ApiQueryRevisions.php
23:10 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'enable forced https for mobile login/account creation on testwiki'
22:57 logmsgbot: spage Finished syncing Wikimedia installation... : E3 deployment Extension:GettingStarted and its i18n msgs
22:11 logmsgbot: spage Started syncing Wikimedia installation... : E3 deployment Extension:GettingStarted and its i18n msgs
21:41 logmsgbot: olivneh synchronized php-1.21wmf7/extensions/E3Experiments
21:34 logmsgbot: olivneh synchronized php-1.21wmf7/extensions/EventLogging
21:06 logmsgbot: olivneh synchronized php-1.21wmf7/extensions/EventLogging
20:58 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
20:39 binasher: nas1-a thumbs vol ran out of inodes, increased total by 10x
19:34 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
18:29 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
18:19 logmsgbot: reedy synchronized wmf-config/
18:00 logmsgbot: demon synchronized php-1.21wmf7/extensions/ExtensionDistributor 'New version of ExtensionDistributor'
14:38 MaxSem: Updating Solr index on all wikis
14:15 hashar: cleaned up all content in /mnt/jenkins-tmp been replaced by /var/lib/jenkins/tmpfs
13:57 hashar: gallium: manually reinstalled Zuul and restarted it.
13:25 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'Rm GeoData exceptions'
11:48 hashar: gallium: manually installed luajit, libluajit-5.1-dev and g++ packages . Pending gerrit change is 43182
10:11 logmsgbot: maxsem synchronized wmf-config/mobile.php 'https://gerrit.wikimedia.org/r/#/c/43163/ - Try disabling noindex on some enwiki pages per discussion with Google'
02:51 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Thu Jan 10 02:51:34 UTC 2013
02:27 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Thu Jan 10 02:27:16 UTC 2013

January 9

22:41 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
22:16 logmsgbot: aaron synchronized php-1.21wmf7/extensions/UploadWizard/resources/mw.UploadWizardDetails.js 'deployed 7482243e3249ae38d6aedbaf06db7107dc3516f5'
21:00 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
19:53 logmsgbot: reedy synchronized wmf-config/
19:47 cmjohnson1: authdns update adding mc1016/mc1017 to zone file
19:13 logmsgbot: aaron Finished syncing Wikimedia installation... :
19:09 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf7
19:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Fix sewiki typo'
18:24 logmsgbot: aaron Started syncing Wikimedia installation... :
18:08 logmsgbot: aaron synchronized php-1.21wmf7/extensions/UploadWizard 'deployed bd33047d1cb938f3b4923a51862d887e6c831b65'
17:43 logmsgbot: aaron synchronized php-1.21wmf7/includes/ 'deployed 5185259101b27e4780618b3cd7718b9a0c51e1c4'
17:33 logmsgbot: aaron synchronized php-1.21wmf7/includes/upload/AssembleUploadChunks.php 'deployed 4bb28e000bb6609e23046b611be335382aa74618'
16:37 cmjohnson1: authdns update adding osm-cp1003/4 production to zone files
15:35 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Deploying Id2755235'
15:04 mark: Started memcached on virt0
14:53 logmsgbot: nikerabbit synchronized wmf-config/CommonSettings.php 'Translation memory tweak'
11:16 logmsgbot: reedy synchronized wmf-config/
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Wed Jan 9 02:52:56 UTC 2013
02:26 mutante: wiping archive of 'ee' mailing list (RT-4294)
02:26 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Wed Jan 9 02:26:08 UTC 2013
01:19 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php
01:10 logmsgbot: preilly synchronized php-1.21wmf6/extensions/MobileFrontend
01:08 logmsgbot: preilly synchronized php-1.21wmf7/extensions/MobileFrontend
00:36 logmsgbot: maxsem synchronized php-1.21wmf6/resources/Resources.php 'https://gerrit.wikimedia.org/r/#/c/42894/'
00:34 logmsgbot: maxsem synchronized php-1.21wmf6/includes/resourceloader 'https://gerrit.wikimedia.org/r/#/c/42894/'
00:30 mutante: powercycling frozen magnesium
00:03 logmsgbot: maxsem synchronized php-1.21wmf6/extensions/MobileFrontend/

January 8

23:56 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
23:06 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
22:49 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
22:17 logmsgbot: tstarling synchronized php-1.21wmf7/includes/objectcache/SqlBagOStuff.php
22:17 logmsgbot: tstarling synchronized php-1.21wmf7/includes/objectcache/RedisBagOStuff.php
22:16 logmsgbot: tstarling synchronized php-1.21wmf7/includes/ArrayUtils.php
22:16 logmsgbot: tstarling synchronized php-1.21wmf7/includes/AutoLoader.php
21:53 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
21:32 logmsgbot: reedy synchronized php-1.21wmf7/extensions/TimedMediaHandler
19:22 logmsgbot: reedy synchronized wmf-config/
18:51 logmsgbot: reedy synchronized php-1.21wmf7/extensions/TimedMediaHandler
17:38 logmsgbot: olivneh synchronized docroot/wikivoyage.org/favicon.ico 'New favicon.ico for wikivoyage'
17:37 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'New logo for wikivoyage & eswikivoyage'
15:59 hashar: upgraded Zuul
15:12 hashar: updating Zuul code base : 6c4d51a...23ec1ba (forced update)
14:53 hashar: gallium: restarted Zuul which could not recover its Gerrit connection :/
14:37 cmjohnson1: rebooting solr2 to fix HT in bios
14:26 hashar: gallium: running git fsck in /var/lib/git/mediawiki/extensions/Wikibase.git . That repository might have ended up being corrupted somehow :(
14:22 cmjohnson1: rebooting solr1 to fix HT in bios
14:17 ^demon: manganese is back up, gerrit's running fine.
14:13 ^demon: rebooting manganese
13:38 apergos: and restarted them
13:38 apergos: stopped and/or shot all swift processes on ms-be2, some were in an odd state, had load spike
12:52 apergos: cleared out /var/lg/messages.[1-x] on older ms-be hosts, several had full root partitions; disabled logging to that facility in rsyslogd conf
10:37 hashar: stopping Zuul
10:37 hashar: shutdowning jenkins for an unscheduled maintenance and upgrading the installation.
10:30 paravoid: reprepro update for jenkins (fetched 1.480.2)
10:29 paravoid: installing new backported reprepro version to {lucid,precise}-wikimedia and upgrading it in brewster
09:58 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '42232 set autoconfirm count to 10 @ fawiki'
09:54 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php 'Increase AbuseFilter disable threshold for AFT entries to 20%'
09:29 logmsgbot: nikerabbit synchronized php-1.21wmf7/extensions/Translate/resources/js/ext.translate.special.translate.js
02:51 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Tue Jan 8 02:51:08 UTC 2013
02:26 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Tue Jan 8 02:26:01 UTC 2013
01:16 Tim: on deployment-squid.pmtpa.wmflabs: started squid
01:05 notpeter: temp stopping puppet on brewster
00:16 notpeter: starting innobackupex on db1004

January 7

23:54 paravoid: starting nginx on ssl3002
23:26 logmsgbot: aaron synchronized php-1.21wmf7/includes/api/ApiUpload.php 'deployed e2df801500f9472f812388006206e3e3261d58d3'
23:22 RobH: solr1003 coming offline for comparison work per rt 4282
22:57 paravoid: powercycling cp1034, dead for 18h, evidence of sudden memory exhaustion
22:38 logmsgbot: aaron synchronized php-1.21wmf7/includes/api/ApiUpload.php 'deployed 542bfaef8f4ca61db04bb60cbd1e617c731db038'
22:37 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'set wfWikiShellCmd hook.'
22:35 cmjohnson1: shutting down solr3 to check bios setup
22:32 Ryan_Lane: repooling ssl1001 with proxy_http_version 1.1 enabled
21:51 binasher: restarted backend varnish on cp102[27]. cp103[0-6]
21:43 binasher: restarting backend varnish on cp1030 (previously restarted cp102[346])
21:42 logmsgbot: maxsem synchronized wmf-config/CommonSettings.php
21:34 logmsgbot: maxsem synchronized wmf-config/CommonSettings.php 'Reduce the load on Solr servers in pmtpa'
21:18 hashar: jenkins: splitting whitespace check out of mediawiki linting job. Will make whitespace a non voting change to avoid rejecting legitimate changes.
21:08 logmsgbot: maxsem synchronized wmf-config/CommonSettings.php 'Multi-server solr'
20:47 sbernardin: replacing ms-be5 with R720xd
20:17 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'Disable GeoData jobs'
20:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary to 1.21wmf7
20:03 Jeff_Green: authdns-update to add tellurium.wikimedia.org
20:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource, wikiversity and remaining wikimedia to 1.21wmf7
19:52 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks and wikquote to 1.21wmf7
19:36 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special, wikinews and wikivoyages to 1.21wmf7
19:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: fishbowl, private and closed to wmf7
18:40 logmsgbot: reedy synchronized php-1.21wmf7/extensions/EducationProgram
18:39 logmsgbot: reedy synchronized php-1.21wmf6/extensions/EducationProgram
14:10 logmsgbot: reedy synchronized wmf-config/
11:07 hashar: pulled on fenari 42526 'Revert "Kill static-master"' thought NOT synced. That simply bring back some symbolic links for 'beta' so that is harmful.
09:45 hashar: Jenkins: EventLogging now receives pep8 linting, 42517 & 42518
09:10 hashar: Jenkins: whitelisted Bryan Tong Minh in Zuul 39850
08:51 hashar: Jenkins: update all jobs. That bring whitespace checking for MediaWiki extensions. See 37803
04:10 Tim: on hume: running purgeList.php to purge all pages on uzwiki from squid
04:08 logmsgbot: tstarling synchronized php-1.21wmf6/maintenance/purgeList.php
02:51 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Mon Jan 7 02:51:10 UTC 2013
02:31 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Mon Jan 7 02:28:58 UTC 2013
02:02 logmsgbot: tstarling synchronized php-1.21wmf7/includes/ImagePage.php
02:01 logmsgbot: tstarling synchronized php-1.21wmf7/includes/Article.php
02:01 logmsgbot: tstarling synchronized php-1.21wmf7/includes/OutputPage.php
02:01 logmsgbot: tstarling synchronized php-1.21wmf7/includes/DefaultSettings.php
02:00 logmsgbot: tstarling synchronized php-1.21wmf6/includes/ImagePage.php
02:00 logmsgbot: tstarling synchronized php-1.21wmf6/includes/OutputPage.php
01:59 logmsgbot: tstarling synchronized php-1.21wmf6/includes/Article.php
01:59 logmsgbot: tstarling synchronized php-1.21wmf6/includes/DefaultSettings.php

January 6

22:46 logmsgbot: reedy synchronized wmf-config/throttle.php
17:46 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Sun Jan 6 17:46:42 UTC 2013
17:23 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Sun Jan 6 17:23:41 UTC 2013
02:48 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Sun Jan 6 02:48:45 UTC 2013
02:26 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Sun Jan 6 02:26:24 UTC 2013

January 5

19:41 Nemo_bis: en.wiki job queue again way over a million and counting
17:08 paravoid: restarting memcached on virt0
02:53 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Sat Jan 5 02:53:30 UTC 2013
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Sat Jan 5 02:29:41 UTC 2013

January 4

23:29 RobH: authdns-update for xenon and caesium dns
22:46 binasher: temporarily setting up mysql replication on shard es1 again (es([1-4]|100[1-4]) for mha testing
22:35 notpeter: es1001 back in prodcution. all external store are innodb!
22:17 RobH: updated asw-a-eqiad vlan for 4/0/44
21:25 mutante: mauntius/torrus: http://wikitech.wikimedia.org/view/Torrus#Deadlock_problem
21:23 RobH: authdns-update for some mgmt name changes
21:23 mutante: restarting torrus-common on manutius
21:01 cmjohnson1: db1042 removing/replacing bad hdd from slot 9
20:52 cmjohnson1: db1038 removing bad hdd from slot6 for replacement
20:02 Ryan_Lane: depooling ssl1001
18:29 logmsgbot: robh synchronized wmf-config/PoolCounterSettings.php
15:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikidatawiki to 1.21wmf7
15:08 logmsgbot: demon Finished syncing Wikimedia installation... :
14:25 logmsgbot: demon Started syncing Wikimedia installation... :
02:49 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Fri Jan 4 02:49:14 UTC 2013
02:47 logmsgbot: asher synchronized wmf-config/db.php 'returning db64 to s3'
02:27 binasher: restarted replication on db64 from db34
02:26 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Fri Jan 4 02:26:53 UTC 2013
02:20 binasher: truncated ( de | it )wikivoyage.searchindex tables
00:37 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo to Master
00:19 notpeter: starting innobackupex from es1002 to es1001
00:15 RobH: modifying kuo bios power profile from DAPC to OS performance per watt setting in attempt to resolve runaway process issue
00:12 notpeter: converting es1001 to innodb
00:11 binasher: rebuilding db64 via hotbackup of db1010
00:02 logmsgbot: asher synchronized wmf-config/db.php 'pulling db64 from s3'
00:02 RobH: taking down kuo, its power settings are freaking it out, will fix and bring back into servie

January 3

23:10 logmsgbot: olivneh synchronized php-1.21wmf6/extensions/EventLogging 'Updating display of rev IDs in schema subtitles'
23:02 RobH: authdns-updateexit
22:59 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo to Master
22:34 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'switched captchas back to swift'
22:18 binasher: reprepro includedeb (lucid|precise)-wikimedia mha4mysql-manager_0.55-0_all.deb mha4mysql-node_0.54-0_all.deb
21:30 logmsgbot: reedy synchronized php-1.21wmf7/cache/interwiki.cdb 'Updating 1.21wmf7 interwiki cache'
21:30 logmsgbot: reedy synchronized php-1.21wmf6/cache/interwiki.cdb 'Updating 1.21wmf6 interwiki cache'
21:30 RobH: reinstalling tarin to internal ip for secondary poolcounter server in tampa
21:13 paravoid: powercycling cp1026, load spike, unresponsive
20:38 logmsgbot: reedy synchronized php-1.21wmf7/cache/interwiki.cdb 'Updating interwiki cache'
20:38 logmsgbot: reedy synchronized php-1.21wmf6/cache/interwiki.cdb 'Updating interwiki cache'
20:02 RobH: authdns-update for iodine
20:01 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable email batch for Echo'
19:49 notpeter: temp stopping puppet on brewster
19:19 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: and aswikisource back again
19:18 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: remove aswikisource temporarily
19:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
19:13 logmsgbot: reedy synchronized wmf-config/
19:12 LeslieCarr: restarting puppetmasterd on stafford
19:04 logmsgbot: reedy synchronized php-1.21wmf7/extensions/WikimediaMaintenance/
18:59 logmsgbot: reedy synchronized wmf-config/db.php
18:54 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
18:53 logmsgbot: reedy synchronized wmf-config/
18:51 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
18:49 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
18:47 logmsgbot: reedy synchronized wmf-config/
18:35 mutante: restarting nagios on spence
18:34 logmsgbot: reedy synchronized php-1.21wmf6/extensions/WikimediaMaintenance/
18:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
02:46 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Thu Jan 3 02:46:37 UTC 2013
02:26 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Thu Jan 3 02:26:45 UTC 2013
01:02 binasher: restarting all bits varnish instances with a 180s stagger between each (applying shm_reclen change)
01:01 RobH: dns update to move tarin to internal ip
00:54 RobH: poolcounter test shows numbers rising on ersch
00:54 logmsgbot: robh synchronized wmf-config/PoolCounterSettings.php
00:32 logmsgbot: olivneh synchronized php-1.21wmf6/extensions/EventLogging/includes/JsonSchemaHooks.php

January 2

23:20 logmsgbot: maxsem synchronized php-1.21wmf7/extensions/MobileFrontend 'Revert MobileFrontend to the same revision as 1.21wmf6'
22:20 paravoid: starting nginx on ssl3003
22:13 paravoid: deploying squid config for Accept-Language/X-Carrier, RT 2745 & 3158
22:03 paravoid: downgrading squid from 3.1 to 2.7 on fenari
22:03 paravoid: reprepro copysrc precise-wikimedia lucid-wikimedia squid
21:39 Ryan_Lane: powercycled ssl3001
21:34 paravoid: starting nginx on ssl3002
20:33 Reedy: Modified user_groups.ug_group to varbinary(255)
20:29 Reedy: Modified user_former_groups.ufg_group to varbinary(255)
20:24 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikidatawiki back to 1.21wmf6
20:03 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki, wikidatawiki and mediawikiwiki to 1.21wmf7
19:53 logmsgbot: reedy synchronized php-1.21wmf7/extensions/
19:48 logmsgbot: aaron synchronized php-1.21wmf7/extensions/ConfirmEdit 'deployed 8fddb4a805ee3e79ef07ef3ca78a7ae7df5f4bba'
19:26 Ryan_Lane: make that labs.
19:26 Ryan_Lane: changing novaadmin password in labvs
19:21 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild message cache for 1.21wmf7
18:26 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild message cache for 1.21wmf7
18:23 logmsgbot: aaron synchronized php-1.21wmf6/extensions/ConfirmEdit 'deployed 74c5543fead84a8460be51ffa0b16104cfc3abd1'
18:17 Reedy: Running sync-common on srv191
18:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki to 1.21wmf7 before rebuilding message cache
18:14 logmsgbot: reedy synchronized live-1.5/
18:14 cmjohnson1: powercycling srv191
18:13 logmsgbot: reedy synchronized wmf-config/
18:12 logmsgbot: reedy synchronized php-1.21wmf7/ 'Initial sync'
18:09 logmsgbot: aaron synchronized wmf-config/CommonSettings.php
17:35 cmjohnson1: db1014 removing bad hdd from slot11 to replace w/new rt 4039
16:54 LeslieCarr: pulling sfp uplink module on asw-c-eqiad
15:15 cmjohnson1: swapping ethernet cable for ms-be1003
14:34 Nemo_bis: MaxSem> can someone restart labsconsole's memcached which appears to be down again? 14.14 paravoid> done
09:36 logmsgbot: krinkle synchronized php-1.21wmf6/resources/mediawiki.page/mediawiki.page.watch.ajax.js 'I2b1b34c9'
02:27 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Wed Jan 2 02:27:26 UTC 2013

January 1

19:51 apergos: shot hung converts on all image scalers
02:32 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Tue Jan 1 02:32:04 UTC 2013
01:20 logmsgbot: aaron synchronized wmf-config/CommonSettings.php
00:42 logmsgbot: aaron synchronized php-1.21wmf6/extensions/ConfirmEdit
00:16 logmsgbot: aaron synchronized wmf-config/CommonSettings.php

December 31

16:27 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Deploying I9be1e7ac, testwiki permission changes'
02:26 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Mon Dec 31 02:26:11 UTC 2012

December 30

22:56 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
20:35 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
15:40 logmsgbot: reedy synchronized php-1.21wmf6/extensions/ParserFunctions
09:05 Nemo_bis: Search reported broken with no results at all returned on en.wikt, (en|ru).source etc. "Lucene on search14 is CRITICAL" since 3h ago.
02:26 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Sun Dec 30 02:26:16 UTC 2012

December 29

02:27 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Sat Dec 29 02:27:24 UTC 2012

December 28

18:07 Jeff_Green: restart udp2log on emery to clean up 838 defunct processes
16:47 logmsgbot: maxsem synchronized php-1.21wmf6/extensions/PageImages/PageImages.body.php 'https://gerrit.wikimedia.org/r/#/c/40812/'
16:26 cmjohnson1: authdns-update adding mc17/18 to pmtpa mgmt zonefile
05:54 apergos: restarted varnish front and back ends on cp1023, it was swapping
02:57 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Fri Dec 28 02:28:21 UTC 2012

December 27

22:57 logmsgbot: tstarling Finished syncing Wikimedia installation... :
22:53 logmsgbot: tstarling Started syncing Wikimedia installation... :
22:51 logmsgbot: tstarling Finished syncing Wikimedia installation... :
22:47 logmsgbot: tstarling Started syncing Wikimedia installation... :
22:42 logmsgbot: tstarling Finished syncing Wikimedia installation... :
22:39 logmsgbot: tstarling Started syncing Wikimedia installation... :
22:28 logmsgbot: tstarling Started syncing Wikimedia installation... :
22:27 logmsgbot: tstarling Finished syncing Wikimedia installation... :
22:13 logmsgbot: tstarling Started syncing Wikimedia installation... :
22:02 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php
15:33 cmjohnson1: shutting down mw57 to troubleshoot DIMM/Mem issue
06:47 logmsgbot: tstarling synchronized live-1.5/mobileRedirect.php
02:21 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Thu Dec 27 02:21:50 UTC 2012
02:09 Tim: on db64: killed FlaggedRevsStats::getEditReviewTimes queries, were running for 40 days. Client host was hume but no process was found with the relevant ephemeral port.
00:21 binasher: powercycled cp1044, couldn't ssh in
00:14 logmsgbot: reedy synchronized php-1.21wmf6/cache/l10n

December 26

23:27 binasher: restarted varnishhtcpd on all eqiad upload varnish hosts; looking back fondly on vacation
23:22 jeremyb: 2012-12-26 22:51:58 <+logmsgbot> !log maxsem synchronized wmf-config/InitialiseSettings.php 'Enable PageImages everywhere'
23:21 jeremyb: 2012-12-26 22:31:37 <+logmsgbot> !log maxsem synchronized wmf-config/InitialiseSettings.php 'Enable PageImages on test2'
23:21 jeremyb: 2012-12-26 22:25:53 <+logmsgbot> !log awjrichards synchronized php-1.21wmf6/extensions/MobileFrontend/ 'touch files'
23:21 jeremyb: 2012-12-26 22:23:19 <+logmsgbot> !log awjrichards Finished syncing Wikimedia installation... : MobileFrontend updates per https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-12-26
23:21 jeremyb: 2012-12-26 21:59:47 <+logmsgbot> !log awjrichards Started syncing Wikimedia installation... : MobileFrontend updates per https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-12-26
23:21 jeremyb: 2012-12-26 21:23:38 <+logmsgbot> !log maxsem synchronized wmf-config/CommonSettings.php 'Fix PageImages inclusion'
23:21 jeremyb: 2012-12-26 21:11:03 <+logmsgbot> !log awjrichards synchronized wmf-config/InitialiseSettings.php 'Enabling PageImages on testwiki'
23:21 jeremyb: 2012-12-26 21:10:34 <+logmsgbot> !log awjrichards synchronized wmf-config/CommonSettings.php 'Enabling PageImages on testwiki'
23:20 jeremyb: 2012-12-26 02:05:45 <+logmsgbot> !log LocalisationUpdate failed: git pull of extensions failed
23:20 jeremyb: 2012-12-25 11:03:46 <+logmsgbot> !log maxsem synchronized wmf-config/CommonSettings.php 'Disable broken Swift captchas on mw.o too'
23:20 jeremyb: 2012-12-25 10:58:38 <+logmsgbot> !log maxsem synchronized wmf-config/CommonSettings.php 'Captchas broken on every wiki'
23:20 jeremyb: 2012-12-25 05:45:08 < Ryan_Lane> !log restarting gerrit. also blocked a bad bot
23:20 jeremyb: 2012-12-25 02:23:37 <+logmsgbot> !log LocalisationUpdate completed (1.21wmf6) at Tue Dec 25 02:23:37 UTC 2012
23:19 jeremyb: 2012-12-24 14:49:25 < apergos> !log rebooting cp1043 which had fallen over
23:19 jeremyb: 2012-12-24 13:28:34 <+logmsgbot> !log reedy synchronized live-1.5/
23:19 jeremyb: !logging what morebots missed
23:08 LeslieCarr: restarted morebots

December 24

02:29 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Mon Dec 24 02:29:30 UTC 2012
00:44 logmsgbot: tstarling synchronized php-1.21wmf6/extensions/ProofreadPage/ProofreadPage.body.php

December 23

23:24 logmsgbot: reedy synchronized php-1.21wmf6/includes/filebackend/FSFileBackend.php
22:31 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:04 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Sun Dec 23 02:29:31 UTC 2012

December 22

17:02 logmsgbot: reedy synchronized wmf-config/
16:50 Reedy: Created WikiLove tables on itwikivoyage
06:39 apergos: shot a bunch of hung converts again on the imagescalers
02:26 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Sat Dec 22 02:26:37 UTC 2012

December 21

21:33 Ryan_Lane: also upgrading: libzmq3, python-zmq on all minions
21:33 Ryan_Lane: upgrading salt on all minions
18:45 mutante: re-enabled password protection on ganglia
18:39 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Switched all wikis to swift for captchas'
14:55 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'Enable GeoData jobs everywhere'
14:43 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'Try GeoData jobs on enwiki'
14:23 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'Enable GeoData on wikiedias and explicitly on test2'
14:18 hashar: Enabling whitespace checking on mediawiki-core-lint
14:16 MaxSem: Load test passed successfully
13:58 MaxSem: Load-testing geosearch
13:46 MaxSem: Created GeoData tables for wikis in special.dblist
13:42 hashar: Zuul pipeline gate-and-submit will now add a message in Gerrit when it start. Would let people know the jobs have started after they voted CR+2 39584
13:36 logmsgbot: hashar synchronized wmf-config/CommonSettings-labs.php 'beta: 39742'
10:32 logmsgbot: hashar synchronized all-labs.dblist
10:32 logmsgbot: hashar synchronized wmf-config
10:23 hashar: Deploying MediaWiki config changes for labs, basically renaming files from -wmflabs to -labs. 39059 and 39060
09:35 logmsgbot: maxsem synchronized wmf-config/CommonSettings.php 'Switch GeoData to solr1001'
09:10 MaxSem: Populating solr1001 with enwiki coordinates
08:59 logmsgbot: maxsem synchronized php-1.21wmf5/extensions/GeoData/ 'Update GeoData to master to fix job problems'
08:58 logmsgbot: maxsem synchronized php-1.21wmf6/extensions/GeoData/ 'Update GeoData to master to fix job problems'
02:27 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Fri Dec 21 02:27:16 UTC 2012
00:11 hashar: Doc to apply a change to Zuul / reload it is at https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Changing_the_configuration (logging there so people find it)

December 20

23:44 logmsgbot: olivneh synchronized php-1.21wmf6/extensions/GettingStarted/resources/ext.gettingstarted.accountcreation.js 'Fix-up: reference correct ID in selector'
23:22 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Enabling PostEdit on test2wiki'
22:34 hashar: updating Zuul code base and running puppet manually on gallium
22:23 mutante: dumping pilot_1_16wmf4 database on kaulen, putting dump on tridge and removing mysqld from kaulen
22:13 logmsgbot: olivneh synchronized php-1.21wmf6/extensions/GettingStarted 'Interface update, pt. 2'
22:00 logmsgbot: olivneh synchronized php-1.21wmf6/extensions/GettingStarted 'Interface update'
21:54 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Enabling Extension:PostEdit for astwiki & trwiki'
21:43 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Enabling Extension:PostEdit for itwikivoyage'
21:35 logmsgbot: olivneh synchronized php-1.21wmf6/extensions/E3Experiments
21:27 logmsgbot: olivneh synchronized php-1.21wmf6/extensions/EventLogging 'Roll out stricter schema validation'
20:52 logmsgbot: mlitn synchronized php-1.21wmf6/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
20:16 ottomata: copied js/ and css/ dirs from nickel:/srv/org/wikimedia/ganglia-web-3.5.1 to nickel:/srv/org/wikimedia/ganglia-web-3.5.4
17:55 cmjohnson1: authdns update adding pappas mgmt to zone file
17:15 MaxSem: Cleaned up job queue from stuck solrUpdate jobs
16:58 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/39595'
14:50 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/39576/'
13:53 hashar: Jenkins: deleting the old MediaWiki-* jobs which used the Gerrit Trigger plugin.
13:45 logmsgbot: hashar synchronized php-1.21wmf6/extensions/WikimediaMaintenance 'Bring in 32169 : per-realm and per-datacenter configuration'
13:23 hashar: reconfiguring all Jenkins jobs
09:49 logmsgbot: hashar synchronized wmf-config/ 'ext file now uses realm for consistency 39058. That makes CheckUser to load only in production. Checked on test.wp.o'
09:38 hashar: ran sync-common on hume
09:38 logmsgbot: hashar synchronized wmf-config 'use $wmfRealm in switches instead of legacy $cluster 39057'
09:35 logmsgbot: hashar synchronized live-1.5/robots.php 'use $wmfRealm in switches instead of legacy $cluster 39057'
09:21 logmsgbot: hashar synchronized multiversion/MWMultiVersion.php 'now requires multiversion/MWRealm.php 39543'
09:20 logmsgbot: hashar synchronized multiversion/activeMWVersions 'now requires multiversion/MWRealm.php 39543'
09:20 logmsgbot: hashar synchronized multiversion/backupWikiversions 'now requires multiversion/MWRealm.php 39543'
09:19 logmsgbot: hashar synchronized refresh-dblist 'now requires multiversion/MWRealm.php 39543'
08:31 logmsgbot: hashar synchronized tests/multiversion/MWRealmTest.php
08:31 logmsgbot: hashar synchronized multiversion/MWRealm.sh
08:31 logmsgbot: hashar synchronized multiversion/MWRealm.php
08:29 hashar: going to deploy Mediawiki config changes related to the multiple datacenter support.
02:27 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Thu Dec 20 02:27:33 UTC 2012
01:48 logmsgbot: kaldari synchronized php-1.21wmf6/extensions/Echo 'Update Echo extension'
01:20 logmsgbot: kaldari synchronized php-1.21wmf6/extensions/Echo 'Update Echo extension'
00:37 logmsgbot: kaldari synchronized php-1.21wmf6/extensions/Echo 'Update Echo extension'
00:20 andrewbogott: removed some mariadb packages from brewster: precise-wikimedia libmysqlclient18, mariadb-common, mysql-common, all in precise-wikimedia. They were causing dependency problems that broke the install of (for example) generic::mysql::server
00:19 AaronSchulz: Running refreshFileHeaders.php on all wikis

December 19

23:53 mutante: neon - install nagios-nrpe-plugin with --no-install-recommends
23:07 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'use swift for captchas for testwiki/mediawikiwiki'
23:07 logmsgbot: aaron About 200K "fancy" captchas added (there were ~10K before)
20:44 cmjohnson1: authdns update solr1001/1002 correction
20:28 logmsgbot: demon synchronized php-1.21wmf6/extensions/ParserFunctions/ 'Rolling back pfuncs to d96a17c'
19:59 cmjohnson1: authdns update correcting solr1003 entry
19:00 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: All 'pedias to 1.21wmf6
16:57 logmsgbot: reedy synchronized php-1.21wmf6/extensions/Wikibase
16:39 logmsgbot: reedy synchronized php-1.21wmf6/extensions/CategoryTree
15:37 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php
15:37 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php
15:36 logmsgbot: hashar synchronized wmf-config/CommonSettings.php
14:03 hashar: gallium: updated PHPUnit to 3.7.10 thus solving bug 42724
13:48 hashar: gallium: apt-get install pep8 v1.3.3 (backported from raring)
11:53 MaxSem: Updated solr, cleaning up killlist
09:31 hashar: Jenkins: enabled unit test run on mw/core for some whitelisted people 39310
07:06 apergos: shot a bunch of converts on the image scalers, looks like a couple started flapping about 7-8 hours ago
04:03 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable Echo on test2 and mediawiki'
03:59 logmsgbot: kaldari Finished syncing Wikimedia installation... :
03:31 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Wed Dec 19 03:31:24 UTC 2012
03:24 logmsgbot: kaldari Started syncing Wikimedia installation... :
02:48 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Wed Dec 19 02:48:16 UTC 2012
02:02 logmsgbot: awjrichards synchronized php-1.21wmf5/extensions/MobileFrontend/javascripts/modules/mf-toggle.js 'touch file
02:01 logmsgbot: awjrichards synchronized php-1.21wmf5/extensions/MobileFrontend/javascripts/common/main.js
01:44 logmsgbot: awjrichards synchronized php-1.21wmf6/extensions/MobileFrontend/stylesheets/modules/mf-cleanuptemplates.css 'touch file'
01:44 Ryan_Lane: finished upgrading salt on all production minions
01:41 logmsgbot: awjrichards synchronized php-1.21wmf6/extensions/MobileFrontend/javascripts/modules/mf-cleanuptemplates.js
01:16 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable Echo on test server'
01:14 logmsgbot: demon synchronized php-1.21wmf6/extensions/Wikibase/lib/resources/templates.js 'Deploying I641725a2'
00:46 notpeter: taking down for another reinstall (this time with raid!)
00:33 logmsgbot: awjrichards synchronized php-1.21wmf6/extensions/MobileFrontend/javascripts/common/main.js 'touch file'
00:26 Ryan_Lane: make that modifying project groups in labs by running syncProjectGroups.php maintenance script in OpenStackManager
00:25 Ryan_Lane: modifying project groups

December 18

23:40 logmsgbot: maxsem Finished syncing Wikimedia installation... : https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-12-18
22:59 notpeter: temp putting ganglia.w.o behind htaccess for sec reasons
22:54 logmsgbot: maxsem Started syncing Wikimedia installation... : https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-12-18
22:46 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'switched new captcha setting from testwiki -> test2wiki'
22:43 logmsgbot: aaron synchronized wmf-config/filebackend.php 'set captcha directory.'
22:42 Ryan_Lane: deploying Andrew Otto's group changes to OpenStackManager to labsconsole
22:07 logmsgbot: aaron synchronized php-1.21wmf6/extensions/ConfirmEdit/captcha.py
21:32 cmjohnson1: authdns-update new dns entries for frack bastion host (tellurium)
21:08 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Turns off Echo temporarily on test and test2'
20:48 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php 'Update Echo config file'
20:47 logmsgbot: py gracefulled all apaches
20:44 notpeter: gracefulling all apaches to pick up https://gerrit.wikimedia.org/r/#/c/38521/ (tested good on srv193)
20:44 hashar: running puppet on gallium.
20:44 hashar: Zuul: applying "filters events by user email" to our Zuul deployment https://review.openstack.org/#/c/17609/
20:27 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php
20:26 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php
18:31 LeslieCarr: asw-c-eqiad ae bundles went down, working on fixing
15:45 MaxSem: Testing done, 40 concurrent processes hitting around the worst-case point kept the load on yttrium at 20%. Average response time ~430ms
15:19 MaxSem: Load-testing spatial search
14:14 hashar: restarting Zuul with https://gerrit.wikimedia.org/r/39082 so it starts voting Verified+2
14:13 ^demon: restarting gerrit on manganese to pick up VRIF+2
13:50 hashar: restarted puppet on gallium (some apt-get process was a zombie)
09:25 logmsgbot: nikerabbit synchronized wmf-config/CommonSettings.php 'Bug 43075'
06:30 andrewbogott: switched all labs instances to mount /home via gluster on next reboot
06:29 andrewbogott: rsynced all labs homedirs to gluster volumes
02:46 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Tue Dec 18 02:45:58 UTC 2012
02:25 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Tue Dec 18 02:25:22 UTC 2012
01:28 mutante: fixing duplicate UID issue on stat1 for maryana
00:57 LeslieCarr: asw-c-eqiad unreachable due to lacp issue
00:32 mutante: fixing fenari permissions for gwicke.. (pre-puppet age UID)
00:26 LeslieCarr: starting upgrade of asw-c-eqiad.mgmt - connectivity to row c machines may be affected
00:11 notpeter: temp stopping lsave on es1009 and es1010 for upcoming networking downtime

December 17

23:12 LeslieCarr: restarted pybal on lvs1001-1003 in order to restart their bgp peering
22:44 notpeter: taking fenari down for upgarde to precise (not upgrading, not reimaging)
22:37 LeslieCarr: cr1-eqiad being upgraded and rebooted
22:30 Ryan_Lane: labstore1 is locked up, powercycling
22:29 Nemo_bis: en.wiki job queue spiked from 1 to 3 millions in last 3 hours
21:22 Ryan_Lane: rebooting labstore4
21:15 logmsgbot: reedy synchronized php-1.21wmf6/includes/filebackend/FSFileBackend.php
21:11 Ryan_Lane: rebooting labstore3
21:03 Ryan_Lane: rebooting labstore2
20:48 Ryan_Lane: restarting labstore1
20:33 cmjohnson1: auth-dns update to add internal ip's for solr1-3
19:35 hashar: regenerating Jenkins job mediawiki-core-install-sqlite
19:12 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf6
18:15 andrewbogott: beginning labs $HOME migration from NFS to gluster
16:16 cmjohnson1: auth-dns update adding mgmt for solr1-3, solr1001-3 and internal ip's for solr1001-3
13:13 hashar: set Jenkins to use /bin/bash as a default shell (intend of /bin/sh)
04:02 paravoid: killall -9 convert on imagescalers; uploading 120px generated thumbnail directly to swift
02:50 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Mon Dec 17 02:50:11 UTC 2012
02:27 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Mon Dec 17 02:27:34 UTC 2012

December 16

17:14 paravoid: "killall -9 convert" on all imagescalers
15:55 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
15:54 logmsgbot: reedy synchronized images/sul/wikimania.png
15:36 logmsgbot: reedy synchronized images/sul/
02:47 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Sun Dec 16 02:47:42 UTC 2012
02:45 LeslieCarr: mwscript.php appears to be broken
02:40 LeslieCarr: checking memcache servers due to spate of mediawiki servers flapping
02:26 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Sun Dec 16 02:26:23 UTC 2012

December 15

14:37 logmsgbot: maxsem synchronized /php-1.21wmf5/includes/Message.php 'https://gerrit.wikimedia.org/r/#/c/38822/'
14:30 paravoid: powercycling ms-be1
12:18 apergos: ms-be1 can't be reached form mgmt console to check on it (Connection refused)
02:47 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Sat Dec 15 02:47:17 UTC 2012
02:45 LeslieCarr: drained all transit traffic from cr1-eqiad -- restore by load override /var/home/lcarr/cr1-eqiad.undrained if traffic levels seem too high
02:25 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Sat Dec 15 02:25:26 UTC 2012
00:36 LeslieCarr: appears that cr1-eqiad's fpc rebooted

December 14

23:25 AaronSchulz: Compiled texvc into uncommon/php-1.21wmf6 dir for all apaches
23:08 K4-713: synchronized payments to 4f5296e43b1c708
22:40 hashar: I am not sure what is happening between Gerrit and Zuul. Seems Gerrit might not have sent all its event via stream-event. I have restarted Zuul and it seems to be happy (test: https://gerrit.wikimedia.org/r/#/c/38800/ ). Will monitor over the weekend.
22:34 mutante: fixing symlink on kaulen to reflect new bugzilla version 4.2.4
22:30 hashar: restarting Zuul on gallium
20:54 mutante: deleting /var/lib/git/integration on gallium
18:54 mutante: appyling mini patch on bugzilla per http://bzr.mozilla.org/bugzilla/4.2/revision/8176
18:31 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable clicktracking on test2wiki'
16:56 sbernardin: sq48 going down for HDD controller card replacement-rt4041
15:40 logmsgbot: demon synchronized wmf-config/throttle.php 'Deploying I330230e6'
09:30 logmsgbot: mlitn synchronized php-1.21wmf5/extensions/ArticleFeedbackv5 'Fix AFTv5 toolbox popups'
02:47 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Fri Dec 14 02:46:58 UTC 2012
02:26 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Fri Dec 14 02:25:58 UTC 2012
01:50 mutante: restarting pdns on ns0
01:35 mutante: creating new self-signed SSL cert for wikitech - valid for another year
01:03 logmsgbot: csteipp synchronized php-1.21wmf6/extensions/CategoryTree/CategoryTreeFunctions.php
00:15 logmsgbot: olivneh synchronized php-1.21wmf6/extensions/AbuseFilter
00:15 mutante: prototype.wikimedia is also removed ... bye or please complain to us about it
00:13 logmsgbot: olivneh synchronized php-1.21wmf5/extensions/AbuseFilter
00:09 mutante: DNS update - killing a bunch of .tesla.

December 13

23:55 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Enabling GettingStarted on enwiki'
23:42 logmsgbot: olivneh synchronized wmf-config/abusefilter.php '(bug 42758) enable AbuseFilter events for enwiki irc feed'
23:40 mutante: gallium mysql (jenkins/CI) back up and running after brief db server issue
23:34 logmsgbot: olivneh synchronized php-1.21wmf6/extensions/AbuseFilter
23:33 logmsgbot: olivneh synchronized php-1.21wmf5/extensions/AbuseFilter
23:30 mutante: mysql stopped on gallium due to dpkg issue.. investigating
23:28 mutante: installing package upgrades on gallium, incl. newer mysql packages
23:20 logmsgbot: olivneh synchronized php-1.21wmf6/extensions/E3Experiments/experiments/acux/accountCreationUX.js
23:17 logmsgbot: olivneh synchronized php-1.21wmf5/extensions/E3Experiments/experiments/acux/accountCreationUX.js
23:14 logmsgbot: spage Finished syncing Wikimedia installation... : E3 deploy new Special:GettingStarted and E3 logging
23:03 LeslieCarr: switched asw-c-eqiad to cr2-eqiad aggregated bundle from lacp to static
22:34 logmsgbot: spage Started syncing Wikimedia installation... : E3 deploy new Special:GettingStarted and E3 logging
20:10 logmsgbot: mlitn synchronized php-1.21wmf5/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
19:39 notpeter: restarting all lucene instances in pmtpa
19:27 notpeter: stopping puppet on brewster
19:01 notpeter: upgrading lucene to 2.1.6wm1 on all search boxes in pmtpa
18:55 notpeter: upgrading lucene to 2.1.6wm1 on all search boxes in eqiad
18:51 notpeter: taking down hume for disk replacement/reimage/etc
18:50 paravoid: nas1-a/nas1001-a: resizing originals volume to 37T and thumbs volume to 6T
16:54 logmsgbot: nikerabbit synchronized php-1.21wmf6/extensions/Translate/ 'Translate bugfix'
14:08 ^demon: live-hacked out manganese -> gallium gerrit replication. spamming logs with "reject HostKey: gallium.wikimedia.org". restarted gerrit on manganese.
13:53 MaxSem: Updated enwiki Solr index
10:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
02:47 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Thu Dec 13 02:47:24 UTC 2012
02:26 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Thu Dec 13 02:26:13 UTC 2012
00:59 logmsgbot: aaron synchronized php-1.21wmf5/includes/job/JobQueueDB.php 'deployed c3147eb053e0fb4c60f8feddc7de1bc363210d02'
00:50 RoanKattouw: Force-running puppet on celsus
00:08 logmsgbot: aaron synchronized php-1.21wmf6/includes/job/JobQueueDB.php 'deployed 2c177fa087bb09bc0915413ea2f628dae5c0e986'

December 12

23:22 logmsgbot: catrope synchronized php-1.21wmf6/extensions/VisualEditor/ 'Updating VisualEditor to master'
23:20 logmsgbot: catrope synchronized php-1.21wmf5/extensions/VisualEditor/ 'Updating VisualEditor to master'
22:45 mutante: wikipedia.co.za redirects to af.wikipedia.org now
22:43 logmsgbot: dzahn gracefulled all apaches
22:42 mutante: restarting Apaches to enable wikipedia.co.za
21:57 logmsgbot: reedy synchronized php-1.21wmf6/includes/api/
21:20 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commons back to wmf6 for testding
21:05 logmsgbot: nikerabbit synchronized php-1.21wmf6/extensions/Translate 'Translate bug fix'
20:55 logmsgbot: reedy synchronized wmf-config/
20:39 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote to wmf6
20:37 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks, wikimedia and wikivoyage to 1.21wmf6
20:26 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity and wikinews to wmf6
20:12 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Reverted commonswiki to 1.21wmf5
19:52 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
19:41 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource and wiktionary to 1.21wmf6
19:37 logmsgbot: catrope synchronized php-1.21wmf6/cache/l10n/
19:32 mutante: removing test-star cert and key from all ssl hosts
19:22 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Closed, special and private to 1.21wmf6
18:42 logmsgbot: reedy synchronized wmf-config/
18:21 logmsgbot: catrope synchronized php-1.21wmf6/extensions/VisualEditor/ 'Looks like this did not get synced right earlier'
18:16 RoanKattouw: Rebuilding l10n cache for wmf6 to fight broken messages for VisualEditor on wmf6 wikis
18:13 logmsgbot: reedy synchronized wmf-config/
18:03 logmsgbot: reedy synchronized wmf-config/
11:53 MaxSem: done: real 8m1.864s user 1m32.940s sys 0m0.900s
11:44 MaxSem: Populating Solr index for enwiki
11:34 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
11:29 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
11:26 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
11:22 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/38300'
11:06 MaxSem: Populated Solr index on testwiki
03:26 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable VisualEditor on enwiki'
03:12 logmsgbot: catrope synchronized php-1.21wmf5/extensions/VisualEditor 'touch'
03:06 logmsgbot: catrope synchronized php-1.21wmf5/resources/startup.js 'touch'
03:04 logmsgbot: catrope synchronized php-1.21wmf5/extensions/VisualEditor/modules/ve/init/mw/targets/ve.init.mw.ViewPageTarget.js 'touch'
02:48 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Wed Dec 12 02:48:46 UTC 2012
02:43 logmsgbot: catrope synchronized php-1.21wmf5/extensions/VisualEditor/modules/ve/init/mw/targets/ve.init.mw.ViewPageTarget.js 'touch'
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Wed Dec 12 02:28:55 UTC 2012
02:17 logmsgbot: catrope synchronized php-1.21wmf6/extensions/VisualEditor 'Updating VisualEditor'
02:17 logmsgbot: catrope synchronized php-1.21wmf5/extensions/VisualEditor 'Updating VisualEditor'
02:07 logmsgbot: catrope synchronized php-1.21wmf6/extensions/VisualEditor 'Updating VisualEditor'
02:07 logmsgbot: catrope synchronized php-1.21wmf5/extensions/VisualEditor 'Updating VisualEditor'
00:10 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Send parsoid requests to celsus (aa2903d81515d543f09dd41d1ad844d1b5e79cf3)'
00:08 logmsgbot: catrope synchronized php-1.21wmf6/extensions/VisualEditor 'Updating VisualEditor'
00:07 logmsgbot: catrope synchronized php-1.21wmf5/extensions/VisualEditor 'Updating VisualEditor'

December 11

23:33 LeslieCarr: reverted the priority, messed with BGP_tweak_LP to re-acheive desired results
23:28 logmsgbot: awjrichards synchronized php-1.21wmf5/extensions/MobileFrontend/javascripts/modules/mf-tables.js
23:20 LeslieCarr: changed vrrp priority of row a to have cr1-eqiad for mastership in order to rebalance transit traffic
22:53 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly MobileFrontend deployment + GeoData updates
22:41 mutante: rebuilding archives of wikilovesmonuments list
22:38 Ryan_Lane: rebooting virt0
22:33 Ryan_Lane: apt upgrading virt0
22:15 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly MobileFrontend deployment + GeoData updates
21:38 MaxSem: fixing GeoData tables
20:17 logmsgbot: asher synchronized wmf-config/db.php 'pooling db59 with full weight'
20:15 logmsgbot: asher synchronized wmf-config/db.php 'pooling db59 with a low weight for warmup'
20:14 logmsgbot: asher synchronized wmf-config/db.php 'returning db59 to s1'
19:31 binasher: shutting down mysql on db59 for upgrade
19:30 RoanKattouw: Force-running puppet on celsus and constable for like the millionth time
19:29 logmsgbot: asher synchronized wmf-config/CommonSettings.php 're-enabling mysql pcache'
19:20 logmsgbot: asher synchronized wmf-config/CommonSettings.php 'emergency disable mysql pcache, pushing again'
19:20 RoanKattouw: Also disabled mwext-Translate-install
19:17 RoanKattouw: Disabled mwext-VisualEditor-install in Jenkins, totally broken
19:14 logmsgbot: asher synchronized wmf-config/CommonSettings.php 'emergency disable mysql pcache'
19:03 logmsgbot: asher synchronized wmf-config/db.php 'pulling db59 from s1 for test'
18:44 RoanKattouw: Force-running puppet on celsus and constable
18:25 RoanKattouw: Force-running puppet on celsus and constable again
18:24 mutante: giving all RT accounts a real name value to reduce confusion (nickname,unix login,username,language bla bla is optional)
18:21 RoanKattouw: Edited celsus:/etc/varnish/default.vcl to point to 10.2.1.28:8000
18:12 RoanKattouw: Installing Varnish in celsus
17:43 RoanKattouw: Force-running puppet on constable
16:50 cmjohnson1: correction...mc1010 and mc1002 replacing 10g NIC
16:43 cmjohnson1: mc1009 and mc1002 shutting down to replace 10g NIC
16:26 logmsgbot: reedy synchronized php-1.21wmf6/extensions/ParserFunctions/
15:32 logmsgbot: demon synchronized php-1.21wmf6/extensions/Wikibase/client/includes/store/sql/WikiPageEntityLookup.php
15:27 logmsgbot: demon synchronized php-1.21wmf6/includes/Revision.php 'Syncing I14a7ebb8'
14:44 logmsgbot: maxsem synchronized wmf-config
14:25 sbernardin: sr266 going down for main board replacement rt2896
14:25 sbernardin: sr266 going down for main board replacement
12:32 hashar: changing operations-puppet-validate Jenkins job.
12:25 Nemo_bis: All !Wikipedia and !Wikimedia projects are now up and editable again, s3 was read-only for a few hours for slow DB http://ur1.ca/bz8v0 [1]
12:07 logmsgbot: reedy synchronized php-1.21wmf6/extensions/LabeledSectionTransclusion
09:35 mark: Power cycled nescio
08:43 hashar: restarting Jenkins, it still knows about jobs that no more exist :/
08:40 apergos: stopped job runners temporarily while we look at still-increasing replag on s3
08:03 binasher: restarted all job runners
07:56 logmsgbot: aaron synchronized php-1.21wmf5/includes/job/JobQueueDB.php 'deployed 6f03ada325f985fc0b14c0d0144a42c94268941f'
07:55 logmsgbot: aaron synchronized php-1.21wmf6/includes/job/JobQueueDB.php 'deployed eabed08c91538ffbb651ed7d8867f9966afc2ae5 '
07:42 logmsgbot: aaron synchronized php-1.21wmf6/includes/job/JobQueueDB.php 'deployed 26b9a472df430dc3abaead0e6f5a9f640b075b99'
07:34 binasher: restarting all job runners
07:32 Aaron|home: Also deployed 919b9d8ba3ee7d6a16214b0853e82fa733a0605a and dcd314e837ccf3d2f7c2ca7dc94b7d1e6accfde4
07:31 logmsgbot: aaron synchronized php-1.21wmf5/includes/job/JobQueueDB.php 'reverted live hacks to use slave DB'
07:02 logmsgbot: catrope synchronized php-1.21wmf6/extensions/VisualEditor/modules/ve/init/mw/targets/ve.init.mw.ViewPageTarget.js 'touch'
07:02 logmsgbot: catrope synchronized php-1.21wmf5/extensions/VisualEditor/modules/ve/init/mw/targets/ve.init.mw.ViewPageTarget.js 'touch'
07:00 logmsgbot: olivneh synchronized php-1.21wmf5/extensions/E3Experiments/Experiments.php
06:56 logmsgbot: olivneh synchronized php-1.21wmf5/extensions/EventLogging 'Syncing to avoid obsolete CACHE_MEMCACHE use in prod.'
06:49 logmsgbot: asher synchronized wmf-config/db.php 'pulling db39 from s3, crazy behind on purging'
05:52 logmsgbot: aaron synchronized php-1.21wmf5/includes/objectcache/MemcachedPhpBagOStuff.php
05:51 logmsgbot: aaron synchronized php-1.21wmf5/includes/objectcache/MemcachedPhpBagOStuff.php
05:22 binasher: killed all wikiadmin threads on db39
05:11 logmsgbot: catrope synchronized php-1.21wmf6/extensions/VisualEditor 'Updating VisualEditor to master'
05:11 logmsgbot: catrope synchronized php-1.21wmf5/extensions/VisualEditor 'Updating VisualEditor to master'
04:41 binasher: same with db66
04:38 binasher: killed 22 long running FlaggedRevsStats::getEditReviewTimes on db39, run from hume against enwikinews
04:04 RoanKattouw: Updating Parsoid to master
03:53 mutante: powercycling mexia
03:29 logmsgbot: aaron synchronized php-1.21wmf5/includes/job/JobQueueDB.php
03:29 logmsgbot: aaron synchronized php-1.21wmf5/includes/job/jobs/RefreshLinksJob.php
03:29 RoanKattouw: mexia still hasn't come back up. That combined with the weird behavior I saw before rebooting it makes me suspect a hardware issue
03:12 logmsgbot: aaron synchronized php-1.21wmf6/includes/job/JobQueueDB.php
03:08 logmsgbot: aaron synchronized php-1.21wmf6/includes/job/jobs/RefreshLinksJob.php
03:02 RoanKattouw: Rebooting mexia, seems to have been hit by some kernel bug
02:00 logmsgbot: LocalisationUpdate failed: git pull of core failed
01:11 logmsgbot: pgehres synchronized wmf-config/CommonSettings.php 'Putting donation pipeline back to normal'
01:02 cmjohnson1: replacing disk on slot 8 db1008
00:48 logmsgbot: pgehres synchronized wmf-config/CommonSettings.php 'Putting donation pipeline in maintenance mode, part 2'
00:44 logmsgbot: pgehres synchronized wmf-config/CommonSettings.php 'Putting donation pipeline in maintenance mode'
00:32 K4-713: synchronized payments cluster to 325a919967709359c8
00:05 logmsgbot: pgehres synchronized php-1.21wmf6/extensions/FundraiserLandingPage/ 'Updating FundraiserLandingPage to master'
00:04 logmsgbot: pgehres synchronized php-1.21wmf6/extensions/ContributionTracking/ 'Updating ContributionTracking to master'

December 10

23:17 logmsgbot: pgehres synchronized php-1.21wmf5/extensions/ContributionTracking/ 'Updating ContributionTracking to master'
23:10 K4-713: synchronized payments to 72c1985a2b9b1db
22:45 LeslieCarr: removed /home/wikipedia/common/php-1.21wmf1/ from fenari
22:08 mutante: mobile and non-mobile Wikipedia now use just ONE certificate.. a new "star.wp" with an ".m." in it, valid until 2016 by DigiCert
21:57 mutante: one more nginx restart on ssl boxes for mobile cert
21:18 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: wikidatawiki & mediawikiwiki to 1.21wmf6
21:03 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki also on 1.21wmf6
21:03 logmsgbot: demon Finished syncing Wikimedia installation... :
20:27 logmsgbot: demon Started syncing Wikimedia installation... :
20:19 logmsgbot: asher synchronized wmf-config/CommonSettings.php 're-enabling pc1 for sql: the bag o stuffening'
20:11 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files:
20:07 logmsgbot: demon Finished syncing Wikimedia installation... :
20:03 logmsgbot: demon Started syncing Wikimedia installation... :
20:01 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.21wmf5
19:59 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki to 1.21wmf6
19:58 logmsgbot: demon ran sync-common-all
19:50 mutante: replaced wikipedia SSL cert with new DigiCert (rather than RapidSSL) (RT-3639)
19:40 mutante: restarting nginx on all ssl hosts
19:38 mutante: repooling ssl4
19:11 mutante: fixing test-star vs. old-star and new-star WP certificate/key confusion on ssl hosts.. arrg
18:51 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'New link for Special:Version'
18:25 mutante: temp. depooling ssl4
17:58 logmsgbot: demon synchronized wmf-config/extdist/svn-invoker.conf 'More futile efforts to fix extdist'
12:13 logmsgbot: maxsem synchronized wmf-config/CommonSettings.php 'https://gerrit.wikimedia.org/r/#/c/37823/'
02:24 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Mon Dec 10 02:24:44 UTC 2012

December 9

12:05 Nemo_bis: job queue snapshot for bug 42614: en.wiki 900k still raising steadily, Commons 70k, fr.wiki 4.5M, Meta 2k, other tops 1k at most
05:22 Ryan_Lane: rebooted wikitech and cleared some logs to free up /var (which was full)

December 8

14:32 logmsgbot: reedy synchronized live-1.5/
14:31 logmsgbot: reedy synchronized wmf-config/
14:04 logmsgbot: reedy synchronized php-1.21wmf5/cache/l10n
13:50 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Sat Dec 8 13:50:26 UTC 2012
02:26 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Sat Dec 8 02:26:46 UTC 2012

December 7

23:46 logmsgbot: olivneh synchronized php-1.21wmf5/extensions/EventLogging 'Fix-up of mtime issue'
22:07 mutante: DNS update - add icinga.wm as cname for neon
21:37 logmsgbot: lcarr synchronized wmf-config/throttle.php
21:22 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'EventLogging log group -> '
21:02 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Updating wfDebugLogGroups dest for EventLogging'
20:18 logmsgbot: olivneh synchronized php-1.21wmf5/extensions/E3Experiments
20:00 K4-713: synchronized payments cluster to 351b8cfa58dc99
20:00 binasher: rebooting pc1
19:29 binasher: upgrading pc1 to precise and newer mysql-facebook build
18:52 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'disabled use of pc1'
18:05 RoanKattouw: Ran killall -u extdist on fenari per Chad's request
17:33 logmsgbot: reedy synchronized php-1.21wmf5/extensions/CentralAuth
16:36 logmsgbot: maxsem synchronized php-1.21wmf5/extensions/GeoData 'https://gerrit.wikimedia.org/r/#/c/37446/'
16:22 RobH: authdns-update for durant mgmt
15:57 hashar: Jenkins now auto submit changes in mediawiki/core.git whenever tests are successful 37420
15:39 logmsgbot: reedy synchronized php-1.21wmf5/extensions/WikimediaMaintenance
15:22 logmsgbot: reedy synchronized php-1.21wmf5/extensions/WikimediaMaintenance/
14:14 logmsgbot: demon synchronized php-1.21wmf5/extensions/Wikibase/client/includes/store/sql/WikiPageEntityLookup.php 'More debugging, Iaf3cfff9'
14:11 logmsgbot: demon synchronized php-1.21wmf5/extensions/Wikibase/client/includes/store/sql/WikiPageEntityLookup.php 'More debugging'
14:04 logmsgbot: demon synchronized php-1.21wmf5/extensions/Wikibase/client/includes/store/sql/WikiPageEntityLookup.php
13:28 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Deploying wikibase debug log groups'
13:14 logmsgbot: demon synchronized php-1.21wmf5/extensions/Wikibase 'Syncing wikibase to 24d8471'
09:54 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c/37399/'
09:48 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/37398'
05:25 logmsgbot: aaron synchronized php-1.21wmf5/includes/job/JobQueueDB.php 'deployed 78c63d4cafb4937e289856f66ac0e524fda79acb'
04:59 logmsgbot: kaldari synchronized php-1.21wmf5/extensions/UploadWizard/resources/mw.UploadWizardDetails.js 'fixing live geo bug in UploadWizard'
02:24 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Fri Dec 7 02:24:07 UTC 2012
01:03 logmsgbot: reedy synchronized php-1.21wmf5/extensions/ParserFunctions/Expr.php
00:56 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'turning on experimental Flickr uploading for admins on Commons'
00:54 binasher: reprepro changes: add precise-wikimedia deb universe amd64 mariadb-server-5.5 5.5.28-mariadb-wmf201212041~precise <woot>
00:32 mutante: DNS update - adding wikivoyage.net (link to .org)
00:27 logmsgbot: dzahn gracefulled all apaches

December 6

23:53 logmsgbot: reedy synchronized php-1.21wmf5/extensions/ParserFunctions/Expr.php
23:41 logmsgbot: awjrichards synchronized php-1.21wmf5/extensions/MobileFrontend/javascripts/common/mf-application.js 'Touch file'
23:21 logmsgbot: awjrichards synchronized php-1.21wmf5/extensions/MobileFrontend/ 'fix for MobileFrontend bug 42749'
22:53 logmsgbot: aaron synchronized php-1.21wmf5/includes/job/JobQueueDB.php 'deployed 471f6840613ae5137464b6449db88bc30608e85f'
22:41 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Deploying Ia1208778, Ib80a42fa'
22:34 logmsgbot: olivneh synchronized php-1.21wmf5/extensions/EventLogging
22:25 logmsgbot: catrope synchronized php-1.21wmf5/resources/mediawiki/mediawiki.Uri.js '67de9f2cd4e37b7b7081d47a2652aa0158680c5c'
22:25 logmsgbot: catrope synchronized php-1.21wmf5/includes/EditPage.php '67de9f2cd4e37b7b7081d47a2652aa0158680c5c'
22:24 logmsgbot: catrope synchronized php-1.21wmf5/includes/Title.php '67de9f2cd4e37b7b7081d47a2652aa0158680c5c'
21:57 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Updating Extension:EventLogging config'
21:50 logmsgbot: reedy synchronized php-1.21wmf5/includes/media/DjVuImage.php
20:42 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'Turning off wgResourceLoaderExperimentalAsyncLoading on test2'
20:31 logmsgbot: kaldari Finished syncing Wikimedia installation... :
20:24 Reedy: Running sync-common on hume
19:42 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Deploying Idf3b7f0c'
19:41 logmsgbot: aaron synchronized wmf-config 'deployed 7c21802fc48ef4c5ee0edab3118d03cbbdc1866d'
19:12 logmsgbot: kaldari Started syncing Wikimedia installation... :
19:06 RobH: authdns-update for celsus and constable servers
18:34 ^demon: running `mwscript extensions/Wikibase/repo/maintenance/rebuildEntityPerPage.php --wiki=wikidatawiki` in screen on hume. If it causes problems, kill it and ping me
18:25 RoanKattouw: Reweighted pmtpa Parsoid pool according to # of cores
10:59 Nemo_bis: job queue snapshot for bug 42614: en.wiki 380k still raising steadily, Commons 85k down from 120+, fr.wiki 4M+, pl.wiki 18k, nl.wiki 8k, other tops 2k at most
10:08 hashar: applying new CI workflow on mediawiki/core.git . Changes now needs CR+2 to trigger unit tests.
03:27 logmsgbot: asher synchronized wmf-config/db.php 'setting db59 to full weight'
03:26 logmsgbot: asher synchronized wmf-config/db.php 'lowering db59 weight for warmup'
03:25 logmsgbot: asher synchronized wmf-config/db.php 'returning db59 to s1'
02:24 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Thu Dec 6 02:24:10 UTC 2012
01:19 binasher: rebooting db59
00:56 binasher: shutting down mysql on db59, upgrading to precise
00:55 logmsgbot: asher synchronized wmf-config/db.php 'pulling db59 from s1 for system upgrade'
00:43 RoanKattouw: Added mexia, tola, kuo, lardner to /h/w/common/docroot/noc/pybal/pmtpa/parsoid , they appear to be pooled just fine now
00:38 binasher: testing mariadb-server-5.5 5.5.28-mariadb-wmf201212041~precise on db1043 (slaving enwiki)
00:06 logmsgbot: aaron synchronized php-1.21wmf5/extensions/SwiftCloudFiles/php-cloudfiles-wmf/cloudfiles_http.php
00:05 binasher: rebooting db1043
00:00 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/37157/'
00:00 RoanKattouw: *build-essential

December 5

23:59 RoanKattouw: and build-essentials too
23:59 RoanKattouw: Temporarily installing git-core and npm on bast1001 by hand with Peter's blessing
23:59 Ryan_Lane: make that upgrading labsconsole to wmf/1.21wmf5
23:56 Ryan_Lane: upgrading labsconsole to wmf/1.21wmf6
23:45 MaxSem: created GeoData tables in all wikivoyages
23:42 MaxSem: creating GeoData tables in all wikipedias
23:37 binasher: shutting down mysql on db1043 for upgrade and testing
23:33 logmsgbot: maxsem synchronized wmf-config/CommonSettings.php 'GeoData tweaks'
23:32 logmsgbot: maxsem synchronized php-1.21wmf5/extensions/GeoData/
22:57 Ryan_Lane: upgraded labsconsole successfully
22:47 MaxSem: upgraded GeoData tables on testwiki by recreation
22:38 Ryan_Lane: running update.php on labsconsole
22:37 Ryan_Lane: attempting upgrade of mediawiki on labsconsole to wmf/1.21wmf4
22:28 logmsgbot: awjrichards synchronized php-1.21wmf5/extensions/MobileFrontend/ 'Bug fixes for MobileFrontend'
22:04 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Enable mobile beta photo uploads to commons from testwiki'
21:53 RobH: authdns-update for new misc servers
21:51 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable wikibaseclient on test2wiki'
20:47 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'wikidatawiki not readonly'
20:37 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'wikidatawiki not readonly'
20:30 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild localisationcache for wikidatawiki
20:12 notpeter: temp stopping puppet on brewster
19:50 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild localisationcache for wikidatawiki
19:40 logmsgbot: reedy synchronized wmf-config/
19:18 logmsgbot: reedy synchronized wmf-config/ 'Wikidata config updates'
19:18 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: git pull is helpful. Actually put everything else to 1.21wmf5
19:16 logmsgbot: reedy synchronized php-1.21wmf5/extensions/Wikibase
19:15 logmsgbot: reedy synchronized php-1.21wmf5/extensions/Diff
19:12 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'wikidatawiki readonly for maintenance'
19:10 logmsgbot: reedy synchronized php-1.21wmf5/extensions/DataValues
19:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.21wmf5
16:05 Reedy: THIS DOESN'T WORK IN HERE
15:34 apergos: powercycled sq48, hmm it was up for 214 days... fishy squid!
15:17 apergos: pulled ms-be5:sde1 and ms-be12:sdi from rings, they were borking obj replication across the cluster
14:53 hashar: Phasing out the Jenkins job linting operations/puppet.git , replacing it with https://integration.mediawiki.org/ci/job/operations-puppet-validate/
14:03 logmsgbot: reedy synchronized php-1.21wmf5/includes/AutoLoader.php
14:03 logmsgbot: reedy synchronized php-1.21wmf5/includes/db
14:02 logmsgbot: reedy synchronized php-1.21wmf5/includes/dao
08:40 apergos: powercycling snapshot1, was in swapdeath
08:04 apergos: cleaning up stuff on ms-be5 /srv/swift-storage/sde1/objects/ again, data somehow wound up on the underlying root filesystem, root partition was full
07:22 Nemo_bis: job queue snapshot for bug 42614: en.wiki 285k still raising steadily, Commons 111k, fr.wiki 4M, pt.wiki 40k, other tops 5k at most
02:44 logmsgbot: LocalisationUpdate completed (1.21wmf4) at Wed Dec 5 02:44:51 UTC 2012
02:24 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Wed Dec 5 02:24:05 UTC 2012
01:45 mutante: gallium - puppet runs fine again, has our librsvg 2.36-1wm1 packages and Android SDK stuff should also work (@hashar)
00:48 logmsgbot: mlitn synchronized php-1.21wmf4/extensions/ArticleFeedbackv5/modules 're-sync aftv5 js&css files with newer modification date'
00:45 logmsgbot: awjrichards synchronized wmf-config/mobile.php 'Fix api endpoint definition for mobile beta'
00:42 K4-713: synchronized payments cluster to f2c33676270c
00:37 notpeter: rebooting es1004 for kernel upgrade before repooling
00:13 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Disable use of commons api for mobile beta photo uploads on testwiki and test2wiki'
00:06 logmsgbot: awjrichards synchronized wmf-config/mobile.php 'Enabling use of commons API for mobile beta photo upload'
00:05 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Preparing to use commons API for mobile beta photo upload'

December 4

23:50 logmsgbot: awjrichards synchronized php-1.21wmf5/extensions/MobileFrontend/ 'touch files'
23:46 logmsgbot: anomie synchronized php-1.21wmf4/extensions/Vector/modules/ext.vector.collapsibleNav.css 'CSS fixes backported from 1.21wmf5'
23:45 logmsgbot: anomie synchronized php-1.21wmf4/skins/vector/screen.css 'CSS fixes backported from 1.21wmf5'
23:44 logmsgbot: anomie synchronized php-1.21wmf4/skins/modern/main.css 'CSS fixes backported from 1.21wmf5'
23:41 logmsgbot: awjrichards Finished syncing Wikimedia installation... : Updating MobileFrontend per regular weekly deployment
23:32 mutante: re-siging neon on sockpuppet, remove from ssh_known_hosts, run puppet ...
23:27 logmsgbot: py synchronized wmf-config/db.php 's7: repooling db56'
22:58 logmsgbot: mlitn synchronized php-1.21wmf5/extensions/ArticleFeedbackv5 'apply some patches to aft for dewiki'
22:58 logmsgbot: mlitn synchronized php-1.21wmf4/extensions/ArticleFeedbackv5 'apply some patches to aft for dewiki'
22:43 mutante: reinstalling neon
22:39 logmsgbot: awjrichards Started syncing Wikimedia installation... : Updating MobileFrontend per regular weekly deployment
22:21 notpeter: taking down hume for upgrade to precise
22:03 logmsgbot: awjrichards synchronized php-1.21wmf5/extensions/MobileFrontend/stylesheets/specials/watchlist.css 'touch file'
21:51 notpeter: starting innoback from db1041 to db56
21:34 notpeter: bast1001 back up with clean puppet runs
21:15 LeslieCarr: rebooting sockpuppet for kernel upgrades
20:51 logmsgbot: py synchronized wmf-config/db.php 's7: depooling db56 for reimage'
20:48 LeslieCarr: upgrading sockpuppet's 57 pending security upgrades
20:31 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php 're-enable aftv5 on dewiki after having created missing tables'
20:07 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Fix dewiki breakage until tables are created and everyone's bikeshedding over the proper way to deploy'
19:52 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php
19:51 logmsgbot: mlitn synchronized wmf-config/CommonSettings.php
19:51 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php
19:50 logmsgbot: mlitn synchronized php-1.21wmf5/extensions/ArticleFeedbackv5 'desc'
19:49 logmsgbot: mlitn synchronized php-1.21wmf4/extensions/ArticleFeedbackv5 'desc'
19:36 LeslieCarr: committing to pfw1-eqiad - has occasionally lost a few packets in the past
19:24 mutante: upgraded Bugzilla to 4.2.4
18:47 notpeter: taking down bast1001 for reimaging
18:20 logmsgbot: reedy synchronized php-1.21wmf5/extensions/GoogleNewsSitemap
17:43 logmsgbot: reedy synchronized php-1.21wmf5/extensions/GoogleNewsSitemap/
17:40 cmjohnson1: cp1031 powering down for fan assembly/main board swap rt3614
16:48 cmjohnson1: es1004 replacing bad hdd slot 1 (rt3954)
15:58 cmjohnson1: authdns update...adding pc1001-1003
14:45 mark: Installed ms-fe3001
14:08 Nemo_bis: job queue snapshot for bug 42614: Commons 112k, fr.wiki 4M, pl.wiki 21k, pt.wiki 65k, other tops 5k at most
09:28 logmsgbot: nikerabbit synchronized php-1.21wmf5/extensions/Narayam/ 'i18n deploy'
05:02 K4-713: revision-locked payments1004 to 4c15a56c46bf5, all other payments hosts updated to ec2a05d768355
02:44 logmsgbot: LocalisationUpdate completed (1.21wmf4) at Tue Dec 4 02:44:31 UTC 2012
02:23 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Tue Dec 4 02:23:29 UTC 2012
01:31 logmsgbot: spage synchronized php-1.21wmf5/extensions/E3Experiments 'E3 backports to wmf5'
01:00 mutante: gallium - removing multiarch support (32bit on 64bit systems) which causes issues with librsvg package
00:47 logmsgbot: pgehres synchronized php-1.21wmf4/extensions/CentralNotice/ 'Updating CentralNotice to master. Adding API for CN logs'
00:43 logmsgbot: pgehres synchronized php-1.21wmf5/extensions/CentralNotice/ 'Updating CentralNotice to master. Adding API for CN logs'
00:36 Tim: killing long-running runJobs.php processes affected by LST bug
00:15 logmsgbot: asher synchronized wmf-config/db.php 'adding db68 to s7'
00:00 mutante: gallium - restarting Apache, enabling new ServerAlias

December 3

23:57 logmsgbot: aaron synchronized php-1.21wmf5/extensions/CentralAuth/CentralAuthUser.php 'revert live hacks, exception already dumped in bugzilla'
23:34 Tim: installed php-luasandbox on fenari
22:48 LeslieCarr: restarting pybal on lvs1003 for parsoid vip
22:40 mutante: DNS update - removing broken dvd.wikimeedia.org
22:40 LeslieCarr: restarted pybal on lvs1006 and lvs1005
22:30 LeslieCarr: restarting pybal on lvs1006 for parsoid vip
22:26 LeslieCarr: modify above - restarting PYBAL on lvs for parsoid vip
22:26 LeslieCarr: restarting lvs4 for parsoid vip
22:22 LeslieCarr: restarting pybal on lvs3
22:11 K4-713: updated payments cluster to d7946a804d763
22:07 mutante: gallium - removed librsvg2-2:i386, installed librsvg2-2, librsvg2-bin
22:07 logmsgbot: aaron synchronized wmf-config/CommonSettings.php
22:03 K4-713: updated payments to 8eb328e96a1ef6ac
21:53 K4-713: updated payments to 54d3f8f0f9c7bdc
21:36 logmsgbot: aaron synchronized php-1.21wmf5/extensions/CentralAuth/CentralAuthUser.php 'added exception'
21:28 mutante: DNS update - adding doc and integration CNAMEs
21:04 Ryan_Lane: adding mwang to the ops ldap group
20:48 paravoid: deploying squid config for the updated mobile ACL
20:48 hashar: jenkins / gallium : updating Zuul repository 2747501..f73a04d , should be updated by puppet automatically
20:42 logmsgbot: kaldari synchronized php-1.21wmf5/extensions/Vector/modules/ext.vector.collapsibleNav.css 'syncing vector css for menus'
20:33 binasher: added max_user_connections to wikiadmin grants on s1
20:32 logmsgbot: kaldari synchronized php-1.21wmf5/resources/startup.js 'syncing startup.js per Roan'
20:23 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'wikidatawiki back to writeable'
19:29 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'wikidatawiki marked readonly'
19:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf5
18:40 mutante: deleting broken link to nagios3.conf on spence and restarting apache
18:25 mutante: installing package upgrades on spence
18:16 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'deployed b999dbca1f8ca0a76fe327dd1ae30beb6eed462f'
17:30 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
16:57 mark: Installed ms-be3003
16:49 mark: Installed ms-be3004
15:35 cmjohnson1: mw16 shutting down to troubleshoot DIMM error rt3951
15:14 pp-pdf1: upgrade mwlib to 0.14.2
13:12 logmsgbot: demon synchronized wmf-config/throttle.php 'Deploy I826b9114'
12:58 mark: Shutdown slauerhoff's switchports
02:46 logmsgbot: LocalisationUpdate completed (1.21wmf4) at Mon Dec 3 02:46:35 UTC 2012
02:24 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Mon Dec 3 02:24:47 UTC 2012

December 2

02:43 logmsgbot: LocalisationUpdate completed (1.21wmf4) at Sun Dec 2 02:43:34 UTC 2012
02:22 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Sun Dec 2 02:21:55 UTC 2012

December 1

02:45 logmsgbot: LocalisationUpdate completed (1.21wmf4) at Sat Dec 1 02:45:47 UTC 2012
02:24 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Sat Dec 1 02:24:14 UTC 2012
01:17 logmsgbot: catrope synchronized php-1.21wmf5/cache/l10n/ 'Rebuild l10n cache'
01:04 logmsgbot: catrope synchronized php-1.21wmf4/cache/l10n/ 'Rebuild l10n cache'
00:52 logmsgbot: aaron synchronized php-1.21wmf5/includes/filebackend/SwiftFileBackend.php 'deployed db4574540c993aeab8f0840de916e47fdac6199f'
00:41 logmsgbot: olivneh synchronized php-1.21wmf5/extensions/EventLogging/content/JsonSchemaHooks.php 'Assign NS_SCHEMA rights to editinterface'
00:21 logmsgbot: catrope synchronized php-1.21wmf5/extensions/VisualEditor 'Updating VisualEditor to master'
00:20 logmsgbot: catrope synchronized php-1.21wmf4/extensions/VisualEditor 'Updating VisualEditor to master'
00:10 logmsgbot: olivneh synchronized php-1.21wmf5/extensions/EventLogging/content/JsonSchemaHooks.php 'Update SCHEMA_NS perms on wmf5'
00:03 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Returned itwikisource to 1.21wmf5

November 30

23:57 logmsgbot: aaron synchronized php-1.21wmf5/extensions/LabeledSectionTransclusion 'deployed 23e25751e59fa5ce7b4e337780d13818c2e36260'
22:20 binasher: running patch-uploadstash-us_props.sql migration on all wikis
20:55 mutante: for log completeness - wikivoyage.com usable, redirects to .org
19:30 notpeter: temp stopping puppet on brewster
19:24 logmsgbot: pgehres synchronized php-1.21wmf4/resources/startup.js 'Touching to update RL'
19:23 logmsgbot: pgehres synchronized php-1.21wmf4/extensions/CentralNotice/ 'Updating CN to master. More banner buckets for A/B testing'
19:21 logmsgbot: pgehres synchronized php-1.21wmf5/resources/startup.js 'Touching to update RL'
19:18 logmsgbot: pgehres synchronized php-1.21wmf5/extensions/CentralNotice/ 'Updating CN to master. More banner buckets for A/B testing'
19:11 RobH: authdns-update for server antimony
18:50 cmjohnson1: analytics1007 troubleshooting hdd/pxe boot issue (going to power cycle several times)
18:21 RobH: authdns-update to add wtp1001 info to dns
03:18 K4-713: updated payments cluster to 4c15a56c46bf52a
02:49 logmsgbot: LocalisationUpdate completed (1.21wmf4) at Fri Nov 30 02:49:43 UTC 2012
02:26 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Fri Nov 30 02:26:44 UTC 2012
01:40 logmsgbot: olivneh synchronized php-1.21wmf5/extensions/EventLogging/content/JsonSchemaHooks.php
01:14 logmsgbot: py synchronized wmf-config/db.php 's7: db26 out, db68 in, db56 lower weight'
00:27 logmsgbot: olivneh Finished syncing Wikimedia installation... :
00:16 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'LST for test2wiki'

November 29

23:51 logmsgbot: dzahn gracefulled all apaches
23:35 logmsgbot: olivneh Started syncing Wikimedia installation... :
23:32 logmsgbot: dzahn gracefulled all apaches
23:20 logmsgbot: olivneh synchronized php-1.21wmf5/extensions/EventLogging
23:16 logmsgbot: spage synchronized php-1.21wmf4/extensions/E3Experiments 'E3 deploy ACUX at 100%'
23:14 mutante: DNS update - add wikivoyage.com
22:58 notpeter: restarting incremental indexer on searchidx2
22:56 logmsgbot: kaldari synchronized php-1.21wmf5/extensions/UploadWizard/resources/mw.UploadWizardDetails.js 'resyncing UploadWizard patch from yesterday'
22:45 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Enabling EventLogging on metawiki'
22:44 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Updating configuration of EventLogging'
22:38 logmsgbot: olivneh synchronized php-1.21wmf4/extensions/EventLogging/modules/ext.EventLogging.js
22:14 notpeter: starting innobackup from db1041 to db68 (s7)
21:48 logmsgbot: asher synchronized wmf-config/db.php 'pooling db66 as a new s3 slave'
21:27 mutante: old swift logs on iron, some rm'ed, some gzipped, archiving ben's home contents on tridge, clean up iron
19:49 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: itwikisource back to 1.21wmf4
19:48 mutante: installing package upgrades on iron
19:41 mutante: removing mysql server from iron
19:29 RobH: authdns-update to add dns for tin
19:25 logmsgbot: demon synchronized php-1.21wmf5/extensions/LabeledSectionTransclusion/ 'Updating LST to master'
19:22 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'TemplateSandbox on all wikis'
19:21 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'TemplateSandbox on all wikis'
18:13 logmsgbot: aaron synchronized wmf-config/PrivateSettings.php 'Set Swift auth URL to the IP address host.'
17:30 cmjohnson1: es1004 has raid error...power cycling to enter raid bios
15:37 cmjohnson1: provisioning labsdb1002
14:37 cmjohnson1: provisioning labsdb1001
06:09 Tim: fixing searchidx2 by increasing RMI timeout
02:45 logmsgbot: LocalisationUpdate completed (1.21wmf4) at Thu Nov 29 02:45:27 UTC 2012
02:24 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Thu Nov 29 02:24:32 UTC 2012
00:48 logmsgbot: pgehres synchronized php-1.21wmf4/extensions/CentralNotice/ 'Updating CentralNotice to master to hide banners on mobile devices'
00:45 logmsgbot: pgehres synchronized php-1.21wmf5/extensions/CentralNotice/ 'Updating CentralNotice to master to hide banners on mobile devices'
00:12 Tim: uploaded new lucene-search-2 with fixed init script, will upgrade it on the search servers

November 28

23:30 logmsgbot: kaldari synchronized php-1.21wmf5/skins/vector/screen.css 'updating vector skin'
23:30 logmsgbot: kaldari synchronized php-1.21wmf5/skins/modern/main.css 'updating modern skin'
23:29 logmsgbot: kaldari synchronized php-1.21wmf5/extensions/Vector/modules/ext.vector.collapsibleNav.css 'adding forward-compat to Vector for bug 42452'
22:59 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: Moving all non-pedias back to wmf5
22:46 cmjohnson1: running authdns update
22:45 cmjohnson1: adding DNS entry labsdb1001/1002 .eqiad.wmnet
22:41 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: metawiki to wmf5
22:33 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: Rolling wikiversions.dat back to de90c27d, undoing todays deploy
22:18 logmsgbot: aaron synchronized php-1.21wmf4/extensions/SwiftCloudFiles
22:01 Ryan_Lane: disabled page counters on labsconsole
21:27 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: Commonswiki back to 1.21wmf5
21:13 logmsgbot: kaldari synchronized php-1.21wmf5/extensions/UploadWizard/resources/mw.UploadWizardDetails.js 'fixing uplaodwizrd for ie8'
20:41 logmsgbot: aaron synchronized php-1.21wmf5/extensions/SwiftCloudFiles 'deployed d50d754971991bb2362455fe47eaadb20aff3a80'
20:41 RoanKattouw: Restarted logmsgbot
18:48 notpeter: temp stopping puppet on brewster
14:45 ^demon: fenari:/home/wikipedia/common now reflects Ia319794c - rm'd all traces of submodule
05:47 Tim: on searchidx2: killed extra IncrementalUpdater daemon
04:12 Tim: on searchidx2: restarted incremental updater
00:16 MaxSem: Scap complete

November 27

23:14 Ryan_Lane: restarted gerrit; was getting WARN org.eclipse.jetty.util.log : Dispatched Failed! errors
22:27 MaxSem: scapping as a part of usual mobile deployment
22:23 mutante: starting squid backend on knsq18 (cron spam/nagios)
21:52 Reedy: reedy synchronized php-1.21wmf5/extensions/LabeledSectionTransclusion
21:50 robla: logmsgbot doesn't seem to be working for sync-dir, scap, etc
20:46 Ryan_Lane: upgrading salt in production, in batches of 10
20:18 Ryan_Lane: upgrading salt on all labs minions
20:07 notpeter: temp stopping puppet on brewster
19:31 mutante: upgrade libc* packages on sodium and singer
19:29 mutante: installing a bunch of pkg upgrades on zirconium
16:07 paravoid: ignore previous labs-related entry
16:06 paravoid: bastion disabled hazard-sj's cron, as it was accidentally DoSing bastion
14:15 apergos: snapshot1002 down for reinstall and testing
12:32 mark: Removed ms6 out of the loop as backend server for esams upload squids
04:11 pgehres: TimStarling: restarting lucene-search-2 on search1016 due to init script bug
04:11 pgehres: synchronized wmf-config/reporting-setup.php "Setting S:FundraiserStatistics to use db1013 as the slave"

November 26

19:29 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.21wmf5
19:27 logmsgbot_: reedy synchronized php-1.21wmf5/extensions/UploadWizard
19:24 logmsgbot_: reedy synchronized php-1.21wmf5/cache/l10n/
19:10 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuild localisation cache for 1.21wmf5
18:58 logmsgbot_: reedy synchronized php-1.21wmf5/cache/l10n/
18:47 logmsgbot_: reedy synchronized php-1.21wmf5/extensions/Wikibase
18:45 logmsgbot_: reedy synchronized php-1.21wmf5/extensions/Diff
18:37 logmsgbot_: aaron synchronized php-1.21wmf5/includes/filerepo/file/LocalFile.php 'deployed 1fa41352d4e0b3cd2e53e7d8d485ac9098b8bfcc'
18:31 logmsgbot_: reedy Started syncing Wikimedia installation... : Build localisation cache for 1.21wmf5
18:16 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.21wmf5 for l10n stuffs
18:15 logmsgbot_: aaron synchronized php-1.21wmf4/thumb.php 'deployed 81c1d0133f74eec600277ab744a9730c4e974e47'
18:14 logmsgbot_: reedy synchronized php-1.21wmf5 'Initial sync of php-1.21wmf5'
18:03 logmsgbot_: aaron synchronized php-1.21wmf5/extensions/FlaggedRevs
18:02 logmsgbot_: reedy synchronized wmf-config/ExtensionMessages-1.21wmf5.php
18:01 logmsgbot_: reedy synchronized live-1.5/static-1.21wmf5/
17:58 logmsgbot_: aaron synchronized php-1.21wmf4/extensions/FlaggedRevs
17:55 mutante: authdns-update - adding wikipedia.bg
16:29 mark: Dist-upgraded & rebooted eqiad upload Varnish servers
15:26 Jeff_Green: dist-upgrades & reboots for payments100[1-4], pay-lvs100[12]
02:24 logmsgbot_: LocalisationUpdate completed (1.21wmf4) at Mon Nov 26 02:23:58 UTC 2012

November 25

22:07 logmsgbot_: reedy synchronized wmf-config/
19:23 logmsgbot_: reedy synchronized wmf-config/flaggedrevs.php
19:01 Reedy: Pointed php symlink to php-1.21wmf4
08:11 apergos: from about half an hour ago, restarted lucene search on search13 and forgot to log it
02:23 logmsgbot_: LocalisationUpdate completed (1.21wmf4) at Sun Nov 25 02:23:41 UTC 2012

November 24

02:24 logmsgbot_: LocalisationUpdate completed (1.21wmf4) at Sat Nov 24 02:24:00 UTC 2012

November 23

23:15 logmsgbot_: olivneh synchronized wmf-config/InitialiseSettings.php 'Enabling PostEdit on ptwiki & svwiki'
18:28 logmsgbot_: maxsem synchronized php-1.21wmf4/includes/Message.php 'Debugging'
16:20 logmsgbot_: maxsem synchronized php-1.21wmf4/includes/Message.php 'Debugging'
15:09 mark: Dist-upgraded and rebooted all eqiad bits servers (new Varnish package)
15:04 logmsgbot_: maxsem synchronized php-1.21wmf4/extensions/CategoryTree/ 'https://gerrit.wikimedia.org/r/#/c/34668/'
14:57 mark: Dist-upgraded and rebooted all pmtpa bits servers (new Varnish package)
14:33 mark: Dist-upgraded and rebooted all esams bits servers (new Varnish package)
14:05 mark: Built new varnish 3.0.3plus~rc1-wm6 packages with fixed epoll deadlock, and inserted it into the precise-wikimedia APT repository
04:20 Tim: on fenari: updated the "search" dsh node group based on nmap -sP and fixed the remaining search servers
04:04 Tim: oh yeah, and I upgraded lucene to my version with the timeouts, deployed to pmtpa only via puppet
04:02 Tim: many lucene search servers failed to bind to port 1099 when they were restarted by the upgrade, restarting manually
02:24 logmsgbot_: LocalisationUpdate completed (1.21wmf4) at Fri Nov 23 02:24:34 UTC 2012

November 22

14:52 apergos: msbe8 and msbe10 installed, not yet deployed
14:12 hashar: jenkins: configure jobbuilder-bot user permissions
13:39 hashar: Jenkins: updated zuul-bot user permission so it can trigger jobs.
12:16 hashar: Installed plugins on Jenkins for Zuul deployment: notification, build-timeout
12:06 hashar: restarted Jenkins to load in the new zuul jobs.
09:23 apergos: fixed perms on srv284 php-1.*wmf* and wmf-config, test (were 700 instead of 777)
02:24 logmsgbot_: LocalisationUpdate completed (1.21wmf4) at Thu Nov 22 02:24:22 UTC 2012
01:28 mutante: sync-apache srv284-only, start apache (was missing all.conf), repool
01:15 mutante: rebooting srv193 (test.wp) for upgrade
00:56 mutante: repooled mw58,mw59 (upgrades) srv284 (hw ticket was resolved, reinstalled)
00:07 mutante: powercycling kaulen, this time no console output at all

November 21

23:47 mutante: tmp. depooling and rebooting a few mw5x servers for kernel upgrades, one by one
23:37 logmsgbot_: preilly synchronized php-1.21wmf4/extensions/ZeroRatedMobileAccess 'update post deploy'
23:34 logmsgbot_: preilly synchronized php-1.21wmf3/extensions/ZeroRatedMobileAccess 'update post deploy'
22:58 mutante: disabled swap on kaulen per Tim's advice
22:55 mutante: bugzilla back up
22:50 mutante: powercycling kaulen
22:25 logmsgbot_: asher synchronized wmf-config/throttle.php 'increased api limit for bug 42319'
21:51 mutante: installing package upgrades in the mw55-mw59 range
21:17 logmsgbot_: preilly Finished syncing Wikimedia installation... : update zero rated mobile access
20:56 Ryan_Lane: deploying Change If5f6bc33: (bug 42334) on labsconsole
20:47 Ryan_Lane: applying changes to fix no creds issue when setting preferences on labsconsole
20:47 logmsgbot_: preilly Started syncing Wikimedia installation... : update zero rated mobile access
20:34 logmsgbot_: preilly synchronized php-1.21wmf4/extensions/ZeroRatedMobileAccess 'update post deploy'
20:34 logmsgbot_: preilly synchronized php-1.21wmf3/extensions/ZeroRatedMobileAccess 'update post deploy'
20:00 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: rest of wikipedias to 1.21wmf4
19:24 mutante: installing package upgrades in the srv193-srv199 range
18:55 mutante: re-signing srv284 on puppetmaster, package installs...
18:45 AaronSchulz: Copying over all math/timeline files to nas1 that are not already there for DR
18:34 AaronSchulz: prepared all wikivoyage file directories and containers
17:41 Reedy: Running scap-recompile on all hosts in mediawiki-installation group
16:38 mutante: powering on analytics1007
16:29 mutante: reinstalling srv284 "last lucid standing"
16:24 mark: Takeback by nas1001-a completed, snapmirror relationships back in sync
16:15 mark: Aborted synchronous snapmirror relationships between nas1-a and nas1001-a to initiate cf giveback
16:08 mutante: powercycling srv284
16:05 mutante: remove known troublemakers srv266 and srv284 from dsh group "apaches"
15:50 mark: Initiated takeover of nas1001-a to nas1001-b
15:49 mark: nas1001-b back up and running
14:46 mark: Halted nas1001-b for NIC upgrade
09:18 notpeter: repooling mw20-mw34
07:50 notpeter: repooling mw70-mw74
07:21 apergos: restarted lucene search on search14
03:01 notpeter: depooling mw70-mw74 for upgrade to precise
03:01 notpeter: depooling mw20-mw34 for upgrade to precise
02:45 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Wed Nov 21 02:45:42 UTC 2012
02:24 logmsgbot_: LocalisationUpdate completed (1.21wmf4) at Wed Nov 21 02:24:47 UTC 2012
02:17 K4-713: updated payments cluster to 6ea97a5cbc06
00:39 logmsgbot_: preilly synchronized php-1.21wmf4/extensions/ZeroRatedMobileAccess 'update post deploy'
00:37 logmsgbot_: preilly synchronized php-1.21wmf3/extensions/ZeroRatedMobileAccess 'update post deploy'
00:06 logmsgbot_: maxsem synchronized php-1.21wmf4/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/34440/'
00:03 logmsgbot_: maxsem synchronized php-1.21wmf3/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/34440/'
00:03 notpeter: depooling mw66-mw69 for upgrade to apache
00:03 notpeter: depooling mw35-mw49 for upgrade to apache

November 20

23:52 logmsgbot_: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
23:35 K4-713: updated payments cluster to b5a80e04086c3a7
23:32 binasher: depooled srv203
23:00 logmsgbot_: reedy synchronized php-1.21wmf4/
22:56 logmsgbot_: reedy synchronized php-1.21wmf3/
22:45 logmsgbot_: kaldari synchronized wmf-config/InitialiseSettings.php 'disabling experimental flickr uploading on commons until some bugs are fixed'
22:44 logmsgbot_: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
22:35 logmsgbot_: asher synchronized wmf-config/InitialiseSettings.php 'now only 30 wikis that need wgOldChangeTagsIndex = true, changing default to false'
21:02 notpeter: depooling mw17-mw19 and mw50-mw54 for upgrade to precise
20:59 notpeter: depooling mw62-mw65 for upgrade to precise
20:37 Jeff_Green: switching payments back to eqiad
20:23 binasher: running change_tag index migrations on the 757 wikis with <7k ct rows
20:09 logmsgbot_: kaldari synchronized wmf-config/InitialiseSettings.php 'turning on flickr uploading for sysops on Commons'
19:42 apergos: scribbled over /dev/sda partition table on ms-be7 so I could watch the install break, left in shell form installer
16:14 notpeter: depooling srv281-srv289 for upgrades to precise
09:57 hashar: Upgrading Jenkins plugin xUnit which introduce a potential back compatibility issue <failedIfNotNew> renamed to <failIfNotNew>. Will update configs files.
09:37 hashar: restarting Jenkins
09:22 hashar: upgrading Jenkins plugins
09:18 logmsgbot_: nikerabbit synchronized wmf-config/InitialiseSettings.php 'i18n deploy'
08:15 logmsgbot_: tstarling Finished syncing Wikimedia installation... :
07:32 paravoid: rebooting ms-be5, strange df output for /
07:01 logmsgbot_: tstarling Started syncing Wikimedia installation... :
06:37 logmsgbot_: tstarling synchronized wmf-config/CommonSettings.php
06:36 logmsgbot_: tstarling synchronized wmf-config/InitialiseSettings.php
06:36 logmsgbot_: tstarling synchronized wmf-config/filebackend.php
06:34 logmsgbot_: tstarling synchronized php-1.21wmf4/extensions/Score
06:33 logmsgbot_: tstarling synchronized php-1.21wmf3/extensions/Score
05:54 Ryan_Lane: rebooting ssl3001
05:19 Ryan_Lane: depooling ssl3001
05:19 Ryan_Lane: repooling ssl3002 (upgrade complete)
05:14 paravoid: deploying squid config for score
05:11 Ryan_Lane: rebooting ssl3002
04:48 Ryan_Lane: depooling ssl3002 for upgrade to precise
04:48 Ryan_Lane: repooling ssl3003 (upgrade finished)
04:42 Ryan_Lane: rebooting ssl3003 (upgraded via ssh, not console)
04:25 notpeter: repooling srv258-srv280
04:04 Ryan_Lane: repooling ssl1 (upgrade complete)
03:51 Tim: on brewster: reduced rotate count for squid logs to zero and ran logrotate -f
03:44 Tim: on brewster: root partition is full, removing some useless squid logs
02:52 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Tue Nov 20 02:52:09 UTC 2012
02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf4) at Tue Nov 20 02:28:20 UTC 2012
02:10 notpeter: depooling srv296-srv301 for upgrades to precise
02:08 notpeter: repooling srv290-srv295
01:21 Ryan_Lane: depooling ssl1
01:21 Ryan_Lane: repooling ssl2 (upgrade complete)
01:10 Ryan_Lane: depooling ssl3003 to upgrade to precise
01:02 MaxSem: Scap hung and had to be aborted. Since what was being deployed wasn't enabled, no clusters were harmed.
00:46 Ryan_Lane: all eqiad https hosts upgraded to precise
00:46 Ryan_Lane: repooling ssl1001 (upgrade complete)
00:23 Ryan_Lane: depooling ssl2 for upgrade to precise
00:23 Ryan_Lane: repooling ssl4 (upgrade complete)
00:21 logmsgbot_: maxsem Started syncing Wikimedia installation... : Black-deployment of ext:Solarium, not enabled anywhere yet
00:01 Ryan_Lane: depooling ssl1001
00:01 Ryan_Lane: repooling ssl1002 (upgrade complete)

November 19

23:59 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
23:37 Ryan_Lane: repooling ssl3 depooling ssl4
23:08 logmsgbot_: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/34220/ and https://gerrit.wikimedia.org/r/#/c/33786/'
23:06 Ryan_Lane: depooling ssl1002 for upgrade to precise
23:06 Ryan_Lane: repooling ssl1003 (upgrade to precise complete)
22:59 Ryan_Lane: make that rebooting ssl1003
22:59 Ryan_Lane: rebooting ssl1004
22:53 Ryan_Lane: rebooting ssl3
22:51 Ryan_Lane: re-pooling ssl4 and depooling ssl3
22:35 Ryan_Lane: depooling ssl4
22:35 Ryan_Lane: upgrading ssl1003 to precise
22:13 Ryan_Lane: depooling ssl1003
20:14 mark: Restoring bits.esams traffic
19:56 mark: Moving bits.esams traffic to pmtpa
19:20 notpeter: depooling srv290-srv301 for upgrade to precise
19:20 notpeter: depooling srv258-srv280 for upgrade to precise
19:11 logmsgbot_: py synchronized wmf-config/db.php 'repooling es1'
18:45 logmsgbot_: maxsem synchronized wmf-config/mobile.php 'https://gerrit.wikimedia.org/r/#/c/34105/'
18:14 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf4
17:10 logmsgbot_: reedy synchronized php-1.21wmf3/cache/interwiki.cdb
17:09 logmsgbot_: reedy synchronized php-1.21wmf4/cache/interwiki.cdb
15:58 paravoid: powercycling ms-be3, unresponsive
15:48 logmsgbot_: reedy synchronized php-1.21wmf4/extensions/OAI
14:48 logmsgbot_: reedy synchronized php-1.21wmf3/cache/interwiki.cdb 'Updating interwiki cache'
14:48 logmsgbot_: reedy synchronized php-1.21wmf4/cache/interwiki.cdb 'Updating interwiki cache'
14:23 paravoid: putting ms-fe1 back into the pool
08:36 hashar: running sync-common on mw46
06:56 apergos: restarted lucene search on search14
02:43 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Mon Nov 19 02:43:07 UTC 2012
02:23 logmsgbot_: LocalisationUpdate completed (1.21wmf4) at Mon Nov 19 02:23:19 UTC 2012
00:35 Tim: on hume: running fixBug41778.php on all wikis
00:31 logmsgbot_: tstarling synchronized php-1.21wmf3/extensions/WikimediaMaintenance/fixBug41778.php
00:30 logmsgbot_: tstarling synchronized php-1.21wmf4/extensions/WikimediaMaintenance/fixBug41778.php

November 18

22:53 logmsgbot_: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/34034/'
21:44 logmsgbot_: maxsem synchronized php-1.21wmf4/extensions/TocTree/ 'https://gerrit.wikimedia.org/r/#/c/33987/ to fix RL warnings'
21:33 logmsgbot_: maxsem synchronized php-1.21wmf4/includes/resourceloader/ResourceLoader.php 'Debugging over'
21:26 logmsgbot_: maxsem synchronized php-1.21wmf3/extensions/Narayam/ 'Deploying https://gerrit.wikimedia.org/r/#/c/34024/ to unbreak RL'
21:25 logmsgbot_: maxsem synchronized php-1.21wmf4/extensions/Narayam/ 'Deploying https://gerrit.wikimedia.org/r/#/c/34024/ to unbreak RL'
20:58 logmsgbot_: maxsem synchronized php-1.21wmf4/includes/resourceloader/ResourceLoader.php 'Debugging'
09:26 logmsgbot_: aaron synchronized php-1.21wmf4/includes/upload/UploadFromChunks.php
09:16 logmsgbot_: aaron synchronized php-1.21wmf4/includes/upload/UploadFromChunks.php 'debug logging'
09:09 logmsgbot_: aaron synchronized php-1.21wmf4/includes/upload/UploadFromChunks.php 'debug logging'
08:11 apergos: restarted lucene search on search13
07:22 logmsgbot_: aaron synchronized php-1.21wmf4/includes/upload/UploadFromChunks.php
02:53 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Sun Nov 18 02:52:59 UTC 2012
02:29 logmsgbot_: LocalisationUpdate completed (1.21wmf4) at Sun Nov 18 02:29:14 UTC 2012

November 17

16:46 binasher: restarting lsearchd on all eqiad hosts
16:21 logmsgbot_: asher synchronized wmf-config/lucene.php 'failing pool3+4 to pmtpa'
16:18 logmsgbot_: asher synchronized wmf-config/lucene.php 'failing pool1+2 to pmtpa'
02:52 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Sat Nov 17 02:52:30 UTC 2012
02:27 logmsgbot_: LocalisationUpdate completed (1.21wmf4) at Sat Nov 17 02:27:00 UTC 2012

November 16

23:00 logmsgbot_: maxsem synchronized php-1.21wmf3/extensions/TimedMediaHandler 'https://gerrit.wikimedia.org/r/#/c/33819/'
22:58 logmsgbot_: maxsem synchronized php-1.21wmf4/extensions/TimedMediaHandler 'https://gerrit.wikimedia.org/r/#/c/33819/'
22:44 notpeter: repooling srv231-srv247
22:21 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
22:18 logmsgbot_: catrope synchronized php-1.21wmf4/resources/jquery/jquery.tablesorter.js 'Deploying 9cb43905cd7189aa9a42f942f24a7aea50b4f0e9'
22:18 notpeter: repooling srv214-srv218
22:17 logmsgbot_: catrope synchronized php-1.21wmf3/resources/jquery/jquery.tablesorter.js 'Deploying 2405956a8d6f305f6d3f9b0a1bda4deb7e917ec8'
21:30 notpeter: starting innobackupex from es1004 to es1
21:09 logmsgbot_: py synchronized wmf-config/db.php 'pulling es1'
20:40 binasher: running patch-job_attempts.sql migration on all wikis
20:30 binasher: purging zhwiki refreshLinks2 jobs
20:03 notpeter: depooling srv231-srv247 from apaches pool for upgrade to precise
20:02 notpeter: depooling srv214-srv218 from api pool for upgrade to precise
19:48 logmsgbot_: py synchronized wmf-config/db.php 'es2 going live'
18:56 notpeter: temp stopping puppet on brewster
17:41 paravoid: depooling ms-fe1
17:15 logmsgbot_: hashar synchronized wmf-config/CommonSettings.php 'Redis-setup is unavailable for wmflabs, use memcached 33721'
16:49 Reedy: Running sync-common on mw52
15:34 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/EducationProgram/
15:33 logmsgbot_: reedy synchronized php-1.21wmf4/extensions/EducationProgram/
15:16 paravoid: pooling ms-fe1 back with the modified rewrite.py
12:37 Reedy: Running sync common on srv252, lots of Failed opening required fatals...
04:58 K4-713: updated payments cluster to c898ce38e162516e73
03:11 Tim: deleted corrupted memcached value enwiki:gadgets-definition:6
02:48 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Fri Nov 16 02:48:37 UTC 2012
02:37 RoanKattouw: Forcing puppet run on wtp1 to reinstall Parsoid
02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf4) at Fri Nov 16 02:28:52 UTC 2012
01:49 logmsgbot_: aaron synchronized php-1.21wmf4/thumb.php 'remove hacks'
01:34 logmsgbot_: aaron synchronized php-1.21wmf4/thumb.php 'more debugging'
01:33 logmsgbot_: aaron synchronized php-1.21wmf4/thumb.php
01:32 logmsgbot_: aaron synchronized php-1.21wmf4/thumb.php
01:17 logmsgbot_: aaron synchronized php-1.21wmf4/thumb.php
01:10 logmsgbot_: aaron synchronized php-1.21wmf4/thumb.php 'debug logging'
00:35 logmsgbot_: tstarling synchronized wmf-config/InitialiseSettings.php
00:35 logmsgbot_: tstarling synchronized wmf-config/CommonSettings.php
00:30 logmsgbot_: tstarling synchronized wmf-config/InitialiseSettings.php
00:29 logmsgbot_: tstarling synchronized wmf-config/InitialiseSettings.php
00:28 logmsgbot_: tstarling synchronized small.dblist
00:21 logmsgbot_: tstarling synchronized php-1.21wmf4/extensions/UniversalLanguageSelector

November 15

22:41 logmsgbot_: reedy synchronized php-1.21wmf4/extensions/ReaderFeedback/
22:36 logmsgbot_: reedy synchronized php-1.21wmf4/extensions/WikimediaIncubator/
22:29 Reedy: Ran sync-docroot
22:08 notpeter: repooling srv25-srv257
22:06 notpeter: repooling srv190-srv192, srv225-srv230
21:33 binasher: starting apache on srv249
21:32 binasher: stopping apache on srv249
21:30 binasher: ran chmod 755 /usr/local/apache/common-local/wmf-config on srv249
20:46 Jeff_Green: adjusted key_buffer_size for fundraisingdbs from 1M to 1G
20:42 notpeter: removing srv225-srv230 for upgrade to precise
20:34 notpeter: reimaging srv190-srv192 as regular apaches
20:09 logmsgbot_: reedy synchronized php-1.21wmf3/
20:05 logmsgbot_: reedy synchronized php-1.21wmf4/
19:37 notpeter: repooling srv200-srv213
19:23 notpeter: removing srv251-srv257 for upgrade to precise
19:19 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
18:59 paravoid: upgrading srv{190,219,220,221,222,223,224}.pmtpa.wmnet for libtiff vulnerabilities (USN-1631-1)
17:40 cmjohnson1: wtp1 raid1 config is wrong; bringing server down to fix
17:12 notpeter: removing srv248, srv249, and srv250 from various pools for upgrade to precise
16:47 paravoid: running 'reprepro update' to fetch ceph packages
16:47 cmjohnson1: shutting down srv250 to verify drac config
15:01 logmsgbot_: reedy synchronized php-1.21wmf4/extensions/SiteMatrix/
14:32 notpeter: removing srv200-srv213 from pybal for upgrades to precise (and this time I mean it)
13:48 logmsgbot_: py synchronized wmf-config/db.php 'pulling es2 for conversion to inno'
11:26 logmsgbot_: kaldari synchronized php-1.21wmf4/extensions/UploadWizard/resources/mw.UploadWizardUploadInterface.js 'refixing for Safari, part 2'
11:26 logmsgbot_: kaldari synchronized php-1.21wmf4/extensions/UploadWizard/resources/mw.UploadWizardDetails.js 'refixing for Safari, part 1'
11:15 logmsgbot_: kaldari synchronized php-1.21wmf4/extensions/UploadWizard/resources/mw.UploadWizardUploadInterface.js 'refixing for Safari, part 2'
11:14 logmsgbot_: kaldari synchronized php-1.21wmf4/extensions/UploadWizard/resources/mw.UploadWizardDetails.js 'refixing for Safari, part 1'
11:04 logmsgbot_: kaldari synchronized php-1.21wmf4/extensions/UploadWizard/resources/mw.UploadWizardDetails.js 'rolling back my fix for Safari, part 1'
11:03 logmsgbot_: kaldari synchronized php-1.21wmf4/extensions/UploadWizard/resources/mw.UploadWizardUploadInterface.js 'rolling back my fix for Safari, part 2'
10:48 logmsgbot_: kaldari synchronized php-1.21wmf4/extensions/UploadWizard/resources/mw.UploadWizardUploadInterface.js 'syncing UploadWizard fix for Safari, part 2'
10:47 logmsgbot_: kaldari synchronized php-1.21wmf4/extensions/UploadWizard/resources/mw.UploadWizardDetails.js 'syncing UploadWizard fix for Safari, part 1'
09:14 logmsgbot_: aaron synchronized php-1.21wmf4/maintenance/nextJobDB.php 'deployed 2e0c24df43588bd4ceba5522bde1b3f06fbd05b0'
07:25 Nemo_bis: Job queue still suspect since the wmf4 deploy, job runners activity halved; see bug 41656
04:20 K4-713: updated payments cluster to a24d39a120f61cb
02:54 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Thu Nov 15 02:54:48 UTC 2012
02:34 logmsgbot_: LocalisationUpdate completed (1.21wmf4) at Thu Nov 15 02:34:15 UTC 2012
02:20 K4-713: updated payments cluster to 46bcee539b93f7b
01:49 logmsgbot_: catrope synchronized php-1.21wmf4/extensions/VisualEditor 'Updating VE to master'
01:49 logmsgbot_: catrope synchronized php-1.21wmf3/extensions/VisualEditor 'Updating VE to master'
01:45 logmsgbot_: catrope synchronized php-1.21wmf4/extensions/VisualEditor 'Updating VE to master'
01:44 logmsgbot_: catrope synchronized php-1.21wmf3/extensions/VisualEditor 'Updating VE to master'
00:46 K4-713: updated payments cluster to ab3d3615bf1ddd0c
00:35 logmsgbot_: catrope synchronized php-1.21wmf4/extensions/VisualEditor 'Updating VE to master'
00:35 logmsgbot_: catrope synchronized php-1.21wmf3/extensions/VisualEditor 'Updating VE to master'
00:23 K4-713: updated payments cluster to 4ccd348df7661
00:11 logmsgbot_: asher synchronized wmf-config/db.php 'pulling db11'

November 14

23:59 K4-713: Updated payments cluster to 5edbe2cdc1689d14ebb883
23:50 logmsgbot_: catrope synchronized php-1.21wmf4/extensions/VisualEditor 'Updating VE to master'
23:49 logmsgbot_: catrope synchronized php-1.21wmf3/extensions/VisualEditor 'Updating VE to master'
23:27 logmsgbot_: catrope synchronized wmf-config/InitialiseSettings.php 'Dark launch VisualEditor on enwiki'
23:11 logmsgbot_: catrope synchronized php-1.21wmf4/extensions/VisualEditor 'Updating VE to master'
23:10 logmsgbot_: catrope synchronized php-1.21wmf3/extensions/VisualEditor 'Updating VE to master'
22:55 logmsgbot_: aaron synchronized wmf-config/filebackend.php
22:55 RoanKattouw: Belatedly starting VisualEditor dark launch, going a bit over time
22:52 logmsgbot_: aaron synchronized wmf-config/filebackend.php 're-enabled writes to thumbs2/'
22:43 logmsgbot_: aaron synchronized wmf-config/filebackend.php
22:41 logmsgbot_: aaron synchronized wmf-config/filebackend.php
22:22 logmsgbot_: aaron synchronized wmf-config/filebackend.php 'revert for a moment'
22:19 logmsgbot_: aaron synchronized wmf-config/filebackend.php 'Multiwrite thumbs to /thumbs2 and math/timeline files to nas as well.'
21:55 logmsgbot_: reedy synchronized php-1.21wmf4/extensions/Echo/
21:48 logmsgbot_: pgehres synchronized wmf-config/CommonSettings.php
21:41 logmsgbot_: pgehres synchronized php-1.21wmf4/extensions/ContributionReporting/
21:32 notpeter: repooling srv200-srv213 due to lack of automated imaging :(
21:19 binasher: restarted lsearchd on all pmtpa lucene hosts
20:44 notpeter: removing srv200-srv213 from apaches pool for upgarde to precise
20:42 binasher: fixed private wiki search index builds in pmtpa (standby cluster) and building initial wikivoyagedata indices there
20:38 Reedy: Killing /tmp/mw-cache-1.20wmf* from all apaches
20:20 mutante: installing some package upgrades on sodium (lists)
20:10 Reedy: Running sync-common on mw53
19:56 Reedy: srv269 has a full /tmp. Cleaning up
19:55 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else (non wikipedia) to 1.21wmf4
19:52 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks and wikinews to 1.21wmf4
19:46 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: private, readonly, closed and special to 1.21wmf4
19:40 paravoid: depooling ms-fe1 for rewrite.py testing
19:03 Reedy: Running sync-common on mw48
18:28 notpeter: temp stopping puppet on brewster
18:24 mutante: patching Bugzilla to 4.0.9
17:56 logmsgbot_: aaron synchronized php-1.21wmf3/extensions/TimedMediaHandler 'deployed 275aa9524ea6e1e15d7eef3d81c68dde3ffd5693'
17:46 Reedy: oai.oaiaudit cleaned out on S3
16:39 RobH: server wmf5815 named 'otto' after Jan Otto, name suggested by DannyB
16:22 logmsgbot_: py synchronized wmf-config/db-secondary.php 'updating comments in db-secondary.php'
12:52 mark: Reinstall ms-be3001
04:11 Tim: on zhwiki: removing refreshLinks2 jobs for templates "Side box", "Sister", "Flag/core"
03:25 Tim: on zhwiki: removing 970k country data jobs
02:50 logmsgbot_: LocalisationUpdate completed (1.21wmf4) at Wed Nov 14 02:50:04 UTC 2012
02:29 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Wed Nov 14 02:29:53 UTC 2012
00:54 logmsgbot_: aaron synchronized php-1.21wmf4/extensions/TimedMediaHandler 'deployed 2784d7ab1b0e0a56b905ff4b9c4070f63f867d25'
00:39 logmsgbot_: aaron synchronized wmf-config/filebackend.php 'Set "swiftTempUrlKey"'
00:28 AaronSchulz: Set X-Account-Meta-Temp-Url-Key for swift mw:thumb account
00:27 logmsgbot_: aaron synchronized wmf-config/PrivateSettings.php
00:21 logmsgbot_: asher synchronized wmf-config/mc.php 'disable general memcached multiwriting: nothing but pecl'
00:19 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'remove apache memchached's from the parsercache multiwriter'

November 13

22:56 logmsgbot_: kaldari synchronized wmf-config/PrivateSettings.php 'syncing PrivateSettings back to previous version'
22:41 binasher: converted puppet.updates_update table to innodb - received the most number of of any puppet db table and was myisam, yikes. this might help with concurrency.
22:31 LeslieCarr: restarting gmetad on nickel
22:22 Reedy: Publishing schema for lucene/oai on mw.o....
22:07 mutante: mailman lists: renaming wikimedia-th to wikimedia-th-private
22:06 logmsgbot_: kaldari synchronized wmf-config/PrivateSettings.php 'syncing PrivateSettings for Flickr API key'
18:43 RobH: analytics1007 now pxe boots
18:41 RobH: analytics1007.mgmt locked up, had to remove all power to fix, now working
18:33 RobH: testing analytics1002, pxe boots no problem
17:53 paravoid: setting up media1:thumbs on nas1001-a and semi-sync snapmirror from nas1-a
17:24 Jeff_Green: wikimedia.org zone update, new DKIM keys for silverpop
16:54 paravoid: snapmirror between nas1-a/nas1001-a for all volumes in-sync again
16:50 paravoid: running "cf giveback" on nas1001-b to restore nas1001-a; had to "partner snapmirror off" first.
16:46 apergos: had to restart varnish frontend by hand after reboot, also the varnishncsa for emery, locke, multicast
16:30 apergos: rebooting cp1042 (XFS: possible memory allocation deadlock in kmem_alloc), lots of stuck /bin/ps form nagios, load average ridiculous
16:10 paravoid: creating media1:thumbs volume at nas1-a and setting up /etc/export rules for it
16:08 cmjohnson1: provision labsdb3
15:32 cmjohnson1: provisioning labsdb2
15:14 hashar: restarting jenkins on gallium
08:57 logmsgbot_: reedy synchronized php-1.21wmf4/extensions/TimedMediaHandler
08:56 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler
02:39 logmsgbot_: LocalisationUpdate completed (1.21wmf4) at Tue Nov 13 02:39:21 UTC 2012
02:22 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Tue Nov 13 02:22:06 UTC 2012

November 12

22:46 Tim: deleted refreshLinks jobs corresponding to my recent edit to enwiki's Template:Citation/core
21:47 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
21:45 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
21:43 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
21:32 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
21:17 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
21:15 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
21:11 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
19:22 RoanKattouw: Updated Parsoid on wtp1
19:15 logmsgbot_: reedy synchronized wmf-config/
19:13 logmsgbot_: reedy synchronized php-1.21wmf4/extensions/EducationProgram
19:02 logmsgbot_: reedy synchronized php-1.21wmf4/extensions/Wikibase
18:53 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki, testwiki and wikidatawiki to 1.21wmf4
18:53 paravoid: upgrading tmh1/tmh2 and srv{190,219,220,221,222,223,224} for libav vulnerability (USN-1630-1)
18:52 logmsgbot_: reedy synchronized php-1.21wmf4/cache/l10n/
18:36 logmsgbot_: reedy gracefulled all apaches
18:33 logmsgbot_: reedy gracefulled all apaches
18:29 logmsgbot_: reedy synchronized live-1.5/
18:19 sbernardin: changing ssd's on labsdb3
18:05 logmsgbot_: reedy synchronized live-1.5/
18:04 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/EducationProgram/
17:59 logmsgbot_: reedy synchronized live-1.5/
17:34 logmsgbot_: reedy Started syncing Wikimedia installation... : Build 1.21wmf4 messages
17:14 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.21wmf4
17:07 logmsgbot_: reedy synchronized live-1.5/
17:05 logmsgbot_: reedy synchronized wmf-config/ExtensionMessages-1.21wmf4.php
17:03 logmsgbot_: reedy synchronized php-1.21wmf4/cache
17:02 logmsgbot_: reedy synchronized php-1.21wmf4/ 'Initial sync'
15:28 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/EducationProgram/
15:27 logmsgbot_: reedy synchronized php-1.21wmf3/includes/api/ApiEditPage.php
14:59 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/CentralNotice/
14:14 logmsgbot_: py synchronized wmf-config/db.php 're-adding es4'
11:43 paravoid: apt: removing swift 1.5 from precise-wikimedia
02:32 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Mon Nov 12 02:32:41 UTC 2012

November 11

21:51 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
21:44 paravoid: deploying squid config, adding wikivoyage to the swift regexp
21:26 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'rv'
21:22 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Disable mysql search for wikidata wiki...'
20:01 Reedy: Running sync-common on tmh1
19:58 mutante: powercycling tmh1
19:51 logmsgbot_: reedy synchronized php-1.21wmf3/maintenance/refreshLinks.php
19:49 logmsgbot_: reedy synchronized wmf-config/
19:49 logmsgbot_: demon synchronized wmf-config/InitialiseSettings.php
19:47 logmsgbot_: reedy synchronized php-1.21wmf3/maintenance/refreshLinks.php
19:43 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
17:55 logmsgbot_: dzahn synchronized ./wmf-config/InitialiseSettings.php
15:12 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler
14:48 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/CentralAuth
14:34 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/WikimediaIncubator/
14:17 logmsgbot_: reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
14:16 logmsgbot_: reedy synchronized search-redirect.php
02:32 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Sun Nov 11 02:32:27 UTC 2012
01:21 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
01:08 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
00:05 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
00:02 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php

November 10

23:25 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/CreditsSource/
22:44 mutante: DNS - raise wikivoyage.org TTL back to regular 3600 (1h)
20:51 binasher: indexing {de,fr,it,nl,ru,sv}wikivoyage
20:36 binasher: indexing wikidatawiki
20:12 binasher: attempting to build a lucene index for enwikivoyage
20:10 logmsgbot_: reedy synchronized php-1.21wmf3/cache/interwiki.cdb
20:10 logmsgbot_: reedy synchronized php-1.21wmf3/cache/interwiki.cdb
20:06 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
18:57 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler/
18:55 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Fix wikivoyage sul image location'
18:16 logmsgbot_: reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
18:02 logmsgbot_: reedy synchronized php-1.21wmf3/cache/interwiki.cdb 'Updating interwiki cache'
17:52 logmsgbot_: dzahn gracefulled all apaches
17:40 logmsgbot_: reedy synchronized extract2.php
17:36 logmsgbot_: dzahn gracefulled all apaches
17:27 logmsgbot_: reedy synchronized images/sul/
17:16 logmsgbot_: dzahn gracefulled all apaches
17:00 logmsgbot_: reedy synchronized extract2.php
16:41 logmsgbot_: dzahn gracefulled all apaches
16:39 mutante: sync-apache and graceful to redirect wikivoyage.de
16:38 logmsgbot_: reedy synchronized wmf-config/
16:37 logmsgbot_: reedy synchronized wmf-config/
16:36 logmsgbot_: reedy synchronized images/sul/wikivoyage.png
16:29 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
16:28 mutante: authdns-update for wikivoyage.org - project launch! wee
15:57 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
15:44 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Disable MF for wikivoyage'
15:35 logmsgbot_: reedy synchronized php-1.21wmf3/includes/Article.php
14:05 logmsgbot_: reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
13:31 Reedy: Deleting 1.21wmf2 l10n cache
13:27 Reedy: Pointing php symlink to 1.21wmf3
13:23 logmsgbot_: reedy synchronized php-1.21wmf3/includes/Revision.php
12:59 logmsgbot_: reedy synchronized php-1.21wmf3/includes/resourceloader/ResourceLoaderWikiModule.php
12:58 logmsgbot_: reedy synchronized php-1.21wmf3/includes/
12:55 logmsgbot_: reedy synchronized php-1.21wmf3/cache/interwiki.cdb
12:41 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
11:19 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
02:30 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Sat Nov 10 02:30:06 UTC 2012
00:55 mutante: authdns update to set MX for wikivoyage-old to us

November 9

23:17 binasher: added some redis metrics to ganglia for mc1-16
22:41 cmjohnson1: provisioning labsdb1
22:11 logmsgbot_: dzahn synchronized ./wmf-config/InitialiseSettings.php
21:20 logmsgbot_: reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
20:43 notpeter: repooling mw60 in bits apaches
20:26 sbernardin: installing ssd's in labsdb1 and labsdb2
19:53 mutante: git reset in /h/w/common/ to revert local disabling of "voy" wikis and synced dblist again
19:40 mutante: doing a git pull in wmf-config
19:17 mutante: running sync-dblist
19:15 notpeter: rebooting mw60 for upgarde to precise
19:05 mutante: enabling wikivoyage subdomains in apache for QA check
19:05 logmsgbot_: dzahn gracefulled all apaches
18:54 cmjohnson1: authdns update "Adding .pmtpa.wmnet entries for labsdb1-3"
18:45 mutante: wikivoyage imports: "de" (and all for now) done
18:38 mutante: wikivoyage imports: "en" done, "it" done
16:54 sbernardin: shutting down ms-be7 to row C sdtpa
16:53 sbernardin: racking new ms-be7 to row C sdtpa
16:51 cmjohnson1: ms-be7 going offline to make room for new 720 servers
14:36 cmjohnson1: authodns committing mgmt ip for labsdb3
12:03 apergos: ms-be6 deployed and back in swift rings on shiny new 720xd
05:00 logmsgbot_: tstarling synchronized wmf-config/InitialiseSettings.php 'enabling patrol on wikidatawiki'
02:47 logmsgbot_: olivneh synchronized php-1.21wmf3/extensions/E3Experiments/Experiments.hooks.php 'Fixing OpenTask event log bug'
02:23 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Fri Nov 9 02:23:01 UTC 2012
01:58 binasher: converted all s3 searchindex tables to innodb
01:58 notpeter: starting innobackupex from es3 to es4
01:49 logmsgbot_: olivneh synchronized php-1.21wmf3/extensions/E3Experiments/experiments/openTask.js
01:36 logmsgbot_: py synchronized wmf-config/db.php 'pulling es4'
01:18 mutante: installing package upgrades on kaulen, half-configured apache2 brought down bugzilla ...
00:56 Tim: on fenari: killed stuck git processes run by ExtensionDistributor and LocalisationUpdate
00:52 logmsgbot_: LocalisationUpdate failed: git pull of extensions failed
00:31 logmsgbot_: pgehres synchronized wmf-config
00:25 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/SiteMatrix/

November 8

23:56 logmsgbot_: pgehres synchronized wmf-config
23:51 K4-713: updated payments cluster to 7d92a1a35f0
23:51 logmsgbot_: pgehres synchronized php-1.21wmf3/extensions/CentralNotice/ 'Updating CentralNotice to master'
23:29 logmsgbot_: olivneh synchronized php-1.21wmf3/extensions/E3Experiments
21:18 cmjohnson1: ms-be6 initial puppet run
21:14 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'multiwrite pcache to new memcached servers, only use redis for sessions'
21:08 logmsgbot_: olivneh synchronized php-1.21wmf3/extensions/EventLogging
21:01 logmsgbot_: olivneh synchronized php-1.21wmf3/extensions/E3Experiments
21:01 mutante: restarted nginx on all SSL proxies without issues now. puppet runs again and does not recreate configs for data/voyage on esams.
20:55 mutante: removing wikidata/voyage sites config from esams ssl hosts
20:54 cmjohnson1: ms-be6 another OS install
20:51 mutante: fixed wikipedia ssl chained cert on ssl1 (all other hosts had it recreated as expected)
20:50 apergos: ignore the fail event on ms-be6, this was caused manually and is a bogus warning
20:31 logmsgbot_: demon synchronized wmf-config/CommonSettings.php 'Enabling lucene for wikidatawiki'
19:29 logmsgbot_: dzahn synchronized ./wmf-config/throttle.php
18:16 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler
18:15 mutante: removing wm-nl@wikimedia.org as list admin of "press-nl" because it is an OTRS queue (per thehelpfulone)
15:55 paravoid: starting ms-be3 {object,container,account}-server
14:42 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Thu Nov 8 14:42:45 UTC 2012
14:13 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Thu Nov 8 14:13:54 UTC 2012
12:23 mark: Reinstalling sq70 with Precise
10:57 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
10:57 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/ProofreadPage/
09:49 mark: Killed show_bug.cgi processes on kaulen
09:21 mark: Power cycled cp1041
07:20 apergos: restarted lucene search on search1015
06:26 apergos: restarted lucene search on searchpool1016
03:03 logmsgbot_: aaron synchronized php-1.21wmf3/includes/objectcache/MemcachedPeclBagOStuff.php 'deployed efda0b27befc28ec13dae23f48e8baea42f4a7f3'
02:00 logmsgbot_: LocalisationUpdate failed: git pull of extensions failed
00:52 logmsgbot_: reedy synchronized wmf-config/
00:42 notpeter: repooling palladium
00:40 mutante: wikivoyage imports: "fr" - done
00:39 logmsgbot_: aaron synchronized php-1.21wmf3/extensions/SwiftCloudFiles/php-cloudfiles-wmf/cloudfiles.php 'removed debug logging'
00:37 logmsgbot_: reedy synchronized all.dblist
00:35 logmsgbot_: aaron synchronized php-1.21wmf3/extensions/SwiftCloudFiles/php-cloudfiles-wmf/cloudfiles.php
00:34 paravoid: temporarily stopping {account,container,object}-server on ms-be3 while it syncs up from its downtime
00:34 paravoid: starting memcached on virt0
00:30 logmsgbot_: aaron synchronized php-1.21wmf3/extensions/SwiftCloudFiles/php-cloudfiles-wmf/cloudfiles.php
00:26 logmsgbot_: dzahn gracefulled all apaches
00:26 paravoid: powercycling ms-be3

November 7

23:48 notpeter: removing palladium from eqiad bits cache pool for upgrade to precise
23:43 notpeter: repooling niobium
23:02 notpeter: removing niobium from eqiad bits cache pool for upgrade to precise
22:59 logmsgbot_: asher synchronized wmf-config/db.php 'returning db64 to s3'
22:51 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'move redis to the front of the session multiwrite array'
22:41 notpeter: re-pooling strontium
22:28 logmsgbot_: asher synchronized wmf-config/db.php 'pulling db64'
22:08 mutante: powercycling db64 (frozen)
22:02 logmsgbot_: aaron synchronized wmf-config/filebackend.php
21:57 notpeter: removing strontium from eqiad bits cache pool for upgrade to precise
21:30 logmsgbot_: asher synchronized wmf-config/mc.php 'disable persistent connections for memcached-pecl'
21:26 logmsgbot_: asher synchronized wmf-config/mc.php 'persistent connections for memcached-pecl'
21:10 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'enabling session multiwrite to redis on mc1-16'
20:59 mutante: wikivoyage imports: "ru" - done
20:54 binasher: upgraded redis on mc1-16
20:50 logmsgbot_: aaron synchronized php-1.21wmf3/includes/filerepo/file/LocalFile.php 'debug logging.'
20:45 mutante: wikivoyage imports: "sv" - done
20:42 binasher: added new redis-2.6.3 pkg to precise-wikimedia and upgraded mc1
20:34 mutante: wikivoyage imports: "nl" - done
20:31 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler
20:09 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'disabling session multiwrite to redis on mc1'
19:56 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
19:53 paravoid: powercycling zinc, unresponsive to network and serial console
19:51 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.21wmf3
19:46 mutante: point wlm.wikimedia.org to the cluster rather than stand-alone host yttrium
19:25 notpeter: re-adding arsenic to bits cache pool
19:00 logmsgbot_: dzahn gracefulled all apaches
18:42 sbernardin: racking new ms-be6 to row B sdtpa
17:43 cmjohnson1: racking ms-be6 (720xd)
17:36 Reedy: Creating transcode_key_idx on all transcode tables on all wikis
17:08 logmsgbot_: demon synchronized wmf-config/InitialiseSettings.php 'Enabling TMH for commonswiki'
17:08 logmsgbot_: demon synchronized wmf-config/CommonSettings.php 'Enabling TMH for commonswiki'
15:23 paravoid: reactivating tele2 ipv6 peering and re-exporting routes to it
14:46 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler/
13:26 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler/
13:11 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler/
12:52 mark: Starting load testing of cp3003 and cp3004
11:57 mark: Increased weight of cp3004 to 9
11:16 mark: Pooled cp3004 with weight 1
07:14 logmsgbot_: dzahn gracefulled all apaches
02:49 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Wed Nov 7 02:49:00 UTC 2012
02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Wed Nov 7 02:28:51 UTC 2012
01:48 logmsgbot_: reedy synchronized all.dblist 'frwikivoyage'
01:46 logmsgbot_: dzahn gracefulled all apaches
01:13 logmsgbot_: demon synchronized php-1.21wmf3/extensions/ExtensionDistributor/ExtensionDistributor_body.php 'Revert debugging'
01:05 logmsgbot_: demon synchronized php-1.21wmf3/extensions/ExtensionDistributor/ExtensionDistributor_body.php 'Debugging'
00:56 logmsgbot_: demon synchronized php-1.21wmf3/extensions/ExtensionDistributor/ExtensionDistributor_body.php 'Debugging'

November 6

23:57 mutante: added gpg key for hexmode to keys.html/keys.txt
23:51 logmsgbot_: dzahn gracefulled all apaches
23:49 logmsgbot_: reedy synchronized all.dblist 'Kill wikivoyage from all.dblist'
23:44 logmsgbot_: reedy synchronized closed.dblist
23:42 logmsgbot_: reedy synchronized closed.dblist
23:41 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
23:31 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
23:30 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
23:30 logmsgbot_: spage synchronized php-1.21wmf3/extensions/E3Experiments 'ACUX interactive validation'
23:16 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'setting s3 to writeable again'
22:49 mutante: creating http://dumps.wikimedia.org/mediawiki/1.20/ per hexmode
22:36 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'setting s3 to read-only for emergency db maintenance'
22:14 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler/
22:01 mutante: adding wikivoyage.org/.de alias file on mchenry
21:00 mutante: imported frwikivoyage db dumps
20:53 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
20:03 logmsgbot_: dzahn gracefulled all apaches
19:57 logmsgbot_: dzahn gracefulled all apaches
19:56 Jeff_Green: restarted udp2log on locke
19:24 AaronSchulz: finished cleanupBug41615.php script runs for affected wikis (s2,s3,s4,s7)
18:57 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/WikimediaMaintenance/
18:46 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
18:44 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
18:39 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
18:39 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
18:35 logmsgbot_: reedy synchronized wmf-config/
18:30 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
18:27 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
18:22 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
18:18 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/WikimediaMaintenance/
17:03 logmsgbot_: reedy synchronized multiversion/ 'Resync multiversion after changing to a submodule'
17:01 logmsgbot_: reedy synchronized multiversion/ 'Resync multiversion after changing to a submodule'
16:59 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler/
16:06 logmsgbot_: demon synchronized php-1.21wmf3/extensions/Wikibase/repo/Wikibase.php 'Syncing out 856bad9d'
16:05 ^demon: cleaned up fenari:/h/w/c/php-1.21wmf3 history so it can properly ff on pull
11:37 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler/
11:33 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
10:33 logmsgbot_: nikerabbit synchronized php-1.21wmf3/extensions/WebFonts/resources/ext.webfonts.fontlist.js
10:22 logmsgbot_: nikerabbit synchronized php-1.21wmf3/extensions/Translate/utils/MessageGroupStats.php
09:12 logmsgbot_: hashar synchronized docroot/mediawiki/xml 'import updated export-0.8.xsd'
02:55 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Tue Nov 6 02:55:06 UTC 2012
02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Tue Nov 6 02:28:55 UTC 2012
01:57 logmsgbot_: reedy synchronized wmf-config/
01:14 binasher: slamming the apache cluster by wiping out all mobile caching yet again
00:59 logmsgbot_: awjrichards synchronized php-1.21wmf2/extensions/MobileFrontend/
00:59 logmsgbot_: awjrichards synchronized php-1.21wmf3/extensions/MobileFrontend/javascripts/ 'touch files'
00:49 logmsgbot_: awjrichards synchronized php-1.21wmf3/extensions/MobileFrontend/javascripts/modules/mf-references.js 'touch file'
00:44 logmsgbot_: awjrichards synchronized php-1.21wmf3/extensions/MobileFrontend
00:40 logmsgbot_: reedy synchronized php-1.21wmf3/cache/l10n/ 'Update l10n cache'
00:18 Tim: on gallium: moving data directories to a tmpfs
00:13 K4-713: updated the payments cluster to bd673e77d6adde3
00:04 logmsgbot_: reedy synchronized wmf-config/

November 5

23:52 logmsgbot_: reedy synchronized multiversion
23:43 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/UserMerge
23:36 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwikivoyage to 1.21wmf3
23:35 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
23:34 logmsgbot_: reedy synchronized wmf-config/
23:27 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
22:23 binasher: rebooting srv191,225,230 for kernel upgrades
22:10 binasher: rebooting mw21,25,40 for kernel upgrades
21:54 logmsgbot_: aaron synchronized wmf-config/mc.php 'Switched all wikis to memcached-multiwrite.'
21:54 logmsgbot_: aaron synchronized wmf-config/InitialiseSettings.php 'Disabled spammy (with pecl memcached) memcached.log, the useful one is memcached-serious.'
21:49 logmsgbot_: asher gracefulled all apaches
21:36 binasher: reboot mw49 for kernel upgrade
21:19 binasher: rebooted mw28
21:19 mutante: copied interwiki.cdb from prod to wikitech, used $wgInterwikiCache to point to .cdb, mw does not use mysql table anymore. long iw links work again, "shortcuts" are different though and still an issue
21:09 logmsgbot_: aaron synchronized wmf-config/mc.php 'Added (zh|de)wiki to memc multiwrite and switched read order.'
20:42 notpeter: re-adding sq67 to bits cache pool w/o upgarde and removing arsenic from eqiad bits cache pool for actual upgrade to precise
20:40 notpeter: removing sq67 from bits cache pool for upgrade to precise
19:48 Jeff_Green: payments flipped to eqiad cluster
19:02 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf3
18:22 logmsgbot_: aaron synchronized wmf-config/InitialiseSettings.php 'TMH deploy too all except commons.'
17:58 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler/
17:57 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/TimedMediaHandler/
17:56 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/MwEmbedSupport/
17:35 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler/
17:33 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/MwEmbedSupport
17:21 logmsgbot_: py gracefulled all apaches
17:20 notpeter: gracefulling all apaches to pick up https://gerrit.wikimedia.org/r/#/c/31670/
17:19 mutante: DNS update - push wikivoyage.de (link)
16:57 mark: Reinstalling sq69
16:17 mark: Reinstalling sq68 with Precise
12:38 mark: Performance testing cp3003
11:38 apergos: synced apache/common-local by hand to tmh1 and tmh2 and restart job runners on both hosts (the directory was previously empty)
09:49 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'wikidatawiki to use noticeproject::wikimedia banner'
09:36 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php '(bug 41717) Update default (language-neutral) Wiktionary logo'
09:26 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php '(bug 41774) New wgImportSource for ro.wikibooks'
09:17 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php '(bug 38134) Enable Extension:GoogleNewsSitemap on es wikinews'
09:15 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php '(bug 40212) Mass update Wiktionary favicons'
09:02 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'Configure Babel category for Wikidata 31608'
09:00 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'whitespace fix 31205'
08:57 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php '(bug 40212) Mass update Wiktionary favicons'
08:53 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php '(bug 32411) Transwiki import to multilingual wikisource broken'
08:40 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php '(bug 41167) Namespace configuration for ba.wikipedia'
08:25 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php '(bug 40717) Namespace configuration for th.wiktionary and th.wikibooks'
08:24 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php '(bug 40717) Namespace configuration for th.wiktionary and th.wikibooks'
02:51 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Mon Nov 5 02:51:13 UTC 2012
02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Mon Nov 5 02:28:01 UTC 2012

November 4

05:31 logmsgbot_: catrope synchronized php-1.21wmf3/extensions/TimedMediaHandler/
05:30 logmsgbot_: catrope synchronized php-1.21wmf2/extensions/TimedMediaHandler/
02:53 K4-713: updated the payments cluster to 3a315eafc09015e
02:49 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Sun Nov 4 02:49:49 UTC 2012
02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Sun Nov 4 02:28:44 UTC 2012

November 3

17:46 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
17:34 logmsgbot_: reedy synchronized php-1.21wmf3/includes/
06:50 apergos: (from half an hour ago) restarted lucene search on search1015. stupid shift key
02:50 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Sat Nov 3 02:50:07 UTC 2012
02:29 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Sat Nov 3 02:29:23 UTC 2012

November 2

23:45 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/ 'Dark deploy wikivoyage extensions'
22:34 AaronSchulz: Created TMH table on all wikis
21:55 logmsgbot_: dzahn gracefulled all apaches
21:55 mutante: adding apache config for wikivoyage
21:24 mutante: authdns-update, adding new zone for wikivoyage-old
21:08 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/ProofreadPage
19:49 cmjohnson1: srv266 troubleshooting cpu
18:39 logmsgbot_: aaron synchronized php-1.21wmf3/thumb.php 'deployed a16d25a721b502f83f6d9354d482f8f2bc08176b'
18:38 logmsgbot_: aaron synchronized php-1.21wmf2/thumb.php 'deployed 0b7378c966ead76422d03fb0c663fa714ad8a48a'
17:07 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
17:00 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
16:23 logmsgbot_: maxsem synchronized wmf-config/InitialiseSettings.php 'Close wikimania2010wiki'
16:19 logmsgbot_: maxsem synchronized closed.dblist 'https://gerrit.wikimedia.org/r/#/c/31379/'
14:39 logmsgbot_: reedy synchronized wmf-config/
13:09 paravoid: temporarily deactivating IPv6 peering with tele2 on cr1-eqiad due to routing issues
10:34 Tim: on wikitech, running compressOld.php
10:21 Tim: converting wikitech database from latin1 to binary
09:57 Tim: upgraded wikitech to MW 1.19.2
06:40 apergos: some files on wikitech (old backups etc) moved to tridge in /data/wikitech-nov2012 to give us a couple more gb of room
06:20 Tim: upgrading wikitech to MW 1.17 branch head
06:13 Tim: on wikitech: disabled debug log
05:46 Tim: wikitech.wikimedia.org had a full root partition. I resized it to fill up the current account allocation
02:26 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Fri Nov 2 02:26:03 UTC 2012
01:03 logmsgbot_: pgehres synchronized php-1.21wmf2/extensions/CentralNotice/ 'Updating CentralNotice, fixing Bug 41632'
00:29 logmsgbot_: spage Finished syncing Wikimedia installation... : PostEdit i18n, E3 ACUX updates, fixing Cross site reqest problem in CentralNotice
00:12 logmsgbot_: reedy synchronized wmf-config/

November 1

23:42 logmsgbot_: spage Started syncing Wikimedia installation... : PostEdit i18n, E3 ACUX updates, fixing Cross site reqest problem in CentralNotice
23:24 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Revert that, didn't fix the issue'
23:22 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Temporarily disable translate and translation notification on wikidatawiki'
22:59 Reedy: Dropped contest tables from testwiki
22:56 logmsgbot_: reedy synchronized wmf-config/ 'Disable Contest extension'
22:48 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
22:43 Reedy: Created translate tables on wikidatawiki
22:41 logmsgbot_: spage synchronized php-1.21wmf3/extensions/E3Experiments 'ACUX bump versions'
22:33 logmsgbot_: spage synchronized php-1.21wmf3/extensions/PostEdit 'latest PostEdit'
22:30 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Enable wikidata import from metawiki'
21:27 spagewmf: spagewmf ran `sync-dir php-1.21wmf3/extensions/E3Experiments 'ACUX updates'`
21:25 logmsgbot_: spage synchronized php-1.21wmf3/extensions/E3Experiments 'ACUX updates'
21:17 mutante: freeing disk space on wikitech
19:37 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/ProofreadPage
19:01 LeslieCarr: (modifying above ^^) pulled IPv6 (only) announcements from AS1257 on cr1-eqiad ONLY due to routing issues
19:00 LeslieCarr: pulled IPv6 (only) announcements from AS1257 due to routing issues
18:39 mutante: deleting duplicate docroot for wikidata in /h/w/common/docroot/
18:14 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Test
18:02 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/TimedMediaHandler
17:49 logmsgbot_: reedy synchronized php-1.21wmf2/includes/
17:48 logmsgbot_: mlitn synchronized php-1.21wmf3/extensions/ArticleFeedbackv5 'desc'
17:47 logmsgbot_: mlitn synchronized php-1.21wmf2/extensions/ArticleFeedbackv5 'desc'
17:41 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Re-enable TMH on enwiki'
17:34 logmsgbot_: reedy synchronized php-1.21wmf3/includes/cache/MessageCache.php
17:34 logmsgbot_: reedy synchronized php-1.21wmf3/includes/Message.php
17:18 logmsgbot_: reedy synchronized php-1.21wmf3/includes/
17:17 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler
17:14 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Enable PageTriage on test2wiki'
17:10 Reedy: Created PageTriage tables on test2wiki
17:08 hashar: gallium is sent to swap from time to time. The cause seems to be the Ext-Wikibase job for which update.php eat all memory. I have disabled the job meanwhile. In case of trouble, simply kill the wild php process.
16:50 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Disable TMH on enwiki'
16:42 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Enable TMH and mwembed on enwiki'
16:36 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/MwEmbedSupport/
16:35 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/MwEmbedSupport/
16:35 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/TimedMediaHandler/ 'Update to master'
16:34 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler/ 'Update to master'
16:32 mark: Restarted backend varnish instance on cp1029
16:14 mark: Reinstalling sq67 with Precise
15:56 logmsgbot_: reedy synchronized php-1.21wmf3/includes/specials/SpecialUndelete.php
15:56 logmsgbot_: reedy synchronized php-1.21wmf3/includes/Revision.php
15:54 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/Wikibase/lib/includes/Utils.php
15:41 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/Wikibase
14:06 mark: Restarted backend varnish instance on cp1027
13:12 mark: Repooled cp3003
12:35 mark: Depooled cp3003, high rate of 500 responses
12:29 mark: Pooled cp3003 as upload Varnish cluster in PyBal with weight 1
10:55 mark: Restarted backend varnish instance on cp1026
08:46 mark: Restarted backend varnish instance on cp1025
08:40 apergos: running some nonpriority jobs manually on mw12 (so people later don't get weirded out bu the ganglia graphs)
08:21 Nemo_bis: Job queue runners almost idle again since 5 UTC; en.wiki queue at 145k, zh.wiki 260k, Commons 60k now
07:40 mark: Killed and restarted lucene on search1016
04:47 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: reverted wmf3 deployment made earlier today.
03:59 logmsgbot_: tstarling synchronized php-1.21wmf3/includes/job/JobQueueDB.php 'fix deadlocks'
02:54 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Thu Nov 1 02:54:45 UTC 2012
02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Thu Nov 1 02:28:03 UTC 2012
00:12 mutante: installing package upgrades on marmontel (blog)

October 31

23:39 logmsgbot_: reedy synchronized wmf-config/
23:23 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/Wikibase
23:10 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
22:37 logmsgbot_: reedy synchronized php-1.21wmf3/maintenance/nextJobDB.php
21:51 logmsgbot_: reedy synchronized php-1.21wmf3/maintenance/nextJobDB.php
21:35 K4-713: payments synchronized to 5d40dbe27d05
21:12 logmsgbot_: reedy synchronized php-1.21wmf3/includes/site/SiteObject.php
20:42 logmsgbot_: aaron synchronized php-1.21wmf2/extensions/TimedMediaHandler 'deployed cbe526b06ec1876c933f8a04c2526b137642a879'
20:41 logmsgbot_: aaron synchronized php-1.21wmf3/extensions/TimedMediaHandler 'deployed bf098d262dce290ac4b0cf70f74182c98aeca4d9'
20:09 Reedy: Restarting Jenkins
20:09 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Bug 41528 - need more memory for video thumbnails'
19:35 logmsgbot_: preilly Finished syncing Wikimedia installation... : update zero rated mobile access
19:34 mark: Restarted backend varnish instance on cp1024
19:33 K4-713: updated payments cluster to 5d8309e21
18:54 logmsgbot_: preilly Started syncing Wikimedia installation... : update zero rated mobile access
18:41 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else non wikipedia or wikivoyage to 1.21wmf3
18:40 logmsgbot_: preilly synchronized php-1.21wmf3/extensions/ZeroRatedMobileAccess 'update post deploy'
18:39 logmsgbot_: preilly synchronized php-1.21wmf2/extensions/ZeroRatedMobileAccess 'update post deploy'
18:38 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary and wikinews to 1.21wmf3
18:31 cmjohnson1: rebuilding software raid for storage3 and provisioning
18:30 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/ReaderFeedback
18:29 mutante: srv286 - installing php5-memcached and dist-upgrade
18:27 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity, wikiquote and wikibooks to 1.21wmf3
18:25 mutante: srv286 - kill frozen apt-get update process, update sources, install package upgrades
18:18 mutante: mw18 - dpkg --configure -a to fix interrupted dpkg, configures php-luasandbox, install php5-memcached
18:17 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/ReaderFeedback
18:07 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special, closed and private to 1.21wmf3
18:01 logmsgbot_: reedy synchronized php-1.21wmf3/
18:00 logmsgbot_: reedy synchronized php-1.21wmf2/
17:56 mutante: now redirecting wikimania.wm to wikimania2013.wm
17:54 logmsgbot_: dzahn gracefulled all apaches
17:52 logmsgbot_: preilly synchronized php-1.21wmf3/extensions/MobileFrontend 'update after deploy'
17:51 logmsgbot_: preilly synchronized php-1.21wmf2/extensions/MobileFrontend 'update after deploy'
17:41 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
16:57 Reedy: Truncating wikibase.wb_changes
16:25 mark: Restarted backend varnish instance on cp1023
15:50 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'wgULSIMEEnabled = false'
15:49 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/UniversalLanguageSelector/ 'Update to master'
15:00 mark: Restarted backend varnish instance on cp1022
14:45 paravoid: putting ms-fe4 back in the pool
14:26 paravoid: temporarily depooling ms-fe4 to apply a temporary local hack
14:12 paravoid: putting ms-fe4 back in the pool
13:27 mark: Restarted cp1021 with persistent storage
13:26 paravoid: depooling ms-fe4 to upgrade to precise/swift 1.7.4
11:35 paravoid: putting ms-fe3 back in the pool
11:28 ^demon: restarted gerrit services on manganese & formey to pick up replication changes
07:59 paravoid: depooling ms-fe3 to upgrade to precise/swift 1.7.4
07:09 apergos: restarted lucene search on search1015 and search 1003
02:47 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Wed Oct 31 02:47:40 UTC 2012
02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Wed Oct 31 02:28:50 UTC 2012
02:05 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
02:01 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Re-enable TMH'
01:43 logmsgbot_: reedy synchronized wmf-config/
01:23 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Disable TMH everywhere (and mwembed at the same time)'
01:15 K4-713: synchronized the payments cluster to 85e9007aef3e75
01:06 logmsgbot_: awjrichards synchronized php-1.21wmf3/extensions/MobileFrontend/javascripts/actions/mf-edit.js 'touch file'
00:58 logmsgbot_: reedy synchronized php-1.21wmf3/includes/job/JobQueueDB.php
00:57 logmsgbot_: awjrichards synchronized php-1.21wmf2/extensions/MobileFrontend/javascripts/actions/mf-edit.js 'touch file'
00:42 logmsgbot_: kaldari synchronized php-1.21wmf2/extensions/CentralNotice 'Update CentralNotice on meta and en.wiki to fix banner override bug'
00:30 logmsgbot_: maxsem Finished syncing Wikimedia installation... :

October 30

23:44 logmsgbot_: maxsem Started syncing Wikimedia installation... :
23:08 mutante: replaced wikipedia SSL cert with new one by GeoTrust
22:16 logmsgbot_: kaldari synchronized php-1.21wmf3/extensions/CentralNotice 'autoloading ApiCentralNoticeAllocations for CentralNotice'
22:15 LeslieCarr: failing overRE mastership on cr2-eqiad to upgrade re1
22:02 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'memcached logging group'
22:00 logmsgbot_: asher synchronized wmf-config/mc.php 'testing MemcachedPeclBagOStuff on test2wiki (serverlist fix)'
21:57 logmsgbot_: asher synchronized wmf-config/mc.php 'testing MemcachedPeclBagOStuff on test2wiki'
21:47 mutante: restarting nginx on ssl hosts for cert update
21:43 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
21:30 logmsgbot_: kaldari synchronized php-1.21wmf3/extensions/CentralNotice 'updating centralNotice to autoload ApiCentralNoticeAllocations class'
21:24 logmsgbot_: kaldari synchronized php-1.21wmf2/extensions/CentralNotice 'updating centralNotice to autoload ApiCentralNoticeAllocations class'
21:23 logmsgbot_: kaldari Finished syncing Wikimedia installation... :
21:14 logmsgbot_: kaldari synchronized php-1.21wmf2/extensions/CentralNotice 'updating centralNotice to autoload ApiCentralNoticeAllocations class'
21:09 logmsgbot_: kaldari synchronized php-1.21wmf2/extensions/CentralNotice/CentralNotice.php 'updating centralNotice to autoload ApiCentralNoticeAllocations class'
20:54 Jeff_Green: stopping mysql on db1013 for cloning
20:37 logmsgbot_: kaldari Started syncing Wikimedia installation... :
20:21 logmsgbot_: kaldari synchronized wmf-config/CommonSettings.php 'updating wgCentralBannerDispatcher for CentralNotice'
19:46 logmsgbot_: kaldari synchronized wmf-config/CommonSettings.php 'updating settings for CentralNotice banner loading'
18:31 binasher: completed centralnotice patch-bucketing.sql migration
18:24 mutante: giving sbernardin racktables permissions
17:53 mark: Installed Precise on new upload esams servers cp3003-cp3010
17:20 mutante: running sync-apache on srv199 only
17:19 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'wgMaxAnimatedGifArea = 2.5e7'
17:15 logmsgbot_: reedy synchronized wmf-config/omgtestfile
17:06 Reedy: Running sync-common on srv199
17:01 Reedy: Running sync-common on srv194
16:54 logmsgbot_: reedy synchronized wmf-config/ExtensionMessages-1.21wmf3.php
16:38 apergos: stpped stray jobrunner on srv278
14:38 paravoid: powercycling srv229, dead for unknown reason so far
10:08 logmsgbot_: nikerabbit synchronized php-1.21wmf2/extensions/NewUserMessage/NewUserMessage.class.php 'I0f93ee53'
09:41 logmsgbot_: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Temporarily enable beta mappings for am wikis'
09:19 logmsgbot_: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Narayam and Webfonts: bug 41460, bug 39200, bug 41359'
09:07 mark: Fixed wikidata.org language cnames issue
07:44 mark: Fixed IPv6 addresses on new esams bits servers
07:33 mark: Replaced 2 bits @ esams servers with 4 new servers cp3019-cp3022
02:49 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Tue Oct 30 02:49:48 UTC 2012
02:24 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Tue Oct 30 02:24:04 UTC 2012
01:09 logmsgbot_: olivneh synchronized wmf-config/InitialiseSettings.php
01:09 logmsgbot_: olivneh synchronized php-1.21wmf2/extensions/EventLogging
01:02 logmsgbot_: olivneh synchronized php-1.21wmf3/extensions/EventLogging 'Updating EventLogging on test2'
00:43 logmsgbot_: olivneh synchronized php-1.21wmf2/extensions/EventLogging
00:29 logmsgbot_: aaron synchronized php-1.21wmf2/extensions/TimedMediaHandler 'deployed 18e51f3b06b84d1d5fbf47272d9ebfc5008dc879'
00:28 logmsgbot_: aaron synchronized php-1.21wmf3/extensions/TimedMediaHandler 'deployed c1ac05640377f4f99cbe2a094e80d3d25d63b93d'
00:27 logmsgbot_: olivneh synchronized php-1.21wmf3/extensions/EventLogging 'Updating EventLogging on test2'

October 29

21:28 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
21:25 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Revert'
21:24 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Change wikidata search config even more'
21:20 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Change wikidata search config even more'
21:11 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Disable lucene on wikidatawiki for now'
20:44 logmsgbot_: mlitn synchronized php-1.21wmf2/extensions/ArticleFeedbackv5
19:39 logmsgbot_: reedy synchronized live-1.5/
19:36 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.21wmf3
19:19 Reedy: Deleting php-1.20wmf11 files
19:16 logmsgbot_: reedy synchronized php-1.21wmf3/includes/Revision.php
18:25 logmsgbot_: aaron synchronized php-1.21wmf3/extensions/TimedMediaHandler 'deployed 05118462e3ddbfc507772cc3c4e1117a8527e0b5'
18:19 Reedy: Run Wikibase/repo/maintenance/createBlacklistedItems.php on wikidatawiki
18:16 logmsgbot_: catrope synchronized php-1.21wmf3/extensions/VisualEditor/ 'Update VisualEditor'
18:04 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Add wikidata to wgCrossSiteAJAXdomains'
17:54 Reedy: Ran patch-job_token.sql on wikidatawiki
17:52 logmsgbot_: demon synchronized wmf-config/CommonSettings.php 'Turning SUL back on for wikidata.org'
17:51 logmsgbot_: catrope gracefulled all apaches
17:20 Reedy: Running \Wikibase\Utils::insertDefaultSites(); on wikidatawiki
17:08 logmsgbot_: reedy synchronized php-1.21wmf3/cache/interwiki.cdb 'Updating interwiki cache'
17:07 logmsgbot_: reedy synchronized php-1.21wmf2/cache/interwiki.cdb 'Updating interwiki cache'
17:02 Reedy: Applied Wikibase/lib/sql/WikibaseLib.sql on wikidatawiki
16:50 logmsgbot_: reedy synchronized php-1.21wmf3/includes/
16:44 logmsgbot_: reedy synchronized php-1.21wmf3/includes/WikiPage.php
16:43 logmsgbot_: reedy synchronized php-1.21wmf3/includes/Revision.php
16:41 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikidatawiki to 1.21wmf3
16:38 logmsgbot_: reedy synchronized php-1.21wmf3/cache/l10n/ 'Sync'
16:30 logmsgbot_: mark synchronized wmf-config/CommonSettings.php 'Add new bits server IPs'
16:12 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuilding localisation cache for 1.21wmf3
16:08 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuilding localisation cache for 1.21wmf3
16:01 logmsgbot_: reedy Finished syncing Wikimedia installation... : Rebuilding localisation cache for 1.21wmf3
15:54 logmsgbot_: reedy synchronized wmf-config/
15:52 Reedy: Running sync-common on searchidx1001
15:51 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/PostEdit/
15:50 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/EventLogging/
15:46 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuilding localisation cache for 1.21wmf3
15:41 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.21wmf3
15:40 logmsgbot_: reedy synchronized php-1.21wmf3 'Initial file sync-out'
13:38 ^demon: flushed gerrit's account_byemail cache, error log was complaining
13:19 logmsgbot_: demon synchronized wmf-config/CommonSettings.php 'Disabling wikidata for SUL until we fix SSL'
02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Mon Oct 29 02:28:02 UTC 2012

October 28

22:37 logmsgbot_: olivneh synchronized php-1.21wmf2/extensions/EventLogging 'Updating Extension:EventLogging (enabled only on test & test2)'
22:33 logmsgbot_: olivneh synchronized wmf-config/InitialiseSettings.php 'Enable Extension:EventLogging on test2'
21:24 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'arwiki logo'
02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Sun Oct 28 02:28:08 UTC 2012

October 27

21:19 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Set geoservice to geoiplookup.wm.o'
20:51 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Comment out wgULSGeoService'
10:02 paravoid: repooling sq82 frontend in pybal
08:49 paravoid: reinstalling sq82 with lucid
07:41 apergos: depooled sq82 in pybal, squid3 was running on it again and it's still precise
02:33 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Sat Oct 27 02:33:26 UTC 2012
00:25 mutante: restarting nginx on ssl hosts for changed server name for wikidata
00:17 mutante: reloading nginx on ssl1001

October 26

23:46 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/UniversalLanguageSelector/
23:23 Ryan_Lane: powercycling kaulen
23:09 cmjohnson1: tinkering with storage3
22:24 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Set wgULSGeoService to false'
22:03 Ryan_Lane: repooing ssl1004
21:44 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'wgContentHandlerUseDB back to true. Mainpage fixed!'
21:41 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'wgContentHandlerUseDB to false for wikidatawiki'
21:27 cmjohnson1: imaging db42
20:59 Ryan_Lane: rebooting ssl1004
20:54 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Revert that'
20:52 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Disabling wikidata related extensions temporarily'
20:50 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
20:50 logmsgbot_: reedy synchronized images/sul/wikidata.png
20:43 logmsgbot_: reedy synchronized php-1.21wmf2/includes/content/ContentHandler.php
20:43 logmsgbot_: reedy synchronized php-1.21wmf2/cache/l10n/
20:43 logmsgbot_: reedy synchronized php-1.21wmf2/includes/EditPage.php
20:37 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
20:36 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
20:28 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuild localisation cache for wikidata related extensions
20:25 logmsgbot_: aaron synchronized php-1.21wmf2/extensions/MwEmbedSupport 'deployed de1079e3eec4eae4f129d7bcffc71596f415ae74'
20:15 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Enable wikibaserepo on wikidatawiki'
20:13 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
20:13 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
20:10 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Temporary hacking in of ns' for wikidatawiki'
20:07 Reedy: Adding extensions/Wikibase/repo/includes/store/sql/Wikibase.sql to wikidatawiki
20:01 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
20:00 logmsgbot_: aaron synchronized php-1.21wmf2/extensions/TimedMediaHandler 'deployed 31a1d21e8ebfb90ece5ef02075fc85f197a50e53'
19:59 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'live hacky hack'
19:55 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'live hacky hack'
19:53 cmjohnson1: rebooting sq82
19:52 logmsgbot_: reedy synchronized wmf-config/ 'gotta touch them all'
19:45 logmsgbot_: reedy synchronized wmf-config/
19:18 cmjohnson1: stopping puppet on brewster temporarily
19:17 logmsgbot_: root gracefulled all apaches
19:12 mutante: deleting files older than 3 days from /tmp/ in "mw" dsh group
19:05 mutante: deleting /tmp/mw-cache-1.20* from mw dsh group
18:57 Reedy: Cleaned /tmp on mw48
18:54 Reedy: Cleaned /tmp on mw34
18:51 Reedy: Cleaned /tmp on mw45
18:31 Jeff_Green: aluminium dist upgrade and reboot
18:27 Jeff_Green: erzurumi dist-upgrade and reboot
18:02 logmsgbot_: aaron synchronized php-1.21wmf2/thumb.php 'done'
17:58 logmsgbot_: aaron synchronized php-1.21wmf2/thumb.php 'improve path mismatch error message'
17:50 logmsgbot_: aaron synchronized php-1.21wmf2/includes/upload/UploadStash.php 'done'
17:44 logmsgbot_: aaron synchronized php-1.21wmf2/includes/upload/UploadStash.php
17:42 logmsgbot_: aaron synchronized php-1.21wmf2/includes/upload/UploadStash.php 'debug logging'
17:35 logmsgbot_: dzahn gracefulled all apaches
17:32 mutante: sync-apache new wikidata config
16:49 logmsgbot_: demon synchronized wmf-config/InitialiseSettings.php 'Disabling EventLogging on enwiki'
14:28 logmsgbot_: demon synchronized php-1.21wmf2/extensions/AbuseFilter/modules/ext.abuseFilter.tools.js 'Fixing JS for AbuseFilter'
14:28 logmsgbot_: demon synchronized php-1.21wmf2/extensions/AbuseFilter/modules/ext.abuseFilter.examine.js 'Fixing JS for AbuseFilter'
14:08 logmsgbot_: demon synchronized php-1.21wmf2/extensions/AbuseFilter/modules 'Updating AbuseFilter to f52f940e'
13:41 logmsgbot_: maxsem synchronized wmf-config/mobile.php 'https://gerrit.wikimedia.org/r/#/c/30070/'
13:07 logmsgbot_: demon synchronized wmf-config/InitialiseSettings.php 'Syncing I85721736'
12:46 paravoid: stopping squid3(!) on sq82, box was reprovisioned with precise but we don't support squid3/squid on precise yet
11:20 logmsgbot_: reedy synchronized wmf-config/
11:03 Reedy: Created WikiLove tables on urwiki
05:42 paravoid: powering off ms-fe1001 & ms-fe1002
02:38 logmsgbot_: kaldari synchronized php-1.21wmf2/extensions/CentralNotice/CentralNotice.db.php 'updating CentralNotice.db.php'
02:32 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Fri Oct 26 02:32:33 UTC 2012
01:37 logmsgbot_: kaldari synchronized php-1.21wmf2/extensions/CentralNotice 'updating CentralNotice'
01:21 logmsgbot_: kaldari synchronized php-1.21wmf2/extensions/CentralNotice/CentralNotice.db.php 'updating CentralNotice.db.php'
00:15 logmsgbot_: olivneh synchronized php-1.21wmf2/extensions/ArticleFeedbackv5/ArticleFeedbackv5.hooks.php
00:14 logmsgbot_: pgehres synchronized php-1.21wmf2/extensions/ContributionTracking/ 'Updating ContributionTracking'
00:13 logmsgbot_: pgehres synchronized php-1.21wmf2/extensions/FundraiserLandingPage/ 'Updating FundraiserLandingPage'
00:04 logmsgbot_: kaldari synchronized php-1.21wmf2/extensions/CentralNotice/modules/ext.centralNotice.bannerController/bannerController.js 'fixing bannerController'

October 25

23:55 logmsgbot_: kaldari Finished syncing Wikimedia installation... :
23:49 cmjohnson1: installing base OS on sq82
23:20 logmsgbot_: kaldari Started syncing Wikimedia installation... :
22:58 logmsgbot_: olivneh synchronized php-1.21wmf2/extensions/E3Experiments
22:40 logmsgbot_: aaron synchronized php-1.21wmf2/extensions/OpenSearchXml/ApiOpenSearchXml.php 'use slave DB for revision queries.'
22:36 logmsgbot_: aaron synchronized php-1.21wmf2/extensions/MWSearch/MWSearch_body.php 'use slave DB for revision queries.'
22:30 logmsgbot_: aaron synchronized php-1.21wmf2/includes/Revision.php 'done'
22:30 logmsgbot_: aaron synchronized php-1.21wmf2/includes/Revision.php 'logging one more thing.'
22:24 logmsgbot_: aaron synchronized php-1.21wmf2/includes/Revision.php 'done'
22:23 logmsgbot_: aaron synchronized php-1.21wmf2/includes/Revision.php 'tweak wfGetCaller param a level'
22:22 logmsgbot_: aaron synchronized php-1.21wmf2/includes/Revision.php
22:21 logmsgbot_: aaron synchronized php-1.21wmf2/includes/Revision.php 'temp query debug logging'
22:18 logmsgbot_: olivneh synchronized php-1.21wmf2/extensions/EventLogging
21:55 logmsgbot_: mlitn synchronized php-1.21wmf2/extensions/AbuseFilter 'desc'
21:55 logmsgbot_: mlitn synchronized php-1.21wmf2/extensions/ArticleFeedbackv5 'desc'
21:55 logmsgbot_: olivneh synchronized php-1.21wmf2/extensions/PostEdit
21:55 logmsgbot_: mlitn synchronized wmf-config 'desc'
21:46 logmsgbot_: mlitn synchronized php-1.21wmf2/extensions/AbuseFilter 'desc'
21:45 logmsgbot_: mlitn synchronized php-1.21wmf2/extensions/ArticleFeedbackv5 'desc'
21:01 logmsgbot_: demon synchronized wmf-config/CommonSettings.php 'Syncing I39c9693a'
21:00 logmsgbot_: demon synchronized wmf-config/wgConf.php 'Syncing I39c9693a'
20:30 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/ 'Dark deploy Wikibase, Diff and ULS'
19:47 logmsgbot_: pgehres synchronized php-1.21wmf2/extensions/ContributionTracking/ 'Updating ContributionTracking to master'
19:44 logmsgbot_: kaldari synchronized wmf-config/CommonSettings.php 'adding new var for CentralNotice'
19:41 logmsgbot_: pgehres synchronized php-1.21wmf2/extensions/LandingCheck/ 'Updating LandingCheck to master'
18:36 logmsgbot_: preilly synchronized php-1.21wmf2/extensions/MobileFrontend 'update after deploy'
18:36 logmsgbot_: preilly synchronized php-1.21wmf1/extensions/MobileFrontend 'update after deploy'
18:31 Ryan_Lane: Put in a live-hack on labsconsole to remove m1.tiny from the list of instance types
18:29 cmjohnson1: installing base OS on srv194
18:17 Ryan_Lane: updated OpenStackManager on labsconsole to master version
18:16 binasher: flushed mobile varnish cache per preilly
18:09 Jeff_Green: restarted gmetad on nickel
17:53 logmsgbot_: preilly synchronized php-1.21wmf2/extensions/MobileFrontend 'update after deploy'
17:52 logmsgbot_: preilly synchronized php-1.21wmf1/extensions/MobileFrontend 'update after deploy'
16:56 paravoid: powercycling copper, load 700
14:57 notpeter: silencing wikivoyage checks in nagios until deployed
14:54 cmjohnson1: shutting down sq82 to replace /dev/sda
14:52 cmjohnson1: shutting down srv194 to replace disk
14:05 mark: Sending all non-European upload traffic to eqiad
12:01 mark: Sending US upload traffic to eqiad
12:01 logmsgbot_: demon synchronized wmf-config/wgConf.php 'Syncing out new prefixes for wikidata/wikivoyage'
10:56 mark: Sending Argentina upload traffic to eqiad
10:37 mark: Sending Brazil upload traffic to eqiad
10:24 mark: Sending Canadian upload traffic to eqiad
10:03 paravoid: powercycling nickel, OOM & unable to login
09:42 mark: Hacked Nagios back up
08:43 mark: Added georecords for wikidata-lb.wikimedia.org and wikivoyage-lb.wikimedia.org, their geomaps containing just a default entry pointing to eqiad
08:33 mark: Added LVS service IPs for wikidata and wikivoyage (pmtpa/eqiad) to DNS
07:12 mark: Stopping backend squid on sq82, sda I/O errors
00:34 paravoid: upgrading oggvideotools to 0.8a on all imagescalers, fixing regression on the lucid->precise upgrade
00:27 binasher: deploying updated squid mobile redirector
00:24 logmsgbot_: tstarling synchronized wmf-config/CommonSettings.php

October 24

23:12 binasher: restarting pdns on ns0
23:09 cmjohnson1: db78 loading base OS
23:06 cmjohnson1: authdns update adding db78.pmtpa.wmnet
22:53 cmjohnson1: authdns changes adding db63-db78 mgmt
21:37 logmsgbot_: aaron synchronized php-1.21wmf2/includes/EditPage.php 'deployed 51eede5af0fe0bb28d7abdf81a04292ba8a4864c'
20:48 logmsgbot_: demon synchronized php-1.21wmf2/includes/Title.php 'Syncing Ic2d3f0b8'
20:44 logmsgbot_: demon synchronized php-1.21wmf2/includes/EditPage.php 'Deploying I4c2055be'
19:59 logmsgbot_: dzahn gracefulled all apaches
18:49 logmsgbot_: catrope synchronized php-1.21wmf2/includes/upload/UploadStash.php 'rm debugging live hacks'
18:46 logmsgbot_: catrope synchronized php-1.21wmf2/includes/upload/UploadStash.php 'Debugging exception'
18:44 logmsgbot_: catrope synchronized php-1.21wmf2/includes/upload/UploadStash.php 'Debugging exception'
18:39 logmsgbot_: catrope synchronized php-1.21wmf2/includes/upload/UploadStash.php 'Debugging exception'
18:17 Reedy: Running sync-common on mw40
18:02 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.21wmf2
17:53 notpeter: upgrading srv223 and srv224 to precise
17:41 logmsgbot_: preilly synchronized php-1.21wmf2/extensions/ZeroRatedMobileAccess 'update post deploy'
17:40 logmsgbot_: preilly synchronized php-1.21wmf1/extensions/ZeroRatedMobileAccess 'update post deploy'
17:38 logmsgbot_: lcarr synchronized wmf-config/throttle.php 'I updated a throttle'
17:26 logmsgbot_: preilly synchronized php-1.21wmf2/extensions/MobileFrontend 'update after deploy'
17:25 logmsgbot_: preilly synchronized php-1.21wmf1/extensions/MobileFrontend 'update after deploy'
17:21 logmsgbot_: lcarr synchronized wmf-config/throttle.php 'I updated a throttle'
17:10 cmjohnson1: db62 loading OS
16:14 cmjohnson1: shutting down db61 for rack relocation
16:03 cmjohnson1: removing h310 controller from db62 and adding original h700 card
14:21 notpeter: upgarding srv221 and srv222 to precise
13:39 cmjohnson1: installing base OS on db62
11:56 paravoid: putting ms-fe2 (upgraded to precise/swift 1.7.4) back into the pool
05:59 logmsgbot_: tstarling synchronized wmf-config/CommonSettings.php
02:48 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Wed Oct 24 02:48:06 UTC 2012
02:27 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Wed Oct 24 02:27:30 UTC 2012
00:50 logmsgbot_: maxsem synchronized php-1.21wmf1/extensions/MobileFrontend
00:26 mutante: flushing varnish mobile cache
00:21 logmsgbot_: maxsem synchronized php-1.21wmf1/extensions/MobileFrontend/
00:18 logmsgbot_: maxsem synchronized php-1.21wmf2/extensions/MobileFrontend/
00:09 logmsgbot_: maxsem Finished syncing Wikimedia installation... : Weekly MobileFrontend and Zero deployment

October 23

23:25 logmsgbot_: maxsem Started syncing Wikimedia installation... : Weekly MobileFrontend and Zero deployment
23:09 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/ExtensionDistributor
23:02 logmsgbot_: reedy synchronized php-1.21wmf2/includes/api/
22:57 logmsgbot_: reedy synchronized php-1.21wmf2/includes/WikiPage.php
22:57 logmsgbot_: reedy synchronized wmf-config/
22:54 logmsgbot_: reedy synchronized php-1.21wmf2/includes/WikiPage.php 'debugging'
22:53 logmsgbot_: reedy synchronized php-1.21wmf2/includes/WikiPage.php 'debugging'
22:13 logmsgbot_: mlitn Finished syncing Wikimedia installation... : updating ArticleFeedbackv5, PageTriage & E3Experiments
22:02 logmsgbot_: reedy synchronized php-1.21wmf2/includes/api/
21:41 logmsgbot_: reedy synchronized php-1.21wmf2/includes/api/
21:39 logmsgbot_: mlitn Started syncing Wikimedia installation... : updating ArticleFeedbackv5, PageTriage & E3Experiments
20:57 Ryan_Lane: restarting gmetad on nickel
20:56 Ryan_Lane: restarting gmetad
20:40 logmsgbot_: reedy synchronized php-1.21wmf2/includes/
20:31 Jeff_Green: dist-upgrade & rebooting payments[1-4]
20:05 logmsgbot_: olivneh synchronized php-1.21wmf2/extensions/PostEdit
20:02 cmjohnson1: fixing db62 raid cnfg
19:10 logmsgbot_: asher synchronized wmf-config/db.php 'returning es10 to service'
18:40 binasher: shutting down es10 for dimm replacement
17:39 RobH: analytics1011 now offline and wiping as well
17:15 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'wgMaxImageArea to 25MP'
17:11 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/PagedTiffHandler
16:36 cmjohnson1: db56 replacing disk 7 (slot6)
16:14 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/PagedTiffHandler/
16:08 logmsgbot_: reedy synchronized php-1.21wmf2/includes/db/Database.php
16:07 logmsgbot_: reedy synchronized php-1.21wmf2/includes/Message.php
16:06 cmjohnson1: db45 replacing disk2 (slot 1)
15:55 RobH: analytics1012-1022 are starting wipe and having network cable pulled
15:47 RobH: analytics1011-1022 will be dying today, byebye c2100
15:31 logmsgbot_: reedy synchronized php-1.21wmf2/includes/EditPage.php
15:21 paravoid: manually patching swift on ms-be3 to comment-out the drop_cache() calls
14:57 logmsgbot_: reedy synchronized php-1.21wmf2/includes/api/ApiParse.php 'Dont parse special pages'
14:50 paravoid: temporarily stopping puppet on brewster
14:49 mark: Inserted new varnish build 3.0.3plus~rc1-wm4 into precise-wikimedia, fixing duplicate header issue
14:49 notpeter: removing srv194 from apaches pool for drive error
14:49 logmsgbot_: reedy synchronized php-1.21wmf2/includes/api/ApiParse.php 'Dont parse special pages'
14:32 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/PostEdit 'Update PostEdit to master'
12:53 mark: Inserted new varnish 3.0.3plus~rc1-wm3 packages into the precise-wikimedia APT repository, with a duplicate content-length header fix on streaming
11:20 paravoid: depooling ms-fe2, in preparation for ubuntu/swift upgrade
06:42 logmsgbot_: krinkle synchronized php-1.21wmf1/resources/mediawiki/mediawiki.searchSuggest.js 'Ibcc56f64'
06:40 logmsgbot_: krinkle synchronized php-1.21wmf2/resources/mediawiki/mediawiki.searchSuggest.js 'Ibcc56f64'
05:17 logmsgbot_: tstarling synchronized php-1.21wmf2/includes/SpecialPage.php
02:55 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Tue Oct 23 02:55:38 UTC 2012
02:29 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Tue Oct 23 02:29:31 UTC 2012

October 22

23:04 logmsgbot_: reedy synchronized php-1.21wmf2/includes/EditPage.php
23:00 LeslieCarr: switching RE mastership on cr2-eqiad
22:43 LeslieCarr: draining cr2-eqiad again for downgrade
22:37 logmsgbot_: asher synchronized wmf-config/db.php 'raising weight of db65'
22:33 LeslieCarr: switching cr2-eqiad routing engine mastership
22:19 logmsgbot_: asher synchronized wmf-config/db.php 'replacing db22 with db65'
22:06 LeslieCarr: undraining cr2-eqiad
22:01 Jeff_Green: stopping puppet on aluminium so fr-tech can local-test a config change
21:48 LeslieCarr: switch routing engine mastership on cr2-eqiad
21:23 LeslieCarr: commit full on cr2-eqiad which will restart all routing processes
21:20 cmjohnson1: replacing disk13 on tridge array
21:19 logmsgbot_: olivneh synchronized php-1.21wmf2/extensions/Vector
21:17 logmsgbot_: olivneh synchronized php-1.21wmf2/extensions/PostEdit
20:49 LeslieCarr: draining traffic from cr2-eqiad in preparation for maintenance and upgrade
20:21 Reedy: Running p[opulateFileArchiveSha1.php on enwiki
19:06 logmsgbot_: reedy synchronized php-1.21wmf2/includes/api/ApiParse.php
18:53 binasher: streaming hotbackup of db1038 to db65, preparing to replace to db22
18:28 logmsgbot_: reedy synchronized php-1.21wmf2/skins/CologneBlue.php
18:16 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf2
16:49 logmsgbot_: py gracefulled all apaches
16:47 Jeff_Green: stopping mysql on db1025 to clone it to db1013
16:02 logmsgbot_: reedy synchronized wmf-config/
16:01 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/ExtensionDistributor
15:48 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/ExtensionDistributor
15:43 notpeter: re-adding srv219 and srv220 to rendering pool
15:04 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/ExtensionDistributor
13:32 notpeter: removing srv219 and srv220 from rendering pool for upgarde to precise
09:35 mark: Built new varnish 3.0.3plus~rc1-wm2 packages and inserted them into the precise-wikimedia APT repository
02:51 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Mon Oct 22 02:51:32 UTC 2012
02:31 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Mon Oct 22 02:30:57 UTC 2012

October 21

22:50 logmsgbot_: tstarling synchronized php-1.21wmf2/bin/ulimit4.sh
22:39 Reedy: populateFilearchiveSha1.php finished on commonswiki
14:18 Reedy: Running mwscript populateFilearchiveSha1.php commonswiki
02:49 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Sun Oct 21 02:49:42 UTC 2012
02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Sun Oct 21 02:28:50 UTC 2012
01:35 paravoid: powercycling srv219, swap-died while trying to get a core dump
01:34 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'wgMaxImageArea to 20MP'
01:33 paravoid: powercycling srv220/srv224, swapdeath

October 20

18:51 Reedy: Ran populateFilearchiveSha1.php on metawiki
18:21 paravoid: powercycling srv221 & srv222, both dead since ~5h ago
17:55 RobH_: updated otrs motd per requst by otrs admins (it referenced a now closed request for comment)
17:42 Reedy: Running foreachwiki cleanupUploadStash.php
03:13 K4-713: updated the payments cluster to 97e1ba74a39376
02:41 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Sat Oct 20 02:41:10 UTC 2012
02:21 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Sat Oct 20 02:21:41 UTC 2012
01:36 K4-713: updated the payments cluster to bb30dab91a7afe862
00:46 mutante: upgrade Bugzilla to 4.0.8 (security patch)

October 19

23:36 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'wgMaxImageArea to 17MP'
23:31 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/SwiftCloudFiles
23:30 LeslieCarr: shutting down sfo-monitor1
23:27 logmsgbot_: reedy synchronized php-1.21wmf2/includes/api/ApiParse.php
23:15 binasher: returning srv194/200 to apache lvs pool
22:50 mutante: authdns-update, renaming travel-guide-lb to wikivoyage-lb
21:10 binasher: pulled srv194 and srv200 from lvs for memc testing
21:02 binasher: upgrading percona-toolkit to 2.1.5
20:18 logmsgbot_: reedy synchronized php-1.21wmf2/includes/api/ApiParse.php
19:33 logmsgbot_: aaron synchronized php-1.21wmf2/includes/filebackend/FileBackendStore.php 'deployed 65e47037f6e9da2e81db2bc1374cdac375a82db5'
19:26 paravoid: powercycling srv223 too, locked up, nothing in serial console
19:25 paravoid: powercycling srv221, locked up, nothing in serial console
16:21 RobH: msw-a2-eqiad back online
15:54 RobH: msw-a2-eqiad is coming down for replacement per RT 3683. all mgmt connectivity (only) will be lost to items in A2-EQIAD
14:43 paravoid: putting ms-fe1 back into the pool
14:14 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commonswiki to 1.21wmf2
13:59 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: metawiki to 1.21wmf2
13:50 cmjohnson1: shutting down msbe1001-1012 to remove ssd's and package for dell return
13:48 logmsgbot_: reedy synchronized php-1.21wmf2/includes/EditPage.php
04:15 logmsgbot_: tstarling synchronized php-1.21wmf2/includes/User.php
02:59 logmsgbot_: tstarling synchronized php-1.21wmf2/extensions/AbuseFilter/AbuseFilter.class.php
02:57 logmsgbot_: tstarling synchronized php-1.21wmf2/includes/User.php
02:47 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Fri Oct 19 02:47:49 UTC 2012
02:27 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Fri Oct 19 02:27:57 UTC 2012
02:27 logmsgbot_: tstarling synchronized php-1.21wmf2/extensions/AbuseFilter/AbuseFilter.class.php
02:23 logmsgbot_: tstarling synchronized php-1.21wmf2/extensions/AbuseFilter/AbuseFilter.class.php
02:10 logmsgbot_: tstarling synchronized php-1.21wmf2/includes/User.php
01:59 logmsgbot_: tstarling synchronized php-1.21wmf2/includes/User.php
01:56 logmsgbot_: tstarling synchronized php-1.21wmf2/extensions/CentralAuth/CentralAuthHooks.php
01:53 K4-713: updated payments cluster to ff8a1e791eaff47313
01:53 logmsgbot_: tstarling synchronized php-1.21wmf2/extensions/CentralAuth/CentralAuthHooks.php
01:50 logmsgbot_: tstarling synchronized php-1.21wmf2/extensions/CentralAuth/CentralAuthHooks.php
01:46 logmsgbot_: tstarling synchronized php-1.21wmf2/extensions/CentralAuth/CentralAuthHooks.php
01:45 logmsgbot_: kaldari Finished syncing Wikimedia installation... :
01:44 logmsgbot_: tstarling synchronized php-1.21wmf2/extensions/CentralAuth/CentralAuthHooks.php
01:20 logmsgbot_: kaldari Started syncing Wikimedia installation... :
00:26 logmsgbot_: olivneh synchronized wmf-config/InitialiseSettings.php
00:26 logmsgbot_: olivneh synchronized php-1.21wmf1/extensions/PostEdit
00:22 logmsgbot_: reedy synchronized wmf-config/
00:21 paravoid: disabling puppet on (the already depooled) ms-fe1
00:09 logmsgbot_: olivneh Finished syncing Wikimedia installation... :
00:04 logmsgbot_: olivneh Started syncing Wikimedia installation... :
00:03 logmsgbot_: olivneh synchronized php-1.21wmf1/extensions/PostEdit

October 18

23:42 logmsgbot_: reedy synchronized wmf-config/
23:31 logmsgbot_: spage Finished syncing Wikimedia installation... : updating 1.21wmf1 & wmf2 with E3Experiments (and 1.21wmf1 with PostEdit)
23:26 Reedy: populateFilearchiveSha1.php ran for all wikis on 1.21wmf2
23:16 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
23:14 logmsgbot_: spage Started syncing Wikimedia installation... : updating 1.21wmf1 & wmf2 with E3Experiments (and 1.21wmf1 with PostEdit)
23:13 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikimedia.dblist to 1.21wmf2
23:11 Reedy: Added fa_sha1 to liquidthreads_labswikimedia
23:10 Reedy: Ran populateFilearchiveSha1.php on private, wiktionary, wikiversity and wikinews wikis
23:07 logmsgbot_: tstarling synchronized php-1.21wmf2/includes/User.php 'debugging hack'
23:04 Reedy: Added fa_sha1 to fdcwiki
23:00 logmsgbot_: tstarling synchronized php-1.21wmf2/includes/User.php 'debugging hack'
22:32 logmsgbot_: aaron synchronized php-1.21wmf2/includes/filebackend/FileBackendStore.php 'deployed 15520f489b9adab8bcfd84e6b6a5d48280a85556'
22:05 logmsgbot_: reedy synchronized php-1.21wmf2/includes/api/ApiQueryRevisions.php
21:26 logmsgbot_: aaron synchronized php-1.21wmf2/includes/filebackend/FileBackendStore.php 'deployed ff03172cccad1ce7017c7dc44e508317cc73975f'
21:11 logmsgbot_: kaldari Finished syncing Wikimedia installation... :
21:01 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php '$wgMaxImageArea to 15'
20:18 logmsgbot_: kaldari Started syncing Wikimedia installation... :
20:04 Reedy: Running scap-recompile on all mediawiki-installation
19:28 Jeff_Green: adjusted a few hostnames for silverpop
19:21 notpeter: created ulsfo rt queue
19:19 logmsgbot_: reedy synchronized php-1.21wmf2/includes/EditPage.php
18:53 logmsgbot_: reedy synchronized php-1.21wmf2/includes/
17:54 Jeff_Green: adjusted banner log filter to include non-fundraising banners
17:01 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Rest of planned wikis to 1.21wmf2 minus commons and meta
16:53 K4-713: synchronized payments cluster to efcecc41c6f3c26
15:19 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary and wikiversity to 1.21wmf2
15:17 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
15:15 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews to 1.21wmf2
15:07 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: private wikis to 1.21wmf2
14:19 notpeter: re-adding srv190 to rendering pool
02:50 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Thu Oct 18 02:50:46 UTC 2012
02:26 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Thu Oct 18 02:26:24 UTC 2012

October 17

22:31 logmsgbot_: reedy synchronized php-1.21wmf2/includes/specials/
20:58 mutante: nuking wmfsocial mailing list
20:48 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: revert test
20:47 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test
20:42 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Fix tetwiki
20:40 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.21wmf2
20:31 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki testwiki and test2wiki back to 1.21wmf2
20:23 ^demon: restarted gerrit service, was missing documentation from UI
19:33 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: private and special wikis back to 1.21wmf1
19:19 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Increase $wgMaxImageArea to 2.5e7 on testwiki'
19:14 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Increase $wgMaxImageArea to 1.4e7'
18:54 binasher: fixed pc1 innodb_free_space logging on pc1 via "GRANT SELECT ON `parsercache`.`pc000` TO 'dbstats'@'localhost';" - (it had only been granted access to the objectcache table which no longer exists)
18:53 logmsgbot_: olivneh synchronized php-1.21wmf1/extensions/E3Experiments/lib/event/eventlog.js
18:44 cmjohnson1: racking and setting up replacements for ms-be6/7
18:22 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: metawiki back to 1.21wmf1
18:18 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Closed, special and private to 1.21wmf2
18:08 logmsgbot_: reedy synchronized php-1.21wmf2/includes/api/ApiEditPage.php
18:05 LeslieCarr: killed and restarted all varnish processes on cp1041
18:00 LeslieCarr: restarted varnish-frontend on cp1041
17:53 LeslieCarr: restarting varnish on cp1041
17:24 mark: Installed experimental Varnish package with range support in streaming mode on cp1021
16:50 logmsgbot_: midom synchronized wmf-config/InitialiseSettings.php
16:21 LeslieCarr: restarting varnish on cp1042 , gone into swapdeath spiral
16:15 logmsgbot_: reedy synchronized php-1.21wmf2/cache/l10n/
16:01 logmsgbot_: reedy synchronized php-1.21wmf1/cache/l10n/
15:59 notpeter: increasing weight on mw17-mw59 by 33.3%
15:46 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuilding localisation cache for educationprogram
15:36 Jeff_Green: switched donate.wikimedia.org to geodns
15:31 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/EducationProgram
15:23 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/EducationProgram/
14:34 logmsgbot_: midom synchronized wmf-config/InitialiseSettings.php
06:09 apergos: restarted lucene search on search1015
04:10 Tim: restarted swift frontends
03:41 Tim: seems to have worked, also truncating pc161-174 to be on the safe side
03:38 Tim: on pc1: truncated tables pc158, pc159, pc160 due to flood of "table full" errors in dberror.log indicating innodb space exhaustion
02:49 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Wed Oct 17 02:49:53 UTC 2012
02:26 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Wed Oct 17 02:26:34 UTC 2012

October 16

23:41 logmsgbot_: maxsem synchronized php-1.21wmf1/extensions/MobileFrontend/ 'https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-10-15%2616'
23:33 logmsgbot_: maxsem synchronized php-1.21wmf2/extensions/MobileFrontend/ 'https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-10-15%2616'
22:50 binasher: redeployed squid refresh_pattern no-cache tweaks
22:46 MaxSem: Running mw-update-l10n and friends to catch up newest MobileFrontend messages
21:45 paravoid: deploying squid configs, reverting a change that was too aggressive on caching content and broke https-by-default wikis
21:40 logmsgbot_: reedy synchronized php-1.21wmf2/includes/specials/SpecialUndelete.php
20:08 cmjohnson1: storage3 down for troubleshooting
19:57 Reedy: Ran namespaceDupes.php on azwikibooks
19:20 binasher: deployed squid refresh_pattern tweaks for testing
19:19 Jeff_Green: adding dns and geodns entries for donate-lb
18:58 cmjohnson1: shutting down db67 to add h800 card
14:57 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'touch'
14:56 logmsgbot_: hashar synchronized wmf-config/throttle.php '(bug 41069) Delhi 2012-10-17 workshop throttle rule'
14:56 logmsgbot_: hashar synchronized wmf-config/throttle.php '(bug 41069) Delhi 2012-10-17 workshop throttle rule'
12:31 logmsgbot_: reedy synchronized php-1.21wmf2/includes/
09:06 logmsgbot_: nikerabbit synchronized wmf-config/InitialiseSettings.php '(bug 41015) Enable Narayam on hi.wiktionary'
02:53 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Tue Oct 16 02:53:28 UTC 2012
02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Tue Oct 16 02:28:10 UTC 2012
00:57 RoanKattouw: chmod -R g+w /home/wikipedia/common/php-1.21wmf2/.git
00:48 RoanKattouw: I mean 002
00:48 RoanKattouw: Created ~spage/.bashrc with 'umask 0002'
00:03 binasher: completed patch-job_token.sql migrations
00:00 binasher: running patch-job_token.sql migration against all wikis

October 15

23:04 mutante: power cycling cp1040
21:55 logmsgbot_: catrope synchronized php-1.21wmf2/extensions/VisualEditor/ 'f25aaf5bc4449ea8c1ac749a96487485b5f0c6b3'
21:54 logmsgbot_: catrope synchronized php-1.21wmf2/includes/resourceloader/ResourceLoaderFileModule.php '1f761e3bf349df8d48592f71ed4afff63471c6b4'
21:44 mwalker: resuming purge of CentralNotice cache (now working) with rate limit of 50hps. estimated completion in 3hrs.
21:29 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php '(bug 41016) Temporarily reenable wgUseNPPatrol on fi.wikipedia'
20:53 logmsgbot_: reedy synchronized php-1.21wmf2/resources/
20:53 logmsgbot_: reedy synchronized php-1.21wmf1/resources/
20:52 cmjohnson1: shutting down es10 for per rt3724
20:49 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/ArticleFeedback
20:49 mutante: flushing mobile varnish cache
20:43 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/ArticleFeedback
20:37 logmsgbot_: reedy synchronized php-1.21wmf1/includes/resourceloader/ResourceLoaderFileModule.php
20:37 mwalker: rate limiting for CentralNotice purge is working too well... ie: script is not doing anything -- debugging
20:35 logmsgbot_: awjrichards synchronized php-1.21wmf1/extensions/MobileFrontend/ 'touch files'
20:21 mwalker: Attempting to purge out another 500k pages with CentralNotice skin problem (this time not crashing the cluster by filtering out media files and rate limiting to no more than 50hps). Estimated run time is 3 hours.
20:20 logmsgbot_: awjrichards synchronized php-1.21wmf1/extensions/MobileFrontend/ 'touch files'
20:10 logmsgbot_: asher synchronized wmf-config/db.php 'pulling es10, third crash in three weeks'
20:07 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection/ 'Revert back to 9f37005e78a068821c45d9d313394c6728f55b4a'
20:05 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/Collection/ 'Revert back to 9f37005e78a068821c45d9d313394c6728f55b4a'
19:57 cmjohnson1: powering down srv266 troubleshooting per rt2896
19:04 binasher: restarted udp2log on oxygen with correct --recv-queue
18:58 logmsgbot_: reedy synchronized wmf-config/
18:55 logmsgbot_: reedy synchronized php-1.21wmf1/includes/resourceloader/ResourceLoaderFileModule.php
18:49 RoanKattouw: Created fluorine:/a/mw-log/udp2log/resourceloader.log chmod 644 and owned by udp2log:udp2log as root per Reedy's request
18:19 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/CodeReview
18:11 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.21wmf2
18:07 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/MobileFrontend
17:32 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'wgDebugLogGroups for resourceloader'
17:30 logmsgbot_: reedy synchronized php-1.21wmf1/includes/resourceloader/ResourceLoaderFileModule.php
16:28 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/MobileFrontend/
16:26 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/MobileFrontend/
16:16 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Re-enable education program on enwiki'
16:06 logmsgbot_: reedy Finished syncing Wikimedia installation... : Build localisation cache for 1.21wmf2
15:43 ottomata: stopping puppet on oxygen, looking for cause of recent packet loss
15:19 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/MobileFrontend
15:17 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/MobileFrontend
15:12 logmsgbot_: reedy Started syncing Wikimedia installation... : Build localisation cache for 1.21wmf2
14:56 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.21wmf2
14:55 logmsgbot_: reedy synchronized wmf-config/
14:51 logmsgbot_: reedy synchronized php-1.21wmf2/ 'Initial sync'
13:35 Reedy: Killing php-1.20wmf11 l10n cache from apaches
13:34 Reedy: Killing php-1.20wmf11 l10n cache from apaches
13:31 Reedy: Pointing php to php-1.21wmf1
13:29 logmsgbot_: reedy synchronized wmf-config/throttle.php 'Updated from gitrepo'
13:28 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Remove extdist conditional for mount path'
11:49 mark: Deployed range_offset_limit 64 MB change (from 0) to all upload backend squids
11:24 paravoid: restarting pdns @ ns2
11:16 paravoid: switching European bits back to esams
09:48 paravoid: restarting all imagescaler apaches, did not recover after swift outage
09:24 paravoid: live-hacking swift on ms-be10, ms-be12, ms-be1 to remove fadvise calls
07:53 Tim: increasing worker count to 100 on all swift backends, via puppet
07:37 Tim: ms-be11 showed an immediate improvement in bandwidth out, but wchan still indicates that 48 is not enough, increasing to 100
07:33 Tim: experimentally doubling the worker count on ms-be11 since wchan indicates that the worker pool is exhausted
05:49 apergos: ms-be12 reporting errors on disk /dev/sdi, I've rt'ed it an unmounted it in the meantime
04:34 Tim: on ms-fe2: restarted swift-proxy due to mem leak
04:28 Tim: on ms-fe3: restarting swift-proxy due to swap
04:26 LeslieCarr: restarted swift proxy on ms-fe4 - it was in swap doom
02:25 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Mon Oct 15 02:25:12 UTC 2012

October 14

21:57 logmsgbot_: reedy synchronized php-1.21wmf1/includes/specials/SpecialUserlogin.php
14:26 logmsgbot_: reedy synchronized php-1.21wmf1/cache/l10n/
14:22 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Sun Oct 14 14:22:54 UTC 2012
02:00 logmsgbot_: LocalisationUpdate failed: git pull of extensions failed

October 13

21:39 logmsgbot_: preilly Finished syncing Wikimedia installation... : update zero rated mobile access
21:18 logmsgbot_: preilly Started syncing Wikimedia installation... : update zero rated mobile access
21:06 logmsgbot_: preilly synchronized php-1.21wmf1/extensions/MobileFrontend 'update for landing page'
18:51 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to master'
18:41 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to original again..'
18:38 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to f980f2380b830e712edfdbce065784d1bcba82fd'
18:36 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to 4f898444ce5d9c2c9a5491f2303db15a3143a5c5'
18:34 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to 72b7d4d645489dd092605a2183a944fac33cb40b'
18:26 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to original'
18:25 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to 8415a4564bf429b76e113368a3e5e1c19fb84a47'
18:23 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to 72b7d4d645489dd092605a2183a944fac33cb40b'
18:21 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to 9f37005e78a068821c45d9d313394c6728f55b4a'
16:34 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection
16:27 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection
13:17 logmsgbot_: reedy synchronized wmf-config/throttle.php
02:00 logmsgbot_: LocalisationUpdate failed: git pull of extensions failed

October 12

23:51 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'enabling redis debug log'
23:40 LeslieCarr: moving traffic back to esams via authdns-scenario
23:40 LeslieCarr: reenabled nagios alerts
23:20 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'enabling session multiwrites to redis'
23:16 LeslieCarr: rebooting esams core switch stack, expect reachability problems to esams
23:01 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'disabling session multiwrites to redis'
22:36 LeslieCarr: switching europe traffic to pmtpa via authdns-scenario
22:36 LeslieCarr: muted nagios
20:20 AaronSchulz: Confirmed all swift -> netapp sync scripts finished (s4 one kept dying to random http errors)
19:58 logmsgbot_: reedy synchronized wmf-config/
19:01 cmjohnson1: powering down search32 to replace cpu1
18:33 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'AFT (non v5) doesn't use clicktracking now'
18:28 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/ArticleFeedback/
18:19 cmjohnson1: troubleshooting db42 w/IBM tech
17:46 cmjohnson1: pc3 shutting down to replace hdd's with ssd's rt3703
17:29 cmjohnson1: pc2 shutting down to replaced hdd w/ ssd rt3703
16:27 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Fix extdist mount related warnings'
16:14 cmjohnson1: running installer on storage3
15:54 paravoid: moving bits from esams to pmtpa (authdns normal scenario)
15:52 cmjohnson1: storage 3 replacing disk 12 and rebuilding raid
15:32 mark: Disabled csw1-esams:e8/2
14:37 mark: Rebooting cr1-esams
14:17 mark: Upgrading JunOS on cr1-esams to 11.4R5.5, preparing for reboot
13:59 mark: Activated aggregate generation on cr1-esams
13:49 mark: Moved AS38930 transit from csw1-esams:e8/4 to cr1-esams:xe-1/3/0
13:24 cmjohnson1: storage3 powering down
12:27 mark: Upgraded new stack member asw-oe11-esams to junos 11.4R5.5 and rebooted it
12:16 mark: Added new EX4200-48T asw-oe11-esams to csw2-esams stack
11:08 mark: Disabled csw2-oe12-esams:vcp-1
02:01 logmsgbot_: LocalisationUpdate failed: git pull of extensions failed
01:21 mutante: moving ancient copy of bugzilla out of /h/w/htdocs (to /root/backup on fenari temp. just in case)
01:15 mutante: deleting old document root dirs on fenari. /home/wikipedia/htdocs/noc and noc.old (use /h/w/common/docroot/noc and mediawiki-config repo)
01:08 mutante: restarting Apache on fenari for noc.wm change
00:42 Jeff_Green: switched payments back to pmtpa and increased TTL to 5 min
00:37 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'And back to master'
00:31 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'Revert due to error'
00:29 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'Update collection to master'
00:09 cmjohnson1: storage3 being reimaged

October 11

23:19 mutante: manually copy index.html for noc from /common/docroot/noc to /h/w/htdocs/noc/
23:15 logmsgbot_: dzahn synchronized docroot/noc/index.html
23:13 logmsgbot_: dzahn synchronized docroot/noc/index.html
23:08 logmsgbot_: olivneh synchronized php-1.21wmf1/extensions/E3Experiments
22:16 logmsgbot_: olivneh synchronized php-1.21wmf1/extensions/E3Experiments
21:58 logmsgbot_: bsitu Finished syncing Wikimedia installation... : Update ArticleFeedbackv5, MoodBar, PageTriage to master
21:36 logmsgbot_: bsitu Started syncing Wikimedia installation... : Update ArticleFeedbackv5, MoodBar, PageTriage to master
20:48 Jeff_Green: flipping payments.wikimedia.org to eqiad cluster to live-test
20:12 mutante: rebooting analytics1007
19:02 notpeter: stopping puppet on brewster
18:50 RobH: i need to pull virt1001-1003 out of nagios later, as they are now renamed
16:52 notpeter: restarting pdns on ns0
16:50 logmsgbot_: preilly synchronized php-1.21wmf1/extensions/MobileFrontend 'update for landing page'
16:50 notpeter: dist-upgrade and reboot on oxygen
16:49 K4-713: updated the payments cluster to 4d0223ebb48f9b
16:39 RobH: authdns update for pc1001-1003 mgmt ip
15:35 RobH: working on analytics1007 per rt 3582
14:55 RobH: shutting down db1012 to remove evaluation ssd card
14:14 paravoid: apt: including php-parsekit, python-async, python-git, python-gitdb, python-smmap backports for precise
08:59 apergos: restarted swift proxy on ms-fe3 and ms-fe2 with some delay between. note I had to shoot stale processes from Sept on all these boxes as well
08:50 apergos: restarted ms-fe4 swift-proxy
04:13 mutante: activated W3 Total Cache performance plugin in Wordpress (blog.wikimedia.org)
04:12 mutante: marmontel (blog): fixed some file permissions in ./blog/wp-content for www-data the wordpress caching plugin needs to be re-activated
02:27 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Thu Oct 11 02:27:51 UTC 2012
01:55 LeslieCarr: moved traffic back to normal with authdns-scenario
01:48 LeslieCarr: reenabling nagios notifications
01:22 LeslieCarr: upgrading csw2-esams member 1
01:22 LeslieCarr: upgrading csw2-esams member 0
01:13 binasher: disabled nagios notifications
01:04 LeslieCarr: upgrading csw2-esams member 2
01:02 LeslieCarr: upgrading csw2-esams member 3
00:44 binasher: restarted pdns on ns2
00:44 LeslieCarr: restarting dns on ns2
00:33 LeslieCarr: draining esams via authdns-scenarios
00:11 logmsgbot_: reedy synchronized php-1.21wmf1/includes/api/ApiUpload.php

October 10

23:02 RobH: db1008.mgmt returned to service
23:02 RobH: msw-a2-eqiad has a bad set of ports, needs replacement per rt 3683
22:46 RobH: reboot seems to have restored system (is posting) but drac still not responding, possible bad cable, will test shortly
22:40 RobH: db1008 being cold reset (removing power cords)
22:05 mutante: apt-get dist-upgrade on marmontel (blog)
21:20 LeslieCarr: undraining esams via authdns-scenario normal
20:43 LeslieCarr: draining esams via authdns-scenario for upgrade
20:20 Jeff_Green: returning payments to pmtpa
20:14 Jeff_Green: flipped payments.wikimedia.org to the new eqiad cluster, live testing
20:06 logmsgbot_: awjrichards synchronized php-1.21wmf1/extensions/MobileFrontend/javascripts/common/mf-application.js 'touchfile'
18:46 cmjohnson1: storage3 power cycling
18:24 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Rest of wikipedia projects to 1.21wmf1
17:58 logmsgbot_: reedy synchronized php-1.20wmf12/cache/interwiki.cdb 'Update interwiki cache'
17:58 logmsgbot_: reedy synchronized php-1.21wmf1/cache/interwiki.cdb 'Update interwiki cache'
17:15 notpeter: created indices for fdcwiki on searchidx2 and searchidx1001
17:13 logmsgbot_: reedy synchronized docroot/fdc/
16:57 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Add meta as import source for fdcwiki'
16:43 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Set wgServer for fdcwiki'
15:24 logmsgbot_: reedy ran sync-common-all
14:48 logmsgbot_: reedy synchronized s3.dblist
14:39 logmsgbot_: reedy synchronized wmf-config/
14:37 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
12:40 mark: Throttled SnapMirror originals transfer to 75 MB/s
12:20 mark: Increased SnapMirror TCP window size on nas1-a and nas1001-a for the originals snapmirror relationship, and restarted the initial transfer at the last checkpoint
06:28 apergos: restarted lucene search on search1016
03:01 binasher: completed fa_sha1 migration
02:41 logmsgbot_: LocalisationUpdate completed (1.20wmf12) at Wed Oct 10 02:41:09 UTC 2012
02:20 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Wed Oct 10 02:20:42 UTC 2012
00:02 binasher: started filearchive.fa_sha1 migration, starting with s1

October 9

23:34 logmsgbot_: olivneh synchronized php-1.21wmf1/extensions/E3Experiments
23:32 logmsgbot_: tstarling synchronized php-1.21wmf1/includes/parser/ParserCache.php
23:31 logmsgbot_: tstarling synchronized php-1.20wmf12/includes/parser/ParserCache.php
23:30 logmsgbot_: reedy synchronized php-1.20wmf12/resources/startup.js 'touch'
23:30 logmsgbot_: reedy synchronized php-1.20wmf12/resources/jquery/jquery.js '1.8.2'
23:28 logmsgbot_: tstarling synchronized wmf-config/InitialiseSettings.php
23:20 mutante: flushing mobile varnish cache
23:18 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Add outreachwiki to import sources of enwiki'
23:17 logmsgbot_: awjrichards synchronized php-1.21wmf1/extensions/MobileFrontend/
23:11 logmsgbot_: awjrichards synchronized wmf-config/InitialiseSettings.php 'Enable sticky cookie fix on all wikis; Update logo asset path for mobile footer'
23:06 logmsgbot_: dzahn synchronized docroot/fdc/w
22:51 logmsgbot_: dzahn gracefulled all apaches
22:50 logmsgbot_: dzahn synchronized docroot/fdc/index.php
22:41 logmsgbot_: dzahn synchronized /docroot/fdc
22:38 logmsgbot_: reedy synchronized php-1.21wmf1/resources/startup.js 'touch'
22:26 logmsgbot_: reedy synchronized php-1.21wmf1/resources/jquery/jquery.js 'jQuery 1.8.2'
22:21 mutante: creating new doc root for fdc wiki /usr/local/apache/common/docroot/fdc
22:10 notpeter: readded mw61 to bits backends
21:53 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection
21:29 cmjohnson1: swapping disk 12 out on storage 3
21:07 LeslieCarr: restarting apache2 on srv220
20:31 mutante: authdns-update on dobson, adding fdc.wikimedia.org DNS entry
19:36 Jeff_Green: added a couple temporary filters to oxygen for fr-tech
17:28 notpeter: reimaging mw61 with precise for bits test
16:29 Aaron|home: Set abbrvThreshold to 160 for thumbnails
16:27 logmsgbot_: aaron synchronized wmf-config/filebackend.php 'Set abbrvThreshold to 160 for thumbnails.'
16:22 logmsgbot_: aaron synchronized wmf-config/filebackend.php 'Set abbrvThreshold to 180 for thumbnails.'
16:18 logmsgbot_: aaron synchronized wmf-config/filebackend.php 'Set abbrvThreshold to 200 for thumbnails.'
13:31 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Pre-emptively set $wgContentHandlerUseDB = false;'
11:35 mark: Setup sync SnapMirror relationship from nas1-a:originals to nas1001-a:originals and started transfer
11:29 mark: Increased maxfiles of nas1001-a:originals to 63753378
11:27 mark: Created 36 drive media1 raid-dp aggregate on nas1001-a, hosting new 40T volume 'originals'
10:59 paravoid: rolling restart of swift proxies (ms-fe2/3/4) because of the memory leaks
09:32 logmsgbot_: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Bug 40366 - Narayam'
09:25 logmsgbot_: nikerabbit synchronized php-1.21wmf1/extensions/Translate/api/ApiGroupReview.php
02:50 logmsgbot_: LocalisationUpdate completed (1.20wmf12) at Tue Oct 9 02:50:29 UTC 2012
02:25 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Tue Oct 9 02:25:49 UTC 2012

October 8

20:47 Aaron|home: Started syncFileBackend.php (swift -> nas) on screens on hume (s4)
20:41 logmsgbot_: maxsem synchronized php-1.21wmf1/extensions/MobileFrontend/MobileFrontend.php 'Live hack to unbreak prop=extracts on 1.21wmf1'
20:36 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/ConfirmEdit/FancyCaptcha.class.php 'Revert cache key change live hack'
20:02 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Set wgOldChangeTagsIndex to false for all wikis running with new indexes'
19:22 Aaron|home: Started syncFileBackend.php (swift -> nas) on screens on hume (s1-s3,s5-s7)
19:13 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'wgOldChangeTagsIndex to false for outreachwiki'
19:07 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
19:05 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
18:44 Reedy: Updated liquidthreads_labswikimedia CategoryLinks schema with patch-categorylinks-better-collation.sql
18:36 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf1
16:43 logmsgbot_: reedy synchronized wmf-config/
11:49 apergos: manually cleaned up /tmp on mw49, forced a puppet run too (isn't there a cron job to clean up /tmp?), stopped job runner afterwards since it had been stopped already by someone
11:31 Reedy: Manually running sync-common on mw49
11:14 logmsgbot_: ariel synchronized wmf-config 'move from ms7 upload6 to nas1 upload7'
02:52 logmsgbot_: LocalisationUpdate completed (1.20wmf12) at Mon Oct 8 02:52:16 UTC 2012
02:27 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Mon Oct 8 02:27:45 UTC 2012

October 7

18:09 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'move captchadir to upload7'
10:11 logmsgbot_: maxsem synchronized wmf-config 'Raise creation throttle for India event even more'
02:48 logmsgbot_: LocalisationUpdate completed (1.20wmf12) at Sun Oct 7 02:47:58 UTC 2012
02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Sun Oct 7 02:28:29 UTC 2012

October 6

15:56 logmsgbot_: lcarr synchronized wmf-config/throttle.php 'I updated a throttle'
14:14 logmsgbot_: lcarr synchronized wmf-config/throttle.php 'I updated a throttle'
03:11 logmsgbot_: LocalisationUpdate completed (1.20wmf12) at Sat Oct 6 03:11:10 UTC 2012
02:39 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Sat Oct 6 02:39:31 UTC 2012
02:17 Reedy: Manually re-running localisation update
02:02 logmsgbot_: LocalisationUpdate failed: git pull of extensions failed

October 5

20:44 pp-pdf1: updated mwlib.zim to 0.3.0
20:02 notpeter: removing srv190 from imagescaling pool due to misbuilt pacakge
18:21 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/ProofreadPage
17:22 Reedy: Live hacked 1.21wmf1 ConfirmEdit/FancyCaptcha.class.php cache key for directory list. Needs reverting after 24 hours or so
17:17 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/ConfirmEdit/FancyCaptcha.class.php 'New memcached key'
17:06 logmsgbot_: reedy synchronized wmf-config/
14:00 notpeter: restarting puppet on brewster
13:54 mark: Rollback of upload-lb.eqiad; sending traffic back to pmtpa
13:48 notpeter: stopping puppet on brewster
13:42 logmsgbot_: ariel synchronized wmf-config 'rollback ms7 to nas1 upload6 to upload7 migration'
12:17 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'wgCacheEpoch to 20120908000000'
12:04 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Update wgCacheEpoch and wgThumbnailEpoch to the start of 2012 (+ 1 year)'
12:04 apergos: started one last rsync op upload/ from ms7 to nas1 with --update
11:46 mark: Moved /mnt/upload7/upload/* to /mnt/upload7, removed upload /mnt/upload7/upload dir
11:42 logmsgbot_: ariel synchronized wmf-config 'migrate media etc from ms7 upload6 to nas1 upload7'
11:41 apergos: running sync-dir wmf-config to migrate media etc from ms7 upload6 to nas1 upload7
02:46 logmsgbot_: LocalisationUpdate completed (1.20wmf12) at Fri Oct 5 02:45:59 UTC 2012
02:27 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Fri Oct 5 02:27:21 UTC 2012
01:03 mutante: updating a lot of mailing list descriptions per BZ:37537 / thehelpfulone

October 4

23:59 mutante: manually fixed brion's ssh key on fenari. session opened for user brion. welcome back :)
22:06 mutante: powercycling downed cp1044
21:40 logmsgbot_: olivneh synchronized php-1.20wmf12/extensions/E3Experiments
21:28 logmsgbot_: lcarr synchronized wmf-config/throttle.php 'I updated a throttle'
20:41 logmsgbot_: olivneh synchronized php-1.20wmf12/extensions/ClickTracking/modules/jquery.clickTracking.js
20:18 logmsgbot_: olivneh synchronized php-1.20wmf12/extensions/E3Experiments
18:09 logmsgbot_: catrope synchronized php-1.21wmf1/languages/messages/MessagesEn.php 'Ib8746ece3a34b5e577dc08596b0ff7b1b96b5d73'
18:08 logmsgbot_: catrope synchronized php-1.21wmf1/resources/mediawiki/mediawiki.feedback.js 'Ib8746ece3a34b5e577dc08596b0ff7b1b96b5d73'
16:16 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
15:30 logmsgbot_: reedy synchronized wmf-config/
14:02 Reedy: Manually ran sync-common on srv247
13:34 notpeter: removing srv191 from bits pool for test upgrade to precise
10:39 Tim: gave myself an account on sanger's exim so that I can send outbound mail through it
05:24 apergos: restarted lucene search on search1016 again
02:49 logmsgbot_: LocalisationUpdate completed (1.20wmf12) at Thu Oct 4 02:49:23 UTC 2012
02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Thu Oct 4 02:28:12 UTC 2012
00:14 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection/Collection.body.php

October 3

22:47 mutante: reinstalling analytics1007
21:38 mutante: deleting education-ar mailman list per bug 40749
20:55 logmsgbot_: dzahn synchronized wmf-config/throttle.php 'IP cap lift for Grace Hopper event, bug 40669'
20:18 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else non wikipedia to 1.21wmf1
20:15 mutante: powercycling analytics1007
20:15 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikimedia and wiktionary to 1.21wmf1
20:13 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks and wikinews to 1.21wmf1
20:12 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity and wikiquote to 1.21wmf1
20:09 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special, private and closed to 1.21wmf1
19:33 logmsgbot_: preilly synchronized php-1.21wmf1/extensions/MobileFrontend 'update for landing page'
19:32 logmsgbot_: preilly synchronized php-1.20wmf12/extensions/MobileFrontend 'update for landing page'
19:29 logmsgbot_: preilly synchronized php-1.21wmf1/extensions/ZeroRatedMobileAccess 'update for landing page'
19:28 logmsgbot_: preilly synchronized php-1.20wmf12/extensions/ZeroRatedMobileAccess 'update for landing page'
15:43 Reedy: Close strategywiki, zawiktionary, zh_min_nanwikibooks
15:41 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'WikiLove on itwikiquote'
15:36 Reedy: Created wikilove tables to itwikiquote
15:18 Jeff_Green: test-deploying SPF DNS records
09:49 mark: Doubled number of inodes on nas1-a:originals from 31876689 to 63753378
08:38 apergos: stopped ms7 to ms8 replication
08:18 apergos: rebooting ms8 to unstick deadlocked zfs recv and zfs list processes
07:20 Tim: upgrading luasandbox on all apaches to version 1.1-1
06:28 apergos: restarted lucene search on search1016
05:22 logmsgbot_: tstarling synchronized php-1.21wmf1/cache/trusted-xff.cdb
05:19 logmsgbot_: tstarling synchronized php-1.20wmf12/cache/trusted-xff.cdb
03:11 logmsgbot_: tstarling synchronized wmf-config/db.php 'es10 back up'
03:03 Tim: on es10: started mysqld
02:51 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Wed Oct 3 02:51:35 UTC 2012
02:45 Tim: es10 not responding to ssh, rebooting
02:43 logmsgbot_: tstarling synchronized wmf-config/db.php
02:31 logmsgbot_: LocalisationUpdate completed (1.20wmf12) at Wed Oct 3 02:31:56 UTC 2012
02:09 logmsgbot_: tstarling synchronized wmf-config/InitialiseSettings.php 'API log'
01:35 logmsgbot_: krinkle synchronized live-1.5
01:29 logmsgbot_: tstarling synchronized live-1.5
01:25 Tim: on srv247, srv270, srv271, srv286, mw69: killing more stuck apt-get processes, will run apt-get upgrade
01:19 Tim: on srv233: killed frozen apt-get, running since August 17, running apt-get upgrade
01:17 logmsgbot_: krinkle Finished syncing Wikimedia installation... : Icd721011b40bb8d2c20aefa8b359a3e45413a07f
01:09 Tim: upgrading wikimedia-task-appserver on all apaches
01:06 Tim: on brewster: updated wikimedia-task-appserver
00:42 mutante: moved logmsgbot to -operations channel
00:34 logmsgbot: krinkle Started syncing Wikimedia installation... : Icd721011b40bb8d2c20aefa8b359a3e45413a07f
00:25 Reedy: Restarted Jenkins

October 2

23:45 logmsgbot: olivneh synchronized php-1.20wmf12/extensions/E3Experiments/experiments/openTask.js 'Updating event format for OpenTask funnel'
23:44 logmsgbot: reedy synchronized php-1.21wmf1/extensions/Collection/
23:19 Ryan_Lane: deploying Damianz's OpenStackManager change for adding/removing bastion users on shell access on labsconsole
16:17 logmsgbot: asher synchronized wmf-config/db.php 'adding db64 to s3 in place of db25'
16:02 mark: Reclaimed space on brewster:/ - restarted squdi
15:19 mark: Sending ALL upload traffic to upload-lb.eqiad (Varnish)
14:57 mark: Sending traffic from India, Korea, China, Singapore, Thailand, Vietnam to upload-lb.eqiad (Varnish)
14:26 mark: Sending Mexican upload traffic to upload-lb.eqiad (Varnish)
14:12 mark: Sending Brazil upload traffic to upload-lb.eqiad (Varnish)
14:10 mark: Restarted pdns on linne
14:03 mark: Sending Japanese upload traffic to upload-lb.eqiad (Varnish)
13:57 notpeter: rebooting srv194-199 and mw55-59 to pick up kernel upgrades, and then re-adding them to apaches pool
13:10 notpeter: rebooting mw1-16 to pick up kernel upgrades before re-enabling them as (precise) jobrunners
13:05 notpeter: enabling srv190 in imagescaler pool at half-weight to test precise image-scaling
11:52 paravoid: apt: include new php5 built with libicu42 (precise-wikimedia/main), include libicu42
11:46 mark: Moving US upload traffic to upload-lb.eqiad (Varnish)
09:06 paravoid: squid deploy all: switching math & timeline to be served from swift
05:23 binasher: started replicating db64 from the s3 master
03:51 logmsgbot: olivneh synchronized php-1.20wmf12/extensions/E3Experiments/experiments/openTask.js
03:28 logmsgbot: olivneh synchronized php-1.20wmf12/extensions/E3Experiments/experiments/openTask.js
02:37 logmsgbot: LocalisationUpdate completed (1.21wmf1) at Tue Oct 2 02:37:57 UTC 2012
02:21 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Tue Oct 2 02:21:28 UTC 2012
01:56 Tim: send sessions to both the existing memcached and also redis, to fill redis and allow for a graceful complete switch later
01:55 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
01:41 Tim: restarting all apaches to get redis PHP extension
01:35 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
01:33 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
01:16 binasher: streaming hotbackup of db1019 to db64 (s3)
01:12 binasher: stopping mysql on db25 and powering down - logging lots of ecc chipkill errors and periodically unresponsive. building db64 as a perm replacement
01:00 logmsgbot: awjrichards Finished syncing Wikimedia installation... : Updating MobileFrontend, per https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-10-01
00:50 logmsgbot: asher synchronized wmf-config/db.php 'pulling db25, returning es10'
00:07 logmsgbot: awjrichards Started syncing Wikimedia installation... : Updating MobileFrontend, per https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-10-01

October 1

22:10 logmsgbot: awjrichards synchronized wmf-config/mobile.php 'Eanble wgMFEnableDesktopResources on testwiki'
22:09 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Eanble wgMFEnableDesktopResources on testwiki'
21:40 logmsgbot: preilly synchronized php-1.21wmf1/extensions/ZeroRatedMobileAccess/ 'zero weekly carrier test'
21:18 logmsgbot: preilly synchronized php-1.20wmf12/extensions/ZeroRatedMobileAccess/ 'zero weekly carrier test'
20:44 logmsgbot: reedy synchronized php-1.21wmf1/includes/specials/SpecialUserlogin.php 'bug 40668'
20:23 logmsgbot: reedy synchronized php-1.21wmf1/includes/OutputPage.php
20:21 logmsgbot: reedy synchronized php-1.21wmf1/includes/OutputPage.php
20:16 logmsgbot: reedy synchronized php-1.21wmf1/includes/OutputPage.php
20:13 logmsgbot: reedy synchronized php-1.21wmf1/includes/OutputPage.php
19:33 paravoid: pushing new rewrite.py to swift servers (r24303, r24514, r24576)
19:31 logmsgbot: reedy synchronized php-1.21wmf1/includes/job/Job.php
19:24 logmsgbot: reedy synchronized php-1.20wmf12/includes/job/Job.php
18:51 notpeter: stopping puppet on brewster
18:22 logmsgbot: asher synchronized wmf-config/db.php 'pulling es10, crashed over weekend, needs to catchup on repl'
18:01 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.21wmf1
15:52 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.21wmf1, rebuild cache
15:40 ^demon: restarted gerrit on manganese.
15:36 ^demon: gracefully shut down gerrit until we figure out why manganese is swapping
14:18 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.21wmf1, rebuild cache
14:08 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.21wmf1, rebuild cache
14:06 mark: Added cp1029 and cp1030 to the frontend varnish upload pool in eqiad (PyBal)
13:59 logmsgbot: reedy synchronized php-1.21wmf1/
13:53 logmsgbot: reedy synchronized live-1.5/
13:53 logmsgbot: reedy synchronized wmf-config/
13:48 logmsgbot: reedy synchronized php-1.21wmf1
13:10 cmjohnson1: powering down db62 to replace raid controller card
12:38 logmsgbot: reedy synchronized live-1.5
12:34 Reedy: Make that, deleting wmf10 php source files
12:32 Reedy: Deleting wmf12 php source files
12:30 Reedy: Deleting wmf11 localisation cache
11:11 mark: Added cp1029 and cp1030 to the Varnish upload.eqiad pool (10 mins ago)
09:17 mark: Rebooting cp3001, both ports in LACP block
02:25 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Mon Oct 1 02:25:14 UTC 2012

September 30

23:58 MaxSem: Ran mwscript extensions/WikimediaMaintenance/migrateWlmUploads.php testwiki --do-it>wlm.log
19:26 logmsgbot: maxsem synchronized wmf-config 'Remove WLM banner / https://gerrit.wikimedia.org/r/#/c/25771/'
02:23 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Sun Sep 30 02:23:14 UTC 2012

September 29

10:39 mark: Started rsync of /export/upload (respecting /etc/rsync.includes) to /mnt (nas1-a:vol/originals/) on ms7 in a screen session
10:35 mark: Manually mounted nas1-a:/vol/originals on ms7, fenari
10:33 mark: Created 40 TB volume 'originals' inside 36-drive aggregate 'media1' on nas1-a
04:07 mutante: reinstalled cp1031 (new mainboard, thx RobH)
02:24 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Sat Sep 29 02:24:45 UTC 2012
01:59 mutante: restarting cp1031 after hardware replacement
01:24 mutante: checking/fixing puppet runs on cp103x servers

September 28

23:54 binasher: rebooting db1047 to new kernel
21:28 apergos: disbaled puppet and stopped swift processes on ms-be6 again. noticed four drives in "Unconfigured" state in megacli output after disk replacement,don'tknow more details about how that went.
20:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
20:04 mark: Destroyed nas1001-a:images volume, containing aggregate test0, and started zeroing drives
20:02 mark: Destroyed test0 aggregate on nas1-a, zeroing disks
19:55 mark: Destroyed volume nas1-a:images
19:54 MaxSem: And one more autocreation + re-running the migration script
19:51 mark: Destroyed snapmirror relationship between nas1001-a:images -> nas1-a:images and deleted related snapshots
19:51 MaxSem: Autocreated Commons accounts for 3 testwiki users to migrate their images
19:24 MaxSem: running mwscript extensions/WikimediaMaintenance/migrateWlmUploads.php testwiki --do-it
19:03 logmsgbot: reedy synchronized wmf-config/
19:01 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
18:55 logmsgbot: reedy synchronized wmf-config/
18:38 logmsgbot: reedy synchronized wmf-config/
18:04 binasher: stopping mysql on db1047 for upgrades
16:19 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Set wgCopyUploadProxy and enable copyupload on testwiki'
16:17 logmsgbot: reedy synchronized php-1.20wmf12/includes/upload/UploadFromUrl.php
16:16 logmsgbot: reedy synchronized php-1.20wmf12/includes/DefaultSettings.php
15:46 RobH: done with db62. it was in the ubuntu installer when I took it over, so its just sitting now
14:53 RobH: db62 being pulled for drac7 work
14:39 Jeff_Green: rebooting aluminium
13:11 mark: Moved BGP transit sessions to AS1257 from cr2-eqiad (over equinix exchange) to cr1-eqiad (dedicated link)
02:26 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Fri Sep 28 02:26:27 UTC 2012

September 27

23:13 logmsgbot: olivneh synchronized php-1.20wmf12/extensions/Vector/modules/ext.vector.footerCleanup.css
22:50 logmsgbot: olivneh synchronized wmf-config
22:05 logmsgbot: olivneh synchronized php-1.20wmf12/extensions/E3Experiments
22:00 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update PageTriage and ArticleFeedbackv5 to master
21:32 logmsgbot: bsitu Started syncing Wikimedia installation... : Update PageTriage and ArticleFeedbackv5 to master
20:48 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'turning off moodbar on testwiki'
20:09 RobH: authdns-update for new misc servers mgmt in eqiad
19:10 cmjohnson1: search32 going down to swap cpu's (dell request)
18:58 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
17:42 cmjohnson1: bring db62 down to swap disk controller card per ct/asher
17:16 logmsgbot: demon synchronized wmf-config/throttle.php 'Syncing I8c44415b: Account creation throttle for ptwikiversity event'
16:42 cmjohnson1: ms-be6 powering down
16:37 RobH: cp1031 mainboard replaced per rt3614, will need reinstall
15:59 RobH: mc1006 memory repaired rt3613
15:53 RobH: mc1002 repaired rt3612
15:33 RobH: mc1002 (rt3612) & mc1006 (rt3613) offline for memory repair/swap
15:20 RobH: authdns-update for wikimediafoundation.info
15:06 Jeff_Green: indium dist-upgrade & reboot
13:58 Jeff_Green: dist-upgrade & reboot grosley
10:55 apergos: ignore ms-be6 messages, I'm trying to get into the bleeping lsi raid util
04:04 logmsgbot: olivneh synchronized php-1.20wmf12/extensions/E3Experiments/experiments/postEditFeedback.js
02:36 Jeff_Green: powercycle fenari, it's nonresponsive even via drac terminal

September 26

20:42 RobH: mc1011 drac fixed
20:39 RobH: mc1009, 1010 drac fixed
20:30 RobH: cp1031 will remain offline until replacement parts arrive tomorrow rt3614
18:44 RobH: correction rt 3613
18:43 RobH: mc1006 testing done, single bad dimm, support ticket filed rt3614
18:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: 285 remaining wikis to 1.20wmf12
18:06 RobH: mc1006 offline for memory troubleshooting per rt 3613
16:52 RobH: mc1002 offline, memory bad per rt3612, new memory will arrive tomorrow
16:14 RobH: carbon rebooting for console redirection issues, any eqiad based installs will fail during this downtime
15:57 Jeff_Green: dist-upgrade & reboot most payments boxes
15:21 RobH: bast1001 link speed fixed, done working on it
15:19 RobH: troubleshooting bast1001 link speed per 3414
15:12 cmjohnson1: sq37 has fatal error...powering down to replace disk controller card
13:56 cmjohnson1: ms-be8 powercycling
02:48 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Wed Sep 26 02:48:22 UTC 2012
02:23 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Wed Sep 26 02:23:25 UTC 2012
01:00 mutante: reinstalling cp1030 with precise
00:34 mutante: fixing puppet on singer..finally

September 25

23:25 binasher: ran "varnishadm param.set nuke_limit 300" on all mobile varnish front and back instances to match new default config
23:20 binasher: repooled cp1044
23:16 binasher: depooling cp1044 from lvs for testing
22:44 binasher: stopping puppet on cp1044, experimenting with varnish lru / nuke params to solve allocation failures
22:28 mutante: generating locales on singer, add ro UTF-8 locale, run ro.planet
21:50 Jeff_Green: dropped apparently-unused IP alias on loudon, deprecated from old haproxy install
21:50 Jeff_Green: apt-update and reboot loudon
21:36 Jeff_Green: replaced iptables ruleset for loudon
21:02 notpeter: stopping puppet on brewster
20:49 mutante: sync-common-file new favicon.ico for wikidata.org
20:49 logmsgbot: dzahn synchronized docroot/www.wikidata.org/favicon.ico 'updating wikidata favicon per Lydia'
17:57 Jeff_Green: restarted udp2log on locke to clean up ~20 defunct procs
17:56 notpeter: temp stopping puppet on brewster
17:27 mutante: adding otto to ops group in LDAP
14:39 cmjohnson1: ms-be7 shutting down to check jumpers
14:35 cmjohnson1: ms-be8 shutting down to check jumpers
12:14 mark: Powercycled carbon, has been down for 4d
05:58 Tim: updated Scribunto for bug 40480
05:45 paravoid: upgrading all PDF boxes for USN-1581-1/CVE-2012-4405
05:45 paravoid: upgrading all imagescalers for USN-1581-1/CVE-2012-4405

September 24

19:44 RobH: db1012 card install done, system powered down for asher
19:36 RobH: db1012 shutting down for pci card install per rt3558
19:14 logmsgbot: reedy synchronized php-1.20wmf12/extensions/TimedMediaHandler 'Updating to master'
18:54 pgehres: added new columns to fundraising contribution_tracking table
18:20 notpeter: temp stopping puppet on brewster
18:15 logmsgbot: hashar synchronized wmf-config/filebackend-wmflabs.php 'beta: disable commons as a default foreign file repo'
18:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf12
17:03 cmjohnson1: powering up and toying with ms-be8 for h/w checks
16:35 RobH: mc1001-1008 memory upgrade complete
16:35 cmjohnson1: ms-be6 shutting down for h/w testing
15:51 RobH: mc1001-mc1008 are having memory upgrades installed. hosts already offline.
13:53 pp-pdf1: restart all services
13:52 pp-pdf1: updated mwlib.epub to 0.14.2
13:52 pp-pdf1: updated mwlib.rl to 014.1
13:52 pp-pdf1: update mwlib to 0.14.1
13:51 pp-pdf1: updated timelib to 0.2.4
13:50 pp-pdf1: updated pip to 1.2.1
13:50 pp-pdf1: updated distribute to 0.6.28
13:17 LeslieCarr: switching unnumbered GRE tunnel to numbered GRE Tunnel between cr2-knams and cr2-eqiad for routability reasons
12:45 LeslieCarr: restarting powerdns on ns2
09:34 logmsgbot: hashar synchronized docroot/bits/static-master/ '(bug 40419) extension assets not available on beta (aka add docroot/bits/static-master'
09:23 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 40436) Namespaces configuration for se.wikipedia'
09:19 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 40186) Favicon for es.wiktionary'
02:53 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Mon Sep 24 02:53:05 UTC 2012
02:27 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Mon Sep 24 02:27:55 UTC 2012
01:53 Tim: on hume: running updateCollation.php with --force for bug 40462
01:50 Tim: depooled precise appservers and stopped job runners on precise servers

September 23

23:26 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'Reduce wgMaxGeneratedPPNodeCount to 1.5M'
17:29 binasher: powercycled cp1043
02:45 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Sun Sep 23 02:45:21 UTC 2012
02:26 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Sun Sep 23 02:26:13 UTC 2012

September 22

09:26 mark: Restarted lucene on search1015
02:46 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Sat Sep 22 02:46:32 UTC 2012
02:27 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Sat Sep 22 02:27:27 UTC 2012

September 21

20:55 notpeter: removing srv190 from apaches pool to use to test build as an imagescaler
20:38 logmsgbot: asher synchronized wmf-config/db.php 'moving ES writes from es1 (cluster23) to es2 (cluster24) and es3 (cluster25)'
20:29 logmsgbot: asher synchronized wmf-config/db.php 'db.php with new es shards added, but not being written to'
20:29 ^demon: restarted gerrit like 10m ago to deploy I03de7c74 (apache timeout increase)
20:17 Jeff_Green: apt-get upgrade on manganese, restart apache for adjusted 5->10m gerrit timeout
20:15 mutante: analytics1007 seems dead. powercycle but never see any console output on mgmt at all
14:49 hashar: labs: cleaned out a 4GB file out of labs-nfs1:/export/home/deployment-prep
02:49 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Fri Sep 21 02:49:27 UTC 2012
02:25 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Fri Sep 21 02:25:33 UTC 2012
01:39 logmsgbot: krinkle synchronized php-1.20wmf12/extensions/LiquidThreads 'I5a098a42'
01:37 logmsgbot: krinkle synchronized php-1.20wmf12/skins/common/preview.js 'Ia9673a75'
00:16 binasher: configured pt-heartbeat on es2 and es3 shards

September 20

23:18 logmsgbot: kaldari synchronized php-1.20wmf12/extensions/PageTriage/modules/ext.pageTriage.defaultTagsOptions/ext.pageTriage.defaultTagsOptions.js
23:17 logmsgbot: kaldari synchronized php-1.20wmf11/extensions/PageTriage/modules/ext.pageTriage.defaultTagsOptions/ext.pageTriage.defaultTagsOptions.js
22:58 Ryan_Lane: restarting gerrit
22:36 notpeter: stopping puppet on brewster
22:20 logmsgbot: aaron synchronized wmf-config/filebackend.php
22:18 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Enabled short thumbnails for testwiki'
22:15 logmsgbot: olivneh synchronized php-1.20wmf11/extensions/E3Experiments/experiments
22:02 logmsgbot: kaldari Finished syncing Wikimedia installation... :
21:32 logmsgbot: kaldari Started syncing Wikimedia installation... :
20:32 ottomata: reinstalling analytics1002-analytics1010 with Ubuntu precise
20:25 logmsgbot: aaron synchronized php-1.20wmf12/includes/filebackend/SwiftFileBackend.php
20:25 logmsgbot: aaron synchronized php-1.20wmf11/includes/filebackend/SwiftFileBackend.php
20:13 mutante: bugs.mediawiki.org|bugzilla.wikipedia.org|bugzilla.mediawiki.org all redirect to https:// rather than http now
20:12 logmsgbot: dzahn gracefulled all apaches
20:10 mutante: sync-apache and pushing out redirects.conf change for Bugzilla redirects to https
20:01 logmsgbot: kaldari Finished syncing Wikimedia installation... :
19:50 logmsgbot: aaron synchronized php-1.20wmf11/includes/filebackend/SwiftFileBackend.php 'deployed 8001d1ebf5b79b5a6ea359646284347d31e5442b'
19:38 logmsgbot: aaron synchronized php-1.20wmf12/includes/filebackend/SwiftFileBackend.php 'deployed 4246f85ac2a68a19b9fa8f4954c5ff7aabea2922'
19:33 logmsgbot: py synchronized wmf-config/lucene.php
19:32 notpeter: pushing all search traffic back to eqiad
19:18 logmsgbot: kaldari Started syncing Wikimedia installation... :
18:58 binasher: ran scap on mw58 and returning to the lvs pool
17:08 logmsgbot: aaron synchronized php-1.20wmf12/thumb.php 'deployed fd26ac3603c544c9f2273408b14034cd22b56a52'
17:04 notpeter: temp stopping puppet on brewster
10:41 hashar: UploadWizard has been broken in some cases following the release of wmf12. Restored by reverting a change. See bug 40380 for details.
10:13 logmsgbot: hashar synchronized php-1.20wmf12/resources/mediawiki.language '(bug 40380) inexistent language files are loaded'
08:12 logmsgbot: hashar synchronized php-1.20wmf12/includes/EditPage.php '(bug 40098) Wrong edit summary generation when adding a new discussion section'
08:11 logmsgbot: hashar synchronized php-1.20wmf11/includes/EditPage.php '(bug 40098) Wrong edit summary generation when adding a new discussion section'
08:03 hashar: Jenkins: installing "Gerrit Trigger plugin" 2.6 which replaces our 2.5.3 snapshot
05:44 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
05:40 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
05:26 logmsgbot: tstarling synchronized php-1.20wmf12/includes/parser/Parser.php 'node limit log'
05:23 logmsgbot: tstarling synchronized php-1.20wmf11/includes/parser/Preprocessor_DOM.php 'node limit'
05:22 logmsgbot: tstarling synchronized php-1.20wmf11/includes/parser/Parser.php 'node limit'
05:22 logmsgbot: tstarling synchronized php-1.20wmf11/includes/parser/ParserOptions.php 'node limit'
05:13 Tim: fixing common-local on mw58, has wrong permissions
05:11 logmsgbot: tstarling synchronized php-1.20wmf11/includes/DefaultSettings.php 'node limit'
03:23 Ryan_Lane: purged search data from dataset2 on diederik's request
02:44 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Thu Sep 20 02:44:42 UTC 2012
02:21 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Thu Sep 20 02:20:57 UTC 2012

September 19

22:48 binasher: pulling mw58 from lvs pool and deploy groups for temporary testing
22:07 logmsgbot: olivneh synchronized php-1.20wmf11/extensions/E3Experiments
21:50 notpeter: temp stopping puppet on brewster
21:28 apergos1: ms-be10: after chris installed the disks, filesystems were created ok by puppet and mounted but on a couple of reboots, there were boot messages about two not identical lists of disk not present or not ready; waiting a little bit allowed them to come up but it's not great.
20:56 Jeff_Green: restarting gmetad on nickel
20:32 Reedy: Deleting php-1.20wmf10 localisation cache from mediawiki-installation
20:00 Reedy: Deleting php-1.20wmf9 from mediawiki-installation
19:43 logmsgbot: reedy synchronized php-1.20wmf12/extensions/CategoryTree 'Category tree to master'
18:12 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: Swapping all non-pedias to 1.20wmf12
18:07 cmjohnson1: ms-be10 powering down to replace 12 disk drives
17:38 logmsgbot: py synchronized wmf-config/lucene.php
17:30 logmsgbot: py synchronized wmf-config/lucene.php
17:26 logmsgbot: aaron synchronized php-1.20wmf12/thumb.php 'deployed 7be7e496ae2d6e4b082278ab4ace8c1b4b08e433'
17:24 notpeter: moving search traffic to pmtpa for eqiad search package upgrades, starting with en
15:22 cmjohnson1: updating authdns. adding 12 new misc servers mgmt ip's
13:31 mark: Deactivated subnet for vlan 104 (Tampa)
12:28 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 39264) Add Tudalen: and Indecs: namespaces to cy.wikipedia'
12:24 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 38840) Namespaces configuration on uz.wikipedia'
12:18 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 38840) Namespaces configuration on uz.wikipedia'
12:13 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 39206) Namespaces configuration for se.wikipedia'
12:02 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 40163) Try to fix ltwiki import source for betawikiversity'
06:01 paravoid: depooling ms-fe1; broken after precise upgrade; pending further investigation
04:46 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild localisation caches for pagetriage
04:04 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild localisation caches for pagetriage
02:48 logmsgbot: reedy synchronized php-1.20wmf12/extensions/PageTriage
02:48 logmsgbot: reedy synchronized php-1.20wmf11/extensions/PageTriage
02:42 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Wed Sep 19 02:42:01 UTC 2012
02:22 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Wed Sep 19 02:22:47 UTC 2012
01:27 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
01:17 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php

September 18

20:17 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '23795 disable anon edits on wikimania2012 wiki'
20:13 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 39652) Fix "autoreviewer" restriction level on ptwiki'
20:09 logmsgbot: hashar: ran "mwscript updateArticleCount.php --wiki guwikisource --update" for bug 40173
20:07 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 40173) Set wgArticleCountMethod for gu.wikisource'
20:04 logmsgbot: hashar synchronized wmf-config/CommonSettings-wmflabs.php '17365 beta: Removes transcoding stub'
19:50 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'cleanup sep11wiki left over'
19:50 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'cleanup sep11wiki left over'
19:50 logmsgbot: hashar synchronized closed.dblist 'cleanup sep11wiki left over'
19:48 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'Cleaning Berlin hackathon tutorial configuration.'
19:45 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 39942) Disables patrol on fi.wikipedia'
19:43 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 39905) Create interface editor group on pt.wikipedia'
19:38 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '24207 enable EPUB collection format on ALL wiki
19:34 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'ran stylize.php on InitialiseSettings.php 23998'
19:13 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'Removing live hack to enable EPUB format on all wiki, should use $wmgCollectionUseEpub instead (see our internal mailing list)'
19:01 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 40186) New logo for es.wiktionary'
17:16 logmsgbot: hashar synchronized wmf-config/InitialiseSettings-wmflabs.php 'fix AFTv5 on beta'
17:07 notpeter: putting srv190 and mw56-59 back into apaches pool
16:52 cmjohnson1: search32 going down to check if it's DIMM error or possible board or cpu1 error
15:41 cmjohnson1: srv249 powercycling to check drac configuration
14:08 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 40122) disable GlobalBlocking on fishbowl, private wikis'
13:53 logmsgbot: hashar synchronized php-1.20wmf11/includes/Sanitizer.php 'bug 40306 - Only convert align to float for table'
13:52 logmsgbot: hashar synchronized php-1.20wmf11/tests/phpunit/includes/SanitizerTest.php 'bug 40306 - Only convert align to float for table'
13:48 apergos1: ms-fe1 back in swift frontend pool, upgrade to precise complete.
13:41 hashar: 23798 manually deleted /apache/common-local/docroot/wikipedia.org/{million,test}.txt files from all mediawiki-installation hosts
12:53 logmsgbot: hashar synchronized php-1.20wmf12/includes/Sanitizer.php 'bug 40306 - Only convert align to float for table'
12:53 logmsgbot: hashar synchronized php-1.20wmf12/tests/phpunit/includes/SanitizerTest.php 'bug 40306 - Only convert align to float for table'
12:22 apergos1: reboot ms-fe1, upgrading to precise
05:59 Aaron|home: Last of the math/ file copy scripts has finished
02:42 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Tue Sep 18 02:42:15 UTC 2012
02:26 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Tue Sep 18 02:26:08 UTC 2012
00:20 Ryan_Lane: adding an autofs direct map to LDAP, pointing at the gluster storage for testing home directory move

September 17

23:43 paravoid: adding a modified ixgbe.ko to precise-installer's initrd.gz; adding ixgbe-dkms to apt
23:30 notpeter: starting puppet on brewster
23:17 notpeter: stopping puppet on brewster
21:45 apergos: (from 5 hours ago) last of the swift proxy services restarted, it had reached 8.6gb of swap use. good for a few more days now
20:52 Jeff_Green: stopped puppet on aluminium for config testing
20:45 logmsgbot: preilly synchronized wmf-config/CommonSettings.php 'enable EPUB'
18:55 notpeter: removing srv190 and mw55-59 from apaches pool for upgrade to precise
18:48 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.20wmf12
18:47 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild localisation cache
18:20 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild localisation cache
18:13 logmsgbot: aaron synchronized php-1.20wmf12/extensions/timeline/Timeline.php
18:10 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enabled html5 on all wikis'
18:07 binasher: granted 'research' read access to user rollup/filter dbs on dbs 42,1047
18:06 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.20wmf12 and rebuild localisation cachae
16:36 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.20wmf12 and rebuild localisation cachae
16:21 logmsgbot: reedy synchronized php-1.20wmf11/extensions/ArticleFeedbackv5/ 'Bring in logging related fixes etc'
16:17 logmsgbot: reedy synchronized php-1.20wmf12 'Initial sync'
15:31 Reedy: Deleted 1.20wmf10 l10n cache
15:28 Reedy: Pointing php to php-1.20wmf11
08:21 apergos1: swap was using 7gb on ms-fe1... restarted swift-proxy over there. wlll prolly restart it on ms-fe2 through 4 over the next day or two
02:26 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Mon Sep 17 02:26:30 UTC 2012

September 16

14:53 mark: powercycled srv256
14:53 mark: powercycled srv250
14:49 mark: cleared profiling
02:20 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Sun Sep 16 02:20:02 UTC 2012

September 15

19:04 mark: Power cycled srv212, 211, 226, 273, 237
19:04 logmsgbot: tstarling synchronized wmf-config/mc.php 'swapping out other slow servers'
18:53 logmsgbot: tstarling synchronized wmf-config/mc.php 'swap out srv237 for srv249'
18:46 logmsgbot: tstarling synchronized php-1.20wmf11/includes/objectcache/MemcachedClient.php 'memcached profiling'
18:37 mark: Reduced MaxRequestsPerChild from 300 to 4 in apache.conf
18:29 LeslieCarr: logs stopped on fluorine between 18:11 and 18:27 UTC
14:03 LeslieCarr: powercycling srv259 (unresponsive)
13:57 LeslieCarr: rebooting srv270 (unresponsive)
13:36 LeslieCarr: rebooting srv288
11:54 logmsgbot: andrew synchronized wmf-config/throttle.php 'Fix undeclared variable error'
09:56 logmsgbot: reedy synchronized wmf-config/throttle.php 'Bug 40263 - Request to lift IP address for mass creation - Philippines SFD 2012'
08:52 thedj: someone should deploy Planets update, or add blogs to the old config because otherwise WLM has passed before the blogs are added to the stream.
04:04 LeslieCarr: started ircd using the instructions here - http://wikitech.wikimedia.org/view/IRCD on ekr�em
03:55 LeslieCarr: powercycling ekrem
02:29 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Sat Sep 15 02:29:57 UTC 2012

September 14

15:32 mark: Powercycled sodium
15:18 logmsgbot: hashar synchronized docroot/noc/conf/wikiversions.dat 'publish wikiversions.dat on noc.wm.org'
04:43 Aaron|home: Copying missing timeline files from swift to nfs on hume
04:36 logmsgbot: aaron synchronized php-1.20wmf11/extensions/timeline/Timeline.php 'Switched to doOperations().'
03:24 logmsgbot: aaron synchronized php-1.20wmf11/extensions/timeline 'deployed 6e7076f35b38c16579342c69e4f19f8df4ed0ab6'
02:22 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Fri Sep 14 02:22:36 UTC 2012

September 13

23:04 logmsgbot: dzahn gracefulled all apaches
23:04 mutante: apache-graceful-all, pushing out redirect for wikimedia.biz
21:02 mutante: fixing UID for spage on stat1 (/etc/passwd and files), was used twice
20:34 logmsgbot: reedy synchronized php-1.20wmf11/extensions/Collection/
18:47 RobH: authdns-update for wikimedia.biz
17:05 mutante: flushing mobile varnish cache
17:03 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Enable wlm banner'
17:02 logmsgbot: awjrichards synchronized php-1.20wmf11/extensions/MobileFrontend/stylesheets/banner.css 'Touching banner.css'
02:20 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Thu Sep 13 02:20:30 UTC 2012
01:51 logmsgbot: aaron synchronized php-1.20wmf11/thumb.php 'debug logging'
01:46 logmsgbot: aaron synchronized php-1.20wmf11/thumb.php 'debug logging'
01:44 logmsgbot: awjrichards synchronized php-1.20wmf11/extensions/MobileFrontend/stylesheets/banner.css 'Update timestamp on banner.css'
01:41 logmsgbot: aaron synchronized php-1.20wmf11/thumb.php 'debug logging'
01:32 logmsgbot: aaron synchronized php-1.20wmf11/thumb.php 'debug logging'
01:01 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 're-disable wlm banner on enwiki'
00:58 logmsgbot: awjrichards synchronized php-1.20wmf11/extensions/MobileFrontend/
00:55 logmsgbot: awjrichards synchronized wmf-config/mobile.php 'Easier toggle of MF banner'
00:54 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Easier toggle of MF banner'
00:34 logmsgbot: awjrichards synchronized wmf-config/mobile.php 'Disabling wlm banner for mobile frontend'
00:18 logmsgbot: catrope synchronized php-1.20wmf11/extensions/Vector 'c9c0c3098c5bc6c15362332a1ed18e8545b1ab07'
00:17 logmsgbot: catrope synchronized php-1.20wmf11/skins/common/shared.css 'f9242f450ae16cf4f0a99e6fa78248e13ccb4ee0'
00:12 logmsgbot: aaron synchronized php-1.20wmf11/extensions/timeline/Timeline.php 'debug logging'
00:07 logmsgbot: maxsem Finished syncing Wikimedia installation... : WLM app banner - MF, WikimediaMessages and config

September 12

23:52 logmsgbot: reedy synchronized php-1.20wmf11/extensions/MobileFrontend/
23:38 logmsgbot: maxsem Started syncing Wikimedia installation... : WLM app banner - MF, WikimediaMessages and config
23:09 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'migrate favicons from upload to bits'
23:08 logmsgbot: hashar synchronized docroot/strategyapps/favicon.ico
23:07 logmsgbot: hashar synchronized docroot/strategy/favicon.ico
23:06 logmsgbot: hashar synchronized docroot/steward/favicon.ico
23:06 logmsgbot: hashar synchronized docroot/outreach/favicon.ico
23:06 logmsgbot: hashar synchronized docroot/checkuser/favicon.ico
23:06 logmsgbot: aaron synchronized php-1.20wmf11/extensions/timeline 'deployed f79a67ac344801acc57edbc95158bf5ecc87481b'
23:03 logmsgbot: hashar synchronized docroot/bits/favicon
23:02 logmsgbot: hashar synchronized docroot/bits/favicon
22:24 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Enabled auto-resyncing for file backends.'
22:12 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Disabled directory operations on local-temp zones.'
22:12 logmsgbot: reedy synchronized closed.dblist
21:48 logmsgbot: aaron synchronized php-1.20wmf11/includes/filebackend/FileBackendMultiWrite.php 'deployed 6a6a0427160e1493617789ce320020b21d0836dd'
21:48 logmsgbot: aaron synchronized php-1.20wmf11/languages/messages/MessagesEn.php 'deployed 6a6a0427160e1493617789ce320020b21d0836dd'
21:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Subpagesortkey on ptwikibooks and enwikibooks'
21:13 RoanKattouw_: On ms7: find /export/upload/math ! -user apache -exec /root/fixownership2 \{\} \;
21:10 logmsgbot: reedy synchronized php-1.20wmf11/includes/api/ApiQueryLogEvents.php
20:51 logmsgbot: reedy synchronized php-1.20wmf11/extensions/TimedMediaHandler/
20:50 logmsgbot: reedy synchronized php-1.20wmf11/extensions/MwEmbedSupport/
19:05 notpeter: reimaging mw1-16
19:00 hashar: jenkins: hashar installed a snapshot of Gerrit Trigger plugin
18:57 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else over to 1.20wmf11
18:47 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki back to 1.20wmf11
18:09 notpeter: adding mw59 to apaches pool
18:03 logmsgbot: reedy synchronized php-1.20wmf10
17:58 logmsgbot: reedy synchronized php-1.20wmf11
17:53 notpeter: re-adding srv194-199 to apaches pool
17:38 notpeter: removing mw1-16 from apaches pool to reimage them as precise jobrunner boxes
14:46 paravoid: restarted swift on ms-fe1-4, proxy-server was leaking memory and had been swapping
02:52 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Wed Sep 12 02:52:12 UTC 2012
02:27 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Wed Sep 12 02:27:16 UTC 2012

September 11

22:29 notpeter: removing srv195-199 from apaches pool for upgarde to precise
20:51 RobH: authdns-update per rt 1326
18:31 apergos: second reboot had whines of device not present/not ready for /dev/sdn1 and /dev/sdh1 (note the list is not the same as the previous list), waited a little and it booted up with all disks mounted
18:26 apergos: first reboot of msbe10 showed complaints of not ready on devices sdi, k, n but they did eventually mount when the box came up fully. rebooting a second time to see what happens
18:13 apergos: rebooting ms-be10 after recreation and remount of swift filesystems (except those on ssds), see if they stick
17:55 apergos: new degradedarray event on ms-be6, booooo
17:54 apergos: rebooting ms-bs10 to see if we can get some reasonable device names, in prep for trying the same trick on it as on ms-be6 and 7
17:43 Reedy: Running sync-common on mw8
17:42 Reedy: mw8 seems to have out of date mw files
17:42 apergos: rebooting ms-be7 after recreating all filesystems (except those on the ssds) and remounting them, to see if the disks stay visible
17:38 logmsgbot: reedy synchronized php-1.20wmf11/extensions/ConfirmEdit/FancyCaptcha.class.php
17:31 apergos: puppet disabled on ms-be7 via /etc/default/puppet, swift processes shut down while we test
17:10 apergos: one more reboot of ms-be6 after raid repaired, see if it and all mounts stay ok
15:57 apergos: rebuilding the degraded raid array on ms-be6
15:15 apergos: stopping swift processes on ms-be6 til we decide what's next; all disks came up and were mounted (but we still do have degraded raid array, not so awesome)
15:12 apergos: redid other ms-be6 filesystems so labels would be correct with disk layout without ssds. reboot again to see if it comes up properly
14:44 apergos: manually recreated filesystems and mounted missing drives on ms-be6 (without ssds cabled), seems to have worked without errors. rebooting to see what happens.
14:18 apergos: stopped swift processes on ms-be6 and disabled puppet (/etc/default/puppet) so it won't restart them. still doing testing of hw over there so we need the box up.
04:46 logmsgbot: reedy synchronized php-1.20wmf11/extensions/ConfirmEdit/ '9643ccd3964827651e8a23a19a92ba3d240e60ee'
02:42 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Tue Sep 11 02:41:59 UTC 2012
02:22 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Tue Sep 11 02:22:02 UTC 2012
01:30 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki back to 1.20wmf10
01:02 paravoid: adjusting spence's firewall to include all eqiad subnets

September 10

22:53 Ryan_Lane: adding all current labsconsole users to mediawiki shell group
22:47 logmsgbot: maxsem synchronized php-1.20wmf11/extensions/MobileFrontend/ 'MF update'
22:46 logmsgbot: maxsem synchronized php-1.20wmf10/extensions/MobileFrontend/ 'MF update'
22:40 paravoid: swift: weight 66->100 for ms-be9, 11, 12
21:40 Ryan_Lane: added salt and dependent packages to lucid-wikimedia and precise-wikimedia repos
21:39 logmsgbot: maxsem synchronized php-1.20wmf11/extensions/MobileFrontend/api/ApiMobileView.php 'Bug 39489'
19:42 mutante: push change for glwiki logo to cluster (gerrit 23175)
19:42 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
18:32 RobH: authdns-update for wicipediacymraeg.org
18:01 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files:
17:08 Jeff_Green: adjusted locke:/etc/udp2log/squid to fix pattern match for fundraiser banners
02:44 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Mon Sep 10 02:44:38 UTC 2012
02:26 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Mon Sep 10 02:26:50 UTC 2012

September 9

17:41 logmsgbot: aaron synchronized php-1.20wmf11/includes/upload/UploadFromChunks.php
17:40 logmsgbot: aaron synchronized php-1.20wmf11/includes/upload/UploadFromChunks.php 'deployed 9824650d70a0b2362e35eff3a5ba632d923404f6'
17:39 logmsgbot: aaron synchronized php-1.20wmf11/includes/filerepo/FileRepo.php 'deployed 9824650d70a0b2362e35eff3a5ba632d923404f6'
02:47 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Sun Sep 9 02:47:05 UTC 2012
02:26 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Sun Sep 9 02:26:48 UTC 2012
01:13 Jeff_Green: restarted udp2log on locke

September 8

07:54 hashar: restarted Jenkins to fix up a plugin
02:45 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Sat Sep 8 02:45:17 UTC 2012
02:27 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Sat Sep 8 02:27:32 UTC 2012

September 7

22:22 mutante: creating new gerrit project for the wikibugs IRC bot as a .deb (operations/debs/wikibugs)
21:43 mutante: for more reference, mw8 is down for DIMM errors (RT-3499)
21:42 RoanKattouw: For reference, the dsh groups Rob removed mw8 from are mediawiki-installation and apaches
21:39 RobH: pulled mw8 out of dsh groups since its down
19:59 cmjohnson1: loading new OS on mw8 after mother board swap
16:02 logmsgbot: hashar synchronized wmf-config/ext-wmflabs.php '22172 - Deploying OnlineStatusBar on beta'
15:21 logmsgbot: py synchronized wmf-config/lucene.php
15:17 notpeter: moving pool2,3,4 and prefix search traffic to eqiad
15:14 logmsgbot: py synchronized wmf-config/lucene.php
15:07 notpeter: moving en search traffic from pmtpa to eqiad
13:26 logmsgbot: hashar: bug 38036 cp /mnt/upload6/wikipedia/favicon-si.ico /mnt/upload6/wiktionary/favicon-si.ico
11:07 mark: Sending Canadian upload traffic to upload-lb.eqiad (Varnish)
07:12 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'change cortado location back again'
06:59 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'change cortado location'
04:26 Tim: professor has stopped responding to ssh and is not reporting to ganglia, rebooting
02:37 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Fri Sep 7 02:37:41 UTC 2012
02:19 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Fri Sep 7 02:19:43 UTC 2012
01:02 AaronSchulz: Started copying math/ files into swift
00:58 logmsgbot: aaron synchronized wmf-config/CommonSettings.php
00:49 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Removed excess math global'
00:28 logmsgbot: aaron synchronized wmf-config/ 'Write math files to swift and NFS'
00:19 logmsgbot: aaron synchronized wmf-config/filebackend.php 'deployed 5aa6ab06fd4e0e76bd6a64e468c9eacb0f46f680'
00:06 Ryan_Lane: remove cn from uniqueness check in LDAP. it's a global check and causes problems
00:04 RobH: authdns-update to support wicipediacymraeg.org

September 6

23:23 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php '...and anonnotice too (5f7cdae1da72895bcdb485ee7983944099f5af15)'
23:18 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'wgForceUIMsgAsContentMsg for bewikimedia (c1ef8fc89889a9802845c5f73381285a75a79eb3)'
21:56 logmsgbot: kaldari Finished syncing Wikimedia installation... :
21:03 logmsgbot: kaldari Started syncing Wikimedia installation... :
19:53 ottomata: stopping puppet on oxygen. (puppet upgraded udp-filter before I was ready, I have to make sure ip filtering still works before I can turn it back on)
19:38 logmsgbot: reedy synchronized php-1.20wmf11/includes/media/FormatMetadata.php
18:40 RobH: authdns-update for ersch ip and removing old decom servers from pdns templates
18:29 RobH: installing ersch as poolcounter server
18:17 notpeter: stopping search indexing on searchidx1001 and searchidx2 to sync to eqiad
17:49 RoanKattouw: Pushing master of operations/mediawiki-config.git into gerrit from fenari
17:39 logmsgbot: aaron synchronized php-1.20wmf11/includes/filebackend/SwiftFileBackend.php 'deployed 3b2518d0284267100a61445b5d811940c8891395'
17:39 logmsgbot: aaron synchronized php-1.20wmf11/includes/filebackend/FileBackend.php 'deployed 3b2518d0284267100a61445b5d811940c8891395'
16:14 mutante: srv219 - apt-get auto-remove unneeded packages, running puppet
16:11 mutante: srv219 - apt-get -f install, removing cm-super, dist-upgrading,
16:05 mark: Sending upload traffic from Argentina to upload-lb.eqiad (Varnish with streaming and persistence patches)
14:55 cmjohnson1: ms-be6 going down to remove ssd drives
14:23 notpeter: reimaging search1001, 1002, 1003
02:45 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Thu Sep 6 02:45:50 UTC 2012
02:28 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Thu Sep 6 02:28:18 UTC 2012
01:39 mutante: renaming mailing list chaptercommittee-l to affcom , rebuilding archives...
01:27 Ryan_Lane: enabling $wgEmailConfirmToEdit on labsconsole
01:21 Ryan_Lane: enabling ConfirmEdit with FancyCaptcha on labsconsole
01:09 Ryan_Lane: upgrading OATHAuth to 795cef09cab6ecb0e9ded35df06b2877ccc22c1a on labsconsole
00:19 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable LQT on lqtlabs'
00:19 Ryan_Lane: added SpamBlacklist to labsconsole
00:12 Ryan_Lane: added scribunto to labsconsole
00:12 RoanKattouw: Added rev_sha1 to revision table on liquidthreads_labswikimedia
00:09 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'and wgCanonicalServer too'
00:07 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Add wgServer for liquidthreads_labswikimedia'

September 5

23:59 logmsgbot: catrope rebuilt wikiversions.cdb and synchronized wikiversions files:
23:29 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php '$wgDisplayTitle=false; on bewikimedia (114ff229f4eb15c1dc88b6da238c7aeea987c025)'
23:21 RobH: done with local hacks on brewster, resuming puppet
23:19 logmsgbot: catrope synchronized php-1.20wmf11/extensions/VisualEditor 'Update VisualEditor (55b78f34db13c0abd43bcdcc8798a8c3425f14ba)'
23:06 RobH: stopping puppet on brewster for local hack testing on ms-be6
23:00 logmsgbot: reedy synchronized php-1.20wmf11/includes/api/ApiQuery.php
22:36 mutante: renaming mailing list "wsor" to "wmfresearch"
22:34 RobH: blog update successful, all plugins also updated.
22:31 RobH: blog software is outdated by a few versions, updating.
22:12 apergos: shutdown ms-be6, prep for puppet changes for reinstall w/o ssds
22:09 Ryan_Lane: pulling in change 22690 to labsconsole
20:45 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
20:30 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
20:15 logmsgbot: reedy synchronized php-1.20wmf11/includes/media/FormatMetadata.php
20:12 logmsgbot: aaron synchronized wmf-config/filebackend.php 'deployed c48c6c189ce576186457bb916f42f56e9d35d6f6'
20:11 Jeff_Green: rebooting storage3
19:57 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commonswiki back to 1.20wmf11
19:56 mutante: added new Nagios monitor to Watchmouse
19:53 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
19:43 mutante: deleting Ben as a Watchmouse contact
19:41 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Unbreak the site'
19:39 logmsgbot: catrope synchronized php-1.20wmf11/includes/ImagePage.php 'Fix for bug 40018 (ac25a9d1127848269b6c9766262c23f961b35c5b)'
19:39 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
19:38 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2 back to wmf11
19:36 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2 back to wmf10 for a few mins..
19:23 logmsgbot: krinkle synchronized docroot/default/index.html 'Ia7ca3eaf'
19:23 logmsgbot: krinkle synchronized docroot/default/wmf_logo_with_text.png 'Ia7ca3eaf'
19:00 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'revert all back to old cortado.jar'
18:59 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
18:56 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
18:48 logmsgbot: reedy synchronized wmf-config
18:40 Reedy: commonswiki to stay on 1.20wmf10 for the meantime
18:38 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commonswiki back to 1.20wmf10 for testing
18:32 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Go back to older cortado to check thumbnail isue'
18:27 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Use an updated version of cortado for testwiki and test2wiki'
18:25 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else non wikipedia to 1.20wmf11
18:22 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote and wiktionary to 1.20wmf11
18:19 logmsgbot: reedy synchronized php-1.20wmf11/includes/Timestamp.php '0000...'
18:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews, wikiversity and wikimedia to 1.20wmf11
18:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Misc wikis to 1.20wmf11
17:52 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Enable CORS for the API (347f386e2144688bcd4c80d24bd9f3be7d135d74)'
17:51 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable CORS for the API (347f386e2144688bcd4c80d24bd9f3be7d135d74)'
17:28 logmsgbot: aaron synchronized php-1.20wmf11/thumb.php 'deployed 8a63ff8c8558b1ca9f2265da71376e494053c234'
17:28 logmsgbot: aaron synchronized php-1.20wmf11/includes/filerepo/file/File.php 'deployed 8a63ff8c8558b1ca9f2265da71376e494053c234'
17:27 logmsgbot: aaron synchronized php-1.20wmf11/includes/StreamFile.php 'deployed 8a63ff8c8558b1ca9f2265da71376e494053c234'
15:54 Jeff_Green: updated MOTD for OTRS.
15:01 cmjohnson1: search32 still borked ..going down for troubleshooting
14:48 logmsgbot: reedy synchronized php-1.20wmf11/extensions/CheckUser 'CheckUser to master'
04:10 apergos: running d/d8 container consistency check fr ms7 from bast1001 in screen session as ariel
02:51 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Wed Sep 5 02:51:43 UTC 2012
02:26 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Wed Sep 5 02:26:57 UTC 2012
02:17 RobH: nagios crashed, fixing
01:05 Ryan_Lane: enabling DynamicSidebar extensions on labsconsole

September 4

23:34 AaronSchulz: Started copying timeline/ files into swift
23:34 logmsgbot: reedy synchronized wmf-config/
23:30 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Write timeline files to swift and NFS'
22:45 logmsgbot: catrope synchronized php-1.20wmf11/extensions/VisualEditor/ 'Update VisualEditor (59101ea3c93021c22da9796bf1715603967693fe)'
22:43 apergos: running heads on commons d/d7 on bast1001 as ariel from screen, data consistency check
22:07 notpeter: adding srv266 and srv281 back to apaches pool (to see if they break...)
21:04 logmsgbot: py synchronized wmf-config/lucene.php
21:01 notpeter: moving en search traffic back to pmtpa
20:46 logmsgbot: py synchronized wmf-config/lucene.php
20:45 notpeter: moving en search back to eqiad
20:41 Ryan_Lane: added oliver mail alias for okeyes
20:37 cmjohnson1: srv291 shutting down for troubleshooting
19:09 apergos: swift consistency check (heads of various objects) running from bastion1001 as ariel in screen session, about 10 HEADS/sec, if this starts to impact swift just shoot it
18:38 cmjohnson1: srv281 going down for troubleshooting
17:54 RobH: helium being reinstalled to poolcounter server per RT#3407
17:41 Ryan_Lane: pulled latest update of branch 1.20wmf10 on labsconsole
17:22 cmjohnson1: search32 shutting down to replace processor
16:26 notpeter: temp stopping puppet on brewster
16:26 notpeter: reimaging searchidx1001
15:26 logmsgbot: reedy synchronized wmf-config/ 'Staging for subpagesortkey'
15:24 logmsgbot: reedy synchronized wmf-config/
15:22 logmsgbot: reedy synchronized php-1.20wmf11/extensions/SubpageSortkey/
14:29 logmsgbot: reedy synchronized php-1.20wmf11/resources/startup.js 'touch'
14:28 logmsgbot: reedy synchronized php-1.20wmf10/resources/startup.js 'touch'
14:27 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'touch'
14:23 logmsgbot: reedy synchronized php-1.20wmf10/resources/jquery/jquery.js 'Revert back to 1.7.2 for wmf10 wikis'
14:19 logmsgbot: reedy synchronized php-1.20wmf11/resources/jquery/jquery.js 'Revert back to 1.7.2 for wmf11 wikis'
12:55 mark: Pushed varnish 3.0.3plus-rc1 packages into the precise-wikimedia APT repository
02:51 paravoid: powering down ms-be6, broken hardware
02:39 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Tue Sep 4 02:39:34 UTC 2012
02:24 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Tue Sep 4 02:24:01 UTC 2012
01:28 logmsgbot: tstarling synchronized wmf-config/flaggedrevs.php

September 3

18:34 logmsgbot: reedy synchronized live-1.5/
18:26 Reedy: Deleting php-1.20wmf8 from mediawiki-installation
18:25 Reedy: Deleting 1.20wmf9 l10n cache from mediawiki-installation
18:22 Reedy: Repointed php symlink from wmf9 to wmf10
18:12 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.20wmf11
17:48 logmsgbot: reedy Finished syncing Wikimedia installation... : rebuild 1.20wmf11 localisation cache
17:42 paravoid: powering down mw8; broken DIMM (#3499), has been flapping for days
17:32 paravoid: killing power via ipmi on ms-be7; broken hardware, stale data, should not be on rotation
17:18 logmsgbot: reedy Started syncing Wikimedia installation... : rebuild 1.20wmf11 localisation cache
17:16 paravoid: swift: distributing new ring; disable ms-be7, ms-be8, ms-be11 sdi, increase ms-be9/11/12 weight from 31 to 66
17:06 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.20wmf11 and rebuild localisation cache
15:59 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.20wmf11 and rebuild localisation cache
15:44 logmsgbot: reedy synchronized php-1.20wmf11/cache/
15:44 logmsgbot: reedy synchronized wmf-config/
15:42 Reedy: sync-docroot
15:41 logmsgbot: reedy synchronized php-1.20wmf11 'Initial sync-out'
15:09 notpeter: reinstalling all eqiad search frontends as precise
13:55 hashar: Sent one live hack made to 1.20wmf10 : https://gerrit.wikimedia.org/r/#/c/22461/
13:52 hashar: Rebased local 1.20wmf10 directory on top of its current origin/1.20wmf10 ( 472ec09 )
13:50 hashar: installed Gerrit commitmsg hook in /h/w/c/php-1.20wmf10/.git/hooks
13:36 logmsgbot: reedy synchronized php-1.20wmf10/resources/jquery/jquery.js 'jQuery 1.8.1'
09:43 hashar: manually cleaning out /tmp on gallium. It is being filled by faulty tests not cleaning the files they are creating.
02:18 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Mon Sep 3 02:18:28 UTC 2012

September 2

21:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
02:25 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Sun Sep 2 02:25:51 UTC 2012

September 1

02:49 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Deploy 22ceabed668b96cd88b48f6f7482c86a53645fb2'
02:49 logmsgbot: catrope synchronized wmf-config/filebackend.php 'Deploy 988ea2f71d84e1f62269f6e3d72eaf7b03206906'
02:49 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Deploy 988ea2f71d84e1f62269f6e3d72eaf7b03206906'
02:25 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Sat Sep 1 02:25:35 UTC 2012

August 31

22:04 Ryan_Lane: removing attribute uid from the attribute uniqueness plugin. It's the naming attribute and it's already ensured to be unique. It causes problems when it's also in the plugin.
21:54 Ryan_Lane: enabling attribute uniqueness plugin on virt0 and virt1000 for attributes: cn, uid, uidNumber
21:51 mutante: running rm -rf /var/lib/testswarm on gallium in a screen ..takes forever
21:51 Ryan_Lane: adding equality index to uidNumber on virt0 and virt1000 opendj
21:35 mutante: ms-be6 - just for the record, still could not mount same disks (sdn1, sdl1, sdi1, sdf1) after firmware upgrade
20:32 cmjohnson1: ms-be6 down for firmware upgrade
20:28 mutante: deleting /var/lib/testswarm on gallium (testswarm is gone)
12:11 logmsgbot: hashar synchronized wmf-config/CommonSettings.php '5c84fca $wgDBerrorLogInUTC -> $wgDBerrorLogTZ'
12:11 logmsgbot: hashar synchronized live-1.5/robots.php 'f0bf7a5 - disallow robots on beta'
12:11 hashar: forgot to pull before sync :-)
12:11 logmsgbot: hashar synchronized wmf-config/CommonSettings.php '5c84fca $wgDBerrorLogInUTC -> $wgDBerrorLogTZ'
12:10 logmsgbot: hashar synchronized live-1.5/robots.php 'f0bf7a5 - disallow robots on beta'
04:02 logmsgbot: tstarling synchronized php-1.20wmf10/tests/parser/parserTests.txt
04:01 logmsgbot: tstarling synchronized php-1.20wmf10/includes/specials/SpecialBlock.php
04:01 logmsgbot: tstarling synchronized php-1.20wmf10/includes/api/ApiFormatBase.php
04:00 logmsgbot: tstarling synchronized php-1.20wmf10/includes/Linker.php
04:00 logmsgbot: tstarling synchronized php-1.20wmf10/includes/User.php
04:00 logmsgbot: tstarling synchronized php-1.20wmf10/includes/AuthPlugin.php
03:59 logmsgbot: tstarling synchronized php-1.20wmf10/includes/DefaultSettings.php
02:59 paravoid: swift: removing ms-be8 sdc1, sdg1, sdk1, sdl1, sdn1 from ring; broken hardware (again)
02:18 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Fri Aug 31 02:18:02 UTC 2012

August 30

22:54 logmsgbot: maxsem synchronized php-1.20wmf10/includes/api/ApiMain.php 'WLM API logging https://gerrit.wikimedia.org/r/#/c/21748/'
22:27 Ryan_Lane: ran a maintenance script on labsconsole to update all instance pages
22:05 logmsgbot: kaldari Finished syncing Wikimedia installation... :
21:45 logmsgbot: kaldari Started syncing Wikimedia installation... :
21:17 notpeter: removing srv194 from apache pool
21:13 logmsgbot: olivneh synchronized php-1.20wmf10/extensions/E3Experiments/experiments
20:39 cmjohnson1: srv291 shutting down to reseat DIMM
20:39 cmjohnson1: srv291 shutting down to reseat DIMM
20:28 logmsgbot: krinkle synchronized live-1.5/static-1.20wmf10 'Creating static aliases for wmf10'
20:25 Krinkle: Created static-1.20wmf10 and extensions,resources,skins subdir symlinks in /home/wikipedia/commons/live-1.5, was not done by the script apparently
20:07 logmsgbot: reedy synchronized php-1.20wmf10/resources/jquery
20:02 mutante: reinstalling cp1024-cp1028
19:28 Ryan_Lane: finished upgrading labsconsole to 1.20wmf10
19:17 notpeter: all search traffic now pointed back to pmtpa
19:17 logmsgbot: py synchronized wmf-config/lucene.php
18:58 Ryan_Lane: upgrading labsconsole to 1.20wmf10
18:57 logmsgbot: py synchronized wmf-config/lucene.php
18:41 paravoid: deploying squid config all for thumb_handler.php addition to thumb_php
18:31 logmsgbot: preilly synchronized wmf-config/lucene.php
17:47 logmsgbot: aaron synchronized php-1.20wmf10/includes/filebackend/SwiftFileBackend.php 'deployed faec65ea23a8d17f08488950fccb0911f7b63086'
17:33 apergos: and rebooting ms-be7 again in the slim hope that we get back the missing drives... meh
17:32 logmsgbot: aaron synchronized php-1.20wmf10/includes/filebackend/SwiftFileBackend.php 'deployed 3bc61905b704096789376927b6cc3d7ca78b0670'
17:32 logmsgbot: aaron synchronized php-1.20wmf10/includes/filebackend/FSFileBackend.php 'deployed 3bc61905b704096789376927b6cc3d7ca78b0670'
16:45 notpeter: removing srv281 from apaches pool because it's a broken piece of garbage.
16:23 apergos: power cycle ms-be7 so it picks up the replaced drive with the right device name
15:46 cmjohnson1: mw8 shutting down for DIMM troubleshooting
14:09 RobH: authdns-update for helium/potassium (poolcounter servers)
10:53 hashar: gerrit-wm irc bot is apparently no more sending anything to #mediawiki :( IRC user has been inactive since midnight UTC. Opened bug 39797
03:35 Tim: restarting apaches to get php.ini change of pcre.backtrack_limit
02:18 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Thu Aug 30 02:18:56 UTC 2012
01:36 K4-713: updated the payments cluster to 95907e42e57
01:10 logmsgbot: aaron synchronized wmf-config/filebackend.php 'deployed fix for bogus var'
00:48 mutante: starting nagios on spence

August 29

23:02 logmsgbot: catrope synchronized wmf-config/filebackend.php 'Deploy d5b73b216317c705f58d1a1eef8bc0db21a90314'
21:25 logmsgbot: krinkle synchronized php-1.20wmf10/includes/EditPage.php '(bug 39584) Deploy MediaWiki:Previewnote bug fix (I2144f026d3b301edb521bf)'
21:12 logmsgbot: krinkle synchronized php-1.20wmf10/resources/ 'Update jquery.ui and jquery.effects to jQuery UI 1.8.23 (I35ed086e9a8c6d)'
20:29 Ryan_Lane: changing apache config and wiki location on virt0
20:27 logmsgbot: krinkle synchronized php-1.20wmf10/resources/jquery/jquery.js '(8dc8ee84) jQuery 1.8'
18:41 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: Move last remaining wiki to 1.20wmf10, so long 1.20wmf9
18:38 Ryan_Lane: deleting global role groups from LDAP. They aren't needed in keystone's LDAP DIT
18:00 mutante: reinstalling cp1022 and up
17:56 notpeter: returning srv281 to apaches pool
17:36 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: Switching all 'pedias to 1.20wmf10
17:11 notpeter: removing srv281 once again and reimaging
17:09 notpeter: returning srv281 to apaches pool
17:01 cmjohnson1: swapping out disk7 (dev/sdg) on ms-be7
16:36 logmsgbot: root Finished syncing Wikimedia installation... :
16:28 notpeter: removing srv281 from apaches pool
16:20 logmsgbot: root Started syncing Wikimedia installation... :
16:11 logmsgbot: mark synchronized wmf-config/mc.php
16:10 notpeter: enabling srv194 and srv281 in pybal for apache/precise live test
16:04 mark: Fixed serial console on hume
16:00 mark: Powering up fenari
15:49 mark: Powering up hume
15:48 mark: Final rsync finished.
13:37 mark: Mounted /home read-only on nfs1, started final rsync to nas1-a
13:28 mark: Changed /home mount from nfs1 to nas1-a on srv193 and spence
13:05 mark: Shutdown fenari and hume
12:48 hashar: reloading Jenkins to fix up Ext-TitleBlacklist misconfiguration
12:42 hashar: enabling Jenkins on TitleBacklist extension (job: Ext-TitleBlacklist )
08:55 logmsgbot: nikerabbit synchronized php-1.20wmf9/includes/Linker.php
08:54 logmsgbot: nikerabbit synchronized php-1.20wmf10/includes/Linker.php
02:45 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Wed Aug 29 02:45:36 UTC 2012
02:27 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Wed Aug 29 02:27:21 UTC 2012

August 28

23:45 LeslieCarr: there are some network issues with one of our ipv6 carriers, causing some european ipv6 users to have trouble reaching our US sites.
23:17 Ryan_Lane: fixed scripts for management home directories and gluster shares for labs by changing the project search filter to match keystone's LDAP DIT
21:44 logmsgbot: bsitu synchronized php-1.20wmf10/extensions/PageTriage/tools/cleanupPageTriagePageTags.php 'Fix the cleanup cron'
21:06 K4-713: updated the payments cluster to 9bcd8961fecb5
20:59 mutante: git pull and sync-file InitialiseSettings.php to cluster to push namespace fix on ar.wikimedia
20:59 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
20:52 logmsgbot: mlitn Finished syncing Wikimedia installation... : Updating ArticleFeedbackv5
20:16 logmsgbot: mlitn Started syncing Wikimedia installation... : Updating ArticleFeedbackv5
19:55 logmsgbot: catrope synchronized php-1.20wmf10/extensions/ProofreadPage 'Deploy 65fcb5984fd7d52ecd1887b248fda608b9dc4987'
19:53 logmsgbot: catrope synchronized php-1.20wmf10/tests/phpunit/includes/RevisionStorageTest.php 'Deploy e1cfe5ada99f3ed4594c5cfc33906538f4e02bdf'
18:52 mutante: rebooting cp1021 into PXE
17:41 paravoid: include debs for swift 1.5 and friends to apt (lucid/precise)
16:15 Jeff_Green: added payments100[1-4] & pay-lvs100[12] to nagios nsca config, restarted nagios on spence
14:03 logmsgbot: nikerabbit synchronized wmf-config/CommonSettings.php 'ttmserver part 2'
14:02 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'ttmserver part 1'
13:14 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Syncing 95c87a0b for AFT (mlitn said its ok)'
13:13 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Syncing f58063db'
13:11 logmsgbot: demon synchronized .gitignore 'Syncing .gitignore'
12:29 mark: Built varnish 3.0.3plus~rc1 packages and deployed them on cp1041 (mobile)
10:02 logmsgbot: nikerabbit synchronized php-1.20wmf10/extensions/Translate/ 'Picking up Solarium'
09:23 logmsgbot: nikerabbit synchronized php-1.20wmf10/extensions/wikihiero/ 'RTL fix'
04:01 Ryan_Lane: rebooting virt5
03:56 Ryan_Lane: rebooting virt4
03:41 Ryan_Lane: upgrading virt1 to precise
03:38 Ryan_Lane: upgrading virt4 to precise
03:37 Ryan_Lane: upgrading virt5 to precise
03:06 Ryan_Lane: bringing virt7 instances up
02:48 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Tue Aug 28 02:48:54 UTC 2012
02:45 Ryan_Lane: bringing virt8 instances up
02:37 logmsgbot: tstarling Finished syncing Wikimedia installation... :
02:30 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Tue Aug 28 02:30:01 UTC 2012
02:24 Ryan_Lane: force running puppet on virt7
02:24 Ryan_Lane: force running puppet on virt8
02:17 Ryan_Lane: rebooting virt8
02:17 Ryan_Lane: rebooting virt 7
02:08 logmsgbot: tstarling Started syncing Wikimedia installation... :
01:51 Ryan_Lane: upgrading virt8 to precise
01:48 logmsgbot: tstarling synchronized php-1.20wmf10/includes/GlobalFunctions.php
01:48 logmsgbot: tstarling synchronized php-1.20wmf10/thumb.php
01:47 logmsgbot: tstarling synchronized php-1.20wmf10/includes/upload/UploadFromUrl.php
01:46 Ryan_Lane: upgrading virt7 to precise
01:32 Ryan_Lane: bringing all instances up on virt6
01:01 Ryan_Lane: attempting to bring up some instances on virt6

August 27

23:31 Ryan_Lane: restarting virt6
23:08 Ryan_Lane: upgrading virt6 to precise. This action will cause downtime for a while.
22:49 mutante: fixed gmq.planet.wm updates
22:34 Ryan_Lane: rebooting virt3
22:24 Ryan_Lane: upgrading virt3 to precise
22:23 Ryan_Lane: disabling nova-compute on virt2
21:01 dschoon: ${whoami}
21:00 Ryan_Lane: enabling OpenStackManager and LdapAuthentication on labsconsole temporarily for testing
21:00 Ryan_Lane: enabling nova-compute on virt2 for testing instance creation
20:47 ottomata: starting puppet back up on emery
20:32 logmsgbot: aaron synchronized php-1.20wmf9/extensions/MobileFrontend 'deployed f2551fa97ba686101745d5cb97ef4678838cd3bb'
20:29 logmsgbot: aaron synchronized php-1.20wmf10/extensions/MobileFrontend 'deployed 75104cf6d3bc6d528d2b85e09761005081fb5cf4'
20:20 paravoid: deploying swift origs to all pmtpa backend squids
20:09 Ryan_Lane: force running puppet on virt2
20:04 Ryan_Lane: rebooting virt2
19:54 paravoid: deploying swift origs on sq51 (1h ago), sq52, sq53
19:54 Ryan_Lane: upgrading virt2 to precise
19:38 Ryan_Lane: upgrading OpenStackManager on labsconsole
19:33 Ryan_Lane: adding keystone services and endpoints
19:32 Ryan_Lane: upgrading nova, glance and keystone database schemas
19:18 ottomata: temporarily stopping puppet on emery to test log2udp AFT clicktracking relay to vanadium
19:06 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf10
18:41 Ryan_Lane: force running puppet on virt0
18:36 Ryan_Lane: restarting virt0
18:09 Ryan_Lane: upgrading virt0 to precise
17:54 Ryan_Lane: running a fresh backup of opendj on virt0
17:54 Ryan_Lane: backing up all databases on virt0
17:46 Ryan_Lane: stopping rabbitmq on virt0
17:43 Ryan_Lane: disabling openstack services on the virt infrastructure, that is
17:43 Ryan_Lane: disabling all openstack services
17:42 Ryan_Lane: disabling OpenStackManager and LdapAuthentication on labsconsole
17:37 Ryan_Lane: disabling puppet on all virt nodes
17:33 paravoid: rolled back sq51's config; depooled sq51 (10' ago)
17:18 paravoid: modified sq51 to serve origs from swift; added sq51 back to the pool
16:45 paravoid: depooling sq51 backend
15:55 paravoid: upgraded swift on ms-be12
15:54 paravoid: upgraded swift on ms-be5
15:47 paravoid: upgraded swift on ms-be11
14:51 paravoid: upgraded swift on ms-be3
14:40 paravoid: upgraded swift on ms-be4
14:30 paravoid: upgraded swift on ms-be10; (still depooled, no effect)
09:38 mark: Decommissioning bayes, shut it down
02:52 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Mon Aug 27 02:52:23 UTC 2012
02:27 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Mon Aug 27 02:27:16 UTC 2012
02:16 Tim: on spence: fixed /apache symlink, was pointing to a non-standard location, breaking sync scripts
01:32 logmsgbot: tstarling synchronized php-1.20wmf10/extensions/Scribunto/engines/LuaCommon/LuaCommon.php
01:32 logmsgbot: tstarling synchronized php-1.20wmf10/extensions/Scribunto/Scribunto.i18n.php
01:31 Tim: updating Scribunto
01:31 logmsgbot: tstarling synchronized php-1.20wmf10/extensions/Scribunto/Scribunto.namespaces.php

August 26

02:53 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Sun Aug 26 02:53:09 UTC 2012
02:27 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Sun Aug 26 02:27:02 UTC 2012

August 25

02:46 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Sat Aug 25 02:46:16 UTC 2012
02:27 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Sat Aug 25 02:27:08 UTC 2012

August 24

23:08 notpeter: temp stopping puppet on iron
23:02 maplebed: upgrading swift backend ms-be9
22:15 maplebed: upgrading swift backend ms-be7
21:54 maplebed: upgrading swift back end ms-be2
21:30 mutante: removing decom'ed srv206 and srv217 from ALL dsh groups
21:24 maplebed: upgrading swift backend host ms-be1
21:22 mutante: breaking test.wp :) ..and rolling back
21:11 maplebed: upgraded swift backend host ms-be8
20:10 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php (bug 34817)
19:57 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
19:01 RoanKattouw: (That was me)
19:01 logmsgbot: Restarted logmsgbot, was broken again
19:00 ottomata: starting puppet back up on brewster
18:58 mutante: sync-file ./wmf-config/InitialiseSettings.php
18:57 mutante: git pull in /h/w/common/wmf-config
18:32 ottomata: temporarily stopping puppet on brewster to test out partman change for analytics1023-1027
14:57 RobH: restart puppeted on brewster
14:48 RobH: analytics1023-1027 rotating down for removal of extra nic
14:28 RobH: stopping puppet on brewster to do a local nonpuppetized test change
13:08 paravoid: repooling ms-fe1/2 with all new swift
12:40 apergos: on ms5, running from screen session as root: tossing non-standard thumb sizes > 100 px for commons/x/xx to see what space that gives us
12:32 paravoid: depooling ms-fe1/2 for the 1.5 upgrade
02:47 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Fri Aug 24 02:47:46 UTC 2012
02:26 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Fri Aug 24 02:26:10 UTC 2012
02:08 logmsgbot: kaldari Finished syncing Wikimedia installation... :
01:55 logmsgbot: kaldari Started syncing Wikimedia installation... :

August 23

23:56 logmsgbot: kaldari synchronized php-1.20wmf10/extensions/UploadWizard
23:49 K4-713: updated paypal IPN to r2028
22:40 maplebed: putting ms-fe3 and 4 into rotation running swift v1.5.0-3
22:40 K4-713: updated production civicrm to r2024
22:06 logmsgbot: kaldari Finished syncing Wikimedia installation... :
21:53 logmsgbot: kaldari Started syncing Wikimedia installation... :
21:33 logmsgbot: aaron synchronized php-1.20wmf10/extensions/SwiftCloudFiles 'deployed de4a83af7750750fcaa49e81b7faa771c8601546'
21:32 logmsgbot: aaron synchronized php-1.20wmf9/extensions/SwiftCloudFiles 'deployed de4a83af7750750fcaa49e81b7faa771c8601546'
20:42 logmsgbot: olivneh synchronized php-1.20wmf9/extensions/E3Experiments/experiments/trackEdits.js
20:40 logmsgbot: olivneh synchronized php-1.20wmf9/extensions/E3Experiments/lib/e3.js
20:35 logmsgbot: kaldari Finished syncing Wikimedia installation... :
20:12 logmsgbot: kaldari Started syncing Wikimedia installation... :
18:44 maplebed: rebooting ms-be3
18:34 apergos: deleting about 200k thumb dirs and their contents from ms5, unused on any project, covering june and july uploads
18:06 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Disable auth caching for testwiki'
17:40 logmsgbot: aaron synchronized php-1.20wmf10/includes/filebackend/FSFileBackend.php 'debug logging.'
17:21 logmsgbot: aaron synchronized wmf-config/PrivateSettings.php 'Make testwiki use ms-fe4'
17:09 logmsgbot: aaron synchronized wmf-config/filebackend.php
16:54 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Only write thumbnails to the master backend.'
16:41 logmsgbot: aaron synchronized php-1.20wmf10/includes/filebackend/FileBackendMultiWrite.php 'deployed 36c0c1edb6e713a797e7fde52f716ca17f548866'
16:40 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/backend/FileBackendMultiWrite.php 'deployed 36c0c1edb6e713a797e7fde52f716ca17f548866'
11:46 paravoid: depooling ms-fe3 to stage 1.5 upgrade
11:27 paravoid: depooling ms-fe4 to stage 1.5 upgrade
04:21 logmsgbot: aaron synchronized php-1.20wmf9/includes/media/MediaTransformOutput.php 'deployed ed9d124c37f57ab7d7c9678908af54363290b1f4'
04:20 logmsgbot: aaron synchronized php-1.20wmf10/includes/media/MediaTransformOutput.php 'deployed ed9d124c37f57ab7d7c9678908af54363290b1f4'
04:12 logmsgbot: aaron synchronized php-1.20wmf10/includes/media/MediaTransformOutput.php
03:49 logmsgbot: aaron synchronized php-1.20wmf10/includes/StreamFile.php
03:38 logmsgbot: aaron synchronized php-1.20wmf10/includes/StreamFile.php 'debug logging'
02:53 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Thu Aug 23 02:53:10 UTC 2012
02:27 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Thu Aug 23 02:27:31 UTC 2012
00:04 logmsgbot: asher synchronized wmf-config/db.php 'returning db35 to s5'

August 22

23:54 binasher: rebooting db35 for kernel upgrade
23:50 binasher: new s5 master repl pos - MASTER_LOG_FILE='db45-bin.000187', MASTER_LOG_POS=807116168
23:49 logmsgbot: asher synchronized wmf-config/db.php 'setting s5 to writeable'
23:47 logmsgbot: asher synchronized wmf-config/db.php 'setting s5 to read-only and switching master to db45, return db39 to s3'
23:47 binasher: preparing to rotate the s5 master to db45 (last of these for now)
23:36 mutante: check_job_queue check fixed after Tim synced spence. zhwiki has large queue: zhwiki (118614). Nagios did not talk because it was a transition from one critical to another
23:26 Tim: added spence to the mediawiki-installation dsh group
23:25 binasher: rebooting db39 for kernel upgrade
23:19 binasher: new s3 master repl pos: MASTER_LOG_FILE='db34-bin.000864', MASTER_LOG_POS=466194762
23:18 logmsgbot: asher synchronized wmf-config/db.php 'setting s3 to writeable'
23:16 logmsgbot: asher synchronized wmf-config/db.php 'setting s3 to read-only, switching master to db34'
23:15 binasher: preparing to rotate the s3 master to db34
23:14 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Restored math directory to its intended value.'
23:13 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'Restored math directory to its intended value.'
23:06 K4-713: updated production civicrm to r2013
22:42 mutante: rebooting srv208,srv238
22:35 mutante: re-enabling puppet on srv208 and srv238, dist-upgrading while at it
22:26 K4-713: updated production civicrm to r2011
22:10 mutante: srv208,srv238 - puppet running but it's not. looks like puppet is disabled using puppet agent --disable, on purpose?
21:39 logmsgbot: catrope synchronized php-1.20wmf10/resources 'jQuery revert'
21:03 mutante: adding Bugzilla https monitor to watchmouse
20:53 mutante: dist-upgrading kaulen, new kernel (not rebooted yet), fixed SSL cert path in puppet, restart puppet
20:37 mutante: stopped puppet on kaulen. manually fixed path to SSL cert so Apache would start. Bugzilla back for now
20:32 mutante: powercycling kaulen. OOM death.
20:29 mutante: kaulen unresponsive, bugzilla affected
20:26 logmsgbot: aaron synchronized php-1.20wmf9/extensions/SwiftCloudFiles/php-cloudfiles-wmf/cloudfiles.php 'deployed 8fea21b10eaba28773c30c6952fc792f08bf09a8'
20:24 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Re-enabled swift token caching.'
20:20 logmsgbot: aaron synchronized php-1.20wmf10/extensions/SwiftCloudFiles/php-cloudfiles-wmf/cloudfiles.php 'deployed 8fea21b10eaba28773c30c6952fc792f08bf09a8'
19:17 logmsgbot: aaron synchronized php-1.20wmf10/includes/Revision.php
19:13 logmsgbot: aaron synchronized php-1.20wmf10/includes/Revision.php 'bit logic is fun'
19:11 logmsgbot: aaron synchronized php-1.20wmf10/includes/Revision.php 'debug logging'
19:08 mutante: reinstalling zirconium to use LVM
19:05 logmsgbot: aaron synchronized php-1.20wmf10/includes/Revision.php 'debug logging'
18:55 logmsgbot: Restarted ircecho on fenari, logmsgbot was broken --Roan
18:52 RoanKattouw: 1.20wmf10 deployed to all non-Wikipedia projects around 20:04 UTC
17:44 mutante: installing package upgrades on bast1001
17:19 paravoid: stopped supervisord & puppet on silver/zhen per preilly's request
17:00 apergos: oh, for people following along, shot the delete job. apparently even deletes is enough to push ms5 over the edge, yay
16:50 apergos: removing thumbs on ms5 unused on any project, for june-july uploads. running in screen as root
16:37 ^demon: bugzilla went down, but came back up. magic?
16:35 Jeff_Green: restarted ns1 b/c it fell down after authdns-update
16:33 Jeff_Green: adding dns for pay-lvs100[12]
16:32 paravoid: deploy new squid config that removes ms5 from a swift fallback for thumbs
16:06 cmjohnson1: ms-be6 powering down for memory testing
15:32 notpeter: rebooting srv193, upgrade to precise
11:38 mark: Removed /homewmf NFS mount on kaulen
06:14 binasher: new s1 repl position: MASTER_LOG_FILE='db63-bin.000049', MASTER_LOG_POS=646903534
06:14 logmsgbot: asher synchronized wmf-config/db.php 'returing db38 as an s1 slave'
06:02 binasher: shutting down mysql on db38 and rebooting for kernel upgrade
05:50 logmsgbot: asher synchronized wmf-config/db.php 'setting s1 writeable on new master'
05:49 logmsgbot: asher synchronized wmf-config/db.php 'setting s1 to read-only, switching master to db63'
05:47 binasher: preparing to switch s1 master to db63 (upgrade to 96GB ram and precise) in a minute
05:29 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'enabling Scribunto on mediawiki.org'
05:07 Tim: on ms7, running a script to remerge the split math directories back into /mnt/upload6/math, using rsync
02:55 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Wed Aug 22 02:55:22 UTC 2012
02:27 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Wed Aug 22 02:27:55 UTC 2012
00:19 Tim: running updateCollation.php for ptwiki in a screen on hume
00:15 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'setting $wgCategoryCollation=uca-default on ptwiki'
00:00 binasher: completed user.user_email index migrations

August 21

23:46 maplebed: deployed change to swift proxy rewrite to only send thumb 404s to the image scalers gerrit change 20916
23:44 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Enabled token/url caching for swift.'
23:26 Tim: updated email address for enwiki user account "Giano"
23:17 binasher: running user.user_email index migration on s6-7
23:16 binasher: running user.user_email index migration on s5
23:14 binasher: truncated enwiki.flagged(images|templates)
23:10 binasher: running user.user_email index migration on s4
22:44 logmsgbot: asher synchronized wmf-config/db.php 'returning db52 as an s2 slave'
22:24 binasher: s2 master swap - new repl position MASTER_LOG_FILE='db54-bin.000302', MASTER_LOG_POS=870283287
22:23 logmsgbot: asher synchronized wmf-config/db.php 'setting s2 to writeable'
22:22 logmsgbot: asher synchronized wmf-config/db.php 'setting s2 to read-only for master swap to db54'
22:09 logmsgbot: reedy synchronized php-1.20wmf10/extensions/EducationProgram
21:59 binasher: running purgeParserCache.php for all dbs
21:26 binasher: resumed s3 user.user_email migrations
20:34 Jeff_Green: hand-tweezed spence:/etc/nagios/nsca_payments.cfg b/c puppet is broken for spence, restarted nagios on spence
20:15 Jeff_Green: added hosts boron/indium to nagios's nsca config
19:39 logmsgbot: reedy synchronized php-1.20wmf10/resources/ 'jQuery UI 1.8.23'
19:26 paravoid: rolling back sq51 switch to originals
18:52 paravoid: putting sq51 (modified with originals pointing to swift) back into the backend pool
18:33 RobH: authdns-update for mc1009-1016 mgmt
18:16 RobH: db1047 back online, presends sdb and sdc at 3597.3 GB each
18:11 RobH: db1047 disk shelf added, two raid10 arrays of 12 disks each, booting now into OS.
18:00 RobH: db1047 shutting down
17:58 RobH: db1047 mysql shutting down (going to add new disk shelf to server)
16:44 AaronSchulz: Running setZoneAccess.php for all wikis
16:40 apergos: shot morebots so the restarter script would restart it
05:41 binasher: halting s3 user migration on hiwiktionary, will resume in the morning (pst)
04:02 binasher: adding user.user_email index on s3
03:44 binasher: adding user.user_email index on s2
02:37 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Tue Aug 21 02:37:10 UTC 2012
02:21 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Tue Aug 21 02:21:32 UTC 2012
01:47 binasher: adding user.user_email index to enwiki via osc
00:32 maplebed: removed ms-be6 from rotation due to bad memory and repeated kernel panics
00:23 mutante: fenari - install a couple lib* package upgrades
00:10 logmsgbot: preilly synchronized php-1.20wmf10/extensions/MobileFrontend 'weekly deploy'
00:09 logmsgbot: preilly synchronized php-1.20wmf9/extensions/MobileFrontend 'weekly deploy'

August 20

23:48 logmsgbot: kaldari Finished syncing Wikimedia installation... :
23:46 logmsgbot: tstarling synchronized php-1.20wmf9/maintenance/benchmarks/obama.php
23:18 logmsgbot: kaldari Started syncing Wikimedia installation... :
22:50 logmsgbot: reedy synchronized php-1.20wmf10/extensions/VisualEditor/
22:49 logmsgbot: reedy synchronized php-1.20wmf10/includes/GlobalFunctions.php
22:15 logmsgbot: reedy synchronized php-1.20wmf10/skins/
21:33 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Enabling file uploads on testwiki for regular user accounts (exception to bug 12556)'
20:26 Jeff_Green: fixing typo in pdns-templates/wmnet and reloading nameservers
18:57 logmsgbot: catrope synchronized php-1.20wmf10/extensions/VisualEditor/ 'Actually push out VE fixes'
18:52 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Update VE config for Parsoid prefix change'
18:27 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki and mediawikiwiki to 1.20wmf10
18:22 paravoid: cleaning sq51's cache, poisoned after swift changes staging
16:06 logmsgbot: reedy synchronized php-1.20wmf10/cache/l10n/
16:04 mark: Upgrading JUNOS on asw2-a5-eqiad to 11.4R2.14
15:49 mark: Upgrading JUNOS on asw2-d3-sdtpa to 11.4R2.14
15:47 logmsgbot: reedy Started syncing Wikimedia installation... :
15:38 logmsgbot: reedy Finished syncing Wikimedia installation... : testwiki to 1.20wmf10 and rebuild localisation cache for it..
15:01 logmsgbot: reedy Started syncing Wikimedia installation... : testwiki to 1.20wmf10 and rebuild localisation cache for it..
14:46 logmsgbot: reedy synchronized php-1.20wmf10/
14:39 logmsgbot: reedy synchronized wmf-config/
14:38 logmsgbot: reedy synchronized php-1.20wmf10/
14:36 logmsgbot: reedy synchronized live-1.5/
14:30 Reedy: Removing php-1.20wmf7 from mediawiki-installation group
14:26 logmsgbot: reedy synchronized php-1.20wmf10/ 'staging 1.20wmf10 directory'
09:49 paravoid: powering off mw8, faulty (#3425), has been flapping a lot
09:24 paravoid: restarted gmond in cp3002
02:28 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Mon Aug 20 02:28:20 UTC 2012

August 19

20:54 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
02:24 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Sun Aug 19 02:24:13 UTC 2012

August 18

03:46 Tim: copying old logs from bast1001 to fluorine
02:28 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Sat Aug 18 02:28:03 UTC 2012

August 17

23:17 Ryan_Lane: added cn=essextest project to labs LDAP for essex testing. this project isn't compatible with diablo
22:16 notpeter: rebooting/reimaging zhen and silver for upgrade to precise
22:01 mutante: rebuilding wikitech wiki l10n cache
22:00 mutante: rebuilding wikitech-l archives
20:15 Reedy: Pointing php to php-1.20wmf9
20:13 Reedy: Deleting php-1.20wmf8/cache/l10n from mediawiki-installation
17:48 RobH: once all apaches update, will migrate old logs from nfs to fluorine
17:48 RobH: apache logs confirmed hitting fluorine, all will transition over as they puppet update
17:24 logmsgbot: robh synchronized wmf-config/InitialiseSettings.php
17:24 logmsgbot: robh synchronized wmf-config/CommonSettings.php
17:23 RobH: moving apache logs back to fluorine
17:00 cmjohnson1: shutting down search23 to repair DIMM
16:59 cmjohnson1: mc1 shutting down to repair DIMM
16:31 mark: Enabled all 3 new 802.3ad aggregated links for asw-d1-sdtpa to asw-d3-sdtpa
16:15 RobH: authdns-update to correct typo, nothing really broke badly, i r lucky
15:54 RobH: fluorine now coming down for reinstall, all log files backed up to nfs
15:47 mark: Created static 802.3ad trunks on csw1-sdtpa for asw-d1-sdtpa to asw-d3-sdtpa uplinks, deployed with only the existing port enabled
15:44 mark: Created static 802.3ad trunks on asw-d1-sdtpa and asw-d2-sdtpa for uplinks
15:31 RobH: puppet is taking forever on nagios, as soon as it completes, will remove fluorine from decom and rerun
15:11 RobH: authdns-update for fluorine ip update
15:04 RobH: lvm restructure
15:04 RobH: all data copied, all traffic moved, fluorine coming down for reinstallation into internal vlan and lvs restructure
15:03 RobH: all log traffic appears to have moved back to nfs1, copying fluorine logs back to nfs for reformat
14:53 mark: Setup static 802.3ad trunk on asw-d3-sdtpa with one port up
14:40 cmjohnson1: swapping disk3 db44
14:27 logmsgbot: robh synchronized wmf-config/InitialiseSettings.php
14:27 logmsgbot: robh synchronized wmf-config/CommonSettings.php
13:48 logmsgbot: robh ran sync-common-all
13:34 RobH: moving apache logs from pushing to fluorine back to nfs1 for now
05:12 Tim: queued refreshlinks jobs for 1330 commonswiki image description pages with coord templates which weren't refreshed in the previous manual attempt, probably due to failed jobs
03:34 logmsgbot: aaron synchronized php-1.20wmf9/includes/media/DjVu.php 'deployed 3ae632c3c9111b16f019574bcafec58fa3ee408d'
03:34 logmsgbot: aaron synchronized php-1.20wmf9/includes/media/SVG.php 'deployed 3ae632c3c9111b16f019574bcafec58fa3ee408d '
03:16 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/backend/FileBackendStore.php
03:14 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/backend/FileBackendStore.php
03:10 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/backend/FileBackendStore.php 'Profiling'
03:08 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/backend/FileBackendStore.php 'Profiling'
02:42 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/backend/SwiftFileBackend.php 'Profiling'
02:37 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Fri Aug 17 02:37:38 UTC 2012
01:12 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/backend/TempFSFile.php 'deployed 08ea8adc95a4edba0b1f2f13c34f2a57725f1a58'
01:10 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/file/File.php 'deployed 08ea8adc95a4edba0b1f2f13c34f2a57725f1a58'
00:13 Tim: copying MW log archives from NFS to fluorine via rsync on fenari

August 16

23:39 Tim: on fluorine: chown'd log files and restarted udp2log to get new uid
23:39 binasher: reslaved es1002 after conversion to innodb
23:31 logmsgbot: olivneh synchronized php-1.20wmf9/extensions/E3Experiments/Experiments.hooks.php
23:26 mutante: fix date/NTP on sq36
23:14 logmsgbot: kaldari Finished syncing Wikimedia installation... :
23:12 mutante: killing nagios-wm. it stopped talking even though stuff gets written to --infile
22:50 logmsgbot: kaldari Started syncing Wikimedia installation... :
22:25 logmsgbot: preilly Finished syncing Wikimedia installation... : update mobile frontend
22:04 logmsgbot: preilly Started syncing Wikimedia installation... : update mobile frontend
22:03 notpeter: restarting puppet on nfs1/2
21:54 notpeter: restarting puppet on locke
21:51 notpeter: restarting puppet on emery
21:39 notpeter: restarting puppet on oxygen
21:37 cmjohnson1: shutting ms-be6 down for hardware testing/replacing
21:26 notpeter: stopping puppet on nfs2 and restarting both udp2log instances to switch user. will keep puppet off until I merge new init script into puppet
21:20 notpeter: stopping puppet on nfs1 and restarting both udp2log instances to switch user. will keep puppet off until I merge new init script into puppet
21:12 maplebed: ms-be6 down for bad disk repair
20:58 Ryan_Lane: attempting to upgrade virt1000 to precise
20:47 cmjohnson1: ms-be6 shutting down to replace 2 disks
20:38 notpeter: stopping puppet on locke and restarting both udp2log instances to switch user. will keep puppet off until I merge new init script into puppet
20:12 RobH: ms-be1004 is ready to be installed into service
19:57 RobH: ms-be1004 down for ssd install
19:44 notpeter: stopping puppet on emery and restarting both udp2log instances to switch user. will keep puppet off until I merge new init script into puppet
19:34 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Set $readFromSwiftShared = true;'
19:30 logmsgbot: aaron synchronized wmf-config/filebackend.php
19:29 notpeter: stopping puppet on oxygen and restarting both udp2log instances to switch user. will keep puppet off until I merge new init script into puppet
19:23 logmsgbot: aaron synchronized wmf-config/filebackend.php
19:17 binasher: running purgeParserCache.php again
19:05 RobH: puppet restarted on brewster
18:58 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Fixed container mapping for shared repo'
18:28 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Use NFS for reads for shared-multiwrite'
18:14 logmsgbot: aaron synchronized wmf-config/swift.php 'Disabled redundant thumb copy/purge hooks.'
18:01 RobH: authdns-update for new es1005-1010
17:59 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Make internal reads come from swift for all wikis'
17:50 apergos: powercycled ms-be3 ... in theory
16:52 cmjohnson1: shutting down mw18 and mw19 to replace DIMM
16:29 ottomata: Stopping puppet on brewster again. Sigh. PARTMAAAAN!
16:22 logmsgbot: reedy synchronized php-1.20wmf9/includes/QueryPage.php
16:07 cmjohnson1: shutting down mw8 to replace bad DIMM
15:58 ottomata: starting puppet on brewster. (Woo, partman looking better!)
14:15 ottomata: stopping puppet on brewster to continue partman troubleshooting for analytics dells
13:26 mark: Enabled and started SIS deduplication on home_pmtpa on nas1-a
12:05 mark: Removed OSPF metric on xe-5/2/1.0 on cr2-eqiad, to move eqiad->pmtpa traffic to the lower latency link
11:27 pp-pdf2: restarted all services
11:27 pp-pdf1: restarted all services
11:27 pp-pdf3: restarted all services
11:26 pp-pdf3: updated mwlib.epub to 0.14.1
11:26 pp-pdf2: updated mwlib.epub to 0.14.1
11:25 pp-pdf1: updated mwlib.epub to 0.14.1
10:50 mark: Setup semi-sync snapmirror from nas1-a:home_pmtpa to nas1001-a:home_pmtpa
03:57 mutante: nagios back up after adding missing monitor groups misc_pmtpa in appserver role (srv194)
02:56 K4-713: synchronized payments cluster to 37e31eddf5a4
02:28 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Thu Aug 16 02:28:33 UTC 2012
02:12 Ryan_Lane: pushed in large puppet change for ldap, openstack, gerrit and ldap pdns to make it more modular and to add support for eqiad region
02:10 Ryan_Lane: fixed arecord issues with labsconsole by adding an exception handling live hack for the jobs
01:21 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'remove wikimedia philippines event exemption'
00:40 mutante: installing package upgrades on singer
00:36 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/backend/FileBackendMultiWrite.php 'deployed c51a9a288b6dd5c0023a77f324c04707b23501c6'
00:24 binasher: running "mwscript purgeParserCache.php --wiki=$db --age=1814400" instead
00:19 binasher: running mwscript purgeParserCache.php --wiki=enwiki --age=1209600
00:02 RoanKattouw: find /export/upload/wik*/*/{0,1,2,3,4,5,6,7,8,9,a,b,c,d,e,f,archive,math,temp,timeline} ! -user apache -exec /root/fixownership2 \{\} \; where fixownership2 = chown -h apache $1
00:02 RoanKattouw: Restarting the find-chown, this time with -h so symlinks are handled correctly (for some reason there are a bunch of broken symlinks with weird characters out there...)

August 15

23:52 binasher: powercycled srv259
23:44 mutante: brought nagios back up by removing "misc_pmtpa" host and servicegroups from srv194 manually
23:35 mutante: restarted pdns-recursor on ns0
23:11 mutante: authdns-update, moving zirconium to public IP and removing selenium
22:33 logmsgbot: reedy synchronized php-1.20wmf9/extensions/OpenSearchXml/
21:19 RoanKattouw: find /export/upload/wik*/*/{0,1,2,3,4,5,6,7,8,9,a,b,c,d,e,f,archive,math,temp,timeline} ! -user apache -exec chown apache \{\} \;
21:17 RoanKattouw: chown on ms7 has finished, running another find to find remaining bad files
20:57 K4-713: synchronized payments cluster to 79275e152b76f
20:26 ottomata: staring puppet on brewster. PARTMAN IS THE ENEMY. will continue the fight tomorrow.
20:22 mutante: http://en.wikipedia.org/w/ now redirects to Main_page on all languages. It was a 403 before.
20:20 mutante: apache-graceful-all
20:15 mutante: sync-apache pushing out new redirects from /w/ to Main_Page
20:04 ottomata: stopping puppet on brewster to experiment with partman on new analytics dells
19:29 mark: Removed OSPF/OSPFv3 metric 60 on cr1-sdtpa:xe-1/1/0 (eqiad link)
19:17 mark: Moved VRRP mastership from cr2-pmtpa to cr1-sdtpa by reassigning VRRP priorities, relieving the sdtpa-pmtpa link
18:44 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Remaining wikis over to 1.20wmf9
17:11 maplebed: ms-be1 dropped offline yesterday; powercycling.
16:11 mark: Setup SnapMirror replication from nas1-a:fr_archive to nas1001-a:fr_archive
16:10 mark: Created 4 TB volumes 'fr_archive' in aggregate prod1 on both nas1-a nad nas1001-a
16:10 mark: Increased aggregate 'prod1' by 4 drives on both nas1-a and nas1001-a
16:06 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Wed Aug 15 16:06:26 UTC 2012
15:40 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Wed Aug 15 15:40:21 UTC 2012
14:33 cmjohnson1: authdns-update for es9 and es10
13:46 RobH: authdns-update for wikipedia.co.za
12:15 mark: Started rsync of /home to nas1-a:/vol/home_pmtpa on nfs1
12:11 mark: Restricted NFS mounting of /vol/root on all NetApps
11:39 mark: Destroyed aggregate 'labs' on nas1-a
10:00 Tim: deploying bugfix version of wmerrors
09:51 Tim: moved old MW debug logs out of /home/wikipedia/logs to avoid confusion, live logs are now on fluorine
05:22 Tim: moving MediaWiki logs to fluorine
05:17 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
05:15 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
04:38 Tim: added rate limit exemption for 27.108.200.111
04:38 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
03:00 Tim: built a udplog package for precise and uploaded it
02:35 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'enabling Scribunto and CodeEditor on test2wiki'
02:28 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
02:15 logmsgbot: tstarling Finished syncing Wikimedia installation... :
02:01 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
01:46 logmsgbot: tstarling Started syncing Wikimedia installation... :
01:00 logmsgbot: tstarling Started syncing Wikimedia installation... :
00:47 logmsgbot: tstarling Started syncing Wikimedia installation... :
00:39 Tim: running scap to for extension-list update

August 14

23:53 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Move VE Parsoid from cadmium to wtp1'
21:25 logmsgbot: kaldari synchronized php-1.20wmf9/extensions/PageTriage/tools/cleanupPageTriage.php 'adding PageTriage maintainence script'
21:05 logmsgbot: kaldari Finished syncing Wikimedia installation... :
20:45 logmsgbot: kaldari Started syncing Wikimedia installation... :
20:38 mutante: killed duplicate nagios-wm
20:20 cmjohnson1: db44 swapping out disk1 rt 3424
20:08 ottomata: installed precise and puppetized stat1001
19:26 cmjohnson1: search23 shutting down to replace DIMM A2 rt 3423
19:00 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Deploying https://gerrit.wikimedia.org/r/19399'
18:52 RoanKattouw: Running /root/fixownership < /root/badownershipfiles in a screen on ms7
18:14 notpeter: stopping puppet on brewster for partman experimentation
17:34 logmsgbot: catrope synchronized php-1.20wmf9/.gitmodules
17:33 RoanKattouw: Yesterday's find on ms7 is done, found 1.8 million files with bad ownership. Will run a batch chown on them soon
17:32 logmsgbot: catrope synchronized php-1.20wmf9/extensions/UploadWizard 'Deploy 9d9189458ad8d966d952d295918847f00e304cb5'
09:17 logmsgbot: nikerabbit synchronized php-1.20wmf9/extensions/Translate/ 'I18ndeploy: Translate bugs'
07:46 logmsgbot: tstarling Finished syncing Wikimedia installation... :
07:07 logmsgbot: tstarling Started syncing Wikimedia installation... :
06:48 Tim: running scap to push out CodeEditor and Scribunto sources, not enabled yet
05:45 logmsgbot: olivneh synchronized php-1.20wmf9/extensions/E3Experiments/ 'Update E3Experiments to master'
03:34 Tim: restarting all apaches to get LuaSandbox extension
03:26 Tim: on mw8: removed broken package php5-memcached
02:45 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Tue Aug 14 02:45:30 UTC 2012
02:28 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Tue Aug 14 02:28:13 UTC 2012
01:06 Tim: restarting apaches to get new wmerrors configuration

August 13

23:53 Tim: uploaded a new version of the php5-wmerrors package for lucid and precise
23:18 mutante: removed #wikimedia-tech from /usr/local/bin/start-nagios-bot
23:01 mutante: killing nagios-wm process which isn't supposed to be on #wikimedia-tech anymore
21:58 cmjohnson1: running authdns-update for labsdb1 & 2
21:19 RoanKattouw: ...piping result into ms7:/root/badownershipfiles
21:18 RoanKattouw: ms7# find /export/upload/wik*/*/{0,1,2,3,4,5,6,7,8,9,a,b,c,d,e,f,archive,math,temp,timeline} ! -user apache
21:17 RoanKattouw: Searching for files with wrong ownership on ms7 using
21:13 RobH: reinstalling virt1004 to see if the raid issue re-appears
20:55 notpeter: updating root auth keys file on ms7 by hand.
20:36 logmsgbot: catrope synchronized php-1.20wmf9/resources/jquery/jquery.tablesorter.js 'Deploy 1fafaef3aa157f49cbbf4b9cffb03544bcf72a08'
20:04 preilly: updating zero for custom banner colors
20:03 logmsgbot: preilly synchronized php-1.20wmf9/extensions/ZeroRatedMobileAccess 'update for landing page'
20:03 logmsgbot: preilly synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess 'update for landing page'
19:42 logmsgbot: preilly synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess 'update for landing page'
19:41 preilly: update to zero for banner text link
19:41 logmsgbot: preilly synchronized php-1.20wmf9/extensions/ZeroRatedMobileAccess 'update for landing page'
19:36 logmsgbot: preilly synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess 'update for landing page'
19:32 RobH: powering off analytics1011-1014 to remove the add on nics (disrupting installer)
18:25 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved enwiki to 1.20wmf9
17:21 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Switched backend reads to swift for testwikis and mw.org'
15:18 paravoid: reprovisioning srv281 with a new fs layout
13:05 mark: Doing NDU of IOM3 shelf firmware on nas1001-a
12:59 mark: Doing NDU of IOM3 shelf firmware on nas1-a
12:47 mark: Setup NTP on all NetApps
12:24 mark: Restored SnapMirror relationship between nas1-a and nas1001-a, 70 days lagged, transfer initiated
12:04 paravoid: apt: include php-memcached 2.0.1-6~wmf+lucid2 @ lucid-wikimedia
11:58 paravoid: apt: include php5 5.3.10-1ubuntu3.2+wmf1 @ precise-wikimedia
11:08 paravoid: apt: remove wikidiff2/0.0.1wm1 (obsolete), copysrc php-wikidiff2 {lucid,precise}-wikimedia
11:08 paravoid: apt: remove php-wikidiff2/1.1.0-2 from lucid-wikimedia/universe (lucid-wikimedia/main has 1.1.2-1)
10:49 apergos: starting initial rsync from ms8 to ms10 (should take approximately forever)
06:26 Tim: uploaded czmq and zpubsub packages to brewster for Ori
02:37 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Mon Aug 13 02:37:53 UTC 2012
02:20 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Mon Aug 13 02:20:27 UTC 2012

August 12

03:56 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Sun Aug 12 03:56:33 UTC 2012
03:38 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Sun Aug 12 03:38:33 UTC 2012
03:19 Reedy: Running localisation update manually in screen session on fenari
02:00 logmsgbot: LocalisationUpdate failed: git pull of extensions failed

August 11

02:01 logmsgbot: LocalisationUpdate failed: git pull of extensions failed

August 10

21:54 Ryan_Lane: live hacked nova on virt2 with parts of nova git commit 4584e552a653904c36cf04cb295a7bf09d2def28
21:22 logmsgbot: preilly synchronized php-1.20wmf9/extensions/ZeroRatedMobileAccess 'update for landing page'
21:22 logmsgbot: preilly synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess 'update for landing page'
21:21 preilly: update to zero needed to support carrier testing
21:21 logmsgbot: preilly synchronized php-1.20wmf9/extensions/MobileFrontend 'update for zero'
21:21 preilly: update to zero needed to support carrier testing
21:20 logmsgbot: preilly synchronized php-1.20wmf8/extensions/MobileFrontend 'update for zero'
18:42 Jeff_Green: adding frack.eqiad.wmnet hosts to DNS (wmnet and 10..in-addr.arpa)
18:11 K4-713: Payments cluster updated to 133b2b6fb5
17:07 notpeter: temporarily stopping puppet on brewster
16:43 cmjohnson1: enabling pxe boot for 10GB Nic cards mc1-16
16:32 mark: Started OnTap upgrade to 8.1 on nas1-a and nas1-b
14:39 mark: OnTap upgrade on nas1001-a and -b complete
14:11 mark: Started OnTap upgrade to 8.1 on nas1001-b
13:55 mark: Started OnTap upgrade to 8.1 on nas1001-a
13:10 mark: Upgraded SP firmware to 1.3 on nas1001-a and nas1001-b
11:55 mark: nas1001-b disk fw upgrade completed
11:51 mark: nas1001-a disk firmware upgrade completed. Started disk firmware upgrade on nas1001-b
11:46 mark: Started disk firmware update on nas1001-a
02:39 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Fri Aug 10 02:39:57 UTC 2012
02:22 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Fri Aug 10 02:22:22 UTC 2012
01:07 Tim: on virt0: enabling mysql general log to debug "detached session" errors on nova
00:05 Tim: added myself as a labsconsole sysop, added myself to the packaging project, created a "php" instance for building php-related packages

August 9

22:05 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Added swift to multiwrite backend for all remaining wikis'
22:02 RobH: db1046 back online
21:40 RobH: shutting down db1046 to migrate its position in the rack
21:27 logmsgbot: olivneh synchronized php-1.20wmf8/extensions/E3Experiments/experiments/communityClicks.js
21:20 binasher: resumed enwiki replication in eqiad
21:13 binasher: stopping mysql on db1017 for a minute (and all enwiki eqiad replication with it)
20:52 logmsgbot: kaldari synchronized wmf-config/extension-list 'Removing deprecated CustomUserSignup extension'
20:52 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'Removing deprecated CustomUserSignup extension and turning on Curation Toolbar for en.wiki'
20:52 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'Removing deprecated CustomUserSignup extension'
20:09 logmsgbot: kaldari Finished syncing Wikimedia installation... :
19:48 logmsgbot: catrope synchronized php-1.20wmf9/extensions/VisualEditor/ 'VE bugfix'
19:37 logmsgbot: kaldari Started syncing Wikimedia installation... :
19:27 logmsgbot: catrope synchronized wmf-config/CommonSettings.php
19:22 binasher: rebooting db1009, db1010 for upgrades
19:20 binasher: rebooting db1011 for upgrade
19:17 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Unbreak links in VE'
18:01 cmjohnson1: re-imaging db68-77
16:54 cmjohnson1: loading OS on db65-67
16:06 logmsgbot: preilly synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess 'update for zero'
16:05 logmsgbot: preilly synchronized php-1.20wmf9/extensions/ZeroRatedMobileAccess 'update for zero'
14:53 RobH: cp1001 returned to service, resolving rt 3212
14:30 RobH: cp1001 squid services going offline for troubleshooting
02:37 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Thu Aug 9 02:37:33 UTC 2012
02:20 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Thu Aug 9 02:19:57 UTC 2012

August 8

21:30 logmsgbot: kaldari Finished syncing Wikimedia installation... :
20:49 logmsgbot: kaldari Started syncing Wikimedia installation... :
20:02 binasher: streaming a hotbackup of db1017 to db1050
19:54 ottomata: updating DNS entries for analytics1011-1027, they are all in row-c
18:47 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikimedia to 1.20wmf9
18:45 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Fix $wgVisualEditorParsoidPrefix'
18:44 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews and wikibooks to 1.20wmf9
18:36 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary and wikiquote over to 1.20wmf9
18:35 binasher: completed metawiki centralnotice schema migration for fundraising
18:31 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity and wikisource to 1.20wmf9
18:26 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special, fishbowl, closed and private wikis to 1.20wmf9
18:25 Reedy: special, fishbowl, closed and private wikis to 1.20wmf9
18:16 apergos: shutting down ms10, going to move it to public ip
17:21 logmsgbot: preilly synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess 'update for zero'
17:20 logmsgbot: preilly synchronized php-1.20wmf9/extensions/ZeroRatedMobileAccess 'update for zero'
16:58 mutante: freeing disk space on wikitech
16:53 logmsgbot: preilly synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess 'update for zero'
16:52 logmsgbot: preilly synchronized php-1.20wmf9/extensions/ZeroRatedMobileAccess 'update for zero'
16:46 logmsgbot: preilly synchronized php-1.20wmf9/extensions/MobileFrontend 'update for zero'
16:45 logmsgbot: preilly synchronized php-1.20wmf8/extensions/MobileFrontend 'update for zero'
16:40 cmjohnson1: swapping out main board on search32
15:21 logmsgbot: catrope synchronized php-1.20wmf9/extensions/VisualEditor/.git 'Sync .git metadata so Special:Version is accurate'
14:43 logmsgbot: reedy synchronized wmf-config/
11:49 mark: Upgraded Observium to latest SVN
02:40 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Wed Aug 8 02:40:06 UTC 2012
02:21 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Wed Aug 8 02:21:44 UTC 2012
01:04 mutante: a couple package upgrades on bast1001

August 7

23:22 logmsgbot: catrope synchronized php-1.20wmf8/resources/mediawiki/mediawiki.js 'Deploying 6b6466f948d29520e1e3ab2592b940ce52415300'
23:19 logmsgbot: catrope synchronized php-1.20wmf9/extensions/VisualEditor/ 'Update VisualEditor'
22:48 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'syncing CommonSettings.php for Curation Toolbar'
22:47 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'syncing InitialiseSettings.php for Curation Toolbar'
22:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disabled echo due to bug 39085'
22:12 logmsgbot: reedy synchronized php-1.20wmf8/extensions/E3Experiments/
22:05 binasher: streaming a hotbackup of db1001 to db1049 (new enwiki snapshot host in eqiad)
21:48 logmsgbot: reedy synchronized php-1.20wmf9/extensions/EducationProgram/
21:45 Reedy: Updated EducationProgram schema on both enwiki and test2wiki
20:15 logmsgbot: reedy synchronized php-1.20wmf9/includes/api/ApiQuery.php
16:23 LeslieCarr: moving bits traffic from pmtpa to eqiad
15:45 mark: Moving text squid traffic from pmtpa to eqiad
15:08 first reports of sites being very slow
15:07 nagios-wm: Apache HTTP on ... is CRITICAL: CRITICAL - Socket timeout after 10 seconds
14:35 Jeff_Green: starting otrs dump on db49
07:11 paravoid: powercycled srv272, swapdeath
03:09 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Tue Aug 7 03:09:28 UTC 2012
02:37 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Tue Aug 7 02:37:34 UTC 2012
00:53 mutante: planet (en and fr) configs Updated to revision 115645.
00:32 Tim: on cp3002: set tcp_tw_recycle back to zero

August 6

23:49 mutante: wikitech back to old version after failed upgrade attempt
23:48 mutante: test
22:27 mutante: dist-upgrading wikitech instance
22:19 logmsgbot: aaron synchronized php-1.20wmf8/includes/filerepo/backend/SwiftFileBackend.php
22:14 binasher: rebooting db1028, upgrading
22:13 binasher: rebooting db1027, upgrade to precise
22:10 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/backend/SwiftFileBackend.php 'deployed 0a5c71d4dd6405502b1d1d0a02b5d0927d519986'
21:59 binasher: rebooting db1026, upgrading to precise
21:54 mutante: saving space on wikitech linode - gzipping old .sql dumps and stuff
21:32 logmsgbot: catrope synchronized php-1.20wmf9/extensions/VisualEditor/ 'Updating VisualEditor'
21:18 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Fix Parsoid URL'
21:09 logmsgbot: catrope synchronized php-1.20wmf9/extensions/VisualEditor 'Updating VisualEditor'
20:23 logmsgbot: catrope synchronized php-1.20wmf9/cache/l10n/ 'Sync out l10n cache again now that it is properly rebuilt'
19:47 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki to 1.20wmf9
19:45 RoanKattouw: Rebuilt ExtensionMessages-1.20wmf9.php and then l10n cache for wmf9 to pick up addition of FundraiserLandingPage.i18n.magic.php , somehow wasn't picked up during initial scap
19:44 RobH: asw-c8-eqiad PEM1 power reseated, cleared alarm rt 3204
19:39 logmsgbot: catrope synchronized wmf-config/ExtensionMessages-1.20wmf9.php 'Updated'
19:30 RobH: db1047 back online
19:18 RobH: db1047 shutting down
19:15 RobH: db1047 mysql and system shutdown per rt 3084 for bad memory swap
19:14 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files:
19:12 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.20wmf9
19:11 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.20wmf9 and scap to rebuild message cache
18:20 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.20wmf9 and scap to rebuild message cache
18:12 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.20wmf9 and scap to rebuild message cache
18:06 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/backend/FileBackendMultiWrite.php 'deployed b55e9652fce3051d621356c5c87c47f44515a367'
18:02 logmsgbot: aaron synchronized wmf-config/filebackend.php
17:59 logmsgbot: aaron synchronized php-1.20wmf8/includes/filerepo/backend/FileBackendMultiWrite.php 'deployed b55e9652fce3051d621356c5c87c47f44515a367'
17:11 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Adding Swift to local multiwrite backend for testwikis+mw.org'
16:57 cmjohnson1: powering down mc1 to reseat DIMM
16:50 logmsgbot: reedy synchronized .
16:33 logmsgbot: reedy synchronized php-1.20wmf9/
15:08 paravoid: switched European bits traffic back to esams
14:50 Tim: on cp3002: temporarily set net.ipv4.tcp_tw_recycle=1
14:41 Tim: on cp3002 increasing tcp_max_tw_buckets
14:33 Tim: on cp3001: date -s `"date"` to fix leap second issue
13:26 Nemo_bis: Reedy says: Status: Down - routing issues to Tampa !Wikipedia !Wikimedia
13:17 Nemo_bis: more or less everything at !Wikimedia / !Wikipedia seems down
02:27 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Mon Aug 6 02:27:00 UTC 2012

August 5

21:54 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Update per rearrangements etc'
21:16 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'more cp updates to $wgSquidServersNoPurge'
20:56 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Add some eqiad internal ips for squids to wgSquidServersNoPurge'
02:24 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Sun Aug 5 02:24:31 UTC 2012

August 4

02:27 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Sat Aug 4 02:27:51 UTC 2012

August 3

23:50 logmsgbot: reedy synchronized wmf-config/secure.php 'fix bits usages'
21:53 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 38988 - Enable Extension:Collection for Wikimedia Sverige'
18:58 maplebed: updated DNS to fix *.svc.{esams,eqiad}.wmnet forward resolution
18:49 andrewbogott: turned off nova-compute on virt1, virt2, virt3, virt4
18:44 maplebed: updating DNS adding swift eqiad LVS address
17:42 logmsgbot: aaron synchronized php-1.20wmf8/extensions/TimedMediaHandler 'Updated to master (0207962155b810b11100bc6d05c3562949e7c1a9)'
07:58 hashar: GlusterFS upgrade of labs project storage seems to have been completed. Fixed the issues I had with it on 'beta' project, yeah!!!!!
02:25 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Fri Aug 3 02:25:37 UTC 2012
01:54 domas: bumped memcached timeout to be slightly above min_rto, 200ms->250ms, should eliminate most of stupid memcached errors because of datacenter network failures
01:53 logmsgbot: midom synchronized wmf-config/mc.php

August 2

22:48 Ryan_Lane: upgrading glusterfs for project storage to 3.3
22:05 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Moved all wikis to multiwrite backend'
22:05 mutante: dist-upgrading sodium
21:47 hashar: added Dsc to the Gerrit 'integration' group. David now got access to the various integration/* repositories.
20:34 logmsgbot: andrew synchronized php-1.20wmf8/extensions/Echo/modules/base/ext.echo.base.css
20:30 logmsgbot: asher synchronized wmf-config/mc.php 'setting wgMemCachedPersistent back to false, no change observed to current issue'
20:12 logmsgbot: asher synchronized wmf-config/mc.php 'setting wgMemCachedPersistent = true as an experiment'
20:06 Andrew: Leaving Echo deployment there for now.
20:04 Andrew: Rolled back LQT updates, they seem to break other notifications. We'll figure out the bugs with that in another window.
20:02 logmsgbot: andrew synchronized php-1.20wmf8/extensions/LiquidThreads
19:56 Andrew: deployed Echo to mediawiki.org
19:56 logmsgbot: andrew synchronized wmf-config/InitialiseSettings.php
19:55 logmsgbot: andrew synchronized wmf-config/CommonSettings.php
19:55 logmsgbot: andrew synchronized php-1.20wmf8/extensions/Echo
19:54 logmsgbot: andrew synchronized php-1.20wmf8/extensions/LiquidThreads
19:44 logmsgbot: andrew synchronized wmf-config/InitialiseSettings.php
19:44 logmsgbot: andrew synchronized php-1.20wmf8/extensions/Echo
19:43 logmsgbot: andrew synchronized php-1.20wmf8/extensions/LiquidThreads
19:43 Andrew: Deploying Echo to test2wiki instead
18:38 Andrew: Turned Echo back on on testwiki
18:35 Andrew: Stealing E2's deployment window to have another crack at Echo
16:44 paravoid: sync-apache/apache-gracefull-all for BZ #38905 (ShortUrl on !Wikipedia)
16:20 RobH: authdns-update for smokeping
13:05 logmsgbot: nikerabbit synchronized php-1.20wmf8/extensions/Translate/specials/ 'Translate bug fixes'
04:45 logmsgbot: tstarling synchronized php-1.20wmf8/skins/common/commonElements.css
02:27 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Thu Aug 2 02:27:17 UTC 2012
02:11 Ryan_Lane: deployed OATHAuth 8992e6f541cd7adf8111ccd87f25a481d3759b33 on labsconsole
02:01 logmsgbot: reedy synchronized php-1.20wmf8/extensions/E3Experiments/
01:26 logmsgbot: reedy synchronized php-1.20wmf8/extensions/SubPageList3/
01:16 logmsgbot: reedy synchronized php-1.20wmf8/extensions/SubPageList3/SubPageList3.php
00:55 logmsgbot: asher synchronized wmf-config/db.php 'moving watchlist/special contrib queries to db63, preparing to decom db12'
00:47 logmsgbot: reedy synchronized php-1.20wmf8/extensions/SubPageList3/SubPageList3.php
00:43 logmsgbot: asher synchronized wmf-config/db.php 'adding db63 to s1 at a low weight'

August 1

23:59 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:27 mutante: rebuilding wikitech-l archives
23:16 logmsgbot: andrew Finished syncing Wikimedia installation... :
22:59 Andrew: pulling the plug on Echo deployment due to replication-lag related bug.
22:57 logmsgbot: andrew Started syncing Wikimedia installation... :
22:32 Andrew: Running scap to deploy Echo to test
22:31 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Made mw.org use the multiwrite backend.'
22:23 logmsgbot: aaron synchronized live-1.5/MWVersion.php 'logging for /home being used for thumbs for testwiki'
22:14 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Removed read-only flag from testwiki backends.'
22:08 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Made testwikis use multiwrite backend.'
21:27 Andrew: Added Echo tables to testwiki
21:11 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Configured $wgTimedTextForeignNamespaces.'
21:08 logmsgbot: aaron synchronized php-1.20wmf8/extensions/TimedMediaHandler/handlers/TextHandler/TextHandler.php 'Deployed a247b6264c99cff66e4e3ae3428a46f971fab3a2'
21:07 logmsgbot: aaron synchronized php-1.20wmf8/extensions/TimedMediaHandler/TimedMediaHandler.php 'Deployed a247b6264c99cff66e4e3ae3428a46f971fab3a2'
21:01 logmsgbot: reedy synchronized php-1.20wmf8/extensions/E3Experiments/
20:48 logmsgbot: andrew synchronized wmf-config/CommonSettings.php
20:47 Andrew: srv281: rsync: write failed on "/apache/common-local/wmf-config/InitialiseSettings.php": No space left on device (28)
20:47 logmsgbot: andrew synchronized wmf-config/InitialiseSettings.php
20:46 Andrew: syncing {Common,Initialise}Settings.php
20:46 Andrew: Added configuration skeleton for Echo in {Common,Initialise}Settings.php
20:46 Andrew: Created tables for Echo on mediawikiwiki
19:16 MaxSem: Ran extensions/GeoData/updateIndexGranularity.php on testwiki
19:04 binasher: started hotbackup of db1017 to db63
19:04 mark: Ran reprepro --delete clearvanished on brewster to cleanup removed repositories karmic-wikimedia and oneiric-wikimedia
19:03 mutante: installing a couple lib upgrades on fenari
18:15 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: everything else over to 1.20wmf8
18:04 logmsgbot: aaron synchronized php-1.20wmf8/extensions/TimedMediaHandler 'Updating to 4b7c77d63d7798a024c7f8b4ba01e2f3e7203550'
17:37 maplebed: pushing dns typo correction
10:32 Tim: on dobson: restarted ntpd
10:30 Tim: on linne: restarted ntpd in an attempt to fix the leap indicator
02:49 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Wed Aug 1 02:49:33 UTC 2012
02:25 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Wed Aug 1 02:25:43 UTC 2012
01:53 logmsgbot: catrope synchronized php-1.20wmf8/cache/l10n
01:05 logmsgbot: catrope synchronized php-1.20wmf8/extensions/TimedMediaHandler/MwEmbedModules/EmbedPlayer/resources/mw.EmbedPlayer.js 'touch'
00:33 Tim: leap second event plus one month is causing an apparent 1s step in time reported by linne/dobson as seen by some clients, causing nagios errors etc. Will step.

July 31

23:12 logmsgbot: aaron Finished syncing Wikimedia installation... :
22:32 logmsgbot: aaron Started syncing Wikimedia installation... :
22:19 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Disable ogghandler for wikis using TMH.'
22:18 logmsgbot: catrope synchronized php-1.20wmf7/extensions/TitleBlacklist/TitleBlacklist.hooks.php 'php-1.20wmf8/extensions/TitleBlacklist/TitleBlacklist.hooks.php Deploying 3d45bc25577f33bc220e17a7671d391977f9fbf9 and 4c7db40395b6b2d273e08f6b0e59c5a74cb5ed34'
22:17 RobH: authdns-update for ms-be1006 and ms-be1012
22:04 binasher: added memcached_1.4.14-0wmf1_amd64 to precise-wikimedia
21:55 logmsgbot: catrope synchronized php-1.20wmf8/extensions/TitleBlacklist/TitleBlacklist.hooks.php
21:54 logmsgbot: catrope synchronized php-1.20wmf8/extensions/TitleBlacklist/TitleBlacklist.hooks.php
21:49 logmsgbot: catrope synchronized php-1.20wmf8/extensions/TitleBlacklist/TitleBlacklist.hooks.php
21:49 logmsgbot: aaron Finished syncing Wikimedia installation... :
21:45 logmsgbot: aaron Started syncing Wikimedia installation... :
21:08 logmsgbot: aaron Finished syncing Wikimedia installation... :
21:03 logmsgbot: aaron Started syncing Wikimedia installation... :
20:09 logmsgbot: mlitn Finished syncing Wikimedia installation... :
19:35 logmsgbot: mlitn Started syncing Wikimedia installation... :
19:26 RobH: authdns-update for wtp1 info
17:40 AaronSchulz: Initial swift originals migration for s1-s7 done
17:29 maplebed: finished swift deploy - image scaling requests now go straight to the rendering cluster
16:16 logmsgbot: aaron synchronized live-1.5/MWVersion.php 'Recognize thumb_handler for testwiki and /home check.'
16:09 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Deployed handlerUrl for thumb zone.'
16:00 maplebed: beginning swift deploy to make thumbnail requests bypass ms5 and go straight from swift to the rendering cluster
14:33 mark: Repooled cp1042
14:33 mark: Reinstalled cp1042 with Precise
13:16 Tim: compiling phpllvm tests on bast1001
12:19 mark: Depooled cp1042 for reinstall with Precise
08:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'touch'
08:45 logmsgbot: reedy synchronized flaggedrevs.dblist 'Disable FR on eswikibooks'
02:54 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Tue Jul 31 02:54:35 UTC 2012
02:27 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Tue Jul 31 02:27:34 UTC 2012

July 30

23:49 binasher: adding revised enwiki.aft_article_feedback af_user_id_user_ip index via osc
23:48 binasher: dropping enwiki.aft_article_feedback af_user_id_user_ip index
21:48 logmsgbot: reedy synchronized php-1.20wmf7/extensions/Collection/.git 'Just for saper..'
20:51 binasher: adding enwiki.aft_article_feedback af_user_id_ip index
20:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'touch'
20:30 logmsgbot: reedy synchronized php-1.20wmf8/extensions/Collection
20:29 logmsgbot: reedy synchronized php-1.20wmf7/extensions/Collection
19:26 logmsgbot: reedy synchronized .
19:22 logmsgbot: reedy synchronized php-1.20wmf8/extensions/UploadWizard/
18:25 notpeter: removing srv194 from apaches pool due to logging issues
18:06 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf8
17:42 notpeter: re-adding srv194 to pmtpa apaches pool for basic testing of precise
17:02 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Remove duplicate routing code'
16:55 logmsgbot: reedy synchronized wmf-config/
16:32 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable shorturl on tawikis'
16:31 Reedy: Created ShortUrl table on numerous tawiki* wikis
16:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable ShortUrl on tawiki, hiwiki and orwiki'
16:26 Reedy: Add ShortUrl table to hiwiki, orwiki and tawiki
16:05 paravoid: sync-apache/apache-graceful-all for RT #2121
16:02 Reedy: Created FlaggedRevs tables on eswikiboooks
15:58 logmsgbot: reedy synchronized wmf-config/
15:53 Reedy: Created FlaggedRevs tables on eswiki
08:11 hashar: gallium/jenkins: deployed job for the WLMMobile nightly builds
07:40 hashar: gallium/jenkins: updating Android SDKs
02:46 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Mon Jul 30 02:45:58 UTC 2012
02:23 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Mon Jul 30 02:23:05 UTC 2012
01:00 Tim: installed mytop on fenari at faidon's request
00:52 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'docs'
00:49 Tim: killed all AFTv5 queries another few times
00:49 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'disabled properly'
00:45 Tim: killed ArticleFeedback queries on db12
00:44 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'disabled ArticleFeedback since it caused an overload on db12 and general site slowness'
00:32 Tim: on hume: kill -CONT
00:28 Tim: on hume: stopped populateRevisionSha1.php with kill -STOP due to excessive (800s) lag on db12

July 29

23:13 logmsgbot: reedy synchronized php-1.20wmf8/includes/WebRequest.php
02:45 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Sun Jul 29 02:45:34 UTC 2012
02:23 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Sun Jul 29 02:23:12 UTC 2012

July 28

06:29 logmsgbot: catrope synchronized php-1.20wmf8/extensions/Vector/modules/ext.vector.collapsibleNav.css 'touch'
06:29 logmsgbot: catrope synchronized php-1.20wmf7/extensions/Vector/modules/ext.vector.collapsibleNav.css 'touch'
02:51 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Sat Jul 28 02:51:29 UTC 2012
02:25 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Sat Jul 28 02:25:46 UTC 2012

July 27

21:16 RobH: updated dns for mgmt on wtp1.mgmt
20:42 cmjohnson1: setting db66
20:23 RobH: authdns-udpate for yttrium/wlm.w.o per rt 3221
20:12 cmjohnson1: setting up tmh2 rt3298
19:51 cmjohnson1: setting up tmh1 rt 3298
19:40 RobH: setting up yttrium per rt 3221
19:32 logmsgbot: aaron synchronized php-1.20wmf7/thumb.php 'deployed 3f79e142beade8135514a3ac2a1af2cdd8a30901'
19:31 logmsgbot: aaron synchronized php-1.20wmf7/includes/DefaultSettings.php 'deployed 3f79e142beade8135514a3ac2a1af2cdd8a30901'
19:31 logmsgbot: aaron synchronized php-1.20wmf7/includes/filerepo/FileRepo.php 'deployed 3f79e142beade8135514a3ac2a1af2cdd8a30901'
19:26 logmsgbot: aaron synchronized php-1.20wmf8/thumb.php 'deployed 718e305c2213125b1df41323f22b7db400a77139'
19:25 logmsgbot: aaron synchronized php-1.20wmf8/includes/filerepo/FileRepo.php 'deployed 718e305c2213125b1df41323f22b7db400a77139'
19:24 logmsgbot: aaron synchronized php-1.20wmf8/includes/DefaultSettings.php 'deployed 718e305c2213125b1df41323f22b7db400a77139'
19:00 RobH: allocating potassium and zirconium to rt 3342 (misc db use) and shutting them down until they are installed.
18:37 RobH: another authdns-update, forgot to update asset tags to servernames for mgmt
18:33 RobH: authdns-update for tmh1/2
14:07 hashar: srv281: rsync: write failed on "/apache/common-local/wmf-config/InitialiseSettings.php": No space left on device (28)
14:07 hashar: srv281: rsync: write failed on "/apache/common-local/wmf-config/InitialiseSettings.php": No space left on device (28)
14:07 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '16817 - fix nycwikimedia wgImportSources'
13:40 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Deploying I39d1ca7c: import sources for nycwikimedia'
12:51 Reedy: Created tgs_lang indexes on wikis that have translate installed
02:53 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Fri Jul 27 02:53:30 UTC 2012
02:31 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Fri Jul 27 02:31:05 UTC 2012

July 26

22:26 logmsgbot: kaldari synchronized php-1.20wmf7/extensions/E3Experiments/Experiments.hooks.php 'syncing Experiments.hooks.php for wmf7'
22:25 logmsgbot: kaldari synchronized php-1.20wmf8/extensions/E3Experiments/Experiments.hooks.php 'syncing Experiments.hooks.php for wmf8'
22:00 logmsgbot: reedy synchronized wmf-config/
21:55 logmsgbot: reedy synchronized php-1.20wmf7/LocalSettings.php
21:48 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'turning E3Experiments on for en.wiki'
21:38 logmsgbot: kaldari Finished syncing Wikimedia installation... :
21:17 logmsgbot: kaldari Started syncing Wikimedia installation... :
21:09 logmsgbot: kaldari synchronized php-1.20wmf8/extensions/E3Experiments 'sync E3Experiments stuff for en.wiki'
21:02 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'turning on E3Experiments for en.wiki'
19:40 RobH: changed to calcium instead, as yttrium is a 610 and i only need a 310
19:37 RobH: claiming yttrium for smokeping install
19:23 RobH: authdns update for new services
18:59 logmsgbot: reedy synchronized php-1.20wmf8/extensions/WikiEditor
16:53 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
16:52 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
16:46 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Special -'
16:43 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'enable shorturl on testwiki and test2wiki'
16:42 Reedy: Created ShortUrl tables on test2wiki
13:44 hashar: deployment-prep rsync finished for both apache and upload6. Remounting and restarting apaches
13:23 cmjohnson1: search32 has hardware issues..powering down
08:43 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php '(bug 32516) Enable Narayam on pawiki.'
08:38 logmsgbot: nikerabbit synchronized php-1.20wmf7/extensions/Narayam/ 'Deploying https://gerrit.wikimedia.org/r/16721'
02:46 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Thu Jul 26 02:46:34 UTC 2012
02:24 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Thu Jul 26 02:24:01 UTC 2012
00:21 logmsgbot: catrope synchronized php-1.20wmf8/skins/common/commonElements.css 'Deploy 0cc26386cec1955acf2e1f7341eaaa466ea220b2 and 162a25e96cf48cbd2545f6c3d54f0c20f1bcd520'
00:08 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Undo testing change'

July 25

23:46 logmsgbot: reedy synchronized php-1.20wmf8/extensions/AbuseFilter/
23:30 binasher: completed abuse_filter_log migration for all wikis
22:54 logmsgbot: aaron synchronized php-1.20wmf7/thumb.php
22:29 LeslieCarr: fixed puppetmaster on stafford (bad stafford, no cookie!)
22:20 logmsgbot: aaron synchronized php-1.20wmf7/thumb.php 'url extraction fix'
22:15 logmsgbot: aaron synchronized php-1.20wmf7/thumb.php 'debug logging'
22:08 logmsgbot: aaron synchronized php-1.20wmf7/thumb.php 'url regex tweak'
22:06 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Temporarily set $wmgArticleFeedbackv5OversightEmails to my work e-mail for debugging'
22:05 binasher: started second step of abuse_filter_log migration via osc, indexes for new columns
21:56 logmsgbot: aaron synchronized php-1.20wmf7/thumb.php 'debug logging'
21:08 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'Re-implementing wgNoticeFundraisingUrl override for wmf7 wikis'
21:02 logmsgbot: aaron synchronized live-1.5/thumb_handler.php 'entry point to thumb.php'
20:43 binasher: started abuse_filter_log migration via osc. first step, adding afl_rev_id, afl_log_id columns. running in cluster order starting with s1 (enwiki)
20:30 logmsgbot: reedy synchronized php-1.20wmf8/includes/objectcache/MemcachedClient.php 'Bring in reversion to shut up warnings'
20:26 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: all none wikipedia projects to 1.20wmf8
20:21 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
19:49 logmsgbot: reedy synchronized php-1.20wmf8/extensions/AbuseFilter/ 'Revert back to 0eeafeab44fb7a177ebf929c313999855656701f'
18:25 logmsgbot: reedy synchronized php-1.20wmf8/extensions/Quiz/
18:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity, closed and special to 1.20wmf8
18:10 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews to 1.20wmf8
18:02 binasher: streaming hot backup of db1033 to db63, intended db12 replacement and first precise enwiki db
17:10 mark: Added asw-c-eqiad to Torrus, RANCID, Observium
16:52 cmjohnson1: srv266 still working on it...going down
15:46 logmsgbot: reedy synchronized wmf-config/
15:34 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
15:33 Reedy: Created wikilove tables on svwikinews
15:24 Reedy: Created FlaggedRevs tables on cawikinews
15:23 logmsgbot: reedy synchronized wmf-config/
14:37 cmjohnson1: srv266 shutting down for attempted h/w repairs
02:46 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Wed Jul 25 02:46:06 UTC 2012
02:23 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Wed Jul 25 02:23:44 UTC 2012

July 24

23:23 logmsgbot: awjrichards Finished syncing Wikimedia installation... : Synchronizing updates to MobileFrontend per http://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-06-24
22:52 logmsgbot: awjrichards Started syncing Wikimedia installation... : Synchronizing updates to MobileFrontend per http://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-06-24
22:26 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Fixing footer logo for enwiki on mobile view'
20:10 binasher: upgrading versions of xtrabackup and percona toolkit on all coredbs
19:45 logmsgbot: mlitn Finished syncing Wikimedia installation... :
19:10 logmsgbot: mlitn Started syncing Wikimedia installation... :
18:30 RoanKattouw: root@fenari:~# chmod g+w /home/wikipedia/common/php-1.20wmf8/cache/l10n/
18:30 RoanKattouw: root@fenari:~# chown l10nupdate:wikidev /home/wikipedia/common/php-1.20wmf8/cache/l10n
18:04 logmsgbot: reedy synchronized php-1.20wmf8/extensions/AbuseFilter/
16:28 logmsgbot: reedy synchronized php-1.20wmf8/extensions/AbuseFilter/
16:08 RobH: srv278 hardware worked on by chris, placing back in service to see if its going to stay fixed
14:30 cmjohnson1: correcting enviromental leads on powestrips in sdtpa
14:06 paravoid: force puppet run on all LVS servers for the ganglia change
09:10 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Bug 36104 - Narayam on bnwikisource'
02:48 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Tue Jul 24 02:48:21 UTC 2012
02:26 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Tue Jul 24 02:26:52 UTC 2012
00:03 binasher: upgrading db1045 to precise for testing

July 23

23:11 AaronSchulz: Started originals migrations for remaining wikis
22:07 binasher: deploying new mobile redirector to eqiad squids
20:44 cmjohnson1: ms-be10 taking off-line for some hardware testing
20:13 logmsgbot: reedy synchronized php-1.20wmf7/includes/specials/SpecialUserlogin.php
20:12 logmsgbot: reedy synchronized php-1.20wmf8/includes/specials/SpecialUserlogin.php
19:31 RobH: authdns-update for transcode name updates
19:28 RobH: powering down old transcode1, was camera gateway sandbox, reclaiming name
18:57 maplebed: took ms-be10 out of rotation because it ate itself
18:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.20wmf8
17:03 RobH: reobooting potassium, shouldnt be in use, had tons of cruft ssh connections from month ago
16:26 Reedy: running ddsh -cM -g mediawiki-installation -o -oSetupTimeout=30 -F30 -- "sudo -u mwdeploy rm -rf /usr/local/apache/common/php-1.20wmf5"
16:23 Reedy: running ddsh -cM -g mediawiki-installation -o -oSetupTimeout=30 -F30 -- "sudo -u mwdeploy rm -rf /usr/local/apache/common/php-1.20wmf4"
16:22 logmsgbot: reedy Finished syncing Wikimedia installation... : 1.20wmf8 messages
16:04 paravoid: reinstalling srv281; disk full, hasn't run puppet for 300 days, depooled for ages
15:54 logmsgbot: reedy Started syncing Wikimedia installation... : 1.20wmf8 messages
15:44 logmsgbot: reedy Started syncing Wikimedia installation... : 1.20wmf8 messages
15:42 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.20wmf8 and rebuild localisation cache
15:23 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.20wmf8 and rebuild localisation cache
15:21 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: rm chwikimedia we dont host it
15:19 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: chwikimedia to 1.20wmf7
15:14 Reedy: srv281 has a full / and hasn't had a puppet run in over 434298 minutes
15:10 logmsgbot: reedy synchronized php-1.20wmf8/cache/
15:06 logmsgbot: reedy synchronized php-1.20wmf8/ 'Initial sync out of 1.20wmf8'
13:44 RobH: authdns-update for ms-be eqiad hosts
12:46 maplebed: rebooting ms-be10 for xfs errors and a clean boot
12:38 maplebed: rebalanced swift rings moving more content to new object servers
12:26 logmsgbot: reedy synchronized flaggedrevs.dblist
12:26 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php
12:25 Reedy: Created trwikiquote flagged revs tables
12:23 logmsgbot: reedy synchronized wmf-config/
02:22 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Mon Jul 23 02:22:45 UTC 2012
02:22 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Mon Jul 23 02:22:33 UTC 2012

July 22

22:56 Ryan_Lane: restarted opendj on sanger. The process OOM'd due to heap size.
02:23 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Sun Jul 22 02:23:02 UTC 2012
02:22 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Sun Jul 22 02:22:49 UTC 2012

July 21

17:42 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'deployed ffd469239a80b8bb9813eb029e4cd71a3ea52db9'
02:23 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Sat Jul 21 02:23:19 UTC 2012
02:23 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Sat Jul 21 02:23:06 UTC 2012

July 20

18:57 paravoid: powerin on owa1/owa2 again, they're being used as ganglia aggregators for swift
18:11 Ryan_Lane: starting gerrit
17:59 binasher: db48 is now replicating from db1048 (and db1048 from db48)
17:26 RobHalsell: authdns-update run for new servers
15:29 paravoid: powering off the rest of mw10[0-9][0-9]
14:35 mark: Added server hydrogen to the dns_rec eqiad LVS pool
14:22 paravoid: powering off owa1/2/3, unused
13:54 paravoid: powering off all of mw1[0-9][0-9][0-9].eqiad.wmnet, unused
13:36 mark: Reinstalling hydrogen
13:19 paravoid: powercycling hydrogen, down since yesterday
12:59 paravoid: powercycling nescio/ns2, unresponsive network & console
10:38 mark: Built new varnish 3.0.3~rc1+persistent1-wm1 packages and inserted them into the precise-wikimedia APT repository
02:23 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Fri Jul 20 02:23:39 UTC 2012
02:23 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Fri Jul 20 02:23:27 UTC 2012

July 19

22:48 logmsgbot: catrope Finished syncing Wikimedia installation... :
22:10 logmsgbot: catrope Started syncing Wikimedia installation... :
22:02 binasher: deploying new mobile redirector to pmtpa text squids (currently inactive)
21:39 logmsgbot: kaldari Started syncing Wikimedia installation... :
21:36 binasher: deploying new mobile redirector to eqiad text squids
21:36 logmsgbot: catrope synchronized php-1.20wmf7/extensions/LastModified 'Remove E3Experiments cruft from LastModified'
21:07 binasher: deploying new mobile redirector to esams text squids
18:52 preilly: fixing subdomain check for zero vs m
18:52 logmsgbot: preilly synchronized wmf-config/mobile.php 'fix subdomain check'
18:21 RobH: updating dns with new zonefiles for legally won domain names
18:15 Jeff_Green: storage3 dist-upgrade and reboot
17:01 AaronSchulz: Running copyFileBackend.php for commons (shards c-f)
16:28 cmjohnson1: mw60 powering down to replace DIMM B1 rt3287
15:48 RobH: hydrogen repaired per rt 3243
15:37 RobH: rebooting hydrogen to set bios redirection
14:18 logmsgbot: reedy synchronized wmf-config/
13:42 Jeff_Green: dist-upgrade and reboot hosts in payments cluster
13:12 Reedy: Pointed /h/w/c/php to php-1.20wmf7
13:10 Reedy: Deleted php-1.20wmf6/cache/l10n from mediawiki-installation
02:31 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Thu Jul 19 02:31:02 UTC 2012
02:23 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Thu Jul 19 02:23:27 UTC 2012

July 18

22:02 notpeter: ok, changed my mind one more time. removing srv194 from apaches pool for precise testing
21:41 notpeter: scratch that. removing srv289 from apaches pool for precise testing, not mw1
21:39 notpeter: removing mw1 from apaches pool to do precise test install
21:28 notpeter: adding php5 packages to precise-wikimedia repo
19:45 cmjohnson1: srv281 powering down for HW checks
19:09 cmjohnson1: srv278 powering down for hardware problems -- random reboot system already depooled
18:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: 295 remaining wikis to 1.20wmf7
17:37 cmjohnson1: mw8 powering down to replace DIMM A3 rt 3273
17:25 cmjohnson1: search32 powering down to replace DIMM B1 rt 3076
16:55 logmsgbot: reedy synchronized php-1.20wmf7/includes/api/ApiQuerySiteinfo.php
12:52 pp-pdf1: upgraded mwlib to 0.13.11, restarted all services
07:58 hashar: gallium: added Firefox 14 to Testswarm, disabled Firefox 13.
02:47 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Wed Jul 18 02:47:23 UTC 2012
02:24 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Wed Jul 18 02:24:14 UTC 2012

July 17

23:24 Tim: on srv193: removed core dump files, disabled core dumping, restarted apache
19:52 logmsgbot: mlitn Finished syncing Wikimedia installation... :
19:09 logmsgbot: mlitn Started syncing Wikimedia installation... :
15:18 mark: lvs1002 is back up and idling
15:15 mark: Fixed serial console redirection after boot to OFF on lvs1002
15:06 mark: lvs4 is back up and serving traffic
14:51 mark: Reinstalling lvs4 with Ubuntu Precise
14:48 mark: Stopped PyBal on lvs4, failing over traffic to lvs3
14:47 mark: Reinstalling lvs1002 with Ubuntu Precise
14:46 mark: Stopped PyBal on lvs1002, failing over traffic to lvs1005
14:44 mark: lvs1001 is back up and serving traffic
14:11 mark: Stopped PyBal on lvs1001, failing over traffic to lvs1004
13:09 maplebed: changed auth URL for swift to use load balancer rather than round robin DNS
13:08 mark: lvs1003 is back up and serving traffic
13:07 maplebed: changed order on ganglia swift view graphs to group by metric rather than host
13:02 logmsgbot: reedy synchronized wmf-config/
12:53 mark: Fixed boot order on lvs1003
12:34 mark: Reinstalling lvs1003 with Ubuntu Precise
12:28 mark: Stopped PyBal on lvs1003, failing over traffic to lvs1006
12:10 mark: amslvs1 is back up and serving traffic
11:44 mark: Reinstalling amslvs1 with Ubuntu Precise
11:40 mark: Stopped PyBal on amslvs1, failing over traffic to amslvs3
11:32 mark: amslvs2 is back up and serving traffic
10:47 mark: Reinstalling amslvs2 with Ubuntu Precise
10:36 mark: Stopped PyBal on amslvs2, failing over traffic to amslvs4
07:30 Tim: testing envvars/apache2.conf change on srv193
07:09 Tim: restarted apache on srv193
02:58 Tim: graceful restart of all apaches
02:52 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
02:47 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Tue Jul 17 02:47:06 UTC 2012
02:23 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Tue Jul 17 02:23:41 UTC 2012
00:44 Tim: deploying new redirects.conf
00:28 Tim: testing new redirects.conf on mw1

July 16

23:03 pp-pdf1: restarted nserve
23:02 pp-pdf1: fixed nserve handling of filenames with whitespace
22:46 pp-pdf1: restart all services
22:45 pp-pdf2: restart all services
22:45 pp-pdf3: restart all services
22:45 pp-pdf1: update mwlib to 0.13.9
22:45 pp-pdf2: update mwlib to 0.13.9
22:45 pp-pdf3: update mwlib to 0.13.9
22:44 pp-pdf2: update greenlet to 0.4.0
22:44 pp-pdf1: update greenlet to 0.4.0
22:44 pp-pdf3: update greenlet to 0.4.0
22:44 pp-pdf2: upgrade qserve to 0.2.8
22:44 pp-pdf3: upgrade qserve to 0.2.8
22:43 pp-pdf1: upgrade qserve to 0.2.8
22:20 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Disabling epub generation from Collection extension on all wikis except for simple and test'
22:19 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Making epub generation for Collection extension configurable'
18:26 AaronSchulz: Running copyFileBackend.php for commons (shards 8-b)
18:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf7
17:08 LeslieCarr: reactivating XO transit connections on cr1-sdtpa
17:03 LeslieCarr: draining cr2-eqiad to cr1-sdtpa link for moving of fiber
15:56 mark: lvs3 is back up, and idling
15:56 mutante: fixing pt.planet after missing locale has been added
15:48 mutante: installing package upgrades on singer (planet)
15:43 cmjohnson1: mw8 shutting down and taking offline to run Dell's dset program
15:38 mutante: planet - svn up the config
15:29 mark: Reinstalling lvs3 with Ubuntu Precise
15:17 mark: lvs5 is back up and serving traffic
14:49 mark: Reinstalling lvs5 with Ubuntu Precise
14:45 mark: Stopped PyBal on lvs5 to failover traffic to lvs1
14:35 maplebed: adjusted swift rings; set new object servers to 20, new container servers to 100
14:33 mark: lvs6 is back up and serving traffic
14:12 mark: Reinstalling lvs6 with Ubuntu Precise
14:02 mark: Stopped PyBal on lvs6 to failover traffic to lvs2
13:58 mutante: mw8 - alright, most likely just needs new DIMM per cmjohnson
13:50 mutante: mw8 PHP fatal errors, running out of memory
12:56 mutante: sync-apache, graceful-all to fix wikipedia.cz redirect
05:31 apergos: reboot to upgrade kernel etc on mw8 since it's been flapping anyways
02:46 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Mon Jul 16 02:46:14 UTC 2012
02:23 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Mon Jul 16 02:23:04 UTC 2012

July 15

20:19 Aaron|home: Running copyFileBackend.php for commons (shard 7)
02:46 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Sun Jul 15 02:46:33 UTC 2012
02:23 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Sun Jul 15 02:23:27 UTC 2012

July 14

18:46 LeslieCarr: powercycling frozen db1029
18:01 LeslieCarr: rebooting unresponsive mw1111
16:41 logmsgbot: midom synchronized php-1.20wmf6/includes/GlobalFunctions.php 'message key analysis'
16:18 logmsgbot: midom synchronized php-1.20wmf6/includes/GlobalFunctions.php 'message key analysis'
03:07 Aaron|home: Running copyFileBackend.php for commons (shards 5-6)
02:47 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Sat Jul 14 02:47:32 UTC 2012
02:23 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Sat Jul 14 02:23:14 UTC 2012
00:25 logmsgbot: reedy synchronized php-1.20wmf7/includes/api/ApiDelete.php

July 13

18:08 logmsgbot: reedy synchronized php-1.20wmf7/includes/GlobalFunctions.php
18:08 logmsgbot: reedy synchronized php-1.20wmf6/includes/GlobalFunctions.php
17:58 logmsgbot: reedy synchronized wmf-config/
17:57 logmsgbot: reedy synchronized php-1.20wmf7/
17:56 logmsgbot: reedy synchronized php-1.20wmf6/
17:32 logmsgbot: reedy synchronized wmf-config/
17:21 AaronSchulz: Running copyFileBackend.php for commons (shards 1-4) (actually started yesterday)
15:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable aftv5 on en_labswikimedia due to out of date schema'
15:21 logmsgbot: reedy synchronized php-1.20wmf7/includes/GlobalFunctions.php
15:20 logmsgbot: reedy synchronized php-1.20wmf6/includes/GlobalFunctions.php
15:20 logmsgbot: reedy synchronized php-1.20wmf7/includes/DefaultSettings.php
15:19 logmsgbot: reedy synchronized php-1.20wmf6/includes/DefaultSettings.php
15:17 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Setting wgDBerrorLogInUTC = true'
15:07 hasharDeadmau5: unknown column 'af_is_featured' on en_labswikimedia@db25
14:55 mark: Increased all mobile varnish server weights from 10 to 100 to aid chash
14:54 mark: Added cp1041.eqiad.wmnet back into the mobile LVS pool
10:10 mutante: mw1016 - was down, reinstalling with precise
02:46 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Fri Jul 13 02:45:59 UTC 2012

July 12

22:15 Aaron|home: Running copyFileBackend.php for commons (shards 0,c-f)
20:58 Aaron|home: Running copyFileBackend.php for commons (shards 0,8-b)
14:22 mutante: apache children on srv193 keep segfaulting since yesterday (test.wp)
14:00 RobHalsell: shutting down srv206 for chris per rt241
11:55 logmsgbot: reedy synchronized wmf-config/
10:40 mark: Depooled mobile varnish server cp1041 for vlan/hostname change and reinstallation with precise
04:38 binasher: started hot backup of db48 to db1048
02:50 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Thu Jul 12 02:50:49 UTC 2012
02:29 maplebed: adjusted the swift rings a bit to move some more traffic to the new hosts. set object partitions to weight 10, account and container 60.
02:27 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Thu Jul 12 02:26:57 UTC 2012

July 11

22:52 AaronSchulz: Running copyFileBackend.php for commons (shards 0,4-7)
22:06 AaronSchulz: Running copyFileBackend.php for commons (shards 0,0-3)
21:36 binasher: running hotbackup of db48 -> db49 (otrs / external misc)
21:31 binasher: rebooting db49 for kernel upgrade
21:23 binasher: rebuilding db49 (otrs slave)
21:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
20:16 logmsgbot: reedy synchronized php-1.20wmf7/extensions/CentralNotice
19:33 cmjohnson1: db10 swapping disk1 out rt3251
19:33 RobH: shutting down srv203 for hardware checking per rt3110
19:12 hashar: finally stopped using my production ssh key on labs.
19:11 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote and wikiversity to 1.20wmf7
19:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks and wikinews to 1.20wmf7
19:06 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary and wikisource to 1.20wmf7
19:03 cmjohnson1: search35 shutting down to reseat DIMM rt-3260
18:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Switch special, private and closed wikis from 1.20wmf6 to 1.20wmf7
15:16 logmsgbot: robh synchronized wmf-config/mc.php 'moving srv208 to down'
15:14 logmsgbot: robh synchronized wmf-config/mc.php
14:30 cmjohnson1: shutting down search32 to reseat DIMM rt-3076
11:16 mutante: yay @ svn removal
11:13 hashar: /home/wikipedia/conf/httpd cleaned out the svn repository. That directory is now 100% under git tracking yeah!!! Thanks TIm!
11:11 mutante: rebooting srv266
11:05 mutante: purged wikipedie.cz/Experti_na_prirodu URLs with purgeList.php
10:52 mutante: apache-graceful-all
10:48 mutante: sync-apache to push fixed wikipedie.cz redirect
08:36 hashar: synced /h/w/conf/httpd git and svn repositories
08:33 mutante: chmod -R g+w /home/wikipedia/conf/httpd on fenari to fix group write on .git/objects
07:59 mutante: srv266 - package/kernel upgrades
07:35 mutante: powercycled downed owa3, installed kernel upgrades on owa1-3 (but they are also waiting to be repurposed, RT:2511)
04:05 Tim: on hume: running updateCollation.php --dry-run on ptwiki to determine whether there will be any key truncation issues with bug 35632
02:48 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Wed Jul 11 02:48:18 UTC 2012
02:25 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Wed Jul 11 02:25:19 UTC 2012

July 10

23:47 AaronSchulz: Doing shards c-f
23:47 maplebed: put two new swift front ends into rotation ms-fe3 and ms-fe4
23:40 AaronSchulz: Doing shards 8-b
23:34 AaronSchulz: Doing shards 4-7
23:31 AaronSchulz: copyFileBackend.php run rate for above processes is at /home/aaron/NFStoSwiftCopyRate, currently 50
23:26 AaronSchulz: Running copyFileBackend.php for zhwiki public zone shards 0-3 on hume
23:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:17 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:12 logmsgbot: asher synchronized wmf-config/CommonSettings.php 'moving parsercache from db40 to pc1'
22:05 maplebed: put swift backends ms-be9-12 in rotation for containers (weight 30) and objects (weight 5)
21:17 maplebed: rebooting new ms-be6-8 to change BIOS setting to boot from disk
20:26 logmsgbot: mlitn Finished syncing Wikimedia installation... :
20:26 RobH: shutting down sq36 for hardware troubleshooting
19:38 LeslieCarr: restarted frozen pdns on ns2
19:28 logmsgbot: mlitn Started syncing Wikimedia installation... :
19:15 cmjohnson1: sq36-requires a hard shutdown-unresponsive to mgmt. rt-3254
19:05 RoanKattouw: chmod g+w /home/wikipedia/common/php-1.20wmf7/cache/l10n/
19:03 RoanKattouw: chown l10nupdate:wikidev /home/wikipedia/common/php-1.20wmf7/cache/l10n/
18:58 hashar: reworked the git repository in /home/wikipedia/conf/httpd , manually synced changes from svn to the git repo
18:12 RobH: srv266 shutdown for chris
18:06 cmjohnson1: srv266 being brought down for an extended period of time to run diagnostic tests
17:25 cmjohnson1: srv266 shutting down for HW troubleshooting rt-2896
16:33 mutante: package upgrades and kernel on niobium
16:28 mutante: powercycling niobium
15:49 mutante: powercycling mw1008,mw1070,mw1073
14:10 logmsgbot: reedy synchronized hastidy
13:26 logmsgbot: tstarling synchronized live-1.5/robots.php
13:17 logmsgbot: tstarling synchronized live-1.5/favicon.php
13:13 logmsgbot: tstarling synchronized live-1.5/favicon.php
12:53 Tim: deploying favicon.php test alias
12:41 logmsgbot: tstarling synchronized live-1.5/favicon.php
12:30 logmsgbot: tstarling synchronized live-1.5/favicon.php
12:11 logmsgbot: tstarling synchronized live-1.5/favicon.php
12:10 logmsgbot: tstarling synchronized live-1.5/favicon.php
12:03 logmsgbot: tstarling synchronized live-1.5/favicon.php
12:02 logmsgbot: tstarling synchronized live-1.5/favicon.php
11:58 logmsgbot: tstarling synchronized live-1.5/favicon.php
10:10 logmsgbot: reedy synchronized images/sul/ 'crushed'
10:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
07:49 logmsgbot: tstarling synchronized live-1.5/favicon.php
07:46 logmsgbot: tstarling synchronized live-1.5/favicon.php
07:41 logmsgbot: tstarling Finished syncing Wikimedia installation... :
06:32 logmsgbot: tstarling Started syncing Wikimedia installation... :
06:14 logmsgbot: tstarling Started syncing Wikimedia installation... :
06:06 Tim: removing php-*/cache/l10n/l10nupdate* and running scap with the new version of scap from I8bcd2817
05:01 Tim: on fenari: running git submodule update --init in /var/lib/l10nupdate/mediawiki/
03:56 Tim: testing new scheme for LU involving not pushing out LU files to all apaches
02:49 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Tue Jul 10 02:49:16 UTC 2012
01:53 brion: test.wikipedia.org fails with 'TrustedXFF: hosts file missing.'
01:44 Tim: on manganese: ran date -s "`date`" to make sure that isn't the cause of the high CPU usage during clone, it wasn't
01:44 Tim: running rebuildLocalisationCache.php for testwiki
01:43 brion: test.wikipedia.org broken again, "No localisation cache found for English."
00:51 brion: test.wikipedia is broken loading Cite.php

July 9

23:53 Tim: deleting and recreating php-1.20wmf7 to test a script
20:51 logmsgbot: reedy synchronized php-1.20wmf7/includes/media/FormatMetadata.php
20:51 logmsgbot: reedy synchronized php-1.20wmf6/includes/media/FormatMetadata.php
20:30 pp-pdf3: upgraded mwlib.epub
20:30 pp-pdf1: upgraded mwlib.epub
20:30 pp-pdf2: upgraded mwlib.epub
20:27 cmjohnson1: db10 swapping disk1 for new disk
19:41 logmsgbot: reedy synchronized php-1.20wmf6/includes/WikiPage.php
19:37 logmsgbot: reedy synchronized php-1.20wmf7/includes/WikiPage.php
18:13 logmsgbot: reedy synchronized php-1.20wmf7/extensions/CodeReview/ 'CR to trunk to fix fatals'
18:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki and testwiki to 1.20wmf7
15:45 logmsgbot: reedy synchronized wmf-config/ExtensionMessages-1.20wmf7.php
15:43 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.20wmf7 and rebuilding localisation caches
15:01 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.20wmf7 and rebuilding localisation caches
14:54 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.20wmf7 and rebuilding localisation caches
14:47 logmsgbot: reedy synchronized php-1.20wmf7
14:17 Reedy: Copying php-1.20wmf7 from /tmp to /h/w/c on fenari
13:26 Reedy: Running ddsh -cM -g mediawiki-installation -o -oSetupTimeout=30 -F30 -- "sudo -u mwdeploy rm -rf /usr/local/apache/common/php-1.20wmf3"
13:23 Reedy: Running ddsh -cM -g mediawiki-installation -o -oSetupTimeout=30 -F30 -- "sudo -u mwdeploy rm -rf /usr/local/apache/common/php-1.20wmf2"
13:13 Reedy: Removed wmf2 and wmf3 from bits docroots
13:11 mutante: powercycling and upgrading a couple more mw* servers
13:04 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
13:01 mutante: labs.wikimedia.org is now a redirect to labsconsole
12:49 mutante: authdns-update to add labs.wm entry for redirect to labsconsole
08:34 logmsgbot: preilly synchronized wmf-config/CommonSettings.php 'adjust $wgCollectionFormats'
08:16 hashar: pallium, updating jenkins build script with 14666 & 14667
02:26 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Mon Jul 9 02:25:57 UTC 2012

July 8

12:17 apergos: removed HTCPpurger.log.1 and current log ater restart of purger on ms6, /was full. people reporting thumb issues from europe
02:26 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Sun Jul 8 02:26:21 UTC 2012

July 7

16:39 logmsgbot: reedy synchronized docroot/bits/DolphinBrowser/js/ 'Remove BOM'
16:10 logmsgbot: reedy synchronized php-1.20wmf6/extensions/cldr/ 'remove bom'
02:27 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Sat Jul 7 02:27:22 UTC 2012

July 6

21:49 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 38227 - Please enable WikiLove on Turkish Wikipedia'
21:48 binasher: truncating pagetriage tables on enwiki (per bsitu)
21:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 38227 - Please enable WikiLove on Turkish Wikipedia'
21:46 Reedy: Created WikiLove tables on trwiki
21:39 logmsgbot: aaron synchronized php-1.20wmf6/includes/WikiPage.php
21:07 maplebed: removed a bit more load from the swift spinning media container servers by adjusting the ring weights
20:45 logmsgbot: aaron synchronized php-1.20wmf6/includes/WikiPage.php 'rev debug logging'
20:22 logmsgbot: aaron synchronized php-1.20wmf6/includes/Revision.php 'es debug logging'
20:20 logmsgbot: aaron synchronized php-1.20wmf6/includes/WikiPage.php 'es debug logging'
19:49 logmsgbot: aaron synchronized php-1.20wmf6/includes/WikiPage.php 'debug logging'
19:30 logmsgbot: aaron synchronized php-1.20wmf6/includes/WikiPage.php 'debug logging'
19:24 logmsgbot: aaron synchronized php-1.20wmf6/includes/WikiPage.php 'debug logging'
17:28 maplebed: adjusted swift ring files to move container listings off spinning disks (ms-be1-4)
17:04 logmsgbot: reedy synchronized wmf-config/
16:53 Reedy: Running updateCollation.php in foreachwiki on fenari in screen as reedy
16:45 Reedy: Running updateCollation.php against enwiki on fenari in screen as reedy
16:39 Reedy: Running updateCollation.php against commonswiki on fenari in screen as reedy
16:08 logmsgbot: reedy synchronized images/wiki-en.png
15:54 logmsgbot: reedy synchronized images/sul/meta.png
15:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
15:36 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
15:21 cmjohnson1: ms-be5 removing disk0 to replace with good disk
14:38 RobHalsell: authdns-update to correct typo in mgmt dns entry
13:22 mark: Inserted new pybal 1.04 package in the precise-wikimedia APT repository, and upgraded all precise LVS servers
11:31 logmsgbot: tstarling Finished syncing Wikimedia installation... :
11:25 logmsgbot: tstarling Started syncing Wikimedia installation... :
11:20 Tim: running scap to delete docroot files per I48502d90 and Ie3afd137
08:58 mutante: dist-upgrading (unused) db10xx servers
07:43 Tim: also trusted-xff.phps
07:39 Tim: removing some untracked junk from /home/wikipedia/common
07:29 mutante: continue to restart and upgrade downed mw10xx servers
02:26 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Fri Jul 6 02:26:22 UTC 2012
01:35 binasher: updated squid redirector to cover wiki(quotes|books|versity)
00:13 Ryan_Lane: restarting gerrit
00:09 Ryan_Lane: force running puppet on manganese, it'll restart gerrit

July 5

23:54 Ryan_Lane: force running puppet on formey and manganese, since a config change is involved, it's going to restart
23:45 Ryan_Lane: restarting gerrit on manganese
23:38 Ryan_Lane: upgrading gerrit on formey
23:21 Ryan_Lane: updating database for gerrit
23:09 Ryan_Lane: upgrading gerrit on manganese
23:08 Ryan_Lane: stopping gerrit on manganese and disabling puppet
23:07 Ryan_Lane: stopping gerrit on formey and disabling puppet
22:47 logmsgbot: preilly synchronized wmf-config/mobile.php 'add subdomain check'
22:34 logmsgbot: reedy synchronized wmf-config/
22:17 logmsgbot: reedy synchronized wmf-config/ 'Tidying config for randomrootpage'
21:55 logmsgbot: reedy synchronized wmf-config/ 'Strike 2'
21:50 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'revert'
21:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Random root page enabled everywhere but wikipedias'
21:47 logmsgbot: kaldari Finished syncing Wikimedia installation... :
21:34 logmsgbot: kaldari Started syncing Wikimedia installation... :
21:18 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'Bumping AFTv5 lottery percentage to 20% of en.wiki'
21:15 LeslieCarr: powercycling unresponsive mw1116
21:09 maplebed: added new ms-be pmtpa hosts to DNS
20:03 logmsgbot: preilly synchronized php-1.20wmf6/extensions/MobileFrontend 'fix cache issue with token'
19:14 Jeff_Green: *somebody* purged binary logs on blondel
18:58 logmsgbot: preilly synchronized php-1.20wmf6/extensions/MobileFrontend 'fix cache issue'
18:54 mark: Upgraded pybal on all precise LVS servers
18:43 mark: Built new pybal_1.03 package and inserted it into the precise-wikimedia APT repository
17:41 mutante: powercycling db1048, mw1136, mw1046
17:32 logmsgbot: midom synchronized wmf-config/InitialiseSettings.php 'reenabling db40'
17:30 mutante: more powercycling and upgrading: mw1036, mw1134, mw1043 ..
17:26 mutante: powercycling db1009,db1010
17:22 mutante: powercycling db1027,db1028
17:14 mutante: powercycling db1013
16:50 mutante: powercycling downed db1015
16:44 mutante: powercycling and upgrading more mw10xx servers, 1017,1023,1025 ...
16:27 mutante: mw1002, mw1007,mw1009,mw1011 - crashed,powercycling,dist-upgrading+kernel,reboot
16:18 mutante: powercycling mw1002
15:37 mutante: argon back up with new kernel,mysql,grub,.. looks happy afaict
15:33 mutante: argon (limesurvey) fscked, dist-upgrading
15:28 mutante: powercycling argon
15:25 hashar: updated Jenkins configuration on gallium : Updating f407ebe..4b669b9
14:08 cmjohnson1: HUME replacing disk 0
14:07 logmsgbot: midom synchronized wmf-config/InitialiseSettings.php 'disabling parser cache for now'
11:54 mark: Inserted new pybal_1.02 package into APT distribution precise-wikimedia
11:17 mark: Installed new pybal snapshot build for testing on lvs1005
11:13 mutante: add missing Russian locales on singer, run localegen, run ru.planet update
02:26 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Thu Jul 5 02:26:45 UTC 2012

July 4

16:32 mutante: wikidata.org on now - redirect purged from squids
15:43 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'imports'
15:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable LQT on fiwikimedia'
15:39 Reedy: Killed php-1.20wmf5 localisation cache from mediawiki-installation group
15:37 Reedy: Created LQT tables on fiwikimedia
15:25 mutante: wikidata.org works now, besides old redirect may still be cached on cp* boxes (not purged by purgeList.php via multicast?). http://www.wikidata.org/?notcached
15:06 mutante: sync-common-file extract2.php, apache-graceful-all
15:04 logmsgbot: dzahn synchronized extract2.php
13:23 mutante: apache-graceful-all to add wikidata.org virtual host
13:21 mutante: svn commiting gerrit 9874, sync-apache
13:14 mutante: git pull in /h/w/common/docroot . adding wikidata.org files on fenari. , then "sync-docroot"
12:44 logmsgbot: dzahn synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
12:44 mutante: updating/syncing interwiki cache
09:32 hashar: swift-container-auditor seems to get down from time. Nagios reporting 0 processes at 8:15am and 9:25am UTC (I guess it get restarted automatically by puppet)
07:34 hashar: updating Jenkins copy of integration/jenkins from 0f069c3 to e264d1b. Bring new ant script + update to testswarm fetcher
07:09 Tim: on srv193: ran dpkg --set-selections to revert holds on php5 packages and ran apt-get upgrade
07:07 Tim: on srv193: fixing broken PHP packages causing puppet failure, nothing in the server admin log about them so I assume they were installed by accident
06:21 Tim: deployed Idb6d9a8b and restarting apaches
06:11 Tim: deployed Id7008681 and restarting apaches
05:45 Tim: reniced apache processes to level 0
05:04 Tim: deploying apache nice level change per RT #664
03:59 Tim: on mw1: experimenting with renice methods for RT 664
02:26 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Wed Jul 4 02:26:39 UTC 2012

July 3

23:03 logmsgbot: mlitn synchronized php-1.20wmf6/extensions/ArticleFeedbackv5/
22:36 logmsgbot: preilly synchronized php-1.20wmf6/extensions/MobileFrontend 'weekly update'
22:27 hashar: updating testswarm submitter on gallium
21:01 logmsgbot: mlitn Finished syncing Wikimedia installation... :
20:37 logmsgbot: mlitn Started syncing Wikimedia installation... :
20:18 logmsgbot: asher synchronized wmf-config/db.php 'lowering db32 weight'
19:34 logmsgbot: asher synchronized wmf-config/db.php 'lowering db36 weight'
19:31 logmsgbot: asher synchronized wmf-config/db.php 're-add db36, db32 (low weight), es3 (innodb)'
18:11 RobH: virt1006 mgmt serial not set correctly, fixed
17:51 RobH: investigating stat1001 power issue
17:22 RobH: fluorine offlining to test disks
17:22 RobH: pulling helium offline for disk testing with fluorine disks
17:10 RobH: db1047 disk0 rebuild in progress
17:04 RobH: replacing bad disk in db1047
16:45 logmsgbot: reedy synchronized wmf-config/ 'Various config changes'
16:30 logmsgbot: reedy synchronized docroot/mediawiki/xml/export-0.7.xsd
16:15 Jeff_Green: silicon gets dist-upgrade & reboot
16:15 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wgCheckSerialized is deaded'
03:53 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 're-enable API action=purge on commonswiki'
02:27 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Tue Jul 3 02:27:02 UTC 2012

July 2

22:33 binasher: rebooted db36 for kernel upgrade
22:25 logmsgbot: asher synchronized wmf-config/db.php 'temp pulling db36'
22:02 brion: fun with routers in tampa, wikis down
21:48 maplebed: rebooted emery - it's been unresponsive for 3 days.
18:34 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
18:30 hashar: set up ignore file in httpd configuration directory
18:23 logmsgbot: reedy synchronized wmf-config/ 'Enable WikimediaShopLink on enwiki'
18:11 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: 273 wikipedias to 1.20wmf6
18:03 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf6
15:21 logmsgbot: hashar synchronized wmf-config/CommonSettings.php '/etc/wikimedia-realm detection https://gerrit.wikimedia.org/r/13888'
15:18 logmsgbot: hashar synchronized docroot/bits/static-master '(bug 37245) docroot 'static-master' for beta bits'
15:04 mutante: authdns-update to switch jobs.wm redirect to wikimedia-lb to fix SSL cert mismatch (RT-3071)
14:55 mark: Reboot of cr1-sdtpa did not fix the RE packet loss issue... therefore unlikely to be leap second related
14:41 mark: Rebooting cr1-sdtpa
14:37 mark: Shutdown PyBal BGP sessions on cr1-sdtpa
14:34 mark: Shutdown BGP session to 2828 on cr1-sdtpa
13:36 hashar: db12 suffering some 1400sec (and growing) replag. mysqldump in progress on that host.
12:35 mutante: installing upgrades on fenari (linux-firmware linux-libc-dev..)
12:27 mutante: rebooting gallium one more time to install kernel
12:26 mutante: upgrading kernel on gallium
12:23 logmsgbot: hashar synchronized live-1.5/CREDITS
11:31 mark: Now we have packet loss within pmtpa/sdtpa... reverting change
10:57 mark: Problems on one of two pmtpa-eqiad waves; raised OSPF metric to 60 to failover traffic to the other link
10:50 Tim: fixing leap second issue on bastion1 by rebooting it
10:47 Tim: fixed leap second issue on bastion-restricted
09:56 Tim: fixing leap second issue on virt1,virt2,virt3,virt4,virt5
09:52 Tim: fixing leap second issue on aluminium,gallium,manganese
09:47 Tim: fixing leap second issue on formey,grosley,hooper,sanger,sockpuppet
09:43 Tim: on fenari: fixed leap second issue with the mozilla method
09:39 apergos: rebooting gallium, it's pretty unhappy (maybe related to leap second issue)
08:14 logmsgbot: hashar: srv190 srv266 srv281 timeouts on sync-file
08:14 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'Bug 37457 - fix import sources for viwikibooks'
08:11 hashar: Stopped Jenkins on gallium. It is not doing anything anyway. Asked to reboot box RT #3208
02:53 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Mon Jul 2 02:53:51 UTC 2012
02:28 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Mon Jul 2 02:28:48 UTC 2012
01:48 Tim: kill -CONT on populateRevisionSha1.php processes
00:47 Tim: on nfs1: trying leap second fix suggested at https://bugzilla.mozilla.org/show_bug.cgi?id=769972#c5
00:26 logmsgbot: tstarling synchronized wmf-config/db.php 'reduce db32 read load to zero due to persistent lag'
00:12 Tim: switched enwiki back to r/w
00:12 logmsgbot: tstarling synchronized wmf-config/db.php
00:06 Tim: on hume: stopped all populateRevisionSha1.php processes with kill -STOP
00:03 logmsgbot: reedy synchronized wmf-config/db.php 's1/enwiki into readonly'

July 1

19:12 logmsgbot: reedy synchronized php-1.20wmf6/extensions/WikimediaMaintenance/ 'Update to master for hashar'
17:55 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php 'more logging'
17:45 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php 'more logging'
17:43 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php 'more logging'
17:32 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php
17:30 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php
16:53 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php
16:48 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php 'logging'
12:54 notpeter: also going to reboot all pmtpa search nodes. not in prod, but are still freaking out from leap second bug.
05:33 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php 'logging'
04:25 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Sun Jul 1 04:25:25 UTC 2012
04:06 Ryan_Lane: virt1000 is back up, rebooting virt0
04:01 Ryan_Lane: rebooting virt1000
03:16 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Sun Jul 1 03:16:39 UTC 2012
01:43 notpeter: that worked. restarting all remaining search nodes.
01:39 notpeter: problem with lucene persisting through service restart, but not node restart. restarting en pool nodes.
01:20 paravoid: restarting opendj (nfs1/nfs2), load spike, possibly related to leap second
00:51 notpeter: search1004 dead. powercycling.
00:50 notpeter: based on ganglia evidence, lucene seems to have been affected by leap second bug. restartig each instance, one minute wait in between

2000s

Archive 1: 2004 Jun - 2004 Sep
Archive 2: 2004 Oct - 2004 Nov
Archive 3: 2004 Dec - 2005 Mar
Archive 4: 2005 Apr - 2005 Jul
Archive 5: 2005 Aug - 2005 Oct, with revision history 2004-06-23 to 2005-11-25
Archive 6: 2005 Nov - 2006 Feb
Archive 7: 2006 Mar - 2006 Jun
Archive 8: 2006 Jul - 2006 Sep
Archive 9: 2006 Oct - 2007 Jan, with revision history 2005-11-25 to 2007-02-21
Archive 10: 2007 Feb - 2007 Jun
Archive 11: 2007 Jul - 2007 Dec
Archive 12: 2008 Jan - 2008 Jul
Archive 12a: 2008 Aug
Archive 12b: 2008 Sept
Archive 13: 2008 Oct - 2009 Jun
Archive 14: 2009 Jun - 2009 Dec

2010s

2020s