Server admin log/Archive 21

From Wikitech
Jump to: navigation, search

January 29

  • 22:27 RobH: torrus is all happy again, huzzah
  • 22:25 logmsgbot: maxsem Finished syncing Wikimedia installation... : https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2013-01-29
  • 22:11 logmsgbot: maxsem Started syncing Wikimedia installation... : https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2013-01-29
  • 22:02 Ryan_Lane: rebooting labstore2
  • 22:01 RobH: updating torrus files via puppet, sorry if torrus crashes out, its bitchy
  • 22:00 notpeter: temp stopping puppet on brewster
  • 21:58 Ryan_Lane: rebooting labstore1
  • 21:57 Ryan_Lane: stopping all gluster services
  • 21:38 cmjohnson1: authdns update adding mgmt ip's to zone files for hpm misc servers in eqiad
  • 21:33 Ryan_Lane: stopping gluster services on labstore1, this will affect nfs mounts (like ssh keys and public data exports)
  • 20:43 RobH: updated rt config, pmtpa comments now come from pmtpa-comment, not rt-comment to match the rest of the queues configuration
  • 19:51 cmjohnson1: ms-be1004 going down for h/w replacement
  • 19:47 hashar: jenkins: updating all jobs
  • 19:31 cmjohnson1: ms-be1003 going down to replace h/w
  • 19:29 Ryan_Lane: upgrading glusterfs to 3.3.1 from 3.3.0 and replacing old glusterfs package with new packages
  • 19:09 cmjohnson1: ms-be1002 going down to replace h/w
  • 18:37 cmjohnson1: ms-be1001 going down to replace h/w
  • 17:57 andrewbogott: restarted apache and memcached on virt0 (apparently fixing main page errors)
  • 17:10 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php
  • 14:14 mark: powercycled sodium
  • 08:20 logmsgbot: pgehres synchronized php-1.21wmf8/extensions/CentralNotice/ 'Updating CentralNotice to master'
  • 04:58 logmsgbot: tstarling synchronized php-1.21wmf6/cache/trusted-xff.cdb
  • 04:57 logmsgbot: tstarling synchronized php-1.21wmf7/cache/trusted-xff.cdb
  • 04:53 logmsgbot: tstarling synchronized php-1.21wmf8/cache/trusted-xff.cdb
  • 04:51 logmsgbot: tstarling synchronized php-1.21wmf8/cache/trusted-xff.cdb
  • 04:47 logmsgbot: tstarling synchronized php-1.21wmf8/cache/trusted-xff.cdb
  • 04:36 logmsgbot: tstarling synchronized php-1.21wmf8/cache/trusted-xff.cdb 'fixing corrupted CDB file'
  • 04:36 logmsgbot: tstarling synchronized php-1.21wmf7/cache/trusted-xff.cdb 'fixing corrupted CDB file'
  • 02:55 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Tue Jan 29 02:55:44 UTC 2013
  • 02:30 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Tue Jan 29 02:30:42 UTC 2013
  • 02:17 Ryan_Lane: deleting sysadmin and netadmin roles from ldap
  • 01:55 Ryan_Lane: adding ldap entries for combined sysadmin/netadmin role, projectadmin
  • 01:29 K4-713: updated payments cluster to ec4d3f24862559
  • 00:44 RobH: typo, knsq18
  • 00:44 RobH: squid backend on knsq48 rebuilding
  • 00:36 RobH: crazy memory and cpu spike on knsq18 for no apparent reason, restarting process (memory leak?)

January 28

  • 23:27 K4-713: synchronized payments cluster to 1948266b33e905b
  • 22:40 logmsgbot: pgehres synchronized wmf-config/CommonSettings.php 'Enabling CN:Translate on testwiki'
  • 22:37 hashar: jenkins: applied pep8 and pyflakes jobs to sartoris repository.
  • 22:35 logmsgbot: pgehres Finished syncing Wikimedia installation... : Dark deploying Translate integration with CentralNotice
  • 22:20 logmsgbot: pgehres Started syncing Wikimedia installation... : Dark deploying Translate integration with CentralNotice
  • 21:26 logmsgbot: pgehres synchronized php-1.21wmf8/extensions/LandingCheck 'Updating LandingCheck to master'
  • 20:42 LeslieCarr: cleared a lot of 0 size badly named logs on fluorine:/a/mw-log
  • 20:41 sbernardin1: taking down osm-web1 for h/w installation
  • 19:25 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 19:09 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf8
  • 19:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 18:31 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 18:08 paravoid: upgrading ffmpeg/libav for USN-1705-1/USN-1706-1 on all image/video scalers in eqiad/pmtpa
  • 18:00 sbernardin1: shutting down ms-be11 for h/w replacement
  • 15:29 paravoid: upgraded & rebooting emery, 208 days bug
  • 15:06 mark: Pooled cp3009 and cp3010 in the esams upload pool
  • 02:52 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Mon Jan 28 02:52:53 UTC 2013
  • 02:30 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Mon Jan 28 02:29:55 UTC 2013

January 27

  • 23:50 logmsgbot: reedy synchronized php-1.21wmf7/extensions/CentralAuth/
  • 23:24 logmsgbot: reedy synchronized php-1.21wmf8/extensions/CentralAuth/
  • 23:11 drdee: enabled sampling on Thank_You_Main and a country filter on Emery
  • 07:16 logmsgbot: aaron synchronized php-1.21wmf8/maintenance/nextJobDB.php 'deployed f0f869dfabc1cedcedd5df972fbe6c279789a498 '
  • 04:48 logmsgbot: aaron synchronized php-1.21wmf8/maintenance/nextJobDB.php 'removed live hack'
  • 04:47 logmsgbot: aaron synchronized php-1.21wmf8/maintenance/runJobs.php
  • 02:55 logmsgbot: aaron synchronized php-1.21wmf8/maintenance/runJobs.php
  • 02:51 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Sun Jan 27 02:51:05 UTC 2013
  • 02:48 logmsgbot: aaron synchronized php-1.21wmf8/maintenance/nextJobDB.php 'quick debugging'
  • 02:27 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Sun Jan 27 02:26:57 UTC 2013
  • 01:30 logmsgbot: aaron synchronized php-1.21wmf8/maintenance/nextJobDB.php 'deployed 93869b8d37b6b7999190b378cb6813f6317b0b10'
  • 01:02 LeslieCarr: rpd core caused bgp to restart (even though it wasn't supposed to) - causing routing churn
  • 00:55 LeslieCarr: core dumping cr1-sdtpa - 1% chance of minor network issue

January 26

  • 20:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 17:37 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 02:54 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Sat Jan 26 02:54:55 UTC 2013
  • 02:38 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: aawiki to 1.21wmf8
  • 02:29 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Sat Jan 26 02:29:39 UTC 2013
  • 02:28 logmsgbot: aaron synchronized php-1.21wmf8/maintenance/nextJobDB.php 'logging'
  • 02:21 logmsgbot: aaron synchronized php-1.21wmf8/includes/job/JobQueueDB.php
  • 01:21 LeslieCarr: htcp purging across datacenters now "works". dobson is now receiving purge requests on multicast group 239.128.0.112 port 4827 and transmitting them via udpmcast.py (started by rc.local) to hooft in esams
  • 01:14 LeslieCarr: deactivating multicast for 1 minute in order to try and flush the multicast forwarding table

January 25

  • 23:57 mutante: installing mysql-client/common package upgrades on kaulen (bugzilla)
  • 23:07 LeslieCarr: on cr1 and cr2 on eqiad allowing for unicast-reverse-path feasible-paths
  • 22:29 LeslieCarr: clearing multicast joins in order to troubleshoot multicast pruning
  • 20:47 logmsgbot: reedy synchronized live-1.5/
  • 20:47 logmsgbot: reedy synchronized docroot
  • 19:13 Nemo_bis: Job queue not being run somewhat: no FuzzyBot on mw:Communication for a hour, en.wiki and Commons unhappy.
  • 18:54 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 18:49 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 18:47 Reedy: Created translate tables on brwikimedia
  • 18:09 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 17:31 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 16:51 ^demon: formey: mediawiki svn repository is now read-only. long live git.
  • 02:53 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Fri Jan 25 02:53:00 UTC 2013
  • 02:27 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Fri Jan 25 02:27:20 UTC 2013
  • 01:15 binasher: moved s[2-7] analytics slave cnames to pmtpa dbs
  • 01:07 mutante: planet is now an IPv6-enabled service
  • 00:38 mutante: DNS update - set non-autoconfig IPv6 and reverse for zirconium
  • 00:11 mutante: DNS update - push reverse IPv6 for zirconium

January 24

  • 23:49 mutante: installing package upgrades on zirconium
  • 23:49 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/GettingStarted
  • 23:48 logmsgbot: olivneh synchronized php-1.21wmf7/extensions/GettingStarted
  • 22:55 mutante: updating all planets
  • 22:55 andrewbogott: changed all lab sudoer policies to !authenticate
  • 22:55 andrewbogott: changed all labs sudoers from "sudoUser: ALL" to "sudoUser: project-<projname>"
  • 22:28 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php
  • 22:04 logmsgbot: olivneh synchronized php-1.21wmf7/extensions/E3Experiments
  • 22:03 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/E3Experiments
  • 22:02 logmsgbot: olivneh synchronized php-1.21wmf7/extensions/EventLogging
  • 21:59 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/EventLogging
  • 21:39 paravoid: deploying squid config for upload's /monitoring/ switch from url_regex to urlpath_regex
  • 21:28 binasher: fixed dbtree to support our current situation of db-pmpta containing eqiad write masters
  • 21:22 sbernardin: taking srv278 down for decommissioning per paravoid
  • 21:03 K4|foodish: Synchronized payments cluster to fe035f6b5a55946b729f
  • 20:56 paravoid: gradual restart of ms-fe[1-4] proxies (switching scaling to eqiad)
  • 19:24 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend/includes/skins/SkinMobile.php 'https://gerrit.wikimedia.org/r/#/c/45562/'
  • 19:23 logmsgbot: maxsem synchronized php-1.21wmf7/extensions/MobileFrontend/includes/skins/SkinMobile.php 'https://gerrit.wikimedia.org/r/#/c/45562/'
  • 19:19 logmsgbot: aaron synchronized php-1.21wmf8/thumb.php 'deployed c6d478f73f33f051c1060ffe4b9c5855e07d358f'
  • 18:42 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Removed unused NFS file backends'
  • 18:42 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php
  • 18:16 logmsgbot: awjrichards synchronized php-1.21wmf7/extensions/MobileFrontend 'touch files'
  • 18:16 logmsgbot: awjrichards synchronized php-1.21wmf8/extensions/MobileFrontend 'touch files'
  • 18:15 logmsgbot: awjrichards Finished syncing Wikimedia installation... : Redploy rolled back MobileFrontend changes from Jan 17
  • 18:00 cmjohnson1: authdns update "fixing typo on db1060.mgmt.eqiad"
  • 17:56 logmsgbot: awjrichards Started syncing Wikimedia installation... : Redploy rolled back MobileFrontend changes from Jan 17
  • 11:06 logmsgbot: mlitn synchronized wmf-config/CommonSettings.php 'Reinstate AFTv5 test groups'
  • 10:25 logmsgbot: mlitn synchronized wmf-config/CommonSettings.php 'Reinstate AFTv5 test groups'
  • 04:08 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
  • 04:08 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
  • 03:28 Ryan_Lane: powercycling formey
  • 03:11 mutante: DNS update - switching planet over to zirconium / new planet
  • 02:53 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Thu Jan 24 02:52:58 UTC 2013
  • 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Thu Jan 24 02:28:21 UTC 2013

January 23

  • 23:57 logmsgbot: dzahn gracefulled all apaches
  • 22:53 RobH: calcium is my testbed server, ignore its bitching
  • 22:28 mutante: starting NTP on mw1008 (Tick tock, on the clock But the party don't stop, no)
  • 22:20 paravoid: netapp: nas1001-b cf giveback
  • 22:10 RobH: authdns-update correcting camera fqdn
  • 22:00 Ryan_Lane: restarting pdns on ns0
  • 21:56 cmjohnson1: authdns update adding mgmt ip's to zone files for 10 misc servers in eqiad
  • 21:30 Ryan_Lane: restarting opendj on virt0
  • 21:12 mutante: restarting pdns on ns1
  • 21:03 RobH: authdns-update to push db61 into sandbox
  • 20:56 RobH: swapped port 3/39 on csw1-sdtpa from vlan 2 to 102
  • 20:24 mutante: removing mw1072 (hardware fail, RT-4381) from "apaches" dsh group
  • 20:22 logmsgbot: dzahn gracefulled all apaches
  • 19:43 mutante: puppet signing mw1085 (was sitting there like a fresh install)
  • 19:41 RobH: stopping puppet on brewster cuz i wanna locally hack and reboot 1 time on db61 without partman auto kicking in
  • 19:28 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Switched all non-wikipedias to 1.21-wmf8
  • 19:07 RobH: db61 coming offline, and going into otrs sandbox service
  • 19:05 RobH: authdns-update
  • 18:03 binasher: amended wikiadmin grants on s7 to allow new jobrunners access to centralauth
  • 17:44 paravoid: running authdns-update - a bunch of cleanups, r3861:3867
  • 15:09 mark: Added missing pim interface ae0.400 statement on cr2-pmtpa which prevented eqiad purges from reaching pmtpa
  • 04:11 logmsgbot: tstarling synchronized wmf-config/db-eqiad.php 'cleanup, added pmtpa host/IP mappings'
  • 04:09 logmsgbot: tstarling synchronized wmf-config/db-pmtpa.php 'back to read/write mode (with eqiad master)'
  • 02:49 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Wed Jan 23 02:49:23 UTC 2013
  • 02:26 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Wed Jan 23 02:26:04 UTC 2013
  • 02:23 Tim: reloading all apaches
  • 02:19 Tim: running sync-apache to fix stale eqiad apache configuration
  • 02:14 Tim: added eqiad apaches to /etc/dsh/group/apaches
  • 01:23 logmsgbot: aaron synchronized php-1.21wmf8/includes/api/ApiUpload.php 'deployed 2a4ad3e32cf4f6814554ebcf09ac0250546a2549'
  • 01:22 logmsgbot: aaron synchronized php-1.21wmf7/includes/api/ApiUpload.php 'deployed 2a4ad3e32cf4f6814554ebcf09ac0250546a2549'
  • 01:07 Tim: ran scap-1 on tmh1001 and tmh1002
  • 01:05 Tim: added tmh1001 and tmh1002 to mediawiki-installation

January 22

  • 23:58 Tim: on db1006: fixing permissions for root@fenari
  • 23:54 Tim: wikiadmin permissions for eqiad were broken, attempting to use CIDR notation, fixing them on all masters
  • 23:04 Ryan_Lane: Alex Monk's echo changes to OpenStackManager deployed on labsconsole
  • 23:04 Ryan_Lane: echo enabled on labsconsole
  • 23:03 Ryan_Lane: labsconsole upgraded to 1.21wmf8
  • 22:38 Tim: on neon: cleaned up /var/lib/puppet/clientbucket/* since it was out of disk space
  • 22:29 logmsgbot: catrope synchronized php-1.21wmf8/resources/jquery/jquery.client.js '8a06466e1697f58717e92532437be642a796d5a1'
  • 22:29 logmsgbot: catrope synchronized php-1.21wmf7/resources/jquery/jquery.client.js '8a06466e1697f58717e92532437be642a796d5a1'
  • 21:55 cmjohnson1: msw-a7-eqiad power was accidently removed while switch was being moved
  • 21:45 ^demon: gerrit restarting. don't panic.
  • 20:26 logmsgbot: catrope synchronized php-1.21wmf8/includes/api/ApiQueryImageInfo.php 'Fix API error in imageinfo'
  • 20:26 logmsgbot: catrope synchronized php-1.21wmf7/includes/api/ApiQueryImageInfo.php 'Fix API error in imageinfo'
  • 20:12 AaronSchulz: Recompiled textvc on all hosts
  • 19:37 Ryan_Lane: upgrading labsconsole to 1.21wmf8
  • 19:03 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Disabling ReaderFeedback'
  • 19:03 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Disabling ReaderFeedback'
  • 18:59 mutante: restarted pdns on ns1, failed during update
  • 18:58 mutante: DNS update - adding ca./ja. planet to zirconium for testing new planet
  • 18:51 mutante: DNS update - deleting wikimania2006/2007
  • 18:14 logmsgbot: root synchronized wmf-config/db-eqiad.php 'eqiad read-write!'
  • 17:59 notpeter: moving jobrunning to eqiad
  • 17:54 logmsgbot: aaron synchronized php-1.21wmf7/includes/Block.php 'deployed b0d5ff9bfb09f15ccbab6849564511d372ecd8d4'
  • 17:46 notpeter: authdns-update
  • 17:42 Jeff_Green: db1013 wiped, reimaged, powered off, and un-monitored
  • 17:40 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'setting pmtpa to readonly'
  • 17:14 notpeter: dns svn repo on sockpuppet has changes staged and checked in, not yet deployed on dobson
  • 17:05 notpeter: staging cname switch for pmtpa/eqiad dbs on sockpuppet
  • 15:22 mark: Changed AS14907->AS43821 routing
  • 12:26 ^demon: restarted ircecho on manganese
  • 10:30 Tim: also replaced /etc/sudoers on fenari and hume. Hume needs a puppet change which will follow shortly
  • 10:18 hashar: gallium : restarted puppet
  • 10:14 hashar: jenkins updated all plugins and restarting
  • 10:09 Tim: on searchidx2: replaced /etc/sudoers with the distro default so that /etc/sudoers.d/* from puppet can take effect
  • 06:48 Tim: on eqiad upload varnishes: purged /wikipedia/it/b/bc/Wiki.png at user request. varnishhtcpd appears to be totally broken.
  • 02:48 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Tue Jan 22 02:48:37 UTC 2013
  • 02:26 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Tue Jan 22 02:26:14 UTC 2013

January 21

  • 23:11 Tim: on sockpuppet: fixed puppet checkout, switching branch from mikepatch1 to production, and then did fetch&&rebase for good measure
  • 22:52 Ryan_Lane: adding virt9-11 entries in dns
  • 22:49 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 20:22 mark: Rerouted AS43821->AS14907 traffic
  • 19:41 mark: Restarting knsq* upload frontends manually
  • 19:34 paravoid: deploying squid config for upload's /monitoring/
  • 19:14 mark: Restarting amssq* upload frontends in a slow loop
  • 19:11 mark: Restarted oversized frontend on amssq50
  • 18:55 paravoid: starting knsq19 backend squid
  • 18:50 mark: Restarted knsq16 backend minus two disks
  • 18:40 mark: Started amssq56 squid instances
  • 18:38 mark: Started knsq16 minus one disk
  • 18:37 mark: Started knsq18 minus one disk
  • 18:32 mark: power cycled amssq56
  • 18:28 mark: Rebooting knsq24
  • 18:05 mark: Rerouted AS43821->AS14907 traffic
  • 17:53 paravoid: stopping knsq19 backend squid
  • 17:46 paravoid: restarting knsq19 backend squid
  • 17:35 paravoid: restarting pybal on lvs1002
  • 17:35 cmjohnson1: db1038 swapping bad disk (slot 2) with new disk
  • 17:27 mark: Rerouted AS14907->AS43821 traffic
  • 16:36 paravoid: depooling, restarting and repooling ms-fe2/3/4 one by one
  • 16:30 paravoid: repooling ms-fe1
  • 16:29 mark: Rerouted AS43821->AS14907 traffic
  • 15:48 logmsgbot: reedy synchronized php-1.21wmf8/includes/EditPage.php
  • 15:45 logmsgbot: reedy synchronized php-1.21wmf7/includes/EditPage.php
  • 15:32 paravoid: depooling ms-fe1 for testing
  • 14:17 hashar: gallium: manually installed pyflakes 44974
  • 11:10 apergos: nagios was dead over the weekend (config broken), fixed in puppet and on spence, now back in action
  • 09:57 hashar: relaying Ryan: he restarted ldap on virt0 (was hung after server restart). nscld was properly falling back to virt1000 but ldap was stuck there too. DNS got restarted.
  • 09:52 hashar: jenkins: jobs refresh completed.
  • 08:14 hashar: jenkins: updating all Jenkins jobs based on d31c92e of integration/jenkins-job-builder-config.git
  • 05:51 Tim: on marmontel: removed MW-specific packages php5-wmerrors, php-luasandbox, php-wikidiff2
  • 02:50 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Mon Jan 21 02:50:20 UTC 2013
  • 02:27 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Mon Jan 21 02:26:59 UTC 2013

January 20

  • 23:56 logmsgbot: tstarling Finished syncing Wikimedia installation... :
  • 23:45 logmsgbot: tstarling Started syncing Wikimedia installation... :
  • 23:40 Ryan_Lane: rebooting virt0 to determine which dimm is bad
  • 19:57 binasher: authdns scenario back to normal
  • 19:40 LeslieCarr: deactivating 2828 as a preferred intermediate route between 14907 and 43821
  • 19:30 binasher: seeing major packetloss between esams and pmtpa, failing out of esams
  • 02:48 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Sun Jan 20 02:48:05 UTC 2013
  • 02:25 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Sun Jan 20 02:25:48 UTC 2013

January 19

  • 22:24 MaxSem: Created ProofredPage table on aswikisource
  • 19:22 logmsgbot: pgehres synchronized wmf-config/reporting-setup.php 'Switching S:FundraiserStatistics read slave back to db1025'
  • 02:46 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Sat Jan 19 02:46:06 UTC 2013
  • 02:24 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Sat Jan 19 02:24:18 UTC 2013
  • 01:54 binasher: testing completed, no longer sending live traffic to eqiad
  • 01:53 binasher: i was sending >40 reqs/sec from europe and asia to eqiad apaches for the last hour for real world non-english testing
  • 00:27 mutante: DNS update - add wikivoyager.org/.de
  • 00:08 RobH: blog is fixed, in that its the software equivalent of spinning plates
  • 00:04 RoanKattouw: Running git fsck on mediawiki/core.git as nobody on manganese

January 18

  • 23:42 RobH: blog is partly offline (unreliable loads) investigating
  • 23:38 RobH: stupid w3 caching on blog won't let me use new wp-victor directory, has to match old .git name or caching errors result
  • 23:34 RobH: blog is being bitchy about new theme versus old caching, reverting to stock and clearing local cache on marmontel
  • 23:26 RobH: blog software updated a few hours ago, forgot to log. just updated theme to git update from gerrit repo.
  • 23:21 logmsgbot: asher synchronized wmf-config/CommonSettings.php 'dc specific poolcounter conf'
  • 23:17 notpeter: stopping all jobrunners including tmh boxes per asher's request while troubleshooting
  • 23:16 mutante: DNS update to add wikimania.asia
  • 23:11 logmsgbot: asher synchronized wmf-config 'eqiad/pmtpa config variances'
  • 22:37 logmsgbot: asher synchronized wmf-config 'revert'
  • 22:36 logmsgbot: asher synchronized wmf-config 'revert'
  • 22:34 logmsgbot: asher synchronized wmf-config 'eqiad/pmtpa config variances'
  • 20:19 logmsgbot: root Finished syncing Wikimedia installation... :
  • 19:49 RobH: authdns-update colby back into sandbox
  • 19:08 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Added ceph file backend configuration'
  • 19:08 logmsgbot: root Started syncing Wikimedia installation... :
  • 19:00 mark: rm -rf /usr/local/apache/common-local on eqiad apaches
  • 18:54 RobH: restarted pdns on linne, it borked on authdns-update
  • 18:48 RobH: authdns-update
  • 18:17 logmsgbot: root Finished syncing Wikimedia installation... :
  • 18:04 logmsgbot: root Started syncing Wikimedia installation... :
  • 18:02 logmsgbot: root Finished syncing Wikimedia installation... :
  • 18:00 mark: Removed mw1085 from mediawiki_installation (acting up)
  • 17:54 logmsgbot: root Started syncing Wikimedia installation... :
  • 17:26 logmsgbot: root Started syncing Wikimedia installation... :
  • 16:11 hashar: gallium jenkins: updated tools/fetch-mw-ext script to wipe the extension destination directory 44648 updated: 09bbb44..e41843b
  • 15:59 hashar: gallium jenkins: updated the tools/fetch-mw-ext script which was not properly fetching the extension. {{gerrit|44644}] updated: cfa82ff..09bbb44
  • 15:18 Reedy: recentchanges.rc_params is now a blob on all wikis
  • 15:14 mark: Deployed MediaWiki on eqiad image scalers
  • 15:11 logmsgbot: root Finished syncing Wikimedia installation... :
  • 15:05 logmsgbot: root Started syncing Wikimedia installation... :
  • 15:00 logmsgbot: root Finished syncing Wikimedia installation... :
  • 14:57 paravoid: running /usr/local/bin/foreachwiki maintenance/cleanupUploadStash.php on hume
  • 14:57 logmsgbot: root Started syncing Wikimedia installation... :
  • 14:49 logmsgbot: reedy synchronized php-1.21wmf8/maintenance/cleanupUploadStash.php
  • 14:49 logmsgbot: reedy synchronized php-1.21wmf7/maintenance/cleanupUploadStash.php
  • 14:31 mark: Corrected srv191:/tmp permissions
  • 14:21 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 13:56 mark: Set rt_cache_rebuild_count to -1 on lvs2 and lvs6
  • 13:50 mark: Powercycled lvs6
  • 13:26 logmsgbot: demon synchronized php-1.21wmf7/includes/resourceloader/ResourceLoaderFileModule.php 'Deploying Idc83a0fe'
  • 13:25 logmsgbot: demon synchronized php-1.21wmf7/includes/resourceloader/ResourceLoader.php 'Deploying Idc83a0fe'
  • 13:25 logmsgbot: demon synchronized php-1.21wmf7/includes/MessageBlobStore.php 'Deploying Idc83a0fe'
  • 12:59 logmsgbot: root Started syncing Wikimedia installation... :
  • 12:57 Tim: added eqiad apaches to mediawiki-installation and started scap, it will take a while
  • 12:31 logmsgbot: tstarling Finished syncing Wikimedia installation... :
  • 12:29 logmsgbot: tstarling Started syncing Wikimedia installation... :
  • 12:19 logmsgbot: root Finished syncing Wikimedia installation... :
  • 12:10 logmsgbot: root Started syncing Wikimedia installation... :
  • 12:03 logmsgbot: root Started syncing Wikimedia installation... :
  • 11:58 mark: Mounted nfs1:/home noatime
  • 11:58 logmsgbot: root Started syncing Wikimedia installation... :
  • 10:31 RoanKattouw: Purged all URLs containing 'wikidata' from Varnish
  • 10:22 logmsgbot: tstarling Started syncing Wikimedia installation... :
  • 05:28 Tim: running sync-apache and gracefulling apaches to hopefully fix reports of occasional wikidata redirect issues
  • 05:04 logmsgbot: tstarling synchronized live-1.5/load.php 're-enabling bits for MF'
  • 03:39 logmsgbot: awjrichards synchronized php-1.21wmf8/extensions/MobileFrontend 'Roll back today's MobileFrontend deployment to 1ded0894cdbc089370a58ba41aeb4db53e5c7345'
  • 03:38 logmsgbot: awjrichards synchronized php-1.21wmf7/extensions/MobileFrontend 'Roll back today's MobileFrontend deployment to 1ded0894cdbc089370a58ba41aeb4db53e5c7345'
  • 03:30 logmsgbot: tstarling synchronized live-1.5/load.php
  • 03:24 Tim: attempting to block MF in bits apache configuration
  • 02:50 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Fri Jan 18 02:50:35 UTC 2013
  • 02:48 Tim: restarted bits apaches due to overload, were totally down
  • 02:27 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Fri Jan 18 02:26:57 UTC 2013
  • 01:44 LeslieCarr: purged mobile varnish cache for awjr
  • 01:43 logmsgbot: awjrichards synchronized php-1.21wmf8/extensions/MobileFrontend 'touch files'
  • 01:42 logmsgbot: awjrichards Finished syncing Wikimedia installation... : Updating MobileFrontend per https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2013-01-17
  • 01:42 logmsgbot: awjrichards synchronized php-1.21wmf7/extensions/MobileFrontend/ 'touch files'
  • 01:35 logmsgbot: dzahn gracefulled all apaches
  • 01:30 logmsgbot: reedy synchronized wmf-config/
  • 01:25 cmjohnson1: authdns update adding db1051-1060 to eqiad mgmt zone file
  • 01:02 logmsgbot: aaron synchronized wmf-config/PrivateSettings.php
  • 00:43 logmsgbot: dzahn gracefulled all apaches
  • 00:42 logmsgbot: dzahn synchronized ./wmf-config/CommonSettings.php
  • 00:42 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
  • 00:40 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
  • 00:20 logmsgbot: dzahn gracefulled all apaches
  • 00:10 logmsgbot: aaron synchronized php-1.21wmf8/extensions/UploadWizard/resources/mw.UploadWizardDetails.js
  • 00:08 logmsgbot: dzahn gracefulled all apaches
  • 00:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'fix whitespace'
  • 00:01 logmsgbot: dzahn gracefulled all apaches

January 17

  • 23:50 logmsgbot: awjrichards Started syncing Wikimedia installation... : Updating MobileFrontend per https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2013-01-17
  • 23:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 23:33 mutante: rolled wikibugs back to last working version from SVN ..Revision 93776
  • 23:21 RobH: authdns-update
  • 23:12 Tim: on srv191: wiped /tmp and started apache
  • 23:08 Tim: fixing broken /tmp partition on srv191. Stopping apache
  • 23:04 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/EventLogging
  • 22:58 logmsgbot: olivneh synchronized php-1.21wmf7/extensions/EventLogging
  • 22:53 logmsgbot: aaron synchronized php-1.21wmf8/includes/Block.php 'deployed d69578ee88a2542cbaec3bd7904026a4edba86b1'
  • 22:18 logmsgbot: reedy synchronized php-1.21wmf8/cache/interwiki.cdb 'Updating 1.21wmf8 interwiki cache'
  • 22:18 logmsgbot: reedy synchronized php-1.21wmf7/cache/interwiki.cdb 'Updating 1.21wmf7 interwiki cache'
  • 21:59 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
  • 21:41 logmsgbot: dzahn gracefulled all apaches
  • 21:41 mutante: dropping www from wikidata in mw and apache configs as requested
  • 21:39 logmsgbot: dzahn synchronized ./wmf-config/CommonSettings.php
  • 21:39 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
  • 21:10 mutante: manually deployed changes to wikibugs bot, which is in git meanwhile
  • 20:03 logmsgbot: bsitu synchronized php-1.21wmf8/extensions/Echo 'Update Echo to master'
  • 19:47 logmsgbot: py synchronized wmf-config/lucene.php 'moving all search traffic to eqiad'
  • 19:46 binasher: stopped all replication on es1 (es1-4,1001-1004) and ensured all nodes are read-only. replication was enabled there briefly for mha testing.
  • 19:42 logmsgbot: py synchronized wmf-config/lucene.php 'moving enwiki search traffic to eqiad'
  • 19:38 binasher: fixed eqiad root + wikiuser grants on es shards
  • 19:16 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Disabled multiwrite to nas-1'
  • 18:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 17:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 17:41 binasher: moved dewiki db master to eqiad with mha and back to pmtpa in < 6 seconds, only one http request wanting to write hit the read-only window
  • 17:38 Reedy: Created moodbar tables for itwikivoyage
  • 17:32 logmsgbot: asher synchronized wmf-config/db.php 'setting s5/dewiki to writeable'
  • 17:31 logmsgbot: asher synchronized wmf-config/db.php 'setting s5/dewiki to read-only for several minutes'
  • 17:06 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'micro design for itwikivoyage'
  • 16:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 15:55 logmsgbot: reedy synchronized wmf-config/
  • 15:51 Reedy: Created wikilove tables on iswiki
  • 15:17 logmsgbot: reedy synchronized extract2.php
  • 15:16 logmsgbot: reedy synchronized live-1.5/
  • 15:13 logmsgbot: reedy synchronized docroot
  • 15:03 logmsgbot: demon synchronized php-1.21wmf8/extensions/Wikibase/repo/resources/wikibase.ui.entityViewInit.js 'Deploying Ie22a7213'
  • 15:03 logmsgbot: demon synchronized php-1.21wmf8/extensions/Wikibase/repo/resources/Resources.php 'Deploying Ie22a7213'
  • 12:56 hashar: Gerrit: created repository operations/debs/python-voluptuous to package http://pypi.python.org/pypi/voluptuous/
  • 11:43 paravoid: "mwscriptwikiset cleanupUploadStash.php all.dblist" on hume (kill it if swift overloads)
  • 11:37 Tim: installed rsyncd on 6 apache servers for network-aware scap testing
  • 03:56 binasher: resumed replication on db67
  • 03:49 binasher: stopped replication on db67.pmtpa (enwiki research slave), re-replicating off the pmtpa master instead of cross datacenter. original config isn't well supported by mha.
  • 02:45 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Thu Jan 17 02:45:26 UTC 2013
  • 02:27 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Thu Jan 17 02:26:58 UTC 2013
  • 01:10 mutante: DNS update to remove some useless stuff from esams zone (RT-4331)
  • 00:44 mutante: installing package upgrades on ms1002
  • 00:38 mutante: powercycling ms1002
  • 00:37 LeslieCarr: reloading unresponsive ms1002
  • 00:00 logmsgbot: reedy synchronized php-1.21wmf7/maintenance/mcc.php

January 16

  • 23:59 logmsgbot: reedy synchronized php-1.21wmf7/maintenance/mctest.php
  • 23:59 logmsgbot: reedy synchronized php-1.21wmf8/maintenance/mctest.php
  • 23:49 logmsgbot: laner synchronized wmf-config/lucene.php 'Switching search prefix pool to pmtpa'
  • 22:25 LeslieCarr: removing ganglios requirements to try to make it happily run in precise with icinga
  • 22:22 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 22:21 logmsgbot: reedy synchronized php-1.21wmf8/resources/startup.js
  • 21:58 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikidatawiki to 1.21wmf8
  • 21:58 logmsgbot: reedy synchronized php-1.21wmf8/extensions/
  • 21:52 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki to 1.21wmf8
  • 21:48 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki to wmf8
  • 21:46 logmsgbot: reedy synchronized php-1.21wmf8/extensions/Wikibase
  • 21:43 LeslieCarr: restarted nginx on ssl3001 (was depooled)
  • 21:42 logmsgbot: reedy synchronized php-1.21wmf8/cache/trusted-xff.cdb
  • 21:41 logmsgbot: reedy synchronized php-1.21wmf8/cache/interwiki.cdb
  • 21:39 logmsgbot: reedy synchronized php-1.21wmf8/cache/l10n/
  • 21:17 binasher: https://twitter.com/ashbash/status/291655249075240960
  • 21:16 RobH: authdns-update
  • 21:13 logmsgbot: reedy synchronized php-1.21wmf7/extensions/Wikibase
  • 21:13 binasher: set global read_only=0; on enwiki master
  • 21:13 logmsgbot: asher synchronized wmf-config/db.php 'taking s1 out of read-only'
  • 21:10 binasher: set global read_only=1; on enwiki master for "we have no unit tests, so lets test read-only in production"� testing
  • 21:09 logmsgbot: asher synchronized wmf-config/db.php 'setting s1 to read-only for several minutes'
  • 21:01 cmjohnson1: reinstalling sq48 w/lucid
  • 20:53 mutante: "bak" directory from unclean SVN copied to /root on sockpuppet as dns_bak.tar.gz just in case
  • 20:52 mutante: removing uncommitted stuff from DNS SVN, creating RT for wikimedia.jp.net and softwarewikipedia.com
  • 20:42 logmsgbot: reedy synchronized wmf-config/
  • 20:30 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild message cache for 1.21wmf8
  • 20:17 logmsgbot: reedy synchronized live-1.5/
  • 20:16 logmsgbot: reedy synchronized docroot
  • 20:15 logmsgbot: reedy synchronized php-1.21wmf8
  • 20:15 logmsgbot: reedy synchronized php-1.21wmf8
  • 20:13 logmsgbot: reedy synchronized php-1.21wmf8
  • 18:45 LeslieCarr: turning up xe-1/1/2 on asw-c-eqiad (was disabled due ot faulty equipment)
  • 18:44 LeslieCarr: turning up xe-1/1/0 on asw-c-eqiad (was disabled due to faulty equipment)
  • 18:41 mutante: putting logging-only iptables rule on sanger to check out what rules we actually need
  • 18:37 mark: restarted knsq19 earlier
  • 18:24 mark: Rerouting 43821->14907 traffic away from AS1257
  • 18:24 mutante: updating pt.planet
  • 18:21 RobH: praseodymium being odd, its coming down
  • 18:08 mark: Rerouting 14907->43821 traffic away from AS1257
  • 17:40 Reedy: Swapped slot1 from 1.21wmf6 to 1.21wmf8
  • 17:34 cmjohnson1: powering down titanium to verify console redirection status
  • 15:37 ottomata: deployed new webstatscollector filter to collect stats on wikivoyage domains and restarted udp2log on locke
  • 15:30 cmjohnson1: ms-be1005 & ms-be1006 going down
  • 15:16 hashar: restarted Jenkins again, somehow got stuck  :/
  • 14:58 hashar: restarted Jenkins on gallium to apply plugin updates.
  • 13:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 13:23 hashar: gallium: reapplied Tim's patch to PHPUnit install in /usr/share/php/PHPUnit/Util/Test.php to prevent PHP segfaulting while running tests. bug 43972
  • 11:34 mark: Added eqiad apaches and image scalers to the LVS pools
  • 11:28 mark: Added eqiad API apaches to the LVS pool
  • 11:12 mark: Restarted PyBal on lvs1003 and lvs1006
  • 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Wed Jan 16 02:28:11 UTC 2013
  • 00:46 Ryan_Lane: initializing all deployment destinations in pmtpa
  • 00:16 Ryan_Lane: testing deployment destination initialization on hume

January 15

  • 23:59 LeslieCarr: copied ganglios from lucid-wikimedia to precise-wikimedia
  • 23:59 LeslieCarr: restarted morebots
  • 23:35 Ryan_Lane: testing deployment destination initialization on fenari
  • 23:28 RobH: ignore msfe1001/msfe1002 errors (the names without the dash) as they are turned off and renamed
  • 23:21 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 44015 - Add en.wikivoyage autopatrolled group'
  • 22:20 mutante: powercycling msfe1002
  • 22:19 RobH: authdns-update
  • 22:14 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wgUseRCPatrol on for wikivoyages'
  • 21:32 mutante: setting knsq17 to enabled: False in pybal esams/uploads-squids as it is broken per RT-4321
  • 21:17 RobH: cerium & titanium being installed as parsoid caching servers, ignore any alerts for now.
  • 20:20 paravoid: stopping squid3 on sq48
  • 20:00 Nemo_bis: nikerabbit rebuilt message index on all Translate-enabled wikis (half a hour ago)
  • 19:54 RobH: authdns-update
  • 19:15 RobH: authdns-update
  • 18:36 mutante: stopping Apache on sanger
  • 16:01 hashar: Made a mistake in Zuul configuration. A slight outage while correcting it :/
  • 12:55 logmsgbot: nikerabbit synchronized php-1.21wmf7/extensions/Translate/ 'Translate to master again'
  • 12:09 logmsgbot: nikerabbit synchronized wmf-config/CommonSettings.php 'Translate config fix'
  • 10:26 logmsgbot: nikerabbit synchronized php-1.21wmf7/extensions/Translate 'Translate to master'
  • 10:18 logmsgbot: nikerabbit synchronized php-1.21wmf7/includes/HTMLForm.php
  • 10:18 logmsgbot: nikerabbit synchronized php-1.21wmf7/includes/Preferences.php
  • 10:17 logmsgbot: nikerabbit synchronized php-1.21wmf7/includes/AutoLoader.php
  • 10:04 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Enable ULS and disable WebFonts/Narayam on Translate wikis'
  • 10:03 logmsgbot: nikerabbit synchronized wmf-config/CommonSettings.php 'Updated ULS configuration'
  • 09:42 logmsgbot: nikerabbit synchronized php-1.21wmf7/extensions/UniversalLanguageSelector/ 'ULS to master'
  • 08:33 hashar: gallium : installing liblua5.1-0-dev package.
  • 04:50 binasher: set pc[1-3] to replicate from pc100[1-3], full master/master all the way across the sky
  • 02:26 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Tue Jan 15 02:26:37 UTC 2013
  • 02:16 LeslieCarr: rebooting neon
  • 02:02 LeslieCarr: created a new lvm for /var/log on neon, copied files over, and restarted processes. some file permission issues and incorrect file owners may remain
  • 01:39 LeslieCarr: restarting neon
  • 01:23 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'Set sqlbagostuff log'
  • 01:11 mutante: neon - changing logrotate config for atop to rotate 7 instead of 14
  • 01:09 mutante: deleting atop.log.1* in /var/log on neon to free disk space
  • 00:47 binasher: the production persistent parsercache is now on pc[1-3], replicated to pc100[1-3]
  • 00:43 logmsgbot: asher synchronized wmf-config/CommonSettings.php 'deploying db parsercache, sharded and replicated'

January 14

  • 23:18 Tim: changed CNAME for search.wikimedia.org to point to wikimedia-lb instead of ekrem
  • 22:46 mutante: installing apache,mysql-common,perl,sudo package updates on sanger
  • 22:45 mutante: killed java process on sanger, restarted opendj
  • 22:10 hashar: pullen 43849 on fenari (simply changes some comments)
  • 22:04 Reedy: Modified rc_params on huwiki to make it a blob
  • 21:30 Jeff_Green: tearing down community-analytics vhost and cname
  • 21:26 binasher: rebooting pc1
  • 20:36 logmsgbot: reedy cleared profiling data
  • 20:33 RobH: xenon & caesium added to parsoid pybal config
  • 20:27 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable WikibaseClient on huwiki'
  • 20:19 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 20:07 logmsgbot: reedy synchronized wmf-config/
  • 19:24 logmsgbot: reedy synchronized php-1.21wmf7/extensions/Wikibase
  • 19:09 cmjohnson1: provision sq48
  • 19:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Rest of the 'pedias to 1.21wmf7
  • 18:29 Ryan_Lane: adding glusterfs-client glusterfs-common glusterfs-server and glusterfs-dbg for lucid and precise to our repo
  • 17:20 logmsgbot: reedy synchronized php-1.21wmf7/includes/ChangesList.php
  • 16:55 ottomata: packet loss on oxygen after adding new wikipedia zero filters. Stopping puppet to temporarily comment them out and make sure they are the problem.
  • 16:53 cmjohnson1: powercycling db1037 to verify raid cfg
  • 16:48 Reedy: Populated huwiki sites and site_identifier tables from testwiki tables
  • 13:34 hashar: upgrade PEAR packages on gallium bug 43957
  • 13:33 hashar: upgrade PEAR packages on gallium
  • 13:20 hashar: jenkins: gating jobs now always report a message, even if tests complete after the change has been merged in 43407
  • 08:29 hashar: gallium: removed /mnt/jenkins-tmp tmpfs, replaced by /var/lib/jenkins/tmpfs bug 43825
  • 04:05 logmsgbot: tstarling synchronized docroot
  • 03:53 logmsgbot: tstarling synchronized docroot/search.wikimedia.org/index.php
  • 02:50 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Mon Jan 14 02:50:08 UTC 2013
  • 02:25 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Mon Jan 14 02:25:38 UTC 2013
  • 01:25 logmsgbot: tstarling synchronized php-1.21wmf7/includes/GlobalFunctions.php
  • 01:24 logmsgbot: tstarling synchronized php-1.21wmf7/includes/DefaultSettings.php
  • 01:24 logmsgbot: tstarling synchronized php-1.21wmf7/bin/ulimit5.sh
  • 01:22 logmsgbot: tstarling synchronized php-1.21wmf6/includes/GlobalFunctions.php
  • 01:21 logmsgbot: tstarling synchronized php-1.21wmf6/includes/DefaultSettings.php
  • 01:21 logmsgbot: tstarling synchronized php-1.21wmf6/bin/ulimit5.sh
  • 00:43 Tim: patched @requires handling out of phpunit on gallium so that it stops segfaulting and accepts Id9f2fea7

January 13

  • 22:45 Tim: killed stuck convert processes on the image scalers
  • 22:35 mark: restarted apache on srv222
  • 22:35 LeslieCarr: restarted ircecho on spence
  • 20:30 cmjohnson1: provisioning sq48
  • 17:11 apergos: restarted morebots
  • 16:43 paravoid: stopping squid on knsq17 (disk broken RT #4321)
  • 16:44 paravoid: restarting backend varnish: cp1021, cp1023, cp1025, cp1026, cp1028, cp1030, cp1031, cp1033, cp1034, cp1036 (varnish bug)
  • 08:28 Ori: synchronized wmf-config/throttle.php to extend throttle exemption for Pune summit per bug 43856

January 12

  • 09:29 logmsgbot: asher synchronized wmf-config/CommonSettings.php 'pulling parsercache'
  • 02:53 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Sat Jan 12 02:53:39 UTC 2013
  • 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Sat Jan 12 02:28:21 UTC 2013
  • 01:20 binasher: added mysql grants for eqiad to all shards

January 11

  • 20:08 logmsgbot: asher synchronized wmf-config/db.php 'repooling db59'
  • 19:48 logmsgbot: anomie synchronized php-1.21wmf7/maintenance/rebuildLocalisationCache.php 'Backported for git-deploy l10n work'
  • 19:47 logmsgbot: anomie synchronized php-1.21wmf6/maintenance/rebuildLocalisationCache.php 'Backported for git-deploy l10n work'
  • 18:55 RobH: xenon & caesium installing as parsoid
  • 17:01 hashar: creating Jenkins jobs for Semantic* extensions
  • 16:35 hashar: creating Jenkins jobs for GettingStarted and PHPExcel extensions.
  • 15:41 logmsgbot: maxsem synchronized wmf-config/throttle.php 'https://gerrit.wikimedia.org/r/#/c/43432/'
  • 14:08 hashar: updating Zuul 23ec1ba..ff79197 . Bring up better support for LOST builds.
  • 13:40 logmsgbot: reedy synchronized php-1.21wmf7/extensions/FlaggedRevs
  • 06:09 logmsgbot: tstarling synchronized wmf-config/db.php 'depool db59'
  • 05:44 logmsgbot: aaron synchronized php-1.21wmf7/includes/job/jobs/HTMLCacheUpdateJob.php 'deployed f0f528e254346f04d4afdb85b439cd20c0df93fe'
  • 05:38 logmsgbot: aaron synchronized php-1.21wmf7/includes/cache/HTMLCacheUpdate.php 'deployed 9709a7d7e423c21e38a0436faa233ac0dcf04226 '
  • 05:38 logmsgbot: aaron synchronized php-1.21wmf7/includes/DefaultSettings.php 'deployed 9709a7d7e423c21e38a0436faa233ac0dcf04226 '
  • 04:38 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Set $wgMaxBacklinksInvalidate.'
  • 04:16 logmsgbot: spage synchronized php-1.21wmf6/extensions/EventLogging 'EventLogging meant to be on 1.21wmf6 wikis as well'
  • 02:52 Tim: killed convert processes on all image scalers
  • 02:49 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Fri Jan 11 02:49:45 UTC 2013
  • 02:33 Tim: killed convert processes on srv222
  • 02:26 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Fri Jan 11 02:26:00 UTC 2013
  • 01:35 Reedy: Ran namespaceDupes.php on cswikisource
  • 01:34 logmsgbot: reedy synchronized php-1.21wmf7/extensions/ProofreadPage
  • 01:33 logmsgbot: spage synchronized php-1.21wmf6/extensions/EventLogging/EventLogging.hooks.php 'Log field fix for mobile'
  • 01:21 logmsgbot: spage synchronized php-1.21wmf7/extensions/EventLogging/EventLogging.hooks.php 'Log field fix for mobile'
  • 00:39 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Force https for mobile login and account creation everywhere'
  • 00:38 logmsgbot: asher synchronized wmf-config/db.php 'setting db59 weight to zero'
  • 00:34 logmsgbot: awjrichards synchronized php-1.21wmf6/extensions/MobileFrontend/
  • 00:33 logmsgbot: awjrichards synchronized php-1.21wmf7/extensions/MobileFrontend/

January 10

  • 23:44 logmsgbot: awjrichards synchronized php-1.21wmf6/extensions/ZeroRatedMobileAccess/ 'Updating ZeroRatedMobileAccess to master on php-1.21wmf6'
  • 23:35 logmsgbot: awjrichards synchronized php-1.21wmf7/extensions/ZeroRatedMobileAccess/ 'Updating ZeroRatedMobileAccess to head of master'
  • 23:12 logmsgbot: anomie synchronized php-1.21wmf7/includes/api/ApiQueryRevisions.php
  • 23:10 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'enable forced https for mobile login/account creation on testwiki'
  • 22:57 logmsgbot: spage Finished syncing Wikimedia installation... : E3 deployment Extension:GettingStarted and its i18n msgs
  • 22:11 logmsgbot: spage Started syncing Wikimedia installation... : E3 deployment Extension:GettingStarted and its i18n msgs
  • 21:41 logmsgbot: olivneh synchronized php-1.21wmf7/extensions/E3Experiments
  • 21:34 logmsgbot: olivneh synchronized php-1.21wmf7/extensions/EventLogging
  • 21:06 logmsgbot: olivneh synchronized php-1.21wmf7/extensions/EventLogging
  • 20:58 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 20:39 binasher: nas1-a thumbs vol ran out of inodes, increased total by 10x
  • 19:34 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 18:29 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 18:19 logmsgbot: reedy synchronized wmf-config/
  • 18:00 logmsgbot: demon synchronized php-1.21wmf7/extensions/ExtensionDistributor 'New version of ExtensionDistributor'
  • 14:38 MaxSem: Updating Solr index on all wikis
  • 14:15 hashar: cleaned up all content in /mnt/jenkins-tmp been replaced by /var/lib/jenkins/tmpfs
  • 13:57 hashar: gallium: manually reinstalled Zuul and restarted it.
  • 13:25 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'Rm GeoData exceptions'
  • 11:48 hashar: gallium: manually installed luajit, libluajit-5.1-dev and g++ packages . Pending gerrit change is 43182
  • 10:11 logmsgbot: maxsem synchronized wmf-config/mobile.php 'https://gerrit.wikimedia.org/r/#/c/43163/ - Try disabling noindex on some enwiki pages per discussion with Google'
  • 02:51 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Thu Jan 10 02:51:34 UTC 2013
  • 02:27 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Thu Jan 10 02:27:16 UTC 2013

January 9

  • 22:41 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 22:16 logmsgbot: aaron synchronized php-1.21wmf7/extensions/UploadWizard/resources/mw.UploadWizardDetails.js 'deployed 7482243e3249ae38d6aedbaf06db7107dc3516f5'
  • 21:00 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 19:53 logmsgbot: reedy synchronized wmf-config/
  • 19:47 cmjohnson1: authdns update adding mc1016/mc1017 to zone file
  • 19:13 logmsgbot: aaron Finished syncing Wikimedia installation... :
  • 19:09 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf7
  • 19:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Fix sewiki typo'
  • 18:24 logmsgbot: aaron Started syncing Wikimedia installation... :
  • 18:08 logmsgbot: aaron synchronized php-1.21wmf7/extensions/UploadWizard 'deployed bd33047d1cb938f3b4923a51862d887e6c831b65'
  • 17:43 logmsgbot: aaron synchronized php-1.21wmf7/includes/ 'deployed 5185259101b27e4780618b3cd7718b9a0c51e1c4'
  • 17:33 logmsgbot: aaron synchronized php-1.21wmf7/includes/upload/AssembleUploadChunks.php 'deployed 4bb28e000bb6609e23046b611be335382aa74618'
  • 16:37 cmjohnson1: authdns update adding osm-cp1003/4 production to zone files
  • 15:35 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Deploying Id2755235'
  • 15:04 mark: Started memcached on virt0
  • 14:53 logmsgbot: nikerabbit synchronized wmf-config/CommonSettings.php 'Translation memory tweak'
  • 11:16 logmsgbot: reedy synchronized wmf-config/
  • 02:52 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Wed Jan 9 02:52:56 UTC 2013
  • 02:26 mutante: wiping archive of 'ee' mailing list (RT-4294)
  • 02:26 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Wed Jan 9 02:26:08 UTC 2013
  • 01:19 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php
  • 01:10 logmsgbot: preilly synchronized php-1.21wmf6/extensions/MobileFrontend
  • 01:08 logmsgbot: preilly synchronized php-1.21wmf7/extensions/MobileFrontend
  • 00:36 logmsgbot: maxsem synchronized php-1.21wmf6/resources/Resources.php 'https://gerrit.wikimedia.org/r/#/c/42894/'
  • 00:34 logmsgbot: maxsem synchronized php-1.21wmf6/includes/resourceloader 'https://gerrit.wikimedia.org/r/#/c/42894/'
  • 00:30 mutante: powercycling frozen magnesium
  • 00:03 logmsgbot: maxsem synchronized php-1.21wmf6/extensions/MobileFrontend/

January 8

  • 23:56 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
  • 23:06 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
  • 22:49 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
  • 22:17 logmsgbot: tstarling synchronized php-1.21wmf7/includes/objectcache/SqlBagOStuff.php
  • 22:17 logmsgbot: tstarling synchronized php-1.21wmf7/includes/objectcache/RedisBagOStuff.php
  • 22:16 logmsgbot: tstarling synchronized php-1.21wmf7/includes/ArrayUtils.php
  • 22:16 logmsgbot: tstarling synchronized php-1.21wmf7/includes/AutoLoader.php
  • 21:53 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
  • 21:32 logmsgbot: reedy synchronized php-1.21wmf7/extensions/TimedMediaHandler
  • 19:22 logmsgbot: reedy synchronized wmf-config/
  • 18:51 logmsgbot: reedy synchronized php-1.21wmf7/extensions/TimedMediaHandler
  • 17:38 logmsgbot: olivneh synchronized docroot/wikivoyage.org/favicon.ico 'New favicon.ico for wikivoyage'
  • 17:37 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'New logo for wikivoyage & eswikivoyage'
  • 15:59 hashar: upgraded Zuul
  • 15:12 hashar: updating Zuul code base : 6c4d51a...23ec1ba (forced update)
  • 14:53 hashar: gallium: restarted Zuul which could not recover its Gerrit connection :/
  • 14:37 cmjohnson1: rebooting solr2 to fix HT in bios
  • 14:26 hashar: gallium: running git fsck in /var/lib/git/mediawiki/extensions/Wikibase.git . That repository might have ended up being corrupted somehow :(
  • 14:22 cmjohnson1: rebooting solr1 to fix HT in bios
  • 14:17 ^demon: manganese is back up, gerrit's running fine.
  • 14:13 ^demon: rebooting manganese
  • 13:38 apergos: and restarted them
  • 13:38 apergos: stopped and/or shot all swift processes on ms-be2, some were in an odd state, had load spike
  • 12:52 apergos: cleared out /var/lg/messages.[1-x] on older ms-be hosts, several had full root partitions; disabled logging to that facility in rsyslogd conf
  • 10:37 hashar: stopping Zuul
  • 10:37 hashar: shutdowning jenkins for an unscheduled maintenance and upgrading the installation.
  • 10:30 paravoid: reprepro update for jenkins (fetched 1.480.2)
  • 10:29 paravoid: installing new backported reprepro version to {lucid,precise}-wikimedia and upgrading it in brewster
  • 09:58 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '42232 set autoconfirm count to 10 @ fawiki'
  • 09:54 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php 'Increase AbuseFilter disable threshold for AFT entries to 20%'
  • 09:29 logmsgbot: nikerabbit synchronized php-1.21wmf7/extensions/Translate/resources/js/ext.translate.special.translate.js
  • 02:51 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Tue Jan 8 02:51:08 UTC 2013
  • 02:26 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Tue Jan 8 02:26:01 UTC 2013
  • 01:16 Tim: on deployment-squid.pmtpa.wmflabs: started squid
  • 01:05 notpeter: temp stopping puppet on brewster
  • 00:16 notpeter: starting innobackupex on db1004

January 7

  • 23:54 paravoid: starting nginx on ssl3002
  • 23:26 logmsgbot: aaron synchronized php-1.21wmf7/includes/api/ApiUpload.php 'deployed e2df801500f9472f812388006206e3e3261d58d3'
  • 23:22 RobH: solr1003 coming offline for comparison work per rt 4282
  • 22:57 paravoid: powercycling cp1034, dead for 18h, evidence of sudden memory exhaustion
  • 22:38 logmsgbot: aaron synchronized php-1.21wmf7/includes/api/ApiUpload.php 'deployed 542bfaef8f4ca61db04bb60cbd1e617c731db038'
  • 22:37 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'set wfWikiShellCmd hook.'
  • 22:35 cmjohnson1: shutting down solr3 to check bios setup
  • 22:32 Ryan_Lane: repooling ssl1001 with proxy_http_version 1.1 enabled
  • 21:51 binasher: restarted backend varnish on cp102[27]. cp103[0-6]
  • 21:43 binasher: restarting backend varnish on cp1030 (previously restarted cp102[346])
  • 21:42 logmsgbot: maxsem synchronized wmf-config/CommonSettings.php
  • 21:34 logmsgbot: maxsem synchronized wmf-config/CommonSettings.php 'Reduce the load on Solr servers in pmtpa'
  • 21:18 hashar: jenkins: splitting whitespace check out of mediawiki linting job. Will make whitespace a non voting change to avoid rejecting legitimate changes.
  • 21:08 logmsgbot: maxsem synchronized wmf-config/CommonSettings.php 'Multi-server solr'
  • 20:47 sbernardin: replacing ms-be5 with R720xd
  • 20:17 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'Disable GeoData jobs'
  • 20:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary to 1.21wmf7
  • 20:03 Jeff_Green: authdns-update to add tellurium.wikimedia.org
  • 20:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource, wikiversity and remaining wikimedia to 1.21wmf7
  • 19:52 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks and wikquote to 1.21wmf7
  • 19:36 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special, wikinews and wikivoyages to 1.21wmf7
  • 19:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: fishbowl, private and closed to wmf7
  • 18:40 logmsgbot: reedy synchronized php-1.21wmf7/extensions/EducationProgram
  • 18:39 logmsgbot: reedy synchronized php-1.21wmf6/extensions/EducationProgram
  • 14:10 logmsgbot: reedy synchronized wmf-config/
  • 11:07 hashar: pulled on fenari 42526 'Revert "Kill static-master"' thought NOT synced. That simply bring back some symbolic links for 'beta' so that is harmful.
  • 09:45 hashar: Jenkins: EventLogging now receives pep8 linting, 42517 & 42518
  • 09:10 hashar: Jenkins: whitelisted Bryan Tong Minh in Zuul 39850
  • 08:51 hashar: Jenkins: update all jobs. That bring whitespace checking for MediaWiki extensions. See 37803
  • 04:10 Tim: on hume: running purgeList.php to purge all pages on uzwiki from squid
  • 04:08 logmsgbot: tstarling synchronized php-1.21wmf6/maintenance/purgeList.php
  • 02:51 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Mon Jan 7 02:51:10 UTC 2013
  • 02:31 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
  • 02:29 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Mon Jan 7 02:28:58 UTC 2013
  • 02:02 logmsgbot: tstarling synchronized php-1.21wmf7/includes/ImagePage.php
  • 02:01 logmsgbot: tstarling synchronized php-1.21wmf7/includes/Article.php
  • 02:01 logmsgbot: tstarling synchronized php-1.21wmf7/includes/OutputPage.php
  • 02:01 logmsgbot: tstarling synchronized php-1.21wmf7/includes/DefaultSettings.php
  • 02:00 logmsgbot: tstarling synchronized php-1.21wmf6/includes/ImagePage.php
  • 02:00 logmsgbot: tstarling synchronized php-1.21wmf6/includes/OutputPage.php
  • 01:59 logmsgbot: tstarling synchronized php-1.21wmf6/includes/Article.php
  • 01:59 logmsgbot: tstarling synchronized php-1.21wmf6/includes/DefaultSettings.php

January 6

  • 22:46 logmsgbot: reedy synchronized wmf-config/throttle.php
  • 17:46 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Sun Jan 6 17:46:42 UTC 2013
  • 17:23 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Sun Jan 6 17:23:41 UTC 2013
  • 02:48 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Sun Jan 6 02:48:45 UTC 2013
  • 02:26 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Sun Jan 6 02:26:24 UTC 2013

January 5

  • 19:41 Nemo_bis: en.wiki job queue again way over a million and counting
  • 17:08 paravoid: restarting memcached on virt0
  • 02:53 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Sat Jan 5 02:53:30 UTC 2013
  • 02:29 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Sat Jan 5 02:29:41 UTC 2013

January 4

  • 23:29 RobH: authdns-update for xenon and caesium dns
  • 22:46 binasher: temporarily setting up mysql replication on shard es1 again (es([1-4]|100[1-4]) for mha testing
  • 22:35 notpeter: es1001 back in prodcution. all external store are innodb!
  • 22:17 RobH: updated asw-a-eqiad vlan for 4/0/44
  • 21:25 mutante: mauntius/torrus: http://wikitech.wikimedia.org/view/Torrus#Deadlock_problem
  • 21:23 RobH: authdns-update for some mgmt name changes
  • 21:23 mutante: restarting torrus-common on manutius
  • 21:01 cmjohnson1: db1042 removing/replacing bad hdd from slot 9
  • 20:52 cmjohnson1: db1038 removing bad hdd from slot6 for replacement
  • 20:02 Ryan_Lane: depooling ssl1001
  • 18:29 logmsgbot: robh synchronized wmf-config/PoolCounterSettings.php
  • 15:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikidatawiki to 1.21wmf7
  • 15:08 logmsgbot: demon Finished syncing Wikimedia installation... :
  • 14:25 logmsgbot: demon Started syncing Wikimedia installation... :
  • 02:49 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Fri Jan 4 02:49:14 UTC 2013
  • 02:47 logmsgbot: asher synchronized wmf-config/db.php 'returning db64 to s3'
  • 02:27 binasher: restarted replication on db64 from db34
  • 02:26 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Fri Jan 4 02:26:53 UTC 2013
  • 02:20 binasher: truncated ( de | it )wikivoyage.searchindex tables
  • 00:37 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo to Master
  • 00:19 notpeter: starting innobackupex from es1002 to es1001
  • 00:15 RobH: modifying kuo bios power profile from DAPC to OS performance per watt setting in attempt to resolve runaway process issue
  • 00:12 notpeter: converting es1001 to innodb
  • 00:11 binasher: rebuilding db64 via hotbackup of db1010
  • 00:02 logmsgbot: asher synchronized wmf-config/db.php 'pulling db64 from s3'
  • 00:02 RobH: taking down kuo, its power settings are freaking it out, will fix and bring back into servie

January 3

  • 23:10 logmsgbot: olivneh synchronized php-1.21wmf6/extensions/EventLogging 'Updating display of rev IDs in schema subtitles'
  • 23:02 RobH: authdns-updateexit
  • 22:59 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo to Master
  • 22:34 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'switched captchas back to swift'
  • 22:18 binasher: reprepro includedeb (lucid|precise)-wikimedia mha4mysql-manager_0.55-0_all.deb mha4mysql-node_0.54-0_all.deb
  • 21:30 logmsgbot: reedy synchronized php-1.21wmf7/cache/interwiki.cdb 'Updating 1.21wmf7 interwiki cache'
  • 21:30 logmsgbot: reedy synchronized php-1.21wmf6/cache/interwiki.cdb 'Updating 1.21wmf6 interwiki cache'
  • 21:30 RobH: reinstalling tarin to internal ip for secondary poolcounter server in tampa
  • 21:13 paravoid: powercycling cp1026, load spike, unresponsive
  • 20:38 logmsgbot: reedy synchronized php-1.21wmf7/cache/interwiki.cdb 'Updating interwiki cache'
  • 20:38 logmsgbot: reedy synchronized php-1.21wmf6/cache/interwiki.cdb 'Updating interwiki cache'
  • 20:02 RobH: authdns-update for iodine
  • 20:01 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable email batch for Echo'
  • 19:49 notpeter: temp stopping puppet on brewster
  • 19:19 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: and aswikisource back again
  • 19:18 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: remove aswikisource temporarily
  • 19:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 19:13 logmsgbot: reedy synchronized wmf-config/
  • 19:12 LeslieCarr: restarting puppetmasterd on stafford
  • 19:04 logmsgbot: reedy synchronized php-1.21wmf7/extensions/WikimediaMaintenance/
  • 18:59 logmsgbot: reedy synchronized wmf-config/db.php
  • 18:54 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 18:53 logmsgbot: reedy synchronized wmf-config/
  • 18:51 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 18:49 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 18:47 logmsgbot: reedy synchronized wmf-config/
  • 18:35 mutante: restarting nagios on spence
  • 18:34 logmsgbot: reedy synchronized php-1.21wmf6/extensions/WikimediaMaintenance/
  • 18:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 02:46 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Thu Jan 3 02:46:37 UTC 2013
  • 02:26 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Thu Jan 3 02:26:45 UTC 2013
  • 01:02 binasher: restarting all bits varnish instances with a 180s stagger between each (applying shm_reclen change)
  • 01:01 RobH: dns update to move tarin to internal ip
  • 00:54 RobH: poolcounter test shows numbers rising on ersch
  • 00:54 logmsgbot: robh synchronized wmf-config/PoolCounterSettings.php
  • 00:32 logmsgbot: olivneh synchronized php-1.21wmf6/extensions/EventLogging/includes/JsonSchemaHooks.php

January 2

  • 23:20 logmsgbot: maxsem synchronized php-1.21wmf7/extensions/MobileFrontend 'Revert MobileFrontend to the same revision as 1.21wmf6'
  • 22:20 paravoid: starting nginx on ssl3003
  • 22:13 paravoid: deploying squid config for Accept-Language/X-Carrier, RT 2745 & 3158
  • 22:03 paravoid: downgrading squid from 3.1 to 2.7 on fenari
  • 22:03 paravoid: reprepro copysrc precise-wikimedia lucid-wikimedia squid
  • 21:39 Ryan_Lane: powercycled ssl3001
  • 21:34 paravoid: starting nginx on ssl3002
  • 20:33 Reedy: Modified user_groups.ug_group to varbinary(255)
  • 20:29 Reedy: Modified user_former_groups.ufg_group to varbinary(255)
  • 20:24 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikidatawiki back to 1.21wmf6
  • 20:03 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki, wikidatawiki and mediawikiwiki to 1.21wmf7
  • 19:53 logmsgbot: reedy synchronized php-1.21wmf7/extensions/
  • 19:48 logmsgbot: aaron synchronized php-1.21wmf7/extensions/ConfirmEdit 'deployed 8fddb4a805ee3e79ef07ef3ca78a7ae7df5f4bba'
  • 19:26 Ryan_Lane: make that labs.
  • 19:26 Ryan_Lane: changing novaadmin password in labvs
  • 19:21 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild message cache for 1.21wmf7
  • 18:26 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild message cache for 1.21wmf7
  • 18:23 logmsgbot: aaron synchronized php-1.21wmf6/extensions/ConfirmEdit 'deployed 74c5543fead84a8460be51ffa0b16104cfc3abd1'
  • 18:17 Reedy: Running sync-common on srv191
  • 18:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki to 1.21wmf7 before rebuilding message cache
  • 18:14 logmsgbot: reedy synchronized live-1.5/
  • 18:14 cmjohnson1: powercycling srv191
  • 18:13 logmsgbot: reedy synchronized wmf-config/
  • 18:12 logmsgbot: reedy synchronized php-1.21wmf7/ 'Initial sync'
  • 18:09 logmsgbot: aaron synchronized wmf-config/CommonSettings.php
  • 17:35 cmjohnson1: db1014 removing bad hdd from slot11 to replace w/new rt 4039
  • 16:54 LeslieCarr: pulling sfp uplink module on asw-c-eqiad
  • 15:15 cmjohnson1: swapping ethernet cable for ms-be1003
  • 14:34 Nemo_bis: MaxSem> can someone restart labsconsole's memcached which appears to be down again? 14.14 paravoid> done
  • 09:36 logmsgbot: krinkle synchronized php-1.21wmf6/resources/mediawiki.page/mediawiki.page.watch.ajax.js 'I2b1b34c9'
  • 02:27 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Wed Jan 2 02:27:26 UTC 2013

January 1

  • 19:51 apergos: shot hung converts on all image scalers
  • 02:32 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Tue Jan 1 02:32:04 UTC 2013
  • 01:20 logmsgbot: aaron synchronized wmf-config/CommonSettings.php
  • 00:42 logmsgbot: aaron synchronized php-1.21wmf6/extensions/ConfirmEdit
  • 00:16 logmsgbot: aaron synchronized wmf-config/CommonSettings.php

December 31

  • 16:27 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Deploying I9be1e7ac, testwiki permission changes'
  • 02:26 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Mon Dec 31 02:26:11 UTC 2012

December 30

  • 22:56 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
  • 20:35 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 15:40 logmsgbot: reedy synchronized php-1.21wmf6/extensions/ParserFunctions
  • 09:05 Nemo_bis: Search reported broken with no results at all returned on en.wikt, (en|ru).source etc. "Lucene on search14 is CRITICAL" since 3h ago.
  • 02:26 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Sun Dec 30 02:26:16 UTC 2012

December 29

  • 02:27 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Sat Dec 29 02:27:24 UTC 2012

December 28

  • 18:07 Jeff_Green: restart udp2log on emery to clean up 838 defunct processes
  • 16:47 logmsgbot: maxsem synchronized php-1.21wmf6/extensions/PageImages/PageImages.body.php 'https://gerrit.wikimedia.org/r/#/c/40812/'
  • 16:26 cmjohnson1: authdns-update adding mc17/18 to pmtpa mgmt zonefile
  • 05:54 apergos: restarted varnish front and back ends on cp1023, it was swapping
  • 02:57 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
  • 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Fri Dec 28 02:28:21 UTC 2012

December 27

  • 22:57 logmsgbot: tstarling Finished syncing Wikimedia installation... :
  • 22:53 logmsgbot: tstarling Started syncing Wikimedia installation... :
  • 22:51 logmsgbot: tstarling Finished syncing Wikimedia installation... :
  • 22:47 logmsgbot: tstarling Started syncing Wikimedia installation... :
  • 22:42 logmsgbot: tstarling Finished syncing Wikimedia installation... :
  • 22:39 logmsgbot: tstarling Started syncing Wikimedia installation... :
  • 22:28 logmsgbot: tstarling Started syncing Wikimedia installation... :
  • 22:27 logmsgbot: tstarling Finished syncing Wikimedia installation... :
  • 22:13 logmsgbot: tstarling Started syncing Wikimedia installation... :
  • 22:02 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php
  • 15:33 cmjohnson1: shutting down mw57 to troubleshoot DIMM/Mem issue
  • 06:47 logmsgbot: tstarling synchronized live-1.5/mobileRedirect.php
  • 02:21 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Thu Dec 27 02:21:50 UTC 2012
  • 02:09 Tim: on db64: killed FlaggedRevsStats::getEditReviewTimes queries, were running for 40 days. Client host was hume but no process was found with the relevant ephemeral port.
  • 00:21 binasher: powercycled cp1044, couldn't ssh in
  • 00:14 logmsgbot: reedy synchronized php-1.21wmf6/cache/l10n

December 26

  • 23:27 binasher: restarted varnishhtcpd on all eqiad upload varnish hosts; looking back fondly on vacation
  • 23:22 jeremyb: 2012-12-26 22:51:58 <+logmsgbot> !log maxsem synchronized wmf-config/InitialiseSettings.php 'Enable PageImages everywhere'
  • 23:21 jeremyb: 2012-12-26 22:31:37 <+logmsgbot> !log maxsem synchronized wmf-config/InitialiseSettings.php 'Enable PageImages on test2'
  • 23:21 jeremyb: 2012-12-26 22:25:53 <+logmsgbot> !log awjrichards synchronized php-1.21wmf6/extensions/MobileFrontend/ 'touch files'
  • 23:21 jeremyb: 2012-12-26 22:23:19 <+logmsgbot> !log awjrichards Finished syncing Wikimedia installation... : MobileFrontend updates per https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-12-26
  • 23:21 jeremyb: 2012-12-26 21:59:47 <+logmsgbot> !log awjrichards Started syncing Wikimedia installation... : MobileFrontend updates per https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-12-26
  • 23:21 jeremyb: 2012-12-26 21:23:38 <+logmsgbot> !log maxsem synchronized wmf-config/CommonSettings.php 'Fix PageImages inclusion'
  • 23:21 jeremyb: 2012-12-26 21:11:03 <+logmsgbot> !log awjrichards synchronized wmf-config/InitialiseSettings.php 'Enabling PageImages on testwiki'
  • 23:21 jeremyb: 2012-12-26 21:10:34 <+logmsgbot> !log awjrichards synchronized wmf-config/CommonSettings.php 'Enabling PageImages on testwiki'
  • 23:20 jeremyb: 2012-12-26 02:05:45 <+logmsgbot> !log LocalisationUpdate failed: git pull of extensions failed
  • 23:20 jeremyb: 2012-12-25 11:03:46 <+logmsgbot> !log maxsem synchronized wmf-config/CommonSettings.php 'Disable broken Swift captchas on mw.o too'
  • 23:20 jeremyb: 2012-12-25 10:58:38 <+logmsgbot> !log maxsem synchronized wmf-config/CommonSettings.php 'Captchas broken on every wiki'
  • 23:20 jeremyb: 2012-12-25 05:45:08 < Ryan_Lane> !log restarting gerrit. also blocked a bad bot
  • 23:20 jeremyb: 2012-12-25 02:23:37 <+logmsgbot> !log LocalisationUpdate completed (1.21wmf6) at Tue Dec 25 02:23:37 UTC 2012
  • 23:19 jeremyb: 2012-12-24 14:49:25 < apergos> !log rebooting cp1043 which had fallen over
  • 23:19 jeremyb: 2012-12-24 13:28:34 <+logmsgbot> !log reedy synchronized live-1.5/
  • 23:19 jeremyb: !logging what morebots missed
  • 23:08 LeslieCarr: restarted morebots

December 24

  • 02:29 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Mon Dec 24 02:29:30 UTC 2012
  • 00:44 logmsgbot: tstarling synchronized php-1.21wmf6/extensions/ProofreadPage/ProofreadPage.body.php

December 23

  • 23:24 logmsgbot: reedy synchronized php-1.21wmf6/includes/filebackend/FSFileBackend.php
  • 22:31 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 22:04 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 02:29 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Sun Dec 23 02:29:31 UTC 2012

December 22

  • 17:02 logmsgbot: reedy synchronized wmf-config/
  • 16:50 Reedy: Created WikiLove tables on itwikivoyage
  • 06:39 apergos: shot a bunch of hung converts again on the imagescalers
  • 02:26 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Sat Dec 22 02:26:37 UTC 2012

December 21

  • 21:33 Ryan_Lane: also upgrading: libzmq3, python-zmq on all minions
  • 21:33 Ryan_Lane: upgrading salt on all minions
  • 18:45 mutante: re-enabled password protection on ganglia
  • 18:39 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Switched all wikis to swift for captchas'
  • 14:55 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'Enable GeoData jobs everywhere'
  • 14:43 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'Try GeoData jobs on enwiki'
  • 14:23 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'Enable GeoData on wikiedias and explicitly on test2'
  • 14:18 hashar: Enabling whitespace checking on mediawiki-core-lint
  • 14:16 MaxSem: Load test passed successfully
  • 13:58 MaxSem: Load-testing geosearch
  • 13:46 MaxSem: Created GeoData tables for wikis in special.dblist
  • 13:42 hashar: Zuul pipeline gate-and-submit will now add a message in Gerrit when it start. Would let people know the jobs have started after they voted CR+2 39584
  • 13:36 logmsgbot: hashar synchronized wmf-config/CommonSettings-labs.php 'beta: 39742'
  • 10:32 logmsgbot: hashar synchronized all-labs.dblist
  • 10:32 logmsgbot: hashar synchronized wmf-config
  • 10:23 hashar: Deploying MediaWiki config changes for labs, basically renaming files from -wmflabs to -labs. 39059 and 39060
  • 09:35 logmsgbot: maxsem synchronized wmf-config/CommonSettings.php 'Switch GeoData to solr1001'
  • 09:10 MaxSem: Populating solr1001 with enwiki coordinates
  • 08:59 logmsgbot: maxsem synchronized php-1.21wmf5/extensions/GeoData/ 'Update GeoData to master to fix job problems'
  • 08:58 logmsgbot: maxsem synchronized php-1.21wmf6/extensions/GeoData/ 'Update GeoData to master to fix job problems'
  • 02:27 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Fri Dec 21 02:27:16 UTC 2012
  • 00:11 hashar: Doc to apply a change to Zuul / reload it is at https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Changing_the_configuration (logging there so people find it)

December 20

  • 23:44 logmsgbot: olivneh synchronized php-1.21wmf6/extensions/GettingStarted/resources/ext.gettingstarted.accountcreation.js 'Fix-up: reference correct ID in selector'
  • 23:22 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Enabling PostEdit on test2wiki'
  • 22:34 hashar: updating Zuul code base and running puppet manually on gallium
  • 22:23 mutante: dumping pilot_1_16wmf4 database on kaulen, putting dump on tridge and removing mysqld from kaulen
  • 22:13 logmsgbot: olivneh synchronized php-1.21wmf6/extensions/GettingStarted 'Interface update, pt. 2'
  • 22:00 logmsgbot: olivneh synchronized php-1.21wmf6/extensions/GettingStarted 'Interface update'
  • 21:54 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Enabling Extension:PostEdit for astwiki & trwiki'
  • 21:43 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Enabling Extension:PostEdit for itwikivoyage'
  • 21:35 logmsgbot: olivneh synchronized php-1.21wmf6/extensions/E3Experiments
  • 21:27 logmsgbot: olivneh synchronized php-1.21wmf6/extensions/EventLogging 'Roll out stricter schema validation'
  • 20:52 logmsgbot: mlitn synchronized php-1.21wmf6/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
  • 20:16 ottomata: copied js/ and css/ dirs from nickel:/srv/org/wikimedia/ganglia-web-3.5.1 to nickel:/srv/org/wikimedia/ganglia-web-3.5.4
  • 17:55 cmjohnson1: authdns update adding pappas mgmt to zone file
  • 17:15 MaxSem: Cleaned up job queue from stuck solrUpdate jobs
  • 16:58 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/39595'
  • 14:50 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/39576/'
  • 13:53 hashar: Jenkins: deleting the old MediaWiki-* jobs which used the Gerrit Trigger plugin.
  • 13:45 logmsgbot: hashar synchronized php-1.21wmf6/extensions/WikimediaMaintenance 'Bring in 32169 : per-realm and per-datacenter configuration'
  • 13:23 hashar: reconfiguring all Jenkins jobs
  • 09:49 logmsgbot: hashar synchronized wmf-config/ 'ext file now uses realm for consistency 39058. That makes CheckUser to load only in production. Checked on test.wp.o'
  • 09:38 hashar: ran sync-common on hume
  • 09:38 logmsgbot: hashar synchronized wmf-config 'use $wmfRealm in switches instead of legacy $cluster 39057'
  • 09:35 logmsgbot: hashar synchronized live-1.5/robots.php 'use $wmfRealm in switches instead of legacy $cluster 39057'
  • 09:21 logmsgbot: hashar synchronized multiversion/MWMultiVersion.php 'now requires multiversion/MWRealm.php 39543'
  • 09:20 logmsgbot: hashar synchronized multiversion/activeMWVersions 'now requires multiversion/MWRealm.php 39543'
  • 09:20 logmsgbot: hashar synchronized multiversion/backupWikiversions 'now requires multiversion/MWRealm.php 39543'
  • 09:19 logmsgbot: hashar synchronized refresh-dblist 'now requires multiversion/MWRealm.php 39543'
  • 08:31 logmsgbot: hashar synchronized tests/multiversion/MWRealmTest.php
  • 08:31 logmsgbot: hashar synchronized multiversion/MWRealm.sh
  • 08:31 logmsgbot: hashar synchronized multiversion/MWRealm.php
  • 08:29 hashar: going to deploy Mediawiki config changes related to the multiple datacenter support.
  • 02:27 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Thu Dec 20 02:27:33 UTC 2012
  • 01:48 logmsgbot: kaldari synchronized php-1.21wmf6/extensions/Echo 'Update Echo extension'
  • 01:20 logmsgbot: kaldari synchronized php-1.21wmf6/extensions/Echo 'Update Echo extension'
  • 00:37 logmsgbot: kaldari synchronized php-1.21wmf6/extensions/Echo 'Update Echo extension'
  • 00:20 andrewbogott: removed some mariadb packages from brewster: precise-wikimedia libmysqlclient18, mariadb-common, mysql-common, all in precise-wikimedia. They were causing dependency problems that broke the install of (for example) generic::mysql::server
  • 00:19 AaronSchulz: Running refreshFileHeaders.php on all wikis

December 19

  • 23:53 mutante: neon - install nagios-nrpe-plugin with --no-install-recommends
  • 23:07 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'use swift for captchas for testwiki/mediawikiwiki'
  • 23:07 logmsgbot: aaron About 200K "fancy" captchas added (there were ~10K before)
  • 20:44 cmjohnson1: authdns update solr1001/1002 correction
  • 20:28 logmsgbot: demon synchronized php-1.21wmf6/extensions/ParserFunctions/ 'Rolling back pfuncs to d96a17c'
  • 19:59 cmjohnson1: authdns update correcting solr1003 entry
  • 19:00 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: All 'pedias to 1.21wmf6
  • 16:57 logmsgbot: reedy synchronized php-1.21wmf6/extensions/Wikibase
  • 16:39 logmsgbot: reedy synchronized php-1.21wmf6/extensions/CategoryTree
  • 15:37 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php
  • 15:37 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php
  • 15:36 logmsgbot: hashar synchronized wmf-config/CommonSettings.php
  • 14:03 hashar: gallium: updated PHPUnit to 3.7.10 thus solving bug 42724
  • 13:48 hashar: gallium: apt-get install pep8 v1.3.3 (backported from raring)
  • 11:53 MaxSem: Updated solr, cleaning up killlist
  • 09:31 hashar: Jenkins: enabled unit test run on mw/core for some whitelisted people 39310
  • 07:06 apergos: shot a bunch of converts on the image scalers, looks like a couple started flapping about 7-8 hours ago
  • 04:03 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable Echo on test2 and mediawiki'
  • 03:59 logmsgbot: kaldari Finished syncing Wikimedia installation... :
  • 03:31 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Wed Dec 19 03:31:24 UTC 2012
  • 03:24 logmsgbot: kaldari Started syncing Wikimedia installation... :
  • 02:48 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Wed Dec 19 02:48:16 UTC 2012
  • 02:02 logmsgbot: awjrichards synchronized php-1.21wmf5/extensions/MobileFrontend/javascripts/modules/mf-toggle.js 'touch file
  • 02:01 logmsgbot: awjrichards synchronized php-1.21wmf5/extensions/MobileFrontend/javascripts/common/main.js
  • 01:44 logmsgbot: awjrichards synchronized php-1.21wmf6/extensions/MobileFrontend/stylesheets/modules/mf-cleanuptemplates.css 'touch file'
  • 01:44 Ryan_Lane: finished upgrading salt on all production minions
  • 01:41 logmsgbot: awjrichards synchronized php-1.21wmf6/extensions/MobileFrontend/javascripts/modules/mf-cleanuptemplates.js
  • 01:16 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable Echo on test server'
  • 01:14 logmsgbot: demon synchronized php-1.21wmf6/extensions/Wikibase/lib/resources/templates.js 'Deploying I641725a2'
  • 00:46 notpeter: taking down for another reinstall (this time with raid!)
  • 00:33 logmsgbot: awjrichards synchronized php-1.21wmf6/extensions/MobileFrontend/javascripts/common/main.js 'touch file'
  • 00:26 Ryan_Lane: make that modifying project groups in labs by running syncProjectGroups.php maintenance script in OpenStackManager
  • 00:25 Ryan_Lane: modifying project groups

December 18

  • 23:40 logmsgbot: maxsem Finished syncing Wikimedia installation... : https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-12-18
  • 22:59 notpeter: temp putting ganglia.w.o behind htaccess for sec reasons
  • 22:54 logmsgbot: maxsem Started syncing Wikimedia installation... : https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-12-18
  • 22:46 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'switched new captcha setting from testwiki -> test2wiki'
  • 22:43 logmsgbot: aaron synchronized wmf-config/filebackend.php 'set captcha directory.'
  • 22:42 Ryan_Lane: deploying Andrew Otto's group changes to OpenStackManager to labsconsole
  • 22:07 logmsgbot: aaron synchronized php-1.21wmf6/extensions/ConfirmEdit/captcha.py
  • 21:32 cmjohnson1: authdns-update new dns entries for frack bastion host (tellurium)
  • 21:08 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Turns off Echo temporarily on test and test2'
  • 20:48 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php 'Update Echo config file'
  • 20:47 logmsgbot: py gracefulled all apaches
  • 20:44 notpeter: gracefulling all apaches to pick up https://gerrit.wikimedia.org/r/#/c/38521/ (tested good on srv193)
  • 20:44 hashar: running puppet on gallium.
  • 20:44 hashar: Zuul: applying "filters events by user email" to our Zuul deployment https://review.openstack.org/#/c/17609/
  • 20:27 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php
  • 20:26 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php
  • 18:31 LeslieCarr: asw-c-eqiad ae bundles went down, working on fixing
  • 15:45 MaxSem: Testing done, 40 concurrent processes hitting around the worst-case point kept the load on yttrium at 20%. Average response time ~430ms
  • 15:19 MaxSem: Load-testing spatial search
  • 14:14 hashar: restarting Zuul with https://gerrit.wikimedia.org/r/39082 so it starts voting Verified+2
  • 14:13 ^demon: restarting gerrit on manganese to pick up VRIF+2
  • 13:50 hashar: restarted puppet on gallium (some apt-get process was a zombie)
  • 09:25 logmsgbot: nikerabbit synchronized wmf-config/CommonSettings.php 'Bug 43075'
  • 06:30 andrewbogott: switched all labs instances to mount /home via gluster on next reboot
  • 06:29 andrewbogott: rsynced all labs homedirs to gluster volumes
  • 02:46 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Tue Dec 18 02:45:58 UTC 2012
  • 02:25 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Tue Dec 18 02:25:22 UTC 2012
  • 01:28 mutante: fixing duplicate UID issue on stat1 for maryana
  • 00:57 LeslieCarr: asw-c-eqiad unreachable due to lacp issue
  • 00:32 mutante: fixing fenari permissions for gwicke.. (pre-puppet age UID)
  • 00:26 LeslieCarr: starting upgrade of asw-c-eqiad.mgmt - connectivity to row c machines may be affected
  • 00:11 notpeter: temp stopping lsave on es1009 and es1010 for upcoming networking downtime

December 17

  • 23:12 LeslieCarr: restarted pybal on lvs1001-1003 in order to restart their bgp peering
  • 22:44 notpeter: taking fenari down for upgarde to precise (not upgrading, not reimaging)
  • 22:37 LeslieCarr: cr1-eqiad being upgraded and rebooted
  • 22:30 Ryan_Lane: labstore1 is locked up, powercycling
  • 22:29 Nemo_bis: en.wiki job queue spiked from 1 to 3 millions in last 3 hours
  • 21:22 Ryan_Lane: rebooting labstore4
  • 21:15 logmsgbot: reedy synchronized php-1.21wmf6/includes/filebackend/FSFileBackend.php
  • 21:11 Ryan_Lane: rebooting labstore3
  • 21:03 Ryan_Lane: rebooting labstore2
  • 20:48 Ryan_Lane: restarting labstore1
  • 20:33 cmjohnson1: auth-dns update to add internal ip's for solr1-3
  • 19:35 hashar: regenerating Jenkins job mediawiki-core-install-sqlite
  • 19:12 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf6
  • 18:15 andrewbogott: beginning labs $HOME migration from NFS to gluster
  • 16:16 cmjohnson1: auth-dns update adding mgmt for solr1-3, solr1001-3 and internal ip's for solr1001-3
  • 13:13 hashar: set Jenkins to use /bin/bash as a default shell (intend of /bin/sh)
  • 04:02 paravoid: killall -9 convert on imagescalers; uploading 120px generated thumbnail directly to swift
  • 02:50 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Mon Dec 17 02:50:11 UTC 2012
  • 02:27 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Mon Dec 17 02:27:34 UTC 2012

December 16

  • 17:14 paravoid: "killall -9 convert" on all imagescalers
  • 15:55 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 15:54 logmsgbot: reedy synchronized images/sul/wikimania.png
  • 15:36 logmsgbot: reedy synchronized images/sul/
  • 02:47 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Sun Dec 16 02:47:42 UTC 2012
  • 02:45 LeslieCarr: mwscript.php appears to be broken
  • 02:40 LeslieCarr: checking memcache servers due to spate of mediawiki servers flapping
  • 02:26 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Sun Dec 16 02:26:23 UTC 2012

December 15

  • 14:37 logmsgbot: maxsem synchronized /php-1.21wmf5/includes/Message.php 'https://gerrit.wikimedia.org/r/#/c/38822/'
  • 14:30 paravoid: powercycling ms-be1
  • 12:18 apergos: ms-be1 can't be reached form mgmt console to check on it (Connection refused)
  • 02:47 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Sat Dec 15 02:47:17 UTC 2012
  • 02:45 LeslieCarr: drained all transit traffic from cr1-eqiad -- restore by load override /var/home/lcarr/cr1-eqiad.undrained if traffic levels seem too high
  • 02:25 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Sat Dec 15 02:25:26 UTC 2012
  • 00:36 LeslieCarr: appears that cr1-eqiad's fpc rebooted

December 14

  • 23:25 AaronSchulz: Compiled texvc into uncommon/php-1.21wmf6 dir for all apaches
  • 23:08 K4-713: synchronized payments to 4f5296e43b1c708
  • 22:40 hashar: I am not sure what is happening between Gerrit and Zuul. Seems Gerrit might not have sent all its event via stream-event. I have restarted Zuul and it seems to be happy (test: https://gerrit.wikimedia.org/r/#/c/38800/ ). Will monitor over the weekend.
  • 22:34 mutante: fixing symlink on kaulen to reflect new bugzilla version 4.2.4
  • 22:30 hashar: restarting Zuul on gallium
  • 20:54 mutante: deleting /var/lib/git/integration on gallium
  • 18:54 mutante: appyling mini patch on bugzilla per http://bzr.mozilla.org/bugzilla/4.2/revision/8176
  • 18:31 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable clicktracking on test2wiki'
  • 16:56 sbernardin: sq48 going down for HDD controller card replacement-rt4041
  • 15:40 logmsgbot: demon synchronized wmf-config/throttle.php 'Deploying I330230e6'
  • 09:30 logmsgbot: mlitn synchronized php-1.21wmf5/extensions/ArticleFeedbackv5 'Fix AFTv5 toolbox popups'
  • 02:47 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Fri Dec 14 02:46:58 UTC 2012
  • 02:26 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Fri Dec 14 02:25:58 UTC 2012
  • 01:50 mutante: restarting pdns on ns0
  • 01:35 mutante: creating new self-signed SSL cert for wikitech - valid for another year
  • 01:03 logmsgbot: csteipp synchronized php-1.21wmf6/extensions/CategoryTree/CategoryTreeFunctions.php
  • 00:15 logmsgbot: olivneh synchronized php-1.21wmf6/extensions/AbuseFilter
  • 00:15 mutante: prototype.wikimedia is also removed ... bye or please complain to us about it
  • 00:13 logmsgbot: olivneh synchronized php-1.21wmf5/extensions/AbuseFilter
  • 00:09 mutante: DNS update - killing a bunch of .tesla.

December 13

  • 23:55 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Enabling GettingStarted on enwiki'
  • 23:42 logmsgbot: olivneh synchronized wmf-config/abusefilter.php '(bug 42758) enable AbuseFilter events for enwiki irc feed'
  • 23:40 mutante: gallium mysql (jenkins/CI) back up and running after brief db server issue
  • 23:34 logmsgbot: olivneh synchronized php-1.21wmf6/extensions/AbuseFilter
  • 23:33 logmsgbot: olivneh synchronized php-1.21wmf5/extensions/AbuseFilter
  • 23:30 mutante: mysql stopped on gallium due to dpkg issue.. investigating
  • 23:28 mutante: installing package upgrades on gallium, incl. newer mysql packages
  • 23:20 logmsgbot: olivneh synchronized php-1.21wmf6/extensions/E3Experiments/experiments/acux/accountCreationUX.js
  • 23:17 logmsgbot: olivneh synchronized php-1.21wmf5/extensions/E3Experiments/experiments/acux/accountCreationUX.js
  • 23:14 logmsgbot: spage Finished syncing Wikimedia installation... : E3 deploy new Special:GettingStarted and E3 logging
  • 23:03 LeslieCarr: switched asw-c-eqiad to cr2-eqiad aggregated bundle from lacp to static
  • 22:34 logmsgbot: spage Started syncing Wikimedia installation... : E3 deploy new Special:GettingStarted and E3 logging
  • 20:10 logmsgbot: mlitn synchronized php-1.21wmf5/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
  • 19:39 notpeter: restarting all lucene instances in pmtpa
  • 19:27 notpeter: stopping puppet on brewster
  • 19:01 notpeter: upgrading lucene to 2.1.6wm1 on all search boxes in pmtpa
  • 18:55 notpeter: upgrading lucene to 2.1.6wm1 on all search boxes in eqiad
  • 18:51 notpeter: taking down hume for disk replacement/reimage/etc
  • 18:50 paravoid: nas1-a/nas1001-a: resizing originals volume to 37T and thumbs volume to 6T
  • 16:54 logmsgbot: nikerabbit synchronized php-1.21wmf6/extensions/Translate/ 'Translate bugfix'
  • 14:08 ^demon: live-hacked out manganese -> gallium gerrit replication. spamming logs with "reject HostKey: gallium.wikimedia.org". restarted gerrit on manganese.
  • 13:53 MaxSem: Updated enwiki Solr index
  • 10:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 02:47 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Thu Dec 13 02:47:24 UTC 2012
  • 02:26 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Thu Dec 13 02:26:13 UTC 2012
  • 00:59 logmsgbot: aaron synchronized php-1.21wmf5/includes/job/JobQueueDB.php 'deployed c3147eb053e0fb4c60f8feddc7de1bc363210d02'
  • 00:50 RoanKattouw: Force-running puppet on celsus
  • 00:08 logmsgbot: aaron synchronized php-1.21wmf6/includes/job/JobQueueDB.php 'deployed 2c177fa087bb09bc0915413ea2f628dae5c0e986'

December 12

  • 23:22 logmsgbot: catrope synchronized php-1.21wmf6/extensions/VisualEditor/ 'Updating VisualEditor to master'
  • 23:20 logmsgbot: catrope synchronized php-1.21wmf5/extensions/VisualEditor/ 'Updating VisualEditor to master'
  • 22:45 mutante: wikipedia.co.za redirects to af.wikipedia.org now
  • 22:43 logmsgbot: dzahn gracefulled all apaches
  • 22:42 mutante: restarting Apaches to enable wikipedia.co.za
  • 21:57 logmsgbot: reedy synchronized php-1.21wmf6/includes/api/
  • 21:20 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commons back to wmf6 for testding
  • 21:05 logmsgbot: nikerabbit synchronized php-1.21wmf6/extensions/Translate 'Translate bug fix'
  • 20:55 logmsgbot: reedy synchronized wmf-config/
  • 20:39 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote to wmf6
  • 20:37 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks, wikimedia and wikivoyage to 1.21wmf6
  • 20:26 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity and wikinews to wmf6
  • 20:12 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Reverted commonswiki to 1.21wmf5
  • 19:52 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 19:41 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource and wiktionary to 1.21wmf6
  • 19:37 logmsgbot: catrope synchronized php-1.21wmf6/cache/l10n/
  • 19:32 mutante: removing test-star cert and key from all ssl hosts
  • 19:22 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Closed, special and private to 1.21wmf6
  • 18:42 logmsgbot: reedy synchronized wmf-config/
  • 18:21 logmsgbot: catrope synchronized php-1.21wmf6/extensions/VisualEditor/ 'Looks like this did not get synced right earlier'
  • 18:16 RoanKattouw: Rebuilding l10n cache for wmf6 to fight broken messages for VisualEditor on wmf6 wikis
  • 18:13 logmsgbot: reedy synchronized wmf-config/
  • 18:03 logmsgbot: reedy synchronized wmf-config/
  • 11:53 MaxSem: done: real 8m1.864s user 1m32.940s sys 0m0.900s
  • 11:44 MaxSem: Populating Solr index for enwiki
  • 11:34 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
  • 11:29 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
  • 11:26 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
  • 11:22 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/38300'
  • 11:06 MaxSem: Populated Solr index on testwiki
  • 03:26 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable VisualEditor on enwiki'
  • 03:12 logmsgbot: catrope synchronized php-1.21wmf5/extensions/VisualEditor 'touch'
  • 03:06 logmsgbot: catrope synchronized php-1.21wmf5/resources/startup.js 'touch'
  • 03:04 logmsgbot: catrope synchronized php-1.21wmf5/extensions/VisualEditor/modules/ve/init/mw/targets/ve.init.mw.ViewPageTarget.js 'touch'
  • 02:48 logmsgbot: LocalisationUpdate completed (1.21wmf6) at Wed Dec 12 02:48:46 UTC 2012
  • 02:43 logmsgbot: catrope synchronized php-1.21wmf5/extensions/VisualEditor/modules/ve/init/mw/targets/ve.init.mw.ViewPageTarget.js 'touch'
  • 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Wed Dec 12 02:28:55 UTC 2012
  • 02:17 logmsgbot: catrope synchronized php-1.21wmf6/extensions/VisualEditor 'Updating VisualEditor'
  • 02:17 logmsgbot: catrope synchronized php-1.21wmf5/extensions/VisualEditor 'Updating VisualEditor'
  • 02:07 logmsgbot: catrope synchronized php-1.21wmf6/extensions/VisualEditor 'Updating VisualEditor'
  • 02:07 logmsgbot: catrope synchronized php-1.21wmf5/extensions/VisualEditor 'Updating VisualEditor'
  • 00:10 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Send parsoid requests to celsus (aa2903d81515d543f09dd41d1ad844d1b5e79cf3)'
  • 00:08 logmsgbot: catrope synchronized php-1.21wmf6/extensions/VisualEditor 'Updating VisualEditor'
  • 00:07 logmsgbot: catrope synchronized php-1.21wmf5/extensions/VisualEditor 'Updating VisualEditor'

December 11

  • 23:33 LeslieCarr: reverted the priority, messed with BGP_tweak_LP to re-acheive desired results
  • 23:28 logmsgbot: awjrichards synchronized php-1.21wmf5/extensions/MobileFrontend/javascripts/modules/mf-tables.js
  • 23:20 LeslieCarr: changed vrrp priority of row a to have cr1-eqiad for mastership in order to rebalance transit traffic
  • 22:53 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly MobileFrontend deployment + GeoData updates
  • 22:41 mutante: rebuilding archives of wikilovesmonuments list
  • 22:38 Ryan_Lane: rebooting virt0
  • 22:33 Ryan_Lane: apt upgrading virt0
  • 22:15 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly MobileFrontend deployment + GeoData updates
  • 21:38 MaxSem: fixing GeoData tables
  • 20:17 logmsgbot: asher synchronized wmf-config/db.php 'pooling db59 with full weight'
  • 20:15 logmsgbot: asher synchronized wmf-config/db.php 'pooling db59 with a low weight for warmup'
  • 20:14 logmsgbot: asher synchronized wmf-config/db.php 'returning db59 to s1'
  • 19:31 binasher: shutting down mysql on db59 for upgrade
  • 19:30 RoanKattouw: Force-running puppet on celsus and constable for like the millionth time
  • 19:29 logmsgbot: asher synchronized wmf-config/CommonSettings.php 're-enabling mysql pcache'
  • 19:20 logmsgbot: asher synchronized wmf-config/CommonSettings.php 'emergency disable mysql pcache, pushing again'
  • 19:20 RoanKattouw: Also disabled mwext-Translate-install
  • 19:17 RoanKattouw: Disabled mwext-VisualEditor-install in Jenkins, totally broken
  • 19:14 logmsgbot: asher synchronized wmf-config/CommonSettings.php 'emergency disable mysql pcache'
  • 19:03 logmsgbot: asher synchronized wmf-config/db.php 'pulling db59 from s1 for test'
  • 18:44 RoanKattouw: Force-running puppet on celsus and constable
  • 18:25 RoanKattouw: Force-running puppet on celsus and constable again
  • 18:24 mutante: giving all RT accounts a real name value to reduce confusion (nickname,unix login,username,language bla bla is optional)
  • 18:21 RoanKattouw: Edited celsus:/etc/varnish/default.vcl to point to 10.2.1.28:8000
  • 18:12 RoanKattouw: Installing Varnish in celsus
  • 17:43 RoanKattouw: Force-running puppet on constable
  • 16:50 cmjohnson1: correction...mc1010 and mc1002 replacing 10g NIC
  • 16:43 cmjohnson1: mc1009 and mc1002 shutting down to replace 10g NIC
  • 16:26 logmsgbot: reedy synchronized php-1.21wmf6/extensions/ParserFunctions/
  • 15:32 logmsgbot: demon synchronized php-1.21wmf6/extensions/Wikibase/client/includes/store/sql/WikiPageEntityLookup.php
  • 15:27 logmsgbot: demon synchronized php-1.21wmf6/includes/Revision.php 'Syncing I14a7ebb8'
  • 14:44 logmsgbot: maxsem synchronized wmf-config
  • 14:25 sbernardin: sr266 going down for main board replacement rt2896
  • 14:25 sbernardin: sr266 going down for main board replacement
  • 12:32 hashar: changing operations-puppet-validate Jenkins job.
  • 12:25 Nemo_bis: All !Wikipedia and !Wikimedia projects are now up and editable again, s3 was read-only for a few hours for slow DB http://ur1.ca/bz8v0 [1]
  • 12:07 logmsgbot: reedy synchronized php-1.21wmf6/extensions/LabeledSectionTransclusion
  • 09:35 mark: Power cycled nescio
  • 08:43 hashar: restarting Jenkins, it still knows about jobs that no more exist :/
  • 08:40 apergos: stopped job runners temporarily while we look at still-increasing replag on s3
  • 08:03 binasher: restarted all job runners
  • 07:56 logmsgbot: aaron synchronized php-1.21wmf5/includes/job/JobQueueDB.php 'deployed 6f03ada325f985fc0b14c0d0144a42c94268941f'
  • 07:55 logmsgbot: aaron synchronized php-1.21wmf6/includes/job/JobQueueDB.php 'deployed eabed08c91538ffbb651ed7d8867f9966afc2ae5 '
  • 07:42 logmsgbot: aaron synchronized php-1.21wmf6/includes/job/JobQueueDB.php 'deployed 26b9a472df430dc3abaead0e6f5a9f640b075b99'
  • 07:34 binasher: restarting all job runners
  • 07:32 Aaron|home: Also deployed 919b9d8ba3ee7d6a16214b0853e82fa733a0605a and dcd314e837ccf3d2f7c2ca7dc94b7d1e6accfde4
  • 07:31 logmsgbot: aaron synchronized php-1.21wmf5/includes/job/JobQueueDB.php 'reverted live hacks to use slave DB'
  • 07:02 logmsgbot: catrope synchronized php-1.21wmf6/extensions/VisualEditor/modules/ve/init/mw/targets/ve.init.mw.ViewPageTarget.js 'touch'
  • 07:02 logmsgbot: catrope synchronized php-1.21wmf5/extensions/VisualEditor/modules/ve/init/mw/targets/ve.init.mw.ViewPageTarget.js 'touch'
  • 07:00 logmsgbot: olivneh synchronized php-1.21wmf5/extensions/E3Experiments/Experiments.php
  • 06:56 logmsgbot: olivneh synchronized php-1.21wmf5/extensions/EventLogging 'Syncing to avoid obsolete CACHE_MEMCACHE use in prod.'
  • 06:49 logmsgbot: asher synchronized wmf-config/db.php 'pulling db39 from s3, crazy behind on purging'
  • 05:52 logmsgbot: aaron synchronized php-1.21wmf5/includes/objectcache/MemcachedPhpBagOStuff.php
  • 05:51 logmsgbot: aaron synchronized php-1.21wmf5/includes/objectcache/MemcachedPhpBagOStuff.php
  • 05:22 binasher: killed all wikiadmin threads on db39
  • 05:11 logmsgbot: catrope synchronized php-1.21wmf6/extensions/VisualEditor 'Updating VisualEditor to master'
  • 05:11 logmsgbot: catrope synchronized php-1.21wmf5/extensions/VisualEditor 'Updating VisualEditor to master'
  • 04:41 binasher: same with db66
  • 04:38 binasher: killed 22 long running FlaggedRevsStats::getEditReviewTimes on db39, run from hume against enwikinews
  • 04:04 RoanKattouw: Updating Parsoid to master
  • 03:53 mutante: powercycling mexia
  • 03:29 logmsgbot: aaron synchronized php-1.21wmf5/includes/job/JobQueueDB.php
  • 03:29 logmsgbot: aaron synchronized php-1.21wmf5/includes/job/jobs/RefreshLinksJob.php
  • 03:29 RoanKattouw: mexia still hasn't come back up. That combined with the weird behavior I saw before rebooting it makes me suspect a hardware issue
  • 03:12 logmsgbot: aaron synchronized php-1.21wmf6/includes/job/JobQueueDB.php
  • 03:08 logmsgbot: aaron synchronized php-1.21wmf6/includes/job/jobs/RefreshLinksJob.php
  • 03:02 RoanKattouw: Rebooting mexia, seems to have been hit by some kernel bug
  • 02:00 logmsgbot: LocalisationUpdate failed: git pull of core failed
  • 01:11 logmsgbot: pgehres synchronized wmf-config/CommonSettings.php 'Putting donation pipeline back to normal'
  • 01:02 cmjohnson1: replacing disk on slot 8 db1008
  • 00:48 logmsgbot: pgehres synchronized wmf-config/CommonSettings.php 'Putting donation pipeline in maintenance mode, part 2'
  • 00:44 logmsgbot: pgehres synchronized wmf-config/CommonSettings.php 'Putting donation pipeline in maintenance mode'
  • 00:32 K4-713: synchronized payments cluster to 325a919967709359c8
  • 00:05 logmsgbot: pgehres synchronized php-1.21wmf6/extensions/FundraiserLandingPage/ 'Updating FundraiserLandingPage to master'
  • 00:04 logmsgbot: pgehres synchronized php-1.21wmf6/extensions/ContributionTracking/ 'Updating ContributionTracking to master'

December 10

  • 23:17 logmsgbot: pgehres synchronized php-1.21wmf5/extensions/ContributionTracking/ 'Updating ContributionTracking to master'
  • 23:10 K4-713: synchronized payments to 72c1985a2b9b1db
  • 22:45 LeslieCarr: removed /home/wikipedia/common/php-1.21wmf1/ from fenari
  • 22:08 mutante: mobile and non-mobile Wikipedia now use just ONE certificate.. a new "star.wp" with an ".m." in it, valid until 2016 by DigiCert
  • 21:57 mutante: one more nginx restart on ssl boxes for mobile cert
  • 21:18 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: wikidatawiki & mediawikiwiki to 1.21wmf6
  • 21:03 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki also on 1.21wmf6
  • 21:03 logmsgbot: demon Finished syncing Wikimedia installation... :
  • 20:27 logmsgbot: demon Started syncing Wikimedia installation... :
  • 20:19 logmsgbot: asher synchronized wmf-config/CommonSettings.php 're-enabling pc1 for sql: the bag o stuffening'
  • 20:11 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 20:07 logmsgbot: demon Finished syncing Wikimedia installation... :
  • 20:03 logmsgbot: demon Started syncing Wikimedia installation... :
  • 20:01 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.21wmf5
  • 19:59 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki to 1.21wmf6
  • 19:58 logmsgbot: demon ran sync-common-all
  • 19:50 mutante: replaced wikipedia SSL cert with new DigiCert (rather than RapidSSL) (RT-3639)
  • 19:40 mutante: restarting nginx on all ssl hosts
  • 19:38 mutante: repooling ssl4
  • 19:11 mutante: fixing test-star vs. old-star and new-star WP certificate/key confusion on ssl hosts.. arrg
  • 18:51 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'New link for Special:Version'
  • 18:25 mutante: temp. depooling ssl4
  • 17:58 logmsgbot: demon synchronized wmf-config/extdist/svn-invoker.conf 'More futile efforts to fix extdist'
  • 12:13 logmsgbot: maxsem synchronized wmf-config/CommonSettings.php 'https://gerrit.wikimedia.org/r/#/c/37823/'
  • 02:24 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Mon Dec 10 02:24:44 UTC 2012

December 9

  • 12:05 Nemo_bis: job queue snapshot for bug 42614: en.wiki 900k still raising steadily, Commons 70k, fr.wiki 4.5M, Meta 2k, other tops 1k at most
  • 05:22 Ryan_Lane: rebooted wikitech and cleared some logs to free up /var (which was full)

December 8

  • 14:32 logmsgbot: reedy synchronized live-1.5/
  • 14:31 logmsgbot: reedy synchronized wmf-config/
  • 14:04 logmsgbot: reedy synchronized php-1.21wmf5/cache/l10n
  • 13:50 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Sat Dec 8 13:50:26 UTC 2012
  • 02:26 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Sat Dec 8 02:26:46 UTC 2012

December 7

  • 23:46 logmsgbot: olivneh synchronized php-1.21wmf5/extensions/EventLogging 'Fix-up of mtime issue'
  • 22:07 mutante: DNS update - add icinga.wm as cname for neon
  • 21:37 logmsgbot: lcarr synchronized wmf-config/throttle.php
  • 21:22 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'EventLogging log group -> '
  • 21:02 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Updating wfDebugLogGroups dest for EventLogging'
  • 20:18 logmsgbot: olivneh synchronized php-1.21wmf5/extensions/E3Experiments
  • 20:00 K4-713: synchronized payments cluster to 351b8cfa58dc99
  • 20:00 binasher: rebooting pc1
  • 19:29 binasher: upgrading pc1 to precise and newer mysql-facebook build
  • 18:52 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'disabled use of pc1'
  • 18:05 RoanKattouw: Ran killall -u extdist on fenari per Chad's request
  • 17:33 logmsgbot: reedy synchronized php-1.21wmf5/extensions/CentralAuth
  • 16:36 logmsgbot: maxsem synchronized php-1.21wmf5/extensions/GeoData 'https://gerrit.wikimedia.org/r/#/c/37446/'
  • 16:22 RobH: authdns-update for durant mgmt
  • 15:57 hashar: Jenkins now auto submit changes in mediawiki/core.git whenever tests are successful 37420
  • 15:39 logmsgbot: reedy synchronized php-1.21wmf5/extensions/WikimediaMaintenance
  • 15:22 logmsgbot: reedy synchronized php-1.21wmf5/extensions/WikimediaMaintenance/
  • 14:14 logmsgbot: demon synchronized php-1.21wmf5/extensions/Wikibase/client/includes/store/sql/WikiPageEntityLookup.php 'More debugging, Iaf3cfff9'
  • 14:11 logmsgbot: demon synchronized php-1.21wmf5/extensions/Wikibase/client/includes/store/sql/WikiPageEntityLookup.php 'More debugging'
  • 14:04 logmsgbot: demon synchronized php-1.21wmf5/extensions/Wikibase/client/includes/store/sql/WikiPageEntityLookup.php
  • 13:28 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Deploying wikibase debug log groups'
  • 13:14 logmsgbot: demon synchronized php-1.21wmf5/extensions/Wikibase 'Syncing wikibase to 24d8471'
  • 09:54 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c/37399/'
  • 09:48 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/37398'
  • 05:25 logmsgbot: aaron synchronized php-1.21wmf5/includes/job/JobQueueDB.php 'deployed 78c63d4cafb4937e289856f66ac0e524fda79acb'
  • 04:59 logmsgbot: kaldari synchronized php-1.21wmf5/extensions/UploadWizard/resources/mw.UploadWizardDetails.js 'fixing live geo bug in UploadWizard'
  • 02:24 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Fri Dec 7 02:24:07 UTC 2012
  • 01:03 logmsgbot: reedy synchronized php-1.21wmf5/extensions/ParserFunctions/Expr.php
  • 00:56 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'turning on experimental Flickr uploading for admins on Commons'
  • 00:54 binasher: reprepro changes: add precise-wikimedia deb universe amd64 mariadb-server-5.5 5.5.28-mariadb-wmf201212041~precise <woot>
  • 00:32 mutante: DNS update - adding wikivoyage.net (link to .org)
  • 00:27 logmsgbot: dzahn gracefulled all apaches

December 6

  • 23:53 logmsgbot: reedy synchronized php-1.21wmf5/extensions/ParserFunctions/Expr.php
  • 23:41 logmsgbot: awjrichards synchronized php-1.21wmf5/extensions/MobileFrontend/javascripts/common/mf-application.js 'Touch file'
  • 23:21 logmsgbot: awjrichards synchronized php-1.21wmf5/extensions/MobileFrontend/ 'fix for MobileFrontend bug 42749'
  • 22:53 logmsgbot: aaron synchronized php-1.21wmf5/includes/job/JobQueueDB.php 'deployed 471f6840613ae5137464b6449db88bc30608e85f'
  • 22:41 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Deploying Ia1208778, Ib80a42fa'
  • 22:34 logmsgbot: olivneh synchronized php-1.21wmf5/extensions/EventLogging
  • 22:25 logmsgbot: catrope synchronized php-1.21wmf5/resources/mediawiki/mediawiki.Uri.js '67de9f2cd4e37b7b7081d47a2652aa0158680c5c'
  • 22:25 logmsgbot: catrope synchronized php-1.21wmf5/includes/EditPage.php '67de9f2cd4e37b7b7081d47a2652aa0158680c5c'
  • 22:24 logmsgbot: catrope synchronized php-1.21wmf5/includes/Title.php '67de9f2cd4e37b7b7081d47a2652aa0158680c5c'
  • 21:57 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Updating Extension:EventLogging config'
  • 21:50 logmsgbot: reedy synchronized php-1.21wmf5/includes/media/DjVuImage.php
  • 20:42 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'Turning off wgResourceLoaderExperimentalAsyncLoading on test2'
  • 20:31 logmsgbot: kaldari Finished syncing Wikimedia installation... :
  • 20:24 Reedy: Running sync-common on hume
  • 19:42 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Deploying Idf3b7f0c'
  • 19:41 logmsgbot: aaron synchronized wmf-config 'deployed 7c21802fc48ef4c5ee0edab3118d03cbbdc1866d'
  • 19:12 logmsgbot: kaldari Started syncing Wikimedia installation... :
  • 19:06 RobH: authdns-update for celsus and constable servers
  • 18:34 ^demon: running `mwscript extensions/Wikibase/repo/maintenance/rebuildEntityPerPage.php --wiki=wikidatawiki` in screen on hume. If it causes problems, kill it and ping me
  • 18:25 RoanKattouw: Reweighted pmtpa Parsoid pool according to # of cores
  • 10:59 Nemo_bis: job queue snapshot for bug 42614: en.wiki 380k still raising steadily, Commons 85k down from 120+, fr.wiki 4M+, pl.wiki 18k, nl.wiki 8k, other tops 2k at most
  • 10:08 hashar: applying new CI workflow on mediawiki/core.git . Changes now needs CR+2 to trigger unit tests.
  • 03:27 logmsgbot: asher synchronized wmf-config/db.php 'setting db59 to full weight'
  • 03:26 logmsgbot: asher synchronized wmf-config/db.php 'lowering db59 weight for warmup'
  • 03:25 logmsgbot: asher synchronized wmf-config/db.php 'returning db59 to s1'
  • 02:24 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Thu Dec 6 02:24:10 UTC 2012
  • 01:19 binasher: rebooting db59
  • 00:56 binasher: shutting down mysql on db59, upgrading to precise
  • 00:55 logmsgbot: asher synchronized wmf-config/db.php 'pulling db59 from s1 for system upgrade'
  • 00:43 RoanKattouw: Added mexia, tola, kuo, lardner to /h/w/common/docroot/noc/pybal/pmtpa/parsoid , they appear to be pooled just fine now
  • 00:38 binasher: testing mariadb-server-5.5 5.5.28-mariadb-wmf201212041~precise on db1043 (slaving enwiki)
  • 00:06 logmsgbot: aaron synchronized php-1.21wmf5/extensions/SwiftCloudFiles/php-cloudfiles-wmf/cloudfiles_http.php
  • 00:05 binasher: rebooting db1043
  • 00:00 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/37157/'
  • 00:00 RoanKattouw: *build-essential

December 5

  • 23:59 RoanKattouw: and build-essentials too
  • 23:59 RoanKattouw: Temporarily installing git-core and npm on bast1001 by hand with Peter's blessing
  • 23:59 Ryan_Lane: make that upgrading labsconsole to wmf/1.21wmf5
  • 23:56 Ryan_Lane: upgrading labsconsole to wmf/1.21wmf6
  • 23:45 MaxSem: created GeoData tables in all wikivoyages
  • 23:42 MaxSem: creating GeoData tables in all wikipedias
  • 23:37 binasher: shutting down mysql on db1043 for upgrade and testing
  • 23:33 logmsgbot: maxsem synchronized wmf-config/CommonSettings.php 'GeoData tweaks'
  • 23:32 logmsgbot: maxsem synchronized php-1.21wmf5/extensions/GeoData/
  • 22:57 Ryan_Lane: upgraded labsconsole successfully
  • 22:47 MaxSem: upgraded GeoData tables on testwiki by recreation
  • 22:38 Ryan_Lane: running update.php on labsconsole
  • 22:37 Ryan_Lane: attempting upgrade of mediawiki on labsconsole to wmf/1.21wmf4
  • 22:28 logmsgbot: awjrichards synchronized php-1.21wmf5/extensions/MobileFrontend/ 'Bug fixes for MobileFrontend'
  • 22:04 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Enable mobile beta photo uploads to commons from testwiki'
  • 21:53 RobH: authdns-update for new misc servers
  • 21:51 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable wikibaseclient on test2wiki'
  • 20:47 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'wikidatawiki not readonly'
  • 20:37 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'wikidatawiki not readonly'
  • 20:30 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild localisationcache for wikidatawiki
  • 20:12 notpeter: temp stopping puppet on brewster
  • 19:50 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild localisationcache for wikidatawiki
  • 19:40 logmsgbot: reedy synchronized wmf-config/
  • 19:18 logmsgbot: reedy synchronized wmf-config/ 'Wikidata config updates'
  • 19:18 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: git pull is helpful. Actually put everything else to 1.21wmf5
  • 19:16 logmsgbot: reedy synchronized php-1.21wmf5/extensions/Wikibase
  • 19:15 logmsgbot: reedy synchronized php-1.21wmf5/extensions/Diff
  • 19:12 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'wikidatawiki readonly for maintenance'
  • 19:10 logmsgbot: reedy synchronized php-1.21wmf5/extensions/DataValues
  • 19:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.21wmf5
  • 16:05 Reedy: THIS DOESN'T WORK IN HERE
  • 15:34 apergos: powercycled sq48, hmm it was up for 214 days... fishy squid!
  • 15:17 apergos: pulled ms-be5:sde1 and ms-be12:sdi from rings, they were borking obj replication across the cluster
  • 14:53 hashar: Phasing out the Jenkins job linting operations/puppet.git , replacing it with https://integration.mediawiki.org/ci/job/operations-puppet-validate/
  • 14:03 logmsgbot: reedy synchronized php-1.21wmf5/includes/AutoLoader.php
  • 14:03 logmsgbot: reedy synchronized php-1.21wmf5/includes/db
  • 14:02 logmsgbot: reedy synchronized php-1.21wmf5/includes/dao
  • 08:40 apergos: powercycling snapshot1, was in swapdeath
  • 08:04 apergos: cleaning up stuff on ms-be5 /srv/swift-storage/sde1/objects/ again, data somehow wound up on the underlying root filesystem, root partition was full
  • 07:22 Nemo_bis: job queue snapshot for bug 42614: en.wiki 285k still raising steadily, Commons 111k, fr.wiki 4M, pt.wiki 40k, other tops 5k at most
  • 02:44 logmsgbot: LocalisationUpdate completed (1.21wmf4) at Wed Dec 5 02:44:51 UTC 2012
  • 02:24 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Wed Dec 5 02:24:05 UTC 2012
  • 01:45 mutante: gallium - puppet runs fine again, has our librsvg 2.36-1wm1 packages and Android SDK stuff should also work (@hashar)
  • 00:48 logmsgbot: mlitn synchronized php-1.21wmf4/extensions/ArticleFeedbackv5/modules 're-sync aftv5 js&css files with newer modification date'
  • 00:45 logmsgbot: awjrichards synchronized wmf-config/mobile.php 'Fix api endpoint definition for mobile beta'
  • 00:42 K4-713: synchronized payments cluster to f2c33676270c
  • 00:37 notpeter: rebooting es1004 for kernel upgrade before repooling
  • 00:13 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Disable use of commons api for mobile beta photo uploads on testwiki and test2wiki'
  • 00:06 logmsgbot: awjrichards synchronized wmf-config/mobile.php 'Enabling use of commons API for mobile beta photo upload'
  • 00:05 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Preparing to use commons API for mobile beta photo upload'

December 4

  • 23:50 logmsgbot: awjrichards synchronized php-1.21wmf5/extensions/MobileFrontend/ 'touch files'
  • 23:46 logmsgbot: anomie synchronized php-1.21wmf4/extensions/Vector/modules/ext.vector.collapsibleNav.css 'CSS fixes backported from 1.21wmf5'
  • 23:45 logmsgbot: anomie synchronized php-1.21wmf4/skins/vector/screen.css 'CSS fixes backported from 1.21wmf5'
  • 23:44 logmsgbot: anomie synchronized php-1.21wmf4/skins/modern/main.css 'CSS fixes backported from 1.21wmf5'
  • 23:41 logmsgbot: awjrichards Finished syncing Wikimedia installation... : Updating MobileFrontend per regular weekly deployment
  • 23:32 mutante: re-siging neon on sockpuppet, remove from ssh_known_hosts, run puppet ...
  • 23:27 logmsgbot: py synchronized wmf-config/db.php 's7: repooling db56'
  • 22:58 logmsgbot: mlitn synchronized php-1.21wmf5/extensions/ArticleFeedbackv5 'apply some patches to aft for dewiki'
  • 22:58 logmsgbot: mlitn synchronized php-1.21wmf4/extensions/ArticleFeedbackv5 'apply some patches to aft for dewiki'
  • 22:43 mutante: reinstalling neon
  • 22:39 logmsgbot: awjrichards Started syncing Wikimedia installation... : Updating MobileFrontend per regular weekly deployment
  • 22:21 notpeter: taking down hume for upgrade to precise
  • 22:03 logmsgbot: awjrichards synchronized php-1.21wmf5/extensions/MobileFrontend/stylesheets/specials/watchlist.css 'touch file'
  • 21:51 notpeter: starting innoback from db1041 to db56
  • 21:34 notpeter: bast1001 back up with clean puppet runs
  • 21:15 LeslieCarr: rebooting sockpuppet for kernel upgrades
  • 20:51 logmsgbot: py synchronized wmf-config/db.php 's7: depooling db56 for reimage'
  • 20:48 LeslieCarr: upgrading sockpuppet's 57 pending security upgrades
  • 20:31 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php 're-enable aftv5 on dewiki after having created missing tables'
  • 20:07 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Fix dewiki breakage until tables are created and everyone's bikeshedding over the proper way to deploy'
  • 19:52 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php
  • 19:51 logmsgbot: mlitn synchronized wmf-config/CommonSettings.php
  • 19:51 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php
  • 19:50 logmsgbot: mlitn synchronized php-1.21wmf5/extensions/ArticleFeedbackv5 'desc'
  • 19:49 logmsgbot: mlitn synchronized php-1.21wmf4/extensions/ArticleFeedbackv5 'desc'
  • 19:36 LeslieCarr: committing to pfw1-eqiad - has occasionally lost a few packets in the past
  • 19:24 mutante: upgraded Bugzilla to 4.2.4
  • 18:47 notpeter: taking down bast1001 for reimaging
  • 18:20 logmsgbot: reedy synchronized php-1.21wmf5/extensions/GoogleNewsSitemap
  • 17:43 logmsgbot: reedy synchronized php-1.21wmf5/extensions/GoogleNewsSitemap/
  • 17:40 cmjohnson1: cp1031 powering down for fan assembly/main board swap rt3614
  • 16:48 cmjohnson1: es1004 replacing bad hdd slot 1 (rt3954)
  • 15:58 cmjohnson1: authdns update...adding pc1001-1003
  • 14:45 mark: Installed ms-fe3001
  • 14:08 Nemo_bis: job queue snapshot for bug 42614: Commons 112k, fr.wiki 4M, pl.wiki 21k, pt.wiki 65k, other tops 5k at most
  • 09:28 logmsgbot: nikerabbit synchronized php-1.21wmf5/extensions/Narayam/ 'i18n deploy'
  • 05:02 K4-713: revision-locked payments1004 to 4c15a56c46bf5, all other payments hosts updated to ec2a05d768355
  • 02:44 logmsgbot: LocalisationUpdate completed (1.21wmf4) at Tue Dec 4 02:44:31 UTC 2012
  • 02:23 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Tue Dec 4 02:23:29 UTC 2012
  • 01:31 logmsgbot: spage synchronized php-1.21wmf5/extensions/E3Experiments 'E3 backports to wmf5'
  • 01:00 mutante: gallium - removing multiarch support (32bit on 64bit systems) which causes issues with librsvg package
  • 00:47 logmsgbot: pgehres synchronized php-1.21wmf4/extensions/CentralNotice/ 'Updating CentralNotice to master. Adding API for CN logs'
  • 00:43 logmsgbot: pgehres synchronized php-1.21wmf5/extensions/CentralNotice/ 'Updating CentralNotice to master. Adding API for CN logs'
  • 00:36 Tim: killing long-running runJobs.php processes affected by LST bug
  • 00:15 logmsgbot: asher synchronized wmf-config/db.php 'adding db68 to s7'
  • 00:00 mutante: gallium - restarting Apache, enabling new ServerAlias

December 3

  • 23:57 logmsgbot: aaron synchronized php-1.21wmf5/extensions/CentralAuth/CentralAuthUser.php 'revert live hacks, exception already dumped in bugzilla'
  • 23:34 Tim: installed php-luasandbox on fenari
  • 22:48 LeslieCarr: restarting pybal on lvs1003 for parsoid vip
  • 22:40 mutante: DNS update - removing broken dvd.wikimeedia.org
  • 22:40 LeslieCarr: restarted pybal on lvs1006 and lvs1005
  • 22:30 LeslieCarr: restarting pybal on lvs1006 for parsoid vip
  • 22:26 LeslieCarr: modify above - restarting PYBAL on lvs for parsoid vip
  • 22:26 LeslieCarr: restarting lvs4 for parsoid vip
  • 22:22 LeslieCarr: restarting pybal on lvs3
  • 22:11 K4-713: updated payments cluster to d7946a804d763
  • 22:07 mutante: gallium - removed librsvg2-2:i386, installed librsvg2-2, librsvg2-bin
  • 22:07 logmsgbot: aaron synchronized wmf-config/CommonSettings.php
  • 22:03 K4-713: updated payments to 8eb328e96a1ef6ac
  • 21:53 K4-713: updated payments to 54d3f8f0f9c7bdc
  • 21:36 logmsgbot: aaron synchronized php-1.21wmf5/extensions/CentralAuth/CentralAuthUser.php 'added exception'
  • 21:28 mutante: DNS update - adding doc and integration CNAMEs
  • 21:04 Ryan_Lane: adding mwang to the ops ldap group
  • 20:48 paravoid: deploying squid config for the updated mobile ACL
  • 20:48 hashar: jenkins / gallium : updating Zuul repository 2747501..f73a04d , should be updated by puppet automatically
  • 20:42 logmsgbot: kaldari synchronized php-1.21wmf5/extensions/Vector/modules/ext.vector.collapsibleNav.css 'syncing vector css for menus'
  • 20:33 binasher: added max_user_connections to wikiadmin grants on s1
  • 20:32 logmsgbot: kaldari synchronized php-1.21wmf5/resources/startup.js 'syncing startup.js per Roan'
  • 20:23 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'wikidatawiki back to writeable'
  • 19:29 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'wikidatawiki marked readonly'
  • 19:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf5
  • 18:40 mutante: deleting broken link to nagios3.conf on spence and restarting apache
  • 18:25 mutante: installing package upgrades on spence
  • 18:16 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'deployed b999dbca1f8ca0a76fe327dd1ae30beb6eed462f'
  • 17:30 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 16:57 mark: Installed ms-be3003
  • 16:49 mark: Installed ms-be3004
  • 15:35 cmjohnson1: mw16 shutting down to troubleshoot DIMM error rt3951
  • 15:14 pp-pdf1: upgrade mwlib to 0.14.2
  • 13:12 logmsgbot: demon synchronized wmf-config/throttle.php 'Deploy I826b9114'
  • 12:58 mark: Shutdown slauerhoff's switchports
  • 02:46 logmsgbot: LocalisationUpdate completed (1.21wmf4) at Mon Dec 3 02:46:35 UTC 2012
  • 02:24 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Mon Dec 3 02:24:47 UTC 2012

December 2

  • 02:43 logmsgbot: LocalisationUpdate completed (1.21wmf4) at Sun Dec 2 02:43:34 UTC 2012
  • 02:22 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Sun Dec 2 02:21:55 UTC 2012

December 1

  • 02:45 logmsgbot: LocalisationUpdate completed (1.21wmf4) at Sat Dec 1 02:45:47 UTC 2012
  • 02:24 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Sat Dec 1 02:24:14 UTC 2012
  • 01:17 logmsgbot: catrope synchronized php-1.21wmf5/cache/l10n/ 'Rebuild l10n cache'
  • 01:04 logmsgbot: catrope synchronized php-1.21wmf4/cache/l10n/ 'Rebuild l10n cache'
  • 00:52 logmsgbot: aaron synchronized php-1.21wmf5/includes/filebackend/SwiftFileBackend.php 'deployed db4574540c993aeab8f0840de916e47fdac6199f'
  • 00:41 logmsgbot: olivneh synchronized php-1.21wmf5/extensions/EventLogging/content/JsonSchemaHooks.php 'Assign NS_SCHEMA rights to editinterface'
  • 00:21 logmsgbot: catrope synchronized php-1.21wmf5/extensions/VisualEditor 'Updating VisualEditor to master'
  • 00:20 logmsgbot: catrope synchronized php-1.21wmf4/extensions/VisualEditor 'Updating VisualEditor to master'
  • 00:10 logmsgbot: olivneh synchronized php-1.21wmf5/extensions/EventLogging/content/JsonSchemaHooks.php 'Update SCHEMA_NS perms on wmf5'
  • 00:03 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Returned itwikisource to 1.21wmf5

November 30

  • 23:57 logmsgbot: aaron synchronized php-1.21wmf5/extensions/LabeledSectionTransclusion 'deployed 23e25751e59fa5ce7b4e337780d13818c2e36260'
  • 22:20 binasher: running patch-uploadstash-us_props.sql migration on all wikis
  • 20:55 mutante: for log completeness - wikivoyage.com usable, redirects to .org
  • 19:30 notpeter: temp stopping puppet on brewster
  • 19:24 logmsgbot: pgehres synchronized php-1.21wmf4/resources/startup.js 'Touching to update RL'
  • 19:23 logmsgbot: pgehres synchronized php-1.21wmf4/extensions/CentralNotice/ 'Updating CN to master. More banner buckets for A/B testing'
  • 19:21 logmsgbot: pgehres synchronized php-1.21wmf5/resources/startup.js 'Touching to update RL'
  • 19:18 logmsgbot: pgehres synchronized php-1.21wmf5/extensions/CentralNotice/ 'Updating CN to master. More banner buckets for A/B testing'
  • 19:11 RobH: authdns-update for server antimony
  • 18:50 cmjohnson1: analytics1007 troubleshooting hdd/pxe boot issue (going to power cycle several times)
  • 18:21 RobH: authdns-update to add wtp1001 info to dns
  • 03:18 K4-713: updated payments cluster to 4c15a56c46bf52a
  • 02:49 logmsgbot: LocalisationUpdate completed (1.21wmf4) at Fri Nov 30 02:49:43 UTC 2012
  • 02:26 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Fri Nov 30 02:26:44 UTC 2012
  • 01:40 logmsgbot: olivneh synchronized php-1.21wmf5/extensions/EventLogging/content/JsonSchemaHooks.php
  • 01:14 logmsgbot: py synchronized wmf-config/db.php 's7: db26 out, db68 in, db56 lower weight'
  • 00:27 logmsgbot: olivneh Finished syncing Wikimedia installation... :
  • 00:16 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'LST for test2wiki'

November 29

  • 23:51 logmsgbot: dzahn gracefulled all apaches
  • 23:35 logmsgbot: olivneh Started syncing Wikimedia installation... :
  • 23:32 logmsgbot: dzahn gracefulled all apaches
  • 23:20 logmsgbot: olivneh synchronized php-1.21wmf5/extensions/EventLogging
  • 23:16 logmsgbot: spage synchronized php-1.21wmf4/extensions/E3Experiments 'E3 deploy ACUX at 100%'
  • 23:14 mutante: DNS update - add wikivoyage.com
  • 22:58 notpeter: restarting incremental indexer on searchidx2
  • 22:56 logmsgbot: kaldari synchronized php-1.21wmf5/extensions/UploadWizard/resources/mw.UploadWizardDetails.js 'resyncing UploadWizard patch from yesterday'
  • 22:45 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Enabling EventLogging on metawiki'
  • 22:44 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Updating configuration of EventLogging'
  • 22:38 logmsgbot: olivneh synchronized php-1.21wmf4/extensions/EventLogging/modules/ext.EventLogging.js
  • 22:14 notpeter: starting innobackup from db1041 to db68 (s7)
  • 21:48 logmsgbot: asher synchronized wmf-config/db.php 'pooling db66 as a new s3 slave'
  • 21:27 mutante: old swift logs on iron, some rm'ed, some gzipped, archiving ben's home contents on tridge, clean up iron
  • 19:49 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: itwikisource back to 1.21wmf4
  • 19:48 mutante: installing package upgrades on iron
  • 19:41 mutante: removing mysql server from iron
  • 19:29 RobH: authdns-update to add dns for tin
  • 19:25 logmsgbot: demon synchronized php-1.21wmf5/extensions/LabeledSectionTransclusion/ 'Updating LST to master'
  • 19:22 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'TemplateSandbox on all wikis'
  • 19:21 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'TemplateSandbox on all wikis'
  • 18:13 logmsgbot: aaron synchronized wmf-config/PrivateSettings.php 'Set Swift auth URL to the IP address host.'
  • 17:30 cmjohnson1: es1004 has raid error...power cycling to enter raid bios
  • 15:37 cmjohnson1: provisioning labsdb1002
  • 14:37 cmjohnson1: provisioning labsdb1001
  • 06:09 Tim: fixing searchidx2 by increasing RMI timeout
  • 02:45 logmsgbot: LocalisationUpdate completed (1.21wmf4) at Thu Nov 29 02:45:27 UTC 2012
  • 02:24 logmsgbot: LocalisationUpdate completed (1.21wmf5) at Thu Nov 29 02:24:32 UTC 2012
  • 00:48 logmsgbot: pgehres synchronized php-1.21wmf4/extensions/CentralNotice/ 'Updating CentralNotice to master to hide banners on mobile devices'
  • 00:45 logmsgbot: pgehres synchronized php-1.21wmf5/extensions/CentralNotice/ 'Updating CentralNotice to master to hide banners on mobile devices'
  • 00:12 Tim: uploaded new lucene-search-2 with fixed init script, will upgrade it on the search servers

November 28

  • 23:30 logmsgbot: kaldari synchronized php-1.21wmf5/skins/vector/screen.css 'updating vector skin'
  • 23:30 logmsgbot: kaldari synchronized php-1.21wmf5/skins/modern/main.css 'updating modern skin'
  • 23:29 logmsgbot: kaldari synchronized php-1.21wmf5/extensions/Vector/modules/ext.vector.collapsibleNav.css 'adding forward-compat to Vector for bug 42452'
  • 22:59 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: Moving all non-pedias back to wmf5
  • 22:46 cmjohnson1: running authdns update
  • 22:45 cmjohnson1: adding DNS entry labsdb1001/1002 .eqiad.wmnet
  • 22:41 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: metawiki to wmf5
  • 22:33 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: Rolling wikiversions.dat back to de90c27d, undoing todays deploy
  • 22:18 logmsgbot: aaron synchronized php-1.21wmf4/extensions/SwiftCloudFiles
  • 22:01 Ryan_Lane: disabled page counters on labsconsole
  • 21:27 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: Commonswiki back to 1.21wmf5
  • 21:13 logmsgbot: kaldari synchronized php-1.21wmf5/extensions/UploadWizard/resources/mw.UploadWizardDetails.js 'fixing uplaodwizrd for ie8'
  • 20:41 logmsgbot: aaron synchronized php-1.21wmf5/extensions/SwiftCloudFiles 'deployed d50d754971991bb2362455fe47eaadb20aff3a80'
  • 20:41 RoanKattouw: Restarted logmsgbot
  • 18:48 notpeter: temp stopping puppet on brewster
  • 14:45 ^demon: fenari:/home/wikipedia/common now reflects Ia319794c - rm'd all traces of submodule
  • 05:47 Tim: on searchidx2: killed extra IncrementalUpdater daemon
  • 04:12 Tim: on searchidx2: restarted incremental updater
  • 00:16 MaxSem: Scap complete

November 27

  • 23:14 Ryan_Lane: restarted gerrit; was getting WARN org.eclipse.jetty.util.log : Dispatched Failed! errors
  • 22:27 MaxSem: scapping as a part of usual mobile deployment
  • 22:23 mutante: starting squid backend on knsq18 (cron spam/nagios)
  • 21:52 Reedy: reedy synchronized php-1.21wmf5/extensions/LabeledSectionTransclusion
  • 21:50 robla: logmsgbot doesn't seem to be working for sync-dir, scap, etc
  • 20:46 Ryan_Lane: upgrading salt in production, in batches of 10
  • 20:18 Ryan_Lane: upgrading salt on all labs minions
  • 20:07 notpeter: temp stopping puppet on brewster
  • 19:31 mutante: upgrade libc* packages on sodium and singer
  • 19:29 mutante: installing a bunch of pkg upgrades on zirconium
  • 16:07 paravoid: ignore previous labs-related entry
  • 16:06 paravoid: bastion disabled hazard-sj's cron, as it was accidentally DoSing bastion
  • 14:15 apergos: snapshot1002 down for reinstall and testing
  • 12:32 mark: Removed ms6 out of the loop as backend server for esams upload squids
  • 04:11 pgehres: TimStarling: restarting lucene-search-2 on search1016 due to init script bug
  • 04:11 pgehres: synchronized wmf-config/reporting-setup.php "Setting S:FundraiserStatistics to use db1013 as the slave"

November 26

  • 19:29 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.21wmf5
  • 19:27 logmsgbot_: reedy synchronized php-1.21wmf5/extensions/UploadWizard
  • 19:24 logmsgbot_: reedy synchronized php-1.21wmf5/cache/l10n/
  • 19:10 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuild localisation cache for 1.21wmf5
  • 18:58 logmsgbot_: reedy synchronized php-1.21wmf5/cache/l10n/
  • 18:47 logmsgbot_: reedy synchronized php-1.21wmf5/extensions/Wikibase
  • 18:45 logmsgbot_: reedy synchronized php-1.21wmf5/extensions/Diff
  • 18:37 logmsgbot_: aaron synchronized php-1.21wmf5/includes/filerepo/file/LocalFile.php 'deployed 1fa41352d4e0b3cd2e53e7d8d485ac9098b8bfcc'
  • 18:31 logmsgbot_: reedy Started syncing Wikimedia installation... : Build localisation cache for 1.21wmf5
  • 18:16 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.21wmf5 for l10n stuffs
  • 18:15 logmsgbot_: aaron synchronized php-1.21wmf4/thumb.php 'deployed 81c1d0133f74eec600277ab744a9730c4e974e47'
  • 18:14 logmsgbot_: reedy synchronized php-1.21wmf5 'Initial sync of php-1.21wmf5'
  • 18:03 logmsgbot_: aaron synchronized php-1.21wmf5/extensions/FlaggedRevs
  • 18:02 logmsgbot_: reedy synchronized wmf-config/ExtensionMessages-1.21wmf5.php
  • 18:01 logmsgbot_: reedy synchronized live-1.5/static-1.21wmf5/
  • 17:58 logmsgbot_: aaron synchronized php-1.21wmf4/extensions/FlaggedRevs
  • 17:55 mutante: authdns-update - adding wikipedia.bg
  • 16:29 mark: Dist-upgraded & rebooted eqiad upload Varnish servers
  • 15:26 Jeff_Green: dist-upgrades & reboots for payments100[1-4], pay-lvs100[12]
  • 02:24 logmsgbot_: LocalisationUpdate completed (1.21wmf4) at Mon Nov 26 02:23:58 UTC 2012

November 25

  • 22:07 logmsgbot_: reedy synchronized wmf-config/
  • 19:23 logmsgbot_: reedy synchronized wmf-config/flaggedrevs.php
  • 19:01 Reedy: Pointed php symlink to php-1.21wmf4
  • 08:11 apergos: from about half an hour ago, restarted lucene search on search13 and forgot to log it
  • 02:23 logmsgbot_: LocalisationUpdate completed (1.21wmf4) at Sun Nov 25 02:23:41 UTC 2012

November 24

  • 02:24 logmsgbot_: LocalisationUpdate completed (1.21wmf4) at Sat Nov 24 02:24:00 UTC 2012

November 23

  • 23:15 logmsgbot_: olivneh synchronized wmf-config/InitialiseSettings.php 'Enabling PostEdit on ptwiki & svwiki'
  • 18:28 logmsgbot_: maxsem synchronized php-1.21wmf4/includes/Message.php 'Debugging'
  • 16:20 logmsgbot_: maxsem synchronized php-1.21wmf4/includes/Message.php 'Debugging'
  • 15:09 mark: Dist-upgraded and rebooted all eqiad bits servers (new Varnish package)
  • 15:04 logmsgbot_: maxsem synchronized php-1.21wmf4/extensions/CategoryTree/ 'https://gerrit.wikimedia.org/r/#/c/34668/'
  • 14:57 mark: Dist-upgraded and rebooted all pmtpa bits servers (new Varnish package)
  • 14:33 mark: Dist-upgraded and rebooted all esams bits servers (new Varnish package)
  • 14:05 mark: Built new varnish 3.0.3plus~rc1-wm6 packages with fixed epoll deadlock, and inserted it into the precise-wikimedia APT repository
  • 04:20 Tim: on fenari: updated the "search" dsh node group based on nmap -sP and fixed the remaining search servers
  • 04:04 Tim: oh yeah, and I upgraded lucene to my version with the timeouts, deployed to pmtpa only via puppet
  • 04:02 Tim: many lucene search servers failed to bind to port 1099 when they were restarted by the upgrade, restarting manually
  • 02:24 logmsgbot_: LocalisationUpdate completed (1.21wmf4) at Fri Nov 23 02:24:34 UTC 2012

November 22

  • 14:52 apergos: msbe8 and msbe10 installed, not yet deployed
  • 14:12 hashar: jenkins: configure jobbuilder-bot user permissions
  • 13:39 hashar: Jenkins: updated zuul-bot user permission so it can trigger jobs.
  • 12:16 hashar: Installed plugins on Jenkins for Zuul deployment: notification, build-timeout
  • 12:06 hashar: restarted Jenkins to load in the new zuul jobs.
  • 09:23 apergos: fixed perms on srv284 php-1.*wmf* and wmf-config, test (were 700 instead of 777)
  • 02:24 logmsgbot_: LocalisationUpdate completed (1.21wmf4) at Thu Nov 22 02:24:22 UTC 2012
  • 01:28 mutante: sync-apache srv284-only, start apache (was missing all.conf), repool
  • 01:15 mutante: rebooting srv193 (test.wp) for upgrade
  • 00:56 mutante: repooled mw58,mw59 (upgrades) srv284 (hw ticket was resolved, reinstalled)
  • 00:07 mutante: powercycling kaulen, this time no console output at all

November 21

  • 23:47 mutante: tmp. depooling and rebooting a few mw5x servers for kernel upgrades, one by one
  • 23:37 logmsgbot_: preilly synchronized php-1.21wmf4/extensions/ZeroRatedMobileAccess 'update post deploy'
  • 23:34 logmsgbot_: preilly synchronized php-1.21wmf3/extensions/ZeroRatedMobileAccess 'update post deploy'
  • 22:58 mutante: disabled swap on kaulen per Tim's advice
  • 22:55 mutante: bugzilla back up
  • 22:50 mutante: powercycling kaulen
  • 22:25 logmsgbot_: asher synchronized wmf-config/throttle.php 'increased api limit for bug 42319'
  • 21:51 mutante: installing package upgrades in the mw55-mw59 range
  • 21:17 logmsgbot_: preilly Finished syncing Wikimedia installation... : update zero rated mobile access
  • 20:56 Ryan_Lane: deploying Change If5f6bc33: (bug 42334) on labsconsole
  • 20:47 Ryan_Lane: applying changes to fix no creds issue when setting preferences on labsconsole
  • 20:47 logmsgbot_: preilly Started syncing Wikimedia installation... : update zero rated mobile access
  • 20:34 logmsgbot_: preilly synchronized php-1.21wmf4/extensions/ZeroRatedMobileAccess 'update post deploy'
  • 20:34 logmsgbot_: preilly synchronized php-1.21wmf3/extensions/ZeroRatedMobileAccess 'update post deploy'
  • 20:00 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: rest of wikipedias to 1.21wmf4
  • 19:24 mutante: installing package upgrades in the srv193-srv199 range
  • 18:55 mutante: re-signing srv284 on puppetmaster, package installs...
  • 18:45 AaronSchulz: Copying over all math/timeline files to nas1 that are not already there for DR
  • 18:34 AaronSchulz: prepared all wikivoyage file directories and containers
  • 17:41 Reedy: Running scap-recompile on all hosts in mediawiki-installation group
  • 16:38 mutante: powering on analytics1007
  • 16:29 mutante: reinstalling srv284 "last lucid standing"
  • 16:24 mark: Takeback by nas1001-a completed, snapmirror relationships back in sync
  • 16:15 mark: Aborted synchronous snapmirror relationships between nas1-a and nas1001-a to initiate cf giveback
  • 16:08 mutante: powercycling srv284
  • 16:05 mutante: remove known troublemakers srv266 and srv284 from dsh group "apaches"
  • 15:50 mark: Initiated takeover of nas1001-a to nas1001-b
  • 15:49 mark: nas1001-b back up and running
  • 14:46 mark: Halted nas1001-b for NIC upgrade
  • 09:18 notpeter: repooling mw20-mw34
  • 07:50 notpeter: repooling mw70-mw74
  • 07:21 apergos: restarted lucene search on search14
  • 03:01 notpeter: depooling mw70-mw74 for upgrade to precise
  • 03:01 notpeter: depooling mw20-mw34 for upgrade to precise
  • 02:45 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Wed Nov 21 02:45:42 UTC 2012
  • 02:24 logmsgbot_: LocalisationUpdate completed (1.21wmf4) at Wed Nov 21 02:24:47 UTC 2012
  • 02:17 K4-713: updated payments cluster to 6ea97a5cbc06
  • 00:39 logmsgbot_: preilly synchronized php-1.21wmf4/extensions/ZeroRatedMobileAccess 'update post deploy'
  • 00:37 logmsgbot_: preilly synchronized php-1.21wmf3/extensions/ZeroRatedMobileAccess 'update post deploy'
  • 00:06 logmsgbot_: maxsem synchronized php-1.21wmf4/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/34440/'
  • 00:03 logmsgbot_: maxsem synchronized php-1.21wmf3/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/34440/'
  • 00:03 notpeter: depooling mw66-mw69 for upgrade to apache
  • 00:03 notpeter: depooling mw35-mw49 for upgrade to apache

November 20

  • 23:52 logmsgbot_: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
  • 23:35 K4-713: updated payments cluster to b5a80e04086c3a7
  • 23:32 binasher: depooled srv203
  • 23:00 logmsgbot_: reedy synchronized php-1.21wmf4/
  • 22:56 logmsgbot_: reedy synchronized php-1.21wmf3/
  • 22:45 logmsgbot_: kaldari synchronized wmf-config/InitialiseSettings.php 'disabling experimental flickr uploading on commons until some bugs are fixed'
  • 22:44 logmsgbot_: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
  • 22:35 logmsgbot_: asher synchronized wmf-config/InitialiseSettings.php 'now only 30 wikis that need wgOldChangeTagsIndex = true, changing default to false'
  • 21:02 notpeter: depooling mw17-mw19 and mw50-mw54 for upgrade to precise
  • 20:59 notpeter: depooling mw62-mw65 for upgrade to precise
  • 20:37 Jeff_Green: switching payments back to eqiad
  • 20:23 binasher: running change_tag index migrations on the 757 wikis with <7k ct rows
  • 20:09 logmsgbot_: kaldari synchronized wmf-config/InitialiseSettings.php 'turning on flickr uploading for sysops on Commons'
  • 19:42 apergos: scribbled over /dev/sda partition table on ms-be7 so I could watch the install break, left in shell form installer
  • 16:14 notpeter: depooling srv281-srv289 for upgrades to precise
  • 09:57 hashar: Upgrading Jenkins plugin xUnit which introduce a potential back compatibility issue <failedIfNotNew> renamed to <failIfNotNew>. Will update configs files.
  • 09:37 hashar: restarting Jenkins
  • 09:22 hashar: upgrading Jenkins plugins
  • 09:18 logmsgbot_: nikerabbit synchronized wmf-config/InitialiseSettings.php 'i18n deploy'
  • 08:15 logmsgbot_: tstarling Finished syncing Wikimedia installation... :
  • 07:32 paravoid: rebooting ms-be5, strange df output for /
  • 07:01 logmsgbot_: tstarling Started syncing Wikimedia installation... :
  • 06:37 logmsgbot_: tstarling synchronized wmf-config/CommonSettings.php
  • 06:36 logmsgbot_: tstarling synchronized wmf-config/InitialiseSettings.php
  • 06:36 logmsgbot_: tstarling synchronized wmf-config/filebackend.php
  • 06:34 logmsgbot_: tstarling synchronized php-1.21wmf4/extensions/Score
  • 06:33 logmsgbot_: tstarling synchronized php-1.21wmf3/extensions/Score
  • 05:54 Ryan_Lane: rebooting ssl3001
  • 05:19 Ryan_Lane: depooling ssl3001
  • 05:19 Ryan_Lane: repooling ssl3002 (upgrade complete)
  • 05:14 paravoid: deploying squid config for score
  • 05:11 Ryan_Lane: rebooting ssl3002
  • 04:48 Ryan_Lane: depooling ssl3002 for upgrade to precise
  • 04:48 Ryan_Lane: repooling ssl3003 (upgrade finished)
  • 04:42 Ryan_Lane: rebooting ssl3003 (upgraded via ssh, not console)
  • 04:25 notpeter: repooling srv258-srv280
  • 04:04 Ryan_Lane: repooling ssl1 (upgrade complete)
  • 03:51 Tim: on brewster: reduced rotate count for squid logs to zero and ran logrotate -f
  • 03:44 Tim: on brewster: root partition is full, removing some useless squid logs
  • 02:52 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Tue Nov 20 02:52:09 UTC 2012
  • 02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf4) at Tue Nov 20 02:28:20 UTC 2012
  • 02:10 notpeter: depooling srv296-srv301 for upgrades to precise
  • 02:08 notpeter: repooling srv290-srv295
  • 01:21 Ryan_Lane: depooling ssl1
  • 01:21 Ryan_Lane: repooling ssl2 (upgrade complete)
  • 01:10 Ryan_Lane: depooling ssl3003 to upgrade to precise
  • 01:02 MaxSem: Scap hung and had to be aborted. Since what was being deployed wasn't enabled, no clusters were harmed.
  • 00:46 Ryan_Lane: all eqiad https hosts upgraded to precise
  • 00:46 Ryan_Lane: repooling ssl1001 (upgrade complete)
  • 00:23 Ryan_Lane: depooling ssl2 for upgrade to precise
  • 00:23 Ryan_Lane: repooling ssl4 (upgrade complete)
  • 00:21 logmsgbot_: maxsem Started syncing Wikimedia installation... : Black-deployment of ext:Solarium, not enabled anywhere yet
  • 00:01 Ryan_Lane: depooling ssl1001
  • 00:01 Ryan_Lane: repooling ssl1002 (upgrade complete)

November 19

  • 23:59 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
  • 23:37 Ryan_Lane: repooling ssl3 depooling ssl4
  • 23:08 logmsgbot_: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/34220/ and https://gerrit.wikimedia.org/r/#/c/33786/'
  • 23:06 Ryan_Lane: depooling ssl1002 for upgrade to precise
  • 23:06 Ryan_Lane: repooling ssl1003 (upgrade to precise complete)
  • 22:59 Ryan_Lane: make that rebooting ssl1003
  • 22:59 Ryan_Lane: rebooting ssl1004
  • 22:53 Ryan_Lane: rebooting ssl3
  • 22:51 Ryan_Lane: re-pooling ssl4 and depooling ssl3
  • 22:35 Ryan_Lane: depooling ssl4
  • 22:35 Ryan_Lane: upgrading ssl1003 to precise
  • 22:13 Ryan_Lane: depooling ssl1003
  • 20:14 mark: Restoring bits.esams traffic
  • 19:56 mark: Moving bits.esams traffic to pmtpa
  • 19:20 notpeter: depooling srv290-srv301 for upgrade to precise
  • 19:20 notpeter: depooling srv258-srv280 for upgrade to precise
  • 19:11 logmsgbot_: py synchronized wmf-config/db.php 'repooling es1'
  • 18:45 logmsgbot_: maxsem synchronized wmf-config/mobile.php 'https://gerrit.wikimedia.org/r/#/c/34105/'
  • 18:14 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf4
  • 17:10 logmsgbot_: reedy synchronized php-1.21wmf3/cache/interwiki.cdb
  • 17:09 logmsgbot_: reedy synchronized php-1.21wmf4/cache/interwiki.cdb
  • 15:58 paravoid: powercycling ms-be3, unresponsive
  • 15:48 logmsgbot_: reedy synchronized php-1.21wmf4/extensions/OAI
  • 14:48 logmsgbot_: reedy synchronized php-1.21wmf3/cache/interwiki.cdb 'Updating interwiki cache'
  • 14:48 logmsgbot_: reedy synchronized php-1.21wmf4/cache/interwiki.cdb 'Updating interwiki cache'
  • 14:23 paravoid: putting ms-fe1 back into the pool
  • 08:36 hashar: running sync-common on mw46
  • 06:56 apergos: restarted lucene search on search14
  • 02:43 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Mon Nov 19 02:43:07 UTC 2012
  • 02:23 logmsgbot_: LocalisationUpdate completed (1.21wmf4) at Mon Nov 19 02:23:19 UTC 2012
  • 00:35 Tim: on hume: running fixBug41778.php on all wikis
  • 00:31 logmsgbot_: tstarling synchronized php-1.21wmf3/extensions/WikimediaMaintenance/fixBug41778.php
  • 00:30 logmsgbot_: tstarling synchronized php-1.21wmf4/extensions/WikimediaMaintenance/fixBug41778.php

November 18

  • 22:53 logmsgbot_: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/34034/'
  • 21:44 logmsgbot_: maxsem synchronized php-1.21wmf4/extensions/TocTree/ 'https://gerrit.wikimedia.org/r/#/c/33987/ to fix RL warnings'
  • 21:33 logmsgbot_: maxsem synchronized php-1.21wmf4/includes/resourceloader/ResourceLoader.php 'Debugging over'
  • 21:26 logmsgbot_: maxsem synchronized php-1.21wmf3/extensions/Narayam/ 'Deploying https://gerrit.wikimedia.org/r/#/c/34024/ to unbreak RL'
  • 21:25 logmsgbot_: maxsem synchronized php-1.21wmf4/extensions/Narayam/ 'Deploying https://gerrit.wikimedia.org/r/#/c/34024/ to unbreak RL'
  • 20:58 logmsgbot_: maxsem synchronized php-1.21wmf4/includes/resourceloader/ResourceLoader.php 'Debugging'
  • 09:26 logmsgbot_: aaron synchronized php-1.21wmf4/includes/upload/UploadFromChunks.php
  • 09:16 logmsgbot_: aaron synchronized php-1.21wmf4/includes/upload/UploadFromChunks.php 'debug logging'
  • 09:09 logmsgbot_: aaron synchronized php-1.21wmf4/includes/upload/UploadFromChunks.php 'debug logging'
  • 08:11 apergos: restarted lucene search on search13
  • 07:22 logmsgbot_: aaron synchronized php-1.21wmf4/includes/upload/UploadFromChunks.php
  • 02:53 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Sun Nov 18 02:52:59 UTC 2012
  • 02:29 logmsgbot_: LocalisationUpdate completed (1.21wmf4) at Sun Nov 18 02:29:14 UTC 2012

November 17

  • 16:46 binasher: restarting lsearchd on all eqiad hosts
  • 16:21 logmsgbot_: asher synchronized wmf-config/lucene.php 'failing pool3+4 to pmtpa'
  • 16:18 logmsgbot_: asher synchronized wmf-config/lucene.php 'failing pool1+2 to pmtpa'
  • 02:52 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Sat Nov 17 02:52:30 UTC 2012
  • 02:27 logmsgbot_: LocalisationUpdate completed (1.21wmf4) at Sat Nov 17 02:27:00 UTC 2012

November 16

  • 23:00 logmsgbot_: maxsem synchronized php-1.21wmf3/extensions/TimedMediaHandler 'https://gerrit.wikimedia.org/r/#/c/33819/'
  • 22:58 logmsgbot_: maxsem synchronized php-1.21wmf4/extensions/TimedMediaHandler 'https://gerrit.wikimedia.org/r/#/c/33819/'
  • 22:44 notpeter: repooling srv231-srv247
  • 22:21 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 22:18 logmsgbot_: catrope synchronized php-1.21wmf4/resources/jquery/jquery.tablesorter.js 'Deploying 9cb43905cd7189aa9a42f942f24a7aea50b4f0e9'
  • 22:18 notpeter: repooling srv214-srv218
  • 22:17 logmsgbot_: catrope synchronized php-1.21wmf3/resources/jquery/jquery.tablesorter.js 'Deploying 2405956a8d6f305f6d3f9b0a1bda4deb7e917ec8'
  • 21:30 notpeter: starting innobackupex from es1004 to es1
  • 21:09 logmsgbot_: py synchronized wmf-config/db.php 'pulling es1'
  • 20:40 binasher: running patch-job_attempts.sql migration on all wikis
  • 20:30 binasher: purging zhwiki refreshLinks2 jobs
  • 20:03 notpeter: depooling srv231-srv247 from apaches pool for upgrade to precise
  • 20:02 notpeter: depooling srv214-srv218 from api pool for upgrade to precise
  • 19:48 logmsgbot_: py synchronized wmf-config/db.php 'es2 going live'
  • 18:56 notpeter: temp stopping puppet on brewster
  • 17:41 paravoid: depooling ms-fe1
  • 17:15 logmsgbot_: hashar synchronized wmf-config/CommonSettings.php 'Redis-setup is unavailable for wmflabs, use memcached 33721'
  • 16:49 Reedy: Running sync-common on mw52
  • 15:34 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/EducationProgram/
  • 15:33 logmsgbot_: reedy synchronized php-1.21wmf4/extensions/EducationProgram/
  • 15:16 paravoid: pooling ms-fe1 back with the modified rewrite.py
  • 12:37 Reedy: Running sync common on srv252, lots of Failed opening required fatals...
  • 04:58 K4-713: updated payments cluster to c898ce38e162516e73
  • 03:11 Tim: deleted corrupted memcached value enwiki:gadgets-definition:6
  • 02:48 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Fri Nov 16 02:48:37 UTC 2012
  • 02:37 RoanKattouw: Forcing puppet run on wtp1 to reinstall Parsoid
  • 02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf4) at Fri Nov 16 02:28:52 UTC 2012
  • 01:49 logmsgbot_: aaron synchronized php-1.21wmf4/thumb.php 'remove hacks'
  • 01:34 logmsgbot_: aaron synchronized php-1.21wmf4/thumb.php 'more debugging'
  • 01:33 logmsgbot_: aaron synchronized php-1.21wmf4/thumb.php
  • 01:32 logmsgbot_: aaron synchronized php-1.21wmf4/thumb.php
  • 01:17 logmsgbot_: aaron synchronized php-1.21wmf4/thumb.php
  • 01:10 logmsgbot_: aaron synchronized php-1.21wmf4/thumb.php 'debug logging'
  • 00:35 logmsgbot_: tstarling synchronized wmf-config/InitialiseSettings.php
  • 00:35 logmsgbot_: tstarling synchronized wmf-config/CommonSettings.php
  • 00:30 logmsgbot_: tstarling synchronized wmf-config/InitialiseSettings.php
  • 00:29 logmsgbot_: tstarling synchronized wmf-config/InitialiseSettings.php
  • 00:28 logmsgbot_: tstarling synchronized small.dblist
  • 00:21 logmsgbot_: tstarling synchronized php-1.21wmf4/extensions/UniversalLanguageSelector

November 15

  • 22:41 logmsgbot_: reedy synchronized php-1.21wmf4/extensions/ReaderFeedback/
  • 22:36 logmsgbot_: reedy synchronized php-1.21wmf4/extensions/WikimediaIncubator/
  • 22:29 Reedy: Ran sync-docroot
  • 22:08 notpeter: repooling srv25-srv257
  • 22:06 notpeter: repooling srv190-srv192, srv225-srv230
  • 21:33 binasher: starting apache on srv249
  • 21:32 binasher: stopping apache on srv249
  • 21:30 binasher: ran chmod 755 /usr/local/apache/common-local/wmf-config on srv249
  • 20:46 Jeff_Green: adjusted key_buffer_size for fundraisingdbs from 1M to 1G
  • 20:42 notpeter: removing srv225-srv230 for upgrade to precise
  • 20:34 notpeter: reimaging srv190-srv192 as regular apaches
  • 20:09 logmsgbot_: reedy synchronized php-1.21wmf3/
  • 20:05 logmsgbot_: reedy synchronized php-1.21wmf4/
  • 19:37 notpeter: repooling srv200-srv213
  • 19:23 notpeter: removing srv251-srv257 for upgrade to precise
  • 19:19 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 18:59 paravoid: upgrading srv{190,219,220,221,222,223,224}.pmtpa.wmnet for libtiff vulnerabilities (USN-1631-1)
  • 17:40 cmjohnson1: wtp1 raid1 config is wrong; bringing server down to fix
  • 17:12 notpeter: removing srv248, srv249, and srv250 from various pools for upgrade to precise
  • 16:47 paravoid: running 'reprepro update' to fetch ceph packages
  • 16:47 cmjohnson1: shutting down srv250 to verify drac config
  • 15:01 logmsgbot_: reedy synchronized php-1.21wmf4/extensions/SiteMatrix/
  • 14:32 notpeter: removing srv200-srv213 from pybal for upgrades to precise (and this time I mean it)
  • 13:48 logmsgbot_: py synchronized wmf-config/db.php 'pulling es2 for conversion to inno'
  • 11:26 logmsgbot_: kaldari synchronized php-1.21wmf4/extensions/UploadWizard/resources/mw.UploadWizardUploadInterface.js 'refixing for Safari, part 2'
  • 11:26 logmsgbot_: kaldari synchronized php-1.21wmf4/extensions/UploadWizard/resources/mw.UploadWizardDetails.js 'refixing for Safari, part 1'
  • 11:15 logmsgbot_: kaldari synchronized php-1.21wmf4/extensions/UploadWizard/resources/mw.UploadWizardUploadInterface.js 'refixing for Safari, part 2'
  • 11:14 logmsgbot_: kaldari synchronized php-1.21wmf4/extensions/UploadWizard/resources/mw.UploadWizardDetails.js 'refixing for Safari, part 1'
  • 11:04 logmsgbot_: kaldari synchronized php-1.21wmf4/extensions/UploadWizard/resources/mw.UploadWizardDetails.js 'rolling back my fix for Safari, part 1'
  • 11:03 logmsgbot_: kaldari synchronized php-1.21wmf4/extensions/UploadWizard/resources/mw.UploadWizardUploadInterface.js 'rolling back my fix for Safari, part 2'
  • 10:48 logmsgbot_: kaldari synchronized php-1.21wmf4/extensions/UploadWizard/resources/mw.UploadWizardUploadInterface.js 'syncing UploadWizard fix for Safari, part 2'
  • 10:47 logmsgbot_: kaldari synchronized php-1.21wmf4/extensions/UploadWizard/resources/mw.UploadWizardDetails.js 'syncing UploadWizard fix for Safari, part 1'
  • 09:14 logmsgbot_: aaron synchronized php-1.21wmf4/maintenance/nextJobDB.php 'deployed 2e0c24df43588bd4ceba5522bde1b3f06fbd05b0'
  • 07:25 Nemo_bis: Job queue still suspect since the wmf4 deploy, job runners activity halved; see bug 41656
  • 04:20 K4-713: updated payments cluster to a24d39a120f61cb
  • 02:54 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Thu Nov 15 02:54:48 UTC 2012
  • 02:34 logmsgbot_: LocalisationUpdate completed (1.21wmf4) at Thu Nov 15 02:34:15 UTC 2012
  • 02:20 K4-713: updated payments cluster to 46bcee539b93f7b
  • 01:49 logmsgbot_: catrope synchronized php-1.21wmf4/extensions/VisualEditor 'Updating VE to master'
  • 01:49 logmsgbot_: catrope synchronized php-1.21wmf3/extensions/VisualEditor 'Updating VE to master'
  • 01:45 logmsgbot_: catrope synchronized php-1.21wmf4/extensions/VisualEditor 'Updating VE to master'
  • 01:44 logmsgbot_: catrope synchronized php-1.21wmf3/extensions/VisualEditor 'Updating VE to master'
  • 00:46 K4-713: updated payments cluster to ab3d3615bf1ddd0c
  • 00:35 logmsgbot_: catrope synchronized php-1.21wmf4/extensions/VisualEditor 'Updating VE to master'
  • 00:35 logmsgbot_: catrope synchronized php-1.21wmf3/extensions/VisualEditor 'Updating VE to master'
  • 00:23 K4-713: updated payments cluster to 4ccd348df7661
  • 00:11 logmsgbot_: asher synchronized wmf-config/db.php 'pulling db11'

November 14

  • 23:59 K4-713: Updated payments cluster to 5edbe2cdc1689d14ebb883
  • 23:50 logmsgbot_: catrope synchronized php-1.21wmf4/extensions/VisualEditor 'Updating VE to master'
  • 23:49 logmsgbot_: catrope synchronized php-1.21wmf3/extensions/VisualEditor 'Updating VE to master'
  • 23:27 logmsgbot_: catrope synchronized wmf-config/InitialiseSettings.php 'Dark launch VisualEditor on enwiki'
  • 23:11 logmsgbot_: catrope synchronized php-1.21wmf4/extensions/VisualEditor 'Updating VE to master'
  • 23:10 logmsgbot_: catrope synchronized php-1.21wmf3/extensions/VisualEditor 'Updating VE to master'
  • 22:55 logmsgbot_: aaron synchronized wmf-config/filebackend.php
  • 22:55 RoanKattouw: Belatedly starting VisualEditor dark launch, going a bit over time
  • 22:52 logmsgbot_: aaron synchronized wmf-config/filebackend.php 're-enabled writes to thumbs2/'
  • 22:43 logmsgbot_: aaron synchronized wmf-config/filebackend.php
  • 22:41 logmsgbot_: aaron synchronized wmf-config/filebackend.php
  • 22:22 logmsgbot_: aaron synchronized wmf-config/filebackend.php 'revert for a moment'
  • 22:19 logmsgbot_: aaron synchronized wmf-config/filebackend.php 'Multiwrite thumbs to /thumbs2 and math/timeline files to nas as well.'
  • 21:55 logmsgbot_: reedy synchronized php-1.21wmf4/extensions/Echo/
  • 21:48 logmsgbot_: pgehres synchronized wmf-config/CommonSettings.php
  • 21:41 logmsgbot_: pgehres synchronized php-1.21wmf4/extensions/ContributionReporting/
  • 21:32 notpeter: repooling srv200-srv213 due to lack of automated imaging :(
  • 21:19 binasher: restarted lsearchd on all pmtpa lucene hosts
  • 20:44 notpeter: removing srv200-srv213 from apaches pool for upgarde to precise
  • 20:42 binasher: fixed private wiki search index builds in pmtpa (standby cluster) and building initial wikivoyagedata indices there
  • 20:38 Reedy: Killing /tmp/mw-cache-1.20wmf* from all apaches
  • 20:20 mutante: installing some package upgrades on sodium (lists)
  • 20:10 Reedy: Running sync-common on mw53
  • 19:56 Reedy: srv269 has a full /tmp. Cleaning up
  • 19:55 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else (non wikipedia) to 1.21wmf4
  • 19:52 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks and wikinews to 1.21wmf4
  • 19:46 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: private, readonly, closed and special to 1.21wmf4
  • 19:40 paravoid: depooling ms-fe1 for rewrite.py testing
  • 19:03 Reedy: Running sync-common on mw48
  • 18:28 notpeter: temp stopping puppet on brewster
  • 18:24 mutante: patching Bugzilla to 4.0.9
  • 17:56 logmsgbot_: aaron synchronized php-1.21wmf3/extensions/TimedMediaHandler 'deployed 275aa9524ea6e1e15d7eef3d81c68dde3ffd5693'
  • 17:46 Reedy: oai.oaiaudit cleaned out on S3
  • 16:39 RobH: server wmf5815 named 'otto' after Jan Otto, name suggested by DannyB
  • 16:22 logmsgbot_: py synchronized wmf-config/db-secondary.php 'updating comments in db-secondary.php'
  • 12:52 mark: Reinstall ms-be3001
  • 04:11 Tim: on zhwiki: removing refreshLinks2 jobs for templates "Side box", "Sister", "Flag/core"
  • 03:25 Tim: on zhwiki: removing 970k country data jobs
  • 02:50 logmsgbot_: LocalisationUpdate completed (1.21wmf4) at Wed Nov 14 02:50:04 UTC 2012
  • 02:29 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Wed Nov 14 02:29:53 UTC 2012
  • 00:54 logmsgbot_: aaron synchronized php-1.21wmf4/extensions/TimedMediaHandler 'deployed 2784d7ab1b0e0a56b905ff4b9c4070f63f867d25'
  • 00:39 logmsgbot_: aaron synchronized wmf-config/filebackend.php 'Set "swiftTempUrlKey"'
  • 00:28 AaronSchulz: Set X-Account-Meta-Temp-Url-Key for swift mw:thumb account
  • 00:27 logmsgbot_: aaron synchronized wmf-config/PrivateSettings.php
  • 00:21 logmsgbot_: asher synchronized wmf-config/mc.php 'disable general memcached multiwriting: nothing but pecl'
  • 00:19 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'remove apache memchached's from the parsercache multiwriter'

November 13

  • 22:56 logmsgbot_: kaldari synchronized wmf-config/PrivateSettings.php 'syncing PrivateSettings back to previous version'
  • 22:41 binasher: converted puppet.updates_update table to innodb - received the most number of of any puppet db table and was myisam, yikes. this might help with concurrency.
  • 22:31 LeslieCarr: restarting gmetad on nickel
  • 22:22 Reedy: Publishing schema for lucene/oai on mw.o....
  • 22:07 mutante: mailman lists: renaming wikimedia-th to wikimedia-th-private
  • 22:06 logmsgbot_: kaldari synchronized wmf-config/PrivateSettings.php 'syncing PrivateSettings for Flickr API key'
  • 18:43 RobH: analytics1007 now pxe boots
  • 18:41 RobH: analytics1007.mgmt locked up, had to remove all power to fix, now working
  • 18:33 RobH: testing analytics1002, pxe boots no problem
  • 17:53 paravoid: setting up media1:thumbs on nas1001-a and semi-sync snapmirror from nas1-a
  • 17:24 Jeff_Green: wikimedia.org zone update, new DKIM keys for silverpop
  • 16:54 paravoid: snapmirror between nas1-a/nas1001-a for all volumes in-sync again
  • 16:50 paravoid: running "cf giveback" on nas1001-b to restore nas1001-a; had to "partner snapmirror off" first.
  • 16:46 apergos: had to restart varnish frontend by hand after reboot, also the varnishncsa for emery, locke, multicast
  • 16:30 apergos: rebooting cp1042 (XFS: possible memory allocation deadlock in kmem_alloc), lots of stuck /bin/ps form nagios, load average ridiculous
  • 16:10 paravoid: creating media1:thumbs volume at nas1-a and setting up /etc/export rules for it
  • 16:08 cmjohnson1: provision labsdb3
  • 15:32 cmjohnson1: provisioning labsdb2
  • 15:14 hashar: restarting jenkins on gallium
  • 08:57 logmsgbot_: reedy synchronized php-1.21wmf4/extensions/TimedMediaHandler
  • 08:56 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler
  • 02:39 logmsgbot_: LocalisationUpdate completed (1.21wmf4) at Tue Nov 13 02:39:21 UTC 2012
  • 02:22 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Tue Nov 13 02:22:06 UTC 2012

November 12

  • 22:46 Tim: deleted refreshLinks jobs corresponding to my recent edit to enwiki's Template:Citation/core
  • 21:47 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
  • 21:45 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
  • 21:43 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
  • 21:32 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 21:17 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 21:15 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 21:11 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 19:22 RoanKattouw: Updated Parsoid on wtp1
  • 19:15 logmsgbot_: reedy synchronized wmf-config/
  • 19:13 logmsgbot_: reedy synchronized php-1.21wmf4/extensions/EducationProgram
  • 19:02 logmsgbot_: reedy synchronized php-1.21wmf4/extensions/Wikibase
  • 18:53 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki, testwiki and wikidatawiki to 1.21wmf4
  • 18:53 paravoid: upgrading tmh1/tmh2 and srv{190,219,220,221,222,223,224} for libav vulnerability (USN-1630-1)
  • 18:52 logmsgbot_: reedy synchronized php-1.21wmf4/cache/l10n/
  • 18:36 logmsgbot_: reedy gracefulled all apaches
  • 18:33 logmsgbot_: reedy gracefulled all apaches
  • 18:29 logmsgbot_: reedy synchronized live-1.5/
  • 18:19 sbernardin: changing ssd's on labsdb3
  • 18:05 logmsgbot_: reedy synchronized live-1.5/
  • 18:04 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/EducationProgram/
  • 17:59 logmsgbot_: reedy synchronized live-1.5/
  • 17:34 logmsgbot_: reedy Started syncing Wikimedia installation... : Build 1.21wmf4 messages
  • 17:14 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.21wmf4
  • 17:07 logmsgbot_: reedy synchronized live-1.5/
  • 17:05 logmsgbot_: reedy synchronized wmf-config/ExtensionMessages-1.21wmf4.php
  • 17:03 logmsgbot_: reedy synchronized php-1.21wmf4/cache
  • 17:02 logmsgbot_: reedy synchronized php-1.21wmf4/ 'Initial sync'
  • 15:28 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/EducationProgram/
  • 15:27 logmsgbot_: reedy synchronized php-1.21wmf3/includes/api/ApiEditPage.php
  • 14:59 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/CentralNotice/
  • 14:14 logmsgbot_: py synchronized wmf-config/db.php 're-adding es4'
  • 11:43 paravoid: apt: removing swift 1.5 from precise-wikimedia
  • 02:32 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Mon Nov 12 02:32:41 UTC 2012

November 11

  • 21:51 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 21:44 paravoid: deploying squid config, adding wikivoyage to the swift regexp
  • 21:26 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'rv'
  • 21:22 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Disable mysql search for wikidata wiki...'
  • 20:01 Reedy: Running sync-common on tmh1
  • 19:58 mutante: powercycling tmh1
  • 19:51 logmsgbot_: reedy synchronized php-1.21wmf3/maintenance/refreshLinks.php
  • 19:49 logmsgbot_: reedy synchronized wmf-config/
  • 19:49 logmsgbot_: demon synchronized wmf-config/InitialiseSettings.php
  • 19:47 logmsgbot_: reedy synchronized php-1.21wmf3/maintenance/refreshLinks.php
  • 19:43 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
  • 17:55 logmsgbot_: dzahn synchronized ./wmf-config/InitialiseSettings.php
  • 15:12 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler
  • 14:48 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/CentralAuth
  • 14:34 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/WikimediaIncubator/
  • 14:17 logmsgbot_: reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
  • 14:16 logmsgbot_: reedy synchronized search-redirect.php
  • 02:32 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Sun Nov 11 02:32:27 UTC 2012
  • 01:21 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 01:08 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 00:05 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 00:02 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php

November 10

  • 23:25 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/CreditsSource/
  • 22:44 mutante: DNS - raise wikivoyage.org TTL back to regular 3600 (1h)
  • 20:51 binasher: indexing {de,fr,it,nl,ru,sv}wikivoyage
  • 20:36 binasher: indexing wikidatawiki
  • 20:12 binasher: attempting to build a lucene index for enwikivoyage
  • 20:10 logmsgbot_: reedy synchronized php-1.21wmf3/cache/interwiki.cdb
  • 20:10 logmsgbot_: reedy synchronized php-1.21wmf3/cache/interwiki.cdb
  • 20:06 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 18:57 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler/
  • 18:55 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Fix wikivoyage sul image location'
  • 18:16 logmsgbot_: reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
  • 18:02 logmsgbot_: reedy synchronized php-1.21wmf3/cache/interwiki.cdb 'Updating interwiki cache'
  • 17:52 logmsgbot_: dzahn gracefulled all apaches
  • 17:40 logmsgbot_: reedy synchronized extract2.php
  • 17:36 logmsgbot_: dzahn gracefulled all apaches
  • 17:27 logmsgbot_: reedy synchronized images/sul/
  • 17:16 logmsgbot_: dzahn gracefulled all apaches
  • 17:00 logmsgbot_: reedy synchronized extract2.php
  • 16:41 logmsgbot_: dzahn gracefulled all apaches
  • 16:39 mutante: sync-apache and graceful to redirect wikivoyage.de
  • 16:38 logmsgbot_: reedy synchronized wmf-config/
  • 16:37 logmsgbot_: reedy synchronized wmf-config/
  • 16:36 logmsgbot_: reedy synchronized images/sul/wikivoyage.png
  • 16:29 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 16:28 mutante: authdns-update for wikivoyage.org - project launch! wee
  • 15:57 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 15:44 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Disable MF for wikivoyage'
  • 15:35 logmsgbot_: reedy synchronized php-1.21wmf3/includes/Article.php
  • 14:05 logmsgbot_: reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
  • 13:31 Reedy: Deleting 1.21wmf2 l10n cache
  • 13:27 Reedy: Pointing php symlink to 1.21wmf3
  • 13:23 logmsgbot_: reedy synchronized php-1.21wmf3/includes/Revision.php
  • 12:59 logmsgbot_: reedy synchronized php-1.21wmf3/includes/resourceloader/ResourceLoaderWikiModule.php
  • 12:58 logmsgbot_: reedy synchronized php-1.21wmf3/includes/
  • 12:55 logmsgbot_: reedy synchronized php-1.21wmf3/cache/interwiki.cdb
  • 12:41 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
  • 11:19 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 02:30 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Sat Nov 10 02:30:06 UTC 2012
  • 00:55 mutante: authdns update to set MX for wikivoyage-old to us

November 9

  • 23:17 binasher: added some redis metrics to ganglia for mc1-16
  • 22:41 cmjohnson1: provisioning labsdb1
  • 22:11 logmsgbot_: dzahn synchronized ./wmf-config/InitialiseSettings.php
  • 21:20 logmsgbot_: reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
  • 20:43 notpeter: repooling mw60 in bits apaches
  • 20:26 sbernardin: installing ssd's in labsdb1 and labsdb2
  • 19:53 mutante: git reset in /h/w/common/ to revert local disabling of "voy" wikis and synced dblist again
  • 19:40 mutante: doing a git pull in wmf-config
  • 19:17 mutante: running sync-dblist
  • 19:15 notpeter: rebooting mw60 for upgarde to precise
  • 19:05 mutante: enabling wikivoyage subdomains in apache for QA check
  • 19:05 logmsgbot_: dzahn gracefulled all apaches
  • 18:54 cmjohnson1: authdns update "Adding .pmtpa.wmnet entries for labsdb1-3"
  • 18:45 mutante: wikivoyage imports: "de" (and all for now) done
  • 18:38 mutante: wikivoyage imports: "en" done, "it" done
  • 16:54 sbernardin: shutting down ms-be7 to row C sdtpa
  • 16:53 sbernardin: racking new ms-be7 to row C sdtpa
  • 16:51 cmjohnson1: ms-be7 going offline to make room for new 720 servers
  • 14:36 cmjohnson1: authodns committing mgmt ip for labsdb3
  • 12:03 apergos: ms-be6 deployed and back in swift rings on shiny new 720xd
  • 05:00 logmsgbot_: tstarling synchronized wmf-config/InitialiseSettings.php 'enabling patrol on wikidatawiki'
  • 02:47 logmsgbot_: olivneh synchronized php-1.21wmf3/extensions/E3Experiments/Experiments.hooks.php 'Fixing OpenTask event log bug'
  • 02:23 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Fri Nov 9 02:23:01 UTC 2012
  • 01:58 binasher: converted all s3 searchindex tables to innodb
  • 01:58 notpeter: starting innobackupex from es3 to es4
  • 01:49 logmsgbot_: olivneh synchronized php-1.21wmf3/extensions/E3Experiments/experiments/openTask.js
  • 01:36 logmsgbot_: py synchronized wmf-config/db.php 'pulling es4'
  • 01:18 mutante: installing package upgrades on kaulen, half-configured apache2 brought down bugzilla ...
  • 00:56 Tim: on fenari: killed stuck git processes run by ExtensionDistributor and LocalisationUpdate
  • 00:52 logmsgbot_: LocalisationUpdate failed: git pull of extensions failed
  • 00:31 logmsgbot_: pgehres synchronized wmf-config
  • 00:25 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/SiteMatrix/

November 8

  • 23:56 logmsgbot_: pgehres synchronized wmf-config
  • 23:51 K4-713: updated payments cluster to 7d92a1a35f0
  • 23:51 logmsgbot_: pgehres synchronized php-1.21wmf3/extensions/CentralNotice/ 'Updating CentralNotice to master'
  • 23:29 logmsgbot_: olivneh synchronized php-1.21wmf3/extensions/E3Experiments
  • 21:18 cmjohnson1: ms-be6 initial puppet run
  • 21:14 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'multiwrite pcache to new memcached servers, only use redis for sessions'
  • 21:08 logmsgbot_: olivneh synchronized php-1.21wmf3/extensions/EventLogging
  • 21:01 logmsgbot_: olivneh synchronized php-1.21wmf3/extensions/E3Experiments
  • 21:01 mutante: restarted nginx on all SSL proxies without issues now. puppet runs again and does not recreate configs for data/voyage on esams.
  • 20:55 mutante: removing wikidata/voyage sites config from esams ssl hosts
  • 20:54 cmjohnson1: ms-be6 another OS install
  • 20:51 mutante: fixed wikipedia ssl chained cert on ssl1 (all other hosts had it recreated as expected)
  • 20:50 apergos: ignore the fail event on ms-be6, this was caused manually and is a bogus warning
  • 20:31 logmsgbot_: demon synchronized wmf-config/CommonSettings.php 'Enabling lucene for wikidatawiki'
  • 19:29 logmsgbot_: dzahn synchronized ./wmf-config/throttle.php
  • 18:16 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler
  • 18:15 mutante: removing wm-nl@wikimedia.org as list admin of "press-nl" because it is an OTRS queue (per thehelpfulone)
  • 15:55 paravoid: starting ms-be3 {object,container,account}-server
  • 14:42 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Thu Nov 8 14:42:45 UTC 2012
  • 14:13 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Thu Nov 8 14:13:54 UTC 2012
  • 12:23 mark: Reinstalling sq70 with Precise
  • 10:57 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 10:57 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/ProofreadPage/
  • 09:49 mark: Killed show_bug.cgi processes on kaulen
  • 09:21 mark: Power cycled cp1041
  • 07:20 apergos: restarted lucene search on search1015
  • 06:26 apergos: restarted lucene search on searchpool1016
  • 03:03 logmsgbot_: aaron synchronized php-1.21wmf3/includes/objectcache/MemcachedPeclBagOStuff.php 'deployed efda0b27befc28ec13dae23f48e8baea42f4a7f3'
  • 02:00 logmsgbot_: LocalisationUpdate failed: git pull of extensions failed
  • 00:52 logmsgbot_: reedy synchronized wmf-config/
  • 00:42 notpeter: repooling palladium
  • 00:40 mutante: wikivoyage imports: "fr" - done
  • 00:39 logmsgbot_: aaron synchronized php-1.21wmf3/extensions/SwiftCloudFiles/php-cloudfiles-wmf/cloudfiles.php 'removed debug logging'
  • 00:37 logmsgbot_: reedy synchronized all.dblist
  • 00:35 logmsgbot_: aaron synchronized php-1.21wmf3/extensions/SwiftCloudFiles/php-cloudfiles-wmf/cloudfiles.php
  • 00:34 paravoid: temporarily stopping {account,container,object}-server on ms-be3 while it syncs up from its downtime
  • 00:34 paravoid: starting memcached on virt0
  • 00:30 logmsgbot_: aaron synchronized php-1.21wmf3/extensions/SwiftCloudFiles/php-cloudfiles-wmf/cloudfiles.php
  • 00:26 logmsgbot_: dzahn gracefulled all apaches
  • 00:26 paravoid: powercycling ms-be3

November 7

  • 23:48 notpeter: removing palladium from eqiad bits cache pool for upgrade to precise
  • 23:43 notpeter: repooling niobium
  • 23:02 notpeter: removing niobium from eqiad bits cache pool for upgrade to precise
  • 22:59 logmsgbot_: asher synchronized wmf-config/db.php 'returning db64 to s3'
  • 22:51 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'move redis to the front of the session multiwrite array'
  • 22:41 notpeter: re-pooling strontium
  • 22:28 logmsgbot_: asher synchronized wmf-config/db.php 'pulling db64'
  • 22:08 mutante: powercycling db64 (frozen)
  • 22:02 logmsgbot_: aaron synchronized wmf-config/filebackend.php
  • 21:57 notpeter: removing strontium from eqiad bits cache pool for upgrade to precise
  • 21:30 logmsgbot_: asher synchronized wmf-config/mc.php 'disable persistent connections for memcached-pecl'
  • 21:26 logmsgbot_: asher synchronized wmf-config/mc.php 'persistent connections for memcached-pecl'
  • 21:10 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'enabling session multiwrite to redis on mc1-16'
  • 20:59 mutante: wikivoyage imports: "ru" - done
  • 20:54 binasher: upgraded redis on mc1-16
  • 20:50 logmsgbot_: aaron synchronized php-1.21wmf3/includes/filerepo/file/LocalFile.php 'debug logging.'
  • 20:45 mutante: wikivoyage imports: "sv" - done
  • 20:42 binasher: added new redis-2.6.3 pkg to precise-wikimedia and upgraded mc1
  • 20:34 mutante: wikivoyage imports: "nl" - done
  • 20:31 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler
  • 20:09 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'disabling session multiwrite to redis on mc1'
  • 19:56 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 19:53 paravoid: powercycling zinc, unresponsive to network and serial console
  • 19:51 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.21wmf3
  • 19:46 mutante: point wlm.wikimedia.org to the cluster rather than stand-alone host yttrium
  • 19:25 notpeter: re-adding arsenic to bits cache pool
  • 19:00 logmsgbot_: dzahn gracefulled all apaches
  • 18:42 sbernardin: racking new ms-be6 to row B sdtpa
  • 17:43 cmjohnson1: racking ms-be6 (720xd)
  • 17:36 Reedy: Creating transcode_key_idx on all transcode tables on all wikis
  • 17:08 logmsgbot_: demon synchronized wmf-config/InitialiseSettings.php 'Enabling TMH for commonswiki'
  • 17:08 logmsgbot_: demon synchronized wmf-config/CommonSettings.php 'Enabling TMH for commonswiki'
  • 15:23 paravoid: reactivating tele2 ipv6 peering and re-exporting routes to it
  • 14:46 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler/
  • 13:26 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler/
  • 13:11 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler/
  • 12:52 mark: Starting load testing of cp3003 and cp3004
  • 11:57 mark: Increased weight of cp3004 to 9
  • 11:16 mark: Pooled cp3004 with weight 1
  • 07:14 logmsgbot_: dzahn gracefulled all apaches
  • 02:49 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Wed Nov 7 02:49:00 UTC 2012
  • 02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Wed Nov 7 02:28:51 UTC 2012
  • 01:48 logmsgbot_: reedy synchronized all.dblist 'frwikivoyage'
  • 01:46 logmsgbot_: dzahn gracefulled all apaches
  • 01:13 logmsgbot_: demon synchronized php-1.21wmf3/extensions/ExtensionDistributor/ExtensionDistributor_body.php 'Revert debugging'
  • 01:05 logmsgbot_: demon synchronized php-1.21wmf3/extensions/ExtensionDistributor/ExtensionDistributor_body.php 'Debugging'
  • 00:56 logmsgbot_: demon synchronized php-1.21wmf3/extensions/ExtensionDistributor/ExtensionDistributor_body.php 'Debugging'

November 6

  • 23:57 mutante: added gpg key for hexmode to keys.html/keys.txt
  • 23:51 logmsgbot_: dzahn gracefulled all apaches
  • 23:49 logmsgbot_: reedy synchronized all.dblist 'Kill wikivoyage from all.dblist'
  • 23:44 logmsgbot_: reedy synchronized closed.dblist
  • 23:42 logmsgbot_: reedy synchronized closed.dblist
  • 23:41 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 23:31 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 23:30 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 23:30 logmsgbot_: spage synchronized php-1.21wmf3/extensions/E3Experiments 'ACUX interactive validation'
  • 23:16 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'setting s3 to writeable again'
  • 22:49 mutante: creating http://dumps.wikimedia.org/mediawiki/1.20/ per hexmode
  • 22:36 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'setting s3 to read-only for emergency db maintenance'
  • 22:14 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler/
  • 22:01 mutante: adding wikivoyage.org/.de alias file on mchenry
  • 21:00 mutante: imported frwikivoyage db dumps
  • 20:53 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 20:03 logmsgbot_: dzahn gracefulled all apaches
  • 19:57 logmsgbot_: dzahn gracefulled all apaches
  • 19:56 Jeff_Green: restarted udp2log on locke
  • 19:24 AaronSchulz: finished cleanupBug41615.php script runs for affected wikis (s2,s3,s4,s7)
  • 18:57 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/WikimediaMaintenance/
  • 18:46 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 18:44 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 18:39 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 18:39 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 18:35 logmsgbot_: reedy synchronized wmf-config/
  • 18:30 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 18:27 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 18:22 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 18:18 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/WikimediaMaintenance/
  • 17:03 logmsgbot_: reedy synchronized multiversion/ 'Resync multiversion after changing to a submodule'
  • 17:01 logmsgbot_: reedy synchronized multiversion/ 'Resync multiversion after changing to a submodule'
  • 16:59 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler/
  • 16:06 logmsgbot_: demon synchronized php-1.21wmf3/extensions/Wikibase/repo/Wikibase.php 'Syncing out 856bad9d'
  • 16:05 ^demon: cleaned up fenari:/h/w/c/php-1.21wmf3 history so it can properly ff on pull
  • 11:37 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler/
  • 11:33 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
  • 10:33 logmsgbot_: nikerabbit synchronized php-1.21wmf3/extensions/WebFonts/resources/ext.webfonts.fontlist.js
  • 10:22 logmsgbot_: nikerabbit synchronized php-1.21wmf3/extensions/Translate/utils/MessageGroupStats.php
  • 09:12 logmsgbot_: hashar synchronized docroot/mediawiki/xml 'import updated export-0.8.xsd'
  • 02:55 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Tue Nov 6 02:55:06 UTC 2012
  • 02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Tue Nov 6 02:28:55 UTC 2012
  • 01:57 logmsgbot_: reedy synchronized wmf-config/
  • 01:14 binasher: slamming the apache cluster by wiping out all mobile caching yet again
  • 00:59 logmsgbot_: awjrichards synchronized php-1.21wmf2/extensions/MobileFrontend/
  • 00:59 logmsgbot_: awjrichards synchronized php-1.21wmf3/extensions/MobileFrontend/javascripts/ 'touch files'
  • 00:49 logmsgbot_: awjrichards synchronized php-1.21wmf3/extensions/MobileFrontend/javascripts/modules/mf-references.js 'touch file'
  • 00:44 logmsgbot_: awjrichards synchronized php-1.21wmf3/extensions/MobileFrontend
  • 00:40 logmsgbot_: reedy synchronized php-1.21wmf3/cache/l10n/ 'Update l10n cache'
  • 00:18 Tim: on gallium: moving data directories to a tmpfs
  • 00:13 K4-713: updated the payments cluster to bd673e77d6adde3
  • 00:04 logmsgbot_: reedy synchronized wmf-config/

November 5

  • 23:52 logmsgbot_: reedy synchronized multiversion
  • 23:43 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/UserMerge
  • 23:36 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwikivoyage to 1.21wmf3
  • 23:35 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 23:34 logmsgbot_: reedy synchronized wmf-config/
  • 23:27 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 22:23 binasher: rebooting srv191,225,230 for kernel upgrades
  • 22:10 binasher: rebooting mw21,25,40 for kernel upgrades
  • 21:54 logmsgbot_: aaron synchronized wmf-config/mc.php 'Switched all wikis to memcached-multiwrite.'
  • 21:54 logmsgbot_: aaron synchronized wmf-config/InitialiseSettings.php 'Disabled spammy (with pecl memcached) memcached.log, the useful one is memcached-serious.'
  • 21:49 logmsgbot_: asher gracefulled all apaches
  • 21:36 binasher: reboot mw49 for kernel upgrade
  • 21:19 binasher: rebooted mw28
  • 21:19 mutante: copied interwiki.cdb from prod to wikitech, used $wgInterwikiCache to point to .cdb, mw does not use mysql table anymore. long iw links work again, "shortcuts" are different though and still an issue
  • 21:09 logmsgbot_: aaron synchronized wmf-config/mc.php 'Added (zh|de)wiki to memc multiwrite and switched read order.'
  • 20:42 notpeter: re-adding sq67 to bits cache pool w/o upgarde and removing arsenic from eqiad bits cache pool for actual upgrade to precise
  • 20:40 notpeter: removing sq67 from bits cache pool for upgrade to precise
  • 19:48 Jeff_Green: payments flipped to eqiad cluster
  • 19:02 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf3
  • 18:22 logmsgbot_: aaron synchronized wmf-config/InitialiseSettings.php 'TMH deploy too all except commons.'
  • 17:58 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler/
  • 17:57 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/TimedMediaHandler/
  • 17:56 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/MwEmbedSupport/
  • 17:35 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler/
  • 17:33 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/MwEmbedSupport
  • 17:21 logmsgbot_: py gracefulled all apaches
  • 17:20 notpeter: gracefulling all apaches to pick up https://gerrit.wikimedia.org/r/#/c/31670/
  • 17:19 mutante: DNS update - push wikivoyage.de (link)
  • 16:57 mark: Reinstalling sq69
  • 16:17 mark: Reinstalling sq68 with Precise
  • 12:38 mark: Performance testing cp3003
  • 11:38 apergos: synced apache/common-local by hand to tmh1 and tmh2 and restart job runners on both hosts (the directory was previously empty)
  • 09:49 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'wikidatawiki to use noticeproject::wikimedia banner'
  • 09:36 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php '(bug 41717) Update default (language-neutral) Wiktionary logo'
  • 09:26 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php '(bug 41774) New wgImportSource for ro.wikibooks'
  • 09:17 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php '(bug 38134) Enable Extension:GoogleNewsSitemap on es wikinews'
  • 09:15 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php '(bug 40212) Mass update Wiktionary favicons'
  • 09:02 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'Configure Babel category for Wikidata 31608'
  • 09:00 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'whitespace fix 31205'
  • 08:57 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php '(bug 40212) Mass update Wiktionary favicons'
  • 08:53 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php '(bug 32411) Transwiki import to multilingual wikisource broken'
  • 08:40 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php '(bug 41167) Namespace configuration for ba.wikipedia'
  • 08:25 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php '(bug 40717) Namespace configuration for th.wiktionary and th.wikibooks'
  • 08:24 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php '(bug 40717) Namespace configuration for th.wiktionary and th.wikibooks'
  • 02:51 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Mon Nov 5 02:51:13 UTC 2012
  • 02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Mon Nov 5 02:28:01 UTC 2012

November 4

  • 05:31 logmsgbot_: catrope synchronized php-1.21wmf3/extensions/TimedMediaHandler/
  • 05:30 logmsgbot_: catrope synchronized php-1.21wmf2/extensions/TimedMediaHandler/
  • 02:53 K4-713: updated the payments cluster to 3a315eafc09015e
  • 02:49 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Sun Nov 4 02:49:49 UTC 2012
  • 02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Sun Nov 4 02:28:44 UTC 2012

November 3

  • 17:46 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 17:34 logmsgbot_: reedy synchronized php-1.21wmf3/includes/
  • 06:50 apergos: (from half an hour ago) restarted lucene search on search1015. stupid shift key
  • 02:50 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Sat Nov 3 02:50:07 UTC 2012
  • 02:29 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Sat Nov 3 02:29:23 UTC 2012

November 2

  • 23:45 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/ 'Dark deploy wikivoyage extensions'
  • 22:34 AaronSchulz: Created TMH table on all wikis
  • 21:55 logmsgbot_: dzahn gracefulled all apaches
  • 21:55 mutante: adding apache config for wikivoyage
  • 21:24 mutante: authdns-update, adding new zone for wikivoyage-old
  • 21:08 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/ProofreadPage
  • 19:49 cmjohnson1: srv266 troubleshooting cpu
  • 18:39 logmsgbot_: aaron synchronized php-1.21wmf3/thumb.php 'deployed a16d25a721b502f83f6d9354d482f8f2bc08176b'
  • 18:38 logmsgbot_: aaron synchronized php-1.21wmf2/thumb.php 'deployed 0b7378c966ead76422d03fb0c663fa714ad8a48a'
  • 17:07 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
  • 17:00 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
  • 16:23 logmsgbot_: maxsem synchronized wmf-config/InitialiseSettings.php 'Close wikimania2010wiki'
  • 16:19 logmsgbot_: maxsem synchronized closed.dblist 'https://gerrit.wikimedia.org/r/#/c/31379/'
  • 14:39 logmsgbot_: reedy synchronized wmf-config/
  • 13:09 paravoid: temporarily deactivating IPv6 peering with tele2 on cr1-eqiad due to routing issues
  • 10:34 Tim: on wikitech, running compressOld.php
  • 10:21 Tim: converting wikitech database from latin1 to binary
  • 09:57 Tim: upgraded wikitech to MW 1.19.2
  • 06:40 apergos: some files on wikitech (old backups etc) moved to tridge in /data/wikitech-nov2012 to give us a couple more gb of room
  • 06:20 Tim: upgrading wikitech to MW 1.17 branch head
  • 06:13 Tim: on wikitech: disabled debug log
  • 05:46 Tim: wikitech.wikimedia.org had a full root partition. I resized it to fill up the current account allocation
  • 02:26 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Fri Nov 2 02:26:03 UTC 2012
  • 01:03 logmsgbot_: pgehres synchronized php-1.21wmf2/extensions/CentralNotice/ 'Updating CentralNotice, fixing Bug 41632'
  • 00:29 logmsgbot_: spage Finished syncing Wikimedia installation... : PostEdit i18n, E3 ACUX updates, fixing Cross site reqest problem in CentralNotice
  • 00:12 logmsgbot_: reedy synchronized wmf-config/

November 1

  • 23:42 logmsgbot_: spage Started syncing Wikimedia installation... : PostEdit i18n, E3 ACUX updates, fixing Cross site reqest problem in CentralNotice
  • 23:24 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Revert that, didn't fix the issue'
  • 23:22 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Temporarily disable translate and translation notification on wikidatawiki'
  • 22:59 Reedy: Dropped contest tables from testwiki
  • 22:56 logmsgbot_: reedy synchronized wmf-config/ 'Disable Contest extension'
  • 22:48 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 22:43 Reedy: Created translate tables on wikidatawiki
  • 22:41 logmsgbot_: spage synchronized php-1.21wmf3/extensions/E3Experiments 'ACUX bump versions'
  • 22:33 logmsgbot_: spage synchronized php-1.21wmf3/extensions/PostEdit 'latest PostEdit'
  • 22:30 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Enable wikidata import from metawiki'
  • 21:27 spagewmf: spagewmf ran `sync-dir php-1.21wmf3/extensions/E3Experiments 'ACUX updates'`
  • 21:25 logmsgbot_: spage synchronized php-1.21wmf3/extensions/E3Experiments 'ACUX updates'
  • 21:17 mutante: freeing disk space on wikitech
  • 19:37 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/ProofreadPage
  • 19:01 LeslieCarr: (modifying above ^^) pulled IPv6 (only) announcements from AS1257 on cr1-eqiad ONLY due to routing issues
  • 19:00 LeslieCarr: pulled IPv6 (only) announcements from AS1257 due to routing issues
  • 18:39 mutante: deleting duplicate docroot for wikidata in /h/w/common/docroot/
  • 18:14 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Test
  • 18:02 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/TimedMediaHandler
  • 17:49 logmsgbot_: reedy synchronized php-1.21wmf2/includes/
  • 17:48 logmsgbot_: mlitn synchronized php-1.21wmf3/extensions/ArticleFeedbackv5 'desc'
  • 17:47 logmsgbot_: mlitn synchronized php-1.21wmf2/extensions/ArticleFeedbackv5 'desc'
  • 17:41 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Re-enable TMH on enwiki'
  • 17:34 logmsgbot_: reedy synchronized php-1.21wmf3/includes/cache/MessageCache.php
  • 17:34 logmsgbot_: reedy synchronized php-1.21wmf3/includes/Message.php
  • 17:18 logmsgbot_: reedy synchronized php-1.21wmf3/includes/
  • 17:17 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler
  • 17:14 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Enable PageTriage on test2wiki'
  • 17:10 Reedy: Created PageTriage tables on test2wiki
  • 17:08 hashar: gallium is sent to swap from time to time. The cause seems to be the Ext-Wikibase job for which update.php eat all memory. I have disabled the job meanwhile. In case of trouble, simply kill the wild php process.
  • 16:50 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Disable TMH on enwiki'
  • 16:42 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Enable TMH and mwembed on enwiki'
  • 16:36 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/MwEmbedSupport/
  • 16:35 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/MwEmbedSupport/
  • 16:35 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/TimedMediaHandler/ 'Update to master'
  • 16:34 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler/ 'Update to master'
  • 16:32 mark: Restarted backend varnish instance on cp1029
  • 16:14 mark: Reinstalling sq67 with Precise
  • 15:56 logmsgbot_: reedy synchronized php-1.21wmf3/includes/specials/SpecialUndelete.php
  • 15:56 logmsgbot_: reedy synchronized php-1.21wmf3/includes/Revision.php
  • 15:54 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/Wikibase/lib/includes/Utils.php
  • 15:41 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/Wikibase
  • 14:06 mark: Restarted backend varnish instance on cp1027
  • 13:12 mark: Repooled cp3003
  • 12:35 mark: Depooled cp3003, high rate of 500 responses
  • 12:29 mark: Pooled cp3003 as upload Varnish cluster in PyBal with weight 1
  • 10:55 mark: Restarted backend varnish instance on cp1026
  • 08:46 mark: Restarted backend varnish instance on cp1025
  • 08:40 apergos: running some nonpriority jobs manually on mw12 (so people later don't get weirded out bu the ganglia graphs)
  • 08:21 Nemo_bis: Job queue runners almost idle again since 5 UTC; en.wiki queue at 145k, zh.wiki 260k, Commons 60k now
  • 07:40 mark: Killed and restarted lucene on search1016
  • 04:47 logmsgbot_: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: reverted wmf3 deployment made earlier today.
  • 03:59 logmsgbot_: tstarling synchronized php-1.21wmf3/includes/job/JobQueueDB.php 'fix deadlocks'
  • 02:54 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Thu Nov 1 02:54:45 UTC 2012
  • 02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Thu Nov 1 02:28:03 UTC 2012
  • 00:12 mutante: installing package upgrades on marmontel (blog)

October 31

  • 23:39 logmsgbot_: reedy synchronized wmf-config/
  • 23:23 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/Wikibase
  • 23:10 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
  • 22:37 logmsgbot_: reedy synchronized php-1.21wmf3/maintenance/nextJobDB.php
  • 21:51 logmsgbot_: reedy synchronized php-1.21wmf3/maintenance/nextJobDB.php
  • 21:35 K4-713: payments synchronized to 5d40dbe27d05
  • 21:12 logmsgbot_: reedy synchronized php-1.21wmf3/includes/site/SiteObject.php
  • 20:42 logmsgbot_: aaron synchronized php-1.21wmf2/extensions/TimedMediaHandler 'deployed cbe526b06ec1876c933f8a04c2526b137642a879'
  • 20:41 logmsgbot_: aaron synchronized php-1.21wmf3/extensions/TimedMediaHandler 'deployed bf098d262dce290ac4b0cf70f74182c98aeca4d9'
  • 20:09 Reedy: Restarting Jenkins
  • 20:09 logmsgbot_: aaron synchronized wmf-config/CommonSettings.php 'Bug 41528 - need more memory for video thumbnails'
  • 19:35 logmsgbot_: preilly Finished syncing Wikimedia installation... : update zero rated mobile access
  • 19:34 mark: Restarted backend varnish instance on cp1024
  • 19:33 K4-713: updated payments cluster to 5d8309e21
  • 18:54 logmsgbot_: preilly Started syncing Wikimedia installation... : update zero rated mobile access
  • 18:41 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else non wikipedia or wikivoyage to 1.21wmf3
  • 18:40 logmsgbot_: preilly synchronized php-1.21wmf3/extensions/ZeroRatedMobileAccess 'update post deploy'
  • 18:39 logmsgbot_: preilly synchronized php-1.21wmf2/extensions/ZeroRatedMobileAccess 'update post deploy'
  • 18:38 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary and wikinews to 1.21wmf3
  • 18:31 cmjohnson1: rebuilding software raid for storage3 and provisioning
  • 18:30 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/ReaderFeedback
  • 18:29 mutante: srv286 - installing php5-memcached and dist-upgrade
  • 18:27 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity, wikiquote and wikibooks to 1.21wmf3
  • 18:25 mutante: srv286 - kill frozen apt-get update process, update sources, install package upgrades
  • 18:18 mutante: mw18 - dpkg --configure -a to fix interrupted dpkg, configures php-luasandbox, install php5-memcached
  • 18:17 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/ReaderFeedback
  • 18:07 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special, closed and private to 1.21wmf3
  • 18:01 logmsgbot_: reedy synchronized php-1.21wmf3/
  • 18:00 logmsgbot_: reedy synchronized php-1.21wmf2/
  • 17:56 mutante: now redirecting wikimania.wm to wikimania2013.wm
  • 17:54 logmsgbot_: dzahn gracefulled all apaches
  • 17:52 logmsgbot_: preilly synchronized php-1.21wmf3/extensions/MobileFrontend 'update after deploy'
  • 17:51 logmsgbot_: preilly synchronized php-1.21wmf2/extensions/MobileFrontend 'update after deploy'
  • 17:41 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 16:57 Reedy: Truncating wikibase.wb_changes
  • 16:25 mark: Restarted backend varnish instance on cp1023
  • 15:50 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'wgULSIMEEnabled = false'
  • 15:49 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/UniversalLanguageSelector/ 'Update to master'
  • 15:00 mark: Restarted backend varnish instance on cp1022
  • 14:45 paravoid: putting ms-fe4 back in the pool
  • 14:26 paravoid: temporarily depooling ms-fe4 to apply a temporary local hack
  • 14:12 paravoid: putting ms-fe4 back in the pool
  • 13:27 mark: Restarted cp1021 with persistent storage
  • 13:26 paravoid: depooling ms-fe4 to upgrade to precise/swift 1.7.4
  • 11:35 paravoid: putting ms-fe3 back in the pool
  • 11:28 ^demon: restarted gerrit services on manganese & formey to pick up replication changes
  • 07:59 paravoid: depooling ms-fe3 to upgrade to precise/swift 1.7.4
  • 07:09 apergos: restarted lucene search on search1015 and search 1003
  • 02:47 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Wed Oct 31 02:47:40 UTC 2012
  • 02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Wed Oct 31 02:28:50 UTC 2012
  • 02:05 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
  • 02:01 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Re-enable TMH'
  • 01:43 logmsgbot_: reedy synchronized wmf-config/
  • 01:23 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Disable TMH everywhere (and mwembed at the same time)'
  • 01:15 K4-713: synchronized the payments cluster to 85e9007aef3e75
  • 01:06 logmsgbot_: awjrichards synchronized php-1.21wmf3/extensions/MobileFrontend/javascripts/actions/mf-edit.js 'touch file'
  • 00:58 logmsgbot_: reedy synchronized php-1.21wmf3/includes/job/JobQueueDB.php
  • 00:57 logmsgbot_: awjrichards synchronized php-1.21wmf2/extensions/MobileFrontend/javascripts/actions/mf-edit.js 'touch file'
  • 00:42 logmsgbot_: kaldari synchronized php-1.21wmf2/extensions/CentralNotice 'Update CentralNotice on meta and en.wiki to fix banner override bug'
  • 00:30 logmsgbot_: maxsem Finished syncing Wikimedia installation... :

October 30

  • 23:44 logmsgbot_: maxsem Started syncing Wikimedia installation... :
  • 23:08 mutante: replaced wikipedia SSL cert with new one by GeoTrust
  • 22:16 logmsgbot_: kaldari synchronized php-1.21wmf3/extensions/CentralNotice 'autoloading ApiCentralNoticeAllocations for CentralNotice'
  • 22:15 LeslieCarr: failing overRE mastership on cr2-eqiad to upgrade re1
  • 22:02 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'memcached logging group'
  • 22:00 logmsgbot_: asher synchronized wmf-config/mc.php 'testing MemcachedPeclBagOStuff on test2wiki (serverlist fix)'
  • 21:57 logmsgbot_: asher synchronized wmf-config/mc.php 'testing MemcachedPeclBagOStuff on test2wiki'
  • 21:47 mutante: restarting nginx on ssl hosts for cert update
  • 21:43 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 21:30 logmsgbot_: kaldari synchronized php-1.21wmf3/extensions/CentralNotice 'updating centralNotice to autoload ApiCentralNoticeAllocations class'
  • 21:24 logmsgbot_: kaldari synchronized php-1.21wmf2/extensions/CentralNotice 'updating centralNotice to autoload ApiCentralNoticeAllocations class'
  • 21:23 logmsgbot_: kaldari Finished syncing Wikimedia installation... :
  • 21:14 logmsgbot_: kaldari synchronized php-1.21wmf2/extensions/CentralNotice 'updating centralNotice to autoload ApiCentralNoticeAllocations class'
  • 21:09 logmsgbot_: kaldari synchronized php-1.21wmf2/extensions/CentralNotice/CentralNotice.php 'updating centralNotice to autoload ApiCentralNoticeAllocations class'
  • 20:54 Jeff_Green: stopping mysql on db1013 for cloning
  • 20:37 logmsgbot_: kaldari Started syncing Wikimedia installation... :
  • 20:21 logmsgbot_: kaldari synchronized wmf-config/CommonSettings.php 'updating wgCentralBannerDispatcher for CentralNotice'
  • 19:46 logmsgbot_: kaldari synchronized wmf-config/CommonSettings.php 'updating settings for CentralNotice banner loading'
  • 18:31 binasher: completed centralnotice patch-bucketing.sql migration
  • 18:24 mutante: giving sbernardin racktables permissions
  • 17:53 mark: Installed Precise on new upload esams servers cp3003-cp3010
  • 17:20 mutante: running sync-apache on srv199 only
  • 17:19 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'wgMaxAnimatedGifArea = 2.5e7'
  • 17:15 logmsgbot_: reedy synchronized wmf-config/omgtestfile
  • 17:06 Reedy: Running sync-common on srv199
  • 17:01 Reedy: Running sync-common on srv194
  • 16:54 logmsgbot_: reedy synchronized wmf-config/ExtensionMessages-1.21wmf3.php
  • 16:38 apergos: stpped stray jobrunner on srv278
  • 14:38 paravoid: powercycling srv229, dead for unknown reason so far
  • 10:08 logmsgbot_: nikerabbit synchronized php-1.21wmf2/extensions/NewUserMessage/NewUserMessage.class.php 'I0f93ee53'
  • 09:41 logmsgbot_: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Temporarily enable beta mappings for am wikis'
  • 09:19 logmsgbot_: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Narayam and Webfonts: bug 41460, bug 39200, bug 41359'
  • 09:07 mark: Fixed wikidata.org language cnames issue
  • 07:44 mark: Fixed IPv6 addresses on new esams bits servers
  • 07:33 mark: Replaced 2 bits @ esams servers with 4 new servers cp3019-cp3022
  • 02:49 logmsgbot_: LocalisationUpdate completed (1.21wmf3) at Tue Oct 30 02:49:48 UTC 2012
  • 02:24 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Tue Oct 30 02:24:04 UTC 2012
  • 01:09 logmsgbot_: olivneh synchronized wmf-config/InitialiseSettings.php
  • 01:09 logmsgbot_: olivneh synchronized php-1.21wmf2/extensions/EventLogging
  • 01:02 logmsgbot_: olivneh synchronized php-1.21wmf3/extensions/EventLogging 'Updating EventLogging on test2'
  • 00:43 logmsgbot_: olivneh synchronized php-1.21wmf2/extensions/EventLogging
  • 00:29 logmsgbot_: aaron synchronized php-1.21wmf2/extensions/TimedMediaHandler 'deployed 18e51f3b06b84d1d5fbf47272d9ebfc5008dc879'
  • 00:28 logmsgbot_: aaron synchronized php-1.21wmf3/extensions/TimedMediaHandler 'deployed c1ac05640377f4f99cbe2a094e80d3d25d63b93d'
  • 00:27 logmsgbot_: olivneh synchronized php-1.21wmf3/extensions/EventLogging 'Updating EventLogging on test2'

October 29

  • 21:28 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
  • 21:25 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Revert'
  • 21:24 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Change wikidata search config even more'
  • 21:20 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Change wikidata search config even more'
  • 21:11 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Disable lucene on wikidatawiki for now'
  • 20:44 logmsgbot_: mlitn synchronized php-1.21wmf2/extensions/ArticleFeedbackv5
  • 19:39 logmsgbot_: reedy synchronized live-1.5/
  • 19:36 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.21wmf3
  • 19:19 Reedy: Deleting php-1.20wmf11 files
  • 19:16 logmsgbot_: reedy synchronized php-1.21wmf3/includes/Revision.php
  • 18:25 logmsgbot_: aaron synchronized php-1.21wmf3/extensions/TimedMediaHandler 'deployed 05118462e3ddbfc507772cc3c4e1117a8527e0b5'
  • 18:19 Reedy: Run Wikibase/repo/maintenance/createBlacklistedItems.php on wikidatawiki
  • 18:16 logmsgbot_: catrope synchronized php-1.21wmf3/extensions/VisualEditor/ 'Update VisualEditor'
  • 18:04 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Add wikidata to wgCrossSiteAJAXdomains'
  • 17:54 Reedy: Ran patch-job_token.sql on wikidatawiki
  • 17:52 logmsgbot_: demon synchronized wmf-config/CommonSettings.php 'Turning SUL back on for wikidata.org'
  • 17:51 logmsgbot_: catrope gracefulled all apaches
  • 17:20 Reedy: Running \Wikibase\Utils::insertDefaultSites(); on wikidatawiki
  • 17:08 logmsgbot_: reedy synchronized php-1.21wmf3/cache/interwiki.cdb 'Updating interwiki cache'
  • 17:07 logmsgbot_: reedy synchronized php-1.21wmf2/cache/interwiki.cdb 'Updating interwiki cache'
  • 17:02 Reedy: Applied Wikibase/lib/sql/WikibaseLib.sql on wikidatawiki
  • 16:50 logmsgbot_: reedy synchronized php-1.21wmf3/includes/
  • 16:44 logmsgbot_: reedy synchronized php-1.21wmf3/includes/WikiPage.php
  • 16:43 logmsgbot_: reedy synchronized php-1.21wmf3/includes/Revision.php
  • 16:41 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikidatawiki to 1.21wmf3
  • 16:38 logmsgbot_: reedy synchronized php-1.21wmf3/cache/l10n/ 'Sync'
  • 16:30 logmsgbot_: mark synchronized wmf-config/CommonSettings.php 'Add new bits server IPs'
  • 16:12 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuilding localisation cache for 1.21wmf3
  • 16:08 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuilding localisation cache for 1.21wmf3
  • 16:01 logmsgbot_: reedy Finished syncing Wikimedia installation... : Rebuilding localisation cache for 1.21wmf3
  • 15:54 logmsgbot_: reedy synchronized wmf-config/
  • 15:52 Reedy: Running sync-common on searchidx1001
  • 15:51 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/PostEdit/
  • 15:50 logmsgbot_: reedy synchronized php-1.21wmf3/extensions/EventLogging/
  • 15:46 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuilding localisation cache for 1.21wmf3
  • 15:41 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.21wmf3
  • 15:40 logmsgbot_: reedy synchronized php-1.21wmf3 'Initial file sync-out'
  • 13:38 ^demon: flushed gerrit's account_byemail cache, error log was complaining
  • 13:19 logmsgbot_: demon synchronized wmf-config/CommonSettings.php 'Disabling wikidata for SUL until we fix SSL'
  • 02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Mon Oct 29 02:28:02 UTC 2012

October 28

  • 22:37 logmsgbot_: olivneh synchronized php-1.21wmf2/extensions/EventLogging 'Updating Extension:EventLogging (enabled only on test & test2)'
  • 22:33 logmsgbot_: olivneh synchronized wmf-config/InitialiseSettings.php 'Enable Extension:EventLogging on test2'
  • 21:24 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'arwiki logo'
  • 02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Sun Oct 28 02:28:08 UTC 2012

October 27

  • 21:19 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Set geoservice to geoiplookup.wm.o'
  • 20:51 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Comment out wgULSGeoService'
  • 10:02 paravoid: repooling sq82 frontend in pybal
  • 08:49 paravoid: reinstalling sq82 with lucid
  • 07:41 apergos: depooled sq82 in pybal, squid3 was running on it again and it's still precise
  • 02:33 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Sat Oct 27 02:33:26 UTC 2012
  • 00:25 mutante: restarting nginx on ssl hosts for changed server name for wikidata
  • 00:17 mutante: reloading nginx on ssl1001

October 26

  • 23:46 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/UniversalLanguageSelector/
  • 23:23 Ryan_Lane: powercycling kaulen
  • 23:09 cmjohnson1: tinkering with storage3
  • 22:24 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Set wgULSGeoService to false'
  • 22:03 Ryan_Lane: repooing ssl1004
  • 21:44 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'wgContentHandlerUseDB back to true. Mainpage fixed!'
  • 21:41 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'wgContentHandlerUseDB to false for wikidatawiki'
  • 21:27 cmjohnson1: imaging db42
  • 20:59 Ryan_Lane: rebooting ssl1004
  • 20:54 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Revert that'
  • 20:52 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Disabling wikidata related extensions temporarily'
  • 20:50 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 20:50 logmsgbot_: reedy synchronized images/sul/wikidata.png
  • 20:43 logmsgbot_: reedy synchronized php-1.21wmf2/includes/content/ContentHandler.php
  • 20:43 logmsgbot_: reedy synchronized php-1.21wmf2/cache/l10n/
  • 20:43 logmsgbot_: reedy synchronized php-1.21wmf2/includes/EditPage.php
  • 20:37 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 20:36 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 20:28 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuild localisation cache for wikidata related extensions
  • 20:25 logmsgbot_: aaron synchronized php-1.21wmf2/extensions/MwEmbedSupport 'deployed de1079e3eec4eae4f129d7bcffc71596f415ae74'
  • 20:15 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Enable wikibaserepo on wikidatawiki'
  • 20:13 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
  • 20:13 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
  • 20:10 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Temporary hacking in of ns' for wikidatawiki'
  • 20:07 Reedy: Adding extensions/Wikibase/repo/includes/store/sql/Wikibase.sql to wikidatawiki
  • 20:01 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 20:00 logmsgbot_: aaron synchronized php-1.21wmf2/extensions/TimedMediaHandler 'deployed 31a1d21e8ebfb90ece5ef02075fc85f197a50e53'
  • 19:59 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'live hacky hack'
  • 19:55 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'live hacky hack'
  • 19:53 cmjohnson1: rebooting sq82
  • 19:52 logmsgbot_: reedy synchronized wmf-config/ 'gotta touch them all'
  • 19:45 logmsgbot_: reedy synchronized wmf-config/
  • 19:18 cmjohnson1: stopping puppet on brewster temporarily
  • 19:17 logmsgbot_: root gracefulled all apaches
  • 19:12 mutante: deleting files older than 3 days from /tmp/ in "mw" dsh group
  • 19:05 mutante: deleting /tmp/mw-cache-1.20* from mw dsh group
  • 18:57 Reedy: Cleaned /tmp on mw48
  • 18:54 Reedy: Cleaned /tmp on mw34
  • 18:51 Reedy: Cleaned /tmp on mw45
  • 18:31 Jeff_Green: aluminium dist upgrade and reboot
  • 18:27 Jeff_Green: erzurumi dist-upgrade and reboot
  • 18:02 logmsgbot_: aaron synchronized php-1.21wmf2/thumb.php 'done'
  • 17:58 logmsgbot_: aaron synchronized php-1.21wmf2/thumb.php 'improve path mismatch error message'
  • 17:50 logmsgbot_: aaron synchronized php-1.21wmf2/includes/upload/UploadStash.php 'done'
  • 17:44 logmsgbot_: aaron synchronized php-1.21wmf2/includes/upload/UploadStash.php
  • 17:42 logmsgbot_: aaron synchronized php-1.21wmf2/includes/upload/UploadStash.php 'debug logging'
  • 17:35 logmsgbot_: dzahn gracefulled all apaches
  • 17:32 mutante: sync-apache new wikidata config
  • 16:49 logmsgbot_: demon synchronized wmf-config/InitialiseSettings.php 'Disabling EventLogging on enwiki'
  • 14:28 logmsgbot_: demon synchronized php-1.21wmf2/extensions/AbuseFilter/modules/ext.abuseFilter.tools.js 'Fixing JS for AbuseFilter'
  • 14:28 logmsgbot_: demon synchronized php-1.21wmf2/extensions/AbuseFilter/modules/ext.abuseFilter.examine.js 'Fixing JS for AbuseFilter'
  • 14:08 logmsgbot_: demon synchronized php-1.21wmf2/extensions/AbuseFilter/modules 'Updating AbuseFilter to f52f940e'
  • 13:41 logmsgbot_: maxsem synchronized wmf-config/mobile.php 'https://gerrit.wikimedia.org/r/#/c/30070/'
  • 13:07 logmsgbot_: demon synchronized wmf-config/InitialiseSettings.php 'Syncing I85721736'
  • 12:46 paravoid: stopping squid3(!) on sq82, box was reprovisioned with precise but we don't support squid3/squid on precise yet
  • 11:20 logmsgbot_: reedy synchronized wmf-config/
  • 11:03 Reedy: Created WikiLove tables on urwiki
  • 05:42 paravoid: powering off ms-fe1001 & ms-fe1002
  • 02:38 logmsgbot_: kaldari synchronized php-1.21wmf2/extensions/CentralNotice/CentralNotice.db.php 'updating CentralNotice.db.php'
  • 02:32 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Fri Oct 26 02:32:33 UTC 2012
  • 01:37 logmsgbot_: kaldari synchronized php-1.21wmf2/extensions/CentralNotice 'updating CentralNotice'
  • 01:21 logmsgbot_: kaldari synchronized php-1.21wmf2/extensions/CentralNotice/CentralNotice.db.php 'updating CentralNotice.db.php'
  • 00:15 logmsgbot_: olivneh synchronized php-1.21wmf2/extensions/ArticleFeedbackv5/ArticleFeedbackv5.hooks.php
  • 00:14 logmsgbot_: pgehres synchronized php-1.21wmf2/extensions/ContributionTracking/ 'Updating ContributionTracking'
  • 00:13 logmsgbot_: pgehres synchronized php-1.21wmf2/extensions/FundraiserLandingPage/ 'Updating FundraiserLandingPage'
  • 00:04 logmsgbot_: kaldari synchronized php-1.21wmf2/extensions/CentralNotice/modules/ext.centralNotice.bannerController/bannerController.js 'fixing bannerController'

October 25

  • 23:55 logmsgbot_: kaldari Finished syncing Wikimedia installation... :
  • 23:49 cmjohnson1: installing base OS on sq82
  • 23:20 logmsgbot_: kaldari Started syncing Wikimedia installation... :
  • 22:58 logmsgbot_: olivneh synchronized php-1.21wmf2/extensions/E3Experiments
  • 22:40 logmsgbot_: aaron synchronized php-1.21wmf2/extensions/OpenSearchXml/ApiOpenSearchXml.php 'use slave DB for revision queries.'
  • 22:36 logmsgbot_: aaron synchronized php-1.21wmf2/extensions/MWSearch/MWSearch_body.php 'use slave DB for revision queries.'
  • 22:30 logmsgbot_: aaron synchronized php-1.21wmf2/includes/Revision.php 'done'
  • 22:30 logmsgbot_: aaron synchronized php-1.21wmf2/includes/Revision.php 'logging one more thing.'
  • 22:24 logmsgbot_: aaron synchronized php-1.21wmf2/includes/Revision.php 'done'
  • 22:23 logmsgbot_: aaron synchronized php-1.21wmf2/includes/Revision.php 'tweak wfGetCaller param a level'
  • 22:22 logmsgbot_: aaron synchronized php-1.21wmf2/includes/Revision.php
  • 22:21 logmsgbot_: aaron synchronized php-1.21wmf2/includes/Revision.php 'temp query debug logging'
  • 22:18 logmsgbot_: olivneh synchronized php-1.21wmf2/extensions/EventLogging
  • 21:55 logmsgbot_: mlitn synchronized php-1.21wmf2/extensions/AbuseFilter 'desc'
  • 21:55 logmsgbot_: mlitn synchronized php-1.21wmf2/extensions/ArticleFeedbackv5 'desc'
  • 21:55 logmsgbot_: olivneh synchronized php-1.21wmf2/extensions/PostEdit
  • 21:55 logmsgbot_: mlitn synchronized wmf-config 'desc'
  • 21:46 logmsgbot_: mlitn synchronized php-1.21wmf2/extensions/AbuseFilter 'desc'
  • 21:45 logmsgbot_: mlitn synchronized php-1.21wmf2/extensions/ArticleFeedbackv5 'desc'
  • 21:01 logmsgbot_: demon synchronized wmf-config/CommonSettings.php 'Syncing I39c9693a'
  • 21:00 logmsgbot_: demon synchronized wmf-config/wgConf.php 'Syncing I39c9693a'
  • 20:30 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/ 'Dark deploy Wikibase, Diff and ULS'
  • 19:47 logmsgbot_: pgehres synchronized php-1.21wmf2/extensions/ContributionTracking/ 'Updating ContributionTracking to master'
  • 19:44 logmsgbot_: kaldari synchronized wmf-config/CommonSettings.php 'adding new var for CentralNotice'
  • 19:41 logmsgbot_: pgehres synchronized php-1.21wmf2/extensions/LandingCheck/ 'Updating LandingCheck to master'
  • 18:36 logmsgbot_: preilly synchronized php-1.21wmf2/extensions/MobileFrontend 'update after deploy'
  • 18:36 logmsgbot_: preilly synchronized php-1.21wmf1/extensions/MobileFrontend 'update after deploy'
  • 18:31 Ryan_Lane: Put in a live-hack on labsconsole to remove m1.tiny from the list of instance types
  • 18:29 cmjohnson1: installing base OS on srv194
  • 18:17 Ryan_Lane: updated OpenStackManager on labsconsole to master version
  • 18:16 binasher: flushed mobile varnish cache per preilly
  • 18:09 Jeff_Green: restarted gmetad on nickel
  • 17:53 logmsgbot_: preilly synchronized php-1.21wmf2/extensions/MobileFrontend 'update after deploy'
  • 17:52 logmsgbot_: preilly synchronized php-1.21wmf1/extensions/MobileFrontend 'update after deploy'
  • 16:56 paravoid: powercycling copper, load 700
  • 14:57 notpeter: silencing wikivoyage checks in nagios until deployed
  • 14:54 cmjohnson1: shutting down sq82 to replace /dev/sda
  • 14:52 cmjohnson1: shutting down srv194 to replace disk
  • 14:05 mark: Sending all non-European upload traffic to eqiad
  • 12:01 mark: Sending US upload traffic to eqiad
  • 12:01 logmsgbot_: demon synchronized wmf-config/wgConf.php 'Syncing out new prefixes for wikidata/wikivoyage'
  • 10:56 mark: Sending Argentina upload traffic to eqiad
  • 10:37 mark: Sending Brazil upload traffic to eqiad
  • 10:24 mark: Sending Canadian upload traffic to eqiad
  • 10:03 paravoid: powercycling nickel, OOM & unable to login
  • 09:42 mark: Hacked Nagios back up
  • 08:43 mark: Added georecords for wikidata-lb.wikimedia.org and wikivoyage-lb.wikimedia.org, their geomaps containing just a default entry pointing to eqiad
  • 08:33 mark: Added LVS service IPs for wikidata and wikivoyage (pmtpa/eqiad) to DNS
  • 07:12 mark: Stopping backend squid on sq82, sda I/O errors
  • 00:34 paravoid: upgrading oggvideotools to 0.8a on all imagescalers, fixing regression on the lucid->precise upgrade
  • 00:27 binasher: deploying updated squid mobile redirector
  • 00:24 logmsgbot_: tstarling synchronized wmf-config/CommonSettings.php

October 24

  • 23:12 binasher: restarting pdns on ns0
  • 23:09 cmjohnson1: db78 loading base OS
  • 23:06 cmjohnson1: authdns update adding db78.pmtpa.wmnet
  • 22:53 cmjohnson1: authdns changes adding db63-db78 mgmt
  • 21:37 logmsgbot_: aaron synchronized php-1.21wmf2/includes/EditPage.php 'deployed 51eede5af0fe0bb28d7abdf81a04292ba8a4864c'
  • 20:48 logmsgbot_: demon synchronized php-1.21wmf2/includes/Title.php 'Syncing Ic2d3f0b8'
  • 20:44 logmsgbot_: demon synchronized php-1.21wmf2/includes/EditPage.php 'Deploying I4c2055be'
  • 19:59 logmsgbot_: dzahn gracefulled all apaches
  • 18:49 logmsgbot_: catrope synchronized php-1.21wmf2/includes/upload/UploadStash.php 'rm debugging live hacks'
  • 18:46 logmsgbot_: catrope synchronized php-1.21wmf2/includes/upload/UploadStash.php 'Debugging exception'
  • 18:44 logmsgbot_: catrope synchronized php-1.21wmf2/includes/upload/UploadStash.php 'Debugging exception'
  • 18:39 logmsgbot_: catrope synchronized php-1.21wmf2/includes/upload/UploadStash.php 'Debugging exception'
  • 18:17 Reedy: Running sync-common on mw40
  • 18:02 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.21wmf2
  • 17:53 notpeter: upgrading srv223 and srv224 to precise
  • 17:41 logmsgbot_: preilly synchronized php-1.21wmf2/extensions/ZeroRatedMobileAccess 'update post deploy'
  • 17:40 logmsgbot_: preilly synchronized php-1.21wmf1/extensions/ZeroRatedMobileAccess 'update post deploy'
  • 17:38 logmsgbot_: lcarr synchronized wmf-config/throttle.php 'I updated a throttle'
  • 17:26 logmsgbot_: preilly synchronized php-1.21wmf2/extensions/MobileFrontend 'update after deploy'
  • 17:25 logmsgbot_: preilly synchronized php-1.21wmf1/extensions/MobileFrontend 'update after deploy'
  • 17:21 logmsgbot_: lcarr synchronized wmf-config/throttle.php 'I updated a throttle'
  • 17:10 cmjohnson1: db62 loading OS
  • 16:14 cmjohnson1: shutting down db61 for rack relocation
  • 16:03 cmjohnson1: removing h310 controller from db62 and adding original h700 card
  • 14:21 notpeter: upgarding srv221 and srv222 to precise
  • 13:39 cmjohnson1: installing base OS on db62
  • 11:56 paravoid: putting ms-fe2 (upgraded to precise/swift 1.7.4) back into the pool
  • 05:59 logmsgbot_: tstarling synchronized wmf-config/CommonSettings.php
  • 02:48 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Wed Oct 24 02:48:06 UTC 2012
  • 02:27 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Wed Oct 24 02:27:30 UTC 2012
  • 00:50 logmsgbot_: maxsem synchronized php-1.21wmf1/extensions/MobileFrontend
  • 00:26 mutante: flushing varnish mobile cache
  • 00:21 logmsgbot_: maxsem synchronized php-1.21wmf1/extensions/MobileFrontend/
  • 00:18 logmsgbot_: maxsem synchronized php-1.21wmf2/extensions/MobileFrontend/
  • 00:09 logmsgbot_: maxsem Finished syncing Wikimedia installation... : Weekly MobileFrontend and Zero deployment

October 23

  • 23:25 logmsgbot_: maxsem Started syncing Wikimedia installation... : Weekly MobileFrontend and Zero deployment
  • 23:09 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/ExtensionDistributor
  • 23:02 logmsgbot_: reedy synchronized php-1.21wmf2/includes/api/
  • 22:57 logmsgbot_: reedy synchronized php-1.21wmf2/includes/WikiPage.php
  • 22:57 logmsgbot_: reedy synchronized wmf-config/
  • 22:54 logmsgbot_: reedy synchronized php-1.21wmf2/includes/WikiPage.php 'debugging'
  • 22:53 logmsgbot_: reedy synchronized php-1.21wmf2/includes/WikiPage.php 'debugging'
  • 22:13 logmsgbot_: mlitn Finished syncing Wikimedia installation... : updating ArticleFeedbackv5, PageTriage & E3Experiments
  • 22:02 logmsgbot_: reedy synchronized php-1.21wmf2/includes/api/
  • 21:41 logmsgbot_: reedy synchronized php-1.21wmf2/includes/api/
  • 21:39 logmsgbot_: mlitn Started syncing Wikimedia installation... : updating ArticleFeedbackv5, PageTriage & E3Experiments
  • 20:57 Ryan_Lane: restarting gmetad on nickel
  • 20:56 Ryan_Lane: restarting gmetad
  • 20:40 logmsgbot_: reedy synchronized php-1.21wmf2/includes/
  • 20:31 Jeff_Green: dist-upgrade & rebooting payments[1-4]
  • 20:05 logmsgbot_: olivneh synchronized php-1.21wmf2/extensions/PostEdit
  • 20:02 cmjohnson1: fixing db62 raid cnfg
  • 19:10 logmsgbot_: asher synchronized wmf-config/db.php 'returning es10 to service'
  • 18:40 binasher: shutting down es10 for dimm replacement
  • 17:39 RobH: analytics1011 now offline and wiping as well
  • 17:15 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'wgMaxImageArea to 25MP'
  • 17:11 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/PagedTiffHandler
  • 16:36 cmjohnson1: db56 replacing disk 7 (slot6)
  • 16:14 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/PagedTiffHandler/
  • 16:08 logmsgbot_: reedy synchronized php-1.21wmf2/includes/db/Database.php
  • 16:07 logmsgbot_: reedy synchronized php-1.21wmf2/includes/Message.php
  • 16:06 cmjohnson1: db45 replacing disk2 (slot 1)
  • 15:55 RobH: analytics1012-1022 are starting wipe and having network cable pulled
  • 15:47 RobH: analytics1011-1022 will be dying today, byebye c2100
  • 15:31 logmsgbot_: reedy synchronized php-1.21wmf2/includes/EditPage.php
  • 15:21 paravoid: manually patching swift on ms-be3 to comment-out the drop_cache() calls
  • 14:57 logmsgbot_: reedy synchronized php-1.21wmf2/includes/api/ApiParse.php 'Dont parse special pages'
  • 14:50 paravoid: temporarily stopping puppet on brewster
  • 14:49 mark: Inserted new varnish build 3.0.3plus~rc1-wm4 into precise-wikimedia, fixing duplicate header issue
  • 14:49 notpeter: removing srv194 from apaches pool for drive error
  • 14:49 logmsgbot_: reedy synchronized php-1.21wmf2/includes/api/ApiParse.php 'Dont parse special pages'
  • 14:32 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/PostEdit 'Update PostEdit to master'
  • 12:53 mark: Inserted new varnish 3.0.3plus~rc1-wm3 packages into the precise-wikimedia APT repository, with a duplicate content-length header fix on streaming
  • 11:20 paravoid: depooling ms-fe2, in preparation for ubuntu/swift upgrade
  • 06:42 logmsgbot_: krinkle synchronized php-1.21wmf1/resources/mediawiki/mediawiki.searchSuggest.js 'Ibcc56f64'
  • 06:40 logmsgbot_: krinkle synchronized php-1.21wmf2/resources/mediawiki/mediawiki.searchSuggest.js 'Ibcc56f64'
  • 05:17 logmsgbot_: tstarling synchronized php-1.21wmf2/includes/SpecialPage.php
  • 02:55 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Tue Oct 23 02:55:38 UTC 2012
  • 02:29 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Tue Oct 23 02:29:31 UTC 2012

October 22

  • 23:04 logmsgbot_: reedy synchronized php-1.21wmf2/includes/EditPage.php
  • 23:00 LeslieCarr: switching RE mastership on cr2-eqiad
  • 22:43 LeslieCarr: draining cr2-eqiad again for downgrade
  • 22:37 logmsgbot_: asher synchronized wmf-config/db.php 'raising weight of db65'
  • 22:33 LeslieCarr: switching cr2-eqiad routing engine mastership
  • 22:19 logmsgbot_: asher synchronized wmf-config/db.php 'replacing db22 with db65'
  • 22:06 LeslieCarr: undraining cr2-eqiad
  • 22:01 Jeff_Green: stopping puppet on aluminium so fr-tech can local-test a config change
  • 21:48 LeslieCarr: switch routing engine mastership on cr2-eqiad
  • 21:23 LeslieCarr: commit full on cr2-eqiad which will restart all routing processes
  • 21:20 cmjohnson1: replacing disk13 on tridge array
  • 21:19 logmsgbot_: olivneh synchronized php-1.21wmf2/extensions/Vector
  • 21:17 logmsgbot_: olivneh synchronized php-1.21wmf2/extensions/PostEdit
  • 20:49 LeslieCarr: draining traffic from cr2-eqiad in preparation for maintenance and upgrade
  • 20:21 Reedy: Running p[opulateFileArchiveSha1.php on enwiki
  • 19:06 logmsgbot_: reedy synchronized php-1.21wmf2/includes/api/ApiParse.php
  • 18:53 binasher: streaming hotbackup of db1038 to db65, preparing to replace to db22
  • 18:28 logmsgbot_: reedy synchronized php-1.21wmf2/skins/CologneBlue.php
  • 18:16 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf2
  • 16:49 logmsgbot_: py gracefulled all apaches
  • 16:47 Jeff_Green: stopping mysql on db1025 to clone it to db1013
  • 16:02 logmsgbot_: reedy synchronized wmf-config/
  • 16:01 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/ExtensionDistributor
  • 15:48 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/ExtensionDistributor
  • 15:43 notpeter: re-adding srv219 and srv220 to rendering pool
  • 15:04 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/ExtensionDistributor
  • 13:32 notpeter: removing srv219 and srv220 from rendering pool for upgarde to precise
  • 09:35 mark: Built new varnish 3.0.3plus~rc1-wm2 packages and inserted them into the precise-wikimedia APT repository
  • 02:51 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Mon Oct 22 02:51:32 UTC 2012
  • 02:31 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Mon Oct 22 02:30:57 UTC 2012

October 21

  • 22:50 logmsgbot_: tstarling synchronized php-1.21wmf2/bin/ulimit4.sh
  • 22:39 Reedy: populateFilearchiveSha1.php finished on commonswiki
  • 14:18 Reedy: Running mwscript populateFilearchiveSha1.php commonswiki
  • 02:49 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Sun Oct 21 02:49:42 UTC 2012
  • 02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Sun Oct 21 02:28:50 UTC 2012
  • 01:35 paravoid: powercycling srv219, swap-died while trying to get a core dump
  • 01:34 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'wgMaxImageArea to 20MP'
  • 01:33 paravoid: powercycling srv220/srv224, swapdeath

October 20

  • 18:51 Reedy: Ran populateFilearchiveSha1.php on metawiki
  • 18:21 paravoid: powercycling srv221 & srv222, both dead since ~5h ago
  • 17:55 RobH_: updated otrs motd per requst by otrs admins (it referenced a now closed request for comment)
  • 17:42 Reedy: Running foreachwiki cleanupUploadStash.php
  • 03:13 K4-713: updated the payments cluster to 97e1ba74a39376
  • 02:41 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Sat Oct 20 02:41:10 UTC 2012
  • 02:21 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Sat Oct 20 02:21:41 UTC 2012
  • 01:36 K4-713: updated the payments cluster to bb30dab91a7afe862
  • 00:46 mutante: upgrade Bugzilla to 4.0.8 (security patch)

October 19

  • 23:36 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'wgMaxImageArea to 17MP'
  • 23:31 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/SwiftCloudFiles
  • 23:30 LeslieCarr: shutting down sfo-monitor1
  • 23:27 logmsgbot_: reedy synchronized php-1.21wmf2/includes/api/ApiParse.php
  • 23:15 binasher: returning srv194/200 to apache lvs pool
  • 22:50 mutante: authdns-update, renaming travel-guide-lb to wikivoyage-lb
  • 21:10 binasher: pulled srv194 and srv200 from lvs for memc testing
  • 21:02 binasher: upgrading percona-toolkit to 2.1.5
  • 20:18 logmsgbot_: reedy synchronized php-1.21wmf2/includes/api/ApiParse.php
  • 19:33 logmsgbot_: aaron synchronized php-1.21wmf2/includes/filebackend/FileBackendStore.php 'deployed 65e47037f6e9da2e81db2bc1374cdac375a82db5'
  • 19:26 paravoid: powercycling srv223 too, locked up, nothing in serial console
  • 19:25 paravoid: powercycling srv221, locked up, nothing in serial console
  • 16:21 RobH: msw-a2-eqiad back online
  • 15:54 RobH: msw-a2-eqiad is coming down for replacement per RT 3683. all mgmt connectivity (only) will be lost to items in A2-EQIAD
  • 14:43 paravoid: putting ms-fe1 back into the pool
  • 14:14 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commonswiki to 1.21wmf2
  • 13:59 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: metawiki to 1.21wmf2
  • 13:50 cmjohnson1: shutting down msbe1001-1012 to remove ssd's and package for dell return
  • 13:48 logmsgbot_: reedy synchronized php-1.21wmf2/includes/EditPage.php
  • 04:15 logmsgbot_: tstarling synchronized php-1.21wmf2/includes/User.php
  • 02:59 logmsgbot_: tstarling synchronized php-1.21wmf2/extensions/AbuseFilter/AbuseFilter.class.php
  • 02:57 logmsgbot_: tstarling synchronized php-1.21wmf2/includes/User.php
  • 02:47 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Fri Oct 19 02:47:49 UTC 2012
  • 02:27 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Fri Oct 19 02:27:57 UTC 2012
  • 02:27 logmsgbot_: tstarling synchronized php-1.21wmf2/extensions/AbuseFilter/AbuseFilter.class.php
  • 02:23 logmsgbot_: tstarling synchronized php-1.21wmf2/extensions/AbuseFilter/AbuseFilter.class.php
  • 02:10 logmsgbot_: tstarling synchronized php-1.21wmf2/includes/User.php
  • 01:59 logmsgbot_: tstarling synchronized php-1.21wmf2/includes/User.php
  • 01:56 logmsgbot_: tstarling synchronized php-1.21wmf2/extensions/CentralAuth/CentralAuthHooks.php
  • 01:53 K4-713: updated payments cluster to ff8a1e791eaff47313
  • 01:53 logmsgbot_: tstarling synchronized php-1.21wmf2/extensions/CentralAuth/CentralAuthHooks.php
  • 01:50 logmsgbot_: tstarling synchronized php-1.21wmf2/extensions/CentralAuth/CentralAuthHooks.php
  • 01:46 logmsgbot_: tstarling synchronized php-1.21wmf2/extensions/CentralAuth/CentralAuthHooks.php
  • 01:45 logmsgbot_: kaldari Finished syncing Wikimedia installation... :
  • 01:44 logmsgbot_: tstarling synchronized php-1.21wmf2/extensions/CentralAuth/CentralAuthHooks.php
  • 01:20 logmsgbot_: kaldari Started syncing Wikimedia installation... :
  • 00:26 logmsgbot_: olivneh synchronized wmf-config/InitialiseSettings.php
  • 00:26 logmsgbot_: olivneh synchronized php-1.21wmf1/extensions/PostEdit
  • 00:22 logmsgbot_: reedy synchronized wmf-config/
  • 00:21 paravoid: disabling puppet on (the already depooled) ms-fe1
  • 00:09 logmsgbot_: olivneh Finished syncing Wikimedia installation... :
  • 00:04 logmsgbot_: olivneh Started syncing Wikimedia installation... :
  • 00:03 logmsgbot_: olivneh synchronized php-1.21wmf1/extensions/PostEdit

October 18

  • 23:42 logmsgbot_: reedy synchronized wmf-config/
  • 23:31 logmsgbot_: spage Finished syncing Wikimedia installation... : updating 1.21wmf1 & wmf2 with E3Experiments (and 1.21wmf1 with PostEdit)
  • 23:26 Reedy: populateFilearchiveSha1.php ran for all wikis on 1.21wmf2
  • 23:16 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 23:14 logmsgbot_: spage Started syncing Wikimedia installation... : updating 1.21wmf1 & wmf2 with E3Experiments (and 1.21wmf1 with PostEdit)
  • 23:13 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikimedia.dblist to 1.21wmf2
  • 23:11 Reedy: Added fa_sha1 to liquidthreads_labswikimedia
  • 23:10 Reedy: Ran populateFilearchiveSha1.php on private, wiktionary, wikiversity and wikinews wikis
  • 23:07 logmsgbot_: tstarling synchronized php-1.21wmf2/includes/User.php 'debugging hack'
  • 23:04 Reedy: Added fa_sha1 to fdcwiki
  • 23:00 logmsgbot_: tstarling synchronized php-1.21wmf2/includes/User.php 'debugging hack'
  • 22:32 logmsgbot_: aaron synchronized php-1.21wmf2/includes/filebackend/FileBackendStore.php 'deployed 15520f489b9adab8bcfd84e6b6a5d48280a85556'
  • 22:05 logmsgbot_: reedy synchronized php-1.21wmf2/includes/api/ApiQueryRevisions.php
  • 21:26 logmsgbot_: aaron synchronized php-1.21wmf2/includes/filebackend/FileBackendStore.php 'deployed ff03172cccad1ce7017c7dc44e508317cc73975f'
  • 21:11 logmsgbot_: kaldari Finished syncing Wikimedia installation... :
  • 21:01 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php '$wgMaxImageArea to 15'
  • 20:18 logmsgbot_: kaldari Started syncing Wikimedia installation... :
  • 20:04 Reedy: Running scap-recompile on all mediawiki-installation
  • 19:28 Jeff_Green: adjusted a few hostnames for silverpop
  • 19:21 notpeter: created ulsfo rt queue
  • 19:19 logmsgbot_: reedy synchronized php-1.21wmf2/includes/EditPage.php
  • 18:53 logmsgbot_: reedy synchronized php-1.21wmf2/includes/
  • 17:54 Jeff_Green: adjusted banner log filter to include non-fundraising banners
  • 17:01 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Rest of planned wikis to 1.21wmf2 minus commons and meta
  • 16:53 K4-713: synchronized payments cluster to efcecc41c6f3c26
  • 15:19 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary and wikiversity to 1.21wmf2
  • 15:17 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 15:15 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews to 1.21wmf2
  • 15:07 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: private wikis to 1.21wmf2
  • 14:19 notpeter: re-adding srv190 to rendering pool
  • 02:50 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Thu Oct 18 02:50:46 UTC 2012
  • 02:26 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Thu Oct 18 02:26:24 UTC 2012

October 17

  • 22:31 logmsgbot_: reedy synchronized php-1.21wmf2/includes/specials/
  • 20:58 mutante: nuking wmfsocial mailing list
  • 20:48 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: revert test
  • 20:47 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test
  • 20:42 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Fix tetwiki
  • 20:40 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.21wmf2
  • 20:31 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki testwiki and test2wiki back to 1.21wmf2
  • 20:23 ^demon: restarted gerrit service, was missing documentation from UI
  • 19:33 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: private and special wikis back to 1.21wmf1
  • 19:19 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Increase $wgMaxImageArea to 2.5e7 on testwiki'
  • 19:14 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Increase $wgMaxImageArea to 1.4e7'
  • 18:54 binasher: fixed pc1 innodb_free_space logging on pc1 via "GRANT SELECT ON `parsercache`.`pc000` TO 'dbstats'@'localhost';" - (it had only been granted access to the objectcache table which no longer exists)
  • 18:53 logmsgbot_: olivneh synchronized php-1.21wmf1/extensions/E3Experiments/lib/event/eventlog.js
  • 18:44 cmjohnson1: racking and setting up replacements for ms-be6/7
  • 18:22 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: metawiki back to 1.21wmf1
  • 18:18 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Closed, special and private to 1.21wmf2
  • 18:08 logmsgbot_: reedy synchronized php-1.21wmf2/includes/api/ApiEditPage.php
  • 18:05 LeslieCarr: killed and restarted all varnish processes on cp1041
  • 18:00 LeslieCarr: restarted varnish-frontend on cp1041
  • 17:53 LeslieCarr: restarting varnish on cp1041
  • 17:24 mark: Installed experimental Varnish package with range support in streaming mode on cp1021
  • 16:50 logmsgbot_: midom synchronized wmf-config/InitialiseSettings.php
  • 16:21 LeslieCarr: restarting varnish on cp1042 , gone into swapdeath spiral
  • 16:15 logmsgbot_: reedy synchronized php-1.21wmf2/cache/l10n/
  • 16:01 logmsgbot_: reedy synchronized php-1.21wmf1/cache/l10n/
  • 15:59 notpeter: increasing weight on mw17-mw59 by 33.3%
  • 15:46 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuilding localisation cache for educationprogram
  • 15:36 Jeff_Green: switched donate.wikimedia.org to geodns
  • 15:31 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/EducationProgram
  • 15:23 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/EducationProgram/
  • 14:34 logmsgbot_: midom synchronized wmf-config/InitialiseSettings.php
  • 06:09 apergos: restarted lucene search on search1015
  • 04:10 Tim: restarted swift frontends
  • 03:41 Tim: seems to have worked, also truncating pc161-174 to be on the safe side
  • 03:38 Tim: on pc1: truncated tables pc158, pc159, pc160 due to flood of "table full" errors in dberror.log indicating innodb space exhaustion
  • 02:49 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Wed Oct 17 02:49:53 UTC 2012
  • 02:26 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Wed Oct 17 02:26:34 UTC 2012

October 16

  • 23:41 logmsgbot_: maxsem synchronized php-1.21wmf1/extensions/MobileFrontend/ 'https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-10-15%2616'
  • 23:33 logmsgbot_: maxsem synchronized php-1.21wmf2/extensions/MobileFrontend/ 'https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-10-15%2616'
  • 22:50 binasher: redeployed squid refresh_pattern no-cache tweaks
  • 22:46 MaxSem: Running mw-update-l10n and friends to catch up newest MobileFrontend messages
  • 21:45 paravoid: deploying squid configs, reverting a change that was too aggressive on caching content and broke https-by-default wikis
  • 21:40 logmsgbot_: reedy synchronized php-1.21wmf2/includes/specials/SpecialUndelete.php
  • 20:08 cmjohnson1: storage3 down for troubleshooting
  • 19:57 Reedy: Ran namespaceDupes.php on azwikibooks
  • 19:20 binasher: deployed squid refresh_pattern tweaks for testing
  • 19:19 Jeff_Green: adding dns and geodns entries for donate-lb
  • 18:58 cmjohnson1: shutting down db67 to add h800 card
  • 14:57 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'touch'
  • 14:56 logmsgbot_: hashar synchronized wmf-config/throttle.php '(bug 41069) Delhi 2012-10-17 workshop throttle rule'
  • 14:56 logmsgbot_: hashar synchronized wmf-config/throttle.php '(bug 41069) Delhi 2012-10-17 workshop throttle rule'
  • 12:31 logmsgbot_: reedy synchronized php-1.21wmf2/includes/
  • 09:06 logmsgbot_: nikerabbit synchronized wmf-config/InitialiseSettings.php '(bug 41015) Enable Narayam on hi.wiktionary'
  • 02:53 logmsgbot_: LocalisationUpdate completed (1.21wmf2) at Tue Oct 16 02:53:28 UTC 2012
  • 02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Tue Oct 16 02:28:10 UTC 2012
  • 00:57 RoanKattouw: chmod -R g+w /home/wikipedia/common/php-1.21wmf2/.git
  • 00:48 RoanKattouw: I mean 002
  • 00:48 RoanKattouw: Created ~spage/.bashrc with 'umask 0002'
  • 00:03 binasher: completed patch-job_token.sql migrations
  • 00:00 binasher: running patch-job_token.sql migration against all wikis

October 15

  • 23:04 mutante: power cycling cp1040
  • 21:55 logmsgbot_: catrope synchronized php-1.21wmf2/extensions/VisualEditor/ 'f25aaf5bc4449ea8c1ac749a96487485b5f0c6b3'
  • 21:54 logmsgbot_: catrope synchronized php-1.21wmf2/includes/resourceloader/ResourceLoaderFileModule.php '1f761e3bf349df8d48592f71ed4afff63471c6b4'
  • 21:44 mwalker: resuming purge of CentralNotice cache (now working) with rate limit of 50hps. estimated completion in 3hrs.
  • 21:29 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php '(bug 41016) Temporarily reenable wgUseNPPatrol on fi.wikipedia'
  • 20:53 logmsgbot_: reedy synchronized php-1.21wmf2/resources/
  • 20:53 logmsgbot_: reedy synchronized php-1.21wmf1/resources/
  • 20:52 cmjohnson1: shutting down es10 for per rt3724
  • 20:49 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/ArticleFeedback
  • 20:49 mutante: flushing mobile varnish cache
  • 20:43 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/ArticleFeedback
  • 20:37 logmsgbot_: reedy synchronized php-1.21wmf1/includes/resourceloader/ResourceLoaderFileModule.php
  • 20:37 mwalker: rate limiting for CentralNotice purge is working too well... ie: script is not doing anything -- debugging
  • 20:35 logmsgbot_: awjrichards synchronized php-1.21wmf1/extensions/MobileFrontend/ 'touch files'
  • 20:21 mwalker: Attempting to purge out another 500k pages with CentralNotice skin problem (this time not crashing the cluster by filtering out media files and rate limiting to no more than 50hps). Estimated run time is 3 hours.
  • 20:20 logmsgbot_: awjrichards synchronized php-1.21wmf1/extensions/MobileFrontend/ 'touch files'
  • 20:10 logmsgbot_: asher synchronized wmf-config/db.php 'pulling es10, third crash in three weeks'
  • 20:07 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection/ 'Revert back to 9f37005e78a068821c45d9d313394c6728f55b4a'
  • 20:05 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/Collection/ 'Revert back to 9f37005e78a068821c45d9d313394c6728f55b4a'
  • 19:57 cmjohnson1: powering down srv266 troubleshooting per rt2896
  • 19:04 binasher: restarted udp2log on oxygen with correct --recv-queue
  • 18:58 logmsgbot_: reedy synchronized wmf-config/
  • 18:55 logmsgbot_: reedy synchronized php-1.21wmf1/includes/resourceloader/ResourceLoaderFileModule.php
  • 18:49 RoanKattouw: Created fluorine:/a/mw-log/udp2log/resourceloader.log chmod 644 and owned by udp2log:udp2log as root per Reedy's request
  • 18:19 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/CodeReview
  • 18:11 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.21wmf2
  • 18:07 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/MobileFrontend
  • 17:32 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'wgDebugLogGroups for resourceloader'
  • 17:30 logmsgbot_: reedy synchronized php-1.21wmf1/includes/resourceloader/ResourceLoaderFileModule.php
  • 16:28 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/MobileFrontend/
  • 16:26 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/MobileFrontend/
  • 16:16 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Re-enable education program on enwiki'
  • 16:06 logmsgbot_: reedy Finished syncing Wikimedia installation... : Build localisation cache for 1.21wmf2
  • 15:43 ottomata: stopping puppet on oxygen, looking for cause of recent packet loss
  • 15:19 logmsgbot_: reedy synchronized php-1.21wmf2/extensions/MobileFrontend
  • 15:17 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/MobileFrontend
  • 15:12 logmsgbot_: reedy Started syncing Wikimedia installation... : Build localisation cache for 1.21wmf2
  • 14:56 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.21wmf2
  • 14:55 logmsgbot_: reedy synchronized wmf-config/
  • 14:51 logmsgbot_: reedy synchronized php-1.21wmf2/ 'Initial sync'
  • 13:35 Reedy: Killing php-1.20wmf11 l10n cache from apaches
  • 13:34 Reedy: Killing php-1.20wmf11 l10n cache from apaches
  • 13:31 Reedy: Pointing php to php-1.21wmf1
  • 13:29 logmsgbot_: reedy synchronized wmf-config/throttle.php 'Updated from gitrepo'
  • 13:28 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Remove extdist conditional for mount path'
  • 11:49 mark: Deployed range_offset_limit 64 MB change (from 0) to all upload backend squids
  • 11:24 paravoid: restarting pdns @ ns2
  • 11:16 paravoid: switching European bits back to esams
  • 09:48 paravoid: restarting all imagescaler apaches, did not recover after swift outage
  • 09:24 paravoid: live-hacking swift on ms-be10, ms-be12, ms-be1 to remove fadvise calls
  • 07:53 Tim: increasing worker count to 100 on all swift backends, via puppet
  • 07:37 Tim: ms-be11 showed an immediate improvement in bandwidth out, but wchan still indicates that 48 is not enough, increasing to 100
  • 07:33 Tim: experimentally doubling the worker count on ms-be11 since wchan indicates that the worker pool is exhausted
  • 05:49 apergos: ms-be12 reporting errors on disk /dev/sdi, I've rt'ed it an unmounted it in the meantime
  • 04:34 Tim: on ms-fe2: restarted swift-proxy due to mem leak
  • 04:28 Tim: on ms-fe3: restarting swift-proxy due to swap
  • 04:26 LeslieCarr: restarted swift proxy on ms-fe4 - it was in swap doom
  • 02:25 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Mon Oct 15 02:25:12 UTC 2012

October 14

  • 21:57 logmsgbot_: reedy synchronized php-1.21wmf1/includes/specials/SpecialUserlogin.php
  • 14:26 logmsgbot_: reedy synchronized php-1.21wmf1/cache/l10n/
  • 14:22 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Sun Oct 14 14:22:54 UTC 2012
  • 02:00 logmsgbot_: LocalisationUpdate failed: git pull of extensions failed

October 13

  • 21:39 logmsgbot_: preilly Finished syncing Wikimedia installation... : update zero rated mobile access
  • 21:18 logmsgbot_: preilly Started syncing Wikimedia installation... : update zero rated mobile access
  • 21:06 logmsgbot_: preilly synchronized php-1.21wmf1/extensions/MobileFrontend 'update for landing page'
  • 18:51 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to master'
  • 18:41 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to original again..'
  • 18:38 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to f980f2380b830e712edfdbce065784d1bcba82fd'
  • 18:36 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to 4f898444ce5d9c2c9a5491f2303db15a3143a5c5'
  • 18:34 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to 72b7d4d645489dd092605a2183a944fac33cb40b'
  • 18:26 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to original'
  • 18:25 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to 8415a4564bf429b76e113368a3e5e1c19fb84a47'
  • 18:23 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to 72b7d4d645489dd092605a2183a944fac33cb40b'
  • 18:21 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to 9f37005e78a068821c45d9d313394c6728f55b4a'
  • 16:34 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection
  • 16:27 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection
  • 13:17 logmsgbot_: reedy synchronized wmf-config/throttle.php
  • 02:00 logmsgbot_: LocalisationUpdate failed: git pull of extensions failed

October 12

  • 23:51 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'enabling redis debug log'
  • 23:40 LeslieCarr: moving traffic back to esams via authdns-scenario
  • 23:40 LeslieCarr: reenabled nagios alerts
  • 23:20 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'enabling session multiwrites to redis'
  • 23:16 LeslieCarr: rebooting esams core switch stack, expect reachability problems to esams
  • 23:01 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'disabling session multiwrites to redis'
  • 22:36 LeslieCarr: switching europe traffic to pmtpa via authdns-scenario
  • 22:36 LeslieCarr: muted nagios
  • 20:20 AaronSchulz: Confirmed all swift -> netapp sync scripts finished (s4 one kept dying to random http errors)
  • 19:58 logmsgbot_: reedy synchronized wmf-config/
  • 19:01 cmjohnson1: powering down search32 to replace cpu1
  • 18:33 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'AFT (non v5) doesn't use clicktracking now'
  • 18:28 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/ArticleFeedback/
  • 18:19 cmjohnson1: troubleshooting db42 w/IBM tech
  • 17:46 cmjohnson1: pc3 shutting down to replace hdd's with ssd's rt3703
  • 17:29 cmjohnson1: pc2 shutting down to replaced hdd w/ ssd rt3703
  • 16:27 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Fix extdist mount related warnings'
  • 16:14 cmjohnson1: running installer on storage3
  • 15:54 paravoid: moving bits from esams to pmtpa (authdns normal scenario)
  • 15:52 cmjohnson1: storage 3 replacing disk 12 and rebuilding raid
  • 15:32 mark: Disabled csw1-esams:e8/2
  • 14:37 mark: Rebooting cr1-esams
  • 14:17 mark: Upgrading JunOS on cr1-esams to 11.4R5.5, preparing for reboot
  • 13:59 mark: Activated aggregate generation on cr1-esams
  • 13:49 mark: Moved AS38930 transit from csw1-esams:e8/4 to cr1-esams:xe-1/3/0
  • 13:24 cmjohnson1: storage3 powering down
  • 12:27 mark: Upgraded new stack member asw-oe11-esams to junos 11.4R5.5 and rebooted it
  • 12:16 mark: Added new EX4200-48T asw-oe11-esams to csw2-esams stack
  • 11:08 mark: Disabled csw2-oe12-esams:vcp-1
  • 02:01 logmsgbot_: LocalisationUpdate failed: git pull of extensions failed
  • 01:21 mutante: moving ancient copy of bugzilla out of /h/w/htdocs (to /root/backup on fenari temp. just in case)
  • 01:15 mutante: deleting old document root dirs on fenari. /home/wikipedia/htdocs/noc and noc.old (use /h/w/common/docroot/noc and mediawiki-config repo)
  • 01:08 mutante: restarting Apache on fenari for noc.wm change
  • 00:42 Jeff_Green: switched payments back to pmtpa and increased TTL to 5 min
  • 00:37 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'And back to master'
  • 00:31 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'Revert due to error'
  • 00:29 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection 'Update collection to master'
  • 00:09 cmjohnson1: storage3 being reimaged

October 11

  • 23:19 mutante: manually copy index.html for noc from /common/docroot/noc to /h/w/htdocs/noc/
  • 23:15 logmsgbot_: dzahn synchronized docroot/noc/index.html
  • 23:13 logmsgbot_: dzahn synchronized docroot/noc/index.html
  • 23:08 logmsgbot_: olivneh synchronized php-1.21wmf1/extensions/E3Experiments
  • 22:16 logmsgbot_: olivneh synchronized php-1.21wmf1/extensions/E3Experiments
  • 21:58 logmsgbot_: bsitu Finished syncing Wikimedia installation... : Update ArticleFeedbackv5, MoodBar, PageTriage to master
  • 21:36 logmsgbot_: bsitu Started syncing Wikimedia installation... : Update ArticleFeedbackv5, MoodBar, PageTriage to master
  • 20:48 Jeff_Green: flipping payments.wikimedia.org to eqiad cluster to live-test
  • 20:12 mutante: rebooting analytics1007
  • 19:02 notpeter: stopping puppet on brewster
  • 18:50 RobH: i need to pull virt1001-1003 out of nagios later, as they are now renamed
  • 16:52 notpeter: restarting pdns on ns0
  • 16:50 logmsgbot_: preilly synchronized php-1.21wmf1/extensions/MobileFrontend 'update for landing page'
  • 16:50 notpeter: dist-upgrade and reboot on oxygen
  • 16:49 K4-713: updated the payments cluster to 4d0223ebb48f9b
  • 16:39 RobH: authdns update for pc1001-1003 mgmt ip
  • 15:35 RobH: working on analytics1007 per rt 3582
  • 14:55 RobH: shutting down db1012 to remove evaluation ssd card
  • 14:14 paravoid: apt: including php-parsekit, python-async, python-git, python-gitdb, python-smmap backports for precise
  • 08:59 apergos: restarted swift proxy on ms-fe3 and ms-fe2 with some delay between. note I had to shoot stale processes from Sept on all these boxes as well
  • 08:50 apergos: restarted ms-fe4 swift-proxy
  • 04:13 mutante: activated W3 Total Cache performance plugin in Wordpress (blog.wikimedia.org)
  • 04:12 mutante: marmontel (blog): fixed some file permissions in ./blog/wp-content for www-data the wordpress caching plugin needs to be re-activated
  • 02:27 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Thu Oct 11 02:27:51 UTC 2012
  • 01:55 LeslieCarr: moved traffic back to normal with authdns-scenario
  • 01:48 LeslieCarr: reenabling nagios notifications
  • 01:22 LeslieCarr: upgrading csw2-esams member 1
  • 01:22 LeslieCarr: upgrading csw2-esams member 0
  • 01:13 binasher: disabled nagios notifications
  • 01:04 LeslieCarr: upgrading csw2-esams member 2
  • 01:02 LeslieCarr: upgrading csw2-esams member 3
  • 00:44 binasher: restarted pdns on ns2
  • 00:44 LeslieCarr: restarting dns on ns2
  • 00:33 LeslieCarr: draining esams via authdns-scenarios
  • 00:11 logmsgbot_: reedy synchronized php-1.21wmf1/includes/api/ApiUpload.php

October 10

  • 23:02 RobH: db1008.mgmt returned to service
  • 23:02 RobH: msw-a2-eqiad has a bad set of ports, needs replacement per rt 3683
  • 22:46 RobH: reboot seems to have restored system (is posting) but drac still not responding, possible bad cable, will test shortly
  • 22:40 RobH: db1008 being cold reset (removing power cords)
  • 22:05 mutante: apt-get dist-upgrade on marmontel (blog)
  • 21:20 LeslieCarr: undraining esams via authdns-scenario normal
  • 20:43 LeslieCarr: draining esams via authdns-scenario for upgrade
  • 20:20 Jeff_Green: returning payments to pmtpa
  • 20:14 Jeff_Green: flipped payments.wikimedia.org to the new eqiad cluster, live testing
  • 20:06 logmsgbot_: awjrichards synchronized php-1.21wmf1/extensions/MobileFrontend/javascripts/common/mf-application.js 'touchfile'
  • 18:46 cmjohnson1: storage3 power cycling
  • 18:24 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Rest of wikipedia projects to 1.21wmf1
  • 17:58 logmsgbot_: reedy synchronized php-1.20wmf12/cache/interwiki.cdb 'Update interwiki cache'
  • 17:58 logmsgbot_: reedy synchronized php-1.21wmf1/cache/interwiki.cdb 'Update interwiki cache'
  • 17:15 notpeter: created indices for fdcwiki on searchidx2 and searchidx1001
  • 17:13 logmsgbot_: reedy synchronized docroot/fdc/
  • 16:57 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Add meta as import source for fdcwiki'
  • 16:43 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Set wgServer for fdcwiki'
  • 15:24 logmsgbot_: reedy ran sync-common-all
  • 14:48 logmsgbot_: reedy synchronized s3.dblist
  • 14:39 logmsgbot_: reedy synchronized wmf-config/
  • 14:37 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 12:40 mark: Throttled SnapMirror originals transfer to 75 MB/s
  • 12:20 mark: Increased SnapMirror TCP window size on nas1-a and nas1001-a for the originals snapmirror relationship, and restarted the initial transfer at the last checkpoint
  • 06:28 apergos: restarted lucene search on search1016
  • 03:01 binasher: completed fa_sha1 migration
  • 02:41 logmsgbot_: LocalisationUpdate completed (1.20wmf12) at Wed Oct 10 02:41:09 UTC 2012
  • 02:20 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Wed Oct 10 02:20:42 UTC 2012
  • 00:02 binasher: started filearchive.fa_sha1 migration, starting with s1

October 9

  • 23:34 logmsgbot_: olivneh synchronized php-1.21wmf1/extensions/E3Experiments
  • 23:32 logmsgbot_: tstarling synchronized php-1.21wmf1/includes/parser/ParserCache.php
  • 23:31 logmsgbot_: tstarling synchronized php-1.20wmf12/includes/parser/ParserCache.php
  • 23:30 logmsgbot_: reedy synchronized php-1.20wmf12/resources/startup.js 'touch'
  • 23:30 logmsgbot_: reedy synchronized php-1.20wmf12/resources/jquery/jquery.js '1.8.2'
  • 23:28 logmsgbot_: tstarling synchronized wmf-config/InitialiseSettings.php
  • 23:20 mutante: flushing mobile varnish cache
  • 23:18 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Add outreachwiki to import sources of enwiki'
  • 23:17 logmsgbot_: awjrichards synchronized php-1.21wmf1/extensions/MobileFrontend/
  • 23:11 logmsgbot_: awjrichards synchronized wmf-config/InitialiseSettings.php 'Enable sticky cookie fix on all wikis; Update logo asset path for mobile footer'
  • 23:06 logmsgbot_: dzahn synchronized docroot/fdc/w
  • 22:51 logmsgbot_: dzahn gracefulled all apaches
  • 22:50 logmsgbot_: dzahn synchronized docroot/fdc/index.php
  • 22:41 logmsgbot_: dzahn synchronized /docroot/fdc
  • 22:38 logmsgbot_: reedy synchronized php-1.21wmf1/resources/startup.js 'touch'
  • 22:26 logmsgbot_: reedy synchronized php-1.21wmf1/resources/jquery/jquery.js 'jQuery 1.8.2'
  • 22:21 mutante: creating new doc root for fdc wiki /usr/local/apache/common/docroot/fdc
  • 22:10 notpeter: readded mw61 to bits backends
  • 21:53 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection
  • 21:29 cmjohnson1: swapping disk 12 out on storage 3
  • 21:07 LeslieCarr: restarting apache2 on srv220
  • 20:31 mutante: authdns-update on dobson, adding fdc.wikimedia.org DNS entry
  • 19:36 Jeff_Green: added a couple temporary filters to oxygen for fr-tech
  • 17:28 notpeter: reimaging mw61 with precise for bits test
  • 16:29 Aaron|home: Set abbrvThreshold to 160 for thumbnails
  • 16:27 logmsgbot_: aaron synchronized wmf-config/filebackend.php 'Set abbrvThreshold to 160 for thumbnails.'
  • 16:22 logmsgbot_: aaron synchronized wmf-config/filebackend.php 'Set abbrvThreshold to 180 for thumbnails.'
  • 16:18 logmsgbot_: aaron synchronized wmf-config/filebackend.php 'Set abbrvThreshold to 200 for thumbnails.'
  • 13:31 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Pre-emptively set $wgContentHandlerUseDB = false;'
  • 11:35 mark: Setup sync SnapMirror relationship from nas1-a:originals to nas1001-a:originals and started transfer
  • 11:29 mark: Increased maxfiles of nas1001-a:originals to 63753378
  • 11:27 mark: Created 36 drive media1 raid-dp aggregate on nas1001-a, hosting new 40T volume 'originals'
  • 10:59 paravoid: rolling restart of swift proxies (ms-fe2/3/4) because of the memory leaks
  • 09:32 logmsgbot_: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Bug 40366 - Narayam'
  • 09:25 logmsgbot_: nikerabbit synchronized php-1.21wmf1/extensions/Translate/api/ApiGroupReview.php
  • 02:50 logmsgbot_: LocalisationUpdate completed (1.20wmf12) at Tue Oct 9 02:50:29 UTC 2012
  • 02:25 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Tue Oct 9 02:25:49 UTC 2012

October 8

  • 20:47 Aaron|home: Started syncFileBackend.php (swift -> nas) on screens on hume (s4)
  • 20:41 logmsgbot_: maxsem synchronized php-1.21wmf1/extensions/MobileFrontend/MobileFrontend.php 'Live hack to unbreak prop=extracts on 1.21wmf1'
  • 20:36 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/ConfirmEdit/FancyCaptcha.class.php 'Revert cache key change live hack'
  • 20:02 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Set wgOldChangeTagsIndex to false for all wikis running with new indexes'
  • 19:22 Aaron|home: Started syncFileBackend.php (swift -> nas) on screens on hume (s1-s3,s5-s7)
  • 19:13 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'wgOldChangeTagsIndex to false for outreachwiki'
  • 19:07 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
  • 19:05 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
  • 18:44 Reedy: Updated liquidthreads_labswikimedia CategoryLinks schema with patch-categorylinks-better-collation.sql
  • 18:36 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf1
  • 16:43 logmsgbot_: reedy synchronized wmf-config/
  • 11:49 apergos: manually cleaned up /tmp on mw49, forced a puppet run too (isn't there a cron job to clean up /tmp?), stopped job runner afterwards since it had been stopped already by someone
  • 11:31 Reedy: Manually running sync-common on mw49
  • 11:14 logmsgbot_: ariel synchronized wmf-config 'move from ms7 upload6 to nas1 upload7'
  • 02:52 logmsgbot_: LocalisationUpdate completed (1.20wmf12) at Mon Oct 8 02:52:16 UTC 2012
  • 02:27 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Mon Oct 8 02:27:45 UTC 2012

October 7

  • 18:09 logmsgbot_: asher synchronized wmf-config/CommonSettings.php 'move captchadir to upload7'
  • 10:11 logmsgbot_: maxsem synchronized wmf-config 'Raise creation throttle for India event even more'
  • 02:48 logmsgbot_: LocalisationUpdate completed (1.20wmf12) at Sun Oct 7 02:47:58 UTC 2012
  • 02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Sun Oct 7 02:28:29 UTC 2012

October 6

  • 15:56 logmsgbot_: lcarr synchronized wmf-config/throttle.php 'I updated a throttle'
  • 14:14 logmsgbot_: lcarr synchronized wmf-config/throttle.php 'I updated a throttle'
  • 03:11 logmsgbot_: LocalisationUpdate completed (1.20wmf12) at Sat Oct 6 03:11:10 UTC 2012
  • 02:39 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Sat Oct 6 02:39:31 UTC 2012
  • 02:17 Reedy: Manually re-running localisation update
  • 02:02 logmsgbot_: LocalisationUpdate failed: git pull of extensions failed

October 5

  • 20:44 pp-pdf1: updated mwlib.zim to 0.3.0
  • 20:02 notpeter: removing srv190 from imagescaling pool due to misbuilt pacakge
  • 18:21 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/ProofreadPage
  • 17:22 Reedy: Live hacked 1.21wmf1 ConfirmEdit/FancyCaptcha.class.php cache key for directory list. Needs reverting after 24 hours or so
  • 17:17 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/ConfirmEdit/FancyCaptcha.class.php 'New memcached key'
  • 17:06 logmsgbot_: reedy synchronized wmf-config/
  • 14:00 notpeter: restarting puppet on brewster
  • 13:54 mark: Rollback of upload-lb.eqiad; sending traffic back to pmtpa
  • 13:48 notpeter: stopping puppet on brewster
  • 13:42 logmsgbot_: ariel synchronized wmf-config 'rollback ms7 to nas1 upload6 to upload7 migration'
  • 12:17 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'wgCacheEpoch to 20120908000000'
  • 12:04 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Update wgCacheEpoch and wgThumbnailEpoch to the start of 2012 (+ 1 year)'
  • 12:04 apergos: started one last rsync op upload/ from ms7 to nas1 with --update
  • 11:46 mark: Moved /mnt/upload7/upload/* to /mnt/upload7, removed upload /mnt/upload7/upload dir
  • 11:42 logmsgbot_: ariel synchronized wmf-config 'migrate media etc from ms7 upload6 to nas1 upload7'
  • 11:41 apergos: running sync-dir wmf-config to migrate media etc from ms7 upload6 to nas1 upload7
  • 02:46 logmsgbot_: LocalisationUpdate completed (1.20wmf12) at Fri Oct 5 02:45:59 UTC 2012
  • 02:27 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Fri Oct 5 02:27:21 UTC 2012
  • 01:03 mutante: updating a lot of mailing list descriptions per BZ:37537 / thehelpfulone

October 4

  • 23:59 mutante: manually fixed brion's ssh key on fenari. session opened for user brion. welcome back :)
  • 22:06 mutante: powercycling downed cp1044
  • 21:40 logmsgbot_: olivneh synchronized php-1.20wmf12/extensions/E3Experiments
  • 21:28 logmsgbot_: lcarr synchronized wmf-config/throttle.php 'I updated a throttle'
  • 20:41 logmsgbot_: olivneh synchronized php-1.20wmf12/extensions/ClickTracking/modules/jquery.clickTracking.js
  • 20:18 logmsgbot_: olivneh synchronized php-1.20wmf12/extensions/E3Experiments
  • 18:09 logmsgbot_: catrope synchronized php-1.21wmf1/languages/messages/MessagesEn.php 'Ib8746ece3a34b5e577dc08596b0ff7b1b96b5d73'
  • 18:08 logmsgbot_: catrope synchronized php-1.21wmf1/resources/mediawiki/mediawiki.feedback.js 'Ib8746ece3a34b5e577dc08596b0ff7b1b96b5d73'
  • 16:16 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
  • 15:30 logmsgbot_: reedy synchronized wmf-config/
  • 14:02 Reedy: Manually ran sync-common on srv247
  • 13:34 notpeter: removing srv191 from bits pool for test upgrade to precise
  • 10:39 Tim: gave myself an account on sanger's exim so that I can send outbound mail through it
  • 05:24 apergos: restarted lucene search on search1016 again
  • 02:49 logmsgbot_: LocalisationUpdate completed (1.20wmf12) at Thu Oct 4 02:49:23 UTC 2012
  • 02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Thu Oct 4 02:28:12 UTC 2012
  • 00:14 logmsgbot_: reedy synchronized php-1.21wmf1/extensions/Collection/Collection.body.php

October 3

  • 22:47 mutante: reinstalling analytics1007
  • 21:38 mutante: deleting education-ar mailman list per bug 40749
  • 20:55 logmsgbot_: dzahn synchronized wmf-config/throttle.php 'IP cap lift for Grace Hopper event, bug 40669'
  • 20:18 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else non wikipedia to 1.21wmf1
  • 20:15 mutante: powercycling analytics1007
  • 20:15 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikimedia and wiktionary to 1.21wmf1
  • 20:13 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks and wikinews to 1.21wmf1
  • 20:12 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity and wikiquote to 1.21wmf1
  • 20:09 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special, private and closed to 1.21wmf1
  • 19:33 logmsgbot_: preilly synchronized php-1.21wmf1/extensions/MobileFrontend 'update for landing page'
  • 19:32 logmsgbot_: preilly synchronized php-1.20wmf12/extensions/MobileFrontend 'update for landing page'
  • 19:29 logmsgbot_: preilly synchronized php-1.21wmf1/extensions/ZeroRatedMobileAccess 'update for landing page'
  • 19:28 logmsgbot_: preilly synchronized php-1.20wmf12/extensions/ZeroRatedMobileAccess 'update for landing page'
  • 15:43 Reedy: Close strategywiki, zawiktionary, zh_min_nanwikibooks
  • 15:41 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'WikiLove on itwikiquote'
  • 15:36 Reedy: Created wikilove tables to itwikiquote
  • 15:18 Jeff_Green: test-deploying SPF DNS records
  • 09:49 mark: Doubled number of inodes on nas1-a:originals from 31876689 to 63753378
  • 08:38 apergos: stopped ms7 to ms8 replication
  • 08:18 apergos: rebooting ms8 to unstick deadlocked zfs recv and zfs list processes
  • 07:20 Tim: upgrading luasandbox on all apaches to version 1.1-1
  • 06:28 apergos: restarted lucene search on search1016
  • 05:22 logmsgbot_: tstarling synchronized php-1.21wmf1/cache/trusted-xff.cdb
  • 05:19 logmsgbot_: tstarling synchronized php-1.20wmf12/cache/trusted-xff.cdb
  • 03:11 logmsgbot_: tstarling synchronized wmf-config/db.php 'es10 back up'
  • 03:03 Tim: on es10: started mysqld
  • 02:51 logmsgbot_: LocalisationUpdate completed (1.21wmf1) at Wed Oct 3 02:51:35 UTC 2012
  • 02:45 Tim: es10 not responding to ssh, rebooting
  • 02:43 logmsgbot_: tstarling synchronized wmf-config/db.php
  • 02:31 logmsgbot_: LocalisationUpdate completed (1.20wmf12) at Wed Oct 3 02:31:56 UTC 2012
  • 02:09 logmsgbot_: tstarling synchronized wmf-config/InitialiseSettings.php 'API log'
  • 01:35 logmsgbot_: krinkle synchronized live-1.5
  • 01:29 logmsgbot_: tstarling synchronized live-1.5
  • 01:25 Tim: on srv247, srv270, srv271, srv286, mw69: killing more stuck apt-get processes, will run apt-get upgrade
  • 01:19 Tim: on srv233: killed frozen apt-get, running since August 17, running apt-get upgrade
  • 01:17 logmsgbot_: krinkle Finished syncing Wikimedia installation... : Icd721011b40bb8d2c20aefa8b359a3e45413a07f
  • 01:09 Tim: upgrading wikimedia-task-appserver on all apaches
  • 01:06 Tim: on brewster: updated wikimedia-task-appserver
  • 00:42 mutante: moved logmsgbot to -operations channel
  • 00:34 logmsgbot: krinkle Started syncing Wikimedia installation... : Icd721011b40bb8d2c20aefa8b359a3e45413a07f
  • 00:25 Reedy: Restarted Jenkins

October 2

  • 23:45 logmsgbot: olivneh synchronized php-1.20wmf12/extensions/E3Experiments/experiments/openTask.js 'Updating event format for OpenTask funnel'
  • 23:44 logmsgbot: reedy synchronized php-1.21wmf1/extensions/Collection/
  • 23:19 Ryan_Lane: deploying Damianz's OpenStackManager change for adding/removing bastion users on shell access on labsconsole
  • 16:17 logmsgbot: asher synchronized wmf-config/db.php 'adding db64 to s3 in place of db25'
  • 16:02 mark: Reclaimed space on brewster:/ - restarted squdi
  • 15:19 mark: Sending ALL upload traffic to upload-lb.eqiad (Varnish)
  • 14:57 mark: Sending traffic from India, Korea, China, Singapore, Thailand, Vietnam to upload-lb.eqiad (Varnish)
  • 14:26 mark: Sending Mexican upload traffic to upload-lb.eqiad (Varnish)
  • 14:12 mark: Sending Brazil upload traffic to upload-lb.eqiad (Varnish)
  • 14:10 mark: Restarted pdns on linne
  • 14:03 mark: Sending Japanese upload traffic to upload-lb.eqiad (Varnish)
  • 13:57 notpeter: rebooting srv194-199 and mw55-59 to pick up kernel upgrades, and then re-adding them to apaches pool
  • 13:10 notpeter: rebooting mw1-16 to pick up kernel upgrades before re-enabling them as (precise) jobrunners
  • 13:05 notpeter: enabling srv190 in imagescaler pool at half-weight to test precise image-scaling
  • 11:52 paravoid: apt: include new php5 built with libicu42 (precise-wikimedia/main), include libicu42
  • 11:46 mark: Moving US upload traffic to upload-lb.eqiad (Varnish)
  • 09:06 paravoid: squid deploy all: switching math & timeline to be served from swift
  • 05:23 binasher: started replicating db64 from the s3 master
  • 03:51 logmsgbot: olivneh synchronized php-1.20wmf12/extensions/E3Experiments/experiments/openTask.js
  • 03:28 logmsgbot: olivneh synchronized php-1.20wmf12/extensions/E3Experiments/experiments/openTask.js
  • 02:37 logmsgbot: LocalisationUpdate completed (1.21wmf1) at Tue Oct 2 02:37:57 UTC 2012
  • 02:21 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Tue Oct 2 02:21:28 UTC 2012
  • 01:56 Tim: send sessions to both the existing memcached and also redis, to fill redis and allow for a graceful complete switch later
  • 01:55 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
  • 01:41 Tim: restarting all apaches to get redis PHP extension
  • 01:35 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
  • 01:33 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
  • 01:16 binasher: streaming hotbackup of db1019 to db64 (s3)
  • 01:12 binasher: stopping mysql on db25 and powering down - logging lots of ecc chipkill errors and periodically unresponsive. building db64 as a perm replacement
  • 01:00 logmsgbot: awjrichards Finished syncing Wikimedia installation... : Updating MobileFrontend, per https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-10-01
  • 00:50 logmsgbot: asher synchronized wmf-config/db.php 'pulling db25, returning es10'
  • 00:07 logmsgbot: awjrichards Started syncing Wikimedia installation... : Updating MobileFrontend, per https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-10-01

October 1

  • 22:10 logmsgbot: awjrichards synchronized wmf-config/mobile.php 'Eanble wgMFEnableDesktopResources on testwiki'
  • 22:09 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Eanble wgMFEnableDesktopResources on testwiki'
  • 21:40 logmsgbot: preilly synchronized php-1.21wmf1/extensions/ZeroRatedMobileAccess/ 'zero weekly carrier test'
  • 21:18 logmsgbot: preilly synchronized php-1.20wmf12/extensions/ZeroRatedMobileAccess/ 'zero weekly carrier test'
  • 20:44 logmsgbot: reedy synchronized php-1.21wmf1/includes/specials/SpecialUserlogin.php 'bug 40668'
  • 20:23 logmsgbot: reedy synchronized php-1.21wmf1/includes/OutputPage.php
  • 20:21 logmsgbot: reedy synchronized php-1.21wmf1/includes/OutputPage.php
  • 20:16 logmsgbot: reedy synchronized php-1.21wmf1/includes/OutputPage.php
  • 20:13 logmsgbot: reedy synchronized php-1.21wmf1/includes/OutputPage.php
  • 19:33 paravoid: pushing new rewrite.py to swift servers (r24303, r24514, r24576)
  • 19:31 logmsgbot: reedy synchronized php-1.21wmf1/includes/job/Job.php
  • 19:24 logmsgbot: reedy synchronized php-1.20wmf12/includes/job/Job.php
  • 18:51 notpeter: stopping puppet on brewster
  • 18:22 logmsgbot: asher synchronized wmf-config/db.php 'pulling es10, crashed over weekend, needs to catchup on repl'
  • 18:01 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.21wmf1
  • 15:52 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.21wmf1, rebuild cache
  • 15:40 ^demon: restarted gerrit on manganese.
  • 15:36 ^demon: gracefully shut down gerrit until we figure out why manganese is swapping
  • 14:18 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.21wmf1, rebuild cache
  • 14:08 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.21wmf1, rebuild cache
  • 14:06 mark: Added cp1029 and cp1030 to the frontend varnish upload pool in eqiad (PyBal)
  • 13:59 logmsgbot: reedy synchronized php-1.21wmf1/
  • 13:53 logmsgbot: reedy synchronized live-1.5/
  • 13:53 logmsgbot: reedy synchronized wmf-config/
  • 13:48 logmsgbot: reedy synchronized php-1.21wmf1
  • 13:10 cmjohnson1: powering down db62 to replace raid controller card
  • 12:38 logmsgbot: reedy synchronized live-1.5
  • 12:34 Reedy: Make that, deleting wmf10 php source files
  • 12:32 Reedy: Deleting wmf12 php source files
  • 12:30 Reedy: Deleting wmf11 localisation cache
  • 11:11 mark: Added cp1029 and cp1030 to the Varnish upload.eqiad pool (10 mins ago)
  • 09:17 mark: Rebooting cp3001, both ports in LACP block
  • 02:25 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Mon Oct 1 02:25:14 UTC 2012

September 30

  • 23:58 MaxSem: Ran mwscript extensions/WikimediaMaintenance/migrateWlmUploads.php testwiki --do-it>wlm.log
  • 19:26 logmsgbot: maxsem synchronized wmf-config 'Remove WLM banner / https://gerrit.wikimedia.org/r/#/c/25771/'
  • 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Sun Sep 30 02:23:14 UTC 2012

September 29

  • 10:39 mark: Started rsync of /export/upload (respecting /etc/rsync.includes) to /mnt (nas1-a:vol/originals/) on ms7 in a screen session
  • 10:35 mark: Manually mounted nas1-a:/vol/originals on ms7, fenari
  • 10:33 mark: Created 40 TB volume 'originals' inside 36-drive aggregate 'media1' on nas1-a
  • 04:07 mutante: reinstalled cp1031 (new mainboard, thx RobH)
  • 02:24 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Sat Sep 29 02:24:45 UTC 2012
  • 01:59 mutante: restarting cp1031 after hardware replacement
  • 01:24 mutante: checking/fixing puppet runs on cp103x servers

September 28

  • 23:54 binasher: rebooting db1047 to new kernel
  • 21:28 apergos: disbaled puppet and stopped swift processes on ms-be6 again. noticed four drives in "Unconfigured" state in megacli output after disk replacement,don'tknow more details about how that went.
  • 20:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 20:04 mark: Destroyed nas1001-a:images volume, containing aggregate test0, and started zeroing drives
  • 20:02 mark: Destroyed test0 aggregate on nas1-a, zeroing disks
  • 19:55 mark: Destroyed volume nas1-a:images
  • 19:54 MaxSem: And one more autocreation + re-running the migration script
  • 19:51 mark: Destroyed snapmirror relationship between nas1001-a:images -> nas1-a:images and deleted related snapshots
  • 19:51 MaxSem: Autocreated Commons accounts for 3 testwiki users to migrate their images
  • 19:24 MaxSem: running mwscript extensions/WikimediaMaintenance/migrateWlmUploads.php testwiki --do-it
  • 19:03 logmsgbot: reedy synchronized wmf-config/
  • 19:01 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 18:55 logmsgbot: reedy synchronized wmf-config/
  • 18:38 logmsgbot: reedy synchronized wmf-config/
  • 18:04 binasher: stopping mysql on db1047 for upgrades
  • 16:19 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Set wgCopyUploadProxy and enable copyupload on testwiki'
  • 16:17 logmsgbot: reedy synchronized php-1.20wmf12/includes/upload/UploadFromUrl.php
  • 16:16 logmsgbot: reedy synchronized php-1.20wmf12/includes/DefaultSettings.php
  • 15:46 RobH: done with db62. it was in the ubuntu installer when I took it over, so its just sitting now
  • 14:53 RobH: db62 being pulled for drac7 work
  • 14:39 Jeff_Green: rebooting aluminium
  • 13:11 mark: Moved BGP transit sessions to AS1257 from cr2-eqiad (over equinix exchange) to cr1-eqiad (dedicated link)
  • 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Fri Sep 28 02:26:27 UTC 2012

September 27

  • 23:13 logmsgbot: olivneh synchronized php-1.20wmf12/extensions/Vector/modules/ext.vector.footerCleanup.css
  • 22:50 logmsgbot: olivneh synchronized wmf-config
  • 22:05 logmsgbot: olivneh synchronized php-1.20wmf12/extensions/E3Experiments
  • 22:00 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update PageTriage and ArticleFeedbackv5 to master
  • 21:32 logmsgbot: bsitu Started syncing Wikimedia installation... : Update PageTriage and ArticleFeedbackv5 to master
  • 20:48 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'turning off moodbar on testwiki'
  • 20:09 RobH: authdns-update for new misc servers mgmt in eqiad
  • 19:10 cmjohnson1: search32 going down to swap cpu's (dell request)
  • 18:58 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 17:42 cmjohnson1: bring db62 down to swap disk controller card per ct/asher
  • 17:16 logmsgbot: demon synchronized wmf-config/throttle.php 'Syncing I8c44415b: Account creation throttle for ptwikiversity event'
  • 16:42 cmjohnson1: ms-be6 powering down
  • 16:37 RobH: cp1031 mainboard replaced per rt3614, will need reinstall
  • 15:59 RobH: mc1006 memory repaired rt3613
  • 15:53 RobH: mc1002 repaired rt3612
  • 15:33 RobH: mc1002 (rt3612) & mc1006 (rt3613) offline for memory repair/swap
  • 15:20 RobH: authdns-update for wikimediafoundation.info
  • 15:06 Jeff_Green: indium dist-upgrade & reboot
  • 13:58 Jeff_Green: dist-upgrade & reboot grosley
  • 10:55 apergos: ignore ms-be6 messages, I'm trying to get into the bleeping lsi raid util
  • 04:04 logmsgbot: olivneh synchronized php-1.20wmf12/extensions/E3Experiments/experiments/postEditFeedback.js
  • 02:36 Jeff_Green: powercycle fenari, it's nonresponsive even via drac terminal

September 26

  • 20:42 RobH: mc1011 drac fixed
  • 20:39 RobH: mc1009, 1010 drac fixed
  • 20:30 RobH: cp1031 will remain offline until replacement parts arrive tomorrow rt3614
  • 18:44 RobH: correction rt 3613
  • 18:43 RobH: mc1006 testing done, single bad dimm, support ticket filed rt3614
  • 18:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: 285 remaining wikis to 1.20wmf12
  • 18:06 RobH: mc1006 offline for memory troubleshooting per rt 3613
  • 16:52 RobH: mc1002 offline, memory bad per rt3612, new memory will arrive tomorrow
  • 16:14 RobH: carbon rebooting for console redirection issues, any eqiad based installs will fail during this downtime
  • 15:57 Jeff_Green: dist-upgrade & reboot most payments boxes
  • 15:21 RobH: bast1001 link speed fixed, done working on it
  • 15:19 RobH: troubleshooting bast1001 link speed per 3414
  • 15:12 cmjohnson1: sq37 has fatal error...powering down to replace disk controller card
  • 13:56 cmjohnson1: ms-be8 powercycling
  • 02:48 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Wed Sep 26 02:48:22 UTC 2012
  • 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Wed Sep 26 02:23:25 UTC 2012
  • 01:00 mutante: reinstalling cp1030 with precise
  • 00:34 mutante: fixing puppet on singer..finally

September 25

  • 23:25 binasher: ran "varnishadm param.set nuke_limit 300" on all mobile varnish front and back instances to match new default config
  • 23:20 binasher: repooled cp1044
  • 23:16 binasher: depooling cp1044 from lvs for testing
  • 22:44 binasher: stopping puppet on cp1044, experimenting with varnish lru / nuke params to solve allocation failures
  • 22:28 mutante: generating locales on singer, add ro UTF-8 locale, run ro.planet
  • 21:50 Jeff_Green: dropped apparently-unused IP alias on loudon, deprecated from old haproxy install
  • 21:50 Jeff_Green: apt-update and reboot loudon
  • 21:36 Jeff_Green: replaced iptables ruleset for loudon
  • 21:02 notpeter: stopping puppet on brewster
  • 20:49 mutante: sync-common-file new favicon.ico for wikidata.org
  • 20:49 logmsgbot: dzahn synchronized docroot/www.wikidata.org/favicon.ico 'updating wikidata favicon per Lydia'
  • 17:57 Jeff_Green: restarted udp2log on locke to clean up ~20 defunct procs
  • 17:56 notpeter: temp stopping puppet on brewster
  • 17:27 mutante: adding otto to ops group in LDAP
  • 14:39 cmjohnson1: ms-be7 shutting down to check jumpers
  • 14:35 cmjohnson1: ms-be8 shutting down to check jumpers
  • 12:14 mark: Powercycled carbon, has been down for 4d
  • 05:58 Tim: updated Scribunto for bug 40480
  • 05:45 paravoid: upgrading all PDF boxes for USN-1581-1/CVE-2012-4405
  • 05:45 paravoid: upgrading all imagescalers for USN-1581-1/CVE-2012-4405

September 24

  • 19:44 RobH: db1012 card install done, system powered down for asher
  • 19:36 RobH: db1012 shutting down for pci card install per rt3558
  • 19:14 logmsgbot: reedy synchronized php-1.20wmf12/extensions/TimedMediaHandler 'Updating to master'
  • 18:54 pgehres: added new columns to fundraising contribution_tracking table
  • 18:20 notpeter: temp stopping puppet on brewster
  • 18:15 logmsgbot: hashar synchronized wmf-config/filebackend-wmflabs.php 'beta: disable commons as a default foreign file repo'
  • 18:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf12
  • 17:03 cmjohnson1: powering up and toying with ms-be8 for h/w checks
  • 16:35 RobH: mc1001-1008 memory upgrade complete
  • 16:35 cmjohnson1: ms-be6 shutting down for h/w testing
  • 15:51 RobH: mc1001-mc1008 are having memory upgrades installed. hosts already offline.
  • 13:53 pp-pdf1: restart all services
  • 13:52 pp-pdf1: updated mwlib.epub to 0.14.2
  • 13:52 pp-pdf1: updated mwlib.rl to 014.1
  • 13:52 pp-pdf1: update mwlib to 0.14.1
  • 13:51 pp-pdf1: updated timelib to 0.2.4
  • 13:50 pp-pdf1: updated pip to 1.2.1
  • 13:50 pp-pdf1: updated distribute to 0.6.28
  • 13:17 LeslieCarr: switching unnumbered GRE tunnel to numbered GRE Tunnel between cr2-knams and cr2-eqiad for routability reasons
  • 12:45 LeslieCarr: restarting powerdns on ns2
  • 09:34 logmsgbot: hashar synchronized docroot/bits/static-master/ '(bug 40419) extension assets not available on beta (aka add docroot/bits/static-master'
  • 09:23 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 40436) Namespaces configuration for se.wikipedia'
  • 09:19 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 40186) Favicon for es.wiktionary'
  • 02:53 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Mon Sep 24 02:53:05 UTC 2012
  • 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Mon Sep 24 02:27:55 UTC 2012
  • 01:53 Tim: on hume: running updateCollation.php with --force for bug 40462
  • 01:50 Tim: depooled precise appservers and stopped job runners on precise servers

September 23

  • 23:26 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'Reduce wgMaxGeneratedPPNodeCount to 1.5M'
  • 17:29 binasher: powercycled cp1043
  • 02:45 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Sun Sep 23 02:45:21 UTC 2012
  • 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Sun Sep 23 02:26:13 UTC 2012

September 22

  • 09:26 mark: Restarted lucene on search1015
  • 02:46 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Sat Sep 22 02:46:32 UTC 2012
  • 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Sat Sep 22 02:27:27 UTC 2012

September 21

  • 20:55 notpeter: removing srv190 from apaches pool to use to test build as an imagescaler
  • 20:38 logmsgbot: asher synchronized wmf-config/db.php 'moving ES writes from es1 (cluster23) to es2 (cluster24) and es3 (cluster25)'
  • 20:29 logmsgbot: asher synchronized wmf-config/db.php 'db.php with new es shards added, but not being written to'
  • 20:29 ^demon: restarted gerrit like 10m ago to deploy I03de7c74 (apache timeout increase)
  • 20:17 Jeff_Green: apt-get upgrade on manganese, restart apache for adjusted 5->10m gerrit timeout
  • 20:15 mutante: analytics1007 seems dead. powercycle but never see any console output on mgmt at all
  • 14:49 hashar: labs: cleaned out a 4GB file out of labs-nfs1:/export/home/deployment-prep
  • 02:49 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Fri Sep 21 02:49:27 UTC 2012
  • 02:25 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Fri Sep 21 02:25:33 UTC 2012
  • 01:39 logmsgbot: krinkle synchronized php-1.20wmf12/extensions/LiquidThreads 'I5a098a42'
  • 01:37 logmsgbot: krinkle synchronized php-1.20wmf12/skins/common/preview.js 'Ia9673a75'
  • 00:16 binasher: configured pt-heartbeat on es2 and es3 shards

September 20

  • 23:18 logmsgbot: kaldari synchronized php-1.20wmf12/extensions/PageTriage/modules/ext.pageTriage.defaultTagsOptions/ext.pageTriage.defaultTagsOptions.js
  • 23:17 logmsgbot: kaldari synchronized php-1.20wmf11/extensions/PageTriage/modules/ext.pageTriage.defaultTagsOptions/ext.pageTriage.defaultTagsOptions.js
  • 22:58 Ryan_Lane: restarting gerrit
  • 22:36 notpeter: stopping puppet on brewster
  • 22:20 logmsgbot: aaron synchronized wmf-config/filebackend.php
  • 22:18 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Enabled short thumbnails for testwiki'
  • 22:15 logmsgbot: olivneh synchronized php-1.20wmf11/extensions/E3Experiments/experiments
  • 22:02 logmsgbot: kaldari Finished syncing Wikimedia installation... :
  • 21:32 logmsgbot: kaldari Started syncing Wikimedia installation... :
  • 20:32 ottomata: reinstalling analytics1002-analytics1010 with Ubuntu precise
  • 20:25 logmsgbot: aaron synchronized php-1.20wmf12/includes/filebackend/SwiftFileBackend.php
  • 20:25 logmsgbot: aaron synchronized php-1.20wmf11/includes/filebackend/SwiftFileBackend.php
  • 20:13 mutante: bugs.mediawiki.org|bugzilla.wikipedia.org|bugzilla.mediawiki.org all redirect to https:// rather than http now
  • 20:12 logmsgbot: dzahn gracefulled all apaches
  • 20:10 mutante: sync-apache and pushing out redirects.conf change for Bugzilla redirects to https
  • 20:01 logmsgbot: kaldari Finished syncing Wikimedia installation... :
  • 19:50 logmsgbot: aaron synchronized php-1.20wmf11/includes/filebackend/SwiftFileBackend.php 'deployed 8001d1ebf5b79b5a6ea359646284347d31e5442b'
  • 19:38 logmsgbot: aaron synchronized php-1.20wmf12/includes/filebackend/SwiftFileBackend.php 'deployed 4246f85ac2a68a19b9fa8f4954c5ff7aabea2922'
  • 19:33 logmsgbot: py synchronized wmf-config/lucene.php
  • 19:32 notpeter: pushing all search traffic back to eqiad
  • 19:18 logmsgbot: kaldari Started syncing Wikimedia installation... :
  • 18:58 binasher: ran scap on mw58 and returning to the lvs pool
  • 17:08 logmsgbot: aaron synchronized php-1.20wmf12/thumb.php 'deployed fd26ac3603c544c9f2273408b14034cd22b56a52'
  • 17:04 notpeter: temp stopping puppet on brewster
  • 10:41 hashar: UploadWizard has been broken in some cases following the release of wmf12. Restored by reverting a change. See bug 40380 for details.
  • 10:13 logmsgbot: hashar synchronized php-1.20wmf12/resources/mediawiki.language '(bug 40380) inexistent language files are loaded'
  • 08:12 logmsgbot: hashar synchronized php-1.20wmf12/includes/EditPage.php '(bug 40098) Wrong edit summary generation when adding a new discussion section'
  • 08:11 logmsgbot: hashar synchronized php-1.20wmf11/includes/EditPage.php '(bug 40098) Wrong edit summary generation when adding a new discussion section'
  • 08:03 hashar: Jenkins: installing "Gerrit Trigger plugin" 2.6 which replaces our 2.5.3 snapshot
  • 05:44 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
  • 05:40 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
  • 05:26 logmsgbot: tstarling synchronized php-1.20wmf12/includes/parser/Parser.php 'node limit log'
  • 05:23 logmsgbot: tstarling synchronized php-1.20wmf11/includes/parser/Preprocessor_DOM.php 'node limit'
  • 05:22 logmsgbot: tstarling synchronized php-1.20wmf11/includes/parser/Parser.php 'node limit'
  • 05:22 logmsgbot: tstarling synchronized php-1.20wmf11/includes/parser/ParserOptions.php 'node limit'
  • 05:13 Tim: fixing common-local on mw58, has wrong permissions
  • 05:11 logmsgbot: tstarling synchronized php-1.20wmf11/includes/DefaultSettings.php 'node limit'
  • 03:23 Ryan_Lane: purged search data from dataset2 on diederik's request
  • 02:44 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Thu Sep 20 02:44:42 UTC 2012
  • 02:21 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Thu Sep 20 02:20:57 UTC 2012

September 19

  • 22:48 binasher: pulling mw58 from lvs pool and deploy groups for temporary testing
  • 22:07 logmsgbot: olivneh synchronized php-1.20wmf11/extensions/E3Experiments
  • 21:50 notpeter: temp stopping puppet on brewster
  • 21:28 apergos1: ms-be10: after chris installed the disks, filesystems were created ok by puppet and mounted but on a couple of reboots, there were boot messages about two not identical lists of disk not present or not ready; waiting a little bit allowed them to come up but it's not great.
  • 20:56 Jeff_Green: restarting gmetad on nickel
  • 20:32 Reedy: Deleting php-1.20wmf10 localisation cache from mediawiki-installation
  • 20:00 Reedy: Deleting php-1.20wmf9 from mediawiki-installation
  • 19:43 logmsgbot: reedy synchronized php-1.20wmf12/extensions/CategoryTree 'Category tree to master'
  • 18:12 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: Swapping all non-pedias to 1.20wmf12
  • 18:07 cmjohnson1: ms-be10 powering down to replace 12 disk drives
  • 17:38 logmsgbot: py synchronized wmf-config/lucene.php
  • 17:30 logmsgbot: py synchronized wmf-config/lucene.php
  • 17:26 logmsgbot: aaron synchronized php-1.20wmf12/thumb.php 'deployed 7be7e496ae2d6e4b082278ab4ace8c1b4b08e433'
  • 17:24 notpeter: moving search traffic to pmtpa for eqiad search package upgrades, starting with en
  • 15:22 cmjohnson1: updating authdns. adding 12 new misc servers mgmt ip's
  • 13:31 mark: Deactivated subnet for vlan 104 (Tampa)
  • 12:28 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 39264) Add Tudalen: and Indecs: namespaces to cy.wikipedia'
  • 12:24 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 38840) Namespaces configuration on uz.wikipedia'
  • 12:18 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 38840) Namespaces configuration on uz.wikipedia'
  • 12:13 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 39206) Namespaces configuration for se.wikipedia'
  • 12:02 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 40163) Try to fix ltwiki import source for betawikiversity'
  • 06:01 paravoid: depooling ms-fe1; broken after precise upgrade; pending further investigation
  • 04:46 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild localisation caches for pagetriage
  • 04:04 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild localisation caches for pagetriage
  • 02:48 logmsgbot: reedy synchronized php-1.20wmf12/extensions/PageTriage
  • 02:48 logmsgbot: reedy synchronized php-1.20wmf11/extensions/PageTriage
  • 02:42 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Wed Sep 19 02:42:01 UTC 2012
  • 02:22 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Wed Sep 19 02:22:47 UTC 2012
  • 01:27 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
  • 01:17 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php

September 18

  • 20:17 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '23795 disable anon edits on wikimania2012 wiki'
  • 20:13 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 39652) Fix "autoreviewer" restriction level on ptwiki'
  • 20:09 logmsgbot: hashar: ran "mwscript updateArticleCount.php --wiki guwikisource --update" for bug 40173
  • 20:07 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 40173) Set wgArticleCountMethod for gu.wikisource'
  • 20:04 logmsgbot: hashar synchronized wmf-config/CommonSettings-wmflabs.php '17365 beta: Removes transcoding stub'
  • 19:50 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'cleanup sep11wiki left over'
  • 19:50 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'cleanup sep11wiki left over'
  • 19:50 logmsgbot: hashar synchronized closed.dblist 'cleanup sep11wiki left over'
  • 19:48 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'Cleaning Berlin hackathon tutorial configuration.'
  • 19:45 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 39942) Disables patrol on fi.wikipedia'
  • 19:43 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 39905) Create interface editor group on pt.wikipedia'
  • 19:38 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '24207 enable EPUB collection format on ALL wiki
  • 19:34 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'ran stylize.php on InitialiseSettings.php 23998'
  • 19:13 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'Removing live hack to enable EPUB format on all wiki, should use $wmgCollectionUseEpub instead (see our internal mailing list)'
  • 19:01 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 40186) New logo for es.wiktionary'
  • 17:16 logmsgbot: hashar synchronized wmf-config/InitialiseSettings-wmflabs.php 'fix AFTv5 on beta'
  • 17:07 notpeter: putting srv190 and mw56-59 back into apaches pool
  • 16:52 cmjohnson1: search32 going down to check if it's DIMM error or possible board or cpu1 error
  • 15:41 cmjohnson1: srv249 powercycling to check drac configuration
  • 14:08 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 40122) disable GlobalBlocking on fishbowl, private wikis'
  • 13:53 logmsgbot: hashar synchronized php-1.20wmf11/includes/Sanitizer.php 'bug 40306 - Only convert align to float for table'
  • 13:52 logmsgbot: hashar synchronized php-1.20wmf11/tests/phpunit/includes/SanitizerTest.php 'bug 40306 - Only convert align to float for table'
  • 13:48 apergos1: ms-fe1 back in swift frontend pool, upgrade to precise complete.
  • 13:41 hashar: 23798 manually deleted /apache/common-local/docroot/wikipedia.org/{million,test}.txt files from all mediawiki-installation hosts
  • 12:53 logmsgbot: hashar synchronized php-1.20wmf12/includes/Sanitizer.php 'bug 40306 - Only convert align to float for table'
  • 12:53 logmsgbot: hashar synchronized php-1.20wmf12/tests/phpunit/includes/SanitizerTest.php 'bug 40306 - Only convert align to float for table'
  • 12:22 apergos1: reboot ms-fe1, upgrading to precise
  • 05:59 Aaron|home: Last of the math/ file copy scripts has finished
  • 02:42 logmsgbot: LocalisationUpdate completed (1.20wmf12) at Tue Sep 18 02:42:15 UTC 2012
  • 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Tue Sep 18 02:26:08 UTC 2012
  • 00:20 Ryan_Lane: adding an autofs direct map to LDAP, pointing at the gluster storage for testing home directory move

September 17

  • 23:43 paravoid: adding a modified ixgbe.ko to precise-installer's initrd.gz; adding ixgbe-dkms to apt
  • 23:30 notpeter: starting puppet on brewster
  • 23:17 notpeter: stopping puppet on brewster
  • 21:45 apergos: (from 5 hours ago) last of the swift proxy services restarted, it had reached 8.6gb of swap use. good for a few more days now
  • 20:52 Jeff_Green: stopped puppet on aluminium for config testing
  • 20:45 logmsgbot: preilly synchronized wmf-config/CommonSettings.php 'enable EPUB'
  • 18:55 notpeter: removing srv190 and mw55-59 from apaches pool for upgrade to precise
  • 18:48 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.20wmf12
  • 18:47 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild localisation cache
  • 18:20 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild localisation cache
  • 18:13 logmsgbot: aaron synchronized php-1.20wmf12/extensions/timeline/Timeline.php
  • 18:10 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enabled html5 on all wikis'
  • 18:07 binasher: granted 'research' read access to user rollup/filter dbs on dbs 42,1047
  • 18:06 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.20wmf12 and rebuild localisation cachae
  • 16:36 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.20wmf12 and rebuild localisation cachae
  • 16:21 logmsgbot: reedy synchronized php-1.20wmf11/extensions/ArticleFeedbackv5/ 'Bring in logging related fixes etc'
  • 16:17 logmsgbot: reedy synchronized php-1.20wmf12 'Initial sync'
  • 15:31 Reedy: Deleted 1.20wmf10 l10n cache
  • 15:28 Reedy: Pointing php to php-1.20wmf11
  • 08:21 apergos1: swap was using 7gb on ms-fe1... restarted swift-proxy over there. wlll prolly restart it on ms-fe2 through 4 over the next day or two
  • 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Mon Sep 17 02:26:30 UTC 2012

September 16

  • 14:53 mark: powercycled srv256
  • 14:53 mark: powercycled srv250
  • 14:49 mark: cleared profiling
  • 02:20 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Sun Sep 16 02:20:02 UTC 2012

September 15

  • 19:04 mark: Power cycled srv212, 211, 226, 273, 237
  • 19:04 logmsgbot: tstarling synchronized wmf-config/mc.php 'swapping out other slow servers'
  • 18:53 logmsgbot: tstarling synchronized wmf-config/mc.php 'swap out srv237 for srv249'
  • 18:46 logmsgbot: tstarling synchronized php-1.20wmf11/includes/objectcache/MemcachedClient.php 'memcached profiling'
  • 18:37 mark: Reduced MaxRequestsPerChild from 300 to 4 in apache.conf
  • 18:29 LeslieCarr: logs stopped on fluorine between 18:11 and 18:27 UTC
  • 14:03 LeslieCarr: powercycling srv259 (unresponsive)
  • 13:57 LeslieCarr: rebooting srv270 (unresponsive)
  • 13:36 LeslieCarr: rebooting srv288
  • 11:54 logmsgbot: andrew synchronized wmf-config/throttle.php 'Fix undeclared variable error'
  • 09:56 logmsgbot: reedy synchronized wmf-config/throttle.php 'Bug 40263 - Request to lift IP address for mass creation - Philippines SFD 2012'
  • 08:52 thedj: someone should deploy Planets update, or add blogs to the old config because otherwise WLM has passed before the blogs are added to the stream.
  • 04:04 LeslieCarr: started ircd using the instructions here - http://wikitech.wikimedia.org/view/IRCD on ekr�em
  • 03:55 LeslieCarr: powercycling ekrem
  • 02:29 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Sat Sep 15 02:29:57 UTC 2012

September 14

  • 15:32 mark: Powercycled sodium
  • 15:18 logmsgbot: hashar synchronized docroot/noc/conf/wikiversions.dat 'publish wikiversions.dat on noc.wm.org'
  • 04:43 Aaron|home: Copying missing timeline files from swift to nfs on hume
  • 04:36 logmsgbot: aaron synchronized php-1.20wmf11/extensions/timeline/Timeline.php 'Switched to doOperations().'
  • 03:24 logmsgbot: aaron synchronized php-1.20wmf11/extensions/timeline 'deployed 6e7076f35b38c16579342c69e4f19f8df4ed0ab6'
  • 02:22 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Fri Sep 14 02:22:36 UTC 2012

September 13

  • 23:04 logmsgbot: dzahn gracefulled all apaches
  • 23:04 mutante: apache-graceful-all, pushing out redirect for wikimedia.biz
  • 21:02 mutante: fixing UID for spage on stat1 (/etc/passwd and files), was used twice
  • 20:34 logmsgbot: reedy synchronized php-1.20wmf11/extensions/Collection/
  • 18:47 RobH: authdns-update for wikimedia.biz
  • 17:05 mutante: flushing mobile varnish cache
  • 17:03 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Enable wlm banner'
  • 17:02 logmsgbot: awjrichards synchronized php-1.20wmf11/extensions/MobileFrontend/stylesheets/banner.css 'Touching banner.css'
  • 02:20 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Thu Sep 13 02:20:30 UTC 2012
  • 01:51 logmsgbot: aaron synchronized php-1.20wmf11/thumb.php 'debug logging'
  • 01:46 logmsgbot: aaron synchronized php-1.20wmf11/thumb.php 'debug logging'
  • 01:44 logmsgbot: awjrichards synchronized php-1.20wmf11/extensions/MobileFrontend/stylesheets/banner.css 'Update timestamp on banner.css'
  • 01:41 logmsgbot: aaron synchronized php-1.20wmf11/thumb.php 'debug logging'
  • 01:32 logmsgbot: aaron synchronized php-1.20wmf11/thumb.php 'debug logging'
  • 01:01 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 're-disable wlm banner on enwiki'
  • 00:58 logmsgbot: awjrichards synchronized php-1.20wmf11/extensions/MobileFrontend/
  • 00:55 logmsgbot: awjrichards synchronized wmf-config/mobile.php 'Easier toggle of MF banner'
  • 00:54 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Easier toggle of MF banner'
  • 00:34 logmsgbot: awjrichards synchronized wmf-config/mobile.php 'Disabling wlm banner for mobile frontend'
  • 00:18 logmsgbot: catrope synchronized php-1.20wmf11/extensions/Vector 'c9c0c3098c5bc6c15362332a1ed18e8545b1ab07'
  • 00:17 logmsgbot: catrope synchronized php-1.20wmf11/skins/common/shared.css 'f9242f450ae16cf4f0a99e6fa78248e13ccb4ee0'
  • 00:12 logmsgbot: aaron synchronized php-1.20wmf11/extensions/timeline/Timeline.php 'debug logging'
  • 00:07 logmsgbot: maxsem Finished syncing Wikimedia installation... : WLM app banner - MF, WikimediaMessages and config

September 12

  • 23:52 logmsgbot: reedy synchronized php-1.20wmf11/extensions/MobileFrontend/
  • 23:38 logmsgbot: maxsem Started syncing Wikimedia installation... : WLM app banner - MF, WikimediaMessages and config
  • 23:09 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'migrate favicons from upload to bits'
  • 23:08 logmsgbot: hashar synchronized docroot/strategyapps/favicon.ico
  • 23:07 logmsgbot: hashar synchronized docroot/strategy/favicon.ico
  • 23:06 logmsgbot: hashar synchronized docroot/steward/favicon.ico
  • 23:06 logmsgbot: hashar synchronized docroot/outreach/favicon.ico
  • 23:06 logmsgbot: hashar synchronized docroot/checkuser/favicon.ico
  • 23:06 logmsgbot: aaron synchronized php-1.20wmf11/extensions/timeline 'deployed f79a67ac344801acc57edbc95158bf5ecc87481b'
  • 23:03 logmsgbot: hashar synchronized docroot/bits/favicon
  • 23:02 logmsgbot: hashar synchronized docroot/bits/favicon
  • 22:24 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Enabled auto-resyncing for file backends.'
  • 22:12 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Disabled directory operations on local-temp zones.'
  • 22:12 logmsgbot: reedy synchronized closed.dblist
  • 21:48 logmsgbot: aaron synchronized php-1.20wmf11/includes/filebackend/FileBackendMultiWrite.php 'deployed 6a6a0427160e1493617789ce320020b21d0836dd'
  • 21:48 logmsgbot: aaron synchronized php-1.20wmf11/languages/messages/MessagesEn.php 'deployed 6a6a0427160e1493617789ce320020b21d0836dd'
  • 21:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Subpagesortkey on ptwikibooks and enwikibooks'
  • 21:13 RoanKattouw_: On ms7: find /export/upload/math ! -user apache -exec /root/fixownership2 \{\} \;
  • 21:10 logmsgbot: reedy synchronized php-1.20wmf11/includes/api/ApiQueryLogEvents.php
  • 20:51 logmsgbot: reedy synchronized php-1.20wmf11/extensions/TimedMediaHandler/
  • 20:50 logmsgbot: reedy synchronized php-1.20wmf11/extensions/MwEmbedSupport/
  • 19:05 notpeter: reimaging mw1-16
  • 19:00 hashar: jenkins: hashar installed a snapshot of Gerrit Trigger plugin
  • 18:57 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else over to 1.20wmf11
  • 18:47 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki back to 1.20wmf11
  • 18:09 notpeter: adding mw59 to apaches pool
  • 18:03 logmsgbot: reedy synchronized php-1.20wmf10
  • 17:58 logmsgbot: reedy synchronized php-1.20wmf11
  • 17:53 notpeter: re-adding srv194-199 to apaches pool
  • 17:38 notpeter: removing mw1-16 from apaches pool to reimage them as precise jobrunner boxes
  • 14:46 paravoid: restarted swift on ms-fe1-4, proxy-server was leaking memory and had been swapping
  • 02:52 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Wed Sep 12 02:52:12 UTC 2012
  • 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Wed Sep 12 02:27:16 UTC 2012

September 11

  • 22:29 notpeter: removing srv195-199 from apaches pool for upgarde to precise
  • 20:51 RobH: authdns-update per rt 1326
  • 18:31 apergos: second reboot had whines of device not present/not ready for /dev/sdn1 and /dev/sdh1 (note the list is not the same as the previous list), waited a little and it booted up with all disks mounted
  • 18:26 apergos: first reboot of msbe10 showed complaints of not ready on devices sdi, k, n but they did eventually mount when the box came up fully. rebooting a second time to see what happens
  • 18:13 apergos: rebooting ms-be10 after recreation and remount of swift filesystems (except those on ssds), see if they stick
  • 17:55 apergos: new degradedarray event on ms-be6, booooo
  • 17:54 apergos: rebooting ms-bs10 to see if we can get some reasonable device names, in prep for trying the same trick on it as on ms-be6 and 7
  • 17:43 Reedy: Running sync-common on mw8
  • 17:42 Reedy: mw8 seems to have out of date mw files
  • 17:42 apergos: rebooting ms-be7 after recreating all filesystems (except those on the ssds) and remounting them, to see if the disks stay visible
  • 17:38 logmsgbot: reedy synchronized php-1.20wmf11/extensions/ConfirmEdit/FancyCaptcha.class.php
  • 17:31 apergos: puppet disabled on ms-be7 via /etc/default/puppet, swift processes shut down while we test
  • 17:10 apergos: one more reboot of ms-be6 after raid repaired, see if it and all mounts stay ok
  • 15:57 apergos: rebuilding the degraded raid array on ms-be6
  • 15:15 apergos: stopping swift processes on ms-be6 til we decide what's next; all disks came up and were mounted (but we still do have degraded raid array, not so awesome)
  • 15:12 apergos: redid other ms-be6 filesystems so labels would be correct with disk layout without ssds. reboot again to see if it comes up properly
  • 14:44 apergos: manually recreated filesystems and mounted missing drives on ms-be6 (without ssds cabled), seems to have worked without errors. rebooting to see what happens.
  • 14:18 apergos: stopped swift processes on ms-be6 and disabled puppet (/etc/default/puppet) so it won't restart them. still doing testing of hw over there so we need the box up.
  • 04:46 logmsgbot: reedy synchronized php-1.20wmf11/extensions/ConfirmEdit/ '9643ccd3964827651e8a23a19a92ba3d240e60ee'
  • 02:42 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Tue Sep 11 02:41:59 UTC 2012
  • 02:22 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Tue Sep 11 02:22:02 UTC 2012
  • 01:30 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki back to 1.20wmf10
  • 01:02 paravoid: adjusting spence's firewall to include all eqiad subnets

September 10

  • 22:53 Ryan_Lane: adding all current labsconsole users to mediawiki shell group
  • 22:47 logmsgbot: maxsem synchronized php-1.20wmf11/extensions/MobileFrontend/ 'MF update'
  • 22:46 logmsgbot: maxsem synchronized php-1.20wmf10/extensions/MobileFrontend/ 'MF update'
  • 22:40 paravoid: swift: weight 66->100 for ms-be9, 11, 12
  • 21:40 Ryan_Lane: added salt and dependent packages to lucid-wikimedia and precise-wikimedia repos
  • 21:39 logmsgbot: maxsem synchronized php-1.20wmf11/extensions/MobileFrontend/api/ApiMobileView.php 'Bug 39489'
  • 19:42 mutante: push change for glwiki logo to cluster (gerrit 23175)
  • 19:42 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
  • 18:32 RobH: authdns-update for wicipediacymraeg.org
  • 18:01 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 17:08 Jeff_Green: adjusted locke:/etc/udp2log/squid to fix pattern match for fundraiser banners
  • 02:44 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Mon Sep 10 02:44:38 UTC 2012
  • 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Mon Sep 10 02:26:50 UTC 2012

September 9

  • 17:41 logmsgbot: aaron synchronized php-1.20wmf11/includes/upload/UploadFromChunks.php
  • 17:40 logmsgbot: aaron synchronized php-1.20wmf11/includes/upload/UploadFromChunks.php 'deployed 9824650d70a0b2362e35eff3a5ba632d923404f6'
  • 17:39 logmsgbot: aaron synchronized php-1.20wmf11/includes/filerepo/FileRepo.php 'deployed 9824650d70a0b2362e35eff3a5ba632d923404f6'
  • 02:47 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Sun Sep 9 02:47:05 UTC 2012
  • 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Sun Sep 9 02:26:48 UTC 2012
  • 01:13 Jeff_Green: restarted udp2log on locke

September 8

  • 07:54 hashar: restarted Jenkins to fix up a plugin
  • 02:45 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Sat Sep 8 02:45:17 UTC 2012
  • 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Sat Sep 8 02:27:32 UTC 2012

September 7

  • 22:22 mutante: creating new gerrit project for the wikibugs IRC bot as a .deb (operations/debs/wikibugs)
  • 21:43 mutante: for more reference, mw8 is down for DIMM errors (RT-3499)
  • 21:42 RoanKattouw: For reference, the dsh groups Rob removed mw8 from are mediawiki-installation and apaches
  • 21:39 RobH: pulled mw8 out of dsh groups since its down
  • 19:59 cmjohnson1: loading new OS on mw8 after mother board swap
  • 16:02 logmsgbot: hashar synchronized wmf-config/ext-wmflabs.php '22172 - Deploying OnlineStatusBar on beta'
  • 15:21 logmsgbot: py synchronized wmf-config/lucene.php
  • 15:17 notpeter: moving pool2,3,4 and prefix search traffic to eqiad
  • 15:14 logmsgbot: py synchronized wmf-config/lucene.php
  • 15:07 notpeter: moving en search traffic from pmtpa to eqiad
  • 13:26 logmsgbot: hashar: bug 38036 cp /mnt/upload6/wikipedia/favicon-si.ico /mnt/upload6/wiktionary/favicon-si.ico
  • 11:07 mark: Sending Canadian upload traffic to upload-lb.eqiad (Varnish)
  • 07:12 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'change cortado location back again'
  • 06:59 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'change cortado location'
  • 04:26 Tim: professor has stopped responding to ssh and is not reporting to ganglia, rebooting
  • 02:37 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Fri Sep 7 02:37:41 UTC 2012
  • 02:19 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Fri Sep 7 02:19:43 UTC 2012
  • 01:02 AaronSchulz: Started copying math/ files into swift
  • 00:58 logmsgbot: aaron synchronized wmf-config/CommonSettings.php
  • 00:49 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Removed excess math global'
  • 00:28 logmsgbot: aaron synchronized wmf-config/ 'Write math files to swift and NFS'
  • 00:19 logmsgbot: aaron synchronized wmf-config/filebackend.php 'deployed 5aa6ab06fd4e0e76bd6a64e468c9eacb0f46f680'
  • 00:06 Ryan_Lane: remove cn from uniqueness check in LDAP. it's a global check and causes problems
  • 00:04 RobH: authdns-update to support wicipediacymraeg.org

September 6

  • 23:23 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php '...and anonnotice too (5f7cdae1da72895bcdb485ee7983944099f5af15)'
  • 23:18 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'wgForceUIMsgAsContentMsg for bewikimedia (c1ef8fc89889a9802845c5f73381285a75a79eb3)'
  • 21:56 logmsgbot: kaldari Finished syncing Wikimedia installation... :
  • 21:03 logmsgbot: kaldari Started syncing Wikimedia installation... :
  • 19:53 ottomata: stopping puppet on oxygen. (puppet upgraded udp-filter before I was ready, I have to make sure ip filtering still works before I can turn it back on)
  • 19:38 logmsgbot: reedy synchronized php-1.20wmf11/includes/media/FormatMetadata.php
  • 18:40 RobH: authdns-update for ersch ip and removing old decom servers from pdns templates
  • 18:29 RobH: installing ersch as poolcounter server
  • 18:17 notpeter: stopping search indexing on searchidx1001 and searchidx2 to sync to eqiad
  • 17:49 RoanKattouw: Pushing master of operations/mediawiki-config.git into gerrit from fenari
  • 17:39 logmsgbot: aaron synchronized php-1.20wmf11/includes/filebackend/SwiftFileBackend.php 'deployed 3b2518d0284267100a61445b5d811940c8891395'
  • 17:39 logmsgbot: aaron synchronized php-1.20wmf11/includes/filebackend/FileBackend.php 'deployed 3b2518d0284267100a61445b5d811940c8891395'
  • 16:14 mutante: srv219 - apt-get auto-remove unneeded packages, running puppet
  • 16:11 mutante: srv219 - apt-get -f install, removing cm-super, dist-upgrading,
  • 16:05 mark: Sending upload traffic from Argentina to upload-lb.eqiad (Varnish with streaming and persistence patches)
  • 14:55 cmjohnson1: ms-be6 going down to remove ssd drives
  • 14:23 notpeter: reimaging search1001, 1002, 1003
  • 02:45 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Thu Sep 6 02:45:50 UTC 2012
  • 02:28 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Thu Sep 6 02:28:18 UTC 2012
  • 01:39 mutante: renaming mailing list chaptercommittee-l to affcom , rebuilding archives...
  • 01:27 Ryan_Lane: enabling $wgEmailConfirmToEdit on labsconsole
  • 01:21 Ryan_Lane: enabling ConfirmEdit with FancyCaptcha on labsconsole
  • 01:09 Ryan_Lane: upgrading OATHAuth to 795cef09cab6ecb0e9ded35df06b2877ccc22c1a on labsconsole
  • 00:19 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable LQT on lqtlabs'
  • 00:19 Ryan_Lane: added SpamBlacklist to labsconsole
  • 00:12 Ryan_Lane: added scribunto to labsconsole
  • 00:12 RoanKattouw: Added rev_sha1 to revision table on liquidthreads_labswikimedia
  • 00:09 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'and wgCanonicalServer too'
  • 00:07 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Add wgServer for liquidthreads_labswikimedia'

September 5

  • 23:59 logmsgbot: catrope rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 23:29 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php '$wgDisplayTitle=false; on bewikimedia (114ff229f4eb15c1dc88b6da238c7aeea987c025)'
  • 23:21 RobH: done with local hacks on brewster, resuming puppet
  • 23:19 logmsgbot: catrope synchronized php-1.20wmf11/extensions/VisualEditor 'Update VisualEditor (55b78f34db13c0abd43bcdcc8798a8c3425f14ba)'
  • 23:06 RobH: stopping puppet on brewster for local hack testing on ms-be6
  • 23:00 logmsgbot: reedy synchronized php-1.20wmf11/includes/api/ApiQuery.php
  • 22:36 mutante: renaming mailing list "wsor" to "wmfresearch"
  • 22:34 RobH: blog update successful, all plugins also updated.
  • 22:31 RobH: blog software is outdated by a few versions, updating.
  • 22:12 apergos: shutdown ms-be6, prep for puppet changes for reinstall w/o ssds
  • 22:09 Ryan_Lane: pulling in change 22690 to labsconsole
  • 20:45 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 20:30 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 20:15 logmsgbot: reedy synchronized php-1.20wmf11/includes/media/FormatMetadata.php
  • 20:12 logmsgbot: aaron synchronized wmf-config/filebackend.php 'deployed c48c6c189ce576186457bb916f42f56e9d35d6f6'
  • 20:11 Jeff_Green: rebooting storage3
  • 19:57 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commonswiki back to 1.20wmf11
  • 19:56 mutante: added new Nagios monitor to Watchmouse
  • 19:53 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 19:43 mutante: deleting Ben as a Watchmouse contact
  • 19:41 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Unbreak the site'
  • 19:39 logmsgbot: catrope synchronized php-1.20wmf11/includes/ImagePage.php 'Fix for bug 40018 (ac25a9d1127848269b6c9766262c23f961b35c5b)'
  • 19:39 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 19:38 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2 back to wmf11
  • 19:36 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2 back to wmf10 for a few mins..
  • 19:23 logmsgbot: krinkle synchronized docroot/default/index.html 'Ia7ca3eaf'
  • 19:23 logmsgbot: krinkle synchronized docroot/default/wmf_logo_with_text.png 'Ia7ca3eaf'
  • 19:00 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'revert all back to old cortado.jar'
  • 18:59 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 18:56 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 18:48 logmsgbot: reedy synchronized wmf-config
  • 18:40 Reedy: commonswiki to stay on 1.20wmf10 for the meantime
  • 18:38 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commonswiki back to 1.20wmf10 for testing
  • 18:32 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Go back to older cortado to check thumbnail isue'
  • 18:27 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Use an updated version of cortado for testwiki and test2wiki'
  • 18:25 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else non wikipedia to 1.20wmf11
  • 18:22 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote and wiktionary to 1.20wmf11
  • 18:19 logmsgbot: reedy synchronized php-1.20wmf11/includes/Timestamp.php '0000...'
  • 18:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews, wikiversity and wikimedia to 1.20wmf11
  • 18:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Misc wikis to 1.20wmf11
  • 17:52 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Enable CORS for the API (347f386e2144688bcd4c80d24bd9f3be7d135d74)'
  • 17:51 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable CORS for the API (347f386e2144688bcd4c80d24bd9f3be7d135d74)'
  • 17:28 logmsgbot: aaron synchronized php-1.20wmf11/thumb.php 'deployed 8a63ff8c8558b1ca9f2265da71376e494053c234'
  • 17:28 logmsgbot: aaron synchronized php-1.20wmf11/includes/filerepo/file/File.php 'deployed 8a63ff8c8558b1ca9f2265da71376e494053c234'
  • 17:27 logmsgbot: aaron synchronized php-1.20wmf11/includes/StreamFile.php 'deployed 8a63ff8c8558b1ca9f2265da71376e494053c234'
  • 15:54 Jeff_Green: updated MOTD for OTRS.
  • 15:01 cmjohnson1: search32 still borked ..going down for troubleshooting
  • 14:48 logmsgbot: reedy synchronized php-1.20wmf11/extensions/CheckUser 'CheckUser to master'
  • 04:10 apergos: running d/d8 container consistency check fr ms7 from bast1001 in screen session as ariel
  • 02:51 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Wed Sep 5 02:51:43 UTC 2012
  • 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Wed Sep 5 02:26:57 UTC 2012
  • 02:17 RobH: nagios crashed, fixing
  • 01:05 Ryan_Lane: enabling DynamicSidebar extensions on labsconsole

September 4

  • 23:34 AaronSchulz: Started copying timeline/ files into swift
  • 23:34 logmsgbot: reedy synchronized wmf-config/
  • 23:30 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Write timeline files to swift and NFS'
  • 22:45 logmsgbot: catrope synchronized php-1.20wmf11/extensions/VisualEditor/ 'Update VisualEditor (59101ea3c93021c22da9796bf1715603967693fe)'
  • 22:43 apergos: running heads on commons d/d7 on bast1001 as ariel from screen, data consistency check
  • 22:07 notpeter: adding srv266 and srv281 back to apaches pool (to see if they break...)
  • 21:04 logmsgbot: py synchronized wmf-config/lucene.php
  • 21:01 notpeter: moving en search traffic back to pmtpa
  • 20:46 logmsgbot: py synchronized wmf-config/lucene.php
  • 20:45 notpeter: moving en search back to eqiad
  • 20:41 Ryan_Lane: added oliver mail alias for okeyes
  • 20:37 cmjohnson1: srv291 shutting down for troubleshooting
  • 19:09 apergos: swift consistency check (heads of various objects) running from bastion1001 as ariel in screen session, about 10 HEADS/sec, if this starts to impact swift just shoot it
  • 18:38 cmjohnson1: srv281 going down for troubleshooting
  • 17:54 RobH: helium being reinstalled to poolcounter server per RT#3407
  • 17:41 Ryan_Lane: pulled latest update of branch 1.20wmf10 on labsconsole
  • 17:22 cmjohnson1: search32 shutting down to replace processor
  • 16:26 notpeter: temp stopping puppet on brewster
  • 16:26 notpeter: reimaging searchidx1001
  • 15:26 logmsgbot: reedy synchronized wmf-config/ 'Staging for subpagesortkey'
  • 15:24 logmsgbot: reedy synchronized wmf-config/
  • 15:22 logmsgbot: reedy synchronized php-1.20wmf11/extensions/SubpageSortkey/
  • 14:29 logmsgbot: reedy synchronized php-1.20wmf11/resources/startup.js 'touch'
  • 14:28 logmsgbot: reedy synchronized php-1.20wmf10/resources/startup.js 'touch'
  • 14:27 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'touch'
  • 14:23 logmsgbot: reedy synchronized php-1.20wmf10/resources/jquery/jquery.js 'Revert back to 1.7.2 for wmf10 wikis'
  • 14:19 logmsgbot: reedy synchronized php-1.20wmf11/resources/jquery/jquery.js 'Revert back to 1.7.2 for wmf11 wikis'
  • 12:55 mark: Pushed varnish 3.0.3plus-rc1 packages into the precise-wikimedia APT repository
  • 02:51 paravoid: powering down ms-be6, broken hardware
  • 02:39 logmsgbot: LocalisationUpdate completed (1.20wmf11) at Tue Sep 4 02:39:34 UTC 2012
  • 02:24 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Tue Sep 4 02:24:01 UTC 2012
  • 01:28 logmsgbot: tstarling synchronized wmf-config/flaggedrevs.php

September 3

  • 18:34 logmsgbot: reedy synchronized live-1.5/
  • 18:26 Reedy: Deleting php-1.20wmf8 from mediawiki-installation
  • 18:25 Reedy: Deleting 1.20wmf9 l10n cache from mediawiki-installation
  • 18:22 Reedy: Repointed php symlink from wmf9 to wmf10
  • 18:12 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.20wmf11
  • 17:48 logmsgbot: reedy Finished syncing Wikimedia installation... : rebuild 1.20wmf11 localisation cache
  • 17:42 paravoid: powering down mw8; broken DIMM (#3499), has been flapping for days
  • 17:32 paravoid: killing power via ipmi on ms-be7; broken hardware, stale data, should not be on rotation
  • 17:18 logmsgbot: reedy Started syncing Wikimedia installation... : rebuild 1.20wmf11 localisation cache
  • 17:16 paravoid: swift: distributing new ring; disable ms-be7, ms-be8, ms-be11 sdi, increase ms-be9/11/12 weight from 31 to 66
  • 17:06 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.20wmf11 and rebuild localisation cache
  • 15:59 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.20wmf11 and rebuild localisation cache
  • 15:44 logmsgbot: reedy synchronized php-1.20wmf11/cache/
  • 15:44 logmsgbot: reedy synchronized wmf-config/
  • 15:42 Reedy: sync-docroot
  • 15:41 logmsgbot: reedy synchronized php-1.20wmf11 'Initial sync-out'
  • 15:09 notpeter: reinstalling all eqiad search frontends as precise
  • 13:55 hashar: Sent one live hack made to 1.20wmf10 : https://gerrit.wikimedia.org/r/#/c/22461/
  • 13:52 hashar: Rebased local 1.20wmf10 directory on top of its current origin/1.20wmf10 ( 472ec09 )
  • 13:50 hashar: installed Gerrit commitmsg hook in /h/w/c/php-1.20wmf10/.git/hooks
  • 13:36 logmsgbot: reedy synchronized php-1.20wmf10/resources/jquery/jquery.js 'jQuery 1.8.1'
  • 09:43 hashar: manually cleaning out /tmp on gallium. It is being filled by faulty tests not cleaning the files they are creating.
  • 02:18 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Mon Sep 3 02:18:28 UTC 2012

September 2

  • 21:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 02:25 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Sun Sep 2 02:25:51 UTC 2012

September 1

  • 02:49 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Deploy 22ceabed668b96cd88b48f6f7482c86a53645fb2'
  • 02:49 logmsgbot: catrope synchronized wmf-config/filebackend.php 'Deploy 988ea2f71d84e1f62269f6e3d72eaf7b03206906'
  • 02:49 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Deploy 988ea2f71d84e1f62269f6e3d72eaf7b03206906'
  • 02:25 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Sat Sep 1 02:25:35 UTC 2012

August 31

  • 22:04 Ryan_Lane: removing attribute uid from the attribute uniqueness plugin. It's the naming attribute and it's already ensured to be unique. It causes problems when it's also in the plugin.
  • 21:54 Ryan_Lane: enabling attribute uniqueness plugin on virt0 and virt1000 for attributes: cn, uid, uidNumber
  • 21:51 mutante: running rm -rf /var/lib/testswarm on gallium in a screen ..takes forever
  • 21:51 Ryan_Lane: adding equality index to uidNumber on virt0 and virt1000 opendj
  • 21:35 mutante: ms-be6 - just for the record, still could not mount same disks (sdn1, sdl1, sdi1, sdf1) after firmware upgrade
  • 20:32 cmjohnson1: ms-be6 down for firmware upgrade
  • 20:28 mutante: deleting /var/lib/testswarm on gallium (testswarm is gone)
  • 12:11 logmsgbot: hashar synchronized wmf-config/CommonSettings.php '5c84fca $wgDBerrorLogInUTC -> $wgDBerrorLogTZ'
  • 12:11 logmsgbot: hashar synchronized live-1.5/robots.php 'f0bf7a5 - disallow robots on beta'
  • 12:11 hashar: forgot to pull before sync :-)
  • 12:11 logmsgbot: hashar synchronized wmf-config/CommonSettings.php '5c84fca $wgDBerrorLogInUTC -> $wgDBerrorLogTZ'
  • 12:10 logmsgbot: hashar synchronized live-1.5/robots.php 'f0bf7a5 - disallow robots on beta'
  • 04:02 logmsgbot: tstarling synchronized php-1.20wmf10/tests/parser/parserTests.txt
  • 04:01 logmsgbot: tstarling synchronized php-1.20wmf10/includes/specials/SpecialBlock.php
  • 04:01 logmsgbot: tstarling synchronized php-1.20wmf10/includes/api/ApiFormatBase.php
  • 04:00 logmsgbot: tstarling synchronized php-1.20wmf10/includes/Linker.php
  • 04:00 logmsgbot: tstarling synchronized php-1.20wmf10/includes/User.php
  • 04:00 logmsgbot: tstarling synchronized php-1.20wmf10/includes/AuthPlugin.php
  • 03:59 logmsgbot: tstarling synchronized php-1.20wmf10/includes/DefaultSettings.php
  • 02:59 paravoid: swift: removing ms-be8 sdc1, sdg1, sdk1, sdl1, sdn1 from ring; broken hardware (again)
  • 02:18 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Fri Aug 31 02:18:02 UTC 2012

August 30

  • 22:54 logmsgbot: maxsem synchronized php-1.20wmf10/includes/api/ApiMain.php 'WLM API logging https://gerrit.wikimedia.org/r/#/c/21748/'
  • 22:27 Ryan_Lane: ran a maintenance script on labsconsole to update all instance pages
  • 22:05 logmsgbot: kaldari Finished syncing Wikimedia installation... :
  • 21:45 logmsgbot: kaldari Started syncing Wikimedia installation... :
  • 21:17 notpeter: removing srv194 from apache pool
  • 21:13 logmsgbot: olivneh synchronized php-1.20wmf10/extensions/E3Experiments/experiments
  • 20:39 cmjohnson1: srv291 shutting down to reseat DIMM
  • 20:39 cmjohnson1: srv291 shutting down to reseat DIMM
  • 20:28 logmsgbot: krinkle synchronized live-1.5/static-1.20wmf10 'Creating static aliases for wmf10'
  • 20:25 Krinkle: Created static-1.20wmf10 and extensions,resources,skins subdir symlinks in /home/wikipedia/commons/live-1.5, was not done by the script apparently
  • 20:07 logmsgbot: reedy synchronized php-1.20wmf10/resources/jquery
  • 20:02 mutante: reinstalling cp1024-cp1028
  • 19:28 Ryan_Lane: finished upgrading labsconsole to 1.20wmf10
  • 19:17 notpeter: all search traffic now pointed back to pmtpa
  • 19:17 logmsgbot: py synchronized wmf-config/lucene.php
  • 18:58 Ryan_Lane: upgrading labsconsole to 1.20wmf10
  • 18:57 logmsgbot: py synchronized wmf-config/lucene.php
  • 18:41 paravoid: deploying squid config all for thumb_handler.php addition to thumb_php
  • 18:31 logmsgbot: preilly synchronized wmf-config/lucene.php
  • 17:47 logmsgbot: aaron synchronized php-1.20wmf10/includes/filebackend/SwiftFileBackend.php 'deployed faec65ea23a8d17f08488950fccb0911f7b63086'
  • 17:33 apergos: and rebooting ms-be7 again in the slim hope that we get back the missing drives... meh
  • 17:32 logmsgbot: aaron synchronized php-1.20wmf10/includes/filebackend/SwiftFileBackend.php 'deployed 3bc61905b704096789376927b6cc3d7ca78b0670'
  • 17:32 logmsgbot: aaron synchronized php-1.20wmf10/includes/filebackend/FSFileBackend.php 'deployed 3bc61905b704096789376927b6cc3d7ca78b0670'
  • 16:45 notpeter: removing srv281 from apaches pool because it's a broken piece of garbage.
  • 16:23 apergos: power cycle ms-be7 so it picks up the replaced drive with the right device name
  • 15:46 cmjohnson1: mw8 shutting down for DIMM troubleshooting
  • 14:09 RobH: authdns-update for helium/potassium (poolcounter servers)
  • 10:53 hashar: gerrit-wm irc bot is apparently no more sending anything to #mediawiki :( IRC user has been inactive since midnight UTC. Opened bug 39797
  • 03:35 Tim: restarting apaches to get php.ini change of pcre.backtrack_limit
  • 02:18 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Thu Aug 30 02:18:56 UTC 2012
  • 01:36 K4-713: updated the payments cluster to 95907e42e57
  • 01:10 logmsgbot: aaron synchronized wmf-config/filebackend.php 'deployed fix for bogus var'
  • 00:48 mutante: starting nagios on spence

August 29

  • 23:02 logmsgbot: catrope synchronized wmf-config/filebackend.php 'Deploy d5b73b216317c705f58d1a1eef8bc0db21a90314'
  • 21:25 logmsgbot: krinkle synchronized php-1.20wmf10/includes/EditPage.php '(bug 39584) Deploy MediaWiki:Previewnote bug fix (I2144f026d3b301edb521bf)'
  • 21:12 logmsgbot: krinkle synchronized php-1.20wmf10/resources/ 'Update jquery.ui and jquery.effects to jQuery UI 1.8.23 (I35ed086e9a8c6d)'
  • 20:29 Ryan_Lane: changing apache config and wiki location on virt0
  • 20:27 logmsgbot: krinkle synchronized php-1.20wmf10/resources/jquery/jquery.js '(8dc8ee84) jQuery 1.8'
  • 18:41 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: Move last remaining wiki to 1.20wmf10, so long 1.20wmf9
  • 18:38 Ryan_Lane: deleting global role groups from LDAP. They aren't needed in keystone's LDAP DIT
  • 18:00 mutante: reinstalling cp1022 and up
  • 17:56 notpeter: returning srv281 to apaches pool
  • 17:36 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: Switching all 'pedias to 1.20wmf10
  • 17:11 notpeter: removing srv281 once again and reimaging
  • 17:09 notpeter: returning srv281 to apaches pool
  • 17:01 cmjohnson1: swapping out disk7 (dev/sdg) on ms-be7
  • 16:36 logmsgbot: root Finished syncing Wikimedia installation... :
  • 16:28 notpeter: removing srv281 from apaches pool
  • 16:20 logmsgbot: root Started syncing Wikimedia installation... :
  • 16:11 logmsgbot: mark synchronized wmf-config/mc.php
  • 16:10 notpeter: enabling srv194 and srv281 in pybal for apache/precise live test
  • 16:04 mark: Fixed serial console on hume
  • 16:00 mark: Powering up fenari
  • 15:49 mark: Powering up hume
  • 15:48 mark: Final rsync finished.
  • 13:37 mark: Mounted /home read-only on nfs1, started final rsync to nas1-a
  • 13:28 mark: Changed /home mount from nfs1 to nas1-a on srv193 and spence
  • 13:05 mark: Shutdown fenari and hume
  • 12:48 hashar: reloading Jenkins to fix up Ext-TitleBlacklist misconfiguration
  • 12:42 hashar: enabling Jenkins on TitleBacklist extension (job: Ext-TitleBlacklist )
  • 08:55 logmsgbot: nikerabbit synchronized php-1.20wmf9/includes/Linker.php
  • 08:54 logmsgbot: nikerabbit synchronized php-1.20wmf10/includes/Linker.php
  • 02:45 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Wed Aug 29 02:45:36 UTC 2012
  • 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Wed Aug 29 02:27:21 UTC 2012

August 28

  • 23:45 LeslieCarr: there are some network issues with one of our ipv6 carriers, causing some european ipv6 users to have trouble reaching our US sites.
  • 23:17 Ryan_Lane: fixed scripts for management home directories and gluster shares for labs by changing the project search filter to match keystone's LDAP DIT
  • 21:44 logmsgbot: bsitu synchronized php-1.20wmf10/extensions/PageTriage/tools/cleanupPageTriagePageTags.php 'Fix the cleanup cron'
  • 21:06 K4-713: updated the payments cluster to 9bcd8961fecb5
  • 20:59 mutante: git pull and sync-file InitialiseSettings.php to cluster to push namespace fix on ar.wikimedia
  • 20:59 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
  • 20:52 logmsgbot: mlitn Finished syncing Wikimedia installation... : Updating ArticleFeedbackv5
  • 20:16 logmsgbot: mlitn Started syncing Wikimedia installation... : Updating ArticleFeedbackv5
  • 19:55 logmsgbot: catrope synchronized php-1.20wmf10/extensions/ProofreadPage 'Deploy 65fcb5984fd7d52ecd1887b248fda608b9dc4987'
  • 19:53 logmsgbot: catrope synchronized php-1.20wmf10/tests/phpunit/includes/RevisionStorageTest.php 'Deploy e1cfe5ada99f3ed4594c5cfc33906538f4e02bdf'
  • 18:52 mutante: rebooting cp1021 into PXE
  • 17:41 paravoid: include debs for swift 1.5 and friends to apt (lucid/precise)
  • 16:15 Jeff_Green: added payments100[1-4] & pay-lvs100[12] to nagios nsca config, restarted nagios on spence
  • 14:03 logmsgbot: nikerabbit synchronized wmf-config/CommonSettings.php 'ttmserver part 2'
  • 14:02 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'ttmserver part 1'
  • 13:14 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Syncing 95c87a0b for AFT (mlitn said its ok)'
  • 13:13 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Syncing f58063db'
  • 13:11 logmsgbot: demon synchronized .gitignore 'Syncing .gitignore'
  • 12:29 mark: Built varnish 3.0.3plus~rc1 packages and deployed them on cp1041 (mobile)
  • 10:02 logmsgbot: nikerabbit synchronized php-1.20wmf10/extensions/Translate/ 'Picking up Solarium'
  • 09:23 logmsgbot: nikerabbit synchronized php-1.20wmf10/extensions/wikihiero/ 'RTL fix'
  • 04:01 Ryan_Lane: rebooting virt5
  • 03:56 Ryan_Lane: rebooting virt4
  • 03:41 Ryan_Lane: upgrading virt1 to precise
  • 03:38 Ryan_Lane: upgrading virt4 to precise
  • 03:37 Ryan_Lane: upgrading virt5 to precise
  • 03:06 Ryan_Lane: bringing virt7 instances up
  • 02:48 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Tue Aug 28 02:48:54 UTC 2012
  • 02:45 Ryan_Lane: bringing virt8 instances up
  • 02:37 logmsgbot: tstarling Finished syncing Wikimedia installation... :
  • 02:30 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Tue Aug 28 02:30:01 UTC 2012
  • 02:24 Ryan_Lane: force running puppet on virt7
  • 02:24 Ryan_Lane: force running puppet on virt8
  • 02:17 Ryan_Lane: rebooting virt8
  • 02:17 Ryan_Lane: rebooting virt 7
  • 02:08 logmsgbot: tstarling Started syncing Wikimedia installation... :
  • 01:51 Ryan_Lane: upgrading virt8 to precise
  • 01:48 logmsgbot: tstarling synchronized php-1.20wmf10/includes/GlobalFunctions.php
  • 01:48 logmsgbot: tstarling synchronized php-1.20wmf10/thumb.php
  • 01:47 logmsgbot: tstarling synchronized php-1.20wmf10/includes/upload/UploadFromUrl.php
  • 01:46 Ryan_Lane: upgrading virt7 to precise
  • 01:32 Ryan_Lane: bringing all instances up on virt6
  • 01:01 Ryan_Lane: attempting to bring up some instances on virt6

August 27

  • 23:31 Ryan_Lane: restarting virt6
  • 23:08 Ryan_Lane: upgrading virt6 to precise. This action will cause downtime for a while.
  • 22:49 mutante: fixed gmq.planet.wm updates
  • 22:34 Ryan_Lane: rebooting virt3
  • 22:24 Ryan_Lane: upgrading virt3 to precise
  • 22:23 Ryan_Lane: disabling nova-compute on virt2
  • 21:01 dschoon: ${whoami}
  • 21:00 Ryan_Lane: enabling OpenStackManager and LdapAuthentication on labsconsole temporarily for testing
  • 21:00 Ryan_Lane: enabling nova-compute on virt2 for testing instance creation
  • 20:47 ottomata: starting puppet back up on emery
  • 20:32 logmsgbot: aaron synchronized php-1.20wmf9/extensions/MobileFrontend 'deployed f2551fa97ba686101745d5cb97ef4678838cd3bb'
  • 20:29 logmsgbot: aaron synchronized php-1.20wmf10/extensions/MobileFrontend 'deployed 75104cf6d3bc6d528d2b85e09761005081fb5cf4'
  • 20:20 paravoid: deploying swift origs to all pmtpa backend squids
  • 20:09 Ryan_Lane: force running puppet on virt2
  • 20:04 Ryan_Lane: rebooting virt2
  • 19:54 paravoid: deploying swift origs on sq51 (1h ago), sq52, sq53
  • 19:54 Ryan_Lane: upgrading virt2 to precise
  • 19:38 Ryan_Lane: upgrading OpenStackManager on labsconsole
  • 19:33 Ryan_Lane: adding keystone services and endpoints
  • 19:32 Ryan_Lane: upgrading nova, glance and keystone database schemas
  • 19:18 ottomata: temporarily stopping puppet on emery to test log2udp AFT clicktracking relay to vanadium
  • 19:06 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf10
  • 18:41 Ryan_Lane: force running puppet on virt0
  • 18:36 Ryan_Lane: restarting virt0
  • 18:09 Ryan_Lane: upgrading virt0 to precise
  • 17:54 Ryan_Lane: running a fresh backup of opendj on virt0
  • 17:54 Ryan_Lane: backing up all databases on virt0
  • 17:46 Ryan_Lane: stopping rabbitmq on virt0
  • 17:43 Ryan_Lane: disabling openstack services on the virt infrastructure, that is
  • 17:43 Ryan_Lane: disabling all openstack services
  • 17:42 Ryan_Lane: disabling OpenStackManager and LdapAuthentication on labsconsole
  • 17:37 Ryan_Lane: disabling puppet on all virt nodes
  • 17:33 paravoid: rolled back sq51's config; depooled sq51 (10' ago)
  • 17:18 paravoid: modified sq51 to serve origs from swift; added sq51 back to the pool
  • 16:45 paravoid: depooling sq51 backend
  • 15:55 paravoid: upgraded swift on ms-be12
  • 15:54 paravoid: upgraded swift on ms-be5
  • 15:47 paravoid: upgraded swift on ms-be11
  • 14:51 paravoid: upgraded swift on ms-be3
  • 14:40 paravoid: upgraded swift on ms-be4
  • 14:30 paravoid: upgraded swift on ms-be10; (still depooled, no effect)
  • 09:38 mark: Decommissioning bayes, shut it down
  • 02:52 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Mon Aug 27 02:52:23 UTC 2012
  • 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Mon Aug 27 02:27:16 UTC 2012
  • 02:16 Tim: on spence: fixed /apache symlink, was pointing to a non-standard location, breaking sync scripts
  • 01:32 logmsgbot: tstarling synchronized php-1.20wmf10/extensions/Scribunto/engines/LuaCommon/LuaCommon.php
  • 01:32 logmsgbot: tstarling synchronized php-1.20wmf10/extensions/Scribunto/Scribunto.i18n.php
  • 01:31 Tim: updating Scribunto
  • 01:31 logmsgbot: tstarling synchronized php-1.20wmf10/extensions/Scribunto/Scribunto.namespaces.php

August 26

  • 02:53 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Sun Aug 26 02:53:09 UTC 2012
  • 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Sun Aug 26 02:27:02 UTC 2012

August 25

  • 02:46 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Sat Aug 25 02:46:16 UTC 2012
  • 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Sat Aug 25 02:27:08 UTC 2012

August 24

  • 23:08 notpeter: temp stopping puppet on iron
  • 23:02 maplebed: upgrading swift backend ms-be9
  • 22:15 maplebed: upgrading swift backend ms-be7
  • 21:54 maplebed: upgrading swift back end ms-be2
  • 21:30 mutante: removing decom'ed srv206 and srv217 from ALL dsh groups
  • 21:24 maplebed: upgrading swift backend host ms-be1
  • 21:22 mutante: breaking test.wp :) ..and rolling back
  • 21:11 maplebed: upgraded swift backend host ms-be8
  • 20:10 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php (bug 34817)
  • 19:57 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
  • 19:01 RoanKattouw: (That was me)
  • 19:01 logmsgbot: Restarted logmsgbot, was broken again
  • 19:00 ottomata: starting puppet back up on brewster
  • 18:58 mutante: sync-file ./wmf-config/InitialiseSettings.php
  • 18:57 mutante: git pull in /h/w/common/wmf-config
  • 18:32 ottomata: temporarily stopping puppet on brewster to test out partman change for analytics1023-1027
  • 14:57 RobH: restart puppeted on brewster
  • 14:48 RobH: analytics1023-1027 rotating down for removal of extra nic
  • 14:28 RobH: stopping puppet on brewster to do a local nonpuppetized test change
  • 13:08 paravoid: repooling ms-fe1/2 with all new swift
  • 12:40 apergos: on ms5, running from screen session as root: tossing non-standard thumb sizes > 100 px for commons/x/xx to see what space that gives us
  • 12:32 paravoid: depooling ms-fe1/2 for the 1.5 upgrade
  • 02:47 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Fri Aug 24 02:47:46 UTC 2012
  • 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Fri Aug 24 02:26:10 UTC 2012
  • 02:08 logmsgbot: kaldari Finished syncing Wikimedia installation... :
  • 01:55 logmsgbot: kaldari Started syncing Wikimedia installation... :

August 23

  • 23:56 logmsgbot: kaldari synchronized php-1.20wmf10/extensions/UploadWizard
  • 23:49 K4-713: updated paypal IPN to r2028
  • 22:40 maplebed: putting ms-fe3 and 4 into rotation running swift v1.5.0-3
  • 22:40 K4-713: updated production civicrm to r2024
  • 22:06 logmsgbot: kaldari Finished syncing Wikimedia installation... :
  • 21:53 logmsgbot: kaldari Started syncing Wikimedia installation... :
  • 21:33 logmsgbot: aaron synchronized php-1.20wmf10/extensions/SwiftCloudFiles 'deployed de4a83af7750750fcaa49e81b7faa771c8601546'
  • 21:32 logmsgbot: aaron synchronized php-1.20wmf9/extensions/SwiftCloudFiles 'deployed de4a83af7750750fcaa49e81b7faa771c8601546'
  • 20:42 logmsgbot: olivneh synchronized php-1.20wmf9/extensions/E3Experiments/experiments/trackEdits.js
  • 20:40 logmsgbot: olivneh synchronized php-1.20wmf9/extensions/E3Experiments/lib/e3.js
  • 20:35 logmsgbot: kaldari Finished syncing Wikimedia installation... :
  • 20:12 logmsgbot: kaldari Started syncing Wikimedia installation... :
  • 18:44 maplebed: rebooting ms-be3
  • 18:34 apergos: deleting about 200k thumb dirs and their contents from ms5, unused on any project, covering june and july uploads
  • 18:06 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Disable auth caching for testwiki'
  • 17:40 logmsgbot: aaron synchronized php-1.20wmf10/includes/filebackend/FSFileBackend.php 'debug logging.'
  • 17:21 logmsgbot: aaron synchronized wmf-config/PrivateSettings.php 'Make testwiki use ms-fe4'
  • 17:09 logmsgbot: aaron synchronized wmf-config/filebackend.php
  • 16:54 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Only write thumbnails to the master backend.'
  • 16:41 logmsgbot: aaron synchronized php-1.20wmf10/includes/filebackend/FileBackendMultiWrite.php 'deployed 36c0c1edb6e713a797e7fde52f716ca17f548866'
  • 16:40 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/backend/FileBackendMultiWrite.php 'deployed 36c0c1edb6e713a797e7fde52f716ca17f548866'
  • 11:46 paravoid: depooling ms-fe3 to stage 1.5 upgrade
  • 11:27 paravoid: depooling ms-fe4 to stage 1.5 upgrade
  • 04:21 logmsgbot: aaron synchronized php-1.20wmf9/includes/media/MediaTransformOutput.php 'deployed ed9d124c37f57ab7d7c9678908af54363290b1f4'
  • 04:20 logmsgbot: aaron synchronized php-1.20wmf10/includes/media/MediaTransformOutput.php 'deployed ed9d124c37f57ab7d7c9678908af54363290b1f4'
  • 04:12 logmsgbot: aaron synchronized php-1.20wmf10/includes/media/MediaTransformOutput.php
  • 03:49 logmsgbot: aaron synchronized php-1.20wmf10/includes/StreamFile.php
  • 03:38 logmsgbot: aaron synchronized php-1.20wmf10/includes/StreamFile.php 'debug logging'
  • 02:53 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Thu Aug 23 02:53:10 UTC 2012
  • 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Thu Aug 23 02:27:31 UTC 2012
  • 00:04 logmsgbot: asher synchronized wmf-config/db.php 'returning db35 to s5'

August 22

  • 23:54 binasher: rebooting db35 for kernel upgrade
  • 23:50 binasher: new s5 master repl pos - MASTER_LOG_FILE='db45-bin.000187', MASTER_LOG_POS=807116168
  • 23:49 logmsgbot: asher synchronized wmf-config/db.php 'setting s5 to writeable'
  • 23:47 logmsgbot: asher synchronized wmf-config/db.php 'setting s5 to read-only and switching master to db45, return db39 to s3'
  • 23:47 binasher: preparing to rotate the s5 master to db45 (last of these for now)
  • 23:36 mutante: check_job_queue check fixed after Tim synced spence. zhwiki has large queue: zhwiki (118614). Nagios did not talk because it was a transition from one critical to another
  • 23:26 Tim: added spence to the mediawiki-installation dsh group
  • 23:25 binasher: rebooting db39 for kernel upgrade
  • 23:19 binasher: new s3 master repl pos: MASTER_LOG_FILE='db34-bin.000864', MASTER_LOG_POS=466194762
  • 23:18 logmsgbot: asher synchronized wmf-config/db.php 'setting s3 to writeable'
  • 23:16 logmsgbot: asher synchronized wmf-config/db.php 'setting s3 to read-only, switching master to db34'
  • 23:15 binasher: preparing to rotate the s3 master to db34
  • 23:14 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Restored math directory to its intended value.'
  • 23:13 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'Restored math directory to its intended value.'
  • 23:06 K4-713: updated production civicrm to r2013
  • 22:42 mutante: rebooting srv208,srv238
  • 22:35 mutante: re-enabling puppet on srv208 and srv238, dist-upgrading while at it
  • 22:26 K4-713: updated production civicrm to r2011
  • 22:10 mutante: srv208,srv238 - puppet running but it's not. looks like puppet is disabled using puppet agent --disable, on purpose?
  • 21:39 logmsgbot: catrope synchronized php-1.20wmf10/resources 'jQuery revert'
  • 21:03 mutante: adding Bugzilla https monitor to watchmouse
  • 20:53 mutante: dist-upgrading kaulen, new kernel (not rebooted yet), fixed SSL cert path in puppet, restart puppet
  • 20:37 mutante: stopped puppet on kaulen. manually fixed path to SSL cert so Apache would start. Bugzilla back for now
  • 20:32 mutante: powercycling kaulen. OOM death.
  • 20:29 mutante: kaulen unresponsive, bugzilla affected
  • 20:26 logmsgbot: aaron synchronized php-1.20wmf9/extensions/SwiftCloudFiles/php-cloudfiles-wmf/cloudfiles.php 'deployed 8fea21b10eaba28773c30c6952fc792f08bf09a8'
  • 20:24 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Re-enabled swift token caching.'
  • 20:20 logmsgbot: aaron synchronized php-1.20wmf10/extensions/SwiftCloudFiles/php-cloudfiles-wmf/cloudfiles.php 'deployed 8fea21b10eaba28773c30c6952fc792f08bf09a8'
  • 19:17 logmsgbot: aaron synchronized php-1.20wmf10/includes/Revision.php
  • 19:13 logmsgbot: aaron synchronized php-1.20wmf10/includes/Revision.php 'bit logic is fun'
  • 19:11 logmsgbot: aaron synchronized php-1.20wmf10/includes/Revision.php 'debug logging'
  • 19:08 mutante: reinstalling zirconium to use LVM
  • 19:05 logmsgbot: aaron synchronized php-1.20wmf10/includes/Revision.php 'debug logging'
  • 18:55 logmsgbot: Restarted ircecho on fenari, logmsgbot was broken --Roan
  • 18:52 RoanKattouw: 1.20wmf10 deployed to all non-Wikipedia projects around 20:04 UTC
  • 17:44 mutante: installing package upgrades on bast1001
  • 17:19 paravoid: stopped supervisord & puppet on silver/zhen per preilly's request
  • 17:00 apergos: oh, for people following along, shot the delete job. apparently even deletes is enough to push ms5 over the edge, yay
  • 16:50 apergos: removing thumbs on ms5 unused on any project, for june-july uploads. running in screen as root
  • 16:37 ^demon: bugzilla went down, but came back up. magic?
  • 16:35 Jeff_Green: restarted ns1 b/c it fell down after authdns-update
  • 16:33 Jeff_Green: adding dns for pay-lvs100[12]
  • 16:32 paravoid: deploy new squid config that removes ms5 from a swift fallback for thumbs
  • 16:06 cmjohnson1: ms-be6 powering down for memory testing
  • 15:32 notpeter: rebooting srv193, upgrade to precise
  • 11:38 mark: Removed /homewmf NFS mount on kaulen
  • 06:14 binasher: new s1 repl position: MASTER_LOG_FILE='db63-bin.000049', MASTER_LOG_POS=646903534
  • 06:14 logmsgbot: asher synchronized wmf-config/db.php 'returing db38 as an s1 slave'
  • 06:02 binasher: shutting down mysql on db38 and rebooting for kernel upgrade
  • 05:50 logmsgbot: asher synchronized wmf-config/db.php 'setting s1 writeable on new master'
  • 05:49 logmsgbot: asher synchronized wmf-config/db.php 'setting s1 to read-only, switching master to db63'
  • 05:47 binasher: preparing to switch s1 master to db63 (upgrade to 96GB ram and precise) in a minute
  • 05:29 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'enabling Scribunto on mediawiki.org'
  • 05:07 Tim: on ms7, running a script to remerge the split math directories back into /mnt/upload6/math, using rsync
  • 02:55 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Wed Aug 22 02:55:22 UTC 2012
  • 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Wed Aug 22 02:27:55 UTC 2012
  • 00:19 Tim: running updateCollation.php for ptwiki in a screen on hume
  • 00:15 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'setting $wgCategoryCollation=uca-default on ptwiki'
  • 00:00 binasher: completed user.user_email index migrations

August 21

  • 23:46 maplebed: deployed change to swift proxy rewrite to only send thumb 404s to the image scalers gerrit change 20916
  • 23:44 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Enabled token/url caching for swift.'
  • 23:26 Tim: updated email address for enwiki user account "Giano"
  • 23:17 binasher: running user.user_email index migration on s6-7
  • 23:16 binasher: running user.user_email index migration on s5
  • 23:14 binasher: truncated enwiki.flagged(images|templates)
  • 23:10 binasher: running user.user_email index migration on s4
  • 22:44 logmsgbot: asher synchronized wmf-config/db.php 'returning db52 as an s2 slave'
  • 22:24 binasher: s2 master swap - new repl position MASTER_LOG_FILE='db54-bin.000302', MASTER_LOG_POS=870283287
  • 22:23 logmsgbot: asher synchronized wmf-config/db.php 'setting s2 to writeable'
  • 22:22 logmsgbot: asher synchronized wmf-config/db.php 'setting s2 to read-only for master swap to db54'
  • 22:09 logmsgbot: reedy synchronized php-1.20wmf10/extensions/EducationProgram
  • 21:59 binasher: running purgeParserCache.php for all dbs
  • 21:26 binasher: resumed s3 user.user_email migrations
  • 20:34 Jeff_Green: hand-tweezed spence:/etc/nagios/nsca_payments.cfg b/c puppet is broken for spence, restarted nagios on spence
  • 20:15 Jeff_Green: added hosts boron/indium to nagios's nsca config
  • 19:39 logmsgbot: reedy synchronized php-1.20wmf10/resources/ 'jQuery UI 1.8.23'
  • 19:26 paravoid: rolling back sq51 switch to originals
  • 18:52 paravoid: putting sq51 (modified with originals pointing to swift) back into the backend pool
  • 18:33 RobH: authdns-update for mc1009-1016 mgmt
  • 18:16 RobH: db1047 back online, presends sdb and sdc at 3597.3 GB each
  • 18:11 RobH: db1047 disk shelf added, two raid10 arrays of 12 disks each, booting now into OS.
  • 18:00 RobH: db1047 shutting down
  • 17:58 RobH: db1047 mysql shutting down (going to add new disk shelf to server)
  • 16:44 AaronSchulz: Running setZoneAccess.php for all wikis
  • 16:40 apergos: shot morebots so the restarter script would restart it
  • 05:41 binasher: halting s3 user migration on hiwiktionary, will resume in the morning (pst)
  • 04:02 binasher: adding user.user_email index on s3
  • 03:44 binasher: adding user.user_email index on s2
  • 02:37 logmsgbot: LocalisationUpdate completed (1.20wmf10) at Tue Aug 21 02:37:10 UTC 2012
  • 02:21 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Tue Aug 21 02:21:32 UTC 2012
  • 01:47 binasher: adding user.user_email index to enwiki via osc
  • 00:32 maplebed: removed ms-be6 from rotation due to bad memory and repeated kernel panics
  • 00:23 mutante: fenari - install a couple lib* package upgrades
  • 00:10 logmsgbot: preilly synchronized php-1.20wmf10/extensions/MobileFrontend 'weekly deploy'
  • 00:09 logmsgbot: preilly synchronized php-1.20wmf9/extensions/MobileFrontend 'weekly deploy'

August 20

  • 23:48 logmsgbot: kaldari Finished syncing Wikimedia installation... :
  • 23:46 logmsgbot: tstarling synchronized php-1.20wmf9/maintenance/benchmarks/obama.php
  • 23:18 logmsgbot: kaldari Started syncing Wikimedia installation... :
  • 22:50 logmsgbot: reedy synchronized php-1.20wmf10/extensions/VisualEditor/
  • 22:49 logmsgbot: reedy synchronized php-1.20wmf10/includes/GlobalFunctions.php
  • 22:15 logmsgbot: reedy synchronized php-1.20wmf10/skins/
  • 21:33 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Enabling file uploads on testwiki for regular user accounts (exception to bug 12556)'
  • 20:26 Jeff_Green: fixing typo in pdns-templates/wmnet and reloading nameservers
  • 18:57 logmsgbot: catrope synchronized php-1.20wmf10/extensions/VisualEditor/ 'Actually push out VE fixes'
  • 18:52 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Update VE config for Parsoid prefix change'
  • 18:27 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki and mediawikiwiki to 1.20wmf10
  • 18:22 paravoid: cleaning sq51's cache, poisoned after swift changes staging
  • 16:06 logmsgbot: reedy synchronized php-1.20wmf10/cache/l10n/
  • 16:04 mark: Upgrading JUNOS on asw2-a5-eqiad to 11.4R2.14
  • 15:49 mark: Upgrading JUNOS on asw2-d3-sdtpa to 11.4R2.14
  • 15:47 logmsgbot: reedy Started syncing Wikimedia installation... :
  • 15:38 logmsgbot: reedy Finished syncing Wikimedia installation... : testwiki to 1.20wmf10 and rebuild localisation cache for it..
  • 15:01 logmsgbot: reedy Started syncing Wikimedia installation... : testwiki to 1.20wmf10 and rebuild localisation cache for it..
  • 14:46 logmsgbot: reedy synchronized php-1.20wmf10/
  • 14:39 logmsgbot: reedy synchronized wmf-config/
  • 14:38 logmsgbot: reedy synchronized php-1.20wmf10/
  • 14:36 logmsgbot: reedy synchronized live-1.5/
  • 14:30 Reedy: Removing php-1.20wmf7 from mediawiki-installation group
  • 14:26 logmsgbot: reedy synchronized php-1.20wmf10/ 'staging 1.20wmf10 directory'
  • 09:49 paravoid: powering off mw8, faulty (#3425), has been flapping a lot
  • 09:24 paravoid: restarted gmond in cp3002
  • 02:28 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Mon Aug 20 02:28:20 UTC 2012

August 19

  • 20:54 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 02:24 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Sun Aug 19 02:24:13 UTC 2012

August 18

  • 03:46 Tim: copying old logs from bast1001 to fluorine
  • 02:28 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Sat Aug 18 02:28:03 UTC 2012

August 17

  • 23:17 Ryan_Lane: added cn=essextest project to labs LDAP for essex testing. this project isn't compatible with diablo
  • 22:16 notpeter: rebooting/reimaging zhen and silver for upgrade to precise
  • 22:01 mutante: rebuilding wikitech wiki l10n cache
  • 22:00 mutante: rebuilding wikitech-l archives
  • 20:15 Reedy: Pointing php to php-1.20wmf9
  • 20:13 Reedy: Deleting php-1.20wmf8/cache/l10n from mediawiki-installation
  • 17:48 RobH: once all apaches update, will migrate old logs from nfs to fluorine
  • 17:48 RobH: apache logs confirmed hitting fluorine, all will transition over as they puppet update
  • 17:24 logmsgbot: robh synchronized wmf-config/InitialiseSettings.php
  • 17:24 logmsgbot: robh synchronized wmf-config/CommonSettings.php
  • 17:23 RobH: moving apache logs back to fluorine
  • 17:00 cmjohnson1: shutting down search23 to repair DIMM
  • 16:59 cmjohnson1: mc1 shutting down to repair DIMM
  • 16:31 mark: Enabled all 3 new 802.3ad aggregated links for asw-d1-sdtpa to asw-d3-sdtpa
  • 16:15 RobH: authdns-update to correct typo, nothing really broke badly, i r lucky
  • 15:54 RobH: fluorine now coming down for reinstall, all log files backed up to nfs
  • 15:47 mark: Created static 802.3ad trunks on csw1-sdtpa for asw-d1-sdtpa to asw-d3-sdtpa uplinks, deployed with only the existing port enabled
  • 15:44 mark: Created static 802.3ad trunks on asw-d1-sdtpa and asw-d2-sdtpa for uplinks
  • 15:31 RobH: puppet is taking forever on nagios, as soon as it completes, will remove fluorine from decom and rerun
  • 15:11 RobH: authdns-update for fluorine ip update
  • 15:04 RobH: lvm restructure
  • 15:04 RobH: all data copied, all traffic moved, fluorine coming down for reinstallation into internal vlan and lvs restructure
  • 15:03 RobH: all log traffic appears to have moved back to nfs1, copying fluorine logs back to nfs for reformat
  • 14:53 mark: Setup static 802.3ad trunk on asw-d3-sdtpa with one port up
  • 14:40 cmjohnson1: swapping disk3 db44
  • 14:27 logmsgbot: robh synchronized wmf-config/InitialiseSettings.php
  • 14:27 logmsgbot: robh synchronized wmf-config/CommonSettings.php
  • 13:48 logmsgbot: robh ran sync-common-all
  • 13:34 RobH: moving apache logs from pushing to fluorine back to nfs1 for now
  • 05:12 Tim: queued refreshlinks jobs for 1330 commonswiki image description pages with coord templates which weren't refreshed in the previous manual attempt, probably due to failed jobs
  • 03:34 logmsgbot: aaron synchronized php-1.20wmf9/includes/media/DjVu.php 'deployed 3ae632c3c9111b16f019574bcafec58fa3ee408d'
  • 03:34 logmsgbot: aaron synchronized php-1.20wmf9/includes/media/SVG.php 'deployed 3ae632c3c9111b16f019574bcafec58fa3ee408d '
  • 03:16 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/backend/FileBackendStore.php
  • 03:14 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/backend/FileBackendStore.php
  • 03:10 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/backend/FileBackendStore.php 'Profiling'
  • 03:08 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/backend/FileBackendStore.php 'Profiling'
  • 02:42 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/backend/SwiftFileBackend.php 'Profiling'
  • 02:37 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Fri Aug 17 02:37:38 UTC 2012
  • 01:12 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/backend/TempFSFile.php 'deployed 08ea8adc95a4edba0b1f2f13c34f2a57725f1a58'
  • 01:10 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/file/File.php 'deployed 08ea8adc95a4edba0b1f2f13c34f2a57725f1a58'
  • 00:13 Tim: copying MW log archives from NFS to fluorine via rsync on fenari

August 16

  • 23:39 Tim: on fluorine: chown'd log files and restarted udp2log to get new uid
  • 23:39 binasher: reslaved es1002 after conversion to innodb
  • 23:31 logmsgbot: olivneh synchronized php-1.20wmf9/extensions/E3Experiments/Experiments.hooks.php
  • 23:26 mutante: fix date/NTP on sq36
  • 23:14 logmsgbot: kaldari Finished syncing Wikimedia installation... :
  • 23:12 mutante: killing nagios-wm. it stopped talking even though stuff gets written to --infile
  • 22:50 logmsgbot: kaldari Started syncing Wikimedia installation... :
  • 22:25 logmsgbot: preilly Finished syncing Wikimedia installation... : update mobile frontend
  • 22:04 logmsgbot: preilly Started syncing Wikimedia installation... : update mobile frontend
  • 22:03 notpeter: restarting puppet on nfs1/2
  • 21:54 notpeter: restarting puppet on locke
  • 21:51 notpeter: restarting puppet on emery
  • 21:39 notpeter: restarting puppet on oxygen
  • 21:37 cmjohnson1: shutting ms-be6 down for hardware testing/replacing
  • 21:26 notpeter: stopping puppet on nfs2 and restarting both udp2log instances to switch user. will keep puppet off until I merge new init script into puppet
  • 21:20 notpeter: stopping puppet on nfs1 and restarting both udp2log instances to switch user. will keep puppet off until I merge new init script into puppet
  • 21:12 maplebed: ms-be6 down for bad disk repair
  • 20:58 Ryan_Lane: attempting to upgrade virt1000 to precise
  • 20:47 cmjohnson1: ms-be6 shutting down to replace 2 disks
  • 20:38 notpeter: stopping puppet on locke and restarting both udp2log instances to switch user. will keep puppet off until I merge new init script into puppet
  • 20:12 RobH: ms-be1004 is ready to be installed into service
  • 19:57 RobH: ms-be1004 down for ssd install
  • 19:44 notpeter: stopping puppet on emery and restarting both udp2log instances to switch user. will keep puppet off until I merge new init script into puppet
  • 19:34 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Set $readFromSwiftShared = true;'
  • 19:30 logmsgbot: aaron synchronized wmf-config/filebackend.php
  • 19:29 notpeter: stopping puppet on oxygen and restarting both udp2log instances to switch user. will keep puppet off until I merge new init script into puppet
  • 19:23 logmsgbot: aaron synchronized wmf-config/filebackend.php
  • 19:17 binasher: running purgeParserCache.php again
  • 19:05 RobH: puppet restarted on brewster
  • 18:58 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Fixed container mapping for shared repo'
  • 18:28 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Use NFS for reads for shared-multiwrite'
  • 18:14 logmsgbot: aaron synchronized wmf-config/swift.php 'Disabled redundant thumb copy/purge hooks.'
  • 18:01 RobH: authdns-update for new es1005-1010
  • 17:59 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Make internal reads come from swift for all wikis'
  • 17:50 apergos: powercycled ms-be3 ... in theory
  • 16:52 cmjohnson1: shutting down mw18 and mw19 to replace DIMM
  • 16:29 ottomata: Stopping puppet on brewster again. Sigh. PARTMAAAAN!
  • 16:22 logmsgbot: reedy synchronized php-1.20wmf9/includes/QueryPage.php
  • 16:07 cmjohnson1: shutting down mw8 to replace bad DIMM
  • 15:58 ottomata: starting puppet on brewster. (Woo, partman looking better!)
  • 14:15 ottomata: stopping puppet on brewster to continue partman troubleshooting for analytics dells
  • 13:26 mark: Enabled and started SIS deduplication on home_pmtpa on nas1-a
  • 12:05 mark: Removed OSPF metric on xe-5/2/1.0 on cr2-eqiad, to move eqiad->pmtpa traffic to the lower latency link
  • 11:27 pp-pdf2: restarted all services
  • 11:27 pp-pdf1: restarted all services
  • 11:27 pp-pdf3: restarted all services
  • 11:26 pp-pdf3: updated mwlib.epub to 0.14.1
  • 11:26 pp-pdf2: updated mwlib.epub to 0.14.1
  • 11:25 pp-pdf1: updated mwlib.epub to 0.14.1
  • 10:50 mark: Setup semi-sync snapmirror from nas1-a:home_pmtpa to nas1001-a:home_pmtpa
  • 03:57 mutante: nagios back up after adding missing monitor groups misc_pmtpa in appserver role (srv194)
  • 02:56 K4-713: synchronized payments cluster to 37e31eddf5a4
  • 02:28 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Thu Aug 16 02:28:33 UTC 2012
  • 02:12 Ryan_Lane: pushed in large puppet change for ldap, openstack, gerrit and ldap pdns to make it more modular and to add support for eqiad region
  • 02:10 Ryan_Lane: fixed arecord issues with labsconsole by adding an exception handling live hack for the jobs
  • 01:21 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'remove wikimedia philippines event exemption'
  • 00:40 mutante: installing package upgrades on singer
  • 00:36 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/backend/FileBackendMultiWrite.php 'deployed c51a9a288b6dd5c0023a77f324c04707b23501c6'
  • 00:24 binasher: running "mwscript purgeParserCache.php --wiki=$db --age=1814400" instead
  • 00:19 binasher: running mwscript purgeParserCache.php --wiki=enwiki --age=1209600
  • 00:02 RoanKattouw: find /export/upload/wik*/*/{0,1,2,3,4,5,6,7,8,9,a,b,c,d,e,f,archive,math,temp,timeline} ! -user apache -exec /root/fixownership2 \{\} \; where fixownership2 = chown -h apache $1
  • 00:02 RoanKattouw: Restarting the find-chown, this time with -h so symlinks are handled correctly (for some reason there are a bunch of broken symlinks with weird characters out there...)

August 15

  • 23:52 binasher: powercycled srv259
  • 23:44 mutante: brought nagios back up by removing "misc_pmtpa" host and servicegroups from srv194 manually
  • 23:35 mutante: restarted pdns-recursor on ns0
  • 23:11 mutante: authdns-update, moving zirconium to public IP and removing selenium
  • 22:33 logmsgbot: reedy synchronized php-1.20wmf9/extensions/OpenSearchXml/
  • 21:19 RoanKattouw: find /export/upload/wik*/*/{0,1,2,3,4,5,6,7,8,9,a,b,c,d,e,f,archive,math,temp,timeline} ! -user apache -exec chown apache \{\} \;
  • 21:17 RoanKattouw: chown on ms7 has finished, running another find to find remaining bad files
  • 20:57 K4-713: synchronized payments cluster to 79275e152b76f
  • 20:26 ottomata: staring puppet on brewster. PARTMAN IS THE ENEMY. will continue the fight tomorrow.
  • 20:22 mutante: http://en.wikipedia.org/w/ now redirects to Main_page on all languages. It was a 403 before.
  • 20:20 mutante: apache-graceful-all
  • 20:15 mutante: sync-apache pushing out new redirects from /w/ to Main_Page
  • 20:04 ottomata: stopping puppet on brewster to experiment with partman on new analytics dells
  • 19:29 mark: Removed OSPF/OSPFv3 metric 60 on cr1-sdtpa:xe-1/1/0 (eqiad link)
  • 19:17 mark: Moved VRRP mastership from cr2-pmtpa to cr1-sdtpa by reassigning VRRP priorities, relieving the sdtpa-pmtpa link
  • 18:44 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Remaining wikis over to 1.20wmf9
  • 17:11 maplebed: ms-be1 dropped offline yesterday; powercycling.
  • 16:11 mark: Setup SnapMirror replication from nas1-a:fr_archive to nas1001-a:fr_archive
  • 16:10 mark: Created 4 TB volumes 'fr_archive' in aggregate prod1 on both nas1-a nad nas1001-a
  • 16:10 mark: Increased aggregate 'prod1' by 4 drives on both nas1-a and nas1001-a
  • 16:06 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Wed Aug 15 16:06:26 UTC 2012
  • 15:40 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Wed Aug 15 15:40:21 UTC 2012
  • 14:33 cmjohnson1: authdns-update for es9 and es10
  • 13:46 RobH: authdns-update for wikipedia.co.za
  • 12:15 mark: Started rsync of /home to nas1-a:/vol/home_pmtpa on nfs1
  • 12:11 mark: Restricted NFS mounting of /vol/root on all NetApps
  • 11:39 mark: Destroyed aggregate 'labs' on nas1-a
  • 10:00 Tim: deploying bugfix version of wmerrors
  • 09:51 Tim: moved old MW debug logs out of /home/wikipedia/logs to avoid confusion, live logs are now on fluorine
  • 05:22 Tim: moving MediaWiki logs to fluorine
  • 05:17 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
  • 05:15 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
  • 04:38 Tim: added rate limit exemption for 27.108.200.111
  • 04:38 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
  • 03:00 Tim: built a udplog package for precise and uploaded it
  • 02:35 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'enabling Scribunto and CodeEditor on test2wiki'
  • 02:28 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
  • 02:15 logmsgbot: tstarling Finished syncing Wikimedia installation... :
  • 02:01 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
  • 01:46 logmsgbot: tstarling Started syncing Wikimedia installation... :
  • 01:00 logmsgbot: tstarling Started syncing Wikimedia installation... :
  • 00:47 logmsgbot: tstarling Started syncing Wikimedia installation... :
  • 00:39 Tim: running scap to for extension-list update

August 14

  • 23:53 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Move VE Parsoid from cadmium to wtp1'
  • 21:25 logmsgbot: kaldari synchronized php-1.20wmf9/extensions/PageTriage/tools/cleanupPageTriage.php 'adding PageTriage maintainence script'
  • 21:05 logmsgbot: kaldari Finished syncing Wikimedia installation... :
  • 20:45 logmsgbot: kaldari Started syncing Wikimedia installation... :
  • 20:38 mutante: killed duplicate nagios-wm
  • 20:20 cmjohnson1: db44 swapping out disk1 rt 3424
  • 20:08 ottomata: installed precise and puppetized stat1001
  • 19:26 cmjohnson1: search23 shutting down to replace DIMM A2 rt 3423
  • 19:00 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Deploying https://gerrit.wikimedia.org/r/19399'
  • 18:52 RoanKattouw: Running /root/fixownership < /root/badownershipfiles in a screen on ms7
  • 18:14 notpeter: stopping puppet on brewster for partman experimentation
  • 17:34 logmsgbot: catrope synchronized php-1.20wmf9/.gitmodules
  • 17:33 RoanKattouw: Yesterday's find on ms7 is done, found 1.8 million files with bad ownership. Will run a batch chown on them soon
  • 17:32 logmsgbot: catrope synchronized php-1.20wmf9/extensions/UploadWizard 'Deploy 9d9189458ad8d966d952d295918847f00e304cb5'
  • 09:17 logmsgbot: nikerabbit synchronized php-1.20wmf9/extensions/Translate/ 'I18ndeploy: Translate bugs'
  • 07:46 logmsgbot: tstarling Finished syncing Wikimedia installation... :
  • 07:07 logmsgbot: tstarling Started syncing Wikimedia installation... :
  • 06:48 Tim: running scap to push out CodeEditor and Scribunto sources, not enabled yet
  • 05:45 logmsgbot: olivneh synchronized php-1.20wmf9/extensions/E3Experiments/ 'Update E3Experiments to master'
  • 03:34 Tim: restarting all apaches to get LuaSandbox extension
  • 03:26 Tim: on mw8: removed broken package php5-memcached
  • 02:45 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Tue Aug 14 02:45:30 UTC 2012
  • 02:28 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Tue Aug 14 02:28:13 UTC 2012
  • 01:06 Tim: restarting apaches to get new wmerrors configuration

August 13

  • 23:53 Tim: uploaded a new version of the php5-wmerrors package for lucid and precise
  • 23:18 mutante: removed #wikimedia-tech from /usr/local/bin/start-nagios-bot
  • 23:01 mutante: killing nagios-wm process which isn't supposed to be on #wikimedia-tech anymore
  • 21:58 cmjohnson1: running authdns-update for labsdb1 & 2
  • 21:19 RoanKattouw: ...piping result into ms7:/root/badownershipfiles
  • 21:18 RoanKattouw: ms7# find /export/upload/wik*/*/{0,1,2,3,4,5,6,7,8,9,a,b,c,d,e,f,archive,math,temp,timeline} ! -user apache
  • 21:17 RoanKattouw: Searching for files with wrong ownership on ms7 using
  • 21:13 RobH: reinstalling virt1004 to see if the raid issue re-appears
  • 20:55 notpeter: updating root auth keys file on ms7 by hand.
  • 20:36 logmsgbot: catrope synchronized php-1.20wmf9/resources/jquery/jquery.tablesorter.js 'Deploy 1fafaef3aa157f49cbbf4b9cffb03544bcf72a08'
  • 20:04 preilly: updating zero for custom banner colors
  • 20:03 logmsgbot: preilly synchronized php-1.20wmf9/extensions/ZeroRatedMobileAccess 'update for landing page'
  • 20:03 logmsgbot: preilly synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess 'update for landing page'
  • 19:42 logmsgbot: preilly synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess 'update for landing page'
  • 19:41 preilly: update to zero for banner text link
  • 19:41 logmsgbot: preilly synchronized php-1.20wmf9/extensions/ZeroRatedMobileAccess 'update for landing page'
  • 19:36 logmsgbot: preilly synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess 'update for landing page'
  • 19:32 RobH: powering off analytics1011-1014 to remove the add on nics (disrupting installer)
  • 18:25 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved enwiki to 1.20wmf9
  • 17:21 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Switched backend reads to swift for testwikis and mw.org'
  • 15:18 paravoid: reprovisioning srv281 with a new fs layout
  • 13:05 mark: Doing NDU of IOM3 shelf firmware on nas1001-a
  • 12:59 mark: Doing NDU of IOM3 shelf firmware on nas1-a
  • 12:47 mark: Setup NTP on all NetApps
  • 12:24 mark: Restored SnapMirror relationship between nas1-a and nas1001-a, 70 days lagged, transfer initiated
  • 12:04 paravoid: apt: include php-memcached 2.0.1-6~wmf+lucid2 @ lucid-wikimedia
  • 11:58 paravoid: apt: include php5 5.3.10-1ubuntu3.2+wmf1 @ precise-wikimedia
  • 11:08 paravoid: apt: remove wikidiff2/0.0.1wm1 (obsolete), copysrc php-wikidiff2 {lucid,precise}-wikimedia
  • 11:08 paravoid: apt: remove php-wikidiff2/1.1.0-2 from lucid-wikimedia/universe (lucid-wikimedia/main has 1.1.2-1)
  • 10:49 apergos: starting initial rsync from ms8 to ms10 (should take approximately forever)
  • 06:26 Tim: uploaded czmq and zpubsub packages to brewster for Ori
  • 02:37 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Mon Aug 13 02:37:53 UTC 2012
  • 02:20 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Mon Aug 13 02:20:27 UTC 2012

August 12

  • 03:56 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Sun Aug 12 03:56:33 UTC 2012
  • 03:38 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Sun Aug 12 03:38:33 UTC 2012
  • 03:19 Reedy: Running localisation update manually in screen session on fenari
  • 02:00 logmsgbot: LocalisationUpdate failed: git pull of extensions failed

August 11

  • 02:01 logmsgbot: LocalisationUpdate failed: git pull of extensions failed

August 10

  • 21:54 Ryan_Lane: live hacked nova on virt2 with parts of nova git commit 4584e552a653904c36cf04cb295a7bf09d2def28
  • 21:22 logmsgbot: preilly synchronized php-1.20wmf9/extensions/ZeroRatedMobileAccess 'update for landing page'
  • 21:22 logmsgbot: preilly synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess 'update for landing page'
  • 21:21 preilly: update to zero needed to support carrier testing
  • 21:21 logmsgbot: preilly synchronized php-1.20wmf9/extensions/MobileFrontend 'update for zero'
  • 21:21 preilly: update to zero needed to support carrier testing
  • 21:20 logmsgbot: preilly synchronized php-1.20wmf8/extensions/MobileFrontend 'update for zero'
  • 18:42 Jeff_Green: adding frack.eqiad.wmnet hosts to DNS (wmnet and 10..in-addr.arpa)
  • 18:11 K4-713: Payments cluster updated to 133b2b6fb5
  • 17:07 notpeter: temporarily stopping puppet on brewster
  • 16:43 cmjohnson1: enabling pxe boot for 10GB Nic cards mc1-16
  • 16:32 mark: Started OnTap upgrade to 8.1 on nas1-a and nas1-b
  • 14:39 mark: OnTap upgrade on nas1001-a and -b complete
  • 14:11 mark: Started OnTap upgrade to 8.1 on nas1001-b
  • 13:55 mark: Started OnTap upgrade to 8.1 on nas1001-a
  • 13:10 mark: Upgraded SP firmware to 1.3 on nas1001-a and nas1001-b
  • 11:55 mark: nas1001-b disk fw upgrade completed
  • 11:51 mark: nas1001-a disk firmware upgrade completed. Started disk firmware upgrade on nas1001-b
  • 11:46 mark: Started disk firmware update on nas1001-a
  • 02:39 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Fri Aug 10 02:39:57 UTC 2012
  • 02:22 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Fri Aug 10 02:22:22 UTC 2012
  • 01:07 Tim: on virt0: enabling mysql general log to debug "detached session" errors on nova
  • 00:05 Tim: added myself as a labsconsole sysop, added myself to the packaging project, created a "php" instance for building php-related packages

August 9

  • 22:05 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Added swift to multiwrite backend for all remaining wikis'
  • 22:02 RobH: db1046 back online
  • 21:40 RobH: shutting down db1046 to migrate its position in the rack
  • 21:27 logmsgbot: olivneh synchronized php-1.20wmf8/extensions/E3Experiments/experiments/communityClicks.js
  • 21:20 binasher: resumed enwiki replication in eqiad
  • 21:13 binasher: stopping mysql on db1017 for a minute (and all enwiki eqiad replication with it)
  • 20:52 logmsgbot: kaldari synchronized wmf-config/extension-list 'Removing deprecated CustomUserSignup extension'
  • 20:52 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'Removing deprecated CustomUserSignup extension and turning on Curation Toolbar for en.wiki'
  • 20:52 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'Removing deprecated CustomUserSignup extension'
  • 20:09 logmsgbot: kaldari Finished syncing Wikimedia installation... :
  • 19:48 logmsgbot: catrope synchronized php-1.20wmf9/extensions/VisualEditor/ 'VE bugfix'
  • 19:37 logmsgbot: kaldari Started syncing Wikimedia installation... :
  • 19:27 logmsgbot: catrope synchronized wmf-config/CommonSettings.php
  • 19:22 binasher: rebooting db1009, db1010 for upgrades
  • 19:20 binasher: rebooting db1011 for upgrade
  • 19:17 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Unbreak links in VE'
  • 18:01 cmjohnson1: re-imaging db68-77
  • 16:54 cmjohnson1: loading OS on db65-67
  • 16:06 logmsgbot: preilly synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess 'update for zero'
  • 16:05 logmsgbot: preilly synchronized php-1.20wmf9/extensions/ZeroRatedMobileAccess 'update for zero'
  • 14:53 RobH: cp1001 returned to service, resolving rt 3212
  • 14:30 RobH: cp1001 squid services going offline for troubleshooting
  • 02:37 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Thu Aug 9 02:37:33 UTC 2012
  • 02:20 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Thu Aug 9 02:19:57 UTC 2012

August 8

  • 21:30 logmsgbot: kaldari Finished syncing Wikimedia installation... :
  • 20:49 logmsgbot: kaldari Started syncing Wikimedia installation... :
  • 20:02 binasher: streaming a hotbackup of db1017 to db1050
  • 19:54 ottomata: updating DNS entries for analytics1011-1027, they are all in row-c
  • 18:47 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikimedia to 1.20wmf9
  • 18:45 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Fix $wgVisualEditorParsoidPrefix'
  • 18:44 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews and wikibooks to 1.20wmf9
  • 18:36 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary and wikiquote over to 1.20wmf9
  • 18:35 binasher: completed metawiki centralnotice schema migration for fundraising
  • 18:31 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity and wikisource to 1.20wmf9
  • 18:26 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special, fishbowl, closed and private wikis to 1.20wmf9
  • 18:25 Reedy: special, fishbowl, closed and private wikis to 1.20wmf9
  • 18:16 apergos: shutting down ms10, going to move it to public ip
  • 17:21 logmsgbot: preilly synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess 'update for zero'
  • 17:20 logmsgbot: preilly synchronized php-1.20wmf9/extensions/ZeroRatedMobileAccess 'update for zero'
  • 16:58 mutante: freeing disk space on wikitech
  • 16:53 logmsgbot: preilly synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess 'update for zero'
  • 16:52 logmsgbot: preilly synchronized php-1.20wmf9/extensions/ZeroRatedMobileAccess 'update for zero'
  • 16:46 logmsgbot: preilly synchronized php-1.20wmf9/extensions/MobileFrontend 'update for zero'
  • 16:45 logmsgbot: preilly synchronized php-1.20wmf8/extensions/MobileFrontend 'update for zero'
  • 16:40 cmjohnson1: swapping out main board on search32
  • 15:21 logmsgbot: catrope synchronized php-1.20wmf9/extensions/VisualEditor/.git 'Sync .git metadata so Special:Version is accurate'
  • 14:43 logmsgbot: reedy synchronized wmf-config/
  • 11:49 mark: Upgraded Observium to latest SVN
  • 02:40 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Wed Aug 8 02:40:06 UTC 2012
  • 02:21 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Wed Aug 8 02:21:44 UTC 2012
  • 01:04 mutante: a couple package upgrades on bast1001

August 7

  • 23:22 logmsgbot: catrope synchronized php-1.20wmf8/resources/mediawiki/mediawiki.js 'Deploying 6b6466f948d29520e1e3ab2592b940ce52415300'
  • 23:19 logmsgbot: catrope synchronized php-1.20wmf9/extensions/VisualEditor/ 'Update VisualEditor'
  • 22:48 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'syncing CommonSettings.php for Curation Toolbar'
  • 22:47 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'syncing InitialiseSettings.php for Curation Toolbar'
  • 22:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disabled echo due to bug 39085'
  • 22:12 logmsgbot: reedy synchronized php-1.20wmf8/extensions/E3Experiments/
  • 22:05 binasher: streaming a hotbackup of db1001 to db1049 (new enwiki snapshot host in eqiad)
  • 21:48 logmsgbot: reedy synchronized php-1.20wmf9/extensions/EducationProgram/
  • 21:45 Reedy: Updated EducationProgram schema on both enwiki and test2wiki
  • 20:15 logmsgbot: reedy synchronized php-1.20wmf9/includes/api/ApiQuery.php
  • 16:23 LeslieCarr: moving bits traffic from pmtpa to eqiad
  • 15:45 mark: Moving text squid traffic from pmtpa to eqiad
  • 15:08 first reports of sites being very slow
  • 15:07 nagios-wm: Apache HTTP on ... is CRITICAL: CRITICAL - Socket timeout after 10 seconds
  • 14:35 Jeff_Green: starting otrs dump on db49
  • 07:11 paravoid: powercycled srv272, swapdeath
  • 03:09 logmsgbot: LocalisationUpdate completed (1.20wmf9) at Tue Aug 7 03:09:28 UTC 2012
  • 02:37 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Tue Aug 7 02:37:34 UTC 2012
  • 00:53 mutante: planet (en and fr) configs Updated to revision 115645.
  • 00:32 Tim: on cp3002: set tcp_tw_recycle back to zero

August 6

  • 23:49 mutante: wikitech back to old version after failed upgrade attempt
  • 23:48 mutante: test
  • 22:27 mutante: dist-upgrading wikitech instance
  • 22:19 logmsgbot: aaron synchronized php-1.20wmf8/includes/filerepo/backend/SwiftFileBackend.php
  • 22:14 binasher: rebooting db1028, upgrading
  • 22:13 binasher: rebooting db1027, upgrade to precise
  • 22:10 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/backend/SwiftFileBackend.php 'deployed 0a5c71d4dd6405502b1d1d0a02b5d0927d519986'
  • 21:59 binasher: rebooting db1026, upgrading to precise
  • 21:54 mutante: saving space on wikitech linode - gzipping old .sql dumps and stuff
  • 21:32 logmsgbot: catrope synchronized php-1.20wmf9/extensions/VisualEditor/ 'Updating VisualEditor'
  • 21:18 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Fix Parsoid URL'
  • 21:09 logmsgbot: catrope synchronized php-1.20wmf9/extensions/VisualEditor 'Updating VisualEditor'
  • 20:23 logmsgbot: catrope synchronized php-1.20wmf9/cache/l10n/ 'Sync out l10n cache again now that it is properly rebuilt'
  • 19:47 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki to 1.20wmf9
  • 19:45 RoanKattouw: Rebuilt ExtensionMessages-1.20wmf9.php and then l10n cache for wmf9 to pick up addition of FundraiserLandingPage.i18n.magic.php , somehow wasn't picked up during initial scap
  • 19:44 RobH: asw-c8-eqiad PEM1 power reseated, cleared alarm rt 3204
  • 19:39 logmsgbot: catrope synchronized wmf-config/ExtensionMessages-1.20wmf9.php 'Updated'
  • 19:30 RobH: db1047 back online
  • 19:18 RobH: db1047 shutting down
  • 19:15 RobH: db1047 mysql and system shutdown per rt 3084 for bad memory swap
  • 19:14 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files:
  • 19:12 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.20wmf9
  • 19:11 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.20wmf9 and scap to rebuild message cache
  • 18:20 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.20wmf9 and scap to rebuild message cache
  • 18:12 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.20wmf9 and scap to rebuild message cache
  • 18:06 logmsgbot: aaron synchronized php-1.20wmf9/includes/filerepo/backend/FileBackendMultiWrite.php 'deployed b55e9652fce3051d621356c5c87c47f44515a367'
  • 18:02 logmsgbot: aaron synchronized wmf-config/filebackend.php
  • 17:59 logmsgbot: aaron synchronized php-1.20wmf8/includes/filerepo/backend/FileBackendMultiWrite.php 'deployed b55e9652fce3051d621356c5c87c47f44515a367'
  • 17:11 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Adding Swift to local multiwrite backend for testwikis+mw.org'
  • 16:57 cmjohnson1: powering down mc1 to reseat DIMM
  • 16:50 logmsgbot: reedy synchronized .
  • 16:33 logmsgbot: reedy synchronized php-1.20wmf9/
  • 15:08 paravoid: switched European bits traffic back to esams
  • 14:50 Tim: on cp3002: temporarily set net.ipv4.tcp_tw_recycle=1
  • 14:41 Tim: on cp3002 increasing tcp_max_tw_buckets
  • 14:33 Tim: on cp3001: date -s `"date"` to fix leap second issue
  • 13:26 Nemo_bis: Reedy says: Status: Down - routing issues to Tampa !Wikipedia !Wikimedia
  • 13:17 Nemo_bis: more or less everything at !Wikimedia / !Wikipedia seems down
  • 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Mon Aug 6 02:27:00 UTC 2012

August 5

  • 21:54 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Update per rearrangements etc'
  • 21:16 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'more cp updates to $wgSquidServersNoPurge'
  • 20:56 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Add some eqiad internal ips for squids to wgSquidServersNoPurge'
  • 02:24 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Sun Aug 5 02:24:31 UTC 2012

August 4

  • 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Sat Aug 4 02:27:51 UTC 2012

August 3

  • 23:50 logmsgbot: reedy synchronized wmf-config/secure.php 'fix bits usages'
  • 21:53 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 38988 - Enable Extension:Collection for Wikimedia Sverige'
  • 18:58 maplebed: updated DNS to fix *.svc.{esams,eqiad}.wmnet forward resolution
  • 18:49 andrewbogott: turned off nova-compute on virt1, virt2, virt3, virt4
  • 18:44 maplebed: updating DNS adding swift eqiad LVS address
  • 17:42 logmsgbot: aaron synchronized php-1.20wmf8/extensions/TimedMediaHandler 'Updated to master (0207962155b810b11100bc6d05c3562949e7c1a9)'
  • 07:58 hashar: GlusterFS upgrade of labs project storage seems to have been completed. Fixed the issues I had with it on 'beta' project, yeah!!!!!
  • 02:25 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Fri Aug 3 02:25:37 UTC 2012
  • 01:54 domas: bumped memcached timeout to be slightly above min_rto, 200ms->250ms, should eliminate most of stupid memcached errors because of datacenter network failures
  • 01:53 logmsgbot: midom synchronized wmf-config/mc.php

August 2

  • 22:48 Ryan_Lane: upgrading glusterfs for project storage to 3.3
  • 22:05 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Moved all wikis to multiwrite backend'
  • 22:05 mutante: dist-upgrading sodium
  • 21:47 hashar: added Dsc to the Gerrit 'integration' group. David now got access to the various integration/* repositories.
  • 20:34 logmsgbot: andrew synchronized php-1.20wmf8/extensions/Echo/modules/base/ext.echo.base.css
  • 20:30 logmsgbot: asher synchronized wmf-config/mc.php 'setting wgMemCachedPersistent back to false, no change observed to current issue'
  • 20:12 logmsgbot: asher synchronized wmf-config/mc.php 'setting wgMemCachedPersistent = true as an experiment'
  • 20:06 Andrew: Leaving Echo deployment there for now.
  • 20:04 Andrew: Rolled back LQT updates, they seem to break other notifications. We'll figure out the bugs with that in another window.
  • 20:02 logmsgbot: andrew synchronized php-1.20wmf8/extensions/LiquidThreads
  • 19:56 Andrew: deployed Echo to mediawiki.org
  • 19:56 logmsgbot: andrew synchronized wmf-config/InitialiseSettings.php
  • 19:55 logmsgbot: andrew synchronized wmf-config/CommonSettings.php
  • 19:55 logmsgbot: andrew synchronized php-1.20wmf8/extensions/Echo
  • 19:54 logmsgbot: andrew synchronized php-1.20wmf8/extensions/LiquidThreads
  • 19:44 logmsgbot: andrew synchronized wmf-config/InitialiseSettings.php
  • 19:44 logmsgbot: andrew synchronized php-1.20wmf8/extensions/Echo
  • 19:43 logmsgbot: andrew synchronized php-1.20wmf8/extensions/LiquidThreads
  • 19:43 Andrew: Deploying Echo to test2wiki instead
  • 18:38 Andrew: Turned Echo back on on testwiki
  • 18:35 Andrew: Stealing E2's deployment window to have another crack at Echo
  • 16:44 paravoid: sync-apache/apache-gracefull-all for BZ #38905 (ShortUrl on !Wikipedia)
  • 16:20 RobH: authdns-update for smokeping
  • 13:05 logmsgbot: nikerabbit synchronized php-1.20wmf8/extensions/Translate/specials/ 'Translate bug fixes'
  • 04:45 logmsgbot: tstarling synchronized php-1.20wmf8/skins/common/commonElements.css
  • 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Thu Aug 2 02:27:17 UTC 2012
  • 02:11 Ryan_Lane: deployed OATHAuth 8992e6f541cd7adf8111ccd87f25a481d3759b33 on labsconsole
  • 02:01 logmsgbot: reedy synchronized php-1.20wmf8/extensions/E3Experiments/
  • 01:26 logmsgbot: reedy synchronized php-1.20wmf8/extensions/SubPageList3/
  • 01:16 logmsgbot: reedy synchronized php-1.20wmf8/extensions/SubPageList3/SubPageList3.php
  • 00:55 logmsgbot: asher synchronized wmf-config/db.php 'moving watchlist/special contrib queries to db63, preparing to decom db12'
  • 00:47 logmsgbot: reedy synchronized php-1.20wmf8/extensions/SubPageList3/SubPageList3.php
  • 00:43 logmsgbot: asher synchronized wmf-config/db.php 'adding db63 to s1 at a low weight'

August 1

  • 23:59 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 23:27 mutante: rebuilding wikitech-l archives
  • 23:16 logmsgbot: andrew Finished syncing Wikimedia installation... :
  • 22:59 Andrew: pulling the plug on Echo deployment due to replication-lag related bug.
  • 22:57 logmsgbot: andrew Started syncing Wikimedia installation... :
  • 22:32 Andrew: Running scap to deploy Echo to test
  • 22:31 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Made mw.org use the multiwrite backend.'
  • 22:23 logmsgbot: aaron synchronized live-1.5/MWVersion.php 'logging for /home being used for thumbs for testwiki'
  • 22:14 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Removed read-only flag from testwiki backends.'
  • 22:08 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Made testwikis use multiwrite backend.'
  • 21:27 Andrew: Added Echo tables to testwiki
  • 21:11 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Configured $wgTimedTextForeignNamespaces.'
  • 21:08 logmsgbot: aaron synchronized php-1.20wmf8/extensions/TimedMediaHandler/handlers/TextHandler/TextHandler.php 'Deployed a247b6264c99cff66e4e3ae3428a46f971fab3a2'
  • 21:07 logmsgbot: aaron synchronized php-1.20wmf8/extensions/TimedMediaHandler/TimedMediaHandler.php 'Deployed a247b6264c99cff66e4e3ae3428a46f971fab3a2'
  • 21:01 logmsgbot: reedy synchronized php-1.20wmf8/extensions/E3Experiments/
  • 20:48 logmsgbot: andrew synchronized wmf-config/CommonSettings.php
  • 20:47 Andrew: srv281: rsync: write failed on "/apache/common-local/wmf-config/InitialiseSettings.php": No space left on device (28)
  • 20:47 logmsgbot: andrew synchronized wmf-config/InitialiseSettings.php
  • 20:46 Andrew: syncing {Common,Initialise}Settings.php
  • 20:46 Andrew: Added configuration skeleton for Echo in {Common,Initialise}Settings.php
  • 20:46 Andrew: Created tables for Echo on mediawikiwiki
  • 19:16 MaxSem: Ran extensions/GeoData/updateIndexGranularity.php on testwiki
  • 19:04 binasher: started hotbackup of db1017 to db63
  • 19:04 mark: Ran reprepro --delete clearvanished on brewster to cleanup removed repositories karmic-wikimedia and oneiric-wikimedia
  • 19:03 mutante: installing a couple lib upgrades on fenari
  • 18:15 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: everything else over to 1.20wmf8
  • 18:04 logmsgbot: aaron synchronized php-1.20wmf8/extensions/TimedMediaHandler 'Updating to 4b7c77d63d7798a024c7f8b4ba01e2f3e7203550'
  • 17:37 maplebed: pushing dns typo correction
  • 10:32 Tim: on dobson: restarted ntpd
  • 10:30 Tim: on linne: restarted ntpd in an attempt to fix the leap indicator
  • 02:49 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Wed Aug 1 02:49:33 UTC 2012
  • 02:25 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Wed Aug 1 02:25:43 UTC 2012
  • 01:53 logmsgbot: catrope synchronized php-1.20wmf8/cache/l10n
  • 01:05 logmsgbot: catrope synchronized php-1.20wmf8/extensions/TimedMediaHandler/MwEmbedModules/EmbedPlayer/resources/mw.EmbedPlayer.js 'touch'
  • 00:33 Tim: leap second event plus one month is causing an apparent 1s step in time reported by linne/dobson as seen by some clients, causing nagios errors etc. Will step.

July 31

  • 23:12 logmsgbot: aaron Finished syncing Wikimedia installation... :
  • 22:32 logmsgbot: aaron Started syncing Wikimedia installation... :
  • 22:19 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Disable ogghandler for wikis using TMH.'
  • 22:18 logmsgbot: catrope synchronized php-1.20wmf7/extensions/TitleBlacklist/TitleBlacklist.hooks.php 'php-1.20wmf8/extensions/TitleBlacklist/TitleBlacklist.hooks.php Deploying 3d45bc25577f33bc220e17a7671d391977f9fbf9 and 4c7db40395b6b2d273e08f6b0e59c5a74cb5ed34'
  • 22:17 RobH: authdns-update for ms-be1006 and ms-be1012
  • 22:04 binasher: added memcached_1.4.14-0wmf1_amd64 to precise-wikimedia
  • 21:55 logmsgbot: catrope synchronized php-1.20wmf8/extensions/TitleBlacklist/TitleBlacklist.hooks.php
  • 21:54 logmsgbot: catrope synchronized php-1.20wmf8/extensions/TitleBlacklist/TitleBlacklist.hooks.php
  • 21:49 logmsgbot: catrope synchronized php-1.20wmf8/extensions/TitleBlacklist/TitleBlacklist.hooks.php
  • 21:49 logmsgbot: aaron Finished syncing Wikimedia installation... :
  • 21:45 logmsgbot: aaron Started syncing Wikimedia installation... :
  • 21:08 logmsgbot: aaron Finished syncing Wikimedia installation... :
  • 21:03 logmsgbot: aaron Started syncing Wikimedia installation... :
  • 20:09 logmsgbot: mlitn Finished syncing Wikimedia installation... :
  • 19:35 logmsgbot: mlitn Started syncing Wikimedia installation... :
  • 19:26 RobH: authdns-update for wtp1 info
  • 17:40 AaronSchulz: Initial swift originals migration for s1-s7 done
  • 17:29 maplebed: finished swift deploy - image scaling requests now go straight to the rendering cluster
  • 16:16 logmsgbot: aaron synchronized live-1.5/MWVersion.php 'Recognize thumb_handler for testwiki and /home check.'
  • 16:09 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Deployed handlerUrl for thumb zone.'
  • 16:00 maplebed: beginning swift deploy to make thumbnail requests bypass ms5 and go straight from swift to the rendering cluster
  • 14:33 mark: Repooled cp1042
  • 14:33 mark: Reinstalled cp1042 with Precise
  • 13:16 Tim: compiling phpllvm tests on bast1001
  • 12:19 mark: Depooled cp1042 for reinstall with Precise
  • 08:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'touch'
  • 08:45 logmsgbot: reedy synchronized flaggedrevs.dblist 'Disable FR on eswikibooks'
  • 02:54 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Tue Jul 31 02:54:35 UTC 2012
  • 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Tue Jul 31 02:27:34 UTC 2012

July 30

  • 23:49 binasher: adding revised enwiki.aft_article_feedback af_user_id_user_ip index via osc
  • 23:48 binasher: dropping enwiki.aft_article_feedback af_user_id_user_ip index
  • 21:48 logmsgbot: reedy synchronized php-1.20wmf7/extensions/Collection/.git 'Just for saper..'
  • 20:51 binasher: adding enwiki.aft_article_feedback af_user_id_ip index
  • 20:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'touch'
  • 20:30 logmsgbot: reedy synchronized php-1.20wmf8/extensions/Collection
  • 20:29 logmsgbot: reedy synchronized php-1.20wmf7/extensions/Collection
  • 19:26 logmsgbot: reedy synchronized .
  • 19:22 logmsgbot: reedy synchronized php-1.20wmf8/extensions/UploadWizard/
  • 18:25 notpeter: removing srv194 from apaches pool due to logging issues
  • 18:06 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf8
  • 17:42 notpeter: re-adding srv194 to pmtpa apaches pool for basic testing of precise
  • 17:02 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Remove duplicate routing code'
  • 16:55 logmsgbot: reedy synchronized wmf-config/
  • 16:32 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable shorturl on tawikis'
  • 16:31 Reedy: Created ShortUrl table on numerous tawiki* wikis
  • 16:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable ShortUrl on tawiki, hiwiki and orwiki'
  • 16:26 Reedy: Add ShortUrl table to hiwiki, orwiki and tawiki
  • 16:05 paravoid: sync-apache/apache-graceful-all for RT #2121
  • 16:02 Reedy: Created FlaggedRevs tables on eswikiboooks
  • 15:58 logmsgbot: reedy synchronized wmf-config/
  • 15:53 Reedy: Created FlaggedRevs tables on eswiki
  • 08:11 hashar: gallium/jenkins: deployed job for the WLMMobile nightly builds
  • 07:40 hashar: gallium/jenkins: updating Android SDKs
  • 02:46 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Mon Jul 30 02:45:58 UTC 2012
  • 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Mon Jul 30 02:23:05 UTC 2012
  • 01:00 Tim: installed mytop on fenari at faidon's request
  • 00:52 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'docs'
  • 00:49 Tim: killed all AFTv5 queries another few times
  • 00:49 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'disabled properly'
  • 00:45 Tim: killed ArticleFeedback queries on db12
  • 00:44 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'disabled ArticleFeedback since it caused an overload on db12 and general site slowness'
  • 00:32 Tim: on hume: kill -CONT
  • 00:28 Tim: on hume: stopped populateRevisionSha1.php with kill -STOP due to excessive (800s) lag on db12

July 29

  • 23:13 logmsgbot: reedy synchronized php-1.20wmf8/includes/WebRequest.php
  • 02:45 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Sun Jul 29 02:45:34 UTC 2012
  • 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Sun Jul 29 02:23:12 UTC 2012

July 28

  • 06:29 logmsgbot: catrope synchronized php-1.20wmf8/extensions/Vector/modules/ext.vector.collapsibleNav.css 'touch'
  • 06:29 logmsgbot: catrope synchronized php-1.20wmf7/extensions/Vector/modules/ext.vector.collapsibleNav.css 'touch'
  • 02:51 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Sat Jul 28 02:51:29 UTC 2012
  • 02:25 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Sat Jul 28 02:25:46 UTC 2012

July 27

  • 21:16 RobH: updated dns for mgmt on wtp1.mgmt
  • 20:42 cmjohnson1: setting db66
  • 20:23 RobH: authdns-udpate for yttrium/wlm.w.o per rt 3221
  • 20:12 cmjohnson1: setting up tmh2 rt3298
  • 19:51 cmjohnson1: setting up tmh1 rt 3298
  • 19:40 RobH: setting up yttrium per rt 3221
  • 19:32 logmsgbot: aaron synchronized php-1.20wmf7/thumb.php 'deployed 3f79e142beade8135514a3ac2a1af2cdd8a30901'
  • 19:31 logmsgbot: aaron synchronized php-1.20wmf7/includes/DefaultSettings.php 'deployed 3f79e142beade8135514a3ac2a1af2cdd8a30901'
  • 19:31 logmsgbot: aaron synchronized php-1.20wmf7/includes/filerepo/FileRepo.php 'deployed 3f79e142beade8135514a3ac2a1af2cdd8a30901'
  • 19:26 logmsgbot: aaron synchronized php-1.20wmf8/thumb.php 'deployed 718e305c2213125b1df41323f22b7db400a77139'
  • 19:25 logmsgbot: aaron synchronized php-1.20wmf8/includes/filerepo/FileRepo.php 'deployed 718e305c2213125b1df41323f22b7db400a77139'
  • 19:24 logmsgbot: aaron synchronized php-1.20wmf8/includes/DefaultSettings.php 'deployed 718e305c2213125b1df41323f22b7db400a77139'
  • 19:00 RobH: allocating potassium and zirconium to rt 3342 (misc db use) and shutting them down until they are installed.
  • 18:37 RobH: another authdns-update, forgot to update asset tags to servernames for mgmt
  • 18:33 RobH: authdns-update for tmh1/2
  • 14:07 hashar: srv281: rsync: write failed on "/apache/common-local/wmf-config/InitialiseSettings.php": No space left on device (28)
  • 14:07 hashar: srv281: rsync: write failed on "/apache/common-local/wmf-config/InitialiseSettings.php": No space left on device (28)
  • 14:07 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '16817 - fix nycwikimedia wgImportSources'
  • 13:40 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Deploying I39d1ca7c: import sources for nycwikimedia'
  • 12:51 Reedy: Created tgs_lang indexes on wikis that have translate installed
  • 02:53 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Fri Jul 27 02:53:30 UTC 2012
  • 02:31 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Fri Jul 27 02:31:05 UTC 2012

July 26

  • 22:26 logmsgbot: kaldari synchronized php-1.20wmf7/extensions/E3Experiments/Experiments.hooks.php 'syncing Experiments.hooks.php for wmf7'
  • 22:25 logmsgbot: kaldari synchronized php-1.20wmf8/extensions/E3Experiments/Experiments.hooks.php 'syncing Experiments.hooks.php for wmf8'
  • 22:00 logmsgbot: reedy synchronized wmf-config/
  • 21:55 logmsgbot: reedy synchronized php-1.20wmf7/LocalSettings.php
  • 21:48 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'turning E3Experiments on for en.wiki'
  • 21:38 logmsgbot: kaldari Finished syncing Wikimedia installation... :
  • 21:17 logmsgbot: kaldari Started syncing Wikimedia installation... :
  • 21:09 logmsgbot: kaldari synchronized php-1.20wmf8/extensions/E3Experiments 'sync E3Experiments stuff for en.wiki'
  • 21:02 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'turning on E3Experiments for en.wiki'
  • 19:40 RobH: changed to calcium instead, as yttrium is a 610 and i only need a 310
  • 19:37 RobH: claiming yttrium for smokeping install
  • 19:23 RobH: authdns update for new services
  • 18:59 logmsgbot: reedy synchronized php-1.20wmf8/extensions/WikiEditor
  • 16:53 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 16:52 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 16:46 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Special -'
  • 16:43 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'enable shorturl on testwiki and test2wiki'
  • 16:42 Reedy: Created ShortUrl tables on test2wiki
  • 13:44 hashar: deployment-prep rsync finished for both apache and upload6. Remounting and restarting apaches
  • 13:23 cmjohnson1: search32 has hardware issues..powering down
  • 08:43 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php '(bug 32516) Enable Narayam on pawiki.'
  • 08:38 logmsgbot: nikerabbit synchronized php-1.20wmf7/extensions/Narayam/ 'Deploying https://gerrit.wikimedia.org/r/16721'
  • 02:46 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Thu Jul 26 02:46:34 UTC 2012
  • 02:24 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Thu Jul 26 02:24:01 UTC 2012
  • 00:21 logmsgbot: catrope synchronized php-1.20wmf8/skins/common/commonElements.css 'Deploy 0cc26386cec1955acf2e1f7341eaaa466ea220b2 and 162a25e96cf48cbd2545f6c3d54f0c20f1bcd520'
  • 00:08 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Undo testing change'

July 25

  • 23:46 logmsgbot: reedy synchronized php-1.20wmf8/extensions/AbuseFilter/
  • 23:30 binasher: completed abuse_filter_log migration for all wikis
  • 22:54 logmsgbot: aaron synchronized php-1.20wmf7/thumb.php
  • 22:29 LeslieCarr: fixed puppetmaster on stafford (bad stafford, no cookie!)
  • 22:20 logmsgbot: aaron synchronized php-1.20wmf7/thumb.php 'url extraction fix'
  • 22:15 logmsgbot: aaron synchronized php-1.20wmf7/thumb.php 'debug logging'
  • 22:08 logmsgbot: aaron synchronized php-1.20wmf7/thumb.php 'url regex tweak'
  • 22:06 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Temporarily set $wmgArticleFeedbackv5OversightEmails to my work e-mail for debugging'
  • 22:05 binasher: started second step of abuse_filter_log migration via osc, indexes for new columns
  • 21:56 logmsgbot: aaron synchronized php-1.20wmf7/thumb.php 'debug logging'
  • 21:08 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'Re-implementing wgNoticeFundraisingUrl override for wmf7 wikis'
  • 21:02 logmsgbot: aaron synchronized live-1.5/thumb_handler.php 'entry point to thumb.php'
  • 20:43 binasher: started abuse_filter_log migration via osc. first step, adding afl_rev_id, afl_log_id columns. running in cluster order starting with s1 (enwiki)
  • 20:30 logmsgbot: reedy synchronized php-1.20wmf8/includes/objectcache/MemcachedClient.php 'Bring in reversion to shut up warnings'
  • 20:26 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: all none wikipedia projects to 1.20wmf8
  • 20:21 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 19:49 logmsgbot: reedy synchronized php-1.20wmf8/extensions/AbuseFilter/ 'Revert back to 0eeafeab44fb7a177ebf929c313999855656701f'
  • 18:25 logmsgbot: reedy synchronized php-1.20wmf8/extensions/Quiz/
  • 18:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity, closed and special to 1.20wmf8
  • 18:10 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews to 1.20wmf8
  • 18:02 binasher: streaming hot backup of db1033 to db63, intended db12 replacement and first precise enwiki db
  • 17:10 mark: Added asw-c-eqiad to Torrus, RANCID, Observium
  • 16:52 cmjohnson1: srv266 still working on it...going down
  • 15:46 logmsgbot: reedy synchronized wmf-config/
  • 15:34 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 15:33 Reedy: Created wikilove tables on svwikinews
  • 15:24 Reedy: Created FlaggedRevs tables on cawikinews
  • 15:23 logmsgbot: reedy synchronized wmf-config/
  • 14:37 cmjohnson1: srv266 shutting down for attempted h/w repairs
  • 02:46 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Wed Jul 25 02:46:06 UTC 2012
  • 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Wed Jul 25 02:23:44 UTC 2012

July 24

  • 23:23 logmsgbot: awjrichards Finished syncing Wikimedia installation... : Synchronizing updates to MobileFrontend per http://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-06-24
  • 22:52 logmsgbot: awjrichards Started syncing Wikimedia installation... : Synchronizing updates to MobileFrontend per http://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2012-06-24
  • 22:26 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Fixing footer logo for enwiki on mobile view'
  • 20:10 binasher: upgrading versions of xtrabackup and percona toolkit on all coredbs
  • 19:45 logmsgbot: mlitn Finished syncing Wikimedia installation... :
  • 19:10 logmsgbot: mlitn Started syncing Wikimedia installation... :
  • 18:30 RoanKattouw: root@fenari:~# chmod g+w /home/wikipedia/common/php-1.20wmf8/cache/l10n/
  • 18:30 RoanKattouw: root@fenari:~# chown l10nupdate:wikidev /home/wikipedia/common/php-1.20wmf8/cache/l10n
  • 18:04 logmsgbot: reedy synchronized php-1.20wmf8/extensions/AbuseFilter/
  • 16:28 logmsgbot: reedy synchronized php-1.20wmf8/extensions/AbuseFilter/
  • 16:08 RobH: srv278 hardware worked on by chris, placing back in service to see if its going to stay fixed
  • 14:30 cmjohnson1: correcting enviromental leads on powestrips in sdtpa
  • 14:06 paravoid: force puppet run on all LVS servers for the ganglia change
  • 09:10 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Bug 36104 - Narayam on bnwikisource'
  • 02:48 logmsgbot: LocalisationUpdate completed (1.20wmf8) at Tue Jul 24 02:48:21 UTC 2012
  • 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Tue Jul 24 02:26:52 UTC 2012
  • 00:03 binasher: upgrading db1045 to precise for testing

July 23

  • 23:11 AaronSchulz: Started originals migrations for remaining wikis
  • 22:07 binasher: deploying new mobile redirector to eqiad squids
  • 20:44 cmjohnson1: ms-be10 taking off-line for some hardware testing
  • 20:13 logmsgbot: reedy synchronized php-1.20wmf7/includes/specials/SpecialUserlogin.php
  • 20:12 logmsgbot: reedy synchronized php-1.20wmf8/includes/specials/SpecialUserlogin.php
  • 19:31 RobH: authdns-update for transcode name updates
  • 19:28 RobH: powering down old transcode1, was camera gateway sandbox, reclaiming name
  • 18:57 maplebed: took ms-be10 out of rotation because it ate itself
  • 18:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.20wmf8
  • 17:03 RobH: reobooting potassium, shouldnt be in use, had tons of cruft ssh connections from month ago
  • 16:26 Reedy: running ddsh -cM -g mediawiki-installation -o -oSetupTimeout=30 -F30 -- "sudo -u mwdeploy rm -rf /usr/local/apache/common/php-1.20wmf5"
  • 16:23 Reedy: running ddsh -cM -g mediawiki-installation -o -oSetupTimeout=30 -F30 -- "sudo -u mwdeploy rm -rf /usr/local/apache/common/php-1.20wmf4"
  • 16:22 logmsgbot: reedy Finished syncing Wikimedia installation... : 1.20wmf8 messages
  • 16:04 paravoid: reinstalling srv281; disk full, hasn't run puppet for 300 days, depooled for ages
  • 15:54 logmsgbot: reedy Started syncing Wikimedia installation... : 1.20wmf8 messages
  • 15:44 logmsgbot: reedy Started syncing Wikimedia installation... : 1.20wmf8 messages
  • 15:42 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.20wmf8 and rebuild localisation cache
  • 15:23 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.20wmf8 and rebuild localisation cache
  • 15:21 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: rm chwikimedia we dont host it
  • 15:19 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: chwikimedia to 1.20wmf7
  • 15:14 Reedy: srv281 has a full / and hasn't had a puppet run in over 434298 minutes
  • 15:10 logmsgbot: reedy synchronized php-1.20wmf8/cache/
  • 15:06 logmsgbot: reedy synchronized php-1.20wmf8/ 'Initial sync out of 1.20wmf8'
  • 13:44 RobH: authdns-update for ms-be eqiad hosts
  • 12:46 maplebed: rebooting ms-be10 for xfs errors and a clean boot
  • 12:38 maplebed: rebalanced swift rings moving more content to new object servers
  • 12:26 logmsgbot: reedy synchronized flaggedrevs.dblist
  • 12:26 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php
  • 12:25 Reedy: Created trwikiquote flagged revs tables
  • 12:23 logmsgbot: reedy synchronized wmf-config/
  • 02:22 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Mon Jul 23 02:22:45 UTC 2012
  • 02:22 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Mon Jul 23 02:22:33 UTC 2012

July 22

  • 22:56 Ryan_Lane: restarted opendj on sanger. The process OOM'd due to heap size.
  • 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Sun Jul 22 02:23:02 UTC 2012
  • 02:22 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Sun Jul 22 02:22:49 UTC 2012

July 21

  • 17:42 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'deployed ffd469239a80b8bb9813eb029e4cd71a3ea52db9'
  • 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Sat Jul 21 02:23:19 UTC 2012
  • 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Sat Jul 21 02:23:06 UTC 2012

July 20

  • 18:57 paravoid: powerin on owa1/owa2 again, they're being used as ganglia aggregators for swift
  • 18:11 Ryan_Lane: starting gerrit
  • 17:59 binasher: db48 is now replicating from db1048 (and db1048 from db48)
  • 17:26 RobHalsell: authdns-update run for new servers
  • 15:29 paravoid: powering off the rest of mw10[0-9][0-9]
  • 14:35 mark: Added server hydrogen to the dns_rec eqiad LVS pool
  • 14:22 paravoid: powering off owa1/2/3, unused
  • 13:54 paravoid: powering off all of mw1[0-9][0-9][0-9].eqiad.wmnet, unused
  • 13:36 mark: Reinstalling hydrogen
  • 13:19 paravoid: powercycling hydrogen, down since yesterday
  • 12:59 paravoid: powercycling nescio/ns2, unresponsive network & console
  • 10:38 mark: Built new varnish 3.0.3~rc1+persistent1-wm1 packages and inserted them into the precise-wikimedia APT repository
  • 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Fri Jul 20 02:23:39 UTC 2012
  • 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Fri Jul 20 02:23:27 UTC 2012

July 19

  • 22:48 logmsgbot: catrope Finished syncing Wikimedia installation... :
  • 22:10 logmsgbot: catrope Started syncing Wikimedia installation... :
  • 22:02 binasher: deploying new mobile redirector to pmtpa text squids (currently inactive)
  • 21:39 logmsgbot: kaldari Started syncing Wikimedia installation... :
  • 21:36 binasher: deploying new mobile redirector to eqiad text squids
  • 21:36 logmsgbot: catrope synchronized php-1.20wmf7/extensions/LastModified 'Remove E3Experiments cruft from LastModified'
  • 21:07 binasher: deploying new mobile redirector to esams text squids
  • 18:52 preilly: fixing subdomain check for zero vs m
  • 18:52 logmsgbot: preilly synchronized wmf-config/mobile.php 'fix subdomain check'
  • 18:21 RobH: updating dns with new zonefiles for legally won domain names
  • 18:15 Jeff_Green: storage3 dist-upgrade and reboot
  • 17:01 AaronSchulz: Running copyFileBackend.php for commons (shards c-f)
  • 16:28 cmjohnson1: mw60 powering down to replace DIMM B1 rt3287
  • 15:48 RobH: hydrogen repaired per rt 3243
  • 15:37 RobH: rebooting hydrogen to set bios redirection
  • 14:18 logmsgbot: reedy synchronized wmf-config/
  • 13:42 Jeff_Green: dist-upgrade and reboot hosts in payments cluster
  • 13:12 Reedy: Pointed /h/w/c/php to php-1.20wmf7
  • 13:10 Reedy: Deleted php-1.20wmf6/cache/l10n from mediawiki-installation
  • 02:31 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Thu Jul 19 02:31:02 UTC 2012
  • 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Thu Jul 19 02:23:27 UTC 2012

July 18

  • 22:02 notpeter: ok, changed my mind one more time. removing srv194 from apaches pool for precise testing
  • 21:41 notpeter: scratch that. removing srv289 from apaches pool for precise testing, not mw1
  • 21:39 notpeter: removing mw1 from apaches pool to do precise test install
  • 21:28 notpeter: adding php5 packages to precise-wikimedia repo
  • 19:45 cmjohnson1: srv281 powering down for HW checks
  • 19:09 cmjohnson1: srv278 powering down for hardware problems -- random reboot system already depooled
  • 18:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: 295 remaining wikis to 1.20wmf7
  • 17:37 cmjohnson1: mw8 powering down to replace DIMM A3 rt 3273
  • 17:25 cmjohnson1: search32 powering down to replace DIMM B1 rt 3076
  • 16:55 logmsgbot: reedy synchronized php-1.20wmf7/includes/api/ApiQuerySiteinfo.php
  • 12:52 pp-pdf1: upgraded mwlib to 0.13.11, restarted all services
  • 07:58 hashar: gallium: added Firefox 14 to Testswarm, disabled Firefox 13.
  • 02:47 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Wed Jul 18 02:47:23 UTC 2012
  • 02:24 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Wed Jul 18 02:24:14 UTC 2012

July 17

  • 23:24 Tim: on srv193: removed core dump files, disabled core dumping, restarted apache
  • 19:52 logmsgbot: mlitn Finished syncing Wikimedia installation... :
  • 19:09 logmsgbot: mlitn Started syncing Wikimedia installation... :
  • 15:18 mark: lvs1002 is back up and idling
  • 15:15 mark: Fixed serial console redirection after boot to OFF on lvs1002
  • 15:06 mark: lvs4 is back up and serving traffic
  • 14:51 mark: Reinstalling lvs4 with Ubuntu Precise
  • 14:48 mark: Stopped PyBal on lvs4, failing over traffic to lvs3
  • 14:47 mark: Reinstalling lvs1002 with Ubuntu Precise
  • 14:46 mark: Stopped PyBal on lvs1002, failing over traffic to lvs1005
  • 14:44 mark: lvs1001 is back up and serving traffic
  • 14:11 mark: Stopped PyBal on lvs1001, failing over traffic to lvs1004
  • 13:09 maplebed: changed auth URL for swift to use load balancer rather than round robin DNS
  • 13:08 mark: lvs1003 is back up and serving traffic
  • 13:07 maplebed: changed order on ganglia swift view graphs to group by metric rather than host
  • 13:02 logmsgbot: reedy synchronized wmf-config/
  • 12:53 mark: Fixed boot order on lvs1003
  • 12:34 mark: Reinstalling lvs1003 with Ubuntu Precise
  • 12:28 mark: Stopped PyBal on lvs1003, failing over traffic to lvs1006
  • 12:10 mark: amslvs1 is back up and serving traffic
  • 11:44 mark: Reinstalling amslvs1 with Ubuntu Precise
  • 11:40 mark: Stopped PyBal on amslvs1, failing over traffic to amslvs3
  • 11:32 mark: amslvs2 is back up and serving traffic
  • 10:47 mark: Reinstalling amslvs2 with Ubuntu Precise
  • 10:36 mark: Stopped PyBal on amslvs2, failing over traffic to amslvs4
  • 07:30 Tim: testing envvars/apache2.conf change on srv193
  • 07:09 Tim: restarted apache on srv193
  • 02:58 Tim: graceful restart of all apaches
  • 02:52 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 02:47 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Tue Jul 17 02:47:06 UTC 2012
  • 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Tue Jul 17 02:23:41 UTC 2012
  • 00:44 Tim: deploying new redirects.conf
  • 00:28 Tim: testing new redirects.conf on mw1

July 16

  • 23:03 pp-pdf1: restarted nserve
  • 23:02 pp-pdf1: fixed nserve handling of filenames with whitespace
  • 22:46 pp-pdf1: restart all services
  • 22:45 pp-pdf2: restart all services
  • 22:45 pp-pdf3: restart all services
  • 22:45 pp-pdf1: update mwlib to 0.13.9
  • 22:45 pp-pdf2: update mwlib to 0.13.9
  • 22:45 pp-pdf3: update mwlib to 0.13.9
  • 22:44 pp-pdf2: update greenlet to 0.4.0
  • 22:44 pp-pdf1: update greenlet to 0.4.0
  • 22:44 pp-pdf3: update greenlet to 0.4.0
  • 22:44 pp-pdf2: upgrade qserve to 0.2.8
  • 22:44 pp-pdf3: upgrade qserve to 0.2.8
  • 22:43 pp-pdf1: upgrade qserve to 0.2.8
  • 22:20 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Disabling epub generation from Collection extension on all wikis except for simple and test'
  • 22:19 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Making epub generation for Collection extension configurable'
  • 18:26 AaronSchulz: Running copyFileBackend.php for commons (shards 8-b)
  • 18:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf7
  • 17:08 LeslieCarr: reactivating XO transit connections on cr1-sdtpa
  • 17:03 LeslieCarr: draining cr2-eqiad to cr1-sdtpa link for moving of fiber
  • 15:56 mark: lvs3 is back up, and idling
  • 15:56 mutante: fixing pt.planet after missing locale has been added
  • 15:48 mutante: installing package upgrades on singer (planet)
  • 15:43 cmjohnson1: mw8 shutting down and taking offline to run Dell's dset program
  • 15:38 mutante: planet - svn up the config
  • 15:29 mark: Reinstalling lvs3 with Ubuntu Precise
  • 15:17 mark: lvs5 is back up and serving traffic
  • 14:49 mark: Reinstalling lvs5 with Ubuntu Precise
  • 14:45 mark: Stopped PyBal on lvs5 to failover traffic to lvs1
  • 14:35 maplebed: adjusted swift rings; set new object servers to 20, new container servers to 100
  • 14:33 mark: lvs6 is back up and serving traffic
  • 14:12 mark: Reinstalling lvs6 with Ubuntu Precise
  • 14:02 mark: Stopped PyBal on lvs6 to failover traffic to lvs2
  • 13:58 mutante: mw8 - alright, most likely just needs new DIMM per cmjohnson
  • 13:50 mutante: mw8 PHP fatal errors, running out of memory
  • 12:56 mutante: sync-apache, graceful-all to fix wikipedia.cz redirect
  • 05:31 apergos: reboot to upgrade kernel etc on mw8 since it's been flapping anyways
  • 02:46 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Mon Jul 16 02:46:14 UTC 2012
  • 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Mon Jul 16 02:23:04 UTC 2012

July 15

  • 20:19 Aaron|home: Running copyFileBackend.php for commons (shard 7)
  • 02:46 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Sun Jul 15 02:46:33 UTC 2012
  • 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Sun Jul 15 02:23:27 UTC 2012

July 14

  • 18:46 LeslieCarr: powercycling frozen db1029
  • 18:01 LeslieCarr: rebooting unresponsive mw1111
  • 16:41 logmsgbot: midom synchronized php-1.20wmf6/includes/GlobalFunctions.php 'message key analysis'
  • 16:18 logmsgbot: midom synchronized php-1.20wmf6/includes/GlobalFunctions.php 'message key analysis'
  • 03:07 Aaron|home: Running copyFileBackend.php for commons (shards 5-6)
  • 02:47 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Sat Jul 14 02:47:32 UTC 2012
  • 02:23 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Sat Jul 14 02:23:14 UTC 2012
  • 00:25 logmsgbot: reedy synchronized php-1.20wmf7/includes/api/ApiDelete.php

July 13

  • 18:08 logmsgbot: reedy synchronized php-1.20wmf7/includes/GlobalFunctions.php
  • 18:08 logmsgbot: reedy synchronized php-1.20wmf6/includes/GlobalFunctions.php
  • 17:58 logmsgbot: reedy synchronized wmf-config/
  • 17:57 logmsgbot: reedy synchronized php-1.20wmf7/
  • 17:56 logmsgbot: reedy synchronized php-1.20wmf6/
  • 17:32 logmsgbot: reedy synchronized wmf-config/
  • 17:21 AaronSchulz: Running copyFileBackend.php for commons (shards 1-4) (actually started yesterday)
  • 15:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable aftv5 on en_labswikimedia due to out of date schema'
  • 15:21 logmsgbot: reedy synchronized php-1.20wmf7/includes/GlobalFunctions.php
  • 15:20 logmsgbot: reedy synchronized php-1.20wmf6/includes/GlobalFunctions.php
  • 15:20 logmsgbot: reedy synchronized php-1.20wmf7/includes/DefaultSettings.php
  • 15:19 logmsgbot: reedy synchronized php-1.20wmf6/includes/DefaultSettings.php
  • 15:17 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Setting wgDBerrorLogInUTC = true'
  • 15:07 hasharDeadmau5: unknown column 'af_is_featured' on en_labswikimedia@db25
  • 14:55 mark: Increased all mobile varnish server weights from 10 to 100 to aid chash
  • 14:54 mark: Added cp1041.eqiad.wmnet back into the mobile LVS pool
  • 10:10 mutante: mw1016 - was down, reinstalling with precise
  • 02:46 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Fri Jul 13 02:45:59 UTC 2012

July 12

  • 22:15 Aaron|home: Running copyFileBackend.php for commons (shards 0,c-f)
  • 20:58 Aaron|home: Running copyFileBackend.php for commons (shards 0,8-b)
  • 14:22 mutante: apache children on srv193 keep segfaulting since yesterday (test.wp)
  • 14:00 RobHalsell: shutting down srv206 for chris per rt241
  • 11:55 logmsgbot: reedy synchronized wmf-config/
  • 10:40 mark: Depooled mobile varnish server cp1041 for vlan/hostname change and reinstallation with precise
  • 04:38 binasher: started hot backup of db48 to db1048
  • 02:50 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Thu Jul 12 02:50:49 UTC 2012
  • 02:29 maplebed: adjusted the swift rings a bit to move some more traffic to the new hosts. set object partitions to weight 10, account and container 60.
  • 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Thu Jul 12 02:26:57 UTC 2012

July 11

  • 22:52 AaronSchulz: Running copyFileBackend.php for commons (shards 0,4-7)
  • 22:06 AaronSchulz: Running copyFileBackend.php for commons (shards 0,0-3)
  • 21:36 binasher: running hotbackup of db48 -> db49 (otrs / external misc)
  • 21:31 binasher: rebooting db49 for kernel upgrade
  • 21:23 binasher: rebuilding db49 (otrs slave)
  • 21:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 20:16 logmsgbot: reedy synchronized php-1.20wmf7/extensions/CentralNotice
  • 19:33 cmjohnson1: db10 swapping disk1 out rt3251
  • 19:33 RobH: shutting down srv203 for hardware checking per rt3110
  • 19:12 hashar: finally stopped using my production ssh key on labs.
  • 19:11 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote and wikiversity to 1.20wmf7
  • 19:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks and wikinews to 1.20wmf7
  • 19:06 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary and wikisource to 1.20wmf7
  • 19:03 cmjohnson1: search35 shutting down to reseat DIMM rt-3260
  • 18:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Switch special, private and closed wikis from 1.20wmf6 to 1.20wmf7
  • 15:16 logmsgbot: robh synchronized wmf-config/mc.php 'moving srv208 to down'
  • 15:14 logmsgbot: robh synchronized wmf-config/mc.php
  • 14:30 cmjohnson1: shutting down search32 to reseat DIMM rt-3076
  • 11:16 mutante: yay @ svn removal
  • 11:13 hashar: /home/wikipedia/conf/httpd cleaned out the svn repository. That directory is now 100% under git tracking yeah!!! Thanks TIm!
  • 11:11 mutante: rebooting srv266
  • 11:05 mutante: purged wikipedie.cz/Experti_na_prirodu URLs with purgeList.php
  • 10:52 mutante: apache-graceful-all
  • 10:48 mutante: sync-apache to push fixed wikipedie.cz redirect
  • 08:36 hashar: synced /h/w/conf/httpd git and svn repositories
  • 08:33 mutante: chmod -R g+w /home/wikipedia/conf/httpd on fenari to fix group write on .git/objects
  • 07:59 mutante: srv266 - package/kernel upgrades
  • 07:35 mutante: powercycled downed owa3, installed kernel upgrades on owa1-3 (but they are also waiting to be repurposed, RT:2511)
  • 04:05 Tim: on hume: running updateCollation.php --dry-run on ptwiki to determine whether there will be any key truncation issues with bug 35632
  • 02:48 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Wed Jul 11 02:48:18 UTC 2012
  • 02:25 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Wed Jul 11 02:25:19 UTC 2012

July 10

  • 23:47 AaronSchulz: Doing shards c-f
  • 23:47 maplebed: put two new swift front ends into rotation ms-fe3 and ms-fe4
  • 23:40 AaronSchulz: Doing shards 8-b
  • 23:34 AaronSchulz: Doing shards 4-7
  • 23:31 AaronSchulz: copyFileBackend.php run rate for above processes is at /home/aaron/NFStoSwiftCopyRate, currently 50
  • 23:26 AaronSchulz: Running copyFileBackend.php for zhwiki public zone shards 0-3 on hume
  • 23:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 22:17 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 22:12 logmsgbot: asher synchronized wmf-config/CommonSettings.php 'moving parsercache from db40 to pc1'
  • 22:05 maplebed: put swift backends ms-be9-12 in rotation for containers (weight 30) and objects (weight 5)
  • 21:17 maplebed: rebooting new ms-be6-8 to change BIOS setting to boot from disk
  • 20:26 logmsgbot: mlitn Finished syncing Wikimedia installation... :
  • 20:26 RobH: shutting down sq36 for hardware troubleshooting
  • 19:38 LeslieCarr: restarted frozen pdns on ns2
  • 19:28 logmsgbot: mlitn Started syncing Wikimedia installation... :
  • 19:15 cmjohnson1: sq36-requires a hard shutdown-unresponsive to mgmt. rt-3254
  • 19:05 RoanKattouw: chmod g+w /home/wikipedia/common/php-1.20wmf7/cache/l10n/
  • 19:03 RoanKattouw: chown l10nupdate:wikidev /home/wikipedia/common/php-1.20wmf7/cache/l10n/
  • 18:58 hashar: reworked the git repository in /home/wikipedia/conf/httpd , manually synced changes from svn to the git repo
  • 18:12 RobH: srv266 shutdown for chris
  • 18:06 cmjohnson1: srv266 being brought down for an extended period of time to run diagnostic tests
  • 17:25 cmjohnson1: srv266 shutting down for HW troubleshooting rt-2896
  • 16:33 mutante: package upgrades and kernel on niobium
  • 16:28 mutante: powercycling niobium
  • 15:49 mutante: powercycling mw1008,mw1070,mw1073
  • 14:10 logmsgbot: reedy synchronized hastidy
  • 13:26 logmsgbot: tstarling synchronized live-1.5/robots.php
  • 13:17 logmsgbot: tstarling synchronized live-1.5/favicon.php
  • 13:13 logmsgbot: tstarling synchronized live-1.5/favicon.php
  • 12:53 Tim: deploying favicon.php test alias
  • 12:41 logmsgbot: tstarling synchronized live-1.5/favicon.php
  • 12:30 logmsgbot: tstarling synchronized live-1.5/favicon.php
  • 12:11 logmsgbot: tstarling synchronized live-1.5/favicon.php
  • 12:10 logmsgbot: tstarling synchronized live-1.5/favicon.php
  • 12:03 logmsgbot: tstarling synchronized live-1.5/favicon.php
  • 12:02 logmsgbot: tstarling synchronized live-1.5/favicon.php
  • 11:58 logmsgbot: tstarling synchronized live-1.5/favicon.php
  • 10:10 logmsgbot: reedy synchronized images/sul/ 'crushed'
  • 10:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 07:49 logmsgbot: tstarling synchronized live-1.5/favicon.php
  • 07:46 logmsgbot: tstarling synchronized live-1.5/favicon.php
  • 07:41 logmsgbot: tstarling Finished syncing Wikimedia installation... :
  • 06:32 logmsgbot: tstarling Started syncing Wikimedia installation... :
  • 06:14 logmsgbot: tstarling Started syncing Wikimedia installation... :
  • 06:06 Tim: removing php-*/cache/l10n/l10nupdate* and running scap with the new version of scap from I8bcd2817
  • 05:01 Tim: on fenari: running git submodule update --init in /var/lib/l10nupdate/mediawiki/
  • 03:56 Tim: testing new scheme for LU involving not pushing out LU files to all apaches
  • 02:49 logmsgbot: LocalisationUpdate completed (1.20wmf7) at Tue Jul 10 02:49:16 UTC 2012
  • 01:53 brion: test.wikipedia.org fails with 'TrustedXFF: hosts file missing.'
  • 01:44 Tim: on manganese: ran date -s "`date`" to make sure that isn't the cause of the high CPU usage during clone, it wasn't
  • 01:44 Tim: running rebuildLocalisationCache.php for testwiki
  • 01:43 brion: test.wikipedia.org broken again, "No localisation cache found for English."
  • 00:51 brion: test.wikipedia is broken loading Cite.php

July 9

  • 23:53 Tim: deleting and recreating php-1.20wmf7 to test a script
  • 20:51 logmsgbot: reedy synchronized php-1.20wmf7/includes/media/FormatMetadata.php
  • 20:51 logmsgbot: reedy synchronized php-1.20wmf6/includes/media/FormatMetadata.php
  • 20:30 pp-pdf3: upgraded mwlib.epub
  • 20:30 pp-pdf1: upgraded mwlib.epub
  • 20:30 pp-pdf2: upgraded mwlib.epub
  • 20:27 cmjohnson1: db10 swapping disk1 for new disk
  • 19:41 logmsgbot: reedy synchronized php-1.20wmf6/includes/WikiPage.php
  • 19:37 logmsgbot: reedy synchronized php-1.20wmf7/includes/WikiPage.php
  • 18:13 logmsgbot: reedy synchronized php-1.20wmf7/extensions/CodeReview/ 'CR to trunk to fix fatals'
  • 18:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki and testwiki to 1.20wmf7
  • 15:45 logmsgbot: reedy synchronized wmf-config/ExtensionMessages-1.20wmf7.php
  • 15:43 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.20wmf7 and rebuilding localisation caches
  • 15:01 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.20wmf7 and rebuilding localisation caches
  • 14:54 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.20wmf7 and rebuilding localisation caches
  • 14:47 logmsgbot: reedy synchronized php-1.20wmf7
  • 14:17 Reedy: Copying php-1.20wmf7 from /tmp to /h/w/c on fenari
  • 13:26 Reedy: Running ddsh -cM -g mediawiki-installation -o -oSetupTimeout=30 -F30 -- "sudo -u mwdeploy rm -rf /usr/local/apache/common/php-1.20wmf3"
  • 13:23 Reedy: Running ddsh -cM -g mediawiki-installation -o -oSetupTimeout=30 -F30 -- "sudo -u mwdeploy rm -rf /usr/local/apache/common/php-1.20wmf2"
  • 13:13 Reedy: Removed wmf2 and wmf3 from bits docroots
  • 13:11 mutante: powercycling and upgrading a couple more mw* servers
  • 13:04 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 13:01 mutante: labs.wikimedia.org is now a redirect to labsconsole
  • 12:49 mutante: authdns-update to add labs.wm entry for redirect to labsconsole
  • 08:34 logmsgbot: preilly synchronized wmf-config/CommonSettings.php 'adjust $wgCollectionFormats'
  • 08:16 hashar: pallium, updating jenkins build script with 14666 & 14667
  • 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Mon Jul 9 02:25:57 UTC 2012

July 8

  • 12:17 apergos: removed HTCPpurger.log.1 and current log ater restart of purger on ms6, /was full. people reporting thumb issues from europe
  • 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Sun Jul 8 02:26:21 UTC 2012

July 7

  • 16:39 logmsgbot: reedy synchronized docroot/bits/DolphinBrowser/js/ 'Remove BOM'
  • 16:10 logmsgbot: reedy synchronized php-1.20wmf6/extensions/cldr/ 'remove bom'
  • 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Sat Jul 7 02:27:22 UTC 2012

July 6

  • 21:49 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 38227 - Please enable WikiLove on Turkish Wikipedia'
  • 21:48 binasher: truncating pagetriage tables on enwiki (per bsitu)
  • 21:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 38227 - Please enable WikiLove on Turkish Wikipedia'
  • 21:46 Reedy: Created WikiLove tables on trwiki
  • 21:39 logmsgbot: aaron synchronized php-1.20wmf6/includes/WikiPage.php
  • 21:07 maplebed: removed a bit more load from the swift spinning media container servers by adjusting the ring weights
  • 20:45 logmsgbot: aaron synchronized php-1.20wmf6/includes/WikiPage.php 'rev debug logging'
  • 20:22 logmsgbot: aaron synchronized php-1.20wmf6/includes/Revision.php 'es debug logging'
  • 20:20 logmsgbot: aaron synchronized php-1.20wmf6/includes/WikiPage.php 'es debug logging'
  • 19:49 logmsgbot: aaron synchronized php-1.20wmf6/includes/WikiPage.php 'debug logging'
  • 19:30 logmsgbot: aaron synchronized php-1.20wmf6/includes/WikiPage.php 'debug logging'
  • 19:24 logmsgbot: aaron synchronized php-1.20wmf6/includes/WikiPage.php 'debug logging'
  • 17:28 maplebed: adjusted swift ring files to move container listings off spinning disks (ms-be1-4)
  • 17:04 logmsgbot: reedy synchronized wmf-config/
  • 16:53 Reedy: Running updateCollation.php in foreachwiki on fenari in screen as reedy
  • 16:45 Reedy: Running updateCollation.php against enwiki on fenari in screen as reedy
  • 16:39 Reedy: Running updateCollation.php against commonswiki on fenari in screen as reedy
  • 16:08 logmsgbot: reedy synchronized images/wiki-en.png
  • 15:54 logmsgbot: reedy synchronized images/sul/meta.png
  • 15:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 15:36 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
  • 15:21 cmjohnson1: ms-be5 removing disk0 to replace with good disk
  • 14:38 RobHalsell: authdns-update to correct typo in mgmt dns entry
  • 13:22 mark: Inserted new pybal 1.04 package in the precise-wikimedia APT repository, and upgraded all precise LVS servers
  • 11:31 logmsgbot: tstarling Finished syncing Wikimedia installation... :
  • 11:25 logmsgbot: tstarling Started syncing Wikimedia installation... :
  • 11:20 Tim: running scap to delete docroot files per I48502d90 and Ie3afd137
  • 08:58 mutante: dist-upgrading (unused) db10xx servers
  • 07:43 Tim: also trusted-xff.phps
  • 07:39 Tim: removing some untracked junk from /home/wikipedia/common
  • 07:29 mutante: continue to restart and upgrade downed mw10xx servers
  • 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Fri Jul 6 02:26:22 UTC 2012
  • 01:35 binasher: updated squid redirector to cover wiki(quotes|books|versity)
  • 00:13 Ryan_Lane: restarting gerrit
  • 00:09 Ryan_Lane: force running puppet on manganese, it'll restart gerrit

July 5

  • 23:54 Ryan_Lane: force running puppet on formey and manganese, since a config change is involved, it's going to restart
  • 23:45 Ryan_Lane: restarting gerrit on manganese
  • 23:38 Ryan_Lane: upgrading gerrit on formey
  • 23:21 Ryan_Lane: updating database for gerrit
  • 23:09 Ryan_Lane: upgrading gerrit on manganese
  • 23:08 Ryan_Lane: stopping gerrit on manganese and disabling puppet
  • 23:07 Ryan_Lane: stopping gerrit on formey and disabling puppet
  • 22:47 logmsgbot: preilly synchronized wmf-config/mobile.php 'add subdomain check'
  • 22:34 logmsgbot: reedy synchronized wmf-config/
  • 22:17 logmsgbot: reedy synchronized wmf-config/ 'Tidying config for randomrootpage'
  • 21:55 logmsgbot: reedy synchronized wmf-config/ 'Strike 2'
  • 21:50 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'revert'
  • 21:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Random root page enabled everywhere but wikipedias'
  • 21:47 logmsgbot: kaldari Finished syncing Wikimedia installation... :
  • 21:34 logmsgbot: kaldari Started syncing Wikimedia installation... :
  • 21:18 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'Bumping AFTv5 lottery percentage to 20% of en.wiki'
  • 21:15 LeslieCarr: powercycling unresponsive mw1116
  • 21:09 maplebed: added new ms-be pmtpa hosts to DNS
  • 20:03 logmsgbot: preilly synchronized php-1.20wmf6/extensions/MobileFrontend 'fix cache issue with token'
  • 19:14 Jeff_Green: *somebody* purged binary logs on blondel
  • 18:58 logmsgbot: preilly synchronized php-1.20wmf6/extensions/MobileFrontend 'fix cache issue'
  • 18:54 mark: Upgraded pybal on all precise LVS servers
  • 18:43 mark: Built new pybal_1.03 package and inserted it into the precise-wikimedia APT repository
  • 17:41 mutante: powercycling db1048, mw1136, mw1046
  • 17:32 logmsgbot: midom synchronized wmf-config/InitialiseSettings.php 'reenabling db40'
  • 17:30 mutante: more powercycling and upgrading: mw1036, mw1134, mw1043 ..
  • 17:26 mutante: powercycling db1009,db1010
  • 17:22 mutante: powercycling db1027,db1028
  • 17:14 mutante: powercycling db1013
  • 16:50 mutante: powercycling downed db1015
  • 16:44 mutante: powercycling and upgrading more mw10xx servers, 1017,1023,1025 ...
  • 16:27 mutante: mw1002, mw1007,mw1009,mw1011 - crashed,powercycling,dist-upgrading+kernel,reboot
  • 16:18 mutante: powercycling mw1002
  • 15:37 mutante: argon back up with new kernel,mysql,grub,.. looks happy afaict
  • 15:33 mutante: argon (limesurvey) fscked, dist-upgrading
  • 15:28 mutante: powercycling argon
  • 15:25 hashar: updated Jenkins configuration on gallium : Updating f407ebe..4b669b9
  • 14:08 cmjohnson1: HUME replacing disk 0
  • 14:07 logmsgbot: midom synchronized wmf-config/InitialiseSettings.php 'disabling parser cache for now'
  • 11:54 mark: Inserted new pybal_1.02 package into APT distribution precise-wikimedia
  • 11:17 mark: Installed new pybal snapshot build for testing on lvs1005
  • 11:13 mutante: add missing Russian locales on singer, run localegen, run ru.planet update
  • 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Thu Jul 5 02:26:45 UTC 2012

July 4

  • 16:32 mutante: wikidata.org on now - redirect purged from squids
  • 15:43 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'imports'
  • 15:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable LQT on fiwikimedia'
  • 15:39 Reedy: Killed php-1.20wmf5 localisation cache from mediawiki-installation group
  • 15:37 Reedy: Created LQT tables on fiwikimedia
  • 15:25 mutante: wikidata.org works now, besides old redirect may still be cached on cp* boxes (not purged by purgeList.php via multicast?). http://www.wikidata.org/?notcached
  • 15:06 mutante: sync-common-file extract2.php, apache-graceful-all
  • 15:04 logmsgbot: dzahn synchronized extract2.php
  • 13:23 mutante: apache-graceful-all to add wikidata.org virtual host
  • 13:21 mutante: svn commiting gerrit 9874, sync-apache
  • 13:14 mutante: git pull in /h/w/common/docroot . adding wikidata.org files on fenari. , then "sync-docroot"
  • 12:44 logmsgbot: dzahn synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
  • 12:44 mutante: updating/syncing interwiki cache
  • 09:32 hashar: swift-container-auditor seems to get down from time. Nagios reporting 0 processes at 8:15am and 9:25am UTC (I guess it get restarted automatically by puppet)
  • 07:34 hashar: updating Jenkins copy of integration/jenkins from 0f069c3 to e264d1b. Bring new ant script + update to testswarm fetcher
  • 07:09 Tim: on srv193: ran dpkg --set-selections to revert holds on php5 packages and ran apt-get upgrade
  • 07:07 Tim: on srv193: fixing broken PHP packages causing puppet failure, nothing in the server admin log about them so I assume they were installed by accident
  • 06:21 Tim: deployed Idb6d9a8b and restarting apaches
  • 06:11 Tim: deployed Id7008681 and restarting apaches
  • 05:45 Tim: reniced apache processes to level 0
  • 05:04 Tim: deploying apache nice level change per RT #664
  • 03:59 Tim: on mw1: experimenting with renice methods for RT 664
  • 02:26 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Wed Jul 4 02:26:39 UTC 2012

July 3

  • 23:03 logmsgbot: mlitn synchronized php-1.20wmf6/extensions/ArticleFeedbackv5/
  • 22:36 logmsgbot: preilly synchronized php-1.20wmf6/extensions/MobileFrontend 'weekly update'
  • 22:27 hashar: updating testswarm submitter on gallium
  • 21:01 logmsgbot: mlitn Finished syncing Wikimedia installation... :
  • 20:37 logmsgbot: mlitn Started syncing Wikimedia installation... :
  • 20:18 logmsgbot: asher synchronized wmf-config/db.php 'lowering db32 weight'
  • 19:34 logmsgbot: asher synchronized wmf-config/db.php 'lowering db36 weight'
  • 19:31 logmsgbot: asher synchronized wmf-config/db.php 're-add db36, db32 (low weight), es3 (innodb)'
  • 18:11 RobH: virt1006 mgmt serial not set correctly, fixed
  • 17:51 RobH: investigating stat1001 power issue
  • 17:22 RobH: fluorine offlining to test disks
  • 17:22 RobH: pulling helium offline for disk testing with fluorine disks
  • 17:10 RobH: db1047 disk0 rebuild in progress
  • 17:04 RobH: replacing bad disk in db1047
  • 16:45 logmsgbot: reedy synchronized wmf-config/ 'Various config changes'
  • 16:30 logmsgbot: reedy synchronized docroot/mediawiki/xml/export-0.7.xsd
  • 16:15 Jeff_Green: silicon gets dist-upgrade & reboot
  • 16:15 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wgCheckSerialized is deaded'
  • 03:53 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 're-enable API action=purge on commonswiki'
  • 02:27 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Tue Jul 3 02:27:02 UTC 2012

July 2

  • 22:33 binasher: rebooted db36 for kernel upgrade
  • 22:25 logmsgbot: asher synchronized wmf-config/db.php 'temp pulling db36'
  • 22:02 brion: fun with routers in tampa, wikis down
  • 21:48 maplebed: rebooted emery - it's been unresponsive for 3 days.
  • 18:34 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
  • 18:30 hashar: set up ignore file in httpd configuration directory
  • 18:23 logmsgbot: reedy synchronized wmf-config/ 'Enable WikimediaShopLink on enwiki'
  • 18:11 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: 273 wikipedias to 1.20wmf6
  • 18:03 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf6
  • 15:21 logmsgbot: hashar synchronized wmf-config/CommonSettings.php '/etc/wikimedia-realm detection https://gerrit.wikimedia.org/r/13888'
  • 15:18 logmsgbot: hashar synchronized docroot/bits/static-master '(bug 37245) docroot 'static-master' for beta bits'
  • 15:04 mutante: authdns-update to switch jobs.wm redirect to wikimedia-lb to fix SSL cert mismatch (RT-3071)
  • 14:55 mark: Reboot of cr1-sdtpa did not fix the RE packet loss issue... therefore unlikely to be leap second related
  • 14:41 mark: Rebooting cr1-sdtpa
  • 14:37 mark: Shutdown PyBal BGP sessions on cr1-sdtpa
  • 14:34 mark: Shutdown BGP session to 2828 on cr1-sdtpa
  • 13:36 hashar: db12 suffering some 1400sec (and growing) replag. mysqldump in progress on that host.
  • 12:35 mutante: installing upgrades on fenari (linux-firmware linux-libc-dev..)
  • 12:27 mutante: rebooting gallium one more time to install kernel
  • 12:26 mutante: upgrading kernel on gallium
  • 12:23 logmsgbot: hashar synchronized live-1.5/CREDITS
  • 11:31 mark: Now we have packet loss within pmtpa/sdtpa... reverting change
  • 10:57 mark: Problems on one of two pmtpa-eqiad waves; raised OSPF metric to 60 to failover traffic to the other link
  • 10:50 Tim: fixing leap second issue on bastion1 by rebooting it
  • 10:47 Tim: fixed leap second issue on bastion-restricted
  • 09:56 Tim: fixing leap second issue on virt1,virt2,virt3,virt4,virt5
  • 09:52 Tim: fixing leap second issue on aluminium,gallium,manganese
  • 09:47 Tim: fixing leap second issue on formey,grosley,hooper,sanger,sockpuppet
  • 09:43 Tim: on fenari: fixed leap second issue with the mozilla method
  • 09:39 apergos: rebooting gallium, it's pretty unhappy (maybe related to leap second issue)
  • 08:14 logmsgbot: hashar: srv190 srv266 srv281 timeouts on sync-file
  • 08:14 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'Bug 37457 - fix import sources for viwikibooks'
  • 08:11 hashar: Stopped Jenkins on gallium. It is not doing anything anyway. Asked to reboot box RT #3208
  • 02:53 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Mon Jul 2 02:53:51 UTC 2012
  • 02:28 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Mon Jul 2 02:28:48 UTC 2012
  • 01:48 Tim: kill -CONT on populateRevisionSha1.php processes
  • 00:47 Tim: on nfs1: trying leap second fix suggested at https://bugzilla.mozilla.org/show_bug.cgi?id=769972#c5
  • 00:26 logmsgbot: tstarling synchronized wmf-config/db.php 'reduce db32 read load to zero due to persistent lag'
  • 00:12 Tim: switched enwiki back to r/w
  • 00:12 logmsgbot: tstarling synchronized wmf-config/db.php
  • 00:06 Tim: on hume: stopped all populateRevisionSha1.php processes with kill -STOP
  • 00:03 logmsgbot: reedy synchronized wmf-config/db.php 's1/enwiki into readonly'

July 1

  • 19:12 logmsgbot: reedy synchronized php-1.20wmf6/extensions/WikimediaMaintenance/ 'Update to master for hashar'
  • 17:55 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php 'more logging'
  • 17:45 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php 'more logging'
  • 17:43 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php 'more logging'
  • 17:32 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php
  • 17:30 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php
  • 16:53 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php
  • 16:48 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php 'logging'
  • 12:54 notpeter: also going to reboot all pmtpa search nodes. not in prod, but are still freaking out from leap second bug.
  • 05:33 logmsgbot: aaron synchronized php-1.20wmf5/includes/WikiPage.php 'logging'
  • 04:25 logmsgbot: LocalisationUpdate completed (1.20wmf5) at Sun Jul 1 04:25:25 UTC 2012
  • 04:06 Ryan_Lane: virt1000 is back up, rebooting virt0
  • 04:01 Ryan_Lane: rebooting virt1000
  • 03:16 logmsgbot: LocalisationUpdate completed (1.20wmf6) at Sun Jul 1 03:16:39 UTC 2012
  • 01:43 notpeter: that worked. restarting all remaining search nodes.
  • 01:39 notpeter: problem with lucene persisting through service restart, but not node restart. restarting en pool nodes.
  • 01:20 paravoid: restarting opendj (nfs1/nfs2), load spike, possibly related to leap second
  • 00:51 notpeter: search1004 dead. powercycling.
  • 00:50 notpeter: based on ganglia evidence, lucene seems to have been affected by leap second bug. restartig each instance, one minute wait in between

Archives