Server Admin Log/Archive 16

October 31

17:21 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 25719 - Add missing slash in timezone'

October 30

23:05 apergos: test of logging (sorry)
21:22 mark: Deploying a sudoers file for NRPE using Puppet
20:48 mark: Running apt-get upgrade on db17
20:48 mark: Pushed updated wikimedia-raid-utils package into the APT repository, with a newer arcconf that should work on Lucid
15:53 atglenn: powercycled mobile2, it was unresponsive to ssh and pings, ganaglia showed no activity
03:05 domas: ms1 can't snapshot either, I suspect kernel bugs. we either have to roll back to 2.6.28 or move forward, or actually try rebuilding filesystems from scratch with new kernels...

October 29

23:21 domas: lol repaired myisam tables on db9, call if data has been lost, hehe
22:58 domas: resynced srv154, was running with months old configuration/code.
22:58 domas: was db22 disabled silently by someone? or not reenabled? :) reenabled now...
22:55 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
18:33 apergos: restarted torrus on streber, after reports that it was not responding
17:46 apergos: domas ran "reset-mysql-slave db18" (from fenari) which clears out *all* old relay logs, and restarts the slaves.
17:34 apergos: removed some old relay logs from /a/sqldata on db18 to get space back, it was at 95%
15:22 RoanKattouw: Followers on Twitter: view missing entries between Sep 2 and today at http://identi.ca/wikimediatech
15:22 RoanKattouw: Re-established identi.ca->Twitter bridge for wikimediatech, broken since September 2
15:21 RobH: repaired the sessions table, rt is now happy
15:09 RobH: rt is being odd, looking into it

October 28

21:34 RobH: powercycled sq69, ran puppet, its back online
21:24 RobH: sq69 is borked, powercycling
17:51 Ryan_Lane: running checksetup.pl on kaulen for bugzilla
17:50 Ryan_Lane: running mysqlcheck --autorepair on bugzilla database on db9 for the bug_fulltext table
16:26 atglenn: as soon as someone is available we need to repair a table for bugzilla (probably due to db9 restart): Table './bugzilla3/bugs_fulltext' is marked as crashed and should be repaired [for Statement "INSERT INTO bugs_fulltext (bug_id, short_desc)
15:23 atglenn: reenabled logging for fundraising on locke
14:50 atglenn: I see a lot of lot of ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO) after reboot of db9... not awake enough to try to look at it; services seem to be running ok
14:46 atglenn: powercycled db9, it was unreachable by ssh, ganglia showed load and wait_cpu through the roof
14:46 RoanKattouw: db9 back after having been powercycled by Ariel
14:18 RoanKattouw: db9 down. Responds to ping but doesn't respond to anything else

October 27

18:32 mark: Shutdown yongle for decommissioning
17:49 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25668 - Allow bureaucrats to de-sysop and de-bureaucrat on fiwikimedia'
17:43 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25659 - Activation of Special'
17:41 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25659 - Activation of Special'
17:29 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25652 - Allow importing from frwiki on arzwiki'
11:06 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Set $wgOldChangeTagsIndex=false on fawikinews'

October 26

22:34 RobH: removed borked tenwiki deployments entirely from cluster, time to start over
18:33 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'readding ms1'
18:04 RobH: dropping out all the tenwikipedia stuff again, internet hates this wiki
18:00 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'rob makes typos some more'
17:59 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'rob makes typos'
17:30 RoanKattouw: Removed 10wikipedia from all.dblist and ran sync-dblist
17:08 logmsgbot: robh synchronized php-1.5/wmf-config/mc.php 'sync cuz i said so'
17:03 RobH: kicking srv183 around
16:56 logmsgbot: robh synchronized php-1.5/wmf-config/mc.php 'srv183 fell over'
16:51 logmsgbot: robh ran sync-common-all
16:26 RobH: sync-docroot run, nothing blew up, yay
14:34 apergos: powercycled mobile3. hey in case anyone ever needs to know this again, mobile3 is a dell r410 with the new drac 6.
12:47 hcatlin: mobile3 is not responding to anything. needs restart. attempting to contact ops.

October 25

21:50 Ryan_Lane: pdns on ns0 not responding, restarting
21:47 Ryan_Lane: fixing test-payments.tesla entry in dns
19:09 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25604 - Project namespace creation on frwiktionary'
19:05 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25627 - Please create various namespaces for fawikinews'
18:54 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '22283 - Add "Author" namespace to Telugu Wikisource'
18:47 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25635 - Please set the Crimean Tatar Wikipedia logo'
18:44 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25636 - Abuse filter in Persian wikinews'
18:42 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25576 - Install AbuseFilter on simplewiki'
18:34 logmsgbot: jeluf ran sync-common-all
18:32 logmsgbot: jeluf synchronized php-1.5/wmf-config/flaggedrevs.php 'bug 25634 - Please install flagged revisions on fawikinews'
18:30 logmsgbot: jeluf synchronized flaggedrevs.dblist
16:09 apergos: after restart of mysqld on db9, it now accepts requests from singer, blogs appear to be back
15:41 apergos: powercycled singer, it was unresponsive to ssh, huge cpu spikes

October 24

20:44 mark: Shutting down mayflower for decommissioning
20:40 mark: Changed root password on lily, iris, ragweed, mayflower and sage
20:37 mark: Changed root password on browne and srv2
19:31 RobH: installed python-openssl on spence for testing
19:11 tomaszf: removing downtime with init script on loudon
18:21 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
18:20 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
17:44 Ryan_Lane: reinstalling ersch for openstack testing
17:29 mark: Changed root password hash in puppet; being deployed to all systems in the next 30-60 minutes
17:21 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
16:56 Ryan_Lane: reinstalling alsted, to try stable openstack from scratch
16:43 Ryan_Lane: rebooting ersch and alsted
14:26 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
14:25 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
12:34 mark: Restarted varnish on knsq5
12:33 mark: Restarted varnish on knsq1
12:31 mark: Powercycled knsq4
12:31 mark: Restarted varnish on knsq2
12:31 mark: Removed puppet lock on knsq2

October 23

20:48 domas: powercycled srv154, saw mgmt network problems yet again
20:30 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Turning of CentralNotice for qualitywiki'
18:32 RobH: modifying the sync-common-all to take out logging while I am futzing with it.
18:27 logmsgbot: root ran sync-common-all
18:24 logmsgbot: root ran sync-common-all
18:21 logmsgbot: root ran sync-common-all
18:19 logmsgbot: root ran sync-common-all
18:17 logmsgbot: root ran sync-common-all
18:13 domas: increased maxclients 10->20 on scalers
18:10 logmsgbot: root ran sync-common-all
17:37 Ryan_Lane: updating the dns entry for svn.wikimedia.org
17:37 Ryan_Lane: rsynd'd /svnroot to formey
17:37 Ryan_Lane: remounted /svnroot ro on mayflower
17:37 Ryan_Lane: moving svn from mayflower to formey
17:03 domas: restarted apaches that failed to start after apt-get upgrade :(
17:01 logmsgbot: root ran sync-common-all
16:58 RobH: working on rsync settings and running crap as root for testing, disregard
16:57 tomaszf: upgrading activemq to 5.4.1 on erzurumi
16:51 logmsgbot: tstarling synchronized php-1.5/includes/memcached-client.php 'profiling memcached compression'
16:48 logmsgbot: tstarling synchronized php-1.5/includes/memcached-client.php 'profiling memcached compression'
16:46 logmsgbot: catrope synchronized php-1.5/extensions/CodeReview/backend/RepoStats.php 'r75268'
16:36 logmsgbot: root ran sync-common-all
16:33 domas: upgrading scalers to new imagemagick and whatever is pending
16:20 RobH: having to run the sync as root to fix permissions with roan, not normal use
16:19 logmsgbot: root ran sync-common-all
16:10 logmsgbot: root ran sync-common-all
15:22 mark: Deployed wikimedia.vcl (varnish) which fixes a memleak in the geoip code
14:27 logmsgbot: catrope synchronized php-1.5/extensions/CodeReview/ui/CodeRevisionView.php 'r75239'
14:09 Ryan_Lane: updated nova-* on ersch and alsted for stable release - rebooted
11:15 mark: Unstuck srv220 and srv222

October 22

17:25 RobH: at this time no issues are apparent and site access appears normal again.
17:25 RobH: Reported issues with accessing cluster hitting esams, possible network issues, further investigation shows no server errors. If network links continue to cause issues, we may redirect traffic stateside
02:36 apergos: shot some hung converts on the image scalers, that should hold us for awhile

October 21

21:10 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'Whitelisting new landing pages'
20:37 Ryan_Lane: adding account for jdavis on mchenry and sanger
18:31 mark: Powercycled sq67 and sq68
18:10 mark: Unstuck srv219, 223, 224
18:05 AaronSchulz: ran runBatchedQuery.php to finish enwiki rename 'Voice of All' => 'Aaron Schulz' on archive/revision tables
17:58 Ryan_Lane|food: enabled eth0 interface on test-payments, and changed the IP to the new IP
17:56 RobH: added dns entry for test-payments on prototype server
16:47 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaLicenseTexts.i18n.php 'r75158'
15:26 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
15:14 RoanKattouw: Commented out srv154 in /etc/dsh/groups/mediawiki-installation because it's down and blocking sync-file
15:13 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js 'r75156'
01:02 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'Readding payments sidebar links after outage'

October 20

22:03 mark: Removed squid.conf log line sending packets to yaseo
21:53 RobH: tired of misdated cron emails from dataset1, fixed the date again, it will go off date on its own eventually
21:52 Ryan_Lane: restarting opendj on nfs1
21:23 Ryan_Lane: adding nis indexes on sanger, and restarting the ldap server
21:22 Ryan_Lane: updating opendj on sanger
21:22 Ryan_Lane: updating opendj on nfs1
21:21 Ryan_Lane: restarting the ldap servers one at a time
21:21 Ryan_Lane: creating nis indexes in the ldap server on nfs1 and nfs2
21:09 Ryan_Lane: upgrading opendj on nfs2
21:05 Ryan_Lane: updating opendj package in repository to opendj_2.4.0-7
20:40 logmsgbot: laner synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disabling local uploads for outreachwiki, and enabling subpages on namespace 0'
20:19 logmsgbot: jeluf synchronized php-1.5/wmf-config/flaggedrevs.php 'Bug 24304 - Reconfigure English Wikibooks'
18:10 mark: Punched hole in loudon firewall for squid stats from streber
17:46 Ryan_Lane: setting svn.wikimedia.org to 60 seconds ttl in dns
17:25 mark: Added loudon to torrus (CDN/Squid)
17:24 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'taking outage on payments server'

October 19

21:36 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'Adding new Appeal pages'
20:52 Ryan_Lane: restarting varnish on all bits
17:12 JeLuF: uninstalled broken php5-xdebug package on srv181
15:23 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php
15:22 logmsgbot: catrope synchronized php-1.5/extensions/CentralNotice/SpecialBannerController.php
15:08 JeLuF: killed old convert jobs on srv100
14:17 JeLuF: stopped job runner on srv 258-289, checking impact on ext stores
10:59 JeLuF: added srv258-289 as job runners. We now have 69 active job runners
10:34 JeLuF: added srv181-190 as job runners. We now have 38 active job runners
06:44 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Turning central notice off for closed advisory wki'

October 18

20:10 Ryan_Lane: adding ldap management scripts to formey via puppet class ldap::client::sdk
15:44 JeLuF: Set default charset for http://noc.wikimedia.org/conf/ to utf-8
14:52 JeLuF: killed stuck runJobs.php tasks on srv{150-170}
14:36 logmsgbot: jeluf synchronized php-1.5/wmf-config/db.php 'removed db28 from s5'
14:32 logmsgbot: jeluf synchronized php-1.5/wmf-config/db.php 'removed db13 from s2'
13:44 RoanKattouw: Removed trailing ?> from mc.php, InitialiseSettings.php, lucene.php, wgConf.php, secure.php, checkers.php
06:44 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25557 - Index namespace on Telugu Wikisource'
06:41 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25554 - Change the "upload file" link on jawiki'

October 17

22:06 apergos: edited /etc/apache2/envvars in place and set APACHE_ULIMIT_MAX_FILES to 2000, hoping that will address the "too many open files" issue
21:29 hcatlin: mobile3 is having file limit issues. Turned off apache so that is de-pooled. Issue description sent to Ariel for RT submission.
18:17 mark: Restarted all varnish processes on sq67-70
18:10 mark: authdns-scenario normal
18:07 apergos: stopped and restarted varnish on knsq1, 4... loads back to normal there now
18:05 mark: Powercycled sq69
18:00 Ryan_Lane: restarting varnish on esams bits
17:50 JeLuF: Setting scenario to esams-down.
17:47 JeLuF: restarted varnish on knsq1, load average was 5149.87
15:39 logmsgbot: jeluf ran sync-common-all 'Bug 25056 - Create new wiki for Persian Wikinews'
15:15 logmsgbot: jeluf ran sync-common-all 'Bug 25130 - Create Wikipedia in Komi-Permyak'
14:42 logmsgbot: jeluf ran sync-common-all 'Bug 25307 - Create Wikipedia in Banjar'
14:03 logmsgbot: jeluf ran sync-common-all 'Bug 25544 - Create a new Wikipedia in Hill Mari'
12:59 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25345 - outreachwiki import sources'

October 16

16:44 logmsgbot: catrope synchronized php-1.5/extensions/CodeReview/ui/CodeRevisionView.php 'r74785'
16:43 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
16:42 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r74788'

October 15

20:30 RoanKattouw: Ran a DB query to set status of 'new' pre-r50000 revs to 'old'. 8,327 revs affected
00:50 Ryan_Lane: add ldap::client::nss configuration to formey, so it can see ldap users
00:48 Ryan_Lane: add all mayflower users and keys to wmf ldap
00:47 Ryan_Lane: add sudo and openssh-lpk schema to nfs1 and nfs2 ldap servers

October 14

20:36 Ryan_Lane: also added rkattouw to the engineering alias
20:34 Ryan_Lane: adding ^demon and myself to the engineering alias
20:00 Ryan_Lane: adding ldap::client::wmf-cluster include to formey in puppet. Adding openldap client as well.
19:59 Ryan_Lane: enabling opendj replication on nfs1 and nfs2
18:01 hcatlin: mobiles are acting up. keep an eye on them!
17:20 mark: Fixed mobile by restarting apache on mobile2
17:15 hcatlin: LVS is down for mobile... not sure why
17:05 hcatlin: mobile deploy is complete!
15:52 hcatlin: deploying updates to mobile cluster
13:13 mark: Deployed logrotate file for squid-frontend
12:34 mark: Fixed gmond on sq61
12:31 mark: restarted scaling on srv222

October 13

23:20 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'White listing new Appeal pages'
22:55 Ryan_Lane: added ldap::client::trust and ldap::client::openldap classes to nfs1 and nfs2, so that they can manage their own ldap servers
19:51 Ryan_Lane: commented out #monitor_service { "disk space":... line in misc::nfs-server::home::monitoring class in misc-servers.pp; was already defined in base.pp.
19:36 Ryan_Lane: installing opendj on nfs1 and nfs2
16:20 Ryan_Lane: reinstalling opendj and java on sanger. recreating the opendj instance.
16:12 Ryan_Lane: fixed sanger's ip address in /etc/hosts
14:02 mark: Killed ircd on browne - hopefully people are connecting to irc.wikimedia.org :P
14:01 logmsgbot: mark synchronized php-1.5/wmf-config/InitialiseSettings.php 'Move UDP IRC RC stream from browne to ekrem'
10:45 mark: Powercycled sq68
10:43 logmsgbot: mark synchronized php-1.5/wmf-config/db.php 'Depooling db19, ECC errors'
10:42 logmsgbot: mark synchronized php-1.5/wmf-config/db.php 'Repooling db15, db18, db19'

October 12

21:16 Ryan_Lane: make that starting opendj instance on sanger
21:15 Ryan_Lane: starting opends instance on sanger
21:15 Ryan_Lane: created opendj instance on sanger
19:38 Ryan_Lane: adding new package version of opendj to repository
19:14 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'Updating white list of banners'
18:19 RobH: db19 memory error, rebooting and will work on it later
18:15 RobH: mobile2 moved and back online
17:22 RobH: leaving pdf2 for now, just moble2 (rack needs to be wired)
17:22 RobH: mobile2 coming down for relocation pdf2 coming down for relocation
15:59 Ryan_Lane: updating opendj package in repository to a hardy built version
15:45 RobH: mysql restarted on db15
15:44 RobH: restarted mysql on db18 db19
15:39 RobH: db15 db18 db19 moved to b2-sdtpa wired and booting.
15:36 Ryan_Lane: with puppet
15:36 Ryan_Lane: installing opendj on sanger
15:27 Ryan_Lane: adding opendj package to repository
15:07 RobH: db15 db18 db19 commented out of db.php and coming down for relocation.
15:07 logmsgbot: robh synchronized php-1.5/wmf-config/db.php 'commented out db15 db18 db19 for their relocation'
14:43 logmsgbot: catrope synchronized php-1.5/extensions/CodeReview/api/ApiCodeRevisions.php 'r74661'
14:41 RobH: db22 had bad fan, fan replaced, opening case with oracle for replacement of the bad fan (used on site spare)
14:36 RobH: db22 unexpected power loss due to power cables binding when checking fan failure, is s4 slave
13:46 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'r74655'
13:46 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r74655'
13:00 logmsgbot: catrope synchronized php-1.5/extensions/CodeReview/ui/CodeRevisionListView.php 'r74654'
01:22 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'setting outreach wiki notice name'
01:21 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'setting outreach wiki notice name'

October 11

19:07 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'Whitelisting banners for thursday test'
17:36 logmsgbot: catrope synchronized php-1.5/extensions/CodeReview/ui/CodeAuthorListView.php 'r74641'
17:13 logmsgbot: catrope synchronized php-1.5/extensions/CodeReview/ui/CodeCommentsListView.php 'r74636'
17:08 logmsgbot: catrope synchronized php-1.5/languages/Language.php 'r74634'
16:58 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 74633
16:57 RoanKattouw: Running scap to deploy CodeReview update (r74630, r74633)
07:46 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix for jQuery 1.4.2 upgrade'
07:45 logmsgbot: catrope synchronized php-1.5/skins/common/jquery.min.js 'jQuery 1.4.2 upgrade'
07:45 logmsgbot: catrope synchronized php-1.5/skins/common/jquery.js 'jQuery 1.4.2 upgrade'
00:19 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'Re-adding donate sidebar links'

October 10

19:51 mark: Put European load back on esams
19:41 mark: Restarted Varnish on knsq4
19:26 mark: Restarted pdns on nescio
19:26 mark: Temporarily moved european traffic to pmtpa to stabilize esams
18:49 mark: Set backup static routes to LVS service IPs on csw5-pmtpa
18:31 mark: Restarted PyBal on lvs3, its BGP session had no prefixes announced
18:26 Ryan_Lane: repooling srv100
18:25 Ryan_Lane: killing convert and apache processes on the srv219-224. restarting apache on them.
18:15 Ryan_Lane|away: depooling srv100

October 9

19:25 Ryan_Lane|away: killed convert ulimit4 and apache on srv 100 and 219. restarted apache

October 8

22:36 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'Taking outage on payments'
02:51 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionTracking/ContributionTracking_body.php 'Updating fall back returnto'

October 7

21:53 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'white listing new banners'
19:56 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25332 - Logo update for Azerbaijani Wikibooks'
19:55 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25323 - Logo for Azerbaijani Wikisource'
19:43 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25218 - please update the wiktionary logo of mg.wiktionary.org'
19:38 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25333 - Portal namespace for Azerbaijani Wikipedia'
19:30 logmsgbot: jeluf synchronized php-1.5/wmf-config/CommonSettings.php
19:29 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25422 - Set $wgDisableHardRedirects = false; on wikimediafoundation.org'
19:15 logmsgbot: jeluf ran sync-common-all
19:07 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25423 - AbuseFilter for the Portuguese Wiktionary'
19:05 logmsgbot: jeluf synchronized php-1.5/wmf-config/abusefilter.php
18:56 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25420 - Enabling $wgBlockAllowsUTEdit for Persian Wikipedia'
17:56 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionTracking/ContributionTracking_body.php 'Picking up 1click tracking changes'
00:49 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialBannerController.php 'Fixing for anonymous views'

October 6

20:46 Ryan_Lane: killing convert, apache2, and ulimit4.sh on srv219-224; restarting apache on them
20:26 Ryan_Lane: applying memcached::disabled class to srv154, srv155, srv157, srv158, and srv170
20:26 Ryan_Lane: added new memcached::disabled class to puppet
20:04 Ryan_Lane: restarted varnish on all bits servers to reset memory
18:12 Ryan_Lane: rebuilding alsted for openstack
18:03 Ryan_Lane: rebuilding ersch, trying openstack again
16:43 mark: Moved irc.wikimedia.org CNAME from browne to ekrem
16:09 mark: Setup modified ratbox ircd on server ekrem, and linked it to the ircd on browne
15:06 Ryan_Lane: relabeled slot1 for Wikimedia-Daily tapes on tridge to fix backup errors
10:27 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
00:35 logmsgbot: midom synchronized php-1.5/wmf-config/db.php

October 5

23:19 RobH: lvs1 moved to b1-sdtpa, no network only mgmt
23:15 RobH: db16 isnt in cluster, no mysql start, db20 mysql isnt started
23:12 RobH: mysql restarted db17
23:12 RobH: db16, db17, db20 in new rack locations.
21:44 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialNoticeTemplate.php 'Picking up fixes for geo'
21:43 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/CentralNotice.php 'Picking up fixes for geo'
21:42 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialBannerListLoader.php 'Picking up fixes for geo'
21:41 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/centralnotice.js 'Picking up fixes for geo'
21:40 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/CentralNotice.db.php 'Picking up fixes for geo'
21:38 RobH: db20 wants to go with db16/17, its shutting down too
21:34 logmsgbot: robh synchronized php-1.5/wmf-config/db.php 'commenting out db17'
21:31 RobH: db16 and db17 going offline to move
21:27 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/CentralNotice.i18n.php 'Picking up fixes for geo'
21:08 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
21:01 RobH: db11 was already moved, opps, rebooting.
20:50 logmsgbot: tfinc synchronized php-1.5/skins/common/images/closewindow.png 'Picking up new close button image'
20:41 RobH: db11 and db14 also moving
20:41 logmsgbot: robh synchronized php-1.5/wmf-config/db.php 'once more with my actual changes'
20:39 logmsgbot: robh synchronized php-1.5/wmf-config/db.php 'if this doesnt work its not my fault'
20:34 RobH: moving db13 & db14, they are shutting down for a short duration
18:17 RobH: db11 shutting down
18:05 RobH: storage1 back online
17:35 mark: Setup ports 3/35 to 3/44 for db11 to db20 respectively
17:31 domas: s3 switched to db27-bin.001:79
17:31 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
17:31 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
17:28 RobH: moved dataset1 network connection, all good
16:32 RobH: dataset on the brain, storage1/2
16:32 RobH: dataset1 shutting down for relocation into b1-sdtpa, dataset2 already offline, moving as well
16:15 logmsgbot: catrope synchronized php-1.5/wmf-config/checkers.php 'Remove no-UA exemption for yongle'
16:14 mark: Setup csw1-sdtpa:11/2 for amaranth
16:12 mark: Setup csw1-sdtpa:11/1 for dataset1
16:06 domas: switched commons master to 5.1, pos: db31-bin.000001:106
16:06 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
16:06 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
16:02 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
16:02 logmsgbot: catrope synchronized php-1.5/extensions/ArticleAssessmentPilot/js/ArticleAssessment.combined.min.js 'r74312'
15:52 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'db27 back'
15:52 logmsgbot: catrope synchronized php-1.5/extensions/ArticleAssessmentPilot/js/ArticleAssessment.combined.min.js 'r74310'
15:33 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'db31 in, db27 out'
15:25 domas: fixed space on srv178, conf changes made apparmor audit log blow up
15:02 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'db31 mnt'
15:00 mark: Restarted Apache on singer
14:55 mark: Migrated the Apple Dictionary bridge from yongle to ekrem, and moved the search.wikimedia.org CNAME accordingly
14:51 mark: ekrem reboot test
13:37 mark: Had puppet install php5-curl on ekrem
13:29 mark: Migrated the old mobile (WAP) site from yongle to ekrem and updated the Squid config
11:56 mark: Installed Lucid on new server ekrem
11:23 mark: Reused public ip address of isidore for new server ekrem
10:53 mark: Reinstalled knsq1 as bits varnish server and had it join the esams bits cluster
09:34 mark: Removed AS43821 outbound double prepend on BGP session with AS13030
09:30 mark: Moving European bits traffic back to the European cluster

October 4

19:52 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
19:52 logmsgbot: catrope synchronized php-1.5/extensions/ArticleAssessmentPilot/js/ArticleAssessment.combined.min.js 'r74268'
19:51 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/PrefSwitch/SpecialPrefSwitch.php 'r74268'
11:57 mark: Stopped PyBal on amslvs4 as a test
11:47 logmsgbot: mark synchronized php-1.5/wmf-config/mc.php 'Replace srv87'
10:53 mark: Setup BGP state monitoring for csw5-pmtpa and csw1-sdtpa as well
10:48 mark: Setup BGP state monitoring for csw1-esams and csw2-esams
10:36 mark: Restarted Apache on srv223
10:24 mark: Fixed Nagios by removing db34 from the listed hosts section in conf.php
09:47 mark: Downpreffed AS13030 transit to local-pref 90 on br1-knams

October 3

19:25 logmsgbot: catrope synchronized php-1.5/wmf-config/checkers.php
18:59 logmsgbot: catrope synchronized php-1.5/wmf-config/checkers.php 'Block search abuse reported by Robert'
17:43 mark: Restarted apache on srv219-221, 224
17:38 mark: Stopped IPVS backup sync daemon on amslvs3
17:37 mark: Rebooting amslvs1
17:37 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
17:32 mark: Stopped PyBal on amslvs1
17:30 mark: Stopped IPVS backup sync daemon on amslvs4
17:27 mark: Rebooting amslvs2
17:19 mark: Stopped PyBal on amslvs2
17:14 mark: Drive failed in thistle
17:09 mark: Started temporary LVS state syncing between amslvs1->amslvs3 and amslvs2->amslvs4, preparing for reboot of amslvs1-2
16:51 mark: Rebooted (backup) LVS servers amslvs3 and amslvs4
16:36 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'adding db28 to s5'
16:20 mark: Selectively updated pybal/bgp.py on lvs2-4 and restarted PyBal
16:13 mark: Upgraded PyBal to r0.1+74215 on amslvs1-4
14:27 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
14:12 domas: db22 was first wikimedia slave initialized using xtrabackup, hehe
14:08 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'adding db22 as commons 5.1 slave'
14:06 mark: Restored nameservers order for mobile serverswq
13:36 mark: lvs3 service ips were unreachable from outside the subnet for some reason. Restarting PyBal (and therefore the BGP session to the router) fixed it
13:22 mark: Started pdns recursor on dobson again
13:19 mark: Once again stopped pnds-recursor on dobson
13:18 mark: Restarted Apache on the mobile servers one by one, to put the new resolv.conf in effect
13:10 mark: Swapped nameservers for the mobile servers, so they don't crash the mobile site while I reinstall the primary recursor
13:02 logmsgbot: catrope synchronized php-1.5/wmf-config/checkers.php 'Log no-UA requests for exempt IPs to /h/w/l/nouaexempt.log'
12:45 mark: Mobile site broke with missing primary DNS recursor, restarted it
12:44 mark: Stopped DNS recursor on dobson again
12:20 mark: Installed DNS recursors on all LVS servers to reduce SPOFfiness
11:46 domas: doing some DB maintenance
11:44 mark: Installed nscd on lvs4 and restarted PyBal
11:30 mark: Restarted pybal on lvs4
11:25 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
11:21 mark: Shutdown primary pmtpa DNS recursor (dobson)
11:20 logmsgbot: catrope synchronized php-1.5/skins/common/wikibits.js 'r74198'

October 2

17:01 apergos: we were hoping that a reboot of dataset1 would clear up the ipv6 + lighty issue that it had after os upgrade; no dice. this needs further investiagtion. ipv6 left disabled in lighty conf for now.
16:43 RobH: dataset1 relocation and drive work done, system is back online
16:20 RobH: dataset1 moved, boots back online, still working on its degraded disk replacement
15:55 RobH: rebooted scs-a1-sdtpa since its web interface isnt accessible, not sure if its the device or the network
15:06 RobH: dataset1 downtime window starts now

October 1

20:03 RoanKattouw: Batch-deleted ~1200 categories on cswiktionary, requested by Danny B
15:54 mark: Restarted apache on srv219-223, and killed stuck convert processes
13:20 mark: Removed unix accounts brion and fvassard (with puppet)

September 30

23:14 Ryan_Lane: deleting /srv-old on prototype
23:11 Ryan_Lane: added a new 50G disk to prototype. rsync'd data from /srv to new mount. moved /srv to /srv-old. mounted new drive on /srv. rsync'd data from /srv-old to /srv.
21:27 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'White listing new banners'
17:13 awjr: upgraded kernel on grosley.wikimedia.org to 2.6.24-28
00:57 Ryan_Lane: added mail alias ryan -> rlane
00:55 Ryan_Lane: added mail alias carrie -> csmith on mchenry

September 29

22:59 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Turning CN back on as default'
22:55 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/CentralNotice.php 'Picking up fix for db lookup'
22:55 Ryan_Lane: rebooting ersch and alsted
22:54 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'Turning test wikipedia to non infrastructure mode'
22:41 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'disabling cn'
22:39 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/CentralNotice.php 'Picking up fixes from 73993'
22:38 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialBannerController.php 'Picking up fixes from 73993'
22:36 logmsgbot: tfinc synchronized php-1.5/includes/Skin.php 'Picking up fixes from 73993'
21:31 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialBannerController.php 'Picking up fix for message translation'
21:29 logmsgbot: tfinc synchronized php-1.5/includes/MessageCache.php 'Picking up fix for message translation'
21:28 logmsgbot: tfinc synchronized php-1.5/includes/parser/CoreParserFunctions.php 'Picking up fix for message translation'
21:27 logmsgbot: tfinc synchronized php-1.5/includes/parser/ParserOptions.php 'Picking up fix for message translation'
19:21 mark: Permanently doubled sys.net.ipv4.tcp_max_tw_buckets to 360000, as the Varnish systems are constantly close to the limit (visible in /proc/net/sockstat)
19:04 mark: ...on all squid and varnish servers, in puppet sysctl.conf
19:03 mark: Increasing net.ipv4.tcp_max_syn_backlog = 262144
18:57 mark: Experimentally doubled sys.net.ipv4.tcp_max_tw_buckets to 360000 on sq70 (manually, so not in sysctl.conf)
16:45 Ryan_Lane: make that the *new* tesla cluster subnet
16:44 Ryan_Lane: updating dhcpd.conf on brewster to include the tesla subnet (153.192)
16:29 mark: Setup IP helper-address to brewster for vlan 103 / ve 7 on csw5-pmtpa
02:31 Ryan_Lane: unmounted old data partition and replaced it with the newer, larger data partition and updated the fstab on tridge.
00:51 Ryan_Lane: destroyed resized data1 logical volume, and recreated the jfs filesystem on tridge. copying the data from /data to new partition; will remount data as new partition and update fstab
00:49 awjr: Upgraded kernel on CC server to 2.6.32-24
00:49 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'Re-enabling CC server on variable page post kernel upgrade'
00:49 Ryan_Lane: install kvm-pxe on alsted
00:49 Ryan_Lane: configured eth1 with vlan tag 103, and added it to a new bridge, br103 on alsted and ersch
00:36 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'Taking outage on CC server for kernel upgrade'

September 28

21:57 Ryan_Lane: changing testvm1 to tesla namespace. adding new origin for tesla
21:52 Ryan_Lane: adding testvm1.tesla to dns
19:58 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'Switching from landing page 9 to 1 for donate sidebar link'
02:00 Ryan_Lane: added puppet vim syntax highlighting to root's .vim directory on sockpuppet
01:17 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enaling centralnotice for all projects' Project Page
01:12 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enaling dewiki for centralnotice'
01:05 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Re-enaling cswiki for centralnotice'
01:03 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/CentralNotice.php 'Picking up autoload fix'
00:57 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/CentralNotice.php 'Picking up db class fix'
00:51 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialBannerListLoader.php 'Picking up db class fix'
00:51 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/CentralNotice.php 'Picking up db class fix'
00:44 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'picking up new cc form variants for thursday fundraiser'
00:35 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialBannerListLoader.php 'Turning on debugging for CN'
00:27 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Turning cs wiki centralnotice off'
00:10 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Turning central notice back on for en wikiversity'

September 27

23:38 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Turning central notice back on for cs wiki'
23:03 logmsgbot: tfinc synchronizing Wikimedia installation... Revision: 73744
20:23 Ryan_Lane: updated puppet configuration to require php version 5.2.4-2ubuntu5.12wm1 for all apaches
20:21 Ryan_Lane: testing updated puppet config for php on srv212
20:18 mark: Changed MIME type for geoiplookup.wikimedia.org in wikimedia.vcl (using puppet)
20:17 Ryan_Lane: updated puppet configuration to update php on all apaches
20:12 mark: Powercycled sq67
20:06 Ryan_Lane: testing updated php packages deployed from puppet on srv 211
20:05 Ryan_Lane: updating wikis on prototype
19:36 Ryan_Lane: reinstalled wikimedia-task-appserver on srv210
19:27 Ryan_Lane: testing updated php from repository on srv210
18:45 Ryan_Lane: pushing updated php to the repository

September 26

21:13 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'disabled wgFlaggedRevsComments on hewikisource'
17:47 mark: Sending esams bits traffic to bits.pmtpa
17:28 mark: Sending esams bits traffic to text.esams
17:28 mark: Sending esams bits traffic to bits.esams
17:18 RobH: aware of bits issues, looking into them
07:40 apergos: (25/09/2010 10:17:43 UTC) restarted apache on mobile3. hope that holds us for awhile. lvs was depooling alternately mobile3 and then mobile2
00:42 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'Re-adding cc donate sidebar links'

September 25

19:56 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 're-deployment'
15:56 apergos: restarted apache on mobile2, it had fallen over
11:41 logmsgbot: catrope synchronized php-1.5/extensions/ArticleAssessmentPilot/api/ApiArticleAssessment.php 'r73744'

September 24

23:19 Ryan_Lane: rebuilding ersch for open nebula testing
22:53 rainman-sr: disk was full on searchidx1, cleared up some disc space by deleting some old backup indexes
22:42 RobH: bayes back online
22:40 Ryan_Lane: enabling https on owa1 and owa2
22:39 Ryan_Lane: changing lvs scheduler on lvs1.tesla.usability.wikimedia.org to use wlc, since session stickyness is no longer needed
22:28 Ryan_Lane: added egencia to email whitelist on mchenry
22:15 RobH: updating firmware on bayes, leave it alone
22:07 Ryan_Lane: starting testing of open nebula on alsted
22:06 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'Removing payments links for outage on loudon'
21:32 JeLuF: restarted apache on srv211, was non-responsive.
20:48 Ryan_Lane: installed php5 packages on test wikipedia, restarting apache
20:41 Ryan_Lane: testing php5_5.2.4-2ubuntu5.12wm1 packages on test.wikipedia.org
19:58 RobH: bayes is goin down for rack relocation, weeeeee
18:58 Ryan_Lane: installing ganglia on loudon
14:45 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
14:45 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/PrefSwitch/PrefSwitch.js 'r73655'
14:44 logmsgbot: catrope synchronized php-1.5/extensions/ArticleAssessmentPilot/ArticleAssessmentPilot.hooks.php 'r73658'
14:44 logmsgbot: catrope synchronized php-1.5/extensions/ArticleAssessmentPilot/ArticleAssessmentPilot.php 'r73658'
14:44 logmsgbot: catrope synchronized php-1.5/extensions/ArticleAssessmentPilot/js/ArticleAssessment.combined.min.js 'r73658'
14:42 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Add settings for r73658; will work once that's deployed'
14:08 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Temp fix for missing jQuery UI CSS in ArticleAssessment feedback dialog. Proper software-side fix underway'
12:17 mark: Doing another reboot test of streber
12:11 mark: Started lighttpd on streber
12:10 mark: Ran apt-get upgrade && reboot on linne
12:05 mark: Rebooting streber for kernel upgrade

September 23

23:41 Ryan_Lane: upgraded drupal on prototype.wikimedia.org
22:49 Ryan_Lane: started profile-collector on spence... properly this time
22:38 Ryan_Lane: started profile-collector on spence
22:24 Ryan_Lane: added cgi module to apache config on fenari so that report.py will run
21:29 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'Whitelisting banners for thursday test'
21:12 Ryan_Lane: fixed apache configuration on noc. highlight.php is working again.
20:57 tomaszf: ran sync-docroot to pick up new version of bannerImpression.php r46 fixing firefox mime type issues
20:33 mark: Deployed ^/wiki/Special:Banner(Controller|ListLoader) exemption from header_replace Cache-Control on the text frontend squids (ACL wpstatic)
20:25 logmsgbot: tfinc synchronized php-1.5/extensions/VariablePage/VariablePage.i18n.php 'Picking up fixes for tracking'
20:24 logmsgbot: tfinc synchronized php-1.5/extensions/VariablePage/VariablePage.alias.php 'Picking up fixes for tracking'
20:24 logmsgbot: tfinc synchronized php-1.5/extensions/VariablePage/VariablePage.body.php 'Picking up fixes for tracking'
20:23 logmsgbot: tfinc synchronized php-1.5/extensions/VariablePage/VariablePage.php 'Picking up fixes for tracking'
20:21 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'Adding new config option in prep for variable page push'
20:05 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25180 - upload disabled on zhwikinews'
20:03 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25142 - Set $wgGroupPermissions['*']['createpage'] = true; zhwiki (Allowed IP createpage)'
20:00 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25266 - Please set the Tatar Wiktionary logo'
19:25 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '17627 - Allow autoconfirmed users to patrol on ar Wikisource'
18:36 RobH: resetting the SP on following servers to fix the fact no LEDs are lit on the front panels, yet servers are up (known sun lom issue for us) db21,db23,db24,db29 all SPs back online and LEDs working properly
16:55 RobH: ignore storage2 flapping, it is offline, trying to fix it enough to recover things from it.
15:07 hcatlin: deploying mobile changes... UDP fix
06:56 Tim: rebooting bayes, Erik Z doesn't seem to be running any scripts on it at the moment
06:45 Tim: rebooting tarin, tridge and williams for kernel upgrades
05:53 Tim: rebooting sockpuppet, stafford, spence for kernel upgrade

September 22

nimishg: created user 'file_mover' on locke and dataset1 for automating file movement between these machines
nimishg: added logging on locke to stress-test packet loss for 24 hrs
23:51 logmsgbot: laner synchronized php-1.5/wmf-config/InitialiseSettings.php 'Changing wikispecies logo for bug 25026'
18:24 Ryan_Lane: disabled firewall on alsted, commencing more openstack work
18:18 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable ArticleAssessment on enwiki for Category:Article_Assessment_Pilot'
18:17 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Kill dark launch hack'
18:11 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
18:10 logmsgbot: catrope synchronized php-1.5/extensions/ArticleAssessmentPilot/js/ArticleAssessment.combined.min.js 'r73553'
17:41 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php
17:39 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php
17:33 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'Adding Sequence namespace to commons per Erik and mdale's request'
17:22 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Enable AA for me only on enwiki'
17:18 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable ArticleAssessment on enwiki, but without a trigger category'
17:10 RoanKattouw: Dark-launching ArticleAssessment+SimpleSurvey on enwiki, such that it's only visible to me
16:51 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'bug 25243: Change Image: to File: in filerepo config to save HTTP redirects'
15:48 RobH: added memcached roles to all apaches srv150+ that are not image scalers. Once puppet updates them all, they will be tested and moved into the spares group in mc.php
15:06 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 'removed ms1 from ES rotation to dump for TS'
14:29 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 73534
14:28 RoanKattouw: Running scap to deploy updates to ArticleAssessment and SimpleSurvey. Still only enabled on testwiki for now
14:26 logmsgbot: catrope synchronized php-1.5/includes/DefaultSettings.php 'r73514'
14:25 logmsgbot: catrope synchronized php-1.5/includes/api/ApiMain.php 'r73514'
11:53 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Reenable action=parse on Commons now ImageAnnotator is disabled'
08:46 Tim: deploying shortened error messages (Wikimedia_en) on all squids, since they all seem to have the updated package now
08:42 Tim: fixed RCS/permissions mess in /home/wikipedia/conf/squid
08:23 logmsgbot: tstarling synchronized php-1.5/includes/api/ApiMain.php 'r73511'
08:23 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'disable action=parse on commons'
08:20 logmsgbot: tstarling synchronized php-1.5/includes/DefaultSettings.php 'r73511'
01:43 Ryan_Lane: installed openstack components on alsted; firewalled off for now

September 21

20:32 RoanKattouw: Applying PrefSwitch schema change (pss_user_text field) on enwiki
20:01 RoanKattouw: Ran prefswitch.pss_user_text SQL patch on testwiki DB
19:45 RoanKattouw: Applying SQL patch for ArticleAssessment to testwiki DB
19:31 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 73472
19:30 RoanKattouw: Running scap to deploy ArticleAssessment and SimpleSurvey extensions. Not enabled anywhere yet
19:16 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/PrefSwitch/PrefSwitch.classes.php 'r73471'
19:14 JeLuF: fixed broken installation of wikimedia-task-appserver on fenari (missing package libapache2-mod-php5 was installed automatically)
19:11 logmsgbot: catrope synchronized php-1.5/includes/api/ApiBase.php 'r73469'
19:04 logmsgbot: catrope synchronized php-1.5/skins/vector/main-rtl.css 'Deploying r72939, which apparently never got done'
19:02 logmsgbot: catrope synchronized php-1.5/skins/vector/main-ltr.css 'Deploying r72939, which apparently never got done'
18:36 Ryan_Lane: changed eiximenis's ufw configuration to only forward port 9000 to port 80 if the destination address is eiximenis
18:30 Ryan_Lane: changing firewall rules on eiximenis
16:11 mark: Restarted sq32 and sq33
15:39 logmsgbot: tstarling synchronized php-1.5/includes/api/ApiMain.php 'disabling action=parse'
15:21 mark: restarted backend squid on sq31
15:16 mark: Lowered cache_mem from 2500 to 2000 on sq31-33
09:35 Tim: on grosley: tried to fix broken hudson-labs apt source, apt-get dist-upgrade
09:31 Tim: apt-get dist-upgrade on sanger, bayes
09:19 Tim: on mchenry: apt-get dist-upgrade, and chmod 755 /var/spool/exim4 so that NRPE works again
09:03 Tim: configured kaulen to get its packages from the local mirror, and removed the o=Wikimedia pin to make it not use the PHP packages which were compiled on hardy and which conflict with everything
08:28 Tim: apt-get dist-upgrade on brewster, ersch, erzurumi, formey
08:21 Tim: on alsted: apt-get dist-upgrade
07:13 Tim: on hume: apt-get dist-upgrade
05:05 logmsgbot: tstarling synchronized php-1.5/wmf-config/secure.php 'removed downtime notice'
04:58 Tim: on yongle: fixed broken sources.list, upgraded 187 packages
04:54 Tim: upgraded packages on tarin, tridge, williams
04:42 Tim: on streber: removed python-pysnmp2, conflicted with python-pysnmp4 and apparently wasn't being used
04:28 Tim: upgraded packages on stafford, streber. python-pysnmp-common on streber is broken
04:14 Tim: upgraded packages on singer
04:07 Tim: upgrading packages on spence
04:03 Tim: upgrading packages on sockpuppet
03:53 logmsgbot: laner synchronized php-1.5/wmf-config/InitialiseSettings.php
03:53 Tim: scheduled reboot of singer at 05:00 using atd
03:50 logmsgbot: tstarling synchronized php-1.5/wmf-config/secure.php 'secure.wikimedia.org scheduled reboot notice'
03:46 logmsgbot: laner synchronizing Wikimedia installation... Revision: 72920
03:22 Ryan_Lane: updated the kernel on oldusability.wikimedia.org linode server, and rebooted
02:55 Tim: upgrading kernel on wikitech linode to latest 2.6 paravirt, will reboot
02:45 logmsgbot: laner synchronized php-1.5/extensions/CentralNotice/CentralNotice.php 'removing debug info'
02:44 logmsgbot: laner synchronized php-1.5/extensions/CentralNotice/SpecialBannerListLoader.php 'removing debug info'
02:43 logmsgbot: laner synchronized php-1.5/extensions/CentralNotice/SpecialCentralNotice.php 'removing debug info'
02:39 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialCentralNotice.php 'removing debug info'
02:13 logmsgbot: tfinc synchronizing Wikimedia installation... Revision: 72920
02:12 Tim: rebooting locke and sanger for kernel upgrade
02:09 Tim: rebooting hume for kernel upgrade
01:33 logmsgbot: tfinc ran sync-common-all
01:12 Tim-away: fixed syntax error in InitialiseSettings.php, which caused the site to go down

September 20

22:23 Ryan_Lane: finished kernel upgrades on tesla VMs
20:54 Ryan_Lane: moved mysql data from /var/lib/mysql to /a/sqldata on prototype
18:33 Ryan_Lane: Correction: previous ufw logs were for eiximenis
18:33 Ryan_Lane: added ufw application configuration for etherpad. Added allow rules for OpenSSH and Etherpad in ufw config. Enabled ufw.
18:32 Ryan_Lane: added ufw prerouting rules for 80->9000 for etherpad.
18:32 Ryan_Lane: restarted etherpad on eiximenis
17:46 Ryan_Lane: upgrading kernel on all tesla VMs
16:30 Ryan_Lane: running database recovery on torrus on streber, and restarting the service
14:29 Tim: rebooting mchenry for kernel upgrade
13:39 Tim: rebooting ersch, erzurumi, formey and kaulen for new kernel
12:34 Tim: rebooting brewster and eiximenis for new kernel
12:25 Tim: updated browne /etc/motd
08:49 Tim: rebooting alsted for new kernel
07:35 Tim: rebooting fenari to get new kernel
07:03 Tim: upgrading everything on fenari

September 17

20:56 RobH: db7 & db8 no longer under warranty, replaced failed 73gb 10k with 146gb 10k drive on site. raid rebuilding on both
20:35 RobH: locke no longer under warranty, replaced failed 73gb 10k with 146gb 10k drive on site. raid rebuilding, closing rt#114
19:36 RobH: db28 coming down for fanboard replacement (not hot swappable since its the controller)
19:30 RobH: all SP back online for search1-search12 reporting error free.
19:25 RobH: resetting the SP on search1-search12 to make them forget the spare powersupply they really do not have.
19:16 RobH: full power reset cleared drac errors, noted for followup, system back online.
19:07 RobH: srv206 has all kinds of errors, working on it, ignore nagios flaps.
18:49 RobH: db34 online and ready to be setup into database deployments.
00:15 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'Fixing placement of VariablePage extension'

September 16

21:39 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'adding enwiki to variable page extension'
21:38 Ryan_Lane: fixed puppet issue with ganglia on memcache servers (a bad puppet file had previously been pushed)
21:35 logmsgbot: tfinc ran sync-common-all
21:27 logmsgbot: tfinc synchronized php-1.5/extensions/VariablePage/VariablePage.i18n.php 'Adding variable page extension for donate link'
17:36 Ryan_Lane: took sq33 out of lvs rotation
16:02 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 25142: Redisable anon page creation, there was no consensus'
14:23 RobH: sq60,sq73,sq75 coming back into service
14:13 RobH: fixed sq33, working on backend on sq60,sq73,sq75, all having cache cleaned
13:54 RobH: taking another look at sq33, pulled from service
13:49 RobH: working on srv206
13:46 RobH: sq73 & sq75 up
13:35 RobH: sq38 & sq60 back in service
13:29 RobH: rebooted sq38 & sq60 to bring them back online
12:45 mark: Powercycled amssq52
12:37 mark: knsq3 declared dead
11:43 mark: Inserted extra switch fabric and spare 4x 10G line card modules into csw1-esams

September 15

21:16 Ryan_Lane: doing initial repo copies from mayflower to formey
21:16 Ryan_Lane: added svn users to formey.wikimedia.org
18:43 mark: Moving back bits.esams traffic
18:21 mark: Moving traffic for bits.esams to bits.pmtpa
18:08 mark: Restarted varnish on knsq5
18:07 mark: Changed LVS scheduler from 'wlc' to 'wrr' for bits on amslvs1
18:02 mark: Restarting pybal on amslvs1, with proxyfetch disabled for varnish/bits
17:47 mark: Restarted varnish servers on knsq2, 4 and 5
17:37 mark: temporarily disabled the proxyfetch monitor on amslvs1 to stabilize
17:06 mark: Restarted pdns instances on dobson
16:59 mark: Shutdown powerdns (auth) and pdns-recursor on dobson, preparing for reinstall
16:19 mark: Temporarily making ns1 the DNS master, to rebuild dobson
15:52 mark: Fixed regular expressions in puppet site.pp
14:48 mark: ms2 upgrade to Lucid is complete
14:47 mark: Replaced apparmor profile on ms2 by newer one from ms1, restarted replication
13:25 mark: Starting ms2 upgrade to Lucid
01:09 Ryan_Lane: finished rebuilding ersch
00:58 Ryan_Lane: rebuilding ersch
00:57 Ryan_Lane: finished rebuilding alsted
00:39 Ryan_Lane: rebuilding alsted

September 14

22:55 awjr: Rolled back PHP 5.3.2 to PHP 5.2.10 on payments.wikimedia.org due to outstanding PHP bug 50394
20:58 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'disabled $wgFlaggedRevsComments on huwiki (unused)'
20:06 awjr: restarting payments.wikimedia.org post ubuntu 10.04 upgrade
19:41 awjr: performing upgrade from ubuntu 8.04->10.04 on payments.wikimedia.org
19:06 mark: Restarted pdns on dobson/ns0
19:05 mark: Fixing wikimedia.org MX records in DNS
18:59 RobH: mail server issues are being investigated.
18:48 Ryan_Lane: removed public IP address to test-payments
18:45 awjr: Restarting test-payments post upgrade
18:40 Ryan_Lane: added public IP address to test-payments
18:37 awjr: Performing Ubuntu upgrade from 8.04 -> 10.04 on test-payments on tesla
18:37 RobH: db34 is running memtest86, relocating the memory moved it, who knows if its due to reseating or relocation? hence memtest is in progress.
18:07 RobH: db34 is truely the spawn of the devil. after two trips back to repair, it now appears not only did it need a new mainboard, new cpu, new mainboard again for good measure, new power supply logic board, it also now has two dimm errors. pulling it down to migrate the memory around and confirm its the dimms.
18:00 RobH: db34 system board replaced, and also bad cpu replaced, back in rack and installing os.
17:55 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25128 - Books namespace for fiwiki'
17:53 Ryan_Lane: delegated corp.wikimedia.org to ns1.corp.wikimedia.org and ns2.corp.wikimedia.org
17:49 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25160 - Please create an "autopatrolled" usergroup at metawiki'
17:46 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25142 - Set $wgGroupPermissions['*']['createpage'] = true; zhwiki (Allowed IP createpage)'
17:43 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25166 - patroller on Fawiki'
17:39 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php
17:36 JeLuF: closed swwikibooks (bug 25170)
17:36 logmsgbot: jeluf ran sync-common-all
17:10 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25125 - Please update logo for Kurdish Wikipedia'
11:43 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 'PagedTiffHandler'
11:42 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'enabling PagedTiffHandler on all wikis instead of just commons'
11:41 logmsgbot: fixed logmsgbot UDP listener
09:58 rainman-sr: search10-19 show down in ganglia and nagios with message "Backend Squid HTTP", although they are alive and well and not squids. Could someone look into this?
02:37 Tim: on locke: filtered mobile packets out of packet-loss input

September 13

20:32 Ryan_Lane: rebuilt formey as 10.04
20:30 RobH: disregard that I cannot spell disregard.
20:30 RobH: disreagrd formey popping up and down, Ryan_Lane is working on it.
18:17 RoanKattouw: logmsgbot on IRC but broken
18:17 RoanKattouw: Synced CommonSettings.php to bump style version appendix
18:17 RoanKattouw: Synced extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js for r72920
14:33 RoanKattouw: Started logmsgbot as catrope. Filed RT #217 for cleanup of logmsgbot processes
14:28 RoanKattouw: Synced CommonSettings.php for bug 25106 (allow .sla uploads on outreachwiki)
14:27 RoanKattouw: logmsgbot MIA
11:14 hcatlin: deploying a "perm disable" bugfix for m.
11:06 hcatlin: deployed m. language update
07:31 Tim: deploying 72894

September 12

07:43 domas: reinstalling db20

September 11

15:21 RobH: seems srv224 has been stuck on conversions and required manual restart of apache before.
15:20 RobH: srv224 apache stuck on some conversion, wont restart, offlined in lvs but is now back online
02:53 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'removed redundant settings'

September 10

23:51 RobH: bringing back online srv154, it should start up, run puppet, and put itself back into service
23:50 RobH: hey look, the nagios-wm change worked, it logs to both channels, all is well.
23:46 RobH: srv154 may flap, its intentional, i assure you
23:43 RobH: nagios-wm reports in multiple channels, woot
23:21 RobH: tired of messing with nagios bot, its back online
23:13 RobH: trying to bounce the nagios bot to make it report in different channels.
20:23 Ryan_Lane: manually modified /etc/gmond.conf on srv218 temporarily to use an actually existing directory for ganglia plugin configuration
20:22 Ryan_Lane: manually added memcache ganglia plugin to srv218 temporarily for testing before pushing to all memcache servers
20:21 Ryan_Lane: added patched python-memcache package to srv218 for memcache ganglia plugin testing
20:21 Ryan_Lane: created test-payments.tesla.usability.wikimedia.org for payment processing testing (no public IP)
19:06 RobH: mobile2 back online and in lvs pool, mobile pool optimal (as optimal as a 3 server cluster can be)
19:01 RobH: mobile2 unresponsive to ssh, shows partically up in lvs4, wont respond to serial console, rebooting it
18:59 RobH: mobile3 back online.
18:59 RobH: restarted apache/memcached on mobile3
16:17 RobH: db20 back online, but has one power supply while other is replaced
15:38 RobH: db20 all kinds of messed up, ignore nagios flaps.
15:38 RobH: db16 running memory tests until Monday, leave it alone

September 9

21:32 RobH: knsq21, knsq22 online, rt#5 resolved
20:31 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'Adding thursday banners to whitelist for contrib stats'
20:17 RobH: knsq21 & knsq22 coming down for reinstallation, set to false in pybal
20:14 RobH: knsq17-knsq20 reinstalled and back in service
19:53 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23081 - Remove "makesysop" right from bureaucrats and stewards groups'
19:50 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25011 - Need Namespace name change in tamil wikinews'
19:43 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24865 - [[special'
19:35 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23186 - Enable upload in portuguese Wikipedia'
19:32 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24852 - New namespace for WikiProject on jawiki'
19:16 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25022 - Add NewUserMessage extension on ru.wikiversity'
18:47 RobH: test test
18:47 RobH: test test
17:44 RobH: knsq14, knsq15 both back online and in lvs pool, still working on knsq16, knsq17
17:00 RobH: knsq14-knsq17 reinstalled, getting added and updated by puppet, then they will be put back into service
16:24 RobH: knsq12 back in lvs pool, set knsq14-17 to false for reinstallation
16:11 hcatlin: deployed XSS fix to m.wiki. thanks to gcouprie and tstarling.
15:52 RobH: holding off on that, knsq12 is not up and online, fixing.
15:51 RobH: knsq8 in cluster, setting knsq14-knsq17 to false, then taking them down for reinstallation
15:49 RobH: knsq8 fixed. setting to true in lvs config
15:45 RobH: knsq8 being fixed and pushed into service, then resuming reinstalltions for rt#5 knsq14-knsq22
13:44 Tim: edited "permanently disable" link out of live m.wikipedia.org code to fix XSS
00:45 ^demon: blocked empty commit summaries in subversion, bug 25025

September 8

21:57 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24849 - Enable the Collection extension for creating books on the Finnish Wikipedia [fiwiki]'
21:52 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24936 - Localizing "Portal" namespace in si.wikipedia'
21:48 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24929 - Make Author namespace for srwikisource'
21:44 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24825 - Create "Livro'
21:39 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24576 - New namespaces for ml.wikisource'
21:28 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24394 - Install AbuseFilter on Hindi Wikipedia'
21:22 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25103 - Create "Pielikums" namespace on lv.wiktionary'
21:18 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24852 - New namespace for WikiProject on jawiki'
21:10 logmsgbot: jeluf synchronized php-1.5/wmf-config/abusefilter.php '25082 - Grant abusefilter-log-detail to only sysop and abusefilter group on jawiki '
18:08 domas: testing testing
18:07 RobH: domas added support for multiple channels into morebots and its now back, yayyyyy
17:54 RobH: killing morebots to restart it, i may end up breaking it completely and having to revert my changes
13:34 hcatlin: m. deployment complete and everything is looking good
13:33 hcatlin: deploying TStarling's UDP patch for mobile
05:55 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Remove BOM'
05:44 logmsgbot: jeluf synchronized php-1.5/wmf-config/abusefilter.php '24394 - Install AbuseFilter on Hindi Wikipedia'
05:36 logmsgbot: jeluf synchronized php-1.5/wmf-config/abusefilter.php '25082 - Grant abusefilter-log-detail to only sysop and abusefilter group on jawiki '
05:29 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25051 - Localize namespaces on newiki'

September 7

21:55 RobH: sq69, sq70 both back online and in lvs pool
21:51 Ryan_Lane: repooled sq68 in pybal bits config
21:49 mark: Powercycled sq68
21:48 RobH: rebooting sq69, sq70
21:41 Ryan_Lane: set sq68 to false in pybal's bits configuration
19:56 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesFrr.php 'r72545'
19:46 apergos: restarted en wiki full history (with prefetch this time).
19:29 logmsgbot: robh synchronized php-1.5/wmf-config/mc.php 'adding in srv258-289 into memcached spares, all test online'
18:47 Ryan_Lane: split pdf1-3 and mobile1-3 away from misc cluster in ganglia; made two new clusters (pdf and mobile)
18:33 apergos: starting xml full history dumps stage (paralellized) for en wik on snapshot2 in screen session from /backups-testing
17:38 Ryan_Lane: changed password for Utilisateur:Bot de paille on frwiki
14:37 mark: Powercycled mobile1
05:13 apergos: started meta-current phase of xml dumps for en wiki on snapshot2 in screen session
03:39 Tim: installed packet loss monitor on locke

September 6

20:23 apergos: started parallelized pages-articles xml dump for en wiki on snapshot2 in screen session.
11:23 apergos1: started parallel stubs dump for en wiki on snapshot2 in screen session
05:49 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/i18n/Lqt.namespaces.php
05:49 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/i18n/Lqt.namespaces.php
05:35 logmsgbot: andrew synchronizing Wikimedia installation... Revision: 72153
05:32 Andrew: Running l10nupdate, scap and installing LiquidThreads on huwiki

September 5

22:53 domas: why is srv193 not in prod?
22:52 logmsgbot: midom synchronized php-1.5/wmf-config/mc.php 'srv123 was not replaced mc.php, despite previous SAL messages'
21:04 logmsgbot: midom synchronized php-1.5/wmf-config/InitialiseSettings.php 'rolling back logo for axis of eee... nevermind'
04:46 apergos: started the abstracts dump, it works out to 11 jobs in parallel. (screen session on snapshot2) if this seems to bog down the db serves too much, someone can shoot it

September 4

23:13 apergos: starting enwiki xml dump run one stage at a time, with parallelization of later stages. running table dumps now from snapshot2 screen session, using /backups-testing directory as work area
07:16 apergos: tried restarting lighty with tpc6 enabled on dataset1 now that tcp6 connections are closed... no dice. back to ipv4 only
00:34 Ryan_Lane: added initial ldap configuration to puppet.

September 3

23:44 Ryan_Lane: added philippe to the wmffundraising alias
18:56 RobH: srv123 pulled, wiping. also all memcached in updated pool test fine.
18:56 logmsgbot: robh synchronized php-1.5/wmf-config/mc.php 'removing srv123 from memcached due to decommissioning'
18:38 RobH: rt#54 db16 seems to not be throwing errors... installs ok, passed all VTS testing and memtesting. reinstalled the OS
11:45 tomaszf: leaving lighttpd in non ipv6 mode till currently open sockets clear
10:39 tomaszf: disabling ipv6 for lighttpd on dataset1 as its not working post upgrade
10:20 tomaszf: upgrading to ubuntu 10.04 LTS on dataset1
01:44 Ryan_Lane: added pre-commit and post-commit hooks to puppet's svn to check syntax on pre-commit, and svn up on post-commit

September 2

23:09 Ryan_Lane: restarted mysql on db10
23:09 Ryan_Lane: set expire_logs_days to 14 on db10
23:04 Ryan_Lane: fixed password issue for debian-sys-maint on db10
22:56 Ryan_Lane: commented out skip_slave_start in the my.cnf on db10
22:54 Ryan_Lane: restarted db10 (specifically the mysql database)
22:54 Ryan_Lane: restarted db10
19:38 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'adding new thursday test banner to whitelist'
16:53 RobH: reinstalling storage2
16:09 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/PrefSwitch/SpecialPrefSwitch.php 'Attempt to fix DB error'
16:06 RobH: db16 is running VTS, DO NOT TOUCH THIS SERVER
15:51 RobH: db34 unracked, going back for repair with Oracle Field Engineer.
15:02 RobH: ignore errrors from db16, db28, db34. Today is sun server troubleshooting day.
14:58 RobH: db28 is not in rotation, taking it down to continue hardware testing/repair
14:25 RobH: srv206 back online and in cluster.
14:25 logmsgbot: robh synchronized php-1.5/wmf-config/mc.php 'moved srv206 from down to spare'
13:49 RobH: srv206 offline, looking into it.

September 1

22:42 mark: Shutdown zwinger
22:10 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/PrefSwitch/SpecialPrefSwitch.php 'Fix fatal'
21:58 Ryan_Lane: created article assessment wiki on prototype
21:31 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/PrefSwitch/SpecialPrefSwitch.php 'Attempted fix for skin overwriting problem'
21:19 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/PrefSwitch/SpecialPrefSwitch.php 'Attempted fix for DB issue'
20:56 Ryan_Lane: removed unneeded files from /var/tmp on prototype to free up space
20:43 Ryan_Lane: svn upped bugzilla/skins/custom/buglist.css and bugzilla/skins/custom/show_bug.css
20:42 Ryan_Lane: added an older version of PatchReader to kaulen for bugzilla
20:37 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialNoticeText.php 'Picking up changes from r72135'
20:37 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialNoticeLocal.php 'Picking up changes from r72135'
20:36 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialCentralNotice.php 'Picking up changes from r72135'
20:36 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/rebuildLocalTemplates.php 'Picking up changes from r72135'
20:36 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/rebuildTemplates.php 'Picking up changes from r72135'
20:35 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/NoticePage.php 'Picking up changes from r72135'
19:07 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/PrefSwitch/SpecialPrefSwitch.php 'Fix fatal in global opt-out'
18:56 RoanKattouw: Switchover is all done, everyone 's free to touch stuff again as far as I'm concerned
18:55 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Vector switchover, thumb size change. enable global opt-out'
18:54 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Vector switchover, thumb size change. enable global opt-out'
18:54 RoanKattouw: Deploying config for Vector switchover now
18:49 RobH: all new api servers mgmt accessible, still need OS installation.
18:48 RobH: pushing dns change, there was a typo in the zonefile for srv290 mgmt
18:37 RoanKattouw: Put config update for Vector switch in wmf-config/ and synced do test. DO NOT SYNC THIS until I confirm it's working
18:24 logmsgbot: catrope synchronized php-1.5/includes/UserRightsProxy.php 'r72132'
18:22 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/PrefSwitch/SpecialPrefSwitch.php 'r72127'
18:21 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/PrefSwitch/PrefSwitch.i18n.php 'r72127'
18:21 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/PrefSwitch/PrefSwitch.php 'r72127'
18:20 logmsgbot: catrope synchronized php-1.5/skins/Vector.php 'r72127'
18:20 RoanKattouw: Deploying r72127 and r72132 with individual syncs
18:03 RoanKattouw: Setting default skin to Vector on testwiki
17:53 RoanKattouw: Testing r72127 on testwiki
17:46 RobH: srv280-srv289 back online. all power balancing in b5-sdtpa complete.
17:22 RobH: srv270-srv279 back online, srv280-srv289 is next (and the last batch)
17:06 RobH: also taking down srv279 in last batch
17:03 RobH: srv264-srv269 back online, now bringing down srv270-srv278
16:49 RobH: morebots needs better parsing to read my mind.
16:49 RobH: srv258-srv263 back online. Now bringing down srv264-srv269
16:31 RobH: srv258-srv263 are being shutdown to have their power rebalanced within the rack. They will be back online shortly.

August 31

23:40 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialNoticeTemplate.php 'Picking up all changes in r72057'
23:40 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialCentralNotice.php 'Picking up all changes in r72057'
23:39 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/CentralNotice.i18n.php 'Picking up all changes in r72057'
23:39 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/CentralNotice.db.php 'Picking up all changes in r72057'
23:04 Ryan_Lane: added wikidev group to mchenry and sanger
22:02 mark: Puppetised jjones account with new ssh key
22:01 Ryan_Lane: modified ssh configuration on cluster to disable password login, except while inside of the cluster.
20:14 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 72043
20:14 RoanKattouw: Running scap to deploy Siebrand's recent commits to 1.16wmf4 (messages files and Names.php)
19:37 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24902 - tawikinews'
19:13 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24957 - enable editinterface user group on hindi wiki and grant bureacrats to add/remove this group'
19:11 Ryan_Lane: added review.tesla.usability.wikimedia.org to forward and reverse DNS
18:42 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24882 - Make NS aliases in Korean Wikinews.'
17:35 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 're-enabling MobileRedirect.js'
17:34 logmsgbot: tstarling synchronized php-1.5/extensions/WikimediaMobile/MobileRedirect.js 're-enabling'
17:15 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'bumping mobile JS version'
17:12 Tim: disabling mobile redirect since it's been down for a long time now
17:11 logmsgbot: tstarling synchronized php-1.5/extensions/WikimediaMobile/MobileRedirect.js
15:21 Tim: fixed mobile1 and mobile2 temporarily with killall -9 -u deploy && /etc/init.d/apache2 start
15:01 hcatlin: rolled back the lastest m. code release because of catostrophic cluster failure. hoping for recovery.
14:52 Tim: mobile mostly down, restored hcatlin's sudo access on mobile1
09:20 logmsgbot: root synchronizing Wikimedia installation... Revision: 71885
09:19 logmsgbot: tstarling synchronizing Wikimedia installation... Revision: 71885
09:11 Tim: added --ignore-externals to /h/w/b/l10nupdate to avoid SSL certificate warning when it attempts to update geshi
09:07 Tim: running LocalisationUpdate/update.php
04:56 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24956 - enable flood flag user on hindi wiki and grant bureacrats to add or remove this group'
04:52 logmsgbot: jeluf synchronized php-1.5/wmf-config/flaggedrevs.php 'Bug 24842 - Enable flagged revisions in category namespace on Hungarian Wikipedia'
04:50 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24957 - enable editinterface user group on hindi wiki and grant bureacrats to add/remove this group'
04:47 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23186 - Enable upload in portuguese Wikipedia'

August 30

23:57 mark: Changed root/enable passwords on mr1-pmtpa, msw1-sdtpa, msw1-pmtpa, asw3-pmtpa(asw-b3-pmtpa)
23:48 mark: Changed root/enable passwords on mr1-pmtpa
23:47 mark: Changed root/enable passwords on asw-a2-pmtpa, asw-b1-pmtpa, asw-b2-pmtpa, asw-b4-pmtpa, asw-c3-pmtpa, asw-c4-pmtpa
23:40 mark: Changed root/enable passwords on asw-b-sdtpa, asw-a4-sdtpa, asw-a5-sdtpa
23:33 mark: Changed root/enable passwords on csw1-esams, csw2-esams, br1-knams
23:19 mark: Changed root/enable passwords on csw1-sdtpa and csw5-pmtpa
20:47 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Enabling jQuery on every page view'
20:47 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enabling jQuery on every page view'
20:38 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24947 - ReaderFeedback Extension requested for Turkish news'
20:21 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '[Bug 24942] Disable file uploading for the Hungarian Wiktionary'
19:51 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'hiwki flaggedrevs'
19:45 logmsgbot: robh synchronized php-1.5/wmf-config/flaggedrevs.php 'setting hiwiki flaggedrevs reviewer group change assign from sysop to crat'
19:42 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24942 - Disable file uploading for the Hungarian Wiktionary'
19:40 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24846 - Activation of the 'autopatrolled' group on nn WP'
19:35 logmsgbot: jeluf synchronized closed.dblist
19:34 logmsgbot: robh synchronized php-1.5/wmf-config/flaggedrevs.php 'setting hiwiki flaggedrevs to same as enwiki'
19:27 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24458 - Enable subpages on frwikisource'
19:25 Ryan_Lane: added owa.tesla.usability.wikimedia.org to forward and reverse dns
18:52 logmsgbot: robh synchronized php-1.5/wmf-config/flaggedrevs.php
18:47 logmsgbot: robh synchronized php-1.5/wmf-config/flaggedrevs.php
17:39 logmsgbot: robh synchronized php-1.5/wmf-config/flaggedrevs.php
17:28 RobH: pushed changes for bug 24622
17:28 logmsgbot: robh ran sync-common-all
05:14 logmsgbot: tstarling synchronized php-1.5/extensions/PagedTiffHandler/PagedTiffHandler_body.php 'r71927'
04:42 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 're-enabling PagedTiffHandler'
02:56 logmsgbot: tstarling synchronized php-1.5/extensions/PagedTiffHandler/PagedTiffHandler_body.php 'r71924'

August 28

14:35 mark: Restarted pdns on ns2
12:14 logmsgbot: andrew synchronizing Wikimedia installation... Revision: 71783
12:14 Andrew: Finished testing AbuseFilter log entry deletion, rolling out to all wikis with scap
12:04 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'Testing abusefilter rights on testwiki'
11:57 Andrew: Setting up for rollout of abusefilter log entry deletion. Will be assigned to the oversight group
11:40 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'LQT for sewikimedia and svwikisource'
11:32 Andrew: deploying LiquidThreads to svwikimedia and svwikisource
01:14 brion: reset list admins & password for WikiPT per IRC request due to old admin being inactive. Set to the admins from WikimediaPT.

August 27

21:18 logmsgbot: jeluf synchronized langlist 'Bug 16077 - Remove nomcom from langlist'
21:14 logmsgbot: jeluf synchronized php-1.5/languages/messages/MessagesFrr.php 'Bug 24663 - linktrail, merge changes from TRUNK'
20:45 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24877 - Enable File Uploading in Korean Wikinews'
20:41 logmsgbot: jeluf synchronized php-1.5/cache/interwiki.cdb 'Updating interwiki cache'
20:35 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24878 - Change Korean Wikinews Logo'
20:34 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24791 - Enable subpage for "Templat" namespace in id.wikisource'
20:30 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24867 - Set logo ckb wiki'
20:28 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24930 - Change the logo image in Sinhala Wikipedia'
20:26 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24958 - Update default old en-logo of frrwiki to new localized logo version'
20:21 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24933 - Allow bureacrats to remove sysop on hindi wiki'
18:15 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24663 - Create North Frisian Wikipedia (wp/frr)'
15:19 RoanKattouw: convertUserOptions.php runs finished on s2 and s3 (s4-6 finished yesterday after ~5 hrs), s1 still running
12:21 apergos: image uploads were broken since the PagedTiffHandler ext was enabled. Part of its image checking logic was being applied to all images, not just tifs. Disabled for now, uploads working again
12:19 logmsgbot: ariel synchronized php-1.5/wmf-config/CommonSettings.php 'disabling PagedTiffHandler pending fixes'
09:00 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php '$wgTiffUseTiffinfo=true'
08:53 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'enabling PagedTiffHandler'
08:46 logmsgbot: root synchronizing Wikimedia installation... Revision: 71783
08:46 logmsgbot: tstarling synchronizing Wikimedia installation... Revision: 71783
08:45 Tim: deploying r71783: latest PagedTiffHandler changes
08:24 Tim: added "umask 002" to tomasz's .bashrc, so that he doesn't break the subversion working copy
05:57 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'Adding landing page five'
00:19 logmsgbot: tfinc synchronized php-1.5/includes/Pager.php 'Picking up fixes in r71098'
00:18 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/TemplatePager.php 'Picking up fixes in r71739'

August 26

21:51 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'Adding new banners for thursday test'
21:16 Ryan_Lane: rebooting ci.tesla
20:44 Ryan_Lane: redoing the backend IP scheme for the tesla nodes. Shouldn't affect anything unless you are logged into a node through the backend.
20:42 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialCentralNotice.php 'picking up GMT fix for CN'
19:50 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/CentralNotice.i18n.php 'CN Phase1 Deployment'
19:49 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/rebuildTemplates.php 'CN Phase1 Deployment'
19:47 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/centralnotice.js 'CN Phase1 Deployment'
19:47 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/centralnotice.css 'CN Phase1 Deployment'
19:46 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/TemplatePager.php 'CN Phase1 Deployment'
19:46 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialNoticeText.php 'CN Phase1 Deployment'
19:46 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialNoticeTemplate.php 'CN Phase1 Deployment'
19:45 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialNoticeLocal.php 'CN Phase1 Deployment'
19:45 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialCentralNotice.php 'CN Phase1 Deployment'
19:44 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/NoticePage.php 'CN Phase1 Deployment'
19:44 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/CentralNotice.php 'CN Phase1 Deployment'
19:43 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/CentralNotice.db.php 'CN Phase1 Deployment'
19:18 Ryan_Lane: reconfiguring tesla's backend networks. Selenium grid nodes may be down periodically
18:57 Fred: rebooting pdf1. Hard down.
18:45 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
18:44 logmsgbot: catrope synchronized php-1.5/skins/common/wikibits.js 'r71723'
18:43 RoanKattouw: Commented out srv217 in mediawiki-installation node list cause it's down
16:28 RoanKattouw: Starting six instances of convertUserOptions.php , one for each DB cluster, on hume
11:17 mark: Discarding mail to bugzilla-daemon@wikimedia.org on mchenry

August 25

23:30 RoanKattouw: Running convertUserOptions.php on eswiki, metawiki and enwikinews as a test run
16:28 RoanKattouw: Starting import of another batch of geograph images: 1.1GB in 9,665 files
06:15 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 'fawiki milestone logo attempt 3'
06:13 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 'fawiki milestone logo attempt 2'
06:02 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 'reverting fawiki change, source image deleted'
05:52 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 'milestone logo for fawiki'

August 24

19:18 Fred: puppetized Ryan Lane's account and privileges.
18:53 Fred: mobile3 has been deployed and is now in rotation.
18:01 Fred: adding mobile3.w.o to DNS
17:43 Fred: starting deployment of Mobile3
17:39 RobH: racked mobile3, handing off to fred to install
15:34 mark: Reran Nagios sync script to get MySQL monitoring back
13:07 mark: Removed unused IPs in the squid+lvs and public services subnets from DNS
08:51 logmsgbot: tstarling synchronized php-1.5/includes/media/Bitmap.php 'r71547'
08:19 logmsgbot: tstarling synchronized php-1.5/includes/media/Bitmap.php 'live patch to chain -resize with -thumbnail to fix bug 24824'

August 23

13:19 Tim: on ms4: fastcgi re-enabled
13:06 mark: Finished removing all 2009 daily zfs snapshots on ms4
12:08 Tim: on ms4: disabling fastcgi to try to get basic cached service
12:03 Tim: on ms4: increasing max thread count to 256
12:00 mark: Removing all the 2009-10 daily zfs snapshots on ms4
11:46 mark: Removing all the 2009-09 daily zfs snapshots on ms4
11:36 mark: Removing all the 2009-08 daily zfs snapshots on ms4
11:33 mark: Removed oldest daily thumbs zfs snapshot on ms4
10:43 Tim: on ms4: restarting webserver7 with fcgi re-enabled, reduced thread pool count to 4
10:42 mark: Added swap2 to /etc/vfstab on ms4
10:36 mark: zfs create -V 4gb rpool/swap2; swap -a /dev/zol/dsk/rpool/swap2 (on ms4)
10:11 Tim: trying to start webserver7 on ms4, will see if it crashes it again
09:56 mark: svcadm disable puppetd on ms4
09:11 Tim: ms4 back up, after some mucking around with /etc/vfstab
08:48 Tim: ms4 timeout on http, squid serving "cannot forward", will reboot
08:44 Tim: ms4 not responding to ssh, giving "stub start error" on http, trying serial console, very slow
03:05 domas: 'ps -ef | grep php-cgi | awk '$3==1 { print $2 }' | xargs kill; rm /tmp/https-ms4-5351d5c9/stub.pid' to recover from ms4 fastcgi death, not sure what are the causes yet
00:50 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'oops, wrong cluster'
00:49 logmsgbot: midom synchronized php-1.5/wmf-config/db.php

August 22

20:55 apergos: should probably debug the ms4 issue but I will sleep soon. yet another restart.
18:25 apergos: restarted webserver7 on ms4
11:16: apergos: reports that new thumbnails are not being generated on demand. See http://commons.wikimedia.org/w/index.php?title=Special:NewFiles&until=20100822111026. hard restarted apaches on the scalers.
10:02 apergos: had some converts on File:Approximated_*_Inscribed_in_a_Circle.gif stuck again on the image scalers, shot them

August 20

05:44 logmsgbot: andrew synchronized php-1.5/languages/Language.php 'Deploying r71329'
05:43 Andrew: not deploying r71327, actually deploying r71329 because r71327 does not work on 1.16wmf4
05:36 Andrew: deploying r71327

August 19

22:39 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'fixing case on banner bnames'
21:42 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'Adding new banners and appeal pages'
21:34 RobH: bug 24664 for mk chapter done
21:30 logmsgbot: robh ran sync-common-all
21:20 RobH: pushed live project ko.wikinews.org, no apache or dns changes needed since ko langcode was already in dns
21:18 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php
21:17 logmsgbot: robh ran sync-common-all
20:59 RobH: created new project frr.wikipedia.org, dns, apache, etc..
20:53 logmsgbot: robh ran sync-common-all
20:40 mark: Downpreffed AS16265 transit routes to local-pref 90
20:28 RobH: pushed dns changes and apache changes for the bookshelf project url, bug # 24872
20:28 mark: Turned up AS157 transit on 10G link e1/3 on br1-knams
14:34 Tim: killed hung convert on all image scalers

August 18

20:03 RobH: sq57 drive replaced, but raid didnt work (seems like grub wasnt copied to both drives) leaving offline for now, will investigate later
19:42 RobH: sq57 set to false in lvs, replacing bad disk.
18:33 RobH: kicking around db16, trying to fix it
10:37 mark: Restored VRRP priorities to original state
10:33 mark: authdns-scenario normal
10:30 mark: Enabled ve1 on csw1-esams
02:31 Tim: also edited /etc/gai.conf on fenari to prefer IPv4, to fix ExtensionDistributor
02:28 Tim: edited /etc/gai.conf on kaulen to avoid broken IPv6 connection to mayflower, so CR will start working again

August 17

22:57 mark: Shutdown ve1 on csw1 to force VRRP backup
22:53 mark: Packet loss, authdns-scenario esams-down
22:48 mark: authdns-scenario normal
22:43 mark: Configured all VRRP instances on csw1-esams to have priority 1, to reliably stay in backup mode
22:11 RobH: dns changed to route traffic to tampa
22:10 river: authdns-scenario esams-down
20:13 RobH: set srv278 to false in lvs, taking it down for hardware testing per rt#24
18:06 rainman-sr: disabling interwiki search on all wikis, not only en.wp until we figure out what is going on
16:49 rainman-sr: search11 is fully up with all features, and seems to work fine .. will keep an eye on it
16:43 RobH: srv230 online
16:41 rainman-sr: all of search up, still fiddling with search11 to see why it gave strange I/O spikes during the batch2 migration
16:37 RobH: investigating srv230.
16:32 RobH: srv230 back online with memory replacement, synced and back in cluster
16:31 logmsgbot: robh synchronized php-1.5/wmf-config/lucene.php
16:29 logmsgbot: robh synchronized php-1.5/wmf-config/lucene.php 'Returning all search values to normal, should restore full search functionality.'
16:22 rainman-sr: bringing up search5,12, 13-20
16:21 RobH: shutting down srv230 to swap out bad memory
15:50 RobH: search13-search20 relocated to b3-sdtpa. All servers are online, working to bring search back to full deployment.
14:36 rainman-sr: search5,12 will also show offline because they run parts of the services that are temporarly disabled
14:30 RobH: search13-search20 will show offline during their relocation, approx until 16:00 if all things go well
14:09 logmsgbot: robh synchronized php-1.5/wmf-config/lucene.php 'Disabling wgEnableLucenePrefixSearch on projects while search servers are relocated'
14:04 rainman-sr: starting relocation of second batch of search servers
13:05 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/i18n/Lqt.namespaces.php
13:05 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/i18n/Lqt.namespaces.php
12:19 logmsgbot: andrew synchronizing Wikimedia installation... Revision: 71190
12:19 Andrew: running scap to update LiquidThreads localisation
11:01 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'Add LiquidThreads to ptwikibooks'
10:34 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/LiquidThreads.php
10:34 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/i18n/Lqt.namespaces.php
10:33 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/i18n/Lqt.i18n.php
10:33 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/LiquidThreads.php
10:33 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/i18n/Lqt.namespaces.php
10:33 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/i18n/Lqt.i18n.php
05:56 logmsgbot: jeluf synchronized php-1.5/wmf-config/CommonSettings.php '22109 - add fourth level subdomains of wikimedia to '
05:42 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24808 - Localize Wikipedia name for newiki'
05:34 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24355 - Namespace changes - si.wikipedia'
05:31 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24255 - Change meta namespace names on siwikibooks'
05:24 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24579 - Reconfigure Turkish Wikibooks'
05:07 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24822 - XML-import for de-wiktionary'
05:05 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24823 - alias for de-wiktionary'
04:09 Tim: wikimedia-task-appserver on srv224 was half-configured because parentless convert processes stuck in a deadlock were holding the *:80 listen filehandle open, preventing apache from starting. Killed the convert processes and reinstalled.
04:05 Tim: deployment of the new wikimedia-task-appserver led to nagios alerts about half-configured status on 16 servers due to a warning in rsync sync-common about file permissions. Fixed by deleting Mark's vim swapfile and reinstalling.
03:36 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'disabling PagedTiffHandler, no identify present on non-scalers so upload fails'
03:26 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'enabling PagedTiffHandler'
03:23 logmsgbot: tstarling synchronizing Wikimedia installation... Revision: 71190
03:22 Tim: doing another scap to update ExtensionMessages.php
03:16 logmsgbot: root synchronizing Wikimedia installation... Revision: 71190
03:14 Tim: doing svn up/scap to r71188 for PagedTiffHandler and supporting changes

August 16

21:44 RobH: dns update for login vmhost on tesla, rt#108
20:38 Fred: rebooting srv217 ->oom
19:15 logmsgbot: mark synchronized php-1.5/extensions/MWSearch/MWSearch_body.php 'Reenabling the disabled section, as rainman has disabled interwiki results in the lucene backend'
18:51 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24304 - Reconfigure English Wikibooks'
18:50 logmsgbot: jeluf synchronized php-1.5/wmf-config/flaggedrevs.php '24304 - Reconfigure English Wikibooks'
18:49 rainman-sr: restarting search1,3,4,9
18:46 logmsgbot: root synchronized php-1.5/extensions/MWSearch/MWSearch_body.php 'Temporarily disable part of MWSearch'
18:45 JeLuF: archived admin logs, moved January-June 2010 to Archive 15
18:45 logmsgbot: root synchronized php-1.5/extensions/MWSearch/MWSearch_body.php 'Temporarily disable part of MWSearch'
18:43 logmsgbot: root synchronized php-1.5/extensions/MWSearch/MWSearch_body.php 'Temporarily disable part of MWSearch'
18:38 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '17627 - Allow autoconfirmed users to patrol on ar Wikisource'
18:08 Fred: MWSearch extension is having issues...
17:36 Fred: API servers all started segfaulting. Restarting apache for the time being
04:50 logmsgbot: jeluf synchronized php-1.5/wmf-config/abusefilter.php '24304 - Reconfigure English Wikibooks'
04:48 logmsgbot: jeluf synchronized php-1.5/wmf-config/flaggedrevs.php '24304 - Reconfigure English Wikibooks'

August 15

22:42 domas: resynced db11 and db17 from db27, db33 from ixia, db19 from db1, with accompanying BIOS flashing to 3.0 and OS reinstalls. decommissioned ixia and db1
22:40 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'drumrolllll'
22:04 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
21:56 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
21:49 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
21:39 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24520 - Transwiki import source for ml.wikiquote.org'
20:55 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24458 - Enable subpages on frwikisource'
20:39 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24623 - Enable 'eliminator' flag on ptwiki'
20:36 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24304 - Reconfigure English Wikibooks'
20:36 logmsgbot: jeluf synchronized php-1.5/wmf-config/flaggedrevs.php '24304 - reconfigure enwikibooks'
20:35 logmsgbot: jeluf synchronized php-1.5/wmf-config/abusefilter.php '24304 - reconfigure enwikibooks'
20:32 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
20:14 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24623 - Enable 'eliminator' flag on ptwiki'
20:09 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24321 - ml.wikiquote.org lost its project namespace'
20:04 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
19:54 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24777 - Request for a patrolling function on the Nynorsk (nn) Wikipedia'
19:48 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24535 - Enable "mark as patrolled" feature in hindi wiki'
19:44 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24394 - Install AbuseFilter on Hindi Wikipedia'
19:41 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24336 - Change Simplewiki's (EN) autoconfirm time/edit rates'
19:36 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24790 - Localize Wikipedia sitename in devanagari'
19:33 JeLuF: fixed zero byte thumbnail of commons:Shkval_head.jpg
19:19 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24789 - Enable AbuseFilter for ja.wikipedia'
19:18 logmsgbot: jeluf synchronized php-1.5/wmf-config/abusefilter.php '24789 - Enable AbuseFilter for ja.wikipedia'
19:14 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
18:44 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
18:24 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
17:44 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
16:23 logmsgbot: midom synchronized php-1.5/wmf-config/db.php

August 14

20:15 mark: Decommissioning srv150
19:56 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24789 - Enable AbuseFilter for ja.wikipedia'
19:52 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24626 - Add an "autopatrolled" status for frwiktionary'
15:37 mark: dobson has failed RAID1 array member /dev/sda. Running long SMART self test on /dev/sda
14:18 logmsgbot: mark synchronized php-1.5/wmf-config/db.php 'Add ms2 and ms1 to clusters rc1 an cluster22'
14:06 mark: FLUSH TABLES WITH READ LOCK on ms1 for testing
13:59 mark: Stopping mysql on ms1 as monitoring test
13:59 mark: Granted SELECT on mysql.* to nagios on ms3
10:57 mark: Removed oldest LVM snapshot on ixia
09:43 mark: Fixed apparmor profile /etc/apparmor.d/usr.sbin.mysqld on ms1, restarted mysql under apparmor
09:39 mark: START SLAVE on ms1, catching up with ms3
09:38 mark: RESET SLAVE on db5
09:37 mark: STOP SLAVE on db5
09:35 mark: Stopped apparmor on ms1
08:41 Andrew: Leaving as-is for now, hoping somebody with appropriate permissions can fix it later.
08:40 Andrew: STOP SLAVE on db5 gives me ERROR 1045 (00000): Access denied for user: 'wikiadmin@208.80.152.%' (Using password: NO)
08:34 Andrew: Slave is supposedly still running on db5. Assuming Roan didn't stop it when he switched masters a few days ago. Going to text somebody to confirm that stopping is correct course of action.
08:24 Andrew: db5 can't be lagged, it's the master ;-). Obviously something wrong with wfWaitForSlaves.
08:19 Andrew: db5 lagged 217904 seconds
05:09 Andrew: Ran thread_pending_relationship and thread_reaction schema changes on all LiquidThreads wikis
05:06 logmsgbot: andrew synchronizing Wikimedia installation... Revision: 70933
05:04 Andrew: About to update LiquidThreads production version to the alpha.

August 13

22:03 mark: API logins on commons (only) are reported broken
21:45 mark: Set correct $cluster variable for reinstalled knsq* squids
21:03 mark: Increased cache_mem from 1000 to 2500 on sq33, like the other API backend squids
20:58 mark: Stopping backend squid on sq33
20:50 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24769 - Import source addition for tpi.wikipedia.org'
17:46 Fred: and srv100
17:45 Fred: restarted apache on srv219 and srv222
15:57 logmsgbot: mark synchronized php-1.5/wmf-config/mc.php 'Remove some to-be-decommissioned from the down list'
15:56 logmsgbot: mark synchronized php-1.5/wmf-config/mc.php 'Remove some to-be-decommissioned hosts from the down list'
15:53 RobH: srv146 removed from puppet and nodelists, slated for wipe, decommissioned.
15:47 mark: Sent srv146 to death using echo b > /proc/sysrq-trigger. It had a read-only filesystem and is therefore decommissioned.
15:38 mark: Restarted backend squid on sq33
15:36 logmsgbot: mark synchronized php-1.5/wmf-config/mc.php 'Remove some to-be-decommissioned hosts from the down list'
15:25 mark: Reinstalled sq32 with Lucid
15:01 mark: Removed sq86 and sq87 from API LVS pool
14:55 mark: sq80 had been down for a long time. Brought it back up and synced it
14:54 rainman-sr: all of the search cluster restored to pre-relocation configuration
14:34 logmsgbot: robh synchronized php-1.5/wmf-config/lucene.php 'reverting search13 to search11'
13:55 mark: /dev/sda on sq57 is busted
13:54 RobH: removed search17 from search_pool_3
13:50 mark: Set idleconnection.timeout = 300 (NOT idlecommand.timeout) on all LVS services on lvs3, restarting pybal
13:44 mark: powercycled sq57, which was stuck in [16538652.048532] BUG: soft lockup - CPU#3 stuck for 61s! [gmond:15746]
13:42 mark: sq58 was down for a long long time. Brought it back up and synced it
13:37 RobH: added search7 back into search_pool_3, kept search17 in as well
13:27 RobH: changed search_pool_3 back from search7 to search17 since it failed
13:25 logmsgbot: robh synchronized php-1.5/wmf-config/lucene.php 'Re-enabling LucenePrefixSearch - pushed changes on lvs3 to put search back to normal use'
12:45 mark: API squid cluster is too flaky to my taste. Converting sq33 into an API backend squid as well
12:40 mark: Shutdown puppet and backend squid on sq32
11:41 mark: Corrected changed hostname for api.svc.pmtpa.wmnet in text squid config files
11:37 mark: Temporarily rejecting requests to sq31 backend to give it some breathing room while it's reading its COSS dirs
11:32 mark: Reinstalled sq31 with Lucid
10:25 mark: Shutting down backend squid on sq31 to see the load impact
10:18 mark: Setup backend request statistics for the API on torrus
09:15 rainman-sr: bringing up search1-12 and doing some initial index warmups
01:53 RobH: searchidx1, search1-search12 relocated and online, not in cluster until Robert can fix in the morning. The other half will have to move on a different day, 12 hours in the datacenter is long enough.
01:40 RobH: finished moving searchidx1 and search1-12, bringin them back up now

August 12

23:10 RobH: shutting down searchidx1, search1-12 for move
22:40 logmsgbot: robh synchronized php-1.5/wmf-config/lucene.php 'swapped search13 and search18 for migration'
22:37 logmsgbot: robh synchronized php-1.5/wmf-config/lucene.php 'reverting so search13 and search18 can change roles'
22:22 logmsgbot: robh synchronized php-1.5/wmf-config/lucene.php 'changes back in place to migrate searchidx1 and search1-10'
22:19 RobH: puppet updated on all search servers, confirmed all have all three lvs ip addresses
21:55 mark: Configured puppet to bind all LVS service IPs to all search servers
21:54 RobH: reverted search_pool changes on lvs
21:54 logmsgbot: robh synchronized php-1.5/wmf-config/lucene.php 'rolling it back'
21:48 logmsgbot: robh synchronized php-1.5/wmf-config/lucene.php 'changing settings for migration of searchidx1 and search1-search12'
21:43 RobH: changing lvs3 search pool settings for server relocations
20:33 logmsgbot: robh synchronized php-1.5/wmf-config/lucene.php 'commented out wgEnableLucenePrefixSearch for search server relocation'
19:30 RobH: srv281 reinstall done but not online as puppet has multiple package issues, leaving out of lvs
19:09 RobH: srv230 is on, but set to false in lvs. do not push back into rotation until after new memory arrives and is installed tomorrow (rt#69)
18:59 logmsgbot: robh synchronized php-1.5/wmf-config/mc.php 'updating without srv230'
18:53 RobH: srv230 coming down for memory testing
18:49 RobH: set srv230 to false in lvs, need to test memory
18:04 RobH: reinstalling srv281
17:59 RobH: nix that, srv125 was ex-es, leaving those for now.
17:58 RobH: pulling srv103 & srv125 for wipe (pulling stuff with temp warnings first)
17:53 logmsgbot: robh synchronized php-1.5/wmf-config/mc.php 'removed srv103, replacing it with srv244'
17:47 RobH: pulling srv95 for wipe
17:38 RobH: srv110 removed from lvs3 config
17:36 mark: Removed all apaches up to srv150 from the appserver LVS pool on lvs3
17:21 Fred: restarting apache on webservers (220,221,222,224)
16:45 RobH: wipe running on adler and amane, and they have been removed from puppet and dsh node groups
16:12 logmsgbot: jeluf synchronized docroot/bits/index.html
15:41 mark: Setup ports ge-2/0/0 to ge-2/0/20 for search servers on asw-b-sdtpa
15:03 mark: Shutdown BGP session to AS1257 130.244.6.249 on port 2/5 of br1-knams, preparing for cable move
13:08 mark: Recovered backend squid on knsq11
12:53 mark: Reassembling RAID arrays md0 and md1 on knsq11
12:40 mark: Running apt-get upgrade && reboot on amssq31
11:17 mark: Shutdown knsq1 and knsq11 for swapping drives
09:34 logmsgbot: catrope synchronized php-1.5/extensions/TitleBlacklist/TitleBlacklist.hooks.php 'r70933'
09:08 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 24710: Enable $wgVectorShowVariantName on srwiki'
06:00 logmsgbot: jeluf synchronized php-1.5/cache/interwiki.cdb 'Updating interwiki cache'
05:42 JeLuF: Bug 24736 - Update wikimania.wikimedia.org to point to wm2011
02:43 RobH: dataset1 back online, serving http, any jobs in process are borked (sorry ariel!)
02:39 RobH: dataset1 unresponsive to physical console, serial console, had to do a hard reset
02:05 RobH: dataset1 crashed while querying its raid controller about a bad disk, in route to dc to fix.

August 11

21:56 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24561 - Quiz on Polish Wikibooks'
21:53 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '21375 - new wiki as internal working space for the fiwiki arbcom'
21:48 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24486 - Create Appendix namespace on the Luxembourgish Wiktionary'
21:46 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24374 - Create new usergroups at commons'
21:28 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24460 - Set up transwiki import for lb.wiktionary'
21:10 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24626 - Add an "autopatrolled" status for frwiktionary'
21:06 RobH: db16 will remain offline until replacement parts arrive from Sun. rt#54
21:02 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24570 - Request'
20:42 mark: Fixed ganglia mess on ms1
20:37 mark: Started rsync of ms2:/a to ms1:/a
20:32 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24685 - Author, Index, Page namespace for id.wikisource'
20:30 mark: FLUSH TABLES WITH READ LOCK on ms2
20:27 mark: Readded spare /dev/mdak1 to /dev/md1 on ms1. Why do spares go missing all the time...
20:26 mark: Upgraded ms1 to Lucid, rebooted it
20:26 RobH: working on db16
20:24 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24719 - Extension'
20:07 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
20:07 domas: promoted db5 to slave on s4
19:53 mark: Upgrading ms1 to Lucid
19:44 mark: Readded missing spare drive to /dev/md1 on ms1
17:25 logmsgbot: robh synchronized php-1.5/wmf-config/mc.php 'removed srv95 as it has temp warnings and is going to go away soon.'
16:41 RobH_dc: pulled network on srv110 and started wipe, byebye
16:35 RobH_dc: db19 eth1 disconnected per dc tasks
16:24 RobH_dc: ms1 power cable was messing it up, rerouted the cable to be securely in place and system is now operating normally (no more sudden shutdowns hopefully) there was no evidence of hardware failure in logs, but power issues, so this should fix it.
16:02 RobH_dc: working on ms1
15:22 RobH_dc: bad disk replaced in db7, raid is currently rebuilding, system still online.
15:10 RobH_dc: pulled hdd5 from db7 for replacement
15:02 mark: Shutdown clematis for decommissioning
14:18 RobH: knsq11,knsq12,knsq13 are post os reinstall, pre squid deployment config, will finish them in a bit
14:07 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 24441 - Enable Rollback in Quechua Wikipedia'
13:33 RobH: knsq11-knsq13 coming down for reinstallation
12:23 Tim: deployed non-threaded version of imagemagick on all image scalers
11:43 logmsgbot: tstarling synchronized php-1.5/includes/media/Bitmap.php 'OMP_NUM_THREADS=1'
11:21 mark: Reconfigured wikimedia-lvs-realserver on hume, so wikimedia-task-appserver install succeeds
11:19 logmsgbot: tstarling synchronized php-1.5/includes/media/Bitmap.php 'reduced magick memory limit from 100M to 50M to stop hanging with vsize limit 300M'
10:46 mark: Removed pattern check from nagios check_http
09:42 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php
09:38 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php
09:35 Tim: rebooting srv223, went OOM and mostly died
09:32 logmsgbot: tstarling synchronized php-1.5/includes/media/Bitmap.php 'temporary patch to stop scalers going OOM'
09:19 Tim: temporarily increased memory limit on the image scalers, since the new convert tends to hang when it runs out of memory instead of crashing nicely
09:17 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'more memory for image scalers'
08:56 Tim: upgrading imagemagick on image scalers to 6.6.2.6-1wm1, package recently committed to svn
02:48 Tim: on techblog, disabled WP_DEBUG since it was messing up the admin panels with E_NOTICE messages
02:42 Tim: disabled WP-SpamFree on techblog due to bug 19540

August 10

23:12 Fred: upgraded Tridge to Lucid. Now rebooting.
22:04 RobH: knsq10 back online
20:59 RobH: knsq10 reinstalling
20:44 RobH: knsq9 online
19:37 RobH: handed off knsq8 to mark, reinstalling knsq9
19:02 ^demon: disabled svn post-commit hook for parser tests, long-since broken
18:57 mark: Stopping backend squid on amssq60 for testing
15:24 RobH: knsq8 reinstalled, not yet online, will push online shortly
14:56 mark: Setup RT on rt.wikimedia.org (streber)
14:32 RobH: knsq30 online and in cluster, knsq8 coming down for work
14:18 RobH: updated wordpress versions on blog.wikimedia.org and techblog.wikimedia.org
13:35 RobH: finishing install on knsq30
12:50 Tim: installed schroot on stafford, for hardy versions of uupdate etc.
11:19 mark: Fixed broken hourly cron job mw-serve
11:18 mark: Changed su www-data into su mwlib in cleanup cronjob on pdf1
10:23 mark: Removed broken daily system health report on srv178
10:22 mark: Removed broken daily system health report on db4
07:13 logmsgbot: andrew synchronized php-1.5/extensions/CommunityApplications/SpecialCommunityApplications.php 'Merge r70798'
07:13 logmsgbot: andrew synchronized php-1.5/extensions/CommunityApplications/CommunityApplications.i18n.php 'Merge r70798'
04:02 RobH: knsq30 set to false in pybal, install half done, will finish tomorrow morning.
02:44 RobH: knsq29 online and in cluster
02:30 RobH: knsq30 reinstalling
00:09 RobH: knsq28 back online
00:03 RobH: knsq27 back online
00:03 RobH: knsq29 reinstalling

August 9

23:34 RobH: knsq28 reinstalling
23:32 RobH: knsq26 online
23:32 RobH: knsq25 online
23:12 RobH: continuing reinstallation, ignore errors for knsq27, reinstalling
22:33 RobH: knsq23, knsq24 back online, knsq25, knsq26 still being reinstalled, knsq27-30 still online not yet reinstalled
21:51 mark: Added Nagios router interfaces check for br1-knams (using puppet)
21:11 mark: Unmounted /dev/sda6 (/a) on srv171, replaced it by /dev/mapper/nonredundant-data (LV with the same data and more space)
21:02 RobH: knsq24, knsq25, knsq26, knsq27 coming down for reinstall and puppetfication
20:49 RobH: knsq23 reinstall done and pushed back into cluster
18:40 mark: Running apt-get upgrade on db9
18:29 mark: Fixed ganglia mess on sq45
18:24 mark: Powercycled sq45
18:18 mark: Added a new MegaCli64 to wikimedia-raid-utils, made check-raid.py use it instead (we have all 64 bit servers anyway), and deployed the new package to the repository. Puppet will upgrade it everywhere.
16:26 Fred: fixed DPKG issue on transcode... another one of those conflicting gmond install
16:14 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 24735: Sanitize private/fishbowl config'
16:03 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 24732: Portal and Book namespaces for yowiki'
13:59 mark: srv110 decommissioned itself
13:55 RobH: knsq23 coming down for reinstallation
13:31 mark: Changed broken HTTP nagios check for Squid on brewster into a TCP port check
13:28 mark: Stopped MySQL on srv171, created LVM PV,VG and LV on unused drive /dev/sdb. Copying MySQL data onto it.
13:09 mark: START SLAVE on srv171 to get rid of relay binlogs
12:56 mark: Shutdown db3 for decommissioning
12:56 RoanKattouw: Mark 12:53 Shutdown db2 for decommissioning
12:56 RoanKattouw: 12:52 mark synchronized php-1.5/wmf-config/db.php 'Remove db3 from rotation, decommissioning'
12:55 RoanKattouw: Mark 12:47 Power cycled pdf3, out of memory
12:55 RoanKattouw: Mark 12:44 Restarted Apache on srv91
12:55 RoanKattouw: Mark 12:39 Relaxed NTP peers check for dobson and linne (NTP servers)
12:55 RoanKattouw: Mark 12:36 Shutdown adler for decommissioning
12:54 RoanKattouw: Mark 12:18 Made disk space on mchenry by DELETING LOTS OF OLD BACKUPS
12:53 RoanKattouw: Restarted morebots

August 8

20:43 mark: Setup new aggregated graphs at http://torrus.wikimedia.org/torrus/CDN?path=%2FTotals%2F (including bits)
09:49 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 24715: Fix typo'

August 7

11:58 logmsgbot: mark synchronized php-1.5/wmf-config/db.php 'New master: db18, r/w'
11:57 mark: Changed master for s3 to db18 on db11, db27, db25
11:49 mark: New master db18 log position: db18.bin.001 pos 79
11:32 logmsgbot: mark synchronized php-1.5/wmf-config/db.php 'Setting s3 to read-only'
11:21 mark: Stopping mysql on db17
11:21 mark: For reference, SHOW SLAVE STATUS on db18 before the switch:

      Master_Log_File: db17-bin.368
  Read_Master_Log_Pos: 650276717
       Relay_Log_File: db18-relay-bin.048
        Relay_Log_Pos: 650276247
Relay_Master_Log_File: db17-bin.368
     Slave_IO_Running: No
    Slave_SQL_Running: Yes

11:02 RoanKattouw: All s3 slaves down, master serving all read load and getting overloaded
11:00 RoanKattouw: db17 (s3 master) has full disk

August 6

22:59 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
22:59 logmsgbot: catrope synchronized php-1.5/includes/GlobalFunctions.php 'r70605'
22:58 logmsgbot: catrope synchronized php-1.5/skins/vector/main-rtl.css 'r70605'
22:56 logmsgbot: catrope synchronized php-1.5/skins/vector/main-ltr.css 'r70605'
15:43 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesCs.php 'r70573'
15:08 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 24688: Namespace aliases for kowiki'
10:45 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaLicenseTexts.i18n.php 'r70550'
10:11 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Add foundationwiki addgroups in correct section'

August 5

21:14 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 24678: Set $wgAddGroups, $wgRemoveGroups on foundationwiki'
14:30 RobH: replacing certificate file on sanger

August 4

22:57 rainman-sr: search1 somehow got stuck, restarting
22:42 Fred: restarted Nagios bot
16:28 mark: Reverted Fred's automatic security upgrades in puppet
16:25 mark: base::puppet and base::apt were not being included on every Linux host, fixed the case statement in the Puppet base class
11:35 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 24652: Allow bureaucrats to add/remove communityapps group on officewiki'

August 3

22:58 mark: Changed /etc/default/exim4 to make exim listen on SMTP
22:56 mark: s/srv9/grosley/ on /etc/exim4/exim4.conf on grosley
21:20 RobH: grosley exim config was overwritten when converted to puppet control (default install uses simple exim setup now). defined host specifically, adding in ganglia details and removing exim control from puppet
20:21 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 24652: Archive namespace for officewiki'
20:10 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
20:09 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js 'r70409'
15:59 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesEo.php 'r70387'
14:24 RobH: knsq7 coming down for reinstall
14:24 RobH: kndq7 coming down for reinstall
04:57 Andrew: scap
04:57 logmsgbot: andrew synchronizing Wikimedia installation... Revision: 70064
04:55 Andrew: Preparing to update LiquidThreads alpha to trunk with r70106 and r70100 unmerged.
01:30 mark: Restored exim config on williams (OTRS)
00:35 RobH: otrs emails still bouncing, working on it.
00:31 RobH: exim was sitting idle? on williams for otrs delivery after complaining of a failed database connection. restarted exim, it appears to be working on the delivery backlog now, will check back on it in 30 minutes or so

August 2

23:55 guillom: afaict, all emails sent to OTRS are being rejected with the "retry time not reached for any host after a long failure period" message. The issue seems to have started a few hours ago.
22:34 tomaszf: upgrading civicrm to 3.1.6
18:20 RobH: pulling knsq6 for reinstallation and such
17:47 RobH: sq33-sq40 reinstalled, online, serving requests
15:54 RobH: reinstalling sq34-sq40
14:55 mark: Installed gmetad on streber for collecting ganglia information in local RRDs
14:34 mark: Upgraded streber to Ubuntu 10.04
14:24 RobH: restarted pdns on linne
14:20 RobH: updated dns for tesla host ci
12:46 mark: Fixed ganglia package mess on search12-20
12:10 mark: Depooled all text squids from the bits.esams LVS pool
12:02 mark: Reinstalled knsq2 and knsq5 with Ubuntu 10.04, set them up as Varnish bits caches, and pooled them in LVS
11:50 mark: Fixed ganglia mess on kaulen
11:43 mark: Added missing Wikimedia APT repository to kaulen. Why was it not there? Was this host installed in some nonstandard way?
11:35 mark: Set up exim::simple-mail-sender classes for kaulen
11:35 mark: Fixed puppet for snapshot*, base classes were not included
11:19 mark: Turned off knsq3; broken HBA and out of warranty
10:55 mark: Started puppetd on spence inside gdb

August 1

17:21 mark: Reinstalled linne.wikimedia.org with Lucid
15:41 mark: Fixed the haproxy for puppetmaster on brewster, was broken by the upgrade
15:19 mark: Upgraded brewster to Ubuntu 10.04 Lucid
15:18 mark: Removed Wikimedia repository default pinning on brewster, as it's doing more harm than good
15:07 mark: Explicitly install and deinstall gmond / ganglia-monitor packages in puppet, depending on the ubuntu version
14:52 mark: Removed broken amanda backup client from brewster. Broken package install/dpkg state, no logs, no documentation

July 31

17:31 mark: Setup bits.esams varnish cluster, pooled knsq4 (varnish) with all the text squids
17:06 mark: Reinstalled knsq4 with lucid, redeploying it for varnish
16:24 mark: Depooled knsq1-knsq4 in squid config

July 30

20:00 domas: hotfixed db12 build to have faster mysqldumps
19:01 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 24466 reversion'
18:29 RobH: gracefulled apaches a few times, had a rogue typo
17:45 RobH: dns push successful, all nameservers still online and up to date
17:45 RobH: updated dns for movementroleswiki
17:43 logmsgbot: robh ran sync-common-all
16:51 RobH: added movement roles information to apache, still setting up other stuff for it
05:53 Tim: svn up/scap r70064

July 28

15:26 Rob: running updates on sockpuppet
13:08 logmsgbot: andrew synchronizing Wikimedia installation... Revision: 70064
12:57 Andrew: about to update LiquidThreads alpha to trunk state
07:54 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'adding 1.16 branch to ExtensionDistributor'
05:01 apergos: added bayes to list of clients for exports on dataset1; added /data to fstab on bayes for stats use
04:23 logmsgbot: tstarling synchronized php-1.5/includes/api/ApiBase.php
04:22 logmsgbot: tstarling synchronized php-1.5/includes/api/ApiMain.php
04:22 Tim: deploying r70063 and r70064 to fix API fatals
02:09 logmsgbot: root synchronizing Wikimedia installation... Revision: 70061
02:04 Tim: doing svn up/scap to r70061, to get the API cache header fix

July 27

23:27 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 24466'
21:39 mark: Added wikimedia-base to the standard packages list in puppet
21:34 Rob: pushed hu language file change on survey.wikimedia.org
21:16 Rob: singer is being a pain, ssh isnt running, so forth... working on it
19:07 Rob: racktables physical audit of pmtpa done (ignore the bottom of rack a5, will remove them later today when I have a damned mouse)
18:49 Rob: physical audit pmtpa row b complete
18:40 Rob: physical audit pmtpa row c complete
18:18 Rob: uplink successfully moved for asw-b5-sdtpa
18:17 Rob: moving the uplink
18:17 Rob: all the apaches in sdtpa-b5 are migrated from old to new asw-b5-sdtpa.
18:05 Rob: bugzilla admin note: disregard the email bounces for the scireview domain user, their mail server will be fixed shortly
17:58 Rob: srv284 fixed, drac online, needs setup
17:54 Rob: !log srv281 was shutdown when I came into the DC. popped case to use as base for rebuilding another system. need to investigate its initial shutdown
17:17 mark: Reinstalled sq68
16:41 atglenn: started up several more workers on snapshot3 doing xml dumps
14:43 Tim: running svn cleanup on some ExtensionDistributor working copies
14:29 Tim: cleaning up old ext-dist tarballs, removing all that were older than a month
11:48 mark: Stracing puppet on spence
08:01 apergos: started one thread of xml dumps from screen session on snapshot3 as root, if these look good tomorrow we'll crank up more of 'em

July 26

18:29 Rob: moved up to srv268, all working
18:25 Rob: moved srv258-srv261 to new asw-b5-sdtpa ports, all seems shiny
17:59 Rob: hooked up additional ports for sq67/sq68
17:17 mark: Installed Lucid on sq69 and sq70
14:41 mark: Moved db19 back to vlan 2
14:34 mark: Pooled sq68 in the bits.pmtpa varnish LVS pool
14:06 mark: Installed lucid on sq68 and deployed varnish for bits.pmtpa
12:22 mark: Started puppet on spence
05:21 Tim: removed a log on search12 to give it a tiny bit more space

July 25

21:55 mark: Pooled sq67 (Varnish) in the bits.pmtpa LVS pool along with the text squids
21:54 mark: Moved bits.pmtpa.wikimedia.org DNS from 208.80.152.2 (text squids) to 208.80.152.118 (dedicated LVS service)
21:21 mark: Started puppetd on spence
20:50 mark: Setup Nagios monitoring for bits.pmtpa
20:33 mark: Added LVS service for bits.pmtpa.wikimedia.org on lvs4
19:53 mark: Added LVS service ip 208.80.152.118 (bits.pmtpa.wikimedia.org) to all text squids and sq67
19:18 mark: Fixed ganglia varnish monitoring on sq67
13:01 domas: thumbs get i/o errors across multiple clients
10:42 domas: xmltypecheck loop filled / with error logging on srv243 ( http://p.defau.lt/?KtZMFW9x7xaEa2eDwuqMEw ) - had to do some cleanup. didn't livehack anything yet.

July 24

21:31 mark: Deployed varnish with configuration for bits.wikimedia.org on sq67 - not active yet
03:59 apergos: running xml dump file consistency checking script on snapshot2 and snapshot3 in screen as root, expect these to run all night, maybe through the next day as well

July 23

19:17 Rob: synced for Bug 24470 - Enable NewUserMessage extension on lv.wikipedia
19:17 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php
18:21 tomaszf: moving otrs-783-529404994.sql to tridge to free up space
18:21 kaldari: Backing up dev_civicrm database in preparation for drupal upgrade
11:33 RoanKattouw: Started Apache on srv278, had died mysteriously
09:30 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 24505: Editprotected group for hiwiki'
02:02 Rob: dig tested against all three nameservers after update, all nominal
01:59 Rob: pushing dns update to fix pointer for bugs.wikimedia.org
01:30 Tim: running some statistics queries on db38

July 22

19:31 Rob: mail server works, bypass expiration issue, rob will fix when all of the employees are not hitting the mail server.
18:13 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 'put ixia back'
17:50 Rob: wmf mail server cert expired, we know, working on replacing it now
17:29 mark: Fixed ganglia data sources by qualifying all hostnames in gmetad_pmtpa.conf; this must have broken by the resolv.conf change last week
17:15 logmsgbot: catrope synchronized php-1.5/skins/MonoBook.php 'r69735'
17:15 logmsgbot: catrope synchronized php-1.5/skins/Vector.php 'r69735'
17:11 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Hide Navigable TOC preference'
17:10 atglenn: xml dumps stopped in preparation for removal of bad dumps and xml code fix push
17:09 atglenn: snapshot3 added back to sync cluster in preparation for xml dumps fixes
16:50 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 'put db27 back'
16:37 mark: Installed Lucid on sq67
16:26 mark: Made Lucid the default distribution for new installs
16:21 mark: Fixed sq73
15:56 mark: Fixed ganglia cron job on spence.
15:46 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 'put db7 back'
15:29 RoanKattouw: Looks like that resolved the zombie issue on srv86
15:15 mark: Fixed puppet for app servers
15:05 RoanKattouw: Defunct Apache process running on srv86, Apache won't start. Need root to kill zombie
15:04 RoanKattouw: Started Apache on srv270, srv163, had died for some reason
15:04 RoanKattouw: Deployed r69728, r69729 (UsabilityInitiative, Vector updates) about 30 minutes ago
15:03 RoanKattouw: <RoanKattouw> !log Deploying r69728, r69729 (UsabilityInitiative, Vector updates) to test.wikipedia.org
15:03 RoanKattouw: <mark> !log Upgraded Locke to Ubuntu 10.04 Lucid
15:03 RoanKattouw: <logmsgbot> !log kate synchronized php-1.5/wmf-config/db.php 'removed ixia to dump s4 for TS'
15:03 RoanKattouw: <logmsgbot> !log kate synchronized php-1.5/wmf-config/db.php 'removed db7 to dump s6 for TS'
15:03 RoanKattouw: <logmsgbot> !log kate synchronized php-1.5/wmf-config/db.php 'removing db27 to dump s3 for TS'
15:02 RoanKattouw: Started morebots as catrope. werdnum has a defunt instance running that I can't kill

July 21

20:02 logmsgbot: catrope synchronized php-1.5/extensions/CodeReview/ui/CodeRevisionListView.php 'r69705'
19:44 Rob: all 5 new misc servers have mgmt working, will allocate network port in a bit and get them loaded up
19:43 mark: Setup logging for Special:Book on locke
19:42 Rob: all dns servers back online and happy
19:38 Rob: grosley back online providing services
19:30 Rob: grosley was down due to my dns issue, rebooting it, sorry about that.
19:29 Rob: restarted pdns on linne
19:25 Rob: fixed letter reversal in dns, repushed
19:13 Rob: linne pdns restarted
19:06 Rob: updating dns for 5 new misc servers mgmt
16:35 Rob: lots more time wasted diagnosing evident mainboard failures on srv284, hopefully replacement will soon be inroute
16:10 mark: Done basic setup of asw-b-sdtpa, added to RANCID and torrus
14:27 Rob: ms1 is back, passing it off to someone who doesnt hate it
14:06 Rob: working on ms1, no touchy
09:56 logmsgbot: catrope synchronized php-1.5/includes/api/ApiLogin.php 'r69661'
02:23 Tim: increasing nagios retry count for NTP from 8 to 15

July 20

15:55 Rob: pushed blog and techblog updates on existing plugins, core wordpress, but not themes (because I dont feel like rehacking our css)
15:41 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 24374 - Create new usergroups at commons'
15:35 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 24364 - Install Extension:Collection for PDF export on foundation wiki'
15:16 Rob: synced for bug 24449 disable file talk pages on cs wikis (since they all have upload disabled)
15:15 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php
14:34 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'noboard_chapterswikimedia logo'
14:16 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Change dbname noboardwiki -> noboard_chapterswikimedia'
14:12 logmsgbot: robh ran sync-common-all
13:54 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'communityapplications is too long for user_groups table'
13:48 logmsgbot: andrew synchronizing Wikimedia installation... Revision: 69504
13:48 Andrew: scapping to deploy CommunityApplications extension
13:47 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'New group for viewing community applications'
13:42 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'New group for viewing community applications'
13:32 logmsgbot: robh ran sync-common-all
12:21 logmsgbot: andrew synchronized php-1.5/extensions/CommunityHiring/SpecialCommunityHiring.php 'r69606'
12:20 logmsgbot: andrew synchronized php-1.5/extensions/CommunityHiring/CommunityHiring.php
12:20 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'Configuration changes for r69606'
11:53 Andrew: Adding CommunityHiring tables to officewiki database

July 19

20:05 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 24448 - w:cs: upload settings'
18:58 aZaFred_OSCON: started slave on db10 (sync had stopped on July 8th)
16:59 Rob: upded tfinc email quota cuz i was tired of seeing the alerts to postmaster
13:09 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 24440: New namespaces for dawikisource'
10:20 logmsgbot: midom synchronized wmf-deployment/cache/trusted-xff.cdb
08:32 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 24435: Enable $wgBlockAllowsUTEdit on hiwiki'
07:32 Tim: changing max retry count on NTP nagios monitoring to avoid constant flapping
06:45 Tim: restarted ntpd on mchenry, wasn't responding on IPv4

July 18

21:26 domas: fixed opensearch caching, unbreaked API
21:18 logmsgbot: midom synchronized php-1.5/includes/api/ApiMain.php 'fix broken API caching'
15:10 logmsgbot: mark synchronizing Wikimedia installation... Revision: 69504
14:59 mark: Making puppet upgrade wikimedia-raid-utils on all servers
12:58 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaLicenseTexts.i18n.php 'r69504'
12:51 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'New groups for hiwiki (bugs 24416, 24417, 24418, 24419)'
08:13 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'db13 and db15 go live'

July 17

22:01 mark: Apparently package wikimedia-raid-utils truncated /etc/sudoers on many hosts; having puppet put a proper sudoers file back on the application servers
21:42 mark: Fixed gmond mess on search12
21:03 RoanKattouw: Importing another batch of ~10,500 files (~22GB) concurrently with the first one
19:38 RoanKattouw: Import is running in a screen on fenari
19:38 RoanKattouw: Importing ~10,500 files (aggregate size ~21GB) with importImages.php for commons:User:OrdnanceSurveyBot
18:49 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesLad.php 'r69487'
18:48 mark: Having puppet install wikimedia-raid-utils on all servers
18:12 mark: srv278 rebooted for no apparent reason
17:21 mark: Fixed ganglia mess on search19
16:19 mark: Having puppet install NRPE on all internal servers by default
15:52 logmsgbot: root synchronized php-1.5/wmf-config/lucene.php 'Use fixed LVS service search-pool3 instead of search7 directly'
15:31 mark: Removed fscking nscd from spence
15:20 mark: Added internal LVS services to pmtpa.wmnet DNS
15:11 mark: Doing reboot test of nescio (esams DNS)
15:08 mark: Fixed puppet exchange of ssh hostkeys which has been broken for a while
14:30 mark: Truncated all tables in the puppet db on db9
12:30 mark: Implemented domain search list in resolv.conf for fenari (use sparingly!)
12:24 mark: Put /etc/resolv.conf under Puppet management on all servers. Setting timeout option to 3s, to avoid PyBal depools due to 5s timeout when the primary resolver is down.
11:38 mark: Removed rkhunter and chkrootkit on bayes. What is the point with 2 year old software? Just creating more cron spam? :)
10:06 mark: Fixed degraded raid on nescio
09:53 mark: Removed X.org and gdm from bayes. Why was it installed?
09:30 domas: doing maintenance on db13 and db15 (BIOS, OS, MySQL upgrades, resync of data)

July 16

21:05 mark: Removed old-style DNS monitoring from Nagios conf.php, now fully Puppet managed
21:03 mark: Deployed authoritative DNS on nescio.esams, and moved the service IP
19:32 mark: Deployed PowerDNS recursor on nescio, and moved the 91.198.174.6 service ip to it
19:16 mark: Installed lucid on new server nescio.esams
13:49 mark: Fixed entry in upload.wikimedia.org georecord, sent to text squids for a few ip ranges
12:27 mark: Pointed country code 'eu' (127.0.255.1 or 65281) to esams in geodns; since geobackend uses a signed short, I had to mask it to 0x7ff / 32513 in the director map
10:49 mark: Shutdown fuchsia for decommissioning
07:57 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 69416
07:56 RoanKattouw: Running scap to deploy r69416
01:48 logmsgbot: tstarling synchronized php-1.5/includes/diff/DifferenceInterface.php 'r69414'

July 15

19:23 mark: Replaced all occurrences of 'rr.wikimedia.org' with 'text.wikimedia.org' in DNS
19:14 mark: Updated IP of deprecated record rr.esams.wikimedia.org
19:10 mark: Started PyBal on amslvs1 with a new config; it automatically picked up the traffic for both text.esams (91.198.174.232) and bits.esams (91.198.174.233)
19:07 mark: Stopped PyBal on amslvs1, BGP and OSPF did an automatic failover of bits.esams (91.198.174.233) to amslvs3
18:59 mark: Removed IP 91.198.174.2 (old text squids service ip) from amslvs1. Anyone still using the old IP after weeks will now be unable to reach our sites.
18:56 mark: Depooled knsq1-knsq7 in PyBal
17:38 Fred: fixed nfs mounts on Bayes.
15:35 apergos: chowned /mnt/upload6/private/ExtensionDistributor/mw-snapshot/trunk/extensions tree to extdist. ExtensionDistributor apparently working now
15:01 apergos: running svn cleanup on /mnt/upload6/private/ExtensionDistributor/mw-snapshot/trunk/extensions as extdist user
12:34 logmsgbot: tstarling synchronizing Wikimedia installation... Revision: 69381
12:18 Tim: svn up/scap to r69380
05:13 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24321 - ml.wikiquote.org lost its project namespace'

July 14

23:44 Fred: re-added ccron job to periodically save rrds on our ganglia server. (cron job seems to have vanished for some reason)
17:59 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Favicon for wikimaniateamwiki per Guillaume'
16:06 Fred: restarted apache on mobile1 (had begun to return 500)
14:07 mark: Fixed memcached on srv110
12:19 mark: Fixed ganglia and puppet on stafford
11:54 mark: Migrated DNS monitoring to puppet
10:31 mark: Migrated ZFS RAID nagios check to puppet
10:14 mark: Migrated monitoring of lucene to puppet
09:37 mark: Migrated monitoring of image scalers to puppet
08:49 Tim: using stafford for some pbuilder experimentation

July 13

22:02 mark: Migrated monitoring of application servers to Puppet
20:29 mark: Fixed puppet on ms4
20:16 mark: Hacked up nagios conf.php to not create host entries for most servers (now in puppet), except special cases
19:58 mark: Hacked up nagios conf.php to not create host entries
16:51 mark: Migrated Squid Nagios monitoring to puppet, commented some functionality in nagios conf.php
15:51 mark: Split puppet nagios config over multiple files

July 12

16:54 Fred: changed LONGQUERIES check threshold
16:08 Fred: restarting morebots since it had died.
16:08 Fred: restarting Nagios since it was down.
14:29 mark: Added "cfg_file=/etc/nagios/puppet_hosts.cfg" to nagios.cfg
13:25 JeLuF: added disk space monitoring for apaches
12:51 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24306 - Create namespaces for Lithuanian Wiktionary'
12:48 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24321 - ml.wikiquote.org lost its project namespace'
12:46 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24321 - ml.wikiquote.org lost its project namespace'
12:41 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24344 - Namespace changes - si.wiktionary'
11:45 JeLuF: fixed broken ganglia-metrics installation on srv146 (chown gmetric /var/log/gmetricd/gmetricd.log)
11:41 JeLuF: added DPKG status monitoring for all app servers to nagios. Reports all packages that are not in state 'rc' or 'ii'.
10:43 JeLuF: lots of false alerts from nagios due to missing SSL setup for NRPE. Working on it.
09:53 JeLuF: changed puppet config to install nrpe on all app servers
09:28 JeLuF: replacing opsview-nrpe agents by nagios-nrpe agents (image_scalers, some other apaches). Most apaches already use nagios-nrpe
07:40 Tim: set up NRPE disk space monitoring on ms4, discovered that /mnt2 is full
04:54 Tim: updated NFS host/service groups to monitor the actual NFS servers, not a random collection of miscellaneous ex-NFS servers
04:46 Tim: installed NRPE on nfs1 and nfs2
04:08 Tim: adding rendering, m, bits.esams, recursor0, recursor1, recursor0.esams to nagios
04:02 Tim: added forward DNS entry for recursor0.esams, modified reverse DNS entry resolver0.esams -> recursor0.esams
03:55 Tim: fixed reverse DNS entries for recursor0 and recursor1, were set incorrectly to non-existent hostnames "resolver0" and "recursor1"
03:36 Tim: renamed db6.mgmt to locke.mgmt

July 10

14:14 rainman-sr: search7 disk was full, deleting some old unneccessary indexes
12:50 Fred: applied security updates on all machine running Karmic or Lucid (per USN-959-1)

July 9

18:07 domas: forgot to log, rebooted locke, put startup stuff to rc.local, maybe Tim changed it afterwards, hehe. beer is good too.
15:31 Rob: wikimania2011wiki is now using vector
15:31 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php
12:48 logmsgbot: robh ran sync-common-all
01:06 logmsgbot: tstarling synchronized php-1.5/includes/filerepo/RepoGroup.php
01:04 logmsgbot: tstarling synchronized php-1.5/includes/filerepo/RepoGroup.php
01:04 logmsgbot: root synchronized php-1.5/includes/filerepo/RepoGroup.php
01:03 logmsgbot: tstarling synchronized php-1.5/includes/filerepo/RepoGroup.php
00:59 logmsgbot: tstarling synchronized php-1.5/includes/filerepo/RepoGroup.php

July 8

22:27 apergos: powercycled db9 fromm drac after shutdown failed
22:20 Fred: re-imaging srv225 back to normal until wikimedia-task*can be ported to lucid.
22:15 apergos: rebooting db9, mysqld was defunct but the port was in use so couldn't restart it the nice way
17:06 mark: Set temporary 91.198.174.0/24 null0 route on br1-knams, to investigate prefix announcement problems
16:10 Rob: updated puppet to add zak to the mortals admin group and allowed access to shell on fenari as non-root
04:10 Tim: starting upload of BnF images, using importImages.php in screen on fenari

July 7

21:38 Fred: RIP sfoservices. (box not booting at all anymore)
17:10 Fred: re-imaging srv225 to the apache cluster.
16:26 mark: Fixed puppet on srv193
15:56 mark: Fixed horrible gmond mess on searchidx1
15:42 mark: Fixed puppet on sr255
15:30 mark: Mounted /mnt/upload6 on srv255
14:23 mark: Fixed /home backup on nfs1/nfs2 to tridge
07:57 Rob: srv193 is refusing to take my updates, removed it from pybal so it doesnt serve out of data information
07:53 logmsgbot: robh ran sync-common-all
07:50 Rob: updated dns for wikimania wiki
07:38 Rob: adding wikimaniawiki apache support, sycning lots of apaches and docroots.
01:49 Tim: downloading the DjVu files via rsync/ssh for http://www.wikimedia.fr/wikim%C3%A9dia-france-signe-un-partenariat-avec-la-bnf

July 6

13:43 mark: Fixed puppet on nfs1 and nfs2
11:11 mark: Removed config cache on srv110
10:32 mark: Fixed puppet on srv110
10:26 mark: Stopped apache on srv110
00:28 Tim: restarted mailman on lily
00:23 Tim: killed all mailman processes on lily in an attempt to save it from swap death (swapping severely since 00:07)
00:12 Tim: fixed stale /home on searchidx1 and restarted indexer
00:02 Tim: codereview-proxy is up now. Pinging CR update API for all recent revisions

July 5

23:55 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'new URL for codereview proxy'
23:33 Tim: changed CNAME for codereview-proxy to kaulen
23:20 Tim: moving codereview-proxy to kaulen to replace isidore (which is down)
22:53 Tim: on srv124: remounted /home to fix test.wikipedia.org
18:56 logmsgbot: jeluf synchronized php-1.5/wmf-config/flaggedrevs.php '24010 - id.wikipedia requesting FlaggedRevs'
17:01 mark: Did BGP soft clear outbound on all AMS-IX sessions; no prefixes were being announced as of two weeks ago
16:01 mark: Made puppet ensure apache is running on the app servers; running "sync-common" upon start
15:38 mark: Fixed puppet on srv145
11:39 mark: Remounted /home on hume
11:30 logmsgbot: root synchronizing Wikimedia installation... Revision: 68850
11:29 logmsgbot: mark synchronized php-1.5/wmf-config/CommonSettings.php 'CommonSettings.php out of sync on a few apaches'
09:17 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24264 - Create a namespace aliases on zhwiki'
06:26 ronabop: Kudos to the team who rebuilt a multi-hundred node system under extreme pressure.

Was there extreme pressure? :-) --domas

06:23 Tim: fixed broken ircd auth configuration, irc.wikimedia.org now working again
05:13 Tim: on browne: reinstalled udprec to fix IRC server
04:16 Tim: switched nagios monitoring for search to less flappy TCP connection check instead of HTTP
03:58 domas: s3 pos: db17-bin.321:0
03:58 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 's3 rw'
03:54 Tim: fixed search monitoring in nagios
03:47 Tim: started lsearchd on search1-20
03:44 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'rw s4 s4'
03:42 Tim: fixed search1: just needed /home remounted
03:38 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
03:32 domas: new repl positions, s2: db30-bin.000015:1227, s4: db16-bin.019:0
03:17 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
03:13 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
03:04 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 's3 fake master r/o'
02:54 Tim: mysql status: s2 and s4 have replication broken with "Client requested master to start replication from impossible position". s3: still waiting for innodb recovery on master. Other clusters good.
02:49 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php
02:48 Tim: on db8: read_only=1 again and setting wiki to r/o
02:46 Tim: on db8: read_only=0, started up r/o (s4)
02:42 Tim: putting s2 into read-only mode due to replication issues
02:35 RobH_: search server defaults to sitting on grub screen for search13-search20, will fix later, for now they are booting back up.
02:30 Tim: fixed m.wikipedia.org on lvs4
02:26 RobH_: search13 back up, working on the others
02:24 mark: Moved bits.pmtpa to point to Text squids in DNS
02:08 Tim: starting mysqld on a lot of DB servers
02:08 RobH_: seems like a power outage, not an AC issue.
02:07 RobH_: email back online
01:52 Tim: on nfs1: river fixed the filesystem with fsck
01:33 Tim: (about 5 minutes ago) started mysqld on db17
01:26 Tim: on lvs4: removed dead squids from text list
01:16 Tim: started mysql on db8
01:15 Tim: started mysqld on db5
01:11 Tim: power went off briefly again, lvs4 came back up properly this time, starting pybal on it again
00:59 Tim: got LVS set up and working on lvs4
00:56 Tim: s/nfs4/lvs4
00:55 Tim: got nfs4 back online

July 3

01:25 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
00:40 logmsgbot: midom synchronized php-1.5/wmf-config/db.php

July 2

22:17 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'style version'
22:14 logmsgbot: andrew synchronizing Wikimedia installation... Revision: 68850
21:47 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24119 - Change the logo image in Sinhala wiktionary'
21:46 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24119 - Change the logo image in Sinhala wiktionary'
21:42 logmsgbot: andrew synchronized php-1.5/skins/common/shared.css
21:19 logmsgbot: andrew synchronized php-1.5/wmf-config/ExtensionMessages.php
21:17 logmsgbot: andrew synchronizing Wikimedia installation... Revision: 68850
20:27 Fred: added replag nagios check for all slave DBs.
19:56 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24025 - Enable NewUserMessage extension on ko.wikipedia'
19:50 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24067 - New namespace for gl.wiktionary'
19:40 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24048 - Change sitename for Osetian Wikipedia.'
19:28 mark: Slave SQL thread stopped on db40 due to lock wait timeout (?), restarted
17:58 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ParserFunctions.php
17:47 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24157 - Enable RevisionMove on testwiki'
11:30 JeLuF: mark: !log Upgraded asw-b3-sdtpa, asw-b4-sdtpa and asw-b5-sdtpa to newer JunOS
00:54 Fred: starting MTA on list server again...
00:51 Fred: Cause of spam: spammers using wiki@wikimedia.org as originating address. result: we are getting hit by the responses.
00:46 Fred: purging mail queue for spam 'replies' on list server.
00:21 Fred: mailing list server down while fixing .

July 1

22:58 tomaszf: setting watchdog timer on db9 to 60sec for process kill
21:08 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '17531 - Add subpage feature to the article namespace on nowikimedia'
21:05 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '17510 - Give patroller-group access to suppressredirect on nowiki'
20:57 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24065 - Add some interwiki links in Special'
20:55 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24018 - Change the logo image in Sinhala wikibooks'
20:53 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24138 - Logo of the fr.wikisource'
20:49 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24205 - New namespace for de.arbcom'
20:28 logmsgbot: catrope synchronized php-1.5/extensions/CodeReview/backend/CodeRevision.php 'r68850'
20:27 logmsgbot: catrope synchronized php-1.5/extensions/CodeReview/CodeReview.php 'r68850'
18:11 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ParserFunctions.php
18:09 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
18:09 Rob: every single access switch is now accessible via serial mgmt
18:08 Rob: only switch in pmtpa-row a is asw-a2-pmtpa, which was not responsive to serial, fixed scs settings and properly labeled the port
18:07 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r68842'
18:06 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js 'r68842'
17:35 Rob: pmtpa-rowc switches connected to scs-c1-pmtpa
17:03 Rob: asw-b1,asw-b2,asw-b3,asw-b4,asw-b5-pmtpa connected to scs-c1-pmtpa
16:30 mark: Rob moved management fiber on pmtpa end from csw5-pmtpa:8/24 to msw1-pmtpa:0/1/1
16:05 Rob: moved the primary serial mgmt interface of csw5-pmtpa from scs-ext to scs-c1-pmtpa.mgmt.pmtpa.wmnet, Also ran the permanent connections to the same serial console for msw1-pmtpa and mr1
14:04 mark: Configured Exim to bypass spamd for wiki@wikimedia.org recipient
13:51 mark: Restarted exim4 and spamassassin on mchenry
07:51 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Second attempt at restoring thumb size on svwiki'
07:48 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Change thumb size on svwiki back to 250px'
04:49 Tim: on singer: configured "php_admin_flag engine off" in all planet vhosts
03:24 Tim: added a user account for myself on filesrv1, in tech group, I figure I've been here long enough to deserve one

2000s

Archive 1: 2004 Jun - 2004 Sep
Archive 2: 2004 Oct - 2004 Nov
Archive 3: 2004 Dec - 2005 Mar
Archive 4: 2005 Apr - 2005 Jul
Archive 5: 2005 Aug - 2005 Oct, with revision history 2004-06-23 to 2005-11-25
Archive 6: 2005 Nov - 2006 Feb
Archive 7: 2006 Mar - 2006 Jun
Archive 8: 2006 Jul - 2006 Sep
Archive 9: 2006 Oct - 2007 Jan, with revision history 2005-11-25 to 2007-02-21
Archive 10: 2007 Feb - 2007 Jun
Archive 11: 2007 Jul - 2007 Dec
Archive 12: 2008 Jan - 2008 Jul
Archive 12a: 2008 Aug
Archive 12b: 2008 Sept
Archive 13: 2008 Oct - 2009 Jun
Archive 14: 2009 Jun - 2009 Dec

2010s

2020s