Server Admin Log/Archive 17

This page archive server admin logs from November 1st 2010 till December 31st 2010

December 31

21:55 tomaszf: kaldari aborted alter tables on db9 due to high i/o wait
20:37 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.hooks.php 'r79366'
20:01 Ryan_Lane: freeing up space on db17 by clearing out an old error log
17:27 apergos: copying snapshot1:/data (which appears to be static html dumps of en and pl wiki) to tridge:/data/snapshot1, rsync running from tridge
17:13 logmsgbot: catrope synchronized php-1.5/includes/specials/SpecialLinkSearch.php 'r79354'

December 30

23:36 tomaszf: pushed r79304 to payments cluster .. updating tshirt images
22:40 tomaszf: pushed r79297 to payments cluster .. adding wordmarks
20:36 Ryan_Lane: powercycling mobile2 - it's dead
19:44 apergos: add xml data fs to nfs in puppet, include that in snapshot hosts stanzas
19:43 apergos: remove andrew from snapshot stanzas in puppet, he should already be convered by home-no-service
19:33 awjr: pushed r79287 to payments cluster, replacing bad payflowpro_gateway.i18n.php with the good
19:05 awjr: pushed r79282 of DonationInterface to payments cluster, updating credit card validation on cc form, picking up i18n changes and new css for premiums
18:49 logmsgbot: catrope synchronized docroot/mediawiki/xml/export-0.4.xsd 'Update XSD file for XML dumps'
15:59 RobH: all blogs updated successfully, all plugins updated and reactivated
15:48 RobH: created backups of both blogs, pushing updates to new versions for security reasons
11:27 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
11:25 logmsgbot: catrope synchronized php-1.5/skins/common/shared.css 'r79244'
06:40 Tim: fixed my VOIP account

December 29

21:01 richcole: drive15 from IBM db42
19:22 Ryan_Lane: granted sudo privileges for demon on formey to create/modify/delete svn (LDAP) users
02:03 tomaszf: updating payments cluster to r79148

December 28

21:54 Ryan_Lane: destroying 1TB logical volume on storage3
21:54 Ryan_Lane: umounting 1TB logical volume from /archive on storage3 and mounting 4TB xfs volume (data was moved prior to this)
19:09 Ryan_Lane: moving archive data from /archive to /archive1 on storage3, will replace mount afterwards
19:08 Ryan_Lane: creating new 4TB logical volume "archive1" on storage3, and putting an xfs filesystem on it
16:34 mark: Defined special disk space check (with SMS notification) for mysql core databases
15:52 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'adding db11, db18 and db39'
14:30 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'enabling db37'
13:16 domas: resumed replication on db11, db18, db37 and db39
00:19 Ryan_Lane: restarting apache on srv217 - puppet updated its apache2.conf file, but didn't restart it
00:12 Ryan_Lane: powercycling amssq53 - it's dead
00:09 Ryan_Lane: powercycling sq76 - it's dead
00:08 Ryan_Lane: knsq14 isn't coming back up - may be broken
00:08 Ryan_Lane: powercycling knsq24 - it's dead
00:02 Ryan_Lane: powercycling knsq14 - it's dead

December 27

23:54 Ryan_Lane: starting apache on srv217, running puppet to ensure code is synched
23:52 Ryan_Lane: restarting apache on srv231
23:50 logmsgbot: laner synchronized php-1.5/wmf-config/mc.php 'Fixing IP address for srv226'
23:48 logmsgbot: laner synchronized php-1.5/wmf-config/mc.php 'Fixing IP address for srv227'
23:39 Ryan_Lane: powercycling srv217 - it's dead
20:38 Ryan_Lane: adding puppet schema to opendj on nova-controller.tesla
19:51 Ryan_Lane: specifically chgrp'd all files to svnadm and added write permissions for the group
19:51 Ryan_Lane: opened up file permissions on /srv/org/wikimedia/svn on formey so that ^demon can edit files
19:48 Ryan_Lane: disabled selenium service on windows7-1, and launched selenium services manually, while logged in as the selenium user
18:04 Ryan_Lane: installing puppet, puppet-el, puppetmaster, puppetmaster-passenger, and vim-puppet on nova-controller.tesla
17:56 Ryan_Lane: adding new 1TB "archive" logical volume on storage3, formatting as ext3, and mounting at /archive
17:49 Ryan_Lane: purging some bin logs on db9 to free up space
16:27 Ryan_Lane: installing graphviz on formey via puppet for bug 26404
09:21 apergos: truncated log-all and log-index in /a/search/log on searchidx1 to get some space back

December 26

18:57 Ryan_Lane: powercycling mobile1, as it is dead
12:49 apergos: restarted apache on srv227, it was seemingly the source of the DOM related errors, many msgs like "PHP Fatal error: Cannot access property started with '\0' in /usr/local/apache/common-local/wmf-deployment/includes/parser/Preprocessor_DOM.php on line 1393" in /var/log/messages
02:29 Ryan_Lane: running /home/wikipedia/common/wmf-deployment/maintenance/purgeStaleMemcachedFlaggedRevs.php on s3/s7 flaggedrevs enabled wikis
00:08 logmsgbot: ariel synchronized wmf-deployment/wmf-config/db.php 'edits enabled for s3/s7, people should purge if they see weird things, flagged revs may still cause problems, hewiki still has issues'

December 25

23:39 apergos: ran Platonides' script "purgeStaleMemcachedText.php" on s3/s7 projects, see /home/wikipedia/common/wmf-deployment/maintenance/purgeStaleMemcachedText.php on fenari (not added to the local branch yet)
21:14 logmsgbot: ariel synchronized wmf-deployment/wmf-config/db.php 'read-only for s3/s7 while we sort out edit sync and cache issues'
20:17 mark: New master bin-log db17-bin.003, position 406
20:16 mark: Switched master on s3, s7 from db27 to db17, just one slave db25
20:16 logmsgbot: mark synchronized php-1.5/wmf-config/db.php 'Going read-write on s3, s7'
20:13 logmsgbot: mark synchronized php-1.5/wmf-config/db.php 'Switch to new master db17, slave db25 for s3/s7'
20:03 logmsgbot: mark synchronized php-1.5/wmf-config/db.php
20:02 logmsgbot: mark synchronized php-1.5/wmf-config/db.php 'Remove db39 from rotation'
19:58 mark: Switching master to db17
19:58 logmsgbot: mark synchronized php-1.5/wmf-config/db.php
19:43 Ryan_Lane: purging old bin logs on db27
14:32 apergos: turned off Indexes option on wikitech in apache config files, just like on the cluster
12:17 apergos: powercycling searchidx1, can't get in through management console, nonresponsive to ssh, google says the weird ilom messages are a firmware issue, so... reset.
11:11 rainman-sr: searchidx1 is down for about 5 days now, can someone take a look at it? Cannot ssh to it. Also search group dissapeared from ganglia.
00:22 mark: Running apt-get dist-upgrade on mobile3
00:20 mark: Rebalanced LVS weights of mobile servers on lvs4 from 20/50/30 to 30/50/50
00:06 mark: powercycled mobile3

December 24

18:18 apergos: changed cname for dumps.wikimedia.org to point to dataset2 (so all related aliases will also point there)
04:47 Ryan_Lane: adding 512MB ram to nova-controller.tesla; shutting down VM to do so

December 23

18:23 apergos: added export of /data for download host to class in puppet
17:57 Ryan_Lane: ignore that last message
17:57 Cobi: TODO: Give Cobi root on servers
17:54 Ryan_Lane: added vivek and aditya to nova-controller.tesla, nova-compute1, nova-compute2, and added them to sudoers
17:46 apergos: added misc::download class for dataset2 in puppet (handles lighttpd for download.wm)
14:02 hcatlin: deploying code update to m.
10:42 logmsgbot: tfinc synchronized php-1.5/wmf-config/reporting-setup.php 'Flipping back stats page to db10 since weve cleared up our space issues'
01:39 apergos: dataset2 racked and installed (thanks rich, robh, mark). now copying data back off tridge, rsync in screen session from tridge as root.
00:40 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionTracking/ContributionTracking_body.php
00:40 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionTracking/ContributionTracking.i18n.php
00:23 tomaszf_: updating IPN listener on grosley to r83

December 22

23:58 tomaszf: updating civicrm thank_you module to pick up email fix
23:35 nelson__: snow
23:28 tomaszf: updating thank_you and queue2civicrm module on grosley
23:28 Ryan_Lane: started profile-collector on spence
23:21 logmsgbot: tfinc synchronized php-1.5/extensions/LandingCheck/SpecialLandingCheck.php
22:05 logmsgbot: demon synchronizing Wikimedia installation... Revision: 78853: Deploy various FR/PC changes, r78852
21:30 RobH: udpated dns with dataset2 info
21:24 tomaszf: fixing perms on grosley:/srv/org.wikimedia.civicrm/sites/all/modules to point to www-data instead of awjrichards
16:30 Ryan_Lane: apache-graceful-all was me ;)
16:23 Ryan_Lane: reloading apache on srv*, incrementally
15:48 logmsgbot: catrope synchronized php-1.5/extensions/ContributionReporting/FundraiserStatistics_body.php 'r78799'
15:48 logmsgbot: catrope synchronized php-1.5/extensions/ContributionReporting/ContributionReporting.php 'r78799'
11:22 Tim: added Roan to security@wikimedia.org alias list

December 21

21:42 RobH: db41 online and slated for owa project
21:17 tomaszf: dropping old faulkner db on db9
01:02 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 26357 - Flood flag for foundationwiki'
01:01 tomaszf: moving over hume banner impression logs : 2010-12-08 -> 2010-12-13 to storage3 : /archive/udplogs
00:49 tomaszf: moving over hume banner impression old logs : 2010-12-04 -> 2010-12-07 to storage3 : /archive/udplogs
00:49 Ryan_Lane: changing expiration times of text/javascript and application/x-javascript from A2592000 to A345600 in apache config as a short-term fix for bannerlist loading
00:24 tomaszf_: starting copy from hume -> storage3 to free up space
00:20 logmsgbot: catrope synchronized php-1.5/languages/Names.php 'r78543'

December 20

21:28 RobH: bayes is back online, updated
18:43 Ryan_Lane: configuring pdns on nova-controller.tesla to use strict mode for LDAP
13:16 logmsgbot: tstarling synchronized wmf-deployment/cache/trusted-xff.cdb
13:14 Tim: updating TrustedXFF
12:14 apergos: one hour later, 6.62GB has been copied of the XML dumps, out of 1T. Let's see how long it takes to get the whole thing over.
11:13 apergos: started copy to google storage of the last complete xml dumps for each project (note that "complete" may include "completed with failures"), from dataset1, in root screen session

December 19

18:58 mark: Powercycled amssq42, 48, 57, 58
18:22 mark: Fixed puppet on ekrem
16:20 mark: Fixed ganglia on spence again
14:49 mark: Fixed Ganglia on Spence
14:21 mark: Fixed puppet payments3 exported resource override issue
14:02 mark: Rebooting spence
13:44 mark: Starting dist upgrade from hardy to lucid on spence
13:43 mark: Stopping nagios on spence
13:42 mark: Removed mysql-server from spence
13:29 mark: Disabled Merlin Nagios module
13:11 mark: Rebooting spence
13:02 mark: Running apt-get dist-upgrade on spence
12:57 mark: Power cycled spence

December 18

23:11 RobH: srv187-srv189 relocated and powering up
22:27 logmsgbot: robh synchronized php-1.5/wmf-config/db.php 'srv181-srv186 in'
22:26 mark: Updated smokeping with new rack layouts and contents
22:21 RobH: srv181-srv186 relocated and coming back online
22:21 RobH: srv187-srv189 coming down for relocation
22:00 logmsgbot: robh synchronized php-1.5/wmf-config/db.php 'srv153 srv154 in'
21:58 RobH: srv153, srv154 powering back up
21:53 RobH: shutting down srv153, srv154 power redistribution
21:51 logmsgbot: robh synchronized php-1.5/wmf-config/db.php 'srv154-srv153 out'
21:28 RobH: srv181-srv186 going offline for relocation
21:27 logmsgbot: robh synchronized php-1.5/wmf-config/db.php 'srv181-srv186 pulled when applicable'
21:23 logmsgbot: hashar synchronized php-1.5/wmf-config/abusefilter.php 'resyncing abusefilter.php for the offline apaches'
21:20 logmsgbot: robh synchronized php-1.5/wmf-config/db.php
21:15 RobH: srv157 accidental reboot, opps =P
21:13 logmsgbot: robh synchronized php-1.5/wmf-config/db.php 'commented out srv156, its coming down for power rebalancing'
21:11 logmsgbot: robh synchronized php-1.5/wmf-config/db.php 'srv178-srv180 back in'
21:05 RobH: srv175-srv180 relocated to new rack and coming online
20:53 logmsgbot: robh synchronized php-1.5/wmf-config/db.php 'srv177 back in'
20:42 logmsgbot: robh synchronized php-1.5/wmf-config/db.php 'srv175 srv176 back in'
20:06 logmsgbot: hashar synchronized php-1.5/wmf-config/abusefilter.php 'bug 26364: abuse filter privacy issue on hiwiki'
20:02 RobH: srv175-srv180 coming down for relocation
20:02 logmsgbot: robh synchronized php-1.5/wmf-config/db.php 'putting ms2 back into ES rotation, removing srv175-srv180'
19:39 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 26364: abuse filter privacy issue on hiwiki'
19:28 RobH: hume back online in new rack
18:38 mark: DRBD failover to nfs2 successful, nfs2 is now primary
18:38 RobH: pdf2 relocated, powering up
18:32 mark: Starting DRBD failover of nfs1 to nfs2
18:20 RobH: nfs2 reracked, powered up, online (replication check delayed until later)
18:19 RobH: pdf2 coming down for relocation, will be back online shortly
17:51 RobH: shutting down nfs2 for relocation
17:42 RobH: tridge moved, powering back up
17:00 RobH: streber & williams moved, powering up
16:43 RobH: williams and streber coming down for relocation (otrs and rt will be offline during this transition)
16:33 RobH: mchenry & sanger moved, powering up
16:09 RobH: shutting down mchenry & sanger to relocate them
16:03 mark: Started slave on ms1
15:59 mark: Renamed asw-b2-pmtpa to asw-d1-sdtpa in DNS
15:30 logmsgbot: robh synchronized php-1.5/wmf-config/db.php 'remvoing ms5 for relocation, will add back again shortly'
15:28 RobH: ms1 is going to power down and be relocated
15:26 mark: Failed over traffic back from amslvs4 to amslvs2
15:23 RobH: ms5 moved and powering up
15:20 mark: Running apt-get dist-upgrade && reboot on amslvs2
15:11 mark: Failed over upload.esams traffic from amslvs2 to amslvs4
15:04 mark: Running apt-get dist-upgrade && reboot on amslvs4
14:51 mark: Running apt-get dist-upgrade && reboot on amslvs3
14:48 mark: Failed over traffic back from amslvs3 to amslvs1
14:46 RobH: ms5 shutting down for relocation
14:32 mark: Running apt-get dist-upgrade && reboot on amslvs1
14:27 mark: Made puppet install kernel 2.6.36 on LVS balancers
14:22 mark: ES clusters 7, 20, 21 copies finished
14:22 mark: ES cluster 8 copy finished
14:07 mark: Imported package "linux-image-2.6.36-1-server" from the kernel PPA into the Wikimedia APT repository for lucid-wikimedia, section universe

December 17

23:38 RobH: srv169 is not racked, has a dead hard disk, and had to use its rail for another server until we drill the stuck rail out of old rack
23:38 logmsgbot: robh synchronized php-1.5/wmf-config/db.php 'pushing srv170-174 back into file'
23:36 RobH: srv170-srv174 racked and powered, mysql running, putting back into db.php, puppet currently running to bring apache online
23:25 apergos: added default values for tick and freq to /etc/default/adjtimex on dataset1 manually (ubuntu install of package is broken and leaves broken conf file, known bug, etc.)
23:12 mark: ES cluster 6 copy done
23:00 RobH: shutting down and moving srv169-srv174
22:59 logmsgbot: robh synchronized php-1.5/wmf-config/db.php 'srv169-srv174 out for relocation'
22:46 mark: Started copy of ES cluster 21 data from srv161 to tridge (screen on tridge)
22:43 mark: Started copy of ES cluster 20 data from srv160 to tridge (screen on tridge)
22:41 mark: ES cluster 10 copy done
22:40 logmsgbot: robh synchronized php-1.5/wmf-config/db.php 'srv157 back in es rotation'
22:39 mark: Started copy of ES cluster 8 data from srv156 to tridge (screen on tridge)
22:38 mark: ES cluster 5 copy done
22:31 mark: Stopped apache & puppet on srv170 to speed up last bit of the copy
22:26 logmsgbot: robh synchronized php-1.5/wmf-config/db.php 'removing srv157 for relocation'
22:25 RobH: shutting down srv157 for relocation
22:25 logmsgbot: robh synchronized php-1.5/wmf-config/db.php 'putting srv163-srv168 back into rotation'
22:22 RobH: srv163-168 relocated, puppet will run automatically to repool apache, mysql is already running, pushing them back into ES cluster
22:18 mark: ES cluster 9 copy done
22:16 apergos: trying new approach to clock drift on dataset1: installed adjtimex, set adjtimex --tick 10853 ( after doing adjtimex --compare), set hwclock -s and restarted ntpd. hwclock seems to be much more accurate than system clock. note changing clock source from tsc to hpet made no difference, so changed that back.
21:59 mark: Stopped puppet and apache on srv157 to speed up copy
21:33 RobH: shutting down srv163-srv168 for relocation
21:32 logmsgbot: robh synchronized php-1.5/wmf-config/db.php 'srv163-srv168 depooled for relocation'
21:28 logmsgbot: robh synchronized php-1.5/wmf-config/db.php 'putting srv158-srv162 back into service'
21:27 RobH: srv158-srv162 back in service, pushing back into db.php
21:26 mark: ES cluster 4 copy finished
21:26 mark: ES cluster 3 copy finished
20:35 RobH: shutting down srv158-srv162, will relocate srv157 after the ES data is backed up to tridge
20:32 logmsgbot: robh synchronized php-1.5/wmf-config/db.php 'pulling srv158-srv162 from ES rotation for relocation'
20:28 logmsgbot: robh synchronized php-1.5/wmf-config/db.php 'srv182 is online pushing it back into es service'
20:28 mark: Started copy of ES cluster 10 data from srv170 to tridge (screen on tridge)
20:23 mark: Started copy of ES cluster 9 data from srv157 to tridge (screen on tridge)
20:16 RobH: working on srv169
20:16 mark: Started copy of ES cluster 7 data from srv155 to tridge (screen on tridge)
20:05 mark: Powercycled sq69
19:45 mark: Started copy of ES cluster 6 data from srv154 to tridge (screen on tridge)
19:37 mark: Started copy of ES cluster 5 data from srv153 to tridge (screen on tridge)
19:37 mark: Started copy of ES cluster 4 data from srv152 to tridge (screen on tridge)
19:28 mark: Started copy of ES cluster 3 data from srv151 to tridge (screen on tridge)
19:18 RobH: srv151-srv156 back in service
19:17 logmsgbot: robh synchronized php-1.5/wmf-config/db.php 'put srv156 back into service'
18:39 logmsgbot: awjrichards synchronized php-1.5/wmf-config/reporting-setup.php 'Updating maximum amount for contribution reporting'
18:31 mark: Setup all new PDUs
18:17 RobH: shutting down srv151-srv156 for relocation
17:36 logmsgbot: robh synchronized php-1.5/wmf-config/mc.php 'moved srv254-srv257 into spares group'
17:33 RobH: srv254-srv257 back in rotation
16:50 mark: Set up PDUs ps1-d2-sdtpa and ps1-d3-sdtpa
16:39 mark: Updated Torrus power monitoring with the new PDUs
16:38 RobH: srv254-srv257 going offline for relocation
16:38 logmsgbot: robh synchronized php-1.5/wmf-config/mc.php 'srv254-srv257 replaced in active rotation for relocation'
16:03 mark: Ignored replication of db bugzilla3 on storage3, repaired bugzilla3 tables, and restarted replication
15:55 mark: Setup DNS for new PDUs
15:16 mark: Updating dns for asw-b1-pmtpa and asw-b4-pmtpa name change
04:08 RobH: srv247-srv253 back in rotation and relocated into d2 from b2 (will update racktables shortly)
03:26 logmsgbot: robh synchronized php-1.5/wmf-config/mc.php 'srv240-srv246 in srv247-srv253 out'
03:21 RobH: srv247-srv253 removing from memcached rotation and shutting down for rack relocation
03:20 RobH: srv240-srv246 online, pushing back into memcached rotation
02:30 logmsgbot: robh synchronized php-1.5/wmf-config/mc.php 'srv233-srv239 in, srv240-srv246 out for relocation'
02:27 RobH: removing srv240-srv246 from mc rotation and shutting down for relocation
02:27 RobH: srv233-srv239 successfully relocated, pushing them into mc rotation
01:18 logmsgbot: robh synchronized php-1.5/wmf-config/mc.php 'srv233-srv239 moved out of memcached rotation'
01:14 RobH: srv233-srv239 being removed from memcached pool and shutdown for rack relocation. srv228-srv232 being put back into memcached
01:06 RobH: srv232 moved and powering up
00:46 Ryan_Lane: rsyncing data from storage3 to db10
00:43 RobH: srv226-srv231 powering up in their new home in d2-sdtpa
00:31 Ryan_Lane: stopping mysql on storage3
00:31 Ryan_Lane: update puppet config for storage3 to update ganglia and nagios
00:25 logmsgbot: tfinc synchronized php-1.5/wmf-config/reporting-setup.php
00:15 Ryan_Lane: copying snapshot data to /a/sqldata (excluding faulkner db)
00:11 Ryan_Lane: deleting databases on db10
00:10 Ryan_Lane: stopping mysql on db10 and restoring data from snapshot
00:06 Ryan_Lane: starting mysql on db10
00:05 Ryan_Lane: deleting faulkner db on db10

December 16

23:48 Ryan_Lane: starting mysql on storage3
23:42 Ryan_Lane: stopping mysql on storage3 to replace the faulkner db
23:33 Ryan_Lane: starting mysql on storage3
23:07 RobH: srv226-srv232 coming down for relocation
23:06 logmsgbot: robh synchronized php-1.5/wmf-config/mc.php 'moved servers around for relocation'
22:38 RobH: srv281 reinstalling
22:24 RobH: srv281 coming down for repair (isnt in memcached active pool)
22:11 Ryan_Lane: rebooting storage3
22:10 Ryan_Lane: restating apparmor on storage3 to reload profiles
21:09 RobH: all power strips in row c and d sdtpa have been wired for serial
20:18 Ryan_Lane: stopping mysql on db10 and copying data to storage3
20:11 Ryan_Lane: installing lvm on storage3
20:11 tomaszf: killing long running civicrm queries on db9
19:59 Ryan_Lane: creating LVM configuration using sda
19:48 Ryan_Lane: reformatting /dev/sda1 as xfs on storage3
19:48 Ryan_Lane: unmounting /data on storage3 and mounting as /a
19:44 CodeBlock: installing puppetmaster and puppet on nova-controller
19:35 Ryan_Lane: installing mysql-client mysql-server mysql-common on storage3

December 15

22:26 Ryan_Lane: restarted slave on db10
22:25 Ryan_Lane: killed mysqlcheck process on db10, as it was eating all available cpu
22:09 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 26222 - Set $wgLogo to local logo on rowiki'
21:55 Ryan_Lane: restarted slave on db10
21:16 Ryan_Lane: restarting mysqld on db10
21:16 Ryan_Lane: enabling long query log on db10
20:43 Ryan_Lane: stoped and started slave on db10
19:39 RobH: pushing dns change to switch dumps to dataset1 from singer (download and all its various incarnations are all pointing at dumps
19:16 RobH: amaranth accidental reboot due to power removal
18:50 RobH: msw-b1-sdtpa power cycled due to power cable being moved within the rack
18:41 RobH: storage3 powering down to have its power distro fixed in rack
18:30 RobH: bringing down db41/db42, the power in rack B1-sdtpa is not balanced properly.
17:52 logmsgbot: demon synchronized php-1.5/extensions/CodeReview/backend/CodeRevision.php 'r78443'
17:45 logmsgbot: demon synchronized php-1.5/extensions/CodeReview/backend/CodeRevision.php 'Debugging pywikipedia failure'
09:48 domas: upgraded stuff on secure to fix ssl warnings
02:37 phuzion: Wrote configs for powerdns with ldap backend and restarted powerdns on nova-controller.tesla.
02:34 Ryan_Lane: restarting opendj on nova-controller.tesla
02:34 Ryan_Lane: adding the dnsdomain2 schema to opendj on nova-controller.tesla
02:13 phuzion: Instaled pdns-server and pdns-backend-ldap on nova-controller.tesla
01:15 logmsgbot: awjrichards synchronized php-1.5/extensions/ContributionReporting/FundraiserStatistics.js 'updating to r78421 so fundraising stats page defaults to 2010 fundraiser'
01:15 logmsgbot: awjrichards synchronized php-1.5/extensions/ContributionReporting/FundraiserStatistics_body.php 'updating to r78421 so fundraising stats page defaults to 2010 fundraiser'
00:09 logmsgbot: awjrichards synchronized php-1.5/extensions/ContributionReporting/FundraiserStatistics.js 'Revering to r64689 due to broken JS'

December 14

23:59 logmsgbot: awjrichards synchronized php-1.5/extensions/ContributionReporting/FundraiserStatistics.js 'updating to r78414 so fundraising stats page defaults to 2010 fundraiser'
22:36 Ryan_Lane: installing python-crypto on nova-controller.tesla
20:09 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Raise $wgAPIMaxResultSize to 12MB temporarily because there are 10MB revs on enwiki even though $wgMaxArticleSize is 2MB'
12:26 Tim: spent the 5 minutes required to get MediaWiki tarball downloads working again from download.wikimedia.org

December 13

23:47 apergos: dataset1 is up, ssh accessible, with the data partition mounted read only, rsync on tridge in a screen session dong rsync as root of /data/xmldatadumps from dataset1 to /data/XMLdumps on tridge.
21:37 Ryan_Lane: restarting pdns on ns1
21:36 Ryan_Lane: changing IP address for payments.wikimedia.org in DNS
20:54 Ryan_Lane: restarting pybal on lvs4
20:33 Ryan_Lane: intalling wikimedia-lvs-realserver on payments3 and payments4
20:31 Ryan_Lane: adding payments configuration to pybal on lvs4
20:11 Ryan_Lane: moving payments4 into public subnet and restarting network services
20:10 Ryan_Lane: moving payments4 into squid vlan
20:09 mark: Failed over text.esams and bits.esams traffic to amslvs3 using an increased BGP metric for amslvs1 routes on csw1-esams
20:03 mark: Configured amslvs3 with option conn_tab_size=20 for module ip_vs
20:01 mark: Rebooting amslvs3 again to test
19:53 mark: Rebooting amslvs3
19:51 mark: Running apt-get upgrade on amslvs3
19:50 mark: Added "kernel-ppa" PPA to amslvs3, installing linux 2.6.36
18:34 Ryan_Lane: renaming payments3 and payments4 in puppet
18:27 Ryan_Lane: restarting network and services on payments3
18:00 Ryan_Lane: moving payments3 and payments4 to a public subnet
17:59 Ryan_Lane: disabling payments3 and payments4 from haproxy config on loudon
12:42 mark: Revived down servers

December 12

22:01 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'removed dead settings'
18:47 apergos: bayes unreachable by mgmt or ping, needs power cycled somehow
18:47 apergos: bringing snapshot1 back up (not intended, but it was sitting there in the bios, I was actually looking for the bayes mgmt console as its current ip for that is nonresponsive... seems like snapshot1 got the old one though.
15:55 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Belated style version bump for last sync'
15:38 logmsgbot: catrope synchronized wmf-deployment/extensions/UploadWizard/resources/combined.min.js 'r78249'
14:26 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'Reorganized autopromote settings to be more sane'
05:52 Ryan_Lane: installing php5-uuid on nova-controller.tesla

December 10

23:29 Ryan_Lane: adding generic::geoip to owa*
20:24 RobH: updating dns with emery entry
20:20 RobH: snapshot1 powering back up, emery being pulled for temp storage use
18:54 RobH: snapshot1 going offline to add capacity
13:40 mark: Deployed new version exim 4.69-2ubuntu0.1wm2 with a str_format vuln fixed from exim 4.70 (lily, mchenry, sanger, grosley, williams)
13:40 mark: Ran apt-get upgrade on sockpuppet
13:40 mark: Ran apt-get upgrade on lily
05:57 Ryan_Lane: cleaning /tmp on pdf1-3
05:45 Ryan_Lane: rebooting pdf3
05:38 Ryan_Lane: running mw-serve-ctl in cleanup mode on pdf3
02:10 Ryan_Lane: installing ldap scripts into /usr/local/sbin on nova-controller.tesla
02:03 Ryan_Lane: installing ldap-utils on nova-controller.tesla
01:06 Ryan_Lane: installing php5-curl on nova-controller.tesla
00:27 Ryan_Lane: installing php5-memcache on nova-controller.tesla
00:27 Ryan_Lane: installing php-apc and memcached on nova-controller.tesla
00:18 mark: Deployed slightly modified exim4-daemon-heavy 4.69-2ubuntu0.1wm1 package on mchenry, sanger, lily, grosley

December 9

21:51 logmsgbot: awjrichards synchronized php-1.5/extensions/ContributionTracking/ContributionTracking_body.php 'Updating ContributionTracking to have alternate item names for recurring and one-time paypal donations; Also now using i18n messages for item-name text'
21:50 logmsgbot: awjrichards synchronized php-1.5/extensions/ContributionTracking/ContributionTracking.i18n.php 'Updating i18n file to allow for translatable item-name for paypal payments'
21:10 logmsgbot: awjrichards synchronized php-1.5/wmf-config/CommonSettings.php 'Re-updating common settings to use new configuration style and to play nicely with new contribution-tracking-setup.php'
21:09 logmsgbot: awjrichards synchronized php-1.5/wmf-config/contribution-tracking-setup.php 'Adding separate contribution-tracking-setup.php configuration file contribution tracking so inclusion of COntributionTracking.php does not override values set in reporting-setup.php'
21:02 logmsgbot: awjrichards synchronized php-1.5/wmf-config/CommonSettings.php 'Fixing config revert due to misspelled extension path'
21:01 logmsgbot: awjrichards synchronized php-1.5/wmf-config/CommonSettings.php 'Rolling back config changes do to config problems - looking for database in wrong place'
20:56 logmsgbot: awjrichards synchronized php-1.5/wmf-config/CommonSettings.php 'Updating ContributionTracking configuration to use style and also updating config vars for ContributionTracking to use new recurring payments IPN listener'
20:55 logmsgbot: awjrichards synchronized php-1.5/wmf-config/InitialiseSettings.php 'Updating ContributionTracking configuration to use style and also updating config vars for ContributionTracking to use new recurring payments IPN listener'
20:48 RobH: updating dns for storage3
20:47 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'removed unused config var'
20:37 richcole: rebooting dataset1 to run scripts
20:36 logmsgbot: awjrichards synchronized php-1.5/extensions/ContributionTracking/ContributionTracking_body.php 'Updating ContributionTracking extension to be able to handle recurring donations via PayPal'
20:36 logmsgbot: awjrichards synchronized php-1.5/extensions/ContributionTracking/ContributionTracking.php 'Updating ContributionTracking extension to be able to handle recurring donations via PayPal'
20:36 logmsgbot: awjrichards synchronized php-1.5/extensions/ContributionTracking/ContributionTracking_OWA_ref.sql 'Updating ContributionTracking extension to be able to handle recurring donations via PayPal'
20:35 logmsgbot: awjrichards synchronized php-1.5/extensions/ContributionTracking/patch-owa.sql 'Updating ContributionTracking extension to be able to handle recurring donations via PayPal'
20:35 logmsgbot: awjrichards synchronized php-1.5/extensions/ContributionTracking/ContributionTracking.sql 'Updating ContributionTracking extension to be able to handle recurring donations via PayPal'
20:35 logmsgbot: awjrichards synchronized php-1.5/extensions/ContributionTracking/ContributionTracking.pg.sql 'Updating ContributionTracking extension to be able to handle recurring donations via PayPal'
20:34 logmsgbot: awjrichards synchronized php-1.5/extensions/ContributionTracking/ContributionTracking.i18n.php 'Updating ContributionTracking extension to be able to handle recurring donations via PayPal'
20:34 logmsgbot: awjrichards synchronized php-1.5/extensions/ContributionTracking/ContributionTracking.alias.php 'Updating ContributionTracking extension to be able to handle recurring donations via PayPal'
20:17 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'cleaned up tag config to use newer format'
19:48 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'removed unused config var'
19:33 Ryan_Lane: powercycling srv281
19:31 Ryan_Lane: powercycling srv266
19:28 Ryan_Lane: powercycling sq64
19:25 Ryan_Lane: powercycling sq49
04:02 Tim: reset list admin password for cu-ombuds-l
01:17 RobH: storage3 online to pull off old data
01:05 RobH: dns updated for storage3 mgmt

December 8

23:31 Ryan_Lane: Adding additional directory and alias for owa resources via puppet on owa*
22:05 richcole: moving storage 3 from PMtpa to SDtpa row B rack 1
22:04 hcatlin: deploying code update to m.wiki
22:04 Ryan_Lane: changing the * cert on owa* to the more current one
21:37 Ryan_Lane: adding a virtualhost for https://owa.wikimedia.org on owa*
19:48 Ryan_Lane: adding *.wikimedia.org certificate and key to owa* nodes
19:18 Ryan_Lane: changing DNS ttl for payments.wikimedia.org
07:50 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevision.php 'deployed r78053'
06:01 logmsgbot: tstarling synchronized php-1.5/includes/DjVuImage.php 'r78047'

December 7

22:03 Ryan_Lane: upgrading imagemagick on pdf1, pdf2, and pdf3
21:37 Ryan_Lane: restarting puppet on srv187 and srv177
21:35 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Resync because Arthur's sync didn't get picked up on all servers'
21:23 logmsgbot: awjrichards synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disabling VariablePage extension on all wikis'
19:48 Ryan_Lane: moving awjrichards from restricted to mortals in admins.pp
18:56 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaLicenseTexts.i18n.php 'r77997'
01:08 logmsgbot: tstarling synchronized wmf-deployment/cache/trusted-xff.cdb

December 6

22:40 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'removed redundant config var'
20:57 mark: Restarted pybal on amslvs3; moved traffic back to amslvs1
20:48 logmsgbot: catrope synchronized php-1.5/extensions/CodeReview/CodeReview.i18n.php 'r77904'
20:21 logmsgbot: catrope synchronized php-1.5/extensions/ExtensionDistributor/ExtensionDistributor_body.php 'r77901'
20:18 mark: Restarting varnish on knsq1-5
20:17 RobH: rebooting sq68
20:17 RobH: sq68 unresponsive to console and ssh
20:11 mark: Moved bits.esams traffic from amslvs1 to amslvs3 (on csw1-esams)
19:13 logmsgbot: robh synchronized php-1.5/wmf-config/mc.php
19:10 logmsgbot: robh synchronized php-1.5/wmf-config/mc.php 'srv266 down'
14:12 domas: master switch on s2 to New position: db13-bin.000001:106
14:12 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
14:11 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
12:13 mark: Shutdown db4's switchport (csw5-pmtpa:1/8)
11:29 mark: Powercycled srv217

December 5

20:28 logmsgbot: demon synchronized php-1.5/extensions/CodeReview/ui/CodeRepoStatsView.php 'r77827'
16:07 Ryan_Lane: modified /etc/apache2/envvars on mobile3 to look like mobile2
15:54 Ryan_Lane: restarting apache on mobile3
15:00 hcatlin: Deploying mobile update to fix change in Palm Pre useragent
10:53 logmsgbot: catrope synchronized php-1.5/extensions/CodeReview/ui/CodeRevisionListView.php 'r77792'
00:57 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
00:39 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'awwwww crap'

December 4

23:43 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FRDependencyUpdate.php 'deployed r77749'
22:43 logmsgbot: aaron synchronized php-1.5/includes/Article.php 'Deployed r77747'
22:22 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'taking out db11 and db17'
19:54 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevs.hooks.php 'Deploy 77740'
19:53 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevs.php 'Deploy 77740'
19:04 logmsgbot: demon synchronized php-1.5/extensions/CodeReview/ui/CodeRevisionTagView.php 'r77736'
18:55 logmsgbot: demon synchronized php-1.5/includes/api/ApiQueryRevisions.php 'r77735'
01:02 logmsgbot: catrope synchronized php-1.5/extensions/CodeReview/backend/CodeRevision.php 'r77708'
00:56 logmsgbot: catrope synchronized php-1.5/extensions/CodeReview/backend/CodeRevision.php 'r77705'
00:48 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Fix settings for r77701'
00:48 logmsgbot: catrope synchronized php-1.5/extensions/CodeReview/backend/CodeRevision.php 'r77701'
00:47 logmsgbot: catrope synchronized php-1.5/extensions/CodeReview/CodeReview.php 'r77701'
00:40 logmsgbot: catrope synchronized php-1.5/extensions/CodeReview/backend/CodeRevision.php 'r77699'
00:21 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Send all code comments to mediawiki-codereview@lists.wikimedia.org'

December 3

22:38 Ryan_Lane: powercycling sq70, since it is very dead
20:20 Ryan_Lane: fixing deadlock problem with torrus
18:27 mark: Shutdown csw5-pmtpa:7/9 (to asw-b4-pmtpa) - so rack B4 is now offline
18:26 mark: Shutdown csw5-pmtpa:7/3 (to asw-b3-pmtpa)
14:09 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'sending most of s3 load to db39'
14:08 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'adding db39, reduced s3 host, 5.1'
14:03 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'send most of s7 load to db37'
13:56 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'initial s7 server and wiki sets defined, db37 in as s7-only slave'
10:31 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 18307 - Autopatrolled group for frwikisource'
02:53 Ryan_Lane: rebooting nova-compute1.tesla
00:41 logmsgbot: tfinc synchronized php-1.5/extensions/LandingCheck/SpecialLandingCheck.php
00:31 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php
00:30 Andrew: Granted 'disableaccount' right to users on arbcom_enwiki, according to email conversations
00:28 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php
00:24 logmsgbot: andrew synchronizing Wikimedia installation... Revision: 77622:
00:23 Andrew: Running scap.
00:17 Andrew: About to deploy updates for Special:DisableAccount

December 2

23:11 Ryan_Lane: installing user-mode-linux on nova-compute1/2
23:10 Ryan_Lane: setting --libvirt_type=qemu in the nova.conf on nova-*.tesla
22:25 Ryan_Lane: adding icmp to the default security list's exceptions on nova-controller.tesla using "euca-authorize default -P icmp -t -1:-1 -s 0.0.0.0/0"
22:22 Ryan_Lane: adding ssh to the default security list's exceptions using "euca-authorize default -P tcp -p 22 -s 0.0.0.0/0"
22:07 Ryan_Lane: adding a lucid image to the lucid-64-1 bucket using uec-publish-tarball
22:02 Ryan_Lane: installing unzip on nova-controller.tesla
21:59 Ryan_Lane: installing cloud-utils on nova-controller.tesla
21:29 Ryan_Lane: giving eth2 a static IP assignment on nova-controller.tesla; no bridging is necessary on the controller, only compute nodes
21:21 Ryan_Lane: installing python-mysqldb on nova-compute1 and 2
21:17 Ryan_Lane: removed template, keys, and ca path flags from nova-*.tesla, and added state_path and dhcp_bridge flags
21:12 Ryan_Lane: updating nova packages on nova-controller.tesla
20:53 Ryan_Lane: made owa1-3 real servers. depooling owa3
20:44 mark: Setup owa.wikimedia.org LVS service on lvs4
17:56 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 26159 groups for hiwiki'
17:53 Ryan_Lane: restarting rabbit on nova-controller.tesla
17:53 CodeBlock: installed mysql-client on nova-compute1 and 2
17:52 Ryan_Lane: restarting nova services on nova-controller.tesla
17:44 CodeBlock: installed nova-common nova-doc python-nova nova-compute nova-volume on nova-compute1, as well.
17:43 mark: Shutdown port csw5-pmtpa:2/47 (to msw-a1-pmtpa) - so rack A1 is now fully offline
17:41 CodeBlock: installed nova-common nova-doc python-nova nova-compute nova-volume on nova-compute2
17:40 mark: Shutdown ports csw5-pmtpa:7/23 (to msw-b1-pmtpa) and 7/7 (to asw-b1-pmtpa) - so rack B1 is now fully offline
17:24 Ryan_Lane: restarting nova services on nova-controller.tesla after configuring them to use the ldap driver
13:25 hcatlin: deploying fixes to mobile cluster
12:01 domas: please don't run any ddl on s3 (no new wiki creations either) until further notice, consider your data lost otherwise
02:51 richcole: rebuilding raid array DB15
02:38 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'disabling db15'
02:15 richcole: swapping drive 15 on DB 19
02:11 Ryan_Lane: stopping nova services on nova-*.tesla
02:11 Ryan_Lane: adding mount for /var/lib/nova on nova-*.tesla
02:06 Ryan_Lane: added another nic to all nova-* systems, on a new virtual nic called "nova vm network". To be used for VMs with no access to the outside world.
02:05 CodeBlock: added phuzion to sudoers on nova-*.tesla
01:51 RobH: db15 down for reinstallation, do not take db13 or db24 offline
01:43 Tim: added a user account for hashar in the wikidev group, with the same UID and home directory as in 2006-2008
01:13 richcole: reboot DB15
01:09 RobH: db15 is not recovering from lockup, lost a lot of drives in array, rich is investigating
01:02 RobH: db15 unresponsive to console, rebooting
01:01 RobH: db15 drive was swapped, now is down, investigating

December 1

23:16 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'Updating fundraiser counter source for 2010'
22:50 Ryan_Lane: rebooted mobile3
21:17 mark: Removed jjones account for shell access
21:06 Ryan_Lane: installing python-libvirt on nova-controller.tesla
20:58 Ryan_Lane: restarting nova services on nova-controller.tesla
20:55 Ryan_Lane: installing rabbitmq-server on nova-controller.tesla
20:19 Ryan_Lane: installing ldap-utils on nova-controller.tesla
19:54 Ryan_Lane: installing php5-cli on nova-controller.tesla
19:54 Ryan_Lane: installing php5, php5-ldap, and php5-mysql on nova-controller.tesla
19:54 Ryan_Lane: installing apache2 on nova-controller.tesla
19:54 Ryan_Lane: installing opendj on nova-controller.tesla
19:47 CodeBlock: also installed mysql-server on nova-controller.tesla
19:46 CodeBlock: added nova's PPA to nova-controller.tesla, and installed nova-common nova-doc python-nova nova-api nova-network nova-objectstore nova-scheduler python-greenlet python-mysqldb
19:45 Ryan_Lane: added hosts entries on nova-*.tesla for backend IP addresses
19:20 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 26159 - Enable some user groups on hi wiki'
19:07 RobH: updated flaggedrevs and pushed for bug 24622
19:07 logmsgbot: robh ran sync-common-all
17:57 Ryan_Lane: installing vmware tools on nova-*.tesla
11:03 logmsgbot: tstarling synchronized wmf-deployment/cache/trusted-xff.cdb
10:04 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
03:51 CodeBlock: installed htop on nova-controller
01:37 Ryan_Lane: powered off, but did not delete ds1.tesla, storage1.tesla, and storage2.tesla. Re-used their resources for nova-*.tesla though.
01:36 Ryan_Lane: gave shell and sudo access to codeblock on nova-*.tesla systems
01:35 Ryan_Lane: added vms nova-controller.tesla, nova-compute1.tesla, and nova-compute2.tesla

November 30

22:05 Ryan_Lane: correction that's ds1.tesla to nova-controller.tesla
22:03 Ryan_Lane: changing ds1 to nova-controller in dns
21:23 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25650 - Please add rollback group on fa.wikinews'
21:14 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '16315 - Change favicon for Wiktionaries'
21:03 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '16315 - Change favicon for Wiktionaries'
20:51 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26171 - Change logo for Esperanto Wikibooks'
16:54 Ryan_Lane: destroying ds1.tesla, storage1.tesla, and storage2.tesla to reclaim memory and an IP address for OpenStack testing with volunteer ops/dev community members
16:16 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Temporarily allow sysops to -flood anyone on zhwiki, because $wgGroupsRemoveFromSelf seems to be broken'
16:06 Ryan_Lane: removing amanda-client from tridge, it isn't needed afterall
15:54 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 26117 - Fix wrong syntax in zhwiki entries in $wgGroupsAddToSelf and $wgGroupsRemoveFromSelf'
15:51 Ryan_Lane: adding amanda-client to tridge
15:29 ^demon: reinstated pre-commit hook for php lint test
15:20 ^demon: disabled pre-commit hook for php -l while I debug some more, hitting issues with svn del's
14:59 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 26151 - Change project namespace on mrwiktionary'
14:57 ^demon: bug 26172, added php lint check to public svn pre-commit
13:18 mark: moved noc monitoring out of LVS (I want to sleep at night)
13:05 logmsgbot: catrope synchronized php-1.5/wmf-config/abusefilter.php 'Set $wgAbuseFilterBlockDuration to 1 month on hiwiki'
12:55 logmsgbot: catrope synchronized php-1.5/wmf-config/abusefilter.php 'bug 24394 - Allow block and unblock as abuse filter actions on hiwiki'
12:24 mark: Enabled Nagios SMS notifications for critical services to all paid Ops staff and Danese
12:08 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
12:07 logmsgbot: catrope synchronized wmf-deployment/extensions/UploadWizard/resources/combined.min.js 'r77473'
11:29 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable UploadWizard on commonswiki'
11:28 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
11:22 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 77471:
11:22 RoanKattouw: Running scap to deploy UploadWizard changes cluster-wide. Only enabled on test right now, will go to Commons soon
11:07 logmsgbot: catrope synchronized wmf-deployment/extensions/UploadWizard/resources/combined.min.js
10:51 logmsgbot: catrope synchronized php-1.5/extensions/UploadWizard/UploadWizard.i18n.php
10:44 logmsgbot: catrope synchronized wmf-deployment/extensions/UploadWizard/resources/combined.min.js
10:41 logmsgbot: catrope synchronized wmf-deployment/extensions/UploadWizard/resources/combined.min.js
10:31 logmsgbot: catrope synchronized wmf-deployment/extensions/UploadWizard/resources/combined.min.js
10:22 logmsgbot: catrope synchronized wmf-deployment/extensions/UploadWizard/resources/combined.min.js
05:14 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26117 - Enable flood flag on zhwiki'
00:32 Ryan_Lane: done checking nagios monitoring of LDAP on nfs2
00:32 Ryan_Lane: checking nagios monitoring of LDAP on nfs2

November 29

22:49 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionTracking/ContributionTracking_body.php
22:47 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionTracking/ContributionTracking.i18n.php
22:22 Ryan_Lane: changed check for noc.wikimedia.org to use http, not https
21:46 Ryan_Lane: finished testing manifestcheck. The pre-commit hook for puppet's svn is more complete now. It will pull all manifests and check site.pp and all includes.
21:34 Ryan_Lane: testing new pre-commit manifestcheck script for puppet svn. errors may be reported for ldap.pp
19:21 Ryan_Lane: even better... testing with a file not included in site.pp
19:21 Ryan_Lane: testing svn pre-commit hook for puppet. ldap.pp may show some errors...
18:16 Ryan_Lane: added http monitoring for noc.wikimedia.org
17:58 Ryan_Lane: added ldap and ldaps monitoring to nfs1/2
17:45 Ryan_Lane: adding backup::client class to nfs1/2 for ldap backups
13:55 domas: changed db10 lag threshold to alert at 10 minutes \o/

November 28

19:20 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php
19:18 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25693 - Change logo on tawikisource'
19:11 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26074 - please enable the Collection extension for the albanian wikipedia'
19:10 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26096 - setup logo for br.wikimedia.org'
19:08 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25780 - Create "autopatroller" User Group and granting admins(sysop) to add/remove this user group in ca.wikipedia'
18:55 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26117 - Enable flood flag on zhwiki'
18:46 logmsgbot: jeluf ran sync-common-all 'closing cowikiquote'
18:38 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26060 - Change logo for Esperanto Wikinews'
18:36 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26153 - Please restrict anonymous users from creating new pages at es.wikibooks'
18:33 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26155 - Please enable the "autopatrolled" group on it.wikipedia'
18:30 logmsgbot: jeluf synchronized php-1.5/wmf-config/flaggedrevs.php '26154 - Activate Flagged Revs For NS_CATEGORY'
11:51 logmsgbot: midom synchronized php-1.5/wmf-config/db.php

November 27

22:49 logmsgbot: catrope synchronized php-1.5/wmf-config/db.php 'Put db18 back into rotation, it's caught up'
22:41 logmsgbot: catrope synchronized php-1.5/wmf-config/db.php 'Set db18 load to 0 while while it's lagged'
20:57 logmsgbot: midom synchronized php-1.5/extensions/OAI/OAIRepo_body.php 'removing audit'
12:54 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
11:25 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'db25 maintenance'

November 26

14:52 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesBh.php 'r77331'
13:58 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
13:57 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js 'r77329'
13:31 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 25923 - Namespace aliases for brwikisource'
11:52 mark: Fixed packages on fenari
03:42 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'disabling proxy for RSS extension, to allow requests to blog.wikimedia.org'

November 25

21:28 RoanKattouw: And Ariel restarted Apache too
21:27 RoanKattouw: Apache on fenari was down, caused by wikimedia-task-appserver being half-installed. Ariel ran apt-get install on it manually, which seems to have fixed it
20:07 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/api/ApiReview.php 'Deploy r77298'
13:35 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesExt.php 'r77287'
13:30 logmsgbot: catrope synchronized php-1.5/includes/api/ApiQueryUsers.php 'r77285'
12:42 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'In RSS extension, use url-downloader.wikimedia.org'
12:40 logmsgbot: tstarling synchronized php-1.5/extensions/RSS/RSSParser.php
12:40 logmsgbot: tstarling synchronized php-1.5/extensions/RSS/RSS.php
12:26 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'enabling RSS extension on wikimediafoundation.org'
12:26 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php
12:22 logmsgbot: tstarling synchronizing Wikimedia installation... Revision: 77242:
12:21 Tim: pushing out RSS extension to update extension messages, not enabled yet
10:46 richcole: dataset1 being taken down for service
01:30 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/maintenance/updateStats.php 'deployed r77270'
01:30 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/api/ApiReview.php 'deployed r77270'
00:35 richcole: dataset1 going down for part replacement
00:28 tomasz: turning off replication for faulkner db on db10. setting db10 to rw per nimish talk with domas.

November 24

23:13 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/FundraiserStatistics_body.php
23:13 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/ContributionReporting.i18n.php
23:05 logmsgbot: catrope synchronized php-1.5/includes/media/Bitmap.php
23:02 logmsgbot: catrope synchronized php-1.5/includes/media/Bitmap.php 'r77263'
22:59 logmsgbot: catrope synchronized php-1.5/includes/media/Bitmap.php 'Back out r77262, is broken'
22:55 logmsgbot: catrope synchronized php-1.5/includes/media/Bitmap.php 'r77262'
22:50 JeLuF: added srv230 in /etc/dsh/group/mediawiki-installation
22:29 logmsgbot: catrope synchronized php-1.5/thumb.php
22:19 atglenn: ms4 thumb_handler.php synced to latest from svn, temp dir thumb handling (see rev 77257)
22:18 logmsgbot: catrope synchronized php-1.5/thumb.php 'Live hack for temporary thumbs'
21:01 tomasz: moving faulkner db from faulkner to un replicated fundraiser_history on db10
15:53 Ryan_Lane: powercycling mobile2
15:49 Ryan_Lane: restarting apache on mobile3
15:46 RoanKattouw: mobile3 is broken, serving 500s for http://m.wikipedia.org . mobile1 is fine; this causes LVS HTTP status to flap in Nagios
03:57 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 77216:
03:50 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 77216:
03:15 logmsgbot: demon synchronized php-1.5/extensions/FlaggedRevs/api/ApiReview.php 'r77216'
03:15 logmsgbot: demon synchronized php-1.5/extensions/FlaggedRevs/forms/RevisionReviewForm.php 'r77216'
02:55 logmsgbot: demon synchronized php-1.5/extensions/FlaggedRevs/FlaggedArticleView.php 'r77213'
02:43 Ryan_Lane: restarted memcached on owa1/2 to read new config file
00:19 logmsgbot: demon synchronizing Wikimedia installation... Revision: 77209:
00:17 ^demon: syncing FlaggedRevs changes to all wikis

November 23

23:11 logmsgbot: demon synchronized php-1.5/includes/specials/SpecialRecentchangeslinked.php 'r76485'
23:10 logmsgbot: demon synchronized php-1.5/includes/specials/SpecialRecentchanges.php 'r76485'
23:09 logmsgbot: demon synchronized php-1.5/includes/diff/DifferenceInterface.php 'r76444'
21:58 ^demon: restarted wmfsvndumper with new dump location
21:44 ^demon: running svndump test in screen on formey
21:43 Ryan_Lane: installed screen on formey
21:42 Ryan_Lane: added demon to svnadm
21:41 Ryan_Lane: changed ownership of svn config files from adm to svnadm
21:41 Ryan_Lane: added an svnadm group to ldap
15:07 mark: Finished testing, amslvs1 is primary again
14:40 mark: Shutdown BGP session to amslvs1 on csw1-esams for testing, traffic failing over to amslvs3
13:50 mark: Removed redistribution of BGP LVS routes into OSPF; rely on iBGP exclusively for LVS routes
12:06 logmsgbot: catrope synchronized php-1.5/includes/api/ApiDelete.php 'r77146'
02:27 RobH: had to restart morebots, it had left all channels. also had to change admin pass for listserv

November 22

22:09 mark: Restarted varnish on knsq4
22:05 mark: Restarting varnish on knsq1
21:44 Ryan_Lane: err - restarting varnish on knsq2,4,5
21:44 Ryan_Lane: restarting varnish on knsq2,3,4
21:43 Ryan_Lane: restarting knsq2,3,4
21:29 mark: Restarted varnish on knsq2,4,5
21:24 mark: Restarting Varnish on knsq1
21:09 mark: Set static routes 91.198.174.232/31 to 91.198.174.247 (csw1-esams) on br1-knams
21:03 mark: Started pybal on amslvs1 again
20:53 mark: Killed PyBal on amslvs1
20:39 mark: Lowered cache_mem and cache_dir sizes on amssq31 for testing
20:28 Ryan_Lane: depooling knsq24
20:25 mark: Restarted pybal on amslvs1 with depool-threshold = 1 (temporarily)
17:27 Ryan_Lane: restarting nagios
17:27 Ryan_Lane: adding warning notifications via IRC to nagios
16:40 mark: Suppressed announcements to AS16265 on csw1-esams
16:30 mark: Suppressed BGP announcements to AS13030 on br1-knams

November 21

08:29 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25929 - Import sources for Spanish Wikipedia'
05:49 tomasz: purging binary logs on db9 to mysql-bin.001600
05:43 tomasz: syncing db9 mysql-bin.001562 - mysql-bin.001599 to tridge

November 20

21:12 JeLuF: added download.wikipedia.org as ServerAlias for download.wikimedia.org
20:14 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25982 - Enable subpages in the main namespace for ten.wikipedia.org'
20:11 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25923 - Namespaces on br.wikisource'
20:09 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25929 - Import sources for Spanish Wikipedia'
20:06 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25996 - Import sources for pfl.wikipedia'
13:00 JeLuF: added more error patterns to puppetmon

November 19

23:04 Ryan_Lane: changed https check on payments to 1 retry in nagios, via puppet
20:59 atglenn: restarted torrus. guess why :-P
17:52 mark: Readded notification of Service[nagios] when changing nagios types in puppet
13:59 richcole: set raid 10 up on DB41
08:36 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'bug 25850 - Hide Take me Back link on all wikis'
07:32 JeLuF: started puppet on sq42 sq41 sq47 sq45 sq46 sq44 sq50 sq52 sq53 sq51 sq48 sq55 sq54 sq43 sq56 sq58 sq64 sq62 sq61 sq65 sq66 sq60 sq63 sq72 sq75 sq71 sq73 sq74 sq76 sq78 sq77 sq81 sq80 sq79 sq83 sq84 sq82 sq85 sq86
05:21 apergos: on sq85 I was seeing complaints from cron about restart of puppet: unknown option -w. removed that from /etc/default/puppet and restart, but that fails: Could not parse for environment production: Could not find file /agent.pp
01:00 atglenn: added monitoring mechanism in root's crontab on sq85 (don't need it everywhere) that will sms me when ms4 is acting up. I'd do it in puppet if someone told me how they would want it to be added there.
00:22 tomasz: turning db9 watchdog back on. setting at 5minutes

November 18

21:57 richcole: DB42 shutdown for service
21:40 JeLuF: CORRECTION: started puppet manually on sq59 sq61 sq73 sq60 sq62 sq65 sq77 sq63 sq64 sq72 sq75 sq74 sq76 sq71 sq78 sq66, startup script is broken.
21:40 JeLuF: started squid manually on sq59 sq61 sq73 sq60 sq62 sq65 sq77 sq63 sq64 sq72 sq75 sq74 sq76 sq71 sq78 sq66, startup script is broken.
21:26 mark: Fixed puppet on formey
21:24 mark: Fixed puppet on linne
19:57 JeLuF: blocked UDP from srv124 on nfs1 aka syslog
17:00 JeLuF: restarted puppet on srv215, srv235, srv244, srv257, srv262, srv288
16:33 JeLuF: fixed puppet on srv185 and srv200
15:00 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'Set FR_INCLUDES_CURRENT on mediawikiwiki'

November 17

20:19 JeLuF: syslog is being spammed with one week old messages from srv124
20:19 RobH: owa1/2/3 online with base OS install and puppet updates
17:59 RobH: updated dns for new databases servers
17:15 richcole: owa1 going down for repair
15:52 Ryan_Lane: moved the nagios purge stuff out of puppet, and into nagios's init script. Pulled the nagios init script into puppet
10:03 tomasz_: adding single field index on converted amount under public_reporting within civirm db on db9
10:03 tomasz_: adding single field indexes to utm_source, utm_medium, and utm_campaign under contribution_tracking table within drupal db on db9
03:44 atglenn: restarted apache on ekrem, many processes hung in "graceful close" state for a long period of time
03:06 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialBannerController.php
03:04 Tim: in puppet, disabled nagios::purge since it breaks puppet entirely on fenari. Removed Aaron's obsolete ssh public key by adding an ensure=>absent to puppet.
01:34 Tim: on ekrem: ran logrotate -f, since log rotation previously failed due to disk full
01:28 Tim: on ekrem: root partition full, deleted old apache access logs
00:20 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php

November 16

19:05 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 76812:
19:03 RoanKattouw: Running scap to deploy UploadWizard backend changes (core only)
17:04 Ryan_Lane: adding run stages to puppet config; adding apt-get update to first stage, and nagios resource purging to last stage
17:01 logmsgbot: catrope synchronized php-1.5/maintenance/nextJobDB.php 'Fix memcached usage for nextJobDB.php, broken since Sep 09. Should speed up job queue processing'
16:38 RobH: updated dns
15:50 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Add deletedtext, deletedhistory rights to eliminator group on hiwiki'
15:04 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 25374 - Eliminator group for hiwiki'
14:36 JeLuF: 25871 - fixed logo for pflwiki
14:05 RobH: temp fixed nagios

November 15

22:31 Ryan_Lane: repooling sq70
21:43 Ryan_Lane: pushing change to varnish to send cache-control header for geoip lookup
21:43 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Adding categories to $wmgArticleAssessmentCategory'
21:37 logmsgbot: catrope synchronized php-1.5/extensions/ArticleAssessmentPilot/ArticleAssessmentPilot.hooks.php 'r76709'
21:17 Ryan_Lane: depooling sq70
21:17 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25569 - Create the Gagauz Wikipedia (wp/gag)'
21:01 mark: Lowered CARP weight of esams text amssq* squids from 20 to 10, equal to the older knsq* squids
20:33 Ryan_Lane: setting authdns-scenario normal
20:05 RobH: current slowdowns reported for folks hitting AMS squids. Moving traffic to US datacenter should fix major slowdowns on !Wikipedia & !Wikimedia
20:04 Ryan_Lane: setting authdns-scenario esams-down
19:56 RobH: fixed nagios again
19:51 RobH: updating dns for new owa processing nodes
18:54 RobH: srv298 now online in api pool
18:21 Ryan_Lane: fixing puppet manually on sq34, sq36, sq37, sq39, sq40, and knsq13
18:18 RobH: gilman to secure gateway project stalled, needs network checks done
18:07 Ryan_Lane: puppetizing /etc/default/puppet, since some hosts had START=no, instead of START=yes
17:46 RobH: gilman needed hard reset, ilom responsive now (thx rich!)
17:35 Ryan_Lane: restarting puppet again on all nodes using -M flag for ddsh to see system names (checking for errors)
17:23 Ryan_Lane: restarting puppet on all nodes
17:12 RobH: sq57 disk replaced, reinstalled, back in service
17:09 mark: Restarted apache on sockpuppet with concurrency 4 instead of 3
17:04 RobH: puppet is now failing to work properly on sq57, why did we upgrade puppet again?
16:59 RobH: sq57 reinstalled and doing post installation configuration
16:40 Ryan_Lane: upping configtimeout setting in puppet to 8 minutes, globally
16:33 Ryan_Lane: trying to add puppet.conf to puppet again
16:24 Ryan_Lane: undoing puppet.conf changes
16:20 RobH: sq57 coming down for reinstallation
16:19 RobH: db13 back online, restarted mysql, but its currently commented out of db.php
16:12 RobH: not sure why db13 is borked, but its down, poking at it
16:09 Ryan_Lane: added puppet.conf to puppet. pushing change out
16:00 RobH: torrus is up again
15:59 richcole: swaped sq57 sdb bad drive
15:56 RobH: torrus is down, again, restarting and cleaning up its services
15:52 RobH: manually purged spence nagios, started manually, working until puppet borks it again
15:10 RobH: nagios is down, investigating

November 14

23:28 mark: Fixed Nagios
20:00 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25918 - Namespaces on vec.wikisource.org'
14:57 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25904 - Create the Swedish Wikiversity (wv/sv)'
14:56 logmsgbot: jeluf ran sync-common-all '25904 - Create the Swedish Wikiversity (wv/sv)'
14:28 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25918 - Namespaces on vec.wikisource.org'
08:26 domas: ran purge-nagios-resources.py manually to bring up nagios
07:14 domas: reduced passenger pool size to 4 on sockpuppet
04:42 Ryan_Lane: moving /etc/nagios/puppet_services.cfg to .bak and rerunning puppet
03:05 Ryan_Lane: modified nagios puppet manifest to purge decommisioned servers from the services configuration
01:48 logmsgbot: jeluf synchronized php-1.5/cache/interwiki.cdb 'Updating interwiki cache'
01:33 Ryan_Lane: temporarily upped configtimeout in /etc/puppet/puppet.conf to 8 minutes on spence so that puppet would run

November 13

22:03 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/ContributionHistory_body.php
20:42 mark: Ran dist-upgrade on sq68
20:37 mark: powercycled sq68
19:29 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25871 - Create the Palatinate German Wikipedia (wp/pfl)'
19:29 logmsgbot: jeluf ran sync-common-all '25871 - Create the Palatinate German Wikipedia (wp/pfl)'
18:39 mark: Fixed puppet on db16
18:24 mark: Installed script reporting the last Puppet run in MOTD (Karmic and higher only)
18:03 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25774 - Create Wikinews in Esperanto'
18:02 logmsgbot: jeluf ran sync-common-all '25774 - Create Wikinews in Esperanto'
17:43 logmsgbot: jeluf ran sync-common-all '25773 - Create Wikibooks in Limburgish'
17:37 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25743 - Create the Breton Wikisource (ws/br)'
17:27 logmsgbot: jeluf ran sync-common-all '25743 - Create the Breton Wikisource (ws/br)'
17:10 mark: Installed cron job that removes puppetdlock files over a day old; these prevent puppet from doing runs forever otherwise
17:01 apergos: removed "-n" from mw-tor-list on hume, otherwise it (I guess) terminates early, at any rate it produces an empty tor node list. If this turns out to be too big a burden on hume's resources we can look at some other approach
16:49 mark: Upgrading puppet agent from 0.25 to 2.6 across the cluster
16:24 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25696 - Create vec.wikisource.org'
16:04 logmsgbot: jeluf ran sync-common-all 'added gag.wikipedia and vec.wikisource'
15:48 mark: Replaced the Wikimedia APT repository by a new per-distribution-version repository managed by 'reprrepro' on brewster; the old repository is available as http://apt.wikimedia.org/wikimedia-old/
14:29 logmsgbot: jeluf synchronized php-1.5/cache/interwiki.cdb 'Updating interwiki cache'
10:51 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25714 - Adding sources wikis for [[Special'
09:30 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25156 - Requesting an alias for project namespace on Persian Wikipedia'
08:58 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25859 - Enable Collection on gl.wikipedia'
08:45 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25893 - Wikimania Logo for WikimaniaTeam Wiki'

November 12

22:56 Ryan_Lane: added file_mover user to hume
21:22 mark: Fixed torrus
21:11 mark: Setup amanda backups of brewster:/srv/{wikimedia,autoinstall,tftpboot}
18:31 logmsgbot: jeluf ran sync-common-all '25737 - Closure of Nauruan Wikibooks'
18:06 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/ContributionReporting.php
17:46 Ryan_Lane: adding demon to shell accounts as mortal
17:35 Ryan_Lane: rebooting ersch
17:34 Ryan_Lane: rebooting alsted
15:58 mark: Shutdown srv126 for decommissioning
15:27 Ryan_Lane: deleting svnuser.pp manifest, and any references to it, since we are now using ldap for svn users instead.
14:11 RobH: srv230 shows memory error in SEL. it reboots and sees all memory. opening a ticket to ensure its not showing the memory error on its LCD
12:22 mark: Rebooting sockpuppet
12:20 mark: Running apt-get dist-upgrade on sockpuppet
12:05 mark: Converted puppetmaster install on sockpuppet from mongrel based to passenger based
11:19 mark: Upgraded puppet and puppetmaster on sockpuppet to 2.6.1
07:09 JeLuF: restarted crashed backend squids on sq41 and sq42
01:32 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/FundraiserStatistics.css
00:44 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialBannerAllocation.php
00:27 logmsgbot: tfinc synchronized php-1.5/extensions/LandingCheck/SpecialLandingCheck.php
00:27 logmsgbot: tfinc synchronized php-1.5/extensions/LandingCheck/LandingCheck.php
00:27 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'Changing landing check to be wg var'

November 11

23:59 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Adding country codes to landing check'
22:09 RobH: torrus wasnt recording items, restarted
18:14 apergos: restarted varnish on storage1, seemed it might have gone out to lunch
17:41 mark: Powercycled sq84
17:01 mark: powercycled sq80
16:52 mark: Powercycled amssq50
16:42 mark: Powercycled sq68
16:37 mark: Powercycled sq59
16:34 mark: Powercycled sq57
15:38 mark: Removed ex-fedora data on ms2, after backing it up to tridge
10:33 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
10:32 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js 'r76511'
09:58 Ryan_Lane: restarted apache on fenari
04:02 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/ContributionReporting.php 'Updating for 2010'
03:50 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'removing test since its in the extension config'
03:46 logmsgbot: tfinc synchronizing Wikimedia installation... Revision: 76474
02:30 atglenn: so another restart of torrus. seriously...
00:43 domas: what Rob meant was that they went away by themselves, as it was upstream provider issue.
00:39 RobH: !wikipedia and !wikimedia network issues resolved, all projects should be fine now
00:35 domas: #network #failwhale #lol
00:31 RobH: looking into the current slowdown/inaccessibilty issues for folks on !Wikipedia and !Wikimedia
00:25 domas: flapping network in pmtpa

November 10

22:32 rfaulk: installed "scipy" python package on grosley.wikimedia.org with apt-get - statistical analysis in python
22:23 atglenn: restarted torrus, it had deadlocked again. is it my imagination or is this happening really often lately?
21:26 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
21:25 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js 'r76474'
21:21 logmsgbot: nimishg synchronized php-1.5/extensions/ContributionReporting/ContributionReporting.i18n.php 'r76472'
21:16 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
21:16 RoanKattouw: Removed srv124 from mediawiki-installation node group as it's slated to be decommissioned
21:14 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js 'r76471'
20:44 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
20:44 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js 'r76469'
20:32 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
20:31 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r76467'
19:40 RobH: singer config restarted, will host download.w.o & dumps.w.o as well as a number of other things that refer to those two entries in dns
19:39 RobH: changed dns for dumps.wikimedia.org to go to singer instead of dataset1 during its downtime
18:59 atglenn: someone was polite and didn't name me in the above comment :-P I commented out the script that ships logs to both dammit.lt and dataset1 instead of looking at the script itself
18:58 domas: unbroke pagecounts shipment (someone broke it and said "yes you can blame me, it was my f*ckup, people should know that")
14:59 apergos: rebooting dataset1 so we can get web service going over there (can't be restarted in the usual way after kernel panic)
10:38 RoanKattouw: Published MW 1.16 tarball on noc.wm.o because download.wm.o is still down http://noc.wikimedia.org/mediawiki-1.16.0.tar.gz
06:47 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'Turning cc gateway back on with the sidebar'
05:20 apergos: stopped rsync of pagecount stats from locke to dataset1 for now til disk/fs issue is resolved
04:59 apergos: shot all dump processes on dataset1; note a kernel panic in logs from within __destroy_inode, going to reboot and leave rsync of pagecounts and dumps off
00:41 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'Taking outage on cc cluster'

November 9

22:42 RobH: running puppet on spence to remove all the old apaches that are no longer in any kind of service
22:39 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24539 - Transwiki import source for ml.wikisource.org'
22:36 RobH: didnt log my change that I ran about 25 minutes ago to change test.w.o from srv124 to srv193 in squid settings and deployed
22:32 logmsgbot: jeluf ran sync-common-all
22:27 logmsgbot: jeluf synchronized closed.dblist
21:31 logmsgbot: robh synchronized php-1.5/wmf-config/mc.php 'removed srv193 from potential memcached pool as it will shortly become the new test.w.o server'
20:53 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Set tenwiki logo to local Wiki.png'
17:26 logmsgbot: robh synchronized php-1.5/wmf-config/abusefilter.php 'bugzilla#24394'
17:17 logmsgbot: robh synchronized php-1.5/wmf-config/abusefilter.php
17:15 RobH: that actually ran 15 minutes ago and was stuck on a broken server at the end, all other hosts had synced
17:15 logmsgbot: robh synchronized php-1.5/wmf-config/mc.php 'srv230 unresponsive to ssh, needs to reboot, swapped it out for working spare'
15:51 RobH: removed srv* under srv151 from pybal, left entry for srv124 as its test.w.o, even though its set to false
15:46 RobH: srv229 puppet was hanging, manually ran apt-get update and reran puppet, now its happy
15:22 RobH: srv229 rebooted, wouldnt let me ssh in, coming back up now with puppet run
15:21 logmsgbot: robh synchronized php-1.5/wmf-config/mc.php 'srv281 is not behaving, swapped it out'
15:17 logmsgbot: robh synchronized php-1.5/wmf-config/mc.php 'replacing a server that i am working on'
15:01 mark: Reduced CARP weight of new amssq squids from 30 to 20, until they get SSDs
12:16 mark: Started copy of ms2:/a/ex-fedora data to tridge
12:15 mark: Included class base for server tridge in Puppet
12:09 mark: Cleaned up temporary files on image scalers
06:33 apergos: cleared out some old bin log files on db27 to get back some space
05:20 apergos: reboot of ms4 successful, we need to monitor performance over the next several days. If people are still seeing 503's for thumbs that's a problem
05:09 apergos: er, ms4!!
05:09 apergos: rebooting ms5 into alternate boot environ, now with new improved patches (:-P), let's see if it works
01:00 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'Bumping style version for meta udp2log fix'

November 8

23:50 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'updating tenwiki logo'
23:18 logmsgbot: tstarling ran sync-common-all
22:32 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25801 - New logo for et.wikimedia.org'
22:27 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25742 Please set the Buryat Wikipedia logo'
22:25 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25742 Please set the Buryat Wikipedia logo'
22:20 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25836 - Commons'
22:13 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25779 - Create new namespace "Institution"/"Museum" for Commons'
21:54 JeLuF: Fixed typo: Replaced the second "srv290" in node group "apaches" by srv296.
21:10 logmsgbot: jeluf synchronizing Wikimedia installation... Revision: 76208
21:10 JeLuF: srv154 didn't receive any updates in the last few days, it was missing in the mediawiki-installation nodegroup
20:19 mark: Moved uplink of asw-b-sdtpa from temporary GigE link csw1-sdtpa:3/48 to 2x 10G (LACP) links csw1-sdtpa:16/2 and 16/3, shutdown 3/48
19:44 logmsgbot: nimishg synchronized php-1.5/extensions/LandingCheck/SpecialLandingCheck.php 'r76156'
19:44 logmsgbot: nimishg synchronized php-1.5/extensions/LandingCheck/LandingCheck.php 'r76156'
19:22 RobH: updated dns, seems it borked, but pdns is running on nescio so it should clear up
17:47 logmsgbot: robh synchronized php-1.5/wmf-config/abusefilter.php 'bug'
16:38 RobH: esams squid flapping on text squids is due to disk i/o use, they will be replaced with SSD soon
06:04 apergos: brought ms4 back up in primary boot environment, testing concluded for tonight, results being sent to Oracle
05:50 apergos: doing reboot of ms4 into alternate boot environ for testing
02:34 apergos: restarted torrus, it was out to lunch again

November 7

17:39 RoanKattouw: Starting makeArbcomList.php in a screen on fenari. Tim says this'll take about a day
12:43 logmsgbot: catrope synchronized php-1.5/includes/Profiler.php 'r76243'
12:04 logmsgbot: catrope synchronized php-1.5/wmf-config/abusefilter.php 'More perm changes for frwiktionary'
11:59 logmsgbot: catrope synchronized php-1.5/wmf-config/abusefilter.php 'More perm changes for frwiktionary'
11:56 logmsgbot: catrope synchronized php-1.5/wmf-config/abusefilter.php 'More perm changes for frwiktionary'
11:35 logmsgbot: catrope synchronized php-1.5/includes/Profiler.php
11:33 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 25711 - enable AbuseFilter on frwiktionary'
11:33 logmsgbot: catrope synchronized php-1.5/wmf-config/abusefilter.php 'bug 25711 - enable AbuseFilter on frwiktionary'
11:17 logmsgbot: catrope synchronized php-1.5/includes/Profiler.php 'Temp hack to debug fatals in Profiler.php'
10:48 logmsgbot: catrope ran sync-common-all
10:47 RoanKattouw: Enabling FlaggedRevs on sqwiki per bug 25822 and disabling new page patrolling
10:16 RoanKattouw: srv230 SSH is broken from fenari, Nagios disagrees. Commenting out srv230 from /etc/dsh/group/mediawiki-installation . After fixing srv230, uncomment it and resync the box
10:13 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 25674 - Enable $wgBlockAllowsUIEdit on frwiktionary'

November 6

18:59 logmsgbot: catrope synchronized php-1.5/extensions/CodeReview/ui/CodeRevisionView.php 'r76209'
18:56 logmsgbot: catrope synchronized php-1.5/extensions/CodeReview/ui/CodeReleaseNotes.php 'r76208'
18:56 logmsgbot: catrope synchronized php-1.5/extensions/CodeReview/ui/CodeAuthorListView.php 'r76208'
18:56 logmsgbot: catrope synchronized php-1.5/extensions/CodeReview/ui/CodeRevisionListView.php 'r76208'
18:56 logmsgbot: catrope synchronized php-1.5/extensions/CodeReview/api/ApiCodeUpdate.php 'r76208'
18:55 logmsgbot: catrope synchronized php-1.5/extensions/CodeReview/api/ApiCodeRevisions.php 'r76208'
18:55 logmsgbot: catrope synchronized php-1.5/extensions/CodeReview/api/ApiCodeDiff.php 'r76208'
18:55 logmsgbot: catrope synchronized php-1.5/extensions/CodeReview/api/ApiCodeComments.php 'r76208'
18:55 logmsgbot: catrope synchronized php-1.5/extensions/CodeReview/svnImport.php 'r76208'
18:55 logmsgbot: catrope synchronized php-1.5/extensions/CodeReview/CodeReview.i18n.php 'r76208'
18:55 RoanKattouw: Syncing CodeReview update

November 5

23:38 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Turning central notice back on for everyone'
23:09 logmsgbot: tfinc synchronizing Wikimedia installation... Revision: 76127
23:09 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Turnning off cn on all but testing wikis before scap'
23:08 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'Turnning off cn on all but testing wikis before scap'
23:03 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'Setting wgCentralDBname to meta'
20:34 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'Picking up url fix so that udp2log doesnt double count on meta'
20:33 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialBannerController.php 'Picking up url fix so that udp2log doesnt double count on meta'
20:15 RobH: There are no longer any memcached servers in the decommissioned server range. If there are any issues from the changes, the original mc.php is named mc.php.old and will be removed in 72 hours if there are no mishaps
20:14 logmsgbot: robh synchronized php-1.5/wmf-config/mc.php 'of course one dies AS i sync it'
20:12 logmsgbot: robh synchronized php-1.5/wmf-config/mc.php 'on secondary review, missed two old servers, removed and updated'
20:08 RobH: tested new memcached config, all servers working
20:08 logmsgbot: robh synchronized php-1.5/wmf-config/mc.php 'removed the older servers below srv150 and replaced with tested good new memcached servers'
19:55 logmsgbot: catrope synchronized images/wikimedia-button.png 'Let's try that with an actual PNG file rather than HTML'
19:50 logmsgbot: catrope synchronized images/wikimedia-button.png 'New Powered by Wikimedia button'
19:11 logmsgbot: catrope synchronized php-1.5/skins/common/images/poweredby_mediawiki_88x31.png 'r76126'
19:00 rfaulk: Added "httpagentparser" Python package on grosley.wikimedia.org from publicly avaialable distutils distribution - this package assists in parsing user-agent header strings found in the 2010/11 fundraiser squid logs
18:58 rfaulk: Added "setuptools" Python package on grosley.wikimedia.org with apt-get for Wikimedia 2010/11 fundraiser work - This package enables installation of python packages distributed with Python distutils
18:22 RobH: srv284 is having some booting issues, seems to be harddisk related, but since drac output is slightly garbled, unable to confirm. new rt# 376
17:54 RobH: srv284 unresponsive to console, rebooting and fixing it to bring it back into service
17:53 RobH: srv266 back online and in service
17:52 mark: Shutdown browne and srv2 for decommissioning - thereby removing the last traces of Fedora from the cluster. Goodbye!
17:43 RobH: srv266 unresponsive to remote console, rebooting and updating
17:42 RobH: srv206 fixed, pushed back into lvs
17:25 RobH: working on srv206, disregard any errors it throws
16:40 RobH: issue with the new api servers is fixed and they are now back in service
16:04 RobH: some new api servers are not working right, depooled until they are fixed
15:58 mark: Removed ibis IPs from Squid ACLs; invalid requests issue has been resolved
15:57 mark: Fixed NFS mounts on apaches that had them missing since the wikimedia-task-appserver upgrade
15:26 RobH: working on sq57, disregard flapping
15:24 RobH: new api apackes srv290-srv301 are online, except srv298 which needs drac correction before installation
15:22 RobH: dropping old entry for tenwiki in apache config and resyncing/restarting apaches to eliminate error message
15:18 RobH: pushing srv291-srv301 into lvs
15:11 RobH: doing puppet runs on srv292-srv301 before pushing them into service
14:57 mark: Hacked out the 'remotemount' lines in /var/lib/dpkg/info/wikimedia-task-appserver.postrm files to prevent apaches from being without NFS mounts during/between puppet runs and package upgrades
14:23 mark: Deploying new package wikimedia-task-appserver 1.46 across the cluster, which removes configuration files (now handled by Puppet)
11:59 logmsgbot: catrope synchronized php-1.5/includes/api/ApiLogin.php 'Revert r76078'
11:49 logmsgbot: catrope synchronized php-1.5/includes/api/ApiLogin.php 'r76078'
05:57 apergos: failure booting into be3 on ms4, had to back out. so, no progress, we are back to where we were before the reboots.
05:40 apergos: cleared up luactivate error, shutdown ms4 again, trying to boot into alt boot environment
05:16 apergos: used shutdown on ms4, be3 showed as "active on reboot" but it booted into be0 (old boot environment) nonetheless. *grumble*
05:06 apergos: rebooted ms4 into alt boot environment with current patches applied
00:18 RobH: new api servers are not coping down the data correctly and not reflecting config changes in puppet, so they fail, srv290+ not online yet

November 4

23:06 RobH: running puppet across the new api servers srv290-srv301 then will push them in service later when i figure out why they are not doing what I want ;P
20:13 RobH: sq51 hatees me
20:11 RobH: new api servers srv290-301 are installed and showing in ganglia, having issues getting the first couple to pool into lvs before i push the rest into service
20:09 RobH: fixed sq51
19:29 RoanKattouw: Strike that, have backed out changes
19:06 RoanKattouw: Until Mark's made sure they're good, that is
19:06 RoanKattouw: Changing some files in wmf-deployment/includes/media . DO NOT RUN SCAP or otherwise deploy these changes!
18:35 RobH: added dns entries for payments
17:59 RobH: doing puppet runs and final setup for srv290-srv301
16:56 rfaulk: Added numpy Python package to grosley.wikimedia.org with apt_get ... For use in the 2010/11 fundraiser to facilitate stats gathering by providing scientific computing functionality in Python
16:43 rfaulk: Added MySQLdb Python package to on grosley.wikimedia.org with apt-get ... This package will be used to access fundraising databases to facilitate the gathering and synthesis of relevant statistics for the 2010/11 Wikimedia findraiser
16:23 mark: Set storage1 (varnish) as upload backend on sq41-50, instead of ms4
16:14 RobH: sq59 is being bitchy and wont clean the cache, possible hdd issue? will investigate later
15:42 RobH: sq35 back in rotation
15:34 mark: Added storage1 (varnish->ms4) as an HTTP backend to sq45's squid config
15:34 RobH: commenting out sq35, trying to make it work again in pybal
15:16 RobH: poking at sq59
15:06 RobH: sq35 back online, pushed into lvs, partially up - may need to wait up to 5 for idleconnect timer
14:46 RobH: pushed dns updates for new payments boxes and correcting owadb1/2 to db31/32
14:28 RobH: sq35 set to false in pybal until i determine whats wrong with it
14:09 mark: Reduced CARP weight of sq41-50 from 10 to 5
13:37 RobH: sq35 may flag, disregard
13:30 RoanKattouw: Removed uploadwizard test wiki on prototype, gonna set it up on the Commons prototype instead
04:17 atglenn: ganglia 3.1 now running on ms4 and ms5
01:44 RobH: srv217 back in cluster
00:36 RobH: torrus back online
00:29 RobH: fixing torrus deadlock, no touchy
00:18 tomaszf: upped open file descriptors on loudon to 4096 for squid
00:17 RobH: kicking srv217 for reinstall

November 3

21:22 RobH: updated puppet to properly remove memcached from memcached::false entries and removed the host memcached check for servers no longer running memcached, hup'd nagios to take the change
21:21 atglenn: rebooting ms5 after OS update. note that we were unable to get some of the more recent patches, they are probably from after the sun->oracle transition
21:02 logmsgbot: nimishg synchronized php-1.5/extensions/LandingCheck/LandingCheck.i18n.php 'r75890'
21:02 logmsgbot: nimishg synchronized php-1.5/extensions/LandingCheck/LandingCheck.alias.php 'r75890'
21:01 logmsgbot: nimishg synchronized php-1.5/extensions/LandingCheck/SpecialLandingCheck.php 'r75890'
21:01 logmsgbot: nimishg synchronized php-1.5/extensions/LandingCheck/LandingCheck.php 'r75890'
20:31 atglenn: removed about 1.5T of stuff off of /export on ms4 (old backups, solaris isos, etc)
19:41 logmsgbot: catrope synchronized php-1.5/README 'Dummy sync so I can document what the errors look like'
19:32 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'Backing out config change for stats fix'
19:31 RobH: srv281 still down, setting to false in pybal just so it doesnt keep trying to use it
18:31 RobH: reinstalling srv281, tired of lookin at it in red
17:18 mark: Upgraded storage1 to Lucid
16:42 mark: Removing 2010-03 snapshots on ms4
16:01 mark: Fixed sshd on ms4
15:46 mark: Removing 2010-02 snapshots on ms4
15:45 mark: Disabled gmetric cron jobs on ms4
15:43 mark: Disabled daily snapshot generation on ms4
15:27 mark: Restarted gmond on ms4
15:24 mark: Upgraded puppet on ms4
15:13 mark: Powercycled knsq2
14:52 mark: Removing daily snapshots for 2010-10 on ms4
14:24 mark: Restored /etc/sudoers file on DB machines butchered by old versions of wikimedia-raid-utils
05:34 logmsgbot: tstarling synchronized php-1.5/includes/Math.php 'r75909'
04:52 apergos: oh btw, I notice that when / on the squids fills, we don't see it in ganglia, it must report an aggregate or something. it would sure be nice to get notified.
04:18 apergos: lather rinse repeat for sq47, I hope that's all of 'em
03:46 apergos: repeated on sq45...
03:13 apergos: same old story on sq46... restarted syslog, reloaded squid, got back some space on /
02:41 apergos: er... and deleted the log file :-P
02:38 apergos: moved ginormous cache.log out of the way on sg48 and reloaded squid over there since it wasn't done earlier
02:32 apergos: cleaned up / on sq41, restarted syslog, reloaded squid
00:59 logmsgbot: nimishg synchronized php-1.5/wmf-config/InitialiseSettings.php
00:53 logmsgbot: nimishg synchronizing Wikimedia installation... Revision: 75891
00:33 apergos1: also 44 and 43
00:30 apergos1: cleaning up space on other / full squids: sq42

November 2

23:22 apergos: same story on sq50, cleared out some space, tried upping that to 300 but started seeing TCP connection to 208.80.152.156 (208.80.152.156:80) failed in the logs so backed off to 200
23:13 apergos: trying adjusting max-conn on sq49 for conns to ms4... tried 200, it maxed out. trying 300 now...
23:08 apergos: hupped squid on sq49, restarted syslog, / was full from "Failed to select source" errors, cleared out some space
23:08 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'Updating sidebar links'
22:40 apergos: added in the amssq47 through amssq62 to /etc/squid/cachemgr.conf on fenari
19:48 RobH: torrus back online
19:44 RobH: following procedure on wikitech to fix torrus
16:46 RobH: sq42 & sq44 behaving normally now, cleaning cache on sq48 and killing squid for restart as it is flapping and at high load, due to earlier nfs issue
16:38 RobH: restarting and cleaning backend squid on sq44 and sq42 which were complaining in lvs
16:35 RobH: sq43 was flapping since the nfs mount on ms4 was borked. restarted it
16:07 apergos: NFSD_SERVERS=2048 in /etc/default on ms4
16:06 apergos: note that the variables rpcmod:cotsmaxdupreqs has been changed to 2048 in /etc/system, and
15:54 apergos: hard reset on ms4, reboot was not getting the job done
15:47 apergos: rebootint ms4, nfsd hung and couldn't be restarted or killed.
14:04 RobH: restarted pdns on linne due to crash from authdns update
14:02 RobH: updated dns with new mgmt entries for payments, owasrvs, and owadbs
03:45 domas: added srv193 back to apaches pool on lvs

November 1

23:55 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialBannerController.php 'Picking up fixes for Bug #25564'
23:54 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/CentralNotice.php 'Picking up fixes for Bug #25564'
20:42 domas: ms4 mildly loaded (disks go to >100i/s each) throwing nfs timeouts, I bumped up NFSD_SERVERS to 2048
19:05 Ryan_Lane: powercycling srv207
16:18 RoanKattouw: Something weird's going on with srv207: Nagios says its SSH is up but it times out on SSH from fenari
16:15 logmsgbot: catrope synchronized php-1.5/includes/api/ApiBase.php 'r75798'

2000s

Archive 1: 2004 Jun - 2004 Sep
Archive 2: 2004 Oct - 2004 Nov
Archive 3: 2004 Dec - 2005 Mar
Archive 4: 2005 Apr - 2005 Jul
Archive 5: 2005 Aug - 2005 Oct, with revision history 2004-06-23 to 2005-11-25
Archive 6: 2005 Nov - 2006 Feb
Archive 7: 2006 Mar - 2006 Jun
Archive 8: 2006 Jul - 2006 Sep
Archive 9: 2006 Oct - 2007 Jan, with revision history 2005-11-25 to 2007-02-21
Archive 10: 2007 Feb - 2007 Jun
Archive 11: 2007 Jul - 2007 Dec
Archive 12: 2008 Jan - 2008 Jul
Archive 12a: 2008 Aug
Archive 12b: 2008 Sept
Archive 13: 2008 Oct - 2009 Jun
Archive 14: 2009 Jun - 2009 Dec

2010s

2020s