Server Admin Log/Archive 15

June 30

21:44 logmsgbot: andrew synchronizing Wikimedia installation... Revision: 68749
21:44 Andrew: deploying new version of LiquidThreads to labs
17:57 Andrew: running thread_pending_relationship schema update for LiquidThreads on liquidthreads_labswikimedia
17:50 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'New logos for fawiki, arwiki, arzwiki'
17:44 tomaszf: adding generic wikimedia svn depot for display wall project
17:30 RoanKattouw: Rolled back switchover on ukwiki on request
17:29 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Whitespace change for sync-dblist'
17:19 RoanKattouw: Messed up by not doing stuff in the right order; switchover complete now
17:18 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Whitespace change'
17:12 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Whitespace change'
17:11 RoanKattouw: Syncing dblist to make switchover work
17:09 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Switch all wikis in switchover-jun30.dblist to Vector, UsabilityInitiative, new thumb size and new logo'
17:09 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Switch all wikis in switchover-jun30.dblist to Vector, UsabilityInitiative, new thumb size and new logo'
15:57 domas: domas' oprofile killed db40's mysqld :)
15:56 Rob: removing out of band connection from scs-ext and moving to mr1
04:54 logmsgbot: jeluf synchronizing Wikimedia installation... Revision: 68749
02:44 river: stopping ES replication on ms2 to dump for ES backup

June 29

22:22 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
22:20 logmsgbot: catrope synchronized php-1.5/skins/vector/main-rtl.css 'r68749'
22:13 logmsgbot: catrope synchronized php-1.5/skins/vector/main-ltr.css 'r68749'
21:45 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'Bump UI CSS style version'
21:43 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
21:41 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 68745
21:41 RoanKattouw: Running scap to deploy UsabilityInitiative and Vector fixes, for real this time
21:27 RoanKattouw: Strike that, putting the code on test first
21:26 RoanKattouw: Running scap to deploy UsabilityInitiative and Vector fixes
20:46 Fred: removed A record for volunteer.w.o since it was pointed out that it linked to a winamp playlist.
19:14 Fred: removed Naoko from lists to prevent bounces :(
18:46 Fred: removed cron entry from isidore for bz-reporter -> installed mail on Kaulen which will now handle the weekly reports.
17:54 Rob: connected mr1-pmtpa port 0/1 to csw5 port 7/1
17:48 Rob: and msw1-pmtpa:prt24 to msw-b2 (port unimportant)
17:47 Rob: new mgmt connections: mgmt-gateway: serial to scs-ca-pmtpa:port1; mgmt-gateway:0/0 to msw1-pmta:port23, msw1-pmtpa:mgmt to scs-c1-pmtpa port2
16:23 Rob: dataset1 bad disk swapped and array shows rebuilding
15:37 Rob: srv89 resurrected and playing nicely with the other server kids
15:15 Rob: srv146 updated (puppet,sync-docroot,sync-apache), working, and back in lvs cluster
15:09 Rob: fixing srv146, removed from lvs until fixed
08:19 logmsgbot: midom synchronized php-1.5/wmf-config/CommonSettings.php
08:14 logmsgbot: midom synchronized php-1.5/includes/db/DatabaseMysql.php '68713 live'
08:13 logmsgbot: midom synchronized php-1.5/includes/DefaultSettings.php '68713 live'
08:08 logmsgbot: midom synchronized php-1.5/includes/GlobalFunctions.php 'removing msg profiling'
06:00 apergos: restarted morebots on wikitech

June 28

17:43 Rob: ignore sq33 flapping, its disabled in lvs and I am working on it.
16:43 Fred: restarted apache on mobile1. Had locked up for some reason (probably too many threads).
15:57 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/forms/RevisionReviewForm.php 'Deployed r68663'
15:54 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevs.class.php 'Deployed r68663'

June 27

08:12 domas: locke locked up for no apparent reason :(

June 25

18:13 Rob: srv104 disk failure, shutting it down (via ticket to the noc)
15:01 Rob: srv284 has E1410 System Fatal Error, will place ticket later today
14:59 Rob: db28 is borked, needs more replacement memory, reopened ticket, will remain offline for now.
14:59 Rob: fixed srv281 issues, needs reinstallation later today
14:58 Rob: tested db34, LOM is doa, calling in later today
14:03 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Switch private wikis to Vector per Erik M'
13:53 Rob: ignore db28 flapping, working on it.
09:43 domas: updated password formats for wikiuser on s1 and s2
08:20 logmsgbot: midom synchronized php-1.5/wmf-config/mc.php 'swapping out srv104, putting in srv242'
07:38 JeLuF: enabled PHP modules on fenari so that noc.wikimedia.org deliveres syntax highlighted config files.
00:10 logmsgbot: andrew synchronized php-1.5/extensions/AbuseFilter/SpecialAbuseLog.php

June 23

22:18 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php
22:11 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 'Finishing touch'
21:58 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 'Testing'
21:49 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 'Testing'
21:42 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 'Merge r68488'
14:23 domas: 'fixed' abusefilterlog performance issue with a livehack, hehehe: http://p.defau.lt/?55XRFbMETCGrQHRI5a5PDQ
14:22 logmsgbot: midom synchronized php-1.5/extensions/AbuseFilter/SpecialAbuseLog.php
13:49 domas: reduced db9 killing threshold from 30s to 10s :)

June 22

22:55 tomaszf: switching cname of download. from storage2 to dataset1
14:47 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Reenable CollapsibleNav on officewiki per Guillaume'
14:38 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Reenable EditWarning'
14:35 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Modules/EditWarning/EditWarning.i18n.php 'r68406'
14:32 Tim: on mayflower: disabled sshd TCP forwarding
14:40 river: dumping ES to dataset1
14:05 river: stopped MySQL on clematis to move it to slauerhoff
14:03 logmsgbot: midom synchronized php-1.5/wmf-config/CommonSettings.php 'disabled editwarning'
14:03 Tim: upgrading libtiff4 on all apaches due to USN-954-1
13:35 logmsgbot: midom synchronized php-1.5/includes/GlobalFunctions.php 'msg profiling hook'
13:17 mark: (and domas) Restarted DNS on ns0 and ns2
10:21 mark: Depreffed _1299_16265_43821_ route

June 21

18:56 tomaszf: sync'ing public dir from storage2 to dataset1
18:29 tomaszf: importing prod civi database onto dev
12:36 domas: so easy to earn a beer!
12:32 domas: powercycled pdf1, OOM

June 20

16:27 JeLuF: lighttpd issue on lily, it only listens for port 443 on IPv4, not on IPv6, server.use-ipv6 = "enable" is set, it listens to port 80 on both v4 and v6.

June 19

20:42 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24042 - Namespace Aliases for English Wikibooks'
19:10 logmsgbot: catrope synchronized php-1.5/includes/StringUtils.php 'Revert live hack'
18:57 logmsgbot: catrope synchronized php-1.5/includes/StringUtils.php 'Live hack for debugging OOM in '
18:53 JeLuF: fixed broken wikimedia-task-appserver package on fenari. PHP maintenance scripts were no longer running.
08:47 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedArticleView.php

June 18

23:07 atglenn: add pdhanda to core svn committers
22:22 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 68239
22:14 AaronSchulz: deployed r68251
22:14 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 68239
20:52 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
20:52 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js 'r68239'
17:26 logmsgbot: catrope synchronized php-1.5/includes/DefaultSettings.php 'r68222'
17:25 logmsgbot: catrope synchronized php-1.5/skins/vector/main-rtl.css 'r68222'
17:25 logmsgbot: catrope synchronized php-1.5/skins/vector/main-ltr.css 'r68222'
16:40 Rob: updated dns with analytics and donation ip assignments on the tesla dev box
04:44 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version'
03:18 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 'Merge r68206'
03:13 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 'Merge r68204'
03:05 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 'Fix column ambiguity not found on my local install'
03:03 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 'Merge r68202'
02:48 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 'Merge r68200'
02:34 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 'Merge r68198'
02:21 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 'Deploy r68196'
02:20 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy.php 'Deploy r68196'

June 17

21:22 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
21:22 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js 'r68180'
21:09 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
21:09 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js 'r68180'
20:33 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24010 - id.wikipedia requesting FlaggedRevs'
20:27 logmsgbot: jeluf synchronized flaggedrevs.dblist '24010 - id.wikipedia requesting FlaggedRevs'
20:06 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'enabling mobile redirect for eo,ko,krc,xal,ca,mr,ja wiki'
19:34 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'enabling mobile redirect for th,vi,zh,id,hsb,dsb,bn,huw wiki'
19:24 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'enabling mobile redirect for sv,da,et,el,ml,ar wiki'
19:15 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'enabling mobile redirect for cs,eo,he,nn,it wiki'
19:12 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'enabling mobile redirect for pt,nl,fi wiki'
16:33 domas: did a horrible hack to please philippe in LQT.
16:32 logmsgbot: midom synchronized php-1.5/extensions/LiquidThreads/classes/Thread.php

June 16

21:25 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'adding mobile redirect for es,ru,fr"'
20:47 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'adding mobile redirect for de'
16:20 logmsgbot: jeluf synchronized docroot/wikibooks.org/favicon.ico '17049 - New favicon for Wikibooks projects'
15:50 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 24001: Make Vector, usability features default on ukwikimedia'
15:23 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 24001: Make Vector, usability features default on officewiki'
14:43 logmsgbot: midom synchronized php-1.5/wmf-config/flaggedrevs.php
06:17 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '21451 - Change favicon on outreach.wikimedia.org'
05:30 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23988 - Switch the right that crats are able to give and take away to Transwiki Importer instead of Importer'
04:48 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23918 - Enable 'researcher' permission on testwiki'
01:31 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/maintenance/wikimedia-periodic-update.sh
01:29 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'captcha whitelist, bug 23982'
01:27 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'captcha whitelist, bug 23982'
01:26 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php
01:02 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedArticleView.php 'deployed r68112'

June 15

23:47 Tim: on enwiki master, was seeing 14 threads queued for up to 50 seconds waiting to do Title::invalidateCache() in response to FlaggedArticleView::addToDiffView(). Disabled that caller, fixed now
23:43 logmsgbot: tstarling synchronized php-1.5/extensions/FlaggedRevs/FlaggedArticleView.php 'disabling cache update'
23:26 AaronSchulz: enabled Pending Changes on enwiki
23:23 logmsgbot: aaron ran sync-common-all
23:16 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'Get enwiki settings here first'
21:24 AaronSchulz: deployed r68107
21:22 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 66620
18:40 logmsgbot: tfinc synchronized php-1.5/extensions/WikimediaMobile/WikimediaMobile.php
17:51 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 23978: Enable subpages in Outreach namespace on officewiki'
14:57 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'Fixed $wgFlaggedRestrictions on hewikisource. Changed editor -> review; there is no editor right. Probably broken for ages.'
14:41 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'Made autoreview right work without review right present.'
14:25 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'removed clearly redundant $wgFlagRestrictions values'
12:45 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedArticleView.php 'deployed r68068'
12:44 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevs.hooks.php 'deployed r68068'
12:44 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevision.php 'deployed r68068'
11:51 domas: expect higher purging rate for now, as I had to drop corrupted dampen db :)
11:49 domas: restarted dampen via separate udp2log process (the communication between main udp2log and dampen is now async)
11:26 logmsgbot: midom synchronized php-1.5/wmf-config/flaggedrevs.php 'removing talk namespaces from hebrew wikisource flaggedrevs configuration, doh'
11:15 domas: disabled dampen.py/vector purge on locke, as it was blocking on i/o and stopping whole logging pipeline
06:09 JeLuF: fixed "16994 - Normalize *.org portal page URLs to www.*.org"
05:03 logmsgbot: jeluf synchronized php-1.5/wmf-config/flaggedrevs.php '23948 - Remove autoreview bit from editor usergroup on enwikinews'
03:58 AaronSchulz: deployed r68058
03:58 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 66620
03:10 logmsgbot: tstarling synchronized live-1.5/404.php 'more errors'
03:09 logmsgbot: tstarling synchronized 404.html 'fixed spelling error'
03:03 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevs.php 'deployed r68056'
01:08 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 66620
00:55 logmsgbot: aaron synchronized php-1.5/wmf-config/CommonSettings.php 'removed labs msg load hack'
00:54 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'removed wmgUseFlaggedRevsAlpha'
00:53 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'removed labs hacks, uses same FR branch as other sites now'

June 14

23:50 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php
23:39 logmsgbot: aaron ran sync-common-all
23:38 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'switch to _alpha again'
23:28 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 66620
23:28 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'reverted previous switch'
23:25 atglenn: ran /home/wikipedia/bin/l10nupdate for flaggedrevs alpha rolllout
23:06 tomaszf: killing running xml snapshots on snapshot2
23:04 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 66620
23:04 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'switched all wikis to _alpha'
21:36 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23961 - Change Project namescape in Abkhazian wikipedia'
21:21 JeLuF: "apt-get remove mysql-server-5.0" on srv169
21:13 JeLuF: fixed dpkg --configure problems on ~100 servers caused by /var/log/gmetricd/gmetricd.log being owned by root
21:10 tomaszf: rsync enwiki 20100514 from dataset1 to storage2
21:08 tomaszf: installing lighttpd on dataset1
21:00 JeLuF: "apt-get update" was hanging on ca 10 servers, killed them and run puppetd --test afterwards
20:58 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/FlaggedArticleView.php 'deployed r68032'
20:58 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/FlaggedRevs.hooks.php 'deployed r68032'
20:57 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/FlaggedRevs.class.php 'deployed r68032'
20:49 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
20:49 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'r68027'
20:47 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/js2stopgap/jui.combined.min.js 'r68027'
20:47 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r68027'
20:46 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js 'r68027'
20:45 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/images/wikiEditor/toolbar/format-bold-ru.png 'r68027'
20:43 logmsgbot: catrope synchronized php-1.5/skins/vector/main-rtl.css 'r68027'
20:43 logmsgbot: catrope synchronized php-1.5/skins/vector/main-ltr.css 'r68027'
20:43 logmsgbot: catrope synchronized php-1.5/skins/common/wikibits.js 'r68027'
20:43 logmsgbot: catrope synchronized php-1.5/skins/common/shared.css 'r68027'
20:42 logmsgbot: catrope synchronized php-1.5/skins/monobook/main.css 'r68027'
20:41 logmsgbot: catrope synchronized php-1.5/skins/Vector.php 'r68027'
20:09 JeLuF: srv217: dpkg reported broken gmond package, refused further updates. Fixed. Run puppetd --test.
17:35 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '22616 - Create aliases for Romanian namespaces'
16:47 logmsgbot: jeluf synchronized php-1.5/wmf-config/flaggedrevs.php '23948 - Remove autoreview bit from editor usergroup on enwikinews'
16:44 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23953 - Add rollback permission to the patrollers group in the French Wikisource'
15:28 AaronSchulz: deployed r67999 (Flaggedrevs_alpha)
15:27 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 66620
07:31 domas: established 90-day purge policy on locke's archives via /etc/cron.daily/purge-archive
04:04 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/FlaggedRevs.hooks.php 'deployed r67978'
02:49 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'added NS_PROJECT to review namespaces on flaggedrevs_labswikimedia'
02:35 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/forms/RevisionReviewForm.php 'deployed r67978'
02:34 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/FlaggedRevs.class.php 'deployed r67978'
02:04 logmsgbot: tstarling synchronized php-1.5/includes/Html.php 'r67975, disable HTML 5 form validation'
01:48 AaronSchulz: Deployed 67968,67973 (flaggedrevs_alpha)
01:47 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 66620

June 13

23:30 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Remove autopatrol permission from reviewer group on enwiki'
15:39 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23939 - Enable Collection extension on ka.wikipedia.org'
13:39 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 'Merge r67943'
13:32 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 'Merge r67941'
13:32 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy.php 'Merge r67941'
12:15 RoanKattouw: Had JeLuF change l10nupdate to run as catrope rather than brion, should shut up the SSL error and make it easier for me to fix in the future
08:03 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'add reviewer group back to flaggedrevs_labswikimedia'
02:01 logmsgbot: LocalisationUpdate failed

June 12

21:33 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23935 - Autopatroller group for Swedish Wikisource (svwikisource)'
21:28 JeLuF: #23928 - change nyc.wikimedia.org redirect target in redirects.conf
20:19 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23918 - Enable 'researcher' permission on testwiki'
20:00 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '22663 - Add some interwiki links in Special'
19:36 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23644 - Create new user group to disable new article creation'
18:47 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23644 - Create new user group to disable new article creation'
18:26 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23644 - Create new user group to disable new article creation'
18:14 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23644 - Create new user group to disable new article creation'
17:53 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23644 - Create new user group to disable new article creation'
14:13 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 23922: Add reviewer group on enwiki'
04:38 JeLuF: (bug 23912)
04:37 JeLuF: added "linux-libertine" font package to the requirements for image scalers in puppet
02:02 logmsgbot: LocalisationUpdate failed

June 11

17:54 Rob: working on db28, ignore its alerts, it does not serve any production traffic presently
17:08 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23833 - Request for creation/activation of portal namespace'
17:06 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23874 - Change usability wiki sitename to "Wikimedia Usability Initiative"'
17:05 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23883 - Add rights to Reviewer in Testwiki'
16:53 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23892 - double entries in config file for arbcom_enwiki'
13:11 Rob: got email from johannes, pdf2 needed reboot done
06:24 logmsgbot: ariel synchronized php-1.5/wmf-config/InitialiseSettings.php 'en wiktionary flood group tweaks again'
05:57 logmsgbot: ariel synchronized php-1.5/wmf-config/InitialiseSettings.php 'en wiktionary flood group tweaks'
05:29 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'add flood group to en wiktionary, bug 23832'
05:04 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'add commons to en wiktionary import sources, bug 23900'
02:02 logmsgbot: LocalisationUpdate failed

June 10

23:23 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'adding thank you banners to tracking page'
18:00 mark: Setup live purging of all requests for vector enabled wikis, to remove some broken pages of this morning from the cache
14:52 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'fixing wgCollectionArticleNamespaces'
14:51 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 'fixing wgCollectionArticleNamespaces'
09:03 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/PrefSwitch/PrefSwitch.i18n.php 'r67789'
08:59 mark: Purge done.
08:34 logmsgbot: catrope synchronized php-1.5/cache/l10n/l10nupdate-it.cache 'Rebuilt'
08:33 logmsgbot: catrope synchronized php-1.5/cache/l10n/l10nupdate-ja.cache 'Rebuilt'
08:33 logmsgbot: catrope synchronized php-1.5/cache/l10n/l10nupdate-fr.cache 'Rebuilt'
08:33 logmsgbot: catrope synchronized php-1.5/cache/l10n/l10nupdate-de.cache 'Rebuilt'
08:33 logmsgbot: catrope synchronized php-1.5/cache/l10n/l10nupdate-pt.cache 'Rebuilt'
08:33 logmsgbot: catrope synchronized php-1.5/cache/l10n/l10nupdate-pl.cache 'Rebuilt'
08:33 logmsgbot: catrope synchronized php-1.5/cache/l10n/l10nupdate-ru.cache 'Rebuilt'
08:32 logmsgbot: catrope synchronized php-1.5/cache/l10n/l10nupdate-es.cache 'Rebuilt'
08:32 mark: Florida squid purge done, esams cache dump done. Started purging esams pages.
08:22 RoanKattouw: Rebuilding LU cache for es, ru, pl, pt, de, fr, ja, it, (nl already done) to fix weird PrefSwitch messages
08:20 logmsgbot: catrope synchronized php-1.5/cache/l10n/l10nupdate-nl.cache 'Rebuilt, hopefully this works'
08:05 logmsgbot: catrope synchronized php-1.5/cache/l10n/l10nupdate-nl.cache 'Attempt at making i18n behave'
07:54 logmsgbot: LocalisationUpdate failed
07:53 RoanKattouw: Running l10nupdate on hume for real now
07:47 mark: Started purging the Florida squid caches, while the esams squids are still dumping their caches
07:45 RoanKattouw: Running LocalisationUpdate hoping that'll resolve weirdness
07:34 RoanKattouw: Creating prefswitch_survey DB table on all wikis
07:27 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/PrefSwitch/PrefSwitch.i18n.php 'resync'
07:23 mark: Dumping Squid caches for top-9 wikipedias
07:23 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Vector rollout, logo change, thumbnail size change on top 10 wikis sans enwiki'
07:23 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Vector rollout, logo change, thumbnail size change on top 10 wikis sans enwiki'
07:04 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Disable expandablesearch'
06:42 RoanKattouw: Applying extensions/UsabilityInitiative/ClickTracking/patch-additional_info.sql on all wikis
06:35 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'Bump CSS style version'
06:34 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
06:30 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php
06:30 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 66620
06:29 RoanKattouw: Gonna run scap to deploy UsabilityInitiative and Vector updates
02:49 Tim: re-enabled apache and mod_php5 on fenari so that noc.wikimedia.org will work
02:27 Tim: installed wikimedia-task-appserver on fenari, this needs to be present for mediawiki to work correctly
01:03 logmsgbot: tfinc synchronized php-1.5/extensions/WikimediaMobile/MobileRedirect.js

June 9

22:05 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php
22:00 logmsgbot: ariel synchronized php-1.5/wmf-config/InitialiseSettings.php 'tweak wgCollectionNamespaces setting again'
21:49 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'added grants on metawiki to wgCollectionNamespaces'
17:55 Fred: mobile1 reimaged / tested. Putting back in rotation.
17:35 Fred: mobile1 going down for some lucid goodness
17:18 Fred: briefly depooling mobile1 while it get upgraded to Lucid.
01:28 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 'enabling WikimediaMobile on enwiki'
01:20 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'fixed mobile redirect again'
01:10 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 'enabling WikimediaMobile on plwiki'
01:00 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'fixed error in $wgWikimediaMobileUrl'
00:49 AaronSchulz: deployed r67676, r67677 (Flaggedrevs_alpha)
00:46 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 66620

June 8

23:27 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'Adding global support page to tracking'
23:05 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/ContributionTrackingStatistics_body.php 'adding range support to Special:ContributionTrackingStatistics'
22:36 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'turning off fr redirect till fix is in place'
22:08 logmsgbot: fvassard synchronized php-1.5/wmf-config/InitialiseSettings.php 'enabling mobile redirect on FRwiki'
21:34 Fred: mobile LVS setup. Traffic to mobile now balanced between mobile1 (30%) and mobile2 (70%)
19:58 Rob: updated civicrm certificate file, had to apache2ctl restart on grosley
19:03 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Add .omniplan to extension whitelist for private wikis. Requested by Pete'
18:56 Fred: new wikimedia-task-dns-auth deployed to nameservers. langlist regenerated. *lang*.m.wikipedia.org is now cnamed to m.wikimedia.org
16:41 Rob: pushed updates to wordpress plugins on singer
16:40 Fred: I foobared. need to redirect traffic to mobile2 so mobile1 can be re-imaged.
15:20 Rob: updated singer and dns records for austin's setup of wm10reg
01:29 Tim: running throughput test on mobile2

June 7

23:12 Fred: modified puppet receipe for mobile to allow for easier passenger gem update.
22:04 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
22:04 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js 'r67568'
20:55 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'rollback for de_labs editors'
19:46 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialNoticeTemplate.php 'Picking up fix for tmp_id'
19:40 logmsgbot: catrope synchronized php-1.5/extensions/CentralNotice/SpecialNoticeTemplate.php 'r67558'
19:12 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialNoticeTemplate.php 'Picking up pagination from r67553'
18:56 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialNoticeText.php 'Picking up pagination from r67553'
16:57 Fred: Limiting message size to 5M for lists.
16:43 Tim: moved mediawiki-cvs archive files away to prevent further archiving
16:26 Tim: disabled archiving on problematic list mediawiki-cvs
14:07 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php
14:04 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php
13:49 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php
13:48 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php
13:46 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php
07:43 Tim: on lily: fixed broken sources.list and upgraded python, installed python symbols
07:25 Tim: on lily: restarted mailman (including ArchRunner)
07:07 Tim: on lily: there are three instances of ArchRunner which are in a tight loop using CPU but not doing any syscalls, presumably dead. Renicing them, will try to make backtraces
03:41 Tim: srv281 back up and resynced. However it crashed after being up only 3 days and shows more machine check errors. Suggest RMA.
03:25 Fred: setup ganglia for lily.
03:07 Tim: srv281 down for 41 hours, trying reboot
02:54 Fred: upgraded gmond on lily. Modified gmond.conf to handle new modular arch.
02:46 Tim: on lily: experimentally disabling broken bayes feature in spamassassin
02:35 Tim: on lily: restarting spamd and fixing lock file errors
01:25 Tim: on mobile1: Reduced PassengerMaxRequests to 5000 to reduce memory leakage

June 6

11:33 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'r67456'
11:33 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/css/combined.min.css 'r67456'
07:33 Tim: on mobile1: use PassengerMaxRequests instead of PassengerPoolIdleTime, to avoid oscillation between 200 and 30 processes, once every half an hour due to unknown slowdown
03:10 Tim: serve a 404 error for requests to the mobile server for domains other than *.m.wikipedia.org. DNS points here for lots of domains.
02:14 Tim: installed a redirect from en.m.wikipedia.com to .org
01:18 Tim: changed the mobile1 apache access log format to something more useful (and squid-like)
00:47 Tim: on mobile1: installed logrotate script for apache2 (via puppet)

June 5

21:21 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'fr_labs config cleanup'
21:15 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php
20:55 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'fr_labs config cleanup'
03:29 Tim: fixed mobile1 in nagios, added mobile2
02:09 Tim: restarting gmond on all miscellaneous cluster servers, to make mobile2 reappear in ganglia (broken by IP renumber)

June 4

22:52 tomaszf: starting webstats with new binary
22:50 tomaszf: stopping webstats in prep for update to track mobile stats
19:30 atglenn: moved bad snapshots (apr 11 through may 6 2010) to /mnt/dumps/public/bad so public index shows only good dumps and so there will be no prefetch against them
18:47 Fred: moved mobile2 to squid vlan / re-ip'ed / dns changed. mobile1 => 115 mobile2 => 116
18:35 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix. Gotta kill this thing some time'
18:35 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js 'r67355'
18:34 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r67355'
12:11 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 'WikimediaMobile'
11:37 Tim: mobile down for 15 minutes, possibly apache threads exhausted, restarting apache
09:56 logmsgbot: catrope synchronized php-1.5/extensions/ContactPage/SpecialContact.php 'r67333'
09:56 domas: deployments manage to kill apache processes sometimes
09:50 logmsgbot: tstarling synchronizing Wikimedia installation... Revision: 66620
09:50 Tim: pushing out WikimediaMobile (r67331) in preparation for deployment on testwiki
08:44 domas: decreased keepalivetimeout and timeout on mobile1
08:35 Tim: on mobile1: reduced max passenger pool size to 200, Domas and I think it's about right, shouldn't exceed allowable memory, should give us close to 100% CPU.
08:26 Tim: on mobile1: domas fixed file limit, now 50k
08:10 Tim: increasing MaxClients on mobile1 to 1500
05:01 Fred: Added apache2.conf, memcached.conf to puppet receipe for mobile.
03:43 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23784 - Modify add/remove rights for bureaucrats on officewiki'
02:46 Tim: mobile1: increased ServerLimit to 1500 and reduced MaxClients to 500
02:35 Tim: on mobile1: increased memcached memory limit from 64M to 5000M
02:15 Tim: switched mobile1 over from apache2-mpm-worker to apache2-mpm-prefork (via puppet)
01:03 Tim: set ganglia host_dmax to 1 day

June 3

21:57 Fred: mobile1 re-imaged and puppetized. Changed subnet for mobile1. Changed DNS for mobile1. m pointing to newly imaged mobile1 (until transition is completed)
20:55 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23689 - Enable Collection extension on Thai Wikipedia'
20:22 AaronSchulz: deployed r67296 FlaggedRevs_alpha
20:21 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 66620
19:39 mark: Moved mobile1 switchport from vlan 101 to 100
19:36 mark: Reverted DNS change of mobile1, back to .157
17:21 Fred: mobile1 going to be unreacheable while re-ip'ing
14:05 logmsgbot: midom synchronized php-1.5/wmf-config/InitialiseSettings.php 'timezone change for bat-smg'
11:53 mark: Made m.wikipedia.org CNAME m.wikipedia.org, m.wikipedia.org A to mobile1/2 in RR
10:57 hcatlin: mobile2 has been rebuilt and is featuring the new apache/mobile stack taking 40% of all mobile traffic. pls help monitor on ganglia.
09:04 Tim: cleaning COSS on sq45, resynced its configuration, will start squid when done
08:58 Tim: kernel reports degraded RAID on sq33, sq34, sq35, sq37, sq38, sq40
08:39 Tim: checked all serial consoles, all nonresponsive, rebooted all
08:23 Tim: sq33, sq34, sq35, sq37, sq38, sq40, sq45 have been down for 16-28 days, apparently for no good reason, can't find any log or DT entries. Will try reboots.
07:56 Tim: added new squids to nagios
06:36 Tim: cleaning cache directories on sq56 to avoid resurrection of expired content
06:35 Tim: adding monitoring for rather important service IPs: upload.esams and text.esams
06:22 Tim: sq56 not responding to ping or serial console (for 4 days), nothing in racadm getsel, rebooting
06:07 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 'disabling ClickTracking due to CR r58099'
05:24 Tim: started apache on srv216, was stopped for some reason
03:57 Fred: shutting down mailman on list for a few minutes while exim and spamd catch up
01:42 Tim: adding forward and reverse DNS for mobile.tesla.usability.wikimedia.org, 208.80.152.245

June 2

23:48 Fred: re-imaging mobile2 again ;p
21:12 logmsgbot: robh ran sync-common-all
20:57 Rob: srv134 has temp errors and multiple bad fans, out of warranty, decommissioning and removing from nagios/lvs/dsh
20:52 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23756 - Enable "Rollbacker" group on itwiki'
20:44 Rob: how many ops does it take to bring srv281 back online? one + a vounteer to point out he did it wrong.
20:42 logmsgbot: robh ran sync-common-all
20:36 logmsgbot: robh ran sync-common-all
20:26 logmsgbot: jeluf synchronizing Wikimedia installation... Revision: 66620
20:15 Rob: srv281 back online and apache files updated
20:08 Rob: lomaria is down, because it was throwing crap around and generally broken, so its shutdown now.
19:59 Rob: had to swap power cable on msw-b5-sdtpa
14:02 Rob: stay off db28, fixing its firmware.
13:44 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php
13:39 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php
13:38 Tim: ArchRunner on lily was stuck in a busy loop of some kind, had to restart it with kill -9
13:36 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php
13:35 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php
13:32 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php
13:32 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ParserFunctions.php
13:31 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy.i18n.php
13:31 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy.php
06:49 logmsgbot: tstarling synchronized php-1.5/extensions/CodeReview/backend/Subversion.php 'r67213'
06:45 Tim: updated codereview-proxy on isidore to r67211
06:13 Tim: attempting to set up CodeReview for pywikipedia
05:48 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23705 - Change namespaces in ja.wikipedia'
05:33 logmsgbot: jeluf synchronized php-1.5/wmf-config/flaggedrevs.php '23742 - make category namespace flaggable on en.wikinews'
05:29 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23572 - Imagemap cannot accept dynamic links.'

June 1

23:10 Fred: mobile2 re-installed and puppetized
22:44 Fred: re-imaging mobile2 for total lucid puppetization.
20:17 Rob: mobile1 decided no more ssh server, its reinstalled and defined in puppet so it is ok now.
19:42 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesCkb_arab.php 'r67198'
14:26 logmsgbot: catrope synchronized php-1.5/extensions/CategoryTree/CategoryTreeFunctions.php 'r67183'
14:10 domas: livehacked the CategoryTreeFunctions.php to fix a fatal error :)
14:09 logmsgbot: midom synchronized php-1.5/extensions/CategoryTree/CategoryTreeFunctions.php
12:02 logmsgbot: tstarling synchronizing Wikimedia installation... Revision: 66620
12:01 Tim: updating CategoryTree to r67181
05:56 logmsgbot: midom synchronized php-1.5/includes/db/Database.php
05:52 logmsgbot: midom synchronized php-1.5/extensions/CategoryTree/CategoryTreeFunctions.php

May 31

20:45 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23646 - Flagged Revisions on Tamil Wikinews (ta.wikinews.org)'
20:44 logmsgbot: jeluf synchronized flaggedrevs.dblist
20:39 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23705 - Change namespaces in ja.wikipedia'
19:52 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23728 - Local time for Croatian Wiktionary'
19:45 JeLuF: sync-apache doesn't update /etc/apache2/apache2.conf on all hosts, two different versions are currently in use. Probably due to missing /home NFS mount on some apaches
19:17 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23737 - Enabling extension AbuseFilter for Upper Sorbian and Lower Sorbian Wikipedia'
19:13 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '15063 - wpSpamRegex entry for large image tables'
19:08 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23728 - Local time for Croatian Wiktionary'
09:36 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable $wgEnotifWatchlist on outreachwiki'
06:57 Tim: installed NRPE on db22
06:53 Tim: installed NRPE on db14
06:51 Tim: installed NRPE on db16
06:24 Tim: db16 has the same problem, killed 6 times today, taking the same action (27GB to 24GB)
06:16 Tim: mysql on db14 has been killed 4 times today by the OOM killer, reducing innodb_buffer_pool_size from 28GB to 24GB
06:13 Tim: mysql on db14 killed by kernel OOM killer, recovered without intervention

May 30

16:03 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaLicenseTexts.i18n.php 'r67084'
16:02 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaLicenseTexts.php 'r67084'
16:02 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaMessages.i18n.php 'r67084'
16:02 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaMessages.php 'r67084'
15:48 logmsgbot: catrope synchronized php-1.5/wmf-config/ExtensionMessages.php 'Regenerated'
15:47 RoanKattouw: Removed nonexistent ActiveTaskForces.php from extension-list to unbreak it
15:46 logmsgbot: catrope synchronized php-1.5/extensions/ContactPageFundraiser/SpecialContact.php 'r67081'
15:45 logmsgbot: catrope synchronized php-1.5/extensions/ContactPageFundraiser/ContactPage.php 'r67081'
15:36 logmsgbot: catrope synchronized php-1.5/extensions/ContactPage/ContactPage.i18n.php 'Resync, didn't seem to be picked up right'
15:32 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22741: Enable $wgContactIncludeIP on nlwiki'
15:17 logmsgbot: catrope synchronized php-1.5/extensions/ContactPage/SpecialContact.php 'r67075'
15:16 logmsgbot: catrope synchronized php-1.5/extensions/ContactPage/ContactPage.i18n.php 'r67075'
15:16 logmsgbot: catrope synchronized php-1.5/extensions/ContactPage/ContactPage.alias.php 'r67075'
15:15 logmsgbot: catrope synchronized php-1.5/extensions/ContactPage/ContactPage.php 'r67075'
08:38 Tim-away: restarted memcached on srv194

May 29

23:17 apergos: started third worker on snapshot3. I'm going to stop logging these now.
16:15 apergos: started second worker doing xml dumps on snapshot3 (same screen session as root)
03:57 apergos: xml dumps resumed: running one copy of "worker" on snapshot3, in a screen session as root, along with "monitor" modified in place to not meddle with lock files (so that it leaves the enwiki dump, in progress on snapshot2, undisturbed)

May 28

23:37 atglenn: taking snapshot3 out of sync updates for a few days (commented out in mediawiki-installation node file)
21:55 JeLuF: creating new captcha images on hume
16:54 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23486 - Enable Collection on gdwiki'
13:35 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php
13:30 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ParserFunctions.php 'Missing svn add'
13:27 logmsgbot: andrew synchronizing Wikimedia installation... Revision: 66620
13:27 Andrew: running scap to deploy strategywiki extension updates
13:23 logmsgbot: andrew synchronized php-1.5/includes/specials/SpecialRevisiondelete.php
09:00 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'moving over db16 to commons duty, db14 to dewiki, reenabling db4 and db5'
07:53 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'removing db5 and db4 for copies'
07:11 logmsgbot: tstarling synchronized php-1.5/includes/Sanitizer.php 'r66994'
06:31 domas: upgrading LOM/BIOS/OS on db14,db16,db22 to 3.0/10.4
02:41 Tim: srv92 locked up for a few minutes with syslog showing SATA errors. May be near death.
01:17 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 're-enabled CategoryTree, unless there is some emergency, you can't just disable extensions like this'

May 27

21:35 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable CollapsibleNav on outreachwiki'
21:22 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php
21:09 AaronSchulz: deployed r66980 (Flaggedrevs_alpha)
21:07 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 66620
20:24 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable Vector and usability features on outreachwiki and foundationwiki'
20:17 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable CollapsibleNav for beta users on all wikis'
20:17 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Enable SimpleSearch for beta users on all wikis'
11:59 domas: opened a new sub-categories related bug at https://bugzilla.wikimedia.org/show_bug.cgi?id=23682 ;-)
11:30 logmsgbot: midom synchronized php-1.5/wmf-config/CommonSettings.php 'disable category tree'
04:13 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'vector as default skin for flaggedrevs_labswikimeida'
02:40 Tim: ran namespaceDupes.php with suffix on jawiki to fix talk namespace duplicates

May 26

23:59 Rob: updated blog to vector earlier today, didnt log... just pushed another update to the theme, blog is still happah
21:43 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'updated foundationwiki default skin from monobook to vector'
17:02 Tim: on mobile1 nginx: set proxy_next_upstream=off to avoid having the site go down regularly for 10s at a time
16:30 Tim: changing mobile1 nginx log format
06:39 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23667 - Local time for Macedonian Wikipedia'
03:39 Tim: set up SSLCertificateChainFile on williams (otrs) per bug 23631, used the cut down version of gd_bundle.crt posted on the bug report
02:08 logmsgbot: andrew synchronized php-1.5/includes/LogEventsList.php
02:07 logmsgbot: andrew synchronized php-1.5/includes/specials/SpecialRevisiondelete.php
02:07 logmsgbot: andrew synchronized php-1.5/languages/messages/MessagesEn.php
02:05 Andrew: Deploying r66793 and r66823, first to test.wikipedia.org for final testing, then to all sites.

May 25

17:02 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Re-add style version appendix now that borked usability JS was purged from Varnish'
12:51 domas: what happened - server goes down, configuration gets updated, server comes up, builds new cache out of old conf, sync has timestamp older than cache, mw has stale config
12:47 domas: cleaned /tmp/mw-cache for some servers - apparently sync-common is not enough to get stuff in sync anymore :) we have now monobook pages in caches, I guess.
12:26 rainman-sr: freed up some space and restarted indexing on searchidx1, indexing was stuck because the disk was full
11:16 mark: Downgraded puppet on db36
06:56 domas: resynced srv206/srv193 - seems to have lost critical configuration updates while they were down. Again, why servers are not resynced after downtime?
01:24 Tim: fixed redirect from http://racktables to https://racktables, was broken due to IP address change
01:15 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'removed cruft from wgStyleVersion'
01:12 logmsgbot: tstarling synchronized php-1.5/skins/common/wikibits.js 'r66861'
00:19 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'removed Editor group from flaggedrevs_labswikimedia'
00:02 AaronSchulz: unhide the username of edits by "The Thing That Should Not Be" after accidental account locking (and unlock didn't work)

May 24

15:01 Rob: sq39 will stay offline for a bit, have a configuration concern with replacement disk performance.
15:01 Rob: pushed updates to srv206 and srv193, they should be back online and ok now.
14:54 Rob: sq39 has disk replaced and reinstalled, but its failing squid backend due to configuration. pulling out of rotation to look at later today.
14:51 Rob: working on srv206 and srv193
14:21 Rob: replaced bad disk in sq39, reinstalling.
12:16 domas: db32, db36, db38 and db40 are now serving enwiki (+ db12 and db26 )
12:15 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
09:25 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'db22 being removed too'
09:23 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'removing db14, db16 from enwiki duty, will host different clusters, db26 upgrade'
09:17 domas: master switch db16->db36, new master position - db36-bin.000001:106
09:16 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
09:16 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
09:09 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
09:06 domas: srv141 was in prod, but not in mw_inst
07:24 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
07:09 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
07:06 domas: srv134 was having i/o issues, probably bad disk, rebooted for it to stop flapping, sorry Jens :)

May 23

15:12 apergos: turned off overcommit on bayes (left overcommit ratio at default = 50, let's see how that goes)
14:57 apergos: rebooted bayes after OOM
13:18 domas: forward-ported (cp from hardy to wikimedia repo ;-) libstdc++5 to unbreak wikimedia-raid-utils and few other things. \o/
12:40 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
08:01 JeLuF: increased size of filesystem /var/backup on mchenry by 10 GB
06:44 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23545 - Rights changes for frwiktionary'
06:39 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
06:00 domas: reinstalled bunch of new DBs with lucidlynx

May 21

22:53 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23545 - Rights changes for frwiktionary'
19:36 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Use separate Apple Touch icon for testwiki'
18:17 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
18:16 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r66729'
16:33 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
16:33 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r66726'
16:18 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
16:17 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Resync, seems this didn't get updated properly last time'
09:53 RoanKattouw: Deployed new favicon and apple toch icon on test; overwrote the latter locally on srv124, will be overwritten by scap until we change the wikipedia.org-wide default

May 20

21:31 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'Bump CSS style version'
21:31 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
21:02 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 66620
21:01 RoanKattouw: Running scap to deploy r66708
19:50 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23596 - Add the 'Vikiprojekts' namespace and a related 'Vikiprojekta diskusija' namespace to the Latvian Wikipedia'
19:36 AaronSchulz: deployed r66704 (Flaggedrevs_alpha)
19:28 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 66620
19:02 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/PrefSwitch/PrefSwitch.i18n.php 'r66701'
17:25 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '20074 - Add an Apple Touch icon for MediaWiki.org'
17:21 logmsgbot: jeluf synchronized docroot/mediawiki/Apple-touch-icon.png
13:55 mark: Unshut BGP session to 13030, and reenabled announcement of our prefixes on the session
13:41 mark: Changed IP address of text.esams.wikimedia.org (i.e. the main IP address of our wiki farm at esams) from 91.198.174.2 to 91.198.174.232
13:14 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveTaskForces/ActiveTaskForces_body.php
13:11 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveTaskForces/ActiveTaskForces_body.php 'Testing'
13:09 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveTaskForces/ActiveTaskForces_body.php 'ActiveTaskForces: Merge r66686'
13:06 mark: Added new LVS service IPs to the older esams text squids
10:53 mark: Removed /etc/cron.hourly/mw-cleanup-tmp on all appservers

May 19

21:34 mark: Manually updated countries.nerd.dk databases with new versions on the auth dns servers
20:16 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23568 - Allow bureaucrats to grant and remove patroller, autopatroller, rollbacker on eswiktionary'
20:11 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23592 - Disable creating new articles for IP-users in ckbWikipedia'
20:07 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23576 - Alias of Potal talk namespace on jawiki, jawikinews'
15:39 Rob: mchenry's drive replaced, it will start auto mirroring shortly
15:33 Rob: pulled mchenry's hot swap spare to put in a 1.5TB for migration
15:33 Rob: welcome back morebots

May 18

20:46 Rob: got distracted, but mailman is back online now.
20:43 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23545 - Rights changes for frwiktionary'
20:23 Rob: have to pull a message from a list, stopping mailmain on lily to edit the mbox (will restart and log its restart shortly)
19:50 logmsgbot: catrope synchronized php-1.5/includes/DefaultSettings.php 'r66618'
19:50 logmsgbot: catrope synchronized php-1.5/skins/vector/main-rtl.css 'r66618'
19:49 logmsgbot: catrope synchronized php-1.5/skins/vector/main-ltr.css 'r66618'
19:41 atglenn: added jeluf to core committer group on mayflower svn (why wasn't he there? who knows)
18:22 RoanKattouw: Started morebots, didn't come back up when wikitech did
18:21 RoanKattouw: <logmsgbot> robh ran sync-common-all
18:21 RoanKattouw: <logmsgbot> catrope synchronized php-1.5/languages/messages/MessagesRo.php 'r66615'
14:32 Andrew: 13:50 <+logmsgbot> !log andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'Activate single-revision deletion for admins on all projects'
03:45 logmsgbot: tstarling synchronized php-1.5/maintenance/addwiki.php

May 17

23:38 logmsgbot: aaron synchronized php-1.5/StartProfiler.php 'gave flaggedrevs_labswikimedia a profiling ID'
20:48 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23569 - Change name of Portal talk namespace on jawiki, jawikinews'
20:40 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'touch'
20:40 logmsgbot: jeluf synchronized php-1.5/wmf-config/abusefilter.php '22008 - Abusefilter-log-detail for Arabic Wikipedia'
20:35 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23562 - New Namespace for ko.wikisource'
20:20 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23404 - Create new namespaces on trwikibooks'
20:14 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23328 - New namespace requested on fr.wikt'
20:10 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '22921 - New namespaces and aliases for ml.wikibooks'
19:50 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23244 - Enable FlaggedRevs/patrol extension on mkwiki'
19:42 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23351 - Please remove the 'reviewer' user group from en wikinews if possible'
19:41 logmsgbot: jeluf synchronized php-1.5/wmf-config/flaggedrevs.php '23351 - Please remove the 'reviewer' user group from en wikinews if possible'
19:29 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23488 - Add rename user group on Commons'
19:22 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23545 - Rights changes for frwiktionary'
16:52 logmsgbot: catrope synchronizing Wikimedia installation...
16:51 RoanKattouw: Running scap to deploy r66569
09:17 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Reenable SimpleSearch in config (still disabled in JS), bump style version appendix'
09:16 RoanKattouw: Commented out srv141 in /etc/dsh/mediawiki-installation to stop sync-file from hanging forever
09:15 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js 'Disable SimpleSearch in JS with if(false)'
08:41 domas: banned a sucker via badbadip
08:22 domas: site had some overload for ~15mins, two memcached/app servers didn't recover afterwards (I rebooted srv181, srv141 is still in oom or some other limbo)
08:13 logmsgbot: midom synchronized php-1.5/wmf-config/mc.php
05:24 logmsgbot: ariel synchronized wmf-deployment/wmf-config/InitialiseSettings.php 'enable lqt on en wikt, per bug 23417'

May 16

18:25 JeLuF: fixed doxygen cronjob on mayflower. Also added logfile rotation
15:04 mark: sq39 died due to missing drive /dev/sdb
11:11 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23543 - Add download as PDF in ru.wikiversity'
10:45 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'typo'
06:54 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23396'
06:43 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23481 - Add MediaWiki, Wikimania wikis to the "wikimedia" project in CentralNotice'
06:26 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23531 - Turn on $wgVectorShowVariantName for zhwiki'

May 15

21:44 JeLuF: added nyc.wikimedia.org to DNS and redirects.conf
19:32 mark: Changed IPv4 address of upload.esams.wikimedia.org from 91.198.174.3 to 91.198.174.234
18:02 logmsgbot: catrope synchronized php-1.5/includes/api/ApiQueryAllUsers.php 'r66486'
16:50 mark: Emergency switch of upload LVS (91.198.174.3) to amslvs4
16:37 mark: Removed amssq61 from the LVS pool as "it's weird"
13:53 mark: Pooled new upload squids amssq47-62 with normal LVS weight 30
13:37 mark: Pooled new upload squids amssq47-62 with normal LVS weight 12
13:32 mark: Pooled new upload squids amssq47-62 with low LVS weight 1
13:20 mark: Pooled new upload squids amssq47-62 with low CARP weight 1
12:01 mark: Fixed an issue with puppet on all hosts, trying to import ssh keys with an empty title
11:56 mark: Upgraded server sockpuppet to 10.04 LTS
10:20 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Disable SimpleSearch temporarily until bugs are resolved'
05:50 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23502 - Install Quiz on French Wiktionary'

May 14

15:22 mark: Installed Lucid on amssq47..62
12:14 river: restarted pdns on ns1
11:54 domas: (and mark) fixing derailed squid conf for ~16 servers that went down

May 13

22:09 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
22:08 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js 'r66404'
22:08 logmsgbot: catrope synchronized php-1.5/skins/Vector.php 'r66404'
21:31 mark: Setup amslvs2 and amslvs4, created fenari:/home/w/conf/pybal/esams/ for having a central location where PyBal can fetch realserver lists from
19:49 logmsgbot: catrope synchronized php-1.5/skins/Vector.php 'r66381'
19:38 mark: Downgraded puppet on server alsted (running 10.04 LTS); the puppet in 10.04 is too new
17:43 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix for reports of weirdness'
17:24 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Hide template pref for real'
17:22 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Remove template collapsing pref'
14:38 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/FlaggedRevsXML.php
14:32 mark: Copied the standard Squid error messages over the Wikimedia ones on API squids sq31 and sq32, until we upgrade them
14:22 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/FlaggedRevsXML.php
13:55 AaronSchulz: deployed r66342 (Flaggedrevs_alpha)
13:52 logmsgbot: aaron synchronizing Wikimedia installation...
12:18 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/PrefSwitch/PrefSwitch.i18n.php 'r66336'
12:17 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
12:17 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r66336'
12:16 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js 'r66336'
10:41 mark: Increased purge sample rate to 1
10:33 mark: Increased purge sample rate to 1/5
08:46 RoanKattouw: refreshCategoryCount.php running nicely on Commons now, should finish in about 2 hours
08:29 RoanKattouw: Running refreshCategoryCount.php on Commons again. Hacked script to print more progress markers, hacked Category.php to not do LOCK IN SHARE MODE
08:23 RoanKattouw: Nope, that didn't work
08:22 RoanKattouw: Attempting to run refreshCategoryCounts.php on Commons, last attempt failed with DB deadlock almost immediately
08:04 mark: Sample rate of purging increased to 1/10
08:02 mark: First stage of purging (all squid objects with known URL) done, now enabling live purging of requested enwiki pages, with 1/100 sample rate
07:38 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/PrefSwitch/PrefSwitch.i18n.php 'r66327, r66329'
07:10 mark: Upped speed of purge rate limiting to approx 1500 purges/s (running on a screen on locke)
07:02 mark: Started purging enwiki squid pages
06:58 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'r66325'
06:57 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/css/combined.min.css 'r66325'
06:52 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Change logo URL for enwiki to point to new logo'
06:45 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
06:14 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable Vector, usability features by default on enwiki'
06:08 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Enable SimpleSearch on Vector-default wikis, bump style version appendix'
06:05 mark: Restarted apache on srv159
06:02 logmsgbot: catrope synchronizing Wikimedia installation...
06:01 RoanKattouw: Running scap to deploy Vector&UsabilityInitiative updates
05:34 mark: /dev/sdb on knsq2 has 20 uncorrected errors, about to die
05:28 RoanKattouw: Deploying r66322, r66323 on test.wikipedia.org

May 12

18:44 Rob: new apaches seem to be handling things just fine
18:34 Rob: new apaches are online, pooled one with no issues, so just pushed the rest (srv258, then srv259-srv289)
12:09 JeLuF: There are reports about issues with .de domains, DENIC seems to have problems with the DNS server for .de
07:01 Tim: running SecurePoll voter list script for checkuser and oversight elections, in screen on hume

May 11

22:37 Rob: adding new hosts to nagios, prepare for flapping from them (but they are not in service so can be ignored for now)
21:55 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '22139 - Install Labeled Section Transclusion on wikt'
21:17 logmsgbot: robh synchronized php-1.5/wmf-config/CommonSettings.php 'added back with proper syntax'
21:12 logmsgbot: robh synchronized php-1.5/wmf-config/CommonSettings.php 'revert'
21:11 domas: unbreaking Rob's change.
21:10 logmsgbot: midom synchronized php-1.5/wmf-config/CommonSettings.php 'lol'
21:05 logmsgbot: robh synchronized php-1.5/wmf-config/CommonSettings.php 'adding file extensions for foundationwiki per cary'
20:09 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php
20:08 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23462 - please allow Crats to add/remove the Transwiki importer right on SimpleWiki'
19:55 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php
19:43 RoanKattouw: Running refreshCategoryCounts.php on the remaining wikis, 6 parallel instances on hume
19:38 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php
19:34 AaronSchulz: cleared out de_labswikimedia pages
19:12 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php
17:32 mark: Moved bits.esams.wikimedia.org to 91.198.174.233 in DNS. In case of problems, move back to .2
16:50 mark: Allocated 4 out-of-subnet service IPs for LVS, to test whether this confuses the foundry RXes less
13:21 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 23374'
13:02 mark: Set up an LVS service for bits.esams.wikimedia.org (91.198.174.8), and changed it in DNS. Currently served by new server amslvs1
12:18 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'configured de_labswikimedia like dewiki'
10:24 mark: Installed Lucid on amslvs1-4
01:20 tomaszf: starting new logstream of pagecounts to dataset1

May 10

23:05 atglenn: power cycled pdf2 from the drac, was not responding to ping
22:17 RoanKattouw: pdf2 down, pinging from fenari yields Host unreachable
22:06 RoanKattouw: Stopped refreshCategoryCounts.php run about an hour ago, will resume tomorrow
20:32 JeLuF: running 4 instances of populateRevisionLength.php in parallel
20:17 RoanKattouw: Now running 8 parallel instances of refreshCategoryCounts.php on hime
20:04 RoanKattouw: Running refreshCategoryCounts.php on all wikis on hume, two instances in parallel
19:12 JeLuF: running populateRevisionLength.php on all wikis (bug 12188)
18:15 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
18:14 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js 'r66149'
11:30 hcatlin: deployed new UDP logger for mobile. someone needs to keep an eye out on the parser!
09:36 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Fix Portail namespace on frwikisource'
07:44 mark: Trippled CARP weight on the new ams squids
05:21 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '22663 - Add some interwiki links in Special:Import of fr.wiktionary, fr.wikibooks'

May 9

22:41 mark: Pooled amssq31-46 in LVS with weight equal to the older squids
22:23 mark: Pooled amssq31-46 in LVS with weight 1
20:25 mark: Installed Lucid on amssq32-46
17:12 mark: Pooled amssq31 in CARP with low weight
16:38 mark: Pooled amssq31 in LVS
15:30 apergos: added respository dumps link to index.html on mayflower
15:25 logmsgbot: mark synchronized php-1.5/wmf-config/CommonSettings.php 'Add amssq31-62 to the proxies list'
12:43 mark: Setup OSPF on the esams management network
09:44 apergos: added directory for demon's weekly svn dumps to apacke config on mayflower

May 8

23:14 mark: Setup quick hack on brewster to proxy puppet connections from esams to sockpuppet
20:04 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php ' Bug 23317 - Enable subpages for Template namespace at English Wikibooks'
19:57 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23251 - Turn on subpages in main namespace of Wikimedia Norge's wiki'
19:31 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23389 - Disable anonymous edits on chapterwiki no.wikimedia'
19:24 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23390 - Namespace alias VP -> NS_PROJECT on eowiki'
19:17 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23392 - Allow bureaucrats to de-sysop on English Wikinews'
19:10 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23409 - Add Portail namespace for French Wikisource'
19:04 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23418 - Change English Wiktionary logo'
18:59 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23410 - Set logo on Bhojpuri Wikipedia'
18:54 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23423 - Enable Collection extension in Upper Sorbian and Lower Sorbian Wikipedia'
18:51 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23434 - Activation of the extension Collection on eowiki'
18:49 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23437 - Collection extension on Ukrainian Wikipedia'
18:44 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23357 - Disable Login for German ArbCom wiki'
10:30 RoanKattouw: Upped $wgMaxShellMemory to 200M (from 100M) on wikitech, unbreaks ImageMagick
02:12 Andrew: Forced query page update on strategywiki, added ActiveTaskForces to extension-list
02:11 logmsgbot: andrew synchronizing Wikimedia installation...
02:08 logmsgbot: andrew synchronizing Wikimedia installation...
01:02 logmsgbot: andrew synchronizing Wikimedia installation...
01:02 Andrew: Per Eugene's request, installing ActiveTaskForces on strategywiki

May 7

22:15 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
22:09 RoanKattouw: My name is morebots and I'm alive and well
22:05 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
21:58 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
21:47 RoanKattouw: Switched default skin on wikitech to Vector again
21:46 RoanKattouw: Upgraded mwclient on wikitech for login CSRF fix
20:42 RoanKattouw: Testing morebots
20:23 RoanKattouw: Installed JPEG delegate for imagemagick on wikitech, now reinstalling imagemagick
20:21 RoanKattouw: Installed JPEG delegate for imagemagick on wikitech, now reinstalling imagemagick
20:21 RoanKattouw: Enabled PdfHandler on wikitech
20:11 RoanKattouw: Enabled PdfHandler on wikitech
20:07 RoanKattouw: Enabled PdfHandler on wikitech
19:57 RoanKattouw: Switching wikitech to 1.16wmf4, was running wmf-deployment
19:56 RoanKattouw: Set $wgDefaultSkin = 'vector'; on wikitech
17:56 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
17:55 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js 'r66031'
17:36 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
16:57 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
16:52 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
16:48 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
16:41 RoanKattouw: Running svn cleanup again after patching some permissions issues (created catrope-owned copies of affected dirs)
16:14 RoanKattouw: Running svn cleanup in /mnt/upload6/private/ExtensionDistributor/mw-snapshot/branches/REL1_15/extensions
08:23 logmsgbot: mark synchronized php-1.5/wmf-config/db.php 'Repooling db12'
05:25 domas: running test build on db40, serving enwiki duty
05:24 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'normalizing load a bit hehe'
05:15 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
05:14 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
03:38 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'preparing for a copy'
01:46 logmsgbot: tstarling synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevs.hooks.php 'more profiling'
01:32 logmsgbot: tstarling synchronized php-1.5/includes/parser/Parser.php 'profiling tweak'
01:29 logmsgbot: tstarling synchronized php-1.5/includes/parser/Parser.php 'profiling tweak'

May 6

22:49 mark: STOPped mysqldump on snapshot1 (holding table locks) and killed the currently running mysql query. db12 is catching up again
22:25 logmsgbot: mark synchronized php-1.5/wmf-config/db.php 'depool db12 (as opposed to setting load to 0)'
21:53 logmsgbot: mark synchronized php-1.5/wmf-config/db.php 'depool db12'
21:43 RoanKattouw: db12 lag at 1464 seconds and rising
20:00 AaronSchulz: deployed r65996 (Flaggedrevs_alpha)
20:00 logmsgbot: aaron synchronizing Wikimedia installation...
19:42 mark: Shutdown BGP session to AS 13030
18:38 Rob: had wrong setting in name for dns on new scs, updated dns to correct
18:18 Rob: updated dns for scs-b1-pmtpa
17:14 schmir: schmir drinks beer
17:08 Rob: project2 had two bad fans. if it fails a lot more, it may have system damage from the overheating
16:55 Rob: replicated package installation from pdf2 to pdf3
16:14 Rob: pdf3 online and setup with identical users to pdf1/2, handed off to pediapress for mwlib magics
14:59 Rob: updated dns due to linne and gilman having same IP in zone file, but linne was not listed in the reverse dns file. changing gilman IP info since its not online and in cluster just yet.
14:39 Rob: updated dns for pdf3 server
14:08 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable PDF creation for anons on enwiki'
00:54 logmsgbot: tstarling synchronizing Wikimedia installation...
00:53 Tim: updating collection extension to trunk
00:38 logmsgbot: tstarling synchronized php-1.5/includes/parser/Parser.php 'r65974'

May 5

21:56 logmsgbot: catrope synchronized php-1.5/includes/DefaultSettings.php 'r65966'
21:56 logmsgbot: catrope synchronized php-1.5/skins/vector/main-rtl.css 'r65966'
21:56 logmsgbot: catrope synchronized php-1.5/skins/vector/main-ltr.css 'r65966'
19:49 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable CollapsibleNav on commons'
19:48 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'Bump CSS style version'
19:47 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
19:45 logmsgbot: catrope synchronizing Wikimedia installation...
19:45 RoanKattouw: Running scap to deploy r65959 (Vector, UsabilityInitiative fixes)
19:05 RoanKattouw: Deploying r65959 on testwiki
12:52 mark: DRAC of server multatuli not working - aborts ssh login with error
04:32 river: adding vlan 302 for toolserver web cluster private interconnect
01:48 logmsgbot: tstarling synchronizing Wikimedia installation...
01:47 Tim: deploying r65931, MIME type fixes especially .oga
01:26 Tim: installed ca-certificates on fenari to fix bug 23179
00:45 Tim: deploying oggvideotools r168 (large page size fix, bug 23402) to all image scalers

May 4

21:54 logmsgbot: catrope synchronized php-1.5/RELEASE-NOTES 'Dummy sync in an attempt to find problematic servers'
01:30 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php
01:29 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php
01:12 Tim: installed oggThumb on remaining image scalers. Deployed r65889 with oggThumb enabled.
01:12 logmsgbot: tstarling synchronizing Wikimedia installation...

May 3

15:11 logmsgbot: catrope synchronized php-1.5/extensions/ProofreadPage/ProofreadPage.php 'r65858'
15:10 logmsgbot: catrope synchronized php-1.5/extensions/ProofreadPage/proofread_index.js 'r65857'
10:09 RoanKattouw: Syncing srv99 in response to comments on bug 23373
09:28 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/Threads.php 'r65846'
09:27 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Threads.php 'r65846'
09:20 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/Threads.php 'r65843'
09:20 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Threads.php 'r65843'

May 1

21:35 RoanKattouw: Apache not running on srv246
20:25 logmsgbot: catrope synchronized php-1.5/includes/specials/SpecialUserlogin.php 'r65761'
20:24 logmsgbot: catrope synchronized php-1.5/includes/templates/Userlogin.php 'r65761'
13:05 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/FlaggedArticleView.php
13:02 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/FlaggedArticleView.php
08:42 AaronSchulz: deployed r65728 (Flaggedrevs_alpha)
08:42 logmsgbot: aaron synchronizing Wikimedia installation...

April 30

23:39 Tim-away: installed the new oggvideotools on srv224, source install from /root/src
20:00 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21375: Allow bcrats to remove sysop, bcrat on arbcom_fiwiki'
16:51 Rob: ryan rebooted prototype vm on tesla, it was giving oom errors, software on system needs to be adjusted, as the system has 4GB dedicated to that virtual host.
05:38 logmsgbot: tstarling synchronized php-1.5/includes/media/Bitmap.php 'r65682'

April 29

20:59 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'updating logo imports language for bug 19609'
20:06 atglenn: restarted (sort of) morebots

April 28

03:25 AaronSchulz: deployed r65602 (Flaggedrevs_alpha)
03:25 logmsgbot: aaron synchronizing Wikimedia installation...
00:57 logmsgbot: midom synchronized php-1.5/includes/User.php 'getAllCallers for user::invalidate'
00:33 logmsgbot: midom synchronized php-1.5/includes/Title.php 'verbose fname hook for invalidation'
00:07 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'removing srv169 from cluster9 ES config - seems to be freshly imaged server without the datastore atm, needs resync'

April 27

22:47 Tim: removed overly-broad block of all user agents containing "download" from common-acls.conf
16:36 Rob: tridge coming down for disk array addition
14:26 Rob: replaced disk 16 and 25 in dataset1

April 26

21:16 logmsgbot: ariel synchronized wmf-deployment/wmf-config/InitialiseSettings.php 'remove ip throttle exception for en pedia new user seminar, ended'
15:55 apergos: shutting down back end squid on knsq2, sdb bad
12:51 ^demon: Setup cron on mayflower for /home/demon/svn-dump-update every saturday at 1am.
07:45 mark: Removed sendmail from mobile1, so it stops spamming with cron messages. Why was it installed?
02:51 logmsgbot: midom synchronized php-1.5/wmf-config/db.php

April 25

23:19 logmsgbot: andrew synchronized php-1.5/extensions/TorBlock/TorBlock.class.php 'MFT r65535'
23:13 logmsgbot: andrew synchronized php-1.5/includes/HttpFunctions.php 'Deploy r65532'
22:56 Andrew: Deployed r65529, HttpFunctions fix.
22:56 logmsgbot: andrew synchronized php-1.5/includes/HttpFunctions.php
17:24 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'taking out db14 for some migrations'
11:58 JeLuF: added image/vnd.djvu to the mime.types file of ms7, refreshed webserver (bug 23318)
10:46 logmsgbot: catrope synchronized php-1.5/extensions/GlobalBlocking/SpecialGlobalBlockStatus.php 'r65520'
09:46 JeLuF: deleted oprofiled.log on mobile1 (17GB)

April 24

19:03 apergos: so on ombile1 there is now an "extra" error log in /var/log/nginx/error.log.toobig.gz (it needed to be compressed, sorry)
18:51 apergos: cleared some space from mobile1 (hd full), trying restarts again
17:52 Fred: mobile1 went oom again. Rebooting the server.
16:58 Rob: unracked csw-a4-pmtpa
16:48 Rob: new msw1-sdtpa switch is now working on serial console (scs was set to 57600, changed to 9600)
13:18 mark: Set up all sdtpa row B ServerTech PDUs
10:50 RoanKattouw: Rerunning refreshLinks for svwikisource
06:17 apergos: stopping back end squid on knsq11, sdb is bad

April 23

23:15 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php ' 23259 - Disable Extension:NewUserMessage on kawiki'
22:38 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23233 - rollbacker group on farsi wikipedia'
22:35 Fred: manually starting merb on mobile1.
22:25 Fred: mobile1 went OOM and shot itslef in the foot. Rebooted.
22:24 JeLuF: running refreshLinks for svwikisource
22:23 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23287 - $wgCategoryPrefixedDefaultSortkey on Swedish Wikisource'
22:20 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23244 - Enable FlaggedRevs/patrol extension on mkwiki'
22:17 logmsgbot: jeluf synchronized flaggedrevs.dblist '23244 - Enable FlaggedRevs/patrol extension on mkwiki'
22:14 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23299 - Allow imports from Wikimedia Commons to Hebrew Wikipedia'
22:12 Fred: mobile1 went dead. Looking into it.

April 22

17:55 Rob: added wikidev group to snapshot servers, which includes all roots and also specifically included andrew
17:18 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'Turn LiquidThreads on on wikimania2010wiki'
01:45 AaronSchulz: deployed r65418 (Flaggedrevs_alpha)
01:37 logmsgbot: aaron synchronizing Wikimedia installation...
00:00 atglenn: trying to run memtest on dataset1... little issue with the serial port line speed :-P

April 21

23:47 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'Set oldChangeTagsIndex var to false on flaggedrevs_labswikimedia'
22:30 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'reenabling ixia and db30 - both got disabled weeks ago by someone...'
22:11 domas: installed hardy on db31-db40 sans db34
14:55 logmsgbot: tstarling synchronized php-1.5/includes/Article.php 'r65394 fix pagetitle-view-mainpage'
14:25 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'Turn on live preview on testwiki'
14:25 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'Turn on live preview on testwiki'
14:23 logmsgbot: andrew synchronized php-1.5/includes/Preferences.php 'r65392'
13:56 logmsgbot: andrew synchronized php-1.5/skins/common/preview.js 'MFT r65388'
13:49 logmsgbot: andrew synchronized php-1.5/skins/common/preview.js 'MFT r65384'
11:00 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
10:59 logmsgbot: catrope synchronized php-1.5/skins/vector/main-rtl.css 'r65374'
10:59 logmsgbot: catrope synchronized php-1.5/skins/vector/main-ltr.css 'r65374'
10:17 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/View.php 'r65369'
10:17 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'r65369'
10:08 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/NewMessagesController.php 'r65367'
10:07 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/NewMessagesController.php 'r65367'
06:29 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'enabled abusefilter for flaggedrevs_labswikimedia'
02:07 domas: how did I install db31?

April 20

18:18 Rob: see that, me syncing files, not like you cheaters fred and ariel!
18:18 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 23231'
16:55 Rob: replaced bad disk on db5
13:19 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Touch this file to purge config cache'
11:53 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/View.php 'Merge r65305'
11:52 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'Merge r65305'
11:49 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'Merge r65302'
11:49 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/View.php 'Merge r65302'
11:28 ^demon: svn dump successfully completed. Dumped revs 1-65293 to /home/demon/svn/mediawiki-20100418-dump-with-deltas. Took up ~1.1G
11:13 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style vers'
11:13 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/lqt.css 'Deploy r65297'
11:11 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/lqt.css 'Deploy r65297'
10:40 logmsgbot: andrew synchronized php-1.5/extensions/NewUserMessage/NewUserMessage.class.php 'Deploy r65295'
10:27 ^demon: Running 'svnadmin dump --deltas' in a screen on Mayflower. Writing to storage2 Tomasz mount�ed at /home/demon/svn. Already to ~r30000

April 19

23:14 atglenn: one more time with elwiki, cswiki and ltwikt worker threads (shooting parent processes as well this time) on snapshot3
23:08 domas: started slow query killer on db9 - it was constantly alerting in #wikimedia-tech
22:32 atglenn: killed fetchText.php on snapshot3 for ltwikt, cswiki, thwiki, all hung on writes (nfs issue ? they were stuck from April 16 on... coincidentally when cname change went out for download.wm -> dumps.wm) Dumps seem to be running normally now
19:56 Tim: upgraded libavcodec and libavformat on image_scalers node group due to USN-931-1
00:18 Fred: made room on McHenry to resume backups from lily and sanger.

April 18

07:37 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '21243 - Request for PDF Version & Page collection'
07:27 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php ' 23218 - Add CAT: as namespace alias for Category: on enwikinews'

April 16

23:02 JeLuF: 16412 - Create a redirect from hu.wikimedia.org to http://wiki.media.hu
21:24 JeLuF: running cleanupTitles.php on *.wiktionary.org
21:01 JeLuF: updated site_stats:ss_images on plwiki
20:48 JeLuF: Redirect download.wikipedia.org and other aliases to dumps.wikimedia.org
20:07 JeLuF: 17008 - Redirecting from eo.wikisource.org
19:37 JeLuF: enabled on-demand SVG compression on upload.wikimedia.org
19:06 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '22157 - Enable Extension:NewUserMessage on ka.wiki'
18:46 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '20947 - Enable RevisionDelete on Hungarian Wikipedia (hu.wikipedia)'
18:31 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '22342 - Need a new namespace for the thesaurus in Bengali Wiktionary'
18:27 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23048 - Change name of Portal talk namespace on Tamil Wikipedia'
18:21 logmsgbot: jeluf synchronized php-1.5/wmf-config/flaggedrevs.php '23166 - New namespace for pl.wikisource'
18:18 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23166 - New namespace for pl.wikisource'
18:06 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '22426 - Change eo.wikt user namespace to "Uzanto"'
18:05 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '22426 - Change eo.wikt user namespace to "Uzanto"'
17:36 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '22319 - $wgBlockDisablesLogin = true; for otrs_wikiwiki and chapcomwiki'
17:32 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#23197 need to have Nominating Committee wiki deleted '
17:31 logmsgbot: jeluf synchronized all.dblist 'Deleting nomcomwiki'
17:31 logmsgbot: jeluf synchronized closed.dblist 'Deleting nomcomwiki'
11:55 Tim: running cleanupImages.php on mlwiki
09:46 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Change Czech talk namespace names per Danny B'
09:34 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesCs.php 'r65113 Change talk namespaces names'
08:57 hcatlin: changed mobile1 so logrotate is now run hourly instead of daily. should stop the disk from overflowing

April 15

11:55 logmsgbot: catrope synchronized php-1.5/includes/specials/SpecialRandompage.php 'r65056'
11:16 RoanKattouw: Running l10nupdate expecting it to fail, testing failure logging
10:31 RoanKattouw: Running l10nupdate by hand to debug issue reported at translatewiki
09:27 logmsgbot: tstarling synchronized php-1.5/extensions/CentralAuth/SpecialCentralAuth.php 'r65048'
09:26 logmsgbot: tstarling synchronized php-1.5/extensions/CentralAuth/CentralAuth.i18n.php 'r65048'

April 14

20:04 logmsgbot: ariel synchronized php-1.5/wmf-config/InitialiseSettings.php 'Add group IP to wgRateLimitsExcludedIPs for a wp for new users conf. To be removed Apr 20 2010'
18:27 atglenn: (yesterday's news) dataset1 back up with smaller ext3 partition replacing the xfs one, for testing
15:42 logmsgbot: catrope synchronized php-1.5/extensions/CentralAuth/CentralAuthUser.php 'r65033'
15:08 logmsgbot: catrope synchronized php-1.5/includes/specials/SpecialUpload.php 'r65031'
14:18 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesEn.php 'r65028'
14:17 logmsgbot: catrope synchronized php-1.5/includes/UserMailer.php 'r65028'
13:59 logmsgbot: catrope synchronized php-1.5/includes/specials/SpecialUpload.php 'r65026'
13:09 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
13:08 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js 'r65023'
12:58 logmsgbot: catrope synchronized php-1.5/includes/DefaultSettings.php 'r65022'
12:57 logmsgbot: catrope synchronized php-1.5/includes/specials/SpecialUpload.php 'r65022'
12:57 logmsgbot: catrope synchronized php-1.5/skins/common/upload.js 'r65022'
11:14 logmsgbot: catrope synchronized php-1.5/skins/Vector.php 'r65016'
11:14 logmsgbot: catrope synchronized php-1.5/skins/MonoBook.php 'r65016'

April 12

22:31 logmsgbot: ariel synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22630 BlockDisablesLogin enabled on arbcom-nl'
20:46 atglenn: reboot hangs on dataset1 (probably due to same lock issue on xfs we see in logs), power cycle will have to be done from the dc tomorrow
20:13 atglenn: removed /data from fstab on dataset1, rebooting
18:20 Rob: sq49 shiny
17:52 Rob: sq50 shiny
17:25 Rob: updated squid upload-setttings removing the specific server settings of sq49 & sq50, defaulting those hosts back to pmtpa setttings
17:07 Rob: sq48 shiny, sq50 reinstall
16:44 Rob: sq47 shiny, sq49 reinstalling
16:01 Rob: sq45 shiny, sq46 reinstalling
15:46 Rob: sq44 shiny, sq46 reinstalling
15:25 Rob: sq43 shiny, sq45 reinstalling
15:05 Rob: sq42 back online, sq44 reinstalling
14:33 Rob: sq43 reinstalling
14:02 Rob: sq42 being reinstalled

April 11

22:33 domas: s/DRAC/ILOM/
22:32 domas: setting up new DB machine DRAC with following configuration: http://p.defau.lt/?azoQvnt25YpR4Sa7hIwXOg
20:20 RoanKattouw: Fixing incomplete rename on frwiktionary, CaBot -> CaBot Senior, 4641 edits
20:11 mark: Unshut e8/1 on csw5-pmtpa
20:05 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '22625 - reviving deleted projects, restored to status "closed"'
20:05 logmsgbot: jeluf synchronized all.dblist
19:56 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '22625 - reviving deleted projects, restored to status "closed"'
19:53 logmsgbot: jeluf synchronized all.dblist
19:52 logmsgbot: jeluf synchronized all.dblist
19:49 logmsgbot: jeluf synchronized all.dblist
18:48 mark: Restored DNS scenario to normal: European traffic back to esams
18:45 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix, reports of empty CSS files'
18:07 mark: For reference: cable SMF00292 can be cleaned up (MPLS link)
15:44 mark: Switched traffic back to Florida ;-)
15:44 river: shut e8/1 HGTN on csw5-pmtpa because it was saturated
15:44 mark: River switched traffic back to Europe due to port congestion problem
15:14 mark: Moving all European traffic to Florida for a core router reload
15:14 mark: Moving all
12:18 logmsgbot: catrope synchronized php-1.5/includes/OutputPage.php 'r64929/r64930 LanguageConverter fixes'
12:18 logmsgbot: catrope synchronized php-1.5/includes/parser/Parser.php 'r64929/r64930 LanguageConverter fixes'
12:18 logmsgbot: catrope synchronized php-1.5/includes/Article.php 'r64929/r64930 LanguageConverter fixes'
12:18 logmsgbot: catrope synchronized php-1.5/languages/Language.php 'r64929/r64930 LanguageConverter fixes'
12:18 logmsgbot: catrope synchronized php-1.5/languages/LanguageConverter.php 'r64929/r64930 LanguageConverter fixes'
10:33 mark: Enabled RSTP on Toolserver vlans 102, 300, 301
10:24 mark: Implemented BFD between br1-knams and csw2-esams
09:40 logmsgbot: catrope synchronized php-1.5/includes/specials/SpecialAllmessages.php 'r64926 Fix Special:Allmessages sorting'
09:29 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Add researcher group on enwiki'
07:56 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23143 - Enable flagged revisions on sr: wikinews'
07:55 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23143 - Enable flagged revisions on sr: wikinews'
07:55 logmsgbot: jeluf synchronized flaggedrevs.dblist
03:18 Tim: restarted morebots
03:17 Tim: running cleanupMl.php in a screen on fenari, to fix the mess left by cleanupTitles.php
01:16 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/View.php
01:12 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/api/ApiFeedLQTThreads.php
01:09 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/pages/ThreadPermalinkView.php
01:09 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Thread.php
01:08 Andrew: Deployed trunk revision r64920
01:08 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php
00:22 Tim: running cleanupTitles.php on all wiktionaries

April 10

22:28 mark: Enabled Graceful Routing Engine Switchover (GRES) on csw2-esams
20:42 mark: Implemented IPv6 gateway failover using IPv6 DAD (Duplicate Address Detection)
20:23 mark: Implemented VRRP between csw1-esams and csw2-esams for VLANs 100, 101, 102 and 301
17:34 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/SpecialGlobalUsage.php 'r64890'
15:20 RoanKattouw: Deleting about 10k redundant redirects on mlwiki using deleteBatch.php
14:29 logmsgbot: catrope synchronized php-1.5/includes/EditPage.php 'r64882 for bug 23139'
12:52 mark: Set up BGP prefix advertisements through summarization with aggregate addresses, instead of iBGP announcement of the summary addresses of AS43821
12:11 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/PrefSwitch/PrefSwitch.i18n.php 'r64869'
11:32 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
11:31 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js 'r64864'
11:31 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r64864'
11:13 logmsgbot: catrope synchronized php-1.5/includes/specials/SpecialEmailuser.php 'r64863'
10:09 logmsgbot: catrope synchronized php-1.5/includes/specials/SpecialEmailuser.php 'r64861 Shut up fatal'
09:53 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 23128 Rename Project namespace on outreachwiki'
03:43 logmsgbot: tstarling synchronized php-1.5/includes/specials/SpecialContributions.php
03:27 logmsgbot: tstarling synchronized php-1.5/includes/specials/SpecialUpload.php 'r64847: fix $1 in uploadtext'
02:44 logmsgbot: tstarling synchronized php-1.5/includes/specials/SpecialContributions.php 'r64845, diff/hist link swap'
01:27 Tim: running cleanupTitles.php on ml wikis (starting with ml.wiktionary)

April 9

22:35 logmsgbot: catrope synchronized php-1.5/includes/parser/Parser.php 'r64839 Language converter bugfix'
22:32 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Set initialCaptial to true for Commons ForeignFileRepo setups. Chad made me do it'
22:18 logmsgbot: catrope synchronized php-1.5/includes/specials/SpecialUserlogin.php 'r64838 Fix fatal'
20:46 Rob: sockpuppet back online and shiny
20:40 Rob: tired of seeing pls reboot msgs on sockpuppet, rebooting it
20:39 Rob: byebye 412 days of uptime on sq41, coming down for OS reinstallation
19:53 logmsgbot: catrope synchronized php-1.5/includes/parser/Parser.php 'r64824 Recover lost namespace names'
19:31 logmsgbot: catrope synchronized php-1.5/includes/parser/Parser.php 'r64821 Language converter bugfix'
19:00 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#20045 - Logo change for pih.wikipedia'
18:16 logmsgbot: catrope synchronized php-1.5/includes/parser/Parser.php 'r64815 Language converter bug'
17:15 logmsgbot: fvassard synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug23118 Grant Bureaucrat all rights removal on outreach wiki'
16:52 logmsgbot: catrope synchronized php-1.5/includes/api/ApiRollback.php 'Live hack for bug 23117'
15:59 logmsgbot: jeluf synchronized php-1.5/wmf-config/abusefilter.php '23099 - Enable EditFilter on Spanish Wikipedia'
15:47 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23099 - Enable EditFilter on Spanish Wikipedia'
14:45 logmsgbot: catrope synchronized php-1.5/includes/diff/DifferenceInterface.php 'r64803 for bug 23113'
14:20 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesPt.php 'r64801 Restore name of User namespace in pt'
12:45 logmsgbot: tstarling synchronized php-1.5/includes/Title.php 'revert'
12:43 logmsgbot: tstarling synchronized php-1.5/includes/Title.php 'r64793'
12:42 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesPt.php 'r64791 restore pt specialpage aliases to 1.16wmf3 state'
12:17 RoanKattouw: Creating tables for FlaggedRevs on bswiki, were missing and causing DB errors
12:16 Tim: added /sr variant alias, apparently this is the default now on srwiki and the lack of it was causing a redirect loop
12:07 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesEs.php 'r64787 misspelled alias'
11:55 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesEs.php 'r64784: Restore ORDERNAR: alias'
10:39 logmsgbot: catrope synchronized php-1.5/includes/specials/SpecialUserlogin.php 'r64783'
09:43 logmsgbot: tstarling synchronized php-1.5/languages/messages/MessagesMl.php
09:35 RoanKattouw: Touched MessagesEn.php on srv252 to shut up "not a valid magic thingie" errors
09:34 logmsgbot: catrope synchronized php-1.5/includes/api/ApiBase.php 'Deploy r64778 to shut up PHP warning'
07:33 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 'all repooled'
07:20 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php
07:19 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'read/write'
07:18 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 'depooling servers still lagged'
06:55 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php
06:54 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 'all wikis read-only'
06:48 Tim: preloading centralauth.globaluser into RAM on s3 slave servers
06:37 logmsgbot: tstarling synchronized php-1.5/wmf-config/ExtensionMessages.php
06:36 logmsgbot: tstarling synchronized php-1.5/wmf-config/ExtensionMessages.php
06:33 logmsgbot: root synchronizing Wikimedia installation...
06:33 logmsgbot: root synchronizing Wikimedia installation...
06:33 Tim: deploying MW 1.16wmf4
05:33 Tim: svn switch svn+ssh://svn.wikimedia.org/svnroot/mediawiki/branches/wmf/1.16wmf4
05:05 JeLuF: 15357 - Redirect dk.wiktionary and dk.wikibooks to da.wiktionary and da.wikibooks respectively
04:53 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#11766 - Install Labeled Section Transclusion extension on fr.wikipedia'
00:01 Tim: pilot updated for r64769

April 8

23:50 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/Thread.php
23:50 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Thread.php
23:49 Andrew: Deploying r64767
23:46 Tim: on kaulen: svn up 1.16wmf4
23:38 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/specialpages/Stabilization_body.php 'deployed r64765'
23:06 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/Thread.php
23:06 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php
23:05 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Thread.php
23:05 Andrew: Deploying r64762
22:16 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesAce.php 'r64760'
20:14 mark: Set up p2p link (vlan) between csw1-esams and csw2-esams over the 2x10G trunk and configured OSPF
19:41 mark: Made csw1-esams and br1-knams originate default routes (v4/v6) in OSPF
19:39 logmsgbot: aaron synchronizing Wikimedia installation...
19:39 AaronSchulz: deployed r64754 (FlaggedRevs_alpha)
19:34 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#23038 - Set sitename of Belorussian Wiktionary'
19:24 atglenn: missing.php update was copy paste from meta in accordance with bug 11125
19:23 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22974 - Enable FlaggedRevs on bswiki'
19:23 logmsgbot: jeluf synchronized flaggedrevs.dblist '#22974 - Enable FlaggedRevs on bswiki'
19:23 logmsgbot: ariel synchronized php-1.5/wmf-config/missing.php
18:22 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23077 - Extension:Collection for English Wikiversity'
18:19 logmsgbot: jeluf synchronized php-1.5/wmf-config/CommonSettings.php '23081 - Remove "makesysop" right from bureaucrats and stewards groups'
17:42 mark: Implemented OSPFv3 on AS43821
17:00 Rob: db31-db40 racked and lom setup (except db34 which has an unresponsive bmc fresh from factory.
16:10 mark: Rebooted br1-knams after it got confused about OSPF area 0 vs 0.0.0.0
15:10 mark: Changed OSPF reference bandwidth from 100 Mbps to 100 Gbps on all AS43821 routers
06:25 logmsgbot: kate synchronized php-1.5/wmf-config/db.php
06:25 river: removing lomaria to dump s5 for TS
06:22 logmsgbot: kate synchronized php-1.5/wmf-config/db.php
06:22 river: removing db30 to dump s2 for TS
05:53 The_Thing: Can I test this?
02:59 logmsgbot: andrew synchronized php-1.5/includes/media/Bitmap.php
02:58 logmsgbot: andrew synchronized php-1.5/includes/DefaultSettings.php
02:58 Andrew: Seems to work correctly, deploying fix to all wikis.
02:55 Andrew: Deploying r64727 on testwiki for testing GIF scaling fixes.
01:43 Tim: on kaulen: reset the root mysql password and saved it in /root/.my.cnf for future reference
01:05 Tim: installing PHP on kaulen and various MW dependencies

April 7

21:56 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php
21:55 Andrew: Sticking LiquidThreads on officewiki, by Erik's request.
20:52 Fred: ganglia shut down for a couple minutes while I grow the tmpfs.
18:13 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Change PrefStats tracking for wikis with Vector/toolbar enabled by default'
08:54 logmsgbot: catrope synchronized php-1.5/includes/api/ApiLogin.php 'r64695'
02:47 logmsgbot: tstarling synchronizing Wikimedia installation...
00:16 logmsgbot: tstarling synchronizing Wikimedia installation...
00:16 Tim: deploying r64679, CSRF fix, see bug 23076

April 6

22:44 mark: Implemented OSPF on AS43821
16:22 mark: Disabled the automatic purging of Commons pages
16:00 apergos: turned off profiling on db22, removed stuff in samples, disk was full
08:17 logmsgbot: mark synchronized php-1.5/wmf-config/CommonSettings.php 'Set 220px as default thumbnail size on Commons'
08:16 logmsgbot: mark synchronized php-1.5/wmf-config/InitialiseSettings.php 'Add 220px thumbnail size option for Commons'
07:43 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable Vector, toolbar by default on Commons'
07:43 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Enable dialogs too when $wmgUsabilityEnforce set'
07:36 logmsgbot: catrope synchronized php-1.5/wmf-config/ExtensionMessages.php 'Added PrefSwitch'
07:29 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable PrefSwitch on test'
07:29 mark: Removed ldap from nsswitch.conf on zwinger and browne
07:24 mark: With srv1 dead, we're without an LDAP server. Removed zwinger from the mediawiki-installation node group
07:24 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix, add conditional code for PrefSwitch'
07:22 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r64652'
07:18 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js
07:13 mark: RIP: srv1. It reached the respectable uptime age of 1447 days. It will be sorely missed, especially by its twin brother, srv2.
07:06 mark: Started apache on srv240
07:03 Andrew: Running ~andrew/populateGIFMetadata.php on all wikis in a screen on fenari. Feel free to kill it if it causes any problems.
07:01 mark: Restarted apache on srv228, srv235, srv241,
06:57 mark: Started apache on srv252
06:55 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/images/wikiEditor/toolbar/button-sprite.png 'Forgotten file'
06:51 mark: Restarted apache on srv158
06:48 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'Bump CSS style version'
06:47 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
06:46 mark: Set up purging of all commons hits, removing duplicates using 'dampen.py' in locke:/root
06:45 logmsgbot: catrope synchronizing Wikimedia installation...
06:45 RoanKattouw: Scapping to deploy new version of UsabilityInitiative extension
04:39 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'GIF scaling working correctly on testwiki, pushing to other wikis again'
04:20 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'Turn GIF thumbnailing off except on testwiki for further investigation'
04:16 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php
04:15 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#19831 - Add an icon to Wikinews for iPhone's & iPod Touch's'
04:13 Andrew: Trying re-activating GIF scaling for images in the limits
04:11 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php
04:10 Andrew: setting $wgMaxAnimatedGifArea back to its default to prevent animated GIFs from being scaled as single frames.

April 5

22:01 Fred: modified site.pp to change ganglia squids aggregators.
21:10 mark: Changed purge sample rate to 1/10
21:06 tomaszf: lowering xml snapshot workers count down to three to prepare for storage server move
20:56 mark: Set up automatic purging of 1 out of every 100 Commons requests, to test for the migration tomorrow
20:44 tomaszf: removing the 20091128 xml snapshot dataset to free up space.
18:18 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#23050 - Change ab.wikipedia site logo'

April 3

19:59 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'null edit, to update langlist cache'
19:58 logmsgbot: jeluf synchronized langlist 'remove tokipona'
19:23 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22912 - Change sitename, add namespace alias on ukwikisource'
19:05 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#21068 - Some wikis have missing logo images on secure server'
18:40 logmsgbot: jeluf synchronized php-1.5/cache/interwiki.cdb 'Updating interwiki cache for new wikitech entry'
18:24 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22994 - Logo switch on vi.wikiquote'
18:11 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#23032 - Enable NewUserMessage on ro.wikinews.org'

April 2

21:05 logmsgbot: kate synchronized php-1.5/wmf-config/db.php
21:05 river: repooled db5
20:34 mark: connected scs1-esams serial console to knsq16's com1
16:59 logmsgbot: kate synchronized php-1.5/wmf-config/db.php
16:59 river: repooled db1
16:26 river: depooling db5 to dump s4
16:26 logmsgbot: kate synchronized php-1.5/wmf-config/db.php
09:49 logmsgbot: kate synchronized php-1.5/wmf-config/db.php
09:49 river: repooled db27
09:02 river: removing db1 to dump s6 for TS
09:02 logmsgbot: kate synchronized php-1.5/wmf-config/db.php

April 1

22:02 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
22:00 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins/jquery.wikiEditor.html 'r64505'
21:30 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r64464'
21:27 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js 'r64464'
20:33 logmsgbot: kate synchronized php-1.5/wmf-config/db.php
20:33 river: removing db27 from s3 for TS dump
20:08 AaronSchulz: deployed r64498 for FlaggedRevs_alpha
20:07 logmsgbot: aaron synchronizing Wikimedia installation...
18:25 IShadowed: DOMAS OMG WIKI DOWN

March 31

21:16 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Don't totally disable collapsiblenav, keep it as a hidden pref'
20:31 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
20:30 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'r64455'
20:30 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/css/combined.min.css 'r64455'
20:29 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r64455'
20:29 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js 'r64455'
19:55 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Disable collapsiblenav explicitly in wake of usability deployment'
18:19 RoanKattouw: Manually corrected time on prototype, was 6 hours behind. Guess we need NTP there too
16:56 Rob: rebooted srv119
16:31 Rob: srv89 had temp warnings and failure to respond to console, replaced two bad fans in chassis with fans from srv127
16:22 Rob: srv98 was unresponsive to console, rebooted and appears to be back online and working just fine for now.
16:20 Rob: srv127 decomissioned. dead hard disks, dead fans, temp warnings, no lom, and out of warranty. will be parted out to other servers
16:08 Rob: srv127 throwing both hard disk errors and temp warnings
15:00 Rob: db28 is not in any current use to to memory issue, investigating memory issue now for hardware warranty replacement
05:33 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Tim says this helps'
05:02 logmsgbot: jeluf ran sync-common-all
04:59 logmsgbot: jeluf ran sync-common-all
00:50 Tim: increased apache TimeOut setting on singer from 5 to 30 seconds, to hopefully fix multiple reports of high error rates on secure.wikimedia.org

March 30

20:18 Rob: new server gilman racked for future secure gateway use and dns updated for server fqdn
20:17 Rob: back to work morebots
17:22 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '22650 - bureaucrats should only be allowed to remove abusefilter and bot, not the other stuff'
09:21 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version'
09:20 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/lqt.css 'Deploy r64384'
09:20 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/lqt.css 'Deploy r64384'
05:28 logmsgbot: jeluf ran sync-common-all
04:39 logmsgbot: tstarling synchronized php-1.5/extensions/InputBox/InputBox.classes.php
04:29 Tim: removed the ex-fedora instances from srv122, srv124, srv142 from ms1
00:29 Tim: copying half the data back from ms1 to ms2 so I can have a backup, with half the data on each server

March 29

23:57 Tim: deleting obsolete ex-fedora instances from ms3 and ms2
23:50 Tim: shutting down all ex-fedora mysql instances except cluster1/2
20:32 atglenn: removed dead upload5 mounts from hume, fenari, snapshot1, should be the last traces
19:50 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
19:50 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js 'r64357'
14:29 RoanKattouw: prototype.wikimedia.org VM seems to have crashed, prototype down

March 28

21:25 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'Add a new blocked email address'
18:49 RoanKattouw: Ganglia monitoring of Flordia Squids broken, reports all boxes down
16:33 apergos: upped maxclients and turned keepalive back on on singer (these were changed last week during the site outage)

March 27

18:21 JeLuF: upgraded ViewVC to 1.1.4
07:51 JeLuF: roled out new apache config with a catch-all for www.*.(wikisource|wiktionary|wikiversity).org and www.(meta|commons).wikimedia.org, see bug 1698
00:47 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php
00:42 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'cleaned up testwiki config'
00:36 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'removed redundant configs'
00:26 AaronSchulz: added plwikisource flaggedrevs tables

March 26

21:46 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22919 - Enable FlaggedRevs for pl.wikisource'
21:45 logmsgbot: jeluf synchronized flaggedrevs.dblist
21:44 logmsgbot: jeluf synchronized php-1.5/wmf-config/flaggedrevs.php
21:41 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
21:41 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins/jquery.wikiEditor.html 'r64223'
21:39 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22965 - wgUploadNavigationUrl for nl.wikipedia'
20:40 Fred: restarted DNS server on ns2
20:34 Rob: added dns support for new misc server tarin
19:44 Rob: ignore nagioswarnings for sq11-sq30, they are being decommissioned
18:49 Fred: removed srv217 from pybal (api) since the server is not ssh-accessible.
18:48 Fred: removed srv89 from pybal since the server is not ssh-accessible.
18:44 Rob: removing sq11-sq30 from nagios and node lists
18:41 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
13:14 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
13:13 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r64216'
13:10 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'More fixes'
12:59 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r64215'
07:34 logmsgbot: tstarling synchronized php-1.5/extensions/CodeReview/ui/CodeRevisionListView.php 'more path'

March 25

23:10 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'bump style version appendix'
23:08 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins/jquery.wikiEditor.html 'r64195'
23:08 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'config for r63650'
21:54 atglenn: added nimish to wmf-deployment committers for current usability testing sessions
20:28 JeLuF: commons and meta now also use mediawiki-based robots.txt generation, based on robots.php (#22040)
20:23 AaronSchulz: deployed r64173 (flaggedrevs_alpha)
20:00 JeLuF: set up svn in /home/wikipedia/conf/httpd
17:36 Rob: copied meta favicon to outreachwiki
13:35 Rob: pdf2 become unaccessible, required reboot
12:07 RoanKattouw: Deployed r64154 around 11:30 UTC
12:07 RoanKattouw: Andrew restarted morebots, was down

March 24

23:11 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
23:08 RoanKattouw: Running scap to deploy UsabilityInitiative updates
22:10 Tim: moved mayflower:/home/mwdocs/phase3 to /var/mwdocs to reduce space usage in /home. Updated apache conf and crontab.
22:07 mark: Sending European traffic back to esams
22:01 Rob: sq46 was unresponsive and rebooted, now is working fine
21:30 Rob: secure.wikimedia.org back online
20:47 mark: Renamed rr.wikimedia.org to text.wikimedia.org
19:38 mark: Brought esams Sun V20z servers back up - restored rr.esams and upload.esams LVS
19:24 mark: Bringing lily back up
19:07 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22924 - Enable the Collection extension at el.wikipedia'
18:40 logmsgbot: jeluf synchronized live-1.5/robots.php
18:39 domas: disabled keepalive on singer
18:37 river: delegated 1.24.10.in-addr.arpa to ha-dns-auth.toolserver.org
18:17 Rob: techdetail: lowered maxclients on singer apache2 and disabled secure site in apache configuration, do not rollback changes to normal until after esams is back online
18:16 Rob: techlbog back up, secure is NOT up, and is not expected to come back up, until the esams outage is resolved
18:01 Rob: disabled secure on singer, as everyone trying to use it overloaded both secure and blogs.
18:01 Rob: Secure is offline
17:51 mark: http://techblog.wikimedia.org/2010/03/global-outage-cooling-failure-and-dns/
17:50 mark: Reduced Negative TTL setting in SOA record of wikimedia.org from 1H to 10M
17:40 domas: NXDOMAINs with 1h TTL served on all records for short moment
17:40 Rob: globally (robs bad)
17:40 Rob: pending techblog post regarding the issue with accessibility in EU
17:25 mark: DNS scenario dir esams-down was empty, all zone files missing. Fixed.
17:18 mark: esams down due to overheating issues; moving traffic to Florida
14:18 mark: csw2-oe12-esams connected to iris's serial console (temp)
04:24 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php
04:21 Andrew: Running scap to update LiquidThreads beta
04:17 Andrew: Preparing to update LiquidThreads beta (production) to alpha version.

March 23

22:16 apergos: latest patches and reboot for ms5
22:13 Rob: sq40 back online, sq41 reinstalling
21:50 Rob: reinstalling sq40 & sq41
21:27 Rob: sq38 & sq39 reinstalled and online
20:43 Rob: taking down sq38 & sq39 for reinstallation
20:43 Rob: sq36 & sq37 back online and in rotation
19:39 Rob: sq36 & sq37 being reinstalled
19:35 logmsgbot: jeluf synchronized live-1.5/robots.php 'change charset to utf8'
19:32 Rob: sq34 and sq35 (not 25)
19:32 Rob: sq34 and sq25 back online and in rotation
19:27 logmsgbot: jeluf synchronized all.dblist
19:24 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.hooks.php 'r64086'
19:24 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.php 'r64086'
19:22 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Add vector-collapsiblenav as a hiddenpref'
19:21 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'removed more group permission for closed wikis'
18:34 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Remove all simplewikibooks configs from Initialisesettings'
18:32 logmsgbot: jeluf synchronized php-1.5/wmf-config/CommonSettings.php 'REVERT Test case for simplewikibooks'
18:29 logmsgbot: jeluf synchronized php-1.5/wmf-config/CommonSettings.php 'Test case for simplewikibooks'
18:28 Rob: sq34 & sq35 coming down for reinstallation
18:23 logmsgbot: jeluf synchronized php-1.5/wmf-config/CommonSettings.php 'REVERT Test case for simplewikibooks'
18:22 logmsgbot: jeluf synchronized php-1.5/wmf-config/CommonSettings.php 'Test case for simplewikibooks'
18:19 Rob: sq33 is reinstalled and doing quite fine.
16:02 Rob: puppet stopped on sq34-sq50, DO NOT RESTART PUPPET. These hosts need reinstallation before puppet can be restarted, or squid changes can be deployed to them.
15:52 Rob: changed the setting for pmtpa to change partitioning schema and up the cache size. Do not deploy old squids with this information until Rob finishes reinstalling them
15:52 Rob: changed squid config in text squids for hard disks update on reinstalls
12:01 logmsgbot: jeluf ran sync-common-all
11:59 logmsgbot: jeluf synchronized closed.dblist

March 22

21:02 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#21213 - logo for ace wikipedia'
18:16 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22657 updating logo'

March 21

20:55 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22882 - Add a patroller group to fr.wikisource'
20:51 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22454 - Patroller and autopatrolled flags on es.wikipedia'
16:48 domas: db28 has more memory errors: http://p.defau.lt/?IvvVKaNdi05v1yfwLiHWVw
16:44 domas: dist-upgraded db28 to karmic
12:14 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'bgwiki group permissions fixed'
11:37 logmsgbot: jeluf synchronized closed.dblist
11:04 JeLuF: removed tokipona wikis from all.dblist
11:02 logmsgbot: jeluf ran sync-common-all
10:25 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22446 - Change the WT namespace to an alias for Wiktionary, for Swedish Wiktionary'
10:03 logmsgbot: jeluf ran sync-common-all
09:53 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'UNDO: 22446 - Change the WT namespace to an alias for Wiktionary, for Swedish Wiktionary - pages still exist in these namespaces'
09:40 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#16857 - Request to switch over logo for sanskrit language wikipedia localisation'
09:34 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22283 - Add "Author" namespace to Telugu Wikisource'
09:26 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22329 - Redirect WV to Wikiversity space for English Wikiversity'
09:22 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22446 - Change the WT namespace to an alias for Wiktionary, for Swedish Wiktionary'
09:13 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '22481 - Namespace alias at mwl.wikipedia.org'
09:06 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '22481 - Namespace alias at mwl.wikipedia.org'
06:57 JeLuF: bugzilla.wikimedia.org
06:57 JeLuF: Changed kaulen config to include ServerAlias bugzilla.wikipedia.org for

March 20

21:21 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'removed sysop access to stabilization except on wikis already have config rows. This page should not be enabled by default.'
19:49 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22549 - Enable email on watchlist change on fr.WV'
19:42 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22699 - Autopatroler at cs.wikipedia'
18:17 logmsgbot: jeluf synchronized php-1.5/wmf-config/mc.php 'srv98 down'
18:13 logmsgbot: jeluf synchronized php-1.5/wmf-config/mc.php 'srv119 down'
14:53 apergos: tossed a few files from tridge to make space, a very short-term solution
14:13 JeLuF: Bug 22681: migrated users on bgwiki dawiki enwiktionary hewiki hrwiki itwiktionary from groups autopatrol and autopatroller to group autopatrolled
14:12 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22681 - Clean up/unify autopatrol/autopatrolled/autopatroller group names'
13:38 JeLuF: created Italian planet at planet.it.wikimedia.org
12:57 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22909 revert'
12:15 JeLuF: restarted morebots
12:15 JeLuF: added it.planet.wikimedia.org to DNS
12:15 JeLuF: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22669 - Logo change for az.wikiquote.org'
12:15 JeLuF: jeluf synchronized php-1.5/wmf-config/abusefilter.php
12:15 JeLuF: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22909 Liquidthreads for huwiki'
12:15 JeLuF: jeluf synchronized php-1.5/languages/messages/MessagesKrc.php 'MessagesKrc from trunk, for bug:22657'

March 19

17:53 logmsgbot: tfinc synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'iframe display bugfix'
17:53 logmsgbot: tfinc synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.js 'iframe display bugfix'
17:52 logmsgbot: tfinc synchronized php-1.5/extensions/UsabilityInitiative/js/plugins/jquery.wikiEditor.js 'iframe display bugfix'
17:44 Fred: getting BUFFER IO ERRORS on srv169. JFS refuses to fix itself.
17:39 Fred: srv169 is having problems with sda6 (/a)... working on it and trying to salvage.
07:36 logmsgbot: tstarling synchronized php-1.5/includes/Revision.php 'bug 22624 hack'
07:05 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 're-adding cluster1+2 for bug 22624'

March 18

23:36 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/language/FlaggedRevs.i18n.php 'deployed r63938'
23:35 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/FlaggedArticleView.php 'deployed r63938'
22:27 logmsgbot: root ran sync-common-all
21:11 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/FlaggedArticleView.php 'deployed r63929'
20:34 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'update flaggedrevs_labs config'
20:31 AaronSchulz: deployed r63927 (flaggedrevs_alpha)
16:14 Fred: security updates of apaches in progress
06:22 logmsgbot: tstarling synchronized php-1.5/includes/StubObject.php 'reverted profiling hack, I assume whatever information you wanted to get out of this, you've got now'
06:19 logmsgbot: tstarling synchronized php-1.5/extensions/ContributionReporting/FundraiserStatistics.css 'revert whitespace-only change'
05:57 logmsgbot: tstarling synchronized php-1.5/extensions/CentralNotice/SpecialNoticeText.php 'revert whitespace-only change'
05:55 logmsgbot: tstarling synchronized php-1.5/extensions/LiquidThreads_alpha/pages/ThreadPermalinkView.php 'reverted debugging hack'
00:33 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Adding override for strategy and usability wiki for wmgNoticeProject'
00:05 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Changing mapping of commosn under wmgNoticeProject to just the string commons'
00:03 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Commenting out commons from wmgNoticeProject to fix loading'

March 17

22:54 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'Bump style version'
22:51 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
22:27 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r63888'
22:20 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r63887'
22:02 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/OptIn/OptIn.i18n.php 'r63885'
22:00 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'bump style version appendix'
22:00 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r63883'
21:41 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 21005'
21:30 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix for usability deployment'
21:22 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Enable experimental Usability features but add them to $wgHiddenPrefs'
21:08 RoanKattouw: Running scap
21:05 RoanKattouw: Updating UsabilityInitiative to trunk state cluster-wide
20:39 RoanKattouw: Pushing UsabilityInitiative software update to testwiki

March 16

20:55 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 'removed ex-fedora clusters, should all be unreferenced now due to RCT completion'
19:55 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php ' Bug 22650 - Enable the AbuseFilter on the French Wikipedia'
19:55 logmsgbot: jeluf synchronized php-1.5/wmf-config/abusefilter.php ' Bug 22650 - Enable the AbuseFilter on the French Wikipedia'
17:19 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'removed redundant tab settings'
08:05 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version'
08:05 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/lqt.js 'Reactivate LiquidThreads live preview'
05:01 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22691 - some fixes on de_wikiquote'

March 15

23:49 Fred: upgraded puppet on ms7 to the recent version that doesn't have scary stupid bugs in it!
21:58 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22663 - Import sources for frwiktionary'
21:52 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22694 - definition of import sources for mg.wiktionary'
21:34 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php ' Bug 22817 - Create Artikeldata namespace on dawiki'
21:31 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php ' Bug 22825 - Set $wgNamespacesWithSubpages[NS_MAIN] = true on sewikimedia'
21:25 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22833 - Enable multipage upload form for English Wikiversity'
21:01 logmsgbot: jeluf synchronized live-1.5/robots.php
20:48 logmsgbot: jeluf synchronized docroot/meta/favicon.ico
20:22 logmsgbot: jeluf synchronized docroot/incubator/favicon.ico
20:08 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22648 - Add portal namespace to kw wikipedia'
20:03 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php ' Bug 22674 - Update Special:Import options at English Wikibooks'
19:55 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php ' Bug 22406 - Enable Extension:Quiz on zhwiki'
19:52 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22615 - Import sources for sv.wikibooks'
19:49 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22787 - TZ for Estonian projets'
19:43 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22831 - wgBlockAllowsUTEdit for kowiki'
19:39 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22827 - Logo change for pcdwiki'
17:53 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Resyncing because I suspect some boxes didn't pick this up'
17:00 hcatlin: [mobile] code push complete!
16:52 hcatlin: [mobile] pushing code update for adding ext, cz, mr, ca, bn, kl, mn, scn, si, and ta langauge codes
16:29 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22834'
15:51 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22473'
15:27 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22791'

March 14

04:52 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'Make Vector and usability enhancements default on liquidthreads_labswikimedia'
04:46 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'Bumping style version'
04:43 Andrew: Scapping to deploy LiquidThreads alpha update
04:40 Andrew: Preparing for LiquidThreads alpha update to trunk state

March 13

18:42 Rob: sanger is back online and working
18:33 Rob: sanger moved, coming back online
18:21 Rob: thats sanger, not singer.
18:21 Rob: taking down singer to move it up in the rack, during this downtime (approx 15 min), wikimedia.org email will be unavailable.
18:20 Rob: mchenry is back online and dealing with its 10 minute mail backlog.
18:14 Rob: mchenry moved, bringing it back online
18:01 Rob: mchenry host coming down to move its rack location. Downtime will be approx. 15 minutes. (all mailing list relays may not work and thus new mailing list entries will back up until this move is completed.)

March 12

19:23 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'Enabling .odg extension for uploads'
18:49 Fred: moved stats.w.o from Zwinger to Spence
12:00 RoanKattouw: storage2 (download.wikimedia.org) was down between 11:40 and ~11:53 UTC, coincides with load spike
07:43 apergos: really edited the right copy of worker.py this time (the copy in /backups :-P)
06:19 apergos: editd worker.py in place on snapshot3 to remove --opt and replace all except lock options individually (it was running with that on db12 again, with high lag). shot that worker process, lag and thread count going down now
02:49 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php
02:49 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 'reducing load on db12 again, seems to be lagging again'
02:38 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 'increased general load on db12'
02:34 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php
02:32 Tim: db12 caught up, repooling
02:15 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'subpages for Grants namespace on meta (bug 22810)'
02:02 tomaszf: killing 20100312 xml snapshot run for enwiki due to unnecessary locking of pagelinks table on db12
01:43 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 'moving db12 back into the "dump" query group so we don't kill db26 too'
01:41 Tim: note: mysqldump query for pagelinks table is running on db12, presumably that is the cause of the slowness and network spike
01:37 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php
01:32 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 'brought db26 into rotation as enwiki watchlist/contribs server, to replace db12 after cache warming'
01:30 Tim: network spike on db12 at 01:00. Going to depool it in case it's going to try the same trick as last time
01:21 logmsgbot: ariel synchronized php-1.5/wmf-config/InitialiseSettings.php 'resyncing for bug 22810 (stupid ssh-agent :-P)'
00:47 logmsgbot: ariel synchronized php-1.5/wmf-config/InitialiseSettings.php 'Grants namespace on meta, bug 22810'
00:17 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Removing foundation wiki from wmgDonationInterface'

March 11

22:03 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22669 rollback'
22:02 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22669'
21:36 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22404'
21:19 logmsgbot: ariel synchronized php-1.5/wmf-config/InitialiseSettings.php 'subpages for templates on foundationwiki (#22484)'
20:37 atglenn: re-enabled replication on media server (ms7 -> ms8), we are back in business
16:43 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22089'
12:27 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version'
11:22 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/View.php 'Deploy r63591, fixes for LiquidThreads comment digestion'
11:21 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'Deploy r63591, fixes for LiquidThreads comment digestion'
01:34 Tim: reinstalled wikimedia-task-appserver on fenari to fix symlinks /apache and /usr/local/apache/common
00:26 atglenn: playing catchup on replication now from ms7-ms8 zfs send/recvs running as root

March 10

22:59 atglenn: patched and rebooted ms8, working on snapshot issue
21:07 logmsgbot: catrope synchronized php-1.5/extensions/ContactPage/ContactPage.i18n.php 'r63575'
21:07 logmsgbot: catrope synchronized php-1.5/extensions/ContactPage/ContactPage.php 'r63575'
21:07 logmsgbot: catrope synchronized php-1.5/extensions/ContactPage/SpecialContact.php 'r63575'
18:54 Fred: removed duplicate ACL for 208.80.152.157 from text-squids

March 9

23:32 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/View.php 'Merge r63523 into LiquidThreads production'
23:26 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'Merge r63524 into LiquidThreads alpha'
23:26 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'Merge r63523 into LiquidThreads production'
23:24 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'Merge r63524 into LiquidThreads alpha'
23:24 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'Merge r63523 into LiquidThreads production'
23:15 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/View.php 'Merge r63154 into LiquidThreads production'
23:13 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'Merge r63154 into LiquidThreads alpha'
20:22 rainman-sr: this morning's search api outage seems to be connected with index update on search8 which triggered high GC load and stuck threads, removing enwp spell index from search8
15:37 hcatlin: message before was referring to mobile site.
15:37 hcatlin: deployed new languages. cs homepages.
14:58 domas: forgot to log - yesterday's API fail was repeated today too, all API nodes were blocked on talking to lucene, and search cluster was idle too, and...
14:54 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesEt.php 'force recache to solve l10ncache issue'
14:51 logmsgbot: catrope synchronized php-1.5/includes/LocalisationCache.php 'r63417'
00:41 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/FlaggedRevs.hooks.php 'deployed r63447'
00:40 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/FlaggedRevs.class.php 'deployed r63447'

March 8

23:59 logmsgbot: tstarling synchronized php-1.5/includes/Sanitizer.php 'r63426'
23:57 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php
23:52 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'removed labs config cruft'
23:32 AaronSchulz: deployed updates MFT to FlaggedRevs_alpha (r63444)
23:30 Fred: updated search nodefile to contain search[13..20].
23:10 AaronSchulz: finally got labs msgs working it seems
23:09 logmsgbot: aaron synchronized php-1.5/wmf-config/CommonSettings.php 'do not override flaggedrevs_alpha msg dirs for lab sites'
22:40 logmsgbot: tstarling synchronized php-1.5/thumb.php 'r63432'
22:04 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/language/OldReviewedPages.i18n.php
21:54 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaLicenseTexts.i18n.php 'r63423'
21:41 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php
21:38 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php
21:36 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'disable LU on labs sites'
21:09 logmsgbot: catrope synchronized php-1.5/includes/LocalisationCache.php 'r63417'
20:57 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'revert, no effect on msg issue'
20:53 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'settings labs cache dir'
20:31 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'Switched labs sites to FlaggedRevs_alpha'
20:30 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'custom cache dir config for labs sites'
19:18 logmsgbot: tstarling synchronized wmf-deployment/cache/trusted-xff.cdb
18:15 Fred: disabled regular http check for mobile1 (leaving the longer timeout one in place)
13:58 logmsgbot: catrope synchronized php-1.5/cache/l10n/l10nupdate-et.cache 'Sync file *after* touching it'
13:55 logmsgbot: catrope synchronized php-1.5/cache/l10n/l10nupdate-et.cache 'Sync LU cache file'
13:52 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesEt.php 'Try again'
13:48 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesNds_nl.php 'Fix message that didn't get picked up'
13:46 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesEt.php 'See if this works'
13:20 RoanKattouw: Rebuilding all l10n caches to prep debugging a LocalisationUpdate issue
13:16 domas: something was wrong with sq31/sq32 - cpu/network shot through roof, can't correlate load
10:42 domas: *sigh*, squid conf: added highPrecedence to defaults, so it is picked up by configuration builder. also, checked in files into RCS (had to take over locks from mr.Starling, sorry!). unbroke test.wikipedia API
09:43 domas: where did test api,php fix go?!
04:38 Fred: removed Bart's ACL in squid cache conf since it was a duplicate ACL.

March 6

22:59 mark: Increased membufs on all backend squids
06:58 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'disabled FlaggedRevs_alpha due to outdated messages issues'
05:40 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php '$wgFlaggedRevsStylePath for labs'
05:35 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php
05:33 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'Set labs site to use flaggedrevs_alpha'
05:26 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'Set labs site to use flaggedrevs_alpha'
05:10 logmsgbot: aaron ran sync-common-all
00:18 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22728 - create new user group on Greek Wiktionary'

March 5

18:48 Fred: removed duplicate ACL from squid backend config.
18:11 Rob: restarted npre daemon on ms1 since it wasnt transmitting to nagios.
00:51 logmsgbot: aaron ran sync-common-all
00:23 logmsgbot: mark synchronized php-1.5/wmf-config/InitialiseSettings.php 'Restore Monobook for nl.wikimedia.org'

March 4

19:44 Tim: installed NRPE and RAID utils on ms1
19:40 Tim: removed db19 from the mysql node group and nagios since it is being used for varnish now
19:29 Tim: fixed prototype.wikimedia.org (nagios critical), was waiting 10s on every request due to memcached being down

March 3

16:16 apergos: reenables gmetric on ms8 a little bit ago, scrub finished with no errors (yeah right(
06:06 apergos: replication off again tonight, zfs nicely borked on ms8, zfs destroy snapshot X gives message "cannot destroy X, dataset already exists" . time for sun contract and call them
05:53 apergos: gmetrics turned off again on ms8, running zpool scrub export
05:40 apergos: reset SYs on ms8 worked (slowly). gmetrics reenabled. (and I thought morebots was just being a bad bot)
05:39 apergos: ( :-P ) starting catchup for the last day's lost replication ms7 -> ms8
05:22 apergos: reboot failed, bootadm hung probably waiting on zfs. stop SYS claims to work but subsequent show SYS shows power state on.
05:14 apergos: rebootiing ms8 to clear zfs deadlock

March 2

23:32 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/OptIn/SpecialOptIn.php 'r63191'
23:31 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/OptIn/OptIn.php 'r63191'
23:31 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Tweak OptIn settings'
22:36 RoanKattouw: Rebuilding localisation cache
20:15 Tim: on hume: started recompressTracked.php for all wikis, to fix any newly-tracked pointers from fixBug20757.php and to complete enwiki
03:48 atglenn: leaving replication and gmetrics turned off on ms8 for thenight, will go back to looking at scat and mdb output tomorrow.
02:27 atglenn: turning off gmetric on ms8, the dfs are piling up behind a deadlocked zfs list
02:13 atglenn: turn of ms7 replication to ms8 temporarily to work out a zfs deadlock issue on ms8

March 1

12:53 logmsgbot: kate synchronized php-1.5/wmf-config/db.php
12:51 river: removing db17 from rotation for ts dump
12:51 logmsgbot: kate synchronized php-1.5/wmf-config/db.php
11:48 domas: recovered locke, ran log collection with per-process limit of 200MB
11:39 domas: locke was killed by awks using gigabytes of memory, mwahaha.

February 27

02:09 Tim: rewrote domas's api.php fix and deployed my version
01:56 logmsgbot: midom synchronized php-1.5/includes/LocalisationCache.php 'removing profiling hooks'
01:42 domas: fixed api.php on test/srv124

February 26

23:40 Tim: disabled the temporary squid purge script, I'm assuming it's done its job now
19:43 logmsgbot: catrope synchronized php-1.5/includes/api/ApiBase.php 'r63014, r63015'
18:24 Fred: bugzilla now residing on Kaulen... Updating server roles.
18:07 Fred: switching bugzilla cname to new server. (both servers are pointing to the same DB, so switch should be transparent to most)
17:28 logmsgbot: midom synchronized php-1.5/includes/LocalisationCache.php 'profiling'
17:26 logmsgbot: midom synchronized php-1.5/includes/diff/DifferenceInterface.php 'removing profiling hook'
17:23 logmsgbot: midom synchronized php-1.5/includes/StubObject.php 'tracehooks'
17:21 logmsgbot: midom synchronized php-1.5/includes/StubObject.php 'tracehooks'
14:38 RoanKattouw: logmsgbot is missing in action AGAIN, starting it as me on fenari
14:37 RoanKattouw: Synced r63008
12:17 domas: *sigh*, recovered locke after OOM, added few entries to limits.conf

February 25

23:48 Tim: running the new fixBug20757.php on all wikis
21:48 RoanKattouw: Removed empty line from the end of all.dblist, broke account creation ^^
20:18 RoanKattouw: Removed those closed wikis again, db lists back to normal
20:08 RoanKattouw: Temp adding some closed wikis to all.dblist to be able to dump them (again)
19:01 Rob: removed old servers will, bart, etc from DNS
18:32 Rob: pushing more dns changes for LOM on new servers.
00:59 Tim: made the squid purge script on locke also purge redirects from /w/index.php?, for bug 22639
00:32 Tim: installed ack-grep on fenari

February 24

20:48 Rob: IP conflicts on alsted/ersch/formey/kaulen. I must have typo'd something in one of their DRAC configurations because they are all messed up. will go into DC and physically review settings via console.
16:10 Rob: had formey and alsted ip info mixed up in dns, updating dns
13:13 RoanKattouw: Adding optin_survey table to wikis that didn't have it
01:38 atglenn: ganglia 3.1 packages up on ms7 and ms8 (see atg for solaris packages for other boxen)
01:20 domas: that was fun, let's do it again!
01:17 mark: Implemented udp2log filter on locke that pipes redirected URLs to a modified version of purgeList to clean the squid caches
00:59 Rob: added austin to the mortals user category for shell access on puppet, updating singer's account list
00:30 mark: Fixing wikipedia.be vhost ServerAlias setting in redirects.conf
00:14 domas: redefined tiertwo acl in squid to allow fenari/zwinger purges, purged special:random, restarted mobile1 stuff

February 23

23:41 domas: stopped nginx on en.m until we resolve special:random issue
20:53 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22628 - Logo of Chamorro Wikipedia'
17:23 Rob: drac setup on new servers ersch, formey, alsted, and kaulen. still need port allocation.
16:47 Rob: forgot to add mgmt, updating dns
16:43 Rob: updating dns for new servers ersch, formey, alsted, & kaulen
15:37 Rob: FINALLY got in the last RMA drive for storage1, arrays were toast (losing 1/2 the drives in a raid10 and having the unlucky chance of them being paired), reinstall in progress
12:50 RoanKattouw: Cleaned up my dblist voodoo, everything should be back to normal now
12:41 RoanKattouw: Fixed sync-dblist, was broken
12:31 RoanKattouw: Syncing pmtpa.dblist anyway, seems I have to
12:24 RoanKattouw: Temporarily adding wikis listed in bug 20325 to pmtpa.dblist so I can export their contents. Please don't sync this

February 22

23:40 Tim: restarted replication on ms2
23:39 Tim: ms1 is up with a full copy of ms3 data, now replicating cluster22/rc1
21:09 logmsgbot: mark synchronized php-1.5/wmf-config/InitialiseSettings.php 'Revert'
20:30 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22522 - Allow bureaucrats to remove sysop flag on Simple English Wikitionary'
20:09 Rob: finished the setup of srv217 (just needed puppet fixed and files updated)
20:02 logmsgbot: root ran sync-common-all
20:01 logmsgbot: mark synchronized php-1.5/wmf-config/InitialiseSettings.php 'Set vector as default skin on nlwikimedia'
19:44 logmsgbot: root ran sync-common-all
18:10 Fred: ganglia is going to hiccup a little for the next few minutes while I restore data... "Don't panic"
07:53 Tim: rebooted srv206 via drac
07:48 logmsgbot: tstarling synchronized php-1.5/wmf-config/mc.php 'removing srv206, is down'
06:19 Tim: increased multicast TTL from 1 to 3 in gmond_template.erb, to fix ganglia on ms*
05:54 Tim: started ms2 -> ms1 copy using nc+tarpipe
05:30 Tim: shutting down mysql on ms2 for copy to ms1
05:09 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 'depooling ms2 for copy to ms1'

February 21

02:43 Fred: puppettized user gmetric

February 20

22:37 mark: Installed Ubuntu 8.04 on ms1 - now ready for ES deployment

February 19

18:48 Fred: removed memcached node srv193 and replaced with srv233
16:52 Rob: updated InitiliseSettings with bug 22585
01:31 Fred: puppetized gmond.conf generation using templates

February 18

23:51 RoanKattouw: Fixed search script on yongle, removed UA check exemption for yongle
23:51 RoanKattouw: logmsgbot is broken again
04:07 Tim: re-running storageTypeStats.php on enwiki to identify rows with old_flags='object,utf-8', which need to be handled properly in fixBug20757.php

February 17

17:25 Rob: pushed updates to wordpress installations
17:04 RoanKattouw: Synced wmf-config/checkers.php 'Allow missing UA from yongle'
17:04 RoanKattouw: Restarted logmsgbot
16:47 Fred: test
07:24 Fred: there is a problem with ganglia. It is being worked on.
00:15 RoanKattouw: Resynced srv151, was returning empty responses

February 16

23:31 Fred: upgrading gmond to 3.1.2 everywhere. However, due to the newish module structure, there is a potential that ganglia will hickup while puppet does its job...
22:03 logmsgbot_: mark synchronized php-1.5/wmf-config/checkers.php 'Exceptions'
20:30 Rob: srv127 is online, but not in LVS. Its cert was accepted on sockpuppet, but puppetd --test results in a cert failure on retrieval from sockpuppet.
20:17 Rob: rebooting srv127
20:15 mark: Increased membufs to 40 per COSS dir on the pmtpa upload backend squids
19:51 mark: Increased membufs per COSS dir from 10 to 20 on the new pmtpa squids
18:10 apergos: but documentation can save people precious time when things are on fire
18:02 mark: Documentation is not a substitution for thinking
16:44 mark: Fixed puppet on most servers
16:23 domas: anyone knows why mysqldump on snapshot3 is locking tables? maybe --single-transaction could work better?!!!?
16:22 RoanKattouw: Strike my last, I hear it'll fix itself in a day
16:21 RoanKattouw: Oops, meant to type srv187-189
16:21 RoanKattouw: Ganglia not picking up data for srv1987-189, 193, 214-218, 250-253, 257 even though the boxes are up and gmond is running; been like this for 3 days
15:21 mark: Removing all puppet certs/private keys on all machines
14:55 mark: Puppetmaster is screwed up by a wrong command that deleted all files under /var/lib/puppet
14:25 logmsgbot_: midom synchronized php-1.5/includes/diff/DifferenceInterface.php 'instrumenting costs by revision pairs'
13:46 logmsgbot_: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22444 and 22330 enabling collection on dawiki and ruwiki'
13:22 domas: oh wait, it does work, something else....
13:16 domas: apparently my squid rule for Apple browsers didn't really work, hence the continuing spikes
03:16 domas: blocked MacOSX Atom syndication on the edge, this will save terabytes of diskspace on unsuspecting Safari user computers :-)
02:13 logmsgbot_: tstarling synchronized php-1.5/wmf-config/checkers.php

February 15

23:42 logmsgbot_: midom synchronized php-1.5/includes/StubObject.php 'uninstrumenting'
23:09 logmsgbot_: midom synchronized php-1.5/wmf-config/checkers.php
23:09 domas: banned all UA-less requests
22:53 logmsgbot_: midom synchronized php-1.5/wmf-config/checkers.php 'un-disabling UA check on API'
22:14 RoanKattouw: Resynced srv114, was returning empty responses
22:10 logmsgbot_: mark synchronized php-1.5/wmf-config/CommonSettings.php 'Test with 220px default thumb size on enwiki (bug #21117)'
22:10 logmsgbot_: mark synchronized php-1.5/wmf-config/InitialiseSettings.php 'Test with 220px default thumb size on enwiki (bug #21117)'
22:02 logmsgbot_: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
21:58 logmsgbot_: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js 'r62553'
21:39 logmsgbot_: catrope synchronized php-1.5/includes/api/ApiMain.php 'r62550'
21:30 logmsgbot_: mark synchronized php-1.5/wmf-config/InitialiseSettings.php 'raised wgSearchSuggestCacheExpiry from 1200 to 86400'
21:27 logmsgbot_: catrope synchronized php-1.5/includes/api/ApiOpenSearch.php 'r62548'
21:27 logmsgbot_: catrope synchronized php-1.5/includes/api/ApiMain.php 'r62548'
21:27 logmsgbot_: catrope synchronized php-1.5/includes/DefaultSettings.php 'r62548'
20:54 logmsgbot_: catrope synchronized php-1.5/includes/api/ApiBase.php 'r62546'
20:54 logmsgbot_: catrope synchronized php-1.5/api.php 'r62546'
20:53 domas: Roan and me are doing some perf engineering on opensearch & related stuff. \o/
20:40 logmsgbot_: midom synchronized php-1.5/includes/StubObject.php 'readding stub trace'
20:34 logmsgbot_: midom synchronized php-1.5/wmf-config/checkers.php 'perf comment action'
20:30 logmsgbot_: midom synchronized php-1.5/includes/StubObject.php 'removing trace hooks'
20:28 logmsgbot_: midom synchronized php-1.5/includes/StubObject.php
20:24 logmsgbot_: catrope synchronized php-1.5/includes/api/ApiOpenSearch.php 'r62543'
19:32 mark: Allowing API requests to be sent to the regular app servers temporarily
19:27 mark: Restarted sq31 and sq31 API backend squids with slightly reduced memory settings
19:18 logmsgbot_: mark synchronized php-1.5/wmf-config/InitialiseSettings.php
17:51 Rob: set srv127 to false on lvs3, will re-enabled once in DC to reboot it manually
17:49 logmsgbot_: root synchronized php-1.5/wmf-config/mc.php 'srv127 down taking out of mc rotation'
16:02 RoanKattouw: Resynced srv134 to fix HTTP 500
15:13 RoanKattouw: Resynced srv232, was serving HTTP 500s
02:02 mark: Fixed stale PHP file removal apache cron job in Puppet

February 14

19:25 mark: srv206 spontaneously rebooted
17:12 mark: Added sq32 to the API backend squid group (with an empty cache)
16:38 mark: Confined all API requests to sq31 backend Squid
12:54 RoanKattouw: logmsgbot is broken

February 13

17:44 mark: Made 64 bit java exception for search13 in puppet
17:37 mark: Converted srv214-218 and srv86-87 into API servers
17:02 mark: Throwing all API requests at the new API app server cluster
16:49 mark: Converted srv187-srv189 and srv250-srv253 into API app servers
15:46 mark: Created new Ganglia group "API application servers", deploying with Puppet
15:31 mark: Created API app servers group in Puppet, deploying on srv254-257 to start with
15:24 mark: Created API LVS service on lvs3 and restarted PyBal
15:10 mark: Added DNS entry for api.svc.pmpta.wmnet (LVS ip)
14:18 mark: Enabled more verbose logging on all squids: debug message level 1 for all message categories
14:18 mark: Disabled cache digests on all squids
13:58 mark: Removing sq30 and lower from the Text CARP pool
13:51 mark: Reducing persistent_connection_timeout and request_timeout from 1 min to 20s on all frontend squids, to reduce FD usage
12:50 mark: Restarting knsq* frontend squids with FD limit raised from 32k to 64k
12:23 mark: Grown filesystem /dev/data/ES (/a) on ms3 by 90G, leaving another 98G free
00:17 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'Sync CSS style versions'
00:16 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
00:10 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r62403'
00:02 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js 'r62400'
00:01 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r62400'

February 12

23:55 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Try deploying r62397 again'
23:46 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/css/combined.min.css 'r62397'
23:46 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r62397'
23:22 mark: Deployed sq86
22:31 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
22:30 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r62388'
21:12 mark: Implemented new routing policy for AS43821
21:01 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
20:59 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r62382'
19:15 Rob: sq67 has both disks installed now (had to get one replaced) and has no OS installation yet. holding off on install, not sure if mark wants this up as a squid or for varnish development
18:22 atglenn: resstarting transfer of data (one chunk) from ms8->ms5->ms6, running in screen as root on all three hosts using nc
16:18 Rob: setup project1 and project2 boxes for flaggedrevsdevelopment and project mangement suite testing
15:12 mark: Upgraded our AMS-IX connection from 2x 1G to 10G along the way
15:12 mark: Reloaded br1-knams twice to fix a CAM partitioning problem; not sufficient IP next hops for 8-path routes
11:34 mark: Filtering all prefixes to/from AMS-IX peers
02:55 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 're-enabling WikiEditor'
02:04 Tim: installed openssh-server on bayes, Erik Z apparently uninstalled it with aptitude
01:37 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'disabled WikiEditor due to complaints about bug 22428'
01:06 atglenn: shoveling over some data from ms8 to ms6 via ms5 :-/
00:48 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/lqt.js 'Disable live preview entirely, broken by CentralNotice and DismissableSiteNotice'
00:37 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable DismissableSiteNotice for lqt.labs, breaks LiquidThreads live preview'
00:17 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version'
00:07 Andrew: Running scap
00:02 Andrew: Updating LiquidThreads alpha to trunk state, deploying with scap in a few minutes

February 11

22:06 Fred: implemented disk space check for tridge
21:10 mark: Pooled sq71-85 with full CARP weight (30)
21:06 mark: Pooled sq79-85 frontends in LVS with full load (30)
20:59 mark: Pooled sq79-85 frontends in LVS with load 1
20:51 mark: Pooling sq79-85 in CARP on all frontends, with low CARP weight 10 instead of 30
20:30 mark: Pooling sq79-85 in CARP on frontend sq51 to seed the cache
19:39 atglenn: replication every 15 minutes enabled from ms7 to ms8
19:31 mark: Removed sq16-30 from the text frontend squid pool in LVS - these servers will soon be decommissioned
19:28 mark: Pooled sq71-78 frontend squids in LVS with full load (30)
19:21 mark: Pooled sq71-78 frontend squids in LVS with low load (1) to seed caches
18:30 mark: Pooling sq71-78 backend squids on all frontends, with lower CARP weight (10 instead of 30) to seed the caches
18:04 mark: Deployed new squid config to sq58-66 frontend squid, to seed the caches of backend squids sq71-78
17:27 mark: Deployed new squid config to sq66 frontend squid, to seed the caches of backend squids sq71-78
17:24 mark: Deployed Squid with correct configuration on sq71-78
11:08 RoanKattouw: Load spike on 4CPU Apaches ended, rr.knams back up, downtime seems to be over
11:05 RoanKattouw: Another load spike on the 8 CPU Apaches, approx 11:02-11:-05 UTC
11:00 RoanKattouw: Apache load spike started around 10:42 UTC and coincides with a load dip followed by a monitoring blackout on the Kennisnet Squids and a load spike on the Tampa Squids
10:58 RoanKattouw: 8CPU Apaches stopped going bonkers all of a sudden according to Ganglia, 4CPU ones still have high CPU usage
10:56 RoanKattouw: Site went down, all Apaches have high CPU usage

February 10

22:31 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix to deploy usability changes sitewide'
22:22 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploy r62275 to test'
21:28 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploy r62264 for real. Running svn up first helps'
21:19 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploy r62264'
21:03 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php
21:02 Andrew: Switching strategywiki to opt-out LiquidThreads, rather than opt-in
20:46 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js 'Deploy r62262'
20:46 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploy r62262'
19:22 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
19:21 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploying r62257'
19:05 Rob: storage1 disks replaced, except for the newly dead one, rma placed.
18:39 Rob: shutting down storage1 to replace its bad OS mirror disk
16:52 mark: Fixed MySQL permissions on srv186 as well
16:15 mark: Rigged puppet to deploy a PHP mail.ini file to the apaches, which sets it to call sendmail with -f <> (sending mail with empty envelope sender, like bounces)

February 9

22:06 Andrew: LiquidThreads production software updates completed successfully.
21:58 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/Dispatch.php
21:57 mark: Firewalled one ip on singer which was overloading Apache, looked like a Wordpress attack
21:50 mark: Lowered MaxClients setting to 250 on singer
21:47 Andrew: Running scap to deploy LiquidThreads updates
21:45 Andrew: Updating LiquidThreads production deployments to the alpha version in the next few minutes.
21:44 mark: Power cycled singer
21:19 mark: Fixed MySQL system user & permissions on srv151-srv185
21:17 mark: Fixed MySQL instance on srv183
20:47 mark: Fixed MySQL instance on srv156
19:39 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
19:29 logmsgbot: catrope synchronized php-1.5/skins/common/edit.js 'Deploy r62190. Only for test for now, thank God for style versions'
17:01 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
17:01 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploying r62187'
16:52 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
16:51 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploying r62184'
14:19 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
14:19 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploying r62181'
02:14 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/lqt.js 'Merge r62158'

February 8

23:25 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix for r62041. Varnish needs this or Tampa will serve stale JS off bits'
23:06 mark: Deploying Exim on all application servers as well
22:06 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/OptIn/SpecialOptIn.php 'Deploy r62141'
21:51 mark: Added search3 and search9 to search_pool_1 on lvs3
21:48 mark: Deploying Exim as a simple mail sending-only MTA (with queuing enabled) on squids, image scalers, search servers and core databases
21:38 Andrew: LiquidThreads wikinews rollout complete, though there was some buggage related to replication that had to be resolved.
21:29 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Dispatch.php
21:29 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Dispatch.php
21:27 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Dispatch.php
21:24 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Dispatch.php
21:22 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Dispatch.php
21:21 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Dispatch.php
21:18 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php
21:18 logmsgbot: andrew synchronized php-1.5/wmf-config/liquidthreads.php
21:13 Andrew: Starting LiquidThreads deployment on enwikinews.
20:00 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Track NTOC and dialogs as well in PrefStats'
13:04 Tim: running fixBug20757.php on enwiki, in a screen on hume
12:51 Tim: locke went into swap half an hour ago and is now dead. Trying reboot.
07:38 Tim: stopped recompressTracked.php for now due to bug 20757
07:04 logmsgbot: tstarling synchronized php-1.5/includes/Revision.php 'r62120'
06:18 Tim: cleared out db9 root partition (which had 370MB free) by moving 4GB of SQL files from /root and /home/tfinc to /a/backup/junk
06:01 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 're-added ms2 as an ES slave for rc1,cluster22 and the ex-fedora clusters, was depooled for ~6 months'
05:55 Tim: added grant for root@fenari to ms3/2
05:31 Tim: started apache on srv213, srv161 and srv240, was stopped for no apparent reason
04:52 logmsgbot: andrew synchronized php-1.5/includes/ChangesList.php 'Merge r62117'
04:37 Andrew: Deploying LiquidThreads alpha updates with scap
04:23 Andrew: Planning to deploy LiquidThreads alpha (liquidthreads.labs and test) in the next few minutes
01:15 Tim: added mysql and gmetric users on srv155 temporarily, to get those services back up
00:42 Tim: ran apt-get upgrade on srv155. Doing reboot test for new kernel.

February 7

23:36 Andrew: Clarification: Somebody should fix it, not I think it's fixed now.
23:35 Andrew: The issue of centralnotice pulling notices from the wrong place should be fixed now, though
23:34 Andrew: Moved /mnt/upload5/centralnotice/centralnotice.js and /mnt/upload5/centralnotice/wikipedia/centralnotice.js to centralnotice-old.js, should stop fundraising banners from appearing.
23:33 Andrew: Issue with fundraising banners appearing on random pages caused by clients with the URL http://upload.wikimedia.org/centralnotice/wikipedia//centralnotice.js?257z6
22:11 Andrew: Reports that fundraising banners are appearing on random pages for some users.
21:32 domas: we're running on single ES copy.
21:32 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'disabling broken storage1'
20:05 domas: removing all fixes I've been doing to recover mail. WMF staff will handle this tomorrow, or the problems will be gone by then.
19:00 domas: blackholed few thousand IPs on mchenry via OUTPUT chain (using -j REJECT action)

February 6

19:55 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php
01:09 logmsgbot: andrew synchronized php-1.5/extensions/CentralAuth/CentralAuth.php 'Merge r61717, because Casey Brown asked me nicely.'
01:08 logmsgbot: andrew synchronized php-1.5/extensions/CentralAuth/ApiQueryGlobalUserInfo.php 'Merge r61717, because Casey Brown asked me nicely.'
00:58 Andrew: Scapping to deploy r62055 'For usability initiative: merge r62041, r62043'

February 5

06:00 logmsgbot: andrew synchronized php-1.5/wmf-config/liquidthreads.php 'Re-activate LiquidThreads email notifications, DoS has passed'
05:22 Tim: freed up a little space on ms3 by reducing expire_logs_days from 14 to 7, and running flush logs
02:45 apergos: cleared out /tmp on srv169...
00:23 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'Bump style version for jQuery UI stylesheet'
00:06 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'Bump style version for combined.min.css'
00:03 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/css/combined.min.css 'Resync this, doesn't seem to have been picked up right'

February 4

23:55 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
23:55 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js 'r62002'
23:50 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
23:37 RoanKattouw: Running scap to deploy new UsabilityInitiative code
22:55 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump UsabilityInitiative_alpha style version appendix'
22:54 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins.combined.min.js 'r61995'
22:28 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable UsabilityInitiative_alpha on usabilitywiki'
22:20 logmsgbot: tstarling synchronized php-1.5/includes/api/ApiUpload.php 'disabled chunk upload'
22:09 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump UsabilityInitiative_alpha style version appendix'
22:07 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.hooks.php
22:06 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/UsabilityInitiative.hooks.php
22:06 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/css/combined.min.css
22:06 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.combined.min.js
22:06 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins.combined.min.js
22:05 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/images/wikiEditor/dialogs/insert-link-external-rtl.png
22:04 RoanKattouw: Deploying r61993 with individual sync-files
21:15 atglenn: turned off hourly snaps on ms7, we'll have daily plus the standard replication
18:49 atglenn: cleared /tmp on srv175 blah blah
18:36 atglenn: cleared out php* files from /tmp on srv163 (need cron job)
10:35 mark: Removed sq11-15 from the upload LVS pool
10:30 mark: Removed sq11-15 from the upload CARP pool
03:28 apergos: added /usr/local/apache/common/php to include path of php on fenari

February 3

23:56 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump UsabilityInitiative_alpha style version appendix'
23:55 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.hooks.php
23:55 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins/jquery.wikiEditor.html
23:55 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins.combined.min.js
23:55 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.combined.min.js
23:55 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/css/combined.min.css
23:54 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/UsabilityInitiative.hooks.php
23:54 RoanKattouw: Deploying r61959 using individual sync-files
22:35 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Explicitly enable EditWarning by default here. Previously done in extension file'
17:32 mark: Set root password on all servers using Puppet
17:26 Rob: loaded sq68 full with 6 ssd, had to 'borrow' drives from the unracked squids in pmtpa awaiting the rail issue resolution.
17:26 Rob: connected sq67 and sq68 to secondary network ports as requested
16:43 Rob: had to update gmetad and add sq14 and sq17 into puppet as ganglia aggregator hosts. Currently not showing in ganglia, but should when puppet updates their files shortly.
16:37 Rob: updated gmetad_pmtpa in NFS store, as well as on zwinger and spence.
16:20 Rob: !log cleaned sq1-sq10 certificates off sockpuppet
16:10 mark: Rebooting sockpuppet for security upgrades
16:06 Rob: updating DNS to remove lvs3.wikimedia.org, as its technically lvs3.pmtpa.wmnet
16:03 Rob: removed sq1-sq10 in dsh groups, server roles, pybal (already done, just doublechecked), and now starting wipe and removing their network connections.
15:37 Rob: updated video plugin on techblog, had to comment out sumotv support line due to errors.
13:30 mark: Removed sq1-10 from the upload backend Squid pool, preparing for decommissioning
12:27 mark: Removed eiximenis from the text backend pool
12:21 mark: Restarted sq51 with hyperthreading disabled
11:31 Tim-away: exim was flooding its mainlog with errors like "User 'exim' has exceeded the 'max_user_connections' resource (current value: 150)". OTRS is apparently broken as a result. Set the limit to infinity to fix it.
11:10 Tim-away: debugging mysql connection errors from mchenry to db9
11:02 Tim-away: restarting exim4 on mchenry
10:57 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/Vector/Vector.hooks.php 'r61922'
10:47 RoanKattouw: Mail delivery reported broken again; last mail delivery to mediawiki-cvs was 3h22m ago
10:14 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump UsabilityInitiative_alpha style version'
10:13 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins/jquery.wikiEditor.html
10:13 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/UsabilityInitiative.hooks.php
10:12 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins.combined.min.js
10:12 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/Vector/Vector.hooks.php
10:12 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/Vector/Vector.combined.min.js
10:12 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.hooks.php
10:12 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.combined.min.js
10:09 RoanKattouw: Deploying r61919 with individual sync-files
07:45 Tim: killed eximstats on mchenry, was sending the machine into swap
06:02 Andrew: Running scap to deploy LiquidThreads alpha updates
05:09 Andrew: Planning to update LiquidThreads alpha to trunk state in the next few minutes.
03:31 Tim: removing refreshLinks2 jobs from the enwiki job queue with namespace=0, 100k of them put there by a nasty biography template
00:34 domas: restarted varnish on db19

February 2

23:26 mark: PyBal got into a confused state due to a duplicate LVS realserver entry. Set up LVS manually/statically for 30 mins to debug the problem. PyBal is now active again.
23:04 Tim: pmtpa text squids down
22:33 mark: Pooled sq59-68 in LVS (Text)
22:27 apergos: restarted apache on srv170 (corrupted apc cache)
22:25 mark: Increased CARP weight of the new Upload squids sq51-58 from 10 to 30
22:17 mark: Added sq59-66 to the Text pool of Squids, with low CARP weight (10) to seed the caches
21:44 logmsgbot: andrew synchronized php-1.5/wmf-config/liquidthreads.php
21:42 Andrew: temporarily disabling $wgLqtEnotif to mitigate mail server DoS impact on LiquidThreads post/reply speed.
21:42 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/NewMessagesController.php 'Deploy r61879'
21:42 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/NewMessagesController.php 'Deploy r61879'
21:04 mark: Removed sq2-10 from the LVS pool
20:46 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version addition for usability alpha'
20:45 logmsgbot: catrope synchronized php-1.5/skins/common/jquery.min.js 'Deploing r61700'
20:38 mark: Added sq52-58 to the backend Squid pool with low CARP weight, to seed the caches
20:37 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22351: Enable Vector and toolbar on srwikinews'
20:36 mark: Increased Exim load average threshold for delivery on mchenry
19:09 Fred: rebooted singer (went down for unknown- at this time- reasons).
18:25 Fred: OS-installed sq52-ssq58
18:14 mark: OS-installed sq59-sq66
17:23 mark: Thrown sq66 backend squid in the Squid pool, CARP weight 30
06:01 Tim: deploying r61846 via scap
04:25 Andrew: Schema change successfully applied.
04:24 Andrew: Applying schema change thread_signature.sql to mediawikiwiki, strategywiki, testwiki
04:22 Andrew: Applying schema change thread_signature.sql to liquidthreads_labswikimedia
01:31 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Fix for UsabilityInitiative_alpha config'
01:18 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.combined.min.js 'r61842'
01:13 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Append to $wgStyleVersion when running usability alpha'
01:10 logmsgbot: catrope synchronized php-1.5/includes/OutputPage.php 'r61841 for real this time'
01:09 logmsgbot: catrope ran sync-common-all
01:07 logmsgbot: catrope synchronized php-1.5/includes/OutputPage.php 'r61841'
01:05 logmsgbot: catrope synchronized php-1.5/includes/OutputPage.php 'r61839'
00:51 RoanKattouw: Running scap
00:48 logmsgbot: catrope synchronized php-1.5/includes/OutputPage.php 'r61837'
00:40 RoanKattouw: Synced extensions/UsabilityInitiative_alpha by hand
00:07 logmsgbot: catrope synchronized php-1.5/skins/common/jquery.min.js
00:07 RoanKattouw: Syncing some CSS and JS files for usability deployment so bits.wikimedia.org will pick them up
00:06 logmsgbot: catrope synchronized php-1.5/skins/common/wikibits.js 'Sync changes to wikibits.js in r61558'

February 1

23:30 RoanKattouw: Testing new usability code on test
23:27 mark: Pooled sq51 frontend squid in the upload LVS cluster, weight 30
23:03 mark: Thrown sq51 backend squid in the Squid pool, CARP weight 30
20:50 Rob: srv217 shutdown until checked
20:50 Rob: srv217 not responsive to network, but LOM works and system is jut not working online, possible bad cable, will check in DC tomorrow
20:43 logmsgbot: mark synchronized php-1.5/wmf-config/CommonSettings.php 'Added 36 new Squid servers, removed bart'
20:39 mark: Deployed new wikimedia-base package that no longer manages sysctl.conf post Hardy, letting puppet handle it
20:10 atglenn: restarted webserver on ms7 to read new conf (fix bug 22321)
20:09 mark: Granted cary access to the OTRS db from fenari
19:27 RoanKattouw: srv9 reported down, pinging yields Host Unreachable. srv9 is not in Nagios but seems to be in use
18:51 Fred: swapped memcache node from srv145 to srv196

January 31

21:18 domas: db28 has a failed dimm

January 30

22:29 RoanKattouw: Scheduling more batches for import overnight using at(1), see http://commons.wikimedia.org/wiki/Commons:Batch_uploading/Geograph#Progress
22:13 RoanKattouw: Running image imports overnight on hume, three parallel import scripts started each five hours, see http://commons.wikimedia.org/wiki/Commons:Batch_uploading/Geograph for details
22:12 apergos: restarted morebots, must have died after last night's freenode irc move
04:40 Tim: fixed broken /etc/rc.local on wikitech
04:18 Tim: restarting mysqld on db12
04:00 Tim: analysis: mysqld on db12 hit a bug at 02:45 and froze, most threads in futex. The mysql client failed to set a read timeout, leading to db12 sucking up all available apache threads. Several squids became overloaded, presumably due to the large size of our 503 error messages.
03:21 Tim: restarted all apaches, thereby killing the reads which started before 3:11, possibly as early as 02:45. Site service resumes immediately. ps -lL on db12: http://p.defau.lt/?Jw1EBE0fnV4Rpxe9OP8yAw
03:18 Tim: finally thought to answer the question "are the apaches waiting for something or idle?" Strace confirms that they are waiting for db12 in read().
03:11 Tim: Having trouble believing db12 could be responsible for the CPU spike on the squids. Depooled db12 just in case.
02:56 Andrew phones Tim
02:45 retrolog: db12 ganglia graphs go flat. Apache CPU goes down to near zero. CPU saturates on some of the text squids: sq16, sq17, sq18, sq21, sq22, sq23, sq25, sq26. Frontend squids serve 503s.
02:00 retrolog: Network spike on db12

January 29

23:21 domas: restarted sq37 for being in bad shape.
22:25 tomaszf: starting xml snapshots back up on snapshots3
21:31 atglenn: cleared out cruft from /tmp on srv158 (how about a cron job? :-P )
18:39 logmsgbot: catrope synchronized php-1.5/maintenance/importImages.php 'Deploy r59908'
16:20 Rob: db28 mainboard replaced, updated LOM to work, updated dhcp with new mac address info
02:50 apergos: started copy of media data from ms7 to ms8 (nc running in screen as root on both hosts)
01:51 logmsgbot: andrew synchronized php-1.5/cache/interwiki.cdb 'Updating interwiki cache for new wikitech entry'

January 28

20:15 Rob: racked sq51-sq68, all drac accessible, wired, ready for install
19:05 Rob: updated dns for management ranges on new squid servers
13:56 mark: Added DNS entries for 36 new Squid servers
13:28 mark: Storage1 went down due to I/O failure
13:00 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Remove aliases made obsolete by r61632'
12:55 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesMl.php 'Deploying r61632'
10:57 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'supressredirect for rollbackers on ruwikisource'
09:30 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'Removed redundant huwiki settings'

January 27

18:04 mark: Fixed ExtensionDistributor, which broke due to the image server migration
18:02 logmsgbot: mark synchronized php-1.5/extensions/ExtensionDistributor/svn-invoker.php 's/upload5/upload6/'
16:10 mark: Added a timeout to puppet's apt-get update execution; hopefully this will prevent things from getting stuck for weeks
16:08 mark: Shutdown sq20 once again, please decommission this broken box
16:05 mark: Fixed puppet on db7
16:05 mark: Killed stuck apt-get update processes on all servers
15:58 mark: Made puppet manage NFS mounts /mnt/upload6, /mnt/thumbs and the absence of /mnt/upload5 on all application servers and image scalers
13:37 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Fix Category talk alias on mlwiki'
11:57 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Also add a Unicode 5.1 alias for Category talk on mlwiki'
11:52 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22286: Unicode 5.1 namespace alias for Category on mlwiki'
11:46 logmsgbot: catrope synchronized php-1.5/includes/api/ApiQueryBacklinks.php 'Deploying r61571'
08:12 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/lqt.js
08:11 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/lqt.js

January 26

23:44 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'Enable file uploads.'
23:34 Fred: restarted morebot
23:31 atglenn: turning off web server on ms1 (previously: moved uploads to ms1, mounted ms7 everywhere, updated squid conf)
21:10 atglenn: copying uploaded data since last replication, from ms1 to ms7
21:05 atglenn: turned off file uploads temporarily
21:04 logmsgbot: ariel synchronized wmf-deployment/wmf-config/CommonSettings.php
20:50 Rob: rebooted srv125 per dc task entry by fred
20:35 tomaszf: enabling global file uploads notice
20:29 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/GlobalUsage.php 'Deploying r61534'
20:29 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/GlobalUsageHooks.php 'Deploying r61534'
18:38 Rob: shutting down pdf2 to relocate its rack position.
18:37 Rob: !log mobile2 port not moved yet, as its port assignment on wikitech is invalid
18:08 Rob: shutting down mobile2 to relocate its rack location
18:07 Rob: bayes back online
18:00 Rob: bayes management lom had ip issues, resolved, still working on host
17:52 Rob: bayes is not coming back online properly, checking.
17:47 Rob: bayes rack location moved and updated in racktables, powering back up
17:34 Rob: shutting down bayes to move its rack location from b2-pmtpa to a2-pmtpa
14:42 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22274: Set $wgAutoConfirmCount to 10 on itwiktionary'
00:56 Tim: on hume: increased RCT proc count, reduced max lag to 1s, increased lag check frequency

January 25

22:36 Tim: ms3 is reporting "table full" for rc1 enwiki.blobs. Running ALTER TABLE.
19:33 Fred: stopped apache on srv225
19:01 Fred: adjusted apache.conf on srv225 to make it work again...
16:07 Rob: powering srv224 & srv225 back up
16:01 Rob: shutting down srv225 & srv224 to swap their power cables from single to Y split.
15:45 Rob: srv196 memory replaced, back online and in service
15:45 logmsgbot: root synchronized php-1.5/wmf-config/mc.php 'pushing out change to two hosts that were down a few minutes ago'
15:40 Rob: shutting down db28 to swap memory around in diagnostics (bad memory in system)
15:25 logmsgbot: root synchronized php-1.5/wmf-config/mc.php 'took down srv196 for hardware replacement'
15:20 Rob: shutting down srv196 to replace faulty memory dimm
12:19 Tim: started recompressTracked.sh on hume

January 23

00:16 aZaFred_: implemented ganglia detailled mysql stats for db10.
00:16 aZaFred_: implemented DB Lag detection script for DB10

January 22

23:08 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/ClickTracking/ClickTracking.hooks.php 'Deploy r61401'
23:07 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/CollapsibleTabs/CollapsibleTabs.hooks.php 'Deploy r61403'
21:29 brion: freed some space on srv218 removing old upload temp files from /tmp but / is still pretty tight (445m free)
20:19 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaLicenseTexts.i18n.php 'Deploying r61387'
19:05 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22201: Add WZ: alias on itwiktionary'
15:22 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable $wgEnotifWatchlist on usabilitywiki'
11:38 mark: Deployed new gmond 3.0.3-2 package that conflicts with the incompatible ganglia 3.1 packages in Ubuntu, to avoid problems in our upcoming upgrade

January 21

14:13 mark: Added srv210 to LVS
14:11 mark: Reenabled srv245 in LVS
13:58 mark: Restarted Apache on srv207, srv210, srv233, srv257, srv245
09:30 mark: Replaced srv121 (decommissioned) by search13 in smokeping
07:59 AaronSchulz: Removed empty ct_tag rows from code_tags
07:47 Tim: cleaned up /tmp on srv180 (was out of disk space)

January 20

20:36 logmsgbot: root synchronized php-1.5/wmf-config/mc.php
16:38 Fred: killed long running queries created by civicrm on db9.
13:56 domas: did lots of changes on wikitech wiki. :)
02:37 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable API logging on enwiki'
01:20 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enabling api logging for enwiki'
01:16 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disabling API request logging on dewiki'
00:03 RoanKattouw: Strike my last, was actually *enabling* it on dewiki
00:03 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable API logging test on dewiki'
00:02 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable API logging test on frwiki'

January 19

21:37 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enabling API request logging on frwiki'
21:36 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Enabling API request logging on frwiki'
20:34 Fred: bugzilla upgrade completed. It might be up to an hour before attachments can be added due to DNS propagation.
19:41 Fred: Bugzilla as you know it is going down. Be back shortly.
18:21 Rob: updated blog.wikimedia.org wp-stats plugin to 1.6.1
16:03 Rob: ssds installed in db28, during boot it displays a memory error, investigating.
15:20 Rob: db28 offline while disks are swapped to solid state for testing.

January 18

05:02 Tim: removed some old xff logs to free up space for logrotate which I ran with the -f flag
05:00 Tim: fixed logrotate on nfs1, broken due to duplicated entries between /etc/logrotate.d/rsyslog and /etc/logrotate.d/syslog-ng. Left the syslog-ng ones in for now.
04:02 RoanKattouw: NFS /home is full

January 16

14:55 mark: Started backend squid on sq50
14:26 mark: sq19 has bad drive /dev/sdc
14:22 mark: dist-upgrade & reboot on sq19
14:14 mark: Shutdown sq20, bad disk /dev/sda
14:13 mark: Reenabled sq24 frontend in PyBal
14:03 Tim-away: disk space critical on srv167, cleaned up /tmp
06:07 apergos: "recovery", my *ss... cleaned up /tmp on srv181 to get some space back

January 15

23:53 tomaszf: starting test dewiki snapshot on snapshot2
17:17 Fred: modified gmetad config on zwinger and spence to reflect new apache 4cpu aggregator
17:16 Fred: added srv149 as a gmond aggregator in puppet.

January 14

17:28 domas: load-tested and fixed db19 to handle full bits workload (~22k/s), now again serving just tampa part.
01:32 apergos: moved bits to .2 again, all nameservers seem to be up and reflect the change (bits unresponsive again)

January 13

18:30 Andrew: [andrew@zwinger ~]$ sync-file wmf-config/CommonSettings.php 'Disable GIF scaling again, due to issues reported on village pump, bug 22041'
17:05 Rob: updated InitialiseSettings for bug 21174 and 21077
16:24 Rob: updated InitialiseSettings.php Bug 20508 Please enable Extention:NewUserMessage on en.Wikinews
11:50 domas: started overflow-watchdog ( http://p.defau.lt/?4_Y_Lrl9tVKS6fEYwkswKA ) on db19, sent bits load at it again
01:23 atglenn: using .2 for bits.pmtpa, did authdns-update, let's see what happens (bits was failing to respond)

January 12

19:52 rainman-sr: could someone please put search3,9 into search_pool 1 (with search1,4) on lvs3
19:37 Fred: Puppet: set spence as a ganglia aggregator for Misc tree.
19:23 Rob: usability/prototype linode was crashed, had to reboot
17:39 Fred: set wgDefaultSkin back to monobook on wikitech since vector is not operational.
16:08 mark: Arcor clients appear to have problems reaching our sites, traffic to Arcor over AMS-IX has been low since midnight UTC
15:52 Rob: srv222 & srv223 power reestablished, booting.
15:47 Rob: shutting down srv223 & srv222 to change out power cords.
15:46 Rob: db10 moved and back online.
15:36 Rob: db10 moved to sdtpa a2, powering up.
14:50 Rob: taking down db10 to relocate from pmtpa-b1 to sdtpa-a2
14:50 Rob: fixed issues with transcode2 and transcode3, completing base installation.
05:48 logmsgbot: tstarling synchronized php-1.5/includes/HTMLCacheUpdate.php
05:48 logmsgbot: tstarling synchronized php-1.5/includes/BacklinkCache.php
05:47 Tim: deploying r60962
01:17 Tim: on streber: removed a corrupt torrus DB file so it could be rebuilt, torrus should be working now
00:57 Tim: killed frozen torrus cron jobs and ran "torrus compile --tree=Network --force"
00:51 Tim: maybe torrus collector is still broken, trying /etc/init.d/torrus-common force-reload
00:46 Tim: with mpm-prefork managed to debug it fairly easily. Moved away permanently locked DB file render_cache.db, torrus.wikimedia.org is now fixed
00:39 Fred: restarting pdns on ns1
00:38 Tim: switching streber to apache2-mpm-prefork, can't work out why it's not working
00:22 Tim: trying "apache2 -X" on streber
00:00 Tim: restarting apache on streber

January 11

23:38 domas: logging the fact that we had cache layer meltdown at some point in time during the day
22:30 domas: leaving bits.pmtpa on db19's varnish, in case of troubles - uncomment bits.pmtpa .2 record in /etc/powerdns/templates/wikimedia.org and run authdns-update
19:43 logmsgbot: fvassard synchronized php-1.5/wmf-config/mc.php 'Swapped memcached from srv125 to srv232'
19:06 Rob: new apaches srv255, srv257 deployed. Updated node groups and synced nagios
19:03 Rob: new apache server srv254 deployed
18:24 atglenn: copy backlog of image data from ms1 to ms7 (running in screen as root on both boxes)
14:43 mark: Rebooting fuchsia, locked up again
14:24 mark: Increased load on knsq16-22 by upping lvs weight from 10 to 15

January 10

23:02 logmsgbot: midom synchronized php-1.5/wmf-config/lucene.php 'rainman asked, rainman guilty, hehehe'
23:01 logmsgbot: midom synchronized php-1.5/wmf-config/secure.php
17:36 rainman-sr: search limit raised to 500 again, interwiki search re-enabled for "other" wikis
16:07 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 'take ixia back out'
16:06 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 'put ixia back'
16:06 rainman-sr: restarting search cluster to deploy search13-19
15:58 domas: all bits serving switched back to text cluster, we have problems with all threads blocking on write(): http://p.defau.lt/?dOBxveiHj_ukjzupEBX3rA
15:19 rainman-sr: configuring search13-19, will leave search20 as spare
14:22 domas: apparently varnish worker threads are blocking on network output, ... :)
12:40 domas: full bits pmtpa load sent to sq1
12:04 domas: sending half of bits load to sq1
12:02 domas: set up separate geo balancing for bits via bits-geo.wikimedia.org
10:45 logmsgbot: midom synchronized php-1.5/wmf-config/CommonSettings.php 'setting extension asset path to bits.wm'
10:42 logmsgbot: midom synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php
10:29 logmsgbot: midom synchronized php-1.5/includes/Setup.php
10:29 logmsgbot: midom synchronized php-1.5/includes/DefaultSettings.php
09:36 domas: moving over static assets to 'bits.wikimedia.org'
09:11 logmsgbot: midom synchronized php-1.5/wmf-config/CommonSettings.php
09:09 logmsgbot: midom synchronized php-1.5/wmf-config/secure.php
08:26 logmsgbot: kate synchronized php-1.5/wmf-config/db.php
08:25 river: taking ixia out of rotation to dump commons

January 9

17:31 mark: Upgraded pdns-recursor to 3.1.7.2 on dobson, mchenry, lily
15:32 mark: Temporarily filtering all prefixes from 1299 on br1-knams, due to some balanced link blackholing issue
13:10 logmsgbot: midom ran sync-common-all
12:45 domas: restarted, that is :)
12:44 domas: fixed ns1, was deadlocked

January 8

21:48 Rob: nagios is flapping errors for esams hosts, but they are still up and functional. Perhaps due to new transit setup earlier today.
20:10 Rob: pushing dns update for flaggedrevssandbox project
16:09 Rob: finished mobile2 initial setup, gave hcatlin sudo rights to server
15:46 mark: Brought up transit session to 1257 on br1-knams
14:06 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
11:42 logmsgbot: midom synchronized php-1.5/wmf-config/db.php

January 7

20:22 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22038: Enable Collection extension on skwiki'
20:21 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 19418: Fix $wgUploadNavigationUrl on bnwiki'
20:16 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21517: Enable patrolling on frwikiversity'
19:54 Rob: base install done on mobile2, pdf2, transcode1, transcode2
16:55 Rob: srv254-srv257 racked, drac online, network attached. needs installs
15:46 Rob: racked srv254-srv257. DRAC setup, cables run. Network not plugged in until mark provisions the ports.
02:51 Tim: restarted apache2 on wikitech to fix swapping
02:27 Fred: rebooted pascal as it was once again hung
01:57 mark: Prepended AS 43821 once on incoming prefixes from AS 1299

January 6

22:30 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable OptIn on trwikimedia'
17:17 Rob: updated dns for tr.wikimedia.org
14:47 Rob: updated techblog software to newest stable revision
14:01 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22027: Enable AbuseFilter and CAPTCHA on trwiki'
13:27 river: restarted slave on db26
12:56 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 19418: Set $wgUploadNavigationUrl on bnwiki'
12:37 logmsgbot: catrope synchronized php-1.5/includes/GlobalFunctions.php 'remove debugging code from yesterday'
12:34 RoanKattouw: Resuming GlobalUsage rebuild
08:48 logmsgbot: kate synchronized php-1.5/wmf-config/db.php
08:47 river: removing db26 from s1 to dump

January 5

23:57 RoanKattouw: GlobalUsage refresh script properly handling replag now. Aborted it, will resume in the morning (CET)
23:56 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/refreshGlobalimagelinks.php 'remove debugging code'
23:54 RoanKattouw: Resuming GlobalUsage refresh script
23:53 logmsgbot: catrope synchronized php-1.5/includes/db/LoadBalancer.php 'Deploying r60705'
23:53 logmsgbot: catrope synchronized php-1.5/includes/GlobalFunctions.php 'Deploying r60705'
22:21 logmsgbot: catrope synchronized php-1.5/includes/GlobalFunctions.php 'Debugging'
22:11 RoanKattouw: Aborting GU refresh *again*, somehow it's not detecting slave lag right
22:07 RoanKattouw: Resuming GlobalUsage refresh script
22:04 logmsgbot: catrope synchronized php-1.5/includes/GlobalFunctions.php 'Deploy r60695 (wfWaitForSlaves fix)'
21:44 RoanKattouw: Running refreshGlobalimagelinks.php on all wikis from the smallest up
20:50 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/SpecialGlobalUsage.php 'Deploy r60692 (GlobalUsage fixes)'
20:14 logmsgbot: fvassard synchronized php-1.5/wmf-config/InitialiseSettings.php 'added outreach namespace to officewiki'
19:54 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21517: Add patroller group on frwikiversity'
19:28 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable GlobalUsage on all public wikis'
17:10 logmsgbot: fvassard synchronized php-1.5/wmf-config/mc.php 'Swapping srv110 for spare srv231.'
16:41 Fred: restarted memcached on srv110
16:10 mark: Installed search13-20 as new search servers
15:50 mark: Installed karmic on search19
15:45 Rob: updating dns for mobile2 and pdf2 servers
12:10 mark: dist-upgrade and reboot on sockpuppet
12:01 river: copied old wmvids to ~kate/wmvids
11:37 mark: Installed OS on search18
07:25 logmsgbot: tstarling synchronized php-1.5/extensions/CodeReview/ui/CodeRevisionView.php
07:25 logmsgbot: tstarling synchronized php-1.5/extensions/CodeReview/api/ApiCodeDiff.php
07:24 logmsgbot: tstarling synchronized php-1.5/extensions/CodeReview/CodeReview.i18n.php
07:24 logmsgbot: tstarling synchronized php-1.5/extensions/CodeReview/CodeReview.php
06:48 Tim: restarted apache on srv199, APC cache corruption
02:33 tomaszf: compressing support-requests on locke to free up space
02:14 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 'disabled $wgCopyUpload and $wgHTTPProxy on commons and test, breaks Lucene Search'

January 4

22:15 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22002: Add Opinions namespace on bgwikinews'
21:51 RoanKattouw: Running rebuildLocalisationCache.php in an attempt to shut up exceptions thrown by srv218
21:29 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'bug 20312: Re-enable GIF scaling on all wikis'
21:09 RoanKattouw: srv205 is throwing fatal errors because the geoip PHP module is not installed
20:51 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'fix typo'
20:45 RoanKattouw: srv218 has a full disk
20:45 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'plus proxy setting'
20:44 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20512: Enable $wgCopyUploads on test and commons'
20:44 mark: OS-installed Karmic on search13, search14, search16, search17, search20
20:40 RoanKattouw: Restarting refreshGlobalimageusage.php on Commons after it mysteriously disappeared
20:00 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Enabling GIF scaling on testwiki'
19:39 RoanKattouw: Running refreshGlobalimagelinks.php on Commons again
19:38 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Enabling GlobalUsage on commons again'
19:35 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Turning on GlobalUsage on testwiki'
19:33 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/GlobalUsageQuery.php 'Seems I forgot this file'
19:20 RoanKattouw: Running rebuildLocalisationCache.php in an attempt to kill "MagicWordArray::parseMatch: parameter not found" error
19:08 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Disable GlobalUsage again, blank pages on commons'
19:06 RoanKattouw: Running rebuildGlobalimagelinks.php on Commons
19:03 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Re-enabling GlobalUsage'
18:57 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/GlobalUsage.i18n.php 'r60611'
18:56 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/SpecialGlobalUsage.php 'r60611'
18:56 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/GlobalUsage.php 'r60611'
18:56 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/GlobalUsageHooks.php 'r60611'
18:56 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/GlobalUsage_body.php 'r60611'
18:56 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/ApiQueryGlobalUsage.php 'r60611'
18:55 RoanKattouw: Deploying r60611 (GlobalUsage update) with individual sync-file's
18:49 RoanKattouw: Truncating globalimagelinks table on commonswiki
18:48 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Temp disable GlobalUsage'
18:46 RoanKattouw: Disabling GlobalUsage temporarily for software update
18:41 logmsgbot: catrope synchronized php-1.5/wmf-config/ExtensionMessages.php 'Enabling WikimediaLicenseTexts on Commons'
18:40 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enabling WikimediaLicenseTexts on Commons'
18:31 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enabling WikimediaLicenseTexts on testwiki'
18:31 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Enabling WikimediaLicenseTexts on testwiki'
18:24 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaLicenseTexts.i18n.php 'r60608'
18:24 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaMessages.i18n.php 'r60608'
18:24 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaMessages.php 'r60608'
18:24 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaLicenseTexts.php 'r60608'
18:24 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaGrammarForms.php 'r60608'
18:23 RoanKattouw: Deploying r60608 (WikimediaMessages update) using sync-file
17:47 Rob: search13-search20 racked and ready for install
17:17 Rob: updating dns for new search servers
14:57 logmsgbot: mark synchronized php-1.5/wmf-config/mc.php 'Replace downed memcached box srv84'
14:45 Rob: moved power for asw-b3-pmtpa, srv86 and srv87 may flap.
14:43 logmsgbot: mark synchronized php-1.5/wmf-config/mc.php 'Replace downed memcached boxes srv120, srv121'
14:31 Rob: decommissioned srv120, srv121 to make room for new search servers.
14:29 Rob: pulling srv34 from rack to decommission, need the space for new search servers.
14:09 logmsgbot: mark synchronized php-1.5/wmf-config/mc.php 'Replace memcached on just decommissioned host srv82'
13:57 Rob: srv82, srv83, srv84 decommissioned to make room for new search servers in rack
12:56 Tim: started squid-frontend on knsq8, knsq10, knsq11, knsq13, knsq14, knsq15, all crashed at roughly the same time as knsq9
12:52 Tim: started squid-frontend on knsq9, died at ~17:30 on the 3rd. Syslog shows many crashes, followed by "out of socket memory" a couple of hundred times, then silence

January 3

20:17 logmsgbot: catrope synchronized php-1.5/includes/api/ApiQueryAllUsers.php 'Deploying r60588, r60590'
17:29 logmsgbot: midom synchronized php-1.5/includes/api/ApiQueryAllUsers.php 'added die(); somewher in there'
14:38 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
14:30 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'ixia is being put on a bench'

January 2

19:13 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21954: Enable import from Commons on brwikimedia'

December 31

19:45 logmsgbot: tfinc synchronized php-1.5/wmf-config/reporting-setup.php 'Moving tracking back to db9 as it requires insert'
18:38 logmsgbot: tfinc synchronized php-1.5/wmf-config/reporting-setup.php 'Switching to query the slave db10 instead of the master db9'

December 30

23:37 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/FundraiserStatistics_body.php
23:35 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/ContributionReporting.i18n.php
12:51 mark: Set /proc/sys/vm/min_free_kbytes to 65535 on brewster and streber, to see if it helps with the swapper page allocation failure bug
07:57 Tim: fixed private+secure file access, URLs were broken for months at least
07:50 logmsgbot: tstarling synchronized php-1.5/wmf-config/secure.php
07:48 logmsgbot: tstarling synchronized php-1.5/wmf-config/secure.php 'possible resolution for boardwiki problem'
07:45 logmsgbot: tstarling synchronized php-1.5/wmf-config/secure.php 'debugging hack'
07:34 Tim: created an account for myself on boardwiki to debug file upload issue

December 29

23:42 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'added tracking for 2009_notice51.'
18:27 tomaszf: dropping retention on storage2 to 4 xml snapshots
13:04 logmsgbot: catrope synchronized php-1.5/includes/api/ApiQueryAllUsers.php 'Deploying r60468'
12:50 RoanKattouw: Started logmsgbot (running as catrope instead of nobody)
12:32 RoanKattouw: logmsgbot down since fenari reboot, needs to be restarted by a root

December 28

21:02 RoanKattouw: Restarting refreshLinks for enwikibooks on hume
20:49 mark: Created 10G of swap space on fenari
20:07 domas: powercycled fenari, console not responsive
15:00 mark: Installed Karmic on LVS1, setting up a test network for LVS performance testing
14:55 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 17338: Disable $wgRestrictDisplayTitle and enable subpages in the main namespace on rmwiki'
13:54 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21566: Enable Collection on viwiki'
13:42 RoanKattouw: Running refreshLinks on enwikinews per bug 19404
13:41 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 19404: Set $wgCategoryPrefixedDefaultSortkey=false on enwikinews'
13:18 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21958: Create Book namespace on enwiki'

December 27

19:11 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21954: Import sources, subpages for brwikimedia'
11:26 domas: db20 offlined disk array, after reboot booted into netinstall (but saw the array), after reset /SYS, and some operator's no-operation in BIOS and RAID controller setup screens, it booted up properly

December 26

19:59 DaBPunkt: (non-dev-entry) Many apaches in the US died, some db-server reported overusing. Problem fixed itself after some minutes.

December 25

20:24 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php
20:09 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'fixed frwiktionary config (sysops should be able to remove patroller)'

December 24

22:52 RoanKattouw: Running updateArticleCount.php on arwiki and arwiktionary
20:21 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'wrong var AGAIN'
20:19 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'wrong var'
20:18 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'bug 21859: Enable Doublewiki on frwiktionary'
19:03 logmsgbot: catrope ran sync-common-all
18:56 RoanKattouw: Running sync-docroot, sync-apache, sync-common-all, apache-graceful-all
18:45 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Explicitly add langcodes for recently created wikis'
18:27 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 19534: Add Page and Index namespaces on sourceswiki'
17:02 logmsgbot: catrope ran sync-common-all
16:57 RoanKattouw: Syncing updates for wiki creations
16:57 RoanKattouw: Not creating fiwikimedia, already created
16:54 RoanKattouw: Adding fiwikimedia (sans DNS entry) per bug 20502
16:41 RoanKattouw: Adding arbcom_fiwiki (sans DNS entry) per bug 21375
16:32 RoanKattouw: Adding dkwikimedia (sans DNS entry) per bug 21009
16:23 RoanKattouw: Adding brwikimedia (sans DNS entry) per bug 21149
16:14 RoanKattouw: Adding pcdwiki (sans DNS entry) per bug 21634
16:06 RoanKattouw: Made raw, non-highlighted versions of .php files visible through noc.wikimedia.org/conf
15:55 RoanKattouw: Made abusefilter.php visible through noc.wikimedia.org/conf
15:34 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20515, bug 20551, bug 20923, bug 21214: Enable AbuseFilter on barwiki, eswikibooks, ltwiktionary, ptwikibooks'
15:34 logmsgbot: catrope synchronized php-1.5/wmf-config/abusefilter.php 'bug 20515, bug 20551, bug 20923, bug 21214: Enable AbuseFilter on barwiki, eswikibooks, ltwiktionary, ptwikibooks'
15:15 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Fix name of oldwikisource -> sourceswiki'
15:11 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable ProofreadPage for oldwikisource, which strangely is not part of the wikisource group'
15:06 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20402: Enable Collection extension on mswiki'
15:03 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21389: Set $wgDisabledVariants on Chinese wikis'
14:55 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'bug 21527: Enable Special:Cite on itwiktionary'
14:28 RoanKattouw: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21100: Add Tac_gia namespace to $wgNamespacesToBeSearchedDefault on viwikisource'
14:19 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21011: Restrict uploads, change $wgUploadNavigationUrl on enwikibooks'
14:10 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21236: Set $wgEnableNewpagesUserFilter=true; on plwiki'
14:06 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21370: Create custom namespaces on etwikisource, fix namespace alias on arwikisource'
13:51 RoanKattouw: Running updateArticleCount.php on zhwikisource for bug 20998
13:48 domas: s1 and s2 had rolling restarts to mysql#3193 build
13:48 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21109: Rename autopatroller group to autoreviewer on zhwiki'
13:44 domas: s2 master switched to db15-bin.000001:106
13:44 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
13:44 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
13:32 logmsgbot: catrope synchronized php-1.5/wmf-config/abusefilter.php 'misspelled right name'
13:29 logmsgbot: catrope synchronized php-1.5/wmf-config/abusefilter.php 'bug 20721: Add abusefilter-viewprivate right for sysops on enwiki'
13:11 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20561: Change sitename on tkwiki'
12:54 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 15237: Localize sitename and Wikipedia namespace on nahwiki'

December 23

16:02 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
14:57 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
14:35 Rob: db27 back online with new fan controller board.
13:53 Rob: shutting down mysql on db27 for hardware replacement.
12:56 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21517: Actually enable patrolling on frwikibooks and frwiktionary'
08:22 Tim: on isidore, also disabled the CentralNotice job that was running every 20 minutes from /etc/crontab
05:16 Tim: disabled CentralNotice rebuilds for donate.dev.wikimedia.org, was overloading isidore (which is only a single-core pentium)

December 22

23:45 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'Added 2009_Notice49 to tracking.'

December 21

19:25 Rob: ms8 still borked.
19:24 Rob: restarted mysql on db27
19:12 Rob: removed the fan boards in db27 and cleaned connections (were filthy with toner or dust or something) and replaced them. system booting back online. (Will watch it for errors for the next week.)
18:36 Rob: shutting down db27 mysql manually for troubleshooting hardware on the system
18:04 Rob: db28 mainboard and fans replaced, booting back online.
17:35 Rob: restarted mysql on srv185, since it appears to be a ext. storage slave.
17:32 Rob: srv185 memory replaced, back online.
17:20 Rob: shutting down srv185 to swap bad dimm1
16:52 Rob: swapped out bad disk in db30, all leds are green now.
16:22 Rob: swapped out bad memory in sq32, booting it back up.
16:13 Rob: shutting down sq32 to swap out bad memory
07:29 logmsgbot: tstarling synchronized php-1.5/extensions/intersection/DynamicPageList.php 'deploying r60254'

December 20

21:29 RoanKattouw: Clean up entries with rev_user_text='conversion script' (capitalize C) on dewiki for bug 21910
21:24 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'forgot $wgRemoveGroups'
21:18 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21517: Add patroller group on frwikibooks and frwiktionary'
21:03 RoanKattouw: Running namespaceDupes on simplewiktionary for bug 21906
21:03 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21906: Add WT: -> Wiktionary: alias on simplewiktionary'
10:25 domas: um, no more stalls?

December 19

20:53 RoanKattouw: Running namespaceDupes on zhwiki
00:41 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'Adding JimmyAppeal[8-9] to tracking'

December 18

22:25 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'revert my last: logo too large'
22:22 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21213: Logo for acewiki'
21:57 logmsgbot: catrope ran sync-common-all
21:56 RoanKattouw: Closing nlwikinews per bug 20325
21:45 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Fix mistake (autopatrol vs. autopatroller)'
20:24 mark: Updated firewall for allowing ssh access from fenari on loudon
13:21 mark: Stopped oprofile on db22
11:22 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21109: Add autopatroller group, allow sysops to add/remove confirmed on zhwiki'
11:18 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20910: Set $wgUploadNavigationUrl on nlwiki'
11:18 RoanKattouw: srv205 has full disk
11:09 domas: s1 switched master to db16-bin.000001
11:08 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
11:08 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
11:07 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
11:07 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
11:00 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
00:14 aZa_: restarted srv141. Puppet went crazy cpu hungry.

December 17

23:54 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/ContributionTrackingStatistics_body.php
23:53 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/ContributionReporting.i18n.php
23:15 RoanKattouw: Strike that, frwikinews
23:07 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21436: Activate flood flag on frwikibooks'
20:32 Rob: sq20 reinstalled, squid package installed. Not back in service until I check with Mark about some puppet settings.
20:07 Rob: resynced nagios to remove sq27
20:03 Rob: sq20 disk died, replaced and reinstalling.
20:01 Rob: sq27 decomissioned (stole drive to fix sq20)
19:59 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21071: Allow sysops to add/remove patroller and autopatroller on hewiki'
18:54 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php
17:21 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21158: Enable $wgBlockAllowsUTEdit on enwikiquote'
16:54 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21121: Enable flood flag on simplewikibooks'
16:22 RoanKattouw: Running namespaceDupes on iawiktionary
16:21 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21241: Add Appendice namespace on iawiktionary'
16:17 Rob: pushing update to dns for search13 dns
16:09 RoanKattouw: Running namespaceDupes on iawiki for bug 21241
16:08 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21241: Add Appendice namespace on iawiki'
16:04 RoanKattouw: Running namespaceDupes on brwiki for bug 21417
16:03 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21417: Change name of project namespace on brwiki, add old name as alias'
15:52 mark: Killed memcached on browne
15:51 mark: Started ircd on browne
15:51 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21041: Allow sysops to grant and bcrats to remove transwiki on nowiki'
15:46 Rob: browne back online after move.
15:46 Rob: nagios synced to node group files without will.
15:45 Rob: will decommissioned, pulled from rack
15:40 Rob: shutting down browne to move its rack location, will be back online shortly.
15:36 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21071: Add patroller and autopatroller groups on hewiki'
15:26 mark: Moved udpmcast from browne to dobson
13:11 mark: Fixed Racktables rack thumb problem by installing php5-gd; it was just serving cached thumbs
11:06 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21306: Allow sysops to add/remove patroller and autopatrol on hrwiki'
11:00 RoanKattouw: Running namespaceDueps on mlwiki for bug 21277
10:56 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21277: Add Portal and Portal talk as aliases on mlwiki'
10:53 RoanKattouw: Running namespaceDupes.php on cawikibooks for bug 20980
10:53 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20980: Add Viquiprojecte namespace on cawikibooks'
10:47 RoanKattouw: Resyncing srv194 and rebuilding caches
10:46 domas: srv194 had 5G-sized oprofile error log, hehe hehehe
10:39 RoanKattouw: srv194 disk still full
10:39 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21134: Allow sysops to add/remove rollbacker on gawiki'

December 16

20:41 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'fix mistake'
20:32 RoanKattouw: srv194 disk full
20:32 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21134: Add rollbacker group on gawiki'
19:56 logmsgbot: andrew synchronized php-1.5/extensions/UsabilityInitiative/js/js2/jquery-ui-1.7.2.js 'Deploying r60130'
19:55 logmsgbot: andrew synchronized php-1.5/extensions/UsabilityInitiative/js/js2.combined.min.js 'Deploying r60130'
19:55 logmsgbot: andrew synchronized php-1.5/extensions/UsabilityInitiative/js/js2.combined.js 'Deploying r60130'
19:53 logmsgbot: fvassard synchronized php-1.5/wmf-config/InitialiseSettings.php 'changed groupOvrrides for fiwiki for arbcom.'
17:44 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'FR labs config - remove excess level'
16:36 hcatlin: new and improved mobile with compressed memcached + new homepags + better utf-8 handling
16:24 hcatlin: taking down mobile1 for a large software update
15:59 mark: Disabled NIS on zwinger
15:54 mark: Disabled NIS on ms1
15:53 mark: Disabled NIS on ms4
15:47 mark: Installed puppet on ms1 and ms7
15:27 Andrew: srv129, srv123, srv120, srv95 seem to be in swapdeath
15:25 logmsgbot: andrew ran sync-common-all
15:24 Andrew: sync-common-all caused memory spike on apaches again, site seems to still be up though
15:17 Andrew: Updating LiquidThreads to trunk state, using sync-common-all
12:58 RoanKattouw: Tim ran rebuildTemplates.php on hume for all languages
12:52 Andrew: Fixed morebots, was choking because the Server Admin Log was archived.
12:50 Andrew: Moved Morebots init script into a line in rc.local. Morebots seems to have been down due to Freenode's DDoS problem, maybe it isn't exiting properly when disconnected from the server
12:46 Tim: ran rebuildTemplates.php on hume for all languages

2000s

Archive 1: 2004 Jun - 2004 Sep
Archive 2: 2004 Oct - 2004 Nov
Archive 3: 2004 Dec - 2005 Mar
Archive 4: 2005 Apr - 2005 Jul
Archive 5: 2005 Aug - 2005 Oct, with revision history 2004-06-23 to 2005-11-25
Archive 6: 2005 Nov - 2006 Feb
Archive 7: 2006 Mar - 2006 Jun
Archive 8: 2006 Jul - 2006 Sep
Archive 9: 2006 Oct - 2007 Jan, with revision history 2005-11-25 to 2007-02-21
Archive 10: 2007 Feb - 2007 Jun
Archive 11: 2007 Jul - 2007 Dec
Archive 12: 2008 Jan - 2008 Jul
Archive 12a: 2008 Aug
Archive 12b: 2008 Sept
Archive 13: 2008 Oct - 2009 Jun
Archive 14: 2009 Jun - 2009 Dec

2010s

2020s