Server Admin Log/Archive 22: Difference between revisions

Browse history interactively

← Older edit Newer edit →

Content deleted Content added

VisualWikitext

Inline

Revision as of 13:14, 11 April 2013

April 11

13:14 logmsgbot: reedy synchronized wmf-config/ 'Disable ClickTracking :D'
13:08 mark: Inserted python-extras packages into the precise-wikimedia APT repository (backport by hashar)
10:10 hashar: Jenkins is back up
09:23 hashar: restarting Jenkins to get the git verbose mode :(
08:44 hashar: jenkins: stop git cloning mw/core over and over by reusing existing clone 58502. That should help with I/O starvation on gallium.
08:40 Nikerabbit: translation memory database rebuild finished
08:17 Nikerabbit: emptying Translate solr instance and starting rebuilds
08:15 logmsgbot: nikerabbit synchronized php-1.22wmf1/extensions/Translate/ 'to master'
07:11 logmsgbot: hashar synchronized docroot/noc/conf/highlight.php 'Revert highlight.php recent modifications bug 47112 58664'
07:02 hashar: Put a live hack on fenari for bug 47112
02:17 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Thu Apr 11 02:16:35 UTC 2013
01:10 Ryan_Lane: disabling puppet on labstore3/4 - making sure gluster won't die without them and we're going to rebuild them

April 10

23:31 LeslieCarr: deactivated XO ipv6 transit
22:25 Ryan_Lane: added live hack to OpenStackManager on wikitech, ensuring that DynamicSidebar support is only enabled for logged-in users
22:24 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php 'moved refreshLinks to redis.'
22:20 ^demon|sick: restarting gerrit, was hung
22:15 Ryan_Lane: enable DynamicSidebar support in OpenStackManager
22:14 Ryan_Lane: temporarily disabling DynamicSidebar support in OpenStackManager
21:59 notpeter: gracefulling apache on stafford
21:36 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php 'Moved htmlCacheUpdate jobs to redis'
21:19 ottomata: added python-statsd to apt repo for hashar
21:18 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php
21:07 ottomata: added python-voluptuous to apt repo (for hashar)
20:46 ottomata: added python-jsonschema to wikimedia apt repo: ori-l :)
20:00 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php 'Moved async upload jobs to redis'
19:13 notpeter: running puppetd --enable on all nodes via salt
19:09 mwalker: eqiad frack is now using clustered memcache and has site local configuration for DB, ActiveMQ, MemCacheD shtuff
18:24 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php 'Enabled use of redis for null test jobs'
18:06 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Switched all remaning wikis to 1.22wmf1
17:45 RobH: authdns-update to rename db1012 to stat1002
17:29 logmsgbot: aaron synchronized php-1.22wmf1/maintenance/copyJobQueue.php 'deployed 54f74111901ddebdc69f26847e3904140b3723d5'
17:24 LeslieCarr: restart unresponsive cp1037
17:14 RobH: authdns-update
15:48 mark: Ran dist-upgrade (for varnish upgrade) on cp1021-1036
15:40 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Nostalgia ext'
15:40 logmsgbot: demon synchronized wmf-config/extension-list 'Nostalgia ext'
15:38 logmsgbot: demon synchronized php-1.22wmf1/extensions/Nostalgia
14:33 logmsgbot: maxsem synchronized wmf-config/mobile.php
14:19 logmsgbot: maxsem synchronized wmf-config 'Enable $wgMFVaryResources on test'
14:12 mark: Inserted varnish 3.0.3plus-rc1-wm10 packages into the APT repository
12:23 apergos: removed outright puppet.log on neon, was > 7gb and no room left on device
07:53 logmsgbot: krinkle synchronized php-1.22wmf1/resources/startup.js 'Ia54dd738b3ce0995fa'
05:57 Tim: on stafford: swapdeath repeat narrowly averted via killall ruby && /etc/init.d/apache2 stop
05:34 Tim: on brewster: proxy down due to full root partition, again. Will fix it properly this time.
05:28 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Add 'Zero' & 'Zero talk' namespaces to metawiki (If6b3ce5e4)'
04:53 Tim: stafford went into swapdeath. I killed all ruby processes a couple of times, and then eventually stopped apache2 for a while, while it recovered
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Wed Apr 10 02:28:06 UTC 2013
02:17 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Wed Apr 10 02:17:19 UTC 2013
00:21 mutante: removing xenon and db1012 from icinga configs, running puppetstoredconfigclean.rb on them, restarting icinga
00:07 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend/includes/MobileFrontend.hooks.php 'https://gerrit.wikimedia.org/r/#/c/58436/'
00:06 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend/includes/MobileFrontend.hooks.php 'https://gerrit.wikimedia.org/r/#/c/58436/'

April 9

23:29 logmsgbot: tstarling synchronized php-1.22wmf1/extensions/Nostalgia
23:21 notpeter: running upgrades on db56 and rebooting
23:19 notpeter: running upgrades on db64 and rebooting
22:55 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend/
22:53 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend/
22:36 logmsgbot: reedy synchronized wmf-config/
22:34 LeslieCarr: clearing a few varnish caches - fyi if db load is seen to increase
22:17 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
21:45 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
21:45 hashar: Zuul is slowly dequeuing a large number of jobs. There is currently rougly 30 minutes lag between change submission and actually processing of the job. No idea why but investigating.
19:56 Ryan_Lane: installed Nuke on wikitech
18:11 K4-713: updated payments anti-fraud rules
13:07 hashar: jenkins plugin upgrade is tracked in bug 47040. Luckily git plugin has been fixed just in time ;-D Just need to migrate the config history.
12:37 hashar: jenkins plugins seems to have been upgraded overnight :(
10:26 mark: Deployed experimental varnish build (3.0.3plus~rc1-wm10) on dysprosium and cp1022
08:54 hashar: Jenkins tests for mediawiki/core are broken (bug 47031) due to a faulty change that has been merged in master branch. Root cause is the gating job not testing the proper change bug 46723 :/
08:43 Nikerabbit: Got rid of local changes and undeployed commit related to noc
08:41 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Narayam and WebFonts'
05:42 Tim: on mw1149: enabled apache access log
02:31 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Tue Apr 9 02:30:59 UTC 2013
02:19 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Tue Apr 9 02:18:53 UTC 2013
00:26 ^demon: jenkins is back. java gc took awhile.
00:19 ^demon: jenkins is hung, looking into it.
00:16 awight: payments cluster updated from 22943dc28 to d3954768e4dd

April 8

23:16 logmsgbot: catrope synchronized php-1.22wmf1/extensions/WikiEditor 'Deploying 17a90bc74e87341199a50ebe5348576e05be028a'
23:15 logmsgbot: catrope synchronized php-1.21wmf12/extensions/WikiEditor 'Deploying 17a90bc74e87341199a50ebe5348576e05be028a'
22:55 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Changing from emery to vanadium'
22:13 logmsgbot: demon synchronized php-1.21wmf12/includes/AutoLoader.php 'Backported Poolcounter convenience class'
22:13 logmsgbot: demon synchronized php-1.21wmf12/includes/PoolCounter.php 'Backported Poolcounter convenience class'
21:59 logmsgbot: demon synchronized wmf-config/PoolCounterSettings-pmtpa.php 'Enable poolcounter for search'
21:59 logmsgbot: demon synchronized wmf-config/PoolCounterSettings-eqiad.php 'Enable poolcounter for search'
21:42 logmsgbot: demon synchronized php-1.21wmf12/extensions/MWSearch/MWSearch_body.php 'Deploying Poolcounter updates'
21:17 notpeter: temp stopping slave on db55
20:53 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Bumped file journal ttl to a year'
20:18 AaronSchulz: Moved ceph sync scripts to terbium and started a second pass
20:15 logmsgbot: aaron synchronized php-1.22wmf1/extensions/FlaggedRevs 'deployed c61688baef682b7d97068fefdcc399256786a387'
19:08 ^demon: gerrit back
19:07 ^demon: gerrit restarting, config change to fix idle timeouts.
19:00 ottomata: removing some unused filters from emery's udp2log instance
18:34 Jeff_Green: deployed 1.22 on payments
18:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.22wmf1
17:09 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'adding db1058 to s1 at full weight'
17:05 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'adding db1058 to s1 at a low warmup rate'
11:57 ^demon: gerrit back, deployed 2.6-rc0-154-gfcdb34b which contains a temporary fix for the stream-events timeout issues. See bug 46917 for info.
11:53 ^demon: bringing gerrit down for urgent update
10:31 hashar: gerrit: created operations/debs/libvpx for j^ . Initialized pristine-tar and upstream branches using empty commits and force push.
07:48 hashar: restarting Zuul for demo purposes :-)
04:00 Tim: on brewster: root partition was full, removed squid access.log and store.log and started squid
03:55 Tim: on all apaches: upgrading libpoppler
02:25 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Mon Apr 8 02:25:07 UTC 2013
02:18 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Mon Apr 8 02:18:18 UTC 2013

April 7

22:46 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
22:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:31 logmsgbot: reedy synchronized wmf-config/ 'Kill readerfeedback config, it aint coming back'
22:26 logmsgbot: reedy synchronized wmf-config/
22:20 logmsgbot: reedy synchronized php-1.22wmf1/extensions/CentralAuth
11:35 ^demon: restarting gerrit, stream-events seized up again.
07:26 hashar: gallium: restarted Zuul just to be sure. Gerrit on manganese is stuck again and need to be restarted bug 46917
02:17 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Sun Apr 7 02:17:09 UTC 2013
02:11 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Sun Apr 7 02:11:21 UTC 2013

April 6

12:27 ^demon: also restarted zuul
12:19 ^demon: restarting gerrit again, stream-events halted like friday.
02:23 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Sat Apr 6 02:22:48 UTC 2013
02:12 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Sat Apr 6 02:11:56 UTC 2013
01:55 binasher: iwlinks iwl_prefix_from_title migration completed for all projects
00:52 K4-713: adjusted anti-fraud rules on payments cluster

April 5

23:11 binasher: running iwlinks index migration on s4 (commons)
21:27 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1021 to full weight, pulling db1023'
21:13 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1021 at a low weight'
20:05 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1021 for upgrade'
19:53 binasher: streaming backup of db1056 to db1050 (converting s1 snapshot slave to file_per_table)
18:58 logmsgbot: asher synchronized wmf-config/db-eqiad.php 's1: moving watchlist to db1049, pulling db1043, adding db1056'
18:50 Jeff_Green: increased jvm heap size on silicon (payments activemq)
18:42 notpeter: restarting gerrit
18:22 logmsgbot: reedy synchronized php-1.21wmf12 'Killing active users in 1.21wmf12'
09:39 hashar: Ariel kindly restarted gerrit on manganese which unbroken the stream-events process in Zuul. It again receives events.
09:33 hashar: restarted Zuul
09:23 Nemo_bis: varnish esams purging still broken, e.g. https://upload.wikimedia.org/wikipedia/commons/f/fe/Wikipedia-logo-v2-km.png
08:58 hashar: for Jenkins / Gerrit issue see bug 46917
08:48 hashar: Jenkins no more receiving ANY events from Gerrit. Effect: no jobs are triggered.
02:23 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Fri Apr 5 02:23:11 UTC 2013
02:13 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Fri Apr 5 02:12:34 UTC 2013
00:49 logmsgbot: csteipp synchronized php-1.21wmf12/includes/
00:46 logmsgbot: csteipp synchronized php-1.22wmf1/includes/
00:30 Tim: on fenari: testing a new version of mw-update-l10n which doesn't use stdout of mergeMessagesFileList.php
00:20 logmsgbot: tstarling synchronized wmf-config/ExtensionMessages-1.22wmf1.php
00:19 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo to master
00:07 Tim: all 1.22wmf1 wikis briefly showed error messages due to display_errors being enabled when mergeMessageFileList.php generated ExtensionMessages-1.22wmf1.php. Fixed temporarily.
00:04 logmsgbot: tstarling synchronized wmf-config/ExtensionMessages-1.22wmf1.php
00:00 logmsgbot: mflaschen synchronized wmf-config/extension-list 'Remove E3Experiments and LastModified'

April 4

23:51 logmsgbot: demon synchronized wmf-config/extension-list 'Remove e3 since it was deleted'
23:49 logmsgbot: mflaschen synchronized wmf-config/CommonSettings.php
23:46 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1043'
23:46 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo to master
23:45 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1043'
23:44 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1043'
23:25 binasher: shutting down mysql on db1043, preparing to reboot
23:24 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1043'
23:17 mutante: wikinews.com now redirects to wikinews.org
23:11 mutante: DNS update - adding wikinews.com
23:08 logmsgbot: dzahn gracefulled all apaches
22:52 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Re-enable Echo on testwiki'
22:49 mutante: running updateSpecialPages on cswikinews per request
22:14 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Disable Echo on testwiki'
22:10 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Bump Echo EventLogging schema for new notification types.'
22:02 logmsgbot: mflaschen synchronized php-1.22wmf1/extensions/GuidedTour/ 'GuidedTour 1.22wmf1 deployment for E3'
22:02 logmsgbot: mflaschen synchronized php-1.22wmf1/extensions/GettingStarted/ 'GettingStarted 1.22wmf1 deployment for E3'
22:01 logmsgbot: mflaschen synchronized php-1.21wmf12/extensions/GuidedTour/ 'GuidedTour 1.21wmf12 deployment for E3'
22:00 logmsgbot: mflaschen synchronized php-1.21wmf12/extensions/GettingStarted/ 'GettingStarted deployment for E3'
20:21 LeslieCarr: stopped cp3001 varnish
20:18 LeslieCarr: restarting varnish on cp3001
20:15 LeslieCarr: restarted udpmcast on hooft
20:13 mutante: puppetstoredconfigclean.rb xenon.eqiad.wmnet - Killing xenon ...
19:34 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/57525/'
19:29 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/57525/'
19:20 binasher: enabled 1:100 full query logging on db1042/1043, disabled puppet [planning to revert in 4-8 hours]
19:09 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/57528'
19:01 binasher: resuming iwlinks migrations on s3
17:31 logmsgbot: mlitn synchronized php-1.22wmf1/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
17:31 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
17:27 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php 'Updating ArticleFeedbackv5 frwiki config'
16:16 paravoid: upgrading image & video scalers for libav
15:55 ottomata: now generating webstatscollector dumps on gadolinium. snapshot1 syncs dumps from there over to dataset2 for dumps.wikimedia.org
15:38 K4-713: adjusted anti-fraud rules on the payments cluster
13:26 ^demon: running jgit gc on all repos
05:43 Ryan_Lane: clearing squid logs and restarting squid on brewster
04:16 Ryan_Lane: opendj upgrade on virt1000 complete
03:56 Ryan_Lane: upgrading opendj on virt1000
02:21 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Thu Apr 4 02:21:01 UTC 2013
02:13 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Thu Apr 4 02:13:31 UTC 2013

April 3

23:19 mwalker: Updating CentralNotice schema on testwiki & metawiki with patch-centralnotice-2_3.sql
22:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
21:45 binasher: started iwlinks index migrations on all wikis [gerrit 43389]
21:32 cmjohnson1: Running DNS update
20:34 ottomata: udpated webstatscollector package in apt repo to 0.1-3
18:54 mutante: DNS update - also adding softwarewikipedia.org
18:49 RoanKattouw: Removed caesium and xenon from /home/wikipedia/common/docroot/noc/pybal/eqiad/parsoid
18:43 mutante: restarting pdns on ns1
18:42 mutante: DNS update - adding softwarewikipedia.com for RT-4355
18:41 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote and wikibooks to 1.22wmf1
18:40 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary to 1.22wmf1
18:35 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews and wikisource to 1.22wmf1
18:31 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity and wikivoyage to 1.22wmf1
18:27 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special, wikimania and wikimedia wikis to 1.22wmf1
18:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: private, closed and fishbowl to 1.22wmf1
17:14 logmsgbot: aaron synchronized php-1.22wmf1/includes/objectcache/SqlBagOStuff.php 'deployed b61053ca4b554f6bd18fb6408967839cdb5ccde2'
17:12 logmsgbot: aaron synchronized php-1.21wmf12/includes/objectcache/SqlBagOStuff.php 'deployed 61587acc64cb62400ff7978271c54e8bd8b1f02d'
16:54 logmsgbot: reedy synchronized php-1.21wmf12/includes/Collation.php
16:34 cmjohnson1: authdns update
14:22 logmsgbot: cmjohnson synchronized wmf-config/db-eqiad.php 'setting weight on db1028 to 400'
13:15 mark: Stopped and started MySQL on pc1003 after finding mysql deadlocked
13:10 logmsgbot: reedy synchronized wmf-config/
12:20 mark: Rebooting cp3003 for RAID reconfiguration
10:43 mark: Deactivated AS6908 peering on cr2-knams
06:09 ^demon: gerrit: finished doing jgit gc on mediawiki/core. Repo size 3G -> 323M. Fresh clone time to <5m to localhost, <1.5m to other hosts inside wmf. I rock. Backup's in /home/demon/core.git in case something goes wrong. Bed time.
03:05 LeslieCarr: the final upload varnish is on 2gig instead of 1gig ! win!
02:32 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Wed Apr 3 02:31:44 UTC 2013
02:19 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Wed Apr 3 02:19:30 UTC 2013
01:05 LeslieCarr: applying a big change to varnish interface groups - risk higher than normal
00:05 binasher: removed sda3/varnish.persist on cp1021, restarted varnish
00:00 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/PostEdit

April 2

23:46 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php '(Ia2665c4fe) Enable PostEdit on bn, br, ca, cs, et, ka and zh wikis'
23:46 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php '(Ibc2633f1c) : 10000 => 5000.'
23:25 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving all search traffic back to eqiad'
23:11 logmsgbot: anomie synchronized php-1.22wmf1/includes/api/ApiQueryRecentChanges.php 'Fix for API list=recentchanges rccontinue'
23:10 logmsgbot: anomie synchronized php-1.21wmf12/includes/api/ApiQueryRecentChanges.php 'Fix for API list=recentchanges rccontinue'
23:01 logmsgbot: maxsem synchronized php-1.21wmf12/resources/Resources.php 'https://gerrit.wikimedia.org/r/#/c/57212/'
22:49 logmsgbot: maxsem synchronized php-1.21wmf12/resources/Resources.php 'https://gerrit.wikimedia.org/r/57208'
22:39 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/56696/'
22:34 awight: payments cluster updated from 12bc36c238 to 0ab980b8b
22:24 logmsgbot: maxsem synchronized php-1.21wmf12/includes/resourceloader/ResourceLoaderStartUpModule.php
22:20 LeslieCarr: authdns-update for cp1039 and cp1040 move
22:14 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend 'touch'
22:11 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend
22:08 logmsgbot: maxsem Finished syncing Wikimedia installation... :
21:42 logmsgbot: maxsem Started syncing Wikimedia installation... :
21:38 logmsgbot: reedy synchronized wmf-config/
21:25 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
20:58 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1028'
20:49 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
20:41 logmsgbot: maxsem synchronized php-1.21wmf12/includes/OutputPage.php 'https://gerrit.wikimedia.org/r/#/c/49071/'
20:34 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c/57078/'
20:11 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'moving s1 watchlist to db1052, db1043 to full weight'
20:09 Ryan_Lane: added marc to the ops ldap group
20:01 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1001'
19:49 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
19:22 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:56 logmsgbot: cmjohnson synchronized wmf-config/db-eqiad.php 'adding db1028 back'
18:54 notpeter: actually starting innopack from db65 to db1054 for pre-labsdb db (sanitarium)
18:48 notpeter: starting innopack from db31 to db1054 for pre-labsdb db (sanitarium)
18:39 binasher: completed image img_media_mime migration on all projects
18:35 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
17:42 LeslieCarr: restarted varnishncsa-locke on cp1023
17:41 LeslieCarr: restarted varnishncsa-multicast_relay on cp1028
17:34 cmjohnson1: correction db1028
17:34 cmjohnson1: removing disk 0 from db1048 to replace
17:28 Jeff_Green: disable fundraising banner log rotation on locke
17:21 logmsgbot: cmjohnson synchronized wmf-config/db-eqiad.php 'adding db1001 back to production removing db1028'
17:04 cmjohnson1: setting weight to 100 on db1001
15:42 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Message for ancient skin users'
15:41 logmsgbot: anomie synchronized php-1.21wmf12/cache/l10n/l10n_cache-en.cdb
15:34 logmsgbot: anomie synchronized php-1.21wmf12/extensions/WikimediaMessages/WikimediaTemporaryMessages.i18n.php
15:13 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Tue Apr 2 15:12:45 UTC 2013
15:09 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Tue Apr 2 15:08:37 UTC 2013
14:54 logmsgbot: demon synchronized php-1.21wmf12/extensions/WikimediaMessages/WikimediaTemporaryMessages.i18n.php '8th time is the charm'
14:51 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'I hate l10nupdate'
14:50 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Notice for users of disabled skins'
14:24 paravoid: deploying squid config, diverting all of upload to swift
04:09 binasher: increased 'wikiadmin'@'10.64.%' max connections to 200 on s1
02:17 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Tue Apr 2 02:17:33 UTC 2013
02:10 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Tue Apr 2 02:10:24 UTC 2013
01:15 Ryan_Lane: removing labstore1 and labstore2 entries from projectstorage.wmnet rr dns entry in preparation for shrinking volumes
00:44 logmsgbot: reedy synchronized wmf-config/
00:43 binasher: now running the image img_media_mime migration on commons (the big one)
00:41 mutante: restarting lucene on all pool4 servers (one by one)
00:36 mutante: creating search index for transitionteamwiki
00:30 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
00:14 logmsgbot: reedy synchronized php-1.22wmf1/cache/interwiki.cdb 'Updating 1.22wmf1 interwiki cache'
00:10 logmsgbot: reedy synchronized php-1.21wmf12/cache/interwiki.cdb 'Updating 1.21wmf12 interwiki cache'
00:01 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:

April 1

23:55 LeslieCarr: search1024 is our guinea pig of using sfp-t's and is connected to asw2-a5-eqiad ge 0/0/18
23:48 logmsgbot: dzahn rebuilt wikiversions.cdb and synchronized wikiversions files:
23:40 mutante: creating swift containers for private wiki
23:39 logmsgbot: dzahn synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
23:37 mutante: dumpInterwiki.php to update interwiki cache
23:36 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving all search traffic to pmtpa temporarily'
23:33 mutante: adding transitionteamwiki to private.dblist
23:25 mutante: running addWiki.php for transitionteamwiki - Access denied for user ...
23:21 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
23:14 cmjohnson1: swapping disk1 (slot1) db1001
23:06 mutante: creating upload dir on upload7 for transitionteamwiki
22:57 binasher: beginning image.img_media_mime scehma migrations, starting with s1
22:52 logmsgbot: aaron synchronized php-1.22wmf1/includes/job 'deployed ec8f42daf19f386d9af2eece6dfc5f8fa6fc42f3'
22:26 logmsgbot: aaron synchronized php-1.22wmf1/maintenance 'deployed c7832c8956d79892be93452e5f4d6b52df1d3bf0'
22:26 logmsgbot: aaron synchronized php-1.22wmf1/includes/job 'deployed c7832c8956d79892be93452e5f4d6b52df1d3bf0'
22:26 mutante: recreating transitionteam wiki docroot from skel-1.5 and syncing docroot
22:26 logmsgbot: root synchronized docroot
21:09 LeslieCarr: rebooting lvs1001
21:04 logmsgbot: demon synchronized php-1.22wmf1/cache/l10n 'Manually syncing 1.22wmf1 l10ncache'
20:56 LeslieCarr: dist-upgrading lvs1001
20:48 LeslieCarr: failing over lvs from lvs1001 to lvs1004
20:44 LeslieCarr: CORRECTION restarting sshd on lvs1001
20:44 LeslieCarr: restarting lvs1001
20:21 logmsgbot: demon synchronized php-1.21wmf12/cache/l10n 'Manually syncing 1.21wmf12 l10ncache'
20:20 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Mon Apr 1 20:20:29 UTC 2013
20:00 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Mon Apr 1 19:59:44 UTC 2013
19:54 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Mon Apr 1 19:54:21 UTC 2013
19:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki
19:13 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki and testwiki to 1.22wmf1
19:11 logmsgbot: reedy synchronized php-1.22wmf1/extensions/ 'Wikidata extensions to same as 1.21wmf12'
19:04 logmsgbot: reedy synchronized php-1.22wmf1/includes/
18:56 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.22wmf1 and rebuild l10n cache
18:49 LeslieCarr: testing logbot
18:20 Reedy: Ran scap-recompile to build texvc
18:20 jeremyb_: re!logged from earlier today while morebots was gone (all times UTC)
18:20 jeremyb_: 17:56:29 <logmsgbot> !log reedy synchronized w
18:20 jeremyb_: 17:55:45 <logmsgbot> !log reedy synchronized docroot
18:20 jeremyb_: 17:55:04 <logmsgbot> !log reedy synchronized php-1.22wmf1 'Initial sync of php-1.22wmf1'
18:19 jeremyb_: 17:27:13 <logmsgbot> !log reedy synchronized wmf-config/ExtensionMessages-1.22wmf1.php
18:19 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.22wmf1 and rebuild l10n cache
18:19 jeremyb_: 17:14:31 <Jeff_Green> !log moved aside williams:/opt/otrs-home/.spamassassin/auto-whitelist and restarted spamd to test whether the bloated AWL is resulting in poor filtering
18:19 jeremyb_: 14:39:19 <andrewbogott> !log restarted glusterfs-server on labstore1 and labstore2, because of read errors in public/datasets/public in bots project
18:19 jeremyb_: 13:55:28 <ottomata> !log added webstatscollector-0.1-2 to apt repo
18:18 jeremyb_: 11:54:15 <paravoid> !log repooling ms-fe1; staggered depool/restart/pool of ms-fe2-4
18:17 Ryan_Lane: restarted morebots. wikitech-static had 100% / partition. Have also fixed that and its cause.
18:16 jeremyb_: 07:45:24 <paravoid> !log depooling ms-fe1 for staging
18:16 jeremyb_: 03:58:05 <logmsgbot> !log asher synchronized wmf-config/db-eqiad.php 'returning db1051'
18:16 jeremyb_: 03:44:43 <logmsgbot> !log asher synchronized wmf-config/db-eqiad.php 'pulling db1051'
18:16 jeremyb_: 03:22:54 <binasher> !log reinstated max_user_connections = 80 for wikiadmin on s1
18:15 cmjohnson1: restarted morebots
03:23 binasher: reinstated max_user_connections = 80 for wikiadmin on s1
02:12 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Mon Apr 1 02:12:24 UTC 2013

March 31

02:10 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Sun Mar 31 02:10:18 UTC 2013

March 30

02:14 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Sat Mar 30 02:13:49 UTC 2013
00:01 LeslieCarr: rebuilding mailman archives …. very slowly

March 29

23:51 LeslieCarr: stopping mailman to remove sensitive message
22:38 LeslieCarr: upgrading packages on aluminium
22:25 logmsgbot: reedy synchronized php-1.21wmf12/extensions/Wikibase
21:02 hashar: jenkins: raised the number of execution for deployment-bastion from 2 to 4. Will let us execute more jobs in parallel
20:14 mutante: scheduling a 2-year downtime for db11 :p
19:33 logmsgbot: dzahn gracefulled all apaches
19:30 logmsgbot: dzahn synchronized ./docroot/transitionteam
19:29 mutante: creating transitionteam docroot
19:25 logmsgbot: aaron synchronized php-1.21wmf12/includes 'deployed fa5c0a6a82c9a1cb63f3cedd87e6c4aba59c994d'
18:49 sbernardin: hot-swapping disk on db29
18:04 ^demon: gerrit back up, now running 2.6-rc0-76-g52fb5ae
18:00 mutante: DNS update - adding transitionteam wiki
17:47 sbernardin: taking down db29 to replace bad hw
17:45 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'Re-enabled async upload for commons'
17:23 ^demon: gerrit offline for an update, brb
09:25 logmsgbot: olivneh synchronized wmf-config/throttle.php 'Updating throttle rules for guwiki workshop (Bug 46686)'
04:16 logmsgbot: aaron synchronized php-1.21wmf12/maintenance/runJobs.php 'deployed 651884d4bdc76c29dd717dfdbbd698632223f3b5'
04:15 logmsgbot: aaron synchronized php-1.21wmf12/includes/job/JobQueueGroup.php 'deployed 651884d4bdc76c29dd717dfdbbd698632223f3b5 '
02:17 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Fri Mar 29 02:17:06 UTC 2013
01:24 ori-l: Stopping EventLogging daemons to allow Puppet to change 'eventlogging' user's home dir
00:11 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/MoodBar 'Updating MoodBar to remove ClickTracking integration'
00:03 logmsgbot: mflaschen synchronized php-1.21wmf12/extensions/GuidedTour/GuidedTour.php 'Small bug fix to GuidedTour; removing unneeded dependency. https://gerrit.wikimedia.org/r/#/c/56546/1/GuidedTour.php'
00:00 logmsgbot: mflaschen synchronized php-1.21wmf12/extensions/GuidedTour/GuidedTour.php 'Small bug fix to GuidedTour; removing unneeded dependency. https://gerrit.wikimedia.org/r/#/c/56546/1/GuidedTour.php'

March 28

23:24 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Remove E3Experiments and LastModified (2/2)'
23:24 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Remove E3Experiments and LastModified (1/2)'
22:35 logmsgbot: khorn synchronized php-1.21wmf12/includes/cache/MessageCache.php 'Reverting changes in language fallback behavior'
22:34 logmsgbot: khorn synchronized php-1.21wmf12/languages/Language.php 'Reverting changes in language fallback behavior'
22:20 logmsgbot: spage Finished syncing Wikimedia installation... : E3: Updated GettingStarted and GuidedTour extensions
22:12 LeslieCarr: removed some log files from ms1004 - it was full
22:10 RobH: bugzilla admin stuff: noc@w.o is now a disabled account cuz im tired of folks who arent on the alias attempting to sign up with it
22:02 logmsgbot: spage Started syncing Wikimedia installation... : E3: Updated GettingStarted and GuidedTour extensions
21:46 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/NavigationTiming/modules/ext.navigationTiming.js 'Fix mobileMode property name (was: 'mobile')'
21:36 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/NavigationTiming
20:11 Jeff_Green: add hostname samarium.wikimedia.org and pdns-update
19:51 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1051'
19:44 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1051'
19:44 LeslieCarr: killed gmetad on neon
19:36 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/ArticleFeedback/populateAFRevisions.php 'Fix fatal error triggered by populateAFRevisions.php cronjob when AFT is not loaded.'
19:33 mutante: starting gmetad on neon again
19:24 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/Echo 'syncing Echo on wmf12'
19:17 mutante: manually deleting db11 references from icinga configs and restarting it
19:12 mutante: pkill -u snmptt on neon, restarting snmptt deamon
19:10 binasher: replaced collector on professor with locally built version from I2bc117d5620b9e545a8e5a6cf3b654ee835b75d7 (testing fix for issues only under very high load)
19:10 mutante: stopping gmetad and ganglia-parser on neon
18:53 mutante: adding gjg (greg-g) to wmf ldap group
18:23 mutante: restarting lucene on pool4 search nodes to pickup new wikivoyage indices
18:12 ottomata: adding reportcard and analytics .wikimedia.org CNAMEs to stat1001
18:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
17:56 mutante: creating search indices for new wikivoyages
17:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
11:49 hashar: search19 and search20 issue with enwiki.prefix is in RT #4845
11:42 hashar: search19 and search20 have enwiki.prefix marked as FAILED. (see: curl --silent http://search19.pmtpa.wmnet:8123/status |grep FAILED and curl --silent http://search20.pmtpa.wmnet:8123/status |grep FAILED).
10:39 logmsgbot: nikerabbit synchronized php-1.21wmf12/extensions/Translate/ 'Translate to master'
04:19 paravoid: updating jsduck in apt and upgrading it on gallium
03:04 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
02:59 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
02:19 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Thu Mar 28 02:19:24 UTC 2013
00:47 K4-713: synchronized payments cluster to 12bc36c2385d8e56792c
00:27 mutante: once again disabling notifications for db11
00:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php

March 27

23:59 logmsgbot: reedy synchronized php-1.21wmf12/extensions/Wikibase
23:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:40 Ryan_Lane: finished updating wikitech-static
23:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:38 Ryan_Lane: updating wikitech-static to wmf/1.21wmf12
23:24 Ryan_Lane: finished upgrading wikitech
23:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:23 Ryan_Lane: upgrading wikitech to wmf/1.21wmf12
23:17 logmsgbot: reedy synchronized php-1.21wmf12/cache/interwiki.cdb 'Updating 1.21wmf12 interwiki cache'
23:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
23:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:50 logmsgbot: maxsem synchronized php-1.21wmf11/extensions/MobileFrontend 'touch'
22:47 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend 'touch'
22:24 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
22:22 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
22:05 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment, 2nd attempt
21:48 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment, 2nd attempt
21:12 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'changing s1 slave weights for testing'
20:10 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild localisation cache to fix he related ParserFunctions problems
19:56 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild localisation cache to fix he related ParserFunctions problems
19:43 logmsgbot: reedy synchronized php-1.21wmf12/extensions/ParserFunctions/
19:38 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild message cache for WikiData deployment
19:05 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild message cache for WikiData deployment
18:51 cmjohnson1: pushing mw1209-1220 into service
18:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:39 logmsgbot: reedy synchronized php-1.21wmf12/extensions/Wikibase
18:28 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.21wmf12
18:23 mutante: installing package upgrades on neon (Icinga), incl. ganglia-monitor,gmeta
18:09 mutante: bunch of mw* hosts: fix puppet runs
18:08 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/Echo 'syncing Echo for wmf12'
18:05 mutante: mw1130 - kill and fix puppet runs
17:41 Jeff_Green: authdns-update, pdns on ns1 croaked, restarted
15:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: be_x_oldwiki to 1.21wmf12
14:09 hashar: jenkins has been restarted by mistake :-( Job building will be unavailable for up to half an hour.
13:58 hashar: Jenkins: disabled the old Gerrit Trigger Plugin bug 46415
09:56 Nikerabbit: Testing Translate bug fixes on test.wikipedia.org
05:20 Tim: on fluorine: running mw-log-cleanup once to test Id2196e7b
05:02 Ryan_Lane: disabling Bayes-collector cron on sanger
04:55 Ryan_Lane: purging BayesLearning files older than 70 days on mchenry
02:45 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Wed Mar 27 02:44:56 UTC 2013
02:19 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Wed Mar 27 02:19:11 UTC 2013
00:55 LeslieCarr: ms-be3 was crashed with stack traces for swift-container
00:54 LeslieCarr: rebooting ms-be3
00:38 awight: payments cluster updated from 4862fb8eef to f295d851c
00:34 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'db1052 to full weight'
00:31 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1001, adding db1052 at a warmup weight'
00:11 awight: payments cluster updated from fa2e2 to a06a03f05 wheee!

March 26

23:54 awight: payments cluster rolled back from e758102 to fa2e240b6e9
23:53 awight: payments cluster updated from fa2e240b6e9 to e758102
23:53 Tim: on all apaches: replacing /usr/local/apache/common-local/live-1.5 with a symlink to w
23:44 logmsgbot: tstarling synchronized w
23:35 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/56019/'
23:33 awight: payment cluster rollback, from 4862fb8 to fa2e240b
23:31 mutante: restarting nrpe on professor
23:29 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/NavigationTiming/modules/ext.navigationTiming.js 'Workaround for buggy NavTiming implementation in IE9 (bug 46474)'
23:20 awight: payments cluster updated from fa2e240b to 4862fb8
23:10 mutante: db11 - disabling all notifications
23:07 mutante: mw13 - killing puppet and re-running
23:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:00 mutante: removing mysql packages from db1036 (broken dpkg, status "pi")
22:55 mutante: re-installing ganglia-monitor on mw1135, fix dpkg
22:54 logmsgbot: reedy synchronized docroot
22:52 mutante: starting udpprofile (carbon-cache.py) on professor
22:51 logmsgbot: reedy synchronized wmf-config/
22:49 mutante: search21 - remove sun-j2sdk , replaced by oracle-jdsdk
22:47 mutante: search21 - broken DPKG: iF sun-j2sdk1.6
22:45 mutante: mw2 - broken DPKG: pi librsvg2-2 , pi librsvg2-bin
22:39 LeslieCarr: did a service apache2 start on mw108*
22:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:24 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:10 notpeter: icinga mostly trustworthy again :)
22:08 andrewbogott: restarted nova-api on virt0 because it appeared to have crashed.
21:31 andrewbogott: restarted memcached on virt0 in a futile attempt to address https://bugzilla.wikimedia.org/show_bug.cgi?id=46583
21:30 mutante: killed stuck snmptt procs on neon to fix the others.. apparently :p
21:28 mutante: stopping gmetad on neon
20:52 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'syncing CommonSettings.php for Echo event tracking'
20:41 mutante: more degraded RAIDs: db1001->RT-4832, db1028->4834
20:35 mutante: broken disks/degraded RAID on db servers: db11->RT-4828, db29->RT-4829, db45->RT-4831
20:34 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/ContributionReporting 'syncing ContributionReporting extension'
20:07 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'raising db1051 to full weight'
19:49 logmsgbot: khorn synchronized php-1.21wmf12/extensions/CentralNotice 'Fixing CentralNotice bug in which it was possible to initialize multiple times'
19:46 logmsgbot: khorn synchronized php-1.21wmf11/extensions/CentralNotice 'Fixing CentralNotice bug in which it was possible to initialize multiple times'
19:40 logmsgbot: kaldari synchronized php-1.21wmf12/resources/startup.js 'syncing php-1.21wmf12/resources/startup.js'
19:33 logmsgbot: reedy synchronized php-1.21wmf12/extensions/Wikibase/client
19:27 cmjohnson1: mw1209 re-imaging
19:20 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/Echo/modules/icons 're-syncing Echo icons dir'
19:19 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php 'Include AFTv5 on frwiki'
19:12 logmsgbot: mlitn synchronized wmf-config 'Enable AFTv5 on frwiki'
19:05 logmsgbot: reedy synchronized php-1.21wmf12/extensions/Collection/
19:04 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
19:04 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
19:02 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/Echo/modules/icons 'syncing Echo icons dir'
18:58 mutante: starting gmetad on neon - we failed to resolve data source name ms-fe1001.eqiad.wmnet
18:56 mutante: restarting ircecho on fenari
18:48 mutante: stopping gmetad on neon
18:44 mutante: killing all snmp related processes on neon and restarting
18:43 logmsgbot: reedy synchronized php-1.21wmf12/extensions/MobileFrontend/includes/specials/SpecialMobileWatchlist.php
18:41 logmsgbot: kaldari synchronized php-1.21wmf11/extensions/Echo/modules/icons 'syncing Echo icons dir'
18:39 mutante: restarting snmtrapd on neon
18:25 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php 'Turning Thanks extension on on MediaWiki wiki'
18:13 logmsgbot: krinkle synchronized php-1.21wmf12/resources/mediawiki/mediawiki.js 'I80af730daa815 fixing bug 46575'
18:12 Krinkle: Change I34a488247 (mw.loader debug) has been reverted in wmf/1.21wmf12 and pulled on fenari
17:51 cmjohnson1: removing disk from slot 0 ms-be1012
17:14 Krinkle: Change I34a488247 is now merged in wmf/1.21wmf12 and pulled on fenari for test.wikipedia.org, do not sync.
17:04 ottomata: added (crappy, but better than nothing) .deb for webstatscollector to apt.wikimedia.org
15:24 cmjohnson1: swapping disk 6 ms-be1012
14:34 logmsgbot: hashar synchronized multiversion '47564 wikiversions.cdb now vary by realm'
14:10 cmjohnson1: running sync-common on mw1209
10:23 logmsgbot: nikerabbit synchronized php-1.21wmf12/extensions/UniversalLanguageSelector/UniversalLanguageSelector.php 'Testing fix for wikidata'
09:28 pp-pdf1: updated mwlib to 0.15.4
08:47 logmsgbot: nikerabbit synchronized php-1.21wmf12/extensions/Translate
08:46 logmsgbot: nikerabbit synchronized php-1.21wmf12/extensions/UniversalLanguageSelector
07:27 mark: Restarted swift-proxy on ms-fe*
04:16 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
02:46 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Tue Mar 26 02:45:52 UTC 2013
02:21 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Tue Mar 26 02:21:08 UTC 2013
01:41 mutante: analytics1026 - delete 16G udp2log.log to free disk
01:38 mutante: analytics1003 - gzipping jmxtrans logs to free some disk
01:36 mutante: analytics1003 - gzipping kafka.log to free some disk
00:39 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'adding db1051 at a low warmup weight'

March 25

23:36 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'done'
23:31 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'changing s1 weights, returning db1050'
23:27 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Quick profiling'
22:10 hashar: reloading Zuul
21:44 logmsgbot: khorn synchronized wmf-config/InitialiseSettings.php 'Re-enabling CentralNotice'
21:39 mutante: killing and restarting nagios-nrpe-server on '*'
21:32 logmsgbot: khorn synchronized php-1.21wmf12/extensions/CentralNotice/ 'CentralNotice jQuery fix'
21:28 logmsgbot: khorn synchronized php-1.21wmf11/extensions/CentralNotice/ 'CentralNotice jQuery fix'
21:12 mutante: killing and restarting nagios-nrpe-server on gadolinium
21:07 mutante: --last-lines 20 | RT ..would be nice sometimes:)
20:13 logmsgbot: aaron synchronized wmf-config/CommonSettings.php
20:07 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Use ffmpeg for ogg thumbs as is done for webm'
19:55 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild l10n cache for wikibase
19:33 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild l10n cache for wikibase
18:57 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
18:46 logmsgbot: reedy synchronized php-1.21wmf12/extensions/Wikibase
18:33 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf12
18:30 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable CN everywhere'
18:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: rb
18:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf12
17:25 logmsgbot: reedy synchronized wmf-config/

March 24

18:13 Ryan_Lane: restarting glusterd processes on labstore*
02:31 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Sun Mar 24 02:31:22 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Sun Mar 24 02:08:00 UTC 2013

March 23

02:37 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Sat Mar 23 02:36:53 UTC 2013
02:14 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Sat Mar 23 02:13:44 UTC 2013
00:17 RoanKattouw: Deployed master of Parsoid onto wtp1004; the code is broken but the box is depooled for Gabriel's benchmarking work
00:15 RoanKattouw: That's me messing with wtp1004

March 22

23:33 ottomata: restarting hadoop setting yarn.nodemanager.resource.memory-mb to 16G
23:06 logmsgbot: dzahn rebuilt wikiversions.cdb and synchronized wikiversions files:
23:04 logmsgbot: dzahn rebuilt wikiversions.cdb and synchronized wikiversions files:
23:04 mutante: sync-wikiversions
22:29 logmsgbot: aaron synchronized php-1.21wmf12/extensions/Translate 'deployed 0aab5984415a94bd214088767ff1de33ab57c4e0'
22:12 mutante: killall nrpe via salt, then restart nagios-nrpe-server , runs as wrong user
22:09 mutante: kill and restart nrpe on oxygen
22:07 mutante: installing package upgrades on oxygen
21:40 mutante: killing nagios-nrpe on gallium and testing if puppet restores it properly
21:03 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Set all non-Wikipedias back to 1.21wmf12 again.
20:51 logmsgbot: aaron synchronized php-1.21wmf12/includes/Title.php 'deployed 0f4914bb4ff2d13cb124cad94dc5ded663123a18'
20:46 logmsgbot: aaron synchronized php-1.21wmf12/includes/WikiPage.php 'deployed e372b635ae6d6a589f6d87f14fe4c9452d7a1b4d'
20:38 logmsgbot: aaron synchronized php-1.21wmf12/includes/Title.php 'Reverted follow-up fixes too'
19:32 ottomata: removed unicast udp2logging to gadolinium; gadolinium will consume from multicast stream
18:44 K4-713: synchronized payments cluster to fa2e240b6e92440b
18:28 K4-713: synchronized payments cluster to df3c8dbc937c971
17:33 logmsgbot: aaron synchronized php-1.21wmf12/includes/WikiPage.php 'deployed 1862a57aef08432e8b1194a54dc179649f5cae63'
09:08 MaxSem: Recreating Solr index
02:52 Tim: on searchidx1001: started incremental indexer, apparently it died on March 21 at 02:08 when it ran at the same time as a cron job import
02:50 binasher: power cycling db1050, unresponsive on serial console
02:49 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1050, crashed'
02:47 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1043'
02:42 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Fri Mar 22 02:42:32 UTC 2013
02:37 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1043, oops'
02:30 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Fri Mar 22 02:30:31 UTC 2013
01:28 ori-l: Finished correcting E3Experiments extension's entry point from 'Experiments.php' (which does not match the submodule name) to 'E3Experiments.php'. Updated extension-list, CommonSettings.php, and the extension itself in both wmf11 and 12.
01:24 logmsgbot: olivneh synchronized wmf-config/extension-list 'Remove temporary workaround for E3 entry-point migration (2/2)'
01:23 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Remove temporary workaround for E3 entry-point migration (1/2)'
01:18 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/E3Experiments 'Updating E3Experiment extensions's entry point'
01:18 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/E3Experiments 'Updating E3Experiment extensions's entry point'
01:11 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Temporarily accepting either E3Experiments.php or Experiments.php as extension entry point to facilitate clean-up'
00:55 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/NavigationTiming 'NavigationTiming: log mobile mode'
00:16 logmsgbot: catrope synchronized wmf-config/ExtensionMessages-1.21wmf12.php 'Remove PHP errors'
00:15 logmsgbot: aaron synchronized php-1.21wmf12/includes/Title.php 'deployed cef327e945e2593fe0291880a7d1976cc5a2f248'
00:09 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/Thanks 'syncing Thanks ext'

March 21

23:24 logmsgbot: robh Finished syncing Wikimedia installation... :
23:17 logmsgbot: robh Started syncing Wikimedia installation... :
23:04 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo, PageTriage to master, Add Thanks
22:44 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo, PageTriage to master, Add Thanks
21:50 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'Syncing change to CommonSettings for Thanks ext'
21:49 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'Syncing change to InitialiseSettings for Thanks ext'
21:46 mutante: running puppet and reloading zuul on gallium. enables hashar's new shiny https://integration.wikimedia.org/zuul/
21:38 RobH: authdns-update for terbium
21:05 RoanKattouw: Added fenari back to /etc/dsh/groups/mediawiki-installation per https://rt.wikimedia.org/Ticket/Display.html?id=4794
21:02 logmsgbot: cmjohnson Finished syncing Wikimedia installation... :
20:58 binasher: pulling mw1070, may have network issues
20:57 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/PageTriage/modules/ext.pageTriage.views.list 'syncing PageTriage js on wmf12'
20:57 hashar: fenari disappeared from dsh group mediawiki-installation :-(
20:52 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
20:51 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
20:48 awight: payments cluster updated from 18fd11a4 to df3c8dbc9
20:43 logmsgbot: cmjohnson Started syncing Wikimedia installation... :
20:37 LeslieCarr: restarting gmond on mw1153
20:19 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
20:19 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/NavigationTiming 'Enable NavigationTiming on mobile alpha and beta'
20:19 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
20:19 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/NavigationTiming 'Enable NavigationTiming on mobile alpha and beta'
19:53 logmsgbot: mflaschen synchronized php-1.21wmf11/extensions/GuidedTour/ 'E3 deploy'
19:52 logmsgbot: mflaschen synchronized php-1.21wmf11/extensions/GettingStarted/ 'E3 deploy'
19:52 logmsgbot: mflaschen synchronized php-1.21wmf11/extensions/E3Experiments/ 'E3 deploy'
19:49 logmsgbot: mflaschen synchronized wmf-config/CommonSettings.php 'E3 deployment'
17:39 logmsgbot: csteipp rebuilt wikiversions.cdb and synchronized wikiversions files: Update test2wiki to 1.21wmf12
17:32 logmsgbot: csteipp synchronized php-1.21wmf12/includes/Title.php 'update for loadPageData()'
17:24 notpeter: running puppet on all lucid nodes via salt
17:23 mutante-away: purging wikispecies.net from squid
17:08 notpeter: install timeout package on all lucid nodes to get puppet back on track
12:49 mark: Rebooting palladium with hyperthreading disabled
11:27 mark: Rebooting niobium with hyperthreading disabled
09:29 hashar: created git repo operations/debs/python-statsd
09:15 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Add 'wikidata.org' to $wgCrossSiteAJAXdomains to allow CORS'
09:14 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Add 'wikidata.org' to $wgCrossSiteAJAXdomains to allow CORS'
06:34 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files:
06:34 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files:
06:32 Tim: reverting all wikis to 1.21wmf11 due to move page being totally broken (bug 46397)
06:32 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files:
06:32 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files:
01:10 logmsgbot: catrope synchronized php-1.21wmf12/includes/db/LoadBalancer.php '319ace81250b97dff0e6ee6df4b1f570f468bf5c'
01:10 logmsgbot: catrope synchronized php-1.21wmf12/includes/db/LoadBalancer.php '319ace81250b97dff0e6ee6df4b1f570f468bf5c'
01:10 logmsgbot: catrope synchronized php-1.21wmf11/includes/db/LoadBalancer.php '319ace81250b97dff0e6ee6df4b1f570f468bf5c'
01:10 logmsgbot: catrope synchronized php-1.21wmf11/includes/db/LoadBalancer.php '319ace81250b97dff0e6ee6df4b1f570f468bf5c'

March 20

23:57 RobH: yay peter!
23:56 notpeter: freeing up roughly 500G of ram across our cluster by calling puppet via cron :)
23:52 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning es100[69]'
23:52 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning es100[69]'
23:51 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'returning s2 servers'
23:51 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'returning s2 servers'
23:36 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling es100[69]'
23:36 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling es100[69]'
23:36 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'returning s1 servers'
23:35 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'returning s1 servers'
23:31 RobH: authdns-update for lanthanum
22:45 RobH: ignore mw1209+ they arent deployed
22:04 LeslieCarr: restarting ircecho on fenari to ensure logmsgbot is on operations and tech irc channels
22:03 ottomata: stopping puppet on gadolinium to test for packet loss weirdness
22:01 pgehres: pgehres synchronized php-1.21wmf12/includes/GlobalFunctions.php 'Fix wfWaitForSlaves() so the parameter actually works correctly'
22:01 logmsgbot: pgehres synchronized php-1.21wmf11/includes/GlobalFunctions.php 'Fix wfWaitForSlaves() so the parameter actually works correctly'
21:35 LeslieCarr: restarted ircecho on fenari
21:01 Krinkle: asher synchronized wmf-config/db-eqiad.php 'pulling db1049 from s1'
20:53 pgehres: pgehres synchronized php-1.21wmf11/extensions/CentralAuth 'Updating CentralAuth to master'
20:53 pgehres: pgehres synchronized php-1.21wmf12/extensions/CentralAuth 'Updating CentralAuth to master'
20:42 Reedy: reedy synchronized wmf-config/InitialiseSettings.php
19:59 mutante: DNS update - remove zuul CNAME
19:53 Reedy: wikibooks to 1.21wmf12
19:51 Reedy: sync'd wmf-config/InitialiseSettings.php
19:50 Reedy: wikiversity and wikiquote to 1.21wmf12
19:49 Reedy: wikivoyage and wiktionary to 1.21wmf12
19:47 Reedy: wikinews and wikisource to 1.21wmf12
19:47 Reedy: Synchroni[sz]ed php-1.21wmf12/extensions for Wikidata deploy
19:46 Reedy: special and wikimedia dblists to 1.21wmf12
19:46 Reedy: wikimania, private and fishbowl dblists to 1.21wmf12
19:45 Reedy: closed.dblist to 1.21wmf12
19:38 RobH: sync-common on mw23
19:36 mutante: restarting hung apache on mw1202
19:28 LeslieCarr: sync-common on mw1085
19:24 mutante: .tar.gz'ing /var/mwdocs on formey (SVN) and then manually delete it
19:18 LeslieCarr: upgrading mw118* and then restarting apache2 on these boxes
19:11 LeslieCarr: installing package upgrades on mw1170-1179
19:10 mutante: installing package upgrades on formey
18:50 mutante: apache-graceful-all, remove boards.wm redirect
16:03 ottomata: added python-voluptuous package to apt repo
15:12 mark: Fixed purging for all sites but eqiad (which wasn't broken)
07:56 hashar: gallium: reloaded apache2 to make sure recent changes are applied.
02:44 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Wed Mar 20 02:44:03 UTC 2013
02:32 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Wed Mar 20 02:32:16 UTC 2013

March 19

23:52 K4-713: synchronized payments cluster to fe4fb96bea1378e
23:33 mutante: DNS update - add wikiartpedia.org
23:30 logmsgbot: dzahn gracefulled all apaches
23:08 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php 'Re-enable AFTv5 on dewiki/testwiki'
22:38 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/data/DataModel.php 'Update ArticleFeedbackv5; filter counts are incorrect'
22:36 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/data/DataModel.php 'Update ArticleFeedbackv5; filter counts are incorrect'
22:33 ottomata: adding udp-filter_0.3.22 for precise to apt repo (lucid version already was there)
22:17 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/data/DataModel.php 'Update ArticleFeedbackv5; filter counts are incorrect'
22:15 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/data/DataModel.php 'Update ArticleFeedbackv5; filter counts are incorrect'
21:52 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
21:52 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
21:37 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
21:25 mutante: installing package upgrades on kaulen
21:24 mutante: installing package upgrades on sockpuppet
21:13 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
20:36 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5/SpecialArticleFeedbackv5.php 'Touched AFT Special page file'
20:35 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/SpecialArticleFeedbackv5.php 'Touched AFT Special page file'
19:42 ottomata1: added gadolinium as a udp2log host on frontend caches
19:14 logmsgbot: mlitn synchronized wmf-config/CommonSettings.php 're-include AFTv5'
19:08 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
19:07 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
19:03 paravoid: adding ruby-multi-json (universe), ruby-parallel, ruby-dimensions, ruby-execjs, ruby-jsduck (main) to apt
18:46 mutante: dist-upgrade on gallium
18:03 logmsgbot: robh synchronized docroot
18:01 logmsgbot: robh synchronized docroot
17:21 mark: Ended bits.esams performance testing
16:29 mark: Starting load testing with esams bits servers
16:20 mark: Rebooted cp3019 with hyperthreading and DAPC disabled
16:16 logmsgbot: krinkle synchronized extract2.php
14:29 mark: Rebooting arsenic with hyperthreading disabled
12:54 Tim: on professor: restarted collector
08:46 hashar: gallium: killed stalled puppet agents. Locked on apt-get and update-java-alt
08:41 hashar: gallium: restarted puppet and zuul
08:38 hashar: gallium: stopped zuul and upgrading
08:37 hashar: gallium: stopped puppet to deploy Zuul whenever the queue is empty
08:36 hashar: zuul : giving results processing priority over enqueued jobs. Cherry picked 263fba9 from upstream. That should resolve Zuul being slow to report back to Gerrit bug 46176
03:52 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
03:50 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
03:12 Tim: started incremental updater on searchidx1001 and searchidx2, apparently has not been running since March 13
02:50 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Tue Mar 19 02:50:19 UTC 2013
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Tue Mar 19 02:29:02 UTC 2013
01:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
00:22 Ryan_Lane: adding dns entries for labstore systems in eqiad
00:08 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'returning servers'

March 18

23:34 binasher: i <3 running "dpkg -P mysql-server-5.1 mysqlfb-client-5.1 mysqlfb-common mysqlfb-server-5.1 mysqlfb-server-core-5.1 libmysqlfbclient16 libmysqlclient16"
23:11 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db10[10-11],[26-28] at low weights'
22:45 binasher: converted rogue s3 tables (moodbar_feedback, trackbacks, transcache) to innodb
22:28 ottomata: added python-flask-login-0.1.2-1 to apt
22:23 logmsgbot: robh gracefulled all apaches
22:20 LeslieCarr: intradatacenter link is flapping , switching links, this may cause some higher latency
22:14 logmsgbot: robh gracefulled all apaches
22:11 logmsgbot: robh gracefulled all apaches
21:58 mutante: purging wikimediafoundation.org from squid
21:57 logmsgbot: dzahn gracefulled all apaches
21:56 logmsgbot: robh gracefulled all apaches
21:36 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling a db from s3-7 for upgrade'
21:30 paravoid: authdns-update: removing *.ts.wikimedia.org records
21:29 logmsgbot: dzahn gracefulled all apaches
21:25 binasher: ran "ddsh -g apaches -cM '/etc/init.d/apache2 start'"
21:24 RobH: authdns-update rt4683
21:22 RobH: gracefuling all apaches resulted in rendering cluster overload, they didnt restart apache, odd
21:21 logmsgbot: robh synchronized docroot
21:18 LeslieCarr: service apache2 start on the mw-eqiad group via dsh
21:14 binasher: started apache on all imagescalers
21:00 logmsgbot: robh gracefulled all apaches
21:00 RobH: troubleshooting erros in apache restart script
20:59 logmsgbot: robh gracefulled all apaches
20:36 RobH: authdns-update rt4694
20:20 RobH: authdns-update
20:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
20:00 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild message cache for 1.21wmf12
19:44 ^demon: jetty freaked out again, forcing a gerrit restart
19:32 binasher: upgraded all coredb mariadb replicas to 5.5.30
19:25 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild message cache for 1.21wmf12
19:08 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1043 db1009'
19:04 logmsgbot: reedy synchronized docroot
19:03 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'db1050 for special s1'
18:56 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1043, db1009 for upgrade'
18:40 logmsgbot: reedy synchronized php-1.21wmf12 'Initial sync of php-1.21wmf12'
18:16 awight: payments cluster updated from c19cc66 to fe4fb96b
18:06 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
18:03 logmsgbot: aaron synchronized wmf-config/PrivateSettings.php 'Removed old testing cruft'
16:19 cmjohnson1: removing/replacing disk 4 ms-be1004
16:01 hashar: Hard restarted Zuul which was dead locked again :-(
13:23 paravoid: depooled, upgraded, restarted and pooled again all ms-fe[1-4]
13:02 paravoid: depooling ms-fe1 for new swift version (patched 1.7.4)
11:27 apergos: restarted both the varnishncsas on niobium, they were giant again
02:30 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Mon Mar 18 02:30:03 UTC 2013

March 17

02:31 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Sun Mar 17 02:31:16 UTC 2013

March 16

14:58 logmsgbot: reedy synchronized php-1.21wmf11/extensions/EducationProgram
14:48 logmsgbot: reedy synchronized php-1.21wmf11/extensions/EducationProgram
14:40 logmsgbot: reedy synchronized php-1.21wmf11/extensions/EducationProgram
13:07 hashar: Mar 16, 2013 1:06:47 PM Jenkins is fully up and running
12:55 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/EducationProgram/includes/pages/EducationPage.php 'Fix fatal error in EducationProgram (Bug 46188)'
12:33 hashar: restarting Jenkins on a quiet day to have it using umask 0002 (aka g+w). That needs a full restart since the umask is set by /usr/bin/daemon See 53990.
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Sat Mar 16 02:29:19 UTC 2013

March 15

23:24 logmsgbot: laner synchronized wmf-config/CommonSettings.php 'Disabling aft5'
22:45 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning es1007,10 at lower weights'
22:35 binasher: rebooted es1007,1010 for kernel upgrades
22:33 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'temporarily pulling es1007, es1010'
22:23 Ryan_Lane: hid domain dropdown for account creation and login on wikitech
22:23 Ryan_Lane: changed wikitech configuration to require email address on account creation
22:23 Ryan_Lane: deployed change 54093 to wikitech
22:03 hashar: Zuul : opened a bug about this week slowness, breakage: https://bugzilla.wikimedia.org/show_bug.cgi?id=46176
21:51 logmsgbot: aaron synchronized php-1.21wmf11/includes 'deployed 08295adcdc4a4130954f52fb1eb5377638cd97bc'
21:36 binasher: db59 -> mariadb 5.5.30
21:30 hashar: restarted Zuul with the safe version ff79197
21:28 hashar: downgrading Zuul to safe version ff79197 (aka stop doing `git remote update` on ref-update events)
21:25 binasher: added new build of mariadb 5.5.30 to precise-wikimedia on brewster
21:20 hashar: hard restarted Zuul. It was locked down again :/
21:18 mutante: installing package upgrades on gallium
21:14 hashar: restarting Zuul, it is locked somehow. Definitely need to update it :(
21:09 RobH: professor services seem back up
20:51 RobH: professor shutting down for fan swap, short downtime (hopefully)
20:47 hashar: integration.mediawiki.org is not redirecting to the new integration.wikimedia.org (apache conf is disabled). Patch in 54068
20:33 mutante: puppet runs on neon broken due to mysql puppet issue - mysql-client-5.5 can not create alias mysql-client: object already exists at /var/lib/git/operations/puppet/manifests/mysql.pp:513
20:31 mutante: restarting pdns on ns0
20:30 mutante: DNS update - removing ipv4/ipv6/ipv6and4/results.labs.wm
19:37 mutante: maerlant - re-adding 2620:0:862:1::80:2 to eth0, starting nginx
19:22 sbernardin: taking down raskin for decommissioning
19:13 mutante: maerlant - dpkg: error processing ganglia-monitor | backing up /var/www/ to tridge /data/other then deleting contents | removing sites-enabled/ipv6and4 | nginx can't bind to [2620:0:862:1::80:2]:443
18:50 mutante: installing more package upgrades on maerlant (wikimedia-lvs-realserver, libc6, nginx...)
18:46 mutante: installing package upgrades on formey
18:44 mutante: fixing broken dpkg/APT on maerlant - dpkg --configure -a , re-run puppet and let it install package upgrades
18:34 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/
18:19 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/
18:19 notpeter: ran puppetd --enable on all mw hosts
18:15 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/
18:07 logmsgbot: mlitn Finished syncing Wikimedia installation... : Update ArticleFeedbackv5 to master
17:52 logmsgbot: mlitn Started syncing Wikimedia installation... : Update ArticleFeedbackv5 to master
17:36 hashar: jenkins: updated jenkins configuration to have its URL default to wikimedia.org domain instead of the legacy mediawiki.org
17:22 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/ 'Update ArticleFeedbackv5 to master'
17:21 hashar: gallium : reloading Zuul so it reports the status url with the wikimedia.org domain
17:19 hashar: integration website going down while we deploy a new layout.
17:03 hashar: gallium : restarted puppet
17:03 hashar: gallium : update /srv/ to latest version of integration/docroot.git ( aaa4b0c ) and moved all content from /srv/org/mediawiki/integration to the new /srv/org/wikimedia/integration
17:00 hashar: stopped puppet on gallium to fix some permissions before puppet changes are merged in and applied.
05:09 Tim: restarting squid on cp1010 for squid redirector test
05:06 Tim: testing new squid redirector
02:30 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Fri Mar 15 02:30:02 UTC 2013
02:16 logmsgbot: reedy synchronized docroot
02:13 logmsgbot: dzahn gracefulled all apaches
01:53 logmsgbot: reedy synchronized wmf-config/
01:52 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
01:52 logmsgbot: reedy gracefulled all apaches
01:44 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/GettingStarted/SpecialGettingStarted.php 'Fix deployed bug in check for editability'
01:44 logmsgbot: reedy synchronized wmf-config/
01:43 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
01:35 logmsgbot: reedy synchronized wmf-config/
01:35 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
01:34 logmsgbot: reedy synchronized docroot
01:28 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
01:27 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
01:21 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/GettingStarted/CategoryRoulette.php 'Fix to use single-parameter version of SRANDMEMBER'
01:14 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
01:13 logmsgbot: dzahn gracefulled all apaches
01:12 logmsgbot: reedy synchronized docroot
01:10 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/NavigationTiming 'Missed a file in earlier sync'
00:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'chapcomwiki readonly'
00:47 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/NavigationTiming/modules/ext.navigationTiming.js 'Additional fields for NavTiming'
00:43 logmsgbot: mflaschen Finished syncing Wikimedia installation... : Deployment for Mobile and E3
00:17 logmsgbot: mflaschen Started syncing Wikimedia installation... : Deployment for Mobile and E3

March 14

23:48 mutante: DNS update - add affcom in prep. to replace chapcom (bug 39482)
23:47 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Updating MF copyright logo path for enwiki'
23:47 logmsgbot: awjrichards synchronized php-1.21wmf11/extensions/MobileFrontend/ 'Updating MobileFrontend per https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2013-03-14'
23:44 mutante: installing package upgrades on sockpuppet
23:30 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Updating MF copyright logo for testwiki'
22:52 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:43 mutante: pa.us.wikimedia.org now redirects to pa-us.wikimedia.org (Wikimedia Pennsylvania) (no sub.sub domains)
22:41 logmsgbot: dzahn gracefulled all apaches
22:34 mutante: disabling notifications for puppet freshness on celsus
22:14 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Enabling NavigationTiming at 1:10,000 sampling factor across cluster.'
22:02 logmsgbot: olivneh synchronized wmf-config 'Enable NavigationTiming on test2'
21:57 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/NavigationTiming 'Update of extension to use EventLogging'
21:51 hashar: Gerrit took roughly 5 minutes to merge the change https://gerrit.wikimedia.org/r/#/c/53412/ ( mediawiki/core ) which means Zuul does nothing during that time waiting for the merge to happen.
21:48 hashar: apparently Gerrit takes ages to submit a change that in turn cause a hard lock on Zuul which is waiting for the merge to happen.
21:39 hashar: Zuul is hanging out not reporting changes :/
21:01 mutante: DNS update - add pa-us wiki to replace pa.us
20:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
20:50 cmjohnson1: authdns update (labsdb1002-3 vlan)
20:45 Ryan_Lane: deleting the jenkins user from ldap
20:36 mutante: DNS update - add zuul as CNAME for gallium
20:34 Ryan_Lane: installed WikiLove on wikitech
20:30 ori-l: Added article edit event to EventLogging, which upped events/sec from ~3.5 to ~30. Expect increased load on vanadium and db1047; bits caches unaffected.
20:20 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/EventLogging 'Log ['HTTP_HOST'] as webHost'
20:05 logmsgbot: olivneh synchronized php-1.21wmf11/includes/WikiPage.php 'Adds CategoryAfterPageAdded / CategoryAfterPageRemoved hooks'
19:32 logmsgbot: aaron synchronized php-1.21wmf11/includes/job/jobs 'deployed 6f76ede163cb114724bce7a1c1b8938e1e30606f '
18:49 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'Enable wgEnableAsyncUploads only on testwikis for now'
18:44 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/ 'Update ArticleFeedbackv5 to master'
18:41 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1009 to full weight'
18:32 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1009 at a low weight for warmup'
18:12 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'Enabled $wgEnableAsyncUploads on all wikis'
18:08 binasher: cleaned up myisam cruft (unused t��ables) on s2
17:49 logmsgbot: aaron synchronized wmf-config 'Enabled redis job aggregator fully'
17:48 cmjohnson1: labsdb1001 powering down
17:42 logmsgbot: aaron synchronized wmf-config 'Set redis job aggregator on for testwiki.'
17:27 logmsgbot: reedy synchronized php-1.21wmf11/extensions/Math/
17:24 binasher: shutting down mysql on db1009, upgrading to mariadb
16:59 cmjohnson1: powering down labsdb1002 and labsdb1003
16:59 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1009 for mariadb migration'
16:57 cmjohnson1: addin mw1041 back to dsh groups
16:26 logmsgbot: mlitn synchronized wmf-config 'Prepare AFTv5 config for deployment new features'
13:15 mark: Set OSPF cost to 10 on csw1-esams:ve7 to facilitate ip multipath
13:15 mark: Fixed OSPF on cr2-knams:xe-0/0/0 - csw1-esams:e8/2 (earlier)
12:17 hashar: gallium setting up /srv/ as a copy of integration/docroot.git
09:03 apergos: restarted varnishncsa.vanadium on niobium, it was using 9gb memory or so
02:25 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Thu Mar 14 02:25:19 UTC 2013
02:21 hashar: jenkins: made Zuul to block changes on operations/debs/ircecho whenever pep8/pyflakes fails.
01:35 mutante: running puppet on celsus to investigate puppet freshness reports from icinga - it started parsoid, but also celsus is a Wikimedia DECOMMISSIONED server (base::decommissioned).
01:10 logmsgbot: reedy synchronized php-1.21wmf11/cache/l10n/
01:00 logmsgbot: reedy synchronized php-1.21wmf11/extensions/Scribunto
00:53 logmsgbot: reedy synchronized wmf-config/

March 13

23:44 mutante: installing package upgrades on zirconium
23:41 logmsgbot: demon Finished syncing Wikimedia installation... :
23:38 Ryan_Lane: deploying change 53464 to OpenStackManager on wikitech
23:23 RobH: no one sync apaches, we wre doing crazy magic on them
23:17 logmsgbot: demon Started syncing Wikimedia installation... :
23:05 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving all search traffic back to eqiad. upgrades over.'
23:00 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving all search traffic back to eqiad. upgrades over.'
22:53 logmsgbot: reedy synchronized wmf-config/
22:44 logmsgbot: demon synchronized php-1.21wmf11/extensions/Scribunto/ 'Updating Scribunto to master'
22:30 logmsgbot: robh gracefulled all apaches
21:46 awight: payments cluster updated to 61d3f1f
21:43 pp-pdf1: upgraded mwlib to 0.15.3
21:38 Tim: on professor: restarted collector
21:22 RobH: repooling all ssl servers (except broken ssl3004) in all sites. new unified certificate is now in use.
21:15 RobH: depooling ssl servers 3 and 4 in each dc site
21:14 RobH: repooling the first two ssl servers per site
21:08 logmsgbot: py synchronized wmf-config/lucene-production.php 'temp moving all earch traffic to pmtpa for upgrades in eqiad'
21:05 RobH: setting first 2 ssl servers per dc site to false in pybal
20:21 Ryan_Lane: repooling ssl1001
20:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Re-enable collection'
20:16 Ryan_Lane: depooling ssl1001
20:08 LeslieCarr: cp1022 unresponsive, power cycling
19:44 LeslieCarr: switched preferred path between esams and eqiad
19:41 LeslieCarr: deploying a possibly mobile front end traffic effecting ipv6 apache request firewall filter
19:26 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable Collection extension'
19:08 LeslieCarr: site was popedotted !
18:47 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving all search traffic back to eqaid'
18:31 logmsgbot: py synchronized wmf-config/lucene-production.php 'temp moving all search traffic to pmtpa for upgrades in eqiad'
18:16 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable scribunto EVERYWHERE'
18:09 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.21wmf11
17:23 pp-pdf1: downgrade to 0.14.3
16:39 pp-pdf1: upgraded mwlib to 0.15.1.
16:32 apergos: cleaned up borked puppet agent on ms-be12 and re-enabled/restarted it
12:30 logmsgbot: anomie synchronized php-1.21wmf11/extensions/Scribunto/common/Hooks.php 'Fix for bug 46031'
12:30 logmsgbot: anomie synchronized php-1.21wmf11/extensions/Scribunto/common/Common.php 'Fix for bug 46031'
12:07 Danny_B: reenabled mail delivery to wikibugs 30 mins ago (disabled automatically for excessive bounces)
11:42 apergos: restarted morebots
01:10 Tim: running updateCollation.php on all uca-* wikis in 6 screens on hume
00:45 logmsgbot: dzahn gracefulled all apaches
00:39 Ryan_Lane: depooling ssl1 ssl2 ssl1001 ssl1002 ssl3001 ssl3002
00:35 Ryan_Lane: pooling ssl3 ssl4 ssl1003 ssl1004 ssl3003
00:31 Ryan_Lane: depooling ssl3 ssl4 ssl1003 ssl1004 ssl3003
00:30 Ryan_Lane: pooling ssl1 ssl2 ssl1001 ssl1002 ssl3001 ssl3002
00:16 Ryan_Lane: depooling ssl1 ssl2 ssl1001 ssl1002 ssl3001 ssl3002
00:12 Tim: manually creating missing /etc/apache2/wmf symlinks on servers that lack them
00:07 logmsgbot: dzahn gracefulled all apaches

March 12

23:39 Tim: updating PHP to php5-3.5+wmf1 for new ICU
23:39 notpeter: upgrading pmtpa search nodes to lucene-search-2 2.1.7wm1
23:34 logmsgbot: tstarling synchronized php-1.21wmf11/maintenance/updateCollation.php
23:34 logmsgbot: tstarling synchronized php-1.21wmf10/maintenance/updateCollation.php
22:33 Ryan_Lane: pooling ssl3003 ssl3 ssl4 ssl1003 ssl1004
22:29 Ryan_Lane: depooling ssl3003 ssl3 ssl4 ssl1003 ssl1004
22:28 Ryan_Lane: pooling ssl3001 ssl3002 ssl1 ssl2 ssl1001 and ssl1002
22:23 mutante: pushing new version 2.7 of wikimedia-task-appserver to wikipedia-precise repo (to fix apache-sanity-check)
22:21 Ryan_Lane: depooling ssl3001 ssl3002 ssl1 ssl2 ssl1001 and ssl1002
21:23 Ryan_Lane: seems that the unified certificate is missing mediawiki.org, depooling the ssl servers that are serving it
21:22 Ryan_Lane: installing the new unified ssl certificate on all ssl servers
19:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:54 RobH: authdns-update for europium to gadolinium ip
18:47 RobH: europium shuttin down
18:43 mark: Imported ganglia 3.5.0 packages into the precise-wikimedia APT repository
18:19 RobH: europium reclaimed per rt4689
18:02 RobH: en-wp.com/org now redirect to en.wikipedia per rt 4669
18:00 RobH: ran apache restart as myself, not root, had a bunch of fails. restarted them as root, hopefully nothing noticable to users
17:58 logmsgbot: robh gracefulled all apaches
17:53 Jeff_Green: live-hack trivial OTRS patch to support filtering on X-Spam-Score for RT #4713, will check in once project is copied to git
17:42 logmsgbot: anomie synchronized php-1.21wmf11/extensions/Scribunto 'Update Scribunto to master'
17:29 logmsgbot: reedy synchronized php-1.21wmf10/includes/Collation.php
17:28 logmsgbot: reedy synchronized php-1.21wmf11/includes/Collation.php
15:42 paravoid: apache graceful to fix bz 46018 / RT 4695
15:39 logmsgbot: root gracefulled all apaches
15:32 hashar: Restarted Zuul process. Was stuck somehow waiting for a job that has been canceled.
13:30 hashar: Jenkins restarted!
12:55 hashar: gallium : upgrading Jenkins (unscheduled)
08:32 hashar: jenkins: live hacked the ant build script for MediaWiki jobs. Now using the local replicated git directory instead of a handmade replication.
05:03 logmsgbot: tstarling synchronized php-1.21wmf10/extensions/Math
05:02 logmsgbot: tstarling synchronized php-1.21wmf11/extensions/Math
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Tue Mar 12 02:52:43 UTC 2013
02:30 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Tue Mar 12 02:29:54 UTC 2013
02:16 Ryan_Lane: deploying change 53248 and 52951 to OpenStackManager on wikitech
02:10 Ryan_Lane: deployed change 53304 to OATHAuth on wikitech
00:38 logmsgbot: kaldari synchronized php-1.21wmf11/extensions/UploadWizard/ 'deploying some bugfixes for UploadWizard'
00:15 awight: payments cluster updated to 207a112b5b36a9e125999acc6f4098d9058ef594

March 11

23:48 AaronSchulz: syncFileBackend.php Running for ceph for all wikis now
23:48 AaronSchulz: copyFileBackend.php finished for ceph for all originals and transcodes (plus math/captcha)
23:38 logmsgbot: aaron synchronized php-1.21wmf11/includes/clientpool/RedisConnectionPool.php 'deployed 5af8702a4b4208ac42fab2fb36da69d13f3e3b92'
23:37 logmsgbot: aaron synchronized php-1.21wmf10/includes/clientpool/RedisConnectionPool.php 'deployed b9fdedc57295a4dd1da684859871aeb734bf70fe'
23:10 Ryan_Lane: adminbot upgraded to 1.6.3
23:09 Ryan_Lane: upgrading adminbot on wikitech-static
22:31 LeslieCarr: upgrading ircecho on neon
22:30 Ryan_Lane: upgrading ircecho on manganese
21:20 mutante: gracefulling eqiad Apaches - wikiquote.net now redirects to wikiquote.org
21:19 logmsgbot: dzahn gracefulled all apaches
21:07 Ryan_Lane: deployed change 53267 to OpenStackManager on wikitech
20:59 logmsgbot: reedy synchronized php-1.21wmf10/cache/interwiki.cdb 'Updating 1.21wmf10 interwiki cache'
20:59 logmsgbot: reedy synchronized php-1.21wmf11/cache/interwiki.cdb 'Updating 1.21wmf11 interwiki cache'
20:30 ^demon: Upgraded gerrit to 2.5.2-1636-g353e384, all's well
20:18 Ryan_Lane: deploying change 52950 to OpenStackManager on wikitech
19:22 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf11
19:05 mutante: patching Bugzilla for security - RT-4553
17:45 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
17:34 mwalker: Shields up! Payments deploy from a6fe052a73f1421028b44305908ec6984dbc50fa to head for Japanese localization updates.
17:29 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
17:19 RobH: search23 shutting down for mb sap per rt4323, has been disabled in pybal
17:18 RobH: search23 out of search pool1, other servers in pool online
17:17 RobH: setting search23 to false for mb swap
17:10 sbernardin: taking down search23 for MB replacement: https://rt.wikimedia.org/Ticket/Display.html?id=3423
16:48 logmsgbot_: aaron synchronized php-1.21wmf11/includes/limit.sh 'deployed e77397fc1de324dfc6d7b515e9440bd564311dd9'
15:57 ottomata: deployed new webstatscollector filter on locke. This replaces spaces in urls.
11:39 apergos: ook that was wrong started apache2 on stafford (puppet uses passenger), now things look ok
10:51 apergos: that would be 10.5. can't add.
10:50 apergos: shot old puppet agents hanging around on stafford, also old apaches from dead puppet master, restarted puppet master, it died around 6.5 hours ago
02:52 logmsgbot_: LocalisationUpdate completed (1.21wmf10) at Mon Mar 11 02:52:06 UTC 2013
02:43 logmsgbot_: krinkle synchronized wmf-config/CommonSettings.php 'I7c8174bcf6ca411014a712a6416d3a74e9fecb3d'
02:41 logmsgbot_: krinkle synchronized wmf-config/InitialiseSettings.php 'I7c8174bcf6ca411014a712a6416d3a74e9fecb3d'
02:29 logmsgbot_: LocalisationUpdate completed (1.21wmf11) at Mon Mar 11 02:29:27 UTC 2013

March 10

22:19 logmsgbot_: reedy Finished syncing Wikimedia installation... : Rebuild message cache
21:59 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuild message cache
21:10 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/CentralAuth/
21:00 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php

March 9

02:54 logmsgbot_: LocalisationUpdate completed (1.21wmf10) at Sat Mar 9 02:54:23 UTC 2013
02:37 mutante: deploying blog WMBlog.php changes (gerrit 28835, gerrit 29823)
02:33 Reedy: Created cn_notice_log on enwiki. Not sure it's actually needed, but there's dberrors appearing here and htere
02:31 logmsgbot_: LocalisationUpdate completed (1.21wmf11) at Sat Mar 9 02:31:06 UTC 2013
02:01 mutante: deploying WMBlog.php changes for EventLogging (gerrit 52945)
01:26 logmsgbot_: olivneh synchronized wmf-config/InitialiseSettings.php '(Change If7ab3d016 / bug 45830) Separate slow parse log destination for private wikis'
01:13 Ryan_Lane: cherry-picking in change 52950 on wikitech
01:12 Ryan_Lane: scratch that
01:11 Ryan_Lane: deploying OATHAuth master to wikitech
00:32 mutante: deleting squid logs on brewster to prevent out-of-disk, and setting logrotate to compress
00:25 Ryan_Lane: upping OpenStackManager to master on wikitech
00:22 logmsgbot_: py synchronized wmf-config/db-pmtpa.php 'removing a second node per shard for streaming dumps. stopping slave'
00:00 Ryan_Lane: upping OpenStackManager to master on wikitech

March 8

22:28 logmsgbot_: reedy synchronized wmf-config/
22:28 logmsgbot_: py synchronized wmf-config/lucene-production.php 'moving pool4 and pool5 search traffic back to eqiad'
22:21 logmsgbot_: reedy synchronized php-1.21wmf10/includes/Collation.php
21:47 awight: payments cluster update to a6fe052a73
20:29 Ryan_Lane: enabling quorum feature for all gluster volumes
19:55 RobH: authdns-update, why? because i can.
19:34 Ryan_Lane: added Babel extension to wikitech
19:20 logmsgbot_: py synchronized wmf-config/db-pmtpa.php 'removing a second node per shard for streaming dumps. stopping slave'
15:35 ottomata: adding metrics.wikimedia.org domain, running authdns-update on dobson
06:21 Tim: updated luasandbox to version 1.4 (with pauseUsageTimer)
04:01 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/Translate
02:53 logmsgbot_: LocalisationUpdate completed (1.21wmf10) at Fri Mar 8 02:52:54 UTC 2013
02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf11) at Fri Mar 8 02:28:34 UTC 2013
01:27 mutante: installing package upgrades on ekrem
01:12 logmsgbot_: catrope synchronized wmf-config/InitialiseSettings.php '20388a06 - Normalize VE config on testwiki'
00:57 mutante: added Set(@ReferrerWhitelist, qw(rt.wikimedia.org:443 rt.wikimedia.org:80));
00:55 Ryan_Lane: deploying change 52592 to wikitech
00:55 mutante: killall lighttpd and restart lighttpd and apache on streber
00:19 logmsgbot_: kaldari Finished syncing Wikimedia installation... :
00:06 mutante: installing openjdk/icedtea jre and other package upgrades on gallium (jenkins)
00:05 LeslieCarr: upgrading cp1001

March 7

23:54 logmsgbot_: kaldari Started syncing Wikimedia installation... :
23:47 LeslieCarr: doing tmh upgrades
23:47 mutante: bravely rebooting streber - expect short RT and Observium outage
23:25 LeslieCarr: upgrading solr boxes
23:14 mutante: installing kernel on streber
23:11 ^demon: gerrit: disabling hooks-its plugin for the time being. It's working (yay) but it's overly spammy until we fix it next week.
23:04 LeslieCarr: dist-upgrading pmtpa memcache
23:04 mutante: dist-upgrading zhen
22:38 mutante: rebooting yvon
22:34 mutante: installing package upgrades on yvon despite (defunct apt-get procs)
22:31 mutante: salt pkg.upgrade on yvon
22:28 Ryan_Lane: rebooting marmontel
22:25 mutante: salt pkg.upgrade on spence
22:21 mutante: salt pkg.upgrade on zirconium
22:18 LeslieCarr: upgrading all analytics machiens
22:06 mutante: reboot formey after dist-upgrade via salt
20:05 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php 'fa0a56a89 - parsoidcache VIP in pmtpa'
19:37 logmsgbot_: bsitu synchronized wmf-config/CommonSettings.php 'Add Eventlogging configuration to Echo'
19:17 logmsgbot_: pgehres synchronized php-1.21wmf11/extensions/CentralNotice/ 'Updating CEntralNotice to master, resolving bug 45846'
18:41 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
18:39 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
17:36 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
16:07 cmjohnson1: reseating disk4 on ms-be1004
15:28 mark: Reinstalled cp3003
15:00 mark: Restarted gmetad on nickel
02:53 logmsgbot_: LocalisationUpdate completed (1.21wmf10) at Thu Mar 7 02:53:50 UTC 2013
02:29 logmsgbot_: LocalisationUpdate completed (1.21wmf11) at Thu Mar 7 02:29:38 UTC 2013
01:53 mutante: installing package upgrades on sodium
00:54 RobH: kicked icinga
00:49 LeslieCarr: restarted pybal on lvs3
00:40 RobH: puppet locked on ms1004, killall puppet, refired
00:39 RobH: puppet halted run on cp1003, left in locked state, killed all puppet instances and refired, cleared.
00:33 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/RSS
00:15 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php

March 6

23:19 LeslieCarr: reinstalling celsus as an internal host
23:07 logmsgbot_: reedy synchronized php-1.21wmf10/resources/ 'touch'
23:06 logmsgbot_: reedy synchronized php-1.21wmf11/resources/ 'touch'
23:05 logmsgbot_: reedy synchronized wmf-config 'touch'
22:54 logmsgbot_: reedy synchronized wmf-config/
22:53 logmsgbot_: reedy synchronized wmf-config/
22:40 mutante: starting haproxy on brewster
22:39 logmsgbot_: reedy Finished syncing Wikimedia installation... : Rebuild message cache for Wikibase deploy
22:39 notpeter: reslaving db36 and db38
22:29 RobH: authdns-update
22:20 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/Wikibase
22:20 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuild message cache for Wikibase deploy
22:11 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
22:10 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
22:09 logmsgbot_: reedy synchronized wikidataclient.dblist
22:07 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
22:02 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
21:55 logmsgbot_: reedy cleared profiling data
21:55 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Enable Wikibase Client on all wikipedias'
21:41 awight: updated payments cluster to 39a265489
21:16 logmsgbot_: reedy synchronized php-1.21wmf10/extensions/
20:54 logmsgbot_: reedy synchronized wmf-config/
20:38 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/DataValues/
20:29 Reedy: Creating and populating site_identifiers table on all Wikipedias
20:28 Reedy: Creating and populating sites table on all Wikipedias
20:27 logmsgbot_: olivneh synchronized wmf-config/CommonSettings.php 'Adding redis host config. var for GettingStarted'
20:12 logmsgbot_: reedy synchronized wmf-config/
20:07 logmsgbot_: reedy synchronized wmf-config/
20:04 logmsgbot_: reedy synchronized php-1.21wmf10/extensions/Wikibase
20:04 logmsgbot_: reedy synchronized php-1.21wmf10/extensions/DataValues
20:03 logmsgbot_: reedy synchronized php-1.21wmf10/extensions/Diff
19:57 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/Wikibase
19:56 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/DataValues
19:43 notpeter: restarting all jobrunners
19:34 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
19:29 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
19:28 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource to 1.21wmf11
19:25 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: aawiki back to 1.21wmf10 for Aaron
19:21 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
19:20 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikimedia and wikiquote to 1.21wmf11
19:18 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks and wikinews to 1.21wmf11
19:15 Reedy: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikivoyage, wikiversity and wiktionary to 1.21wmf11
19:10 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Special wikis to 1.21wmf11
19:07 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Closed, private and fishbowl to 1.21wmf11
19:03 mutante: gracefulling apaches to remove 1.17 aliases (gerrit 50609)
19:03 logmsgbot_: dzahn gracefulled all apaches
18:58 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
18:47 mutante: regenerating mailman list archives for wmfreqs
18:44 mutante: renaming mailman list 'allhands' to 'wmfreqs' (RT-4640)
18:36 logmsgbot_: aaron synchronized php-1.21wmf11/maintenance/nextJobDB.php 'deployed cbc98b09fb0a13b387d82bafab825608e2bb0b3b'
18:35 logmsgbot_: aaron synchronized php-1.21wmf11/includes/job/JobQueueGroup.php 'deployed cbc98b09fb0a13b387d82bafab825608e2bb0b3b'
18:35 logmsgbot_: aaron synchronized php-1.21wmf11/includes/DefaultSettings.php 'deployed cbc98b09fb0a13b387d82bafab825608e2bb0b3b'
18:10 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'clear config cache'
18:09 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'clear config cache'
18:01 logmsgbot_: hashar synchronized wmf-config 'deploying 52340 adapt Lucene configuration file to support beta'
17:59 hashar: deploying 52340 adapt Lucene configuration file to support beta
07:08 jeremyb_: 02:52:46 <+logmsgbot> !log LocalisationUpdate completed (1.21wmf11) at Wed Mar 6 02:52:46 UTC 2013
07:08 jeremyb_: 02:36:35 <+logmsgbot> !log reedy synchronized wmf-config/InitialiseSettings.php
07:08 jeremyb_: 02:28:21 <+logmsgbot> !log LocalisationUpdate completed (1.21wmf10) at Wed Mar 6 02:28:21 UTC 2013
07:08 jeremyb_: 01:33:50 <+logmsgbot> !log csteipp synchronized wmf-config/CommonSettings.php 'Revert https patch'
07:07 jeremyb_: 01:08:27 <+logmsgbot> !log reedy synchronized wmf-config/CommonSettings.php 'Revert wgSecureLoginDefaultHTTPS'
07:07 Ryan_Lane: restarted morebots
01:10 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Set wgSecureLoginDefaultHTTPS to false for testing'
00:37 binasher: added new deb of twemproxy 0.2.3 built with --enable-debug=log to precise-wikimedia
00:21 logmsgbot_: csteipp synchronized wmf-config/CommonSettings.php 'Enabling HTTPS login'
00:14 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings-labs.php
00:09 hashar: snapshot1002: rsync: change_dir#3 "/apache/common-local" failed: No such file or directory (2)
00:09 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings-labs.php

March 5

23:57 logmsgbot_: py synchronized wmf-config/db-eqiad.php 'increasing db1009 to full weight'
23:54 LeslieCarr: reinstalling constable as an internal host
23:49 RoanKattouw: Removed decommissioned srv226-234 from /etc/dsh/group/mediawiki-installation
23:47 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php '6fd144c44f26 - Use new service IP for Parsoid in eqiad'
23:46 logmsgbot_: py synchronized wmf-config/db-pmtpa.php 'repooling previously crashed mariadb pmtpa slaves'
23:36 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'ombudsmenwiki logo'
23:26 logmsgbot_: py synchronized wmf-config/db-pmtpa.php 'commenting out one slave per shard that's mysqldumping'
23:18 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
23:15 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
23:13 notpeter: stopping slave and starting mysql dumps on dbs 71, 57, 66, 65, 55, 50, and 68
23:00 logmsgbot_: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
22:40 logmsgbot_: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
22:36 logmsgbot_: py synchronized wmf-config/db-eqiad.php 'repooling db1009 at 25% weight'
22:18 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
22:17 logmsgbot_: reedy synchronized wmf-config/
22:14 LeslieCarr: restarting pybal on lvs3
22:14 LeslieCarr: restarting pybal on lvs4
22:13 LeslieCarr: restarting pybal
22:07 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
22:05 logmsgbot_: reedy synchronized wmf-config/
21:57 LeslieCarr: restarting pybal on lvs1003
21:55 LeslieCarr: restarting pybal on lvs1006
21:44 Coren: tools rejiggered the webproxy config to be smarter about paths not leading to specific tools
21:43 LeslieCarr: adding parsoidcache service to lvs1006
21:42 mutante: gracefulling eqiad Apaches - add ombudsmen.wm (bug 45744)
21:41 logmsgbot_: dzahn gracefulled all apaches
21:35 mutante: restarting pdns on ns2
21:34 mutante: DNS update - add ombudsmen (bug 45744)
20:49 logmsgbot_: reedy synchronized docroot
20:35 mutante: installing package upgrades on hooper
20:26 logmsgbot_: reedy synchronized wmf-config/
20:22 mutante: installing package upgrades on hume
20:00 mutante: wikipedia.ee now redirects to et.wikipedia, wikimedia.ee now redirects to et.wikimedia
19:56 mutante: gracefulling eqiad Apaches, push wiki(m|p)edia.ee redirects
19:56 logmsgbot_: dzahn gracefulled all apaches
19:18 K4-713: updated payments cluster to 9b063ee2e399a3b
18:42 logmsgbot_: mlitn synchronized wmf-config/InitialiseSettings.php 'Disable AFTv4 (entirely) and AFTv5 (leaving only opt-in) on enwiki'
18:04 cmjohnson1: restarting dhcpd3 service brewster
16:55 cmjohnson1: mw1085 powering off to troubleshoot DIMM error
05:35 ori-l: EventLogging db on db1047: ALTER DATABASE log CHARACTER SET utf8 COLLATE utf8_unicode_ci; (was: latin1)
02:49 logmsgbot_: LocalisationUpdate completed (1.21wmf11) at Tue Mar 5 02:48:58 UTC 2013
02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf10) at Tue Mar 5 02:28:16 UTC 2013
02:03 K4-713: updated payments cluster to 5361c795f4bf31980f
00:46 mutante: fixing exim/mail permissions on neon to make Icinga paging work. missing +x on /var/log/exim4 dir

March 4

23:12 mutante: starting dhcp3 on brewster
23:11 LeslieCarr: p.s. bonding on foundry sucks
23:10 LeslieCarr: bringing down virt2 network to bond it
22:18 mutante: gracefulling eqiad Apaches to push indiawikipedia redirect
22:17 logmsgbot_: dzahn gracefulled all apaches
22:03 logmsgbot_: py synchronized wmf-config/db-pmtpa.php 'shuffling around pmtpa enwiki slaves'
21:50 LeslieCarr: messing around with the network of virt2 - expect possible pmtpa labs network issues
21:50 ottomata: staring puppet back up on oxygen, packet loss is better after disabling a filter
21:30 cmjohnson1: authdns update
21:17 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.21wmf11
21:16 logmsgbot_: reedy synchronized php-1.21wmf11
21:15 logmsgbot_: reedy synchronized php-1.21wmf10
21:14 RobH: shutting down professor for a fan swap, should be a short window
21:11 cmjohnson1: authdns update
21:05 ottomata: stopping puppet on oxygen to see if we can disable filters and reduce packet loss
20:55 Ryan_Lane: restarted icinga
20:53 logmsgbot_: reedy Finished syncing Wikimedia installation... : Rebuild Message Cache for 1.21wmf11
20:52 RobH: had to restart morebots, but it wasnt really dead, opps
20:51 RobH: had to restart icinga
20:51 RobH: Why arent you taking all the !log
20:50 RobH: i hate you morebots
20:28 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuild Message Cache for 1.21wmf11
19:55 logmsgbot_: reedy Started syncing Wikimedia installation... : Build 1.21wmf11 message cache
19:41 logmsgbot_: reedy synchronized php-1.21wmf11/
19:34 logmsgbot_: reedy synchronized wmf-config/
19:34 logmsgbot_: reedy synchronized live-1.5/
19:33 logmsgbot_: reedy synchronized docroot
19:30 hashar: on gallium: stopping puppet, cleaning defunct and restarting puppet
19:23 logmsgbot_: reedy synchronized wmf-config/
19:20 mutante: installing package upgrades on sockpuppet
19:13 sbernardin: taking down amaranth for hw replacement: https://rt.wikimedia.org/Ticket/Display.html?id=4497
18:44 cmjohnson1: authdns update
18:17 K4-713: updated payments cluster to 059969c3d29f685c
17:44 cmjohnson1: authdns update
16:56 logmsgbot_: py synchronized wmf-config/db-eqiad.php 'depooling db1009 while investigating error'
16:24 sbernardin: rebooting db70 to reset idrac: https://rt.wikimedia.org/Ticket/Display.html?id=4628
16:14 sbernardin: db27 being taken down for decom per: https://rt.wikimedia.org/Ticket/Display.html?id=4623
10:12 apergos: cleared some space on brewster by rmeoving old squid access/storage logs (2.2gb worth in 3 files!), its root partition was full, restarted squid
10:12 apergos: reinstalled wikimedia-app-taskserver on fenari (how did it get removed?), needed for various cron jobs among other things
02:04 logmsgbot_: LocalisationUpdate failed (1.21wmf10) at Mon Mar 4 02:04:10 UTC 2013

March 3

02:25 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Sun Mar 3 02:25:44 UTC 2013

March 2

02:27 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Sat Mar 2 02:27:35 UTC 2013
01:17 mutante: powercycling emery
00:35 mutante: rebooting emery for kernel upgrade
00:29 mutante: installing package upgrades on emery

March 1

23:59 cmjohnson1: authdns update
23:56 K4-713: updated payments cluster to c8442435a9e2f5c
23:29 notpeter: dropping limesurvey from db9, backup in db9:/a/backup_dumps
22:53 RoanKattouw: Make that apache2-utils, that's the actual name of the package
22:51 RoanKattouw: Temporarily installing apachebench on wtp1004 per Gabriel's request
22:49 RoanKattouw: Depooled wtp1004 so Gabriel can use it for benchmarking
22:46 mwalker: reverting payments to dced5328fd6ae4b395622abc41d2afa0ccad21c8 from 4a3ca6cc36b9d78dbb015c1bf93bbafb14eb0799 because of bug report that some people are not able to make donations due to 'required field city' being missing and then being unable to provide it
22:34 cmjohnson1: authdns update
22:15 cmjohnson1: auth-dns update
22:12 mutante: DNS update - remove status cname
21:56 cmjohnson1: taking wtp1001 down to replace w/new box
21:32 mwalker: pushing 4a3ca6cc36b9d78dbb015c1bf93bbafb14eb0799 to the payments cluster (again, because the instant reviewer, me, was broken)
21:23 mwalker: attempted deploy of payments cluster to new version, but then had to roll back to dced5328fd6ae4b395622abc41d2afa0ccad21c8
20:02 mutante: blocked en.wikivoyage from bugmenot
20:01 logmsgbot: dzahn synchronized ./docroot/wikivoyage.org/block_8708f.html
19:55 logmsgbot: dzahn synchronized ./docroot/wikivoyage.org/block_ab2c3.html
19:55 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'parser cache max age 30d'
19:37 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1043 to s1 watchlist service'
19:31 mutante: DNS update - add CNAME for status.wp/status.wm
19:28 logmsgbot: maxsem synchronized wmf-config/mobile.php 'https://gerrit.wikimedia.org/r/#/c/51291/'
19:13 mutante: install package upgrades on marmontel
19:06 mutante: install package upgrades on streber
19:04 mutante: create missing backup dir on streber to enable RT_Shredder plugin
19:04 Ryan_Lane: maxsem synchronized wmf-config/InitialiseSettings.php 'make upload endpoint for Commons local https://gerrit.wikimedia.org/r/#/c/51612/'
19:03 Ryan_Lane: LocalisationUpdate completed (1.21wmf10) at Fri Mar 1 02:31:13 UTC 2013
19:03 Ryan_Lane: csteipp synchronized php-1.21wmf10/includes
19:03 mutante: db9 - zipping old sql dumps of drupal and civirm to prevent full disk
19:03 Ryan_Lane: dzahn synchronized ./docroot/wikivoyage.org/mywotf83102384bfcd1e52152.html
19:03 Ryan_Lane: restarted glusterd service on labstore1-4
19:03 Ryan_Lane: pgehres Finished syncing Wikimedia installation... : Updating i18n for DonationInterface-langonly
19:02 Ryan_Lane: pgehres Started syncing Wikimedia installation... : Updating i18n for DonationInterface-langonly
19:02 Ryan_Lane: binasher: shutting down mysql on db1043, converting to mariadb
19:02 LeslieCarr: powering back on spence
19:02 Ryan_Lane: asher synchronized wmf-config/db-eqiad.php 'pulling db1043 and shifting watchlist q's to db1050'
19:02 Ryan_Lane: pgehres synchronized php-1.21wmf10/extensions/DonationInterface/ 'Updating DonatonInterface-langonly'
19:02 Ryan_Lane: LeslieCarr: powering off spence
19:01 Ryan_Lane: asher synchronized wmf-config/lucene.php 'sending search-pool4 traffic to pmtpa, where the index/host distribution has been rebalanced'
19:01 Ryan_Lane: notpeter: upgrading mariadb boxes to 5.5.29
19:01 Ryan_Lane: tstarling synchronized wmf-config/InitialiseSettings.php
19:01 Ryan_Lane: Jeff_Green: stopped gmetad on neon for testing
19:01 Ryan_Lane: olivneh synchronized php-1.21wmf10/extensions/GettingStarted 'Update to split test (3/3)'
19:00 Ryan_Lane: olivneh synchronized php-1.21wmf10/extensions/E3Experiments 'Update to split test (2/3)'
19:00 Ryan_Lane: olivneh synchronized php-1.21wmf10/extensions/GuidedTour 'Update to split test (1/3)'
19:00 Ryan_Lane: mutante: create missing dump directory on streber, enables "RT-shredder" plugin which comes with RT since 3.8
19:00 Ryan_Lane: olivneh synchronized php-1.21wmf10/extensions/EventLogging 'Updates to JavaScript API'
19:00 Ryan_Lane: changing A record for wikitech.wm.o and changing labsconsole.wm.o into a cname to wikitech
18:59 Ryan_Lane: disabled editing on wikitech.wikimedia.org
18:59 Ryan_Lane: binasher: resharding search-pool4 in pmtpa - restarted lsearchd on all local nodes and indexers on searchidx2
18:59 mwalker: ... therefore the payments cluster is now running dced5328fd6ae4b395622abc41d2afa0ccad21c8
18:59 mutante: restart nagios-nrpe,gmond,apache,salt-minion on bastion hosts
18:59 Ryan_Lane: binasher: rebuilding db35 from a hotbackup of db55 (s5); testing a new build of mariadb 5.5.29
18:59 Ryan_Lane: reedy synchronized php-1.21wmf10/cache/interwiki.cdb 'Updating 1.21wmf10 interwiki cache'
18:59 Ryan_Lane: reedy synchronized php-1.21wmf10/cache/interwiki.cdb 'Updating 1.21wmf10 interwiki cache'
18:58 Ryan_Lane: reedy synchronized php-1.21wmf10/extensions/WikimediaMaintenance
18:58 Ryan_Lane: apergos: restarted icinga (not sure why it stopped, log file simply said 'caught TERM signal', in any case it complained that Error: Could not create external command file '/var/lib/nagios/rw/nagios.cmd' as named pipe )
18:58 mwalker: enabled sofort forms for CH and DE on the payments cluster
18:39 mutante: installing package upgrades on bast1001
18:37 mutante: installing package upgrades on fenari
17:40 ^demon: rescanned all trackingids for gerrit on manganese
17:31 ^demon: upgrading gerrit to 2.5.2-1506-g278aa9a on manganese/formey
17:07 Ryan_Lane: test

February 28

12:58 apergos: restarted icinga (not sure why it stopped, log file simply said 'caught TERM signal', in any case it complained that Error: Could not create external command file '/var/lib/nagios/rw/nagios.cmd' as named pipe )
06:28 apergos: stabbed searchpool4 and restarted lucene search etc gah hate hate hate
03:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
03:30 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
03:12 logmsgbot: reedy synchronized docroot
03:10 logmsgbot: reedy synchronized docroot
03:09 logmsgbot: reedy synchronized docroot
02:30 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Thu Feb 28 02:30:14 UTC 2013
01:10 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
00:58 RobH: dsh restarted eqiad apaches in progress
00:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
00:57 logmsgbot: robh gracefulled all apaches
00:54 logmsgbot: reedy synchronized wmf-config/
00:32 RobH: authdns-update to support wikinews.de
00:27 logmsgbot: reedy synchronized wmf-config/
00:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Lowering throttle to 5/day for new users, and'

February 27

23:26 Reedy: Running sync-common on mw1111. Reporting failed opening wmf-config/mc.php
22:57 mutante: rebooting zirconium (planet)
22:54 mutante: installing package upgrades on zirconium
22:52 logmsgbot: reedy synchronized wmf-config/
22:46 mutante: rebooting bast1001
22:36 mutante: installing package upgrades on bast1001
22:16 logmsgbot: reedy synchronized wmf-config/
21:44 RobH: rdb1001 & rdb1002 allocated, network setup, and ready for handoff, per rt4608
21:36 RobH: authdns-update for rdb1001-1002
21:36 RobH: authdns-update for sdb1001-1002
21:31 Tim: running resetUserTokens.php on all wikis, in screen on hume, to fix user_token field corruption (bug 41586)
21:26 logmsgbot: tstarling synchronized php-1.21wmf10/includes/User.php 'user_token fixes'
20:05 K4-713: updated payments cluster to fcbf9211c9836b
19:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.21wmf10
18:55 LeslieCarr: rebooting arsenic
18:42 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild message cache for EducationProgram
18:25 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild message cache for EducationProgram
17:57 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/
17:51 K4-713: updated payments cluster to eab489d0889a
17:13 mutante: rebooting fenari for kernel upgrade
08:33 apergos: restarted lucene search on search1015
08:04 logmsgbot: reedy synchronized live-1.5/static-1.21wmf4/
08:04 logmsgbot: reedy synchronized docroot/bits/static-1.21wmf4/
07:58 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
07:05 Krinkle: done adding 1.21wmf3, 1.21wmf4, 1.21wmf5 (bug 44570)
07:04 logmsgbot: krinkle Finished syncing Wikimedia installation... :
06:29 apergos: snapshot1002 down for reinstallation
06:27 logmsgbot: krinkle Started syncing Wikimedia installation... :
06:15 Krinkle: adding 1.21wmf3, 1.21wmf4, 1.21wmf5 (bug 44570)
06:15 Krinkle: added 1.21wmf1, 1.21wmf2 (bug 44570)
05:35 apergos: restarted lucene search on search1016
04:57 logmsgbot: krinkle Finished syncing Wikimedia installation... :
04:37 logmsgbot: krinkle Started syncing Wikimedia installation... :
02:57 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Wed Feb 27 02:57:22 UTC 2013
02:31 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Wed Feb 27 02:31:06 UTC 2013
01:35 Krinkle: Bringing back checkouts of MediaWiki 1.21wmf5 - 1.21wmf1 on fenari for bug 44570
01:35 Jeff_Green: dist-upgrade and reboot grosley
00:25 logmsgbot: tstarling cleared profiling data

February 26

23:29 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend/ 'https://gerrit.wikimedia.org/r/51053 https://gerrit.wikimedia.org/r/51057'
23:27 logmsgbot: maxsem synchronized php-1.21wmf10/extensions/MobileFrontend/ 'https://gerrit.wikimedia.org/r/51053 https://gerrit.wikimedia.org/r/51057'
23:04 K4-713: synchronized payments cluster to 596b854787132c730
22:35 mutante: installing package upgrades on fenari
22:31 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
22:31 LeslieCarr: nagios is dead! spence.wikimedia.org if you want to access it directly
22:16 mutante: gracefulling eqiad Apaches to push redirect for wikivoyage.eu
22:15 logmsgbot: dzahn gracefulled all apaches
22:15 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
20:58 RobH: stat1 puppet runs fail due to dan account error, i am working on it, but i also want to run out and get something to eat, so leaving in broken state for 10 minutes until i return to work on it
20:21 mutante: DNS update - add wikiovyage.eu
19:34 Jeff_Green: package updates & reboots of payments hosts
19:13 paravoid: freezing all exim4 bounces to restore normal email delivery
18:58 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
17:54 paravoid: restarting opendj on sanger
17:45 AaronSchulz: Copying math files into ceph
15:14 mark: suspended i-000001d7 and i-0000039e for sending out malicious emails
15:06 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/GeoData/solrupdate.php
15:05 logmsgbot: maxsem synchronized php-1.21wmf10/extensions/GeoData/solrupdate.php
12:26 apergos: temp blacklist live hack on sodium for exim4 to prevent some spam, see /etc/exim4/blacklist for that
11:20 logmsgbot: mlitn synchronized php-1.21wmf10/extensions/ArticleFeedback/api/ApiArticleFeedback.php 'Update ArticleFeedback to master'
06:53 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/
06:43 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
06:40 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
06:28 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
06:26 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
06:21 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
03:10 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
03:06 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
03:04 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
03:03 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Tue Feb 26 02:52:01 UTC 2013
02:46 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
02:44 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
02:41 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram
02:37 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram
02:29 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Tue Feb 26 02:28:06 UTC 2013
02:11 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram
01:59 Ryan_Lane: test
01:59 Ryan_Lane: test
01:58 Ryan_Lane: test
01:40 K4-713: updated payments cluster to ce28b210374d8
00:20 mutante: mc1001,ms1004,mw1029,mw1059,mw1070,mw1157 - fixed puppet freshness
00:03 mutante: amssq33,amssq44 - fix more puppet runs

February 25

23:54 mutante: knsq26,knsq28 - deleting puppet lock file, fix puppet runs
23:45 logmsgbot: reedy synchronized wmf-config/
23:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable SearchExtraNS on commonswiki'
23:34 mutante: re-enabling disabled puppet on srv253,srv235
22:16 Ryan_Lane: restarting opendj/pdns on virt0
22:02 Reedy: Ran namespaceDupes.php on huwikt
21:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
20:28 logmsgbot: reedy synchronized docroot
20:27 logmsgbot: reedy synchronized wmf-config/
19:50 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuilding message cache to update WikimediaMessages
19:40 mutante: powercycling frozen niobium
19:32 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuilding message cache to update WikimediaMessages
19:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf10
18:54 mutante: wikivoyager.de and wikivoyager.org transferred and both redirect to wikivoyage.org
18:50 mutante: deleting foundation-news-l (deprecated, replaced by WikimediaAnnounce-l)
18:49 logmsgbot: reedy synchronized php-1.21wmf10/extensions/WikimediaMessages
18:31 AaronSchulz: Copying all captchas into ceph
18:11 logmsgbot: reedy synchronized php-1.21wmf10/extensions/Wikibase
06:38 apergos: restarted lucene search on search1016
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Mon Feb 25 02:52:32 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Mon Feb 25 02:28:55 UTC 2013
01:15 mark: Powercycled strontium
01:07 paravoid: powercycling niobium (both unresponsive from the load)
01:06 paravoid: powercycling arsenic

February 24

06:28 apergos: restarted search1016 lucene
02:54 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sun Feb 24 02:54:30 UTC 2013
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Sun Feb 24 02:29:29 UTC 2013

February 23

21:45 logmsgbot: reedy synchronized php-1.21wmf9/cache/interwiki.cdb
21:44 logmsgbot: reedy synchronized php-1.21wmf10/cache/interwiki.cdb
21:43 Reedy: mw110 is asking for a password
06:13 apergos: restarted lucene search on search1016
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sat Feb 23 02:52:10 UTC 2013
02:27 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Sat Feb 23 02:27:17 UTC 2013
00:43 RobH: lvs1003 had a ton of hung puppet processes (no output on them when tracing, so dunno whats up). killed them and kicked manual run

February 22

23:59 RobH: argon is shut down now
23:55 RobH: argon is going offline, killing our limesurvey install
23:29 paravoid: deploying squid config for upload's new transcoded container
23:26 paravoid: deploying squid
22:55 Ryan_Lane: changing dns entry for wikitech-static
22:41 RobH: mw110 throwing odd puppet errors, will come back to it later
22:41 RobH: sq41 reinstalled, throws endless puppet errors during redeployment
22:07 MaxSem: reindexing complete
21:39 RobH: harmon and copper online with OS installed for datadumps misc host support
21:39 MaxSem: Nuking GeoData Solr index and reindexing
21:38 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/GeoData/solrupdate.php 'https://gerrit.wikimedia.org/r/#/c/50388/'
21:35 logmsgbot: maxsem synchronized php-1.21wmf10/extensions/GeoData/solrupdate.php 'https://gerrit.wikimedia.org/r/#/c/50388/'
21:05 RobH: installing harmon and copper into dump generation misc use
20:17 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/GuidedTour/modules/tours/gettingstarted.js 'Resume split test (3/3)'
20:16 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/GettingStarted/resources/ext.gettingstarted.js 'Resume split test (2/3)'
20:16 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/E3Experiments/experiments/openTask.js 'Resume split test (1/3)'
20:14 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/Math 'Updating Math extension to fix db caching behavior (Change-Id: I9b690e55001859c97fd40330272791d49ec6de75)'
20:13 logmsgbot: olivneh synchronized php-1.21wmf10/extensions/Math 'Updating Math extension to fix db caching behavior (Change-Id: I9b690e55001859c97fd40330272791d49ec6de75)'
20:09 RobH: authdns-update
19:58 RobH: sq41 boot order was fubar, fixed, reinstalling (as reboot into installer wiped mbr)
19:56 RobH: wtp1002-1004 os installed
19:34 Jeff_Green: start mysqldump of otrs database on db49
19:18 RobH: kicked ntpd manually on mw1045
19:09 RobH: sq41 reinstalling
18:57 RobH: mw112-125 back into pybal api pool
18:56 RobH: mw86-99 & mw111 back into pybal pool
18:49 logmsgbot: robh Finished syncing Wikimedia installation... :
18:37 logmsgbot: robh Started syncing Wikimedia installation... :
17:53 sbernardin: taking down sq41 to replace controller card.... https://rt.wikimedia.org/Ticket/Display.html?id=4550
16:24 mutante: tartupeedia.ee now redirects to et.wp portal page (bug 44893)
16:12 mutante: gracefulling eqiad Apaches via dsh to push tartupeedia.ee redirect (bug 44893)
16:11 logmsgbot: dzahn gracefulled all apaches
16:07 mutante: mw110 - sits at partioning screen in installer
15:47 hashar: gallium removed live hack in /srv/org/wikimedia/doc/puppetsource/manifests/role/labsmediawiki.pp
14:52 ^demon: restarted gerrit on manganese, jetty flapping again
14:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
13:14 mark: Halted ms-be3001-3003 for controller replacement
13:11 mark: Halted ms-be3004 for controller replacement
13:07 mark: Halted cp3009 for controller replacement
12:20 mark: Halted cp3010 for controller replacement
11:23 mutante: DNS update - add indiawikipedia.com
07:33 apergos: restarted lucene search on search1016
06:49 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
05:33 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
05:33 logmsgbot: tstarling synchronized docroot/bits/favicon/wikipedia.ico
05:20 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
05:18 logmsgbot: tstarling synchronized docroot/bits
02:53 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Fri Feb 22 02:53:05 UTC 2013
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Fri Feb 22 02:29:09 UTC 2013
01:57 binasher: migrating wikivoyage and wikidata external blob stores to innodb
01:08 logmsgbot: aaron synchronized wmf-config/filebackend.php 'deployed f699c5a9ad2f780dc18863592dcae4a51ba6d841'

February 21

23:21 logmsgbot: olivneh synchronized php-1.21wmf10/extensions/CodeEditor 'Syncing patch that disables background linting'
23:18 logmsgbot: mflaschen Finished syncing Wikimedia installation... : Deploy E3Experiments, GettingStarted, and GuidedTour
23:02 logmsgbot: mflaschen Started syncing Wikimedia installation... : Deploy E3Experiments, GettingStarted, and GuidedTour
22:35 logmsgbot: awjrichards synchronized php-1.21wmf9/extensions/MobileFrontend/
22:33 logmsgbot: awjrichards synchronized php-1.21wmf10/extensions/MobileFrontend
22:12 logmsgbot: awjrichards synchronized wmf-config/mobile.php 'Add photo upload schema for event logging'
21:56 notpeter: running do-release-upgrade on dbs 52, 39, 51, and 35
21:49 logmsgbot: olivneh synchronized php-1.21wmf10/extensions/EventLogging
21:48 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'removing 1 slave for s2-5 from db-secondary for upgrades and maria'
21:48 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/EventLogging
21:47 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/CodeEditor
21:37 notpeter: restarting mysql on db1047 to pick up conf changes
21:31 logmsgbot: olivneh synchronized php-1.21wmf10/extensions/SyntaxHighlight_GeSHi 'Disable highlighting of very large files.'
21:23 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/SyntaxHighlight_GeSHi 'Disable highlighting of very large files.'
21:05 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php Enabling CodeEditor for Schema namespace on test2 and metawiki.
20:02 binasher: some wikis have myisam page_props tables. converting all to innodb via osc
18:39 Ryan_Lane: enabled Extension:OpenID as a provider on labsconsole
11:04 hashar: Ariel saved Gerrit by restarting it!
10:51 hashar: huge CPU spike on manganese (gerrit host)
10:06 hashar: Jenkins: reseting git branch specifier :-/
09:32 hashar: Jenkins: removing git branch specifier from all mediawiki-core jobs.
05:50 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'shell cgroup'
02:54 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Thu Feb 21 02:54:17 UTC 2013
02:30 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'new extension1 shard (as an externalload config) - initially for AFTv5'
02:30 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'new extension1 shard (as an externalload config) - initially for AFTv5'
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Thu Feb 21 02:28:20 UTC 2013
00:53 RobH: issues with new mw servers 86+, took back out of pybal until i can troubleshoot
00:49 logmsgbot: reedy synchronized php-1.21wmf9/extensions/WikimediaMaintenance/
00:48 logmsgbot: reedy synchronized php-1.21wmf10/extensions/WikimediaMaintenance/
00:34 binasher: moved wikidatawiki tables on s3 to wikidatawiki_old to keep around for a time just in case; then "drop database wikidatawiki"
00:20 logmsgbot: asher synchronized wmf-config 'enabling wikidatawiki on shard s5'
00:18 logmsgbot: asher synchronized s5.dblist 'adding wikidatawiki'
00:17 logmsgbot: asher synchronized s3.dblist 'removing wikidatawiki'

February 20

22:48 RobH: authdns-update
22:35 binasher: importing wikidatawiki with schema changes into the s5 master, with sql_log_bin disabled
22:24 binasher: importing wikidatawiki with schema changes into the inactive pmtpa side of the s5 shard, with replication of the import just in pmtpa. ignore repl warnings for db{35,44,45,55}
21:48 RobH: authdns-update for wtp1002-1004
21:45 logmsgbot: maxsem synchronized wmf-config/CommonSettings.php 'Take pmtpa Solr servers out of rotation, increase load on master: https://gerrit.wikimedia.org/r/#/c/50056/'
21:41 RobH: ignore db66 errors, asher is working on it
21:25 binasher: running wikidata wb_items / wb_items_per_site schema migrations in parallel on the db66 slave (creating primary keys, cannot be done via online-schema-change)
21:24 RobH: kicked knsq23 to clear puppet freshness alarm
21:24 RobH: maerlant still offline (200 days?). dropped esams ticket for onsite troubleshooting, as mgmt is offline.
21:20 logmsgbot: asher synchronized wmf-config/InitialiseSettings.php 'setting wgReadOnly for wikidatawiki durring maintenance'
21:18 RobH: forced puppet run on lardner, was tired of it showing error for no runs
21:13 RobH: mw112-125 now in service as api apaches in tampa
20:58 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
20:38 RobH: mw86-mw111 tested ok, pushing into apache service in pmtpa
20:32 RobH: mw86-mw111 being added to nodelists, ignore errors on them for next few minutes
19:40 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks to 1.21wmf10
19:37 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity, wikisource and wikiquote to 1.21wmf10
19:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikivoyage, wikinews and wiktionary to 1.21wmf10
19:25 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: closed, special, wikimedia, private and fishbowl to 1.21wmf10
05:34 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 're-enable Scribunto profiler'
04:32 Tim: updating luasandbox to version 1.2, with profiler bug fixes
02:53 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Wed Feb 20 02:53:32 UTC 2013
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Wed Feb 20 02:29:11 UTC 2013
01:28 RobH: mw1161-1208 have wrong power setting, changing all to system.power.hotspare.enable 0
01:00 notpeter: do-release-upgrade-ing on db58
00:57 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'removing db58 from db-secondary for upgrades and maria, replacing db43'

February 19

23:47 notpeter: do-release-upgrade-ing on db43
23:38 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'removing db43 from db-secondary for upgrades and maria'
23:21 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/49946/'
23:04 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
23:03 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend/
22:49 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
22:41 logmsgbot: preilly synchronized php-1.21wmf9/extensions/ArticleFeedback 'fix MAX call with GREATEST'
22:29 RobH: sq41 offline, rt 4550
22:20 RobH: sq41 puppet freshness bad, wont accept ssh connections, serial redirection results in no output. will need to reboot it
22:17 RobH: sq41 not allowing ssh, drac resetting
22:15 sbernardin: taking down ms-be12 for hardware replacement per: https://rt.wikimedia.org/Ticket/Display.html?id=4546
22:15 RobH: sq79/80 puppet in locked state, manually restarted puppet, removed lock file, ran manually
22:11 logmsgbot: preilly synchronized php-1.21wmf9/extensions/ArticleFeedback 'fix mysql 5.5 / ansi sql incompatibility'
21:54 RobH: mw48 was set to puppet agent disabled, re-enabled and fired puppet run
20:04 logmsgbot: mflaschen synchronized . 'Syncing GuidedTour submodule omitted when new branch cut was first cloned.'
19:59 Ryan_Lane: disabling project storage volumes for all labs projects with no instances
15:02 hashar: restarting jenkins to enable some startup setting
14:58 hashar: manganese CPU spike ongoing. Started roughly at 14:40 UTC
13:29 mutante: wikimedia.jp.net now redirects to wikimedia.org | wikimediacommons.jp.net now redirects to commons.wm
13:27 mutante: DNS update - adding wikimedia.jp.net
13:25 logmsgbot: dzahn gracefulled all apaches
13:20 mutante: gracefulling eqiad Apaches via dsh to push redirect change for .jp.net domains
12:39 apergos: powercycling ms-be7 (unresponsive to ping or from console)
11:29 mutante: installing package upgrads on neon/icinga
02:49 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Tue Feb 19 02:49:32 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Tue Feb 19 02:28:25 UTC 2013
01:12 logmsgbot: anomie synchronized php-1.21wmf10/includes/api/ApiQuerySiteinfo.php 'Backport fix for ugly API bug'
00:42 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
00:30 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php

February 18

23:51 logmsgbot: tstarling synchronized php-1.21wmf9/extensions/Scribunto
23:51 logmsgbot: tstarling synchronized php-1.21wmf10/extensions/Scribunto
23:46 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
23:25 Tim: on snapshot1002 -- fixed incorrect owner on some MW files, killed broken apt-get and upgraded wikimedia-task-appserver
23:24 Tim: fixed rsyncd.conf on scap proxies to allow connections from new apaches
23:14 logmsgbot: tstarling Finished syncing Wikimedia installation... :
23:11 logmsgbot: tstarling Started syncing Wikimedia installation... :
22:45 logmsgbot: tstarling Finished syncing Wikimedia installation... :
22:42 logmsgbot: tstarling Started syncing Wikimedia installation... :
22:37 logmsgbot: tstarling synchronized php-1.21wmf9/extensions/Scribunto
22:37 logmsgbot: tstarling synchronized php-1.21wmf10/extensions/Scribunto
22:20 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
20:38 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikidatawiki to 1.21wmf10
20:37 logmsgbot: reedy synchronized php-1.21wmf10/cache/l10n/ 'Scap is a liar'
20:31 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild l10n cache for wikidata related extensions
20:15 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild l10n cache for wikidata related extensions
19:59 logmsgbot: reedy synchronized php-1.21wmf10/extensions/Wikibase
19:59 logmsgbot: reedy synchronized php-1.21wmf10/extensions/Diff
19:58 logmsgbot: reedy synchronized php-1.21wmf10/extensions/DataValues
19:58 logmsgbot: reedy synchronized php-1.21wmf10/extensions/WikimediaMessages
19:44 logmsgbot: reedy synchronized php-1.21wmf10/cache/l10n/
19:19 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.21wmf10
19:03 logmsgbot: reedy Finished syncing Wikimedia installation... : Build 1.21wmf10 message cache, and push it this time
18:45 logmsgbot: reedy Started syncing Wikimedia installation... : Build 1.21wmf10 message cache, and push it this time
17:54 logmsgbot: reedy Finished syncing Wikimedia installation... : Build 1.21wmf10 message cache
17:53 logmsgbot: reedy Started syncing Wikimedia installation... : Build 1.21wmf10 message cache
17:42 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.21wmf10
17:28 logmsgbot: reedy synchronized php-1.21wmf10 'Initial sync out'
17:03 logmsgbot: reedy synchronized wmf-config/
17:02 logmsgbot: reedy synchronized docroot
17:01 logmsgbot: reedy synchronized live-1.5/
12:33 mutante: wikiartpedia.(biz|co|info|me|mobi|net) activated - redirect to wikipedia.org (RT-4240)
12:30 mutante: adding wikiartpedia domains to DNS / authdns-update
12:28 logmsgbot: dzahn gracefulled all apaches
12:24 mutante: gracefulling eqiad Apaches via dsh for redirect.conf change
12:22 mutante: mw1045 - eth0 NO-CARRIER - check cable (RT-4545)
12:13 mutante: mw1041 - down - CPU Machine Chk error on mgmt - removing from dsh groups (RT-4517)
05:45 logmsgbot: tstarling synchronized wmf-config/db-eqiad.php
05:16 logmsgbot: tstarling synchronized wmf-config/db-eqiad.php
05:10 Tim: on db1043: running alter table to convert the relevant field to signed, since I figure this is less disruptive than sql_mode = 'NO_UNSIGNED_SUBTRACTION' and less likely to break again than skipping the query
04:58 Tim: replication broken on db1043, possibly others, due to update of article_feedback_revisions with negative value for afr_total
03:40 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Mon Feb 18 02:29:47 UTC 2013

February 17

02:29 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sun Feb 17 02:29:31 UTC 2013

February 16

13:38 apergos: restarted lucene-search on search1015 about 20 minutes ago
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sat Feb 16 02:28:26 UTC 2013
01:47 logmsgbot: reedy synchronized php-1.21wmf9/
01:11 logmsgbot: reedy synchronized wmf-config/filebackend.php

February 15

23:56 RobH: authdns-update to correct mw110 reverse entry
23:28 Ryan_Lane: *modified them in LDAP
23:28 Ryan_Lane: modified project objects to indicate whether or not they are using gluster volumes for project and/or home storage
23:04 RobH: disregard mw1161 flaps, im working on it
20:15 Ryan_Lane: restarting opendj and pdns on virt0 and virt1000
20:15 Ryan_Lane: removed opendj from nfs1/2
19:07 RobH: no one sign mw86-mw125 puppet certs, im still working on them
18:23 RobH: mw87-mw125 installing
17:33 binasher: reimaging db1029-1031
16:55 mutante: removing mw1085 from dsh groups - broken hardware (RT-4453)
16:00 hashar: Enabled a linter for operations/apache-config . Any apache typo in operations/apache-config will publicly write your name on the typo wall of shame. See 49251 for implementation details and https://integration.mediawiki.org/ci/job/operations-apache-config-lint/ for some build examples.
14:31 mutante: wikimediafoundation.info activated to redirect .org
14:28 mutante: gracefulling Apaches in eqiad using dsh
14:25 logmsgbot: dzahn gracefulled all apaches
14:15 ^demon: [gerrit] updated replication plugin to 20c6fc7e (1.1-SNAPSHOT) which is back in line with upstream
08:02 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/Echo 'Syncing Iab077a77aa75a6ff0cf03acecca38403595dda93 to fix exception 640c60c4'
02:30 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Fri Feb 15 02:30:05 UTC 2013
01:19 paravoid: deploying squid changes for s/knams/esams/ for knsq* squids (see git log for rationale)
01:09 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Enabling Schema NS on test2wiki'

February 14

22:49 RobH: mw1194 online and now in service as api apache
22:48 RobH: mw1200 added to api pool, online and in service
22:44 RobH: mw1189-1199 (sans 1194) now online as eqiad api apaches
22:37 logmsgbot: mflaschen Finished syncing Wikimedia installation... :
22:29 logmsgbot: mflaschen Started syncing Wikimedia installation... :
22:21 sbernardin: taking storage1 offline for decommissioning per: https://rt.wikimedia.org/Ticket/Display.html?id=4529
22:18 RobH: pushing mw1161-1188 into apache service (sans 1161/1164) from false to true in pybal, otherwise all online
22:16 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Updating for test2wiki'
22:14 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/Math
22:03 Ryan_Lane: replacing gluster upstarts on all labstore nodes with a regular init script
22:01 logmsgbot: reedy synchronized docroot
22:00 logmsgbot: olivneh synchronized wmf-config
21:49 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Enabling PostEdit on {be|mediawiki|outreach}wikis'
21:45 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/EventLogging
21:39 sbernardin: taking down srv266 and srv268 for decommissioning ... https://rt.wikimedia.org/Ticket/Display.html?id=4534
21:31 ottomata1: restarted solr jetty on solr2
21:30 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update ArticleFeedbackv5, Echo, PageTriage to master
21:25 ottomata: restarted solr jetty on solr3 and solr1003 for MaxSem
21:03 logmsgbot: bsitu Started syncing Wikimedia installation... : Update ArticleFeedbackv5, Echo, PageTriage to master
20:24 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'updating CommonSettings.php for Echo'
20:15 apergos: rebooting mw1010 after some issues with OOM
19:22 RobH: new servers are NOT in pybal yet, and thus not getting traffic
19:22 RobH: just added mw1161-1200 into node lists for eventual service, some may throw sync script errors as I track them down
19:15 Ryan_Lane: rebooting labstore3
19:08 Ryan_Lane: rebooting labstore4
19:01 Ryan_Lane: rebooting labstore2
18:52 Ryan_Lane: rebooting labstore1
15:22 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Updating wikibase sort order'
15:22 logmsgbot: demon synchronized php-1.21wmf9/extensions/Wikibase/client/includes/SortUtils.php 'Updating wikibase sort order'
14:52 mutante: re-enabled cp1044 in pybal - mobile varnish cache servers all on precise now
14:39 hashar: Jenkins: adding in several extension to Zuul configuration 49041
14:30 hashar: jenkins : updated all jobs using latest version jjb-config
13:35 mark: Fixed fixing
13:27 mutante: disabling cp1044 in pybal, reinstalling with precise
13:22 mutante: re-enabling cp1043 in pybal
13:11 mark: Upgraded varnish to latest version with updated range patch on cp1021-1028
12:16 mark: Inserted varnish 3.0.3plus~rc1-wm7 into the precise-wikimedia APT repository
12:14 mutante: stopping varnish backend on cp1043 and starting reinstall
12:10 mutante: disabling cp1043 in pybal
11:31 mutante: ms1004 - full disk, deleted large HTCPpurger.log file, fixed puppet runs
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Thu Feb 14 02:29:05 UTC 2013
01:17 andrewbogott: updated OpenStackManager to git head on virt0. Previous state is preserved (temporarily) in the local 'premerge' branch.
01:13 paravoid: removing mysql & apache from emery(!)
00:03 RobH: db29 os installed, handed off to peter

February 13

23:50 RobH: db29 reinstalling
23:48 paravoid: reinstalling sq48
23:22 RobH: db27 and db29 will be up and down for reinstallations
23:17 sbernardin: rebooting db29 per https://rt.wikimedia.org/Ticket/Display.html?id=4528
23:17 sbernardin: rebooting db27 per https://rt.wikimedia.org/Ticket/Display.html?id=4528
23:11 RobH: db1013 online and puppet updated, needs to be pushed into some db cluster service
22:45 RobH: mw27 locked up, powercycling
22:36 notpeter: powercycling all eqiad jobrunners
22:13 RobH: adding mw1201-1208 to eqiad apache pool
21:44 Reedy: Running sync-common on mw1203
21:43 logmsgbot: reedy synchronized php-1.21wmf9 'resync to fix file permission errors'
21:32 Reedy: Running sync-common on mw1204
21:30 Reedy: Running sync-common on mw1202
21:27 RobH: db27 rebooting for reinstallation
21:27 RobH: db29 offline due to hardware issue, related ticket 4526
21:19 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
21:11 logmsgbot: csteipp synchronized wmf-config
21:09 logmsgbot: reedy cleared profiling data
20:25 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Uncommenting enwiki from localClientDatabases'
19:59 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable Wikibase Client on enwiki'
19:47 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase/
19:30 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved remaning wikis to 1.21wmf9
19:21 logmsgbot: aaron synchronized php-1.21wmf9/maintenance/runJobs.php 'deployed b03384f0ebd95e7f79638fb14ccd55da9c186d97'
19:12 logmsgbot: py gracefulled all apaches
18:59 cmjohnson1: mw1041 is borked and will be powered down and removed from network for h/w checks
18:27 logmsgbot: aaron synchronized php-1.21wmf9/extensions/TranslationNotifications 'deployed d7bf60b57f472e5f75101f58505f61842848f961'
16:26 Nemo_bis: Job queue weirdness and TranslationNotifications duplicates again noticed on Meta (bug 42715)
13:33 logmsgbot: reedy synchronized php-1.21wmf9/resources/mediawiki
09:17 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/48804/'
09:10 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/48804/'
08:52 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/48797/'
08:25 hashar: jenkins: changing encryption key and regenerating secrets. See bug 44592
05:50 logmsgbot: tstarling synchronized multiversion/MWScript.php
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Wed Feb 13 02:52:39 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Wed Feb 13 02:28:06 UTC 2013
00:42 notpeter: restarting varnishncsa on cp1043
00:03 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend 'Touchhhhh'

February 12

23:54 logmsgbot: awjrichards synchronized php-1.21wmf9/extensions/MobileFrontend/javascripts/specials/donateimage.js 'touch file'
22:58 logmsgbot: maxsem Finished syncing Wikimedia installation... :
22:46 notpeter: disabling innodb-flush-log-at-trx-commit=2 on db1047 w/asher's blessing. will schedule downtime to apply
22:38 logmsgbot: maxsem Started syncing Wikimedia installation... :
22:04 notpeter: creating metrics user on all s1-7 masters
20:25 logmsgbot: reedy synchronized php-1.21wmf9/resources/mediawiki
20:22 binasher: flushed mobile varnish
20:21 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend
20:19 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend
20:06 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
20:05 binasher: flushed mobile varnish cache �:(
19:59 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend
19:57 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend
19:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable WikibaseClient on enwiki'
19:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable WikibaseClient on enwiki'
18:49 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
18:41 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
18:36 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
18:25 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
18:20 cmjohnson1: nas1001-a swapping mainboard
18:10 logmsgbot: aaron synchronized php-1.21wmf9/extensions/UploadWizard 'deployed 4548f9817d3d8025f4c19e279e9fb1f05b178087'
18:09 logmsgbot: catrope synchronized php-1.21wmf9/extensions/Wikibase 'Un-rollback Wikibase'
18:06 RoanKattouw: Reverted Wikibase update earlier, now reapplying
17:53 paravoid: powercycling niobium, SSH & console unresponsive
17:50 mark: Blocking modules=mobile.startup requests with 503 on bits varnish (earlier)
17:50 mark: Forced mw1149-1150 to healthy in bits.eqiad varnish
17:30 logmsgbot: catrope synchronized php-1.21wmf9/extensions/Wikibase 'Rolling back Wikibase'
16:18 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Tue Feb 12 16:18:31 UTC 2013
16:13 Jeff_Green: restarted apache on mw1152
16:01 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
16:00 Reedy: mw1161 and mw1165 are both missing mediawiki deployment files
15:58 logmsgbot: reedy synchronized php-1.21wmf9/includes/site/
15:54 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Tue Feb 12 15:54:11 UTC 2013
14:25 mutante: ran puppet on professor
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Tue Feb 12 02:52:42 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Tue Feb 12 02:28:53 UTC 2013
01:01 ^demon: stopped puppet and gerrit on manganese and formey
00:28 logmsgbot: aaron synchronized php-1.21wmf9/includes/db/Database.php 'deployed d8705542627f006a7ec9f81a9fb488fcc9a367bd'

February 11

23:19 RobH: unpooled mw1161-1188 from pybal config
22:03 logmsgbot: demon synchronized php-1.21wmf9/extensions/VisualEditor/modules/ve/init/mw/ve.init.mw.Platform.js 'Touching file'
21:08 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Wikibase client throwing fatals, disabling on enwiki pending fix'
21:04 cmjohnson1: auth-dns update
21:03 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'wikidata client on enwiki'
21:03 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'wikidata client on enwiki'
20:30 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: huwiki and itwiki to 1.21wmf9
20:27 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: hewiki and itwiki to 1.21wmf9
20:22 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf9
20:11 logmsgbot: demon synchronized php-1.21wmf9/includes/site 'Updating sites code to pull in I8964a03c, I42221832'
18:39 RobH: disregard, pushing one to test, mw1161
18:36 RobH: mw1161-mw1188 being pushed into pybal pool as general apaches
16:58 logmsgbot: demon synchronized php-1.21wmf9/extensions/Wikibase/lib/includes/modules/SitesModule.php 'Touching file'
16:50 logmsgbot: demon synchronized php-1.21wmf9/extensions/TorBlock 'TorBlock to master, now with less annoying logs'
16:41 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Enable logging group for torblock'
16:40 logmsgbot: demon synchronized php-1.21wmf9/extensions/TorBlock 'Updating TorBlock to master'
16:27 cmjohnson1: commenting out srv226-234 on pybal/pmtpa
14:59 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'bug 44251 hardcode $wgDBuser = "wikiuser" 47074'
13:55 hashar: jenkins: regenerating all phpcs-HEAD jobs so they stop failing when no files need to be processed bug 44567 48450
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Mon Feb 11 02:52:39 UTC 2013
02:27 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Mon Feb 11 02:27:44 UTC 2013

February 10

22:16 Tim: fixed syslog.eqiad.wmnet DNS entry, restarting rsyslogd on the eqiad apaches (RT 4500)
19:20 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c/48015/'
02:53 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Sun Feb 10 02:53:56 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sun Feb 10 02:28:46 UTC 2013

February 9

12:52 Reedy: mw1161 is missing /usr/local/apache/common-local
12:51 logmsgbot: reedy synchronized php-1.21wmf8/resources/
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Sat Feb 9 02:52:45 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sat Feb 9 02:28:28 UTC 2013

February 8

23:48 logmsgbot: anomie synchronized php-1.21wmf8/extensions/Scribunto/engines/LuaCommon/LuaCommon.php
23:48 logmsgbot: anomie synchronized php-1.21wmf9/extensions/Scribunto/engines/LuaCommon/LuaCommon.php
23:09 RoanKattouw: and on cerium too
23:08 RoanKattouw: Running puppet on titanium
20:20 RobH: disregard errors for mw1161-1164
20:13 RobH: mw1161-1164 offline for reallocation from job to general apache
20:10 cmjohnson1: decommissioning srvs226-234
19:43 RobH: mw1165-1188 os installed for apache use, initial puppet runs in progress
19:33 RobH: celsus back online
19:23 RobH: celsus rebooting to fix power mgmt cpu spike issue
19:02 RobH: mw1161-1164 in service as eqiad jobrunners
16:32 ottomata: updated vumi packages in apt to vumi_0.5.0~a+143-0~ppa3 and vumi-wikipedia_0.1~a+14-0~ppa3
16:09 mark: Upgraded and rebooted cr2-eqiad (both REs)
15:11 mark: Set cr2-eqiad VRRP prio to 50 to transition to backup role
15:10 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
15:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: srwikibooks back to 1.21wmf9 to test variant fixes
14:39 hashar: gallium installed php-luasandbox manually. Gerrit change is 48127.
05:47 logmsgbot: tstarling synchronized php-1.21wmf9/includes/context/RequestContext.php
02:54 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Fri Feb 8 02:54:49 UTC 2013
02:30 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Fri Feb 8 02:30:42 UTC 2013
02:02 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
01:48 LeslieCarr: cr2-knams is back up on the new code
01:36 LeslieCarr: cr2-knams reloading after upgrade
01:34 LeslieCarr: deactivating european bgp sessions - may cause some connectivity issues as connections reset
01:24 LeslieCarr: disabling notifications on icinga and nagios
01:03 logmsgbot: reedy synchronized php-1.21wmf9/languages/classes/
00:25 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/GettingStarted

February 7

23:31 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Enabling GuidedTour on additional wikis'
23:25 LeslieCarr: manually cleared arp and turned managmeent interface off and on of asw-c-eqiad to solve reachability issue
23:02 logmsgbot: mflaschen Finished syncing Wikimedia installation... : Deploy GuidedTour
22:47 logmsgbot: mflaschen Started syncing Wikimedia installation... : Deploy GuidedTour
22:39 RobH: mw82 pooled into tampa apaches
22:04 ottomata: pulled power and rebooted amaranth toolserver for multichill. Its back online. multichill will notify whomever needs to do more work.
21:58 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/RelatedArticles/ 'Fixes 44761'
21:58 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/RelatedArticles/ 'Fixes 44761'
21:50 RobH: mw81-mw85 in pmtpa apache service
21:39 cmjohnson1: performing a hard reboot on amaranth
21:34 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/E3Experiments
21:33 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/E3Experiments
21:12 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/PostEdit
21:11 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/PostEdit
21:10 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/EventLogging
21:09 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/EventLogging
20:29 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: Being paranoid, node lists were in flux during last sync
20:23 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: All ku/sr wikis back to 1.21wmf8 - lang converter breakage
20:20 Nemo_bis: !Wikipedia and !Wikimedia projects are/were down, issue being investigated
20:14 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: kuwiktionary back to 1.21wmf8
20:07 RobH: removing srv219-224 from rendering config in pmtpa
20:06 mark: Rebooting cr1-sdtpa
19:56 RobH: adding mw75-mw80 into tampa rendering pool
18:42 mark: Rebooted re0.cr1-eqiad
18:05 mark: Rebooting re1.cr1-eqiad
16:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
15:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
15:44 mark: Rebooting cr1-esams
15:40 logmsgbot: reedy synchronized php-1.21wmf8/cache/interwiki.cdb 'Updating 1.21wmf8 interwiki cache'
15:39 logmsgbot: reedy synchronized php-1.21wmf9/cache/interwiki.cdb 'Updating 1.21wmf9 interwiki cache'
14:54 mutante: granting RT group 'toolserver admins' the 'modify ticket' permission in queue esams
14:38 mutante: planet now updating hourly instead of daily
14:13 mutante: planet speaks SSL now for all languages. (it does not force http->https but it can if there is a demand)
12:00 Nemo_bis: amaranth dead/unreachable from Europe, ping dies at xe-5-3-1.cr2-eqiad.wikimedia.org (173.241.131.218)
10:08 hashar: jenkins : regenerating jobs to make phpcs to show the sniff codes being used. 47847
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Thu Feb 7 02:52:26 UTC 2013
02:27 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Thu Feb 7 02:27:52 UTC 2013
02:23 logmsgbot: spage synchronized php-1.21wmf9/extensions/GuidedTour/modules/externals/mediawiki.libs.guiders/mediawiki.libs.guiders.submodule 'sync GuidedTour submodule omitted from 1.21wmf9'
01:57 RoanKattouw: Force-running puppet on wtp1 to figure out why it's broken
01:53 logmsgbot: krinkle synchronized wmf-config/InitialiseSettings.php 'I9fd98309'
00:41 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
00:35 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
00:20 logmsgbot: reedy synchronized php-1.21wmf9/extensions/WikimediaMaintenance
00:00 logmsgbot: reedy synchronized langlist

February 6

23:59 RobH: authdns-update to add min.wikipedia.org via addwiki automation
23:56 logmsgbot: reedy synchronized php-1.21wmf8/cache/interwiki.cdb 'Updating 1.21wmf8 interwiki cache'
23:55 logmsgbot: reedy synchronized php-1.21wmf9/cache/interwiki.cdb 'Updating 1.21wmf9 interwiki cache'
23:50 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
23:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:43 logmsgbot: reedy synchronized langlist
23:38 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
23:37 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:28 RobH: srv190-srv223 & srv225 should be offline, they are being decommissioned, can ignore nagios errors
23:26 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
23:24 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:18 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
23:17 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:13 RobH: specifically (i can spell, maybe)
23:13 RobH: removed from pybal specifially
23:12 cmjohnson1: removing srv192-218 and srv225 per rt 4438
23:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
23:05 logmsgbot: reedy synchronized php-1.21wmf9/extensions/WikimediaMaintenance/
21:36 cmjohnson1: srv193 is back online
21:21 cmjohnson1: srv193 power down to relocate to different rack
19:47 RobH: authdns-update
19:33 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary and wikibooks to 1.21wmf9
19:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource and wikiquote to 1.21wmf9
19:18 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews, wikivoyage and wikiversity to 1.21wmf9
19:12 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special and wikimedia wikis to 1.21mwf9
19:11 cmjohnson1: powerdowing mw1 to reseat DIMM B1
19:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: closed, private and fishbowl wikis to 1.21wmf9
18:43 ^demon: running puppet on manganese, restarting gerrit
18:41 RobH: authdns-update
17:48 LeslieCarr: restarted morebots
02:51 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Wed Feb 6 02:51:36 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Wed Feb 6 02:28:05 UTC 2013
02:05 logmsgbot: reedy synchronized wmf-config/
00:42 RobH: mailing lists returned to normal
00:41 RobH: done with data removal on mailing list server, rebuilding mbox
00:37 RobH: stopping mailman to scrub an entry out of archives, will restart shortly

February 5

23:29 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend/javascripts/modules/mf-photo.js 'https://gerrit.wikimedia.org/r/#/c/47670/'
23:26 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend/javascripts/modules/mf-photo.js 'https://gerrit.wikimedia.org/r/#/c/47670/'
22:44 logmsgbot: maxsem Finished syncing Wikimedia installation... : https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2013-02-05
22:28 logmsgbot: maxsem Started syncing Wikimedia installation... : https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2013-02-05
22:08 RobH: analytics1007 coming down for troubleshooting, disregard nagios errors
22:01 RobH: analytics1001 reinstalled, not puppet signed, back online
22:00 RobH: analytics1001 respects my authority.
21:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 44688 - Please disable MoodBar extension on English Wikipedia'
21:28 logmsgbot: robh synchronized wmf-config/InitialiseSettings.php
21:12 RobH: analytics1001 going down for troubleshooting/reinstallation
20:21 logmsgbot: kaldari Finished syncing Wikimedia installation... :
20:12 RobH: blog update sucessful, caching plugin update nonsuccessful, rolling it back to older plugin
20:07 logmsgbot: kaldari Started syncing Wikimedia installation... :
20:04 RobH: updating the blog software, here goes nothing
19:17 hashar: gallium : Zuul updated!
19:13 hashar: gallium : updating Zuul by cherry-picking a set of changes that let us fetch tags. ff79197..1854e32
17:05 cmjohnson1: authdns update
16:37 LeslieCarr: copied all equinix ashburn IPv4 peering sessions to their new /22 on cr2-eqiad
16:37 Jeff_Green: dist-upgrades for payments systems
16:36 Jeff_Green: restarted morebots :-P
05:37 Ryan_Lane: restarting opendj on all hosts
05:36 Ryan_Lane: remove virt1 replication agreement from all ldap servers
02:51 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Tue Feb 5 02:51:10 UTC 2013
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Tue Feb 5 02:29:02 UTC 2013
01:07 logmsgbot: reedy synchronized php-1.21wmf9/extensions/TimedMediaHandler
00:51 logmsgbot: reedy synchronized php-1.21wmf8/extensions/ZeroRatedMobileAccess
00:39 logmsgbot: reedy synchronized php-1.21wmf8/extensions/ZeroRatedMobileAccess
00:29 logmsgbot: reedy synchronized php-1.21wmf9/extensions/ZeroRatedMobileAccess

February 4

23:49 logmsgbot: reedy synchronized multiversion/
23:46 logmsgbot: reedy synchronized multiversion/
23:45 logmsgbot: reedy synchronized wmf-config/
23:17 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
21:02 hashar: gallium moved a git extension : mv /var/lib/git/rt-authen-externalauth.git /var/lib/git/operations/debs/
19:44 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
19:39 ottomata: manually gracefully reloaded eqiad apaches. See RT 4449
19:37 logmsgbot: root gracefulled all apaches
19:20 logmsgbot: reedy synchronized wmf-config/
19:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki, mediawikiwiki and wikidatawiki to 1.21wmf9
18:35 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
17:56 logmsgbot: reedy Finished syncing Wikimedia installation... : Build messages for 1.21wmf9
17:51 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
17:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
17:24 logmsgbot: reedy Started syncing Wikimedia installation... : Build messages for 1.21wmf9
17:14 logmsgbot: reedy synchronized php-1.21wmf9/extensions/GuidedTour/
17:09 logmsgbot: reedy Started syncing Wikimedia installation... : Build messages for 1.21wmf9
17:06 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
17:06 logmsgbot: reedy Started syncing Wikimedia installation... : Build messages for 1.21wmf9
16:45 logmsgbot: reedy synchronized php-1.21wmf9/ 'Initial sync'
16:22 logmsgbot: reedy synchronized wmf-config/
16:06 Ryan_Lane: force starting any glusterfs volume that has a downed process
15:12 Ryan_Lane: rebooting labstore1
15:11 Ryan_Lane: upgrading labstore1 to precise
14:33 apergos: powercycled ms-be3
14:32 Ryan_Lane: make that rebooting labstore3
14:32 Ryan_Lane: rebooting labstore2
14:07 Ryan_Lane: upgrading labstore3 to precise
11:14 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'cbd432f..82ed797 47393 Vector micro design on mlwiki, 29792 wikipedia translation on pawiki'
07:37 Ryan_Lane: upgrading labstore2 to precise
02:25 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Mon Feb 4 02:25:58 UTC 2013
00:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
00:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php

February 3

20:58 hashar: jenkins : generating mwext-**-phpcs-HEAD jobs
17:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
17:42 logmsgbot: reedy synchronized wmf-config/
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Sun Feb 3 02:28:29 UTC 2013
02:03 logmsgbot: reedy synchronized wmf-config/
02:02 logmsgbot: reedy synchronized images/sul/foundation.png
01:14 logmsgbot: reedy synchronized wmf-config/
00:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
00:22 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php

February 2

23:59 logmsgbot: reedy synchronized wmf-config/
23:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:34 logmsgbot: reedy synchronized wmf-config/
23:30 logmsgbot: reedy synchronized php-1.21wmf8/extensions/SearchExtraNS
23:03 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:35 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
19:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:54 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
16:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
16:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
09:23 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/PageImages/initImageData.php 'https://gerrit.wikimedia.org/r/#/c/44556/'
08:45 MaxSem: Running foreachwikiindblist wikipedia.dblist extensions/PageImages/initImageData.php --earlier-than=20121229000000
03:26 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Sat Feb 2 02:28:18 UTC 2013

February 1

23:02 K4-713: synchronized payments cluster to b421871b43822
20:00 RobH: authdns-update
19:43 RobH: authdns-update
14:45 cmjohnson1: authdns update "adding mw1161-1200 to eqiad mgmt and production zone files
12:04 cmjohnson1: authdns update adding db1051-60 to zone files
02:27 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Fri Feb 1 02:27:09 UTC 2013
01:50 K4-713: updated payments cluster to 45f8a6c6009a4bd7

January 31

23:14 andrewbogott: rebooting virt0 in a fit of optimism and/or desperation
22:44 logmsgbot: spage Finished syncing Wikimedia installation... : take 2: Deploy E3Experiments, GettingStarted, GuidedTour (new), MoodBar, and WikiEditor
22:37 logmsgbot: spage Started syncing Wikimedia installation... : take 2: Deploy E3Experiments, GettingStarted, GuidedTour (new), MoodBar, and WikiEditor
22:26 logmsgbot: mflaschen Started syncing Wikimedia installation... : Deploy E3Experiments, GettingStarted, GuidedTour (new), MoodBar, and WikiEditor
22:20 ottomata: deployed tab delimiter log format change on squids, varnishes and nginxes
21:40 ottomata: reverted tab separartor change to squid
21:37 ottomata: deloyed tab separator log format changes to squids, merged the corresponding puppet changes to varnish and nginx
20:47 paravoid: deploying patched ganglia and removing .htaccess. Ganglia is public again!
14:59 logmsgbot: reedy synchronized php-1.21wmf8/extensions/Wikibase
14:58 notpeter: taking down db1047 for upgrade to precise
13:21 hashar: gallium: installed mercurial manually (puppet change is 46931 PS2)
13:19 hashar: gallium manually removed bzr: apt-get remove bzr python-keyring python-httplib2 python-launchpadlib python-zope.interface python-oauth python-bzrlib bzr python-simplejson python-configobj python-lazr.uri python-lazr.restfulclient python-wadllib
13:19 hashar: gallium: manually installed "bzr"
12:12 logmsgbot: reedy synchronized php-1.21wmf8/extensions/Wikibase
10:24 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Remove strategyappswiki
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Thu Jan 31 02:29:47 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Thu Jan 31 02:28:18 UTC 2013

January 30

22:40 binasher: deployed new text/upload squid confs with eqiad in proxySubnets
22:38 logmsgbot: reedy synchronized wmf-config/
22:23 logmsgbot: tstarling cleared profiling data
22:12 logmsgbot: aaron synchronized php-1.21wmf8/includes/filerepo/file/LocalFile.php 'deployed f884911f5562fc41980664c424e7037bde6ba110'
22:11 logmsgbot: aaron synchronized php-1.21wmf8/includes/filerepo/file/OldLocalFile.php 'deployed f884911f5562fc41980664c424e7037bde6ba110'
21:59 LeslieCarr: killing varnishhtcpd on cp3009 for realzies this time
21:58 RobH: mw23 offline, reinstalled with precise and signed into puppet, puppet throwing errors on initial run.
21:52 binasher: started udplog with correct mw config on fluorine
21:49 LeslieCarr: restarting varnishhtcpd on cp3010 - it appears to be locked up , netstat -l shows a huge Recv-Q for port 4827 and appears to be the previously discovered issue
21:43 binasher: restarting udp2log on fluorine
20:59 logmsgbot: reedy cleared profiling data
20:50 RobH: authdns-update
20:21 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
20:20 RobH: mw23 was stuck in installer, restarting and setting up.
20:09 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
19:45 logmsgbot: reedy synchronized wmf-config/ 'Enabling wikidata client on itwiki and hewiki'
19:42 ottomata: added updated udplog_1.8-5 .debs to apt.wikimedia.org. No real changes, these now include the packet-loss program in the package.
19:22 logmsgbot: reedy synchronized wmf-config/
19:12 logmsgbot: reedy synchronized php-1.21wmf8/extensions/Wikibase/
19:06 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.21wmf8
18:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:48 Ryan_Lane: downgrading memcached to ubuntu version on virt0
17:18 Reedy: Created wb_changes_dispatch on wikidatawiki
16:55 Reedy: Created sites/site_identifier tables on hewiki and itwiki
16:34 cmjohnson1: mw1072 going to down to check h/w state currently failed rt4381
15:38 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 44395) Allow bureaucrats to remove the translateadmin group on wikidatawiki 46011'
15:37 logmsgbot: hashar synchronized wmf-config/throttle.php '(bug 44506) raise throttle for an Israel editthon 46747'
12:54 logmsgbot: maxsem synchronized php-1.21wmf7/extensions/GeoData
12:52 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/GeoData
05:55 Tim: rebooted kaulen, was not responding on network or serial console
03:11 mutante: wikimania.asia redirect now works, it did not before because of a bug in apache-graceful-all not actually restarting Apaches in eqiad
03:07 mutante: reloading Apaches in eqiad using manual dsh (because apache-graceful-all sanity check has a bug)
02:55 logmsgbot: catrope gracefulled all apaches
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Wed Jan 30 02:52:03 UTC 2013
02:39 logmsgbot: dzahn gracefulled all apaches
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Wed Jan 30 02:28:22 UTC 2013
00:44 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
00:43 mutante: sync InitialiseSettings.php for bug 44473
00:19 binasher: restarted pdns-recursor on dobson
00:13 logmsgbot: reedy synchronized wmf-config/

January 29

23:54 logmsgbot: reedy synchronized wmf-config/
23:21 logmsgbot: pgehres synchronized php-1.21wmf8/extensions/CentralNotice/ 'Updating CentralNotice to master, resolving namespace issues'
23:17 logmsgbot: pgehres synchronized php-1.21wmf7/extensions/CentralNotice/ 'Updating CentralNotice to master, resolving namespace issues'
23:00 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'Enable Special:Nearby everywhere'
22:51 logmsgbot: maxsem synchronized php-1.21wmf7/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/46630'
22:48 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/46630'
22:47 Tim: rotated SAL wikitech page - test logging

2000s

Archive 1: 2004 Jun - 2004 Sep
Archive 2: 2004 Oct - 2004 Nov
Archive 3: 2004 Dec - 2005 Mar
Archive 4: 2005 Apr - 2005 Jul
Archive 5: 2005 Aug - 2005 Oct, with revision history 2004-06-23 to 2005-11-25
Archive 6: 2005 Nov - 2006 Feb
Archive 7: 2006 Mar - 2006 Jun
Archive 8: 2006 Jul - 2006 Sep
Archive 9: 2006 Oct - 2007 Jan, with revision history 2005-11-25 to 2007-02-21
Archive 10: 2007 Feb - 2007 Jun
Archive 11: 2007 Jul - 2007 Dec
Archive 12: 2008 Jan - 2008 Jul
Archive 12a: 2008 Aug
Archive 12b: 2008 Sept
Archive 13: 2008 Oct - 2009 Jun
Archive 14: 2009 Jun - 2009 Dec

2010s

2020s

@@ Line 1: / Line 1: @@
 == April 11 ==
+* 13:14 logmsgbot: reedy synchronized wmf-config/  'Disable ClickTracking :D'
 * 13:08 mark: Inserted python-extras packages into the precise-wikimedia APT repository (backport by hashar)
 * 10:10 hashar: Jenkins is back up