Server Admin Log: Difference between revisions

Browse history interactively

← Older edit Newer edit →

Content deleted Content added

VisualWikitext

Inline

Revision as of 00:46, 17 May 2014

May 17

00:23 ori: varnishadm on cp1056 confirms that varnish recognizes mw1151 as "sick"
00:20 ori: stopping apache and disabling puppet on mw1151 so that varnish stops forwarding reqs to it
00:16 Krinkle: On mw1151, Gadget::loadStructuredList() returns false, memcached has no value for 'enwiki:gadgets-definition:7' and is unable to store it.
00:11 ori: Krinkle identified weird RL responses as all originating in mw1151; dmesg shows ata1 disk troubles: "failed command: READ DMA EXT", "sd 0:0:0:0: [sda] Add. Sense: Unrecovered read error - auto reallocate failed"
00:04 logmsgbot: krinkle synchronized php-1.24wmf4/includes/resourceloader/ResourceLoader.php 'I718fcf23d'

May 16

23:07 awight: updated crm from 0b8aa8aa046935b6cfc67c10ebe10396d5e42745 to 65a433b5564f42c3aa4f310cd4bb938ae70f841d
22:56 logmsgbot: aaron synchronized php-1.24wmf4/includes/db/Database.php '182e42c173b9ab0c2bc5d753879a000b1ff39e77'
22:54 logmsgbot: aaron synchronized php-1.24wmf5/includes/db/Database.php '8829ffc72d3332d348a1a2e58d525e54e126bad5'
22:17 awight: tools updated from 85bb7293d83517086e3609f03365aecde9f58c71 to ee31fc94b17c11a48ddac19aabfcdaab69fd2f72
21:37 logmsgbot: ori synchronized php-1.24wmf4/extensions/MultimediaViewer 'Update MultimediaViewer for I0df067a61: Add sampling to unsampled event logging'
21:33 logmsgbot: ori synchronized php-1.24wmf5/extensions/MultimediaViewer 'Update MultimediaViewer for I0df067a61: Add sampling to unsampled event logging'
21:10 mwalker: updating fundraising smashpig from 2fdf982b20f1cbeaf9f57af64ef21b5b69a36f6e to f64f79f13cf4ab560d0bb5bd69690c827a821629
20:41 awight: update crm from 243641de631b712c4a29ca1f3618771b78dadeae to 0b8aa8aa046935b6cfc67c10ebe10396d5e42745
18:43 awight: update tools from a40c0caa18a0efd93bc5d3f7f68386fbc36bf1fa to 85bb7293d83517086e3609f03365aecde9f58c71
18:12 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I3c453b0949f4e: Tweak MediaViewer sampling settings'
18:07 logmsgbot: ori synchronized wmf-config 'Ia43821231: Add sampling control setting for MediaViewer event'
18:04 logmsgbot: ori updated /a/common to Ia43821231: Add sampling control setting for MediaViewer event logging
17:47 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'GeoData to Elastic for all wikivoyages'
17:11 mwalker: updating staging payments servers as well from 5e24b953dcff5305099e152139e6e93daba8aeec to d99177518b741e7fe18ffda86c83f93c72e164a6
17:10 mwalker: and updated to 1.22.6
17:10 mwalker: moved fundraising wiki from 6a1d4983319038edeb88dc34a1c220ecaec1cbde to d99177518b741e7fe18ffda86c83f93c72e164a6 -- including json i18n changes
16:55 manybubbles: "in place" reindexing (for cirrus) all the wikipedias after the deploy train hit them yesterday
16:53 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Removing old WikiEditor settings'
16:44 RobH: partial zirconium downtime
16:44 RobH: i logged into zirconium, but it had recovered by the time I checked it.
16:33 mwalker: updated fundraising civicrm from 7a23465e620211739421cce3ad57c62597eb8cc3 to 75c1a50b8aa7e7b6f218d7c420932a8fc53a0a34 for an exchange rates fix
16:26 qchris: updated gerrit's hooks-bugzilla plugin to version 2.8.1.2 to allow talking to bugzilla-4.4.4
13:54 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'repool db1056 in s4, warm up'
13:17 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'raise db1070 and db1071 to normal load'
10:14 springle: xtrabackup clone db1049 to db1056
10:13 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'reduce db1049 load while cloning'
09:40 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'pool db1070 and db1071 in s1, warm up'
06:20 logmsgbot: ori synchronized php-1.24wmf4/maintenance/compareParserCache.php 'Ica69a3ef2: Added a script to compare current parser output to cache (no impact on prod; syncing for consistency)'
03:55 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 16 03:54:03 UTC 2014 (duration 54m 2s)
03:09 logmsgbot: LocalisationUpdate completed (1.24wmf5) at 2014-05-16 03:08:04+00:00
02:39 logmsgbot: LocalisationUpdate completed (1.24wmf4) at 2014-05-16 02:38:41+00:00
02:07 springle: xtrabackup db1070 to db1071
01:02 logmsgbot: ori synchronized php-1.24wmf5/includes/parser 'I12a60b5cc: Revert "Declare visibility on class properties of includes/parser/"'
00:54 hoo: rebuildItemsPerSite finished running for Wikidata (after about 30h).
00:32 hoo: manually ran rebuildEntityPerPage for Wikidata to fix 2 broken records
00:07 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/131762/'
00:00 logmsgbot: ori synchronized wmf-config/squid.php 'Id188979c1: Use whole subnets in squid.php list for XFF acceptance'

May 15

23:57 logmsgbot: ori updated /a/common to Id188979c1: Use whole subnets in squid.php list for XFF acceptance
23:38 logmsgbot: ori synchronized php-1.24wmf4/includes 'Ia3b12fb9: Speed up CIDR matching from $wgSquidServersNoPurge'
23:19 logmsgbot: ori synchronized php-1.24wmf5/includes 'Ia3b12fb9: Speed up CIDR matching from $wgSquidServersNoPurge'
23:05 logmsgbot: ori synchronized wmf-config/CirrusSearch-production.php 'Iae07852b1: Elasticsearch plugin juggling'
22:54 logmsgbot: ori synchronized wmf-config 'I51a55c4e2, Ia6c01a913, I594848ce0, and I594848ce0'
22:50 logmsgbot: ori updated /a/common to Ifae836de5: Swapping GeoData backend for enwikivoyage
22:28 logmsgbot: ori synchronized php-1.24wmf4/extensions/EventLogging 'Update EventLogging to master for I89819bd943'
22:26 logmsgbot: ori synchronized php-1.24wmf5/extensions/EventLogging 'Update EventLogging to master for I89819bd943'
22:06 awight: updated tools from 93fda5da99674eca221e0abf53ad499583b27cfb to a40c0caa18a0efd93bc5d3f7f68386fbc36bf1fa
22:04 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'GeoData using elasticsearch on enwikivoyage'
21:04 awight: updated all tools from 47407c16d9922b17af70146416913abfe50b728d to 93fda5da99674eca221e0abf53ad499583b27cfb
20:16 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'I5da4aa5db7b5d3c1843a6fd68d0a7c62a2bbfb4e'
19:56 mwalker: updated fundraising tools repo for screenshots, worldpay auditing, live analysis, and... stomp! from 0eb485c8b6db5f06805976860bce7aa8b0d6444b to 47407c16d9922b17af70146416913abfe50b728d
19:09 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf5
18:55 ori: deploying twemproxy module on mw106*, they may complain for a moment
18:52 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf4
18:46 logmsgbot: reedy Finished scap: testwiki to 1.24wmf5 and build l10n cache (duration: 27m 47s)
18:32 mutante: mw1053 was already disabled in pybal though and RT 7408,7435
18:31 mutante: mw1053 sits at disk partitioning dialog (via mgmt)
18:29 Reedy: mw1053 is pingable but not ssh-able
18:18 logmsgbot: reedy Started scap: testwiki to 1.24wmf5 and build l10n cache
17:53 Jeff_Green: adjusted exim conf on mchenry to route donate.wm.o mail to barium instead of aluminium
16:43 mwalker: disabled qc and put site_offline and maintenance_mode on civicrm to true
15:20 logmsgbot: anomie synchronized php-1.24wmf4/extensions/MultimediaViewer 'SWAT: Deploy change 133446 to fix bug 65225 in MultimediaViewer'
14:03 springle: xtrabackup clone db1056 to db1070
13:59 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1056 while cloning'
13:44 cmjohnson1: sodium going down again for a different disk replacement
13:16 cmjohnson1: shutting down sodium to replace sdb
12:56 godog: restarting gerrit on ytterbium, clones over https seemingly stuck
12:24 manybubbles|away: "in place" reindexing group1 wikis after the deployment train updated cirrus yesterday. They'll need a full reindex after that is done which will take some time but is required to fix issues with redirects not showing up off of the main namespace
11:56 godog: installed openjdk-7-jdk on ytterbium to attempt gerrit thread dump
10:15 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1009 for raid tests'
06:44 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'move s5 api traffic to db1005'
05:19 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'move s4 commonswiki api traffic to db1042'
04:20 springle: installed db1073
03:15 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 15 03:14:04 UTC 2014 (duration 14m 3s)
02:27 logmsgbot: LocalisationUpdate completed (1.24wmf4) at 2014-05-15 02:26:09+00:00
02:15 logmsgbot: LocalisationUpdate completed (1.24wmf3) at 2014-05-15 02:14:31+00:00

May 14

23:42 logmsgbot: mwalker synchronized wmf-config/InitialiseSettings.php 'Poking settings to try and apply them'
23:29 logmsgbot: mwalker synchronized visualeditor.dblist 'Another part of 132409 (visual editor)'
23:27 K4-713: updated payments from 78cc4285bdeb6eecba3efc75e4a04c8b886561e4 to 5e24b953dcff5305099e152139e6e93daba8aeec
23:27 logmsgbot: mwalker synchronized wmf-config/ 'SWAT of 132409 (visual editor) and 130274 (abuse filter)'
22:04 logmsgbot: maxsem synchronized php-1.24wmf3/extensions/MobileFrontend/ 'bug 65042'
22:03 marktraceur: cscott deployed a jenkins job change that pushes parsoid git files to beta-labs for version purposes
22:03 logmsgbot: maxsem synchronized php-1.24wmf4/extensions/MobileFrontend/ 'bug 65042'
20:38 awight: updated crm from 3fd3b94834f94529841ad4a695ecd73c98e487bc to 7a23465e620211739421cce3ad57c62597eb8cc3
20:32 bd808: Restarting logstash on logstash1001.eqiad.wmnet due to missing messages from some (all?) logs
19:58 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'No more LQT on wikimania2011wiki'
18:32 Krinkle: integration-slave1001 had its 8GB / /dev/vda1 100% full. Purging /tmp/perf-*.map brought it back to 41%
18:25 Krinkle: integration-slave1001 is having issues writing to disk
17:50 logmsgbot: yurik synchronized php-1.24wmf4/extensions/ZeroRatedMobileAccess/
17:47 logmsgbot: yurik synchronized php-1.24wmf3/extensions/ZeroRatedMobileAccess/
17:30 logmsgbot: yurik synchronized wmf-config/CommonSettings.php
15:44 chasemp: disabling puppet on tungsten to try tweaking carbon settings to affect queue drops (for the better)
14:28 cmjohnson1: mw1053 going down for disk replacement
13:27 bblack: restarting pybals on lvs300x
12:30 _joe_: restarted uwsgi on tungsten
09:46 mark: Started PyBal on lvs300* and established BGP sessions with the routers
09:43 mark: Setup BGP configuration for lvs300* on cr1-esams and cr2-knams, with elevated MEDs to keep them as last resorts
04:18 Tim: deploying apache configuration change with fixes
03:11 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed May 14 03:10:36 UTC 2014 (duration 10m 35s)
03:01 Tim: reverting apache change
02:53 Tim: deploying apache configuration change https://gerrit.wikimedia.org/r/106109
02:26 logmsgbot: LocalisationUpdate completed (1.24wmf4) at 2014-05-14 02:25:08+00:00
02:21 springle: upgrade db1043, rebuild as m3 master
02:14 logmsgbot: LocalisationUpdate completed (1.24wmf3) at 2014-05-14 02:13:23+00:00
00:18 manybubbles: bouncing elasticsearch on elastic1015 to pick up gc logging configuration. it might warn but shouldn't cause any service disrubtion.

May 13

23:27 logmsgbot: maxsem synchronized php-1.24wmf4/includes/exception/MWException.php 'https://gerrit.wikimedia.org/r/#/c/133184/'
23:25 logmsgbot: maxsem synchronized php-1.24wmf3/includes/exception/MWException.php 'https://gerrit.wikimedia.org/r/#/c/133183/'
23:21 MaxSem: Ran namespaceDupes after adding new namespaces to zhwikisource - no problems found
23:18 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/127584'
22:43 chasemp: ms-be1009 rebooted as it had locked up, swift seems to have recoverd
22:35 mutante: created new gerrit projects for phabricator,arcanist and libphutil
22:22 ori: restarting tungsten to verify fix for gdash/graphite initialization
21:33 ori: gdash and graphite currently down; chase & ori debugging
21:29 manybubbles: I caused elasticsearch1015 to drop out of the Elasticsearch cluster by tring to take a heap dump on it. don't do that. It stops the application for many seconds.
19:07 logmsgbot: yurik synchronized php-1.24wmf3/extensions/ZeroRatedMobileAccess/
19:02 logmsgbot: yurik synchronized php-1.24wmf4/extensions/ZeroRatedMobileAccess/
18:49 cmjohnson1: replacing failed disk dataset1001
18:25 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'touch for I1681addaed690b652822c0296b7a3e9b84de93b6'
18:22 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.24wmf4
15:52 logmsgbot: anomie synchronized php-1.24wmf4/resources/Resources.php 'SWAT: Deploy jQuery Migrate to 1.24wmf4'
15:51 logmsgbot: anomie synchronized php-1.24wmf4/resources/lib/jquery/jquery.migrate.js 'SWAT: Deploy jQuery Migrate to 1.24wmf4'
04:09 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue May 13 04:07:53 UTC 2014 (duration 7m 52s)
03:01 logmsgbot: LocalisationUpdate completed (1.24wmf4) at 2014-05-13 03:00:07+00:00
02:31 logmsgbot: LocalisationUpdate completed (1.24wmf3) at 2014-05-13 02:30:33+00:00

May 12

23:28 logmsgbot: yurik synchronized php-1.24wmf3/extensions/ZeroRatedMobileAccess/
23:24 logmsgbot: yurik synchronized php-1.24wmf4/extensions/ZeroRatedMobileAccess/
20:11 gwicke: deployed Parsoid d1c778ea3
17:00 mutante: re-enabled mw1186 in pybal
16:58 logmsgbot: manybubbles Finished scap: scapping again to get ms1186 synced up (duration: 00m 56s)
16:57 logmsgbot: manybubbles Started scap: scapping again to get ms1186 synced up
16:49 mutante: disabled mw1186 in pybal
16:46 mutante: mw1186 - down, powercycling
16:39 logmsgbot: manybubbles Finished scap: fix php symlink (duration: 04m 25s)
16:35 logmsgbot: manybubbles Started scap: fix php symlink
16:09 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php
16:07 logmsgbot: manybubbles Finished scap: update visual editor for swat deploy (duration: 28m 58s)
15:38 logmsgbot: manybubbles Started scap: update visual editor for swat deploy
11:01 springle: killed bunch of slow Flow\Formatter\ContributionsQuery::queryRevisions queries on flowdb
03:14 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon May 12 03:13:40 UTC 2014 (duration 13m 39s)
02:26 logmsgbot: LocalisationUpdate completed (1.24wmf4) at 2014-05-12 02:25:54+00:00
02:15 logmsgbot: LocalisationUpdate completed (1.24wmf3) at 2014-05-12 02:14:23+00:00

May 11

15:26 logmsgbot: reedy synchronized php-1.24wmf3/thumb.php
15:23 logmsgbot: reedy synchronized php-1.24wmf4/thumb.php
14:21 cmjohnson1: power cycling asw-d5-eqiad
03:11 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun May 11 03:10:37 UTC 2014 (duration 10m 36s)
02:23 logmsgbot: LocalisationUpdate completed (1.24wmf4) at 2014-05-11 02:22:02+00:00
02:13 logmsgbot: LocalisationUpdate completed (1.24wmf3) at 2014-05-11 02:12:49+00:00

May 10

19:58 logmsgbot: hoo synchronized php-1.24wmf3/extensions/Wikidata/ 'Resyncing Wikidata for mw1122'
18:31 hoo: approved an oauth request by Aaron Halfaker by making myself oauth admin for a moment
17:36 logmsgbot: hoo synchronized php-1.24wmf4/extensions/Wikidata/ 'Update Wikidata to fix the JSON dump generation'
17:35 logmsgbot: hoo synchronized php-1.24wmf3/extensions/Wikidata/ 'Update Wikidata to fix the JSON dump generation'
17:11 marktraceur: pushed new uploadwizard qunit job to Jenkins
16:30 logmsgbot: reedy updated /a/common to I415e67919: Memory limit to 235M
15:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'I415e679197b97e2babe50544cf1e8c26c13a598a'
13:59 logmsgbot: bsitu synchronized php-1.24wmf4/extensions/Flow 'Update Flow'
13:46 Krinkle: Reloading Zuul to deploy I403760f1f6dd1bc2
12:07 logmsgbot: hoo synchronized php-1.24wmf4/extensions/Wikidata/ 'Update Wikibase to fix performance issues with dumpJson'
12:05 logmsgbot: hoo synchronized php-1.24wmf3/extensions/Wikidata/ 'Update Wikibase to fix performance issues with dumpJson (2nd run)'
12:03 logmsgbot: hoo synchronized php-1.24wmf3/extensions/Wikidata/ 'Update Wikibase to fix performance issues with dumpJson'
08:53 paravoid: rack D5 down, switch unresponsive; minimal impact (mw1201-1203, 1208-1210)
06:56 logmsgbot: aaron synchronized php-1.24wmf3/img_auth.php '264967c58eccb6dae872ab7345d08f8381ac43a7'
06:46 logmsgbot: aaron synchronized php-1.24wmf4/img_auth.php 'b08af402ef2de7b2c79f71d848c2b8ae98b47be0'
03:13 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May 10 03:12:27 UTC 2014 (duration 12m 26s)
02:26 logmsgbot: LocalisationUpdate completed (1.24wmf4) at 2014-05-10 02:25:31+00:00
02:16 logmsgbot: LocalisationUpdate completed (1.24wmf3) at 2014-05-10 02:15:38+00:00
00:02 awight: updated crm from 237d463ed3e275c217a4f497ed30d2f7f20100eb to 3fd3b94834f94529841ad4a695ecd73c98e487bc

May 9

23:09 K4-713: updated payments cluster from 3be44f5d14c00a893a985f3aad86b6b59507a987 to 78cc4285bdeb6eecba3efc75e4a04c8b886561e4
19:24 logmsgbot: reedy synchronized wmf-config/ 'I5265c408443212536a5ed96d910caba50c22e767'
18:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable EducationProgram on ukwiki'
18:52 logmsgbot: reedy updated /a/common to I0d1ea1639: Remove 1.23wmf13 through 1.23wmf20
18:48 logmsgbot: reedy synchronized docroot and w
18:44 logmsgbot: reedy updated /a/common to I25d891030: Do not optimize commons for new highlighter
18:41 Reedy: Created EducationProgram tables on ukwiki
15:45 manybubbles: reindexing commons to unbreak file searches on wikis not using the experimental highlighter
15:43 logmsgbot: demon synchronized wmf-config/CirrusSearch-common.php 'Do not optimize commons for new highlighter on commons'
15:03 manybubbles: rebuilding hewiki's cirrus index so it can pick up hebmorph too
14:52 logmsgbot: spage synchronized php-1.24wmf4/extensions/Flow 'Fix Flow add new topics and reply in 1.24wmf4'
13:40 logmsgbot: reedy updated /a/common to I721c36406: Add en-rtl to wgExtraLanguageNames for beta
13:24 logmsgbot: reedy updated /a/common to I75a80a998: beta: create a RTL english wiki
10:43 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'raise db1067 and db1066 to normal load. depool db1043'
09:32 logmsgbot: hoo synchronized php-1.24wmf3/extensions/Wikidata/ 'Fix Job injection error handling'
09:24 logmsgbot: hoo synchronized php-1.24wmf3/extensions/Wikidata/ 'Fix Job injection error handling'
08:36 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'I4657fe64572fb3db22e3b48a87df7112b2248e35'
08:21 hashar: apt-get upgraded apache on gallium and lanthanum
08:18 hashar: Jenkins: un pooled integration-slave1001 and rebooting the instance.
08:16 hashar: Jenkins: un pooled integration-slave1002 and rebooting the instance.
08:16 hashar: restarting Zuul (seems some jobs are not properly registered)
06:55 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'pool db1066 in s2, db1067 in s1, warm up'
05:00 springle: installed db1067, db1070, db1071
04:09 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 9 04:08:07 UTC 2014 (duration 8m 6s)
03:47 springle: xtrabackup clone db1036 to db1067, db1051 to db1066
03:46 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'reduce db1036 and db1051 load while cloning'
03:24 springle: installed db106[678]
03:09 logmsgbot: LocalisationUpdate completed (1.24wmf4) at 2014-05-09 03:08:55+00:00
02:41 logmsgbot: LocalisationUpdate completed (1.24wmf3) at 2014-05-09 02:40:06+00:00
00:03 logmsgbot: maxsem synchronized php-1.24wmf3/extensions/MobileFrontend/ 'bug 65042'
00:01 logmsgbot: maxsem synchronized php-1.24wmf4/extensions/MobileFrontend/ 'bug 65042'

May 8

23:11 logmsgbot: maxsem synchronized php-1.24wmf3/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/132299'
23:08 logmsgbot: demon synchronized wmf-config/CirrusSearch-common.php 'Replica count for commonswiki_file -- syncing with whats already live'
23:07 logmsgbot: maxsem synchronized php-1.24wmf4/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/132299'
21:30 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wmgVectorBetaPersonalBar to true for all wikis'
21:28 logmsgbot: reedy updated /a/common to I44f67444c: group0 to 1.24wmf4
20:37 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf3 and group0 to 1.24wmf4
20:32 logmsgbot: reedy updated /a/common to I11e5ca294: FUTURE: Fifth batch of pilot sites for Media Viewer
20:05 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'touch'
20:04 logmsgbot: demon synchronized mediaviewer.dblist 'mediaviewer for svwiki, eswiki, jawiki, ptwiki'
18:56 DJ-K4: enabled fredge queue consumption + jenkins job
17:10 logmsgbot: reedy Finished scap: Build l10n cache for 1.24wmf4 and move testwiki (duration: 17m 05s)
16:53 logmsgbot: reedy Started scap: Build l10n cache for 1.24wmf4 and move testwiki
16:52 chasemp: rebooted ms-be1006 since it dropped dead
16:52 logmsgbot: reedy Finished scap: Build l10n cache for 1.24wmf4 and move testwiki (duration: 11m 42s)
16:42 manybubbles: reindexing the hebrew wikis other then hewikipedia now that they are on wmf3 so they can have hebmorph
16:40 logmsgbot: reedy Started scap: Build l10n cache for 1.24wmf4 and move testwiki
16:39 manybubbles: rebuilding enwiki's cirrus index to optimize for new highlighter
16:14 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to wmf3
16:13 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'raise db106[45] to normal load'
15:13 logmsgbot: manybubbles synchronized php-1.24wmf3/extensions/CirrusSearch/ 'updating Cirrus to pick up some fixes'
15:08 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php 'engage new hightlighter on some more wikis'
15:00 manybubbles: rebuilding all hebrew wikis _except_ hebrew wikipedia and hebrew wikisource to pick up hebmorph. hewikisource got it this morning. hewiki will get it this afternoon after the deployment train
13:24 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'warm up db1064 in s4, db1065 in s1'
12:59 manybubbles: rebuilding cirrus index for hewikisource to pick up hebmorph
10:18 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'reduce db1049 and db1051 load while cloning'
10:16 springle: xtrabackup clone db1051 to db1065
09:35 springle: xtrabackup clone db1049 to db1064
08:55 springle: installed db106[45]
08:35 logmsgbot: reedy synchronized docroot and w
08:18 logmsgbot: reedy synchronized php-1.24wmf4 'staging'
08:01 logmsgbot: reedy updated /a/common to I7f2d2b25d: Allow all users on OfficeWiki to send mass messages
03:44 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 8 03:43:39 UTC 2014 (duration 43m 38s)
02:57 logmsgbot: LocalisationUpdate completed (1.24wmf3) at 2014-05-08 02:56:02+00:00
02:29 logmsgbot: LocalisationUpdate completed (1.24wmf2) at 2014-05-08 02:28:17+00:00

May 7

23:36 logmsgbot: demon Finished scap: no-op scap, for ori (duration: 09m 01s)
23:27 logmsgbot: demon Started scap: no-op scap, for ori
23:17 logmsgbot: ori Finished scap: No changes; testing scap to osmium (again) (duration: 03m 25s)
23:14 logmsgbot: ori Started scap: No changes; testing scap to osmium (again)
23:14 logmsgbot: ori scap aborted: No changes; testing scap to osmium (duration: 11m 27s)
23:06 awight: update crm from 1740219e38091ba4f7afe6545ea189b27340bf86 to 237d463ed3e275c217a4f497ed30d2f7f20100eb
23:02 logmsgbot: ori Started scap: No changes; testing scap to osmium
21:37 K4-713: updated payments from a7fa0d64da2c56586c83cf92babb65bac857be2e to 3be44f5d14c00a893a985f3aad86b6b59507a987
21:00 K4-713: synchronized payments from 4811f6d3d80d126c to a7fa0d64da2c56586
20:49 K4-713: revlocked payments to 4811f6d3d80d126c due to strange errors during an attempted deploy of a7fa0d64da2c56586
20:17 awight: drush pm-uninstall wmf_fredge_qc
20:16 awight: bad call: drush en wmf_fredge_qc -- need to rollback the module schema version and try again with "fredge" creds
20:14 awight: crm updated from cfe34fe0b10861167199a8f72bba279b9cac5e6e to 1740219e38091ba4f7afe6545ea189b27340bf86
20:07 subbu: deployed parsoid 71f4e884 (with deploy sha 9a62899d)
17:19 logmsgbot: yurik synchronized php-1.24wmf2/extensions/ZeroRatedMobileAccess/
17:16 logmsgbot: yurik synchronized php-1.24wmf3/extensions/ZeroRatedMobileAccess/
16:33 manybubbles: performing a rolling restart on elasticsearch nodes in production to pick up new plugins: experimental-highlight 0.0.8 and analysis-hebrew 1.1.0
16:30 _joe_: restarted mwprof/profiler-to-carbon on tungsten, stuck somehow
15:40 logmsgbot: demon synchronized wmf-config/CirrusSearch-common.php 'Raised redundancy for commonswiki_file back up, config to match'
15:27 logmsgbot: anomie synchronized wmf-config/InitialiseSettings.php 'SWAT: Remove obsolete $wmgCirrusIsBuilding (no functionality change)'
15:26 logmsgbot: anomie synchronized wmf-config/CirrusSearch-common.php 'SWAT: Remove obsolete $wmgCirrusIsBuilding (no functionality change)'
15:19 anomie: anomie namespaceDupes.php on OfficeWiki done (that was quick)
15:18 anomie: anomie Running maintenance/namespaceDupes.php on OfficeWiki
15:16 logmsgbot: anomie synchronized wmf-config/InitialiseSettings.php 'SWAT: Change wgMetaNamespace for OfficeWiki and add alias'
15:12 logmsgbot: anomie synchronized wmf-config/InitialiseSettings.php 'SWAT: Allow all users on OfficeWiki to send mass messages (for real this time)'
15:09 logmsgbot: anomie synchronized wmf-config/InitialiseSettings.php 'SWAT: Allow all users on OfficeWiki to send mass messages'
15:03 logmsgbot: anomie synchronized wmf-config/InitialiseSettings.php 'SWAT: Set $wgUploadMissingFileUrl for enwiki'
10:21 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'db1049 to full steam'
09:40 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'warm up db1049 in s4'
07:54 hashar: Jenkins: installing Claim plugin (allow folks to comment on builds and mark them)
06:48 springle: again
05:59 springle: powercycled unresponsive neon, swapdeath + oom killer
03:12 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed May 7 03:11:36 UTC 2014 (duration 11m 35s)
02:28 logmsgbot: LocalisationUpdate completed (1.24wmf3) at 2014-05-07 02:27:23+00:00
02:15 logmsgbot: LocalisationUpdate completed (1.24wmf2) at 2014-05-07 02:14:44+00:00
00:02 mutante: upgrading libtiff on imagescalers

May 6

23:18 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/131855'
21:57 mutante: gracefull'ing apaches
20:43 logmsgbot: demon synchronized php-1.24wmf3/extensions/CirrusSearch 'Rolling Cirrus back to known-good state'
20:42 logmsgbot: demon synchronized php-1.24wmf2/extensions/CirrusSearch 'Rolling Cirrus back to known-good state'
20:27 logmsgbot: demon synchronized php-1.24wmf3/extensions/CirrusSearch/includes/Hooks.php 'Fix typehinting'
20:26 logmsgbot: demon synchronized php-1.24wmf2/extensions/CirrusSearch/includes/Hooks.php 'Fix typehinting'
20:19 logmsgbot: demon synchronized php-1.24wmf3/extensions/CirrusSearch/CirrusSearch.php 'I2638b695: fix for page moves'
20:19 logmsgbot: demon synchronized php-1.24wmf2/extensions/CirrusSearch/CirrusSearch.php 'I2638b695: fix for page moves'
20:18 logmsgbot: demon synchronized php-1.24wmf3/extensions/CirrusSearch/includes/Hooks.php 'I2638b695: fix for page moves'
20:17 logmsgbot: demon synchronized php-1.24wmf2/extensions/CirrusSearch/includes/Hooks.php 'I2638b695: fix for page moves'
19:00 mutante: enabling puppet on netmon1001
17:38 hoo: Changed email for global account "ElphiBot".
16:48 logmsgbot: mattflaschen synchronized php-1.24wmf3/extensions/GettingStarted/ 'GettingStarted token and logging deployment'
16:45 logmsgbot: mattflaschen synchronized php-1.24wmf2/extensions/GettingStarted/ 'GettingStarted token and logging deployment'
16:19 hoo: Changed email for global account "Elph".
15:42 akosiaris: killed zuul processes on gallium and restarted the service
09:16 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue May 6 09:15:20 UTC 2014
09:08 logmsgbot: LocalisationUpdate completed (1.22wmf15) at Tue May 6 09:07:45 UTC 2014
09:01 logmsgbot: faidon synchronized wmf-config/squid.php 'add Swift to squid.php'
08:59 logmsgbot: faidon updated /a/common to Ica9086dcd: Add Swift frontends to squid.php
08:45 ottomata: re-enabling puppet agent on analytics1022; kafka broker is caught up there and is fully in all ISRs
08:36 ottomata: temporarily disabling puppet on analytics1026 to troubleshoot a camus import problem
06:55 springle: hammering dbstore1001 with dumps in screen session. ignore replag
06:31 ori: ..on vanadium
06:31 ori: deleting rotated logs in /var/log/eventlogging/archive that are older than 90 days
04:47 springle: mydumper/myloader clone db1042 to db1049
04:06 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'depool db1049 for maintenance'
03:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue May 6 03:47:41 UTC 2014 (duration 47m 40s)
03:03 logmsgbot: LocalisationUpdate completed (1.24wmf3) at 2014-05-06 03:02:03+00:00
02:28 logmsgbot: LocalisationUpdate completed (1.24wmf2) at 2014-05-06 02:27:21+00:00
00:01 logmsgbot: ori synchronized wmf-config/CommonSettings.php 'Update wgFlowCacheVersion to 4.2'

May 5

23:57 logmsgbot: ori updated /a/common to Id1f2e0acf: Drop wgFlowCacheKey from CommonSettings.php
23:54 logmsgbot: ori Finished scap: SWAT deploy for VisualEditor and Flow cherry-picks (duration: 09m 55s)
23:44 logmsgbot: ori Started scap: SWAT deploy for VisualEditor and Flow cherry-picks
23:26 logmsgbot: ori synchronized php-1.24wmf3/extensions/EventLogging 'Update EventLogging for Id23b37fbe for SWAT.'
23:23 logmsgbot: ori synchronized php-1.24wmf2/extensions/EventLogging 'Update EventLogging for Id23b37fbe for SWAT.'
21:07 jgage: trying on analytics1022: https://wikitech.wikimedia.org/wiki/Analytics/Kraken/Kafka/Administration#Recovering_a_laggy_broker_replica
20:58 RobH: ssl1001-1003 now have updated unified cert in service
20:58 jgage: both kafka brokers back in service
20:54 RobH: cp4001-4020 unified cert and nginx service reloaded, back in service
20:50 RobH: ssl1006 and ssl1009 are responsive to nginx and back in service
20:43 RobH: pybal
20:43 RobH: ssl1009 was refusing connections both before and after my ssl cert update. ssl1006 is presently refusing connections post update. they are set to disabled in pubal
20:39 RobH: ssl1008 back into service, ssl1009 already depooled
20:38 jgage: forced kafka broker reelection
20:34 RobH: ssl1007 going back into service, ssl1008 depooling
20:25 RobH: depooled ssl1006/7 for update
20:25 RobH: ssl1004/5 returned to service (and puppet agents enabled)
20:21 RobH: puppet agent has been re-enabled on ssl1001-1003
20:19 RobH: ssl1004/5 disabled for update
20:18 RobH: putting ssl1002/3 back into service
20:15 subbu: deployed parsoid f2f1f1d7 (with deploy sha 71072f8a)
19:58 RobH: ssl1001 back in service, ssl1002-1003 set to disabled in pybal
19:18 RobH: depooling ssl1001 to test new certs live on system
19:09 RobH: disabled puppet on cp40XX, ssl10XX, and ssl30XX
19:08 logmsgbot: bblack synchronized wmf-config/squid.php 'REVERT: Update wgSquidServersNoPurge to use whole subnets for XFF checking'
19:07 logmsgbot: bblack updated /a/common to Iaf4d57d54: Revert "Use whole subnets in squid.php list for XFF acceptance"
19:03 logmsgbot: bblack synchronized wmf-config/squid.php 'Update wgSquidServersNoPurge to use whole subnets for XFF checking'
19:01 logmsgbot: bblack updated /a/common to I5a2d86ef0: Use whole subnets in squid.php list for XFF acceptance
17:05 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Revert "Increased htmlCacheUpdate throttle"'
16:00 logmsgbot: anomie synchronized php-1.24wmf3/extensions/MobileFrontend/ 'SWAT: Backport change 131237 to 1.24wmf3 to fix bug in MobileFrontend'
15:59 logmsgbot: anomie synchronized php-1.24wmf2/extensions/MobileFrontend/ 'SWAT: Backport change 131237 to 1.24wmf2 to fix bug in MobileFrontend'
15:49 logmsgbot: anomie synchronized php-1.24wmf2/includes/specials/SpecialAllmessages.php 'SWAT: Backport change 131041 to 1.24wmf2 to fix bug in Special:AllMessages'
15:37 logmsgbot: anomie synchronized php-1.24wmf2/includes/specials/SpecialAllmessages.php 'SWAT: Backport change 131041 to 1.24wmf2 to fix bug in Special:AllMessages'
15:24 logmsgbot: anomie synchronized php-1.24wmf3/includes/specials/SpecialAllmessages.php 'SWAT: Backport change 131041 to 1.24wmf3 to fix bug in Special:AllMessages'
15:12 logmsgbot: anomie synchronized php-1.24wmf3/includes/api/ApiLogin.php 'SWAT: Backport change 131056 to 1.24wmf3 to fix bug 64727'
15:10 logmsgbot: anomie synchronized php-1.24wmf2/includes/api/ApiLogin.php 'SWAT: Backport change 131056 to 1.24wmf2 to fix bug 64727'
12:45 akosiaris: removing various sdtpa devices from LibreNMS
03:12 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon May 5 03:11:15 UTC 2014 (duration 11m 14s)
02:32 ^demon|away: [gitb]lit's wonkiness but they're certainly not helping matters.
02:32 ^demon|away: antimony: ran very very aggressive repacking on mediawiki/core, operations/puppet, mediawiki/extensions/{UploadWizard,CentralAuth,CentralNotice,DonationInterface,FlaggedRevs,AbuseFilter,BlueSpiceExtensions,Translate,WikimediaMessages,EducationProgram,UniversalLanguageSelector,Wikibase}, pywikibot/{core,compat}, operations/dumps/tests. Basically anything taking up >90MB on disk. Probably not the cause of gitb
02:26 logmsgbot: LocalisationUpdate completed (1.24wmf3) at 2014-05-05 02:25:34+00:00
02:14 logmsgbot: LocalisationUpdate completed (1.24wmf2) at 2014-05-05 02:13:18+00:00

May 4

20:57 logmsgbot: aaron synchronized php-1.24wmf3/thumb.php 'c5ebd2aefce9e3fc5b994053078754021176f411'
20:40 logmsgbot: aaron synchronized php-1.24wmf3/thumb.php '6c230cbbc6ffa4d8909e88961ebf75755cf9c9d9'
19:24 logmsgbot: ori updated /a/common to I2916ef3bd: labs: stream recent changes to redis
09:58 _joe_: restarted gitblit, stuck on GC as usual.
08:40 _joe_: restarted apache on tungsten as it was stuck communicating with uwsgi
03:09 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun May 4 03:08:41 UTC 2014 (duration 8m 40s)
02:26 logmsgbot: LocalisationUpdate completed (1.24wmf3) at 2014-05-04 02:25:06+00:00
02:14 logmsgbot: LocalisationUpdate completed (1.24wmf2) at 2014-05-04 02:12:57+00:00

May 3

19:41 logmsgbot: hoo synchronized wmf-config/ 'Documentation only change'
06:02 ori: disabled puppet on osmium to test hhvm build
03:10 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May 3 03:09:40 UTC 2014 (duration 9m 39s)
02:27 logmsgbot: LocalisationUpdate completed (1.24wmf3) at 2014-05-03 02:26:41+00:00
02:15 logmsgbot: LocalisationUpdate completed (1.24wmf2) at 2014-05-03 02:14:43+00:00

May 2

23:13 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Made private wikis use thumb_handler.php for thumbnails'
22:30 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Removed useless "handlerUrl" config'
21:21 hashar: Jenkins is back :-]
21:14 hashar: restarting Jenkins (making sure the java process properly disappear)
21:14 hashar: zuul jenkins stuck again :(
20:31 logmsgbot: krinkle synchronized php-1.24wmf3/resources/Resources.php 'Ia12998fb11c686'
20:29 logmsgbot: krinkle synchronized php-1.24wmf2/resources/Resources.php 'Ia12998fb11c686'
17:47 RoanKattouw: Restarting stuck jenkins
16:04 _joe_: depooled mw1053 for hardware problems
15:13 andrewbogott: resetting a bunch more UIDs. Running find-and-chown again, but this time not on the swifts: salt -E '^(?!ms-be|labstore|snapshot).*$'
13:46 paravoid: swift @ eqiad: setting zone 5 (ms-be1013/1014/1015) to weight 2000, i.e. 66%
12:06 hashar: updated our Jenkins Job Builder copy abbf318..8df6bab
06:17 ori: re-enabled puppet on osmium and hafnium
04:02 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 2 04:01:04 UTC 2014 (duration 1m 3s)
03:10 logmsgbot: LocalisationUpdate completed (1.24wmf3) at 2014-05-02 03:09:16+00:00
02:40 logmsgbot: LocalisationUpdate completed (1.24wmf2) at 2014-05-02 02:39:39+00:00

May 1

23:43 bd808: Restarted logstash on logstash1001; MaxSem noticed that many recursion-guard logs were not being completely reassembled and JVM had one CPU maxed out.
23:08 logmsgbot: maxsem synchronized php-1.24wmf2/extensions/CommonsMetadata/ 'https://gerrit.wikimedia.org/r/#/c/130971/'
22:58 paravoid: disabling puppet on holmium; manually overriding completely broken varnish config
20:10 bd808: Deployed scap 92ea0e9 via trebuchet (not actively used yet)
19:17 logmsgbot: reedy synchronized wmf-config/
18:55 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf3
18:42 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf2
18:07 manybubbles: upgrading highlighter plugin on elasticsearch machines - the cluster will go yellow for a few hours during the rolling restart
16:43 logmsgbot: reedy Finished scap: testwiki to 1.24wmf3 (duration: 29m 54s)
16:18 subbu: deployed parsoid 5e05c585 (with deploy sha ca2db96d)
16:16 ottomata: reinstalling elastic1008
16:13 logmsgbot: reedy Started scap: testwiki to 1.24wmf3
16:10 logmsgbot: reedy updated /a/common to I832b45db6: Correct a domain in wgCopyUploadsDomains
16:01 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php 'Enable cirrus as a betafeature on all wikis which did not already have it.'
15:51 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php 'SWAT fix GWtoolset url and add some more logos'
15:40 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php 'SWAT fix GWtoolset url and add some more logos'
15:35 andrewbogott: reassigning a ton of UIDs in production; running a couple dozen 'find' commands to chown files
15:34 logmsgbot: manybubbles synchronized php-1.24wmf2/includes/Article.php 'SWAT update to prevent fatal in backwards compatibility method'
15:27 logmsgbot: manybubbles synchronized php-1.24wmf2/extensions/VisualEditor/ 'SWAT update for firefox focus'
15:08 logmsgbot: manybubbles synchronized php-1.24wmf2/extensions/Wikidata/ 'SWAT update for time parsing and formatting'
08:15 springle: switching s1-analytics-slave db1047 enwiki to tokudb
03:18 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 1 03:17:39 UTC 2014 (duration 17m 38s)
02:34 logmsgbot: LocalisationUpdate completed (1.24wmf2) at 2014-05-01 02:33:34+00:00
02:22 logmsgbot: LocalisationUpdate completed (1.24wmf1) at 2014-05-01 02:21:44+00:00

April 30

23:58 bblack: mobile caches now sync zero carriers/proxies from zero.wm.org rather than noc(fenari) temp hack solution
23:14 logmsgbot: ori synchronized php-1.24wmf2/extensions/VisualEditor 'Ibaf0cc823bfe: Update VisualEditor for cherry-picks'
20:34 logmsgbot: yurik synchronized php-1.24wmf2/extensions/ZeroRatedMobileAccess/
20:31 logmsgbot: yurik synchronized php-1.24wmf1/extensions/ZeroRatedMobileAccess/
18:08 logmsgbot: yurik synchronized wmf-config/mobile.php
17:37 yurikR: yurik Added $wmgZeroRatedMobileAccessApiUserName / password
17:36 logmsgbot: yurik synchronized wmf-config/PrivateSettings.php
17:17 logmsgbot: yurik synchronized php-1.24wmf2/extensions/ZeroRatedMobileAccess/
17:11 logmsgbot: yurik synchronized php-1.24wmf1/extensions/ZeroRatedMobileAccess/
17:10 logmsgbot: aaron synchronized php-1.24wmf2/thumb.php '93a33d733fa81a9a5396083ded6aa28a74f08a98'
17:07 logmsgbot: aaron synchronized wmf-config/PoolCounterSettings-eqiad.php 'Removed "TMHTransformFrame" pool counter entry'
16:19 Krinkle: restarting stuck Jenkins
16:16 Krinkle: Deploying Id248bd6706f32a on Zuul and reloading service
09:29 springle: dbstore100[12] replicating m2 eventlogging
06:54 logmsgbot: aaron synchronized wmf-config/PoolCounterSettings-eqiad.php 'Removed GetLocalFileCopy pool counter entry'
03:23 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Apr 30 03:23:13 UTC 2014 (duration 23m 12s)
02:33 logmsgbot: LocalisationUpdate completed (1.24wmf2) at 2014-04-30 02:33:12+00:00
02:22 logmsgbot: LocalisationUpdate completed (1.24wmf1) at 2014-04-30 02:22:23+00:00
01:58 Krinkle: Deploying Ia82779635d762a3 on Zuul, reloading services
00:02 logmsgbot: ebernhardson synchronized php-1.24wmf1/extensions/MultimediaViewer/ 'I684d44a0b5'

April 29

23:59 logmsgbot: ebernhardson synchronized php-1.24wmf2/extensions/Wikidata/ 'I84c2283e07'
23:52 logmsgbot: ebernhardson synchronized php-1.24wmf2/extensions/MultimediaViewer/ 'I84f8e347f'
23:37 logmsgbot: ori synchronized php-1.24wmf2/skins 'I66c56c577bad'
23:37 logmsgbot: ori synchronized php-1.24wmf2/extensions/VisualEditor 'I5818dce62'
23:25 logmsgbot: ebernhardson synchronized wmf-config/InitialiseSettings.php 'Enable MediaViewer survey on Spanish and Dutch Wikipedia'
23:22 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I59e1fa87e: Include language-0 categories for betawikiversity'
23:21 logmsgbot: ori updated /a/common to I59e1fa87e: Include language-0 categories for betawikiversity
23:16 logmsgbot: ebernhardson synchronized php-1.24wmf2/extensions/WikiEditor 'Update WikiEditor to 1.24wmf2'
21:59 logmsgbot: spage synchronized wmf-config/InitialiseSettings.php 'enable Flow on two mw talk pages for James_F'
21:25 Krinkle: Running deleteEqualMessages.php on newwiki (bug 43917)
21:12 AaronSchulz: populateImageSha1 fixer script finished on all wikis
20:47 manybubbles: rebuilding search indexes for group1 wikis after the train upgraded cirrus for them
20:13 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I56ae921ca: Change group name on the Persian Wikipedia'
20:13 logmsgbot: ori updated /a/common to I56ae921ca: Change group name on the Persian Wikipedia
18:16 logmsgbot: reedy synchronized multiversion/
18:09 logmsgbot: reedy synchronized wmf-config/ 'I52293b29a87e2c645735b37215e4113e561e47da'
18:04 logmsgbot: reedy synchronized docroot and w
18:03 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.24wmf2
17:13 manybubbles: raising number of replicas of enwiki's cirrus index from 1 to 2. cluster will probably complain while they allocate
16:53 RobH: osmium install complete, ticket resolved, ready for ^d and ori to take over
16:29 logmsgbot: reedy updated /a/common to Idb2a86791: Increased htmlCacheUpdate throttle
16:18 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Increased htmlCacheUpdate throttle'
16:05 logmsgbot: manybubbles synchronized php-1.24wmf2/extensions/Wikidata/ 'SWAT upgrade wikidata for date parsing fixes'
15:47 manybubbles: rebuilding test2wiki's cirrus index after swat deploy
15:45 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php 'SWAT add autopatrolled group to shwiktionary and draft namespace to chapcomwiki'
15:38 logmsgbot: manybubbles synchronized wmf-config/CirrusSearch-common.php 'SWAT deploy - move group0 wikis to experimental highlighter and give enwiki its redundency back'
15:37 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php 'SWAT deploy - extra setting for cirrus and new groups and sources for gwtoolset'
15:32 manybubbles: cirrus deploys look good, moving on to twkozlowski's requests
15:31 logmsgbot: manybubbles synchronized wmf-config/CirrusSearch-common.php 'SWAT deploy - move group0 wikis to experimental highlighter and give enwiki its redundency back'
15:31 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php 'SWAT deploy - extra setting for cirrus'
15:14 logmsgbot: manybubbles synchronized php-1.24wmf2/extensions/CirrusSearch/ 'SWAT upgrade - improves as yet undeployed highlighter config'
14:40 logmsgbot: faidon synchronized php-1.24wmf2/extensions/GettingStarted/ 'Revert GettingStarted anon tokens'
14:39 logmsgbot: faidon synchronized php-1.24wmf1/extensions/GettingStarted/ 'Revert GettingStarted anon tokens'
13:45 Krinkle: Running deleteEqualMessages.php on lnwiki (bug 43917)
13:41 logmsgbot: bblack synchronized wmf-config/InitialiseSettings.php 'Revert "Unset $wgUseXVO"'
13:10 Krinkle: Running deleteEqualMessages.php on nlwiki (bug 43917)
12:35 Krinkle: Running deleteEqualMessages.php on cswikiversity (bug 43917)
12:08 Krinkle: Running deleteEqualMessages.php on cswiktionary (bug 43917)
11:31 logmsgbot: bblack synchronized wmf-config/InitialiseSettings.php 'Revert "Unset $wgUseXVO"'
10:34 akosiaris: restarted gitblit on antimony
10:20 hashar: restarting Zuul
10:14 hashar: Jenkins / Zuul : upgrading python-gear from 0.4.0-1 to 0.5.4-1 . Should fix a bunch of jobs registrations issues in Zuul Gearman. bug 63760
09:59 akosiaris: update python-gear on apt.wikimedia.org to 0.5.4-1
08:30 akosiaris: Published carbon's IPv6 address in DNS. apt.wikimedia.org and ubuntu.wikimedia.org are now IPv6 enabled
05:25 AaronSchulz: Manually removed a few 10000s of duplicate Cyberbot job duplicates
03:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Apr 29 03:48:29 UTC 2014 (duration 48m 28s)
03:02 logmsgbot: LocalisationUpdate completed (1.24wmf2) at 2014-04-29 03:02:55+00:00
02:37 logmsgbot: LocalisationUpdate completed (1.24wmf1) at 2014-04-29 02:37:21+00:00
02:24 bblack: wiped disk cache (via mkfs) on cp1055 to (hopefully) clear crash-restart cycle, backend back in service now
01:53 Krinkle: Running deleteEqualMessages.php on cswiki (bug 43917)
00:49 Tim: on cp1055: backend varnish is continually panicking and restarting its child, will try to stop/start service
00:22 logmsgbot: aaron synchronized php-1.24wmf1/includes/WikiPage.php '3505cf933d874ea44bd5a3f3ffe210598ef7eec2'
00:14 logmsgbot: aaron synchronized php-1.24wmf2/includes/WikiPage.php '119fd9fc17b3c309b9065b54f4c83ede7d20498b'
00:00 logmsgbot: mwalker Finished scap: SWAT for 129813, 129640, 129708, 129707, and 130246 (duration: 11m 37s)

April 28

23:49 logmsgbot: mwalker Started scap: SWAT for 129813, 129640, 129708, 129707, and 130246
23:26 logmsgbot: aaron synchronized php-1.24wmf2/maintenance/runJobs.php '91dddcaffa58430204e2bf3c612d893b2710f33b'
22:43 jgage: rebooting db1047 due to unpingable and unresponsive on mgmt console
22:28 logmsgbot: mflaschen synchronized php-1.24wmf2/extensions/GettingStarted/ 'Revert token/TrackedPageContentSaveComplete GettingStarted change'
22:28 logmsgbot: mflaschen synchronized php-1.24wmf1/extensions/GettingStarted/ 'Revert token/TrackedPageContentSaveComplete GettingStarted change'
20:47 Krinkle: Running deleteEqualMessages.php on sqwiki (bug 43917)
20:38 paravoid: apache-graceful-all after tuning php.ini's expose_php setting
20:12 logmsgbot: reedy synchronized wmf-config/db-labs.php
20:08 apergos: restarted gmetad on nickel
20:02 gwicke: deployed Parsoid cab9348e using deploy 9e9030d
20:00 logmsgbot: mflaschen synchronized wmf-config/InitialiseSettings.php 'Update GettingStarted config for new format'
19:59 logmsgbot: mflaschen synchronized php-1.24wmf2/extensions/GettingStarted/ 'Sync GettingStarted for Growth team deploy'
19:58 logmsgbot: mflaschen synchronized php-1.24wmf1/extensions/GettingStarted/ 'Sync GettingStarted for Growth team deploy'
19:30 Krinkle: Running deleteEqualMessages.php on simplewiki (bug 43917)
19:25 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I5e0709ef0: Unset $wgUseXVO'
19:25 logmsgbot: ori updated /a/common to I5e0709ef0: Unset $wgUseXVO
19:22 Krinkle: Running deleteEqualMessages.php on rowiktionary (bug 43917)
19:01 Krinkle: Running deleteEqualMessages.php on bat-smgwiki (bug 43917)
18:48 Krinkle: Running deleteEqualMessages.php on afwikiquote (bug 43917)
18:41 hashar: Jenkins disconnected lanthanum slave, killed all jenkins-slave process on it and repooled server.
18:39 Krinkle: Running deleteEqualMessages.php on abwiki (bug 43917)
17:54 manybubbles: deploying a new version of our Elasticsearch highlighter by doing a rolling restart on Elasticsearch machines - should cause no interruption of service
16:51 akosiaris: executed graceful-stop, start for apaches in order to load the new php-luasandbox apache module
15:08 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php 'Add new sources to gwtoolset and namespaces to hewikisource'
12:48 _joe_: restarted apache on wikitech-static
12:29 Krinkle: Running deleteEqualMessages.php on cvwiki (bug 43917)
12:29 Krinkle: Running deleteEqualMessages.php on afwiki (bug 43917)
11:46 Krinkle: Running deleteEqualMessages.php on bpywiki (bug 43917)
11:00 springle: completed schema change, bug 64411, page_props.pp_sortkey
08:49 springle: reloading db1046 from fresh m2 dump
03:12 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Apr 28 03:12:11 UTC 2014 (duration 12m 10s)
03:00 springle: starting online schema change, bug 64411, page_props.pp_sortkey
02:30 logmsgbot: LocalisationUpdate completed (1.24wmf2) at 2014-04-28 02:30:04+00:00
02:22 springle: powercycle db1046 unresponsive
02:20 logmsgbot: LocalisationUpdate completed (1.24wmf1) at 2014-04-28 02:20:39+00:00

April 27

14:49 paravoid: stopping pybal on lvs300[1-4] to avoid the logspam
05:31 springle: mariadb sql dump in progress db1048 /a for rebuilding db1046. ok to kill if necessary
03:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Apr 27 03:08:01 UTC 2014 (duration 8m 0s)
02:28 logmsgbot: LocalisationUpdate completed (1.24wmf2) at 2014-04-27 02:28:32+00:00
02:19 logmsgbot: LocalisationUpdate completed (1.24wmf1) at 2014-04-27 02:19:43+00:00

April 26

23:27 logmsgbot: aaron synchronized php-1.24wmf2/includes/profiler/Profiler.php '7e20cdd2ba0381b81d3b43c8743fa4202a76bd61'
13:43 logmsgbot: hoo synchronized wmf-config/InitialiseSettings-labs.php 'Syncing for cluster consistency'
13:42 logmsgbot: hoo updated /a/common to Ic98928d54: Have Commons on Beta Labs use $stdlogo
13:26 springle: db1016 xfs head behind tail. reverted to last snapshot volume
12:57 springle: powercycle db1016 unresponsive
03:11 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Apr 26 03:11:16 UTC 2014 (duration 11m 15s)
02:31 logmsgbot: LocalisationUpdate completed (1.24wmf2) at 2014-04-26 02:31:41+00:00
02:22 logmsgbot: LocalisationUpdate completed (1.24wmf1) at 2014-04-26 02:22:41+00:00

April 25

20:29 logmsgbot: mwalker synchronized php-1.24wmf2/extensions/WikiEditor/ 'Reverting some faulty WikiEditor code for bug 64289'
20:28 logmsgbot: mwalker synchronized php-1.24wmf1/extensions/WikiEditor/ 'Reverting some faulty WikiEditor code for bug 64289'
18:02 K4-713: adjusted antifraud filters on payments
17:08 Jeff_Green: reenabled puppet and notifications for iodine
16:22 manybubbles: Elasticsearch rolling restart complete.
14:46 Jeff_Green: disabled icinga notifications for iodine too...
14:44 Jeff_Green: puppet stopped on iodine, doing manual spamassassin training
12:58 springle: upgrading db1047 (analytics slave) to mariadb 10
12:28 manybubbles: Performing rolling restart of Cirrus's Elasticsearch servers to upgrade a plugin. Low risk because it won't be used by the general public until Mondayish so a Friday push should be ok.
12:07 ottomata: stopping puppet on analytics1026 to test more frequent runs of Camus
12:02 logmsgbot: reedy synchronized wmf-config/
11:35 logmsgbot: reedy synchronized docroot and w
09:41 logmsgbot: reedy synchronized wmf-config/
09:15 logmsgbot: reedy synchronized wmf-config/ 'I4a68dc8321b7b302f5e89b5adafcff096f2ac35b'
09:13 logmsgbot: reedy synchronized multiversion/ 'I4a68dc8321b7b302f5e89b5adafcff096f2ac35b'
08:43 logmsgbot: reedy synchronized docroot and w
08:39 logmsgbot: reedy updated /a/common to I57b6d055e: Update flow cache version to 4.2
07:22 springle: up to 5x pt-table-sync running on db1048 m2 master for eventlogging migration. ok to kill if necessary
03:44 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Apr 25 03:44:16 UTC 2014 (duration 44m 15s)
03:03 logmsgbot: LocalisationUpdate completed (1.24wmf2) at 2014-04-25 03:03:33+00:00
02:37 logmsgbot: LocalisationUpdate completed (1.24wmf1) at 2014-04-25 02:37:06+00:00
00:47 mwalker: updating payments servers from e6d188f0dfcd57406acb58aa2b5bf45e48117c33 to a7fa0d64da2c56586c83cf92babb65bac857be2e for worldpay
00:29 bd808|deploy: Ori was able to fix permissions and second scap test worked as expected
00:28 logmsgbot: bd808 Finished scap: no-op scap to validate I24149ab and Ie967901 (try 2) (duration: 05m 02s)
00:26 bd808|deploy: File permissions on /srv/scap/scap/*.{py,pyc} were not consistently a+r which is needed for scap-rebuild-cdbs
00:23 logmsgbot: bd808 Started scap: no-op scap to validate I24149ab and Ie967901 (try 2)

April 24

23:55 logmsgbot: bd808 Finished scap: no-op scap to validate I24149ab and Ie967901 (duration: 02m 51s)
23:52 logmsgbot: bd808 Started scap: no-op scap to validate I24149ab and Ie967901
23:46 awight: perform crm schema update 7018
23:36 bd808|deploy: Running scap-rebuild-cdbs on tin to test python port
23:17 logmsgbot: mwalker synchronized wmf-config/CommonSettings.php 'Updating flow configuration 129589'
23:16 logmsgbot: mwalker synchronized php-1.24wmf2/extensions/Flow 'Updating flow for 129589 and 129604'
22:53 AaronSchulz: Running PopulateImageSha1.php for all multi-versioned files on all wikis to fix broken SHA-1s
21:22 springle: eventlogging dump loading on db1048 m2 master in screen. ok to kill if necessary
21:18 hashar: restarting Zuul
21:01 mwalker: updating payments from 4811f6d3d80d126c8b3c89c11d20cc6416cb58f6 to e6d188f0dfcd57406acb58aa2b5bf45e48117c33 for donationinterface / worldpay updates
20:39 paravoid: shutting down sdtpa, cr1-sdtpa, csw1-sdtpa, msw1-sdtpa and other sdtpa hosts gone forever
20:37 Coren: sync-apache for 126969 and 91339
19:59 logmsgbot: reedy synchronized docroot and w
19:54 logmsgbot: aaron synchronized wmf-config/PoolCounterSettings-eqiad.php 'Removed redundant pool counter config'
19:00 ori: eventlogging data streaming into db1048; db1047 consumer decom'd.
19:00 logmsgbot: reedy synchronized wmf-config/
18:59 logmsgbot: reedy synchronized database lists files:
18:43 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Rest of group0 to 1.24wmf2
18:38 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.24wmf1
18:33 springle: begin eventlogging migration db1047 to db1048 (m2), RT #7081
16:42 hashar: restarted both Zuul and Jenkins
16:29 logmsgbot: reedy synchronized php-1.24wmf2/extensions/Wikidata 'I43988505ea0fd7ac6b1278a50237e0e1d3ee0e9e'
16:25 hashar: restarting Zuul. Got asked to be stopped.
16:15 hashar: Jenkins ended up being stalled due to a known unfigured out issue :/
16:13 hashar: Killed a leftover jenkins process on gallium
15:50 akosiaris: scheduled a safe restart of jenkins
15:48 logmsgbot: hoo updated /a/common to I53de8d84b: Add two languages not supported by MediaWiki to testwikidata
15:48 logmsgbot: hoo synchronized wmf-config/InitialiseSettings.php 'Add two languages not supported by MediaWiki to testwikidata'
15:47 andrewbogott: zuul on gallium is dead and I don't know why
15:45 andrewbogott: restarted jenkins and zuul on gallium
14:02 logmsgbot: reedy Finished scap: testwiki to 1.24wmf2 build l10n cache (take 2) (duration: 15m 46s)
13:46 logmsgbot: reedy Started scap: testwiki to 1.24wmf2 build l10n cache (take 2)
13:45 logmsgbot: reedy Finished scap: testwiki to 1.24wmf2 build l10n cache (duration: 08m 16s)
13:37 logmsgbot: reedy Started scap: testwiki to 1.24wmf2 build l10n cache
13:32 logmsgbot: reedy updated /a/common to I543df75e3: Remove $wgDisableTextSearch and $wgDisableSearchUpdate overrides.
12:22 akosiaris: restarted morebots after upgrade of adminbot to 1.7.5
12:00 Krinkle: Running deleteEqualMessages.php on alswiki (bug 43917)
12:00 Krinkle: Running deleteEqualMessages.php on suwiki (bug 43917)
12:00 Krinkle: Running deleteEqualMessages.php on tlwiki (bug 43917)
11:59 Krinkle: Running deleteEqualMessages.php on nahwiktionary (bug 43917)
11:53 paravoid: reenabling ospf3 between cr1-eqiad/cr2-knams
11:50 paravoid: fixing private4/private6 ACLs to be consistent across all routers
06:47 _joe_: also ran puppet node clean to revoke certs, facts, etc (cp3013.esams.wikimedia.org cp3014.esams.wikimedia.org)
06:38 _joe_: ran puppetstoredconfigclean.rb for cp3013.esams.wikimedia.org cp3014.esams.wikimedia.org
05:20 springle: xtrabackup dbstore1001 to dbstore1002
03:29 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Apr 24 03:29:12 UTC 2014 (duration 29m 11s)
02:46 logmsgbot: LocalisationUpdate completed (1.24wmf1) at 2014-04-24 02:45:58+00:00
02:29 logmsgbot: LocalisationUpdate completed (1.23wmf22) at 2014-04-24 02:29:14+00:00

April 23

23:40 logmsgbot: catrope synchronized php-1.24wmf1/extensions/VisualEditor/lib/ve/modules/ve/ui/widgets/ve.ui.SurfaceWidget.js 'Fix surface focusing bug in Firefox'
23:39 logmsgbot: catrope synchronized php-1.24wmf1/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.Target.js 'Unbreak badtoken recovery in mobile VE'
23:38 logmsgbot: catrope synchronized php-1.24wmf1/resources/src/jquery/jquery.suggestions.js 'Handle CSS ellipsis when calculating suggestions widths'
23:37 logmsgbot: catrope synchronized php-1.23wmf22/resources/src/jquery/jquery.suggestions.js 'Handle CSS ellipsis when calculating suggestions widths'
21:55 bblack: moved cp301[34] ethernet ports to private1-esams
20:14 Krinkle: Running deleteEqualMessages.php on mlwiki (bug 43917)
20:14 Krinkle: Running deleteEqualMessages.php on miwiki (bug 43917)
20:11 Krinkle: Running deleteEqualMessages.php on gvwiki (bug 43917)
20:11 Krinkle: Running deleteEqualMessages.php on euwiktionary (bug 43917)
20:04 subbu: deployed Parsoid 9c99b0be (deploy SHA cf5eb4d0)
19:51 Krinkle: Running deleteEqualMessages.php on brwiki (bug 43917)
19:51 Krinkle: Running deleteEqualMessages.php on afwiki (bug 43917)
19:50 Krinkle: Running deleteEqualMessages.php on iawiki (bug 43917)
19:37 Krinkle: Running deleteEqualMessages.php on hrwiktionary (bug 43917)
19:37 Krinkle: Running deleteEqualMessages.php on hrwiki (bug 43917)
19:34 Krinkle: Running deleteEqualMessages.php on dawiki (bug 43917)
19:00 logmsgbot: reedy synchronized wmf-config/ 'I543df75e364171a71a48f18429972b662b542894'
18:58 logmsgbot: reedy updated /a/common to I865a08779: Fix $wmgBetaFeaturesWhitelist for labs too
18:58 Krinkle: Running deleteEqualMessages.php on amwiki (bug 43917)
17:59 logmsgbot: demon synchronized wmf-config/InitialiseSettings-labs.php 'no-op in prod, for completeness'
17:52 logmsgbot: demon synchronized all-labs.dblist 'no-op for prod, syncing for completeness'
17:52 logmsgbot: demon synchronized wikiversions-labs.json 'no-op for prod, syncing for completeness'
17:51 logmsgbot: demon updated /a/common to I960a792bc: Override wgSearchTypeAlternatives for beta to remove lucene
15:58 akosiaris: updated adminbot on apt.wikimedia.org to 1.7.5
15:57 manybubbles: rebuilding commons' cirrus search index
15:55 logmsgbot: manybubbles synchronized php-1.24wmf1/extensions/CirrusSearch/maintenance/updateOneSearchIndexConfig.php 'swat update to fix maintenance script'
15:46 ottomata: temporarily disabling puppet on analytics1003 to test some kafkatee settings
15:32 apergos: rebooting dataset2 hoping to detect the arrays on reboot
14:40 akosiaris: unexported /vol/{originals,thumbs} on nas1001-a, nas1-a
14:30 akosiaris: break replication for volumes originals, thumbs on nas1001-a, nas1-a
14:03 paravoid: adding AS path 1257 6830 (Tele2 -> UPC) to avoided paths @ cr1-esams/cr2-knams, multiple users reporting slowness issues
13:52 akosiaris: unmounted /vol/originals and /vol/thumbs on fenari (was /mnt/upload7, /mnt/thumbs2) see RT #7076
12:54 logmsgbot: marc synchronized wmf-config/interwiki.cdb 'Updating interwiki cache'
12:50 hashar: Jenkins back
12:46 hashar: restarting jenkins
12:46 hashar: Jenkins: upgrading email-ext and JobConfigHistory plugins (the later now supports slaves configs!)
10:35 hashar: Jenkins: update lanthanum slave agent to use java7
10:32 hashar: Jenkins switching integration-slave1001.eqiad.wmflabs java to use Java 7 . In https://integration.wikimedia.org/ci/computer/integration-slave1001/configure changed JavaPath to /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java
07:45 mutante: nfs1 - delete some old kernels and zip mw logs last touched in 2012/13 to free some disk on /
07:32 mutante: nfs1 - re-enabled puppet
07:12 mutante: nfs2 - revoke puppet cert,salt key,stored configs
06:26 mutante: db48,db63 - revoke puppet cert, salt key, kill from storedconfigs
03:55 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Apr 23 03:55:03 UTC 2014 (duration 55m 2s)
03:09 logmsgbot: LocalisationUpdate completed (1.24wmf1) at 2014-04-23 03:08:58+00:00
02:46 logmsgbot: LocalisationUpdate completed (1.23wmf22) at 2014-04-23 02:46:47+00:00
02:25 manybubbles: restarted rebuilding common's Cirrus index after something crashed. going to get more logging out of it if it crashes again. or it'll work. Either way. Like last time the Elasticsearch check might freak out for a bit after it finished because shards are assigning. That can be ignored for an hour or so.

April 22

23:30 logmsgbot: ori Finished scap: I595446dc5, If2c57846f, Iaa232298e (duration: 00m 45s)
23:30 logmsgbot: ori Started scap: I595446dc5, If2c57846f, Iaa232298e
23:28 logmsgbot: ori synchronized php-1.24wmf1/extensions/EventLogging 'Update EventLogging for Iaa232298e: Set line-height for code icon on schema pages (bug 64251)'
23:27 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'If2c57846f: Enable survey option in MediaViewer on a few more wikis'
23:26 logmsgbot: ori updated /a/common to If2c57846f: Enable survey option in MediaViewer on a few more wikis
23:24 logmsgbot: ori synchronized php-1.23wmf22/extensions/MultimediaViewer 'Update MultimediaViewer for I595446dc5: Add more survey languages (fr, de, pt/pr-br)'
23:23 logmsgbot: ori synchronized php-1.24wmf1/extensions/MultimediaViewer 'Update MultimediaViewer for I595446dc5: Add more survey languages (fr, de, pt/pr-br)'
22:28 logmsgbot: aaron synchronized php-1.24wmf1/maintenance/populateImageSha1.php '32d9206'
22:22 logmsgbot: reedy synchronized php-1.24wmf1/extensions/TimedMediaHandler 'I7483c8b7ec75f5149998da2b530ca04'
21:31 logmsgbot: spage synchronized php-1.24wmf1/extensions/Flow/modules/discussion/styles/mixins/collapse.less 'Fix Flow collapsed topics on mw.org'
21:14 logmsgbot: spage synchronized wmf-config/InitialiseSettings.php 'Enable Flow on Compact Personal Bar talk'
21:12 logmsgbot: spage updated /a/common to I851651247: Non wikipedias to 1.24wmf1
20:35 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'No op in prod, disables lsearchd completely for beta'
20:28 ottomata: turning on varnishkafka on text varnishes
20:12 MatmaRex: wikibugs replaced by pywikibugs (https://github.com/valhallasw/pywikibugs) and moved to #wikimedia-dev (at last!)
20:12 manybubbles: rebuilding the search index for a few wikis - might cause the Elasticsearch health check to freak out because it sucks
19:51 MatmaRex: wikibugs is down, let's not bring it back up
19:31 Krinkle: Reloading Zuul to deploy config change I9c2f94b138244ab8
19:05 hashar: Jenkins killed Jenkins java process on deployment-bastion.eqiad.wmflab to free up the executor and threads entirely.
18:55 hashar: restarted Zuul to clean up some stuck jobs from the queue
18:49 hashar: Jenkins deployment-bastion.eqiad.wmflab is back online: Slave successfully connected and online
18:47 logmsgbot: reedy synchronized docroot and w
18:46 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: non wikipedias to 1.24wmf1
18:43 RobH: tridge back and accessible
18:42 hashar: Jenkins deposing / repooling deployment-bastion.eqiad.wmflabs slave locked up somehow, the executors are no more taken in account by Jenkins master
18:33 RobH: resurrecting tridge in pmtpa
18:00 RobH: tridge is coming dow for relocation, shouldnt disrupt anything but backups in progress
17:52 bblack: disable cp301[34] (mobile varnish frontends) in pybal on fenari
17:21 awight: update crm from 7dafce5 to cfe34fe
16:45 mark: Reenabled Apache and puppet on fenari
16:40 logmsgbot: aaron synchronized php-1.24wmf1/includes/filerepo/file/LocalFile.php 'e9807d0'
16:15 logmsgbot: reedy updated /a/common to I55954c612: Commit updated interwiki.cdb file
15:47 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Collection back on, server move over'
15:14 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Icb6b4bad: Updated $wgForceUIMsgAsContentMsg for commonswiki'
14:59 cmjohnson1: shutting down and relocating virt0 and pdf2
14:50 logmsgbot: marc synchronized wmf-config/interwiki.cdb 'Updating interwiki cache'
14:41 logmsgbot: marc synchronized wmf-config/interwiki.cdb 'Updating interwiki cache'
14:33 manybubbles: populating cirrus indexes for all remaining wikis
14:19 akosiaris: added bblack account on all junipers
14:18 manybubbles: building new elasticsearch indexes for the last wikis that didn't have them. the cluster may go red as the indexes are assigned. silly nagios check.
14:15 logmsgbot: manybubbles synchronized wmf-config/ 'cirrus for more wikis and disable collection for more'
14:13 logmsgbot: manybubbles synchronized docroot/noc/createTxtFileSymlinks.sh 'noncirrus is removed'
14:09 cmjohnson1: mchenry and sanger going down for server relocation
13:26 mark: Disabled puppet and apache on fenari
13:25 paravoid: second pass of swiftrepl eqiad->esams
12:04 logmsgbot: faidon synchronized wmf-config/squid.php 'add cp3013/cp3014 IPv6 addresses'
12:04 logmsgbot: faidon updated /a/common to If8f39abee: squid.php: add cp3013/cp3014 IPv6 addresses
10:35 akosiaris: upgraded php-luasandbox to 1.9.1 on beta (deployment-apache0{1,2})
10:25 akosiaris: upgraded php-luasandbox to 1.9-1 on test.wikimedia.org
10:18 mutante: harmon - delete salt key
09:04 mutante: hooper - revoked puppet cert
08:47 mutante: upgrading Bugzilla to 4.4.4
06:58 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Unbreak $wmgBetaFeaturesWhitelist'
06:44 mutante: db77 - revoke puppet cert,salt key,rm from monitoring
05:14 springle: db68 down. s1-analytics-slave cname to db1007
04:35 paravoid: reactivate esams<->HE & eqiad<->HE peerings; issues are confirmed to be resolved
03:32 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Apr 22 03:32:29 UTC 2014 (duration 32m 28s)
02:43 logmsgbot: LocalisationUpdate completed (1.24wmf1) at 2014-04-22 02:43:01+00:00
02:30 logmsgbot: LocalisationUpdate completed (1.23wmf22) at 2014-04-22 02:29:34+00:00
01:16 awight: nevermind previous update.
01:11 awight: update from 7dafce5 to cfe34fe
00:18 logmsgbot: catrope synchronized php-1.24wmf1/extensions/MobileFrontend/javascripts/modules/editor/VisualEditorOverlay.js 'Fix JS error on save'
00:15 Reedy: torrus (on manutius) is down
00:05 RoanKattouw: Restarted ircecho on ekrem, IRC working again now
00:00 K4-713: updated payments from 2819549 --> 4811f6d

April 21

23:57 RoanKattouw: Started ircd on ekrem, startup doesn't seem to be puppetized
23:24 logmsgbot: catrope synchronized php-1.23wmf22/extensions/MultimediaViewer 'SWAT deploy cherry-picks'
23:24 logmsgbot: catrope synchronized php-1.24wmf1/extensions/MobileFrontend 'SWAT deploy cherry-picks'
23:23 logmsgbot: catrope synchronized php-1.24wmf1/extensions/MultimediaViewer 'SWAT deploy cherry-picks'
23:23 logmsgbot: catrope synchronized php-1.24wmf1/extensions/VisualEditor 'SWAT deploy cherry-picks'
23:16 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Lithuanian namespace aliases for betawikiversity'
23:11 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'BetaFeatures whitelist'
23:11 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Beta Features whitelist'
22:50 RoanKattouw: Restarted stuck Jenkins
21:33 awight: updated payments: af35b7b --> 2819549
20:45 andrewbogott: rebooted wtp1018
20:45 subbu: deployed Parsoid ec51e5d1 (deploy SHA 0dd607fc)
20:31 manybubbles: rolling restart on remaining Elasticsearch servers to get the plugin (1010, 1011, 1012, 1015)
19:45 manybubbles: rolling restart on more of the elasticsearch servers to pick up plugins (06, 07, 09)
19:15 cmjohnson1: shutting down and relocating dobson
19:15 cmjohnson1: shutting down and relocating linne
17:39 cmjohnson: shutting down and relocating fenari
17:37 cmjohnson: shutting down mexia to relocate to 12th floor
17:25 logmsgbot: aaron synchronized php-1.24wmf1/includes/filerepo/file/LocalFile.php '2026e4a'
17:22 logmsgbot: aaron synchronized wmf-config/PoolCounterSettings-eqiad.php 'Adjust large file download pool counter config to tie up less workers'
17:18 logmsgbot: aaron synchronized php-1.23wmf22/includes/filerepo/file/LocalFile.php '01ce288'
17:13 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'renderfile-nonstandard throttle config'
17:03 logmsgbot: aaron synchronized php-1.24wmf1/thumb.php '44c4658'
17:02 logmsgbot: aaron synchronized php-1.23wmf22/thumb.php '9591365'
16:34 ottomata: reinstalling elastic1013 (elastic1014 is still coming back online, but I don't want there to an extra eligible master for long)
16:04 ottomata: reinstalling elastic1014
15:22 cmjohnson1: dataset2 going down to be relocated to the 12th floor
13:41 manybubbles: rolling restart on some of the Elasticsearch servers to pick up new plugins. should not cause any trouble.
13:05 Reedy: De-activated status.wm.o monitor for icinga due to false positive from HTTP auth
12:54 paravoid: demoting myself, removing Commons crat/admin rights
12:41 paravoid: escalating myself to Commons bureaucrat/admin, then adding GWToolset privileges
12:40 paravoid: deleting 29 GWToolset XML under Swift's wikipedia-commons-gwtoolset-metadata container for user Fæ/
11:51 logmsgbot: reedy synchronized php-1.23wmf22/extensions/TimedMediaHandler 'I7483c8b7ec75f5149998da2b530ca04'
11:50 paravoid: deactivating esams<->HE peering, >90% packet loss between lon<->nyc
11:49 paravoid: deactivating eqiad<->HE peering, >90% packet loss between lon<->nyc
11:45 logmsgbot: reedy synchronized php-1.23wmf22/extensions/TimedMediaHandler 'I7483c8b7ec75f5149998da2b530ca0467ac70de7'
03:55 springle: reset pc100* slaves previously replicating from pmtpa
03:32 ori: 5.5k fatals over last 20 hrs, of which 3.5k are calls to doTransform() on a non-object at TimedMediaThumbnail.php:201, and 0.9k are Lua API OOMs at LuaSandbox/Engine.php:264
03:30 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Apr 21 03:30:15 UTC 2014 (duration 30m 14s)
03:26 ori: ap_busy_workers spike on image scalers eqiad, started ~2:55, subsided around ~3:20
02:42 logmsgbot: LocalisationUpdate completed (1.24wmf1) at 2014-04-21 02:42:30+00:00
02:29 logmsgbot: LocalisationUpdate completed (1.23wmf22) at 2014-04-21 02:29:49+00:00

April 20

18:51 ori: restarted grrrit-wm by following instructions on https://wikitech.wikimedia.org/wiki/Grrrit-wm#Restarting_the_bot
11:12 Nemo_bis: grrrit dead: 10.28 -!- grrrit-wm [tools.lolr@208.80.155.145] has quit
03:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Apr 20 03:26:48 UTC 2014 (duration 26m 47s)
02:39 logmsgbot: LocalisationUpdate completed (1.24wmf1) at 2014-04-20 02:39:23+00:00
02:28 logmsgbot: LocalisationUpdate completed (1.23wmf22) at 2014-04-20 02:28:09+00:00

April 19

03:27 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Apr 19 03:27:52 UTC 2014 (duration 27m 51s)
02:41 logmsgbot: LocalisationUpdate completed (1.24wmf1) at 2014-04-19 02:41:02+00:00
02:29 logmsgbot: LocalisationUpdate completed (1.23wmf22) at 2014-04-19 02:29:33+00:00

April 18

21:00 hashar: Jenkins renamed mw-jenkinsbot irc bot to wmf-insecte (french for "bug"). Updated IRC conf to point to chat.freenode.net:7000 with SSL.
19:02 bblack: enabled cp30[14] varnish mobile frontends in esams pybal
17:50 bblack: cp301[34] reinstalls complete, should stay ok in monitoring
17:48 ottomata: resinsalling elastic1008
16:20 springle: db48 mysqld shutdown for decom
16:20 bblack: ignore cp301[34] msgs, reinstalling them
16:10 springle: db63 mysqld shutdown for decom
15:53 ottomata: reinstalling elastic1007
15:52 springle: db48 mysqld set read_only, disabled m2 repl to db1048
15:51 ottomata: disabling puppet on elasti1007 and elastic1008 for reformatting
15:45 mutante: DNS update - removing Tampa msbe/msfe
15:38 Jeff_Green: switched mchenry to use db1048/db1049 for OTRS address lookups
15:24 mutante: DNS update - removing all the Tampa mw/srv mgmt
15:15 mutante: DNS update - removing lvs1-6
14:54 mutante: es5,es6 - revoke puppet certs, salt keys, icinga
14:51 ottomata: powering down stat1 for decom
14:43 mutante: ms-fe[14] - shutting down
14:41 ottomata: disabling puppet on stat1 for decom
14:37 mutante: ms-be 1-12, Tampa Swift boxes, shutdown
14:24 mutante: ms-fe[14] - stop puppet,revoke certs,remove icinga
13:54 mutante: ms-be1-12 - removing from puppet,salt,icinga
13:06 mutante: Bugzilla Apache, changed SSL cipher suite in I7e9adc182dc ,might cost a a few % performance but zirconium had plenty
11:48 hashar: removing mw-jenkinsbot (the wikimedia jenkins installation) from #wikimedia-labs
10:10 hashar: Jenkins upgraded to 1.532.3.
10:06 hashar: Upgrading Jenkins to latest LTS version 1.532.3
07:57 mutante: DNS update - remove api.svc, arptest.pmtpa ..
06:31 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Next round of wikis done building Cirrus indexes, throw into beta mode'
04:04 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Apr 18 04:04:21 UTC 2014 (duration 4m 20s)
03:06 logmsgbot: LocalisationUpdate completed (1.24wmf1) at 2014-04-18 03:06:06+00:00
02:39 logmsgbot: LocalisationUpdate completed (1.23wmf22) at 2014-04-18 02:39:51+00:00
00:21 logmsgbot: ori synchronized wmf-config/CommonSettings.php 'Ie9b265be9: Enable GlobalCssJs on testwiki & test2wiki (2/2)'
00:21 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'Ie9b265be9: Enable GlobalCssJs on testwiki & test2wiki (1/2)'
00:20 logmsgbot: ori updated /a/common to Ie9b265be9: Enable GlobalCssJs on testwiki & test2wiki
00:18 logmsgbot: ori Finished scap: Cherry-pick Ibe8e67ebf for MobileFrontend on 1.23wmf22 and 1.24wmf1; add GlobalCssJs extension to 1.24wmf1 and 1.23wmf22 (duration: 32m 53s)

April 17

23:45 logmsgbot: ori Started scap: Cherry-pick Ibe8e67ebf for MobileFrontend on 1.23wmf22 and 1.24wmf1; add GlobalCssJs extension to 1.24wmf1 and 1.23wmf22
23:39 logmsgbot: ori scap failed: CalledProcessError Command '/usr/local/bin/mw-update-l10n' returned non-zero exit status 1 (duration: 00m 24s)
23:38 logmsgbot: ori Started scap: Cherry-pick Ibe8e67ebf for MobileFrontend on 1.23wmf22 and 1.24wmf1; add GlobalCssJs extension to 1.24wmf1
23:37 logmsgbot: ori updated /a/common to I2a2abd7f3: Add GlobalCssJs to extension-list
23:29 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I52378a4b4: Add meta to legalteamwiki import sources'
23:28 logmsgbot: ori updated /a/common to I52378a4b4: Add meta to legalteamwiki import sources
23:27 logmsgbot: ori synchronized wmf-config/CommonSettings.php 'I373df6138: Normalize TextExtracts config handling (2/2)'
23:27 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I373df6138: Normalize TextExtracts config handling (1/2)'
23:26 logmsgbot: ori updated /a/common to I373df6138: Normalize TextExtracts config handling
23:24 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I7841f74b0: Kill all vestiges of $wgMFRemovableClasses (2/2)'
23:24 logmsgbot: ori synchronized wmf-config/mobile.php 'I7841f74b0: Kill all vestiges of $wgMFRemovableClasses (1/2)'
23:23 logmsgbot: ori updated /a/common to I7841f74b0: Kill all vestiges of $wgMFRemovableClasses
23:18 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I1795c70d1: Create a FeaturedFeed for the Tech News bulletin (2/2)'
23:17 logmsgbot: ori synchronized wmf-config/FeaturedFeedsWMF.php 'I1795c70d1: Create a FeaturedFeed for the Tech News bulletin (1/2)'
23:16 logmsgbot: ori updated /a/common to I1795c70d1: Create a FeaturedFeed for the Tech News bulletin
23:14 logmsgbot: ori synchronized php-1.23wmf22/extensions/ApiSandbox 'I9a56b2c5a: Update ApiSandbox'
23:12 K4-713: updates antifraud rules in payments
22:03 andrewbogott: updated default labs precise image (heartbleed fix)
20:41 manybubbles: elastic1016 restarted and not freaking out any more.
20:37 _joe_: restarting gitblit in order to prevent crippling due to the usual memory leak
20:28 manybubbles: restarting elastic1016 - it is freaking out. If it happens again I'll dig deeper, but for now I consider it a fluke of the rolling restarts today....
20:20 RobH: sorry for the misc-web-lb issues folks, they should be resolved at this time (for now)
20:19 paravoid: lvs1002/1005: commenting first resolv.conf entry until we have a more permanent fix, restarting pybal
20:18 paravoid: disabling puppet on lvs1002/lvs1005
19:57 RobH: still working on issue
19:57 RobH: both cp1043 and cp1044 seem online and serving nginx service, but pybal says they are down still working
19:46 ottomata: power off emery
19:40 RobH: replacing ticket.wikimedia.org cert/key, apache may hiccup
19:33 RobH: blog.w.o cert replacement successful
19:30 ottomata: disabling puppet on emery for decommission
19:29 RobH: blog.w.o certificate swap (yes, again ;), apache may hiccup
19:10 logmsgbot: reedy synchronized wmf-config/
19:09 logmsgbot: reedy synchronized database lists files: I6fc44d3eb829d656d352dab652148dd327b06679
19:04 ottomata: reinstalling elastic1001
18:59 logmsgbot: faidon synchronized wmf-config/CommonSettings.php 'reenable CN CrossWiki Hiding'
18:58 logmsgbot: faidon updated /a/common to Ie95165065: Reenable CentralNotice CrossWiki Hiding
18:57 logmsgbot: reedy synchronized wmf-config/
18:34 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Touch for I0c36c65bb9f405e03b84d3f6c6b93acda522c5c9'
18:33 logmsgbot: reedy synchronized database lists files: I0c36c65bb9f405e03b84d3f6c6b93acda522c5c9
18:30 ottomata: switching erbium udp2log instance from consuming multicast relay to unicast direct from varnishes
18:21 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf1
18:10 ottomata: stopping puppet on elastic1001 and elastic1002, reinstalling elastic1002
18:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.23wmf22
16:06 logmsgbot: reedy Finished scap: testwiki to 1.24wmf1 and build l10n cache (duration: 26m 06s)
15:47 logmsgbot: anomie synchronized php-1.23wmf22/extensions/VisualEditor 'SWAT: 126913 - backport to wmf22 of critical fixes for the Math extension's VisualEditor tool'
15:47 logmsgbot: anomie synchronized php-1.23wmf22/extensions/Math 'SWAT: 126913 - backport to wmf22 of critical fixes for the Math extension's VisualEditor tool'
15:40 logmsgbot: reedy Started scap: testwiki to 1.24wmf1 and build l10n cache
15:28 logmsgbot: faidon synchronized wmf-config/CommonSettings.php 'disable CN CrossWiki Hiding again'
15:27 logmsgbot: faidon updated /a/common to If74ba5a52: Revert "Enable CentralNotice CrossWiki Hiding"
15:17 manybubbles: updgraded site plugins on Elasticsearch nodes
15:04 ottomata: reinstalling elastic1016
14:02 logmsgbot: reedy updated /a/common to I290bd1ea6: Remove further pmtpa remnants
13:41 manybubbles: synced experimental highlighter to elasticsearch nodes - they'll pick it up on restart
11:05 logmsgbot: reedy synchronized wmf-config/ 'I290bd1ea628563646c02651041fa2cec4a320b56'
10:56 mutante: lvs3,lvs4,lvs5,lvs6 - shutdown
10:42 mutante: lvs1, lvs2 shutdown
10:15 mutante: re-deleting unaccepted salt keys for virt2,5-11
10:11 mutante: lvs1-6 - disable puppet,salt,revoke certs,keys
08:28 mutante: db35,db38 - shutdown
07:47 mutante: db35,db38, stop puppet and salt, revoke certs,keys
07:45 mutante: restarting gitblit
03:49 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Apr 17 03:49:19 UTC 2014 (duration 49m 18s)
03:06 subbu: deployed Parsoid 0bccf02c (deploy SHA 5e25f3b05) @ 1:30 pm PST, Apr 16th, 2014
03:02 logmsgbot: LocalisationUpdate completed (1.23wmf22) at 2014-04-17 03:02:25+00:00
02:33 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-17 02:33:47+00:00
02:03 springle: stop mysqld on db35 (m1) for decom
01:42 springle: xtrabackup db63 to db60

April 16

23:19 logmsgbot: mwalker Finished scap: SWAT deploy: configuration change 126223 and multimediaviewer 126852 (duration: 04m 07s)
23:15 logmsgbot: mwalker Started scap: SWAT deploy: configuration change 126223 and multimediaviewer 126852
20:03 RobH: osmium cleared from salt, puppetca, and puppetstoredconfig for reinstall with trusty (ignore any icinga alerts, there are no pages)
18:58 ottomata: reinstalling elastic1015
17:54 logmsgbot: reedy synchronized php-1.23wmf22/includes/jobqueue/ 'I4b4dbe4637dc50cd4630ef19d54f01efba10e138'
17:09 paravoid: starting swiftrepl on copper for eqiad->esams copy
17:08 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I7b6e5c2d7: Enable web fonts by default on Hebrew Wikisource'
17:07 logmsgbot: ori synchronized fc-list 'Ib7b2bc21a: updated fonts list and sorted it, rt #810'
17:06 logmsgbot: ori updated /a/common to I7b6e5c2d7: Enable web fonts by default on Hebrew Wikisource
16:37 logmsgbot: reedy synchronized php-1.23wmf21/includes/jobqueue/JobQueueRedis.php 'I678ab55ae3678b5cd944393f2f2048851625f153'
16:36 logmsgbot: reedy synchronized php-1.23wmf22/includes/jobqueue/JobQueueRedis.php 'I678ab55ae3678b5cd944393f2f2048851625f153'
15:38 ottomata: reinstalling elastic1012
13:50 manybubbles: restarting elastic1009 to suck up new config
13:50 manybubbles: raised the number of replicas for labswiki's search directly in elasticsearch because I can't easilly do for cirrus due to access restrictions
13:45 ottomata: reinstalling elastic1011
13:22 mutante: DNS update - remove virt5-15
12:11 mutante: virt5-11 - shut down
11:40 akosiaris: upgraded python-voluptuous on apt.wikimedia.org to 0.8.2-1wmf1
11:39 hashar: Upgraded Zuul to wmf-deploy-20140416-3 (bring in a84f0e4 - "Make queue processing more efficient" which was much needed)
11:29 hashar: upgraded Zuul to wmf-deploy-20140416-2
11:15 mutante: virt5-11 removing from icinga
11:03 mutante: virt5-11 revoked puppet certs and salt keys
10:56 mutante: stopping puppet on virt5-11
10:47 hashar: Upgraded Zuul on gallium to wmf-deploy-20140416 (depends on python-voluptuous 0.7+ , Alexandros packaged 0.8.2 which I manually installed to validate).
09:26 mutante: disabling mw1163 in pybal
07:03 mutante: zirconium - upgrading apache2, php5 packages
06:07 springle: stop mysqld on db38 (x1) for decom
03:46 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Apr 16 03:46:23 UTC 2014 (duration 46m 22s)
02:55 logmsgbot: LocalisationUpdate completed (1.23wmf22) at 2014-04-16 02:55:28+00:00
02:28 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-16 02:28:01+00:00
00:43 K4-713: updated listener credentials on thulium

April 15

23:25 logmsgbot: mwalker synchronized wmf-config/abusefilter.php '126168 more abuse filter configuration fun'
23:21 logmsgbot: mwalker Finished scap: Configuration change 126163 and MultimediaViewer 126158 (duration: 02m 15s)
23:19 logmsgbot: mwalker Started scap: Configuration change 126163 and MultimediaViewer 126158
23:08 logmsgbot: mwalker Finished scap: Configuration changes, 113656, 121834, 126065 (duration: 03m 11s)
23:05 logmsgbot: mwalker Started scap: Configuration changes, 113656, 121834, 126065
23:01 hashar: restarting Zuul to clear leaked file descriptor (know issue, fixed upstream)
22:12 awight: crm updated from e3f2859 to 7dafce5
21:51 manybubbles: restarting elastic1009 again
21:39 hashar: jenkins /var/lib/git cleaned up on gallium
21:16 manybubbles: restarting elastic1009 to test performance changes. cluster will go yellow for a few minutes. might go red (wikitech is busted)
21:15 hashar: Jenkins is processing jobs again
21:14 hashar: cleared /tmp/ on integration-slave1002 (filled up by hhvm job, known issue, bug filled already)
21:12 hashar: Zuul locked again :/ Unpooling and repooling Jenkins slaves.
19:50 RoanKattouw: Restarting stuck Jenkins
19:31 manybubbles: setting refresh interval on elasticsearch indexes to 30s to test effect on load
19:24 logmsgbot: reedy synchronized wmf-config/
19:20 logmsgbot: reedy synchronized php-1.23wmf22/includes/PrefixSearch.php 'I82b5ca65864099c180d915055c43e6839bd4f4a2'
19:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikisources back to 1.23wmf22
19:07 ottomata: reinstalling elastic1010
19:07 logmsgbot: reedy synchronized php-1.23wmf22/extensions/ProofreadPage
18:41 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikisources back to 1.23wmf21 due to ProofreadPage fatal
18:36 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non Wikipedias to 1.23wmf22
17:09 paravoid: stopped pybal on lvs1005
17:06 cmjohnson1: fixing lvs1005 eth1 cable
16:56 cmjohnson1: mw1057 replacing ethernet cable
16:50 manybubbles: raised "new generation" size on elastic1009 to test a performance theory
16:50 cmjohnson1: mw1093 replacing ethernet cable
16:40 cmjohnson1: replacing eth cable on mw1193
16:31 hashar: ... all Jenkins jobs are using /srv/ssd/gerrit instead
16:30 hashar: gallium had two Gerrit replications streams, one of them got removed 122419 thus deleting the target directories under /var/lib/git
16:22 cmjohnson1: shutting down mw1163 to replace DIMM
16:18 cmjohnson1: swapping bad disk slot 4 on dataset1001
16:13 paravoid: moving ms-fe3xxx/ms-be3xxx to private1-esams
16:05 ottomata: reinstalling elastic1009
15:21 logmsgbot: anomie synchronized php-1.23wmf21/extensions/Flow 'SWAT: Flow: Prevent logspam on enwiki 125930'
15:13 logmsgbot: anomie synchronized php-1.23wmf21/extensions/Flow 'SWAT: Flow: Prevent logspam on enwiki 125930'
15:02 mutante: DNS update - removing Tampa service IPs
13:51 hashar: Jenkins compressing console logs of builds. On gallium as user jenkins : find /var/lib/jenkins/jobs -wholename '*/builds/*/log' -type f -exec gzip --best {} \;
13:42 hashar: Command executed (as gerritslave user): find /srv/ssd/gerrit -type d -name '*.git' -exec bash -c 'echo; date; cd {}; echo; pwd; echo; git repack -ad; date;' \;
13:41 hashar: Repacking Gerrit replicated repositories on lanthanum and gallium (both under /srv/ssd/gerrit/ )
13:13 andrewbogott: shutdown and decommissioned virt12
12:19 paravoid: adding ms-be101[345] to Swift eqiad's rings, at 33% weight; old rings kept at ms-fe1001:~/swift-2014-04-14
11:30 mutante: DNS update - removed dbdump.pmtpa.wmnet
11:26 mutante: DNS update - remove db64,db65,db66,db67,db70
10:55 mutante: db64,db67 - powerdown via mgmt
10:51 mutante: db65,db66 - shutdown
10:07 mutante: db70 - powerdown via mgmt
09:47 mutante: db64-67 - puppetstoredconfigclean.rb db${db}.pmtpa.wmnet ; puppetca --clean db${db}.pmtpa.wmnet ; salt-key -d db${db}.pmtpa.wmnet
07:02 springle: shutdown db67 for decom. analytics data is backed up on dbstore1002
06:47 springle: moving pmtpa m1 and x1 slaves to db73 and db69 on 12th floor
03:25 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Apr 15 03:25:52 UTC 2014 (duration 25m 51s)
02:42 logmsgbot: LocalisationUpdate completed (1.23wmf22) at 2014-04-15 02:42:48+00:00
02:22 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-15 02:22:54+00:00
00:09 gwicke: enabled wikinews family in Parsoid with temporary live patch to un-break VE deploy

April 14

23:43 logmsgbot: ori Finished scap: (no message) (duration: 04m 31s)
23:39 ori: scap: php-1.23wmf22/extensions/VisualEditor 2b0979f...0652ad2 (I12e5c9751)
23:38 logmsgbot: ori Started scap: (no message)
23:17 logmsgbot: ori synchronized php-1.23wmf22/skins/vector/variables.less 'Ibcdaff017: Revert body font stack to be just sans-serif'
23:15 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I22f25730d: Enable VisualEditor for opt-in on Meta (2/2)'
23:15 logmsgbot: ori synchronized visualeditor.dblist 'I22f25730d: Enable VisualEditor for opt-in on Meta (1/2)'
23:14 logmsgbot: ori updated /a/common to I22f25730d: Enable VisualEditor for opt-in on Meta
23:12 logmsgbot: ori synchronized visualeditor.dblist 'I59f5a6e0b: Enable VisualEditor on French Wikinews'
23:12 logmsgbot: ori updated /a/common to I59f5a6e0b: Enable VisualEditor on French Wikinews
22:56 logmsgbot: aaron synchronized wmf-config/PoolCounterSettings-eqiad.php 'Limit large (djvu) file downloads for thumbnails'
20:37 mwalker: updating payments wiki for worldpay currencies (from af35b7b to 8a93c17)
20:13 subbu: deployed Parsoid fba548cbf (deploy repo sha d0e12ddf)
17:47 paravoid: fixing /e/n/interfaces for static configuration: gadolinium hafnium labsdb1001 labsdb1002 labsdb1003 labstore1001 searchidx1001 ssl1005 ssl1006 ssl1009 virt1001 ytterbium
17:37 paravoid: fixing /e/n/interfaces for static configuration for cp40xx, lvs40xx
17:14 mutante: brewster - power down, could not revive due to disk or SATA controller fail
16:57 ottomata1: shutting down elastic1006 for reinstall
16:45 mutante: powering brewster back on
16:40 paravoid: powering up brewster
16:13 mutante: deleted old svn apache config on formey, started apache
15:22 paravoid: restarting virt0's salt-master, glance-api, glance-registry, keystone, nova-scheduler
15:11 logmsgbot: manybubbles synchronized wmf-config/CirrusSearch-common.php 'SWAT Cirrus update to improve performance'
15:09 logmsgbot: manybubbles synchronized php-1.23wmf21/extensions/CirrusSearch/ 'SWAT deploy to improve performance'
14:48 paravoid: upgrading all snapshot* hosts
14:38 paravoid: upgrading all packages & staggered restart of all of swift (ms-fe/ms-be)
13:22 logmsgbot: reedy synchronized php-1.23wmf22/includes/api/ApiFeedRecentChanges.php 'I268d0a53067738ba96bee74c593358b0b28cc083'
13:22 logmsgbot: reedy synchronized php-1.23wmf21/includes/api/ApiFeedRecentChanges.php 'I268d0a53067738ba96bee74c593358b0b28cc083'
13:15 paravoid: staggered upgrades for all pending updates on all mw* boxes & restarting apaches/other core services
11:08 mutante: brewster - shut down
10:49 logmsgbot: reedy synchronized wmf-config/interwiki.cdb 'Updating interwiki cache'
10:05 apergos: had to toss extensions/Elastica on virt1000 and run git submodule update --init --recursive seems to be working now
09:26 mutante: deleting huge pybal log on lvs3001
09:01 mutante: brewster - stop lighttpd,bacula-fd,haproxy,dhcp3-server,rsync,nrpe,salt
07:54 mutante: brewster - disabling puppet agent, removed from site.pp, revoke puppet cert
03:23 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Apr 14 03:22:58 UTC 2014 (duration 22m 57s)
02:42 logmsgbot: LocalisationUpdate completed (1.23wmf22) at 2014-04-14 02:42:05+00:00
02:23 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-14 02:22:58+00:00

April 13

03:19 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Apr 13 03:19:45 UTC 2014 (duration 19m 44s)
02:39 logmsgbot: LocalisationUpdate completed (1.23wmf22) at 2014-04-13 02:39:11+00:00
02:20 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-13 02:20:49+00:00

April 12

05:03 logmsgbot: ori updated /a/common to I5f900190c: Replace $channel with $variant; make it Beta-only
03:21 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Apr 12 03:21:43 UTC 2014 (duration 21m 42s)
02:42 logmsgbot: LocalisationUpdate completed (1.23wmf22) at 2014-04-12 02:42:07+00:00
02:23 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-12 02:23:02+00:00

April 11

23:55 RoanKattouw: Restarting stuck Jenkins
23:35 K4-713: synchronized payments to af35b7b
23:25 K4-713: synchronized payments to b321163
19:50 ottomata: upgraded wikitech to MediaWiki 1.23wmf22, applied security patch
18:19 ottomata: rebooting elastic1003
18:14 Krinkle: git-deploy: Deploying integration/slave-scripts I38b90e8c08d7cb
18:08 Krinkle: git-deploy: Deploying integration/slave-scripts I04d8e308daedb3ccb8
17:41 Krinkle: git-deploy: Deploying integration/slave-scripts 'Ia9ee438fa2675170'
14:27 ottomata: reinstalling elastic1005
04:33 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Apr 11 04:33:40 UTC 2014 (duration 33m 39s)
03:47 logmsgbot: LocalisationUpdate completed (1.23wmf22) at 2014-04-11 03:47:01+00:00
02:41 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-11 02:41:33+00:00
02:29 ori_: graphite: carbon instance 'f' saturates a cpu core. it's the instance that mediawiki profiling data gets hashed to. collector should probably emit to statsd and have statsd compute per-minute rollups
00:06 marktraceur: leaving MultimediaViewer slightly broken on enwiki based on the fact that logged-in users seem mostly unaffected and other wikis aren't seeing issues, will investigate more tomorrow and fix on Monday

April 10

23:54 bd808: Enabled beta update Jenkins job (https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/)
23:37 logmsgbot: mwalker Finished scap: Attempting to regenerate i18n keys for multimediaviewer (duration: 03m 33s)
23:34 logmsgbot: mwalker Started scap: Attempting to regenerate i18n keys for multimediaviewer
23:16 logmsgbot: mwalker synchronized wmf-config/filebackend.php
23:09 logmsgbot: mwalker synchronized wmf-config/InitialiseSettings.php 'touched to see if that pushes changes to FileBackend.php'
23:03 mwalker: sync-common for 125340 and 125335
22:53 logmsgbot: krinkle synchronized php-1.23wmf21/extensions/VisualEditor/modules/ve-mw/ui/tools/ 'touch *.js'
22:35 logmsgbot: krinkle synchronized php-1.23wmf21/extensions/VisualEditor/modules/ve-mw/ui/tools/ve.ui.MWReferenceDialogTool.js 'touch'
22:12 logmsgbot: krinkle synchronized php-1.23wmf21/extensions/VisualEditor/lib/ve/modules/ve/ui/ve.ui.Toolbar.js 'touch'
22:10 logmsgbot: krinkle synchronized php-1.23wmf21/extensions/VisualEditor/lib/ve/lib/oojs-ui/oojs-ui.js 'touch'
22:10 logmsgbot: krinkle synchronized php-1.23wmf21/resources/startup.js 'touch'
22:10 logmsgbot: krinkle synchronized php-1.23wmf21/resources/oojs-ui/oojs-ui.js 'touch'
21:50 Krinkle: VisualEditor throws uncaught error on load for 1.23wmf21 wikis (bug 63791)
21:15 bd808: Disabled beta update Jenkins job (https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/) so that scap testing can happen in beta.
19:29 ottomata: reinstalling elastic1004
19:19 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'I5501078cee871fb9df03e085547b7a047ef5bd7e'
19:16 logmsgbot: ori synchronized wmf-config/InitialiseSettings-labs.php 'Ia79b1b848: Work around bug 63780 by specifying a siteParamsCallback'
19:15 logmsgbot: ori updated /a/common to Ia79b1b848: Work around bug 63780 by specifying a siteParamsCallback
18:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'touch'
18:43 logmsgbot: reedy synchronized database lists files: Enable MediaViewer on mediawikiwiki
18:42 logmsgbot: reedy synchronized docroot and w
18:41 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.23wmf22
18:37 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.23wmf21
18:33 logmsgbot: reedy synchronized php-1.23wmf21/extensions/MultimediaViewer
16:55 ottomata: shutting down elastic1003 for reinstall and reformat
16:42 logmsgbot: reedy updated /a/common to Ie72029103: Add/update symlinks
16:40 logmsgbot: reedy Finished scap: testwiki to 1.23wmf22 and build l10n cache (duration: 24m 45s)
16:15 logmsgbot: reedy Started scap: testwiki to 1.23wmf22 and build l10n cache
16:14 logmsgbot: reedy updated /a/common to I2cccebdd7: wikidatawiki back to 1.23wmf21
15:03 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikidatawiki back to 1.23wmf21...
15:00 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Apr 10 15:00:01 UTC 2014 (duration 27m 49s)
14:17 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-10 14:17:16+00:00
13:49 logmsgbot: LocalisationUpdate completed (1.23wmf20) at 2014-04-10 13:49:24+00:00
13:31 logmsgbot: reedy Finished scap: l10n cache update for wikidatawiki (duration: 19m 15s)
13:12 logmsgbot: reedy Started scap: l10n cache update for wikidatawiki
12:42 bblack: removed broken pdns_gmetric cronjob on lvs boxes
09:44 logmsgbot: ori synchronized wmf-config/CommonSettings.php 'I107179a27: Move HHVM extension blacklist below extract($globals) so it isn't simply clobbered'
09:42 logmsgbot: ori updated /a/common to I107179a27: Move HHVM extension blacklist below extract($globals) so it isn't simply clobbered
09:39 hashar: Zuul processed its backlog. Had to disconnect/reconnect the labs slaves. There is some weird bug occurring :-(
09:29 hashar: Jenkins: disabling Gearman client in https://integration.wikimedia.org/ci/configure and reenabling it
09:20 hashar: Jenkins unpooling both slave labs using the web interface and killing the Jenkins client running as jenkins-deploy . Will repool so the job can be reregistered properly [[bugzilla:63760|bug 63760]]
09:11 mutante: DNS update - removing ms6
09:04 hashar: Jenkins bunch of jobs are not being triggered properly. Taking traces.
08:55 mutante: ms6 - shutdown -h now
08:42 mutante: forcing Bugzilla logout for all users
08:19 logmsgbot: aude synchronized php-1.23wmf20/extensions/Wikidata
08:09 logmsgbot: aude synchronized php-1.23wmf20/extensions/Wikidata
07:57 logmsgbot: aude rebuilt wikiversions.cdb and synchronized wikiversions files: Rebuild wikiversions and put wikidata on 1.23wmf20
07:53 logmsgbot: aude synchronized wikiversions.json 'Put Wikidata back on 1.23wmf20, due to localisation cache issues'
07:21 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Apr 10 07:21:16 UTC 2014 (duration 7m 21s)
06:46 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-10 06:45:59+00:00
06:27 logmsgbot: LocalisationUpdate completed (1.23wmf20) at 2014-04-10 06:27:30+00:00
06:27 logmsgbot: ori synchronized wmf-config/CommonSettings.php 'I20bbe05cc: Avoid using bits on beta-hhvm.wmflabs.org'
06:26 logmsgbot: ori updated /a/common to I20bbe05cc: Avoid using bits on beta-hhvm.wmflabs.org
06:15 ori: Some interface messages are missing on wikidata.org. Started a manual l10nupdate.
04:11 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Apr 10 04:11:47 UTC 2014 (duration 11m 46s)
03:19 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-10 03:19:18+00:00
03:01 logmsgbot: ori synchronized multiversion/MWMultiVersion.php 'Ibdbac982b: Update multiversion regexp for *.beta-hhvm.wmflabs.org'
03:01 logmsgbot: ori updated /a/common to Ibdbac982b: Update multiversion regexp for *.beta-hhvm.wmflabs.org
02:22 logmsgbot: LocalisationUpdate completed (1.23wmf20) at 2014-04-10 02:22:11+00:00
01:49 logmsgbot: ori synchronized wmf-config/InitialiseSettings-labs.php 'I697f7e4a6: Use to branch on interpreter'
01:48 logmsgbot: ori synchronized wmf-config/CommonSettings.php 'I697f7e4a6: Use to branch on interpreter'
01:48 logmsgbot: ori updated /a/common to I697f7e4a6: Use '$channel' to branch on interpreter
01:08 K4-713: updated payments to e1d00b61a703
01:06 Krinkle: git-deploy: Deploying integration/slave-scripts If2539c
01:05 Krinkle: Undid local patch to "grunt-lib-phantomjs/phantomjs/main.js" (for bug 63579) in "/srv/deployment/integration/slave-scripts" on gallium
00:20 logmsgbot: yurik synchronized wmf-config/InitialiseSettings.php
00:08 awight: updated crm from e726e42 to e3f2859
00:06 K4-713: updated payments to 70dce8f4bc7

April 9

23:36 logmsgbot: ebernhardson synchronized php-1.23wmf21/extensions/Math/modules/VisualEditor/ve.ui.MWMathInspectorTool.js 'Update Math VE tool to use a command in 1.23wmf21'
23:32 logmsgbot: ebernhardson synchronized wmf-config/CommonSettings.php 'Update Flow cache version'
23:22 logmsgbot: ebernhardson synchronized php-1.23wmf21/extensions/Flow 'Backport fix DB-to-cache pipeline for 1.23wmd21'
23:05 logmsgbot: ebernhardson synchronized wmf-config/InitialiseSettings-labs.php 'Enable math VE plugin on labs'
23:04 Krinkle: Jenkins and Zuul are back up. Queues have not been preserved.
23:01 ^d: gerrit: reloaded bugzilla plugin to force it to log back in
23:00 Krinkle: Restarting Jenkins because I have no clue what is going on and have no time to investigate yet another random clogging of all jobs. Restart ought to fix it.
22:54 Krinkle: Zuul has lots of queued jobs for npm slaves, but neither Jenkins nor integration-slave1001.eqiad.wmflabs and 1002 themselves have anything queued. They're idle, responsive and waiting for jobs.
22:47 Krinkle: Jenkins slaves in labs seem to be down. Zuul is stacking up jobs for hasNpm nodes (integration slaves in labs). Both slaves have 7/7 executors idle.
22:33 hoo: Logged out all Bugzilla users by deleting all session cookie data from mysql
19:15 logmsgbot: csteipp synchronized php-1.23wmf21/extensions/CentralAuth/maintenance
19:10 logmsgbot: yurik synchronized wmf-config/InitialiseSettings.php
17:38 logmsgbot: yurik synchronized wmf-config/InitialiseSettings.php
17:22 manybubbles: regenerating Elasticsearch index from mediawiki for testwiki to soak up geo changes.
16:48 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/124880'
16:41 manybubbles: reindexed testwiki to soak up geo changes
16:37 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/124876'
16:32 logmsgbot: maxsem synchronized php-1.23wmf21/extensions/GeoData
16:28 manybubbles: fiddling with Elasticsearch cluster balancing options trying to get enwiki better balanced
16:17 logmsgbot: aude synchronized php-1.23wmf21/extensions/Wikidata 'Switch Wikidata back to previous version of Wikibase'
15:52 mutante: ms6 - revoke puppet cert, salt key, remove from icinga
15:02 ottomata: stopped puppet on emery to test sqstat on analytics1003
14:48 ottomata: disabling puppet to test sqstat on analytics1003
14:14 RobH: otrs back up, live hacked apache change, now working permanent puppet change (puppet is disabled on iodine at present)
14:02 RobH: yes, otrs is totally ssl borked, robh is working on it
14:00 mutante: adding filippo to ops/wmf LDAP groups
13:58 RobH: updating otrs cert
09:19 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '[] = 'musees.cg70.fr'; 124754 [[bugzilla:63449|bug 63449]]'
08:39 hashar: Gerrit Letting JenkinsBot submit changes on apps/android/*
03:33 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Apr 9 03:33:25 UTC 2014 (duration 33m 24s)
02:43 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-09 02:43:52+00:00
02:19 logmsgbot: LocalisationUpdate completed (1.23wmf20) at 2014-04-09 02:19:25+00:00
01:25 Krinkle: Bug 63579 is still happening occasionally. Leaving patch on gallium in place for now.
01:09 ori: Debugging uWSGI init scripts on tungsten; expect some Graphite / Gdash flapping.
00:15 ori: graphite webapp 502 caused by uwsgi's init script not restarting the service correctly
00:07 Krinkle: graphite.wikimedia.org (e.g. https://graphite.wikimedia.org/render/?) is serving 502 Bad Gateway, ori is investigating
00:04 Krinkle: To investigate bug 63579, manually patched "grunt-lib-phantomjs/phantomjs/main.js" in "/srv/deployment/integration/slave-scripts" on gallium

April 8

23:34 logmsgbot: mwalker synchronized php-1.23wmf21/extensions/MultimediaViewer/ 'Updating MultimediaViewer for 124510'
23:08 logmsgbot: csteipp synchronized php-1.23wmf21/extensions/CentralAuth/maintenance 'Push maintenance script for token reset'
21:21 logmsgbot: bd808 Purged l10n cache for 1.23wmf19
21:20 logmsgbot: bd808 Purged l10n cache for 1.23wmf18
21:12 logmsgbot: bd808 rebuilt wikiversions.cdb and synchronized wikiversions files: group1 to 1.23wmf21
19:58 manybubbles: finished upgrading to Elasticsearch 1.1.0. The process went well with no issues other then some knocking out search in labs 3 times for 30 seconds a piece. And logging lots of nasty warnings to irc. I've started to the process to fix search in labs so it won't happen again.
19:56 manybubbles: upgraded all elasticsearch servers except elastic1008. that is coming now.
18:45 logmsgbot: ori synchronized wmf-config/InitialiseSettings.php 'I4b18e4ce8: Change wgServer and wgCanonicalServer for arbcom wikis'
18:45 logmsgbot: ori updated /a/common to I4b18e4ce8: Change wgServer and wgCanonicalServer for arbcom wikis
18:13 logmsgbot: bd808 Finished scap: group0 wikis to 1.23wmf21 (with patch for bug 63659) (duration: 03m 18s)
18:10 logmsgbot: bd808 Started scap: group0 wikis to 1.23wmf21 (with patch for bug 63659)
18:01 logmsgbot: hoo synchronized wmf-config/InitialiseSettings.php 'Touch to clear config. cache'
17:56 hoo: changed the Wikidata wb_changes_dispatch position of all wikiquote wikis to 118158153
17:37 logmsgbot: hoo synchronized php-1.23wmf20/extensions/Wikidata/extensions/Wikibase/lib/resources/wikibase.Site.js 'touch'
17:37 logmsgbot: aude synchronized wmf-config/Wikibase.php 'bump wgCacheEpoch for wikidata after enabling wikiquote site links'
17:35 ottomata: restarted gmetad on nickel to fix ganglia
17:29 logmsgbot: aude synchronized wikidataclient.dblist 'Enable Wikibase on Wikiquote'
17:29 logmsgbot: aude synchronized wmf-config 'config changes to enable Wikibase on Wikiquote'
17:22 logmsgbot: aude synchronized wmf-config/CirrusSearch-labs.php 'config change for beta, to enable highlighting'
17:16 manybubbles: finished upgrading elastic1001-1006. starting on 1007. yay progress.
17:03 logmsgbot: aude synchronized php-1.23wmf20/extensions/Wikidata 'Update Wikidata build, to allow populating sites table on wikiquote'
16:31 aude: added sites and site_identifiers core tables on wikiquote
16:28 hashar: Jenkins: killed jenkins-slave java process on gallium and repooled gallium slave. It was no more registered in Zuul :-/
14:32 manybubbles: no harm done, just lost time
14:32 manybubbles: woops, just restarted elastic1002. silly me
14:31 manybubbles: upgrading elastic1001
13:54 manybubbles: they'll pick it up during the rolling restart today to upgrade to 1.1.0
13:53 manybubbles: synced first Elasticsearch plugin to production Elasticsearch servers
13:46 RobH: upgraded libssl on holmium
13:39 Jeff_Green: update & reboot tellurium
13:39 RobH: replacing the blog cert, if holmium crashes I didn't do it correctly.
13:37 mutante: restarting gitblit
12:56 logmsgbot: reedy updated /a/common to Id15ddc665: Revert "Group0 wikis to 1.23wmf21"
10:21 Jeff_Green: update & reboot barium
10:15 Jeff_Green: update & reboot samarium
07:47 _joe|away: restarted nginx on cp1044 and cp1043
05:47 apergos: shot many old apache processes running as stats user from 2013, on stat1001 (restarting apache runs it as www-data user)
05:39 apergos: restarted apache on fenari magnesium yterrbium antimony
05:31 _joe_: upgraded openssl on cp10* and cp30* servers as well
04:46 Tim: on dataset1001: upgraded libssl and restarted lighttpd
04:43 Tim: restarted apache on the above list, failed on labs-ns1, virt1000, ytterbium
04:41 Tim: upgraded libssl on zirconium.wikimedia.org,neon.wikimedia.org,netmon1001.wikimedia.org,iodine.wikimedia.org,ytterbium.wikimedia.org,gerrit.wikimedia.org,virt1000.wikimedia.org,labs-ns1.wikimedia.org,stat1001.wikimedia.org
04:38 Ryan_Lane: upgrading libssl on virt0
04:37 Ryan_Lane: upgrading libssl on virt1000
04:15 Tim: also upgraded libssl on cp4001-4019. Restarted nginx on these servers and also the previous list.
04:03 Tim: upgrading libssl on ssl1001,ssl1002,ssl1003,ssl1004,ssl1005,ssl1006,ssl1007,ssl1008,ssl1009,ssl3001.esams.wikimedia.org,ssl3002.esams.wikimedia.org,ssl3003.esams.wikimedia.org
03:11 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Apr 8 03:11:04 UTC 2014 (duration 11m 3s)
02:34 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-08 02:34:56+00:00
02:16 logmsgbot: LocalisationUpdate completed (1.23wmf20) at 2014-04-08 02:15:58+00:00
01:06 logmsgbot: bd808 Finished scap: revert group0 to 1.23wmf21 (testwiki still on 1.23wmf21) (duration: 09m 54s)
00:56 logmsgbot: bd808 Started scap: revert group0 to 1.23wmf21 (testwiki still on 1.23wmf21)
00:54 logmsgbot: bd808 scap aborted: group0 to 1.23wmf21 (testing python change for mwversionsinuse) (again) (duration: 00m 25s)
00:54 logmsgbot: bd808 Started scap: group0 to 1.23wmf21 (testing python change for mwversionsinuse) (again)
00:53 logmsgbot: bd808 scap aborted: group0 to 1.23wmf21 (testing python change for mwversionsinuse) (duration: 02m 57s)
00:50 logmsgbot: bd808 Started scap: group0 to 1.23wmf21 (testing python change for mwversionsinuse)
00:25 logmsgbot: catrope synchronized php-1.23wmf21/extensions/VisualEditor 'it helps if you run git submodule update first'
00:24 logmsgbot: catrope synchronized php-1.23wmf20/extensions/VisualEditor 'it helps if you run git submodule update first'

April 7

23:58 logmsgbot: catrope synchronized php-1.23wmf20/extensions/VisualEditor/ 'VisualEditor bug fixes'
23:57 logmsgbot: catrope synchronized php-1.23wmf20/skins/vector/variables.less 'Remove troublesome fonts from font stack'
23:50 logmsgbot: catrope synchronized php-1.23wmf21/resources/oojs-ui/ 'Update OOJS-UI for bug fixes'
23:50 logmsgbot: catrope synchronized php-1.23wmf21/extensions/VisualEditor/ 'VisualEditor bug fixes'
23:49 logmsgbot: catrope synchronized php-1.23wmf21/skins/vector/variables.less 'Remove troublesome fonts from font stack'
23:45 logmsgbot: catrope synchronized php-1.23wmf21/extensions/VisualEditor/ 'VisualEditor bug fixes'
23:42 logmsgbot: catrope synchronized php-1.23wmf21/resources/oojs-ui/ 'Update OOJS-UI for bug fixes'
23:42 logmsgbot: catrope synchronized php-1.23wmf21/skins/vector/variables.less 'Remove troublesome fonts from font stack'
23:17 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'SWAT changes: other projects bar on frwikisource, import sources'
23:06 logmsgbot: bd808 Finished scap: test2wiki to 1.23wmf20 (duration: 16m 48s)
22:49 logmsgbot: bd808 Started scap: test2wiki to 1.23wmf20
22:38 logmsgbot: bd808 Finished scap: test2wiki to 1.23wmf21 (duration: 12m 07s)
22:26 logmsgbot: bd808 Started scap: test2wiki to 1.23wmf21
22:12 logmsgbot: bd808 Finished scap: Testing 1.23wmf21 l10n changes (duration: 01m 31s)
22:10 logmsgbot: bd808 Started scap: Testing 1.23wmf21 l10n changes
22:07 logmsgbot: bd808 Finished scap: Testing 1.23wmf21 l10n changes (duration: 03m 49s)
22:04 logmsgbot: bd808 Started scap: Testing 1.23wmf21 l10n changes
21:13 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Apr 7 21:13:14 UTC 2014 (duration 1m 59s)
20:40 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-07 20:40:02+00:00
20:23 logmsgbot: LocalisationUpdate completed (1.23wmf20) at 2014-04-07 20:23:14+00:00
19:08 ottomata: temporatily disabling puppet on analytics 1009, 1010, 1019, 1020 to bring up new journalnodes
18:39 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Apr 7 18:39:21 UTC 2014 (duration 15m 30s)
18:23 logmsgbot: aaron synchronized wmf-config/PoolCounterSettings-eqiad.php 'Added "downloadtiff" pool counter config'
18:13 AaronSchulz: shwiki queue finished emptying out in staggered loop on terbium
18:03 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-07 18:03:48+00:00
17:36 logmsgbot: LocalisationUpdate completed (1.23wmf20) at 2014-04-07 17:36:02+00:00
17:24 bd808: Manually running l10nupdate with new --verbose flag to capture log output
14:34 MaxSem: Rebuilding GeoData index
14:09 hashar: Jenkins cleared swap on gallium (swapoff -a && swapon -a). Makes ganglia graph nicer :D
13:15 apergos: reenabled puppet on dataset2, testing done
12:23 apergos: disabled puppet on dataset2, testing
11:19 logmsgbot: reedy Finished scap: because we're scappy... (rebuilding l10n cache for 1.23wmf21 (duration: 18m 04s)
11:01 logmsgbot: reedy Started scap: because we're scappy... (rebuilding l10n cache for 1.23wmf21
10:45 hashar: integration Getting PHP Composer installed on labs slaves. 124305
09:21 paravoid: reactivating peerings with HE, issues reportedly resolved
09:04 hashar: restarted Zuul
08:54 hashar: gallium killed console-kit-daemon process which was eating a lot of memory
08:42 hashar: Restarting Jenkins, out of Java heap space. Something is leaking memory
08:41 hashar: Jenkins being broken for some reason AGAIN !
05:04 ori: Zuul is stuck: <http://i.imgur.com/o5ghCam.jpg> (617kb image)
02:56 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Apr 7 02:56:12 UTC 2014 (duration 56m 11s)
02:20 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-07 02:20:10+00:00
02:13 logmsgbot: LocalisationUpdate completed (1.23wmf20) at 2014-04-07 02:13:42+00:00

April 6

02:53 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Apr 6 02:53:13 UTC 2014 (duration 53m 12s)
02:18 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-06 02:18:43+00:00
02:13 logmsgbot: LocalisationUpdate completed (1.23wmf20) at 2014-04-06 02:12:57+00:00
01:32 jamesofu_: sugar down for move to labs

April 5

04:15 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Apr 5 04:15:00 UTC 2014 (duration 50m 39s)
03:56 logmsgbot: andrew rebuilt wikiversions.cdb and synchronized wikiversions files: Revert mw.org, test2wiki and testwikidatawiki to 1.23wmf20 due to localisation issue
03:51 Andrew: Reverting mw.org, test2 and test.wikidata back to 1.23wmf20
03:41 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-05 03:41:36+00:00
03:36 logmsgbot: LocalisationUpdate completed (1.23wmf20) at 2014-04-05 03:36:04+00:00
03:23 Andrew: Actually, going to rerun l10nupdate first just to check.
03:22 Andrew: Going to revert deployment of 1.23wmf21 again - still broken
03:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Apr 5 03:08:33 UTC 2014 (duration 8m 32s)
02:34 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-05 02:34:54+00:00
02:14 logmsgbot: LocalisationUpdate completed (1.23wmf20) at 2014-04-05 02:14:07+00:00

April 4

21:34 logmsgbot: bd808 Finished scap: Group0 to 1.23wmf21 (again) (duration: 14m 35s)
21:19 logmsgbot: bd808 Started scap: Group0 to 1.23wmf21 (again)
19:28 hashar: Jenkins: unpooled slave agent on lanthanum, killed it the java agent on it and repooled it.
19:22 hashar: Jenkins is processing jobs again. Queue unchanged so it will resume everything
19:16 hashar: restarting Jenkins
19:07 hashar: Jenkins un pooling gallium slave
19:05 hashar: Zuul / Jenkins stalled again.
18:43 csteipp: redeployed updated patch for bug63251 to fix a reported bug
16:10 _joe_: restarting gitlbit, for the last time today
15:06 _joe_: restarting gitblit as it has eaten up all of its ram again and is trashing cpu
12:32 mutante: hume - shutting down
12:06 mutante: hume - disable puppet/salt/monitoring
11:13 mutante: restarting gitblit with new option to use incremental GC in an attempt to fix timeouts caused by GC eating CPU
08:07 paravoid: deactivating cr1-eqiad<->HE peerings, translantic par2<->ash1 is congested
07:25 mutante: restarting gitblit
05:45 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Apr 4 05:45:07 UTC 2014 (duration 18m 25s)
04:56 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-04 04:56:06+00:00
04:45 logmsgbot: LocalisationUpdate completed (1.23wmf20) at 2014-04-04 04:45:01+00:00
04:20 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: unbreak test2.wp and test.wikidata as well
04:17 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: mw.org back to 1.23wmf20
03:43 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Apr 4 03:43:03 UTC 2014 (duration 43m 2s)
03:28 ori: Interface messages are missing on group0 / 1.23wmf21 wikis (mediawikiwiki, testwiki, test2wiki, and testwikidata)
02:50 logmsgbot: LocalisationUpdate completed (1.23wmf21) at 2014-04-04 02:50:26+00:00
02:24 logmsgbot: LocalisationUpdate completed (1.23wmf20) at 2014-04-04 02:24:51+00:00
01:08 logmsgbot: krinkle synchronized php-1.23wmf21/resources 'I6e93d9ab0e4a926c09c'

April 3

22:00 logmsgbot: demon synchronized wmf-config/CirrusSearch-production.php 'lowering cache time, for testing'
21:55 logmsgbot: demon updated /a/common/php-1.23wmf20 to Ic853ebff4: Cherry-pick I550eb4b0a8fa18344e8b0de3ec85d61c2122ffb8
21:54 logmsgbot: demon synchronized php-1.23wmf20/extensions/CirrusSearch 'Cirrus back to master again'
21:50 logmsgbot: ori synchronized multiversion/updateBitsBranchPointers 'updateBitsBranchPointers: get rid of 'static-stable' branch link'
21:50 logmsgbot: ori updated /a/common to Ic1602c045: updateBitsBranchPointers: get rid of 'static-stable' branch link
21:46 logmsgbot: demon synchronized php-1.23wmf20/extensions/CirrusSearch 'Rolling back to 1.23wmf20 branch point from master'
21:38 logmsgbot: demon synchronized php-1.23wmf20/extensions/CirrusSearch 'Updating Cirrus to master'
21:33 logmsgbot: demon synchronized wmf-config/CirrusSearch-production.php 'italian wikis getting interwiki search. they're my favorite beta testers'
19:23 logmsgbot: reedy synchronized docroot and w
19:21 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.23wmf21
19:17 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias actually to 1.23wmf20
19:15 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.23wmf20
19:09 logmsgbot: reedy Finished scap: testwiki to 1.23wmf21 and build l10n cache (duration: 38m 23s)
18:30 logmsgbot: reedy Started scap: testwiki to 1.23wmf21 and build l10n cache
18:23 logmsgbot: reedy updated /a/common to I835c2b1d5: Depool. See RT 7191.
11:10 paravoid: IPv4 eqiad<->esams private link also elevated by ~15ms but no packet loss observed
11:09 paravoid: affects both IPv6 transit at esams (slowdowns) as well as IPv6 eqiad<->esams
11:08 paravoid: deactivating cr1-esams<->HE peering, latency > 160ms, over at 200ms (congestion?); back to 84ms now;
10:51 akosiaris: temporarily stopped squid on brewster
10:26 hashar: Jenkins job mediawiki-core-phpunit-hhvm is back around thanks to 123573
06:28 paravoid: powercycling ms-be1003, unresponsive, no console output
04:43 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'return upgraded DB slaves to normal load'
04:11 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's6 repool db1015, warm up'
04:04 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's6 depool db1015 for upgrade'
04:03 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's5 repool db1037, warm up'
03:53 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's5 depool db1037 for upgrade'
03:53 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Apr 3 03:53:18 UTC 2014 (duration 53m 16s)
03:34 springle: db1020 raid controller dimm ecc errors
03:14 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's4 depool db1020 for upgrade'
03:12 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's3 repool db1019, warm up'
02:57 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's3 depool db1019 for upgrade'
02:56 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's2 repool db1060, warm up'
02:48 logmsgbot: LocalisationUpdate completed (1.23wmf20) at 2014-04-03 02:48:01+00:00
02:47 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's2 depool db1060 for upgrade'
02:45 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's1 repool db1061, warm up'
02:35 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's1 depool db1061 for upgrade'
02:24 logmsgbot: LocalisationUpdate completed (1.23wmf19) at 2014-04-03 02:24:07+00:00

April 2

23:47 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Bumped wgJobBackoffThrottling for htmlCacheUpdate to 15'
23:47 mwalker: ... deploy was for mobile frontend 123454
23:46 logmsgbot: mwalker synchronized php-1.23wmf20/extensions/MobileFrontend 'SWAT deploy for MaxSem'
20:23 subbu: deployed Parsoid 33471172 with deploy repo sha 5c620e54
19:03 logmsgbot: ori synchronized php-1.23wmf20/extensions/WikimediaEvents 'Update WikimediaEvents for I7fdaa5524: Use simple random sampling to log deprecated usage at 1:100'
19:03 logmsgbot: ori synchronized php-1.23wmf19/extensions/WikimediaEvents 'Update WikimediaEvents for I7fdaa5524: Use simple random sampling to log deprecated usage at 1:100'
17:00 andrewbogott: fixed updating crons on wikitech-status, I think. Time will tell...
16:19 logmsgbot: manybubbles synchronized wmf-config/InitialiseSettings.php 'Lower timeout on prefix searches and make the cirrus.dblist sync I just did take effect.'
16:19 logmsgbot: manybubbles synchronized cirrus.dblist 'Cirrus as primary for most of group1'
16:14 akosiaris: banned tools-exec-03.eqiad.wmflabs. using manual iptables on ytterbium
15:20 ottomata: stopping puppet on stat1
14:27 hashar: Jenkins applying label contintLabsSlave on slaves in labs used for ci (integration-slave1001 and 1002)
14:15 hashar: Jenkins deleting pmtpa slaves (they all have been shutdown and jobs got deleted)
14:00 manybubbles: tried restarting some lsearchd services (carefully) to clear out some crashing when searching for a particular query term. It caused pool queue full errors.... serves me right for trying?
11:20 mutante: running CheckUser/maintenance/purgeOldData.php on all wikis
09:42 akosiaris: rsynced brewster /srv to carbon
09:34 mutante: restarting gitblit on antimony
09:14 mutante: DNS update - removing capella
09:09 mutante: DNS update - removing ms10
05:31 logmsgbot: springle synchronized wmf-config/db-eqiad.php 'normal loads for all upraded slaves'
04:53 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's1 repool db1062, warm up'
04:45 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's1 depool db1062 for upgrade'
04:42 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's7 repool db1039, warm up'
04:27 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's7 depool db1039 for upgrade'
03:56 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's6 repool db1006, warm up'
03:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Apr 2 03:48:31 UTC 2014 (duration 48m 30s)
03:46 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's6 depool db1006 for upgrade'
03:43 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's5 repool db1045, warm up'
03:27 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's5 depool db1045 for upgrade'
03:21 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's4 repool db1059, warm up'
03:07 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's4 depool db1059 for upgrade'
03:04 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's2 repool db1063, warm up'
02:55 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's2 depool db1063 for upgrade'
02:52 logmsgbot: LocalisationUpdate completed (1.23wmf20) at 2014-04-02 02:52:48+00:00
02:29 logmsgbot: LocalisationUpdate completed (1.23wmf19) at 2014-04-02 02:29:18+00:00
02:22 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's3 repool db1027, warm up'
02:03 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's3 depool db1027 for upgrade'
01:16 logmsgbot: ori synchronized php-1.23wmf20/extensions/WikimediaEvents 'Undeployed change from earlier SWAT deploy'
01:16 logmsgbot: ori synchronized php-1.23wmf19/extensions/WikimediaEvents 'Undeployed change from earlier SWAT deploy'
01:05 logmsgbot: ori synchronized php-1.23wmf19/extensions/WikimediaEvents
01:04 logmsgbot: ori synchronized php-1.23wmf19/extensions/EventLogging
01:02 logmsgbot: ori synchronized php-1.23wmf20/extensions/WikimediaEvents
01:02 logmsgbot: ori synchronized php-1.23wmf20/extensions/EventLogging

April 1

23:48 logmsgbot: ebernhardson synchronized php-1.23wmf19/extensions/WikimediaEvents/ 'Update WikimediaEvents to master'
23:48 logmsgbot: ebernhardson synchronized php-1.23wmf19/extensions/EventLogging/ 'Update EventLogging to master'
23:47 logmsgbot: ebernhardson synchronized php-1.23wmf20/extensions/EventLogging/ 'Update EventLogging to master'
23:46 logmsgbot: ebernhardson synchronized php-1.23wmf20/extensions/WikimediaEvents/ 'Update WikimediaEvents to master'
23:32 logmsgbot: ebernhardson synchronized docroot and w
21:42 hashar: Ganglia in labs is more or less back in activity: http://ganglia.wmflabs.org/ No clue what it is graphing though
21:27 hashar: jenkins killed stuck build (5 hours+) of beta-update-databases-eqiad . Might have been blocking Jenkins build queue
19:09 Reedy: ori gracefulled mw1018, mw1050, mw1061, mw1070, mw1139, mw1179
18:45 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: zerowiki to 1.23wmf20
18:43 logmsgbot: reedy updated /a/common to If887effe5: Add zerowiki
18:43 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'touch'
18:42 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Add zerowiki
18:41 logmsgbot: reedy synchronized database lists files:
18:37 logmsgbot: reedy synchronized docroot and w
18:36 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: non wikipedias to 1.23wmf20
18:28 mutante: ms10 - shut down
18:19 mutante: ms10 - disable puppet, revoke puppet cert,salt key,icinga..
18:03 logmsgbot: ori synchronized php-1.23wmf20/includes/profiler/ProfilerSimple.php 'Iad91f1d12: Send profiled items under the correct name'
18:02 logmsgbot: ori synchronized php-1.23wmf19/includes/profiler/ProfilerSimple.php 'Iad91f1d12: Send profiled items under the correct name'
17:34 mutante: logging to eqiad wikitech after Andrew switched over
16:05 andrewbogott: switching wikitech to read-only, migrating to eqiad
15:06 logmsgbot: reedy updated /a/common to If3ca3d486: beta: adjust $wgCaptchaDirectory
15:01 hashar: Gerrit super slow again :-(
14:46 mutante: added oblivion to root-auth-keys
14:17 mutante: welcome new shell user oblivion
14:04 hashar: Gerrit flushed a few caches related to user accounts / LDAP
13:43 mutante: adding oblivion to ops and wmf LDAP groups
08:44 mutante: solr1/2 - revoke puppet certs
08:43 mutante: solr3 - delete salt key, puppet cert
03:11 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's7 db1034 full steam'
02:02 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
01:58 springle: killed research queries on db1047. email me
01:35 springle: restarted sanitarium s3 instance for additional private wikis
00:59 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's7 repool db1034, warm up'

2000s

Archive 1: 2004 Jun - 2004 Sep
Archive 2: 2004 Oct - 2004 Nov
Archive 3: 2004 Dec - 2005 Mar
Archive 4: 2005 Apr - 2005 Jul
Archive 5: 2005 Aug - 2005 Oct, with revision history 2004-06-23 to 2005-11-25
Archive 6: 2005 Nov - 2006 Feb
Archive 7: 2006 Mar - 2006 Jun
Archive 8: 2006 Jul - 2006 Sep
Archive 9: 2006 Oct - 2007 Jan, with revision history 2005-11-25 to 2007-02-21
Archive 10: 2007 Feb - 2007 Jun
Archive 11: 2007 Jul - 2007 Dec
Archive 12: 2008 Jan - 2008 Jul
Archive 12a: 2008 Aug
Archive 12b: 2008 Sept
Archive 13: 2008 Oct - 2009 Jun
Archive 14: 2009 Jun - 2009 Dec

2010s

2020-2024

2025-present

@@ Line 1: / Line 1: @@
 == May 17 ==
+* 00:23 ori: varnishadm on cp1056 confirms that varnish recognizes mw1151 as "sick"
+* 00:20 ori: stopping apache and disabling puppet on mw1151 so that varnish stops forwarding reqs to it
+* 00:16 Krinkle: On mw1151, Gadget::loadStructuredList() returns false, memcached has no value for 'enwiki:gadgets-definition:7' and is unable to store it.
+* 00:11 ori: Krinkle identified weird RL responses as all originating in mw1151; dmesg shows ata1 disk troubles: "failed command: READ DMA EXT", "sd 0:0:0:0: [sda]  Add. Sense: Unrecovered read error - auto reallocate failed"
 * 00:04 logmsgbot: krinkle synchronized php-1.24wmf4/includes/resourceloader/ResourceLoader.php  'I718fcf23d'