Server Admin Log/Archive 27

2015-07-31

• 20:14 logmsgbot: ori Synchronized php-1.26wmf16/includes/objectcache/ObjectCacheSessionHandler.php: Uncommitted revert of I4afaecd to test impact on T102199 (duration: 00m 12s)
• 20:11 godog: revert to openjdk8 and restart cassandra on restbase1008
• 19:55 logmsgbot: ori Synchronized php-1.26wmf16/includes/User.php: More debug logging for T102199 (duration: 00m 13s)
• 19:54 godog: revert to openjdk8 and restart cassandra on restbase1007
• 19:51 logmsgbot: ori Synchronized php-1.26wmf16/includes/EditPage.php: More debug logging for T102199 (duration: 00m 12s)
• 19:21 godog: revert to openjdk8 and restart cassandra on restbase1006
• 19:02 godog: revert to openjdk8 and restart cassandra on restbase1005
• 18:44 twentyafterfour: oddly, the symptom was that there were logs about apc cache entries that had been on the GC queue for too long, I guess this is due to phd being stuck
• 18:43 twentyafterfour: restarted phd on iridium. I had to forcefully kill one stuck repository worker to get the daemons to restart properly.
• 18:36 godog: revert to openjdk8 and restart cassandra on restbase1004
• 18:15 mutante: multatuli - installing package upgrades
• 18:08 legoktm: made User:Flow talk page manager a 'bot' on all wikis (except loginwiki)
• 18:08 godog: revert to openjdk8 and restart cassandra on restbase1003
• 17:53 godog: revert to openjdk8 and restart cassandra on restbase1002
• 17:41 godog: revert to openjdk8 and restart cassandra on restbase1001 T104887
• 17:11 greg-g: follow on to previous to be explicit: it's not deployed, it is queued for Monday morning SWAT
• 17:10 aude: wmf/1.26wmf16 core submodule bump for Ic25edf7 (MultimediaViewer) is now on tin
• 17:06 logmsgbot: aude Synchronized php-1.26wmf16/extensions/Wikidata: Fix api xml format (duration: 00m 20s)
• 15:52 bd808: Rebuilt grafana-dashboards index to have 1 shard/2 replicas in logstash cluster
• 15:46 bd808: Rebuilt kibana-int index to have 1 shard/2 replicas in logstash cluster
• 15:45 andrewbogott: rebooting labvirt1005, again (3.16 this time)
• 15:19 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: reverting db1035 load to 10% (duration: 00m 14s)
• 15:03 urandom: bouncing restbase1005 (attempting to reproduce GC trends)
• 14:54 Coren: turned on alerting of backup status on labstore* with (by design) low limits. Expect alarms, and ignore.
• 14:44 kart_: Update cxserver to 9669e19
• 14:38 andrewbogott: bumped the kernel version on labvirt1005, rebooting.
• 14:09 godog: restart cassandra on restbase1004 to apply java downgrade, missed from batch downgrade yesterday
• 12:10 godog: restbase1008 bootstrap finished successfully
• 10:30 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: returning db1035 to 100% load (duration: 00m 12s)
• 08:19 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I7be6dd2f5: Set $wgAjaxEditStash to false, on suspicion of being implicated in T102199 (duration: 00m 12s)
• 07:35 _joe_: powercycling analytics1013, no ssh, console unresponsive
• 04:45 logmsgbot: @tin ResourceLoader cache refresh completed at Fri Jul 31 04:45:41 UTC 2015 (duration 45m 40s)
• 04:09 springle: upgrade/restart dbstore1001
• 03:48 logmsgbot: krenair Synchronized php-1.26wmf16/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/228197/ (duration: 00m 12s)
• 02:31 logmsgbot: @tin LocalisationUpdate completed (1.26wmf16) at 2015-07-31 02:31:20+00:00
• 02:28 logmsgbot: l10nupdate Synchronized php-1.26wmf16/cache/l10n: (no message) (duration: 06m 13s)
• 00:35 logmsgbot: catrope Synchronized php-1.26wmf16/extensions/Flow/includes/Model/WikiReference.php: debugging (duration: 00m 12s)
• 00:34 logmsgbot: catrope Synchronized php-1.26wmf16/extensions/Flow/includes/Model/WikiReference.php: debugging (duration: 00m 12s)
• 00:29 logmsgbot: catrope Synchronized php-1.26wmf16/extensions/Flow/includes/Model/WikiReference.php: debugging (duration: 00m 13s)

2015-07-30

• 23:52 logmsgbot: catrope Synchronized flow.dblist: remove commons (duration: 00m 14s)
• 23:47 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/195886/ (duration: 00m 11s)
• 23:46 logmsgbot: krenair Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/195886/ (duration: 00m 12s)
• 23:41 logmsgbot: catrope Synchronized flow.dblist: Enable Flow on plwiki and commonswiki (duration: 00m 11s)
• 23:30 logmsgbot: ebernhardson Synchronized php-1.26wmf16/extensions/DonationInterface/: Bump DonationInterfae in 1.26wmf16 again...its uses submodules (duration: 00m 15s)
• 23:29 logmsgbot: ebernhardson Synchronized php-1.26wmf16/extensions/DonationInterface/: Bump DonationInterfae in 1.26wmf16 (duration: 00m 16s)
• 23:28 robh: disregard log entry about racktables, never offlined
• 23:22 logmsgbot: ebernhardson Synchronized php-1.26wmf16/includes/specials/SpecialMIMEsearch.php: (no message) (duration: 00m 12s)
• 23:21 logmsgbot: ebernhardson Synchronized php-1.26wmf16/includes/specials/SpecialSearch.php: Fix search-suggest i18n for frwiki in SWAT (duration: 00m 14s)
• 23:21 logmsgbot: ebernhardson Synchronized php-1.26wmf16/extensions/SpamBlacklist/: Update SpamBlacklist for SWAT (duration: 00m 11s)
• 23:12 awight: updating paymentswiki from 02db5f7f77b667da06b882b2f66de9c5546230bc to d4bdce1cae168448b116d75e3dcd3303b0f13dd2
• 23:10 robh: killing apache on magnesium to manually trigger an outage of racktables and test catchpoint alert formatting
• 23:10 logmsgbot: krinkle Synchronized w/rl-test.php: T105255 (duration: 00m 12s)
• 23:06 legoktm: manually merged User:Mirwin's accounts (T107168)
• 22:59 awight: rolling back. paymentswiki.
• 22:59 awight: redeploying sketchy paymentswiki config
• 22:57 awight: updating paymentswiki from 6854683083cabc730f37b6a79d559f23e7ff7b0f to 02db5f7f77b667da06b882b2f66de9c5546230bc
• 22:43 awight: paymentswiki config rolled back
• 22:42 awight: paymentswiki: config the IIIrd
• 22:34 awight: paymentswiki: rolled back again
• 22:31 awight: redeploying paymentswiki config: with password this time
• 22:21 awight: rolled back paymentswiki config
• 22:01 logmsgbot: ori Synchronized php-1.26wmf16/includes/page/WikiPage.php: I73fba15c26c1: Defer the InfoAction purge in onArticleEdit() (duration: 00m 11s)
• 21:58 awight: paymentswiki config: jiggle the handle
• 21:42 awight: updated paymentswiki from fd0060bf86777ee6b7acd205d134066356da69e8 to 6854683083cabc730f37b6a79d559f23e7ff7b0f
• 21:06 logmsgbot: ori Synchronized php-1.26wmf16/includes/Message.php: c72b7c435f: Debug logging for T102199 (take 2) (duration: 00m 11s)
• 21:06 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: I1bbf3f0: Add a debug log channel for bug T102199 (duration: 00m 12s)
• 20:47 mutante: iridium - apt-get clean - 1.7G avail
• 20:02 logmsgbot: ori Synchronized wmf-config/mobile.php: (no message) (duration: 00m 12s)
• 20:00 bblack: starting rolling wipe process on mobile cache contents for T106966 fixup
• 19:48 logmsgbot: ori Synchronized wmf-config: I0990ac5b: Update URL configuration for mobile when entering mobile mode (duration: 00m 12s)
• 19:15 matt_flaschen: Deployed patch for T107170 to wmf/1.26wmf16
• 19:09 logmsgbot: legoktm Synchronized php-1.26wmf16: Revert "Use OOUI HTMLForm for Special:Watchlist" (duration: 01m 46s)
• 18:49 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I6db1771bf4: Use absolute URLs to construct load.php requests (duration: 00m 12s)
• 18:33 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I6665bf31: Use relative URLs to construct load.php requests (duration: 00m 12s)
• 18:02 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: all wikis to 1.26wmf16
• 17:56 cmjohnson1: decom virt1001-virt1009
• 17:45 jynus: killing some long running queries on db1042
• 15:30 logmsgbot: krenair Synchronized php-1.26wmf15/extensions/MobileFrontend/includes/Resources.php: https://gerrit.wikimedia.org/r/#/c/228001/ (duration: 00m 12s)
• 15:30 logmsgbot: krenair Synchronized php-1.26wmf16/extensions/MobileFrontend/includes/Resources.php: https://gerrit.wikimedia.org/r/#/c/228000/ (duration: 00m 11s)
• 15:21 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/227999/ (duration: 00m 12s)
• 15:03 gwicke: disabled old restbase checkout on tin to make sure it doesn't start up
• 15:02 logmsgbot: krenair Synchronized w/static/images/project-logos/commonswiki.png: https://gerrit.wikimedia.org/r/#/c/227962/ (duration: 00m 13s)
• 15:02 godog: bootstrap cassandra on restbase1008
• 15:02 gwicke: manually cleaned up RB code on 1007 and 1008
• 14:37 moritzm: installed openjdk security updates on analytics*
• 14:05 moritzm: restarted opendj on nembus/neptunium to effect OpenJDK security updates
• 13:44 godog: downgrade openjdk-7-jre on restbase1007, nodetool flush and cassandra restart
• 13:39 godog: downgrade openjdk-7-jre on restbase1006, nodetool flush and cassandra restart
• 13:29 godog: downgrade openjdk-7-jre on restbase1005, nodetool flush and cassandra restart
• 13:25 moritzm: installed openjdk updates on gallium, restarting jenkins
• 13:17 godog: downgrade openjdk-7-jre on restbase1004, nodetool flush and cassandra restart
• 13:02 godog: downgrade openjdk-7-jre on restbase1003, nodetool flush and cassandra restart
• 12:47 godog: downgrade openjdk-7-jre on restbase1002, nodetool flush and cassandra restart
• 12:36 godog: downgrade openjdk-7-jre on restbase1001, nodetool flush and cassandra restart
• 09:18 hashar: Upgraded Zuul on all CI slaves. Should be a noop for zuul-cloner.
• 07:10 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jul 30 07:10:39 UTC 2015 (duration 10m 38s)
• 04:06 Krenair: Ignore that last error
• 04:05 logmsgbot: LocalisationUpdate failed: git pull of core failed
• 03:33 mutante: killing processes by ellery on stat1002 - load avg was over 1500 and users reported pagecounts are broken (possibly all other crons as well)
• 03:01 logmsgbot: LocalisationUpdate completed (1.26wmf16) at 2015-07-30 03:01:49+00:00
• 02:59 logmsgbot: l10nupdate Synchronized php-1.26wmf16/cache/l10n: (no message) (duration: 04m 25s)
• 02:40 logmsgbot: LocalisationUpdate completed (1.26wmf15) at 2015-07-30 02:40:38+00:00
• 02:36 logmsgbot: l10nupdate Synchronized php-1.26wmf15/cache/l10n: (no message) (duration: 07m 45s)
• 02:26 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: I3c6217f06: Double $wgMemoryLimit (330 => 660) (duration: 00m 12s)
• 02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jul 30 02:07:40 UTC 2015 (duration 7m 39s)
• 02:03 logmsgbot: LocalisationUpdate failed (1.26wmf16) at 2015-07-30 02:03:29+00:00
• 02:03 logmsgbot: LocalisationUpdate failed (1.26wmf15) at 2015-07-30 02:03:29+00:00
• 01:30 springle: MIMEsearchPage::reallyDoQuery queries with crazy eg, LIMIT 10405000,501, on commonswiki vslow slave, from tide***.microsoft.com bots. log noise is queries hitting 5min limit and auto-killed
• 00:48 logmsgbot: ori Synchronized php-1.26wmf15/includes/Message.php: 160f69871c: Debug logging for T102199 (duration: 00m 13s)
• 00:36 logmsgbot: ori Synchronized php-1.26wmf16/includes/Message.php: eb281630ce: Debug logging for T102199 (duration: 00m 11s)
• 00:10 awight: rolled back config
• 00:09 awight: crazy previous message was all about: I pointed the DonationInterface frontends to mirror limbo messages to a Redis server on localhost.
• 00:08 awight: deployed interesting gc-cc-limbo config

2015-07-29

• 23:43 legoktm: finished fixing Scribunto content models
• 23:30 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/225840/ (duration: 00m 12s)
• 23:30 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/225840/ (duration: 00m 12s)
• 23:23 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/227892/ (duration: 00m 12s)
• 23:20 legoktm: starting script to fix Scribunto content models due to imports on all wikis (T91170)
• 23:14 logmsgbot: bd808 Purged l10n cache for 1.26wmf14
• 23:14 logmsgbot: bd808 Purged l10n cache for 1.26wmf13
• 23:13 logmsgbot: bd808 Purged l10n cache for 1.26wmf12
• 23:03 mutante: snapshot1001 - apt-get clean - 107M avail
• 23:02 Krenair: snapshot1001 - No space left on device
• 23:02 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/227879/ (duration: 00m 12s)
• 22:27 legoktm: update page set page_content_model ="wikitext" where page_id=12134769; on wikidatawiki
• 21:22 legoktm: fixed Module:*/doc pages on wikidatawiki
• 20:44 legoktm: update page set page_content_model="Scribunto" where page_id=12134769; on wikidatawiki
• 20:42 arlolra: updated Parsoid to version 6e095a92
• 20:41 legoktm: manually fixed content models for wikidata's Module namespace (T107340)
• 20:31 logmsgbot: ori Synchronized php-1.26wmf16/extensions/Wikidata/extensions/Wikibase/repo/includes/actions/SubmitEntityAction.php: Live-hack stats increment call for session_fail_preview (duration: 00m 12s)
• 20:30 logmsgbot: ori Synchronized php-1.26wmf16/extensions/Wikidata/extensions/Wikibase/repo/includes/EditEntity.php: Live-hack stats increment call for session_fail_preview (duration: 00m 12s)
• 20:26 urandom: bouncing cassandra on restbase1006 to apply logstash config
• 20:18 urandom: bouncing cassandra on restbase1005 to apply logstash config
• 20:15 urandom: bouncing cassandra on restbase1004 to apply logstash config
• 20:11 urandom: bouncing cassandra on restbase1003 to apply logstash config
• 20:04 urandom: bouncing cassandra on restbase1002 to apply logstash config
• 19:59 urandom: restarting restbase1001 to apply logstash config
• 19:51 twentyafterfour: scap sync failed on snapshot1001 due to full disk
• 19:48 logmsgbot: twentyafterfour Finished scap: group1 wikis to 1.26wmf16 (duration: 45m 12s)
• 19:03 logmsgbot: twentyafterfour Started scap: group1 wikis to 1.26wmf16
• 18:36 legoktm: fixed content models of MediaWiki and Module namespace pages on azbwiki
• 18:24 legoktm: manually attached User:Flow talk page manager accounts
• 17:38 logmsgbot: aude Synchronized php-1.26wmf16/extensions/Wikidata: fix focus when entering site links (duration: 00m 22s)
• 17:37 logmsgbot: aude Synchronized php-1.26wmf16/thumb.php: 2c9518ed78: Add Content-Length header to thumb.php redirects (duration: 00m 13s)
• 16:14 andrewbogott: re-imaging labnodepool1001
• 16:13 ori: depooled Precise image scalers (mw1159 / mw1160)to see if 2c9518ed78 helped.
• 16:12 logmsgbot: ori Synchronized wmf-config: Revert "No need for wgSecureLogin on our wikis, HTTPS is forced everywhere" (duration: 00m 13s)
• 16:11 logmsgbot: ori Synchronized php-1.26wmf15/thumb.php: 2c9518ed78: Add Content-Length header to thumb.php redirects (duration: 00m 12s)
• 16:11 logmsgbot: ori Synchronized php-1.26wmf16/thumb.php: 2c9518ed78: Add Content-Length header to thumb.php redirects (duration: 00m 12s)
• 16:01 moritzm: installed qemu security updates on labvirt*
• 15:36 logmsgbot: krenair Synchronized tests/dblistTest.php: (no message) (duration: 00m 10s)
• 15:36 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 12s)
• 15:36 logmsgbot: krenair Synchronized database lists: (no message) (duration: 00m 12s)
• 15:33 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 12s)
• 15:30 logmsgbot: krenair Synchronized wikisource.dblist: https://gerrit.wikimedia.org/r/#/c/194549/ (duration: 00m 12s)
• 15:27 logmsgbot: krenair Synchronized tests/dblistTest.php: https://gerrit.wikimedia.org/r/#/c/194549/ (duration: 00m 13s)
• 15:26 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/194549/ (duration: 00m 13s)
• 15:26 logmsgbot: krenair Synchronized database lists: https://gerrit.wikimedia.org/r/#/c/194549/ (duration: 00m 11s)
• 15:21 logmsgbot: krenair Synchronized wikipedia.dblist: https://gerrit.wikimedia.org/r/#/c/227718/3 (duration: 00m 12s)
• 15:21 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/227718/3 (duration: 00m 12s)
• 15:20 logmsgbot: aude Synchronized php-1.26wmf15/extensions/Wikidata: rv usage tracking change (duration: 00m 20s)
• 15:18 logmsgbot: krenair Synchronized wikipedia.dblist: https://gerrit.wikimedia.org/r/#/c/227718/3 (duration: 00m 12s)
• 15:17 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/227718/3 (duration: 00m 12s)
• 14:28 logmsgbot: aude Synchronized usagetracking.dblist: Enable usage tracking on ptwiki and azbwiki (duration: 00m 12s)
• 14:14 logmsgbot: aude Synchronized php-1.26wmf15/extensions/Wikidata: rv add usage tracking job (duration: 00m 20s)
• 14:13 logmsgbot: aude Synchronized php-1.26wmf15/extensions/Wikidata: add usage tracking job (duration: 00m 20s)
• 14:11 logmsgbot: aude Synchronized php-1.26wmf16/extensions/Wikidata: add usage tracking job (duration: 00m 24s)
• 13:27 bblack: repooling cp3030 with wiped caches
• 13:19 bblack: depooling cp3030 (all layers)
• 10:51 _joe_: restarted apertium-apy on sca1001, freed 54 GB of RAM (processes were OOMing)
• 10:18 _joe_: repooling the zend imagescalers until https://gerrit.wikimedia.org/r/#/c/227676 is reviewed and deployed
• 09:14 _joe_: depooling mw1159-60 from the imagescalers pool
• 08:02 hashar_: disabled puppet on labnodepool1001.eqiad.wmnet
• 07:41 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jul 29 07:41:54 UTC 2015 (duration 41m 53s)
• 04:43 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: rv myself (duration: 00m 13s)
• 04:42 logmsgbot: demon Synchronized database lists: rv myself (duration: 00m 12s)
• 04:00 logmsgbot: demon Synchronized database lists: moving special wikipedias to wikipedia.dblist (duration: 00m 13s)
• 04:00 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: moving special wikipedias to wikipedia.dblist (duration: 00m 12s)
• 03:25 springle: upgrade reboot db1011 trusty
• 03:15 logmsgbot: LocalisationUpdate completed (1.26wmf16) at 2015-07-29 03:15:56+00:00
• 03:09 logmsgbot: l10nupdate Synchronized php-1.26wmf16/cache/l10n: (no message) (duration: 10m 47s)
• 02:43 logmsgbot: LocalisationUpdate completed (1.26wmf15) at 2015-07-29 02:43:27+00:00
• 02:37 logmsgbot: l10nupdate Synchronized php-1.26wmf15/cache/l10n: (no message) (duration: 10m 08s)
• 02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jul 29 02:07:17 UTC 2015 (duration 7m 16s)
• 02:03 logmsgbot: LocalisationUpdate failed (1.26wmf16) at 2015-07-29 02:03:04+00:00
• 02:03 logmsgbot: LocalisationUpdate failed (1.26wmf15) at 2015-07-29 02:03:03+00:00
• 00:43 logmsgbot: ori Synchronized php-1.26wmf15/extensions/AbuseFilter: Revert "Revert "Conversion to using getMainStashInstance()"" (duration: 00m 12s)
• 00:02 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Iccd317c6: Switch over the 'sessions' ObjectCache to nutcracker (T106986) (duration: 00m 13s)
• 00:01 ori: Switching over the sessions ObjectCache instance to use nutcracker. Users with an existing edit session in progress will have their session reset and will need to re-login.

2015-07-28

• 23:50 logmsgbot: ori Synchronized php-1.26wmf15/includes/objectcache/RedisBagOStuff.php: I3812ec5a0b: RedisBagOStuff: if no alternatives, skip master link status check (duration: 00m 12s)
• 23:50 logmsgbot: ori Synchronized php-1.26wmf16/includes/objectcache/RedisBagOStuff.php: I3812ec5a0b: RedisBagOStuff: if no alternatives, skip master link status check (duration: 00m 12s)
• 23:36 bblack: rebooting cp20xx.codfw.wmnet for kernel updates (downtimed)
• 23:20 logmsgbot: krenair Synchronized php-1.26wmf16/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.ApiResponseCache.js: https://gerrit.wikimedia.org/r/#/c/227607/ (duration: 00m 12s)
• 23:02 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/227496/ (duration: 00m 12s)
• 22:55 ejegg: updated payments from bdc4afaa7699904ac30c1f6d3bb3fbc6bac5e87e to fd0060bf86777ee6b7acd205d134066356da69e8
• 22:51 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.26wmf16
• 22:40 logmsgbot: krinkle Synchronized w/rl-test.php: T105255 (duration: 00m 12s)
• 22:23 Tim: on mw1203 restarted hhvm due to StatCache lockup
• 22:08 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Iecddb3bf24: Add nutcracker-redis object cache instance, unused for now (duration: 00m 11s)
• 22:05 logmsgbot: twentyafterfour Finished scap: new branch: testwiki to 1.26wmf16 (duration: 26m 26s)
• 22:01 gwicke: restbase ca30b69 deployed to eqiad cluster
• 21:48 gwicke: canary restbase ca30b69 deploy to restbase1001.eqiad
• 21:39 logmsgbot: twentyafterfour Started scap: new branch: testwiki to 1.26wmf16
• 21:14 matt_flaschen: Deployed patch for T107170 to wmf/1.26wmf15 and wmf/1.26wmf16
• 20:39 ori: Upgraded nutcracker to 0.4.1-1+wm1 across fleet
• 18:57 logmsgbot: bblack Synchronized wmf-config/InitialiseSettings-labs.php: remove wgSecureLogin (duration: 00m 12s)
• 18:56 logmsgbot: bblack Synchronized wmf-config/InitialiseSettings.php: remove wgSecureLogin (duration: 00m 12s)
• 18:44 ori: Twiddling with nutcracker on mw1041
• 18:33 andrewbogott: disabling puppet and nova-network on labnet1002 to avoid possible conflict between two different dhcp servers
• 17:04 godog: start cassandra on restbase1007, tentative bootstrap
• 16:24 YuviPanda: bounced create-dbusers on labstore1002
• 16:03 bd808: logstash1002 conversion to jessie done; log event volume returning to normal in index
• 16:01 godog: bounce cassandra on xenon to test logstash logging
• 15:52 bd808: installed logstash on logstash1002; forced puppet run
• 15:03 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor for 5% of new accounts on enwiki gerrit:226338 (duration: 00m 12s)
• 14:43 cmjohnson1: powering down logstash1002 to remove disk and install jessie
• 14:28 moritzm: restarted zookeeper on conf1003 to effect OpenJDK security update
• 14:16 _joe_: re-enabled puppet on mw1152 for testing
• 14:16 moritzm: restarted zookeeper on conf1002 to effect OpenJDK security update
• 13:58 paravoid: upgrading baham to gdnsd 2.2.0
• 13:41 _joe_: disabled puppet on mw1152, thumb_handler testing
• 13:40 moritzm: restarted zookeeper on conf1001 to effect OpenJDK security update
• 13:13 jynus: temporarily changing master of db1069(s1) to db1051 in order to fix some labsdb inconsistencies on enwiki_p
• 12:29 godog: reenable puppet on restbase1001 after merging https://gerrit.wikimedia.org/r/#/c/227355/
• 10:31 paravoid: merging a series of mail-related patches; ping me personally if problems arise
• 10:03 mobrovac: citoid deploying d57ec96
• 09:41 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Increasing db1035 weight (duration: 00m 13s)
• 08:13 moritzm: added elasticsearch-1.7.0 to carbon for jessie and trusty
• 07:30 YuviPanda: dropped others20150724190859 on labstore1002
• 06:53 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul 28 06:53:21 UTC 2015 (duration 53m 20s)
• 02:30 logmsgbot: LocalisationUpdate completed (1.26wmf15) at 2015-07-28 02:30:24+00:00
• 02:26 logmsgbot: l10nupdate Synchronized php-1.26wmf15/cache/l10n: (no message) (duration: 07m 29s)
• 02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul 28 02:07:52 UTC 2015 (duration 7m 51s)
• 02:03 logmsgbot: LocalisationUpdate failed (1.26wmf15) at 2015-07-28 02:03:41+00:00
• 01:11 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/227371/ (duration: 00m 11s)
• 00:35 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/227381/ (duration: 00m 13s)
• 00:30 logmsgbot: krenair Synchronized php-1.26wmf15/extensions/SiteMatrix/SiteMatrix_body.php: https://gerrit.wikimedia.org/r/#/c/227379/ (duration: 00m 12s)
• 00:00 logmsgbot: catrope Finished scap: SWAT (duration: 22m 15s)

2015-07-27

• 23:53 ori: Re-pooling mw1159 and mw1160
• 23:38 logmsgbot: catrope Started scap: SWAT
• 23:24 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings.php: SWAT (duration: 00m 12s)
• 23:23 logmsgbot: catrope Synchronized w/static/images/project-logos/suwikiquote.png: Localized logo for suwikiquote (duration: 00m 12s)
• 23:17 ejegg: updated crm from 83cacfa1e0852ffaf47d2f02e7d843cf6f3bcda4 to db417a28a247a3fdf3e3023a700d6266e04f3e9d
• 22:19 andrewbogott: rebooting labvirt1005
• 21:50 bd808: updated scap to dc8eda5 (Don't exclude PHP files from being synced)
• 21:34 logmsgbot: ori Synchronized php-1.26wmf15/extensions/AbuseFilter: I13d29ea6: Revert "Conversion to using getMainStashInstance()" (duration: 00m 12s)
• 21:24 andrewbogott: rebooting labnet1002, just to see if I can
• 20:57 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I1ca47ebc4: $wgEventLoggingSchemaApiUri: http -> https (duration: 00m 12s)
• 20:54 bd808: installed libbcprov-java and restarted logstash on logstash1001
• 20:33 subbu: deployed parsoid version 92f1cd6d
• 20:17 ori: (A rise in 503s/minute expected. I'll keep it brief.)
• 20:16 ori: Depooled Precise scalers (mw1159 and mw1160) again, for testing.
• 20:07 godog: bounce rsyslog on mw in eqiad in batches
• 19:58 godog: bounce rsyslog on mw in codfw in batches
• 19:54 logmsgbot: twentyafterfour Synchronized w/: deploy https://gerrit.wikimedia.org/r/#/c/227326/ (duration: 00m 12s)
• 19:47 godog: bounce rsyslog on mw1235
• 19:37 bd808: godog fixed salt key for logstash1001 which fixed trebuchet install of kibana
• 19:31 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/227273/ (duration: 00m 13s)
• 19:17 robh: etherpad was giving errors, apache restart fixed
• 18:56 bd808: rsyslog forwarded hhvm and apache2 logs still not hitting logstash1001; rsyslog restarts may be needed
• 18:53 legoktm: restarted populateContentModel.php --wiki=enwiki on terbium with modification to occassionally clear the link cache so it doesn't OOM.
• 18:49 godog: stop jobrunner/jobchron/hhvm on mw1011
• 18:41 bd808: manually ran sync-common on mw1011
• 18:40 bd808: fatalmonitor full of errors from mw1011
• 18:38 logmsgbot: bd808 Synchronized wmf-config/InitialiseSettings.php: logstash: change ip address for logstash1001 and logstash1003 (duration: 00m 12s)
• 18:33 bd808: logstash1003 salt key not accepted by master
• 18:25 bd808: No mediawiki, hhvm or apache2 logs going to logstash1001:10514
• 18:20 bd808: logstash1001 back up and running
• 17:08 moritzm: updated mc200[34] to linux 3.19.3-7 for some testing on hardware
• 16:34 bblack: switched operations/dns to ff-only like operations/puppet in gerrit config
• 16:29 bblack: restarted gitblit on antimony (AGAIN...)
• 15:47 bd808: Added bgerstile and coreyfloyd to github "owners" team
• 15:43 _joe_: upgrading the jobrunners to the latest HHVM packlage
• 15:39 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable EducationProgram extension at French Wikisource gerrit:225019 (duration: 00m 12s)
• 15:26 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Quiz extension at French Wikibooks gerrit:225021 (duration: 00m 12s)
• 15:09 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Set wgCategoryCollation to uca-default on cswiktionary gerrit:226483 (duration: 00m 12s)
• 15:07 bd808: logstash1001 and logstash1003 offline for physical move and reimaging to jessie. kibana data will be degraded until they are back
• 15:04 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor for auto-created accounts on enwiki gerrit:226337 (duration: 00m 13s)
• 14:14 cmjohnson1: logstash1001 going down to relocate to row A
• 13:55 moritzm: uploaded linux 3.19.3-7 (based on 3.19.8-ckt4 plus the recent NMI security fixes) to carbon
• 13:20 cmjohnson1: powering down logstash1003 to relocate to rack d3
• 12:51 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Repool db1035 after maintenance (duration: 00m 12s)
• 12:07 twentyafterfour: deployed https://gerrit.wikimedia.org/r/#/c/227205/ and restarted apache2 on iridium
• 10:04 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Depool db1035 (duration: 00m 12s)
• 09:54 godog: reimage restbase1009, new disks
• 09:24 godog: reimage restbase1007, new disks installed
• 09:09 hashar: Allowed JenkinsBot to submit changes on operations/software/conftool for CI purposes.
• 07:54 moritzm: installed java security updates on xenon, cerium, praseodymium, maps-test*
• 06:59 _joe_: upgrading hhvm to the latest package across the cluster
• 05:47 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jul 27 05:47:31 UTC 2015 (duration 47m 30s)
• 05:00 gwicke: restarted cassandra on restbase1003
• 03:39 springle: upgrade & restart dbstore1002
• 02:27 logmsgbot: LocalisationUpdate completed (1.26wmf15) at 2015-07-27 02:27:00+00:00
• 02:22 logmsgbot: l10nupdate Synchronized php-1.26wmf15/cache/l10n: (no message) (duration: 07m 20s)
• 02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jul 27 02:07:15 UTC 2015 (duration 7m 14s)
• 02:03 logmsgbot: LocalisationUpdate failed (1.26wmf15) at 2015-07-27 02:03:04+00:00
• 01:18 ori: Re-pooling mw1159 and mw1160; ran out of time for debugging.
• 00:43 ori: Depooled Precise image scalers (mw1159 and mw1160); watching for errors.

2015-07-26

• 22:13 legoktm: killed populateContentModel.php for enwiki on terbium due to alerts
• 21:02 logmsgbot: ori Synchronized docroot/wikimedia.org/WikipediaMobileFirefoxOS: Update WikipediaMobileFirefoxOS submodule for URL changes (duration: 00m 16s)
• 20:51 logmsgbot: ori Synchronized docroot: I5f8b8b54a: Move WikipediaMobileFirefoxOS from bits to wikimedia.org docroot (Bug: T98373) (duration: 00m 17s)
• 05:30 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jul 26 05:30:10 UTC 2015 (duration 30m 9s)
• 03:38 robh: ulsfo network issues, faidon depooled via https://gerrit.wikimedia.org/r/#/c/227067/
• 02:26 logmsgbot: LocalisationUpdate completed (1.26wmf15) at 2015-07-26 02:26:47+00:00
• 02:22 logmsgbot: l10nupdate Synchronized php-1.26wmf15/cache/l10n: (no message) (duration: 07m 12s)
• 02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jul 26 02:07:01 UTC 2015 (duration 7m 0s)
• 02:02 logmsgbot: LocalisationUpdate failed (1.26wmf15) at 2015-07-26 02:02:51+00:00

2015-07-25

• 20:51 gwicke: rolling restart of restbase instances
• 16:53 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Repool db1035 at 100% capacity (duration: 00m 40s)
• 16:30 _joe_: repooling mw1159,mw1160
• 14:33 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Repool db1035 with lower weight (duration: 00m 13s)
• 13:57 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Depool db1035 (duration: 00m 12s)
• 13:56 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Depool db1035 (duration: 00m 12s)
• 13:42 jynus: db1035 restarted, temporarilly increasing db error rates on s3
• 07:05 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jul 25 07:05:08 UTC 2015 (duration 5m 7s)
• 02:41 logmsgbot: LocalisationUpdate completed (1.26wmf15) at 2015-07-25 02:41:09+00:00
• 02:35 logmsgbot: l10nupdate Synchronized php-1.26wmf15/cache/l10n: (no message) (duration: 09m 52s)
• 02:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jul 25 02:08:04 UTC 2015 (duration 8m 3s)
• 02:03 logmsgbot: LocalisationUpdate failed (1.26wmf15) at 2015-07-25 02:03:54+00:00

2015-07-24

• 21:57 legoktm: running mwscript populateContentModel.php --wiki=enwiki --ns=all --table=page
• 20:36 logmsgbot: krenair Synchronized php-1.26wmf15/extensions/VisualEditor/modules/ve-mw/ui: https://gerrit.wikimedia.org/r/#/c/226907/ (duration: 00m 12s)
• 19:40 awight: updated DjangoBannerStats from 3db799dc8705c728c7261ae433e8197f5498fa1b to 57a0392b3f43b65050b01a0465e120ed609a769e
• 19:08 YuviPanda: remove others20150724183453 on labstore1002
• 18:39 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Ib7c7861e: Point to a no-op /beacon URL rather than Special:RecordImpression (duration: 00m 12s)
• 18:38 ori: Merging Ib7c7861e: Point to a no-op /beacon URL rather than Special:RecordImpression
• 18:30 ori: Depooled Precise image scalers (mw1159 and mw1160)
• 18:29 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Idfe1fa60: testwiki: Point to a no-op /beacon URL rather than Special:RecordImpression (duration: 00m 12s)
• 18:17 YuviPanda: removed labstore/others20150724 on labstore1002
• 18:15 YuviPanda: running others20150724 on labstore1002
• 16:51 bd808: Upgraded logstash1006 to elasticsearch 1.7.0
• 16:48 bd808: Upgraded logstash1005 to elasticsearch 1.7.0
• 16:36 bd808: Upgraded logstash1004 to elasticsearch 1.7.0
• 16:27 bd808: Upgraded logstash1003 to elasticsearch 1.7.0
• 16:26 bd808: Upgraded logstash1002 to elasticsearch 1.7.0
• 16:25 bd808: Upgraded logstash1001 to elasticsearch 1.7.0
• 13:44 cmjohnson1: swapping failed disk db1058
• 13:11 cmjohnson1: swapping ssds in restbase1007
• 12:47 hashar: restarting Jenkins
• 12:47 hashar: Jenkins: switching gearman plugin from our custom compiled 0.1.1-9-g08e9c42-change_192429_2 to upstream 0.1.2. They are actually the exact same versions.
• 10:23 logmsgbot: legoktm Synchronized php-1.26wmf15/extensions/AbuseFilter/: Special:AbuseFilter on all large Wikipedias is returning errors - T106798 (duration: 00m 13s)
• 08:40 hashar: upgrading zuul to zuul_2.0.0-327-g3ebedde-wmf3precise1 to fix a regression ( https://phabricator.wikimedia.org/T106531 )
• 05:53 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jul 24 05:53:16 UTC 2015 (duration 53m 15s)
• 05:52 Krinkle: Added rl-test.php on testwiki (mw1017) to gather stats about cache-control rollover (Catrope, Krinkle). Used by testwiki/test2wiki/mediawikiwiki Common.js (sampled). See T105255.
• 02:29 logmsgbot: LocalisationUpdate completed (1.26wmf15) at 2015-07-24 02:29:25+00:00
• 02:26 urandom: restarting restbase on restbase1006
• 02:25 logmsgbot: l10nupdate Synchronized php-1.26wmf15/cache/l10n: (no message) (duration: 07m 12s)
• 02:06 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jul 24 02:06:41 UTC 2015 (duration 6m 40s)
• 02:02 logmsgbot: LocalisationUpdate failed (1.26wmf15) at 2015-07-24 02:02:31+00:00
• 00:21 ori: Re-enabled Puppet on mw1153

2015-07-23

• 23:31 logmsgbot: catrope Synchronized php-1.26wmf15/extensions/WikimediaEvents: SWAT (duration: 00m 12s)
• 23:31 logmsgbot: catrope Synchronized php-1.26wmf15/extensions/CirrusSearch: SWAT (duration: 00m 12s)
• 23:30 logmsgbot: catrope Synchronized php-1.26wmf14/extensions/WikimediaEvents: SWAT (duration: 00m 12s)
• 23:30 logmsgbot: catrope Synchronized php-1.26wmf14/extensions/CirrusSearch: SWAT (duration: 00m 13s)
• 23:16 logmsgbot: catrope Synchronized flow.dblist: Enable Flow on viwiki (duration: 00m 12s)
• 23:14 logmsgbot: catrope Synchronized wmf-config/: SWAT (duration: 00m 11s)
• 23:14 logmsgbot: catrope Synchronized w/static/images/: SWAT (duration: 00m 12s)
• 23:11 ori: Restarting Apache on mw1153
• 23:09 ori: T84842: Requests to thumb_handler.php/.* don't match the ProxyPass rule and get handled by Zend instead. To see how HHVM actually handles these requests, I'm disabling Puppet on mw1153 and dropping the '$' anchor from the ProxyPass rules.
• 23:02 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings.php: Enable geo feature usage tracking on all wikis (duration: 00m 12s)
• 21:19 hashar: is already a nice improvement
• 20:33 twentyafterfour: deployed hotfix for T106716, restarted apache on iridium
• 18:46 logmsgbot: catrope Synchronized php-1.26wmf15/resources/src/mediawiki.less/mediawiki.ui/mixins.less: Unbreak quiet button styles (duration: 00m 13s)
• 18:10 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: all wikis to 1.26wmf15
• 17:56 logmsgbot: jynus Synchronized wmf-config/db-codfw.php: Repooling es2004 after hardware maintenance (duration: 00m 11s)
• 17:56 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Repooling es2004 after hardware maintenance (duration: 00m 12s)
• 17:38 legoktm: running foreachwikiindblist /home/legoktm/largebutnotenwiki.dblist populateContentModel.php --ns=all --table=page
• 16:27 ori: restarted hhvm on mw1221
• 16:16 logmsgbot: thcipriani Finished scap: SWAT: Add azb interwiki sorting, Add Southern Luri, and Fix name of S and W Balochi (duration: 06m 13s)
• 16:14 urandom: restarting Cassandra on restbase1001 to (temporarily) enable GC logging
• 16:10 logmsgbot: thcipriani Started scap: SWAT: Add azb interwiki sorting, Add Southern Luri, and Fix name of S and W Balochi
• 15:38 moritzm: added jenkins-debian-glue 0.13.0 to apt.wikimedia.org (jessie-wikimedia)
• 15:35 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: fix references to non-existent wikis gerrit:226470 (duration: 00m 13s)
• 15:31 _joe_: rebooting ms-be1003, stuck in kernel locks
• 15:31 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove reference to nonexistent ru_sibwiki.png gerrit:226469 (duration: 00m 14s)
• 15:26 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Add wgSitename and wgMetaNamespace for pnbwiki gerrit:226543 (duration: 00m 12s)
• 15:15 logmsgbot: thcipriani Synchronized wmf-config/CommonSettings.php: SWAT: Set a different wmgContentTranslationDefaultSourceLanguage for English part II gerrit:224031 (duration: 00m 12s)
• 15:14 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Set a different wmgContentTranslationDefaultSourceLanguage for English part I gerrit:224031 (duration: 00m 13s)
• 15:04 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Add wgSitename and wgMetaNamespace for pnbwikipedia gerrit:225322 (duration: 00m 12s)
• 13:08 mobrovac: graphoid deploying 81b9633
• 10:56 jynus: disabling puppet on maps-test hosts to debug service issue
• 07:28 _joe_: upgrading hhvm on the canary appservers
• 06:59 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jul 23 06:59:44 UTC 2015 (duration 59m 43s)
• 06:42 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1070, warm up (duration: 00m 13s)
• 04:25 logmsgbot: ori Synchronized php-1.26wmf15/extensions/Scribunto/common/Base.php: (no message) (duration: 00m 13s)
• 04:24 logmsgbot: ori Synchronized php-1.26wmf14/extensions/Scribunto/common/Base.php: (no message) (duration: 00m 12s)
• 04:04 springle: upgrade & reboot db1070
• 03:04 logmsgbot: LocalisationUpdate completed (1.26wmf15) at 2015-07-23 03:04:48+00:00
• 03:00 logmsgbot: l10nupdate Synchronized php-1.26wmf15/cache/l10n: (no message) (duration: 07m 24s)
• 02:39 springle: temporarily silenced backup4001 check_disk space icinga noise; seems important, but not exploding-any-minute-now
• 02:37 logmsgbot: LocalisationUpdate completed (1.26wmf14) at 2015-07-23 02:37:55+00:00
• 02:34 logmsgbot: l10nupdate Synchronized php-1.26wmf14/cache/l10n: (no message) (duration: 07m 13s)
• 02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jul 23 02:07:12 UTC 2015 (duration 7m 11s)
• 02:05 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1070 (duration: 00m 12s)
• 02:03 logmsgbot: LocalisationUpdate failed (1.26wmf15) at 2015-07-23 02:03:03+00:00
• 02:03 logmsgbot: LocalisationUpdate failed (1.26wmf14) at 2015-07-23 02:03:02+00:00
• 01:45 logmsgbot: ori Synchronized php-1.26wmf15/includes/libs/objectcache/APCBagOStuff.php: I4b2cf1715538 (duration: 00m 12s)
• 01:45 logmsgbot: ori Synchronized php-1.26wmf14/includes/libs/objectcache/APCBagOStuff.php: I4b2cf1715538 (duration: 00m 12s)
• 01:05 twentyafterfour: phab is back
• 01:03 logmsgbot: ori Synchronized php-1.26wmf14/includes/libs/objectcache/APCBagOStuff.php: I4b2cf1715 (duration: 00m 12s)
• 01:01 legoktm: twentyafterfour is upgrading phabricator
• 00:50 yurik: deployed kartotherian fix, still not starting as a service, and no idea why. Have no access to logs. Frustrated.
• 00:46 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/225515/ (duration: 00m 12s)
• 00:23 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: fix extra dollar mark in https://gerrit.wikimedia.org/r/#/c/226336/1/wmf-config/InitialiseSettings.php (duration: 00m 12s)
• 00:02 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/225541/ (duration: 00m 13s)
• 00:02 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/225541/ (duration: 00m 12s)

2015-07-22

• 23:56 cwdent: updated civicrm from 292ad137f6b3ffc818a3bd617ca4f335931091f3 to 83cacfa1e0852ffaf47d2f02e7d843cf6f3bcda4
• 23:55 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: re-try reverted portion of https://gerrit.wikimedia.org/r/#/c/118654/ using NS IDs instead of not-necessarily-defined constants which were causing warning flood (duration: 00m 13s)
• 23:51 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: partially revert https://gerrit.wikimedia.org/r/#/c/118654/ (duration: 00m 12s)
• 23:47 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://wikitech.wikimedia.org/w/index.php?title=Deployments&diff=171578&oldid=171570 (duration: 00m 12s)
• 23:47 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://wikitech.wikimedia.org/w/index.php?title=Deployments&diff=171578&oldid=171570 (duration: 00m 12s)
• 23:40 yurik: deployed kartotherian
• 23:24 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/224393/ (duration: 00m 12s)
• 23:24 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/224393/ (duration: 00m 13s)
• 23:19 logmsgbot: krenair Synchronized php-1.26wmf15/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/226447/ (duration: 00m 13s)
• 22:52 Reedy: populateSitesTable.php finished
• 22:09 Reedy: running in screen as reedy on tin foreachwikiindblist wikidataclient.dblist extensions/Wikidata/extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https
• 22:09 logmsgbot: reedy Synchronized database lists: Add azbwiki to wikidataclient.dblist (duration: 00m 11s)
• 20:55 cscott: updated Parsoid to version 6befc44e
• 20:26 logmsgbot: twentyafterfour Synchronized php-1.26wmf15/includes/libs/MultiHttpClient.php: Deploy https://gerrit.wikimedia.org/r/#/c/226388/ (duration: 00m 12s)
• 19:57 legoktm: re-attributed edits to User:Mirwin~enwiki (T106069)
• 19:34 logmsgbot: demon Finished scap: azbwiki namespace stuff (duration: 42m 57s)
• 19:30 moritzm: updated remaining Ubuntu systems for openssl/export grade update
• 18:51 logmsgbot: demon Started scap: azbwiki namespace stuff
• 18:49 logmsgbot: demon Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 13s)
• 18:48 logmsgbot: demon Synchronized langlist: azbwiki++ (duration: 00m 12s)
• 18:48 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: azbwiki++ (duration: 00m 12s)
• 18:47 logmsgbot: demon Synchronized w/static/images/project-logos/azbwiki.png: azbwiki++ (duration: 00m 12s)
• 18:45 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: azbwiki++
• 18:44 logmsgbot: demon Synchronized database lists: azbwiki++ (duration: 00m 13s)
• 18:18 legoktm: running populateContentModel.php --ns=all --table=page on all medium wikis
• 18:08 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf15
• 18:08 logmsgbot: twentyafterfour Synchronized php-1.26wmf15/extensions/MobileFrontend/includes/MobileFrontend.hooks.php: deploy https://gerrit.wikimedia.org/r/#/c/226313/ (duration: 00m 13s)
• 16:03 _joe_: installed the hhvm 3.6.5 on deployment-prep
• 15:52 _joe_: uploaded hhvm_3.6.5+dfsg1-1+wm1 to reprepro
• 15:47 logmsgbot: thcipriani Synchronized w/static/images/project-logos/lrcwiki.png: SWAT: Update the logo of lrcwiki gerrit:220358 (duration: 00m 13s)
• 15:27 logmsgbot: jynus Synchronized wmf-config: removing db-secondary.php (duration: 00m 12s)
• 15:26 logmsgbot: jynus Synchronized docroot/noc: removing db-secondary.php from the list of symlinks to maintain (duration: 00m 12s)
• 14:20 hashar: enabling puppet on labnodepool1001.eqiad.wmnet
• 14:04 moritzm: added cython_0.20.1+git90-g0e6e38e-1ubuntu2~precise1 to precise-wikimedia on carbon (required for activemq backport on precise)
• 11:37 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: raise db1071 to normal load (duration: 00m 12s)
• 08:03 _joe_: repooling mw1158-60
• 07:22 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jul 22 07:22:36 UTC 2015 (duration 22m 35s)
• 05:22 logmsgbot: ori Synchronized php-1.26wmf14/extensions/Scribunto/common/Base.php: Cherry-pick I53dd1ecb (duration: 00m 13s)
• 05:22 logmsgbot: ori Synchronized php-1.26wmf15/extensions/Scribunto/common/Base.php: Cherry-pick I53dd1ecb (duration: 00m 13s)
• 04:43 logmsgbot: ori Synchronized php-1.26wmf14/extensions/Scribunto/common/Base.php: Revert: Live-hack I53dd1ecb to test impact (duration: 00m 12s)
• 04:35 gwicke: deployed small restbase hotfix d96210f2
• 04:28 logmsgbot: ori Synchronized php-1.26wmf14/extensions/Scribunto/common/Base.php: Live-hack I53dd1ecb to test impact (duration: 00m 13s)
• 04:25 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1071, warm up (duration: 00m 12s)
• 04:14 springle: upgrade db1071 trusty
• 03:10 logmsgbot: LocalisationUpdate completed (1.26wmf15) at 2015-07-22 03:10:23+00:00
• 03:04 logmsgbot: l10nupdate Synchronized php-1.26wmf15/cache/l10n: (no message) (duration: 10m 33s)
• 02:52 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1071 (duration: 00m 11s)
• 02:37 logmsgbot: LocalisationUpdate completed (1.26wmf14) at 2015-07-22 02:37:45+00:00
• 02:33 logmsgbot: l10nupdate Synchronized php-1.26wmf14/cache/l10n: (no message) (duration: 07m 01s)
• 02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jul 22 02:07:33 UTC 2015 (duration 7m 32s)
• 02:03 logmsgbot: LocalisationUpdate failed (1.26wmf15) at 2015-07-22 02:03:19+00:00
• 02:03 logmsgbot: LocalisationUpdate failed (1.26wmf14) at 2015-07-22 02:03:18+00:00

2015-07-21

• 23:45 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings.php: Set $wgVectorResponsive = true on testwiki (duration: 00m 12s)
• 23:39 logmsgbot: catrope Synchronized php-1.26wmf14/extensions/VisualEditor: SWAT (duration: 00m 13s)
• 23:37 logmsgbot: catrope Synchronized php-1.26wmf15/extensions/VisualEditor: SWAT (duration: 00m 13s)
• 23:08 logmsgbot: catrope Synchronized wmf-config/CommonSettings.php: Enable tracking of geo feature usage on enwiki (duration: 00m 12s)
• 23:07 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings.php: Enable tracking of geo feature usage on enwiki (duration: 00m 13s)
• 23:05 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: trying this again: group0 to 1.26wmf15
• 22:59 logmsgbot: twentyafterfour Finished scap: test: syncing 1.26wmf15 again (duration: 20m 51s)
• 22:54 chasemp: 22:50 < chasemp> "then git reset --hard 9588d0a6844fc9cc68372f4bf3e1eda3cffc8138 in /etc/zuul/wikimedia"
• 22:51 chasemp: gallium 'service zuul stop && service zuul-merger stop && sudo apt-get install zuul=2.0.0-304-g685ca22-wmf1precise1' DOWNGRADE due to errors
• 22:39 logmsgbot: twentyafterfour Started scap: test: syncing 1.26wmf15 again
• 22:27 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: revert group0 to 1.26wmf15
• 22:26 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.26wmf15
• 22:20 ori: Accepted mw1090's minion key on palladium
• 21:21 logmsgbot: twentyafterfour Finished scap: sync 1.26wmf15 branch + localization cache, remove wmf8 (duration: 27m 32s)
• 20:53 logmsgbot: twentyafterfour Started scap: sync 1.26wmf15 branch + localization cache, remove wmf8
• 20:53 logmsgbot: twentyafterfour Purged l10n cache for 1.26wmf11
• 20:52 logmsgbot: twentyafterfour Purged l10n cache for 1.26wmf10
• 20:51 logmsgbot: twentyafterfour Purged l10n cache for 1.26wmf9
• 20:28 hasharConfcall: Zuul no more report any result back to Gerrit :( Fix being deployed
• 19:56 ori: Dropping AccountAudit table on all wikis (T105894)
• 19:45 logmsgbot: ori Synchronized wmf-config: I3887fd6c: Disable AccountAudit (duration: 00m 12s)
• 18:07 logmsgbot: ori Synchronized php-1.26wmf14/extensions/Scribunto: I0e5f2d3b2: Updated mediawiki/core Project: mediawiki/extensions/Scribunto 5af0350e2d09444db279f58504967d0e9b154534 (duration: 00m 13s)
• 18:06 logmsgbot: ori Synchronized php-1.26wmf14/extensions/WikimediaEvents: I0e5f2d3b2: Updated mediawiki/core Project: mediawiki/extensions/WikimediaEvents 968890f1a256a08a02925e4bdb53a8e8d64aacea (duration: 00m 13s)
• 17:08 _joe_: restarted logmsgbot, ircecho on neon
• 16:20 logmsgbot: thcipriani Synchronized php-1.26wmf14/extensions/Wikidata: SWAT: Update Wikibase: Add api featureLog for ungroupedlist param gerrit:226086 (duration: 00m 20s)
• 16:01 logmsgbot: thcipriani Synchronized php-1.26wmf13/extensions/Wikidata: SWAT: Update Wikibase: Add api featureLog for ungroupedlist param gerrit:226086 (duration: 00m 20s)
• 15:37 godog: cleanup ganglia temp files on uranium
• 15:34 logmsgbot: thcipriani Synchronized php-1.26wmf14/includes/filerepo/file/File.php: SWAT: Thumbnail logging and stats part II gerrit:225936 (duration: 00m 12s)
• 15:34 logmsgbot: thcipriani Synchronized php-1.26wmf14/thumb.php: SWAT: Thumbnail logging and stats part I gerrit:225936 (duration: 00m 12s)
• 15:29 logmsgbot: thcipriani Synchronized php-1.26wmf14/includes/filerepo/file/File.php: SWAT: Thumbnail logging and stats part II gerrit:225936 (duration: 00m 13s)
• 15:28 logmsgbot: thcipriani Synchronized php-1.26wmf14/thumb.php: SWAT: Thumbnail logging and stats part I gerrit:225936 (duration: 00m 11s)
• 15:20 cmjohnson1: re-installing mw1090
• 15:12 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Offer 400px as a thumbnail size available in Special:Preferences gerrit:226051 (duration: 00m 12s)
• 15:08 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Assign thumbnail access log to Monolog debug channel gerrit:225935 (duration: 00m 13s)
• 13:57 _joe_: depooling mw1158-60 from the imagescaler pool, to test HHVM-only imagescalers
• 05:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul 21 05:08:32 UTC 2015 (duration 8m 31s)
• 02:27 logmsgbot: LocalisationUpdate completed (1.26wmf14) at 2015-07-21 02:26:59+00:00
• 02:23 logmsgbot: l10nupdate Synchronized php-1.26wmf14/cache/l10n: (no message) (duration: 06m 55s)
• 02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul 21 02:07:22 UTC 2015 (duration 7m 21s)
• 02:03 logmsgbot: LocalisationUpdate failed (1.26wmf14) at 2015-07-21 02:03:11+00:00

2015-07-20

• 23:43 gwicke: removed experimental nodes (1008, 1009) from system.peers on production C* nodes
• 21:29 ejegg: updated fundraising/tools from 9a9e7881d25f101cc612cfae6375c0a1c9b0f55d to 3e0e3ae799a507b378d0ece3e71631b10b361329
• 20:55 XenoRyet: updated payments from ebb1a9e52172a4793cf5feb33220b4d7edfcad70 to 152a64a035a59e67b4469223b8f83609bae523a3
• 19:40 gwicke: (eevans, gwicke) removed *.hprof heap dumps from /var/lib/cassandra, freeing up a lot of space especially on 1004 & 1005
• 18:22 gwicke: deployed restbase 0951a6d to remaining nodes
• 17:55 gwicke: canary restbase deploy of 0951a6d on restbase1001
• 16:44 godog: powercycle mw1090, no console no anything
• 15:31 ejegg: updated AstroPay curl timeout setting on payments to 12 seconds
• 05:32 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jul 20 05:32:31 UTC 2015 (duration 32m 30s)
• 02:28 logmsgbot: LocalisationUpdate completed (1.26wmf14) at 2015-07-20 02:28:03+00:00
• 02:24 logmsgbot: l10nupdate Synchronized php-1.26wmf14/cache/l10n: (no message) (duration: 07m 07s)
• 02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jul 20 02:07:34 UTC 2015 (duration 7m 33s)
• 02:03 logmsgbot: LocalisationUpdate failed (1.26wmf14) at 2015-07-20 02:03:24+00:00
• 00:02 mutante: DNS update - adding language "azb" to langlist

2015-07-19

• 20:52 logmsgbot: krenair Synchronized w/static/images/project-logos/arbcom_enwiki.png: https://gerrit.wikimedia.org/r/#/c/225822/ (duration: 00m 12s)
• 19:10 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: Ic0573f26: Follow-up for I189d748: whitelist 'archive.org' too (duration: 00m 12s)
• 19:06 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: I189d748a: Whitelist *.archive.org for wgCopyUploadsDomains (T106293) (duration: 00m 13s)
• 18:29 logmsgbot: hoo Synchronized wmf-config/CommonSettings.php: Enable IP user page creation on fawiki's Draft ns (duration: 00m 11s)
• 18:18 logmsgbot: ori Synchronized php-1.26wmf14/includes/site/SiteSQLStore.php: I0e5f2d3b2: Use CACHE_ACCEL for SiteLists if on HHVM (duration: 00m 12s)
• 17:37 logmsgbot: ori Synchronized wmf-config: Ib508a440: Undeploy VectorBeta (Task: T87489) (duration: 00m 13s)
• 17:27 logmsgbot: krenair Synchronized w/static/images/project-logos/arbcom_enwiki.png: https://gerrit.wikimedia.org/r/#/c/225718/ (duration: 00m 12s)
• 17:21 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/225705/ (duration: 00m 12s)
• 17:14 logmsgbot: krenair Synchronized w/static/images/project-logos/arbcom_enwiki.png: https://gerrit.wikimedia.org/r/#/c/225705/ (duration: 00m 12s)
• 05:10 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jul 19 05:10:10 UTC 2015 (duration 10m 9s)
• 02:27 logmsgbot: LocalisationUpdate completed (1.26wmf14) at 2015-07-19 02:27:35+00:00
• 02:23 logmsgbot: l10nupdate Synchronized php-1.26wmf14/cache/l10n: (no message) (duration: 07m 04s)
• 02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jul 19 02:07:15 UTC 2015 (duration 7m 14s)
• 02:03 logmsgbot: LocalisationUpdate failed (1.26wmf14) at 2015-07-19 02:03:05+00:00

2015-07-18

• 20:58 logmsgbot: legoktm Synchronized wmf-config/InitialiseSettings-labs.php: labs only (duration: 00m 12s)
• 20:44 YuviPanda: restarted etherpad
• 18:56 akosiaris: reinstall labsdb1004
• 16:36 paravoid: Ganglia is up :)
• 16:09 Krenair: Ganglia seems down
• 15:42 Krenair: Doing T44180
• 05:28 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jul 18 05:28:25 UTC 2015 (duration 28m 24s)
• 02:34 logmsgbot: LocalisationUpdate completed (1.26wmf14) at 2015-07-18 02:34:29+00:00
• 02:30 logmsgbot: l10nupdate Synchronized php-1.26wmf14/cache/l10n: (no message) (duration: 07m 19s)
• 02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jul 18 02:07:38 UTC 2015 (duration 7m 37s)
• 02:03 logmsgbot: LocalisationUpdate failed (1.26wmf14) at 2015-07-18 02:03:29+00:00
• 00:49 ejegg: restored recurring globalcollect batch size of 250
• 00:09 ejegg: updated civicrm from 78de1b9b74934984af3099afe9192fa53011bdaa to 292ad137f6b3ffc818a3bd617ca4f335931091f3

2015-07-17

• 21:51 ejegg: updated civicrm from 0acac037ce0c9a64e94a475463deb2d47e84193a to 78de1b9b74934984af3099afe9192fa53011bdaa
• 20:53 matt_flaschen: Manually fixed issue in mediawikiwiki LQT thread table with rename of Ecliptica to Entropy. https://phabricator.wikimedia.org/T106122#1461380
• 20:03 hashar: stopping Zuul to get rid of a faulty registered function "build:Global-Dev Dashboard Data". Job is gone already.
• 17:50 ejegg: updated civicrm from fa724dd2e2e69545d81015c943cb7f52cf6de8e1 to 0acac037ce0c9a64e94a475463deb2d47e84193a
• 16:49 gwicke: restarted restbase on restbase1001
• 15:04 gwicke: restarted RB thinner scripts, see https://phabricator.wikimedia.org/T105706
• 14:10 urandom: restart restbase service on restbase1006
• 14:07 urandom: restart restbase service on restbase1003
• 14:05 urandom: restart restbase service on restbase1002
• 13:56 godog: apache2ctl graceful on fluorine antimony argon caesium helium
• 13:43 godog: apache2ctl graceful on netmon1001
• 11:24 hashar: rebooted labnodepool1001.eqiad.wmnet . Accidentally deleted the whole /dev which freeze everything :(
• 10:21 _joe_: repooling mw1158
• 09:08 _joe_: depooling mw1158, repooling mw1156,7
• 07:51 _joe_: depooled mw1156,7 for reimaging
• 04:53 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jul 17 04:53:56 UTC 2015 (duration 53m 55s)
• 03:31 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1030 (duration: 00m 12s)
• 02:30 logmsgbot: LocalisationUpdate completed (1.26wmf14) at 2015-07-17 02:30:03+00:00
• 02:26 logmsgbot: l10nupdate Synchronized php-1.26wmf14/cache/l10n: (no message) (duration: 05m 55s)
• 02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jul 17 02:07:22 UTC 2015 (duration 7m 20s)
• 02:03 logmsgbot: LocalisationUpdate failed (1.26wmf14) at 2015-07-17 02:03:12+00:00
• 01:30 mutante: git pull origin on strontium

2015-07-16

• 21:27 ori: bounced nutcracker on mw1139 as well. hashar noticed flood of errors from these hosts on https://logstash.wikimedia.org/#/dashboard/elasticsearch/mediawiki-errors . lack of monitoring / alerts is troubling.
• 21:26 ori: bounced nutcracker on mw1128 and mw1134
• 20:50 mutante: iegreview tool - short maintenance downtime
• 19:39 YuviPanda: imported aspell-id from ubuntu to jessie-wikimedia - needed by ores, simple package that I am not sure why it is not in jessie
• 19:20 logmsgbot: twentyafterfour Synchronized php-1.26wmf14/includes/db/LoadMonitor.php: Deploying Hotfix for T105373 (duration: 00m 13s)
• 18:40 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: all wikis to 1.26wmf14
• 18:26 ejegg: changed batch size from 250 to 1 in RGC jenkins job
• 18:22 ejegg: updated civicrm from 24e0fc854433ea4982e94a0fd2f8bdad8f8dcad7 to fa724dd2e2e69545d81015c943cb7f52cf6de8e1
• 16:56 Jeff_Green: authdns update to rename lutetium.wm.o
• 16:08 hashar_: kept nodepool stopped on labnodepool1001.eqiad.wmnet because it spams the cron log
• 15:57 logmsgbot: demon Synchronized multiversion/MWMultiVersion.php: prod no-op, beta change (duration: 00m 13s)
• 15:54 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/224975/ (duration: 00m 12s)
• 15:27 logmsgbot: thcipriani Synchronized php-1.26wmf14/extensions/Math/MathMathML.php: SWAT: Fix: Undefined variable passed hook gerrit:225058 (duration: 00m 12s)
• 15:03 ejegg: updated payments from 4ca95d55a9745c05ccfbb16ee6f23a6f75328824 to ebb1a9e52172a4793cf5feb33220b4d7edfcad70
• 12:21 dcausse: es1.6 upgrade: all done
• 11:32 dcausse: restarted gmond on elastic1024
• 11:06 mobrovac: citoid deploying ff90869
• 10:56 dcausse: es1.6 upgrade: upgrade elastic1031
• 10:25 mobrovac: citoid rolled back to ffbaf6d
• 10:10 mobrovac: citoid deploying 5aeb0fc
• 10:05 dcausse: es1.6 upgrade: upgrade elastic1030
• 09:38 dcausse: es1.6 upgrade: upgrade elastic1029
• 08:42 dcausse: es1.6 upgrade: upgrade elastic1028
• 07:31 dcausse: es1.6 upgrade: upgrade elastic1027
• 07:22 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jul 16 07:22:49 UTC 2015 (duration 22m 48s)
• 05:53 dcausse: es1.6 upgrade: upgrade elastic1026
• 05:31 logmsgbot: krenair Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 12s)
• 05:24 logmsgbot: krenair Synchronized php-1.26wmf14/extensions/WikimediaMaintenance/dumpInterwiki.php: https://gerrit.wikimedia.org/r/#/c/225008/ (duration: 00m 13s)
• 04:38 logmsgbot: krenair Synchronized php-1.26wmf13/extensions/WikimediaMaintenance/dumpInterwiki.php: https://gerrit.wikimedia.org/r/#/c/225006/ (duration: 00m 13s)
• 03:54 manybubbles: es1.6 upgrade: upgrade elastic1025
• 03:19 logmsgbot: LocalisationUpdate completed (1.26wmf14) at 2015-07-16 03:19:37+00:00
• 03:13 logmsgbot: l10nupdate Synchronized php-1.26wmf14/cache/l10n: (no message) (duration: 10m 23s)
• 02:46 logmsgbot: LocalisationUpdate completed (1.26wmf13) at 2015-07-16 02:46:03+00:00
• 02:43 manybubbles: es1.6 upgrade: upgrade elastic1024
• 02:39 logmsgbot: l10nupdate Synchronized php-1.26wmf13/cache/l10n: (no message) (duration: 10m 50s)
• 02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jul 16 02:07:55 UTC 2015 (duration 7m 54s)
• 02:03 logmsgbot: LocalisationUpdate failed (1.26wmf14) at 2015-07-16 02:03:31+00:00
• 02:03 logmsgbot: LocalisationUpdate failed (1.26wmf13) at 2015-07-16 02:03:30+00:00
• 01:41 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/214981/ (duration: 00m 12s)
• 01:22 manybubbles: es1.6 upgrade: upgrade elastic1023

2015-07-15

• 23:36 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/221885/ (duration: 00m 13s)
• 23:22 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/209840/ (duration: 00m 12s)
• 23:16 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/194075/ (duration: 00m 12s)
• 23:10 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/224799/ (duration: 00m 13s)
• 23:09 logmsgbot: krenair Synchronized docroot/noc: https://gerrit.wikimedia.org/r/#/c/175755/ (duration: 00m 13s)
• 23:06 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/175755/ (duration: 00m 12s)
• 22:23 csteipp: deploy patch for T105305 to wmf13/14
• 22:06 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/223843/ (duration: 00m 12s)
• 21:59 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/222584/ (duration: 00m 13s)
• 21:54 manybubbles: es1.6 upgrade: upgrade elastic1022
• 21:37 manybubbles: es1.6 upgrade: upgrade elastic1021
• 21:09 logmsgbot: twentyafterfour Synchronized php-1.26wmf14: Really Sync If0237cdd0d66634d75b2bab8bc4292c0f3ef75ef this time (duration: 01m 32s)
• 20:41 bblack: restarted salt-master service on palladium
• 20:33 bblack: globally cleaning up dangling symlinks left in /etc/certs from before Id7d2447 via salted 'find /etc/ssl/certs -type l -xtype l|xargs rm'
• 20:30 logmsgbot: twentyafterfour Synchronized php-1.26wmf14: Sync If0237cdd0d66634d75b2bab8bc4292c0f3ef75ef (revert Count API module instantiations and Hook runs) (duration: 01m 48s)
• 20:20 manybubbles: es1.6 upgrade: upgrade elastic1020
• 20:18 RoanKattouw: Running FlowCreateMentionTemplate.php on all Flow wikis
• 20:06 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf14
• 19:50 ejegg: updated civicrm from e29cc5f20b5069afcaff794e628596c1f70d69a3 to 24e0fc854433ea4982e94a0fd2f8bdad8f8dcad7
• 19:06 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/224408/ (duration: 00m 12s)
• 19:01 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/222792/ (duration: 00m 13s)
• 19:00 logmsgbot: krenair Synchronized wmf-config/wikitech.php: https://gerrit.wikimedia.org/r/#/c/222792/ (duration: 00m 12s)
• 18:58 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/222776/ (duration: 00m 13s)
• 18:57 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/222776/ (duration: 00m 13s)
• 18:40 ejegg: updated civicrm from f4219bc8eca5e4db633da07b6ac9e2505cfbae16 to e29cc5f20b5069afcaff794e628596c1f70d69a3
• 18:39 logmsgbot: krenair Synchronized wmf-config/throttle.php: throttle labswiki account creations from hackathon at 500 (duration: 00m 12s)
• 18:39 logmsgbot: twentyafterfour Finished scap: group0 to 1.26wmf14 (duration: 32m 34s)
• 18:21 manybubbles: es1.6 upgrade: upgrading elastic1019
• 18:20 Jeff_Green: authdns-update shifting to service-oriented hostnames for fundraising cluster
• 18:06 logmsgbot: twentyafterfour Started scap: group0 to 1.26wmf14
• 17:55 ejegg: updated civicrm from 6560cefa8d7e68e35e30b310d6691ab57798a4c9 to f4219bc8eca5e4db633da07b6ac9e2505cfbae16
• 17:34 Jeff_Green: authdns-update to remove boron.wm.o
• 17:22 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: partially revert https://gerrit.wikimedia.org/r/#/c/224420/1/wmf-config/CommonSettings.php - doesnt quite work (duration: 00m 13s)
• 17:17 Jeff_Green: authdns-update to remove aluminium, also lanthanum by preexisting commit
• 16:45 andrewbogott: rebooting labvirt1005
• 16:43 mutante: accepting unaccepted salt keys for ganeti VMs ,planet, bromine, krypton
• 16:39 mutante: krypton - signing puppet cert, initial run
• 16:26 andrewbogott: woo, first try!
• 16:23 andrewbogott: trying to kill labvirt1005 via repeated instance suspend/resume
• 16:04 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/224420/ (duration: 00m 12s)
• 16:03 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/224420/ (duration: 00m 12s)
• 16:01 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/224808/ (duration: 00m 12s)
• 15:58 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/222581/ (duration: 00m 11s)
• 15:35 logmsgbot: krenair Synchronized database lists: (no message) (duration: 00m 11s)
• 15:29 logmsgbot: krenair Synchronized docroot/noc/createTxtFileSymlinks.sh: https://gerrit.wikimedia.org/r/#/c/139326/ (duration: 00m 12s)
• 15:27 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/139326/ (duration: 00m 12s)
• 15:20 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/139326/ (duration: 00m 11s)
• 14:33 logmsgbot: legoktm Synchronized wmf-config/CommonSettings.php: Set $wgCentralAuthStrict = true; (duration: 00m 12s)
• 14:22 legoktm: sync failed on mw1090.eqiad.wmnet, read only filesystem
• 14:20 logmsgbot: legoktm Synchronized php-1.26wmf13/extensions/CentralAuth/includes/CentralAuthPlugin.php: Add log entry for $wgCentralAuthStrict failures if SULMigration is enabled (duration: 00m 13s)
• 13:55 dcausse: es1.6 upgrade: upgrade elastic1018
• 13:24 springle: entry below not mw1216 fault, but r/o filesystem error on mw1090
• 13:15 springle: sync-common on mw1216 after sync-file from tin failed non-zero exit status 12
• 13:12 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1022 T105879 (duration: 00m 12s)
• 11:43 dcausse: es1.6 upgrade: upgrade elastic1017
• 08:27 dcausse: es1.6 upgrade: upgrade elastic1016
• 06:31 dcausse: es1.6 upgrade: upgrade elastic1015
• 05:40 dcausse: es1.6 upgrade: upgrade elastic1014
• 05:10 springle: db1030 busy removing table partitioning
• 04:28 manybubbles: es1.6 upgrade: lowered the shard transfer settings back to our normal rate. going to bed.
• 04:12 manybubbles: es1.6 upgrade: upgrade elastic1013
• 03:49 springle: upgrade db1030 trusty
• 03:29 manybubbles: es1.6 upgrade: upgrade elastic1012
• 03:14 logmsgbot: LocalisationUpdate completed (1.26wmf13) at 2015-07-15 03:14:21+00:00
• 03:10 logmsgbot: reedy Synchronized php-1.26wmf13/cache/l10n: (no message) (duration: 13m 32s)
• 03:03 manybubbles: es1.6 upgrade: raised limits on shard migration rate - should speed up the restart. we should lower it before we do restarts during europe's morning
• 02:10 Reedy: Running LU manually to see what's wrong with it
• 02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jul 15 02:07:48 UTC 2015 (duration 7m 47s)
• 02:02 logmsgbot: LocalisationUpdate failed (1.26wmf13) at 2015-07-15 02:02:55+00:00

2015-07-14

• 23:46 manybubbles: es1.6 upgrade: upgraded elastic1011
• 23:22 bblack: updating nginx to 1.9.3-1+wmf1 on cp*
• 23:17 bblack: reprepro: nginx for jessie-wikimedia/main bumped to 1.9.3-1+wmf1
• 22:22 ejegg: updated civicrm from 04efc7d5c7bbb068f907125f2184692aee676123 to 6560cefa8d7e68e35e30b310d6691ab57798a4c9
• 21:29 Reedy: mw1090 fs is ro
• 21:28 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Fix testwiki
• 21:05 _joe|AFK: depooling mw1090, ext4 errors in syslog, filesystem mounted read-only
• 21:01 logmsgbot: twentyafterfour Synchronized wmf-config/CommonSettings.php: revert LCStoreStaticArray (duration: 00m 12s)
• 20:59 logmsgbot: twentyafterfour Finished scap: testwiki to 1.26wmf14 and rebuild localization cache (duration: 72m 45s)
• 20:42 bblack: undoing LCStoreStaticArray because appservers look unhealthy, using ori's command: 'salt -G deployment_target:scap/scap cmd.run "rm /etc/lcstore"'
• 19:46 logmsgbot: twentyafterfour Started scap: testwiki to 1.26wmf14 and rebuild localization cache
• 19:23 manybubbles: es1.6 step iforget: upgrade elasticsearch on elastic1010
• 17:41 mutante: terbium: /usr/local/bin/foreachwiki extensions/Echo/maintenance/processEchoEmailBatch.php
• 17:10 dcausse: es1.6 step 10: upgrade elastic1009
• 16:23 mutante: bromine - apt-get upgrade
• 15:08 logmsgbot: manybubbles Synchronized php-1.26wmf13/extensions/UniversalLanguageSelector/: SWAT add some hooks to extension.json (duration: 00m 13s)
• 14:34 gwicke: started RESTBase revision thin-out script for html and data-parsoid on wikimedia domains
• 14:01 dcausse: es1.6 step 9: upgrade elastic1008
• 12:48 _joe_: reimaging mw1155
• 12:17 ori: Logging a message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log.
• 11:28 dcausse: es1.6 step 8: upgrade elastic1007
• 11:25 _joe_: repooling mw1154 with HHVM
• 10:12 _joe_: stopped poolcounter on mw1154
• 10:06 _joe_: reimaging mw1154
• 07:49 dcausse: es1.6 step 7: upgrade elastic1006
• 07:09 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul 14 07:09:10 UTC 2015 (duration 9m 9s)
• 06:48 dcausse: es1.6 step 6: upgrade elastic1005
• 06:41 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I9c9bf0f4: Use LCStoreStaticArray unconditionally (duration: 03m 02s)
• 05:26 ori: Cleaned up now-unused hhbc files from /run/hhvm/cache on job runners
• 04:58 ori: Enabling LCStoreStaticArray in production. May be reverted by running: 'salt -G deployment_target:scap/scap cmd.run "rm /etc/lcstore"' on palladium.
• 04:48 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Follow-up for Ieb62ee050e: allow LCStoreStaticArray in server mode (duration: 00m 13s)
• 02:35 logmsgbot: LocalisationUpdate completed (1.26wmf13) at 2015-07-14 02:35:21+00:00
• 02:31 logmsgbot: l10nupdate Synchronized php-1.26wmf13/cache/l10n: (no message) (duration: 07m 27s)
• 02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul 14 02:07:32 UTC 2015 (duration 7m 30s)
• 02:02 logmsgbot: LocalisationUpdate failed (1.26wmf13) at 2015-07-14 02:02:33+00:00
• 01:22 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1037; depool db1030 (duration: 00m 13s)

2015-07-13

• 23:22 logmsgbot: catrope Synchronized php-1.26wmf13/extensions/VisualEditor: SWAT (duration: 00m 11s)
• 23:11 logmsgbot: catrope Synchronized php-1.26wmf13/extensions/Flow/includes/Parsoid/Utils.php: Add title to Parsoid exception logging (duration: 00m 12s)
• 22:45 logmsgbot: legoktm Synchronized wmf-config: Revert "Set $wgCentralAuthStrict = true;" (duration: 00m 13s)
• 22:41 logmsgbot: legoktm Synchronized wmf-config/CommonSettings.php: Set $wgCentralAuthStrict = true; (duration: 00m 13s)
• 22:41 logmsgbot: legoktm Synchronized wmf-config/InitialiseSettings.php: Set $wgCentralAuthStrict = true; (duration: 00m 12s)
• 22:16 logmsgbot: legoktm Synchronized php-1.26wmf13/includes/User.php: Add 'AuthPluginStrict' log to identify users who are unable to authenticate (duration: 00m 13s)
• 22:15 logmsgbot: legoktm Synchronized php-1.26wmf13/includes/api/ApiMain.php: Revert "Revert "Revert Count API module instantiations and Hook runs"" (duration: 00m 12s)
• 22:15 logmsgbot: legoktm Synchronized php-1.26wmf13/includes/Hooks.php: Revert "Revert "Revert Count API module instantiations and Hook runs"" (duration: 00m 13s)
• 22:13 ejegg: updated payments from ec34ebf61e5962f66b807abdcb519ff323d41e8e to 4ca95d55a9745c05ccfbb16ee6f23a6f75328824
• 22:00 manybubbles: es1.6 step 4: upgrade elastic1003
• 21:54 ori: Debugging metric issue on graphite1001, brief stats drop possible
• 21:32 legoktm: renaming ~3k users who were originally missed for SULF
• 21:08 logmsgbot: ori Synchronized php-1.26wmf13/includes/Hooks.php: (no message) (duration: 00m 12s)
• 21:08 logmsgbot: ori Synchronized php-1.26wmf13/includes/api/ApiMain.php: (no message) (duration: 00m 13s)
• 20:42 logmsgbot: ori Synchronized php-1.26wmf13/includes/api/ApiMain.php: f9c89d2814: Revert "Revert Count API module instantiations and Hook runs" (duration: 00m 13s)
• 20:30 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Ieb62ee05: Temporary hack to facilitate migration of l10n cache implementations (duration: 00m 11s)
• 19:42 hoo: Updated Wikidata's property suggester with data from today's json dump
• 19:24 manybubbles_: es1.6 step 3: upgrade elastic1002
• 19:08 legoktm: running populateContentModel.php --table=page on all small wikis
• 19:01 andrewbogott: two of two
• 19:01 mutante: morebots - are you 1.7.11 ?
• 19:01 andrewbogott: one of two
• 18:52 legoktm: running populateContentModel.php --table=page on testwiki
• 18:29 manybubbles_: es1.6 step 2: shut down extra instance of elasticsearch on elastic1021
• 17:39 andrewbogott: this is the second test log of three
• 17:39 andrewbogott: this is the first test log of three
• 17:36 mutante: included adminbot_1.7.11 in APT repo
• 16:31 andrewbogott: wikidata-dev updated local puppet and rebooting property-suggester
• 16:08 logmsgbot: krenair Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/224087/ (duration: 00m 12s)
• 16:07 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/224087/ (duration: 00m 12s)
• 15:11 manybubbles_: all done SWATing.
• 15:09 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT enable footer contact link on ukwiki (duration: 00m 11s)
• 14:55 manybubbles_: after upgrading elasticsearch its init script no longer shuts down the old version of elasticsearch. so you have to manually kill it. that means the upgrade instructions will be "special" this time around. hopefully this is a one time thing.
• 14:45 manybubbles_: es1.6 step 1: upgrade elasticsearch on elastic1001 -starting
• 14:45 manybubbles_: es1.6 step 0: successfully synced new versions of plugins
• 14:30 manybubbles_: es1.6 step 0: sync new versions of plugins
• 14:30 manybubbles_: starting the elasticsearch 1.6.0 upgrade
• 13:13 bblack: updating nginx/bind on cp*
• 13:07 bblack: updating openssl on cp*
• 13:02 logmsgbot: krenair Synchronized php-1.26wmf13/extensions/Cite/extension.json: https://gerrit.wikimedia.org/r/#/c/224407/ - unbreak VE mobile, https://phabricator.wikimedia.org/T105686 (duration: 00m 12s)
• 10:58 mobrovac: restbase deploying 6dec79d
• 10:22 logmsgbot: ori Synchronized php-1.26wmf13/maintenance/rebuildLocalisationCache.php: 117f60a171: rebuildLocalisationCache: don't limit memory usage (duration: 00m 12s)
• 08:52 godog: bounce graphite-web on graphite1001
• 08:51 godog: bounce carbon daemons on graphite1001
• 08:50 godog: upgrade graphite to 0.9.13 on graphite1001 and bounce one instance of carbon/cache
• 07:29 logmsgbot: ori Synchronized php-1.26wmf13/includes/cache/LCStoreStaticArray.php: I3f63594a4: Fix variable name (follows Ib2c5856d) (duration: 00m 11s)
• 06:25 logmsgbot: LocalisationUpdate failed: git pull of core failed
• 06:24 ori: Experimenting with altering the localisation cache implementation for testwiki, operations/mediawiki-config on tin will have a local hack for a little bit
• 05:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jul 13 05:07:32 UTC 2015 (duration 7m 31s)
• 02:25 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jul 13 02:25:58 UTC 2015 (duration 25m 57s)
• 02:23 logmsgbot: LocalisationUpdate completed (1.26wmf13) at 2015-07-13 02:23:43+00:00
• 02:20 logmsgbot: l10nupdate Synchronized php-1.26wmf13/cache/l10n: (no message) (duration: 06m 16s)
• 02:10 logmsgbot: LocalisationUpdate completed (1.26wmf13) at 2015-07-13 02:10:25+00:00
• 02:10 logmsgbot: l10nupdate Synchronized php-1.26wmf13/cache/l10n: (no message) (duration: 00m 34s)
• 01:47 springle: restarted labsdb1002 mysqld while troubleshooting replication

2015-07-12

• 14:59 bblack: upgraded most packages on sodium
• 14:48 bblack: upgraded apache2 to 2.2.22-1ubuntu1.9 on: antimony argon caesium fluorine helium iodine logstash1001 logstash1003 magnesium neon netmon1001 rhodium stat1001 ytterbium
• 04:49 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jul 12 04:49:08 UTC 2015 (duration 49m 7s)
• 02:26 logmsgbot: LocalisationUpdate completed (1.26wmf13) at 2015-07-12 02:26:52+00:00
• 02:25 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jul 12 02:25:33 UTC 2015 (duration 25m 32s)
• 02:23 logmsgbot: l10nupdate Synchronized php-1.26wmf13/cache/l10n: (no message) (duration: 06m 12s)
• 02:10 logmsgbot: LocalisationUpdate completed (1.26wmf13) at 2015-07-12 02:10:00+00:00
• 02:09 logmsgbot: l10nupdate Synchronized php-1.26wmf13/cache/l10n: (no message) (duration: 00m 34s)

2015-07-11

• 19:48 jynus: stopping labsdb1002 after table corruption has been detected
• 19:37 urandom: from restbase1002, starting revision culling process (node thin_out_key_rev_value_data.js `hostname -i` local_group_wikimedia_T_parsoid_html 2>&1 | tee >(gzip -c > local_group_wikimedia_T_parsoid_html.log.`date +%s`.gz))
• 19:33 urandom: restbase: setting gc_grace_seconds to 604800 (1 week) on local_group_wikipedia_T_parsoid_html.data
• 04:55 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jul 11 04:55:56 UTC 2015 (duration 55m 55s)
• 04:21 bd808: Logstash cluster upgrade complete! Kibana working again
• 04:21 bd808: Upgraded Elasticsearch to 1.6.0 on logstash1006
• 04:12 bd808: rebooting logstash1006
• 04:06 bd808: logstash1005 fully recovered all shards
• 03:21 logmsgbot: mattflaschen Synchronized php-1.26wmf13/extensions/Flow/includes/Parsoid/Utils.php: Bump Flow to encode page name when sending to Parsoid (duration: 00m 13s)
• 02:28 logmsgbot: LocalisationUpdate completed (1.26wmf13) at 2015-07-11 02:28:18+00:00
• 02:25 logmsgbot: l10nupdate Synchronized php-1.26wmf13/cache/l10n: (no message) (duration: 06m 07s)
• 02:25 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jul 11 02:25:19 UTC 2015 (duration 25m 18s)
• 02:09 logmsgbot: LocalisationUpdate completed (1.26wmf13) at 2015-07-11 02:09:45+00:00
• 02:09 logmsgbot: l10nupdate Synchronized php-1.26wmf13/cache/l10n: (no message) (duration: 00m 35s)
• 00:46 bd808: Upgraded Elasticsearch to 1.6.0 on logstash1005; replicas recovering now
• 00:34 bd808: rebooting logstash1005
• 00:30 bd808: logstash1004 fully recovered all shards

2015-07-10

• 22:51 mutante: tendril: very short maintenance downtime
• 20:10 bd808: `service elasticsearch start` not starting on logstash1004; investigating
• 20:07 bd808: ran apt-get upgrade on logstash1004
• 19:52 mutante: adminbot - built and imported 1.7.10 into APT repo
• 19:43 bd808: rebooting logstash1004
• 19:40 bd808: Kibana seems to be broken by mixed 1.6.0/1.3.9 cluster
• 19:32 bd808: kibana not seeing indices after upgrading elasticsearch to 1.6.0; investigating
• 19:26 bd808: Upgraded logstash1003 to elasticsearch 1.6.0
• 19:22 bd808: Upgraded logstash1002 to elasticsearch 1.6.0
• 19:19 bd808: Upgraded logstash1001 to elasticsearch 1.6.0
• 19:10 logmsgbot: krenair Synchronized php-1.26wmf13/extensions/VisualEditor/lib/ve/src/ce/nodes/ve.ce.TableNode.js: https://gerrit.wikimedia.org/r/#/c/224122/ (duration: 00m 12s)
• 18:11 gwicke: ansible -i production restbase -a 'nodetool setcompactionthroughput 120'
• 18:00 gwicke: ansible -i production restbase -a 'nodetool setcompactionthroughput 90'
• 17:49 gwicke: rolling restart of the cassandra cluster to apply https://gerrit.wikimedia.org/r/#/c/224114/
• 17:32 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: prevent race condition on writing settings (duration: 00m 13s)
• 17:26 moritzm: installed python security updates on mc*
• 17:25 Coren: rebooting labstore2001 (experiments with the new raid setup caused the mapper table to fill)
• 16:35 mobrovac: restbase deploying hotfix for T105509
• 15:29 mobrovac: restbase restarted restabse on restbase1004
• 15:25 godog: bounce cassandra on restbae1004
• 13:43 godog: bounce cassandra on restbae1004
• 13:37 _joe_: temporarily repooled mw1031
• 12:40 godog: bounce cassandra on restbae1004
• 07:43 godog: reimage ms-be2013 T105213
• 04:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jul 10 04:36:49 UTC 2015 (duration 36m 48s)
• 04:33 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1037; repool db1030 (revert below) (duration: 00m 12s)
• 04:28 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1037; depool db1030 (duration: 00m 13s)
• 03:14 mutante: re-enabling puppet on tools-exec-1213, working around adminbot package install fail
• 02:59 elee: please log this with the year
• 02:53 andrewbogott: testing the log by logging a test
• 01:50 gwicke: bounced cassandra on restbase1004
• 01:38 jgage: cassandra restarted on restbase1004
• 00:39 urandom: starting restbase1004
• 00:35 logmsgbot: krenair Synchronized php-1.26wmf13/extensions/VisualEditor/modules/ve-mw/ui/inspectors/ve.ui.MWLinkAnnotationInspector.js: https://gerrit.wikimedia.org/r/#/c/223983/ (duration: 00m 12s)
• 00:15 hoo: Updated WikibaseQualityConstraints data on wikidata (wikidatawiki.wbqc_constraints)

July 9

• 23:41 legoktm: deployed patch for T105413
• 23:07 gwicke: bounced cassandra on restbase1004
• 23:02 logmsgbot: catrope Synchronized wmf-config/CommonSettings.php: TitleBlacklist: Don't block account auto-creation (duration: 00m 13s)
• 22:09 logmsgbot: oblivian Synchronized wmf-config/PoolCounterSettings-eqiad.php: I don't think we want to keep poolcounter running on an imagescaler (duration: 00m 12s)
• 21:30 logmsgbot: tgr Synchronized php-1.26wmf13/extensions/OAuth/api/MWOAuthAPI.setup.php: no canonical redirects for requests with OAuth headers (duration: 00m 12s)
• 21:05 tgr: backporting https://gerrit.wikimedia.org/r/#/c/223952/- fixes OAuth which is broken for 1.26wmf13
• 20:47 gwicke: temporarily disabled puppet on cassandra nodes while tweaking settings
• 19:53 legoktm: manually fixing global merge of Yuvipanda->YuviPanda (T104686)
• 19:04 gwicke: bounced cassandra on restbase1004
• 18:29 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: all wikis to 1.26wmf13
• 17:54 gwicke: bounced restbase on restbase1005
• 17:32 ori: installed poolcounter on mw1154
• 17:31 logmsgbot: ori Synchronized wmf-config/PoolCounterSettings-eqiad.php: (no message) (duration: 00m 12s)
• 17:22 cmjohnson1: shutting down helium for a few minutes to move within the same row
• 16:53 gwicke: bounced cassandra on restbase1004
• 16:48 godog: reboot ms-be2013 T105213
• 16:38 gwicke: bounced cassandra on restbase1006
• 16:07 _joe_: repooling mw1152
• 15:57 godog: restart cassandra on restbase1002
• 15:34 gwicke: bounced cassandra on restbase1004
• 15:24 logmsgbot: krenair Synchronized php-1.26wmf12/extensions/ContentTranslation: https://gerrit.wikimedia.org/r/#/c/223739/ (duration: 00m 12s)
• 15:23 logmsgbot: krenair Synchronized php-1.26wmf13/extensions/ContentTranslation: https://gerrit.wikimedia.org/r/#/c/223737/ (duration: 00m 12s)
• 15:23 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/223742/ (duration: 00m 12s)
• 15:09 gwicke: bounced cassandra on restbase1004
• 14:44 gwicke: re-enabled compaction throttling (60mb/s) on cassandra nodes
• 14:44 bblack: reprepro: jessie-wikimedia/backports openssl pkg, 1.0.2c-1 => 1.0.2d-1~wmf1
• 14:29 _joe_: reimaging mw1152 for wiping any leftover local hacks. Depooling, scheduling downtime
• 14:28 moritzm: installed python-django security updates on labmon, netmon and californium
• 14:24 godog: really upgrade python-django on graphite2001
• 13:48 mobrovac: restbase cassandra rolling restart to apply https://gerrit.wikimedia.org/r/223774
• 13:02 godog: upgrade python-django on graphite1001 and graphite2001 following http://www.ubuntu.com/usn/usn-2671-1/
• 11:34 godog: restart cassandra on restbase1001
• 11:22 logmsgbot: krinkle Synchronized php-1.26wmf13/resources/src/mediawiki/mediawiki.util.js: T105265 (duration: 00m 11s)
• 11:21 logmsgbot: krinkle Synchronized php-1.26wmf13/includes/GlobalFunctions.php: T105265 (duration: 00m 12s)
• 11:09 mobrovac: restbase deploying https://gerrit.wikimedia.org/r/#/c/223297/ which bumps the back-end module version ( https://github.com/wikimedia/restbase-mod-table-cassandra/pull/117 )
• 10:53 mobrovac: restbase started thinner 15 days for wikimedia group
• 10:37 mark: Shutdown AMS-IX route server BGP sessions on cr1-esams
• 07:48 logmsgbot: oblivian Synchronized php-1.26wmf13/thumb.php: Re-add fix for thumb.php 404s on HHVM (duration: 00m 13s)
• 06:27 twentyafterfour: restarted apache2 on iridium to fix phab exception
• 06:15 springle: db1037 is repartitioning tables; it will lag intermittently for a day
• 06:05 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jul 9 06:05:30 UTC 2015 (duration 5m 29s)
• 05:23 gwicke: dynamically limited cassandra compaction throughput to 80mb/s; please review https://gerrit.wikimedia.org/r/#/c/223722/ to make this permanent
• 03:01 logmsgbot: LocalisationUpdate completed (1.26wmf13) at 2015-07-09 03:01:13+00:00
• 02:58 logmsgbot: l10nupdate Synchronized php-1.26wmf13/cache/l10n: (no message) (duration: 05m 29s)
• 02:42 logmsgbot: LocalisationUpdate completed (1.26wmf12) at 2015-07-09 02:42:56+00:00
• 02:40 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jul 9 02:40:16 UTC 2015 (duration 40m 15s)
• 02:36 logmsgbot: l10nupdate Synchronized php-1.26wmf12/cache/l10n: (no message) (duration: 10m 32s)
• 02:28 twentyafterfour: restarted phd
• 02:28 twentyafterfour: moved phd log to free disk space on iridium
• 02:24 logmsgbot: LocalisationUpdate completed (1.26wmf13) at 2015-07-09 02:24:00+00:00
• 02:23 logmsgbot: l10nupdate Synchronized php-1.26wmf13/cache/l10n: (no message) (duration: 00m 34s)
• 02:17 logmsgbot: LocalisationUpdate completed (1.26wmf12) at 2015-07-09 02:17:02+00:00
• 02:16 logmsgbot: l10nupdate Synchronized php-1.26wmf12/cache/l10n: (no message) (duration: 00m 47s)
• 02:00 springle: pkg upgrade and restart db1037
• 01:49 gwicke: switched remaining cassandra nodes to JDK8
• 01:37 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1037 (duration: 00m 11s)
• 01:07 mutante: uranium - deleted apache logs older than 90 days
• 00:45 RoanKattouw: Running populateContentModel.php --wiki=cawiki --table=revision --ns=5
• 00:20 RoanKattouw: Ran populateContentModel.php --table=revision for odd-numbered namespaces on officewiki for T105245

July 8

• 23:07 logmsgbot: catrope Synchronized php-1.26wmf13/extensions/Flow: SWAT (duration: 00m 14s)
• 23:06 bd808: Restarted logstash on logstash1001; no hhvm input seen for last hour
• 22:56 gwicke: finished rolling restart of cassandra cluster to apply https://gerrit.wikimedia.org/r/#/c/223495/
• 22:45 mutante: zirconium - stop puppet for role switch
• 22:33 logmsgbot: legoktm Synchronized php-1.26wmf13/includes/changes/EnhancedChangesList.php: Unbreak missing flags in enhanced RC (duration: 00m 12s)
• 22:08 logmsgbot: hoo Synchronized php-1.26wmf13/extensions/Wikidata/: Update Wikibase: Fix JavaScript ULS usage (duration: 00m 20s)
• 21:51 logmsgbot: manybubbles Synchronized php-1.26wmf12/extensions/CirrusSearch/: Stop some fatals in cirrus (duration: 00m 13s)
• 21:41 logmsgbot: bd808 Synchronized php-1.26wmf13/includes/api/ApiMain.php: Revert Count API module instantiations and Hook runs (2/2) (duration: 00m 12s)
• 21:40 logmsgbot: bd808 Synchronized php-1.26wmf13/includes/Hooks.php: Revert Count API module instantiations and Hook runs (1/2) (duration: 00m 12s)
• 21:39 logmsgbot: bd808 Synchronized php-1.26wmf13/extensions/CirrusSearch/includes/CirrusSearch.php: Suppress interwiki results when they would break (duration: 00m 12s)
• 21:08 bblack: graphite: wiped /var/log/upstart/statsite* logs, restarted statsite processes
• 20:56 csteipp: deployed patches for T103022 & T103023
• 20:53 csteipp: deployed patch for T94116 for wmf12/wmf13
• 20:30 gwicke: added explicit exit 1 in /etc/init.d/cassandra on restbase1008 to prevent cassandra from starting up there; is puppet restarting it?
• 20:29 subbu: deployed parsoid sha c4cfc527
• 20:15 gwicke: bounced cassandra on restbase1001
• 20:05 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jul 8 20:05:09 UTC 2015 (duration 5m 8s)
• 19:32 gwicke: stopped cassandra on restbase1008
• 19:27 logmsgbot: twentyafterfour Synchronized php-1.26wmf13: deploying UniversalLanguageSelector commit 2e0990ac9879 (duration: 01m 58s)
• 19:26 urandom: restbase rolling restart
• 18:21 jgage: ran 'kafka preferred-replica-election' to promote analytics1021 back to Leader
• 18:05 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf13
• 17:16 moritzm: installed libwmf security updates on various systems
• 17:09 gwicke: bounced cassandra on restbase1004
• 15:25 mutante: handing over adminship of the "test" mailman list to John F. Lewis (was: Thehelpfulone) due to inactivity
• 13:36 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: raise db1041 load (duration: 00m 13s)
• 12:58 paravoid: manually dpkg -P ferm on potassium
• 12:52 paravoid: rmmod all iptables/netfilter-related modules from potassium
• 11:23 godog: bounce cassandra on restbase1004, heap space
• 11:12 _joe_: mw1153 passed the smoke tests, repooling
• 11:08 godog: bounce cassandra on restbase1004 and restbase1005 'cannot achieve consistency level quorum'
• 10:50 godog: bounce cassandra on restbase1004, death by compaction
• 09:43 ori: _joe_: starting reimaging of mw1153, depooling it and scheduling downtime (at 9:21 UTC)
• 09:42 ori: Nuked /var/lib/carbon/whisper/ResourceLoader on graphite[12]001. Data prior to rollout of I55f0c44cd considered bogus.
• 09:42 ori: morebots, are you OK?
• 09:41 godog: bounce nutcracker on silver
• 09:33 _joe_: starting reimaging of mw1153, depooling it and scheduling downtime (at 9:21 UTC)
• 09:26 hashar: upgraded plugins on jenkins and restarting it
• 09:06 hashar: Jenkins registering jobs with Zuul
• 08:41 hashar: Jenkins is migrating old build histories. Lot of disk IO happening
• 08:11 hashar: shutdowning Jenkins for upgrade.
• 05:57 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jul 8 05:57:10 UTC 2015 (duration 57m 9s)
• 05:46 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1041, warm up (duration: 00m 13s)
• 02:31 logmsgbot: LocalisationUpdate completed (1.26wmf13) at 2015-07-08 02:31:24+00:00
• 02:16 logmsgbot: LocalisationUpdate completed (1.26wmf12) at 2015-07-08 02:16:50+00:00
• 02:16 logmsgbot: l10nupdate Synchronized php-1.26wmf12/cache/l10n: (no message) (duration: 00m 48s)

July 7

• 23:54 jgage: kafka brokers 1018 & 1021 were demoted; i have triggered a leader election and they are leaders again
• 23:05 logmsgbot: catrope Synchronized visualeditor-default.dblist: Enable VE by default on labswiki (duration: 00m 12s)
• 21:56 hoo: Restarted hhvm on mw1003 "Fatal error: Function already defined: wmfLoadInitialiseSettings in /srv/mediawiki/wmf-config/CommonSettings.php on line 187"
• 21:16 logmsgbot: krinkle Synchronized php-1.26wmf13/includes/resourceloader/ResourceLoader.php: T104769 (duration: 00m 13s)
• 20:53 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.26wmf13
• 20:00 logmsgbot: twentyafterfour Finished scap: testwiki to php-1.26wmf13 and rebuild l10n cache (duration: 39m 41s)
• 19:47 gwicke: restarted cassandra on restbase1005
• 19:20 logmsgbot: twentyafterfour Started scap: testwiki to php-1.26wmf13 and rebuild l10n cache
• 19:15 moritzm: installed PHP security updates on all trusty hosts
• 18:58 ejegg: updated payments from a17ee221db0dbde70c92e24fc188379b6dbad613 to ec34ebf61e5962f66b807abdcb519ff323d41e8e
• 18:08 twentyafterfour: restarted apache2 on iridium (phab hotfix)
• 17:10 robh: OTRS update appears to be functioning normally. As such, ending maintenance window.
• 17:06 robh: otrs is now using the new sha256 cert
• 17:00 robh: starting otrs maint window
• 16:58 _joe_: restarted HHVM on mw1026, near to OOM
• 16:47 twentyafterfour: applied hotfix for phabricator bug: https://secure.phabricator.com/D13544
• 16:36 mutante: protactinium - manual iptables rules replaced by puppet/ferm rules
• 16:11 logmsgbot: thcipriani Synchronized php-1.26wmf12/extensions/ContentTranslation/extension.json: Remove default value for ContentTranslationCampaigns (duration: 00m 12s)
• 15:33 jynus: manually editing table mediawiki.ipblocks to fully solve a former software bug
• 15:12 Jeff_Green: ptr records for frack/codfw and authdns-update
• 15:10 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Enable ContentTranslation in enwiki gerrit:222991 (duration: 00m 13s)
• 14:21 jynus: dropping optin_survey_old table from enwiki
• 13:23 akosiaris: restarting gitblit on antimony
• 11:31 mobrovac: restbase restarted cassandra on rb1005
• 11:26 godog: restart cassandra on restbase1004, heap exhausted
• 10:49 godog: restarted cassandra on restbase1005, mutations through the roof
• 08:27 godog: set operations/puppet/cassandra git submodule repo as hidden
• 06:11 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul 7 06:11:46 UTC 2015 (duration 11m 45s)
• 05:51 logmsgbot: krinkle Synchronized php-1.26wmf12/extensions/WikiEditor/modules/jquery.wikiEditor.toolbar.js: I3e965dda1c4 (duration: 00m 12s)
• 02:27 logmsgbot: LocalisationUpdate completed (1.26wmf12) at 2015-07-07 02:27:55+00:00
• 02:24 logmsgbot: l10nupdate Synchronized php-1.26wmf12/cache/l10n: (no message) (duration: 06m 09s)
• 01:12 ori: Re-pooled mw1152 at 20:46 UTC, did not log it then.
• 00:41 springle: upgrade db1041 trusty
• 00:37 logmsgbot: krenair Synchronized php-1.26wmf12/extensions/CentralAuth/includes/CreateLocalAccountJob.php: https://gerrit.wikimedia.org/r/#/c/223211/ (duration: 00m 13s)

July 6

• 23:50 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/221989/ (duration: 00m 12s)
• 23:49 logmsgbot: krenair Synchronized w/static/images/project-logos/mrwikisource.png: https://gerrit.wikimedia.org/r/#/c/221989/ (duration: 00m 13s)
• 23:35 logmsgbot: krenair Synchronized wmf-config/abusefilter.php: https://gerrit.wikimedia.org/r/#/c/223179/ - should be labs-only (duration: 00m 12s)
• 23:32 logmsgbot: krenair Synchronized README: https://gerrit.wikimedia.org/r/#/c/222941/ - ... (duration: 00m 13s)
• 23:27 logmsgbot: krenair Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/221809/ - should be a noop, just doc changes (duration: 00m 13s)
• 23:25 logmsgbot: krenair Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/221808/ (duration: 00m 13s)
• 23:17 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/223185/ (duration: 00m 12s)
• 23:06 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/220970/ (duration: 00m 14s)
• 21:46 gwicke: restarted cassandra instance on restbase1003; was low on memory and constantly writing small chunks
• 21:30 andrewbogott: rebooting labvirt1005, again. Somehow virtualization is turned off again
• 21:12 subbu: deployed parsoid version 87a746e6
• 21:04 logmsgbot: ori Synchronized php-1.26wmf12/thumb.php: cdc75debaf: Add Content-Length header to thumb.php error responses (duration: 00m 13s)
• 21:02 mutante: purging static-bz URL on varnish ...
• 20:39 akosiaris: upload php5_5.3.10-1ubuntu3.19-wmf1 on apt.wikimedia.org/precise-wikimedia
• 20:15 gwicke: restart cassandra instance on 1005
• 20:04 mobrovac: restbase restart cassandra on rb1005
• 19:28 logmsgbot: krenair Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/223040/ (duration: 00m 12s)
• 19:11 gwicke: reduced compaction throughput from 160 to 100 mb/s across the cassandra cluster via 'nodetool -h <host> setcompactionthroughput 100'
• 18:51 gwicke: restarted cassandra on restbase1001 with jdk8, see T104888
• 18:22 gwicke: restarted cassandra on restbase1004 with jdk8
• 17:54 Jeff_Green: authdns-update for new rigel A record
• 17:42 logmsgbot: jynus Synchronized wmf-config/db-codfw.php: increase db2029 traffic to normal levels (duration: 00m 12s)
• 17:37 gwicke: upgraded restbase1005 to jdk8
• 17:35 gwicke: restarting cassandra instance on restbase1005: out of heap
• 17:10 logmsgbot: jynus Synchronized wmf-config/db-codfw.php: repool db2029 again after conf upgrade(2/2) (duration: 00m 11s)
• 17:09 logmsgbot: jynus Synchronized wmf-config/db-codfw.php: repool db2029 again after conf upgrade (duration: 00m 11s)
• 16:38 jynus: upgrade and restart of db2029
• 16:35 ori: depooled mw1152
• 15:29 logmsgbot: krenair Finished scap: https://gerrit.wikimedia.org/r/#/c/222993/ (duration: 22m 09s)
• 15:21 _joe_: repooling mw1152
• 15:20 _joe_: attempting dump-apc on mw1060
• 15:09 _joe_: depooled the HHVM imagescaler again
• 15:07 logmsgbot: krenair Started scap: https://gerrit.wikimedia.org/r/#/c/222993/
• 15:02 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/222617/ (duration: 00m 12s)
• 14:48 moritzm: installed python security updates on analytics*, lab* and virt*
• 14:46 moritzm: added python-diskimage-builder 0.1.46-1+wmf1 for jessie-wikimedia on carbon
• 14:43 _joe_: depooled the HHVM imagescaler, spitting 503s again.
• 14:18 mobrovac: restbase started thinning out parsoid data (local_group_wikipedia_T_parsoid_dataDVIsgzJSne8k) for >= 22 days
• 14:07 YuviPanda: restart apache on labcontrol1001 to pick up parser function change
• 12:57 moritzm: installed python security updates on mw*, es* and db*
• 12:18 logmsgbot: hoo Synchronized wmf-config/: Enable WikibaseQuality and WikibaseQualityConstraints on wikidata (duration: 00m 13s)
• 12:15 logmsgbot: hoo Finished scap: Update WikibaseQuality and WikibaseQualityConstraint (duration: 25m 56s)
• 11:49 logmsgbot: hoo Started scap: Update WikibaseQuality and WikibaseQualityConstraint
• 11:40 hoo: Created the `wbqc_constraints` table on wikidatawiki
• 09:02 _joe_: restarted the appserver on mw1059 with hhvm.server.apc.expire_on_sets = true, restarted the heap profiling to confirm my hypothesis on T104769
• 08:31 _joe_: restarted cassandra on rb1004. again.
• 05:01 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1034, depool db1041 (duration: 00m 12s)
• 05:00 springle: stash/pull/apply CommonSettings.php on tin, which was left with modifications
• 04:35 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jul 6 04:35:45 UTC 2015 (duration 35m 44s)
• 02:22 logmsgbot: LocalisationUpdate completed (1.26wmf12) at 2015-07-06 02:22:12+00:00
• 02:18 logmsgbot: l10nupdate Synchronized php-1.26wmf12/cache/l10n: (no message) (duration: 06m 07s)

July 5

• 22:30 bd808: Restarted logstash on logstah1001; Hung due to OOM errors
• 22:03 mobrovac: restbase rolling restart of restbase
• 18:11 logmsgbot: krenair Synchronized docroot/noc: https://gerrit.wikimedia.org/r/#/c/222932/ (duration: 00m 12s)
• 17:49 logmsgbot: krenair Synchronized docroot/noc/conf: https://gerrit.wikimedia.org/r/#/c/222290/ (duration: 00m 13s)
• 17:44 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/221600/ (duration: 00m 12s)
• 15:16 YuviPanda: restarted nutcracker on silver.
• 12:55 mobrovac: restbase rolling restart of cassandra to apply the 16G heap change https://gerrit.wikimedia.org/r/222899
• 11:21 _joe_: restarted cassandra on restbase1004 (again), seemingly crashed for a bad request
• 11:03 _joe_: restarting cassandra on rb1003,4 and restbase on rb1002,3
• 09:43 bblack: restarted restbase on restbase1005
• 08:40 _joe_: collecting heaps on an api appserver, mw1115, as comparison
• 08:29 _joe_: restaarted HHVM on mw1059 with heap profiling enabled, collecting data (will stop this evening).
• 08:27 bblack: FYI: 08:15 < grrrit-wm> (CR) BBlack: [C: 2 V: 2] filter S:RI from wm2015register T45250 [puppet] - https://gerrit.wikimedia.org/r/222879 (owner: BBlack)
• 08:23 _joe_: restarted hhvm because of ooms, not apache
• 08:23 _joe_: restarted apache on mw1105,mw1092,90,82,78
• 07:09 bblack: restarted cassandra on restbase1004
• 07:07 bblack: restarted cassandra + restbase on restbase1005
• 07:01 jynus: Restarted HHVM for mw1112,1028,1057,1061,1069,1070,1084,1086
• 02:57 logmsgbot: LocalisationUpdate completed (1.26wmf12) at 2015-07-05 02:57:28+00:00

July 4

• 23:49 Krenair: Ran "mwscript updateSpecialPages.php labswiki --override --only=Wantedpages" on silver, completed in 0.44 seconds
• 23:44 Krenair: test morebots
• 21:22 YuviPanda: restarted cassandra on restbase1004 per urandom
• 19:15 YuviPanda: restarted cassandra on restbase1001
• 17:15 _joe_: restarted cassandra on restbase1001
• 16:12 logmsgbot: krenair Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 10m 35s)
• 12:56 logmsgbot: krinkle Synchronized php-1.26wmf12/resources/src/mediawiki/mediawiki.Title.js: I1dae1e63e47 (duration: 00m 17s)
• 05:01 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jul 4 05:01:43 UTC 2015 (duration 1m 42s)
• 03:11 ori: Promoted Krinkle and Krenair to admin, cloudadmin on wikitech, because duh.
• 02:39 logmsgbot: LocalisationUpdate completed (1.26wmf12) at 2015-07-04 02:39:41+00:00
• 02:29 logmsgbot: l10nupdate Synchronized php-1.26wmf12/cache/l10n: (no message) (duration: 09m 59s)
• 01:00 springle: reload haproxy dbproxy1004

July 3

• 23:59 logmsgbot: legoktm Synchronized php-1.26wmf12/extensions/Translate/: Translate+UserMerge fixes (duration: 00m 17s)
• 23:55 logmsgbot: legoktm Synchronized php-1.26wmf12/extensions/WikiLove/: WikiLove+UserMerge fixes (duration: 00m 18s)
• 23:24 logmsgbot: ori Synchronized w/404.php: Force 'Transfer-Encoding: Chunked' header on 404 responses (duration: 00m 31s)
• 22:36 Krenair: restarted apache on silver to see if it would make https://gerrit.wikimedia.org/r/#/c/221969/ take effect for T104360. It did not.
• 21:46 ori: depooled mw1152
• 20:12 ori: restarted cassandra on restbase1001
• 17:28 ori: pooled mw1152 (HHVM image scaler) for debugging.
• 17:05 logmsgbot: krenair Synchronized php-1.26wmf12/extensions/Collection/RenderingAPI.php: https://gerrit.wikimedia.org/r/#/c/222616/ - hoping this fixes T104708 (duration: 00m 44s)
• 15:35 YuviPanda: cd /mnt/backup/others-20150703/ ; tar --acls --xattrs -cpf - . | pv -L 80M -C -p -r -e -b -t -B 32M -T | ssh -c chacha20-poly1305@openssh.com -i ~/.ssh/id_labstore root@labstore2001.codfw.wmnet "pv -C -B 32M | tar --acls --xattrs -xpf - -C /srv/backup-others-20150703" on labstore1002
• 15:35 YuviPanda: mount /dev/mapper/backup-others--20150703 /srv/backup-others-20150703/ on labstore2001
• 15:34 YuviPanda: mkdir /srv/backup-others-20150703 on labstore2001
• 15:33 YuviPanda: mkfs -t ext4 /dev/mapper/backup-others--20150703 on labstore2001 completed
• 15:33 YuviPanda: run mount -o ro /dev/mapper/labstore-others--20150703 /mnt/backup/others-20150703/ on labstore1002
• 15:32 YuviPanda: run mkdir /mnt/backup/others-20150703 on labstore1002
• 15:31 YuviPanda: run lvcreate -L 640G -s -n others-20150703 labstore/others on labstore1002
• 15:29 YuviPanda: running mkfs -t ext4 /dev/mapper/backup-others--20150703 on labstore2001
• 15:28 YuviPanda: run lvcreate -L 3.5T -n others-20150703 backup on labstore2001
• 15:25 YuviPanda: begin process of backing up others (all labs projects except tools) on to labstore2001 from labstore1002
• 14:06 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool db1022 (low traffic) (duration: 00m 54s)
• 13:27 logmsgbot: jynus Synchronized wmf-config/db-codfw.php: repool db2047 after maintenance (duration: 00m 22s)
• 13:27 YuviPanda: run cd /mnt/backup/tools-20150703/ ; tar --acls --xattrs -cpf - . | pv -L 80M -C -p -r -e -b -t -B 32M -T | ssh -c chacha20-poly1305@openssh.com -i ~/.ssh/id_labstore root@labstore2001.codfw.wmnet "pv -C -B 32M | tar --acls --xattrs -xpf - -C /srv/backup-tools-20150703" on labstore1002
• 13:27 YuviPanda: interrupting tar |ssh | tar script and cleaning out destination again
• 13:17 YuviPanda: clean out tar | ssh | tar target on labstore2001
• 13:15 YuviPanda: /dev/null filled up on labstore1002, aborting pipe of valuable user data into it.
• 13:13 YuviPanda: run cd /mnt/backup/tools-20150703/ ; tar --acls --xattrs -cpf - . | pv -L 80M -C -p -r -e -b -t -B 32M -T > /dev/null on labstore1002
• 13:02 YuviPanda: run cd /mnt/backup/tools-20150703/ ; tar --acls --xattrs -cpf - . | pv -L 80M -C -p -r -e -b -t -B 32M -T | ssh -i ~/.ssh/id_labstore root@labstore2001.codfw.wmnet "pv -C -B 32M | tar --acls --xattrs -xpf - -C /srv/backup-tools-20150703" on labstore1002
• 13:02 YuviPanda: interrupt tar | ssh | tar on labstore1002 and killed dest on labstore2001
• 12:43 YuviPanda: cd /mnt/backup/tools-20150703/ ; tar --acls --xattrs -cpf - . | pv -L 80M -p -r -e -b -t -B 32M -T | ssh -i ~/.ssh/id_labstore root@labstore2001.codfw.wmnet "pv -B 32M | tar --acls --xattrs -xpf - -C /srv/backup-tools-20150703" on screen on labstore1002
• 12:43 mobrovac: restbase deploying restbase/deploy @ 1a826a5
• 12:42 YuviPanda: interrupt tar | ssh | tar on labstore1002, clean out destination on labstore2001
• 12:36 YuviPanda: interrupted tar | ssh | tar on labstore1002 and cleaned out dest on labstore2001
• 12:35 YuviPanda: cd /mnt/backup/tools-20150703/ ; tar --acls --xattrs -cpf - . | pv -L 80M -p -r -e -b -t -B 16M | ssh -i ~/.ssh/id_labstore root@labstore2001.codfw.wmnet "pv -B 16M | tar --acls --xattrs -xpf - -C /srv/backup-tools-20150703" in screen on labstore1002
• 12:33 YuviPanda: rm -rf /srv/backup-tools-20150703/* on labstore2001
• 12:31 mark: labstore2001: mount /srv/backup -o remount,ro
• 12:31 YuviPanda: interrupt tar | ssh | tar on labstore1002
• 12:29 YuviPanda: cd /mnt/backup/tools-20150703/ ; tar --acls --xattrs -cpf - . | ssh -i ~/.ssh/id_labstore root@labstore2001.codfw.wmnet "pv -L 80M -p -r -e -b -t -B 16M | tar --acls --xattrs -xpf - -C /srv/backup-tools-20150703" on labstore1002
• 12:28 YuviPanda: cd /mnt/backup/tools-20150703/ ; tar --acls --xattrs cpf - . | ssh -i ~/.ssh/id_labstore root@labstore2001.codfw.wmnet "pv -L 80M -p -r -e -b -t -B 16M | tar --acls --xattrs xpf - -C /srv/backup-tools-20150703" on labstore1002
• 12:09 YuviPanda: running mount -o ro /dev/mapper/labstore-tools--20150703 /mnt/backup/tools-20150703/ now
• 11:57 YuviPanda: run lvcreate -L 640G -s -n tools-20150703 labstore/tools on labstore1002
• 11:50 YuviPanda: running lvcreate -L 640G -s tools -n tools-20150703 labstore on labstore1002
• 11:26 YuviPanda: umount /mnt/backup/project/tools/ on labstore1002
• 11:24 YuviPanda: ran mount /dev/mapper/backup-tools--20150703 /srv/backup-tools-20150703/ on labstore2001
• 11:22 YuviPanda: mkdir /srv/backup-tools-20150703 on labstore2001
• 11:13 YuviPanda: run mkfs -t ext4 /dev/mapper/backup-tools--20150703 on labstore2001
• 11:09 YuviPanda: lvcreate -L 6TB -n tools-20150703 backup on labstore2001
• 11:09 jynus: reimports finished on dbstore2* hosts and puppet reenabled after T104471 was fixed
• 10:56 mobrovac: restbase disabling puppet on restbase1005 to tweak JVM params for cassandra
• 10:50 YuviPanda: started du of maps project on labstore2001
• 09:36 mobrovac: restbase restarting cassandra on rb1002
• 06:19 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jul 3 06:19:02 UTC 2015 (duration 19m 1s)
• 02:50 urandom: restbase rolling restart
• 02:49 logmsgbot: LocalisationUpdate completed (1.26wmf12) at 2015-07-03 02:49:31+00:00
• 02:42 logmsgbot: l10nupdate Synchronized php-1.26wmf12/cache/l10n: (no message) (duration: 11m 43s)
• 02:06 logmsgbot: ori Synchronized php-1.26wmf12/extensions/CentralAuth: I0e5f2d3b2: Updated mediawiki/core Project: mediawiki/extensions/CentralAuth 7f8da7139714dd5089dd03e8679aba25c2c89c4d (duration: 00m 15s)

July 2

• 22:34 logmsgbot: legoktm Synchronized php-1.26wmf12/extensions/CentralAuth/: Made use of new USE_MULTI_COMMIT flag in user merge jobs (duration: 00m 18s)
• 22:31 logmsgbot: legoktm Synchronized php-1.26wmf12/extensions/UserMerge/: Added USE_MULTI_COMMIT flag to enable query batching (duration: 00m 26s)
• 21:51 logmsgbot: legoktm Synchronized php-1.26wmf12/extensions/Interwiki/Interwiki_body.php: Add missing global $wgInterwikiViewOnly declaration (duration: 00m 15s)
• 21:37 twentyafterfour: restarted apache2 or iridium after applying hotfix for phabricator css issue
• 21:22 logmsgbot: legoktm Synchronized php-1.26wmf12/extensions/CentralNotice/: https://gerrit.wikimedia.org/r/222484 (duration: 00m 15s)
• 21:16 cwdent: updated civicrm from 4fe0648ea9f36282731bf651a59ca1a617db6c08 to 04efc7d5c7bbb068f907125f2184692aee676123
• 20:47 logmsgbot: legoktm Synchronized wmf-config/CommonSettings.php: Disable global merge (duration: 00m 14s)
• 20:13 andrewbogott: restarted keystone on labcontrol1001
• 18:54 bd808: Running sync-common on mw1111; fatal log showed it to be running 1.26wmf9
• 18:30 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: all wikis to 1.26wmf12
• 18:02 YuviPanda: running exportfs -ra on labstore1002
• 16:40 bd808: Restarted logstash on logstash1001 due to OOM
• 16:05 bblack: cp1065 undowntimed/repooled
• 16:04 YuviPanda: clean out exports.d in labstore1002, will get regenerated. backup in /root/exports.backup
• 15:18 logmsgbot: anomie Synchronized php-1.26wmf12/extensions/Wikidata/: SWAT: Update Wikibase: SearchEntities return 'aliases' when not same as label gerrit:222311 (duration: 00m 20s)
• 15:18 YuviPanda: killed icinga-wm again
• 15:17 bblack: depooled cp1065 in pybal/puppet
• 14:57 mutante: restarting gitblit on antimony for the 123443th time
• 14:54 mutante: restarted apache on strontium
• 14:50 YuviPanda: killed icinga-wm for a bit
• 14:43 YuviPanda: kicked puppetmaster on palladium
• 14:28 YuviPanda: restarted apache on labcontrol1001
• 14:14 logmsgbot: jynus Synchronized wmf-config/db-codfw.php: depool db2029 again: T104573 (duration: 00m 12s)
• 13:58 urandom: restarted restbase1005.eqiad
• 13:49 logmsgbot: jynus Synchronized wmf-config/db-codfw.php: repool db2029; depool db2047 for maintenance (duration: 00m 13s)
• 11:19 mobrovac: restbase restarting cassandra on rb1005
• 07:06 logmsgbot: krinkle Synchronized w/touch.php: T104538 (duration: 00m 11s)
• 07:05 logmsgbot: krinkle Synchronized w/favicon.php: T104538 (duration: 00m 11s)
• 06:34 logmsgbot: jynus Synchronized wmf-config/db-codfw.php: Emergency depool of db2029 (duration: 00m 12s)
• 06:27 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jul 2 06:27:57 UTC 2015 (duration 27m 56s)
• 04:18 ori: depooled mw1152.
• 03:38 logmsgbot: krinkle Synchronized docroot/default/index.html: 6d49d229806 (duration: 00m 12s)
• 03:37 logmsgbot: krinkle Synchronized 404.html: 6d49d229806 (duration: 00m 12s)
• 03:14 logmsgbot: legoktm Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 12s)
• 02:54 logmsgbot: LocalisationUpdate completed (1.26wmf12) at 2015-07-02 02:54:06+00:00
• 02:52 logmsgbot: krinkle Synchronized docroot and w: 245a1ff (duration: 00m 12s)
• 02:51 logmsgbot: l10nupdate Synchronized php-1.26wmf12/cache/l10n: (no message) (duration: 05m 19s)
• 02:37 logmsgbot: LocalisationUpdate completed (1.26wmf11) at 2015-07-02 02:37:03+00:00
• 02:30 logmsgbot: l10nupdate Synchronized php-1.26wmf11/cache/l10n: (no message) (duration: 10m 23s)
• 00:44 ori: Repooling mw1152 (HHVM image scaler) for testing)

July 1

• 23:30 springle: restart mysqld dbstore2002 T104471
• 23:06 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/222202/ (duration: 00m 11s)
• 21:39 godog: bounce gitblit
• 20:38 jgage: restarted gitblit on antimony
• 19:50 ori: restarted gitblit on antimony
• 19:49 ori: mw1152 not actually re-pooled because of ongoing work on palladium. I'm undoing the change and hanging back now.
• 19:41 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf12
• 19:36 logmsgbot: twentyafterfour Synchronized php-1.26wmf12: sync 1.26wmf12 branch revert of "Implement support for Google reCAPTCHA 2.0" 90665a737bc25ff3c859044755d662c6cd700573 (duration: 02m 04s)
• 19:31 jynus: replication issues for shard s7 on dbstore2001 and dbstore2002, production applications *not* affected
• 19:31 urandom: from restbase1002; node thin_out_key_rev_value_data.js `hostname -i` local_group_wikipedia_T_parsoid_html 2>&1 | pv --line-mode | gzip -c > wikipedia_T_parsoid_html.log.gz
• 19:28 ori: Repooling mw1152 for further testing of HHVM scaler
• 19:03 logmsgbot: hoo Synchronized php-1.26wmf12/extensions/Wikidata/: Update DataModel to fix SnakList (duration: 00m 20s)
• 18:42 logmsgbot: hoo Synchronized wmf-config/mobile-labs.php: consistency (duration: 00m 12s)
• 18:41 logmsgbot: hoo Synchronized wmf-config/InitialiseSettings-labs.php: consistency (duration: 00m 31s)
• 18:02 andrewbogott: restarted keystone on labcontrol1001
• 17:03 jgage: beginning puppet CA replacement procedure
• 16:06 ejegg: enabled queue consumers
• 16:05 akosiaris: re-enabling ntp everywhere
• 15:59 ejegg: disabled queue consumers
• 15:30 logmsgbot: hoo Synchronized php-1.26wmf12/extensions/Wikidata/: Remove alias uniqueness constraints (duration: 00m 21s)
• 15:06 urandom: restbase1002: PWD=/home/eevans/restbase-mod-table-cassandra/maintenance; node thin_out_key_rev_value_data.js `hostname -i` local_group_wikimedia_T_parsoid_html 2>&1 | pv --line-mode | gzip -c > wikimedia_T_parsoid_html.log.gz
• 15:05 bblack: re-enabling puppet on caches
• 14:59 bblack: disabling puppet on caches (because puppet always breaks when you move files/modules around...)
• 13:57 bblack: rebooting cp2001 (test kernel update)
• 11:32 YuviPanda: rsync on labstore1002 finished, restarting to see what was skipped + errors
• 10:47 moritzm: installed patch security updates on 862 hosts
• 10:42 hashar: restarting Jenkins: upgrading Jenkins gearman plugin from 0.1.1-8-gf2024bd to 0.1.1-9-g08e9c42-change_192429_2 https://phabricator.wikimedia.org/T72597#1416913
• 07:48 mobrovac: restbase restarting cassandra on rb1005
• 05:28 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jul 1 05:28:38 UTC 2015 (duration 28m 37s)
• 05:27 csteipp: deployed patch for T103765
• 04:41 logmsgbot: krinkle Synchronized php-1.26wmf12/includes/resourceloader/ResourceLoader.php: Iee884208c5c4b minify cache key (duration: 00m 11s)
• 03:10 mutante: git pull on strontium
• 03:00 logmsgbot: LocalisationUpdate completed (1.26wmf12) at 2015-07-01 03:00:21+00:00
• 02:53 logmsgbot: l10nupdate Synchronized php-1.26wmf12/cache/l10n: (no message) (duration: 10m 12s)
• 02:26 logmsgbot: LocalisationUpdate completed (1.26wmf11) at 2015-07-01 02:26:55+00:00
• 02:23 logmsgbot: l10nupdate Synchronized php-1.26wmf11/cache/l10n: (no message) (duration: 06m 50s)
• 02:12 springle: upgrade db1034 trusty
• 01:37 ori: Depooled mw1152. Req error dashboard shows elevated 5xx rates correlating with the server getting pooled, but the logs don't appear to corroborate it. Odd.
• 01:03 ori: Disabling Puppet on mw1152 for 12h to hack apache config to log locally
• 00:42 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I9a8018981: Double $wgMaxShellMemory on HHVM scalers (512 Mb => 1024 Mb) (duration: 00m 12s)
• 00:34 ori: pooled mw1152 (HHVM rendering) at weight 10 for testing
• 00:33 gwicke: rolling cassandra restart done
• 00:23 gwicke: starting rolling restart of cassandra nodes to apply new config
• 00:01 greg-g: we're still here

June 30

• 23:30 logmsgbot: hoo Synchronized php-1.26wmf12/extensions/Wikidata/: Fix EntityParserOutputGenerator (duration: 00m 21s)
• 22:55 ori: depooled mw1152
• 22:52 ori: Pooled HHVM image scaler (mw1152) at weight 1 for testing.
• 22:52 gwicke: updated restbase1004 to openjdk-8
• 22:46 bblack: restarting gitblit on antimony, because Java is so 1996
• 22:43 tgr: running eval.php (along the lines of https://gerrit.wikimedia.org/r/#/c/221783) on commonswiki to fix T104395
• 22:13 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings.php: Flow-occupy Wikipedia talk namespace on cawiki (duration: 00m 11s)
• 22:09 matt_flaschen: Done converting wikitext namespace to Flow on Catalan Wikipedia
• 22:03 matt_flaschen: Started convertNamespaceFromWikitext.php for Project_talk on Catalan Wikipedia
• 21:46 RoanKattouw: Also ran populateContentModel.php --table=archive for talk namespaces on officewiki
• 21:45 RoanKattouw: Ran populateContentModel.php --table=archive --ns=5 on officewiki
• 21:29 RoanKattouw: Ran populateContentModel.php --table=page --ns=5 on cawiki
• 21:19 logmsgbot: catrope Synchronized php-1.26wmf12/extensions/Flow: (no message) (duration: 00m 14s)
• 21:19 logmsgbot: catrope Synchronized php-1.26wmf11/extensions/Flow: (no message) (duration: 00m 14s)
• 21:14 logmsgbot: catrope Synchronized php-1.26wmf12/extensions/Flow: (no message) (duration: 00m 14s)
• 21:14 logmsgbot: catrope Synchronized php-1.26wmf11/extensions/Flow: (no message) (duration: 00m 13s)
• 21:01 RoanKattouw: Running populateContentModel.php on officewiki for page table in namespaces occupied by Flow (1,3,5,7,9,11,13,15,91,93,101,111,113,829)
• 20:58 logmsgbot: catrope Synchronized php-1.26wmf12/maintenance/: Add populateContentModel maintenance script (duration: 00m 13s)
• 20:58 logmsgbot: catrope Synchronized php-1.26wmf11/maintenance/: Add populateContentModel maintenance script (duration: 00m 17s)
• 20:53 logmsgbot: hoo Synchronized wmf-config/InitialiseSettings.php: Log 'wbq_evaluation' (duration: 00m 12s)
• 20:46 logmsgbot: hoo Synchronized wmf-config/InitialiseSettings.php: Enable WikibaseQuality extensions on testwikidata (duration: 00m 14s)
• 20:39 hoo: Created `wbqc_constraints` on testwikidatawiki (s3).
• 20:23 logmsgbot: thcipriani rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.26wmf12
• 20:15 logmsgbot: thcipriani Purged l10n cache for 1.26wmf6
• 20:14 logmsgbot: thcipriani Purged l10n cache for 1.26wmf7
• 20:14 logmsgbot: thcipriani Purged l10n cache for 1.26wmf8
• 20:13 logmsgbot: thcipriani Purged l10n cache for 1.26wmf9
• 20:13 logmsgbot: thcipriani Purged l10n cache for 1.26wmf10
• 20:05 logmsgbot: thcipriani Finished scap: testwiki to php-1.26wmf12 and rebuild l10n cache (duration: 34m 58s)
• 19:41 ostriches: OAI: disabled unused accounts
• 19:30 logmsgbot: thcipriani Started scap: testwiki to php-1.26wmf12 and rebuild l10n cache
• 19:00 logmsgbot: demon Synchronized php-1.26wmf11/includes/WebResponse.php: rv my test (duration: 00m 12s)
• 18:55 logmsgbot: demon Synchronized php-1.26wmf11/includes/WebResponse.php: (no message) (duration: 00m 12s)
• 18:36 cmjohnson1: labcontrol1002 going down for a few minutes
• 18:33 mutante: tendril - short downtime for switch to new repo
• 18:17 gwicke: restarted cassandra on restbase1005 with g1gc GC and larger heap
• 18:16 gwicke: restarted cassandra on restbase1004 with g1gc GC and larger heap
• 17:02 akosiaris: enabled and ran puppet on lvs400X, lvs300X, lvs100[123]. noops
• 16:58 bblack: re-enabling puppet on caches
• 16:52 bblack: disabling puppet on cache clusters
• 16:48 akosiaris: enabled an ran puppet on all lvs servers @ codfw
• 16:22 akosiaris: enabled and ran puppet on lvs1004. noop as well
• 16:19 akosiaris: enabled and running puppet on lvs1005
• 16:11 akosiaris: enabling and running puppet on lvs1006
• 16:09 akosiaris: disabling puppet on all lvs and neon
• 16:07 gwicke: restarting cassandra instance on restbase1004
• 15:12 logmsgbot: thcipriani Synchronized wmf-config: SWAT: Standardise a ton of ticket comments gerrit:221803 (duration: 00m 13s)
• 15:04 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Enable CX all wikipedias except enwiki gerrit:221831 (duration: 00m 13s)
• 14:46 kart_: Update cxserver to 0d21a80
• 14:10 mobrovac: restbase restarting cassandra on restbase1005
• 11:29 mobrovac: restbase restarting cassandra on restbase1005
• 10:41 mobrovac: restbase restarting on all nodes
• 09:54 mobrovac: restbase restarting cassandra on restbase1004
• 08:53 mobrovac: restbase restrting cassandra on restbase1004
• 08:05 jynus: applying schema changes for Gather extension
• 06:56 jynus: initiating query profiling on db1018
• 05:21 gwicke: restarting cassandra instance on restbase1004; was in small-write mode
• 05:17 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1034 (duration: 00m 12s)
• 04:37 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jun 30 04:37:00 UTC 2015 (duration 36m 59s)
• 02:22 logmsgbot: LocalisationUpdate completed (1.26wmf11) at 2015-06-30 02:22:00+00:00
• 02:18 logmsgbot: l10nupdate Synchronized php-1.26wmf11/cache/l10n: (no message) (duration: 06m 09s)
• 02:11 logmsgbot: krenair Synchronized wmf-config/wikitech.php: (no message) (duration: 00m 12s)
• 01:56 logmsgbot: krenair Synchronized wmf-config/wikitech.php: (no message) (duration: 00m 11s)
• 01:41 logmsgbot: krinkle Synchronized php-1.26wmf11/includes/resourceloader/ResourceLoader.php: I7761242f01 (duration: 00m 14s)
• 00:37 godog: restbase1* upgrade to cassandra 2.1.7 completed

June 29

• 23:57 robh: mw2027 was offline (blank screen on serial console). mgmt powercycled
• 23:48 godog: start upgrading restbase1* to cassandra 2.1.7
• 23:41 gwicke: restarted cassandra instance on restbase1004.eqiad; log showed many small writes and clients saw timeouts
• 23:29 gwicke: deployed restbase 32db4ce1e1
• 23:21 logmsgbot: ori Synchronized php-1.26wmf11/includes/resourceloader: I0e5f2d3b2: resourceloader: Add timing metrics for key operations (duration: 01m 12s)
• 23:15 logmsgbot: catrope Synchronized wmf-config/: wikitech cleanup (duration: 01m 08s)
• 23:11 RoanKattouw: ssh: connect to host mw2027.codfw.wmnet port 22: Connection timed out
• 23:11 RoanKattouw: Synced wmf-config/CommonSettings.php: Remove survey access point in Popups
• 23:09 godog: stop ircecho on neon, icinga spam
• 22:53 gwicke: canary deploy of restbase 32db4ce1e1 on restbase1001.eqiad
• 21:30 urandom: restarting restbase1004 to apply new metrics reporting interval
• 20:19 subbu: deployed parsoid sha ea98be88
• 18:18 logmsgbot: ori Synchronized php-1.26wmf11/includes/db/LoadBalancer.php: I0e5f2d3b2: Use APC for caching slave lag times (duration: 01m 09s)
• 18:00 cmjohnson1: powering down ms-be1015
• 16:06 bblack: re-enabling puppet on caches
• 15:51 bblack: disabling puppet on caches temporarily ...
• 15:49 logmsgbot: krenair Synchronized php-1.26wmf11/extensions/OpenStackManager: https://gerrit.wikimedia.org/r/#/c/221648/ (duration: 00m 13s)
• 15:29 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/221405/ (duration: 00m 15s)
• 15:26 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/221612/ (duration: 00m 12s)
• 15:24 logmsgbot: krenair Synchronized w/static/images/project-logos/zhwiki-hans-2x.png: https://gerrit.wikimedia.org/r/#/c/221113/ (duration: 00m 14s)
• 15:24 logmsgbot: krenair Synchronized w/static/images/project-logos/zhwiki-hans-1.5x.png: https://gerrit.wikimedia.org/r/#/c/221113/ (duration: 00m 12s)
• 15:23 logmsgbot: krenair Synchronized w/static/images/project-logos/zhwiki-hans.png: https://gerrit.wikimedia.org/r/#/c/221113/ (duration: 00m 12s)
• 15:20 logmsgbot: krenair Synchronized wmf-config/wikitech.php: https://gerrit.wikimedia.org/r/#/c/221009/ (duration: 00m 11s)
• 15:18 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/221047/ (duration: 00m 13s)
• 15:12 logmsgbot: krenair Synchronized php-1.26wmf11/extensions/ContentTranslation/modules/tools/ext.cx.tools.link.js: https://gerrit.wikimedia.org/r/#/c/221605 (duration: 00m 13s)
• 15:02 logmsgbot: krenair Synchronized php-1.26wmf11/extensions/ContentTranslation/modules/tools/ext.cx.tools.formatter.js: https://gerrit.wikimedia.org/r/#/c/221604/ (duration: 00m 14s)
• 14:34 jynus: rebooting and reinstalling db1022
• 12:06 YuviPanda: restarting rsync with new exclusions file on labstore1002 to codfw
• 12:06 YuviPanda: excluded maps, mwoffliner and video project from rsync of broken FS to speed it up
• 11:59 YuviPanda: interupt rsync on labstore1001 to prevent it from copying mwofflienr files
• 11:00 _joe_: shutting down etcd1003, cleaning exported resources
• 10:32 _joe_: effectively removing etcd1003 from the cluster
• 10:17 _joe_: starting removal of etcd1003 from the etcd cluster
• 08:49 _joe_: joined conf1003 to the etcd cluster
• 08:20 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Depool db1022 for reinstall (duration: 00m 12s)
• 08:12 _joe_: adding conf1002 to the etcd cluster as a member
• 07:46 akosiaris: disabling ntp everywhere expect selected hosts in anticipation for the leap second
• 04:51 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jun 29 04:51:48 UTC 2015 (duration 51m 47s)
• 03:08 jgage: jmxtrans filled disks on all kafka brokers, 21GB log files. removed logs and restarted services.
• 02:23 logmsgbot: LocalisationUpdate completed (1.26wmf11) at 2015-06-29 02:23:47+00:00
• 02:20 logmsgbot: l10nupdate Synchronized php-1.26wmf11/cache/l10n: (no message) (duration: 05m 53s)
• 00:52 springle: restart eventlogging auto-purge on m4
• 00:51 springle: restart replication on dbstore2002
• 00:00 springle: pausing replication on dbstore2002

June 28

• 23:51 logmsgbot: ori Synchronized php-1.26wmf11/extensions/CentralNotice/modules/ext.centralNotice.bannerController/bannerController.js: I6ffdc977e87: Parse older format of Geo cookies (duration: 00m 13s)
• 04:30 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jun 28 04:30:54 UTC 2015 (duration 30m 53s)
• 02:20 logmsgbot: LocalisationUpdate completed (1.26wmf11) at 2015-06-28 02:20:52+00:00
• 02:17 logmsgbot: l10nupdate Synchronized php-1.26wmf11/cache/l10n: (no message) (duration: 05m 56s)

June 27

• 23:30 bd808: Deleted corrupt shards on logstash1004 and logstash1005. Recovery in process
• 20:12 ori: Delegated full access to Google Webmaster Tools for myself (olivneh@).
• 04:58 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jun 27 04:58:46 UTC 2015 (duration 58m 45s)
• 02:23 logmsgbot: LocalisationUpdate completed (1.26wmf11) at 2015-06-27 02:23:40+00:00
• 02:20 logmsgbot: l10nupdate Synchronized php-1.26wmf11/cache/l10n: (no message) (duration: 05m 46s)

June 26

• 23:57 bd808: Logstash log ingestion working again after forcing recovery of replicas for logstash-2015.06.26; new logs were being rejected with only a primary shard available
• 23:54 bd808: re-enabled allocation on logstash elasticsearch cluster
• 23:05 bblack: restarted gitblit on antimony, AGAIN
• 22:57 mutante: restarted gitblit
• 22:43 logmsgbot: catrope Synchronized php-1.26wmf11/extensions/Flow: Temporarily make subpages in Flow-occupied namespaces non-Flow again (duration: 00m 14s)
• 22:36 bd808: set indices.recovery.concurrent_streams to 4 on logstash ES cluster
• 22:36 godog: set indices.recovery.max_bytes_per_sec to 10mb on logstash ES cluster
• 22:25 godog: set indices.recovery.max_bytes_per_sec to 50mb on logstash ES cluster
• 22:25 jamesofur: Reset email address of User:Chwms identity verified in person at editathon
• 22:09 bd808: restarted logstash on logstash1001
• 21:10 urandom: taking xenon down to be rebootstrapped
• 20:10 bd808: Deleted 4 corrupt indices (logstash-2015.05.30 logstash-2015.05.31 logstash-2015.06.03 logstash-2015.06.06) on logstash1004
• 19:58 bd808: stopping elasticsearch on logstash1004 to cleanup corrupt shards
• 17:05 mutante: zirconium - manual cleanup, removing planet
• 17:04 godog: reverted cronolog puppetmaster patch, restarting apache
• 14:17 Krenair: Deployed patch for T103391
• 12:23 logmsgbot: krenair Synchronized wmf-config/wikitech.php: https://gerrit.wikimedia.org/r/#/c/221105/ (duration: 00m 12s)
• 12:18 _joe_: added conf1001 to the etcd cluster
• 07:57 logmsgbot: krinkle Synchronized php-1.26wmf11/extensions/Popups: T103610 (duration: 00m 11s)
• 06:04 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jun 26 06:04:14 UTC 2015 (duration 4m 13s)
• 05:22 twentyafterfour: restarted apache on iridium to fix phabricator fatal
• 02:33 logmsgbot: LocalisationUpdate completed (1.26wmf11) at 2015-06-26 02:33:33+00:00
• 02:30 logmsgbot: l10nupdate Synchronized php-1.26wmf11/cache/l10n: (no message) (duration: 05m 36s)
• 00:51 gwicke: reverted restbase1001 canary to 90817c2a
• 00:36 logmsgbot: ori Synchronized php-1.26wmf11/extensions/SyntaxHighlight_GeSHi: I0e5f2d3b2: Updated mediawiki/core Project: mediawiki/extensions/SyntaxHighlight_GeSHi (duration: 00m 11s)
• 00:16 logmsgbot: krinkle Synchronized wmf-config/InitialiseSettings.php: T102852 (duration: 00m 12s)
• 00:15 logmsgbot: krinkle Synchronized w/static/images/project-logos/zhwiki-2x.png: T102852 (duration: 00m 13s)
• 00:14 logmsgbot: krinkle Synchronized w/static/images/project-logos/zhwiki-1.5x.png: T102852 (duration: 00m 12s)
• 00:05 logmsgbot: krinkle Synchronized php-1.26wmf11/extensions/SyntaxHighlight_GeSHi/modules/pygments.wrapper.css: I5d1510dc80d6d4712ca8411 (duration: 00m 12s)

June 25

• 23:53 mutante: planet1001 (ganeti) - signing puppet cert, initial run
• 23:31 mutante: apt-get upgrade on zirconium
• 23:28 logmsgbot: krenair Synchronized wmf-config/wikitech.php: https://gerrit.wikimedia.org/r/#/c/220847/ (duration: 00m 12s)
• 23:27 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/220847/ (duration: 00m 11s)
• 23:24 logmsgbot: krenair Synchronized php-1.26wmf11/extensions/SyntaxHighlight_GeSHi: https://gerrit.wikimedia.org/r/#/c/220997/ (duration: 00m 13s)
• 23:20 gwicke: canary update of restbase on restbase1001 to 4b961f166 (deploy d1c4d9961)
• 23:16 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/218926/ (duration: 00m 12s)
• 23:11 logmsgbot: krenair Synchronized wmf-config/logging.php: https://gerrit.wikimedia.org/r/#/c/220784/ (duration: 00m 13s)
• 23:03 legoktm: fixed content models on lrcwiki for Module namespace
• 23:02 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/220485/ (duration: 00m 16s)
• 22:02 logmsgbot: hoo Synchronized php-1.26wmf11/extensions/Wikidata/: Update Wikidata: Use SELECT FOR UPDATE in SqlIdGenerator (duration: 00m 20s)
• 21:29 godog: rm /var/lib/git/operations/puppet/modules/cassandra from labcontrol1001 labcontrol1002
• 21:10 godog: rm /var/lib/git/operations/puppet/modules/cassandra from rhodium
• 21:07 godog: rm /var/lib/git/operations/puppet/modules/cassandra from strontium and palladium
• 21:06 godog: push puppet.git after module/cassandra removal T92560
• 20:41 mutante: deleted SVN monitor from watchmouse
• 20:18 mutante: bye SVN - subversion URLs now redirect to phab or doc
• 20:08 logmsgbot: nikerabbit Finished scap: T103888 CX aliases (duration: 22m 37s)
• 19:46 logmsgbot: nikerabbit Started scap: T103888 CX aliases
• 18:09 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: all wikis to 1.26wmf11
• 17:46 logmsgbot: krenair Synchronized wmf-config: (no message) (duration: 00m 31s)
• 17:43 logmsgbot: krenair Synchronized wmf-config/wikitech.php: https://gerrit.wikimedia.org/r/#/c/218098/ (duration: 00m 12s)
• 17:43 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/218098/ (duration: 00m 12s)
• 17:18 logmsgbot: ori Synchronized php-1.26wmf11/resources/src/mediawiki.skinning/elements.css: Ieab6b1473e6ce: תיקון טעות (duration: 00m 12s)
• 15:59 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/219599/ (duration: 00m 12s)
• 15:57 logmsgbot: krenair Synchronized wmf-config/CommonSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/217539/ - noop for prod, labs only part (duration: 00m 12s)
• 15:56 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/217539/ (duration: 00m 13s)
• 15:51 logmsgbot: krenair Synchronized wmf-config/flaggedrevs.php: https://gerrit.wikimedia.org/r/#/c/203370/ (duration: 00m 12s)
• 15:49 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/218539/ (duration: 00m 15s)
• 15:32 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/220068/ - noop for prod, just labs (duration: 00m 12s)
• 15:30 logmsgbot: krenair Synchronized commonsuploads.dblist: https://gerrit.wikimedia.org/r/#/c/220715/ (duration: 00m 12s)
• 15:24 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/220747/ (duration: 00m 12s)
• 15:16 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/220408/ (duration: 00m 12s)
• 15:12 logmsgbot: krenair Synchronized php-1.26wmf11/extensions/SemanticForms/includes/SF_AutoeditAPI.php: https://gerrit.wikimedia.org/r/#/c/220765/ (duration: 00m 12s)
• 15:04 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/220706/ (duration: 00m 12s)
• 15:02 logmsgbot: krenair Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/220653/ (duration: 00m 12s)
• 13:30 logmsgbot: jynus Synchronized wmf-config/db-codfw.php: Repool es2003 (but not es2004) after maintenance (duration: 00m 12s)
• 10:57 jynus: rebooting es2003 and es2004
• 10:40 logmsgbot: jynus Synchronized wmf-config/db-codfw.php: depool es2003 and es2004 for maintenance (duration: 00m 13s)
• 10:09 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool db1018 (duration: 00m 12s)
• 09:02 jynus: restarting mysqld on db1018
• 08:42 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool db1018 for maintenance (duration: 00m 13s)
• 08:33 logmsgbot: ori Synchronized php-1.26wmf11/resources/src/mediawiki.skinning/elements.css: I0e5f2d3b2: Wrap lines in <pre> and .mw-code by default (duration: 00m 12s)
• 06:59 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun 25 06:59:13 UTC 2015 (duration 59m 12s)
• 04:04 ori: restarted apache2 on palladium
• 03:11 logmsgbot: LocalisationUpdate completed (1.26wmf11) at 2015-06-25 03:11:01+00:00
• 03:04 logmsgbot: l10nupdate Synchronized php-1.26wmf11/cache/l10n: (no message) (duration: 10m 19s)
• 02:40 bblack: puppet re-enabled on caches
• 02:37 logmsgbot: LocalisationUpdate completed (1.26wmf10) at 2015-06-25 02:37:44+00:00
• 02:34 logmsgbot: l10nupdate Synchronized php-1.26wmf10/cache/l10n: (no message) (duration: 06m 44s)
• 02:04 bblack: disabling puppet on cp* caches for patch-testing
• 00:43 awight: update crm from bd8a00196071ddd04efbff7b30567dd9357c9000 to e923225e423948bd70440e2d1131460b10cefac1
• 00:38 godog: upgrade cassandra to 2.1.7 on restbase1008
• 00:30 twentyafterfour: phabricator upgrade completed
• 00:28 godog: upgrade cassandra to 2.1.7 on restbase1004
• 00:12 legoktm: <twentyafterfour> Phabricator upgrade happening now. Will be down for a few minutes.

June 24

• 23:18 logmsgbot: rmoen Synchronized wmf-config/mobile.php: Enable browse experiment on test and enwiki (duration: 00m 14s)
• 23:17 logmsgbot: rmoen Synchronized wmf-config/InitialiseSettings.php: Enable browse experiment on test and enwiki (duration: 00m 12s)
• 23:13 urandom: rolling restart of Cassandra staging cluster
• 23:04 logmsgbot: legoktm Synchronized php-1.26wmf11/extensions/CentralAuth: https://gerrit.wikimedia.org/r/#/c/220637/ (duration: 00m 13s)
• 23:03 logmsgbot: legoktm Synchronized php-1.26wmf11/extensions/UserMerge: https://gerrit.wikimedia.org/r/#/c/220638/ (duration: 00m 13s)
• 22:32 mutante: zirconium - stop using 443 at all, rm NameVirtualHost *:443
• 22:30 mutante: zirconium - deleting unused apache configs, bugzilla, etherpad, ...
• 21:09 godog: start cassandra on restbase1008
• 18:41 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf11
• 18:02 logmsgbot: legoktm Synchronized php-1.26wmf11/extensions/Flow/includes/Specials/SpecialEnableFlow.php: https://gerrit.wikimedia.org/r/#/c/220514/ (duration: 00m 15s)
• 17:24 logmsgbot: jynus Synchronized wmf-config/db-codfw.php: repool es2001 and es2002 after maintenance (duration: 00m 13s)
• 17:05 thcipriani: scap completed with the exception of snapshot1001 that's disk is full
• 17:04 logmsgbot: thcipriani scap failed: OSError [Errno 2] No such file or directory: '/var/lock/scap' (duration: 41m 33s)
• 16:22 logmsgbot: thcipriani Started scap: SWAT: Automatically add to shell group when adding to a project gerrit:220468
• 16:10 logmsgbot: ori Synchronized php-1.26wmf11/includes/page/Article.php: I0e5f2d3b2: Revert r47388 / 8d9243cf3: Use Title::getLocalURL() for rel=canonical links (duration: 00m 13s)
• 15:57 logmsgbot: thcipriani Synchronized wmf-config: SWAT: Revert Enable browse prototype on test- and enwiki (duration: 00m 15s)
• 15:49 jynus: rebooting es2001 and es2002
• 15:44 logmsgbot: thcipriani Synchronized wmf-config: SWAT: Enable browse prototype on test- and enwiki gerrit:219451 (duration: 00m 12s)
• 15:24 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ContentTranslation in testwiki gerrit:220385 (duration: 00m 12s)
• 15:17 logmsgbot: thcipriani Synchronized php-1.26wmf11/extensions/ContentTranslation: SWAT: Enable publish button when the preference is not to use initial translation (duration: 00m 12s)
• 15:14 andrewbogott: disabled puppet on labcontrol1001 to hotfix https://gerrit.wikimedia.org/r/#/c/220476/
• 15:08 logmsgbot: thcipriani Synchronized php-1.26wmf10/extensions/ContentTranslation: SWAT: Enable publish button when the preference is not to use initial translation (duration: 00m 13s)
• 14:53 logmsgbot: jynus Synchronized wmf-config/db-codfw.php: depool es2001 and es 2002 for maintenance (duration: 00m 13s)
• 14:12 logmsgbot: krenair Synchronized php-1.26wmf10/extensions/SemanticForms/includes/SF_AutoeditAPI.php: T103653 live hack (duration: 00m 13s)
• 10:44 _joe_: restarting jmxtrans on analytics1021
• 10:31 jgage: restarting kafka on analytics1021
• 10:10 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Switchover master es1008 -> es1009 (duration: 00m 12s)
• 09:24 hashar: removing java 6 from gallium and lanthanum https://phabricator.wikimedia.org/T103491
• 09:17 hashar: apt-get upgrade on gallium and lanthanum
• 09:16 jynus: performing a master failover of es1008 into es1009
• 08:27 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Repool es1004 (duration: 00m 14s)
• 05:46 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jun 24 05:46:32 UTC 2015 (duration 46m 31s)
• 05:12 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1045 (duration: 00m 13s)
• 05:03 jgage: removed old logs and did 'apt-get clean' on analytics1021 to make space
• 03:00 logmsgbot: LocalisationUpdate completed (1.26wmf11) at 2015-06-24 03:00:45+00:00
• 02:54 logmsgbot: l10nupdate Synchronized php-1.26wmf11/cache/l10n: (no message) (duration: 10m 34s)
• 02:28 logmsgbot: LocalisationUpdate completed (1.26wmf10) at 2015-06-24 02:28:16+00:00
• 02:24 logmsgbot: l10nupdate Synchronized php-1.26wmf10/cache/l10n: (no message) (duration: 07m 21s)
• 01:39 logmsgbot: ori Synchronized php-1.26wmf11/extensions/SyntaxHighlight_GeSHi: I0e5f2d3b2 (duration: 00m 13s)
• 01:01 gwicke: rolling restart of cassandra instances to rule out a single node in funky state causing elevated p99 latency
• 00:43 ori: experimenting with httpd on mw1041 again
• 00:19 gwicke: rolling restart of restbase instances to rule out backend connections as a source for high p99 latencies
• 00:14 ori: experimenting with HHVM shutdown via /stop on the admin server on mw1041

June 23

• 23:38 logmsgbot: ori Finished scap: scapping to all apaches for --restart test (duration: 07m 03s)
• 23:30 logmsgbot: ori Started scap: scapping to all apaches for --restart test
• 23:24 bblack: nginxes all updated for ssl stapling bugfix
• 23:24 logmsgbot: ori Finished scap: scapping to scap-test dsh group for --restart test (duration: 06m 02s)
• 23:18 logmsgbot: ori Started scap: scapping to scap-test dsh group for --restart test
• 23:16 logmsgbot: ori scap aborted: scapping to scap-test dsh group for --restart test (duration: 00m 06s)
• 23:16 logmsgbot: ori Started scap: scapping to scap-test dsh group for --restart test
• 22:14 logmsgbot: legoktm Synchronized php-1.26wmf11/extensions/SyntaxHighlight_GeSHi/SyntaxHighlight_GeSHi.class.php: RejectParserCacheValue may pass a WikiPage or Article (duration: 00m 13s)
• 22:07 mutante: tmp. disabling puppet on mw1033
• 21:53 logmsgbot: legoktm Synchronized php-1.26wmf11/extensions/SyntaxHighlight_GeSHi/SyntaxHighlight_GeSHi.class.php: (no message) (duration: 00m 15s)
• 21:50 logmsgbot: ori Synchronized php-1.26wmf11/includes/parser/ParserCache.php: (no message) (duration: 00m 12s)
• 21:40 mutante: starting instance planet1001 on ganeti1003 - cant get console
• 21:40 logmsgbot: legoktm Synchronized php-1.26wmf11/includes/parser/ParserCache.php: (no message) (duration: 00m 13s)
• 21:36 bd808: updated scap to 33f3002 (Ensure that the minimum batch size used by cluster_ssh is 1)
• 21:34 logmsgbot: ori Synchronized php-1.26wmf11/extensions/SyntaxHighlight_GeSHi: 3c8bb2c493: Update SyntaxHighlight_GeSHi for cherry-pick (duration: 00m 13s)
• 20:32 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.26wmf11
• 20:19 logmsgbot: mattflaschen Synchronized wmf-config/InitialiseSettings-labs.php: Beta-only change to add Flow_test to enwiki (duration: 00m 11s)
• 19:59 logmsgbot: ori scap failed: OSError [Errno 10] No child processes (duration: 01m 46s)
• 19:58 logmsgbot: ori Started scap: (no message)
• 19:52 ori: updated scap to master
• 19:11 ori: running apache graceful-stop on mw1042 to test mod_status behavior during graceful stop
• 19:02 logmsgbot: twentyafterfour Finished scap: New deployment branch: 1.26wmf11 try #2 (13 apaches failed) (duration: 03m 50s)
• 18:58 logmsgbot: twentyafterfour Started scap: New deployment branch: 1.26wmf11 try #2 (13 apaches failed)
• 18:53 logmsgbot: twentyafterfour Finished scap: New deployment branch: 1.26wmf11 (duration: 26m 37s)
• 18:31 godog: start rolling-downgrade of cassandra to 2.1.3 T102015
• 18:27 logmsgbot: twentyafterfour Started scap: New deployment branch: 1.26wmf11
• 18:13 logmsgbot: ori Finished scap: (no message) (duration: 04m 34s)
• 18:11 paravoid: reloading nginx on all cp* for reuseport
• 18:08 logmsgbot: ori Started scap: (no message)
• 17:57 ori: repooled scap-test servers (mw1170-mw1175 and mw1270-mw1275)
• 17:16 logmsgbot: ori Finished scap: (no message) (duration: 01m 42s)
• 17:14 logmsgbot: ori Started scap: (no message)
• 17:10 logmsgbot: ori Finished scap: (no message) (duration: 01m 34s)
• 17:09 logmsgbot: ori Started scap: (no message)
• 17:06 logmsgbot: ori scap aborted: (no message) (duration: 01m 23s)
• 17:04 logmsgbot: ori Started scap: (no message)
• 16:53 logmsgbot: bd808 Finished scap: no-op sync to scap-test dsh group; Testing HHVM restart take 4 (duration: 01m 30s)
• 16:52 logmsgbot: bd808 Started scap: no-op sync to scap-test dsh group; Testing HHVM restart take 4
• 16:45 cscott: updated OCG to version db7a56965233a74c73917c78b5c8c84c867321d9
• 16:37 logmsgbot: bd808 Finished scap: no-op sync to scap-test dsh group; Testing HHVM restart take 3 (duration: 01m 12s)
• 16:35 logmsgbot: bd808 Started scap: no-op sync to scap-test dsh group; Testing HHVM restart take 3
• 16:35 bd808: updated scap to da64a65 (Cast pid read from file to an int)
• 16:26 logmsgbot: bd808 Finished scap: no-op sync to scap-test dsh group; Testing HHVM restart take 2 (duration: 01m 26s)
• 16:25 logmsgbot: bd808 Started scap: no-op sync to scap-test dsh group; Testing HHVM restart take 2
• 16:22 bd808: updated scap to 947b93f (Fix reference to _get_apache_list)
• 16:12 logmsgbot: bd808 scap failed: AttributeError 'Scap' object has no attribute '_get_apache_list' (duration: 02m 15s)
• 16:10 logmsgbot: bd808 Started scap: no-op sync to scap-test dsh group; Testing HHVM restart
• 16:01 paravoid: staggered upgrade of cp* fleet to nginx 1.9.2
• 15:57 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: Follow-up 94e5fd2: Default wmgUseContentTranslation true only on Wikipedias gerrit:220161 (duration: 00m 16s)
• 15:49 jynus: rebooting es1004
• 15:09 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Enable CX as default except where it is not deployed gerrit:220078 (duration: 00m 12s)
• 15:04 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable 'frwiki-recommender' campaign in frwiki gerrit:220071 (duration: 00m 13s)
• 14:54 paravoid: reprepro: including nginx 1.9.2-1~bpo8+1 to jessie-wikimedia/backports
• 14:39 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Repool es1003, depool es1004 (duration: 00m 12s)
• 14:04 cscott: reverted OCG to version ca4f64852de5b1de782b292b50038fbd2dd84266 (bundler failing with exit code 8)
• 13:57 cscott: updated OCG to version d7c698d5bf730d34057945e912ac75dc542dd788
• 13:44 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/209744/ (duration: 00m 13s)
• 13:44 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/209744/ (duration: 00m 12s)
• 12:54 moritzm: ssh on precise hosts has been updated to a backport of 6.6p1-2ubuntu2 (the version from trusty). this allows us to use modern crypto (plus labs can simplify key handling)
• 12:45 jynus: rebooting es1003
• 12:18 moritzm: uploaded openssh_6.6p1-2ubuntu2~wmfprecise2 to precise-wikimedia on apt.wikimedia.org
• 12:10 logmsgbot: hoo Synchronized arbitraryaccess.dblist: Arbitrary access for ruwiki and cswiki. T102122 (duration: 00m 12s)
• 11:33 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Repool es1002, depool es1003 (part 2/2) (duration: 00m 12s)
• 11:25 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Repool es1002, depool es1003 (duration: 00m 12s)
• 09:41 moritzm: updated jsch on gallium and lanthanum to support modern SSH key exchange in Jenkins (actually that happened yesterday, but I forgot to log it back then)
• 09:41 moritzm: added jsch_0.1.50-1ubuntu1~wmfprecise1 to precise-wikimedia on carbon
• 09:09 akosiaris: failing over etherpad to db1016
• 04:53 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jun 23 04:53:17 UTC 2015 (duration 53m 16s)
• 03:33 springle: xtrabackup clone db2023 to db1045
• 02:26 logmsgbot: LocalisationUpdate completed (1.26wmf10) at 2015-06-23 02:26:44+00:00
• 02:23 logmsgbot: l10nupdate Synchronized php-1.26wmf10/cache/l10n: (no message) (duration: 06m 47s)
• 01:17 logmsgbot: krinkle Synchronized docroot and w: (no message) (duration: 00m 12s)
• 01:00 bd808: Pruned virt1000 from trebuchet minions list: redis-cli srem "deploy:scap/scap:minions" virt1000.wikimedia.org

June 22

• 23:42 gwicke: restarted Cassandra on restbase1006
• 23:27 logmsgbot: catrope Synchronized php-1.26wmf10/extensions/MobileFrontend: For real this time (duration: 00m 14s)
• 23:27 logmsgbot: catrope Synchronized php-1.26wmf10/extensions/Gather: For real this time (duration: 00m 13s)
• 23:17 logmsgbot: catrope Synchronized php-1.26wmf10/extensions/Gather: SWAT (duration: 00m 12s)
• 23:17 logmsgbot: catrope Synchronized php-1.26wmf10/extensions/MobileFrontend/: SWAT (duration: 00m 15s)
• 23:12 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings.php: Enable TinyRGB ICC profile swapping on testwiki (duration: 00m 13s)
• 22:51 logmsgbot: ori Synchronized php-1.26wmf10/resources/src/mediawiki/mediawiki.Title.js: I0e5f2d3b2: Fix undeclared dependency on jquery.mwExtension (duration: 00m 12s)
• 22:45 gwicke: restarting Cassandra on restbase1005 to get the metrics back
• 22:37 gwicke: restarting Cassandra on restbase1004 to get the metrics back
• 22:33 gwicke: restarting Cassandra on restbase1003 to get the metrics back
• 22:24 gwicke: restarting Cassandra on restbase1002 to get the metrics back
• 22:19 bd808: scap error "@ERROR: access denied to common from localhost (127.0.0.1)" from mw2187 and mw2080 on sync-file test.
• 22:17 logmsgbot: bd808 Synchronized README: Testing sync-file after scap update (duration: 00m 12s)
• 22:08 RoanKattouw: Deployed patch for T103054
• 21:59 godog: reboot restbase1008
• 21:56 bd808: updated scap to 81b7c14 (Move dsh group file names to config)
• 21:55 bd808: trebuchet checkout for scap/scap failed on 23 hosts: mw1104, mw1222, mw2009, mw2011, mw2021, mw2028, mw2031, mw2034, mw2069, mw2076, mw2080, mw2086, mw2095, mw2099, mw2120, mw2127, mw2131, mw2136, mw2170, mw2187, mw2189, mw2197, virt1000
• 21:50 bd808: trebuchet fetch for scap/scap failed on mw2086.codfw.wmnet, mw1222.eqiad.wmnet and virt1000.wikimedia.org
• 21:41 gwicke: restarting Cassandra on restbase1001 to get the metrics back
• 21:20 ori: Depooled mw1170-mw1175 and mw1270-mw1275 for testing Idddcfe46
• 21:07 chasemp: rebooting mw1101 the hard way
• 20:28 cscott: updated Parsoid to version d488783e
• 19:34 akosiaris: delete pad:ips from etherpad
• 19:01 jynus: rebooting es1002
• 18:52 logmsgbot: ori Synchronized php-1.26wmf10/includes/OutputPage.php: I0e5f2d3b2: Construct clean canonical URLs for wiki pages, ignoring request URL (T67402) (duration: 00m 14s)
• 18:01 legoktm: live-hacking mw1017 to debug T103053
• 17:49 mutante: Bugzilla has left the building
• 16:31 jynus: reseting wikitech-static mysql contents to improve fragmentation
• 16:26 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Repool es1001, depool es1002 (duration: 00m 14s)
• 16:12 andrewbogott: shutting down virt1000
• 16:08 andrewbogott: disabling puppet on virt1000
• 16:07 ottomata: deploying eventlogging 0.9. This includes changes for arbitrary eventlogging URIs in all eventlogging stages, as well as support for schema based kafka topic URIs.
• 15:24 logmsgbot: thcipriani Synchronized php-1.26wmf10/extensions/WikiEditor: SWAT: Reduce 'Edit' EventLogging schema sampling rate to 6.25% (1/16th) gerrit:219837 (duration: 00m 13s)
• 15:04 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: Default wmgUseWikibaseQuality on beta to true. gerrit:219630 (duration: 00m 14s)
• 14:32 hashar: restarting Jenkins
• 13:26 jynus: rebooting es1001 for regular maintenance
• 12:08 paravoid: powercycled ms-be1002, stuck at console
• 11:12 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Depool es1001 (duration: 00m 13s)
• 11:06 _joe_: restarting hhvm on the low-memory appservers (main and api)
• 09:23 hashar: upgrading Jenkins gearman plugin from 0.1.1 to latest master (f2024bd). Restarting Jenkins.
• 05:11 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jun 22 05:11:22 UTC 2015 (duration 11m 21s)
• 02:31 logmsgbot: LocalisationUpdate completed (1.26wmf10) at 2015-06-22 02:31:32+00:00
• 02:27 logmsgbot: l10nupdate Synchronized php-1.26wmf10/cache/l10n: (no message) (duration: 07m 27s)
• 00:44 jgage: restarted gitblit on antimony again

June 21

• 11:28 jynus: restarting apache on mw1110
• 06:55 gwicke: restarted bootstrap on restbase1009 earlier today; hardware hasn't died yet
• 05:01 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jun 21 05:01:07 UTC 2015 (duration 1m 6s)
• 02:27 logmsgbot: LocalisationUpdate completed (1.26wmf10) at 2015-06-21 02:27:13+00:00
• 02:23 logmsgbot: l10nupdate Synchronized php-1.26wmf10/cache/l10n: (no message) (duration: 10m 23s)
• 01:39 jgage: restarted gitblit on antimony at 00:43 UTC
• 01:37 Krenair: testing morebots

June 20

• 22:50 bblack: restarted gitblit java service on antimony
• 04:27 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jun 20 04:27:14 UTC 2015 (duration 27m 13s)
• 02:21 logmsgbot: LocalisationUpdate completed (1.26wmf10) at 2015-06-20 02:21:30+00:00
• 02:18 logmsgbot: l10nupdate Synchronized php-1.26wmf10/cache/l10n: (no message) (duration: 07m 02s)

June 19

• 23:32 gwicke: upgraded restbase1006 to cassandra 2.1.7
• 23:30 gwicke: starting cassandra bootstrap on restbase1009
• 21:37 gwicke: upgraded cassandra on 1003 to 2.1.7 (pre-release, likely going out on Monday)
• 18:32 godog: stop cassandra on restbase1008
• 17:45 logmsgbot: krenair Synchronized private/PrivateSettings.php: sync 4a30446e for wikitech cleanup - T102361 (duration: 00m 12s)
• 17:24 godog: install linux 3.19 on restbase100[789]
• 17:12 ori: salt -t30 -G 'php:hhvm' cmd.run 'rm -f /usr/local/bin/check_tc_space' (https://gerrit.wikimedia.org/r/#/c/219102/)
• 16:54 moritzm: updated/rebooted nescio/maerlant to 3.19
• 13:40 andrewbogott: test test test
• 02:19 logmsgbot: LocalisationUpdate completed (1.26wmf10) at 2015-06-19 02:19:33+00:00
• 02:16 logmsgbot: l10nupdate Synchronized php-1.26wmf10/cache/l10n: (no message) (duration: 05m 08s)
• 00:49 springle: killed storm of research queries on dbstore1002, load avg 90+, replag, likely explosion, etc. emailing analytics@
• 00:13 logmsgbot: ebernhardson Synchronized php-1.26wmf10/extensions/Flow/tests/: no-op sync of flow test cases in wmf10 (duration: 00m 17s)
• 00:11 logmsgbot: ebernhardson Synchronized php-1.26wmf10/skins/Vector/: Bump Vector submodule in 1.26wmf10 for swat (duration: 00m 12s)

June 18

• 23:37 logmsgbot: ebernhardson Synchronized php-1.26wmf9/skins/Vector: Bump Vector in 1.26wmf9 for SWAT (duration: 00m 16s)
• 23:22 logmsgbot: ebernhardson Synchronized wmf-config/: Actually enable the feedback link on Special:Search (duration: 00m 17s)
• 23:08 logmsgbot: ebernhardson Synchronized wmf-config/InitialiseSettings.php: Enable wgCirrusSearchFeedbackLink on enwiki (duration: 00m 13s)
• 21:07 godog: start (bootstrap) cassandra on restbase1008
• 20:43 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-urd-hin_0.1.0+svn~r60389-1
• 20:17 akosiaris: restarted salt on sca1001, truncate log files. keep a sample in /tmp/
• 20:03 chasemp: apache && hhvm restart for mw 1243 1250 1254 1256 1257
• 20:00 chasemp: apache && hhvm restart for mw...1256 1255 1254 1250 1243 1242 1071 1021
• 19:58 mutante: restarting hhvm on mw1021, mw1071
• 19:27 godog: bounce cassandra on restbase1003, new logging configuration
• 19:26 akosiaris: puppet-merged on strontium
• 19:15 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.26wmf10
• 19:06 godog: upgrade cassandra to 2.1.6 on restbase1003
• 18:56 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-urd_0.1.0~r57551-1
• 18:56 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-hin_0.1.0~r57344-1
• 18:56 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-cy-en_0.1.1~r57554-1
• 18:43 legoktm: fixed content model of MediaWiki:Common.css@lrcwiki
• 18:18 YuviPanda: restarted nutcracker on wikitech
• 18:16 YuviPanda: restarted keystone on labcontrol1001
• 17:13 gwicke: bouncing cassandra on restbase1002
• 17:11 godog: restart cassandra on restbase1004
• 15:53 gwicke: updated restbase to 7ffaf94b
• 15:13 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Hovercards: Disable test release on Catalan and Greek Wikipedias gerrit:215932 (duration: 00m 13s)
• 15:06 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Add wikis for deployment on 20150618 gerrit:218886 (duration: 00m 14s)
• 11:14 akosiaris: powercycling labstore2001
• 09:08 moritzm: added firejail_0.9.26-1~wmfjessie1 and firejail_0.9.26-1~wmftrusty1 to apt.wikimedia.org
• 08:45 jynus: very brief replication stop for s7, already corrected
• 06:51 Coren: rebooting labstore2001
• 06:32 legoktm: live hacking mw1017 for T102915
• 05:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun 18 05:26:01 UTC 2015 (duration 26m 0s)
• 02:48 logmsgbot: LocalisationUpdate completed (1.26wmf10) at 2015-06-18 02:48:44+00:00
• 02:46 logmsgbot: l10nupdate Synchronized php-1.26wmf10/cache/l10n: (no message) (duration: 05m 03s)
• 02:32 logmsgbot: LocalisationUpdate completed (1.26wmf9) at 2015-06-18 02:32:45+00:00
• 02:28 logmsgbot: l10nupdate Synchronized php-1.26wmf9/cache/l10n: (no message) (duration: 05m 56s)
• 02:04 springle: applied T99941 scema change to all remaining affected (ie, old) wikis
• 02:01 tgr: ran https://gerrit.wikimedia.org/r/#/c/159350/7/backend/schema/mysql/developer_agreement.sql on mediawikiwiki
• 01:32 ejegg: updated payments from f33d0a8687a120a2057a7e6acad67da63b17f97e to a17ee221db0dbde70c92e24fc188379b6dbad613
• 01:20 logmsgbot: ori Synchronized php-1.26wmf10/resources/src/mediawiki.action/mediawiki.action.edit.stash.js: 0c21a14a6e: Revert StashEdit: Use postWithToken (duration: 00m 13s)
• 01:06 twentyafterfour: applied hotfix for T102276 and restarted apache on iridium
• 00:00 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.26wmf10

June 17

• 23:35 logmsgbot: catrope Synchronized php-1.26wmf10/extensions/Gather: SWAT (duration: 00m 14s)
• 23:35 gwicke: rolled back restbase to 90817c2a
• 23:24 logmsgbot: catrope Synchronized php-1.26wmf9/extensions/MobileFrontend: SWAT (duration: 00m 15s)
• 23:23 logmsgbot: catrope Synchronized php-1.26wmf9/extensions/Flow: SWAT (duration: 00m 15s)
• 22:45 gwicke: rolling restart of cassandra nodes
• 22:09 gwicke: rolling restart of restbase instances to apply puppet change after puppet actually ran on all nodes
• 21:58 gwicke: rolling restart of restbase instances to apply config change
• 21:56 godog: restart nutcracker on mw1145
• 21:35 gwicke: restarting cassandra on restbase1005
• 20:47 mutante: temp. stopped icinga-wm
• 20:37 gwicke: deployed RESTBase 7ffaf94bfc
• 20:24 cscott: updated Parsoid to version 402ddf66
• 20:01 ottomata: resized antimony's / LV from 30G to 100G. looks like /var/lib/git was getting filled up
• 19:43 jynus: rolling schema changes on hewiki
• 19:29 godog: downgrade and restart cassandra to 2.1.3 on restbase1001, metrics not being pushed to graphite with 2.1.6
• 19:05 godog: bounce cassandra on xenon
• 18:46 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: Ic03b152de: Make $wgUploadPath for commons https only for benefit instant commons (duration: 00m 14s)
• 18:11 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf10
• 17:45 godog: bounce cassandra on restbase1001
• 17:39 mutante: repooled mw1234
• 17:24 ottomata: starting reinstall of Zookeeper analytics nodes (analytics102[345]): https://phabricator.wikimedia.org/T101713
• 17:16 godog: bounce cassandra on restbase1001
• 17:14 jynus: rolling schema changes on ruwiki master
• 17:13 mutante: running puppet via salt on api appservers in batches, switch to ganglia_new and carbon
• 17:12 godog: cassandra stopped sending graphite metrics after restart, investigating (test cluster works fine tho)
• 16:58 jynus: rolling schema changes on ruwiki slaves
• 16:28 godog: start upgrading restbase1001 to cassandra 2.1.6 T102015
• 16:02 logmsgbot: thcipriani Finished scap: Wikitech-Ldap host record roll-out (duration: 24m 35s)
• 15:37 logmsgbot: thcipriani Started scap: Wikitech-Ldap host record roll-out
• 15:19 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Give patrolmarks right to "*" on dewiki gerrit:218901 (duration: 00m 13s)
• 15:17 logmsgbot: anomie Synchronized wmf-config/throttle.php: SWAT: Add a throttle exception for United Islands of Prague gerrit:217413 (duration: 00m 14s)
• 15:15 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable captcha on labswiki for now gerrit:218908 (duration: 00m 13s)
• 15:10 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Add extra namespace aliases for Italian Wikipedia gerrit:215708 (duration: 00m 13s)
• 15:08 anomie: SWAT: Enable anti-abuse features on labswiki gerrit:218903
• 15:08 jynus: testing some schema changes on testwiki
• 15:00 logmsgbot: aude Synchronized usagetracking.dblist: Enable Wikibase usage tracking on nowiki and plwiki (duration: 00m 13s)
• 13:56 logmsgbot: aude Synchronized usagetracking.dblist: Enable Wikibase usage tracking on fiwiki and idwiki (duration: 00m 13s)
• 13:26 logmsgbot: aude Synchronized usagetracking.dblist: Enable Wikibase usage tracking on bgwiki and eowiki (duration: 00m 13s)
• 10:52 akosiaris: reload pybal on lvs1006
• 10:50 mobrovac: finished deploying mathoid I40ef68 on SCA
• 10:48 akosiaris: repooled mathoid.svc.eqiad.wmnet: sca1002 backend
• 10:44 akosiaris: enable puppet on sca1002
• 10:43 akosiaris: enable puppet
• 10:43 akosiaris: depool sca1002 for mathoid.svc.eqiad.wmnet
• 10:43 akosiaris: reloaded pybal on lvs1003
• 10:28 akosiaris: repool sca1002, depool sca1001
• 10:18 mark: Halting pvmove of md124 on labstore1001
• 09:30 akosiaris: disable puppet on sca1001
• 09:09 akosiaris: depool sca1001, resource: mathoid
• 09:09 akosiaris: puppet disabled on sca1002
• 08:37 YuviPanda: run sudo salt -t 20 -b 100 '*' cmd.run 'sudo service salt-minion restart' on virt1000, attempt to get them to answer on labcontrol1001 instead
• 06:52 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jun 17 06:52:58 UTC 2015 (duration 52m 57s)
• 02:56 logmsgbot: LocalisationUpdate completed (1.26wmf10) at 2015-06-17 02:56:49+00:00
• 02:55 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1045 (duration: 00m 13s)
• 02:54 springle: found wikiversions.json modified on tin since 2015-06-16 23:27 (catrope?); stashed and reapplied the file in order to do a pull
• 02:54 logmsgbot: l10nupdate Synchronized php-1.26wmf10/cache/l10n: (no message) (duration: 04m 44s)
• 02:35 logmsgbot: LocalisationUpdate completed (1.26wmf9) at 2015-06-17 02:35:23+00:00
• 02:32 logmsgbot: l10nupdate Synchronized php-1.26wmf9/cache/l10n: (no message) (duration: 06m 12s)
• 02:21 logmsgbot: ori Synchronized php-1.26wmf9/extensions/CentralNotice/modules/ext.centralNotice.bannerController/bannerController.js: I480cbc7ad (duration: 00m 12s)
• 02:21 logmsgbot: ori Synchronized php-1.26wmf10/extensions/CentralNotice/modules/ext.centralNotice.bannerController/bannerController.js: I480cbc7ad (duration: 00m 12s)
• 00:10 paravoid: draining esams because of upcoming network maintenance window

June 16

• 23:28 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings.php: Enable local upload on fawikivoyage; enable logging for T76305 (duration: 00m 13s)
• 23:28 logmsgbot: catrope Synchronized wmf-config/CommonSettings.php: Set previous values for password length policies (duration: 00m 16s)
• 23:17 logmsgbot: twentyafterfour Finished scap: testwiki to 1.26wmf10 (duration: 43m 04s)
• 23:02 godog: restore INFO cassandra logging level on restbase1003
• 22:44 godog: start cassandra on restbase1008
• 22:43 godog: enable back some cassandra debugging on restbase1003
• 22:33 logmsgbot: twentyafterfour Started scap: testwiki to 1.26wmf10
• 22:26 urandom: restored default logging level on restbase1003
• 22:22 urandom: enabling even more debugging on restbase1003
• 22:14 urandom: enable (some) debug logging on restbase1003
• 21:57 logmsgbot: twentyafterfour scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="testwiki" --list-file="/srv/mediawiki-staging/wmf-config/extension-list" --output="/tmp/tmp.SxGNHsmVYP" ' returned non-zero exit status 1 (duration: 01m 24s)
• 21:56 logmsgbot: twentyafterfour Started scap: testwiki to 1.26wmf10
• 20:34 logmsgbot: krinkle Synchronized php-1.26wmf9/extensions/WikimediaEvents/modules/ext.wikimediaEvents.resourceloader.js: T101806 live hack (duration: 00m 12s)
• 19:24 Coren: labstore1001 pvmove of slice2 to slice 51 started; some bursts of iowait expected but should have minimal enduser impact)
• 18:36 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Fix usage tracking setting (duration: 00m 14s)
• 18:03 godog: bounce statsite on graphite1001, stuck while writing to graphite
• 17:30 ejegg: update SmashPig on listener from e1e925c9fc2a60c1e14ef01d8b653dc09512f51f to 258f2c917b1ae50b01231927bcd6f58ecaa8940b
• 17:23 logmsgbot: krinkle Synchronized php-1.26wmf9/includes/resourceloader/ResourceLoader.php: undo live hack (duration: 00m 13s)
• 17:09 logmsgbot: aude Synchronized arbitraryaccess.dblist: Enable arbitrary access on gomwiki and lrcwiki (duration: 00m 13s)
• 17:09 logmsgbot: aude Synchronized usagetracking.dblist: Enable Wikibase usage tracking on second batch of s3 wikis (duration: 00m 13s)
• 17:03 logmsgbot: bblack Synchronized wmf-config/InitialiseSettings.php: wgCanonicalServer: HTTPS for all (duration: 00m 15s)
• 16:44 logmsgbot: krenair Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 13s)
• 16:43 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 13s)
• 16:43 logmsgbot: krenair Synchronized w/static/images/project-logos/gomwiki.png: (no message) (duration: 00m 14s)
• 16:42 logmsgbot: krenair Synchronized langlist: gomwiki (duration: 00m 13s)
• 16:41 logmsgbot: krenair rebuilt wikiversions.cdb and synchronized wikiversions files: (no message)
• 16:40 logmsgbot: krenair Synchronized database lists: (no message) (duration: 00m 13s)
• 16:29 logmsgbot: krenair Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 13s)
• 16:27 logmsgbot: krenair Synchronized langlist: (no message) (duration: 00m 14s)
• 16:25 logmsgbot: krenair Synchronized w/static/images/project-logos/lrcwiki.png: (no message) (duration: 00m 13s)
• 16:21 moritzm: updated copper, oxygen, labstore2001 and labnodepool1001 to the 3.19 kernel
• 16:11 logmsgbot: krenair Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 13s)
• 16:10 logmsgbot: krenair Synchronized wmf-config: (no message) (duration: 00m 14s)
• 16:06 logmsgbot: krenair rebuilt wikiversions.cdb and synchronized wikiversions files: (no message)
• 16:05 logmsgbot: krenair Synchronized database lists: (no message) (duration: 00m 15s)
• 15:43 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: templateeditor: add templateeditor right in hewiki gerrit:218426 (duration: 00m 13s)
• 15:09 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Turn on wgGenerateThumbnailOnParse for wikitech. gerrit:218553 (duration: 00m 12s)
• 15:03 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Add wikis for CX deployment on 20150616 gerrit:218341 (duration: 00m 12s)
• 14:18 cmjohnson: barium is going down for disk replacement
• 13:38 logmsgbot: aude Synchronized usagetracking.dblist: Enable Wikibase usage tracking on dewiki (duration: 00m 15s)
• 13:18 akosiaris: rebooted etherpad1001 for kernel upgrades
• 12:51 logmsgbot: jynus Synchronized wmf-config/db-codfw.php: Repool es2005, es2006 and es2007 after maintenance (duration: 00m 13s)
• 12:44 logmsgbot: aude Synchronized usagetracking.dblist: Enable Wikibase usage tracking on cswiki (duration: 00m 14s)
• 12:20 logmsgbot: aude Synchronized usagetracking.dblist: Enable usage tracking on ruwiki (duration: 00m 15s)
• 11:21 paravoid: restarting the puppetmaster
• 11:19 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1073, warm up (duration: 00m 13s)
• 10:36 akosiaris: rebooting ganeti200{1..6}.codfw.wmnet for kernel upgrades
• 09:33 logmsgbot: jynus Synchronized wmf-config/db-codfw.php: Depool es2005, es2006 and es2007 for maintenance (duration: 00m 14s)
• 09:10 YuviPanda: deleted huge puppet-master.log on labcontrol1001
• 08:05 jynus: added m5-slave to dns servers
• 07:52 paravoid: restarting hhvm on mw1121
• 07:52 moritzm: blacklisted the overlayfs kernel module (prevents a reliable local root exploit on all Ubuntu systems). no systems in the fleet had an overlaysfs mount present or the kernel module loaded, so there should be no impact on existing systems. Note: This is a bandaid, I'll create a Phab task to deploy this via puppet in the future (and to also blacklist additional desktopy kernel modules which increase our attack
• 07:39 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Repool es1005 (duration: 00m 14s)
• 06:24 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jun 16 06:24:04 UTC 2015 (duration 24m 3s)
• 06:18 godog: restore ES replication throttling to 20mb/s
• 06:13 godog: restore ES replication throttling to 40mb/s
• 06:08 logmsgbot: filippo Synchronized wmf-config/PoolCounterSettings-common.php: unthrottle ES (duration: 00m 14s)
• 05:56 godog: bump ES replication throttling to 60mb/s
• 05:50 manybubbles: ok - we're yellow and recovering. ops can take this from here. We have a root cause and we have things I can complain about to the elastic folks I plan to meet with today anyway. I'm going to finish waking up now.
• 05:49 manybubbles: reenabling puppet agent on elasticsearch machines
• 05:46 manybubbles: I expect them to be red for another few minutes during the initial master recovery
• 05:45 manybubbles: started all elasticsearch nodes and now they are recovering.
• 05:41 godog: restart gmond on elastic1007
• 05:39 logmsgbot: filippo Synchronized wmf-config/PoolCounterSettings-common.php: throttle ES (duration: 00m 13s)
• 05:25 manybubbles: shutting down all the elasticsearch on the elasticsearch nodes against - another full cluster restart should fix it like it did last time...............
• 05:11 godog: restart elasticsearch on elastic1031
• 03:06 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1073 (duration: 00m 12s)
• 02:27 logmsgbot: LocalisationUpdate completed (1.26wmf9) at 2015-06-16 02:27:51+00:00
• 02:24 logmsgbot: l10nupdate Synchronized php-1.26wmf9/cache/l10n: (no message) (duration: 05m 52s)
• 00:55 tgr: running extensions/Gather/maintenance/updateCounts.php for gather wikis - https://phabricator.wikimedia.org/T101460
• 00:52 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1057, warm up (duration: 00m 13s)
• 00:46 godog: killed bacula-fd on graphite1001, shouldn't be running and consuming bandwidth (cc akosiaris)
• 00:27 godog: kill python stats on cp1052, filling /tmp

June 15

• 23:42 ori: Cleaning up renamed jobqueue metrics on graphite{1,2}001
• 23:01 godog: killed bacula-fd on graphite2001, shouldn't be running and consuming bandwidth (cc akosiaris)
• 22:54 logmsgbot: hoo Synchronized wmf-config/filebackend.php: Fix commons image inclusion after commons went https only (duration: 00m 14s)
• 22:18 godog: run disk stress-test on restbase1007 / restbase1009
• 22:06 logmsgbot: twentyafterfour Synchronized hhvm-fatal-error.php: deploy: Guard header() call in error page (duration: 00m 15s)
• 22:05 logmsgbot: twentyafterfour Synchronized wmf-config/InitialiseSettings-labs.php: deploy: Never use wgServer/wgCanonicalServer values from production in labs (duration: 00m 12s)
• 20:37 logmsgbot: yurik Synchronized docroot/bits/WikipediaMobileFirefoxOS: Bumping FirefoxOS app to latest (duration: 00m 14s)
• 20:30 godog: bounce cassandra on restbase1003
• 20:18 godog: start cassandra on restbase1008, bootstrapping
• 20:04 godog: sign restbase1008 key, run puppet
• 20:00 godog: powercycle restbase1007, investigate disk issue
• 19:07 logmsgbot: ori Synchronized php-1.26wmf9/includes/jobqueue: 0a32aa3be4: jobqueue: use more sensible metric key names (duration: 00m 13s)
• 16:57 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Grant cloudadmins the 'editallhiera' right gerrit:218115 (duration: 00m 14s)
• 16:48 logmsgbot: thcipriani Synchronized php-1.26wmf9/extensions/OpenStackManager/OpenStackManagerHooks.php: SWAT: refer to user the right way (duration: 00m 13s)
• 16:48 godog: powercycle graphite1002, no ssh, unresponsive console
• 16:19 jynus: upgrading es1005 mysql service while depooled
• 16:12 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Grant cloudadmins the 'editallhiera' right gerrit:218115 (duration: 00m 12s)
• 16:10 bblack: pybal restarts complete, all ok
• 16:09 logmsgbot: thcipriani Finished scap: SWAT: Openstack manager and language updates (duration: 21m 27s)
• 15:47 logmsgbot: thcipriani Started scap: SWAT: Openstack manager and language updates
• 15:46 bblack: starting pybal restart process for config changes ( https://gerrit.wikimedia.org/r/#/c/218285/ ), inactives first w/ manual verification of ok-ness
• 15:11 bblack: rebooting cp3041 (downtimed)
• 15:00 _joe_: ES is green
• 14:38 logmsgbot: aude Synchronized php-1.26wmf9/extensions/Wikidata: Fix property label constraints bug (duration: 00m 24s)
• 14:27 logmsgbot: aude Synchronized arbitraryaccess.dblist: Enable arbitrary access on s7 wikis (duration: 00m 13s)
• 13:47 jynus: enabling puppet on all elastic* nodes, should enable also ganglia
• 13:11 logmsgbot: demon Synchronized wmf-config/PoolCounterSettings-common.php: all the search (duration: 00m 12s)
• 13:04 _joe_: re-scaling down the recovery index bandwidth in ES to 20 mb/s
• 12:52 logmsgbot: demon Synchronized wmf-config/PoolCounterSettings-common.php: partially turn search back on (duration: 00m 13s)
• 11:54 _joe_: raised the ES index replica bandwidth limit to 60mb
• 11:31 akosiaris: migrating etherpad.wikimedia.org to etherpad1001.eqiad.wmnet
• 11:15 _joe_: raised the max bytes for ES recovery to 40mbps
• 10:49 manybubbles: and we're yellow right now.
• 10:49 manybubbles: the initial primaries stage - the red stage of the rolling restart - recovers quick-ish
• 10:48 manybubbles: soon we should see it go yellow and stay that way while the replicas recover
• 10:48 manybubbles: manybubbles is confident his mighty bitch slap of the elasticsearch cluster has set it further to the road to recovery
• 10:46 jynus: disabled puppet on all elasticsearch nodes to avoid restarting services and other magic
• 10:44 _joe_: disabled hot threads logging, ganglia on es nodes
• 10:44 manybubbles: started Elasticsearch on all elasticsearch nodes
• 10:38 manybubbles: stopping all elasticsearch servers - going for a full cluster resstart.
• 10:11 manybubbles: restarting elasticsearch on elasticsearch1021 - that one is in a gc death spiral
• 09:26 logmsgbot: oblivian Synchronized wmf-config/PoolCounterSettings-common.php: temporarily throttle down cirrussearch (duration: 00m 13s)
• 09:12 logmsgbot: oblivian Synchronized wmf-config/PoolCounterSettings-common.php: temporarily throttle down cirrussearch (duration: 00m 13s)
• 07:35 _joe_: attempting a fast restart of elastic1020
• 07:21 logmsgbot: ori Synchronized php-1.26wmf9/extensions/CirrusSearch/includes/Util.php: I504dac0c3: Add missing 'use \Status;' to includes/Util.php (duration: 00m 13s)
• 04:56 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jun 15 04:56:39 UTC 2015 (duration 56m 38s)
• 03:31 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1057 (duration: 00m 12s)
• 02:22 logmsgbot: LocalisationUpdate completed (1.26wmf9) at 2015-06-15 02:22:56+00:00
• 02:19 logmsgbot: l10nupdate Synchronized php-1.26wmf9/cache/l10n: (no message) (duration: 05m 46s)

June 14

• 10:39 YuviPanda: running du -d 2 on /srv/project in a screen sesssion on labstore1001
• 04:33 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jun 14 04:33:20 UTC 2015 (duration 33m 19s)
• 02:42 logmsgbot: reedy Synchronized wmf-config/extension-list: noop (duration: 00m 13s)
• 02:40 logmsgbot: krenair Synchronized wmf-config/squid-labs.php: sync random labs-only file to test per irc (duration: 00m 13s)
• 02:21 logmsgbot: LocalisationUpdate completed (1.26wmf9) at 2015-06-14 02:21:28+00:00
• 02:18 logmsgbot: l10nupdate Synchronized php-1.26wmf9/cache/l10n: (no message) (duration: 05m 47s)

June 13

• 19:30 bblack: repooled cp1071, cp3040
• 18:53 bblack: rebooting cp1071, cp3040 to look at BIOS-level things (depooled, icinga-downed)
• 17:08 logmsgbot: krinkle Synchronized php-1.26wmf9/extensions/WikimediaEvents: T101806 (duration: 00m 12s)
• 15:47 paravoid: labstore1001: stopping manage-nfs-volumes daemon
• 04:41 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jun 13 04:41:57 UTC 2015 (duration 41m 56s)
• 03:51 Krinkle: Running deleteEqualMessages.php for sawiki (T45917)
• 03:49 Krinkle: Running deleteEqualMessages.php for cewiki (T45917)
• 02:21 logmsgbot: LocalisationUpdate completed (1.26wmf9) at 2015-06-13 02:20:58+00:00
• 02:18 logmsgbot: l10nupdate Synchronized php-1.26wmf9/cache/l10n: (no message) (duration: 05m 19s)
• 00:17 gwicke: restarted cassandra on restbase1001
• 00:13 gwicke: restarted cassandra on restbase1002

June 12

• 22:57 ejegg: rolled back SmashPig on listener from 15acdafef9d9682c417632e5ac5a5f2e5380f92e to e1e925c9fc2a60c1e14ef01d8b653dc09512f51f
• 22:40 ejegg: updated SmashPig on listener from e1e925c9fc2a60c1e14ef01d8b653dc09512f51f to 15acdafef9d9682c417632e5ac5a5f2e5380f92e
• 22:24 godog: upgrade and bounce carbon daemons on graphite2001 to investigate T101572
• 21:16 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: I3694489ba: wgCanonicalServer->https for new HTTPS domains (duration: 00m 14s)
• 20:33 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/217878/1 (duration: 00m 13s)
• 20:32 logmsgbot: krenair Synchronized w/static/images/project-logos/dawiki-200k.png: https://gerrit.wikimedia.org/r/#/c/217878/1 (duration: 00m 16s)
• 20:15 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/217670/ (duration: 00m 12s)
• 19:28 ejegg: updated SmashPig on payments-listener from f9c3eaa99fa0fe8ef098d0fc876091d3676aa039 to 5a463400bc74706ba7bf6256cd0101014e792acb
• 19:28 ejegg: updated SmashPig on payments-listener ccepting New Patients:
• 18:47 ejegg: updated SmashPig on payments-listener from 7fed22ad933a6d3e371d60dfc6f8fdd0f9131510 to f9c3eaa99fa0fe8ef098d0fc876091d3676aa039
• 18:45 logmsgbot: faidon Synchronized wmf-config/InitialiseSettings.php: remove wmgHTTPSBlacklistCountries (duration: 00m 12s)
• 18:45 logmsgbot: faidon Synchronized wmf-config/CommonSettings.php: remove CanIPUseHTTPS hook (duration: 00m 13s)
• 17:39 moritzm: updated cerium, xenon and praseodymium to 3.19 kernel
• 17:08 ejegg: enabled queue consumer
• 17:08 ejegg: updated crm from d13aaa4e9e937b0b1ae1f5de61ea7ff1f316d58f to bd8a00196071ddd04efbff7b30567dd9357c9000
• 16:53 ejegg: disabled donations queue consumer
• 15:52 logmsgbot: faidon Synchronized wmf-config/CommonSettings.php: hide prefershttps user pref (duration: 00m 13s)
• 15:40 logmsgbot: faidon Synchronized docroot/search.wikimedia.org/index.php: unbreak search.wikimedia.org due to HTTPS (duration: 00m 12s)
• 15:27 jynus: mysql load issues on labsdb1003, investigating
• 13:39 moritzm: updated etcd* to 3.19 kernel
• 12:11 jynus: restarting mariadb at labsdb1003
• 11:58 moritzm: updated rdb200* to 3.19 kernel
• 11:31 jynus: db2068 up but all services and console login unresponsive, powercycling
• 10:06 springle: killed a bunch of queries hammering labsdb1003 for days
• 09:58 moritzm: updated mc2004 to mc2016 to 3.19 kernel
• 06:06 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jun 12 06:06:55 UTC 2015 (duration 6m 54s)
• 04:37 logmsgbot: ori Synchronized php-1.26wmf9/extensions/FlaggedRevs: I4cfb47b41: Avoid post-redirect parse for certain edits (duration: 00m 14s)
• 02:40 logmsgbot: LocalisationUpdate completed (1.26wmf9) at 2015-06-12 02:40:36+00:00
• 02:34 logmsgbot: l10nupdate Synchronized php-1.26wmf9/cache/l10n: (no message) (duration: 10m 00s)
• 00:40 logmsgbot: legoktm Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/217759 (duration: 00m 15s)
• 00:07 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings-labs.php: (no message) (duration: 00m 14s)

June 11

• 23:59 logmsgbot: legoktm Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/217753 (duration: 00m 16s)
• 23:54 logmsgbot: ori Synchronized php-1.26wmf9/includes/EditPage.php: cf7df757f2: Instrument edit failures (duration: 00m 14s)
• 23:41 logmsgbot: ebernhardson Synchronized php-1.26wmf9/extensions/MobileFrontend: Bump MobileFrontend in 1.26wmf9 for SWAT (duration: 00m 14s)
• 23:40 ejegg: updated civicrm from 7ffe0cefb019828a09c9369187f14518847b5f41 to d13aaa4e9e937b0b1ae1f5de61ea7ff1f316d58f
• 23:24 logmsgbot: ebernhardson Synchronized php-1.26wmf9/extensions/CirrusSearch/: Fix prefer-recent queries in cirrussearch (duration: 00m 13s)
• 23:02 ejegg: updated SmashPig on the rest of the cluster from 477e8a8be5ea895262031c147330de5a651cc3ac to 7fed22ad933a6d3e371d60dfc6f8fdd0f9131510
• 22:17 godog: temporary bump php memory_limit on magnesium to test T102092
• 22:11 ejegg: updated SmashPig on payments-listener from 477e8a8be5ea895262031c147330de5a651cc3ac to 7fed22ad933a6d3e371d60dfc6f8fdd0f9131510
• 21:54 ori: Widespread TC cache exhaustion again, doing rolling restart of HHVMs
• 21:46 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I3d3ed7647: Test LCStoreStaticArray on test2wiki (duration: 00m 14s)
• 21:01 godog: NPE while trying to make restbase1007 (cassandra 2.1.5) join the cluster, trying matching the same cassandra version (2.1.3)
• 20:57 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: fix last commit, did not have any affect (duration: 00m 16s)
• 20:55 ejegg: updated payments from 43c7952d2a31deaea97e8319f5612d644dce43c8 to f33d0a8687a120a2057a7e6acad67da63b17f97e
• 20:54 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/217688/1 (duration: 00m 13s)
• 20:10 godog: sign restbase1007 puppet key and first puppet run
• 19:10 logmsgbot: legoktm Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/217591 (duration: 00m 13s)
• 18:58 logmsgbot: legoktm Synchronized wmf-config/InitialiseSettings-labs.php: beta only change - https://gerrit.wikimedia.org/r/217560 (duration: 00m 12s)
• 18:55 logmsgbot: krinkle Synchronized php-1.26wmf9/extensions/WikimediaEvents: T101806 (duration: 00m 14s)
• 18:43 logmsgbot: twentyafterfour Synchronized php-1.26wmf9/includes/AjaxResponse.php: Hotfix Iafff9982bbbee893c13f891901dde88f998db7a6 (duration: 00m 14s)
• 18:16 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: all wikis to 1.26wmf9
• 17:44 ejegg: rolled back payments to 43c7952d2a31deaea97e8319f5612d644dce43c8
• 17:41 ejegg: updated payments from 43c7952d2a31deaea97e8319f5612d644dce43c8 to 15f24d24b150d5d774314b0c1b40ae26a73185f2
• 17:00 moritzm: updated mc200[1-3] to linux 3.19
• 16:28 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Use arbitrary access tag (duration: 00m 12s)
• 16:27 logmsgbot: aude Synchronized wmf-config/CommonSettings.php: Add arbitrary access group tag (duration: 00m 13s)
• 16:27 logmsgbot: aude Synchronized arbitraryaccess.dblist: Add dblist for arbitrary access wikis (duration: 00m 13s)
• 16:24 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Use usagetracking tag (duration: 00m 13s)
• 16:23 logmsgbot: aude Synchronized wmf-config/CommonSettings.php: Add usagetracking group tag (duration: 00m 16s)
• 16:23 ori: Scap + deployments exhausted TC cache on Apaches; performed a rolling restart of HHVM
• 16:21 logmsgbot: aude Synchronized usagetracking.dblist: Add dblist for usage tracking wikis (duration: 00m 25s)
• 16:19 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Disable Parsoid update jobs (duration: 00m 14s)
• 16:18 logmsgbot: thcipriani Finished scap: SWAT: Update namespaces and special pages for Northern Luri (lrc) from translatewiki gerrit:216533 gerrit:217327 (duration: 32m 11s)
• 15:46 logmsgbot: thcipriani Started scap: SWAT: Update namespaces and special pages for Northern Luri (lrc) from translatewiki gerrit:216533 gerrit:217327
• 15:27 logmsgbot: thcipriani Synchronized php-1.26wmf9/extensions/OpenStackManager: SWAT: update OpenStackManager to disable unused sudoer features gerrit:217407 (duration: 00m 13s)
• 15:11 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Make VisualEditor access RESTbase directly on all public wikis gerrit:214833 (duration: 00m 12s)
• 15:05 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Add wikis for deployment on 20150611 gerrit:217460 (duration: 00m 12s)
• 14:33 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable usage tracking on jawiki (duration: 00m 12s)
• 13:40 _joe_: rolling restart of all the restbase instances
• 13:33 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable usage tracking on frwiki (duration: 00m 12s)
• 13:32 _joe_: running puppet on all restbase hosts
• 13:19 _joe_: running puppet on restbase1001
• 13:16 _joe_: disabling puppet on restbase hosts in anticipation for merging https://gerrit.wikimedia.org/r/217431
• 13:11 paravoid: removing gdnsd from apt: precise-wikimedia (1.9.0-1~precise1/2.1.0-1~precise1), trusty-wikimedia (2.1.0-1), jessie-wikimedia (2.1.2-1~deb8u1)
• 12:13 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable arbitrary access on Wikivoyage and Wikiquote (duration: 00m 13s)
• 11:48 YuviPanda: reboot labvirt1005 for kernel upgrade
• 11:46 YuviPanda: installing linux-image-generic-lts-vivid on labvirt1005 to get a 3.19 kernel
• 09:51 akosiaris: uploaded ruby-jsduck_5.3.4 and ruby-rkelly-remix_0.0.6 on apt.wikimedia.org/jessie-wikimedia/main
• 08:18 akosiaris: recreating jessie chroots on copper
• 06:21 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun 11 06:21:53 UTC 2015 (duration 21m 52s)
• 04:44 twentyafterfour: upgraded phabricator at 1:50 UTC (belatedly logged...)
• 03:01 logmsgbot: LocalisationUpdate completed (1.26wmf9) at 2015-06-11 03:01:48+00:00
• 03:00 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1057, warm up (duration: 01m 16s)
• 02:59 logmsgbot: l10nupdate Synchronized php-1.26wmf9/cache/l10n: (no message) (duration: 05m 59s)
• 02:43 logmsgbot: LocalisationUpdate completed (1.26wmf8) at 2015-06-11 02:43:34+00:00
• 02:30 logmsgbot: l10nupdate Synchronized php-1.26wmf8/cache/l10n: (no message) (duration: 09m 13s)

June 10

• 23:23 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings.php: Add www.limis.lt to $wgCopyUploadsDomains (duration: 00m 19s)
• 22:07 logmsgbot: twentyafterfour Synchronized php-1.26wmf9/extensions/MobileFrontend/includes/skins/banners.mustache: Deploying https://gerrit.wikimedia.org/r/#/c/217417/ (duration: 00m 16s)
• 20:38 logmsgbot: ori Synchronized php-1.26wmf8/includes/Hooks.php: d6802ad7d6: Avoid section profiling in Hooks::run due to high overhead (duration: 00m 14s)
• 20:37 logmsgbot: ori Synchronized php-1.26wmf9/includes/Hooks.php: e552f4942d: Avoid section profiling in Hooks::run due to high overhead (duration: 00m 17s)
• 20:36 logmsgbot: ori Synchronized php-1.26wmf9/includes/User.php: 2f4f1e279d: Fixed "wfTimestamp() fed bogus time value" errors (duration: 00m 12s)
• 20:36 logmsgbot: ori Synchronized php-1.26wmf8/includes/User.php: 55e18123ca: Fixed "wfTimestamp() fed bogus time value" errors (duration: 00m 15s)
• 18:07 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: Group1 wikis to 1.26wmf9
• 16:14 godog: reboot ms-be2008 to check disk swap config
• 15:50 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: retry (duration: 01m 08s)
• 15:34 Krenair: sync failed to something like 25 hosts, cannot directly log into any of them either
• 15:17 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/215030/ - no code change, just docs - should not have to wait 9 days for this (duration: 01m 08s)
• 13:16 moritzm: installed curl security updates on elastic*, wtp*, db*, virt*, labs*, labmon*, labstore*, es*
• 12:38 paravoid: zirconium: rm -rf /var/log2 (last log there from Mar 20th 2014)
• 10:55 jynus: disruption for maintenance starting on labsdb1002 https://lists.wikimedia.org/pipermail/labs-l/2015-June/003766.html
• 03:02 logmsgbot: ori Synchronized php-1.26wmf8/includes/User.php: 55e18123ca: Fixed "wfTimestamp() fed bogus time value" (duration: 01m 07s)
• 03:01 logmsgbot: ori Synchronized php-1.26wmf9/includes/User.php: 2f4f1e279d: Fixed "wfTimestamp() fed bogus time value" (duration: 01m 08s)
• 02:36 logmsgbot: LocalisationUpdate completed (1.26wmf8) at 2015-06-10 02:35:44+00:00
• 02:31 logmsgbot: l10nupdate Synchronized php-1.26wmf8/cache/l10n: (no message) (duration: 07m 20s)
• 01:33 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1057 (duration: 01m 08s)
• 01:13 logmsgbot: ori Synchronized php-1.26wmf8/extensions/FlaggedRevs: 433fae7f23: Update FlaggedRevs for cherry-picks (duration: 01m 09s)
• 01:10 logmsgbot: ori Synchronized php-1.26wmf9/extensions/FlaggedRevs: 2cfc8c9f2b: Update FlaggedRevs for cherry-picks (duration: 01m 09s)

June 9

• 23:57 logmsgbot: catrope Synchronized php-1.26wmf8/includes/: Avoid parser cache miss that often occurs post-save (duration: 01m 14s)
• 23:29 logmsgbot: catrope Synchronized php-1.26wmf8/resources/src/mediawiki/mediawiki.js: touch (duration: 01m 08s)
• 23:23 logmsgbot: catrope Synchronized php-1.26wmf9/includes/resourceloader/ResourceLoaderOOUIImageModule.php: Fix OOUI image variants (duration: 01m 08s)
• 23:22 ori: Deleting unused metrics on graphite2001 (sum_sq and stddev) as well
• 23:21 logmsgbot: catrope Synchronized php-1.26wmf9/resources/src/mediawiki/mediawiki.js: Add logging for T101806 private modules (duration: 01m 08s)
• 23:20 ori: Deleting unused metrics in graphite1001 (sum_sq and stddev)
• 23:19 logmsgbot: catrope Synchronized php-1.26wmf8/resources/src/mediawiki/mediawiki.js: Add logging for T101806 private modules (duration: 01m 08s)
• 23:16 logmsgbot: catrope Synchronized wmf-config/CirrusSearch-common.php: fix total breakage of search in wmf9 (duration: 01m 08s)
• 22:44 andrewbogott: moving labs-ns0 from virt1000 to labcontrol1001
• 22:43 andrewbogott: stopping almost everything on virt1000
• 20:31 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.26wmf9
• 20:27 logmsgbot: twentyafterfour Finished scap: testwiki to php-1.26wmf9 and rebuild l10n cache (duration: 29m 24s)
• 19:58 logmsgbot: twentyafterfour Started scap: testwiki to php-1.26wmf9 and rebuild l10n cache
• 19:42 mutante: einsteinium - no console output after reboot command, powercycled, booting again
• 19:36 mutante: rebooting einsteinium
• 19:28 mutante: restarted apache on mw1227
• 17:30 mutante: wikitech-static: installing bunch of package upgrades on the external wikitech-static VM
• 17:13 cmjohnson1: db1058 replacing failed disk 7
• 16:20 cmjohnson1: analytics1028 going down for troubleshooting
• 16:17 kart_: updated cxserver to 4a71145
• 15:37 logmsgbot: thcipriani Synchronized php-1.26wmf8/extensions/Wikidata: SWAT: Update Wikidata - forward compat for usage tracking gerrit:216967 (duration: 01m 17s)
• 15:20 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT take II: Enabled Guided Tour on th.wikipedia gerrit:216950 (duration: 01m 08s)
• 15:19 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Enabled Guided Tour on th.wikipedia gerrit:216950 (duration: 01m 08s)
• 15:05 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Add wikis for deployment on 20150609 gerrit:216622 (duration: 01m 09s)
• 11:09 Krenair: Email set for User:GifTagger@commonswiki per phab:T100889
• 09:05 akosiaris: uploaded etherpad-lite_1.5.6-2 on apt.wikimedia.org/jessie-wikimedia/main component
• 08:22 akosiaris: upload etherpad-lite_1.5.6-1 on apt.wikimedia.org, jessie-wikimedia dist, main component
• 04:35 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jun 9 04:34:08 UTC 2015 (duration 34m 7s)
• 02:28 logmsgbot: LocalisationUpdate completed (1.26wmf8) at 2015-06-09 02:27:30+00:00
• 02:23 logmsgbot: l10nupdate Synchronized php-1.26wmf8/cache/l10n: (no message) (duration: 07m 12s)
• 01:42 godog: stop icinga-wm on neon

June 8

• 23:43 bblack: repooled cp3030/cp1065 in pybal
• 23:11 logmsgbot: ebernhardson Synchronized php-1.26wmf8/extensions/UploadWizard/: Bump UploadWizard in 1.26wmf8 for evening SWAT (duration: 01m 09s)
• 22:21 bblack: depooled cp3030, cp1065 in pybal for ipsec
• 20:17 subbu: deployed parsoid sha 131554ba
• 19:18 jynus: RAID degradation (disk failure) on s5 master (db1058), no production impact, replacement on the way
• 17:13 ottomata: restarted eventlogging services on eventlog1001 after disabling kafka pieces
• 16:13 _joe_: powercycling tmh1001, console blank, unresponsive to pings
• 16:00 logmsgbot: thcipriani Synchronized commonsuploads.dblist: SWAT: Revert Temporarily re-enable uploads on Marathi Wikipedia, for real gerrit:216719 (duration: 01m 07s)
• 15:58 logmsgbot: thcipriani Synchronized commonsuploads.dblist: SWAT: Revert Temporarily re-enable uploads on Marathi Wikipedia gerrit:216719 (duration: 01m 08s)
• 15:40 logmsgbot: thcipriani Synchronized php-1.26wmf8/extensions/Cite: SWAT: Revert Do all of Cite's real work during unstrip and followup gerrit:216715 (duration: 01m 08s)
• 15:19 Coren: T96063: process halted for now as store/backup is unmovable and on slice5
• 15:17 logmsgbot: thcipriani Synchronized w/static/images/project-logos/pflwiki.png: SWAT: Fix transparency of pflwiki logo gerrit:216595 (duration: 01m 08s)
• 15:15 akosiaris: disabled ircecho on neon for a while
• 14:53 Coren: T96063: starting pvmove from slice5 to slice2
• 14:48 Coren: T96063: dropped volume slice1 from vg store
• 14:46 Coren: T96063: dropped store/project
• 14:44 Coren: starting https://phabricator.wikimedia.org/T96063 on labstore1001
• 14:24 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool es1005 (duration: 01m 08s)
• 14:23 Coren: rsync in progress between labstore1001:store/backup and labstore1002:backup/backup (at ionice idle)
• 14:13 Coren: created store/backup snapshot on labstore1001 for backup copy
• 13:03 moritzm: added strongswan_5.3.0-1+wmf2 to jessie-wikimedia on carbon
• 11:42 _joe_: purging squid cache on carbon
• 11:26 moritzm: updated mc2* to 2:2.8.17-1+deb8u1
• 10:55 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool es1007 (duration: 01m 08s)
• 10:27 akosiaris: disabled puppet on uranium, investigating ganglia problems
• 10:05 akosiaris: ganglia gmetad problems
• 05:25 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jun 8 05:24:08 UTC 2015 (duration 24m 7s)
• 02:26 logmsgbot: LocalisationUpdate completed (1.26wmf8) at 2015-06-08 02:25:12+00:00
• 02:21 logmsgbot: l10nupdate Synchronized php-1.26wmf8/cache/l10n: (no message) (duration: 07m 07s)

June 7

• 23:27 godog: reboot ms-be2008 sdg failed, xfs unhappy
• 07:03 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1073, warm up (duration: 01m 09s)
• 05:16 andrewbogott: we did a whole lot of things to labstore1001 while morebots was away
• 05:14 andrewbogott: service nfs-kernel-server restart on labstore1001
• 02:26 logmsgbot: LocalisationUpdate completed (1.26wmf8) at 2015-06-07 02:25:13+00:00
• 02:21 logmsgbot: l10nupdate Synchronized php-1.26wmf8/cache/l10n: (no message) (duration: 07m 09s)

June 6

• 23:46 subbu: deployed parsoid 5172a446 (cherry-pick of 719c736f) -- hotfix for T101599
• 05:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jun 6 05:47:40 UTC 2015 (duration 47m 39s)
• 02:31 logmsgbot: LocalisationUpdate completed (1.26wmf8) at 2015-06-06 02:30:24+00:00
• 02:26 logmsgbot: l10nupdate Synchronized php-1.26wmf8/cache/l10n: (no message) (duration: 07m 10s)

June 5

• 22:42 godog: powercycle graphite2001, no console no ssh
• 22:06 andrewbogott: restarted apache on virt1000
• 20:49 ori: Upgrading hhvm-fss on application servers to 1.1.7; expect brief 5xx spike.
• 20:14 logmsgbot: demon Synchronized php-1.26wmf8: live hack (duration: 02m 32s)
• 20:10 mutante: apt-get upgrade on terbium
• 19:52 godog: bounce redis on rdb1001/rdb1003 to pick up new slave limits
• 19:51 mutante: chown root:root / on terbium
• 19:50 godog: bounce redis on rdb1002/rdb1004 to pick up new slave limits
• 19:29 godog: bounce redis again on rdb1003 after increasing the slave limits more
• 19:17 godog: bounce redis on rdb1003 after bumping slave limits
• 19:07 godog: redis master logs shows periodic 'cmd=sync scheduled to be closed ASAP for overcoming of output buffer limits.' indicating the slave fails to sync
• 18:40 godog: spike in redis network starting at ~15.00 UTC, correlates with ocg failures
• 18:01 moritzm: restarted gerrit on ytterbium for java update
• 14:43 jynus: short lag period on db1049, traffic automatically redirected to other slave and back to normal
• 14:07 moritzm: added ubuntu-meta-1.325+wmf1 for trusty-wikimedia to apt.wikimedia.org (T100004)
• 14:07 moritzm: added ubuntu-meta-1.267.1+wmf1 for precise-wikimedia to apt.wikimedia.org (T100004)
• 12:44 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Depool es1007 (duration: 01m 08s)
• 12:08 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Repool es1009 (duration: 01m 08s)
• 11:30 _joe_: uploaded new HHVM package, installing on mw1025 for testing
• 09:17 moritzm: added redis_2.6.13-1+wmf1 to precise-wikimedia on apt.wikimedia.org
• 06:24 moritzm: added redis_2.8.4-2+wmf1 to trusty-wikimedia on apt.wikimedia.org
• 05:23 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jun 5 05:22:50 UTC 2015 (duration 22m 49s)
• 04:10 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1073 (duration: 01m 08s)
• 02:26 logmsgbot: LocalisationUpdate completed (1.26wmf8) at 2015-06-05 02:25:20+00:00
• 02:21 logmsgbot: l10nupdate Synchronized php-1.26wmf8/cache/l10n: (no message) (duration: 07m 09s)
• 01:27 tgr: deploying schema changes for Gather on enwiki, enwikivoyage, hewiki (T98490, T101460)
• 00:08 logmsgbot: catrope Synchronized php-1.26wmf8/vendor/oojs/oojs-ui/php/Tag.php: Fix OOUI fatals (T99210) (duration: 00m 13s)

June 4

• 23:40 logmsgbot: catrope Synchronized php-1.26wmf8/extensions/MobileFrontend: SWAT (duration: 00m 13s)
• 23:28 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings.php: Disable VE A/B test for new accounts on enwiki (duration: 00m 13s)
• 22:39 ejegg: updated payments from d22e44e3fab2b937707c2776384cb93a49b4cfd3 to 43c7952d2a31deaea97e8319f5612d644dce43c8
• 22:21 ottomata: doing controlled restart of kafka brokers services to apply auto create topic config
• 21:48 jgage: analyics1013 crashed, rebooted
• 21:42 logmsgbot: ori Synchronized php-1.26wmf8/includes/libs/ReplacementArray.php: 1b20d62c26: Revert "awful hack: disable fss on zhwiki only, except on mw1017" (duration: 00m 13s)
• 21:34 ori: performing rolling restart of HHVMs for hhvm-fss upgrade
• 21:27 bd808: restarted logstash and elasticsearch on logstash100[1-3] to pick up latest jre updates
• 18:48 mutante: restarted apache on silver/wikitech
• 18:20 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Depool es1009 and master-slave switchover (duration: 00m 13s)
• 18:01 awight: Enabling PayPal audit parser job
• 17:57 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Repool es1008 (duration: 00m 15s)
• 17:44 logmsgbot: jynus Synchronized wmf-config/db-codfw.php: Repool es2008 and its slaves (duration: 00m 13s)
• 17:21 ori: Disabling Puppet and nutcracker on mw1017 to control for parser cache
• 17:18 logmsgbot: jynus Synchronized wmf-config/db-codfw.php: Depool es2008 and its slaves (duration: 00m 13s)
• 17:17 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Depool es1008 (duration: 00m 12s)
• 16:33 logmsgbot: kartik Finished scap: Update ContentTranslation (duration: 09m 17s)
• 16:23 logmsgbot: kartik Started scap: Update ContentTranslation
• 15:54 moritzm: added redis_2.8.4-2+wmf1 to trusty-wikimedia on apt.wikimedia.org
• 15:48 logmsgbot: anomie Synchronized php-1.26wmf8/includes/jobqueue/: SWAT: jobqueue: Record stats on how long it takes before a job is run gerrit:215748 (duration: 00m 14s)
• 15:38 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ApiFeatureUsage everywhere gerrit:215901 (duration: 00m 19s)
• 15:36 logmsgbot: anomie Synchronized wmf-config/CommonSettings.php: SWAT: Remove obsolete 'ValidateExtendedMetadataCache' hook gerrit:215900 (duration: 00m 12s)
• 15:35 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Added staff-recommender campaign gerrit:215865 (duration: 00m 12s)
• 15:30 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Add wikis for deployment on 20150406 gerrit:215281 (duration: 00m 12s)
• 15:12 logmsgbot: ori Synchronized php-1.26wmf8/includes/libs/ReplacementArray.php: Ia5f3dc84605: awful hack: disable fss on zhwiki only, except on mw1017 (duration: 00m 17s)
• 15:09 _joe_: puppet disabled, fss disabled on mw1017
• 14:42 YuviPanda: running sudo sed -i 's/GlobalSign_CA.pem/ca-certificates.crt/' /etc/ldap/ldap.conf on all labs nodes
• 14:36 awight: Disable PayPal audit parsing job
• 12:19 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1072, warm up (duration: 00m 13s)
• 05:12 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun 4 05:11:32 UTC 2015 (duration 11m 31s)
• 02:30 logmsgbot: LocalisationUpdate completed (1.26wmf8) at 2015-06-04 02:28:54+00:00
• 02:25 logmsgbot: l10nupdate Synchronized php-1.26wmf8/cache/l10n: (no message) (duration: 07m 22s)

June 3

• 23:42 logmsgbot: kaldari Synchronized wmf-config/InitialiseSettings.php: syncing ImportSource change for meta (duration: 00m 13s)
• 23:34 logmsgbot: kaldari Synchronized wmf-config/InitialiseSettings.php: syncing config change for mediawiki logo on mobile, take 2 (duration: 00m 12s)
• 23:26 logmsgbot: kaldari Synchronized wmf-config/InitialiseSettings.php: syncing config change for mediawiki logo on mobile (duration: 00m 12s)
• 23:25 logmsgbot: kaldari Synchronized images/mobile/mediawiki.png: syncing mediawiki logo for mobile (duration: 00m 12s)
• 22:02 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable Wikibase usage tracking on ukwiki and viwiki (duration: 00m 15s)
• 21:58 mutante: restarted gitblit
• 21:53 logmsgbot: ori Synchronized php-1.26wmf8/includes/resourceloader/ResourceLoader.php: 7f49853fc9: ResourceLoader::filter: use APC when running under HHVM (did not sync correct file previously) (duration: 00m 12s)
• 21:20 andrewbogott: restarting pdns on virt1000 and labcontrol1001
• 21:05 Jamesofur: decryption key for Board Election insert into voteWiki
• 20:58 bblack: repooling ns0 -> radon AuthDNS
• 20:55 bblack: depooling ns0 -> radon AuthDNS (rebooting for kernel update)
• 20:50 hashar: restarted zuul entirely to remove some stalled jobs
• 20:29 paravoid: kafka preferred-replica-election on an1021
• 20:28 hashar: Restarting Jenkins to release a deadlock
• 20:23 logmsgbot: ori Synchronized php-1.26wmf8/resources/Resources.php: 7f49853fc9: ResourceLoader::filter: use APC when running under HHVM (duration: 00m 13s)
• 20:19 subbu: deployed parsoid sha ab675400
• 19:08 bblack: changed ops/puppet repo to ff-only in gerrit config, feel free to scream/revert if necc!
• 18:46 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: All wikis to 1.26wmf8, no new branch until next Tuesday, June 9th
• 18:42 logmsgbot: twentyafterfour Finished scap: Delete stale branch symlinks (1.26wmf1,1.26wmf2) (duration: 07m 14s)
• 18:35 logmsgbot: twentyafterfour Started scap: Delete stale branch symlinks (1.26wmf1,1.26wmf2)
• 15:16 logmsgbot: legoktm Synchronized wmf-config/InitialiseSettings.php: Remove references to $wgEchoCohortInterval (duration: 00m 12s)
• 15:16 logmsgbot: legoktm Synchronized wmf-config/CommonSettings.php: Change default extension distributor branch to REL1_25 (duration: 00m 15s)
• 15:15 bblack: repooling ns1->baham DNS traffic
• 15:07 bblack: depooling ns1->baham DNS traffic for kernel update
• 15:00 moritzm: added linux 3.19.3-5 for jessie-wikimedia on apt.wikimedia.org
• 14:46 bblack: restarted hhvm on mw1195, seems to be a case of https://phabricator.wikimedia.org/T89912
• 14:32 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable Wikibase usage tracking on huwiki (duration: 00m 12s)
• 14:29 logmsgbot: jynus Synchronized wmf-config/db-codfw.php: Repool es2008, es2009 and es2010 (duration: 00m 14s)
• 14:10 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable Wikibase usage tracking on eswiki (duration: 00m 13s)
• 13:38 logmsgbot: jynus Synchronized wmf-config/db-codfw.php: Depool es2008, es2009 and es2010 (duration: 00m 14s)
• 13:12 paravoid: reimaging rubidium with trusty, as spare
• 13:02 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable Wikibase usage tracking on arwiki and cawiki (duration: 00m 15s)
• 12:56 paravoid: permanently switching ns0 to radon instead of rubidium
• 12:53 logmsgbot: jynus Synchronized wmf-config/db-codfw.php: Repool es2009 (duration: 00m 15s)
• 11:04 paravoid: kafka preferred-replica-election on an1021
• 10:55 logmsgbot: jynus Synchronized wmf-config/db-codfw.php: Depool es2009 (duration: 00m 13s)
• 10:43 paravoid: powercycling ms-be1005
• 10:28 logmsgbot: jynus Synchronized wmf-config/db-codfw.php: repool es2010 (duration: 00m 14s)
• 10:24 moritzm: added linux-meta 1.2 for jessie-wikimedia on carbon.wikimedia.org
• 10:09 hashar: Jenkins: refreshing all jobs to get rid of an obsolete http notification to Zuul bug T93321
• 09:48 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool es1008 (duration: 00m 13s)
• 09:00 logmsgbot: jynus Synchronized wmf-config/db-codfw.php: depool es2010 (duration: 00m 13s)
• 08:51 moritzm: removed fuse/ntfs-3g from wtp*
• 07:47 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool es1008 (duration: 00m 14s)
• 05:42 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jun 3 05:41:31 UTC 2015 (duration 41m 30s)
• 02:50 logmsgbot: LocalisationUpdate completed (1.26wmf8) at 2015-06-03 02:48:55+00:00
• 02:45 logmsgbot: l10nupdate Synchronized php-1.26wmf8/cache/l10n: (no message) (duration: 06m 37s)
• 02:28 logmsgbot: LocalisationUpdate completed (1.26wmf7) at 2015-06-03 02:27:38+00:00
• 02:25 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1072 (duration: 00m 12s)
• 02:23 logmsgbot: l10nupdate Synchronized php-1.26wmf7/cache/l10n: (no message) (duration: 07m 13s)
• 01:57 springle: replicate m3 to codfw dbstore2001
• 01:37 springle: start sync m4 eventlogging to codfw dbstore2002
• 00:35 logmsgbot: mattflaschen Synchronized php-1.26wmf8/extensions/Calendar/: Sync Calendar 1.26wmf8 for module position (duration: 00m 12s)
• 00:20 logmsgbot: mattflaschen Synchronized php-1.26wmf8/includes/User.php: Fixed $flags bit operation precedence fail in User::loadFromDatabase() (duration: 00m 14s)

June 2

• 23:56 logmsgbot: mattflaschen Synchronized php-1.26wmf8/extensions/Flow/: Sync Flow 1.26wmf8 for import fix (duration: 00m 15s)
• 23:43 logmsgbot: mattflaschen Synchronized wmf-config/InitialiseSettings.php: Disable WikiGrok (duration: 00m 13s)
• 23:33 logmsgbot: mattflaschen Synchronized php-1.26wmf8/includes/resourceloader/ResourceLoaderStartUpModule.php: Don't cache minification of user.tokens (duration: 00m 15s)
• 23:33 logmsgbot: mattflaschen Synchronized php-1.26wmf8/includes/resourceloader/ResourceLoader.php: Don't cache minification of user.tokens (duration: 00m 13s)
• 23:33 logmsgbot: mattflaschen Synchronized php-1.26wmf8/includes/OutputPage.php: Don't cache minification of user.tokens (duration: 00m 14s)
• 23:31 logmsgbot: mattflaschen Synchronized php-1.26wmf7/includes/resourceloader/ResourceLoaderStartUpModule.php: Don't cache minification of user.tokens (duration: 00m 13s)
• 23:31 logmsgbot: mattflaschen Synchronized php-1.26wmf7/includes/resourceloader/ResourceLoader.php: Don't cache minification of user.tokens (duration: 00m 14s)
• 23:31 logmsgbot: mattflaschen Synchronized php-1.26wmf7/includes/OutputPage.php: Don't cache minification of user.tokens (duration: 00m 13s)
• 21:44 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I263aa9542: Set $wgExtDistUseEventLogging = true; (duration: 00m 13s)
• 21:43 logmsgbot: ori Synchronized php-1.26wmf8/extensions/ExtensionDistributor: cdd033e7d8: Update ExtensionDistributor for cherry-picks (duration: 00m 13s)
• 19:24 logmsgbot: ori Synchronized wmf-config/StartProfiler.php: I7810b72d5: Sample profiling data at 1:10,000 (duration: 00m 12s)
• 19:19 logmsgbot: ori Synchronized wmf-config: I35255f357 and I026dfdbf68 (duration: 00m 12s)
• 19:15 logmsgbot: aude Synchronized wmf-config/Wikibase.php: bump cache epoch for wikidata (duration: 00m 13s)
• 19:06 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: wgMaxCredits to 0 (duration: 00m 13s)
• 18:53 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: Group1 wikis to 1.26wmf8
• 18:46 robh: sodium has resumed normal service. all items on https://phabricator.wikimedia.org/T100711 addressed
• 17:56 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool es1010 (duration: 00m 12s)
• 17:18 robh: mailing list traffic halted for list renames
• 17:07 robh: lists.wikimedia.org is now sha256 cert
• 17:04 robh: starting the lists.wikimedia.org certificate update, archives will offline during this process
• 15:44 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool es1010 (duration: 00m 13s)
• 15:03 logmsgbot: thcipriani Synchronized wmf-config/wikitech.php: SWAT: No longer set use_dnsmasq for new instances. gerrit:215317 (duration: 00m 12s)
• 12:31 twentyafterfour: merged https://gerrit.wikimedia.org/r/#/c/214288/ and deployed scap
• 12:18 moritzm: installed linux-tools-3.19.8-1 for jessie-wikimedia on carbon
• 07:36 logmsgbot: nikerabbit Synchronized wmf-config/InitialiseSettings.php: Fixed wiki id for fiu_vro for CX beta feature (duration: 00m 13s)
• 05:41 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jun 2 05:39:57 UTC 2015 (duration 39m 56s)
• 02:49 logmsgbot: LocalisationUpdate completed (1.26wmf8) at 2015-06-02 02:48:23+00:00
• 02:44 logmsgbot: l10nupdate Synchronized php-1.26wmf8/cache/l10n: (no message) (duration: 05m 45s)
• 02:28 logmsgbot: LocalisationUpdate completed (1.26wmf7) at 2015-06-02 02:27:42+00:00
• 02:23 logmsgbot: l10nupdate Synchronized php-1.26wmf7/cache/l10n: (no message) (duration: 06m 26s)
• 02:06 logmsgbot: krinkle Synchronized php-1.26wmf7/resources/src/mediawiki/mediawiki.js: backport rl-fix I717b86573 (duration: 00m 14s)
• 00:33 ejegg: updated payments-wiki from a4fef65ec1dd3db1fb1d7ceb797b2c7485c722d2 to d22e44e3fab2b937707c2776384cb93a49b4cfd3
• 00:07 ori: Updated jobrunner for I1d351d8d1: Made periodictasks stats calls more useful
• 00:02 logmsgbot: ori Synchronized php-1.26wmf8/extensions/RSS/RSSParser.php: Ice44740fb: Don't rely on strip marker uniqueness (T10104) (duration: 00m 14s)
• 00:01 logmsgbot: ori Synchronized php-1.26wmf7/extensions/RSS/RSSParser.php: Ice44740fb: Don't rely on strip marker uniqueness (T10104) (duration: 00m 13s)

June 1

• 23:36 mutante: restarted gitblit ..
• 23:15 ori: Deployed jobchron / jobrunner change Icab05090b and restarted jobchron / jobrunner on job queue runners.
• 22:51 ejegg: updated payments from 60c160110a20cf763b82677ff1501e9ce0c919bc to a4fef65ec1dd3db1fb1d7ceb797b2c7485c722d2
• 21:36 godog: doing some local testing on carbon for T100636 fwiw, thus puppet disabled
• 21:35 ejegg: update paymentswiki from aa66797553fbcfb63f7cf29abccc44d060b65db0 to 60c160110a20cf763b82677ff1501e9ce0c919bc
• 21:13 logmsgbot: ori Synchronized php-1.26wmf7/languages/LanguageConverter.php: 1d054ce6d3: Use a fixed marker prefix string in the Parser and MWTidy (duration: 00m 14s)
• 20:40 logmsgbot: ori Synchronized php-1.26wmf8/languages/LanguageConverter.php: 1d054ce6d3: Use a fixed marker prefix string in the Parser and MWTidy (duration: 00m 13s)
• 20:29 twentyafterfour: disabled several no-longer-existent repositories in phabricator which apparently have been deleted in gerrit
• 20:26 subbu: deployed parsoid sha 73445bfd
• 20:05 twentyafterfour: restarted apache2 and phd on iridium (phabricator)
• 19:52 MaxSem: Repopulated gis.spatial_ref_sys on labsdb1004 with postgis 2.1 data, old contents backed up as spatial_ref_sys_bak
• 18:55 logmsgbot: ori Synchronized php-1.26wmf7/extensions/SemanticForms/includes/SF_FormUtils.php: I7ed3996a1: Stop using StripState (duration: 00m 13s)
• 18:55 logmsgbot: ori Synchronized php-1.26wmf8/extensions/SemanticForms/includes/SF_FormUtils.php: I7ed3996a1: Stop using StripState (duration: 00m 15s)
• 17:46 yurik: deployed graphoid service update - grafana logging cleanup
• 16:40 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool pc1003 (duration: 00m 15s)
• 16:06 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: T99491, T100925: Sysops to add users to import group on maiwiki, newiki (duration: 00m 14s)
• 15:47 logmsgbot: thcipriani Synchronized php-1.26wmf8/extensions/CodeReview: SWAT: Backport CodeReview module position fix gerrit:215043 (duration: 00m 13s)
• 15:24 logmsgbot: thcipriani Synchronized php-1.26wmf8/includes/resourceloader/ResourceLoaderWikiModule.php: SWAT: Make ResourceLoaderWikiModule support custom position gerrit:214741 (duration: 00m 15s)
• 15:23 logmsgbot: thcipriani Synchronized php-1.26wmf8/extensions/WikiEditor: SWAT: Make ResourceLoaderWikiModule support custom position gerrit:214741 (duration: 00m 13s)
• 15:22 logmsgbot: thcipriani Synchronized php-1.26wmf8/extensions/VectorBeta: SWAT: Make ResourceLoaderWikiModule support custom position gerrit:214741 (duration: 00m 15s)
• 15:21 logmsgbot: thcipriani Synchronized php-1.26wmf8/extensions/SyntaxHighlight_GeSHi: SWAT: Make ResourceLoaderWikiModule support custom position gerrit:214741 (duration: 00m 14s)
• 15:20 logmsgbot: thcipriani Synchronized php-1.26wmf8/extensions/MobileFrontend: SWAT: Make ResourceLoaderWikiModule support custom position gerrit:214741 (duration: 00m 13s)
• 15:18 logmsgbot: thcipriani Synchronized php-1.26wmf8/extensions/Gather: SWAT: Make ResourceLoaderWikiModule support custom position gerrit:214741 (duration: 00m 13s)
• 14:42 cmjohnson1: powering down analytics1028 to swap the bad DIMM
• 14:38 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool pc1003 (duration: 00m 12s)
• 13:48 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable arbitrary access on wikisource and itwiki, and make other projects sidebar feature default for ptwiki (for real) (duration: 00m 12s)
• 13:45 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable arbitrary access on wikisource and itwiki, and make other projects sidebar feature default for ptwiki (duration: 00m 15s)
• 13:31 logmsgbot: aude Synchronized php-1.26wmf8/extensions/Wikidata: css compatibility fixes for wmf8 (duration: 00m 24s)
• 13:00 logmsgbot: krenair Synchronized php-1.26wmf8/extensions/WikimediaMessages/WikimediaMessages.hooks.php: https://gerrit.wikimedia.org/r/#/c/215011/ - fix EditPageCopyrightWarning (duration: 00m 16s)
• 12:22 moritzm: added firmware-nonfree 0.44~wmf1 for jessie-wikimedia on carbon
• 09:32 yurik: deployed latest graphoid service to sca100x
• 08:18 hashar: Jenkins: upgrading git plugin from 1.5.0 to latest
• 08:12 mobrovac: restbase restart cassandra on restbase1006
• 08:09 mobrovac: restbase restart cassandra on restbase1005
• 08:07 mobrovac: restbase restart cassandra on restbase1004
• 08:05 mobrovac: restbase restart cassandra on restbase1003
• 08:00 mobrovac: restbase restart cassandra on restbase1002
• 07:59 mobrovac: restbase restart cassandra on restbase1001
• 05:19 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jun 1 05:18:18 UTC 2015 (duration 18m 17s)
• 02:47 logmsgbot: LocalisationUpdate completed (1.26wmf8) at 2015-06-01 02:46:32+00:00
• 02:43 logmsgbot: l10nupdate Synchronized php-1.26wmf8/cache/l10n: (no message) (duration: 05m 37s)
• 02:27 logmsgbot: LocalisationUpdate completed (1.26wmf7) at 2015-06-01 02:26:03+00:00
• 02:22 logmsgbot: l10nupdate Synchronized php-1.26wmf7/cache/l10n: (no message) (duration: 06m 35s)

May 31

• 22:35 jgage: graphite2001 keeps falling off the net due to OOM; swap 100% in use. dist-upgraded & rebooted. dmesg in ~gage/dmesg.2015-05-31
• 18:37 logmsgbot: krinkle Synchronized php-1.26wmf8/resources/src/mediawiki/mediawiki.js: rl live fix - I717b86573 (duration: 00m 12s)
• 17:36 Krinkle: Confirmed RL problem solved. The jquery|mediawiki&version=bizqqnC request was cached with an old mw.loader implementation somehow. After the touch and sync, the version is now dQAzAsdU and the implementation is up to date.
• 17:33 logmsgbot: krinkle Synchronized php-1.26wmf7/resources: touch mediawiki.js (duration: 00m 13s)
• 17:20 Krinkle: Investigating RL issues (clients are loading mediawiki.notification&version=19700101T000000Z, mw.loader.moduleRegistry contains NaN for versions)
• 17:12 gwicke: performed a rolling restart of RESTBase Cassandra nodes to address elevated request error rates apparently related to schema disagreement
• 05:35 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun May 31 05:34:36 UTC 2015 (duration 34m 35s)
• 02:47 logmsgbot: LocalisationUpdate completed (1.26wmf8) at 2015-05-31 02:46:41+00:00
• 02:43 logmsgbot: l10nupdate Synchronized php-1.26wmf8/cache/l10n: (no message) (duration: 05m 51s)
• 02:26 logmsgbot: LocalisationUpdate completed (1.26wmf7) at 2015-05-31 02:25:44+00:00
• 02:21 logmsgbot: l10nupdate Synchronized php-1.26wmf7/cache/l10n: (no message) (duration: 06m 41s)

May 30

• 21:07 bd808: Upgraded Elasticsearch cluster to 1.3.9 on logstash100[1-6]
• 18:35 logmsgbot: hoo Synchronized php-1.26wmf7/extensions/UploadWizard/: Touch js… (duration: 00m 18s)
• 17:06 logmsgbot: legoktm Synchronized php-1.26wmf8/extensions/WikiEditor/extension.json: Explicitly define module position (duration: 00m 13s)
• 05:32 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May 30 05:31:02 UTC 2015 (duration 31m 1s)
• 02:56 logmsgbot: LocalisationUpdate completed (1.26wmf8) at 2015-05-30 02:55:22+00:00
• 02:52 logmsgbot: l10nupdate Synchronized php-1.26wmf8/cache/l10n: (no message) (duration: 05m 40s)
• 02:36 logmsgbot: LocalisationUpdate completed (1.26wmf7) at 2015-05-30 02:34:55+00:00
• 02:30 logmsgbot: l10nupdate Synchronized php-1.26wmf7/cache/l10n: (no message) (duration: 06m 50s)
• 01:15 ori: Deployed rcstream I797bc1244: Handle invalid JSON gracefully
• 00:08 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/212436/ - docs only, no code change (how was this waiting 10 days?) (duration: 00m 14s)

May 29

• 23:56 logmsgbot: ori Synchronized w/static/images/project-logos: Ic62747f37: Optimise project logos added since I8c9a6a56 (duration: 00m 13s)
• 21:21 logmsgbot: ori Synchronized wmf-config/throttle.php: Ife45684c5: Add another IP address for Santiago edit-a-thon (duration: 00m 13s)
• 20:43 logmsgbot: ori Synchronized robots.txt: I7b321b62d: allow robots to use RL on domains (duration: 00m 14s)
• 17:18 mutante: fix client_max_body_size syntax error in nginx config of payments1001
• 15:19 logmsgbot: anomie Synchronized php-1.26wmf8/extensions/ConfirmEdit/: Update ConfirmEdit to fix API breakage gerrit:214620 (duration: 00m 14s)
• 14:52 paravoid: re-redirecting ns0 traffic back to rubidium
• 14:17 jynus: Moving pdns and designate databases from m1 to m5
• 13:30 logmsgbot: aude Synchronized php-1.26wmf8/extensions/Wikidata: touch js and css files to try to fix issues on test.wikidata (duration: 00m 26s)
• 13:17 godog: roll-restart cassandra on cerium / xenon / praseodymium following java upgrade
• 11:53 paravoid: reimaging rubidium
• 11:45 _joe_: restart nutcracker on mw1150
• 11:41 paravoid: redirecting ns0 traffic to baham (= ns1) in preparation for rubidium upgrade
• 06:52 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 29 06:51:45 UTC 2015 (duration 51m 44s)
• 06:13 logmsgbot: ori Synchronized php-1.26wmf7/includes/deferred/SiteStatsUpdate.php: Icc12c07ab: Update context stats in SiteStatsUpdate (duration: 00m 13s)
• 06:12 logmsgbot: ori Synchronized php-1.26wmf8/includes/deferred/SiteStatsUpdate.php: Icc12c07ab: Update context stats in SiteStatsUpdate (duration: 00m 14s)
• 06:03 apergos: salt keys regenerated on all production hosts (minions, not master key)
• 03:09 logmsgbot: LocalisationUpdate completed (1.26wmf8) at 2015-05-29 03:08:15+00:00
• 03:02 logmsgbot: l10nupdate Synchronized php-1.26wmf8/cache/l10n: (no message) (duration: 10m 08s)
• 02:36 logmsgbot: LocalisationUpdate completed (1.26wmf7) at 2015-05-29 02:35:10+00:00
• 02:31 logmsgbot: l10nupdate Synchronized php-1.26wmf7/cache/l10n: (no message) (duration: 06m 54s)
• 00:07 logmsgbot: ori Synchronized php-1.26wmf7/includes/diff/UnifiedDiffFormatter.php: d95cac90c7: Make the output of UnifiedDiffFormatter match diff -u (duration: 00m 14s)
• 00:06 logmsgbot: ori Synchronized php-1.26wmf7/extensions/Echo/includes/DiffParser.php: 41d27c4a26: Update Echo for cherry-picks (duration: 00m 13s)

May 28

• 23:33 jgage: restarted nutcracker on mw1056 due to errors, per bd808
• 23:18 logmsgbot: catrope Synchronized php-1.26wmf7/includes/EditPage.php: Fix regression with URL-specified edit tags (duration: 00m 13s)
• 23:18 logmsgbot: catrope Synchronized php-1.26wmf6/includes/EditPage.php: Fix regression with URL-specified edit tags (duration: 00m 13s)
• 23:04 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings.php: Enable A/B test of VE for new accounts on enwiki (duration: 00m 13s)
• 22:48 logmsgbot: hoo Synchronized php-1.26wmf7/: Touching some JS, re-syncing resource definitions to rule out causes for Wikidata JS problem. (duration: 01m 00s)
• 21:52 logmsgbot: ori Synchronized php-1.26wmf7/resources/src/mediawiki/mediawiki.toc.js: Touching file on unconfirmed suspicion of stale cache (duration: 00m 16s)
• 21:51 logmsgbot: ori Synchronized php-1.26wmf8/resources/src/mediawiki/mediawiki.toc.js: Touching file on unconfirmed suspicion of stale cache (duration: 00m 15s)
• 20:24 mutante: killed nodejs on wtp1023,wtp1016
• 20:11 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable Wikibase usage tracking on Wikivoyage (duration: 00m 13s)
• 20:03 cscott: updated Parsoid to version 497da30e ; canary restart of wtp1001; observed network TX spike (possibly UDP, possibly logging); reverted to 8ed6fd0b and restarted all parsoids.
• 19:33 mutante: temp. stopped icinga-wm
• 19:05 logmsgbot: legoktm Synchronized php-1.26wmf8/extensions/Gadgets/: Explicitly define module position (duration: 00m 14s)
• 18:32 logmsgbot: legoktm Synchronized php-1.26wmf7/extensions/GlobalCssJs/: Explicitly define module position (duration: 00m 12s)
• 18:24 logmsgbot: legoktm Synchronized php-1.26wmf8/extensions/GlobalCssJs/: Explicitly define module position (duration: 00m 13s)
• 18:22 logmsgbot: krenair Synchronized php-1.26wmf6/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/214397/ - in case we have to go back to wmf6 again for whatever reason (duration: 00m 15s)
• 18:20 logmsgbot: krenair Synchronized php-1.26wmf8/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/214396/ (duration: 00m 13s)
• 18:17 logmsgbot: krenair Synchronized php-1.26wmf7/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/214395/ (duration: 00m 14s)
• 17:29 logmsgbot: twentyafterfour Finished scap: Group0 to 1.26wmf8, everything else to 1.26wmf7 (duration: 28m 16s)
• 17:01 logmsgbot: twentyafterfour Started scap: Group0 to 1.26wmf8, everything else to 1.26wmf7
• 16:59 paravoid: reimaging baham
• 16:52 paravoid: redirecting ns1 traffic to rubidium (= ns0) in preparation for baham upgrade
• 15:54 logmsgbot: kartik Finished scap: Update ContentTranslation (duration: 03m 19s)
• 15:50 logmsgbot: kartik Started scap: Update ContentTranslation
• 15:47 logmsgbot: thcipriani Synchronized wmf-config/abusefilter.php: SWAT: Modify AbuseFilter block configuration on eswikibooks gerrit:206510 (duration: 00m 15s)
• 15:40 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Prevent indexing of User: namespace on ukwiki gerrit:210680 (duration: 00m 14s)
• 15:35 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable NewUserMessage on sa.wikipedia gerrit:212724 (duration: 00m 13s)
• 15:28 godog: set operations/debs/python-statsd as hidden in gerrit -- deprecated
• 15:24 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Extension:NewUserMessage on ta.wikipedia gerrit:213841 (duration: 00m 12s)
• 15:13 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable SandboxLink for cswiki gerrit:214247 (duration: 00m 15s)
• 15:11 godog: set operations/debs/txstatsd as hidden in gerrit -- deprecated
• 15:05 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Add wikis for CX deployment on 20150528 gerrit:213992 (duration: 00m 15s)
• 15:00 bblack: merged up https://gerrit.wikimedia.org/r/214345 - look here if IPv6 problems!
• 14:37 cmjohnson1: powering down dataset1001 to add disk array
• 14:17 bblack: deploying https://gerrit.wikimedia.org/r/214341 - keep in mind if ipv6-related issues arise!
• 13:50 akosiaris: started ircecho (icinga-wm) on neon
• 13:46 hashar: upgrading Jenkins git plugin from 1.4.6+wmf1 to 1.7.1 bug T100655 and restarting Jenkins
• 13:25 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool pc1003 (not to confuse with db1003) after warmup (duration: 00m 15s)
• 13:11 akosiaris: killed ircecho service on neon
• 09:48 _joe_: depooling the HHVM appserver. 503s reduced slightly but still non-irrelevant
• 09:37 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: Depool pc1003 (duration: 00m 15s)
• 09:35 _joe_: pooling mw1152 into the imagescalers pool after fixes made in Lyon
• 06:11 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 28 06:09:56 UTC 2015 (duration 9m 55s)
• 04:22 springle: reload dbstore1002 s7
• 02:41 logmsgbot: LocalisationUpdate completed (1.26wmf6) at 2015-05-28 02:40:00+00:00
• 02:36 logmsgbot: l10nupdate Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 06m 46s)
• 02:20 springle: set global read_only=0 on pc1001 pc1002. this config broke in the recent upgrade
• 00:59 logmsgbot: legoktm Synchronized php-1.26wmf8/resources/: Revert "Convert mediawiki.toc and mediawiki.user to using mw.cookie" (duration: 00m 17s)
• 00:58 logmsgbot: legoktm Synchronized php-1.26wmf7/resources/: Revert "Convert mediawiki.toc and mediawiki.user to using mw.cookie" (duration: 00m 13s)
• 00:07 logmsgbot: twentyafterfour Synchronized rpc/RunJobs.php: deploy I98b8a4ddbcdd58d1f2f23e4b1bf154f10b6b279e (duration: 00m 17s)

May 27

• 23:46 awight: updated payments from 858b87319daa3d66f62eb32e08cefc6b061748d1 to aa66797553fbcfb63f7cf29abccc44d060b65db0
• 23:31 logmsgbot: twentyafterfour Finished scap: scap, now with 10% less fail (duration: 22m 07s)
• 23:26 awight: payments rolled back to 858b87319daa3d66f62eb32e08cefc6b061748d1
• 23:24 awight: updated payments from 858b87319daa3d66f62eb32e08cefc6b061748d1 to aa66797553fbcfb63f7cf29abccc44d060b65db0
• 23:09 logmsgbot: twentyafterfour Started scap: scap, now with 10% less fail
• 22:57 logmsgbot: ori rebuilt wikiversions.cdb and synchronized wikiversions files: (no message)
• 21:49 mutante: restarted hhvm on mw1250,mw1254,mw1256
• 21:47 mutante: restarted hhvm on mw1017,mw1243,mw1244
• 21:42 bblack: restarting hhvm everywhere on 30s intervals between hosts
• 21:10 logmsgbot: twentyafterfour Synchronized php-1.26wmf8: Fix ConfirmEdit fatal Change-Id: I22353669a85391c3d9760a5253cac1263e895cf9 (duration: 01m 08s)
• 20:46 logmsgbot: twentyafterfour Purged l10n cache for 1.26wmf6
• 20:45 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.26wmf8
• 20:41 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.26wmf7
• 20:36 logmsgbot: twentyafterfour Finished scap: testwiki to php-1.26wmf8 and rebuild l10n cache (duration: 67m 53s)
• 19:40 akosiaris: removed operations/puppet/varnish from gerrit, git.wikimedia.org and github. The repo was used as a git submodule but the workflow turned out to be cumbersome approximately a year ago and was no longer updated. Up to a few minutes ago, it only served as a source of confusion. It no longer does.
• 19:28 logmsgbot: twentyafterfour Started scap: testwiki to php-1.26wmf8 and rebuild l10n cache
• 19:22 logmsgbot: twentyafterfour scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_1863397713" --threads=4 --lang en --quiet' returned non-zero exit status 255 (duration: 03m 38s)
• 19:18 logmsgbot: twentyafterfour Started scap: testwiki to php-1.26wmf8 and rebuild l10n cache
• 18:12 moritzm: Uploaded gridengine_6.2u5-4+wmf2 for precise-wikimedia to apt.wikimedia.org
• 17:55 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool pc1002 (duration: 00m 13s)
• 17:42 paravoid: rebooting asw-d2-eqiad
• 17:41 ottomata: initiating controlled shutdown of kafka broker analytics1018 in anticipation of switch reboot
• 15:33 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool pc1002 (duration: 00m 13s)
• 15:02 cmjohnson1: powering down cp1069 to relocate within the same rack
• 14:47 cmjohnson1: powering down cp1070 to relocate within the same rack
• 13:30 hashar: All Jenkins slaves are disconnected due to some ssh error. CI is down.
• 13:27 hashar: restarting Jenkins for java upgrade
• 13:13 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool pc1001 (duration: 00m 13s)
• 11:16 akosiaris: rebooting ganeti100{1..4} for bridge networking configuration
• 09:59 paravoid: powercycling ms-be1001; dead, console unresponsive
• 06:35 springle: clone dbstore2001 data to dbstore2002
• 05:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed May 27 05:47:25 UTC 2015 (duration 47m 24s)
• 02:53 logmsgbot: LocalisationUpdate completed (1.26wmf7) at 2015-05-27 02:52:25+00:00
• 02:48 logmsgbot: l10nupdate Synchronized php-1.26wmf7/cache/l10n: (no message) (duration: 06m 52s)
• 02:29 logmsgbot: LocalisationUpdate completed (1.26wmf6) at 2015-05-27 02:28:34+00:00
• 02:24 logmsgbot: l10nupdate Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 06m 45s)

May 26

• 18:21 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: Group1 wikis to 1.26wmf7
• 17:13 logmsgbot: krenair Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 15s)
• 17:10 logmsgbot: krenair Synchronized multiversion/MWMultiVersion.php: open cnwikimedia (duration: 00m 13s)
• 16:27 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 13s)
• 16:12 logmsgbot: krenair rebuilt wikiversions.cdb and synchronized wikiversions files: add cnwikimedia
• 16:08 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 15s)
• 16:07 logmsgbot: krenair Synchronized database lists: (no message) (duration: 00m 15s)
• 16:07 logmsgbot: krenair Synchronized w/static/images/project-logos/cnwikimedia.png: (no message) (duration: 00m 19s)
• 15:52 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool db1063 (duration: 00m 14s)
• 15:32 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool db1063 (warm period) (duration: 00m 13s)
• 15:24 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/213652/ (duration: 00m 15s)
• 15:23 logmsgbot: krenair Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/213257/ (duration: 00m 14s)
• 14:54 bblack: restarted ganglia-monitor on all cp* (many were obviously-broken, probably most recently from bad startup after the reboots last week)
• 14:14 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool db1063 (duration: 00m 12s)
• 08:24 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool pc1001 (duration: 00m 13s)
• 05:53 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue May 26 05:52:50 UTC 2015 (duration 52m 49s)
• 03:02 logmsgbot: LocalisationUpdate completed (1.26wmf7) at 2015-05-26 03:01:12+00:00
• 02:55 logmsgbot: l10nupdate Synchronized php-1.26wmf7/cache/l10n: (no message) (duration: 09m 31s)
• 02:29 logmsgbot: LocalisationUpdate completed (1.26wmf6) at 2015-05-26 02:28:08+00:00
• 02:24 logmsgbot: l10nupdate Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 06m 44s)
• 01:35 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1026, warm up (duration: 00m 14s)

May 25

• 16:36 jynus: running diagnostics on mariadb@pc1001: a very small amount of requests may experience extra latency
• 14:17 duh: intentionally not scapping right now, will let l10nupdate sync it out
• 14:16 logmsgbot: legoktm Synchronized php-1.26wmf7/extensions/WikimediaMessages/i18n/: ExtensionDistributor message updates (duration: 00m 17s)
• 13:53 logmsgbot: legoktm Synchronized php-1.26wmf7/extensions/ExtensionDistributor: Update ExtensionDistributor to master (duration: 00m 13s)
• 13:38 logmsgbot: jynus Synchronized wmf-config/InitialiseSettings-labs.php: restbase change from yurik (duration: 00m 14s)
• 13:37 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool db1018 (warm cache) (duration: 00m 13s)
• 13:09 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool db1018 (duration: 00m 14s)
• 10:31 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool db1018 (duration: 00m 13s)
• 08:36 YuviKTM: running du -d 1 -h > du-may-25-2015 on /exp/project/tools on labstore1001 to audit tools' NFS usage
• 05:12 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon May 25 05:11:47 UTC 2015 (duration 11m 46s)
• 02:50 logmsgbot: LocalisationUpdate completed (1.26wmf7) at 2015-05-25 02:49:45+00:00
• 02:45 logmsgbot: l10nupdate Synchronized php-1.26wmf7/cache/l10n: (no message) (duration: 06m 32s)
• 02:27 logmsgbot: LocalisationUpdate completed (1.26wmf6) at 2015-05-25 02:26:39+00:00
• 02:22 logmsgbot: l10nupdate Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 06m 36s)

May 24

• 17:18 springle: stop mysqld db1002 db1003 db1004 db1005 db1006 db1007
• 10:00 ^d: gerrit: manually gc'd all repos to help with clone times
• 08:55 godog: resize existing whisper files with new retention on graphite2001
• 05:42 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun May 24 05:41:35 UTC 2015 (duration 41m 34s)
• 02:58 logmsgbot: LocalisationUpdate completed (1.26wmf7) at 2015-05-24 02:57:17+00:00
• 02:53 logmsgbot: l10nupdate Synchronized php-1.26wmf7/cache/l10n: (no message) (duration: 06m 57s)
• 02:34 logmsgbot: LocalisationUpdate completed (1.26wmf6) at 2015-05-24 02:33:23+00:00
• 02:29 logmsgbot: l10nupdate Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 06m 34s)

May 23

• 23:30 logmsgbot: ori Synchronized php-1.26wmf7/extensions/Gadgets: b592efa5fe: Update Gadgets for I6da3eede0: Conversion to using WAN cache (duration: 00m 13s)
• 12:54 godog: remove MediaWiki.xhprof to pick up new retention schema
• 12:53 godog: bounce carbon on graphite1001 to pick up new retention schema
• 11:16 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Ic258d01a7: Revert "Change StatsD port to another value temporarily" (duration: 00m 13s)
• 10:22 ori: Metrics from MediaWiki to graphite are temporarily suspended while xhprof profiling work is ongoing.
• 10:21 logmsgbot: ori Synchronized wmf-config/StartProfiler.php: Exclude xhprof.run_init from being reported (duration: 00m 13s)
• 10:03 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 13s)
• 09:57 logmsgbot: ori Synchronized wmf-config/StartProfiler.php: Ia7549d45: Re-enable xhprof profiling (duration: 00m 14s)
• 09:52 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I311c989e9: Change StatsD port to another value temporarily (duration: 00m 14s)
• 05:13 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May 23 05:12:44 UTC 2015 (duration 12m 43s)
• 02:45 logmsgbot: LocalisationUpdate completed (1.26wmf7) at 2015-05-23 02:44:48+00:00
• 02:41 logmsgbot: l10nupdate Synchronized php-1.26wmf7/cache/l10n: (no message) (duration: 05m 56s)
• 02:24 logmsgbot: LocalisationUpdate completed (1.26wmf6) at 2015-05-23 02:23:36+00:00
• 02:20 logmsgbot: l10nupdate Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 06m 02s)
• 00:33 mutante: adding cwdent to WMF LDAP group per https://www.mediawiki.org/wiki/User:CDentinger_%28WMF%29
• 00:04 logmsgbot: ori Synchronized php-1.26wmf6/includes: 9bf0236c20, 2d3c9233ed (duration: 00m 17s)

May 22

• 20:59 logmsgbot: ori Synchronized php-1.26wmf7/includes: 4632aff034 (duration: 00m 18s)
• 19:19 logmsgbot: ori Synchronized php-1.26wmf6/includes/profiler: 0d9c4dd8fe, ec22d6e6c3, 4127b1a315: Profiler improvements (duration: 00m 16s)
• 19:18 logmsgbot: ori Synchronized php-1.26wmf7/includes/profiler: a69ee4a0f7, a3773b4d8b, ab19be9d99: Profiler improvements (duration: 00m 15s)
• 17:16 yuvipanda: rebooted labvirt1005 from mgmt see what's up with disk array
• 16:53 yuvipanda: rebooted labvirt1005 for T99738
• 15:01 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/211696/ - disable VE A/B test (duration: 00m 12s)
• 13:57 jynus: schema change on x1 shard https://phabricator.wikimedia.org/T94427 No downtime expected
• 10:55 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool db1036 (duration: 00m 12s)
• 07:58 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool db1036 (duration: 00m 13s)
• 06:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 22 06:47:25 UTC 2015 (duration 47m 23s)
• 05:50 springle: upgrade db1026 trusty mariadb 10, mydumper reload
• 03:09 logmsgbot: LocalisationUpdate completed (1.26wmf7) at 2015-05-22 03:08:51+00:00
• 03:02 logmsgbot: l10nupdate Synchronized php-1.26wmf7/cache/l10n: (no message) (duration: 10m 14s)
• 02:43 logmsgbot: hoo Synchronized php-1.26wmf6/extensions/Wikidata/: Update Wikidata: Make wbmergeitems respect the bot parameter (duration: 00m 19s)
• 02:38 logmsgbot: hoo Synchronized php-1.26wmf7/extensions/Wikidata/: Update Wikidata from wmf4 to wmf6 branch. (duration: 00m 22s)
• 02:36 logmsgbot: LocalisationUpdate completed (1.26wmf6) at 2015-05-22 02:35:33+00:00
• 02:32 logmsgbot: l10nupdate Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 05m 56s)

May 21

• 23:50 logmsgbot: hoo Synchronized wmf-config/InitialiseSettings.php: Re-enable subpages for the template namespace on officewiki (duration: 00m 13s)
• 23:35 logmsgbot: hoo Synchronized wmf-config/InitialiseSettings.php: Enable NewUserMessage on hif.wikipedia (duration: 00m 14s)
• 23:30 logmsgbot: hoo Synchronized wmf-config/InitialiseSettings.php: Configure import sources for hif.wikipedia (duration: 00m 12s)
• 23:26 logmsgbot: hoo Synchronized wmf-config/InitialiseSettings.php: Site name configuration on ast.wiktionary (duration: 00m 12s)
• 23:08 logmsgbot: ori Synchronized php-1.26wmf6/includes: 7238213e6d: Defer some updates in doEditUpdates() (duration: 00m 16s)
• 23:07 logmsgbot: ori Synchronized php-1.26wmf7/includes: da79b19b88: Defer some updates in doEditUpdates() (duration: 00m 16s)
• 17:01 mutante: mw1123: apt-get autoclean, rebooting for kernel upgrade
• 16:57 mutante: dist-upgrade on mw1123
• 16:34 logmsgbot: kartik Finished scap: Update ContentTranslation (duration: 23m 25s)
• 16:10 logmsgbot: kartik Started scap: Update ContentTranslation
• 16:04 mutante: armed keyholder on mira
• 15:56 kart_: Updated cxserver
• 15:32 Tim: removed max-registration properties from 2015 board elections on metawiki and votewiki per my comment on T97924
• 15:09 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/212281/ (duration: 00m 10s)
• 15:06 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/211116/ (duration: 00m 16s)
• 15:00 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/205778/ - enable VE A/B test (duration: 00m 14s)
• 14:58 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/205778/ - VE A/B test on enwiki (duration: 00m 11s)
• 14:37 bblack: enabling puppet on caches for varnish retries changes...
• 11:51 logmsgbot: twentyafterfour Finished scap: 1.26wmf7 symlinks (duration: 05m 16s)
• 11:49 twentyafterfour: I'm investigating some inconsistencies in symlinks in /srv/mediawiki, ref https://phabricator.wikimedia.org/T99886
• 11:46 logmsgbot: twentyafterfour Started scap: 1.26wmf7 symlinks
• 11:31 paravoid: troubleshooting analytics1036, includes reboots
• 07:49 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia distribution jessie-wikimedia: php-luasandbox_2.0.9
• 07:21 _joe_: cleaning the bytecode cache database everywhere
• 06:43 _joe_: cleaning up the bytecode caches of a few appservers
• 06:28 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 21 06:27:09 UTC 2015 (duration 27m 8s)
• 04:55 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Ia5239c1e: Unset $wgDiff, so we stop shelling out to diff (duration: 00m 12s)
• 03:10 logmsgbot: LocalisationUpdate completed (1.26wmf7) at 2015-05-21 03:09:49+00:00
• 03:06 logmsgbot: l10nupdate Synchronized php-1.26wmf7/cache/l10n: (no message) (duration: 06m 13s)
• 02:45 logmsgbot: LocalisationUpdate completed (1.26wmf6) at 2015-05-21 02:44:18+00:00
• 02:38 logmsgbot: l10nupdate Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 09m 36s)
• 00:38 logmsgbot: ori Synchronized php-1.26wmf7/includes/MediaWiki.php: adacd7b35c: Pass a message key to MalformedTitleException constructor (duration: 00m 11s)
• 00:37 logmsgbot: ori Synchronized php-1.26wmf6/includes/MediaWiki.php: b13721b5cb: Pass a message key to MalformedTitleException constructor (duration: 00m 12s)
• 00:20 logmsgbot: ori Synchronized php-1.26wmf6/includes/jobqueue/JobQueueGroup.php: 1e43c05283: Revert "Undefer push() in lazyPush() temporarily" (duration: 00m 12s)

May 20

• 23:07 logmsgbot: legoktm Synchronized php-1.26wmf7/extensions/SyntaxHighlight_GeSHi/: https://gerrit.wikimedia.org/r/212456 (duration: 00m 14s)
• 23:05 logmsgbot: legoktm Synchronized wmf-config/: Disable WikiGrok in WMF production (duration: 00m 13s)
• 22:14 logmsgbot: twentyafterfour Purged l10n cache for 1.26wmf5
• 21:51 logmsgbot: ori Synchronized php-1.26wmf6/includes: I32a3cfabc: Made pushLazyJobs() handle all queue groups (duration: 00m 18s)
• 21:25 logmsgbot: legoktm Synchronized php-1.26wmf7/extensions/SyntaxHighlight_GeSHi: https://gerrit.wikimedia.org/r/#/c/212450/ (duration: 00m 13s)
• 21:18 logmsgbot: twentyafterfour Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 14s)
• 21:01 cscott: updated OCG to version ca4f64852de5b1de782b292b50038fbd2dd84266
• 20:59 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.26wmf7
• 20:58 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.26wmf6
• 20:50 logmsgbot: twentyafterfour Finished scap: retry: testwiki to php-1.26wmf7 and rebuild l10n cache (duration: 26m 02s)
• 20:42 ebernhardson: restarted gmond on elastic10{01..31}.eqiad.wmnet
• 20:24 logmsgbot: twentyafterfour Started scap: retry: testwiki to php-1.26wmf7 and rebuild l10n cache
• 20:12 subbu: deployed parsoid version 8ed6fd0b
• 19:35 logmsgbot: twentyafterfour scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_3448528422" --threads=4 --lang en --quiet' returned non-zero exit status 255 (duration: 03m 22s)
• 19:32 logmsgbot: twentyafterfour Started scap: testwiki to php-1.26wmf7 and rebuild l10n cache
• 17:41 bblack: esams+eqiad upload varnish caches will be downtimed+rebooted today, experimenting with depool effects as well (next several hours)
• 16:03 logmsgbot: manybubbles Synchronized php-1.26wmf5/extensions/Flow/: SWAT update flow for wmf5 to fix two issues (duration: 00m 14s)
• 15:54 godog: rolling restart restbase on restbase1003-1006
• 15:52 mobrovac: restbase restarted on restbase1002
• 15:47 godog: restbase restarted on restbase1001
• 15:35 logmsgbot: manybubbles Synchronized php-1.26wmf6/extensions/Flow/: SWAT update flow for wmf6 to fix two issues (duration: 00m 12s)
• 15:22 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT new namespaces for ptwikinews (duration: 00m 11s)
• 15:18 logmsgbot: manybubbles Synchronized wmf-config/throttle.php: SWAT clean old throttle rule and add a new one for an upcoming festival (duration: 00m 13s)
• 15:14 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT update urwikiquote logo 2/2 (duration: 00m 11s)
• 15:13 logmsgbot: manybubbles Synchronized w/static/images/project-logos/urwikiquote.png: SWAT update urwikiquote logo 1/2 (duration: 00m 13s)
• 15:06 springle: db1045 pt-osc reindexing (should be low load, ~2hr)
• 14:36 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable usage tracking on itwiki and wikiquote (duration: 00m 16s)
• 14:25 milimetric: Deployed Event Logging Server with better batch insertion on Monday, May 18 (apologies for late notice)
• 13:13 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1045; depool db1026 (duration: 00m 13s)
• 10:18 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool db1063 (duration: 00m 11s)
• 09:43 _joe_: stopping puppet, fiddling with HHVM parameters on mw1114
• 09:37 Coren: tools kicked grrrit-wm in the diodes.
• 09:35 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool db1063 (duration: 00m 12s)
• 06:45 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool db1063 for maintenance (duration: 00m 11s)
• 06:43 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed May 20 06:42:22 UTC 2015 (duration 42m 21s)
• 03:13 logmsgbot: LocalisationUpdate completed (1.26wmf6) at 2015-05-20 03:12:31+00:00
• 03:06 logmsgbot: l10nupdate Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 09m 40s)
• 02:41 logmsgbot: LocalisationUpdate completed (1.26wmf5) at 2015-05-20 02:40:07+00:00
• 02:36 logmsgbot: l10nupdate Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 06m 30s)
• 01:14 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1045 (duration: 00m 15s)
• 00:43 logmsgbot: ebernhardson Synchronized wmf-config/: Per-user poolcounter triggered many more times than expected (duration: 00m 15s)
• 00:42 logmsgbot: ebernhardson Synchronized wmf-config/PoolCounterSettings-common.php: Enable per-user poolcounter in CirrusSearch on all wikis (duration: 00m 14s)
• 00:41 logmsgbot: ebernhardson Synchronized wmf-config/InitialiseSettings.php: Enable per-user poolcounter in CirrusSearch on all wikis (duration: 00m 12s)
• 00:40 logmsgbot: ebernhardson Synchronized php-1.26wmf5/extensions/NavigationTiming/: Update NavigationTiming for cherry-picks in 1.26wmf5 (duration: 00m 12s)
• 00:39 logmsgbot: ebernhardson Synchronized php-1.26wmf6/extensions/NavigationTiming/: Update NavigationTiming for cherry-picks in 1.26wmf6 (duration: 00m 12s)
• 00:36 logmsgbot: ebernhardson Synchronized php-1.26wmf5/extensions/CirrusSearch/: Bump CirrusSearch in 1.26wmf5 for poolcounter error message updates (duration: 00m 11s)
• 00:35 logmsgbot: ebernhardson Synchronized php-1.26wmf6/extensions/CirrusSearch/: Bump CirrusSearch in 1.26wmf6 for poolcounter error message updates (duration: 00m 13s)
• 00:34 logmsgbot: ebernhardson Synchronized php-1.26wmf5/extensions/CirrusSearch/: Bump CirrusSearch in 1.26wmf5 for poolcounter error message updates (duration: 00m 12s)
• 00:32 logmsgbot: ebernhardson Synchronized php-1.26wmf6/extensions/CirrusSearch/: Bump CirrusSearch in 1.26wmf6 for poolcounter error message updates (duration: 00m 12s)

May 19

• 23:35 gwicke: deployed RESTBase 90817c2a
• 23:20 logmsgbot: bd808 Synchronized wmf-config/InitialiseSettings.php: logstash: Exclude jobrunner debug messages (duration: 00m 12s)
• 23:10 logmsgbot: bd808 Synchronized wmf-config/InitialiseSettings.php: Enable NewUserMessage on maiwiki and pawiki (duration: 00m 12s)
• 22:06 ejegg: updated payment from e89d18ee20abcb1ca3c455e6a298bf8a6aa84442 to 858b87319daa3d66f62eb32e08cefc6b061748d1
• 21:16 logmsgbot: kaldari Synchronized php-1.26wmf6/extensions/MobileFrontend: syncing MobileFrontend for 1.26wmf6 (duration: 00m 11s)
• 21:15 logmsgbot: kaldari Synchronized php-1.26wmf6/extensions/Gather: syncing Gather for 1.26wmf6 (duration: 00m 12s)
• 21:07 robh: merging fixes to sodium, mailing list outage fixed
• 20:51 andrewbogott: rebooting/reimaging virt1005, virt1006, 1007
• 20:22 mutante: mailman: killed processes by user "list". started mailman
• 19:40 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Ia6a2cb7: Removed "refreshLinks" from $wgJobBackoffThrottling (duration: 00m 12s)
• 19:37 logmsgbot: anomie Finished scap: Step 2 for deploying ApiFeatureUsage: sync the config, and l10n data again because I don't think it did last time (duration: 44m 34s)
• 19:25 robh: mailman permission errors abound! had to take it offline again and fixing
• 19:02 robh: mailman is back to routing mail normally (still testing rename parts)
• 18:53 logmsgbot: anomie Started scap: Step 2 for deploying ApiFeatureUsage: sync the config, and l10n data again because I don't think it did last time
• 18:51 logmsgbot: anomie Finished scap: Step 1 for deploying ApiFeatureUsage: sync the code and l10n data (duration: 05m 39s)
• 18:46 logmsgbot: anomie Started scap: Step 1 for deploying ApiFeatureUsage: sync the code and l10n data
• 18:38 yuvipanda: issuing start command for all hosts on labvirt1006, just to make sure
• 18:35 yuvipanda: labvirt1006 rebooting, long POST
• 18:31 yuvipanda: restarted labvirt1006
• 18:20 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group1 to 1.26wmf6
• 18:15 robh: stopping mailman again for further planned work T99098
• 17:43 robh: mailing lists still down, scrubbing list archives is painful and error prone
• 17:33 ottomata: starting reboots of analytics worker nodes in order to enable hyperthreading Bug: https://phabricator.wikimedia.org/T90640
• 17:04 robh: puppet stopped on sodium (dont need it restarting mailman while im working)
• 17:04 robh: starting mailman downtime window to scrub content off list archive per T99098
• 16:58 bblack: automated reboots of esams/eqiad non-upload caches starting (should auto-downtime, should be no real impact)...
• 15:51 logmsgbot: anomie Synchronized php-1.26wmf5/extensions/AbuseFilter/: SWAT: Fix boolean response in API action=abusefiltercheckmatch gerrit:211743 (duration: 00m 12s)
• 15:50 logmsgbot: anomie Synchronized php-1.26wmf6/extensions/AbuseFilter/: SWAT: Fix boolean response in API action=abusefiltercheckmatch gerrit:211744 (duration: 00m 10s)
• 15:31 logmsgbot: anomie Synchronized php-1.26wmf5/includes/skins/SkinTemplate.php: SWAT: Revert "output mw-content-{ltr,rtl} unconditionally" gerrit:211893 (duration: 00m 12s)
• 15:28 logmsgbot: anomie Synchronized php-1.26wmf6/includes/skins/SkinTemplate.php: SWAT: Revert "output mw-content-{ltr,rtl} unconditionally" gerrit:211894 (duration: 00m 13s)
• 15:16 logmsgbot: anomie Synchronized php-1.26wmf5/includes/registration/ExtensionRegistry.php: SWAT: registration: Don't array_unique() over the queue before loading it [[gerrit:211948] (duration: 00m 12s)
• 15:15 logmsgbot: anomie Synchronized php-1.26wmf6/includes/registration/ExtensionRegistry.php: SWAT: registration: Don't array_unique() over the queue before loading it [[gerrit:211947] (duration: 00m 12s)
• 14:43 jynus: back to read/write after virt1000 database migration - migration seems ok
• 14:41 godog: purge cassandra system CF metrics from graphite1001
• 14:29 jynus: temporarily going read-only for virt1000 for database migration
• 14:24 mobrovac: enabled puppet on restbase1001
• 14:19 mobrovac: restbase group1 wiki keyspaces created
• 14:15 mobrovac: starting manually RB with group1 wikis enabled on restbase1001
• 14:11 mobrovac: restbase100x: removed superfluous keyspaces by hand from Cassandra
• 13:47 bblack: done with cp40xx reboot process
• 13:32 bblack: rebooting ulsfo caches (cp40xx - currently depooled from all traffic + downtimed in icinga)
• 13:09 mobrovac: disabled puppet on restbase100x
• 12:51 godog: bounce hhvm on mw1152
• 08:26 _joe_: restarting a few HHVM instances with a full TC space
• 05:05 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue May 19 05:03:56 UTC 2015 (duration 3m 55s)
• 02:46 logmsgbot: LocalisationUpdate completed (1.26wmf6) at 2015-05-19 02:45:17+00:00
• 02:43 logmsgbot: krinkle Synchronized php-1.26wmf6/includes/resourceloader/ResourceLoader.php: Ic0df4fb5cff (duration: 00m 12s)
• 02:42 logmsgbot: l10nupdate Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 05m 43s)
• 02:26 logmsgbot: LocalisationUpdate completed (1.26wmf5) at 2015-05-19 02:25:05+00:00
• 02:21 logmsgbot: l10nupdate Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 06m 11s)
• 00:37 logmsgbot: ebernhardson Synchronized php-1.26wmf5/includes/jobqueue/JobQueueGroup.php: Undefer push() in lazyPush() temporarily (duration: 00m 12s)
• 00:36 logmsgbot: ebernhardson Synchronized php-1.26wmf6/includes/jobqueue/JobQueueGroup.php: Undefer push() in lazyPush() temporarily (duration: 00m 12s)

May 18

• 23:49 yuvipanda: restarted nutcracker on mw1053 and mw1107 for bd808
• 23:47 bd808: nutcracker needs restart on mw1053 and mw1107
• 23:37 yuvipanda: restarting hhvm on mw1123
• 23:36 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Revert "Removed "refreshLinks" from $wgJobBackoffThrottling" (duration: 00m 14s)
• 23:29 logmsgbot: ebernhardson Synchronized wmf-config/CommonSettings.php: removed refreshlinks from #wgJobBackoffThrottling (duration: 00m 14s)
• 23:21 hoo: Reverting my changes to the sites and site_identifiers tables from earlier on... apparently the export/importSites.php maintenance scripts don't work as advertised
• 23:03 logmsgbot: ori Synchronized php-1.26wmf6/extensions/Echo: 8609cb6b90: Update Echo for cherry-picks (duration: 00m 30s)
• 23:02 logmsgbot: ori Synchronized php-1.26wmf5/extensions/Echo: 8c619b99a6: Update Echo for cherry-picks (duration: 00m 57s)
• 22:46 hoo: Updating the sites table on all wikis to reflect the language code change of bhwiki (from bh to bho). I have a backup of the old table from Wikidata in my home, should things go wrong.
• 20:38 mforns: upgraded and restarted EventLogging server: 19b5b7ae719321c4b8fb112890b574051b090571
• 20:12 subbu: deployed parsoid version 8ed3e503
• 19:42 yurik: restarted graphoid service to pick up the new config https://gerrit.wikimedia.org/r/#/c/211450/
• 19:35 ori: restarted statsv on hafnium
• 18:29 logmsgbot: ori Synchronized php-1.26wmf6/includes: 335f8a257d, e3b2255d9c (for UBN! T99468) (duration: 00m 28s)
• 18:28 logmsgbot: ori Synchronized php-1.26wmf5/includes: 335f8a257d, e3b2255d9c (for UBN! T99468) (duration: 01m 26s)
• 18:06 ori: restarted HHVM on mw1107 with libjemalloc heap profiling enabled
• 17:55 ori: Enabling heap profiling on mw11107 to troubleshoot T99525
• 17:08 andrewbogott: starting all instances on labvirt1001 (well, the ones that were running before)
• 16:59 andrewbogott_: dist-upgrading labvirt1001 since it’s down anyway and we may be due for kernel updates.
• 16:53 andrewbogott_: rebooting labvirt1001, and frowning a lot
• 15:59 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/209286/ and https://gerrit.wikimedia.org/r/#/c/211407/ - should be no-ops (duration: 00m 20s)
• 15:36 logmsgbot: marktraceur Synchronized php-1.26wmf6/includes/: [SWAT] [wmf6] resourceloader: Don't cache minification of user.tokens (duration: 00m 19s)
• 15:24 logmsgbot: marktraceur Synchronized php-1.26wmf6/includes/Title.php: [SWAT] [wmf6] Log callers that trigger Title::newFromText $text type warning (duration: 00m 46s)
• 15:23 logmsgbot: marktraceur Synchronized php-1.26wmf5/includes/Title.php: [SWAT] [wmf5] Log callers that trigger Title::newFromText $text type warning (duration: 00m 15s)
• 15:07 logmsgbot: marktraceur Synchronized wmf-config/InitialiseSettings.php: [SWAT] [config] Add wikis for deployment on 2015-05-18 (duration: 00m 29s)
• 14:35 andrewbogott: disabling puppet on labnet1001 to debug dnsmasq
• 14:07 _joe_: restarting HHVM on mw1107 - memory leak probably happening
• 13:38 logmsgbot: aude Synchronized wmf-config/InitialiseSettings-labs.php: Remove beta-specific Graph settings (duration: 01m 46s)
• 13:34 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable arbitrary access on enwikivoyage, fawiki, and hewiki, and graph extension everywhere (duration: 00m 57s)
• 13:31 logmsgbot: aude Synchronized php-1.26wmf6/extensions/Wikidata: Fix rdf dump script (duration: 03m 23s)
• 13:27 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool db1063 after warmup period (duration: 01m 01s)
• 13:01 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: repool db1063 (duration: 00m 17s)
• 11:13 yurik: deployed graphoid update to fix https://phabricator.wikimedia.org/T99349
• 11:10 logmsgbot: jynus Synchronized wmf-config/db-eqiad.php: depool db1063 (duration: 01m 00s)
• 11:07 jynus: depooling db1063 from cluster for maintenance
• 09:02 godog: loss on ulsfo-eqiad, depooled ulsfo
• 05:18 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon May 18 05:17:50 UTC 2015 (duration 17m 49s)
• 02:46 logmsgbot: LocalisationUpdate completed (1.26wmf6) at 2015-05-18 02:45:52+00:00
• 02:42 logmsgbot: l10nupdate Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 05m 35s)
• 02:26 logmsgbot: LocalisationUpdate completed (1.26wmf5) at 2015-05-18 02:25:54+00:00
• 02:21 logmsgbot: l10nupdate Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 06m 24s)

May 17

• 05:06 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun May 17 05:05:16 UTC 2015 (duration 5m 15s)
• 02:44 logmsgbot: LocalisationUpdate completed (1.26wmf6) at 2015-05-17 02:43:13+00:00
• 02:39 logmsgbot: l10nupdate Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 05m 18s)
• 02:25 logmsgbot: LocalisationUpdate completed (1.26wmf5) at 2015-05-17 02:24:09+00:00
• 02:20 logmsgbot: l10nupdate Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 06m 10s)

May 16

• 13:27 manybubbles: that was the last server in the elasticsearch rolling restart. all done. now we have new versions of the plugins. Lets try not to do that again.
• 13:25 manybubbles: es-tool restart-fast on elastic1031
• 09:15 godog: bounce hhvm on mw1196
• 09:10 godog: bounce hhvm on mw1141
• 07:49 godog: restart hhvm on mw1234, still pushing xhprof metrics
• 06:03 _joe_: killed nrpe on labvirt1003 - see T99341
• 05:02 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May 16 05:01:02 UTC 2015 (duration 1m 1s)
• 04:11 andrewbogott: restarting sshd and generally poking around on labvirt1003
• 02:47 logmsgbot: LocalisationUpdate completed (1.26wmf6) at 2015-05-16 02:46:08+00:00
• 02:43 logmsgbot: l10nupdate Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 04m 55s)
• 02:29 logmsgbot: LocalisationUpdate completed (1.26wmf5) at 2015-05-16 02:28:37+00:00
• 02:25 logmsgbot: l10nupdate Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 05m 55s)

May 15

• 22:35 ejegg: updated crm from 03eb4cff1b009e8abaceec250f9a1c5d1f3c6b18 to 7ffe0cefb019828a09c9369187f14518847b5f41
• 19:44 manybubbles: elastic1027 es-tool restart-fast
• 19:37 awight: update crm from 2a2336655737a2cd1d3cc24624d1e8475e4cf039 to 03eb4cff1b009e8abaceec250f9a1c5d1f3c6b18
• 18:29 manybubbles: elastic1026 es-tool restart-fast
• 18:28 godog: bounce hhvm on mw1118
• 17:55 jynus: migrating of db service from virt1000 to m5-master aborted, service continues on virt1000
• 17:44 manybubbles: rolling restart almost done on elastic1025 - 1026 is next!
• 17:33 andrewbogott: updating qemu binaries on labvirt1001
• 17:29 godog: clean up remaining xhprof metrics from graphite1001
• 17:19 godog: bounce hhvm on mw1017
• 17:07 godog: still seeing metrics from xhprof creating, looking for source
• 16:29 godog: bounce carbon on graphite1001
• 16:23 manybubbles: elastic1023 and elastic1024 (skipped one log) es-tool restart-fast
• 16:16 godog: bounce statsdlb on graphite1001
• 14:49 jynus: migrating mariadb service from virt1000 to m5-master
• 14:37 manybubbles: elastic1021 es-tool restart-fast
• 14:26 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1053 in s1, warm up (duration: 00m 13s)
• 12:21 manybubbles: elastic1020 es-tool restart-fast
• 10:19 godog: bounce statsite and uwsgi on graphite1001
• 09:29 godog: restart carbon on graphite1001
• 09:15 godog: restart hhvm on mw1018, straggling
• 09:07 godog: rm MediaWiki.run_init from graphite1001 / graphite2001
• 09:04 ori: restarted hhvm / jobrunner on jobrunners to force them to pick up I6a516a0da ; re-cleared /var/lib/carbon/whisper/MediaWiki/query_* on graphite1001 and graphite2001
• 08:49 kart_: Updated cxserver to 1cb6cec
• 08:21 jynus: reenabling icinga check for MySQL on db1009
• 08:15 logmsgbot: oblivian Synchronized wmf-config/StartProfiler.php: Null-sync to touch the file (duration: 00m 12s)
• 07:20 ori: rm -rf /var/lib/carbon/whisper/MediaWiki/query_* on graphite1001 and graphite2001, as follow-up cleanup for I6a516a0da
• 07:14 logmsgbot: ori Synchronized wmf-config/StartProfiler.php: I6a516a0da: Don't send profiling data to graphite for now (duration: 00m 11s)
• 06:23 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 15 06:22:19 UTC 2015 (duration 22m 18s)
• 05:38 jynus: temporarily opening mysql port on firewall from db1009 to virt1000
• 04:37 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1018, warm up (duration: 00m 11s)
• 02:58 logmsgbot: LocalisationUpdate completed (1.26wmf6) at 2015-05-15 02:56:59+00:00
• 02:55 springle: xtrabackup clone db1057 to db1053
• 02:54 logmsgbot: l10nupdate Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 04m 37s)
• 02:42 springle: upgrade db1053 trusty
• 02:34 logmsgbot: LocalisationUpdate completed (1.26wmf5) at 2015-05-15 02:33:18+00:00
• 02:33 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1019; depool db1053 (duration: 00m 13s)
• 02:30 logmsgbot: l10nupdate Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 05m 39s)
• 02:12 manybubbles|away: elastic1019 es-tool restart-fast
• 01:12 manybubbles|away: elastic1018 es-tool restart-fast
• 00:07 manybubbles|away: elastic1017 es-tool restart-fast

May 14

• 23:35 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 11s)
• 23:20 ori: Depooled mw1169; HHVM deadlock à la T89912. Leaving it depooled to investigate.
• 23:05 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 11s)
• 23:05 logmsgbot: demon Synchronized w/static/images/project-logos/urwikiquote.png: (no message) (duration: 00m 14s)
• 23:03 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 17s)
• 22:26 logmsgbot: ori Synchronized wmf-config/StartProfiler.php: Icbf826a7: 1:1000 request profiling via xhprof (duration: 00m 12s)
• 22:23 gwicke: deployed RESTBase v0.6.3 (fd942ac38ad)
• 22:20 logmsgbot: ori Synchronized php-1.26wmf6/extensions/CirrusSearch/includes/ElasticsearchIntermediary.php: (no message) (duration: 00m 15s)
• 21:39 manybubbles: I'm going to be done doing rolling restarts for a couple of hours. If someone wants to pick them up and do the next one after the cluster goes green again then be my guest.
• 21:35 manybubbles: es-tool restart-fast on elastic1016
• 21:27 logmsgbot: ori Synchronized php-1.26wmf6/extensions/CirrusSearch/includes/ElasticsearchIntermediary.php: (no message) (duration: 00m 12s)
• 21:27 logmsgbot: ori Synchronized php-1.26wmf5/extensions/CirrusSearch/includes/ElasticsearchIntermediary.php: (no message) (duration: 00m 12s)
• 21:14 logmsgbot: ori Synchronized php-1.26wmf6/extensions/CirrusSearch/includes/ElasticsearchIntermediary.php: I3df6713a1: Log request times to StatsD (duration: 00m 13s)
• 21:14 logmsgbot: ori Synchronized php-1.26wmf5/extensions/CirrusSearch/includes/ElasticsearchIntermediary.php: I3df6713a1: Log request times to StatsD (duration: 00m 15s)
• 21:11 manybubbles: elastic1015 es-tool restart-fast
• 19:43 robh: mass unsubcription in listadmins list, resulting in unsupressed mass unsubscribe notices to all listadmin email address (sorry about the emails!)
• 19:24 logmsgbot: legoktm Synchronized php-1.26wmf5/skins/Nostalgia/skin.json: touch (duration: 00m 17s)
• 19:15 legoktm: debugging on tin / mw1017 for nostalgiawiki issue
• 16:59 ^d: elasticsearch: set transient cluster.routing.allocation.node_concurrent_recoveries on prod cluster to 8 (default: 2) to speed up recoveries.
• 16:52 logmsgbot: kartik Finished scap: Update ContentTranslation (duration: 44m 07s)
• 16:28 andrewbogott: disabling puppet on labnet1001 for testing
• 16:13 godog: es-tool restart-fast on elastic1014
• 16:08 logmsgbot: kartik Started scap: Update ContentTranslation
• 15:46 logmsgbot: thcipriani Synchronized php-1.26wmf5/extensions/Translate: SWAT update translate to a6f0a63 gerrit:210919 (duration: 00m 15s)
• 15:12 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT enable new article campaign except bawiki gerrit:210916 (duration: 00m 12s)
• 15:04 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: Open external links on votewiki in new tab gerrit:210849 (duration: 00m 12s)
• 15:00 godog: es-tool restart-fast on elastic1013
• 14:48 logmsgbot: andyrussg Synchronized php-1.26wmf6/extensions/CentralNotice/: Update CentralNotice (duration: 00m 13s)
• 14:34 paravoid: reimaging multatuli
• 14:34 jynus: migrating data db from virt1000 to db1009
• 14:23 bblack: restarted ganglia-monitor on eeden
• 14:21 logmsgbot: andyrussg Synchronized php-1.26wmf5/extensions/CentralNotice/: Update CentralNotice (duration: 00m 12s)
• 14:16 godog: es-tool restart-fast on elastic1012
• 14:12 paravoid: switching ns2 back to eeden
• 13:56 cmjohnson1: upgrading tellurium to trusty
• 13:41 cmjohnson1: power cycling barium
• 13:40 godog: es-root restart-fast on elastic1011
• 13:21 paravoid: reimaging eeden with jessie
• 12:59 paravoid: switching ns2 to multatuli
• 12:53 jynus: disabling temporarily Ichinga check for MySQL running on db1009 until data is migrated from virt1000 and host sent to production
• 12:40 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-pt-gl_0.9.2~r60358-1
• 12:36 godog: es-tool restart-fast on elastic1010
• 11:40 manybubbles: restarting elasticsearch on elastic1009
• 05:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 14 05:06:09 UTC 2015 (duration 6m 8s)
• 02:55 manybubbles: restarting elasticsearch on elastic1008
• 02:50 logmsgbot: LocalisationUpdate completed (1.26wmf6) at 2015-05-14 02:49:53+00:00
• 02:47 logmsgbot: l10nupdate Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 04m 16s)
• 02:44 springle: xtrabackup clone db1056 to db1019
• 02:29 logmsgbot: LocalisationUpdate completed (1.26wmf5) at 2015-05-14 02:28:02+00:00
• 02:24 logmsgbot: l10nupdate Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 05m 51s)
• 01:48 manybubbles: sorry - restarting elasticsearch on elastic1007
• 01:47 manybubbles: restarting elastic1007
• 01:33 logmsgbot: springle Synchronized wmf-config/db-codfw.php: pool new codfw slaves (duration: 00m 11s)
• 01:28 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1060, warm up (duration: 00m 14s)
• 00:49 manybubbles: restarting elasticsearch on elastic1006
• 00:03 logmsgbot: ebernhardson Synchronized php-1.26wmf5/extensions/Gather/: SWAT Submodule bump for Gather extension (duration: 00m 12s)

May 13

• 23:52 awight: payments config: correct memcache location
• 23:40 logmsgbot: ebernhardson Synchronized wmf-config/CirrusSearch-common.php: SWAT deploy cirrus config change (duration: 00m 12s)
• 22:26 logmsgbot: twentyafterfour Purged l10n cache for 1.26wmf4
• 22:25 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: Group 0 to 1.26wmf6
• 22:21 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.26wmf5
• 22:17 twentyafterfour: restarted phd on iridium (phabricator) to sync the daemons' configuration
• 21:28 manybubbles: restarting elasticsearch on elastic1005
• 21:12 cscott: updated OCG to version c7c75e5b03ad9096571dc6dbfcb7022c924ccb4f
• 21:03 awight: updated payments from f97f8f99268974cfdb0182f178955bd627137842 to e89d18ee20abcb1ca3c455e6a298bf8a6aa84442
• 20:28 subbu: deployed parsoid version a8108fe6
• 20:15 manybubbles: restarted elasticsearch on elastic1004
• 20:12 logmsgbot: twentyafterfour Finished scap: testwiki to php-1.26wmf6 and rebuild l10n cache (duration: 47m 24s)
• 20:11 manybubbles: cancel that - I just realized I can't do that.
• 20:10 manybubbles: elastic1003 restarted elasticsearch just fine. the cluster restart is going awesome. I'm going to rig the other 28 to restart via a script, one after the other. Expect nagios to complain about them some.
• 20:03 bblack: restarting hhvm on mw1190
• 19:25 logmsgbot: twentyafterfour Started scap: testwiki to php-1.26wmf6 and rebuild l10n cache
• 19:11 awight: paymens rolled back to f97f8f99268974cfdb0182f178955bd627137842
• 19:10 awight: payments updated from f97f8f99268974cfdb0182f178955bd627137842 to 5c326a521120a904a2012654e9287757dc5a8ca2
• 19:00 manybubbles: elastic1002 restart went well - starting elastic1003
• 18:45 awight: rolled back payments to f97f8f99268974cfdb0182f178955bd627137842
• 18:43 awight: update payments from f97f8f99268974cfdb0182f178955bd627137842 to 5c326a521120a904a2012654e9287757dc5a8ca2
• 18:05 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: undo all the nostalgia (duration: 00m 10s)
• 17:21 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: something something skins are broken (duration: 00m 11s)
• 17:14 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: because sometimes moving code helps (duration: 00m 15s)
• 17:10 manybub|lunch: elastic1002 restarted and rejoined the cluster - now the cluster is repaining. hurray.
• 17:08 manybub|lunch: elastic1001 restarted and rejoined the cluster hapilly while I was at lunch. it looks good - no errors beyond the ones we have fixes in flight for. So I'm going to do elastic1002
• 17:03 hashar: Zuul clone failures solved. Was due to network traffic being interrupted between labs and prod.
• 16:53 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/209967/ (duration: 00m 14s)
• 16:51 hashar: Zuul clone failure https://phabricator.wikimedia.org/T98980
• 16:49 andrewbogott: re-enabling puppet on labnet1001
• 16:46 mutante: es2010 failed disk, reopening ticket for last fail in January
• 16:41 jynus: Enabling puppet agent in db1009.eqiad after reinstall
• 16:40 logmsgbot: ori Synchronized php-1.26wmf4/includes/resourceloader/ResourceLoader.php: I30b490e5b: ResourceLoader::filter: use APC when running under HHVM (duration: 00m 11s)
• 16:38 logmsgbot: ori Synchronized php-1.26wmf5/includes/resourceloader/ResourceLoader.php: I30b490e5b: ResourceLoader::filter: use APC when running under HHVM (duration: 00m 14s)
• 16:28 andrewbogott: disabling puppet on labnet1001 to tinker with nova config
• 15:44 mark: Disregard cr2-knams:xe-0/0/0; we're working on it
• 15:21 manybubbles: I think the elasticsearch cluster got stuck with alloation disabled after the rolling restart. Funky. Haven't seen that one before. Probably a problem with our instructions. Anyway, unstuck it and recovery is going faster now
• 15:17 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: didn't work, undoing previous sync (duration: 00m 12s)
• 15:15 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: trying something (duration: 00m 12s)
• 14:53 manybubbles: elasticsearch restart on elastic1001 going well. cluster still in recovering state as expect. I'll give it an hour to soak.
• 14:48 manybubbles: ok - time to start the rolling restart. I'm going to to elastic1001 first non-automated and watch it
• 14:36 manybubbles: s/gitfit/gitfat/ oh well
• 14:35 manybubbles: first attempt at syncing elasticsearch plugins didn't work 100%. syncing again. gitfit/gitdeploy is betraying me
• 14:32 manybubbles: syncing new versions of elsaticsearch plugins to prod. no restarts yet.
• 14:04 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable usage tracking for Wikisource (duration: 00m 14s)
• 13:57 aude: added wbc_entity_usage table on all Wikibase Client wikis
• 13:56 jynus: jcrespo Disabling puppet agent in db1009.eqiad in preparation for reinstall
• 13:45 logmsgbot: aude Synchronized php-1.26wmf5/extensions/Wikidata: Update maintenance script (duration: 00m 20s)
• 12:45 springle: xtrabackup clone db1060 to db1018
• 12:39 springle: upgrade and restart db1060
• 09:20 jamesofur: inserting FDC election encryption key
• 06:21 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed May 13 06:19:59 UTC 2015 (duration 19m 58s)
• 05:53 springle: reinstall db1018
• 04:50 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1018 (duration: 00m 12s)
• 03:11 logmsgbot: LocalisationUpdate completed (1.26wmf5) at 2015-05-13 03:10:31+00:00
• 03:07 logmsgbot: l10nupdate Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 05m 43s)
• 02:46 logmsgbot: LocalisationUpdate completed (1.26wmf4) at 2015-05-13 02:45:28+00:00
• 02:39 logmsgbot: l10nupdate Synchronized php-1.26wmf4/cache/l10n: (no message) (duration: 10m 08s)
• 01:56 damagecat: Started 'jobs' screen in tin to drain refreshLinks for enwiki using --nothrottle (T98621)
• 01:29 logmsgbot: legoktm Synchronized wmf-config/CommonSettings.php: Hardcode UploadWizard max upload size - T98933 (duration: 00m 12s)
• 01:23 logmsgbot: legoktm Synchronized php-1.26wmf5/extensions/GWToolset/: Check php max_file_size limit directly from PHP $_FILES (duration: 00m 12s)
• 01:21 logmsgbot: legoktm Synchronized php-1.26wmf4/extensions/GWToolset/: Check php max_file_size limit directly from PHP $_FILES (duration: 00m 12s)
• 01:07 gwicke: added commons to supported projects in RESTBase API
• 00:16 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I5ebedfdfb: Set $wgGadgetsCacheType to CACHE_ACCEL (duration: 00m 12s)
• 00:13 logmsgbot: ori Synchronized php-1.26wmf4/includes/jobqueue/jobs/RefreshLinksJob.php: 914d71f3cc: Temporary hack to drain excess refreshLinks jobs (duration: 00m 14s)
• 00:12 logmsgbot: ori Synchronized php-1.26wmf4/extensions/Gadgets: 7539873979: Update Gadgets for cherry-pick (duration: 00m 12s)
• 00:10 logmsgbot: ori Synchronized php-1.26wmf5/extensions/Gadgets: cbb9b1e475: Update Gadgets for cherry-pick (duration: 00m 12s)

May 12

• 23:40 ori: Upgraded all Apaches to HHVM 3.6.1+dfsg1-1+wm2 and Apache 2.4.7-1ubuntu4.4
• 23:26 logmsgbot: demon Synchronized php-1.26wmf4/extensions/CirrusSearch/: (no message) (duration: 00m 12s)
• 23:24 logmsgbot: demon Synchronized php-1.26wmf4/includes/jobqueue/jobs/RefreshLinksJob.php: (no message) (duration: 00m 11s)
• 23:23 logmsgbot: demon Synchronized php-1.26wmf5/includes/jobqueue/jobs/RefreshLinksJob.php: (no message) (duration: 00m 12s)
• 23:23 logmsgbot: demon Synchronized php-1.26wmf5/includes/media/DjVu.php: (no message) (duration: 00m 12s)
• 23:18 ori: Upgrading more HHVMs; DPKG alerts likely but they will be transient.
• 23:10 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 11s)
• 23:03 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: swat (duration: 00m 12s)
• 21:48 logmsgbot: kaldari Finished scap: updating i18n for Gather (1.26wmf5) (duration: 23m 17s)
• 21:25 logmsgbot: kaldari Started scap: updating i18n for Gather (1.26wmf5)
• 21:24 logmsgbot: kaldari Synchronized php-1.26wmf5/extensions/Gather: Updating Gather for 1.26wmf5 (duration: 00m 12s)
• 21:06 apergos: manually installed trigger-trebuchet update on tin after accidental salt upgrade there woops :-D
• 20:56 mutante: upgrading salt packages on tin
• 19:50 ori: Upgrading several app servers to new version of HHVM, expect transient 'DPKG CRITICAL' alerts
• 18:19 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: Group1 wikis to 1.26wmf5
• 17:38 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Ie4641b6e4: Set $wgWMEStatsdBaseUri to host-relative beacon/ path (duration: 00m 12s)
• 16:24 yurik: graphoid service synced, now supports Cache Control headers
• 16:19 ori: restarted HHVM on mw1061; T89912
• 15:20 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT Add *.sl.nsw.gov.au to wgCopyUploadsDomains gerrit:210356 (duration: 00m 11s)
• 15:15 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT Namespaces configuration on or.wiktionary gerrit:210350 (duration: 00m 12s)
• 15:10 hashar: mediawiki-phpunit-hhvm Jenkins job is broken due to an hhvm upgrade bug T98876
• 15:07 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT enable NewUserMessage on bh.wikipedia gerrit:209146 (duration: 00m 13s)
• 13:55 akosiaris: temporarily blocked an IP on uranium firewall. It was the cause of requests causing CPU load. http://ganglia.wikimedia.org/latest/graph.php?r=day&z=xlarge&h=uranium.wikimedia.org&m=cpu_report&s=descending&mc=2&g=cpu_report&c=Miscellaneous+eqiad
• 11:06 twentyafterfour: restarted apache on iridium to clear php opecode cache
• 09:53 akosiaris: restarted gitblit on antimony
• 06:58 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue May 12 06:57:17 UTC 2015 (duration 57m 16s)
• 06:15 springle: pt-kill on 3600s running on dbstore1002 until repl streams recover
• 06:05 springle: killed 100+ 3-day unindexed research queries on dbstore1002, all repl streams lagging and /tmp unhappy
• 03:01 logmsgbot: LocalisationUpdate completed (1.26wmf5) at 2015-05-12 03:00:22+00:00
• 02:57 logmsgbot: l10nupdate Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 05m 47s)
• 02:35 logmsgbot: LocalisationUpdate completed (1.26wmf4) at 2015-05-12 02:34:30+00:00
• 02:30 logmsgbot: l10nupdate Synchronized php-1.26wmf4/cache/l10n: (no message) (duration: 06m 33s)
• 00:39 logmsgbot: mattflaschen Synchronized wmf-config/InitialiseSettings.php: Update Wikipedia word mark and related config (duration: 00m 11s)
• 00:38 logmsgbot: mattflaschen Synchronized images/mobile/wikipedia-wordmark-en.png: Update Wikipedia word mark and related config (duration: 00m 13s)
• 00:30 logmsgbot: mattflaschen Synchronized wmf-config/InitialiseSettings.php: Add www.jacar.go.jp to wgCopyUploadsDomains (duration: 00m 11s)
• 00:30 yuvipanda: restarted nutcracker on silver
• 00:28 logmsgbot: mattflaschen Synchronized wmf-config/InitialiseSettings.php: Deploy Catalan Wikinews flood group (duration: 00m 13s)
• 00:19 logmsgbot: mattflaschen Synchronized php-1.26wmf5/includes/page/WikiPage.php: Job queue changes for triggerOpportunisticLinksUpdate (duration: 00m 12s)
• 00:18 logmsgbot: mattflaschen Synchronized php-1.26wmf5/includes/jobqueue/: Job queue changes for triggerOpportunisticLinksUpdate (duration: 00m 12s)
• 00:17 logmsgbot: mattflaschen Synchronized php-1.26wmf4/includes/jobqueue/: Job queue changes for triggerOpportunisticLinksUpdate (duration: 00m 13s)
• 00:15 yuvipanda: restarted apache on silver
• 00:01 logmsgbot: mattflaschen Synchronized php-1.26wmf4/includes/page/WikiPage.php: Job queue changes for triggerOpportunisticLinksUpdate (duration: 00m 11s)
• 00:00 logmsgbot: mattflaschen Synchronized php-1.26wmf4/includes/jobqueue/: Job queue changes for triggerOpportunisticLinksUpdate (duration: 00m 12s)

May 11

• 23:46 logmsgbot: mattflaschen Synchronized wmf-config: Sync wmf-config for CirrusSearch PoolCounter change; applies to group 0 initially (duration: 00m 12s)
• 23:37 logmsgbot: kaldari Synchronized wmf-config/InitialiseSettings-labs.php: sync InitialiseSettings-labs.php for Browse experiment in mobile (duration: 00m 13s)
• 23:34 logmsgbot: mattflaschen Synchronized php-1.26wmf5/extensions/Flow/: Deploy Flow metadataonly fix (duration: 00m 14s)
• 23:32 yuvipanda: andrewbogott_afk playing around with upgrading virt*** boxes, which are non-live labs boxen.
• 23:31 logmsgbot: mattflaschen Synchronized php-1.26wmf4/extensions/Flow/: Deploy Flow metadataonly fix (duration: 00m 13s)
• 23:17 logmsgbot: mattflaschen Synchronized wmf-config/CommonSettings.php: Make VE default editor for Flow (duration: 00m 13s)
• 23:13 legoktm: manually renamed and migrated User:~~@nlwiki --> User:~~-~nlwiki@global (T98155)
• 22:55 logmsgbot: ori Synchronized php-1.26wmf4/extensions/Josa: dd2db67d9b: Update Josa for cherry-picks (duration: 00m 13s)
• 22:54 logmsgbot: ori Synchronized php-1.26wmf5/extensions/Josa: a0b561da25: Update Josa for cherry-picks (duration: 00m 11s)
• 22:05 twentyafterfour: removed /var/run/phab_repo_lock_libext_Sprint on iridium to allow sprint repo sync
• 22:01 logmsgbot: bd808 Synchronized wmf-config/InitialiseSettings-labs.php: Add common wikitag for all beta cluster wikis (duration: 00m 12s)
• 21:54 ori: Restarting HHVM on mw1036; threads stuck on HPHP::StatCache::refresh
• 21:48 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: I45c1c76d4: Deploy Josa extension to production (enabling) (duration: 00m 13s)
• 21:47 logmsgbot: ori Finished scap: I45c1c76d4: Deploy Josa extension to production (but not enabling yet) (duration: 46m 54s)
• 21:43 ori: Restarting HHVM on mw1110; threads stuck on HPHP::StatCache::refresh
• 21:00 logmsgbot: ori Started scap: I45c1c76d4: Deploy Josa extension to production (but not enabling yet)
• 20:49 hoo: Resolved T98695 by setting the email of the global account to the former enwiki email address.
• 19:37 hoo: Updated Wikidata's property suggester with data from today's json dump
• 18:49 legoktm: renamed a bunch more invalid usernames (https://phabricator.wikimedia.org/T5507)
• 18:41 ori: Deployed I4e3f42ea7, which increases jobrunner::runners_basic from 14 -> 20
• 18:41 logmsgbot: yurik Synchronized wmf-config: patch 210111 - Cleaned Graph, enabled wmgGraphImgServiceAlways (duration: 00m 13s)
• 18:15 logmsgbot: yurik Synchronized php-1.26wmf4/extensions/Graph: Bump Graph to master (duration: 00m 11s)
• 18:14 logmsgbot: yurik Synchronized php-1.26wmf5/extensions/Graph: Bump Graph to master (duration: 00m 14s)
• 17:16 logmsgbot: manybubbles Finished scap: SWAT js config vargs changes (duration: 14m 55s)
• 17:01 logmsgbot: manybubbles Started scap: SWAT js config vargs changes
• 17:01 logmsgbot: manybubbles scap aborted: SWAT js config vargs changes (duration: 27m 58s)
• 16:33 logmsgbot: manybubbles Started scap: SWAT js config vargs changes
• 15:59 manybubbles: waiting a few minutes after that last set of patches before we're sure that the load is down and then, hopefully, we'll scap to get the core changes that are already merged and sitting on tin that we had to ignore while we handled the trafic spike.
• 15:53 logmsgbot: manybubbles Synchronized php-1.26wmf4/includes/media/DjVu.php: SWAT: 10 mb djvu files are expensive to thumbnail (wmf4) (duration: 00m 13s)
• 15:52 logmsgbot: manybubbles Synchronized php-1.26wmf5/includes/media/DjVu.php: SWAT: 10 mb djvu files are expensive to thumbnail (wmf5) (duration: 00m 11s)
• 15:33 manybubbles: stopping SWAT due to some incident that just picked up. Right now Ib990f00ebe974008cea4dccbaa212ec20c846674 and Ida3fd5f8808202892001f66c4a534c1725e769a6 are merged awaiting a scap.
• 15:26 logmsgbot: manybubbles Synchronized wmf-config/CommonSettings.php: SWAT cleanup wgGraphImgServiceAlways 3/3 (duration: 00m 12s)
• 15:26 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT cleanup wgGraphImgServiceAlways 2/3 (duration: 00m 12s)
• 15:25 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings-labs.php: SWAT cleanup wgGraphImgServiceAlways 1/3 (duration: 00m 12s)
• 15:05 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT: send all mediawiki events from all wikis to logstash (duration: 00m 12s)
• 15:03 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: enable graph extension in beta. this should be a noop (duration: 00m 13s)
• 14:01 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable arbitrary Wikibase access for nlwiki and frwikisource (duration: 00m 16s)
• 13:49 logmsgbot: aude Synchronized php-1.26wmf4/extensions/Wikidata: Fix interaction with AbuseFilter (duration: 00m 20s)
• 13:46 logmsgbot: aude Synchronized php-1.26wmf5/extensions/Wikidata: Fix interaction with AbuseFilter (duration: 00m 19s)
• 05:10 ori: upgrading canary appservers to 3.6.1+dfsg1-1+wm2
• 04:55 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon May 11 04:53:58 UTC 2015 (duration 53m 57s)
• 04:17 springle: restarted hhvm on mw1020. lots of fatal noise about N4HPHP13DataBlockFullE
• 02:43 logmsgbot: LocalisationUpdate completed (1.26wmf5) at 2015-05-11 02:42:42+00:00
• 02:39 logmsgbot: l10nupdate Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 05m 37s)
• 02:23 logmsgbot: LocalisationUpdate completed (1.26wmf4) at 2015-05-11 02:22:25+00:00
• 02:18 logmsgbot: l10nupdate Synchronized php-1.26wmf4/cache/l10n: (no message) (duration: 06m 19s)

May 10

• 17:45 ori: App server traffic coincides with spike on S4 dbs, lots of commons sleeper queries, fatal log contains many references to User:Richenza/gallery, so nuking.
• 17:20 ori: Inbound app server traffic more than doubled over the past 12 hrs: http://ganglia.wikimedia.org/latest/graph.php?r=week&z=xlarge&c=Application+servers+eqiad&m=cpu_report&s=by+name&mc=2&g=network_report
• 05:17 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun May 10 05:16:10 UTC 2015 (duration 16m 9s)
• 02:45 logmsgbot: LocalisationUpdate completed (1.26wmf5) at 2015-05-10 02:44:48+00:00
• 02:41 logmsgbot: l10nupdate Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 05m 26s)
• 02:25 logmsgbot: LocalisationUpdate completed (1.26wmf4) at 2015-05-10 02:24:40+00:00
• 02:20 logmsgbot: l10nupdate Synchronized php-1.26wmf4/cache/l10n: (no message) (duration: 06m 16s)

May 9

• 20:55 logmsgbot: krenair Synchronized php-1.26wmf4/extensions/VisualEditor/modules/ve-mw/ui/tools/ve.ui.MWEditModeTool.js: https://gerrit.wikimedia.org/r/#/c/209950/ (duration: 00m 12s)
• 20:53 logmsgbot: krenair Synchronized php-1.26wmf5/extensions/VisualEditor/modules/ve-mw/ui/tools/ve.ui.MWEditModeTool.js: https://gerrit.wikimedia.org/r/#/c/209949/ (duration: 00m 11s)
• 05:06 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May 9 05:05:16 UTC 2015 (duration 5m 15s)
• 02:44 logmsgbot: LocalisationUpdate completed (1.26wmf5) at 2015-05-09 02:43:07+00:00
• 02:39 logmsgbot: l10nupdate Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 05m 21s)
• 02:24 logmsgbot: LocalisationUpdate completed (1.26wmf4) at 2015-05-09 02:23:15+00:00
• 02:19 logmsgbot: l10nupdate Synchronized php-1.26wmf4/cache/l10n: (no message) (duration: 06m 11s)

May 8

• 23:45 logmsgbot: bd808 Synchronized wmf-config/CommonSettings.php: beta: switch $wmfUdp2logDest to deployment-fluorine.eqiad.wmflabs (duration: 00m 12s)
• 22:11 mutante: gzipping some user data on lutetium
• 21:17 logmsgbot: yurik Synchronized wmf-config/CommonSettings.php: Disable security header for Graphs on zerowiki (duration: 00m 12s)
• 21:14 logmsgbot: yurik Synchronized wmf-config/InitialiseSettings.php: Disable security header for Graphs on zerowiki (duration: 00m 12s)
• 21:02 logmsgbot: mattflaschen Synchronized wmf-config/InitialiseSettings-labs.php: Sync out change that only affects Beta Cluster (duration: 00m 11s)
• 19:18 logmsgbot: yurik Synchronized php-1.26wmf4/extensions/CentralAuth: Bumping CentralAuth (duration: 00m 13s)
• 19:18 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I6236f5e2c: Use $wgServer to construct static-asset URLs (duration: 00m 12s)
• 19:12 logmsgbot: yurik Synchronized php-1.26wmf5/extensions/CentralAuth: Bumping CentralAuth (duration: 00m 12s)
• 18:42 csteipp: deployed patch for T98313 for wmf4/5
• 18:14 logmsgbot: yurik Synchronized php-1.26wmf4/extensions/Graph/: Bumping graph (duration: 00m 14s)
• 18:14 logmsgbot: yurik Synchronized php-1.26wmf5/extensions/Graph/: Bumping graph (duration: 00m 14s)
• 16:53 logmsgbot: bd808 Synchronized w/static/images/project-logos/labswiki.png: Add missing labswiki.png (duration: 00m 13s)
• 15:37 Krenair: restarted apache on silver -again- to deal with reports of session errors
• 15:28 greg-g: wikitech's session data errors are transient, hitting save multiple times will eventually work
• 15:26 greg-g: multiple independent reports of wikitech wiki having session data errors
• 14:13 logmsgbot: bblack Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 13s)
• 13:17 logmsgbot: faidon Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 12s)
• 13:17 logmsgbot: faidon Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 14s)
• 13:14 logmsgbot: faidon Synchronized wmf-config/InitialiseSettings.php: revert bits.wm.org change (duration: 00m 12s)
• 13:14 logmsgbot: faidon Synchronized wmf-config/CommonSettings.php: revert bits.wm.org change (duration: 00m 12s)
• 13:03 logmsgbot: faidon Synchronized wmf-config/CommonSettings.php: Switch assets back to bits.wikimedia.org (duration: 00m 15s)
• 13:03 logmsgbot: faidon Synchronized wmf-config/InitialiseSettings.php: Switch assets back to bits.wikimedia.org (duration: 00m 14s)
• 11:49 godog: deploy librenms 2fa805ff
• 09:39 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-kaz_0.1.0~r60155-1
• 09:39 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-dan-nor_1.0.0~r48173-1
• 05:14 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 8 05:13:23 UTC 2015 (duration 13m 22s)
• 04:16 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: I4c70ce4d0: Fix wikiname: roa-rupwiki -> roa_rupwiki (duration: 00m 12s)
• 03:33 logmsgbot: legoktm Synchronized w/static/images/project-logos/wikimania2015wiki.png: Use png for wikimania2015wiki logo (duration: 00m 12s)
• 02:49 logmsgbot: LocalisationUpdate completed (1.26wmf5) at 2015-05-08 02:48:15+00:00
• 02:45 logmsgbot: l10nupdate Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 05m 47s)
• 02:29 logmsgbot: LocalisationUpdate completed (1.26wmf4) at 2015-05-08 02:28:07+00:00
• 02:24 logmsgbot: l10nupdate Synchronized php-1.26wmf4/cache/l10n: (no message) (duration: 06m 06s)
• 00:00 logmsgbot: rmoen Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 11s)

May 7

• 23:54 logmsgbot: rmoen Synchronized php-1.26wmf4/extensions/VisualEditor/: Update VE with Cherry-picks (duration: 00m 12s)
• 23:51 logmsgbot: rmoen Synchronized php-1.26wmf5/extensions/VisualEditor/: Update VE for cherry-picks (duration: 00m 11s)
• 23:41 logmsgbot: rmoen Synchronized php-1.26wmf4/extensions/Flow/: Bump flow with cherry-picks (duration: 00m 13s)
• 23:39 logmsgbot: rmoen Synchronized php-1.26wmf5/extensions/Flow: Bump Flow with cherry-picks (duration: 00m 14s)
• 23:31 logmsgbot: rmoen Synchronized php-1.26wmf4/extensions/Gather: Update Gather with cherry-picks (duration: 00m 14s)
• 23:20 logmsgbot: rmoen Synchronized php-1.26wmf5/extensions/Gather/: Update Gather with Cherry-picks (duration: 00m 15s)
• 22:58 andrewbogott: restarting all instances on labvirt1008, crossing fingers
• 22:38 andrewbogott: rebooting labvirt1008, running dist-upgrade, rebooting again
• 21:29 awight: updated payments from 3ab89e2b14eb449f7ceddf2325493d6235395ecd to f97f8f99268974cfdb0182f178955bd627137842
• 21:25 gwicke: deployed RESTBase 6043e3ada (v0.6.2)
• 21:01 apergos: dumps are interrupted on snapshot1004 while I do a manual run for testing/debugging purposes. please let it run and don't start any other processes on the box, thanks
• 20:53 bd808: Updated kibana to bb9fcf6 (Merge remote-tracking branch 'upstream/kibana3')
• 20:36 legoktm: renaming users with invalid usernames (https://phabricator.wikimedia.org/T5507)
• 20:18 logmsgbot: ori Synchronized wmf-config: I3846e34ed, I1fcb3f17d, I8c9a6a567, I1a73c83f7, and Iacbd92931: serve optimized, cacheable logos from /static (duration: 00m 19s)
• 20:14 bd808: updated scap to 5d681af (Better handling for php lint checks)
• 20:14 bd808: Trebuchet checkout failed for scap/scap on mw1222.eqiad.wmnet, mw1113.eqiad.wmnet, mw1104.eqiad.wmnet
• 20:13 bd808: Trebuchet fetch for scap/scap failed on mw1222.eqiad.wmnet
• 19:17 logmsgbot: legoktm Synchronized php-1.26wmf4/extensions/CentralAuth/: https://gerrit.wikimedia.org/r/209538 and https://gerrit.wikimedia.org/r/209539 (duration: 00m 16s)
• 19:16 logmsgbot: legoktm Synchronized php-1.26wmf5/extensions/CentralAuth/: https://gerrit.wikimedia.org/r/209538 and https://gerrit.wikimedia.org/r/209539 (duration: 00m 16s)
• 16:56 bd808: sync-common on snapshot1004 finished in 12:36
• 16:49 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: Enable shortURL on saprojects gerrit:201216 (duration: 00m 14s)
• 16:43 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: Enable ShortUrl on newiki gerrit:206736 (duration: 00m 21s)
• 16:37 bd808: Running sync-common manually on snapshot1004.eqiad.wmnet
• 16:36 thcipriani: create shorturl table in sawiki, sawikisource, sawikiquote, sawiktionary, sawikibooks
• 16:36 logmsgbot: kartik Finished scap: Update ContentTranslation (duration: 16m 21s)
• 16:23 thcipriani: populateShortUrlTable on newiki
• 16:20 thcipriani: creating newiki shorturl table
• 16:19 logmsgbot: kartik Started scap: Update ContentTranslation
• 15:48 logmsgbot: thcipriani Synchronized php-1.26wmf4/extensions/CentralAuth/includes/LocalRenameJob/LocalRenameUserJob.php: Update CentralAuth gerrit:209493 (duration: 00m 21s)
• 15:34 logmsgbot: thcipriani Synchronized php-1.26wmf5/extensions/CentralAuth/includes/LocalRenameJob/LocalRenameUserJob.php: Update CentralAuth gerrit:209492 (duration: 00m 17s)
• 15:27 springle: db connection EINTR noise in logs, see T98489
• 15:16 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: CX enable content translations gerrit:209207 (duration: 00m 12s)
• 14:39 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1019 (duration: 00m 14s)
• 13:55 moritzm: uploaded to apt.wikimedia.org jessie-wikimedia: linux-meta_1.1
• 13:02 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-tat_0.1.0~r57462-1
• 13:02 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-pt-gl_0.9.2~r57551-1
• 13:02 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-oc-es_1.0.6~r60161-1
• 13:02 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-oc-ca_1.0.6~r60158-1
• 13:02 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-fr-es_0.9.2~r27040-1
• 13:02 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-eus_0.1.0-1
• 13:02 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-eu-es_0.3.3~r56159-1
• 13:02 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-eu-en_0.3.1~r60155-1
• 13:01 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-es-gl_1.0.8~r57542-1
• 13:01 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-es-ast_1.1.0~r60158-1
• 13:01 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-es-an_0.3.0~r60158-1
• 13:01 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-en-gl_0.5.2~r57551-1
• 13:01 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-dan_0.1.0-1
• 12:30 bblack: rebooting cp1070
• 12:26 godog: bounce uwsgi on graphite1001
• 12:25 godog: bounce uwsgi on graphite1001
• 10:26 godog: bounce uwsgi on graphite1001
• 10:01 mark: Decreased labstore1001 md125 sync_speed_min from 80000 to 40000
• 09:35 mark: Increased /sys/block/md125/md/sync_speed_min from 4000 to 40000
• 09:29 mark: Increased /sys/block/md125/md/sync_speed_min from 1000 to 4000
• 05:40 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 7 05:39:36 UTC 2015 (duration 39m 35s)
• 03:03 logmsgbot: LocalisationUpdate completed (1.26wmf5) at 2015-05-07 03:02:50+00:00
• 02:59 logmsgbot: l10nupdate Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 08m 35s)
• 02:36 logmsgbot: LocalisationUpdate completed (1.26wmf4) at 2015-05-07 02:35:43+00:00
• 02:35 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1054 in s2, warm up (duration: 01m 09s)
• 02:29 logmsgbot: l10nupdate Synchronized php-1.26wmf4/cache/l10n: (no message) (duration: 09m 27s)
• 02:14 logmsgbot: krenair Synchronized wmf-config: update interwiki.cdb, T98429 (duration: 00m 24s)
• 01:50 bblack: we're still hitting cap on Zayo as of shortly-ago in graphs and seeing smokeping loss, moved california to eqiad
• 00:13 mutante: running refreshLinks.php for s2
• 00:11 logmsgbot: catrope Synchronized php-1.26wmf4/extensions/MobileFrontend/: SWAT (duration: 00m 42s)
• 00:11 gwicke: deployed RESTBase 8865b9c48

May 6

• 23:43 logmsgbot: catrope Synchronized php-1.26wmf5/extensions/VisualEditor: SWAT (duration: 00m 18s)
• 23:43 logmsgbot: catrope Synchronized php-1.26wmf5/extensions/MobileFrontend: SWAT (duration: 00m 34s)
• 23:19 RoanKattouw: Running populateShortUrl.phg on knwiki
• 23:16 RoanKattouw: Running namespaceDupes.php on tewikiquote
• 23:15 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings.php: SWAT (duration: 00m 17s)
• 23:12 RoanKattouw: Created shorturls table on knwiki
• 20:39 logmsgbot: twentyafterfour Purged l10n cache for 1.26wmf3
• 20:37 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.26wmf5
• 20:32 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.26wmf4
• 20:29 apergos: salt upgraded to 2014.7.5 on all precise/trusty/jessie hosts in production except for: labcontrol2001, tin, virt1000 (deferred) and dysprosium/labvirt1005/labstore1002 (down)
• 20:15 logmsgbot: twentyafterfour Synchronized php-1.26wmf5/extensions/MobileFrontend/javascripts/modules/search/init.js: Temporarily disable MobileWebSearch logging (duration: 00m 36s)
• 20:14 twentyafterfour: ignore all rumors of scap failures, the scaps were successful, with the exception of snapshot1004.eqiad.wmnet which hangs every time
• 20:14 logmsgbot: twentyafterfour Synchronized php-1.26wmf4/extensions/MobileFrontend/javascripts/modules/search/init.js: Temporarily disable MobileWebSearch logging (duration: 00m 37s)
• 20:12 logmsgbot: twentyafterfour scap failed: OSError [Errno 2] No such file or directory: '/var/lock/scap' (duration: 27m 49s)
• 19:44 logmsgbot: twentyafterfour Started scap: testwiki to php-1.26wmf5 and rebuild l10n cache
• 18:39 mutante: restarting apache on rhodium
• 18:34 bblack: rebooting cp3030
• 18:14 andrewbogott: restarted gmetad on uranium
• 17:41 andrewbogott: powering down virt1005 and virt1006
• 17:38 andrewbogott: depuppeting and decommissioning virt1005 and virt1006
• 17:24 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable Wikibase usage tracking on enwikivoyage, fawiki and hewiki (duration: 00m 18s)
• 17:03 jgage: hadoop active namenode switched back to analytics1001 after rack C4 switch replacement
• 16:43 apergos: done with all trusty salt updates in pro except for labcontrol1002 (?), doing jessie now in very tiny batches, it's being trouble
• 15:29 bd808: Stashed uncommitted change to scap on tin that disabled php opening tag check for sync-file
• 15:27 bd808: Updated scap to 57036d2 (Update statsd events)
• 15:27 bd808: trebuchet checkout for scap/scap failed for mw1113.eqiad.wmnet, mw1222.eqiad.wmnet, mw1104.eqiad.wmnet
• 15:25 bd808: trebuchet fetch for scap/scap failed on mw1222.eqiad.wmnet
• 15:04 logmsgbot: bd808 Synchronized wmf-config/InitialiseSettings.php: Send group0 + group1 MediaWiki events to logstash 209170 (duration: 00m 16s)
• 14:32 cmjohnson1: shutting down db1054 for maintenance
• 14:22 _joe_: depooling the HHVM imagescaler
• 14:20 Nemo_bis: phabricator went down again for some minutes, seems ok now?
• 14:17 _joe_: pooling the HHVM imagescalers to test if the issues are solved now.
• 14:15 andrewbogott: rebooting labvirt1009 one last time
• 13:53 _joe_: upgrading the hhvm imagescaler (mw1152) to HHVM 3.6.1
• 13:47 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1021 in s2, warm up (duration: 00m 27s)
• 13:42 apergos: all precise hosts are upgraded to salt except for tin and virt1000; in the middle of trusty updates now, in batches
• 13:38 _joe_: uploading HHVM 3.6.1 and all the related extensions to apt.wikimedia.org
• 13:01 paravoid: replacing asw-c4-eqiad (T93730)
• 12:45 logmsgbot: krenair Synchronized php-1.26wmf4/extensions/SemanticMediaWiki/specials/QueryPages/SMW_QueryPage.php: https://gerrit.wikimedia.org/r/#/c/209212/ (duration: 00m 21s)
• 08:12 logmsgbot: legoktm Synchronized wmf-config/CommonSettings-labs.php: no-op (duration: 00m 24s)
• 07:20 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I019944f42: Change EventLogging endpoint to /beacon/event (duration: 00m 14s)
• 06:51 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed May 6 06:50:27 UTC 2015 (duration 50m 26s)
• 03:14 logmsgbot: LocalisationUpdate completed (1.26wmf4) at 2015-05-06 03:13:28+00:00
• 03:09 logmsgbot: l10nupdate Synchronized php-1.26wmf4/cache/l10n: (no message) (duration: 08m 46s)
• 02:46 logmsgbot: LocalisationUpdate completed (1.26wmf3) at 2015-05-06 02:45:26+00:00
• 02:36 logmsgbot: l10nupdate Synchronized php-1.26wmf3/cache/l10n: (no message) (duration: 10m 46s)
• 02:27 springle: xtrabackup clone db1060 to db1021
• 02:04 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I83ad6d060: Remove wmgUseBits setting, now that the migration is complete (duration: 00m 18s)
• 02:02 logmsgbot: aude Synchronized php-1.26wmf4/extensions/Wikidata: Fix Wikibase api error output bug - update submoduled (duration: 00m 28s)
• 01:59 logmsgbot: aude Synchronized php-1.26wmf4/extensions/Wikidata: Fix Wikibase api error output bug (duration: 01m 08s)
• 01:52 logmsgbot: ori Synchronized multiversion/MWWikiversions.php: Ib08e36901: MWWikiversions::readDbListFile: allow single-line ("#" or "//") comments (duration: 00m 18s)
• 01:40 springle: upgrade db1021 trusty
• 00:51 springle: schema change running T95179 wikidata, bit unusual, dropping a not-null field
• 00:46 logmsgbot: bd808 Synchronized wmf-config/CommonSettings.php: Add AffCom user group application contact page on meta 207332 (duration: 00m 20s)
• 00:45 logmsgbot: bd808 Synchronized docroot/noc/createTxtFileSymlinks.sh: Add AffCom user group application contact page on meta 207332 (duration: 00m 17s)
• 00:45 logmsgbot: bd808 Synchronized docroot/noc/conf/AffComContactPages.php.txt: Add AffCom user group application contact page on meta 207332 (duration: 00m 15s)
• 00:44 logmsgbot: bd808 Synchronized wmf-config/AffComContactPages.php: Add AffCom user group application contact page on meta 207332 (duration: 00m 33s)
• 00:15 logmsgbot: catrope Synchronized php-1.26wmf3/extensions/Flow: SWAT (duration: 00m 23s)
• 00:15 logmsgbot: catrope Synchronized php-1.26wmf3/extensions/WikiEditor: SWAT (duration: 00m 33s)
• 00:13 bd808: Aborted sync-common on snapshot1004; host is starved for RAM and using swap heavily
• 00:06 logmsgbot: catrope Synchronized php-1.26wmf4/extensions/CirrusSearch: SWAT (duration: 00m 28s)
• 00:06 logmsgbot: catrope Synchronized php-1.26wmf4/extensions/Flow: SWAT (duration: 00m 52s)
• 00:04 logmsgbot: catrope Synchronized php-1.26wmf4/extensions/WikiEditor: SWAT (duration: 00m 42s)

May 5

• 23:57 bd808: aborted and restarted sync-common on snapshot1004.eqiad.wmnet manually after waiting 24 minutes with no progress
• 23:49 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings.php: Use Wiki.svg for wikimania2015wiki logo (duration: 00m 19s)
• 23:47 jgage: switched hadoop active namenode from analytics1001 to analytics1002 for rack C4 switch replacement tomorrow morning (T93730)
• 23:39 logmsgbot: rmoen Finished scap: Updates for Gather and MobileFrontend (duration: 41m 11s)
• 23:33 bd808: running sync-common on snapshot1004.eqiad.wmnet manually after it was aborted in scap by rmoen
• 23:30 bd808: snapshot1004.eqiad.wmnet hanging scap yet again
• 23:23 mutante: deleted 8G recurring_blocked.tsv from lutetium
• 22:58 logmsgbot: rmoen Started scap: Updates for Gather and MobileFrontend
• 22:54 logmsgbot: rmoen Synchronized php-1.26wmf3/extensions/Gather/: Update Gather to master (duration: 00m 36s)
• 22:53 logmsgbot: rmoen Synchronized php-1.26wmf3/extensions/MobileFrontend/: Update MobileFrontend (duration: 00m 31s)
• 22:52 logmsgbot: rmoen Synchronized php-1.26wmf4/extensions/Gather/: Update Gather to master (duration: 00m 25s)
• 22:52 mutante: gzip lutetium-slow.log on lutetium to save disk space
• 22:52 logmsgbot: rmoen Synchronized php-1.26wmf4/extensions/MobileFrontend/: Update MobileFrontend (duration: 00m 39s)
• 22:23 mutante: apt-get clean on lutetium to free disk space
• 19:53 twentyafterfour: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: Group1 wikis to 1.26wmf4 (actual time 18:12 UTC)
• 19:44 logmsgbot: aude Synchronized php-1.26wmf4/extensions/Wikidata: Fix usage tracking issue on Wikidata - with submodule update (duration: 00m 33s)
• 19:41 logmsgbot: aude Synchronized php-1.26wmf4/extensions/Wikidata: Fix usage tracking issue on Wikidata (duration: 00m 40s)
• 19:35 bblack: rebooting cp3030 ...
• 19:23 yuvipanda: disabled puppet on zookeeper hosts
• 18:49 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I5978a3910: Update $wgULSFontRepositoryBasePath for post-bits world (duration: 00m 18s)
• 18:43 logmsgbot: ori Synchronized wmf-config: Ia98fc4c5d: wmgUseBits: false for enwiki (duration: 00m 17s)
• 18:33 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: I2ee277293: wmgUseBits: false for all but enwiki (duration: 00m 13s)
• 17:50 logmsgbot: yurik Synchronized wmf-config/InitialiseSettings.php: Enable graph extension on all wikis except wikidata (duration: 00m 19s)
• 17:43 logmsgbot: yurik Synchronized php-1.26wmf3/extensions/Graph: Cherrypicked Graph ext 209004 (duration: 00m 16s)
• 17:42 logmsgbot: yurik Synchronized php-1.26wmf4/extensions/Graph: Cherrypicked Graph ext 209004 (duration: 00m 20s)
• 17:00 logmsgbot: yurik Synchronized wmf-config/CommonSettings.php: Enable graphoid noscript fallback for graph ext (duration: 00m 20s)
• 16:50 yurik_: deployed latest graphoid 0.1.3 service
• 15:16 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: Add medialib.naturalis.nl to wgCopyUploadsDomains gerrit:208634 (duration: 00m 26s)
• 14:07 godog: shut fluorine to replace sdb
• 13:13 akosiaris: restarted apache2 on palladium
• 13:04 Tim: updating voter list for the FDC election for T97924
• 08:47 paravoid: repooling ulsfo
• 07:59 godog: test reboot fluorine with new disk
• 05:51 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue May 5 05:50:01 UTC 2015 (duration 50m 0s)
• 05:07 logmsgbot: tstarling Synchronized php-1.26wmf3/extensions/SecurePoll/cli/wm-scripts/bv2015/voterList.php: (no message) (duration: 00m 16s)
• 04:43 logmsgbot: tstarling Synchronized php-1.26wmf3/extensions/SecurePoll/cli/wm-scripts/bv2015/voterList.php: (no message) (duration: 00m 19s)
• 02:59 logmsgbot: LocalisationUpdate completed (1.26wmf4) at 2015-05-05 02:57:54+00:00
• 02:54 logmsgbot: l10nupdate Synchronized php-1.26wmf4/cache/l10n: (no message) (duration: 07m 06s)
• 02:31 logmsgbot: LocalisationUpdate completed (1.26wmf3) at 2015-05-05 02:30:45+00:00
• 02:26 logmsgbot: l10nupdate Synchronized php-1.26wmf3/cache/l10n: (no message) (duration: 08m 20s)
• 01:41 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1021, move s5 api to db1049 (duration: 00m 15s)
• 01:20 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1070, warm up (duration: 00m 19s)
• 00:32 yuvipanda: restarted hhvm on mw1197
• 00:24 logmsgbot: aude Synchronized wmf-config/Wikibase.php: Enable Wikibase subscription tracking (duration: 00m 12s)

May 4

• 23:59 logmsgbot: catrope Finished scap: (no message) (duration: 24m 34s)
• 23:34 logmsgbot: catrope Started scap: (no message)
• 23:15 logmsgbot: catrope Synchronized php-1.26wmf3/extensions/MassMessage/: SWAT (duration: 00m 12s)
• 23:14 logmsgbot: catrope Synchronized php-1.26wmf4/extensions/MassMessage/: SWAT (duration: 00m 12s)
• 23:14 logmsgbot: catrope Synchronized php-1.26wmf4/extensions/VisualEditor/: SWAT (duration: 00m 12s)
• 23:13 logmsgbot: catrope Synchronized php-1.26wmf4/includes/skins/SkinTemplate.php: SWAT (duration: 00m 11s)
• 22:37 Krenair: silver: apache2ctl restart for T98084
• 22:26 Tim: on terbium: running voterList.php again, with corrected edit counts
• 21:55 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: Id56e33263: wmgUseBits: false for ru and eswiki (duration: 00m 12s)
• 21:40 logmsgbot: bd808 Finished scap: Update 1.26wmf4 ContactPage and WikimediaMessages for AffCom contact form (duration: 22m 11s)
• 21:34 paravoid: cr{1,2}-{eqiad,ulsfo}: swapping metrics for ulsfo's transport links
• 21:18 logmsgbot: bd808 Started scap: Update 1.26wmf4 ContactPage and WikimediaMessages for AffCom contact form
• 21:03 Coren: checking raid consistency from labstore1002
• 21:03 ottomata: rebooting analytics1037
• 20:27 Coren: Starting NFS server switch - graceful labstore1001 shutdown.
• 20:11 gwicke: deployed restbase v0.6.0 / 76583a07
• 19:56 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: I62dffd271: wmgUseBits: false for nl and dewiki (duration: 00m 11s)
• 19:24 logmsgbot: ori Synchronized w/5xx.php: (no message) (duration: 00m 14s)
• 19:12 awight: update crm from 514e7ea41acd14e1565b31b76621ea840d209e07 to 2a2336655737a2cd1d3cc24624d1e8475e4cf039
• 19:12 logmsgbot: ori Synchronized multiversion: I2d93ede75: Remove FormatJson from mediawiki-config (duration: 00m 13s)
• 18:51 logmsgbot: ori Synchronized multiversion/FormatJson.php: Ice8f1796c: Update FormatJson to 532337e6ff from mediawiki/core (duration: 00m 12s)
• 18:44 cscott: updated Parsoid to version b53a7272
• 18:26 logmsgbot: ori Synchronized wmf-config: I81df3a614, I02b06f8e2, I366561a0f: Use MWWikiversions::readDbListFile to read dblist files; Allow computed dblist expressions; Add group1.dblist (duration: 00m 14s)
• 17:53 legoktm: running delete-wmf-tags (https://phabricator.wikimedia.org/P531) on all extension repos
• 16:58 andrewbogott: reimaging/renaming virt1011 -> labvirt1007
• 15:40 logmsgbot: thcipriani Synchronized php-1.26wmf4/extensions/ContentTranslation: Update ContentTranslation to 0bd91b6 gerrit:208607 (duration: 00m 30s)
• 15:32 logmsgbot: thcipriani Synchronized php-1.26wmf3/extensions/ContentTranslation: Sync-dir for ContentTranslation to 6f81619 gerrit:208605 (duration: 00m 18s)
• 15:23 logmsgbot: thcipriani Synchronized php-1.26wmf3/extensions/ContentTranslation/modules/tools/ext.cx.tools.formatter.js: Update ContentTranslation to 6f81619 gerrit:208605 (duration: 00m 25s)
• 15:17 ottomata: starting upgrade of Analytics Cluster to CDH 5.4: https://phabricator.wikimedia.org/T97453
• 15:05 andrewbogott: halting virt1011 pending its rename to labvirt1007
• 14:51 godog: halt fluorine to fix console and swap sda
• 14:50 paravoid: draining ulsfo, network troubles (internal network packet loss)
• 13:49 paravoid: draining all traffic from the Giglinx/Zayo link to ulsfo
• 05:56 Tim: on terbium: running populateEditCount-fixup.php on all wikis
• 05:53 logmsgbot: tstarling Synchronized php-1.26wmf4/extensions/SecurePoll: Iae874c0403a8362929362ca645f4aca18feb0269 (duration: 00m 19s)
• 05:52 logmsgbot: tstarling Synchronized php-1.26wmf3/extensions/SecurePoll: Iae874c0403a8362929362ca645f4aca18feb0269 (duration: 00m 22s)
• 05:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon May 4 05:35:29 UTC 2015 (duration 35m 28s)
• 02:49 logmsgbot: LocalisationUpdate completed (1.26wmf4) at 2015-05-04 02:48:16+00:00
• 02:44 logmsgbot: l10nupdate Synchronized php-1.26wmf4/cache/l10n: (no message) (duration: 07m 33s)
• 02:27 logmsgbot: LocalisationUpdate completed (1.26wmf3) at 2015-05-04 02:26:00+00:00
• 02:22 logmsgbot: l10nupdate Synchronized php-1.26wmf3/cache/l10n: (no message) (duration: 08m 58s)
• 01:13 bd808: Started logstash cluster relocating indices off of logstash100[1-3] to logstash100[4-6]

May 3

• 19:28 yuvipanda: chown www-data: /var/log/mediawiki/refreshLinks/s3@3.log and s2@2.log for Reedy
• 16:23 logmsgbot: hoo Synchronized wmf-config/: Re-enable global renames (duration: 00m 12s)
• 15:17 _joe_: restarted jobchron, not jobcron, this time for real
• 14:37 bblack: dewiki jobqueue:*:rootjob wipe complete
• 14:37 bblack: enwiki + commonswiki jobqueue:*:rootjob wipe complete
• 14:19 bblack: deleting :rootjob: entries for enwiki from redis too
• 14:16 bblack: deleting :rootjob: entries for commonswiki from redis
• 13:54 _joe_: restarting jobcron on the jobrunners
• 13:27 logmsgbot: hoo Synchronized wmf-config/: Temporary disable global renames (duration: 00m 16s)
• 12:47 _joe_: restarting redis server on rdb1001, lagging on the most basic queries
• 12:38 _joe_: deploying I969fe8d329c1bcbb919a54cb225200ba0e006a03 to the jobrunners trying to make them work again
• 05:14 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun May 3 05:13:13 UTC 2015 (duration 13m 12s)
• 04:28 springle: xtrabackup clone db1049 to db1070
• 04:01 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1070 (duration: 00m 16s)
• 02:48 logmsgbot: LocalisationUpdate completed (1.26wmf4) at 2015-05-03 02:47:30+00:00
• 02:47 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1068, warm up (duration: 00m 15s)
• 02:44 logmsgbot: l10nupdate Synchronized php-1.26wmf4/cache/l10n: (no message) (duration: 07m 11s)
• 02:27 logmsgbot: LocalisationUpdate completed (1.26wmf3) at 2015-05-03 02:26:02+00:00
• 02:22 logmsgbot: l10nupdate Synchronized php-1.26wmf3/cache/l10n: (no message) (duration: 08m 11s)

May 2

• 22:16 ori: Deployed change I3bc87f3a5 to fix UBN! bug T97912. Bug was affecting ability to translate messages needed for running upcoming board election.
• 22:16 logmsgbot: ori Synchronized php-1.26wmf4/extensions/Translate/api/ApiQueryMessageGroups.php: I3bc87f3a5: ApiQueryMessageGroups: mark '_canchange' and '_name' as non-API-metadata (duration: 00m 30s)
• 22:09 logmsgbot: ori Synchronized php-1.26wmf3/extensions/Translate/api/ApiQueryMessageGroups.php: I3bc87f3a5: ApiQueryMessageGroups: mark '_canchange' and '_name' as non-API-metadata (duration: 00m 31s)
• 20:25 windowcat: Updated jobrunners to c95d565e242e6fa3706c088ddab1cc6f716408e1
• 19:31 springle: xtrabackup clone db2048, db2049, db2050, db2051, db2052, db2053, db2054 from codfw masters
• 19:09 springle: upgrade db1068 trusty, xtrabackup clone from db1056
• 19:02 ottomata: resinstalling analytics1004 and analytics1010 as trusty
• 06:08 yuvipanda: signed puppet certs manually on virt1000
• 05:19 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May 2 05:18:29 UTC 2015 (duration 18m 28s)
• 03:24 ori: Granted self admin rights on metawiki temporarily to debug a CentralNotice issue.
• 02:53 logmsgbot: LocalisationUpdate completed (1.26wmf4) at 2015-05-02 02:52:36+00:00
• 02:48 logmsgbot: l10nupdate Synchronized php-1.26wmf4/cache/l10n: (no message) (duration: 07m 01s)
• 02:32 logmsgbot: catrope Synchronized php-1.26wmf3/extensions/WikiEditor: Fix data gathering bug (duration: 00m 25s)
• 02:32 logmsgbot: LocalisationUpdate completed (1.26wmf3) at 2015-05-02 02:31:00+00:00
• 02:27 logmsgbot: l10nupdate Synchronized php-1.26wmf3/cache/l10n: (no message) (duration: 08m 11s)
• 02:15 logmsgbot: catrope Synchronized php-1.26wmf4/extensions/WikiEditor: Fix data gathering bug (duration: 00m 15s)
• 00:02 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 16s)

May 1

• 23:53 logmsgbot: aaron Synchronized php-1.26wmf4/includes/media/DjVu.php: caa2efc0e76c2ba849d465006600d131dc2f78b5 (duration: 00m 21s)
• 23:52 logmsgbot: aaron Synchronized php-1.26wmf3/includes/media/DjVu.php: 6cdb23c5d662151a2b578c2acc8823bc975fc22a (duration: 00m 15s)
• 23:40 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: I02e28db61: Update apple-touch to use static (duration: 00m 23s)
• 21:08 matt_flaschen: Ran FlowUpdateWorkflowPageId.php for all production Flow wikis for https://phabricator.wikimedia.org/T96888
• 20:37 logmsgbot: andyrussg Synchronized php-1.26wmf4/extensions/EducationProgram/: Update EducationProgram (duration: 00m 21s)
• 20:01 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: less realm stuff (duration: 00m 17s)
• 20:00 logmsgbot: andyrussg Synchronized php-1.26wmf3/extensions/EducationProgram/: Update EducatiDonProgram (duration: 00m 30s)
• 18:54 logmsgbot: legoktm Synchronized wikiversions-labs.json: https://gerrit.wikimedia.org/r/#/c/208170/ no-op (duration: 00m 25s)
• 18:53 logmsgbot: legoktm Synchronized all-labs.dblist: https://gerrit.wikimedia.org/r/#/c/208170/ no-op (duration: 00m 18s)
• 18:11 logmsgbot: legoktm Synchronized all-labs.dblist: https://gerrit.wikimedia.org/r/208154 - no-op (duration: 00m 19s)
• 15:58 logmsgbot: anomie Synchronized php-1.26wmf3/includes/: Deploy gerrit:208109 to reduce the complaining about the new feature (duration: 00m 28s)
• 15:50 logmsgbot: anomie Synchronized php-1.26wmf4/includes/: Deploy gerrit:208109 to reduce the complaining about the new feature (duration: 00m 24s)
• 15:29 gwicke: finished restarting cassandra nodes on restbase100*.eqiad
• 15:21 ottomata: doing java security update on kafka brokers, doing rolling restarts
• 14:50 gwicke: slowly restarting restbase100*.eqiad to apply new gen size change
• 10:47 godog: bounce apache2 on strontium
• 10:47 godog: bounce apache2 on palladium, mod_passenger died
• 05:45 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 1 05:44:23 UTC 2015 (duration 44m 22s)
• 03:05 logmsgbot: LocalisationUpdate completed (1.26wmf4) at 2015-05-01 03:04:21+00:00
• 03:01 logmsgbot: l10nupdate Synchronized php-1.26wmf4/cache/l10n: (no message) (duration: 06m 45s)
• 02:38 logmsgbot: LocalisationUpdate completed (1.26wmf3) at 2015-05-01 02:37:20+00:00
• 02:31 logmsgbot: l10nupdate Synchronized php-1.26wmf3/cache/l10n: (no message) (duration: 09m 46s)
• 00:18 logmsgbot: catrope Synchronized php-1.26wmf3/extensions/PageTriage/: SWAT (duration: 00m 30s)
• 00:13 logmsgbot: ori Synchronized wmf-config: Iae2e55a11: wmgUseBits: false for itwiki (duration: 00m 19s)

April 30

• 23:59 logmsgbot: catrope Synchronized php-1.26wmf4/extensions/PageTriage/: SWAT (duration: 00m 30s)
• 23:39 logmsgbot: catrope Synchronized php-1.26wmf4/extensions/CentralAuth: SWAT (duration: 00m 15s)
• 23:39 logmsgbot: catrope Synchronized php-1.26wmf4/extensions/Flow: SWAT (duration: 00m 51s)
• 23:38 logmsgbot: catrope Synchronized php-1.26wmf4/extensions/MobileFrontend: SWAT (duration: 00m 58s)
• 23:35 logmsgbot: catrope Synchronized php-1.26wmf4/includes/skins/SkinTemplate.php: Add mw-content-ltr/rtl for missing pages (duration: 00m 35s)
• 23:33 logmsgbot: catrope Synchronized php-1.26wmf3/extensions/CentralAuth/: SWAT (duration: 00m 31s)
• 23:32 ori: EventLogging events logged client-side appear not to be making it to eventlog1001.eqiad.wmnet; Ori investigating.
• 23:29 logmsgbot: catrope Synchronized php-1.26wmf3/extensions/MobileFrontend/: SWAT (duration: 01m 43s)
• 23:04 RoanKattouw: Created wikilove tables on hywiki
• 23:03 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings.php: Enable WikiLove on hywiki (duration: 00m 49s)
• 22:48 logmsgbot: mattflaschen Finished scap: Deploy Flow changes to 1.26wmf4 facilitate LQT->Flow conversion (duration: 33m 35s)
• 22:19 awight: payments redeployed, revision for payments-wiki changed... from df8aeb5d1c5f595348f77cb56d3975eca19a65a2 to 3ab89e2b14eb449f7ceddf2325493d6235395ecd
• 22:17 awight: payments rolled back from 3ab89e2b14eb449f7ceddf2325493d6235395ecd to df8aeb5d1c5f595348f77cb56d3975eca19a65a2
• 22:14 logmsgbot: mattflaschen Started scap: Deploy Flow changes to 1.26wmf4 facilitate LQT->Flow conversion
• 22:10 awight: updating payments from df8aeb5d1c5f595348f77cb56d3975eca19a65a2 to 3ab89e2b14eb449f7ceddf2325493d6235395ecd
• 21:46 awight: update payments from 83d09e09178c634ad35dbb684d1c3aebbb709969 to df8aeb5d1c5f595348f77cb56d3975eca19a65a2
• 21:05 bd808: Finally got sync-common to run to completion on snapshot1004; runtime 45 minutes!
• 20:43 legoPanda: renaming <2k users who were missed in the original run (SUL finalization)
• 19:23 awight: enabling Thank You job
• 19:23 awight: updated crm from 59f03df6b689ef443cc7b7e31e6f5b2986bc8bc9 to 514e7ea41acd14e1565b31b76621ea840d209e07
• 19:07 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I93cdc4a2e and I9ee6bec1f: Define $wgAssetsHost based on wmgUseBits; use it to reference standard chrome (duration: 00m 16s)
• 18:46 Coren: rebooting labstore1002 in prevision of switch to make sure it starts up cleanly.
• 18:14 K4-713: disabled Thank You mail send
• 17:41 bd808: sync-common on snapshot1004 failed after 33 minutes with rsync timeout
• 17:04 logmsgbot: demon Synchronized php-1.26wmf3/includes/Setup.php: meh, didn't work (duration: 00m 27s)
• 17:01 logmsgbot: demon Synchronized php-1.26wmf3/includes/Setup.php: trying something (duration: 00m 18s)
• 16:59 bd808: aborted sync-common on snapshot1004.eqiad.wmnet after 15 minutes for inactivity; trying again
• 16:44 bd808: started sync-common on snapshot1004 to fix aborted sync
• 16:42 logmsgbot: kartik Finished scap: Update ContentTranslation (duration: 26m 42s)
• 16:16 logmsgbot: kartik Started scap: Update ContentTranslation
• 15:22 logmsgbot: anomie Synchronized php-1.26wmf3/extensions/EducationProgram/: SWAT: EducationProgram: ApiListStudents: Use XML-friendly tag names gerrit:207778 (duration: 00m 39s)
• 15:12 logmsgbot: anomie Synchronized php-1.26wmf4/extensions/EducationProgram/: SWAT: EducationProgram: ApiListStudents: Use XML-friendly tag names gerrit:207779 (duration: 00m 25s)
• 15:09 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable GeoData at cawikibooks gerrit:199930 (duration: 00m 19s)
• 15:08 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Restrict local uploads on mai.wikipedia gerrit:207725 (duration: 00m 14s)
• 15:05 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Content Translation for Deployment 20150430 gerrit:207472 (duration: 00m 18s)
• 15:03 logmsgbot: anomie Synchronized wmf-config/CommonSettings.php: SWAT: Bump timestamp in 'ValidateExtendedMetadataCache' hook for T97469 gerrit:207769 (duration: 00m 30s)
• 12:27 godog: upgrade statsite on ms-be1*
• 12:25 godog: upgrade statsite on ms-fe1*
• 12:09 hashar: restarting Jenkins https://phabricator.wikimedia.org/T96183
• 10:53 godog: delete old /tmp/ganglia-graph from uranium
• 10:36 godog: upgrade statsite on labmon1001
• 08:16 paravoid: repooling esams, network maintenance is over
• 05:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Apr 30 05:47:26 UTC 2015 (duration 47m 25s)
• 05:15 paravoid: draining esams, planned upsteam network maintenance
• 03:04 logmsgbot: LocalisationUpdate completed (1.26wmf4) at 2015-04-30 03:03:09+00:00
• 03:00 logmsgbot: l10nupdate Synchronized php-1.26wmf4/cache/l10n: (no message) (duration: 07m 09s)
• 02:39 logmsgbot: LocalisationUpdate completed (1.26wmf3) at 2015-04-30 02:38:03+00:00
• 02:31 logmsgbot: l10nupdate Synchronized php-1.26wmf3/cache/l10n: (no message) (duration: 10m 59s)
• 01:18 logmsgbot: legoktm Synchronized php-1.26wmf3/includes/api/ApiOpenSearch.php: Restore B/C for ApiOpenSearch json output if warnings are present (duration: 00m 20s)
• 01:17 logmsgbot: legoktm Synchronized php-1.26wmf4/includes/api/ApiOpenSearch.php: Restore B/C for ApiOpenSearch json output if warnings are present (duration: 00m 30s)

April 29

• 23:58 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings.php: Enable direct RESTbase load on all Wikipedias (duration: 00m 21s)
• 23:57 logmsgbot: catrope Synchronized php-1.26wmf4/extensions/MobileFrontend: SWAT (duration: 00m 33s)
• 23:50 logmsgbot: catrope Synchronized php-1.26wmf3/resources/lib/jquery/jquery.js: Update jQuery to 1.11.3 (duration: 00m 31s)
• 23:49 logmsgbot: catrope Synchronized php-1.26wmf3/extensions/VisualEditor: SWAT (duration: 00m 39s)
• 23:49 logmsgbot: catrope Synchronized php-1.26wmf3/extensions/WikiEditor: SWAT (duration: 00m 23s)
• 23:48 logmsgbot: catrope Synchronized php-1.26wmf3/extensions/Gather: SWAT (duration: 00m 32s)
• 23:24 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings.php: Enable Graph extension on sewikimedia (duration: 00m 21s)
• 23:21 logmsgbot: catrope Synchronized wmf-config/CommonSettings.php: Disable Graph namespace on all wikis except the ones that already have it (duration: 00m 22s)
• 23:20 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings.php: Add wmgUseGraphWithNamespace (duration: 00m 28s)
• 23:18 logmsgbot: catrope Synchronized wmf-config/Wikibase.php: Enable use of subscriptions table on testwikidata (duration: 00m 31s)
• 22:48 logmsgbot: legoktm Synchronized php-1.26wmf3/includes/MovePage.php: MovePage: Move target existence check into isValidMove() - https://gerrit.wikimedia.org/r/#/c/207557/ (duration: 00m 26s)
• 22:48 springle: dbstore1002 /srv/tmp filled up. killed queries, fixed mount point, restarted mysqld
• 21:27 logmsgbot: twentyafterfour Purged l10n cache for 1.26wmf2
• 21:23 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.26wmf4
• 21:20 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.26wmf3
• 21:15 logmsgbot: twentyafterfour Finished scap: testwiki to php-1.26wmf4 and rebuild l10n cache - attempt #2 (duration: 33m 13s)
• 21:04 bd808: load avg on snapshot04 11.11; scap slow waiting on it
• 20:41 logmsgbot: twentyafterfour Started scap: testwiki to php-1.26wmf4 and rebuild l10n cache - attempt #2
• 20:41 logmsgbot: twentyafterfour scap aborted: testwiki to php-1.26wmf4 and rebuild l10n cache (duration: 26m 52s)
• 20:34 bd808: /etc/dsh/group/scap-proxies is borken on tin
• 20:17 subbu: reverted deploy to ebdac59b
• 20:17 subbu: attempted deploy of 45b54f63 (failed)
• 20:14 logmsgbot: twentyafterfour Started scap: testwiki to php-1.26wmf4 and rebuild l10n cache
• 20:03 logmsgbot: ori Synchronized README: testing deploy 2 (duration: 00m 22s)
• 20:03 logmsgbot: ori Synchronized README: testing deploy script (duration: 00m 25s)
• 16:22 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 30s)
• 15:58 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable assigning "accountcreator" for newiki gerrit:206093 (duration: 00m 30s)
• 15:55 logmsgbot: anomie Synchronized wmf-config/abusefilter.php: SWAT: Add abusefilter-modify-restricted right to sysop user group for idwiki gerrit:206080 (duration: 00m 25s)
• 15:53 logmsgbot: anomie Synchronized php-1.26wmf2/extensions/MobileFrontend: SWAT: Ah, git rebasing was rebasing the reverted commits on top of the revert... (duration: 00m 21s)
• 15:51 logmsgbot: anomie Synchronized php-1.26wmf2/extensions/MobileFrontend: SWAT: Resync? (duration: 00m 36s)
• 15:47 logmsgbot: anomie Synchronized php-1.26wmf2/extensions/MobileFrontend: SWAT: Revert previous, broke stuff on wmf2 (duration: 00m 39s)
• 15:44 logmsgbot: anomie Synchronized php-1.26wmf2/extensions/MobileFrontend: SWAT: MobileFrontend: API: "editable" is a legacy boolean, don't convert it gerrit:207403 (duration: 00m 23s)
• 15:43 _joe_: restarting HHVM on mw1132 too, same reason.
• 15:41 logmsgbot: anomie Synchronized php-1.26wmf3/extensions/MobileFrontend: SWAT: MobileFrontend: API: "editable" is a legacy boolean, don't convert it gerrit:207403 (duration: 00m 37s)
• 15:40 _joe_: restarting HHVM on mw1232, stuck on __lll_lock_wait from HPHP::StatCache::refresh ()
• 15:30 logmsgbot: anomie Synchronized php-1.26wmf3/includes/api/ApiResult.php: SWAT: API: ApiResult must validate even when using numeric auto-indexes gerrit:207456 (duration: 00m 26s)
• 15:20 logmsgbot: anomie Synchronized php-1.26wmf3/extensions/Wikidata: SWAT: Update Wikidata - fix change subscriptions script gerrit:207448 (duration: 00m 53s)
• 15:08 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove sampling of api.log gerrit:206865 (duration: 00m 29s)
• 15:05 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Load HTML directly from RESTBase on all wikipedias gerrit:206320 (duration: 00m 17s)
• 13:03 paravoid: disabling netflows on cr1/2-ulsfo
• 07:12 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Apr 29 07:11:38 UTC 2015 (duration 11m 37s)
• 05:28 logmsgbot: tstarling Synchronized php-1.26wmf3/extensions/SecurePoll: (no message) (duration: 00m 13s)
• 03:47 logmsgbot: LocalisationUpdate completed (1.26wmf3) at 2015-04-29 03:46:05+00:00
• 03:40 logmsgbot: l10nupdate Synchronized php-1.26wmf3/cache/l10n: (no message) (duration: 39m 55s)
• 02:48 springle: killed eight stalled commonswiki.transcode transactions on db1040
• 02:45 logmsgbot: LocalisationUpdate completed (1.26wmf2) at 2015-04-29 02:43:54+00:00
• 02:40 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable Wikibase usage tracking on nlwiki and frwikisource (duration: 00m 12s)
• 02:40 logmsgbot: l10nupdate Synchronized php-1.26wmf2/cache/l10n: (no message) (duration: 25m 50s)
• 00:38 springle: xtrabackup clone db2029 to db2047
• 00:38 springle: xtrabackup clone db2028 to db2046
• 00:20 logmsgbot: gwicke Synchronized wmf-config/InitialiseSettings.php: VE: Load HTML directly from RESTBase for enwiki (duration: 00m 22s)
• 00:07 logmsgbot: bd808 Synchronized docroot/noc/createTxtFileSymlinks.sh: Revert of AffCom contact form 207328 (duration: 00m 35s)
• 00:06 logmsgbot: bd808 Synchronized wmf-config/CommonSettings.php: Revert of AffCom contact form 207328 (duration: 00m 19s)

April 28

• 23:57 logmsgbot: bd808 Synchronized docroot/noc/conf/AffComContactPages.php.txt: Add AffCom user group application contact page on meta 207319 (duration: 00m 28s)
• 23:51 logmsgbot: bd808 Synchronized wmf-config/CommonSettings.php: Add AffCom user group application contact page on meta 204205 (duration: 00m 11s)
• 23:50 logmsgbot: bd808 Synchronized docroot/noc/createTxtFileSymlinks.sh: Add AffCom user group application contact page on meta 204205 (duration: 00m 21s)
• 23:48 logmsgbot: bd808 Synchronized wmf-config/AffComContactPages.php: Add AffCom user group application contact page on meta 204205 (duration: 00m 25s)
• 23:35 bd808|deploy: mw2031.codfw.wmnet syncing very slowly for SWAT
• 23:35 logmsgbot: bd808 Synchronized wmf-config/InitialiseSettings.php: Shell bugs 207162 206731 203783 207273 207170 (duration: 01m 12s)
• 23:32 logmsgbot: bd808 Synchronized commonsuploads.dblist: Restrict local uploads on mai.wikipedia 207273 (duration: 00m 32s)
• 23:26 logmsgbot: bd808 Synchronized php-1.26wmf3/extensions/VisualEditor: Update VisualEditor for two icon issues 207299 (duration: 00m 27s)
• 23:06 logmsgbot: hoo Synchronized wmf-config/: Do Wikibase setting overrides for test wikis in Wikibase-production.php (duration: 00m 24s)
• 22:58 logmsgbot: legoktm Synchronized php-1.26wmf3/extensions/EventLogging/includes/ApiJsonSchema.php: https://gerrit.wikimedia.org/r/#/c/207297/ (duration: 00m 15s)
• 22:07 Tim: running bv2015/voterList.php on terbium
• 22:05 logmsgbot: tstarling Synchronized php-1.26wmf2/extensions/SecurePoll: for new voterList.php (duration: 00m 23s)
• 21:32 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: expire old metadata cache entries (duration: 00m 26s)
• 21:30 logmsgbot: rmoen Synchronized php-1.26wmf3/extensions/Gather/: Updating gather (duration: 00m 44s)
• 20:58 logmsgbot: anomie Synchronized php-1.26wmf3/includes/media/FormatMetadata.php: Unbreak API imageinfo with extmetadata (mainly on Commons) (duration: 00m 25s)
• 19:34 twentyafterfour: Deployed patch for T97391
• 19:25 logmsgbot: twentyafterfour Synchronized php-1.26wmf3/thumb.php: (no message) (duration: 00m 19s)
• 19:22 logmsgbot: twentyafterfour Synchronized php-1.26wmf2/thumb.php: (no message) (duration: 00m 33s)
• 19:21 mutante: tmp. stopped icinga-wm because puppetmaster fail spam
• 19:21 mutante: restarting apache on palladium
• 18:47 robh: stopping puppet on carbon - livehacking partman recipe testing
• 18:46 legoktm: force merged User:Js@ruwiki to User:Js@global per global-renamers list
• 18:34 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: Group1 wikis to 1.26wmf3
• 18:32 milimetric: upgraded and restarted Eventlogging on eventlog1001 (now at be1e055)
• 18:22 milimetric: upgraded and restarted Eventlogging on hafnium (now at be1e055)
• 17:54 mutante: tungsten - disable in icinga. scheduled the longest downtime. shutdown -h now (T97274)
• 17:49 mutante: tungsten - revoke puppet cert, delete salt-key, delete from stored configs
• 15:55 logmsgbot: anomie Synchronized php-1.26wmf2/extensions/ContentTranslation: SWAT: Update ContentTranslation gerrit:207092 (duration: 00m 58s)
• 15:45 logmsgbot: anomie Synchronized php-1.26wmf3/extensions/ContentTranslation: SWAT: Update ContentTranslation gerrit:207098 (duration: 00m 46s)
• 15:35 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Content Translation in cs, el, kk and zu gerrit:207048 (duration: 00m 27s)
• 15:31 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Content Translation in cs, el, kk and zu gerrit:207048 (duration: 00m 21s)
• 15:23 logmsgbot: anomie Synchronized php-1.26wmf3/includes/api/ApiQuery.php: SWAT: API: Remove metadata keys from indexpageids output gerrit:206861 (duration: 00m 17s)
• 15:13 logmsgbot: anomie Synchronized php-1.26wmf2/extensions/CentralAuth/: SWAT: CentralAuth: Fix missing "&" in onMakeGlobalVariablesScript signature gerrit:207023 (duration: 00m 24s)
• 15:11 logmsgbot: anomie Synchronized php-1.26wmf3/extensions/CentralAuth/: SWAT: CentralAuth: Fix missing "&" in onMakeGlobalVariablesScript signature gerrit:207021 (duration: 00m 29s)
• 14:58 akosiaris: restart pybal on lvs1003
• 14:51 akosiaris: restarted pybal on lvs1006
• 13:51 ottomata: powercycling analytics1015 after crash
• 12:38 springle: xtrabackup clone db2023 to db2045
• 12:36 springle: xtrabackup clone db2019 to db2044
• 12:34 springle: xtrabackup clone db2018 to db2043
• 05:31 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Apr 28 05:30:47 UTC 2015 (duration 30m 46s)
• 02:52 logmsgbot: LocalisationUpdate completed (1.26wmf3) at 2015-04-28 02:51:34+00:00
• 02:50 ottomata: 'kafka preferred-replica-election'
• 02:48 logmsgbot: l10nupdate Synchronized php-1.26wmf3/cache/l10n: (no message) (duration: 08m 32s)
• 02:29 logmsgbot: LocalisationUpdate completed (1.26wmf2) at 2015-04-28 02:28:11+00:00
• 02:25 bblack: restarted apache2 on palladium - it was throwing infinite 500 errors due to some mod_passenger issue...
• 02:24 logmsgbot: l10nupdate Synchronized php-1.26wmf2/cache/l10n: (no message) (duration: 10m 17s)
• 01:45 bblack: rebooting analytics1013 (not 1016)
• 01:45 bblack: rebooting analytics1016
• 00:37 bblack: rebooting cp3030
• 00:13 logmsgbot: catrope Synchronized php-1.26wmf3/extensions/Flow: SWAT (duration: 00m 28s)
• 00:12 logmsgbot: catrope Synchronized php-1.26wmf2/extensions/Flow: SWAT (duration: 00m 41s)
• 00:11 logmsgbot: catrope Synchronized php-1.26wmf3/extensions/WikimediaEvents/: SWAT (duration: 00m 45s)

April 27

• 23:33 logmsgbot: catrope Synchronized wmf-config/CommonSettings.php: Re-enable same-domain RESTbase entry point for VE (duration: 00m 22s)
• 23:29 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings.php: SWAT (duration: 00m 26s)
• 23:28 logmsgbot: catrope Synchronized wmf-config/flaggedrevs.php: Remove autoreview group on frwikinews (duration: 00m 35s)
• 22:43 mutante: racreset on analytics1016 because no console
• 22:36 ottomata: powercycled analytics1016 after it is unreachable.
• 20:36 subbu: deployed parsoid sha ebdac59b
• 19:47 mutante: apt-get upgrade on iron (incl. apt itself, gnupg, ssl)
• 17:53 mutante: temp stopped icinga-wm
• 17:22 logmsgbot: aaron Synchronized php-1.26wmf2/includes/media/DjVu.php: 40d702b8d2d023d6f701e4aeb082b62b7adf2f0f (duration: 00m 19s)
• 17:20 logmsgbot: aaron Synchronized php-1.26wmf3/includes/media/DjVu.php: b980b0a9457b2f98a502cfe36edfc75300c7952f (duration: 00m 27s)
• 17:05 logmsgbot: aaron Synchronized wmf-config/db-eqiad.php: Lowered innodb_lock_wait_timeout from defaults (duration: 00m 27s)
• 17:03 logmsgbot: aaron Synchronized wmf-config/db-codfw.php: Lowered innodb_lock_wait_timeout from defaults (duration: 00m 22s)
• 17:03 logmsgbot: aaron Synchronized wmf-config/jobqueue-eqiad.php: Set to .1 (duration: 00m 11s)
• 17:02 logmsgbot: aaron Synchronized wmf-config/jobqueue-codfw.php: Set to .1 (duration: 00m 27s)
• 16:24 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: Extended SWAT gerrit:206822 (duration: 00m 26s)
• 16:16 godog: boostrap cassandra on xenon
• 16:02 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT gerrit:201897 (duration: 00m 22s)
• 15:55 logmsgbot: thcipriani Synchronized wmf-config/flaggedrevs.php: SWAT gerrit:199321 (duration: 00m 17s)
• 15:51 logmsgbot: thcipriani Synchronized wmf-config/flaggedrevs.php: SWAT gerrit:206650 no-op whitespace changes (duration: 00m 22s)
• 15:42 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT gerrit:206727 and gerrit:206786 (duration: 00m 16s)
• 15:27 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT gerrit:204467 (duration: 00m 29s)
• 15:20 logmsgbot: thcipriani Synchronized wmf-config/flaggedrevs.php: SWAT gerrit:206647 (duration: 00m 14s)
• 15:09 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT gerrit:206648 (duration: 00m 51s)
• 14:50 godog: upgrade statsite on graphite1001
• 14:04 bblack: puppet disabled on caches while apt upgrades run...
• 13:28 paravoid: upgrading pfw-codfw to newer junos
• 12:31 paravoid: upgrading pfw-eqiad to newer junos
• 08:19 godog: ms-be101[678] object weight to 3000
• 05:09 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Apr 27 05:08:23 UTC 2015 (duration 8m 22s)

April 26

• 23:31 paravoid: draining esams for planned upstream network maintenance (00:00-04:00 UTC)
• 08:16 jgage: ms-be1007 was unresponsive for ~6 hours, "soft lockup" output on console. rebooted.
• 05:29 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Apr 26 05:28:11 UTC 2015 (duration 28m 10s)
• 03:37 ori: Previous sync-file was for: If296f3d3c: Set max_execution_time in CommonSettings.php
• 03:36 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 14s)
• 03:05 jgage: mw2027 rebooted unexpectedly, no clues in syslog. afterward i dist-upgraded, including new kernel.
• 02:56 logmsgbot: LocalisationUpdate completed (1.26wmf3) at 2015-04-26 02:55:00+00:00
• 02:52 logmsgbot: l10nupdate Synchronized php-1.26wmf3/cache/l10n: (no message) (duration: 06m 38s)
• 02:35 logmsgbot: LocalisationUpdate completed (1.26wmf2) at 2015-04-26 02:33:59+00:00
• 02:30 logmsgbot: l10nupdate Synchronized php-1.26wmf2/cache/l10n: (no message) (duration: 07m 39s)

April 25

• 15:26 subbu: deployed parsoid version fca17070 (cherry-pick of d2135c6b on parsoid master)
• 09:57 _joe_: nuked User:Niteshift/MVneu/2015_April_21-30 on commonswiki
• 05:18 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Apr 25 05:17:41 UTC 2015 (duration 17m 40s)
• 04:30 logmsgbot: mattflaschen Synchronized wmf-config/CommonSettings-labs.php: Sync Beta Cluster-only change (for MW UI beta feature) (duration: 00m 16s)
• 04:30 logmsgbot: mattflaschen Synchronized wmf-config/InitialiseSettings-labs.php: Sync Beta Cluster-only change (for MW UI beta feature) (duration: 00m 16s)
• 02:42 logmsgbot: LocalisationUpdate completed (1.26wmf3) at 2015-04-25 02:41:54+00:00
• 02:39 logmsgbot: l10nupdate Synchronized php-1.26wmf3/cache/l10n: (no message) (duration: 05m 56s)
• 02:24 logmsgbot: LocalisationUpdate completed (1.26wmf2) at 2015-04-25 02:23:33+00:00
• 02:20 logmsgbot: l10nupdate Synchronized php-1.26wmf2/cache/l10n: (no message) (duration: 07m 48s)

April 24

• 22:14 logmsgbot: krinkle Synchronized php-1.26wmf2/includes/resourceloader/ResourceLoaderModule.php: Ibedc31659ed (duration: 00m 14s)
• 22:13 logmsgbot: krinkle Synchronized php-1.26wmf3/includes/resourceloader/ResourceLoaderModule.php: Ibedc31659ed (duration: 00m 17s)
• 21:11 ottomata: started hdfs balancer run
• 20:34 ori: Deployed I1fa012ca1: HHVM: Limit wall execution time of FCGI reqs to 290s
• 19:53 logmsgbot: aaron Synchronized wmf-config/db-codfw.php: Removed unused "max threads" stuff (duration: 00m 15s)
• 19:52 subbu: revert parsoid deploy to 3311936a
• 19:52 logmsgbot: aaron Synchronized wmf-config/db-eqiad.php: Removed unused "max threads" stuff (duration: 00m 14s)
• 19:42 logmsgbot: demon Synchronized php-1.26wmf2/extensions/CirrusSearch/includes/Searcher.php: undo debugging (duration: 00m 14s)
• 19:40 logmsgbot: demon Synchronized php-1.26wmf2/extensions/CirrusSearch/includes/Searcher.php: debugging (duration: 00m 17s)
• 18:58 ori: restarted puppetmaster on palladium as well
• 18:56 ori: restarted apache2 on palladium
• 16:06 andrewbogott: dist-upgrade (including kernel upgrade to 3.13.0-49-generic) on labvirt1004, rebooting
• 15:56 logmsgbot: demon Synchronized wmf-config/: logging cleanup, mostly for labs (duration: 00m 21s)
• 15:42 andrewbogott: dist-upgrade (including kernel upgrade to 3.13.0-49-generic) on labvirt1003, rebooting
• 15:08 andrewbogott: dist-upgrade (including kernel upgrade to 3.13.0-49-generic) on labvirt1005, rebooting
• 14:24 andrewbogott: dist-upgrade (including kernel upgrade to 3.13.0-49-generic) on labvirt1006, rebooting
• 10:31 akosiaris: nova migrated a couple of etcd's project VMs
• 09:09 _joe_: parsoid restart done
• 08:59 _joe_: restarting parsoid cluster-wide
• 08:47 ori: deployed parsoid/deploy 8b5de6aba / I4d55f6d50: Bump src to d2135c6b69 for deploy
• 08:09 logmsgbot: tstarling Synchronized php-1.26wmf2/includes/filerepo/file/LocalFile.php: reverting live hack (duration: 00m 16s)
• 06:40 ori: nuked http://commons.wikimedia.org/wiki/User:Niteshift/MVneu/2015_April_21-30
• 05:44 logmsgbot: ori Synchronized php-1.26wmf1/includes/filerepo/file/LocalFile.php: Undo local hack on version that is inactive (1.26wmf1). No-op. (duration: 00m 17s)
• 05:35 ori: restart hhvm on mw1222; locked up in pthread_cond_wait, backtrace: https://phabricator.wikimedia.org/P552
• 05:28 ori: nuked https://commons.wikimedia.org/wiki/User:Niteshift/MVneu/2015_April_21-20
• 05:18 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: $wgExportAllowHistory default false, $wgExportMaxHistory default 1000 -> 10 (duration: 00m 16s)
• 05:05 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Apr 24 05:04:42 UTC 2015 (duration 4m 41s)
• 04:47 logmsgbot: ori Synchronized php-1.26wmf2/includes/filerepo/file/LocalFile.php: Short-circuit LocalFile::loadExtraFromDB in attempt to mitigate outage (duration: 00m 12s)
• 04:42 springle: killing LocalFile::loadExtraFromDB wholesale on s4
• 04:32 logmsgbot: ori Synchronized php-1.26wmf1/includes/filerepo/file/LocalFile.php: Short-circuit LocalFile::loadExtraFromDB in attempt to mitigate outage (duration: 00m 14s)
• 04:25 ori: Did a cluster-wide 'service hhvm restart'.
• 02:48 logmsgbot: LocalisationUpdate completed (1.26wmf3) at 2015-04-24 02:47:12+00:00
• 02:44 logmsgbot: l10nupdate Synchronized php-1.26wmf3/cache/l10n: (no message) (duration: 06m 00s)
• 02:30 logmsgbot: LocalisationUpdate completed (1.26wmf2) at 2015-04-24 02:28:58+00:00
• 02:25 logmsgbot: l10nupdate Synchronized php-1.26wmf2/cache/l10n: (no message) (duration: 06m 35s)
• 00:47 logmsgbot: catrope Synchronized wmf-config/CommonSettings.php: Revert RESTbase URL change (duration: 00m 13s)
• 00:21 logmsgbot: catrope Synchronized php-1.26wmf3/extensions/VisualEditor: Fix RESTbase revid bug (duration: 00m 18s)
• 00:21 logmsgbot: catrope Synchronized php-1.26wmf2/extensions/VisualEditor: Fix RESTbase revid bug (duration: 00m 17s)
• 00:07 logmsgbot: catrope Synchronized wmf-config/CommonSettings.php: Use same-domain entry point for RESTbase (duration: 00m 13s)
• 00:07 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings.php: Temp disable direct RESTbase on enwiki (duration: 00m 17s)

April 23

• 23:54 logmsgbot: rmoen Synchronized php-1.26wmf3/extensions/Flow: Bump flow for cherry-pick (duration: 00m 23s)
• 23:46 logmsgbot: mattflaschen Synchronized wmf-config/CommonSettings.php: Bump Flow cache version to 4.7 (1e28cf78e64eb860d6eade775abae43d11c1dd75) (duration: 00m 16s)
• 23:41 andrewbogott: updating labvirt1002 to 3.13.0-49-generic, dist-upgrade, rebooting
• 23:41 andrewbogott: reverted labvirt1001 to 3.13.0-49-generic because 3.16 wouldn’t mount the fs
• 23:30 logmsgbot: rmoen Synchronized php-1.26wmf2/extensions/MobileFrontend/: Update MobileFrontend to cherry picks (duration: 00m 20s)
• 23:30 logmsgbot: rmoen Synchronized php-1.26wmf3/extensions/MobileFrontend/: Update MobileFrontend to cherry picks (duration: 00m 38s)
• 22:48 andrewbogott: upgrading labvirt1001 to linux-image-3.16.0-34-generic, dist-upgrading, and rebooting
• 22:10 logmsgbot: bd808 Synchronized wmf-config/logging.php: logstash: Fix log level detection (c09014d) (duration: 00m 17s)
• 21:56 ori: Additional (planned) outcome of Ie22658727 and Ice65e7e70: xff log flowing to fluorine, causing bytes-in to climb from ~1.2M/s to ~2.1M/s
• 21:54 ori: Syncing Ie22658727 and Ice65e7e70 (which introduce new InitialiseSettings vars) in one go caused a small burst of 500s (peaking at 500/sec and lasting a few seconds) on four app servers.
• 21:42 logmsgbot: ori Synchronized wmf-config: Ie22658727 and Ice65e7e70: use Monolog to configure logging (duration: 00m 15s)
• 21:04 awight: update payments from 88b9f621bfee1de14a8cdef556a90e5567721754 to 83d09e09178c634ad35dbb684d1c3aebbb709969
• 19:31 mutante: restarting icinga-wm for config change
• 18:05 andrewbogott: rebooting labvirt1006
• 17:51 logmsgbot: kartik Synchronized php-1.26wmf2/extensions/ContentTranslation: (no message) (duration: 00m 15s)
• 17:29 logmsgbot: kartik Finished scap: Update ContentTranslation (duration: 26m 11s)
• 17:28 ori: scap stuck on snapshot1004; not accepting mwdeploy key
• 17:03 logmsgbot: kartik Started scap: Update ContentTranslation
• 16:53 logmsgbot: aaron Synchronized php-1.26wmf2/includes/jobqueue/JobRunner.php: d23777e6832f660984ce4445ab04f98b7ff0d25f (duration: 00m 12s)
• 16:33 andrewbogott: rebooting labvirt1005
• 15:03 logmsgbot: manybubbles Synchronized wmf-config/CommonSettings.php: swat: Re-enable Special:SupportedLanguages (duration: 00m 11s)
• 12:29 godog: investigating icinga UNKNOWN for hhvm queue/threads
• 09:15 godog: restart carbon on graphite1001, replace with carbon-c-relay
• 08:31 godog: restart carbon on labmon1001, replace with carbon-c-relay
• 05:22 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Apr 23 05:21:17 UTC 2015 (duration 21m 16s)
• 02:49 logmsgbot: LocalisationUpdate completed (1.26wmf3) at 2015-04-23 02:48:40+00:00
• 02:46 logmsgbot: l10nupdate Synchronized php-1.26wmf3/cache/l10n: (no message) (duration: 03m 46s)
• 02:28 logmsgbot: LocalisationUpdate completed (1.26wmf2) at 2015-04-23 02:27:39+00:00
• 02:24 logmsgbot: l10nupdate Synchronized php-1.26wmf2/cache/l10n: (no message) (duration: 05m 46s)
• 00:15 logmsgbot: kaldari Synchronized wmf-config/InitialiseSettings.php: Turning on WikiGrok on English Wikipedia (for 2 week test) (duration: 00m 11s)
• 00:07 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/206024 (duration: 00m 14s)
• 00:05 logmsgbot: krenair Synchronized php-1.26wmf2/extensions/ZeroBanner/includes/ZeroSpecialPage.php: https://gerrit.wikimedia.org/r/#/c/206023/ (duration: 00m 13s)

April 22

• 23:47 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/206015 (duration: 00m 12s)
• 23:44 logmsgbot: krenair Synchronized php-1.26wmf3/extensions/ZeroBanner/includes/ZeroSpecialPage.php: https://gerrit.wikimedia.org/r/#/c/206017/ (duration: 00m 13s)
• 23:26 logmsgbot: krenair Synchronized php-1.26wmf3/extensions/Flow: https://gerrit.wikimedia.org/r/#/c/206008/ (duration: 00m 13s)
• 23:17 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/205889/ (duration: 00m 12s)
• 23:16 logmsgbot: krenair Synchronized php-1.26wmf2/extensions/OpenStackManager/nova/OpenStackNovaUser.php: https://gerrit.wikimedia.org/r/#/c/205887/ (duration: 00m 12s)
• 22:55 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.26wmf3
• 22:52 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.26wmf2
• 22:47 logmsgbot: twentyafterfour Finished scap: testwiki to php-1.26wmf3 and rebuild l10n cache (duration: 37m 11s)
• 22:44 hoo: Killed demon's "sudo -u www-data php /srv/mediawiki-staging/multiversion/MWScript.php refreshLinks.php --wiki=ptwiki" on terbium, sending the box into swap
• 22:10 logmsgbot: twentyafterfour Started scap: testwiki to php-1.26wmf3 and rebuild l10n cache
• 21:31 Coren: reboot round of deployment-prep done
• 21:05 Coren: Starting deployment-prep rolling reboots
• 20:13 logmsgbot: twentyafterfour scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="testwiki" --list-file="/srv/mediawiki-staging/wmf-config/extension-list" --output="/tmp/tmp.KaXyRl6UJi" ' returned non-zero exit status 1 (duration: 02m 10s)
• 20:10 logmsgbot: twentyafterfour Started scap: testwiki to php-1.26wmf3 and rebuild l10n cache
• 20:08 subbu: deployed parsoid version 3311936a
• 19:51 hashar: Zuul / Jenkins back up and processing the 1+ hour backlog of changes. Will take a while. Multiple causes: Zuul gearmand being stalled on a socket that has no more data to emit and Jenkins being deadlocked due to an IRC plugin
• 19:44 hashar: Killing Jenkins cause .... we know
• 19:27 hashar: zuul gearman server is stalled
• 15:30 gwicke: stopped restbase on restbase1002 in preparation for cmjohnson1 checking the hardware
• 15:30 logmsgbot: demon Finished scap: 1.26wmf2 was tracking master. should be fixed, being paranoid and doing full sync + i18n rebuild (duration: 08m 11s)
• 15:21 logmsgbot: demon Started scap: 1.26wmf2 was tracking master. should be fixed, being paranoid and doing full sync + i18n rebuild
• 15:19 logmsgbot: demon Synchronized php-1.26wmf2/extensions/VisualEditor/: (no message) (duration: 00m 12s)
• 15:19 logmsgbot: demon Synchronized php-1.26wmf2/extensions/WikiEditor/: (no message) (duration: 00m 11s)
• 15:12 logmsgbot: demon Synchronized php-1.26wmf1/extensions/WikiEditor/: (no message) (duration: 00m 13s)
• 13:37 godog: ms-be101[678] weight to 2820
• 13:25 paravoid: switched eqiad<->ulsfo link to Giglinx
• 11:11 godog: begin reimagining xenon, cerium and praseodymium
• 07:39 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Apr 22 07:38:22 UTC 2015 (duration 38m 21s)
• 07:28 legoktm: SULF is done, post-rename notifications are being sent out on the last large wikis
• 03:20 logmsgbot: ori Synchronized hhvm-fatal-error.php: I528e5384c: Increment a counter on fatals (duration: 00m 12s)
• 02:56 logmsgbot: LocalisationUpdate completed (1.26wmf2) at 2015-04-22 02:55:44+00:00
• 02:50 logmsgbot: l10nupdate Synchronized php-1.26wmf2/cache/l10n: (no message) (duration: 08m 31s)
• 02:26 logmsgbot: LocalisationUpdate completed (1.26wmf1) at 2015-04-22 02:25:40+00:00
• 02:22 logmsgbot: l10nupdate Synchronized php-1.26wmf1/cache/l10n: (no message) (duration: 05m 45s)
• 01:50 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1019, warm up (duration: 00m 13s)

April 21

• 23:07 logmsgbot: krenair Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/205640/ (duration: 00m 13s)
• 23:04 logmsgbot: krenair Synchronized php-1.26wmf2/extensions/VisualEditor: https://gerrit.wikimedia.org/r/205774 - should effectively be a no-op until config (duration: 00m 12s)
• 22:24 robh: disabled a bunch of old rt queues from allowing ticket creation, tired of spam
• 20:53 logmsgbot: aaron Synchronized php-1.26wmf1/includes/jobqueue/JobRunner.php: 4285f1921585ee87034e9739b1353fbad35f3a29 (duration: 00m 11s)
• 20:53 logmsgbot: aaron Synchronized php-1.26wmf1/includes/GlobalFunctions.php: bceb4de391bd8a321921a8587988cb1be7b71556 (duration: 00m 11s)
• 20:34 logmsgbot: aaron Synchronized php-1.26wmf2/includes/jobqueue/JobRunner.php: 2f3b7594650162b04f55e63e8df251d3913ab7ca (duration: 00m 11s)
• 18:24 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group1 to 1.26wmf2
• 17:48 logmsgbot: aude Synchronized wmf-config/Wikibase.php: Add subscriptionLookupMode setting for wikidata (duration: 00m 13s)
• 17:32 logmsgbot: aaron Synchronized php-1.26wmf2/includes/GlobalFunctions.php: b5b054e2f5b53e30d5aca21d046aa0ac33d5c407 (duration: 00m 12s)
• 16:40 logmsgbot: legoktm Synchronized php-1.26wmf1/extensions/OAI/OAIHooks.php: Don't try to update up_page=0 if page moves suppressed redirects (duration: 00m 13s)
• 16:40 logmsgbot: legoktm Synchronized php-1.26wmf2/extensions/OAI/OAIHooks.php: Don't try to update up_page=0 if page moves suppressed redirects (duration: 00m 11s)
• 15:45 logmsgbot: legoktm Synchronized php-1.26wmf2/extensions/OAI/OAIHooks.php: better debugging for T96686 (duration: 00m 11s)
• 15:11 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT gerrit:204722 (duration: 00m 11s)
• 14:15 logmsgbot: legoktm Synchronized php-1.26wmf1/extensions/OAI/OAIHooks.php: better debugging for T96686 (duration: 00m 11s)
• 14:15 springle: enwiki master under unusual jobrunner load, not terminal but see https://phabricator.wikimedia.org/T96686
• 08:47 logmsgbot: ori Synchronized php-1.26wmf1/includes/jobqueue: Ifa478996f: Revert 'Added per-wiki queue stats information' (duration: 00m 12s)
• 08:46 logmsgbot: ori Synchronized php-1.26wmf2/includes/jobqueue: Ifa478996f: Revert 'Added per-wiki queue stats information' (duration: 00m 13s)
• 06:37 springle: xtrabackup clone db1027 to db1019
• 06:17 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Apr 21 06:16:06 UTC 2015 (duration 16m 5s)
• 06:01 legoktm: fixed invalid accounts due to bad SULF renames on bat_smgwiki
• 05:53 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1019 (duration: 00m 12s)
• 05:48 logmsgbot: springle Synchronized wmf-config/db-codfw.php: reduce max lag to 10s, gerrit 204843 (duration: 00m 12s)
• 05:47 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: reduce max lag to 10s, gerrit 204843 (duration: 00m 12s)
• 03:49 logmsgbot: catrope Synchronized php-1.26wmf1/extensions/WikiEditor/: SWAT (duration: 00m 13s)
• 03:47 logmsgbot: catrope Synchronized php-1.26wmf2/extensions/WikiEditor/: SWAT (duration: 00m 15s)
• 03:01 logmsgbot: LocalisationUpdate completed (1.26wmf2) at 2015-04-21 03:00:08+00:00
• 02:54 logmsgbot: l10nupdate Synchronized php-1.26wmf2/cache/l10n: (no message) (duration: 08m 25s)
• 02:30 logmsgbot: LocalisationUpdate completed (1.26wmf1) at 2015-04-21 02:29:52+00:00
• 02:26 logmsgbot: l10nupdate Synchronized php-1.26wmf1/cache/l10n: (no message) (duration: 05m 56s)
• 00:43 logmsgbot: ebernhardson Synchronized wmf-config/InitialiseSettings-labs.php: keeping prod in sync with labs-only mediawiki-config changes (duration: 00m 14s)
• 00:09 logmsgbot: krenair Synchronized php-1.26wmf1/extensions/WikiGrok/resources/startup/init.js: https://gerrit.wikimedia.org/r/#/c/205469/ (duration: 00m 13s)
• 00:07 logmsgbot: krenair Synchronized php-1.26wmf2/extensions/WikiGrok/resources/startup/init.js: https://gerrit.wikimedia.org/r/#/c/205470/ (duration: 00m 13s)

April 20

• 23:53 logmsgbot: krenair Synchronized wmf-config/CirrusSearch-common.php: https://gerrit.wikimedia.org/r/#/c/204536/ - disable commons file search on officewiki, per erik (duration: 00m 12s)
• 23:47 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/201915/ (duration: 00m 24s)
• 23:43 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/198822/ (duration: 00m 13s)
• 23:39 logmsgbot: krenair Synchronized php-1.26wmf1/extensions/AbuseFilter: https://gerrit.wikimedia.org/r/205463 (duration: 00m 14s)
• 23:36 logmsgbot: krenair Synchronized php-1.26wmf2/extensions/AbuseFilter: https://gerrit.wikimedia.org/r/205462 (duration: 00m 15s)
• 23:17 logmsgbot: krenair Synchronized php-1.26wmf2: php-1.26wmf2/extensions/Flow https://gerrit.wikimedia.org/r/#/c/205432/ (duration: 01m 11s)
• 22:49 jgage: analytics1021: kafka disconnected from zk at 21:40; preferred-replica-election initiated at 22:48 to bring it back into service
• 21:29 mutante: tagged puppet run on appservers , --tags mw-apache-config
• 21:25 mutante: re-enabling puppet on mw servers for Apache change
• 21:02 mutante: disabling puppet on mw servers for deployment
• 20:35 akosiaris_: upload php5_5.3.10-1ubuntu3.18+wmf1 on precise-wikimedia distribution precise-wikimedia
• 20:18 cscott: updated Parsoid to version 0cabb5b2
• 18:49 awight: update payments from 46076dec9d82faa8660138f3b09342237891298b to 88b9f621bfee1de14a8cdef556a90e5567721754
• 18:13 thcipriani: truncate msg_resource on enwiki to refresh RL messages, seems to have fixed the issue
• 17:25 akosiaris_: remove wmf PHP5 for lucid from apt.wikimedia.org
• 16:14 logmsgbot: thcipriani Finished scap: Morning swat for gerrit:205219 (duration: 21m 13s)
• 16:09 akosiaris_: upload etherpad-lite 1.5.4-1 on apt.wikimedia.org. Not to be used in production yet, testing purposes
• 15:52 logmsgbot: thcipriani Started scap: Morning swat for gerrit:205219
• 15:50 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: scap gerrit:205081 (duration: 00m 12s)
• 15:44 bd808: Updated iegreview to 7303e5a (Update wikitext report for Inspire)
• 15:42 ottomata: analytics1014 offline, due to cpu temp?? attempting to reboot.
• 15:37 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: scap gerrit:196782 (duration: 00m 11s)
• 13:29 godog: bounce carbon-cache on graphite1001
• 08:44 _joe_: upgrading pybal on codfw loadbalancers
• 08:20 godog: swift ms-be101[678] weight to 2600
• 08:19 _joe_: installed pybal 1.07 on lvs2003
• 06:24 _joe_: testing new pybal package on lvs2003
• 05:06 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Apr 20 05:05:04 UTC 2015 (duration 5m 3s)
• 02:43 logmsgbot: LocalisationUpdate completed (1.26wmf2) at 2015-04-20 02:42:17+00:00
• 02:39 logmsgbot: l10nupdate Synchronized php-1.26wmf2/cache/l10n: (no message) (duration: 05m 19s)
• 02:24 logmsgbot: LocalisationUpdate completed (1.26wmf1) at 2015-04-20 02:23:22+00:00
• 02:20 logmsgbot: l10nupdate Synchronized php-1.26wmf1/cache/l10n: (no message) (duration: 05m 36s)

April 19

• 05:12 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Apr 19 05:11:33 UTC 2015 (duration 11m 32s)
• 02:42 logmsgbot: LocalisationUpdate completed (1.26wmf2) at 2015-04-19 02:41:18+00:00
• 02:38 logmsgbot: l10nupdate Synchronized php-1.26wmf2/cache/l10n: (no message) (duration: 05m 01s)
• 02:23 logmsgbot: LocalisationUpdate completed (1.26wmf1) at 2015-04-19 02:22:54+00:00
• 02:19 logmsgbot: l10nupdate Synchronized php-1.26wmf1/cache/l10n: (no message) (duration: 05m 46s)
• 02:12 legoktm: sending post-SULF rename notifications to renamed users on medium wikis
• 01:00 legoktm: running forceRenameUsers.php (SUL finalization) on large wikis (minus dewiki, enwiki)

April 18

• 20:57 legoktm: running forceRenameUsers.php (SUL finalization) on medium wikis starting with mgwiki. skipping mediawikiwiki for now due to T96489
• 09:06 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Apr 18 09:04:53 UTC 2015 (duration 4m 52s)
• 02:56 logmsgbot: LocalisationUpdate completed (1.26wmf2) at 2015-04-18 02:55:17+00:00
• 02:52 logmsgbot: l10nupdate Synchronized php-1.26wmf2/cache/l10n: (no message) (duration: 05m 05s)
• 02:32 logmsgbot: LocalisationUpdate completed (1.26wmf1) at 2015-04-18 02:31:04+00:00
• 02:27 logmsgbot: l10nupdate Synchronized php-1.26wmf1/cache/l10n: (no message) (duration: 05m 56s)
• 02:22 ori: Restarted HHVM on mw1181 and mw1096 after total lock-up; backtrace mw1096:/var/log/hhvm/hhvm.28914.bt

April 17

• 23:47 logmsgbot: awight Finished scap: T94246: Change legal text for recurring donations (duration: 39m 00s)
• 23:08 logmsgbot: awight Started scap: T94246: Change legal text for recurring donations
• 22:55 logmsgbot: awight Synchronized php-1.26wmf2/extensions/DonationInterface/: T94246: change legal text for recurring donation forms (duration: 00m 14s)
• 22:54 logmsgbot: awight Synchronized php-1.26wmf1/extensions/DonationInterface/: T94246: change legal text for recurring donation forms (duration: 00m 14s)
• 22:39 legoktm: fixed bad SULF renames on be_x_oldwiki, cbk_zamwiki, fiu_vrowiki, pa_uswikimedia, roa_rupwiki, roa_rupwiktionary, zh_min_nanwikibooks, zh_min_nanwikiquote, zh_min_nanwikisource
• 22:01 logmsgbot: legoktm Synchronized php-1.26wmf1/extensions/CentralAuth/includes/LocalRenameJob/LocalRenameUserJob.php: LocalRenameUserJob: In force mode, bypass all Title/User validation - https://gerrit.wikimedia.org/r/204945 (duration: 00m 11s)
• 22:00 logmsgbot: legoktm Synchronized php-1.26wmf2/extensions/CentralAuth/includes/LocalRenameJob/LocalRenameUserJob.php: LocalRenameUserJob: In force mode, bypass all Title/User validation - https://gerrit.wikimedia.org/r/204945 (duration: 00m 14s)
• 21:40 awight: update payments from f4ba034a8d55810276bbb7d4f861ceba7dfeaf2b to 46076dec9d82faa8660138f3b09342237891298b
• 19:45 logmsgbot: legoktm Synchronized php-1.26wmf1/extensions/CentralAuth/: LocalRenameUserJob: Don't validate the 'from' username if 'force' is true - https://gerrit.wikimedia.org/r/204846 (duration: 00m 12s)
• 19:44 logmsgbot: legoktm Synchronized php-1.26wmf2/extensions/CentralAuth/: LocalRenameUserJob: Don't validate the 'from' username if 'force' is true - https://gerrit.wikimedia.org/r/204846 (duration: 00m 12s)
• 19:39 legoktm: restarted forceRenameUsers.php (SUL finalization) on bgwiki (and then other medium wikis)
• 19:11 logmsgbot: aaron Synchronized php-1.26wmf2/includes/User.php: 2f1e93058f6247c81835a01b13e7473d5c5d060e (duration: 00m 12s)
• 19:10 ^d: running refreshLinks for ptwiki in screen on terbium for T91401. If it causes problems just kill it and ping me later.
• 18:48 logmsgbot: legoktm Synchronized php-1.26wmf1/extensions/CentralAuth/: forceRenameUsers: Replace _ in database name with - https://gerrit.wikimedia.org/r/204827 (duration: 00m 13s)
• 18:47 logmsgbot: legoktm Synchronized php-1.26wmf2/extensions/CentralAuth/: forceRenameUsers: Replace _ in database name with - https://gerrit.wikimedia.org/r/204827 (duration: 00m 14s)
• 18:37 ejegg: update payments from d37687239fa79842c0d6ea65e9230a3f14cda867 to f4ba034a8d55810276bbb7d4f861ceba7dfeaf2b
• 17:10 legoktm: running forceRenameUsers.php (SUL finalization) on all medium wikis
• 17:04 logmsgbot: ori Synchronized php-1.26wmf2/extensions/Popups: I48fbafe4d: Update Popups for cherry-picks (duration: 00m 11s)
• 17:04 logmsgbot: ori Synchronized php-1.26wmf1/extensions/Popups: Ie92d15985: Update Popups for cherry-picks (duration: 00m 13s)
• 16:53 logmsgbot: ori Synchronized php-1.26wmf2/extensions/Popups: I654c5cf8b: Update Popups for cherry-picks (duration: 00m 12s)
• 16:52 logmsgbot: ori Synchronized php-1.26wmf1/extensions/Popups: Iebaefdcf5: Update Popups for cherry-picks (duration: 00m 11s)
• 16:38 legoktm: making User:Maintenance script a 'bot' on all wikis
• 15:19 Jeff_Green: DNS updates, for a couple of fundraising hosts
• 14:26 logmsgbot: hoo Synchronized wmf-config/InitialiseSettings.php: wgCopyUploadsDomains: add hooserv.net for commons (duration: 00m 11s)
• 12:27 hashar: Zuul should be back up now
• 12:14 hashar: Switching Zuul scheduler on gallium.wikimedia.org to the Debian package version
• 09:02 hashar: apt-get upgrade on gallium and lanthanum
• 08:09 godog: reboot ms-be1009, xfs woes
• 05:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Apr 17 05:47:10 UTC 2015 (duration 47m 9s)
• 04:36 logmsgbot: aaron Synchronized wmf-config/db-eqiad.php: Set "recentchanges" group for s2-s7 (duration: 00m 11s)
• 04:33 logmsgbot: aaron Synchronized wmf-config/db-eqiad.php: (no message) (duration: 00m 12s)
• 03:33 legoktm: restarting forceRenameUsers.php (SUL finalization) on the rest of the small wikis, starting with wm2008wiki
• 03:26 legoktm: attached CheckUser@dewiki,enwiki,metawiki to CheckUser@global
• 03:25 legoktm: attached Checkuser@enwiki to Checkuser@global
• 02:47 logmsgbot: LocalisationUpdate completed (1.26wmf2) at 2015-04-17 02:46:38+00:00
• 02:43 logmsgbot: l10nupdate Synchronized php-1.26wmf2/cache/l10n: (no message) (duration: 05m 10s)
• 02:29 logmsgbot: LocalisationUpdate completed (1.26wmf1) at 2015-04-17 02:28:41+00:00
• 02:25 logmsgbot: l10nupdate Synchronized php-1.26wmf1/cache/l10n: (no message) (duration: 05m 39s)
• 01:49 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: I6fa034f4a: Enable Hovercards by default on Catalan and Greek Wikipedias (T88164) (duration: 00m 12s)
• 01:41 legoktm: paused forceRenameUsers around wm2008wiki
• 01:41 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: I95f8c010e: Popups: enable as beta feature by default (duration: 00m 12s)
• 01:37 legoktm: marked as "Steward" accounts as not to be renamed (utr_status=11)
• 01:34 logmsgbot: ori Synchronized php-1.26wmf1/extensions/Popups: Update Popups for Ie4cc455f: Act as a beta feature if so configured (duration: 00m 12s)
• 01:33 logmsgbot: ori Synchronized php-1.26wmf2/extensions/Popups: Update Popups for Ie4cc455f: Act as a beta feature if so configured (duration: 00m 12s)
• 01:32 logmsgbot: ori Synchronized wmf-config: I7fde63453: PopUps: disabled by default; requires BetaFeatures if set as beta feature (duration: 00m 11s)
• 01:31 legoktm: marked as "Oversight" accounts as not to be renamed (utr_status=11)

April 16

• 23:43 logmsgbot: legoktm Synchronized php-1.26wmf2/extensions/Gather/includes/specials/SpecialGather.php: Make Special:Gather show pages for that user https://gerrit.wikimedia.org/r/#/c/204671/ (duration: 00m 13s)
• 23:27 logmsgbot: legoktm Synchronized php-1.26wmf1/extensions/CentralAuth/includes/CentralAuthUser.php: Fix CentralAuthUser::loadAttached if no accounts are attached (duration: 00m 13s)
• 23:26 logmsgbot: legoktm Synchronized php-1.26wmf2/extensions/CentralAuth/includes/CentralAuthUser.php: Fix CentralAuthUser::loadAttached if no accounts are attached (duration: 00m 13s)
• 23:25 logmsgbot: legoktm Synchronized php-1.26wmf2/extensions/Gather/includes/specials/SpecialGather.php: Error in regex broke User lists pages https://gerrit.wikimedia.org/r/#/c/204499/ (duration: 00m 12s)
• 23:08 logmsgbot: legoktm Synchronized wmf-config/InitialiseSettings.php: Set meta namespace on or.wiktionary (duration: 00m 14s)
• 23:06 logmsgbot: legoktm Synchronized wmf-config/InitialiseSettings.php: User rights configuration on ne.wikipedia - Filemover (duration: 00m 11s)
• 23:05 logmsgbot: legoktm Synchronized wmf-config/: User rights configuration on ne.wikipedia - Abusefilter (duration: 00m 12s)
• 22:25 legoktm: running forceRenameUsers.php (SUL finalization) on all small wikis
• 21:32 logmsgbot: legoktm Synchronized wmf-config/InitialiseSettings.php: Increase $wgMaxNameChars to 85 (duration: 00m 12s)
• 20:37 ori: MediaWiki stats flowing into StatsD again.
• 20:34 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I31c7b2c3d5: Reset port of $wgStatsdServer to default (8125) (duration: 00m 14s)
• 20:32 logmsgbot: ori Synchronized php-1.26wmf1/includes/libs/BufferingStatsdDataFactory.php: 3077a66625: Don't bother buffering a counter update with a delta of zero. (duration: 00m 13s)
• 19:44 blazecat: Updated jobqueue:aggregator:s-wikis:v2 key on 10.64.32.76 to $wgLocalDatabases (sans labswiki)
• 19:15 paravoid: depooling esams, network issues
• 18:53 andrewbogott: rebooting labvirt100x to turn on virtualization in bios
• 18:40 andrewbogott: rebooting labvirt1001
• 18:38 legoktm: creating "Maintenance script" account on all SUL wikis for globaluserpage
• 16:46 csteipp: removed oauth-headers.php since that allowed stealing httponly cookies
• 16:44 logmsgbot: csteipp Synchronized w: (no message) (duration: 00m 11s)
• 16:42 logmsgbot: kartik Finished scap: Update ContentTranslation (duration: 22m 13s)
• 16:19 logmsgbot: kartik Started scap: Update ContentTranslation
• 16:06 legoktm: starting to run forceRenameUsers.php (SUL finalization)
• 15:39 ^d: restarting gerrit
• 15:09 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/203323 (duration: 00m 12s)
• 15:03 logmsgbot: krenair Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/203861/ - should be a no-op, just config file cleanup (duration: 00m 13s)
• 12:35 akosiaris: uploaded etherpad-lite_1.4.1-2 on apt.wikimedia.org
• 11:58 Krenair: restarted apache on silver, wikitech login seems to work again
• 11:56 andrewbogott: disabling puppet on virt1000 so that I can prevent a questionable cron (purging tokens from the keystone db) from running while I sleep.
• 04:00 logmsgbot: legoktm Synchronized php-1.26wmf2/includes/DefaultSettings.php: The 'spambot_username' message is a reserved username (duration: 00m 11s)
• 03:29 logmsgbot: legoktm Synchronized php-1.26wmf1/includes/DefaultSettings.php: The 'spambot_username' message is a reserved username (duration: 00m 12s)
• 03:07 bd808: Updated iegreview to e126f7c (Fix aggregated reports to work on the new reviews system)
• 02:56 logmsgbot: LocalisationUpdate completed (1.26wmf2) at 2015-04-16 02:55:08+00:00
• 02:52 logmsgbot: l10nupdate Synchronized php-1.26wmf2/cache/l10n: (no message) (duration: 04m 38s)
• 02:34 legoktm: starting forceRenameUsers.php (SUL finalization) on non-test*wikis
• 02:33 logmsgbot: LocalisationUpdate completed (1.26wmf1) at 2015-04-16 02:32:26+00:00
• 02:29 logmsgbot: l10nupdate Synchronized php-1.26wmf1/cache/l10n: (no message) (duration: 05m 57s)
• 02:24 andrewbogott: but the ‘token’ table is still too big to manage
• 02:24 andrewbogott: restarted mysql on virt1000 because keystone was stuck. It seems to have helped, eventually
• 02:24 andrewbogott: restarted keystone and nova-scheduler in a failed attempt to unstick things
• 02:23 andrewbogott: testing the log by logging a test

April 15

• 20:28 subbu: deployed parsoid version ac7a01b9
• 18:25 legoktm: running forceRenameUsers.php (SUL finalization) on test* wikis
• 17:15 legoktm: running migrateAccount.php --auto (CentralAuth)
• 16:49 ottomata: rebooting analyics1020
• 16:45 nuria: restarted eventlogging && deployed d241d75ee2fab554bc47cf8d1ba83f5df2130633
• 16:29 logmsgbot: demon Synchronized php-1.26wmf1/extensions/CentralAuth/: (no message) (duration: 00m 13s)
• 16:22 gwicke: running revision render thin-out script on wikipedia HTML
• 15:35 bblack: re-enabling puppet on caches, canary nodes were no-op \o/
• 15:27 bblack: disabling puppet on caches JIC for https://gerrit.wikimedia.org/r/204068 merge
• 14:54 legoktm: running deleteEmptyAccounts.php --fix on metawiki (CentralAuth)
• 13:54 andrewbogott: purging expired keystone tokens on virt1000
• 12:59 andrewbogott: restarted pdns on virt1000 and labcontrol2001 to recover from the opendj restart
• 12:58 akosiaris: restarted keystone, nova services on virt1000
• 12:58 akosiaris: restarted opendj on neptunium
• 12:58 andrewbogott: dropping labswiki and labswiki_eqiad from mysql on virt1000
• 11:20 _joe_: restarted apache2 on virt1000, passenger gone to hell
• 10:31 godog: bounce jobchron on mw1001
• 10:30 godog: restart keystone on virt1000 (#2)
• 08:12 godog: bounce keystone on virt1000
• 04:16 logmsgbot: ebernhardson Synchronized wmf-config/CommonSettings.php: Bump flow cache version (duration: 00m 11s)
• 03:11 logmsgbot: LocalisationUpdate completed (1.26wmf1) at 2015-04-15 03:10:08+00:00
• 03:04 logmsgbot: l10nupdate Synchronized php-1.26wmf1/cache/l10n: (no message) (duration: 09m 22s)
• 02:39 logmsgbot: LocalisationUpdate completed (1.25wmf24) at 2015-04-15 02:37:59+00:00
• 02:32 logmsgbot: l10nupdate Synchronized php-1.25wmf24/cache/l10n: (no message) (duration: 09m 03s)

April 14

• 23:54 logmsgbot: catrope Synchronized php-1.25wmf24/extensions/VisualEditor: SWAT (duration: 00m 12s)
• 23:54 logmsgbot: catrope Synchronized php-1.25wmf24/extensions/CentralAuth: SWAT (duration: 00m 13s)
• 23:54 logmsgbot: catrope Synchronized php-1.25wmf24/extensions/Gather: SWAT (duration: 00m 12s)
• 23:54 logmsgbot: catrope Synchronized php-1.25wmf24/extensions/WikiEditor: SWAT (duration: 00m 11s)
• 23:52 logmsgbot: catrope Synchronized php-1.26wmf1/extensions/VisualEditor: SWAT (duration: 00m 12s)
• 23:52 logmsgbot: catrope Synchronized php-1.26wmf1/extensions/CentralAuth: SWAT (duration: 00m 12s)
• 23:51 logmsgbot: catrope Synchronized php-1.26wmf1/extensions/Flow: SWAT (duration: 00m 13s)
• 23:51 logmsgbot: catrope Synchronized php-1.26wmf1/extensions/WikiEditor: SWAT (duration: 00m 11s)
• 22:06 logmsgbot: rmoen Synchronized php-1.26wmf1/extensions/Gather/: Update gather with cherry picks (duration: 00m 11s)
• 22:05 andrewbogott: rebooting all labvirt100x hosts to enable virtualization in the bios
• 21:16 logmsgbot: rmoen Synchronized wmf-config/CirrusSearch-production.php: enable cirrus search eventlogging in production (duration: 00m 13s)
• 20:23 ottomata: powercycled analytics1020
• 19:28 legoktm: running updateUsersToRename.php (CentralAuth)
• 18:14 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: If7f77996b: Set $wgStatsdServer (duration: 00m 15s)
• 18:04 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group1 to 1.26wmf1
• 17:41 legoktm: running removeHHVMTag on testwiki
• 17:07 logmsgbot: legoktm Synchronized wmf-config/InitialiseSettings.php: Add 'CentralAuthSULRename' log group (duration: 00m 14s)
• 17:06 legoktm: ^ was Set $wgCentralAuthCheckSULMigration = true
• 17:06 logmsgbot: legoktm Synchronized wmf-config/CommonSettings.php: Set = true (duration: 00m 13s)
• 17:04 logmsgbot: legoktm Finished scap: Updating CentralAuth (duration: 42m 28s)
• 16:53 Reedy: Deleting oaiaudit entries that are pre 2015
• 16:22 logmsgbot: legoktm Started scap: Updating CentralAuth
• 15:48 bd808: Restarted logstash on logstash1003.eqiad.wmnet; subbu reported missing parsoid log events
• 15:18 logmsgbot: anomie Synchronized php-1.26wmf1/includes/page/Article.php: SWAT: Continued debugging of phab:T92046 (gerrit:204050) (duration: 00m 11s)
• 13:59 cmjohnson1: barium down for disk swap
• 12:02 _joe_: restarting gitblit
• 09:00 _joe_: restarting gitblit
• 08:49 hoo: Attached Manfred Strumpf@enwiki to the global account of the same name
• 08:48 hashar: Testing log bot

April 13

• 21:57 mutante: restarting gitblot
• 20:20 logmsgbot: demon Synchronized php-1.25wmf24/includes/media/XMP.php: rm useless debugging (duration: 00m 15s)
• 20:18 logmsgbot: demon Synchronized php-1.25wmf24/includes/media/XMP.php: adhocdebug (duration: 00m 12s)
• 20:15 springle: dbstore1001 s2 delayed replication resumed, T95426
• 19:48 springle: dbstore1002 centralauth resync, T95927
• 19:45 springle: dbstore1001 s2 delayed replication resumed, T95426
• 17:20 legoktm: running migratePass0 across all CentralAuth wikis
• 16:48 bblack: re-enabling puppet on caches
• 16:23 bblack: disabling puppet on caches for cache.pp-split merge, will be restarting puppetmasters too...
• 16:16 logmsgbot: thcipriani Finished scap: Morning swat for T94128 (duration: 41m 12s)
• 15:34 logmsgbot: thcipriani Started scap: Morning swat for T94128
• 15:24 logmsgbot: thcipriani Synchronized wmf-config/flaggedrevs.php: Morning swat 203283 (duration: 00m 14s)
• 15:17 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: Morning swat 203279 (duration: 00m 14s)
• 15:09 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: Morning swat 202736 (duration: 00m 11s)
• 15:06 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: Morning swat 199927 and 199923 (duration: 00m 11s)
• 15:00 _joe_: depooled mw1031
• 12:30 godog: bounce statsite on graphite1001
• 10:37 godog: ms-be101[678] object weight to 2250
• 08:59 godog: ms-be10[678] account/container weight to 100
• 08:34 zeljkof: restarting stuck Jenkins
• 05:05 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Apr 13 05:04:22 UTC 2015 (duration 4m 21s)
• 02:47 logmsgbot: LocalisationUpdate completed (1.26wmf1) at 2015-04-13 02:46:33+00:00
• 02:43 logmsgbot: l10nupdate Synchronized php-1.26wmf1/cache/l10n: (no message) (duration: 05m 40s)
• 02:27 logmsgbot: LocalisationUpdate completed (1.25wmf24) at 2015-04-13 02:26:20+00:00
• 02:22 logmsgbot: l10nupdate Synchronized php-1.25wmf24/cache/l10n: (no message) (duration: 06m 32s)
• 00:27 jgage: analytics1017 unresponsive, console reported high temps. rebooted.

April 12

• 15:39 hoo: Attached Manfred Strumpf@commonswiki to the global account of the same name
• 15:22 hoo: Attached Aloiswuest@commonswiki, Aloiswuest@dewikiquote and Aloiswuest@dewiktionary to the global account of the same name
• 15:14 hoo: Attached Srbauer@nowiki and Srbauer@sourceswiki to the global account of the same name
• 15:07 hoo: Attached Yagosaga@dewikibooks and Yagosaga@commonswiki to the global account of the same name
• 14:54 hoo: Attached Peng@dewiktionary to the global account of the same name
• 14:48 hoo_: Attached Bradypus@enwiki and Bradypus@commonswiki to the global account of the same name
• 14:47 hoo_: Attached Helmut Welger@eowiki to the global account of the same name
• 05:30 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Apr 12 05:29:02 UTC 2015 (duration 29m 1s)
• 02:46 logmsgbot: LocalisationUpdate completed (1.26wmf1) at 2015-04-12 02:45:15+00:00
• 02:41 logmsgbot: l10nupdate Synchronized php-1.26wmf1/cache/l10n: (no message) (duration: 05m 44s)
• 02:26 logmsgbot: LocalisationUpdate completed (1.25wmf24) at 2015-04-12 02:25:07+00:00
• 02:21 logmsgbot: l10nupdate Synchronized php-1.25wmf24/cache/l10n: (no message) (duration: 06m 23s)

April 11

• 21:05 logmsgbot: aaron Synchronized php-1.26wmf1/maintenance/Maintenance.php: 103c7f7534b69f7a920edd3b893e25851301e79c (duration: 00m 12s)
• 20:56 logmsgbot: aaron Synchronized php-1.26wmf1/includes/jobqueue/JobRunner.php: 2e96dc28ef225441547f4e61acb8a09cb5c0709e (duration: 00m 12s)
• 12:26 logmsgbot: krinkle Synchronized php-1.26wmf1/includes/Title.php: T95811 (duration: 00m 12s)
• 12:20 logmsgbot: krinkle Synchronized php-1.25wmf24/includes/Title.php: T95811 (duration: 00m 11s)
• 05:30 logmsgbot: springle Synchronized wmf-config/db-codfw.php: reduce max lag to 15s, gerrit 203508 (duration: 00m 12s)
• 05:30 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: reduce max lag to 15s, gerrit 203508 (duration: 00m 11s)
• 05:15 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Apr 11 05:14:10 UTC 2015 (duration 14m 9s)
• 04:02 logmsgbot: aaron Synchronized php-1.26wmf1/includes/jobqueue/JobRunner.php: 65ff16efa7a69dfbec4c70df22d89a1b12c60762 (duration: 00m 11s)
• 02:49 logmsgbot: LocalisationUpdate completed (1.26wmf1) at 2015-04-11 02:48:04+00:00
• 02:44 logmsgbot: l10nupdate Synchronized php-1.26wmf1/cache/l10n: (no message) (duration: 05m 28s)
• 02:29 logmsgbot: LocalisationUpdate completed (1.25wmf24) at 2015-04-11 02:28:24+00:00
• 02:24 logmsgbot: l10nupdate Synchronized php-1.25wmf24/cache/l10n: (no message) (duration: 06m 37s)

April 10

• 18:38 bd808: Trebuchet checkout failed for scap/scap on mw2128, mw1113, mw1222, and mw1104
• 18:38 bd808: Trebuchet fetch failed for scap/scap on mw2128 and mw1222
• 18:38 bd808: Updated scap to f9b9a82 (Remove exotic unicode from ascii logo)
• 18:31 logmsgbot: bd808 Synchronized php-1.26wmf1/includes/debug/logger/LegacyLogger.php: debug: Add missing use DateTimeZone in LegacyLogger.php (2c8f292c) (duration: 00m 14s)
• 18:30 logmsgbot: bd808 Synchronized php-1.26wmf1/includes/Title.php: Title: Add debug logging for I2b36b7a3 and I62fe3f700 (f45a334e) (duration: 00m 12s)
• 12:51 godog: metrics from labs on graphite1001 by mistake, purging
• 12:31 logmsgbot: krenair Synchronized php-1.25wmf24/includes/specialpage/SpecialPageFactory.php: removing debug (duration: 00m 13s)
• 12:29 logmsgbot: krenair Synchronized php-1.25wmf24/includes/specialpage/SpecialPageFactory.php: (no message) (duration: 00m 12s)
• 12:28 logmsgbot: krenair Synchronized php-1.25wmf24/includes/specialpage/SpecialPageFactory.php: (no message) (duration: 00m 12s)
• 12:24 logmsgbot: krenair Synchronized php-1.25wmf24/includes/specialpage/SpecialPageFactory.php: (no message) (duration: 00m 12s)
• 12:22 logmsgbot: krenair Synchronized php-1.25wmf24/includes/specialpage/SpecialPageFactory.php: (no message) (duration: 00m 12s)
• 12:19 logmsgbot: krenair Synchronized php-1.25wmf24/includes/specialpage/SpecialPageFactory.php: (no message) (duration: 00m 11s)
• 12:14 logmsgbot: krenair Synchronized php-1.25wmf24/includes/specialpage/SpecialPageFactory.php: trying something else (duration: 00m 13s)
• 12:08 logmsgbot: krenair Synchronized php-1.25wmf24/includes/specialpage/SpecialPageFactory.php: trying to investigate T90382 with some temp debugging (duration: 00m 12s)
• 09:46 _joe_: stopping and starting pybal on lvs2003, tests for T94822
• 09:01 godog: reboot ms-be1005, new disk didn't show up with the right letter
• 06:34 logmsgbot: tstarling Synchronized wmf-config/InitialiseSettings.php: wmgEnableRandomRootPage (duration: 00m 11s)
• 05:59 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Apr 10 05:58:44 UTC 2015 (duration 58m 43s)
• 03:12 logmsgbot: LocalisationUpdate completed (1.26wmf1) at 2015-04-10 03:10:56+00:00
• 03:05 logmsgbot: l10nupdate Synchronized php-1.26wmf1/cache/l10n: (no message) (duration: 09m 28s)
• 02:39 logmsgbot: LocalisationUpdate completed (1.25wmf24) at 2015-04-10 02:38:30+00:00
• 02:32 logmsgbot: l10nupdate Synchronized php-1.25wmf24/cache/l10n: (no message) (duration: 09m 03s)
• 01:36 mutante: powercycling mw2129
• 01:31 logmsgbot: krenair Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/203269/ - trivial throttle change for event this weekend (duration: 01m 06s)
• 00:39 logmsgbot: rmoen Synchronized php-1.25wmf24/extensions/Gather/: update gather to master (duration: 01m 13s)
• 00:37 logmsgbot: rmoen Synchronized php-1.25wmf24/extensions/MobileFrontend/: sync mobilefrontend for cherry-pick (duration: 01m 07s)
• 00:35 logmsgbot: rmoen Synchronized php-1.26wmf1/extensions/Gather/: update gather to master (duration: 01m 07s)
• 00:33 logmsgbot: rmoen Synchronized php-1.26wmf1/extensions/MobileFrontend/: sync mobilefrontend for cherry-pick (duration: 01m 07s)
• 00:02 Krenair: Deployed fix for T95589

April 9

• 23:53 Krenair: ssh to mw2129.codfw.wmnet still timing out
• 23:52 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/203195/ (duration: 01m 11s)
• 21:50 bblack: esams cache role migrations starting up soon for the evening...
• 19:29 logmsgbot: aaron Synchronized php-1.25wmf24/extensions/AbuseFilter: 4b03cec4574aaece27879e408d545ce7ea0fa2ce (duration: 01m 06s)
• 19:14 logmsgbot: aaron Synchronized wmf-config/PoolCounterSettings-common.php: Add pool counter config for Translate (duration: 01m 11s)
• 18:28 legoktm: mw2129.codfw.wmnet still timing out
• 18:28 logmsgbot: legoktm Synchronized wmf-config/InitialiseSettings.php: Enable SandboxLink on all projects where it is a default gadget https://gerrit.wikimedia.org/r/203109 (duration: 01m 06s)
• 18:20 logmsgbot: legoktm Finished scap: SandboxLink deployment (duration: 39m 55s)
• 18:06 legoktm: 18:05:48 ['/srv/deployment/scap/scap/bin/sync-common', '--no-update-l10n', 'mw1010.eqiad.wmnet', 'mw1033.eqiad.wmnet', 'mw1070.eqiad.wmnet', 'mw1097.eqiad.wmnet', 'mw1216.eqiad.wmnet', 'mw1161.eqiad.wmnet', 'mw1201.eqiad.wmnet', 'mw2001.codfw.wmnet', 'mw2041.codfw.wmnet', 'mw2080.codfw.wmnet', 'mw2119.codfw.wmnet', 'mw2187.codfw.wmnet'] on mw2129.codfw.wmnet returned [255]: ssh: connect to host mw2129.codfw.wmnet port 22: Connection
• 17:40 logmsgbot: legoktm Started scap: SandboxLink deployment
• 16:56 logmsgbot: krenair Synchronized wmf-config: no-ops: https://gerrit.wikimedia.org/r/#/c/201910/ and https://gerrit.wikimedia.org/r/#/c/202965/ (duration: 00m 13s)
• 16:34 logmsgbot: kartik Finished scap: Update ContentTranslation (duration: 22m 07s)
• 16:14 godog: migrate labmon1001 to statsite
• 16:12 logmsgbot: kartik Started scap: Update ContentTranslation
• 16:09 ottomata: decomissioning vanadium, powering it off
• 16:05 kart_: Updated cxserver to 640bcdf
• 15:54 logmsgbot: krenair Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/200037/1 - should be a no-op (duration: 00m 11s)
• 15:46 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/202726 (duration: 00m 12s)
• 15:32 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/202755/ (duration: 00m 12s)
• 15:29 godog: bounce uwsgi on graphite1001
• 15:06 logmsgbot: krenair Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/201940/ (duration: 00m 11s)
• 15:03 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/202021/ (duration: 00m 11s)
• 12:23 godog: bounce icinga-wm
• 10:46 godog: txstatsd replaced on graphite1001, replacing other clients
• 10:21 godog: begin replacing txstatsd with statsite, stop graphite to rename metrics
• 05:43 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Apr 9 05:42:38 UTC 2015 (duration 42m 37s)
• 04:31 springle: dbstore1001 s2 delayed replication paused, T95426
• 04:27 springle: xtrabackup clone db2035 to db2041
• 03:15 jamesofur: changed email address for metawiki:JulieC per request and account verification to allow for merger to global account
• 02:59 logmsgbot: LocalisationUpdate completed (1.26wmf1) at 2015-04-09 02:58:19+00:00
• 02:55 logmsgbot: l10nupdate Synchronized php-1.26wmf1/cache/l10n: (no message) (duration: 04m 40s)
• 02:37 logmsgbot: LocalisationUpdate completed (1.25wmf24) at 2015-04-09 02:36:20+00:00
• 02:32 logmsgbot: l10nupdate Synchronized php-1.25wmf24/cache/l10n: (no message) (duration: 06m 19s)
• 00:46 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings.php: Revert direct RESTbase for non-enwiki Wikipedias (duration: 00m 12s)

April 8

• 23:45 logmsgbot: catrope Synchronized php-1.26wmf1/includes/api/ApiParse.php: SWAT (duration: 00m 12s)
• 23:45 logmsgbot: catrope Synchronized php-1.26wmf1/extensions/VisualEditor/: SWAT (duration: 00m 12s)
• 23:38 logmsgbot: catrope Synchronized php-1.25wmf24/extensions/VisualEditor/: SWAT (duration: 00m 14s)
• 23:38 logmsgbot: catrope Synchronized php-1.25wmf24/includes/api/ApiParse.php: SWAT (duration: 00m 11s)
• 23:18 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings.php: Make VE access RB directly on Wikipedias (duration: 00m 12s)
• 23:17 logmsgbot: catrope Synchronized wmf-config/CommonSettings.php: Manage username blacklist from metawiki only (duration: 00m 14s)
• 23:09 mutante: mw1208 - restarted hhvm
• 23:09 mutante: mw1198 - restarted hhvm
• 22:52 mutante: rbf1002 - power down, gone from icinga, rbf200x revoke salt keys
• 22:42 mutante: rbf1001 - shutdown -h now (https://phabricator.wikimedia.org/T93006#1177448)
• 22:38 logmsgbot: aaron Synchronized php-1.26wmf1/extensions/AbuseFilter: e0c99fa093f23f23310c77524a78adfd3017f79e (duration: 00m 12s)
• 22:30 mutante: rbf1001,rbf1002 - stopping redis-server
• 21:09 logmsgbot: gwicke Synchronized wmf-config/InitialiseSettings.php: Make VisualEditor load HTML directly from rest.wikimedia.org on enwiki (duration: 00m 11s)
• 20:51 logmsgbot: twentyafterfour Purged l10n cache for 1.25wmf23
• 20:49 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.26wmf1
• 20:48 logmsgbot: aaron Synchronized wmf-config/db-eqiad.php: Set "recentchanges" query group (duration: 00m 16s)
• 20:46 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.25wmf24
• 20:32 cscott: updated Parsoid to version a76bd8a3
• 20:25 logmsgbot: twentyafterfour Finished scap: testwiki to php-1.26wmf1 and rebuild l10n cache (duration: 25m 38s)
• 19:59 logmsgbot: twentyafterfour Started scap: testwiki to php-1.26wmf1 and rebuild l10n cache
• 15:59 logmsgbot: legoktm Finished scap: Log promote to global renames in the global rename log https://gerrit.wikimedia.org/r/202742 (duration: 22m 27s)
• 15:36 logmsgbot: legoktm Started scap: Log promote to global renames in the global rename log https://gerrit.wikimedia.org/r/202742
• 15:24 logmsgbot: anomie Synchronized php-1.25wmf24/includes/page/Article.php: SWAT: More debugging for phab:T92046 (gerrit:202602, gerrit:202603) (duration: 00m 13s)
• 15:09 logmsgbot: anomie Synchronized wmf-config/throttle.php: SWAT: Throttle rule for Editatón Ciencia y Tecnología en Chile gerrit:202740 (duration: 00m 11s)
• 15:07 logmsgbot: anomie Synchronized wmf-config/CommonSettings.php: SWAT: Add REL1_25 branches to ExtDist gerrit:202591 (duration: 00m 11s)
• 15:06 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Add fatal log group gerrit:202741 (for real this time) (duration: 00m 13s)
• 15:04 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Add fatal log group gerrit:202741 (duration: 00m 12s)
• 12:47 akosiaris: restarted HHVM on mw1114
• 12:43 hasharLunch: Zuul is back and it is nasty
• 12:23 hasharLunch: Killed Zuul :(
• 11:39 logmsgbot: aude Synchronized php-1.25wmf24/extensions/Wikidata: Fix issue with edit links in diff view (duration: 00m 20s)
• 06:03 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Apr 8 06:02:15 UTC 2015 (duration 2m 14s)
• 03:34 bblack: re-enabling puppet on caches (weight scale looks good!)
• 03:30 bblack: disabling puppet on caches for weight scale deploy/test
• 03:05 logmsgbot: LocalisationUpdate completed (1.25wmf24) at 2015-04-08 03:04:53+00:00
• 03:01 logmsgbot: l10nupdate Synchronized php-1.25wmf24/cache/l10n: (no message) (duration: 06m 15s)
• 02:40 logmsgbot: LocalisationUpdate completed (1.25wmf23) at 2015-04-08 02:39:10+00:00
• 02:33 logmsgbot: l10nupdate Synchronized php-1.25wmf23/cache/l10n: (no message) (duration: 08m 54s)
• 00:55 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: reduce max lag to 20s, gerrit 202626 (duration: 00m 11s)

April 7

• 23:21 logmsgbot: mattflaschen Synchronized php-1.25wmf24/extensions/Flow/: Deploy Flow for LQT/Echo conversion feature (duration: 00m 13s)
• 21:24 hoo: Manually started dumpwikidatattl.sh as datasets on snapshot1003
• 20:53 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: I60ef00d2b: Blackhole the slow parse log on private wikis (duration: 00m 13s)
• 18:54 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group1 to 1.25wmf24
• 18:02 logmsgbot: twentyafterfour Synchronized multiversion/updateGroup1: Deploying https://gerrit.wikimedia.org/r/#/c/134124/ (duration: 00m 12s)
• 17:37 logmsgbot: krenair Synchronized php-1.25wmf23/includes/MediaWiki.php: https://gerrit.wikimedia.org/r/#/c/202302/ (duration: 00m 12s)
• 17:35 logmsgbot: krenair Synchronized php-1.25wmf24/includes/MediaWiki.php: https://gerrit.wikimedia.org/r/#/c/202301/ (duration: 00m 15s)
• 17:27 logmsgbot: krenair Synchronized php-1.25wmf23/includes/api/ApiQuerySiteinfo.php: https://gerrit.wikimedia.org/r/#/c/202332/ (duration: 00m 13s)
• 17:25 logmsgbot: krenair Synchronized php-1.25wmf24/includes/api/ApiQuerySiteinfo.php: actually apply the change this time (duration: 00m 11s)
• 17:21 logmsgbot: krenair Synchronized php-1.25wmf24/includes/api/ApiQuerySiteinfo.php: https://gerrit.wikimedia.org/r/#/c/202333/ (duration: 00m 10s)
• 16:51 logmsgbot: krenair Synchronized php-1.25wmf23/includes/skins/Skin.php: https://gerrit.wikimedia.org/r/#/c/202391/ (duration: 00m 13s)
• 16:48 logmsgbot: krenair Synchronized php-1.25wmf24/includes/skins/Skin.php: https://gerrit.wikimedia.org/r/#/c/202313/ (duration: 00m 12s)
• 16:41 logmsgbot: krenair Synchronized php-1.25wmf24/extensions/VisualEditor/lib/ve: https://gerrit.wikimedia.org/r/#/c/202400/ (duration: 00m 12s)
• 16:20 logmsgbot: krenair Synchronized php-1.25wmf24/extensions/Wikidata: https://gerrit.wikimedia.org/r/#/c/202398/1 (duration: 00m 18s)
• 16:16 Krenair: https://gerrit.wikimedia.org/r/#/c/134124/ was merged but has not been synced
• 16:16 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/169716/ (duration: 00m 14s)
• 15:49 logmsgbot: krenair Synchronized php-1.25wmf24/extensions/Flow: https://gerrit.wikimedia.org/r/#/c/202262/2 (duration: 00m 14s)
• 15:43 logmsgbot: krenair Finished scap: https://gerrit.wikimedia.org/r/#/c/202258/1 (duration: 22m 29s)
• 15:41 bblack: re-weighted pybal esams/upload (all-1 to all-3)
• 15:21 logmsgbot: krenair Started scap: https://gerrit.wikimedia.org/r/#/c/202258/1
• 13:32 cmjohnson1: scheduled downtime for barium to replace disk.
• 11:57 springle: xtrabackup clone db2016 to db2042
• 10:59 springle: install db2042, puppet sign, etc
• 09:54 godog: swift weight ms-be10[678] to 2000
• 07:18 paravoid: powercycling analytics1020, unresponsive
• 07:12 _joe_: powercycling mw2128, network driver crashes
• 06:01 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Apr 7 06:00:13 UTC 2015 (duration 0m 12s)
• 04:49 mutante: rbf - puppetstoredconfigclean.rb, remove from icinga
• 03:07 logmsgbot: LocalisationUpdate completed (1.25wmf24) at 2015-04-07 03:06:13+00:00
• 03:02 logmsgbot: l10nupdate Synchronized php-1.25wmf24/cache/l10n: (no message) (duration: 06m 16s)
• 02:41 logmsgbot: LocalisationUpdate completed (1.25wmf23) at 2015-04-07 02:40:43+00:00
• 02:35 logmsgbot: l10nupdate Synchronized php-1.25wmf23/cache/l10n: (no message) (duration: 09m 00s)
• 01:47 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1035, warm up (duration: 00m 12s)
• 01:14 mutante: rbf2001/rbf2002 - stop redis server
• 01:11 mutante: rbf eqiad and codfw - disable puppet (T95153)
• 00:54 mutante: haedus/capella - shutdown -h
• 00:40 mutante: radon: revoke salt key, puppet cert
• 00:39 logmsgbot: krenair Synchronized php-1.25wmf23/includes/Title.php: debug logging - https://gerrit.wikimedia.org/r/#/c/202218/ (duration: 00m 11s)
• 00:36 logmsgbot: krenair Synchronized php-1.25wmf24/includes/Title.php: debug logging - https://gerrit.wikimedia.org/r/#/c/202290/ (duration: 00m 15s)
• 00:34 mutante: haedus/capella: disabling puppet. reclaim

April 6

• 23:56 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/202071/ (duration: 00m 14s)
• 23:53 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/194084/ (duration: 00m 12s)
• 23:52 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/201196/ (duration: 00m 11s)
• 23:38 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/201943/ - wgUploadNavigationUrl for iswiki (duration: 00m 12s)
• 23:27 logmsgbot: krenair Synchronized flow.dblist: https://gerrit.wikimedia.org/r/#/c/202256/ - flow to wikidatawiki (duration: 00m 12s)
• 20:52 gwicke: deployed restbase 42db7c422f
• 20:47 gwicke: deploying restbase 42db7c422f
• 20:17 arlolra: updated Parsoid to version d5aa726ebe831e6e7d3343f1dd01d8cc11fba1c3
• 19:33 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/202094/ - should basically be a no-op for now (duration: 00m 13s)
• 17:38 nuria: restarted eventlogging to deal with log issues
• 17:01 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: done (duration: 00m 14s)
• 17:00 logmsgbot: krenair Synchronized php-1.25wmf23/includes/libs/MapCacheLRU.php: ok, done (duration: 00m 12s)
• 16:59 logmsgbot: krenair Synchronized php-1.25wmf23/includes/libs/MapCacheLRU.php: debug logging (duration: 00m 12s)
• 16:57 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: debug logging (duration: 00m 12s)
• 16:50 logmsgbot: aude Synchronized php-1.25wmf23/extensions/Wikidata: Fix editlinks bug in client (duration: 00m 21s)
• 16:44 logmsgbot: aude Synchronized php-1.25wmf24/extensions/Wikidata: Fix editlinks bug in client (duration: 00m 21s)
• 16:40 logmsgbot: aude Synchronized php-1.25wmf24/extensions/Wikidata: Update property suggester, valueview and fix editlinks bug in client (duration: 00m 19s)
• 16:33 logmsgbot: aude Synchronized php-1.25wmf23/includes/profiler/TransactionProfiler.php: Track request method in dbperformance.log (duration: 00m 13s)
• 16:31 logmsgbot: aude Synchronized php-1.25wmf24/includes/profiler/TransactionProfiler.php: Track request method in dbperformance.log (duration: 00m 12s)
• 16:12 paravoid: removing higher metric for eqiad-ulsfo GTT link
• 15:50 logmsgbot: anomie Synchronized php-1.25wmf23/extensions/MobileFrontend/: SWAT: MobileFrontend: Debounce resize events gerrit:201840 (duration: 00m 12s)
• 15:47 logmsgbot: anomie Synchronized php-1.25wmf24/extensions/UploadWizard/: SWAT: Backport UploadWizard bugfix (duration: 00m 12s)
• 15:36 logmsgbot: anomie Synchronized wmf-config/: SWAT: Enable ContentTranslation in the Vietnamese and Gujarati Wikipedia, and sync some other changes that naughty people didn't sync themselves but say are safe. (duration: 00m 12s)
• 15:22 logmsgbot: manybubbles Synchronized php-1.25wmf23/includes/page/WikiPage.php: SWAT try and catch funky revision errors 2/2 (duration: 00m 13s)
• 15:21 logmsgbot: manybubbles Synchronized php-1.25wmf23/includes/Revision.php: SWAT try and catch funky revision errors 1/2 (duration: 00m 12s)
• 15:20 logmsgbot: manybubbles Synchronized php-1.25wmf24/includes/page/WikiPage.php: SWAT try and catch funky revision errors 2/2 (duration: 00m 12s)
• 15:19 logmsgbot: manybubbles Synchronized php-1.25wmf24/includes/Revision.php: SWAT try and catch funky revision errors 1/2 (duration: 00m 12s)
• 15:03 logmsgbot: manybubbles Finished scap: earyly-SWAT: Ukrainian translations for EducationProgram (duration: 40m 52s)
• 14:22 logmsgbot: manybubbles Started scap: earyly-SWAT: Ukrainian translations for EducationProgram
• 04:46 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Apr 6 04:45:00 UTC 2015 (duration 44m 59s)
• 02:43 logmsgbot: LocalisationUpdate completed (1.25wmf24) at 2015-04-06 02:42:42+00:00
• 02:39 logmsgbot: l10nupdate Synchronized php-1.25wmf24/cache/l10n: (no message) (duration: 06m 06s)
• 02:22 logmsgbot: LocalisationUpdate completed (1.25wmf23) at 2015-04-06 02:21:26+00:00
• 02:17 logmsgbot: l10nupdate Synchronized php-1.25wmf23/cache/l10n: (no message) (duration: 06m 26s)

April 5

• 06:58 paravoid: double ospf/ospf3 metric for eqiad-ulsfo GTT link; switch to other transport link
• 04:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Apr 5 04:25:52 UTC 2015 (duration 25m 51s)
• 02:43 logmsgbot: LocalisationUpdate completed (1.25wmf24) at 2015-04-05 02:42:47+00:00
• 02:39 logmsgbot: l10nupdate Synchronized php-1.25wmf24/cache/l10n: (no message) (duration: 05m 55s)
• 02:22 logmsgbot: LocalisationUpdate completed (1.25wmf23) at 2015-04-05 02:21:46+00:00
• 02:18 logmsgbot: l10nupdate Synchronized php-1.25wmf23/cache/l10n: (no message) (duration: 06m 23s)

April 4

• 05:35 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Apr 4 05:34:27 UTC 2015 (duration 34m 26s)
• 02:56 logmsgbot: LocalisationUpdate completed (1.25wmf24) at 2015-04-04 02:54:57+00:00
• 02:51 logmsgbot: l10nupdate Synchronized php-1.25wmf24/cache/l10n: (no message) (duration: 05m 58s)
• 02:35 logmsgbot: LocalisationUpdate completed (1.25wmf23) at 2015-04-04 02:34:07+00:00
• 02:28 logmsgbot: l10nupdate Synchronized php-1.25wmf23/cache/l10n: (no message) (duration: 09m 10s)

April 3

• 23:22 bd808: Updated scap to a1a5235 (Add a logo banner to scap)
• 23:10 bd808: updated iegreview to aef8b1e (Use proper label for campaign selector)
• 22:59 ori: Graceful'd Apache on Zirconium for change 3813520 to iegreview (Stop using persistent db connections)
• 22:57 bd808: Updated iegreview to 3813520 (Stop using persistent db connections)
• 20:51 YuviPanda: restart ircecho on neon to test)
• 20:06 mutante: ruthenium - running puppet, no issues (has not for 7 days but wasn't disabled either?)
• 19:50 mutante: restarting gitblit
• 19:42 logmsgbot: twentyafterfour Synchronized php-1.25wmf24/extensions/OpenStackManager/nova/OpenStackNovaUser.php: sync security patch (duration: 00m 12s)
• 18:55 mutante: restarted grrrit-wm for config change
• 18:40 ori: Restarted nutcracker on HHVM and mw1147 and repooled
• 18:35 ori: Depooled mw1147. Spamming fluorine:/a/mw-log/memcache-serious.log. Some nutcracker issue most likely.
• 17:49 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I9c4de264: Send server side eventlogging logs to eventlog1001 instead of vanadium (duration: 00m 11s)
• 15:14 logmsgbot: kartik Synchronized php-1.25wmf24/extensions/ContentTranslation: (no message) (duration: 00m 14s)
• 15:14 logmsgbot: kartik Synchronized php-1.25wmf23/extensions/ContentTranslation: (no message) (duration: 00m 17s)
• 11:14 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: db1049 to normal load (duration: 00m 11s)
• 10:37 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1049, warm up (duration: 00m 12s)
• 10:32 paravoid: depooled mw1234
• 10:20 paravoid: staggered restart of the API cluster (sans mw1234, left for further debugging)
• 09:32 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1049 (duration: 00m 20s)
• 09:24 springle: mw1114 critical, no ssh, no console, powercycle
• 09:19 springle: tin sync-file: mw1114.eqiad.wmnet returned [-15]
• 09:18 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: reduce db1049 load (duration: 06m 26s)
• 04:56 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Apr 3 04:55:08 UTC 2015 (duration 55m 7s)
• 03:31 logmsgbot: mattflaschen Synchronized php-1.25wmf24/includes/libs/normal/UtfNormalUtil.php: Fix UtfNormal shim so account creations work (duration: 00m 12s)
• 03:29 logmsgbot: mattflaschen Synchronized php-1.25wmf24/includes/libs/normal/UtfNormalUtil.php: Fix UtfNormal shim so account creations work (duration: 00m 12s)
• 03:03 logmsgbot: LocalisationUpdate completed (1.25wmf24) at 2015-04-03 03:02:30+00:00
• 02:58 logmsgbot: l10nupdate Synchronized php-1.25wmf24/cache/l10n: (no message) (duration: 06m 00s)
• 02:38 logmsgbot: LocalisationUpdate completed (1.25wmf23) at 2015-04-03 02:37:21+00:00
• 02:31 logmsgbot: l10nupdate Synchronized php-1.25wmf23/cache/l10n: (no message) (duration: 09m 08s)
• 02:03 YuviPanda: restarted hhvm on mw1209
• 02:02 YuviPanda: restarted hhvm on mw1249 and mw1065
• 01:54 logmsgbot: catrope Synchronized w: (no message) (duration: 00m 12s)
• 01:30 logmsgbot: ori Synchronized php-1.25wmf24/extensions/ConfirmEdit: 7cb7ef4e6f: Update ConfirmEdit for Id4798364d (duration: 00m 12s)
• 01:11 logmsgbot: catrope Synchronized php-1.25wmf24/extensions/ContentTranslation/modules/campaigns/ext.cx.campaigns.contributionsmenu.js: touch (duration: 00m 13s)
• 01:11 logmsgbot: catrope Synchronized php-1.25wmf23/extensions/ContentTranslation/modules/campaigns/ext.cx.campaigns.contributionsmenu.js: touch (duration: 00m 12s)
• 01:08 logmsgbot: catrope Synchronized php-1.25wmf23/includes: SWAT (duration: 00m 15s)
• 01:06 logmsgbot: catrope Synchronized php-1.25wmf23/autoload.php: SWAT (duration: 00m 12s)
• 01:04 logmsgbot: catrope Synchronized php-1.25wmf24/includes: SWAT (duration: 00m 15s)
• 01:03 logmsgbot: catrope Synchronized php-1.25wmf24/autoload.php: (no message) (duration: 00m 11s)
• 01:01 logmsgbot: catrope Synchronized php-1.25wmf24/extensions/Gather: SWAT (duration: 00m 13s)
• 01:00 ori: restart HHVM on mw1120
• 00:48 logmsgbot: catrope Synchronized php-1.25wmf24/extensions/VisualEditor: SWAT (duration: 00m 12s)
• 00:47 logmsgbot: catrope Synchronized php-1.25wmf24/extensions/Flow: SWAT (duration: 00m 14s)
• 00:47 logmsgbot: catrope Synchronized php-1.25wmf24/extensions/ConfirmEdit: SWAT (duration: 00m 13s)
• 00:47 logmsgbot: catrope Synchronized php-1.25wmf23/extensions/Flow: SWAT (duration: 00m 12s)
• 00:47 logmsgbot: catrope Synchronized php-1.25wmf23/extensions/ConfirmEdit: SWAT (duration: 00m 11s)
• 00:44 logmsgbot: catrope Synchronized php-1.25wmf23/extensions/Gather: SWAT (duration: 00m 11s)

April 2

• 23:11 mutante: temp. disabling puppet on restbase servers
• 22:50 bd808: lots of SYSTEM ERROR responses from nutcracker on mw1147
• 22:13 greg-g: Account creation is broken/not working for either iOS or Android WP apps, investigation in -mobile
• 19:37 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: I3bbf2418d: Set $wgLogoHD for enwiki (duration: 00m 12s)
• 18:52 mutante: running puppet on mw2095 - proxy error
• 18:43 RoanKattouw: Running clearMessageBlobs.php
• 18:37 logmsgbot: demon Synchronized wmf-config/CirrusSearch-labs.php: cleanups for labs, no-op (duration: 00m 12s)
• 18:37 logmsgbot: demon Synchronized wmf-config/CirrusSearch-common.php: turn off "yay new search!!" msg. old news now (duration: 00m 11s)
• 17:15 logmsgbot: kartik Synchronized php-1.25wmf23/extensions/ContentTranslation/modules/campaigns/ext.cx.campaigns.contributionsmenu.js: (no message) (duration: 00m 15s)
• 16:52 logmsgbot: kartik Finished scap: Update ContentTranslation (duration: 40m 01s)
• 16:25 godog: reload uwsgi on graphite1001
• 16:12 logmsgbot: kartik Started scap: Update ContentTranslation
• 15:51 manybubbles: actually that last patch seems to be working too. cool. sweet. still running the cirrus script just in case.
• 15:50 manybubbles: last sync accidentally picked up 'Add 100/106 namespaces to be searched by default at frwiktionary' - that one might require a cirrus script to finish running before its working properly
• 15:49 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT Set $wgRestrictDisplayTitle to false at cawikimedia (duration: 00m 11s)
• 15:42 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT disable mobile ip editing at kowiki 2/2 (duration: 00m 12s)
• 15:42 logmsgbot: manybubbles Synchronized wmf-config/CommonSettings.php: SWAT disable mobile ip editing at kowiki 1/2 (duration: 00m 11s)
• 15:40 logmsgbot: manybubbles Synchronized php-1.25wmf23/extensions/OpenStackManager/: SWAT update openstackmanager extension (duration: 00m 14s)
• 15:36 logmsgbot: manybubbles Synchronized php-1.25wmf24/extensions/OpenStackManager/: SWAT update openstackmanager extension (duration: 00m 11s)
• 15:18 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT enable newarticle campaign on cawiki 3/3 (duration: 00m 14s)
• 15:17 logmsgbot: manybubbles Synchronized wmf-config/CommonSettings-labs.php: SWAT enable newarticle campaign on cawiki 2/3 (duration: 00m 12s)
• 15:17 logmsgbot: manybubbles Synchronized wmf-config/CommonSettings.php: SWAT enable newarticle campaign on cawiki 1/3 (duration: 00m 13s)
• 15:16 manybubbles: ignore last log - its a noop failure on my part
• 15:16 logmsgbot: manybubbles Synchronized wmf-config/CommonSettings.php: SWAT enable newarticle campaign on cawiki 1/3 (duration: 00m 12s)
• 15:15 logmsgbot: manybubbles Synchronized php-1.25wmf23/includes/User.php: SWAT user preferences load from the master by default (duration: 00m 12s)
• 14:13 hashar: Jenkins: migrated Zuul cloner on Precise labs slaves (100[1-4] to a version provided by a Debian package. Jobs console output should now shows Zuul version: 2.0.0-304-g685ca22-wmf1precise1
• 12:51 andrewbogott: restarted opendj, pdns on neptunium, nembus, virt1000, labcontrol2001
• 12:48 paravoid: repooling esams
• 12:42 paravoid: upgrading junos on mr1-esams
• 12:15 mark: Shutting down cp3014 for 10G upgrade
• 11:02 mark: Shutting down cp3012 for 10G upgrade
• 10:38 _joe_: stopping pybal on lvs2003, running manually to help debugging
• 09:43 paravoid: asw-d-eqiad: routing-engine backup switch FPC 8 -> FPC 4
• 09:42 paravoid: asw-d-eqiad: routing-engine backup switch FPC 7 -> FPC 5, master switchover FPC 8 -> FPC 5
• 08:31 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Apr 2 08:30:42 UTC 2015 (duration 30m 41s)
• 07:48 springle: on sync-file from tin: mw2213.codfw.wmnet returned [255]: Host key verification failed
• 07:46 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1027, warm up (duration: 00m 11s)
• 06:40 bblack: re-depooled esams ...
• 06:34 Jamesofur: manually merged Ximilian global account per request and account confirmation
• 06:13 bblack: re-pooling esams (GTT event never happened AFAICS)
• 05:13 springle: xtrabackup clone db1027 to db1035
• 03:45 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1027 (duration: 00m 11s)
• 03:31 springle: reinstall db1035, reclone data
• 03:18 logmsgbot: LocalisationUpdate completed (1.25wmf24) at 2015-04-02 03:08:52+00:00
• 03:03 logmsgbot: l10nupdate Synchronized php-1.25wmf24/cache/l10n: (no message) (duration: 08m 55s)
• 02:41 logmsgbot: l10nupdate Synchronized php-1.25wmf23/cache/l10n: (no message) (duration: 08m 56s)
• 02:06 legoktm: set email for TheFons@global and attached nlwiki
• 00:03 logmsgbot: krenair Synchronized php-1.25wmf24/extensions/OpenStackManager/nova/OpenStackNovaHost.php: https://gerrit.wikimedia.org/r/201386 (duration: 00m 12s)
• 00:01 logmsgbot: krenair Synchronized php-1.25wmf23/extensions/OpenStackManager/nova/OpenStackNovaHost.php: https://gerrit.wikimedia.org/r/201385 (duration: 00m 13s)

April 1

• 23:59 bblack: depooling esams ahead of 2h planned GTT link outage coming up in 1h
• 23:41 logmsgbot: krenair Synchronized wmf-config: https://gerrit.wikimedia.org/r/200098 (duration: 00m 12s)
• 23:37 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/201374/ (duration: 00m 12s)
• 23:35 logmsgbot: krenair Synchronized php-1.25wmf24/extensions/Flow: https://gerrit.wikimedia.org/r/#/c/201360/ (duration: 00m 13s)
• 23:33 logmsgbot: krenair Synchronized php-1.25wmf24/extensions/Flow/modules/editor/editors/visualeditor/ext.flow.editors.visualeditor.js: https://gerrit.wikimedia.org/r/#/c/201360/ (duration: 00m 11s)
• 23:29 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/201093/ (duration: 00m 12s)
• 23:23 logmsgbot: krenair Synchronized php-1.25wmf23/extensions/OpenStackManager/nova/OpenStackNovaHost.php: https://gerrit.wikimedia.org/r/#/c/201367/ (duration: 00m 15s)
• 23:02 logmsgbot: krenair Synchronized wmf-config/wikitech.php: https://gerrit.wikimedia.org/r/#/c/201197/ (duration: 00m 11s)
• 22:59 awight: update payments from f617326761887ed9a9100b472ea3b5736e2c10e6 to d37687239fa79842c0d6ea65e9230a3f14cda867
• 22:57 logmsgbot: twentyafterfour Purged l10n cache for 1.25wmf22
• 22:57 logmsgbot: kaldari Synchronized wmf-config/InitialiseSettings.php: enabling Gather on enwiki (duration: 00m 13s)
• 22:48 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf24
• 22:31 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.25wmf23
• 22:14 logmsgbot: twentyafterfour Finished scap: once again: testwiki to 1.25wmf24 and rebuild l10n cache (duration: 54m 14s)
• 21:56 bblack: repooled cp107[1234] in pybal (eqiad upload, row D)
• 21:20 logmsgbot: twentyafterfour Started scap: once again: testwiki to 1.25wmf24 and rebuild l10n cache
• 21:19 logmsgbot: twentyafterfour scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_186708348" --threads=4 --lang en --quiet' returned non-zero exit status 255 (duration: 00m 46s)
• 21:18 logmsgbot: twentyafterfour Started scap: testwiki to 1.25wmf24 and rebuild l10n cache
• 20:32 gwicke: finished RESTBase 0.5.0 deployment
• 20:28 gwicke: deploying RESTBase 0.5.0
• 20:22 logmsgbot: twentyafterfour scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_3136974758" --threads=4 --lang en --quiet' returned non-zero exit status 255 (duration: 02m 53s)
• 20:19 logmsgbot: twentyafterfour Started scap: retrying: testwiki to php-1.25wmf24 and rebuild l10n cache
• 20:08 YuviPanda: run chmod -R g+w . on tin with CWD /srv/deployment/scap/scap/.git
• 19:48 logmsgbot: twentyafterfour scap failed: CalledProcessError Command 'cp '/srv/mediawiki-staging/php-1.25wmf24/cache/l10n/'*.cdb '/tmp/scap_l10n_1816369030 returned non-zero exit status 1 (duration: 14m 20s)
• 19:33 logmsgbot: twentyafterfour Started scap: testwiki to php-1.25wmf24 and rebuild l10n cache
• 19:18 paravoid: upgrading cr1/2-eqiad<->asw-d-eqiad capacity (T92914)
• 17:55 mutante: added jmm to ops and wmf LDAP groups
• 17:20 _joe_: restarted hhvm on mw1194, stuck in HPHP::StatCache::refresh
• 17:12 ottomata: initiated kafka replica election
• 17:04 paravoid: repooling esams
• 16:48 bd808: Updated Wikimania Scholarships to bde1a27 (Improve performance of phase2 report query)
• 16:09 godog: bounce cassandra on test cluster
• 16:03 logmsgbot: thcipriani Synchronized php-1.25wmf23/extensions/UniversalLanguageSelector: swat gerrit:201122 (duration: 01m 07s)
• 15:58 paravoid: upgrading junos on cr1-esams (esams is depooled, ignore alerts)
• 15:48 logmsgbot: phuedx Synchronized php-1.25wmf22/extensions/Gather/: Updating the Gather extension for 1.25wmf22 (duration: 01m 06s)
• 15:47 bd808: ssh: connect to host mw2213.codfw.wmnet port 22: Connection timed out during sync-dir initiated from tin
• 15:45 logmsgbot: phuedx Synchronized php-1.25wmf23/extensions/Gather: Updating the Gather extension for 1.25wmf23 (duration: 01m 07s)
• 15:33 godog: disable puppet on xenon, praseodymium, cerium, restbase* to test https://gerrit.wikimedia.org/r/197840
• 15:31 ^d: ran sync-common on mw1017 for testwiki fun
• 14:35 paravoid: upgrading junos on cr2-knams (esams is depooled, ignore alerts)
• 14:32 mark: mark@csw2-esams> request system power-off member 1
• 14:28 mark: asw-esams: mark@asw-esams> request system power-off member 3
• 14:20 mark: Shutting down lvs3003 and lvs3004
• 12:08 paravoid: draining esams
• 07:28 godog: bounce apertium-apy on sca1001/sca1002
• 06:43 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Apr 1 06:42:24 UTC 2015 (duration 42m 23s)
• 06:10 _joe_: manually rotated and compressed syslog and apache logs on uranium, still being spammed by kafka brokers
• 03:05 logmsgbot: LocalisationUpdate completed (1.25wmf23) at 2015-04-01 03:04:00+00:00
• 02:58 logmsgbot: l10nupdate Synchronized php-1.25wmf23/cache/l10n: (no message) (duration: 08m 41s)
• 02:35 logmsgbot: LocalisationUpdate completed (1.25wmf22) at 2015-04-01 02:34:41+00:00
• 02:29 logmsgbot: l10nupdate Synchronized php-1.25wmf22/cache/l10n: (no message) (duration: 08m 46s)
• 01:35 legoktm: started zuul on gallium
• 00:40 urandom: restarting cassandra on restbase1006
• 00:40 urandom: restarting cassandra on restbase1005
• 00:35 urandom: restarting cassandra on restbase1004
• 00:33 urandom: restarting cassandra on restbase1003
• 00:30 urandom: restarting cassandra on restbase1002
• 00:12 logmsgbot: catrope Synchronized php-1.25wmf23/extensions/Gather: SWAT (duration: 01m 07s)
• 00:11 RoanKattouw: ssh: connect to host mw2213.codfw.wmnet port 22: Connection timed out
• 00:11 logmsgbot: catrope Synchronized php-1.25wmf23/extensions/ImageMetrics: SWAT (duration: 01m 07s)
• 00:10 logmsgbot: catrope Synchronized php-1.25wmf23/extensions/VisualEditor: SWAT (duration: 01m 07s)
• 00:05 logmsgbot: catrope Synchronized php-1.25wmf22/extensions/Gather: SWAT (duration: 01m 06s)

March 31

• 23:15 urandom: restarting cassandra on restbase1001
• 18:39 awight: rollback crm from b4268a60225ae11f2c2b58d3b1f1c44e282f9ec6 to 59f03df6b689ef443cc7b7e31e6f5b2986bc8bc9
• 18:09 twentyafterfour: mw2213.codfw.wmnet still timing out
• 18:07 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group1 to $VERSION, mw2213.codfe.wmnet failed, trying one more time
• 18:06 twentyafterfour: sync_wikiversions failed for host mw2213.codfw.wmnet port 22: Connection timed out
• 18:04 twentyafterfour: group1 to VERSION=1.25wmf23
• 18:03 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group1 to VERSION
• 17:54 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/200901/ (duration: 01m 06s)
• 17:20 awight: update crm from 0e4a6ed961ca5a5882d42949c12f74c1d246b55e to b4268a60225ae11f2c2b58d3b1f1c44e282f9ec6
• 16:58 mutante: installing many package upgrades on wikitech-static
• 16:58 awight: updated crm from 4c459f3dbf3c3466cdc26a351ba589f4f1aef587 to 0e4a6ed961ca5a5882d42949c12f74c1d246b55e
• 16:04 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/196885/ (duration: 01m 06s)
• 15:55 robh: ignore any alerts for wtp2001-2020, reinstalling
• 15:37 logmsgbot: marktraceur Synchronized wmf-config/db-codfw.php: [SWAT] [config] Remove another useless wgMasterWaitTimeout reference (duration: 01m 07s)
• 15:32 logmsgbot: marktraceur Synchronized wmf-config/InitialiseSettings.php: [SWAT] [config] Enable ContentTranslation in bg, fr, mk, sh, and sl (duration: 01m 06s)
• 15:28 logmsgbot: marktraceur Synchronized wmf-config/CommonSettings.php: [SWAT] [config] T90704: = true; (duration: 01m 13s)
• 15:24 logmsgbot: marktraceur Synchronized wmf-config/CommonSettings.php: [SWAT] [config] Make Spam Blacklist global file protocol-relative (duration: 01m 13s)
• 15:21 logmsgbot: marktraceur Synchronized wmf-config/InitialiseSettings.php: [SWAT] [config] Add autopatrol protection level to lvwiki (duration: 01m 13s)
• 15:10 logmsgbot: marktraceur Synchronized wmf-config/InitialiseSettings.php: [SWAT] [config] Set to 0.30 at commons (duration: 01m 13s)
• 15:05 logmsgbot: marktraceur Synchronized wmf-config/InitialiseSettings.php: [SWAT] [config] Add Draft namespace on zhwiki (duration: 01m 13s)
• 12:38 Coren: labstore1001 is having issues; preparing to switchover to 1002
• 12:14 _joe_: restarted keystone
• 11:38 godog: swift eqiad-prod ms-be101[678] weight to 80 (account/container)
• 11:19 _joe_: running sync-common on codfw hosts, since they will be back into scap today
• 11:00 godog: swift eqiad-prod ms-be101[678] weight to 1600
• 10:08 godog: powercycle fluorine, unresponsive
• 10:06 godog: fluorine unresponsive after moving files from /a to /srv T94396
• 10:06 _joe_: restarting pybal on lvs2003 for testing
• 07:18 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Mar 31 07:15:22 UTC 2015 (duration 15m 21s)
• 02:53 logmsgbot: LocalisationUpdate completed (1.25wmf23) at 2015-03-31 02:52:30+00:00
• 02:48 logmsgbot: l10nupdate Synchronized php-1.25wmf23/cache/l10n: (no message) (duration: 06m 31s)
• 02:28 logmsgbot: l10nupdate Synchronized php-1.25wmf22/cache/l10n: (no message) (duration: 06m 27s)
• 01:58 logmsgbot: maxsem Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/200799/ (duration: 00m 09s)
• 01:53 logmsgbot: kaldari Synchronized php-1.25wmf23/extensions/Gather: syncing Gather on 1.25wmf23 (duration: 00m 09s)
• 01:51 logmsgbot: kaldari Synchronized php-1.25wmf22/extensions/Gather: syncing Gather on 1.25wmf22 (duration: 00m 09s)
• 00:54 logmsgbot: maxsem Synchronized php-1.25wmf23/includes/Import.php: https://gerrit.wikimedia.org/r/#/c/200771/ (duration: 00m 06s)

March 30

• 23:34 logmsgbot: maxsem Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/200736/ (duration: 00m 08s)
• 23:34 logmsgbot: maxsem Synchronized flow.dblist: Enable on zhwiki (duration: 00m 08s)
• 23:32 logmsgbot: maxsem Synchronized wmf-config/InitialiseSettings.php: touch (duration: 00m 08s)
• 23:28 logmsgbot: maxsem Synchronized flow.dblist: Enable on zhwiki (duration: 00m 07s)
• 23:26 logmsgbot: maxsem Synchronized php-1.25wmf22/extensions/Flow/: (no message) (duration: 00m 09s)
• 23:24 logmsgbot: maxsem Synchronized php-1.25wmf23/extensions/Flow/: (no message) (duration: 00m 09s)
• 23:14 logmsgbot: maxsem Synchronized wmf-config/InitialiseSettings.php: Gather on enwiki (duration: 00m 09s)
• 23:12 logmsgbot: maxsem Synchronized php-1.25wmf22/extensions/Gather/: (no message) (duration: 00m 08s)
• 23:11 logmsgbot: maxsem Synchronized php-1.25wmf23/extensions/Gather/: (no message) (duration: 00m 07s)
• 23:11 logmsgbot: maxsem Synchronized php-1.25wmf23/extensions/MobileFrontend/: (no message) (duration: 00m 07s)
• 23:10 logmsgbot: maxsem Synchronized php-1.25wmf23/extensions/Gather/: (no message) (duration: 00m 08s)
• 22:53 MaxSem: Created Gather tables on enwiki
• 21:28 logmsgbot: aude Synchronized php-1.25wmf22/extensions/Wikidata: Fix JS bugs and change dispatcher issues (duration: 00m 15s)
• 21:23 logmsgbot: aude Synchronized php-1.25wmf23/extensions/Wikidata: Fix JS bugs and change dispatcher issues (duration: 00m 14s)
• 21:12 Coren: Labs filesystem switch in progress - not as smooth as I would have liked. On it.
• 20:09 subbu: deployed parsoid sha 29a5dafb
• 19:12 mutante: subra/suhail: re-add to puppet, initial runs
• 17:56 mutante: subra, wmf-reimage
• 17:53 mutante: suhail, rebooting to fix BIOS settings, reinstall
• 17:29 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/200318/ (duration: 00m 06s)
• 17:20 mutante: subra - rebooting for reinstall
• 17:04 robh: faidon and alex are working on carbon, puppet is disabled
• 16:59 logmsgbot: aaron Synchronized php-1.25wmf23/includes/User.php: I3b733a0221462350f3a24d54ffe814357f379512 (duration: 00m 06s)
• 16:57 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/199854/ (duration: 00m 08s)
• 15:52 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/198220/ - BounceHandler to Wikipedias (duration: 00m 07s)
• 15:50 logmsgbot: krenair Synchronized php-1.25wmf23/resources/src/mediawiki.action/mediawiki.action.edit.preview.js: https://gerrit.wikimedia.org/r/#/c/200044/ (duration: 00m 06s)
• 15:33 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/198691/ (duration: 00m 06s)
• 15:31 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/198691/ (duration: 00m 08s)
• 15:30 paravoid: reimaging nescio
• 15:06 logmsgbot: krenair Synchronized wmf-config/logging.php: https://gerrit.wikimedia.org/r/#/c/200286/2 - should be a no-op (duration: 00m 08s)
• 15:02 logmsgbot: krenair Synchronized visualeditor-default.dblist: https://gerrit.wikimedia.org/r/#/c/196984/ - VE phase 5 (duration: 00m 07s)
• 14:04 godog: reload apache on iodine
• 06:25 springle: db1035 restart failed, root fs errors
• 06:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Mar 30 06:06:26 UTC 2015 (duration 6m 25s)
• 05:53 springle: upgrade db1035 trusty
• 05:34 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1035 (duration: 00m 07s)
• 05:21 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1044, warm up (duration: 00m 07s)
• 05:17 springle: upgrade db1044 trusty
• 05:17 springle: restarted production logbot
• 05:15 YuviPanda: restarted apache on silver
• 02:43 logmsgbot: LocalisationUpdate completed (1.25wmf23) at 2015-03-30 02:42:03+00:00
• 02:40 logmsgbot: l10nupdate Synchronized php-1.25wmf23/cache/l10n: (no message) (duration: 03m 52s)
• 02:28 logmsgbot: LocalisationUpdate completed (1.25wmf22) at 2015-03-30 02:27:18+00:00
• 02:24 logmsgbot: l10nupdate Synchronized php-1.25wmf22/cache/l10n: (no message) (duration: 05m 06s)
• 01:14 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1044 (duration: 00m 09s)

March 29

• 06:00 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Mar 29 05:59:39 UTC 2015 (duration 59m 38s)
• 02:35 logmsgbot: LocalisationUpdate completed (1.25wmf23) at 2015-03-29 02:34:28+00:00
• 02:32 logmsgbot: l10nupdate Synchronized php-1.25wmf23/cache/l10n: (no message) (duration: 03m 20s)
• 02:22 logmsgbot: LocalisationUpdate completed (1.25wmf22) at 2015-03-29 02:21:25+00:00
• 02:19 logmsgbot: l10nupdate Synchronized php-1.25wmf22/cache/l10n: (no message) (duration: 05m 22s)

March 28

• 23:04 hoo: Gave sysop and checkuser to Jalexander@labswiki via shell from silver after doing it via meta failed. (T94319)
• 15:06 andrewbogott: and restarted keystone on virt1000
• 15:06 andrewbogott: graceful’d apache2 on virt1000
• 10:45 godog: powercycle ms-be1009
• 07:53 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Mar 28 07:51:56 UTC 2015 (duration 51m 55s)
• 02:52 logmsgbot: LocalisationUpdate completed (1.25wmf23) at 2015-03-28 02:51:45+00:00
• 02:48 logmsgbot: l10nupdate Synchronized php-1.25wmf23/cache/l10n: (no message) (duration: 06m 50s)
• 02:27 logmsgbot: LocalisationUpdate completed (1.25wmf22) at 2015-03-28 02:26:33+00:00
• 02:23 logmsgbot: l10nupdate Synchronized php-1.25wmf22/cache/l10n: (no message) (duration: 07m 03s)
• 00:20 hoo: Attached local accounts to "Advance", per request: enwiki, commonswiki, metawiki, nlwiktionary and nlwikinews

March 27

• 22:17 robh: manually restarted zotero service on sca100[1-2]
• 21:18 logmsgbot: demon Synchronized wmf-config/CommonSettings-labs.php: for completeness (duration: 00m 09s)
• 21:10 csteipp: redeploy security patches to wmf22
• 21:09 logmsgbot: demon Synchronized wmf-config/logging-labs.php: shut up icinga, you're drunk (duration: 00m 07s)
• 21:02 csteipp: redeploy security patches to wmf23
• 20:10 gwicke: thinning out old renders in restbase, keeping only the latest per revision; starting with group0, followed by wikipedia once done
• 16:59 mutante: mount /mnt/data on praseodymium to fix cassandra
• 14:28 _joe_: restarted mw1034, stuck in HPHP::StatCache::refresh
• 11:40 godog: reboot ms-be1009, xfs stuck
• 07:15 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Mar 27 07:14:37 UTC 2015 (duration 14m 36s)
• 02:46 logmsgbot: LocalisationUpdate completed (1.25wmf23) at 2015-03-27 02:45:22+00:00
• 02:43 logmsgbot: l10nupdate Synchronized php-1.25wmf23/cache/l10n: (no message) (duration: 03m 08s)
• 02:27 logmsgbot: LocalisationUpdate completed (1.25wmf22) at 2015-03-27 02:26:36+00:00
• 02:24 logmsgbot: l10nupdate Synchronized php-1.25wmf22/cache/l10n: (no message) (duration: 05m 03s)
• 01:01 awight: updated payments from 32e860bd304763ccedc7110dee828249daa2b154 to f617326761887ed9a9100b472ea3b5736e2c10e6
• 00:10 gwicke: updated fstab data array name from md2 to md127 on cerium, xenon and praseodymium; naming changed after reboot; should probably use uuid instead
• 00:02 mutante: remounted /mnt/data on xenon

March 26

• 23:47 logmsgbot: ebernhardson Synchronized php-1.25wmf22/extensions/EventLogging/: Bump EventLogging in 1.25wmf22 for SWAT (duration: 00m 07s)
• 23:44 logmsgbot: ebernhardson Synchronized php-1.25wmf23/extensions/EventLogging/: Bump EventLogging in 1.25wmf23 for SWAT (duration: 00m 08s)
• 23:42 mutante: starting ferm service on holmium
• 23:40 ejegg: updated dash from 038bdc4c60697ac738eaeae384d91579710ff85a to 5a6b2dda71e6ce76d7bbba853acae8dc9416052c
• 23:34 mutante: cerium, xenon, praseodymium - stuck at boot because /mnt/data not ready, skipped mounting to reboot
• 23:26 gwicke: rebooted xenon, cerium, praseodymium to reload the firewall from scratch
• 23:23 logmsgbot: ebernhardson Synchronized php-1.25wmf22/extensions/Flow: Bump flow submodule in 1.25wmf22 for swat (duration: 00m 08s)
• 23:21 logmsgbot: ebernhardson Synchronized php-1.25wmf23/extensions/Flow/: Bump flow submodule for 1.25wmf23 (duration: 00m 09s)
• 23:19 logmsgbot: ebernhardson Synchronized php-1.25wmf23/extensions/Flow/: Bump flow submodule for 1.25wmf23 (duration: 00m 09s)
• 22:39 logmsgbot: maxsem Synchronized wmf-config/InitialiseSettings.php: Gather on test & test2 (duration: 00m 07s)
• 22:36 logmsgbot: maxsem Synchronized php-1.25wmf23/extensions/Gather/: (no message) (duration: 00m 07s)
• 22:35 logmsgbot: maxsem Synchronized php-1.25wmf22/extensions/Gather/: (no message) (duration: 00m 08s)
• 22:21 logmsgbot: maxsem Finished scap: Enable Gather (duration: 31m 02s)
• 21:50 logmsgbot: maxsem Started scap: Enable Gather
• 21:39 MaxSem: Created Gather tables on test and test2
• 20:15 legoktm: set email for User:ThistleDew172@enwiki and attached to global
• 19:54 mutante: praseodymium - fix firewalling
• 19:51 mutante: praseodymium - log in via mgmt, run puppet to restore flushed iptables rules
• 19:34 bd808: Updated iegreview to 7797bfc (Change email address used for sending out grants-related mails) for T92391
• 19:33 bd808: Applied schema changes to iegreview@m2-master.eqiad.wmnet for T92391
• 18:48 paravoid: replacing pfw1-codfw/pfw2/codfw
• 18:08 logmsgbot: kartik Synchronized php-1.25wmf23/extensions/ContentTranslation: (no message) (duration: 00m 08s)
• 18:07 logmsgbot: kartik Synchronized php-1.25wmf22/extensions/ContentTranslation: (no message) (duration: 00m 08s)
• 17:24 logmsgbot: kartik Finished scap: Update ContentTranslation (duration: 33m 14s)
• 17:04 ejegg: updated dash from 6d9acd60bb833c6dd57ab8b424afc8077b0c9f03 to 038bdc4c60697ac738eaeae384d91579710ff85a
• 16:51 logmsgbot: kartik Started scap: Update ContentTranslation
• 15:13 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: SWAT gerrit:199823 (duration: 00m 06s)
• 09:57 godog: restart keystone on virt1000
• 06:55 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Mar 26 06:53:58 UTC 2015 (duration 53m 57s)
• 05:19 Krenair: Ran cleanup script for T92775
• 02:47 logmsgbot: LocalisationUpdate completed (1.25wmf23) at 2015-03-26 02:46:37+00:00
• 02:45 logmsgbot: l10nupdate Synchronized php-1.25wmf23/cache/l10n: (no message) (duration: 02m 57s)
• 02:29 logmsgbot: LocalisationUpdate completed (1.25wmf22) at 2015-03-26 02:28:47+00:00
• 02:26 logmsgbot: l10nupdate Synchronized php-1.25wmf22/cache/l10n: (no message) (duration: 05m 03s)
• 00:57 superm401: Done running FlowFixEditCount in production
• 00:03 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/199692/ (duration: 00m 07s)
• 00:02 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/194913/ (duration: 00m 08s)
• 00:01 logmsgbot: krenair Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/199574/ (duration: 00m 08s)

March 25

• 23:59 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/198786/ (duration: 00m 07s)
• 23:58 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/198753/ (duration: 00m 09s)
• 23:57 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/198749/ (duration: 00m 08s)
• 23:55 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/194908 (duration: 00m 07s)
• 23:53 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/193662/ (duration: 00m 07s)
• 23:52 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/198460/ (duration: 00m 08s)
• 23:50 logmsgbot: krenair Synchronized wmf-config: re-sync that last one... (duration: 00m 08s)
• 23:49 logmsgbot: krenair Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/156078/ (duration: 00m 07s)
• 23:40 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/199779/ (duration: 00m 08s)
• 23:35 logmsgbot: krenair Synchronized php-1.25wmf23/extensions/Flow/includes: https://gerrit.wikimedia.org/r/#/c/199684/1 (duration: 00m 09s)
• 23:26 logmsgbot: krenair Synchronized php-1.25wmf22/extensions/Flow: https://gerrit.wikimedia.org/r/#/c/199686/ (duration: 00m 09s)
• 23:23 logmsgbot: krenair Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/196068/ (duration: 00m 09s)
• 23:11 logmsgbot: krenair Synchronized wmf-config: trying again (duration: 00m 08s)
• 23:10 logmsgbot: krenair Synchronized flow.dblist: (oops) (duration: 00m 08s)
• 23:06 logmsgbot: krenair Synchronized wmf-config: rv (duration: 00m 07s)
• 23:03 logmsgbot: krenair Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/194809/ - dblist for flow (duration: 00m 08s)
• 22:57 logmsgbot: twentyafterfour Purged l10n cache for 1.25wmf21
• 22:56 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf23
• 22:51 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.25wmf22
• 22:50 mutante: disabled notifications for dsh group checks in icinga - reenable me after T93958
• 22:39 logmsgbot: twentyafterfour Finished scap: testwiki to php-1.25wmf23 (this time without codfw) (duration: 03m 33s)
• 22:35 logmsgbot: twentyafterfour Started scap: testwiki to php-1.25wmf23 (this time without codfw)
• 22:27 paravoid: reformatting berkelium & curium
• 22:18 logmsgbot: twentyafterfour Finished scap: testwiki to php-1.25wmf23 and rebuild l10n cache (attempt #2) (duration: 149m 25s)
• 22:12 bd808: robh killed the stuck scap ssh connections to codfw and scap moved on to the next step
• 22:01 bd808: scap sync-common step stuck with 58 codfw hosts not syncing at any reasonable speed
• 20:45 ejegg: updated dash from b2db5e415ec75d289a4da2e1dd6af4a1bf5ab9b1 to 6d9acd60bb833c6dd57ab8b424afc8077b0c9f03
• 20:05 subbu: deployed parsoid sha 0313fcc7
• 19:48 logmsgbot: twentyafterfour Started scap: testwiki to php-1.25wmf23 and rebuild l10n cache (attempt #2)
• 19:44 ejegg: updated dash from 393facdb6c1a53bfbac3246f2fd3e1c4f51dc1cc to b2db5e415ec75d289a4da2e1dd6af4a1bf5ab9b1
• 19:41 awight: updated payments from 7ffe008fb8964acb1382820d129d784d5b6dd9de to 32e860bd304763ccedc7110dee828249daa2b154
• 19:22 logmsgbot: twentyafterfour scap failed: CalledProcessError Command 'cp '/srv/mediawiki-staging/php-1.25wmf23/cache/l10n/'*.cdb '/tmp/scap_l10n_2482639127 returned non-zero exit status 1 (duration: 02m 19s)
• 19:20 logmsgbot: twentyafterfour Started scap: testwiki to php-1.25wmf23 and rebuild l10n cache
• 16:21 logmsgbot: aude Finished scap: Wikidata bug fixes and fix rollback bug in core (duration: 23m 01s)
• 15:58 logmsgbot: aude Started scap: Wikidata bug fixes and fix rollback bug in core
• 15:19 bd808: Updated scap to include 4a63a63 (Copy l10n CDB files to rebuildLocalisationCache.php tmp dir)
• 15:19 bd808: trebuchet checkout of scap failed on mw1113, mw1222, and mw1104 with return code 30
• 15:18 bd808: trebuchet fetch of scap failed on mw1222 with return code 128
• 07:43 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Mar 25 07:42:46 UTC 2015 (duration 42m 45s)
• 03:11 logmsgbot: LocalisationUpdate completed (1.25wmf22) at 2015-03-25 03:10:38+00:00
• 03:06 logmsgbot: l10nupdate Synchronized php-1.25wmf22/cache/l10n: (no message) (duration: 06m 44s)
• 02:48 logmsgbot: LocalisationUpdate completed (1.25wmf21) at 2015-03-25 02:47:07+00:00
• 02:41 logmsgbot: l10nupdate Synchronized php-1.25wmf21/cache/l10n: (no message) (duration: 09m 12s)
• 01:32 logmsgbot: aude Synchronized php-1.25wmf22/extensions/Wikidata: Fix change dispatcher issues (duration: 00m 18s)

March 24

• 23:56 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/192894/ - should be a noop (duration: 00m 11s)
• 23:53 logmsgbot: krenair Synchronized php-1.25wmf21/extensions/RestBaseUpdateJobs/RestbaseUpdateJob.php: https://gerrit.wikimedia.org/r/#/c/199526/ (duration: 00m 14s)
• 23:46 logmsgbot: krenair Synchronized php-1.25wmf22/extensions/RestBaseUpdateJobs/RestbaseUpdateJob.php: https://gerrit.wikimedia.org/r/#/c/199527/1 (duration: 00m 11s)
• 23:04 logmsgbot: maxsem Synchronized php-1.25wmf22/extensions/WikiGrok: (no message) (duration: 00m 12s)
• 23:03 logmsgbot: maxsem Synchronized php-1.25wmf21/extensions/WikiGrok: (no message) (duration: 00m 13s)
• 22:34 logmsgbot: maxsem Synchronized php-1.25wmf22/extensions/WikiGrok/: bump (duration: 00m 11s)
• 22:27 logmsgbot: maxsem Synchronized php-1.25wmf21/extensions/WikiGrok/: bump (duration: 00m 11s)
• 22:20 MaxSem: Created wikigrok_claims and wikigrok_responses tables on wikidatawiki and testwikidatawiki. Before that, accidentally created on enwiki, so had to uncreate.
• 22:07 ejegg: Re-enabled Jenkins civi jobs
• 22:02 ejegg: updated civicrm from f8fb0f61531431348f3a8a3ee107056a864d537b to 4c459f3dbf3c3466cdc26a351ba589f4f1aef587
• 22:01 ejegg: disabled Jenkins civi jobs
• 20:35 Coren: rebooting labstore2001 to look at its bios
• 18:25 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group1 to php-1.25wmf22
• 18:18 twentyafterfour: Starting deployment train: group1 to 1.25wmf22
• 18:08 legoktm: manually attached User:Secret@enwiki to global
• 18:00 logmsgbot: demon Synchronized wmf-config/extension-list: (no message) (duration: 00m 12s)
• 17:56 legoktm: set email for User:ProGTX@global, attached enwiki
• 17:45 YuviPanda: restart gitblit on antimony
• 16:42 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Whitelist domain for GWT (duration: 00m 13s)
• 16:41 logmsgbot: aude Synchronized php-1.25wmf22/extensions/Wikidata: Update Wikidata - includes security fix and bug fixes (duration: 00m 19s)
• 16:22 _joe_: manually deleting puppet reports
• 16:08 logmsgbot: demon Synchronized php-1.25wmf22/extensions/ContentTranslation: (no message) (duration: 00m 12s)
• 16:07 logmsgbot: demon Synchronized php-1.25wmf21/extensions/ContentTranslation: (no message) (duration: 00m 11s)
• 16:07 logmsgbot: demon Synchronized php-1.25wmf22/extensions/UniversalLanguageSelector: (no message) (duration: 00m 11s)
• 16:07 logmsgbot: demon Synchronized php-1.25wmf21/extensions/UniversalLanguageSelector: (no message) (duration: 00m 11s)
• 16:04 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: dewikiversity content namespaces (duration: 00m 12s)
• 15:36 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: import sources for ptwiki (duration: 00m 11s)
• 15:33 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: once more, with feeling (duration: 00m 12s)
• 15:09 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: typofix in sitenotice (duration: 00m 11s)
• 14:16 _joe_: restarting pybal on lvs2003
• 13:55 Coren: reinstalling labstore2001 with Jessie
• 12:21 ori: disabling puppet on osmium for an hour to avoid perturbing a VE benchmarking suite
• 12:04 godog: bounce elasticsearch on logstash1001, shards unallocated/initializing
• 11:48 godog: remove per-partition iostat data from graphite1001, obsolete
• 10:50 logmsgbot: hoo Synchronized wmf-config/: Deploy Capiunto on beta, for consistency (duration: 01m 44s)
• 09:50 _joe_: running scap sync-common on all codfw mw* servers so that they don't kill scap on next deploy
• 08:35 hashar: restarting Jenkins for some plugins upgrades
• 06:45 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Mar 24 06:44:46 UTC 2015 (duration 44m 45s)
• 02:59 mutante: zirconium - tmp. disable puppet, tmp. enable contacts to make dump, make myself admin of civicrm
• 02:54 logmsgbot: LocalisationUpdate completed (1.25wmf22) at 2015-03-24 02:53:01+00:00
• 02:50 logmsgbot: l10nupdate Synchronized php-1.25wmf22/cache/l10n: (no message) (duration: 05m 04s)
• 02:31 logmsgbot: LocalisationUpdate completed (1.25wmf21) at 2015-03-24 02:30:42+00:00
• 02:28 logmsgbot: l10nupdate Synchronized php-1.25wmf21/cache/l10n: (no message) (duration: 04m 57s)
• 01:48 twentyafterfour: deployed scap/scap-sync-20150324-014557
• 00:38 urandom: restarting cassandra on restbase1006
• 00:01 logmsgbot: krenair Synchronized php-1.25wmf21/extensions/Flow/container.php: https://gerrit.wikimedia.org/r/#/c/199168/ (duration: 00m 05s)

March 23

• 23:57 logmsgbot: krenair Synchronized php-1.25wmf22/extensions/Flow/container.php: https://gerrit.wikimedia.org/r/#/c/199167/1 (duration: 00m 07s)
• 23:50 logmsgbot: krenair Synchronized php-1.25wmf22/extensions/CentralAuth/includes/specials/SpecialGlobalRenameQueue.php: https://gerrit.wikimedia.org/r/#/c/199157/1 (duration: 00m 07s)
• 23:44 logmsgbot: krenair Synchronized php-1.25wmf21/extensions/CentralAuth/includes/specials/SpecialGlobalRenameQueue.php: https://gerrit.wikimedia.org/r/#/c/199158/1 (duration: 00m 05s)
• 23:32 logmsgbot: krenair Synchronized php-1.25wmf21/extensions/RestBaseUpdateJobs: https://gerrit.wikimedia.org/r/#/c/199138/ (duration: 00m 07s)
• 23:24 logmsgbot: krenair Synchronized php-1.25wmf22/extensions/RestBaseUpdateJobs: https://gerrit.wikimedia.org/r/#/c/199137/ (duration: 00m 10s)
• 23:18 hasharDinner: Stopping Jenkins for an upgrade
• 23:06 logmsgbot: krenair Synchronized wmf-config/mobile.php: https://gerrit.wikimedia.org/r/#/c/198195/ (duration: 00m 06s)
• 20:48 cscott: updated OCG to version 11f096b6e45ef183826721f5c6b0f933a387b1bb
• 20:24 cscott: updated Parsoid to version a5d7483f
• 19:39 hoo: Manually created the following global accounts (name@homewiki), per Keegan: Lugal@enwiki, Aoe@enwiki, and Moonkey@eswiki
• 18:26 bblack: depooled mw1135 (eqiad api)
• 17:36 godog: ms-be101[678] weight to 1000
• 15:42 logmsgbot: demon Finished scap: VE + wikieditor + new msg for core (duration: 22m 34s)
• 15:38 chasemp: restarted hhvm on mw1193 -- done this for this particular host a few times now?
• 15:37 nuria: Eventlogging deployment & restart "28a0bf667a3869e95af0997c90af28dd329f6485"
• 15:20 logmsgbot: demon Started scap: VE + wikieditor + new msg for core
• 15:17 logmsgbot: demon Synchronized php-1.25wmf21/extensions/Flow: (no message) (duration: 00m 08s)
• 15:17 logmsgbot: demon Synchronized php-1.25wmf21/extensions/Echo: (no message) (duration: 00m 06s)
• 15:17 logmsgbot: demon Synchronized php-1.25wmf22/extensions/Echo: (no message) (duration: 00m 07s)
• 14:40 gwicke: restarted cassandra nodes to stop repair
• 11:49 godog: restart txstatsd on graphite1001 to drop old diamond metrics
• 10:50 godog: downgrade rsync to 3.0.9-1ubuntu1 on ms-be101[678] (precise's version) problems when senders are on 3.0.9 but receivers 3.1
• 10:48 hoo: Manually attached frwiki:Otets to the global account Otets
• 09:22 godog: deploy new swift ring including ms-be101[678]
• 06:03 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Mar 23 06:02:14 UTC 2015 (duration 2m 13s)
• 03:07 logmsgbot: krinkle Synchronized php-1.25wmf21/includes/TemplateParser.php: Ie90074e4885de7340e (duration: 00m 06s)
• 02:26 logmsgbot: LocalisationUpdate completed (1.25wmf22) at 2015-03-23 02:25:13+00:00
• 02:25 logmsgbot: l10nupdate Synchronized php-1.25wmf22/cache/l10n: (no message) (duration: 00m 03s)
• 02:24 logmsgbot: LocalisationUpdate completed (1.25wmf21) at 2015-03-23 02:23:26+00:00
• 02:24 logmsgbot: krinkle Synchronized php-1.25wmf22/includes/TemplateParser.php: Ie90074e4885de7340e (duration: 00m 06s)
• 02:21 logmsgbot: l10nupdate Synchronized php-1.25wmf21/cache/l10n: (no message) (duration: 04m 51s)

March 22

• 22:48 Krenair: Deployed patch for T93543
• 16:11 andrewbogott: restarted nova-api on labnet1001 because it was timing out
• 05:46 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Mar 22 05:45:32 UTC 2015 (duration 45m 31s)
• 02:22 logmsgbot: LocalisationUpdate completed (1.25wmf22) at 2015-03-22 02:21:46+00:00
• 02:21 logmsgbot: l10nupdate Synchronized php-1.25wmf22/cache/l10n: (no message) (duration: 00m 03s)
• 02:20 logmsgbot: LocalisationUpdate completed (1.25wmf21) at 2015-03-22 02:19:53+00:00
• 02:17 logmsgbot: l10nupdate Synchronized php-1.25wmf21/cache/l10n: (no message) (duration: 04m 54s)
• 01:22 gwicke: running `nodetool cleanup` on restbase1005

March 21

• 20:10 gwicke: performing slow rolling restart of restbase cassandra cluster to apply config changes from puppet
• 06:33 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Mar 21 06:32:22 UTC 2015 (duration 32m 21s)
• 02:36 logmsgbot: LocalisationUpdate completed (1.25wmf22) at 2015-03-21 02:35:37+00:00
• 02:35 logmsgbot: l10nupdate Synchronized php-1.25wmf22/cache/l10n: (no message) (duration: 00m 03s)
• 02:33 logmsgbot: LocalisationUpdate completed (1.25wmf21) at 2015-03-21 02:32:38+00:00
• 02:29 logmsgbot: l10nupdate Synchronized php-1.25wmf21/cache/l10n: (no message) (duration: 06m 43s)

March 20

• 23:56 mutante: suhail - new install, signing puppet cert, initial run
• 23:08 mutante: gdash.wikimedia.org now enforcing protocol redirect to https
• 22:51 logmsgbot: maxsem Synchronized php-1.25wmf21/extensions/MobileFrontend: touch (duration: 00m 09s)
• 22:37 logmsgbot: maxsem Synchronized php-1.25wmf21/includes/TemplateParser.php: https://gerrit.wikimedia.org/r/#/c/198409/ (duration: 00m 06s)
• 22:33 logmsgbot: demon Synchronized php-1.25wmf21/includes/TemplateParser.php: (no message) (duration: 00m 07s)
• 22:33 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 07s)
• 21:59 logmsgbot: demon Synchronized php-1.25wmf21/includes/TemplateParser.php: (no message) (duration: 00m 05s)
• 21:59 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 11s)
• 19:45 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 07s)
• 19:43 logmsgbot: krenair Synchronized wmf-config: retry, think that was a caching issue? (duration: 00m 07s)
• 19:28 logmsgbot: krenair Synchronized wmf-config: rv (duration: 00m 07s)
• 19:27 logmsgbot: krenair Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/198221/ for beta - should be a noop in prod (duration: 00m 08s)
• 18:48 bblack: reinstalling cp1058 (ignore cp1057 message above, it's a typo!)
• 18:47 bblack: reinstalling cp1057
• 17:54 mutante: killing tola, reinstall as suhail
• 17:45 mutante: subra,suhail - powercycling to BIOS
• 17:27 legoktm: re-inserted log users_to_rename rows
• 15:52 hoo: Deployed patch for T93365
• 13:46 akosiaris: uploaded apertium-hbs-mkd_0.1.0~r57554-1 on apt.wikimedia.org component: trusty-wikimedia
• 07:19 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Mar 20 07:18:31 UTC 2015 (duration 18m 30s)
• 04:16 legoktm: set email for Hmscott@global, attached enwiki account
• 02:46 logmsgbot: LocalisationUpdate completed (1.25wmf22) at 2015-03-20 02:45:00+00:00
• 02:41 logmsgbot: l10nupdate Synchronized php-1.25wmf22/cache/l10n: (no message) (duration: 06m 36s)
• 02:31 logmsgbot: LocalisationUpdate completed (1.25wmf21) at 2015-03-20 02:30:50+00:00
• 02:28 logmsgbot: l10nupdate Synchronized php-1.25wmf21/cache/l10n: (no message) (duration: 04m 56s)
• 01:46 subbu: deployed parsoid sha 99d1b214
• 01:09 urandom: restarting cassandra on restbase1001
• 00:47 urandom: restarting cassandra on restbase1002
• 00:12 urandom: restarting cassandra on restbase1003
• 00:11 logmsgbot: krenair Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/160494/ - labs changes only (duration: 00m 09s)
• 00:07 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/187869 - comment change only, noop (duration: 00m 07s)

March 19

• 23:58 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/194966/ - rm toolserver.org from whitelist (duration: 00m 07s)
• 23:53 logmsgbot: krenair Synchronized robots.txt: https://gerrit.wikimedia.org/r/#/c/195097/ - typo fix (duration: 00m 06s)
• 23:47 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/198126/ (duration: 00m 07s)
• 23:39 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/196988/ (duration: 00m 05s)
• 23:36 Krenair: Run mwscript sql.php --wiki=ukwiki php-1.25wmf21/extensions/WikiLove/patches/WikiLoveLog.sql for https://gerrit.wikimedia.org/r/#/c/196988/
• 23:27 logmsgbot: krenair Synchronized php-1.25wmf21/extensions/Flow/maintenance/FlowUpdateRevisionContentLength.php: https://gerrit.wikimedia.org/r/#/c/198125/ (duration: 00m 05s)
• 23:24 nuria: eventlogging re-start due to vanadium disk filling up. Moved logs to "/srv/"
• 23:22 logmsgbot: krenair Synchronized php-1.25wmf22/extensions/Flow: https://gerrit.wikimedia.org/r/#/c/198127/ (duration: 00m 07s)
• 23:08 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/197466/ (duration: 00m 07s)
• 22:22 urandom: restarting cassandra on restbase1004
• 21:56 logmsgbot: krenair Synchronized php-1.25wmf21/extensions/WikiEditor/modules/jquery.wikiEditor.js: https://gerrit.wikimedia.org/r/198106 (duration: 00m 06s)
• 21:48 legoktm: set email for User:Phrazz@global and attached commonswiki account
• 20:14 legoktm: manually merged accounts for User:Babel AutoCreate
• 19:31 bblack: repooled cp1047 in pybal
• 19:07 urandom: restarting cassandra on restbase1005
• 18:56 logmsgbot: legoktm Synchronized php-1.25wmf21/extensions/Renameuser/: Move logging inside of RenameuserSQL (duration: 00m 08s)
• 18:56 logmsgbot: legoktm Synchronized php-1.25wmf22/extensions/Renameuser/: Move logging inside of RenameuserSQL (duration: 00m 07s)
• 18:46 logmsgbot: legoktm Finished scap: Update CentralAuth to master (duration: 30m 36s)
• 18:32 urandom: restarting cassandra on restbase1006
• 18:15 logmsgbot: legoktm Started scap: Update CentralAuth to master
• 17:50 legoktm: manually attached accounts for User:MediaWiki default, required clearing password+email on dewiki and cswiki
• 17:46 awight: update payments from ebaa05b1987f366897dd32bccc5653485fc62113 to 7ffe008fb8964acb1382820d129d784d5b6dd9de
• 17:33 Nikerabbit: ran fix-stats.php on all wikis with ContentTranslation
• 17:31 legoktm: manually attached all of User:MediaWiki message delivery's accounts
• 17:30 legoktm: manually attached all of User:FuzzyBot's accounts
• 17:27 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings-labs.php: Actually disable RESTbase in labs (duration: 00m 06s)
• 17:01 logmsgbot: nikerabbit Synchronized wmf-config/CommonSettings.php: Update wgLocalisationUpdateDirectory to match l10nupdate-1 (duration: 00m 05s)
• 16:52 logmsgbot: kartik Finished scap: Update ContentTranslation (duration: 38m 43s)
• 16:43 subbu: deployed parsoid sha f5f5f0ede
• 16:13 logmsgbot: kartik Started scap: Update ContentTranslation
• 15:56 logmsgbot: krenair Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/197925/ (duration: 00m 07s)
• 15:53 logmsgbot: krenair Synchronized php-1.25wmf22/extensions/WikiEditor/WikiEditor.hooks.php: https://gerrit.wikimedia.org/r/#/c/197904/ (duration: 00m 05s)
• 15:50 logmsgbot: krenair Synchronized php-1.25wmf21/extensions/WikiEditor/WikiEditor.hooks.php: https://gerrit.wikimedia.org/r/#/c/197905/ (duration: 00m 07s)
• 15:27 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/194856/ (duration: 00m 06s)
• 15:18 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/197883/ (duration: 00m 05s)
• 15:10 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/197907/ (duration: 00m 06s)
• 15:07 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/197354/ (duration: 00m 08s)
• 15:07 akosiaris: uploaded python-virtualenv_1.11.4-1 on apt.wikimedia.org precise-wikimedia
• 15:02 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/197381/ (duration: 00m 06s)
• 14:14 logmsgbot: demon Synchronized wmf-config/: memc for codfw (duration: 00m 08s)
• 14:13 akosiaris: uploaded apertium-hbs-slv_0.5.0~r43858-1 on apt.wikimedia.org
• 14:05 akosiaris: uploaded apertium-hbs-eng_0.1.0~r57554-1 on apt.wikimedia.org
• 13:50 akosiaris: uploaded apertium-mk-bg_0.2.0~r49489-1 on apt.wikimedia.org
• 13:39 akosiaris: uploaded apertium-hbs_0.5.0~r57197-2 on apt.wikimedia.org
• 13:02 logmsgbot: demon Synchronized wmf-config/jobqueue-codfw.php: codfw support (duration: 00m 05s)
• 12:53 logmsgbot: demon Synchronized wmf-config/squid.php: codfw (duration: 00m 07s)
• 12:44 logmsgbot: demon Synchronized wmf-config: poolcounter for codfw (duration: 00m 10s)
• 11:14 godog: test-run tftpd-hpa on carbon vs atftpd
• 10:30 YuviPanda: sudo mv eventlogging_processor-client-side-events.log.1 /srv on vanadium, make space in /
• 10:25 YuviPanda: 50G of logs in /var/log/upstart/eventlogging_processor-client-side-events.log.1
• 07:14 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Mar 19 07:13:03 UTC 2015 (duration 13m 2s)
• 04:19 legoktm: manually globalized User:WeeJay
• 02:37 logmsgbot: LocalisationUpdate completed (1.25wmf22) at 2015-03-19 02:36:03+00:00
• 02:36 logmsgbot: l10nupdate Synchronized php-1.25wmf22/cache/l10n: (no message) (duration: 00m 03s)
• 02:24 logmsgbot: LocalisationUpdate completed (1.25wmf21) at 2015-03-19 02:23:13+00:00
• 02:23 logmsgbot: l10nupdate Synchronized php-1.25wmf21/cache/l10n: (no message) (duration: 00m 04s)
• 00:29 hoo: Set email for frwiki account "Sarcelles" to the one of the global account with the same name.
• 00:23 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings.php: Enable RESTbase on frwiki (duration: 00m 06s)
• 00:10 logmsgbot: catrope Finished scap: SWAT (duration: 30m 20s)

March 18

• 23:40 logmsgbot: catrope Started scap: SWAT
• 23:36 logmsgbot: catrope Synchronized wmf-config/mobile.php: SWAT (duration: 00m 05s)
• 23:36 logmsgbot: catrope Synchronized wmf-config/CommonSettings.php: SWAT (duration: 00m 07s)
• 23:36 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings.php: SWAT (duration: 00m 07s)
• 22:35 logmsgbot: gwicke Synchronized wmf-config/InitialiseSettings.php: Enable RESTBase on ptwiki and ruwiki (duration: 00m 05s)
• 21:40 logmsgbot: legoktm Synchronized php-1.25wmf21/extensions/CentralAuth/includes/UsersToRename/UsersToRenameDatabaseUpdates.php: https://gerrit.wikimedia.org/r/#/c/197755/ (duration: 00m 06s)
• 21:39 logmsgbot: legoktm Synchronized php-1.25wmf22/extensions/CentralAuth/includes/UsersToRename/UsersToRenameDatabaseUpdates.php: https://gerrit.wikimedia.org/r/#/c/197754/ (duration: 00m 05s)
• 21:25 urandom: increasing compaction throughput on restbase100[1-6]
• 20:39 gwicke: deployed restbase 73cc02abdb
• 20:32 logmsgbot: twentyafterfour Finished scap: Security patches to php-1.25wmf22 (duration: 21m 56s)
• 20:26 ejegg: updated payments from 2c5e99cb6de54a6a4e6e2334d533e8ef36c2090c to ebaa05b1987f366897dd32bccc5653485fc62113
• 20:15 subbu: deployed parsoid sha b48f6e25
• 20:11 logmsgbot: twentyafterfour Started scap: Security patches to php-1.25wmf22
• 19:52 logmsgbot: twentyafterfour Purged l10n cache for 1.25wmf20
• 19:50 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf22
• 19:47 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.25wmf21
• 19:43 ejegg: rolled back payments to 2c5e99cb6de54a6a4e6e2334d533e8ef36c2090c
• 19:39 ejegg: updated payments from 2c5e99cb6de54a6a4e6e2334d533e8ef36c2090c to 189a0ef97d8311f29d5e6e724540f34e1e6be7aa
• 19:33 logmsgbot: twentyafterfour Finished scap: testwiki to php-1.25wmf22 and rebuild l10n cache (duration: 52m 23s)
• 19:19 hoo: Updated email of global account "Ar-ras". The email I set for it on February 17 was outdated.
• 19:18 ori: Stopping uWSGI on labmon1001 to troubleshoot T93083
• 18:40 logmsgbot: twentyafterfour Started scap: testwiki to php-1.25wmf22 and rebuild l10n cache
• 18:23 urandom: starting nodetool clean on restbase1005
• 17:58 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: cawikimedia language code (duration: 00m 08s)
• 17:52 twentyafterfour: branching 1.25wmf22
• 17:42 tgr: messing with live code on testwiki to test a fix for https://phabricator.wikimedia.org/T93009
• 17:29 akosiaris: uploaded php5_5.3.10-1ubuntu3.17+wmf1ubuntu1 on apt.wikimedia.org for precise-wikimedia
• 17:22 akosiaris: uploaded apertium-mkd_0.1.0-1 on apt.wikimedia.org
• 17:02 mutante: ms1001 - short maintenance downtime for bonding networks interfaces
• 16:53 godog: remount cgroup on silver with 1M of space
• 16:18 awight: update fundraising/tools from 9fd0a885e84074f215082aad689649a0684660f9 to 9a9e7881d25f101cc612cfae6375c0a1c9b0f55d
• 15:57 Coren: Started uwsgi on labmon1001 by hand (which works) so that graphite isn't broken during debugging.
• 15:44 akosiaris: uploaded apertium-af-nl 0.2.0 on apt.wikimedia.org
• 15:44 logmsgbot: legoktm Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/197503/ (duration: 00m 05s)
• 15:34 logmsgbot: demon Synchronized wmf-config/db-codfw.php: initial codfw support (duration: 00m 06s)
• 15:34 logmsgbot: demon Synchronized multiversion/MWRealm.php: initial codfw support (duration: 00m 06s)
• 15:25 logmsgbot: manybubbles Synchronized php-1.25wmf21/extensions/CirrusSearch/includes/Util.php: SWAT fix some batch scripts in cirrus (duration: 00m 07s)
• 15:24 Coren: Fiddling with uwsgi on labmon1001, ignore errors
• 15:20 logmsgbot: legoktm Synchronized php-1.25wmf21/includes/registration/ExtensionRegistry.php: https://gerrit.wikimedia.org/r/#/c/197630/ (duration: 00m 05s)
• 15:15 logmsgbot: legoktm Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/197427 (duration: 00m 53s)
• 14:20 paravoid: powercycling rhenium
• 05:34 ori: Disabled puppet on osmium for testing a chromium thingy
• 05:05 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Mar 18 05:04:23 UTC 2015 (duration 4m 22s)
• 02:28 logmsgbot: LocalisationUpdate completed (1.25wmf21) at 2015-03-18 02:27:48+00:00
• 02:24 logmsgbot: l10nupdate Synchronized php-1.25wmf21/cache/l10n: (no message) (duration: 06m 51s)
• 02:04 logmsgbot: LocalisationUpdate completed (1.25wmf20) at 2015-03-18 02:03:05+00:00
• 02:04 bd808: Updated scap to I58e817b (Improved test for content preceeding <?php opening tag)
• 02:03 logmsgbot: l10nupdate Synchronized php-1.25wmf20/cache/l10n: (no message) (duration: 00m 04s)
• 01:26 logmsgbot: legoktm Synchronized php-1.25wmf21/tests/parser/parserTests.txt: testing syntax error bug (duration: 00m 07s)
• 01:24 logmsgbot: legoktm Synchronized php-1.25wmf20/tests/parser/parserTests.txt: testing syntax error bug (duration: 00m 07s)
• 01:19 bd808: Updated scap to Ie1d1642 (Have utils.check_php_opening_tag check the file extension suffix)
• 01:16 mutante: mw2008 rebooting to fix BIOS HT setting
• 01:16 bd808: Trebuchet error from mw1222 for scap deploy (status code 128), no response from mw2003
• 01:06 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Idf3491140: Drop support for 75 languages in SyntaxHighlighter_GeSHi (duration: 00m 05s)
• 01:05 logmsgbot: kaldari Synchronized php-1.25wmf21/extensions/VisualEditor: syncing update to VE to fix mobile (duration: 00m 06s)
• 01:02 logmsgbot: legoktm Synchronized php-1.25wmf20/tests/parser/: testing syntax error bug (duration: 00m 05s)
• 00:56 logmsgbot: legoktm scap aborted: (no message) (duration: 02m 04s)
• 00:55 legoktm: testing scap for syntax errors bug
• 00:54 logmsgbot: legoktm Started scap: (no message)
• 00:14 logmsgbot: krenair Synchronized php-1.25wmf21/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/197415/ (duration: 00m 08s)
• 00:00 logmsgbot: krenair Synchronized php-1.25wmf21/extensions/Flow: https://gerrit.wikimedia.org/r/#/c/197434/ (duration: 00m 08s)

March 17

• 23:45 logmsgbot: krenair Synchronized wmf-config/mobile.php: 194354, 194373, 194378, 194503 (duration: 00m 05s)
• 23:31 logmsgbot: krenair Synchronized php-1.25wmf21/extensions/GeoData/GeoDataHooks.php: https://gerrit.wikimedia.org/r/#/c/197410/ (duration: 00m 06s)
• 23:22 logmsgbot: krenair Synchronized php-1.25wmf21/resources: https://gerrit.wikimedia.org/r/#/c/197428/ (duration: 00m 05s)
• 23:22 logmsgbot: krenair Synchronized php-1.25wmf21/includes/Linker.php: https://gerrit.wikimedia.org/r/#/c/197428/ (duration: 00m 05s)
• 23:16 logmsgbot: krenair Synchronized php-1.25wmf20/resources: https://gerrit.wikimedia.org/r/#/c/197430/ (duration: 00m 05s)
• 23:16 logmsgbot: krenair Synchronized php-1.25wmf20/includes/Linker.php: https://gerrit.wikimedia.org/r/#/c/197430/ (duration: 00m 05s)
• 22:30 csteipp: deployed updated patch for T73394
• 22:06 csteipp: deployed patches for T85848 to wmf20 and 21
• 22:00 logmsgbot: csteipp Synchronized php-1.25wmf21/includes/: (no message) (duration: 00m 11s)
• 21:53 ori: running varnishncsa in a shell on cp1056 to analyze usage of ext.geshi.language.* modules.
• 21:28 robh: apaches updated per https://phabricator.wikimedia.org/T92547 and appear stable
• 21:09 robh: updating apche redirects, disabling puppet on mw hosts
• 21:09 bd808: Updated scap to include I6301816 (Check for content before <?php) and I61dcf7a (Run rebuildLocalisationCache.php as www-data)
• 21:07 bd808: trebuchet checkout errors from mw1104, mw1113, mw1222. No response from mw2003
• 21:05 bd808: mw1222.eqiad.wmnet and mw2003.codfw.wmnet not responding to trebuchet fetch for scap
• 20:04 logmsgbot: yuvipanda Synchronized private: (no message) (duration: 00m 06s)
• 20:00 YuviPanda: stashing TimStarling’s changes to scap, re-enabling puppet on tin
• 19:22 bblack: temporarily disabling puppet on all cp*
• 18:57 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group1 to 1.25wmf21
• 18:39 twentyafterfour: Deploying 1.25wmf21 to group1 wikis.
• 18:04 _joe_: updating libavcodec53 and libavformat53 on the imagescalers and videoscalers
• 17:21 godog: repool restbase1006
• 17:14 legoktm: started sendForceRenameNotification.php (CentralAuth/SULF) for all wikis
• 17:07 legoktm: migrateAccount.php --auto finished
• 15:50 logmsgbot: demon Synchronized php-1.25wmf21/includes/jobqueue/: (no message) (duration: 00m 09s)
• 15:43 logmsgbot: demon Synchronized php-1.25wmf20/includes/jobqueue/: (no message) (duration: 00m 07s)
• 15:42 godog: restart carbon/relay on graphite1001
• 15:41 logmsgbot: demon Synchronized php-1.25wmf21/extensions/VisualEditor/: (no message) (duration: 00m 06s)
• 15:34 logmsgbot: demon Synchronized php-1.25wmf21/extensions/Citoid: (no message) (duration: 00m 07s)
• 15:20 logmsgbot: demon Synchronized php-1.25wmf20/includes/specials/SpecialUploadStash.php: (no message) (duration: 00m 07s)
• 15:20 logmsgbot: demon Synchronized php-1.25wmf21/includes/specials/SpecialUploadStash.php: (no message) (duration: 00m 07s)
• 15:18 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: restbase off for wikipedias (duration: 00m 06s)
• 15:17 gwicke: restarted cassandra on restbase1006 after clearing the data & removing it from its own seeds
• 15:09 logmsgbot: demon Synchronized wmf-config/abusefilter.php: (no message) (duration: 00m 07s)
• 15:04 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: draft namespace for kowiki (duration: 00m 05s)
• 14:59 bblack: rebooting cp1072-4, cp3030-49 (none in production)
• 12:57 _joe_: updating sudo across all production
• 12:05 _joe_: upgraded libicu48 and mediawiki-math-texvc across the cluster
• 11:16 YuviPanda: ran chown -R gitpuppet:gitpuppet /var/lib/git/operations/puppet on palladium, fix permission issues
• 11:14 YuviPanda: ran chown -R gitpuppet:gitpuppet /var/lib/git/operations/puppet on strontium, fix permission issues
• 11:10 akosiaris: chown gitpuppet:gitpuppet /var/lib/git/operations/puppet/.git/logs/refs/remotes/origin/production on strontium, palladium. Somehow it was owned by root
• 10:59 godog: depool restbase1006, provisioning
• 07:40 _joe_: powercycled mw2027, went down with an unresponsive console
• 07:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Mar 17 07:06:36 UTC 2015 (duration 6m 35s)
• 04:46 ejegg: updated tools from 84442d51a841af4265ff103827cda83d5dd9dc54 to 9fd0a885e84074f215082aad689649a0684660f9
• 02:36 logmsgbot: LocalisationUpdate completed (1.25wmf21) at 2015-03-17 02:35:00+00:00
• 02:34 logmsgbot: l10nupdate Synchronized php-1.25wmf21/cache/l10n: (no message) (duration: 00m 04s)
• 02:22 logmsgbot: LocalisationUpdate completed (1.25wmf20) at 2015-03-17 02:20:58+00:00
• 02:20 logmsgbot: l10nupdate Synchronized php-1.25wmf20/cache/l10n: (no message) (duration: 00m 03s)
• 02:03 ori: applied I98d383a629 locally on mw1017
• 01:45 logmsgbot: ori scap failed: CalledProcessError Command 'cp -r "/tmp/scap_l10n_0.713383032704/*" "/srv/mediawiki-staging/php-1.25wmf20/cache/l10n"' returned non-zero exit status 1 (duration: 00m 10s)
• 01:45 logmsgbot: ori Started scap: (no message)
• 01:44 logmsgbot: ori scap failed: TypeError temp_dir() takes exactly 2 arguments (1 given) (duration: 00m 10s)
• 01:44 logmsgbot: ori Started scap: (no message)
• 01:31 logmsgbot: tstarling Finished scap: (no message) (duration: 30m 19s)
• 01:21 ori: restarted HHVM on mw1139; bt in https://phabricator.wikimedia.org/P406
• 01:04 mutante: disabled contacts.wikimedia.org - if you are an (unexpected) user please contact me or T90679
• 01:01 logmsgbot: tstarling Started scap: (no message)
• 00:58 ori: Increased memory limit on HHVM app servers from 300M to 500M in an attempt to reduce the rate at which T89918 occurs
• 00:57 logmsgbot: tstarling scap failed: CalledProcessError Command 'cp -r "/tmp/scap_l10n_2149279197/*" "/srv/mediawiki-staging/php-1.25wmf20/cache/l10n"' returned non-zero exit status 1 (duration: 04m 27s)
• 00:52 logmsgbot: tstarling Started scap: (no message)
• 00:48 logmsgbot: tstarling scap failed: ValueError unsupported format character '/' (0x2f) at index 18 (duration: 04m 29s)
• 00:44 tgr: running extensions/GlobalUsage/refreshGlobalimagelinks.php --wiki=plwiki --pages=nonexisting
• 00:43 logmsgbot: tstarling Started scap: (no message)
• 00:18 logmsgbot: tstarling scap failed: CalledProcessError Command 'mkdir "/tmp/scap_l10n_110284512" && /usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="cawikibooks" --outdir="/tmp/scap_l10n_110284512" --threads=4 %(force)s --quiet' returned non-zero exit status 2 (duration: 00m 08s)
• 00:17 logmsgbot: tstarling Started scap: (no message)
• 00:11 logmsgbot: tstarling scap failed: CalledProcessError Command 'mkdir "/tmp/scap_l10n_3829571451" && /usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="cawikibooks" --outdir="/tmp/scap_l10n_3829571451" --threads=4 %(force)s --quiet' returned non-zero exit status 2 (duration: 00m 07s)
• 00:11 logmsgbot: tstarling Started scap: (no message)
• 00:10 logmsgbot: tstarling scap failed: TypeError cannot concatenate 'str' and 'int' objects (duration: 00m 07s)
• 00:10 logmsgbot: tstarling Started scap: (no message)
• 00:08 logmsgbot: tstarling scap failed: NameError global name 'random' is not defined (duration: 00m 08s)
• 00:08 logmsgbot: tstarling Started scap: (no message)
• 00:01 Tim: on tin: disabling puppet for scap test. Patching scap locally

March 16

• 23:58 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/195014/ (duration: 00m 07s)
• 23:54 logmsgbot: krenair Synchronized php-1.25wmf21/includes/logging: https://gerrit.wikimedia.org/r/#/c/196845/ (duration: 00m 07s)
• 23:52 logmsgbot: krenair Synchronized php-1.25wmf20/includes/logging: https://gerrit.wikimedia.org/r/#/c/196846/ (duration: 00m 05s)
• 23:41 logmsgbot: krenair Synchronized php-1.25wmf21/extensions/Wikidata: https://gerrit.wikimedia.org/r/#/c/197050/ (duration: 00m 13s)
• 23:34 logmsgbot: krenair Synchronized php-1.25wmf20/extensions/Wikidata: https://gerrit.wikimedia.org/r/#/c/197049/ (duration: 00m 13s)
• 23:29 logmsgbot: krenair Synchronized php-1.25wmf21/includes/content/JsonContent.php: https://gerrit.wikimedia.org/r/#/c/197215/ (duration: 00m 11s)
• 23:19 logmsgbot: krenair Synchronized php-1.25wmf20/includes/content/JsonContent.php: https://gerrit.wikimedia.org/r/#/c/197216/ (duration: 00m 06s)
• 23:16 logmsgbot: krenair Synchronized php-1.25wmf21/extensions/Citoid/Citoid.php: https://gerrit.wikimedia.org/r/#/c/197236/ (duration: 00m 07s)
• 23:06 logmsgbot: krenair Synchronized php-1.25wmf21/extensions/GlobalUsage/refreshGlobalimagelinks.php: https://gerrit.wikimedia.org/r/#/c/196994/ (duration: 00m 05s)
• 23:04 logmsgbot: krenair Synchronized php-1.25wmf20/extensions/GlobalUsage/refreshGlobalimagelinks.php: https://gerrit.wikimedia.org/r/#/c/196993/ (duration: 00m 05s)
• 21:19 akosiaris: installing a non-puppetized version of the puppet cronjob on nescio, sodium. The new well thought out puppet-run can not run on lucid hosts since https://gerrit.wikimedia.org/r/#/c/196162/ . Given they go away soon, it is better to not do weird puppet tricks to accomodate for just 2 old, soon to be deprecated, boxes.
• 20:33 logmsgbot: krenair Finished scap: https://gerrit.wikimedia.org/r/#/c/197127/ - and also try to fix citoid i18n on test wikis (duration: 01m 45s)
• 20:32 logmsgbot: krenair Started scap: https://gerrit.wikimedia.org/r/#/c/197127/ - and also try to fix citoid i18n on test wikis
• 20:12 subbu: deployed parsoid sha ccf4c140
• 20:05 logmsgbot: gwicke Synchronized wmf-config/InitialiseSettings.php: Use RESTBase with VE on all wikipedias (duration: 00m 08s)
• 19:42 paravoid: working on mailman issues
• 19:41 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 05s)
• 19:39 logmsgbot: krenair Finished scap: Citoid extension deployment (duration: 02m 54s)
• 19:36 logmsgbot: krenair Started scap: Citoid extension deployment
• 19:22 mobrovac: restart citoid on sca1002
• 19:17 logmsgbot: krenair scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="cawikibooks" --list-file="/srv/mediawiki-staging/wmf-config/extension-list" --output="/tmp/tmp.zUFETE2jD1" ' returned non-zero exit status 1 (duration: 00m 27s)
• 19:16 logmsgbot: krenair Started scap: Citoid extension deployment
• 18:44 logmsgbot: gwicke Synchronized wmf-config/InitialiseSettings.php: Use RESTBase with VE on frwiki (duration: 00m 08s)
• 18:27 logmsgbot: gwicke Synchronized wmf-config/InitialiseSettings.php: Use RESTBase with VE on itwiki and plwiki (duration: 00m 07s)
• 18:04 logmsgbot: gwicke Synchronized wmf-config/InitialiseSettings.php: Use RESTBase with VE on ptwiki and ruwiki (duration: 00m 05s)
• 17:48 legoktm: running CentralAuth's migrateAccount.php --auto on all unattached accounts
• 17:34 hoo: Updated entity suggester data on wikidata (with data from today's dump)
• 16:52 YuviPanda: running sync-common on silver to have it catch up
• 16:45 logmsgbot: yuvipanda Synchronized README: testing silver firewall hole (duration: 00m 05s)
• 16:29 bblack: cp3030-3049 downtimed in icinga through 2015-04-01 for now, not in production traffic flow
• 16:28 nuria: Eventlogging deploy and restart, reduced batch size. Changeset: 3c987f67a0355c613aa042704a1c3422d0fcd55b
• 16:06 logmsgbot: anomie Synchronized php-1.25wmf20/extensions/BounceHandler/: SWAT: BounceHandler: Removed repititive un-subscribe action on a global user gerrit:196878 (duration: 01m 06s)
• 16:04 logmsgbot: anomie Synchronized php-1.25wmf20/extensions/RestBaseUpdateJobs/: SWAT: RestBaseUpdateJobs: Set HTTP headers as an associative array gerrit:197041 (duration: 01m 03s)
• 16:00 logmsgbot: anomie Synchronized php-1.25wmf21/extensions/RestBaseUpdateJobs/: SWAT: RestBaseUpdateJobs: Set HTTP headers as an associative array gerrit:197042 (duration: 01m 03s)
• 15:53 logmsgbot: anomie Synchronized php-1.25wmf21/extensions/BounceHandler/: SWAT: BounceHandler: Removed repititive un-subscribe action on a global user gerrit:196877 (duration: 01m 04s)
• 15:37 nuria: Eventlogging deploy & restart: 4399dfc3240c0d27fdf6c517c7bf3239fc2da924
• 15:35 logmsgbot: anomie Synchronized php-1.25wmf20/extensions/Flow/: SWAT: Flow: base href fix gerrit:196995 gerrit:196997 (duration: 01m 05s)
• 15:33 logmsgbot: anomie Synchronized php-1.25wmf21/extensions/Flow/: SWAT: Flow: base href fix and dependency gerrit:196996 (duration: 01m 10s)
• 15:29 logmsgbot: anomie Synchronized php-1.25wmf20/includes/Html.php: SWAT: Fix for mediawiki.ui style for wpTextbox1 and wpSummary in preview if text includes inbutbox element gerrit:196897 (duration: 01m 03s)
• 15:25 logmsgbot: anomie Synchronized php-1.25wmf21/includes/Html.php: SWAT: Fix for mediawiki.ui style for wpTextbox1 and wpSummary in preview if text includes inbutbox element gerrit:196896 (duration: 01m 03s)
• 15:23 logmsgbot: anomie Synchronized php-1.25wmf21/includes/Html.php: SWAT: Fix for mediawiki.ui style for wpTextbox1 and wpSummary in preview if text includes inbutbox element gerrit:196896 (duration: 01m 03s)
• 15:09 logmsgbot: anomie Synchronized php-1.25wmf21/extensions/WikiEditor/: SWAT: WikiEditor: fix Edit schema validation issues gerrit:196715 gerrit:196716 gerrit:196727 (duration: 01m 04s)
• 15:07 YuviPanda: citoid down on sca1001, not coming back after restart. mobrovac investigating
• 15:05 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Beta Features: Remove VisualEditor language tool (deployed everywhere) gerrit:193762 (duration: 01m 04s)
• 14:45 godog: reboot ms-be2009, xfs hosed
• 10:02 hashar: restarting jenkins
• 02:18 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Mar 16 02:17:38 UTC 2015 (duration 17m 37s)
• 02:06 logmsgbot: LocalisationUpdate completed (1.25wmf21) at 2015-03-16 02:05:32+00:00
• 02:05 logmsgbot: l10nupdate Synchronized php-1.25wmf21/cache/l10n: (no message) (duration: 00m 04s)
• 02:05 logmsgbot: LocalisationUpdate completed (1.25wmf20) at 2015-03-16 02:03:57+00:00
• 02:03 logmsgbot: l10nupdate Synchronized php-1.25wmf20/cache/l10n: (no message) (duration: 00m 04s)

March 15

• 21:36 bblack: restarted keystone service on virt1001 to fix wikitech login, still no idea why that was necessary or what was broken
• 02:23 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Mar 15 02:22:41 UTC 2015 (duration 22m 40s)
• 02:11 logmsgbot: LocalisationUpdate completed (1.25wmf21) at 2015-03-15 02:10:51+00:00
• 02:10 logmsgbot: l10nupdate Synchronized php-1.25wmf21/cache/l10n: (no message) (duration: 00m 03s)
• 02:10 logmsgbot: LocalisationUpdate completed (1.25wmf20) at 2015-03-15 02:09:18+00:00
• 02:09 logmsgbot: l10nupdate Synchronized php-1.25wmf20/cache/l10n: (no message) (duration: 00m 04s)

March 14

• 07:28 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Mar 14 07:27:49 UTC 2015 (duration 27m 48s)
• 02:54 logmsgbot: LocalisationUpdate completed (1.25wmf21) at 2015-03-14 02:53:19+00:00
• 02:49 logmsgbot: l10nupdate Synchronized php-1.25wmf21/cache/l10n: (no message) (duration: 06m 28s)
• 02:29 logmsgbot: LocalisationUpdate completed (1.25wmf20) at 2015-03-14 02:28:10+00:00
• 02:24 logmsgbot: l10nupdate Synchronized php-1.25wmf20/cache/l10n: (no message) (duration: 06m 37s)
• 00:51 logmsgbot: legoktm Synchronized php-1.25wmf20/extensions/MassMessage/includes/job/MassMessageServerSideJob.php: https://gerrit.wikimedia.org/r/#/c/196729/ (duration: 00m 09s)
• 00:50 logmsgbot: legoktm Synchronized php-1.25wmf21/extensions/MassMessage/includes/job/MassMessageServerSideJob.php: https://gerrit.wikimedia.org/r/#/c/196729/ (duration: 00m 06s)

March 13

• 23:47 logmsgbot: legoktm Synchronized php-1.25wmf20/extensions/CentralAuth/: https://gerrit.wikimedia.org/r/#/c/196717/ (duration: 00m 08s)
• 23:44 logmsgbot: legoktm Synchronized php-1.25wmf21/extensions/CentralAuth/: https://gerrit.wikimedia.org/r/#/c/196718/ (duration: 00m 09s)
• 22:47 logmsgbot: aaron Synchronized wmf-config/InitialiseSettings.php: Added jobqueue federated log (duration: 00m 11s)
• 21:12 mutante: rbf2001 - re-signed puppet, re-enable icinga
• 21:05 logmsgbot: legoktm Synchronized php-1.25wmf20/extensions/MassMessage/: https://gerrit.wikimedia.org/r/196648 (duration: 00m 08s)
• 21:04 logmsgbot: legoktm Synchronized php-1.25wmf20/extensions/CentralAuth/: https://gerrit.wikimedia.org/r/196654 (duration: 00m 08s)
• 21:03 logmsgbot: legoktm Synchronized php-1.25wmf21/extensions/CentralAuth/: https://gerrit.wikimedia.org/r/#/c/196649/ (duration: 00m 08s)
• 21:01 mutante: rbf2001 - reinstalled, wmf-reimage
• 21:00 logmsgbot: legoktm Synchronized php-1.25wmf21/extensions/MassMessage/: https://gerrit.wikimedia.org/r/196649 (duration: 00m 09s)
• 19:59 logmsgbot: aaron Synchronized wmf-config/mc.php: Disabled bloom filter (duration: 00m 08s)
• 18:49 mutante: rbf2001 - powercycling, PXE boot
• 18:45 bblack: reinstalling cp3008
• 18:13 mutante: cp1049 - repooled in pybal - all eqiad upload caches now jessie
• 17:41 mutante: cp1049 (upload) - depooled in pybal
• 16:25 bblack: rebooting cp4019
• 15:26 bblack: reinstalling cp1044
• 11:43 _joe_: installing the new libicu48 package on the canary appservers
• 07:40 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Mar 13 07:39:11 UTC 2015 (duration 39m 10s)
• 02:58 logmsgbot: LocalisationUpdate completed (1.25wmf21) at 2015-03-13 02:57:48+00:00
• 02:54 logmsgbot: l10nupdate Synchronized php-1.25wmf21/cache/l10n: (no message) (duration: 07m 02s)
• 02:33 logmsgbot: LocalisationUpdate completed (1.25wmf20) at 2015-03-13 02:32:12+00:00
• 02:28 logmsgbot: l10nupdate Synchronized php-1.25wmf20/cache/l10n: (no message) (duration: 07m 08s)
• 00:44 logmsgbot: gwicke Synchronized wmf-config/InitialiseSettings.php: Enable RESTBase updates (duration: 00m 09s)
• 00:42 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/196496 (duration: 00m 09s)
• 00:38 logmsgbot: krenair Synchronized php-1.25wmf21/extensions/MobileFrontend/javascripts/modules/mediaViewer/ImageOverlay.js: https://gerrit.wikimedia.org/r/#/c/196497/ (duration: 00m 09s)
• 00:31 logmsgbot: krenair Synchronized php-1.25wmf21/includes/api: https://gerrit.wikimedia.org/r/#/c/196313/ (duration: 00m 08s)
• 00:21 logmsgbot: krenair Synchronized php-1.25wmf20/includes/api: https://gerrit.wikimedia.org/r/#/c/196317/ (duration: 00m 12s)
• 00:20 mutante: starting redis on rbf1002
• 00:17 logmsgbot: krenair Synchronized php-1.25wmf21/resources/src/mediawiki.ui/components/inputs.less: https://gerrit.wikimedia.org/r/#/c/196308/ (duration: 00m 07s)
• 00:07 logmsgbot: krenair Synchronized php-1.25wmf21/extensions/TemplateData: https://gerrit.wikimedia.org/r/#/c/196439/ (duration: 00m 12s)

March 12

• 23:55 logmsgbot: krenair Synchronized php-1.25wmf21/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/196436/ (duration: 00m 09s)
• 23:47 mutante: cp1050 - repooled in pybal
• 23:44 logmsgbot: krenair Synchronized php-1.25wmf20/extensions/VisualEditor/lib/ve/src/ce/ve.ce.Surface.js: https://gerrit.wikimedia.org/r/#/c/196435/ (duration: 00m 06s)
• 23:32 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/196428/ (duration: 00m 06s)
• 23:28 logmsgbot: krenair Synchronized php-1.25wmf21/extensions/Flow/includes/Formatter/RecentChanges.php: https://gerrit.wikimedia.org/r/#/c/196475/ (duration: 00m 08s)
• 23:18 mutante: cp1050 depooled for reinstall
• 23:15 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/196180/ (duration: 00m 08s)
• 23:11 andrewbogott: imported designate, designate-api, designate-agent, designate-central, designate-common, designate-doc, designate-sink, python-designate to trusty-wikimedia universe, build from gerrit repo ‘openstack-designate’ branch debian/unstable
• 23:09 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/196183 (duration: 00m 14s)
• 22:56 logmsgbot: ori Synchronized php-1.25wmf20/extensions/RestBaseUpdateJobs: (no message) (duration: 00m 06s)
• 22:56 logmsgbot: ori Synchronized php-1.25wmf21/extensions/RestBaseUpdateJobs: (no message) (duration: 00m 08s)
• 22:45 andrewbogott: imported python-pecan_0.6.1-2_all.deb and openstack-pkg-tools_22_all.deb to trusty-wikimedia universe, directly from sid �
• 22:37 logmsgbot: gwicke Synchronized wmf-config/InitialiseSettings.php: Temporarily disable RESTBase updates (duration: 00m 11s)
• 22:27 awight: disabled WorldPay gateway
• 22:19 mutante: cp1051 - repooled in pybal
• 21:52 logmsgbot: mobrovac Synchronized wmf-config/InitialiseSettings.php: RESTBase VRS -> testwikis + mwwiki, RESTBase update ext to all small wikis (duration: 00m 09s)
• 21:46 mutante: cp1051 - disable in pybal, reinstalling
• 20:56 ejegg: updated payments from 673e11f54c613163e7fcf1259935ebb8f9343a73 to 2c5e99cb6de54a6a4e6e2334d533e8ef36c2090c
• 20:49 mutante: repooled cp1066 in pybal - text varnishes in eqiad now 100% Debian
• 20:19 mutante: cp1066 - comment in pybal, reinstall
• 19:49 ejegg: updated payments from a6c451c89620f531c590ddc6d954ac2b808da3df to 673e11f54c613163e7fcf1259935ebb8f9343a73
• 19:41 chasemp: restarting hhvm on mw1120
• 19:20 logmsgbot: rush Synchronized wmf-config/session.php: re-reenable mc1014 (duration: 00m 06s)
• 18:49 awight: updating payments from cbaf66e7705789f37117ec6edc4d936c6174d511 to a6c451c89620f531c590ddc6d954ac2b808da3df
• 18:15 logmsgbot: ori Synchronized wmf-config/session.php: I29542c0965 (duration: 00m 08s)
• 18:09 logmsgbot: rush Synchronized wmf-config/session.php: mc1014 enable (duration: 00m 06s)
• 18:04 akosiaris: started uWSGI on graphite2001
• 17:51 legoktm: restarted populateListOfUsersToBeRenamed.php on terbium (CentralAuth)
• 17:35 awight: payments rolled back from bfa2d27cd9715f7d151c9e1600987fab0d5165e3 to cbaf66e7705789f37117ec6edc4d936c6174d511
• 17:31 awight: update payments from cbaf66e7705789f37117ec6edc4d936c6174d511 to bfa2d27cd9715f7d151c9e1600987fab0d5165e3
• 17:08 mutante: rdb2004 .. but then gets the 'malformed IP address' warning like on rbf2001
• 17:07 mutante: rdb2004 - changed serial settings in bios, boots into installer now (T92011)
• 16:42 logmsgbot: kartik Finished scap: Update ContentTranslation (duration: 02m 25s)
• 16:40 logmsgbot: kartik Started scap: Update ContentTranslation
• 15:21 logmsgbot: catrope Synchronized php-1.25wmf21/extensions/Flow/: SWAT (duration: 00m 07s)
• 10:52 _joe_: depooled mw1152 again
• 10:33 _joe_: repooling mw1152
• 10:32 logmsgbot: oblivian Synchronized wmf-config/CommonSettings.php: Fix for svg conversion on HHVM (duration: 00m 05s)
• 09:15 _joe_: depooling the HHVM imagescaler
• 08:53 _joe_: pooling mw1152, the HHVM imagescaler, into production
• 05:23 logmsgbot: legoktm Synchronized README: testing that Yuvi didnt break anything (duration: 00m 05s)
• 03:45 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Mar 12 03:44:38 UTC 2015 (duration 50m 35s)
• 03:13 logmsgbot: LocalisationUpdate completed (1.25wmf21) at 2015-03-12 03:11:58+00:00
• 03:11 logmsgbot: tstarling Synchronized php-1.25wmf21/cache/l10n: (no message) (duration: 00m 04s)
• 02:58 logmsgbot: LocalisationUpdate completed (1.25wmf20) at 2015-03-12 02:57:01+00:00
• 02:56 logmsgbot: tstarling Synchronized php-1.25wmf20/cache/l10n: (no message) (duration: 00m 03s)
• 02:41 hoo: Manually started 7 Wikibase dispatchChanges instances on terbium after cron failed to start them.
• 02:31 logmsgbot: tstarling Synchronized README: (no message) (duration: 00m 06s)
• 02:15 logmsgbot: tstarling Synchronized README: (no message) (duration: 00m 01s)
• 02:12 logmsgbot: tstarling Synchronized README: (no message) (duration: 00m 01s)
• 02:00 Tim: on tin: disabled puppet for l10nupdate testing
• 02:00 mutante: rbf2001 reboot from busybox :p
• 01:55 logmsgbot: LocalisationUpdate completed (1.25wmf21) at 2015-03-12 01:53:58+00:00
• 01:53 logmsgbot: tstarling Synchronized php-1.25wmf21/cache/l10n: (no message) (duration: 00m 01s)
• 01:53 logmsgbot: LocalisationUpdate completed (1.25wmf20) at 2015-03-12 01:52:11+00:00
• 01:51 logmsgbot: tstarling Synchronized php-1.25wmf20/cache/l10n: (no message) (duration: 00m 02s)
• 01:27 Tim: on tin: testing l10nupdate
• 01:00 tgr: running extensions/GlobalUsage/refreshGlobalimagelinks.php --pages=nonexisting for all wikis (T65594)
• 00:57 tgr: doing refreshGlobalimagelinks.php test runs
• 00:55 logmsgbot: tgr Synchronized php-1.25wmf20/extensions/GlobalUsage/refreshGlobalimagelinks.php: fix script before running for T65594 (duration: 00m 06s)
• 00:42 mutante: rdb2001 attempting another reinstall after fixed netboot
• 00:40 Tim: on tin: fixing ownership and permissions of /tmp/mw-cache-*
• 00:21 logmsgbot: krenair Synchronized php-1.25wmf21/extensions/WikiGrok/includes/Hooks.php: https://gerrit.wikimedia.org/r/#/c/196106/ (duration: 00m 05s)
• 00:11 logmsgbot: krenair Synchronized php-1.25wmf20/extensions/WikiGrok/includes/Hooks.php: https://gerrit.wikimedia.org/r/#/c/196122/ (duration: 00m 07s)

March 11

• 23:59 mutante: powercycling rbf2001, attempt reinstall (wrong IP?)
• 23:42 logmsgbot: krenair Synchronized php-1.25wmf20/extensions/WikiGrok/includes/Hooks.php: revert (duration: 00m 05s)
• 23:41 logmsgbot: krenair Synchronized php-1.25wmf20/extensions/WikiGrok/includes/Hooks.php: https://gerrit.wikimedia.org/r/#/c/196103/ (duration: 00m 08s)
• 23:37 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/195197/3 (duration: 00m 06s)
• 23:21 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/196083/ (duration: 00m 07s)
• 23:11 mutante: reinstalling rdb2001
• 22:11 mutante: cp1052 - comment in pybal, reinstalling
• 21:40 twentyafterfour: finished train deployment
• 21:30 logmsgbot: twentyafterfour Purged l10n cache for 1.25wmf19
• 21:30 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf21
• 21:29 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.25wmf20
• 21:18 logmsgbot: twentyafterfour Finished scap: Sync security patches (duration: 16m 14s)
• 21:02 logmsgbot: twentyafterfour Started scap: Sync security patches
• 20:52 mutante: cp1061 repooled in pybal
• 20:44 logmsgbot: mobrovac Synchronized wmf-config/CommonSettings.php: Activate the RESTBase Virtual REST Service on test.wp (duration: 00m 06s)
• 20:43 logmsgbot: mobrovac Synchronized wmf-config/InitialiseSettings.php: Activate the RESTBase Virtual REST Service on test.wp (duration: 00m 07s)
• 20:42 logmsgbot: twentyafterfour Finished scap: testwiki to php-1.25wmf21 and rebuild l10n cache (duration: 20m 59s)
• 20:21 logmsgbot: twentyafterfour Started scap: testwiki to php-1.25wmf21 and rebuild l10n cache
• 20:11 subbu: deployed parsoid sha 73bf3162
• 19:49 mutante: cp1061 - comment in pybal, reinstalling
• 18:44 mutante: cp1053 - reinstalling, PXE boot
• 18:31 mutante: cp1053 - comment in pybal for reinstall
• 18:09 twentyafterfour: branching wmf/1.25wmf21
• 17:18 Coren: trying other ways to restart uwsgi on labmod1001
• 16:21 logmsgbot: catrope Synchronized php-1.25wmf20/extensions/VisualEditor/: Update and unbreak VE (duration: 00m 06s)
• 05:47 YuviPanda: testing sync-file to make sure I didn’t break anything
• 05:47 logmsgbot: yuvipanda Synchronized README: (no message) (duration: 00m 07s)
• 02:31 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Mar 11 02:30:39 UTC 2015 (duration 30m 38s)
• 02:11 logmsgbot: ori Synchronized php-1.25wmf20/extensions/WikimediaEvents: 2nd iteration of HTTPS test (duration: 00m 05s)
• 02:11 logmsgbot: ori Synchronized php-1.25wmf19/extensions/WikimediaEvents: 2nd iteration of HTTPS test (duration: 00m 05s)
• 02:07 logmsgbot: LocalisationUpdate completed (1.25wmf20) at 2015-03-11 02:05:58+00:00
• 02:05 logmsgbot: l10nupdate Synchronized php-1.25wmf20/cache/l10n: (no message) (duration: 00m 01s)
• 02:05 logmsgbot: LocalisationUpdate completed (1.25wmf19) at 2015-03-11 02:04:22+00:00
• 02:04 logmsgbot: l10nupdate Synchronized php-1.25wmf19/cache/l10n: (no message) (duration: 00m 02s)
• 01:22 bblack: reinstalling cp4007 + cp4015
• 00:55 logmsgbot: ori Synchronized docroot/foundation/misc/blank.gif: (no message) (duration: 00m 05s)

March 10

• 23:31 logmsgbot: ebernhardson Synchronized php-1.25wmf19/extensions/RestBaseUpdateJobs/: Update RestBaseUpdateJobs to master in 1.25wmf19 (duration: 00m 09s)
• 23:30 logmsgbot: ebernhardson Synchronized php-1.25wmf20/extensions/RestBaseUpdateJobs: Update RestBaseUpdateJobs to master in 1.25wmf20 (duration: 00m 06s)
• 23:19 logmsgbot: ebernhardson Synchronized php-1.25wmf19/extensions/Flow: Bump flow submodule in 1.25wmf19 for SWAT (duration: 00m 07s)
• 23:17 logmsgbot: ebernhardson Synchronized php-1.25wmf20/extensions/Flow: Bump flow submodule in 1.25wmf20 for SWAT (duration: 00m 08s)
• 21:27 andrewbogott: erased some api-feature-usage.logs from fluorine to make breathing room; merged a patch that will purge _all_ such logs older than 90 days.
• 21:16 mutante: cp1057 - repooled, all bits eqiad are jessie now
• 20:31 mutante: cp1057 - disabled in pybal, reinstalling
• 20:10 twentyafterfour: finished train deployment, logs look ok
• 19:45 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group1 to 1.25wmf20 for real this time
• 19:44 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group1 to 1.25wmf20
• 19:18 gwicke: re-enabled puppet on cerium, xenon and praseodymium
• 18:27 twentyafterfour: starting the Tuesday "train" deployment
• 17:51 mutante: cp1056 - disabled in pybal, reboot to PXE for reinstall
• 15:41 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: Morning SWAT gerrit:112590 (duration: 00m 06s)
• 15:36 logmsgbot: thcipriani Synchronized wmf-config/InitialiseSettings.php: Morning swat sync of gerrit:195565 (duration: 00m 06s)
• 15:23 logmsgbot: thcipriani Synchronized database lists: (no message) (duration: 00m 07s)
• 15:17 thcipriani: Delete vewikimedia deployed via morning swat gerrit:171219
• 02:20 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Mar 10 02:19:34 UTC 2015 (duration 19m 33s)
• 02:07 logmsgbot: LocalisationUpdate completed (1.25wmf20) at 2015-03-10 02:06:03+00:00
• 02:06 logmsgbot: l10nupdate Synchronized php-1.25wmf20/cache/l10n: (no message) (duration: 00m 01s)
• 02:05 logmsgbot: LocalisationUpdate completed (1.25wmf19) at 2015-03-10 02:04:08+00:00
• 02:04 logmsgbot: l10nupdate Synchronized php-1.25wmf19/cache/l10n: (no message) (duration: 00m 02s)
• 01:24 ori: I749477ac1 follow-up: chmodded 0755 /tmp/mw-cache-* and 0644 /tmp/mw-cache-*/conf-*
• 01:06 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I749477ac1: More secure permissions on conf cache (duration: 00m 06s)

March 9

• 23:34 logmsgbot: legoktm Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 05s)
• 23:33 logmsgbot: legoktm Synchronized php-1.25wmf19/extensions/WikimediaMaintenance/dumpInterwiki.php: https://gerrit.wikimedia.org/r/#/c/195466/ (duration: 00m 06s)
• 23:33 logmsgbot: legoktm Synchronized php-1.25wmf20/extensions/WikimediaMaintenance/dumpInterwiki.php: https://gerrit.wikimedia.org/r/#/c/195467/ (duration: 00m 05s)
• 23:30 logmsgbot: legoktm Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/195450/ (duration: 00m 05s)
• 23:28 logmsgbot: legoktm Synchronized wmf-config/mobile.php: https://gerrit.wikimedia.org/r/#/c/194919/ (duration: 00m 07s)
• 23:27 logmsgbot: legoktm Synchronized php-1.25wmf20/extensions/Thanks/tests/: https://gerrit.wikimedia.org/r/#/c/195290/ (duration: 00m 06s)
• 23:25 logmsgbot: legoktm Synchronized php-1.25wmf20/extensions/VisualEditor/lib/ve/src/ce/nodes/ve.ce.TableCellNode.js: https://gerrit.wikimedia.org/r/#/c/195290/ (duration: 00m 06s)
• 23:11 logmsgbot: legoktm Synchronized php-1.25wmf19/extensions/ImageMetrics/resources/: https://gerrit.wikimedia.org/r/#/c/195447/ (duration: 00m 05s)
• 23:09 logmsgbot: legoktm Synchronized php-1.25wmf20/extensions/ImageMetrics/resources/: https://gerrit.wikimedia.org/r/#/c/195449/ (duration: 00m 06s)
• 22:29 tgr: doing an extensions/GlobalUsage/refreshGlobalimagelinks.php --pages=nonexistent test run on aawiki
• 20:29 logmsgbot: mobrovac Synchronized wmf-config/CommonSettings.php: Set the correct RESTBase server for the RESTBaseUpdateJobs extension (duration: 00m 07s)
• 20:28 logmsgbot: mobrovac Synchronized wmf-config/InitialiseSettings.php: Enable the RESTBaseUpdateJobs extension on testwiki (duration: 00m 06s)
• 20:09 arlolra: updated Parsoid to version c8370a480636c3a0d47ed5090dd29efcb72591e2
• 17:58 akosiaris: restarting pybal on lvs1003 to pick up https://gerrit.wikimedia.org/r/195301
• 17:39 akosiaris: restarting pybal on lvs1006 to pick up https://gerrit.wikimedia.org/r/195301
• 16:44 akosiaris: restarting pybal on lvs1003, lvs1006 for LVS zotero change
• 15:35 ori: restarted HHVM on mw1119; ^d reports TC cache full
• 15:29 logmsgbot: thcipriani Synchronized php-1.25wmf20/extensions/CirrusSearch/: morning swat (duration: 00m 08s)
• 02:21 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Mar 9 02:20:28 UTC 2015 (duration 20m 27s)
• 02:08 logmsgbot: LocalisationUpdate completed (1.25wmf20) at 2015-03-09 02:07:13+00:00
• 02:07 logmsgbot: l10nupdate Synchronized php-1.25wmf20/cache/l10n: (no message) (duration: 00m 01s)
• 02:06 logmsgbot: LocalisationUpdate completed (1.25wmf19) at 2015-03-09 02:05:40+00:00
• 02:05 logmsgbot: l10nupdate Synchronized php-1.25wmf19/cache/l10n: (no message) (duration: 00m 01s)

March 8

• 22:40 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1068 T91920 (duration: 00m 06s)
• 17:21 logmsgbot: marktraceur Synchronized wmf-config/throttle.php: Account creation throttle exemption for Walker Art Center - hopefully soon enough (duration: 00m 06s)
• 15:58 logmsgbot: marc Synchronized wmf-config/InitialiseSettings.php: Adding pool.publicdomainproject.org to wgCopyUploadsDomains (T91927) (duration: 00m 07s)
• 02:16 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Mar 8 02:15:23 UTC 2015 (duration 15m 22s)
• 02:04 logmsgbot: LocalisationUpdate completed (1.25wmf20) at 2015-03-08 02:03:45+00:00
• 02:03 logmsgbot: l10nupdate Synchronized php-1.25wmf20/cache/l10n: (no message) (duration: 00m 02s)
• 02:03 logmsgbot: LocalisationUpdate completed (1.25wmf19) at 2015-03-08 02:02:13+00:00
• 02:02 logmsgbot: l10nupdate Synchronized php-1.25wmf19/cache/l10n: (no message) (duration: 00m 01s)
• 01:33 legoktm: running checkLocalNames.php --delete on commonswiki & wikidatawiki (CentralAuth)

March 7

• 10:13 legoktm: started checkLocalUser.php and checkLocalNames.php scripts (CentralAuth)
• 03:06 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Mar 7 03:05:47 UTC 2015 (duration 5m 46s)
• 02:25 legoktm: manually finished global rename for Just.isabella on commonswiki
• 02:08 logmsgbot: LocalisationUpdate completed (1.25wmf20) at 2015-03-07 02:07:16+00:00
• 02:07 logmsgbot: l10nupdate Synchronized php-1.25wmf20/cache/l10n: (no message) (duration: 00m 01s)
• 02:05 logmsgbot: LocalisationUpdate completed (1.25wmf19) at 2015-03-07 02:03:59+00:00
• 02:03 logmsgbot: l10nupdate Synchronized php-1.25wmf19/cache/l10n: (no message) (duration: 00m 04s)
• 01:51 bblack: repooled cp4014 in pybal
• 00:46 bblack: depooled cp4014 in pybal
• 00:38 hoo: Set wb_changes_dispatch.chd_disabled = 1 for all closed wikis on wikidata
• 00:07 logmsgbot: hoo Synchronized wmf-config/Wikibase.php: Turns out trim is actually needed... (duration: 00m 05s)

March 6

• 23:47 legoktm: started running populateListofUsersToRename.php (CentralAuth)
• 23:21 logmsgbot: demon Synchronized php-1.25wmf19/includes/: db profiling backport (duration: 00m 09s)
• 23:10 logmsgbot: legoktm Synchronized php-1.25wmf19/extensions/CentralAuth/includes/CentralAuthUser.php: https://gerrit.wikimedia.org/r/#/c/194709/ (duration: 00m 05s)
• 23:10 logmsgbot: legoktm Synchronized php-1.25wmf20/extensions/CentralAuth/includes/CentralAuthUser.php: https://gerrit.wikimedia.org/r/#/c/194709/ (duration: 00m 08s)
• 22:27 logmsgbot: hoo Synchronized wmf-config/Wikibase.php: Don't dispatch Wikibase changes to closed Wikis (duration: 00m 06s)
• 21:35 logmsgbot: demon Synchronized docroot/noc/index.html: (no message) (duration: 00m 07s)
• 21:25 logmsgbot: demon Synchronized docroot/noc/: (no message) (duration: 00m 09s)
• 20:56 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: no-op, style fixes (duration: 00m 05s)
• 20:39 andrewbogott: rebooting californium to see what memcached does on startup
• 20:03 logmsgbot: legoktm Finished scap: WikimediaMessages updates (duration: 05m 06s)
• 19:58 logmsgbot: legoktm Started scap: WikimediaMessages updates
• 19:50 mutante: chown demon:releasers-mediawiki 1.24 and below (belonged the removed user 1232/mah)
• 17:50 logmsgbot: legoktm Finished scap: no-op to update messages take 3 (duration: 01m 19s)
• 17:49 logmsgbot: legoktm Started scap: no-op to update messages take 3
• 17:48 logmsgbot: legoktm Synchronized wmf-config/session.php: https://gerrit.wikimedia.org/r/#/c/194897/ (duration: 00m 06s)
• 17:47 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Mar 6 17:46:35 UTC 2015 (duration 17m 32s)
• 17:36 mutante: Key for minion californium.eqiad.wmnet deleted. Key for minion californium.wikimedia.org accepted.
• 17:36 logmsgbot: LocalisationUpdate completed (1.25wmf20) at 2015-03-06 17:35:01+00:00
• 17:34 logmsgbot: legoktm Synchronized php-1.25wmf20/cache/l10n: (no message) (duration: 00m 04s)
• 17:34 logmsgbot: LocalisationUpdate completed (1.25wmf19) at 2015-03-06 17:33:15+00:00
• 17:33 logmsgbot: legoktm Synchronized php-1.25wmf19/cache/l10n: (no message) (duration: 00m 01s)
• 17:29 legoktm: running l10nupdate
• 17:28 logmsgbot: legoktm Finished scap: no-op to update messages take 2 (duration: 01m 15s)
• 17:26 logmsgbot: legoktm Started scap: no-op to update messages take 2
• 17:25 logmsgbot: legoktm Finished scap: no-op to update messages (duration: 02m 23s)
• 17:24 ^d: tin: /srv/mediawiki-staging/ now uses https instead of ssh for origin
• 17:23 logmsgbot: legoktm Started scap: no-op to update messages
• 17:14 logmsgbot: krenair Synchronized php-1.25wmf20/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.Target.js: https://gerrit.wikimedia.org/r/#/c/194870/ (duration: 00m 05s)
• 17:01 chasemp: rebooting mc1014 as totally hung box
• 16:58 logmsgbot: reedy Synchronized private/: Unbreak uploads (duration: 00m 06s)
• 16:57 logmsgbot: reedy Synchronized wmf-config/: Unbreak uploads (duration: 00m 07s)
• 16:44 logmsgbot: krenair Synchronized php-1.25wmf19/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.Target.js: https://gerrit.wikimedia.org/r/#/c/194869/ (duration: 00m 07s)
• 16:17 logmsgbot: demon Synchronized wmf-config/: restructured swift config for multi-dc (duration: 00m 07s)
• 16:17 logmsgbot: demon Synchronized private/PrivateSettings.php: restructure swift config for multi-dc, with b/c (duration: 00m 07s)
• 13:32 springle: db2017 testing innodb_use_native_aio=0 due to InnoDB assertion failure on kernel 3.13
• 09:17 hasharConf: Jenkins: upgrading and restarting. Wish me luck.
• 06:55 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Mar 6 06:54:11 UTC 2015 (duration 54m 10s)
• 06:26 bblack: repooled amssq3[24] + cp1063 in pybal
• 06:01 bblack: depool cp1063 in pybal
• 05:57 bblack: repooled cp1068 in pybal
• 05:30 bblack: depooled cp1068 in pybal
• 05:13 bblack: repooled cp4015 in pybal
• 03:55 superm401: Completed running FlowAddMissingModerationLogs.php and FlowFixLog.php on all Flow wikis
• 03:39 bblack: depooled cp4015 in pybal
• 02:57 bblack: amssq3[24] depooled in pybal
• 02:27 logmsgbot: LocalisationUpdate completed (1.25wmf20) at 2015-03-06 02:26:12+00:00
• 02:26 logmsgbot: l10nupdate Synchronized php-1.25wmf20/cache/l10n: (no message) (duration: 00m 04s)
• 02:26 bblack: cp3017 repooled in pybal
• 02:25 logmsgbot: LocalisationUpdate completed (1.25wmf19) at 2015-03-06 02:24:04+00:00
• 02:24 logmsgbot: l10nupdate Synchronized php-1.25wmf19/cache/l10n: (no message) (duration: 00m 01s)
• 01:32 bblack: depooled cp3017 in pybal
• 01:28 bblack: repooled cp3014,cp3022 in pybal
• 01:27 springle: reindexing s7 pagelinks T89630
• 01:13 springle: killing query storm on s7, SpecialWhatLinksHere::showIndirectLinks
• 00:33 logmsgbot: legoktm Finished scap: Flow and WikimediaMessages updates (duration: 14m 53s)
• 00:18 logmsgbot: legoktm Started scap: Flow and WikimediaMessages updates
• 00:15 bblack: depooled cp3013 in pybal

March 5

• 23:44 bblack: depooled cp3022 in pybal
• 23:38 logmsgbot: demon Synchronized docroot/noc/index.html: (no message) (duration: 00m 06s)
• 21:41 andrewbogott: moved californium to a public ip on labs-hosts1-b-eqiad, rebooted
• 21:15 hashar: restarting Jenkins (and kill -9 ing it)
• 19:02 bblack: repooled cp301[48] in pybal
• 18:43 bblack: depool cp3018 esams pybal
• 18:24 bblack: depooled cp3014 frontend-only (esams upload)
• 17:46 twentyafterfour: pushing 1.25wmf20 branches which were missed by yesterday's deployment
• 17:31 ^d: updated php-1.25wmf(19|20) remotes to use https instead of ssh
• 17:31 logmsgbot: demon Synchronized multiversion/checkoutMediaWiki.php: (no message) (duration: 00m 06s)
• 17:28 logmsgbot: kartik Finished scap: Update ContentTranslation (duration: 15m 37s)
• 17:12 logmsgbot: kartik Started scap: Update ContentTranslation
• 16:43 kart_: Updated cxserver to 2695a31
• 16:27 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Enable Content Translation in kywiki and pawiki (duration: 00m 07s)
• 16:15 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: CX: remove labs customization (duration: 00m 07s)
• 16:14 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Publish translations to the Main namespace by default (duration: 00m 05s)
• 16:03 bblack: repooled cp301[48] in pybal
• 15:31 akosiaris: restarted phd (phabricator daemon) on iridium
• 15:15 bblack: depool cp3018 in esams pybal
• 14:50 bblack: depooled cp3014 in esams pybal
• 14:24 logmsgbot: krenair Synchronized wmf-config/logging-labs.php: https://gerrit.wikimedia.org/r/#/c/194508/ (duration: 00m 07s)
• 14:05 logmsgbot: krenair Synchronized wmf-config/interwiki.cdb: Interwiki cache update (duration: 00m 06s)
• 04:14 mutante: started nagios-nrpe on rhenium
• 02:38 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Mar 5 02:37:32 UTC 2015 (duration 37m 31s)
• 02:18 logmsgbot: LocalisationUpdate completed (1.25wmf20) at 2015-03-05 02:17:37+00:00
• 02:17 logmsgbot: l10nupdate Synchronized php-1.25wmf20/cache/l10n: (no message) (duration: 00m 02s)
• 02:06 logmsgbot: LocalisationUpdate completed (1.25wmf19) at 2015-03-05 02:05:17+00:00
• 02:05 logmsgbot: l10nupdate Synchronized php-1.25wmf19/cache/l10n: (no message) (duration: 00m 02s)
• 00:26 logmsgbot: maxsem Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/194074/ (duration: 00m 06s)

March 4

• 23:59 gwicke: stopping cassandra cluster for cleanup
• 23:10 chasemp: Enable test/phase0 and *.wikipedia.org wikis in restbase https://gerrit.wikimedia.org/r/#/c/194244/
• 22:09 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf20
• 21:51 logmsgbot: twentyafterfour Finished scap: Wikipedias to 1.25wmf19, testwiki to 1.25wmf20 and rebuild l10n cache (duration: 17m 26s)
• 21:33 logmsgbot: twentyafterfour Started scap: Wikipedias to 1.25wmf19, testwiki to 1.25wmf20 and rebuild l10n cache
• 21:09 subbu: deployed parsoid version 06c8cf33
• 20:29 hoo: Manually completed the global rename Gabriel2517 -> WikiGuy2517 (was stuck on WD.o)
• 19:03 twentyafterfour: Creating new deployment branch 1.25wmf20
• 16:09 logmsgbot: anomie Synchronized wmf-config: SWAT: Beta-only change: CX: Add wgContentTranslationCampaigns gerrit:194265 (duration: 00m 07s)
• 14:46 logmsgbot: anomie Synchronized php-1.25wmf19/extensions/Graph: early SWAT: Update Graph extension to fix IE bug gerrit:194326 (duration: 00m 06s)
• 13:11 bblack: depooled amssq31 in esams for reinstall
• 07:24 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Mar 4 07:23:48 UTC 2015 (duration 23m 47s)
• 02:29 logmsgbot: LocalisationUpdate completed (1.25wmf19) at 2015-03-04 02:28:15+00:00
• 02:28 logmsgbot: l10nupdate Synchronized php-1.25wmf19/cache/l10n: (no message) (duration: 00m 01s)
• 02:16 logmsgbot: LocalisationUpdate completed (1.25wmf18) at 2015-03-04 02:15:10+00:00
• 02:15 logmsgbot: l10nupdate Synchronized php-1.25wmf18/cache/l10n: (no message) (duration: 00m 01s)
• 00:44 logmsgbot: demon Finished scap: evening swat: centralauth, VE, user.php fix (duration: 30m 19s)
• 00:14 logmsgbot: demon Started scap: evening swat: centralauth, VE, user.php fix
• 00:09 gwicke: running concurrent test dumps of enwiki and dewiki through xenon

March 3

• 23:38 mutante: ran puppet on ruthenium (keeps showing up in icinga but then no issue when you run it)
• 22:56 ori: adding four additional txstatsd backends to graphite1001 to cope with load
• 21:49 logmsgbot: demon Synchronized docroot/mediawiki/keys/: (no message) (duration: 00m 05s)
• 21:45 gwicke: re-initializing cassandra on test hosts xenon, praseodymium and cerium for new test run; expect some downtime
• 20:05 bblack: cp10(60|64|65|70),amssq42 repooled in pybal
• 19:30 logmsgbot: demon Synchronized multiversion/MWMultiVersion.php: moar debugging (duration: 00m 07s)
• 19:30 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group1 to 1.25wmf19
• 19:05 twentyafterfour: Starting the tuesday train deploy ( Group1 wikis to 1.25wmf19 )
• 18:39 bblack: depooled cp10(60|65|70),amssq42 temporarily in pybal for reinstalls
• 17:03 logmsgbot: marktraceur Synchronized wmf-config/InitialiseSettings.php: [SWAT] [config] Enable BounceHandler on non-wikipedias (duration: 00m 05s)
• 16:53 logmsgbot: marktraceur Synchronized wmf-config/InitialiseSettings.php: [SWAT] [config] Remove redundant namespace aliases for Nepali Wikipedia (newiki) (duration: 00m 05s)
• 16:41 logmsgbot: marktraceur Synchronized php-1.25wmf19/extensions/Graph/: [SWAT] [wmf19] Update Graph to 25.19 (duration: 00m 07s)
• 16:30 logmsgbot: marktraceur Synchronized wmf-config/InitialiseSettings.php: [SWAT] [config] Add namespace aliases for Nepali Wikipedia (newiki) (duration: 00m 08s)
• 16:22 logmsgbot: marktraceur Synchronized wmf-config/InitialiseSettings-labs.php: [SWAT] [config] Kartik forgot to sync this beta config patch for CX (duration: 00m 10s)
• 16:22 logmsgbot: marktraceur Synchronized wmf-config/CommonSettings-labs.php: [SWAT] [config] Enable Flickr uploads on betacommons (duration: 00m 06s)
• 03:31 springle: codfw replag coming up, schema changes
• 02:29 logmsgbot: andyrussg Synchronized php-1.25wmf18/extensions/CentralNotice/: CenralNotice update (duration: 00m 07s)
• 02:18 logmsgbot: andyrussg Synchronized php-1.25wmf19/extensions/CentralNotice/: CenralNotice update (duration: 00m 09s)
• 02:17 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Mar 3 02:16:47 UTC 2015 (duration 16m 46s)
• 02:06 MaxSem: Creating geo_tags table everywhere it's not yet present
• 02:06 logmsgbot: LocalisationUpdate completed (1.25wmf19) at 2015-03-03 02:04:56+00:00
• 02:04 logmsgbot: l10nupdate Synchronized php-1.25wmf19/cache/l10n: (no message) (duration: 00m 01s)
• 02:04 logmsgbot: LocalisationUpdate completed (1.25wmf18) at 2015-03-03 02:03:26+00:00
• 02:03 logmsgbot: l10nupdate Synchronized php-1.25wmf18/cache/l10n: (no message) (duration: 00m 02s)
• 00:51 ejegg: enabled queue consumers
• 00:42 ejegg: updated crm from 3c002f32e04652ae56a4fe791bc6158ab981ed8d to f8fb0f61531431348f3a8a3ee107056a864d537b
• 00:39 ejegg: disabled queue consumers
• 00:23 logmsgbot: demon Synchronized php-1.25wmf18/extensions/Flow: (no message) (duration: 00m 07s)
• 00:15 logmsgbot: demon Synchronized wmf-config/: cors logging for beta (duration: 00m 06s)
• 00:11 logmsgbot: demon Synchronized php-1.25wmf19/extensions/VisualEditor: (no message) (duration: 00m 06s)
• 00:11 logmsgbot: demon Synchronized php-1.25wmf18/extensions/VisualEditor: (no message) (duration: 00m 05s)
• 00:09 logmsgbot: demon Synchronized php-1.25wmf19/includes/Linker.php: (no message) (duration: 00m 05s)
• 00:07 logmsgbot: demon Synchronized docroot/noc/: rm dbtree (duration: 00m 06s)

March 2

• 23:44 ejegg: updated civi-staging to f8fb0f61531431348f3a8a3ee107056a864d537b
• 23:42 logmsgbot: maxsem Synchronized wmf-config/: https://gerrit.wikimedia.org/r/193988 (duration: 00m 07s)
• 22:49 bblack: repooled cp1064 frontend (upload eqiad)
• 21:22 ori: deployed patch for T88361
• 21:17 subbu: deployed parsoid version 08643f53
• 20:31 ori: On graphite1001, updated statsdlb to 0.2-1
• 20:12 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 05s)
• 20:10 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 07s)
• 20:08 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 08s)
• 20:05 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 08s)
• 19:56 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 06s)
• 19:48 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 05s)
• 19:46 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 05s)
• 19:36 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 05s)
• 19:25 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 06s)
• 19:19 manybubbles: looks like that didn't cover the whole range - expanding the range of reindexed data - starting against for enwiki
• 19:18 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: collection on all wikis (duration: 00m 07s)
• 19:15 manybubbles: starting on all other wikis
• 19:15 manybubbles: finished Cirrus outage recovery job script for enwiki
• 18:34 manybubbles: starting script to reindex search changes made yesterday night on enwiki (script is https://wikitech.wikimedia.org/wiki/Search#Recovering_from_an_Elasticsearch_outage.2Finterruption_in_updates)
• 16:46 manybubbles: correction to last sync -message - was totally wrong - patch instead did this: "Enable NewUserMessage extension for fawiktionary "
• 16:45 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT - Enable WikiLove extension at newiki (duration: 00m 07s)
• 16:45 logmsgbot: manybubbles Synchronized wmf-config/abusefilter.php: SWAT - AbuseFilter config change for ukwiki (duration: 00m 07s)
• 16:41 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT - AbuseFilter config change for ukwiki (duration: 00m 07s)
• 16:39 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT - Change templateeditor user group rights on fawiki (duration: 00m 07s)
• 16:35 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT - Set $wgBabelCategoryNames true at outreachwiki (duration: 00m 06s)
• 16:05 logmsgbot: manybubbles Synchronized wmf-config/Wikibase.php: SWAT wikidata - add badge items for beta (duration: 00m 06s)
• 16:04 logmsgbot: manybubbles Synchronized wmf-config/Wikibase-production.php: SWAT wikidata - add badge items for beta (duration: 00m 07s)
• 16:04 logmsgbot: manybubbles Synchronized wmf-config/Wikibase-labs.php: SWAT wikidata - add badge items for beta (duration: 00m 06s)
• 15:15 bblack: temporarily depooling cp1064 (eqiad upload) for reinstall
• 05:54 logmsgbot: tstarling Synchronized langlist: (no message) (duration: 00m 06s)
• 05:33 Tim: on terbium: fixed permissions on /srv/mediawiki/multiversion
• 05:32 logmsgbot: tstarling Finished scap: Ieb27df7ef470cbda06b5b0f5bfb372bd7279c183 (duration: 02m 17s)
• 05:29 logmsgbot: tstarling Started scap: Ieb27df7ef470cbda06b5b0f5bfb372bd7279c183
• 05:29 Tim: on tin: updating deployment branches for Ieb27df7ef470cbda06b5b0f5bfb372bd7279c183

March 1

• 06:23 andrewbogott: logging a test to test the logging
• 02:17 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Mar 1 02:16:24 UTC 2015 (duration 16m 23s)
• 02:06 logmsgbot: LocalisationUpdate completed (1.25wmf19) at 2015-03-01 02:05:02+00:00
• 02:04 logmsgbot: l10nupdate Synchronized php-1.25wmf19/cache/l10n: (no message) (duration: 00m 01s)
• 02:04 logmsgbot: LocalisationUpdate completed (1.25wmf18) at 2015-03-01 02:03:30+00:00
• 02:03 logmsgbot: l10nupdate Synchronized php-1.25wmf18/cache/l10n: (no message) (duration: 00m 01s)
• 00:56 gwicke: stopped cassandra on cerium and praseodymium temporarily for testing

February 28

• 02:18 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Feb 28 02:17:47 UTC 2015 (duration 17m 46s)
• 02:06 logmsgbot: LocalisationUpdate completed (1.25wmf19) at 2015-02-28 02:05:51+00:00
• 02:05 logmsgbot: l10nupdate Synchronized php-1.25wmf19/cache/l10n: (no message) (duration: 00m 01s)
• 02:05 logmsgbot: LocalisationUpdate completed (1.25wmf18) at 2015-02-28 02:04:14+00:00
• 02:04 logmsgbot: l10nupdate Synchronized php-1.25wmf18/cache/l10n: (no message) (duration: 00m 01s)
• 00:16 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: If2704b0f7: Change metric prefix from 'mw' back to 'MediaWiki', for back-compat (duration: 00m 06s)

February 27

• 23:47 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I8fa0649ab: Set $wgUDPProfilerPort back to 8125 (duration: 00m 06s)
• 23:33 ori: pushing a config change to txstatsd on graphite1001, the service may complain briefly
• 22:19 logmsgbot: reedy Synchronized docroot and w: nooop for dbtree ( already reverted by prior deploy ) (duration: 00m 05s)
• 17:07 legoktm: running CentralAuth's migratePass0.php on all wikis
• 14:59 hoo: Ran mysql:wikiadmin@db1033 [metawiki]> UPDATE ipblocks SET ipb_deleted = 1 WHERE ipb_id = 16659; to actually suppress a suppressed name
• 08:41 andrewbogott: upgraded virt1012 to Trusty; starting all instances
• 06:25 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Feb 27 06:24:00 UTC 2015 (duration 23m 59s)
• 05:41 andrewbogott: upgrading virt1012 to Trusty because labs networking failed twice in two hours, and how could it be worse?
• 02:19 logmsgbot: LocalisationUpdate completed (1.25wmf19) at 2015-02-27 02:18:19+00:00
• 02:18 logmsgbot: l10nupdate Synchronized php-1.25wmf19/cache/l10n: (no message) (duration: 00m 01s)
• 02:17 logmsgbot: LocalisationUpdate completed (1.25wmf18) at 2015-02-27 02:16:14+00:00
• 02:16 logmsgbot: l10nupdate Synchronized php-1.25wmf18/cache/l10n: (no message) (duration: 00m 02s)
• 01:40 springle: switch db1046 to master of m4 (eventlogging). deployed dbproxy1004 with m4-master CNAME
• 01:18 logmsgbot: krenair Synchronized php-1.25wmf19/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.Target.js: https://gerrit.wikimedia.org/r/#/c/193313/ (duration: 00m 06s)
• 00:58 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/193273/ and https://gerrit.wikimedia.org/r/#/c/193274/ (duration: 00m 06s)
• 00:48 logmsgbot: krenair Synchronized wmf-config: touch (duration: 00m 06s)
• 00:47 logmsgbot: krenair Synchronized wmf-config: https://gerrit.wikimedia.org/r/193308 (duration: 00m 10s)
• 00:43 logmsgbot: krenair Synchronized php-1.25wmf19/skins/Vector/skinStyles/mediawiki.sectionAnchor.less: https://gerrit.wikimedia.org/r/#/c/193310/ (duration: 00m 05s)
• 00:15 logmsgbot: krenair Synchronized wmf-config: touched initialsettings (duration: 00m 07s)
• 00:09 logmsgbot: krenair Synchronized wmf-config: rv (duration: 00m 06s)
• 00:07 logmsgbot: krenair Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/193283/ (duration: 00m 06s)

February 26

• 22:59 Krinkle: git-deploy: Deploying integration/slave-scripts b532a9a..05a5593
• 21:55 gwicke: disabled puppet on cassandra test hosts cerium and praseodymium as well (in addition to xenon) to manually fix incompatible puppet config & re-initialize cluster after cluster name change; see https://phabricator.wikimedia.org/T90955 for upgrade to jessie
• 21:00 ^d: mw1161 is complaining about permissions on setting mtime during rsync
• 20:58 logmsgbot: demon Synchronized multiversion/MWMultiVersion.php: moar debug (duration: 00m 06s)
• 20:40 gwicke: issue with cassandra test cluster is actually that it's still running cassandra 2.1.2, which is incompatible with the current puppet config; should probably update the test cluster to jessie soon
• 20:38 gwicke: cassandra on test cluster seems to be broken, investigating
• 20:16 gwicke: disabled puppet on xenon to test bulk db creation with restbase
• 20:03 logmsgbot: demon Synchronized multiversion/MWMultiVersion.php: debuggg (duration: 00m 06s)
• 17:34 logmsgbot: kartik Finished scap: Update ContentTranslation (duration: 15m 58s)
• 17:18 logmsgbot: kartik Started scap: Update ContentTranslation
• 17:03 logmsgbot: brion Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 05s)
• 17:03 logmsgbot: demon Synchronized README: look ma, no key forwarding (duration: 00m 05s)
• 16:55 kart_: Updated cxserver to 4e09ee8
• 16:36 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/193112/ (duration: 00m 07s)
• 16:33 Krenair: ran sql.php --wiki=ruwiki php-1.25wmf18/extensions/EducationProgram/sql/EducationProgram.sql
• 16:30 logmsgbot: krenair Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/193110/ (duration: 00m 08s)
• 16:25 Krenair: Running updateCollation.php on hsbwiki
• 16:23 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/192803 (duration: 00m 07s)
• 16:17 logmsgbot: krenair Synchronized php-1.25wmf19/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.Target.js: https://gerrit.wikimedia.org/r/#/c/193024/ (duration: 00m 05s)
• 16:06 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/192764/ (duration: 00m 07s)
• 15:57 ottomata: restarted resourcemanager on analytics1001 to load new fairscheduler settings
• 15:14 logmsgbot: demon Synchronized php-1.25wmf18/extensions/RestBaseUpdateJobs: (no message) (duration: 00m 06s)
• 06:22 Tim: on mw1088 restarting hhvm
• 05:52 springle: pre-empt m3/m4 shard split and reclaim disk space on db2011
• 02:42 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Feb 26 02:41:08 UTC 2015 (duration 41m 7s)
• 02:22 logmsgbot: LocalisationUpdate completed (1.25wmf19) at 2015-02-26 02:21:38+00:00
• 02:21 logmsgbot: l10nupdate Synchronized php-1.25wmf19/cache/l10n: (no message) (duration: 00m 01s)
• 02:10 logmsgbot: LocalisationUpdate completed (1.25wmf18) at 2015-02-26 02:09:28+00:00
• 02:09 logmsgbot: l10nupdate Synchronized php-1.25wmf18/cache/l10n: (no message) (duration: 00m 02s)
• 01:18 logmsgbot: maxsem Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/193018/ (duration: 00m 07s)
• 01:13 logmsgbot: hoo Synchronized php-1.25wmf19/extensions/Wikidata/: Update Wikidata to fix EntityViewPlaceholderExpander (duration: 00m 12s)
• 00:59 logmsgbot: maxsem Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/192987/ (duration: 00m 06s)
• 00:57 logmsgbot: maxsem Synchronized php-1.25wmf18/extensions/WikiGrok/: (no message) (duration: 00m 07s)
• 00:53 logmsgbot: maxsem Synchronized php-1.25wmf19/extensions/WikiGrok/: (no message) (duration: 00m 07s)
• 00:32 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/192709/ (duration: 00m 05s)
• 00:28 logmsgbot: ebernhardson Synchronized php-1.25wmf18/extensions/Flow: Bump flow submodule in 1.25wmf18 for infinite scroll fix (duration: 00m 09s)

February 25

• 23:33 logmsgbot: maxsem Synchronized docroot and w: (no message) (duration: 00m 06s)
• 23:22 andrewbogott: upgrading virt1005 to trusty
• 22:53 hoo: Ran rebuildEntityPerPage.php on wikidatawiki to clean up after wikigrok database mess
• 22:51 andrewbogott: rebooting virt1005 in anticipation of an exprimental upgrade to Trusty. (There are no VMs on virt1005 other than a testing host)
• 22:44 twentyafterfour: Done deploying - uploaded release notes for 1.25wmf19
• 22:36 logmsgbot: twentyafterfour Purged l10n cache for 1.25wmf17
• 22:32 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf19
• 22:26 robh: finished with my deployment window for redirects, tested and is now live with no issues (so far)
• 22:25 logmsgbot: twentyafterfour Synchronized ./wmf-config/InitialiseSettings.php: mw1204 still logging errors (duration: 00m 05s)
• 22:22 logmsgbot: twentyafterfour Synchronized ./wmf-config/InitialiseSettings.php: flooding logs (duration: 00m 07s)
• 22:19 logmsgbot: twentyafterfour Synchronized ./wmf-config/InitialiseSettings.php: (no message) (duration: 00m 05s)
• 22:16 robh: localtesting of change on mw1001 shows no issues, so pushing out to rest of apaches
• 22:13 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.25wmf18
• 22:08 Reedy: morebots is dead
• 22:03 robh: disabling puppet on all mw systems for redirects update
• 21:54 logmsgbot: twentyafterfour Finished scap: testwiki to php-1.25wmf19 and rebuild l10n cache (duration: 30m 20s)
• 21:23 logmsgbot: twentyafterfour Started scap: testwiki to php-1.25wmf19 and rebuild l10n cache
• 21:12 arlolra: updated Parsoid to version 5a3aaf712c334190a97a1d224a9efc0fb340f6af
• 20:43 andrewbogott: restarted nova-compute on virt1002
• 19:47 gwicke: restarted restbase1005 with new GC settings
• 19:17 gwicke: restarted cassandra on restbase1003 with new GC settings from puppet
• 16:59 logmsgbot: krenair Synchronized php-1.25wmf18/extensions/VisualEditor/modules/ve-mw/ui/dialogs/ve.ui.MWTemplateDialog.js: * https://gerrit.wikimedia.org/r/#/c/192750/ (duration: 00m 06s)
• 16:05 logmsgbot: demon Synchronized commonsuploads.dblist: (no message) (duration: 00m 07s)
• 16:05 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 05s)
• 15:52 cmjohnson1: replacing PEM2 cr1-eqiad
• 15:49 cmjohnson1: replacing PEM1 on cr1-eqiad
• 15:41 mutante: welcome legoktm as a contint admin
• 10:00 _joe_: restarted hhvm on mw1229, stuck in __lll_lock_wait from HPHP::hphp_session_init
• 07:04 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Feb 25 07:03:41 UTC 2015 (duration 3m 40s)
• 06:22 greg-g: 06:20 < twentyaft> that log was bogus, just me testing but not actually syncing
• 06:17 logmsgbot: twentyafterfour Synchronized php-1.25wmf18/cache/l10n: (no message) (duration: 00m 02s)
• 04:38 springle: s/db1001/dbproxy1001/g on zirconium drupal contacts. seems unpuppetized
• 02:35 logmsgbot: LocalisationUpdate completed (1.25wmf18) at 2015-02-25 02:34:23+00:00
• 02:34 logmsgbot: l10nupdate Synchronized php-1.25wmf18/cache/l10n: (no message) (duration: 00m 03s)
• 02:22 logmsgbot: LocalisationUpdate completed (1.25wmf17) at 2015-02-25 02:21:53+00:00
• 02:21 logmsgbot: l10nupdate Synchronized php-1.25wmf17/cache/l10n: (no message) (duration: 00m 02s)

February 24

• 23:37 springle: killed runaway RecentChangesUpdateJob::purgeExpiredRows transactions from jobrunners. db1033 db1038 db1040 db1052 db1058
• 22:58 logmsgbot: aaron Synchronized php-1.25wmf17/includes/jobqueue/jobs/RecentChangesUpdateJob.php: 6f6d7e57be0ccff8ed2473b7d250e77703c7a6dd (duration: 00m 09s)
• 22:43 logmsgbot: aude Synchronized docroot/mediawiki/xml/: Add sitelist export-import docs (duration: 00m 07s)
• 22:28 logmsgbot: aude Synchronized wikidataclient.dblist: Enable Wikibase Client on Wikibooks (duration: 00m 06s)
• 22:27 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Wikibase config for Wikibooks (duration: 00m 06s)
• 22:25 logmsgbot: aude Synchronized wmf-config/Wikibase.php: Bump cache epoch for Wikidata (duration: 00m 06s)
• 22:20 logmsgbot: aude Synchronized wmf-config/Wikibase.php: Enable Wikibooks sitelinks on Wikidata (duration: 00m 06s)
• 22:08 logmsgbot: aude Finished scap: Updates for enabling Wikibase on Wikibooks (duration: 16m 03s)
• 21:52 logmsgbot: aude Started scap: Updates for enabling Wikibase on Wikibooks
• 21:42 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: sync-wikiversions group1 to 1.25wmf18
• 21:25 logmsgbot: twentyafterfour Synchronized php-1.25wmf18/extensions/Popups: hotfix for Popups (see https://gerrit.wikimedia.org/r/#/c/192465/) (duration: 00m 06s)
• 21:24 ottomata: increased Hadoop nodemanager cpu-vcores to facter $processcount - 1, this should increase hadoop cluster utilization
• 20:31 paravoid: restarting cr1-eqiad/re1 chassis-control; should not be traffic-disrupting
• 20:05 qchris: Updated gerrit plugin its-phabricator-from-bugzilla to 03b936b2cd8fa6adfdbee0ef68eb4b31944936c2
• 20:05 qchris: Updated gerrit plugin its-phabricator to 25a34d7564cffb90a87110a971782195ba2db467
• 17:55 Coren: Shutting down labstore1001 - planned outage for expansion
• 14:59 godog: rolling restart cassandra to pick up metrics configuration
• 07:55 andrewbogott: ‘nova reset-state —active’ and ‘nova reboot’ for EVERY instance on virt1012
• 07:13 andrewbogott: suspending all instances on virt1012
• 06:40 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Feb 24 06:39:18 UTC 2015 (duration 39m 17s)
• 06:29 YuviPanda: stopped nova-compute on virt1005
• 06:28 YuviPanda: starting nova-compute on virt1005
• 04:07 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: raise db1066 load (duration: 00m 07s)
• 02:50 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1066, warm up (duration: 00m 06s)
• 02:19 logmsgbot: LocalisationUpdate completed (1.25wmf18) at 2015-02-24 02:18:03+00:00
• 02:17 logmsgbot: l10nupdate Synchronized php-1.25wmf18/cache/l10n: (no message) (duration: 00m 01s)
• 02:17 logmsgbot: LocalisationUpdate completed (1.25wmf17) at 2015-02-24 02:16:30+00:00
• 02:16 logmsgbot: l10nupdate Synchronized php-1.25wmf17/cache/l10n: (no message) (duration: 00m 02s)
• 01:52 logmsgbot: ori Synchronized php-1.25wmf17/extensions/MobileFrontend/includes/modules/MobileUserModule.php: Reverting live-hack (duration: 00m 07s)
• 01:47 logmsgbot: ori Synchronized php-1.25wmf17/extensions/MobileFrontend/includes/modules/MobileUserModule.php: Testing a theory for T90411 with a live-hack to MobileFrontend. Will revert momentarily. (duration: 00m 07s)
• 00:56 Tim: on osmium, removing the packages I just installed since I will do it in a chroot instead
• 00:51 logmsgbot: twentyafterfour Synchronized php-1.25wmf18/cache/l10n: (no message) (duration: 00m 03s)
• 00:43 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 07s)
• 00:10 logmsgbot: demon Synchronized php-1.25wmf18/extensions/MultimediaViewer: (no message) (duration: 00m 09s)
• 00:10 logmsgbot: demon Synchronized php-1.25wmf17/extensions/MultimediaViewer: (no message) (duration: 00m 06s)

February 23

• 23:28 Tim: on osmium installing packages necessary for building hhvm
• 21:06 subbu: deployed parsoid version d9ac8c21
• 20:43 awight: update crm from f594a66694d52af1c604b1813ac94e9592b6c81e to 3c002f32e04652ae56a4fe791bc6158ab981ed8d
• 18:19 ^d: created education program tables for hewiktionary
• 18:18 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: T89393 (duration: 00m 06s)
• 17:45 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: T90340 (duration: 00m 05s)
• 17:40 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: T89346 (duration: 00m 07s)
• 17:31 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: T88591 (duration: 00m 06s)
• 17:27 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: T89147 (duration: 00m 05s)
• 17:11 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 05s)
• 16:33 paravoid: updating jessie d-i image to currently nightly
• 16:26 _joe_: depooling mw1062 for testing for T86652
• 09:26 godog: reboot ms-be1009, xfs hosed
• 05:41 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1066 (duration: 00m 06s)
• 02:16 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Feb 23 02:15:36 UTC 2015 (duration 15m 35s)
• 02:05 logmsgbot: LocalisationUpdate completed (1.25wmf18) at 2015-02-23 02:04:00+00:00
• 02:03 logmsgbot: l10nupdate Synchronized php-1.25wmf18/cache/l10n: (no message) (duration: 00m 01s)
• 02:03 logmsgbot: LocalisationUpdate completed (1.25wmf17) at 2015-02-23 02:02:28+00:00
• 02:02 logmsgbot: l10nupdate Synchronized php-1.25wmf17/cache/l10n: (no message) (duration: 00m 01s)

February 22

• 22:50 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: db1065 raise load (duration: 00m 07s)
• 22:37 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1065, warm up (duration: 00m 05s)
• 22:08 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1065 (duration: 00m 06s)
• 10:33 godog: reboot ms-be1007, xfs hosed
• 10:23 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1065 (duration: 00m 05s)
• 02:17 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Feb 22 02:16:04 UTC 2015 (duration 16m 3s)
• 02:05 logmsgbot: LocalisationUpdate completed (1.25wmf18) at 2015-02-22 02:04:55+00:00
• 02:04 logmsgbot: l10nupdate Synchronized php-1.25wmf18/cache/l10n: (no message) (duration: 00m 01s)
• 02:04 logmsgbot: LocalisationUpdate completed (1.25wmf17) at 2015-02-22 02:03:23+00:00
• 02:03 logmsgbot: l10nupdate Synchronized php-1.25wmf17/cache/l10n: (no message) (duration: 00m 01s)
• 01:49 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1065 (duration: 00m 06s)

February 21

• 06:51 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Feb 21 06:50:49 UTC 2015 (duration 50m 48s)
• 02:18 logmsgbot: LocalisationUpdate completed (1.25wmf18) at 2015-02-21 02:17:15+00:00
• 02:17 logmsgbot: l10nupdate Synchronized php-1.25wmf18/cache/l10n: (no message) (duration: 00m 01s)
• 02:16 logmsgbot: LocalisationUpdate completed (1.25wmf17) at 2015-02-21 02:15:31+00:00
• 02:15 logmsgbot: l10nupdate Synchronized php-1.25wmf17/cache/l10n: (no message) (duration: 00m 02s)

February 20

• 23:12 logmsgbot: ori Synchronized php-1.25wmf17/extensions/VisualEditor: f14dc93302: Update VisualEditor for cherry-picks (duration: 00m 06s)
• 23:11 logmsgbot: ori Synchronized php-1.25wmf18/extensions/VisualEditor: 5c4457a555: Update VisualEditor for cherry-picks (duration: 00m 05s)
• 19:20 awight|specter: enabled scheduled reminders in production CiviCRM
• 19:05 robh: neon runs puppet fine, back to full service
• 18:54 robh: i broke puppet on neon, workign to fix
• 18:39 robh: killing icinga-admin.w.o url support per T90002
• 18:39 robh: killing icinga-admin.w.o url support
• 18:03 ori: added mobrovac to mediawiki and services gerrit groups
• 16:25 awight: updated crm from a1e604b93342f5555427eaeb81092bfa431ff093 to f594a66694d52af1c604b1813ac94e9592b6c81e
• 15:39 cmjohnson1: virt1002 removing disk 0 which should be /dev/sda
• 10:02 godog: reboot restbase1006 after disk reseat
• 09:16 logmsgbot: twentyafterfour Synchronized wmf-config/CommonSettings.php: wgTranslateBlacklist (duration: 00m 07s)
• 06:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Feb 20 06:06:39 UTC 2015 (duration 6m 38s)
• 02:25 logmsgbot: LocalisationUpdate completed (1.25wmf18) at 2015-02-20 02:24:52+00:00
• 02:24 logmsgbot: l10nupdate Synchronized php-1.25wmf18/cache/l10n: (no message) (duration: 00m 01s)
• 02:23 logmsgbot: LocalisationUpdate completed (1.25wmf17) at 2015-02-20 02:22:50+00:00
• 02:22 logmsgbot: l10nupdate Synchronized php-1.25wmf17/cache/l10n: (no message) (duration: 00m 01s)
• 00:33 logmsgbot: maxsem Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 05s)
• 00:33 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings.php: Pull WikiGrok from wikidata for now (duration: 00m 05s)
• 00:28 logmsgbot: catrope Synchronized php-1.25wmf18/extensions/ZeroBanner: SWAT (duration: 00m 06s)
• 00:27 logmsgbot: catrope Synchronized php-1.25wmf17/extensions/ZeroBanner: SWAT (duration: 00m 06s)
• 00:25 logmsgbot: tstarling Synchronized php-1.25wmf17/extensions/Collection/Collection.body.php: (no message) (duration: 00m 07s)
• 00:21 logmsgbot: tstarling Synchronized php-1.25wmf17/extensions/Collection/Collection.body.php: (no message) (duration: 00m 05s)
• 00:07 logmsgbot: catrope Synchronized wmf-config/mobile.php: SWAT (duration: 00m 06s)
• 00:07 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings.php: SWAT (duration: 00m 08s)

February 19

• 23:06 logmsgbot: ori Synchronized php-1.25wmf17/extensions/WikimediaEvents: (no message) (duration: 00m 06s)
• 23:06 logmsgbot: ori Synchronized php-1.25wmf17/extensions/VisualEditor: (no message) (duration: 00m 06s)
• 23:05 logmsgbot: ori Synchronized php-1.25wmf18/extensions/WikimediaEvents: (no message) (duration: 00m 07s)
• 23:05 logmsgbot: ori Synchronized php-1.25wmf18/extensions/VisualEditor: (no message) (duration: 00m 06s)
• 21:57 logmsgbot: hoo Synchronized php-1.25wmf18/extensions/Wikidata/: Update Wikibase to fix langlink updates in the client API et al (duration: 00m 12s)
• 21:57 logmsgbot: hoo Synchronized php-1.25wmf17/extensions/Wikidata/: Update Wikibase to fix langlink updates in the client API et al (duration: 00m 14s)
• 21:28 gwicke: restbase now up on all live (3 of 6) prod nodes
• 21:20 gwicke: cleanly re-initialized prod cassandra cluster after puppet run; picked up local dc from property file
• 20:49 chasemp: restart ntp on mw1009
• 20:15 mutante: readding mw1062 to puppet, signing new cert and salt-key
• 19:54 mutante: reinstalling mw1062 after disk has been replaced
• 19:28 mutante: ran puppet on ruthenium
• 17:46 logmsgbot: demon Synchronized php-1.25wmf17/extensions/DoubleWiki/DoubleWiki_body.php: shut up warnings finally (duration: 00m 05s)
• 17:42 logmsgbot: oblivian Synchronized wmf-config/session.php: mc1011-12 IP change (duration: 00m 05s)
• 17:35 logmsgbot: kartik Finished scap: ContentTranslation update (duration: 25m 31s)
• 17:25 ^d: jenkins stuck communicating to beta, restarting
• 17:10 logmsgbot: kartik Started scap: ContentTranslation update
• 16:53 _joe_: shutting down mc1009 and mc1010
• 16:39 logmsgbot: oblivian Synchronized wmf-config/session.php: mc1009-10 IP change (duration: 00m 05s)
• 16:23 kart_: Updated cxserver to 395be27
• 16:16 kart_: started cxserver deployment
• 16:01 logmsgbot: demon Synchronized wmf-config/CommonSettings-labs.php: (no message) (duration: 00m 07s)
• 15:49 logmsgbot: oblivian Synchronized wmf-config/session.php: mc1007-8 IP change (duration: 00m 06s)
• 15:21 _joe_: restarting hhvm on mw1103, mw1078,mw1032 - TC full as well.
• 15:17 _joe_: restarting hhvm on mw1027, TC full
• 15:11 _joe_: powering down mc1007,mc1008
• 09:58 AaronS: Deleted more bogus GlobalUserPage purge job queues
• 06:11 springle: bacula-director restart to pick up m1-master CNAME
• 06:07 springle: ran RT update-rt-siteconfig + apache restart to pick up m1-master CNAME
• 05:55 springle: etherpad-lite restart to pick up m1-master CNAME
• 05:49 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Feb 19 05:48:40 UTC 2015 (duration 48m 39s)
• 04:32 ori: txstatsd on graphite1001: disabled profiling and returned service to normal state
• 04:23 ori: restarting txstatsd on graphite1001 with --profile; will disable profiling in a few minutes.
• 02:41 mutante: restarted hhvm on mw1141 (locked up, T89912?)
• 02:36 logmsgbot: LocalisationUpdate completed (1.25wmf18) at 2015-02-19 02:35:41+00:00
• 02:35 logmsgbot: l10nupdate Synchronized php-1.25wmf18/cache/l10n: (no message) (duration: 00m 01s)
• 02:21 logmsgbot: LocalisationUpdate completed (1.25wmf17) at 2015-02-19 02:20:01+00:00
• 02:19 logmsgbot: l10nupdate Synchronized php-1.25wmf17/cache/l10n: (no message) (duration: 00m 01s)
• 02:18 mutante: restbase1004 - starting restbase service, running puppet
• 02:11 mutante: restbase1004/1005 systemctl daemon-reload to run systemd-sysv-generator to make it create missing unit for restbase and unbreak puppet running the service
• 01:12 logmsgbot: ejegg Synchronized wmf-config/CommonSettings.php: Use URLs without mobile redirects for CentralNotice (duration: 00m 07s)
• 00:54 logmsgbot: demon Finished scap: global user page extension-list fix + l10n rebuild (duration: 15m 21s)
• 00:39 AaronS: Deleted labswiki redis jobs (labswiki uses the db queue) for GlobalUserPage and flushed the queue aggregator
• 00:38 logmsgbot: demon Started scap: global user page extension-list fix + l10n rebuild
• 00:33 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 06s)
• 00:26 logmsgbot: demon Synchronized php-1.25wmf17/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.Target.js: (no message) (duration: 00m 06s)
• 00:16 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: RL image debug logs (duration: 00m 07s)
• 00:14 logmsgbot: demon Synchronized php-1.25wmf17/includes/skins/SkinTemplate.php: (no message) (duration: 00m 05s)
• 00:14 logmsgbot: demon Synchronized php-1.25wmf17/includes/skins/Skin.php: (no message) (duration: 00m 05s)
• 00:12 logmsgbot: demon Synchronized php-1.25wmf18/includes/resourceloader/ResourceLoaderImage.php: fix up svg handling in RL (duration: 00m 07s)
• 00:12 logmsgbot: demon Synchronized php-1.25wmf17/includes/resourceloader/ResourceLoaderImage.php: fix up svg handling in RL (duration: 00m 07s)
• 00:09 logmsgbot: demon Synchronized wmf-config/InitialiseSettings-labs.php: no-op, for completeness (duration: 00m 05s)
• 00:04 logmsgbot: aaron Synchronized php-1.25wmf16/includes/db/LoadBalancer.php: 9dc01855bca9ba322f6cb15092b29c654d74cecc (duration: 00m 05s)

February 18

• 23:51 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: rm no-op profile calls (duration: 00m 06s)
• 23:48 logmsgbot: aaron Synchronized php-1.25wmf17/includes/db/LoadBalancer.php: 42a56404328547a0b8bd07f001b1c4dff67b3498 (duration: 00m 05s)
• 23:46 logmsgbot: legoktm Synchronized wmf-config: Enable GlobalUserPage extension on all public, CentralAuth wikis (duration: 00m 05s)
• 23:39 twentyafterfour: fixed symlinks. uploaded release notes. deployment finished 1.5 hours behind schedule
• 23:25 logmsgbot: twentyafterfour Purged l10n cache for 1.25wmf16
• 23:23 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf18
• 23:23 ori: HHVM on mw1141 locked up (threads stuck in __lll_lock_wait). Depooling for further investigation.
• 23:17 logmsgbot: twentyafterfour Finished scap: testwiki to php-1.25wmf18 and rebuild l10n cache (duration: 42m 58s)
• 22:34 logmsgbot: twentyafterfour Started scap: testwiki to php-1.25wmf18 and rebuild l10n cache
• 22:30 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: rollback group0 to 1.25wmf17
• 22:28 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf18
• 22:24 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.25wmf17
• 21:41 subbu: deployed parsoid version 17f68256
• 19:24 bd808|LUNCH: pruned stale members from trebuchet minions set for scap/scap: redis-cli srem "deploy:scap/scap:minions" fenari.wikimedia.org virt0.wikimedia.org nickel.wikimedia.org searchidx1001.eqiad.wmnet
• 19:01 godog: restart txstatsd on graphite1001 to flush old metrics
• 18:40 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Id215ff962: Change $wgUDPProfilerPort to 8135. (duration: 00m 05s)
• 18:14 logmsgbot: oblivian Synchronized wmf-config/session.php: mc1013 IP change (duration: 00m 05s)
• 18:11 logmsgbot: oblivian Synchronized wmf-config/session.php: mc1013 IP change (duration: 00m 07s)
• 17:58 _joe_: fixed scap on mw1158, moving /srv/deployment/scap away made puppet perform the redeploy
• 17:56 _joe_: fixed scap on mw1154, moving /srv/deployment/scap away made puppet perform the redeploy
• 17:51 bd808: fixing scap on mw1158 and mw1154 will take a root to fix bad trebuchet git clones -- cd /src/deployment/scap; sudo mv scap scap-broken; sudo salt-call deploy.fetch 'scap/scap'; sudo salt-call deploy.checkout 'scap/scap'
• 17:22 _joe_: shutting down mc1014, moving to a different rack
• 17:19 _joe_: mw1158 and mw1154 report broken python imports during scap
• 17:18 logmsgbot: oblivian Synchronized wmf-config/session.php: mc1014 IP change (duration: 00m 07s)
• 17:18 logmsgbot: oblivian Synchronized wmf-config/session.php: mc1014 IP change (duration: 00m 07s)
• 16:56 logmsgbot: demon Synchronized README: testing scap update (duration: 00m 07s)
• 16:50 _joe_: moving mc1014 to a new row
• 16:44 logmsgbot: oblivian Synchronized wmf-config/session.php: mc1015 IP change (duration: 00m 05s)
• 16:16 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: translationadmin for sysops on mw.org (duration: 00m 08s)
• 16:03 logmsgbot: demon Synchronized wmf-config/CommonSettings-labs.php: for completeness, no-op (duration: 00m 07s)
• 15:56 logmsgbot: oblivian Synchronized wmf-config/session.php: mc1016 IP change (duration: 00m 07s)
• 15:25 chasemp: phabricator updated for T86772
• 15:11 _joe_: shutting down mc1016 for movement to a new row
• 14:34 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: remove dl() of php_utfnormal (duration: 00m 07s)
• 10:33 hoo: Manually switched wikidatawiki's sites table entry for ruwiki from protocol relative to https URIs
• 06:59 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1054 T89801 (duration: 00m 06s)
• 05:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Feb 18 05:06:53 UTC 2015 (duration 6m 52s)
• 02:39 logmsgbot: LocalisationUpdate completed (1.25wmf17) at 2015-02-18 02:38:47+00:00
• 02:38 logmsgbot: l10nupdate Synchronized php-1.25wmf17/cache/l10n: (no message) (duration: 00m 01s)
• 02:23 logmsgbot: LocalisationUpdate completed (1.25wmf16) at 2015-02-18 02:22:02+00:00
• 02:21 logmsgbot: l10nupdate Synchronized php-1.25wmf16/cache/l10n: (no message) (duration: 00m 02s)
• 02:07 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 05s)
• 01:58 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: reduce db1065 load (duration: 00m 05s)
• 00:56 logmsgbot: maxsem Synchronized php-1.25wmf17/extensions/MobileFrontend/: SWAT (duration: 00m 06s)
• 00:44 logmsgbot: maxsem Synchronized wmf-config/: https://gerrit.wikimedia.org/r/189863 - labs only (duration: 00m 06s)
• 00:27 logmsgbot: maxsem Synchronized php-1.25wmf16/extensions/WikiGrok/: https://gerrit.wikimedia.org/r/190562 (duration: 00m 06s)
• 00:26 logmsgbot: maxsem Synchronized php-1.25wmf17/extensions/WikiGrok/: https://gerrit.wikimedia.org/r/190562 (duration: 00m 07s)
• 00:22 logmsgbot: maxsem Synchronized wmf-config/mobile.php: https://gerrit.wikimedia.org/r/188731 (duration: 00m 05s)
• 00:21 springle: db1043 m3-master restart mysqld T89274
• 00:16 logmsgbot: maxsem Synchronized wmf-config/mobile.php: https://gerrit.wikimedia.org/r/187823 (duration: 00m 06s)
• 00:13 springle: db1048 m3-slave restart mysqld T89274
• 00:11 logmsgbot: maxsem Synchronized php-1.25wmf16/extensions/Echo/: SWAT (duration: 00m 06s)
• 00:05 logmsgbot: maxsem Synchronized php-1.25wmf17/extensions/Echo/: SWAT (duration: 00m 07s)

February 17

• 23:51 mutante: apt-get upgrade on gallium
• 22:37 csteipp: deploy fixes for T85850, T88310, T85855
• 22:13 ejegg: updated payments-wiki-staging from ce73ed11de9775a596c51acdc036503751961bc8 to cbaf66e7705789f37117ec6edc4d936c6174d511
• 21:42 hoo: Set email for dewiki account "Ar-ras" to the email of the commons account with the same name
• 20:41 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group1 to $VERSION
• 19:57 logmsgbot: ori Synchronized wmf-config/StartProfiler.php: I6fbd48e6b: Revert "Revert "Revert "Use ProfilerSectionOnly to handle DB/filebackend entries and the like""" (duration: 00m 05s)
• 19:15 logmsgbot: yurik scap failed: OSError [Errno 2] No such file or directory: '/var/lock/scap' (duration: 33m 42s)
• 18:52 andrewbogott: cold-migrating all instances from virt1005 to virt1012
• 18:41 logmsgbot: yurik Started scap: (no message)
• 18:23 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Ie5879ec6a: Set $wgUDPProfilerPort to 8125 (duration: 00m 07s)
• 18:21 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Icd6766440: Correct StatsFormatString so it emits valid statsd data (duration: 00m 07s)
• 18:02 andrewbogott: adding virt1012 to the nova virt pool
• 17:25 andrewbogott: powering down virt1005, waiting a few seconds, power on
• 17:14 logmsgbot: marktraceur Synchronized php-1.25wmf17/extensions/SecurePoll/includes/ballots/Ballot.php: [SWAT] [wmf17] Backport SecurePoll_BallotStatus fix (duration: 00m 05s)
• 17:05 logmsgbot: marktraceur Synchronized php-1.25wmf17/tests/phpunit/includes/StatusTest.php: [SWAT] [wmf17] Make sure Commons file deletion is still working later today (duration: 00m 06s)
• 17:04 logmsgbot: marktraceur Synchronized php-1.25wmf17/includes/Status.php: [SWAT] [wmf17] Make sure Commons file deletion is still working later today (duration: 00m 06s)
• 16:55 logmsgbot: marktraceur Synchronized php-1.25wmf17/includes/filerepo/FileRepo.php: [SWAT] [wmf17] Make sure Commons uploading is still working later today (duration: 00m 05s)
• 16:52 _joe_: upgrading testwiki to use www-data, may cause a brief downtime
• 16:46 logmsgbot: marktraceur Synchronized php-1.25wmf17/extensions/MultimediaViewer/resources/mmv/ui/: [SWAT] [wmf17] Media Viewer share/embed fix (duration: 00m 07s)
• 16:45 logmsgbot: marktraceur Synchronized php-1.25wmf16/extensions/MultimediaViewer/resources/mmv/ui/: [SWAT] [wmf16] Media Viewer share/embed fix (duration: 00m 05s)
• 16:35 logmsgbot: marktraceur Synchronized wmf-config/: [SWAT] [config] Set = true; on all wikis (duration: 00m 06s)
• 16:34 bd808: mw1062.eqiad.wmnet not accepting ssh login by bd808 (key refused)
• 16:33 logmsgbot: marktraceur Synchronized wmf-config/Wikibase.php: [SWAT] [config] Adjust , update property id blacklist (duration: 00m 05s)
• 16:31 bd808: mw1159.eqiad.wmnet has ancient scap version (2014-10-09)
• 16:27 logmsgbot: marktraceur Synchronized wmf-config/CommonSettings.php: [SWAT] [config] Update wgContentTranslationSiteTemplates (duration: 00m 05s)
• 16:19 logmsgbot: marktraceur Synchronized wmf-config/InitialiseSettings.php: [SWAT] [config] Enable Main namespace publishing for idwiki, ptwiki (duration: 00m 06s)
• 16:16 logmsgbot: marktraceur Synchronized wmf-config/logging.php: No-op test for bd808 (duration: 00m 05s)
• 16:13 bd808: updated scap to 54a2713 (www-data user)
• 16:12 bd808: hosts failing to fetch for scap trebuchet deploy: fenari.wikimedia.org, mw1062.eqiad.wmnet, nickel.wikimedia.org, searchidx1001.eqiad.wmnet, mw1222.eqiad.wmnet, virt0.wikimedia.org, mw1159.eqiad.wmnet
• 15:13 _joe_: disabled manually all crons in the 'apache' crontab on terbium
• 15:06 godog: restart pybal on lvs1003
• 15:00 godog: restart pybal on lvs1006
• 14:02 _joe_: updating the jobrunners to use www-data
• 11:24 _joe_: rolling transition of api appservers to www-data beginning as well
• 10:24 _joe_: converting all appservers to www-data
• 10:05 godog: testing cassandra-metrics on xenon
• 09:39 _joe_: repooling mw1029-1039
• 09:24 _joe_: depooling mw1029-1039
• 09:18 _joe_: repooling mw1019-28
• 09:06 godog: rolling restart of elastic1023 -> elastic1031
• 08:32 _joe_: depooling mw1019-28 for T78076
• 05:16 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Feb 17 05:15:45 UTC 2015 (duration 15m 44s)
• 03:56 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1065, warm up (duration: 00m 05s)
• 02:31 logmsgbot: LocalisationUpdate completed (1.25wmf17) at 2015-02-17 02:30:14+00:00
• 02:30 logmsgbot: l10nupdate Synchronized php-1.25wmf17/cache/l10n: (no message) (duration: 00m 01s)
• 02:21 logmsgbot: ori Synchronized wmf-config: Revert Ie91add33f: Temporarily log message key lookups on four app servers (duration: 00m 05s)
• 02:17 logmsgbot: LocalisationUpdate completed (1.25wmf16) at 2015-02-17 02:16:02+00:00
• 02:16 logmsgbot: l10nupdate Synchronized php-1.25wmf16/cache/l10n: (no message) (duration: 00m 03s)
• 01:51 logmsgbot: ori Synchronized wmf-config: Ie91add33f: Temporarily log message key lookups on four app servers (duration: 00m 05s)

February 16

• 22:34 logmsgbot: catrope Synchronized php-1.25wmf16/includes/MediaWiki.php: I34028206 (duration: 00m 05s)
• 22:33 ori: reloaded nginx on dumps with original config; re-enabled puppet.
• 22:04 ori: reloading nginx on dataset1001 for same
• 22:02 ori: disabled puppet on dataset1001 to experiment w/ https://gerrit.wikimedia.org/r/190940
• 21:50 logmsgbot: catrope Synchronized php-1.25wmf17/includes/MediaWiki.php: I34028206 (duration: 00m 06s)
• 21:12 logmsgbot: reedy Synchronized database lists: Update size related dblists (duration: 00m 06s)
• 21:09 subbu: updated Parsoid to version 86e76a30
• 16:13 logmsgbot: bd808 Synchronized wmf-config/logging-labs.php: Switch beta to syslog logging, try #2 (45d25e2) (duration: 00m 05s)
• 15:54 hoo: Updated Wikidata property suggester with data from today's dump
• 14:54 ottomata: shutting down hadoop cluster, starting upgrade to CDH 5.3.1
• 12:35 akosiaris: GIT_SSH=../../ssh git pull to update labs/private on deployment-salt
• 10:03 godog: resume elasticsearch rolling restart - elastic1012 -> elastic1022 in turn
• 08:02 _joe_: repooling mw1018
• 06:34 springle: messing with phab boolean fulltext syntax T89274
• 04:51 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Feb 16 04:50:10 UTC 2015 (duration 50m 9s)
• 04:20 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1065 (duration: 00m 06s)
• 02:29 logmsgbot: LocalisationUpdate completed (1.25wmf17) at 2015-02-16 02:28:39+00:00
• 02:28 logmsgbot: l10nupdate Synchronized php-1.25wmf17/cache/l10n: (no message) (duration: 00m 01s)
• 02:17 springle: db1046 restart, table maintenance
• 02:15 logmsgbot: LocalisationUpdate completed (1.25wmf16) at 2015-02-16 02:14:29+00:00
• 02:14 logmsgbot: l10nupdate Synchronized php-1.25wmf16/cache/l10n: (no message) (duration: 00m 02s)

February 15

• 02:13 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Feb 15 02:12:53 UTC 2015 (duration 12m 52s)
• 02:04 logmsgbot: LocalisationUpdate completed (1.25wmf17) at 2015-02-15 02:03:44+00:00
• 02:03 logmsgbot: l10nupdate Synchronized php-1.25wmf17/cache/l10n: (no message) (duration: 00m 01s)
• 02:03 logmsgbot: LocalisationUpdate completed (1.25wmf16) at 2015-02-15 02:02:13+00:00
• 02:02 logmsgbot: l10nupdate Synchronized php-1.25wmf16/cache/l10n: (no message) (duration: 00m 02s)

February 14

• 04:54 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Feb 14 04:53:02 UTC 2015 (duration 53m 1s)
• 02:32 logmsgbot: LocalisationUpdate completed (1.25wmf17) at 2015-02-14 02:31:02+00:00
• 02:30 logmsgbot: l10nupdate Synchronized php-1.25wmf17/cache/l10n: (no message) (duration: 00m 01s)
• 02:17 logmsgbot: LocalisationUpdate completed (1.25wmf16) at 2015-02-14 02:16:47+00:00
• 02:16 logmsgbot: l10nupdate Synchronized php-1.25wmf16/cache/l10n: (no message) (duration: 00m 02s)
• 00:47 logmsgbot: awight Synchronized php-1.25wmf17/extensions/CentralNotice: CentralNotice debug logging for T89258 (duration: 00m 06s)
• 00:47 logmsgbot: awight Synchronized php-1.25wmf16/extensions/CentralNotice: CentralNotice debug logging for T89258 (duration: 00m 06s)
• 00:06 logmsgbot: bd808 Synchronized wmf-config/logging-labs.php: Revert: Switch beta to syslog logging (d9dcccb) (duration: 00m 06s)

February 13

• 23:59 logmsgbot: awight Synchronized php-1.25wmf17/extensions/CentralNotice: CentralNotice debug logging for T89258 (duration: 00m 05s)
• 23:58 logmsgbot: bd808 Synchronized wmf-config/logging-labs.php: Switch beta to syslog logging (d9dcccb) (duration: 00m 06s)
• 23:56 logmsgbot: awight Synchronized php-1.25wmf16/extensions/CentralNotice: CentralNotice debug logging for T89258 (duration: 00m 06s)
• 23:26 logmsgbot: legoktm Synchronized php-1.25wmf16/extensions/CentralAuth/includes/CentralAuthUser.php: https://gerrit.wikimedia.org/r/#/c/190579/ (duration: 00m 06s)
• 23:25 logmsgbot: legoktm Synchronized php-1.25wmf17/extensions/CentralAuth/includes/CentralAuthUser.php: https://gerrit.wikimedia.org/r/#/c/190579/ (duration: 00m 06s)
• 23:21 logmsgbot: awight Synchronized php-1.25wmf17/extensions/CentralNotice: CentralNotice debug logging for T89258 (duration: 00m 08s)
• 23:20 logmsgbot: awight Synchronized php-1.25wmf16/extensions/CentralNotice: CentralNotice debug logging for T89258 (duration: 00m 07s)
• 23:15 logmsgbot: awight Synchronized php-1.25wmf17/extensions/CentralNotice: CentralNotice debug logging for T89258 (duration: 00m 05s)
• 23:06 logmsgbot: awight Synchronized wmf-config: Set up a new debug logging group for T89258 (take 2) (duration: 00m 06s)
• 22:56 logmsgbot: awight Synchronized wmf-config: Set up a new debug logging group for T89258 (duration: 00m 06s)
• 22:53 logmsgbot: awight Synchronized php-1.25wmf17/extensions/CentralNotice: CentralNotice fixes for T89258 and T45250 (duration: 00m 06s)
• 22:52 logmsgbot: awight Synchronized php-1.25wmf16/extensions/CentralNotice: CentralNotice fixes for T89258 and T45250 (duration: 00m 07s)
• 21:38 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: debug time over (duration: 00m 05s)
• 21:37 logmsgbot: demon Synchronized php-1.25wmf16/includes/resourceloader/ResourceLoaderImage.php: debug time over (duration: 00m 05s)
• 21:24 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: Debug fun (duration: 00m 05s)
• 21:22 logmsgbot: demon Synchronized php-1.25wmf16/includes/resourceloader/ResourceLoaderImage.php: Debug fun (duration: 00m 05s)
• 18:17 robh: morebots, you doing yer thing?
• 15:59 godog: es-tool restart-fast on elastic1011
• 15:08 godog: correction, elastic1010
• 15:08 godog: es-tool restart-fast on elastic1019
• 14:41 godog: es-tool restart-fast on elastic1009
• 13:25 hoo: Started rebuildItemsPerSite for wikidata on terbium
• 11:34 godog: restart elasticsearch on logstash1001 logstash1002 logstash1003
• 11:26 paravoid: mw1095/mw1192: service hhvm restart, alerts for 10h30/9h35 respectively
• 11:18 godog: es-tool restart-fast on elastic1008
• 04:56 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Feb 13 04:55:10 UTC 2015 (duration 55m 9s)
• 02:32 logmsgbot: LocalisationUpdate completed (1.25wmf17) at 2015-02-13 02:31:34+00:00
• 02:31 logmsgbot: l10nupdate Synchronized php-1.25wmf17/cache/l10n: (no message) (duration: 00m 01s)
• 02:19 ori: ran redis commands 'HDEL jobqueue:aggregator:h-queue-types:v2 LocalGlobalUserPageCacheUpdateJob/labswiki' and 'HDEL jobqueue:aggregator:h-queue-types:v2 LocalGlobalUserPageCacheUpdateJob' on rdb1001
• 02:18 logmsgbot: LocalisationUpdate completed (1.25wmf16) at 2015-02-13 02:17:27+00:00
• 02:17 logmsgbot: l10nupdate Synchronized php-1.25wmf16/cache/l10n: (no message) (duration: 00m 01s)
• 02:04 logmsgbot: legoktm Synchronized wmf-config/CommonSettings.php: Set ['LocalGlobalUserPageCacheUpdateJob'] = 'NullJob' to clear queues (duration: 00m 06s)
• 01:52 logmsgbot: legoktm Synchronized php-1.25wmf17/extensions/GlobalUserPage: Revert GlobalUserPage updates (duration: 00m 06s)
• 01:43 logmsgbot: maxsem Synchronized wmf-config/: Let there be mobile on wikitech (duration: 00m 06s)
• 01:36 ori: Correcting path reference in private/PrivateSettings.php required restarting HHVM on job runners. StatCache bug?
• 01:32 ori: restarting jobrunners
• 01:30 logmsgbot: ori Synchronized private/PrivateSettings.php: Correct path reference, for real this time (duration: 00m 07s)
• 01:26 logmsgbot: ori Synchronized private/PrivateSettings.php: Correct path reference (duration: 00m 06s)
• 01:01 logmsgbot: maxsem Synchronized wmf-config/InitialiseSettings.php: Shutting the warning off (duration: 00m 06s)
• 00:35 logmsgbot: maxsem Synchronized php-1.25wmf16/extensions/Wikidata/: SWAT (duration: 00m 12s)
• 00:34 logmsgbot: maxsem Synchronized php-1.25wmf17/extensions/GlobalUserPage/: fix SWAT (duration: 00m 06s)
• 00:31 logmsgbot: maxsem Synchronized php-1.25wmf17/extensions/Wikidata/: touch (duration: 00m 18s)
• 00:29 mutante: phab service restart for config change
• 00:29 logmsgbot: maxsem Synchronized php-1.25wmf17/extensions/Wikidata/: SWAT (duration: 00m 12s)
• 00:21 logmsgbot: maxsem Synchronized php-1.25wmf17/extensions/GlobalUserPage/: SWAT (duration: 00m 06s)

February 12

• 22:55 mutante: restarting phab for config change
• 20:05 mutante: moving servermon behind misc-web
• 19:18 logmsgbot: demon Synchronized php-1.25wmf16/extensions/CentralNotice/special/SpecialBannerRandom.php: rm live hack leftovers, now being worked on (duration: 00m 05s)
• 18:47 logmsgbot: demon Synchronized php-1.25wmf16/extensions/CentralNotice/special/SpecialBannerRandom.php: rm live hack, have our data (duration: 00m 06s)
• 18:44 logmsgbot: demon Synchronized php-1.25wmf16/extensions/CentralNotice/special/SpecialBannerRandom.php: live hack (duration: 00m 08s)
• 18:11 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: Remove random banner references (duration: 00m 05s)
• 17:34 logmsgbot: oblivian Synchronized wmf-config/session.php: Adding mc1018 to the sessions redis pool (duration: 00m 07s)
• 17:25 logmsgbot: oblivian Synchronized wmf-config/session.php: Adding mc1017 to the sessions redis pool (duration: 00m 05s)
• 17:13 _joe_: adding mc1018 to the nutcracker pool, this time without forcing a puppet run
• 16:57 logmsgbot: demon Synchronized php-1.25wmf16/extensions/CentralNotice/includes/BannerChooser.php: rm live hack for debugging (duration: 00m 05s)
• 16:54 logmsgbot: demon Synchronized php-1.25wmf16/extensions/CentralNotice/includes/BannerChooser.php: live hack for debugging (duration: 00m 06s)
• 16:44 _joe_: triggering a puppet run to insert mc1017 in the nutcracker pool
• 16:40 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/188887/ (duration: 00m 07s)
• 16:35 logmsgbot: krenair Synchronized wmf-config: trying that last sync again, I forgot to actually run the merge (duration: 00m 06s)
• 16:33 logmsgbot: krenair Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/190230/ (duration: 00m 07s)
• 16:26 logmsgbot: krenair Synchronized wmf-config: rv (duration: 00m 06s)
• 16:24 logmsgbot: krenair Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/187730/ (duration: 00m 05s)
• 16:11 godog: es-tool restart-fast on elastic1007
• 16:08 ottomata: re-enabling bits varnishkafka instances
• 16:01 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/190133/ (duration: 00m 07s)
• 14:15 Krenair: Manually logged a missing cross-wiki rights log change entry on meta "Avraham changed group membership for User:Bencmq@zhwiki from bureaucrat, check user and administrator to bureaucrat and administrator (requested)". See T89205 for details
• 11:28 godog: es-tool restart-fast on elastic1006
• 10:12 hashar: gallium and lanthanum: dpkg --purge locate
• 10:09 hashar: gallium: uninstalling locate package from gallium. Has been installed on 2015-01-30 00:31:39 apparently manually by root@iron.wikimedia.org
• 10:02 godog: es-tool fast-restart on elastic1005
• 08:28 hashar: puppet-lint now complains on error (not warnings) \O/ {{bug:T87132}}
• 04:54 springle: broke puppet db grant. fixed puppet db grant
• 04:53 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Feb 12 04:52:16 UTC 2015 (duration 52m 15s)
• 04:09 springle: sign puppet cert dbproxy1003, first run
• 03:02 andrewbogott: restarting wikitech-static. shinken works!
• 02:53 andrewbogott: breaking wikitech-static on purpose to test the shinken alert
• 02:42 logmsgbot: LocalisationUpdate completed (1.25wmf17) at 2015-02-12 02:41:01+00:00
• 02:40 logmsgbot: l10nupdate Synchronized php-1.25wmf17/cache/l10n: (no message) (duration: 00m 01s)
• 02:24 logmsgbot: LocalisationUpdate completed (1.25wmf16) at 2015-02-12 02:23:30+00:00
• 02:23 logmsgbot: l10nupdate Synchronized php-1.25wmf16/cache/l10n: (no message) (duration: 00m 01s)
• 00:33 logmsgbot: catrope Synchronized php-1.25wmf17/resources/lib/oojs-ui: SWAT (duration: 00m 08s)
• 00:15 logmsgbot: catrope Synchronized php-1.25wmf16/resources/lib/oojs-ui: SWAT (duration: 00m 06s)
• 00:13 logmsgbot: aaron Synchronized wmf-config/StartProfiler.php: Use ProfilerSectionOnly to handle DB/filebackend entries (duration: 00m 05s)

February 11

• 23:02 logmsgbot: twentyafterfour Purged l10n cache for 1.25wmf15
• 23:00 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf17
• 22:57 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.25wmf16
• 22:47 logmsgbot: twentyafterfour Finished scap: testwiki to php-1.25wmf17 and rebuild l10n cache (duration: 48m 29s)
• 22:45 mutante: apt-get upgrading zirconium
• 22:12 mutante: deactivated ocg1003 in pybal
• 22:05 andrewbogott: updated wikitech-static to wmf/1.25wmf15
• 21:58 logmsgbot: twentyafterfour Started scap: testwiki to php-1.25wmf17 and rebuild l10n cache
• 21:38 subbu: deployed parsoid version 4fc3b43d
• 20:09 bblack: eqiad-upload-https -> back to even weighting
• 19:46 bblack: all eqiad-upload-https -> cp1064
• 19:36 subbu: temporarily turn off logging to logstash till logstash isssues are resolved.
• 19:34 bblack: repooled cp1070 (eqiad bits) in pybal
• 19:18 bd808: restarted elasticsearch on logstash1002 after OOM
• 19:15 mutante: moved docroots on zirconium to new logical volume for /srv
• 19:09 godog: powerdown graphite1002 T88992
• 18:20 bd808: restarted Elasticsearch on logstash1003; preventative, other nodes restarted today
• 17:59 mutante: ran puppet on virt1000 - finished just fine, not sure why icinga said fail
• 17:54 mutante: running puppet on ruthenium (last was 2 days ago but also not admin disabled..)
• 17:49 godog: logging test
• 10:18 godog: restart elasticsearch on logstash1003, OOM
• 09:58 hashar: restarting Jenkins to upgrade the Credentials plugin
• 06:29 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1057 (duration: 00m 05s)
• 04:54 springle: restarting labsdb1002 https://lists.wikimedia.org/pipermail/labs-l/2015-February/003354.html
• 04:35 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Feb 11 04:34:30 UTC 2015 (duration 34m 29s)
• 03:21 springle: restarting labsdb1001 https://lists.wikimedia.org/pipermail/labs-l/2015-February/003354.html
• 02:48 hoo: Manually logged a missing global rights log change entry on meta "Ajraddatz changed global group membership for Benoit Rochon from (none) to OTRS-member with the following comment: request". See also T89205
• 02:27 logmsgbot: LocalisationUpdate completed (1.25wmf16) at 2015-02-11 02:26:55+00:00
• 02:26 logmsgbot: l10nupdate Synchronized php-1.25wmf16/cache/l10n: (no message) (duration: 00m 02s)
• 02:13 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-02-11 02:12:40+00:00
• 02:12 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 02s)
• 01:56 logmsgbot: krenair Synchronized php-1.25wmf16/includes/UserRightsProxy.php: https://gerrit.wikimedia.org/r/#/c/189879/ - same thing for interwiki user rights logs (duration: 00m 07s)
• 01:52 logmsgbot: krenair Synchronized php-1.25wmf16/extensions/CentralAuth/includes/CentralAuthGroupMembershipProxy.php: https://gerrit.wikimedia.org/r/#/c/189888/ - fix lack of global group membership change logging (duration: 00m 05s)
• 01:44 springle: puppet disabled on lanbdsb1001 labsdb1002. needs restart
• 00:24 logmsgbot: krenair Synchronized php-1.25wmf16/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/189867/ (duration: 00m 06s)
• 00:14 logmsgbot: krenair Synchronized php-1.25wmf16/extensions/UploadWizard/resources/mw.ApiUploadFormDataHandler.js: https://gerrit.wikimedia.org/r/#/c/189860/ (duration: 00m 05s)
• 00:03 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/188554/ (duration: 00m 07s)

February 10

• 23:38 logmsgbot: andyrussg Synchronized php-1.25wmf16/extensions/CentralNotice/: Update CentralNotice (duration: 00m 06s)
• 22:00 bblack: repooled cp1064 eqiad upload frontends in pybal
• 21:38 bblack: repooled cp1065 eqiad text frontend in pybal
• 21:16 bblack: rebooting cp1064 for experimental kernel (is depooled)
• 21:04 bblack: cp1065 frontend disabled in pybal temporarily
• 21:01 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: fixing upload path for labswiki (duration: 00m 06s)
• 20:51 bblack: cp1064 frontend disabled in pybal
• 20:28 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group1 to 1.25wmf16
• 20:20 mutante: restarting phd on iridium (phab) for config change
• 17:10 logmsgbot: demon Finished scap: No code changes, bringing silver in as deploy target (duration: 17m 31s)
• 16:53 logmsgbot: demon Started scap: No code changes, bringing silver in as deploy target
• 16:52 logmsgbot: demon Synchronized wmf-config/wikitech.php: Testing silver sync (duration: 00m 05s)
• 16:24 logmsgbot: anomie Synchronized wmf-config/CommonSettings.php: SWAT: Revert "Whitelist application/x-gzip on private wikis to fully allow dia files", wasn't a correct fix for the issue (duration: 00m 05s)
• 16:18 godog: temporarily disable puppet on carbon
• 16:15 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Quiz extension at cawikibooks gerrit:187913 (duration: 00m 07s)
• 16:13 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wmgUseFloatedToc to false at dewikivoyage gerrit:187408 (duration: 00m 06s)
• 16:10 godog: stop replication on elasticsearch cluster and restart ES on elastic1002
• 16:07 logmsgbot: anomie Synchronized wmf-config/CommonSettings.php: SWAT: Whitelist application/x-gzip on private wikis to fully allow dia files gerrit:188557 (duration: 00m 05s)
• 10:57 logmsgbot: hoo Synchronized wmf-config/InitialiseSettings-labs.php: (no message) (duration: 00m 06s)
• 10:43 godog: reimage graphite1002
• 07:10 _joe_: restarting HHVM on mw1128, in a deadlock in HPHP::RequestInjectionData::onSessionInit ()
• 07:09 _joe_: restarting HHVM on mw1139, in a deadlock in HPHP::StatCache::refresh ()
• 04:43 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Feb 10 04:42:17 UTC 2015 (duration 42m 16s)
• 02:54 andrewbogott: finished wikitech move to silver.
• 01:28 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/189634/ (duration: 00m 10s)
• 01:23 logmsgbot: krenair Synchronized php-1.25wmf16/resources/lib/oojs-ui: https://gerrit.wikimedia.org/r/#/c/189147/ (duration: 00m 08s)
• 01:21 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/189211/ per James_F in -dev (duration: 00m 06s)
• 01:11 logmsgbot: krenair Synchronized php-1.25wmf16/extensions/VisualEditor/modules/ve-mw: https://gerrit.wikimedia.org/r/#/c/189144/ (duration: 00m 05s)
• 01:00 logmsgbot: krenair Synchronized php-1.25wmf16/extensions/Echo/tests/phpunit/includes/DiscussionParserTest.php: https://gerrit.wikimedia.org/r/#/c/189638/ (duration: 00m 08s)
• 01:00 logmsgbot: krenair Synchronized php-1.25wmf16/extensions/Echo/includes/DiscussionParser.php: https://gerrit.wikimedia.org/r/#/c/189638/ (duration: 00m 05s)
• 00:46 logmsgbot: krenair Synchronized php-1.25wmf16/extensions/Echo/tests/phpunit/includes/DiscussionParserTest.php: https://gerrit.wikimedia.org/r/#/c/189549/ (duration: 00m 06s)
• 00:45 logmsgbot: krenair Synchronized php-1.25wmf16/extensions/Echo/includes/DiscussionParser.php: https://gerrit.wikimedia.org/r/#/c/189549/ (duration: 00m 06s)
• 00:33 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/181358/ (duration: 00m 05s)
• 00:33 logmsgbot: krenair Synchronized wmf-config/CommonSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/181358/ (duration: 00m 05s)
• 00:29 logmsgbot: yurik Finished scap: syncing ZeroBanner i18n (duration: 34m 06s)
• 00:13 bd808: restarted elasticsearch on logstash1001; OOM

February 9

• 23:55 logmsgbot: yurik Started scap: syncing ZeroBanner i18n
• 23:54 logmsgbot: yurik Synchronized php-1.25wmf15/extensions/ZeroBanner: cherry-picking 189617 (duration: 00m 05s)
• 23:54 logmsgbot: yurik Synchronized php-1.25wmf16/extensions/ZeroBanner: cherry-picking 189617 (duration: 00m 07s)
• 23:01 logmsgbot: yurik Synchronized php-1.25wmf16/extensions/ZeroBanner: cherry-picking 189553 (duration: 00m 06s)
• 23:00 logmsgbot: yurik Synchronized php-1.25wmf15/extensions/ZeroBanner: cherry-picking 189553 (duration: 00m 06s)
• 17:46 bd808: restarted elasticsearch on logstash1003; OOM
• 16:53 logmsgbot: marktraceur Synchronized php-1.25wmf15/extensions/UploadWizard/resources/mw.FlickrChecker.js: [SWAT] [wmf15] Re-add flickrreview template to files imported from Flickr by UploadWizard (duration: 00m 05s)
• 16:48 logmsgbot: marktraceur Synchronized php-1.25wmf16/extensions/UploadWizard/: [SWAT] [wmf16] Trying to force UploadWizard to update (duration: 00m 05s)
• 16:48 logmsgbot: marktraceur Synchronized php-1.25wmf15/extensions/UploadWizard/: [SWAT] [wmf15] Trying to force UploadWizard to update (duration: 00m 06s)
• 16:47 jgage: restarted eventlogging on hafnium (with deploy from master on tin this time)
• 16:38 logmsgbot: marktraceur Synchronized php-1.25wmf16/extensions/UploadWizard/resources/mw.FlickrChecker.js: [SWAT] [wmf16] Re-add flickrreview template to files imported from Flickr by UploadWizard (duration: 00m 06s)
• 16:37 logmsgbot: marktraceur Synchronized php-1.25wmf15/extensions/UploadWizard/resources/mw.FlickrChecker.js: [SWAT] [wmf15] Re-add flickrreview template to files imported from Flickr by UploadWizard (duration: 00m 05s)
• 16:22 jgage: restarted eventlogging on hafnium for nuria via ~root/upgrade-eventlogging --no-update
• 16:18 logmsgbot: marktraceur Synchronized wmf-config/throttle.php: [SWAT] [config] Add throttle rules for two workshops (duration: 00m 07s)
• 16:14 logmsgbot: marktraceur Synchronized wmf-config/: [SWAT] [config] Un-subscribe frequently failing recipients (duration: 00m 05s)
• 16:12 logmsgbot: marktraceur Synchronized php-1.25wmf16/extensions/OAuth/: [SWAT] [wmf16] OAuth: Support ListDefinedTags and ChangeTagsListActive hooks (duration: 00m 11s)
• 15:00 cmjohnson1: cp1070 down for h/w troubleshooting. Already depooled by bblack
• 11:58 godog: bounce mwprof-profiler-to-carbon on tungsten
• 10:47 hoo: Manually removed wikidatawiki.wb_changes_dispatch entries for test wikis (test2wiki, testwiki, testwikidata).
• 09:11 gwicke: cassandra load testing on xenon, praseodymium and cerium; disk space is tight, might run out on one of those boxes but they are purely test boxes right now, so np
• 05:32 gwicke: stopped puppet on cerium, praseodymium & xenon
• 05:31 gwicke: manually updated cassandra on cerium, praseodymium & xenon to 2.1.2 (see https://phabricator.wikimedia.org/T88956)
• 03:50 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Feb 9 03:48:57 UTC 2015 (duration 48m 56s)
• 02:13 logmsgbot: LocalisationUpdate completed (1.25wmf16) at 2015-02-09 02:12:45+00:00
• 02:12 logmsgbot: l10nupdate Synchronized php-1.25wmf16/cache/l10n: (no message) (duration: 00m 02s)
• 02:12 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-02-09 02:11:13+00:00
• 02:11 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s)

February 8

• 03:53 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Feb 8 03:52:49 UTC 2015 (duration 52m 48s)
• 02:14 logmsgbot: LocalisationUpdate completed (1.25wmf16) at 2015-02-08 02:13:11+00:00
• 02:13 logmsgbot: l10nupdate Synchronized php-1.25wmf16/cache/l10n: (no message) (duration: 00m 01s)
• 02:12 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-02-08 02:11:41+00:00
• 02:11 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s)

February 7

• 15:35 apergos: started nginx on daaset1001, it was not running for some reason
• 09:40 bblack: depooled cp1070 in pybal
• 09:33 bblack: rebooting cp1070 (dead network, dead console)
• 05:10 subbu: deployed parsoid hotfiix 8ca7ef40 (cherry-pick of 447a0565)
• 04:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Feb 7 04:47:30 UTC 2015 (duration 47m 29s)
• 03:13 gwicke: restarting parsoid cluster
• 02:34 logmsgbot: LocalisationUpdate completed (1.25wmf16) at 2015-02-07 02:33:09+00:00
• 02:33 logmsgbot: l10nupdate Synchronized php-1.25wmf16/cache/l10n: (no message) (duration: 00m 01s)
• 02:20 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-02-07 02:19:09+00:00
• 02:19 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 02s)
• 02:11 qchris: Ran kafka leader re-election as analytics1021 dropped out of it's partition leader role.
• 01:48 bblack: leaving cp1064 (jessie upload eqiad) pooled front+back. it's experimental but looks stable. if upload-related 503 spikes and I'm not around, feel free to depool it.
• 00:18 qchris: Manually bumping heap for the Hadoop namenodes and revived them after both of them running out of heap and not coming back.

February 6

• 22:53 logmsgbot: marktraceur Synchronized wmf-config/: [friday] beta config change for tgr (duration: 00m 09s)
• 22:53 subbu: restarted parsoid service to kill several stuck processes on multiple nodes
• 20:05 robh: ms1004 coming offline, shouldnt page (but disregard if it does)
• 19:19 subbu: deployed parsoid hotfiix a9dbd4fc (cherry-pick of 76d6658c)
• 16:54 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: Adding cdm16062.contentdm.oclc.org to wgCopyUploadsDomains (duration: 00m 05s)
• 16:40 godog: cancel downtime on graphite1001, enable downtime on tungsten pending full decomission
• 16:05 godog: bounce ocg on ocg1001 and stop additional ocg instance running
• 15:50 bblack: depool -> repool cp1064 varnish-frontend, reduced cache size to 16G, re-enabled compact_memory
• 15:50 godog: restart ocg on ocg1003 to pick up statsd dns changes
• 15:48 godog: restart ocg on ocg1002 to pick up statsd dns changes
• 14:33 bblack: starting up a fresh round of SSL testing on eqiad upload pooling (cp1064)
• 14:12 godog: bounce diamond on lvs2004/lvs2005
• 13:55 cmjohnson1: upgrading boron to trusty
• 10:51 godog: reimage ms-be2014
• 07:50 _joe_: restarting the parsoid cluster, one node at a time, some processes are stuck.
• 02:04 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
• 01:01 ori: restarting xenon on fluorine
• 00:14 logmsgbot: krenair Synchronized php-1.25wmf15/includes/CategoryViewer.php: https://gerrit.wikimedia.org/r/#/c/188944/1 (duration: 00m 06s)
• 00:13 logmsgbot: krenair Synchronized php-1.25wmf16/includes/CategoryViewer.php: https://gerrit.wikimedia.org/r/#/c/188945/1 (duration: 00m 06s)

February 5

• 23:53 bd808: Updated Wikimania Scholarships to 0852585 (re-enable language selection) + local hack in trebuchet repo to remove incomplete translations
• 22:53 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: Enable Parsoid on wikitech (duration: 00m 05s)
• 21:08 logmsgbot: reedy Purged l10n cache for 1.25wmf13
• 21:03 logmsgbot: reedy Synchronized php-1.25wmf16/extensions/CheckUser/: (no message) (duration: 00m 07s)
• 20:59 logmsgbot: reedy Synchronized php-1.25wmf16: (no message) (duration: 00m 52s)
• 20:59 bblack: cp1064 upload b ackend re-enabled in cache.pp; if upload-related 503s ensue later today and I'm not around, feel free to re-disable it
• 20:57 mutante: radon - reinstalling, scheduled downtime
• 20:42 Reedy: mw1092 giving file has vanished: "/wmf-config/.InitialiseSettings.php.KSg3AF" (in common)
• 20:42 logmsgbot: reedy Synchronized wmf-config/: GlobalUserPage and I33a855cecfbe25003fe9e4f5e2fab2f928c79da4 (duration: 00m 07s)
• 20:41 logmsgbot: reedy Synchronized wmf-config/: GlobalUserPage and I33a855cecfbe25003fe9e4f5e2fab2f928c79da4 (duration: 00m 05s)
• 20:35 logmsgbot: reedy Synchronized wmf-config/: GlobalUserPage and I33a855cecfbe25003fe9e4f5e2fab2f928c79da4 (duration: 00m 08s)
• 20:34 logmsgbot: reedy Synchronized php-1.25wmf16/includes/EditPage.php: Id376f9e75c43c5bd0fa910b04d066e6aa37c73d1 (duration: 00m 07s)
• 20:30 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf16
• 20:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.25wmf15
• 20:21 logmsgbot: reedy Finished scap: testwiki to 1.25wmf16 (duration: 38m 37s)
• 20:08 _joe_: re-exported nfs exports on dataset1001, remounted /mnt/data on snapshot1001
• 19:42 logmsgbot: reedy Started scap: testwiki to 1.25wmf16
• 19:41 logmsgbot: reedy scap aborted: testwiki to 1.25wmf16 (duration: 03m 27s)
• 19:37 logmsgbot: reedy Started scap: testwiki to 1.25wmf16
• 19:09 legoktm: clearing bad sidebar memcache entries on commonswiki
• 18:08 _joe_: restarting nutcracker on jobrunners
• 18:06 _joe_: restarting nutcracker on api appservers
• 18:01 ori: restarting nutcracker on all appservers
• 17:54 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: I4f28205e6: Set $wmgUseMonologLogger to false (duration: 00m 06s)
• 17:47 logmsgbot: ori Synchronized wmf-config/logging.php: Live hack: disable Logstash logging on suspicion that it is acting up (duration: 00m 05s)
• 17:34 paravoid: restarting HHVM on all appservers/API appservers in 10%/6s batches
• 17:26 bblack: repooled cp1063 frontend-only
• 16:21 godog: bounce jmxtrans on analytics1018, analytics1021 and analytics1022
• 16:15 godog: bounce jmxtrans on analytics1012
• 16:03 godog: re-enabled puppet on graphite1001, bounce uwsgi
• 14:34 godog: upload txstatsd 1.0.0-3 to trusty-wikimedia
• 12:42 paravoid: cp*/amssq*: salt rm /etc/logrotate.d/varnishkafka-frontend-stats to fix cronspam
• 12:30 hashar: Upgrading Jenkins and restarting it
• 06:43 springle: upgrade silver to mariadb 10
• 04:55 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Feb 5 04:54:00 UTC 2015 (duration 53m 59s)
• 02:37 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Ia59e654e8: Set a statsd-compatible $wgStatsFormatString (duration: 00m 07s)
• 02:35 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-02-05 02:34:02+00:00
• 02:34 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s)
• 02:20 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-02-05 02:19:31+00:00
• 02:19 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 02s)
• 01:41 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I7b270eb8a: Set $wgUDPProfilerHost to service alias rather than hard-code IP (duration: 00m 05s)
• 00:54 bd808: truncated redis input queues for logstash on all 3 hosts to see if cluster can keep up now with 3 elasticsearch writer threads
• 00:08 Krinkle: Added 'dduvall' to integration group ACL on Gerrit
• 00:06 springle: xtrabackup clone virt1000 to silver

February 4

• 23:38 mutante: starting memcached on virt1000
• 23:21 qchris: Manual failover of Hadoop namenode from analytics1001 to analytics1002, as analytics1001 had Heap space errors
• 22:50 ejegg: updated payments from 1e9b78e9a8bf557a710988620bd6f1a335787173 to cbaf66e7705789f37117ec6edc4d936c6174d511
• 22:49 manybubbles: this is certainly a bug in Elasticsearch, but I imagine its one solved in newer versions. i hope, more like.
• 22:49 manybubbles: not sure what happened but now space if freeing up on 1001. the disk was never in danger of filling up but it was full enough not to allocate more to it. Now that stuff is allocating elsewhere elasticsearch is clearing the used space.
• 22:41 manybubbles: looks like elastics1001 doesn't have much free space left. I think that might have something to do with this....
• 22:38 manybubbles: Elasticsearch wasn't initializing shards to elastic1001 after its restart. Didn't check why. Set allocation to primaries then back to all and that unstuck it.
• 21:16 arlolra: updated Parsoid to version dd4721f4
• 20:33 logmsgbot: ori rebuilt wikiversions.cdb and synchronized wikiversions files: I4fb67945b: Revert "[Regression] Revert "Non wikipedias to 1.25wmf15"
• 20:18 logmsgbot: aude Synchronized wmf-config/Wikibase.php: set useLegacyChangesSubscription to true for Wikidata (duration: 00m 07s)
• 18:30 godog: bounce txstatsd on cache hosts in eqiad
• 18:17 godog: bounce txstatsd on cache hosts in ulsfo
• 18:08 godog: bounce txstatsd on cache hosts in esams
• 17:30 logmsgbot: marktraceur Synchronized php-1.25wmf14/extensions/UploadWizard/: Touching pretty much everything in UploadWizard, maybe it will help (duration: 00m 07s)
• 17:22 logmsgbot: marktraceur Synchronized php-1.25wmf14/extensions/UploadWizard/resources/mw.UploadWizard.js: Touch an UploadWizard file to try and fix caching (duration: 00m 07s)
• 16:58 robh: replacing the intermediary cert on dumps.w.o (so nginx will flap on it shortly)
• 16:56 godog: restart ES on elastic1001
• 15:43 logmsgbot: marktraceur Synchronized php-1.25wmf14/extensions/UploadWizard/resources/controller/uw.controller.Upload.js: Touch an UploadWizard file to try and fix caching (duration: 00m 07s)
• 15:25 logmsgbot: marktraceur Synchronized php-1.25wmf14/extensions/UploadWizard/resources/controller/uw.controller.Upload.js: Touch an UploadWizard file to try and fix caching (duration: 00m 05s)
• 15:22 godog: graphite move close to completion, updating dashboards
• 15:16 godog: bounce diamond in batches in eqiad
• 14:50 logmsgbot: marktraceur Synchronized php-1.25wmf15/extensions/UploadWizard/resources/controller/uw.controller.Upload.js: Touch an UploadWizard file to try and fix caching (duration: 00m 05s)
• 14:14 godog: bounce webperf-related services on hafnium too: ve, statsd-mw-js-deprecate, statsv, asset-check
• 14:10 godog: bounce navtiming on hafnium to pick up dns changes
• 12:42 godog: stop bacula-fd on tungsten, backups running during migration
• 12:41 _joe_: installing the new HHVM package on jobrunners
• 12:28 godog: bounce txstatsd on ms-fe*
• 12:28 godog: bounce txstatsd on ms-be*
• 12:00 godog: bounce diamond in batches in ulsfo
• 11:57 godog: bounce diamond in batches in esams
• 11:51 godog: bounce mwprof on tungsten to force picking up dns changes
• 11:35 _joe_: installing the new hhvm package on api, one at a time
• 11:23 godog: start migrating graphite from tungsten to graphite1001 https://gerrit.wikimedia.org/r/#/c/188036/1 https://gerrit.wikimedia.org/r/#/c/188035/1 https://phabricator.wikimedia.org/T85909
• 10:14 logmsgbot: ori Finished scap: I78446aacb: [Regression] Revert "Non wikipedias to 1.25wmf15" (duration: 31m 34s)
• 10:06 godog: start migrating graphite from tungsten to graphite1001 https://gerrit.wikimedia.org/r/#/c/188036/1 https://gerrit.wikimedia.org/r/#/c/188035/1 https://phabricator.wikimedia.org/T85909
• 10:06 ori: restarted hung HHVM on mw1039
• 09:42 logmsgbot: ori Started scap: I78446aacb: [Regression] Revert "Non wikipedias to 1.25wmf15"
• 09:42 logmsgbot: ori scap aborted: I78446aacb: [Regression] Revert "Non wikipedias to 1.25wmf15" (duration: 00m 02s)
• 09:42 logmsgbot: ori Started scap: I78446aacb: [Regression] Revert "Non wikipedias to 1.25wmf15"
• 08:57 _joe_: installing the new hhvm package on all appservers, one at a time
• 07:49 qchris: Manual failover of Hadoop namenode from analytics1002 to analytics1001, as analytics1002 had Heap space errors
• 05:16 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1057 (duration: 00m 06s)
• 04:49 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Feb 4 04:48:42 UTC 2015 (duration 48m 41s)
• 02:48 logmsgbot: tstarling Synchronized php-1.25wmf15/includes/specials/SpecialUserrights.php: Unbreak interwiki user rights granting (duration: 00m 05s)
• 02:30 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-02-04 02:29:17+00:00
• 02:29 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 02s)
• 02:16 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-02-04 02:15:35+00:00
• 02:15 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 03s)
• 01:47 bd808: restarted elasticsearch on logstash1001; rolling restart part 3 of 3
• 01:39 bd808: restarted elasticsearch on logstash1002; rolling restart of cluster part 2 of 3
• 00:38 robh: replacing dumps.w.o sha1 cert with sha256

February 3

• 22:54 bd808: restarted elasticsearch on logstash1003
• 22:53 bd808: starting rolling restart of logstash elasticsearch cluster to pick up index.merge.scheduler.max_thread_count puppet change
• 22:52 robh: magnesium apache reload for rt cert replacement
• 22:43 robh: replacing etherpad sha1 with sha256 cert
• 22:13 logmsgbot: reedy Synchronized php-1.25wmf15/extensions/WikimediaMaintenance: tmp script (duration: 00m 07s)
• 21:38 logmsgbot: maxsem Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/188300/ (duration: 00m 07s)
• 21:32 bblack: repooled cp10[67]0
• 21:29 mutante: restarted gitblit
• 21:17 andrewbogott: increased opendj lookthrough-limit to 12000 on both ldap hosts. We just hit lucky 5000 users and some queries stopped working.
• 20:59 bblack: depooling cp1060, cp1070 (1 each bits + mobile) for reinstall
• 20:59 robh: updating gerrit.wikimedia.org cert
• 19:30 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.25wmf15
• 18:17 YuviPanda: set up new images for ubuntu trusty / precise on labs, for https://phabricator.wikimedia.org/T87003
• 17:23 bblack: repooled cp1064
• 17:07 bd808: restarted elasticsearch on logstash1002; OOM
• 16:55 bblack: cp1064 (eqiad upload cache) depooled in pybal
• 15:39 _joe_: installing a new package to canary servers
• 14:55 _joe_: uploaded a new hhvm package version, deploying to testwiki and beta
• 08:50 logmsgbot: reedy Synchronized wmf-config/: Bye bye Solarium (duration: 00m 06s)
• 08:20 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: Ie7b32e3d8: Add log group for T87645 (duration: 00m 05s)
• 08:19 logmsgbot: ori Synchronized php-1.25wmf14/includes/EditPage.php: Id376f9e75: Hack for T87645, since maybe it is still happening (duration: 00m 07s)
• 08:17 logmsgbot: ori Synchronized php-1.25wmf15/includes/EditPage.php: Id376f9e75: Hack for T87645, since maybe it is still happening (duration: 00m 05s)
• 08:14 paravoid: radium: upgrade tor to the latest torproject.org version
• 08:10 springle: wikitech mysql restart to fix novaold errors
• 05:14 springle: wikitech virt1000 test db dump T88311
• 04:49 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Feb 3 04:48:38 UTC 2015 (duration 48m 37s)
• 02:49 ori: deployed https://gerrit.wikimedia.org/r/#/c/187304/ (php-set X-Analytics header) to both production branches.
• 02:42 bblack: cp1065 re-pooled in pybal
• 02:31 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-02-03 02:30:18+00:00
• 02:30 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 02s)
• 02:19 mutante: installing package upgrades on radium
• 02:17 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-02-03 02:16:37+00:00
• 02:16 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 03s)
• 01:59 bblack: depool cp1065 (text eqiad in pybal -> jessie)
• 01:59 bd808: Manually created apifeatureusage-2015.02.02 and apifeatureusage-2015.02.03 indices in elasticsearch; clsuter needs rolling restart for autocreate to work for these names
• 01:51 bd808: restarted logstash on logstash1001
• 01:51 bd808: restarted elasticsearch on logstash1003
• 00:03 mutante: rbf2002 - error while setting up RAID during installer (rbf2001 did not have this? or did it?)
• 00:02 mutante: rbf2001 - initial puppet run, adding users

February 2

• 23:59 mutante: signing puppet cert for rbf2001, PXE booting rbf2002
• 22:18 logmsgbot: ori Synchronized php-1.25wmf14/extensions/XAnalytics: (no message) (duration: 00m 05s)
• 22:18 logmsgbot: ori Synchronized php-1.25wmf15/extensions/XAnalytics: (no message) (duration: 00m 05s)
• 21:16 subbu: deployed parsoid version e3c9ae99
• 19:59 ejegg: updated payments from ce73ed11de9775a596c51acdc036503751961bc8 to 1e9b78e9a8bf557a710988620bd6f1a335787173
• 19:11 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable usage tracking on Wikidata (duration: 00m 07s)
• 18:15 mutante: restarted hhvm on mw1207
• 18:04 mutante: started nginx on ms1001, dataset1001
• 18:00 mutante: restarted gitblit
• 17:20 logmsgbot: marktraceur Synchronized php-1.25wmf15/extensions/UploadWizard/resources/mw.FormDataTransport.js: [SWAT] [wmf15] Fix UploadWizard for ogg files (duration: 00m 07s)
• 17:20 logmsgbot: marktraceur Synchronized php-1.25wmf14/extensions/UploadWizard/resources/mw.FormDataTransport.js: [SWAT] [wmf14] Fix UploadWizard for ogg files (duration: 00m 06s)
• 16:54 logmsgbot: anomie Synchronized wmf-config: SWAT: Added BounceHandler extension to group0 wikis gerrit:186242 (duration: 00m 07s)
• 16:52 yuvipanda: kill opendj on virt1000, it shouldn't have been running there in the first place
• 16:34 logmsgbot: anomie Synchronized wmf-config: SWAT: Have ContentTranslate publish article to Main namespace for cawiki gerrit:186358 (duration: 00m 07s)
• 16:11 aude: added and populated wbc_entity_usage table for wikidatawiki
• 10:54 logmsgbot: hoo Synchronized wmf-config/Wikibase.php: Exempt Item and Property namespaces from ConfirmEdit (duration: 00m 07s)
• 10:44 logmsgbot: hoo Synchronized php-1.25wmf14/extensions/Wikidata/: Update Wikibase: Fixes for UsageTracking and the anon edit warning (duration: 00m 14s)
• 10:43 logmsgbot: hoo Synchronized php-1.25wmf15/extensions/Wikidata/: Update Wikibase: Fixes for UsageTracking and the anon edit warning (duration: 00m 12s)
• 10:28 yuvipanda: been restarting pdns, opendj, apache, mysql, keystone left and right on virt1000 all day.
• 10:28 hashar: Synchronized wmf-config/InitialiseSettings.php: Add www.doria.fi to $wgCopyUploadsDomains bug T87104 https://gerrit.wikimedia.org/r/#/c/187914/ (duration: 00m 07s)
• 10:27 yuvipanda: foo
• 04:09 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Feb 2 04:08:14 UTC 2015 (duration 8m 13s)
• 02:44 springle: virt1000 mysqld restart, shrink buffer pool
• 02:20 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-02-02 02:19:21+00:00
• 02:19 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 02s)
• 02:11 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-02-02 02:09:59+00:00
• 02:09 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 04s)
• 00:13 subbu: restarted parsoid service on the parsoid cluster to free up leaked memory on several processes (seems to have happened in the 21:30 - 22:30 UTC on 31st Jan time frame)

February 1

• 04:12 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Feb 1 04:11:30 UTC 2015 (duration 11m 29s)
• 02:21 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-02-01 02:20:24+00:00
• 02:20 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s)
• 02:12 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-02-01 02:10:58+00:00
• 02:10 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 02s)

January 31

• 14:20 logmsgbot: hoo Synchronized wmf-config/CommonSettings-labs.php: (no message) (duration: 00m 06s)
• 04:12 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jan 31 04:11:31 UTC 2015 (duration 11m 29s)
• 02:25 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-01-31 02:24:27+00:00
• 02:24 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 02s)
• 02:12 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-31 02:10:58+00:00
• 02:10 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 02s)

January 30

• 22:38 subbu: deployed parsoid version 2abd0eb6
• 21:52 bblack: re-pooling cp3020 (bits cache esams) - reinstalled, looks sane...
• 21:38 YuviPanda|flight: killed diamond taking up 100% on labstore1001
• 21:04 logmsgbot: demon Synchronized docroot/noc/conf/logging.php.txt: (no message) (duration: 00m 06s)
• 21:02 logmsgbot: demon Synchronized docroot and w: (no message) (duration: 00m 10s)
• 20:58 bblack: reinstalling cp3020 (seems to have fs corruption issues, but may not be hardware...)
• 19:27 logmsgbot: phuedx Synchronized php-1.25wmf15/extensions/MobileFrontend/: No-op deployment training (duration: 00m 06s)
• 19:02 mutante: cp1039, cp1040 - shut down
• 18:56 bblack: rebooting cp3020 again (still depooled)
• 18:51 mutante: cp1037,cp1038 - shut down
• 18:40 godog: initial rsync from tungsten to graphite1001 T85909
• 18:36 cmjohnson1: removing cp1063 from pybal
• 18:34 logmsgbot: andyrussg Synchronized php-1.25wmf15/extensions/CentralNotice: Revert update to CentralNotice (duration: 00m 06s)
• 18:27 bblack: rebooting cp3020, something's all wrong there...
• 18:16 mutante: cp1037,cp1038,cp1039,cp1040 - disabled puppet, removed from icinga, revoked certs and salt key etc. decom
• 17:38 logmsgbot: andyrussg Synchronized php-1.25wmf15/extensions/CentralNotice: Update CentralNotice (duration: 00m 06s)
• 17:16 ^d: running puppet on ytterbium, gerrit shall restart
• 16:48 bblack: expect more icinga "CRITICAL: DPKG CRITICAL" on cache nodes for a while; applying backlog of upstream pkg updates slowly to all
• 16:18 bblack: restarting frontend varnishes to apply increased cache sizes from https://gerrit.wikimedia.org/r/#/c/186816/ over the next ~9H
• 10:04 paravoid: labstore1001: setting /proc/sys/sunrpc/{nfs,rpc}_debug to 0; rm /var/log/{kern.log,syslog.1,syslog}
• 04:21 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jan 30 04:20:12 UTC 2015 (duration 20m 11s)
• 02:25 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-01-30 02:23:56+00:00
• 02:24 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 03s)
• 02:20 awight: updated crm from a6b28d0a5e90f7ca68988a15f311465bbb5ae5e6 to ff28f69392fdab74520bf94e4a46586637914480
• 02:15 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-30 02:14:31+00:00
• 02:14 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 02s)
• 01:36 mutante: cp1047 - DIMM fail -> T88045
• 01:30 mutante: powercycling cp1047

January 29

• 23:59 awight: updated payments from 4743b32c9091d6f2169bea3173e75aa5d2f36eb7 to ce73ed11de9775a596c51acdc036503751961bc8
• 22:10 Krinkle: git-deploy: Deploying integration/slave-scripts I5aa76b0, I4d94af46735c, I66fbce3fa
• 19:49 bd808: Updated Wikimania Scholarships (2f4a99f) "Add Sakha to list of languages"
• 18:29 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: Disable Extension:Graph on cawiki (duration: 00m 06s)
• 04:53 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jan 29 04:51:55 UTC 2015 (duration 51m 54s)
• 02:35 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-01-29 02:34:03+00:00
• 02:34 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 02s)
• 02:21 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-29 02:20:02+00:00
• 02:20 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 02s)
• 01:12 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Id5186348f: Set $wgResourceLoaderStorageEnabled to false on osmium (duration: 00m 07s)
• 00:19 logmsgbot: demon Synchronized wmf-config/: (no message) (duration: 00m 06s)
• 00:18 logmsgbot: demon Synchronized php-1.25wmf14/includes/Title.php: (no message) (duration: 00m 06s)
• 00:17 logmsgbot: demon Synchronized php-1.25wmf15/includes/Title.php: (no message) (duration: 00m 06s)
• 00:17 logmsgbot: demon Synchronized php-1.25wmf15/includes/api/ApiPageSet.php: (no message) (duration: 00m 05s)
• 00:17 logmsgbot: demon Synchronized php-1.25wmf14/includes/api/ApiPageSet.php: (no message) (duration: 00m 06s)
• 00:11 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: flooders on officewiki (duration: 00m 08s)

January 28

• 22:55 mutante: restarting torrus on netmon1001
• 21:21 subbu: synced parsoid version 88605a4a for restart
• 21:20 subbu: synced parsoid version 88605a4a for restart
• 21:19 YuviPanda: restarting parsoid across wtp* hosts for subbu
• 21:00 YuviPanda: restarted hhvm on mw1069
• 20:40 logmsgbot: reedy Finished scap: mostly nooop, but adding graph to l10n cache (duration: 31m 33s)
• 20:09 logmsgbot: reedy Started scap: mostly nooop, but adding graph to l10n cache
• 20:03 logmsgbot: reedy Synchronized wmf-config/extension-list: Add Graph to extension-list (duration: 00m 07s)
• 20:03 mutante: shutdown remaining amslvs2-4
• 19:39 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Exempt osmium's script URL from bits rewriting (duration: 00m 05s)
• 19:36 mutante: shutting down amslvs1
• 19:19 logmsgbot: reedy Synchronized wmf-config/: Various config updates (duration: 00m 06s)
• 19:10 mutante: decom amslvs1-4, removing from puppet
• 18:20 ejegg|away: updated crm from 4fa10ec9e3afbf65e6cbd523138cdc4b4485c482 to a6b28d0a5e90f7ca68988a15f311465bbb5ae5e6
• 17:37 bd808: restarted elasticsearch on logstash1001; OOM
• 10:45 bblack: cp301[56] frontends repooled
• 07:07 bblack: repool amssq42 for text-https
• 04:52 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jan 28 04:51:31 UTC 2015 (duration 51m 30s)
• 02:35 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-01-28 02:34:42+00:00
• 02:34 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 02s)
• 02:21 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-28 02:20:45+00:00
• 02:20 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 02s)
• 02:07 mutante: started nodejs-ocg on ocg1001 (didnt listen on 8000 as opposed to ocg1002)
• 01:52 Tim: did hotfix on fluorine for incorrect udp2log conf file location

January 27

• 23:56 bblack: cp3016 out of service for now, needs reinstall (precise!)
• 23:05 mutante: rebooting terbium
• 22:52 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool pc1001 (duration: 00m 05s)
• 22:42 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool pc1001 (duration: 00m 11s)
• 20:58 andrewbogott: rebooting virt1002
• 20:40 andrewbogott: rebooting virt1001
• 20:32 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool pc1003 (duration: 00m 05s)
• 20:03 springle: reboot pc1003
• 19:58 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool pc1003 (duration: 00m 07s)
• 19:46 andrewbogott: rebooting virt1006
• 19:35 godog: (after the fact) reboot gadolinium, currently not coming back
• 19:23 mutante: brought ircd back up on argon
• 19:19 YuviPanda: run sysctl -w net.netfilter.nf_conntrack_max=131072 on labnet1001
• 19:19 YuviPanda: run sysctl -w net.netfilter.nf_conntrack_max=131072 on labmon1001
• 19:15 Krinkle: irc.wikimedia.org is down. "Connection refused."
• 19:14 Krenair: IRC RC seems broken
• 18:31 YuviPanda: rebooting tungstun
• 18:27 godog: reboot swift in esams
• 17:59 YuviPanda: rebooting labmon1001
• 17:49 godog: reboot all swift machines in eqiad, in turn
• 17:47 bblack: rebooting various LVSes...
• 17:23 marktraceur: I am consciously leaving NavigationTiming unsynced because nobody seems that concerned about it, and nobody is here to shepherd the patch. If you *are* concerned about it, then contact ori.
• 17:22 akosiaris: rebooting sca1001, sca1002, chromium, oxygen
• 17:21 logmsgbot: marktraceur Synchronized php-1.25wmf14/extensions/GWToolset/: [SWAT] [wmf14] GWToolset HHVM fixes (duration: 00m 06s)
• 17:11 logmsgbot: marktraceur Synchronized php-1.25wmf14/extensions/GWToolset/includes/: [SWAT] [wmf14] GWToolset HHVM fixes (duration: 00m 07s)
• 17:08 logmsgbot: marktraceur Synchronized wmf-config/InitialiseSettings.php: [SWAT] [config] Rename portal namespace in kowiki (duration: 00m 05s)
• 16:54 logmsgbot: marktraceur Synchronized php-1.25wmf15/extensions/GWToolset/includes/: [SWAT] [wmf15] GWToolset HHVM fixes (duration: 00m 06s)
• 04:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jan 27 04:08:16 UTC 2015 (duration 8m 15s)
• 02:19 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-01-27 02:19:11+00:00
• 02:19 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 03s)
• 02:11 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-27 02:10:49+00:00
• 02:10 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 02s)
• 00:58 logmsgbot: reedy Synchronized wmf-config/: Noop for citoid for beta (duration: 00m 05s)
• 00:50 logmsgbot: reedy Synchronized wmf-config/: Noop for bouncehandler for beta (duration: 00m 06s)

January 26

• 23:57 _joe_: depooling mw1018 for testing with user changing
• 20:00 akosiaris: restarted apache on palladium/strontium, cleared the, created on Jan 23, pid files from puppetmaster
• 19:59 _joe_: restarted apache on palladium
• 19:31 YuviPanda: restarted keystone on virt1000, ^d couldn’t log in
• 19:15 logmsgbot: reedy Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 06s)
• 04:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jan 26 04:08:01 UTC 2015 (duration 8m 0s)
• 02:23 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-01-26 02:23:10+00:00
• 02:23 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 03s)
• 02:15 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-26 02:14:49+00:00
• 02:14 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 02s)

January 25

• 20:50 ori: depooled mw1118 while investigating T85428
• 18:55 bd808: high rate of nutcracker "SYSTEM ERROR" errors on mw1118
• 18:43 bd808: trimmed Logstash redis input queues to 0 events; dropped ~4M backlogged events
• 04:14 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jan 25 04:14:33 UTC 2015 (duration 14m 32s)
• 02:23 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-01-25 02:22:58+00:00
• 02:22 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 02s)
• 02:10 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-25 02:10:30+00:00
• 02:10 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 02s)

January 24

• 22:40 bd808: Emptied logstash redis lists on all 3 hosts
• 22:27 bd808: Full restart of logstash elasticsearch cluster
• 06:27 paravoid: mass-restarting hhvm across the cluster
• 06:22 logmsgbot: faidon Synchronized wmf-config: touched config (duration: 00m 07s)
• 06:04 logmsgbot: faidon Synchronized wmf-config/StartProfiler.php: fix for T87497/r186578 (duration: 00m 06s)
• 04:52 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jan 24 04:52:42 UTC 2015 (duration 52m 41s)
• 02:30 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-01-24 02:30:10+00:00
• 02:30 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 02s)
• 02:17 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-24 02:17:08+00:00
• 02:17 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 02s)
• 01:05 hashar: restarting Jenkins (deadlock on deployment-bastion slave)
• 00:50 logmsgbot: reedy Synchronized wmf-config/: Kill wmgVectorBetaPersonalBar (duration: 00m 08s)
• 00:23 YuviPanda: begin re-imaging osmium

January 23

• 23:53 logmsgbot: reedy Synchronized wmf-config/: More config updates (duration: 00m 06s)
• 23:43 logmsgbot: reedy Synchronized wmf-config/: Config updates (duration: 00m 06s)
• 23:34 godog: halting lsearchd machines
• 23:23 _joe_: repooling 10 appservers depooled for no apparent reason, and no entry in the SAL
• 22:57 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I50931db37: Route all counter stats to tungsten:3811 (duration: 00m 07s)
• 22:57 Reedy: updateCollation.php for plwikisource complete
• 22:52 godog: restarting graphite on tungsten, webapp stuck and no graphs
• 22:49 Reedy: running mwscript updateCollation.php --wiki=plwikisource --previous-collation=uppercase in screen as reedy on tin
• 22:48 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: plwikisource collation update (duration: 00m 06s)
• 22:41 logmsgbot: reedy Synchronized 404.html: updates (duration: 00m 06s)
• 22:41 logmsgbot: reedy Synchronized docroot and w: Update docroot files (duration: 00m 05s)
• 22:38 logmsgbot: reedy Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 05s)
• 21:55 godog: removed puppet stored config for search* and searchidx*
• 21:34 _joe_: installing a new hhvm package on the canary appservers
• 21:21 _joe_: uploaded new hhvm package to apt.w.o
• 20:30 logmsgbot: rmoen Synchronized php-1.25wmf15/extensions/MobileFrontend/: updating mobilefrontend credits (duration: 00m 06s)
• 20:09 csteipp: deployed patch for T64685
• 19:44 ejegg: updated payments from 46bd1611c56d6e31594d01aa345c35dd45dcf676 to 4743b32c9091d6f2169bea3173e75aa5d2f36eb7
• 19:03 logmsgbot: ebernhardson Synchronized php-1.25wmf14/extensions/Flow: Bump flow submodule in 1.25wmf14 (duration: 00m 07s)
• 19:02 logmsgbot: ebernhardson Synchronized php-1.25wmf15/extensions/Flow/: Bump flow submodule in 1.25wmf15 (duration: 00m 08s)
• 17:59 mutante: apt-get clean, free some diskspace on antimony
• 17:17 bd808: logstash1002 elasticsearch rejoined cluster after restart
• 17:14 bd808: logstash elasticsearch cluster split brained; logstash1002 thinks it is a lone master
• 07:20 ori: restarted puppetmaster and apache2 on palladium.
• 04:03 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jan 23 04:03:45 UTC 2015 (duration 3m 44s)
• 02:19 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-01-23 02:19:45+00:00
• 02:19 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s)
• 02:11 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-23 02:11:43+00:00
• 02:11 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s)

January 22

• 23:27 legoktm: restarted the job runner service
• 19:37 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 05s)
• 18:39 ori: ran 'hdel jobqueue:aggregator:h-ready-queues:v2 LocalRenameUserJob/vewikimedia' on rdb1001
• 18:08 Krenair: Deployed patch for T87304
• 12:58 Krenair: Run https://phabricator.wikimedia.org/T87040#984282 again to stop GWToolset extension flooding commons log
• 04:15 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jan 22 04:15:02 UTC 2015 (duration 15m 1s)
• 02:19 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-01-22 02:19:19+00:00
• 02:19 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 04s)
• 02:11 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-22 02:11:09+00:00
• 02:11 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s)
• 01:25 springle: ongoing schema changes T86415 externallinks
• 01:12 akosiaris: applied base::firewall to all bastions hosts. This is bast*, iron, hooft
• 01:11 springle: dump db2019 to dbstore2001
• 01:09 springle: dump db2018 to dbstore2001
• 00:58 springle: dump db2017 to dbstore2001

January 21

• 23:07 springle: xtrabackup clone db1051 to db2016
• 20:59 _joe_: restarting hhvm on mw1192, stuck in a deadlock inside HPHP::f_ini_set ()
• 19:13 paravoid: restarting puppetmasters
• 04:39 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jan 21 04:39:39 UTC 2015 (duration 39m 38s)
• 02:39 logmsgbot: ori Synchronized README: Testing I68b5e1c2f (duration: 00m 07s)
• 02:38 logmsgbot: ori Synchronized README: Testing I68b5e1c2f (duration: 00m 02s)
• 02:28 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-01-21 02:28:50+00:00
• 02:28 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s)
• 02:16 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-21 02:16:38+00:00
• 02:16 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s)
• 02:08 logmsgbot: hoo Synchronized wmf-config/: Remove the Bug54847.php script - 2nd (duration: 00m 06s)
• 02:04 logmsgbot: hoo Synchronized wmf-config/: Remove the Bug54847.php script (duration: 00m 07s)
• 01:53 superm401: Re-ran GettingStarted populate_categories.php, also populating ptwiki for the first time.
• 01:43 logmsgbot: mattflaschen Finished scap: Turning off WikiGrok, enabling GettingStarted copyediting suggestions on ptwiki, and upgrading ContentTranslation (duration: 28m 26s)
• 01:15 logmsgbot: mattflaschen Started scap: Turning off WikiGrok, enabling GettingStarted copyediting suggestions on ptwiki, and upgrading ContentTranslation

January 20

• 23:28 logmsgbot: mattflaschen Synchronized wmf-config/InitialiseSettings.php: Enable Flow on ptwiki (duration: 00m 05s)
• 21:48 logmsgbot: krenair Synchronized php-1.25wmf14/extensions/TemplateData/modules/ext.templateDataGenerator.ui.tdDialog.js: https://gerrit.wikimedia.org/r/#/c/185999/ (duration: 00m 05s)
• 21:39 logmsgbot: krenair Synchronized php-1.25wmf15/extensions/TemplateData/modules/ext.templateDataGenerator.ui.tdDialog.js: https://gerrit.wikimedia.org/r/#/c/185998/ (duration: 00m 07s)
• 21:16 logmsgbot: krenair Synchronized php-1.25wmf15/extensions/VisualEditor/modules/ve-mw/ui/pages/ve.ui.MWSettingsPage.js: https://gerrit.wikimedia.org/r/#/c/185991/1 (duration: 00m 05s)
• 21:11 logmsgbot: krenair Synchronized php-1.25wmf14/extensions/VisualEditor/modules/ve-mw/ui/pages/ve.ui.MWSettingsPage.js: https://gerrit.wikimedia.org/r/#/c/185992/1 (duration: 00m 07s)
• 20:21 legoktm: deleted localnames/localuser entries in centralauth db that referred to vewikimedia (https://phabricator.wikimedia.org/T87264)
• 19:27 logmsgbot: reedy Finished scap: Add magic word for pl (duration: 14m 21s)
• 19:13 logmsgbot: reedy Started scap: Add magic word for pl
• 16:27 paravoid: restarting puppetmasters (palladium/strontium)
• 16:14 logmsgbot: ebernhardson Synchronized wmf-config/: tues morning swat config changes (duration: 00m 07s)
• 16:14 logmsgbot: ebernhardson Synchronized wmf-config/: tues morning swat config changes (duration: 00m 07s)
• 15:26 _joe_: started manually dumpwikidatajson on snapshot1003, in a root-owned screen session
• 15:25 _joe_: restarted apache on palladium
• 04:01 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jan 20 04:00:59 UTC 2015 (duration 0m 58s)
• 02:43 akosiaris: restarted puppet on strontium
• 02:19 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-01-20 02:19:04+00:00
• 02:19 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 03s)
• 02:11 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-20 02:10:56+00:00
• 02:10 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s)
• 01:06 paravoid: rebooting netmon1001, upstart stuck(?!)
• 01:02 paravoid: power-cycling rhenium

January 19

• 23:46 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1063 (duration: 00m 06s)
• 23:41 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1060 (duration: 00m 07s)
• 23:14 logmsgbot: reedy Synchronized wmf-config/CommonSettings.php: Remove old texvccheck config (duration: 00m 05s)
• 21:51 bd808: Updated Wikimania scholarships to 1f8d8f8 (disable PDO persistent connections; review display fixes)
• 19:40 hoo: Set email of commons user Tatobot to the email of the owning account
• 19:30 akosiaris: https://gerrit.wikimedia.org/r/185610 merged, tested on wtp1024, wtp1023, caused 0 problems, rolling out to the rest of parsoid machines
• 19:23 akosiaris: disable puppet on wtp* hosts for https://gerrit.wikimedia.org/r/#/c/185610/ merge
• 17:27 akosiaris: manually running wikidatajsondump.sh in a screen on datasets1003 after https://gerrit.wikimedia.org/r/185840 was merged
• 16:21 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1060 (duration: 00m 05s)
• 14:57 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1054, warm up (duration: 00m 05s)
• 03:56 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jan 19 03:56:15 UTC 2015 (duration 56m 14s)
• 02:20 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-01-19 02:20:40+00:00
• 02:20 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s)
• 02:12 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-19 02:12:37+00:00
• 02:12 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s)
• 00:43 andrewbogott: restarted keystone service on virt1000

January 18

• 19:39 springle Synchronized wmf-config/db-eqiad.php: depool db1054 (duration: 00m 05s)
• 04:03 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jan 18 04:03:03 UTC 2015 (duration 3m 2s)
• 02:18 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-01-18 02:17:57+00:00
• 02:17 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s)
• 02:10 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-18 02:10:18+00:00
• 02:10 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 02s)

January 17

• 06:36 YuviPanda: restarted dnsmasq on labnet1001 (see https://wikitech.wikimedia.org/wiki/Labs_DNS#DHCP_and_internal_DNS for how to)
• 04:47 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jan 17 04:47:42 UTC 2015 (duration 47m 41s)
• 02:31 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-01-17 02:30:57+00:00
• 02:30 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s)
• 02:18 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-17 02:18:24+00:00
• 02:18 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s)
• 02:17 ori: mw1148: threads in stuck in __lll_lock_wait (); restarted HHVM.
• 02:16 logmsgbot: krinkle Synchronized php-1.25wmf14/includes/content/JsonContent.php: Ic1d10393912fcefa22d (duration: 00m 06s)
• 02:16 logmsgbot: krinkle Synchronized php-1.25wmf14/resources/src/mediawiki/mediawiki.content.json.css: Ic1d10393912fcefa22d (duration: 00m 06s)
• 02:14 logmsgbot: krinkle Synchronized php-1.25wmf15/includes/content/JsonContent.php: Ic1d10393912fcefa22d (duration: 00m 05s)
• 02:14 logmsgbot: krinkle Synchronized php-1.25wmf15/resources/src/mediawiki/mediawiki.content.json.css: Ic1d10393912fcefa22d (duration: 00m 06s)
• 02:10 logmsgbot: ori Synchronized wmf-config/StartProfiler.php: I4e3871d3d: xenon: Annotate file scope and closure scope with filename (duration: 00m 05s)
• 00:13 bd808: restarted elasticserch on logstash1001 & logstash1003; OOM

January 16

• 23:33 bd808: ran `LTRIM logstash -50000 9999999` on redis queues to drop ~4M events in backlog
• 22:14 bd808: restarted elasticsearch on logstash1001; OOM errors
• 21:21 bd808: restarted elasticsearch on logstash1001
• 21:18 logmsgbot: marktraceur Finished scap: Fix UploadWizard regression and EventLogging errors (duration: 31m 06s)
• 21:17 bd808: OOM for elasticsearch on logstash1001 caused a dropped shard and icinga alerts
• 20:47 logmsgbot: marktraceur Started scap: Fix UploadWizard regression and EventLogging errors
• 20:17 logmsgbot: bd808 Synchronized wmf-config/InitialiseSettings.php: Allow wgDebugLogGroups to exclude logstash append (e808e690) (duration: 00m 05s)
• 20:17 logmsgbot: bd808 Synchronized wmf-config/logging.php: Allow wgDebugLogGroups to exclude logstash append (e808e690) (duration: 00m 07s)
• 18:13 bd808: document count not changing for logstash-2015.01.16 index
• 17:59 logmsgbot: bd808 Synchronized wmf-config/logging-labs.php: beta: Allow wgDebugLogGroups to exclude logstash append (03c3ab27) (duration: 00m 06s)
• 17:50 bblack: depooled amssq42 text cache in esams
• 17:44 ejegg: updated tools from 88b57fea517d2232e8ae906df550f426b6574f24 to 84442d51a841af4265ff103827cda83d5dd9dc54
• 17:24 logmsgbot: demon Synchronized wmf-config/: (no message) (duration: 00m 05s)
• 17:21 ejegg: updated civicrm from d648ededf5c9fc2b0ebf989300ca2037956418e3 to 4fa10ec9e3afbf65e6cbd523138cdc4b4485c482
• 17:17 logmsgbot: demon Synchronized wmf-config/: (no message) (duration: 00m 06s)
• 16:48 ottomata: finished hadoop namenode migration. Hadoop cluster is back online
• 16:48 bd808: Upgraded elasticsearch and restarted on all logstash nodes
• 16:43 bd808: shutdown whole elasticsearch cluster for logstash
• 16:39 bd808: restarted elasticsearch on logstash1001
• 16:07 ottomata: stopping hadoop cluster
• 08:01 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1051 db1056, warm up (duration: 00m 10s)
• 06:15 jgage: Icinga test of Mediawiki Apple Dictionary Bridge as https://search.wikimedia.org/?lang=en&site=wikipedia&search=Wikimedia_Foundation&limit=1 returns an error since shortly after l10n update at 02:31 UTC, though URL works without &limit=1 and end user osx dictionary lookups are still working.
• 05:54 ori: <jgage> mtr shows me packet loss between cr2-eqiad.wikimedia.org and 206.126.236.21 aka eqixva-google-gige.google.com
• 04:40 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jan 16 04:40:10 UTC 2015 (duration 40m 9s)
• 04:22 Tim: on mw1228 doing some tests to figure out why incorrect Expires header is being sent on requests for /images/*
• 03:09 logmsgbot: ori Synchronized php-1.25wmf14/includes/content/JsonContent.php: I2f4f9cb343: Let subclasses specify content model in JsonContent (duration: 00m 06s)
• 03:01 springle: xtrabackup clone db1020 to db1046
• 02:31 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-01-16 02:31:37+00:00
• 02:31 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s)
• 02:19 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-16 02:19:04+00:00
• 02:19 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s)
• 02:06 ori: EventLogging syncs were of I335ad42bb: JsonSchemaContent: Fix html rendering of objects and arrays
• 02:03 logmsgbot: ori Synchronized php-1.25wmf14/extensions/EventLogging: (no message) (duration: 00m 05s)
• 02:03 logmsgbot: ori Synchronized php-1.25wmf15/extensions/EventLogging: (no message) (duration: 00m 06s)
• 00:46 mutante: on both puppetmasters: chown gitpuppet /var/lib/git/operations/puppet/.git/logs/refs/heads/production & .git/logs/HEAD & .git/logs/refs/remotes/origin to fix puppet-merge. git pulled on strontium
• 00:46 mutante: restarted morebots

January 15

• 23:09 bd808: Updated scholarships.wikimedia.org to d598e0d
• 22:08 bd808: restarted elasticsaerch on logstash1003; died from OOM
• 21:06 subbu: deployed parsoid version 2fdf9298
• 20:38 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: I250ecfceb: Switch all wikis to monolog logger (duration: 00m 05s)
• 20:04 bd808: logstash redis queue backlog 384k events and climbing; likely related to the elasticsearch cluster flapping
• 19:53 Coren: aborting labs filesystem move (not enough contiguous free space) and postponing until new shelf
• 18:59 YuviPanda: this works?
• 18:23 csteipp: deployed patches for T85349 T85850 T86711
• 17:26 ejegg: updated crm from bb05adf9279bd7a795906ca476e1850a85c21711 to d648ededf5c9fc2b0ebf989300ca2037956418e3
• 16:51 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 06s)
• 16:09 bd808: Deleted 2015-12-* indices from logstash elasticsearch cluster
• 16:07 logmsgbot: anomie Synchronized php-1.25wmf14/extensions/FlaggedRevs/api/actions/ApiReview.php: SWAT: Fix FlaggedRevs action=review for binary flagging gerrit:185180 (duration: 00m 07s)
• 16:01 bd808: Elasticsearch cluster for logstash has indices for events dated 2015-12-* again
• 15:49 Jeff_Green: many frack host package updates and reboots
• 10:09 logmsgbot: aude Synchronized php-1.25wmf14/extensions/Wikidata: fix noexternallanglinks bug (duration: 00m 13s)
• 08:24 qchris: Ran kafka leader re-election to bring analytics1021 back into the set of leaders
• 08:03 _joe_: restarted ES on logstash1002, not joining the cluster
• 07:47 _joe_: restarted HHVM on mw1119, all threads stuck in a lock for HPHP::RequestInjectionData::onSessionInit
• 07:45 ori: re-enabled puppet on osmium. i disabled it three hours ago to debug zhwiki key errors in memcached-serious.log.
• 07:19 YuviPanda: set home of mwdeploy to /home/mwdeploy in LDAP
• 07:09 YuviPanda: changed ldap mwdeploy user shell to /bin/bash to match puppet
• 06:21 YuviPanda: ldap mass modification: Changing everyone with shell set to sillyshell to /bin/bash
• 04:56 springle: upgrade db1051 db1056 trusty
• 04:35 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jan 15 04:35:40 UTC 2015 (duration 35m 39s)
• 03:03 logmsgbot: tstarling Synchronized wmf-config/InitialiseSettings.php: TitleBlacklist log (duration: 00m 06s)
• 02:32 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-01-15 02:32:40+00:00
• 02:32 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s)
• 02:26 springle: tin mw1084 sync-file failed socket error, manual sync-common
• 02:23 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1051 db1056 (duration: 00m 05s)
• 02:20 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-15 02:20:40+00:00
• 02:20 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s)
• 01:53 springle: raise mysql max_connections to 250 on virt1000. lots of nova persistent connections, little activity
• 01:26 logmsgbot: kaldari Synchronized php-1.25wmf15/extensions/Thanks/: syncing Thanks for wmf15 (duration: 00m 05s)
• 01:26 logmsgbot: maxsem Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/185106 (duration: 00m 06s)

January 14

• 22:57 logmsgbot: ori Synchronized wmf-config/StartProfiler.php: I5bd397456: Restrict "forceprofile" to requests that set X-Wikimedia-Debug header (duration: 00m 06s)
• 22:47 logmsgbot: kartik Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 05s)
• 22:40 logmsgbot: kartik Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 06s)
• 22:31 logmsgbot: reedy Synchronized php-1.25wmf14/includes/libs/virtualrest/ParsoidVirtualRESTService.php: (no message) (duration: 00m 05s)
• 22:18 logmsgbot: reedy Synchronized php-1.25wmf15/includes/libs/virtualrest/ParsoidVirtualRESTService.php: (no message) (duration: 00m 05s)
• 22:18 logmsgbot: reedy Synchronized php-1.25wmf15/includes/libs/virtualrest/ParsoidVirtualRESTService.php: (no message) (duration: 00m 05s)
• 22:13 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: CT on testwiki (duration: 00m 06s)
• 22:09 mutante: made an /a/tmp/ on fluorine to let wikidev write to for log analysis
• 22:08 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: touch (duration: 00m 06s)
• 22:03 mutante: apt-get clean on fluorine to get a little disk space
• 21:45 logmsgbot: kartik Synchronized wmf-config: Enable ContentTranslation (duration: 00m 06s)
• 21:25 subbu: reverted parsoid deploy to previous deployment (2cd6fefa) -- seeing a lot of dirty diffs post-deploy
• 21:24 logmsgbot: reedy Finished scap: Add ContentTranslation messages (duration: 29m 14s)
• 21:11 subbu: deployed parsoid version 45b0aafb (deploy sha 88525538)
• 20:55 logmsgbot: reedy Started scap: Add ContentTranslation messages
• 20:55 logmsgbot: hashar scap aborted: (no message) (duration: 00m 01s)
• 20:54 logmsgbot: hashar Started scap: (no message)
• 20:37 hashar: Restarting Zuul
• 20:36 hashar: Zuul applied Ori patch to fix a git lock contention in Zuul-cloner bug T86730 . Tagged wmf-deploy-20150114-1
• 20:32 logmsgbot: reedy Synchronized wmf-config/: Update IW cache (duration: 00m 06s)
• 20:31 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf15
• 20:30 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.25wmf14
• 20:22 logmsgbot: reedy Finished scap: testwiki to 1.25wmf15 and build l10n caches. Some extension bumps in wmf14 (duration: 19m 53s)
• 20:02 logmsgbot: reedy Started scap: testwiki to 1.25wmf15 and build l10n caches. Some extension bumps in wmf14
• 20:01 logmsgbot: reedy scap failed: CalledProcessError Command '('/usr/bin/git', 'merge-base', 'HEAD', 'gerrit\norigin')' returned non-zero exit status 128 (duration: 00m 21s)
• 20:00 logmsgbot: reedy Started scap: verbose
• 19:57 logmsgbot: reedy scap failed: CalledProcessError Command '('/usr/bin/git', 'merge-base', 'HEAD', 'gerrit\norigin')' returned non-zero exit status 128 (duration: 01m 34s)
• 19:56 logmsgbot: reedy Started scap: testwiki to 1.25wmf15 and build l10n caches. Some extension bumps in wmf14. take 2
• 19:45 logmsgbot: reedy scap failed: CalledProcessError Command '('/usr/bin/git', 'merge-base', 'HEAD', 'gerrit\norigin')' returned non-zero exit status 128 (duration: 06m 47s)
• 19:38 logmsgbot: reedy Started scap: testwiki to 1.25wmf15 and build l10n caches. Some extension bumps in wmf14
• 18:19 mutante: deleted /etc/logrotate.d/dumpwikidatajson on snapshot1003 (gerrit 182173)
• 17:30 Reedy: mw1152: Permission denied (publickey).
• 17:29 logmsgbot: reedy Purged l10n cache for 1.25wmf12
• 17:27 Reedy: deleted php-1.25wmf5 through php-1.25wmf9 from /srv/mediawiki
• 17:27 YuviPanda: sync-common && scap-rebuild-cdbs completed on virt1000, wikitech still seems up. Tending to the wounded.
• 17:23 YuviPanda: rm -rf /srv/mediawiki-staging/php-1.25wmf5 on tin
• 16:57 YuviPanda: running sync-common —verbose on virt1000
• 16:55 logmsgbot: krenair Synchronized wmf-config/wikitech.php: https://gerrit.wikimedia.org/r/#/c/184635/ (duration: 00m 06s)
• 16:48 logmsgbot: krenair Synchronized php-1.25wmf14/extensions/MultimediaViewer/resources/mmv/mmv.lightboxinterface.js: https://gerrit.wikimedia.org/r/#/c/184633/ (duration: 00m 07s)
• 16:38 logmsgbot: krenair Synchronized php-1.25wmf14: https://gerrit.wikimedia.org/r/#/c/184818/ (duration: 06m 14s)
• 15:46 ottomata: stopping all varnishkafka bits instances
• 10:15 logmsgbot: ori Synchronized wmf-config/mc.php: typo fix for nutcracker socket (duration: 00m 05s)
• 10:05 logmsgbot: ori Synchronized wmf-config/mc.php: nutcracker: use UNIX domain socket only if on HHVM (duration: 00m 07s)
• 09:57 logmsgbot: ori Synchronized wmf-config/mc.php: Memcached: make remaining app servers use UNIX domain socket (duration: 00m 06s)
• 08:24 _joe_: repooling mw1225, depooled for a long time
• 07:44 logmsgbot: ori Synchronized wmf-config/mc.php: use UNIX domain socket on mw12* app servers (duration: 00m 06s)
• 04:15 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jan 14 04:15:41 UTC 2015 (duration 15m 40s)
• 02:24 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-14 02:24:36+00:00
• 02:24 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s)
• 02:16 logmsgbot: LocalisationUpdate completed (1.25wmf13) at 2015-01-14 02:16:18+00:00
• 02:16 logmsgbot: l10nupdate Synchronized php-1.25wmf13/cache/l10n: (no message) (duration: 00m 01s)
• 01:17 logmsgbot: maxsem Synchronized wmf-config: https://gerrit.wikimedia.org/r/184812 (duration: 00m 06s)
• 01:01 logmsgbot: maxsem Synchronized wmf-config: touch (duration: 00m 08s)
• 00:59 logmsgbot: maxsem Synchronized wmf-config: touch (duration: 00m 07s)
• 00:50 logmsgbot: maxsem Synchronized wmf-config/: https://gerrit.wikimedia.org/r/184810 https://gerrit.wikimedia.org/r/184757 (duration: 00m 06s)
• 00:46 logmsgbot: maxsem Synchronized php-1.25wmf14/extensions/Wikidata/: https://gerrit.wikimedia.org/r/#/c/184807/ (duration: 00m 11s)
• 00:28 logmsgbot: maxsem Synchronized php-1.25wmf14/extensions/Wikidata/: https://gerrit.wikimedia.org/r/#/c/184807/ (duration: 00m 12s)

January 13

• 23:50 ori: Updated nutcracker on application servers to 0.4.0+dfsg-1+wm1.
• 22:52 logmsgbot: ori Synchronized wmf-config/mc.php: Use UNIX domain socket for nutcracker on mw1030 & mw1031 (duration: 00m 05s)
• 22:37 hashar: Restarted Zuul, deadlocked waiting for Gerrit
• 22:37 ejegg: updated payments from c23cf16407ef200da446d81fb990abbe429fd378 to 46bd1611c56d6e31594d01aa345c35dd45dcf676
• 20:48 Reedy: running mwscript extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki=mediawikiwiki --reindexAndRemoveOk --indexIdentifier=now
• 20:48 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: mediawikiwiki content namespaces (duration: 00m 05s)
• 20:42 logmsgbot: reedy Synchronized wmf-config/: Config updates (duration: 00m 06s)
• 20:41 logmsgbot: reedy Synchronized database lists: wikidata dblist update (duration: 00m 06s)
• 19:38 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: Add Renameuser debug log group (duration: 00m 09s)
• 19:24 logmsgbot: reedy Synchronized wmf-config/Wikibase.php: bump cache epoch (duration: 00m 06s)
• 19:09 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: monolog: enable for group0 + group1 wikis (duration: 00m 07s)
• 19:06 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non Wikipedias to 1.25wmf14
• 18:40 ori: mw1062: sync-file failed, read-only file system. Host should be removed from dsh group.
• 18:38 logmsgbot: ori Synchronized wmf-config/StartProfiler.php: xenon: Skip frames that don't have a 'phpStack' key (duration: 00m 06s)
• 17:48 hashar: If Zuul status page ( https://integration.wikimedia.org/zuul/ ) shows a lot of changes with completed jobs and the number of results growing, Zuul is deadlocked waiting for Gerrit. Have to restart it on gallium.wikimedia.org with /etc/init.d/zuul restart
• 17:39 hashar: Zuul back in action. Got recheck or +2 again the changes that have been discarded.
• 17:37 hashar: Restarting deadlocked Zuul , which drops ALL events. Reason is Gerrit lost connection with its database which is not handled by Zuul . See https://wikitech.wikimedia.org/wiki/Incident_documentation/20150106-Zuul
• 16:12 logmsgbot: demon Synchronized wmf-config/: (no message) (duration: 00m 05s)
• 16:12 logmsgbot: demon Synchronized flaggedrevs.dblist: (no message) (duration: 00m 05s)
• 16:02 YuviPanda: restart mysql on virt1000, wikitech acting up again
• 15:23 godog: upgrade and restart poolcounter on helium
• 15:17 godog: upload poolcounter 1.0.3 to precise-wikimedia
• 15:17 YuviPanda: restarted mysql on virt1000, wikitech failing with db errors. seems fine now (4 minutes ago)
• 15:17 YuviPanda: test
• 15:14 _joe_: jobrunners 100% on HHVM
• 15:14 _joe_: reimaging mw1007 mw1008
• 14:36 bblack: ssl runtime config updated for +3DES/-RC4 ( I87616455abd58c986aa960348fc20c017f097716 )
• 14:07 _joe_: reimaging mw1005,mw1006
• 13:45 Jeff_Green: package updates and reboots on many fundraising hosts
• 11:30 _joe_: reimaging mw1003, mw1004
• 11:30 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1067, warm up (duration: 00m 05s)
• 11:20 springle: upgrade db1067 trusty
• 10:30 hoo: Set email for global account "Liberipedia" as per https://phabricator.wikimedia.org/T76321
• 10:19 _joe_: reimaging mw1001, mw1002
• 10:02 _joe_: net.ipv4.tcp_tw_reuse = 1 on mw1223
• 08:58 _joe_: raising the net.ipv4.ip_local_port_range on mw1196
• 08:43 _joe_: raising the net.ipv4.ip_local_port_range on mw1230
• 07:30 springle: ongoing schema changes T86415 externallinks, codfw first
• 04:55 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1067 (duration: 00m 05s)
• 04:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jan 13 04:07:54 UTC 2015 (duration 7m 52s)
• 02:29 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-13 02:29:21+00:00
• 02:29 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 03s)
• 02:17 logmsgbot: LocalisationUpdate completed (1.25wmf13) at 2015-01-13 02:17:08+00:00
• 02:17 logmsgbot: l10nupdate Synchronized php-1.25wmf13/cache/l10n: (no message) (duration: 00m 03s)
• 00:44 logmsgbot: ori Finished scap: Updates to MobileFrontend, CentralAuth, EventLogging and WikimediaEvents (duration: 06m 27s)
• 00:38 logmsgbot: ori Started scap: Updates to MobileFrontend, CentralAuth, EventLogging and WikimediaEvents

January 12

• 23:06 ejegg: updated crm from d8a1160bca99354a856b1595cedf5c33f9ac255c to bb05adf9279bd7a795906ca476e1850a85c21711
• 21:38 hoo: Set email for global account "Carol.Christiansen" after having it confirmed by a steward and a dewiki bureaucrat (also based on old OTRS records)
• 21:12 subbu: deployed parsoid version 2cd6fefa
• 18:48 hoo: Ran sync-common on osmium
• 18:21 mutante: purging 'mlocate' package from neon as well to fix Icinga DPKG crits
• 18:04 bd808: Deployed scholarships at hash a5bc6fd
• 18:04 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 06s)
• 18:03 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 08s)
• 18:01 bd808: Applied 2015 schema changes to scholarships database on m2-master
• 17:33 hoo: mw1010: rsync: failed to set times on "/srv/mediawiki/.": Read-only file system (30)
• 17:31 logmsgbot: hoo Synchronized php-1.25wmf13/extensions/CentralAuth/: Only test passwords once in CentralAuthUser::prepareMigration - 2nd try (duration: 00m 07s)
• 17:31 logmsgbot: hoo Synchronized php-1.25wmf14/extensions/CentralAuth/: Only test passwords once in CentralAuthUser::prepareMigration (duration: 00m 06s)
• 17:31 logmsgbot: hoo Synchronized php-1.25wmf13/extensions/CentralAuth/: Only test passwords once in CentralAuthUser::prepareMigration (duration: 00m 06s)
• 17:22 mutante: restarted icinga-wm to join -releng
• 17:02 mutante: labmon1001 - purging mlocate package that was status 'rc'
• 16:30 godog: stop/start graphite-web on tungsten to clear logs
• 16:28 bd808: deleted 2014-01-* and 2015-12-* indices from logstash elasticsearch cluster
• 16:13 bd808: logs on logstash1001 reporting elasticserch connection errors; restarted logstash service
• 16:10 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable "Other projects sidebar" by default on frwiki gerrit:183288 (duration: 00m 05s)
• 16:09 bd808: logstash elasticsearch cluster has strange indices dated 2014-01-* and 2015-12-* again
• 16:05 bd808: restarted elasticsearch on logstash1001
• 16:05 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable thumbnail prerendering in production gerrit:183885 (duration: 00m 06s)
• 16:03 _joe_: depooling mw1062, disk errors
• 16:03 bd808: elasticsearch on logstash1001 not responding to http requests
• 16:01 logmsgbot: aude Synchronized wmf-config/Wikibase.php: Enable usage tracking on test.wikidata and testwiki (duration: 00m 05s)
• 16:00 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 05s)
• 15:59 bd808: logstash not showing any events at all since 2015-01-12T13:58:59.728Z
• 15:58 logmsgbot: aude Finished scap: Update Wikidata and WikimediaMessages (duration: 32m 06s)
• 15:35 hashar: Enabling test/gate of several extensions together. 180494 , RFC extensions continuous integration bug T1350
• 15:26 aude: Added and populated wbc_entity_usage table on testwiki and testwikidatawiki
• 15:25 logmsgbot: aude Started scap: Update Wikidata and WikimediaMessages
• 15:10 logmsgbot: hashar Synchronized php-1.25wmf13/extensions/Echo/tests/phpunit/includes/cache/TitleLocalCacheTest.php: php-1.25wmf14/extensions/Echo/tests/phpunit/includes/cache/TitleLocalCacheTest.php (duration: 00m 05s)
• 15:02 hashar: restarting Zuul. Deadlocked due to Gerrit database
• 12:20 _joe_: upgrading HHVM on the API cluster
• 10:45 _joe_: restarting hhvm on mw1126, stuck in HPHP::StatCache::refresh
• 10:22 _joe_: upgrading HHVM on all appservers
• 10:11 logmsgbot: hashar Synchronized wmf-config/throttle.php: Tel-Hai Academic College event - Bug: T85773 (duration: 00m 07s)
• 09:23 hashar: Restarting Zuul
• 09:06 hashar: Tweak Zuul configuration to pin python-daemon <= 2.0 and deploying tag wmf-deploy-20150112-1. bug T86513
• 07:24 andrewbogott: on virt1005 and virt1006, ran 'ln -s /usr/bin/qemu-system-x86_64 /usr/bin/kvm' that allows nova to migrate instances between hosts.
• 03:54 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jan 12 03:54:34 UTC 2015 (duration 54m 33s)
• 02:17 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-12 02:17:48+00:00
• 02:17 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s)
• 02:11 logmsgbot: LocalisationUpdate completed (1.25wmf13) at 2015-01-12 02:10:54+00:00
• 02:10 logmsgbot: l10nupdate Synchronized php-1.25wmf13/cache/l10n: (no message) (duration: 00m 01s)

January 11

• 16:44 springle: db1050 dberror log noise was https://phabricator.wikimedia.org/T86482
• 16:29 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1050, warm up (duration: 00m 05s)
• 16:08 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1050, mysqld got TERM somehow (duration: 00m 05s)
• 03:59 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jan 11 03:59:26 UTC 2015 (duration 59m 25s)
• 02:18 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-11 02:18:10+00:00
• 02:18 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 04s)
• 02:11 logmsgbot: LocalisationUpdate completed (1.25wmf13) at 2015-01-11 02:11:13+00:00
• 02:11 logmsgbot: l10nupdate Synchronized php-1.25wmf13/cache/l10n: (no message) (duration: 00m 02s)

January 10

• 23:20 _joe_: restarted the puppetmaster on palladium
• 19:44 logmsgbot: hoo Synchronized wmf-config/Bug54847.php: Don't use protected CentralAuthUser::getPasswordObject (duration: 00m 06s)
• 18:18 logmsgbot: hoo Synchronized wmf-config/Bug54847.php: Bug54847.php: Replace removed CentralAuthUser::getPasswordHash (duration: 00m 06s)
• 18:10 hoo: Ran mysql:wikiadmin@db1033 [centralauth]> DELETE FROM bug_54847_password_resets WHERE r_username = 'Tk';
• 04:03 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jan 10 04:03:52 UTC 2015 (duration 3m 50s)
• 02:24 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-10 02:24:48+00:00
• 02:24 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s)
• 02:17 logmsgbot: LocalisationUpdate completed (1.25wmf13) at 2015-01-10 02:17:29+00:00
• 02:17 logmsgbot: l10nupdate Synchronized php-1.25wmf13/cache/l10n: (no message) (duration: 00m 04s)
• 00:09 hoo: Set email of global account "Classicfilms" to the commonswiki, enwikiquote and enwiktionary accounts of the same name.

January 9

• 22:45 ejegg: updated crm from 8ded33737337d0c842d8c194b5cb15b25fc99c2e to d8a1160bca99354a856b1595cedf5c33f9ac255c
• 21:58 ejegg: enabled queue consumers
• 21:48 ejegg: disabled queue consumers
• 20:16 ottomata: stopping esams bits varnishkafka instances
• 17:19 logmsgbot: marktraceur Synchronized php-1.25wmf14/includes/filerepo/file/File.php: Remove silly debug line from File class (duration: 00m 07s)
• 17:18 logmsgbot: marktraceur Synchronized php-1.25wmf13/includes/filerepo/file/File.php: Remove silly debug line from File class (duration: 00m 08s)
• 16:58 Reedy: CREATE INDEX /*i*/br_timestamp ON /*_*/bounce_records(br_timestamp); for bounce_records on wikishared on extension1
• 15:24 Jeff_Green: deployed DNS dmarc record for wikipedia.*
• 10:19 _joe_: reimaging mw1152 as a HAT imagescaler
• 05:36 ori: repooled mw123[12]
• 05:32 logmsgbot: ori Synchronized wmf-config/mc.php: I33ff81e6a: memcached: set server address to localhost rather than 127.0.0.1 on mw123* (duration: 00m 05s)
• 04:32 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jan 9 04:32:31 UTC 2015 (duration 32m 30s)
• 03:05 springle: upgrade db1016 trusty
• 03:01 MaxSem: Running mwscript extensions/WikiGrok/maintenance/refreshCampaigns.php --wiki=enwiki --version=1 in screen session on terbium, feel free to kill if causes problems
• 02:50 logmsgbot: maxsem Synchronized php-1.25wmf13/extensions/MobileFrontend/: (no message) (duration: 00m 07s)
• 02:49 logmsgbot: maxsem Synchronized php-1.25wmf13/extensions/Mantle: (no message) (duration: 00m 07s)
• 02:42 logmsgbot: maxsem Synchronized php-1.25wmf14/extensions/MobileFrontend/: (no message) (duration: 00m 06s)
• 02:42 logmsgbot: maxsem Synchronized php-1.25wmf13/extensions/MobileFrontend/: (no message) (duration: 00m 07s)
• 02:42 logmsgbot: maxsem Synchronized php-1.25wmf13/extensions/Mantle: (no message) (duration: 00m 05s)
• 02:30 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1003 db1005 db1006 db1009. repool db1050 in s6, db1015 in s3 (duration: 00m 06s)
• 02:24 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-09 02:24:15+00:00
• 02:24 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s)
• 02:18 logmsgbot: LocalisationUpdate completed (1.25wmf13) at 2015-01-09 02:18:23+00:00
• 02:18 logmsgbot: l10nupdate Synchronized php-1.25wmf13/cache/l10n: (no message) (duration: 00m 01s)
• 01:49 logmsgbot: maxsem Synchronized php-1.25wmf13/extensions/MobileFrontend: touch (duration: 00m 09s)
• 01:25 logmsgbot: maxsem Finished scap: SWAT: MobileFrontend and WikiGrok updates (duration: 17m 36s)
• 01:07 logmsgbot: maxsem Started scap: SWAT: MobileFrontend and WikiGrok updates
• 00:53 hoo: Updated the Wikidata property suggester with data from Monday's JSON dump
• 00:46 logmsgbot: legoktm Synchronized wmf-config/InitialiseSettings.php: SWAT: https://gerrit.wikimedia.org/r/#/c/180451/ (duration: 00m 06s)
• 00:45 logmsgbot: legoktm Synchronized closed.dblist: SWAT: https://gerrit.wikimedia.org/r/#/c/180451/ (duration: 00m 08s)
• 00:42 logmsgbot: legoktm Synchronized php-1.25wmf13/extensions/CentralAuth/: SWAT: https://gerrit.wikimedia.org/r/#/c/183554/ (duration: 00m 06s)
• 00:40 logmsgbot: legoktm Synchronized php-1.25wmf14/extensions/CentralAuth/: SWAT: https://gerrit.wikimedia.org/r/#/c/183554/ (duration: 00m 06s)
• 00:37 logmsgbot: legoktm Synchronized php-1.25wmf13/extensions/Scribunto/engines/LuaCommon/lualib/mw.title.lua: SWAT: https://gerrit.wikimedia.org/r/#/c/183552/ again (duration: 00m 07s)
• 00:36 logmsgbot: legoktm Synchronized php-1.25wmf14/extensions/Scribunto/engines/LuaCommon/lualib/mw.title.lua: SWAT: https://gerrit.wikimedia.org/r/#/c/183552/ again (duration: 00m 06s)
• 00:29 logmsgbot: legoktm Synchronized php-1.25wmf13/extensions/Scribunto/engines/LuaCommon/lualib/mw.title.lua: SWAT: https://gerrit.wikimedia.org/r/#/c/183552/ (duration: 00m 07s)
• 00:27 logmsgbot: legoktm Synchronized php-1.25wmf14/extensions/Scribunto/engines/LuaCommon/lualib/mw.title.lua: SWAT: https://gerrit.wikimedia.org/r/#/c/183552/ (duration: 00m 06s)

January 8

• 23:46 logmsgbot: ori Synchronized wmf-config/mc.php: (no message) (duration: 00m 07s)
• 23:32 logmsgbot: ori Synchronized wmf-config/mc.php: Revert: I4c4691e26: memcached: use a unix socket instead of a tcp connection on selected hosts (duration: 00m 06s)
• 23:30 logmsgbot: ori Synchronized wmf-config/mc.php: I4c4691e26: memcached: use a unix socket instead of a tcp connection on selected hosts (duration: 00m 06s)
• 23:26 ori: depooling mw1230 and mw1231 for a couple of minutes for I4c4691e26
• 21:53 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: nooop to test scap update (duration: 00m 06s)
• 21:52 mutante: fixing scap permissions on mediawiki-installation servers via dsh
• 21:29 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: nooop to test scap update (duration: 00m 06s)
• 21:04 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: nooop to test scap update (duration: 00m 09s)
• 21:00 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: nooop to test scap update (duration: 00m 06s)
• 20:39 Reedy: Scap deployed at a78ddec
• 20:22 Reedy: moved /srv/deployment/scap to scap.old as git repo seems busted. Hoping puppet puts it back again correctly...
• 19:47 Reedy: scap-rebuild-cdbs finished
• 19:44 Reedy: running dsh -g mediawiki-installation -M -F 40 -- "sudo -u mwdeploy /srv/deployment/scap/scap/bin/scap-rebuild-cdbs"
• 19:43 logmsgbot: reedy Synchronized php-1.25wmf14/cache/l10n/: l10nupdate (duration: 03m 39s)
• 19:39 logmsgbot: reedy Synchronized php-1.25wmf13/cache/l10n/: (no message) (duration: 00m 05s)
• 19:38 logmsgbot: reedy Synchronized php-1.25wmf13/cache/l10n/: l10nupdate (duration: 03m 54s)
• 19:26 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-08 19:26:46+00:00
• 19:25 logmsgbot: reedy Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s)
• 19:14 logmsgbot: LocalisationUpdate completed (1.25wmf13) at 2015-01-08 19:14:37+00:00
• 19:13 logmsgbot: reedy Synchronized php-1.25wmf13/cache/l10n: (no message) (duration: 00m 01s)
• 18:58 Reedy: Attempting manual run of l10nupdate
• 17:08 logmsgbot: marktraceur Finished scap: [SWAT] [AbuseFilter] Add file_size variable (duration: 33m 27s)
• 16:51 hashar: Restarted Zuul. Same issue as https://wikitech.wikimedia.org/wiki/Incident_documentation/20150106-Zuul
• 16:35 logmsgbot: marktraceur Started scap: [SWAT] [AbuseFilter] Add file_size variable
• 16:25 logmsgbot: marktraceur Synchronized php-1.25wmf14/extensions/AbuseFilter: [SWAT] [AbuseFilter] Add file_size variable (duration: 00m 06s)
• 16:14 logmsgbot: marktraceur Synchronized php-1.25wmf14/extensions/VisualEditor/: [SWAT] VisualEditor fixes for T86046 and T86056 (duration: 00m 05s)
• 16:02 logmsgbot: marktraceur Synchronized wmf-config/logging.php: [SWAT] Honor log sampling and levels for logstash on group0 wikis (duration: 00m 05s)
• 15:44 _joe_: restarting hhvm on mw1226, TC full
• 15:37 _joe_: restarting hhvm on mw1113, stuck in parsing the ini file (HPHP::is_valid_var_name)
• 15:08 logmsgbot: aude Synchronized wmf-config/Wikibase.php: Bump cache epoch for test.wikidata (duration: 00m 06s)
• 15:01 logmsgbot: aude Finished scap: Update group0 to wmf/1.25wmf14 Wikidata extension branch (duration: 26m 43s)
• 14:35 logmsgbot: aude Started scap: Update group0 to wmf/1.25wmf14 Wikidata extension branch
• 14:34 Jeff_Green: samarium package updates and reboot
• 11:37 godog: reboot ms-be1011, xfs hosed :(
• 11:07 _joe_: repooled the canary servers
• 10:55 _joe_: installing a new hhvm package (with the correct libicu dependence) on canary hosts
• 10:02 godog: removing backend hosts from LVS for search pools
• 09:42 godog: restart uwsgi on tungsten
• 08:15 _joe_: depooled the canary appservers while a new package version is rebuilt
• 08:11 _joe_: installing a new hhvm package version on the canary pools
• 03:06 springle: xtrabackup clone db1037 to db1050
• 01:04 bd808: cleaned up logstash indices dated 2014-01-* and 2015-12-* that look to have been created by some sort of syslog input parsing bug
• 01:01 bd808: accidentally deleted 2015-01-07 logstash index when cleaning up rogue indices for 2014-01-*
• 00:40 bd808: restarted elasticsearch on logstash1002 to heal split brain in cluster
• 00:38 bd808: elasticsearch cluster for logstash is split brain.
• 00:37 logmsgbot: maxsem Synchronized php-1.25wmf13/extensions/WikiGrok/: https://gerrit.wikimedia.org/r/#/c/183186/ (duration: 00m 07s)
• 00:37 logmsgbot: maxsem Synchronized php-1.25wmf14/extensions/WikiGrok/: https://gerrit.wikimedia.org/r/#/c/183186/ (duration: 00m 08s)
• 00:34 bd808: restarted logstash on logstash1001
• 00:10 K4-713: updated payments settings

January 7

• 21:11 subbu: deployed parsoid version 904fab9e
• 19:51 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: And disable error log again (duration: 00m 06s)
• 19:49 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: Enabling error log for a few minutes (duration: 00m 15s)
• 19:44 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf14
• 19:42 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.25wmf13
• 19:36 logmsgbot: reedy Finished scap: testwiki to 1.25wmf14... (duration: 28m 32s)
• 19:07 logmsgbot: reedy Started scap: testwiki to 1.25wmf14...
• 18:42 godog: reboot ms-be2011, megacli in a funny state and unable to bring new drive in service
• 17:20 logmsgbot: ori Synchronized php-1.25wmf13/extensions/EventLogging/modules/ext.eventLogging.core.js: I5470424: Correct events to send schema name (duration: 00m 05s)
• 17:20 logmsgbot: ori Synchronized php-1.25wmf12/extensions/EventLogging/modules/ext.eventLogging.core.js: I5470424: Correct events to send schema name (duration: 00m 06s)
• 17:15 godog: reboot ms-be2003
• 15:36 springle: xtrabackup clone codfw slaves db2034 db2035 db2036 db2037 db2038 db2039 db2040 from other codfw slaves
• 14:40 _joe_: upgrading hhvm on testwiki
• 14:16 springle: xtrabackup clone db1027 to db1015
• 11:42 _joe_: reimaging mw1009-mw1012
• 10:19 godog: reboot ms-be2003, deleted LD should disappear
• 09:47 hashar: restarting Jenkins to resolve a deadlocks with the beta cluster jobs
• 07:53 _joe_: reimaging jobrunners mw1013-mw1016 (in batch of two)
• 06:50 springle: xtrabackup clone es2008 to es2010
• 06:50 springle: xtrabackup clone es2006 to es2007
• 02:05 logmsgbot: kaldari Synchronized php-1.25wmf13/extensions/WikiGrok/: Fixing campaign generation in WikiGrok (duration: 00m 05s)
• 01:57 logmsgbot: ori Synchronized php-1.25wmf13/extensions/EventLogging: I69c8daf: Update EventLogging for cherry-picks (duration: 00m 05s)
• 01:57 logmsgbot: ori Synchronized php-1.25wmf12/extensions/EventLogging: I07d9bc8: Update EventLogging for cherry-picks (duration: 00m 06s)
• 01:33 springle: xtrabackup clone es2008 to es2009
• 01:24 logmsgbot: kaldari Synchronized php-1.25wmf13/extensions/MobileFrontend/: Sync MobileFrontend in 1.25wmf13 for VE fix (duration: 00m 05s)
• 01:11 springle: xtrabackup clone es2006 to es2005

January 6

• 21:49 awight: update crm from 80241fd2a43f03796b416d728661470f875a590a to 8ded33737337d0c842d8c194b5cb15b25fc99c2e
• 20:24 logmsgbot: reedy Synchronized wmf-config/: (no message) (duration: 00m 06s)
• 20:16 Reedy: Ran namespaceDupes.php on ndswiktionary
• 20:16 logmsgbot: reedy Synchronized wmf-config/: Config updates (duration: 00m 06s)
• 20:00 logmsgbot: reedy Synchronized docroot and w: update noc index (duration: 00m 06s)
• 19:42 logmsgbot: demon Finished scap: mwsearch is no more (duration: 35m 40s)
• 19:06 logmsgbot: demon Started scap: mwsearch is no more
• 19:03 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: nooop
• 19:01 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.25wmf13
• 18:30 godog: upload txstatsd 1.0.0-2 to trusty-wikimedia
• 16:22 logmsgbot: anomie Synchronized php-1.25wmf13/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.MobileViewTarget.js: SWAT: Update setupToolbar signature in mobile target gerrit:182993 (duration: 00m 06s)
• 16:01 logmsgbot: anomie Synchronized wmf-config/flaggedrevs.php: SWAT: Add Flexion to $wgFlaggedRevsNamespaces for dewiktionary gerrit:180813 (duration: 00m 08s)
• 10:36 hashar: The Zuul issues have their root cause in Gerrit which cannot open ReviewDB. Filled as https://phabricator.wikimedia.org/T85916
• 10:21 hashar: Restarted Zuul. Gerrit transiently died out just like ~10 hours ago which locked Zuul entirely
• 09:41 springle: started an analytics ETL run on dbstore1002. to disable: set global event_scheduler=0;
• 09:41 godog: starting hhvm-profiler-to-carbon on tungsten T85641
• 09:16 mutante: upgraded python version on zirconium
• 09:13 logmsgbot: hashar Synchronized wmf-config/CommonSettings-labs.php: (no message) (duration: 00m 05s)
• 08:42 hashar: Zuul scheduler was stuck while reporting a change back to Gerrit waiting for data to be received. For some reason none came back and Zuul halted entirely. Restarting Gerrit killed the stalled connection and made Zuul to drop all events and resume operations.
• 08:38 andrewbogott: restarted gerrit service on ytterbium
• 08:11 hashar: Zuul stalled for some reason :(
• 08:07 andrewbogott: restarted pdns on virt1000 and labcontrol2001 to handle the change to nembus (just in case pdns is upset by change!)
• 08:07 andrewbogott: moved codfw ldap service to nembus
• 02:03 logmsgbot: aude Finished scap: Add Wikidata other projects message (duration: 37m 18s)
• 01:26 logmsgbot: aude Started scap: Add Wikidata other projects message
• 01:23 logmsgbot: aude Synchronized php-1.25wmf13/extensions/WikimediaMessages: Add Wikidata other projects message (duration: 00m 06s)
• 01:16 logmsgbot: aude Synchronized php-1.25wmf13/extensions/MobileFrontend: Fix MobileFrontend bugs (duration: 00m 06s)
• 00:57 logmsgbot: aude Synchronized php-1.25wmf12/extensions/MoodBar: Bug fixes for MoodBar (duration: 00m 09s)
• 00:49 logmsgbot: aude Synchronized php-1.25wmf13/extensions/MoodBar: Bug fixes for MoodBar (duration: 00m 06s)
• 00:44 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Update collabwiki bureaucrat permissions (duration: 00m 06s)
• 00:25 greg-g: restarting jenkins, hope that kicks it enough
• 00:23 awight: updated payments from 62c81d4574e5e994ff8f3cac7115eff335bd5265 to c23cf16407ef200da446d81fb990abbe429fd378
• 00:01 logmsgbot: ori Synchronized wmf-config/StartProfiler.php: I341a50bef: 'Re-enable xhprof for single-request profiling' (duration: 00m 06s)

January 5

• 22:06 subbu: deployed parsoid version 0e2997d2
• 21:40 bblack: hard rebooted wtp1020, unresponsive in every way
• 19:14 hoo: Made the ruwikinews sites table entry on wikidatawiki use https URLs rather than protocol relative ones
• 18:41 YuviPanda: imported trusty pyparsing package into precise-wikimedia
• 17:44 logmsgbot: hoo Synchronized php-1.25wmf13/extensions/Wikidata/: Update Wikibase: Fix SpecialEntityData and enhance populateSitesTable (duration: 00m 24s)
• 17:43 logmsgbot: hoo Synchronized php-1.25wmf12/extensions/Wikidata/: Update Wikibase: Fix SpecialEntityData and enhance populateSitesTable (duration: 00m 14s)
• 17:22 logmsgbot: bd808 Synchronized wmf-config/logging-labs.php: Beta logging config change (5b628827) (duration: 00m 06s)
• 17:09 logmsgbot: bd808 Synchronized wmf-config/logging-labs.php: Beta logging config change (b47ee787) (duration: 00m 06s)
• 17:00 bd808: restarted logstash on logstash1001 to see if that will make syslog events come back
• 16:58 bd808: syslog events not being recorded in logstash as expected (apache2, hhvm)
• 16:21 logmsgbot: manybubbles Synchronized php-1.25wmf13/extensions/VisualEditor/: SWAT fix switching between wikitext and VE on mobile (duration: 00m 14s)
• 16:14 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT disable creating books in the wikipedia namespace AND shuffle some upload permissions on kowiki (duration: 00m 05s)
• 16:12 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT disable creating books in the wikipedia namespace (duration: 00m 06s)
• 16:03 logmsgbot: manybubbles Synchronized wmf-config/Wikibase.php: SWAT Display links to Wikidata in the other project sidebar (duration: 00m 06s)
• 10:40 springle: xtrabackup clone: db1037 to db1061, db1039 to db1062
• 10:23 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1061 db1062 (duration: 00m 06s)
• 10:14 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: pool db1057, warm up (duration: 00m 07s)

January 4

• 17:35 springle: xtrabackup clone db1061 to db1057
• 16:49 springle: restarted zuul
• 15:59 springle: upgrade db1057 trusty
• 15:23 springle: limiting exim/otrs concurrent connections on m2-master to 250
• 14:29 springle: xtrabackup clone db1020 to db2011
• 13:49 springle: dbproxy1002 failed m2-master traffic over to m2-slave. services up. investigating cause

January 3

• 23:23 subbu: Try #2: hotfix synced to parsoid cores (to return 500 for urwiki:نام_مقامات_اے); git sha 85d8818ec1b692aaab440630a119c539d63d5ca5
• 22:38 YuviPanda: restarted parsoid on wtp1010
• 22:38 YuviPanda: restarted parsoid on wtp1006
• 22:37 YuviPanda: restarted parsoid on wtp1004
• 22:29 subbu: hotfix synced to parsoid cores (to return 500 for urwiki:نام_مقامات_اے); restart coming next
• 22:15 YuviPanda: restarted parsoid on wtp* hosts agian
• 21:19 YuviPanda: restarting parsoid on wtp* hosts again
• 20:46 YuviPanda: restarting parsoid on wtp* again
• 20:29 YuviPanda: manually restarted parsoid on wtp1012
• 20:12 YuviPanda: restarting parsoid on all wtp* hosts
• 20:06 YuviPanda: restarting parsoid on wtp1008
• 17:13 _joe_: restarting parsoid across the cluster

January 2

• 21:19 qchris: Ran kafka leader re-election to bring analytics1021 back into the set of leaders
• 11:48 godog: reboot es2004, debugging gmond stuck on start/stop
• 04:59 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1061, warm up (duration: 00m 06s)
• 03:29 springle: clone and deploy es2002 es2003 es2004

January 1

• 15:19 springle: upgrade db1061 trusty

2000s

• Archive 1: 2004 Jun - 2004 Sep
• Archive 2: 2004 Oct - 2004 Nov
• Archive 3: 2004 Dec - 2005 Mar
• Archive 4: 2005 Apr - 2005 Jul
• Archive 5: 2005 Aug - 2005 Oct, with revision history 2004-06-23 to 2005-11-25
• Archive 6: 2005 Nov - 2006 Feb
• Archive 7: 2006 Mar - 2006 Jun
• Archive 8: 2006 Jul - 2006 Sep
• Archive 9: 2006 Oct - 2007 Jan, with revision history 2005-11-25 to 2007-02-21
• Archive 10: 2007 Feb - 2007 Jun
• Archive 11: 2007 Jul - 2007 Dec
• Archive 12: 2008 Jan - 2008 Jul
• Archive 12a: 2008 Aug
• Archive 12b: 2008 Sept
• Archive 13: 2008 Oct - 2009 Jun
• Archive 14: 2009 Jun - 2009 Dec

2010s

• Archive 15: 2010 Jan - 2010 Jun
• Archive 16: 2010 Jul - 2010 Oct
• Archive 17: 2010 Nov - 2010 Dec
• Archive 18: 2011 Jan - 2011 Jun
• Archive 19: 2011 Jul - 2011 Dec
• Archive 20: 2011 Dec - 2012 Jun, with revision history 2007-02-21 to 2012-03-27
• Archive 21: 2012 Jul - 2013 Jan
• Archive 22: 2013 Jan - 2013 Jul
• Archive 23: 2013 Aug - 2013 Dec
• Archive 24: 2014 Jan - 2014 Mar
• Archive 25: 2014 April - 2014 September
• Archive 26: 2014 October - 2014 December
• Archive 27: 2015 January - 2015 July
• Archive 28: 2015 August - 2015 December
• Archive 29: 2016 January - 2016 May
• Archive 30: 2016 June - 2016 August
• Archive 31: 2016 September - 2016 December
• Archive 32: 2017 January - 2017 July
• Archive 33: 2017 August - 2017 December
• Archive 34: 2018 January - 2018 April
• Archive 35: 2018 May - 2018 August
• Archive 36: 2018 September - 2018 December
• Archive 37: 2019 January - 2019 April
• Archive 38: 2019 May - 2019 August
• Archive 39: 2019 September - 2019 December

2020s

• Archive 40: 2020 January - 2020 April
• Archive 41: 2020 May - 2020 July
• Archive 42: 2020 August - 2020 November
• Archive 43: 2020 December
• Archive 44: 2021 January - 2021 April
• Archive 45: 2021 May - 2021 July
• Archive 46: 2021 August - 2021 October
• Archive 47: 2021 November - 2021 December
• Archive 48: 2022 January
• Archive 49: 2022 February
• Archive 50: 2022 March
• Archive 51: 2022 April 1-15
• Archive 52: 2022 April 16-30
• Archive 53: 2022 May
• Archive 54: 2022 June
• Archive 55: 2022 July
• Archive 56: 2022 August
• Archive 57: 2022 September
• Archive 58: 2022 October
• Archive 59: 2022 November 1-15
• Archive 60: 2022 November 16-30
• Archive 61: 2022 December
• Archive 62: 2023 January
• Archive 63: 2023 February
• Archive 64: 2023 March
• Archive 65: 2023 April
• Archive 66: 2023 May
• Archive 67: 2023 June
• Archive 68: 2023 July
• Archive 69: 2023 August 1-15
• Archive 70: 2023 August 16-31
• Archive 71: 2023 September
• Archive 72: 2023 October
• Archive 73: 2023 November
• Archive 74: 2023 December
• Archive 75: 2024 January
• Archive 76: 2024 February
• Archive 77: 2024 March
• Archive 78: 2024 April
• Archive 79: 2024 May 1-15
• Archive 80: 2024 May 16-31
• Archive 81: 2024 June 1-15
• Archive 82: 2024 June 16-30
• Archive 83: 2024 July
• Archive 84: 2024 August
• Archive 85: 2024 September
• Archive 86: 2024 October
• Archive 87: 2024 November