Server Admin Log

March 1

• 06:23 andrewbogott: logging a test to test the logging
• 02:17 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Mar 1 02:16:24 UTC 2015 (duration 16m 23s)
• 02:06 logmsgbot: LocalisationUpdate completed (1.25wmf19) at 2015-03-01 02:05:02+00:00
• 02:04 logmsgbot: l10nupdate Synchronized php-1.25wmf19/cache/l10n: (no message) (duration: 00m 01s)
• 02:04 logmsgbot: LocalisationUpdate completed (1.25wmf18) at 2015-03-01 02:03:30+00:00
• 02:03 logmsgbot: l10nupdate Synchronized php-1.25wmf18/cache/l10n: (no message) (duration: 00m 01s)
• 00:56 gwicke: stopped cassandra on cerium and praseodymium temporarily for testing

February 28

• 02:18 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Feb 28 02:17:47 UTC 2015 (duration 17m 46s)
• 02:06 logmsgbot: LocalisationUpdate completed (1.25wmf19) at 2015-02-28 02:05:51+00:00
• 02:05 logmsgbot: l10nupdate Synchronized php-1.25wmf19/cache/l10n: (no message) (duration: 00m 01s)
• 02:05 logmsgbot: LocalisationUpdate completed (1.25wmf18) at 2015-02-28 02:04:14+00:00
• 02:04 logmsgbot: l10nupdate Synchronized php-1.25wmf18/cache/l10n: (no message) (duration: 00m 01s)
• 00:16 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: If2704b0f7: Change metric prefix from 'mw' back to 'MediaWiki', for back-compat (duration: 00m 06s)

February 27

• 23:47 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I8fa0649ab: Set $wgUDPProfilerPort back to 8125 (duration: 00m 06s)
• 23:33 ori: pushing a config change to txstatsd on graphite1001, the service may complain briefly
• 22:19 logmsgbot: reedy Synchronized docroot and w: nooop for dbtree ( already reverted by prior deploy ) (duration: 00m 05s)
• 17:07 legoktm: running CentralAuth's migratePass0.php on all wikis
• 14:59 hoo: Ran mysql:wikiadmin@db1033 [metawiki]> UPDATE ipblocks SET ipb_deleted = 1 WHERE ipb_id = 16659; to actually suppress a suppressed name
• 08:41 andrewbogott: upgraded virt1012 to Trusty; starting all instances
• 06:25 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Feb 27 06:24:00 UTC 2015 (duration 23m 59s)
• 05:41 andrewbogott: upgrading virt1012 to Trusty because labs networking failed twice in two hours, and how could it be worse?
• 02:19 logmsgbot: LocalisationUpdate completed (1.25wmf19) at 2015-02-27 02:18:19+00:00
• 02:18 logmsgbot: l10nupdate Synchronized php-1.25wmf19/cache/l10n: (no message) (duration: 00m 01s)
• 02:17 logmsgbot: LocalisationUpdate completed (1.25wmf18) at 2015-02-27 02:16:14+00:00
• 02:16 logmsgbot: l10nupdate Synchronized php-1.25wmf18/cache/l10n: (no message) (duration: 00m 02s)
• 01:40 springle: switch db1046 to master of m4 (eventlogging). deployed dbproxy1004 with m4-master CNAME
• 01:18 logmsgbot: krenair Synchronized php-1.25wmf19/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.Target.js: https://gerrit.wikimedia.org/r/#/c/193313/ (duration: 00m 06s)
• 00:58 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/193273/ and https://gerrit.wikimedia.org/r/#/c/193274/ (duration: 00m 06s)
• 00:48 logmsgbot: krenair Synchronized wmf-config: touch (duration: 00m 06s)
• 00:47 logmsgbot: krenair Synchronized wmf-config: https://gerrit.wikimedia.org/r/193308 (duration: 00m 10s)
• 00:43 logmsgbot: krenair Synchronized php-1.25wmf19/skins/Vector/skinStyles/mediawiki.sectionAnchor.less: https://gerrit.wikimedia.org/r/#/c/193310/ (duration: 00m 05s)
• 00:15 logmsgbot: krenair Synchronized wmf-config: touched initialsettings (duration: 00m 07s)
• 00:09 logmsgbot: krenair Synchronized wmf-config: rv (duration: 00m 06s)
• 00:07 logmsgbot: krenair Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/193283/ (duration: 00m 06s)

February 26

• 22:59 Krinkle: git-deploy: Deploying integration/slave-scripts b532a9a..05a5593
• 21:55 gwicke: disabled puppet on cassandra test hosts cerium and praseodymium as well (in addition to xenon) to manually fix incompatible puppet config & re-initialize cluster after cluster name change; see https://phabricator.wikimedia.org/T90955 for upgrade to jessie
• 21:00 ^d: mw1161 is complaining about permissions on setting mtime during rsync
• 20:58 logmsgbot: demon Synchronized multiversion/MWMultiVersion.php: moar debug (duration: 00m 06s)
• 20:40 gwicke: issue with cassandra test cluster is actually that it's still running cassandra 2.1.2, which is incompatible with the current puppet config; should probably update the test cluster to jessie soon
• 20:38 gwicke: cassandra on test cluster seems to be broken, investigating
• 20:16 gwicke: disabled puppet on xenon to test bulk db creation with restbase
• 20:03 logmsgbot: demon Synchronized multiversion/MWMultiVersion.php: debuggg (duration: 00m 06s)
• 17:34 logmsgbot: kartik Finished scap: Update ContentTranslation (duration: 15m 58s)
• 17:18 logmsgbot: kartik Started scap: Update ContentTranslation
• 17:03 logmsgbot: brion Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 05s)
• 17:03 logmsgbot: demon Synchronized README: look ma, no key forwarding (duration: 00m 05s)
• 16:55 kart_: Updated cxserver to 4e09ee8
• 16:36 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/193112/ (duration: 00m 07s)
• 16:33 Krenair: ran sql.php --wiki=ruwiki php-1.25wmf18/extensions/EducationProgram/sql/EducationProgram.sql
• 16:30 logmsgbot: krenair Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/193110/ (duration: 00m 08s)
• 16:25 Krenair: Running updateCollation.php on hsbwiki
• 16:23 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/192803 (duration: 00m 07s)
• 16:17 logmsgbot: krenair Synchronized php-1.25wmf19/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.Target.js: https://gerrit.wikimedia.org/r/#/c/193024/ (duration: 00m 05s)
• 16:06 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/192764/ (duration: 00m 07s)
• 15:57 ottomata: restarted resourcemanager on analytics1001 to load new fairscheduler settings
• 15:14 logmsgbot: demon Synchronized php-1.25wmf18/extensions/RestBaseUpdateJobs: (no message) (duration: 00m 06s)
• 06:22 Tim: on mw1088 restarting hhvm
• 05:52 springle: pre-empt m3/m4 shard split and reclaim disk space on db2011
• 02:42 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Feb 26 02:41:08 UTC 2015 (duration 41m 7s)
• 02:22 logmsgbot: LocalisationUpdate completed (1.25wmf19) at 2015-02-26 02:21:38+00:00
• 02:21 logmsgbot: l10nupdate Synchronized php-1.25wmf19/cache/l10n: (no message) (duration: 00m 01s)
• 02:10 logmsgbot: LocalisationUpdate completed (1.25wmf18) at 2015-02-26 02:09:28+00:00
• 02:09 logmsgbot: l10nupdate Synchronized php-1.25wmf18/cache/l10n: (no message) (duration: 00m 02s)
• 01:18 logmsgbot: maxsem Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/193018/ (duration: 00m 07s)
• 01:13 logmsgbot: hoo Synchronized php-1.25wmf19/extensions/Wikidata/: Update Wikidata to fix EntityViewPlaceholderExpander (duration: 00m 12s)
• 00:59 logmsgbot: maxsem Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/192987/ (duration: 00m 06s)
• 00:57 logmsgbot: maxsem Synchronized php-1.25wmf18/extensions/WikiGrok/: (no message) (duration: 00m 07s)
• 00:53 logmsgbot: maxsem Synchronized php-1.25wmf19/extensions/WikiGrok/: (no message) (duration: 00m 07s)
• 00:32 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/192709/ (duration: 00m 05s)
• 00:28 logmsgbot: ebernhardson Synchronized php-1.25wmf18/extensions/Flow: Bump flow submodule in 1.25wmf18 for infinite scroll fix (duration: 00m 09s)

February 25

• 23:33 logmsgbot: maxsem Synchronized docroot and w: (no message) (duration: 00m 06s)
• 23:22 andrewbogott: upgrading virt1005 to trusty
• 22:53 hoo: Ran rebuildEntityPerPage.php on wikidatawiki to clean up after wikigrok database mess
• 22:51 andrewbogott: rebooting virt1005 in anticipation of an exprimental upgrade to Trusty. (There are no VMs on virt1005 other than a testing host)
• 22:44 twentyafterfour: Done deploying - uploaded release notes for 1.25wmf19
• 22:36 logmsgbot: twentyafterfour Purged l10n cache for 1.25wmf17
• 22:32 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf19
• 22:26 robh: finished with my deployment window for redirects, tested and is now live with no issues (so far)
• 22:25 logmsgbot: twentyafterfour Synchronized ./wmf-config/InitialiseSettings.php: mw1204 still logging errors (duration: 00m 05s)
• 22:22 logmsgbot: twentyafterfour Synchronized ./wmf-config/InitialiseSettings.php: flooding logs (duration: 00m 07s)
• 22:19 logmsgbot: twentyafterfour Synchronized ./wmf-config/InitialiseSettings.php: (no message) (duration: 00m 05s)
• 22:16 robh: localtesting of change on mw1001 shows no issues, so pushing out to rest of apaches
• 22:13 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.25wmf18
• 22:08 Reedy: morebots is dead
• 22:03 robh: disabling puppet on all mw systems for redirects update
• 21:54 logmsgbot: twentyafterfour Finished scap: testwiki to php-1.25wmf19 and rebuild l10n cache (duration: 30m 20s)
• 21:23 logmsgbot: twentyafterfour Started scap: testwiki to php-1.25wmf19 and rebuild l10n cache
• 21:12 arlolra: updated Parsoid to version 5a3aaf712c334190a97a1d224a9efc0fb340f6af
• 20:43 andrewbogott: restarted nova-compute on virt1002
• 19:47 gwicke: restarted restbase1005 with new GC settings
• 19:17 gwicke: restarted cassandra on restbase1003 with new GC settings from puppet
• 16:59 logmsgbot: krenair Synchronized php-1.25wmf18/extensions/VisualEditor/modules/ve-mw/ui/dialogs/ve.ui.MWTemplateDialog.js: * https://gerrit.wikimedia.org/r/#/c/192750/ (duration: 00m 06s)
• 16:05 logmsgbot: demon Synchronized commonsuploads.dblist: (no message) (duration: 00m 07s)
• 16:05 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 05s)
• 15:52 cmjohnson1: replacing PEM2 cr1-eqiad
• 15:49 cmjohnson1: replacing PEM1 on cr1-eqiad
• 15:41 mutante: welcome legoktm as a contint admin
• 10:00 _joe_: restarted hhvm on mw1229, stuck in __lll_lock_wait from HPHP::hphp_session_init
• 07:04 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Feb 25 07:03:41 UTC 2015 (duration 3m 40s)
• 06:22 greg-g: 06:20 < twentyaft> that log was bogus, just me testing but not actually syncing
• 06:17 logmsgbot: twentyafterfour Synchronized php-1.25wmf18/cache/l10n: (no message) (duration: 00m 02s)
• 04:38 springle: s/db1001/dbproxy1001/g on zirconium drupal contacts. seems unpuppetized
• 02:35 logmsgbot: LocalisationUpdate completed (1.25wmf18) at 2015-02-25 02:34:23+00:00
• 02:34 logmsgbot: l10nupdate Synchronized php-1.25wmf18/cache/l10n: (no message) (duration: 00m 03s)
• 02:22 logmsgbot: LocalisationUpdate completed (1.25wmf17) at 2015-02-25 02:21:53+00:00
• 02:21 logmsgbot: l10nupdate Synchronized php-1.25wmf17/cache/l10n: (no message) (duration: 00m 02s)

February 24

• 23:37 springle: killed runaway RecentChangesUpdateJob::purgeExpiredRows transactions from jobrunners. db1033 db1038 db1040 db1052 db1058
• 22:58 logmsgbot: aaron Synchronized php-1.25wmf17/includes/jobqueue/jobs/RecentChangesUpdateJob.php: 6f6d7e57be0ccff8ed2473b7d250e77703c7a6dd (duration: 00m 09s)
• 22:43 logmsgbot: aude Synchronized docroot/mediawiki/xml/: Add sitelist export-import docs (duration: 00m 07s)
• 22:28 logmsgbot: aude Synchronized wikidataclient.dblist: Enable Wikibase Client on Wikibooks (duration: 00m 06s)
• 22:27 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Wikibase config for Wikibooks (duration: 00m 06s)
• 22:25 logmsgbot: aude Synchronized wmf-config/Wikibase.php: Bump cache epoch for Wikidata (duration: 00m 06s)
• 22:20 logmsgbot: aude Synchronized wmf-config/Wikibase.php: Enable Wikibooks sitelinks on Wikidata (duration: 00m 06s)
• 22:08 logmsgbot: aude Finished scap: Updates for enabling Wikibase on Wikibooks (duration: 16m 03s)
• 21:52 logmsgbot: aude Started scap: Updates for enabling Wikibase on Wikibooks
• 21:42 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: sync-wikiversions group1 to 1.25wmf18
• 21:25 logmsgbot: twentyafterfour Synchronized php-1.25wmf18/extensions/Popups: hotfix for Popups (see https://gerrit.wikimedia.org/r/#/c/192465/) (duration: 00m 06s)
• 21:24 ottomata: increased Hadoop nodemanager cpu-vcores to facter $processcount - 1, this should increase hadoop cluster utilization
• 20:31 paravoid: restarting cr1-eqiad/re1 chassis-control; should not be traffic-disrupting
• 20:05 qchris: Updated gerrit plugin its-phabricator-from-bugzilla to 03b936b2cd8fa6adfdbee0ef68eb4b31944936c2
• 20:05 qchris: Updated gerrit plugin its-phabricator to 25a34d7564cffb90a87110a971782195ba2db467
• 17:55 Coren: Shutting down labstore1001 - planned outage for expansion
• 14:59 godog: rolling restart cassandra to pick up metrics configuration
• 07:55 andrewbogott: ‘nova reset-state —active’ and ‘nova reboot’ for EVERY instance on virt1012
• 07:13 andrewbogott: suspending all instances on virt1012
• 06:40 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Feb 24 06:39:18 UTC 2015 (duration 39m 17s)
• 06:29 YuviPanda: stopped nova-compute on virt1005
• 06:28 YuviPanda: starting nova-compute on virt1005
• 04:07 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: raise db1066 load (duration: 00m 07s)
• 02:50 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1066, warm up (duration: 00m 06s)
• 02:19 logmsgbot: LocalisationUpdate completed (1.25wmf18) at 2015-02-24 02:18:03+00:00
• 02:17 logmsgbot: l10nupdate Synchronized php-1.25wmf18/cache/l10n: (no message) (duration: 00m 01s)
• 02:17 logmsgbot: LocalisationUpdate completed (1.25wmf17) at 2015-02-24 02:16:30+00:00
• 02:16 logmsgbot: l10nupdate Synchronized php-1.25wmf17/cache/l10n: (no message) (duration: 00m 02s)
• 01:52 logmsgbot: ori Synchronized php-1.25wmf17/extensions/MobileFrontend/includes/modules/MobileUserModule.php: Reverting live-hack (duration: 00m 07s)
• 01:47 logmsgbot: ori Synchronized php-1.25wmf17/extensions/MobileFrontend/includes/modules/MobileUserModule.php: Testing a theory for T90411 with a live-hack to MobileFrontend. Will revert momentarily. (duration: 00m 07s)
• 00:56 Tim: on osmium, removing the packages I just installed since I will do it in a chroot instead
• 00:51 logmsgbot: twentyafterfour Synchronized php-1.25wmf18/cache/l10n: (no message) (duration: 00m 03s)
• 00:43 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 07s)
• 00:10 logmsgbot: demon Synchronized php-1.25wmf18/extensions/MultimediaViewer: (no message) (duration: 00m 09s)
• 00:10 logmsgbot: demon Synchronized php-1.25wmf17/extensions/MultimediaViewer: (no message) (duration: 00m 06s)

February 23

• 23:28 Tim: on osmium installing packages necessary for building hhvm
• 21:06 subbu: deployed parsoid version d9ac8c21
• 20:43 awight: update crm from f594a66694d52af1c604b1813ac94e9592b6c81e to 3c002f32e04652ae56a4fe791bc6158ab981ed8d
• 18:19 ^d: created education program tables for hewiktionary
• 18:18 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: T89393 (duration: 00m 06s)
• 17:45 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: T90340 (duration: 00m 05s)
• 17:40 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: T89346 (duration: 00m 07s)
• 17:31 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: T88591 (duration: 00m 06s)
• 17:27 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: T89147 (duration: 00m 05s)
• 17:11 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 05s)
• 16:33 paravoid: updating jessie d-i image to currently nightly
• 16:26 _joe_: depooling mw1062 for testing for T86652
• 09:26 godog: reboot ms-be1009, xfs hosed
• 05:41 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1066 (duration: 00m 06s)
• 02:16 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Feb 23 02:15:36 UTC 2015 (duration 15m 35s)
• 02:05 logmsgbot: LocalisationUpdate completed (1.25wmf18) at 2015-02-23 02:04:00+00:00
• 02:03 logmsgbot: l10nupdate Synchronized php-1.25wmf18/cache/l10n: (no message) (duration: 00m 01s)
• 02:03 logmsgbot: LocalisationUpdate completed (1.25wmf17) at 2015-02-23 02:02:28+00:00
• 02:02 logmsgbot: l10nupdate Synchronized php-1.25wmf17/cache/l10n: (no message) (duration: 00m 01s)

February 22

• 22:50 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: db1065 raise load (duration: 00m 07s)
• 22:37 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1065, warm up (duration: 00m 05s)
• 22:08 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1065 (duration: 00m 06s)
• 10:33 godog: reboot ms-be1007, xfs hosed
• 10:23 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1065 (duration: 00m 05s)
• 02:17 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Feb 22 02:16:04 UTC 2015 (duration 16m 3s)
• 02:05 logmsgbot: LocalisationUpdate completed (1.25wmf18) at 2015-02-22 02:04:55+00:00
• 02:04 logmsgbot: l10nupdate Synchronized php-1.25wmf18/cache/l10n: (no message) (duration: 00m 01s)
• 02:04 logmsgbot: LocalisationUpdate completed (1.25wmf17) at 2015-02-22 02:03:23+00:00
• 02:03 logmsgbot: l10nupdate Synchronized php-1.25wmf17/cache/l10n: (no message) (duration: 00m 01s)
• 01:49 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1065 (duration: 00m 06s)

February 21

• 06:51 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Feb 21 06:50:49 UTC 2015 (duration 50m 48s)
• 02:18 logmsgbot: LocalisationUpdate completed (1.25wmf18) at 2015-02-21 02:17:15+00:00
• 02:17 logmsgbot: l10nupdate Synchronized php-1.25wmf18/cache/l10n: (no message) (duration: 00m 01s)
• 02:16 logmsgbot: LocalisationUpdate completed (1.25wmf17) at 2015-02-21 02:15:31+00:00
• 02:15 logmsgbot: l10nupdate Synchronized php-1.25wmf17/cache/l10n: (no message) (duration: 00m 02s)

February 20

• 23:12 logmsgbot: ori Synchronized php-1.25wmf17/extensions/VisualEditor: f14dc93302: Update VisualEditor for cherry-picks (duration: 00m 06s)
• 23:11 logmsgbot: ori Synchronized php-1.25wmf18/extensions/VisualEditor: 5c4457a555: Update VisualEditor for cherry-picks (duration: 00m 05s)
• 19:20 awight|specter: enabled scheduled reminders in production CiviCRM
• 19:05 robh: neon runs puppet fine, back to full service
• 18:54 robh: i broke puppet on neon, workign to fix
• 18:39 robh: killing icinga-admin.w.o url support per T90002
• 18:39 robh: killing icinga-admin.w.o url support
• 18:03 ori: added mobrovac to mediawiki and services gerrit groups
• 16:25 awight: updated crm from a1e604b93342f5555427eaeb81092bfa431ff093 to f594a66694d52af1c604b1813ac94e9592b6c81e
• 15:39 cmjohnson1: virt1002 removing disk 0 which should be /dev/sda
• 10:02 godog: reboot restbase1006 after disk reseat
• 09:16 logmsgbot: twentyafterfour Synchronized wmf-config/CommonSettings.php: wgTranslateBlacklist (duration: 00m 07s)
• 06:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Feb 20 06:06:39 UTC 2015 (duration 6m 38s)
• 02:25 logmsgbot: LocalisationUpdate completed (1.25wmf18) at 2015-02-20 02:24:52+00:00
• 02:24 logmsgbot: l10nupdate Synchronized php-1.25wmf18/cache/l10n: (no message) (duration: 00m 01s)
• 02:23 logmsgbot: LocalisationUpdate completed (1.25wmf17) at 2015-02-20 02:22:50+00:00
• 02:22 logmsgbot: l10nupdate Synchronized php-1.25wmf17/cache/l10n: (no message) (duration: 00m 01s)
• 00:33 logmsgbot: maxsem Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 05s)
• 00:33 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings.php: Pull WikiGrok from wikidata for now (duration: 00m 05s)
• 00:28 logmsgbot: catrope Synchronized php-1.25wmf18/extensions/ZeroBanner: SWAT (duration: 00m 06s)
• 00:27 logmsgbot: catrope Synchronized php-1.25wmf17/extensions/ZeroBanner: SWAT (duration: 00m 06s)
• 00:25 logmsgbot: tstarling Synchronized php-1.25wmf17/extensions/Collection/Collection.body.php: (no message) (duration: 00m 07s)
• 00:21 logmsgbot: tstarling Synchronized php-1.25wmf17/extensions/Collection/Collection.body.php: (no message) (duration: 00m 05s)
• 00:07 logmsgbot: catrope Synchronized wmf-config/mobile.php: SWAT (duration: 00m 06s)
• 00:07 logmsgbot: catrope Synchronized wmf-config/InitialiseSettings.php: SWAT (duration: 00m 08s)

February 19

• 23:06 logmsgbot: ori Synchronized php-1.25wmf17/extensions/WikimediaEvents: (no message) (duration: 00m 06s)
• 23:06 logmsgbot: ori Synchronized php-1.25wmf17/extensions/VisualEditor: (no message) (duration: 00m 06s)
• 23:05 logmsgbot: ori Synchronized php-1.25wmf18/extensions/WikimediaEvents: (no message) (duration: 00m 07s)
• 23:05 logmsgbot: ori Synchronized php-1.25wmf18/extensions/VisualEditor: (no message) (duration: 00m 06s)
• 21:57 logmsgbot: hoo Synchronized php-1.25wmf18/extensions/Wikidata/: Update Wikibase to fix langlink updates in the client API et al (duration: 00m 12s)
• 21:57 logmsgbot: hoo Synchronized php-1.25wmf17/extensions/Wikidata/: Update Wikibase to fix langlink updates in the client API et al (duration: 00m 14s)
• 21:28 gwicke: restbase now up on all live (3 of 6) prod nodes
• 21:20 gwicke: cleanly re-initialized prod cassandra cluster after puppet run; picked up local dc from property file
• 20:49 chasemp: restart ntp on mw1009
• 20:15 mutante: readding mw1062 to puppet, signing new cert and salt-key
• 19:54 mutante: reinstalling mw1062 after disk has been replaced
• 19:28 mutante: ran puppet on ruthenium
• 17:46 logmsgbot: demon Synchronized php-1.25wmf17/extensions/DoubleWiki/DoubleWiki_body.php: shut up warnings finally (duration: 00m 05s)
• 17:42 logmsgbot: oblivian Synchronized wmf-config/session.php: mc1011-12 IP change (duration: 00m 05s)
• 17:35 logmsgbot: kartik Finished scap: ContentTranslation update (duration: 25m 31s)
• 17:25 ^d: jenkins stuck communicating to beta, restarting
• 17:10 logmsgbot: kartik Started scap: ContentTranslation update
• 16:53 _joe_: shutting down mc1009 and mc1010
• 16:39 logmsgbot: oblivian Synchronized wmf-config/session.php: mc1009-10 IP change (duration: 00m 05s)
• 16:23 kart_: Updated cxserver to 395be27
• 16:16 kart_: started cxserver deployment
• 16:01 logmsgbot: demon Synchronized wmf-config/CommonSettings-labs.php: (no message) (duration: 00m 07s)
• 15:49 logmsgbot: oblivian Synchronized wmf-config/session.php: mc1007-8 IP change (duration: 00m 06s)
• 15:21 _joe_: restarting hhvm on mw1103, mw1078,mw1032 - TC full as well.
• 15:17 _joe_: restarting hhvm on mw1027, TC full
• 15:11 _joe_: powering down mc1007,mc1008
• 09:58 AaronS: Deleted more bogus GlobalUserPage purge job queues
• 06:11 springle: bacula-director restart to pick up m1-master CNAME
• 06:07 springle: ran RT update-rt-siteconfig + apache restart to pick up m1-master CNAME
• 05:55 springle: etherpad-lite restart to pick up m1-master CNAME
• 05:49 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Feb 19 05:48:40 UTC 2015 (duration 48m 39s)
• 04:32 ori: txstatsd on graphite1001: disabled profiling and returned service to normal state
• 04:23 ori: restarting txstatsd on graphite1001 with --profile; will disable profiling in a few minutes.
• 02:41 mutante: restarted hhvm on mw1141 (locked up, T89912?)
• 02:36 logmsgbot: LocalisationUpdate completed (1.25wmf18) at 2015-02-19 02:35:41+00:00
• 02:35 logmsgbot: l10nupdate Synchronized php-1.25wmf18/cache/l10n: (no message) (duration: 00m 01s)
• 02:21 logmsgbot: LocalisationUpdate completed (1.25wmf17) at 2015-02-19 02:20:01+00:00
• 02:19 logmsgbot: l10nupdate Synchronized php-1.25wmf17/cache/l10n: (no message) (duration: 00m 01s)
• 02:18 mutante: restbase1004 - starting restbase service, running puppet
• 02:11 mutante: restbase1004/1005 systemctl daemon-reload to run systemd-sysv-generator to make it create missing unit for restbase and unbreak puppet running the service
• 01:12 logmsgbot: ejegg Synchronized wmf-config/CommonSettings.php: Use URLs without mobile redirects for CentralNotice (duration: 00m 07s)
• 00:54 logmsgbot: demon Finished scap: global user page extension-list fix + l10n rebuild (duration: 15m 21s)
• 00:39 AaronS: Deleted labswiki redis jobs (labswiki uses the db queue) for GlobalUserPage and flushed the queue aggregator
• 00:38 logmsgbot: demon Started scap: global user page extension-list fix + l10n rebuild
• 00:33 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 06s)
• 00:26 logmsgbot: demon Synchronized php-1.25wmf17/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.Target.js: (no message) (duration: 00m 06s)
• 00:16 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: RL image debug logs (duration: 00m 07s)
• 00:14 logmsgbot: demon Synchronized php-1.25wmf17/includes/skins/SkinTemplate.php: (no message) (duration: 00m 05s)
• 00:14 logmsgbot: demon Synchronized php-1.25wmf17/includes/skins/Skin.php: (no message) (duration: 00m 05s)
• 00:12 logmsgbot: demon Synchronized php-1.25wmf18/includes/resourceloader/ResourceLoaderImage.php: fix up svg handling in RL (duration: 00m 07s)
• 00:12 logmsgbot: demon Synchronized php-1.25wmf17/includes/resourceloader/ResourceLoaderImage.php: fix up svg handling in RL (duration: 00m 07s)
• 00:09 logmsgbot: demon Synchronized wmf-config/InitialiseSettings-labs.php: no-op, for completeness (duration: 00m 05s)
• 00:04 logmsgbot: aaron Synchronized php-1.25wmf16/includes/db/LoadBalancer.php: 9dc01855bca9ba322f6cb15092b29c654d74cecc (duration: 00m 05s)

February 18

• 23:51 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: rm no-op profile calls (duration: 00m 06s)
• 23:48 logmsgbot: aaron Synchronized php-1.25wmf17/includes/db/LoadBalancer.php: 42a56404328547a0b8bd07f001b1c4dff67b3498 (duration: 00m 05s)
• 23:46 logmsgbot: legoktm Synchronized wmf-config: Enable GlobalUserPage extension on all public, CentralAuth wikis (duration: 00m 05s)
• 23:39 twentyafterfour: fixed symlinks. uploaded release notes. deployment finished 1.5 hours behind schedule
• 23:25 logmsgbot: twentyafterfour Purged l10n cache for 1.25wmf16
• 23:23 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf18
• 23:23 ori: HHVM on mw1141 locked up (threads stuck in __lll_lock_wait). Depooling for further investigation.
• 23:17 logmsgbot: twentyafterfour Finished scap: testwiki to php-1.25wmf18 and rebuild l10n cache (duration: 42m 58s)
• 22:34 logmsgbot: twentyafterfour Started scap: testwiki to php-1.25wmf18 and rebuild l10n cache
• 22:30 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: rollback group0 to 1.25wmf17
• 22:28 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf18
• 22:24 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.25wmf17
• 21:41 subbu: deployed parsoid version 17f68256
• 19:24 bd808|LUNCH: pruned stale members from trebuchet minions set for scap/scap: redis-cli srem "deploy:scap/scap:minions" fenari.wikimedia.org virt0.wikimedia.org nickel.wikimedia.org searchidx1001.eqiad.wmnet
• 19:01 godog: restart txstatsd on graphite1001 to flush old metrics
• 18:40 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Id215ff962: Change $wgUDPProfilerPort to 8135. (duration: 00m 05s)
• 18:14 logmsgbot: oblivian Synchronized wmf-config/session.php: mc1013 IP change (duration: 00m 05s)
• 18:11 logmsgbot: oblivian Synchronized wmf-config/session.php: mc1013 IP change (duration: 00m 07s)
• 17:58 _joe_: fixed scap on mw1158, moving /srv/deployment/scap away made puppet perform the redeploy
• 17:56 _joe_: fixed scap on mw1154, moving /srv/deployment/scap away made puppet perform the redeploy
• 17:51 bd808: fixing scap on mw1158 and mw1154 will take a root to fix bad trebuchet git clones -- cd /src/deployment/scap; sudo mv scap scap-broken; sudo salt-call deploy.fetch 'scap/scap'; sudo salt-call deploy.checkout 'scap/scap'
• 17:22 _joe_: shutting down mc1014, moving to a different rack
• 17:19 _joe_: mw1158 and mw1154 report broken python imports during scap
• 17:18 logmsgbot: oblivian Synchronized wmf-config/session.php: mc1014 IP change (duration: 00m 07s)
• 17:18 logmsgbot: oblivian Synchronized wmf-config/session.php: mc1014 IP change (duration: 00m 07s)
• 16:56 logmsgbot: demon Synchronized README: testing scap update (duration: 00m 07s)
• 16:50 _joe_: moving mc1014 to a new row
• 16:44 logmsgbot: oblivian Synchronized wmf-config/session.php: mc1015 IP change (duration: 00m 05s)
• 16:16 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: translationadmin for sysops on mw.org (duration: 00m 08s)
• 16:03 logmsgbot: demon Synchronized wmf-config/CommonSettings-labs.php: for completeness, no-op (duration: 00m 07s)
• 15:56 logmsgbot: oblivian Synchronized wmf-config/session.php: mc1016 IP change (duration: 00m 07s)
• 15:25 chasemp: phabricator updated for T86772
• 15:11 _joe_: shutting down mc1016 for movement to a new row
• 14:34 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: remove dl() of php_utfnormal (duration: 00m 07s)
• 10:33 hoo: Manually switched wikidatawiki's sites table entry for ruwiki from protocol relative to https URIs
• 06:59 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1054 T89801 (duration: 00m 06s)
• 05:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Feb 18 05:06:53 UTC 2015 (duration 6m 52s)
• 02:39 logmsgbot: LocalisationUpdate completed (1.25wmf17) at 2015-02-18 02:38:47+00:00
• 02:38 logmsgbot: l10nupdate Synchronized php-1.25wmf17/cache/l10n: (no message) (duration: 00m 01s)
• 02:23 logmsgbot: LocalisationUpdate completed (1.25wmf16) at 2015-02-18 02:22:02+00:00
• 02:21 logmsgbot: l10nupdate Synchronized php-1.25wmf16/cache/l10n: (no message) (duration: 00m 02s)
• 02:07 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 05s)
• 01:58 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: reduce db1065 load (duration: 00m 05s)
• 00:56 logmsgbot: maxsem Synchronized php-1.25wmf17/extensions/MobileFrontend/: SWAT (duration: 00m 06s)
• 00:44 logmsgbot: maxsem Synchronized wmf-config/: https://gerrit.wikimedia.org/r/189863 - labs only (duration: 00m 06s)
• 00:27 logmsgbot: maxsem Synchronized php-1.25wmf16/extensions/WikiGrok/: https://gerrit.wikimedia.org/r/190562 (duration: 00m 06s)
• 00:26 logmsgbot: maxsem Synchronized php-1.25wmf17/extensions/WikiGrok/: https://gerrit.wikimedia.org/r/190562 (duration: 00m 07s)
• 00:22 logmsgbot: maxsem Synchronized wmf-config/mobile.php: https://gerrit.wikimedia.org/r/188731 (duration: 00m 05s)
• 00:21 springle: db1043 m3-master restart mysqld T89274
• 00:16 logmsgbot: maxsem Synchronized wmf-config/mobile.php: https://gerrit.wikimedia.org/r/187823 (duration: 00m 06s)
• 00:13 springle: db1048 m3-slave restart mysqld T89274
• 00:11 logmsgbot: maxsem Synchronized php-1.25wmf16/extensions/Echo/: SWAT (duration: 00m 06s)
• 00:05 logmsgbot: maxsem Synchronized php-1.25wmf17/extensions/Echo/: SWAT (duration: 00m 07s)

February 17

• 23:51 mutante: apt-get upgrade on gallium
• 22:37 csteipp: deploy fixes for T85850, T88310, T85855
• 22:13 ejegg: updated payments-wiki-staging from ce73ed11de9775a596c51acdc036503751961bc8 to cbaf66e7705789f37117ec6edc4d936c6174d511
• 21:42 hoo: Set email for dewiki account "Ar-ras" to the email of the commons account with the same name
• 20:41 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group1 to $VERSION
• 19:57 logmsgbot: ori Synchronized wmf-config/StartProfiler.php: I6fbd48e6b: Revert "Revert "Revert "Use ProfilerSectionOnly to handle DB/filebackend entries and the like""" (duration: 00m 05s)
• 19:15 logmsgbot: yurik scap failed: OSError [Errno 2] No such file or directory: '/var/lock/scap' (duration: 33m 42s)
• 18:52 andrewbogott: cold-migrating all instances from virt1005 to virt1012
• 18:41 logmsgbot: yurik Started scap: (no message)
• 18:23 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Ie5879ec6a: Set $wgUDPProfilerPort to 8125 (duration: 00m 07s)
• 18:21 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Icd6766440: Correct StatsFormatString so it emits valid statsd data (duration: 00m 07s)
• 18:02 andrewbogott: adding virt1012 to the nova virt pool
• 17:25 andrewbogott: powering down virt1005, waiting a few seconds, power on
• 17:14 logmsgbot: marktraceur Synchronized php-1.25wmf17/extensions/SecurePoll/includes/ballots/Ballot.php: [SWAT] [wmf17] Backport SecurePoll_BallotStatus fix (duration: 00m 05s)
• 17:05 logmsgbot: marktraceur Synchronized php-1.25wmf17/tests/phpunit/includes/StatusTest.php: [SWAT] [wmf17] Make sure Commons file deletion is still working later today (duration: 00m 06s)
• 17:04 logmsgbot: marktraceur Synchronized php-1.25wmf17/includes/Status.php: [SWAT] [wmf17] Make sure Commons file deletion is still working later today (duration: 00m 06s)
• 16:55 logmsgbot: marktraceur Synchronized php-1.25wmf17/includes/filerepo/FileRepo.php: [SWAT] [wmf17] Make sure Commons uploading is still working later today (duration: 00m 05s)
• 16:52 _joe_: upgrading testwiki to use www-data, may cause a brief downtime
• 16:46 logmsgbot: marktraceur Synchronized php-1.25wmf17/extensions/MultimediaViewer/resources/mmv/ui/: [SWAT] [wmf17] Media Viewer share/embed fix (duration: 00m 07s)
• 16:45 logmsgbot: marktraceur Synchronized php-1.25wmf16/extensions/MultimediaViewer/resources/mmv/ui/: [SWAT] [wmf16] Media Viewer share/embed fix (duration: 00m 05s)
• 16:35 logmsgbot: marktraceur Synchronized wmf-config/: [SWAT] [config] Set = true; on all wikis (duration: 00m 06s)
• 16:34 bd808: mw1062.eqiad.wmnet not accepting ssh login by bd808 (key refused)
• 16:33 logmsgbot: marktraceur Synchronized wmf-config/Wikibase.php: [SWAT] [config] Adjust , update property id blacklist (duration: 00m 05s)
• 16:31 bd808: mw1159.eqiad.wmnet has ancient scap version (2014-10-09)
• 16:27 logmsgbot: marktraceur Synchronized wmf-config/CommonSettings.php: [SWAT] [config] Update wgContentTranslationSiteTemplates (duration: 00m 05s)
• 16:19 logmsgbot: marktraceur Synchronized wmf-config/InitialiseSettings.php: [SWAT] [config] Enable Main namespace publishing for idwiki, ptwiki (duration: 00m 06s)
• 16:16 logmsgbot: marktraceur Synchronized wmf-config/logging.php: No-op test for bd808 (duration: 00m 05s)
• 16:13 bd808: updated scap to 54a2713 (www-data user)
• 16:12 bd808: hosts failing to fetch for scap trebuchet deploy: fenari.wikimedia.org, mw1062.eqiad.wmnet, nickel.wikimedia.org, searchidx1001.eqiad.wmnet, mw1222.eqiad.wmnet, virt0.wikimedia.org, mw1159.eqiad.wmnet
• 15:13 _joe_: disabled manually all crons in the 'apache' crontab on terbium
• 15:06 godog: restart pybal on lvs1003
• 15:00 godog: restart pybal on lvs1006
• 14:02 _joe_: updating the jobrunners to use www-data
• 11:24 _joe_: rolling transition of api appservers to www-data beginning as well
• 10:24 _joe_: converting all appservers to www-data
• 10:05 godog: testing cassandra-metrics on xenon
• 09:39 _joe_: repooling mw1029-1039
• 09:24 _joe_: depooling mw1029-1039
• 09:18 _joe_: repooling mw1019-28
• 09:06 godog: rolling restart of elastic1023 -> elastic1031
• 08:32 _joe_: depooling mw1019-28 for T78076
• 05:16 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Feb 17 05:15:45 UTC 2015 (duration 15m 44s)
• 03:56 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1065, warm up (duration: 00m 05s)
• 02:31 logmsgbot: LocalisationUpdate completed (1.25wmf17) at 2015-02-17 02:30:14+00:00
• 02:30 logmsgbot: l10nupdate Synchronized php-1.25wmf17/cache/l10n: (no message) (duration: 00m 01s)
• 02:21 logmsgbot: ori Synchronized wmf-config: Revert Ie91add33f: Temporarily log message key lookups on four app servers (duration: 00m 05s)
• 02:17 logmsgbot: LocalisationUpdate completed (1.25wmf16) at 2015-02-17 02:16:02+00:00
• 02:16 logmsgbot: l10nupdate Synchronized php-1.25wmf16/cache/l10n: (no message) (duration: 00m 03s)
• 01:51 logmsgbot: ori Synchronized wmf-config: Ie91add33f: Temporarily log message key lookups on four app servers (duration: 00m 05s)

February 16

• 22:34 logmsgbot: catrope Synchronized php-1.25wmf16/includes/MediaWiki.php: I34028206 (duration: 00m 05s)
• 22:33 ori: reloaded nginx on dumps with original config; re-enabled puppet.
• 22:04 ori: reloading nginx on dataset1001 for same
• 22:02 ori: disabled puppet on dataset1001 to experiment w/ https://gerrit.wikimedia.org/r/190940
• 21:50 logmsgbot: catrope Synchronized php-1.25wmf17/includes/MediaWiki.php: I34028206 (duration: 00m 06s)
• 21:12 logmsgbot: reedy Synchronized database lists: Update size related dblists (duration: 00m 06s)
• 21:09 subbu: updated Parsoid to version 86e76a30
• 16:13 logmsgbot: bd808 Synchronized wmf-config/logging-labs.php: Switch beta to syslog logging, try #2 (45d25e2) (duration: 00m 05s)
• 15:54 hoo: Updated Wikidata property suggester with data from today's dump
• 14:54 ottomata: shutting down hadoop cluster, starting upgrade to CDH 5.3.1
• 12:35 akosiaris: GIT_SSH=../../ssh git pull to update labs/private on deployment-salt
• 10:03 godog: resume elasticsearch rolling restart - elastic1012 -> elastic1022 in turn
• 08:02 _joe_: repooling mw1018
• 06:34 springle: messing with phab boolean fulltext syntax T89274
• 04:51 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Feb 16 04:50:10 UTC 2015 (duration 50m 9s)
• 04:20 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1065 (duration: 00m 06s)
• 02:29 logmsgbot: LocalisationUpdate completed (1.25wmf17) at 2015-02-16 02:28:39+00:00
• 02:28 logmsgbot: l10nupdate Synchronized php-1.25wmf17/cache/l10n: (no message) (duration: 00m 01s)
• 02:17 springle: db1046 restart, table maintenance
• 02:15 logmsgbot: LocalisationUpdate completed (1.25wmf16) at 2015-02-16 02:14:29+00:00
• 02:14 logmsgbot: l10nupdate Synchronized php-1.25wmf16/cache/l10n: (no message) (duration: 00m 02s)

February 15

• 02:13 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Feb 15 02:12:53 UTC 2015 (duration 12m 52s)
• 02:04 logmsgbot: LocalisationUpdate completed (1.25wmf17) at 2015-02-15 02:03:44+00:00
• 02:03 logmsgbot: l10nupdate Synchronized php-1.25wmf17/cache/l10n: (no message) (duration: 00m 01s)
• 02:03 logmsgbot: LocalisationUpdate completed (1.25wmf16) at 2015-02-15 02:02:13+00:00
• 02:02 logmsgbot: l10nupdate Synchronized php-1.25wmf16/cache/l10n: (no message) (duration: 00m 02s)

February 14

• 04:54 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Feb 14 04:53:02 UTC 2015 (duration 53m 1s)
• 02:32 logmsgbot: LocalisationUpdate completed (1.25wmf17) at 2015-02-14 02:31:02+00:00
• 02:30 logmsgbot: l10nupdate Synchronized php-1.25wmf17/cache/l10n: (no message) (duration: 00m 01s)
• 02:17 logmsgbot: LocalisationUpdate completed (1.25wmf16) at 2015-02-14 02:16:47+00:00
• 02:16 logmsgbot: l10nupdate Synchronized php-1.25wmf16/cache/l10n: (no message) (duration: 00m 02s)
• 00:47 logmsgbot: awight Synchronized php-1.25wmf17/extensions/CentralNotice: CentralNotice debug logging for T89258 (duration: 00m 06s)
• 00:47 logmsgbot: awight Synchronized php-1.25wmf16/extensions/CentralNotice: CentralNotice debug logging for T89258 (duration: 00m 06s)
• 00:06 logmsgbot: bd808 Synchronized wmf-config/logging-labs.php: Revert: Switch beta to syslog logging (d9dcccb) (duration: 00m 06s)

February 13

• 23:59 logmsgbot: awight Synchronized php-1.25wmf17/extensions/CentralNotice: CentralNotice debug logging for T89258 (duration: 00m 05s)
• 23:58 logmsgbot: bd808 Synchronized wmf-config/logging-labs.php: Switch beta to syslog logging (d9dcccb) (duration: 00m 06s)
• 23:56 logmsgbot: awight Synchronized php-1.25wmf16/extensions/CentralNotice: CentralNotice debug logging for T89258 (duration: 00m 06s)
• 23:26 logmsgbot: legoktm Synchronized php-1.25wmf16/extensions/CentralAuth/includes/CentralAuthUser.php: https://gerrit.wikimedia.org/r/#/c/190579/ (duration: 00m 06s)
• 23:25 logmsgbot: legoktm Synchronized php-1.25wmf17/extensions/CentralAuth/includes/CentralAuthUser.php: https://gerrit.wikimedia.org/r/#/c/190579/ (duration: 00m 06s)
• 23:21 logmsgbot: awight Synchronized php-1.25wmf17/extensions/CentralNotice: CentralNotice debug logging for T89258 (duration: 00m 08s)
• 23:20 logmsgbot: awight Synchronized php-1.25wmf16/extensions/CentralNotice: CentralNotice debug logging for T89258 (duration: 00m 07s)
• 23:15 logmsgbot: awight Synchronized php-1.25wmf17/extensions/CentralNotice: CentralNotice debug logging for T89258 (duration: 00m 05s)
• 23:06 logmsgbot: awight Synchronized wmf-config: Set up a new debug logging group for T89258 (take 2) (duration: 00m 06s)
• 22:56 logmsgbot: awight Synchronized wmf-config: Set up a new debug logging group for T89258 (duration: 00m 06s)
• 22:53 logmsgbot: awight Synchronized php-1.25wmf17/extensions/CentralNotice: CentralNotice fixes for T89258 and T45250 (duration: 00m 06s)
• 22:52 logmsgbot: awight Synchronized php-1.25wmf16/extensions/CentralNotice: CentralNotice fixes for T89258 and T45250 (duration: 00m 07s)
• 21:38 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: debug time over (duration: 00m 05s)
• 21:37 logmsgbot: demon Synchronized php-1.25wmf16/includes/resourceloader/ResourceLoaderImage.php: debug time over (duration: 00m 05s)
• 21:24 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: Debug fun (duration: 00m 05s)
• 21:22 logmsgbot: demon Synchronized php-1.25wmf16/includes/resourceloader/ResourceLoaderImage.php: Debug fun (duration: 00m 05s)
• 18:17 robh: morebots, you doing yer thing?
• 15:59 godog: es-tool restart-fast on elastic1011
• 15:08 godog: correction, elastic1010
• 15:08 godog: es-tool restart-fast on elastic1019
• 14:41 godog: es-tool restart-fast on elastic1009
• 13:25 hoo: Started rebuildItemsPerSite for wikidata on terbium
• 11:34 godog: restart elasticsearch on logstash1001 logstash1002 logstash1003
• 11:26 paravoid: mw1095/mw1192: service hhvm restart, alerts for 10h30/9h35 respectively
• 11:18 godog: es-tool restart-fast on elastic1008
• 04:56 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Feb 13 04:55:10 UTC 2015 (duration 55m 9s)
• 02:32 logmsgbot: LocalisationUpdate completed (1.25wmf17) at 2015-02-13 02:31:34+00:00
• 02:31 logmsgbot: l10nupdate Synchronized php-1.25wmf17/cache/l10n: (no message) (duration: 00m 01s)
• 02:19 ori: ran redis commands 'HDEL jobqueue:aggregator:h-queue-types:v2 LocalGlobalUserPageCacheUpdateJob/labswiki' and 'HDEL jobqueue:aggregator:h-queue-types:v2 LocalGlobalUserPageCacheUpdateJob' on rdb1001
• 02:18 logmsgbot: LocalisationUpdate completed (1.25wmf16) at 2015-02-13 02:17:27+00:00
• 02:17 logmsgbot: l10nupdate Synchronized php-1.25wmf16/cache/l10n: (no message) (duration: 00m 01s)
• 02:04 logmsgbot: legoktm Synchronized wmf-config/CommonSettings.php: Set ['LocalGlobalUserPageCacheUpdateJob'] = 'NullJob' to clear queues (duration: 00m 06s)
• 01:52 logmsgbot: legoktm Synchronized php-1.25wmf17/extensions/GlobalUserPage: Revert GlobalUserPage updates (duration: 00m 06s)
• 01:43 logmsgbot: maxsem Synchronized wmf-config/: Let there be mobile on wikitech (duration: 00m 06s)
• 01:36 ori: Correcting path reference in private/PrivateSettings.php required restarting HHVM on job runners. StatCache bug?
• 01:32 ori: restarting jobrunners
• 01:30 logmsgbot: ori Synchronized private/PrivateSettings.php: Correct path reference, for real this time (duration: 00m 07s)
• 01:26 logmsgbot: ori Synchronized private/PrivateSettings.php: Correct path reference (duration: 00m 06s)
• 01:01 logmsgbot: maxsem Synchronized wmf-config/InitialiseSettings.php: Shutting the warning off (duration: 00m 06s)
• 00:35 logmsgbot: maxsem Synchronized php-1.25wmf16/extensions/Wikidata/: SWAT (duration: 00m 12s)
• 00:34 logmsgbot: maxsem Synchronized php-1.25wmf17/extensions/GlobalUserPage/: fix SWAT (duration: 00m 06s)
• 00:31 logmsgbot: maxsem Synchronized php-1.25wmf17/extensions/Wikidata/: touch (duration: 00m 18s)
• 00:29 mutante: phab service restart for config change
• 00:29 logmsgbot: maxsem Synchronized php-1.25wmf17/extensions/Wikidata/: SWAT (duration: 00m 12s)
• 00:21 logmsgbot: maxsem Synchronized php-1.25wmf17/extensions/GlobalUserPage/: SWAT (duration: 00m 06s)

February 12

• 22:55 mutante: restarting phab for config change
• 20:05 mutante: moving servermon behind misc-web
• 19:18 logmsgbot: demon Synchronized php-1.25wmf16/extensions/CentralNotice/special/SpecialBannerRandom.php: rm live hack leftovers, now being worked on (duration: 00m 05s)
• 18:47 logmsgbot: demon Synchronized php-1.25wmf16/extensions/CentralNotice/special/SpecialBannerRandom.php: rm live hack, have our data (duration: 00m 06s)
• 18:44 logmsgbot: demon Synchronized php-1.25wmf16/extensions/CentralNotice/special/SpecialBannerRandom.php: live hack (duration: 00m 08s)
• 18:11 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: Remove random banner references (duration: 00m 05s)
• 17:34 logmsgbot: oblivian Synchronized wmf-config/session.php: Adding mc1018 to the sessions redis pool (duration: 00m 07s)
• 17:25 logmsgbot: oblivian Synchronized wmf-config/session.php: Adding mc1017 to the sessions redis pool (duration: 00m 05s)
• 17:13 _joe_: adding mc1018 to the nutcracker pool, this time without forcing a puppet run
• 16:57 logmsgbot: demon Synchronized php-1.25wmf16/extensions/CentralNotice/includes/BannerChooser.php: rm live hack for debugging (duration: 00m 05s)
• 16:54 logmsgbot: demon Synchronized php-1.25wmf16/extensions/CentralNotice/includes/BannerChooser.php: live hack for debugging (duration: 00m 06s)
• 16:44 _joe_: triggering a puppet run to insert mc1017 in the nutcracker pool
• 16:40 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/188887/ (duration: 00m 07s)
• 16:35 logmsgbot: krenair Synchronized wmf-config: trying that last sync again, I forgot to actually run the merge (duration: 00m 06s)
• 16:33 logmsgbot: krenair Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/190230/ (duration: 00m 07s)
• 16:26 logmsgbot: krenair Synchronized wmf-config: rv (duration: 00m 06s)
• 16:24 logmsgbot: krenair Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/187730/ (duration: 00m 05s)
• 16:11 godog: es-tool restart-fast on elastic1007
• 16:08 ottomata: re-enabling bits varnishkafka instances
• 16:01 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/190133/ (duration: 00m 07s)
• 14:15 Krenair: Manually logged a missing cross-wiki rights log change entry on meta "Avraham changed group membership for User:Bencmq@zhwiki from bureaucrat, check user and administrator to bureaucrat and administrator (requested)". See T89205 for details
• 11:28 godog: es-tool restart-fast on elastic1006
• 10:12 hashar: gallium and lanthanum: dpkg --purge locate
• 10:09 hashar: gallium: uninstalling locate package from gallium. Has been installed on 2015-01-30 00:31:39 apparently manually by root@iron.wikimedia.org
• 10:02 godog: es-tool fast-restart on elastic1005
• 08:28 hashar: puppet-lint now complains on error (not warnings) \O/ {{bug:T87132}}
• 04:54 springle: broke puppet db grant. fixed puppet db grant
• 04:53 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Feb 12 04:52:16 UTC 2015 (duration 52m 15s)
• 04:09 springle: sign puppet cert dbproxy1003, first run
• 03:02 andrewbogott: restarting wikitech-static. shinken works!
• 02:53 andrewbogott: breaking wikitech-static on purpose to test the shinken alert
• 02:42 logmsgbot: LocalisationUpdate completed (1.25wmf17) at 2015-02-12 02:41:01+00:00
• 02:40 logmsgbot: l10nupdate Synchronized php-1.25wmf17/cache/l10n: (no message) (duration: 00m 01s)
• 02:24 logmsgbot: LocalisationUpdate completed (1.25wmf16) at 2015-02-12 02:23:30+00:00
• 02:23 logmsgbot: l10nupdate Synchronized php-1.25wmf16/cache/l10n: (no message) (duration: 00m 01s)
• 00:33 logmsgbot: catrope Synchronized php-1.25wmf17/resources/lib/oojs-ui: SWAT (duration: 00m 08s)
• 00:15 logmsgbot: catrope Synchronized php-1.25wmf16/resources/lib/oojs-ui: SWAT (duration: 00m 06s)
• 00:13 logmsgbot: aaron Synchronized wmf-config/StartProfiler.php: Use ProfilerSectionOnly to handle DB/filebackend entries (duration: 00m 05s)

February 11

• 23:02 logmsgbot: twentyafterfour Purged l10n cache for 1.25wmf15
• 23:00 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf17
• 22:57 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.25wmf16
• 22:47 logmsgbot: twentyafterfour Finished scap: testwiki to php-1.25wmf17 and rebuild l10n cache (duration: 48m 29s)
• 22:45 mutante: apt-get upgrading zirconium
• 22:12 mutante: deactivated ocg1003 in pybal
• 22:05 andrewbogott: updated wikitech-static to wmf/1.25wmf15
• 21:58 logmsgbot: twentyafterfour Started scap: testwiki to php-1.25wmf17 and rebuild l10n cache
• 21:38 subbu: deployed parsoid version 4fc3b43d
• 20:09 bblack: eqiad-upload-https -> back to even weighting
• 19:46 bblack: all eqiad-upload-https -> cp1064
• 19:36 subbu: temporarily turn off logging to logstash till logstash isssues are resolved.
• 19:34 bblack: repooled cp1070 (eqiad bits) in pybal
• 19:18 bd808: restarted elasticsearch on logstash1002 after OOM
• 19:15 mutante: moved docroots on zirconium to new logical volume for /srv
• 19:09 godog: powerdown graphite1002 T88992
• 18:20 bd808: restarted Elasticsearch on logstash1003; preventative, other nodes restarted today
• 17:59 mutante: ran puppet on virt1000 - finished just fine, not sure why icinga said fail
• 17:54 mutante: running puppet on ruthenium (last was 2 days ago but also not admin disabled..)
• 17:49 godog: logging test
• 10:18 godog: restart elasticsearch on logstash1003, OOM
• 09:58 hashar: restarting Jenkins to upgrade the Credentials plugin
• 06:29 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1057 (duration: 00m 05s)
• 04:54 springle: restarting labsdb1002 https://lists.wikimedia.org/pipermail/labs-l/2015-February/003354.html
• 04:35 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Feb 11 04:34:30 UTC 2015 (duration 34m 29s)
• 03:21 springle: restarting labsdb1001 https://lists.wikimedia.org/pipermail/labs-l/2015-February/003354.html
• 02:48 hoo: Manually logged a missing global rights log change entry on meta "Ajraddatz changed global group membership for Benoit Rochon from (none) to OTRS-member with the following comment: request". See also T89205
• 02:27 logmsgbot: LocalisationUpdate completed (1.25wmf16) at 2015-02-11 02:26:55+00:00
• 02:26 logmsgbot: l10nupdate Synchronized php-1.25wmf16/cache/l10n: (no message) (duration: 00m 02s)
• 02:13 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-02-11 02:12:40+00:00
• 02:12 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 02s)
• 01:56 logmsgbot: krenair Synchronized php-1.25wmf16/includes/UserRightsProxy.php: https://gerrit.wikimedia.org/r/#/c/189879/ - same thing for interwiki user rights logs (duration: 00m 07s)
• 01:52 logmsgbot: krenair Synchronized php-1.25wmf16/extensions/CentralAuth/includes/CentralAuthGroupMembershipProxy.php: https://gerrit.wikimedia.org/r/#/c/189888/ - fix lack of global group membership change logging (duration: 00m 05s)
• 01:44 springle: puppet disabled on lanbdsb1001 labsdb1002. needs restart
• 00:24 logmsgbot: krenair Synchronized php-1.25wmf16/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/189867/ (duration: 00m 06s)
• 00:14 logmsgbot: krenair Synchronized php-1.25wmf16/extensions/UploadWizard/resources/mw.ApiUploadFormDataHandler.js: https://gerrit.wikimedia.org/r/#/c/189860/ (duration: 00m 05s)
• 00:03 logmsgbot: krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/188554/ (duration: 00m 07s)

February 10

• 23:38 logmsgbot: andyrussg Synchronized php-1.25wmf16/extensions/CentralNotice/: Update CentralNotice (duration: 00m 06s)
• 22:00 bblack: repooled cp1064 eqiad upload frontends in pybal
• 21:38 bblack: repooled cp1065 eqiad text frontend in pybal
• 21:16 bblack: rebooting cp1064 for experimental kernel (is depooled)
• 21:04 bblack: cp1065 frontend disabled in pybal temporarily
• 21:01 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: fixing upload path for labswiki (duration: 00m 06s)
• 20:51 bblack: cp1064 frontend disabled in pybal
• 20:28 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: group1 to 1.25wmf16
• 20:20 mutante: restarting phd on iridium (phab) for config change
• 17:10 logmsgbot: demon Finished scap: No code changes, bringing silver in as deploy target (duration: 17m 31s)
• 16:53 logmsgbot: demon Started scap: No code changes, bringing silver in as deploy target
• 16:52 logmsgbot: demon Synchronized wmf-config/wikitech.php: Testing silver sync (duration: 00m 05s)
• 16:24 logmsgbot: anomie Synchronized wmf-config/CommonSettings.php: SWAT: Revert "Whitelist application/x-gzip on private wikis to fully allow dia files", wasn't a correct fix for the issue (duration: 00m 05s)
• 16:18 godog: temporarily disable puppet on carbon
• 16:15 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Quiz extension at cawikibooks gerrit:187913 (duration: 00m 07s)
• 16:13 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wmgUseFloatedToc to false at dewikivoyage gerrit:187408 (duration: 00m 06s)
• 16:10 godog: stop replication on elasticsearch cluster and restart ES on elastic1002
• 16:07 logmsgbot: anomie Synchronized wmf-config/CommonSettings.php: SWAT: Whitelist application/x-gzip on private wikis to fully allow dia files gerrit:188557 (duration: 00m 05s)
• 10:57 logmsgbot: hoo Synchronized wmf-config/InitialiseSettings-labs.php: (no message) (duration: 00m 06s)
• 10:43 godog: reimage graphite1002
• 07:10 _joe_: restarting HHVM on mw1128, in a deadlock in HPHP::RequestInjectionData::onSessionInit ()
• 07:09 _joe_: restarting HHVM on mw1139, in a deadlock in HPHP::StatCache::refresh ()
• 04:43 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Feb 10 04:42:17 UTC 2015 (duration 42m 16s)
• 02:54 andrewbogott: finished wikitech move to silver.
• 01:28 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/189634/ (duration: 00m 10s)
• 01:23 logmsgbot: krenair Synchronized php-1.25wmf16/resources/lib/oojs-ui: https://gerrit.wikimedia.org/r/#/c/189147/ (duration: 00m 08s)
• 01:21 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/189211/ per James_F in -dev (duration: 00m 06s)
• 01:11 logmsgbot: krenair Synchronized php-1.25wmf16/extensions/VisualEditor/modules/ve-mw: https://gerrit.wikimedia.org/r/#/c/189144/ (duration: 00m 05s)
• 01:00 logmsgbot: krenair Synchronized php-1.25wmf16/extensions/Echo/tests/phpunit/includes/DiscussionParserTest.php: https://gerrit.wikimedia.org/r/#/c/189638/ (duration: 00m 08s)
• 01:00 logmsgbot: krenair Synchronized php-1.25wmf16/extensions/Echo/includes/DiscussionParser.php: https://gerrit.wikimedia.org/r/#/c/189638/ (duration: 00m 05s)
• 00:46 logmsgbot: krenair Synchronized php-1.25wmf16/extensions/Echo/tests/phpunit/includes/DiscussionParserTest.php: https://gerrit.wikimedia.org/r/#/c/189549/ (duration: 00m 06s)
• 00:45 logmsgbot: krenair Synchronized php-1.25wmf16/extensions/Echo/includes/DiscussionParser.php: https://gerrit.wikimedia.org/r/#/c/189549/ (duration: 00m 06s)
• 00:33 logmsgbot: krenair Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/181358/ (duration: 00m 05s)
• 00:33 logmsgbot: krenair Synchronized wmf-config/CommonSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/181358/ (duration: 00m 05s)
• 00:29 logmsgbot: yurik Finished scap: syncing ZeroBanner i18n (duration: 34m 06s)
• 00:13 bd808: restarted elasticsearch on logstash1001; OOM

February 9

• 23:55 logmsgbot: yurik Started scap: syncing ZeroBanner i18n
• 23:54 logmsgbot: yurik Synchronized php-1.25wmf15/extensions/ZeroBanner: cherry-picking 189617 (duration: 00m 05s)
• 23:54 logmsgbot: yurik Synchronized php-1.25wmf16/extensions/ZeroBanner: cherry-picking 189617 (duration: 00m 07s)
• 23:01 logmsgbot: yurik Synchronized php-1.25wmf16/extensions/ZeroBanner: cherry-picking 189553 (duration: 00m 06s)
• 23:00 logmsgbot: yurik Synchronized php-1.25wmf15/extensions/ZeroBanner: cherry-picking 189553 (duration: 00m 06s)
• 17:46 bd808: restarted elasticsearch on logstash1003; OOM
• 16:53 logmsgbot: marktraceur Synchronized php-1.25wmf15/extensions/UploadWizard/resources/mw.FlickrChecker.js: [SWAT] [wmf15] Re-add flickrreview template to files imported from Flickr by UploadWizard (duration: 00m 05s)
• 16:48 logmsgbot: marktraceur Synchronized php-1.25wmf16/extensions/UploadWizard/: [SWAT] [wmf16] Trying to force UploadWizard to update (duration: 00m 05s)
• 16:48 logmsgbot: marktraceur Synchronized php-1.25wmf15/extensions/UploadWizard/: [SWAT] [wmf15] Trying to force UploadWizard to update (duration: 00m 06s)
• 16:47 jgage: restarted eventlogging on hafnium (with deploy from master on tin this time)
• 16:38 logmsgbot: marktraceur Synchronized php-1.25wmf16/extensions/UploadWizard/resources/mw.FlickrChecker.js: [SWAT] [wmf16] Re-add flickrreview template to files imported from Flickr by UploadWizard (duration: 00m 06s)
• 16:37 logmsgbot: marktraceur Synchronized php-1.25wmf15/extensions/UploadWizard/resources/mw.FlickrChecker.js: [SWAT] [wmf15] Re-add flickrreview template to files imported from Flickr by UploadWizard (duration: 00m 05s)
• 16:22 jgage: restarted eventlogging on hafnium for nuria via ~root/upgrade-eventlogging --no-update
• 16:18 logmsgbot: marktraceur Synchronized wmf-config/throttle.php: [SWAT] [config] Add throttle rules for two workshops (duration: 00m 07s)
• 16:14 logmsgbot: marktraceur Synchronized wmf-config/: [SWAT] [config] Un-subscribe frequently failing recipients (duration: 00m 05s)
• 16:12 logmsgbot: marktraceur Synchronized php-1.25wmf16/extensions/OAuth/: [SWAT] [wmf16] OAuth: Support ListDefinedTags and ChangeTagsListActive hooks (duration: 00m 11s)
• 15:00 cmjohnson1: cp1070 down for h/w troubleshooting. Already depooled by bblack
• 11:58 godog: bounce mwprof-profiler-to-carbon on tungsten
• 10:47 hoo: Manually removed wikidatawiki.wb_changes_dispatch entries for test wikis (test2wiki, testwiki, testwikidata).
• 09:11 gwicke: cassandra load testing on xenon, praseodymium and cerium; disk space is tight, might run out on one of those boxes but they are purely test boxes right now, so np
• 05:32 gwicke: stopped puppet on cerium, praseodymium & xenon
• 05:31 gwicke: manually updated cassandra on cerium, praseodymium & xenon to 2.1.2 (see https://phabricator.wikimedia.org/T88956)
• 03:50 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Feb 9 03:48:57 UTC 2015 (duration 48m 56s)
• 02:13 logmsgbot: LocalisationUpdate completed (1.25wmf16) at 2015-02-09 02:12:45+00:00
• 02:12 logmsgbot: l10nupdate Synchronized php-1.25wmf16/cache/l10n: (no message) (duration: 00m 02s)
• 02:12 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-02-09 02:11:13+00:00
• 02:11 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s)

February 8

• 03:53 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Feb 8 03:52:49 UTC 2015 (duration 52m 48s)
• 02:14 logmsgbot: LocalisationUpdate completed (1.25wmf16) at 2015-02-08 02:13:11+00:00
• 02:13 logmsgbot: l10nupdate Synchronized php-1.25wmf16/cache/l10n: (no message) (duration: 00m 01s)
• 02:12 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-02-08 02:11:41+00:00
• 02:11 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s)

February 7

• 15:35 apergos: started nginx on daaset1001, it was not running for some reason
• 09:40 bblack: depooled cp1070 in pybal
• 09:33 bblack: rebooting cp1070 (dead network, dead console)
• 05:10 subbu: deployed parsoid hotfiix 8ca7ef40 (cherry-pick of 447a0565)
• 04:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Feb 7 04:47:30 UTC 2015 (duration 47m 29s)
• 03:13 gwicke: restarting parsoid cluster
• 02:34 logmsgbot: LocalisationUpdate completed (1.25wmf16) at 2015-02-07 02:33:09+00:00
• 02:33 logmsgbot: l10nupdate Synchronized php-1.25wmf16/cache/l10n: (no message) (duration: 00m 01s)
• 02:20 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-02-07 02:19:09+00:00
• 02:19 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 02s)
• 02:11 qchris: Ran kafka leader re-election as analytics1021 dropped out of it's partition leader role.
• 01:48 bblack: leaving cp1064 (jessie upload eqiad) pooled front+back. it's experimental but looks stable. if upload-related 503 spikes and I'm not around, feel free to depool it.
• 00:18 qchris: Manually bumping heap for the Hadoop namenodes and revived them after both of them running out of heap and not coming back.

February 6

• 22:53 logmsgbot: marktraceur Synchronized wmf-config/: [friday] beta config change for tgr (duration: 00m 09s)
• 22:53 subbu: restarted parsoid service to kill several stuck processes on multiple nodes
• 20:05 robh: ms1004 coming offline, shouldnt page (but disregard if it does)
• 19:19 subbu: deployed parsoid hotfiix a9dbd4fc (cherry-pick of 76d6658c)
• 16:54 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: Adding cdm16062.contentdm.oclc.org to wgCopyUploadsDomains (duration: 00m 05s)
• 16:40 godog: cancel downtime on graphite1001, enable downtime on tungsten pending full decomission
• 16:05 godog: bounce ocg on ocg1001 and stop additional ocg instance running
• 15:50 bblack: depool -> repool cp1064 varnish-frontend, reduced cache size to 16G, re-enabled compact_memory
• 15:50 godog: restart ocg on ocg1003 to pick up statsd dns changes
• 15:48 godog: restart ocg on ocg1002 to pick up statsd dns changes
• 14:33 bblack: starting up a fresh round of SSL testing on eqiad upload pooling (cp1064)
• 14:12 godog: bounce diamond on lvs2004/lvs2005
• 13:55 cmjohnson1: upgrading boron to trusty
• 10:51 godog: reimage ms-be2014
• 07:50 _joe_: restarting the parsoid cluster, one node at a time, some processes are stuck.
• 02:04 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
• 01:01 ori: restarting xenon on fluorine
• 00:14 logmsgbot: krenair Synchronized php-1.25wmf15/includes/CategoryViewer.php: https://gerrit.wikimedia.org/r/#/c/188944/1 (duration: 00m 06s)
• 00:13 logmsgbot: krenair Synchronized php-1.25wmf16/includes/CategoryViewer.php: https://gerrit.wikimedia.org/r/#/c/188945/1 (duration: 00m 06s)

February 5

• 23:53 bd808: Updated Wikimania Scholarships to 0852585 (re-enable language selection) + local hack in trebuchet repo to remove incomplete translations
• 22:53 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: Enable Parsoid on wikitech (duration: 00m 05s)
• 21:08 logmsgbot: reedy Purged l10n cache for 1.25wmf13
• 21:03 logmsgbot: reedy Synchronized php-1.25wmf16/extensions/CheckUser/: (no message) (duration: 00m 07s)
• 20:59 logmsgbot: reedy Synchronized php-1.25wmf16: (no message) (duration: 00m 52s)
• 20:59 bblack: cp1064 upload b ackend re-enabled in cache.pp; if upload-related 503s ensue later today and I'm not around, feel free to re-disable it
• 20:57 mutante: radon - reinstalling, scheduled downtime
• 20:42 Reedy: mw1092 giving file has vanished: "/wmf-config/.InitialiseSettings.php.KSg3AF" (in common)
• 20:42 logmsgbot: reedy Synchronized wmf-config/: GlobalUserPage and I33a855cecfbe25003fe9e4f5e2fab2f928c79da4 (duration: 00m 07s)
• 20:41 logmsgbot: reedy Synchronized wmf-config/: GlobalUserPage and I33a855cecfbe25003fe9e4f5e2fab2f928c79da4 (duration: 00m 05s)
• 20:35 logmsgbot: reedy Synchronized wmf-config/: GlobalUserPage and I33a855cecfbe25003fe9e4f5e2fab2f928c79da4 (duration: 00m 08s)
• 20:34 logmsgbot: reedy Synchronized php-1.25wmf16/includes/EditPage.php: Id376f9e75c43c5bd0fa910b04d066e6aa37c73d1 (duration: 00m 07s)
• 20:30 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf16
• 20:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.25wmf15
• 20:21 logmsgbot: reedy Finished scap: testwiki to 1.25wmf16 (duration: 38m 37s)
• 20:08 _joe_: re-exported nfs exports on dataset1001, remounted /mnt/data on snapshot1001
• 19:42 logmsgbot: reedy Started scap: testwiki to 1.25wmf16
• 19:41 logmsgbot: reedy scap aborted: testwiki to 1.25wmf16 (duration: 03m 27s)
• 19:37 logmsgbot: reedy Started scap: testwiki to 1.25wmf16
• 19:09 legoktm: clearing bad sidebar memcache entries on commonswiki
• 18:08 _joe_: restarting nutcracker on jobrunners
• 18:06 _joe_: restarting nutcracker on api appservers
• 18:01 ori: restarting nutcracker on all appservers
• 17:54 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: I4f28205e6: Set $wmgUseMonologLogger to false (duration: 00m 06s)
• 17:47 logmsgbot: ori Synchronized wmf-config/logging.php: Live hack: disable Logstash logging on suspicion that it is acting up (duration: 00m 05s)
• 17:34 paravoid: restarting HHVM on all appservers/API appservers in 10%/6s batches
• 17:26 bblack: repooled cp1063 frontend-only
• 16:21 godog: bounce jmxtrans on analytics1018, analytics1021 and analytics1022
• 16:15 godog: bounce jmxtrans on analytics1012
• 16:03 godog: re-enabled puppet on graphite1001, bounce uwsgi
• 14:34 godog: upload txstatsd 1.0.0-3 to trusty-wikimedia
• 12:42 paravoid: cp*/amssq*: salt rm /etc/logrotate.d/varnishkafka-frontend-stats to fix cronspam
• 12:30 hashar: Upgrading Jenkins and restarting it
• 06:43 springle: upgrade silver to mariadb 10
• 04:55 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Feb 5 04:54:00 UTC 2015 (duration 53m 59s)
• 02:37 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Ia59e654e8: Set a statsd-compatible $wgStatsFormatString (duration: 00m 07s)
• 02:35 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-02-05 02:34:02+00:00
• 02:34 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s)
• 02:20 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-02-05 02:19:31+00:00
• 02:19 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 02s)
• 01:41 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I7b270eb8a: Set $wgUDPProfilerHost to service alias rather than hard-code IP (duration: 00m 05s)
• 00:54 bd808: truncated redis input queues for logstash on all 3 hosts to see if cluster can keep up now with 3 elasticsearch writer threads
• 00:08 Krinkle: Added 'dduvall' to integration group ACL on Gerrit
• 00:06 springle: xtrabackup clone virt1000 to silver

February 4

• 23:38 mutante: starting memcached on virt1000
• 23:21 qchris: Manual failover of Hadoop namenode from analytics1001 to analytics1002, as analytics1001 had Heap space errors
• 22:50 ejegg: updated payments from 1e9b78e9a8bf557a710988620bd6f1a335787173 to cbaf66e7705789f37117ec6edc4d936c6174d511
• 22:49 manybubbles: this is certainly a bug in Elasticsearch, but I imagine its one solved in newer versions. i hope, more like.
• 22:49 manybubbles: not sure what happened but now space if freeing up on 1001. the disk was never in danger of filling up but it was full enough not to allocate more to it. Now that stuff is allocating elsewhere elasticsearch is clearing the used space.
• 22:41 manybubbles: looks like elastics1001 doesn't have much free space left. I think that might have something to do with this....
• 22:38 manybubbles: Elasticsearch wasn't initializing shards to elastic1001 after its restart. Didn't check why. Set allocation to primaries then back to all and that unstuck it.
• 21:16 arlolra: updated Parsoid to version dd4721f4
• 20:33 logmsgbot: ori rebuilt wikiversions.cdb and synchronized wikiversions files: I4fb67945b: Revert "[Regression] Revert "Non wikipedias to 1.25wmf15"
• 20:18 logmsgbot: aude Synchronized wmf-config/Wikibase.php: set useLegacyChangesSubscription to true for Wikidata (duration: 00m 07s)
• 18:30 godog: bounce txstatsd on cache hosts in eqiad
• 18:17 godog: bounce txstatsd on cache hosts in ulsfo
• 18:08 godog: bounce txstatsd on cache hosts in esams
• 17:30 logmsgbot: marktraceur Synchronized php-1.25wmf14/extensions/UploadWizard/: Touching pretty much everything in UploadWizard, maybe it will help (duration: 00m 07s)
• 17:22 logmsgbot: marktraceur Synchronized php-1.25wmf14/extensions/UploadWizard/resources/mw.UploadWizard.js: Touch an UploadWizard file to try and fix caching (duration: 00m 07s)
• 16:58 robh: replacing the intermediary cert on dumps.w.o (so nginx will flap on it shortly)
• 16:56 godog: restart ES on elastic1001
• 15:43 logmsgbot: marktraceur Synchronized php-1.25wmf14/extensions/UploadWizard/resources/controller/uw.controller.Upload.js: Touch an UploadWizard file to try and fix caching (duration: 00m 07s)
• 15:25 logmsgbot: marktraceur Synchronized php-1.25wmf14/extensions/UploadWizard/resources/controller/uw.controller.Upload.js: Touch an UploadWizard file to try and fix caching (duration: 00m 05s)
• 15:22 godog: graphite move close to completion, updating dashboards
• 15:16 godog: bounce diamond in batches in eqiad
• 14:50 logmsgbot: marktraceur Synchronized php-1.25wmf15/extensions/UploadWizard/resources/controller/uw.controller.Upload.js: Touch an UploadWizard file to try and fix caching (duration: 00m 05s)
• 14:14 godog: bounce webperf-related services on hafnium too: ve, statsd-mw-js-deprecate, statsv, asset-check
• 14:10 godog: bounce navtiming on hafnium to pick up dns changes
• 12:42 godog: stop bacula-fd on tungsten, backups running during migration
• 12:41 _joe_: installing the new HHVM package on jobrunners
• 12:28 godog: bounce txstatsd on ms-fe*
• 12:28 godog: bounce txstatsd on ms-be*
• 12:00 godog: bounce diamond in batches in ulsfo
• 11:57 godog: bounce diamond in batches in esams
• 11:51 godog: bounce mwprof on tungsten to force picking up dns changes
• 11:35 _joe_: installing the new hhvm package on api, one at a time
• 11:23 godog: start migrating graphite from tungsten to graphite1001 https://gerrit.wikimedia.org/r/#/c/188036/1 https://gerrit.wikimedia.org/r/#/c/188035/1 https://phabricator.wikimedia.org/T85909
• 10:14 logmsgbot: ori Finished scap: I78446aacb: [Regression] Revert "Non wikipedias to 1.25wmf15" (duration: 31m 34s)
• 10:06 godog: start migrating graphite from tungsten to graphite1001 https://gerrit.wikimedia.org/r/#/c/188036/1 https://gerrit.wikimedia.org/r/#/c/188035/1 https://phabricator.wikimedia.org/T85909
• 10:06 ori: restarted hung HHVM on mw1039
• 09:42 logmsgbot: ori Started scap: I78446aacb: [Regression] Revert "Non wikipedias to 1.25wmf15"
• 09:42 logmsgbot: ori scap aborted: I78446aacb: [Regression] Revert "Non wikipedias to 1.25wmf15" (duration: 00m 02s)
• 09:42 logmsgbot: ori Started scap: I78446aacb: [Regression] Revert "Non wikipedias to 1.25wmf15"
• 08:57 _joe_: installing the new hhvm package on all appservers, one at a time
• 07:49 qchris: Manual failover of Hadoop namenode from analytics1002 to analytics1001, as analytics1002 had Heap space errors
• 05:16 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1057 (duration: 00m 06s)
• 04:49 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Feb 4 04:48:42 UTC 2015 (duration 48m 41s)
• 02:48 logmsgbot: tstarling Synchronized php-1.25wmf15/includes/specials/SpecialUserrights.php: Unbreak interwiki user rights granting (duration: 00m 05s)
• 02:30 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-02-04 02:29:17+00:00
• 02:29 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 02s)
• 02:16 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-02-04 02:15:35+00:00
• 02:15 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 03s)
• 01:47 bd808: restarted elasticsearch on logstash1001; rolling restart part 3 of 3
• 01:39 bd808: restarted elasticsearch on logstash1002; rolling restart of cluster part 2 of 3
• 00:38 robh: replacing dumps.w.o sha1 cert with sha256

February 3

• 22:54 bd808: restarted elasticsearch on logstash1003
• 22:53 bd808: starting rolling restart of logstash elasticsearch cluster to pick up index.merge.scheduler.max_thread_count puppet change
• 22:52 robh: magnesium apache reload for rt cert replacement
• 22:43 robh: replacing etherpad sha1 with sha256 cert
• 22:13 logmsgbot: reedy Synchronized php-1.25wmf15/extensions/WikimediaMaintenance: tmp script (duration: 00m 07s)
• 21:38 logmsgbot: maxsem Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/188300/ (duration: 00m 07s)
• 21:32 bblack: repooled cp10[67]0
• 21:29 mutante: restarted gitblit
• 21:17 andrewbogott: increased opendj lookthrough-limit to 12000 on both ldap hosts. We just hit lucky 5000 users and some queries stopped working.
• 20:59 bblack: depooling cp1060, cp1070 (1 each bits + mobile) for reinstall
• 20:59 robh: updating gerrit.wikimedia.org cert
• 19:30 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.25wmf15
• 18:17 YuviPanda: set up new images for ubuntu trusty / precise on labs, for https://phabricator.wikimedia.org/T87003
• 17:23 bblack: repooled cp1064
• 17:07 bd808: restarted elasticsearch on logstash1002; OOM
• 16:55 bblack: cp1064 (eqiad upload cache) depooled in pybal
• 15:39 _joe_: installing a new package to canary servers
• 14:55 _joe_: uploaded a new hhvm package version, deploying to testwiki and beta
• 08:50 logmsgbot: reedy Synchronized wmf-config/: Bye bye Solarium (duration: 00m 06s)
• 08:20 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: Ie7b32e3d8: Add log group for T87645 (duration: 00m 05s)
• 08:19 logmsgbot: ori Synchronized php-1.25wmf14/includes/EditPage.php: Id376f9e75: Hack for T87645, since maybe it is still happening (duration: 00m 07s)
• 08:17 logmsgbot: ori Synchronized php-1.25wmf15/includes/EditPage.php: Id376f9e75: Hack for T87645, since maybe it is still happening (duration: 00m 05s)
• 08:14 paravoid: radium: upgrade tor to the latest torproject.org version
• 08:10 springle: wikitech mysql restart to fix novaold errors
• 05:14 springle: wikitech virt1000 test db dump T88311
• 04:49 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Feb 3 04:48:38 UTC 2015 (duration 48m 37s)
• 02:49 ori: deployed https://gerrit.wikimedia.org/r/#/c/187304/ (php-set X-Analytics header) to both production branches.
• 02:42 bblack: cp1065 re-pooled in pybal
• 02:31 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-02-03 02:30:18+00:00
• 02:30 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 02s)
• 02:19 mutante: installing package upgrades on radium
• 02:17 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-02-03 02:16:37+00:00
• 02:16 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 03s)
• 01:59 bblack: depool cp1065 (text eqiad in pybal -> jessie)
• 01:59 bd808: Manually created apifeatureusage-2015.02.02 and apifeatureusage-2015.02.03 indices in elasticsearch; clsuter needs rolling restart for autocreate to work for these names
• 01:51 bd808: restarted logstash on logstash1001
• 01:51 bd808: restarted elasticsearch on logstash1003
• 00:03 mutante: rbf2002 - error while setting up RAID during installer (rbf2001 did not have this? or did it?)
• 00:02 mutante: rbf2001 - initial puppet run, adding users

February 2

• 23:59 mutante: signing puppet cert for rbf2001, PXE booting rbf2002
• 22:18 logmsgbot: ori Synchronized php-1.25wmf14/extensions/XAnalytics: (no message) (duration: 00m 05s)
• 22:18 logmsgbot: ori Synchronized php-1.25wmf15/extensions/XAnalytics: (no message) (duration: 00m 05s)
• 21:16 subbu: deployed parsoid version e3c9ae99
• 19:59 ejegg: updated payments from ce73ed11de9775a596c51acdc036503751961bc8 to 1e9b78e9a8bf557a710988620bd6f1a335787173
• 19:11 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Enable usage tracking on Wikidata (duration: 00m 07s)
• 18:15 mutante: restarted hhvm on mw1207
• 18:04 mutante: started nginx on ms1001, dataset1001
• 18:00 mutante: restarted gitblit
• 17:20 logmsgbot: marktraceur Synchronized php-1.25wmf15/extensions/UploadWizard/resources/mw.FormDataTransport.js: [SWAT] [wmf15] Fix UploadWizard for ogg files (duration: 00m 07s)
• 17:20 logmsgbot: marktraceur Synchronized php-1.25wmf14/extensions/UploadWizard/resources/mw.FormDataTransport.js: [SWAT] [wmf14] Fix UploadWizard for ogg files (duration: 00m 06s)
• 16:54 logmsgbot: anomie Synchronized wmf-config: SWAT: Added BounceHandler extension to group0 wikis gerrit:186242 (duration: 00m 07s)
• 16:52 yuvipanda: kill opendj on virt1000, it shouldn't have been running there in the first place
• 16:34 logmsgbot: anomie Synchronized wmf-config: SWAT: Have ContentTranslate publish article to Main namespace for cawiki gerrit:186358 (duration: 00m 07s)
• 16:11 aude: added and populated wbc_entity_usage table for wikidatawiki
• 10:54 logmsgbot: hoo Synchronized wmf-config/Wikibase.php: Exempt Item and Property namespaces from ConfirmEdit (duration: 00m 07s)
• 10:44 logmsgbot: hoo Synchronized php-1.25wmf14/extensions/Wikidata/: Update Wikibase: Fixes for UsageTracking and the anon edit warning (duration: 00m 14s)
• 10:43 logmsgbot: hoo Synchronized php-1.25wmf15/extensions/Wikidata/: Update Wikibase: Fixes for UsageTracking and the anon edit warning (duration: 00m 12s)
• 10:28 yuvipanda: been restarting pdns, opendj, apache, mysql, keystone left and right on virt1000 all day.
• 10:28 hashar: Synchronized wmf-config/InitialiseSettings.php: Add www.doria.fi to $wgCopyUploadsDomains bug T87104 https://gerrit.wikimedia.org/r/#/c/187914/ (duration: 00m 07s)
• 10:27 yuvipanda: foo
• 04:09 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Feb 2 04:08:14 UTC 2015 (duration 8m 13s)
• 02:44 springle: virt1000 mysqld restart, shrink buffer pool
• 02:20 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-02-02 02:19:21+00:00
• 02:19 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 02s)
• 02:11 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-02-02 02:09:59+00:00
• 02:09 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 04s)
• 00:13 subbu: restarted parsoid service on the parsoid cluster to free up leaked memory on several processes (seems to have happened in the 21:30 - 22:30 UTC on 31st Jan time frame)

February 1

• 04:12 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Feb 1 04:11:30 UTC 2015 (duration 11m 29s)
• 02:21 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-02-01 02:20:24+00:00
• 02:20 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s)
• 02:12 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-02-01 02:10:58+00:00
• 02:10 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 02s)

January 31

• 14:20 logmsgbot: hoo Synchronized wmf-config/CommonSettings-labs.php: (no message) (duration: 00m 06s)
• 04:12 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jan 31 04:11:31 UTC 2015 (duration 11m 29s)
• 02:25 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-01-31 02:24:27+00:00
• 02:24 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 02s)
• 02:12 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-31 02:10:58+00:00
• 02:10 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 02s)

January 30

• 22:38 subbu: deployed parsoid version 2abd0eb6
• 21:52 bblack: re-pooling cp3020 (bits cache esams) - reinstalled, looks sane...
• 21:38 YuviPanda|flight: killed diamond taking up 100% on labstore1001
• 21:04 logmsgbot: demon Synchronized docroot/noc/conf/logging.php.txt: (no message) (duration: 00m 06s)
• 21:02 logmsgbot: demon Synchronized docroot and w: (no message) (duration: 00m 10s)
• 20:58 bblack: reinstalling cp3020 (seems to have fs corruption issues, but may not be hardware...)
• 19:27 logmsgbot: phuedx Synchronized php-1.25wmf15/extensions/MobileFrontend/: No-op deployment training (duration: 00m 06s)
• 19:02 mutante: cp1039, cp1040 - shut down
• 18:56 bblack: rebooting cp3020 again (still depooled)
• 18:51 mutante: cp1037,cp1038 - shut down
• 18:40 godog: initial rsync from tungsten to graphite1001 T85909
• 18:36 cmjohnson1: removing cp1063 from pybal
• 18:34 logmsgbot: andyrussg Synchronized php-1.25wmf15/extensions/CentralNotice: Revert update to CentralNotice (duration: 00m 06s)
• 18:27 bblack: rebooting cp3020, something's all wrong there...
• 18:16 mutante: cp1037,cp1038,cp1039,cp1040 - disabled puppet, removed from icinga, revoked certs and salt key etc. decom
• 17:38 logmsgbot: andyrussg Synchronized php-1.25wmf15/extensions/CentralNotice: Update CentralNotice (duration: 00m 06s)
• 17:16 ^d: running puppet on ytterbium, gerrit shall restart
• 16:48 bblack: expect more icinga "CRITICAL: DPKG CRITICAL" on cache nodes for a while; applying backlog of upstream pkg updates slowly to all
• 16:18 bblack: restarting frontend varnishes to apply increased cache sizes from https://gerrit.wikimedia.org/r/#/c/186816/ over the next ~9H
• 10:04 paravoid: labstore1001: setting /proc/sys/sunrpc/{nfs,rpc}_debug to 0; rm /var/log/{kern.log,syslog.1,syslog}
• 04:21 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jan 30 04:20:12 UTC 2015 (duration 20m 11s)
• 02:25 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-01-30 02:23:56+00:00
• 02:24 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 03s)
• 02:20 awight: updated crm from a6b28d0a5e90f7ca68988a15f311465bbb5ae5e6 to ff28f69392fdab74520bf94e4a46586637914480
• 02:15 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-30 02:14:31+00:00
• 02:14 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 02s)
• 01:36 mutante: cp1047 - DIMM fail -> T88045
• 01:30 mutante: powercycling cp1047

January 29

• 23:59 awight: updated payments from 4743b32c9091d6f2169bea3173e75aa5d2f36eb7 to ce73ed11de9775a596c51acdc036503751961bc8
• 22:10 Krinkle: git-deploy: Deploying integration/slave-scripts I5aa76b0, I4d94af46735c, I66fbce3fa
• 19:49 bd808: Updated Wikimania Scholarships (2f4a99f) "Add Sakha to list of languages"
• 18:29 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: Disable Extension:Graph on cawiki (duration: 00m 06s)
• 04:53 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jan 29 04:51:55 UTC 2015 (duration 51m 54s)
• 02:35 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-01-29 02:34:03+00:00
• 02:34 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 02s)
• 02:21 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-29 02:20:02+00:00
• 02:20 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 02s)
• 01:12 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Id5186348f: Set $wgResourceLoaderStorageEnabled to false on osmium (duration: 00m 07s)
• 00:19 logmsgbot: demon Synchronized wmf-config/: (no message) (duration: 00m 06s)
• 00:18 logmsgbot: demon Synchronized php-1.25wmf14/includes/Title.php: (no message) (duration: 00m 06s)
• 00:17 logmsgbot: demon Synchronized php-1.25wmf15/includes/Title.php: (no message) (duration: 00m 06s)
• 00:17 logmsgbot: demon Synchronized php-1.25wmf15/includes/api/ApiPageSet.php: (no message) (duration: 00m 05s)
• 00:17 logmsgbot: demon Synchronized php-1.25wmf14/includes/api/ApiPageSet.php: (no message) (duration: 00m 06s)
• 00:11 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: flooders on officewiki (duration: 00m 08s)

January 28

• 22:55 mutante: restarting torrus on netmon1001
• 21:21 subbu: synced parsoid version 88605a4a for restart
• 21:20 subbu: synced parsoid version 88605a4a for restart
• 21:19 YuviPanda: restarting parsoid across wtp* hosts for subbu
• 21:00 YuviPanda: restarted hhvm on mw1069
• 20:40 logmsgbot: reedy Finished scap: mostly nooop, but adding graph to l10n cache (duration: 31m 33s)
• 20:09 logmsgbot: reedy Started scap: mostly nooop, but adding graph to l10n cache
• 20:03 logmsgbot: reedy Synchronized wmf-config/extension-list: Add Graph to extension-list (duration: 00m 07s)
• 20:03 mutante: shutdown remaining amslvs2-4
• 19:39 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: Exempt osmium's script URL from bits rewriting (duration: 00m 05s)
• 19:36 mutante: shutting down amslvs1
• 19:19 logmsgbot: reedy Synchronized wmf-config/: Various config updates (duration: 00m 06s)
• 19:10 mutante: decom amslvs1-4, removing from puppet
• 18:20 ejegg|away: updated crm from 4fa10ec9e3afbf65e6cbd523138cdc4b4485c482 to a6b28d0a5e90f7ca68988a15f311465bbb5ae5e6
• 17:37 bd808: restarted elasticsearch on logstash1001; OOM
• 10:45 bblack: cp301[56] frontends repooled
• 07:07 bblack: repool amssq42 for text-https
• 04:52 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jan 28 04:51:31 UTC 2015 (duration 51m 30s)
• 02:35 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-01-28 02:34:42+00:00
• 02:34 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 02s)
• 02:21 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-28 02:20:45+00:00
• 02:20 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 02s)
• 02:07 mutante: started nodejs-ocg on ocg1001 (didnt listen on 8000 as opposed to ocg1002)
• 01:52 Tim: did hotfix on fluorine for incorrect udp2log conf file location

January 27

• 23:56 bblack: cp3016 out of service for now, needs reinstall (precise!)
• 23:05 mutante: rebooting terbium
• 22:52 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool pc1001 (duration: 00m 05s)
• 22:42 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool pc1001 (duration: 00m 11s)
• 20:58 andrewbogott: rebooting virt1002
• 20:40 andrewbogott: rebooting virt1001
• 20:32 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool pc1003 (duration: 00m 05s)
• 20:03 springle: reboot pc1003
• 19:58 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool pc1003 (duration: 00m 07s)
• 19:46 andrewbogott: rebooting virt1006
• 19:35 godog: (after the fact) reboot gadolinium, currently not coming back
• 19:23 mutante: brought ircd back up on argon
• 19:19 YuviPanda: run sysctl -w net.netfilter.nf_conntrack_max=131072 on labnet1001
• 19:19 YuviPanda: run sysctl -w net.netfilter.nf_conntrack_max=131072 on labmon1001
• 19:15 Krinkle: irc.wikimedia.org is down. "Connection refused."
• 19:14 Krenair: IRC RC seems broken
• 18:31 YuviPanda: rebooting tungstun
• 18:27 godog: reboot swift in esams
• 17:59 YuviPanda: rebooting labmon1001
• 17:49 godog: reboot all swift machines in eqiad, in turn
• 17:47 bblack: rebooting various LVSes...
• 17:23 marktraceur: I am consciously leaving NavigationTiming unsynced because nobody seems that concerned about it, and nobody is here to shepherd the patch. If you *are* concerned about it, then contact ori.
• 17:22 akosiaris: rebooting sca1001, sca1002, chromium, oxygen
• 17:21 logmsgbot: marktraceur Synchronized php-1.25wmf14/extensions/GWToolset/: [SWAT] [wmf14] GWToolset HHVM fixes (duration: 00m 06s)
• 17:11 logmsgbot: marktraceur Synchronized php-1.25wmf14/extensions/GWToolset/includes/: [SWAT] [wmf14] GWToolset HHVM fixes (duration: 00m 07s)
• 17:08 logmsgbot: marktraceur Synchronized wmf-config/InitialiseSettings.php: [SWAT] [config] Rename portal namespace in kowiki (duration: 00m 05s)
• 16:54 logmsgbot: marktraceur Synchronized php-1.25wmf15/extensions/GWToolset/includes/: [SWAT] [wmf15] GWToolset HHVM fixes (duration: 00m 06s)
• 04:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jan 27 04:08:16 UTC 2015 (duration 8m 15s)
• 02:19 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-01-27 02:19:11+00:00
• 02:19 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 03s)
• 02:11 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-27 02:10:49+00:00
• 02:10 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 02s)
• 00:58 logmsgbot: reedy Synchronized wmf-config/: Noop for citoid for beta (duration: 00m 05s)
• 00:50 logmsgbot: reedy Synchronized wmf-config/: Noop for bouncehandler for beta (duration: 00m 06s)

January 26

• 23:57 _joe_: depooling mw1018 for testing with user changing
• 20:00 akosiaris: restarted apache on palladium/strontium, cleared the, created on Jan 23, pid files from puppetmaster
• 19:59 _joe_: restarted apache on palladium
• 19:31 YuviPanda: restarted keystone on virt1000, ^d couldn’t log in
• 19:15 logmsgbot: reedy Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 06s)
• 04:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jan 26 04:08:01 UTC 2015 (duration 8m 0s)
• 02:23 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-01-26 02:23:10+00:00
• 02:23 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 03s)
• 02:15 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-26 02:14:49+00:00
• 02:14 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 02s)

January 25

• 20:50 ori: depooled mw1118 while investigating T85428
• 18:55 bd808: high rate of nutcracker "SYSTEM ERROR" errors on mw1118
• 18:43 bd808: trimmed Logstash redis input queues to 0 events; dropped ~4M backlogged events
• 04:14 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jan 25 04:14:33 UTC 2015 (duration 14m 32s)
• 02:23 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-01-25 02:22:58+00:00
• 02:22 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 02s)
• 02:10 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-25 02:10:30+00:00
• 02:10 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 02s)

January 24

• 22:40 bd808: Emptied logstash redis lists on all 3 hosts
• 22:27 bd808: Full restart of logstash elasticsearch cluster
• 06:27 paravoid: mass-restarting hhvm across the cluster
• 06:22 logmsgbot: faidon Synchronized wmf-config: touched config (duration: 00m 07s)
• 06:04 logmsgbot: faidon Synchronized wmf-config/StartProfiler.php: fix for T87497/r186578 (duration: 00m 06s)
• 04:52 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jan 24 04:52:42 UTC 2015 (duration 52m 41s)
• 02:30 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-01-24 02:30:10+00:00
• 02:30 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 02s)
• 02:17 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-24 02:17:08+00:00
• 02:17 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 02s)
• 01:05 hashar: restarting Jenkins (deadlock on deployment-bastion slave)
• 00:50 logmsgbot: reedy Synchronized wmf-config/: Kill wmgVectorBetaPersonalBar (duration: 00m 08s)
• 00:23 YuviPanda: begin re-imaging osmium

January 23

• 23:53 logmsgbot: reedy Synchronized wmf-config/: More config updates (duration: 00m 06s)
• 23:43 logmsgbot: reedy Synchronized wmf-config/: Config updates (duration: 00m 06s)
• 23:34 godog: halting lsearchd machines
• 23:23 _joe_: repooling 10 appservers depooled for no apparent reason, and no entry in the SAL
• 22:57 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I50931db37: Route all counter stats to tungsten:3811 (duration: 00m 07s)
• 22:57 Reedy: updateCollation.php for plwikisource complete
• 22:52 godog: restarting graphite on tungsten, webapp stuck and no graphs
• 22:49 Reedy: running mwscript updateCollation.php --wiki=plwikisource --previous-collation=uppercase in screen as reedy on tin
• 22:48 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: plwikisource collation update (duration: 00m 06s)
• 22:41 logmsgbot: reedy Synchronized 404.html: updates (duration: 00m 06s)
• 22:41 logmsgbot: reedy Synchronized docroot and w: Update docroot files (duration: 00m 05s)
• 22:38 logmsgbot: reedy Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 05s)
• 21:55 godog: removed puppet stored config for search* and searchidx*
• 21:34 _joe_: installing a new hhvm package on the canary appservers
• 21:21 _joe_: uploaded new hhvm package to apt.w.o
• 20:30 logmsgbot: rmoen Synchronized php-1.25wmf15/extensions/MobileFrontend/: updating mobilefrontend credits (duration: 00m 06s)
• 20:09 csteipp: deployed patch for T64685
• 19:44 ejegg: updated payments from 46bd1611c56d6e31594d01aa345c35dd45dcf676 to 4743b32c9091d6f2169bea3173e75aa5d2f36eb7
• 19:03 logmsgbot: ebernhardson Synchronized php-1.25wmf14/extensions/Flow: Bump flow submodule in 1.25wmf14 (duration: 00m 07s)
• 19:02 logmsgbot: ebernhardson Synchronized php-1.25wmf15/extensions/Flow/: Bump flow submodule in 1.25wmf15 (duration: 00m 08s)
• 17:59 mutante: apt-get clean, free some diskspace on antimony
• 17:17 bd808: logstash1002 elasticsearch rejoined cluster after restart
• 17:14 bd808: logstash elasticsearch cluster split brained; logstash1002 thinks it is a lone master
• 07:20 ori: restarted puppetmaster and apache2 on palladium.
• 04:03 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jan 23 04:03:45 UTC 2015 (duration 3m 44s)
• 02:19 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-01-23 02:19:45+00:00
• 02:19 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s)
• 02:11 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-23 02:11:43+00:00
• 02:11 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s)

January 22

• 23:27 legoktm: restarted the job runner service
• 19:37 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 05s)
• 18:39 ori: ran 'hdel jobqueue:aggregator:h-ready-queues:v2 LocalRenameUserJob/vewikimedia' on rdb1001
• 18:08 Krenair: Deployed patch for T87304
• 12:58 Krenair: Run https://phabricator.wikimedia.org/T87040#984282 again to stop GWToolset extension flooding commons log
• 04:15 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jan 22 04:15:02 UTC 2015 (duration 15m 1s)
• 02:19 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-01-22 02:19:19+00:00
• 02:19 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 04s)
• 02:11 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-22 02:11:09+00:00
• 02:11 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s)
• 01:25 springle: ongoing schema changes T86415 externallinks
• 01:12 akosiaris: applied base::firewall to all bastions hosts. This is bast*, iron, hooft
• 01:11 springle: dump db2019 to dbstore2001
• 01:09 springle: dump db2018 to dbstore2001
• 00:58 springle: dump db2017 to dbstore2001

January 21

• 23:07 springle: xtrabackup clone db1051 to db2016
• 20:59 _joe_: restarting hhvm on mw1192, stuck in a deadlock inside HPHP::f_ini_set ()
• 19:13 paravoid: restarting puppetmasters
• 04:39 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jan 21 04:39:39 UTC 2015 (duration 39m 38s)
• 02:39 logmsgbot: ori Synchronized README: Testing I68b5e1c2f (duration: 00m 07s)
• 02:38 logmsgbot: ori Synchronized README: Testing I68b5e1c2f (duration: 00m 02s)
• 02:28 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-01-21 02:28:50+00:00
• 02:28 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s)
• 02:16 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-21 02:16:38+00:00
• 02:16 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s)
• 02:08 logmsgbot: hoo Synchronized wmf-config/: Remove the Bug54847.php script - 2nd (duration: 00m 06s)
• 02:04 logmsgbot: hoo Synchronized wmf-config/: Remove the Bug54847.php script (duration: 00m 07s)
• 01:53 superm401: Re-ran GettingStarted populate_categories.php, also populating ptwiki for the first time.
• 01:43 logmsgbot: mattflaschen Finished scap: Turning off WikiGrok, enabling GettingStarted copyediting suggestions on ptwiki, and upgrading ContentTranslation (duration: 28m 26s)
• 01:15 logmsgbot: mattflaschen Started scap: Turning off WikiGrok, enabling GettingStarted copyediting suggestions on ptwiki, and upgrading ContentTranslation

January 20

• 23:28 logmsgbot: mattflaschen Synchronized wmf-config/InitialiseSettings.php: Enable Flow on ptwiki (duration: 00m 05s)
• 21:48 logmsgbot: krenair Synchronized php-1.25wmf14/extensions/TemplateData/modules/ext.templateDataGenerator.ui.tdDialog.js: https://gerrit.wikimedia.org/r/#/c/185999/ (duration: 00m 05s)
• 21:39 logmsgbot: krenair Synchronized php-1.25wmf15/extensions/TemplateData/modules/ext.templateDataGenerator.ui.tdDialog.js: https://gerrit.wikimedia.org/r/#/c/185998/ (duration: 00m 07s)
• 21:16 logmsgbot: krenair Synchronized php-1.25wmf15/extensions/VisualEditor/modules/ve-mw/ui/pages/ve.ui.MWSettingsPage.js: https://gerrit.wikimedia.org/r/#/c/185991/1 (duration: 00m 05s)
• 21:11 logmsgbot: krenair Synchronized php-1.25wmf14/extensions/VisualEditor/modules/ve-mw/ui/pages/ve.ui.MWSettingsPage.js: https://gerrit.wikimedia.org/r/#/c/185992/1 (duration: 00m 07s)
• 20:21 legoktm: deleted localnames/localuser entries in centralauth db that referred to vewikimedia (https://phabricator.wikimedia.org/T87264)
• 19:27 logmsgbot: reedy Finished scap: Add magic word for pl (duration: 14m 21s)
• 19:13 logmsgbot: reedy Started scap: Add magic word for pl
• 16:27 paravoid: restarting puppetmasters (palladium/strontium)
• 16:14 logmsgbot: ebernhardson Synchronized wmf-config/: tues morning swat config changes (duration: 00m 07s)
• 16:14 logmsgbot: ebernhardson Synchronized wmf-config/: tues morning swat config changes (duration: 00m 07s)
• 15:26 _joe_: started manually dumpwikidatajson on snapshot1003, in a root-owned screen session
• 15:25 _joe_: restarted apache on palladium
• 04:01 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jan 20 04:00:59 UTC 2015 (duration 0m 58s)
• 02:43 akosiaris: restarted puppet on strontium
• 02:19 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-01-20 02:19:04+00:00
• 02:19 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 03s)
• 02:11 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-20 02:10:56+00:00
• 02:10 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s)
• 01:06 paravoid: rebooting netmon1001, upstart stuck(?!)
• 01:02 paravoid: power-cycling rhenium

January 19

• 23:46 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1063 (duration: 00m 06s)
• 23:41 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1060 (duration: 00m 07s)
• 23:14 logmsgbot: reedy Synchronized wmf-config/CommonSettings.php: Remove old texvccheck config (duration: 00m 05s)
• 21:51 bd808: Updated Wikimania scholarships to 1f8d8f8 (disable PDO persistent connections; review display fixes)
• 19:40 hoo: Set email of commons user Tatobot to the email of the owning account
• 19:30 akosiaris: https://gerrit.wikimedia.org/r/185610 merged, tested on wtp1024, wtp1023, caused 0 problems, rolling out to the rest of parsoid machines
• 19:23 akosiaris: disable puppet on wtp* hosts for https://gerrit.wikimedia.org/r/#/c/185610/ merge
• 17:27 akosiaris: manually running wikidatajsondump.sh in a screen on datasets1003 after https://gerrit.wikimedia.org/r/185840 was merged
• 16:21 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1060 (duration: 00m 05s)
• 14:57 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1054, warm up (duration: 00m 05s)
• 03:56 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jan 19 03:56:15 UTC 2015 (duration 56m 14s)
• 02:20 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-01-19 02:20:40+00:00
• 02:20 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s)
• 02:12 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-19 02:12:37+00:00
• 02:12 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s)
• 00:43 andrewbogott: restarted keystone service on virt1000

January 18

• 19:39 springle Synchronized wmf-config/db-eqiad.php: depool db1054 (duration: 00m 05s)
• 04:03 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jan 18 04:03:03 UTC 2015 (duration 3m 2s)
• 02:18 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-01-18 02:17:57+00:00
• 02:17 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s)
• 02:10 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-18 02:10:18+00:00
• 02:10 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 02s)

January 17

• 06:36 YuviPanda: restarted dnsmasq on labnet1001 (see https://wikitech.wikimedia.org/wiki/Labs_DNS#DHCP_and_internal_DNS for how to)
• 04:47 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jan 17 04:47:42 UTC 2015 (duration 47m 41s)
• 02:31 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-01-17 02:30:57+00:00
• 02:30 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s)
• 02:18 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-17 02:18:24+00:00
• 02:18 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s)
• 02:17 ori: mw1148: threads in stuck in __lll_lock_wait (); restarted HHVM.
• 02:16 logmsgbot: krinkle Synchronized php-1.25wmf14/includes/content/JsonContent.php: Ic1d10393912fcefa22d (duration: 00m 06s)
• 02:16 logmsgbot: krinkle Synchronized php-1.25wmf14/resources/src/mediawiki/mediawiki.content.json.css: Ic1d10393912fcefa22d (duration: 00m 06s)
• 02:14 logmsgbot: krinkle Synchronized php-1.25wmf15/includes/content/JsonContent.php: Ic1d10393912fcefa22d (duration: 00m 05s)
• 02:14 logmsgbot: krinkle Synchronized php-1.25wmf15/resources/src/mediawiki/mediawiki.content.json.css: Ic1d10393912fcefa22d (duration: 00m 06s)
• 02:10 logmsgbot: ori Synchronized wmf-config/StartProfiler.php: I4e3871d3d: xenon: Annotate file scope and closure scope with filename (duration: 00m 05s)
• 00:13 bd808: restarted elasticserch on logstash1001 & logstash1003; OOM

January 16

• 23:33 bd808: ran `LTRIM logstash -50000 9999999` on redis queues to drop ~4M events in backlog
• 22:14 bd808: restarted elasticsearch on logstash1001; OOM errors
• 21:21 bd808: restarted elasticsearch on logstash1001
• 21:18 logmsgbot: marktraceur Finished scap: Fix UploadWizard regression and EventLogging errors (duration: 31m 06s)
• 21:17 bd808: OOM for elasticsearch on logstash1001 caused a dropped shard and icinga alerts
• 20:47 logmsgbot: marktraceur Started scap: Fix UploadWizard regression and EventLogging errors
• 20:17 logmsgbot: bd808 Synchronized wmf-config/InitialiseSettings.php: Allow wgDebugLogGroups to exclude logstash append (e808e690) (duration: 00m 05s)
• 20:17 logmsgbot: bd808 Synchronized wmf-config/logging.php: Allow wgDebugLogGroups to exclude logstash append (e808e690) (duration: 00m 07s)
• 18:13 bd808: document count not changing for logstash-2015.01.16 index
• 17:59 logmsgbot: bd808 Synchronized wmf-config/logging-labs.php: beta: Allow wgDebugLogGroups to exclude logstash append (03c3ab27) (duration: 00m 06s)
• 17:50 bblack: depooled amssq42 text cache in esams
• 17:44 ejegg: updated tools from 88b57fea517d2232e8ae906df550f426b6574f24 to 84442d51a841af4265ff103827cda83d5dd9dc54
• 17:24 logmsgbot: demon Synchronized wmf-config/: (no message) (duration: 00m 05s)
• 17:21 ejegg: updated civicrm from d648ededf5c9fc2b0ebf989300ca2037956418e3 to 4fa10ec9e3afbf65e6cbd523138cdc4b4485c482
• 17:17 logmsgbot: demon Synchronized wmf-config/: (no message) (duration: 00m 06s)
• 16:48 ottomata: finished hadoop namenode migration. Hadoop cluster is back online
• 16:48 bd808: Upgraded elasticsearch and restarted on all logstash nodes
• 16:43 bd808: shutdown whole elasticsearch cluster for logstash
• 16:39 bd808: restarted elasticsearch on logstash1001
• 16:07 ottomata: stopping hadoop cluster
• 08:01 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1051 db1056, warm up (duration: 00m 10s)
• 06:15 jgage: Icinga test of Mediawiki Apple Dictionary Bridge as https://search.wikimedia.org/?lang=en&site=wikipedia&search=Wikimedia_Foundation&limit=1 returns an error since shortly after l10n update at 02:31 UTC, though URL works without &limit=1 and end user osx dictionary lookups are still working.
• 05:54 ori: <jgage> mtr shows me packet loss between cr2-eqiad.wikimedia.org and 206.126.236.21 aka eqixva-google-gige.google.com
• 04:40 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jan 16 04:40:10 UTC 2015 (duration 40m 9s)
• 04:22 Tim: on mw1228 doing some tests to figure out why incorrect Expires header is being sent on requests for /images/*
• 03:09 logmsgbot: ori Synchronized php-1.25wmf14/includes/content/JsonContent.php: I2f4f9cb343: Let subclasses specify content model in JsonContent (duration: 00m 06s)
• 03:01 springle: xtrabackup clone db1020 to db1046
• 02:31 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-01-16 02:31:37+00:00
• 02:31 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s)
• 02:19 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-16 02:19:04+00:00
• 02:19 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s)
• 02:06 ori: EventLogging syncs were of I335ad42bb: JsonSchemaContent: Fix html rendering of objects and arrays
• 02:03 logmsgbot: ori Synchronized php-1.25wmf14/extensions/EventLogging: (no message) (duration: 00m 05s)
• 02:03 logmsgbot: ori Synchronized php-1.25wmf15/extensions/EventLogging: (no message) (duration: 00m 06s)
• 00:46 mutante: on both puppetmasters: chown gitpuppet /var/lib/git/operations/puppet/.git/logs/refs/heads/production & .git/logs/HEAD & .git/logs/refs/remotes/origin to fix puppet-merge. git pulled on strontium
• 00:46 mutante: restarted morebots

January 15

• 23:09 bd808: Updated scholarships.wikimedia.org to d598e0d
• 22:08 bd808: restarted elasticsaerch on logstash1003; died from OOM
• 21:06 subbu: deployed parsoid version 2fdf9298
• 20:38 logmsgbot: ori Synchronized wmf-config/InitialiseSettings.php: I250ecfceb: Switch all wikis to monolog logger (duration: 00m 05s)
• 20:04 bd808: logstash redis queue backlog 384k events and climbing; likely related to the elasticsearch cluster flapping
• 19:53 Coren: aborting labs filesystem move (not enough contiguous free space) and postponing until new shelf
• 18:59 YuviPanda: this works?
• 18:23 csteipp: deployed patches for T85349 T85850 T86711
• 17:26 ejegg: updated crm from bb05adf9279bd7a795906ca476e1850a85c21711 to d648ededf5c9fc2b0ebf989300ca2037956418e3
• 16:51 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 06s)
• 16:09 bd808: Deleted 2015-12-* indices from logstash elasticsearch cluster
• 16:07 logmsgbot: anomie Synchronized php-1.25wmf14/extensions/FlaggedRevs/api/actions/ApiReview.php: SWAT: Fix FlaggedRevs action=review for binary flagging gerrit:185180 (duration: 00m 07s)
• 16:01 bd808: Elasticsearch cluster for logstash has indices for events dated 2015-12-* again
• 15:49 Jeff_Green: many frack host package updates and reboots
• 10:09 logmsgbot: aude Synchronized php-1.25wmf14/extensions/Wikidata: fix noexternallanglinks bug (duration: 00m 13s)
• 08:24 qchris: Ran kafka leader re-election to bring analytics1021 back into the set of leaders
• 08:03 _joe_: restarted ES on logstash1002, not joining the cluster
• 07:47 _joe_: restarted HHVM on mw1119, all threads stuck in a lock for HPHP::RequestInjectionData::onSessionInit
• 07:45 ori: re-enabled puppet on osmium. i disabled it three hours ago to debug zhwiki key errors in memcached-serious.log.
• 07:19 YuviPanda: set home of mwdeploy to /home/mwdeploy in LDAP
• 07:09 YuviPanda: changed ldap mwdeploy user shell to /bin/bash to match puppet
• 06:21 YuviPanda: ldap mass modification: Changing everyone with shell set to sillyshell to /bin/bash
• 04:56 springle: upgrade db1051 db1056 trusty
• 04:35 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jan 15 04:35:40 UTC 2015 (duration 35m 39s)
• 03:03 logmsgbot: tstarling Synchronized wmf-config/InitialiseSettings.php: TitleBlacklist log (duration: 00m 06s)
• 02:32 logmsgbot: LocalisationUpdate completed (1.25wmf15) at 2015-01-15 02:32:40+00:00
• 02:32 logmsgbot: l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s)
• 02:26 springle: tin mw1084 sync-file failed socket error, manual sync-common
• 02:23 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1051 db1056 (duration: 00m 05s)
• 02:20 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-15 02:20:40+00:00
• 02:20 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s)
• 01:53 springle: raise mysql max_connections to 250 on virt1000. lots of nova persistent connections, little activity
• 01:26 logmsgbot: kaldari Synchronized php-1.25wmf15/extensions/Thanks/: syncing Thanks for wmf15 (duration: 00m 05s)
• 01:26 logmsgbot: maxsem Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/185106 (duration: 00m 06s)

January 14

• 22:57 logmsgbot: ori Synchronized wmf-config/StartProfiler.php: I5bd397456: Restrict "forceprofile" to requests that set X-Wikimedia-Debug header (duration: 00m 06s)
• 22:47 logmsgbot: kartik Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 05s)
• 22:40 logmsgbot: kartik Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 06s)
• 22:31 logmsgbot: reedy Synchronized php-1.25wmf14/includes/libs/virtualrest/ParsoidVirtualRESTService.php: (no message) (duration: 00m 05s)
• 22:18 logmsgbot: reedy Synchronized php-1.25wmf15/includes/libs/virtualrest/ParsoidVirtualRESTService.php: (no message) (duration: 00m 05s)
• 22:18 logmsgbot: reedy Synchronized php-1.25wmf15/includes/libs/virtualrest/ParsoidVirtualRESTService.php: (no message) (duration: 00m 05s)
• 22:13 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: CT on testwiki (duration: 00m 06s)
• 22:09 mutante: made an /a/tmp/ on fluorine to let wikidev write to for log analysis
• 22:08 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: touch (duration: 00m 06s)
• 22:03 mutante: apt-get clean on fluorine to get a little disk space
• 21:45 logmsgbot: kartik Synchronized wmf-config: Enable ContentTranslation (duration: 00m 06s)
• 21:25 subbu: reverted parsoid deploy to previous deployment (2cd6fefa) -- seeing a lot of dirty diffs post-deploy
• 21:24 logmsgbot: reedy Finished scap: Add ContentTranslation messages (duration: 29m 14s)
• 21:11 subbu: deployed parsoid version 45b0aafb (deploy sha 88525538)
• 20:55 logmsgbot: reedy Started scap: Add ContentTranslation messages
• 20:55 logmsgbot: hashar scap aborted: (no message) (duration: 00m 01s)
• 20:54 logmsgbot: hashar Started scap: (no message)
• 20:37 hashar: Restarting Zuul
• 20:36 hashar: Zuul applied Ori patch to fix a git lock contention in Zuul-cloner bug T86730 . Tagged wmf-deploy-20150114-1
• 20:32 logmsgbot: reedy Synchronized wmf-config/: Update IW cache (duration: 00m 06s)
• 20:31 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf15
• 20:30 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.25wmf14
• 20:22 logmsgbot: reedy Finished scap: testwiki to 1.25wmf15 and build l10n caches. Some extension bumps in wmf14 (duration: 19m 53s)
• 20:02 logmsgbot: reedy Started scap: testwiki to 1.25wmf15 and build l10n caches. Some extension bumps in wmf14
• 20:01 logmsgbot: reedy scap failed: CalledProcessError Command '('/usr/bin/git', 'merge-base', 'HEAD', 'gerrit\norigin')' returned non-zero exit status 128 (duration: 00m 21s)
• 20:00 logmsgbot: reedy Started scap: verbose
• 19:57 logmsgbot: reedy scap failed: CalledProcessError Command '('/usr/bin/git', 'merge-base', 'HEAD', 'gerrit\norigin')' returned non-zero exit status 128 (duration: 01m 34s)
• 19:56 logmsgbot: reedy Started scap: testwiki to 1.25wmf15 and build l10n caches. Some extension bumps in wmf14. take 2
• 19:45 logmsgbot: reedy scap failed: CalledProcessError Command '('/usr/bin/git', 'merge-base', 'HEAD', 'gerrit\norigin')' returned non-zero exit status 128 (duration: 06m 47s)
• 19:38 logmsgbot: reedy Started scap: testwiki to 1.25wmf15 and build l10n caches. Some extension bumps in wmf14
• 18:19 mutante: deleted /etc/logrotate.d/dumpwikidatajson on snapshot1003 (gerrit 182173)
• 17:30 Reedy: mw1152: Permission denied (publickey).
• 17:29 logmsgbot: reedy Purged l10n cache for 1.25wmf12
• 17:27 Reedy: deleted php-1.25wmf5 through php-1.25wmf9 from /srv/mediawiki
• 17:27 YuviPanda: sync-common && scap-rebuild-cdbs completed on virt1000, wikitech still seems up. Tending to the wounded.
• 17:23 YuviPanda: rm -rf /srv/mediawiki-staging/php-1.25wmf5 on tin
• 16:57 YuviPanda: running sync-common —verbose on virt1000
• 16:55 logmsgbot: krenair Synchronized wmf-config/wikitech.php: https://gerrit.wikimedia.org/r/#/c/184635/ (duration: 00m 06s)
• 16:48 logmsgbot: krenair Synchronized php-1.25wmf14/extensions/MultimediaViewer/resources/mmv/mmv.lightboxinterface.js: https://gerrit.wikimedia.org/r/#/c/184633/ (duration: 00m 07s)
• 16:38 logmsgbot: krenair Synchronized php-1.25wmf14: https://gerrit.wikimedia.org/r/#/c/184818/ (duration: 06m 14s)
• 15:46 ottomata: stopping all varnishkafka bits instances
• 10:15 logmsgbot: ori Synchronized wmf-config/mc.php: typo fix for nutcracker socket (duration: 00m 05s)
• 10:05 logmsgbot: ori Synchronized wmf-config/mc.php: nutcracker: use UNIX domain socket only if on HHVM (duration: 00m 07s)
• 09:57 logmsgbot: ori Synchronized wmf-config/mc.php: Memcached: make remaining app servers use UNIX domain socket (duration: 00m 06s)
• 08:24 _joe_: repooling mw1225, depooled for a long time
• 07:44 logmsgbot: ori Synchronized wmf-config/mc.php: use UNIX domain socket on mw12* app servers (duration: 00m 06s)
• 04:15 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jan 14 04:15:41 UTC 2015 (duration 15m 40s)
• 02:24 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-14 02:24:36+00:00
• 02:24 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s)
• 02:16 logmsgbot: LocalisationUpdate completed (1.25wmf13) at 2015-01-14 02:16:18+00:00
• 02:16 logmsgbot: l10nupdate Synchronized php-1.25wmf13/cache/l10n: (no message) (duration: 00m 01s)
• 01:17 logmsgbot: maxsem Synchronized wmf-config: https://gerrit.wikimedia.org/r/184812 (duration: 00m 06s)
• 01:01 logmsgbot: maxsem Synchronized wmf-config: touch (duration: 00m 08s)
• 00:59 logmsgbot: maxsem Synchronized wmf-config: touch (duration: 00m 07s)
• 00:50 logmsgbot: maxsem Synchronized wmf-config/: https://gerrit.wikimedia.org/r/184810 https://gerrit.wikimedia.org/r/184757 (duration: 00m 06s)
• 00:46 logmsgbot: maxsem Synchronized php-1.25wmf14/extensions/Wikidata/: https://gerrit.wikimedia.org/r/#/c/184807/ (duration: 00m 11s)
• 00:28 logmsgbot: maxsem Synchronized php-1.25wmf14/extensions/Wikidata/: https://gerrit.wikimedia.org/r/#/c/184807/ (duration: 00m 12s)

January 13

• 23:50 ori: Updated nutcracker on application servers to 0.4.0+dfsg-1+wm1.
• 22:52 logmsgbot: ori Synchronized wmf-config/mc.php: Use UNIX domain socket for nutcracker on mw1030 & mw1031 (duration: 00m 05s)
• 22:37 hashar: Restarted Zuul, deadlocked waiting for Gerrit
• 22:37 ejegg: updated payments from c23cf16407ef200da446d81fb990abbe429fd378 to 46bd1611c56d6e31594d01aa345c35dd45dcf676
• 20:48 Reedy: running mwscript extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki=mediawikiwiki --reindexAndRemoveOk --indexIdentifier=now
• 20:48 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: mediawikiwiki content namespaces (duration: 00m 05s)
• 20:42 logmsgbot: reedy Synchronized wmf-config/: Config updates (duration: 00m 06s)
• 20:41 logmsgbot: reedy Synchronized database lists: wikidata dblist update (duration: 00m 06s)
• 19:38 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: Add Renameuser debug log group (duration: 00m 09s)
• 19:24 logmsgbot: reedy Synchronized wmf-config/Wikibase.php: bump cache epoch (duration: 00m 06s)
• 19:09 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: monolog: enable for group0 + group1 wikis (duration: 00m 07s)
• 19:06 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non Wikipedias to 1.25wmf14
• 18:40 ori: mw1062: sync-file failed, read-only file system. Host should be removed from dsh group.
• 18:38 logmsgbot: ori Synchronized wmf-config/StartProfiler.php: xenon: Skip frames that don't have a 'phpStack' key (duration: 00m 06s)
• 17:48 hashar: If Zuul status page ( https://integration.wikimedia.org/zuul/ ) shows a lot of changes with completed jobs and the number of results growing, Zuul is deadlocked waiting for Gerrit. Have to restart it on gallium.wikimedia.org with /etc/init.d/zuul restart
• 17:39 hashar: Zuul back in action. Got recheck or +2 again the changes that have been discarded.
• 17:37 hashar: Restarting deadlocked Zuul , which drops ALL events. Reason is Gerrit lost connection with its database which is not handled by Zuul . See https://wikitech.wikimedia.org/wiki/Incident_documentation/20150106-Zuul
• 16:12 logmsgbot: demon Synchronized wmf-config/: (no message) (duration: 00m 05s)
• 16:12 logmsgbot: demon Synchronized flaggedrevs.dblist: (no message) (duration: 00m 05s)
• 16:02 YuviPanda: restart mysql on virt1000, wikitech acting up again
• 15:23 godog: upgrade and restart poolcounter on helium
• 15:17 godog: upload poolcounter 1.0.3 to precise-wikimedia
• 15:17 YuviPanda: restarted mysql on virt1000, wikitech failing with db errors. seems fine now (4 minutes ago)
• 15:17 YuviPanda: test
• 15:14 _joe_: jobrunners 100% on HHVM
• 15:14 _joe_: reimaging mw1007 mw1008
• 14:36 bblack: ssl runtime config updated for +3DES/-RC4 ( I87616455abd58c986aa960348fc20c017f097716 )
• 14:07 _joe_: reimaging mw1005,mw1006
• 13:45 Jeff_Green: package updates and reboots on many fundraising hosts
• 11:30 _joe_: reimaging mw1003, mw1004
• 11:30 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1067, warm up (duration: 00m 05s)
• 11:20 springle: upgrade db1067 trusty
• 10:30 hoo: Set email for global account "Liberipedia" as per https://phabricator.wikimedia.org/T76321
• 10:19 _joe_: reimaging mw1001, mw1002
• 10:02 _joe_: net.ipv4.tcp_tw_reuse = 1 on mw1223
• 08:58 _joe_: raising the net.ipv4.ip_local_port_range on mw1196
• 08:43 _joe_: raising the net.ipv4.ip_local_port_range on mw1230
• 07:30 springle: ongoing schema changes T86415 externallinks, codfw first
• 04:55 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1067 (duration: 00m 05s)
• 04:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jan 13 04:07:54 UTC 2015 (duration 7m 52s)
• 02:29 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-13 02:29:21+00:00
• 02:29 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 03s)
• 02:17 logmsgbot: LocalisationUpdate completed (1.25wmf13) at 2015-01-13 02:17:08+00:00
• 02:17 logmsgbot: l10nupdate Synchronized php-1.25wmf13/cache/l10n: (no message) (duration: 00m 03s)
• 00:44 logmsgbot: ori Finished scap: Updates to MobileFrontend, CentralAuth, EventLogging and WikimediaEvents (duration: 06m 27s)
• 00:38 logmsgbot: ori Started scap: Updates to MobileFrontend, CentralAuth, EventLogging and WikimediaEvents

January 12

• 23:06 ejegg: updated crm from d8a1160bca99354a856b1595cedf5c33f9ac255c to bb05adf9279bd7a795906ca476e1850a85c21711
• 21:38 hoo: Set email for global account "Carol.Christiansen" after having it confirmed by a steward and a dewiki bureaucrat (also based on old OTRS records)
• 21:12 subbu: deployed parsoid version 2cd6fefa
• 18:48 hoo: Ran sync-common on osmium
• 18:21 mutante: purging 'mlocate' package from neon as well to fix Icinga DPKG crits
• 18:04 bd808: Deployed scholarships at hash a5bc6fd
• 18:04 logmsgbot: demon Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 06s)
• 18:03 logmsgbot: demon Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 08s)
• 18:01 bd808: Applied 2015 schema changes to scholarships database on m2-master
• 17:33 hoo: mw1010: rsync: failed to set times on "/srv/mediawiki/.": Read-only file system (30)
• 17:31 logmsgbot: hoo Synchronized php-1.25wmf13/extensions/CentralAuth/: Only test passwords once in CentralAuthUser::prepareMigration - 2nd try (duration: 00m 07s)
• 17:31 logmsgbot: hoo Synchronized php-1.25wmf14/extensions/CentralAuth/: Only test passwords once in CentralAuthUser::prepareMigration (duration: 00m 06s)
• 17:31 logmsgbot: hoo Synchronized php-1.25wmf13/extensions/CentralAuth/: Only test passwords once in CentralAuthUser::prepareMigration (duration: 00m 06s)
• 17:22 mutante: restarted icinga-wm to join -releng
• 17:02 mutante: labmon1001 - purging mlocate package that was status 'rc'
• 16:30 godog: stop/start graphite-web on tungsten to clear logs
• 16:28 bd808: deleted 2014-01-* and 2015-12-* indices from logstash elasticsearch cluster
• 16:13 bd808: logs on logstash1001 reporting elasticserch connection errors; restarted logstash service
• 16:10 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable "Other projects sidebar" by default on frwiki gerrit:183288 (duration: 00m 05s)
• 16:09 bd808: logstash elasticsearch cluster has strange indices dated 2014-01-* and 2015-12-* again
• 16:05 bd808: restarted elasticsearch on logstash1001
• 16:05 logmsgbot: anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable thumbnail prerendering in production gerrit:183885 (duration: 00m 06s)
• 16:03 _joe_: depooling mw1062, disk errors
• 16:03 bd808: elasticsearch on logstash1001 not responding to http requests
• 16:01 logmsgbot: aude Synchronized wmf-config/Wikibase.php: Enable usage tracking on test.wikidata and testwiki (duration: 00m 05s)
• 16:00 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 05s)
• 15:59 bd808: logstash not showing any events at all since 2015-01-12T13:58:59.728Z
• 15:58 logmsgbot: aude Finished scap: Update Wikidata and WikimediaMessages (duration: 32m 06s)
• 15:35 hashar: Enabling test/gate of several extensions together. 180494 , RFC extensions continuous integration bug T1350
• 15:26 aude: Added and populated wbc_entity_usage table on testwiki and testwikidatawiki
• 15:25 logmsgbot: aude Started scap: Update Wikidata and WikimediaMessages
• 15:10 logmsgbot: hashar Synchronized php-1.25wmf13/extensions/Echo/tests/phpunit/includes/cache/TitleLocalCacheTest.php: php-1.25wmf14/extensions/Echo/tests/phpunit/includes/cache/TitleLocalCacheTest.php (duration: 00m 05s)
• 15:02 hashar: restarting Zuul. Deadlocked due to Gerrit database
• 12:20 _joe_: upgrading HHVM on the API cluster
• 10:45 _joe_: restarting hhvm on mw1126, stuck in HPHP::StatCache::refresh
• 10:22 _joe_: upgrading HHVM on all appservers
• 10:11 logmsgbot: hashar Synchronized wmf-config/throttle.php: Tel-Hai Academic College event - Bug: T85773 (duration: 00m 07s)
• 09:23 hashar: Restarting Zuul
• 09:06 hashar: Tweak Zuul configuration to pin python-daemon <= 2.0 and deploying tag wmf-deploy-20150112-1. bug T86513
• 07:24 andrewbogott: on virt1005 and virt1006, ran 'ln -s /usr/bin/qemu-system-x86_64 /usr/bin/kvm' that allows nova to migrate instances between hosts.
• 03:54 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jan 12 03:54:34 UTC 2015 (duration 54m 33s)
• 02:17 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-12 02:17:48+00:00
• 02:17 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s)
• 02:11 logmsgbot: LocalisationUpdate completed (1.25wmf13) at 2015-01-12 02:10:54+00:00
• 02:10 logmsgbot: l10nupdate Synchronized php-1.25wmf13/cache/l10n: (no message) (duration: 00m 01s)

January 11

• 16:44 springle: db1050 dberror log noise was https://phabricator.wikimedia.org/T86482
• 16:29 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1050, warm up (duration: 00m 05s)
• 16:08 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1050, mysqld got TERM somehow (duration: 00m 05s)
• 03:59 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jan 11 03:59:26 UTC 2015 (duration 59m 25s)
• 02:18 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-11 02:18:10+00:00
• 02:18 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 04s)
• 02:11 logmsgbot: LocalisationUpdate completed (1.25wmf13) at 2015-01-11 02:11:13+00:00
• 02:11 logmsgbot: l10nupdate Synchronized php-1.25wmf13/cache/l10n: (no message) (duration: 00m 02s)

January 10

• 23:20 _joe_: restarted the puppetmaster on palladium
• 19:44 logmsgbot: hoo Synchronized wmf-config/Bug54847.php: Don't use protected CentralAuthUser::getPasswordObject (duration: 00m 06s)
• 18:18 logmsgbot: hoo Synchronized wmf-config/Bug54847.php: Bug54847.php: Replace removed CentralAuthUser::getPasswordHash (duration: 00m 06s)
• 18:10 hoo: Ran mysql:wikiadmin@db1033 [centralauth]> DELETE FROM bug_54847_password_resets WHERE r_username = 'Tk';
• 04:03 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jan 10 04:03:52 UTC 2015 (duration 3m 50s)
• 02:24 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-10 02:24:48+00:00
• 02:24 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s)
• 02:17 logmsgbot: LocalisationUpdate completed (1.25wmf13) at 2015-01-10 02:17:29+00:00
• 02:17 logmsgbot: l10nupdate Synchronized php-1.25wmf13/cache/l10n: (no message) (duration: 00m 04s)
• 00:09 hoo: Set email of global account "Classicfilms" to the commonswiki, enwikiquote and enwiktionary accounts of the same name.

January 9

• 22:45 ejegg: updated crm from 8ded33737337d0c842d8c194b5cb15b25fc99c2e to d8a1160bca99354a856b1595cedf5c33f9ac255c
• 21:58 ejegg: enabled queue consumers
• 21:48 ejegg: disabled queue consumers
• 20:16 ottomata: stopping esams bits varnishkafka instances
• 17:19 logmsgbot: marktraceur Synchronized php-1.25wmf14/includes/filerepo/file/File.php: Remove silly debug line from File class (duration: 00m 07s)
• 17:18 logmsgbot: marktraceur Synchronized php-1.25wmf13/includes/filerepo/file/File.php: Remove silly debug line from File class (duration: 00m 08s)
• 16:58 Reedy: CREATE INDEX /*i*/br_timestamp ON /*_*/bounce_records(br_timestamp); for bounce_records on wikishared on extension1
• 15:24 Jeff_Green: deployed DNS dmarc record for wikipedia.*
• 10:19 _joe_: reimaging mw1152 as a HAT imagescaler
• 05:36 ori: repooled mw123[12]
• 05:32 logmsgbot: ori Synchronized wmf-config/mc.php: I33ff81e6a: memcached: set server address to localhost rather than 127.0.0.1 on mw123* (duration: 00m 05s)
• 04:32 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jan 9 04:32:31 UTC 2015 (duration 32m 30s)
• 03:05 springle: upgrade db1016 trusty
• 03:01 MaxSem: Running mwscript extensions/WikiGrok/maintenance/refreshCampaigns.php --wiki=enwiki --version=1 in screen session on terbium, feel free to kill if causes problems
• 02:50 logmsgbot: maxsem Synchronized php-1.25wmf13/extensions/MobileFrontend/: (no message) (duration: 00m 07s)
• 02:49 logmsgbot: maxsem Synchronized php-1.25wmf13/extensions/Mantle: (no message) (duration: 00m 07s)
• 02:42 logmsgbot: maxsem Synchronized php-1.25wmf14/extensions/MobileFrontend/: (no message) (duration: 00m 06s)
• 02:42 logmsgbot: maxsem Synchronized php-1.25wmf13/extensions/MobileFrontend/: (no message) (duration: 00m 07s)
• 02:42 logmsgbot: maxsem Synchronized php-1.25wmf13/extensions/Mantle: (no message) (duration: 00m 05s)
• 02:30 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1003 db1005 db1006 db1009. repool db1050 in s6, db1015 in s3 (duration: 00m 06s)
• 02:24 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-09 02:24:15+00:00
• 02:24 logmsgbot: l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s)
• 02:18 logmsgbot: LocalisationUpdate completed (1.25wmf13) at 2015-01-09 02:18:23+00:00
• 02:18 logmsgbot: l10nupdate Synchronized php-1.25wmf13/cache/l10n: (no message) (duration: 00m 01s)
• 01:49 logmsgbot: maxsem Synchronized php-1.25wmf13/extensions/MobileFrontend: touch (duration: 00m 09s)
• 01:25 logmsgbot: maxsem Finished scap: SWAT: MobileFrontend and WikiGrok updates (duration: 17m 36s)
• 01:07 logmsgbot: maxsem Started scap: SWAT: MobileFrontend and WikiGrok updates
• 00:53 hoo: Updated the Wikidata property suggester with data from Monday's JSON dump
• 00:46 logmsgbot: legoktm Synchronized wmf-config/InitialiseSettings.php: SWAT: https://gerrit.wikimedia.org/r/#/c/180451/ (duration: 00m 06s)
• 00:45 logmsgbot: legoktm Synchronized closed.dblist: SWAT: https://gerrit.wikimedia.org/r/#/c/180451/ (duration: 00m 08s)
• 00:42 logmsgbot: legoktm Synchronized php-1.25wmf13/extensions/CentralAuth/: SWAT: https://gerrit.wikimedia.org/r/#/c/183554/ (duration: 00m 06s)
• 00:40 logmsgbot: legoktm Synchronized php-1.25wmf14/extensions/CentralAuth/: SWAT: https://gerrit.wikimedia.org/r/#/c/183554/ (duration: 00m 06s)
• 00:37 logmsgbot: legoktm Synchronized php-1.25wmf13/extensions/Scribunto/engines/LuaCommon/lualib/mw.title.lua: SWAT: https://gerrit.wikimedia.org/r/#/c/183552/ again (duration: 00m 07s)
• 00:36 logmsgbot: legoktm Synchronized php-1.25wmf14/extensions/Scribunto/engines/LuaCommon/lualib/mw.title.lua: SWAT: https://gerrit.wikimedia.org/r/#/c/183552/ again (duration: 00m 06s)
• 00:29 logmsgbot: legoktm Synchronized php-1.25wmf13/extensions/Scribunto/engines/LuaCommon/lualib/mw.title.lua: SWAT: https://gerrit.wikimedia.org/r/#/c/183552/ (duration: 00m 07s)
• 00:27 logmsgbot: legoktm Synchronized php-1.25wmf14/extensions/Scribunto/engines/LuaCommon/lualib/mw.title.lua: SWAT: https://gerrit.wikimedia.org/r/#/c/183552/ (duration: 00m 06s)

January 8

• 23:46 logmsgbot: ori Synchronized wmf-config/mc.php: (no message) (duration: 00m 07s)
• 23:32 logmsgbot: ori Synchronized wmf-config/mc.php: Revert: I4c4691e26: memcached: use a unix socket instead of a tcp connection on selected hosts (duration: 00m 06s)
• 23:30 logmsgbot: ori Synchronized wmf-config/mc.php: I4c4691e26: memcached: use a unix socket instead of a tcp connection on selected hosts (duration: 00m 06s)
• 23:26 ori: depooling mw1230 and mw1231 for a couple of minutes for I4c4691e26
• 21:53 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: nooop to test scap update (duration: 00m 06s)
• 21:52 mutante: fixing scap permissions on mediawiki-installation servers via dsh
• 21:29 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: nooop to test scap update (duration: 00m 06s)
• 21:04 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: nooop to test scap update (duration: 00m 09s)
• 21:00 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: nooop to test scap update (duration: 00m 06s)
• 20:39 Reedy: Scap deployed at a78ddec
• 20:22 Reedy: moved /srv/deployment/scap to scap.old as git repo seems busted. Hoping puppet puts it back again correctly...
• 19:47 Reedy: scap-rebuild-cdbs finished
• 19:44 Reedy: running dsh -g mediawiki-installation -M -F 40 -- "sudo -u mwdeploy /srv/deployment/scap/scap/bin/scap-rebuild-cdbs"
• 19:43 logmsgbot: reedy Synchronized php-1.25wmf14/cache/l10n/: l10nupdate (duration: 03m 39s)
• 19:39 logmsgbot: reedy Synchronized php-1.25wmf13/cache/l10n/: (no message) (duration: 00m 05s)
• 19:38 logmsgbot: reedy Synchronized php-1.25wmf13/cache/l10n/: l10nupdate (duration: 03m 54s)
• 19:26 logmsgbot: LocalisationUpdate completed (1.25wmf14) at 2015-01-08 19:26:46+00:00
• 19:25 logmsgbot: reedy Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s)
• 19:14 logmsgbot: LocalisationUpdate completed (1.25wmf13) at 2015-01-08 19:14:37+00:00
• 19:13 logmsgbot: reedy Synchronized php-1.25wmf13/cache/l10n: (no message) (duration: 00m 01s)
• 18:58 Reedy: Attempting manual run of l10nupdate
• 17:08 logmsgbot: marktraceur Finished scap: [SWAT] [AbuseFilter] Add file_size variable (duration: 33m 27s)
• 16:51 hashar: Restarted Zuul. Same issue as https://wikitech.wikimedia.org/wiki/Incident_documentation/20150106-Zuul
• 16:35 logmsgbot: marktraceur Started scap: [SWAT] [AbuseFilter] Add file_size variable
• 16:25 logmsgbot: marktraceur Synchronized php-1.25wmf14/extensions/AbuseFilter: [SWAT] [AbuseFilter] Add file_size variable (duration: 00m 06s)
• 16:14 logmsgbot: marktraceur Synchronized php-1.25wmf14/extensions/VisualEditor/: [SWAT] VisualEditor fixes for T86046 and T86056 (duration: 00m 05s)
• 16:02 logmsgbot: marktraceur Synchronized wmf-config/logging.php: [SWAT] Honor log sampling and levels for logstash on group0 wikis (duration: 00m 05s)
• 15:44 _joe_: restarting hhvm on mw1226, TC full
• 15:37 _joe_: restarting hhvm on mw1113, stuck in parsing the ini file (HPHP::is_valid_var_name)
• 15:08 logmsgbot: aude Synchronized wmf-config/Wikibase.php: Bump cache epoch for test.wikidata (duration: 00m 06s)
• 15:01 logmsgbot: aude Finished scap: Update group0 to wmf/1.25wmf14 Wikidata extension branch (duration: 26m 43s)
• 14:35 logmsgbot: aude Started scap: Update group0 to wmf/1.25wmf14 Wikidata extension branch
• 14:34 Jeff_Green: samarium package updates and reboot
• 11:37 godog: reboot ms-be1011, xfs hosed :(
• 11:07 _joe_: repooled the canary servers
• 10:55 _joe_: installing a new hhvm package (with the correct libicu dependence) on canary hosts
• 10:02 godog: removing backend hosts from LVS for search pools
• 09:42 godog: restart uwsgi on tungsten
• 08:15 _joe_: depooled the canary appservers while a new package version is rebuilt
• 08:11 _joe_: installing a new hhvm package version on the canary pools
• 03:06 springle: xtrabackup clone db1037 to db1050
• 01:04 bd808: cleaned up logstash indices dated 2014-01-* and 2015-12-* that look to have been created by some sort of syslog input parsing bug
• 01:01 bd808: accidentally deleted 2015-01-07 logstash index when cleaning up rogue indices for 2014-01-*
• 00:40 bd808: restarted elasticsearch on logstash1002 to heal split brain in cluster
• 00:38 bd808: elasticsearch cluster for logstash is split brain.
• 00:37 logmsgbot: maxsem Synchronized php-1.25wmf13/extensions/WikiGrok/: https://gerrit.wikimedia.org/r/#/c/183186/ (duration: 00m 07s)
• 00:37 logmsgbot: maxsem Synchronized php-1.25wmf14/extensions/WikiGrok/: https://gerrit.wikimedia.org/r/#/c/183186/ (duration: 00m 08s)
• 00:34 bd808: restarted logstash on logstash1001
• 00:10 K4-713: updated payments settings

January 7

• 21:11 subbu: deployed parsoid version 904fab9e
• 19:51 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: And disable error log again (duration: 00m 06s)
• 19:49 logmsgbot: reedy Synchronized wmf-config/InitialiseSettings.php: Enabling error log for a few minutes (duration: 00m 15s)
• 19:44 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf14
• 19:42 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.25wmf13
• 19:36 logmsgbot: reedy Finished scap: testwiki to 1.25wmf14... (duration: 28m 32s)
• 19:07 logmsgbot: reedy Started scap: testwiki to 1.25wmf14...
• 18:42 godog: reboot ms-be2011, megacli in a funny state and unable to bring new drive in service
• 17:20 logmsgbot: ori Synchronized php-1.25wmf13/extensions/EventLogging/modules/ext.eventLogging.core.js: I5470424: Correct events to send schema name (duration: 00m 05s)
• 17:20 logmsgbot: ori Synchronized php-1.25wmf12/extensions/EventLogging/modules/ext.eventLogging.core.js: I5470424: Correct events to send schema name (duration: 00m 06s)
• 17:15 godog: reboot ms-be2003
• 15:36 springle: xtrabackup clone codfw slaves db2034 db2035 db2036 db2037 db2038 db2039 db2040 from other codfw slaves
• 14:40 _joe_: upgrading hhvm on testwiki
• 14:16 springle: xtrabackup clone db1027 to db1015
• 11:42 _joe_: reimaging mw1009-mw1012
• 10:19 godog: reboot ms-be2003, deleted LD should disappear
• 09:47 hashar: restarting Jenkins to resolve a deadlocks with the beta cluster jobs
• 07:53 _joe_: reimaging jobrunners mw1013-mw1016 (in batch of two)
• 06:50 springle: xtrabackup clone es2008 to es2010
• 06:50 springle: xtrabackup clone es2006 to es2007
• 02:05 logmsgbot: kaldari Synchronized php-1.25wmf13/extensions/WikiGrok/: Fixing campaign generation in WikiGrok (duration: 00m 05s)
• 01:57 logmsgbot: ori Synchronized php-1.25wmf13/extensions/EventLogging: I69c8daf: Update EventLogging for cherry-picks (duration: 00m 05s)
• 01:57 logmsgbot: ori Synchronized php-1.25wmf12/extensions/EventLogging: I07d9bc8: Update EventLogging for cherry-picks (duration: 00m 06s)
• 01:33 springle: xtrabackup clone es2008 to es2009
• 01:24 logmsgbot: kaldari Synchronized php-1.25wmf13/extensions/MobileFrontend/: Sync MobileFrontend in 1.25wmf13 for VE fix (duration: 00m 05s)
• 01:11 springle: xtrabackup clone es2006 to es2005

January 6

• 21:49 awight: update crm from 80241fd2a43f03796b416d728661470f875a590a to 8ded33737337d0c842d8c194b5cb15b25fc99c2e
• 20:24 logmsgbot: reedy Synchronized wmf-config/: (no message) (duration: 00m 06s)
• 20:16 Reedy: Ran namespaceDupes.php on ndswiktionary
• 20:16 logmsgbot: reedy Synchronized wmf-config/: Config updates (duration: 00m 06s)
• 20:00 logmsgbot: reedy Synchronized docroot and w: update noc index (duration: 00m 06s)
• 19:42 logmsgbot: demon Finished scap: mwsearch is no more (duration: 35m 40s)
• 19:06 logmsgbot: demon Started scap: mwsearch is no more
• 19:03 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: nooop
• 19:01 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.25wmf13
• 18:30 godog: upload txstatsd 1.0.0-2 to trusty-wikimedia
• 16:22 logmsgbot: anomie Synchronized php-1.25wmf13/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.MobileViewTarget.js: SWAT: Update setupToolbar signature in mobile target gerrit:182993 (duration: 00m 06s)
• 16:01 logmsgbot: anomie Synchronized wmf-config/flaggedrevs.php: SWAT: Add Flexion to $wgFlaggedRevsNamespaces for dewiktionary gerrit:180813 (duration: 00m 08s)
• 10:36 hashar: The Zuul issues have their root cause in Gerrit which cannot open ReviewDB. Filled as https://phabricator.wikimedia.org/T85916
• 10:21 hashar: Restarted Zuul. Gerrit transiently died out just like ~10 hours ago which locked Zuul entirely
• 09:41 springle: started an analytics ETL run on dbstore1002. to disable: set global event_scheduler=0;
• 09:41 godog: starting hhvm-profiler-to-carbon on tungsten T85641
• 09:16 mutante: upgraded python version on zirconium
• 09:13 logmsgbot: hashar Synchronized wmf-config/CommonSettings-labs.php: (no message) (duration: 00m 05s)
• 08:42 hashar: Zuul scheduler was stuck while reporting a change back to Gerrit waiting for data to be received. For some reason none came back and Zuul halted entirely. Restarting Gerrit killed the stalled connection and made Zuul to drop all events and resume operations.
• 08:38 andrewbogott: restarted gerrit service on ytterbium
• 08:11 hashar: Zuul stalled for some reason :(
• 08:07 andrewbogott: restarted pdns on virt1000 and labcontrol2001 to handle the change to nembus (just in case pdns is upset by change!)
• 08:07 andrewbogott: moved codfw ldap service to nembus
• 02:03 logmsgbot: aude Finished scap: Add Wikidata other projects message (duration: 37m 18s)
• 01:26 logmsgbot: aude Started scap: Add Wikidata other projects message
• 01:23 logmsgbot: aude Synchronized php-1.25wmf13/extensions/WikimediaMessages: Add Wikidata other projects message (duration: 00m 06s)
• 01:16 logmsgbot: aude Synchronized php-1.25wmf13/extensions/MobileFrontend: Fix MobileFrontend bugs (duration: 00m 06s)
• 00:57 logmsgbot: aude Synchronized php-1.25wmf12/extensions/MoodBar: Bug fixes for MoodBar (duration: 00m 09s)
• 00:49 logmsgbot: aude Synchronized php-1.25wmf13/extensions/MoodBar: Bug fixes for MoodBar (duration: 00m 06s)
• 00:44 logmsgbot: aude Synchronized wmf-config/InitialiseSettings.php: Update collabwiki bureaucrat permissions (duration: 00m 06s)
• 00:25 greg-g: restarting jenkins, hope that kicks it enough
• 00:23 awight: updated payments from 62c81d4574e5e994ff8f3cac7115eff335bd5265 to c23cf16407ef200da446d81fb990abbe429fd378
• 00:01 logmsgbot: ori Synchronized wmf-config/StartProfiler.php: I341a50bef: 'Re-enable xhprof for single-request profiling' (duration: 00m 06s)

January 5

• 22:06 subbu: deployed parsoid version 0e2997d2
• 21:40 bblack: hard rebooted wtp1020, unresponsive in every way
• 19:14 hoo: Made the ruwikinews sites table entry on wikidatawiki use https URLs rather than protocol relative ones
• 18:41 YuviPanda: imported trusty pyparsing package into precise-wikimedia
• 17:44 logmsgbot: hoo Synchronized php-1.25wmf13/extensions/Wikidata/: Update Wikibase: Fix SpecialEntityData and enhance populateSitesTable (duration: 00m 24s)
• 17:43 logmsgbot: hoo Synchronized php-1.25wmf12/extensions/Wikidata/: Update Wikibase: Fix SpecialEntityData and enhance populateSitesTable (duration: 00m 14s)
• 17:22 logmsgbot: bd808 Synchronized wmf-config/logging-labs.php: Beta logging config change (5b628827) (duration: 00m 06s)
• 17:09 logmsgbot: bd808 Synchronized wmf-config/logging-labs.php: Beta logging config change (b47ee787) (duration: 00m 06s)
• 17:00 bd808: restarted logstash on logstash1001 to see if that will make syslog events come back
• 16:58 bd808: syslog events not being recorded in logstash as expected (apache2, hhvm)
• 16:21 logmsgbot: manybubbles Synchronized php-1.25wmf13/extensions/VisualEditor/: SWAT fix switching between wikitext and VE on mobile (duration: 00m 14s)
• 16:14 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT disable creating books in the wikipedia namespace AND shuffle some upload permissions on kowiki (duration: 00m 05s)
• 16:12 logmsgbot: manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT disable creating books in the wikipedia namespace (duration: 00m 06s)
• 16:03 logmsgbot: manybubbles Synchronized wmf-config/Wikibase.php: SWAT Display links to Wikidata in the other project sidebar (duration: 00m 06s)
• 10:40 springle: xtrabackup clone: db1037 to db1061, db1039 to db1062
• 10:23 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: depool db1061 db1062 (duration: 00m 06s)
• 10:14 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: pool db1057, warm up (duration: 00m 07s)

January 4

• 17:35 springle: xtrabackup clone db1061 to db1057
• 16:49 springle: restarted zuul
• 15:59 springle: upgrade db1057 trusty
• 15:23 springle: limiting exim/otrs concurrent connections on m2-master to 250
• 14:29 springle: xtrabackup clone db1020 to db2011
• 13:49 springle: dbproxy1002 failed m2-master traffic over to m2-slave. services up. investigating cause

January 3

• 23:23 subbu: Try #2: hotfix synced to parsoid cores (to return 500 for urwiki:نام_مقامات_اے); git sha 85d8818ec1b692aaab440630a119c539d63d5ca5
• 22:38 YuviPanda: restarted parsoid on wtp1010
• 22:38 YuviPanda: restarted parsoid on wtp1006
• 22:37 YuviPanda: restarted parsoid on wtp1004
• 22:29 subbu: hotfix synced to parsoid cores (to return 500 for urwiki:نام_مقامات_اے); restart coming next
• 22:15 YuviPanda: restarted parsoid on wtp* hosts agian
• 21:19 YuviPanda: restarting parsoid on wtp* hosts again
• 20:46 YuviPanda: restarting parsoid on wtp* again
• 20:29 YuviPanda: manually restarted parsoid on wtp1012
• 20:12 YuviPanda: restarting parsoid on all wtp* hosts
• 20:06 YuviPanda: restarting parsoid on wtp1008
• 17:13 _joe_: restarting parsoid across the cluster

January 2

• 21:19 qchris: Ran kafka leader re-election to bring analytics1021 back into the set of leaders
• 11:48 godog: reboot es2004, debugging gmond stuck on start/stop
• 04:59 logmsgbot: springle Synchronized wmf-config/db-eqiad.php: repool db1061, warm up (duration: 00m 06s)
• 03:29 springle: clone and deploy es2002 es2003 es2004

January 1

• 15:19 springle: upgrade db1061 trusty

Archives

• Server admin log/Archive 1 (2004 Jun - 2004 Sep)
• Server admin log/Archive 2 (2004 Oct - 2004 Nov)
• Server admin log/Archive 3 (2004 Dec - 2005 Mar)
• Server admin log/Archive 4 (2005 Apr - 2005 Jul)
• Server admin log/Archive 5 (2005 Aug - 2005 Oct), with history 2004-06-23–2005-11-25
• Server admin log/Archive 6 (2005 Nov - 2006 Feb)
• Server admin log/Archive 7 (2006 Mar - 2006 Jun)
• Server admin log/Archive 8 (2006 Jul - 2006 Sep)
• Server admin log/Archive 9 (2006 Oct - 2007 Jan), with history 2005-11-25–2007-02-21
• Server admin log/Archive 10 (2007 Feb - 2007 Jun)
• Server admin log/Archive 11 (2007 Jul - 2007 Dec)
• Server admin log/Archive 12 (2008 Jan - 2008 Jul)
• Server admin log/2008-08
• Server admin log/2008-09
• Server admin log/Archive 13 (2008 Oct - 2009 Jun)
• Server admin log/Archive 14 (2009 Jun - 2009 Dec)
• Server admin log/Archive 15 (2010 Jan - 2010 Jun)
• Server admin log/Archive 16 (2010 Jul - 2010 Oct)
• Server admin log/Archive 17 (2010 Nov - 2010 Dec)
• Server admin log/Archive 18 (2011 Jan - 2011 Jun)
• Server admin log/Archive 19 (2011 Jul - 2011 Dec)
• Server admin log/Archive 20 (2011 Dec - 2012 Jun), with history 2007-02-21–2012-03-27
• Server admin log/Archive 21 (2012 Jul - 2013 Jan)
• Server admin log/Archive 22 (2013 Jan - 2013 Jul)
• Server admin log/Archive 23 (2013 Aug - 2013 Dec)
• Server admin log/Archive 24 (2014 Jan - 2014 Mar)
• Server admin log/Archive 25 (2014 April - 2014 September)
• Server admin log/Archive 26 (2014 October - 2014 December)