Server Admin Log

2016-09-29

21:14 ejegg: updated SmashPig from 077ffcc3c4485f62a2f9c80eb4843ef8c72d0c4f to 2169b71016e2deb1114655d0256d3286ff057943
18:25 ejegg: disabled CiviCRM dedupe jobs
18:24 bd808: https://tools.wmflabs.org/sal/ missing some entries for 2016-09-29; consider https://wikitech.wikimedia.org/wiki/Server_Admin_Log canonical
18:21 cwd: rolled forward PaymentListeners again
17:54 cwd: updated smashpig from 0d88feaf8ecab0286e36a91303bb234c68fd6384 to 077ffcc3c4485f62a2f9c80eb4843ef8c72d0c4f
17:25 cwd: rolled back PaymentListeners
17:05 cwd: updated PaymentListeners from b4d77a991e100f97d98fcd72eaf03940a4e1845d to 21647c8f4b781b74ae2dc4377334410b4eed7e3c
16:59 cwd: updated smashpig from 40c4a7c664dc53f16943aa0b83f30ab1ce435c15 to 0d88feaf8ecab0286e36a91303bb234c68fd6384
16:51 ejegg: enabled adyen job runner
16:48 cwd: updated smashpig from 3458f93599084815da46a2540e9ed762c8b120ce to 40c4a7c664dc53f16943aa0b83f30ab1ce435c15
16:34 elukey: executed 'sudo salt -C 'G@cluster:imagescaler and G@site:eqiad' cmd.run 'find /var/log/hhvm/ -type f -user root -exec chown www-data:www-data {} \;' to reduce cronspam
16:32 elukey: executed 'sudo salt -C 'G@cluster:imagescaler and G@site:codfw' cmd.run 'find /var/log/hhvm/ -type f -user root -exec chown www-data:www-data {} \;' to reduce cronspam
16:32 urandom: T133395: restbase staging: starting bootstrap of restbase-test2001-b.codfw.wmnet (test of decomm/bootstrap under time-windowed compaction)
15:18 urandom: T133395: restbase staging: decommissioning restbase-test2001-b.codfw.wmnet (test of decomm/bootstrap under time-windowed compaction)
13:29 cwd|afk: disabled Adyen job runner
10:28 hashar: Upgrading Jenkins plugins with zeljkof :]
09:32 robh: received notification of ulsfo.1.23.pdu flapping power status via united layer icinga, yet checking router shows no power interruption for cr1-ulsfo. seems to be a monitoring false alarm (from united layers end, not ours)
08:38 logmsgbot: reedy@tin Synchronized wmf-config/mobile-labs.php: Remove transfers of non existent $wmg variables (duration: 00m 48s)
02:32 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Sep 29 02:32:47 UTC 2016 (duration 4m 46s)
02:28 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.20) (duration: 12m 42s)
02:14 eileen: civicrm upgraded from 86371233a9218a526f85cd76d7008f966042fde7 to 6768613907598c52b0af1ffae35faa9078f15f63
01:04 cwd: rolled smashpig back because JSON_UNESCAPED_UNICODE is unavailable in php5.3 and don't want consumers to explode
00:56 cwd: updated SmashPig from 3458f93599084815da46a2540e9ed762c8b120ce to 3d0a76b5918f280602e5dabeeb0373c1a118590d

2016-09-28

23:51 ejegg: updated SmashPig from d2589271d43d8895fbb928f54f8feb43fcbc43c9 to 3458f93599084815da46a2540e9ed762c8b120ce
22:06 cwd: PaymentListeners rolled back
21:54 cwd: PaymentListeners updated from b4d77a991e100f97d98fcd72eaf03940a4e1845d to 21647c8f4b781b74ae2dc4377334410b4eed7
21:33 Krenair: Fixed labs 205.21.68.10.in-addr.arpa. entry to remove another broken contintcloud name, unbreaking beta scap
21:25 XenoRyet: update SmashPig from 372cd4008fee3fd02ad2eae9163cf7b28d2ef7c8 to d2589271d43d8895fbb928f54f8feb43fcbc43c9
20:51 logmsgbot: apergos is awesome and made the bot work again by restarting it
20:50 apergos: restarted logmsgbot on neon
20:47 bd808: logmsgbot seems to be down: "error: [Errno 111] Connection refused" from scap sync-file
20:46 bd808: scap sync-file wmf-config/throttle.php "IP cap lift for eswiki on 2016-09-30 (T146788)"
19:48 awight: update fundraising CRM from 6b2bd98fce5006030423bccc4f4b7fd9b5d14821 to 86371233a9218a526f85cd76d7008f966042fde7
19:22 XenoRyet: reverted SmashPig from 4b930ada57d99070713be18e235238078e1c1e48 to 372cd4008fee3fd02ad2eae9163cf7b28d2ef7c8
17:22 jynus: restarting db1069.s3 (stagnant replication)
16:42 awight: update CRM from 88000acc826592e535491bde6a47f8976deaf07a to 6b2bd98fce5006030423bccc4f4b7fd9b5d14821
08:15 twentyafterfour: twentyafterfour@iridium:/srv/phab/phabricator$ sudo bin/search index --type PhabricatorProject --force
03:34 eileen: tools upgrade from b0be0f9ca04191c4bab869bb81191c5c77c432ca to 5427a601ed64ecfe01ef84eda13f28c0dd08a3af
01:20 eileen: updating CiviCRM from 26e5214e66f2e4b5de89c27dd5121dfde43269e2 to 88000acc826592e535491bde6a47f8976deaf07a
01:05 eileen: civicrm upgrade from d30a5e454e6614214faef8354a3a23d71d573c7f to 26e5214e66f2e4b5de89c27dd5121dfde43269e2

2016-09-27

05:32 _joe_: rebooting ms-be1002, stuck in a failed disk

2016-09-26

19:37 awight: enabling awight_test5 banner at 1% of nlwiki
18:09 ejegg: rolled back paypal IPN listener to b4d77a991e100f97d98fcd72eaf03940a4e1845d
17:59 ejegg: updated standalone paypal IPN listener from b4d77a991e100f97d98fcd72eaf03940a4e1845d to 21647c8f4b781b74ae2dc4377334410b4eed7e3c
17:47 ejegg: rolled back paypal IPN listener to b4d77a991e100f97d98fcd72eaf03940a4e1845d
17:39 ejegg: updated standalone paypal IPN listener from b4d77a991e100f97d98fcd72eaf03940a4e1845d to 21647c8f4b781b74ae2dc4377334410b4eed7e3c
13:52 marostegui: phabricator is back in write mode - search is degraded. we are regenerating the indexes
13:52 chasemp: iridium phab ./bin/search index --all
03:39 cwdent_: disabled civicrm dedupe contacts job

2016-09-25

16:26 Amir1: deploying 9cc2009 on ORES nodes (T146581)
15:24 elukey: executed https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hadoop/Administration#Fixing_HDFS_mount_at_.2Fmnt.2Fhdfs on stat1002 (fusermount didn't succeed to umount though)
14:39 gehel: restarting blazegraph on wdqs1002 - T146576
14:30 gehel: putting wdqs1002 in maintenance mode, server looks unstable, investigating... - T146576
14:28 godog: temporarily redirect ores-celery-worker logs to /srv/log/celery/syslog.log and remove old daemon.log.1 to avoid scb100* filling up the / filesystem

2016-09-24

19:30 ema: hhvm 1283-1290 rolling restart
12:21 godog: apply temporary cleanup of old (+20m) thumbor temporary files - T146262
10:47 _joe_: systemctl restart thumbor-instances.service on thumbor1001 freed 3 GB of space
02:45 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Sep 24 02:44:59 UTC 2016 (duration 5m 57s)
02:39 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.20) (duration: 16m 49s)

2016-09-23

22:05 matt_flaschen: Deployed patch for T146425
21:42 logmsgbot: ebernhardson@tin Synchronized php-1.28.0-wmf.20/extensions/CirrusSearch/includes/ElasticsearchIntermediary.php: Additional logging to track down autocomplete timing regression (duration: 00m 50s)
20:52 gehel: cleaning up leftover system unit files on wdqs1*
18:41 gehel: killing stuck tilerator notification processes on maps1001 - T145534
17:57 mutante: mira restarted cron
17:53 ejegg: updated SmashPig from 8ac116037440746eaf64b9e99e1ee962d5d33475 to 372cd4008fee3fd02ad2eae9163cf7b28d2ef7c8
17:46 logmsgbot: thcipriani@tin Synchronized README: Test sync for new mira (duration: 01m 27s)
17:43 mutante: mira - changing UID of l10nupdate to 10002, chown'ing files (1001 -> 10002)
17:35 logmsgbot: ebernhardson@tin Synchronized php-1.28.0-wmf.20/extensions/CirrusSearch/includes/ElasticsearchIntermediary.php: Add timing marks to narrow down autocomplete timing regression (duration: 00m 50s)
17:31 logmsgbot: ebernhardson@tin Synchronized php-1.28.0-wmf.20/extensions/CirrusSearch/includes/CompletionSuggester.php: Add timing marks to narrow down autocomplete timing regression (duration: 18m 43s)
17:04 mutante: stat1002 - before it was hanging and then fixed due to https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hadoop/Administration#Fixing_HDFS_mount_at_.2Fmnt.2Fhdfs
17:03 mutante: stat1002 - starting nagios-nrpe-server
14:55 jynus: deployed dns update (removing db1010) T129395
12:20 moritzm: rearmed keyholder on mira
12:03 _joe_: rolling restart of mw1280-90, high cpu usage due to memory leaks.
10:16 moritzm: reimaging mira to jessie (again, previously installer config still pointed to trusty)
10:05 Amir1: ladsgroup@terbium:~$ mwscript extensions/ORES/maintenance/PopulateDatabase.php --wiki=wikidatawiki (T146461) and for 'trwiki', 'plwiki', 'fawiki', 'nlwiki', 'ruwiki', 'ptwiki'
10:00 Amir1: ladsgroup@terbium:~$ mwscript extensions/ORES/maintenance/PopulateDatabase.php --wiki=enwiki
09:58 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.20/extensions/ORES/includes/Cache.php: No int typehinting (causes jobs to crash) T146461 (duration: 00m 42s)
09:58 moritzm: rearmed keyholder on mira
09:48 jynus: disabling alerts and shutting down db1010 in preparation for decommissioning T129395
09:08 moritzm: reimaging mira to jessie
09:06 elukey: reboot eventlog2001.codfw.wmnet for kernel upgrades
08:52 elukey: upgrading varnishkafka to 1.0.12-1 in cache:misc
08:44 ema: depooled nginx restart on cp4003 and cp1045 for libssl upgrade
08:30 elukey: upgrading varnishkafka to 1.0.12-1 in cache:maps
07:34 elukey: executed 'find /var/log/hhvm/ -type f -user root -exec chown www-data:www-data {} \;' for all the api and appservers to remove/prevent cronspam (root:adm files also related to new reimaged hosts, Rsyslog needs to be configured before hhvm) - T132324
07:02 moritzm: rebooting francium for kernel security update
04:03 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.20/includes/deferred: 5af1b93db1bb3d14844c55e4e3ed17fe963de551 (duration: 00m 48s)
04:02 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.20/includes/libs/rdbms: 5af1b93db1bb3d14844c55e4e3ed17fe963de551 (duration: 00m 51s)
02:46 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Sep 23 02:46:04 UTC 2016 (duration 6m 10s)
02:39 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.20) (duration: 17m 04s)
02:13 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.20/extensions/SecurePoll/: https://gerrit.wikimedia.org/r/#/c/312450/1 (duration: 00m 51s)
02:10 mutante: mw1206, mw1224 - restarted hhvm and apache
01:49 bblack: depooled mw1224 service apache2
00:38 Krenair: mw1224 apache stuck, not restarting for now in case someone wants to investigate later. possibly T89912?
00:17 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/312339 (duration: 00m 48s)
00:16 logmsgbot: krenair@tin Synchronized wmf-config/mobile.php: https://gerrit.wikimedia.org/r/312339 (duration: 00m 47s)
00:08 logmsgbot: krenair@tin Synchronized php-1.28.0-wmf.20/extensions/FlaggedRevs/business/RevisionReviewForm.php: https://gerrit.wikimedia.org/r/#/c/312423/ (duration: 00m 48s)

2016-09-22

23:46 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/310483 (duration: 00m 48s)
23:19 logmsgbot: krenair@tin Synchronized php-1.28.0-wmf.20/resources/src/mediawiki.less/mediawiki.ui/mixins.less: https://gerrit.wikimedia.org/r/#/c/312340/ (duration: 00m 48s)
22:49 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.20/includes/libs/rdbms/loadbalancer/LoadBalancer.php: a73a7ef9286275f797411646f9c5af60d4894c73 (duration: 01m 04s)
22:18 mutante: added slaporte and zhousquared to wmf LDAP group (T146227)
21:24 hasharAway: Nodepool is all back and operational. Reduced amount of queries to the OpenStack API by more than 10%
21:14 yurik: deployed tilerator https://gerrit.wikimedia.org/r/#/c/312329/
21:05 hasharAway: stopped nodepooled and restarted it with 0.1.1-wmf5
21:04 mutante: upgraded nodepool to 0.1.1-wmf5 on labnodepool1001
21:04 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.20/resources/src/mediawiki/mediawiki.js: T146099 (duration: 00m 48s)
21:02 mutante: imported nodepool_0.1.1-wmf5_amd64 into jessie-wikimedia (T145142)
20:52 urandom: T133395: RESTBase Staging: starting dumps (3, eqiad)
20:47 urandom: T133395: RESTBase Staging: altering table to set TWCS on wikipedia parsoid.html table
20:20 urandom: T133395: RESTBase Staging: Restarting Cassandra to pick up TWCS jar in classpath
20:08 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.20
20:00 thcipriani: rolling out wmf.20 to all wikis
19:23 SMalyshev: Deploying new version of WDQS GUI
19:09 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.20
19:01 thcipriani: wmf.20 to group1 will watch until 20 UTC and move forward to all wikis
18:56 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.18/extensions/CentralNotice: SWAT: Update extensions/CentralNotice submodule (T144952) (duration: 00m 50s)
18:50 yuvipanda: enabling puppet on labcontrol1001, run on labtestcontrol2001 seems ok
18:47 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.20/extensions/CentralNotice: SWAT: Update extensions/CentralNotice submodule (T144952) (duration: 00m 52s)
18:46 yuvipanda: disable puppet on labcontrol1001 for https://gerrit.wikimedia.org/r/#/c/312301/
18:38 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.20/includes/libs/rdbms/database/Database.php: 844cfd568a7c7953faa6ac69acebff1cee943b7f & 014a420b4525798b1202cc488b337acdaf09c49a (duration: 00m 49s)
18:31 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.19/extensions/TimedMediaHandler/MwEmbedModules: SWAT: Update ogv.js to 1.2.0 (T145983) (duration: 00m 48s)
18:28 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.18/extensions/TimedMediaHandler/MwEmbedModules: SWAT: Update ogv.js to 1.2.0 (T145983) (duration: 00m 51s)
17:24 moritzm: rebooting ms-be1016, high load caused by XFS bug
17:09 moritzm: rolling reboot of trusty swift backend servers in eqiad completed
16:40 elukey: forced logrotation for /etc/logrotate.d/upstart on labvirt1014 to investigate cronspam
16:17 godog: offline sdd on ms-be1004 via megacli T144499
15:29 mobrovac: restbase deploy end of d96fbc1
15:10 mobrovac: restbase deploy start of d96fbc1
15:02 bblack: upgrading openssl on cp*
13:22 moritzm: resume rolling reboot of trusty swift backend servers in eqiad for kernel security update
13:02 moritzm: uploaded openssl 1.0.2i for jessie-wikimedia to carbon
12:31 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.20/extensions/Popups: Merge mw.popups.experiment into mw.popups.core T146035 (duration: 00m 49s)
12:27 mobrovac: restbase deploy end of d5538ad
12:25 elukey: installing varnishkafka 1.0.12 on cache:upload ulsfo and eqiad
12:24 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-es-ro_0.7.3~r57551-2+wmf1
12:13 hashar: Early SWAT for mobile team ( https://gerrit.wikimedia.org/r/#/c/311977/ )
12:11 mobrovac: restbase deploy start of d5538ad
11:34 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: Repool db1082 with some light weight (duration: 00m 52s)
10:24 moritzm: rolling reboot of trusty swift backend servers in eqiad for kernel security update
09:59 moritzm: rebooting subra/suhail for kernel security update
09:50 hashar: updated jobrunner code to a0e82166 (tweak errors reporting in logs) | Does not include 51014242 "Batch stats to statsd" (poke addshore )
09:19 gehel: upgrade / restart of elasticsearch eqiad cluster done T145404 / T146123
09:02 elukey: installing varnishkafka 1.0.12 on cache:upload codfw
08:49 marostegui: Deploying schema change on S7 master - T141951
08:43 elukey: installing varnishkafka 1.0.12 on cache:upload esams
08:40 elukey: installed varnishkafka 1.0.12 on cp1099
08:35 elukey: restarted varnishkafka on cp1099 (log abandoned )
08:19 hashar: Cleanup jobrunner list of minions in redis ( "deploy:jobrunner/jobrunner:minions" )
08:09 hashar: Resyncing all jobrunner deployment installations since only 41/68 minions have completed fetch/checkout
08:01 elukey: rolling restart of the whole Analytics Hadoop cluster for kernel upgrades (analytics* hosts)
07:58 elukey: uploaded varnishkafka 1.0.12-1 to reprepro
07:52 elukey: rebooted stat100[23] for kernel upgrades
07:40 moritzm: rolling restart of trusty swift frontend servers in codfw for kernel security update
07:33 elukey: rebooting stat1004 for kernel upgrades
06:45 elukey: Puppet disabled on analytics1027 to stop periodic Java daemons (prep step for Hadoop cluster reboots)
03:18 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Sep 22 03:18:48 UTC 2016 (duration 6m 48s)
03:12 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.20) (duration: 17m 08s)
02:38 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.18) (duration: 16m 39s)
02:03 eileen: all back on
01:50 eileen: turned off jenkins jobs to run dbupdate: dedupe(s) thank-you & donate import
01:40 eileen: update civicrm from 5393a13727e8dbad05ffef9ddd44e965eb9282d1 to d30a5e454e6614214faef8354a3a23d71d573c7f
01:35 twentyafterfour: reboot successful, iridium is back online
01:28 twentyafterfour: Rebooting iridium to apply kernel update
00:39 awight: update paymentswiki from d572ee9e3ef0a97c044f02e9866469a8f3fa5858 to cc27f83f31ecc609d4400050e73905b7364f1d42; mirror unsubscribe queue

2016-09-21

23:32 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Blacklist minerva from showing Related Articles in the footer (T144912, currently no-op) (duration: 00m 47s)
23:31 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Blacklist minerva from showing Related Articles in the footer (T144912, currently no-op) (duration: 00m 49s)
22:57 mobrovac: change-prop deploying ea8cdf8
22:35 logmsgbot: awight@tin Synchronized wmf-config/InitialiseSettings.php: Add CentralNotice debug log bucket for T144952 (duration: 00m 48s)
22:33 logmsgbot: awight@tin Synchronized php-1.28.0-wmf.20/extensions/CentralNotice: Correct CentralNotice logging for T144952 (duration: 00m 51s)
22:31 logmsgbot: awight@tin Synchronized php-1.28.0-wmf.18/extensions/CentralNotice: Correct CentralNotice logging for T144952 (duration: 00m 51s)
22:08 cwd: updated SmashPig from f308ba490635c75b6fe735e7dc1c558250202365 to 8ac116037440746eaf64b9e99e1ee962d5d33475
20:39 bearND: deployed mobileapps bf6943b
20:36 yurik: deployed kartotherian geoshape lines support - https://gerrit.wikimedia.org/r/#/c/312097/
20:35 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.28.0-wmf.20
20:31 bearND: starting mobileapps deploy
20:24 logmsgbot: thcipriani@tin Finished scap: testwiki to php-1.28.0-wmf.20 and rebuild l10n cache (duration: 52m 01s)
20:17 arlolra: updated Parsoid to version a802de0
20:05 arlolra: starting Parsoid deploy
19:32 logmsgbot: thcipriani@tin Started scap: testwiki to php-1.28.0-wmf.20 and rebuild l10n cache
19:24 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.18/resources/src/mediawiki/mediawiki.js: T146099 (duration: 01m 41s)
17:41 bblack: bits.wikimedia.org hostname removed from DNS (if related real complaints/problems occur, revert https://gerrit.wikimedia.org/r/305533 )
16:46 Krenair: running P3833 script against designate to clean up existing T120797 mess
16:46 mobrovac: restbase deploy end of a75510d
16:30 mobrovac: restbase deploy start of a75510d
15:03 moritzm: installing wireshark security updates
13:59 akosiaris: disabled puppet on neon, puppet migration in progress
13:48 hashar: European SWAT completed
13:45 logmsgbot: hashar@tin Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 46s)
13:44 logmsgbot: hashar@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 46s)
13:43 logmsgbot: hashar@tin Synchronized wmf-config/InitialiseSettings-labs.php: (no message) (duration: 00m 48s)
13:39 logmsgbot: hashar@tin Synchronized wmf-config/mobile.php: For phuedx or is that for yurik? (duration: 00m 47s)
13:37 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.18/extensions/Kartographer: For yurik or phuedx? :D (duration: 00m 48s)
13:37 gehel: adding planet_osm_lines and roads indexes on maps*
13:34 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.19/extensions/Wikidata: (no message) (duration: 02m 22s)
13:31 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.19/extensions/Kartographer/: (no message) (duration: 00m 50s)
13:17 logmsgbot: hashar@tin Synchronized wmf-config: (no message) (duration: 00m 49s)
13:13 logmsgbot: hashar@tin Synchronized wmf-config: New wikitext editor: Enable the Beta Feature in Beta Cluster (duration: 00m 50s)
13:10 logmsgbot: hashar@tin Synchronized wmf-config: New wikitext editor: Enable the Beta Feature in Beta Cluster (duration: 00m 51s)
12:30 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: Repool db1094 after the ALTER table - T141951 (duration: 00m 47s)
12:09 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: Depool db1094 for an ALTER table - T141951 (duration: 00m 47s)
11:45 moritzm: rolling restart of trusty swift backend servers in codfw for kernel security update
11:25 mobrovac: restbase deploy end of ca55669
11:16 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: Repool db1086 after the ALTER table - T141951 (duration: 00m 47s)
11:07 mobrovac: restbase deploy start of ca55669
11:04 marostegui: Rebuilding tables in db1082 (non pooled) - T137191
11:03 elukey: adding mw1197 back to serving live traffic after the reimage
10:51 elukey: restarted varnishkafka on cp1048 (VSLQ_Dispatch: Varnish Log abandoned or overrun.)
10:45 elukey: adding mw1196 back to serving live traffic after the reimage
10:06 moritzm: rebooting lithium for kernel security update
09:39 moritzm: reimaging mw1173-mw1175 to jessie
09:26 marostegui: Stopping mysql at db1019 for a few days as it will be decommissioned - T146265
09:19 gehel: powercycling elastic1027 - T145404
09:09 godog: reimage bast3001.wikimedia.org with separate /srv
08:31 marostegui: schema change on S7 - T141951
08:30 elukey: reimagining mw1196-7 to jessie
08:29 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: Depooling db1086 for an alter table - T141951 (duration: 00m 49s)
07:19 elukey: Moved some hhvm logs (/var/log/hhvm) from root:adm to www-data:www-data on mw127[678] to remove cronspam (T132324)
07:18 moritzm: reimaging mw1170-mw1172 to jessie
06:59 marostegui: dropping tables in S1,S3,S4 - T54924
06:21 elukey: removing aqs100[123] from live traffic - aqs.svc.eqiad.wmnet - T144497
06:03 awight|afk: update SmashPig from 6651835c816ce62112cbfea96863e4244f133c17 to f308ba490635c75b6fe735e7dc1c558250202365
03:58 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.18/resources/src/mediawiki/mediawiki.js: I221cd6c2b (duration: 00m 46s)
03:56 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.18/resources/src/mediawiki/mediawiki.requestIdleCallback.js: I221cd6c2b (duration: 00m 48s)
03:54 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.19/resources/src/mediawiki/mediawiki.requestIdleCallback.js: I221cd6c2b (duration: 00m 47s)
03:31 awight: update SmashPig from 4530dc9113a83bc5492a7da6f8e9ee3a6c789cda to 6651835c816ce62112cbfea96863e4244f133c17
03:13 awight: update SmashPig from f1f55096ac20899fa9700aa0f80dece5b47a7fd3 to 4530dc9113a83bc5492a7da6f8e9ee3a6c789cda
02:46 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Sep 21 02:46:22 UTC 2016 (duration 7m 11s)
02:45 mutante: thumbor1002 moved nginx access logs to /srv for more space on /
02:39 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.18) (duration: 16m 28s)
02:11 mutante: thumbor1001/1002 - moved logs from /var/log/thumbor to /srv/thumborlogs to free some space, the actual issue is in /tmp though. lots of systemd-private-* dirs with large sizes. like https://bugzilla.redhat.com/show_bug.cgi?id=1183684 ?
01:58 awight: update SmashPig from 285d8ce25f22d8538c693c1b3c3e7ce9e688f22c to f1f55096ac20899fa9700aa0f80dece5b47a7fd3
01:51 mutante: thumbor servers ran out of disk space
01:15 awight: updated SmashPig to 285d8ce25f22d8538c693c1b3c3e7ce9e688f22c
00:45 eileen: jobs enabled again - update failed to run due to trigger - which didn't affect staging maybe not on - will edit & try again

2016-09-20

23:49 eileen: disabled Dedupe CiviCRM contacts
23:47 eileen: disabled Project Dedupe Major gifts contacts
23:47 eileen: disabled CiviCRM contacts (high numbers)
23:46 eileen: disabled Thank you mail send
23:46 eileen: disabled Donations queue consume
23:45 eileen: CiviCRM update from de1df9ec4fe58487eeb61cc69160f228e009f2cf to 5393a13727e8dbad05ffef9ddd44e965eb9282d1
23:35 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.18/resources/src/mediawiki/mediawiki.js: Always use requestIdleCallback polyfill for batchEval (T146099) (duration: 00m 46s)
23:22 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.19/extensions/Echo/: SWAT (duration: 00m 55s)
23:20 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.19/extensions/TimedMediaHandler: SWAT (duration: 00m 50s)
23:17 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.18/extensions/TimedMediaHandler: SWAT (duration: 00m 50s)
22:26 mobrovac: change-prop deploying 4417255
21:41 mutante: powercycled mw1294 (down, frozen console)
21:26 thcipriani: starting branch cut for wmf.20
21:17 chasemp: rsync initial transfer of others on labstore1001 to labstore1004
21:08 awight: update SmashPig from db68be988194c960aebca691d0fd8e6a6d24246a to 285d8ce25f22d8538c693c1b3c3e7ce9e688f22c
20:26 Pchelolo: restbase deploy ca41acd3f
20:20 Pchelolo: restbase deploy ca41acd3f canary on restbase1007
20:13 Pchelolo: restbase deploy ca41acd3f to staging
20:07 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.18/extensions/ProofreadPage/modules/page/ext.proofreadpage.page.edit.js: Initializes the zoom widget after page loading (duration: 00m 47s)
19:51 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.18/extensions/ProofreadPage/modules/page/ext.proofreadpage.page.edit.js: Makes sure that the zoom widget is initialized before zooming in/out - https://gerrit.wikimedia.org/r/#/c/311765/ (duration: 00m 48s)
17:47 Pchelolo: update RESTBase to 4829630f
17:27 Pchelolo: update RESTBase to 4829630f canary on restbase1007
17:01 elukey: adding aqs1006 to live traffic - aqs.svc.eqiad.wmnet - T144497
16:58 elukey: adding aqs1005 to live traffic - aqs.svc.eqiad.wmnet - T144497
16:48 gehel: increase recovery bandwidth on elasticsearch eqiad to match codfw - T145404
16:32 elukey: restarting cassandra on aqs100[56] (started the work earlier on today, stopped due to T146130)
14:31 dcausse: restarting relforge100[12].eqiad.wmnet servers for kernel upgrade and java settings change
14:23 moritzm: installing tomcat security updates on Ubuntu servers
13:41 jynus: disabling puppet on labtestweb2001
13:40 ottomata: merged --until flag change in check_graphite script (this could affect all graphite based alerts)
13:37 zeljkof: executed script: mwscript maintenance/updateArticleCount.php --wiki=wikidatawiki --update
13:34 zeljkof: EU SWAT finished
13:28 gehel: restarting for elasticsearch and kernel upgrade - eqiad cluster - T145404 / T146123
13:05 logmsgbot: zfilipin@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Change $wgArticleCountMethod in Wikidata from default (link) to any (T144687) (duration: 00m 47s)
11:41 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: (no message) (duration: 00m 46s)
11:38 moritzm: upgrading ganeti2001 to Linux 4.4 (ganeti2006 has been promoted to new master node)
11:05 moritzm: upgrading ganeti2006 to Linux 4.4
10:49 moritzm: upgrading ganeti2005 to Linux 4.4
10:44 moritzm: reimaging app servers mw1240-mw1242 and API servers mw1194/mw1195 to jessie
10:43 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: (no message) (duration: 00m 48s)
10:36 moritzm: upgrading ganeti2004 to Linux 4.4
10:28 akosiaris: force mw2232 to use palladium for report handler testing
10:25 jynus: deploying schema change on s1 hosts T139090
10:15 moritzm: upgrading ganeti2003 to Linux 4.4
10:12 mobrovac: change-prop deploying e1ef51e
09:57 moritzm: upgrading ganeti2002 to Linux 4.4
09:13 moritzm: reimaging API servers mw1192/mw1193 to jessie
08:56 moritzm: reimaging mw1243-mw1245 to jessie
07:36 elukey: restart cassandra on aqs100[456] for T130861 - only aqs1004 is taking live traffic
02:47 eileen: CiviCRM update from c9157ba6aac3ad3c2fd61c7bb0475851fb8ec421 to de1df9ec4fe58487eeb61cc69160f228e009f2cf
02:40 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Sep 20 02:40:17 UTC 2016 (duration 6m 52s)
02:33 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.18) (duration: 10m 35s)
00:43 mutante: wtp2019 - down again, powercycled, probably damaged RAM

2016-09-19

23:22 Pchelolo: restart restbase in staging
23:01 awight: update orphan rectifier config for T145848
22:58 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: Restore testwiki to 1.28.0-wmf.18
22:53 dapatrick: Deployed patch for T144573 to wmf18 and wmf19
22:48 logmsgbot: thcipriani@tin Finished scap: testwiki to php-1.28.0-wmf.19 and rebuild l10n cache (duration: 52m 03s)
22:31 bblack: cache_upload: pooling cp1099 (storage experiment - T145661)
22:02 awight: update paymentswiki from 4cd18776ca67fe0db04b4173cb47f39dda59f43b to d572ee9e3ef0a97c044f02e9866469a8f3fa5858
22:00 bblack: cp1099: depooling varnish backends for storage size experimentation
21:56 logmsgbot: thcipriani@tin Started scap: testwiki to php-1.28.0-wmf.19 and rebuild l10n cache
21:41 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: Revert group0 wikis to 1.28.0-wmf.19
21:30 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 wikis to 1.28.0-wmf.19
20:40 hoo: Removed today's Wikidata json dumps: All shards succeeded, but final dump composition apparently failed.
20:14 chasemp: reboot labstore1004
20:08 Pchelolo: restbase deploy 4829630f staging
18:49 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: SWAT: Throttle for RCL (T145838) (duration: 00m 47s)
18:43 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: ORES default threshold to high for wikidatawiki (T144784) (duration: 00m 47s)
18:32 jynus: emergency/unscheduled restart of mariadb @ labsdb1003 - close to OOM, unusable
18:14 thcipriani: ran on terbium: mwscript extensions/ShortUrl/populateShortUrlTable.php --wiki=bdwikimedia
17:47 awight: reenable banner history queue consumer
17:46 awight: update civicrm from 1df25962834885121bd7a9ba856c0c69c2b9cfda to c9157ba6aac3ad3c2fd61c7bb0475851fb8ec421
17:39 ejegg: updated payments-wiki from 392d67520d14998d61823755b22df50ab45afb35 to 4cd18776ca67fe0db04b4173cb47f39dda59f43b
17:36 Krenair: Reset wikitech/horizon 2fa for Greg per request
15:50 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-nno-nob_1.1.0~r66076-1+wmf1
15:50 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-br-fr_0.5.0~r61325-1+wmf1
14:40 chasemp: testing nfs export performance on labstore1004/1005 cluster
14:35 yurik: depl graphoid https://gerrit.wikimedia.org/r/#/c/311374/
14:28 moritzm: uploaded apache 2.4.10-10+deb8u7+wmf1 for jessie-wikimedia to carbon
14:21 elukey: adding aqs1004 to live traffic - aqs.svc.eqiad.wmnet - T144497
14:10 moritzm: reimaging mw1246-mw1248 to jessie
14:10 hashar: European SWAT is complete.
14:05 zeljkof: EU SWAT finished
14:05 logmsgbot: zfilipin@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add WT namespace alias to NS_PROJECT in mywiktionary (T140998) (duration: 00m 47s)
14:01 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.19/extensions/Graph: Fixed wikiraw: protocol bug T146010 (duration: 00m 47s)
14:00 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.18/extensions/Graph: Fixed wikiraw: protocol bug T146010 (duration: 00m 48s)
13:58 logmsgbot: zfilipin@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable WikidataPageBanner on itwikiwoyage (T145328) (duration: 00m 48s)
13:55 hashar: Europe SWAT extended as we still have some patches to process
13:47 logmsgbot: zfilipin@tin Synchronized wmf-config/throttle.php: SWAT: Throttling rule for RCL (T145838) [throttle] Allow the same number accounts Throttle for RCL (duration: 00m 47s)
13:18 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.19/includes/api/ApiQueryBacklinksprop.php: API: Force straight join for prop=linkshere|transcludedin|fileusage T145079 (duration: 00m 47s)
13:18 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.18/includes/api/ApiQueryBacklinksprop.php: API: Force straight join for prop=linkshere|transcludedin|fileusage T145079 (duration: 00m 50s)
12:59 moritzm: installing wget updates from jessie 8.6 point update
12:51 elukey: adding mw1191 back to serving traffic after reimage
11:43 mobrovac: restbase cassandra truncating local_group_wikipedia_T_feed_aggregated.data
11:32 jynus: rebooting again db1061 for upgrade
11:13 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: Temporarily depool db1062 and repool db1034, in order to be able to ALTER a large table. T141951 (duration: 00m 48s)
10:49 moritzm: reimaging mw1249, mw1250, mw1258 to jessie
10:23 jynus: powercycle db1061, unresponsive since ~1am
09:20 moritzm: invalidated squid cache on carbon
08:49 akosiaris: increase /var/lib/puppet to 50GB on puppetmaster1002, puppetmaster2001, puppetmaster2002
08:48 logmsgbot: addshore@tin Synchronized wmf-config/CommonSettings.php: 311118 NOOP Some inline comments added (duration: 00m 58s)
08:04 marostegui: renaming tables in S1, S4 and S4 in eqiad before dropping them T54924
07:50 elukey: reimaging mw1191.eqiad.wmnet to jessie
07:49 moritzm: installing updates for file/libmagic from jessie 8.6 point update
07:42 moritzm: reimaging mw1255-mw1257 to jessie
02:28 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Sep 19 02:28:46 UTC 2016 (duration 5m 4s)
02:23 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.18) (duration: 10m 39s)

2016-09-18

20:37 bblack: restart upload backend: cp3036
20:22 bblack: restart upload backend: cp3039
19:58 bblack: restart upload backend: cp1074 (stats indicate LRU_Fail imminent)
19:28 bblack: restart upload backend: cp1064 (already in LRU_Fail, caught early)
19:28 bblack: restart up
18:06 bblack: restart upload varnish backend: cp1050 (already in LRU_Fail)
17:58 bblack: restart upload varnish backend: cp2008
17:42 bblack: restart upload varnish backend: cp1071
17:32 bblack: restart upload varnish backend: cp2020
17:06 bblack: restart upload varnish backend: cp2026
16:35 bblack: restarting upload varnish backend: cp2011
15:43 bblack: restarting upload varnish backend: cp2017
15:13 bblack: restarting upload varnish backend: cp2005
14:52 bblack: restarting upload varnish backend: cp1049
14:42 bblack: restarting upload varnish backend: cp2022
14:17 bblack: restarting varnish backend on cp1073 (503 LRU_Fail pattern, has been up a few days...)
13:29 bblack: disabling puppet on cp1074, to experiment with vhtcpd regex filter
11:17 ema: repooling varnish-be in codfw
11:00 ema: varnish-backend restart on cp3037
10:58 ema: varnish-backend restart on cp3044
10:54 ema: repooling varnish on cp1050
10:53 ema: repooling varnish on cp1062
10:52 ema: repooling varnish on cp1064
10:50 ema: repooling varnish on cp1071
10:50 ema: repooling varnish on cp1072
10:49 ema: repooling varnish on cp1073
10:49 ema: repooling varnish on cp1074
09:47 _joe_: varnish-backend-restart on cp1063
09:30 _joe_: varnish-backend-restart on cp1048
06:09 ejegg|afk: updated civicrm from 5ba6976f2552564b51085abff0afd5f76195229b to 1df25962834885121bd7a9ba856c0c69c2b9cfda
02:32 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Sep 18 02:32:49 UTC 2016 (duration 5m 58s)
02:26 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.18) (duration: 10m 18s)

2016-09-17

16:40 _joe_: rolling restart of HHVM on part fo the API cluster in eqiad, T133674
08:15 _joe_: enlarged puppet partition on puppetmaster1001, rendered full by reports
07:06 p858snake: set +z on -operations, allows messages sent by +b or +q users (normally blocked) to be seen by users that currently op'ed
06:55 p858snake: see T145924 or email to ops list for more info
06:54 p858snake: silenced (+q) icinga-wm in operations channel, due to channel spam from low disk space on puppetm1001
02:34 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Sep 17 02:34:33 UTC 2016 (duration 7m 3s)
02:27 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.18) (duration: 11m 40s)

2016-09-16

23:09 mutante: titanium - shutdown -h now
21:23 logmsgbot: aaron@tin Synchronized wmf-config/InitialiseSettings.php: Set some database logging groups to log (duration: 00m 47s)
20:34 logmsgbot: reedy@tin Synchronized wmf-config/: Load CN via extension registration. Only load jsonconfig once (duration: 00m 56s)
20:28 logmsgbot: reedy@tin Synchronized wmf-config/extension-list: Couple more to extension.json (duration: 00m 47s)
19:32 mutante: fermium disabled puppet again
19:31 mutante: fermium starting mailman qrunner (T144933)
19:26 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.19/includes/jobqueue/JobQueueGroup.php: 01254b0a72a8619117ecad103427e2431e89cc52 (duration: 00m 47s)
18:11 mutante: fermium - re-enabled puppet (after merging gerrit 310746(
17:55 mutante: gallium rm /etc/sudoers.d/jenkins-slave (to go with gerrit 311161)
17:03 Pchelolo: deploy changeprop to apply gerrit 311153 config change
16:16 yuvipanda: puppet developers are you reading this? just checking...
15:18 akosiaris: enable puppet on puppetmaster1001 again
15:03 akosiaris: disabling puppet on puppetmaster1001
15:00 hashar: gallium: dpkg --purge php5-mysql (mysql got removed)
14:46 gehel: disabling shard allocation check on relforge to test shard allocation issues
13:56 elukey: mw1189 back serving traffic after reimage
13:41 logmsgbot: akosiaris@tin Synchronized wmf-config/db-eqiad.php: (no message) (duration: 00m 46s)
13:35 hashar: gallium: removing MySQL which is no more defined in puppet and running puppet. Did: apt-get remove mysql-common mysql-server mysql-server-core-5.5
13:19 logmsgbot: akosiaris@tin Synchronized wmf-config/db-eqiad.php: (no message) (duration: 00m 48s)
12:50 logmsgbot: hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: All wikis back to 1.28.0-wmf.18 :( T145819
12:40 hashar: Going to rollback all Wikis back to 1.28.0-wmf.18 . Despite much investigation, a bunch of jobs are broken due to T145819 which includes Special:CreateAccount :(
12:37 elukey: mw1190 back serving traffic after the reimage
12:24 gehel: rolling restart of codfw elasticsearch cluster completed - T145404
12:20 moritzm: installing security updates for mysql 5.5 (one off systems running mysql as packaged by Ubuntu/Debian and not running wmf-mariadb10)
12:15 moritzm: installing python-imaging security updates on precise
11:24 akosiaris: silence icinga-wm for a while
11:22 akosiaris: restarted puppetmaster on all puppetmasters
11:22 akosiaris: stop puppetmaster on all puppetmasters, resizing /var/lib/puppet
10:21 marostegui: renaming tables before dropping them in codfw S1,S3,S4 - T54924
09:34 elukey: reimage mw1189-90 to Jessie (trying Riccardo's script!)
09:02 moritzm: reimaging mw1252-mw1254 to jessie
08:59 moritzm: installing tomcat7 security updates
08:52 moritzm: installing tomcat8 security updates
08:43 moritzm: installing libidn security updates in eqiad
07:36 elukey: forced logrotation with debug of /etc/logrotate.d/graphite-web on graphite1001 to find cronspam source
03:21 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Sep 16 03:21:54 UTC 2016 (duration 5m 18s)
03:16 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.19) (duration: 18m 44s)
02:41 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.18) (duration: 18m 25s)
01:28 mutante: mw1294 - down and frozen, powercycled
00:53 awight: remove "pending" from AMQ old message consumer
00:50 awight: update paymentswiki config to disable legacy orphan rectifier

2016-09-15

23:38 legoktm: legoktm@terbium:~$ foreachwiki extensions/WikimediaMaintenance/createExtensionTables.php babel # T145366
23:24 awight: rolled back paymentswiki to 392d67520d14998d61823755b22df50ab45afb35
23:22 awight: updating paymentswiki to possible broken 609c1e5f03ddcd9d72f562d7911cf77cc459371b
23:00 awight: update paymentswiki to from 996ca30076946c6148d9688a905c82e2e346e165 to 392d67520d14998d61823755b22df50ab45afb35 (reverted DI submodule update)
22:55 awight: Reenabled donations and fredge consumers
22:50 awight: update fundraising CRM from f381bd1c15e30e4d47fe372e128143baee6a7c7a to 5ba6976f2552564b51085abff0afd5f76195229b
22:30 awight: rollback paymentswiki from 609c1e5f03ddcd9d72f562d7911cf77cc459371b to 996ca30076946c6148d9688a905c82e2e346e165
22:28 awight: update paymentswiki from 996ca30076946c6148d9688a905c82e2e346e165 to 609c1e5f03ddcd9d72f562d7911cf77cc459371b
22:19 ejegg: updated SmashPig from af19422065c08669269179019fea1e7c208d8a7e to db68be988194c960aebca691d0fd8e6a6d24246a
22:10 ejegg: updated SmashPig from 12a7b78cf280f96ff3d2314abb2a9290506ae454 to af19422065c08669269179019fea1e7c208d8a7e
21:41 ejegg: updated SmashPig from e11af5793df9a0e2dbcf9ac138a8f2a3fa1bf574 to 12a7b78cf280f96ff3d2314abb2a9290506ae454
20:55 hashar: All wikis are on 1.28.0-wmf.19 wikidatawiki / testwikidatawiki stick to .18 for now.
20:35 gehel: increasing number of shards per node for dewiki_content index to 2 on elasticsearch codfw
20:28 logmsgbot: hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: All wiki to .19. Keep testwikidata and wikidata at .18 (commits: 38603f0 770d336)
20:14 yuvipanda: remove self from github wikimedia org, was getting spammed for each new repo creation
20:13 ema: varnish-be esams cache_upload: rolling depool and restart
20:11 ejegg: updated SmashPig from e11af5793df9a0e2dbcf9ac138a8f2a3fa1bf574 to 12a7b78cf280f96ff3d2314abb2a9290506ae454
20:08 gehel: increasing number of shards per node for enwiki_content index to 2 on elasticsearch codfw
20:01 yuvipanda: restart puppetmaster on labcontrol1001 to pick up hiera changes
19:59 ema: depool and restart varnish-be on cp1048
19:56 ema: depool and restart varnish-be on cp1064
19:52 logmsgbot: hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: (no message)
19:50 ema: depool and restart varnish-be on cp1062
19:45 ema: depool and restart varnish-be on cp1073
19:33 ema: depool and restart varnish-be on cp1050
19:20 ema: depool and restart varnish-be on cp1063
19:09 logmsgbot: hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.19
19:08 ema: depool and restart varnish-be on cp1072
19:08 awight: disabled fredge, donations, and banner-history consumers
19:07 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.18/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php: To unblock renames stuck on mediawiki.org T145596 (duration: 00m 47s)
19:06 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.19/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php: To unblock renames stuck on mediawiki.org T145596 (duration: 00m 47s)
19:05 logmsgbot: thcipriani@tin Finished scap: SWAT: Add missing close button title message (T145774) and Revert "Remove jquery.arrowSteps module" (T144974) (duration: 28m 08s)
18:59 ema: depool and restart varnish-be on cp1099
18:43 ejegg: updated smashpig consume pending job with new queue name
18:37 logmsgbot: thcipriani@tin Started scap: SWAT: Add missing close button title message (T145774) and Revert "Remove jquery.arrowSteps module" (T144974)
18:33 mutante: titanium - stop salt, stop puppet, revoke puppet cert, delete salt key
18:23 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.19/extensions/Kartographer: SWAT: Fix map popup CSS (T145716) (duration: 00m 56s)
18:23 ema: depool and restart varnish-be on cp1074
17:56 awight: Purging GC messages from pending with timestamp < '2016-09-15 13:44:55'
17:44 ema: depool and restart varnish-be on cp1049
17:18 Pchelolo: change-prop deploy 310877
16:51 Pchelolo: change-prop deploy gerrit 310873
16:49 akosiaris: uploaded to apt.wikimedia.org precise-wikimedia: zuul_2.5.0-8-gcbc7f62-wmf3precise1
16:49 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: zuul_2.5.0-8-gcbc7f62-wmf3jessie1
15:55 moritzm: uploaded trebuchet-trigger 0.5.6-1~jessie1 to carbon (no change rebuild for jessie)
15:45 kart_: Update cxserver to a1949e9
14:59 _joe_: starting a noop run on all nodes to puppetmaster2001 to test puppetdb
14:57 elukey: deployed new-aqs-cluster branch (--rev new-aqs-cluster) to aqs100[456] (new AQS cluster not serving live traffic)
14:30 _joe_: removing old reports from the puppet directory
14:03 godog: empty big log file on thumbor1001 /var/log/thumbor/thumbor.log
13:31 zeljkof: EU SWAT done!
13:28 logmsgbot: zfilipin@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Update outdated comment for Wikibase (duration: 00m 48s)
13:25 logmsgbot: zfilipin@tin Synchronized wmf-config/CommonSettings.php: SWAT: Remove $wgTranslateEC (duration: 00m 48s)
12:43 logmsgbot: addshore@tin Finished scap: Update RevisionSlider i18n (duration: 30m 26s)
12:13 logmsgbot: addshore@tin Started scap: Update RevisionSlider i18n
11:11 hashar: CI is catching up. It is starved processing a long serie of dependent changes in Gerrit
11:08 hashar: CI / Jenkins is starved. Investigating
10:09 moritzm: reimaging mw1251 to jessie
09:35 moritzm: reimaging mw1250 to jessie
08:11 marostegui: altering tables in S7 - eqiad hosts - T141951
07:01 moritzm: installing libidn security updates
06:54 marostegui: renaming tables before dropping them - T145487
06:29 moritzm: installing chromium security updates on osmium
06:24 _joe_: turning off nitrogen for memory reduction, reimage
03:23 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Sep 15 03:23:49 UTC 2016 (duration 6m 44s)
03:17 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.19) (duration: 18m 25s)
02:39 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.18) (duration: 17m 49s)
02:04 mutante: ms-be1022 - down per icinga, but also mgmt is not reachable

2016-09-14

23:44 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.19/extensions/Kartographer/tests/phpunit/KartographerTest.php: Always serve all the data on preview (T145615, 2/2, no-op part) (duration: 00m 50s)
23:43 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.19/extensions/Kartographer/includes/Tag/TagHandler.php: Always serve all the data on preview (T145615, 1/2) (duration: 00m 47s)
23:34 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Add logging channel for NewUserMessage (T131957) (duration: 00m 47s)
22:36 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.19/includes/resourceloader/ResourceLoaderWikiModule.php: T145673 (duration: 00m 47s)
21:18 Pchelolo: RESTBase update to fd43f3a58
21:13 Pchelolo: RESTBase update to fd43f3a58 canary on restbase1007
20:57 Pchelolo: RESTBase update to fd43f3a58 staging
20:28 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.19/extensions/CentralAuth/includes/LocalRenameJob/LocalRenameJob.php: Fix LocalRenameJob transaction owner to match JobRunner T143328 T145596 (duration: 00m 48s)
20:23 yuvipanda: manually raise max_connections on labtestcontrol2001, see T145679 for ticket
20:13 Pchelolo: revert RESTBase is staging to d10d759d42
20:11 arlolra: updated Parsoid to version aed15dda
20:05 Pchelolo: update RESTBase to 5ae9a506 - staging
20:03 arlolra: starting Parsoid deploy
19:25 logmsgbot: hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: Revert group1. Hebrew wiki has templates on the wrong side / CSS is off
19:14 logmsgbot: hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.19
19:09 Pchelolo: revert RESTBase to d10d759
19:08 awight: update paymentswiki config to 9919bad6897f2eb7199dd076f6fc04a60c713cf8
19:07 mutante: titanium - puppet node clean
18:59 Pchelolo: RESTBase deploy d39580f14
18:57 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.18/extensions/Kartographer/modules/box/Map.js: SWAT: Map should take viewport width/height instead of body width/height (T145521) (duration: 00m 47s)
18:53 Pchelolo: RESTBase deploy d39580f14 canary on restbase1007
18:50 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.18/extensions/ZeroBanner/modules: SWAT: Display edit icon and page actions (duration: 00m 47s)
18:45 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.19/extensions/Kartographer/modules/box/Map.js: SWAT: Map should take viewport width/height instead of body width/height (T145521) (duration: 00m 47s)
18:23 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.19/includes/pager/ReverseChronologicalPager.php: SWAT: Partially reverting I8e684f06 to restore some legacy behavior (T145597) (duration: 00m 48s)
18:17 urandom: T133805: Renabling Pupppet, forcing run, and restarting Cassandra to restore 8M region size on restbase1013-a.eqiad.wmnet
17:57 AaronSchulz: Deleted big pages per https://meta.wikimedia.org/w/index.php?title=Steward_requests/Miscellaneous&oldid=15908701#Deleting_a_pages_with_a_.3E5000_revisions_in_ruwiki
17:53 mutante: meitnerium - oops, an unrelated rsyncd is supposed to be running on this, puppet re-created files
17:50 mutante: meitnerium - stop rsyncd, remove config fragments
17:20 mobrovac: change-prop deploying 19e2d51
17:19 volans: reimage mw2198 as it failed before
16:32 jynus: stopping mysql and shutting down db1082
16:05 elukey: restarting cassandra on aqs100[23] T130861
16:01 jynus: starting mysql on db1082
15:57 elukey: restarting cassandra on aqs1001 T130861
15:54 akosiaris: updated cr1-eqiad,cr2-eqiad puppet rules
15:50 hoo: Ran T132839-Workarounds.sh from my home in terbium (see T132839)
15:49 urandom: T130861: Performing rolling Cassandra restart, restbase staging
15:45 urandom: T130861: Restarting Cassandra, xenon.eqiad.wmnet
15:41 urandom: T130861: Forcing puppet run in restbase staging
14:38 mobrovac: change-prop deploying ddc091e
14:30 gehel: increasing delayed allocation to 10m on elasticsearch codfw to speed up cluster restart - T145404
14:26 gehel: upgrading elasticsearch codfw to elasticsearch 2.3.5 - T145404
13:33 gehel: upgrading logstash to elasticsearch 2.3.5 - T145404
13:20 marostegui: renaming tables in s3 codfw - T132837
13:11 logmsgbot: hashar@tin Synchronized portals: Bumping portals to master T128546 (duration: 00m 47s)
13:10 logmsgbot: hashar@tin Synchronized portals/prod/wikipedia.org/assets: Bumping portals to master T128546 (duration: 00m 48s)
13:07 akosiaris: stop ircecho (icinga-wm) temporarily on neon
12:12 akosiaris: stop ircecho (icinga-wm) temporarily on neon
12:05 _joe_: restarting apache on puppetmaster1001
10:31 akosiaris: stopped temporarily ircecho (icinga-wm) on neon
10:25 ema: varnish-be restarted on cp4005
10:03 marostegui: alter localuser table in db2054 - T141951
09:47 marostegui: Renaming tables before dropping them T54924
08:43 marostegui: alter localuser table in db2047 - T141951
07:35 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.19/includes/MediaWiki.php: Use cpPosTime cookie for same-domain redirects on DB change - https://gerrit.wikimedia.org/r/#/c/310494/ (duration: 00m 45s)
07:34 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.19/includes/db/ChronologyProtector.php: Use cpPosTime cookie for same-domain redirects on DB change - https://gerrit.wikimedia.org/r/#/c/310494/ (duration: 00m 47s)
07:32 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.19/includes/db/loadbalancer/LBFactory.php: Use cpPosTime cookie for same-domain redirects on DB change - https://gerrit.wikimedia.org/r/#/c/310494/ (duration: 00m 46s)
03:22 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Sep 14 03:22:13 UTC 2016 (duration 7m 6s)
03:15 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.19) (duration: 18m 08s)
02:46 mutante: restarted grrrrit-wm
02:44 mutante: gerrit back to normal
02:42 mutante: gerrit restarting to apply config changes 256663 and 308885
02:40 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.18) (duration: 17m 56s)
01:56 logmsgbot: aaron@tin Synchronized wmf-config: Set $wgAPIMaxLagThreshold => 3 and "max lag" => 6 (duration: 00m 51s)

2016-09-13

23:32 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.18/resources/lib/moment/locale: T145382 (duration: 00m 47s)
23:19 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.19/resources/lib/moment/locale: T145382 (duration: 00m 49s)
22:44 awight: update paymentswiki config to da01ae9045af7b8b5f3d34a5f81419fde3bde1d5
22:13 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.19/includes/MediaWiki.php: Avoid stupid warnings on url parsing (duration: 00m 47s)
21:45 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.19/includes/DefaultSettings.php: for lego <3 (duration: 00m 47s)
20:46 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.19/extensions/Echo: For Roan <3 (duration: 00m 54s)
20:23 Krenair: restarted designate-api on labtestservices2001, now designate in labtest is working again
20:21 Krenair: restart rabbitmq-server on labtestcontrol2001
19:01 logmsgbot: demon@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.19
18:35 gehel: starting data import on wdqs200?
18:24 logmsgbot: demon@tin Finished scap: testwiki to wmf.19 + l10n bootstrap (try 2) (duration: 47m 21s)
18:08 gehel: moving to scap deployed configuration for wdqs - T144380
17:58 mutante: wtp2019 - powercycled, back up without the error, services started
17:55 mutante: wtp2019 Uncorrectable Memory Error
17:55 mutante: wtp2019 - down sinc a couple days. console says "Alert! System fatal error during previous boot"
17:53 volans: reimaging mw2198 that failed early today
17:36 logmsgbot: demon@tin Started scap: testwiki to wmf.19 + l10n bootstrap (try 2)
17:24 logmsgbot: demon@tin scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_4282891950" --threads=4 --lang en --quiet' returned non-zero exit status 1 (duration: 09m 23s)
17:15 logmsgbot: demon@tin Started scap: testwiki to wmf.19 + l10n bootstrap
16:59 jynus: beta dbs back in rw mode
16:54 mobrovac: restbase deploy end of d10d759
16:53 logmsgbot: demon@tin Synchronized multiversion/updateWikiversions: unbreak myself (duration: 00m 48s)
16:41 mobrovac: restbase deploy start of d10d759
16:25 mobrovac: change-prop deploying d701a69
16:24 godog: dump hhvm backtrace on mw1162 and restart hhvm, apache gets connection refused
16:17 urandom: T144826: Removing compaction rate limit, increasing compactor threads (from 10 to 20), and beginning scrub of local_group_wikipedia_T_parsoid_html.data (restbase2004-b.codfw.wmnet)
16:06 jynus: power resetting db1082
16:02 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1082 (duration: 01m 00s)
15:46 gehel: test_* indices remvoed on relforge cluster, cluster restarted
15:44 jynus: setting deployment-db1 and deployment-db1 mysqls in read only mode
15:40 gehel: shutting down relforge cluster for indices cleanup
15:18 marxarelli: starting 2-hour read-only maintenance window for beta cluster migration
14:50 godog: drain and reboot restbase2004
14:26 gehel: deleting test_* indices on relforge cluster
14:24 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-kaz-tat_0.2.1~r57554-1+wmf1
14:03 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-kaz_0.1.0~r61338-1+wmf1
13:33 logmsgbot: hashar@tin Synchronized wmf-config/CommonSettings.php: Stop logging xff from 127.0.0.1 T129982 (duration: 00m 47s)
13:28 hashar: Pulling "Stop logging xff from 127.0.0.1" patch on mw1299 and mw1161-mw1169 T129982
13:21 hashar: Pulling "Stop logging xff from 127.0.0.1" patch on mw1300-1303 T129982
13:19 logmsgbot: addshore@tin Synchronized php-1.28.0-wmf.18/extensions/UploadWizard/resources/uw.EventFlowLogger.js: SWAT: [[gerrit:310180|uw.EventFlowLogger: Fix NS_ERROR_NOT_AVAILABLE debug logging (duration: 00m 49s)
13:15 logmsgbot: hashar@tin Synchronized portals: Bumping portals to master (duration: 00m 50s)
13:10 logmsgbot: addshore@tin Synchronized dblists/clldefault.dblist: SWAT: Deploy Compact Language Links out of beta for Tulu Wikipedia (duration: 00m 46s)
13:08 logmsgbot: addshore@tin Synchronized wmf-config/flaggedrevs.php: SWAT: Fix illegal wgFlaggedRevsWhitelist for arwiki (duration: 00m 47s)
13:04 logmsgbot: addshore@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable RevisionSlider BetaFeature on all wikis (duration: 00m 49s)
12:47 marostegui: alter localuser table in db2040 - T141951
12:24 jynus: putting db1024 under maintenance (potential lag, etc.) to test solutions for T145079
12:06 mobrovac: zotero translators deploying cad95af
11:49 gehel: restarting elasticseaarch on relforge1001 - OOM heap space
11:18 volans: reimaging mw2198 and mw2199 to test the automation script T143536
10:58 mobrovac: citoid deployed e79430f for T144597
10:58 godog: finished rolling restart swift-proxy for thumbor change T139606
10:48 elukey: zuul upgraded to zuul_2.5.0-8-gcbc7f62-wmf2jessie1 on scandium (T145057)
10:45 elukey: uploaded 2.5.0-8-gcbc7f62-wmf2jessie1 to jessie-wikimedia/thirdparty (T145057)
10:10 godog: enable shadow requests to thumbor for small wikis T139606
09:39 marostegui: alter localuser table in dbstore2002 - T141951
08:52 gehel: relforge is taking more time than expected to recover after upgrade, most probably related to ~3k indices that were created for test purpose
08:46 gehel: relforge is taking more time than expected to recover after upgrade, most probably related to >10k indices that were created for test purpose
08:33 marostegui: renaming tables in db1015 - T145487
08:22 marostegui: alter localuser table in https://tendril.wikimedia.org/host/view/dbstore2001.codfw.wmnet/3306 - T141951
08:04 gehel: upgrading elasticsearch & plugins to 2.3.5 on relforge - T145404
07:52 elukey: wrong package name for my prev entry - remove kafkatee from stat1002 - not in puppet and causing cronspam (T132324)
07:50 elukey: remove kafkacat from stat1002 - not in puppet and causing cronspam (T132324)
07:07 moritzm: installing openjdk6 security updates
02:46 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Sep 13 02:46:54 UTC 2016 (duration 7m 12s)
02:39 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.18) (duration: 17m 41s)
01:58 logmsgbot: aaron@tin Synchronized wmf-config/CommonSettings.php: Lower $wgMaxUserDBWriteDuration to 3 (duration: 00m 47s)
01:27 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.18/extensions/GlobalUsage: 1843a856c7c648a310a1983d53139e5b79ea6585 (duration: 00m 48s)
01:24 bblack: cache_upload: reverting codfw to file storage
01:15 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.18/extensions/GeoData: 2dedca3789266b53cbde5bd88913c01175926974 (duration: 00m 49s)
01:14 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.18/includes/jobqueue/utils/PurgeJobUtils.php: (no message) (duration: 00m 52s)
00:50 bblack: cache_upload: reverting eqiad to file storage
00:50 bblack: reverting eqiad to file storage
00:17 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.18/extensions/GeoData: 2dedca3789266b53cbde5bd88913c01175926974 (duration: 00m 48s)
00:16 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.18/includes/jobqueue/utils/PurgeJobUtils.php: 0419831bbae66bda1e4dca8d690b607b493b2f5f (duration: 00m 47s)

2016-09-12

23:40 bblack: switch cache_upload codfw to -sdeprecated_persistent...
23:40 logmsgbot: dereckson@tin Synchronized wmf-config/throttle.php: Women in Science throttle rules (T145115 and T145253) (duration: 00m 47s)
23:33 urandom: T144826: Restarting Cassandra on restbase2004-b.codfw.wmnet (scrub complete, re-joining cluster)
23:14 bblack: switch cache_upload eqiad to -sdeprecated_persistent...
21:54 eileen: from 3f01d93237c2b8af4fbda5e629a4a59c77dab3c1 to f381bd1c15e30e4d47fe372e128143baee6a7c7a
21:17 arlolra: For completeness, "back" in my last log is a mistake. I scap deployed the wrong --rev, but that was ultimately the version we wanted deployed anyways, so no harm no foul. (T145460)
20:40 arlolra: Parsoid back on f7c43009c
20:32 arlolra: Parsoid deploy failed, rolling back
20:28 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1015-b.eqiad.wmnet
20:28 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1014-b.eqiad.wmnet
20:28 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1009-b.eqiad.wmnet
20:21 mobrovac: change-prop deploying 86a60b3
20:13 arlolra: starting Parsoid deploy
19:52 logmsgbot: demon@tin Synchronized multiversion/getMWVersion: for dumps <3 (duration: 00m 46s)
19:44 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1013-c.eqiad.wmnet
19:44 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1012-c.eqiad.wmnet
19:44 urandom: !log T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1008-c.eqiad.wmnet
19:42 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1011-c.eqiad.wmnet
19:42 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1010-c.eqiad.wmnet
19:42 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1007-c.eqiad.wmnet
19:13 urandom: T133805: Restarting Cassandra to apply G1 region size of 32M on restbase1013-a.eqiad.wmnet
19:12 urandom: T133805: Disabling Puppet for GC experiment on restbase1013.eqiad.wmnet
19:10 logmsgbot: thcipriani@tin Synchronized static/images/project-logos: SWAT: Fix HD logos for hewiki (T145017) (duration: 00m 48s)
18:43 ejegg: updated civicrm from 93091637a0a68950efb8955a2cfef031d6ba7883 to 9cbee66e067f788b2634000a33c010dd3f4606b6
18:24 ori: Changing wikiversion for group2 wikis on mw1017 to debug regression (T145359)
18:23 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.18/resources/Resources.php: Add missing dependency to 'mediawiki.Upload.BookletLayout' module (T145315) (duration: 00m 47s)
18:13 gehel: rolled back wdqs to HEAD^1
17:50 gehel: wdqs1001 put in maitnenance, some issue with config file deployment
17:46 gehel: deploying latest wikidata query service
17:29 godog: roll-restart cassandra in eqiad with new CA and certs T143044
16:30 ema: wiping/repooling cp4015
15:41 godog: roll-restart cassandra in codfw with new CA and certs T143044
15:15 godog: drain and restart cassandra instances on restbase2001 with new CA - T143044
14:51 ema: depool cp4015, restart and repool cp4006's backend
14:38 moritzm: powering down mw2017 for hardware maintenance
14:38 mobrovac: change-prop deploying 5d5d39e
14:36 urandom: T144826: Removing compaction rate limit, increasing compactor threads (from 10 to 20), and beginning scrub of local_group_wikipedia_T_parsoid_html.data (restbase2004-b.codfw.wmnet)
14:10 mobrovac: change-prop deploying 404b07c to enable scap config deploys
13:48 logmsgbot: hashar@tin Synchronized wmf-config: Remove upload7 references T129586 (duration: 00m 50s)
13:32 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.18/maintenance/cleanupUploadStash.php: Revert "Clean up user handling in UploadStash" T145228 (duration: 00m 46s)
13:31 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.18/includes/upload/UploadStash.php: Revert "Clean up user handling in UploadStash" T145228 (duration: 00m 46s)
13:27 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.18/extensions/Kartographer: Fix mw.Uri crushing bug T145178 (duration: 00m 49s)
13:21 ema: upgrade cp1099 to varnish 4 T131502
13:17 logmsgbot: hashar@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 48s)
13:16 logmsgbot: hashar@tin Synchronized wmf-config/throttle.php: Add throttling rule for University of Canterbury T145327 (duration: 00m 46s)
13:15 logmsgbot: hashar@tin Synchronized static/images/project-logos/: Add HD logos for hewiki T145017 (duration: 00m 50s)
13:04 ema: upgrade cp1074 to varnish 4 T131502
12:47 ema: upgrade cp1073 to varnish 4 T131502
12:30 ema: upgrade cp1072 to varnish 4 T131502
12:11 ema: upgrade cp1071 to varnish 4 T131502
12:00 ema: upgrade cp1064 to varnish 4 T131502
11:47 ema: upgrade cp1063 to varnish 4 T131502
11:40 mobrovac: change-prop deploying 79b172a
11:32 ema: upgrade cp1062 to varnish 4 T131502
11:17 ema: upgrade cp1050 to varnish 4 T131502
11:03 ema: upgrade cp1049 to varnish 4 T131502
10:48 ema: upgrade cp1048 to varnish 4 T131502
10:28 marostegui: renaming tables in db1015 - T132837
10:19 moritzm: decomissioning mw2061-mw2074 (Bug: T144745)
10:07 volans: reimage mw2198, mw2199 to Jessie (again) T143536
10:04 marostegui: Testing schema change on db1039 - T141951
10:02 jynus: deploying schema change on s4 hosts T139090
09:47 ema: depool cp4006 (503 Could not get storage)
07:25 moritzm: reimaging mw2077-mw2079, mw2017 to jessie
07:16 moritzm: installing openjpeg security updates
04:34 bblack: upgrade cp3049 to varnish 4 T131502
04:20 bblack: upgrade cp3048 to varnish 4 T131502
04:06 bblack: upgrade cp3047 to varnish 4 T131502
03:51 bblack: upgrade cp3046 to varnish 4 T131502
03:36 bblack: upgrade cp3045 to varnish 4 T131502
03:21 bblack: upgrade cp3044 to varnish 4 T131502
03:05 bblack: upgrade cp3039 to varnish 4 T131502
02:49 bblack: upgrade cp3038 to varnish 4 T131502
02:34 bblack: upgrade cp3037 to varnish 4 T131502
02:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Sep 12 02:29:27 UTC 2016 (duration 5m 53s)
02:23 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.18) (duration: 10m 37s)
02:06 ema: upgrade cp3036 to varnish 4 T131502
01:28 ema: upgrade cp3035 to varnish 4 T131502
00:51 ema: upgrade cp3034 to varnish 4 T131502

2016-09-11

22:33 logmsgbot: aaron@tin Synchronized wmf-config/CommonSettings.php: Lower wgMaxUserDBWriteDuration to 4 (duration: 00m 47s)
08:44 Amir1: ladsgroup@tin:~$ mwscript resetUserEmail.php --wiki=fawiki Sinasalek <email removed>
02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Sep 11 02:30:18 UTC 2016 (duration 5m 55s)
02:24 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.18) (duration: 10m 03s)

2016-09-10

22:30 urandom: T144826: Restarting Cassandra on restbase2004-b.codfw.wmnet (scrub complete, re-joining cluster)
12:36 urandom: T144826: Removing compaction rate limit, increasing compactor threads (from 10 to 20), and beginning scrub of local_group_wikipedia_T_parsoid_html.data (restbase2004-b.codfw.wmnet)
02:46 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Sep 10 02:46:34 UTC 2016 (duration 6m 11s)
02:40 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.18) (duration: 17m 53s)

2016-09-09

19:29 logmsgbot: demon@tin Synchronized wmf-config/wikitech.php: bizarro config loading (duration: 00m 46s)
18:23 logmsgbot: demon@tin Synchronized wmf-config/: prune old ext messages files (duration: 00m 52s)
16:54 logmsgbot: demon@tin Synchronized multiversion/: rm one more ugly file (duration: 01m 05s)
16:53 logmsgbot: demon@tin Synchronized docroot/noc/conf/: Updating activeMWVersions data (duration: 00m 47s)
16:29 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.18/extensions/JsonConfig/: Unbreak Zero namespace, Check globals in addition to attributes https://gerrit.wikimedia.org/r/309598 (duration: 00m 51s)
16:10 legoktm: live hacking on mw1017
15:15 urandom: T133805: Renabling Pupppet, forcing run, and restarting Cassandra to restore 8M region size on restbase1013-a.eqiad.wmnet
14:52 Jeff_Green: authdns-update for pay-lvs1001 & pay-lvs1002
14:51 mobrovac: change-prop deployed 34b23e7
13:25 elukey: analytics1032 back in service after disk swap
12:45 elukey: running authdns-update on ns0.w.o to pick up the new domain pivot.wikimedia.org (T138262)
12:27 elukey: reimaging mw213[789] and mw2075 to Jessie
12:05 moritzm: reimaging mw2133-mw2136 to jessie
10:19 moritzm: reimaging mw2080, mw2083-mw2085 to jessie
10:04 volans: reimage mw2132 to Jessie
10:03 logmsgbot: gehel@palladium conftool action : set/pooled=yes; selector: dc=eqiad,cluster=maps,service=kartotherian
09:25 gehel: restarting pybal on lvs1003
09:05 gehel: deploying new LVS configuration for kartotherian.svc.eqiad.wmnet
09:03 elukey: reimage mw2128->mw2131 to Jessie
09:02 godog: reimage ms-be1022 - T140597
08:55 godog: reset power on ms-be2019, cpu "soft lockup"
08:31 moritzm: reimaging mw2124-mw2127 to jessie
07:17 elukey: puppet disabled on analytics1032, Hadoop services stopped - T145170
06:48 moritzm: reimaging mw2120-mw2123 to jessie
05:22 jynus: deploying schema change on s5 hosts T139090
03:32 logmsgbot: aaron@tin Synchronized wmf-config/InitialiseSettings.php: Avoid $wmfMasterDatacenter notices from noc files (duration: 00m 46s)
03:31 logmsgbot: aaron@tin Synchronized docroot/noc/db.php: Avoid $wmfMasterDatacenter notices from noc files (duration: 00m 48s)
02:45 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Sep 9 02:45:47 UTC 2016 (duration 6m 11s)
02:39 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.18) (duration: 17m 37s)
01:38 logmsgbot: aaron@tin Synchronized wmf-config/filebackend-production.php: Bump description text expiry for files (duration: 00m 46s)
01:07 logmsgbot: aaron@tin Synchronized tests/Defines.php: (no message) (duration: 00m 46s)
01:02 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.18/extensions/SpamBlacklist: 56effa952c48725a2665dec72782bc8f7c7915a2 (duration: 00m 49s)
00:06 logmsgbot: hoo@tin Synchronized php-1.28.0-wmf.18/extensions/Wikidata: Don't use multiple return values (T145138) (duration: 02m 24s)

2016-09-08

23:53 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.18/extensions/CentralNotice: Bump production version to 4dbd3f9 (duration: 00m 51s)
23:47 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.18/extensions/Popups/extension.json: ext.popups.core depends on mediawiki.storage (Gerrit:309469) (duration: 00m 46s)
23:41 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.18/extensions/Kartographer/modules/box/Map.js: Switch to geojson for geoshapes srv (T144777) (duration: 00m 48s)
23:36 awight: update fundraising crm from cf19366e7f651785276d0071ce2c944d393c0ad5 to 93091637a0a68950efb8955a2cfef031d6ba7883
23:27 awight: rolling back fundraising crm from 946a3f1338bbaf3b65070b03a5a4c4aff5313a90 to cf19366e7f651785276d0071ce2c944d393c0ad5
23:25 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.18/resources/src/mediawiki/page/rollback.js: RollbackAction: Allow 'from' to be an empty string (T141985, 2/2) (duration: 00m 46s)
23:23 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.18/includes/actions/RollbackAction.php: RollbackAction: Allow 'from' to be an empty string (T141985, 1/2) (duration: 00m 46s)
23:18 awight: update fundraising crm from cf19366e7f651785276d0071ce2c944d393c0ad5 to 946a3f1338bbaf3b65070b03a5a4c4aff5313a90
23:02 yurik_: scaped kartotherian https://gerrit.wikimedia.org/r/#/c/309473/
22:52 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.18/includes/jobqueue/jobs:
21:56 awight: update SmashPig from 7f9eb7475d194c67ff070b5d8bbc9fc6837b462f to e11af5793df9a0e2dbcf9ac138a8f2a3fa1bf574
20:47 gehel: redeploy wdqs on wdqs2001.codfw.wmnet
20:03 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.18/extensions/Echo/includes/SeenTime.php: Trying to stop some duplicate redis fetches (duration: 00m 52s)
19:57 bblack: repooling normal traffic to cache_upload in ulsfo
19:02 logmsgbot: demon@tin rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.18
18:23 logmsgbot: demon@tin Synchronized multiversion/: So much junk to remove (duration: 01m 06s)
17:58 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase200[1-9]-b.codfw.wmnet
17:53 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1013-b.eqiad.wmnet
17:53 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1012-b.eqiad.wmnet
17:53 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1008-b.eqiad.wmnet
17:46 chasemp: reboot labstore1004 & labstore1005
17:45 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1011-b.eqiad.wmnet
17:45 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1010-b.eqiad.wmnet
17:45 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1007-b.eqiad.wmnet
17:41 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1015-a.eqiad.wmnet
17:41 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1014-a.eqiad.wmnet
17:41 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1009-a.eqiad.wmnet
17:11 gehel: reverting deploying new LVS configuration for kartotherian.svc.eqiad.wmnet - puppet error, let's analyse slowly...
17:06 gehel: deploying new LVS configuration for kartotherian.svc.eqiad.wmnet
17:04 bd808: Updated Striker to 7d7c8ee
16:55 logmsgbot: demon@tin Synchronized multiversion/: removing more junk - getMWVersion (duration: 01m 07s)
16:47 logmsgbot: demon@tin Finished scap: removing obsolete p symlink (duration: 04m 25s)
16:43 logmsgbot: demon@tin Started scap: removing obsolete p symlink
16:13 godog: roll-restart cassandra instances on restbase-test cluster T143044
16:07 moritzm: uploaded linux-meta 1.10 to carbon (pointing to the new 4.4.19 kernel image)
14:48 logmsgbot: addshore@tin Finished scap: SWAT: Update jquery.uls from upstream (duration: 50m 36s)
14:41 gehel: deploying new DNS entries for kartotherian.svc.eqiad.wmnet
14:36 godog: bounce restbase-test2001 cassandra-a instance T143044
13:58 logmsgbot: addshore@tin Started scap: SWAT: Update jquery.uls from upstream
13:43 moritzm: powering down mw2075-mw2079 for hardware maintenance (T142726)
13:28 ema: upgrading cache_upload ulsfo to varnish 4, dns depooled T131502
13:15 logmsgbot: addshore@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable mention status notifications everywhere (duration: 00m 47s)
13:12 logmsgbot: addshore@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add massmessage-sender group to urwiki (duration: 00m 47s)
13:08 logmsgbot: addshore@tin Synchronized wmf-config/extension-list: SWAT: RESTBaseUpdateJobs: Un-deploy the extension 3/3 (duration: 00m 47s)
13:07 logmsgbot: addshore@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: RESTBaseUpdateJobs: Un-deploy the extension 2/3 (duration: 00m 46s)
13:06 logmsgbot: addshore@tin Synchronized wmf-config/CommonSettings.php: SWAT: RESTBaseUpdateJobs: Un-deploy the extension 1/3 (duration: 00m 49s)
12:52 gehel: redeploying wdqs on wdqs2002.codfw.wmnet - T144380
12:44 moritzm: uploaded new linux package for jessie (based on 4.4.19 with bumped kernel ABI=2)
11:35 moritzm: reimaging mw2161, mw2162, mw2081, mw2082 to jessie
10:14 mobrovac: change-prop deploying a991e25
09:46 godog: roll-reboot thumbor machines to apply memory cgroup enablement T144938
08:52 gehel: initial data mimport on wdqs codfw cluster - T144380
08:23 marostegui: Drop tables: ImageMetricsLoadingTime_10078363 and ImageMetricsCorsSupport_11686678 - T141407
06:51 moritzm: reimaging mw2212-mw2214 to jessie
06:44 elukey: reimaging mw2208->mw2211 to jessie
03:33 ottomata: merging dns change to point archiva.wikimedia.org at new archiva node meitnerium
03:24 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Sep 8 03:24:15 UTC 2016 (duration 7m 23s)
03:16 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.18) (duration: 18m 14s)
02:40 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.17) (duration: 17m 56s)
00:46 logmsgbot: krenair@tin Synchronized php-1.28.0-wmf.18/extensions/VisualEditor/extension.json: https://gerrit.wikimedia.org/r/#/c/309213/ (duration: 00m 46s)
00:42 yurik: kartotherian synced T145042
00:15 twentyafterfour: upgrade complete. Service restored and everything seems normal.
00:14 twentyafterfour: phabricator upgrade is running database migrations now, taking longer than expected
00:03 twentyafterfour: Phabricator upgrade starting momentarily. Service will be offline for a short time, most likely less than 5 minutes.

2016-09-07

23:55 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Enable Translate on fr.wiktionary (T138972) (duration: 00m 47s)
23:45 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.17/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: Revert "Turn on CirrusSearch bm25 A/B test" (T143588) (duration: 00m 46s)
23:43 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.18/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: Revert "Turn on CirrusSearch bm25 A/B test" (T143588) (duration: 00m 46s)
23:41 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.18/extensions/VisualEditor/modules/ve-mw/ui/pages/ve.ui.MWParameterPage.js: Fix parent constructor call (Gerrit:309180) (duration: 00m 46s)
23:39 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.18/extensions/VisualEditor/lib/ve: Fix bad serialization of DOM elements in cloneElement (through Gerrit:309156) (duration: 00m 47s)
23:38 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.17/extensions/VisualEditor/lib/ve: Fix bad serialization of DOM elements in cloneElement (through Gerrit:309155) (duration: 00m 47s)
23:36 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.18/extensions/ORES/includes/Hooks.php: Get results when the score is not stored too (T144999) (duration: 00m 46s)
23:24 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Correct dblist definition (T143345, 2/2) (duration: 00m 47s)
23:23 logmsgbot: dereckson@tin Synchronized dblists/: Correct dblist definition (T143345, 1/2) (duration: 00m 49s)
22:02 Amir1: ladsgroup@terbium:~$ mwscript extensions/ORES/maintenance/PurgeScoreCache.php --wiki=wikidatawiki --model damaging
21:39 awight: update payments wiki from fafb6b476da3239039780b86d4b8f8d91bb54faa to 996ca30076946c6148d9688a905c82e2e346e165
21:36 Dereckson: Created tables for Translate extension on fr.wiktionary (T138972)
21:35 awight: reprocessing 20160906 PayPal audit files, take 2
21:21 awight: update fundraising-tools from b3ed7ab3deac94c4e465d3768109bca05b6f0a0c to b0be0f9ca04191c4bab869bb81191c5c77c432ca
21:04 urandom: T139961: Stopping RESTBase htmldumper in codfw
21:03 awight: rollback fundraising-tools from b71c504835454572ff70e48297c38b6ca3cbaece to b3ed7ab3deac94c4e465d3768109bca05b6f0a0c
20:50 awight: Reprocessing 20160906 PayPal audit files
20:11 mobrovac: restbase deploy end of 38d8c41
20:07 mobrovac: restbase cassandra truncating local_group_wikipedia_T_feed_aggregated.data for T144990
19:59 mobrovac: mobileapps deploying 2cd4f6a
19:52 mobrovac: restbase deploy start of 38d8c41
19:51 awight: update fundraising-tools from b3ed7ab3deac94c4e465d3768109bca05b6f0a0c to b71c504835454572ff70e48297c38b6ca3cbaece
19:40 urandom: T139961: Actually starting RESTBase htmldumper processes in codfw (read testing)
19:12 logmsgbot: demon@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.18
19:01 urandom: T139961: Starting RESTBase htmldumper processes in codfw (read testing)
18:47 thcipriani: Morning SWAT complete
18:46 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.18/extensions/MobileFrontend/resources/mobile.notifications.overlay/NotificationsOverlay.js: SWAT: Count local unread notifications when mark-all-read is clicked (T141404) (duration: 00m 44s)
18:44 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.18/extensions/Echo/modules/model/mw.echo.dm.ModelManager.js: SWAT: Add method to get local unread notifications in the manager (T141404) (duration: 00m 45s)
18:37 gehel: deploying wdqs, fix for T144913
18:35 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Extension:SandboxLink for tcywiki (T144925) (duration: 00m 47s)
18:29 logmsgbot: thcipriani@tin Synchronized dblists/nowikidatadescriptiontaglines.dblist: SWAT: Remove wikidata descriptions from additional projects (duration: 00m 45s)
18:24 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.18/extensions/UniversalLanguageSelector: SWAT: Revert "Update jquery.uls to a9dc11b" (T144871) (duration: 00m 47s)
18:21 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.18/includes/db/loadbalancer/LoadBalancer.php: ddd35a6ccedb68bad41d17c67e4408afe5ca4ae6 (duration: 00m 45s)
18:13 logmsgbot: thcipriani@tin Synchronized dblists/nowikidatadescriptiontaglines.dblist: SWAT: Revert "Enable Wikidata descriptions on all wikipedias" (duration: 00m 47s)
17:23 logmsgbot: aaron@tin Synchronized wmf-config/redis.php: Avoid pointless ChronologyProtector duplicate key notices (duration: 00m 47s)
16:42 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.18/includes/libs/objectcache/WANObjectCache.php: for aaron <3 (duration: 02m 50s)
16:09 volans: restarted ircecho on neon after rotating the irc.log file
16:03 cmjohnson1: db1020 swapping failed disk slot 5
15:59 gehel: deploying wdqs, fix for T144913
15:53 cmjohnson1: graphite1002 swapping failed disk slot10
15:43 madhuvishy: Rebooting host labstore1004
15:36 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/309015/ (duration: 02m 50s)
15:08 andrewbogott: re-imaging labnet1002 for T136718
15:02 moritzm: shutting down mw2080-mw2085 for hardware maintenance (T142726)
14:55 mobrovac: mobileapps deploying 8b929cfe
14:49 urandom: T144826: Restarting Cassandra on restbase2004-c.codfw.wmnet (scrub complete, re-joining cluster)
14:45 urandom: T144826: Removing compaction rate limit, increasing compactor threads from 10 to 20, and beginning scrub of local_group_globaldomain_T_mathoid_png.data (restbase2004-c.codfw.wmnet)
14:45 urandom: T144826: Removing compaction rate limit, increasing compactor threads from 10 to 20, and beginning scrub of local_group_globaldomain_T_mathoid_png.data
14:29 hoo: Deployed d4ad9ddbdd21a6460bb3f3a6fa6b74998e33d020 of wikidata/query/deploy: UI improvements
14:16 moritzm: shutting down mw2120-mw2139 for hardware maintenance (T142726)
14:13 moritzm: reimaging mw2157-2160 to jessie
13:58 elukey: reimaging mw2204->mw2207 to jessie
13:57 moritzm: upgrading labvirt1014 to Linux 4.4
13:55 mobrovac: restbase start end of 3852f72
13:31 mobrovac: restbase start deploy of 3852f72
13:15 logmsgbot: addshore@tin Synchronized php-1.28.0-wmf.18/extensions/RevisionSlider/modules/ext.RevisionSlider.DiffPage.js: SWAT: [[gerrit:308943|Revert "Do not nest mw-content-text element when reloading a diff" (duration: 00m 47s)
13:10 logmsgbot: addshore@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable mention status notifications on mediawikiwiki and metawiki (duration: 00m 47s)
13:06 logmsgbot: addshore@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable RC patrol for fiwiki and some related changes (duration: 00m 47s)
12:58 gehel: enabling row aware allocation on elasticsearch eqiad - T143571
12:51 ema: depool upload in ulsfo
12:38 moritzm: restarted mailman on fermium
12:35 logmsgbot: filippo@palladium conftool action : set/pooled=yes; selector: ms-fe1001.eqiad.wmnet
12:20 mobrovac: mobileapps deploying fc09d0d
11:48 moritzm: reimaging mw2153-mw2156 to jessie
11:04 logmsgbot: filippo@palladium conftool action : set/pooled=no; selector: ms-fe1001.eqiad.wmnet
10:32 hashar: https://yarn.wikimedia.org/ for the lazies
10:30 elukey: yarn.w.o is now available to all the users in the wmf ldap group (Basic Auth)
10:02 godog: add mw:thumbor to read/write ACLs for thumbnail containers of a subset of wikis T139606
09:05 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: pooled db1064 and removed db1019 which was replacing it - T144723 (duration: 00m 52s)
08:06 moritzm: reimaging mw2200-mw2203 to jessie
07:58 elukey: executed apt-get purge tmpreaper on gallium (T132324)
07:27 elukey: reimaging mw2144->mw2147 to jessie
07:00 gehel: increase cluster_concurrent_rebalance on elasticsearch codfw - T143571
06:40 moritzm: reimaging mw2140-mw2143 to jessie
05:38 gehel: enabling row aware allocation on elasticsearch codfw - T143571
03:16 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.18) (duration: 17m 58s)
02:40 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.17) (duration: 17m 37s)
02:06 urandom: T144826: Restarting Cassandra restbase2004-b.codfw.wmnet (putting back into service)
01:07 hoo: Updated Wikidata's property suggester with data from Monday's json dump and applied the T132839 workarounds
01:04 Krenair: labtest ldap: created dc=codfw,ou=hosts,dc=wikimedia,dc=org
00:13 Dereckson: Ran namespaceDupes maintenance script on frwiki

2016-09-06

23:54 yurik: deployed kartotherian - adjusting geoshapes arg, and bumping deps - https://gerrit.wikimedia.org/r/#/c/308899/
23:52 Dereckson: Ran namespaceDupes maintenance script on skwiki (0 pages to fix, 1 links to fix, fixed) (T143472)
23:47 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Fix User namespace localisation for sk.wikipedia (T143472) (duration: 00m 47s)
23:23 logmsgbot: dereckson@tin Synchronized dblists/nowikidatadescriptiontaglines.dblist: Update wikis where Wikidata descriptions is shown or not (Gerrit:307968 and Gerrit:307969, T143345) (duration: 00m 46s)
23:14 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Do not use $wgExtensionFunctions to set globals (T143055) (duration: 00m 47s)
22:58 arlolra: Parsoid restarted to pick up new wiki config after <maplink> deploy (T144062)
22:52 arlolra: restarting Parsoid
22:26 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/308890/3 (duration: 00m 47s)
22:09 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.17/extensions/Kartographer: https://gerrit.wikimedia.org/r/#/c/308887/ (duration: 00m 48s)
21:48 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.17/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/308300/ (duration: 00m 48s)
21:47 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.18/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/308300/ (duration: 00m 49s)
20:59 urandom: T133805: Restarting Cassandra to apply G1 region size of 16M on restbase1013-a.eqiad.wmnet
20:56 urandom: T133805: Disabling Puppet for GC experiment on restbase1013.eqiad.wmnet
20:53 logmsgbot: demon@tin Finished scap: scap 4 wikidata <3 (duration: 25m 53s)
20:27 logmsgbot: demon@tin Started scap: scap 4 wikidata <3
19:48 logmsgbot: demon@tin Synchronized multiversion/: rm useless file (duration: 01m 05s)
19:43 logmsgbot: demon@tin Finished scap: group0 to wmf.18 (duration: 27m 33s)
19:15 logmsgbot: demon@tin Started scap: group0 to wmf.18
19:05 robh: cache misc updates for wmfusercontent complete
18:58 Pchelolo: restbase deploy 4c239f2fa
18:54 Pchelolo: restbase deploy 4c239f2fa canary on restbase1007
18:49 Pchelolo: restbase deploy 4c239f2fa to staging
18:24 logmsgbot: demon@tin Synchronized multiversion/checkoutMediaWiki: rm branch pointer junk (duration: 00m 45s)
18:23 robh: running updates on cache_misc systems to update wmfusercontent certificate
18:05 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: Enable VisualEditor by default for logged-out users on Indic-script wikipædias (duration: 00m 50s)
17:47 logmsgbot: demon@tin Finished scap: 1.28.0-wmf.18 initial scap for l10n build (testwiki) (duration: 51m 52s)
17:39 ejegg: enabled banner history queue consumer
17:38 arlolra: updated Parsoid to version 7863e6ad (T142617)
17:36 ejegg: updated CiviCRM from 7484c9085209b83850d0802da8beea74dc594749 to cf19366e7f651785276d0071ce2c944d393c0ad5
17:34 ejegg: disabled banner history queue consumer
17:26 arlolra: starting Parsoid deploy
17:20 urandom: T144826: Ephemerally increasing compactor thread count from 10 to 20
17:13 urandom: T144826: Lifting compaction throttle
17:10 urandom: T144826: Starting online scrub
17:02 urandom: T144826: Restaring Cassandra on restbase2004-a.codfw.wmnet
16:55 logmsgbot: demon@tin Started scap: 1.28.0-wmf.18 initial scap for l10n build (testwiki)
16:11 mobrovac: change-prop restarting to pick up https://gerrit.wikimedia.org/r/308230
11:32 mobrovac: change-prop deploying e14892b
11:19 marostegui: disabled puppet on db1064 - going to be reimaged - T144723
10:50 jynus: shutting down db2001-2009
09:45 marostegui: rysnc running from db1064 to dbstore1001 (T144723)
09:23 marostegui: Stopping mysql on db1064 for maintenance - T144723
09:12 moritzm: shutting down mw2153-mw2162 for hardware maintenance (T142726)
08:54 akosiaris: T144174 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-srd-ita_0.9.0~r72554-1+wmf1
08:20 akosiaris: T144174 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-srd_0.9.0~r72792-1+wmf1
08:20 akosiaris: T144174 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-ita_0.9.0~r72553-1+wmf1
07:41 moritzm: correction: shutting down mw2140-mw2147 and mw2200-mw2214 for hardware maintenance (T142726)
07:40 moritzm: shutting down mw2140-mw2214 for hardware maintenance (T142726)
07:22 elukey: Increasing MaxRequestWorkers on Eqiad Imagescalers - mw129[3-8] - from 30 to 100 (one at the time checking metrics)
07:21 moritzm: reimaging mw2170 to jessie
06:54 moritzm: installing chromium security update on osmium
02:45 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Sep 6 02:45:17 UTC 2016 (duration 6m 28s)
02:38 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.17) (duration: 16m 46s)

2016-09-05

23:27 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Set wgMathFileBackend to false for wikitech wikis (T126628) (duration: 00m 48s)
23:23 logmsgbot: ema@palladium conftool action : set/pooled=yes; selector: cp2002.codfw.wmnet
23:17 logmsgbot: ema@palladium conftool action : set/pooled=no; selector: cp2002.codfw.wmnet
23:17 logmsgbot: dereckson@tin Synchronized dblists/closed.dblist: Close wikimania2015 (T139032) dblist update (duration: 00m 47s)
23:17 ema: upgrading cp2002 to varnish 4 T131502
23:14 logmsgbot: dereckson@tin Synchronized wmf-config/: Close wikimania2015 (T139032). So long and thanks for all the fish. (duration: 00m 51s)
22:55 logmsgbot: ema@palladium conftool action : set/pooled=yes; selector: cp2005.codfw.wmnet
22:50 logmsgbot: ema@palladium conftool action : set/pooled=no; selector: cp2005.codfw.wmnet
22:49 ema: upgrading cp2005 to varnish 4 T131502
22:28 logmsgbot: ema@palladium conftool action : set/pooled=yes; selector: cp2008.codfw.wmnet
22:24 logmsgbot: ema@palladium conftool action : set/pooled=no; selector: cp2008.codfw.wmnet
22:24 volans: restarting ircecho on neon to get back icinga-wm
22:24 ema: upgrading cp2008 to varnish 4 T131502
22:08 volans: stopped ircecho on neon to avoid the spam of recovery, monitoring icinga, I'll re-enable it in a bit
22:04 logmsgbot: ema@palladium conftool action : set/pooled=yes; selector: cp2011.codfw.wmnet
22:00 logmsgbot: ema@palladium conftool action : set/pooled=no; selector: cp2011.codfw.wmnet
21:59 ema: upgrading cp2011 to varnish 4 T131502
21:58 volans: restarting ircecho on neon to get back icinga-wm
21:36 logmsgbot: ema@palladium conftool action : set/pooled=yes; selector: cp2014.codfw.wmnet
21:28 logmsgbot: ema@palladium conftool action : set/pooled=no; selector: cp2014.codfw.wmnet
21:28 ema: upgrading cp2014 to varnish 4 T131502
20:56 bd808: Updated striker to b5fdbf9 (T144040, T144296)
20:45 logmsgbot: ema@palladium conftool action : set/pooled=yes; selector: cp2017.codfw.wmnet
20:41 logmsgbot: ema@palladium conftool action : set/pooled=no; selector: cp2017.codfw.wmnet
20:41 ema: upgrading cp2017 to varnish 4 T131502
19:49 logmsgbot: ema@palladium conftool action : set/pooled=yes; selector: cp2020.codfw.wmnet
19:45 logmsgbot: ema@palladium conftool action : set/pooled=no; selector: cp2020.codfw.wmnet
19:45 ema: upgrading cp2020 to varnish 4 T131502
19:26 logmsgbot: ema@palladium conftool action : set/pooled=yes; selector: cp2024.codfw.wmnet
19:22 logmsgbot: ema@palladium conftool action : set/pooled=no; selector: cp2024.codfw.wmnet
19:22 ema: upgrading cp2024 to varnish 4 T131502
18:53 logmsgbot: addshore@tin Synchronized php-1.28.0-wmf.17/extensions/UploadWizard/resources/mw.UploadWizardUpload.js: SWAT: mw.UploadWizardDetails, mw.UploadWizardUpload: Use amenableparser to handle templates in error messages Part 2/2 (duration: 00m 46s)
18:52 logmsgbot: addshore@tin Synchronized php-1.28.0-wmf.17/extensions/UploadWizard/resources/mw.UploadWizardDetails.js: SWAT: mw.UploadWizardDetails, mw.UploadWizardUpload: Use amenableparser to handle templates in error messages Part 1/2 (duration: 00m 48s)
18:51 logmsgbot: addshore@tin Synchronized php-1.28.0-wmf.17/resources/src/mediawiki/mediawiki.Upload.BookletLayout.js: SWAT: mw.Upload.BookletLayout: Use amenableparser to handle templates in error messages (duration: 00m 47s)
18:50 logmsgbot: addshore@tin Synchronized php-1.28.0-wmf.17/resources/src/mediawiki/api/messages.js: SWAT: mw.api.messages: Allow passing extra parameters for the API call (duration: 00m 53s)
18:31 ema: upgrading cp2026 to varnish 4 T131502
17:43 ema: restarting pybal on lvs2002 T134893
17:35 ema: upgrading cp2022 to varnish 4 T131502
17:34 mobrovac: change-prop deploying 222fcf8
14:42 elukey: upgrading apache httpd to the latest version on mw129[3-8] (eqiad image scalers)
14:05 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: Changing db-eqiad config to depool db1064 - T144723 (duration: 00m 48s)
14:00 elukey: upgrading mw1306/mw1299 to the latest version of Apache httpd
13:47 mobrovac: change-prop restarting for https://gerrit.wikimedia.org/r/306308
13:40 elukey: upgrading mw130[012345] to the latest version of Apache httpd (eqiad jobrunners, one at the time)
13:23 moritzm: reimaging mw2087 to jessie
13:12 logmsgbot: addshore@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable the RevisionSlider on test.wikidata.org (duration: 00m 48s)
13:07 logmsgbot: addshore@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add source wikis in import page in sawiki (duration: 00m 53s)
12:49 ema: repool cp4005 with varnish 3
12:47 moritzm: depooling/rebooting/repooling sca1002 for upgrade to Linux 4.4 (T144492)
12:41 ema: downgrading cp4005 to varnish 3 T131502
11:40 elukey: Reimaging mw217[89] and mw219[6789] to Debian jessie
10:37 moritzm: depooling/rebooting/repooling sca1001 for upgrade to Linux 4.4 (T144492)
10:18 moritzm: reimaging mw2192-mw2195 to jessie
09:24 elukey: reimaging mw21(8[89]|9[01]) to Debian Jessie
09:14 elukey: reimaging mw218[4567] to Debian Jessie
09:01 moritzm: reimaging mw2174-mw2177 to jessie
07:12 moritzm: reimaging mw2169-mw2172 to jessie
02:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Sep 5 02:29:51 UTC 2016 (duration 5m 42s)
02:24 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.17) (duration: 10m 50s)

2016-09-04

20:11 elukey: re-enabled ircecho on neon
20:03 elukey: stopped ircecho on neon temporarily
19:32 elukey: restarting apache2 on rhodium (attempt to fix it)
19:14 hashar: Puppet is falling since ~ 18:05 UTC. At least a couple european ops are looking at it
02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Sep 4 02:31:20 UTC 2016 (duration 6m 7s)
02:25 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.17) (duration: 09m 57s)

2016-09-03

14:24 ema: depool cp4005
10:21 jynus: deploying schema change on s7 hosts T139090
02:46 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Sep 3 02:46:12 UTC 2016 (duration 6m 22s)
02:39 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.17) (duration: 17m 35s)

2016-09-02

22:31 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.17/includes/page/WikiPage.php: No-op sync - 880c7f9526610bfd396dddb14f7772f9247850cb (duration: 00m 49s)
16:50 thcipriani: jenkins restarted
16:50 mark: rebooting bast3001
16:49 ema: downgrading cp4015 to varnish 3 T131502
16:40 thcipriani: restarting jenkins shortly for plugin upgrade
16:10 ema: downgrading cp4014 to varnish 3 T131502
15:33 ema: downgrading cp4013 to varnish 3 T131502
14:27 ema: downgrading cp4007 to varnish 3 T131502
14:14 mutante: gallium deleting jenkins/config-history files older than an hour
13:48 moritzm: rebooting labnet1002 for kernel update
12:09 ema: pooling ulsfo
11:37 elukey: upgrading mw1283.eqiad.wmnet to the latest httpd version
10:17 moritzm: reimaging mw2180-mw2183 to jessie
10:10 elukey: reimaging mw216[5-8] to jessie (IPMI fixed)
09:48 mark: Raised OSPF metrics on cr2-ulsfo<-->cr1-codfw link from 388 to 1000 in both directions
09:13 elukey: upgrading httpd on mw127[6789] to 2.4.10-10+deb8u6+wmf2 (eqiad api canaries)
09:03 ema: reboot cp4006 for kernel upgrade
08:38 moritzm: reimaging mw2099, mw2117, mw2163, mw2164 to jessie
08:34 elukey: upgrading httpd on mw1289 to 2.4.10-10+deb8u6+wmf2
08:19 elukey: upgrading httpd on mw128[0124] to 2.4.10-10+deb8u6+wmf2
08:04 elukey: upgrading httpd on mw1290 to 2.4.10-10+deb8u6+wmf2
06:40 moritzm: reimage mw2101 to jessie
06:36 moritzm: reimage mw2149-mw2151 to jessie
04:35 mutante: tin - re-enabled puppet
02:57 awight: Began bulk refund for T144489
02:43 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Sep 2 02:43:50 UTC 2016 (duration 5m 15s)
02:38 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.17) (duration: 16m 12s)
01:06 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Customize wgMathDirectory for wikitech (T126628, 2/2) (duration: 00m 46s)
01:05 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Customize wgMathDirectory for wikitech (T126628, 1/2) (duration: 00m 47s)
00:22 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Enable Math on wikitech (T126338) (duration: 00m 47s)
00:13 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: End lazy loading reference experiments (T144240) (duration: 00m 47s)
00:10 logmsgbot: dereckson@tin Synchronized wmf-config/CirrusSearch-common.php: Disable phrase suggester for wikidata (T143260, 2/2) (duration: 00m 46s)
00:09 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Disable phrase suggester for wikidata (T143260, 1/2) (duration: 00m 46s)
00:03 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.17/extensions/ZeroBanner: Update router code (T143425) (duration: 00m 47s)
00:02 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.17/extensions/MobileFrontend/resources/mobile.startup/Skin.js: Ensure lazy image placeholders without height can be loaded (T143768) (duration: 00m 46s)
00:00 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.17/extensions/Flow/modules/flow/ui/widgets/editor/editors/mw.flow.ui.VisualEditorWidget.js: Flow Fixes related to Visual Editor (T138356 and T139972) (duration: 00m 45s)

2016-09-01

23:57 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.17/extensions/VisualEditor/lib/ve: Update lib/ve submodule for Ib9bbaccfff9 (duration: 00m 47s)
23:51 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.17/extensions/CirrusSearch/includes/Query/FullTextQueryStringQueryBuilder.php: Do not use the suggest reverse field if it's a non local search (Gerrit:307955) (duration: 00m 48s)
23:39 mutante: tin removed mw2187 from /etc/dsh/group/scap-proxies
23:38 mutante: tin stopping puppet
23:32 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings-labs.php: Enable Wikidata descriptions taglines on labs (T143345, no-op in prod) (duration: 02m 52s)
23:15 XenoRyet: updated payments-wiki from ef2f2f85461d25a3938c173d0116fa0d2dee58d1 to fafb6b476da3239039780b86d4b8f8d91bb54faa
23:06 MaxSem: That for https://gerrit.wikimedia.org/r/#/c/308084/
23:06 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 02m 53s)
23:01 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.17/extensions/Kartographer: https://gerrit.wikimedia.org/r/#/c/308085/ (duration: 02m 54s)
22:45 ejegg: enabled fredge queue consumer
22:38 ejegg: updated civicrm from 1678e1f52468dae596882a323cfa5365b966d5f0 to 7484c9085209b83850d0802da8beea74dc594749
22:36 ejegg: disabled fredge queue consumer
22:31 ejegg: updated paymentswiki settings from 393944f179b6823f27dc1a9430b4a3661388ccff to 18451935c65f711e5c50794211a05309fd903a40
22:11 awight: reenabling recurring Ingenico job and kicking one-off run
22:03 awight: updating wmf_civicrm schema to 7240
21:59 awight: update fundraising CRM from 0c6bf3813ee0f2e58d5948fc7c000cf20c114841 to 1678e1f52468dae596882a323cfa5365b966d5f0
21:25 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1013-a
21:25 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1012-a
21:24 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1008-a
21:22 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1011-a
21:22 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1010-a
21:21 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1007-a
20:58 mutante: mw2187 - shut down
20:35 urandom: T143226: Clearing repair status: eqiad, rack 'd' nodes
20:35 urandom: T143226: Clearing repair status: eqiad, rack 'dd' nodes
19:58 hashar: 1.28.0-wmf.17 rolled on group2 and apparently all fine
19:54 gehel: reloading ferm rules on elasticsearch eqiad cluster
19:44 urandom: T143226: Clearing repair status: eqiad, rack 'b' nodes
19:39 urandom: T143226: Clearing repair status restbase1011-c.eqiad.wmnet
19:19 logmsgbot: hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.17
19:14 mutante: tin temp. disabled puppet
19:11 mutante: tin removing mw2167 thru mw2199 from dsh file manually, re-running puppet
18:33 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Test PageAssessments on English Wikivoyage (T142056) (duration: 02m 48s)
18:26 mutante: mw2187 - powercycled
18:19 logmsgbot: gehel@palladium conftool action : set/pooled=yes; selector: name=elastic1029.eqiad.wmnet
18:19 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: REVERT because proxy down SWAT: Test PageAssessments on English Wikivoyage (T142056) (duration: 03m 15s)
18:11 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Test PageAssessments on English Wikivoyage (T142056) (duration: 04m 54s)
17:55 Jeff_Green: switching fundraising database reader from db1008 to frdb1001
17:12 bd808: Updated striker to ac555bd; fixes T144064
16:49 ema: cp4006 repooled after downgrade
16:44 logmsgbot: gehel@palladium conftool action : set/pooled=yes; selector: name=elastic1028.eqiad.wmnet
16:24 ema: restarting pybal on lvs4002 T134893
16:10 ema: downgrading cp4006 to varnish 3 T131502
15:36 logmsgbot: gehel@palladium conftool action : set/pooled=yes; selector: name=elastic102[12].eqiad.wmnet
15:34 chasemp: reboot nova-compute on labvirt1013 as stuck (no logs, not applying any changes or taking any instruction)
14:58 moritzm: powered down several hosts for hardware maintenance (T142726): mw2099, mw2102, mw2117, mw2163-mw2199
14:42 mobrovac: restbase deploy end of 9cca320
14:40 moritzm: powered down several hosts for hardware maintenance (T142726): mw2087, mw2149-mw2151
14:39 elukey: upgrading httpd/apache to 2.4.10-10+deb8u6+wmf2 on mw128[56]
14:27 mobrovac: restbase deploy start of 9cca320
14:11 godog: wipe and reinitialize corrupted xfs on /dev/sdn1 on ms-be1016
13:53 eileen: turned off queue for http://localhost:9000/job/GlobalCollect%20Recurring%20Donations/ in jenkins
13:42 elukey: upgrading httpd/apache to 2.4.10-10+deb8u6+wmf2 on mw128[78]
13:41 bblack: uploaded openssl-1.1.0-1+wmf1 to jessie-wikimedia/experimental
13:39 elukey: not upgrading mw130[01] since I'd need more info before proceeding
13:34 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.17/includes/page/WikiPage.php: T144484 (duration: 00m 49s)
13:30 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.17/includes/page/WikiPage.php: T144484 (duration: 00m 35s)
13:16 elukey: upgrading httpd/apache to 2.4.10-10+deb8u6+wmf2 on mw130[01].eqiad.wmnet
13:01 logmsgbot: gehel@palladium conftool action : set/pooled=yes; selector: name=elastic1047.eqiad.wmnet
12:53 logmsgbot: ema@palladium conftool action : set/pooled=yes; selector: cp4005.ulsfo.wmnet (tags: ['dc=ulsfo', 'cluster=cache_upload', 'service=varnish-be'])
12:12 gehel: rolling restart of ferm on elasticsearch eqiad cluster to account for moved servers - T143685
10:50 moritzm: installing libidn security updates
10:39 godog: reboot ms-be1016, stuck again
10:24 logmsgbot: hashar@tin Synchronized docroot/noc/index.html: link to conftool and wikitech pages on https://noc.wikimedia.org/ (duration: 00m 47s)
10:17 gehel: repooled elastic104[456] - T144450
09:58 jynus: adding marostegui to wmf and ops on wikitech LDAP
09:57 logmsgbot: gehel@palladium conftool action : set/pooled=inactive; selector: elastic1046.eqiad.wmnet
09:57 logmsgbot: gehel@palladium conftool action : set/pooled=inactive; selector: elastic1045.eqiad.wmnet
09:56 logmsgbot: gehel@palladium conftool action : set/pooled=inactive; selector: elastic1044.eqiad.wmnet
09:51 moritzm: reimaging mw2200-2203 to jessie
09:37 moritzm: reimaging mw2061-2064 to jessie
09:35 elukey: reimaging mw2167 -> mw2170 to Jessie
09:24 moritzm: reimaging mw2163-2166 to jessie
09:06 moritzm: installing postgres security updates on labsdb1004/1006/1007
09:04 godog: reboot ms-be1016, stuck and nothing on console
07:45 moritzm: reimaging mw2116-2119 to jessie
03:12 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Sep 1 03:12:34 UTC 2016 (duration 7m 14s)
03:05 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.17) (duration: 17m 59s)
02:30 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.16) (duration: 12m 21s)
02:22 bblack: ulsfo depool
01:00 Jeff_Green: reboot db1025 for kernel update
01:00 Jeff_Green: reboot db1025 for kernel update
00:29 eileen: CiviCRM upgrade from from d067c476da074afa70297c29e3d471743c110e3a to 0c6bf3813ee0f2e58d5948fc7c000cf20c114841
00:13 eileen: fr campaigns disabled
00:09 eileen: stop Donations q consumer job on jenkins

2016-08-31

23:53 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/307886/ (duration: 00m 48s)
23:42 logmsgbot: maxsem@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/268627/ (duration: 00m 47s)
23:40 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/268627/ (duration: 00m 47s)
23:34 urandom: T143226: Clearing repair status: eqiad, rack 'a' nodes
23:34 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: Updating logos, step 2 (duration: 00m 48s)
23:31 logmsgbot: maxsem@tin Synchronized static: Update logos, step 1 (duration: 00m 47s)
23:20 urandom: T143226: Clearing repair status: codfw, rack 'd'
23:15 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.17/extensions/Kartographer: https://gerrit.wikimedia.org/r/#/c/307883/ (duration: 00m 49s)
22:33 logmsgbot: gehel@palladium conftool action : set/pooled=no; selector: name=elastic1047.eqiad.wmnet
22:32 gehel: depooling elastic1047 from LVS
22:17 gehel: restarting elasticsearch on elastic1028
22:12 urandom: T143226: Clearing repair status: restbase2008.codfw.wmnet
22:02 hashar: surge of since 31th 7:00 UTC : Retrying connection to search.svc.eqiad.wmnet
22:01 urandom: T143226: Clearing repair status: restbase2004.codfw.wmnet
21:52 urandom: T143226: Clearing repair status: restbase2003.codfw.wmnet
21:46 urandom: T143226: Clearing repair status: restbase2007.codfw.wmnet
21:44 urandom: T143226: Clearing repair status: restbase2002.codfw.wmnet
21:39 urandom: T143226: Starting restbase2001-{b,c}.codfw.wmnet
21:38 urandom: T143226: Stopping restbase2001-{a,b} to clear tables marked repaired
21:27 urandom: T143226: Stopping restbase2001-a.codfw.wmnet to clear tables marked repaired
19:59 hashar: group1 to 1.28.0-wmf.17 done. There is a couple explicit commit of implicit transactions for Wikidata T144433 T144434 not much of a worry
19:50 apergos: restarted udp2log-mw on fluorine again after deployment of https://gerrit.wikimedia.org/r/#/c/307812/
19:46 logmsgbot: hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.17
19:15 apergos: stopped/started udp2log-mw on fluorine
18:54 gehel: shutting down elasticsearch on elastic1028 to prepare moving server - T143685
18:52 Pchelolo: RESTBase deploy fa1a5aab
18:47 Pchelolo: RESTBase deploy fa1a5aab canary on restbase1007
18:34 Pchelolo: RESTBase deploy fa1a5aab to staging
18:22 logmsgbot: ebernhardson@tin Synchronized php-1.28.0-wmf.16/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: Enable CirrusSearch BM25 AB test (duration: 00m 47s)
18:17 logmsgbot: ebernhardson@tin Synchronized php-1.28.0-wmf.17/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: Enable CirrusSearch BM25 AB test (duration: 00m 50s)
17:41 mobrovac: change-prop deploying bf221599c
14:49 mobrovac: parsoid first scap3 deploy to wtp20xx
14:40 godog: reenable puppet on wtf* post-merge of scap3 conversion
14:39 elukey: the previous entry was related to apache2
14:38 elukey: upgrading the remaining codfw mw* to 2.4.10-10+deb8u6+wmf2
14:38 mobrovac: parsoid first scap3 deploy to wtp2001 only
14:25 godog: stopping puppet on wtp* prior to merging https://gerrit.wikimedia.org/r/#/c/304470/
14:19 bblack: packages updates for cp* (new kernel, new nginx, misc updates)
13:59 bblack: uploaded nginx 1.11.3-1+wmf2 for jessie-wikimedia to carbon (ssl curve variable support)
13:58 jynus: running alter table on db1047 enwiki.categorylinks to mitigate lag issues
13:52 elukey: upgrading the codfw jessie appservers to 2.4.10-10+deb8u6+wmf2
13:40 chasemp: restart rabbitmq on labcontrol1001
13:36 bblack: repooling ulsfo traffic
13:25 moritzm: installing libarchive security updates on jessie
13:13 moritzm: reimaging mw2212-mw2215 to jessie
11:39 godog: swift eqiad-prod: set weight for ms-be1021 sd[h-n] to 0 - T139767
11:10 gehel: restarting elasticsearch104[456] to take new rack configuration into account - T143685
10:33 akosiaris: starting a gradual deployment of wikidiff2 across the mw* fleet. T140443
09:53 mobrovac: change-prop deploying e999f517
09:51 elukey: restarting cassandra on aqs100[123] to verify performances after daemon restart
09:42 moritzm: reimaging mw2108-mw2111 to jessie
08:24 akosiaris: drained ulsfo of traffic, it seems to be unstable currently
07:10 moritzm: removed cached files on stat1001 (/var/www/limn-public-data/caching) after checking with joal, disk space on root partition was depleted
07:07 jynus: removed db1027 from our dns
07:05 moritzm: reimaging mw2104-mw2108 to jessie
04:28 ejegg: updated civicrm from e1feb34ff688e230cced92c46e5c2a78e2b3cffa to d067c476da074afa70297c29e3d471743c110e3a
03:48 ejegg: rolled back civicrm to e1feb34ff688e230cced92c46e5c2a78e2b3cffa
03:43 ejegg: updated civicrm from e1feb34ff688e230cced92c46e5c2a78e2b3cffa to 8895f0e8ac9ad03cdf11b81542c17c5a39ca5cd8
03:17 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Aug 31 03:17:37 UTC 2016 (duration 7m 13s)
03:11 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.17/resources/src/mediawiki/mediawiki.js: Ie4e7464aa811 and I90eea4bfe (duration: 00m 47s)
03:10 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.17) (duration: 19m 02s)
02:57 ejegg: rolled back civicrm to e1feb34ff688e230cced92c46e5c2a78e2b3cffa
02:53 ejegg: updated civicrm from e1feb34ff688e230cced92c46e5c2a78e2b3cffa to c0a7f45bd913ebd28e16c0d78b009a7af95209e9
02:51 ejegg: disabled fredge queue consumer
02:34 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.16) (duration: 12m 45s)
00:06 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.17/extensions/Flow: SWAT (duration: 01m 09s)
00:05 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.17/extensions/Echo: SWAT (duration: 00m 53s)

2016-08-30

23:59 logmsgbot: catrope@tin Synchronized wmf-config/CirrusSearch-common.php: Fix CirrusSearch BM25 A/B test similarity config (duration: 00m 48s)
23:59 RoanKattouw: Whoops, I meant scap sync-file, not scap sync
23:59 logmsgbot: catrope@tin scap aborted: wmf-config/CirrusSearch-common.php Fix CirrusSearch BM25 A/B test similarity config (duration: 00m 01s)
23:59 logmsgbot: catrope@tin Started scap: wmf-config/CirrusSearch-common.php Fix CirrusSearch BM25 A/B test similarity config
23:58 logmsgbot: catrope@tin Synchronized wmf-config/logging.php: Require ack of kafka logging (T135159) (duration: 00m 47s)
23:40 ejegg: updated SmashPig from 4ba14a10518f80fe7e20fadc493d78c796c15921 to 7f9eb7475d194c67ff070b5d8bbc9fc6837b462f
23:35 ejegg: rolled back SmashPig to 4ba14a10518f80fe7e20fadc493d78c796c15921
23:34 logmsgbot: catrope@tin Synchronized wmf-config: Enable Wikidata description taglines on all projects except top 6 wikis (T143344) (duration: 00m 54s)
23:33 logmsgbot: catrope@tin Synchronized dblists/nowikidatadescriptiontaglines.dblist: (no message) (duration: 00m 53s)
23:30 ejegg: updated SmashPig from 4ba14a10518f80fe7e20fadc493d78c796c15921 to b94325bea6158e80f194689e247b16239053baa1
22:19 ejegg: rolled back SmashPig to 4ba14a10518f80fe7e20fadc493d78c796c15921
22:18 mobrovac: change-prop deploying a87a61d
21:57 ejegg: updated SmashPig from 4ba14a10518f80fe7e20fadc493d78c796c15921 to b94325bea6158e80f194689e247b16239053baa1
20:16 gehel: restarting ferm on elasticsearch eqiad cluster after reinstall of elastic104[4567] - T143685
20:09 yuvipanda: cleanup /var/cache/iegreview for bd808
19:52 legoktm: deployed https://gerrit.wikimedia.org/r/306710, moving 4 parsoid CI jobs from nodepool trusty to nodepool jessie
19:47 mutante: rsyncing archiva data from titanium to meitnerium, runs in a screen
19:38 hashar: 1.28.0-wmf.17 successfully pushed to group0
19:34 logmsgbot: hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.28.0-wmf.17 (bis) T144307
19:30 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.16/extensions/CentralAuth/includes/CentralAuthUser.php: Remove verbose cache miss log that was making notices T144307 (duration: 00m 48s)
19:05 logmsgbot: hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: Rollback 1.28.0-wmf.17 T142117
19:03 logmsgbot: hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.28.0-wmf.17 T142117
18:28 gehel: deploying latest version of wikidata query service
18:17 chasemp: restart nodepool
18:12 logmsgbot: thcipriani@tin Synchronized dblists/visualeditor-nondefault.dblist: SWAT: Enable VisualEditor by default for logged-in users on Indic-script Wikipedias (T142586) PART II (duration: 00m 49s)
18:11 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor by default for logged-in users on Indic-script Wikipedias (T142586) PART I (duration: 00m 48s)
18:08 mobrovac: change-prop deploying c793e4a2
17:48 gehel: shutting down elasticsearch on elastic1046 to prepare moving server - T143685
17:08 gehel: shutting down elasticsearch on elastic1045 to prepare moving server - T143685
17:05 gehel: shutting down elasticsearch on elastic1044 to prepare moving server - T143685
16:58 gehel: shutting down elasticsearch on elastic1047 to prepare moving server - T143685
16:13 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.17/includes/EditPage.php: (no message) (duration: 00m 48s)
16:04 logmsgbot: hoo@tin Synchronized wmf-config/InitialiseSettings.php: Enable allowDataAccessInUserLanguage on Wikidata (T122670) (duration: 00m 51s)
15:44 logmsgbot: hashar@tin Finished scap: testwiki to php-1.27.0-wmf.17 T142117 (duration: 53m 28s)
15:35 godog: restart pybal on lvs primaries in codfw/eqiad
14:51 logmsgbot: hashar@tin Started scap: testwiki to php-1.27.0-wmf.17 T142117
14:46 ema: banning objects with status code 200 and content-length 0 from upload frontends in ulsfo T144257
14:44 godog: bounce pybal to pick up prometheus.svc on low-traffic in eqiad/codfw
14:41 godog: bounce pybal to pick up prometheus.svc
14:40 ema: banning objects with status code 200 and content-length 0 from upload backends in ulsfo T144257
14:39 hashar: Applying security patches for 1.28.0-wmf.17 T142117
14:35 hashar: mw2101 running scap pull , it missed bunch of files
14:35 zeljkof: European SWAT is done!
14:34 logmsgbot: filippo@palladium conftool action : set/pooled=yes; selector: prometheus1001.eqiad.wmnet
14:34 logmsgbot: filippo@palladium conftool action : set/pooled=yes; selector: prometheus2001.codfw.wmnet
14:32 logmsgbot: zfilipin@tin Synchronized php-1.28.0-wmf.16/extensions/CirrusSearch/includes/CirrusSearch.php: SWAT: Initialize the UserTesting framework before creating a Connection (duration: 00m 49s)
14:20 logmsgbot: zfilipin@tin Synchronized dblists/s3.dblist: SWAT: Sort s3.dblist in lexicographical order (duration: 02m 44s)
14:14 hashar: Moved mediawiki-core-phpcs job back to Nodepool T143938
14:01 hashar: European SWAT extended
13:53 logmsgbot: ema@palladium conftool action : set/pooled=no; selector: cp4005.ulsfo.wmnet (tags: ['dc=ulsfo', 'cluster=cache_upload', 'service=varnish-be'])
13:48 logmsgbot: zfilipin@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Allow bureaucrats to manage account creators group on ar.wikipedia (T143844) (duration: 00m 50s)
13:11 gehel: cleanup openjdk 7 on maps2002 - T142977
13:06 gehel: removing unused openjdk 7 on maps1001
11:47 gehel: banning elastic10(44|45|46|47) from elasticsearch eqiad cluster - T143685
11:42 moritzm: upgrading remaining jessie-based mw systems to hhvm 3.12.7 (now that the systemd unit override is in place)
11:01 godog: roll-restart xenon/cerium/praseodymium cassandra instances to pick up new certs
10:53 hashar: Cutting MediaWiki branch 1.28.0-wmf.17 T142117
09:57 ema: restarted morebots
02:36 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Aug 30 02:36:06 UTC 2016 (duration 6m 36s)
02:29 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.16) (duration: 11m 57s)

2016-08-29

23:47 chasemp: stop nfs server on labstore1001 to try to fix scratch mount
23:34 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.16/extensions/ORES/includes/Cache.php: Improvements to purging cache (T144216) (duration: 00m 47s)
23:27 Dereckson: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Add Əlavə namespace to az.wiktionary (T143851) (duration: 00m 47s)
23:26 chasemp: restart nfs-kernel-server on labstore1003 and labstore1001
23:25 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: EditSubmitButtonLabelPublish: Temporarily don't do this (currently no-op) (duration: 00m 47s)
23:25 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: ores: Update thresholds (T144101) (duration: 00m 48s)
22:45 ejegg: enabled donations queue consumer
22:40 ejegg: updated CiviCRM from d040549105151ccb66a464effbbd56a3e4bfbb8b to e1feb34ff688e230cced92c46e5c2a78e2b3cffa
22:39 ejegg: disable donations queue consumer
22:39 dapatrick: Deployed patch for T125177 to 1.28.0-wmf.16
21:56 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: T131132 (duration: 00m 48s)
21:32 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1015-a.eqiad.wmnet
21:31 urandom: T143226: Starting restbase1015-a.eqiad.wmnet
21:29 urandom: T143226: Stopping restbase1015-a.eqiad.wmnet to remove repairedAt attribute on a table
21:11 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1007-b.eqiad.wmnet
20:33 akosiaris: restarted ircecho on neon.wikimedia.org (icinga-wm)
20:12 bd808: Updated striker to ffe13c1; see https://wikitech.wikimedia.org/wiki/Toolsadmin.wikimedia.org/Deployments#2016-08-29
20:11 bd808: starting striker deploy
20:10 subbu: finished deploying parsoid sha 48cf803e
20:04 subbu: synced new parsoid code; restarted parsoid on wtp1001 as a canary
20:02 subbu: starting parsoid deploy
19:41 bd808: Taking stashbot offline (hopefully briefly)
19:26 Niharika: Updated iegreview to 29e98bb (Show all users in 'Manage Users' list (blocked and unblocked both))
18:42 logmsgbot: thcipriani@tin Synchronized wmf-config/CirrusSearch-common.php: SWAT: Define variables for AB test for all wikis and Revert "Revert "CirrusSearch BM25 A/B test config"" (duration: 00m 47s)
18:32 chasemp: make greg a phab admin to fight surge of spam bots
18:29 logmsgbot: thcipriani@tin Synchronized wmf-config/CirrusSearch-common.php: REVERT SWAT: CirrusSearch BM25 A/B test config (T143586) (duration: 00m 48s)
18:23 logmsgbot: thcipriani@tin Synchronized wmf-config/CirrusSearch-common.php: SWAT: CirrusSearch BM25 A/B test config (T143586) (duration: 00m 46s)
18:11 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Switch enwiki to uca-default collation (T136150) (duration: 00m 47s)
17:31 logmsgbot: demon@tin Synchronized multiversion/: delete deleteMediawiki (duration: 01m 09s)
16:12 Amir1: mwscript extensions/ORES/maintenance/PopulateDatabase.php --wiki=fawiki (and seven other wikis) T144101
16:07 Amir1: mwscript extensions/ORES/maintenance/PurgeScoreCache.php --wiki=enwiki (and seven other wikis) (T144101)
15:34 halfak: restarted ores-celery-worker service on scb1002
15:23 Amir1: ladsgroup@terbium:~$ mwscript extensions/ORES/maintenance/CheckModelVersions.php on (fa|tr|pl|ru|pt|nl|wikidata|en)wiki
15:12 halfak: deploying ores b8598dd (see T144101)
14:28 logmsgbot: ema@palladium conftool action : set/pooled=yes; selector: cp4015.ulsfo.wmnet
14:20 logmsgbot: ema@palladium conftool action : set/pooled=no; selector: cp4015.ulsfo.wmnet
14:20 ema: upgrading cp4015 to Varnish 4 (T131502)
14:06 gehel: restart elasticsearch on logstash1004 to validate fix for T142357
14:03 hashar: European SWAT deploy is complete
13:59 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.16/includes/api/ApiUpload.php: ApiUpload: Better handle unreasonably large metadata in 'imageinfo' T143993 (duration: 00m 46s)
13:44 logmsgbot: ema@palladium conftool action : set/pooled=yes; selector: cp4014.ulsfo.wmnet
13:37 logmsgbot: ema@palladium conftool action : set/pooled=no; selector: cp4014.ulsfo.wmnet
13:36 ema: upgrading cp4014 to Varnish 4 (T131502)
13:36 logmsgbot: hashar@tin Synchronized wmf-config/missing.php: Do not prepend protocol in missing.php T141208 (duration: 00m 47s)
13:26 logmsgbot: hashar@tin Synchronized wmf-config/throttle.php: Add throttling exception for UBC T143951 (duration: 00m 46s)
13:19 logmsgbot: hashar@tin Synchronized wmf-config/InitialiseSettings.php: Disable ORES for reverted, goodfaith and wp10 models T143988 (duration: 02m 43s)
13:15 hashar: Pulled https://gerrit.wikimedia.org/r/#/c/306951/ on mw1099
13:14 hashar: Pulled https://gerrit.wikimedia.org/r/#/c/306904/ on mw1099
13:11 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.16/extensions/CirrusSearch/includes/Query/FullTextSimpleMatchQueryBuilder.php: Fallback to QueryString if we detect acronyms T143541 (duration: 00m 50s)
13:08 hashar: Pulled https://gerrit.wikimedia.org/r/#/c/307261/ on mw1099
12:51 logmsgbot: ema@palladium conftool action : set/pooled=yes; selector: cp4013.ulsfo.wmnet
12:42 logmsgbot: ema@palladium conftool action : set/pooled=no; selector: cp4013.ulsfo.wmnet
12:42 ema: upgrading cp4013 to Varnish 4 (T131502)
12:05 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-urd-hin_0.1.0~r64379-1+wmf1
12:05 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-sme-nob_0.6.0~r61921-1+wmf1
12:05 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-isl-eng_0.1.0~r66083-1+wmf1
12:05 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-hbs-slv_0.1.0~r59294-1+wmf1
12:05 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-hbs-eng_0.1.0~r57598-1+wmf1
11:52 elukey: reimaging mw209[01] to jessie
11:31 moritzm: reimaging mw2088 to jessie
11:07 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: giella-sme_0.0.20150917~r121176-1+wmf1
11:07 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-isl_0.1.0~r65494-1+wmf1
11:07 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-hbs_0.5.0~r68212-1+wmf1
11:07 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-eus_0.1.0-1+wmf1
11:07 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-en-es_0.8.0+svn~57502-2+wmf1
11:07 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-en-ca_0.9.3~r61328-2+wmf1
10:52 hashar: Upgrading Zuul server on gallium zuul_2.5.0-8-gcbc7f62-wmf1precise1 zuul_2.5.0-8-gcbc7f62-wmf2precise1 T144088
10:09 moritzm: disabling puppet on jessie mediawiki hosts in eqiad for staged merged of https://gerrit.wikimedia.org/r/#/c/306225/
09:58 elukey: Executed 'kafka preferred-replica-election' on kafka1012 to rebalance Kafka broker leaders (T144158)
09:55 moritzm: depooled mw1266, running some tests with systemd unit override and hhvm update
09:44 logmsgbot: ema@palladium conftool action : set/pooled=yes; selector: cp4007.ulsfo.wmnet
09:28 ema: upgrading cp4007 to Varnish 4 (T131502)
08:18 hashar: Upgrading Zuul server on gallium zuul 2.1.0-391-gbc58ea3-wmf2precise1 => zuul_2.5.0-8-gcbc7f62-wmf1precise1 T144088
02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Aug 29 02:31:06 UTC 2016 (duration 5m 53s)
02:25 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.16) (duration: 11m 25s)

2016-08-28

16:51 hoo: Ran T132839-Workarounds.sh from my home in terbium (see T132839)
02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Aug 28 02:30:02 UTC 2016 (duration 5m 58s)
02:24 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.16) (duration: 11m 19s)

2016-08-27

02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Aug 27 02:30:35 UTC 2016 (duration 5m 50s)
02:24 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.16) (duration: 11m 12s)

2016-08-26

22:51 apergos: from a few hours ago: (08:42:47 μμ) bd808: !log Updated striker to fix T143956
22:49 apergos: running html dmp for en wikipedia manually out of screen session (ariel) on francium

2016-08-25

15:13 ema: pooling cp4005 backend (varnish 4 cache_upload) T131502
14:37 elukey: upgrading httpd to 2.4.10-10+deb8u6+wmf2 on mw1269/mw127[01]
14:17 moritzm: re-enabled puppet on kafka*, mc* and rdb* hosts
13:45 hashar: European SWAT is done
13:44 logmsgbot: zfilipin@tin Synchronized php-1.28.0-wmf.16/extensions/UniversalLanguageSelector: SWAT: ext.uls.compactlinks: consistently normalize language codes (T143867) (duration: 00m 47s)
13:42 logmsgbot: zfilipin@tin Synchronized php-1.28.0-wmf.15/extensions/UniversalLanguageSelector: SWAT: ext.uls.compactlinks: consistently normalize language codes (T143867) (duration: 00m 49s)
13:40 moritzm: temporarily disabling puppet on kafka*, mc* and rdb* hosts (to enable ferm ipsec change in controlled stages)
13:09 moritzm: puppet re-enabled on all kafka* hosts
12:00 moritzm: temporarily disabling puppet on kafka* hosts (to enable ferm changes in controlled stages)
10:00 logmsgbot: ema@palladium conftool action : set/pooled=yes; selector: cp4005.ulsfo.wmnet (tags: ['dc=ulsfo', 'cluster=cache_upload', 'service=varnish-fe'])
10:00 logmsgbot: ema@palladium conftool action : set/pooled=yes; selector: cp4005.ulsfo.wmnet (tags: ['dc=ulsfo', 'cluster=cache_upload', 'service=nginx'])
09:52 moritzm: temporarily stop irc bot, until puppet has self-healed
08:17 moritzm: restarted hhvm on mw1191 and mw1216, got stuck
07:07 moritzm: installing harfbuzz security updates
03:28 logmsgbot: tgr@tin Synchronized php-1.28.0-wmf.16/includes/specialpage/AuthManagerSpecialPage.php: UBN fix for T143840 (duration: 00m 49s)
02:55 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Aug 25 02:55:50 UTC 2016 (duration 6m 30s)
02:49 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.16) (duration: 11m 09s)
02:27 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.15) (duration: 10m 41s)
01:49 mutante: iridium - temp. stopping phd service for home dir change
01:08 mutante: radium apt-get autoremove; apt-get upgrade (openssh, openssl, passwd, sudo, libpam, libc6 :)
00:51 urandom: T137474: Stopping dumps in RESTBase staging, and reverting xenon.eqiad.wmnet to Cassandra 2.2.6-wmf1
00:32 logmsgbot: ebernhardson@tin Synchronized php-1.28.0-wmf.16/extensions/CirrusSearch/includes/Job/CheckerJob.php: Fix CirrusSearch CheckerJob stuck in a loop (duration: 00m 47s)
00:08 mutante: chromium back in service - both eqiad DNS recursors now on jessie

2016-08-24

23:43 logmsgbot: dzahn@palladium conftool action : set/pooled=yes; selector: name=chromium.wikimedia.org
23:39 mutante: chromium - install ntpdate, stop ntp, sync time with hydrogen, start ntp, remove ntpdate
23:36 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.15/extensions/UploadWizard/resources: More debug logging for Firefox's 'NS_ERROR_NOT_AVAILABLE' exceptions (T136831) (duration: 00m 49s)
23:35 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.16/extensions/UploadWizard/resources/: More debug logging for Firefox's 'NS_ERROR_NOT_AVAILABLE' exceptions (T136831) (duration: 00m 47s)
23:33 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.16/extensions/ProofreadPage/ProofreadPage.body.php: Fix ProofreadPage::updatePrIndex signature (T143817) (duration: 00m 50s)
22:59 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings-labs.php: labs-only change: https://gerrit.wikimedia.org/r/298919 (duration: 00m 46s)
22:58 logmsgbot: krenair@tin Synchronized wmf-config/LabsServices.php: labs-only change: https://gerrit.wikimedia.org/r/305837 (duration: 00m 48s)
22:49 mutante: chromium - revoking and re-signing puppet certs, salt keys, initial puppet run..
22:43 yuvipanda: force a puppet run on californium
22:27 bd808: Deployed striker (2fdf103) to californium
22:20 mutante: rebooting chromium into PXE
22:15 papaul: installing puppetmaster200[1-2]
22:05 mutante: stopping puppet and pdns-recursor on chromium
21:57 logmsgbot: krinkle@tin Synchronized multiversion/MWMultiVersion.php: Ie9c568a87ae - No-op clean-up (duration: 00m 49s)
21:54 yuvipanda: forcing puppet run on californium
21:50 mutante: running puppet on lvs servers, removing chromium from resolv.conf for reinstall
21:45 yuvipanda: re-enabled puppet, ran puppet and verified ORES is ok on scb1001 / 1002
21:40 yuvipanda: re-enabled puppet, ran puppet and verified ORES is ok on scb2001 / 2002
21:27 logmsgbot: dzahn@palladium conftool action : set/pooled=no; selector: name=chromium.wikimedia.org
21:27 mutante: depooling chromium for reinstall. scheduled downtime for host and service IPs
21:25 yuvipanda: disable puppet on scb2001 and 2002 as well
21:24 yuvipanda: disable puppet on scb1001 and 1002
21:20 yuvipanda: forcing puppet run on tin
21:20 yuvipanda: forcing puppet run on scb1001
21:14 bd808: Forced puppet run on logstash1003 but cron had beaten me to it
21:12 bd808: Forcing puppet run on logstash1002
21:08 hoo: Ran DELETE FROM wbs_propertypairs WHERE pid1 = '641' on Wikidata for T132839
21:07 bd808: Forcing puppet run on logstash1001
20:59 urandom: T137474: Upgrading xenon.eqiad.wmnet to cassandra_2.2.6-wmf2
20:57 bd808: Created initial tables for striker in striker db on m5-master
20:53 yuvipanda: gave bd808 password for striker db on terbium
20:39 yurik: deployed kartotherian https://gerrit.wikimedia.org/r/#/c/306518/
20:31 yurik: deployed tilerator https://gerrit.wikimedia.org/r/#/c/306514/
20:03 urandom: T137474 Starting htmldumper in RESTBase Staging
20:02 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.16/extensions/ProofreadPage/ProofreadPage.body.php: Fix hooks signatures T143817 (duration: 00m 47s)
19:47 mobrovac: change-prop deploying 28a0057
19:00 logmsgbot: hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.16
18:44 logmsgbot: thcipriani@tin Synchronized wmf-config/CirrusSearch-production.php: SWAT: Cirrus: Send more like this queries to default cluster (duration: 00m 46s)
18:37 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Record content security policy events in log stash (duration: 00m 52s)
18:23 logmsgbot: thcipriani@tin Synchronized wmf-config: SWAT: Change ORES threshold to soft as default (T143738) (duration: 01m 01s)
18:17 paravoid: reenabling cr2-codfw:xe-5/0/1 (link to cr2-eqiad), recovered since 17:02 UTC
18:17 ebernhardson: restarted kibana on logstash1001
17:44 jynus: overwritting /etc/default/prometheus-mysqld-exporter on all trusty mysql nodes on codfw
17:27 gehel: deleting logstash indices from before august 1st (T142357)
17:04 logmsgbot: demon@tin Synchronized wmf-config/: Remove old obsolete ExtensionMessages files (duration: 00m 49s)
16:50 godog: bounce uwsgi on labmon1001 T143556
16:15 logmsgbot: ema@palladium conftool action : set/pooled=no; selector: cp4005.ulsfo.wmnet (tags: ['dc=ulsfo', 'cluster=cache_upload', 'service=varnish-fe'])
16:15 logmsgbot: ema@palladium conftool action : set/pooled=no; selector: cp4005.ulsfo.wmnet (tags: ['dc=ulsfo', 'cluster=cache_upload', 'service=nginx'])
15:58 papaul: !log wdqs200[1-2] - signing puppet certs, salt-key, initial run
15:41 logmsgbot: ema@palladium conftool action : set/pooled=yes; selector: cp4005.ulsfo.wmnet (tags: ['dc=ulsfo', 'cluster=cache_upload', 'service=varnish-fe'])
15:41 logmsgbot: ema@palladium conftool action : set/pooled=yes; selector: cp4005.ulsfo.wmnet (tags: ['dc=ulsfo', 'cluster=cache_upload', 'service=nginx'])
15:26 _joe_: attempting to restart apache on rhodium, swapping, load exploding
15:22 papaul: installing wdqs200[1-2]
14:41 logmsgbot: demon@tin Synchronized multiversion/getMWVersion.php: fixup error message (duration: 00m 47s)
13:53 gehel: rolling restart of logstash nodes to validate fix to T142357
13:50 hashar: European SWAT done
13:49 logmsgbot: zfilipin@tin Synchronized wmf-config/InitialiseSettings.php: Fully restrict uploads on ms.wikipedia T141227 (duration: 00m 46s)
13:48 logmsgbot: zfilipin@tin Synchronized dblists/commonsuploads.dblist: Fully restrict uploads on ms.wikipedia T141227 (duration: 00m 46s)
13:39 ema: upgrading cp4005 to Varnish 4 (T131502)
13:27 logmsgbot: zfilipin@tin Synchronized wmf-config/InitialiseSettings.php: Make default ORES threshold soft (higher threshold) T143738 (duration: 00m 59s)
12:56 mobrovac: changeprop deploying c5fd932
12:29 gehel: remove old logstash indices that were not deleted after 31 days
11:10 godog: enable collection of mysqld metrics from prometheus2002 too
10:49 moritzm: start installing hhvm updates in eqiad
10:13 gehel: increase recovery throttling on elasticsearch codfw to reduce rolling restart time
09:53 moritzm: disabled creation of unprivileged user namespaces on trusty systems via sysctl kernel.unprivileged_userns_clone
09:30 gehel: starting rolling restart of elasticearch codfw for JVM and elasticsearch upgrade
09:03 hashar: gallium contint::firewall: Limited to production networks https://gerrit.wikimedia.org/r/301627 . For monitoring do: grep iptables-dropped /var/log/syslog
08:39 ema: dns: re-enabling codfw
08:29 paravoid: disabling cr2-codfw:xe-5/0/1 (link to cr2-eqiad), flapping since 07:31 UTC
08:09 ema: disabling codfw in dns
03:07 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Aug 24 03:07:38 UTC 2016 (duration 6m 55s)
03:00 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.16) (duration: 18m 24s)
02:25 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.15) (duration: 10m 16s)
01:57 mutante: temp stop dhcp service on install2001 - debug

2016-08-23

23:45 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Enable Related Articles on fr.wikinews (T143480) (duration: 00m 53s)
23:28 mutante: restarted grrrit-wm, apache and gerrit on lead
23:23 mutante: gerrit restarting for config changes 303355, 304977
22:04 bd808: Updated tin:/srv/deployment/logstash/plugins to d18b1c6 (Add output plugin for Sentry)
21:27 SMalyshev: redeploying WDQS GUI to fix examples breakage
21:27 logmsgbot: dzahn@palladium conftool action : set/pooled=yes; selector: dc=eqiad,cluster=dns,name=hydrogen.wikimedia.org
21:05 mutante: hydrogen - reinstall finished, re-added to salt, restarted ntpd
20:42 mutante: hydrogen - signing new puppet cert
20:22 mutante: hydrogen - reinstalling one more time, wrong partitioning
19:55 mutante: re-signing new puppet certs for hydrogen, initial run, new salt key
19:34 hashar: 1.28.0-wmf.16 to group0 looks successful.
19:19 logmsgbot: demon@tin rebuilt wikiversions.php and synchronized wikiversions files: fix compile & sync of wikiversions
19:17 logmsgbot: hashar@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 50s)
19:13 logmsgbot: hashar@tin Synchronized wikiversions.json: Group0 to 1.28.0-wmf.16 T141551 (duration: 00m 48s)
19:04 mutante: rebooting hydrogen
19:04 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix UI l10n for Help page link on commons.wikimedia.org (T143564) (duration: 00m 47s)
18:56 logmsgbot: bblack@palladium conftool action : set/pooled=no; selector: dc=eqiad,cluster=dns,name=hydrogen.wikimedia.org
18:26 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.15/resources/src/mediawiki.widgets/mw.widgets.CategoryCapsuleItemWidget.js: SWAT: Debug logging for "queue[title] undefined" (T139130) (duration: 00m 50s)
18:24 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.16/resources/src/mediawiki.widgets/mw.widgets.CategoryCapsuleItemWidget.js: SWAT: Debug logging for "queue[title] undefined" (T139130) (duration: 00m 50s)
18:22 mutante: all mw appservers are installing all the font packages now, not just imagescalers. this should fix some issues with EasyTimeline on zh projects and more
18:13 kaldari: mwscript maintenance/updateCollation.php --wiki=svwiki --force
18:12 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Switching Swedish Wikipedia to uca-sv-u-kn collation (T142113) (duration: 00m 58s)
17:35 mobrovac: changeprop deploying 519ad9d
16:27 jynus: rebooting es2004 for hardware maintenance T143220
14:25 moritzm: installing libgd security updates
14:13 moritzm: rebooting conf1002 for kernel update
14:10 gehel: cleanup list of banned node from elasticsearch eqiad cluster
14:05 logmsgbot: hashar@tin Finished scap: testwiki to php-1.28.0-wmf.16 T141551 (duration: 28m 45s)
13:48 moritzm: rebooting conf1001 for kernel update
13:37 Dereckson: Run namespaceDupes maintenance script on azwiki and azwiktionary (T143580)
13:37 godog: remove some old compilations from puppet compiler filling up compiler02.eqiad.wmflabs disks T143671
13:36 logmsgbot: hashar@tin Started scap: testwiki to php-1.28.0-wmf.16 T141551
13:34 hashar: Restarting Jenkins
13:28 hashar: European SWAT deploy is complete
13:27 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.15/maintenance/namespaceDupes.php: Run LinksDeletionUpdate after commit() in namespaceDupes.php T143631 (duration: 00m 52s)
13:25 moritzm: rebooting conf1003 for kernel update
13:23 logmsgbot: hashar@tin Synchronized wmf-config/CommonSettings.php: Remove no longer relevant $wgTranslateTasks overrides (duration: 00m 48s)
13:23 _joe_: restarted pybal on lvs1011 for testing with etcd reboots
13:21 logmsgbot: hashar@tin Synchronized wmf-config/CommonSettings.php: Remove English for all groups from $wgTranslateBlacklist T124013 (duration: 00m 54s)
13:16 logmsgbot: hashar@tin Synchronized wmf-config/InitialiseSettings.php: Enable T143073 debug log channel (duration: 02m 19s)
12:45 mobrovac: citoid deploying f711219
12:43 hashar: on tin dropping stall versions /srv/mediawiki-staging/php-1.28.0-wmf.{8,9,10} T141551
11:22 hashar: Cutting MediaWiki branch 1.28.0-wmf.16 | T141551
10:07 moritzm: upgrading hhvm on codfw mediawiki servers
09:00 Krenair: updated wikitech-static to MW 1.27.1
08:52 hashar: Jenkins had some deadlocks preventing builds from processing. Resolved by disabling/reenabling the Gearman client
07:29 moritzm: installing botan security updates on trusty systems
05:52 mutante: install2001 - "MD RAID" and "MegaRAID" icinga checks and both fail? new/test? install1001 doesn't have these checks - disabled notifications
03:07 ejegg: updated payments-wiki from 2b027e313ccecc2b93f214a94738b4f94899f347 to a472ff0a8c62f697221d71647f4013e5c2dfcd45
02:28 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Aug 23 02:28:07 UTC 2016 (duration 5m 44s)
02:22 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.15) (duration: 10m 16s)

2016-08-22

23:58 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.15/extensions/AbuseFilter/Views/: Let abusefilter-modify users see history of hidden filters (gerrit:305596+gerrit:306077, T143365) (duration: 00m 50s)
23:22 Dereckson: NamespaceDupes maintenance script run on sk.wikipedia
23:19 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Fix user namespaces on Slovak Wikipedia (T143472) (duration: 00m 58s)
22:43 Dereckson: Run maintenance ssript namespaceDupes.php on azwiktionary (T143580)
21:19 urandom: Restarting restbase staging in codfw
20:29 adamw_: update CRM from a30fc0aa2c2dc3a7a9a0b09bef19d112bdf5f98e to 43cca607ed0d6d2f91e3f18d9df1473021b40f88
20:13 subbu: finished deploying parsoid sha df53a991
20:08 papaul: wezen - signing puppet certs, salt-key, initial run
20:08 subbu: synced new parsoid code; restarted parsoid on wtp1001 as a canary
20:05 subbu: starting parsoid deploy
18:47 logmsgbot: otto@palladium conftool action : set/pooled=yes; selector: kafka1002.eqiad.wmnet
18:45 logmsgbot: otto@palladium conftool action : set/pooled=no; selector: kafka1002.eqiad.wmnet
18:44 logmsgbot: otto@palladium conftool action : set/pooled=yes; selector: kafka1001.eqiad.wmnet
18:39 logmsgbot: otto@palladium conftool action : set/pooled=no; selector: kafka1001.eqiad.wmnet
18:39 logmsgbot: otto@palladium conftool action : set/pooled=yes; selector: kafka2002.codfw.wmnet
18:36 logmsgbot: otto@palladium conftool action : set/pooled=no; selector: kafka2002.codfw.wmnet
18:35 logmsgbot: otto@palladium conftool action : set/pooled=yes; selector: kafka2001.codfw.wmnet
18:32 logmsgbot: otto@palladium conftool action : set/pooled=no; selector: kafka2001.codfw.wmnet
18:32 logmsgbot: otto@palladium conftool action : set/pooled=no; selector: kafka2001.eqiad.wmnet
18:30 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase2009-a.codfw.wmnet
18:29 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase2008-a.codfw.wmnet
18:29 ottomata: deploying eventlogging eventbus and a topic config patch, will depool each node as i do and check that all is well
18:26 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1015-a.eqiad.wmnet
18:26 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1014-a.eqiad.wmnet
18:25 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1009-a.eqiad.wmnet
18:24 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1012-a.eqiad.wmnet
18:24 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1011-a.eqiad.wmnet
18:23 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ORES review tool for English Wikipedia (T140003) (duration: 01m 04s)
18:23 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1010-a.eqiad.wmnet
18:14 urandom: Restarting Cassandra instances to apply new TLS cert restbase-test2003.codfw.wmnet
18:13 urandom: Restarting Cassandra instances to apply new TLS cert restbase-test2002.codfw.wmnet
18:04 urandom: Restarting Cassandra instances to apply new TLS cert restbase-test2001.codfw.wmnet (for reals this time)
17:48 godog: cassandra: replace certs for restbase-test200[123]-[ab] - T120662
17:35 urandom: Restart Cassandra instances to apply updated certificates, restbase-test2001.codfw.wmnet
17:09 gehel: deplyoing latest GUI + updater version on wdqs100? servers
16:57 urandom_: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1013-a.eqiad.wmnet
16:21 papaul: installing wezen new syslog server
14:46 hashar: European swat completed 8/8 100%
14:46 bblack: text caches: geoip testing looks good, re-enabling+running puppet for the rest
14:46 logmsgbot: hashar@tin Synchronized wmf-config/InitialiseSettings.php: Set Flow as default for User talk on kabwiki - T140588 (duration: 00m 59s)
14:36 bblack: text cache puppets disabled, cp1065 testing merged https://gerrit.wikimedia.org/r/253619
13:44 jynus: stopping db1073 to clone compressed Innodb data to db2034
13:43 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.15/extensions/ProofreadPage/ProofreadPage.body.php: Fix unknown constant AS_HOOK_ERROR issue in ProofreadPage (T143471) (duration: 00m 48s)
13:42 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.15/extensions/UniversalLanguageSelector/resources/js/ext.uls.compactlinks.js: Apply toLowerCase when reading featured articles (T143527) (duration: 00m 50s)
13:41 logmsgbot: dereckson@tin scap aborted: php-1.28.0-wmf.15/extensions/UniversalLanguageSelector / resources/js/ext.uls.compactlinks.js Apply toLowerCase when reading featured articles (T143527) (duration: 00m 04s)
13:41 logmsgbot: dereckson@tin Started scap: php-1.28.0-wmf.15/extensions/UniversalLanguageSelector / resources/js/ext.uls.compactlinks.js Apply toLowerCase when reading featured articles (T143527)
13:40 logmsgbot: hashar@tin Synchronized wmf-config/CommonSettings.php: Add Collection render note for articles rdf2latex -t - T135613 (duration: 00m 48s)
13:39 logmsgbot: hashar@tin Synchronized wmf-config/CommonSettings.php: Add Collection render note for articles & rdf2latex -t - T135613 (duration: 00m 49s)
13:35 logmsgbot: hashar@tin Synchronized wmf-config/InitialiseSettings.php: Enable WikidataPageBanner on ro.wikivoyage - T142963 (duration: 00m 49s)
13:25 logmsgbot: hashar@tin Synchronized wmf-config/InitialiseSettings.php: Restrict local upload on ar.wikipedia - T142450 (duration: 00m 49s)
13:16 logmsgbot: hashar@tin Synchronized wmf-config/throttle.php: [cleanup] Remove old throttle rules (duration: 00m 48s)
13:09 logmsgbot: hashar@tin Synchronized wmf-config/InitialiseSettings.php: Enable transwiki upload for tcywiki T143397 (duration: 00m 58s)
12:10 mobrovac: restbase deploy end of e9e5ff1
11:57 mobrovac: restbase deploy start of e9e5ff1
10:17 gehel: starting rolling restart of elasticearch eqiad for JVM and elasticsearch upgrade
09:12 jynus: stopping db2034 for cloning and reimage
08:43 _joe_: restarting hhvm on mw1278, deadlock in HPHP::Treadmill::getAgeOldestRequest
08:41 moritzm: restarted hhvm on mw1162, was deadlocked
02:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Aug 22 02:29:09 UTC 2016 (duration 5m 45s)
02:23 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.15) (duration: 10m 26s)

2016-08-21

15:01 Dereckson: Run deleteEqualMessages maintenance script on urwiki (T45917)
02:27 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Aug 21 02:27:39 UTC 2016 (duration 5m 41s)
02:22 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.15) (duration: 10m 05s)

2016-08-20

13:34 bblack--: rolling, depooled restarts for varnish-frontends: all text, upload in esams, eqiad, codfw.
02:35 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Aug 20 02:35:10 UTC 2016 (duration 5m 58s)
02:29 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.15) (duration: 09m 46s)

2016-08-19

22:58 mutante: works fine, no more "bastiononly" group - all users get automatically added on bastions
22:55 mutante: temp. disabled puppet on bastion hosts, confirming change 301149 works as expected
20:03 mutante: removed 2fa for user Florianschmidtwelzow on labswiki, confirmed via file on tools-login and IRC identity
18:00 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase2006-a.codfw.wmnet
17:52 logmsgbot: dereckson@tin Finished scap: (no message) (duration: 26m 44s)
17:26 Dereckson: Current scap is for Gerrit:305662 (T143402)
17:26 logmsgbot: dereckson@tin Started scap: (no message)
16:20 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase2004-a.codfw.wmnet
16:19 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase2007-a.codfw.wmnet
16:11 bblack: cache_maps: rolling frontend cache restarts, ~5 minute window
13:45 bblack: cache_misc: varnish-frontend global rolling restart (~3 mins to completion)
08:56 jynus: deploying schema change on s2 hosts T139090
07:51 moritzm: depooling mw2215 for some tests with the hhvm systemd unit
07:30 moritzm: installing gnupg security updates
06:32 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.15/includes/OutputPage.php: T143357 (duration: 00m 55s)
02:26 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Aug 19 02:26:42 UTC 2016 (duration 6m 6s)
02:20 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.15) (duration: 08m 59s)
02:08 logmsgbot: mattflaschen@tin Synchronized wmf-config/InitialiseSettings-labs.php: Beta-only change (duration: 00m 54s)
01:31 logmsgbot: mattflaschen@tin Synchronized wmf-config/InitialiseSettings-labs.php: Beta-only change (duration: 00m 51s)
00:31 awight: rollback civicrm to a30fc0aa2c2dc3a7a9a0b09bef19d112bdf5f98e

2016-08-18

23:58 logmsgbot: tgr@tin Synchronized wmf-config/InitialiseSettings.php: SWAT gerrit:305558 Set timezone to Europe/Ljubljana on sl. projects (duration: 00m 49s)
23:52 logmsgbot: tgr@tin Synchronized wmf-config/abusefilter.php: SWAT gerrit:293109 Remove AbuseFilter B/C config (duration: 00m 49s)
23:46 logmsgbot: tgr@tin Synchronized wmf-config/InitialiseSettings.php: SWAT gerrit:303939 Remove $wgDisableAuthManager (duration: 00m 49s)
23:45 logmsgbot: tgr@tin Synchronized wmf-config/CommonSettings.php: SWAT gerrit:303939 Remove $wgDisableAuthManager (duration: 00m 48s)
23:43 logmsgbot: tgr@tin Synchronized wmf-config/wikitech.php: SWAT gerrit:303939 Remove $wgDisableAuthManager (duration: 00m 49s)
23:42 kaldari: mwscript sql.php --wiki=enwikivoyage /srv/mediawiki/php/extensions/PageAssessments/db/addReviewsTable.sql
23:41 kaldari: mwscript sql.php --wiki=enwikivoyage /srv/mediawiki/php/extensions/PageAssessments/db/addProjectsTable.sql
23:36 logmsgbot: tgr@tin Synchronized wmf-config/InitialiseSettings.php: SWAT gerrit:275299 Disable wgUseFilePatrol in huwiki (duration: 01m 00s)
22:44 mutante: phab2001 - puppet re-enabled (but phd service stopped, after gerrit 305591)
22:16 logmsgbot: krenair@tin Synchronized wmf-config/LabsServices.php: for labs, no-op in prod: https://gerrit.wikimedia.org/r/#/c/305589/ (duration: 00m 56s)
21:23 Dereckson: Run deleteEqualMessages maintenance script on eswikibooks, eswikiquote, eswikisource, eswiktionary and eswikiversity (T45917)
21:19 awight: update civicrm from a30fc0aa2c2dc3a7a9a0b09bef19d112bdf5f98e to da572ab911763612a4b7005056821918ac630cbe
20:22 mutante: phab2001 disabled phd service again
20:04 Dereckson: Run initSiteStats maintenance script on mhrwiki and newwiki (T143352)
19:45 mutante: phab2001 - reenabled puppet temp
19:10 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.15
18:49 Guest64769: phab2001 manually removed phabricator crons since puppet is disabled there
18:48 awight: rollback civicrm from 41844071e3018b21f2c36ab692c4b43b92cab8d0 to a30fc0aa2c2dc3a7a9a0b09bef19d112bdf5f98e
18:45 awight: update civicrm from a30fc0aa2c2dc3a7a9a0b09bef19d112bdf5f98e to 41844071e3018b21f2c36ab692c4b43b92cab8d0
17:24 logmsgbot: gehel@tin Synchronized wmf-config/InitialiseSettings.php: switching search back to eqiad (duration: 00m 49s)
17:19 gehel: elasticsearch eqiad recovery throttling back to standard 20mb recovery
17:15 godog: reinstall ms-be1027 after ssd replaced T140374
17:12 chasemp: nodepool restart with new settings for rate/env/ready for jessie from puppet
16:56 mobrovac: mathoid deploying 75606c71
16:16 chasemp: restart nodepool with STATSD_HOST env variable for test
15:52 gehel: reset postgresql replication on maps1003 (correction, not maps1001)
15:51 gehel: reset postgresql replication on maps1001
15:21 logmsgbot: aude@tin Synchronized wmf-config/Wikibase.php: Bump cache epoch for Wikidata (duration: 00m 49s)
15:12 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable lazy loaded images on mobile web (duration: 01m 00s)
14:54 cmjohnson1: ms-be1005 replacing disk Slot Number: 3
14:41 paravoid: cr1-eqiad and cr2-eqiad: replace obsolete PIM BFD statements with new family inet/inet6 ones
14:38 paravoid: cr1-eqiad: JunOS, SCB and linecard upgrade is over
14:34 paravoid: cr1-eqiad: activating PyBal/LVS BGP sessions
14:32 paravoid: cr1-eqiad: activating Transit4/6 BGP sessions
14:31 paravoid: cr1-eqiad: activating Private-Peer4/6 BGP sessions
14:31 paravoid: cr1-eqiad: reenabling Private-Peer/Transit interfaces
14:29 mark: cr2-eqiad: setting ae3.1019 inet vrrp priority to default (from 50)
14:27 paravoid: cr1-eqiad: removing deprioritization of all VRRP groups (priority=50)
14:24 paravoid: cr1-eqiad: reenabling xe-5/0/3 (link to pfw) and the Fundraising BGP group
14:24 paravoid: cr1-eqiad: reenabling xe-4/2/0 (link to cr1-codfw) and xe-4/2/2 (link to cr2-knams)
14:20 paravoid: cr1-eqiad: reenabling all asw row D interfaces
14:19 paravoid: cr1-eqiad: reenabling all asw row B/C interfaces
14:18 gehel: increasing elasticsearch recovery throttling to 40mb to speed up eqiad recovery
14:18 paravoid: cr1-eqiad: reenabling all asw row A interfaces
14:12 paravoid: cr1-eqiad: activate chassis redundancy graceful-switchover
13:57 paravoid: cr1-eqiad: cmjohnson1 replacing both SCBs with SCBE2s and adding a new linecard
13:54 paravoid: cr1-eqiad: shutting down both routing-engines and powering off
13:38 paravoid: cr1-eqiad: setting "chassis network-services enhanced-ip" and rebooting both REs
13:20 paravoid: cr1-eqiad: upgrading re1 and rebooting
13:18 paravoid: cr1-eqiad: toggling mastership between routing-engines (re1->re0)
13:00 jynus: reloaded dbproxy1010's haproxy service to point to the original master
12:58 paravoid: cr1-eqiad: upgrading re0 and rebooting
12:56 paravoid: cr2-eqiad: remove VRRPv3 backwards compatibility (delete protocols vrrp checksum-without-pseudoheader)
12:53 paravoid: cr1-eqiad: deactivate chassis redundancy graceful-switchover
12:52 paravoid: cr1-eqiad: disabling all asw row A-C interfaces
12:44 paravoid: cr1-eqiad: disabling all asw row D interfaces
12:41 logmsgbot: gehel@tin Synchronized wmf-config/InitialiseSettings.php: switching search to codfw (duration: 00m 56s)
12:40 gehel: switching search traffic to codfw
12:13 paravoid: cr1-eqiad: reenabling all asw row A/B/C interfaces
12:05 paravoid: cr1-eqiad: disabling all asw row B/C interfaces
12:04 paravoid: cr1-eqiad: disabling all asw row A interfaces
12:01 paravoid: cr1-eqiad: disabling Transit/Fundraising interfaces
11:56 paravoid: cr1-eqiad: deactivating Fundraising BGP session
11:54 paravoid: cr1-eqiad: deactivating PyBal/LVS backup static routes
11:47 paravoid: cr1-eqiad: deactivating Transit4/6 and Private-Peer4/6 BGP sessions
11:31 moritzm: uploaded new Linux package for jessie-wikimedia to carbon (now based on 4.4.18)
11:06 paravoid: cr1-eqiad: deactivating BGP sessions with PyBal/LVS
11:04 paravoid: cr1-eqiad: disabling xe-4/2/0 (link to cr1-codfw) and xe-4/2/2 (link to cr2-knams)
10:51 paravoid: cr1-eqiad: deprioritizing all groups (priority=50)
10:20 moritzm: upgrading remaining canary application servers to hhvm 3.12.7
08:59 moritzm: enabled scaling of huge SVGs on image scalers (T111815)
08:58 logmsgbot: jmm@tin Synchronized wmf-config/CommonSettings.php: enable SVG scaling of huge files (duration: 00m 48s)
08:10 jynus: stoping mysql on db1042, db2009 for testing (both are depooled and alerts disabled)
03:06 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Aug 18 03:06:44 UTC 2016 (duration 7m 5s)
02:59 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.15) (duration: 08m 44s)
02:34 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.14) (duration: 11m 58s)
00:27 logmsgbot: maxsem@tin Finished scap: https://gerrit.wikimedia.org/r/#/c/305424/ (duration: 52m 22s)

2016-08-17

23:53 XenoRyet: updated payments-wiki from b449f65dba5697905a87261592e934d7f4898a54 to 2b027e313ccecc2b93f214a94738b4f94899f347
23:35 logmsgbot: maxsem@tin Started scap: https://gerrit.wikimedia.org/r/#/c/305424/
23:27 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/304328/ (duration: 00m 55s)
23:12 logmsgbot: maxsem@tin Synchronized wmf-config/CirrusSearch-production.php: https://gerrit.wikimedia.org/r/#/c/304204/3 (duration: 00m 53s)
22:16 chasemp: restart nodepool
22:15 chasemp: openstack quota set --instances 15 contintcloud
22:03 chasemp: restart nodepool with 10s cycle rate
21:59 chasemp: openstack quota set --instances 14 contintcloud
21:59 chasemp: disable puppet on labnodepool for testing of instance threshold
21:50 chasemp: set contintcloud project instance quota to 12 for testing
21:47 cscott: updated OCG to version e3e0fd015ad8fdbf9da1838c830fe4b075c59a29 (T133001, T142226)
21:46 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.15/extensions/CentralAuth: ef4b5d45f9bb59c978e23f21ed09649aa628c4d1 (duration: 00m 59s)
21:43 cscott: starting OCG deploy
20:43 bearND: deployed mobileapps 81bd74f
20:26 urandom: Restarting Cassandra, aqs1004-a.eqiad.wmnet
20:25 bearND: starting mobileapps deploy
20:09 subbu: finished deploying parsoid sha 3cf877bb
20:04 subbu: synced new parsoid code; restarted parsoid on wtp1001 as a canary
20:01 subbu: starting parsoid deploy
19:23 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.15 refs T140971
18:41 ejegg: disabled donations queue consumer, adyen job runner, pending queue consumer for db rename
18:40 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase2005-a.codfw.wmnet
18:37 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase2003-a.codfw.wmnet
18:31 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase2002-a.codfw.wmnet
18:17 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1008-a.eqiad.wmnet
17:31 ottomata: deploying Kafka main-eqiad -> main-codfw 'eqiad.*' topic mirroing
16:02 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.14/extensions/ContentTranslation: SWAT: Avoid deadlock patterns in cx_corpora updates (T134245) (duration: 00m 50s)
15:59 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.15/extensions/ContentTranslation: SWAT: Avoid deadlock patterns in cx_corpora updates (T134245) (duration: 00m 52s)
15:40 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: wmgEchoMentionStatusNotifications true for test/test2wiki (T141995) (duration: 00m 50s)
15:29 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.15/includes/api/ApiUpload.php: SWAT: Do not call the "UploadStashFile" hook for partially uploaded files (T143161) PART II (duration: 00m 50s)
15:27 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.15/includes/upload/UploadBase.php: SWAT: Do not call the "UploadStashFile" hook for partially uploaded files (T143161) PART I (duration: 00m 53s)
15:15 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor by default for logged-out users on Arabic-script Wikipedias (T142587) (duration: 00m 50s)
14:50 jynus: deploying schema change on s6 hosts T139090
14:47 gehel: rolling restart of elasticsearch logstash cluster for elasticsearch upgrade to 2.3.4
14:45 godog: bounce carbon on graphite machines
14:42 gehel: rolling restart of elasticsearch relforge cluster for elasticsearch upgrade to 2.3.4
14:29 gehel: upgrading elasticsearch plugins to 2.3.4 on elasticsearch, relforge and logstash clusters. Rolling restart coming next.
14:26 gehel: upgrading elasticsearch to 2.3.4 on relforge cluster
13:28 logmsgbot: hoo@tin Synchronized wmf-config/InitialiseSettings.php: Enable allowDataAccessInUserLanguage on meta (T122672) (duration: 00m 56s)
13:10 gehel: rolling restart of relforge100* for JVM upgrade. Short downtime expected.
12:57 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-swe-nor_0.2.0~r69544-1+wmf1
12:57 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-swe-dan_0.7.0~r66063-1+wmf1
12:57 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-dan-nor_1.3.0~r67099-2+wmf1
12:18 godog: restart pybal on low-traffic for thumbor - T139606
12:10 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-hin_0.1.0~r59158-1+wmf1
12:03 moritzm: repooling mw1298 with config change to allow scaling of huge SVGs (to testdrive further before enabling this in general)
11:56 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-nob_0.9.0~r69513-1+wmf1
11:55 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-dan_0.5.0~r67099-2+wmf1
10:07 Amir1: ladsgroup@scb[12]00[12]:~$ sudo service celery-ores-worker restart (T143105)
09:45 godog: upload scap3 3.2.3-1 to carbon T127762
09:37 moritzm: upgraded mw1017 to HHVM 3.12.7 (plus patches)
09:31 moritzm: uploaded hhvm 3.12.7+dfsg+wmf1~trusty1 for trusty-wikimedia to carbon (also includes a fix for T137642)
08:00 moritzm: installing openjdk security updates on the elastic* clusters
07:36 gehel: cleanup and shutdown of nobelium before reclaim
06:34 moritzm: installing openjdk security updates on the stat* hosts
03:15 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Aug 17 03:15:05 UTC 2016 (duration 7m 22s)
03:07 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.15) (duration: 17m 59s)
02:33 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.14) (duration: 11m 31s)

2016-08-16

23:57 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.15/includes/OutputPage.php: 653232f90605 (duration: 00m 52s)
23:56 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.15/includes/resourceloader/ResourceLoaderClientHtml.php: 653232f90605 (duration: 00m 48s)
23:55 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.15/includes/skins/SkinTemplate.php: 05c82731c9831c465 (duration: 00m 49s)
23:49 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/304632/2 (duration: 00m 54s)
23:43 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.14/extensions/Kartographer: https://gerrit.wikimedia.org/r/#/c/305080/ (duration: 00m 51s)
23:39 logmsgbot: twentyafterfour@tin Synchronized php-1.28.0-wmf.15/extensions/BetaFeatures: deploy https://gerrit.wikimedia.org/r/#/c/305148/ (duration: 00m 49s)
23:32 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: touch (duration: 00m 52s)
23:25 logmsgbot: maxsem@tin Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/304631/2 (duration: 00m 53s)
23:13 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.15/extensions/Kartographer: https://gerrit.wikimedia.org/r/#/c/305080/ (duration: 00m 51s)
23:11 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.15/extensions/Graph: https://gerrit.wikimedia.org/r/#/c/305145/ (duration: 00m 59s)
22:34 ejegg: restarted donation queue consumer
22:26 ejegg: paused queue consumer to set up pending db comparison testing
22:02 ejegg: updated Civicrm from 1d24161dd3f0f3ad64c0bf77f06022f30a2b3f2f to a30fc0aa2c2dc3a7a9a0b09bef19d112bdf5f98e
21:37 mutante: restarted grrrit-wm for config change 304746
20:48 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 wikis to 1.28.0-wmf.15
20:40 logmsgbot: twentyafterfour@tin Finished scap: sync testwiki to 1.28.0-wmf.15 refs T140971 (duration: 52m 45s)
19:47 logmsgbot: twentyafterfour@tin Started scap: sync testwiki to 1.28.0-wmf.15 refs T140971
18:02 gehel: starting tile generation for zoom levels 0-10 on maps eqiad
17:46 moritzm: uploaded hhvm 3.12.7+dfsg+wmf1 for jessie-wikimedia to carbon (also includes a fix for T137642)
16:14 bblack: rolling depooled restarts of varnish-frontend on ulsfo upload caches
16:09 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.14/extensions/Flow/maintenance/FlowRestoreLQT.php: SWAT: Query wiki DB for logging table, not Flow DB (T119509) (duration: 00m 57s)
16:09 logmsgbot: bblack@palladium conftool action : set/pooled=yes; selector: name=cp4006.ulsfo.wmnet
16:09 bblack: repooling cp4006
15:58 bblack: rebooting cp4006
15:48 logmsgbot: thcipriani@tin Synchronized portals: SWAT: Bumping portals to master (T140153) (duration: 00m 47s)
15:47 logmsgbot: thcipriani@tin Synchronized portals/prod/wikipedia.org/assets: SWAT: Bumping portals to master (T140153) (duration: 00m 51s)
15:34 ema: depooling cp4006
15:27 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ORES review tool in plwiki (T140005) (duration: 00m 55s)
15:14 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Change login cookies (for "Keep me logged in") to a one year expiry. (T68699) (duration: 01m 08s)
12:53 hoo: Put a better workaround for T132839 in place: Only remove property pairs with context = "item". This keeps ref and qualifier pairs for ext ids intact.
12:13 ema: re-enabling puppet on cache hosts (T138546)
10:52 ema: rolling restart of v4 varnishes (T142810)
06:35 moritzm: restarted hhvm on jobrunner mw1162 (deadlocked)
04:02 mutante: restarted grrrit-wm
03:58 mutante: gerrit restarting to apply config change 302229
03:42 mutante: restarted grrrit-wm
03:21 mutante: gerrit restarting to apply config change 304838
03:21 mutante: lead - restarted apache
02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Aug 16 02:31:10 UTC 2016 (duration 5m 42s)
02:25 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.14) (duration: 11m 31s)
00:21 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: remove utc timezone overrides (duration: 00m 48s)
00:17 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Set timezone to Europe/Ljubljana on sl. projects (T142701) (duration: 00m 49s)
00:06 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Add autopatrolled and rollbacker user groups to it.wikinews (T142571) (duration: 00m 52s)

2016-08-15

23:51 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Fix VE namespaces in he, fa and ko Wikipedias (T118060). Remove redundant svwiktionary wmgVisualEditorAvailableNamespaces entry (Gerrit:304936). (duration: 00m 52s)
23:15 logmsgbot: dereckson@tin Synchronized wmf-config/interwiki.php: Update interwiki map (Gerrit:304622) (duration: 00m 49s)
23:12 eileen: upgrading CiviCRM from cad2c59081e53b877f51be4c853f307f3c4d4949 to 1d24161dd3f0f3ad64c0bf77f06022f30a2b3f2f
23:12 kaldari: mwscript maintenance/updateCollation.php --wiki=mkwiki --force
23:08 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Set collation to uca-mk-u-kn on mk.wikipedia (T26953) (duration: 01m 00s)
21:35 awight: update Thank-you letter to point to the new unsubscribe server
21:17 awight: update paymentswiki from 6b1f3c278308f831a149c986d2269342f94e2a16 to b449f65dba5697905a87261592e934d7f4898a54; update config for FundraisingEmailUnsubscribe
20:17 Krenair: labtestcontrol2001: Raise max_connections mysql global to 500 to match real-labs' on m5-master (db1009). old value was 151, see my comment at T132422
20:10 subbu: finished deploying parsoid sha f039dcf6
20:05 subbu: synced new parsoid code; restarted parsoid on wtp1001 as a canary
20:03 subbu: starting parsoid deploy
19:39 urandom: T140008: Starting major compaction (WP parsoid html, split output) on restbase1007-a.eqiad.wmnet
19:27 urandom: T140008: Staring user-defined compaction (10 tables, highest droppable tombstones), restbase2001-b.codfw.wmnet
18:56 awight: roll back paymentswiki from 75e4d2509a53910e5630ca3dca5950474b587a17 to 6b1f3c278308f831a149c986d2269342f94e2a16
18:52 awight: update paymentswiki from 6b1f3c278308f831a149c986d2269342f94e2a16 to 75e4d2509a53910e5630ca3dca5950474b587a17
18:13 ottomata: starting main-eqiad Kafka upgrade to confluent 0.9, will be stopping and starting brokers on kafka1001 and kafka1002
17:03 gehel: deploying latest wdqs on wdqs100[12].eqiad.wmnet
16:16 mobrovac: restbase restarting in eqiad after cassandra restarts for the openjdk upgrade
16:07 urandom: Restarting Cassandra to apply OpenJDK 8u102 upgrade, restbase1015.eqiad.wmnet
16:00 urandom: Restarting Cassandra to apply OpenJDK 8u102 upgrade, restbase1014.eqiad.wmnet
15:52 urandom: Restarting Cassandra to apply OpenJDK 8u102 upgrade, restbase1013.eqiad.wmnet
15:45 urandom: Restarting Cassandra to apply OpenJDK 8u102 upgrade, restbase1012.eqiad.wmnet
15:38 urandom: Restarting Cassandra to apply OpenJDK 8u102 upgrade, restbase1011.eqiad.wmnet
15:30 urandom: Restarting Cassandra to apply OpenJDK 8u102 upgrade, restbase1010.eqiad.wmnet
15:03 ostriches: gerrit: upgrading 2.12.2 -> 2.12.3. Quick restart will happen.
14:24 moritzm: upgrading firejail on mw* to 0.9.40
13:59 moritzm: upgrading mw1293 to firejail 0.9.40
11:08 moritzm: rolling restart of cassandra on restbase1* to pick up openjdk security update
10:54 moritzm: uploaded gerrit 2.12.3 to apt.wikimedia.org
10:49 mobrovac: restbase restring RB in codfw following the jdk upgrade and cassandra restarts
10:06 moritzm: rolling restart of cassandra on restbase2* to pick up openjdk security update
07:29 moritzm: upgrading openjdk on maps clusters (along with cassandra restart)
02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Aug 15 02:30:30 UTC 2016 (duration 5m 51s)
02:24 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.14) (duration: 11m 21s)

2016-08-14

17:12 paravoid: bumping cr2-knams<->cr1-eqiad OSPF/OSPF3 metric to 1820 (thus activating the new cr2-esams<->cr2-eqiad link which has a metric of 840)
02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Aug 14 02:30:34 UTC 2016 (duration 5m 53s)
02:24 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.14) (duration: 11m 15s)

2016-08-13

20:59 Krenair: labtestcontrol2001: restarted mysql to unbreak labtesthorizon login again. we really need to figure out why this becomes necessary
11:38 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: wfLoadExtension all around (duration: 00m 49s)
10:57 logmsgbot: reedy@tin Synchronized wmf-config/: Simplify WikimediaMessages loading, Remove Orphaned OpenStreetMapSlippyMap config (duration: 00m 50s)
10:45 logmsgbot: reedy@tin Synchronized wmf-config/: Simplify WikimediaMessages loading, Remove Orphaned OpenStreetMapSlippyMap config (duration: 00m 52s)
10:33 logmsgbot: reedy@tin Synchronized wmf-config/extension-list: Fix erroneous plural (duration: 00m 50s)
09:52 logmsgbot: reedy@tin Synchronized wmf-config/extension-list: Fix path to FancyCaptcha extension.json in extension-list (duration: 00m 47s)
09:46 logmsgbot: reedy@tin Synchronized wmf-config/extension-list: Many more to extension.json in extension-list (duration: 00m 50s)
09:13 elukey: extended the maximum kafka topic partition size (350GiB) to upload as well (it was only text before)
08:57 elukey: added temporary override to Kafka topic settings to free disk space: retention.bytes=375809638400
08:42 elukey: restarting cassandra on aqs100[456] (non live cluster) for jvm upgrades and new settings
08:22 elukey: added temporary override to Kafka topic settings to free disk space: retention.bytes=429496729600
08:14 elukey: added temporary override to Kafka topic settings to free disk space: retention.bytes=483183820800
07:31 awight: Removed Ingenico audit processing post-build action to immediately start a thank-you mailing job.
02:33 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Aug 13 02:33:32 UTC 2016 (duration 6m 7s)
02:27 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.14) (duration: 11m 28s)

2016-08-12

20:13 halfak: running FLUSHALL on oresrdb1001 T142857
20:07 Amir1: deploying ores 2ef24f2 to all nodes T142857
20:03 logmsgbot: mattflaschen@tin Synchronized wmf-config/CommonSettings-labs.php: T68699: Enable one-year login on Beta Cluster. Scheduled for prod on Tuesday. (duration: 00m 52s)
20:03 Amir1: deploying ores 2ef24f2 to scb2001.codfw.wmnet (canary node) T142857
16:34 chasemp: reboot labstore1005 to test failure mode
15:36 gehel: reset postgresql maps slave for maps1003
15:18 yurik: deployed tilerator to fix krtotherian restarts on maps100*
15:13 moritzm: upgrading openjdk on restbase-test/xenon/praseodymium/cerium) (along with cassandra restart)
14:33 mobrovac: zotero restarted, mem usage was at 11%
14:16 chasemp: reboot labstore1004/1005
13:54 ema: cache_maps varnish backends rolling restart (T142810)
13:21 gehel: initial configuration of maps1002
11:00 elukey: upgrading httpd to 2.4.10-10+deb8u6+wmf2 on mw126[5678]
10:47 jynus: dropping aft tables from all other hosts after archiving its contents T59185
10:06 jynus: dropping aft tables from all enwiki hosts after archiving its contents T59185
09:38 jynus: dropping aft tables from db1052 T59185
09:32 elukey: restarting Druid java daemons on druid100[123] for openjdk upgrades
08:36 akosiaris: roll restart parsoid to apply https://phabricator.wikimedia.org/rOPUPb32dd25950f1499a79e74fc811c050291b9ec6b8
08:25 moritzm: install postgres security updates on maps clusters
02:45 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Aug 12 02:45:40 UTC 2016 (duration 5m 57s)
02:39 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.14) (duration: 18m 02s)
02:04 logmsgbot: mattflaschen@tin Synchronized php-1.28.0-wmf.14/extensions/Echo: Revert self-mentions pending further investigation and discussion, due to accidental self-mentions. (duration: 01m 04s)
00:51 yurik: deployed kartotherian & tilerator. maps100[134].eqiad are still down (non production)
00:21 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.14/extensions/CentralNotice: CentralNotice deployment gerrit:304420 (duration: 00m 49s)
00:15 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.14/: VE: Fix TextState#getChangeTransaction bug (T141573) ; Echo: Revert "Hack around browser bug in IE breaking badge alignment in Monobook" (gerrit:304415) ; Core: Revert CSS fix (gerrit:304412, T142750) (duration: 08m 58s)

2016-08-11

23:47 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Remove 'gather-hidelist' user right gerrit:303803 (duration: 00m 48s)
23:42 ostriches: gerrit: running puppet to pick up config change, gerrit will do a quick restart
23:37 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: UrlShortener: Whitelist *.wikidata.org (T142055) (duration: 00m 47s)
23:24 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Re-enable thank-you-edit notifications (T128249) (duration: 00m 58s)
23:22 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Re-enable Echo footer notice (T141414) (duration: 00m 54s)
20:25 ejegg: updated SmashPig from 5c180de6e424be10f9d61052c2c8dca7e0e825af to 4ba14a10518f80fe7e20fadc493d78c796c15921
20:08 yuvipanda: restart nova-compute on labvirt1010
20:06 yuvipanda: tools restart rabbitmq-server on labcontrol1001
20:04 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group2 wikis to 1.28.0-wmf.14
19:44 logmsgbot: twentyafterfour@tin Synchronized php-1.28.0-wmf.14/extensions/CentralNotice: deploy https://gerrit.wikimedia.org/r/#/c/304130/ (duration: 00m 58s)
18:18 MaxSem: restarted hhvm on mw1017 to catch up with hhvm-wikidiff2 upgrade
17:49 ema: switched LVS schedulers for text, upload, maps and misc port 80 to source hash scheduling T108827
17:37 ema: applying https://gerrit.wikimedia.org/r/#/c/297418/ to eqiad load balancers
17:16 ema: applying https://gerrit.wikimedia.org/r/#/c/297418/ to codfw load balancers
17:02 Amir1: restarting celery-ores-worker service in scb[12]00[12] T142361
17:00 ema: applying https://gerrit.wikimedia.org/r/#/c/297418/ to ulsfo load balancers
16:36 ema: applying https://gerrit.wikimedia.org/r/#/c/297418/ to esams load balancers
15:51 ema: disabling puppet on lvs hosts to test https://gerrit.wikimedia.org/r/#/c/297418/
14:20 jynus: disabling puppet on db2001-2009; preparing for decomission
13:53 chasemp: restarting nodepool to test stats collection
13:02 elukey: restarting cassandra on aqs100[123] for jvm upgrades
10:26 elukey: restarting statsv on hafnium (process stuck after kafka brokers restart)
10:25 elukey: kafka restarted on kafka200[12] for jvm upgrades
09:34 jynus: removing leftovers of unmaintained skrillex tool from mira and tin
09:17 elukey: restarting statsv on hafnium (blocked due to analytics kafka brokers restarts)
09:01 elukey: upgrading httpd to 2.4.10-10+deb8u6+wmf2 on mw126[234]
08:50 elukey: uploaded apache2 2.4.10-10+deb8u6+wmf2 to reprepro
08:18 elukey: restarting kafka on the kafka analytics cluster for jvm upgrades
07:38 elukey: increaseing traffic weight to 30 for mw1261 (current: 5)
02:56 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Aug 11 02:56:27 UTC 2016 (duration 6m 54s)
02:49 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.14) (duration: 10m 24s)
02:27 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.13) (duration: 11m 12s)
00:41 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.14/maintenance/purgeChangedFiles.php: 91baa668219d8a77b1e300c257e969b04b4b1e3d (duration: 00m 47s)
00:39 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.13/maintenance/purgeChangedFiles.php: 4fe4f803541c4b21c5a118eab426d78d6c0c607b (duration: 00m 50s)
00:10 logmsgbot: reedy@tin Synchronized wikiversions.json: noop for mira (duration: 00m 49s)

2016-08-10

23:59 AaronSchulz: Running purgeChangedFiles.php on all wikis on a terbium screen (T142638)
23:46 logmsgbot: reedy@tin rebuilt wikiversions.php and synchronized wikiversions files: Reinstate .14 as T142638 is fixed
23:41 chasemp: stop nodepool & cleaning out nodepool instances for clean start on project ACL src group removal
23:44 mutante: mw2086 - removing node from cluster failed - backend error, request requires authentication
23:40 logmsgbot: reedy@tin Synchronized php-1.28.0-wmf.14/extensions/VipsScaler: Remove old broken config causing T142638 (duration: 00m 50s)
23:39 yuvipanda: restarted rabbitmq on labcontrol1001
23:22 mutante: connected to mw2086.mgmt (which icinga said was down since a couple hours). i saw it booting up..it came back but i did not powercyle or reboot, just view console
23:17 logmsgbot: reedy@tin rebuilt wikiversions.php and synchronized wikiversions files: Revert to .13 to attempt to fix T142638
22:36 andrewbogott: restarting rabbitmq-server on labvirt1001
21:42 mutante: added tbayer(HaeB) to wmf LDAP group
21:30 eileen: updated civicrm from 024a7ef2badb02743659ffed9856571d96604fd2 to cad2c59081e53b877f51be4c853f307f3c4d4949
20:21 Amir1: deploygin 7aad8e9 for ores in all nodes
20:18 Amir1: deploygin 7aad8e9 for ores in canary (scb2001.codfw.wmnet)
20:17 logmsgbot: mattflaschen@tin Synchronized wmf-config/CommonSettings-labs.php: Re-enable thank-you-edit (milestone notifications for 1st, 10th, 100th, etc. edits) on Beta Cluster (duration: 02m 45s)
20:11 subbu: finished deploying parsoid sha 4de49e26
20:07 subbu: finished syncing code; restarted parsoid on wtp1001 as canary
20:05 subbu: starting parsoid deploy
19:34 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: (no message)
19:31 twentyafterfour: sync-wikiversions failed for mw2086.codfw.wmnet
19:30 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: (no message)
17:15 logmsgbot: otto@palladium conftool action : set/pooled=yes; selector: kafka1002.eqiad.wmnet
17:14 logmsgbot: otto@palladium conftool action : set/pooled=no; selector: kafka1002.eqiad.wmnet
17:14 logmsgbot: otto@palladium conftool action : set/pooled=yes; selector: kafka1001.eqiad.wmnet
17:12 logmsgbot: otto@palladium conftool action : set/pooled=no; selector: kafka1001.eqiad.wmnet
17:12 ottomata: restarting eventbus in eqiad to apply error_output change
16:13 logmsgbot: hoo@tin Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on knwiki (T142468) (duration: 02m 55s)
16:08 logmsgbot: hoo@tin Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on cywiki (T140725) (duration: 02m 49s)
15:50 _joe_: restarting parsoid for T137878
15:41 andrewbogott: rebooting labvirt1014 to 3.16.0-77-generic for testing secgroup issues
15:30 elukey: correct version installed on mw1261 is 2.4.10-10+deb8u6+wmf2
15:22 thcipriani: mw2086 ssh from tin as mwdeploy user failing. Will need to run 'scap pull' when it comes back online
15:20 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Promote new language switcher to all wikis (T129505) (duration: 01m 12s)
15:14 logmsgbot: thcipriani@tin Synchronized dblists/visualeditor-nondefault.dblist: SWAT: Enable VisualEditor by default for logged-in users on Arabic-script Wikipedias (T93387) PART II]] (duration: 01m 40s)
15:11 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor by default for logged-in users on Arabic-script Wikipedias (T93387) PART I]] (duration: 02m 49s)
14:40 elukey: depooling mw1261 to install/test apache2_2.4.10-10+deb8u6+wmf1_amd64.deb (T73487). After basic checks the host will get back into service with weight 5.
13:54 moritzm: depooling image scaler mw1298 for some local tests with huge SVGs
13:48 logmsgbot: otto@palladium conftool action : set/pooled=yes; selector: kafka1002.eqiad.wmnet
13:43 logmsgbot: otto@palladium conftool action : set/pooled=no; selector: kafka1002.eqiad.wmnet
13:42 logmsgbot: otto@palladium conftool action : set/pooled=yes; selector: kafka1001.eqiad.wmnet
13:38 logmsgbot: otto@palladium conftool action : set/pooled=no; selector: kafka1001.eqiad.wmnet
13:38 ottomata: deploying eventbus service to kafka100[12], depooling, deploying, and repooling each one at at time
12:57 moritzm: reboot rdb2001-rdb2004 for updates to Linux 4.4
12:19 akosiaris: T140443 uploaded to apt.wikimedia.org trusty-wikimedia: php-wikidiff2_1.4.1
11:29 akosiaris: T140443 uploaded to apt.wikimedia.org trusty-wikimedia: php-wikidiff2_1.4.0
09:55 moritzm: rebooting dubnium for kernel update to 4.4
09:44 moritzm: rebooting pollux for kernel update to 4.4
09:41 _joe_: uploaded puppetdb deb packages for jessie (T142363)
09:09 moritzm: rebooting hafnium for kernel update to 4.4
03:22 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Aug 10 03:22:29 UTC 2016 (duration 7m 26s)
03:15 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.14) (duration: 18m 02s)
02:40 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.13) (duration: 17m 50s)
00:42 logmsgbot: reedy@tin Synchronized php-1.28.0-wmf.14/extensions/timeline/Timeline.body.php: (no message) (duration: 00m 48s)
00:41 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.13/extensions/timeline: (no message) (duration: 00m 47s)
00:33 logmsgbot: maxsem@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/303303/ (duration: 00m 47s)
00:28 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.13/extensions/timeline: https://gerrit.wikimedia.org/r/#/c/303952/ (duration: 00m 50s)
00:25 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.14/extensions/timeline: https://gerrit.wikimedia.org/r/#/c/303950/ (duration: 00m 47s)
00:17 eileen: updated Civicrm from af0048c007ddf9e74c6bd15e50c73c3bd0942492 to 024a7ef2badb02743659ffed9856571d96604fd2
00:11 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.13/extensions/timeline: https://gerrit.wikimedia.org/r/#/c/303947/ (duration: 00m 48s)

2016-08-09

23:44 logmsgbot: maxsem@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/303940/ (duration: 00m 49s)
23:22 logmsgbot: maxsem@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/303936/ (duration: 00m 49s)
23:15 Amir1: ladsgroup@scb2002:~$ sudo service celery-ores-worker restart
23:13 Amir1: ladsgroup@scb2001:~$ sudo service celery-ores-worker restart
23:11 Amir1: ladsgroup@scb1002:~$ sudo service celery-ores-worker restart
23:11 Amir1: sudo service celery-ores-worker restart in scb1001
22:49 paravoid: re-enabling cr2-eqiad:xe-5/2/3 (link to cr2-codfw); vendor reported it as fixed
22:45 ejegg: updated civicrm from d9a765903ac682c0fbd329ced59f5ba3953970c9 to af0048c007ddf9e74c6bd15e50c73c3bd0942492
22:02 ejegg: updated SmashPig from 26a475bf5ae03d88ebc4c2fe9707d562d8e3afe3 to 5c180de6e424be10f9d61052c2c8dca7e0e825af
21:09 logmsgbot: legoktm@tin Synchronized wmf-config/CommonSettings.php: Configure 'sourceUrl' for ExtensionDistributor (duration: 00m 50s)
21:02 twentyafterfour: cleaned up stale branch php-1.28.0-wmf.7
20:40 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings.php: Fix wgAddGroups/wgRemoveGroups for mediawikiwiki - T142492 (duration: 00m 50s)
20:27 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.13/extensions/FlaggedRevs: 3acc2bd85544072b7fd55abd2829f2bdde7aeef8 (duration: 00m 53s)
20:25 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.28.0-wmf.14
20:19 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.14/extensions/FlaggedRevs: 52f54661a84f2066bc4c3a13e603ff8c6e5db357 (duration: 00m 55s)
20:17 mutante: root@tin:/srv/mediawiki-staging# find . -uid 0 -exec chown mwdeploy:wikidev {} \;
20:08 bblack: cr[12]-eqiad: added term relforge for instances->relforge100[12] in labs-support
20:05 logmsgbot: twentyafterfour@tin Finished scap: testwiki to 1.28.0-wmf.14 refs T139217 (duration: 49m 45s)
20:02 ejegg: rolled back SmashPig to 26a475bf5ae03d88ebc4c2fe9707d562d8e3afe3
20:01 awight: update paymentswiki from 33de2ce5d7b040d733141fc12a74f70a81c09925 to 6b1f3c278308f831a149c986d2269342f94e2a16
19:58 ejegg: enabled fundraising pending queue consumer
19:57 ejegg: updated SmashPig from 26a475bf5ae03d88ebc4c2fe9707d562d8e3afe3 to 081493d846ad3c1b063f794ca2303576157b27f2
19:40 andrewbogott: restarting rabbitmq-server on labcontrol1001
19:38 awight|eat: update paymentswiki from 4e6a68f9d0b98d9037f81b0c4d312831d7fe66bb to 33de2ce5d7b040d733141fc12a74f70a81c09925
19:15 logmsgbot: twentyafterfour@tin Started scap: testwiki to 1.28.0-wmf.14 refs T139217
19:05 yurik: deployed/restarted kartotherian https://gerrit.wikimedia.org/r/#/c/303841/ and tilerator https://gerrit.wikimedia.org/r/#/c/303840/
18:56 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.13/extensions/WikimediaEvents/modules/ext.wikimediaEvents.deprecate.js: Log RL splitRequest (duration: 02m 08s)
18:54 ejegg: disable pending-new queue consumer
18:21 awight: update paymentswiki from b737b60c87da82543ab812ece4611c68af01307f to 4e6a68f9d0b98d9037f81b0c4d312831d7fe66bb
18:10 ejegg: rolled back SmashPig to 26a475bf5ae03d88ebc4c2fe9707d562d8e3afe3
18:05 ejegg: updated SmashPig from 26a475bf5ae03d88ebc4c2fe9707d562d8e3afe3 to 081493d846ad3c1b063f794ca2303576157b27f2
17:51 ottomata: restarting kafka broker on kafka1013 to test eventlogging leader changes
17:13 subbu: finished deploying parsoid sha a577d80e
17:03 subbu: synced new code; restarted parsoid on wtp1001 as a canary
17:01 subbu: starting parsoid deploy
16:41 mobrovac: restbase deploy end of b800d343
16:21 mobrovac: restbase deploy start of b800d343
15:59 elukey: switching restbase/cassandra user on aqs100[123] to aqs (T142073) - https://gerrit.wikimedia.org/r/303798 will be applied to one node at the time with depool/pool
15:18 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.13/extensions/CirrusSearch/includes/Search/Result.php: SWAT: Use createFragmentTarget instead of setFragment (T142297) (duration: 00m 54s)
14:37 akosiaris: T135176 pool wtp2002
14:36 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2002.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
14:00 paravoid: setting cr2-eqiad:xe-5/2/3 (link to cr2-codfw) to disable; flapping
13:34 akosiaris: all of wtp2001-wtp2020 except wtp2002 have been pooled. T135176
13:33 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2001.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
13:07 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-fra-cat_1.1.0~r64309-1+wmf1
12:26 moritzm: depooling image scalers mw2086-mw2089 for reimaging with jessie
12:10 moritzm: rolling restart of hhvm on remaining eqiad mw servers to pick up curl security updates
12:05 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-fra_1.0.0~r65786-1+wmf1
11:56 moritzm: installing fontconfig security updates on jessie systems
09:20 moritzm: rolling restart of hhvm in on eqiad canary trusty app servers to pick up curl security updates
08:05 moritzm: rolling restart of hhvm in codfw to pick up curl security updates
07:42 moritzm: installing php5 security updates on remaining four precise systems
07:25 moritzm: installing curl security updates on Ubuntu systems
06:24 moritzm: installing chromium security updates on osmium
06:13 kart_: Updated cxserver to d3c7d64 (T142340)
05:32 _joe_: removing stale nodes from puppet
02:26 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Aug 9 02:26:53 UTC 2016 (duration 5m 51s)
02:21 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.13) (duration: 09m 24s)
01:01 logmsgbot: krinkle@tin Synchronized wmf-config/InitialiseSettings.php: coding style (duration: 00m 56s)
00:23 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Give autopatrol to translationadmin too (duration: 00m 47s)
00:17 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Rename autoreview group to autopatrolled on mw.org (duration: 00m 48s)
00:12 logmsgbot: catrope@tin Synchronized wmf-config: Various config changes for SWAT (duration: 00m 53s)

2016-08-08

23:53 mutante: git pull on dbtree/db1152, there were previous changes that did not get deployed
23:53 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Fix typo in Russian language switcher config (T129505) (duration: 00m 48s)
23:50 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.13/includes/specials/SpecialNewimages.php: Restore the newimagestext message (T142191) (duration: 00m 47s)
23:43 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Set import sources on enwikibooks (T142333) (duration: 00m 49s)
23:42 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.13/extensions/Echo/modules/nojs/mw.echo.badge.monobook.less: Hack around IE bug breaking badge alignment in Monobook (T142053) (duration: 00m 50s)
23:34 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Enable OAuth notifications in config (no-op until Wednesday) (T61172) (T62528) (duration: 00m 50s)
23:19 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Promote language switcher to top of page on ruwiki (T138961) (duration: 00m 59s)
23:11 bblack: openssl-1.0.2h-1~wmf4 -> caches
23:05 bblack: uploaded to carbon jessie-wikimedia: openssl-1.0.2h-1~wmf4 ( https://gerrit.wikimedia.org/r/#/c/303700/ )
21:57 dapatrick: Deployed patch for T130384 to wmf.13
21:55 hoo: Updated Wikidata's property suggester with data from today's json dump and removed the external identifiers as a workaround for T132839
21:53 ejegg: enabled globalcollect recurring job
21:30 ejegg: updated civicrm from 2d68638471aded73d05a796b05cab11809e31c56 to d9a765903ac682c0fbd329ced59f5ba3953970c9
20:21 subbu: aborting parsoid deploy for today (needs akosiaris to take a look at the deployment setup)
20:16 logmsgbot: aaron@tin Synchronized wmf-config/db-eqiad.php: Use pt-heartbeat and GTIDs on remaning sections (duration: 01m 00s)
20:14 logmsgbot: aaron@tin Synchronized wmf-config/db-codfw.php: Use pt-heartbeat and GTIDs on remaning sections (duration: 00m 53s)
20:02 subbu: starting parsoid deploy
19:50 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.13/includes/db/DatabaseMysqlBase.php: 267c62a530530e (duration: 00m 48s)
19:20 logmsgbot: aaron@tin Synchronized wmf-config/db-eqiad.php: Enable MASTER_GTID_WAIT() on s6 (duration: 00m 48s)
19:19 logmsgbot: aaron@tin Synchronized wmf-config/db-codfw.php: Enable MASTER_GTID_WAIT() on s6 (duration: 00m 48s)
19:18 ottomata: restarting kafka broker on kafka1022 to test more eventlogging rebalances
19:15 logmsgbot: aaron@tin Synchronized tests: (no message) (duration: 00m 50s)
19:13 logmsgbot: aaron@tin Synchronized wmf-config/db-eqiad.php: Switched to pt-heartbeat lag detection on s6 (duration: 00m 53s)
19:12 logmsgbot: aaron@tin Synchronized wmf-config/db-codfw.php: Switched to pt-heartbeat lag detection on s6 (duration: 00m 51s)
19:11 ottomata: restarting kafka broker on 1013 to test more eventlogging rebalances
19:00 ottomata: restarting kafka broker on 1013 to test more eventlogging rebalances
18:48 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.13/extensions/WikimediaEvents/modules/ext.wikimediaEvents.rlfeature.js: T141344 Track JSON support (duration: 00m 47s)
18:30 ottomata: restarting kafka broker on kafka1013 to test eventlogging leader rebalances
17:09 gehel: updating WDQS to latest and service restart
17:03 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2001.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:56 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2020.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:56 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2019.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:56 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2018.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:56 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2017.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:56 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2016.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:56 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2015.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:55 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2014.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:55 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2013.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:55 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2012.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:55 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2011.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:55 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2010.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:55 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2009.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:55 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2008.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:55 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2007.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:55 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2006.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:55 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2005.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:55 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2004.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:55 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2003.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:53 urandom: T140008: Starting major compaction, restbase2001-a.codfw.wmnet
15:23 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add extendedconfirmed user group for fawiki (T140839) (duration: 00m 51s)
15:13 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Simplify the VE RB URL config some more, now that we no longer use wgServerName (duration: 00m 48s)
15:13 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-pt-ca_0.8.2+svn~57507-1+wmf1
15:12 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-fr-es_0.9.2~r61322-1+wmf1
15:08 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: MoodBar: Disable on all wikis except nlwiki (T131340) (duration: 01m 08s)
12:52 moritzm: upgrading firejail on scb1002 (along with service restarts except changeprop)
12:19 moritzm: upgrading firejail on scb1001 (along with service restarts except changeprop)
11:45 moritzm: upgrading firejail on scb* in codfw
11:17 _joe_: creating nihal.codfw.wmnet as a VM for puppetdb, T142365
11:08 _joe_: creating nitrogen.eqiad.wmnet as a VM for puppetdb, T142365
10:10 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.13/extensions/Cite/Cite_body.php: Cite::referencesFormatEntry: Avoid Undefined index: key - T132583 (duration: 00m 49s)
10:05 akosiaris: T135176 depool wtp2001-wtp2020
09:58 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2020.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:58 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2019.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:58 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2018.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:58 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2017.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:58 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2016.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:58 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2015.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:58 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2014.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:58 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2013.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:58 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2012.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:58 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2011.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:58 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2010.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:58 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2009.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:58 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2008.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:58 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2007.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:57 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2006.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:57 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2005.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:57 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2004.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:57 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2003.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:57 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2002.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:52 akosiaris: uploaded to apt.wikimedia.org precise-wikimedia: php5_5.3.10-1ubuntu3.24+wmf1
09:51 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: php5_5.3.10-1ubuntu3.24+wmf1
09:48 logmsgbot: legoktm@tin Synchronized wmf-config/CommonSettings.php: Fix missed $wmg -> $wg for $wgRestbaseServer (duration: 00m 49s)
09:41 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.13/includes/revisiondelete/: Fix inconsistent RevDelFileItem visibilities - T142228 (duration: 00m 50s)
09:34 elukey: re-imaging aqs1005 to migrate Cassandra partitions to RAID10 (T142075)
09:13 moritzm: restarting elasticsearch on logstash100[56] to pick up java security updates
07:16 moritzm: installing php5 security updates
02:26 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Aug 8 02:26:21 UTC 2016 (duration 5m 46s)
02:20 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.13) (duration: 09m 21s)

2016-08-07

23:39 logmsgbot: reedy@tin Synchronized wmf-config: last 2 to wfLoadExtension (duration: 00m 59s)
23:12 logmsgbot: reedy@tin Synchronized wmf-config: Handful more extensions to wfLoadExtension (duration: 00m 49s)
22:57 logmsgbot: reedy@tin Synchronized wmf-config: RestbaseUpdateJobs to wfLoadExtension (duration: 00m 51s)
22:55 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: Image Area to 100MP (duration: 00m 48s)
22:44 logmsgbot: reedy@tin Synchronized wmf-config/extension-list: 2 more to extension.json (duration: 00m 48s)
22:35 logmsgbot: reedy@tin Synchronized docroot/noc/conf: Cleanup! (duration: 00m 50s)
22:30 logmsgbot: reedy@tin Synchronized docroot: trusted-xff symlink updates (duration: 00m 50s)
22:29 logmsgbot: reedy@tin Synchronized wmf-config/: Swap trusted-xff from cdb to php (duration: 00m 51s)
20:49 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.13/extensions/GlobalBlocking/extension.json: Adding globalblock-exempt grant for OAuth - T142306 (duration: 00m 57s)
16:22 cwd|afk: disabled globalcollect recurring donations
16:13 akosiaris: restarted apache2 on palladium for full depool to take place
12:47 hashar: root cause of CI outage is T126552
12:41 hashar: CI fully back. Root cause was Jenkins that could not properly create slaves config due to : Could not create rootDir /var/lib/jenkins/config-history/xxxx . Deleting via find /var/lib/jenkins/config-history/nodes/ -path '*_deleted_*' -delete
12:12 hashar: CI stuck spawning instances via Nodepool apparently due to : Quota exceeded for instances: Requested 1, but already used 10 of 10 instances (HTTP 403) --- Though there is only 8 instances ...
12:10 hashar: CI stuck spawning instances via Nodepool apparently due to : Quota exceeded for instances: Requested 1, but already used 10 of 10 instances (HTTP 403) --- Though there is only 8 instances ...
02:24 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Aug 7 02:24:55 UTC 2016 (duration 5m 51s)
02:19 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.13) (duration: 08m 55s)

2016-08-06

23:09 yuvipanda: cleaned and re-accepted salt-key for labvirt1014, minion back up now
22:49 yuvipanda: run 'service mariadb start' on labsdb1003, puppet run didn't do anything
19:43 andrewbogott: rebooting labvirt1012 for a kernel downgrade
19:12 andrewbogott: rebooting labvirt1013 for kernel downgrade
09:36 akosiaris: revert back to old backed up bayes database on mendelevium.eqiad.wmnet (OTRS) to get bayes training working again
02:26 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Aug 6 02:26:00 UTC 2016 (duration 5m 48s)
02:20 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.13) (duration: 08m 46s)
01:02 andrewbogott: re-imaging labvirt1014

2016-08-05

23:39 logmsgbot: tgr@tin Synchronized php-1.28.0-wmf.13/includes/api/ApiLogin.php: temporarily re-add dropped API feature to unbreak Pywikibot T142155 (duration: 00m 48s)
22:37 andrewbogott: rebooting labvirt1014 as part of a protracted iptables/nova-compute investigation
21:03 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: Add transitionary timeline config primarily for beta (duration: 00m 57s)
18:26 andrewbogott: restarting rabbitmq-server on labcontrol1001
17:27 ejegg: rolled back SmashPig to 26a475bf5ae03d88ebc4c2fe9707d562d8e3afe3
17:25 ejegg: updated SmashPig from 26a475bf5ae03d88ebc4c2fe9707d562d8e3afe3 to 2e8a2f4c92840bd999a8742211e0a65d484fde00
16:03 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-spa-arg_0.4.0~r64399-1+wmf1
16:03 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-arg-cat_0.1.0~r64925-1+wmf1
15:16 akosiaris: T135176 pool wtp1019-wtp1024
15:13 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1024.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
15:13 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1023.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
15:13 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1022.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
15:13 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1021.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
15:13 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1020.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
15:13 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1019.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
13:56 akosiaris: strontium has issues, see https://phabricator.wikimedia.org/T142187
13:53 moritzm: uploaded gerrit 2.12.2-wmf2 for jessie-wikipedia to apt.wikimedia.org
13:49 ostriches: gerrit: quick restart to pick up apache and java updates
13:22 paravoid: started spamassassin/exim4 on mendelevium
12:19 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Pool db1079 as an api server; reduce main load (duration: 00m 49s)
12:14 akosiaris: just encountered https://wikitech.wikimedia.org/wiki/OTRS#SpamAssassin_stops_reporting_Bayes_results. Recovered the db with sa-learn --sync and then force spam/ham runs via the web interface
11:06 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Document T142135 and apply workaround (duration: 00m 52s)
11:03 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Document T142135 (duration: 00m 56s)
10:54 gehel: killing elasticsearch on logstash1004 (stuck during shutdown)
10:24 akosiaris: T135176 depool wtp1019-wtp1024
10:24 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1024.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
10:24 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1023.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
10:24 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1022.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
10:24 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1021.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
10:23 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1020.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
10:23 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1019.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
10:23 akosiaris: T135176 pool wtp1013-wtp1018
10:23 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1018.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
10:23 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1017.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
10:23 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1016.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
10:23 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1015.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
10:23 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1014.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
10:23 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1013.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
09:26 dcausse: creating tcywiki indices on elastic@codfw
09:23 logmsgbot: elukey@palladium conftool action : set/pooled=yes:weight=30; selector: mw1261.eqiad.wmnet
09:17 moritzm: installing openjdk security updates on logstash cluster, rolling restart to pick up new JRE
09:12 elukey: upgrading httpd on mw126[78] to 2.4.10-10+deb8u4+wmf3 (T73487)
07:23 moritzm: restarting hhvm on mediawiki canaries to pick up curl security update
07:18 akosiaris: T135176 depool wtp1013-wtp1018
07:15 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1018.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
07:15 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1017.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
07:15 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1016.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
07:15 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1015.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
07:15 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1014.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
07:14 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1013.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
04:30 moritzm: installing curl security updates on jessie systems
04:07 yuvipanda: restarted nova-network on labnet1002 for T142165
02:27 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Aug 5 02:27:30 UTC 2016 (duration 6m 14s)
02:21 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.13) (duration: 08m 43s)

2016-08-04

23:37 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.13/extensions/Echo: https://gerrit.wikimedia.org/r/#/c/303095/ https://gerrit.wikimedia.org/r/#/c/303099/ (duration: 00m 54s)
23:37 Pchelolo: restart restbase to apply gerrit:300214 config change
23:34 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.13/extensions/MobileFrontend: https://gerrit.wikimedia.org/r/#/c/303095/ https://gerrit.wikimedia.org/r/#/c/303099/ (duration: 00m 52s)
23:33 awight: update civicrm from 9a971ff6d74ae8e14c1c9f854155d9829e6a0278 to 2d68638471aded73d05a796b05cab11809e31c56
23:29 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.13/extensions/MobileFrontend: https://gerrit.wikimedia.org/r/#/c/302992/ (duration: 00m 56s)
23:17 logmsgbot: maxsem@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/301746/2 (duration: 00m 53s)
23:15 Dereckson: mwscript extensions/WikimediaMaintenance/filebackend/setZoneAccess.php tcywiki --backend=local-multiwrite
23:12 logmsgbot: dereckson@tin Synchronized wmf-config/interwiki.php: Interwiki map update for tcy.wikipedia.org (Gerrit:303098, T140898) (duration: 01m 05s)
22:43 Dereckson: Run mwscript extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki=tcywiki --baseName=tcywiki
22:31 ejegg: updated SmashPig from b7f5e449aa62cc5518dea580de96b8ed7a2489d0 to 26a475bf5ae03d88ebc4c2fe9707d562d8e3afe3
22:16 Dereckson: Synchronized static/images/project-logos: Logos for tcy.wikipedia.org (T140898) (duration: 00m 48s)
22:12 Dereckson: Synced dblists, wikiversions, langlist, wmf-config/InitialiseSettings.php for tcy.wikipedia
22:07 Dereckson: mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki tcy wikipedia tcywiki
21:50 urandom: T140825,T140869: Rolling restart of Cassandra instances, eqiad Rack `d'
21:48 urandom: T140825,T140869: Rolling restart of Cassandra instances, eqiad Rack `b', complete
21:26 urandom: T140825,T140869: Rolling restart of Cassandra instances, eqiad Rack `b'
21:17 ejegg: updated SmashPig from e6aa6fe6fdcaab8e961a8b0668cc742d4c443c46 to b7f5e449aa62cc5518dea580de96b8ed7a2489d0
21:14 urandom: T140825,T140869: Rolling restart of codfw Cassandra instances complete
20:40 akosiaris: restart apache on palladium. Managed to get it into a deadlock. OFC puppet spam will flood the channel
20:13 urandom: T140825,T140869: Performing rolling restart of codfw Cassandra instances
20:10 urandom: T140825,T140869: Cassandra instance restarts complete: restbase1011.eqiad.wmnet
20:03 urandom: T140825,T140869: Performing Cassandra instance rolling restart of restbase1011.eqiad.wmnet
19:59 logmsgbot: reedy@tin Synchronized wmf-config/extension-list: More to extension.json in extension-list (duration: 00m 52s)
19:58 ottomata: testing some eventbus kafka failure scenarios in codfw with test.event. (short icinga downtime has been scheduled)
19:51 awight: paymentswiki config fix
19:47 logmsgbot: otto@palladium conftool action : set/pooled=yes; selector: kafka2002.codfw.wmnet
19:46 awight: enabling paymentswiki queue mirroring
19:46 akosiaris: restart puppetmaster on palladium to activate loadfactor change. puppet related icinga spam will ensue*
19:46 akosiaris: restart puppetmaster on palladium to activate loadfactor change. puppet related icinga spam will ensure
19:45 logmsgbot: otto@palladium conftool action : set/pooled=no; selector: kafka2002.codfw.wmnet
19:43 logmsgbot: otto@palladium conftool action : set/pooled=yes; selector: kafka2001.codfw.wmnet
19:42 urandom: T140825,T140869: Restarting Cassandra, restbase1010-c.eqiad.wmnet
19:40 logmsgbot: otto@palladium conftool action : set/pooled=no; selector: kafka2001.codfw.wmnet
19:40 urandom: T140825,T140869: Restarting Cassandra, restbase1010-b.eqiad.wmnet
19:39 ottomata: deploying eventlogging-service-eventbus to codfw hosts, depooling and pooling
19:37 urandom: T140825,T140869: Restarting Cassandra, restbase1010-a.eqiad.wmnet
19:06 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.13
18:49 logmsgbot: thcipriani@tin Synchronized README: test logstash host (duration: 00m 51s)
18:36 cwd: updated payments from 3a724bfb1a3e20e17b5886dae0ba7572020abd6b to b737b60c87da82543ab812ece4611c68af01307f
18:12 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Pool db1074 as backup rc node in case db1036 lags (duration: 00m 25s)
18:12 awight: update paymentswiki config to 71cc55194f4465600ce4da0ea9f7dfaefdda5479
17:36 elukey: added the analytics-deploy key to the Keyholder for the Analytics Refinery scap3 migration (also updated https://wikitech.wikimedia.org/wiki/Keyholder)
17:18 ostriches: gerrit: Restarting really quick, trying alternative mysql library.
17:17 urandom: T140825,T140869: Restarting Cassandra, restbase1007-c.eqiad.wmnet
17:14 urandom: T140825,T140869: Restarting Cassandra, restbase1007-b.eqiad.wmnet
17:08 urandom: T140825,T140869: Restarting Cassandra, restbase1007-a.eqiad.wmnet
17:07 elukey: restarting keyholder-proxy on tin to let the new analytics key to be picked up
15:08 logmsgbot: thcipriani@tin Synchronized portals: SWAT: Bumping portals to master. (duration: 00m 37s)
15:07 logmsgbot: thcipriani@tin Synchronized portals/prod/wikipedia.org/assets: SWAT: Bumping portals to master. (duration: 00m 36s)
14:11 godog: manually config s1 dbs to scrape on prometheus2001 as a test
13:51 akosiaris: delete+accept ganeti1004 salt minion key
13:43 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1012.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
13:43 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1011.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
13:43 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1010.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
13:43 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1009.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
13:43 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1008.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
10:59 jynus: applying prometheus grants to pc* hosts
10:33 elukey: upgrading httpd on mw126[56] to 2.4.10-10+deb8u4+wmf3 (T73487)
07:31 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1012.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
07:31 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1011.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
07:31 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1010.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
07:31 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1009.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
07:31 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1008.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
07:30 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1009.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
07:30 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1008.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
02:43 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Aug 4 02:43:16 UTC 2016 (duration 6m 23s)
02:36 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.13) (duration: 05m 52s)
02:24 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.12) (duration: 08m 39s)
00:19 awight: revert paymentswiki config to 70b2ff90d9c8f19716f2e9c07a8dc8cfa17991ca
00:17 mutante: gerrit is restarting for config change 301822 (set default project owners). gerrit apache is restarting for 301829 (redirect /r) 301895 (logs for 10 days) and 301824 (renaming logs)
00:16 awight: update payments-wiki config to 793389ac8fa34cfc6a4ba1df67f2f9fac1ca02fe

2016-08-03

23:41 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.13/extensions/Echo/modules/nojs/: Adjust notification badges for monobook (T141923). Prevent IE from rendering the badge SVGs ridiculously big (T142042). (duration: 00m 29s)
23:38 kaldari: ran "mwscript maintenance/updateCollation.php --wiki=testwiki --force"
23:35 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Test numeric collation on testwiki (T141433) (duration: 00m 26s)
23:24 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.13/extensions/MobileFrontend/includes/skins/SkinMinervaBeta.php: Do not output the 'switch language' action on Main Page in beta (T142016) (duration: 00m 28s)
23:19 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Enable RevisionSlider on plwiki (T141974) (duration: 00m 26s)
23:06 logmsgbot: aude@tin rebuilt wikiversions.php and synchronized wikiversions files: Wikidata back to wmf.13
23:04 logmsgbot: aude@tin Synchronized php-1.28.0-wmf.13/extensions/Wikidata: Update PropertySuggester (duration: 02m 04s)
22:46 mdholloway: mobileapps deployed e7488f6
22:44 mdholloway: starting mobileapps deployment
21:54 bblack: upgrading nginx on cache_text + cache_upload
21:46 bblack: upgrading nginx on cache_misc + cache_maps
21:41 bblack: nginx-1.11.3-1+wmf1 uploaded to carbon jessie-wikimedia
21:02 bearND: deployed mobileapps e48b6a8
21:00 bearND: starting mobileapps deploy
20:05 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: wikidata to 1.28.0-wmf.12
19:38 mutante: ganeti1004, start salt-minion
19:19 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.13
18:59 logmsgbot: aude@tin Synchronized php-1.28.0-wmf.13/extensions/Wikidata: Update PropertySuggester (duration: 02m 02s)
18:53 akosiaris: T135176 pool wtp100[34567] with weight=15
18:53 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1007.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
18:53 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1006.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
18:53 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1005.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
18:53 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1004.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
18:53 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1003.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
18:08 mutante: restarted morebots-production
17:35 mobrovac: citoid deploying 0b9f59fe0
17:01 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1007.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
17:01 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1005.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
17:01 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1004.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
17:01 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1003.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
16:22 akosiaris: reboot wtp1006
16:17 bblack: upgrading openssl on cache_text, cache_upload
16:01 akosiaris: T135176 pool wtp100[3457] with weight=15. wtp1006 does not look so good
16:00 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1007.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
16:00 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1005.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
15:59 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1004.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
15:59 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1003.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
15:59 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1007.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
15:59 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1005.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
15:59 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1004.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
15:59 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1003.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
15:35 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-urd_0.1.0~r61311-1+wmf1
15:35 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-tat_0.1.0~r60887-1+wmf1
15:35 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-swe_0.7.0~r69513-1+wmf1
15:35 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-spa_0.1.0~r65494-1+wmf1
15:35 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-pt-gl_0.9.2~r60358-1
15:35 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-oc-es_1.0.6~r60161-1+wmf1
15:35 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-oc-ca_1.0.6~r60158-1
15:34 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-nno_0.9.0~r69513-2+wmf1
15:34 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-mlt-ara_0.2.0~r62623-1+wmf1
15:34 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-mk-en_0.1.1~r57554-1+wmf1
15:34 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-mk-bg_0.2.0~r49489-1+wmf1
15:34 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-is-sv_0.1.0~r56030-1+wmf1
15:34 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-id-ms_0.1.1+svn~57870-1+wmf1
15:34 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-hbs-mkd_0.1.0~r57554-1+wmf1
15:34 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-eu-es_0.3.3~r56159-1+wmf1
15:34 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-eu-en_0.3.1~r60155-1+wmf1
15:34 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-es-pt_1.1.5+svn~57507-1+wmf1
15:34 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-es-it_0.1.0~r51165-1+wmf1
15:34 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-es-gl_1.0.8~r57542-1+wmf1
15:34 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-es-ca_1.2.1+svn~57448-1+wmf1
15:34 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-es-ast_1.1.0~r60158-1+wmf1
15:34 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-eo-fr_0.9.0~r57551-1
15:34 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-eo-es_0.9.1~r60655-1+wmf1
15:33 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-eo-en_1.0.0~r63833-1+wmf1
15:33 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-eo-ca_0.9.1~r60655-1+wmf1
15:33 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-en-gl_0.5.2~r57551-1+wmf1
15:33 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-cy-en_0.1.1~r57554-3+wmf1
15:33 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-cat_1.0.0~r65787-1+wmf1
15:33 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-ca-it_0.1.1~r57554-1+wmf1
15:33 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-arg_0.1.2~r65494-1+wmf1
15:33 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-af-nl_0.2.0~r58256-1+wmf1
15:32 bblack: upgrading openssl on cache_maps + cache_misc
15:23 bblack: openssl-1.0.2h-1~wmf3 uploaded to carbon jessie-wikimedia ( https://gerrit.wikimedia.org/r/#/c/301920/ )
15:13 logmsgbot: addshore@tin Synchronized php-1.28.0-wmf.13/includes/Linker.php: SWAT: Debug Logging for Undefined index: width in Linker.php (duration: 00m 30s)
14:36 godog: reboot ms-be1022 following firmware upgrade T141756
14:17 mobrovac: restbase deploy end of ff1ee1e7
14:06 mobrovac: restbase deploy start of ff1ee1e7
13:16 godog: reboot ms-be1022 - T140597
12:31 akosiaris: T135176 depool wtp100[34567]
12:30 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1007.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
12:30 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1006.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
12:30 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1005.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
12:30 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1004.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
12:30 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1003.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
09:29 jynus: applying prometheus required grants to all databases T128185
08:43 godog: upload scap 3.2.2-1 to carbon T127762
08:32 jynus: restarting replication on db1018
08:31 elukey: upgrading httpd on mw126[34] to 2.4.10-10+deb8u4+wmf3 (T73487)
08:27 jynus: stopping replication to db1018 (s2-master-eqiad)
08:21 jynus: restarting replication on db1024
08:14 jynus: stopping replication to db1024 (depooled) to test replication alerts
08:12 jynus: restarting slave on db2017
08:07 jynus: stopping replication to s2-master-codfw (db2017) to test replication alerts
07:29 jynus: restarting pt-heartbeat-wikimedia on all database masters
05:38 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.13/extensions/MobileFrontend/: I195f67d061d (duration: 00m 29s)
05:37 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.13/includes/OutputPage.php: I195f67d061d (duration: 00m 30s)
05:36 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.13/resources/Resources.php: I195f67d061d (duration: 00m 30s)
05:36 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.13/autoload.php: I195f67d061d (duration: 00m 42s)
05:35 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.13/includes/resourceloader/: I195f67d061d (duration: 00m 38s)
03:09 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Aug 3 03:09:30 UTC 2016 (duration 6m 58s)
03:02 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.13) (duration: 14m 56s)
02:30 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.12) (duration: 08m 34s)
02:24 mutante: neon - restarted ferm service
02:21 logmsgbot: legoktm@tin Synchronized wmf-config/CommonSettings-labs.php: labs-only, move from ores.wikimedia.org to ores-beta.wmflabs.org (duration: 00m 33s)
02:19 mutante: restarted log bot
01:17 awight: revert paymentswiki config change, new commit 4d0090f50d515047c5e4dfbe565cd04bdd31801d
01:15 awight: update paymentswiki config to f89b8d372753fb2f37c50c062e8cca12c6351e19
01:00 awight: update paymentswiki config to ac5f7a848450f63b4438de926088acd4feacea83
00:27 awight: update paymentswiki config to 3a724bfb1a3e20e17b5886dae0ba7572020abd6b
00:25 logmsgbot: dereckson@tin Synchronized wmf-config: Add $wmgEchoMentionStatusNotifications and enable it in beta labs (no-op, sync labs files) (duration: 00m 28s)
00:20 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Add $wmgEchoMentionStatusNotifications and enable it in beta labs (no-op in prod, T135717, T139623) (duration: 00m 26s)
00:19 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Add $wmgEchoMentionStatusNotifications and enable it in beta labs (no-op in prod, T135717, T139623) (duration: 00m 25s)
00:13 logmsgbot: legoktm@tin Synchronized wmf-config: De-deploy CustomData extension - T140847 (duration: 00m 28s)
00:02 logmsgbot: maxsem@tin Synchronized wmf-config: Labs only (duration: 00m 30s)

2016-08-02

23:59 awight: update paymentswiki config to 9b3ee23d54c481343164c50f172184bb7b99e871
23:51 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Reset ar.wikipedia content namespaces (T141906) (duration: 00m 26s)
23:50 awight: update paymentswiki config to 5429797ec902cd3a6fe098d22e12af7a578e80b8
23:50 awight: update paymentswiki config to
23:39 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Move abusefilter permissions to abusefilter.php for azbwiki (T141860, 2/2) (duration: 00m 27s)
23:38 logmsgbot: dereckson@tin Synchronized wmf-config/abusefilter.php: Move abusefilter permissions to abusefilter.php for azbwiki (T141860, 1/2) (duration: 00m 28s)
23:29 logmsgbot: dereckson@tin Synchronized wmf-config/abusefilter.php: Allow abuse filter editors group to edit tags on en.wikipedia (T141847) (duration: 00m 32s)
21:41 logmsgbot: krinkle@tin Synchronized docroot/noc: Update favicon.ico symlink (duration: 00m 34s)
21:20 mutante: gerrit is restarting to apply config changes: 301898 (warm cache, faster startup) 301894 (double size of conflicts cache) 302129 (avoid breaking full phabricator urls)
21:04 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.28.0-wmf.13
21:01 ejegg: updated civicrm from 4904c4aae3565b65d5f37ecb827ea26c930b72d6 to 9a971ff6d74ae8e14c1c9f854155d9829e6a0278
21:01 logmsgbot: thcipriani@tin Finished scap: testwiki to php-1.28.0-wmf.13 and rebuild l10n cache with Wikidata (duration: 24m 13s)
20:36 logmsgbot: thcipriani@tin Started scap: testwiki to php-1.28.0-wmf.13 and rebuild l10n cache with Wikidata
20:17 logmsgbot: thcipriani@tin Finished scap: testwiki to php-1.28.0-wmf.13 and rebuild l10n cache (duration: 45m 40s)
19:31 logmsgbot: thcipriani@tin Started scap: testwiki to php-1.28.0-wmf.13 and rebuild l10n cache
18:21 akosiaris: T135176 set weight for wtp100[12] to 15
18:20 logmsgbot: akosiaris@palladium conftool action : set/weight=15; selector: wtp1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
18:19 logmsgbot: akosiaris@palladium conftool action : set/weight=15; selector: wtp1001.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
17:01 thcipriani: starting branch-cut for 1.28.0-wmf.13
16:17 andrewbogott: disabling puppet on all lab* hosts for staged upgrade
15:47 mobrovac: restbase rolling restart for firejail upgrade to 0.9.40
15:45 moritzm: upgraded firejail on restbase* to 0.9.40.3
15:32 jynus: performing schema change on heartbeat.heartbeat on all core databases
15:29 logmsgbot: addshore@tin Synchronized php-1.28.0-wmf.12/extensions/RevisionSlider/modules/ext.RevisionSlider.init.js: SWAT: Track the load times of RevisionSlider (duration: 00m 26s)
15:25 logmsgbot: addshore@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove T107711 debug logging (duration: 00m 30s)
15:22 logmsgbot: addshore@tin Synchronized php-1.28.0-wmf.12/extensions/ContentTranslation/modules/tools/ext.cx.tools.link.js: SWAT: Fix: Target links has source link titles (duration: 00m 34s)
14:55 ejegg: updated fundraising tools from 28bc2da677caa795c58f906db76a1f8d612ac899 to b3ed7ab3deac94c4e465d3768109bca05b6f0a0c
14:29 moritzm: upgrading restbase2003 system to firejail 0.9.40.3
14:26 akosiaris: T135176 set weight for wtp100[12] to 8
14:24 ema: setting default_ttl=604800 on cache_upload varnish backends
14:24 moritzm: upgrading restbase staging systems to firejail 0.9.40.3
14:22 logmsgbot: akosiaris@palladium conftool action : set/weight=8; selector: wtp1001.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
14:22 logmsgbot: akosiaris@palladium conftool action : set/weight=8; selector: wtp1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
14:14 jynus: resizing online labsdb1006 postgres fs to increase available disk space
12:34 akosiaris: T135176 repool wtp100[12] with a weight of 1 instead of 15
12:33 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
12:33 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1001.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
12:33 logmsgbot: akosiaris@palladium conftool action : set/weight=1; selector: wtp1001.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
12:33 logmsgbot: akosiaris@palladium conftool action : set/weight=1; selector: wtp1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
11:48 mark: puppet node clean & salt-key -d of cp3022.esams.wmnet
10:44 logmsgbot: krenair@tin Synchronized wmf-config: https://gerrit.wikimedia.org/r/302409 - labs-only changes (duration: 00m 34s)
09:49 godog: temporarily stop puppet and disable check_hpssacli on ms-be1023 T136631
09:42 moritzm: installing chromium security updates on osmium
09:39 godog: upload scap 3.2.1-1 to carbon T127762 (at 9:33)
07:55 akosiaris: T135176 depool wtp100[12]
07:54 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
07:54 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1001.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
02:26 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Aug 2 02:26:32 UTC 2016 (duration 5m 22s)
02:21 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.12) (duration: 08m 25s)

2016-08-01

23:31 eileen: Upgrading CiviCRM from d657255e1edebeccfc0a03bea70b78eb11375cf8 to 4904c4aae3565b65d5f37ecb827ea26c930b72d6
23:27 logmsgbot: dereckson@tin Synchronized wmf-config/: Revert "Add $wmgEchoMentionStatusNotifications and enable it in beta labs" (T135717, T139623) (duration: 00m 27s)
23:25 logmsgbot: dereckson@tin Synchronized wmf-config/: Add $wmgEchoMentionStatusNotifications and enable it in beta labs (T135717, T139623) (duration: 00m 30s)
23:24 eileen: Updating civicrm from d657255e1edebeccfc0a03bea70b78eb11375cf8 to d657255e1edebeccfc0a03bea70b78eb11375cf8
23:21 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Load Elastica extension via extension.json (Gerrit:301856) (duration: 00m 31s)
23:15 mutante: root@tin:/srv/mediawiki-staging# find . -uid 0 -exec chown mwdeploy:wikidev {} \;
23:05 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings-labs.php: Labs: Set CategoryCollation for dewiki to 'uca-de-u-kn' (T128806) (duration: 00m 38s)
22:03 dapatrick: Deployed patch for T139670 to wmf.12
21:59 Amir1: restarting uwsgi-ores in scb1001 and scb1002
21:37 Amir1: for ores
21:37 Amir1: deploying 6790ccb
21:20 Amir1: deploying e8d2475 to scb nodes
21:17 chasemp: iridium sudo -u phd /srv/phab/phabricator/bin/repository update 1912
20:33 Amir1: deploying 624d777 to ores
20:18 subbu: finished deploying parsoid sha abf396eb
20:15 mutante: restarted ganglia aggregators on carbon
20:06 subbu: synced new parsoid code; restarted parsoid on wtp1001 as a canary
20:04 subbu: starting parsoid deploy
19:56 logmsgbot: mobrovac@tin Synchronized wmf-config/LabsServices.php: (no message) (duration: 00m 38s)
19:37 Pchelolo: deploy restbase 840411a4
19:32 Pchelolo: deploy restbase 840411a4 canary on restbase1007
18:49 mutante: rebooting xenon to toggle HT setting in BIOS
18:10 bblack: upgrading openssl on cache_text and cache_upload
17:36 bblack: upgrading openssl package on cache_maps + cache_misc
17:06 Pchelolo: staging deploy restbase 840411a44
16:59 Pchelolo: labs deploy restbase 840411a44
16:10 bd808: Restarted logstash on logstash1001; missing hhvm and service logs; no output to /var/log/logstash/logstash.log for days
15:59 bblack: cp1065: canary upgrade of openssl to 1.0.2h-1~wmf2 (+ nginx upgrade-restart)
15:58 urandom: T134016: Bootstrapping restbase1015-c.eqiad.wmnet
15:57 bblack: uploaded openssl_1.0.2h-1~wmf2 to carbon (jessie-wikimedia) - https://gerrit.wikimedia.org/r/#/c/301903/
15:38 logmsgbot: addshore@tin Synchronized php-1.28.0-wmf.12/includes/Linker.php: Debug Logging for Undefined index: width in Linker.php (duration: 00m 25s)
15:29 logmsgbot: addshore@tin Synchronized wmf-config/InitialiseSettings.php: Remove dewiki_diffstats logging (duration: 00m 26s)
15:27 logmsgbot: addshore@tin Synchronized wmf-config/InitialiseSettings.php: Remove T124356 debug logging (duration: 00m 25s)
15:25 logmsgbot: addshore@tin Synchronized wmf-config/InitialiseSettings.php: Debug logging for T138987 (duration: 00m 26s)
15:23 logmsgbot: addshore@tin Synchronized wmf-config/InitialiseSettings-labs.php: Beta move $wgEchoMentionStatusNotifications to CommonSettings PART 2/2 (duration: 00m 24s)
15:22 logmsgbot: addshore@tin Synchronized wmf-config/CommonSettings-labs.php: Beta move $wgEchoMentionStatusNotifications to CommonSettings PART 1/2 (duration: 00m 24s)
15:17 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Set favicon for mk.wiktionary (T140566) (duration: 00m 25s)
15:11 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Allow sysops on he.wiktionary to remove autopatroller and patroller user rights (T140563) (duration: 00m 30s)
15:04 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: SWAT: Expanding throttle limits for enwiki Edit-a-thon (T141421) (duration: 00m 24s)
14:41 andrewbogott: restarting slapd on serpens to reclaim leaked memory
13:48 godog: reboot ms-be1023 after raid controller fw upgrade T141756
12:46 mobrovac: zotero translators deployed cde2f75
10:11 godog: reboot ms-be2027 after raid controller fw upgrade T141756
09:25 jynus: dropping index name_type_patrolled_timestamp on zhwiki on db1060 and db1054 T140108
03:20 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.12/includes/filerepo/file/LocalFile.php: c4f34e7a12baa9 (duration: 00m 44s)
02:26 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Aug 1 02:26:44 UTC 2016 (duration 5m 38s)
02:21 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.12) (duration: 08m 22s)

2016-07-31

18:07 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.12/includes/page/WikiPage.php: 061737668d729ba1a76 (duration: 00m 24s)
17:49 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.12/includes/page/WikiPage.php: 23330222a2af3fb (duration: 00m 34s)
02:27 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Jul 31 02:27:53 UTC 2016 (duration 5m 34s)
02:22 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.12) (duration: 08m 07s)

2016-07-30

23:36 Jamesofur: deleted 7 files from server for legal compliance
02:25 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Jul 30 02:25:55 UTC 2016 (duration 5m 37s)
02:20 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.12) (duration: 08m 10s)

2016-07-29

23:56 YuviPanda: hardreset xenon
23:44 urandom: Rebooting xenon.eqiad.wmnet
23:13 bblack: installed openssl-1.0.2h-1~wmf2 on pinkunicorn for the weekend (not on carbon yet) - https://gerrit.wikimedia.org/r/301903
21:02 ostriches: gerrit: raised log level on sshd to ERROR from WARN. Irrelevant logspam.
20:56 mutante: restarted grrrit-wm but this time only because it died by itself
20:22 mutante: gerrit restarting to apply config change 301873 - tuning caches
20:11 YuviPanda: restart gerrit-wm bot
19:56 mutante: gerrit restarting to apply config change 300446 - up heap size limit
18:59 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings-labs.php: labs only change to enable mobile language bar (duration: 00m 27s)
18:37 urandom: T134016: Bootstrapping restbase2009-c.codfw.wmnet
16:39 YuviPanda: granted addshore admin on labs grafana
15:17 anomie: starting maintenance script for phab:T140811
13:15 Dereckson: Purged static resources related to mk.wiktionary (T141610)
12:43 elukey: upgrading zuul-merger to zuul_2.1.0-391-gbc58ea3-wmf2jessie1_amd64.deb on scandium
10:23 elukey: restarting cassandra on aqs100[123] to apply the latest config (https://gerrit.wikimedia.org/r/#/c/301780/1 - T140869)
10:14 jynus: applying new grants to all s1 servers
09:38 hashar: Upgrading Zuul to get rid of a forced sleep(300) whenever a patch is merged T93812. zuul_2.1.0-391-gbc58ea3-wmf2precise1
08:51 godog: switch back statsite flush period to 60s T101141
07:30 jynus: schema change continues for s2, s1, s4 and s5 T140108
07:24 jynus: fixing s3 replication lag created by TokuDB insert problem
06:59 jynus: powercycling db2069 T141601
04:04 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.12/extensions/TorBlock/extension.json: Move basic torunblocked line to GrantPermissions, not GroupPermissions, see wikitech-l (duration: 00m 38s)
02:26 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Jul 29 02:26:09 UTC 2016 (duration 5m 57s)
02:20 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.12) (duration: 07m 30s)

2016-07-28

23:29 ejegg: updated payments-wiki from 2d9dd79507a42ced0a99bde87b3c45b804610e40 to 3a724bfb1a3e20e17b5886dae0ba7572020abd6b4
23:23 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/301724/1 (duration: 00m 24s)
23:19 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/301644/ (duration: 00m 29s)
22:42 logmsgbot: maxsem@tin Synchronized wmf-config/: Labs only (duration: 00m 45s)
20:25 urandom: T134016: Bootstrapping restbase2006-c.codfw.wmnet
20:02 Pchelolo: deploy restbase cdd164c4e
19:43 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.12/includes/api/ApiQueryUserContributions.php: Fix Undefined variable issue in ApiQueryUserContributions (duration: 00m 32s)
19:42 Pchelolo: deploy restbase cdd164c4e canary on restbase1007
19:22 urandom: T134016: Bootstrapping restbase1014-c.eqiad.wmnet
19:06 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.12
19:01 mutante: restarted grrrit-wm
17:37 ottomata: powercycling analytics1032
16:50 godog: bounce statsite on graphite1001 T101141
16:38 jynus: stopping dbproxy1001 haproxy service
15:41 logmsgbot: aude@tin Synchronized wmf-config/Wikibase.php: Update entityNamespaces setting (duration: 00m 27s)
15:28 logmsgbot: aude@tin Synchronized php-1.28.0-wmf.12/extensions/Wikidata: Fix exception when undeleting items and fix css bug (duration: 01m 52s)
15:20 logmsgbot: aude@tin Synchronized php-1.28.0-wmf.11/extensions/ContentTranslation: Fix DumpCorpora script (duration: 00m 27s)
15:19 logmsgbot: aude@tin Synchronized php-1.28.0-wmf.12/extensions/ContentTranslation: Fix DumpCorpora script (duration: 00m 31s)
15:15 godog: bounce statsite on graphite1001 - T101141
15:10 logmsgbot: aude@tin Synchronized dblists/clldefault.dblist: Enable content translation on more wikis (duration: 00m 23s)
15:09 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable content translation on more wikis (duration: 00m 25s)
15:07 jynus: adding new index (schema change) to recentchanges T140108
14:59 godog: bounce carbon-cache on graphite1001 - T101141
14:11 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1055 after maintenance (duration: 00m 35s)
12:51 elukey: upgrading zuul-merger to zuul_2.1.0-391-gbc58ea3-wmf1jessie (T140894)
12:14 akosiaris: installing updates on mendelevium
11:18 jynus: deploying schema change to all ores databases T140803
11:17 godog: replace statsdlb with statsd-proxy on graphite1001
11:09 jynus: testing schema change on db2038
09:25 moritzm: installing PHP security updates
08:49 godog: swift eqiad-prod: ms-be102[3456] weight 3000
07:43 elukey: starting decom process for old api servers - mw11(1[4-9]|20|3[0-9]|4[0-8]).eqiad.wmnet (tracked in https://etherpad.wikimedia.org/p/appservers-decom)
07:16 _joe_: regenerated the ssl key for rhodium, 1024 bits
06:54 moritzm: installing java security updates on restbase staging systems
06:44 _joe_: installed puppet 3.8 from backports on rhodium
06:36 moritzm: installing perl security updates on eqiad and codfw jessie systems
06:36 _joe_: refreshed puppet facts for the compiler
05:56 jynus: shutting down db1055 for upgrade
02:47 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Jul 28 02:47:28 UTC 2016 (duration 6m 22s)
02:41 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.12) (duration: 08m 03s)
02:24 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.11) (duration: 08m 49s)
02:13 twentyafterfour: restarted apache2 on iridium to deploy 4305a9bb0300650ea40de433261c7e59cc88e4bc
01:30 twentyafterfour: Deploying #phab-2016.30 (https://phabricator.wikimedia.org/project/profile/2118/) - no downtime is expected.

2016-07-27

23:45 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings-labs.php: Test numeric sorting on Beta Cluster (Gerrit:301520, labs only, no-op in prod) (duration: 00m 23s)
23:22 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.12/extensions/ORES/maintenance/PopulateDatabase.php: Add revision_id to log for errors (T141368, 2/2, no-op) (duration: 00m 29s)
23:21 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.12/extensions/ORES/includes/Cache.php: Add revision_id to log for errors (T141368, 1/2) (duration: 00m 31s)
22:50 Pchelolo: restbase deploy cdd164c4e8 to staging
21:34 mutante: planet2001 tmp disable puppet for testing
21:17 ejegg: updated payments from 79cb53998c41f72d0fa49130ed1f66dc112b478c to 2d9dd79507a42ced0a99bde87b3c45b804610e40
21:13 bearND: deployed mobileapps e561edf
21:08 bearND: starting mobileapps deploy
20:47 andrewbogott: restarting rabbitmq-server on labcontrol1001 because sometimes that fixes a thing
19:59 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.12
19:38 Pchelolo: deploy restbase 8f5e2897e to staging
19:16 andrewbogott: rebuilding labnet1001 (it's a spare and shouldn't affect Labs)
19:04 urandom: T134016: Bootstrapping restbase2005-c.codfw.wmnet
18:50 Pchelolo: restart changeprop to apply config changes 300681 and 301305
18:08 Pchelolo: deploy restbase 8efbc9282e to staging
17:58 urandom: T134016: Restarting Cassandra instance to apply disabled streaming socket timeout (restbase2009-b.codfw.wmnet)
17:51 mutante: restarted grrriit-wm
17:45 mutante: gerrit is restarting to deploy config change 301381, a couple seconds downtime
17:23 jynus: running analyze table on enwiki.logging db1055 (depooled)
17:22 urandom: Truncating "local_group_wikipedia_T_parsoid_section_offsets".data, "local_group_wikipedia_T_parsoid_dataW4ULtxs1oMqJ".data, and "local_group_wikipedia_T_parsoid_html".data in RESTBase staging
17:21 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1055 also from the rc/log role (duration: 00m 28s)
16:33 urandom: T134016: Restarting Cassandra instance to apply disabled streaming socket timeout (restbase2009-a.codfw.wmnet)
16:22 logmsgbot: addshore@tin Synchronized php-1.28.0-wmf.12/extensions/MobileFrontend/includes/skins/SkinMinerva.php: 301387 Fix watchstar for logged-out user (duration: 00m 32s)
16:04 logmsgbot: addshore@tin Synchronized php-1.28.0-wmf.11/includes/EditPage.php: 301356 Count edit conflicts for each namespace separately (duration: 00m 32s)
15:54 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: SWAT: IP cap lift for Wikipedia Edit-a-thon on 2016-08-03 (duration: 00m 23s)
15:49 urandom: T134016: Restarting Cassandra instance to apply disabled streaming socket timeout (restbase2006-b.codfw.wmnet)
15:45 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add "upwizcampeditors" to $wgAddGroups, $wgRemoveGroups for commonswiki (duration: 00m 24s)
15:38 logmsgbot: thcipriani@tin Synchronized wmf-config/CirrusSearch-common.php: SWAT: Turn on textcat based language detection for search PART II (duration: 00m 23s)
15:37 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Turn on textcat based language detection for search PART I (duration: 00m 27s)
15:21 urandom: T134016: Restarting Cassandra instance to apply disabled streaming socket timeout (restbase2006-a.codfw.wmnet)
15:06 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: beta wgEchoMentionStatusNotifications default true (duration: 01m 28s)
14:58 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1055 for database maintenance (duration: 00m 29s)
14:50 urandom: T134016: Restarting Cassandra instance to apply disabled streaming socket timeout (restbase2005-b.codfw.wmnet)
14:16 urandom: T134016: Cancelling bootstrap of restbase2005-c.codfw.wmnet
14:12 urandom: T134016: Restarting Cassandra instance to apply disabled streaming socket timeout (restbase2005-a.codfw.wmnet)
13:29 elukey: Restart Cassandra on aqs100[123] to apply the latest configuration (T140869)
12:50 elukey: disabling puppet on restbase*, aqs* and maps* as extra careful step for https://gerrit.wikimedia.org/r/301083 (no-op but better safe than sorry)
12:28 bblack: starting wipe of cache_misc caches
12:23 akosiaris: puppet enabled on netmon1001 (correction of previous log line)
12:22 akosiaris: puppet enabled on net1001
12:05 akosiaris: disable puppet on netmon1001, debugging servermon
11:25 Dereckson: Run initSiteStats to update statistics count on ast.wikipedia (T141432)
10:42 jynus: add extra grants to db1016 and all of m1 for servermon
10:20 moritzm: restarting slapd on serpens
08:43 elukey: Decomissioning mw1018-25 (T139353)
08:30 logmsgbot: hashar@tin Synchronized wmf-config/InitialiseSettings.php: Lower loglevel for resourceloader to info https://gerrit.wikimedia.org/r/#/c/301336/ (duration: 00m 26s)
07:49 jynus: update m1-master to point to dbproxy1006
07:11 moritzm: installing perl security updates in esams and codfw
06:56 jynus: dropping tables from m4 shard T141407
03:06 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Jul 27 03:06:43 UTC 2016 (duration 6m 49s)
02:59 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.12) (duration: 15m 29s)
02:27 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.11) (duration: 09m 11s)
02:09 mutante: restarted grrrit-wm after removing bugzilla password from gerrit
01:57 mutante: lead removed reviewer_count job from root's crontab
01:41 mutante: ytterbium - shutdown -h now, over and out
01:05 ejegg: rolled back paymentswiki from 79d2b67067fd7e579372b63e0d619eccfa3b9143 to 79cb53998c41f72d0fa49130ed1f66dc112b478c
00:56 mutante: xenon - rsync cassandra-test data to restbase-test2001 /srv/backups/eqiad/
00:53 mutante: restbase-test2001-2003 - test rsyncing, create temp data dir. mkdir -p $(grep path /etc/rsync.d/frag-parsoid-html | cut -d= -f2)

2016-07-26

23:59 logmsgbot: reedy@tin Synchronized php-1.28.0-wmf.12/extensions/MobileFrontend/: Deploy revert for group0 for T141386 (duration: 00m 30s)
23:25 logmsgbot: reedy@tin Synchronized dblists/commonsuploads.dblist: Disabling local uploads on ms.wikipedia.org (duration: 00m 23s)
23:18 logmsgbot: reedy@tin Synchronized wmf-config/event-schemas: Bump event-schemas submodule commit to master (duration: 00m 28s)
23:08 ejegg: updated payments from 79cb53998c41f72d0fa49130ed1f66dc112b478c to 79d2b67067fd7e579372b63e0d619eccfa3b9143
23:07 logmsgbot: reedy@tin Synchronized wmf-config/: Remove rest of ImageMetrics config (duration: 00m 33s)
23:04 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: Undeploy ImageMetrics (duration: 00m 27s)
21:05 logmsgbot: reedy@tin Synchronized wmf-config/extension-list: moar extension.json (duration: 00m 33s)
20:43 mdholloway: mobileapps deployed fd3f33b
20:41 mdholloway: starting mobileapps deployment
20:37 urandom: Bootstrapping restbase2005-c.eqiad.wmnet
20:23 urandom: T134016: Bootstrapping restbase1009-c.eqiad.wmnet
19:58 urandom: T134016, T140825: Restarting Cassandra to disable trickle_fsync and streaming socket timeouts (restbase1015-b.eqiad.wmnet)
19:54 urandom: T134016, T140825: Restarting Cassandra to disable trickle_fsync and streaming socket timeouts (restbase1015-a.eqiad.wmnet)
19:53 urandom: T140825: Setting vm.dirty_background_bytes=24576 (restbase1015.eqiad.wmnet)
19:49 urandom: T134016, T140825: Restarting Cassandra to disable trickle_fsync and streaming socket timeouts (restbase1014-b.eqiad.wmnet)
19:43 urandom: T134016, T140825: Restarting Cassandra to disable trickle_fsync and streaming socket timeouts (restbase1014-a.eqiad.wmnet)
19:42 urandom: T140825: Setting vm.dirty_background_bytes=24576 (restbase1014.eqiad.wmnet)
19:40 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.28.0-wmf.12
19:37 urandom: T134016, T140825: Restarting Cassandra to disable trickle_fsync and streaming socket timeouts (restbase1009-b.eqiad.wmnet)
19:34 logmsgbot: thcipriani@tin Finished scap: testwiki to php-1.28.0-wmf.12 and rebuild l10n cache (duration: 25m 29s)
19:33 urandom: T134016, T140825: Restarting Cassandra to disable trickle_fsync and streaming socket timeouts (restbase1009-a.eqiad.wmnet)
19:33 urandom: T140825: Setting vm.dirty_background_bytes=24576 (restbase1009.eqiad.wmnet)
19:09 logmsgbot: thcipriani@tin Started scap: testwiki to php-1.28.0-wmf.12 and rebuild l10n cache
19:07 logmsgbot: thcipriani@tin Purged l10n cache for 1.28.0-wmf.10
18:36 logmsgbot: addshore@tin Synchronized php-1.28.0-wmf.11/extensions/WikimediaEvents/WikimediaEventsHooks.php: dewiki_diffstats add rev timestamps & feature state 301119 (duration: 00m 28s)
18:28 logmsgbot: addshore@tin Synchronized wmf-config/InitialiseSettings.php: Enable RevisionSlider on mediawikiwiki 301105 (duration: 01m 28s)
18:07 bd808: Restarted elasticsearch on logstash1003, couldn't find master
17:12 subbu: finished deploying parsoid version 285b6983
17:10 thcipriani: starting branch cut for 1.28.0-wmf.12
17:06 subbu: synced new parsoid code; restarted parsoid on wtp1007 as a canary
17:03 subbu: starting parsoid deploy
15:55 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Configuration changes for mk.wiktionary.org PART III (duration: 00m 26s)
15:54 logmsgbot: thcipriani@tin Synchronized static/images/project-logos/mkwiktionary.png: SWAT: Configuration changes for mk.wiktionary.org PART II (duration: 00m 24s)
15:54 logmsgbot: thcipriani@tin Synchronized static/favicon/wiktionary/mk.ico: SWAT: Configuration changes for mk.wiktionary.org PART I (duration: 00m 24s)
15:47 godog: reimage mw1292 as thumbor1002
15:12 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove EchoBundleEmailInterval (T135446) PART II (duration: 00m 26s)
15:12 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Remove EchoBundleEmailInterval (T135446) PART I (duration: 00m 34s)
15:00 paravoid: installed cr2-eqiad FPC 3
14:36 moritzm: uploading openjdk-8 security update (8u102-b14-1~bpo8+1) to carbon
14:32 godog: reimage mw1291 as thumbor1001
13:55 jynus: compressing 300GB table on dbstore2002 (expect warnings, slowdown, lag -but it is a passive analytics slave)
12:42 moritzm: installing perl security updates
11:48 moritzm: installing exim4 updates related to perl security release
11:39 logmsgbot: filippo@palladium conftool action : set/pooled=inactive; selector: name=mw1291.eqiad.wmnet
11:39 logmsgbot: filippo@palladium conftool action : set/pooled=inactive; selector: name=mw1292.eqiad.wmnet
11:29 logmsgbot: filippo@palladium conftool action : set/pooled=no; selector: name=mw1292.*
11:28 logmsgbot: filippo@palladium conftool action : set/pooled=no; selector: name=mw1291.*
10:43 elukey: restarting cassandra on aqs100[456] instances (not serving live traffic)
09:18 moritzm: updating debhelper, cdbs, devscripts, libintl-perl, libmodule-build-perl and libnet-dns-perl on jessie systems for compatibility with perl security update
07:38 kart_: Update cxserver to 447a6c9 - registry: Remove 'en' as target from Apertium MT - disables machine translation to English in ContentTranslation
02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Jul 26 02:31:43 UTC 2016 (duration 6m 8s)
02:25 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.11) (duration: 09m 00s)
01:51 mutante: cerium testing is over?
01:11 Amir1: deploying from 2d9817b to a291da1 for ores in scb nodes
00:53 mutante: lead - stopped rsyncd
00:49 urandom: T134016: Bootstrapping restbase2008-c.codfw.wmnet
00:06 Pchelolo: restbase deploy ae5fbac to staging

2016-07-25

23:11 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Add dewiki_diffstats to wmgMonologChannels (Gerrit:288158, T134861) (duration: 00m 25s)
22:48 legoktm: restarted zuul due to depends-on lockup
22:46 logmsgbot: reedy@tin Synchronized docroot/noc/conf/: Update dblist symlinks (duration: 00m 37s)
21:52 tgr: deployed security patch for T137551
20:37 Pchelolo: restbase deploy 8efbc92
20:10 Pchelolo: restbase deploy 8efbc92 canary deploy to restbase1007
20:05 Pchelolo: restbase deploy 8efbc92 to staging
20:00 Pchelolo: restbase deploy 8efbc92 to deployment-prep
19:21 urandom: T134016: Bootstrapping restbase1013-c.eqiad.wmnet
18:31 ottomata: upgrading kafka to 0.9 in main-codfw, first kafka2001 then 2002
18:15 mutante: ytterbium - revoke puppet cert, delete salt-key, remove from icinga
16:16 urandom: T134016: Restarting Cassandra to apply stream timeout (restbase1013-b.eqiad.wmnet)
16:10 urandom: T134016: Restarting Cassandra to apply stream timeout (restbase1013-a.eqiad.wmnet)
16:06 urandom: T140825, T134016: Restarting Cassandra to apply stream timeout, and disable trickle_fsync (restbase1012-c.eqiad.wmnet)
16:02 urandom: T140825, T134016: Restarting Cassandra to apply stream timeout, and disable trickle_fsync (restbase1012-b.eqiad.wmnet)
15:54 urandom: T140825, T134016: Reststarting Cassandra to apply stream timeout, and disable trickle_fsync (restbase1012-a.eqiad.wmnet)
15:53 urandom: T140825: Setting vm.dirty_background_bytes=24M on restbase1012.eqiad.wmnet
15:43 urandom: T140825, T134016: Reststarting Cassandra to apply stream timeout, and 8MB trickle_fsync (restbase1008-c.eqiad.wmnet)
15:39 urandom: T140825, T134016: Reststarting Cassandra to apply stream timeout, and 8MB trickle_fsync (restbase1008-b.eqiad.wmnet)
15:34 urandom: T140825, T134016: Reststarting Cassandra to apply stream timeout, and 8MB trickle_fsync (restbase1008-a.eqiad.wmnet)
15:28 elukey: Standardized the jmxtrans GC metric names to pick up automatically variations in settings. This introduces metric name changes in Hadoop, Zookeeper, Kafka. (https://gerrit.wikimedia.org/r/#/c/299118/)
12:53 moritzm: installing squid security updates
10:10 _joe_: remove spurious puppet facts
10:10 _joe_: remove spurious puppet facts
10:04 moritzm: installing Django security updates
09:18 godog: swift eqiad-prod: ms-be102[3456] weight 1500
03:26 hashar: scandium: migrating zuul-merger repos from lead to gerrit.wikimedia.org: find /srv/ssd/zuul/git -path '*/.git/config' -print -execdir sed -i -e 's/lead.wikimedia.org/gerrit.wikimedia.org/' config \;
02:28 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Jul 25 02:28:21 UTC 2016 (duration 5m 52s)
02:22 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.11) (duration: 09m 09s)
02:03 ostriches: gerrit: reindexing lucene now that we have new data. searches/dashboards may look a tad weird for a bit
01:53 hashar: starting Zuul
01:51 mutante: restarted grrrit-wm
01:39 ostriches: lead: turning puppet back on, here we go
01:38 jynus: m2 replication on db2011 stopped, master binlog pos: db1020-bin.000968:1013334195
01:37 hashar: scandium: restarted zuul-merger
01:36 ostriches: ytterbium: Stopped puppet, stopped gerrit process.
01:34 mutante: switched gerrit-new to gerrit in DNS
01:30 ostriches: lead: stopped puppet for a few minutes
01:17 hashar: scandium: migrating zuul-merger repos to lead find /srv/ssd/zuul/git -path '*/.git/config' -print -execdir sed -i -e 's/ytterbium.wikimedia.org/lead.wikimedia.org/' config \;
01:10 hashar: stopping CI
01:09 jynus: reviewdb backup finished, available on db1020:/srv/tmp/2016-07-25_00-54-31/
01:02 ostriches: rsyncing latest git data from ytterbium to lead
00:57 mutante: manually deleted reviewer-counts cron from gerrit2 user, runs as root and puppet does not remove crons unless ensure=>absent
00:55 jynus: starting hot backup of db1020's reviewdb

2016-07-24

02:25 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Jul 24 02:25:08 UTC 2016 (duration 4m 34s)
02:20 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.11) (duration: 08m 59s)

2016-07-23

15:38 godog: stop swift in esams test cluster, lots of logging from there
15:37 godog: lithium sudo lvextend --size +10G -r /dev/mapper/lithium--vg-syslog
04:58 ori: Gerrit is back up after service restart; was unavailable between ~ 04:29 - 04:57 UTC
04:56 ori: Restarting Gerrit on ytterbium
04:48 ori: Users report Gerrit is down; on ytterbium java is occupying two cores at 100%
02:26 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Jul 23 02:26:49 UTC 2016 (duration 5m 41s)
02:21 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.11) (duration: 08m 24s)
01:02 logmsgbot: tgr@tin Synchronized php-1.28.0-wmf.11/extensions/CentralAuth/includes/CentralAuthPlugin.php: T141160 (duration: 00m 29s)
01:01 logmsgbot: tgr@tin Synchronized php-1.28.0-wmf.11/extensions/CentralAuth/includes/CentralAuthHooks.php: T141160 (duration: 00m 27s)
01:00 logmsgbot: tgr@tin Synchronized php-1.28.0-wmf.11/extensions/CentralAuth/includes/CentralAuthPrimaryAuthenticationProvider.php: T141160 (duration: 00m 28s)
00:37 tgr: doing an emergency deploy of https://gerrit.wikimedia.org/r/#/c/300679 for T141160, creates dozens of new users per hour to be unattached on loginwiki which probably has weird consequences

2016-07-22

22:19 logmsgbot: aaron@tin Synchronized wmf-config/InitialiseSettings.php: Enable debug logging for DBTransaction (duration: 00m 38s)
21:10 ejegg: updated civicrm from 2f4805fa2d2a7c57881408be2b3a017d26d8f43e to d657255e1edebeccfc0a03bea70b78eb11375cf8
20:58 ejegg: disabled Worldpay audit parser job
18:59 ejegg: rolled back payments from 79d2b67067fd7e579372b63e0d619eccfa3b9143 to 79cb53998c41f72d0fa49130ed1f66dc112b478c
18:54 mutante: restart grrrit-wm
16:05 Jeff_Green: running authdns-update to correct a DKIM public key on wikipedia.org
15:24 anomie: Starting script to populate empty gu_auth_token phab:T140478
15:16 urandom: T140825: Restarting Cassandra to apply 8MB trickle_fsync (restbase1015-a.eqiad.wmnet)
14:21 gehel: rolling restart of logstash100[1-3] - T141063
14:19 urandom: T134016: Boostrapping restbase2004-c.codfw.wmnet
12:42 jynus: applying new m5 db grants
11:12 jynus: reimage dbproxy1009 T140983
11:04 jynus: applying new m2 db grants
10:47 jynus: reimage dbproxy1007 T140983
10:36 jynus: applying new m1 db grants
10:27 hashar: Restarting Jenkins entirely (deadlocked)
10:23 hashar: Jenkins has some random deadlock. Will probably reboot it
09:45 jynus: reimage dbproxy1006
09:36 jynus: applying new m3 db grants
08:19 jynus: reimage dbproxy1008
06:43 jynus: updating dns records: m3-slave to db1043; m2-master to dbproxy1002
04:08 jynus: backing up, shutting down and reimage db1043
03:14 jynus: stopping db1043 db
03:06 twentyafterfour: restarted apache2 and phd on iridium
03:04 jynus: reverting m3-master dns back to the proxy
02:59 jynus: restarted phd on iridium
02:35 jynus: SET GLOBAL read_only=0; on db1048
02:34 jynus: updating m3-master dns
02:33 jynus: setting db1043 as read-only (phabricator/m3)
02:31 jynus: making dbstore1002.eqiad.wmnet:3306 a child of db1048.eqiad.wmnet:3306
02:27 jynus: making db2012.codfw.wmnet:3306 a child of db1048.eqiad.wmnet
02:25 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Jul 22 02:25:53 UTC 2016 (duration 5m 47s)
02:20 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.11) (duration: 08m 23s)
00:53 bd808: Restarted elasticsearch on logstash1003; couldn't find master (even though the master thought 1003 was fine)
00:43 mutante: restarted grrrit-wm
00:01 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings-labs.php: Labs-only cleanups (duration: 00m 25s)

2016-07-21

23:53 Amir1: deploying 2d9817b to ores in scb nodes
23:49 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 24s)
23:46 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#q,298344,n,z (duration: 00m 24s)
23:41 MaxSem: on tin: ran mwscript extensions/ShortUrl/populateShortUrlTable.php --wiki=urwiki
23:39 MaxSem: created ShortUrl tables on urwiki
23:37 ori: Restarted statsv on hafnium (cc Krinkle). 'gaierror: [Errno -3] Temporary failure in name resolution'
23:34 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.11/extensions/CirrusSearch/: https://gerrit.wikimedia.org/r/#q,300430,n,z https://gerrit.wikimedia.org/r/#q,300436,n,z (duration: 00m 32s)
23:32 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.11/extensions/EventBus/: https://gerrit.wikimedia.org/r/#q,300332,n,z (duration: 00m 26s)
23:27 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/299619/ (duration: 00m 24s)
23:22 logmsgbot: maxsem@tin Synchronized wmf-config/: https://gerrit.wikimedia.org/r/#/c/299615/ (duration: 00m 29s)
23:21 Amir1: restarting uwsgi and celery for ores in scb1002
23:20 logmsgbot: maxsem@tin Synchronized dblists/wikidatadescriptions.dblist: https://gerrit.wikimedia.org/r/#/c/299615/ (duration: 00m 24s)
23:19 Amir1: restarting uwsgi and celery for ores in scb 1001
23:09 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/298933/ (duration: 00m 29s)
22:46 ebernhardson: restart elasticsearch on logstash1002
22:22 bd808: Restarted kibana4 on logstash1001 for "node[18588]: segfault at 2fcb25f00009 ip 0000000000ad9846 sp 00007ffe526bbb40 error 4 in node[400000+1383000]"
22:01 mutante: stat1002 - puppetized git pull from "refinery_source" fails
21:11 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: Moved WMF specific SiteMatrix data to CommonSettings (duration: 00m 26s)
20:28 ejegg: re-enabled fundraising campaigns after schema update
19:27 logmsgbot: demon@tin Synchronized wikiversions.json: because sync-wikiversions doesn't care about co-masters ugh (duration: 00m 29s)
19:09 logmsgbot: demon@tin rebuilt wikiversions.php and synchronized wikiversions files: last wikis to wmf.11
19:07 ejegg: disabled fundraising CentralNotice campaigns for paymentswiki schema update
18:34 ejegg: updated payments-wiki from f23f15656eb488f5008b45b940077abbaa779004 to 79d2b67067fd7e579372b63e0d619eccfa3b9143
17:33 mutante: restarted grrrit-wm
17:26 logmsgbot: krinkle@tin Synchronized w/static.php: allow short-lived caching of 400/500 errors (duration: 00m 24s)
17:15 ostriches: gerrit: restarting
17:13 ostriches: gerrit: killed a couple of long-running git-upload-pack's for mediawiki/core
17:07 gehel: cleaning leftover crons on logstash* servers - T140973
16:51 urandom: T134016: Starting bootstrap of restbase2003-c.codfw.wmnet
16:50 urandom: T134016: Restart of codfw rack 'c' instances to apply stream socket timeout complete
16:47 urandom: T134016: Restarting Cassandra to apply new stream timeout (restbase2008-b.codfw.wmnet)
16:46 logmsgbot: ebernhardson@tin Synchronized php-1.28.0-wmf.11/extensions/CirrusSearch/includes/Searcher.php: T140950: Deploy UBN fix to CirruSearch (duration: 00m 31s)
16:46 urandom: T134016: Restarting Cassandra to apply new stream timeout (restbase2008-a.codfw.wmnet)
16:43 urandom: T134016: Restarting Cassandra to apply new stream timeout (restbase2004-b.codfw.wmnet)
16:41 urandom: T134016: Restarting Cassandra to apply new stream timeout (restbase200r-a.codfw.wmnet)
16:38 urandom: T134016: Restarting Cassandra to apply new stream timeout (restbase2003-b.codfw.wmnet)
16:36 urandom: T134016: Restarting Cassandra to apply new stream timeout (restbase2003-a.codfw.wmnet)
16:30 subbu: finished (test) deploy of parsoid sha ed2f8228
16:28 urandom: Cancelling 2003-c bootstrap, and disabling Puppet on restbase2003.codfw.wmnet to keep instance down : T134016
16:27 subbu: synced parsoid code; restarting parsoid on wtp1001 as a canary
16:24 subbu: starting (test) parsoid deployment
16:17 subbu: aborted (test) parsoid deployment
16:13 subbu: starting parsoid deployment
14:58 jynus: stopping dbstore1002 for scheduled maintenace T119488
14:44 paravoid: cr2-eqiad is now upgraded, passing transit and cross-DC traffic and is the VRRP master in eqiad
14:43 paravoid: cr2-eqiad: restoring VRRP priorities
14:40 paravoid: cr2-eqiad: restoring PyBal BGP sessions
14:39 paravoid: cr2-eqiad: reenabling IX interface & BGP
14:37 paravoid: cr2-eqiad: reenabling Transit interfaces & BGP
14:35 paravoid: cr2-eqiad: enabling Fundraising interface & BGP
14:30 paravoid: cr2-eqiad: reenabling xe-4/2/0 (link to cr1-eqord) and xe-5/2/3 (link to cr2-codfw)
14:26 paravoid: cr2-eqiad: reenabling all asw-*-eqiad interfaces
14:14 logmsgbot: demon@tin Synchronized wmf-config/: extension list cleanups (duration: 00m 34s)
14:07 paravoid: cr2-eqiad: halting both routing engines(!)
14:04 paravoid: cr2-eqiad: disabling xe-4/2/0 (link to cr1-eqord)
14:04 paravoid: cr2-eqiad: disabling xe-5/2/3 (link to cr2-codfw)
14:02 paravoid: cr2-eqiad: disabling all asw-*-eqiad interfaces
13:41 paravoid: cr2-eqiad: fabric upgrade bandwidth for FPC 4/5
13:38 paravoid: cr2-eqiad: toggling mastership between routing-engines (re1->re0)
13:31 paravoid: cr2-eqiad: setting scb 0 to offline and replacing it
13:31 paravoid: cr2-eqiad: setting fabric plane 0/1/2/3 to offline
13:30 paravoid: cr2-eqiad: powering off re0 (backup)
13:28 paravoid: cr2-eqiad: toggling mastership between routing-engines (re0->re1)
13:18 mobrovac: mathoid deploying 36be4ea
13:12 paravoid: cr2-eqiad: setting scb 1 to offline and replacing it
13:10 paravoid: cr2-eqiad: setting fabric plane 5/6/7 to offline
13:10 paravoid: cr2-eqiad: setting fabric plane 4 to offline
13:10 paravoid: cr2-eqiad: setting "chassis state cb-upgrade on" and powering off re1 (backup)
13:01 godog: bounce gerrit on ytterbium
12:58 godog: manually flipping m2-master to db1020
12:49 paravoid: cr2-eqiad: re-enabling GRES and toggling mastership between routing-engines (re1->re0)
12:48 paravoid: cr2-eqiad: fixing IPv6 VRRP interoperatbility between the cr1/cr2 ( http://www.juniper.net/documentation/en_US/junos14.2/topics/concept/vrrpv3-junos-support.html )
12:41 mobrovac: citoid deployed 5134e49e
12:36 mobrovac: change-prop deploying b7079fd9c
12:27 paravoid: cr2-eqiad: rebooting backup RE (re0)
12:19 paravoid: cr2-eqiad: toggling mastership between routing-engines (re0->re1)
12:15 paravoid: cr2-eqiad: setting "chassis network-services enhanced-ip" and rebooting re1 (then re0 will follow)
11:49 paravoid: upgrading cr2-eqiad:re1 and rebooting
11:38 paravoid: cr2-eqiad: toggling mastership between routing-engines (re1->re0)
11:24 paravoid: upgrading cr2-eqiad:re0 and rebooting
11:15 paravoid: cr2-eqiad: deactivate chassis redundancy graceful-switchover
10:59 paravoid: cr2-eqiad: disabling IX/Transit/Fundraising interfaces
10:55 paravoid: cr2-eqiad: deactivating Fundraising BGP session
10:52 paravoid: cr2-eqiad: deactivating Transit BGP sessions
10:50 paravoid: cr2-eqiad: deactivating IX BGP sessions
10:35 paravoid: cr2-eqiad: increase cross-datacenter link OSPF metrics
09:47 gehel: reinstalling and configuring relforge1001/1002 - T137256
08:10 _joe_: restarting apache on palladium
07:47 apergos: restarted gerrit on ytterbium, it was refusing to complete git fetches for large repos (mw core, puppet...)
03:03 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Jul 21 03:03:21 UTC 2016 (duration 7m 2s)
02:56 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.11) (duration: 08m 57s)
02:31 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.10) (duration: 09m 33s)

2016-07-20

23:43 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings-labs.php: SWAT, no-op (duration: 00m 24s)
23:42 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.10/extensions/EventBus/EventBus.hooks.php: Add rev_by_bot flag to revision_create event (2/2) (duration: 00m 23s)
23:41 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.10/extensions/EventBus/extension.json: Add rev_by_bot flag to revision_create event (1/2) (duration: 00m 26s)
23:40 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.11/extensions/ORES/: Let ORES extension score for some namespaces instead of all (Gerrit:300083]) (duration: 00m 30s)
23:38 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: ORES score edits in main and Property namespaces in wikidatawiki (Gerrit:300086) (duration: 00m 33s)
21:59 mutante: new language "tcy" (Tulu) has been approved and added today - https://meta.wikimedia.org/wiki/Requests_for_new_languages/Wikipedia_Tulu
21:49 mutante: DNS authdns-gen-zones on all servers to add new language tcy (bug T97051)
20:28 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.11/extensions/PagedTiffHandler/: (no message) (duration: 00m 25s)
20:25 ejegg: updated civicrm from a386eb5a76ec97b3b01c46a49309dfa39bbc58b0 to 2f4805fa2d2a7c57881408be2b3a017d26d8f43e
20:20 awight: update paymentswiki from 7c6fb5a3b90fffdf2229cc903fb546e0e1e47998 to f23f15656eb488f5008b45b940077abbaa779004
20:06 logmsgbot: demon@tin Finished scap: group1 to wmf.11 (3rd bestest try) (duration: 46m 03s)
19:38 urandom: Starting Casssandra on restbase1011-b.eqiad.wmnet
19:20 logmsgbot: demon@tin Started scap: group1 to wmf.11 (3rd bestest try)
19:18 logmsgbot: demon@tin scap aborted: group1 to wmf.11 (2nd best try) (duration: 01m 30s)
19:17 logmsgbot: demon@tin Started scap: group1 to wmf.11 (2nd best try)
19:12 logmsgbot: demon@tin scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="fawiki" --list-file="/srv/mediawiki-staging/wmf-config/extension-list" --output="/tmp/tmp.KjzXNQRvAU" ' returned non-zero exit status 1 (duration: 00m 28s)
19:12 logmsgbot: demon@tin Started scap: group1 to wmf.11
17:23 urandom: Stopping restbase1013-c bootstrap (pending better timeouts) : T134016
16:09 logmsgbot: reedy@tin Synchronized wmf-config/extension-list: More to extension registration for l10n (duration: 00m 27s)
16:09 logmsgbot: thcipriani@tin Finished scap: SWAT: Fix spelling of RevisionSlider (T140875) (duration: 23m 18s)
15:52 urandom: Resuming failed bootstrap on restbase2003.codfw.wmnet : T134016
15:48 _joe_: removed ruthenium from the list of trebuchet minions
15:46 logmsgbot: thcipriani@tin Started scap: SWAT: Fix spelling of RevisionSlider (T140875)
15:41 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.11/extensions/RevisionSlider/modules/ext.RevisionSlider.HelpDialog.js: SWAT: Open links in the "tutorial" in the new window (T140875) (duration: 00m 27s)
15:32 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: Beta: Test OresEnabledNamespaces on enwiki (duration: 00m 25s)
15:28 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Let bureaucrats in fawiki remove sysop user group (T140810) (duration: 00m 25s)
15:26 mafk: SWAT for 299446 done, which fixes T140544
15:20 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Configuration changes for he.wikinews.org (duration: 00m 28s)
15:08 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Compact Language Links: To beta in ruwikivoyage (duration: 00m 33s)
15:04 urandom: Ammending my last to include 'Rack b'
15:03 urandom: RESTBase Cassandra: raising stream throughput to 25Mbit/s; lowering compaction throughput to 10MB/s : T134016
14:40 Jeff_Green: running authdns-update to deploy SPF/DKIM records for wikipedia.org
14:20 urandom: Restarting bootstrap on restbas1013.eqiad.wmnet (duplicate resume?) : T134016
14:05 urandom: Resuming failed bootstrap on restbase1013-c.eqiad.wmnet : T134016
13:30 urandom: Performing rolling RESTBase restart to work-around Cassandra instance restart fallout : T138314 and T138314
13:28 urandom: Restarting restbase1008-a.eqiad.wmnet to apply a (ephemeral) 7200000ms streaming timeout : T138314
13:15 paravoid: cr2-eqiad: setting VRRP priority to 50 for all subnets, effectively switching the VRRP master to cr1-eqiad
13:12 _joe_: transitioning wtp2011-2020
12:35 _joe_: transitioning wtp2002-2010
12:35 logmsgbot: oblivian@palladium conftool action : set/pooled=yes; selector: name=wtp2001.codfw.wmnet
12:26 _joe_: transition ongoing on wtp2001
12:23 logmsgbot: oblivian@palladium conftool action : set/pooled=no; selector: name=wtp2001.codfw.wmnet
12:22 logmsgbot: oblivian@palladium conftool action : set/pooled=no; selector: name=
12:17 _joe_: disabling puppet on all parsoid hosts for the transition to service-runner T90668
07:28 elukey: restarting evenbus on kafka100[12] (T140848)
06:14 _joe_: updating parsoid on wtp100[12]
04:55 mutante: osmium package chromium-browser is missing after upgrade, refered to by jsbench
04:45 mutante: osmium - rsyncing /home , /srv (except /srv/mediawiki created by puppet) back from temp backup on hafnium
04:38 mutante: osmium result: boots into 4.4 kernel which would not work before.. lol
04:35 mutante: osmium edit grub config to boot second entry (3.16), update-grub, reboot
03:00 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Jul 20 03:00:43 UTC 2016 (duration 6m 50s)
02:53 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.11) (duration: 07m 55s)
02:29 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.10) (duration: 08m 41s)
02:28 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: delist codereview (duration: 00m 27s)
01:39 logmsgbot: demon@tin Synchronized tests/SiteConfiguration.php: for completeness (duration: 00m 24s)
01:35 mutante: hafnium stopping rsyncd, deleting configs
01:33 mutante: labstore1005, accepting salt key (reinstall 2016-06-25)
01:29 mutante: rhodium, new puppetmaster, add to salt
01:19 mutante: osmium - revoke old puppet cert, salt-key .. sign new ones
01:19 mutante: osmium - after reinstall with jessie, did not boot with 4.4 kernel, _does_ boot with 3.16.04.. still jessie just booted manually into the older kernel in grub
01:13 logmsgbot: demon@tin Synchronized wmf-config/abusefilter.php: Disable abusefilter profiling on commonswiki (duration: 00m 26s)
00:44 logmsgbot: dereckson@tin Finished scap: wmf-config/ upgrade: Gerrit changes 296770, 296767, 296929, 296930, 292623, 292624 (duration: 45m 42s)
00:42 urandom: Temporarily reducing compaction throughput to 10MB/s on restbase1013-c.eqiad.wmnet : T134016
00:32 mutante: osmium - reboot into PXE, reinstall

2016-07-19

23:58 logmsgbot: dereckson@tin Started scap: wmf-config/ upgrade: Gerrit changes 296770, 296767, 296929, 296930, 292623, 292624
22:17 awight: update paymentswiki to fundraising/REL1_27 from e8b600c518b28e3f350ced85d7d1006a76b86596 to 7c6fb5a3b90fffdf2229cc903fb546e0e1e47998
22:10 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/299870/ (duration: 00m 29s)
21:54 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.10/extensions/AbuseFilter: Backported fix for logspam (duration: 00m 38s)
21:43 gwicke: temporarily lowering compaction throughput on all eqiad restbase cassandra instances from 60mb/s to 20mb/s via `nodetool setcompactionthroughput 20` (T140825)
21:23 gwicke: temporarily lowered compaction throughput on all 1012 instances from 60mb/s to 20mb/s via `nodetool setcompactionthroughput 20` (T140825)
21:08 urandom: Bootstrapping restbase2003-c.codfw.wmnet : T134016
20:45 urandom: Lowering compaction throughput to 20MB/s on restbase1013-{a,b}.eqiad.wmnet : T134016
20:44 urandom: Lowering compaction throughput from 35MB/s to 20MB/s on restbase1013-c.eqiad.wmnet : T134016
20:28 urandom: Throttling stream throughput to 20MB/s on all rack 'b' instances : T134016
20:21 urandom: Lowering compaction throughput from 45MB/s to 35MB/s on restbase1013-c.eqiad.wmnet : T134016
20:06 urandom: Disabling Puppet on restbase2003.codfw.wmnet : T134016
20:03 logmsgbot: demon@tin Finished scap: group0 to wmf.11 (duration: 24m 52s)
19:57 urandom: Reducing stream throughput on restbase1013-{a,b} to 20MB/s : T134016
19:47 urandom: Lowering compaction throughput from 60MB/s to 45MB/s on restbase1013-c.eqiad.wmnet : T134016
19:38 logmsgbot: demon@tin Started scap: group0 to wmf.11
19:31 logmsgbot: demon@tin Purged l10n cache for 1.28.0-wmf.9
19:31 logmsgbot: demon@tin Purged l10n cache for 1.28.0-wmf.8
19:31 logmsgbot: demon@tin Purged l10n cache for 1.28.0-wmf.7
19:31 logmsgbot: demon@tin Purged l10n cache for 1.28.0-wmf.7
19:31 logmsgbot: demon@tin Purged l10n cache for 1.28.0-wmf.6
19:12 urandom: Starting bootstrap of restbase1013-c.eqiad.wmnet : T134016
18:59 urandom: Disabling puppet on restbase1013.eqiad.wmnet : T134016
17:56 paravoid: cr1-eqiad: restart chassis-control immediately (should not be traffic affecting)
17:49 jynus: applying new grants to m3 dbs in preparation for db1043 failover/proxy implementation
17:43 mdholloway: mobileapps deployed aa9115a
17:41 mdholloway: starting mobileapps deployment
17:15 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: Don't set everywhere, breaks internal to us but external to MW requests (eg gerrit, ocg, etc) (duration: 00m 25s)
17:14 cmjohnson1: dbstore1002 swapping disk at slot 6
16:18 logmsgbot: oblivian@palladium conftool action : set/pooled=yes; selector: name=wtp1002.*,cluster=parsoid,dc=eqiad
16:01 cmjohnson1: swapping pem0-3 on cr1-eqiad
15:49 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.10/extensions/ContentTranslation/includes/AbuseFilterCheck.php: SWAT: Avoid accessing private $filters field (T139657) (duration: 00m 26s)
15:41 logmsgbot: oblivian@palladium conftool action : set/pooled=yes; selector: name=wtp1001.*,cluster=parsoid,dc=eqiad
15:19 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove Echo transition flags PART II (duration: 00m 30s)
15:19 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Remove Echo transition flags PART I (duration: 00m 26s)
15:18 cmjohnson1: replacing PEM0-3 cr2-eqiad
14:23 logmsgbot: oblivian@palladium conftool action : set/pooled=no:weight=15; selector: cluster=parsoid,name=wtp100[12].*
14:22 logmsgbot: oblivian@palladium conftool action : set/weight=0; selector: cluster=parsoid,name=wtp100[12].*
14:10 jynus: reboot and reimage dbproxy1003 to jessie T125027 T138460
13:48 jynus: restarting dbproxy1005 for kernel upgrade
13:44 jynus: reloading dbproxy1001 to repool db1001 as pasive backend
10:37 elukey: sent SIGHUP to eventbus on kafka100[12] to reload schemas
10:01 jynus: testing haproxy start sequence on dbproxy1005 (unused proxy)
10:01 mobrovac: scb disabling puppet
09:10 godog: upgrade slapd to 2.4.41+dfsg-1+wmf1 on serpens - T130593
02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Jul 19 02:30:08 UTC 2016 (duration 6m 2s)
02:24 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.10) (duration: 08m 37s)

2016-07-18

23:55 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/292622/ (duration: 00m 25s)
23:54 logmsgbot: maxsem@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/292622/ (duration: 00m 24s)
23:54 logmsgbot: maxsem@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/292622/ (duration: 00m 25s)
23:47 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/292621/ (duration: 00m 29s)
23:46 logmsgbot: maxsem@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/292621/ (duration: 00m 31s)
23:39 logmsgbot: maxsem@tin Synchronized wmf-config/: https://gerrit.wikimedia.org/r/#/c/292620/ part 2 (duration: 00m 27s)
23:38 logmsgbot: maxsem@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/292620/ part 1 (duration: 00m 26s)
23:31 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/292619/ part 2 (duration: 00m 24s)
23:30 logmsgbot: maxsem@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/292619/ part 1 (duration: 00m 26s)
23:19 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.10/extensions/MultimediaViewer/: https://gerrit.wikimedia.org/r/#/c/299560/ (duration: 00m 26s)
23:17 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.10/extensions/Kartographer/: https://gerrit.wikimedia.org/r/#/c/299560/ (duration: 00m 27s)
23:15 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/296673/ (duration: 00m 30s)
22:58 bawolff: deploy fix for T129738 to php-1.28.0-wmf.10
22:57 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: rm deprecated fundraising config (duration: 00m 26s)
22:51 mutante: restarted grrrit-wm
22:48 ostriches: lead: puppet turned back on
22:44 dapatrick: Deployed patches for T133147 to wmf.10
22:40 ostriches: lead: disabled puppet for a bit to test some CSS tweaks live.
22:35 mutante: gerrit-new restarting for config change 298710
22:34 dapatrick: Deployed patch for T132926 to wmf.10
22:32 mutante: gerrit restarting for config change 298710
22:18 bawolff: Deployed patch for T136402 on php-1.28.0-wmf.10
22:07 gehel: elasticsearch / kibana upgrade done
22:01 dapatrick: Deployed patch for T115333 to wmf.10
21:52 ebernhardson: brought up kibana4 on logstash.wikimedia.org
21:46 logmsgbot: demon@tin Finished scap: security, se-curity (duration: 08m 12s)
21:38 logmsgbot: demon@tin Started scap: security, se-curity
21:34 ebernhardson: changed replica count for logstash-2016.06-(01|02|03|15|16|17) indices back to 2
21:09 ebernhardson: changed replica count for logstash-2016.06-(01|02|03|15|16|17) indices to 0 to make room for recovering todays index
21:07 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: whitelist some rss feeds for mw.org (duration: 00m 43s)
21:01 ejegg: updated payments from 8d3873f8d6b0600331775e9ccfc0cf4c6ed1e181 to e8b600c518b28e3f350ced85d7d1006a76b86596
20:18 ebernhardson: installed elasticsearch-2.3.3 to logstash1001-6
20:02 ebernhardson: re-shutdown elasticsearch on logstash1001-6
19:55 ejegg: updated payments from 0c14940f4930e94a9287acae978cc6e661e54ee1 to 8d3873f8d6b0600331775e9ccfc0cf4c6ed1e181
19:54 ebernhardson: re-stopping logstash on logstash1001-3
19:52 gehel: disabling puppet on logstash.* nodes for elasticsearch upgrade
19:49 mutante: ytterbium - fixing Apache config, graceful
19:47 ebernhardson: shutdown elasticsearch on logstash1004-6
19:38 bd808: Dropped logstash-2016.07.04 through logstash-2016.07.14 indices for backing Elasticsearch upgrade
19:36 ebernhardson: shutting down logstash and elasticsearch on logstash1001-03
19:05 logmsgbot: demon@tin Synchronized private/: remove obsolete wikitech config file (duration: 00m 32s)
19:04 gehel: starting elasticsearch upgrade for logstash (T136001)
18:57 logmsgbot: demon@tin Synchronized wmf-config/: Pruning 1.27.0-wmf.N ExtensionMessages files (duration: 00m 34s)
18:08 ejegg: rolled back payments to 0c14940f4930e94a9287acae978cc6e661e54ee1
18:05 ejegg: updated payments from 0c14940f4930e94a9287acae978cc6e661e54ee1 to 8d3873f8d6b0600331775e9ccfc0cf4c6ed1e181
17:28 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: globally set $wgHTTPProxy (duration: 00m 26s)
17:18 mobrovac: moblieapps deploying debb3f6
17:08 gehel: updated wdqs to latest version, new blazegraph version, restart of wdqs-updater
15:23 logmsgbot: oblivian@palladium conftool action : set/pooled=yes; selector: name=mw1261.*
15:21 logmsgbot: oblivian@palladium conftool action : set/pooled=no; selector: name=mw1261.*
15:10 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable global abuse filters on ptwiki (T140395) (duration: 00m 38s)
15:09 hashar: gallium upgrading Zuul: zuul_2.1.0-151-g30a433b-wmf3precise1 zuul_2.1.0-151-g30a433b-wmf4precise1_amd64.deb . To support layout validation when multiple connections are used
14:49 mobrovac: mobileapps deploying dfe5f11f5
14:11 mobrovac: moblieapps deploying fb65cea
12:03 hashar: Gerrit was slow processing requests such as git pull since 11:17 UTC . Fixed by killing all idling/waiting tasks T140604
11:08 godog: swift codfw-prod: ms-be202[567] weight 3000 - T136630
10:10 jynus: hard reset for db2056 T140598
08:54 godog: swift eqiad-prod: ms-be102[3-6] to weight 500 - T136631
08:31 hashar: gallium: upgrading Zuul 2.1.0-95-g66c8e52-wmf1precise1 .. zuul_2.1.0-151-g30a433b-wmf3precise1 T137525
02:26 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Jul 18 02:26:24 UTC 2016 (duration 5m 43s)
02:20 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.10) (duration: 08m 23s)

2016-07-17

10:31 godog: restart slapd on serpens - T130593
02:26 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Jul 17 02:26:21 UTC 2016 (duration 5m 42s)
02:20 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.10) (duration: 08m 24s)

2016-07-16

23:16 Krenair: testing that the SAL bot is still working
20:42 logmsgbot: twentyafterfour@tin Synchronized wmf-config/interwiki.php: (no message) (duration: 00m 26s)
20:41 twentyafterfour: deploying interwiki https config change https://gerrit.wikimedia.org/r/#/c/299299 refs T140206
20:32 awight: update paymentswiki from 25c97ba0f27b61859f90fd205c53d587c2838fec to 0c14940f4930e94a9287acae978cc6e661e54ee1
18:31 awight: enable LogCompleted for Ingenico
18:27 awight: update paymentswiki from 8bf6e911eb43a2d369bf656f07d1b51be0a54f6c to 25c97ba0f27b61859f90fd205c53d587c2838fec
02:27 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Jul 16 02:27:00 UTC 2016 (duration 5m 44s)
02:21 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.10) (duration: 08m 22s)

2016-07-15

21:12 awight: reenable donation queue
21:11 awight: update civicrm from cea316cc57c511c645a92a003028c95e19cac877 to a386eb5a76ec97b3b01c46a49309dfa39bbc58b0
20:32 awight: disable donations queue consumer
18:29 logmsgbot: ori@tin Synchronized wmf-config/InitialiseSettings.php: I8bf7c8dd: Lower default $wgSquidMaxage from 31 days to 14 days (duration: 00m 39s)
15:49 mobrovac: restbase deploy end of 731284b
15:37 mobrovac: restbase deploy start of 731284b
15:29 ottomata: restarting hadoop-mapreduce-historyserver to apply yarn log aggreation retention settings
13:40 godog: stress-test spinning disks on ms-be102[3-6]
12:08 bblack: varnish: rolling frontend restarts for text+upload done
11:56 bblack: varnish: starting rolling, depooled restart of text and upload frontend caches
11:03 godog: swift codfw-prod: ms-be202[567] weight 2500
10:30 twentyafterfour: deployed rPHABacb736547c6595fe09e05bafd7a3b563d3cf67c8 and rPHABcf12fdf248df82dc414d96bddd147c058bc3d636 to address maniphest task dependency graphs. Now related tasks will be shown as a plain list when there are too many tasks to graph.
10:15 mobrovac: restbase deploy end of 018864b
10:04 mobrovac: restbase deploy start of 018864b
09:33 jynus: renabling semisync replication throughout s4
09:31 jynus: restarted circular replication from db2019 -> db1040
09:22 jynus: updating dns record for s4-master.eqiad.wmnet
08:13 jynus: reimporting x1 partial db copy on dbstore1002 from x1-master
07:30 moritzm: installing PHP security updates on jessie systems
07:27 _joe_: powercycling mw1280
06:47 moritzm: installing nspr security updates
06:08 moritzm: installing libarchive security updates
02:35 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Jul 15 02:35:16 UTC 2016 (duration 6m 14s)
02:29 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.10) (duration: 07m 54s)

2016-07-14

23:11 logmsgbot: ebernhardson@tin Synchronized php-1.28.0-wmf.10/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T137169: Turn of TextCat A/B test (duration: 00m 34s)
21:08 matt_flaschen: Started backfillReadBundles.php on all group 2 wikis
21:07 matt_flaschen: Started backfillUnreadWikis.php --rebuild on all group 2 wikis
20:40 yurik: restarted graphoid with the new settings, enabling geoshape protocol
20:02 ottomata: restarting hadoop-yarn-resourcemanager on analytics1002 and then analytics1001 to apply yarn log aggregation change
19:56 logmsgbot: aude@tin Synchronized php-1.28.0-wmf.10/extensions/RevisionSlider: touching js and resource files (duration: 00m 28s)
19:22 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings-labs.php: maps geoshapes stuff for yurik (labs file for completeness) (duration: 00m 27s)
19:21 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: maps geoshapes stuff for yurik (duration: 00m 31s)
19:19 logmsgbot: demon@tin rebuilt wikiversions.php and synchronized wikiversions files: Move remaining wikis to wmf.10
19:00 urandom: Dropping legacy Cassandra system_auth tables in RESTBase production to complete RBAC conversion : T139639
15:52 logmsgbot: aude@tin Synchronized wmf-config/CommonSettings-labs.php: (no message) (duration: 00m 32s)
15:50 logmsgbot: aude@tin Synchronized dblists/clldefault.dblist: Enable compact language lists on more wikis (duration: 00m 51s)
15:40 jynus: shutting down es2018, pc2004, es2005 for hardware maintenance T139714
15:35 logmsgbot: aude@tin Finished scap: Update i18n for RevisionSlider (duration: 46m 58s)
15:08 jynus: shutting down es2014, es2015, es2016 for hardware maintenance T139714
14:56 bblack: cache_misc: manually raised default_ttl to 3600 (to match https://gerrit.wikimedia.org/r/#/c/298970/ without restarts)
14:48 logmsgbot: aude@tin Started scap: Update i18n for RevisionSlider
14:48 jynus: shutting down es2011, es2012, es2013 for hardware maintenance T139714
14:47 logmsgbot: aude@tin scap aborted: (no message) (duration: 00m 02s)
14:47 logmsgbot: aude@tin Started scap: (no message)
14:45 logmsgbot: aude@tin Synchronized wmf-config/CommonSettings.php: Enable RevisionSlider on test wikis (duration: 00m 28s)
14:44 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable RevisionSlider on test wikis (duration: 00m 27s)
14:38 logmsgbot: aude@tin Synchronized wmf-config/extension-list: Add RevisionSlider to extension-list (duration: 00m 42s)
09:58 godog: powercycle ms-be1012, adding back replaced disk
08:40 Amir1: for ores
08:40 Amir1: deploying 0e9555f to scb nodes
08:39 elukey: restarted hhvm on mw1289 mw1280 mw1288 mw1284 mw1287
08:18 jynus: running "megacli -PDOffline -PhysDrv '[32:6]' -aALL" on dbstore1002 to debug issue T140337
08:06 elukey: upgrading cache misc to varnishkafka 1.0.11-1
08:03 _joe_: removing appservers mw1018-25 from service via conftool for decommissioning (T139353)
08:01 elukey: removing api servers mw111[4-9] from service via conftool as first decom step (T139353)
07:55 elukey: removing api servers mw112[0-9] from service via conftool as first decom step (T139353)
06:45 moritzm: restarted hhvm on mw1170
06:07 moritzm: upgrading hhvm in codfw
04:22 twentyafterfour: Phabricator hotfix: applied patch to disable task graph on tasks with > 100 related tasks.
03:15 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Jul 14 03:14:57 UTC 2016 (duration 7m 16s)
03:07 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.10) (duration: 15m 50s)
02:37 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.8) (duration: 15m 17s)
00:36 matt_flaschen: Ran backfillReadBundles on labtestwiki
00:35 matt_flaschen: Started backfillReadBundles on labswiki
00:24 matt_flaschen: Started backfillUnreadWikis --rebuild and backfillReadBundles for all group 0 and group 1 wikis earlier
00:06 twentyafterfour: Phabricator maintenance completed. Service restored
00:01 twentyafterfour: preparing to take Phabricator offline momentarily for scheduled maintenance / upgrade. Service should be restored within a couple of minutes.

2016-07-13

23:31 eileen: update CiviCRM from 0898bb9360fe4a5ddea1a41d4e3f3e9823afee27 to cea316cc57c511c645a92a003028c95e19cac877
23:27 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/298822 (duration: 00m 26s)
23:25 logmsgbot: krenair@tin Synchronized static/images/project-logos: update for https://gerrit.wikimedia.org/r/298819 and https://gerrit.wikimedia.org/r/298822 (duration: 00m 24s)
23:18 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/225509 + https://gerrit.wikimedia.org/r/298899 - create https://meta.wikimedia.org/wiki/Special:Contact/stewards (duration: 00m 26s)
23:17 logmsgbot: ori@tin Synchronized php-1.28.0-wmf.10/includes/resourceloader/ResourceLoaderStartUpModule.php: I882bf7075: ResourceLoader: Update expected length of module version hash (duration: 00m 25s)
21:55 mutante: ytterbium - puppet enabled again, fix deployed
21:48 mutante: ytterbium, disabled puppet, started apache, needs fix
20:40 logmsgbot: demon@tin Synchronized README: no-op to bring co-masters in sync (duration: 00m 28s)
20:28 bearND: deployed mobileapps d1eb1da
20:20 logmsgbot: demon@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.10
20:19 bearND: starting mobileapps deploy
20:03 ebernhard|lunch: Update codfw elasticsearch cluster sttings with cluster.routing.allocation.disk.watermark.low: 70% to match eqiad and reduce free space icinga warnings
19:28 ostriches: gerrit: restarting, puppet back on, issue fixed.
19:23 logmsgbot: anomie@tin Synchronized php-1.28.0-wmf.10/includes/auth/AuthManager.php: Add timing data logging for T119736 (duration: 00m 28s)
19:23 logmsgbot: anomie@tin Synchronized php-1.28.0-wmf.8/includes/auth/AuthManager.php: Add timing data logging for T119736 (duration: 00m 27s)
19:14 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.8/includes/api/ApiQueryRecentChanges.php: API: Remove index forcing in ApiQueryRecentChanges - T140108 (duration: 00m 26s)
19:04 logmsgbot: demon@tin rebuilt wikiversions.php and synchronized wikiversions files: all group0 to wmf.10
18:41 ostriches: gerrit/ytterbium: flapped for a minute because of incompat 2.12/2.8 config. Working, puppet disabled pending real fix.
18:40 mutante: gerrit has a temp problem. maintenance going on
18:34 logmsgbot: krenair@tin Synchronized wmf-config: labs-only change, should be a noop here: https://gerrit.wikimedia.org/r/298812 (duration: 00m 27s)
18:32 mutante: gerrit will restart shortly for a config change. expect a very short downtime
18:22 legoktm: checkLocalUser.php finished, starting run #2 now
17:58 logmsgbot: legoktm@tin Synchronized wmf-config/: PoolCounterClient.php -> extension.json (duration: 00m 32s)
17:44 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.10/extensions/WikimediaEvents/WikimediaEventsHooks.php: Include the namespace for all pages & Include the resolved special page name for special pages - T138500 (duration: 00m 36s)
17:41 logmsgbot: demon@tin Finished scap: wmf.10 code sync + testwiki to wmf.10 for l10n cache gen (once more with feeling) (duration: 47m 06s)
17:35 ejegg: turned on cURL verbose logging for AstroPay requests on payments
17:27 jynus: drop databases fab_migration, percona and test from m3 T138460
17:25 _joe_: restarting hhvm on mw1229 (stuck in HPHP::Treadmill::getAgeOldestRequest)
17:22 urandom: Starting restbase on restbase1013.eqiad.wmnet
16:54 logmsgbot: demon@tin Started scap: wmf.10 code sync + testwiki to wmf.10 for l10n cache gen (once more with feeling)
16:50 ejegg: updated CentralNotice for cookie cleanup
16:48 logmsgbot: ejegg@tin Synchronized php-1.28.0-wmf.8/extensions/CentralNotice/: (no message) (duration: 01m 52s)
16:47 logmsgbot: demon@tin scap failed: OSError [Errno 1] Operation not permitted: '/var/lock/scap' (duration: 00m 00s)
16:41 logmsgbot: demon@tin scap aborted: wmf.10 code sync + testwiki to wmf.10 for l10n cache gen (duration: 00m 37s)
16:40 logmsgbot: demon@tin Started scap: wmf.10 code sync + testwiki to wmf.10 for l10n cache gen
16:39 logmsgbot: demon@tin Purged l10n cache for 1.28.0-wmf.9
16:38 logmsgbot: demon@tin Purged l10n cache for 1.28.0-wmf.7
16:38 logmsgbot: demon@tin Purged l10n cache for 1.28.0-wmf.6
16:37 logmsgbot: demon@tin Purged l10n cache for 1.28.0-wmf.5
16:19 hashar: CI slightly overloaded / backloaded due to a long tail of Wikibase changes sent in Gerrit.
16:03 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: [Beta] Change ORES thresholds in beta (duration: 00m 29s)
16:01 logmsgbot: thcipriani@tin Synchronized wmf-config/LabsServices.php: SWAT: [Beta] Parsoid: direct traffic to deployment-parsoid07 (duration: 00m 26s)
15:30 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ORES review tool for Turkish Wikipedia (T139992) (duration: 00m 28s)
15:11 urandom: Stopping Staging dumps : T139639
14:44 urandom: Starting offset dump runs from {xenon,cerium,praseodymium}.eqiad.wmnet : T139639
14:38 urandom: Restarting Cassandra on xenon.eqiad.wmnet : T139639
14:32 urandom: Restarting RESTBase on xenon.eqiad.wmnet : T139639
14:29 yurik: deployed kartotherian https://gerrit.wikimedia.org/r/#/c/298731/ & tilerator https://gerrit.wikimedia.org/r/#/c/298732/
14:25 moritzm: depooling mw1298 (image scaler) for some tests
14:21 godog: reboot ms-be1012, many mkfs.xfs stuck on broken sdh
14:17 logmsgbot: oblivian@palladium conftool action : delete; selector: cluster=rcstream
14:16 gehel: shutting down elastic1001-1016 (T139758)
14:16 logmsgbot: oblivian@palladium conftool action : delete; selector: cluster=rcstream
14:13 yurik: about to deploy updated kartotherian & tilerator for node 4.4.6
14:09 urandom: Dropping legacy system_auth tables in staging to complete RBAC conversion : T139639
14:05 bblack: rcstream cleanup done, puppet re-enabled on relevant lvs and rcs100x
13:58 gehel: cleanup puppet / salt from old elasticsearch servers elastic1001-1016 (T139758)
13:58 hashar: T137525 reverted Zuul back to zuul_2.1.0-95-g66c8e52-wmf1precise1_amd64.deb . It could not connect to Gerrit reliably
13:45 ottomata: restarting hadoop nodemanagers to apply log aggregation retention check interval change
13:43 bblack: restarting pybal on primary eqiad high-traffic2 (lvs1002)
13:41 moritzm: upgrading hhvm on remaining appservers in eqiad and codfw
13:34 gehel: disabling puppet and stopping elasticsearch on elastic1001-1016 (T139758)
13:30 bblack: disabling puppet on rcs100[12] for rcstream cleanup
13:29 bblack: disabling puppet on eqiad high-traffic2 lvs for rcstream cleanup
13:20 gehel: scheduling icinga downtime on elastic1001-1016 prior to decommissioning (T139758)
13:01 elukey: upgrading cache maps to varnishkafka 1.0.11-1
13:00 elukey: uploaded varnishkafka 1.0.11-1 to jessie-wikimedia experimental
12:53 hashar: CI is processing with Zuul 2.1.0-151-g30a433b. It might stop processing events at anytime though due to T137525
12:36 hashar: T137525 Upgrading Zuul 2.1.0-95-g66c8e52-wmf1precise1 ... zuul_2.1.0-151-g30a433b-wmf1precise1_amd64.deb
11:52 elukey: installing varnishkafka_1.0.11-1 on cp3008.esams to test it before the complete rollout
11:38 paravoid: cr1-ulsfo: "restart snmp" to fix SNMP hiccup after reboot
11:38 paravoid: cr1/2-ulsfo: disabling flow monitoring
04:22 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings.php: revert http logging change (duration: 00m 31s)
04:20 logmsgbot: legoktm@tin Synchronized wmf-config/interwiki.php: Update interwiki map, make them HTTPS (duration: 00m 39s)
03:44 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings.php: Log 'http' at warning level to debug transwiki import errors (duration: 00m 29s)
02:34 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Jul 13 02:34:52 UTC 2016 (duration 6m 8s)
02:28 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.8) (duration: 10m 11s)
00:56 ori: Restarted grrrit-wm
00:41 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.8/extensions/VisualEditor/: https://gerrit.wikimedia.org/r/#/c/298677/ (duration: 01m 01s)
00:07 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php: getCentralAuthToken visibility back to protected (Gerrit:298661) (duration: 00m 27s)

2016-07-12

23:43 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Add flow-create-board for gomwiki sysop (T139226) (duration: 00m 27s)
23:41 mutante: lithium deleted some logs older than 60 days to make space
23:33 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Enable lazy loaded references and images on Thai wikipedia (T136731) (duration: 00m 38s)
23:29 mutante: stat1003 still on every puppet run a mongodb gets started..over and over again
23:12 ejegg: updated payments from d9f7027340e5311f38c4224c2fddde087467df87 to 8bf6e911eb43a2d369bf656f07d1b51be0a54f6c
23:12 ostriches: lead: puppet disabled for a bit while index building is in progress.
23:12 ostriches: ytterbium: puppet enabled again, all happy again
22:56 ostriches: ytterbium: disabled puppet for a moment so we can do a config change w/o gerrit restarting itself
22:06 eileen: upgrade Civicrm from f7434730ebd87f6d542c34c080c61eb3f21ccc6b to 0898bb9360fe4a5ddea1a41d4e3f3e9823afee27
21:52 eileen: Updating CiviCRM from 415d7e62bc3bcbd7c5e3682da64ee4847ad63f5b to f7434730ebd87f6d542c34c080c61eb3f21ccc6b
21:24 logmsgbot: mattflaschen@tin Synchronized php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php: T140144: Echo/CentralAuth: Bail if not fully initialized (duration: 00m 49s)
20:18 urandom: Start revision culling script for local_group_wikipedia_T_parsoid_html, from restbase1009.eqiad.wmnet : T140008
19:30 logmsgbot: mattflaschen@tin Synchronized php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php: T119736: T140144: Troubleshoot why Echo is still triggering CA failures (duration: 00m 39s)
19:02 mutante: git pulled on strontium to sync with palladium
18:48 matt_flaschen: Started checkLocalUser.php at ~2016-07-12 17:45 UTC, killed ~18:06 since Echo apparently is not fully fixed after all.
18:38 legoktm: foreachwiki ../../../../home/legoktm/checkLocalUser.php --delete=1 --verbose=1 on terbium
18:24 logmsgbot: anomie@tin Synchronized php-1.28.0-wmf.9/includes/auth/AuthManager.php: Commit transaction after auto-creating a user gerrit:298541 (duration: 00m 29s)
18:22 logmsgbot: anomie@tin Synchronized php-1.28.0-wmf.8/includes/auth/AuthManager.php: Commit transaction after auto-creating a user gerrit:298540 (duration: 00m 30s)
18:08 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: turn cx back on (duration: 00m 29s)
17:44 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.8/extensions/ContentTranslation/: ping limiter fixes (duration: 00m 29s)
17:37 jynus: out of band ALTER TABLE recentchanges ADD KEY `name_type_patrolled_timestamp` on db1054 T140108
16:58 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: prep pinglimiter config for content translation (duration: 00m 33s)
16:56 Amir1: deploying ores f472f65 to scb
16:51 Amir1: deploying ores f472f65 to scb2001
16:42 godog: disable puppet on ms-fe* and re-enable gradually to apply https://gerrit.wikimedia.org/r/#/c/298297/
16:25 logmsgbot: mattflaschen@tin Synchronized php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php: T119736: ForeignWikiRequest: Bail early for non-global users (duration: 00m 32s)
16:12 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: Disable content translation, outage right now (duration: 00m 29s)
15:44 bblack: cache nodes: salt manual removal of vm compaction cron via sed ( https://gerrit.wikimedia.org/r/298499 )
15:33 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.9/extensions/Echo/includes/ForeignWikiRequest.php: SWAT: ForeignWikiRequest: Bail early for non-global users (T119736) (duration: 00m 31s)
15:29 logmsgbot: thcipriani@tin Synchronized portals: SWAT: Bumping portals to master (T128546) (duration: 00m 29s)
15:28 logmsgbot: thcipriani@tin Synchronized portals/prod/wikipedia.org/assets: SWAT: Bumping portals to master (T128546) (duration: 00m 29s)
15:20 bblack: upgrading nginx to 1.11.2-1+wmf1 on all caches
15:18 logmsgbot: thcipriani@tin Synchronized static/images/project-logos/trwikimedia.png: SWAT: Revert Logo update for trwikimedia (T140015) (duration: 00m 29s)
15:09 logmsgbot: thcipriani@tin Synchronized static/images/project-logos/trwikimedia.png: SWAT: Logo update for trwikimedia (T140015) (duration: 00m 33s)
14:58 bblack: upgrading nginx to 1.11.2-1+wmf1 on cache_maps
14:33 elukey: Rebuild new AQS Cassandra cluster (aqs100[456]) to remove previous testing settings (no prod traffic is served)
14:23 logmsgbot: reedy@tin Synchronized wmf-config/extension-list: even more extension.json (duration: 00m 26s)
14:17 bblack: nginx 1.11.2-1+wmf1 uploaded to carbon
14:17 logmsgbot: reedy@tin Synchronized wmf-config/extension-list: moar extension.json (duration: 00m 26s)
14:04 bblack: lvs nodes: apt-get install linux-meta
13:52 bblack: lvs nodes: apt-get upgrade to latest (various base system packages)
13:49 bblack: cache nodes: apt-get upgrade to latest (just 3.16 kernel)
13:05 ottomata: restarting nodemanagers on analytics 1039 1046 and 1054
10:45 godog: terbium:~# lvextend --size +70G -r /dev/mapper/terbium--vg-root T139786
09:35 gehel: lowering elasticsearch codfw high watermark to rebalance cluster
09:32 godog: reboot ms-be3004 / high load average and xfs unhappy
09:22 godog: progressively delete esams swift containers, unused and not in production
07:49 elukey: removing api servers mw113[0-9] from service via conftool as first decom step (T139353)
06:31 logmsgbot: legoktm@tin Synchronized wmf-config/CommonSettings.php: Don't block logins if CentralAuthUser::queryAttached() fails - T119736 (duration: 00m 27s)
06:00 legoktm: running checkLocalUsers.php on terbium
04:37 ori: Reverted all wikis to wmf8 due to tenfold increase in T119736
04:35 logmsgbot: ori@tin rebuilt wikiversions.php and synchronized wikiversions files: (no message)
02:26 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Jul 12 02:26:41 UTC 2016 (duration 5m 31s)
02:21 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.9) (duration: 08m 44s)

2016-07-11

23:49 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.9/resources/: https://gerrit.wikimedia.org/r/#/c/298402/ (duration: 00m 28s)
23:36 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.9/extensions/Echo/: https://gerrit.wikimedia.org/r/#/c/298400/ (duration: 00m 33s)
23:32 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.9/extensions/Citoid/: https://gerrit.wikimedia.org/r/#/c/298327/ (duration: 00m 27s)
23:30 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.9/extensions/Wikidata: https://gerrit.wikimedia.org/r/#/c/298386/ (duration: 02m 00s)
23:15 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/297964/ (duration: 00m 32s)
23:11 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/297556/ (duration: 00m 28s)
23:07 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.9/extensions/Kartographer/: https://gerrit.wikimedia.org/r/#/c/297556/ (duration: 00m 29s)
22:46 logmsgbot: krenair@tin Synchronized wmf-config: more labs-only changes: https://gerrit.wikimedia.org/r/298393 (duration: 00m 36s)
21:50 logmsgbot: krenair@tin Synchronized wmf-config: sync labs-only change, should be a noop here: https://gerrit.wikimedia.org/r/#/c/298299/ (duration: 00m 39s)
21:40 awight: resuming fundraising donations queue consumer
21:37 mutante: ytterbium - graceful'ed Apache, warning about duplicate NameVirtual host is gone
21:03 bd808: Updated default mapping for logstash-* index creation using json generated by https://gerrit.wikimedia.org/r/#/c/298295/. Should take effect starting with the logstash-2016.07.12 index.
20:44 logmsgbot: reedy@tin Synchronized wmf-config/extension-list-labs: nooop for prod (duration: 00m 32s)
20:31 ottomata: rolling restart of hadoop-yarn-nodemanager to apply log aggregation retention seconds
20:29 mdholloway: mobileapps deployed df16702
20:25 mdholloway: starting mobileapps deployment
20:20 awight: disabled fundraising donation queue consumer...
20:09 subbu: finished deploying parsoid sha e738c415
20:05 subbu: synced new parsoid code; restarted parsoid on wtp1001 as a canary
20:03 subbu: starting parsoid deploy
18:37 chasemp: new hd for failed array in labstore2001
18:23 ejegg: updated payments from 2fc573cbb94e833c4144aa9dad79de8ec374bb09 to d9f7027340e5311f38c4224c2fddde087467df87
18:08 mutante: welcome new mediawiki deployer Brian Wolff (T138635)
17:44 ejegg: updated CiviCRM from f477a42014dd1e6759849b347d5f73d710954d0b to bf029eecb9bfb49d267e60d76344b0170bfa0a83
17:09 awight: reenable fundraising campaigns
17:07 gehel: starting deployment of latest WDQS (second time deploying with scap3)
16:52 twentyafterfour: unclog the phabricator task queue (phd) by cherry-picking upstream fix 12c6f87ca to wmf/stable (+restarted phd on iridium)
15:47 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Update logo settings for the Nepali Wikipedia (T139240) PART II (duration: 00m 26s)
15:46 logmsgbot: thcipriani@tin Synchronized static/images/project-logos: SWAT: Update logo settings for the Nepali Wikipedia (T139240) PART I (duration: 00m 27s)
15:43 elukey: restarted hhvm on mw1170 (Apache errors while reading FCGI headers, HHVM dump debug in /tmp/hhvm.14968.bt.)
15:40 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Allow to import from zh.wikipedia to beta.wikiversity (T139922) (duration: 00m 26s)
15:37 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: State Compact Language Links is not beta anymore (T136677) (duration: 00m 26s)
15:33 logmsgbot: thcipriani@tin Synchronized wmf-config/interwiki.php: SWAT: Update interwiki map (duration: 00m 28s)
15:25 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: HD version for sqwikiquote logos (T139229) PART II (duration: 00m 27s)
15:25 logmsgbot: thcipriani@tin Synchronized static/images/project-logos: SWAT: HD version for sqwikiquote logos (T139229) PART I (duration: 00m 25s)
15:19 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add contentdm.lib.byu.edu to wgCopyUploadsDomains (T139095) (duration: 00m 26s)
15:13 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: SWAT: Remove old throttle rules (duration: 00m 30s)
15:06 elukey: removing mw114[0-8] from service via conftool as first decom step (T139353)
15:06 logmsgbot: thcipriani@tin Synchronized static/images/project-logos/sqwikiquote.png: SWAT: Change Albanian Wikiquote logo (T139229) (duration: 00m 34s)
14:46 ema: upgrading cache_misc to varnish 4.1.3-1wm1
14:39 _joe_: shutting down mw1090-1113,mw1149-51 for decommissioning
14:05 moritzm: rebooting achernar for kernel update
13:31 moritzm: rebooting acamar for kernel update
13:07 ema: upgrading esams cache_maps to varnish 4.1.3-1wm1
12:51 gehel: upgrading nodejs to 4.4.6 on maps2.* servers
12:24 ema: upgrading ulsfo cache_maps to varnish 4.1.3-1wm1
12:14 elukey: restarted hhvm on mw1261
12:03 ema: upgrading codfw cache_maps to varnish 4.1.3-1wm1
11:30 ema: upgrading eqiad cache_maps to varnish 4.1.3-1wm1
11:19 moritzm: installing hhvm updates on canary app servers
10:25 hashar: CI: upgraded Chromium from v49 to v51 (v50 caused qunit jobs to fail / timeout randomly) T136188
10:00 godog: swift codfw-prod: ms-be202[567] weight 2000
09:47 moritzm: installing GCC stable updates on trusty systems (also provides some runtime libs in addition to GCC itself)
09:40 ema: upgrading cp1046 to varnish 4.1.3-1wm1
07:29 mobrovac: change-prop deploying 2b699a6
07:26 mobrovac: graphoid deploying 375d31fd
07:24 mobrovac: mathoid deploying 669cfc0
07:19 mobrovac: cxserver deploying fd8eca47e
07:15 mobrovac: citoid deploying 274c0231d
07:13 mobrovac: mobileapps deploying 6e409f46
06:34 _joe_: restarted hhvm on mw1168
06:13 moritzm: restarted saltmaster on neodymium
02:26 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Jul 11 02:26:55 UTC 2016 (duration 5m 41s)
02:21 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.9) (duration: 08m 41s)

2016-07-10

02:25 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Jul 10 02:25:55 UTC 2016 (duration 5m 43s)
02:20 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.9) (duration: 08m 34s)

2016-07-09

19:54 bd808: restarted logstash on logstash1003 for de-dot plugin update (T136001)
19:52 bd808: restarted logstash on logstash1002 for de-dot plugin update (T136001)
19:50 bd808: restarted logstash on logstash1001 for de-dot plugin update (T136001)
19:46 bd808: Updated logstash/plugins to 18b3f1f (Fix de_dot to process keys with falsey values)
02:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Jul 9 02:29:53 UTC 2016 (duration 6m 7s)
02:23 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.9) (duration: 08m 21s)
00:28 mutante: analytics1027 - icinga said puppet fail, just ran it, recovery, same on neon.. something kafka graphite checks
00:06 mutante: gerrit is back
00:05 ostriches: gerrit: disabled puppet for a minute so I can unbreak gerrit so I can fix gerrit in puppet.

2016-07-08

23:57 mutante: behold, gerrit might restart now for config change
23:52 Amir1: manually restarting uwsgi-ores in scb1001
23:48 logmsgbot: krinkle@tin Synchronized docroot/foundation/: Remove unused docroot/foundation/index.html (duration: 00m 30s)
23:32 Amir1: manually restarting uwsgi-ores on scb1002
22:59 urandom: Restarting restbase1015-b.eqiad.wmnet to cancel running streams : T139362
22:26 eileen: from bdf2afd417b70332c9542fd3ee4f14cb4e6f93cc to f477a42014dd1e6759849b347d5f73d710954d0b
22:13 urandom: Restarting restbase1014-b.eqiad.wmnet to cancel running streams : T139362
22:02 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.9/extensions/CentralAuth/: Fix job serializing (and status display on Special:GlobalRenameProgress) - T137973 (duration: 00m 32s)
21:59 logmsgbot: reedy@tin Synchronized wmf-config/extension-list: Fix RestBaseUpdateJobs in extension-list (duration: 00m 36s)
21:51 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: Use Canonical entry point for RestBaseUpdateJobs (duration: 00m 33s)
21:50 logmsgbot: reedy@tin Synchronized multiversion/MWWikiversions.php: Remove some else if spaces (duration: 00m 46s)
21:48 urandom: Restarting restbase1009-b.eqiad.wmnet to cancel running streams : T139362
21:31 elukey: mw1146 powercycled (memory pressure, no ssh/root login)
21:17 urandom: Restarting restbase1015-a.eqiad.wmnet to cancel running streams : T139362
21:11 urandom: Restarting restbase1014-a.eqiad.wmnet to cancel running streams : T139362
21:04 urandom: Restarting restbase1009-a.eqiad.wmnet to cancel running streams : T139362
20:59 urandom: Forcing node removal (restbase1014-c.eqiad.wmnet) : T139362
20:51 urandom: Throttle RESTBase Cassandra outgoing streams to 1mbit cluster-wide : T139362 (actually happened at 21:26)
20:17 bd808: Deleted old l10nupdate caches manually on tin (T130317)
19:56 urandom: Throttle RESTBase Cassandra outgoing streams to 3mbit cluster-wide : T139362
18:52 anomie: Attempting to resubmit LocalRenameUserJobs for T137973
18:48 urandom: "This is going to hurt me more than it does you."; `nodetool removenode' of restbase1014-c.eqiad.wmnet : T139362
18:39 urandom: Stopping restbase1014-c.eqiad.wmnet : T139362
18:29 mutante: db1042 - temp stop puppet, edit ferm rules to allow testing from lead
14:43 jynus: rechecking data consistency after m3 table fixes (could cause lag)
12:21 moritzm: installing glib updates from jessie point release
12:03 jynus: stopping replication on db1043 (m3-slave) for maintenance
09:33 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: aqs1002.eqiad.wmnet
09:30 logmsgbot: elukey@palladium conftool action : set/pooled=no; selector: aqs1002.eqiad.wmnet
09:30 elukey: upgrading nodejs packages on aqs100[23]
09:10 logmsgbot: elukey@palladium conftool action : set/pooled=yes:weight=5; selector: mw1261.eqiad.wmnet
09:06 logmsgbot: elukey@palladium conftool action : set/pooled=no:weight=5; selector: mw1261.eqiad.wmnet
09:03 logmsgbot: elukey@palladium conftool action : set/pooled=yes:weight=5; selector: mw1261.eqiad.wmnet
09:01 moritzm: rebooting ruthenium for update to Linux 4.4
08:40 hashar: gallium: deleting old log files /var/log/zuul/gearman-server-debug.log*
08:35 hashar: gallium: restarting Zuul to apply logging configuration change https://gerrit.wikimedia.org/r/#/c/291913/
08:18 moritzm: rebooting terbium for kernel security update
08:12 moritzm: rearming keyholder on tin
08:08 moritzm: rebooting tin for kernel security update
08:08 logmsgbot: oblivian@palladium conftool action : set/pooled=no; selector: name=mw1261.eqiad.wmnet
07:15 _joe_: removing 20 gb logfile from terbium, only useless debug info
02:37 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Jul 8 02:37:29 UTC 2016 (duration 6m 22s)
02:31 logmsgbot: dereckson@tin scap sync-l10n completed (1.28.0-wmf.9) (duration: 07m 47s)
02:12 Dereckson: scap pull on terbium (was out of disk space during previous full scap)
02:00 logmsgbot: l10nupdate@tin LocalisationUpdate failed: git pull of extensions failed
01:54 mutante: terbium ran out of disk, deleted rotated nutcracker log
01:48 logmsgbot: dereckson@tin Finished scap: Flow 297914, Echo 297919 297934, ORES 297916 (duration: 27m 48s)
01:26 ostriches: gerrit: readded robots.txt to ytterbium for now
01:20 logmsgbot: dereckson@tin Started scap: Flow 297914, Echo 297919 297934, ORES 297916
00:42 awight: roll back paymentswiki further, to 2fc573cbb94e833c4144aa9dad79de8ec374bb09
00:39 awight: roll back paymentswiki from f54ffb4fad0dc18079a813fbe25813dba36c64aa to c33ddfccf945bd075f0abff9e9de8c09f0174f89
00:27 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.9/includes/api/ApiParse.php: API: Generate head items in the context of the given title (T139565) (duration: 00m 30s)
00:01 awight: CentralNotice campaigns reenabled
00:00 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Enable ORES review tool as a beta feature in ptwiki (T139692) (duration: 00m 28s)

2016-07-07

23:57 Dereckson: Ran extensions/ORES/maintenance/CheckModelVersions.php and extensions/ORES/maintenance/PopulateDatabase.php on ptwiki (T139692)
23:55 awight: Update paymentswiki from c33ddfccf945bd075f0abff9e9de8c09f0174f89 to f54ffb4fad0dc18079a813fbe25813dba36c64aa
23:50 Dereckson: Created table ores_classification on ptwiki from php-1.28.0-wmf.9/extensions/ORES/sql/ores_classification.sql
23:48 Dereckson: Created table ores_model on ptwiki from php-1.28.0-wmf.9/extensions/ORES/sql/ores_model.sql
23:36 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Remove unused deprecated $wgStyleSheetPath (Gerrit:297511) (duration: 00m 27s)
23:23 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: VisualEditor: Move cite out of primary toolbar except on WP/WB/WV (Gerrit:296573) (duration: 00m 30s)
22:33 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.9/extensions/VisualEditor/: https://gerrit.wikimedia.org/r/#/c/297795/ and https://gerrit.wikimedia.org/r/#/c/297908/ (duration: 00m 45s)
21:32 urandom: Bootstrapping restbase1009-c : T139362
20:59 urandom: Fin : T126629
20:52 urandom: Restarting Cassandra instance restbase2009-b.codfw.wmnet : T126629
20:50 urandom: Restarting Cassandra instance restbase2009-a.codfw.wmnet : T126629
20:49 urandom: Upgrading Cassandra to 2.2.6-wmf1 on restbase2009.codfw.wmnet : T126629
20:48 urandom: Cassandra upgrade of restbase2006.codfw.wmnet instances complete : T126629
20:45 urandom: Restarting RESTBase on restbase2001.codfw.wmnet
20:41 urandom: Restarting Cassandra instance restbase2006-b.codfw.wmnet : T126629
20:38 urandom: Restarting Cassandra instance restbase2006-a.codfw.wmnet : T126629
20:36 urandom: Upgrading Cassandra to 2.2.6-wmf1 on restbase2006.codfw.wmnet : T126629
20:33 urandom: Cassandra upgrade of restbase2005.codfw.wmnet instances complete : T126629
20:30 urandom: Restarting Cassandra instance restbase2005-b.codfw.wmnet : T126629
20:27 urandom: Restarting Cassandra instance restbase2005-a.codfw.wmnet : T126629
20:27 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.9
20:26 twentyafterfour: deploying wmf.9 to all wikis refs T138555
20:26 urandom: Upgrading Cassandra to 2.2.6-wmf1 on restbase2005.codfw.wmnet : T126629
20:15 gehel: logstash upgrade aborted, rescheduled to Monday July 11th
20:10 bd808: Restarted logstash on logstash1001; hoping to clear up missing de-dot errors
20:07 bd808: Restarted logstash on logstash1002; hoping to clear up missing de-dot errors
20:07 urandom: Cassandra upgrade of restbase2008.codfw.wmnet instances complete : T126629
20:04 urandom: Restarting Cassandra instance restbase2008-b.codfw.wmnet : T126629
20:02 urandom: Restarting Cassandra instance restbase2008-a.codfw.wmnet : T126629
20:02 logmsgbot: mattflaschen@tin Synchronized php-1.28.0-wmf.9/extensions/Echo: Fixes for notification sorting and message parsing (duration: 00m 38s)
20:01 urandom: Upgrading Cassandra to 2.2.6-wmf1 on restbase2008.codfw.wmnet : T126629
20:01 urandom: Cassandra upgrade of restbase2004.codfw.wmnet instances complete : T126629
19:59 logmsgbot: mattflaschen@tin Synchronized php-1.28.0-wmf.9/extensions/Flow/includes/Notifications/PostReplyPresentationModel.php: flow-post-reply: show compact header on one line (duration: 00m 32s)
19:59 urandom: Restarting Cassandra instance restbase2004-b.codfw.wmnet : T126629
19:57 bd808: Restarted logstash on logstash1003; hoping to clear up missing de-dot errors
19:55 urandom: Restarting Cassandra instance restbase2004-a.codfw.wmnet : T126629
19:55 urandom: Upgrading Cassandra to 2.2.6-wmf1 on restbase2004.codfw.wmnet : T126629
19:53 urandom: Cassandra upgrade of restbase2003.codfw.wmnet instances complete : T126629
19:50 urandom: Restarting Cassandra instance restbase2003-b.codfw.wmnet : T126629
19:46 urandom: Restarting Cassandra instance restbase2003-a.codfw.wmnet : T126629
19:45 urandom: Upgrading Cassandra to 2.2.6-wmf1 on restbase2003.codfw.wmnet : T126629
19:32 urandom: Upgrade of restbase2007.codfw.wmnet instances complete : T126629
19:29 urandom: Restarting Cassandra instance restbase2007-c.codfw.wmnet : T126629
19:27 urandom: Restarting Cassandra instance restbase2007-b.codfw.wmnet : T126629
19:24 urandom: Restarting Cassandra instance restbase2007-a.codfw.wmnet : T126629
19:22 urandom: Upgrading Cassandra to 2.2.6-wmf1 on restbase2007.codfw.wmnet : T126629
19:19 urandom: Upgrade of restbase2002.codfw.wmnet instances complete : T126629
19:16 urandom: Restarting Cassandra instance restbase2002-c.codfw.wmnet : T126629
19:13 urandom: Restarting Cassandra instance restbase2002-b.codfw.wmnet : T126629
19:12 ostriches: gerrit: force all users to log out. sorry ❤️
19:11 urandom: Restarting Cassandra instance restbase2002-a.codfw.wmnet : T126629
19:09 urandom: Upgrading Cassandra to 2.2.6-wmf1 on restbase2002.codfw.wmnet : T126629
19:08 urandom: Cassandra upgrade of restbase2001.codfw.wmnet instances complete : T126629
19:05 urandom: Restarting Cassandra instance restbase2001-c.codfw.wmnet : T126629
19:02 urandom: Restarting Cassandra instance restbase2001-b.codfw.wmnet : T126629
18:56 urandom: Restarting Cassandra instance restbase2001-a.codfw.wmnet : T126629
18:53 urandom: Upgrading Cassandra to 2.2.6-wmf1 on restbase2001.codfw.wmnet : T126629
18:27 urandom: Disabling Puppet on RESTBase codfw nodes : T126629
18:23 awight: Update SmashPig from 917138e159f0341e3dfbb35818c3ce479927875b to e6aa6fe6fdcaab8e961a8b0668cc742d4c443c46
18:15 urandom: Cassandra 2.2.6 upgrade of restbase1015.eqiad.wmnet instances complete : T126629
18:11 urandom: Restarting Cassandra instance restbase1015-b.eqiad.wmnet : T126629
18:08 urandom: Restarting Cassandra instance restbase1015-a.eqiad.wmnet : T126629
18:07 urandom: Upgrading Cassandra to 2.2.6-wmf1 on restbase1015.eqiad.wmnet : T126629
17:12 ostriches: gerrit: flush all caches to pick up account disable & rename
16:55 urandom: Restarting Cassandra instance restbase1014-c.eqiad.wmnet : T126629
16:54 awight: update paymentwiki from 2fc573cbb94e833c4144aa9dad79de8ec374bb09 to c33ddfccf945bd075f0abff9e9de8c09f0174f89
16:48 urandom: Restarting Cassandra instance restbase1014-b.eqiad.wmnet : T126629
16:47 jynus: stopping pc2006 for hardware maintenance T139283
16:39 urandom: Restarting Cassandra instance restbase1014-a.eqiad.wmnet : T126629
16:38 urandom: Upgrading Cassandra to 2.2.6-wmf1 on restbase1014.eqiad.wmnet : T126629
16:29 urandom: Disabling Puppet on restbase101[4-5].eqiad.wmnet : T126629
16:24 gehel: starting elasticsearch and kibana upgrade on logstash cluster (T136001)
16:21 awight: Taking Fundraising campaigns down for maintenance
15:51 elukey: add mw1261 back into service
15:44 bd808: Dropped logstash indices older than logstash-2016.07.01 in preparation for elasticsearch upgrade
15:25 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.9/extensions/Echo/modules/controller/mw.echo.Controller.js: SWAT: Correct section (alert/message/all) (duration: 00m 25s)
15:14 _joe_: uploaded new HHVM package for jessie
15:11 logmsgbot: thcipriani@tin Synchronized dblists/clldefault.dblist: SWAT: Deploy Compact Language Links as default (Stage 4) (T136677) PART II (duration: 00m 34s)
15:11 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Deploy Compact Language Links as default (Stage 4) (T136677) PART I (duration: 00m 55s)
14:43 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: aqs1001.eqiad.wmnet
14:38 logmsgbot: elukey@palladium conftool action : set/pooled=no; selector: aqs1001.eqiad.wmnet
14:37 elukey: depool aqs1001 for nodejs upgrade
12:05 elukey: depooling mw1261 from service (T73487)
11:34 jynus: breaking m3 replication on db1048 (depooled) to check icinga alert changes
10:44 jynus: disabling all mysql lag alerts cross-fleet T122457
10:29 akosiaris: reboot fermium.wikimedia.org hassium.eqiad.wmnet install1001.wikimedia.org krypton.eqiad.wmnet meitnerium.wikimedia.org mendelevium.eqiad.wmnet T134242
10:22 akosiaris: reboot dubnium T134242
10:22 akosiaris: reboot bromine T134242
10:13 akosiaris: reboot bohrium T134242
10:11 elukey: pooling mw1261 back to service with Apache mod-proxy-fcgi set to trace8 (T73487)
10:07 akosiaris: reboot etherpad1001.eqiad.wmnet, kernel upgrade and qemu upgrade, T134242
10:04 gehel: rolling restart of elasticsearch cluster eqiad completed (T138811)
08:51 _joe_: removing all old servers from the appservers pool but the canaries (T139353)
08:47 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Failover s4 recentchanges to db1056 (duration: 00m 38s)
08:33 hoo: Updated Wikidata's property suggester with data from Monday's json dump and removed the external identifiers as a workaround for T132839
07:56 gehel: rolling restart of elasticsearch cluster codfw completed (T138811)
07:16 legoktm: mysql:wikiadmin@db1041 [centralauth]> delete from localuser where lu_name ="Philippe" and lu_wiki ="scnwiki";
04:37 eileen: Update CiviCRM from dd24368a897fd78752178ee253e7a890dd57db41 to bdf2afd417b70332c9542fd3ee4f14cb4e6f93cc
03:12 eileen: CiviCRM upgrade from 5f8f7c3236e6bf12c52deea07093fbca165ef4a7 to dd24368a897fd78752178ee253e7a890dd57db41
03:08 chasemp: silence labvirt1011 flapping for 24h, we have a task, and we are attempting to move vms we can
02:48 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Jul 7 02:48:55 UTC 2016 (duration 6m 44s)
02:42 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.9) (duration: 07m 07s)
02:26 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.8) (duration: 09m 58s)
02:14 chasemp: reboot labvirt1011
01:53 chasemp: start nova-compute again as it seems ot have been killed on labvirt1011 which is acting weird
01:24 andrewbogott: rebooting labvirt1011 (because it is acting crazy)
00:50 eileen: updating CiviCRM from 54e168db2fddc6a9a07036323e01a27dd64333cf to 5f8f7c3236e6bf12c52deea07093fbca165ef4a7
00:23 eileen: Updated CiviCRM from bb9bf136dc0fa82d5d07ebeb33d696e54672b2d6 to 54e168db2fddc6a9a07036323e01a27dd64333cf
00:06 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.8/extensions/CentralAuth/: Make LocalRename jobs run sequentially - T137973 (for real this time) (duration: 00m 30s)
00:05 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.9/extensions/CentralAuth/: Make LocalRename jobs run sequentially - T137973 (duration: 00m 30s)
00:03 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.8/extensions/CentralAuth/: Make LocalRename jobs run sequentially - T137973 (duration: 00m 34s)

2016-07-06

23:56 legoktm: created pageassesments tables on testwiki
23:52 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.9/includes/specials/SpecialContributions.php: Add mediawiki.special.changeslist to SpecialContributions - T139522 (duration: 00m 25s)
23:50 legoktm: running extensions/ORES/maintenance/CheckModelVersions.php and extensions/ORES/maintenance/PopulateDatabase.php on ruwiki
23:49 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings.php: Enable ORES review tool as a beta feature in ruwiki - T139541 (duration: 00m 27s)
23:47 legoktm: created ores_* tables on ruwiki
23:41 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.9/extensions/Echo/: T139321, T139323 (duration: 00m 32s)
23:33 logmsgbot: legoktm@tin Synchronized wmf-config: VisualEditor: Move the citation button out of the primary toolbar on Wikivoyes - T133725 (2/2) (duration: 00m 30s)
23:32 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings.php: VisualEditor: Move the citation button out of the primary toolbar on Wikivoyes - T133725 (1/2) (duration: 00m 26s)
23:30 logmsgbot: legoktm@tin Synchronized wmf-config: touch (duration: 00m 32s)
23:25 logmsgbot: legoktm@tin Synchronized wmf-config: Test PageAssessments extension on test.wikipedia.org - T137918 (duration: 00m 36s)
22:18 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.9/includes/diff: T139526 (duration: 00m 38s)
20:44 urandom: Restarting Cassandra instance restbase1008-b.eqiad.wmnet : T126629
20:41 bearND: deployed mobileapps 7a73789
20:38 bearND: starting mobileapps deploy
20:38 urandom: Restarting Cassandra instance restbase1008-a.eqiad.wmnet : T126629
20:23 urandom: Disable Puppet on restbase1009.eqiad.wmnet : T126629
20:13 urandom: Upgrade of restbase1013.eqiad.wmnet instances complete : T126629
20:06 urandom: Restarting Cassandra instance restbase1013-b.eqiad.wmnet : T126629
20:00 urandom: Restarting Cassandra instance restbase1013-a.eqiad.wmnet : T126629
19:58 twentyafterfour: deployed 1.28.0-wmf.9 to group1 wikis: T138555
19:57 urandom: Upgrading Cassandra package to 2.2.6-wmf1 on restbase1013.eqiad.wmnet : T126629
19:57 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.9
19:30 urandom: Upgrade of restbase1012.eqiad.wmnet instances complete : T126629
19:24 urandom: Restarting Cassandra instance restbase1012-c.eqiad.wmnet : T126629
19:18 urandom: Restarting Cassandra instance restbase1012-b.eqiad.wmnet : T126629
19:09 urandom: Restarting Cassandra instance restbase1012-a.eqiad.wmnet : T126629
19:07 urandom: Upgrading Cassandra package to 2.2.6-wmf1 on restbase1012.eqiad.wmnet : T126629
18:57 mutante: mw1133 puppet fail because out of memory, stop hhvm, run puppet
18:55 urandom: Cassandra upgrade of restbase1008.eqiad.wmnet instances complete : T126629
18:54 mutante: labstore1001 - failed backup
18:54 mutante: mw1261 syntax error in Apache config
18:52 urandom: Restarting Cassandra instance restbase1008-c : T126629
18:51 mutante: labstore2001 - RAID failure in Icinga (is it T102626 ?)
18:47 urandom: Restarting Cassandra instance restbase1008-b : T126629
18:46 mutante: mw1261 restart hhvm service
18:42 urandom: Restarting Cassandra instance restbase1008-a : T126629
18:40 urandom: Upgrading Cassandra package to 2.2.6-wmf1 on restbase1008 : T126629
17:14 urandom: Disabling Puppet on restbase{1008,1012,1013}.eqiad.wmnet in preparation for rack 'b' Cassandra upgrade : T126629
16:54 urandom: Upgrade of restbase1011.eqiad.wmnet instances to Cassandra 2.2.6 complete : T126629
16:50 urandom: Restarting Cassandra for restbase1011-c.eqiad.wmnet : T126629
16:45 urandom: Restarting Cassandra for restbase1011-b.eqiad.wmnet : T126629
16:41 urandom: Restarting Cassandra fro restbase1011-a.eqiad.wmnet : T126629
16:39 urandom: Upgrading Cassandra package to 2.2.6-wmf1 on restbase1011.eqiad.wmnet : T126629
16:26 awight: update orphan rectifier from 2fc573cbb94e833c4144aa9dad79de8ec374bb09 to 70a7baa9f77c2510739bab0ff9d1b51578a59a6e
16:25 urandom: Upgrade of restbase1010.eqiad.wmnet instances complete : T126629
16:25 awight: update orphan rectifier config to add payments 4 to the Redis pool
16:22 urandom: Restarting Cassandra for restbase1010-c.eqiad.wmnet : T126629
15:59 urandom: Restarting Cassandra for restbase1010-b.eqiad.wmnet : T126629
15:47 urandom: Restarting Cassandra for restbase1010-a.eqiad.wmnet : T126629
15:45 urandom: Re-enabling Puppet on restbase1010 : T126629
15:44 urandom: Upgrading Cassandra to 2.2.6-wmf1 on restbase1010 : T126629
15:36 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.9/extensions/Echo/modules/styles: SWAT: Set width to Special:Notifications (T138433) (duration: 00m 30s)
15:29 elukey: restarting the hdfs datanode on each analytics* Hadoop server to force the new -Xmx2048 heap setting to be picked up
15:27 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Echo transition flags everywhere (duration: 00m 26s)
15:22 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable autopatrolled user group at urwiki (T139302) (duration: 00m 30s)
15:18 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.9/includes/diff/DifferenceEngine.php: Show parser output for diffs unless extension aborts (T139433) (duration: 00m 30s)
15:08 urandom: Disabling puppet on restbase101[0-1].eqiad.wmnet in preparation for 2.2.6 upgrade : T126629
15:02 akosiaris: poweroff prometheus2002 from dbrb -> plain conversion
15:00 _joe_: depooling mw1261, installing an apache package with additional fixes (T73487)
14:19 moritzm: rebooting californium for kernel update (hosting horizon.wikimedia.org)
14:02 moritzm: rebooting silver (hosting wikitech)
13:48 gehel: re-enabling puppet on ^(aqs|restbase).* after confirming that Cassandra puppet module change is a noop
13:38 gehel: disabling puppet on ^(aqs|restbase).* before merging changes to Cassandra puppet module
13:35 elukey: depooling mw1261.eqiad to restore previous fcgi logging settings (T73487)
13:17 dcausse: restarting elastic master node (elastic1040)
13:15 dcausse: truncating elastic main logs on elastic1040 and elastic1034
12:19 mobrovac: restbase deploy end of fa4699a
12:09 mobrovac: restbase deploy start of fa4699a
11:54 elukey: depooling mw1261.eqiad.wmnet to raise Apache's mod-fcgi to trace8 for 503 investigation - T73487 (this will probably slow down a bit the host)
11:15 jynus: shutting down db1048 in preparation for upgrade
10:14 moritzm: upgrading restbase cluster in codfw for nodejs 4.4.6
09:39 _joe_: restarting hhvm on mw1236,mw1215 to test for possible TC cache corruption
09:31 moritzm: installing tomcat security updates on Ubuntu systems (jessie already fixed)
09:13 logmsgbot: elukey@palladium conftool action : set/pooled=yes:weight=25; selector: mw1285.eqiad.wmnet
09:10 logmsgbot: elukey@palladium conftool action : set/pooled=yes:weight=25; selector: mw1284.eqiad.wmnet
09:09 elukey: pooling into service the last batch of new API appservers - mw1284->mw1290
08:46 godog: lithium:~$ sudo lvextend --size +50G -t -r /dev/mapper/lithium--vg-syslog
08:32 godog: powercycle ms-be2021, unreachable and nothing on console
08:11 jynus: stopping replication on db1056 and performing alter table
07:58 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Fail commons db servers back to its original configuration (duration: 00m 45s)
07:57 moritzm: rolling reboot of sca clusters for kernel update
07:51 moritzm: restarted hhvm on mw1148
03:09 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Jul 6 03:09:17 UTC 2016 (duration 6m 53s)
03:02 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.9) (duration: 17m 48s)
02:25 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.8) (duration: 09m 16s)
00:23 logmsgbot: maxsem@tin Synchronized wmf-config: No-op (duration: 00m 37s)

2016-07-05

23:57 MaxSem: ran ORES's CheckModelVersions.php and PopulateDatabase.php on nlwiki
23:48 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/297462/ (duration: 00m 29s)
23:42 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.8/extensions/MobileFrontend/: https://gerrit.wikimedia.org/r/#/c/297542/ (duration: 00m 33s)
23:35 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/297542/ (duration: 00m 30s)
23:33 MaxSem: created ORES tables on nlwiki
23:26 chasemp: reboot of labvirt1011 which is being flaky I cannot keep a connection to
23:22 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/296852/ (duration: 00m 53s)
22:07 hasharAway: CI had issue booting instances since 19:50UTC it is operational again as of 21:30 UTC and slowly processing the backlog.
21:34 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: 1.28.0-wmf.9 to group0
20:56 logmsgbot: twentyafterfour@tin Finished scap: Rebuild l10n cache and deploy 1.28.0-wmf.9 to testwiki (duration: 32m 16s)
20:24 logmsgbot: twentyafterfour@tin Started scap: Rebuild l10n cache and deploy 1.28.0-wmf.9 to testwiki
20:01 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: lttoolbox_3.3.3~r68466-2+wmf1
20:01 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: hfst_3.10.0~r2798-1+wmf1
20:01 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: hfst-ospell_0.4.0~r4643-5+wmf1
20:01 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: foma_0.9.18+r243-1+wmf1
20:01 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: cg3_0.9.9~r11624-1+wmf1
20:01 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium_3.4.2~r68466-2+wmf1
20:01 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-lex-tools_0.1.1~r66150-1+wmf1
20:01 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-apy_0.9.1~r343-1
18:54 chasemp: add dpatrick to wmf-nda
18:32 Danny_B: killed wb2-phab @ tools.wikibugs to perform some batch edits
18:30 mutante: added siddharth11 to LDAP group "wmf" per T138369#2425011
16:44 jynus: rebooting pc2006 T139283
16:20 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: mw1024.eqiad.wmnet
16:13 logmsgbot: elukey@palladium conftool action : set/pooled=no; selector: mw1024.eqiad.wmnet
16:12 elukey: depooling mw1024 to restore regular fcgi logging settings
16:09 urandom: Bootstrapping restbase1009-c.eqiad.wmnet : T139362
15:49 akosiaris: reenable alerts from smokeping on codfw
15:46 andrewbogott: rebooting labservices1002 to see if it survives the reboot better this time
15:32 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: touch initialisesettings to clear logspam, hopefully (duration: 00m 27s)
15:20 elukey: powercycled mw1140, memory saturated and not reachable via ssh/mgmt-console
15:20 paravoid: mr1-codfw: "request system snapshot media internal slice alternate" + "request system reboot"
15:08 yurik: depl & restarted tilerator https://gerrit.wikimedia.org/r/#/c/297410/
14:42 logmsgbot: elukey@palladium conftool action : set/pooled=yes:weight=25; selector: mw1283.eqiad.wmnet
14:41 logmsgbot: elukey@palladium conftool action : set/pooled=yes:weight=25; selector: mw1282.eqiad.wmnet
14:37 logmsgbot: elukey@palladium conftool action : set/pooled=yes:weight=25; selector: mw1281.eqiad.wmnet
14:26 urandom: Issuing `nodetool cleanup' for restbase1014-b.eqiad.wmnet
14:12 logmsgbot: elukey@palladium conftool action : set/pooled=yes:weight=25; selector: mw1280.eqiad.wmnet
14:09 logmsgbot: elukey@palladium conftool action : set/pooled=yes:weight=25; selector: mw1279.eqiad.wmnet
14:07 logmsgbot: oblivian@palladium conftool action : set/pooled=inactive; selector: dc=eqiad,cluster=appserver,service=apache2,name=mw1217.*
14:07 elukey: Pooling first batch of new eqiad api-servers - mw1279->mw1283
14:03 _joe_: depooling permanently mw1091-13 from the appservers pool in eqiad
13:58 logmsgbot: elukey@palladium conftool action : set/weight=20; selector: mw2245.codfw.wmnet
13:58 logmsgbot: elukey@palladium conftool action : set/weight=20; selector: mw2244.codfw.wmnet
13:58 logmsgbot: elukey@palladium conftool action : set/weight=20; selector: mw2243.codfw.wmnet
13:58 logmsgbot: elukey@palladium conftool action : set/weight=20; selector: mw2242.codfw.wmnet
13:57 logmsgbot: elukey@palladium conftool action : set/weight=20; selector: mw2241.codfw.wmnet
13:57 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: mw2245.codfw.wmnet
13:57 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: mw2244.codfw.wmnet
13:57 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: mw2243.codfw.wmnet
13:56 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: mw2242.codfw.wmnet
13:56 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: mw2241.codfw.wmnet
13:56 elukey: pooling new codfw appservers - mw224[12345]
12:32 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: mw1024.eqiad.wmnet
12:12 logmsgbot: elukey@palladium conftool action : set/pooled=no; selector: mw1024.eqiad.wmnet
12:11 elukey: depooling/re-pooling mw1024.eqiad.wmnet to temporarily set up trace8 logging (503 investigation - T73487)
12:08 jynus: running schema change on db1019 T73563
11:15 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Failover all commons special roles to db1081 (duration: 00m 24s)
11:00 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Failover commons recentachanges (duration: 00m 36s)
10:45 jynus: SET GLOBAL read_only=0; on db1040, our new m4-master
10:38 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Failover commons master to db1040 (duration: 00m 32s)
10:23 jynus: archiving m3-master phlegal* databases before dropping them
10:21 mobrovac: restbase staging started a no-op dump on cerium to test restbase on node 4.4.6
10:05 logmsgbot: elukey@palladium conftool action : set/weight=30; selector: mw1275.eqiad.wmnet
10:05 logmsgbot: elukey@palladium conftool action : set/weight=30; selector: mw1274.eqiad.wmnet
10:05 logmsgbot: elukey@palladium conftool action : set/weight=30; selector: mw1273.eqiad.wmnet
09:59 logmsgbot: elukey@palladium conftool action : set/weight=30; selector: mw1272.eqiad.wmnet
09:31 _joe_: shutting down mw1009-16 for decommissioning
09:06 _joe_: decommissioning mw1009-16
08:38 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: mw1275.eqiad.wmnet
08:36 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: mw1274.eqiad.wmnet
08:32 gehel: deleting enwikisource_titlesuggest on elasticsearch codfw (index creation issue during cluster restart)
08:31 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: mw1273.eqiad.wmnet
08:24 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: mw1272.eqiad.wmnet
08:21 elukey: adding and pooling new appservers - mw127[2345].eqiad
08:07 godog: swift codfw-prod: ms-be202[567] weight 1500
07:55 jynus: dropping etherpad_restore2 database from m1 T138516
07:40 akosiaris: T138516 forcing a puppet run on cache::misc hosts after merging https://gerrit.wikimedia.org/r/297352
07:29 akosiaris: T138516 stop the secondary etherpad instance on etherpad1001. etherpad-restore.wikimedia.org has served its purpose, killing it
02:44 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Jul 5 02:44:09 UTC 2016 (duration 6m 12s)
02:38 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.8) (duration: 17m 13s)

2016-07-04

20:28 jynus: removing /tmp/joal/sstables on all analytics10* hosts
20:22 jynus: deleted 21GB worth of temporary files from analytics1050
19:58 logmsgbot: aaron@tin Synchronized wmf-config/filebackend-production.php: Increase redis lockmanager timeout to 2 (duration: 00m 31s)
19:57 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.8/extensions/MassMessage/: MassMessage is no longer accepting lists in the MassMessageList content model - T139303 (duration: 00m 39s)
17:37 jynus: testing slave_parallel_threads=5 on db1073
14:27 moritzm: rebooting lithium for kernel update
14:22 moritzm: installing tomcat7/ libservlet3.0-java security update on the kafka brokers
14:06 _joe_: shutting down mw1001-1008 for decommissioning
14:03 gehel: rolling restart of elasticsearch codfw/eqiad for kernel upgrade (T138811)
13:47 _joe_: stopping jobrunner on mw1011-16 as well, befor decommissioning
13:46 moritzm: depooling mw1153-mw1160 (trusty image scalers), replaced by mw1291-mw1298 (jessie image scalers)
13:44 godog: ack all mr1-codfw related alerts in librenms
13:43 akosiaris: restart smokeping on netmon1001, temporarily disabled msw1-codfw
13:38 gehel: resuming writes on Cirrus / elasticsearch, this did not speedup cluster recovery
13:18 godog: bounce redis on rcs1001
13:16 gehel: restarting elastic1021 for kernel upgrade (T138811)
13:07 elukey: Bootstrapping again Cassandra on aqs100[456] (rack awareness + 2.2.6 - testing environment)
13:02 gehel: pausing writes on Cirrus / elasticsearch for faster cluster restart
12:43 hashar: Nodepool back up with 10 instances (instead of 20) to accomodate for labs capacity T139285
12:39 godog: nodetool-b stop -- COMPACTION on restbase1014
12:29 moritzm: rolling reboot of rcs* cluster for kernel security update
12:10 moritzm: rolling reboot of ocg* cluster for kernel security update
11:40 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Failover db1053 to db1072 (duration: 00m 40s)
10:56 moritzm: rolling reboot of swift frontends in eqiad for kernel security update
10:30 yuvipanda: stop nodepool on labnodepool1001 and disable puppet to keep it down, to allow stabilizing labs first
10:28 yuvipanda: restart rabbitmq-server on labcontrol1001
10:14 moritzm: installing chromium security update on osmium
10:07 moritzm: installing xerces-c security updates on Ubuntu systems (jessie already fixed)
10:01 _joe_: stopping jobchron and jobrunner on mw1001-10 before decommission
09:50 godog: reimage ms-be300[234] with jessie
09:44 hashar: Labs infra cant delete instances anymore (impacts CI as well) T139285
09:41 moritzm: installing p7zip security updates
09:38 hashar: CI is out of Nodepool instances, the pool has drained because instances can no more be deleted over the OpenStack API
09:25 elukey: Added new jobrunners in service - mw130[256].eqiad.wmnet (https://etherpad.wikimedia.org/p/jessie-install)
08:16 moritzm: rolling reboot of swift backends in eqiad for kernel security update
07:49 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Failover db1034 to db1062 (duration: 00m 30s)
02:26 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Jul 4 02:26:54 UTC 2016 (duration 5m 42s)
02:21 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.8) (duration: 09m 14s)

2016-07-03

19:27 Reedy: Ran namespaceDupes --fix on gomwiki
14:59 yuvipanda: restart nova-compute process on labvirt1010
14:59 yuvipanda: restart nova-compute process on labvirt10101
09:06 jynus: removing old logs from pc2004
07:42 logmsgbot: legoktm@tin Synchronized static/images/project-logos/: Put high-res enwiktionary logos in the right place - T139255 (duration: 00m 38s)
02:27 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Jul 3 02:27:13 UTC 2016 (duration 5m 38s)
02:21 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.8) (duration: 09m 13s)

2016-07-02

19:15 twentyafterfour: Deployed hotfix to phabricator. Restarted apache2 on iridium
02:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Jul 2 02:29:17 UTC 2016 (duration 5m 40s)
02:23 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.8) (duration: 08m 52s)

2016-07-01

22:23 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.8/extensions/WikimediaEvents/extension.json: T128115 (duration: 00m 37s)
22:22 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.8/extensions/WikimediaEvents/modules/: T128115 (duration: 00m 30s)
21:04 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I7a95c0f4: Bump $wgResourceLoaderMaxQueryLength to 5,000 (duration: 00m 32s)
20:08 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I6eb0ae67: Bump $wgResourceLoaderMaxQueryLength to 4,000 (duration: 00m 26s)
19:17 ori: restarted coal on graphite1001 stopped receiving messages from EL 0mq publisher
19:16 ori: restarted navtiming on hafnium; stopped receiving messages from EL 0mq publisher
18:34 mutante: mw1259 - powercycling
18:32 logmsgbot: krinkle@tin Synchronized docroot/default/: (no message) (duration: 00m 31s)
18:31 logmsgbot: krinkle@tin Synchronized errorpages/: (no message) (duration: 01m 06s)
17:47 ebernhardson: restart elasticsearch on elastic1017 to attempt to clear up a continuous backlog of relocating shards
15:53 godog: temporarily run 3x statsdlb instances on graphite1001 to minimise drops - T101141
14:57 dcausse: upgraded and restarted elastic on nobelium@eqiad
14:23 godog: enable another statsdlb instance temporarily on graphite1001 to investigate drops
14:15 moritzm: rearmed keyholder on mira after reboot
13:56 moritzm: rebooting codfw poolcounters for kernel update
13:47 moritzm: rebooting osmium for kernel update
13:28 cmjohnson1: mw1145 swapped eth0 cable
13:04 moritzm: rebooting mira for kernel update
12:59 moritzm: rebooting francium for kernel update
11:23 godog: bounce statsdlb on graphite1001, drops are back after yesterday's reboot T101141
11:15 moritzm: removed two obsolete, older kernel packages from wtp1002 (had flagged an icinga warning on diskspace on /boot)
09:38 elukey: rebooted eventlog2001.codfw.wmnet for kernel upgrades
09:35 moritzm: rolling reboot of swift backends in codfw
09:15 moritzm: powercycling ms-fe2003, stuck after reboot
09:03 moritzm: powercycling ms-fe2002, stuck after reboot
08:41 moritzm: powercycling ms-fe2001, stuck after reboot
08:32 moritzm: rolling reboot of swift frontends in codfw
06:28 moritzm: resuming rolling reboots of elastic* clusters in eqiad and codfw
06:18 moritzm: rolling reboot of wtp1* for kernel security update
02:44 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Jul 1 02:44:01 UTC 2016 (duration 5m 7s)
02:38 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.8) (duration: 17m 02s)
01:59 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I3a8057a8: Bump $wgResourceLoaderMaxQueryLength to 3,000 (duration: 00m 28s)
01:46 logmsgbot: aaron@tin Synchronized wmf-config/InitialiseSettings.php: Enable LocalFile log (duration: 00m 32s)
01:08 logmsgbot: ori@tin Synchronized wmf-config/InitialiseSettings.php: Ie8a71af5: HD logo for en.wiktionary (2/2) (duration: 00m 27s)
01:07 logmsgbot: ori@tin Synchronized static/images/project-logos: Ie8a71af5: HD logo for en.wiktionary (1/2) (duration: 00m 28s)
01:04 logmsgbot: ori@tin Synchronized php-1.28.0-wmf.8/extensions/WikimediaEvents/modules/ext.wikimediaEvents.deprecate.js: Ie28d823c: Log ResourceLoader URL-splitting (duration: 00m 32s)
00:20 logmsgbot: maxsem@tin Finished scap: https://gerrit.wikimedia.org/r/#/c/296819/ - noop in prod (duration: 27m 27s)

2016-06-30

23:53 logmsgbot: maxsem@tin Started scap: https://gerrit.wikimedia.org/r/#/c/296819/ - noop in prod
23:47 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/296257/ (duration: 00m 26s)
23:43 logmsgbot: maxsem@tin Synchronized portals: (no message) (duration: 00m 27s)
23:43 logmsgbot: maxsem@tin Synchronized portals/prod/wikipedia.org/assets: (no message) (duration: 00m 28s)
23:41 ori: banned /static/images/project-logos/enwiktionary.png and /static/images/project-logos/adywiki.png
23:37 logmsgbot: maxsem@tin Synchronized docroot/mediawiki/xml/sitelist-1.0/index.html: https://gerrit.wikimedia.org/r/#/c/296788/ (duration: 00m 24s)
23:27 logmsgbot: maxsem@tin Synchronized static/images/project-logos/: (no message) (duration: 00m 27s)
23:12 logmsgbot: maxsem@tin Synchronized static/images/project-logos/enwiktionary.png: https://gerrit.wikimedia.org/r/#/c/296757/ (duration: 00m 30s)
22:04 logmsgbot: krinkle@tin Synchronized wmf-config/InitialiseSettings.php: test2wiki wgSquidMaxage (duration: 00m 28s)
21:00 ebernhardson: change cluster.routing.allocation.disk.watermark.high on eqiad elasticsearch cluster to 80%
20:52 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.8/includes/filerepo/file/LocalFile.php: 51d7fb48f2af31d69db115a8b3ed790cdaaf0d2e (duration: 00m 35s)
19:47 logmsgbot: aaron@tin Synchronized wmf-config/InitialiseSettings.php: Set the SaveParse log (duration: 00m 26s)
19:43 twentyafterfour: ran scap pull on mw2123
19:42 twentyafterfour: ran scap pull on mw2098
19:42 gehel: activating statement timeout limitations for kartotherian on maps cluster codfw (T138422)
19:38 ostriches: mw2134: running sync-common, seems out of...sync :)
19:20 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: (no message)
19:19 twentyafterfour: Deploying 1.28.0-wmf.8 to all wikimedia wikis.
19:18 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.8/includes: adc4c90202d6c44aa58756e3c6bc35918afc5f75 (duration: 01m 19s)
18:24 ori: restarted coal on graphite1001 and navtiming on hafnium due to inexplicably stopped metrics; nothing useful in logs.
17:48 yurik: deployed Kartotherian https://gerrit.wikimedia.org/r/#/c/296787/
17:10 yurik: deployed Graphoid https://gerrit.wikimedia.org/r/#/c/296780/
17:08 jynus: stopping slave on db1073 to test InnoDB compression T139055
16:44 dcausse: restarting elastic1036 (master in eqiad)
16:23 logmsgbot: thcipriani@tin Synchronized wmf-config: SWAT:Revert "Use extension registration for TitleBlacklist" (duration: 00m 32s)
16:22 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT:Revert "Use extension registration for TitleBlacklist" (duration: 00m 27s)
16:17 dcausse: truncating elastic logs on elastic1036 and elastic1021
16:16 logmsgbot: thcipriani@tin Synchronized wmf-config: SWAT: Use extension registration for TitleBlacklist (T119117) PART II (duration: 00m 36s)
16:15 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Use extension registration for TitleBlacklist (T119117) PART I (duration: 00m 39s)
15:53 logmsgbot: thcipriani@tin Synchronized wmf-config: SWAT: Use extension registration for LabeledSectionTransclusion (T119117) (duration: 00m 27s)
15:48 logmsgbot: thcipriani@tin Synchronized wmf-config: SWAT: Short array syntax (duration: 00m 30s)
15:39 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: SWAT: Put wikidatawiki back on 1.28.0-wmf.8
15:36 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.7/extensions/Wikidata/extensions/Wikibase/repo/includes/WikibaseRepo.php: SWAT: Update Wikidata - Fix broken editing of statements (T138974) (duration: 00m 25s)
15:35 akosiaris: restarted (actually puppet did) gerrit after merging 4 related changes
15:34 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.8/extensions/Wikidata/extensions/Wikibase/repo/includes/WikibaseRepo.php: SWAT: Update Wikidata - Fix broken editing of statements (T138974) (duration: 00m 31s)
15:27 logmsgbot: thcipriani@tin Synchronized dblists/clldefault.dblist: SWAT: Deploy Compact Language Links as default (Stage 3.5) (T136677) (duration: 00m 26s)
15:18 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: SWAT: Delete old throttle rules (duration: 00m 25s)
15:12 logmsgbot: thcipriani@tin Synchronized static/images/project-logos/enwiktionary.png: SWAT: Revert Change project logo for enwikt (T138801) (duration: 00m 25s)
15:07 logmsgbot: thcipriani@tin Synchronized static/images/project-logos/enwiktionary.png: SWAT: Change project logo for enwikt (T138801) (duration: 00m 25s)
15:04 moritzm: rolling reboot of wtp2 for kernel security update
14:49 mdholloway: mobileapps finished deploying 43538aa
14:46 moritzm: pooling four additional jessie-based image scalers (mw1295-mw1298)
14:45 mdholloway: starting mobileapps deployment
13:42 godog: swift codfw-prod: ms-be202[234] weight 3000
13:10 jynus: upgrading and restarting analytics1003 mysql tables
11:59 moritzm: pooling three additional jessie-based image scalers
10:26 elukey: rebooting stat100[234] and analytics1003 for kernel upgrades
10:20 moritzm: powercycling mw1016, stuck after reboot
09:44 dcausse: truncating current logs again on elastic1045 and elastic1036
09:37 moritzm: powercycling mw1011, stuck after reboot
09:21 dcausse: truncating current logs on elastic1045 and elastic1036
09:19 mobrovac: zotero deployed translators cde2f7531a4
09:13 godog: reboot graphite2001 / graphite1001 to apply trusty kernel update
09:13 dcausse: deleting old logs on elastic1045 and elastic1036
06:57 moritzm: powercycling elastic1015, stuck after reboot
06:37 moritzm: powercycling elastic1014, stuck after reboot
06:26 moritzm: resuming rolling restarts of elasticsearch cluster in eqiad and codfw
06:18 moritzm: rolling restart of mw1001-mw1016 for kernel secuity update
03:00 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Jun 30 03:00:28 UTC 2016 (duration 7m 10s)
02:53 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.8) (duration: 07m 08s)
02:39 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.7) (duration: 17m 23s)
01:33 twentyafterfour: starting phd with only 4 taskmasters to help lighten the load
01:30 twentyafterfour: stopped phd on iridium to investigate large spike in sql insert volume
01:18 mutante: iridium back up, on 3.13.0-91
01:15 mutante: rebooting iridium
00:39 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Don't use always true wmgUseXFFBlocks anymore (2/2) (duration: 00m 25s)
00:38 twentyafterfour: Phabricator upgrade complete, service appears to be stable.
00:32 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Don't use always true wmgUseXFFBlocks anymore (1/2) (duration: 00m 25s)
00:30 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Don't use always true wmgUseXFFBlocks anymore (1/2) (duration: 00m 27s)
00:27 twentyafterfour: Taking phabricator offline momentarily for scheduled update. Expect less than 5 minutes of downtime.
00:25 logmsgbot: maxsem@tin Synchronized wmf-config/: Try again? (duration: 00m 29s)
00:17 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Revert "Cleanup: Move never-altered GlobalBlockingBlockXFF into CommonSettings" (no-op) (duration: 00m 26s)
00:15 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Revert "Cleanup: Move never-altered GlobalBlockingBlockXFF into CommonSettings" (duration: 00m 25s)
00:10 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Clean-up for IS/CS (Gerrit:292615 to Gerrit:292618, no op, 2/2) (duration: 00m 29s)
00:09 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Clean-up for IS/CS (Gerrit:292615 to Gerrit:292618, no op, 1/2) (duration: 00m 28s)
00:08 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.8/extensions/TemplateSandbox: https://gerrit.wikimedia.org/r/#/c/296675/ (duration: 00m 30s)
00:08 logmsgbot: dereckson@tin scap aborted: wmf-config/CommonSettings.php Clean-up for IS/CS (Gerrit:292615 to Gerrit:292618, no op, 1/2) (duration: 00m 20s)
00:07 logmsgbot: dereckson@tin Started scap: wmf-config/CommonSettings.php Clean-up for IS/CS (Gerrit:292615 to Gerrit:292618, no op, 1/2)

2016-06-29

23:21 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Revert "Enable Echo transition flags in production for testing" (duration: 00m 25s)
22:50 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: extdist config for 1.27/1.25 (duration: 00m 31s)
21:43 logmsgbot: krenair@tin Synchronized php-1.28.0-wmf.8/extensions/VisualEditor/ApiVisualEditor.php: https://gerrit.wikimedia.org/r/296661 - VE namespaces issue (duration: 00m 26s)
21:41 chasemp: cleared phab 2fa for ebernhardson for lost phone
21:34 jynus: removing /srv/backups/m2-otrs-* (tranferred to es2001) to make space
21:02 yurik: deployed Graphoid https://gerrit.wikimedia.org/r/#/c/296498/
20:55 yurik: deployed Tilerator https://gerrit.wikimedia.org/r/#/c/296647/
20:50 yurik: deployed Kartotherian https://gerrit.wikimedia.org/r/#/c/296646/
20:21 bearND: mobileapps deployed 1da6bf0
20:15 bearND: starting mobileapps deploy
19:54 logmsgbot: maxsem@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/296627/ (duration: 00m 31s)
19:39 mutante: antimony - shutdown -h now (since it's gone from Icinga now)
19:38 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: testwikidata back to wmf.8
19:36 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: Roll back wikidata and testwikidata to 1.28.0-wmf.7 per request by @aude
19:22 mutante: antimony puppetstoredconfigclean.rb to remove icinga monitor remnants
19:14 ostriches: ytterbium: running puppet and reloading replication plugin
19:13 mutante: antimony - stopping gitblit service
19:07 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.8T138555
18:39 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.7/maintenance/dumpBackup.php: Deploy I94ca4a06 (duration: 00m 25s)
18:39 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.7/maintenance/backup.inc: Deploy I94ca4a06 (duration: 00m 24s)
18:37 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.7/includes/export/WikiExporter.php: Deploy I94ca4a06 (duration: 00m 25s)
18:30 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.8/maintenance/dumpBackup.php: Deploy I94ca4a06 (duration: 00m 27s)
18:29 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.8/maintenance/backup.inc: Deploy I94ca4a06 (duration: 00m 33s)
18:28 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.8/includes/export/WikiExporter.php: Deploy I94ca4a06 (duration: 00m 34s)
18:06 mutante: we stopped using gitblit. git.wikimedia.org URLs P3318 T137224
18:05 mutante: git.wm.org URLs switched from gitblit to phab redirects
17:48 ostriches: gerrit: flushed all caches to pick up rename, things may be slow for the next 15m or so
15:49 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: SWAT: Throttling exemption for enwiki (T138167) (duration: 00m 25s)
15:38 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.7/resources/Resources.php: SWAT: mediawiki.action.edit.stash: Restore dependency to "jquery.getAttrs" (T138931) (duration: 00m 26s)
15:34 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.7/extensions/Flow/maintenance/FlowRemoveOldTopics.php: SWAT: Also delete topics that have more recent updates by (only) talk page manager (T119509) (duration: 00m 25s)
15:29 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.7/extensions/Flow: SWAT: Do not reimport existing header (T119509) (duration: 00m 46s)
15:22 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.7/extensions/Flow/maintenance/FlowRestoreLQT.php: SWAT: Script to restore LQT topics to their pre-import state (T119509) (duration: 00m 26s)
15:11 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Echo transition flags in production for testing (duration: 00m 27s)
15:09 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Add $wmgEchoTransition setting for Echo transition flags PART II (duration: 00m 26s)
15:08 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add $wmgEchoTransition setting for Echo transition flags PART I (duration: 00m 50s)
14:46 moritzm: powercycling elastic1012, stuck on reboot
13:22 elukey: rebooting analytics1027 for kernel upgrades
12:58 moritzm: rebooting dataset1001 for kernel update
12:41 moritzm: continuing rolling restarts of elastic* in eqiad and codfw for kernel security update
12:32 moritzm: powercycling elastic1010, stuck on reboot
12:11 moritzm: powercycling mw1260, stuck on reboot
11:38 jynus: halfway moving otrs backups from dbstore1001 to es2001
11:27 gehel: powercycling elastic1009 - stuck in reboot
11:11 moritzm: powercycling mw1223, stuck on reboot
11:09 gehel: deleting broken dewiki_titlesuggest index from codfw (T138811)
10:31 elukey: rebooting analytics100[12] (Hadoop Yarn/HDFS master and standby) - One at the time forcing failover manually with daemon restarts
09:55 moritzm: powercycling mw1163, stuck on reboot
09:23 gehel: banning elastic1001 to 1016 from cluster to prepare their decommissioning (T138329)
09:20 ema: upgrading diamond to 3.5-6 (T138758)
09:01 elukey: rebooting analytics1028->1057 for kernel upgrades (Hadoop worker nodes)
08:55 moritzm: powercycling mw1111, stuck on reboot
08:44 elukey: puppet stopped on analytics1027 to prevent Camus job to run (prep step for Hadoop kernel upgrades)
08:40 moritzm: powercycling mw1108, stuck on reboot
08:12 moritzm: powercycling mw1099, stuck on reboot
08:12 moritzm: powercycling mw1097, stuck on reboot
08:05 moritzm: powercycling mw1092, stuck on reboot
07:47 moritzm: rolling reboot of appservers in eqiad for kernel security update
07:16 moritzm: powercycling snapshot1002, reboot stuck
07:11 moritzm: powercycling snapshot1001, reboot stuck
06:58 moritzm: rebooting most snapshot hosts for kernel security update
03:28 logmsgbot: krinkle@tin Synchronized wmf-config/InitialiseSettings.php: test2wiki (duration: 00m 33s)
02:56 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Jun 29 02:56:32 UTC 2016 (duration 6m 30s)
02:50 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.8) (duration: 04m 48s)
02:30 chasemp: labstore1004 is replicating NFS/DRBD shares to labstore1005 and they are large and it's taking a long time
02:29 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.7) (duration: 09m 21s)
02:18 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: sync wikiversions.json - group0 to 1.28.0-wmf.8 refs T137492
02:16 twentyafterfour: promoting group0 to 1.28.0-wmf.8
00:02 logmsgbot: twentyafterfour@tin Finished scap: sync new branch, testwiki to php-1.28.0-wmf.8 refs T137492 (duration: 51m 59s)

2016-06-28

23:10 logmsgbot: twentyafterfour@tin Started scap: sync new branch, testwiki to php-1.28.0-wmf.8 refs T137492
23:10 Krenair: wikitech-static working now, poke me on IRC or file a #wikitech.wikimedia.org ticket if you find any issues
23:10 twentyafterfour: syncing new branch 1.28.0-wmf.8 refs T137492
23:04 logmsgbot: ebernhardson@tin Synchronized php-1.28.0-wmf.7/extensions/EventBus/EventBus.php: SWAT: EventBus: Match the expected format of response log key (duration: 00m 31s)
23:01 Krenair: Updating MW version on wikitech-static to 1.27 (LTS) - https://lists.wikimedia.org/pipermail/mediawiki-announce/2016-June/000191.html
21:59 halfak: deploying ores beec291
21:33 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.7
21:31 logmsgbot: twentyafterfour@tin Synchronized php-1.28.0-wmf.7/extensions/AbuseFilter/: deploy https://gerrit.wikimedia.org/r/#/c/296464/ refs T138550 T136973 (duration: 00m 36s)
21:24 twentyafterfour: deploying wmf.7 yet again, once CI finishes testing https://gerrit.wikimedia.org/r/#/c/296464/ refs T138550 T136973
20:24 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: once again rolling back to wmf.6 refs T136973 T138550
20:11 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.7
20:09 logmsgbot: twentyafterfour@tin Synchronized php-1.28.0-wmf.7/extensions/AbuseFilter/: deploying https://gerrit.wikimedia.org/r/#/c/296440/ refs T138550, T136973 (duration: 02m 06s)
20:09 twentyafterfour: deploying https://gerrit.wikimedia.org/r/#/c/296440/ to hopefully unblock wmf.7 deployments. refs T138550, T136973
20:08 gehel: disabling puppet on wdqs100[12] to cleanup after failed scap3 deplyoment
19:33 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: Rolling back to wmf.6: save time regression is still present in wmf.7
19:32 twentyafterfour: Rolling back to wmf.6: T138550 is still a problem
19:24 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.7
19:23 twentyafterfour: Deploying 1.28.0-wmf.7 to all wikis
18:23 mutante: zosma - fresh install, sign puppet certs, initial puppet run
16:16 gehel: starting rolling restart of elasticsearch codfw cluster (T138811)
15:25 logmsgbot: thcipriani@tin Synchronized portals: SWAT: Bumping portals to master (T136874) (duration: 00m 29s)
15:24 logmsgbot: thcipriani@tin Synchronized portals/prod/wikipedia.org/assets: SWAT: Bumping portals to master (T136874) (duration: 00m 24s)
15:16 logmsgbot: thcipriani@tin Synchronized dblists/visualeditor-default.dblist: SWAT: Enable VisualEditor by default for all users of the French (T136993), English (T136992), and German (T136991) Wikivoyage (duration: 00m 24s)
15:09 logmsgbot: thcipriani@tin Synchronized dblists/visualeditor-default.dblist: SWAT: Enable VisualEditor by default for all users of the Italian Wikivoyage (T136994) (duration: 00m 25s)
14:52 gehel: powercycling elastic1004 (server not coming up during restart - T138811)
13:47 godog: bounce carbon on graphite machines after applying https://gerrit.wikimedia.org/r/266567
13:40 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: aqs1001.eqiad.wmnet
12:50 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1023, 24, 33, 35, 39, 44, 52, 61, 64, 68, 63, 67, 72, 73 (duration: 02m 39s)
12:00 gehel: powercycling elastic1002 (server not coming up during restart - T138811)
11:43 gehel: powercycling elastic1001 (server not coming up during restart - T138811)
11:21 gehel: rolling restart of elasticsearch eqiad
10:44 moritzm: rolling reboot of mediawiki in codfw for kernel security update
09:39 moritzm: powercycling mw1021, didn't come up after reboot
09:32 elukey: restarted hhvm on mw1238, memory pressure ok but hhvm stuck (hhvm-dump-debug in /tmp/hhvm.14788.bt.)
09:28 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: aqs1003.eqiad.wmnet
09:25 moritzm: powercycling mw1019, didn't come up after reboot
09:25 logmsgbot: reedy@tin Synchronized wmf-config/interwiki.php: Updated IW map (duration: 00m 49s)
09:13 logmsgbot: elukey@palladium conftool action : set/pooled=no; selector: aqs1003.eqiad.wmnet
08:57 moritzm: powercycling mw1018, didn't come up after reboot
08:47 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Prepare old servers for decom by sending all queries to new servers (duration: 01m 39s)
08:32 moritzm: rolling reboot of mediawiki canaries for kernel security update
08:30 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: aqs1002.eqiad.wmnet
08:17 logmsgbot: elukey@palladium conftool action : set/pooled=no; selector: aqs1002.eqiad.wmnet
08:15 elukey: rebooting aqs100[23].eqiad for kernel upgrades
02:54 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Jun 28 02:54:56 UTC 2016 (duration 7m 16s)
02:47 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.7) (duration: 08m 59s)
02:27 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.6) (duration: 10m 51s)
00:26 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.7/includes/api/ApiMain.php: UsageException to try to catch T138585 issue (duration: 00m 27s)
00:21 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Enable Wikibase descriptions on Catalan and Polish wikis (T135429) (duration: 00m 26s)
00:09 logmsgbot: dereckson@tin Synchronized wmf-config/mobile.php: Introduce config variable to control tagline (T138738, 2/2) (duration: 00m 27s)
00:08 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Introduce config variable to control tagline (T138738, 1/2) (duration: 00m 32s)
00:07 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings-labs.php: Introduce config variable to control tagline (no-op) (duration: 00m 27s)
00:05 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.6/extensions/MobileFrontend/: Introduce config variable to control tagline (T138738) (duration: 00m 29s)
00:02 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.7/extensions/MobileFrontend/: Introduce config variable to control tagline (T138738) (duration: 00m 39s)

2016-06-27

20:13 mdholloway: mobileapps deployed 30cc12e
20:08 subbu: finished deploying parsoid sha dd8e644d
20:04 subbu: synced new parsoid code; restarted parsoid on wtp1001 as a canary
20:01 subbu: starting parsoid deploy
17:23 gehel: deploying new logstash config for transition to elasticsearch 2.x (T138335)
15:21 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Increase move rate limit for extendedmovers in enwiki to 16/60 (duration: 00m 28s)
15:19 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: SWAT: Delete old throttle rules (duration: 00m 26s)
15:16 gehel: banning elastic1001 to prepare its decommissioning (T138329)
15:13 logmsgbot: thcipriani@tin Synchronized dblists/clldefault.dblist: SWAT: Deploy Compact Language Links as default (Stage 3) PART II (duration: 00m 23s)
15:07 logmsgbot: thcipriani@tin Synchronized wmf-config: SWAT: Deploy Compact Language Links as default (Stage 3) (duration: 00m 40s)
15:00 elukey: mw1136 powercycled - not responsive to ssh and root login
14:49 logmsgbot: gehel@palladium conftool action : set/pooled=no; selector: dc=eqiad,cluster=elasticsearch,service=elasticsearch,name=elastic101[0-6].eqiad.wmnet
14:39 logmsgbot: gehel@palladium conftool action : set/pooled=no; selector: dc=eqiad,cluster=elasticsearch,service=elasticsearch,name=elastic100[0-9].eqiad.wmnet
14:37 logmsgbot: gehel@palladium conftool action : get/pooled; selector: dc=eqiad,cluster=elasticsearch,service=elasticsearch,name=elastic100[0-9]..eqiad.wmnet
14:34 gehel: removing old elasticsearch servers in eqiad from LVS (elastic1001-1016 - T138329)
10:10 moritzm: pooled mw1291 (jessie imagescaler)
09:48 jynus: stopping and reimporting db2010 (m1)
09:47 gehel: removing maps-test*.codfw.wmnet servers from LVS (T138092)
09:19 logmsgbot: gehel@palladium conftool action : set/pooled=yes; selector: dc=eqiad,cluster=elasticsearch,service=elasticsearch-ssl,name=elastic104..eqiad.wmnet
09:19 logmsgbot: gehel@palladium conftool action : set/pooled=yes; selector: dc=eqiad,cluster=elasticsearch,service=elasticsearch,name=elastic104..eqiad.wmnet
09:18 logmsgbot: gehel@palladium conftool action : set/pooled=yes; selector: dc=eqiad,cluster=elasticsearch,service=elasticsearch-ssl,name=elastic103..eqiad.wmnet
09:18 logmsgbot: gehel@palladium conftool action : set/pooled=yes; selector: dc=eqiad,cluster=elasticsearch,service=elasticsearch,name=elastic103..eqiad.wmnet
09:10 logmsgbot: gehel@palladium conftool action : get/pooled; selector: elastic10??\.eqiad\.wmnet (tags: ['dc=eqiad', 'cluster=elasticsearch', 'service=elasticsearch'])
09:07 logmsgbot: gehel@palladium conftool action : set/pooled=yes; selector: elastic1032.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=elasticsearch', 'service=elasticsearch-ssl'])
09:06 logmsgbot: gehel@palladium conftool action : set/pooled=yes; selector: elastic1032.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=elasticsearch', 'service=elasticsearch'])
09:00 gehel: adding new elasticsearch servers in eqiad to LVS
08:54 godog: swift codfw-prod ms-be202[234] weight 2000
07:15 elukey: puppet stopped on analytics1049 to remove it completely from the Hadoop cluster - broken disk
02:51 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Jun 27 02:51:41 UTC 2016 (duration 7m 5s)
02:44 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.7) (duration: 08m 09s)
02:27 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.6) (duration: 10m 54s)

2016-06-26

02:52 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Jun 26 02:52:48 UTC 2016 (duration 6m 19s)
02:46 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.7) (duration: 08m 15s)
02:28 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.6) (duration: 10m 48s)

2016-06-25

09:37 mutante: install2001 killing ganglia aggregator processes, running puppet, for debugging
02:51 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Jun 25 02:51:43 UTC 2016 (duration 6m 26s)
02:45 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.7) (duration: 07m 58s)
02:28 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.6) (duration: 10m 53s)
01:07 chasemp: sign labstore1005 puppet certs and bootstrap the server
00:53 chasemp: hand hack apache on labmon to make it work temporarily

2016-06-24

18:41 logmsgbot: krenair@tin Synchronized dblists/mobilemainpagelegacy.dblist: https://gerrit.wikimedia.org/r/#/c/295958/4 - fix mobile main page rendering on a bunch of wikis, effectively putting them back to how they were a few days ago (duration: 00m 37s)
17:19 mobrovac: change-prop deploying df88a75b
17:05 _joe_: re-started changeprop after disabling the dependency module
14:18 paravoid: shutting down ms-fe3002 due to on-site work
14:05 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.7/includes/OutputPage.php: T138586 hotfix (duration: 00m 47s)
14:02 mobrovac: scb100x disabled puppet to clear changeprop queues
13:22 gehel: re-enabling puppet on maps1002 (still in pre-configuration state, only default role)
12:34 hashar: Random resource loader entries are apparently faulty causing issues with css and/or javascript T138586
12:04 logmsgbot: elukey@palladium conftool action : set/pooled=no; selector: aqs1001.eqiad.wmnet
12:03 elukey: rebooting aqs1001.eqiad.wmnet for kernel upgrades
10:55 jynus: updated m1-slave dns to be db1001
10:20 hashar: gallium: restarted apache2 , potentially stuck proxy
10:18 moritzm: upgrade nodejs on scb systems in codfw and restart node-based services
09:59 ema: nginx rolling restart to enable TFO on all tlsproxies (T108827)
09:52 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1059 with low weight, increase weight of db1061, db1062 (duration: 00m 33s)
09:48 moritzm: upgrade nodejs on restbase test systems (xenon/praseodymium/cerium/restbase-test) and restart restbase on those
09:09 mobrovac: scb100x stopping puppet to stop change-prop and clear the queue
08:29 moritzm: uploaded nodejs 4.4.6 for jessie-wikimedia to carbon
07:10 elukey: memcached on mc1007 restarted with growth factor 1.05 (T129963)
03:54 robh: data copy for labmon1001 verified complete with proper permissions, re-enabling and running puppet to start back up services
03:19 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Jun 24 03:19:55 UTC 2016 (duration 7m 4s)
03:12 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.7) (duration: 17m 24s)
02:38 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.6) (duration: 17m 08s)
01:22 bblack: stream.wikimedia.org (RCStream) DNS moved to cache_misc termination. If anyone reports bugs with rcstream services, revert https://gerrit.wikimedia.org/r/295385

2016-06-23

23:17 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/295600/ (duration: 00m 29s)
23:15 logmsgbot: maxsem@tin Synchronized dblists/mobilemainpagelegacy.dblist: https://gerrit.wikimedia.org/r/#/c/295600/ (duration: 00m 28s)
22:33 chasemp: reimage labstore1005 post io testing
22:12 chasemp: powercycle labstore1005
21:24 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group2 wikis to wmf.6
21:11 chasemp: silence alerts for labstore1004 for setup
20:31 ebernhardson: synced out latest logstash-plugins via trebuchet
20:17 Dereckson: Run initSiteStats.php on cebwiki (T138533)
20:04 logmsgbot: jzerebecki@tin Synchronized wmf-config/CommonSettings.php: Log PHP/HHVM errors in CLI mode to stderr, not stdout T138291 (duration: 00m 28s)
20:03 robh: labmon1001 data restore at 100gb 50minutes in, 298gb total for restoration
19:29 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.7
19:24 greg-g: 19:21 < RoanKatto> !log Synced patches for T137288 and T137593
18:31 elukey: mw130[0134] - new jobrunners installed and pooled (happened automatically after the fist puppet run)
18:09 robh: labmon1001 powering down for reimage
17:45 subbu: finished deploying parsoid sha 18022c96
17:40 subbu: synced new code; restarted parsoid on wtp1001 as a canary
17:37 subbu: starting parsoid deploy
17:29 robh: labmon1001 cpy changed back to local usb, errors on network transfer for ownership. resumed rsync with append flag to local usb disk.
17:03 bblack: cache perf tuning marker: start rollout of tcp_no_metrics_save:0
16:27 chasemp: remove old log files on ytterbium for T114395
16:18 godog: swift: add ms-be202[234] weight 1000 - T136630
15:31 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings-labs.php: SWAT: LABS: Enable geoshapes graph protocol (duration: 00m 29s)
15:26 akosiaris: stop etherpad-lite, etherpad is down
15:16 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Deploy Compact Language Links as default (Stage 2) PART III (duration: 00m 24s)
15:16 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Deploy Compact Language Links as default (Stage 2) PART II (duration: 00m 28s)
15:15 logmsgbot: thcipriani@tin Synchronized dblists/clldefault.dblist: SWAT: Deploy Compact Language Links as default (Stage 2) PART I (duration: 00m 41s)
15:11 robh: puppet disabled on labmon1001 along with all icinga alerting. data migration to usb in progress via root screen session
15:05 robh: starting data backup of labmon1001, halting statsite/graphite/carbon-relay on system
14:47 akosiaris: change the default message in etherpad to indicate problems
14:47 mobrovac: change-prop deploying 05c72ed24ca
14:45 akosiaris: debugging etherpad. Started the service with a blank db, looks like it's working
14:38 akosiaris: stopping etherpad-lite on etherpad1001, disabling puppet
14:32 jynus: restarting etherpad-lite.service
13:53 hashar: Zuul/CI are slowly catching up. I had to drop a few changes that got force merged on the SmashPig repo.
13:37 awight: update SmashPig from a435adeb130217bda8b95d3c5c6331ace8ad1228 to 917138e159f0341e3dfbb35818c3ce479927875b
13:36 hashar: CI is slowed down due to surge of jobs and lack of instances to build them on ( T133911 ). Queue is 50 for Jessie and 25 for Trusty.
13:30 jynus: db1059 backup and reimage
13:28 awight: update SmashPig from c0cc2a1a6062ad8d114473ea1a444786a0d50833 to a435adeb130217bda8b95d3c5c6331ace8ad1228
13:16 jynus: running scap pool on mw1301
13:13 mobrovac: restarting zotero on sca, 6g mem
13:13 jynus: running scap pool on mw1300
13:11 mobrovac: citoid deploying 0129ab0b
13:11 elukey: purged some puppet output logs on compiler02.puppet3-diffs.eqiad.wmflabs to free space (disk full)
13:09 moritzm: depooled jessie image scaler (mw1291) again, works fine, to be permanently pooled on Monday
12:49 moritzm: pooling new jessie image scaler mw1291 for short production smoke testing
12:35 awight: update SmashPig from f7d65c54bed3ff9c478b0dbcaa1b2d27cc665ace to c0cc2a1a6062ad8d114473ea1a444786a0d50833
12:18 awight: update SmashPig from 90757321a3bfa1045202e06e3dd1960a0043493a to f7d65c54bed3ff9c478b0dbcaa1b2d27cc665ace
12:07 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1059; Repool db1061 & db1062; increase weight of db1068 (duration: 00m 39s)
11:33 gehel: rolling restart of elasticsearch10(01|30|08|36|13|40) to activate new masters
10:13 andrewbogott: restarting rabbitmq-server on labcontrol1001 (random debugging attempt for T138106)
09:49 godog: reimage ms-be202[567] with incorrect raid settings
09:11 jynus: syncing etherpadlite.store (m1) on db2010, which had 2 bad chunks
08:39 mobrovac: change-prop restarting on scb to pick up ores rules https://gerrit.wikimedia.org/r/295576
08:06 mobrovac: change-prop deploying 45db4f84827
06:59 moritzm: installing spice security updates
02:48 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Jun 23 02:47:59 UTC 2016 (duration 6m 44s)
02:41 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.7) (duration: 07m 05s)
02:26 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.6) (duration: 11m 19s)

2016-06-22

23:24 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/295560/ (duration: 00m 25s)
23:23 logmsgbot: maxsem@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/295560/ (duration: 00m 24s)
23:23 logmsgbot: maxsem@tin Synchronized dblists/mobilemainpagelegacy.dblist: https://gerrit.wikimedia.org/r/#/c/295560/ (duration: 00m 24s)
23:14 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/294247/ (duration: 00m 24s)
23:09 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/295558/ (duration: 00m 40s)
22:25 ori: Ran hacked maintain-replicas.pl on labsdb100[13] for T135029
21:06 bblack: cache perf: start deploy of -autocorking (probably last experiment I can squeeze in today)
21:00 Dereckson: Run namespaceDupes.php on ptwikinews (T138230) and frwikinews (T138442)
20:33 mdholloway: mobileapps: finished deploying 8046ee2
20:26 yurik: deployed & restarted tilerator https://gerrit.wikimedia.org/r/#/c/295447/
20:25 mdholloway: starting mobileapps deployment
20:20 Reedy: created tmplog_begin_devices on tmplog_end_devices on testwiki.cn_template_log
20:18 yurik: deployed & restarted kartotherian https://gerrit.wikimedia.org/r/#/c/295449/
19:32 bblack: start rollout of first batch of cache sysctl stuff (un-mysterious + disable prequeue timestamps)
19:29 jynus: archiving and dropping reviewdb on m1 shard
19:06 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.7
18:46 jynus: shutting down and reimaging db1001
18:20 papaul: ms-be202[3-7] - signing puppet certs, salt-key, initial run
17:23 akosiaris: restart apache on ununpentium for m1 migration. Hosts RT, just did it for good measure
17:21 akosiaris: restarted bacula-director on helium
17:15 jynus: killing puppet, rt, librenms user connections on db1001
17:10 jynus: failovered m1-master from db1001 to db1016
16:20 gehel: new elasticsearch servers elastic1032-1047 are configured and have joined the eqiad cluster
15:26 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.6/extensions/OATHAuth: SWAT: Fixup qrcode-generating js, to stop race condition. (duration: 00m 33s)
15:23 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Improve style (duration: 00m 33s)
15:18 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.7/extensions/OATHAuth: SWAT: Fixup qrcode-generating js, to stop race condition. (duration: 00m 27s)
15:13 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add www.wpc.ncep.noaa.gov to wgCopyUploadsDomains (duration: 00m 54s)
15:01 elukey: rebooting bohrium.eqiad.wmnet (running piwik) for kernel upgrades
14:32 jynus: checksumming m1 databases in preparation for failover
14:29 tgr: running https://phabricator.wikimedia.org/diffusion/ECAU/browse/master/maintenance/checkLocalUser.php for some users T119736
14:04 moritzm: rolling restart of hhvm/apache on app servers in eqiad for expat security update
13:42 godog: add 500G to fluorine /a (almost full)
13:31 gehel: configuring new elasticsearch servers elastic1038-1042 in eqiad
13:03 hashar: Manually moved some missing build records. Restarting Jenkins
12:49 hashar: T80385 Restarting Jenkins with builds dir set to "${JENKINS_HOME}/builds/${ITEM_FULL_NAME}" which is /var/lib/jenkins/builds/XXX
12:35 gehel: starting reimage of mw1292
12:34 _joe_: disabling puppet on mw1017, live-hacking it
12:34 hashar: T80385 stopping Jenkins and migrating all build records to /var/lib/jenkins/builds
12:06 gehel: configuring new elasticsearch servers elastic1033-1037 in eqiad
10:46 godog: upload libphutil/arcanist 0~git20160620-0wmf1 to carbon
10:32 elukey: mw1140 powercycle after freeze issues due to memory pressure (was not able to ssh to it)
10:18 moritzm: rolling restart of restbase in eqiad to pick up firejail change in service::node
09:46 moritzm: rolling restart of restbase in codfw to pick up firejail change in service::node
09:43 legoktm: live-hacking on mw1017 to debug T115119
09:19 jynus: stopping and reconfiguring mysql on dbstore1001
07:59 moritzm: rolling restart of hhvm/apache on canary app servers in eqiad for expat security update
07:30 jynus: stopping, backing up and reimaging db1061 and db1062
07:06 moritzm: restarted hhvm on mw1131
04:29 chasemp: fix salt key on labtestmetal2001
03:12 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Jun 22 03:12:33 UTC 2016 (duration 6m 44s)
03:05 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.7) (duration: 17m 49s)
02:31 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.6) (duration: 10m 24s)

2016-06-21

23:14 yurik: updated/restarted kartotherian & tilerator - https://gerrit.wikimedia.org/r/#/c/295440/ https://gerrit.wikimedia.org/r/#/c/295441/
23:05 tgr: deleted localuser rows for Mahir256@orwikisource and A879071@enwiki for T119736
22:19 bd808: Backfilled missing 2016-06-20 data to https://tools.wmflabs.org/sal/production?d=2016-06-20
22:08 logmsgbot: ori@tin Synchronized static/images/mobile: I8f09e825: Optimize mobile static images (duration: 00m 34s)
19:27 bd808: Restarted dead logstash process on logstash1001. Looks to have stopped itself due to the the Elasticsearch OOM earlier
19:18 logmsgbot: thcipriani@tin Purged l10n cache for 1.28.0-wmf.5
19:17 bd808: Restarted ElasticSearch on logstash1001; dead from OOM
19:14 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.28.0-wmf.7
18:50 bblack: enabled tcp_notsent_lowat optimization on all caches (marking this time for investigation of perf graphs later) - https://gerrit.wikimedia.org/r/#/c/295376/
17:16 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.7/extensions/Graph/lib/graph2.compiled.js: pre-train backport: Updated to latest graph2 lib (duration: 00m 31s)
17:10 yurik_: deployed graphoid https://gerrit.wikimedia.org/r/#/c/295367/
17:06 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: Temporary IP Cap Lift on es.wiki and commons (duration: 00m 24s)
16:33 yurik_: deployed and restarted graphoid with scap3
16:32 gehel: starting installation of new elasticsearch server elastic1032.eqiad.wmnet
15:58 gehel: puppet run on tin to enable scap3 deployment for graphoid
15:53 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.7/extensions/Echo/: (no message) (duration: 00m 33s)
15:44 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Deploy Compact Language Links as default (Stage 1) (duration: 00m 25s)
15:42 logmsgbot: thcipriani@tin Synchronized wmf-config/db-eqiad.php: Repool db1068 with low weight; depool db1061 and db1062 (duration: 00m 30s)
15:20 logmsgbot: hashar@tin Finished scap: testwiki to group0 (previously was labtestwiki which does not work) (duration: 51m 45s)
14:47 moritzm: rolling restart of aqs service on aqs1001-aqs1006 to pick up new firejail settings
14:28 logmsgbot: hashar@tin Started scap: testwiki to group0 (previously was labtestwiki which does not work)
14:14 moritzm: correction: restbase1007 was already depooled for cassandra maintenance, thus only rebooting to 4.4
14:12 moritzm: depooling restbase1007 for upgrade to Linux 4.4
14:09 logmsgbot: hashar@tin scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="labtestwiki" --outdir="/tmp/scap_l10n_87423667" --threads=4 --lang en --quiet' returned non-zero exit status 255 (duration: 02m 58s)
14:06 logmsgbot: hashar@tin Started scap: (no message)
14:03 gehel: disabling alerting for maps100?\.eqiad\.wmnet during initial installation
14:02 logmsgbot: hashar@tin scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="labtestwiki" --outdir="/tmp/scap_l10n_2087727834" --threads=4 --lang en --quiet' returned non-zero exit status 255 (duration: 06m 37s)
13:55 logmsgbot: hashar@tin Started scap: testwiki to 1.28.0-wmf.7 (take three) T136973
13:55 logmsgbot: hashar@tin scap aborted: testwiki to 1.28.0-wmf.7 (take two) T136973 (duration: 01m 35s)
13:53 logmsgbot: hashar@tin Started scap: testwiki to 1.28.0-wmf.7 (take two) T136973
13:53 logmsgbot: hashar@tin scap aborted: testwiki to 1.28.0-wmf.7 T136973 (duration: 04m 17s)
13:48 logmsgbot: hashar@tin Started scap: testwiki to 1.28.0-wmf.7 T136973
13:15 hashar: T136973 applied all security patches to 1.28.0-wmf.7
13:11 RoanKattouw: Running extensions/Echo/maintenance/removeOrphanedEvents.php on all Echo-enabled wikis for T136425
12:57 moritzm: rolling restart of hhvm/apache in codfw for expat security update
12:49 RoanKattouw: Running extensions/Echo/maintenance/backfillReadBundles.php on all Echo-enabled wikis for T136368
12:49 RoanKattouw: Running extensions/Echo/maintenance/backfillReadBundles.php on all Echo-enabled wikis
12:36 hoo: Started a new JSON dump creation on snapshot1003 (after the last one was inconsistent, per T138291)
12:35 gehel: lowering throttling limit for index recovery on codfw elasticsearch cluster
12:33 hoo: Removed Wikidata json dumps from 20160620 (inconsistent, per T138291).
12:30 hashar: T136973 started cut of branch wmf/1.28.0-wmf.7
12:25 gehel: lowering throttling limit for index recovery on eqiad elasticsearch cluster
11:06 jynus: reimaging db1068
10:32 godog: reboot ms-be2003 for disk ordering - T137785
10:22 moritzm: installing expat security updates on Ubuntu systems
10:03 moritzm: installing wget security updates on Ubuntu systems
09:43 gehel: lowering disk high watermark to rebalance elasticsearch eqiad cluster disk space
09:25 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1068; repool db1070 and db1071 as api (duration: 00m 27s)
09:22 moritzm: rolling reboot of logstash cluster to Linux 4.4
07:41 elukey: restarted hhvm on mw1141 - hhvm was getting SEGV (dump in /tmp/hhvm.8735.bt.)
07:39 elukey: restarted hhvm on mw1139 (hhvm-dump in /tmp/hhvm.20736.bt.)
06:41 moritzm: restarted hhvm on mw1252
02:10 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Jun 21 02:10:55 UTC 2016 (duration 6m 36s)
02:04 logmsgbot: l10nupdate@tin LocalisationUpdate failed (1.28.0-wmf.6) at 2016-06-21 02:04:19+00:00

2016-06-20

23:22 Dereckson: `mwscript namespaceDupes.php ptwikinews --fix` (T138230). Some links and revisions are still to fix.
23:16 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Fix pt.wikinews namespace issue (T138230) (duration: 00m 24s)
23:13 logmsgbot: dereckson@tin Synchronized wmf-config/mobile.php: Remove old mobile workaround for Wikidata descriptions (T127250, T138085) (duration: 00m 33s)
21:05 logmsgbot: aude@tin Synchronized php-1.28.0-wmf.6/extensions/Wikidata: Fix property suggester (duration: 01m 59s)
19:50 chasemp: cleaning up /scratch NFS share as it ran out of inodes
19:17 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.6/includes/api/ApiStashEdit.php: 82e14dc66f478fbdb9ca6eab1eeb4f9c68c99bd1 (duration: 00m 36s)
18:09 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Pool db1071 with low weight after maintenance (duration: 00m 26s)
17:32 bd808: https://tools.wmflabs.org/sal missing events between 2016-06-19T12:29 and 2016-06-20T17:26.
17:26 gehel: deploying latest WDQS
17:19 godog: upload libphutil / arcanist 0~git20160616-0wmf1 to jessie-wikimedia T137770
17:18 mark: Rebooting pfw-codfw
17:00 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: revert cll patch (duration: 00m 25s)
15:44 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Allow sysops to add to/remove from confirmed on ca.wikinews (duration: 00m 25s)
15:37 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable NewUserMessage on pl.wikipedia (duration: 00m 25s)
15:31 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.6/extensions/CentralAuth: SWAT: queryAttached into cheap and expensive part (duration: 00m 31s)
15:20 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Flow beta feature on frwikiquote (duration: 00m 28s)
15:13 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Deploy Compact Language Links as default (Stage 1) PART III (duration: 00m 30s)
15:12 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Deploy Compact Language Links as default (Stage 1) PART II (duration: 00m 29s)
15:12 logmsgbot: thcipriani@tin Synchronized dblists/cll-nondefault.dblist: SWAT: Deploy Compact Language Links as default (Stage 1) PART I (duration: 00m 29s)
15:11 logmsgbot: jmm@palladium conftool action : select; selector: name=mw1099.eqiad.wmnet
15:04 logmsgbot: thcipriani@tin Synchronized dblists/visualeditor-default.dblist: SWAT: Enable VisualEditor by default for all users of French Wikinews (duration: 00m 29s)
13:27 elukey: restarted hhvm on mw1145 after temp. freeze due to memory pressure (hhvm debug in /tmp/hhvm.17794.bt.)
13:27 paravoid: reactivating peerings with Telia Carrier/AS1299 (eqiad/codfw/ulsfo)
13:06 Amir1: full deployment for 8e65182 in ores nodes
13:04 Amir1: deploying 8e65182 to scb2001
12:56 gehel: installing maps1001.eqiad.wmnet (secondary cluster, no traffic there yet) - T138092
12:56 paravoid: deactivating peerings with Telia Carrier/AS1299 (eqiad/codfw/ulsfo)
12:41 moritzm: rebooting ms1001 for update to Linux 4.4
12:13 Amir1: started deploying ores in scb2001 bdc1e2bd
11:36 godog: roll-restart swift on ms-be1* to apply https://gerrit.wikimedia.org/r/294691
11:27 Amir1: for ores in scb nodes
11:27 Amir1: rollbacking ae71d842dfc0958e06922062dd09d49243332a6a
11:13 _joe_: restarting uwsgi orse service
10:58 Amir1: deploying bdc1e2b in ores nodes
10:53 godog: roll-restart swift on ms-be2* to apply https://gerrit.wikimedia.org/r/294691
10:44 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1071 completelly (duration: 00m 25s)
10:35 jynus: db1071 stop, backup and reimage
10:31 mobrovac: restbase started mobile-sections dump for eswiki on restbase1009 for T136964
10:05 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1073 at 100% weight; depool db1071 for reimaging (duration: 00m 27s)
09:50 moritzm: rolling reboot of restbase2001/restbase2002 for upgrade to Linux 4.4
08:57 Amir1: deploying 5dfe738 in ores nodes
08:15 moritzm: installing libxlst security updates
07:43 gehel: rebalancing shards on elasticsearch eqiad cluster
06:47 _joe_: activating the jessie jobrunner, mw1299
05:57 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: Id5804a80: Better cache headers for 'Powered by MediaWiki' badge (2/2) (duration: 00m 35s)
05:56 logmsgbot: ori@tin Synchronized static/images: Id5804a80: Better cache headers for 'Powered by MediaWiki' badge (1/2) (duration: 00m 33s)
02:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Jun 20 02:29:01 UTC 2016 (duration 5m 44s)
02:23 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.6) (duration: 09m 54s)

2016-06-19

12:29 elukey: restarted hhvm on mw1138 - trace in /tmp/hhvm.25048.bt, hhvm killed by OOM
12:27 elukey: restarted hhvm on mw1114 - trace in /tmp/hhvm.11092.bt, hhvm killed by OOM
02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Jun 19 02:31:25 UTC 2016 (duration 5m 47s)
02:25 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.6) (duration: 10m 50s)

2016-06-18

02:32 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Jun 18 02:32:26 UTC 2016 (duration 6m 18s)
02:26 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.6) (duration: 10m 04s)

2016-06-17

21:21 urandom: Reenabling puppet and resetting configuration on xenon.eqiad.wmnet : T137419
20:39 urandom: Restarting Cassandra on xenon.eqiad.wmnet to apply -XX:+PreserveFramePointer : T137419
20:35 urandom: Disabling puppet on xenon.eqiad.wmnet : T137419
20:23 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.6/extensions/WikimediaEvents/: https://gerrit.wikimedia.org/r/#/c/294958/ (duration: 00m 33s)
18:56 urandom: Restarting Cassandra on xenon.eqiad.wmnet with -XX:+PreserveFramePointer : T137419
18:32 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1073 with low weight after reimage (duration: 00m 35s)
16:29 moritzm: installing squid security updates on carbon
15:59 urandom: Starting html dumps from xenon.eqiad.wmnet and cerium.eqiad.wmnet : T137419
15:54 urandom: Restarting Cassandra on xenon.eqiad.wmnet to enable large pages : T137419
14:55 mobrovac: scb disabling puppet for stopping change-prop to clear transclusion queues
14:16 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Increase db1072 weight after repooling (duration: 00m 36s)
12:57 jynus: stopping, backuping and reimaging db1073
12:49 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1072 with low weight, depool db1073 (duration: 00m 27s)
12:49 moritzm: rolling reboot of mw1157-mw1160 into new kernels
12:27 moritzm: restarted hhvm on mw1133 and mw1135
11:14 moritzm: stopping puppet on hosts using service::node (restbase, sca, scb, aqs) for step-by-step rollout of two puppet patches for firejail/service::node
09:31 _joe_: powercycling mw1140, OOMd
09:30 moritzm: rolling reboot of mw1153,mw1155,mw1156 into new kernels
08:29 hashar: Restarting Jenkins on gallium. Web interface at least is deadlocked somehow
07:23 jynus: backuping and reimaging db1072
07:18 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1072 for maintenance (duration: 00m 31s)
07:11 mobrovac: restbase started mobile-sections dump on restbase1009 for T136964
07:02 mobrovac: change-prop restarting it to apply https://gerrit.wikimedia.org/r/294880
06:40 moritzm: installing apache update on palladium
06:16 akosiaris: _joe_ restarted zotero on sca1001
06:16 akosiaris: restarted zotero on sca1002
06:04 logmsgbot: root@palladium conftool action : set/weight=25; selector: cluster=api_appserver,name=mw127.*
05:58 logmsgbot: root@palladium conftool action : set/pooled=yes:weight=20; selector: cluster=api_appserver,name=mw127.*
02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Jun 17 02:31:00 UTC 2016 (duration 6m 26s)
02:24 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.6) (duration: 09m 46s)

2016-06-16

23:44 logmsgbot: ebernhardson@tin Synchronized php-1.28.0-wmf.6/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T137167: TextCat A/B test for Language Identification (duration: 00m 25s)
23:24 logmsgbot: ebernhardson@tin Synchronized php-1.28.0-wmf.6/extensions/WikimediaEvents/extension.json: T137167: TextCat A/B test for Language Identification (duration: 00m 24s)
23:19 logmsgbot: ebernhardson@tin Synchronized php-1.28.0-wmf.6/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T137167: TextCat A/B test for Language Identification (duration: 00m 24s)
23:16 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: T137167: search: Dependent config for textcat AB test. (duration: 00m 26s)
23:11 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: T137888: Two permission changes at urwiki (duration: 00m 27s)
23:07 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings-labs.php: T127250: Prepare Wikidata descriptions on mobile for production rollout (duration: 00m 27s)
22:33 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.6/extensions/Kartographer: https://gerrit.wikimedia.org/r/294856 https://gerrit.wikimedia.org/r/294855 (duration: 00m 30s)
22:24 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/294854/ (duration: 00m 26s)
21:15 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.6/extensions/VisualEditor/ApiVisualEditor.php: Pass empty summary to parseAndStash() to avoid warnings T137995 (duration: 00m 39s)
19:05 logmsgbot: hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.6
18:37 tgr: running invalidateUserSessions.php for T137799
18:22 mobrovac: change-prop deploying bc87a1fecfa
16:36 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Set all new slaves to medium weight (300) after warm up (duration: 00m 25s)
15:37 jynus: deleted sqldata.s6 from labsdb1008 - space issues caused by queries creating temporary tables
15:27 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.6/extensions/ORES/includes/Hooks.php: SWAT: Performance boost on hidenondamaging (duration: 00m 35s)
15:23 moritzm: rolling reboot of restbase1008 - restbase1011 for upgrade to Linux 4.4
15:21 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.6/extensions/ORES: SWAT: Skip when an edit is errored in PopulateDatabase.php (duration: 00m 30s)
15:04 logmsgbot: root@palladium conftool action : set/pooled=yes; selector: name=mw1262.eqiad.wmnet
14:31 twentyafterfour: re-enabled and ran puppet agent --test on iridium. Everything appears to be normal.
13:04 mobrovac: scb1001 enabled puppet back
12:57 gehel: rebalancing shards on elasticsearch equiad cluster
12:33 Amir1: manually restarted celery-ores-worker in scb1001
12:32 moritzm: installing apache2 trusty update on graphite1001
12:32 Amir1: manually restarted celery-ores-worker in scb1002
12:10 moritzm: restarted hhvm on mw1137, got stuck
10:44 moritzm: depooling mw1154 for kernel update/reboot
10:14 mobrovac: scb1001 disabling puppet for a while to manually test changeprop with transclusion rules
09:59 mobrovac: restbase deploy end of ebeaa46
09:56 _joe_: powercycling mw1143, unresponsive on ssh, console
09:48 mobrovac: restbase deploy start of ebeaa46
09:18 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.6/extensions/MobileFrontend: MobileFrontend RL registration issue preventing Special:Nearby from working properly T137919 (duration: 00m 36s)
08:41 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Pool db1085, increase weight of all new db servers (duration: 00m 29s)
08:15 jynus: rebooting db1085 before putting it back into production
02:34 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.5) (duration: 15m 49s)
00:57 twentyafterfour: puppet disabled on iridium because https://gerrit.wikimedia.org/r/#/c/294653/ needs to merge (hotfix in preamble.php which puppet will undo if it's allowed to run)
00:43 twentyafterfour: phabricator upgrade/maintenance complete. Everything appears to be back up and running normally.
00:41 twentyafterfour: taking phabricator offline momentarily for scheduled maintenance.
00:24 robh: mw1147 rebooted and manually running scap pull
00:21 robh: mw1147 seems to have died during scap, unresponsive from serial console, powercycled
00:16 logmsgbot: mattflaschen@tin Synchronized php-1.28.0-wmf.6/extensions/Kartographer: Search for maplinks inside and outside of content. (duration: 01m 08s)

2016-06-15

23:38 logmsgbot: mattflaschen@tin Synchronized php-1.28.0-wmf.6/extensions/Echo: Sync Echo fix for cross-wiki notifications: 62324e3 (duration: 00m 33s)
21:32 logmsgbot: aaron@tin Synchronized wmf-config/filebackend-production.php: Set "sync" filebackend replication to measure latency effect (duration: 00m 25s)
21:27 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.6/includes/libs/objectcache/WANObjectCache.php: faff8f1ef1bfefd1804a3f46e58566711faa3224 (duration: 00m 27s)
21:16 dapatrick: Deployed patch for T137264 to wmf.5 and wmf.6
20:17 logmsgbot: hashar@tin Synchronized wmf-config/throttle.php: Temporary IP Cap Lift on es.wiki T137917 (duration: 00m 30s)
20:09 subbu: finished deploying parsoid sha 3445eceb
20:05 bblack: cache frontend restarts complete
20:04 subbu: synced new code; restarted parsoid on wtp1001 as a canary
20:02 subbu: starting parsoid deploy
19:25 bblack: rolling restart of global varnish frontends (salt -b 1: depool -> sleep 15 -> restart -> repool) - estimated ~35 mins to completion - T107236 (...._
19:15 bblack: varnish frontend restart halted - v4 compat issue to address :P
19:11 bblack: rolling restart of global varnish frontends (salt -b 1: depool -> sleep 15 -> restart -> repool) - estimated ~30 mins to completion - T107236
19:05 logmsgbot: hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: (no message)
18:54 ori: Started MySQL on es2019 (T130702)
16:32 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1023; pool db1085 (disabled), db1088, db1092 w/low weight (duration: 00m 25s)
16:07 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix autopatrolled group for ko.wikipedia (duration: 00m 31s)
16:00 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.6/resources/src/mediawiki.special/mediawiki.special.search.styles.css: SWAT: Explicitly specify the width of the search input on Special:Search (duration: 00m 25s)
15:53 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add autopatrolled group in kowiki (duration: 00m 24s)
15:33 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Deploy ORES beta feature in wikidatawiki (duration: 00m 24s)
15:23 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: kafka1002.eqiad.wmnet
15:23 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.5/extensions/ORES: SWAT: Skip when an edit is errored in PopulateDatabase.php (duration: 00m 27s)
15:17 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Send authentication events to logstash (duration: 00m 28s)
15:15 logmsgbot: elukey@palladium conftool action : set/pooled=no; selector: kafka1002.eqiad.wmnet
15:11 logmsgbot: thcipriani@tin Synchronized wmf-config/logging.php: SWAT: Fix logging config for authmanager metrics channel rename (duration: 00m 24s)
15:10 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: kafka1001.eqiad.wmnet
15:06 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: SWAT: Remove old throttle rules (duration: 00m 30s)
15:00 logmsgbot: elukey@palladium conftool action : set/pooled=no; selector: kafka1001.eqiad.wmnet
15:00 mobrovac: scb disabled puppet for stopped change-prop during kafka nodes upgrade
15:00 elukey: rebooting Eqiad Event Bus for kernel upgrades (one node at the time)
14:24 moritzm: installing php security updates on jessie systems
13:55 moritzm: remove unused PHP packages from the recently provisioned jessie app servers (new installation are fixed in puppet to only install php5-cli, but the initial set needs fixed up manually)
13:40 gehel: rolling back update of firejail on maps2001
13:16 _joe_: stopped jobchron, jobrunner on mw1299, masked in systemd
13:15 mobrovac: change-prop deployed 6ad337
13:06 moritzm: installing libav security updates
12:37 _joe_: rebooting mw1299
12:06 gehel: upgrade of firejail on maps server stopped, pending a patch to service::node
11:46 mobrovac: scb enabled puppet back
11:44 gehel: upgrading firejail to 0.9.38 on maps servers
11:32 mobrovac: scb disabled puppet for 5 min to keep change-prop down
11:30 mobrovac: change-prop deploying 353b926
11:29 jynus: stopping db1023 for cloning to new s6 hosts
11:22 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Increase new enwiki dbs weight, depool db1023 for cloning (duration: 00m 27s)
11:13 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1033, first pool of db1079, db1086, db1094 with low weight (duration: 00m 25s)
11:11 moritzm: enabed firejail wrapper for imagemagick's convert (for image scalers and the Score extension)
10:59 paravoid: rebooting install2001 again
10:48 logmsgbot: jmm@tin Synchronized wmf-config/CommonSettings.php: firejail security hardening for image scalers (duration: 00m 26s)
09:48 godog: bounce ms-be2003, xfs high load
09:13 moritzm: repooled mw1154 (kernel still the same ATM)
08:53 moritzm: depooling mw1154 (image scaler) for kernel update
08:29 jynus: turning down db1033 for cloning to new s7 slaves
08:15 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1033 for cloning (duration: 00m 38s)
06:59 moritzm: installing apache trusty updates on eqiad app servers
03:51 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.6/includes/parser/Parser.php: 4e6e1bc1f2de000f0fdd84dcf04f63a21127d24a (duration: 00m 30s)
03:49 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.5/includes/parser/Parser.php: 23bac8905a9d60cdc0a068ca025644e091b9027f (duration: 00m 32s)
03:10 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Jun 15 03:10:57 UTC 2016 (duration 6m 55s)
03:04 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.6) (duration: 16m 29s)
02:30 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.5) (duration: 12m 24s)
02:29 logmsgbot: ori@tin Synchronized php-1.28.0-wmf.5/extensions/Scribunto/engines/LuaCommon/TitleLibrary.php: revert: ad-hoc debug of vary-revision in scribunto (duration: 00m 29s)
02:22 logmsgbot: ori@tin Synchronized php-1.28.0-wmf.5/extensions/Scribunto/engines/LuaCommon/TitleLibrary.php: ad-hoc debug of vary-revision in scribunto (duration: 00m 26s)
01:51 logmsgbot: ori@tin Synchronized php-1.28.0-wmf.5/resources/src/mediawiki.action/mediawiki.action.edit.stash.js: Idfad8407: Improve client-side edit stash change detection (duration: 00m 24s)
01:31 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.6/includes/parser: 78de24a20c4662ea709e1f8af84bb5fae4aea2fa (duration: 00m 33s)
01:30 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.5/includes/parser: 48652dfc27d1bbaab41b3a4d8f7d6be23e2da6b6 (duration: 00m 34s)

2016-06-14

23:40 logmsgbot: ori@tin Synchronized php-1.28.0-wmf.6/resources/src/mediawiki.action/mediawiki.action.edit.stash.js: Idfad8407c8e: Improve client-side edit stash change detection (duration: 00m 25s)
23:30 logmsgbot: ori@tin Synchronized wmf-config/InitialiseSettings.php: Id800a9d35b: Set import sources for he.wikipedia (T137074) and If66f307a2e: Set import sources for pt.wikinews (T137633) (duration: 00m 27s)
23:28 logmsgbot: ori@tin Synchronized php-1.28.0-wmf.6/extensions/AntiSpoof: I2e407a3ac8: Revert "Make sure AntiSpoof mappings are mapping in the correct direction." (duration: 00m 27s)
23:15 logmsgbot: ori@tin Synchronized php-1.28.0-wmf.6/extensions/Echo: If07369cb1: Allow the primary link to set all bundled notifications as read (T136368) (duration: 00m 34s)
23:09 logmsgbot: ori@tin Synchronized wmf-config/abusefilter.php: I4e5e4d227: Set $wgAbuseFilterConditionLimit = 2000 for commonswiki (T132048) (duration: 00m 28s)
22:43 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.6/includes/deferred: 0d038de1414c0b4faed1cc9882151e68d86d3b2d (duration: 00m 25s)
22:15 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.5/includes/deferred: 29863094805baed7a5fa493c99c87745ce041f49 (duration: 00m 27s)
21:50 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.6/resources: 7898fd2fa969342a5cc30df6a5757f4642cd6118 (duration: 00m 28s)
21:44 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.6/includes: 7898fd2fa969342a5cc30df6a5757f4642cd6118 (duration: 01m 12s)
21:33 logmsgbot: gehel@palladium conftool action : set/pooled=no; selector: name=maps-test2.*
21:28 logmsgbot: gehel@palladium conftool action : set/pooled=yes; selector: name=maps-test2*
21:28 gehel: sending traffic back to old maps servers (T137620)
21:10 logmsgbot: gehel@palladium conftool action : set/pooled=no; selector: name=maps-test2*
21:09 logmsgbot: gehel@palladium conftool action : set/pooled=no; selector: maps-test2001.codfw.wmnet (tags: ['dc=codfw', 'cluster=maps', 'service=kartotherian'])
21:09 logmsgbot: gehel@palladium conftool action : set/pooled=yes; selector: maps2004.codfw.wmnet (tags: ['dc=codfw', 'cluster=maps', 'service=kartotherian'])
21:08 logmsgbot: gehel@palladium conftool action : set/pooled=yes; selector: maps2003.codfw.wmnet (tags: ['dc=codfw', 'cluster=maps', 'service=kartotherian'])
21:08 logmsgbot: gehel@palladium conftool action : set/pooled=yes; selector: maps2002.codfw.wmnet (tags: ['dc=codfw', 'cluster=maps', 'service=kartotherian'])
20:58 logmsgbot: gehel@palladium conftool action : set/pooled=yes; selector: maps2001.codfw.wmnet (tags: ['dc=codfw', 'cluster=maps', 'service=kartotherian'])
20:55 gehel: pooling maps2001 (new map server) - T137620
20:50 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.6/includes: ca9068daffb49cc0cdfb84385a29aea34df155cd (duration: 01m 51s)
20:46 gehel: adding new maps servers to LVS
20:09 logmsgbot: demon@tin Finished scap: wikidata submodule update for wmf.6 (duration: 25m 51s)
19:43 logmsgbot: demon@tin Started scap: wikidata submodule update for wmf.6
19:30 logmsgbot: demon@tin Finished scap: group0 to 1.28.0-wmf.6 (duration: 26m 43s)
19:03 logmsgbot: demon@tin Started scap: group0 to 1.28.0-wmf.6
18:56 logmsgbot: demon@tin Purged l10n cache for 1.27.0-wmf.23
18:54 logmsgbot: demon@tin Purged l10n cache for 1.28.0-wmf.4
18:54 logmsgbot: demon@tin Purged l10n cache for 1.28.0-wmf.3
18:54 logmsgbot: demon@tin Purged l10n cache for 1.28.0-wmf.2
18:53 logmsgbot: demon@tin Purged l10n cache for 1.28.0-wmf.1
17:22 Dereckson: Run initSiteStats.php for arcwiki and htwiki (T137827)
16:48 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1040, pool for the first time db1081, db1084, db1091 (duration: 00m 34s)
16:41 godog: reimage ms-fe3002 with jessie T117972
15:57 yurik: deployed & restarted kartotherian (fixing spec.config tests)
15:54 urandom: Restarting cassandra-metrics-collector on restbase1007 : T137304
15:53 logmsgbot: thcipriani@tin Synchronized wmf-config: SWAT: Beta: Enable Compact Language Links for new users (duration: 00m 31s)
15:41 logmsgbot: hashar@tin scap aborted: testwiki to php-1.28.0-wmf.6 and rebuild l10n cache (duration: 01m 31s)
15:40 logmsgbot: hashar@tin Started scap: testwiki to php-1.28.0-wmf.6 and rebuild l10n cache
15:35 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.5/extensions/CentralAuth/includes/CentralAuthHooks.php: SWAT: Account for changed login process (duration: 00m 26s)
15:27 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add nonecho.dblist and echo.dblist PART III (duration: 00m 27s)
15:26 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Add nonecho.dblist and echo.dblist PART II (duration: 00m 26s)
15:26 godog: reimage ms-fe3001 with jessie T117972
15:25 logmsgbot: thcipriani@tin Synchronized dblists: SWAT: Add nonecho.dblist and echo.dblist PART I (duration: 00m 28s)
15:19 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add nonecho.dblist and echo.dblist PART III (duration: 00m 28s)
15:18 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Add nonecho.dblist and echo.dblist PART II (duration: 00m 30s)
15:18 logmsgbot: thcipriani@tin Synchronized dblists: SWAT: Add nonecho.dblist and echo.dblist PART I (duration: 00m 30s)
15:09 yurik: deployed & restarted kartotherian
15:07 logmsgbot: thcipriani@tin Synchronized dblists/visualeditor-default.dblist: SWAT: Enable VisualEditor by default on eleven Wikivoyages (duration: 01m 49s)
13:47 hashar: T136971 Cutting MediaWiki branches 1.28.0-wmf.6
13:40 moritzm: installing apache trusty updates on codfw app servers
13:28 paravoid: rebooting install2001, T137647
12:58 moritzm: installing apache trusty updates on canary app servers
12:55 mobrovac: change-prop deployed f34fb06c99
12:27 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1024, pool for the first time db1090 with low weight (duration: 00m 38s)
11:30 mobrovac: scb disabling puppet for 10 mins or so to keep change-prop down
11:15 akosiaris: T134242 rebooting alsafi.wikimedia.org hassaleh.codfw.wmnet kraz.wikimedia.org mx2001.wikimedia.org planet2001.codfw.wmnet pollux.wikimedia.org pybal-test2001.codfw.wmnet pybal-test2002.codfw.wmnet pybal-test2003.codfw.wmnet for qemu-kvm upgrade
11:13 akosiaris: T134242 install qemu-system-common, qemu-system-x86 1:2.5+dfsg-4~bpo8+1 from jessie-backports on ganeti200{1,2,3,4,5,6}
11:04 _joe_: pooling all the new codfw appservers that have been installed - mw2215-mw2240 (T135466)
10:56 _joe_: pooling the new jessie appservers, mw1263-71
10:52 logmsgbot: oblivian@palladium conftool action : set/weight=30; selector: cluster=appserver,dc=eqiad,name=mw12[67].*
09:27 godog: roll-restart swift proxy in codfw and eqiad
09:04 hashar: gallium: manually removing cron entry zuul_repack from user zuul. Causes cron spam due to zuul merger no more being on gallium T137418
08:59 jynus: stopping db1040 for cloning to new s4 hosts
08:28 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1040 for cloning (duration: 00m 32s)
08:23 _joe_: powercycling mw1154, unresponsive
07:19 jynus: powercycling mw1156, could not regain control after OOM
07:18 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1024, increase weight of db1082, db1087 and db1092 (duration: 10m 50s)
07:05 _joe_: rolling reboot of mw2233-40
06:47 _joe_: rebooting mw2228
06:43 _joe_: rebooting mw2228
06:29 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Pool db1052, db1080, db1083, db1089 (duration: 01m 31s)
02:39 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Jun 14 02:39:50 UTC 2016 (duration 5m 59s)
02:33 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.5) (duration: 12m 14s)

2016-06-13

23:50 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Add ORES to whitelisted beta features (T130211) (duration: 00m 23s)
23:42 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.5/extensions/ORES/includes/Hooks.php: Update links to beta features (duration: 00m 25s)
23:33 ejegg: updated payments from 44102c59ac897c9acab470bf83369d233f9b736f to 2fc573cbb94e833c4144aa9dad79de8ec374bb09
23:29 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Update cross-wiki upload configuration (Gerrit:293355) (duration: 00m 23s)
23:10 logmsgbot: dereckson@tin Synchronized portals: (no message) (duration: 00m 24s)
23:10 logmsgbot: dereckson@tin Synchronized portals/prod/wikipedia.org/assets: (no message) (duration: 00m 24s)
22:51 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: Update extension distributor settings (duration: 00m 24s)
22:42 yurik: switched to scap3 and deployed tilerator. Deployed kartotherian. Restarted.
22:41 dapatrick: Deployed patches for T129738 to wmf5
22:36 awight: update fundraising CRM revert from e684b7823e751558772a4de4ac23819bc601eb74 to bb9bf136dc0fa82d5d07ebeb33d696e54672b2d6
22:11 awight: Updating fundraising CRM from b7b46740d701942507dca0a98a75f3f87b6b31b1 to e684b7823e751558772a4de4ac23819bc601eb74
19:15 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.5/resources: ee2da9c2ae6fac93bf65d17b5ea48e5c47c87d47 (duration: 00m 35s)
18:20 bblack: upgrading nginx (etc) on deployment-prep caches
18:11 gehel: deploying latest GUI on WDQS,
17:58 urandom: Upgrade of restbase1007.eqiad.wmnet (https://people.wikimedia.org/~eevans/debian/cassandra_2.2.6-wmf1_all.deb) complete : T137474
17:55 urandom: Restarting restbase1007-c.eqiad.wmnet : T137474
17:52 urandom: Restarting restbase1007-b.eqiad.wmnet : T137474
17:47 awight: Whitelist Special:PaypalExpressGatewayResult
17:43 godog: enable proxy_http apache module on graphite1003 / graphite2002 and restart apache
17:38 urandom: Restarting restbase1007-a.eqiad.wmnet : T137474
17:37 urandom: Upgrading restbase1007.eqiad.wmnet w/ https://people.wikimedia.org/~eevans/debian/cassandra_2.2.6-wmf1_all.deb : T137474
17:35 awight: update paymentswiki from 63fbe39fbc4d671fd2705ce9e42762b7c49564c2 to 44102c59ac897c9acab470bf83369d233f9b736f
16:51 _joe_: powercycling mw1115
16:49 logmsgbot: thcipriani@tin Finished scap: Update l10n cache for ores (duration: 32m 04s)
16:17 logmsgbot: thcipriani@tin Started scap: Update l10n cache for ores
15:59 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Enable ORES on fawiki PART II (duration: 00m 24s)
15:58 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ORES on fawiki PART I (duration: 00m 25s)
15:58 logmsgbot: thcipriani@tin Synchronized wmf-config/extension-list: SWAT: Add ORES to extension-list (duration: 00m 25s)
15:42 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add images.nypl.org to $wgCopyUploadsDomains for commons (duration: 00m 24s)
15:37 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VE in NS_PROJECT in cswiki (duration: 00m 25s)
15:32 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.5/extensions/Echo: SWAT: Use localized weekdays on Special:Notifications (duration: 00m 32s)
15:26 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable transwiki import for la.wiktionary (duration: 00m 26s)
15:22 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Permission changes in zhwiki (duration: 00m 26s)
15:18 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor by default for logged-out users on four Wikipedias too (duration: 00m 24s)
15:10 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.5/extensions/MobileFrontend: Do Not strip srcset on API mobileview action PART II (duration: 00m 38s)
15:09 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.5/extensions/MobileFrontend/includes/MobileContext.php: Do Not strip srcset on API mobileview action PART I (duration: 00m 49s)
15:05 godog: reboot ms-be2012 to fix disk ordering T136395
14:51 godog: truncate syslog.1 on ms-be2012
14:26 bblack: upgrading cp* nginx (and other oustanding minor package updates)
14:23 bblack: uploaded nginx-1.11.1-1+wmf2 to carbon
13:55 dcausse: restarting logstash on logstash1001
11:59 mobrovac: change-prop deployed 54f98b7
11:31 _joe_: rolling reboot of the new appservers in codfw + scap pull
09:55 _joe_: powercycling mw1138, oom, console non-responsive
09:53 jynus: stopping db1052 and cloning it to db1080, db1083 and db1089
09:43 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1052 for cloning (duration: 00m 26s)
08:51 moritzm: removed /var/log/logstash/logstash.log.1 on logstash1001, depleted disk space on the root partition, fallout of T137400
08:43 jynus: powercycling mw1155.eqiad.wmnet , unresponsive on ssh, serial console
08:31 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Increase weight of db1082, db1087, db1092 (duration: 02m 36s)
08:25 logmsgbot: oblivian@palladium conftool action : set/weight=30; selector: name=mw1261.eqiad.wmnet
08:17 logmsgbot: oblivian@palladium conftool action : set/pooled=yes; selector: name=mw1261.eqiad.wmnet
08:01 logmsgbot: oblivian@palladium conftool action : set/pooled=no:weight=20; selector: name=mw1262.eqiad.wmnet
08:00 logmsgbot: oblivian@palladium conftool action : set/pooled=no:weight=20; selector: name=mw1261.eqiad.wmnet
08:00 logmsgbot: oblivian@palladium conftool action : set/pooled=no; selector: name=mw1261.eqiad.wmnet
06:32 logmsgbot: oblivian@palladium conftool action : set/pooled=yes; selector: name=mw126.*
02:28 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.5) (duration: 13m 02s)

2016-06-12

02:28 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.5) (duration: 12m 24s)

2016-06-11

03:14 logmsgbot: ori@tin Synchronized php-1.28.0-wmf.5/includes/parser/CacheTime.php: remove ad-hoc logging of updateCacheExpiry(0) traces (duration: 00m 23s)
03:11 logmsgbot: ori@tin Synchronized php-1.28.0-wmf.5/includes/parser/CacheTime.php: ad-hoc logging of updateCacheExpiry(0) traces (duration: 00m 25s)
02:36 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Jun 11 02:36:22 UTC 2016 (duration 6m 30s)
02:29 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.5) (duration: 11m 42s)
01:01 mutante: rutherfordium ganeti lockup, gnt-instance console .. and it recovered

2016-06-10

23:37 awight: Update PayPal Express Checkout configuration: add API certificate path
21:58 logmsgbot: ori@tin Synchronized wmf-config/mobile.php: I3d8155d7e14: Remove old config hack that disabled $wgResponsiveImages on mobile (duration: 00m 24s)
19:38 mutante: cp1043/cp1044 - revoke puppet cert, salt key
19:30 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: Fix for ip lift cap for eswiki and Temporary IP Cap Lift for eswiki (duration: 00m 23s)
19:19 logmsgbot: ori@tin Synchronized multiversion: Id432e25c: MWMultiVersion: allow wiki to be specified via the environment (duration: 00m 56s)
17:12 elukey: Updated the puppet compiler with new hosts/facts
16:32 mutante: cp1043,cp1044 shutdown -h, confirmed not in pybal/confctl
16:27 mutante: cp1043/cp1044 - decom'ing, were already "Unused spare system" but running, scheduling downtime in icinga, shutting them down and removing from torrus config and puppet (T133614)
14:17 urandom: Testing patched Cassandra (dpkg -i ...; service cassandra-{a,b} restart) on restbase-test200[1-2] : T137474
14:06 urandom: Testing patched Cassandra (dpkg -i ...; service cassandra-a restart) on restbase-test2001 : T137474
13:59 urandom: Testing patched Cassandra (dpkg -i ...; service cassandra-a restart) on praseodymim : T137474
13:58 urandom: Testing patched Cassandra (dpkg -i ...; service cassandra-a restart) on cerium : T137474
13:15 urandom: Starting html dump(s) in RESTBase staging : T137474
13:13 urandom: Testing patched Cassandra (dpkg -i ...; service cassandra-a restart) on xenon : T137474
11:07 logmsgbot: tgr@tin Synchronized php-1.28.0-wmf.5/includes/specials/SpecialUserLogin.php: deploy gerrit:293704 to fix AuthManager metrics (duration: 00m 32s)
11:06 logmsgbot: tgr@tin Synchronized php-1.28.0-wmf.5/includes/specials/SpecialCreateAccount.php: deploy gerrit:293704 to fix AuthManager metrics (duration: 00m 52s)
10:56 mobrovac: scb100x enabled puppet back
10:05 mobrovac: scb100x disabling puppet and stopping change-prop to look at zookeeper znodes
09:22 elukey: restarted uwsgi-ores on scb200[12] as deployment follow up
08:27 elukey: restarted uwsgi-ores (after a deployment + puppet run) - service was down
08:01 Amir1: deploying 38df031 into scb100[12] for ores service. Expecting some down time
07:59 dcausse: refilling ttmserver index on all ttm enabled wikis
06:42 moritzm: bounced hhvm on mw1264 (backtrace in /tmp/hhvm.2197.bt)
06:28 papaul: mw2215-mw2238 -signing puppet certs, salk-key initial run
05:54 mutante: re-enabling puppet on carbon
04:48 moritzm: installing squid3 security updates on Ubuntu systems
03:17 logmsgbot: aaron@tin Synchronized wmf-config/CommonSettings.php: Lower $wgAPIMaxLagThreshold to 5 (duration: 00m 36s)
02:35 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Jun 10 02:35:15 UTC 2016 (duration 6m 2s)
02:29 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.5) (duration: 11m 32s)
01:44 logmsgbot: tgr@tin Synchronized php-1.28.0-wmf.5/includes/specialpage/LoginSignupSpecialPage.php: deploying gerrit:293668: fix AuthManager warning spam (duration: 00m 25s)
01:43 logmsgbot: tgr@tin Synchronized php-1.28.0-wmf.5/includes/specials/SpecialUserLogin.php: deploying gerrit:293667: fix AuthManager dashboard (duration: 00m 33s)
01:42 logmsgbot: tgr@tin Synchronized php-1.28.0-wmf.5/includes/specials/SpecialCreateAccount.php: deploying gerrit:293667: fix AuthManager dashboard (duration: 00m 25s)
00:56 kaldari: ran mwscript maintenance/updateCollation.php --wiki=tawiki --force
00:40 kaldari: ran mwscript maintenance/updateCollation.php --wiki=tawikibooks --force
00:39 kaldari: ran mwscript maintenance/updateCollation.php --wiki=tawikinews --force
00:36 mutante: git pull on strontium because i merged a non-change
00:31 kaldari: ran mwscript maintenance/updateCollation.php --wiki=tawikiquote --force
00:27 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings-labs.php: ores.wikimedia.org instead of ores.wmflabs.org (duration: 00m 25s)
00:21 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.5/extensions/Wikidata/extensions/ArticlePlaceholder/includes/SearchHookHandler.php: Update Wikidata - Fix uncaught exception in ArticlePlaceholder (3/3) (duration: 00m 25s)
00:20 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.5/extensions/Wikidata/vendor/composer/installed.json: Update Wikidata - Fix uncaught exception in ArticlePlaceholder (2/3, no-op) (duration: 00m 25s)
00:19 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.5/extensions/Wikidata/composer.lock: Update Wikidata - Fix uncaught exception in ArticlePlaceholder (1/3, no-op) (duration: 00m 27s)
00:12 kaldari: ran mwscript maintenance/updateCollation.php --wiki=tawikisource --force
00:07 kaldari: ran mwscript maintenance/updateCollation.php --wiki=tawiktionary --force

2016-06-09

23:57 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Set Tamil projects to use uca-ta collation II (T75453) (duration: 00m 25s)
23:53 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Enable Flow beta feature on frwiki (T136684) (duration: 00m 27s)
23:47 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Remove HiddenPrefs hack for turning off cross-wiki notifications (T135266) (duration: 00m 27s)
23:31 logmsgbot: tgr@tin Synchronized wmf-config/InitialiseSettings.php: enable AuthManager on group2 wikis T135504 (duration: 00m 24s)
23:29 logmsgbot: tgr@tin Synchronized wmf-config/CommonSettings.php: enable use of group1, group2 dblists in config (duration: 00m 23s)
23:28 logmsgbot: tgr@tin Synchronized dblists/group2.dblist: add dblist for group2 (duration: 00m 22s)
23:20 logmsgbot: tgr@tin Synchronized php-1.28.0-wmf.5/includes/specialpage/LoginSignupSpecialPage.php: deploying gerrit:293636 for AuthManager T135504 (duration: 00m 25s)
23:19 logmsgbot: tgr@tin Synchronized php-1.28.0-wmf.5/extensions/MobileFrontend/resources/skins.minerva.special.userlogin.styles/userlogin.less: deploying gerrit:293638 for AuthManager T135504 (duration: 00m 25s)
23:18 logmsgbot: tgr@tin Synchronized php-1.28.0-wmf.5/extensions/ConfirmEdit/FancyCaptcha/resources/ext.confirmEdit.fancyCaptcha.js: deploying gerrit:293637 for AuthManager T135504 (duration: 00m 24s)
22:48 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.5
22:35 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.5/includes/site/DBSiteStore.php: Revert "Map dummy language codes in sites" Part II (duration: 00m 31s)
22:35 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.5/includes/ServiceWiring.php: Revert "Map dummy language codes in sites" Part I (duration: 00m 23s)
21:41 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.5/includes: 904dd4ae088a8f67942c09b2b28178377955d6a6 (duration: 01m 18s)
20:57 logmsgbot: hoo@tin Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on nnwiki (T130997) (duration: 00m 24s)
20:53 logmsgbot: hoo@tin Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on lvwiki (T136100) (duration: 00m 26s)
20:48 logmsgbot: hoo@tin Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on guwiki (T136517) (duration: 00m 24s)
20:40 logmsgbot: hoo@tin Synchronized php-1.28.0-wmf.4/extensions/Wikidata: Update ArticlePlaceholder (duration: 01m 54s)
20:36 logmsgbot: hoo@tin Synchronized php-1.28.0-wmf.5/extensions/Wikidata: Update ArticlePlaceholder (without unrelated T136598 fixes this time) (duration: 01m 51s)
20:33 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.5/includes/user/User.php: c3b1f80a701d61dc57ccac0c8b1dc7daf03fa925 (duration: 00m 29s)
19:59 urandom: Restarting Cassandra on xenon.eqiad.wmnet (removing patched test build; restoring state) : T137474
19:53 logmsgbot: hoo@tin Synchronized php-1.28.0-wmf.5/extensions/Wikidata: revert, possible s5 master overload (duration: 01m 57s)
19:47 logmsgbot: hoo@tin Synchronized php-1.28.0-wmf.5/extensions/Wikidata: Update ArticlePlaceholder (duration: 02m 04s)
19:44 bearND: mobileapps deployed 71ff97c
19:42 bearND: starting mobileapps deploy
19:11 ejegg: updated cancel page settings on payments-wiki
17:43 urandom: Restarting Cassandra on xenon.eqiad.wmnet (use exponentially decaying resevoirs for metrics histograms) : T126629
17:19 mobrovac: change-prop deploying ecfda93f09d
17:10 ejegg: updated payments-wiki from 3dcf58e3b4e1d02ad4f1874a3e87e55b7e169bfe to 053aaa259382c94aa59e4d0da7317fcafab635cd
15:31 elukey: added topic override retention.bytes=536870912000 to Kafka webrequest_text (T136690)
15:22 hashar: Cleaning git-daemon on gallium (was used by zuul-merger) T137418
15:19 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Add *.nara.gov to wgCopyUploadDomains (duration: 00m 40s)
14:47 mobrovac: change-prop stopped on scb1002
14:38 elukey: Tested temp setting retention.bytes=2G for Analytics kafka topic webrequest_misc
14:37 hashar: Removing zuul-merger from gallium
14:33 hashar: stopped / disabled zuul-merger on gallium T137418
14:12 mobrovac: change-prop restarting on scb1001 for update
14:07 urandom: Re-enabling puppet on xenon.eqiad.wmnet, forcing a run, and restarting Cassandra : T137419
13:52 mobrovac: change-prop restarting on scb1002 for update
13:45 mobrovac: change-prop deploying 2161403c
13:26 urandom: Restarting Cassandra on xenon.eqiad.wmnet to apply 2G file cache : T137419
12:51 urandom: Restarting Cassandra on xenon.eqiad.wmnet : T126629
12:33 logmsgbot: tgr@tin Synchronized php-1.28.0-wmf.5/extensions/LdapAuthentication/LdapPrimaryAuthenticationProvider.php: deploy gerrit:293459 to fix wikitech API login / morebots (T137377) (duration: 00m 47s)
12:19 tgr: !log deploying gerrit:293459 to fix morebots (T137377)
12:11 urandom: !log Restarting Cassandra on xenon.eqiad.wmnet : T126629
12:06 urandom: !log Temporarily disabling puppet on xenon.eqiad.wmnet to test settings : T126629
11:33 Amir1: !log manually restarting ores-uwsgi and celery-ores-worker in scb100[12]
10:51 urandom: !log Restarting Cassandra on {cerium,praseodymium}.eqiad.wmnet (RESTBase staging) : T126629
09:16 gehel: !log lowering disk high watermark to rebalance disk usage on elasticsearch eqiad cluster
09:05 Amir1: !log restarting uwsgi-ores celery-ores-worker in scb1001 and scb1002
08:55 moritzm: !log installing libtasn security updates
08:38 moritzm: !log rolling restart of app server canaries for libtasn security update
07:22 moritzm: !log removed /var/log/logstash/logstash.1 on logstash1001, logspam (similar to the what is described in https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/144) depleted the space on the root partition
02:55 logmsgbot: !log l10nupdate@tin ResourceLoader cache refresh completed at Thu Jun 9 02:55:20 UTC 2016 (duration 6m 19s)
02:53 mutante: !log ms-be2012 ran out of disk due to huge syslog, deleted log, restarted rsyslogd
02:49 logmsgbot: !log mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.5) (duration: 11m 02s)
02:26 logmsgbot: !log mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.4) (duration: 11m 00s)
00:03 twentyafterfour: !log Preparing to deploy phabricator update. Tagged release/2016-06-08/1

2016-06-08

23:17 logmsgbot: !log maxsem@tin Synchronized php-1.28.0-wmf.5/extensions/WikimediaEvents/: https://gerrit.wikimedia.org/r/#/c/293439/ (duration: 00m 23s)
23:15 logmsgbot: !log maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/293438 (duration: 00m 25s)
23:07 logmsgbot: !log maxsem@tin Synchronized php-1.28.0-wmf.4/extensions/LiquidThreads/: https://gerrit.wikimedia.org/r/#/c/293247/ (duration: 00m 26s)
23:05 logmsgbot: !log maxsem@tin Synchronized php-1.28.0-wmf.5/extensions/LiquidThreads/: https://gerrit.wikimedia.org/r/#/c/293247/ (duration: 00m 26s)
22:51 hoo: !log Re-started dumpwikidatattl on snapshot1003
22:44 logmsgbot: tgr@tin Synchronized wmf-config/InitialiseSettings.php: enable AuthManager on group1 for reals T135504 (duration: 00m 25s)
22:27 logmsgbot: tgr@tin Synchronized wmf-config/InitialiseSettings.php: enable AuthManager on group1 T135504 (duration: 00m 23s)
22:21 logmsgbot: tgr@tin Synchronized php-1.28.0-wmf.5/extensions/OpenStackManager/: backport gerrit:293130 for AuthManager deploy T135504 (duration: 00m 28s)
22:05 ottomata: starting kafka broker on kafka1012 after swapping disk and copying data directory
22:01 logmsgbot: krinkle@tin Synchronized wmf-config/CommonSettings.php: Bump wgResourceLoaderStorageVersion (T134368) (duration: 00m 28s)
21:12 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.5
21:04 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.5/includes/specials/SpecialSearch.php: Add a visual clear to Special:Search input box and profile-tabs (duration: 00m 23s)
20:57 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.5/extensions/Renameuser/RenameuserSQL.php: Use master DB when touching the user to signal rename end (duration: 00m 22s)
20:50 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.5/includes/libs/objectcache/WANObjectCache.php: Avoid getWithSetCallback() warnings on unversioned key migration (duration: 00m 24s)
20:21 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.5/extensions/Kartographer/styles/kartographer.less: Fixed <maplink> autostyling (duration: 00m 26s)
20:18 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.4/extensions/Kartographer: late SWAT: Fix color extraction (duration: 00m 36s)
19:30 mobrovac: change-prop deploying 08a1b1d
19:27 hashar: gallium enabling puppet again now that zuul/jenkins are back
19:18 hashar: Bringing back Jenkins and Zuul on gallium T137265
18:59 logmsgbot: ori@palladium conftool action : set/pooled=yes; selector: name=scb1002.eqiad.wmnet
18:57 yurik: switched kartotherian to scap3, deployed, restarted
18:20 gehel: switching maps to scap3 deployment
16:50 jynus: cloning /var/lib/jenkins from db1085 to contint1001
16:46 ottomata: stopping kafka broker and puppet on kafka1012 to replace sdf
16:37 ottomata: powercycling scb1002
16:36 hashar: Disabled puppet on contint1001 to prevent it from bringing back Jenkins
16:32 logmsgbot: otto@palladium conftool action : set/pooled=no; selector: scb1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=mathoid'])
16:32 logmsgbot: otto@palladium conftool action : set/pooled=no; selector: scb1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
16:32 logmsgbot: otto@palladium conftool action : set/pooled=no; selector: scb1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=mobileapps'])
16:32 logmsgbot: otto@palladium conftool action : set/pooled=no; selector: scb1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=cxserver'])
16:32 logmsgbot: otto@palladium conftool action : set/pooled=no; selector: scb1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=citoid'])
16:32 logmsgbot: otto@palladium conftool action : set/pooled=no; selector: scb1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=graphoid'])
16:24 ottomata: restarting hadoop-yarn-resourcemanager on analytics1002 to make analytics1001 active
16:07 mobrovac: scb1002 enabling back puppet
16:02 elukey: temporary set a 10TB upperbound to the Kafka webrequest_text topic to free space (T136690)
15:43 ottomata: restarting zk in codfw and eqiad 1 by 1 to apply maxClientCnxns=1024
15:12 ottomata: restarting zookeeper 1 by 1 in eqiad
15:03 _joe_: contint1001: systemctl mask zuul,zuul-merger
14:57 elukey: rolling out the new Varnishkafka version in cache misc (didn't do it before since there was an outage ongoing)
14:53 jynus: rebooting gallium with netboot for hardware maintenance
14:44 mobrovac: scb1001 enabling and running puppet on scb1001
13:44 jynus: running fsck.ext3 /dev/sda2 in read-write mode for gallium
13:42 ottomata: powercycling scb2001 and scb2002
13:30 akosiaris: disabling puppet on scb1001 & scb1002
13:30 mobrovac: change-prop stopped on scb1002
13:29 akosiaris: stopping changeprop on scb1001
13:26 ottomata: powercycling scb1002
13:18 ottomata: powercycling scb1001
13:08 elukey: rolling out new varnishkafka package in cache misc
12:09 jynus: mounted temporarily / partition from gallium sda on db1085:/mnt
10:40 moritzm: uploaded jenkins 1.651.2 for jessie-wikimedia to carbon
10:13 elukey: rolling out the new varnishkafka package to cache maps
10:04 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.5/includes/deferred/LinksDeletionUpdate.php: fd44d649787ede78687b4cd2ef21e44a4c8b843b (duration: 00m 33s)
08:28 hashar: stopping Jenkins / zuul / zuul-merger / puppet on gallium
08:15 elukey: lowering down webrequest_text kafka topic retention time from 7 days to 4 days to free disk space (T136690)
08:14 hashar: Jenkins has bunch of executors dead for what ever reason preventing jobs from running :(
07:53 mobrovac: change-prop deploying 84d56e53a
06:59 moritzm: enabling ferm on palladium (will lead to temporary puppet failures)
02:58 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Jun 8 02:58:28 UTC 2016 (duration 6m 31s)
02:51 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.5) (duration: 06m 49s)
02:51 legoktm: / on gallium is currently read-only for some reason
02:29 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.4) (duration: 11m 11s)
00:11 awight_: update fundraising-tools from b2425aef2154d6b689900f4848cca02880321230 to 28bc2da677caa795c58f906db76a1f8d612ac899

2016-06-07

23:46 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.5/includes/deferred/LinksUpdate.php: 6d85caaa9bb5918cb2888fc82f2c7c346cf746a2 (duration: 00m 25s)
23:36 SMalyshev: redeploying WDQS to update the Updater for T128947 fix
23:35 logmsgbot: tgr@tin Synchronized wmf-config/InitialiseSettings.php: SWAT gerrit:292518 User rights configuration for meta. wmf-supportsafety group (duration: 00m 26s)
23:20 logmsgbot: tgr@tin Finished scap: (no message) (duration: 24m 51s)
23:02 awight: update paymentswiki from 28e10141454ef53085aed4c6619a34d3a4b43c58 to de11bfe2273d0bcaa0e713389b2d91e8b3567a1d; add PP cert
22:56 tgr: scapping AuthManager backports + feature switch enabled on group0 T135504
22:56 logmsgbot: tgr@tin Started scap: (no message)
22:10 mutante: icinga config broken: Error: Could not find any host matching 'relforge1001'
21:35 twentyafterfour: restarted apache on iridium to deploy D250
20:02 andrewbogott: dist-upgrade on labvirt1010, in hopes of resolving a nova-compute lockup (possibly related to a kvm upgrade earlier today)
20:00 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.28.0-wmf.5
19:44 jynus: restarting es2017 due to a bunch of ACPI errors (probably memory-caused)
19:35 logmsgbot: thcipriani@tin Finished scap: testwiki to php-1.28.0-wmf.5 and rebuild l10n cache (duration: 26m 40s)
19:08 logmsgbot: thcipriani@tin Started scap: testwiki to php-1.28.0-wmf.5 and rebuild l10n cache
18:30 andrewbogott: rebooting labvirt1011
17:51 ottomata: restarting broker on kafka1020
17:44 Dereckson: `mwscript initSiteStats.php --wiki kshwiki --update` on Terbium (T137234)
17:33 mutante: furud - shutdown, decom, deleteV VM
17:30 ejegg: updated payments-wiki from 3df3329f75fdbc679baf37bfd3955880091b3ae1 to 28e10141454ef53085aed4c6619a34d3a4b43c58
17:06 logmsgbot: krinkle@tin Synchronized wmf-config/CommonSettings.php: clean-up
17:05 ejegg: rolled back payments-wiki to 3df3329f75fdbc679baf37bfd3955880091b3ae1
17:04 thcipriani: starting branch-cut for mediawiki and extensions for version 1.28.0-wmf.5
17:04 ejegg: updated payments-wiki from 3df3329f75fdbc679baf37bfd3955880091b3ae1 to 413bd3ea92ac570c081532c71891c31391194984
16:01 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Update audit hooks for AuthManager (duration: 00m 24s)
15:53 logmsgbot: thcipriani@tin Synchronized wmf-config/wikitech.php: SWAT: Do not set wgAuth to LdapAuth when AuthManager is enabled (duration: 00m 23s)
15:48 logmsgbot: thcipriani@tin Synchronized portals: SWAT: T135902 adding readme and license to wikipedia.org portal (duration: 00m 25s)
15:48 logmsgbot: thcipriani@tin Synchronized portals/prod/wikipedia.org/assets: SWAT: T135902 adding readme and license to wikipedia.org portal (duration: 00m 25s)
15:41 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: huwiki: Enable Popups A/B test for 50% of users (duration: 00m 24s)
15:32 logmsgbot: thcipriani@tin Synchronized wmf-config: SWAT: Revert "Send wmf.4 search and ttmserver traffic to codfw" (duration: 00m 26s)
15:24 logmsgbot: thcipriani@tin Synchronized wmf-config/PrivateSettings.php: SWAT: Use bot password for TNBot after touch wmf-config/PrivateSettings.php (duration: 00m 25s)
15:16 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Use bot password for TNBot (duration: 00m 34s)
15:15 logmsgbot: thcipriani@tin Synchronized private/PrivateSettings.php: SWAT: password update for Translation Notification Bot (duration: 00m 41s)
14:47 elukey: installing varnishkafka 1.0.10-1 on cp1046 manually to test the new version.
14:23 jynus: stopping mysql and the OS @ es2017 for hardware maintenance
13:53 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Pool new s5 db hosts: db1082, db1087, db1092 with low weight (duration: 00m 23s)
13:52 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Add new coredb servers to alias configuration (duration: 00m 38s)
13:49 jynus: about to pool new dewiki/wikidata servers T133398
12:27 moritzm: rolling out gdk-pixbuf security updates
12:23 moritzm: rolling restart of sca cluster for libxml2 security update
11:27 moritzm: restarting apache2 on californium (hosting horizon dashboard) for libxml2 update
11:23 moritzm: restarting apache2 on silver (hosting wikitech) for libxml2 update
11:08 hashar: restarted apache2 on gallium for libxml2 update
10:53 moritzm: restarting apache2 on iridium (hosting Phabricator) for libxml2 update
10:18 moritzm: rolling restart of hhvm on eqiad appservers to pick up libxml2 update
09:09 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1070 after maintenance (duration: 00m 27s)
09:04 hashar: Upgrading Jenkins IRC plugin 2.25..2.27 and instant messaging plugin 1.34..1.35 . The former should fix a deadlock on shutdowning Jenkins
09:00 moritzm: rolling restart of hhvm on codfw appservers to pick up libxml2 update
08:53 moritzm: rolling restart of hhvm on appserver canaries to pick up libxml2 update
08:28 moritzm: deploying libxml2 security updates on Ubuntu systems (Debian systems already upgraded last week)
07:19 jynus: stopping and cloning db1070 to new s5 servers
07:08 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1070 for cloning (duration: 00m 29s)
02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Jun 7 02:30:57 UTC 2016 (duration 5m 32s)
02:25 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.4) (duration: 09m 36s)
01:10 logmsgbot: aude@tin Synchronized php-1.28.0-wmf.4/extensions/Wikidata: Fix bug (T136093) in display of labels after edit (duration: 02m 03s)
00:39 Krenair: (TXT record for SPF, actually)
00:39 Krenair: Created MX and SPF records directly for wmflabs.org. for https://phabricator.wikimedia.org/T137160#2359786
00:35 ejegg: updated settings on payments-wiki
00:26 logmsgbot: ori@tin Synchronized php-1.28.0-wmf.4/extensions/CentralAuth/includes/CentralAuthHooks.php: I79cbb1dc: Prefetch $wgCentralAuthLoginWiki DNS (T92864) (duration: 00m 29s)

2016-06-06

23:41 logmsgbot: maxsem@tin Synchronized wmf-config/abusefilter.php: https://gerrit.wikimedia.org/r/#/c/292758/ (duration: 00m 24s)
23:32 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.4/extensions/GeoData/: (no message) (duration: 00m 25s)
23:29 logmsgbot: maxsem@tin Synchronized private/PrivateSettings.php: Updated Zero password (duration: 00m 25s)
23:21 Amir1: deploying ae71d84 into ores in prod
23:17 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/293037/ (duration: 00m 24s)
23:14 logmsgbot: maxsem@tin Synchronized portals: https://gerrit.wikimedia.org/r/#/c/292992/ (duration: 00m 31s)
23:13 logmsgbot: maxsem@tin Synchronized portals/prod/wikipedia.org/assets: https://gerrit.wikimedia.org/r/#/c/292992/ (duration: 00m 30s)
23:04 logmsgbot: maxsem@tin Synchronized docroot/wikipedia.org/.well-known/apple-app-site-association: https://gerrit.wikimedia.org/r/#q,287190,n,z (duration: 00m 25s)
22:05 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.4/includes/api/ApiStashEdit.php: 50ce579046e07 (duration: 00m 23s)
20:25 arlolra: updated Parsoid to version e8d6092e
20:09 arlolra: starting Parsoid deploy
19:15 ottomata: restarting kafka broker on kafka1020 to test python consumption client
19:12 bblack: restarted nginx on rcs1002 (was stuck half-shut-down for reload?), started nginx on rcs1001 (wasn't running at all)
19:08 mutante: ran puppet on carbon because icinga said fail, saw it change STS headers, but no fail
19:06 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.4/includes/page/WikiPage.php: 661c22db3a352 (duration: 00m 30s)
18:08 ori: Running rebuildrecentchanges.php for test2wiki for T133225
17:14 gehel: deploying latest GUI for wikidata query service
16:58 logmsgbot: tgr@tin Synchronized wmf-config/PrivateSettings.php: (no message) (duration: 00m 23s)
16:57 logmsgbot: tgr@tin Synchronized private/PrivateSettings.php: (no message) (duration: 00m 23s)
16:44 logmsgbot: tgr@tin Synchronized wmf-config/PrivateSettings.php: (no message) (duration: 00m 23s)
16:39 tgr: PrivateSettings changes were for T135074
16:39 logmsgbot: tgr@tin Synchronized wmf-config/PrivateSettings.php: (no message) (duration: 00m 27s)
16:37 logmsgbot: tgr@tin Synchronized private/PrivateSettings.php: (no message) (duration: 00m 26s)
16:23 _joe_: rebooting mw1262
16:22 logmsgbot: tgr@tin Synchronized wmf-config/CommonSettings.php: creating zeroscript grant group on zerowiki, gerrit: 292951 (duration: 00m 28s)
16:00 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Math: Set wgMathFullRestbaseURL to point to wikimedia.org in production (duration: 00m 24s)
15:45 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: ULS: Stop using /static/current (duration: 00m 24s)
15:37 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.4/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.MobileArticleTarget.js: SWAT: Fix config of mobile surfaces (duration: 00m 24s)
15:32 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Use wfLoadExtension for LocalisationUpdate (duration: 00m 27s)
15:21 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT Switch Wikivoyages to Single Edit Tab mode for VE Beta Feature (duration: 00m 24s)
15:14 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor by default for logged-in users on four Wikipedias PART II (duration: 00m 30s)
15:14 logmsgbot: thcipriani@tin Synchronized dblists/visualeditor-default.dblist: SWAT: Enable VisualEditor by default for logged-in users on four Wikipedias PART I (duration: 00m 29s)
15:04 jynus: dropping old outreach databases on m1
14:10 jynus: dropping old bugzilla databases from m1
14:00 jynus: dropping database blog from m1
12:34 hashar: restarted Jenkins, deadlock in IRC plugin
10:46 elukey: re-added kafka1001 to eventbus.svc.eqiad.wmflabs without rebooting since some concerns were raised from the Services team. Will have a discussion with them before proceeding.
10:45 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: kafka1001.eqiad.wmnet
10:33 moritzm: installing perl updates (bugfixes and CVE-2015-8853)
10:27 logmsgbot: elukey@palladium conftool action : set/pooled=no; selector: kafka1001.eqiad.wmnet
10:25 elukey: rebooting kafka100[12] for kernel upgrades (one at the time with de-pool/re-pool actions)
09:12 moritzm: installing dpkg bugfix updates on jessie systems
08:45 mobrovac: change-prop deployed 9b04e475
08:27 gehel: lowering elasticsearch high watermark on eqiad cluster to rebalance disk space
08:17 _joe_: rebooting mw1262
07:57 jynus: enabling GTID on pending coredb servers on eqiad
06:18 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.4/includes/cache/LinkBatch.php: c2ba764f38e44e7 (duration: 00m 30s)
05:34 robh: db2034 locked up via serial console. details on T137084, rebooting since its unresponsive to ssh or serial.
02:28 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Jun 6 02:28:50 UTC 2016 (duration 5m 56s)
02:22 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.4) (duration: 09m 34s)

2016-06-05

14:55 Dereckson: `mwscript initSiteStats.php --wiki csbwiki --update` (T137060)
02:27 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Jun 5 02:27:38 UTC 2016 (duration 5m 35s)
02:22 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.4) (duration: 08m 55s)

2016-06-04

20:18 apergos: rebooting mw1135, unresponsive to ssh or console login
09:51 elukey: restarted hhvm on mw1144 after the host was hanging (OOM killer restored basic host functionalities but not hhvm)
09:47 elukey: removed temporary Analytics Kafka upload retention override
09:38 elukey: Lowering down temporarily the Analytics kafka upload retention time to 24h to free space (T136690)
02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Jun 4 02:30:50 UTC 2016 (duration 5m 39s)
02:25 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.4) (duration: 09m 08s)

2016-06-03

22:57 Krinkle: Purged https://en.wikipedia.org/static/images/project-logos/bawiki.png
22:53 logmsgbot: krinkle@tin Synchronized static/images/project-logos/bawiki.png: (no message) (duration: 00m 24s)
21:57 YuviPanda: started copying graphite data from usb back
21:27 awight: update paymentswiki from 28b98ec254b2a15c8df61c568b62f221b328222f to 3df3329f75fdbc679baf37bfd3955880091b3ae1
20:47 ejegg: updated payments-wiki de86eadcd98922ee4207a0c46112585f3ba5c48d to 28b98ec254b2a15c8df61c568b62f221b328222f
20:25 ejegg: updated GatewayReady hook on paymentswiki
19:37 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.4/extensions/WikimediaEvents/extension.json: T136920 (duration: 00m 28s)
19:04 mutante: releases apt repo on bromine: export fresh jessie-mediawiki indexes
17:41 mutante: uploaded parsoid 0.5.1 to releases
17:14 robh: bast4001 coming down for second hdd installation. (there are currently no active users on system)
16:58 mutante: magnesium - shutdown -h now, bye
15:30 logmsgbot: tgr@tin Finished scap: revert AbuseFilter + config to pre-extension-registration state T136929 (duration: 06m 13s)
15:24 logmsgbot: tgr@tin Started scap: revert AbuseFilter + config to pre-extension-registration state T136929
14:38 gehel: un-freezing writes from CirrusSearch to eqiad cluster during upgrade (T133126)
13:27 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: kafka2001.codfw.wmnet
13:22 logmsgbot: elukey@palladium conftool action : set/pooled=no; selector: kafka2001.codfw.wmnet
13:22 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: kafka2002.codfw.wmnet
13:16 hasharAway: Reenabling puppet on gallium. Forgot to put it back yesterday
13:14 logmsgbot: elukey@palladium conftool action : set/pooled=no; selector: kafka2002.codfw.wmnet
13:11 elukey: rebooting kafka200[12] (codfw EventBus) for kernel upgrades
11:18 gehel: freezing writes from CirrusSearch to eqiad clsuter during upgrade (T133126)
10:48 gehel: taking elasticsearch eqiad cluster down for upgrade to 2.3 (T133126)
10:39 gehel: Starting upgrade of elasticsearch eqiad cluster to 2.3 (T133126)
10:35 moritzm: restarting apache on bohrium (serving piwik.wikimedia.org) for libxml2 security update
10:23 moritzm: restarting apache on planet1001 (serving planet.wikimedia.org) for libxml2 security update
08:42 moritzm: rolling restart of scb cluster (mathoid, ores-uwsgi) in eqiad to pick up libxml2 security updates
08:38 jynus: archiving again syslog.1 from ms-be2012 on /srv/swift-storage/sdl1/tmp
08:35 jynus: created new LDAP group grafana-admin, gid=1007
08:34 elukey: rebooting kafka1012 for kernel upgrades.
08:08 moritzm: installing libxml2 security updates on jessie systems
07:19 kart_: Update cxserver to 19a71f1
06:29 moritzm: installing nginx security updates on Ubuntu systems (Debian installs updated some days ago)
02:36 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Jun 3 02:36:39 UTC 2016 (duration 5m 58s)
02:30 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.4) (duration: 08m 35s)
01:09 mutante: bromine - puppet currently stopped needs some permission fixes for release upload
01:08 mutante: uploaded parsoid 0.5.0 deb to releases.wm.org
00:24 logmsgbot: awight@tin Finished scap: Deploying labtestwiki AuthManager config; Enabling Popups experiment; CentralNotice fixes for T136408, T136387; Special:Notifications fixes (duration: 25m 08s)

2016-06-02

23:59 logmsgbot: awight@tin Started scap: Deploying labtestwiki AuthManager config; Enabling Popups experiment; CentralNotice fixes for T136408, T136387; Special:Notifications fixes
23:32 logmsgbot: awight@tin Synchronized wmf-config/InitialiseSettings.php: Add namespace translation 'Portal' for diq (duration: 00m 24s)
23:28 logmsgbot: awight@tin Synchronized wmf-config/InitialiseSettings.php: Enable AuthManager on beta wikitech (duration: 00m 25s)
23:24 logmsgbot: awight@tin Synchronized wmf-config/InitialiseSettings.php: Enable Hovercards experiment for 1% of users on huwiki (duration: 00m 24s)
23:23 logmsgbot: awight@tin Synchronized php-1.28.0-wmf.4/extensions/Popups: Do not show Hovercards when NavPopups gadget is enabled on huwiki (duration: 00m 24s)
23:21 logmsgbot: awight@tin Synchronized wmf-config/extension-list-labs: Test PageAssessments on Beta Labs (duration: 00m 25s)
23:20 logmsgbot: awight@tin Synchronized wmf-config/InitialiseSettings-labs.php: Test PageAssessments on Beta Labs (duration: 00m 26s)
23:20 logmsgbot: awight@tin Synchronized wmf-config/CommonSettings-labs.php: Test PageAssessments on Beta Labs (duration: 00m 24s)
22:37 logmsgbot: ori@tin Synchronized wmf-config/InitialiseSettings.php: I9dc532b3: Enable "purge" log group (duration: 00m 42s)
22:20 mutante: removed my gerrit admin flag
20:20 mutante: magnesium (formerly RT) remove from puppet and icinga, revoked cert and salt key, just waiting another day or before shutdown
20:18 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: logging: disable Wikibase\Client\Changes\WikiPageUpdater channel (duration: 00m 26s)
20:12 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.4
19:53 ottomata: stopping kafka broker and restarting kafka1014
19:52 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.4/extensions/CheckUser/specials/SpecialCheckUser.php: Fix Special:Checkuser for log entries when cuc_title = "" (duration: 00m 31s)
19:37 ejegg: re-enabled adyen job runner
19:35 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: scb2002.codfw.wmnet (tags: ['dc=codfw', 'cluster=scb', 'service=ores'])
19:35 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: scb2001.codfw.wmnet (tags: ['dc=codfw', 'cluster=scb', 'service=ores'])
19:35 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: scb1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
19:35 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: scb1001.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
18:57 ejegg: disabled adyen job runner
18:47 jynus: restarting replication on db1016
18:43 YuviPanda: powercycle labmon1001 again, get into bios
18:29 YuviPanda: going to try to intentionally trip the NFS check on tools-checker. This will not page
18:24 YuviPanda: powercycle labmon1001 again
18:19 mutante: db2007, revoke puppet cert, delete salt key, nuke from stored configs / icinga
18:19 bearND: mobileapps deployed b2fee30
18:18 mutante: db2007 shutdown, schedule eternal downtime
18:04 bearND: starting mobileapps deploy
17:40 subbu: finished deploying parsoid version 7188080b
17:34 subbu: synced new code; restarted parsoid on wtp1001 as a canary
17:29 subbu: starting deploy of new parsoid code
17:21 mutante: ran ALTER TABLE character set utf8 .. (https://phabricator.wikimedia.org/T119112#2311402) on RT db
17:16 mutante: running RT database upgrade from 4.0.4 to 4.2.8
17:13 awight: update paymentswiki from d26426c4225080c95f0bd5a6a31c54e4826287b1 to de86eadcd98922ee4207a0c46112585f3ba5c48d
17:05 mutante: stopped exim on magnesium
17:05 jynus: stopping replication from db1001 to db1016 (pasive m1 node) before schema change
16:52 mutante: magnesium (RT), tmp. stopped RT and puppet
16:50 YuviPanda: begin reinstall of labmon1001
15:19 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.4/extensions/Math: SWAT: Use img instead of meta tags for SVGs and Fix iterator in batchGetMathML (duration: 00m 28s)
15:12 logmsgbot: thcipriani@tin Synchronized portals: deploying new localized top-links on wikipedia.org (duration: 00m 31s)
15:11 logmsgbot: thcipriani@tin Synchronized portals/prod/wikipedia.org/assets: deploying new localized top-links on wikipedia.org (duration: 00m 32s)
14:33 jynus: acked ores icinga checks on some scb hosts and pointing to T124201 (it seems the checks arrived before the actual setup)
13:52 moritzm: installing imagemagick security updates on Ubuntu systems (but affected decoders already neutralised by policy changes) (also Debian systems already addressed)
13:34 hashar: Downgrading Zuul back to zuul_2.1.0-95-g66c8e52-wmf1precise1_amd64.deb . Paramiko cant acquire ssh connection with Gerrit for some reason... https://phabricator.wikimedia.org/P3204
12:10 hashar: Upgraded Zuul upstream code being 66c8e52..30a433b package is 2.1.0-151-g30a433b-wmf1precise1
11:39 logmsgbot: jmm@tin Synchronized wmf-config/CommonSettings.php: disable firejail security hardening for image scalers, needs more work for the Score extension (duration: 00m 36s)
10:55 hashar: Restarted Zuul and reenabled puppet on gallium
10:50 hashar: gallium: stopped puppet agent
10:49 hashar: gracefully stopping Zuul, will upgrade / take traces etc over the next half hour or so
10:14 jynus: archiving again syslog.1 from ms-be2012 on /srv/swift-storage/sdl1/tmp
10:08 mobrovac: restbase enabling puppet back in production
08:40 mobrovac: restbase deploy end of 19f25925
08:29 mobrovac: restbase deploy start of 19f25925
08:09 mobrovac: restbase disabling puppet in production for testing https://gerrit.wikimedia.org/r/#/c/292109/ in staging
07:23 moritzm: rebooting etherpad1001 (hosting etherpad.wikimedia.org) for upgrade to Linux 4.4
07:02 jynus: performing schema change for db1057
03:04 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Jun 2 03:04:44 UTC 2016 (duration 6m 40s)
02:58 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.4) (duration: 15m 37s)
02:24 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.3) (duration: 10m 06s)
01:52 mutante: scb1001/2001 ores - connection refused
01:52 mutante: mw1136 service hhvm restart
01:37 mutante: labsdb1001 /etc/init.d/mysql start
01:32 YuviPanda: service mysql start on labsdb1001
01:25 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Set $wgSpamBlacklistEventLogging to true on testwiki (duration: 00m 22s)
01:25 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Set $wgSpamBlacklistEventLogging to true on testwiki (duration: 00m 23s)
01:23 YuviPanda: reboot labsdb1001
01:21 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.4/extensions/Flow/handlebars/: HACK: Hide reply form for locked topics (T135848) (duration: 00m 24s)
01:19 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.4/extensions/Echo/includes/special/NotificationPager.php: Fix notification pager (T136759) (duration: 00m 25s)
01:18 YuviPanda: restart mysql on labsdb1001
01:00 bearND: mobileapps reverted to 8d6d648c943074b7d3999baf31d60ad99249cd51
00:55 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings-labs.php: Revert "Test PageAssessments extension on Labs" (no-op) (duration: 00m 22s)
00:55 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings-labs.php: Revert "Test PageAssessments extension on Labs" (no-op) (duration: 00m 23s)
00:26 logmsgbot: awight@tin Synchronized php-1.28.0-wmf.4/extensions/CentralNotice: Fix for T136387 (duration: 00m 38s)
00:05 urandom: Deploy of cdff5e3 to RESTBase production complete
00:03 YuviPanda: started nfs-exports on labstore1001

2016-06-01

23:57 urandom: Deploying cdff5e3 to RESTBase production
23:51 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Revert Use extension registration for SpamBlacklist (T119117) (duration: 00m 24s)
23:49 urandom: Deploying cdff5e3 to restbase1008.eqiad.wmnet (canary node)
23:44 urandom: Deploy of RESTBase to staging environment complete
23:40 urandom: Deploying RESTBase to staging environment
23:39 urandom: RESTBase deploy to xenon.eqiad.wmnet (canary node) complete
23:38 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings-labs.php: Test PageAssessments extension on Labs (no-op) (duration: 00m 26s)
23:37 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings-labs.php: Test PageAssessments extension on Labs (no-op) (duration: 00m 30s)
23:36 urandom: Deploying RESTBase to xenon.eqiad.wmnet (canary node)
23:26 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.4/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.js: Simplify teardown of toolbar save button (T136421) (duration: 00m 23s)
23:21 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Use full URL in $wgNoticeHideUrls (T130442) (duration: 00m 23s)
23:17 urandom: Deploying d8fa5c0 to RESTBase production
23:10 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Use HTTPS URL to citoid instead of protocol-relative (T136423) (duration: 00m 32s)
23:06 urandom: Update restbase staging to f05b66f
22:36 cwd: updated paymentswiki from 44bd699d6700ac4faf3c2d772ba713b093ae8cb8 to d26426c4225080c95f0bd5a6a31c54e4826287b1
22:30 logmsgbot: twentyafterfour@tin Synchronized php-1.28.0-wmf.4/extensions/CentralNotice/: deploy https://gerrit.wikimedia.org/r/#/c/292279/ (duration: 00m 26s)
21:38 twentyafterfour: train has left the station
21:37 logmsgbot: twentyafterfour@tin Synchronized wmf-config/InitialiseSettings.php: deploy /wmf-config/InitialiseSettings.php for eranroz ( T132972 ) (duration: 00m 25s)
21:31 logmsgbot: twentyafterfour@tin Synchronized php-1.28.0-wmf.4/includes/specials/SpecialPrefixindex.php: sync https://gerrit.wikimedia.org/r/#/c/292228/ ( T136738 ) (duration: 00m 26s)
21:26 logmsgbot: twentyafterfour@tin Synchronized php-1.28.0-wmf.3/includes/specials/SpecialPrefixindex.php: sync https://gerrit.wikimedia.org/r/#/c/292234/ ( T136738 ) (duration: 00m 30s)
20:58 bearND: mobileapps deployed ed0e2e4
20:56 gehel: restarting postgresql on maps2001
20:55 bearND: starting mobileapps deploy
20:49 ejegg: updated paymentswiki from 7d222320b35ad8a44d8c77a4c3019364a49e53f2 to 44bd699d6700ac4faf3c2d772ba713b093ae8cb8
20:44 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.4
20:39 logmsgbot: twentyafterfour@tin Synchronized php-1.28.0-wmf.4/includes/cache/LinkBatch.php: deploy https://gerrit.wikimedia.org/r/#/c/292217/ (duration: 00m 27s)
20:17 subbu: finished deploying parsoid sha afb0d522
20:17 urandom: Rolling restart of RESTBase (redistribute Cassandra client connections?) : T126629
20:10 subbu: synced new code; restarted parsoid on wtp1001 as a canary
20:07 subbu: starting parsoid deploy
19:43 ema: cp* hosts rebooted (T131928)
19:40 bblack: restarting pybals for healthcheck config changes
18:25 urandom: restarting Cassandra on restbase1007.eqiad.wmnet
18:19 ejegg: updated payments-wiki from 5bb160e9898224e1d7d0a5c57fe408edb998a262 to 7d222320b35ad8a44d8c77a4c3019364a49e53f2
18:16 ottomata: stopping kafka broker on kafka1018 and rebooting node
17:51 urandom: Restarting Cassandra on restbase1007.eqiad.wmnet : T126629
17:48 ema: depooled reboot of cp4* hosts (T131928)
17:47 urandom: Temporarily disabling puppet to test setting on restbase1007.eqiad.wmnet : T126629
17:15 ejegg: rolled back payments-wiki from a335a3a6f8909d1e7e1a79877512a12a0561aa2a to 5bb160e9898224e1d7d0a5c57fe408edb998a262
17:06 ejegg: updated payments-wiki from 5bb160e9898224e1d7d0a5c57fe408edb998a262 to a335a3a6f8909d1e7e1a79877512a12a0561aa2a
17:05 akosiaris: powered on lvs2006. disk change did not happen
17:05 akosiaris: powered off lvs2006 for disk swap
16:54 logmsgbot: tgr@tin Synchronized wmf-config/InitialiseSettings-labs.php: T135504: enable AuthManager in beta (duration: 00m 32s)
16:39 logmsgbot: tgr@tin Synchronized php-1.28.0-wmf.4/extensions/NewUserMessage/: backport gerrit:292168 to update NewUserMessage for AuthManager (duration: 00m 29s)
16:22 urandom: Disabling traces on restbase1008-a.eqiad.wmnet : T126629
16:01 logmsgbot: thcipriani@tin Finished scap: SWAT: Update for AuthManager (duration: 26m 05s)
15:58 urandom: Setting trace probability on restbase1008-a.eqiad.wmnet to 5% : T126629
15:58 jynus: updating dns entry for db1080.eqiad.wment
15:58 urandom: Disabling trace probability on restbase1007-a.eqiad.wmnet : T126629
15:48 urandom: Setting trace probability to 5% on restbase1007-a.eqiad.wmnet : T126629
15:35 logmsgbot: thcipriani@tin Started scap: SWAT: Update for AuthManager
15:33 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.4/resources/src/moment-locale-overrides.js: SWAT: Avoid passing integers to mw.RegExp.escape (duration: 00m 24s)
15:29 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Remove centralauth-autoaccount right (duration: 00m 25s)
15:26 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable bot passwords on zerowiki (duration: 00m 24s)
15:19 paravoid: Re-enabling OSPF on all cr1-codfw row subnets
15:18 paravoid: Re-enabling cr1-codfw et-0/* interfaces
15:18 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Enable RC patrol on ta.wikiquote" (duration: 00m 25s)
15:15 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove no longer used Echo configuration PART II (duration: 00m 26s)
15:14 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Remove no longer used Echo configuration PART I (duration: 00m 33s)
15:13 paravoid: Rebooting cr1-codfw FPC 0
15:09 paravoid: Upgrading cr1-codfw FPC 0 all PICs firmware
15:08 logmsgbot: thcipriani@tin Synchronized static/images/sul: SWAT: Make SUL icons square and use global defaults (duration: 00m 41s)
15:07 paravoid: Disabling cr1-codfw et-0/* (all row uplinks)
15:03 akosiaris: restarted grrrit-wm after gerrit restart
15:03 paravoid: Disabling OSPF on all cr1-codfw row subnets to drain FPC0
15:02 akosiaris: restarted gerrit to enforce 100m maxObjectSizeLimit
14:59 paravoid: Restoring VRRP priority on cr2-codfw
14:57 bblack: depooled reboot of cp3048 (T131928)
14:57 paravoid: Re-enabling OSPF on all cr2-codfw row subnets
14:54 paravoid: Re-enabling cr2-codfw et-0/* interfaces
14:49 paravoid: Rebooting cr2-codfw FPC 0
14:48 paravoid: Upgrading cr2-codfw FPC 0 all PICs firmware
14:42 paravoid: Disabling cr2-codfw et-0/2/0, et-0/2/1 (row C/D uplinks)
14:34 paravoid: Disabling cr2-codfw et-0/0/0 (row A uplink)
14:29 paravoid: Disabling cr2-codfw et-0/0/1 (row B uplink)
14:15 paravoid: Disabling OSPF on all cr2-codfw row subnets to drain FPC0
14:08 ema: depooled reboot of cp1* hosts (T131928)
12:49 paravoid: draining cr2-codfw for firmware upgrade
12:26 bblack: upgrade nginx to 1.11.1-1+wmf1 on all clusters
11:50 elukey: rebooting kafka1022 for kernel upgrade (4.4)
11:05 ema: rebooting cp3* spares (T131928)
10:47 Dereckson: Script done for uca-it collation on itwiki: 10 599 758 rows processed
10:47 ema: depooled reboot of cp3046 (T131928)
10:47 ema: depooled reboot of cp3003 (T131928)
10:45 ema: depooled reboot of cp3034 (T131928)
10:39 ema: depooled reboot of cp3005 (T131928)
10:38 ema: depooled reboot of cp3044 (T131928)
10:35 ema: depooled reboot of cp3047 (T131928)
10:31 ema: depooled reboot of cp3004 (T131928)
10:28 ema: depooled reboot of cp3009 (T131928)
10:14 ema: depooled reboot of cp3037 (T131928)
10:11 jynus: moved syslog1 to ms-be2012:/srv/swift-storage/sdl1/tmp to avoid / fillup
10:10 ema: depooled reboot of cp3008 (T131928)
10:09 ema: depooled reboot of cp3035 (T131928)
09:37 moritzm: installing libgd security updates
09:28 ema: depooled reboot of cp3039 (T131928)
09:23 ema: depooled reboot of cp3045 (T131928)
09:21 ema: depooled reboot of cp3010 (T131928)
09:18 ema: depooled reboot of cp3006 (T131928)
09:16 ema: depooled reboot of cp3007 (T131928)
09:10 ema: depooled reboot of cp3036 (T131928)
08:25 mobrovac: mobileapps deploying 8d6d648
08:24 ema: depooled reboot of cp3049 (T131928)
08:22 hashar: Nodepool came back up just fine after labnodepool1001 reboot and is fully operational.
08:15 jynus: deleting mysql logrotate scripts to avoid root spam
08:14 moritzm: reboot labnodepool1001 for update to Linux 4.4
07:56 elukey: event logging restarted on eventlog1001.eqiad.wmnet
07:46 elukey: stopping kafka on kafka1020.eqiad and rebooting the host for Linux 4.4 upgrades
07:43 moritzm: rolling reboot of scb in eqiad for update to Linux 4.4
07:32 moritzm: restarted hhvm on mw1180
07:05 mobrovac: change-prop restarting to apply https://gerrit.wikimedia.org/r/291201
05:41 mobrovac: restbase deploy end of 5c99693
05:26 mobrovac: restbase deploy start of 5c99693
04:31 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.4/includes/: reapplied new version of I03739e94 (duration: 01m 21s)
04:27 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.3/includes/: reapplied new version of I03739e94 (duration: 01m 34s)
03:11 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Jun 1 03:11:11 UTC 2016 (duration 6m 39s)
03:04 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.4) (duration: 15m 42s)
02:30 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.3) (duration: 09m 30s)
00:04 Dereckson: Started `mwscript updateCollation.php itwiki --previous-collation=uppercase` on Terbium (T136647)