Server Admin Log

From Wikitech
(Redirected from Server admin log)
Jump to: navigation, search

2017-02-23

  • 03:02 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Feb 23 03:02:10 UTC 2017 (duration 5m 47s)
  • 02:56 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.13) (duration: 14m 38s)
  • 02:23 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.12) (duration: 08m 40s)
  • 00:19 dereckson@tin: Synchronized wmf-config/throttle.php: Throttle rule for it.wikiversity (T158767) (duration: 00m 40s)
  • 00:18 Krinkle: mwscript deleteEqualMessages.php --wiki simplewikibooks (T45917)

2017-02-22

  • 23:46 ejegg: turned off 3DS requirement for Denmark on payments-wiki
  • 23:17 matt_flaschen: Exported https://meta.wikimedia.org/wiki/Talk:Flow/Developer_test_page to https://meta.wikimedia.org/wiki/Talk:Flow/Developer_test_page/Wikitext using extensions/Flow/maintenance/convertToText.php
  • 23:17 matt_flaschen: Migrated https://meta.wikimedia.org/wiki/Research_talk:ORES_paper to https://www.mediawiki.org/wiki/Talk:ORES/Paper using extensions/Flow/maintenance/dumpBackup.php and importDump.php
  • 22:53 Pchelolo: update RESTBase to 3340714f0
  • 22:52 jynus: stopping dbstore1001 mariadb in preparation for tomorrow's reimage T153768
  • 22:50 Pchelolo: update RESTBase to 3340714f0: canary on restbase1007
  • 22:46 Pchelolo: update RESTBase to 3340714f0: staging
  • 21:57 maxsem@tin: Finished deploy [kartotherian/deploy@81db48c]: Deploying https://gerrit.wikimedia.org/r/#/c/339093/ (duration: 15m 05s)
  • 21:42 maxsem@tin: Started deploy [kartotherian/deploy@81db48c]: Deploying https://gerrit.wikimedia.org/r/#/c/339093/
  • 20:30 demon@tin: Finished scap: group1 to wmf.13 (duration: 25m 39s)
  • 20:04 demon@tin: Started scap: group1 to wmf.13
  • 20:02 gehel@tin: Finished deploy [wdqs/wdqs@7768422]: (no justification provided) (duration: 02m 04s)
  • 19:59 gehel@tin: Started deploy [wdqs/wdqs@7768422]: (no justification provided)
  • 19:56 gehel: deploying latest wdqs version
  • 19:46 godog: roll-HUP rsyslog on mw1* to pick up DNS udplog change - T123728
  • 19:45 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Finish removing "shellmanagers" on Wikitech T158482 (duration: 00m 40s)
  • 19:37 thcipriani@tin: Synchronized php-1.29.0-wmf.13/extensions/Flow: SWAT: Import dump: support importing a board that exist in the farm T154830 (duration: 00m 56s)
  • 19:34 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Removing the "shellmanagers" group from Wikitech T158482 (duration: 00m 49s)
  • 19:14 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Configuration changes for wikitech.wikimedia.org T158516 T158554 T158482 (duration: 00m 40s)
  • 18:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1060 with less weight - T158194 (duration: 00m 39s)
  • 18:17 Dereckson: Last two deployment entries were to rollback portals/ to last known state (T158782)
  • 18:17 dereckson@tin: Synchronized portals: (no justification provided) (duration: 00m 39s)
  • 18:17 dereckson@tin: Synchronized portals/prod/wikipedia.org/assets: (no justification provided) (duration: 00m 40s)
  • 16:29 gehel: reimage of relforge1001 starting
  • 16:21 marostegui: Shutdown db1060 for BBU replacement - T158194
  • 16:20 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1060 - T158194 (duration: 00m 40s)
  • 16:19 ema: cp3006 upgraded to varnish 4.1.5
  • 16:15 ema: cp4019 upgraded to varnish 4.1.5
  • 15:48 moritzm: installing tcpdump security updates on ubuntu systems (jessie already fixed for a while)
  • 15:43 jynus: stopping mariadb replication on db1026 for maintenance T147747
  • 15:21 marostegui: Restart MySQL on db1095 to apply new replication filters - https://phabricator.wikimedia.org/T154485
  • 15:16 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1026 for maintenance (duration: 00m 41s)
  • 15:11 marostegui: Restart MySQL on db1069 to apply new replication filters - https://phabricator.wikimedia.org/T154485
  • 14:50 zeljkof: finished EU SWAT
  • 14:49 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: New throttle rule (T158762) (duration: 00m 41s)
  • 14:30 gehel: resetting to usual values for low/high watermark on elasticsearch eqiad (75% / 80%)
  • 14:17 hashar: Nuked Jenkins workspaces for the job operations-puppet-typos
  • 14:17 zfilipin@tin: Synchronized dblists/compact-language-links.dblist: SWAT: Deploy Compact Language Links in Swedish Wikipedia (T157114) (duration: 00m 50s)
  • 14:17 gehel: temporary raising high/low watermarks on elasticsearch eqiad to allow allocation of all shards
  • 14:04 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic1047.eqiad.wmnet
  • 12:38 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1045 with low load (3rd time a charm) (duration: 00m 39s)
  • 12:22 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1045 with low load (again) (duration: 02m 47s)
  • 12:18 dcausse: rebuild of translation memories index is done
  • 12:05 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1045 with low load (duration: 02m 49s)
  • 12:03 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic10(45|46).eqiad.wmnet
  • 11:48 paravoid: upgrading labmon1001 to grafana 4.1
  • 10:55 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=elastic10(45|46|47).eqiad.wmnet
  • 10:54 moritzm: upgrading remaining mediawiki servers to HHVM 3.12.14
  • 10:54 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic10(35|37|39|43|44).eqiad.wmnet
  • 10:42 elukey: reinstall mw211[89] as MW videoscalers (trusty) and mw2243 as MW jobrunner
  • 10:05 filippo@tin: Synchronized wmf-config/ProductionServices.php: Move udp2log from fluorine to mwlog1001 - T123728 (duration: 00m 41s)
  • 10:01 hashar: enabling puppet on contint1001 and running it
  • 09:56 volans: restarting salt-master on neodymium after openssl upgrade
  • 09:37 ema: cache_text, cache_upload: libssl1.1 upgraded to 1.1.0e-1+wmf1, libevent-2.0-5 upgraded to 2.0.21-stable-2+deb8u1
  • 09:28 hashar: disable puppet on contint1001. Will use contint2001 as a canary
  • 09:14 marostegui: Run pt-table-checksum on s2.nlwiki over some tables - T154485
  • 09:04 dcausse: rebuilding translation memories index - ETA ~4hours (from terbium, logs in ~dcausse/ttm-refresh)
  • 09:02 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=elastic10(35|39|43|44).eqiad.wmnet
  • 08:07 moritzm: upgrading openssl on redis clusters / various base service restarts
  • 07:44 gehel: restart elasticsearch on elastic1035
  • 07:43 gehel: trncating logs on elastic10(35|39|44)
  • 07:23 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2062 - T132416 (duration: 00m 40s)
  • 07:23 marostegui: Deploy alter table enwiki.revision db2062 - T132416
  • 07:18 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2055 - T132416 (duration: 00m 40s)
  • 03:10 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Feb 22 03:10:13 UTC 2017 (duration 5m 46s)
  • 03:04 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.13) (duration: 13m 54s)
  • 02:32 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.12) (duration: 11m 48s)
  • 01:03 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ORES review tool in cswiki T151611 (duration: 00m 39s)
  • 00:48 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Send "exception" channel to logstash Do not send "exception-json" channel to logstash T136849 (duration: 00m 40s)
  • 00:34 thcipriani@tin: Synchronized wmf-config: SWAT: Set $wgSoftBlockRanges T154698 PART II (duration: 00m 42s)
  • 00:33 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wgSoftBlockRanges T154698 PART I (duration: 00m 40s)
  • 00:20 thcipriani@tin: Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: Fix SiteConfiguration array merge syntax T157656 Fix Sentry URL scheme on beta Fix PageViewInfo config T158698 (beta-only changes) (duration: 00m 39s)
  • 00:17 thcipriani@tin: Synchronized php-1.29.0-wmf.12/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Turn off sister search AB test. T157942 (duration: 00m 39s)
  • 00:16 thcipriani@tin: Synchronized php-1.29.0-wmf.13/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Turn off sister search AB test. T157942 (duration: 00m 43s)
  • 00:13 smalyshev@tin: Finished deploy [wdqs/wdqs@7768422]: Deploy 2.1.5RC WAR on 2001 for testing (duration: 00m 25s)
  • 00:13 smalyshev@tin: Started deploy [wdqs/wdqs@7768422]: Deploy 2.1.5RC WAR on 2001 for testing
  • 00:05 demon@tin: Synchronized scap/plugins/clean.py: More code cleanup (duration: 00m 40s)

2017-02-21

  • 23:30 MaxSem: Kartotherian deploy did not happen
  • 23:22 demon@tin: Synchronized scap/plugins/clean.py: Code cleanup (duration: 00m 46s)
  • 23:21 demon@tin: scap aborted: scap/plugins/clean.py Code cleanup (duration: 00m 10s)
  • 23:21 demon@tin: Started scap: scap/plugins/clean.py Code cleanup
  • 22:01 mutante: carbon - removed from icinga, shutdown -h now (T158020)
  • 21:31 mutante: carbon - puppet node clean, node deactivate (T158020)
  • 21:10 demon@tin: Synchronized scap/plugins/prep.py: Completeness (duration: 00m 42s)
  • 20:48 Krinkle: (terbium) sql --write test2wiki 'DELETE FROM module_deps' (3687 rows affected, 0.01 sec) - per T158105.
  • 20:47 Krinkle: (terbium) sql --write testwiki 'DELETE FROM module_deps' (per T158105)
  • 20:44 mutante: carbon - backup /root data to install1002:/root/root-carbon/ before shutdown (T158020)
  • 20:36 mutante: rsyncing /home/ dirs excl. dot files, from carbon to install1002 (T158020)
  • 20:15 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=elastic10(35|39|43|44).eqiad.wmnet
  • 20:08 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.13
  • 19:50 demon@tin: Finished scap: prime wmf.13 - testwiki plus l10n build (pt 3 because ugh) (duration: 17m 17s)
  • 19:32 demon@tin: Started scap: prime wmf.13 - testwiki plus l10n build (pt 3 because ugh)
  • 19:32 demon@tin: scap failed: RuntimeError 2 test canaries had check failures (rerun with --force to override this check) (duration: 15m 00s)
  • 19:17 demon@tin: Started scap: prime wmf.13 - testwiki plus l10n build (pt 2 because T156851)
  • 19:16 demon@tin: Finished scap: prime wmf.13 - testwiki plus l10n build (duration: 26m 15s)
  • 18:49 demon@tin: Started scap: prime wmf.13 - testwiki plus l10n build
  • 18:45 moritzm: installing PHP security updates on iridium (phabricator.wikimedia.org)
  • 18:36 ppchelko@tin: Finished deploy [changeprop/deploy@4706f9d]: Change-Prop: Make ORES return minified responses T157693 (duration: 00m 55s)
  • 18:35 ppchelko@tin: Started deploy [changeprop/deploy@4706f9d]: Change-Prop: Make ORES return minified responses T157693
  • 18:34 Pchelolo: changeprop deploy 4706f9da
  • 18:14 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic10(27|32|34|38|41).eqiad.wmnet
  • 18:12 godog: roll-restart nodepool on labnodepool1001 to pick up statsd.eqiad.wmnet DNS changes - T157022
  • 18:12 godog: roll-restart zuul on cont1001 to pick up statsd.eqiad.wmnet DNS changes - T157022
  • 18:04 godog: roll-restart eventstreams in codfw/eqiad to pick up statsd.eqiad.wmnet DNS changes - T157022
  • 18:03 godog: roll-restart trendingedits in codfw/eqiad to pick up statsd.eqiad.wmnet DNS changes - T157022
  • 17:58 demon@tin: Synchronized tests/multiversion/MWMultiVersionTest.php: No op in prod, completeness, etc (duration: 00m 40s)
  • 17:57 demon@tin: Synchronized multiversion/MWMultiVersion.php: Shut up dumb invalid hostname errors (duration: 00m 52s)
  • 17:50 godog: roll-restart ocg in codfw/eqiad to pick up statsd.eqiad.wmnet DNS changes - T157022
  • 17:47 godog: roll-restart jmxtrans in codfw/eqiad on conf* to pick up statsd.eqiad.wmnet DNS changes - T157022
  • 17:35 godog: roll-restart parsoid in codfw/eqiad to pick up statsd.eqiad.wmnet DNS changes - T157022
  • 17:35 Amir1: done restarting ores services
  • 17:20 Amir1: restarting ores uwsgi and celery services in scb nodes
  • 16:59 ema: cache_misc, cache_maps: libssl1.1 upgraded to 1.1.0e-1+wmf1, libevent-2.0-5 upgraded to 2.0.21-stable-2+deb8u1
  • 16:37 gehel: restarting elasticsearch on elastic1030
  • 16:34 gehel: truncating elasticsearch logs on elastic1023
  • 16:31 gehel: truncating elasticsearch logs on elastic1030
  • 16:18 dcausse: truncated main elastic log, daemon.log and syslog on elastic1023
  • 16:08 moritzm: restarting apache on uranium for openssl update
  • 16:06 dcausse: truncated main log file on elastic1030
  • 15:50 gehel: restarting wdqs-updater on wdqs1002
  • 15:40 elukey: restart eventlogging on kafka200[123] for openssl upgrades
  • 15:40 godog: restart navtiming ve asset-check statsd-mw-js-deprecate on hafnium to pick up statsd.eqiad.wmnet change - T157022
  • 15:39 elukey: restart jmxtrans on kafka[12]00[123] for T157022
  • 15:34 mobrovac@tin: Started restart [mobileapps/deploy@cd3b897]: Restarting for Graphite DNS switch T157022
  • 15:32 elukey: correction on my previous entry: restart eventlogging on kafka100[123] for openssl upgrades
  • 15:30 mobrovac@tin: Started restart [graphoid/deploy@da37386]: Restarting for Graphite DNS switch T157022
  • 15:22 elukey: restart eventlogging on kafka200[123] for openssl upgrades
  • 15:21 mobrovac@tin: Started restart [cxserver/deploy@0e4ae4f]: Restarting for Graphite DNS switch T157022
  • 15:20 moritzm: rolling restart of swift frontend servers to pick up openssl update
  • 15:19 mobrovac@tin: Started restart [citoid/deploy@95df861]: Restarting for Graphite DNS switch T157022
  • 15:18 mobrovac@tin: Started restart [mathoid/deploy@ba3217e]: Restarting for Graphite DNS switch T157022
  • 15:17 hashar: European SWAT complete
  • 15:17 hashar@tin: Synchronized php-1.29.0-wmf.12/extensions/UniversalLanguageSelector/UniversalLanguageSelector.hooks.php: Fix site picks: missing from globals (duration: 01m 00s)
  • 15:12 gehel: restarting kartotherian / tilerator(ui) on maps1*
  • 15:09 gehel: restarting kartotherian / tilerator(ui) on maps2*
  • 15:06 gehel: restarting kartotherian / tilerator(ui) on maps-test*
  • 15:06 godog: roll-restart restbase after statsd move to graphite1001 - T157022
  • 15:06 elukey: Increased manually maximum httpd keep alive requests and timeout on bohrium (piwik) - T154558
  • 14:56 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=elastic10(33|34|38|42).eqiad.wmnet
  • 14:43 hashar@tin: Synchronized php-1.29.0-wmf.12/extensions/UniversalLanguageSelector/maintenance/ULSCompactLinksDisablePref.php: Add a maintenance script for opt-in T133031 (duration: 00m 41s)
  • 14:35 moritzm: upgrading openssl on logstash cluster / various base service restarts
  • 14:29 dcausse: truncated main log file on elastic1030
  • 14:29 hashar@tin: Synchronized portals: (no justification provided) (duration: 00m 40s)
  • 14:29 moritzm: restarting NTP servers on dns_recursors to pick up openssl update (one by one)
  • 14:28 hashar@tin: Synchronized portals/prod/wikipedia.org/assets: (no justification provided) (duration: 00m 40s)
  • 14:25 moritzm: upgrading openssl on memcached clusters / various base service restarts
  • 14:22 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable TwoColConflict extension on arwiki - T158493 (duration: 00m 40s)
  • 14:13 hashar@tin: Synchronized portals: (no justification provided) (duration: 00m 41s)
  • 14:12 hashar@tin: Synchronized portals/prod/wikipedia.org/assets: (no justification provided) (duration: 00m 40s)
  • 14:10 hashar@tin: Synchronized php-1.29.0-wmf.12/extensions/UniversalLanguageSelector/: Fix broken site picks feature for compact language links (duration: 01m 04s)
  • 14:08 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable ReadingDepth logging on Wikipedias - T148262 T155639 (duration: 00m 45s)
  • 14:00 moritzm: upgrading openssl on maps clusters / various base service restarts
  • 13:41 elukey: restarting nodejs on aqs1* to pick up openssl security upgrades
  • 13:21 moritzm: upgrading openssl on aqs cluster / various base service restarts
  • 13:06 moritzm: upgrading openssl on parsoid clusters / various base service restarts
  • 12:55 moritzm: upgrading openssl on database servers / various base service restarts
  • 12:53 volans: re-enabled puppet on neodymium and puppetmaster1001 after Gerrit 330436 was merged T154588
  • 12:51 volans: re-enabled puppet on planet2001, was disabled since a week without reason
  • 12:39 volans: reenabled ircecho aftrer fixing ferm issue and run puppet on affected hosts
  • 12:08 volans: stopped ircecho temporarily while fixing ferm
  • 12:01 volans: temporarily disabled puppet on neodymium and puppetmaster1001 to merge Gerrit 330436 T154588
  • 11:32 moritzm: upgrading openssl on kafka clusters / various base service restarts
  • 11:15 moritzm: upgrading openssl on restbase clusters / various base service restarts
  • 11:05 moritzm: upgrading openssl on hadoop cluster / various base service restarts
  • 11:02 elukey: rolling restart of cassandra-metrics-collector on aqs1* for T157022
  • 10:55 elukey: rolling restart of the analyics jmxtrans daemons for T157022
  • 10:29 moritzm: restarting base services on mw2* after openssl update
  • 10:14 godog: downgrade carbon-c-relay on graphite1001 to trusty's version and bounce daemons
  • 09:58 moritzm: upgrading mira/tin to HHVM 3.12.14
  • 09:46 godog: upgrade graphite on graphite1001 and bounce carbon daemons
  • 09:26 ema: cp3030: libssl1.1 upgraded to 1.1.0e-1+wmf1, libevent-2.0-5 upgraded to 2.0.21-stable-2+deb8u1
  • 08:53 godog: switch statsd/graphite DNS to graphite1001 - T157022
  • 08:32 moritzm: upgrading mw1170-mw1208 to HHVM 3.12.14
  • 08:30 gehel: increasing concurrent recoveries / relocations to 8 on elasticsearch eqiad
  • 08:24 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=elastic10(27|32|37|41).eqiad.wmnet
  • 07:31 marostegui: Deploy alter table enwiki.revision db2055 - T132416
  • 07:29 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2048 and depool db2055 - T132416 (duration: 00m 51s)
  • 02:24 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Feb 21 02:24:37 UTC 2017 (duration 5m 20s)
  • 02:19 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.12) (duration: 07m 20s)
  • 01:17 tstarling@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 42s)

2017-02-20

  • 20:31 gehel: taking threaddumps and restarting elastic1017 (high load)
  • 20:20 gehel: reducing concurrent recoveries / relocations to 4 on elasticsearch eqiad
  • 19:07 ariel@tin: Finished deploy [dumps/dumps@9757356]: fix retries of page content dumps with checkpoint, no dup ranges (duration: 00m 02s)
  • 19:07 ariel@tin: Started deploy [dumps/dumps@9757356]: fix retries of page content dumps with checkpoint, no dup ranges
  • 18:30 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=elastic10(27|32|37|41).eqiad.wmnet
  • 18:29 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic10(26|31|36|40).eqiad.wmnet
  • 17:56 ppchelko@tin: Finished deploy [changeprop/deploy@30873eb]: Update change-prop to 30873ebd5: enabling DNS caching for T158338 (duration: 01m 41s)
  • 17:54 ppchelko@tin: Started deploy [changeprop/deploy@30873eb]: Update change-prop to 30873ebd5: enabling DNS caching for T158338
  • 17:52 Pchelolo: update change-prop to 30873ebd5
  • 16:40 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=elastic10(26|31|36|40).eqiad.wmnet
  • 14:55 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs1002.eqiad.wmnet
  • 14:49 ema: cp2002, cp4008: libssl1.1 upgraded to 1.1.0e-1+wmf1 and libevent-2.0-5 upgraded to 2.0.21-stable-2+deb8u1
  • 14:32 ema: upgrading pinkunicorn to varnish 4.1.5-1wm1
  • 14:30 ema: varnish 4.1.5-1wm1 uploaded to apt.w.o
  • 14:10 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic10(25|28|29|30).eqiad.wmnet
  • 13:52 gehel: resetting ownership of new .wsp files for wdqs1002 on graphite[12]001
  • 13:49 moritzm: installing remaining lcms security updates
  • 13:41 hashar@tin: Synchronized wmf-config/throttle.php: [throttle] New rule - T158312 (duration: 00m 42s)
  • 13:35 marostegui: Transferring dbstore1001:/srv/backups (the last 2 backups) to dbstore2001:/srv/backup/dbstore1001 - T153768
  • 13:17 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=elastic10(25|28|29|30).eqiad.wmnet
  • 13:04 moritzm: installing jasper security updates
  • 12:20 godog: remove syslog from graphite1001, bump max open files for carbon-c-relay
  • 11:00 godog: switch diamond traffic to graphite1001 - T157022
  • 10:54 moritzm: rolling restart of nginx on remaining mediawiki servers in eqiad to pick up openssl update
  • 10:26 ariel@tin: Finished deploy [dumps/dumps@dee43ca]: fix prefetch on retries of partially complete page content dumps (duration: 00m 02s)
  • 10:26 ariel@tin: Started deploy [dumps/dumps@dee43ca]: fix prefetch on retries of partially complete page content dumps
  • 10:24 hashar@tin: Synchronized wmf-config/throttle.php: Add new throttle rule - T158432 (duration: 00m 49s)
  • 09:46 moritzm: upgrading mediawiki servers in codfw to HHVM 3.12.14
  • 09:33 marostegui: Manually deploy gtid_domain_id on s6 hosts - T149418
  • 08:47 gehel: restarting diamond on wdqs1002 after initial data import
  • 08:41 marostegui: Increase 100G dbstore1002 lv /dev/mapper/tank-data
  • 08:17 ariel@tin: Finished deploy [dumps/dumps@d50e129]: cleanup tmp files before checkpoint file rerun (duration: 00m 02s)
  • 08:17 ariel@tin: Started deploy [dumps/dumps@d50e129]: cleanup tmp files before checkpoint file rerun
  • 07:39 marostegui@tin: Synchronized wmf-config/db-codfw.php: Update ticket number for db2048 depool reason (duration: 00m 44s)
  • 07:29 marostegui: Deploy alter table on db2048 enwiki.revision - T132416
  • 07:28 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2048 - T132416 (duration: 00m 41s)
  • 02:25 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Feb 20 02:25:07 UTC 2017 (duration 5m 19s)
  • 02:19 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.12) (duration: 07m 53s)

2017-02-19

  • 20:08 ariel@tin: Finished deploy [dumps/dumps@364470e]: fix private table dumping, report failed runs correctly (duration: 00m 03s)
  • 20:08 ariel@tin: Started deploy [dumps/dumps@364470e]: fix private table dumping, report failed runs correctly
  • 15:54 hashar: Restarted Zuul to clear out a stall function in Gearman server.
  • 02:37 l10nupdate@tin: ResourceLoader cache refresh completed at Sun Feb 19 02:37:07 UTC 2017 (duration 5m 18s)
  • 02:31 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.12) (duration: 13m 08s)

2017-02-18

  • 18:00 reedy@tin: Synchronized wmf-config/CommonSettings.php: Rv reservedusernames addition from CS (duration: 00m 42s)
  • 17:59 reedy@tin: Synchronized php-1.29.0-wmf.12/includes/DefaultSettings.php: Unknown user to reserved usernames in defaultsettings (duration: 00m 45s)
  • 17:29 reedy@tin: Synchronized wmf-config/CommonSettings.php: Add Unknown user to reservedusernames (duration: 00m 48s)
  • 02:23 l10nupdate@tin: ResourceLoader cache refresh completed at Sat Feb 18 02:23:52 UTC 2017 (duration 5m 20s)
  • 02:18 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.12) (duration: 06m 41s)

2017-02-17

  • 21:34 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic1023.eqiad.wmnet
  • 19:20 gehel: upgrading maps-test2004 to nodejs6 for testing - T150354
  • 18:34 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic10(21|22|24).eqiad.wmnet
  • 17:17 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=elastic10(21|22|23|24).eqiad.wmnet
  • 17:17 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic10(17|18|19|20).eqiad.wmnet
  • 16:18 ejegg: updated 3DS rules for DK, SE, and NO
  • 16:15 bblack: restarting cp1074 varnish backend (cron due in 24h, but mb lag looks pretty bad)
  • 15:26 urandom: T155120: Restarting Cassandra on restbase1007-a.eqiad.wmnet to disable Prometheus exporter agent
  • 14:48 _joe_: uploaded clustershell 1.7.3, tqdm, pyparsing to jessie-wikimedia in preparation for cumin
  • 13:47 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2070 - T156478 (duration: 00m 45s)
  • 12:21 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2070 - T156478 (duration: 00m 41s)
  • 12:18 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2070 - T156478 (duration: 00m 41s)
  • 11:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Clean up db1028 old comments - T153300 (duration: 00m 41s)
  • 11:11 moritzm: restarting nginx on sodium (mirrors.wikimedia.org) to pick up openssl update
  • 10:01 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=elastic10(17|18|19|20).eqiad.wmnet
  • 09:54 moritzm: rolling restart of nginx on mediawiki servers in codfw to pick up openssl update
  • 09:33 moritzm: rolling restart of nginx on mw canaries to pick up openssl update
  • 09:32 gehel: upgrade nginx on elasticsearch codfw for ssl upgrade
  • 09:30 gehel: upgrade nginx on elastic1049-1052 for ssl upgrade
  • 09:22 moritzm: restarting nginx on install1002/2002 to pick up new openssl
  • 08:35 moritzm: upgrading mw1262-mw1265 to HHVM 3.12.14
  • 08:35 godog: restart nginx on prometheus in eqiad/codfw to pick up openssl update
  • 08:19 moritzm: restarted nginx/prometheus in esams/ulsfo to pick up openssl update
  • 08:00 moritzm: installing openssl 1.1.0e updates
  • 07:58 moritzm: upgrading mw1261 to HHVM 3.12.14
  • 07:41 moritzm: installing spice security updates
  • 06:59 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2070 - T156478 (duration: 00m 48s)
  • 02:33 l10nupdate@tin: ResourceLoader cache refresh completed at Fri Feb 17 02:33:21 UTC 2017 (duration 5m 32s)
  • 02:27 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.12) (duration: 06m 49s)

2017-02-16

  • 23:18 maxsem@tin: Finished scap: Another time, just ot make sure some files synched cuz lat time there were some mid-air collisions (duration: 15m 44s)
  • 23:02 maxsem@tin: Started scap: Another time, just ot make sure some files synched cuz lat time there were some mid-air collisions
  • 22:58 maxsem@tin: Finished scap: Update messages for https://gerrit.wikimedia.org/r/#/c/338013/ (duration: 24m 29s)
  • 22:46 mutante: tin - apt-get clean - 4.6G avail (T158359)
  • 22:33 maxsem@tin: Started scap: Update messages for https://gerrit.wikimedia.org/r/#/c/338013/
  • 22:32 maxsem@tin: Synchronized php-1.29.0-wmf.12/extensions/JsonConfig/: https://gerrit.wikimedia.org/r/#/c/338013/ (duration: 00m 42s)
  • 22:31 maxsem@tin: Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/338208/ (duration: 00m 53s)
  • 22:17 XenoRyet: updated paymentswiki from 4466b9d to 2a0c3b2
  • 22:04 mutante: phab2001 - start/stop phd, testing gerrit 338163
  • 21:39 Reedy: make that 2017
  • 21:39 Reedy: Deleted around 9500 pre 2013 captchas
  • 21:08 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.29.0-wmf.12
  • 20:13 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.29.0-wmf.12
  • 19:13 chasemp: clean out /var/log/ on labnet1001 as it filled up
  • 19:12 gehel: restarting kartotherian / tilerator on maps-test*
  • 19:07 chasemp: bump up nodepool allocated fixed ips set (I think it exhausted them errantly somehow?)
  • 18:52 chasemp: clean out nodepool instances
  • 18:50 chasemp: stop noodepool to reset state on pool
  • 18:03 halfak@tin: Started deploy [ores/deploy@e9bbda3]: (no justification provided)
  • 18:03 halfak: deploying ores:e9bbda3
  • 17:08 hashar: reenable puppet on contint1001
  • 17:03 hashar: stopped puppet on contint1001 for https://gerrit.wikimedia.org/r/#/c/336978/
  • 16:30 moritzm: uploaded HHVM 3.12.14 to apt.wikimedia.org
  • 16:25 jynus: SET GLOBAL thread_pool_size=64; on db1074's mariadb
  • 16:01 moritzm: upgrading mwdebug1002 to HHVM 3.12.14
  • 15:53 moritzm: upgrading mwdebug1001 to HHVM 3.12.14
  • 14:27 moritzm: uploaded openssl 1.1.0e to apt.wikimedia.org
  • 13:28 marostegui@tin: Synchronized wmf-config/db-codfw.php: Change db2070 IP as it goes to another rack - T156478 (duration: 00m 41s)
  • 13:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Change db2070 IP as it goes to another rack - T156478 (duration: 00m 56s)
  • 13:19 marostegui: Shutdown db2070 for maintenance - T156478
  • 10:18 godog: roll-restart hhvm in eqiad to pick up fluorine -> mwlog1001 changes - T123728
  • 10:17 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic10(48|49|50|51|52).eqiad.wmnet
  • 10:15 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic10(48|49|50|51|52).codfw.wmnet
  • 09:39 moritzm: installing libgc security updates on trusty systems
  • 09:33 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2070 - T156478 (duration: 00m 41s)
  • 08:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore origina db1082 weight - T158188 (duration: 00m 41s)
  • 08:39 godog: roll-restart jobrunner in codfw/eqiad to pick up fluorine -> mwlog1001 redis change - T123728
  • 07:54 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2060 - T156161 (duration: 00m 44s)
  • 07:43 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase load db1082 - T158188 (duration: 00m 42s)
  • 03:11 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Feb 16 03:11:01 UTC 2017 (duration 5m 42s)
  • 03:05 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.12) (duration: 14m 27s)
  • 02:33 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.11) (duration: 11m 46s)
  • 00:59 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1082 with low load (duration: 00m 41s)
  • 00:49 eileen1: Update CiviCRM from 1ffc090 to 20660c4
  • 00:12 maxsem@tin: Synchronized php-1.29.0-wmf.12/extensions/Gadgets: https://gerrit.wikimedia.org/r/#/c/338004/ (duration: 00m 42s)

2017-02-15

  • 22:59 eileen1: update CIviCRM from da6ba1b to 1ffc090
  • 22:55 thcipriani@tin: Synchronized php-1.29.0-wmf.12/includes/libs/rdbms/ChronologyProtector.php: Add version to ChronologyProtector key T158217 (duration: 00m 41s)
  • 22:52 demon@tin: Synchronized php-1.29.0-wmf.12/extensions/Dashiki: prep-type stuff (duration: 00m 50s)
  • 21:41 ejegg: updated civicrm from 0fb289f to da6ba1b
  • 20:06 mutante: contint1001 - logrotate --force /etc/logrotate.d/jenkins to test gerrit:337383
  • 19:50 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Enable Popups by default on se.wikimedia (T68374) (duration: 00m 41s)
  • 18:59 demon@tin: Synchronized multiversion/submodules.json: no-op (duration: 00m 50s)
  • 18:30 marostegui: Stop MySQL and shutdown db2062 for maintenance - T156478
  • 17:58 jynus: stopping labsdb1005 mariadb + puppet in preparation for reimage
  • 17:41 thcipriani@tin: Synchronized php-1.29.0-wmf.11/includes/libs/rdbms/ChronologyProtector.php: init() use instanceof instead of empty() T158127 (duration: 00m 41s)
  • 17:17 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: Group0 to 1.29.0-wmf.12 T155527
  • 17:08 thcipriani@tin: Synchronized php-1.29.0-wmf.12/includes/libs/rdbms/ChronologyProtector.php: init() use instanceof instead of empty() T158127 (duration: 00m 43s)
  • 17:05 thcipriani: starting wmf.12 to group0
  • 16:39 godog: flip xenon redis and apache from fluorine to mwlog1001 - T123728
  • 16:23 Jeff_Green: authdns-update to deploy fundraising host rename db1008->frav1001
  • 16:16 urandom: T155120: restarting Cassandra on restbase1007-a to enable Prometheus exporter (canary)
  • 16:01 dcausse@tin: Synchronized php-1.29.0-wmf.11/extensions/CirrusSearch/: T156234: Fold some problematic whitespaces with completion (duration: 01m 01s)
  • 15:57 marostegui: (Old action but for the sake of getting it logged) Force RAID controller to work on WriteBack even with the broken BBU it has now on db1060 so it can keep up with the replication thread - T158194
  • 15:51 filippo@tin: Synchronized wmf-config/StartProfiler.php: Switch xenon redis to mwlog1001.eqiad.wmnet (duration: 00m 42s)
  • 15:44 hashar: Zuul reducing gate-and-submit minimum amount of changes to process from the wrong 12 down to 2. In case of repeating failures it would end up running jobs for only two jobs which would prevent cancelling jobs for up to 11 changes!
  • 15:37 jynus: stopping slave and repartitioning db1045
  • 14:45 jynus: offlined 2 disks with media + other errors on db1060
  • 14:34 marostegui: Stop MySQL and shutdown db1082 - T158188
  • 14:14 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1082 (duration: 00m 44s)
  • 14:04 hashar@tin: Synchronized php-1.29.0-wmf.12/extensions/CirrusSearch/includes/Maintenance/SuggesterAnalysisConfigBuilder.php: Fold some problematic whitespaces with completion - T156234 (duration: 00m 48s)
  • 13:59 elukey: disabled mod_deflate on bohrium (piwik) and disabled puppet. Testing 503 reduction.
  • 13:16 dereckson@tin: Synchronized wmf-config/throttle.php: Throttle rule for Royal College of Nursing event (T158171) (duration: 00m 43s)
  • 12:56 elukey: restart of jmxtrans on all the analytics kafka brokers
  • 12:33 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1045 (duration: 00m 42s)
  • 11:38 marostegui: Running pt-table-checksum on db1043 (m3 - phabricator master) - T154485
  • 11:35 bblack@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1067.eqiad.wmnet
  • 11:20 godog: upgrade git on tin/mira - T140927
  • 10:59 moritzm: installing PHP security updates on californium (running horizon)
  • 10:49 moritzm: installing PHP security updates on uranium (running ganglia)
  • 10:46 moritzm: installing PHP security updates on siliver (running wikitech)
  • 08:21 moritzm: installing PHP security updates on Ubuntu systems
  • 07:33 marostegui: Deploy alter table on x1 master (db1031) for the echo_notification tables - T136428
  • 07:27 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2062 - T156478 (duration: 00m 42s)
  • 02:40 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Feb 15 02:40:31 UTC 2017 (duration 5m 23s)
  • 02:35 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.11) (duration: 12m 50s)
  • 00:22 ebernhardson@tin: Synchronized php-1.29.0-wmf.11/extensions/CirrusSearch/: Provide per-index settings from configuration for elasticsearch 5 (duration: 00m 55s)
  • 00:16 ebernhardson@tin: Synchronized wmf-config/CirrusSearch-common.php: Configure cirrus per-index setings for elasticsearch 5 (duration: 00m 43s)
  • 00:07 ebernhardson@tin: Synchronized php-1.29.0-wmf.11/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: (no justification provided) (duration: 00m 50s)

2017-02-14

  • 22:23 thcipriani@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Cleanup popups beta cluster config (beta-only-change) (duration: 00m 41s)
  • 22:15 chasemp: start staged nova-fullstack testing daemon on labnet1002 for metric inspection
  • 22:11 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.29.0-wmf.11 for T158127
  • 22:07 otto@tin: Finished deploy [analytics/refinery@4cd6305]: Deploying refinery with another update to drop hourly partitions script (duration: 01m 13s)
  • 22:06 otto@tin: Started deploy [analytics/refinery@4cd6305]: Deploying refinery with another update to drop hourly partitions script
  • 22:05 otto@tin: Finished deploy [analytics/refinery@4cd6305]: Deploying refinery with another update to drop hourly partitions script (duration: 01m 41s)
  • 22:03 otto@tin: Started deploy [analytics/refinery@4cd6305]: Deploying refinery with another update to drop hourly partitions script
  • 22:01 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.29.0-wmf.12
  • 21:47 thcipriani@tin: Synchronized php-1.29.0-wmf.11/includes/libs/rdbms/loadbalancer/LoadBalancer.php: doWait() (duration: 00m 50s)
  • 21:10 thcipriani@tin: Finished scap: testwiki to php-1.29.0-wmf.12 and rebuild l10n cache (wikiversions.json not updated previously) (duration: 09m 25s)
  • 21:00 thcipriani@tin: Started scap: testwiki to php-1.29.0-wmf.12 and rebuild l10n cache (wikiversions.json not updated previously)
  • 20:51 thcipriani@tin: Finished scap: testwiki to php-1.29.0-wmf.12 and rebuild l10n cache (duration: 48m 04s)
  • 20:38 eileen1: update civicrm from 8d04e75 to 0fb289f
  • 20:36 otto@tin: Finished deploy [analytics/refinery@67c3924]: Deploying refinery with update to drop hourly partitions script (duration: 02m 25s)
  • 20:33 otto@tin: Started deploy [analytics/refinery@67c3924]: Deploying refinery with update to drop hourly partitions script
  • 20:22 Dereckson: Update site statistics for pam.wikipedia (T158110, now 454 images)
  • 20:03 thcipriani@tin: Started scap: testwiki to php-1.29.0-wmf.12 and rebuild l10n cache
  • 19:23 krinkle@tin: Synchronized docroot/noc/conf: I67194f (duration: 00m 48s)
  • 19:22 krinkle@tin: Synchronized dblists/: I67194f (duration: 01m 37s)
  • 18:39 arlolra: Updated Parsoid to 79ccfb93 (T58381, T108216)
  • 18:28 thcipriani: starting branch cut for 1.29.0-wmf.12
  • 18:27 arlolra@tin: Finished deploy [parsoid/deploy@1bfb86b]: Updating Parsoid to 79ccfb93 (duration: 09m 58s)
  • 18:17 arlolra@tin: Started deploy [parsoid/deploy@1bfb86b]: Updating Parsoid to 79ccfb93
  • 18:11 MaxSem: Purged https://he.wikipedia.org/static/images/mobile/copyright/wikipedia-wordmark-he.svg with purgeList.php
  • 17:34 bblack@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp1067.eqiad.wmnet
  • 16:18 andrewbogott: rebooting californium
  • 16:15 andrewbogott: dist-upgrade californium (as part of the liberty->mitaka upgrade)
  • 16:06 Niharika: Updated wikimania app to 5c44d06 Removed stale translations
  • 16:02 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Change db2062 IP after its move to another rack - T156478 (duration: 00m 39s)
  • 16:01 marostegui@tin: Synchronized wmf-config/db-codfw.php: Change db2062 IP after its move to another rack - T156478 (duration: 00m 40s)
  • 15:11 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Allow sysops to add/revoke account creator on it.wikiversity (T158062) (duration: 00m 41s)
  • 15:05 marostegui: Shutdown mysql (and later the whole host) on db2062 for maintenance - T156478
  • 15:04 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2062 to change its rack - T156478 (duration: 00m 41s)
  • 15:01 ema: lvs10*: upgrade to pybal 1.13.5 T147425
  • 14:48 moritzm: installing php security updates on einsteinium
  • 14:41 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1073,89,90,91,92 (duration: 00m 41s)
  • 14:38 ema: lvs300[12]: upgrade to jessie 8.7, pybal 1.13.5, reboot into kernel 4.4.2-3+wmf8 T155401 T147425
  • 14:35 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1073,89,90,91,92 (duration: 00m 40s)
  • 14:20 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1055,72,83,56,84,76,78,87,71 (duration: 00m 40s)
  • 14:17 hashar: European SWAT is complete
  • 14:16 hashar@tin: Synchronized php-1.29.0-wmf.11/extensions/CirrusSearch/profiles/SimilarityProfiles.php: Explicitly use BM25 as default for wmf_defaults similarity profile (duration: 00m 47s)
  • 14:14 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1055,72,83,56,84,76,78,87,71 (duration: 00m 41s)
  • 14:11 ema: lvs300[34]: upgrade to jessie 8.7, pybal 1.13.5, reboot into kernel 4.4.2-3+wmf8 T155401 T147425
  • 14:09 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable TwoColConflict on dewiki - T155721 (duration: 00m 40s)
  • 14:06 hashar@tin: Synchronized wmf-config/throttle.php: Throttle rule for cswiki - T158040 (duration: 00m 40s)
  • 13:50 ema: lvs400[12]: upgrade to jessie 8.7, pybal 1.13.5, reboot into kernel 4.4.2-3+wmf8 T155401 T147425
  • 13:32 ema: lvs400[34]: upgrade to jessie 8.7, pybal 1.13.5, reboot into kernel 4.4.2-3+wmf8 T155401 T147425
  • 13:28 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1051,66,80,74,77,56,81,70,82 (duration: 00m 44s)
  • 13:15 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1051,66,80,74,77,56,81,70,82 (duration: 00m 40s)
  • 12:59 jynus: reloading/restarting gerrint on cobalt, too slow
  • 11:57 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw2221.codfw.wmnet
  • 11:30 moritzm: manual fix up of exim spool permissions on krypton (used to run the heavy exim variant)
  • 11:28 jynus: performing schema change on all mariadb servers T150474
  • 10:51 ema: lvs200[123]: upgrade to jessie 8.7, pybal 1.13.5, reboot into kernel 4.4.2-3+wmf8 T155401 T147425
  • 10:48 moritzm: installing tomcat security updates
  • 10:23 ema: lvs200[456]: upgrade to jessie 8.7, pybal 1.13.5, reboot into kernel 4.4.2-3+wmf8 T155401 T147425
  • 10:18 ema: uploading pybal 1.13.5 to apt.w.o T147425
  • 09:05 moritzm: upgrading firejail on sca cluster
  • 08:45 moritzm: installing vim security updates
  • 08:33 moritzm: restarting zookeeper on conf1003
  • 08:23 moritzm: restarting zookeeper on conf1002 to pick up OpenJDK update (restarts were stopped yesterday to further investigate gc behaviour)
  • 06:56 marostegui: Deploy alter table on x1 echo_notification tables - T136428
  • 02:40 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Feb 14 02:40:22 UTC 2017 (duration 5m 19s)
  • 02:35 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.11) (duration: 11m 50s)
  • 00:40 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/337442/ (duration: 00m 40s)
  • 00:39 eileen1: update civicrm from 7b36996 to 8d04e75
  • 00:38 maxsem@tin: Synchronized static/images/mobile/copyright/wikipedia-wordmark-he.svg: https://gerrit.wikimedia.org/r/#/c/337442/ (duration: 00m 40s)
  • 00:22 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/337441/ (duration: 00m 40s)
  • 00:21 maxsem@tin: Synchronized static/images/mobile/copyright/wikipedia-wordmark-hi.svg: https://gerrit.wikimedia.org/r/#/c/337441/ (duration: 00m 42s)
  • 00:16 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/337064/ (duration: 00m 48s)

2017-02-13

  • 23:45 Pchelolo: update RESTBase to 0e9106ab8
  • 23:35 Pchelolo: update RESTBase to 0e9106ab8 - canary on restbase1007
  • 23:29 Pchelolo: update RESTBase to 0e9106ab8 - staging
  • 23:27 bsitzmann@tin: Finished deploy [mobileapps/deploy@cd3b897]: Update mobileapps to 776211b (duration: 03m 19s)
  • 23:24 bsitzmann@tin: Started deploy [mobileapps/deploy@cd3b897]: Update mobileapps to 776211b
  • 22:39 Pchelolo: rollback RESTBase to ea980cc5d - staging
  • 22:26 Pchelolo: update RESTBase to 0e9106ab8 - staging
  • 22:14 kaldari@tin: Synchronized dblists/: Turning on Echo for loginwiki (duration: 00m 41s)
  • 22:14 kaldari: scap sync-dir dblists/ 'Turning on Echo for loginwiki'
  • 21:43 bsitzmann@tin: Finished deploy [mobileapps/deploy@f6b4435]: Update mobileapps to 3af473f (duration: 03m 44s)
  • 21:39 bsitzmann@tin: Started deploy [mobileapps/deploy@f6b4435]: Update mobileapps to 3af473f
  • 20:51 mutante: carbon/install - adjusted Letsencrypt cert creation, deactivated reprepro to protect from accidental use, switching rsync direction from install1002->install2002, disabled cron on carbon (T132757)
  • 20:33 Reedy: dropped old out of date echo tables from extension1.loginwiki T157105
  • 19:38 mutante: carbon - synced /srv/ data to install1002/2002 for the last time, switching apt.wikimedia.org CNAME to install1002 - carbon deprecated (T132757)
  • 19:22 legoktm: running namespaceDupes.php on fiwiki (T103786)
  • 19:11 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: properly set wgCirrusSearchUseIcuFolding T155515 (duration: 00m 41s)
  • 19:00 moritzm: upgrading mw1236 with the security updates it missed while it was powered off
  • 18:33 chasemp: labstore1005 service maintain-dbusers restart
  • 18:18 mutante: scandium - shutdown -h now (T150936)
  • 18:05 mutante: scandium - ex-zuul merger - removing from puppet, revoking puppet cert, salt key..
  • 16:08 ema: lvs1003: upgrade to jessie 8.7, pybal 1.13.4, reboot into kernel 4.4.2-3+wmf8 T155401
  • 15:58 ema: lvs1002: upgrade to jessie 8.7, pybal 1.13.4, reboot into kernel 4.4.2-3+wmf8 T155401
  • 15:52 gehel: copy old blazegraph metrics to new path (wikidata.query.(triple|lag).* -> servers.<server_name>...)
  • 15:36 ema: lvs1001: upgrade to jessie 8.7, pybal 1.13.4, reboot into kernel 4.4.2-3+wmf8 T155401
  • 15:18 moritzm: switched krypton to exim4-daemon-light (the -heavy variant was installed from an earlier role it carried)
  • 15:06 addshore: EU SWAT done
  • 15:06 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T101634 Update Wikiquote talk namespace in Sanskrit Wikisource and Support legacy Wikiquote talk namespace in Sanskrit Wikisource (duration: 00m 40s)
  • 14:37 addshore@tin: Synchronized portals: Updating portal stats Gerrit (duration: 00m 40s)
  • 14:36 addshore@tin: Synchronized portals/prod/wikipedia.org/assets: Updating portal stats Gerrit (duration: 00m 40s)
  • 14:28 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T155721 Enable TwoColConflict on metawiki (duration: 00m 40s)
  • {{safesubst:SAL entry|1=14:23 addshore@tin: Synchronized php-1.29.0-wmf.11/extensions/TwoColConflict/includes/TwoColConflictHooks.php: [[gerrit:337186|Change beta feature info and talk links (duration: 00m 40s)}}
  • 14:19 addshore@tin: Synchronized php-1.29.0-wmf.11/extensions/RevisionSlider/modules/ext.RevisionSlider.lazy.css: T157800 Dont set min-height and min-width for oo-ui buttons 2/2 (duration: 00m 55s)
  • 14:18 addshore@tin: Synchronized php-1.29.0-wmf.11/extensions/RevisionSlider/modules/ext.RevisionSlider.css: T157800 Dont set min-height and min-width for oo-ui buttons 1/2 (duration: 01m 07s)
  • 13:57 moritzm: rolling restart of zookeeper in eqiad to pick up Java security updates
  • 13:57 marostegui: Shutdown db2060 for maintenance - T156161
  • 13:47 ema: lvs100[56]: upgrade to jessie 8.7, pybal 1.13.4, reboot into kernel 4.4.2-3+wmf8 T155401
  • 13:37 ema: lvs1004: upgrade to jessie 8.7, pybal 1.13.4, reboot into kernel 4.4.2-3+wmf8 T155401
  • 13:33 moritzm: rolling restart of zookeeper in codfw to pick up Java security updates
  • 12:32 elukey: updating elastic search ACLs on cr1/cr2 for the analytics-ip4 filter
  • 11:59 moritzm: removing unneeded PHP packages from mw1261-mw1265 (these were installed before we changed puppet trim most PHP packages in favour of HHVM)
  • 11:21 moritzm: installing PHP security updates
  • 11:18 elukey: stopped ircecho on einsteinium
  • 11:11 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs1002.eqiad.wmnet
  • 10:47 godog: remove big/spammy log files from thubmro100[12] - T157949
  • 10:35 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs1001.eqiad.wmnet
  • 09:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1078 - T136428 (duration: 00m 41s)
  • 09:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1078 - T136428 (duration: 00m 40s)
  • 09:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1077 - T136428 (duration: 00m 40s)
  • 09:09 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1077 - T136428 (duration: 00m 40s)
  • 09:00 marostegui: Deploy alter table s3 officewiki and mediawikiwiki for echo_notification tables on eqiad - T136428
  • 08:10 elukey: removed empty log files from elastic1022,1024,2001,1026,1040 to fix logrotate cronspam
  • 07:55 moritzm: upgrade HHVM on remaining mw servers in eqiad
  • 07:03 marostegui: Compressing commonswiki tables on labsdb1010 and labsdb1011 - T153743
  • 02:38 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Feb 13 02:38:43 UTC 2017 (duration 5m 18s)
  • 02:33 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.11) (duration: 13m 18s)

2017-02-12

  • 15:18 reedy@tin: Synchronized php-1.29.0-wmf.11/extensions/ConfirmEdit/maintenance: Instrumentation to script (duration: 00m 41s)
  • 02:24 l10nupdate@tin: ResourceLoader cache refresh completed at Sun Feb 12 02:24:56 UTC 2017 (duration 5m 20s)
  • 02:19 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.11) (duration: 07m 12s)

2017-02-11

  • 10:02 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic20(33|34|35|36).codfw.wmnet
  • 09:53 elukey: mw1236 back in production (scap pull executed before pooled=yes) - T156610
  • 09:52 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1236.eqiad.wmnet
  • 09:35 elukey: rebooting mw1236 to make sure that it comes up cleanly - T156610
  • 09:15 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=elastic20(33|34|35|36).codfw.wmnet
  • 09:09 gehel: cleanup logs on elastic20(01|25) - T139043
  • 02:37 l10nupdate@tin: ResourceLoader cache refresh completed at Sat Feb 11 02:37:10 UTC 2017 (duration 5m 19s)
  • 02:31 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.11) (duration: 11m 31s)
  • 00:00 mutante: switching apt.wikimedia.org from carbon to install1002 - there might be a short time until the LE SSL cert is also adjusted

2017-02-10

  • 23:41 RainbowSprinkles: gerrit: Restarting to pick up config changes
  • 23:12 RainbowSprinkles: gerrit: restarting service
  • 22:42 godog: start rsync of whisper metrics graphite2001 -> graphite1001 - T157022
  • 22:29 mutante: carbon - stopping puppet and most services, adding deprecation warning to motd, rsyncing data one last time (T132757)
  • 22:17 mutante: install1001 - shutdown ganeti instance and deleting it and its disk (T132757)
  • 21:27 mutante: install1001, install2001 - removed from Icinga, shutting down (T84380, T132757)
  • 21:18 mutante: install1001, install2001 - revoke puppet certs, puppet node deactivate, delete salt keys (T84380, T132757)
  • 21:03 ladsgroup@tin: Synchronized php-1.29.0-wmf.11/extensions/Nuke/Nuke_body.php: gerrit:337076 Fixing Special:Nuke (T156112, T156949, T156314) (duration: 00m 58s)
  • 21:03 Amir1: ladsgroup@tin:/srv/mediawiki-staging$ scap sync-file php-1.29.0-wmf.11/extensions/Nuke/Nuke_body.php 'gerrit:337076 Fixing Special:Nuke (T156112, T156949, T156314)'
  • 20:38 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic20(29|30|31|32).codfw.wmnet
  • 20:11 godog: silence graphite1001 for ssd reinstall - T157022
  • 18:27 brion: brion running throttled version of requeueTranscodes.php for low-res transcodes. expect increased load on video scalers but should remain responsive.
  • 18:23 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=elastic20(29|30|31|32).codfw.wmnet
  • 18:08 jynus: renabling delayed replication for dbstore2001 T130128
  • 17:57 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic20(27|28).codfw.wmnet
  • 16:15 mutante: scandium - stopping zuul-merger service (T150936)
  • 15:15 jynus: temporarily disabling mariadb replication lag checks to deploy new version of the icinga check script
  • 15:09 godog: bounce cassandra-a on xenon after https://gerrit.wikimedia.org/r/335826
  • 14:26 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic20(25|26).codfw.wmnet
  • 12:33 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: dc=eqiad,service=nginx,name=mw1227.eqiad.wmnet
  • 12:32 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: dc=eqiad,service=nginx,name=mw1228.eqiad.wmnet
  • 12:32 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: dc=eqiad,service=nginx,name=mw1229.eqiad.wmnet
  • 12:11 elukey: updating firewall rules for analytics on cr1/cr2
  • 12:00 godog: bounce mwerrors on eventlog1001 to pick up statsd cname change - T157022
  • 11:46 godog: roll-restart tileratorui in codfw/eqiad to pick up new statsd.eqiad.wmnet - T157022
  • 11:41 godog: roll-restart trendingedits on scb in eqiad/codfw to pick up new statsd.eqiad.wmnet - T157022
  • 11:36 godog: roll-restart mathoid/citoid/mobileapps/cxserver/eventstreams/graphoid on scb in eqiad/codfw to pick up new statsd.eqiad.wmnet - T157022
  • 11:30 godog: roll-restart changeprop on scb in eqiad/codfw to pick up new statsd.eqiad.wmnet - T157022
  • 11:25 godog: roll-restart nodepool on labnodepool1001 to pick up new statsd.eqiad.wmnet - T157022
  • 11:23 godog: roll-restart parsoid on ruthenium to pick up new statsd.eqiad.wmnet - T157022
  • 11:19 godog: roll-restart jmxtrans on conf* in codfw/eqiad to pick up new statsd.eqiad.wmnet - T157022
  • 11:16 godog: roll-restart tilerator in codfw/eqiad to pick up new statsd.eqiad.wmnet - T157022
  • 11:10 godog: restart navtiming ve asset-check statsd-mw-js-deprecate on hafnium to pick up statsd.eqiad.wmnet change - T157022
  • 10:54 godog: roll-restart karthoterian in codfw/eqiad to pick up new statsd.eqiad.wmnet - T157022
  • 10:39 godog: roll-restart parsoid in codfw/eqiad to pick up new statsd.eqiad.wmnet - T157022
  • 10:37 elukey: roll-restart of aqs to pick up new statsd.eqiad.wmnet - T157022
  • 10:34 godog: roll-restart ocg to pick up new statsd.eqiad.wmnet - T157022
  • 10:30 godog: roll-restart restbase in eqiad to pick up new statsd.eqiad.wmnet - T157022
  • 10:20 godog: restart of jmxtrans on analytics by elukey - T157022
  • 10:10 ema: lvs1007-10: upgrade to jessie 8.7, pybal 1.13.4, reboot into kernel 4.4.2-3+wmf8 T155401
  • 10:06 godog: roll-restart restbase in codfw to pick up new statsd.eqiad.wmnet - T157022
  • 10:03 hashar: rebooting contint2001
  • 09:51 hashar: Reenabling puppet and zuul-merger on contint1001 and contint2001. The git-daemon is running now T140297 T150936. The 'systemctl status git-daemon' thought that the service was running when it was not (filled T157785 )
  • 09:26 hashar: stopped zuul-merger process on contint1001 and contint2001. They lack the git-daemon service to expose the merges.
  • 08:45 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=elastic20(25|26|27|28).codfw.wmnet
  • 08:44 moritzm: upgrading hhvm on mw1200-mw1229
  • 08:41 elukey: restarting kafka mirror maker and jmxtrans of kafka[12]00[123] for java security upgrades
  • 08:30 marostegui: Deploye alter table s3 officewiki.echo_notification and mediawikiwiki.echo_notification tables only on codfw - T136428
  • 04:06 mutante: ganglia - switching aggregators from 1001 to 1002 and 2001 to 2002, there might be minor gaps in the graphs, but hey, it's deprecated anyways
  • 03:48 mutante: icinga - live hack fixing config - due to partially removed decom hosts mc2001-mc2016
  • 02:37 l10nupdate@tin: ResourceLoader cache refresh completed at Fri Feb 10 02:37:59 UTC 2017 (duration 5m 26s)
  • 02:32 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.11) (duration: 11m 53s)
  • 01:48 ebernhardson@tin: Synchronized wmf-config/CirrusSearch-common.php: Setup sister search prefix display types T149806 (duration: 00m 40s)
  • 01:43 ebernhardson@tin: Synchronized wmf-config/CirrusSearch-common.php: Setup sister search prefix display types T149806 (duration: 00m 48s)
  • 01:01 brion: transcode queue back to normal
  • 00:53 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wgPageAssessmentsSubprojects to true on English Wikipedia T157654 (duration: 00m 43s)
  • 00:53 brion: transcode high-prio queue may be briefly blocked by an influx of low-res transcodes queued in bulk. should return to normal in a bit.
  • 00:32 thcipriani@tin: Synchronized php-1.29.0-wmf.11/extensions/WikimediaEvents: SWAT: Enable Sister project search AB test T149806 (duration: 00m 45s)

2017-02-09

  • 22:58 bblack@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp2006.codfw.wmnet
  • 22:57 bblack: cp2006: unresponsive control, powercycled from racadm, normal boot, no evidence in logs - repooling for now
  • 22:48 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.29.0-wmf.11
  • 22:44 thcipriani@tin: Synchronized php-1.29.0-wmf.11/skins/CologneBlue/SkinCologneBlue.php: Revert "Remove warning suppression" (duration: 00m 59s)
  • 22:42 bblack: cp2006 depooled due to icinga report of host-down
  • 22:42 bblack@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp2006.codfw.wmnet
  • 22:02 brion: brion running tests of requeueTranscodes.php on terbium to restart subsets of video scaler work
  • 21:54 mutante: cp3014,cp3020,cp3022 - puppet node deactivate - cp3020 delete salt key (T130883)
  • 20:45 thcipriani@tin: Synchronized php-1.29.0-wmf.11/skins/CologneBlue/SkinCologneBlue.php: Fix a bunch of undefined indexes T157619 (sync actual skin file) (duration: 00m 40s)
  • 20:41 thcipriani@tin: Synchronized php-1.29.0-wmf.11/skins/CologneBlue/CologneBlue.php: Fix a bunch of undefined indexes T157619 (duration: 00m 41s)
  • 20:24 otto@tin: Finished deploy [analytics/refinery@9e689f3]: (no justification provided) (duration: 00m 20s)
  • 20:23 otto@tin: Started deploy [analytics/refinery@9e689f3]: (no justification provided)
  • 20:23 otto@tin: Started deploy [analytics/refinery@9e689f3]: (no justification provided)
  • 20:21 otto@tin: Finished deploy [analytics/refinery@9e689f3]: (no justification provided) (duration: 00m 04s)
  • 20:21 otto@tin: Started deploy [analytics/refinery@9e689f3]: (no justification provided)
  • 20:21 otto@tin: Started deploy [analytics/refinery@9e689f3]: (no justification provided)
  • 20:20 otto@tin: Finished deploy [analytics/refinery@9e689f3]: (no justification provided) (duration: 00m 07s)
  • 20:20 otto@tin: Started deploy [analytics/refinery@9e689f3]: (no justification provided)
  • 20:20 otto@tin: Started deploy [analytics/refinery@9e689f3]: (no justification provided)
  • 20:20 otto@tin: Started deploy [analytics/refinery@9e689f3]: (no justification provided)
  • 20:19 otto@tin: Started deploy [analytics/refinery@9e689f3]: (no justification provided)
  • 20:05 thcipriani@tin: Synchronized php-1.29.0-wmf.11/resources/src/mediawiki.special/mediawiki.special.search.interwikiwidget.styles.less: SWAT: Temporary hax to hide cawiki hacked in search sidebar T149806 (duration: 00m 40s)
  • 20:02 thcipriani@tin: Synchronized php-1.29.0-wmf.11/extensions/ConfirmEdit: SWAT: Add script for counting captchas Use an accurate number of captchas (duration: 00m 43s)
  • 19:57 ottomata: restarting main kafka brokers in codfw and then eqiad to pick up jvm updates
  • 19:50 thcipriani@tin: Synchronized php-1.29.0-wmf.11/extensions/TimedMediaHandler: SWAT: TMH job queue split into low and high priority PART III T155098 (duration: 00m 44s)
  • 19:49 thcipriani@tin: Synchronized php-1.29.0-wmf.11/extensions/TimedMediaHandler/TimedMediaHandler.hooks.php: SWAT: TMH job queue split into low and high priority PART II T155098 (duration: 00m 40s)
  • 19:47 thcipriani@tin: Synchronized php-1.29.0-wmf.11/extensions/TimedMediaHandler/TimedMediaHandler.php: SWAT: TMH job queue split into low and high priority PART I T155098 (duration: 00m 41s)
  • 19:37 smalyshev@tin: Finished deploy [wdqs/wdqs@1a7cd32]: Deploy new WAR build on 1003 (duration: 00m 16s)
  • 19:37 smalyshev@tin: Started deploy [wdqs/wdqs@1a7cd32]: Deploy new WAR build on 1003
  • 19:34 smalyshev@tin: Finished deploy [wdqs/wdqs@1a7cd32]: Deploy new WAR build on 2003 (duration: 00m 26s)
  • 19:34 smalyshev@tin: Started deploy [wdqs/wdqs@1a7cd32]: Deploy new WAR build on 2003
  • 19:29 ladsgroup@tin: Started deploy [ores/deploy@10fa16b]: (no justification provided)
  • 19:28 ladsgroup@tin: Started deploy [ores/deploy@a3a410b]: (no justification provided)
  • 19:26 thcipriani@tin: Synchronized php-1.29.0-wmf.11/extensions/TimedMediaHandler/SpecialTimedMediaHandler.php: SWAT: Only load necessary fields on Special:TimedMediaHandler lists (T157621) (duration: 00m 41s)
  • 19:19 ladsgroup@tin: Started deploy [ores/deploy@a3a410b]: (no justification provided)
  • 19:17 smalyshev@tin: Started deploy [wdqs/wdqs@1a7cd32]: Deploy new WAR build
  • 19:15 smalyshev@tin: Started deploy [wdqs/wdqs@1a7cd32]: (no justification provided)
  • 19:14 bd808: Restarted logstash on logstash1001. Dead since 2017-02-09T06:39:46 with "java.lang.UnsupportedOperationException" crash in worker thread.
  • 19:13 smalyshev@tin: Started deploy [wdqs/wdqs@1a7cd32]: (no justification provided)
  • 19:08 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Setting $wgPageAssessmentsSubprojects to true on testwiki T157654 (duration: 00m 54s)
  • 19:05 ladsgroup@tin: Started deploy [ores/deploy@4fdaf7d]: (no justification provided)
  • 18:50 godog: test bouncing jmxtrans on kafka1012 to pick up statsd changes
  • 18:35 godog: bounce zuul to pick up statsd DNS change - T157022
  • 18:34 ladsgroup@tin: Finished deploy [ores/deploy@e27e845]: (no justification provided) (duration: 04m 33s)
  • 18:30 ladsgroup@tin: Started deploy [ores/deploy@e27e845]: (no justification provided)
  • 18:29 Amir1: starting deploy of ores:e27e845 to canary node
  • 17:55 elukey: proactively restarted statsv on hafnium after the kafka broker restarts
  • 17:42 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic20(18|21|22|23|24).codfw.wmnet
  • 16:38 godog: flip dns records for statsd/carbon to graphite2001 - T157022
  • 16:37 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2040 (duration: 00m 52s)
  • 16:22 ema: lvs1011: upgrade to jessie 8.7, pybal 1.13.4, reboot into kernel 4.4.2-3+wmf8 T155401
  • 16:17 marostegui: Shutdown db2060 for maintenance - T156161
  • 16:15 marostegui: Compressing commonswiki on labsdb1009 - T153743
  • 16:08 ema: lvs1012: upgrade to jessie 8.7, pybal 1.13.4, reboot into kernel 4.4.2-3+wmf8 T155401
  • 16:06 jynus: rolling restart of replication threads for dbstore1002/2001/2002 T111654
  • 15:49 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=aqs1009.eqiad.wmnet
  • 15:42 godog: roll-restart diamond to pick up graphite2001 changes
  • 15:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1086 - T136428 (duration: 00m 44s)
  • 15:23 ema: shutdown cp3020 T130883
  • 15:21 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1086 - T136428 (duration: 00m 40s)
  • 15:19 elukey: restarting all Analytics Kafka brokers for Java security upgrades
  • 15:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1079 - T136428 (duration: 00m 40s)
  • 15:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1079 - T136428 (duration: 00m 43s)
  • 14:53 moritzm: upgrading hhvm on mw1189-mw1199 and mw1293/mw1294
  • 14:48 godog: move diamond traffic to graphite2001 - T157022
  • 14:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1062 - T136428 (duration: 00m 41s)
  • 14:20 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1062 - T136428 (duration: 00m 45s)
  • 14:01 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=elastic20(21|22|23|24).codfw.wmnet
  • 13:45 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic2020.codfw.wmnet
  • 13:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1028 - T153300 (duration: 00m 41s)
  • 13:11 moritzm: upgrading firejail on sca cluster
  • 12:52 gehel: killing salt runs stuck on failing reimage of elastic2018
  • 12:37 mforns@tin: Finished deploy [analytics/refinery@9e689f3]: (no justification provided) (duration: 03m 05s)
  • 12:34 mforns@tin: Started deploy [analytics/refinery@9e689f3]: (no justification provided)
  • 11:56 moritzm: upgrading hhvm on mw1170-mw1188 (also effecting updates of openssl, libgd, lcms, gnutls, sqlite, libxpm and glibc)
  • 11:39 gehel: failed reimage on elastic201[89], restarting
  • 10:54 moritzm: deploy exim and openssh bugfix updates from jessie point release
  • 10:51 moritzm: upgrading java on kafka clusters and druid
  • 10:49 elukey: restarting Java daemons on druid100[123] for security upgrades
  • 10:42 jynus: preparing to reimage db2040 T111654
  • 10:37 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2040 (duration: 00m 40s)
  • 10:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1034 - T111654 (duration: 00m 41s)
  • 10:09 hashar: Restarted Jenkins on contint1001
  • 10:04 hashar: Running package upgrades on contint2001
  • 10:03 elukey: restore Hadoop master to an1001
  • 09:57 elukey: failover Hadoop masters from an1001 to an1002 to allow Java upgrades
  • 09:52 gehel: cleaning up logs on elastic20(01|16) - T139043
  • 09:50 elukey: restarting oozie and hive on analytics1003 for java security upgrades
  • 09:39 marostegui: Deploy alter table on eqiad hosts for s7 metawiki and wiki on the echo_notification tables - T136428
  • 09:38 jynus: upgrading and restarting db1034 T111654
  • 09:34 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1034 (duration: 00m 44s)
  • 09:32 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=elastic20(17|18|19|20).codfw.wmnet
  • 09:20 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2057 (duration: 00m 41s)
  • 09:17 gehel: restarting blazegraph on wdqs1003 to ensure proper war is loaded
  • 09:10 marostegui: Deploy alter table on codfw hosts for s7 metawiki and wiki on the echo_notification tables - T136428
  • 09:08 moritzm: restarting archiva on meitnerium for java security update
  • 09:07 elukey: Executing Cassandra nodetool cleanup on aqs1006-{a,b} (one at the time) and aqs1009-a
  • 09:01 elukey: restarting java daemons on all the Hadoop nodes for security upgrades
  • 08:59 gehel: cleaning empty logs on elastic10(22|24|40) - thanks elukey !
  • 08:51 moritzm: installing Java security updates on Hadoop cluster
  • 08:45 moritzm: installing Java security updates on stat* and contint1001
  • 08:17 marostegui: Compressing commonswiki tables on db1095 - T153743
  • 07:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Added comment for db1064 being master of db1095 - T153743 (duration: 00m 40s)
  • 07:46 elukey: Renamed some logs in /var/log (adding _renamed) on aluminum, elastic102[46]/1040 to avoid cronspam and logrotate failures
  • 07:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1064 - T153743 (duration: 00m 40s)
  • 07:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1073 - T156126 (duration: 00m 42s)
  • 03:14 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Feb 9 03:14:55 UTC 2017 (duration 5m 44s)
  • 03:09 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.11) (duration: 15m 19s)
  • 02:36 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.10) (duration: 15m 05s)
  • 01:10 twentyafterfour: phabricator upgrade finished.
  • 01:07 dereckson@tin: Synchronized wmf-config/throttle.php: Update throttle rules (Gerrit:336552 for it.wikiversity + Gerrit:336741 for cleaning) (duration: 00m 40s)
  • 01:02 twentyafterfour: starting phabricator deployment #phab-2017-02-08
  • 00:57 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Switch to SiteMatrixInterwikiResolver for AB test (Gerrit:336738) (duration: 00m 41s)
  • 00:47 dereckson@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Configuration change for RelatedArticles (labs only, Gerrit:336740) (duration: 00m 40s)
  • 00:34 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Enable Quiz on Spanish Wikibooks (duration: 00m 41s)
  • 00:24 dereckson@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Configuration changes for RelatedArticles (labs only, Gerrit:336732 and Gerrit:336733) (duration: 00m 40s)
  • 00:20 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Prune wgMinervaUseFooterV2 (T157075) (duration: 00m 41s)
  • 00:11 dereckson@tin: Synchronized wmf-config/CommonSettings.php: Use https:// urls when communicating with PediaPress (T157398) (duration: 00m 41s)

2017-02-08

  • 23:24 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic2016.codfw.wmnet
  • 23:04 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to 1.29.0-wmf.11 -- T157621 is not code-change related
  • 22:45 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 back to 1.29.0-wmf.10
  • 22:43 thcipriani: rolling back for wmf.11 from group1 due to T157621
  • 22:33 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic2015.codfw.wmnet
  • 22:26 demon@tin: Synchronized php-1.29.0-wmf.11/includes/WebResponse.php: Debugging fun times (duration: 00m 50s)
  • 22:21 gehel: elastic2016 not coming up after reimage - powercycling
  • 22:01 mholloway-shell@tin: Finished deploy [mobileapps/deploy@0efa7b8]: Update service-mobileapp-node to f45bfff (duration: 02m 55s)
  • 21:58 mholloway-shell@tin: Started deploy [mobileapps/deploy@0efa7b8]: Update service-mobileapp-node to f45bfff
  • 21:43 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic20(13|14).codfw.wmnet
  • 21:42 halfak@tin: Finished deploy [ores/deploy@7c80636]: (no justification provided) (duration: 01m 26s)
  • 21:41 halfak@tin: Started deploy [ores/deploy@7c80636]: (no justification provided)
  • 21:36 halfak@tin: Finished deploy [ores/deploy@7c80636]: (no justification provided) (duration: 03m 45s)
  • 21:32 halfak@tin: Started deploy [ores/deploy@7c80636]: (no justification provided)
  • 20:54 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.29.0-wmf.11
  • 20:52 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=elastic20(13|14|15|16).codfw.wmnet
  • 20:47 thcipriani@tin: Synchronized php-1.29.0-wmf.11/extensions/MobileFrontend/includes/api/ApiMobileView.php: Pass revision id to parseSectionsData to avoid warnings T157515 (duration: 00m 42s)
  • 19:54 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Enable VE on fr.wiktionary Projet: namespace (T156660) (duration: 00m 44s)
  • 19:44 Dereckson: mwscript updateCollation.php --wiki=olowiki --previous-collation=uppercase (T147064, 4238 rows processed)
  • 19:39 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Set category collation for olo.wikipedia (T146612, T147064) (duration: 00m 43s)
  • 19:29 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Namespace configuration for ml. projects (T56951) (duration: 00m 41s)
  • 19:17 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Create autopatrolled and rollbacker permissions for fa.wikiquote (T156163) (duration: 00m 43s)
  • 19:08 dereckson@tin: Synchronized php-1.29.0-wmf.11/extensions/UploadWizard/UploadWizard.config.php: Disable Firefogg support (T157201) (duration: 00m 44s)
  • 19:07 mutante: bastion hosts, people.wm: deluser volkere, let puppet create volker-e, move data, delete old home dir (T157591)
  • 19:05 dereckson@tin: Synchronized php-1.29.0-wmf.10/extensions/UploadWizard/UploadWizard.config.php: Disable Firefogg support (T157201) (duration: 00m 46s)
  • 19:02 mutante: temp. disabling puppet and doing some debugging on bastion hosts, renaming a user
  • 18:33 demon@tin: Synchronized multiversion/: Dropping old MWVersion shim (duration: 00m 57s)
  • 18:05 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic20(09|10|11|12).codfw.wmnet
  • 18:02 jynus: upgrading and restarting db2057 T111654
  • 17:46 jynus@tin: Synchronized wmf-config/db-codfw.php: depool db2057 (duration: 00m 41s)
  • 17:45 elukey: added some annotations to the aqs analytics ACLs on cr1/cr2
  • 17:30 jynus@tin: Synchronized wmf-config/db-eqiad.php: repool db1030 (duration: 00m 40s)
  • 17:04 jynus: rolling restart of replication thread of 29 mysql hosts T111654
  • 17:02 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=elastic20(09|10|11|12).codfw.wmnet
  • 16:32 ema: cp3045 stuck rebooting, power-cycled
  • 16:20 ema: cp2017 stuck rebooting, power-cycled
  • 16:19 jynus: upgrading and restarting db1030 T111654
  • 16:15 ema: pybal 1.13.4 built and uploaded to carbon
  • 16:12 jynus@tin: Synchronized wmf-config/db-eqiad.php: depool db1030 (duration: 00m 41s)
  • 16:10 chasemp: maintain-views and maintain-meta_p full runs on labsdb1009/10/11
  • 15:46 marostegui@tin: Synchronized wmf-config/db-codfw.php: db1073 change IP - T156126 (duration: 00m 40s)
  • 15:45 marostegui@tin: Synchronized wmf-config/db-eqiad.php: db1073 change IP - T156126 (duration: 00m 40s)
  • 15:41 ema: cp2011 stuck rebooting, power-cycled
  • 15:40 jynus@tin: Synchronized wmf-config/db-eqiad.php: repool db1026 (duration: 00m 41s)
  • 15:28 elukey: Eqiad cr1/cr2 - Updated analytics-in4 for new aqs nodes and removed decommed ones
  • 15:20 hoo@tin: Synchronized php-1.29.0-wmf.10/extensions/Wikidata: Wikibase uses multiple EntityPrefetchers (T157380) (duration: 02m 07s)
  • 15:15 hoo@tin: Synchronized php-1.29.0-wmf.11/extensions/Wikidata: Wikibase uses multiple EntityPrefetchers (T157380) (duration: 02m 11s)
  • 14:54 marostegui: Shutdown db1073 for maintenance - https://phabricator.wikimedia.org/T156126
  • 14:31 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic2008.codfw.wmnet
  • 14:30 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic2007.codfw.wmnet
  • 14:30 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic2006.codfw.wmnet
  • 14:29 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic2005.codfw.wmnet
  • 14:26 elukey: restarting nutcracker in all the codfw mw servers to pick up the new shards
  • 14:23 ema: cp2022 stuck rebooting, power-cycled
  • 14:23 gehel: drain shards from elastic20(09|10|11|12) in preparation for reimage - T151326
  • 14:17 jynus: upgrading and restarting db1026 T111654
  • 13:46 elukey: replacing the codfw memcached/redis shards 12->16
  • 13:41 marostegui: Start replication on db1064 - T153743
  • 13:40 marostegui: Enable replication between db1095 and db1064 - T153743
  • 13:39 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1026 for maintenance (duration: 00m 41s)
  • 13:10 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=elastic2008.codfw.wmnet
  • 13:10 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=elastic2007.codfw.wmnet
  • 13:09 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=elastic2006.codfw.wmnet
  • 13:09 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=elastic2005.codfw.wmnet
  • 12:43 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1073 - T156126 (duration: 00m 40s)
  • 12:38 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1037 after maintenance (duration: 00m 41s)
  • 12:17 jynus: upgrading and restarting db1037 T111654
  • 12:16 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1037 for maintenance (duration: 00m 40s)
  • 12:01 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1045 after maintenance (duration: 00m 42s)
  • 10:39 jynus: upgrading and restarting db1045 T111654
  • 10:38 moritzm: upgrading openssl, libgd, lcms, gnutls, sqlite, libxpm and glibc in codfw mediawiki cluster (so get get effected by the restart during the HHVM upgrade)
  • 10:11 moritzm: upgrading hhvm on codfw mediawiki cluster
  • 09:55 hoo: Updated the Wikidata property suggester with data from last Monday's JSON dump and applied the T132839 workarounds
  • 09:44 elukey: boostrapping aqs1009-b (last new AQS Cassandra instance)
  • 09:31 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic2004.codfw.wmnet
  • 09:31 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic2003.codfw.wmnet
  • 08:56 ema: cache_upload: upgrade to jessie 8.7 and reboot into kernel 4.4.2-3+wmf8 T155401
  • 08:42 gehel: drain shards from elastic200[5678] in preparation for reimage - T151326
  • 08:18 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=elastic2004.codfw.wmnet
  • 08:18 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=elastic2003.codfw.wmnet
  • 08:16 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic2002.codfw.wmnet
  • 07:59 marostegui: Adding 100G to the lv on dbstore1001
  • 07:23 marostegui: Restart MySQL db1095 and labsdb1009 for maintenance - T153743
  • 03:08 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Feb 8 03:08:05 UTC 2017 (duration 5m 43s)
  • 03:02 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.11) (duration: 15m 32s)
  • 02:29 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.10) (duration: 07m 35s)
  • 01:02 mutante: mw1294 - run puppet because it popped up in Icinga as failed - removes a bunch of /var/tmp/core/../rsvg-convert.*, all else normal
  • 00:56 thcipriani@tin: Synchronized wmf-config/InitialiseSettings-labs.php: SWAT Setting $wgPageAssessmentsSubprojects to true on beta cluster (housekeeping sync) (duration: 00m 40s)
  • 00:55 mutante: mw1189 service hhvm restart
  • 00:55 mutante: iridum - apache graceful'ed
  • 00:51 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Update footer logos on mobile site for various projects PART II T157476 (duration: 00m 41s)
  • 00:50 thcipriani@tin: Synchronized static/images/mobile/copyright: SWAT: Update footer logos on mobile site for various projects PART I T157476 (duration: 00m 41s)
  • 00:16 thcipriani@tin: Synchronized wmf-config: SWAT: Deploy TextCat Improvements T149324 T142140 (duration: 00m 45s)

2017-02-07

  • 23:42 mutante: carbon - stopping DHCP service (install* should be used)
  • 22:31 otto@tin: Finished deploy [eventstreams/deploy@e86077c]: (no justification provided) (duration: 02m 26s)
  • 22:28 otto@tin: Started deploy [eventstreams/deploy@e86077c]: (no justification provided)
  • 21:17 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.29.0-wmf.11
  • 21:02 eileen1: Update CiviCRM from e45da6d to 7b36996
  • 21:01 thcipriani@tin: Finished scap: testwiki to php-1.29.0-wmf.11 and rebuild l10n cache (duration: 51m 53s)
  • 20:10 thcipriani@tin: Started scap: testwiki to php-1.29.0-wmf.11 and rebuild l10n cache
  • 19:43 gehel: deploying analysis-stempel plugin on relforge and cluster restart
  • 19:34 gehel: drain shards from elastic200[34] in preparation for reimage - T151326
  • 19:30 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1015 after maintenance (duration: 01m 34s)
  • 18:58 jynus: preparing db2043 for reimage T152188
  • 18:55 jynus: restarting and upgrading db1015 T152188
  • 18:55 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1015 after maintenance (duration: 00m 54s)
  • 18:33 thcipriani: starting branch cut for MediaWiki and extensions 1.29.0-wmf.11
  • 18:24 arlolra: Updated Parsoid to f0732260 (T109897)
  • 18:20 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1022 after maintenance (duration: 00m 40s)
  • 18:18 arlolra@tin: Finished deploy [parsoid/deploy@c3a5c55]: Updating Parsoid to f0732260 (duration: 09m 05s)
  • 18:09 arlolra@tin: Started deploy [parsoid/deploy@c3a5c55]: Updating Parsoid to f0732260
  • 17:50 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic2001.codfw.wmnet
  • 17:07 jynus: restarting and upgrading db1022 T152188
  • 17:05 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1022 for maintenance (duration: 00m 40s)
  • 16:22 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1021 (duration: 00m 57s)
  • 16:08 jynus: restarting and upgrading db2064 T152188
  • 14:56 jynus: preparing db2036 for reimage T152188
  • 14:42 gehel: drain shards from elastic2001 / elastic2002 in preperation for reimage - T151326
  • 14:28 hashar: European swat copleted
  • 14:27 elukey: restarting hhvm on mw1304 (load very high, no queue, threads locked - /tmp/hhvm.62070.bt.)
  • 14:22 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Namespace changes for elwikisource - T157187 (duration: 00m 40s)
  • 14:19 elukey: restarting all the Yarn Node Managers on the Hadoop worker nodes to pick up the new config - T156932
  • 14:12 hoo@tin: Synchronized wmf-config/: Search index article placeholders on cywiki up to Q2794 (T144592) (duration: 00m 42s)
  • 14:10 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Introduce $wmgArticlePlaceholderSearchEngineIndexed (duration: 00m 52s)
  • 14:05 marostegui: Importing commonswiki tables on labsdb1009 - T153743
  • 13:54 jynus: restarting and upgrading db1021
  • 13:45 marostegui: Importing commonswiki tables on db1095 - T153743
  • 12:41 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1021 (duration: 00m 41s)
  • 12:16 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1036 (duration: 00m 40s)
  • 11:56 moritzm: restarting hhvm on appserver canaries to pick up lcms, sqlite, libxpm, gnutls and glibc updates (from jessie 8.7 release and security updates)
  • 11:53 godog: stop puppet on ms-be1012 and change rsyslog to avoid local syslog spam - T157237
  • 11:42 moritzm: installing libxpm security updates
  • 11:39 jynus: restarting and upgrading db1036
  • 11:37 elukey: restarting hhvm on mw1226 (hhvm dump debug in /tmp/hhvm.33183.bt.)
  • 11:32 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1036 (duration: 00m 40s)
  • 11:27 moritzm: installing libgd security updates
  • 11:13 moritzm: installing libpng security updates
  • 10:34 jynus: preparing db2046 for reimage T152188
  • 10:02 _joe_: uploaded etcd-mirror 0.0.1 to jessie-wikimedia (T156009)
  • 09:48 moritzm: installing cairo security updates
  • 09:46 elukey: stopped and masked cassandra-{a,b} - T157425
  • 08:40 marostegui: Transferring commonswiki tables from db1064 to db1095 - T153743
  • 07:31 elukey: added "> /dev/null" manually to the carbon's root crontab (rsync job) to avoid cronspam. The change was already merged in https://gerrit.wikimedia.org/r/#/c/336218 but puppet is disabled on carbon.
  • 07:08 marostegui: Transferring commonswiki tables from db1064 to labsdb1009 - T153743
  • 07:06 marostegui: Importing commonswiki tables on labsdb1010 - T153743
  • 06:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1072 - T156226 (duration: 00m 50s)
  • 05:41 volans: ms-be1012 running out of space on /, manually compressed /var/log/swift/server.log.1 and cleaned up apt cache T157237
  • 02:37 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Feb 7 02:37:35 UTC 2017 (duration 5m 16s)
  • 02:35 cwd: imported triggers into staging civi
  • 02:32 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.10) (duration: 13m 23s)
  • 01:31 mutante: prometheus1004 - installed OS, signing puppet cert, initial run.. (T152504)
  • 00:49 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Fix $wmgVisualEditorAvailableNamespaces code style (Gerrit:336346) (no-op) (duration: 00m 40s)
  • 00:31 mutante: install1001 - re-enabled puppet, start DHCP service
  • 00:26 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Show again svwiki logo between 1.5x and 2x zoom (T157387) (duration: 00m 40s)

2017-02-06

  • 23:45 mutante: prometheus1003 - installed OS, signing puppet cert, initial run (T152504)
  • 22:57 Krinkle: Purged https://en.wikipedia.org/static/apple-touch/wikipedia.png (mwscript purgeList.php) for T152538
  • 21:39 bsitzmann@tin: Finished deploy [mobileapps/deploy@9b42448]: Update mobileapps to 034a391 (duration: 03m 59s)
  • 21:37 otto@tin: Finished deploy [eventstreams/deploy@c938a57]: (no justification provided) (duration: 01m 47s)
  • 21:35 otto@tin: Started deploy [eventstreams/deploy@c938a57]: (no justification provided)
  • 21:35 bsitzmann@tin: Started deploy [mobileapps/deploy@9b42448]: Update mobileapps to 034a391
  • 20:49 mutante: cp3011 thru cp3022 - shutdown / poweroff (T130883)
  • 20:39 tgr@tin: Synchronized php-1.29.0-wmf.10/extensions/JsonConfig/includes/JCUtils.php: T155532: Update JsonConfig login API call (duration: 01m 00s)
  • 20:27 mutante: cp3011 thru cp3022 - revoke puppet certs, puppet node deactivate (T130883)
  • 19:37 jynus: restarting db2060 after kernel upgrade
  • 19:30 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: Second half of changing project logo for hi.wikibooks.org (duration: 00m 39s)
  • 19:29 ebernhardson@tin: Synchronized static/images/project-logos/hiwikibooks.png: First part of Changing project logo for hi.wikibooks.org (duration: 00m 39s)
  • 19:25 ebernhardson: re-pulled 336242 to mwdebug1002
  • 19:16 ebernhardson: pulled 336242 to mwdebug1002
  • 19:13 jynus: preparing to reimage db2045 T152188
  • 19:10 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: Configure A/B test for CrossProject search results sidebar (duration: 00m 49s)
  • 19:08 ebernhardson: pulled 334673 to mwdebug1002
  • 18:16 jynus: preparing to reimage db2050 T152188
  • 18:06 marostegui: Start to transfer commonswiki ibd and cfg from db1064 to labsdb1010 - https://phabricator.wikimedia.org/T153743
  • 17:41 mobrovac: restbase end deploy of ea980cc5
  • 17:10 mobrovac: restbase start deploy of ea980cc5
  • 17:03 hashar: Nodepool/CI back up
  • 16:51 marostegui: Stop MySQL and shutdown db1072 for raid and BBU replacement - T156226
  • 16:51 marostegui: Stop MySQL and shutdown db1072 for raid
  • 16:35 hashar: Nodepool Jessie images are back up. Trusty one is being rebuild..
  • 15:55 elukey: mc2029 shutdown for DC ops
  • 15:46 hashar: Stopping Nodepool for maintenance
  • 15:42 oblivian@tin: Finished deploy [changeprop/deploy@5f932a3]: Revert ORES throttling (duration: 03m 49s)
  • 15:39 oblivian@tin: Started deploy [changeprop/deploy@5f932a3]: Revert ORES throttling
  • 15:38 moritzm: installing lcms security updates on mediawiki canaries
  • 15:17 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs1001.eqiad.wmnet
  • 15:16 gehel: starting reimage of wdqs1001 - T144536
  • 15:13 ema: cache_text: upgrade to jessie 8.7 and reboot into kernel 4.4.2-3+wmf8 T155401!log cache_maps: upgrade to jessie 8.7 and reboot into kernel 4.4.2-3+wmf8 T155401
  • 15:10 addshore@tin: Synchronized php-1.29.0-wmf.10/extensions/ORES/extension.json: T157206 ORES - Remove all (except meta) API funcationality hooks (take2) (duration: 00m 54s)
  • 14:51 moritzm: upgrading mw1262-mw1265 to hhvm 3.12.11+dfsg-1+wmf2
  • 14:46 addshore: EU SWAT all done!
  • 14:45 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: PROD-NOOP Enable InterwikiSorting on beta (duration: 00m 39s)
  • 14:42 addshore@tin: Synchronized wmf-config/Wikibase.php: T155995 Rm InterwikiSorting settings from wmgWikibaseClientSettings PT 2/2 (duration: 00m 39s)
  • 14:40 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T155995 Rm InterwikiSorting settings from wmgWikibaseClientSettings PT 1/2 (duration: 00m 41s)
  • 14:33 gehel: resetting analytics-wmde/scripts on stat1002 to the correct "production" branch
  • 14:32 ema: cp2018 stuck rebooting, powercycled
  • 14:30 addshore@tin: Synchronized dblists/compact-language-links.dblist: T157108 & T157112 Deploy Compact Language Links out of beta in French/Dutch Wikipedia (duration: 00m 40s)
  • 14:23 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Copy InterwikiSorting settings from wmgWikibaseClientSettings (duration: 00m 40s)
  • 14:19 marostegui: Stop MySQL and shutdown db2060 for maintenance - T156161
  • 14:15 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T155717 Enable TwoColConflict on mw.org (duration: 00m 40s)
  • 14:09 addshore@tin: Synchronized php-1.29.0-wmf.10/extensions/ORES/extension.json: T157206 ORES - Remove all (except meta) API funcationality hooks (duration: 00m 51s)
  • 14:05 volans: fixed duplicate entries in source.list on db2040 and es2002 (trusty)
  • 13:44 ema: cache_misc: upgrade to jessie 8.7 and reboot into kernel 4.4.2-3+wmf8 T155401
  • 13:30 elukey: applied https://gerrit.wikimedia.org/r/#/c/336203/ manually to analytics1028 (hadoop worker node) as live test - T156932
  • 13:23 gehel: removing stale puppet lock file on elastic10(22|26)
  • 12:03 moritzm: upgrading mwdebug* and mw1261 to hhvm 3.12.11+dfsg-1+wmf2
  • 10:21 oblivian@tin: Finished deploy [changeprop/deploy@ac11ebe]: Deploying ores concurrency/disabling (duration: 00m 38s)
  • 10:20 oblivian@tin: Started deploy [changeprop/deploy@ac11ebe]: Deploying ores concurrency/disabling
  • 10:18 oblivian@tin: Started deploy [changeprop/deploy@ac11ebe]: Deploying ores concurrency/disabling to canary
  • 10:17 gehel: data import complete for wdqs1003, repooling - T152643
  • 10:14 marostegui: Started to transfer commonswiki (ibd and cfg) from db1064 to labsdb1011 - T153743
  • 10:14 ema: cache_maps: upgrade to jessie 8.7 and reboot into kernel 4.4.2-3+wmf8 T155401
  • 09:40 gehel: elasticsearch - reindexing from 2017-02-04T20:00:00Z to 2017-02-05T23:59:00Z - T139043
  • 09:36 hoo: Removed 2fa from an account, per T157191
  • 09:30 marostegui: Stop MySQL Replication on db1064 for maintenance - T153743
  • 09:26 marostegui: Deploy ALTER table db1028 metawiki.pagelinks - T153300
  • 09:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1028 - T153300 (duration: 00m 42s)
  • 08:43 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1064 - T153743 (duration: 00m 41s)
  • 08:19 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=aqs1008.eqiad.wmnet
  • 07:38 elukey: bootstrapping aqs1009-a (new AQS cassandra instance)
  • 07:30 marostegui: Stop MySQL on db1095 to snapshot it to es1017 - T153743
  • 07:03 marostegui: Upgrade mariadb+packages db1039 - T153300
  • 07:01 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2060 - T156161 (duration: 00m 40s)
  • 02:25 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Feb 6 02:25:51 UTC 2017 (duration 5m 20s)
  • 02:20 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.10) (duration: 07m 30s)

2017-02-05

  • 23:42 gehel: truncating elasticsearch logs on elastic10(24|26|40) - T139043
  • 23:41 gehel: truncating elasticsearch logs on elastic1022 - T139043
  • 03:28 Amir1: ladsgroup@scb100[1-4]:~$ sudo service celery-ores-worker restart (T157206)
  • 02:26 l10nupdate@tin: ResourceLoader cache refresh completed at Sun Feb 5 02:26:26 UTC 2017 (duration 5m 18s)
  • 02:21 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.10) (duration: 08m 20s)

2017-02-04

  • 19:24 elukey: Started nodetool-b cleanup on aqs1005 (after 1008-{ab} bootstraps)
  • 11:44 elukey: Started nodetool-a cleanup on aqs1008 (after 1008-{ab} bootstraps)
  • 09:09 elukey: Started nodetool-a cleanup on aqs1005 (after 1008-{ab} bootstraps)
  • 02:28 l10nupdate@tin: ResourceLoader cache refresh completed at Sat Feb 4 02:28:33 UTC 2017 (duration 5m 23s)
  • 02:23 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.10) (duration: 07m 53s)

2017-02-03

  • 22:27 mutante: switching webproxy.*.wmnet CNAMEs from carbon to new install servers (T123733) - watching squid access logs
  • 19:02 ostriches: gerrit: flushed all web_sessions, you'll have to login again. Sorry
  • 18:07 godog: stop carbon-cache on graphite1001 to prevent useless write load
  • 17:05 godog: fail over read traffic from graphite1001 to graphite2001 https://gerrit.wikimedia.org/r/335761 - T157022
  • 16:10 godog: rsync coal data graphite1001 -> graphite2001
  • 15:03 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2061 (duration: 00m 40s)
  • 15:01 jynus: preparing to reimage db2039 T111654
  • 14:35 chasemp: restart apache on graphite1001 to see if it helps sqlite lock isssue
  • 14:30 jynus: upgrade and restart db2061 T111654
  • 14:26 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2061 (duration: 00m 40s)
  • 13:58 jynus: restarting and upgrading db2041 T111654
  • 12:16 gehel: restarting relforge1001 to pick up new master configuration
  • 11:21 jynus: preparing to reimage db2054 T111654
  • 11:01 marostegui: Alter table metawiki.pagelinks on db1039 (depooled) - T153300
  • 10:54 jynus: preparing to reimage db2053 T111654
  • 10:50 moritzm: mwdebug* and mw1261 have been reverted to previous HHVM package
  • 10:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1064 - T153743 (duration: 00m 42s)
  • 10:04 marostegui: Reboot db1064 to pick up the new kernels T153743
  • 09:59 marostegui: Upgrade db1064 from MariaDB 10.0.23 to 10.0.29 - T153743
  • 09:55 gehel: restarting relforge1002 to pick up new master configuration
  • 09:54 jynus: upgrade & restart of db2063 T111654
  • 09:48 marostegui: Restart mysql on db1064 to get its binary log changed to ROW - T153743
  • 09:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1064 - T153743 (duration: 00m 40s)
  • 09:41 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db1064 - T153743 (duration: 00m 40s)
  • 09:39 moritzm: upgraded mwdebug* and mw1261 to the new HHVM package
  • 09:22 moritzm: uploaded hhvm_3.12.11+dfsg-1+wmf2 to apt.wikimedia.org
  • 09:10 elukey: Replace Redis/Memcached shards mc2008->2011 with mc2026->mc2029
  • 08:50 moritzm: installing tomcat regression updates on trusty hosts (jessie update was fine)
  • 08:15 moritzm: restarting prometheus servers to pick up openssl update
  • 08:05 elukey: bootstrapping aqs1008-b (AQS Cassandra instance)
  • 07:41 moritzm: upgrading firejail on remaining wtp/Parsoid hosts
  • 07:15 marostegui: Stop MySQL db1095 to snapshot it to es1013:/srv/tmp - T153743
  • 02:38 l10nupdate@tin: ResourceLoader cache refresh completed at Fri Feb 3 02:38:10 UTC 2017 (duration 5m 3s)
  • 02:33 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.10) (duration: 12m 06s)
  • 00:51 dereckson@tin: Synchronized dblists/related-articles-footer-blacklisted-skins.dblist: Adjust RelatedArticles deployment scale for Mobile English Wikipedia (T154681) (duration: 00m 39s)
  • 00:48 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Set site name for ku.wiktionary (T29878) (duration: 00m 39s)
  • 00:45 dereckson@tin: Synchronized wmf-config/: Adjust RelatedArticles deployment scale for Mobile English Wikipedia (T154681) (duration: 00m 42s)
  • 00:42 dereckson@tin: Synchronized dblists/related-articles-footer-blacklisted-skins.dblist: Enable RelatedArticles on Mobile French Wikipedia (T156362) (duration: 00m 44s)
  • 00:33 dereckson@tin: Synchronized static/apple-touch/wikipedia.png: Update apple touch icon for Wikipedia (T152538) (duration: 00m 39s)
  • 00:24 dereckson@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Limit page images on beta cluster to images in the lead section (no-op in prod) (duration: 00m 41s)
  • 00:17 dereckson@tin: Synchronized wmf-config/interwiki.php: Interwiki map update (duration: 00m 40s)

2017-02-02

  • 23:51 twentyafterfour: 1.29.0-wmf.10 appears to be stable. Train deployment complete.
  • 23:38 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.29.0-wmf.10
  • 23:33 twentyafterfour@tin: Synchronized php-1.29.0-wmf.10/includes/cache/MessageCache.php: deploy I5b84b1 refs T156996 (duration: 00m 45s)
  • 23:11 greg-g: Gerrit: we'll be flushing session caches momentarily, sorry for the inconvenience
  • 21:50 gehel: reimaging relforge1002.eqiad.wmnet
  • 20:35 mutante: carbon - disabling puppet (to stop it from re-adding second IPv6 address causing issues with ferm rules)
  • 20:17 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 wikis to 1.29.0-wmf.9
  • 20:14 twentyafterfour: rolling back to wmf.9 due to T156996
  • 20:10 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.29.0-wmf.10
  • 20:04 twentyafterfour: deploying MediaWiki 1.29.0-wmf.10 to all wikis
  • 19:29 tgr: reset wikimedia 2FA for jdlrobson
  • 19:24 tgr: reset wikitech 2FA for jdlrobson
  • 19:13 hashar: Gracefully restarting Jenkins
  • 19:09 ejegg: updated fundraising tools from 931c8cf to 8a65e38
  • 19:03 ejegg: re-enabled thank you mail job
  • 18:59 ejegg: disabled thank you mail job
  • 18:04 bsitzmann@tin: Finished deploy [mobileapps/deploy@09101f7]: try previous deploy again (at least on canary) (duration: 00m 51s)
  • 18:03 bsitzmann@tin: Started deploy [mobileapps/deploy@09101f7]: try previous deploy again (at least on canary)
  • 18:01 bearND: ^ reverted previous deploy due to incorrect links in the news endpoint
  • 18:00 bsitzmann@tin: Finished deploy [mobileapps/deploy@09101f7]: (no justification provided) (duration: 01m 56s)
  • 17:58 bsitzmann@tin: Started deploy [mobileapps/deploy@09101f7]: (no justification provided)
  • 17:56 jynus: upgrade & restart of db2059 T111654
  • 17:43 jynus: upgrade & restart of db2052 T111654
  • 17:32 mobrovac: restbase deploy end of 634faea2
  • 17:08 mobrovac: restbase deploying 634faea2
  • 16:56 reedy@tin: Synchronized php-1.29.0-wmf.10/extensions/ConfirmEdit/maintenance/GenerateFancyCaptchas.php: Fix inclusion path (duration: 00m 41s)
  • 16:24 elukey: reboot mc2019->mc2025 to see if they come up cleanly (currently codfw replicas of eqiad redis shards)
  • 16:13 elukey: rebooting mc202[6789] (not serving any traffic) to see if they come up cleanly
  • 16:00 elukey: rebooting mc203[01234] (not serving any traffic) to see if they come up cleanly
  • 15:43 moritzm: upgrading firejail on remaining app servers
  • 15:35 moritzm: upgrading firejail on wtp1001
  • 15:15 moritzm: upgrading firejail on eqiad imagescalers
  • 15:11 elukey: rebooting mc203[56] (not taking any traffic) to test if they come up cleanly
  • 15:01 elukey: Replace Redis/Memcached shards mc200[4567] with mc202[2345]
  • 14:51 godog: manually fail sdc on graphite1001 - T157022
  • 14:23 addshore: EU SWAT really finished
  • 14:23 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT Create Wikiprojekti namespace on Finnish Wikipedia T156621 (duration: 00m 41s)
  • 14:14 addshore: EU SWAT finished
  • 14:14 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT Add Wikinews languages (en, pt, ca, fr, de, it) as import sources on eswikinews T156737 (duration: 00m 40s)
  • 14:10 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT Enable ElectronPdfService extension on dewiki T150942 (duration: 00m 41s)
  • 12:53 gehel: starting reimage to jessie of elasticsearch relforge - T151326
  • 11:59 joal@tin: Finished deploy [analytics/refinery@bc4b4ed]: (no justification provided) (duration: 01m 14s)
  • 11:58 joal@tin: Started deploy [analytics/refinery@bc4b4ed]: (no justification provided)
  • 11:40 elukey: Swap mc2002 with mc2020, mc2003 with mc2021 (Redis codfw replicas) - T155755
  • 11:35 joal@tin: Finished deploy [analytics/refinery@bc4b4ed]: (no justification provided) (duration: 03m 03s)
  • 11:32 joal@tin: Started deploy [analytics/refinery@bc4b4ed]: (no justification provided)
  • 10:53 elukey: Swap mc2001 with mc2019 (Redis codfw replicas) - T155755
  • 10:34 moritzm: restarted hadoop-mapreduce-historyserver on analytics1001
  • 10:23 moritzm: rolling restart of nginx tls terminators running on mw* application servers in eqiad to pick up openssl 1.1 update
  • 09:54 moritzm: rolling restart of logstash cluster to pick up openjdk/NSS security updates
  • 09:19 jynus: deploying schema change to page_assessments_projects on enwikivoyage T156305
  • 09:18 moritzm: uograding remaining canary servers to new HHVM packages
  • 09:17 marostegui: Remove dbstore1001:/srv/tmp/db1063.tar.gz after it has been transferred to db1095:/srv/tmp/db1063.tar.gz to get more disk space
  • 08:57 jynus: deploying schema change to page_assessments_projects on enwiki T156305
  • 08:42 filippo@tin: Finished deploy [prometheus/jmx_exporter@23a8f0b]: jmx_exporter deploy (duration: 00m 04s)
  • 08:42 filippo@tin: Started deploy [prometheus/jmx_exporter@23a8f0b]: jmx_exporter deploy
  • 08:30 moritzm: installing ntfs-3g security update on labnodepool (other servers had it deinstalled)
  • 08:26 filippo@tin: Finished deploy [prometheus/jmx_exporter@23a8f0b]: (no justification provided) (duration: 00m 07s)
  • 08:26 filippo@tin: Started deploy [prometheus/jmx_exporter@23a8f0b]: (no justification provided)
  • 08:24 marostegui: Transfer /srv/tmp/db1063.tar.gz from dbstore1001 to db1095:/srv/tmp to gain disk space
  • 08:24 marostegui: Remove /srv/tmp/db1067.tar.gz from dbstore1001 to gain disk space
  • 08:23 filippo@tin: Finished deploy [prometheus/jmx_exporter@23a8f0b]: (no justification provided) (duration: 00m 12s)
  • 08:23 filippo@tin: Started deploy [prometheus/jmx_exporter@23a8f0b]: (no justification provided)
  • 08:10 legoktm@tin: Synchronized php-1.29.0-wmf.10/includes/specials/pagers/ActiveUsersPager.php: Make last remaining user_groups queries honor $wgDisableUserGroupExpiry https://gerrit.wikimedia.org/r/#/c/335587/ (T156995) (duration: 00m 51s)
  • 08:08 legoktm@tin: Synchronized php-1.29.0-wmf.10/includes/api/ApiQueryAllUsers.php: Make last remaining user_groups queries honor $wgDisableUserGroupExpiry https://gerrit.wikimedia.org/r/#/c/335587/ (T156995) (duration: 00m 58s)
  • 07:41 marostegui: Restart MySQL on db2012 to tune some innodb_ft flags - T156905
  • 03:24 eileen1: renable all other jenkins jobs - only some dedupe & one-off jobs disabled
  • 03:17 eileen1: re-enable dedupe jobs
  • 03:14 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Feb 2 03:14:06 UTC 2017 (duration 5m 44s)
  • 03:08 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.10) (duration: 13m 49s)
  • 02:36 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.9) (duration: 14m 39s)
  • 02:32 eileen1: update civicrm from d06db92 to e45da6d
  • 02:23 eileen1: drush dis -y module_missing_message_fixer
  • 02:22 eileen1: drush mmmff --all
  • 02:22 eileen1: run drush en -y module_missing_message_fixer
  • 02:20 eileen1: Update civicrm from 7a86121 to d06db92
  • 02:02 mutante: carbon - remove unmapped IPv6 address making ferm rules fail, use only the _mapped_ IP (ip addr del 2620:0:861:1:7a2b:cbff:fe09:ea0/64 dev eth0) (T84380 T132757)
  • 01:50 eileen1: update CiviCRM from e17622b to 7a86121
  • 01:25 eileen1: jenkins disable Dedupe CiviCRM contacts & Dedupe major gifts 500_
  • 01:04 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/335578/ (duration: 00m 42s)
  • 00:30 maxsem@tin: Synchronized wmf-config/CirrusSearch-common.php: https://gerrit.wikimedia.org/r/#/c/335265/2 (duration: 00m 40s)
  • 00:21 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/335561/ (duration: 01m 00s)

2017-02-01

  • 22:54 mutante: carbon - rsyncing /srv/ data to install1002 (T132757)
  • 22:46 dereckson@tin: Synchronized wmf-config/: Folder sync to get around caching issue in previous deployments (T156942) (duration: 00m 45s)
  • 22:08 jynus: deploying schema change to page_assessments_projects on testwiki T156305
  • 21:49 bsitzmann@tin: Finished deploy [mobileapps/deploy@09101f7]: Update mobileapps to e48a88c (duration: 03m 04s)
  • 21:46 bsitzmann@tin: Started deploy [mobileapps/deploy@09101f7]: Update mobileapps to e48a88c
  • 21:39 twentyafterfour@tin: Synchronized wmf-config/InitialiseSettings.php: sync InitializeSettings to activate change from previous patches refs T156942 (duration: 00m 41s)
  • 21:21 reedy@tin: Synchronized php-1.29.0-wmf.10/includes/api/: Guard more ug_expiry queries (duration: 00m 48s)
  • 20:36 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.29.0-wmf.10
  • 20:29 twentyafterfour@tin: Synchronized wmf-config/abusefilter.php: deploy I0b4e02 refs T156942 (duration: 00m 39s)
  • 20:15 twentyafterfour@tin: Synchronized wmf-config/flaggedrevs.php: deploy I1683b1 refs T156942 (duration: 00m 40s)
  • 20:10 jynus: continuing mariadb rolling restart of db2044, db2051, db2058, db2065
  • 20:01 ejegg: updated payments-wiki from dd8a16d to 4466b9d
  • 19:59 twentyafterfour: scheduled downtime in icinga for phab2001's phd service
  • 19:59 twentyafterfour: Freshening phabricator's elasticsearch index, currently 50% complete
  • 19:27 twentyafterfour: disabled read-only in phabricator
  • 19:25 twentyafterfour: running puppet on iridium to activate the config change
  • 19:20 jynus: reloading haproxy on dbproxy1003
  • 19:11 jynus: remaining 7 minute with phabricator up, but read-only
  • 19:10 ostriches: phabricator: now in read-only mode
  • 19:08 jynus: scheduling 10 minutes of emergency downtime on phabricator
  • 19:06 mobrovac: restbase deploy end of 96a641aa
  • 18:49 joal@tin: Finished deploy [analytics/refinery@2b9a70a]: (no justification provided) (duration: 02m 33s)
  • 18:46 joal@tin: Started deploy [analytics/refinery@2b9a70a]: (no justification provided)
  • 18:34 mobrovac: restbase deploy start of 96a641aa
  • 16:54 marostegui: Optimize table phabricator_search.search_documentfield on db2012 - T156905
  • 16:41 jynus: mariadb rolling restart of db2037, db2044, db2051, db2058, db2065
  • 16:20 elukey: restarting Yarn Node Manager daemons on all the Hadoop nodes to bandaid a memory leak causing OOMs
  • 16:18 marostegui: Optimizing table search_documentfield on db1048 - T156905
  • 15:50 akosiaris: stop ircecho for a while to weather out most of the puppet alert storm
  • 15:46 akosiaris: restart puppetdb on nihal (openjdk upgrade)
  • 15:43 akosiaris: restart puppetdb on nitrogen
  • 15:40 jynus: preparing db1067 for reimage to jessie
  • 15:37 moritzm: upgrading canary app servers to new HHVM package (initially mwdebug and mw1261)
  • 15:17 Dereckson: `mwscript populateCategory.php plwikisource --force` to refresh categories stats (T156670)
  • 15:17 dereckson@tin: Finished scap: Full scap to propagate a core namespace l10n change (duration: 40m 10s)
  • 14:41 godog: upgrade thumbor to 0.1.34
  • 14:37 dereckson@tin: Started scap: Full scap to propagate a core namespace l10n change
  • 14:25 jynus: dropping and replacing events on db1057 - db1052 T156008
  • 14:24 dereckson@tin: Synchronized php-1.29.0-wmf.9/languages/messages/MessagesJv.php: Update namespace localisation in Javanese (T155957) (duration: 00m 40s)
  • 14:21 dereckson@tin: Synchronized php-1.29.0-wmf.10/languages/messages/MessagesJv.php: Update namespace localisation in Javanese (T155957) (duration: 00m 45s)
  • 14:12 moritzm: uploaded hhvm 3.12.12 to carbon
  • 14:10 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Enable ElectronPdfService on meta (T150943) (duration: 00m 48s)
  • 13:39 marostegui: Deploy alter table dbstore1002 metawiki.pagelinks - T153300
  • 13:38 akosiaris: issue sudo hdparm -Y /dev/sdb on bast3001 to force a problematic drive to sleep
  • 13:21 marostegui: Clean up db1043 replication thread (it was replicating from db1048 which looks like an old thing) - T156905
  • 12:09 elukey@tin: Finished deploy [analytics/refinery@e6254a4]: (no justification provided) (duration: 04m 41s)
  • 12:04 elukey@tin: Started deploy [analytics/refinery@e6254a4]: (no justification provided)
  • 11:52 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2061 - T153300 (duration: 00m 40s)
  • 11:25 moritzm: removing ntfs-3g from various trusty servers
  • 11:14 godog: bounce leaking thumbor@8813 on thumbor1001
  • 11:08 kartik@tin: Finished deploy [cxserver/deploy@0e4ae4f]: (no justification provided) (duration: 02m 04s)
  • 11:06 kartik@tin: Started deploy [cxserver/deploy@0e4ae4f]: (no justification provided)
  • 07:53 marostegui: Deploy alter table metawiki.pagelinks db2061 - T153300
  • 07:52 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2061 - T153300 (duration: 00m 53s)
  • 07:43 moritzm: rolling restart of cassandra in eqiad to pick up openjdk and NSS security updates
  • 07:41 elukey: bootstrapping aqs1008-a on aqs1008 (new AQS cassandra node)
  • 07:31 marostegui: Force WB policy on the raid controller db1072 - T156226
  • 07:13 akosiaris: restart thumbor process on thumbor1001, thumbor1002, apply a different LimitNOFILE on thumbo1002
  • 04:17 mutante: carbon - rsyncing entire /srv over to install2002 (T156440)
  • 03:00 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Feb 1 03:00:32 UTC 2017 (duration 5m 35s)
  • 02:54 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.10) (duration: 03m 42s)
  • 02:48 mutante: install1002, install2002 - install jessie, sign puppet certs, initial puppet run (T132757, T156440)
  • 02:34 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.9) (duration: 11m 52s)
  • 02:20 demon@tin: Synchronized scap/plugins/clean.py: no-op (duration: 00m 39s)
  • 01:19 mutante: ganeti: create instance install2002 with 80G disk, 2G RAM (T156440)
  • 01:15 mutante: ganeti: install1001 - remove virtual disk 1 from instance | create instance install1002 instead (T132757)
  • 00:57 mutante: Ganglia is now deprecated in favor of Grafana (https://phabricator.wikimedia.org/T145659#2925104)
  • 00:33 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/334025/ (duration: 00m 40s)
  • 00:32 maxsem@tin: Synchronized php-1.29.0-wmf.9/extensions/CentralNotice/: https://gerrit.wikimedia.org/r/#/c/335263/ (duration: 00m 58s)

2017-01-31

  • 23:52 ppchelko@tin: Finished deploy [changeprop/deploy@e27c3a0]: Update change-prop to fix wikidata rollback rule (duration: 01m 32s)
  • 23:51 ppchelko@tin: Started deploy [changeprop/deploy@e27c3a0]: Update change-prop to fix wikidata rollback rule
  • 22:27 twentyafterfour: cleaned up old branches: wmf.3 and wmf.4
  • 21:58 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 wikis to 1.29.0-wmf.10
  • 21:50 twentyafterfour@tin: Synchronized wmf-config/: sync ExtensionMessages-1.29.0-wmf.10.php (duration: 00m 47s)
  • 21:41 twentyafterfour@tin: scap sync-l10n completed (1.29.0-wmf.10) (duration: 19m 49s)
  • 21:11 twentyafterfour: wmf-config/ExtensionMessages-1.29.0-wmf.10.php is missing refs T155525
  • 21:05 twentyafterfour@tin: Finished scap: (no justification provided) (duration: 27m 16s)
  • 20:37 twentyafterfour@tin: Started scap: (no justification provided)
  • 20:37 twentyafterfour: syncing 1.29.0-wmf.10 to test wikis
  • 20:32 jynus: stopping db1063 mariadb before full host reimage
  • 18:58 arlolra: Updated Parsoid to version 734dc996 (T98960)
  • 18:51 arlolra@tin: Finished deploy [parsoid/deploy@dc2323d]: Updating Parsoid to 734dc996 (duration: 12m 58s)
  • 18:46 ottomata: recentchange events now flowing into Kafka via EventBus T152030
  • 18:45 otto@tin: Synchronized wmf-config/CommonSettings.php: Enabling RCFeed -> EventBus (duration: 00m 42s)
  • 18:44 otto@tin: Synchronized wmf-config/CommonSettings-labs.php: Enabling RCFeed -> EventBus (duration: 00m 43s)
  • 18:38 arlolra@tin: Started deploy [parsoid/deploy@dc2323d]: Updating Parsoid to 734dc996
  • 18:06 jynus: end up tendril and dbtree maintenance, things should be back up, report if you see degradations of service
  • 17:37 jynus: stopping mysql, upgrading and restarting db1011- temporary outage of tendril & dbtree T111654
  • 16:59 robh: disabled puppet on einsteinium while i try to figure out what i broke in my config for icinga
  • 16:11 elukey: started Cassandra nodetool cleanup for aqs1007-a
  • 16:03 elukey: started Cassandra nodetool cleanup for aqs1004-b
  • 14:57 jynus: upgrading and restarting db1095 (sanitarium2)
  • 14:12 elukey: restarting hhvm on mw1204 (dump debug in /tmp/hhvm.29120.bt)
  • 14:07 aude@tin: Synchronized wmf-config/Wikibase.php: Update property suggester config (duration: 00m 42s)
  • 13:59 elukey: rebooted analytics1039 to pick up uuids in fstab - T147879
  • 12:26 addshore: TwoColConflict deploy slot done!
  • 12:26 addshore@tin: Synchronized wmf-config/CommonSettings-labs.php: Enable TwoColConflict on test wikis (T155716) 5/5 (duration: 00m 40s)
  • 12:25 addshore@tin: Synchronized wmf-config/CommonSettings.php: Enable TwoColConflict on test wikis (T155716) 4/5 (duration: 00m 40s)
  • 12:24 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Enable TwoColConflict on test wikis (T155716) 3/5 (duration: 00m 42s)
  • 12:23 addshore@tin: Synchronized wmf-config/extension-list-labs: Enable TwoColConflict on test wikis (T155716) 2/5 (duration: 00m 40s)
  • 12:22 addshore@tin: Synchronized wmf-config/extension-list: Enable TwoColConflict on test wikis (T155716) 1/5 (duration: 00m 40s)
  • 12:16 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Add twocolconflict to wgBetaFeaturesWhitelist (T150184) (duration: 00m 41s)
  • 11:14 elukey: updating the puppet compiler's facts
  • 10:42 gehel: starting reimage of wdqs1003
  • 09:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool hosts in C2 - T155999 (duration: 00m 40s)
  • 09:38 moritzm: rolling restart of cassandra in codfw to pick up openjdk and NSS security updates
  • 09:00 gehel: aligning elasticsearch low watermark to 75% disk space on all clusters (eqiad was at 70%)
  • 08:44 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=aqs1007.eqiad.wmnet
  • 08:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add a warning about a possible bad BBU on db1072 - T156226 (duration: 00m 46s)
  • 08:26 elukey: started Cassandra nodetool cleanup for aqs1004-a
  • 07:54 moritzm: installing chromium security update on osmium
  • 07:49 marostegui: Reboot db1072 to force BBU recharge - T156226
  • 07:10 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2034 - T156478 (duration: 00m 57s)
  • 03:49 andrewbogott: restarted nova-api on labnet1001 which actually fixed some things
  • 03:28 chasemp: (slightly belated) set logging level on serpens higher to see if ldap binding is an issue
  • 02:45 bd808: Setup temporary cron on silver as user bd808 until T156733 is fixed properly
  • 02:33 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Jan 31 02:33:22 UTC 2017 (duration 5m 23s)
  • 02:31 bd808: Manually ran extensions/TorBlock/loadExitNodes.php on silver
  • 02:28 bd808@tin: Synchronized wmf-config/InitialiseSettings.php: Enable TorBlock for Wikitech (duration: 00m 41s)
  • 02:28 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.9) (duration: 07m 09s)
  • 02:16 chasemp: restart uwsgi-keystone-admin and uwsgi-keystone-public on labcontrol1001
  • 00:25 ebernhardson: restarting elasticsearch on elastic1029, got stuck in RemoteTransportException loop again
  • 00:23 mobrovac@tin: Started restart [electron-render/deploy@f1df2d3]: Service restart for firejail upgrade
  • 00:22 mobrovac@tin: Started restart [mobileapps/deploy@7615bf9]: Service restart for firejail upgrade
  • 00:21 mobrovac@tin: Started restart [mathoid/deploy@ba3217e]: Service restart for firejail upgrade
  • 00:20 mobrovac@tin: Started restart [graphoid/deploy@da37386]: Service restart for firejail upgrade
  • 00:17 mobrovac@tin: Started restart [cxserver/deploy@5ae4f8b]: Service restart for firejail upgrade
  • 00:13 mobrovac@tin: Started restart [citoid/deploy@95df861]: Service restart for firejail upgrade
  • 00:10 mobrovac@tin: Started restart [changeprop/deploy@2b980fa]: Service restart for firejail upgrade

2017-01-30

  • 21:49 eileen1: killed long running user-initiated dedupe query
  • 21:24 eileen1: updated civicrm from 6b6f5d6 to e17622b
  • 21:13 mobrovac@tin: Finished deploy [trending-edits/deploy@5735f00]: (no justification provided) (duration: 03m 13s)
  • 21:10 mobrovac@tin: Started deploy [trending-edits/deploy@5735f00]: (no justification provided)
  • 21:09 mobrovac@tin: Finished deploy [trending-edits/deploy@5735f00]: (no justification provided) (duration: 03m 07s)
  • 21:06 mobrovac@tin: Started deploy [trending-edits/deploy@5735f00]: (no justification provided)
  • 20:57 mobrovac@tin: Finished deploy [trending-edits/deploy@5735f00]: Bump memory limit and heartbeat timeout (duration: 01m 48s)
  • 20:55 mobrovac@tin: Started deploy [trending-edits/deploy@5735f00]: Bump memory limit and heartbeat timeout
  • 20:50 godog: uploaded scap 3.5.1-1
  • 20:50 thcipriani@tin: Synchronized README: test scap (duration: 00m 43s)
  • 20:47 eileen1: updated localsettings to a346207
  • 20:45 mobrovac@tin: Finished deploy [trending-edits/deploy@9addcd0]: Bump max_age to 18h for T156411 (duration: 02m 39s)
  • 20:43 mobrovac@tin: Started deploy [trending-edits/deploy@9addcd0]: Bump max_age to 18h for T156411
  • 20:25 eileen1: disable drupal update module on prod. T155084, this should still be on on dev sites so not using update script
  • 20:12 Pchelolo: update RESTBase to 501ea47edc in staging
  • 20:09 ejegg: updated payments-wiki config to d98b30b
  • 19:47 gehel@tin: Finished deploy [wdqs/wdqs@81442a0]: (no justification provided) (duration: 01m 23s)
  • 19:46 gehel@tin: Started deploy [wdqs/wdqs@81442a0]: (no justification provided)
  • 19:44 gehel: deploying latest wdqs gui
  • 18:56 thcipriani: unlocking mediawiki deployments for test
  • 18:53 nuria@tin: Finished deploy [eventlogging/analytics@4b28b14]: (no justification provided) (duration: 00m 11s)
  • 18:53 nuria@tin: Started deploy [eventlogging/analytics@4b28b14]: (no justification provided)
  • 18:50 nuria: rollback deployment to eventlogging
  • 18:48 thcipriani: mediawiki deployments momentarily
  • 18:46 nuria@tin: Finished deploy [eventlogging/analytics@4b28b14]: (no justification provided) (duration: 00m 04s)
  • 18:46 nuria@tin: Started deploy [eventlogging/analytics@4b28b14]: (no justification provided)
  • 18:42 Pchelolo: update RESTBase to cd2b5e019
  • 18:38 Pchelolo: update RESTBase to cd2b5e019: canary on restbase2001
  • 18:36 godog: upload scap 3.5.0-1 - T127762
  • 18:18 gehel: nginx upgrade and wdqs restart complete - sorry for the noise
  • 18:13 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs1002.codfw.wmnet
  • 18:12 Niharika: updated scholarships Fixed some bugs with the login form
  • 18:11 moritzm: upgrading firejail on scb cluster
  • 18:11 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs1001.codfw.wmnet
  • 18:07 Pchelolo: update RESTBase to cd2b5e019: canary on restbase1007
  • 18:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Change db2034 IP - T156478 (duration: 00m 40s)
  • 18:04 marostegui@tin: Synchronized wmf-config/db-codfw.php: Change db2034 IP - T156478 (duration: 00m 40s)
  • 18:04 gehel: rolling restart of nginx and wdqs for updates
  • 17:41 Pchelolo: update RESTBase to cd2b5e019: staging
  • 17:09 legoktm@tin: Finished scap: Build l10n cache for linter (duration: 22m 43s)
  • 17:05 marostegui: Shutdown mysql and poweroff db2034 for maintenance - T156478
  • 16:46 legoktm@tin: Started scap: Build l10n cache for linter
  • 16:41 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2034 for maintenance - T156478 (duration: 00m 40s)
  • 16:34 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs2003.codfw.wmnet
  • 15:42 hashar@tin: Synchronized php-1.29.0-wmf.9/extensions/timeline/Timeline.body.php: debug log EasyTimeline error - T138036 (duration: 00m 46s)
  • 15:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1073 with its original weight - T156226 (duration: 00m 52s)
  • 14:45 hashar@tin: Synchronized php-1.29.0-wmf.9/languages/Language.php: translateBlockExpiry: Duration is block expiry minus current time - T156453 (duration: 00m 42s)
  • 14:29 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable RSS extension at metawiki, enable one feed - T155830 (duration: 00m 42s)
  • 14:15 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Create namespace alias وگ for NS_PROJECT in fawikiquote - T156451 (duration: 00m 40s)
  • 14:10 hashar@tin: Synchronized wmf-config/flaggedrevs.php: Remove flaggedrevs-protect-review page protection from enwiki - T156448 (duration: 00m 41s)
  • 14:08 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable SandboxLink on tgwiki - T156473 (duration: 00m 40s)
  • 14:06 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable SandboxLink on gdwiki - T156281 (duration: 00m 48s)
  • 11:23 gehel: upgrade and restart nginx on elasticsearch eqiad cluster
  • 10:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1073 with less weight - T156226 (duration: 00m 49s)
  • 10:00 gehel: upgrade and restart nginx on elasticsearch codfw cluster
  • 09:58 gehel: upgrade and restart nginx on relforge cluster
  • 09:44 godog: upgrade to thumbor 0.1.33 - T151066
  • 09:37 ariel@tin: Finished deploy [dumps/dumps@4a9e952]: proper md5sum format for adds/changes dumps (duration: 00m 02s)
  • 09:37 ariel@tin: Starting deploy [dumps/dumps@4a9e952]: proper md5sum format for adds/changes dumps
  • 09:25 elukey: bootstrapping new cassandra instance (aqs1007-b) on AQS - https://gerrit.wikimedia.org/r/#/c/334753/
  • 09:19 moritzm: installing tcpdump security updates
  • 09:06 marostegui: Upgrade db2012 to 10.0.29-2 (this was done couple of hours ago, but for the record) - T156373
  • 09:05 marostegui: Start slaves from s1 to s7 on dbstore2001 - T156373
  • 08:54 moritzm: installing NSS security updates on kafka and Hadoop clusters
  • 08:45 elukey: restarting aqs on aqs100[4567] to pick up NSS updates
  • 08:19 elukey: set mw1236.eqiad.wmnet pooled=inactive because powered off (no mentions on the SAL, still trying to find why)
  • 08:05 moritzm: switched application servers in codfw to systemd-timesyncd
  • 08:04 marostegui: Stop mysql db1073 to use it to clone db1072 - T156226
  • 08:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1073 - T156226 (duration: 02m 45s)
  • 02:21 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Jan 30 02:21:21 UTC 2017 (duration 4m 22s)
  • 02:16 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.9) (duration: 06m 11s)

2017-01-29

  • 02:21 l10nupdate@tin: ResourceLoader cache refresh completed at Sun Jan 29 02:21:37 UTC 2017 (duration 4m 23s)
  • 02:17 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.9) (duration: 06m 06s)

2017-01-28

  • 02:22 l10nupdate@tin: ResourceLoader cache refresh completed at Sat Jan 28 02:22:49 UTC 2017 (duration 4m 47s)
  • 02:18 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.9) (duration: 05m 39s)

2017-01-27

  • 22:45 mutante: install1001 - adding a second virtual hard disk, 80G
  • 22:31 mutante: carbon: rsync entire /srv/ to install2001 (this is APT data but also misc things like junos, megacli, firmware, ipmi
  • 21:41 volans: restored watchmouse checks for s5 (de wiki), Main_Page redirect was restored
  • 21:36 mobrovac@tin: Finished deploy [trending-edits/deploy@0e79bec]: Bump max_age to 12h T156411 (duration: 01m 58s)
  • 21:34 mobrovac@tin: Starting deploy [trending-edits/deploy@0e79bec]: Bump max_age to 12h T156411
  • 21:28 mobrovac@tin: Finished deploy [trending-edits/deploy@e0e32bb]: Restart the service to assess the load of replaying the last 6h T156411 (duration: 01m 03s)
  • 21:27 mobrovac@tin: Starting deploy [trending-edits/deploy@e0e32bb]: Restart the service to assess the load of replaying the last 6h T156411
  • 20:24 jynus: restart and upgrade mariadb on db1048
  • 20:14 mutante: db1019, db1042, analytics1015, analytics1026 - puppet node deactivate, remove from icinga, finish decom (T147313, T149793, T146265)
  • 20:07 volans: updated watchmouse checks for s5 (de wiki) because Main_Page was deleted, used the localized page instead
  • 19:52 mutante: db1019 - shutdown -h now (T146265)
  • 19:51 mutante: db1042 - i came to shut it down .. and noticed it had died (or somebody did it) about 3 hours ago .. there it goes (T149793)
  • 18:52 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 wikis to 1.29.0-wmf.9
  • 18:52 twentyafterfour: Rolling forward with group2 to 1.29.0-wmf.9 refs T156364 T154683
  • 17:55 papaul: OS installation on mc2019-mc2036
  • 16:08 marostegui@tin: Synchronized wmf-config/db-eqiad.php: db1072 change IP - T156226 (duration: 00m 40s)
  • 16:07 marostegui@tin: Synchronized wmf-config/db-codfw.php: db1072 change IP - T156226 (duration: 00m 40s)
  • 16:01 jynus: submitted wmf-mariadb10_10.0.29-2 for T156373 fix
  • 15:48 marostegui: Stop mysql and shutdown db1072 for maintenance - T156226
  • 15:45 ema: cache_text: ban req.url == "/apple-app-site-association" && obj.status == 404 (T155504)
  • 14:13 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add rack positions for s7 in codfw (duration: 00m 40s)
  • 14:12 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add rack positions for s7 in eqiad (duration: 00m 40s)
  • 13:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add rack positions for s6 in eqiad (duration: 00m 40s)
  • 13:44 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add rack positions for s6 in codfw (duration: 00m 40s)
  • 13:31 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add rack positions for s5 in eqiad (duration: 00m 40s)
  • 13:30 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add rack positions for s5 in codfw (duration: 00m 40s)
  • 13:14 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add rack positions for s4 in codfw (duration: 00m 41s)
  • 13:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add rack positions for s4 in eqiad (duration: 00m 43s)
  • 13:11 jynus: starting db1048 until db1043-bin.001457:753455353, expect it to stop soon
  • 13:00 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add rack positions for s3 in eqiad (duration: 00m 40s)
  • 13:00 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add rack positions for s3 in codfw (duration: 00m 40s)
  • 12:24 moritzm: upgrading mediawiki canaries to new openssl 1.1 package
  • 12:13 moritzm: upgrading openjdk-7 packages (security updates) on wdqs cluster
  • 11:56 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add rack positions for s2 in codfw (duration: 00m 47s)
  • 11:55 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add rack positions for s2 in eqiad (duration: 00m 59s)
  • 11:11 moritzm: initial installation of openssl bugfix/security updates
  • 10:54 moritzm: uploaded openssl 1.0.2k for jessie-wikimedia to carbon
  • 10:35 paravoid: manually running certspotter -all_time as my user on einstenium (will take a few days to complete)
  • 10:21 legoktm: added addshore to labs-tools-wikibugs2 gerrit group
  • 08:13 moritzm: uploaded openssl 1.1.0d packages for jessie-wikimedia to carbon
  • 02:58 l10nupdate@tin: ResourceLoader cache refresh completed at Fri Jan 27 02:58:52 UTC 2017 (duration 5m 46s)
  • 02:53 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.9) (duration: 14m 14s)
  • 02:21 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.8) (duration: 07m 57s)
  • 01:20 twentyafterfour: deploying hotfix for phabricator refs T154479
  • 00:48 mutante: carbon - moved the 1.5TB /srv/"mirrors.off", which used to be mirrors but is now on sodium, into / to that /srv/ can be synced without this
  • 00:32 dereckson@tin: Synchronized wmf-config/throttle.php: Fix throttle rule for Her Girl Friday + Lenny Unconference (T156278) (duration: 00m 53s)
  • 00:12 volans: re-enabled puppet (with a temporary fix to keep parsoid-vd and parsoid-vd-client stopped) on ruthenium T156177

2017-01-26

  • 23:49 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 wikis to 1.29.0-wmf.8
  • 23:37 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.29.0-wmf.9
  • 23:33 robh: archiva.w.o maint done, uses new LE cert.
  • 23:14 robh: going to try to convert archiva.wikimedia.org from GS to LE cert. will require rehup of nginx
  • 22:58 mutante: db1019, db1042 - revoke puppet certs, delete salt keys, schedule icinga downtime, stop services (T149793, T146265)
  • 22:56 mutante: analytics1015, analytics1026 - puppet node clean (again?) - again having problems to remove decom'ed nodes from Icinga (T147313)
  • 22:53 mutante: analytics1015, analytics1026 - puppet node clean (again?) - again having problems to remove decom'ed nodes from Icinga
  • 22:46 mutante: mw1181, mw1272, mw1212, mw1174 - service hhvm restart
  • 22:06 mobrovac@tin: Finished deploy [trending-edits/deploy@e0e32bb]: Bump replay time to 6h for T156411 (duration: 01m 42s)
  • 22:04 mobrovac@tin: Starting deploy [trending-edits/deploy@e0e32bb]: Bump replay time to 6h for T156411
  • 20:30 andrewbogott: refreshing logins on wikitech
  • 19:13 elukey: restore analytics1001 as RM and HDFS masters
  • 18:56 otto@tin: Finished deploy [eventstreams/deploy@f1a1866]: (no message) (duration: 03m 16s)
  • 18:52 otto@tin: Starting deploy [eventstreams/deploy@f1a1866]: (no message)
  • 18:36 elukey: restarting Yarn node managers on an102[89] and an103[01], impacted by the switch restart
  • 18:32 paravoid: starting pybal on lvs1001/lvs1002/lvs1003
  • 18:15 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1055, 56, 57, 59 (duration: 00m 54s)
  • 18:14 paravoid: rebooting newly provisioned asw-c2-eqiad to enable mixed mode
  • 17:57 elukey: boostrapping aqs1007-a cassandra instance
  • 17:51 paravoid: replacing asw-c2-eqiad
  • 17:46 paravoid: stopping pybal on lvs1001/lvs1002/lvs1003
  • 17:34 elukey@tin: Finished deploy [analytics/aqs/deploy@5917fd4]: (no message) (duration: 02m 25s)
  • 17:31 elukey@tin: Starting deploy [analytics/aqs/deploy@5917fd4]: (no message)
  • 15:43 bblack: cache_misc puppet re-enabled and up to date
  • 15:37 godog: bounce uwsgi on graphite1003 with less workers - T155872
  • 15:35 moritzm: installing gnupg2 updates from jessie point update
  • 15:15 akosiaris: T156242 add /dev/sdb partitions to mdadm devices
  • 15:15 bblack: puppet disabled on cache_misc for merging complicated stuff
  • 15:06 moritzm: installing gnupg updates from jessie point update
  • 14:52 jynus: stopping mysql on db1048 T156373
  • 14:29 zeljkof: finished with eu swat
  • 14:28 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: IP Cap Lift for Edit-a-Thon (T156258) [throttle] Her Girl Friday + Lenny Unconference / Editathon in NYC, 2017-01-28 (T156278) (duration: 00m 41s)
  • 13:54 godog: delete labs 'instances' graphite three for data >30d, graphite low on disk space
  • 13:53 elukey: restarting cassandra on aqs100[56] to complete the openjdk update
  • 13:32 moritzm: rolling restart of maps cluster in eqiad
  • 13:08 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1054 - T156225 (duration: 00m 40s)
  • 13:05 hashar@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 back to 1.29.0-wmf.9 T156310
  • 13:02 godog: reboot ms-be1013
  • 12:54 elukey: restarting the aqs1004-b casandra instance to pick up the new openjdk (last test before complete rollout)
  • 12:28 elukey: restarting the aqs1004-a casandra instance to pick up the new openjdk
  • 12:28 moritzm: upgrading java on maps cluster, rolling restart of maps cluster in codfw
  • 12:17 hashar@tin: Synchronized php-1.29.0-wmf.9/extensions/FlaggedRevs/backend/FlaggedRevision.php: Fix fatal in prod caused by deprecated function removal T156310 (duration: 00m 41s)
  • 12:04 moritzm: installing java security updates on aqs cluster
  • 11:12 hashar@tin: rebuilt wikiversions.php and synchronized wikiversions files: FlaggedRevs is broken in wmf.9 causing blank pages. T156356 T156310
  • 09:55 marostegui: Disable semi-sync on db1057 old s1 master - https://phabricator.wikimedia.org/T156008
  • 09:39 marostegui: Enable semi-sync replication on db1052 (s1 master) - T156008
  • 09:04 marostegui: Change dbstore1002 to replicate from the new s1 master db1052 - T156008
  • 08:57 marostegui: Change db1047 to replicate from the new s1 master db1052 - T156008
  • 08:48 marostegui: Change db1069 to replicate from the new s1 master db1052 - T156008
  • 08:48 jynus: deploying dns CNAME updates due to master swithover
  • 07:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Change s1 master to db1057 - T156008 (duration: 00m 20s)
  • 07:18 jynus: last message was master of db1057
  • 07:18 jynus: change master of db1052 from db2016 to db1052
  • 06:49 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1052 - T156008 (duration: 00m 31s)
  • 03:50 mutante: rsyncing apt.wikimedia.org data from carbon to install2001 (T84380)
  • 02:07 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Jan 26 02:07:05 UTC 2017 (duration 4m 42s)
  • 02:02 l10nupdate@tin: LocalisationUpdate failed (1.29.0-wmf.9) at 2017-01-26 02:02:23+00:00
  • 02:02 l10nupdate@tin: LocalisationUpdate failed (1.29.0-wmf.8) at 2017-01-26 02:02:23+00:00
  • 02:02 twentyafterfour: phabricator update complete
  • 01:47 twentyafterfour: upgrading phabricator, downtime should be minimal but expect the service to be offline for up to a few minutes
  • 01:11 mattflaschen@tin: Synchronized wmf-config/InitialiseSettings.php: No-op documentation change to InitialiseSettings.php (duration: 00m 46s)
  • 00:31 krenair@tin: Synchronized wmf-config: https://gerrit.wikimedia.org/r/298397 part 2 (duration: 00m 42s)
  • 00:29 krenair@tin: Synchronized wmf-config: https://gerrit.wikimedia.org/r/298397 (duration: 00m 43s)
  • 00:26 Krenair: sync 298397 to mwdebug1001
  • 00:11 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: Enable deprecation logging with gerrit:334206 (duration: 00m 53s)
  • 00:09 ebernhardson: sync 334206 to mwdebug1002

2017-01-25

  • 23:39 mutante: mwdebug1002 - service hhvm restart
  • 23:38 mutante: mw1185, mw1268 - service hhvm restart
  • 23:37 ejegg: updated civicrm from cd058a0 to 6b6f5d6
  • 23:21 ejegg: restarted civicrm donation queue consumer and dedupe jobs
  • 22:47 demon@tin: Synchronized w/extract2.php: (no message) (duration: 00m 40s)
  • 22:35 ejegg: paused civicrm dedupe and donation import jobs
  • 22:33 ejegg: updated civicrm from af8d735 to cd058a0
  • 21:39 akosiaris: reload pfw1-codfw node 0 in an effort to debug high RTTs
  • 20:50 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.29.0-wmf.9
  • 20:44 twentyafterfour: deploying mediawiki 1.29.0-wmf.9 to group1 wikis
  • 20:36 volans: upgrading nodejs-legacy (it is just the symlink) to v6 on parsoid hosts T149331
  • 20:11 eileen: renabled dedupe job, disabled major gifts (one should be enough). Will investigate next error
  • 19:26 mutante: analytics1015,analytics1026 - decom: remove DNS names, delete salt keys, revoke puppet certs, puppet node clean (to remove from icinga) (T147313)
  • 18:22 bd808@tin: Finished deploy [striker/deploy@5aa3aa8]: Update Striker to 5aa3aa8 (T144710, T147024, T144712, T144711, T153935) (duration: 00m 24s)
  • 18:22 bd808@tin: Starting deploy [striker/deploy@5aa3aa8]: Update Striker to 5aa3aa8 (T144710, T147024, T144712, T144711, T153935)
  • 18:09 demon@tin: Synchronized docroot/foundation/logos: rm a junk logo (duration: 00m 50s)
  • 18:02 elukey: running authdns-update on ns0.w.o to pick up changes made in https://gerrit.wikimedia.org/r/334040
  • 17:38 jynus: restarting and upgrading db2060
  • 16:57 ostriches: gerrit: everything back up!
  • 16:56 ostriches: gerrit: quick service reboot to pick up new java version
  • 16:31 Jeff_Green: renamed fdb2001 to frdb2001
  • 16:25 marostegui@tin: Synchronized wmf-config/db-codfw.php: Change db1054 IP - T156225 (duration: 00m 40s)
  • 16:24 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Change db1054 IP - T156225 (duration: 00m 41s)
  • 16:07 marostegui: Stop mysql and power off db1054 for maintenance - T156225
  • 15:28 gehel: removing jieba / ltr / swift plugins from elasticsearch relforge - T156150
  • 15:27 gehel: deleting indices using jieba plugin from relforge - T156150
  • 15:19 chasemp: (slightly late) of 'maintain-views --all-databases --table watchlist_count --replace-all' across labsdbs
  • 15:11 godog: graphite1003 / graphite2002 at 94% utilization, increase lv size by 300G
  • 14:42 moritzm: installing ruby2.1 updates from jessie point release
  • 14:23 moritzm: installing wget updates from jessie point release
  • 14:13 dcausse: EU SWAT done
  • 14:11 dcausse@tin: Synchronized wmf-config/InitialiseSettings.php: T156234 Revert [cirrus] properly set wgCirrusSearchUseIcuFolding (duration: 00m 41s)
  • 14:09 moritzm: removed totally outdated openjdk-8 packages from trusty-wikimedia (from 2014) on carbon
  • 13:55 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1054 - T156225 (duration: 00m 50s)
  • 13:52 moritzm: upgrading openjdk-8 on maps-test*
  • 13:11 gehel: pooling new elasticsearch nodes on codfw - T154251
  • 12:38 moritzm: installing libxml security updates
  • 12:04 Dereckson: Refresh site statistics on simple. (T156247)
  • 11:01 moritzm: upgrading restbase staging cluster to new openjdk (also piggyback reboot to latest 4.4 kernel)
  • 10:49 moritzm: uploaded openjdk-8 u121 to apt.wikimedia.org
  • 10:28 moritzm: uploaded ca-certificates-java 20161107~bpo8+1 to apt.wikimedia.org
  • 10:11 ema: repooled codfw
  • 09:25 elukey: updating puppet-compiler facts
  • 08:53 ema: upgrade cp3040 to jessie 8.7 and reboot into kernel 4.4.2-3+wmf8 T155401
  • 08:15 ema: upgrade cp3034 to jessie 8.7 and reboot into kernel 4.4.2-3+wmf8 T155401
  • 08:15 ema: upgrade cp3034 to jessie 8.7 and reboot into kernel 4.4.2-3+wmf8
  • 08:02 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2054 - T153300 (duration: 00m 51s)
  • 07:48 _joe_: restarting pybal on lvs1003
  • 07:28 elukey: upgrading aqs100[56] to node6
  • 06:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1066 - T156005 (duration: 00m 42s)
  • 05:37 mobrovac: zotero restarting zotero, taking 95% of mem ...
  • 02:55 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Jan 25 02:55:57 UTC 2017 (duration 5m 37s)
  • 02:50 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.9) (duration: 12m 53s)
  • 02:20 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.8) (duration: 06m 25s)
  • 00:59 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1052 after maintenance (duration: 00m 40s)
  • 00:37 mutante: planet2001 - re-add new salt key, fix minion
  • 00:11 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Amend import sources for en.wikisource (T155922) (duration: 00m 47s)
  • 00:09 mutante: analytics1015,analytics1026 - revoked puppet cert, removing from puppet, shutting down (T147313)

2017-01-24

  • 23:50 mutante: carbon - stopping puppet, stopping atftpd
  • 23:49 mutante: carbon stopping DHCP
  • 23:49 mutante: analytics1015 (unused spare system) - use for test OS install
  • 23:26 jynus: restarting db1052 for kernel upgrade
  • 23:09 ebernhardson@tin: Synchronized php-1.29.0-wmf.9/includes/specials/SpecialSearch.php: Update special:search security patc h to not fatal (duration: 00m 44s)
  • 23:04 jynus: reimage db1066
  • 22:41 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1066 for reimage (duration: 00m 55s)
  • 22:22 Pchelolo: update RESTBase to 69065e2
  • 22:19 Pchelolo: update RESTBase to 69065e2: canary on restbase1007
  • 22:13 Pchelolo: update RESTBase to 69065e2: staging
  • 21:55 jynus@tin: Synchronized wmf-config/db-eqiad.php: repool db1065 as dump/vslow & clean up s1 comments (duration: 00m 43s)
  • 21:43 twentyafterfour: Finished group0 to wmf/1.29.0-wmf.9 (refs T15525) Changelog: https://www.mediawiki.org/wiki/MediaWiki_1.29/wmf.9/Changelog
  • 21:34 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 wikis to 1.29.0-wmf.9 refs T155525
  • 21:31 demon@tin: Synchronized docroot: Drop labs docroot, unused in prod (duration: 00m 44s)
  • 21:22 twentyafterfour@tin: Finished scap: test wikis to 1.29.0-wmf.9 refs T155525 (duration: 32m 37s)
  • 20:49 twentyafterfour@tin: Started scap: test wikis to 1.29.0-wmf.9 refs T155525
  • 20:49 volans: disabled puppet on ruthenium to avoid the restart of parsoid-vd and parsoid-vd-client processes T156177
  • 20:09 demon@tin: Synchronized docroot: Adding new wikimediafoundation.org docroot (duration: 01m 05s)
  • 19:51 volans: ruthenium: stopped parsoid-vd and parsoid-vd-client to avoid uncontrolled spawning of phantomjs childs
  • 19:39 volans: sudo service parsoid-vd stop on ruthenium
  • 19:37 twentyafterfour: branching 1.29.0-wmf.9 refs T154683
  • 19:35 volans: killed 822 "/srv/visualdiff/node_modules/phantomjs/lib/phantom/bin/phantomjs" processes on ruthenium. RAM and swap full, host unresponsive
  • 19:29 jynus: change replication master of db1095 to db1065
  • 19:07 jynus: change replication master of db1095 to db1052
  • 19:07 demon@tin: Synchronized docroot/foundation/logos: rm some old junk logos (duration: 00m 42s)
  • 18:58 arlolra: Updated Parsoid to version d000fdb4 (T58846, T154804, T152633)
  • 18:45 arlolra@tin: Finished deploy [parsoid/deploy@c1a14c0]: Retry updating Parsoid to d000fdb4 (duration: 04m 14s)
  • 18:41 arlolra@tin: Starting deploy [parsoid/deploy@c1a14c0]: Retry updating Parsoid to d000fdb4
  • 18:40 arlolra@tin: Finished deploy [parsoid/deploy@c1a14c0]: Updating Parsoid to d000fdb4 (duration: 21m 28s)
  • 18:37 demon@tin: Synchronized docroot: tidying up mobileportal docroot stuff (duration: 00m 41s)
  • 18:24 demon@tin: Synchronized docroot: Removing old wikidata docroot (duration: 00m 46s)
  • 18:19 arlolra@tin: Starting deploy [parsoid/deploy@c1a14c0]: Updating Parsoid to d000fdb4
  • 18:10 marostegui: restart mysql db1065 maintenance - https://phabricator.wikimedia.org/T155999)
  • 18:07 mutante: planet2001 - re-adding to puppet, revoke old cert, sign new cert, initial run
  • 18:02 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1065 - T156006 (duration: 00m 49s)
  • 16:54 andrewbogott: tools deleting tools-mail-01
  • 16:52 mutante: planet2001 - reinstalling to test DHCP/TFTP from install2001
  • 16:37 elukey: upgrading aqs1004 to node6
  • 16:26 marostegui: Restart mysql db1072
  • 16:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1072 - T156006 (duration: 00m 41s)
  • 16:18 paravoid: removing lvs4002_T151273 policy from cr1/2-ulsfo
  • 16:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1072 - T156006 (duration: 00m 47s)
  • 16:12 godog: kill stray swift-proxy processes from ms-fe1* T156143
  • 16:07 filippo@puppetmaster1001: conftool action : set/pooled=yes; selector: name=ms-fe1001.eqiad.wmnet
  • 16:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1073 - T155999 (duration: 00m 48s)
  • 15:58 moritzm: upgraded nodejs on thorium to 6.9 / restarted pivot
  • 15:57 papaul: shutting down ms-be2002 for maintenance
  • 15:54 chasemp: drbdadm adjust tools for 1004/1005 w/ 192.168.0.0/30
  • 15:49 moritzm: installing tomcat7 security updates on trusty hosts (jessie already fixed a while ago)
  • 15:14 godog: bounce pybal on lvs1003 - T134893
  • 15:10 chasemp: drbdadm adjust misc for 1004/1005 w/ 192.168.0.0/30
  • 15:09 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=ms-fe1001.eqiad.wmnet
  • 15:08 filippo@puppetmaster1001: conftool action : set/pooled=yes; selector: name=ms-fe1001.eqiad.wmnet
  • 15:07 chasemp: drbdadm adjust test for 1004/1005 w/ 192.168.0.0/30
  • 15:04 chasemp: recabling labstore1004/1005 eth1
  • 14:55 marostegui: Stop replication on db1052 and db1073 for maintenance - T156006
  • 14:54 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=ms-fe1001.eqiad.wmnet
  • 14:50 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1073 - T155999 (duration: 00m 39s)
  • 14:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1051 with less weight - T156004 (duration: 00m 41s)
  • 14:26 dcausse: EU SWAT Done
  • 14:23 dcausse@tin: Synchronized wmf-config/InitialiseSettings.php: T155515 [cirrus] properly set wgCirrusSearchUseIcuFolding (duration: 00m 39s)
  • 14:13 dcausse@tin: Synchronized wmf-config/InitialiseSettings.php: T155142 [cirrus] Increase weigths for content namespaces on mw.org (duration: 00m 39s)
  • 13:49 marostegui@tin: Synchronized wmf-config/db-codfw.php: Change db1052 IP - T156006 (duration: 00m 39s)
  • 13:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Change db1052 IP - T156006 (duration: 00m 39s)
  • 13:41 marostegui: Shutdown db1052 for maintenance - T156006
  • 13:37 marostegui: Shutdown mysql on db1052 for maintenance - T156006
  • 13:16 marostegui@tin: Synchronized wmf-config/db-codfw.php: Change db1051 IP - T156004 (duration: 00m 39s)
  • 13:14 marostegui@tin: Synchronized wmf-config/db-eqiad.php: wmf-config/db-codfw.php Change db1051 IP - T156004 (duration: 00m 39s)
  • 13:00 marostegui: Shutdown db1051 for maintenance - T156004
  • 12:56 marostegui: Shutdown mysql on db1051 for maintenance - T156004
  • 12:55 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1051 - T156004 (duration: 00m 39s)
  • 12:51 moritzm: installing pcsc-lite security updates on trusty hosts (jessie already fixed a while ago)
  • 12:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1051 - T156004 (duration: 00m 39s)
  • 12:23 akosiaris: switch all networks to use install1001, install2001 as DHCP relay endpoint. T156109
  • 12:17 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Revert last (duration: 00m 39s)
  • 12:14 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T155995 Copy InterwikiSorting settings from wmgWikibaseClientSettings noop (duration: 00m 39s)
  • 12:07 addshore@tin: Synchronized wmf-config/CommonSettings.php: T155995 Prepare to enable InterwikiSorting on beta cluster & Populate InterwikiSortingInterwikiSortOrders with WB Client 4/4 noop (duration: 00m 39s)
  • 12:06 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: T155995 Prepare to enable InterwikiSorting on beta cluster 3/4 noop (duration: 00m 39s)
  • 12:05 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T155995 Prepare to enable InterwikiSorting on beta cluster 2/4 noop (duration: 00m 39s)
  • 12:05 addshore@tin: Synchronized wmf-config/extension-list-labs: T155995 Prepare to enable InterwikiSorting on beta cluster 1/4 noop (duration: 00m 39s)
  • 09:53 akosiaris: mark /dev/sdb as faulty on md devices on bast3001 T154603
  • 09:36 akosiaris: add /dev/sdb partitions to md RAID device on mw2251
  • 09:33 addshore@tin: Synchronized wmf-config/CommonSettings.php: T155995 Prepare to enable InterwikiSorting on beta cluster 4/4 noop (duration: 00m 38s)
  • 09:33 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: T155995 Prepare to enable InterwikiSorting on beta cluster 3/4 noop (duration: 00m 40s)
  • 09:32 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T155995 Prepare to enable InterwikiSorting on beta cluster 2/4 noop (duration: 00m 41s)
  • 09:30 addshore@tin: Synchronized wmf-config/extension-list-labs: T155995 Prepare to enable InterwikiSorting on beta cluster 1/4 noop (duration: 00m 53s)
  • 09:21 marostegui: Alter table db2054 metawiki.pagelinks - T153300
  • 09:21 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2054 - T153300 (duration: 00m 39s)
  • 08:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1065 original weight - T156005 (duration: 00m 39s)
  • 08:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add rack positions - T155999 (duration: 00m 41s)
  • 08:26 marostegui@tin: Synchronized wmf-config/db-codfw.php: wmf-config/db-eqiad.php Add rack positions - T155999 (duration: 00m 50s)
  • 06:20 _joe_: repooling mw2098 after scap pull
  • 02:23 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Jan 24 02:23:01 UTC 2017 (duration 4m 23s)
  • 02:18 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.8) (duration: 06m 40s)
  • 01:37 ejegg: updated SmashPig from 03880ce to ab52dbe
  • 01:18 Krinkle: mwscript deleteEqualMessages.php --wiki gotwiki (T45917)
  • 01:16 ejegg: updated payments-wiki from c22353b to dd8a16d
  • 00:26 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1065 with low load after reimage (duration: 00m 45s)

2017-01-23

  • 22:38 Pchelolo: update RESTBase to 598fa56f
  • 22:37 Pchelolo: update RESTBase to 598fa56f: canary on restbase1007
  • 22:22 Pchelolo: update RESTBase to d1663345c
  • 22:21 Pchelolo: update RESTBase to d1663345c: canary on restbase1007
  • 22:18 Pchelolo: update RESTBase to d1663345c: staging
  • 21:54 demon@tin: Synchronized wmf-config: interwiki update, dropping some old ExtensionMessages files (duration: 00m 41s)
  • 21:43 mutante: sca2004 was out of memory but also fixed itself and i could run puppet again a few minutes later
  • 21:41 demon@tin: Synchronized w: Removing wiki.phtml, apache does the rewrites (duration: 00m 48s)
  • 21:39 bsitzmann@tin: Finished deploy [mobileapps/deploy@7615bf9]: Update mobileapps to 66ef3c2 (duration: 03m 16s)
  • 21:36 bsitzmann@tin: Starting deploy [mobileapps/deploy@7615bf9]: Update mobileapps to 66ef3c2
  • 20:29 nuria@tin: Finished deploy [analytics/aqs/deploy@025ef23]: (no message) (duration: 00m 08s)
  • 20:29 nuria@tin: Starting deploy [analytics/aqs/deploy@025ef23]: (no message)
  • 20:28 nuria@tin: Finished deploy [analytics/aqs/deploy@025ef23]: (no message) (duration: 00m 07s)
  • 20:28 nuria@tin: Starting deploy [analytics/aqs/deploy@025ef23]: (no message)
  • 20:18 otto@tin: Finished deploy [analytics/aqs/deploy@025ef23]: (no message) (duration: 00m 01s)
  • 20:18 otto@tin: Starting deploy [analytics/aqs/deploy@025ef23]: (no message)
  • 20:11 gehel@tin: Finished deploy [wdqs/wdqs@fd88fda]: (no message) (duration: 01m 56s)
  • 20:09 otto@tin: Finished deploy [analytics/aqs/deploy@025ef23]: (no message) (duration: 00m 01s)
  • 20:09 otto@tin: Starting deploy [analytics/aqs/deploy@025ef23]: (no message)
  • 20:09 gehel@tin: Starting deploy [wdqs/wdqs@fd88fda]: (no message)
  • 20:06 gehel: deplyoing latest wdqs version (2h behind planned schedule)
  • 20:04 otto@tin: Finished deploy [analytics/aqs/deploy@025ef23]: (no message) (duration: 02m 16s)
  • 20:02 otto@tin: Starting deploy [analytics/aqs/deploy@025ef23]: (no message)
  • 19:26 reedy@tin: Synchronized wmf-config/CommonSettings.php: Make sure CommonSettings-labs is one of the last things loaded so we don't get problems from things being included after (duration: 00m 40s)
  • 18:57 nuria@tin: Finished deploy [analytics/aqs/deploy@025ef23]: (no message) (duration: 00m 35s)
  • 18:56 nuria@tin: Starting deploy [analytics/aqs/deploy@025ef23]: (no message)
  • 18:47 nuria@tin: Finished deploy [analytics/aqs/deploy@56ab863]: (no message) (duration: 00m 01s)
  • 18:47 nuria@tin: Starting deploy [analytics/aqs/deploy@56ab863]: (no message)
  • 18:47 nuria@tin: Finished deploy [analytics/aqs/deploy@56ab863]: (no message) (duration: 00m 01s)
  • 18:47 nuria@tin: Starting deploy [analytics/aqs/deploy@56ab863]: (no message)
  • 18:45 nuria@tin: Finished deploy [analytics/aqs/deploy@56ab863]: (no message) (duration: 01m 08s)
  • 18:44 nuria@tin: Starting deploy [analytics/aqs/deploy@56ab863]: (no message)
  • 18:43 nuria@tin: Finished deploy [analytics/aqs/deploy@56ab863]: (no message) (duration: 00m 01s)
  • 18:43 nuria@tin: Starting deploy [analytics/aqs/deploy@56ab863]: (no message)
  • 18:33 demon@tin: Synchronized wmf-config/CommonSettings-labs.php: no-op, completeness (duration: 00m 40s)
  • 18:32 demon@tin: Synchronized wmf-config/extension-list-labs: no-op, completeness (duration: 00m 40s)
  • 18:30 jynus: reimaging db1065 to jessie
  • 18:18 papaul: shutting down ms-be2010 for maintenance
  • 18:12 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1065 (duration: 00m 39s)
  • 16:31 papaul: shutting down mw2098 for maintenance
  • 15:38 marostegui: Alter tables: flow_topic_list and flow_tree_node on db1031 (x1 master) - T149819
  • 15:19 elukey: whitelisted dbproxy1011 on cr1/cr2 for analytics-in4 input filter
  • 15:18 moritzm: installing mysql 5.5 security updates (as packaged by jessie/trusty, not the internal mariadb packages)
  • 15:17 moritzm: installing pdns-recursor security update on labservices1002
  • 15:17 Dereckson: Fixed namespaces dupes following NS_PROJECT update on sa.wikisource (T101634)
  • 15:15 gehel: reimage elastic2025 - T154251
  • 15:02 Dereckson: EU SWAT done (handled by addshore)
  • 15:00 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Set site name and meta namespace for Sanskrit wikis (T101634) (duration: 00m 40s)
  • 14:34 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T155844 [fix] Add finds.org.uk without wildcard too (duration: 00m 39s)
  • 14:24 addshore@tin: Synchronized wmf-config/Wikibase.php: T150183 Move InterwikiSortOrders to own file PT 2/2 (duration: 00m 39s)
  • 14:23 addshore@tin: Synchronized wmf-config/InterwikiSortOrders.php: T150183 Move InterwikiSortOrders to own file PT 1/2 (duration: 00m 40s)
  • 14:23 gehel: disabling puppet on elastic20(2[5-9]|3[0-6]) prior to reimage - T154251
  • 14:20 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T155916 Amend category collation for de.wikisource to uca-de-u-kn (duration: 00m 39s)
  • 14:20 godog: depool ms-fe200[1234] T152612
  • 14:15 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T155906 Add n, n:es and n:fr as import sources in test2wiki (duration: 00m 39s)
  • 14:14 Dereckson: Fix namespaces dupes on sa.wikisource to prepare T101634 / Gerrit:333640
  • 14:09 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: gerrit:333476 (NOOP) Temporarily set $wgDisableUserGroupExpiry to true on labs (duration: 00m 40s)
  • 14:07 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: gerrit:333294 Add *.finds.org.uk to wgCopyUploadsDomains (duration: 00m 41s)
  • 11:54 elukey: whitelisted dbproxy1010 on cr1/cr2 for analytics-in4 input filter
  • 10:50 moritzm: installing pdns-recursor security updates on trusty systems
  • 10:38 moritzm: installing openjpeg security updates
  • 09:06 marostegui: Compress s2 on dbstore2001 - T151552
  • 07:47 marostegui: Enabling gtid_domain_id on db1047 (eventlogging host) - T149418
  • 07:43 marostegui: Enabling gtid_domain_id on db1046 (eventlogging master) - T149418
  • 07:32 marostegui: Deploy gtid_domain_id db1043 (passive master) - last host pending in m3 - T149418
  • 07:28 marostegui: Compressing cebwiki.templatelinks on db1015 (224G table) - T153739
  • 03:02 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Jan 23 03:02:32 UTC 2017 (duration 4m 44s)
  • 02:57 reedy@tin: scap sync-l10n completed (1.29.0-wmf.8) (duration: 13m 14s)
  • 02:26 Reedy: running l10nupdate manually
  • 02:23 Reedy: cleaned up reCaptcha extension in l10ncache dirs
  • 02:02 l10nupdate@tin: LocalisationUpdate failed: git pull of extensions failed

2017-01-22

  • 23:26 mobrovac: restbase deploying d1663345 - blacklist of a bot log page on enwiki
  • 02:01 l10nupdate@tin: LocalisationUpdate failed: git pull of extensions failed
  • 00:08 krenair@tin: Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/333468 - needed to be handled before next window (duration: 00m 42s)

2017-01-21

  • 21:57 mobrovac@tin: Finished deploy [changeprop/deploy@2b980fa]: (no message) (duration: 00m 54s)
  • 21:56 mobrovac@tin: Starting deploy [changeprop/deploy@2b980fa]: (no message)
  • 20:02 legoktm@tin: Synchronized php-1.29.0-wmf.8/RELEASE-NOTES-1.29: for completeness (duration: 00m 39s)
  • 20:01 legoktm@tin: Synchronized php-1.29.0-wmf.8/resources: Revert "Added reason suggestion in block/delete/protect forms" (1/2) - T34950 (duration: 00m 39s)
  • 20:00 legoktm@tin: Synchronized php-1.29.0-wmf.8/includes: Revert "Added reason suggestion in block/delete/protect forms" (1/2) - T34950 (duration: 01m 31s)
  • 04:03 ema: graphite1003: carbon-cache@c restarted, it's been killed by OOM killer again
  • afk: disabled civicrm dedupe high numbers
  • afk: disabled civicrm dedupe
  • 02:02 l10nupdate@tin: LocalisationUpdate failed: git pull of extensions failed
  • 01:21 mobrovac@tin: Finished deploy [changeprop/deploy@eb27062]: (no message) (duration: 01m 03s)
  • 01:20 mobrovac@tin: Starting deploy [changeprop/deploy@eb27062]: (no message)
  • 00:36 volans: restarted carbon-cache@c on graphite1003 (was killed by oom-killer)
  • 00:20 mobrovac: restbase deploying 7c753fe6

2017-01-20

  • 23:37 mattflaschen@tin: Synchronized docroot: No-op file rename (duration: 00m 46s)
  • 23:36 mattflaschen@tin: Synchronized dblists: No-op file rename (duration: 00m 54s)
  • 19:55 robh: done fixing ulsfo serial in ulsfo
  • 19:45 robh: messing with ulsfo serial connections
  • 19:29 robh: cp4012 donating its redundant power supply to lvs4002 with redundant supplies
  • 19:12 ejegg: re-enabled fundraising Jenkins jobs
  • 19:01 ejegg: disabled fundraising jenkins jobs
  • 17:22 chasemp: shutdown eth1 on labstore1004 for testing
  • 17:17 andrewbogott: graceful'd apache on silver, in hopes that the wikitech instance api will update
  • 16:18 cmjohnson1: swapping cable eth0 labstore1004 (chasemp)
  • 16:03 jynus: restart and upgrade of db2066
  • 14:42 jynus: restart and upgrade of db2067
  • 13:30 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2047 - T153300 (duration: 00m 39s)
  • 12:32 ladsgroup@tin: Synchronized php-1.29.0-wmf.8/extensions/ORES/includes/Hooks.php: ORES database query fix (T155500) (duration: 00m 40s)
  • 12:13 Amir1: deploy wmf.8 in mwdebug1002 (T155500)
  • 10:48 godog: reload swift-proxy on ms-fe100* to pick up https://gerrit.wikimedia.org/r/333222
  • 10:39 elukey: manually forcing a /etc/init.d/apache2 reload on mw1259 (videoscaler) to replicate the effects of a logrotate run and test why alarms go off.
  • 10:15 moritzm: installing exim bugfix updates from latest jessie point release
  • 10:02 godog: reload swift-proxy on ms-fe1001 to pick up https://gerrit.wikimedia.org/r/333222
  • 09:19 jynus: rolling restart and upgrade of labsdb1009/10/11 to mariadb 10.1.21-2
  • 08:41 marostegui: Remove partitions on metawiki.pagelinks db2047 - T153300
  • 08:25 _joe_: restarting pybal on lvs1003/1006 to pick up config changes
  • 07:38 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2047 - T153300 (duration: 00m 48s)
  • 07:09 marostegui: Compress pagelinks tables on db1015 - T153739
  • 02:36 l10nupdate@tin: ResourceLoader cache refresh completed at Fri Jan 20 02:36:44 UTC 2017 (duration 5m 34s)
  • 02:31 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.8) (duration: 11m 23s)
  • 00:52 mutante: tin - keyholder disarm and arm again using new passphrase
  • 00:49 mutante: mira - arming keyholder after setting service/dumps/eventlogging/phabricator key passphrases to the same one (T154943)
  • 00:46 mutante: setting all deployment key passphrases to the one used for mw deploy - update key files in private repo (T154943)
  • 00:40 mobrovac@tin: Finished deploy [parsoid/deploy@465f9c4]: Restarting Parsoid everywhere for Node v6 switch T149331 (duration: 04m 21s)
  • 00:39 thcipriani@tin: Synchronized php-1.29.0-wmf.8/includes/specials/SpecialContributions.php: SWAT: SpecialContributions: Username input is not really required T155780 (duration: 00m 39s)
  • 00:35 mobrovac@tin: Starting deploy [parsoid/deploy@465f9c4]: Restarting Parsoid everywhere for Node v6 switch T149331
  • 00:34 thcipriani@tin: Synchronized php-1.29.0-wmf.8/resources/lib/oojs-ui: SWAT: resources: Update OOjs UI with fixes on top of v0.18.3 T155728 (duration: 00m 41s)
  • 00:24 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Wikidata description taglines shown on English Wikipedia T152743 (duration: 00m 39s)
  • 00:22 volans: apt-upgrading nodejs to v6 on the rest of parsoid hosts (a deploy with restart will follow) T149331

2017-01-19

  • 23:34 mutante: force puppet run on restbase1*
  • 23:32 volans: upgrading node to v6 on wtp1003 T149331
  • 23:31 mutante: force puppet run on restbase2*
  • 23:31 mutante: icinga - replace check command names in puppet_services.cfg for change 333010
  • 23:26 mobrovac@tin: Finished deploy [eventstreams/deploy@0d1d9c6]: Bump preq to 0.5.2 for Node v6 (duration: 02m 11s)
  • 23:24 mobrovac@tin: Starting deploy [eventstreams/deploy@0d1d9c6]: Bump preq to 0.5.2 for Node v6
  • 23:16 mobrovac@tin: Finished deploy [cxserver/deploy@5ae4f8b]: Bump preq to 0.5.2 for Node v6 (duration: 01m 56s)
  • 23:14 mobrovac@tin: Starting deploy [cxserver/deploy@5ae4f8b]: Bump preq to 0.5.2 for Node v6
  • 23:11 mobrovac@tin: Finished deploy [graphoid/deploy@da37386]: Bump preq to 0.5.2 for Node v6 (duration: 02m 21s)
  • 23:10 ejegg: updated SmashPig from f05c9a3 to 03880ce
  • 23:08 mobrovac@tin: Starting deploy [graphoid/deploy@da37386]: Bump preq to 0.5.2 for Node v6
  • 22:59 ppchelko@tin: Finished deploy [eventstreams/deploy@fe77f19]: Deploy for switching to node 6 T149331 (duration: 01m 30s)
  • 22:58 mobrovac@tin: Finished deploy [trending-edits/deploy@0abcf25]: Switching to node 6 T149331 (duration: 01m 59s)
  • 22:58 ppchelko@tin: Starting deploy [eventstreams/deploy@fe77f19]: Deploy for switching to node 6 T149331
  • 22:58 ppchelko@tin: Finished deploy [cxserver/deploy@ff0225e]: Deploy for switching to node 6 T149331 (duration: 02m 12s)
  • 22:56 mobrovac@tin: Finished deploy [electron-render/deploy@f1df2d3]: Switching to node 6 T149331 (duration: 01m 58s)
  • 22:56 mobrovac@tin: Starting deploy [trending-edits/deploy@0abcf25]: Switching to node 6 T149331
  • 22:56 mobrovac@tin: Finished deploy [mobileapps/deploy@cacb3c9]: Switching to node 6 T149331 (duration: 02m 41s)
  • 22:55 ppchelko@tin: Starting deploy [cxserver/deploy@ff0225e]: Deploy for switching to node 6 T149331
  • 22:55 ppchelko@tin: Finished deploy [citoid/deploy@95df861]: Deploy for switching to node 6 T149331 (duration: 02m 18s)
  • 22:54 mobrovac@tin: Starting deploy [electron-render/deploy@f1df2d3]: Switching to node 6 T149331
  • 22:54 mobrovac@tin: Finished deploy [mathoid/deploy@ba3217e]: (no message) (duration: 02m 02s)
  • 22:53 mobrovac@tin: Starting deploy [mobileapps/deploy@cacb3c9]: Switching to node 6 T149331
  • 22:53 mobrovac@tin: Finished deploy [graphoid/deploy@f872f94]: Switching to node 6 T149331 (duration: 01m 39s)
  • 22:53 ppchelko@tin: Starting deploy [citoid/deploy@95df861]: Deploy for switching to node 6 T149331
  • 22:52 ppchelko@tin: Finished deploy [changeprop/deploy@ffd0b8b]: Deploy for switching to node 6 T149331 (duration: 00m 58s)
  • 22:52 mobrovac@tin: Starting deploy [mathoid/deploy@ba3217e]: (no message)
  • 22:51 mobrovac@tin: Starting deploy [graphoid/deploy@f872f94]: Switching to node 6 T149331
  • 22:51 ppchelko@tin: Starting deploy [changeprop/deploy@ffd0b8b]: Deploy for switching to node 6 T149331
  • 22:51 mutante: scb1003,scb1004 - upgrade nodejs
  • 22:51 ppchelko@tin: Finished deploy [changeprop/deploy@ffd0b8b]: Canary deploy for switching to node 6 T149331 (duration: 07m 36s)
  • 22:48 mobrovac@tin: Finished deploy [mathoid/deploy@ba3217e]: (no message) (duration: 03m 24s)
  • 22:47 mobrovac@tin: Finished deploy [graphoid/deploy@f872f94]: Switching to node 6 T149331 (duration: 01m 43s)
  • 22:45 mobrovac@tin: Starting deploy [graphoid/deploy@f872f94]: Switching to node 6 T149331
  • 22:45 mobrovac@tin: Finished deploy [graphoid/deploy@f872f94]: Switching to node 6 T149331 (duration: 01m 10s)
  • 22:45 mobrovac@tin: Starting deploy [mathoid/deploy@ba3217e]: (no message)
  • 22:44 mobrovac@tin: Starting deploy [graphoid/deploy@f872f94]: Switching to node 6 T149331
  • 22:44 ppchelko@tin: Starting deploy [changeprop/deploy@ffd0b8b]: Canary deploy for switching to node 6 T149331
  • 22:41 mutante: scb1001-1004 - upgraded nodejs version
  • 22:38 mutante: scb2003 - repool, scb2001,scb2002 - upgrade nodejs, libuv1 packages
  • 22:35 mutante: scb2003 - depool, upgrade nodejs, libuv1 packages
  • 22:33 mutante: scb2004 - re-pooled
  • 21:35 chasemp: rebooting labstore1004
  • 21:33 chasemp: failover secondary labstore cluster from 1004 to 1004
  • 21:17 chasemp: force non tools on NFS to go ro
  • 21:03 volans: upgrading node to v6 on wtp1002 T149331
  • 20:46 mutante: scb2004 - upgrading nodejs, libuv1
  • 20:45 volans: upgrading node to v6 on wtp2003 T149331
  • 20:44 mutante: depooling scb2004 for nodejs install
  • 20:26 volans: upgrading node to v6 on wtp2002 T149331
  • 20:09 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.8
  • 20:05 dereckson@tin: Synchronized wmf-config/throttle.php: Add throttle rule for BAMU event (T154312) (duration: 00m 39s)
  • 19:44 ejegg: updated SmashPig from 48675c3 to f05c9a3
  • 19:40 mutante: switching dumps.wikimedia.org to Letsencrypt SSL cert
  • 19:20 jynus: restarting db1069:3311 due to query being "stuck" on tokudb table
  • 19:13 demon@tin: Synchronized wmf-config/CommonSettings.php: ContentTranslation: Enable publishing article in testwiki (2/2) (duration: 00m 39s)
  • 19:12 demon@tin: Synchronized wmf-config/InitialiseSettings.php: ContentTranslation: Enable publishing article in testwiki (1/2) (duration: 00m 39s)
  • 19:06 demon@tin: Synchronized wmf-config/CommonSettings.php: Double $wgTranscodeBackgroundTimeLimit to compensate for threading (duration: 00m 47s)
  • 18:54 ema: libvmod-header removed from carbon, varnish-modules provides it
  • 18:51 mutante: dataset1001 - temp disabling puppet, ms1001 - switching to Letsencrypt cert
  • 18:34 jynus: aborting rolling restart on labsdb1010, labsdb1011 due to package bug to be fixed on 10.1.21-2
  • 18:19 mobrovac: restbase updating firejail in production
  • 17:52 jynus: rolling restart and upgrade of labsdb1009/10/11 to mariadb 10.1.21
  • 17:47 Dereckson: Reattach Zlazstadpieroniebomiurwieszkabelodinternetu CentralAuth account (T155184)
  • 17:26 jynus: restarting and upgrading mariadb on labsdb1004 to 10.0.29
  • 14:40 Dereckson: EU SWAT done
  • 14:40 Dereckson: `mwscript namespaceDupes.php sawiki --fix` (T101634)
  • 14:36 ottomata: restarting apache/puppetmaster on labcontrol1001 to try to fix 'invalid byte sequence in US-ASCII' puppet error
  • 14:34 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Fix Portal talk namespace name on Sanskrit Wikipedia (T101634) (duration: 00m 39s)
  • 14:25 dereckson@tin: Synchronized wmf-config: Add noratelimit user right to translation admins on Commons (T155162) (duration: 00m 42s)
  • 13:21 marostegui: Compressing revision,pagelinks and templatelinks tables on db1035 - T110504
  • 11:41 marostegui: Compressing dewiki db1045 - T155399
  • 10:14 marostegui: Compressing templatelinks tables on db1015 - T153739
  • 09:32 moritzm: upgrading firejail on image scalers
  • 08:57 godog: bounce udp2log on fluorine after https://gerrit.wikimedia.org/r/313604
  • 07:45 volans@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=mw2098.codfw.wmnet
  • 07:42 volans@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=mw2098.wmnet
  • 07:28 dereckson@tin: Synchronized wmf-config/throttle.php: Fix throttle rule for KCES IMR edit-a-thon (duration: 02m 42s)
  • 07:18 marostegui: Compressing enwikivoyage.text and shwiki.logging tables on db1044 - T153826
  • 07:14 marostegui: Compressing enwikivoyage.text and shwiki.logging tables on db1038 - T154465
  • 02:28 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.7) (duration: 07m 28s)
  • 01:50 mutante: install - if in private1-c-eqiad, private1-b-codfw, you are using install1001 for both DHCP and TFTP, if in other networks you still use carbon as DHCP but then also install1001 as TFTP
  • 01:47 mutante: install1001/2001 - re-enabled, carbon is still DHCP for some rows
  • 01:23 mutante: install1001 - re-enable puppet - install2001 - same thing, temp disable and live-hack mw2251 to use trusty installer
  • 01:16 mutante: switching mw2251 to trusty-installer for test
  • 01:14 mutante: temp disable puppet on install1001 for papaul debugging
  • 00:24 maxsem@tin: Synchronized php-1.29.0-wmf.8/extensions/Graph: SWAT https://gerrit.wikimedia.org/r/#/c/332916/1 (duration: 00m 40s)

2017-01-18

  • 23:38 demon@tin: Synchronized php-1.29.0-wmf.8/extensions/Collection: Unbreak (duration: 00m 40s)
  • 22:59 demon@tin: Synchronized php-1.29.0-wmf.8/extensions/ProofreadPage/includes/index/ProofreadIndexPage.php: Unbreak, T155682 (duration: 00m 39s)
  • 22:49 demon@tin: Synchronized php-1.29.0-wmf.8/extensions/LiquidThreads/classes/Hooks.php: Unbreak hook mess (duration: 00m 41s)
  • 22:48 demon@tin: Synchronized php-1.29.0-wmf.8/includes/widget/search/FullSearchResultWidget.php: Unbreak hook mess (duration: 00m 45s)
  • 22:46 madhuvishy: Reenabled nfs-exportd and puppet on labstore1004. All of misc being exported as rw now. T154336
  • 21:32 bd808: Updated wikimania-scholarships to 29ba0ec "Add Tulu (tcy) to Communities" (T155666)
  • 21:18 volans: restarted pybal on lvs2001 (active) T134893
  • 21:00 volans: restarted pybal on lvs2004 (passive) T134893
  • 20:48 demon@tin: Finished scap: group1 to wmf.8 (duration: 46m 33s)
  • 20:47 volans: Upgraded nodejs to v6 on wtp1001 T149331
  • 20:31 nuria@tin: Finished deploy [analytics/refinery@666d98d]: (no message) (duration: 02m 19s)
  • 20:28 nuria@tin: Starting deploy [analytics/refinery@666d98d]: (no message)
  • 20:01 demon@tin: Started scap: group1 to wmf.8
  • 19:56 volans: restarted pybal on lvs2003
  • 19:51 volans: restarted pybal on lvs2006
  • 19:26 demon@tin: Synchronized dblists: Remove old compact lang list dblist (duration: 00m 39s)
  • 19:25 volans: Upgrading nodejs to v6 on wtp2001 T149331
  • 19:25 demon@tin: Synchronized docroot/noc/conf: Using new compact lang list dblist (duration: 00m 39s)
  • 19:24 demon@tin: Synchronized tests/cirrusTest.php: Use new compact lang list dblist (duration: 00m 39s)
  • 19:22 papaul: OS installation on mw2251-mw2260
  • 19:22 demon@tin: Synchronized wmf-config: Use new compact lang links dblist (duration: 00m 41s)
  • 19:21 demon@tin: Synchronized dblists/compact-language-links.dblist: New dblist (duration: 00m 39s)
  • 19:12 volans: Upgrading nodejs to v6 on ruthenium T149331
  • 19:06 dereckson@tin: Synchronized wmf-config/throttle.php: Add throttle rule for KCES IMR edit-a-thon (T154312) (duration: 00m 39s)
  • 18:35 demon@tin: Synchronized multiversion/MWMultiVersion.php: Swapping 500 -> 400 when specifying invalid host headers (duration: 00m 39s)
  • 18:28 demon@tin: Synchronized multiversion/MWMultiVersion.php: minor cleanup (duration: 00m 48s)
  • 18:03 madhuvishy: Rolling out https://gerrit.wikimedia.org/r/#/c/332735/ across labs instances T154336
  • 17:54 madhuvishy: Disabled (systemctl disable) nfs-export on labstore1001 and 1004 to prevent auto restart from bringing them back up T154336
  • 17:42 mobrovac: restbase deploying 3027682
  • 17:40 madhuvishy: Starting final sync of latest diff from labstore1001 to labstore-secondary T154336
  • 17:38 madhuvishy: Disabling puppet on labstore1001 and 1004 to make sure nfs exports are not overridden T154336
  • 17:37 madhuvishy: Exporting all misc shares from labstore1004 as RO T154336
  • 17:30 madhuvishy: Stopping nfs-exportd on labstore1004 T154336
  • 17:29 madhuvishy: Exported all misc exports as RO on labstore1001 T154336
  • 17:26 madhuvishy: Stopping nfs-exportd on labstore1001 T154336
  • 17:13 madhuvishy: Disabling puppet across labs instances with NFS (/home and/or /data/project) mounted for T154336
  • 17:12 madhuvishy: Silenced shinken, and icinga on labstore1001 for misc nfs migration T154336
  • 15:45 oblivian@puppetmaster1001: conftool action : set/pooled=inactive; selector: service=nginx,cluster=api_appserver,dc=eqiad,name=mw122[6-9].*
  • 15:45 oblivian@puppetmaster1001: conftool action : set/pooled=inactive; selector: service=nginx,cluster=api_appserver,dc=eqiad,name=mw123[0-5].*
  • 15:41 oblivian@puppetmaster1001: conftool action : set/weight=15; selector: service=apache2,cluster=api_appserver,dc=eqiad,name=mw1(1[8-9]|2[0-1]|22[0-5]).*
  • 15:24 zeljkof: finished EU SWAT
  • 15:23 zfilipin@tin: Synchronized php-1.29.0-wmf.7/maintenance/importImages.php: SWAT: maintenance/importImages: Dont sleep after the last upload (duration: 00m 41s)
  • 15:07 hashar: tin.eqiad.wmnet : committed an uncommitted live hack for php-1.29.0-wmf.7/includes/AutoLoader.php by ostriches
  • 15:06 zeljkof: extending EU SWAT until 332766 is deployed
  • 14:39 zfilipin@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Increase $wgHTTPImportTimeout to 50 seconds (T155209) (duration: 00m 39s)
  • 14:31 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set wgDisableUserGroupExpiry to true on production, false on labs (T155605) (duration: 00m 40s)
  • 14:30 zfilipin@tin: Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: Set wgDisableUserGroupExpiry to true on production, false on labs (T155605) (duration: 00m 40s)
  • 14:03 moritzm: upgrading firejail on aqs cluster
  • 13:12 moritzm: uploaded firejail 0.9.44.6 for jessie-wikimedia to carbon
  • 12:21 marostegui: Enable gtid_domain_id on m3 - T149418
  • 12:06 moritzm: installing libio-socket-ssl-perl bugfix updates from jessie point release
  • 11:42 moritzm: installing sed bugfix updates from jessie point release
  • 11:17 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2064 - T154097 (duration: 00m 39s)
  • 11:09 moritzm: restarting mediawiki canary servers to pick up cairo and libpng updates
  • 10:38 moritzm: installing libxml security updates
  • 10:38 oblivian@puppetmaster1001: conftool action : set/pooled=no; selector: service=nginx,cluster=api_appserver,dc=eqiad,name=mw123[0-5].*
  • 10:11 godog: pool ms-fe200[789] T152612
  • 10:10 marostegui: Restart mysql dbstore2001 to enable gtid_domain_id manually before deploying it on m3 - T149418
  • 10:05 marostegui: Remove partitions from enwiktionary.templatelinks on db2064 - T154097
  • 10:05 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2064 - T154097 (duration: 00m 45s)
  • 09:56 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2063 - T154097 (duration: 00m 48s)
  • 09:40 marostegui: Restart mysql dbstore2002 to enable gtid_domain_id manually before deploying it on m3 - T149418
  • 08:51 marostegui: Remove partitions from enwiktionary.templatelinks on db2063 - T154097
  • 08:50 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2063 - T154097 (duration: 00m 39s)
  • 08:33 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2060 - T154031 (duration: 00m 40s)
  • 08:18 marostegui: Compressing templatelinks tables on db1035 - T154465
  • 08:16 marostegui: Compressing templatelinks tables on db1038 - T154465
  • 08:00 _joe_: restarting pybal on lvs1003
  • 07:56 _joe_: restarting pybal on lvs1003
  • 07:34 oblivian@puppetmaster1001: conftool action : set/pooled=no; selector: service=nginx,cluster=api_appserver,dc=eqiad,name=mw122[6-9].*
  • 07:32 _joe_: depooling mw1226-mw1235 from the https pool in eqiad, T152074
  • 07:30 oblivian@puppetmaster1001: conftool action : set/pooled=inactive; selector: service=nginx,cluster=api_appserver,dc=eqiad,name=mw12[7-9].*
  • 07:25 marostegui: Restart MySQL dbstore2001 to apply InnoDB defaults
  • 03:05 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Jan 18 03:05:47 UTC 2017 (duration 5m 38s)
  • 03:00 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.8) (duration: 13m 22s)
  • 02:29 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.7) (duration: 08m 23s)
  • 02:00 ejegg: updated SmashPig from 03ef6b1 to 48675c3
  • 01:21 mobrovac@tin: Finished deploy [citoid/deploy@9f93a00]: (no message) (duration: 04m 14s)
  • 01:17 mobrovac@tin: Starting deploy [citoid/deploy@9f93a00]: (no message)
  • 00:59 mobrovac@tin: Finished deploy [trending-edits/deploy@1d53b7c]: fixes for T153122 and T145571 (duration: 05m 06s)
  • 00:54 mobrovac@tin: Starting deploy [trending-edits/deploy@1d53b7c]: fixes for T153122 and T145571
  • 00:35 demon@tin: Synchronized w/mobilelanding.php: Last major fix for multiversion (duration: 00m 45s)
  • 00:29 ejegg: updated SmashPig from 3da597f to 03ef6b1
  • 00:00 demon@tin: Synchronized multiversion/MWVersion.php: Swap to using MWMultiVersion and make this a fallback (duration: 00m 39s)

2017-01-17

  • 23:47 demon@tin: Synchronized w: Step 4/∞ of multiversion cleanups (duration: 00m 39s)
  • 23:42 demon@tin: Synchronized multiversion/MWScript.php: Step 3/∞ of multiversion cleanups (duration: 00m 39s)
  • 23:21 demon@tin: Synchronized rpc/RunJobs.php: Step 2/∞ of multiversion cleanups (duration: 00m 39s)
  • 22:30 demon@tin: Synchronized multiversion: Step 1/∞ of multiversion cleanups (duration: 00m 55s)
  • 21:48 demon@tin: Synchronized multiversion/MWVersion.php: Removing old getMediaWikiCli() entry point, unused (duration: 00m 39s)
  • 20:41 demon@tin: Synchronized php-1.29.0-wmf.8/extensions/LiquidThreads/classes/Hooks.php: Fix warning about pass-by-ref (duration: 00m 40s)
  • 20:09 ema: restarting hhvm on mw1227 - hhvm-dump-debug in /tmp/hhvm.25127.bt
  • 20:08 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.8
  • 19:52 demon@tin: Finished scap: testwiki to wmf.8 + rebuild l10n (duration: 46m 21s)
  • 19:36 ejegg: updated payments-wiki from 1f9ea80 to c22353b
  • 19:06 demon@tin: Started scap: testwiki to wmf.8 + rebuild l10n
  • 19:05 demon@tin: Synchronized wmf-config/throttle.php: throttle rule for T155493 (duration: 00m 40s)
  • 19:04 demon@tin: Synchronized wmf-config/InitialiseSettings.php: Enable subpages in NS_MAIN in eswikiversity (duration: 00m 39s)
  • 19:02 demon@tin: Synchronized wmf-config/InitialiseSettings.php: Add *.leventhalmap.org to the copyupload whitelist (duration: 00m 39s)
  • 19:01 demon@tin: Synchronized wmf-config/InitialiseSettings.php: avwiki namespace tweaks, T155321 (duration: 00m 39s)
  • 19:00 demon@tin: Synchronized wmf-config/throttle.php: T155510 throttle rule (duration: 01m 36s)
  • 18:03 mobrovac: restbase deploying a0e542b, switching to Node v6 T149331
  • 17:48 mobrovac: restbase installing node v6.9.1 on the cluster T149331
  • 16:29 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2056 - T154097 (duration: 00m 48s)
  • 16:22 marostegui: Powering off db2060 for maintenance - T154031
  • 16:16 filippo@puppetmaster1001: conftool action : set/pooled=yes; selector: name=ms-fe2005.codfw.wmnet
  • 15:01 moritzm: installing bash security updates
  • 14:10 moritzm: installing bind9 security updates
  • 13:33 moritzm: installing libpng security updates
  • 12:29 marostegui: Remove partitions from enwiktionary.templatelinks on db2056 - T154097
  • 12:27 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2056 - T154097 (duration: 00m 42s)
  • 12:16 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2049 - T154097 (duration: 00m 47s)
  • 11:54 moritzm: installing potrace security updates
  • 11:32 moritzm: installing w3m security updates
  • 11:22 moritzm: installing tre security updates
  • 11:19 moritzm: installing python-werkzeug security updates
  • 11:15 marostegui: Remove partitions from enwiktionary.templatelinks on db2049 - T154097
  • 11:13 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2049 - T154097 (duration: 00m 38s)
  • 10:58 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2041 - T154097 (duration: 00m 38s)
  • 10:43 moritzm: installing file/libmagic security updates
  • 10:35 hashar: CI switched NodeJS from v4 to v6 T155443 T149331
  • 10:22 moritzm: installing jq security updates
  • 10:16 hashar: Updating CI Jessie image for NodeJs 4 -> 6 upgrade. T155443
  • 10:06 moritzm: installing libwmf security updates
  • 10:02 marostegui: Remove partitions from enwiktionary.templatelinks on db2041 - T154097
  • 09:59 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2041 - T154097 (duration: 00m 38s)
  • 09:22 moritzm: installing tomcat security updates
  • 08:42 marostegui: Compressing wikidatawiki on db1026 - https://phabricator.wikimedia.org/T154929
  • 08:33 moritzm: installing tiff security updates
  • 07:50 marostegui: Remove partitions from enwiktionary.templatelinks on dbstore2001 - T154097
  • 07:26 marostegui: Compressing revision tables db1035 (depooled)
  • 06:58 marostegui: Compressing cebwiki/templatelinks (215G) table on db1038 - T154465
  • 02:26 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Jan 17 02:26:07 UTC 2017 (duration 4m 22s)
  • 02:21 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.7) (duration: 07m 20s)

2017-01-16

  • 18:42 moritzm: uploaded nodejs 6.9.1 for jessie-wikimedia to carbon
  • 15:01 elukey: restarting hhvm on mw1167 - hhvm-dump-debug in /tmp/hhvm.20360.bt
  • 14:47 hashar: European SWAT complete
  • 14:44 hashar@tin: Synchronized wmf-config/throttle.php: Add a new throttle rule - T155416 (duration: 00m 38s)
  • 14:26 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Namespace aliases on Bhojpuri Wikipedia (bhwiki) - T155278 (duration: 00m 41s)
  • 14:20 hashar@tin: Synchronized wmf-config/throttle.php: Add one throttle rule + remove obsolete ones T155345 (duration: 00m 38s)
  • 14:18 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: wgBabelMainCategory for cswikiversity to Uživatel %code% T155301 (duration: 00m 39s)
  • 13:12 moritzm: installing pysaml2 security updates
  • 12:26 moritzm: installing pdns-recursor security updates
  • 10:35 marostegui: Compressing templatelinks tables on db1044 (depooled) - T153826
  • 10:30 marostegui: Compressing pagelinks tables on db1038 - T154465
  • 09:13 marostegui: Compressing dewiki on db1026 - T154929
  • 08:56 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2034 - T149553 (duration: 00m 38s)
  • 02:25 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Jan 16 02:25:46 UTC 2017 (duration 4m 21s)
  • 02:21 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.7) (duration: 08m 04s)

2017-01-15

  • 02:25 l10nupdate@tin: ResourceLoader cache refresh completed at Sun Jan 15 02:25:27 UTC 2017 (duration 4m 23s)
  • 02:21 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.7) (duration: 07m 46s)

2017-01-14

  • 02:35 l10nupdate@tin: ResourceLoader cache refresh completed at Sat Jan 14 02:35:07 UTC 2017 (duration 4m 25s)
  • 02:30 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.7) (duration: 12m 03s)

2017-01-13

  • 22:56 godog: delete labs instance data older than 60d from graphite[21]001, low disk space
  • 02:25 l10nupdate@tin: ResourceLoader cache refresh completed at Fri Jan 13 02:25:57 UTC 2017 (duration 5m 16s)
  • 02:20 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.7) (duration: 07m 11s)
  • 01:32 bsitzmann@tin: Finished deploy [trending-edits/deploy@cf388a9]: Update trending-edits to 421fa63 (duration: 01m 53s)
  • 01:30 bsitzmann@tin: Starting deploy [trending-edits/deploy@cf388a9]: Update trending-edits to 421fa63
  • 00:12 demon@tin: Synchronized multiversion: Clean up cli entry point (duration: 00m 54s)

2017-01-12

  • 23:05 demon@tin: Synchronized README: no-op for force co-master sync (duration: 00m 40s)
  • 22:49 demon@tin: Synchronized docroot/foundation: Yay no more powerpoints (duration: 00m 38s)
  • 22:38 demon@tin: Synchronized docroot/foundation/presentations: removing some of these powerpoints (duration: 00m 38s)
  • 22:08 maxsem@tin: Synchronized php-1.29.0-wmf.7/extensions/Graph/includes/ApiGraph.php: Debug for T155057 (duration: 00m 38s)
  • 18:13 demon@tin: Synchronized wmf-config/InitialiseSettings.php: Use HD logos for (nap|os|pl|pt)wiki (duration: 00m 41s)
  • 18:12 demon@tin: Synchronized static/images/project-logos: HD logos for (nap|os|pl|pt)wiki (duration: 00m 39s)
  • 09:16 akosiaris: T155112 upload Vagrant 1.9.1 to apt.wikimedia.org/jessie-wikimedia/thirdparty and apt.wikimedia.org/trusty-wikimedia/thirdparty
  • 08:59 hashar: disabling puppet on contint1001 to live hack apache conf ( T150727 )
  • 02:46 demon@tin: Synchronized wmf-config/interwiki.php: T154225 (duration: 00m 38s)
  • 02:37 demon@tin: Synchronized wmf-config/InitialiseSettings-labs.php: no-op, completeness (duration: 00m 38s)
  • 02:36 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Jan 12 02:36:23 UTC 2017 (duration 5m 15s)
  • 02:31 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.7) (duration: 11m 12s)
  • 01:33 Reedy: running scap pull on tin
  • 00:29 demon@tin: Synchronized wmf-config/InitialiseSettings.php: oathauth group for wikitech (duration: 00m 38s)
  • 00:21 demon@tin: Synchronized docroot/noc/db.php: (no message) (duration: 00m 39s)
  • 00:17 reedy@tin: Synchronized wmf-config: More consistency for various commits (duration: 00m 40s)
  • 00:12 nuria: restarted apache2 and mysql on bohrium to see if mysql no connection errors disappear
  • 00:12 demon@tin: Synchronized wmf-config/InitialiseSettings.php: Wikidata lang config (duration: 00m 38s)

2017-01-11

  • 23:55 demon@tin: Synchronized wmf-config/InitialiseSettings.php: Minerva hacks (duration: 00m 38s)
  • 23:47 reedy@tin: Synchronized wmf-config: consistency (duration: 00m 41s)
  • 23:16 demon@tin: Synchronized wmf-config/CommonSettings.php: video transcode jobqueue stuff for Brion (duration: 00m 38s)
  • 23:10 demon@tin: Synchronized w/static.php: For Timo <3 (duration: 00m 40s)
  • 22:48 demon@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Use new metawiki logo, no-op in prod (duration: 00m 38s)
  • 22:47 demon@tin: Synchronized static/images/project-logos: beta logos (duration: 00m 40s)
  • 22:44 demon@tin: Synchronized wmf-config/CommonSettings.php: commentfix (duration: 00m 38s)
  • 22:39 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: Simplify ores config. Enable sitenotice banners for arwiki (duration: 00m 39s)
  • 22:32 demon@tin: Synchronized wmf-config/abusefilter.php: comment fix (duration: 00m 39s)
  • 22:26 elukey: added mw1239.eqiad.wmnet back to service - T148421
  • 22:20 elukey: restarting hhvm on mw1198 (dump-debug in /tmp/hhvm.9737.bt)
  • 22:13 demon@tin: Synchronized wmf-config/abusefilter.php: Set $wgAbuseFilterNotificationsPrivate = true; for Meta-Wiki (duration: 00m 40s)
  • 22:04 demon@tin: Synchronized wmf-config/flaggedrevs.php: Deprecated variable cleanup (duration: 00m 38s)
  • 21:54 demon@tin: Synchronized scap/plugins/wmf-beta-autoupdate.py: no-op, not yet used (duration: 00m 38s)
  • 21:45 demon@tin: Synchronized wmf-config/CommonSettings-labs.php: no-op (duration: 00m 38s)
  • 21:28 demon@tin: Synchronized wmf-config: massmessage hack cleanup + comments on kartographer wikivoyage mode (duration: 00m 41s)
  • 21:17 demon@tin: Synchronized wmf-config/InitialiseSettings.php: use new HD logos (duration: 00m 38s)
  • 21:16 demon@tin: Synchronized static/images/project-logos/notifications: New HD logos (duration: 00m 38s)
  • 21:10 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: pawiki logos. remove technican from trwikiquote (duration: 00m 38s)
  • 21:09 reedy@tin: Synchronized static/images: pawiki (duration: 00m 42s)
  • 21:03 Reedy: update collation of fiwikivoyage T151570
  • 21:02 reedy@tin: Synchronized php-1.29.0-wmf.7/extensions/CentralAuth/extension.json: fix name (duration: 00m 41s)
  • 21:01 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: Translation namespace for mlwikisource. fiwikivoyage collation (duration: 00m 40s)
  • 20:53 reedy@tin: Synchronized wmf-config/CommonSettings.php: Upgrade Collections license URL to HTTPS (duration: 00m 57s)
  • 20:52 reedy@tin: Synchronized wmf-config/throttle.php: Fix throttle (duration: 00m 42s)
  • 20:44 Dereckson: Reset user e-mail for account for Panam2014
  • 20:43 demon@tin: Synchronized tests/noc-conf/NOCDblistTest.php: No-op (duration: 00m 40s)
  • 20:40 demon@tin: Synchronized wmf-config/InitialiseSettings-labs.php: no-op (duration: 00m 40s)
  • 19:36 demon@tin: Synchronized multiversion: rollback (duration: 00m 56s)
  • 19:34 demon@tin: Synchronized multiversion: MWVersion fallbacks & such (duration: 00m 56s)
  • 19:27 demon@tin: Synchronized php-1.29.0-wmf.7/extensions/FlaggedRevs: Stupid errors (duration: 00m 46s)
  • 18:56 demon@tin: Synchronized multiversion/MWMultiVersion.php: Attempt #2 for Multiversion cleanup (duration: 00m 41s)
  • 18:08 ebernhardson: restart elasticsaerch on relforge100[12] for new test version of ltr plugin
  • 14:12 hashar: European SWAT completed
  • 14:11 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Import source on bd.wikimedia.org T154990 + Turn of patrolling on ruwiki T154285 (duration: 00m 42s)
  • 14:10 hashar@tin: Synchronized composer.json: build: Update PHPUnit from 3.7 to 4.8, add phplint to composer-test - T85947 (duration: 00m 45s)
  • 14:09 hashar@tin: Synchronized composer.lock: build: Update PHPUnit from 3.7 to 4.8, add phplint to composer-test - T85947 (duration: 00m 55s)
  • 14:06 hashar: scap pull on terbium
  • 02:35 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Jan 11 02:35:46 UTC 2017 (duration 4m 31s)
  • 02:31 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.7) (duration: 10m 47s)
  • 00:39 _joe_: restart hhvm on mw1182, stuck on HPHP::Treadmill::getAgeOldestRequest

2017-01-10

  • 23:09 hoo: Ran DELETE FROM wbc_entity_usage WHERE eu_row_id IN(1714177, 1714178, 1714179, 1714180, 1714181, 1714182, 1714183, 1714184, 3914375); on s5 master (T147630)
  • 22:53 demon@tin: Synchronized php-1.29.0-wmf.7/extensions/VisualEditor/ApiVisualEditor.php: T154962 logspam (duration: 00m 41s)
  • 22:46 mutante: gerrit restarting for config change 331553
  • 22:13 reedy@tin: Synchronized php-1.29.0-wmf.7/includes/registration/ExtensionRegistry.php: (no message) (duration: 00m 43s)
  • 22:12 reedy@tin: Synchronized php-1.29.0-wmf.7/extensions/Wikidata: (no message) (duration: 02m 21s)
  • 22:06 demon@tin: Synchronized php-1.29.0-wmf.7/includes/libs/objectcache/WANObjectCache.php: Silence obnoxious replag errors (duration: 00m 42s)
  • 21:33 eileen2: civicrm updated from b26844d to af8d735
  • 21:22 mutante: cp3048 - labservices1001 - ran puppet, in this case it wasn't about gerrit, but recovered too
  • 21:15 mutante: sca2004, labsdb1003 - ran puppet (they wanted to git clone during gerrit restart)
  • 21:04 mutante: gerrit restarting for config change 49993 (T40114)
  • 20:16 dereckson@tin: Synchronized php-1.29.0-wmf.7/extensions/SemanticMediaWiki: Remove deprecated function usages (T147924) (duration: 00m 49s)
  • 20:14 dereckson@tin: Synchronized php-1.29.0-wmf.7/extensions/UploadWizard/resources/transports/mw.FormDataTransport.js: mw.FormDataTransport: Don't remove Unicode characters from temp filename (T155039) (duration: 00m 41s)
  • 19:49 demon@tin: Synchronized scap/plugins/prep.py: another no-op (duration: 00m 41s)
  • 19:39 demon@tin: Synchronized scap/plugins/prep.py: prod no-op, for completeness (duration: 00m 40s)
  • 18:41 legoktm: re-attached User:Fuu5tgsrygr / T154983
  • 04:30 dereckson@tin: Synchronized wmf-config/throttle.php: Fix Dayanand College Solapur event throttle rule (T154312) (duration: 00m 44s)
  • 02:26 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Jan 10 02:26:47 UTC 2017 (duration 4m 22s)
  • 02:22 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.7) (duration: 07m 46s)
  • 01:34 reedy@tin: Synchronized rpc/RunJobs.php: revert (duration: 00m 40s)
  • 01:32 reedy@tin: Synchronized w: revert 0a2a096 (duration: 00m 40s)
  • 01:31 reedy@tin: Synchronized multiversion: revert 0a2a096 (duration: 00m 56s)
  • 01:01 Dereckson: Updated articles count on pl.wikisource: 491 100 (T154711)
  • 00:59 Dereckson: Fixed links with namespaceDupes on pl.wikisource
  • 00:58 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Add Collection namespace to the Polish Wikisource (T154711) (duration: 00m 41s)

2017-01-09

  • 23:38 reedy@tin: Synchronized wmf-config/CommonSettings.php: wfLoadExtension (duration: 00m 40s)
  • 23:34 reedy@tin: Synchronized wmf-config/extension-list: More to extension.json (duration: 00m 40s)
  • 23:29 demon@tin: Synchronized multiversion: Final batch of MWVersion cleanup (in song form) (duration: 00m 56s)
  • 23:28 demon@tin: Synchronized rpc/RunJobs.php: More cleanup songs (duration: 00m 40s)
  • 23:28 mutante: ganglia web - replacing SSL cert with Letsencrypt
  • 23:26 demon@tin: Synchronized w: Cleanup cleanup everybody do your share (duration: 00m 40s)
  • 23:25 demon@tin: Synchronized multiversion/MWMultiVersion.php: Cleanup cleanup everybody everywhere (duration: 00m 40s)
  • 23:09 reedy@tin: Synchronized wmf-config/CommonSettings-labs.php: T154927 (duration: 00m 41s)
  • 23:08 reedy@tin: Synchronized wmf-config/InitialiseSettings-labs.php: T154927 (duration: 00m 42s)
  • 23:07 reedy@tin: Synchronized wmf-config/extension-list-labs: T154927 (duration: 00m 41s)
  • 22:53 robh: updating lists.w.o to use LE cert
  • 21:50 akosiaris: service restart zotero on sca1003, sca1004. Zotero OOMed again as usual
  • 19:49 robh: updating librenms.wikimedia.org cert, netmon1001 only system affected
  • 18:16 paravoid: rebooting and powercycling mira, CPU frequency throttled, suspecting firmware bug
  • 17:51 hoo: Updated the Wikidata property suggester with data from last Monday's JSON dump and applied the T132839 workarounds
  • 15:35 zeljkof: finished EU SWAT!
  • 15:33 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add en.wikinews and es.wikinews as import source in testwiki (T154879) (duration: 02m 38s)
  • 15:26 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable import from cswiki to arbcom_cswiki (T154799) (duration: 02m 38s)
  • 15:10 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add DW alias for NS_PROJECT_TALK in frwiki (T153952) (duration: 02m 36s)
  • 15:04 zeljkof: extending EU SWAT, tree more patches left to deploy
  • 15:00 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: [throttle] Lift for 2017-01-10/12 + minor cleanup (T154312) (duration: 02m 36s)
  • 14:52 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Extension:Babel s category on cswikiversity (T67211) (duration: 02m 36s)
  • 14:39 zfilipin@tin: Synchronized static/images/project-logos: SWAT: Add HD logos for multiple projects (T150618) (duration: 02m 36s)
  • 14:25 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add digitalmedia.fws.gov to the whitelist (T154671) (duration: 02m 38s)
  • 10:38 akosiaris: restart nginx and rcstream on rcs1001.eqiad.wmnet to debug issue with prematurely closed connections and 502 returned to clients. No change witnessed.
  • 07:12 ebernhardson: restart elasticsearch on relforge100[12] to adjust ltr logging settings
  • 02:45 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Jan 9 02:45:57 UTC 2017 (duration 4m 36s)
  • 02:41 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.7) (duration: 27m 03s)

2017-01-08

  • 02:27 l10nupdate@tin: ResourceLoader cache refresh completed at Sun Jan 8 02:27:29 UTC 2017 (duration 4m 40s)
  • 02:22 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.7) (duration: 07m 33s)

2017-01-07

  • 19:07 ejegg: disabled dedupe civicrm contacts
  • afk: re-enabled dedupe civicrm contacts
  • afk: disabled dedupe civicrm contacts
  • 13:31 dcausse: elastic@codfw removing/readding replicas for viwiki_general and zhwiki_content (affected by something similar to https://github.com/elastic/elasticsearch/issues/12661) - T154765
  • 11:35 _joe_: from medelevium
  • 11:35 _joe_: restarted apache/otrs, removed a 8 gb error.log
  • 05:11 Dereckson: Update statistics count on so.wikipedia (T154833)
  • 02:35 l10nupdate@tin: ResourceLoader cache refresh completed at Sat Jan 7 02:35:56 UTC 2017 (duration 5m 20s)
  • 02:30 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.7) (duration: 11m 08s)

2017-01-06

  • 23:08 godog: force puppet run on cache_upload in eqiad to switch thumbs back from codfw
  • 22:38 demon@tin: Synchronized multiversion: updateBranchPointers consolidation (duration: 00m 56s)
  • 20:14 demon@tin: Synchronized w: Dropping old entry point (duration: 00m 41s)
  • 19:35 demon@tin: Synchronized php-1.29.0-wmf.7/extensions/UploadWizard/resources: I32e0b8 (duration: 00m 40s)
  • 19:29 demon@tin: Synchronized php-1.29.0-wmf.7/extensions/UploadWizard/resources/mw.UploadWizard.js: I32e0b8 (duration: 00m 59s)
  • 19:17 ebernhardson: restarting elasticsearch on relforge100[12] to test new search-ltr plugin
  • 19:07 ostriches: gerrit: Started full reindex of all changes, should be background but will be watching
  • 18:59 mutante: gerrit restarting for config change 308753 - will be back in seconds
  • 18:13 mutante: mw1205 - restarted hhvm
  • 18:08 godog: force puppet run on eqiad cache_upload to switch thumbs to codfw
  • 16:17 godog: bounce swift-proxy on ms-fe100[123] leave ms-fe1004 for investigation
  • 16:14 hashar: Restarting Nodepool
  • 16:05 ema: wiping codfw caches T154758
  • 15:29 cmjohnson1: powering off mw1239 to reseat DIMM
  • 15:22 papaul: elastic2025-elastic2036 - signing puppet certs, salt-key, initial run
  • 14:53 mark: papaul powercycled asw-a7-codfw 14:50
  • 14:16 reedy@tin: Finished scap: Rebuild message cache for Echo api messages being missing T154110 (duration: 25m 00s)
  • 13:51 reedy@tin: Started scap: Rebuild message cache for Echo api messages being missing T154110
  • 10:00 ariel@tin: Synchronized wmf-config/throttle.php: test, noop (duration: 02m 45s)
  • 09:24 paravoid: asw-a7-codfw is down, serial console unresponsive
  • 09:12 ariel@tin: Synchronized wmf-config/throttle.php: Adjust throttle rule for Maharashtra 'Edit Wikipedia' workshop (VNGIASS) (duration: 02m 46s)
  • 07:08 moritzm: installing crypto++ security updates on trusty hosts
  • 04:37 matt_flaschen: Finished FlowFixInconsistentBoards.php (production mode) on all wikis
  • 04:27 matt_flaschen: Started FlowFixInconsistentBoards.php (production mode) on all wikis
  • 04:10 mutante: Icinga now using Letsencrypt cert and all good
  • 03:42 mutante: icinga - debugging issue with cert change
  • 03:05 papaul: OS instalaltion on elastic2025-elastic2036
  • 02:34 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.7) (duration: 13m 23s)
  • 01:27 bd808: Restarted logstash on logstash1002 (T154732)
  • 01:06 Krinkle: stream.wikimedia.org problems - nginx responds with HTTP 502 Bad Gateway to most requests
  • 01:01 ostriches: gerrit: back up from upgrade
  • 01:00 ostriches: gerrit: down for upgrade
  • 00:13 mutante: analytics1036, ms-fe1003 - ran puppet to fix Icinga
  • 00:11 mutante: carbon - stopping ganglia-monitor-aggregator for good

2017-01-05

  • 23:49 mutante: rolling out exim4 upgrades (DSA 3747-1) on all remaining eqiad (all-eqiad)
  • 23:46 godog: fix root-owned files on puppetmaster1001:/var/lib/git/operations/private/ causing /srv/private post-commit hook to fail
  • 23:18 mutante: switching eqiad ganglia aggregator - running puppet on install1001 - disabling on carbon, re-enabling puppet across eqiad
  • 23:05 mutante: temp disabling puppet on all eqiad hosts via salt - during ganglia aggregator switch
  • 22:51 demon@tin: Synchronized php-1.29.0-wmf.7/extensions/Echo/includes/api: silence api warnings (duration: 02m 46s)
  • 22:40 demon@tin: Synchronized php-1.29.0-wmf.7/extensions/VisualEditor: silence some api warnings (duration: 02m 48s)
  • 22:22 demon@tin: Synchronized php-1.29.0-wmf.7/extensions/CodeReview/api/ApiQueryCodeComments.php: silence some warnings (duration: 02m 46s)
  • 21:59 mutante: rolling out exim4 upgrades (DSA 3747-1) on all remaning ones in codfw (all-codfw)
  • 21:53 ejegg: updated payments wiki from 21ea9bc to 1f9ea80
  • 21:53 mutante: mx1001 - upgrading exim4 packages, exim4-daemon-heavy, forcing puppet run
  • 21:35 mutante: mx2001 - upgrading exim4 packages, daemon-heavey, forcing puppet run
  • 21:22 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.29.0-wmf.7
  • 20:58 mutante: mendelevium (OTRS) - upgrade exim4 packages, force puppet run
  • 20:45 mutante: iridium (phabricator) - upgrade exim4 packages, force puppet run
  • 20:29 demon@tin: Synchronized docroot/wikipedia.org: removing junk 15 stuff (duration: 04m 50s)
  • 20:18 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.29.0-wmf.7
  • 20:07 demon@tin: Synchronized docroot: Ok last docroot thing for today I promise (duration: 05m 00s)
  • 19:52 demon@tin: Synchronized docroot: Final bit of this round of docroot cleanup (duration: 05m 00s)
  • 19:44 thcipriani@tin: Finished scap: SemanticForms l10n cache rebuild for 1.29.0-wmf.7 (duration: 39m 21s)
  • 19:35 mutante: mira - upgrade exim, python-requests, linux-image
  • 19:29 mutante: rolled out exim4 upgrades (DSA 3747-1) on contint1001/2001, dbmonitor1001/2001, tungsten, seaborgium, hassium, labstore*
  • 19:05 thcipriani@tin: Started scap: SemanticForms l10n cache rebuild for 1.29.0-wmf.7
  • 18:34 mutante: fermium (lists server) - upgrading exim packages, exim4-daemon-heavy, forcing puppet run
  • 18:29 yurik@tin: Finished deploy [graphoid/deploy@d20b00e]: (no message) (duration: 03m 11s)
  • 18:26 yurik@tin: Starting deploy [graphoid/deploy@d20b00e]: (no message)
  • 18:25 arlolra: Updated Parsoid to 974dd5b3 (T143183, T102134, T113044)
  • 18:25 mutante: rolling out exim4 upgrades (DSA 3747-1) on prometheus, mwlog, pollux, labmon, lithium, hassaleh, dubnium, graphite1002, tin, serpens, bromine, dataset1001
  • 18:16 arlolra@tin: Finished deploy [parsoid/deploy@465f9c4]: Updating Parsoid to 974dd5b3 (duration: 10m 46s)
  • 18:15 mutante: rolling out exim4 upgrades (DSA 3747-1) on puppetmaster, yubiauth, oresrdb, (oresrdb1001 - Unknown installation error.. eh.. this is new)
  • 18:05 arlolra@tin: Starting deploy [parsoid/deploy@465f9c4]: Updating Parsoid to 974dd5b3
  • 17:57 robh: shutting down mw2075-2089 for decom per T154621
  • 17:35 robh: diabling puppet on mw2075-2089 to decommission them today.
  • 17:15 mutante: rolling out exim4 upgrades (DSA 3747-1) on ruthenium, einsteinium (icinga), etherpad1001, rhodium, | einsteinium: upgrade python packages, kernel | xenon: apt-get autoremove, upgrade python- arcconf, libs...
  • 17:07 mutante: scandium (zuul merger), upgrade exim, python-requests, kernel version
  • 17:06 mutante: rolling out exim4 upgrades (DSA 3747-1) on kraz, wdqs2003, wezen, zosma, tegmen, rutherfordium. upgrade kernel and python-requests on zosma
  • 16:59 mutante: rolling out exim4 upgrades (DSA 3747-1) on all-db-noncore, all-mw-eqiad, restbase-eqiad, kafka-main
  • 16:52 mutante: rolling out exim4 upgrades (DSA 3747-1) on notebook, lvs-canary, lvs, mw-maintenance, all-mw-codfw
  • 16:33 mutante: cobalt upgrading exim packages
  • 16:28 mutante: rolling out exim4 upgrades (DSA 3747-1) on mw-api, swift-fe, swift-be, sca, scb, misc-analytics
  • 16:21 mutante: rolling out exim4 upgrades (DSA 3747-1) on db-core-eqiad, db-misc-servers, videoscaler
  • 15:59 moritzm: upgrading firejail on thumbor servers
  • 15:44 chasemp: labstore1005 systemctl disable create-dbusers
  • 15:01 mobrovac: restbase restarting for firejail upgrade
  • 14:46 moritzm: upgrading firejail on image scalers
  • 14:19 moritzm: upgrading firejail on restbase production hosts
  • 14:15 hashar@tin: Synchronized php-1.29.0-wmf.7/extensions/ContentTranslation: Workaround to fix restoration for truncated section ids - T154279 (duration: 02m 10s)
  • 14:10 moritzm: upgrading firejail on restbase staging hosts
  • 13:46 moritzm: installing audiofile security updates
  • 13:09 mobrovac@tin: Finished deploy [trending-edits/deploy@c5d239b]: Restart for firejail upgrade (duration: 00m 46s)
  • 13:08 mobrovac@tin: Starting deploy [trending-edits/deploy@c5d239b]: Restart for firejail upgrade
  • 13:07 mobrovac@tin: Finished deploy [electron-render/deploy@b2a820e]: Restart for firejail upgrade (duration: 05m 29s)
  • 13:02 mobrovac@tin: Starting deploy [electron-render/deploy@b2a820e]: Restart for firejail upgrade
  • 13:01 mobrovac@tin: Finished deploy [electron-render/deploy@b2a820e]: Restart for firejail upgrade (duration: 03m 40s)
  • 12:57 mobrovac@tin: Starting deploy [electron-render/deploy@b2a820e]: Restart for firejail upgrade
  • 12:45 mobrovac@tin: Finished deploy [mobileapps/deploy@c39bd1f]: Restart for firejail upgrade (duration: 04m 05s)
  • 12:41 mobrovac@tin: Starting deploy [mobileapps/deploy@c39bd1f]: Restart for firejail upgrade
  • 12:41 mobrovac@tin: Finished deploy [mathoid/deploy@79fdd56]: Restart for firejail upgrade (duration: 00m 41s)
  • 12:40 mobrovac@tin: Starting deploy [mathoid/deploy@79fdd56]: Restart for firejail upgrade
  • 12:40 mobrovac@tin: Finished deploy [graphoid/deploy@151f26c]: Restart for firejail upgrade (duration: 00m 43s)
  • 12:39 mobrovac@tin: Starting deploy [graphoid/deploy@151f26c]: Restart for firejail upgrade
  • 12:38 mobrovac@tin: Finished deploy [cxserver/deploy@0279029]: Restart for firejail upgrade (duration: 00m 39s)
  • 12:37 mobrovac@tin: Starting deploy [cxserver/deploy@0279029]: Restart for firejail upgrade
  • 12:36 mobrovac@tin: Finished deploy [citoid/deploy@da96f4b]: (no message) (duration: 01m 06s)
  • 12:35 mobrovac@tin: Starting deploy [citoid/deploy@da96f4b]: (no message)
  • 12:24 moritzm: installing firejail security updates on scb
  • 11:13 akosiaris: rebooting bast3001, T154603
  • 11:11 moritzm: uploaded firejail 0.9.44+wmf2 for jessie-wikimedia to carbon
  • 07:54 elukey: chown www-data:www-data all the root:adm hhvm log files on mw eqiad hosts (T132324)
  • 07:11 marostegui: Compressing revision tables across all the wikis - db1038 - T154465
  • 07:09 marostegui: Compressing pagelinks tables across all the wikis - db1044 - T153826
  • 07:08 marostegui: Compressing revision tables across all the wikis - db1015 - T153739
  • 06:15 bd808: sudo -u l10nupdate rm /var/lock/scap on tin to clean up lock left by bad l10nupdate locking attempt
  • 05:49 mutante: rolling out exim4 upgrades (DSA 3747-1) on swift-fe-codfw, swift-be-codfw, ALL remaining mw
  • 05:44 mutante: rolling out exim4 upgrades (DSA 3747-1) on db-core-codfw, etcd, graphite, kafka-analytics-canary, kafka-analytics, logstash
  • 05:39 mutante: rolling out exim4 upgrades (DSA 3747-1) on prometheus, aqs, db-es
  • 05:35 mutante: rolling out exim4 upgrades (DSA 3747-1) on cp-eqiad, memcached-eqiad
  • 01:48 mutante: rolling out exim4 upgrades (DSA 3747-1) on redis-codfw (rdb2005 needed manual) and all of dc-ulsfo, dc-esams
  • 01:41 mutante: rolling out exim4 upgrades (DSA 3747-1) on ganeti, cp-ulsfo, wdqs, thumbor, db-es-codfw
  • 01:36 mutante: rolling out exim4 upgrades (DSA 3747-1) on parsoid, maps, cp-esams
  • 01:32 mutante: servermon - after the next update by cron - package data is back
  • 01:13 mattflaschen@tin: Synchronized wmf-config/InitialiseSettings.php: Disable NewUserMessage on gomwiki (duration: 00m 41s)
  • 01:10 robh@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw2090.codfw.wmnet
  • 01:06 demon@tin: Synchronized scap/plugins: (no message) (duration: 00m 40s)
  • 01:04 mattflaschen@tin: Synchronized php-1.29.0-wmf.7/extensions/Flow: Flow script to add more troubleshooting information to a maintenance script (duration: 00m 56s)
  • 00:57 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2090.codfw.wmnet
  • 00:57 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2089.codfw.wmnet
  • 00:57 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2088.codfw.wmnet
  • 00:57 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2087.codfw.wmnet
  • 00:57 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2086.codfw.wmnet
  • 00:56 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2085.codfw.wmnet
  • 00:55 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2084.codfw.wmnet
  • 00:55 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2083.codfw.wmnet
  • 00:55 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2082.codfw.wmnet
  • 00:55 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2081.codfw.wmnet
  • 00:54 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2080.codfw.wmnet
  • 00:54 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2080.codfw.wmnet
  • 00:52 mattflaschen@tin: Synchronized php-1.29.0-wmf.6/extensions/Flow: Two Flow fixes related to production database/content inconsistencies. (duration: 00m 59s)
  • 00:42 mutante: phab2001 - same chmod 751 on exim4 dirs that i manually did on krypton is done by puppet here, fully automatic, not sure why krypton was a one-off
  • 00:41 mutante: planet1001/2001, phab2001 - upgrade exim4, exim4-daemon-heavy
  • 00:37 mutante: servermon - weird behaviour in the "pending package upgrades" list? exim4 package was shown as pending on lots of hosts, after next upgrade it disppears from list, even though about half the servers should still be listed
  • 00:23 mutante: krypton - chmod 751 /var/spool/exim4/ to fix Icinga alerts about unaccesible tmpfs (nagios user could not access), it was 751 on other hosts like ununpentium
  • 00:17 mutante: krypton - stop exim, umount orphaned "scan" tmpfs (there is no clamav here)
  • 00:00 mutante: gerrit slowdown reported around 23:55 UTC, was back to normal after 2 minutes (T148478) - attaching latest jvm_gc log
  • 00:00 aaron@tin: Synchronized wmf-config/logging.php: No-op sync of 7e103f2 (duration: 00m 42s)

2017-01-04

  • 23:55 mutante: krypton - chown Debian-exim:Debian-exim /var/spool/exim4/scan/ to fix Icinga-reported DISK issue - wrong permissions - see puppet/modules/exim4/manifests/init.pp line 57 ff "catch-22 with Puppet vs. package"
  • 23:20 mutante: rolled out exim4 upgrades (DSA 3747-1) on memcached-canary, memcached-codfw, restbase-codfw, cp-codfw
  • 23:11 aaron@tin: Synchronized wmf-config/logging.php: Include DB shard as a logstash column (duration: 00m 41s)
  • 23:09 mutante: rolling out exim4 upgrades (DSA 3747-1) on memcached-canary, memcached-codfw, restbase-codfw
  • 23:06 mutante: rolling out exim4 upgrades (DSA 3747-1) on mw-eqiad
  • 22:52 robh: all my server depools and decoms for the mw range are on T154621
  • 22:51 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2078.codfw.wmnet
  • 22:51 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2077.codfw.wmnet
  • 22:51 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2076.codfw.wmnet
  • 22:50 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2075.codfw.wmnet
  • 22:49 godog: rename / reimage restbase-test1* to restbase-dev1*
  • 22:46 bblack: TLS: unified certificates in esams switching to digicert
  • 22:19 mutante: rolling out exim4 upgrades (DSA 3747-1) on mw-codfw
  • 21:55 Krinkle: mwscript deleteEqualMessages.php --wiki nowikinews (T45917)
  • 21:55 Krinkle: mwscript deleteEqualMessages.php --wiki nowiki (T45917)
  • 21:48 otto@tin: Finished deploy [eventstreams/deploy@a103be2]: (no message) (duration: 01m 32s)
  • 21:46 otto@tin: Starting deploy [eventstreams/deploy@a103be2]: (no message)
  • 21:24 bsitzmann@tin: Finished deploy [mobileapps/deploy@c39bd1f]: Update mobileapps to b43c5d6 (duration: 02m 55s)
  • 21:21 bsitzmann@tin: Starting deploy [mobileapps/deploy@c39bd1f]: Update mobileapps to b43c5d6
  • 21:15 smalyshev@tin: Finished deploy [wdqs/wdqs@3762556]: (no message) (duration: 02m 40s)
  • 21:13 smalyshev@tin: Starting deploy [wdqs/wdqs@3762556]: (no message)
  • 21:01 smalyshev@tin: Finished deploy [wdqs/wdqs@3762556]: (no message) (duration: 00m 59s)
  • 21:00 smalyshev@tin: Starting deploy [wdqs/wdqs@3762556]: (no message)
  • 20:49 madhuvishy: adding temporary IP tables rule on labservices1001 to drop traffic from toolchecker for tests (T152369)
  • 20:39 mutante: rolling out exim4 upgrades (DSA 3747-1) on mw-canary hosts
  • 20:36 mutante: rolling out exim4 upgrades (DSA 3747-1) on stat* and kubernetes hosts
  • 20:05 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.29.0-wmf.7
  • 19:09 thcipriani@tin: Synchronized portals: SWAT: Bumping portal to master T128546 (duration: 00m 42s)
  • 19:08 thcipriani@tin: Synchronized portals/prod/wikipedia.org/assets: SWAT: Bumping portal to master T128546 (duration: 00m 43s)
  • 18:30 mutante: rolling out exim4 upgrades (DSA 3747-1) on misc servers
  • 18:29 ejegg: enabled payment processor audit parser jobs
  • 17:47 matt_flaschen: Ran manual DB update to officewiki for T153320.
  • 15:20 zeljkof: EU SWAT finished
  • 15:19 hashar@tin: Synchronized wmf-config/throttle.php: (no message) (duration: 00m 41s)
  • 15:03 zeljkof: extending eu swat until 330392 is merged
  • 14:56 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: Throttle rules for 2017-01-06/07, tewiki (T154568) (duration: 00m 40s)
  • 14:32 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set valid content language for Norwegian wikis (T126146) (duration: 00m 41s)
  • 14:20 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert $wgMFEEditorOptions[anonymousEditing] = false for kowiki (T119823) (duration: 00m 41s)
  • 11:18 jynus: continuing maintenance on db1035 (mysql replication stopped)
  • 10:12 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2060 - T154031 (duration: 00m 47s)
  • 07:43 marostegui: Compressing tables on db1015 - T153739
  • 07:24 marostegui: Compressing more tables on db1044 - T153826
  • 03:10 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Jan 4 03:10:08 UTC 2017 (duration 5m 33s)
  • 03:05 bd808@tin: Synchronized wmf-config/throttle.php: Add throttle rules for January 2017 events in Maharashtra (T154312) (duration: 00m 42s)
  • 03:04 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.7) (duration: 13m 05s)
  • 02:34 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.6) (duration: 11m 17s)
  • 01:24 maxsem@tin: Synchronized php-1.29.0-wmf.6/extensions/CentralAuth: https://gerrit.wikimedia.org/r/#/c/330345/ (duration: 00m 44s)
  • 01:03 maxsem@tin: Synchronized php-1.29.0-wmf.7/extensions/Flow: https://gerrit.wikimedia.org/r/#/c/330338/ (duration: 00m 58s)
  • 00:42 foks: removed 2fa for account per T154171
  • 00:33 maxsem@tin: Synchronized wmf-config/unitConversionConfig.json: https://gerrit.wikimedia.org/r/#/c/327907/5 (duration: 00m 40s)
  • 00:13 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/330264/3 (duration: 00m 41s)
  • 00:06 eileen: update civicrm from f78c894 to b26844d

2017-01-03

  • 23:57 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.29.0-wmf.7
  • 23:48 thcipriani@tin: Finished scap: testwiki to php-1.29.0-wmf.7 and rebuild l10n cache (duration: 49m 50s)
  • 23:19 ostriches: gerrit: quick restart of services
  • 22:58 thcipriani@tin: Started scap: testwiki to php-1.29.0-wmf.7 and rebuild l10n cache
  • 22:55 chasemp: iptables block of tools-checker-01 to debug DNS SPoF
  • 22:44 maxsem@tin: Synchronized php-1.29.0-wmf.7/extensions/Kartographer: https://gerrit.wikimedia.org/r/#/c/330321/ (duration: 00m 42s)
  • 22:38 maxsem@tin: Synchronized php-1.29.0-wmf.6/extensions/Kartographer: https://gerrit.wikimedia.org/r/#/c/330322/ (duration: 00m 42s)
  • 22:11 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: mapframe on fr: and fi: https://gerrit.wikimedia.org/r/#/c/330311/3 (duration: 00m 41s)
  • 21:00 gehel@tin: Finished deploy [wdqs/wdqs@a25d3aa]: (no message) (duration: 00m 55s)
  • 20:59 gehel@tin: Starting deploy [wdqs/wdqs@a25d3aa]: (no message)
  • 20:57 otto@tin: Finished deploy [eventstreams/deploy@9095b4e]: (no message) (duration: 11m 15s)
  • 20:45 otto@tin: Starting deploy [eventstreams/deploy@9095b4e]: (no message)
  • 20:18 gehel@tin: Finished deploy [wdqs/wdqs@cd7215c]: (no message) (duration: 04m 54s)
  • 20:13 gehel@tin: Starting deploy [wdqs/wdqs@cd7215c]: (no message)
  • 19:56 demon@tin: Synchronized multiversion/updateBranchPointers: Removing unused --dry-run option (duration: 00m 40s)
  • 19:52 thcipriani@tin: Synchronized multiversion: SWAT: Remove checkoutMediaWiki (duration: 00m 58s)
  • 19:13 thcipriani@tin: Synchronized wmf-config/Wikibase-production.php: SWAT: Add badge for "digitaldocument" in Wikibase T153186 (duration: 01m 33s)
  • 18:01 moritzm: installing fontconfig security updates
  • 17:44 mutante: terbium - Notice: /Stage[main]/Mediawiki::Maintenance::Generatecaptcha/Cron[generatecaptcha]/ensure: created(T150029)
  • 17:24 thcipriani: starting branch cut for 1.29.0-wmf.7
  • 17:04 mutante: iridium (phab) - apt-get clean ; find /var/log/account/ -mtime +10 -delete ; find /var/log/atop/ -mtime +10 -delete (T154407)
  • 16:59 thcipriani: enable l10nupdate cron post deployment-freeze
  • 16:49 mutante: iridium (phab) - reduce process accounting from 30 days to 10 days to save disk space used by /var/log/account, run /etc/cron.daily/acct (T154407)
  • 16:37 bd808: Updated scholarships to 1690808 on krypton; needed help from _joe_ to make trebuchet work
  • 15:58 _joe_: rolling restart of restbase on the production cluster
  • 15:49 _joe_: rolling restart of restbase on the test cluster
  • 14:53 akosiaris: reenabling ntpd on the restbase in eqiad
  • 14:43 gehel: upgrade liblogstash-gelf on elastic* - T150408
  • 14:35 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: [2/2] Add HD logos for multiple wikis - T150618 (duration: 00m 40s)
  • 14:33 hashar@tin: Synchronized static/images/project-logos: [1/2] Add HD logos for multiple wikis - T150618 (duration: 00m 40s)
  • 14:29 akosiaris: reenabling ntpd on the rest of the boxes. Leaving restbase only out for last
  • 14:21 hashar@tin: Synchronized wmf-config/throttle.php: New rules + remove obsolete rules T154245 (duration: 00m 40s)
  • 14:19 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable mapframe for nowiki T154021 (duration: 00m 39s)
  • 14:17 akosiaris: reenabling ntpd on aqs boxes
  • 14:17 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Set sortPrepend for gdwiki T153900 (duration: 00m 40s)
  • 14:16 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable SandboxLink on ruwiki T153855 (duration: 00m 40s)
  • 14:10 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable subpages in NS0 for arbcom_cswiki - T154247 (duration: 00m 40s)
  • 14:08 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Add new page protection level on etwiki - T153465 (duration: 00m 53s)
  • 14:08 akosiaris: reenabling ntpd on maps, maps-test boxes
  • 14:07 akosiaris: reenabling ntpd on kafka boxes
  • 13:37 akosiaris: reenabling ntpd on elastic boxes
  • 13:27 akosiaris: reenabling ntpd on rdb boxes
  • 13:22 akosiaris: reenabling ntpd on conf boxes
  • 13:22 akosiaris: reenabling ntpd on es* boxes
  • 13:15 akosiaris: reenabling ntpd on scb* boxes
  • 13:09 moritzm: uploaded Linux 4.4.39 for jessie-wikimedia to carbon
  • 13:09 akosiaris: reenabling ntpd on mc* boxes
  • 13:01 akosiaris: reenabling ntpd on ms-fe boxes
  • 13:00 moritzm: installing libgd security updates
  • 12:55 akosiaris: reenabling ntpd on ms-be boxes
  • 12:52 akosiaris: reenabling ntpd on lvs boxes
  • 12:48 akosiaris: reenabling ntpd on analytics boxes
  • 12:41 moritzm: installing python security updates
  • 12:12 moritzm: installing squid security updates
  • 12:07 akosiaris: reenabling ntpd on pc eqiad & codfw boxes
  • 12:06 akosiaris: reenabling ntpd on ganeti eqiad & codfw boxes
  • 11:54 akosiaris: reenabling ntpd on wtp eqiad boxes
  • 11:52 akosiaris: reenabling ntpd on logstash eqiad boxes
  • 11:51 akosiaris: reenabling ntpd on db* eqiad boxes
  • 11:46 akosiaris: reenabling ntpd on cobalt (gerrit)
  • 11:32 moritzm: installing tar security updates on trusty hosts
  • 11:27 gehel: upgrade liblogstash-gelf on deployment-elastic* - T150408
  • 11:16 gehel: upgrade lilogstash-gelf on relforge - T150408
  • 11:13 akosiaris: reenabling ntpd on db* codfw boxes
  • 11:07 akosiaris: reenabling ntpd on wtp codfw boxes
  • 10:59 akosiaris: reenabling ntpd on mw eqiad boxes
  • 10:53 jynus: stopping mysql replication on db1035 (depooled)
  • 10:50 akosiaris: reenabling ntpd on mw codfw boxes
  • 10:44 akosiaris: reenabling ntpd on eqiad cp boxes
  • 10:39 akosiaris: reenabling ntpd on codfw cp boxes
  • 10:14 akosiaris: start enabling ntpd again across the fleet. Starting with cp boxes on ulsfo and esams
  • 09:23 marostegui: stop MySQL dbstore2002 for maintenance - T151552
  • 09:10 marostegui: stop MySQL dbstore2001 for maintenance - T151552
  • 08:21 marostegui: Run optimize table on db1038 on all the revision,templatelinks and pagelinks tables - T154465
  • 08:00 marostegui: Run optimize table on a few large tables - db1015 - T153739
  • 07:58 elukey: chown www-data:www-data all the root:adm hhvm log files on mw codfw hosts (T132324)
  • 07:54 marostegui: Run optimize table on a few large tables - db1044 - T153826
  • 07:30 marostegui: Stop mysql db2048 and db2034 for maintenance - https://phabricator.wikimedia.org/T149553

2017-01-02

  • 23:09 hoo: Removed 2fa from an account, per T154450
  • 17:20 ema: iridium: removed /var/log/account/pacct.2[0-9].gz to free up more disk space
  • 16:05 ema: removing old kernels and kernel headers from iridium to free up some disk space
  • 13:24 elukey: powercycled mw1280, not pingable and mgmt console frozen

2017-01-01

  • 02:23 chasemp: labservices1001 'racadm serveraction hardreset'
  • 02:23 godog: reboot labservices1001, unresponsive on console and MCE/temperature alerts found on lithium
  • 00:56 filippo@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1286.eqiad.wmnet,service=apache2
  • 00:55 bd808: Restarted logstash on logstash1001 (T154388)
  • 00:46 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw1286.eqiad.wmnet
  • 00:27 godog: dump core file and restart varnish-frontend on cp2026


Archives