Jump to content
Server Admin Log/Archive 33
18:22 ejegg: disabled failing silverpop data fetch jobs
18:31 mutante: gerrit: restart service to apply change from Dec 20th, avoid logspam due to unapplied config change
15:38 ariel@tin: Finished deploy [dumps/dumps@51d4fd6]: enable pageslogging dump in parallel jobs (duration: 00m 02s)
15:38 ariel@tin: Started deploy [dumps/dumps@51d4fd6]: enable pageslogging dump in parallel jobs
14:46 apergos: restarted puppetdb on nitrogen, puppetdb refusing to hand out facts again
08:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1110 (duration: 00m 52s)
08:19 marostegui: Stop replication in sync on dbstore1002.s5 and db1110
08:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1110 to reimport dewiki.imagelinks on dbstore1002 (duration: 00m 52s)
07:43 marostegui: Fixing dbstore1002 dewiki.imagelinks
14:18 marostegui: Power cycle pc2005 as it is down
19:54 godog: power reset ms-be1033
19:47 godog: reboot ms-be1033 - T183724
17:00 twentyafterfour: restarting apache on phab1001
15:56 mobrovac@tin: Started restart [electron-render/deploy@94d27d7]: Bounce Electron, stuck - T174916
16:33 mobrovac@tin: Finished deploy [mathoid/deploy@63b2ddc]: Bring back Mathoid in codfw to v0.6.5 in sync with eqiad - T179419 T172767 (duration: 01m 44s)
16:31 mobrovac@tin: Started deploy [mathoid/deploy@63b2ddc]: Bring back Mathoid in codfw to v0.6.5 in sync with eqiad - T179419 T172767
16:27 mutante: re-generated DNS zones to add new language lfn (Lingua Franca Nova) (T183561 )
12:00 Amir1: ladsgroup@terbium:~$ mwscript extensions/Cognate/maintenance/populateCognatePages.php --wiki hifwiktionary (T180785 )
11:59 Amir1: ladsgroup@terbium:~$ mwscript extensions/Cognate/maintenance/populateCognateSites.php --wiki aawiktionary --site-group wiktionary (T180785 )
11:05 ariel@tin: Finished deploy [dumps/dumps@8c44d0a]: re-enable file size reporting for dump files still being written (duration: 00m 02s)
11:05 ariel@tin: Started deploy [dumps/dumps@8c44d0a]: re-enable file size reporting for dump files still being written
08:34 akosiaris: migrate logstash1007 off ganeti1006 T181121
08:21 akosiaris: migrate logstash1007 off ganeti1006
03:52 madhuvishy: restart pdfrender on scb1002
03:50 madhuvishy: restart pdfrender on scb1004
03:32 madhuvishy: restart pdfrender on scb1003 following icinga page
03:28 madhuvishy: restart pdfrender on scb1001 following icinga page
16:10 _joe_: restarted pdfrenderer on scb1002
18:31 mobrovac@tin: Finished deploy [mathoid/deploy@7c5f8e2]: Better handling of chem rules (codfw only) - T183557 (duration: 02m 45s)
18:28 mobrovac@tin: Started deploy [mathoid/deploy@7c5f8e2]: Better handling of chem rules (codfw only) - T183557
16:42 demon@tin: Pruned MediaWiki: 1.31.0-wmf.11 [keeping static files] (duration: 01m 16s)
16:05 demon@tin: Synchronized README: forcing co-master sync (duration: 00m 51s)
15:56 demon@tin: Synchronized scap/plugins/clean.py: no-op (duration: 00m 51s)
15:52 demon@tin: Pruned MediaWiki: 1.31.0-wmf.10 [keeping static files] (duration: 01m 18s)
15:50 demon@tin: Pruned MediaWiki: 1.31.0-wmf.10 [keeping static files] (duration: 01m 47s)
14:48 bblack: esams TLS cert switch from digicert-2016 to digicert-2017
14:18 mobrovac@tin: Finished deploy [mathoid/deploy@6c29c09]: Update Mathoid to v0.7.0 in CODFW only to prefill storage, take 2 - T179419 T172767 (duration: 04m 23s)
14:13 mobrovac@tin: Started deploy [mathoid/deploy@6c29c09]: Update Mathoid to v0.7.0 in CODFW only to prefill storage, take 2 - T179419 T172767
14:05 mobrovac@tin: Finished deploy [mathoid/deploy@7f39b71]: Update Mathoid to v0.7.0 in CODFW only to prefill storage - T179419 T172767 (duration: 02m 57s)
14:02 mobrovac@tin: Started deploy [mathoid/deploy@7f39b71]: Update Mathoid to v0.7.0 in CODFW only to prefill storage - T179419 T172767
13:37 mobrovac: restbase depool restbase2008 for T179419
13:17 bblack: repooling cp4032 - T183176
06:44 jynus: start manual backup of db1029 onto db1056
19:25 robh: cp4032 going offline for memory swap
17:24 jynus: starting manual backup of x1-master onto db1055
17:01 volans: debugging Icinga notes_url (no side effect expected but logging it in case there will be) T170353
16:56 volans: restarted ircecho to pick up gerrit/399658
15:43 joal@tin: Finished deploy [analytics/refinery@92f9318]: Deploying for new pageview_top_bycountry job (duration: 04m 40s)
15:39 joal@tin: Started deploy [analytics/refinery@92f9318]: Deploying for new pageview_top_bycountry job
12:39 elukey: restart druid historical/broker to apply new jvm settings to druid public workers (druid100[456] - https://gerrit.wikimedia.org/r/399617 )
12:07 elukey@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw1333.*.eqiad.wmnet
11:43 _joe_: refreshing puppet facts on the puppet compilers
10:54 jynus@tin: Synchronized wmf-config/db-eqiad.php: Remove db1055 & db1056 (duration: 00m 51s)
10:52 jynus@tin: Synchronized wmf-config/db-codfw.php: Remove db1055 & db1056 (duration: 00m 51s)
10:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1082 and db1100 original weight - T161294 (duration: 00m 51s)
10:15 elukey: rolling restart of eventbus on kafka* for openssl security updates
10:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1100 and start restoring original weight for db1082 - T161294 (duration: 00m 48s)
09:50 elukey: restart eventbus on kafka2001 for openssl updates
09:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1100 - T161294 (duration: 00m 51s)
09:30 elukey: restart zookeeper on conf100[2,3] for jvm updates - T179943
09:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1100 - T161294 (duration: 00m 51s)
09:20 elukey: restart zookeeper on conf1001 for jvm updates - T179943
08:51 elukey: run kafka preferred-replica-election after maintenance of kafka1023 (fully bootstrapped now)
08:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1100 - T161294 (duration: 00m 51s)
08:34 marostegui: Stop replication in sync on db1100 and dbstore1002 - T161294
08:23 elukey: repool mw1277 after investigation
07:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Slowly repool db1100- T161294 (duration: 00m 52s)
07:21 marostegui: Upgrade MariaDB on db1100
07:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1109 and db1096:3315 - T161294 (duration: 00m 51s)
06:55 _joe_: restarting hhvm on mw1231 and mw1208
06:50 marostegui: Stop replication in sync on dbstore1002 - db1100 - T161294
06:42 marostegui: Stop replication in sync on db1100 - db2052 - T161294
06:36 marostegui: Remove some old files in dbstore1001:/srv/tmp to address the WARNING alert
06:35 marostegui: Stop replication in sync db1100 and db1071 - T161294
06:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1109 - T161294 (duration: 00m 51s)
03:59 mutante: mw1315 kill and restart hhvm | mw1312 stop and start hhvm
03:55 mutante: mw1290 kill and restart hhvm | mw1230 stop and start hhvm
02:20 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.12) (duration: 05m 31s)
01:16 eileen: update CiviCRM civicrm revision changed from 9d6ba74f57 ffa9d7fc7a da6fa4cce9
23:26 twentyafterfour: restarting apache on phab1001 to deploy a hotfix for T144184
21:42 mepps: updated dash from 114131713e b01458b260
21:20 otto@tin: Finished deploy [analytics/refinery@548dad7]: deploying refinery v0.0.56 with JsonRefine fixes to allow Popups schema to be refined. This is a no-op for everything else (duration: 04m 51s)
21:15 otto@tin: Started deploy [analytics/refinery@548dad7]: deploying refinery v0.0.56 with JsonRefine fixes to allow Popups schema to be refined. This is a no-op for everything else
20:31 ebernhardson: T183053 update elasticsearch settings for wikidatawiki_content on codfw to use: index.refresh_interval=5s
20:18 XioNoX: remove local-as from cr2-esams IX4
19:46 XioNoX: remove local-as from cr2-esams IX6
17:57 _joe_: depooling mw1277 for further investigation
17:51 demon@tin: Synchronized wmf-config/InitialiseSettings.php: T172875 (second try, forgot to pull first (duration: 00m 51s)
17:48 elukey: new mw jobrunner in production (mw1334) - T165519
17:47 demon@tin: Synchronized wmf-config/InitialiseSettings.php: T172875 (duration: 00m 51s)
17:46 demon@tin: Synchronized wmf-config/Wikibase-production.php: T180614 (duration: 00m 51s)
17:43 elukey: restart zookeeper on conf2003 for jvm updates - T179943
17:42 elukey: restart hhvm on mw1316
17:38 demon@tin: Synchronized wmf-config/InitialiseSettings.php: sandbox link on Atikamekw 'pedia (duration: 00m 52s)
17:38 elukey: restart hhvm on mw1234
17:32 elukey: restart zookeeper on conf2002 for jvm updates - T179943
17:20 demon@tin: Synchronized wmf-config/CommonSettings.php: more n0-0ps (duration: 00m 52s)
17:16 demon@tin: Synchronized multiversion/submodules.json: no-op (duration: 00m 51s)
17:09 demon@tin: Synchronized README: noop (duration: 00m 51s)
16:27 ema: lvs1003: stop pybal, clean ipvs services, start pybal
16:25 ema: lvs1006: stop pybal, clean ipvs services, start pybal
15:54 moritzm: rolling restart of scb* hosts in eqiad to pick up openssl update
15:47 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1334.eqiad.wmnet
15:38 moritzm: rolling restart of remaining scb* hosts in codfw to pick up openssl update
14:26 ema: start pybal on lvs2003
14:24 ema: stop pybal on lvs2003, clean IPVS table after traffic failover to get rid of trendingedits `TCP 10.2.1.9:6699 wrr`
14:14 moritzm: upgrading openssl on wdqs (along with system service restarts)
14:13 ema: bounce pybal on lvs2006 and clean IPVS table
14:06 elukey: temporarily shutdown kafka on kafka1023 to move some topic partitions on different disk partition (disk space usage alerts)
13:49 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1276.eqiad.wmnet
13:38 elukey@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw1276.eqiad.wmnet
13:38 moritzm: upgrading openssl on wdqs (along with system service restarts)
13:35 moritzm: rolling restart of aqs to pick up openssl security update
13:20 moritzm: upgrading openssl on aqs/druid clusters (along with system service restarts)
13:11 jynus: restart dbproxy1008 to test workaround is working on cold restart
13:01 moritzm: upgrading openssl on hadoop cluster (along with system service restarts)
12:34 marostegui: Enable notifications for db1100 - T161294
12:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 and db1109 - T161294 (duration: 00m 51s)
12:18 moritzm: installing iproute2 bugfix update from stretch point relesae
11:50 godog: create k8s-staging LVs in prometheus/eqiad - T163692
11:37 moritzm: upgrading openssl on elastic* (along with system service restarts)
11:27 akosiaris: repool ganeti1006, rebalance row_A ganeti nodegroup. T181121
11:18 jynus: restart dbproxy1005 to test cold service start
11:11 marostegui: Stop replication in sync on db1100 and db1071 - T161294
10:57 arturo: remove old kernel packages from silver.wikimedia.org to free space
09:56 jynus: restart dbproxy1001 to test cold service start
09:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1105:3311 - T174569 (duration: 00m 51s)
09:24 jynus: disable puppet on dbproxies for gerrit:399359 deployment
08:44 moritzm: powercycling ganeti1005
08:30 moritzm: installing rsync security updates
08:24 jynus: starting reimage of dbproxy1008
08:11 marostegui: Stop replication in sync on db1100 and dbstore1002 - T161294
07:56 jynus: starting reimage of dbproxy1007
07:35 jynus: starting reimage of dbproxy1005
07:24 jynus: upgrading and restarting dbproxy1004
07:09 marostegui: Stop replication in sync on db1096:3315 and db1100 - T161294
07:08 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1096:3315 - T161294 (duration: 00m 51s)
06:53 marostegui: Stop replication in sync on db1101:3318 and db1109 - T161294
06:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1099:3318 depool db1101:3318 - T161294 (duration: 00m 51s)
06:19 marostegui: Deploy schema change on db1105:3311 - T174569
06:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1105:3311 - T174569 (duration: 00m 52s)
06:06 Jamesofur: reset mailman password for tawikisource T183329
02:20 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.12) (duration: 05m 45s)
21:51 XioNoX: removing local-as AS43821 from ams transits - T167840
19:49 mutante: deleted ganglia.wikimedia.org from DNS - webserver was already down since yesterday - not used anymore (T177225 )
19:23 mutante: webperf1001/webperf2001 - rebooting for kernel upgrades (not used yet)
19:18 mutante: gerrit2001 - reboot for kernel upgrade
18:14 moritzm: installing zsh update from stretch point release
17:11 jynus: purging ferm from dbproxy1002, 3, 6, 9, 10 and 11
17:00 moritzm: installing libxv security updates on jessie
16:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1097:3315 - T161294 (duration: 00m 51s)
16:45 marostegui: Defragment s7 databases on db1102 - https://phabricator.wikimedia.org/T172169
16:38 awight@tin: Synchronized wmf-config/CirrusSearch-labs.php: wmf-config/CommonSettings-labs.php wmf-config/db-labs.php wmf-config/InitialiseSettings-labs.php wmf-config/interwiki-labs.php wmf-config/jobqueue-labs.php wmf-config/mc-labs.php wmf-config/mobile-labs.php wmf-config/Wikibase-labs.php Sync out labs config changes (duration: 00m 51s)
16:34 elukey: manually started eventlogging cleaner on db1107 to purge/sanitize data up to 90 days ago (tmux is running for user eventlogcleaner) - T108850
16:22 moritzm: installing ncurses updates from jessie point release
15:26 jgleeson: switched back on donations queue consumer and thank you mailer service
15:25 chasemp: labvirt10[19|20] aptitude install linux-image-4.4.0-81-generic linux-image-extra-4.4.0-81-generic; sudo update-grub; /sbin/reboot T172538
15:10 marostegui: Stop replication in sync on db1109 and db1099:3318 - https://phabricator.wikimedia.org/T161294
15:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1099:3318 and db1109 - T161294 (duration: 00m 51s)
15:09 jgleeson: updated civicrm from e0ee2d189c 9d6ba74f57
15:04 jgleeson: switched off donations queue consumer and thank you mailer service in preparation for new release
15:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1106 - T161294 (duration: 00m 52s)
14:49 moritzm: restarting hhvm on canary app servers to pick up security updates for openssl, icu and libx11
14:44 moritzm: installing request-tracker4 update from jessie point release on ununpentium
14:28 jynus: disabling puppet on dbproxies for 399164 deploy
13:57 moritzm: upgrading pdns-recursor on achernar/acamar to 4.0.4+deb9u3~bpo8+1 (security fix)
13:13 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1317.eqiad.wmnet
13:05 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1318.eqiad.wmnet
13:05 elukey@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw133[0-1].eqiad.wmnet
12:12 hashar: CI: switching mwgate-composer-php70 job from Nodepool to Docker | https://gerrit.wikimedia.org/r/#/c/398921/
11:59 hashar: CI: switching composer-php55 / composer-package-php55 jobs from Nodepool to Docker | https://gerrit.wikimedia.org/r/#/c/398920/
11:29 moritzm: upgrading pdns-recursor on maerlant to 4.0.4+deb9u3~bpo8+1 (security fix)
11:28 hashar@tin: Finished deploy [docker-pkg/deploy@09087ad]: Bumping Jinja2 2.9.6..2.10 (duration: 00m 30s)
11:28 hashar@tin: Started deploy [docker-pkg/deploy@09087ad]: Bumping Jinja2 2.9.6..2.10
11:16 moritzm: uploaded pdns-recursor 4.0.4+deb9u3~bpo8+1 to apt.wikimedia.org
10:54 ariel@tin: Finished deploy [dumps/dumps@2bafffe]: allow dump runs in specified wiki list order, rather than by longest to wait (duration: 00m 02s)
10:54 ariel@tin: Started deploy [dumps/dumps@2bafffe]: allow dump runs in specified wiki list order, rather than by longest to wait
10:52 moritzm: upgrading pdns-recursor on nescio to 4.0.4+deb9u3~bpo8+1 (security fix)
10:47 elukey: restart zookeeper on conf2001 for jvm updates - T179943
10:45 jynus: disabling puppet on dbproxies for 398450 deploy
10:38 godog: rollout updated version of prometheus-nutcracker-exporter
09:12 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1097:3315 - T161294 (duration: 00m 51s)
08:56 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1106 - T161294 (duration: 00m 51s)
08:35 moritzm: reimaging mw1317 (video scaler) to stretch
08:28 marostegui: Stop replication in sync on db2045 and db1109 - T161294
08:21 moritzm: installing openssl security updates
08:05 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw2246.codfw.wmnet
08:05 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw2119.codfw.wmnet
07:00 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1106 - T161294 (duration: 00m 51s)
06:53 mobrovac@tin: Finished deploy [restbase/deploy@2b75a64]: Bug fix: Add the time_to_live config option to the Parsoid module (duration: 04m 26s)
06:51 marostegui: Stop replication in sync on db1106 and db2052 - T161294
06:49 mobrovac@tin: Started deploy [restbase/deploy@2b75a64]: Bug fix: Add the time_to_live config option to the Parsoid module
06:40 marostegui: Stop replication in sync on db1106 and dbstore1002 s5 - T161294
06:29 marostegui: Stop replication in sync on db1100 and db1106 - T161294
06:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1106 - T161294 (duration: 00m 53s)
06:09 marostegui: Deploy schema change on db1065 (s1 sanitarium master) with replication, so some lag will be generated on labs - T174569
05:18 andrewbogott: restarting slapd on seaborgium (in response to ldap complaints on the grid master)
02:24 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.12) (duration: 05m 22s)
00:44 mutante: einsteinium: sudo systemctl restrart ircecho (alias kick-icinga-wm)
22:34 ejegg: updated payments-wiki from f594dfa763 e91db27108
21:00 mutante: uranium - apt-get remove ganglia-webfrontend, apache2
20:53 mutante: ganglia.wikimedia.org shut down just now after a deprecation period - service is out of commission - T177225
20:53 chasemp: reboot labtestvirt2003
20:49 mutante: install1002/2002 - killing all ganglia processes, decoming aggregators
20:48 bawolff@tin: Synchronized php-1.31.0-wmf.12/extensions/TemplateData/TemplateDataBlob.php: T118682 (duration: 00m 52s)
19:49 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp4032.ulsfo.wmnet
18:42 moritzm: installing xml2 updates from stretch point release
18:28 moritzm: installing libxkbcommon updates from stretch point release
18:11 moritzm: installing python updates from stretch point release
17:51 elukey: run kafka preferred-replica-election on the analytics cluster to allow kafka1023 (new node) to become a partition leader
16:23 demon@tin: Pruned MediaWiki: 1.31.0-wmf.11 [keeping static files] (duration: 01m 18s)
16:17 demon@tin: Pruned MediaWiki: 1.31.0-wmf.8 (duration: 04m 58s)
16:15 elukey@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=mw133[0-7].eqiad.wmnet
16:09 thcipriani@tin: Synchronized README: noop sync to test scap 3.7.4-3 (duration: 03m 02s)
16:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 - T174569 (duration: 03m 03s)
15:47 marostegui: Stop MySQL on db1111 to copy its content to db1112 - T180788
15:45 jynus: stop and upgrade db1107 T183123
15:37 marostegui: Stop db1100 and dbstore1002 in sync - T161294
15:28 elukey@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw1329.eqiad.wmnet
15:23 moritzm: uploaded prometheus-blazegraph-exporter, prometheus-wdqs-updater-exporter and prometheus-pdns-exporter to apt.wikimedia.org
15:16 chasemp: reboot labtestvirt2003
14:41 ema: upgrade pinkunicorn to latest jessie point release (8.10) T182656
14:13 elukey: temporarily stopped mysql consumers on eventlog1001 to ease a mysql backup on db1107 - T183123
13:58 jynus: starting one-time backup of eventlogging database on db1107:/srv/backups T183123
13:29 marostegui: Stop replicaiton in sync on db1109 and db2045 - T161294
13:25 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1307.eqiad.wmnet
13:25 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1313.eqiad.wmnet
13:21 jmm@puppetmaster1001: conftool action : set/pooled=no; selector: mw1313.eqiad.wmnet
13:20 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1260.eqiad.wmnet
13:20 jmm@puppetmaster1001: conftool action : set/pooled=no; selector: mw1260.eqiad.wmnet
13:18 marostegui: Stop replication in sync on db1100 and db2052 - T161294
13:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1106 - T161294 (duration: 03m 06s)
13:00 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1106 - T161294 (duration: 03m 03s)
12:58 marostegui: Stop replication in sync on db1106 and db1100 - T161294
12:35 marostegui: Deploy schema change on db1099:331 and db1067 - T174569
12:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1099:3311 - T174569 (duration: 03m 05s)
12:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1066 - T174569 (duration: 03m 06s)
12:20 akosiaris: build scap 3.7.4-3 and upload to jessie-wikimedia, stretch-wikimedia, trusty-wikimedia. T183046 , T182347
11:37 hashar: restarting Jenkins CI to upgrade the monitoring plugin
11:24 _joe_: ran cleanup script on scb* T180384
11:20 mobrovac: stopping the trending edits service - T180384
11:18 moritzm: reimaging mw1307 (video scaler) to stretch
11:08 _joe_: rolling restart of pybal on the low-traffic balancers
11:04 _joe_: disabled notifications for trendingedits.svc T180384
10:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1109 - T161294 (duration: 00m 56s)
10:30 marostegui: Stop replication on db1109 and db2045 in sync - T161294
10:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1109 - T161294 (duration: 00m 56s)
09:47 marostegui: Deploy schema change on db1066 - T174569
09:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1066 - T174569 (duration: 00m 56s)
09:43 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1073 - T174569 (duration: 00m 57s)
09:40 moritzm: installing openssl security updates
09:39 marostegui: Deploy schema change on db1055 (already depooled) - T174569
09:28 gehel: removing initial import datafiles from maps[12]001
08:58 marostegui: Stop replication in sync on db1100 and db2052 - T161294
08:57 elukey: rolling restart of the Yarn nodemanagers (hadoop) on analytics10[456]* to pick up new settings - T182276
08:30 Jamesofur: insert decryption key for 2017 Arb elections
08:08 volans: powercycling ganeti1005
06:57 _joe_: reeanbling puppet across servers with scap
06:44 marostegui: Defragment s2 databases on db1102 - T172169
06:43 _joe_: restarted hhvm on mw1283, still the same kind of lockups
06:33 marostegui: Deploy schema change on db1073 - T174569
06:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1073 - T174569 (duration: 00m 57s)
06:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1106 - T161294 (duration: 00m 56s)
06:17 marostegui: Stop replication in sync on db1106 and db1100 - T161294
06:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1106 - T161294 (duration: 00m 57s)
03:02 eileen: re-enable multiqueue_consumer process-control config revision is 1ae3778278
03:00 eileen: civicrm revision changed from 43d3f4d739 e0ee2d189c
02:32 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.12) (duration: 05m 57s)
01:17 cwd: disabled fredge multiqueue consumer
00:49 eileen: civicrm revision changed from 798e24671b 43d3f4d739
17:27 no_justification: gerrit: Back, might see a few transient puppet failures if git pulls happened during the d/t, but should all recover
17:21 no_justification: gerrit: halting service momentarily for account reindexing
15:00 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1100 (duration: 00m 58s)
12:29 ariel@tin: Finished deploy [dumps/dumps@95dbfe6]: revert previous deploy (duration: 00m 02s)
12:29 ariel@tin: Started deploy [dumps/dumps@95dbfe6]: revert previous deploy
12:16 ariel@tin: Finished deploy [dumps/dumps@faf7de8]: use cat to recombine gzipped files (duration: 00m 02s)
12:16 ariel@tin: Started deploy [dumps/dumps@faf7de8]: use cat to recombine gzipped files
02:04 mutante: labweb1002 - manually downgrade to scap 3.7.4-1 (disabled puppet)
01:49 mutante: reimported scap 3.7.4-1 into APT (jessie-wikimedia) after fixing md5/sha sums in .dsc and .changes files to match orig.tar.gz | copied it from jessie-wikimedia to trusty and stretch-wikimedia. all distributions downgraded to 3.7.4-1 (T183046 )
01:42 mutante: reimported scap 3.7.4-1 into APT (jessie-wikimedia) after fixing md5/sha sums in .dsc and .changes files to match orig.tar.gz
01:15 mutante: reprepro removing scap 3.7.4-2 package, attempting to reimport 3.7.4-1 package
01:11 mutante: reprepro remove trusty-wikimedia scap
00:44 demon@tin: Synchronized php-1.31.0-wmf.12/extensions/LoginNotify/includes/LoginNotify.php: T182867 (duration: 00m 57s)
00:40 demon@tin: Synchronized README: Testing again, this time with feeling (duration: 00m 56s)
00:28 mutante: re-disabled puppet on labweb1001/labweb1002 (as it was before)
00:26 mutante: re-enabling puppet on scap hosts
00:25 demon@tin: Synchronized README: Testing (duration: 00m 57s)
00:10 mutante: no more scap 3.7.4-2 found across 'R:Package = scap' (T183046 )
23:55 mutante: downgrading scap from 3.7.4-2 to 3.7.4-1 where it is installed - cumin -b 10 -s 5 'R:Package = scap' 'if dpkg -l scap | grep "3.7.4.2" && file /var/cache/apt/archives/scap_3.7.4-1_all.deb; then puppet agent --disable; apt-get remove --yes -q scap ; dpkg -i /var/cache/apt/archives/scap_3.7.4-1_all.deb ; fi' targeting 478 hosts (T183046 )
23:37 mutante: aqs1004, analytics1003, downgraded scap to 3.7.4-1
22:55 mutante: tin - apt-get remove scap ; dpkg -i /var/cache/apt/archives/scap_3.7.4-1_all.deb
19:39 chasemp: reboot labtestvirt2003
18:15 jgleeson: rolled back to civicrm to 798e2467
17:32 jynus: stop, upgrade and reboot labsdb1009
17:18 jynus: reloading dbproxy1010
16:44 chasemp: labtestvirt2003:~# /sbin/reboot to pickup new kernel
16:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1084 original weight (duration: 00m 57s)
16:10 elukey: re-enable piwik on bohrium after mysql backup restore
15:11 chasemp: reimage labtestvirt2003.codfw.wmnet
13:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1084 and restore original weight for db1081 (duration: 00m 56s)
13:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1084 and db1081 (duration: 00m 56s)
13:20 chasemp: disable puppet across eqiad lab* things to land a bit of code gracefully
13:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1084 (duration: 00m 56s)
12:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1084 with low weight - T180788 (duration: 00m 57s)
10:59 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1080 - T174569 (duration: 00m 56s)
10:59 jynus: reloading dbproxy1011
10:50 marostegui: Stop MySQL on db1084 to clone db1111 - T180788
10:49 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1084 to clone db1111 - T180788 (duration: 00m 56s)
10:31 elukey: rolling restart of yarn nodemanagers on an103* to apply new config - T182276
10:14 moritzm: uploaded prometheus-dns-rec-exporter 0.3 to apt.wikimedia.org
09:50 elukey: restore piwik database on bohrium after mysql corruption - piwik disabled
09:26 marostegui: Deploy schema change on db1080 - T174569
09:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1080 - T174569 (duration: 00m 56s)
09:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1083 - T174569 (duration: 00m 56s)
08:50 moritzm: powercycling ganeti1005
08:38 marostegui: Stop MySQL on db1034 to decommission it - T182556
08:26 marostegui: Remove db1034 from tendril - T182556
06:50 marostegui: Deploy schema change on db1083 - T174569
06:49 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1083 - T174569 (duration: 00m 56s)
06:43 marostegui: Deploy schema change on dbstore1001 (s1) - T174569
06:29 marostegui: Fix dbstore1001 s5 replication
06:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 and db1109 - T161294 (duration: 01m 16s)
03:01 mutante: db2016 thru db2019 - had to manually kill gmond process to decom ganglia, other db codfw hosts: didnt need it | running puppet on db205* and others in codfw to remove all ganglia (T177225 )
01:23 catrope@tin: Synchronized php-1.31.0-wmf.12/extensions/ORES: Fix broken join conditions (T182936 ) (duration: 00m 57s)
01:22 hoo: Updated the Wikidata property suggester with data from Monday's JSON dump and applied the T132839 workarounds
01:19 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Don't discount file searches on commonswiki (duration: 00m 57s)
01:11 catrope@tin: Synchronized php-1.31.0-wmf.11: Pulling in today's cherry-picks into wmf.11 too (duration: 10m 49s)
00:44 catrope@tin: Synchronized php-1.31.0-wmf.12/resources/src/mediawiki.rcfilters/ui/mw.rcfilters.ui.MenuSelectWidget.js: T182711 (duration: 00m 56s)
00:33 catrope@tin: Synchronized php-1.31.0-wmf.12/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.ArticleTargetEvents.js: Track editor mode on save events (T182610 ) (duration: 00m 56s)
00:25 catrope@tin: Synchronized php-1.31.0-wmf.12/resources/lib/oojs-ui/oojs-ui-core.js: Backport OOjs UI fix for T182359 , T182395 (duration: 00m 57s)
00:14 RoanKattouw: updateCollation.php finished on sewiki (T181503 )
00:13 RoanKattouw: Running updateCollation.php on sewiki
00:13 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Set category collation for sewiki to uppercase-se (duration: 00m 57s)
22:58 demon@tin: rebuilt and synchronized wikiversions files: group2 to wmf.12 (#2)
22:47 demon@tin: Synchronized php-1.31.0-wmf.12/extensions/ORES/includes/Hooks/ApiHooksHandler.php: fix undefined property (duration: 01m 08s)
22:17 demon@tin: rebuilt and synchronized wikiversions files: rollback, ORES breaking stuff on enwiki
22:08 demon@tin: rebuilt and synchronized wikiversions files: group2 to wmf.12
21:53 ejegg: updated CiviCRM from 1086291bb3 41a23f9fc3
20:33 awight: stress testing ores*
20:30 demon@tin: Pruned MediaWiki: 1.31.0-wmf.10 [keeping static files] (duration: 03m 32s)
20:28 jgleeson: turned back on donations queue consume service and thank you mailer service
20:21 awight@tin: Started restart [ores/deploy@b67bba7]: (non-production) Restart ORES services on ores*
20:19 thcipriani@tin: Finished scap: SWAT: Update en/i18n message for multiple-unclosed-formatting-tags (duration: 30m 55s)
20:02 jgleeson: updated civicrm from 798e24671b 1086291bb3
{{safesubst:SAL entry|1=20:02 urandom: lowering cassandra compaction throughput to 5MB/s, restbase101{2,4}-{a,b,c}}}
19:58 jgleeson: Temporarily disabled donations queue consume service and thank you mailer service
19:48 thcipriani@tin: Started scap: SWAT: Update en/i18n message for multiple-unclosed-formatting-tags
19:43 thcipriani@tin: Synchronized wmf-config/throttle.php: SWAT: Define new throttle rule and cleaning expired rules T182889 (duration: 01m 08s)
19:29 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wgNamespaceRobotPolicies for wikidata T181525 (duration: 01m 04s)
19:20 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove single editor tab for plwiki T181045 (duration: 01m 09s)
19:02 awight@tin: Finished deploy [ores/deploy@b67bba7]: Redeploy ORES to scb1001 (duration: 01m 04s)
19:01 awight@tin: Started deploy [ores/deploy@b67bba7]: Redeploy ORES to scb1001
19:01 bsitzmann@tin: Finished deploy [mobileapps/deploy@bf85a55]: Update mobileapps to ff74bb1 T182868 T182774 ) (duration: 12m 26s)
18:54 awight@tin: Started restart [ores/deploy@b67bba7]: Restart ORES services
18:51 arlolra: Updated Parsoid to ca20680 T182793 , T182774 )
18:48 bsitzmann@tin: Started deploy [mobileapps/deploy@bf85a55]: Update mobileapps to ff74bb1 T182868 T182774 )
18:41 arlolra@tin: Finished deploy [parsoid/deploy@13b5cb5]: (no justification provided) (duration: 09m 19s)
18:32 arlolra@tin: Started deploy [parsoid/deploy@13b5cb5]: (no justification provided)
18:24 elukey: replace kafka1018 with kafka1023 (Analytics Kafka cluster)
18:11 awight@tin: Started deploy [ores/deploy@b67bba7]: Update ORES service to b67bba77acb
16:34 bd808: Scholarships: Set application start and close dates via web UI (T181072 )
16:25 niharika29@tin: Finished deploy [scholarships/scholarships@872381d]: Deploy wikimania scholarships app for 2018 T181072 (duration: 00m 02s)
16:25 niharika29@tin: Started deploy [scholarships/scholarships@872381d]: Deploy wikimania scholarships app for 2018 T181072
16:07 bd808: Scholarships: updated database schema with 20171212-add-scholarship-orgs-field.sql (T181072 )
15:48 marostegui: Deploy schema change on dbstore1002 (s1) - T174569
15:25 jynus: stop, upgrade and restart labsdb1011
15:07 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1089 - T174569 (duration: 01m 08s)
14:21 hashar@tin: Synchronized php-1.31.0-wmf.12/resources/src/mediawiki.rcfilters: Swat for RCFilters https://gerrit.wikimedia.org/r/#/c/398242/ https://gerrit.wikimedia.org/r/#/c/398239/ (duration: 01m 09s)
13:47 chasemp: I'm reverting https://gerrit.wikimedia.org/r/#/c/394541/ as it broke the database puppet for labservices (used for powerdns backend)
13:42 godog: upgrade grafana to 4.6.3 - T182294
13:41 elukey: update facts for puppet compiler to pick up new hosts
13:32 mobrovac@tin: Finished deploy [restbase/deploy@187d8ba]: Remove Trending Edits end point and stop storing feed results in Cassandra - T180384 T179412 (duration: 05m 37s)
13:26 mobrovac@tin: Started deploy [restbase/deploy@187d8ba]: Remove Trending Edits end point and stop storing feed results in Cassandra - T180384 T179412
13:10 marostegui: Deploy schema change on db1089 (s1) - T174569
13:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1089 - T174569 (duration: 01m 07s)
12:56 awight: stress testing on ores1*.eqiad.wmnet cluster, T182249
12:42 awight@tin: Finished deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster (duration: 00m 59s)
12:41 awight@tin: Started deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster
12:40 awight@tin: Finished deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster (duration: 00m 45s)
12:39 awight@tin: Started deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster
12:18 gehel: re-initialize cassandra on maps-test2001 - T182583
12:11 jynus: disable puppet on all databases to deploy safely https://gerrit.wikimedia.org/r/398246
12:00 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1091 - T174569 (duration: 01m 08s)
11:27 awight@tin: Finished deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster (duration: 00m 20s)
11:27 awight@tin: Started deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster
11:24 awight@tin: Finished deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster (duration: 00m 05s)
11:23 awight@tin: Started deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster
11:23 awight@tin: Finished deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster (duration: 00m 11s)
11:23 awight@tin: Started deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster
11:16 akosiaris@tin: Finished deploy [ores/deploy@b67bba7]: T181661 (duration: 00m 03s)
11:16 akosiaris@tin: Started deploy [ores/deploy@b67bba7]: T181661
11:14 akosiaris@tin: Finished deploy [ores/deploy@b67bba7]: T181661 (duration: 00m 03s)
11:14 akosiaris@tin: Started deploy [ores/deploy@b67bba7]: T181661
11:13 akosiaris@tin: Finished deploy [ores/deploy@b67bba7]: T181661 (duration: 00m 55s)
11:12 akosiaris@tin: Started deploy [ores/deploy@b67bba7]: T181661
09:27 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1067 (duration: 01m 08s)
08:39 marostegui: Reload dbproxy1011 config
07:18 marostegui: Stop replication and set read-only on labsdb1003 - T142807
07:08 marostegui: Stop replication in sync on db1109 and db1101:3318 - T161294
07:08 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1101:3318 and db1109 - T161294 (duration: 01m 07s)
07:02 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 and db1109 - T161294 (duration: 01m 08s)
06:55 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1101:3318 and db1109 - T161294 (duration: 01m 09s)
06:35 marostegui: Deploy schema change on db1091 (s4) - T174569
06:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1091 - T174569 (duration: 01m 08s)
02:26 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.11) (duration: 08m 05s)
01:46 twentyafterfour: no phabricator deployment tonight, not enough time to prepare and test the update due to a short outage earlier this evening.
00:12 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: T182534 (duration: 01m 08s)
23:54 twentyafterfour: restarted phd on phab1001 (for good measure)
23:23 reedy@tin: Synchronized php-1.31.0-wmf.12/extensions/Flow/Hooks.php: unbreak CheckUser (duration: 01m 08s)
23:22 foks: deleted 22 illegal images from server
22:51 volans: restarting apache2 on phab1001, phabricator timing out
22:23 ppchelko@tin: Finished deploy [cpjobqueue/deploy@044cd23]: Fix sha1-based deduplication (duration: 00m 34s)
22:23 ppchelko@tin: Started deploy [cpjobqueue/deploy@044cd23]: Fix sha1-based deduplication
22:03 mholloway-shell@tin: Finished deploy [mobileapps/deploy@e62d8e3]: Update mobileapps to ddddebb
21:58 mholloway-shell@tin: Started deploy [mobileapps/deploy@e62d8e3]: Update mobileapps to ddddebb
21:54 ejegg: updated civicrm from 85a8526eb8 798e24671b
21:37 mholloway-shell@tin: Finished deploy [mobileapps/deploy@b8082da]: Update mobileapps to bf67c97
21:33 mholloway-shell@tin: Started deploy [mobileapps/deploy@b8082da]: Update mobileapps to bf67c97
21:24 ppchelko@tin: Finished deploy [restbase/deploy@a993556]: Do not fallback if the revision is not specified T182770 (duration: 04m 04s)
21:20 ppchelko@tin: Started deploy [restbase/deploy@a993556]: Do not fallback if the revision is not specified T182770
20:27 demon@tin: Synchronized php-1.31.0-wmf.12/extensions/WikimediaEvents/extension.json: James_F made me do it (duration: 01m 08s)
20:14 demon@tin: rebuilt and synchronized wikiversions files: group1 to wmf.12
20:10 demon@tin: Synchronized php: symlink bump for wmf.12 (duration: 01m 07s)
19:37 ebernhardson@tin: Synchronized php-1.31.0-wmf.12/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.trackSubscriber.js: SWAT: VE trackSubscriber: Add timing data for 'loaded' state (duration: 01m 07s)
19:35 ppchelko@tin: Finished deploy [restbase/deploy@3f4bedc]: Remove references to Cassandra 2 from Parsoid storage T179417 (duration: 04m 43s)
19:34 ebernhardson@tin: Synchronized php-1.31.0-wmf.12/resources/src/mediawiki.rcfilters/mw.rcfilters.UriProcessor.js: SWAT: T182734 : RCLFilters: support target page with a subpage (duration: 01m 07s)
19:30 ppchelko@tin: Started deploy [restbase/deploy@3f4bedc]: Remove references to Cassandra 2 from Parsoid storage T179417
19:25 ebernhardson@tin: Synchronized php-1.31.0-wmf.12/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.trackSubscriber.js: SWAT: VE trackSubscriber: data isn't required (duration: 01m 08s)
19:14 ebernhardson@tin: Synchronized php-1.31.0-wmf.12/resources/src/mediawiki.rcfilters/: SWAT: T182788 : RCFilters: Fix live update (duration: 01m 08s)
19:07 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Turn on cirrus MLR for 4 more wikis (duration: 01m 09s)
19:05 mutante: phab[12]001,mx[12]001,mendelevium,fermium: rm /usr/local/bin/exim-to-gmetric and remove root's crontab lines to follow-up gerrit:382916
18:31 mutante: releases2001: /srv/mediawiki# rm -rf extensions/ skins/ vendor/ | clean up removed repos, let puppet clone, to match releases1001 and fix puppet run
17:33 awight@tin: Finished deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster (duration: 00m 59s)
17:32 awight@tin: Started deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster
17:30 moritzm: installing wireshark security updates
16:30 marostegui: Deploy schema change on s4 on dbstore1001 - T174569
16:19 godog: try again deleting obsolete cassandra metrics from graphite2002 - T181964
15:50 akosiaris@tin: Finished deploy [ores/deploy@b4f2b02]: (no justification provided) (duration: 02m 22s)
15:48 akosiaris@tin: Started deploy [ores/deploy@b4f2b02]: (no justification provided)
15:47 akosiaris@tin: Finished deploy [ores/deploy@b4f2b02]: (no justification provided) (duration: 01m 30s)
15:46 akosiaris@tin: Started deploy [ores/deploy@b4f2b02]: (no justification provided)
15:43 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1034 from config - T182556 (duration: 01m 07s)
15:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1034 from config - T182556 (duration: 01m 09s)
15:39 akosiaris@tin: Finished deploy [ores/deploy@b4f2b02]: (no justification provided) (duration: 04m 46s)
15:34 akosiaris@tin: Started deploy [ores/deploy@b4f2b02]: (no justification provided)
15:21 chasemp: remove and purge vblade-persist and runit from labstore1004 T182781
15:02 jynus: upgrade and restart db1067
15:02 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1067 (duration: 01m 07s)
14:43 zeljkof: EU SWAT finished
14:38 zfilipin@tin: Synchronized portals: SWAT: Bumping portals to master (T128546) (duration: 01m 09s)
14:37 zfilipin@tin: Synchronized portals/prod/wikipedia.org/assets: SWAT: Bumping portals to master (T128546) (duration: 01m 08s)
14:29 dcausse@tin: Synchronized php-1.31.0-wmf.12/extensions/Wikibase: T182293 Extract names of search fields as constants (duration: 02m 05s)
14:22 dcausse@tin: Synchronized wmf-config/InitialiseSettings.php: T182293 [cirrus] tune wikidata similarity configuration 2/2 (duration: 01m 07s)
14:20 dcausse@tin: Synchronized wmf-config/Wikibase.php: T182293 [cirrus] tune wikidata similarity configuration 1/2 (duration: 01m 12s)
14:03 akosiaris: gnt-node remove ganeti1006 T181121
14:01 elukey: restart Yarn nodemanagers on analytics102[8,9] to apply new settings - T182276
13:21 moritzm: uploaded prometheus-pdns-rec-exporter 0.2-1 to apt.wikimedia.org
13:10 marostegui: Deploy schema change on s4 - dbstore1002 - T174569
12:53 marostegui: Deploy alter table on s4 db1064 (sanitarium master) with replication, this will generate lag on labs replicas - T174569
12:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1103:3314 - T174569 (duration: 01m 36s)
12:19 moritzm: uploaded prometheus-ircd-exporter and prometheus-pdns-rec-exporter to apt.wikimedia.org
11:59 elukey: forced remount of /mnt/hdfs after OOM event on stat1005
11:41 akosiaris: empty ganeti1006 for reimage after ltpstress and bios upgrades T181121
11:16 mobrovac@tin: Finished deploy [graphoid/deploy@7979a40]: Update to service-template-node v0.5.4 (duration: 03m 56s)
11:12 mobrovac@tin: Started deploy [graphoid/deploy@7979a40]: Update to service-template-node v0.5.4
10:19 mobrovac@tin: Finished deploy [recommendation-api/deploy@ac66089]: Update to service-template-node v0.5.4 (duration: 02m 20s)
10:17 mobrovac@tin: Started deploy [recommendation-api/deploy@ac66089]: Update to service-template-node v0.5.4
09:41 godog: upload prometheus-elasticsearch-exporter to jessie-wikimedia - T181627
08:38 jynus: migrate away from ganeti1008
06:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1103:3314 - T174569 (duration: 01m 11s)
06:25 marostegui: Deploy schema change on db1103:3314 - T174569
06:12 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1097:3314 - T174569 (duration: 01m 08s)
02:26 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.11) (duration: 07m 02s)
00:49 ebernhardson@tin: Synchronized php-1.31.0-wmf.11/extensions/WikimediaEvents/: SWAT: T182616 : turn on second mlr ab test for hewiki (duration: 01m 08s)
00:44 ebernhardson@tin: Synchronized php-1.31.0-wmf.12/extensions/WikimediaEvents/: SWAT: T182616 : turn on second mlr ab test for hewiki (duration: 01m 08s)
00:40 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Turn on cirrus MLR for most wikis with >1% of search traffic (duration: 01m 08s)
00:35 mholloway-shell@tin: Finished deploy [mobileapps/deploy@5832a8c]: Update mobileapps to bfc3588
00:32 brion: restarted requeueTranscodes.php on terbium for mp3 audio generation backfill (had dropped DB connection)
00:31 mholloway-shell@tin: Started deploy [mobileapps/deploy@5832a8c]: Update mobileapps to bfc3588
00:05 ebernhardson@tin: Synchronized wmf-config/: SWAT: T182616 : Setup MLR AB test for hewiki (duration: 01m 10s)
22:30 mholloway-shell@tin: Finished deploy [mobileapps/deploy@ea8f05d]: Update mobileapps to 94f267b
22:24 mholloway-shell@tin: Started deploy [mobileapps/deploy@ea8f05d]: Update mobileapps to 94f267b
22:02 urandom: setting compaction throughput to 5 MB/s, restbase1010
21:31 mutante: releases1001 - rm mediawiki core repo and let puppet try to recreate it (follow-up issue after gerrit:397891)
21:28 mobrovac@tin: Finished deploy [restbase/deploy@dceab2e]: Switch to Parsoid content v1.6.0 and switch to Cassandra 3 storage - T179417 (duration: 04m 16s)
21:27 demon@tin: Synchronized php-1.31.0-wmf.12/extensions/GlobalBlocking/: (no justification provided) (duration: 01m 07s)
21:26 mholloway-shell@tin: Finished deploy [mobileapps/deploy@28bfda3]: Update mobileapps to d0ee651
21:24 demon@tin: Synchronized php-1.31.0-wmf.11/extensions/GlobalBlocking/: (no justification provided) (duration: 01m 08s)
21:24 mholloway-shell@tin: Started deploy [mobileapps/deploy@28bfda3]: Update mobileapps to d0ee651
21:24 mholloway-shell@tin: Finished deploy [mobileapps/deploy@b2d5b8e]: Update mobileapps to 172abc7
21:24 mholloway-shell@tin: Started deploy [mobileapps/deploy@b2d5b8e]: Update mobileapps to 172abc7
21:23 mobrovac@tin: Started deploy [restbase/deploy@dceab2e]: Switch to Parsoid content v1.6.0 and switch to Cassandra 3 storage - T179417
21:21 ppchelko@tin: Finished deploy [restbase/deploy@dceab2e]: Bump expected parsoid version, but do not switch summaries, take 4 (duration: 02m 45s)
21:18 ppchelko@tin: Started deploy [restbase/deploy@dceab2e]: Bump expected parsoid version, but do not switch summaries, take 4
21:15 ppchelko@tin: Finished deploy [restbase/deploy@dceab2e]: Bump expected parsoid version, but do not switch summaries, take 3 (duration: 02m 04s)
21:13 ppchelko@tin: Started deploy [restbase/deploy@dceab2e]: Bump expected parsoid version, but do not switch summaries, take 3
21:13 ppchelko@tin: Finished deploy [restbase/deploy@dceab2e]: Bump expected parsoid version, but do not switch summaries, take 2 after failed content rerender (duration: 05m 05s)
21:12 mobrovac@tin: Started restart [electron-render/deploy@94d27d7]: Electron hanging - T174916
21:08 ppchelko@tin: Started deploy [restbase/deploy@dceab2e]: Bump expected parsoid version, but do not switch summaries, take 2 after failed content rerender
21:08 ppchelko@tin: Finished deploy [restbase/deploy@dceab2e]: Bump expected parsoid version, but do not switch summaries (duration: 05m 51s)
21:02 ppchelko@tin: Started deploy [restbase/deploy@dceab2e]: Bump expected parsoid version, but do not switch summaries
20:55 ppchelko@tin: Finished deploy [restbase/deploy@d3ca789]: Revert deployment for using MCS for summaries (duration: 01m 36s)
20:53 ppchelko@tin: Started deploy [restbase/deploy@d3ca789]: Revert deployment for using MCS for summaries
20:53 demon@tin: rebuilt and synchronized wikiversions files: group0 to wmf.12
20:51 ppchelko@tin: Finished deploy [restbase/deploy@506047c]: Update expected Parsoid version, switched summary to MCS (duration: 03m 30s)
20:48 ppchelko@tin: Started deploy [restbase/deploy@506047c]: Update expected Parsoid version, switched summary to MCS
20:44 mholloway-shell@tin: Finished deploy [mobileapps/deploy@b2d5b8e]: Update mobileapps to 172abc7
20:39 mholloway-shell@tin: Started deploy [mobileapps/deploy@b2d5b8e]: Update mobileapps to 172abc7
20:35 mholloway-shell@tin: Finished deploy [mobileapps/deploy@0a9d635]: Update mobileapps to 035608d
20:32 mholloway-shell@tin: Started deploy [mobileapps/deploy@0a9d635]: Update mobileapps to 035608d
20:31 mholloway-shell@tin: Finished deploy [mobileapps/deploy@2690678]: Update mobileapps to 5b8796d
19:57 arlolra: Updated Parsoid to 741fc5d T114072 , T181226 , T21910 , T152540 , T103714 , T97093 , T118520 , T181229 , T182338 , T182170 , T169006 )
19:51 mholloway-shell@tin: Started deploy [mobileapps/deploy@2690678]: Update mobileapps to 5b8796d
19:43 arlolra@tin: Finished deploy [parsoid/deploy@98139cb]: (no justification provided) (duration: 14m 57s)
19:35 anomie: Running cleanupUsersWithNoId.php on all wikis (this will take a while), see T181731
19:12 arlolra: Parsoid deploy aborted and rolled back to 01c1fc3
19:06 arlolra@tin: Finished deploy [parsoid/deploy@98139cb]: Updating Parsoid to 741fc5d
19:00 arlolra@tin: Started deploy [parsoid/deploy@98139cb]: Updating Parsoid to 741fc5d
18:48 aaron@tin: Synchronized php-1.31.0-wmf.12/includes/Setup.php: 058c17e702eb0
18:21 moritzm: uploaded prometheus-ircd-exporter to apt.wikimedia.org
17:52 demon@tin: Finished scap: bootstrap wmf.12 (duration: 29m 35s)
17:22 demon@tin: Started scap: bootstrap wmf.12
17:15 demon@tin: Pruned MediaWiki: 1.31.0-wmf.7 (duration: 08m 45s)
17:12 akosiaris@tin: Finished deploy [ores/deploy@b4f2b02]: T181661 (duration: 00m 03s)
17:12 akosiaris@tin: Started deploy [ores/deploy@b4f2b02]: T181661
17:11 akosiaris@tin: Finished deploy [ores/deploy@b4f2b02]: T181661 (duration: 00m 36s)
17:11 akosiaris@tin: Started deploy [ores/deploy@b4f2b02]: T181661
17:03 akosiaris@tin: Started deploy [ores/deploy@b4f2b02]: T181661
16:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1084 and db1081 original weight (duration: 00m 56s)
16:40 jynus: restart and upgrade db1059 (phabricator passive db)
16:35 gehel@tin: Finished deploy [kartotherian/deploy@6e223df]: new kartotherian packaging on maps-test2001 (duration: 00m 19s)
16:35 gehel@tin: Started deploy [kartotherian/deploy@6e223df]: new kartotherian packaging on maps-test2001
16:34 gehel@tin: Finished deploy [tilerator/deploy@29d633e]: new tilerator packaging on maps-test2001 (duration: 00m 20s)
16:34 gehel@tin: Started deploy [tilerator/deploy@29d633e]: new tilerator packaging on maps-test2001
16:30 gehel@tin: Finished deploy [tilerator/deploy@29d633e]: new tilerator packaging on maps-test2002 (duration: 00m 20s)
16:30 gehel@tin: Started deploy [tilerator/deploy@29d633e]: new tilerator packaging on maps-test2002
16:28 gehel@tin: Finished deploy [kartotherian/deploy@6e223df]: new kartotherian packaging on maps-test2002 (duration: 00m 18s)
16:28 gehel@tin: Started deploy [kartotherian/deploy@6e223df]: new kartotherian packaging on maps-test2002
16:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1084 (duration: 00m 53s)
16:24 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2072 (duration: 00m 55s)
16:22 akosiaris@tin: Started deploy [ores/deploy@b4f2b02]: T181661
16:21 akosiaris: failover boron to ganeti1008
16:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1084 (duration: 00m 56s)
15:59 akosiaris@tin: Finished deploy [ores/deploy@b4f2b02]: T181661 (duration: 00m 04s)
15:59 akosiaris@tin: Started deploy [ores/deploy@b4f2b02]: T181661
15:59 akosiaris@tin: Finished deploy [ores/deploy@b4f2b02]: T181661 (duration: 00m 09s)
15:58 akosiaris@tin: Started deploy [ores/deploy@b4f2b02]: T181661
15:46 jynus: stop, upgrade and reboot db2072
15:34 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2072 (duration: 00m 56s)
15:31 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1097:3314 - T174569 (duration: 01m 01s)
15:31 marostegui: Deploy schema change on s4 db1097:3314 - T174569
15:25 gehel: powercycling maps-test2003
15:24 elukey: rename notebook1002 -> kafka1023 - step 3, replace notebook1002 with kafka1023 in the puppet config
15:20 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1084 with low weight (duration: 01m 18s)
15:06 marostegui: Upgrade MySQl on db1084
15:02 elukey: clear recdns records related to notebook1002/kafka1023 (rec_control wipe-cache kafka1023.eqiad.wmnet kafka1023.mgmt.eqiad.wmnet notebook1002.eqiad.wmnet 14.5.64.10.in-addr.arpa 104.3.65.10.in-addr.arpa) - T181518
14:58 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1260.eqiad.wmnet
14:46 elukey: start rename notebook1002 -> kafka1023 - step 2, dns config (host already shutdown) - T181518
14:39 hashar: Rebuild operations-puppet-tests-docker image based on c76d892 T178620 and /cache being owned by root
14:37 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2071 (duration: 00m 56s)
14:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore original weight for db1086 (duration: 00m 56s)
14:20 zeljkof: EU SWAT finished
14:18 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add NS aliases for zh_wikiquote (T181374) (duration: 00m 56s)
14:09 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: Lift account registration on en.wiki (T182665) (duration: 00m 56s)
14:02 herron: upgrading codfw puppet agents
13:59 jynus: stop, upgrade and reboot db2071
13:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1086 (duration: 00m 56s)
13:56 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2071 (duration: 00m 57s)
13:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1086 (duration: 00m 57s)
12:44 akosiaris@puppetmaster1001: conftool action : set/weight=10; selector: scb1004.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
12:10 akosiaris@puppetmaster1001: conftool action : set/weight=10; selector: scb1004.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
12:10 akosiaris@puppetmaster1001: conftool action : set/weight=10; selector: scb1003.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
12:06 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1092 (duration: 00m 55s)
11:56 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1086 with low weight - T163190 (duration: 00m 55s)
11:27 jynus: stop, upgrade and reboot db1092
11:26 marostegui: Upgrade MySQL and kernel on db1086
11:26 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1092 (duration: 00m 56s)
11:12 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2066 (duration: 00m 57s)
11:11 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1084 - T174569 (duration: 00m 57s)
11:11 marostegui: Deploy schema change on db1084 (s4) - T174569
11:05 marostegui: Deploy schema change on db1056 (s4) already depooled - T174569
11:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1081 - T174569 (duration: 00m 56s)
10:46 gehel@tin: Finished deploy [kartotherian/deploy@6e223df]: new kartotherian packaging on maps-test2003 (duration: 00m 10s)
10:46 gehel@tin: Started deploy [kartotherian/deploy@6e223df]: new kartotherian packaging on maps-test2003
10:26 jynus: stop, upgrade and reboot db2066
10:24 gehel@tin: Finished deploy [kartotherian/deploy@6e223df]: new kartotherian packaging on maps-test2004 (duration: 00m 18s)
10:24 gehel@tin: Started deploy [kartotherian/deploy@6e223df]: new kartotherian packaging on maps-test2004
10:23 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2066 (duration: 00m 56s)
10:14 moritzm: reimaging mw1260 (video scaler) to stretch
10:04 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2059 (duration: 00m 56s)
09:51 moritzm: updated stretch installer netboot image after stretch 9.3 point release
09:51 moritzm: updated stretch installer netboot image after jessie 8.10 point release
09:44 marostegui: Stop db1039 and db1086 in sync - T163190
09:43 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1086 - T163190 (duration: 00m 56s)
09:38 gehel: reduce replication factor for cassandra on maps-test cluster and reset cassandra on maps-test2001 to work around limited disk space - T182583
09:34 marostegui: Stop replication in sync on db1034 and db1039 for data consistency check - T163190
09:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1096:3316 after InnoDB there - T178359 (duration: 00m 56s)
09:00 jynus: stop, upgrade and reboot db2059
08:57 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2059 (duration: 00m 56s)
08:47 moritzm: updated jessie installer netboot image after jessie 8.10 point release
07:15 marostegui: Deploy schema change on db1081 - https://phabricator.wikimedia.org/T174569
07:14 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1081 - T174569 (duration: 00m 56s)
06:56 marostegui: stop MySQL on db1055 - T182653
06:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1055 - T178359 T182653 (duration: 00m 56s)
02:27 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.11) (duration: 06m 21s)
00:53 legoktm@tin: Synchronized wmf-config/CommonSettings.php: Remove manual firejailing of Score binaries (T181535 ) (duration: 00m 56s)
00:47 legoktm@tin: Synchronized wmf-config/CommonSettings.php: Have ExtensionDistributor treat REL1_30 as stable (duration: 00m 56s)
00:39 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Re-enable ORES on fawiki (T182354 ) (duration: 00m 56s)
00:25 ejegg: updated CiviCRM from 2db9c76 85a8526
00:23 catrope@tin: Synchronized wmf-config/CommonSettings.php: Fix default for rcOresDamagingPref (duration: 00m 56s)
00:22 ejegg: disabled nightly Ingenico WX audit parsing job
23:37 smalyshev@tin: Finished deploy [wdqs/wdqs@f6b110f]: updater fix and GUI update (duration: 06m 10s)
23:31 smalyshev@tin: Started deploy [wdqs/wdqs@f6b110f]: updater fix and GUI update
23:28 mholloway-shell@tin: Finished deploy [mobileapps/deploy@07293bc]: Update mobileapps to e290b17
23:22 mholloway-shell@tin: Started deploy [mobileapps/deploy@07293bc]: Update mobileapps to e290b17
21:11 mholloway-shell@tin: Finished deploy [mobileapps/deploy@6347d62]: Update mobileapps to 61ca333
21:04 mholloway-shell@tin: Started deploy [mobileapps/deploy@6347d62]: Update mobileapps to 61ca333
19:47 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: T181493 : Enable Page Previews EventLogging instrumentation (duration: 00m 56s)
19:47 mobrovac@tin: Finished deploy [restbase/deploy@be7d72f]: Expose the Reading Lists end points, take #3 - T181107 (duration: 06m 19s)
19:46 ppchelko@tin: Finished deploy [cpjobqueue/deploy@b1beaf1]: Revert dedupe based on sha1 as well as on event ID (duration: 00m 29s)
19:46 ppchelko@tin: Started deploy [cpjobqueue/deploy@b1beaf1]: Revert dedupe based on sha1 as well as on event ID
{{safesubst:SAL entry|1=19:43 urandom: lower compaction throughput to 2 MB/s, restbase1010-{a,b,c} - T178177}}
19:40 mobrovac@tin: Started deploy [restbase/deploy@be7d72f]: Expose the Reading Lists end points, take #3 - T181107
19:34 mobrovac@tin: Finished deploy [restbase/deploy@bce2885]: Expose the Reading Lists end points, take #2 - T181107 (duration: 05m 33s)
19:28 mobrovac@tin: Started deploy [restbase/deploy@bce2885]: Expose the Reading Lists end points, take #2 - T181107
19:28 mobrovac@tin: Finished deploy [restbase/deploy@bce2885]: Expose the Reading Lists end points - T181107 (duration: 01m 26s)
19:26 mobrovac@tin: Started deploy [restbase/deploy@bce2885]: Expose the Reading Lists end points - T181107
19:24 ebernhardson@tin: Synchronized wmf-config/throttle.php: SWAT: T182613 Update throttle rule for McGill University Library (duration: 00m 56s)
19:21 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: T131132 : Switch submit button from save to publish on enwiki (duration: 02m 43s)
19:19 awight@tin: Finished deploy [ores/deploy@1c0ede0]: (non-production) Testing parallel ORES deployment, T181661 (duration: 01m 12s)
19:17 awight@tin: Started deploy [ores/deploy@1c0ede0]: (non-production) Testing parallel ORES deployment, T181661
19:16 ebernhardson@tin: Synchronized php-1.31.0-wmf.11/extensions/CirrusSearch/includes/Search/RescoreBuilders.php: SWAT: Add query string for running Cirrus MLR pre-deploy checks (duration: 00m 57s)
18:56 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: Switch cebwiki, ruwiki and small projects to Kafka for htmlCacheUpdate - T182023 (duration: 00m 57s)
18:55 ppchelko@tin: Finished deploy [cpjobqueue/deploy@e1075af]: Enable htmlCacheUpdate for ceb and ru wiki and small projects T182023 (duration: 00m 34s)
18:54 ppchelko@tin: Started deploy [cpjobqueue/deploy@e1075af]: Enable htmlCacheUpdate for ceb and ru wiki and small projects T182023
18:03 gehel@tin: Finished deploy [wdqs/wdqs@353b3cb]: wdqs: GUI and updater updates (duration: 01m 14s)
18:02 gehel@tin: Started deploy [wdqs/wdqs@353b3cb]: wdqs: GUI and updater updates
16:42 hashar: Restarting Jenkins
15:36 marostegui: Deploy schema change on s2 master (db1054) - T174569
14:51 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Disable the wgTranslateNumerals at hiwikiversity T182584 (duration: 00m 56s)
14:46 niharika29@tin: Synchronized php-1.31.0-wmf.11/includes/specials/SpecialUndelete.php: Revert replacing textarea in Special:Undelete with OOUI T182398 (duration: 00m 57s)
14:34 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Bureaucrats to grant and remove translationadmin rights; sysops to add and remove the same from themselves - T182492 (duration: 00m 56s)
14:32 niharika29@tin: Synchronized wmf-config/interwiki.php: Update the Interwiki map - T182506 (duration: 00m 56s)
14:25 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Create NS_PROJECT and NS_PROJECT_TALK alias for kowikisource (T182487 ) (duration: 00m 56s)
13:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1060 - T174569 (duration: 04m 42s)
12:39 _joe_: trying to get a full core dump from hhvm on mw1200
11:12 _joe_: depooling mw1200 for investigation instead
11:11 _joe_: restarting hhvm on mw1200, stuck in a kernel task
11:08 jdrewniak@tin: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 45s)
11:07 jdrewniak@tin: Synchronized portals/prod/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 44s)
10:49 ema: cp4021: restart varnish-be due to mbox lag
10:04 godog: upgrade grafana to 4.6.2 on labmon1001 - T182294
10:00 jynus: stopping dbstore2001:s5 and dbstore1002 (s5) mysql replication in sync
09:28 akosiaris: upload scap_3.7.4-1 to apt.wikimedia.org/jessie-wikimedia/main
09:16 gehel: cleaning old cassandra dumps on maps-test2001 servers
09:15 gehel: cleaning up old postgres logs on maps-test2001
09:05 elukey: set notebook1002 as role::spare as prep step to reimage it to kafka1023
09:03 jynus: dropping multiple leftover files from db1102
08:52 marostegui: Stop replication in sync on db1034 and db1039 - T163190
08:12 elukey: powercycle ganeti1008 - all vms stuck, console com2 showed a ton of printks without a clear indicator of the root cause
07:49 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1034 - T182556 (duration: 00m 45s)
07:44 _joe_: restarting hhvm on mw1189,mw1229,mw1235,mw1282,mw1285,mw1315,mw1316, all stuck with a kernel hang
06:59 _joe_: restarted hhvm, nginx on mw1280, hanging kernel operations
06:45 marostegui: Deploy schema change on s2 db1060 with replication enabled, this will generate some lag on s2 on labs - T174569
06:45 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1060 - T174569 (duration: 00m 44s)
06:22 marostegui: Compress s6 on db1096 - T178359
06:21 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1096:3316 to compress InnoDB there - T178359 (duration: 00m 45s)
02:43 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.11) (duration: 09m 21s)
20:33 elukey: execute restart-hhvm on mw1312 - hhvm stuck multiple times queueing requests
20:01 elukey: ran kafka preferred-replica-election for the kafka analytics cluster (1012->1022) to re-add kafka1012 to the kafka brokers acting as partition leaders (will spread the load in a better way)
17:00 apergos: restarted hhvm on mw1276, the same old hang with the same old symptoms
16:10 awight@tin: Finished deploy [ores/deploy@1c0ede0]: Reducing ORES Celery log verbosity (take 4\!) (duration: 03m 01s)
16:07 awight@tin: Started deploy [ores/deploy@1c0ede0]: Reducing ORES Celery log verbosity (take 4\!)
16:02 awight@tin: Finished deploy [ores/deploy@1c0ede0]: Reducing ORES Celery log verbosity (duration: 05m 58s)
15:56 awight@tin: Started deploy [ores/deploy@1c0ede0]: Reducing ORES Celery log verbosity
15:55 awight@tin: Finished deploy [ores/deploy@1c0ede0]: Reducing ORES Celery log verbosity (duration: 00m 17s)
15:55 awight@tin: Started deploy [ores/deploy@1c0ede0]: Reducing ORES Celery log verbosity
15:53 awight@tin: Finished deploy [ores/deploy@1c0ede0]: Reducing ORES Celery log verbosity (duration: 00m 31s)
15:53 awight@tin: Started deploy [ores/deploy@1c0ede0]: Reducing ORES Celery log verbosity
15:53 apergos: did same on scb1002,3,4
15:48 awight: Making an emergency deployment to ORES logging config to reduce verbosity.
15:45 apergos: on scb1001 moved daemon.log out of the way, did "service rsyslog rotate", saved the last 5000 entries for use by ores team, removed the log
11:44 apergos: that server list: mw1278, 1277, 1226, 1234, 1230
11:42 apergos: restarted hhvm on api servers after lockup
11:19 legoktm@tin: Synchronized wmf-config/InitialiseSettings.php: Disable ORES in fawiki - T182354 (duration: 00m 45s)
00:11 Jamesofur: removed 2FA from EVinente after verification T182373
23:23 hashar: force ran puppet on contint2001
22:15 madhuvishy: Kicked off rsync of /data/xmldatadumps/public to labstore1006 & 7
22:05 smalyshev@tin: Finished deploy [wdqs/wdqs@353b3cb]: temporary fix for T182464 , better fix coming soon (duration: 05m 55s)
21:59 smalyshev@tin: Started deploy [wdqs/wdqs@353b3cb]: temporary fix for T182464 , better fix coming soon
20:22 aaron@tin: Synchronized php-1.31.0-wmf.11/includes/Setup.php: a319c3e7ab61
18:00 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: Disable GlobalBlocking on fishbowl wikis (duration: 00m 45s)
16:23 urandom: starting cassandra, restbase1010 - T178177
16:22 urandom: disabling smart path, restbase1010, arrays 'b'...'e' - T178177
16:20 urandom: disabling smart path, restbase1010, array 'a' (canary) - T178177
16:15 urandom: shutting down cassandra, restbase1010 - T178177
15:35 marostegui: Fix dbstore1002 s5 replication
15:28 gehel@tin: Finished deploy [tilerator/deploy@29d633e]: testing new tilerator packaging on maps-test2003 (duration: 00m 03s)
15:28 gehel@tin: Started deploy [tilerator/deploy@29d633e]: testing new tilerator packaging on maps-test2003
15:08 gehel@tin: Finished deploy [tilerator/deploy@29d633e]: testing new tilerator packaging on maps-test2003 (duration: 02m 08s)
15:06 gehel@tin: Started deploy [tilerator/deploy@29d633e]: testing new tilerator packaging on maps-test2003
15:05 gehel@tin: Finished deploy [tilerator/deploy@29d633e]: testing new tilerator packaging on maps-test2003 (duration: 00m 42s)
15:05 gehel@tin: Started deploy [tilerator/deploy@29d633e]: testing new tilerator packaging on maps-test2003
14:39 gehel@tin: Finished deploy [tilerator/deploy@e52ea1d]: testing new tilerator packaging on maps-test2003 (duration: 02m 34s)
14:36 gehel@tin: Started deploy [tilerator/deploy@e52ea1d]: testing new tilerator packaging on maps-test2003
11:45 elukey: updated prometheus-druid-exporter on druid* to 0.6
11:39 elukey: upload prometheus-druid-exporter 0.6 to stretch/jessie wikimedia
06:52 marostegui: Fix labsdb1004 replication broken
06:43 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1099:3311 - T178359 (duration: 00m 55s)
04:56 bd808: scholarships: updated db schema for 2018 cycle (T181072 )
00:04 bblack: cp4026 - restart varnish backend, mailbox lag
23:59 tgr@tin: Synchronized wmf-config/InitialiseSettings.php: T181107 enable ReadingLists on all wikis (duration: 00m 46s)
23:54 tgr: ran mwscript extensions/ReadingLists/maintenance/populateProjectsFromSiteMatrix.php --wiki=testwiki
23:47 tgr@tin: Finished scap: T181107 deploy ReadingLists to testwiki (duration: 24m 44s)
23:22 tgr@tin: Started scap: T181107 deploy ReadingLists to testwiki
23:04 tgr: ran mwscript ../../../home/tgr/sql.php --wiki=mediawikiwiki --cluster extension1 --wikidb wikishared /srv/mediawiki-staging/php-1.31.0-wmf.11/extensions/ReadingLists/sql/readinglists.sql
22:33 tgr@tin: Synchronized php-1.31.0-wmf.11/extensions/ReadingLists/: ReadingLists/wmf.11: catching up with master (duration: 00m 45s)
22:29 tgr@tin: Synchronized php-1.31.0-wmf.10/extensions/ReadingLists/: ReadingLists/wmf.10: catching up with master (duration: 00m 46s)
20:39 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.11
20:35 elukey: restart hhvm on mw1235 - hhvm-dump-debug hanging out, not stacktrace available
20:31 elukey: restart hhvm on mw1281 - hhvm stuck (hhvm-dump-debug timing out)
20:22 joal@tin: Finished deploy [analytics/refinery@53bd630]: Regular analytics deploy - Long time no see, deployment :) - post-patch-2 (hopefully last for tonight) (duration: 05m 21s)
20:18 herron: re-pooling eqiad puppet 4 masters via dns puppet.eqiad.wmnet puppet.wikimedia.org
20:17 joal@tin: Started deploy [analytics/refinery@53bd630]: Regular analytics deploy - Long time no see, deployment :) - post-patch-2 (hopefully last for tonight)
20:14 joal@tin: Finished deploy [analytics/refinery@3e52903]: Regular analytics deploy - Long time no see, deployment :) - post-patch (duration: 01m 20s)
20:12 joal@tin: Started deploy [analytics/refinery@3e52903]: Regular analytics deploy - Long time no see, deployment :) - post-patch
19:52 joal@tin: Finished deploy [analytics/refinery@bd9c6cc]: Regualr analytics deploy - Long time no see, deployment :) (duration: 12m 15s)
19:40 joal@tin: Started deploy [analytics/refinery@bd9c6cc]: Regualr analytics deploy - Long time no see, deployment :)
19:35 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Start description usage tracking for commonswiki T106287 (duration: 00m 48s)
19:29 thcipriani@tin: Synchronized php-1.31.0-wmf.11/includes/specialpage/ChangesListSpecialPage.php: SWAT: WLFilters: Correctly check if RCFilters should be enabled on WL T182318 (duration: 00m 48s)
19:24 thcipriani@tin: Synchronized wmf-config/Wikibase-production.php: SWAT: Remove obsolete WikibaseQualityConstraints settings (duration: 00m 48s)
19:14 ppchelko@tin: Finished deploy [changeprop/deploy@3c4f51d]: Long awaited deploy: generic optimizations, gc metric, delay reporting (duration: 01m 15s)
19:13 ppchelko@tin: Started deploy [changeprop/deploy@3c4f51d]: Long awaited deploy: generic optimizations, gc metric, delay reporting
19:12 thcipriani@tin: Synchronized wmf-config/throttle.php: SWAT: Rm all past throttle overrides in throttle.php (duration: 00m 48s)
18:06 mholloway-shell@tin: Finished deploy [mobileapps/deploy@2fa32ed]: Update mobileapps to 71f581c
18:00 mholloway-shell@tin: Started deploy [mobileapps/deploy@2fa32ed]: Update mobileapps to 71f581c
18:00 herron: re-pooling eqiad puppet 4 masters as puppet.ulsfo.wnet
17:33 demon@tin: Synchronized php-1.31.0-wmf.11/extensions/VisualEditor/lib/ve: Ief480487
17:25 milimetric@tin: Finished deploy [analytics/aqs/deploy@4ec13b4]: (no justification provided) (duration: 07m 28s)
17:25 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1314.eqiad.wmnet
17:25 herron: puppetmaster1001 upgraded to puppet 4. re-enabling puppet agents across the fleet
17:18 milimetric@tin: Started deploy [analytics/aqs/deploy@4ec13b4]: (no justification provided)
17:13 herron: upgrading puppetmaster1001 to puppet 4
17:08 herron: temporarily disabling all puppet agents during puppetmaster1001 (puppet ca) upgrade to puppet 4
16:14 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1099:3311 weight - T178359 (duration: 00m 48s)
16:13 herron: upgrading puppetmaster1002 to puppet 4
16:03 moritzm: uploaded openssl 1.0.2n for jessie-wikimedia to apt.wikimedia.org
16:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1093 back as main traffic in s6 - T178359 (duration: 00m 48s)
15:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1099:3311 weight - T178359 (duration: 00m 48s)
15:42 elukey: hhvm-dump-debug for mw1314 saved to /tmp/hhvm.17991.bt.
15:30 elukey@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw1314.eqiad.wmnet
15:24 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1099:3311 weight - T178359 (duration: 00m 48s)
15:19 otto@tin: Finished deploy [analytics/superset/deploy@f0f5adf]: initial deployment (duration: 00m 02s)
15:18 otto@tin: Started deploy [analytics/superset/deploy@f0f5adf]: initial deployment
14:55 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1099:3311 with low weight - T178359 (duration: 00m 47s)
14:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully repool db1074 (duration: 00m 47s)
14:09 zeljkof: EU SWAT finished
14:08 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: alswiki: Set wgRestrictDisplayTitle = false (T182154) (duration: 00m 49s)
13:39 moritzm: reimaging mw2246 to stretch
12:54 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw2152.codfw.wmnet
12:32 addshore@tin: Synchronized wmf-config/Wikibase.php: Move Wikibase dispatchingLockManager to InitialiseSettings PT 2/2 (duration: 00m 48s)
12:31 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Move Wikibase dispatchingLockManager to InitialiseSettings PT 1/2 (duration: 00m 48s)
11:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase API traffic for db1074 (duration: 00m 48s)
11:41 moritzm: reimaging mw2152 to stretch
11:35 marostegui: Compress s8 on db1099 - T178359
11:21 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase API traffic for db1074 (duration: 00m 48s)
11:11 mobrovac@tin: Finished deploy [restbase/deploy@097ba7d]: Add CORS headers to erroneous responses as well - T182103 (duration: 05m 24s)
11:05 mobrovac@tin: Started deploy [restbase/deploy@097ba7d]: Add CORS headers to erroneous responses as well - T182103
10:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1074 with low weight (duration: 00m 48s)
10:50 elukey: powercycle analytics1003 - no serial console, ssh stuck in System is booting up. See pam_nologin(8)
10:28 _joe_: depooling mw1283 for further investigation
10:14 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1098:3316 db1098:3317 - T178359 (duration: 00m 51s)
10:12 elukey: reboot analytics1003 for kernel+jvm updates - T179943
10:01 gehel: upgrade of ELK stack on logstash100* completed - Kibana was unavailable for longer than expected - T178412
09:48 hashar: CI: removed Wikidata from configuration, replaced by Wikibase. wmf/* and REL branches are going to be broken though | https://gerrit.wikimedia.org/r/395704 | T181838
09:46 akosiaris: silence ganeti1006 on icinga T181121
09:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1098:3316 db1098:3317 - T178359 (duration: 00m 52s)
09:28 marostegui: Upgrade MySQL and kernel on db1074
09:22 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=logstash1009.eqiad.wmnet
09:19 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=logstash1009.eqiad.wmnet
09:18 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=logstash1008.eqiad.wmnet
09:14 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=logstash1008.eqiad.wmnet
09:13 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=logstash1007.eqiad.wmnet
09:08 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=logstash1007.eqiad.wmnet
09:05 gehel@tin: Finished deploy [logstash/plugins@b13d2fa]: (no justification provided) (duration: 00m 02s)
09:05 gehel@tin: Started deploy [logstash/plugins@b13d2fa]: (no justification provided)
09:00 gehel: upgrading ELK stack on logstash100* - some log messages might be lost during the upgrade - T178412
08:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1098:3316 db1098:3317 - T178359 (duration: 00m 48s)
08:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1098:3316 db1098:3317 - T178359 (duration: 00m 48s)
08:28 elukey: install prometheus-druid-exporter 0.5 on druid*
08:26 elukey: upload prometheus-druid-exporter 0.5-1 to jessie/stretch-wikimedia
07:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Slowly pool db1098:3316 db1098:3317 - T178359 (duration: 00m 47s)
07:18 marostegui@tin: Synchronized wmf-config/db-codfw.php: Slowly pool db1098:3316 db1098:3317 - T178359 (duration: 00m 48s)
06:45 marostegui: Stop replication on db1099:3311 to reimport: change_tag, tag_summary, user and watchlist tables and recompress again - T178359
06:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1074 - T174569 (duration: 00m 48s)
06:40 marostegui: Deploy schema change on db1074 (s2) - T174569
03:00 catrope@tin: Synchronized php-1.31.0-wmf.11/resources/src/mediawiki.rcfilters/: T182268 (duration: 00m 57s)
02:29 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.10) (duration: 08m 57s)
23:45 eileen: update CiviCRM from 85263f6 2db9c76
23:12 ppchelko@tin: Started restart [changeprop/deploy@065a06e]: (no justification provided)
23:04 eileen: update process-control to b1cd515
22:58 ppchelko@tin: Finished deploy [cpjobqueue/deploy@761524f]: Separate delay and totaldelay metrics T182216 (duration: 00m 31s)
22:57 ppchelko@tin: Started deploy [cpjobqueue/deploy@761524f]: Separate delay and totaldelay metrics T182216
22:55 mutante: stat1003 - re-enabled puppet after putting role::spare on it (T175150 )
22:48 mutante: stat1003 - this host was kind of invisible (not in site, not in icinga) but still up, re-enabling puppet after re-adding it to site
22:03 demon@tin: Synchronized php-1.31.0-wmf.11/extensions/WikidataPageBanner/includes/WikidataPageBanner.hooks.php: unbreak (duration: 00m 48s)
22:00 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.11, again
21:51 hoo@tin: Synchronized php-1.31.0-wmf.10/extensions/Wikibase/lib/includes/Changes/EntityChange.php: Make EntityChange truly forward compatible with compact diffs (T182243 ) (duration: 00m 48s)
21:43 arlolra: Updated Parsoid to 01c1fc3 T178253 , T61840 , T180930 , T179259 )
21:34 arlolra@tin: Finished deploy [parsoid/deploy@dfcc622]: Updating Parsoid to 01c1fc3
21:24 arlolra@tin: Started deploy [parsoid/deploy@dfcc622]: Updating Parsoid to 01c1fc3
20:45 marostegui: Add 400G to labsdb1003 /srv partition
20:15 hoo: Ran "scap pull" on snapshot1001 after T177486 related tests
20:10 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: partial rollback -- wikidata errors
20:07 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.11
20:05 demon@tin: Synchronized php: symlink bump (duration: 00m 47s)
19:32 hoo@tin: Synchronized php-1.31.0-wmf.11/extensions/Wikibase/repo/maintenance/dispatchChanges.php: dispatch: track how long client selecting takes (duration: 00m 48s)
19:24 ppchelko@tin: Finished deploy [cpjobqueue/deploy@8c66189]: Deduplicate based on event sha1 as well as on id, take 2 (duration: 00m 38s)
19:24 ppchelko@tin: Started deploy [cpjobqueue/deploy@8c66189]: Deduplicate based on event sha1 as well as on id, take 2
19:23 hoo@tin: Synchronized php-1.31.0-wmf.10/extensions/Wikibase/repo/maintenance/dispatchChanges.php: dispatch: track how long client selecting takes (duration: 00m 48s)
19:22 ppchelko@tin: Started deploy [cpjobqueue/deploy@8c66189]: Deduplicate based on event sha1 as well as on id, take 2
19:22 ppchelko@tin: Finished deploy [cpjobqueue/deploy@8c66189]: Deduplicate based on event sha1 as well as on id (duration: 00m 11s)
19:22 ppchelko@tin: Started deploy [cpjobqueue/deploy@8c66189]: Deduplicate based on event sha1 as well as on id
19:20 ppchelko@tin: Finished deploy [cpjobqueue/deploy@df72b34]: Deduplicate based on event sha1 as well as on id (duration: 00m 09s)
19:19 ppchelko@tin: Started deploy [cpjobqueue/deploy@df72b34]: Deduplicate based on event sha1 as well as on id
18:44 volans: stopped irc echo on tegmen, re-enabled puppet and run it (it was disabled by a run-no-puppet sync_icinga_state)
18:32 mobrovac@tin: Finished deploy [zotero/translators@3044b3a]: Update translators to 092c7bc T178596 (duration: 00m 07s)
18:32 mobrovac@tin: Started deploy [zotero/translators@3044b3a]: Update translators to 092c7bc T178596
18:16 herron: upgrading rhodium to puppet 4
18:13 mobrovac@tin: Synchronized wmf-config/jobqueue.php: Switch htmlCacheUpdate jobs for wiktionaries to EventBus, file 2/2 - T182023 (duration: 00m 48s)
18:12 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: Switch htmlCacheUpdate jobs for wiktionaries to EventBus, file 1/2 - T182023 (duration: 00m 48s)
18:05 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add https://studiezaal.nijmegen.nl to $wgCopyUploadsDomains (T181713) (duration: 00m 49s)
17:51 ppchelko@tin: Finished deploy [cpjobqueue/deploy@df72b34]: Switch htmlCacheUpdate for wiktionaries, attempt 2 T182023 (duration: 00m 32s)
17:50 ppchelko@tin: Started deploy [cpjobqueue/deploy@df72b34]: Switch htmlCacheUpdate for wiktionaries, attempt 2 T182023
17:44 ppchelko@tin: Finished deploy [cpjobqueue/deploy@3281df1]: Switch htmlCacheUpdate for wiktionaries T182023 (duration: 02m 57s)
17:42 ppchelko@tin: Started deploy [cpjobqueue/deploy@3281df1]: Switch htmlCacheUpdate for wiktionaries T182023
17:36 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1260.eqiad.wmnet
17:05 hoo: Started Wikidata RDF dumps (sudo -b -u datasets bash -c 'dumpwikidatardf.sh all ttl; dumpwikidatardf.sh truthy nt') on snapshot1007
16:33 anomie@terbium: Finished cleanupUsersWithNoId.php on mediawikiwiki
16:29 anomie@terbium: Running cleanupUsersWithNoId.php for mediawikiwiki, see T181731
16:28 anomie@terbium: Finished cleanupUsersWithNoId.php on testwikidatawiki
16:27 anomie@terbium: Running cleanupUsersWithNoId.php for testwikidatawiki, see T181731
16:26 anomie@terbium: Finished cleanupUsersWithNoId.php on test2wiki
16:14 anomie@terbium: Running cleanupUsersWithNoId.php for test2wiki, see T181731
16:11 anomie@terbium: Finished cleanupUsersWithNoId.php on testwiki
16:03 anomie@terbium: Running cleanupUsersWithNoId.php for testwiki, see T181731
15:13 akosiaris: upload kubernetes_1.7.10-1_amd64 on apt.wikimedia.org/stretch-wikimedia/main T181489
14:59 addshore: swat done
14:57 addshore@tin: Synchronized php-1.31.0-wmf.11/extensions/Wikibase/repo/includes/ChangeDispatcher.php: SWAT Tracking within ChangeDispatcher::getPendingChanges (duration: 00m 49s)
14:53 addshore@tin: Synchronized php-1.31.0-wmf.10/extensions/Wikibase/repo/includes/ChangeDispatcher.php: SWAT Tracking within ChangeDispatcher::getPendingChanges (duration: 00m 49s)
14:33 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add https://studiezaal.nijmegen.nl to $wgCopyUploadsDomains (T181713) (duration: 00m 47s)
14:22 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add Portal namespace for mwl.wikipedia (T180052) (duration: 00m 48s)
14:13 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT T180476 wmgUseNewWikiDiff2Extension true for dewiki (duration: 00m 48s)
14:08 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT T181498 T181329 Enable AdvancedSearch on fawiki and huwiki (duration: 00m 50s)
13:40 moritzm: reimaging mw1260 to stretch
10:55 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw2118.codfw.wmnet
10:55 moritzm: reimaging mw2119 to stretch
10:45 mobrovac@tin: Finished deploy [restbase/deploy@b1d7c82]: Use Cass3 for revisions, deprecate trending-edits, fix CX end point - T179421 T180384 T173801 (duration: 06m 02s)
10:39 mobrovac@tin: Started deploy [restbase/deploy@b1d7c82]: Use Cass3 for revisions, deprecate trending-edits, fix CX end point - T179421 T180384 T173801
09:34 gehel: shuttting down logstash / elasticsearch on logstash100[123] in preparation for decommission -T175830
08:05 moritzm: reimaging mw2118 to stretch (now for real, yesterday's reimage logged to SAL was interrupted)
07:33 bblack: cp4024 - backend restart, mailbox lag
02:39 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.10) (duration: 08m 53s)
01:01 eileen: update process-control to 55529ea
00:36 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable description usage tracking for all wikis except commons T106287 (duration: 00m 48s)
00:27 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable RemexHTML on wikis with zero high priority linter errors T182042 (duration: 00m 51s)
23:07 eileen: update civicrm from d2c70d2 85263f6
22:29 eileen: update process_control to 3522186
22:13 mutante: restbase1010 failed at reboot with P6431 , after a cold start (power off, power on) it came back though :) (T178177 T141756 )
22:00 mutante: restbase1010 - rebooting for firmware upgrade
21:58 mutante: restbase1010 - upgraded HP firmware (Flashing Smart Array P440ar in Slot 0 [ 3.56 -> 6.06 ]) T141756 T178177
21:56 urandom: draining cassandra instances, restbase1010 - T178177
21:52 mutante: restbase1010 - upgrading firmware - Flashing Smart Array P440ar in Slot 0 [ 3.56 -> 6.06 ]
21:31 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 shenanigans / wmf.11
21:12 demon@tin: Finished scap: bootstrap wmf.11 (duration: 51m 01s)
20:50 twentyafterfour: phabricator: running `sudo bin/garbage collect --collector search.ferret.ngram`
20:29 twentyafterfour: phabricator: running `sudo bin/search ngrams --threshold 0.2`
20:21 demon@tin: Started scap: bootstrap wmf.11
19:50 mutante: forcing puppet ron on ores eqiad to reduce number of celery workers used for stress test
19:37 ppchelko@tin: Finished deploy [eventlogging/eventbus@6ca0372]: Make the kafka async deliver callback thread-safe. T180017 (duration: 01m 29s)
19:36 ppchelko@tin: Started deploy [eventlogging/eventbus@6ca0372]: Make the kafka async deliver callback thread-safe. T180017
19:20 gehel: moving eventlogging collection by logstash from logstash1003 to logstash1007, no messages **should** be lost - T175830
18:24 ppchelko@tin: Finished deploy [eventlogging/eventbus@6ca0372]: Make the kafka async deliver callback thread-safe. Limited to kafka1001. T180017 (duration: 00m 14s)
18:24 ppchelko@tin: Started deploy [eventlogging/eventbus@6ca0372]: Make the kafka async deliver callback thread-safe. Limited to kafka1001. T180017
17:27 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw2118.codfw.wmnet
15:56 awight: beginning stress test on ores* (non-production)
15:54 awight@tin: Finished deploy [ores/deploy@6baed71]: (non-production) Test ORES deployment to ores100[1-2] (duration: 01m 01s)
15:53 awight@tin: Started deploy [ores/deploy@6baed71]: (non-production) Test ORES deployment to ores100[1-2]
15:15 ottomata: restarrting kafka-jumbo brokers, applying SSL (downtime scheduled)
15:03 urandom: bootstrapping cassandra, restbase1014-c - T179422
14:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully repool db1076 (duration: 00m 43s)
14:41 zeljkof: EU SWAT finished
14:40 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Add category collation for sewiki" (T181503) (duration: 00m 44s)
14:15 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable RemexHTML on itwiki and dewiki (T181188 T181190) (duration: 00m 43s)
13:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1076 weight (duration: 00m 44s)
13:14 moritzm: reimaging mw2118/mw2119 (video scalers) to stretch
12:21 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully repool db1034 (duration: 00m 43s)
12:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1076 with low weight (duration: 00m 44s)
11:58 marostegui: Upgrade MariaDB and kernel on db1076
11:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1034 - T178359 ! (duration: 00m 43s)
10:53 moritzm: rebooting meitnerium/archiva.wikimedia.org for update to 4.9.51
10:46 moritzm: rebooting einsteinium (icinga.wikimedia.org) for update to 4.9,51
10:45 elukey: reboot druid1003 for kernel+jvm updates - T179943
10:33 addshore@tin: Synchronized wmf-config/extension-list: extension-list extension.json entrypoint for ArticlePlaceholder (duration: 00m 43s)
10:33 moritzm: rebooting tegmen for update to 4.9,51
10:07 jynus: stopping dbstore1002 (s5) and dbstore2001 (s5) for maintenance
10:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1034 - T178359 ! (duration: 00m 43s)
10:00 ema: cp4021: restart varnish-be due to mbox lag/fetch failures
09:49 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1034 with low weight - T178359 ! (duration: 00m 43s)
09:42 elukey: reboot analytics100[12] for kernel+jvm updates (Hadoop Master nodes) - T179943
09:39 godog: bootstrap restbase1014-b - T179422
09:20 marostegui: Optimize s7 on db1098 - T178359
09:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1076 - T174569 (duration: 00m 43s)
09:01 marostegui: Deploy schema change on db1076 (s2) - T174569
08:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1090 - T174569 (duration: 00m 44s)
08:54 moritzm: enabling test production traffic for mw1259 (stretch-based video scaler)
08:24 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1259.eqiad.wmnet
08:08 moritzm: installing python updates on trusty
07:27 kartik@tin: Finished deploy [cxserver/deploy@4b74f03]: Update cxserver to 1693bcf
07:23 kartik@tin: Started deploy [cxserver/deploy@4b74f03]: Update cxserver to 1693bcf
06:58 marostegui: Stop MySQL on db1034 to clone db1098:3317 - T178359
06:55 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1034 - T178359 ! (duration: 00m 43s)
06:48 marostegui: Fix dbstore1002 replication
06:20 marostegui: Deploy schema change on db1090 (s2) - T174569
06:20 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1090 - T174569 (duration: 00m 43s)
06:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1103:3312 - T174569 (duration: 00m 44s)
02:29 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.10) (duration: 06m 36s)
00:29 mutante: bast4001 - removing ganglia aggregators, package, config...
23:44 mutante: bast3002 - killall -u ganglia to kill all aggregator procs, apt-get remove --purge ganglia-monitor, rm -rf /etc/ganglia, rm -rf /usr/lib/ganglia, apt-get autoremove
23:28 anomie@tin: Synchronized wmf-config/InitialiseSettings.php: Revert wgCommentTableSchemaMigrationStage change, breaks too much stuff (duration: 00m 44s)
22:08 brion: brion running requeueTranscodes.php on terbium to batch-run .mp3 output on Commons (T181749 )
21:27 ppchelko@tin: Finished deploy [cpjobqueue/deploy@c8bea2e]: (no justification provided) (duration: 00m 42s)
21:26 ppchelko@tin: Started deploy [cpjobqueue/deploy@c8bea2e]: (no justification provided)
21:26 awight@tin: Finished deploy [ores/deploy@6baed71]: Update ORES to 6baed71
21:25 eileen: re-enable dedupe job process control commit 461f482
21:13 awight@tin: Started deploy [ores/deploy@6baed71]: Update ORES to 6baed71
21:07 gehel@tin: Finished deploy [kartotherian/deploy@6e223df]: testing new kartotherian packaging on maps-test2003 (duration: 00m 20s)
21:07 gehel@tin: Started deploy [kartotherian/deploy@6e223df]: testing new kartotherian packaging on maps-test2003
20:07 anomie@tin: Synchronized wmf-config/InitialiseSettings.php: Revert wgCommentTableSchemaMigrationStage change, breaks too much stuff (duration: 00m 43s)
19:31 legoktm@tin: Synchronized wmf-config/InitialiseSettings.php: Set wgCommentTableSchemaMigrationStage = MIGRATION_WRITE_BOTH on test wikis (T166733 ) (duration: 00m 43s)
19:18 gehel@tin: Finished deploy [kartotherian/deploy@e166d87]: testing new kartotherian packaging on maps-test2003 (duration: 00m 03s)
19:18 gehel@tin: Started deploy [kartotherian/deploy@e166d87]: testing new kartotherian packaging on maps-test2003
19:17 legoktm@tin: Synchronized wmf-config/InitialiseSettings.php: Set $wgRestrictionMethod = 'firejail'; everywhere (T173370 ) (duration: 00m 43s)
19:11 legoktm@tin: Synchronized static/images/mobile/copyright/: Add converted copyright svg images as png files - https://gerrit.wikimedia.org/r/#/c/394820/ (duration: 00m 43s)
19:06 legoktm@tin: Synchronized wmf-config/InitialiseSettings-labs.php: beta: Add ORES filter thresholds for simplewiki (duration: 00m 43s)
18:46 hoo: Started dumpwikidatajson.sh on snapshot1007 (T181385 )
18:40 hoo: Ran scap pull on snapshot1001 (T181385 )
18:35 hoo@tin: Synchronized php-1.31.0-wmf.10/includes/objectcache/ObjectCache.php: Only send statsd data for WAN cache in non-CLI mode (T181385 ) (duration: 00m 44s)
18:28 godog: bootstrap restbase1014-a - T179422
18:04 gehel@tin: Finished deploy [wdqs/wdqs@2873745]: wdqs GUI update (duration: 02m 10s)
18:02 gehel@tin: Started deploy [wdqs/wdqs@2873745]: wdqs GUI update
17:41 demon@tin: Synchronized php-1.31.0-wmf.10/extensions/CirrusSearch/maintenance/: fix some deprecated spam (duration: 00m 44s)
17:40 ejegg: disabled CiviCRM bounce processing again
17:12 demon@tin: Pruned MediaWiki: 1.31.0-wmf.6 (duration: 02m 29s)
17:08 demon@tin: Pruned MediaWiki: 1.31.0-wmf.5 (duration: 02m 58s)
16:54 demon@tin: Synchronized docroot/noc/conf/dblists: double checking symlink move (duration: 00m 44s)
16:44 ejegg: re-enabled CiviCRM bounced mail processing
16:41 marostegui: Deploy schema change on db1053 (s2) - T174569
16:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1103:3312 - T174569 (duration: 00m 45s)
16:39 demon@tin: Finished scap: docroot/noc/conf/ Adding new dblist symlink (duration: 02m 36s)
16:39 marostegui: Deploy schema change on db1103:3312 - T174569
16:37 demon@tin: Started scap: docroot/noc/conf/ Adding new dblist symlink
16:37 demon@tin: scap aborted: docroot/noc/conf/dblists (duration: 00m 02s)
16:37 demon@tin: Started scap: docroot/noc/conf/dblists
16:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1105:3312 - T174569 (duration: 00m 45s)
16:32 demon@tin: Finished scap: docroot/noc/conf/ drop some dangling symlinks (duration: 03m 53s)
16:28 demon@tin: Started scap: docroot/noc/conf/ drop some dangling symlinks
16:11 herron: re-enabling puppet agents in eqiad
16:07 demon@tin: Finished scap: Revert "Special:Preferences: Use OOjs UI" and follow-ups (duration: 21m 19s)
16:05 herron: re-enabling puppet agents in codfw
15:46 demon@tin: Started scap: Revert "Special:Preferences: Use OOjs UI" and follow-ups
15:31 herron: temporarily disabling puppet agents in eqiad and codfw while puppetdb catches up with command queue
15:29 herron: disabling ircecho temporarily
15:24 _joe_: restarting puppetdb on nihal, will cause puppet failures
15:23 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1259.eqiad.wmnet
15:18 demon@tin: Synchronized wmf-config/CommonSettings-labs.php: no-op (duration: 00m 45s)
15:01 godog: reimage restbase1014 - T179422
14:51 herron: cutting over all production puppet agents to codfw puppet 4 masters via dns
14:50 Amir1: deployed backward compatibility of entity compact diff transmit T113468
14:49 ladsgroup@tin: Synchronized php-1.31.0-wmf.10/extensions/Wikibase: (no justification provided) (duration: 01m 41s)
14:37 gehel@tin: Finished deploy [kartotherian/deploy@e166d87]: dummy kartotherian deployment to test udp2log config change - T175242 (duration: 00m 03s)
14:37 gehel@tin: Started deploy [kartotherian/deploy@e166d87]: dummy kartotherian deployment to test udp2log config change - T175242
14:30 elukey: reboot druid100[23] for kernel updates
14:28 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Localize sitename and meta NS for wawiktionary (T181782) (duration: 00m 46s)
14:01 elukey: reboot analytics106* (hadoop worker nodes) for kernel+jvm updates - T179943
13:24 paravoid: upgrading bast2001 to stretch
13:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1105:3312 - T174569 (duration: 00m 45s)
13:20 marostegui: Deploy schema change on db1105:3312 (s2) - T174569
13:05 marostegui: Compress s6 on db1098 - T178359
12:44 gehel@tin: Finished deploy [kartotherian/deploy@e166d87]: testing new kartotherian packaging on maps-test2003 (duration: 00m 22s)
12:43 gehel@tin: Started deploy [kartotherian/deploy@e166d87]: testing new kartotherian packaging on maps-test2003
11:06 jdrewniak@tin: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 45s)
11:05 jdrewniak@tin: Synchronized portals/prod/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 45s)
10:37 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2085 (duration: 00m 43s)
10:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1096:3316 - T178359wq! (duration: 00m 45s)
09:55 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1099:3318 from s5 (duration: 00m 44s)
09:51 godog: bootstrap restbase1012-c - T179422
09:32 godog: clear erroneous table metrics from graphite1003 / graphite2002 - T181689
09:24 elukey: reboot analytics105* (hadoop worker nodes) for kernel+jvm updates - T179943
09:19 jynus: rebooting mariadb at labsdb1005
09:12 moritzm: reimaging mw1259 (video scaler) to stretch, will be kept disabled initially (some controlled live tests following)
08:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1096:3315 and 3316 - T178359 (duration: 00m 45s)
08:45 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1096:3316 - T178359 (duration: 00m 45s)
08:44 moritzm: updating tor on radium to 0.3.1.9
08:41 moritzm: updating tor packages to 0.3.1.9
08:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1096:3315 and pool db1096:3316 - T178359 (duration: 00m 45s)
08:12 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db1096:3315 - T178359 (duration: 00m 44s)
08:11 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1096:3315 - T178359 (duration: 00m 45s)
07:53 moritzm: installing curl security updates
07:17 marostegui: Compress s1 on db1099 - T178359
07:08 marostegui: Stop MySQL on db1044 as it will be decommissioned - T181696
07:05 _joe_: playing with puppetdb status for ores2003 (deactivating/reactivating node)
06:40 marostegui: Stop MySQL on db1098 to clone db1096.s6 - T178359
06:39 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1044 from config as it will be decommissioned - T181696 (duration: 00m 45s)
06:38 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1044 from config as it will be decommissioned - T181696 (duration: 00m 45s)
06:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1098 - T178359 (duration: 00m 46s)
06:21 marostegui: Deploy alter table on s3 master (db1075) without replication - T174569
02:32 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.10) (duration: 06m 28s)
15:33 ejegg: disabled CiviCRM bounce processing job
12:17 akosiaris: empty ganeti1006, it had issues this morning per T181121
12:06 marostegui: Fix dbstore1002 replication
07:44 akosiaris: ran puppet on conf2002, etcdmirror-conftool-eqiad-wmnet got started again
05:11 andrewbogott: deleting files on labsdb1003 /srv/tmp older than 30 days
03:57 no_justification: gerrit2001: icinga is flapping on the gerrit process/systemd check, but this is kind of known (not sure why it's doing this all of a sudden). It's not letting me acknowledge it, but it's fine/harmless. Cf T176532
17:55 marostegui: Reboot db1096.s5 to pick up the correct innodb_buffer_pool size after finishing compressing s5 - T178359
03:51 hoo: Ran "scap pull" on snapshot1001, after final T181385 tests
00:03 mutante: tried one more time on db2028,db2029, both trusty. on db2028: gmond was running as user ganglia-monitor, failed, had to manually kill the process, run puppet again then ok. on db2029, gmond was running as "499" but puppet just ran and removed it without manual intervention. (T177225 )
23:15 urandom: starting cassandra bootstrap, restbase1012-b - T179422
21:49 mutante: db2029 - removing ganglia-monitor, testing to kill gmond, running puppet to figure out how to cleanly remove it on trusty
21:12 mutante: db2023 killed gmond (ganglia-monitor) process manually which was still running even though ganglia-monitor package was removed and caused puppet breakage (it seems only on trusty). after that puppet run is clean again and ganglia removed. (T177225 ) (https://gerrit.wikimedia.org/r/#/c/394647/1 )
20:18 awight@tin: Started deploy [ores/deploy@9afbf14]: (non-production) Test ORES deployment to ores100*
20:17 awight@tin: Finished deploy [ores/deploy@9afbf14]: (non-production) Test ORES deployment to ores1001 (duration: 02m 31s)
20:15 awight@tin: Started deploy [ores/deploy@9afbf14]: (non-production) Test ORES deployment to ores1001
20:03 aaron@tin: Synchronized php-1.31.0-wmf.10/includes/libs/objectcache/WANObjectCache.php: f096d0b465b75d
18:59 demon@tin: Synchronized wmf-config/CommonSettings-labs.php: no-op (duration: 00m 46s)
18:22 mutante: Phabricator: restarting Apache for php-curl update
18:21 _joe_: restarting apache2 on the codfw puppetmasters
18:06 marktraceur@tin: Synchronized php-1.31.0-wmf.10/extensions/UploadWizard/resources/controller/uw.controller.Deed.js: (no justification provided) (duration: 00m 46s)
17:49 mutante: phab2001 - restarted apache
17:33 herron: stopped ircecho on einsteinium
17:00 awight@tin: Unlocked for deployment [ores/deploy]: Don't deploy while we're messing with git-lfs (duration: 00m 14s)
17:00 awight@tin: Locking from deployment [ores/deploy]: Don't deploy while we're messing with git-lfs (planned duration: 16666666666m 39s)
17:00 awight@tin: Locking from deployment [ores/deploy]: Don't deploy while we're messing with git-lfs (planned duration: -1m 59s)
16:59 awight@tin: Unlocked for deployment [ores/deploy]: Don't deploy while we're messing with git-lfs (duration: 00m 07s)
16:59 awight@tin: Locking from deployment [ores/deploy]: Don't deploy while we're messing with git-lfs (planned duration: 60m 00s)
16:34 jynus: stopping db2092 to clone s1 to db2085
16:24 urandom: starting cassandra bootstrap, restbase1012-a -- T179422
15:27 godog: bounce uwsgi on labmon1001 - stuck
15:21 moritzm: installing nspr security updates on trusty
15:17 moritzm: installing ffmpeg security updates
15:14 gehel@tin: Finished deploy [kartotherian/deploy@df7ebff]: testing new kartotherian packaging on maps-test2003 (duration: 00m 20s)
15:14 jynus@tin: Synchronized wmf-config/db-codfw.php: Undeploy db2092, use db2085 for s1 (duration: 00m 45s)
15:14 gehel@tin: Started deploy [kartotherian/deploy@df7ebff]: testing new kartotherian packaging on maps-test2003
15:14 moritzm: installing libxcursor security updates on trusty
15:09 jynus@tin: Synchronized wmf-config/db-eqiad.php: Undeploy db2092, use db2085 for s1 (duration: 00m 45s)
14:24 awight@tin: Finished deploy [ores/deploy@532bd0b]: (non-production) Update ORES on new cluster (duration: 02m 06s)
14:22 awight@tin: Started deploy [ores/deploy@532bd0b]: (non-production) Update ORES on new cluster
14:22 akosiaris: upload apertium-crh-tur_0.3.0~r83159-1+wmf1 to apt.wikimedia.org/jessie-wikimedia component main. T181465
14:10 herron: cutting all puppet service records over to codfw puppet 4 masters
12:44 elukey: reboot druid1001 for kernel+jvm updates - T179943
12:11 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2055 (duration: 00m 45s)
11:59 jynus: restarting and upgrading mysql on labsdb1004
11:28 jynus: upgrading and restarting dbstore2001
11:14 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2055 (duration: 00m 46s)
11:08 marostegui: Stop MySQL on db2055 for testing
11:04 marostegui: Update MySQL on db1039 for testing
10:57 elukey: reboot analytics1028 for kernel + jvm updates (Hadoop HDFS journalnode) - T179943
10:56 godog: delete docker diskspace metrics from labs - T181476
10:21 godog: initial purge of old table metrics from graphite2002 - T181689
10:18 moritzm: reimaging mw1259 (video scaler) to stretch, will be kept disabled initially (with some live tests starting next week)
09:55 akosiaris: run memtester 61G on ganeti1008 T181121
09:23 elukey: reboot analytics104* for kernel+jvm updates - T179943
09:23 akosiaris: powercycle ganeti1008 T181121 , it's largely unresponsive
08:41 marostegui: Remove db1046 and db1047 from tendril - T156844
08:40 elukey: reboot the remaining analytics103* hadoop workers to pick up kernel+jvm updates - T179943
08:28 akosiaris: empty ganeti1008, move VMs to ganeti1006 T181121
08:20 akosiaris: repool ganeti1005, ganeti1006 to empty ganeti1008 T181121
07:51 akosiaris: upload apertium-tur_0.2.0~r83161-1+wmf1, apertium-crh_0.2.0~r83161-1+wmf1 to apt.wikimedia.org/jessie-wikimedia component main. T181465
07:14 marostegui: Logging retroactively for the record, restarting MySQL on db1039
05:14 legoktm@tin: Synchronized wmf-config/InitialiseSettings.php: touch (duration: 00m 44s)
01:46 hoo: Ran scap pull on mwdebug1001 after T181385 testing
00:13 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: HTML5 sections be upon us! (duration: 00m 45s)
00:04 hoo: Killed all remaining Wikidata JSON/RDF dumpers, due to T181385 . This means no dumps this week!
23:54 demon@tin: Synchronized wmf-config/extension-list-labs: no-op (duration: 00m 45s)
23:39 addshore@tin: Synchronized wmf-config: wdbuild: T173818 T177060 Add wikidata extensions to extension-list (duration: 00m 46s)
23:35 addshore@tin: Synchronized wmf-config: wdbuild: T173818 Remove Wikibase-buildentry.php config file (empty) (duration: 00m 46s)
23:34 addshore@tin: Synchronized wmf-config/Wikibase.php: wdbuild: T173818 Remove Wikibase-buildentry.php config file (empty) (duration: 00m 45s)
23:30 addshore@tin: Synchronized wmf-config: wdbuild: T173818 Remove wmgUseWikidataBuild (duration: 00m 46s)
23:29 addshore@tin: Synchronized wmf-config/Wikibase-buildentry.php: wdbuild: T173818 Remove wmgUseWikidataBuild (duration: 00m 45s)
23:21 addshore@tin: Synchronized wmf-config: wdbuild: T173818 wdbuild: Stop loading from build on ALL WIKIS The build is dead! Mwahahaaa (duration: 00m 47s)
23:18 bsitzmann@tin: Finished deploy [mobileapps/deploy@4305d96]: Update mobileapps to 4317ea5 T181743 ) (duration: 04m 50s)
23:15 addshore@tin: Synchronized wmf-config: wdbuild: wdbuild: Stop loading from build on all wikis (except the one i really dont want to break) (duration: 00m 46s)
23:14 bsitzmann@tin: Started deploy [mobileapps/deploy@4305d96]: Update mobileapps to 4317ea5 T181743 )
23:06 addshore@tin: Synchronized wmf-config: wdbuild: wdbuild: Stop loading from build on group1 (duration: 00m 46s)
22:55 addshore@tin: Synchronized wmf-config: wdbuild: wdbuild: Stop loading from build on group0 (duration: 00m 46s)
22:42 addshore: BETA ONLY was a lie on that last one ...
22:41 addshore@tin: Synchronized wmf-config: wdbuild: BETA ONLY wdbuild: Stop loading from build on test and testwikidata (duration: 00m 47s)
22:40 demon@tin: Pruned MediaWiki: 1.31.0-wmf.8 [keeping static files] (duration: 02m 33s)
22:27 addshore@tin: Synchronized wmf-config: wdbuild: BETA ONLY gerrit:394411 (duration: 00m 47s)
21:35 addshore@tin: Synchronized wmf-config: wdbuild: BETA ONLY (duration: 00m 47s)
21:31 jynus: upgrading labsdb1010 and restarting mariadb
21:23 mutante: powercycling kafka1018 (was down in Icinga and saw in SAL: reboot kafka10[12-22] for kernel + jvm updates - T179943 )
21:18 jynus: upgrade and restart db2078
21:14 addshore@tin: Synchronized wmf-config: wdbuild: BETA ONLY (duration: 00m 47s)
21:01 addshore@tin: Synchronized wmf-config: wdbuild: T173818 : add switch to ease killing (again) (duration: 00m 46s)
20:59 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: wdbuild: T173818 : add switch to ease killing (again) (duration: 00m 45s)
20:49 XenoRyet: Updated tools from 6e604fd 626fe02
20:40 bd808: Sending Toolforge survey final reminder emails from silver for T177126
20:32 addshore@tin: Synchronized wmf-config: REVERT wdbuild: T173818 : add switch to ease killing (duration: 00m 47s)
20:30 addshore@tin: Synchronized wmf-config: wdbuild: T173818 : add switch to ease killing (duration: 00m 47s)
20:13 ejegg: re-adjusted Civi job timings. QC every odd min for 105 sec, TY every even min for 70 sec after 45 sec delay
20:12 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.10
19:55 ejegg: 105 seconds for each job
19:55 ejegg: adjusted timings of Civi jobs to let TY and QC run concurrently
19:37 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Temporary disable remex html" T178632 (duration: 00m 49s)
19:20 thcipriani@tin: Synchronized php-1.31.0-wmf.10/includes/htmlform/fields/HTMLMultiSelectField.php: SWAT: HTMLMultiSelectField: Allow formatting in section headings in OOUI mode T181698 (duration: 00m 49s)
19:12 thcipriani@tin: Synchronized wmf-config: SWAT: Enable MP3 uploads on Commons T120288 (duration: 00m 51s)
17:57 andrewbogott: upgrading labtestpuppetmaster2001 to puppet 4.8
17:47 moritzm: uploaded prometheus-openldap-exporter 0+git20171128-1 for jessie-wikimedia (T181511 )
17:06 herron: beginning cut over of esams to codfw puppet 4 masters
16:44 urandom: drop (erroneous) legacy tables from -ng cassandra cluster - T181689
16:12 elukey: drain and reboot analytics1031->39 to pick up jvm+kernel updates - T179943
15:59 jynus: setup prometheus with unix_socket on new server db1107 and db1108
15:19 gehel: restart blazegraph on wdqs1004
15:16 jynus@tin: Synchronized wmf-config/db-eqiad.php: Increase db1110 load (duration: 00m 48s)
15:11 herron: beginning cut over of ulsfo to codfw puppet 4 masters
14:50 zeljkof: EU SWAT finished
14:46 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable WikiLove Extension on pa.wiki (T178919) (duration: 00m 49s)
14:28 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add import sources to de.wiki (T181695) (duration: 00m 49s)
14:20 zfilipin@tin: Synchronized php-1.31.0-wmf.10/extensions/ContentTranslation/modules/dashboard/ext.cx.dashboard.js: SWAT: Set default languages after fetching valid languages (duration: 00m 49s)
14:19 marostegui: Deploy schema change on s3 - db1078 - T174569
13:23 addshore@tin: Synchronized php-1.31.0-wmf.10/extensions/AdvancedSearch/modules/ext.advancedSearch.init.js: pre-swat: T181644 Force search profile advanced in AdvancedSearch gerrit:394297 (duration: 00m 48s)
13:22 addshore@tin: Synchronized php-1.31.0-wmf.8/extensions/AdvancedSearch/modules/ext.advancedSearch.init.js: pre-swat: T181644 Force search profile advanced in AdvancedSearch gerrit:394299 (duration: 00m 50s)
13:18 herron: cutting codfw puppet agents over to puppetmaster2001.codfw.wmnet
12:07 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Tackle s4 DB weights to make them more equal (duration: 00m 48s)
11:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1081 and reduce traffic for db1091 (duration: 00m 50s)
10:33 jynus: stopping replication on db1044 and db1095 (s3)
09:16 moritzm: installing exim security updates on stretch (jessie/trusty not affected)
09:14 elukey: drain and reboot analytics1029/1030 for jvm+kernel updates (Hadoop worker canaries)
09:05 godog: add 200G of space to graphite2002 carbon lv
08:25 moritzm: rolling restart of mw canaries to pick up curl security update
08:21 marostegui: Enable GTID on s8 eqiad hosts that do not have it enabled (db1109, db1104, db1101, db1092, db1087, db1063) - T177208
07:48 moritzm: installing curl security updates
07:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1056 (duration: 01m 18s)
06:48 marostegui: Deploy schema change on dbstore1001 s3 - T174569
06:26 marostegui: Deploy schema change on s3 db1077 - T174569
02:48 ebernhardson@tin: Finished deploy [search/mjolnir/deploy@7aa39b7]: (no justification provided) (duration: 03m 15s)
02:45 ebernhardson@tin: Started deploy [search/mjolnir/deploy@7aa39b7]: (no justification provided)
02:29 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.8) (duration: 08m 43s)
01:17 ejegg: updated fundraising tools from 6d4b6f3 6e604fd
01:06 twentyafterfour: finished phabricator upgrade, service is online and appears to be functioning normally
01:03 twentyafterfour: Starting phabricator upgrade & maintenance. Service will be offline for less than 5 minutes.
01:02 bsitzmann@tin: Finished deploy [mobileapps/deploy@fa2a877]: Update mobileapps to dcea7d3 T181004 ) (duration: 06m 08s)
00:56 bsitzmann@tin: Started deploy [mobileapps/deploy@fa2a877]: Update mobileapps to dcea7d3 T181004 )
00:53 maxsem@tin: Finished scap: Message updates for https://gerrit.wikimedia.org/r/#/c/394155/ (duration: 26m 38s)
00:48 ejegg: re-enabled CiviCRM jobs
00:42 ejegg: updated CiviCRM from 0f95f3e e81228f
00:40 ejegg: disabled CiviCRM jobs
00:31 ejegg: updated fundraising dashboard from 6ee6567 1141317
00:27 maxsem@tin: Started scap: Message updates for https://gerrit.wikimedia.org/r/#/c/394155/
00:24 maxsem@tin: Synchronized php-1.31.0-wmf.10/extensions/ContentTranslation/: https://gerrit.wikimedia.org/r/#/c/394206/ (duration: 00m 52s)
00:21 ejegg: added weekly Ingenico audit processing job in makemissing mode
23:20 eileen: update civicrm from a1022cf 0f95f3e
22:23 hashar: Nodepool had some troubles spawning new instances from 21:09 to 21:36, and took a while to recover. Issue similar to T170492 #3581822
21:51 SMalyshev: starting wikidata reindex (T181426 )
21:23 no_justification: docroot sync was for If1afa59a
21:21 demon@tin: Synchronized docroot/: (no justification provided) (duration: 00m 49s)
20:29 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.10
20:26 demon@tin: Synchronized php: symlink bump (duration: 00m 48s)
20:05 urandom: restarting cassandra bootstrap of restbase1012-a (T179422 )
19:55 mutante: restarting gerrit to apply config change and set gitBasicAuth to true to unblock T171758 (gerrit:391865)
19:50 herron: beginning rolling cut over of eqiad scb hosts to codfw puppet 4 masters
19:15 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/394059/3 (duration: 00m 49s)
18:57 herron: beginning cutover of codfw lvs2* systems to codfw puppet 4 masters. first standby nodes, then active nodes
18:12 marostegui: Compress s5 on db1096 - T178359
18:10 jynus@tin: Synchronized wmf-config/db-eqiad.php: Increase db1110 load (duration: 00m 53s)
18:05 awight@tin: Finished deploy [ores/deploy@532bd0b]: (non-production) Update ORES on new cluster (duration: 01m 11s)
18:04 awight@tin: Started deploy [ores/deploy@532bd0b]: (non-production) Update ORES on new cluster
18:03 herron: beginning cut over of codfw cp2* servers to codfw puppet 4 masters
17:40 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool db1110 with low load (duration: 00m 48s)
17:40 godog: bootstrapping restbase1012-a - T179422
17:19 herron: beginning cut over of cp200[12] to codfw puppet 4 masters
17:16 ejegg: increased time limit for donation queue consumer from 60 to 70 seconds, reduced thank you job from 55 to 45 seconds
17:03 ejegg: re-enabled Civi jobs
16:58 ejegg: disabled Civi jobs for Deadlock-retry update
16:46 jynus@tin: Synchronized wmf-config/db-codfw.php: Update db1110 ip (duration: 00m 50s)
16:43 jynus@tin: Synchronized wmf-config/db-eqiad.php: Update db1110 ip (duration: 00m 49s)
15:02 chasemp: purge old qcow2 images from under /home on labtestcontrol2001
15:00 awight: Restarting ORES celery workers manually, T181538
14:39 _joe_: reloading apache on puppetmasters to pick up the configuration changes
14:37 awight@tin: Started restart [ores/deploy@e58bfbf]: Restart ORES services (take 2), T181538
14:36 elukey: reboot druid100[456] for jvm+kernel updates - T179943
14:35 awight@tin: Finished deploy [ores/deploy@e58bfbf]: Restart ORES services, T181538 (duration: 00m 16s)
14:35 awight@tin: Started deploy [ores/deploy@e58bfbf]: Restart ORES services, T181538
14:25 zeljkof: EU SWAT finished
14:18 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: Define throttle rule (T181367) (duration: 00m 49s)
14:14 herron: beginning cut over of cache::misc and codfw cache::canary cp servers to codfw puppet4 masters
14:13 moritzm: rebooting neodymium for kernel update to 4.9.51
14:12 godog: reimage restbase1012 - T179422
14:10 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT T180291 T180128 AdvancedSearch for arwiki (duration: 00m 49s)
14:07 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT T180128 AdvancedSearch for dewiki (duration: 00m 50s)
13:34 awight@tin: Finished deploy [ores/deploy@e58bfbf]: (non-production) Update ORES on new cluster (take 3) (duration: 16m 49s)
13:18 elukey: reboot kafka100[23] for jvm+kernel updates - T179943
13:17 awight@tin: Started deploy [ores/deploy@e58bfbf]: (non-production) Update ORES on new cluster (take 3)
12:34 jynus@tin: Synchronized wmf-config/db-eqiad.php: Setup s8 on eqiad, with no wikis (duration: 00m 48s)
12:24 jynus: scap pull on mwdebug1001
12:23 jynus@tin: Synchronized wmf-config/db-codfw.php: Pool db1096:3315 (duration: 00m 48s)
12:01 marostegui: Deploy schema change on db1072 (sanitarium master) on s3 with replication enabled to replicate to labs - T174569
11:30 elukey: reboot kafka1001 for kernel + jvm updates - T179943
10:15 akosiaris: disable puppet on oresrdb* for merging https://gerrit.wikimedia.org/r/#/c/394022/ . T181563
09:50 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1110 (duration: 00m 48s)
09:46 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1110 (duration: 00m 49s)
09:39 godog: upload cassandra-tools-wmf 1.0.2-1 - T181438
09:28 mobrovac@tin: Finished deploy [electron-render/deploy@94d27d7]: Update to electron v1.7.9 and start using the Charter font - T181200 (duration: 04m 03s)
09:24 mobrovac@tin: Started deploy [electron-render/deploy@94d27d7]: Update to electron v1.7.9 and start using the Charter font - T181200
09:04 godog: bootstrap restbase1007-c - T179422
08:41 moritzm: uploaded prometheus-openldap-exporter 0+git20171128-1 for jessie-wikimedia (T181511 )
08:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1096 - T178359 (duration: 00m 48s)
07:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1099:3318 and db1055 - T178359 (duration: 00m 48s)
07:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1099:3318 traffic - T178359 (duration: 00m 45s)
07:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1099:3318 with low weight - T178359 (duration: 00m 45s)
06:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1055 - T178359 (duration: 00m 48s)
06:42 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1099:3311 and db1099:3318 to the config (depooled) T178359 (duration: 00m 48s)
06:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1099:3311 and db1099:3318 to the config (depooled) T178359 (duration: 00m 49s)
06:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1055 with lower weight - T178359 (duration: 00m 50s)
06:17 ebernhardson@tin: Finished deploy [search/mjolnir/deploy@7aa39b7]: (no justification provided) (duration: 04m 27s)
06:12 ebernhardson@tin: Started deploy [search/mjolnir/deploy@7aa39b7]: (no justification provided)
03:39 ejegg: reduced CiviMail sample rate to 0.35
03:09 ejegg: reduced donation queue consumer time limit from 70 to 60 seconds, increased ty mail batch time limit from 45 to 55 seconds
02:50 ejegg: reduced donation queue consumer time limit from 75 to 70 seconds, increased ty mail batch time limit from 40 to 45 seconds
02:44 ejegg: reduced CiviMail record creation rate from 100% to 50%
02:24 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.8) (duration: 06m 08s)
01:25 mutante: snapshot1001 - closed idle screen session
01:22 mutante: analytics1003 - closed idle screen session
01:21 mutante: mw1276 - run "scap pull" to get in sync after hardware issue, then pooled again (T181397 )
01:09 mutante: restarting ircecho - it stopped talking
01:07 mutante: forcing puppet run on all labvirt* machines to clean out Icinga alerts
00:44 awight@tin: Synchronized php-1.31.0-wmf.10/extensions/ORES: Hotfix to mitigate cache stampeding, T181567 (duration: 00m 49s)
00:33 reedy@tin: Synchronized php-1.31.0-wmf.10/includes/logging/LogPager.php: Fix fatal on Special:Log T181565 (duration: 00m 48s)
00:32 awight@tin: Synchronized php-1.31.0-wmf.8/extensions/ORES: Hotfix to mitigate cache stampeding, T181567 (duration: 00m 50s)
00:18 Jamesofur: deleted 6 archived files from servers for legal compliance
00:08 ejegg: updated fundrasing dashboard from df94248 6ee6567
23:58 hoo: Ran scap pull on mwdebug1001 after T181385 related testing
23:07 akosiaris@tin: Synchronized wmf-config/CommonSettings.php: T181538 (duration: 00m 49s)
22:31 akosiaris@tin: Synchronized wmf-config/CommonSettings.php: (no justification provided) (duration: 00m 49s)
22:31 akosiaris: deploy wmf-config/CommonSettings.php for ORES internal discovery URL, https://gerrit.wikimedia.org/r/#/c/393924/ T181538
21:41 akosiaris: disable ORES queue redis persistency by config set appendonly no on oresrdb1001
21:24 hoo: Manually killed all remaining Wikidata TTL (RDF) dumpers on snapshot1007. Some shards failed due to the db1110 depool.
21:23 hoo: Manually killed all remaining Wikidata JSON dumpers on snapshot1007. Some shards failed due to the db1110 depool.
20:56 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: removing aawiki from group0
20:42 gehel: repooling elastic2004 after RAID controller maintenance - T181412
20:42 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.10
20:28 mutante: forcing puppet run on cache misc to revert "failover ORES to codfw"
20:19 demon@tin: Synchronized scap/plugins/prep.py: no-op (duration: 00m 48s)
20:16 demon@tin: Synchronized dblists/group0.dblist: adding some new wikis (duration: 00m 48s)
18:57 demon@tin: Finished scap: bootstrap wmf.10 (duration: 35m 20s)
18:51 demon@tin: Finished deploy [gerrit/gerrit@571cf4c]: deploying 2.15+ polygerrit style changes (duration: 00m 09s)
18:51 demon@tin: Started deploy [gerrit/gerrit@571cf4c]: deploying 2.15+ polygerrit style changes
18:40 akosiaris: revert weight changes for scb1001, scb1002 T181835
18:39 akosiaris@puppetmaster1001: conftool action : set/weight=10; selector: scb1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
18:39 akosiaris@puppetmaster1001: conftool action : set/weight=10; selector: scb1001.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
18:35 awight@tin: Finished deploy [ores/deploy@e58bfbf]: (non-production) Update ORES on new cluster (take 2) (duration: 04m 30s)
18:32 akosiaris: force puppet run on cache::misc boxes T181538
18:30 awight@tin: Started deploy [ores/deploy@e58bfbf]: (non-production) Update ORES on new cluster (take 2)
18:28 urandom: (re)bootstrapping cassandra, restbase1007-b - T179422
18:22 demon@tin: Started scap: bootstrap wmf.10
18:18 awight@tin: Finished deploy [ores/deploy@e58bfbf]: (non-production) Update ORES on new cluster (duration: 01m 41s)
18:17 awight@tin: Started deploy [ores/deploy@e58bfbf]: (non-production) Update ORES on new cluster
18:00 akosiaris: force stop celery-ores-worker on scb1001
17:58 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: scb1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
17:58 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: scb1001.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
17:46 urandom: decommissioning cassandra, restbase1007-b - T179422
17:42 urandom: restart cassandra, restbase1007, to pickup logstash java deps - T179422
16:45 ejegg: updated fundraising dashboard from d8c86e7 df94248
16:25 bblack: mw1329 boot to PXE (should come up with new .66 IP)
15:39 jynus@tin: Synchronized wmf-config/db-eqiad.php: depool db1110 (duration: 00m 44s)
15:38 bblack: powered off mw1329
15:06 marostegui: Compress s5 on db1099
15:04 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Revert "Comply wikidata with new ores thresholds" (duration: 00m 45s)
14:56 marostegui: Deploy schema change on s3 on dbstore1002 - T174569
14:45 gehel@tin: Finished deploy [kartotherian/deploy@cb9b1ef]: testing new kartotherian packaging on maps-test2003 (duration: 00m 02s)
14:45 gehel@tin: Started deploy [kartotherian/deploy@cb9b1ef]: testing new kartotherian packaging on maps-test2003
14:41 otto@tin: Finished deploy [eventlogging/analytics@c464b8c]: Fixing bug where userAgent set by client producer was not used T178440 (duration: 00m 02s)
14:40 otto@tin: Started deploy [eventlogging/analytics@c464b8c]: Fixing bug where userAgent set by client producer was not used T178440
14:40 otto@tin: Started deploy [eventlogging/analytics@c464b8c]: Fixing bug where userAgent set by client producer was not used
14:39 gehel@tin: Finished deploy [kartotherian/deploy@cb9b1ef]: testing new kartotherian packaging on maps-test2003 (duration: 00m 18s)
14:39 gehel@tin: Started deploy [kartotherian/deploy@cb9b1ef]: testing new kartotherian packaging on maps-test2003
14:37 marostegui: Stop MySQL on db1055 to clone db1099:3311 - T178359
14:34 gehel@tin: Finished deploy [kartotherian/deploy@55c5da4]: (no justification provided) (duration: 00m 11s)
14:34 gehel@tin: Started deploy [kartotherian/deploy@55c5da4]: (no justification provided)
14:21 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1055 - T178359 (duration: 00m 44s)
14:17 elukey: reboot kafka10[12-22] for kernel + jvm updates - T179943
14:03 elukey: reboot kafka200[123] for kernel + jvm updates - T179943
12:25 godog: cleanup wanobjectcache metrics with hashes - T178531
12:14 godog: bounce carbon-frontend-relay after https://gerrit.wikimedia.org/r/393749
11:25 hoo: Manually re-started Wikidata JSON dumps on snapshot1007, got stuck after db1082 went down.
10:55 jynus: stop db1044 replication for db1095 master switchover
10:49 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1082. Depool db1044 (duration: 00m 44s)
10:44 mobrovac@tin: Synchronized wmf-config/jobqueue.php: Process wikibase-addUsagesForPage only via EventBus - T175212 (duration: 00m 44s)
10:08 ppchelko@tin: Finished deploy [cpjobqueue/deploy@c4b9e16]: Enable wikibase-addUsagesForPage with low concurrency (duration: 00m 29s)
10:07 ppchelko@tin: Started deploy [cpjobqueue/deploy@c4b9e16]: Enable wikibase-addUsagesForPage with low concurrency
09:35 jynus: restart db1087 for maintenance
09:22 jynus: restart db1082
09:19 godog: unmask and restart restbase1007-b - T179422
09:15 moritzm: installibg libxml-libxml-perl security updates on trusty (Debian already fixed)
09:11 ppchelko@tin: Finished deploy [cpjobqueue/deploy@b0b1793]: Remove double-processing created for consumer group renaming (duration: 00m 28s)
09:10 ppchelko@tin: Started deploy [cpjobqueue/deploy@b0b1793]: Remove double-processing created for consumer group renaming
09:04 marostegui: Drop database log from dbstore1002 - T156844
08:57 ppchelko@tin: Finished deploy [cpjobqueue/deploy@2212086]: Move enabled jobs config to vars.yaml (duration: 00m 39s)
08:56 ppchelko@tin: Started deploy [cpjobqueue/deploy@2212086]: Move enabled jobs config to vars.yaml
08:38 jynus@tin: Synchronized wmf-config/db-eqiad.php: depool db1082 (duration: 00m 44s)
08:02 mobrovac: bootstrap restbase1007-b - T179422
06:39 marostegui: Stop MySQL on db1099 to copy its content to dbstore1001 and reimage it as multi-instance - T178359
05:23 mutante: labtestpuppetmaster2001 - manually purging ganglia things since puppet is disabled
05:05 mutante: labweb1001/labweb1002: manually purging ganglia package/config/service/unit files because puppet is disabled there (T177225 )
04:08 eileen: update process-control to b40eaec
03:58 demon@tin: Pruned MediaWiki: 1.31.0-wmf.4 (duration: 03m 13s)
03:31 mutante: labservices1001/1002,labtestservices2001 - remove pdns_gmetric cronjobs causing cron spam after ganglia decom from lab*
02:20 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.8) (duration: 05m 23s)
22:29 mutante: labtestnet2001 - re-enabled puppet to decom ganglia, no errors
22:22 mutante: decom'ing Ganglia from all labtest* hosts - packages removed by puppet etc, might cause some false positives in Icinga but it's ok (T177225 )
22:21 otto@tin: Finished deploy [eventlogging/eventbus@e024af3]: deploy revert checked out at 3df06ab T180017 (duration: 00m 10s)
22:21 otto@tin: Started deploy [eventlogging/eventbus@e024af3]: deploy revert checked out at 3df06ab T180017
21:59 awight: clearing ORES threshold caches for ruwiki, frwiki, wikidatawiki.
21:59 awight@tin: Synchronized wmf-config/InitialiseSettings.php: Reenable ORES on frwiki, ruwiki, and wikidata; T181006 (duration: 00m 45s)
21:53 awight@tin: Finished deploy [ores/deploy@e58bfbf]: (codfw) Update ORES for impossible threshold handling and new wikidata editquality model, T179711 T180686 T180450 (duration: 05m 06s)
21:47 awight@tin: Started deploy [ores/deploy@e58bfbf]: (codfw) Update ORES for impossible threshold handling and new wikidata editquality model, T179711 T180686 T180450
21:46 awight@tin: Finished deploy [ores/deploy@e58bfbf]: Update ORES for impossible threshold handling and new wikidata editquality model, T179711 T180686 T180450 (duration: 12m 55s)
21:33 awight@tin: Started deploy [ores/deploy@e58bfbf]: Update ORES for impossible threshold handling and new wikidata editquality model, T179711 T180686 T180450
21:23 awight@tin: Synchronized php-1.31.0-wmf.8/extensions/ORES: ORES error handling for bad thresholds, T181191 (duration: 00m 46s)
21:11 awight@tin: Synchronized wmf-config/InitialiseSettings.php: Temporarily disable ORES on wikidata (duration: 00m 45s)
21:10 awight: Previous “rollback ORES” was from a stale screen session, no actions were taken today.
21:08 awight@tin: Finished deploy [ores/deploy@82a13ae]: Rollback ORES (take 3); 181006 (duration: 9942m 42s)
20:21 demon@tin: i lied, that was for wgStyleVersion removal
20:20 demon@tin: Synchronized wmf-config/CommonSettings.php: cli sapi fixes (duration: 00m 45s)
20:17 ejegg: updated payments-wiki from fea6b37 f594dfa
20:03 kaldari@tin: Synchronized wmf-config/InitialiseSettings-labs.php: (no justification provided) (duration: 00m 45s)
20:02 kaldari@tin: Synchronized wmf-config/CommonSettings.php: (no justification provided) (duration: 00m 45s)
20:01 kaldari@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 46s)
19:54 bblack: crN-ulsfo: remove lvs400[1-4] from PyBal BGP neighbors list
19:42 catrope@tin: Synchronized php-1.31.0-wmf.8/resources/src/mediawiki.rcfilters/mw.rcfilters.UriProcessor.js: T181100 (duration: 00m 45s)
19:41 demon@tin: Synchronized wmf-config/InitialiseSettings.php: babel officewiki thing (duration: 00m 45s)
19:39 catrope@tin: Synchronized php-1.31.0-wmf.8/includes/specials/SpecialRecentchangeslinked.php: T181100 (duration: 00m 45s)
19:34 otto@tin: Finished deploy [eventlogging/eventbus@3df06ab]: Temp deploy to kafka1001 only to catch bug: T180017 (duration: 00m 12s)
19:34 otto@tin: Started deploy [eventlogging/eventbus@3df06ab]: Temp deploy to kafka1001 only to catch bug: T180017
19:21 catrope@tin: Synchronized wmf-config/Wikibase.php: T176903 (duration: 00m 45s)
19:17 AaronSchulz: Removed more bogus md5 wanobjecache/ metric from graphite[12]001
19:11 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: T179241 (duration: 00m 45s)
19:02 volans: restarted ircecho
18:51 ejegg: updated payments-wiki from 6b3019b fea6b37
18:50 dcausse: elastic/cirrus: reindexing english group2 wikis: T179945
18:45 volans: stopped ircecho temporary to avoid spam
18:14 XioNoX: pushing v6 gre firewall filter
18:12 gehel@tin: Finished deploy [wdqs/wdqs@7ad20f3]: (no justification provided) (duration: 02m 10s)
18:11 gehel: deploying blazegraph + GUI updates
18:10 gehel@tin: Started deploy [wdqs/wdqs@7ad20f3]: (no justification provided)
18:08 hoo: Killed truthy nt dumpers stuck at 100% CPU (from last week) on snapshot1007 (T181385 )
18:06 akosiaris: poweroff ganeti1005, ganeti1006 T181121
18:05 bd808: Sending Toolforge survey 1 week reminder emails from silver for T177126
16:07 bblack: lvs400x - puppet disabled for https://gerrit.wikimedia.org/r/#/c/393610
15:52 herron: beginning canary cutover/deployment of codfw prometheus servers to codfw puppet 4 puppetmasters
15:19 chasemp: disable puppet across cloud things for cleanup
15:11 herron: beginning canary cutover/deployment of codfw elasticsearch servers to codfw puppet 4 puppetmasters
14:22 zeljkof: EU SWAT finished
14:19 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: IP cap lift for Semaine contributive 2017-2018 (T181360) (duration: 00m 45s)
14:10 addshore@tin: Synchronized php-1.31.0-wmf.8/extensions/AdvancedSearch: SWAT AdvancedSearch T181175 T181222 , adjust links in beta section & fix usability of mobile search (duration: 00m 46s)
13:31 moritzm: installing nspr security updates on trusty
13:22 elukey: remove eventlogging replication support (log database) from dbstore1002 - T156844
13:17 moritzm: updating nginx on francium
13:01 Pchelolo: stop cpjobqueue in eqiad for backlog accumulation
12:44 Pchelolo: started parsoid linter script to generate load on cpjobqueue (python3 parsoid_reparse.py http://parsoid.discovery.wmnet:8000 --sitematrix --linter-only --skip-closed https://commons.wikimedia.org/w/api.php )
12:44 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: gerrit:393587 BETA wmgMonologChannels FileImporter => debug (duration: 01m 01s)
12:43 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: mw1276.eqiad.wmnet
12:35 moritzm: powercycling mw1276
12:21 addshore@tin: Synchronized wmf-config/CommonSettings-labs.php: gerrit:393582 LABS ONLY Actually call wfLoadExtension for FileExporter & Importer on beta BETA (T181383 ) (duration: 00m 44s)
12:14 marostegui: Stop replication on db1109 to test table rename for s5/s8 failover
12:09 ppchelko@tin: Finished deploy [cpjobqueue/deploy@47d27dc]: Enable keep-alive T181007 (duration: 01m 03s)
12:08 ppchelko@tin: Started deploy [cpjobqueue/deploy@47d27dc]: Enable keep-alive T181007
11:47 addshore@tin: Synchronized wmf-config/CommonSettings-labs.php: gerrit:393577 LABS ONLY Enable FileImporter & FileExporter on BETA PT2/2 (T181383 ) (duration: 00m 45s)
11:46 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: gerrit:393577 LABS ONLY Enable FileImporter & FileExporter on BETA PT1/2 (T181383 ) (duration: 00m 45s)
11:44 addshore@tin: Synchronized wmf-config/extension-list-labs: gerrit:393576 BETA ONLY Add FileExporter & FileImporter to extension-list-labs (duration: 00m 45s)
11:27 hashar: contint1001 enable Icinga "Disks space" notification again. It is no more complaing about Docker partitions | ping mutante | T178454
11:17 godog: bootstrap restbase1007-a - T179422
11:05 jdrewniak@tin: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 45s)
11:05 jdrewniak@tin: Synchronized portals/prod/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 46s)
10:53 moritzm: installing postgresql-common security updates
10:08 ppchelko@tin: Finished deploy [cpjobqueue/deploy@e35aa05]: Revert using keep-alive (duration: 00m 22s)
10:08 ppchelko@tin: Started deploy [cpjobqueue/deploy@e35aa05]: Revert using keep-alive
09:42 ppchelko@tin: Finished deploy [cpjobqueue/deploy@b570d4e]: Make http agent use keep-alive (duration: 00m 48s)
09:41 ppchelko@tin: Started deploy [cpjobqueue/deploy@b570d4e]: Make http agent use keep-alive
09:14 godog: reimage restbase1007 - T179422
08:55 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1097:3314 - T178359 (duration: 00m 43s)
08:40 moritzm: installing openjdk security updates on hadoop, druid and kafka clusters
08:27 marostegui: Deploy schema change on dbstore1002 and dbstore1001 - T174569
08:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1097:3314 weight - T178359 (duration: 00m 45s)
07:20 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1097:3314 weight - T178359 (duration: 00m 45s)
07:10 marostegui: Stop MySQL on db1021 as it will be decommissioned - T181378
06:53 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1021 from the config as it will be decommissioned - T181378 (duration: 00m 44s)
06:52 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1021 from the config as it will be decommissioned - T181378 (duration: 00m 45s)
06:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1097:3314 with low weight - T178359 (duration: 00m 46s)
02:22 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.8) (duration: 05m 44s)
19:31 marostegui: Set 32:3 disk to offline on db1051
16:09 kartik@tin: Finished deploy [cxserver/deploy@11aecc9]: Update cxserver to 0c242c0
16:05 kartik@tin: Started deploy [cxserver/deploy@11aecc9]: Update cxserver to 0c242c0
16:05 godog: unban statsd traffic from scb on graphite1001 - T181333
15:45 ppchelko@tin: Finished deploy [cpjobqueue/deploy@e35aa05]: Rollback. Disable GC metric reporting T181333 (duration: 00m 31s)
15:45 ppchelko@tin: Started deploy [cpjobqueue/deploy@e35aa05]: Rollback. Disable GC metric reporting T181333
15:37 volans: restarted statsd-proxy on graphite1001 (died during investigation) T181333
14:34 godog: rolling restart of cxserver to alleviate metrics leak - T181333
14:26 godog: restart cxserver on scb100[34] - T181333
14:10 godog: roll-restart cpjobqueue to alleviate metrics leak - T181333
13:40 godog: drop incoming statsd from scb to graphite1001 temporarily - T181333
08:32 ariel@tin: Finished deploy [dumps/dumps@ec21673]: fix abstracts recombine job (duration: 00m 02s)
08:32 ariel@tin: Started deploy [dumps/dumps@ec21673]: fix abstracts recombine job
13:52 moritzm: removing git packages from jessie-wikimedia/experimental (replaced by component/git)
13:24 moritzm: installing openjpeg2 updates (original security already got installed after initial release, but there was a binNMU for amd64)
13:17 marostegui: Stop replication on db1097 to reimport and recompress commonswiki.watchlist
12:54 jynus: reenabling puppet on db1071
12:50 jynus: resetting replication on es1011 for consistency with other replica sets
12:40 jynus: setting up s8 topology on eqiad
12:38 jynus: disable puppet on db1071 and stop local s5 heartbeat there
12:32 reedy@tin: Synchronized docroot/mediawiki/keys/: Fixup keys (duration: 00m 45s)
12:13 marostegui: Enable GTID on es2018 - T181293
11:57 marostegui: Disable puppet on es2018 - T181293
11:50 jynus@tin: Synchronized wmf-config/db-codfw.php: depool es2018 T181293 (duration: 00m 45s)
11:48 marostegui: Reboot es2018 after full-upgrade - T181293
11:25 marostegui: Restart mysql on es2018
11:25 jynus@tin: Synchronized wmf-config/db-eqiad.php: db2085:3318, db2086:3318 (duration: 00m 43s)
11:24 jynus@tin: Synchronized wmf-config/db-codfw.php: Pool db2038, db2085:3318, db2086:3318 (duration: 00m 45s)
10:55 marostegui: Restart MySQL on db2086 to move s5 to s8
10:37 jynus: cancelling db2085 restart, only doing mysql:s5
10:35 jynus: restarting db2085 (including both s5 and s3 instances)
10:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool all future s8 slaves for a topology change - T177208 (duration: 00m 45s)
10:22 moritzm: installing ca-cerfificates updates on trusty hosts
09:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1097:3315 and db1092 (duration: 00m 45s)
09:50 jynus: restarting db2045
09:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1101:3318 db1097:3315 abd db1092 (duration: 00m 45s)
08:49 marostegui: Stop MySQL on db1092
08:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1101:3318 in s5 to warm it up and depool db1092 - T178359 T177208 (duration: 00m 45s)
08:40 moritzm: installing java security updates on notebook* hosts
08:38 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Slowly pool db1097:3315 - T178359 (duration: 00m 45s)
08:37 marostegui@tin: Synchronized wmf-config/db-codfw.php: Slowly pool db1097:3315 - T178359 (duration: 00m 45s)
08:20 moritzm: installing java security updates on meitnerium
08:15 moritzm: installing java security updates on stat1004
08:14 hashar: restarting jenkins on contint1001 for a java update
08:07 elukey: re-enabling piwik on bohrium (only VM running on ganeti1006 atm) after mysql tables restore completed
06:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Slowly pool db1101:3318 in s5 to warm it up - T178359 (duration: 00m 45s)
06:46 marostegui@tin: Synchronized wmf-config/db-codfw.php: Slowly pool db1101:3318 in s5 to warm it up - T178359 (duration: 00m 49s)
00:12 reedy@tin: Synchronized docroot/mediawiki/keys/: Add Brian Wolff's key (duration: 00m 45s)
23:42 reedy@tin: Synchronized wmf-config/: Simplify variable assignment (duration: 00m 47s)
23:25 reedy@tin: Synchronized composer.lock: (no justification provided) (duration: 00m 44s)
23:24 reedy@tin: Synchronized composer.json: (no justification provided) (duration: 00m 45s)
23:23 reedy@tin: Synchronized phpcs.xml: (no justification provided) (duration: 00m 45s)
22:39 demon@tin: Synchronized wmf-config/: style fixes, no-op (duration: 00m 47s)
22:19 demon@tin: Synchronized wmf-config/CommonSettings.php: no-op, moving stuff around (duration: 00m 47s)
21:34 ariel@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2251.codfw.wmnet
21:15 apergos: rebooted mw2251 after unresponsive on mgmt console and no ping
20:49 demon@tin: Synchronized static/images/project-logos/nowikimedia.png: logo update (duration: 02m 51s)
17:29 akosiaris: set ganeti1006 as drained. ganeti1005 was already set. That will prevent scheduling VMs on those. T181121
17:01 akosiaris: force powercycle on ganeti1006 T181121
16:30 akosiaris: gnt-node failover ganeti1006
15:50 jynus: disable puppet on db2023, db2038 for deployment whle transfer is ongoing
15:23 moritzm: rebooting mwdebug* servers for update to 4.9.51
14:10 marostegui: Stop Mysql on db1101.s5 to clone db1097 - T178359
14:09 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1053 in s2 as vslow to replace db1021 - T134476 (duration: 00m 45s)
13:13 jynus: sutting down db2023 and db2038 for cloning
12:47 reedy@tin: Synchronized wmf-config/CommonSettings.php: GWToolset migratory config T87928 (duration: 00m 46s)
12:40 moritzm: migrating instances off ganeti2001 / kernel reboot to 4.9.51
12:10 moritzm: migrating instances off ganeti2002 / kernel reboot to 4.9.51
11:52 moritzm: migrating instances off ganeti2003 / kernel reboot to 4.9.51
11:37 kartik@tin: Finished deploy [cxserver/deploy@51b78ce]: T181209 Update cxserver to e8fe3f0
11:35 moritzm: migrating instances off ganeti2004 / kernel reboot to 4.9.51
11:35 moritzm: migrating instances off ganeti200 / kernel reboot to 4.9.51
11:34 kartik@tin: Started deploy [cxserver/deploy@51b78ce]: T181209 Update cxserver to e8fe3f0
11:23 moritzm: migrating instances off ganeti2005 / kernel reboot to 4.9.51
11:15 moritzm: migrating instances off ganeti2006 / kernel reboot to 4.9.51
11:06 moritzm: migrating instances off ganeti2007 / kernel reboot to 4.9.51
10:57 moritzm: migrating instances off ganeti2008 / kernel reboot to 4.9.51
10:07 moritzm: rebooting sarin for update to 4.9.51
09:16 jynus@tin: Synchronized wmf-config/db-codfw.php: mariadb: Switchover s5 codfw master (duration: 00m 45s)
08:56 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1072 (duration: 00m 44s)
08:44 jynus: stopping and restarting db2052
08:31 moritzm: installing bdb security updates on trusty
08:29 jynus: starting switchover of db2023 to db2052
08:21 marostegui: Stop MySQL on db1072 to MySQL upgrade and kernel upgrade
08:21 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1072 (duration: 00m 45s)
07:28 marostegui: Stop MySQL on db1021 and db1053 to clone db1053
07:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1021 - T134476 (duration: 00m 45s)
07:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Move db1053 to s2 to replace db1021 as vslow, dump slave - T134476 (duration: 00m 45s)
06:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1089 - T180045 (duration: 00m 45s)
06:55 marostegui: Drop index on ores_classification on s1 - T180045
06:53 marostegui: Drop index on ores_classification on s2 - T180045
06:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1089 - T180045 (duration: 00m 45s)
06:44 legoktm: legoktm@tin:~$ echo "https://www.mediawiki.org/keys/keys.html " | mwscript purgeList.php
06:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1051 and db1063 in vslow service for s5 to warm them up for the s8 split - T177208 (duration: 00m 46s)
05:58 kartik@tin: Finished deploy [cxserver/deploy@5b35ed5]: Update cxserver to e8fe3f0
05:54 kartik@tin: Started deploy [cxserver/deploy@5b35ed5]: Update cxserver to e8fe3f0
02:23 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.8) (duration: 05m 27s)
01:36 demon@tin: Synchronized wmf-config/InitialiseSettings.php: dropping education program from cswiki (duration: 00m 45s)
01:29 demon@tin: Synchronized wmf-config/CommonSettings.php: Set $wgCentralAuthGlobalBlockInterwikiPrefix (duration: 00m 45s)
01:26 demon@tin: Synchronized wmf-config/InitialiseSettings-labs.php: no-op (duration: 00m 44s)
01:24 demon@tin: Synchronized wmf-config/CommonSettings.php: abusefilter stuff (duration: 00m 45s)
01:21 demon@tin: Synchronized wmf-config/InitialiseSettings.php: abusefilter stuff (duration: 00m 45s)
01:19 demon@tin: Synchronized wmf-config/CommonSettings.php: abusefilter stuff (duration: 00m 45s)
01:05 demon@tin: Synchronized docroot/mediawiki/keys/: update formatting, docs, etc (duration: 00m 46s)
21:15 mutante: running puppet on logstash hosts to apply config change 392897 (add log4j filter)
21:11 demon@tin: Synchronized docroot/noc/conf/index.php: favicon fix (duration: 00m 46s)
20:35 demon@tin: Synchronized dblists/: now with more alphabeticalizedness (duration: 00m 45s)
20:25 demon@tin: Synchronized dblists/Makefile: no-op (duration: 00m 45s)
20:23 demon@tin: Synchronized tests/noc-conf/NOCDblistTest.php: no-op (duration: 00m 45s)
20:19 demon@tin: Synchronized wmf-config/LabsServices.php: no-op (duration: 00m 45s)
20:14 demon@tin: Synchronized wmf-config/InitialiseSettings.php: adding national library of Israel to copy domains (duration: 00m 45s)
20:13 otto@tin: Finished deploy [eventlogging/analytics@57234e7]: no-op: removing now unneeded code that might accidentally serialize userAgent to json string: T179625 (duration: 00m 04s)
20:13 otto@tin: Started deploy [eventlogging/analytics@57234e7]: no-op: removing now unneeded code that might accidentally serialize userAgent to json string: T179625
20:09 demon@tin: Synchronized robots.txt: block a nasty bot 💔 (duration: 00m 44s)
20:02 demon@tin: Synchronized multiversion/bin/expanddblist: fix param warning (duration: 00m 45s)
20:00 mepps: updated civicrm from a16e566 a1022cf
19:58 demon@tin: Synchronized wmf-config/InitialiseSettings.php: comments (duration: 00m 45s)
19:56 demon@tin: Synchronized multiversion/MWScript.php: assume aawiki for purgeUrls (duration: 00m 45s)
19:51 demon@tin: Synchronized docroot/: removing old foundation docroot (duration: 00m 46s)
19:49 urandom: starting cassandra cleanups, restbase-200{1,3,5}-a - T179422
19:41 demon@tin: Synchronized w/extract2.php: removing old portal support (duration: 00m 45s)
18:47 demon@tin: Synchronized dblists/closed.dblist: closed transitionteamwiki (duration: 00m 45s)
18:34 demon@tin: Synchronized wmf-config/InitialiseSettings-labs.php: no-op (duration: 00m 45s)
18:32 demon@tin: Synchronized scap/plugins/updatewikiversions.py: minor fix (duration: 00m 45s)
18:30 demon@tin: Pruned MediaWiki: 1.31.0-wmf.7 [keeping static files] (duration: 01m 46s)
18:12 herron: re-enabling puppet agents after puppetdb postgres security updates
18:09 moritzm: installing postgres security updates on nitrogen/puppetdb
18:05 moritzm: installing postgres security updates on nihal/puppetdb
18:02 herron: disabling puppet agents for puppetdb postgres security update
17:30 demon@tin: Synchronized php-1.31.0-wmf.8/extensions/AdvancedSearch/: fixing layout issues in timeless (duration: 00m 46s)
16:54 mepps: updated payments-wiki from 1ca91b1 6b3019b
16:45 chasemp: disable puppet accross labtest things
16:34 marostegui: Compress s4 on db1097 - T178359
16:28 herron: starting canary deploy/cutover of codfw scb hosts to codfw puppet 4 masters
16:16 elukey: restart druid broker,coordinator,historical daemons on druid100[123] to pick up new logging settings
15:41 jynus: starting manually pt-heartbeat for s8 on db1071
15:22 herron: beginning cut over of codfw db servers (^db2.*) to codfw puppet 4 masters
14:49 jynus@tin: Synchronized wmf-config/db-codfw.php: mariadb: Setup s8 replica set on codfw (duration: 00m 45s)
14:27 moritzm: installing libxml-libxml-perl security updates
14:21 jynus: starting database topology changes for s8 on codfw T177208
14:11 urandom: bootstrapping cassandra, restbase2004-c.codfw.wmnet - T179422
13:43 apergos: one more round of labstore1006 <-- ms1001 rsync catchup
13:37 moritzm: installing imagemagick security updates
12:39 marostegui: Stop MySQL on db1053 to clone db1097.s4 - T178359
12:38 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1053 - T178359 (duration: 00m 45s)
12:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Start adapting the config to move db1097 to s4 and s5 as multi-instance rc slave T178359 (duration: 00m 45s)
12:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1101.s7 - T178359 (duration: 00m 45s)
11:51 jynus: starting dropping incorrectly created database on s7 amwikimedia (not to be confused with production wiki s3 amwikimedia)
11:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1101.s7 - T178359 (duration: 00m 45s)
11:19 akosiaris: gnt-node evacuate -s -f ganeti1005. T181121
11:02 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1101.s7 - T178359 (duration: 00m 45s)
10:54 akosiaris: gnt-node migrate -f ganeti1005. T181121
10:51 marostegui: Drop index from ores_classification on s5 - T180045
10:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1051 and db1063 in vslow service for s5 to warm them up for the s8 split - T177208 (duration: 00m 45s)
10:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1101 to s5 and s7 as recentchanges multi-instance slave - T178359 (duration: 00m 45s)
10:04 moritzm: running "scap pull" on mw1191, it's depooled and marked as "inactive", but health checks are triggering db errors
09:35 bblack: cr[12]-ulsfo - switch static fallback LVS routes from lvs400[12] to lvs400[56]
09:27 bblack: lvs@ulsfo - done switching primaries (host MED config) - lvs400[56] now primary for text/upload traffic
09:11 akosiaris@tin: Finished deploy [parsoid/deploy@b150764]: T180211 (duration: 05m 05s)
09:08 bblack: puppet disabled on lvs400[1256] for switching primaries
09:06 akosiaris@tin: Started deploy [parsoid/deploy@b150764]: T180211
09:04 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp2017.codfw.wmnet
09:00 bblack: lvs4005 - reboot to clear experimental stuff
08:16 bblack: backend restart on cp4024 (upload@ulsfo) - mailbox lag
07:56 marostegui: Drop index from ores_classification on s3 - T180045
07:50 marostegui: Drop index from ores_classification on s6 - T180045
07:48 marostegui: Drop index from ores_classification on s7 - T180045
07:29 _joe_: stopping the additional workers for htmlCacheUpdate (commons and ruwiki), adding one additional runner for refreshLinks on ruwiki
06:43 marostegui: Stop MySQL on db1063 and db1051 (which is going to be recloned) - T177208
06:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Move db1051 from s1 to s5 - T177208 (duration: 00m 53s)
06:28 ejegg: updated payments-wiki from f871160 1ca91b1
03:36 mutante: powercycled mw2251 which had gone down without further comment
02:35 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.8) (duration: 11m 34s)
01:09 hoo: Cleaned out remaining T180934 related log blow up on snapshot1007 (dumpwikidatajson-wikidata-20171120-all-0.log)
00:43 mutante: gerrit restarting service to apply config change
00:40 mutante: gerrit - re-enabling puppet to apply logstash change on cobalt, gerrit restart incoming (T141324 )
00:39 maxsem@tin: Synchronized php-1.31.0-wmf.8/extensions/CentralNotice/: https://gerrit.wikimedia.org/r/#/c/392754/ (duration: 00m 47s)
00:11 maxsem@tin: Synchronized php-1.31.0-wmf.8/extensions/InputBox/: https://gerrit.wikimedia.org/r/#/c/392745/ (duration: 00m 45s)
00:10 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/392576/ (duration: 00m 46s)
00:07 bblack@neodymium: conftool action : set/pooled=yes; selector: name=nescio.wikimedia.org
00:04 bblack@neodymium: conftool action : set/pooled=no; selector: name=nescio.wikimedia.org
00:02 bblack@neodymium: conftool action : set/pooled=yes; selector: name=maerlant.wikimedia.org
00:01 bblack@neodymium: conftool action : set/pooled=no; selector: name=maerlant.wikimedia.org
00:01 bblack@neodymium: conftool action : set/pooled=yes; selector: name=maerlant.wikimedia.org
00:00 bblack@neodymium: conftool action : set/pooled=yes; selector: name=chromium.wikimedia.org
23:55 bblack@neodymium: conftool action : set/pooled=no; selector: name=chromium.wikimedia.org
23:30 mholloway-shell@tin: Finished deploy [mobileapps/deploy@fc01242]: Update mobileapps to 52d6a83
23:26 mholloway-shell@tin: Started deploy [mobileapps/deploy@fc01242]: Update mobileapps to 52d6a83
22:45 bblack@neodymium: conftool action : set/pooled=yes; selector: name=achernar.wikimedia.org
22:38 bblack@neodymium: conftool action : set/pooled=no; selector: name=achernar.wikimedia.org
22:36 bblack@neodymium: conftool action : set/pooled=yes; selector: name=hydrogen.wikimedia.org
22:29 urandom: Bootstrapping Cassandra, restbase2004-b.codfw.wmnet (T179422 )
22:12 mutante: gerrit - temp disable puppet on cobalt (prod gerrit), test switching gerrit logging to logstash on gerrit2001 - gerrit:392079 gerrit:392083 T141324
22:05 bblack@neodymium: conftool action : set/pooled=no; selector: name=hydrogen.wikimedia.org
22:03 bblack@neodymium: conftool action : set/pooled=yes; selector: name=acamar.wikimedia.org
22:02 bblack: repooling acamar for recdns
21:26 ariel@tin: Finished deploy [dumps/dumps@16f92d6]: take 2: gzip namespace and abstract dumps; remove last configfile existence checks (duration: 00m 02s)
21:26 ariel@tin: Started deploy [dumps/dumps@16f92d6]: take 2: gzip namespace and abstract dumps; remove last configfile existence checks
20:45 bblack: recdns: puppet disabled on all, acamar depooled, careful deploys going on for anycast+recdns stuff
20:42 ayounsi@neodymium: conftool action : set/pooled=no; selector: name=acamar.wikimedia.org
20:26 ariel@tin: Finished deploy [dumps/dumps@16f92d6]: gzip namespace and abstract dumps; remove last configfile existence checks (duration: 00m 16s)
20:26 ariel@tin: Started deploy [dumps/dumps@16f92d6]: gzip namespace and abstract dumps; remove last configfile existence checks
20:22 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to 1.31.0-wmf.8
20:00 thcipriani: finishing wmf.8 rollout, starting group2 to wmf.8
19:21 addshore: SWAT done!
19:20 addshore@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Add images.collection.cooperhewitt.org & *.dimu.org to wgCopyUploadsDomains. T180791 T180241 (duration: 00m 48s)
19:11 addshore@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Enable AdvancedSearch on group0 PT2/2 (duration: 00m 49s)
19:10 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable AdvancedSearch on group0 PT1/2 (duration: 00m 50s)
18:40 mholloway-shell@tin: Finished deploy [mobileapps/deploy@dd41387]: Update mobileapps to 9d1602d
18:35 mholloway-shell@tin: Started deploy [mobileapps/deploy@dd41387]: Update mobileapps to 9d1602d
18:28 smalyshev@tin: Finished deploy [wdqs/wdqs@c69c739]: Restore categories vocabulary to V003 (duration: 01m 55s)
18:26 smalyshev@tin: Started deploy [wdqs/wdqs@c69c739]: Restore categories vocabulary to V003
18:25 smalyshev@tin: Finished deploy [wdqs/wdqs@7d951d2]: Restore categories vocabulary to V003 (duration: 00m 11s)
18:25 smalyshev@tin: Started deploy [wdqs/wdqs@7d951d2]: Restore categories vocabulary to V003
16:45 marostegui: Compress s3 on db2085 - T178359
16:35 papaul: powering down wtp2017 for disk replacement
16:19 papaul: updating firmware on db2068
15:31 _joe_: rolling restart of pybal on low-traffic in codfw, eqiad for the new depool thresholds for MW
15:00 ppchelko@tin: Finished deploy [cpjobqueue/deploy@3e948de]: Revert: Temporarily disable deduplication (duration: 00m 26s)
15:00 ppchelko@tin: Started deploy [cpjobqueue/deploy@3e948de]: Revert: Temporarily disable deduplication
14:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore original weight for db1096 (duration: 00m 48s)
14:39 ppchelko@tin: Finished deploy [cpjobqueue/deploy@aac3201]: Temporarily disable deduplication (duration: 00m 29s)
14:39 ppchelko@tin: Started deploy [cpjobqueue/deploy@aac3201]: Temporarily disable deduplication
14:29 zeljkof: EU SWAT finished
14:25 bblack: cr[12]-ulsfo: allow lvs400[567] as PyBal neighbors for BGP
14:18 zfilipin@tin: Synchronized php-1.31.0-wmf.8/extensions/TimedMediaHandler/MwEmbedModules/EmbedPlayer/resources/mw.EmbedPlayerOgvJs.js: SWAT: Disable wasm, use asm.js codec modules for Safari/Edge (T181022) (duration: 00m 49s)
14:12 jdrewniak@tin: Synchronized portals: SWAT: Bumping portals to master (T128546) (duration: 00m 49s)
14:11 jdrewniak@tin: Synchronized portals/prod/wikipedia.org/assets: SWAT: Bumping portals to master (T128546) (duration: 00m 50s)
14:11 ppchelko@tin: Started restart [cpjobqueue/deploy@5341d94]: Restart to try increased maxSockets
14:11 Pchelolo: restart cpjobqueue to try increasing maxSockets
14:08 bblack: not-abnormally-quick reboot on lvs4005
14:08 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: electron stuck - T174916
13:45 ppchelko@tin: Finished deploy [cpjobqueue/deploy@5341d94]: Enable GC metrics reporting (duration: 00m 36s)
13:44 ppchelko@tin: Started deploy [cpjobqueue/deploy@5341d94]: Enable GC metrics reporting
13:39 _joe_: starting 2 manual runners for htmlcacheupdate on commons, 1 for htmlcacheupdate and 1 for refreshlinks on ruwiki, on terbium
13:39 bblack: quick reboot on lvs4005
12:02 kartik@tin: Finished deploy [cxserver/deploy@b87a27a]: Update cxserver to 4301987
11:58 kartik@tin: Started deploy [cxserver/deploy@b87a27a]: Update cxserver to 4301987
11:04 akosiaris: reboot ununpentium for serial tty change
11:03 akosiaris: reboot oresrdb2001 for serial tty change
11:02 akosiaris: reboot netmon1003 for serial tty change
11:01 akosiaris: reboot install2002 for serial tty change
10:57 akosiaris: reboot install1002 for serial tty change
10:35 godog: bootstrap cassandra restbase2004-a - T179422
10:11 ppchelko@tin: Finished deploy [cpjobqueue/deploy@e35aa05]: Set consumer_batch_size to 10 T181007 (duration: 00m 31s)
10:10 ppchelko@tin: Started deploy [cpjobqueue/deploy@e35aa05]: Set consumer_batch_size to 10 T181007
09:39 elukey: upload prometheus-druid-exporter 0.4 to jessie/stretch-wikimedia
09:28 marostegui: Shutdown db2068 for maintenance - T180927
09:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1096 (duration: 00m 49s)
09:02 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1096 (duration: 00m 49s)
08:39 marostegui: Drop index on db1089 enwiki.ores_classification - T180045
08:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1096 with low weight - T178359 (duration: 00m 48s)
08:27 marostegui: Reboot db1096 for kernel and MariaDB upgrade
08:19 marostegui: Compress s5 on db1101 - T178359
08:11 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1109 and db1110 - T180700 (duration: 00m 48s)
07:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1109 and db1110 - T180700 (duration: 00m 49s)
07:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1109 and db1110 in s5 with small weight - T180700 (duration: 00m 48s)
06:50 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1087 - T179106 (duration: 00m 48s)
06:47 marostegui: Remove index wb_terms_language from db1087 - https://phabricator.wikimedia.org/T179106
06:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1087 - T179106 (duration: 00m 48s)
06:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1096 - T178359 (duration: 00m 48s)
06:27 marostegui: Stop MySQL on db1096 to clone db1101.s5 - T178359
06:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1082 original weight - T177208 (duration: 00m 49s)
02:58 mutante: phabricator back up
02:55 mutante: phab1001 (phabricator prod) reboot for kernel upgrade
02:48 krinkle@tin: Synchronized docroot/noc: clean up (duration: 00m 49s)
02:23 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.7) (duration: 05m 42s)
02:03 smalyshev@tin: Started deploy [wdqs/wdqs@7d951d2]: Restore categories vocabulary to V003
00:19 catrope@tin: Synchronized php-1.31.0-wmf.8/resources/src/mediawiki.rcfilters/ui/mw.rcfilters.ui.ItemMenuOptionWidget.js: T180863 (duration: 00m 49s)
23:46 mutante: phab2001 - reboot for kernel upgrade
23:35 legoktm@tin: Synchronized wmf-config/InitialiseSettings.php: emergency disable ORES on frwp/ruwp T181006 (duration: 00m 49s)
23:35 awight: purge cache keys for ORES thresholds on frwiki and ruwiki
23:25 awight@tin: Started deploy [ores/deploy@82a13ae]: Rollback ORES (take 3); 181006
23:19 awight: aborted ORES rollback
23:18 awight@tin: Started deploy [ores/deploy@95cd523]: Rollback ORES (take 2); 181006
23:14 smalyshev@tin: Finished deploy [wdqs/wdqs@7d951d2]: Rollback categories vocabulary version due to a bug (duration: 08m 11s)
23:11 awight: purged memcache key 'ruwiki:ORES:threshold_statistics:goodfaith:1’, T181006
23:06 smalyshev@tin: Started deploy [wdqs/wdqs@7d951d2]: Rollback categories vocabulary version due to a bug
22:55 awight@tin: Finished deploy [ores/deploy@5084251]: Rollback ORES; T179711 (duration: 01m 05s)
22:55 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: no wmf.8 for group2. i hate my life
22:55 awight: rolling back ORES to fix T181006
22:54 eileen: update civicrm from 272580a a16e566
22:54 awight@tin: Started deploy [ores/deploy@5084251]: Rollback ORES; T179711
22:50 Krinkle: MW HTTP 500 spike tracked as https://phabricator.wikimedia.org/T181006
22:49 Krinkle: Sharp rise in HTTP 500 errors as of 22:05 (45 minutes ago)
22:29 aaron@tin: Synchronized php-1.31.0-wmf.7/includes/libs/objectcache/WANObjectCache.php: fix cache key namespace (duration: 00m 49s)
22:27 awight@tin: Finished deploy [ores/deploy@5084251]: Updating ORES to revscoring 2.0.10, T179711 (duration: 49m 54s)
22:26 aaron@tin: Synchronized php-1.31.0-wmf.7/extensions/Collection: cache key name fix (duration: 00m 49s)
22:11 aaron@tin: Synchronized php-1.31.0-wmf.8/includes/libs/objectcache/WANObjectCache.php: 7e74b49
22:04 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.8
21:52 bblack: ns2 back on eeden
21:52 aaron@tin: Synchronized php-1.31.0-wmf.8/extensions/Collection: 3baebf4a
21:44 bblack: rebooting eeden
21:37 awight@tin: Started deploy [ores/deploy@5084251]: Updating ORES to revscoring 2.0.10, T179711
21:33 bblack: routing ns2.wikimedia.org to radon for eeden reboot
21:19 bblack: routing ns0.wikimedia.org back to radon post-reboot
21:13 bblack: rebooting radon
21:07 bblack: re-routing ns0.wikimedia.org traffic to baham for radon reboot
20:17 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Remove overlapping userrights (T101983 ) (duration: 19m 41s)
20:16 bblack: ns1 dns traffic back to normal on baham
20:16 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Revert "Enable Translate extension in amwikimedia (T180879 )" (duration: 00m 48s)
20:11 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Enable Translate extension in amwikimedia (T180879 ) (duration: 00m 49s)
20:11 hashar: CI docker jobs were all broken due to a mistake. Should be back now. T177684
20:07 ladsgroup@tin: Synchronized php-1.31.0-wmf.8/resources/src/mediawiki.rcfilters/ui/mw.rcfilters.ui.ItemMenuOptionWidget.js: RCFilters: Only apply excluded label to namespace items (T180863 ) (duration: 00m 49s)
20:06 bblack: rebooting baham
19:58 bblack: re-routing ns1.wikimedia.org traffic to radon for baham reboot
19:52 chasemp: updating phab mail handler
19:51 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Remove overlapping userrights (T101983 ) (duration: 00m 49s)
19:47 ottomata: restarted coal with fixes for eventcapsule changes in T179625
19:42 ladsgroup@tin: Synchronized wmf-config/throttle.php: Adjust throttle.php for dewiki workshop (T180046 ) (duration: 00m 49s)
19:40 ladsgroup@tin: Synchronized wmf-config/throttle.php: Adjust throttle.php for dewiki workshop (T180046 ) (duration: 00m 48s)
19:36 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Allow admins to remove users from MP3 uploaders user group (T180002 ) (duration: 00m 49s)
19:31 ladsgroup@tin: Synchronized portals: SWAT: Bumping portals to master (T128546) (duration: 00m 51s)
19:30 ladsgroup@tin: Synchronized portals/prod/wikipedia.org/assets: SWAT: Bumping portals to master (T128546) (duration: 00m 49s)
19:22 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Switch submit button from 'save' to 'publish' on dewiki (duration: 00m 50s)
18:51 chasemp: disable puppet across cloud things to rollout https://gerrit.wikimedia.org/r/#/c/392168/ slowly
18:25 smalyshev@tin: Finished deploy [wdqs/wdqs@2e39b69]: Blazegraph and GUI update (duration: 01m 42s)
18:23 smalyshev@tin: Started deploy [wdqs/wdqs@2e39b69]: Blazegraph and GUI update
17:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1082 with low weight - T177208 (duration: 00m 49s)
17:19 bd808: Sending Toolforge survey emails from silver for T177126
17:00 moritzm: uploaded git 2.11-3+deb9u2+bpo8+wmf1 for component/git to apt.wikimedia.org/jessie-wikimedia
16:43 marostegui: Reboot db1082 for kernel upgrade and MariaDB upgrade to 10.0.33
16:39 ppchelko@tin: Finished deploy [cpjobqueue/deploy@174420f]: Revert: Temporary set consumer_batch_size to 50, forgot -f, checkout prev rev (duration: 03m 33s)
16:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1109 and db1110 to the config depooled - T180700 (duration: 00m 48s)
16:38 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1109 and db1110 to the config depooled - T180700 (duration: 00m 48s)
16:36 ppchelko@tin: Started deploy [cpjobqueue/deploy@174420f]: Revert: Temporary set consumer_batch_size to 50, forgot -f, checkout prev rev
16:32 ppchelko@tin: Finished deploy [cpjobqueue/deploy@f0610d3]: Revert: Temporary set consumer_batch_size to 50, forgot -f (duration: 00m 28s)
16:32 ppchelko@tin: Started deploy [cpjobqueue/deploy@f0610d3]: Revert: Temporary set consumer_batch_size to 50, forgot -f
16:24 marostegui: Add db1109 and db1110 to tendril - T180700
16:06 ppchelko@tin: Finished deploy [cpjobqueue/deploy@f0610d3]: Revert: Temporary set consumer_batch_size to 50 (duration: 00m 17s)
16:06 ppchelko@tin: Started deploy [cpjobqueue/deploy@f0610d3]: Revert: Temporary set consumer_batch_size to 50
15:59 urandom: Use lz4 compression instead of deflate (T180804 )
15:45 jynus: shutting down db2068 for maintenance after depool T180927
15:42 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2068 (duration: 00m 49s)
15:34 chasemp: disable puppet for a merge across cloud things
15:31 ppchelko@tin: Finished deploy [cpjobqueue/deploy@f0610d3]: Temporary set consumer_batch_size to 50 (duration: 00m 30s)
15:30 ppchelko@tin: Started deploy [cpjobqueue/deploy@f0610d3]: Temporary set consumer_batch_size to 50
15:03 herron: pointing codfw mw servers at codfw puppet 4 masters via puppetmaster2001
14:39 zeljkof: EU SWAT finished
14:34 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Single edit tab in Catalan Wikipedia (T180660) (duration: 00m 49s)
14:25 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable wgNamespacesWithSubpages for hiwikiversity (T180913) (duration: 00m 48s)
14:22 jynus: enable semi-sync replication on s5
14:14 dcausse@tin: Synchronized wmf-config/CirrusSearch-common.php: T180805 : Revert [cirrus] disable token count router (duration: 00m 49s)
14:05 elukey: upload prometheus-druid-exporter 0.3 to jessie-wikimedia
13:53 Pchelolo: disable puppet on scb100x and stop cpjobqueue to accumulate some backlog
13:39 ppchelko@tin: Finished deploy [cpjobqueue/deploy@174420f]: Optimise committed offset calculation (duration: 00m 28s)
13:39 ppchelko@tin: Started deploy [cpjobqueue/deploy@174420f]: Optimise committed offset calculation
13:30 elukey: upload prometheus-druid-exporter 0.3 to stretch-wikimedia
12:40 marostegui: Optimize db1063 wikidatawiki.wb_terms
12:17 moritzm: uploaded hhvm 3.18.5+dfsg-1+wmf1+icu57 to apt.wikimedia.org (jessie-wikimedia/component/icu57) (HHVM build linked against a co-installable backport of icu57)
11:46 moritzm: rebooting tungsten for update to 4.9.51
11:41 moritzm: rebooting etherpad1001 (etherpad.wikimedia.org) for update to 4.9.51
11:36 moritzm: rebooting pybal-test for update to 4.9.51
11:27 moritzm: rebooting restbase-test cluster for update to 4.9.51
11:23 ppchelko@tin: Finished deploy [cpjobqueue/deploy@bdcef23]: Various performance improvements in committing (duration: 00m 30s)
11:22 ppchelko@tin: Started deploy [cpjobqueue/deploy@bdcef23]: Various performance improvements in committing
11:13 moritzm: rebooting ruthenium for update to 4.9.51
10:57 godog: reimage restbase2004 - T179422
10:51 moritzm: rebooting cerium/praseodymium/xenon for update to 4.9.51
10:42 moritzm: rebooting mwlog1001 for update to 4.9.51
10:39 moritzm: rebooting mwlog2001 for update to 4.9.51
10:23 hoo: Manually re-started the Wikidata entity JSON dump on snapshot1007 (T180934 )
10:19 dcausse: elastic/cirrus: reindexing english group0 and group1 wikis: T179945
10:08 hashar: contint1001: sudo systemctl start jenkins
10:05 moritzm: rebooting contint1001 for update to 4.9.51
09:22 moritzm: rebooting hafnium for update to 4.9.51
08:46 marostegui: Run mydumper for db1047.staging - T156844
07:58 moritzm: installing procmail security updates
07:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1097 - T178359 (duration: 00m 49s)
06:41 marostegui: Stop MySQL on db1082 to clone db1109 and db1110 - T180700
06:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1082 - T177208 (duration: 00m 48s)
06:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore original weights for db1100 and db1071 - T180917 (duration: 00m 49s)
06:15 marostegui: Reboot db2068 - T180927
02:28 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.7) (duration: 06m 26s)
23:20 Jamesofur: removed 2FA for Ask21 T180889
20:28 krinkle@tin: Synchronized docroot/noc/index.html: noc: Link to Grafana (duration: 00m 49s)
19:50 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1100 (duration: 00m 49s)
07:09 mobrovac@tin: Started restart [zotero/translators@a0c41c3]: Zotero eating up memory
07:01 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: electron stuck - T174916
20:06 gehel: upgrade of elasticsearch eqiad complete - T178411
18:03 reedy@tin: Synchronized multiversion/vendor/: bump (duration: 01m 14s)
15:53 marostegui: Compress s5 on db2085 - T178359
14:48 marostegui: Compress s3 on db2092 - T178359
00:30 urandom: Bootstrapping restbase2006-b (T179422 )
20:35 mutante: netmon1002 - rsync smokeping data back from local backup to show measurements made from eqiad as requested on T180812
20:05 mutante: running puppet on all cache misc to switch smokeping web to eqiad
20:02 mutante: T180812 copying smokeping data from 2001 to 1002 - netmon1002: /usr/bin/rsync -avp rsync://netmon2001.wikimedia.org/var-lib-smokeping /var/lib/smokeping/ | switching backend from codfw to eqiad
19:37 mutante: T180812 - @netmon1002:# rsync -avp /var/lib/smokeping/ /root/backup/netmon1002/201711717/var/lib/smokeping/@netmon2001:/# rsync -avp /var/lib/smokeping/ /root/backup/netmon2001/201711717/var/lib/smokeping/
19:36 mutante: @netmon1002:/var/lib/smokeping# rsync -avp /var/lib/smokeping/ /root/backup/netmon1002/201711717/var/lib/smokeping/
18:37 anomie@tin: Synchronized wmf-config/InitialiseSettings.php: Setting wgCommentTableSchemaMigrationStage = MIGRATION_WRITE_BOTH on Beta Cluster, no prod change (duration: 00m 48s)
18:36 anomie@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Setting wgCommentTableSchemaMigrationStage = MIGRATION_WRITE_BOTH on Beta Cluster, no prod change (duration: 00m 49s)
17:43 demon@tin: Synchronized php-1.31.0-wmf.8/extensions/Wikidata/extensions/Wikibase/client/WikibaseClient.php: fix client dependencies (duration: 00m 49s)
17:41 demon@tin: Synchronized php-1.31.0-wmf.8/extensions/Wikibase/client/WikibaseClient.php: fix client dependencies (duration: 00m 50s)
17:21 bblack: enabling new normalization code for all upload@esams (done)
17:19 bblack: enabling new normalization code for all upload@eqiad
17:14 marostegui: Revert schema change on dbstore1001 - T180714
17:12 marostegui: Move dbstore1001 under db1070 - T180714
17:12 bblack: enabling new normalization code for all upload@codfw
16:52 bblack: enabling new normalization code for all upload@ulsfo
16:24 bblack: disabling puppet on all cp* (testing encoding patch)
16:19 moritzm: uploaded boost 1.55.0+dfsg-3+wmf2+icu57 to apt.wikimedia.org for jessie-wikimedia/component/icu57 (needed for HHVM build linked against ICU 57)
16:04 dcausse@tin: Synchronized wmf-config/CirrusSearch-common.php: T180795 [cirrus] disable token count router (duration: 00m 49s)
15:43 chasemp: labservices1001:~# mv /var/zones/tools.eqiad.wmflabs /home/rush T180797
15:26 urandom: Starting restbase2006-c w/ -Dcassandra.replace_address=10.192.48.51 (T179422 )
14:06 jynus: deploy master events to db1070
14:04 chasemp: labstore1003 service nfs-kernel-server restart && service rsync start
13:55 moritzm: uploaded boost 1.55.0+dfsg-3+wmf1+icu57 to apt.wikimedia.org for jessie-wikimedia/component/icu57 (needed for HHVM build linked against ICU 57)
12:15 moritzm: installing openssl updates on poolcounters
12:07 moritzm: installing openssl updates on graphite* hosts
11:46 moritzm: installing openssl updates on etcd* hosts
11:43 moritzm: installing openssl updates on dbproxy hosts
11:33 moritzm: installing openssl updates on puppetmasters
11:01 akosiaris: create webperf1001, webperf2001 in ganeti T179036
10:50 akosiaris@tin: Synchronized wmf-config/db-eqiad.php: (no justification provided) (duration: 00m 49s)
10:49 akosiaris: sync wmf-config/db-eqiad.php for T180724
10:48 akosiaris@tin: scap aborted: (no justification provided) (duration: 00m 02s)
10:48 akosiaris@tin: Started scap: (no justification provided)
10:47 akosiaris: pool mw2251 T180724
10:47 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw2251.codfw.wmnet
10:01 moritzm: rebooting darmstadtium (docker registry) for update to 4.9.51
09:59 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore original traffic for the s5 eqiad hosts that were out for hours due to schema change revert after s5 master crash (duration: 00m 48s)
09:54 moritzm: rebooting labnodepool* for update to 4.9.51
09:38 ppchelko@tin: Finished deploy [cpjobqueue/deploy@2141162]: Bump concurrency even more (duration: 00m 29s)
09:38 ppchelko@tin: Started deploy [cpjobqueue/deploy@2141162]: Bump concurrency even more
09:20 godog: start restbase2006-c instead, restbase2006-b failed and -c shows as "down" - T179422
09:13 jynus: changing master of db1071 (63 -> 70)
09:10 gehel: cleanup leftover titlesuggest indices on elasticsearch eqiad (jawiki, frwiki, ptwiki)
09:05 ppchelko@tin: Finished deploy [cpjobqueue/deploy@cbd25d3]: Bump overall concurrency to get rid of RecordLintJob backlog (duration: 00m 35s)
09:04 ppchelko@tin: Started deploy [cpjobqueue/deploy@cbd25d3]: Bump overall concurrency to get rid of RecordLintJob backlog
08:54 godog: bootstrap restbase2006-b - T179422
08:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for the s5 eqiad hosts that were out for hours due to schema change revert after s5 master crash (duration: 00m 50s)
08:04 elukey: reboot stat100[456] for kernel updates
07:51 marostegui: Revert schema change on dbstore1002 - T180714
07:50 marostegui: Move dbstore1002 under db1070 - T180714
07:33 marostegui: Enable GTID on all the eqiad up-to-date hosts, only pending db1071 - T180714
07:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool s5 eqiad hosts after reverting schema change - T180714 (duration: 00m 50s)
06:34 marostegui: Revert schema changes on s5 codfw master with replication enabled, lag will be generated on codfw s5 - T180714
04:43 krinkle@tin: Synchronized wmf-config/CommonSettings-labs.php: no-op for labs (duration: 00m 51s)
01:32 ejegg: re-enabled donation queue consumer
01:28 ejegg: updated CiviCRM from 8454e06 272580a
01:07 ejegg: updated CiviCRM from 0b8ceea 8454e06
food: disabled donations queue consumer for thank you subject update
00:41 krinkle@tin: Synchronized wmf-config/PhpAutoPrepend.php: I60cce0
00:40 krinkle@tin: Synchronized wmf-config/StartProfiler.php: I60cce0
00:37 krinkle@tin: Synchronized wmf-config/profiler.php: I60cce0
00:23 urandom: Decommissioning Cassandra, restbase1014-c.eqiad.wmnet (T179422 )
00:22 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/390131/3 (duration: 00m 49s)
23:50 mutante: wezen - systemctl restart rsyslog
21:52 mepps: updated civicrm from 22f6532 0b8ceea
21:05 mepps: updated payments-wiki from 210cb37 f871160
20:51 apergos: one more catchup rsync from ms1001 to labstore1006 kicking off
20:15 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool only 'good' servers - third try (duration: 00m 49s)
20:11 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool only 'good' servers - second try (duration: 00m 49s)
20:10 demon@tin: Unlocked for deployment [operations/mediawiki-config]: No deploys, recovering from downtime (more narrow locking) (duration: 06m 49s)
20:03 demon@tin: Locking from deployment [operations/mediawiki-config]: No deploys, recovering from downtime (more narrow locking) (planned duration: 360m 00s)
20:02 twentyafterfour: deploy a9613b4 T180706
19:56 demon@tin: Unlocked for deployment [ALL REPOSITORIES]: No deploys, recovering from downtime (duration: 50m 28s)
19:54 ayounsi@tin: Finished deploy [netbox/deploy@19f4f65]: (no justification provided) (duration: 00m 04s)
19:54 ayounsi@tin: Started deploy [netbox/deploy@19f4f65]: (no justification provided)
19:49 akosiaris: schedule extra downtime for s5 slaves
19:46 urandom: Bootstapping Cassandra restbase2006-a (T179422 )
19:13 jynus: reset slave all on db1070 @ db1063-bin.001382:234464548
19:05 demon@tin: Locking from deployment [ALL REPOSITORIES]: No deploys, recovering from downtime (planned duration: 360m 00s)
19:05 demon@tin: Unlocked for deployment [ALL REPOSITORIES]: Dealing with outage, no deploys for now (duration: 01m 22s)
19:03 demon@tin: Locking from deployment [ALL REPOSITORIES]: Dealing with outage, no deploys for now (planned duration: 60m 00s)
18:48 marostegui: dewikipedia and wikidata currently back to writable
18:44 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool only 'good' servers (duration: 00m 48s)
18:41 mutante: dewikipedia and wikidata currently back to readonly-mode while wikidata is being worked on
18:41 marostegui: Set s5 master read_only
18:25 akosiaris: set mw2251 to inactive. T180724
18:24 akosiaris@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=mw2251.codfw.wmnet
17:57 gehel: silencing wdqs
17:55 jynus@tin: Synchronized wmf-config/db-eqiad.php: Failover db1063 to db1070 (duration: 00m 46s)
17:44 XioNoX: merging netbox CR
17:41 bblack: disable port ge-5/0/39 on asw-c-eqiad (db1063)
17:40 urandom: Decommissioning Cassandra, restbase1014-b.eqiad.wmnet (T179422 )
17:36 jynus: stopping slave on db1070
17:28 urandom: Converting 'enwiki parsoid' to size-tiered compaction (T179422 )
17:19 demon@tin: Finished scap: consistency (duration: 25m 12s)
17:18 bblack: Temporary GeoDNS routing changes (eqsin traffic simulation using ulsfo) - https://gerrit.wikimedia.org/r/#/c/391357/ - expecting ~24h, West Asia latencies will probably increase, spike in cache misses, etc...
17:18 urandom: Converting 'wikipedia parsoid' to size-tiered compaction (T179422 )
17:16 godog: reimage restbase2006 - T179422
17:15 urandom: Converting 'commons mobile' to size-tiered compaction (T179422 )
17:05 urandom: Converting 'others mobile' to size-tiered compaction (T179422 )
16:54 demon@tin: Started scap: consistency
16:54 godog: reimage restbase2006 - T179422
16:48 jynus: stop and restart db1071 for upgrade and reconfiguration
16:48 herron: beginning gradual cutover of codfw mw systems to puppet 4 master puppetmaster2001
16:48 godog: upgrade hpsa firmware to 6.06 on restbase2006 - T141756
16:43 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1034 and db1079 original weight (duration: 00m 48s)
16:29 jynus@tin: Synchronized wmf-config/db-eqiad.php: mariadb: Depool db1071, pool db1104 as api (duration: 00m 49s)
16:21 moritzm: restarting apache on labs puppet masters to pick up openssl updates
16:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1034 weight (duration: 00m 48s)
15:47 moritzm: rebooting ores1* for update to 4.9.51
15:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1034 weight (duration: 00m 49s)
15:33 moritzm: rebooting seaborgium (slapd) for update to 4.9.51
15:22 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2038 - T178359 (duration: 00m 49s)
15:16 milimetric@tin: Finished deploy [analytics/refinery@4ef15d3]: Mainly deploying the interlanguage navigation dataset (duration: 14m 29s)
15:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1034 weight (duration: 00m 48s)
15:10 godog: upgrade prometheus-redis-exporter to 0.13-1 - T148637
15:06 moritzm: installing postgres security updates on labsdb1006/1007
15:02 milimetric@tin: Started deploy [analytics/refinery@4ef15d3]: Mainly deploying the interlanguage navigation dataset
15:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1051 - T177208 (duration: 00m 48s)
15:00 herron: beginning cut over of codfw canary appservers to puppet 4 master puppetmaster2001
14:55 moritzm: rebooting serpens (slapd) for update to 4.9.51
14:50 elukey: updating puppet compiler's facts (following https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource:Puppet3-diffs#FAQ )
14:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1034 weight - T178359 (duration: 00m 49s)
14:26 dcausse: EU swat done
14:23 moritzm: rebooting kubetcd* to 4.9.51
14:18 dcausse@tin: Synchronized wmf-config/CirrusSearch-common.php: T177913 : [cirrus] Add overridden iw prefix for svwiki (2/2) (duration: 00m 48s)
14:16 dcausse@tin: Synchronized wmf-config/InitialiseSettings.php: T177913 : [cirrus] Add overridden iw prefix for svwiki (1/2) (duration: 00m 50s)
14:02 gehel: starting upgrade of elasticsearch eqiad - T178411
13:46 moritzm: rebooting bohrium (piwik host) for update to 4.9.51
13:33 moritzm: installing openssl updates on es* hosts
13:24 moritzm: rebooting dubnium/pollux (openldap corp mirror) for update to 4.9.51
13:12 moritzm: installing openssl updates on pc* hosts
13:07 elukey: restart aqs on aqs100[5-9] to apply localQuorum (https://gerrit.wikimedia.org/r/391765 ) - T164348
13:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1034 weight - T178359 (duration: 00m 49s)
12:54 moritzm: rebooting radium (tor relay) for update to 4.9.51
12:34 moritzm: installing openssl updates on memcached/redis clusters
12:27 hoo: Updated operations/dumps/dcat (ea4e75..7734e04) on snapshot1007
12:09 moritzm: rebooting dbmonitor* hosts for update to 4.9.51
12:03 addshore@tin: Synchronized php-1.31.0-wmf.8/extensions/Wikidata/extensions/Constraints: T180665 Manually applied: Fix SparqlHelper::getCacheMaxAge() (duration: 00m 52s)
11:58 addshore@tin: Synchronized php-1.31.0-wmf.8/extensions/WikibaseQualityConstraints: T180665 Fix SparqlHelper::getCacheMaxAge() (duration: 00m 53s)
11:42 moritzm: installing openssl updates on conf* clusters
11:37 addshore@tin: Synchronized wmf-config/Wikibase-production.php: testwikidata only, T180665 , WBQC configuration for testwikidatawiki (duration: 00m 49s)
11:32 addshore: addshore@terbium:~$ mwscript extensions/WikibaseQualityConstraints/maintenance/ImportConstraintStatements.php --wiki=testwikidatawiki > importConstraintStatements.log
11:21 godog: upgrade prometheus to 1.8.1+ds+k8s-1 in ulsfo/esams/eqiad - T177395
11:09 moritzm: rebooting debug proxies (hassium/hassaleh) for update to 4.9.51
11:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1034 weight - T178359 (duration: 00m 48s)
11:01 jynus: shutting down labsdb1010 to clone to labsdb1009
10:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1034 with low weight - T178359 (duration: 00m 48s)
10:26 addshore: Kill the build deploy slot done!
10:24 addshore@tin: Synchronized php-1.31.0-wmf.8/extensions/Wikibase: T180634 Bring Wikibase up to date with .8 branch of Wikidata build extension (duration: 01m 46s)
10:15 addshore@tin: Synchronized php-1.31.0-wmf.8/extensions/WikibaseQualityConstraints: T180634 Bring WikibaseQualityConstraints up to date with .8 branch of Wikidata build extension (duration: 00m 53s)
09:59 moritzm: rebooting prometheus servers in eqiad for update to 4.9.51
09:44 elukey: restart aqs on aqs1004 to apply localQuorum (https://gerrit.wikimedia.org/r/391765 ) - T164348
09:38 moritzm: rebooting prometheus servers in codfw for update to 4.9.51
09:30 moritzm: uploaded icu57.1-6+wmf2 to jessie-wikimedia/component/icu57
09:23 godog: bootstrap restbase2002-c - T179422
09:19 godog: upgrade grafana to 4.6.1 on https://grafana.wikimedia.org/ - T180428
08:19 marostegui: Deploy schema change on db1092 - T174569
08:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1092 - T174569 (duration: 00m 49s)
07:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1106 (duration: 00m 49s)
07:40 marostegui: Stop MySQL on db1034 to copy its content to db1101.s7 - T178359
07:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1034 - T178359 (duration: 00m 49s)
07:17 ppchelko@tin: Finished deploy [eventlogging/eventbus@872cfb3]: Revert gerrit 302372 due to AssertionError T180017 (duration: 00m 14s)
07:17 ppchelko@tin: Started deploy [eventlogging/eventbus@872cfb3]: Revert gerrit 302372 due to AssertionError T180017
06:54 marostegui: Deploy alter table on db1096 - T174569
06:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1096 - T174569 (duration: 00m 48s)
06:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1099 and db1071 - T174569 (duration: 00m 49s)
06:15 smalyshev@tin: Finished deploy [wdqs/wdqs@b44cf27]: data reload/T176593 (duration: 00m 28s)
06:14 smalyshev@tin: Started deploy [wdqs/wdqs@b44cf27]: data reload/T176593
02:26 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.7) (duration: 07m 54s)
02:09 twentyafterfour: phabricator database migrations complete, service is back online
01:28 twentyafterfour: Phabricator will be offline for a couple of minutes while I apply database migrations.
01:22 twentyafterfour: updating phabricator (belatedly)
01:05 demon@tin: Synchronized scap/plugins/prep.py: no-op, co-master sync (duration: 00m 55s)
00:59 tzatziki: Removing 2FA from AuburnPilot (T180654 )
00:59 tzatziki: Removing 2FA from Twotwo2019 (T180438 )
00:42 eileen: update civicrm from b99a9cf 22f6532
22:48 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: wikidatawiki back to wmf.7
22:21 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.8
22:11 urandom: Setting keyspaces erroneously configured for leveled compaction, to use size-tiered (T180568 )
22:05 madhuvishy: Kicking off re-enabling puppet and puppet runs across Cloud VPS instances
21:33 demon@tin: Synchronized php: symlink swap (duration: 00m 49s)
21:21 madhuvishy: Disabling puppet across cloud VPS through cumin on labpuppetmaster1001
21:01 ottomata: restarting kafka-jumbo brokers to update to Kafka 0.11.0.1
20:52 urandom: Restarting restbase2002-a.codfw.wmnet (T180568 )
20:40 mutante: restarting gerrit to enable 'large file support'-plugin gerrit:391635
20:38 addshore@tin: Finished scap: Update extensions/Wikidata to new wmf/1.31.0-wmf.8 branch (again) T180539 (duration: 21m 48s)
20:17 addshore@tin: Started scap: Update extensions/Wikidata to new wmf/1.31.0-wmf.8 branch (again) T180539
20:11 otto@tin: Finished deploy [eventlogging/eventbus@872cfb3]: deploying kafka-futures change to kafka1001 only, will apply async with https://gerrit.wikimedia.org/r/#/c/391634/ (duration: 00m 20s)
20:11 otto@tin: Started deploy [eventlogging/eventbus@872cfb3]: deploying kafka-futures change to kafka1001 only, will apply async with https://gerrit.wikimedia.org/r/#/c/391634/
19:53 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Add BP and WP aliases for project namespace on mwlwiki (T180052 ) (duration: 00m 48s)
19:48 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable SandboxLink on mwlwiki (T180052 ) (duration: 00m 48s)
19:41 demon@tin: Finished deploy [gerrit/gerrit@bce982f]: adding lfs plugin @ 2.13.9 (duration: 00m 08s)
19:41 demon@tin: Started deploy [gerrit/gerrit@bce982f]: adding lfs plugin @ 2.13.9
19:40 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Disable EventLogging for Popups (T178500 ) (duration: 00m 49s)
19:37 gehel: upgrade of elasticsearch codfw completed, cluster still recovering - T178411
19:34 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable Minerva download icon on all wikis (T179914 ) (duration: 00m 48s)
19:30 chasemp: labstore1004:~# service nfs-kernel-server restart
19:19 catrope@tin: Synchronized php-1.31.0-wmf.8/resources/src/mediawiki.rcfilters/mw.rcfilters.UriProcessor.js: T180577 (duration: 00m 49s)
19:16 catrope@tin: Synchronized php-1.31.0-wmf.7/skins/MinervaNeue/resources/skins.minerva.scripts/init.js: Limit download button to Google Chrome (T179529 , T179914 ) (duration: 00m 49s)
19:14 catrope@tin: Synchronized wmf-config/flaggedrevs.php: Enable RCFilters on all remaining wikis (T177445 ) (duration: 00m 49s)
19:01 urandom: Restarting Cassandra instances on restbase2001.codfw.wmnet (T180568 )
18:38 Jamesofur: removed 2FA from User:Einsbor after verification for votewiki, stewardwiki and SUL
18:20 moritzm: rebooting auth* servers for update to 4.9.51
18:20 moritzm: rebooting auth
17:57 demon@tin: Synchronized docroot/search.wikimedia.org/index.php: removing support for non-wikipedias (duration: 00m 49s)
16:03 addshore@tin: Finished scap: Revert partial scap: Update extensions/Wikidata to new wmf/1.31.0-wmf.8 branch, second try (T180539 ) (duration: 23m 48s)
15:56 oblivian@tin: Finished deploy [docker-pkg/deploy@9b319d2]: Adding do extension to jinja2, needed for contint images (duration: 00m 18s)
15:56 oblivian@tin: Started deploy [docker-pkg/deploy@9b319d2]: Adding do extension to jinja2, needed for contint images
15:55 ema: restart pybal on lvs[12]00[36] for config change https://gerrit.wikimedia.org/r/#/c/389964/
15:43 jynus: shutting down labsdb1009 for maintenance T179244
15:40 addshore@tin: Started scap: Revert partial scap: Update extensions/Wikidata to new wmf/1.31.0-wmf.8 branch, second try (T180539 )
15:29 moritzm: installing perl security updates on trusty (Debian hosts fixed two months ago)
15:14 ladsgroup@tin: scap aborted: Update extensions/Wikidata to new wmf/1.31.0-wmf.8 branch, second try (T180539 ) (duration: 03m 48s)
15:13 moritzm: removing unused kernels from prometheus*
15:11 ladsgroup@tin: Started scap: Update extensions/Wikidata to new wmf/1.31.0-wmf.8 branch, second try (T180539 )
15:05 moritzm: rebooting kubestagetcd* for update to 4.9.51
14:57 ladsgroup@tin: scap aborted: Update extensions/Wikidata to new wmf/1.31.0-wmf.8 branch (T180539 ) (duration: 00m 31s)
14:57 ladsgroup@tin: Started scap: Update extensions/Wikidata to new wmf/1.31.0-wmf.8 branch (T180539 )
14:42 Amir1: deployed ORES change for wikidata (gerrit:391197, phab:T180450 )
14:41 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 49s)
14:37 mobrovac: restbase creating Cassandra 3 revision tables on restbase1009 - T179421
14:30 zfilipin@tin: Synchronized wmf-config/: SWAT: Remove wgContentTranslationEnableSuggestions (duration: 00m 51s)
14:27 ema: cache_upload: upgrade varnish to 4.1.8-1wm2
14:22 zfilipin@tin: Synchronized wmf-config/CommonSettings-labs.php: SWAT: Beta: Explicitly set cookieDomain for ContentTranslationSiteTemplates (T149879) (duration: 00m 49s)
14:18 ema: repool cp4024 T174891
14:17 mutante: wtp2017 - systemctl start ferm (ferm wasnt running due to failed DNS lookup for prometheus2003 sometime in the past)
14:12 zfilipin@tin: Synchronized php-1.31.0-wmf.8/extensions/EventBus/EventBus.php: SWAT: Increase request timeout to match kafka produce timeout (T180017) (duration: 00m 50s)
14:11 godog: upgrade hpsa firmware to 6.06 on restbase2004 - T180562 T141756
14:10 ema: upgrade varnish to 4.1.8-1wm2 on cp4024 (cache_upload, depooled)
14:04 jynus: reload haproxy on dbproxy1010
14:00 moritzm: installing openssl updates on kafka and hadoop clusters
13:34 ema: cache_text: upgrade varnish to 4.1.8-1wm2
13:31 ppchelko@tin: Started restart [changeprop/deploy@065a06e]: Restart to rebalance all rules T179684
13:14 moritzm: rebooting video scalers in eqiad for update to 4.9.51 (and to pick up openssl update)
13:07 ema: upgrade varnish to 4.1.8-1wm2 on cp3030 (cache_text)
12:41 elukey: re-enable eventlogging after maintenance
12:24 jynus: deploying dns change for m4-master
12:19 ema: cache_misc: upgrade varnish to 5.1.3-1wm3
12:09 elukey: executed sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=65 on all jobrunners
12:01 ema: upgrade varnish to 5.1.3-1wm3 on cp3007 (cache_misc)
11:58 ema: varnish 4.1.8-1wm2 uploaded to apt.w.o (main)
11:50 ema: varnish 5.1.3-1wm3 uploaded to apt.w.o (experimental)
11:45 jynus: restart haproxy on dbproxy1009
11:32 marostegui: Stop MySQL on db1046
10:58 moritzm: rebooting job runners in eqiad for update to 4.9.51 (and to pick up openssl update)
10:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1106, optimizing wikidatawiki.wb_terms (duration: 00m 49s)
10:09 ppchelko@tin: Finished deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x attempt 4 T179786 (duration: 04m 51s)
10:04 ppchelko@tin: Started deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x attempt 4 T179786
10:04 ppchelko@tin: Finished deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x attempt 3, force T179786 (duration: 04m 44s)
09:59 ppchelko@tin: Started deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x attempt 3, force T179786
09:59 ppchelko@tin: Finished deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x attempt 2 T179786 (duration: 03m 32s)
09:58 ppchelko@tin: (no justification provided)
09:55 ppchelko@tin: Started deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x attempt 2 T179786
09:55 ppchelko@tin: Finished deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x T179786 (duration: 02m 44s)
09:53 godog: reboot restbase2004 - T180562
09:52 ppchelko@tin: Started deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x T179786
09:52 mobrovac@tin: Started restart [restbase/deploy@c76a665]: Pick up the new seeds definition - T179422
09:52 ppchelko@tin: Started deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x T179786
09:32 godog: restart cassandra on restbase2002-b - T180568
09:30 moritzm: updating openssl on database hosts
09:19 godog: restart cassandra on restbase2001-c - T180568
09:15 marostegui: Stop mysql on db1046 to transfer its content to db1107 - T177405
09:08 elukey: stop eventlogging on eventlog1001, eventlogging replication on db1108/db1047/dbstore1002 as preparation steps to migrate the log db from db1046 to db1107
09:02 jynus: rebooting labsdb1010 for kernel upgrade
08:51 elukey: reboot thorium (hosting all analytics websites) for kernel updates
08:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1101 - going to convert it to mult-instance T178359 (duration: 00m 48s)
07:49 marostegui: Deploy schema change on db1071 and db1099 (s5) - T174569
07:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1099 and db1071 - T174569 (duration: 00m 49s)
07:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1106 and db1100 - T174569 (duration: 00m 49s)
05:33 subbu: subbu@terbium (followup to the linter-reparse.py script log entry) it is safe to kill -9 the script on terbium anytime if there are any problems because of it
05:27 subbu: subbu@terbium running linter-reparse.py script to initialize baseline linter categories for all wikis (12 hours in so far .. expected to run for ~2 weeks). hits parsoid eqiad cluster.
04:06 demon@tin: Synchronized docroot/search.wikimedia.org/robots.txt: go away robots / kill some 404s (duration: 00m 50s)
02:29 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.7) (duration: 05m 57s)
00:59 ejegg: updated payments-wiki from d150287 210cb37
00:53 Reedy: going to restart zuul as it's got backed up
00:33 maxsem@tin: Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/391220/ (duration: 00m 49s)
00:18 ejegg: updated SmashPig payments listener from 5262d53 45aa626
00:12 ejegg: updated CiviCRM from f571c67 b99a9cf
23:29 mutante: releases1001 - chmod g+w /srv/org/wikimedia/releases/mediawiki/1.*
23:01 urandom: Decommissioning Cassandra, restbase1014-a.eqiad.wmnet (T179422 )
22:57 demon@tin: Synchronized w/: Removing mobilelanding from global w/, few sites actually need it (duration: 00m 49s)
22:55 demon@tin: Synchronized docroot/m.wikipedia.org/w/mobilelanding.php: symlink -> real file (duration: 00m 50s)
21:42 no_justification: gerrit: restarted services on master/cobalt, things will flap for a second
21:38 mutante: gerrit2001 - restarted gerrit
21:17 ebernhardson@tin: Finished deploy [search/mjolnir/deploy@494e0c6]: redeploy mjolnir submodule bump to master (duration: 02m 11s)
21:15 ebernhardson@tin: Started deploy [search/mjolnir/deploy@494e0c6]: redeploy mjolnir submodule bump to master
21:03 ebernhardson@tin: Finished deploy [search/mjolnir/deploy@77310bc]: update mjolnir to master (duration: 02m 05s)
21:01 ebernhardson@tin: Started deploy [search/mjolnir/deploy@77310bc]: update mjolnir to master
20:37 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.8
20:15 bblack: reboot cp4024
19:58 ebernhardson@tin: Finished deploy [search/mjolnir/deploy@b20f0da]: test mjolnir deployment (duration: 02m 12s)
19:57 urandom: Bootstrapping restbase2002-b.codfw.wmnet (T179422 )
19:56 ebernhardson@tin: Started deploy [search/mjolnir/deploy@b20f0da]: test mjolnir deployment
19:47 demon@tin: Finished scap: bootstrap wmf.8 (duration: 44m 37s)
19:32 chasemp: clean up instances in error state in testlabs project
19:10 jynus: restart db2034 for mariadb upgrade
19:09 chasemp: for i in `OS_TENANT_NAME=testlabs openstack server list | grep stress | awk '{print $2}'`; do echo $i; OS_TENANT_NAME=testlabs openstack server delete $i; sleep 30; done T171473
19:02 demon@tin: Started scap: bootstrap wmf.8
18:32 arlolra: Updated Parsoid to e71937d0 T178253 )
18:31 mholloway-shell@tin: Finished deploy [mobileapps/deploy@9b10959]: Redeploying: Update mobileapps to c002862
18:28 moritzm: upgraded nginx on notebook* to 1.13.6
18:26 madhuvishy: Upgraded notebook1001 and 1002 to kernel version 4.9.51-1~bpo8+1
18:22 arlolra@tin: Finished deploy [parsoid/deploy@b150764]: Updating Parsoid to e71937d0
18:21 ebernhardson@tin: Finished deploy [search/mjolnir/deploy@e6905f4]: test mjolnir deployment (duration: 01m 13s)
18:19 ebernhardson@tin: Started deploy [search/mjolnir/deploy@e6905f4]: test mjolnir deployment
18:13 arlolra@tin: Started deploy [parsoid/deploy@b150764]: Updating Parsoid to e71937d0
18:08 ebernhardson@tin: Finished deploy [search/mjolnir/deploy@cd6ddda]: (no justification provided) (duration: 04m 28s)
18:07 moritzm: rebooting notebook* hosts for update to 4.9.51
18:04 ebernhardson@tin: Started deploy [search/mjolnir/deploy@cd6ddda]: (no justification provided)
18:02 mholloway-shell@tin: Started deploy [mobileapps/deploy@9b10959]: Redeploying: Update mobileapps to c002862
17:58 ebernhardson@tin: Finished deploy [search/mjolnir/deploy@ceb5c2f]: (no justification provided) (duration: 00m 20s)
17:57 ebernhardson@tin: Started deploy [search/mjolnir/deploy@ceb5c2f]: (no justification provided)
17:56 ebernhardson@tin: Finished deploy [search/MjoLniR/deploy@607adfb]: (no justification provided) (duration: 00m 14s)
17:56 ebernhardson@tin: Started deploy [search/MjoLniR/deploy@607adfb]: (no justification provided)
17:44 herron: restarted puppetdb on nitrogen and nihal to pick up jre updates
17:42 smalyshev@tin: Finished deploy [wdqs/wdqs@b44cf27]: data reload/T176593 (duration: 00m 16s)
17:42 smalyshev@tin: Started deploy [wdqs/wdqs@b44cf27]: data reload/T176593
17:27 demon@tin: Pruned MediaWiki: 1.31.0-wmf.3 (duration: 07m 58s)
17:12 demon@tin: Synchronized wmf-config/CommonSettings.php: Beta-only, no-op (duration: 00m 43s)
17:11 demon@tin: Synchronized wmf-config/extension-list-labs: No-op (duration: 00m 44s)
17:10 demon@tin: Pruned MediaWiki: 1.31.0-wmf.6 [keeping static files] (duration: 02m 48s)
16:32 twentyafterfour: Restarting apache2 on phab1001 (deploy phabricator hotfix: D876 )
16:23 jynus: stop labsdb1010 mariadb to clone it later to labsdb1009 T179244
15:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1105 in s1 and s2 - T178359 (duration: 00m 45s)
15:00 thcipriani@tin: testing IRC logging
15:00 urandom: Decommissioning Cassandra, restbase1012-c.eqiad.wmnet (T179422 )
14:37 moritzm: installing postgres security updates on maps*
14:29 zeljkof: EU SWAT finished
14:07 moritzm: installing postgres security updates on labsdb1004
13:11 moritzm: installing openssl updates
13:10 akosiaris: shutdown install1002 for disk resize
12:57 godog: upgrade grafana to 4.6.1 on https://grafana-labs.wikimedia.org/ - T180428
12:03 ema: restart varnish-be on cp3007, requests failing with 'no backend connection'
11:49 moritzm: rebooting remaining app servers in eqiad for update to Linux 4.9.51 (and to pick up OpenSSL updates)
11:22 moritzm: rebooting remaining API servers in eqiad for update to Linux 4.9.51 (and to pick up OpenSSL updates)
11:04 godog: reimage restbase2002 - T179422
10:50 marostegui: Deploy alter table on s5: db1104 db1100 db1106 - T174569
10:32 elukey: removed old target configs from /srv/prometheus/analytics/targets on prometheus100[34] after https://gerrit.wikimedia.org/r/391179
10:23 moritzm: rebooting image scalers in eqiad for update to Linux 4.9.51 (and to pick up OpenSSL updates)
10:07 godog: upload scap 3.7.2-1 - T127762
09:59 mobrovac@tin: Synchronized wmf-config/jobqueue.php: JobQueue: Migrate RecordLintJob to EventBus - T175212 (duration: 00m 46s)
09:56 marostegui: Deploy alter table on s5 - dbstore1001 - T174569
09:49 foks: Disabled 2FA for Jean-Frédéric
09:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1105 as multi-instance host for s1 and s2 - T178359 (duration: 00m 46s)
09:45 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db1105 as multi-instance host for s1 and s2 - T178359 (duration: 00m 47s)
09:43 godog: upgrade prometheus to 1.8.1 with k8s on prometheus2004 - T177395
09:27 ppchelko@tin: Finished deploy [cpjobqueue/deploy@df5fca9]: Enable RecordLintJob processing (duration: 00m 40s)
09:27 ppchelko@tin: Started deploy [cpjobqueue/deploy@df5fca9]: Enable RecordLintJob processing
09:07 moritzm: rebooting remaining app servers in codfw for update to Linux 4.9.51 (and to pick up OpenSSL updates)
09:05 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw2108.codfw.wmnet
09:05 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: wtp2018.codfw.wmnet
08:31 marostegui: Deploy alter table on s5 - dbstore1002 - T174569
08:22 addshore@tin: Synchronized wmf-config/extension-list-labs: Add AdvancedSearch to extension-list T180147 PT 2/2 LABS ONLY (duration: 00m 46s)
08:21 addshore@tin: Synchronized wmf-config/extension-list: Add AdvancedSearch to extension-list T180147 PT 1/2 (duration: 00m 47s)
08:13 moritzm: rebooting job runners in codfw for update to Linux 4.9.51 (and to pick up OpenSSL updates)
06:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103 on s2 and s4 - T178359 (duration: 00m 47s)
06:18 marostegui: Deploy alter table on s6 primary master (db1061) - T174569
02:30 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Nov 14 02:30:59 UTC 2017 (duration 6m 38s)
02:24 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.7) (duration: 07m 28s)
01:35 mdholloway: mobileapps finished rolling back to 11/8 deployment
01:34 mholloway-shell@tin: Finished deploy [mobileapps/deploy@00e60b2]: (no justification provided) (duration: 02m 27s)
01:31 mholloway-shell@tin: Started deploy [mobileapps/deploy@00e60b2]: (no justification provided)
01:30 mdholloway: mobileapps rolling back today's deployment due to a significant increase in errors.
22:49 bawolff: deployed patch T119158 (Will affect language converter. -{}- no longer allowed in link urls)
22:42 bawolff: deployed patch T124404
22:02 mholloway-shell@tin: Finished deploy [mobileapps/deploy@9b10959]: Update mobileapps to c002862
21:57 mholloway-shell@tin: Started deploy [mobileapps/deploy@9b10959]: Update mobileapps to c002862
21:40 urandom: Decommissioning Cassandra, restbase1012-b.eqiad.wmnet (T179422 )
20:20 ejegg: updated CiviCRM from ddc3881 f571c67
19:20 smalyshev@tin: Finished deploy [wdqs/wdqs@b44cf27]: data reload/T176593 (duration: 00m 20s)
19:20 smalyshev@tin: Started deploy [wdqs/wdqs@b44cf27]: data reload/T176593
19:18 thcipriani@tin: Synchronized wmf-config/abusefilter.php: SWAT: Enable per-filter profiling on enwiki T179323 (duration: 00m 45s)
19:16 thcipriani@tin: Synchronized docroot/search.wikimedia.org/index.php: search.wikimedia.org: Clean up result returning logic (duration: 00m 47s)
19:15 madhuvishy: Kick of second dumps rsync from ms1001 to labstore1006
18:28 gehel: deploy latest blazegraph + GUI on wdqs200[23] to switch vocabulary - T176593
18:20 elukey: drain + shutdown analytics1029 as prep step to replace the BBU - T178742
18:09 hoo: Ran "scap pull" on mwdebug1001/snapshot1001 after (further) tests re T177486
18:02 urandom: Restarting Cassandra, restbase-dev1004.eqiad.wmnet (testing new `c-foreach-restart`)
18:00 demon@tin: Synchronized docroot/search.wikimedia.org/index.php: minor cleanup (duration: 00m 47s)
17:28 hoo: Ran "scap pull" on mwdebug1001 after tests re T177486
17:12 mobrovac: restbase depooling restbase1007, restbase1012, restbase1014, restbase2002, restbase2004, restbase2006 for T179422
16:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103 on s2 and s4 - T178359 (duration: 00m 47s)
16:45 mobrovac@tin: Finished deploy [mathoid/deploy@63b2ddc]: Update to service-template-node v0.5.3 - T151396 (duration: 03m 45s)
16:42 mobrovac@tin: Started deploy [mathoid/deploy@63b2ddc]: Update to service-template-node v0.5.3 - T151396
16:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1085 - T174569 (duration: 00m 46s)
15:58 akosiaris@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=wtp2017.codfw.wmnet
15:58 akosiaris@puppetmaster1001: conftool action : set/pooled=no; selector: name=wtp2017.codfw.wmnet
15:22 otto@tin: Finished deploy [eventlogging/analytics@e024af3]: T179625 (duration: 00m 02s)
15:22 otto@tin: Started deploy [eventlogging/analytics@e024af3]: T179625
15:09 otto@tin: Finished deploy [eventlogging/analytics@03285e4]: Reverting, got an error: userAgent is a <type unicode>. (duration: 00m 02s)
15:09 otto@tin: Started deploy [eventlogging/analytics@03285e4]: Reverting, got an error: userAgent is a <type unicode>.
15:08 urandom: Decommissioning Cassandra, restbase1012-a.eqiad.wmnet (T179422 )
15:06 otto@tin: Finished deploy [eventlogging/analytics@5796c27]: T179625 (duration: 00m 04s)
15:06 otto@tin: Started deploy [eventlogging/analytics@5796c27]: T179625
14:58 herron: upgrading puppetmaster2002 to puppet 4
14:15 zeljkof: EU SWAT finished
14:10 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Whitelist jenkins in test wiki (T167432 ) (duration: 00m 47s)
14:08 marostegui: Deploy alter table on db1102.s6 (with replication - sanitarium master) - T174569
13:49 marostegui: Stop replication on labsdb1010 to copy cebwiki.geo_tags table
13:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103 on s2 and s4 - T178359 (duration: 00m 46s)
13:36 marostegui@tin: Synchronized wmf-config/db-codfw.php: Increase weight for db1103 on s2 and s4 - T178359 (duration: 00m 46s)
13:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1083 - T174569 (duration: 00m 46s)
13:09 marostegui: Deploy schema change on db1083 - T174569
12:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1088 - T174569 (duration: 00m 47s)
12:14 ema: cache_misc: upgrade varnish to 5.1.3-1wm2
12:08 ema: cp3008: upgrade varnish to 5.1.3-1wm2
12:02 ema: cp4021: restart varnish-be (mbox lag)
11:56 moritzm: installing imagemagick security updates
11:48 moritzm: installing irssi security updates
11:18 elukey: restart of all the druid daemons on druid100[1-6] to apply the new prometheus jmx jvm exporters - T177459
10:57 gehel: upgrade elasticsearch on cirrus / codfw - T178411
10:49 marostegui: Deploy schema change on db1088 - T174569
10:49 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1088 - T174569 (duration: 00m 54s)
10:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1093 - T174569 (duration: 00m 46s)
10:28 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db1103 as multi-instance host for s2 and s4 - T178359 (duration: 00m 46s)
10:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1103 as multi-instance host for s2 and s4 - T178359 (duration: 00m 47s)
10:12 moritzm: rebooting remaining Parsoid hosts in eqiad for update to Linux 4.9.51 (and to pick up OpenSSL update)
09:44 godog: test upgrade of prometheus 1.8.1 with k8s on prometheus2003 - T177395
09:29 moritzm: rebooting mw1221-mw1235 for update to Linux 4.9.51 (and to pick up OpenSSL update)
09:02 elukey: restart of druid brokers on druid100[1-6] to apply https://gerrit.wikimedia.org/r/390419 - T177459
08:41 marostegui: Deploy alter table on db1093 - T174569
08:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1093 - T174569 (duration: 00m 46s)
08:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1098 after alter table - T174569 (duration: 00m 47s)
08:33 moritzm: rebooting mw1238-mw1258 for update to Linux 4.9.51 (and to pick up OpenSSL update)
07:48 moritzm: installing ruby2.3 security updates
07:38 marostegui: Deploy alter table to db1104 - T179106
07:33 marostegui: Optimize wb_terms table on db2052 - T179106
06:44 marostegui: Deploy alter table directly on codfw s5 master (db2023), this will generate lag on codfw - T179793
06:27 marostegui: Deploy alter table on db1098 - T174569
06:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1098 - T174569 (duration: 00m 49s)
06:18 marostegui: Deploy alter table db2086 - T179106
04:01 urandom: Decommissioning Cassandra, restbase1007-c.eqiad.wmnet (T179422 )
02:38 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Nov 13 02:38:38 UTC 2017 (duration 6m 42s)
02:31 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.7) (duration: 08m 18s)
19:39 urandom: Decommissioning Cassandra, restbase1007-b.eqiad.wmnet (T179422 )
16:19 bblack: cp4026 - restart backend (mailbox lag)
01:29 urandom: Decommissioning Cassandra, restbase1007-a.eqiad.wmnet (T179422 )
13:52 urandom: Decommissioning Cassandra, restbase2006-c.codfw.wmnet (T179422 )
00:48 urandom: Decommissioning Cassandra, restbase2006-b.codfw.wmnet (T179422 )
18:08 smalyshev@tin: Finished deploy [wdqs/wdqs@ccab8ce]: data reload/T176593 (duration: 00m 34s)
18:08 smalyshev@tin: Started deploy [wdqs/wdqs@ccab8ce]: data reload/T176593
17:20 moritzm: uploaded icu 57.1-6+wmf1 for jessie-wikimedia/component/icu57 (co-installable build for ICU migration)
17:11 moritzm: freed some disk space on install1002
16:47 marostegui: Deploy alter table on s6, dbstore1001, dbstore1002 abd db1030 - T174569
15:03 moritzm: rebooting remaing API servers in codfw to 4.9.51 (and to pick up OpenSSL updates)
14:57 urandom: Decommissioning Cassandra, restbase2006-a.codfw.wmnet (T179422 )
14:17 moritzm: rebooting image scalers in codfw to 4.9.51 (and to pick up OpenSSL updates)
13:10 marostegui: truncate /var/log/nginx/error.log.1 on install1002 as it is filling up
13:09 moritzm: powercycling mw2118, stuck after reboot
13:05 ema: cp4021: restart varnish-be due to mbox lag
12:48 moritzm: rebooting video scalers in codfw to 4.9.51 (and to pick up OpenSSL updates)
11:46 _joe_: restarted all services and repooled scb1001
11:38 moritzm: rebooting mw2163-2199 to 4.9.51 (and to pick up OpenSSL updates)
11:32 _joe_: stopping mobileapps as well on scb1001
11:27 moritzm: rebooting wtp1025 to 4.9.51
11:22 _joe_: stopping changeprop, celery-ores, cpjobqueue on scb1001
11:21 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Disable AdvancedSearch on deployment.beta BETA ONLY T180201 (duration: 00m 46s)
11:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove old comment about db1080 (duration: 00m 46s)
11:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1055 original weight - T178359 (duration: 00m 46s)
10:59 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: wtp2017.codfw.wmnet
10:55 _joe_: depooling scb1001 from all services while it becomes healthy again
10:52 _joe_: restarting ores on scb1001, causing memory exhaustion
10:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1055 weight - T178359 (duration: 00m 47s)
10:49 moritzm: powercycling wtp2017, stuck after reboot
10:24 marostegui: Deploy schema change on db2089 - T179106
10:11 moritzm: rebooting Parsoid servers in codfw to 4.9.51 (and to pick up OpenSSL updates)
10:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1055 weight - T178359 (duration: 00m 58s)
09:54 marostegui: Compress enwiki on db1105.s1 - T178359
09:45 moritzm: powercycling mw2108, stuck after reboot
09:30 hashar: Upgrading operations-puppet-tests-docker jenkins job to stop passing docker --tty and thus have signals forwarded from 'docker run' - T176747
09:23 moritzm: rebooting mw2097-mw2117 to 4.9.51 (and to pick up OpenSSL updates)
09:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1055 with low weight - T178359 (duration: 00m 47s)
09:13 moritzm: powercycling mw2213, stuck after reboot
08:43 moritzm: rebooting mw2200-mw2223 to 4.9.51 (and to pick up OpenSSL updates)
07:50 smalyshev@tin: Finished deploy [wdqs/wdqs@213f864]: (no justification provided) (duration: 00m 33s)
07:49 smalyshev@tin: Started deploy [wdqs/wdqs@213f864]: (no justification provided)
07:27 _joe_: restarting apache on phab1001
07:20 marostegui: Deploy alter table on s3.codfw master (db2018) with replication, this will generate lag on codfw - T174569
06:50 marostegui: Deploy alter table on s5 eqiad master (db1063) - T172207
06:41 marostegui: Stop MySQL on db1055 to copy its content to db1105 - T178359
06:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1055 - T178359 (duration: 00m 49s)
06:39 marostegui: Force a BBU relearn on db1046 - T166141
01:21 aaron@tin: Synchronized php-1.31.0-wmf.7/includes/db: Use the main stash for LBFactory "memStash" parameter (duration: 00m 47s)
01:18 demon@tin: Synchronized docroot/search.wikimedia.org/index.php: minor cleanups, less 500s (duration: 00m 47s)
22:28 urandom: Decommissioning Cassandra, restbase2004-c.codfw.wmnet (T179422 )
20:28 demon@tin: Synchronized php-1.31.0-wmf.7/includes/libs/objectcache/WANObjectCache.php: less spammy error logs (duration: 00m 47s)
20:09 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.7
19:06 ebernhardson@tin: Synchronized php-1.31.0-wmf.7/extensions/WikimediaEvents/modules/all/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Turn off DBN sizing AB test (duration: 00m 51s)
19:00 bblack: cp3030 - end experimentation, puppetizing back to normal config
18:46 bblack: cp3030 - round 2 of ssl_do_wait_shutdown test
18:41 arlolra: Updated Parsoid to 2887b5ad T178253 , T173643 , T176728 , T180010 , T171381 , T179757 )
18:28 arlolra@tin: Finished deploy [parsoid/deploy@d1c7386]: Updating Parsoid to 2887b5ad
18:22 bblack: cp3030: puppet-disabled + manual nginx ssl_do_wait_shutdown config
18:16 arlolra@tin: Started deploy [parsoid/deploy@d1c7386]: Updating Parsoid to 2887b5ad
18:14 moritzm: not rebooting parsoid hosts due to Services deployment window, instead rolling restart of mw2120-mw2139 for kernel update to 4.9.51
18:04 moritzm: rolling restart of parsoid servers in codfw for 4.9.51 kernel update
17:14 urandom: Restarting Cassandra, restbase2005-b.codfw.wmnet (T179419 )
16:33 urandom: Restarting Cassandra, restbase2005-a.codfw.wmnet (T179419 )
15:12 urandom: Creating mathoid schema (T179419 )
15:05 addshore@tin: Synchronized wmf-config/CommonSettings-labs.php: Enable AdvancedSearch on beta LABS / BETA ONLY PT2/2 (duration: 00m 49s)
15:04 addshore: last sync was actually "Enable AdvancedSearch on beta"
15:04 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Add AdvancedSearch to extension-list-labs LABS / BETA ONLY PT1/2 (duration: 00m 50s)
14:57 addshore@tin: Synchronized wmf-config/extension-list-labs: Add AdvancedSearch to extension-list-labs LABS / BETA ONLY (duration: 00m 50s)
14:42 zeljkof: EU SWAT finished
14:40 zfilipin@tin: Synchronized wmf-config/StartProfiler.php: SWAT: xenon: encode the request method as a virtual stack frame (duration: 00m 50s)
14:35 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Use a threshold that ores in frwiki can stand (T180115 ) (duration: 00m 50s)
14:30 zfilipin@tin: Synchronized php-1.31.0-wmf.7/extensions/ContentTranslation/modules/: SWAT: Bring back the overlay support for a specific screen region (T179997) (duration: 00m 50s)
14:25 zfilipin@tin: Synchronized php-1.31.0-wmf.7/extensions/EventBus/EventBus.php: SWAT: Logging improvements Rename logged field to fix logstash mapping (duration: 00m 54s)
14:09 urandom: Decommissioning Cassandra, restbase2004-b.codfw.wmnet (T179422 )
13:04 moritzm: rebooting mw1209-mw1220 (app servers) to 4.9.51 (also to pick up new OpenSSL)
12:38 moritzm: rebooting mw1189-mw1208 (API servers) to 4.9.5 (also to pick up new OpenSSL)
12:37 Amir1: ladsgroup@terbium:/srv/mediawiki-staging/php-1.31.0-wmf.6$ mwscript extensions/ORES/maintenance/CheckModelVersions.php --wiki=frwiki (T180115 )
11:51 moritzm: rebooting mw1180-mw1188 (app servers) to 4.9.5 (also to pick up new OpenSSL)
11:45 _joe_: removed all local hacks from puppetmaster1001, now it uses rhodium again
11:37 _joe_: cleaning up spurious directories /var/lib/puppet/server/ssl/ca from eqiad's puppetmaster backends, generated due to some error on 8/11/2017
11:10 ema: powercycle cp2008, stuck rebooting
11:03 moritzm: rebooting mw1276-mw1279 (API canaries) to 4.9.5 (also to pick up new OpenSSL)
09:50 moritzm: rolling reboot of scb in eqiad for kernel update (also to pick up openssl updates)
07:56 ema: cp1074 failed rebooting, power-cycled
07:46 _joe_: restarting apache on rhodium after setting --profile --trace in the puppet settings
07:04 legoktm@tin: Synchronized php-1.31.0-wmf.7/resources/: Restore jquery.badge and jquery.placeholder modules (duration: 00m 53s)
02:58 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Nov 9 02:58:30 UTC 2017 (duration 6m 59s)
02:51 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.7) (duration: 09m 09s)
02:29 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.6) (duration: 09m 18s)
01:02 twentyafterfour: Not deploying any phabricator updates this week.
23:45 urandom: Decommissioning Cassandra, restbase2004.codfw.wmnet (T179422 )
22:50 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.7 try #2
22:43 ebernhardson@tin: Synchronized php-1.31.0-wmf.6/extensions/CirrusSearch/: Backport cirrus rescore profile refactor to wmf.6 (duration: 01m 02s)
22:21 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 back to wmf.6
22:09 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.7
21:44 bsitzmann@tin: Finished deploy [mobileapps/deploy@00e60b2]: Update mobileapps to 8e82983 T178706 T178708 T178333 T170692 ) (duration: 07m 12s)
21:37 bsitzmann@tin: Started deploy [mobileapps/deploy@00e60b2]: Update mobileapps to 8e82983 T178706 T178708 T178333 T170692 )
21:34 ejegg: re-started donations queue consumer with new thank you letter
21:09 ejegg: updated CiviCRM from b11c591 ddc3881
20:52 ejegg: turned off donations consumer for ty letter update
20:36 ejegg: updated payments-wiki from a539d27 d150287
20:33 aaron@tin: Synchronized php-1.31.0-wmf.6/extensions/CentralAuth: Use the proper cache key method in loadFromCache() (duration: 00m 54s)
20:22 apergos: rsync from ms1001 to labstore1006 of dumps, 17T so expect it to take several days
20:10 herron: depooled rhodium via puppetmaster1001 apache config
20:09 demon@tin: Synchronized php: symlink (duration: 00m 49s)
19:57 demon@tin: Synchronized php-1.31.0-wmf.7/extensions/CentralAuth/includes/CentralAuthUser.php: (no justification provided) (duration: 00m 51s)
19:55 urandom: Creating mathoid schema (T179419 )
19:31 niharika29@tin: Synchronized php-1.31.0-wmf.6/extensions/ORES/: Store stats of accessing ores service for getting thresholds T179862 (duration: 00m 51s)
19:31 urandom: Creating page restrictions schema (T179421 )
19:30 niharika29@tin: Synchronized php-1.31.0-wmf.7/extensions/ORES/: Store stats of accessing ores service for getting thresholds T179862 (duration: 00m 51s)
19:24 niharika29@tin: Synchronized php-1.31.0-wmf.7/extensions/CirrusSearch/: Revert Improve handling of 5xx responses to elasticsearch requests (duration: 01m 02s)
19:14 smalyshev@tin: Finished deploy [wdqs/wdqs@b330bc8]: Update service whitelist (duration: 02m 58s)
19:11 smalyshev@tin: Started deploy [wdqs/wdqs@b330bc8]: Update service whitelist
18:27 demon@tin: Synchronized wmf-config/: Dropping old PrivateSettings symlink (ducks and covers) (duration: 00m 52s)
18:19 demon@tin: Synchronized phpcs.xml: no-op (duration: 00m 50s)
17:58 urandom: Restarting Cassandra, restbase2005-[abc]
17:51 urandom: Clearing snapshots in RESTBase legacy Cassandra cluster (T179417 )
17:47 marostegui: Deploy alter table on s1 codfw primary master (db2048) with replication, this will generate lag on codfw - T174569
17:43 mobrovac: restbase truncate the default parsoid storage group's tables for T179417
17:41 urandom: Restarting Cassandra, restbase2001-[abc]
17:19 urandom: Restarting Cassandra, restbase2003-[abc]
17:09 herron: regenerated rhodium puppet certificate
17:08 urandom: Restarting Cassandra, restbase1009-[abc]
16:58 urandom: Restarting Cassandra, restbase1008-[abc]
16:49 awight@tin: Finished deploy [ores/deploy@82a13ae]: Roll back scb1002 (duration: 02m 37s)
16:47 awight@tin: Started deploy [ores/deploy@82a13ae]: Roll back scb1002
16:45 awight@tin: Finished deploy [ores/deploy@1b0e59f]: Try to purge specter of revscoring 1 (duration: 05m 45s)
16:40 awight@tin: Started deploy [ores/deploy@1b0e59f]: Try to purge specter of revscoring 1
16:37 urandom: Restarting Cassandra, restbase1010-[abc]
16:37 godog: disregard message about thumbor rolling-restart, upgrade already done and only thumbor1001 rebooted now
16:34 awight@tin: Finished deploy [ores/deploy@82a13ae]: Fix ORES on scb1002 (duration: 00m 03s)
16:34 awight@tin: Started deploy [ores/deploy@82a13ae]: Fix ORES on scb1002
16:29 godog: roll-restart thumbor in eqiad for kernel upgrade
16:11 mobrovac@tin: Finished deploy [restbase/deploy@c5dd1e2]: Switch wiktionary definitions to use the next-gen storage, take 2b - T179420 (duration: 07m 22s)
16:04 mobrovac@tin: Started deploy [restbase/deploy@c5dd1e2]: Switch wiktionary definitions to use the next-gen storage, take 2b - T179420
16:02 mobrovac@tin: Finished deploy [restbase/deploy@c5dd1e2]: Switch wiktionary definitions to use the next-gen storage, take 2 - T179420 (duration: 00m 13s)
16:02 mobrovac@tin: Started deploy [restbase/deploy@c5dd1e2]: Switch wiktionary definitions to use the next-gen storage, take 2 - T179420
16:01 otto@tin: Finished deploy [eventlogging/analytics@03285e4]: Reverting EvenCapsule update and fixes, processes got restarted too early (duration: 00m 02s)
16:01 otto@tin: Started deploy [eventlogging/analytics@03285e4]: Reverting EvenCapsule update and fixes, processes got restarted too early
15:33 urandom: Decommissioning restbase2001-c.codfw.wmnet (T179422 )
15:23 _joe_: testing changes on rhodium regarding hostprivkey,hostcert
15:21 ema: eqiad lvs reboots: upgrading kernel to 4.9.51, libssl to 1.0.2m
15:17 otto@tin: Finished deploy [eventlogging/analytics@02c5a6b]: EventCapsule update and fixes, this is no-op as is. T179625 (duration: 00m 04s)
15:17 otto@tin: Started deploy [eventlogging/analytics@02c5a6b]: EventCapsule update and fixes, this is no-op as is. T179625
15:01 otto@tin: Started restart [eventlogging/eventbus@41e3418]: Bumping worker processes to 16 on all targets: T180017
14:55 otto@tin: Started restart [eventlogging/eventbus@41e3418]: Bumping worker processes to 16: T180017
14:54 ema: esams lvs reboots: upgrading kernel to 4.9.51, libssl to 1.0.2m
14:54 otto@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 12s)
14:54 otto@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
14:41 ema: powercycle cp1050 (failed reboot)
14:29 addshore@tin: Synchronized php-1.31.0-wmf.7/docs/uidesign/mediawiki.diff.html: SWAT Add render moved paragraphs marker in diff view PT 2/2 DOCS ONLY (duration: 00m 50s)
14:28 addshore@tin: Synchronized php-1.31.0-wmf.7/resources/src/mediawiki/mediawiki.diff.styles.css: SWAT Add render moved paragraphs marker in diff view PT 1/2 (duration: 00m 51s)
14:16 volans: upgrading cumin to v1.3.0 on prod and WMCS cumin masters
14:10 zfilipin@tin: Synchronized php-1.31.0-wmf.6/extensions/EventBus/EventBus.php: SWAT: Logging improvements. (duration: 00m 52s)
14:08 ema: codfw lvs reboots: upgrading kernel to 4.9.51, libssl to 1.0.2m
13:03 hasharAway: Upgrading jenkins on contint1001/contint2001
12:54 bblack@puppetmaster1001: conftool action : set/pooled=yes; selector: name=phab2001-vcs.codfw.wmnet
12:53 bblack: restart pybal on lvs2002 for git-ssh.codfw deploy
12:51 bblack: restart pybal on lvs2005 for git-ssh.codfw deploy
12:46 mutante: osmium - re-enabling puppet - temp test is over and will be decom'ed
12:41 mutante: krypton (misc PHP apps, scholarships.wm, iegreview.wm, grafana, racktables, burrow) rebooting for kernel upgrade
12:35 mutante: bromine (misc static tistes, annual/transparency/static-bz) - rebooting for kernel upgrade
12:07 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1051 original weight after maintenance - T178359 (duration: 00m 50s)
12:03 mutante: alcyone (url-downloader) rebooting for kernel upgrade
11:53 mutante: netmon1003 (servermon) - rebooting for kernel upgrade
11:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1051 weight after maintenance - T178359 (duration: 00m 50s)
11:48 moritzm: installed openjdk-8/openssl updates and new kernels on restbase*
11:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1051 weight after maintenance - T178359 (duration: 00m 50s)
11:12 moritzm: installing jenkins security update on releases*
11:10 moritzm: imported jenkins 2.73.3 to apt.wikimedia.org
10:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1051 with low weight after maintenance - T178359 (duration: 01m 01s)
10:57 ema: ulsfo lvs reboots: upgrading kernel to 4.9.51, libssl to 1.0.2m
10:42 mutante: actinium - reboot for kernel upgrade (url-downloader)
10:40 ema: varnish 5.1.3-1wm2 built and uploaded to apt.w.o (experimental)
10:39 mutante: aluminium - reboot for kernel upgrade (url-downloader)
10:28 elukey@puppetmaster1001: conftool action : set/pooled=no; selector: name=aqs1005.eqiad.wmnet
10:18 elukey@puppetmaster1001: conftool action : set/pooled=no; selector: name=aqs1004.eqiad.wmnet
10:07 elukey: reboot aqs100[4-9] for jvm and kernel updates
09:37 mutante: planet1001, alsafi (url-downloader) - apt autoremove; reboot for kernel upgrade
09:32 mutante: restarting ircecho (icinga-wm)
09:26 mutante: ununpentium (rt.wikimedia.org): apt-get autoremove; reboot for kernel upgrade
09:23 mutante: rutherforidum (people.wikimedia.org) : apt-get autoremove ; reboot for kernel upgrade
09:22 Pchelolo: restart cassandra-a on restbase1010
09:09 Pchelolo: restart restbase on 1013 and 1015
08:58 mutante: planet2001 - apt autoremove; reboot for kernel upgrade
08:40 ema: resume cache_text/upload rolling reboots: upgrading kernel to 4.9.51, libssl to 1.0.2m and 1.1.0g
07:55 marostegui: Deploy alter table on s1 - on codfw master (db2048) with replication enabled - T172207
07:46 marostegui: Deploy alter table on s2 - on codfw master (db2017) with replication enabled - T172207
07:21 marostegui: Deploy alter table on s3 - on codfw master (db2018) with replication enabled - T172207
07:13 marostegui: Deploy alter table on s7 - on codfw master (db2029) with replication enabled - T172207
07:13 krinkle@tin: Synchronized docroot/noc/conf/: I2e51e783a
06:57 marostegui: Deploy alter table on s6 - on codfw master (db2028) with replication enabled - T172207
06:26 marostegui: Add 330G to db2023 partition to make sure the alter over logging table runs fine - T174569
06:16 Krinkle: Restarted uwsgi-graphite-web service on graphite2001
06:16 Krinkle: Restarted uwsgi-graphite-web service on graphite1001
06:15 marostegui: Stop MySQL on db1051 to copy its content to db1105.s1 - T178359
06:14 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1051 - T178359 (duration: 00m 51s)
05:45 Krinkle: rm -rf /var/lib/carbon/whisper/MediaWiki/wanobjectcache/centralauth_user_* on graphite1001 and graphite2001 for T179999
03:17 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Nov 8 03:17:53 UTC 2017 (duration 7m 13s)
03:10 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.7) (duration: 15m 16s)
02:44 aaron@tin: Synchronized php-1.31.0-wmf.7/tests: Deploy 087f2d579a9f 4432e898be0
02:42 aaron@tin: Synchronized php-1.31.0-wmf.7/includes: Deploy 087f2d579a9f 4432e898be0
02:33 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.6) (duration: 07m 38s)
23:57 urandom: Decommissioning restbase2001-b.codfw.wmnet (T179422 )
22:50 ejegg: re-enabled thank you mailer
22:49 ejegg: updated CiviCRM from 85e89c6 dc1b279
22:39 hoo: Reset a global account's email, per T179950
22:28 ejegg: updated CiviCRM from 122ba65 85e89c6
21:56 ejegg: updated CiviCRM from dc9d054 122ba65
21:20 addshore@tin: Synchronized wmf-config/Wikibase-buildentry.php: T176948 Remove Shared Cache settings from Wikibase-buildentry (duration: 00m 50s)
21:16 addshore@tin: Synchronized wmf-config/Wikibase.php: T176948 Stop using wgWikibaseSharedCacheKeyPrefix from Wikidata build (duration: 00m 49s)
21:09 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: T176948 Remove wmgWikibaseUseConfigFromWikidataBuild PT 3/3 (LABS) (duration: 00m 50s)
21:07 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T176948 Remove wmgWikibaseUseConfigFromWikidataBuild PT 2/3 (duration: 00m 52s)
21:06 addshore@tin: Synchronized wmf-config/Wikibase.php: T176948 Remove wmgWikibaseUseConfigFromWikidataBuild PT 1/3 (duration: 00m 50s)
21:02 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T176948 wmgWikibaseUseConfigFromWikidataBuild flase for all of PROD (duration: 00m 49s)
21:01 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T176948 Load wikibase build from mediawiki-config for wikidataclient (duration: 00m 50s)
20:59 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T176948 Load wikibase build from mediawiki-config for group0 & group1 (duration: 00m 50s)
20:54 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T176948 Load wikibase build from mediawiki-config for hewiki (duration: 00m 49s)
20:51 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T176948 Load wikibase build from mediawiki-config for wikidatawiki (duration: 00m 50s)
20:48 herron: puppet issue cleared after reverting 386666. restarting ircecho on einsteinium
20:45 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: T176948 #1 #2 #3 Load wikibase build from mediawiki-config for BETA ONLY (duration: 00m 50s)
20:28 addshore@tin: Synchronized wmf-config/Wikibase.php: Add loading of wikibase extensions from build PT 3/3 (duration: 00m 50s)
20:27 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Add loading of wikibase extensions from build PT 2/3 (duration: 00m 50s)
20:25 addshore@tin: Synchronized wmf-config/Wikibase-buildentry.php: Add loading of wikibase extensions from build PT 1/3 (duration: 00m 49s)
20:20 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Remove unused wmgUseWikibasePropertySuggester PT 2/2 (duration: 00m 50s)
20:19 addshore@tin: Synchronized wmf-config/CommonSettings.php: Remove unused wmgUseWikibasePropertySuggester PT 1/2 (duration: 00m 50s)
20:06 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.7
19:44 volans: restarted apache2 on rhodium (puppet master failing)
19:31 herron: restart apache2 service on rhodium
19:12 demon@tin: Finished scap: wmf.7 bootstrap (duration: 48m 15s)
19:03 awight: begin stress test on ores*
18:58 awight@tin: Finished deploy [ores/deploy@29905e5]: test deployment to repair ores1009 (non-production) (duration: 00m 31s)
18:57 awight@tin: Started deploy [ores/deploy@29905e5]: test deployment to repair ores1009 (non-production)
18:56 awight@tin: Finished deploy [ores/deploy@29905e5]: test deployment to repair ores1008 (non-production) (duration: 00m 32s)
18:56 awight@tin: Started deploy [ores/deploy@29905e5]: test deployment to repair ores1008 (non-production)
18:50 awight@tin: Finished deploy [ores/deploy@29905e5]: test deployment to repair ores1002 (non-production) (duration: 00m 33s)
18:49 awight@tin: Started deploy [ores/deploy@29905e5]: test deployment to repair ores1002 (non-production)
18:39 volans: slowly running puppet on failed hosts with cumin (concurrency=5)
18:38 awight@tin: Finished deploy [ores/deploy@29905e5]: test deployment to ores* (non-production) (duration: 00m 20s)
18:38 awight@tin: Started deploy [ores/deploy@29905e5]: test deployment to ores* (non-production)
18:36 _joe_: restarting apache2 on rhodium
18:24 demon@tin: Started scap: wmf.7 bootstrap
18:16 awight@tin: Finished deploy [ores/deploy@29905e5]: test deployment to ores* (non-production) (duration: 01m 04s)
18:15 awight@tin: Started deploy [ores/deploy@29905e5]: test deployment to ores* (non-production)
18:06 herron: restarting puppetdb service on nitrogen
17:56 urandom: Decommissioning restbase2001-a.codfw.wmnet (T179422 )
17:55 elukey: stop ircecho on einstenium (puppet shower from nitrogen)
17:40 ema: stop cache_text/upload rolling reboots, resuming tomorrow
{{safesubst:SAL entry|1=17:38 urandom: Restart Cassandra, restbase1010-{a,b,c}.eqiad.wmnet (T178177)}}
17:33 urandom: Clearing Cassandra snapshots (T179422 )
17:32 moritzm: rolling reboot of scb in codfw to pick up new kernel (and openssl updates)
{{safesubst:SAL entry|1=17:02 urandom: Restarting Cassandra, restbase2001-{a,b,c} to apply OpenJDK upgrade}}
16:47 demon@tin: Synchronized multiversion/submodules.json: no op (duration: 00m 47s)
16:12 ema: start cache_text/upload rolling reboots: upgrading kernel to 4.9.51, libssl to 1.0.2m and 1.1.0g
15:46 gehel: rolling restart of cassandra maps-test for logging change
15:43 godog: roll-restart thumbor in eqiad for kernel upgrade
15:37 urandom: T179420 : recreating wiktionary definition schemas
15:29 mobrovac@tin: Finished deploy [restbase/deploy@eab2948]: revert definition switch, wrong schema - T179420 (duration: 06m 46s)
15:22 mobrovac@tin: Started deploy [restbase/deploy@eab2948]: revert definition switch, wrong schema - T179420
15:12 _joe_: added a runner for htmlCacheUpdate on cewiki too
15:10 mobrovac@tin: Finished deploy [restbase/deploy@c5dd1e2]: Switch wiktionary definitions to use the next-gen storage - T179420 (duration: 07m 52s)
15:02 mobrovac@tin: Started deploy [restbase/deploy@c5dd1e2]: Switch wiktionary definitions to use the next-gen storage - T179420
14:52 godog: roll-restart thumbor for kernel upgrades
14:38 mobrovac: restbase creating wiktionary definition schemas for T179420
14:36 godog: reboot wezen for kernel upgrade
14:33 godog: reboot lithium for kernel upgrade
14:26 elukey: rolling restart of kafka on kafka-jumbo* for jvm security updates
14:20 moritzm: rebooting mw canaries to 4.9.51 kernel (also picking up openssl/openssl1.1 updates)
14:19 moritzm: rearming keyholder on naos
14:14 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable ShortUrl on pa.wiki - T178919 (duration: 00m 46s)
14:11 hashar: mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki=pawiki ShortUrl - T178919
14:08 moritzm: rebooting tureis for kernel update
14:07 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: add _ in appendix talk namespace at mywiktionary - T179907 (duration: 00m 45s)
14:02 moritzm: actually holding mw canary reboots until SWAT is over
14:01 moritzm: rebooting mw canaries to 4.9.51 kernel (also picking up openssl/openssl1.1 updates)
13:50 herron: rebooted fermium (lists) for kernel update
13:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add comment about db1105 current status (duration: 00m 47s)
13:43 herron: starting rolling mx reboots for kernel update
13:31 marostegui: Deploy schema change on s5 codfw master (db2023) with replication, this will generate lag on codfw - T174569
13:19 moritzm: rebooting naos for kernel update
13:17 gehel: reboot maps eqiad cluster for upgrades
13:10 moritzm: rebooting wasat for kernel update
12:50 mutante: hafnium, tungsten: groupdel perf-roots to go with gerrit:389663 (T179728 )
12:00 mobrovac@tin: Finished deploy [restbase/deploy@eab2948]: Use the new storage for wikidata.org - T179417 (duration: 08m 14s)
11:52 mobrovac@tin: Started deploy [restbase/deploy@eab2948]: Use the new storage for wikidata.org - T179417
11:43 mobrovac: restbase truncating cassandra 2 non-WP tables for T179417
11:42 mobrovac: restbase truncating cassandra 2 non-WP tables for T179420
11:29 moritzm: installing java security updates/restarting cassandra on restbase2001 (cassandra3 node)
11:07 ema: cache_misc rolling reboots: upgrading kernel to 4.9.51, libssl to 1.0.2m and 1.1.0g
10:59 ema: reboot cp3007: upgrading kernel to 4.9.51, libssl to 1.0.2m and 1.1.0g
10:59 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db2092 as recentchanges multi-instance host on s1 and s3 T178359 (duration: 00m 45s)
10:58 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2092 as recentchanges multi-instance host on s1 and s3 T178359 (duration: 00m 45s)
10:41 moritzm: restarting ntpd on dns recursors to pick up openssl update
10:28 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2091 as recentchanges multi-instance host on s2 and s4 T178359 (duration: 00m 45s)
10:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db2091 as recentchanges multi-instance host on s2 and s4 T178359 (duration: 00m 46s)
10:24 elukey: create staging database on db1108 (researchers scratch pad) - T177405
10:19 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2086 as recentchanges multi-instance host on s5 and s7 T178359 (duration: 00m 45s)
10:09 gehel: reboot of maps codfw cluster for upgrades
09:59 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db2086 as recentchanges multi-instance host on s5 and s7 T178359 (duration: 00m 45s)
09:58 paravoid: updating certspotter to 0.5 in apt and tegmen/einsteinium
09:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db2089 as recentchanges multi-instance host on s6 and s5(s8) T178359 (duration: 00m 46s)
09:39 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2089 as recentchanges multi-instance host on s6 and s5(s8) T178359 (duration: 00m 45s)
09:13 marostegui: Stop MySQL on db1038 - host to be decommissioned - T177911
09:11 moritzm: installing java security updates/restarting cassandra on restbase2002
09:02 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1038 from config - T177911 (duration: 00m 45s)
09:02 ppchelko@tin: Finished deploy [cpjobqueue/deploy@3db0cc4]: Do not set Host header for requests to jobrunner (duration: 00m 49s)
09:02 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1038 from config - T177911 (duration: 00m 47s)
09:01 ppchelko@tin: Started deploy [cpjobqueue/deploy@3db0cc4]: Do not set Host header for requests to jobrunner
08:49 marostegui: Run redact_sanitarium for hifwiktionary on db1095 (sanitarium) - T173647
08:48 marostegui: Optimize pagelinks and templatelinks on s7 master - db1062 - T174509
07:18 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: Electron stuck, restarting - T174916
07:11 ema: reboot pinkunicorn for kernel (4.9.51) and openssl (1.0.2m) upgrades
06:19 marostegui: Deploy alter table on s7 codfw master (db2029) with replication, this will cause lag in codfw - T174569
02:29 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Nov 7 02:29:54 UTC 2017 (duration 6m 39s)
02:23 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.6) (duration: 07m 19s)
22:34 bawolff: Deploy patch for T178451
21:48 arlolra: Updated Parsoid to 6cb77104 T179579 , T175792 )
21:42 arlolra@tin: Finished deploy [parsoid/deploy@d1df3c3]: (no justification provided) (duration: 08m 46s)
21:35 awight: Begin stress testing ores* (non-production)
21:33 arlolra@tin: Started deploy [parsoid/deploy@d1df3c3]: (no justification provided)
21:20 XioNoX: re-activating BGP session to telia in ulsfo
21:16 ladsgroup@tin: Finished deploy [ores/deploy@97a1d80]: Deploying early November (T179837 ) (duration: 10m 47s)
21:05 ladsgroup@tin: Started deploy [ores/deploy@97a1d80]: Deploying early November (T179837 )
20:56 reedy@tin: Synchronized wmf-config/interwiki.php: Update IW Map again (duration: 00m 46s)
20:50 marostegui: Deploy alter table on s2 codfw master (db2017) with replication, this will generate lag in codfw - T174569
20:36 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: hifwiktionary T173643 (duration: 00m 45s)
20:34 reedy@tin: rebuilt wikiversions.php and synchronized wikiversions files: hifwiktionary T173643
20:33 reedy@tin: Synchronized dblists/: hifwiktionary T173643 (duration: 00m 47s)
20:04 reedy@tin: Synchronized wmf-config/interwiki.php: Update IW map (duration: 00m 45s)
19:57 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: electcomwiki T174370 (duration: 00m 45s)
19:56 reedy@tin: rebuilt wikiversions.php and synchronized wikiversions files: electcomwiki T174370
19:55 reedy@tin: Synchronized dblists/: electcomwiki T174370 (duration: 00m 44s)
19:52 demon@tin: Pruned MediaWiki: 1.31.0-wmf.2 (duration: 02m 49s)
19:47 demon@tin: Pruned MediaWiki: 1.31.0-wmf.5 [keeping static files] (duration: 01m 38s)
19:22 Amir1: ladsgroup@terbium:/srv/mediawiki-staging/php-1.31.0-wmf.4$ mwscript extensions/ORES/maintenance/CheckModelVersions.php --wiki=enwiki (T179596 )
19:21 kaldari@tin: Synchronized wmf-config/InitialiseSettings.php: Updating InitialiseSettings for draftquality data (duration: 00m 48s)
18:15 XioNoX: deactivating BGP session to telia in ulsfo
18:09 gehel@tin: Finished deploy [wdqs/wdqs@fa8bb12]: WDQS GUI and Blazegraph update (duration: 01m 53s)
18:07 gehel@tin: Started deploy [wdqs/wdqs@fa8bb12]: WDQS GUI and Blazegraph update
17:50 awight@tin: Finished deploy [ores/deploy@29905e5]: restart services on ores* (non-production) (duration: 17m 18s)
17:48 urandom: T179083 : Restarting Cassandra instances on restbase2001.codfw.wmnet
17:32 awight@tin: Started deploy [ores/deploy@29905e5]: restart services on ores* (non-production)
16:51 marostegui: Optimize pagelinks and templatelinks on s2 master - db1054 - T174509
16:26 bblack: Switching unified TLS certificate in the US
16:25 awight@tin: Finished deploy [ores/deploy@82a13ae]: Redeploy ORES master to scb* (duration: 16m 10s)
16:09 awight@tin: Started deploy [ores/deploy@82a13ae]: Redeploy ORES master to scb*
15:50 akosiaris@tin: Started deploy [ores/deploy@29905e5]: testing deploy
15:49 akosiaris@tin: Started deploy [ores/deploy@29905e5]: testing deploy
15:43 awight@tin: Finished deploy [ores/deploy@29905e5]: restart services on ores* (non-production) (duration: 12m 45s)
15:31 awight@tin: Started deploy [ores/deploy@29905e5]: restart services on ores* (non-production)
15:26 gehel: reboot of maps-test cluster for jvm and kernel upgrades
15:13 ppchelko@tin: Finished deploy [cpjobqueue/deploy@96f55c6]: USe correct regex for job selection (duration: 00m 29s)
15:12 ppchelko@tin: Started deploy [cpjobqueue/deploy@96f55c6]: USe correct regex for job selection
15:03 addshore@tin: Synchronized wmf-config/CommonSettings-labs.php: SWAT Set wgWikiDiff2MovedParagraphDetectionCutoff for group0 PT 3/3 (duration: 00m 46s)
15:01 addshore@tin: Synchronized wmf-config/CommonSettings.php: SWAT Set wgWikiDiff2MovedParagraphDetectionCutoff for group0 PT 2/3 (duration: 00m 46s)
15:00 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT Set wgWikiDiff2MovedParagraphDetectionCutoff for group0 PT 1/3 (duration: 00m 47s)
14:57 gehel: reboot of relforge for kernel + jvm upgrade
14:50 mobrovac@tin: Synchronized wmf-config/jobqueue.php: Switch MessageIndexRebuildJob, flaggedrevs_CacheUpdate and deleteLinks jobs to the EventBus infrastructure - T175210 (duration: 00m 46s)
14:50 ppchelko@tin: Finished deploy [cpjobqueue/deploy@e93feba]: Start processing all 'hearted' jobs T175210 (duration: 00m 44s)
14:49 ppchelko@tin: Started deploy [cpjobqueue/deploy@e93feba]: Start processing all 'hearted' jobs T175210
14:48 ppchelko@tin: Started deploy [cpjobqueue/deploy@e93feba]: (no justification provided)
14:45 awight@tin: Finished deploy [ores/deploy@29905e5]: Force deployment on ores* (non-production) (duration: 05m 43s)
14:45 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs2003.codfw.wmnet
14:45 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs2002.codfw.wmnet
14:40 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs2002.codfw.wmnet
14:40 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs2001.codfw.wmnet
14:39 awight@tin: Started deploy [ores/deploy@29905e5]: Force deployment on ores* (non-production)
14:35 awight@tin: Finished deploy [ores/deploy@29905e5]: restart services on ores* (non-production) (duration: 04m 48s)
14:34 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs2001.codfw.wmnet
14:33 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs1005.eqiad.wmnet
14:30 awight@tin: Started deploy [ores/deploy@29905e5]: restart services on ores* (non-production)
14:27 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs1005.eqiad.wmnet
14:27 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs1004.eqiad.wmnet
14:21 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs1004.eqiad.wmnet
14:21 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs1003.eqiad.wmnet
14:20 awight@tin: Finished deploy [ores/deploy@29905e5]: celery 4 -> ores* (non-production) (duration: 19m 22s)
14:18 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: EventBus: Enable logging - T150106 (duration: 00m 47s)
14:14 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs1003.eqiad.wmnet
14:14 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=wdqs1003.eqiad.wmnet
14:13 mobrovac@tin: Synchronized php-1.31.0-wmf.6/extensions/EventBus/EventBus.php: EventBus - Improve logging for T150106 (duration: 00m 48s)
14:01 gehel: full restart of wdqs eqiad / codfw for multiple upgrades
14:01 awight@tin: Started deploy [ores/deploy@29905e5]: celery 4 -> ores* (non-production)
13:03 elukey: rolling restart of druid historical daemons on druid100[1-6] to apply https://gerrit.wikimedia.org/r/#/c/389429
12:58 moritzm: installing openssl security updates
12:55 moritzm: restarting apache on netmon to pick up openssl update
12:43 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: Electron stuck, restrting - T174916
12:29 moritzm: installing imagemagick security updates
11:45 oblivian@tin: Started deploy [jobrunner/jobrunner@a20d043]: (no justification provided)
11:44 marostegui: Deploy alter table on s6 codfw master (with replication, so this will generate lag on codfw) - T174569
11:19 marostegui: Compress InnoDB on db1103.s2 - T178359
11:16 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2087 as recentchanges multi-instance host on s6 and s7 T178359 (duration: 00m 47s)
11:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db2087 as recentchanges multi-instance host on s6 and s7 T178359 (duration: 00m 48s)
10:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1101 original weight after maintenance - T178359 (duration: 00m 46s)
10:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give db1101 more weight after maintenance - T178359 (duration: 00m 46s)
10:05 moritzm: uploading linux 4.9.51-1~bpo8+1 for jessie-wikimedia to apt.wikimedia.org
09:57 godog: roll-upgrade swift to 2.10.2 in codfw and roll-restart for kernel upgrade - T177739
09:56 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db2088 as recentchanges multi-instance host on s1 and s2 T178359 (duration: 00m 46s)
09:55 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2088 as recentchanges multi-instance host on s1 and s2 T178359 (duration: 00m 46s)
09:37 _joe_: manually running htmlCacheUpdate for commonswiki and ruwiki on terbium, T173710
09:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db2085 as recentchanges multi-instance host on s3 and s5 T178553 T178359 (duration: 00m 46s)
09:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2085 as multi-instance core host on eqiad file T178553 T178359 (duration: 00m 46s)
09:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give db1101 more weight after maintenance - T178359 (duration: 00m 46s)
09:07 moritzm: removed monitoring/RAID packages from stretch-wikimedia/thirdparty, now in thirdparty/hwraid
08:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1101 with low weight after maintenance - T178359 (duration: 00m 47s)
06:46 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2084 as multi-instance core host on eqiad file T178553 T178359 (duration: 00m 46s)
06:38 marostegui: Stop MySQL on db1101 to copy its content to db1103.s4 - T178359
06:31 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1101 - T178359 (duration: 00m 48s)
06:19 marostegui: Optimize pagelinks and templatelinks on s6 master (db1061) - T174509
06:16 marostegui: Deploy alter table on s4 codfw master (with no replication) - T174569
02:44 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Nov 6 02:44:41 UTC 2017 (duration 6m 55s)
02:37 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.6) (duration: 07m 17s)
13:34 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=scb1002.eqiad.wmnet
13:19 elukey@puppetmaster1001: conftool action : set/pooled=no; selector: name=scb1002.eqiad.wmnet
12:36 mobrovac: trending-edits depool it from scb1002 to investigate
00:18 Krinkle: Finished whisper-mass-resize for frontend.navtiming on graphite2001 (T179622 )
20:28 marostegui: Deploy alter table db2058 - T174569
18:52 Krinkle: Starting whisper-mass-resize for frontend.navtiming on graphite2001 (T179622 )
18:51 Krinkle: whisper-mass-resize completed for graphite1001.eqiad.wmnet:/var/lib/carbon/whisper/frontend/navtiming (at 01:34 UTC actually)
17:56 demon@tin: Synchronized php-1.31.0-wmf.6/includes/libs/rdbms/database/DatabaseMysqlBase.php: logging (duration: 00m 47s)
16:17 marostegui: Deploy alter table on db2044 - T174569
16:03 demon@tin: Synchronized php-1.31.0-wmf.6/extensions/ContentTranslation/api/ApiQueryTranslatorStats.php: fix warnings about bad min() calls (duration: 00m 47s)
15:50 cmjohnson1: powering off labvirt1015 to swap CPU (again)
15:49 akosiaris: T177393 enable RBAC for kubernetes in production and staging
15:35 volans: uploaded cumin_1.3.0-1_amd64.deb to apt.wikimedia.org jessie-wikimedia
14:19 moritzm: installing java sec updates / cassandra restarts on xenon/praseodymium (cerium already done earlier)
14:16 hashar: restarting Jenkins on contint1001
14:12 hashar: Upgrading packages on contint1001 / contint2001
13:50 moritzm: installing heimdal security updates on trusty
13:35 ppchelko@tin: Started restart [electron-render/deploy@8dd5f13]: Restarting, service is stuck
13:33 Pchelolo: restart electron render service
12:41 moritzm: uploaded openjdk 8u151-b12 to apt.wikimedia.org/jessie-wikimedia
11:45 moritzm: restarting jenkins on releases* to pick up openjdk security update
11:41 mutante: gerrit - restart Apache - apply gerrit:354078 - only listen on service IP
11:12 moritzm: uploaded openssl 1.0.2m-1 for to apt.wikimedia.org
10:46 marostegui: Compress InnoDB on db1103.s4 - T178359
10:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add comment to db1103 status (duration: 00m 47s)
10:42 mutante: cobalt: (gerrit) temp disable puppet - gerrit2001: restart apache, apply gerrit:354078 stop listening on all interfaces
10:39 oblivian@tin: Synchronized wmf-config/CommonSettings.php: Increase concurrency of htmlCacheUpdate jobs T173710 (duration: 00m 48s)
09:09 mobrovac@tin: Finished deploy [restbase/deploy@a67b4a7]: Use only the new storage for summaries - T179418 (duration: 10m 40s)
08:58 mobrovac@tin: Started deploy [restbase/deploy@a67b4a7]: Use only the new storage for summaries - T179418
08:46 ema: pool cp4021
08:16 elukey: drop CommandInvocation_15243810 and CommandInvocation_15243810_15423246 T166712
08:12 moritzm: installing openjdk-8 security updates on stretch
07:57 marostegui: Deploy alter table on s4.db2037 - T174569
07:29 elukey: depooled mw1191 and powercycle - host down with 'CPU 1 check errors' in racadm getsel
06:39 marostegui: Stop MySQL on db2089.s5 to copy it to db2085 - T178359
06:25 marostegui: Stop MySQL on db1103 to copy it to dbstore1001 and reimage it - T178359
23:36 krinkle@tin: Synchronized php-1.31.0-wmf.6/extensions/NavigationTiming/modules/ext.navigationTiming.js: Fix zero values - T178479 (duration: 00m 47s)
23:10 herron: disabled puppet master debug logging on puppetmaster2001 via /usr/share/puppet/rack/puppet-master/config.ru
22:48 volans: cleaned daemon.log from puppet spam logs on puppetmaster2001 (15GB) - cc herron
22:39 halfak@tin: Started deploy [ores/deploy@82a13ae]: T179621
22:02 Krinkle: Mass-resizing Graphite/Whisper files on graphite1001 and graphite2001 for T179622 (frontend.* namespace)
21:53 ayounsi@tin: Finished deploy [librenms/librenms@ad48b4d]: (no justification provided) (duration: 00m 05s)
21:52 ayounsi@tin: Started deploy [librenms/librenms@ad48b4d]: (no justification provided)
21:50 XioNoX: upgrading librenms
21:38 hoo: Updated the Wikidata property suggester with data from Monday's JSON dump and applied the T132839 workarounds
20:56 madhuvishy: Powercycling labvirt1015
20:18 chasemp: simulate load for labvirt1015 'sudo cumin "name:labvirt1015stresstest*" "stress-ng --cpu 1 --io 2 --vm 1 --vm-bytes 1G &"'
19:45 mepps: updated payments-wiki from b88b805 a539d27
19:28 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.6
18:20 gehel: restarting blazegraph on wdqs1003
18:13 gehel: restarting stuck updater on wdqs1003
17:55 demon@tin: Synchronized php-1.31.0-wmf.6/includes/Message.php: logging fixes (duration: 00m 48s)
17:53 demon@tin: Synchronized php-1.31.0-wmf.6/includes/libs/rdbms: logging improvements (duration: 00m 54s)
17:47 moritzm: uploaded openssl 1.1.0g-1+wmf1 to jessie-wikimedia
17:42 awight@tin: Finished deploy [ores/deploy@0e54a3c]: revscoring 2, T175180 (duration: 33m 50s)
17:08 awight@tin: Started deploy [ores/deploy@0e54a3c]: revscoring 2, T175180
16:14 ema: restart varnish-be on cp4026 due to mbox lag
16:04 mobrovac@tin: Synchronized wmf-config/jobqueue.php: Use only EventBus for processing updateBetaFeatureUserCount - T175210 (duration: 00m 51s)
13:46 chasemp: ugrade dnsmasq-base on labcontrol*
13:36 zeljkof: EU SWAT finished
13:35 zfilipin@tin: Synchronized php-1.31.0-wmf.6/extensions/ContentTranslation/modules/publish/ext.cx.publish.js: SWAT: CX1: Check for template adaptation failures before publishing (T154116) (duration: 00m 50s)
13:11 zfilipin@tin: Synchronized wmf-config/ProductionServices.php: SWAT: use the logstash LVS endpoint (T175242) (duration: 00m 51s)
12:09 moritzm: removed obsolete packages from jessie-wikimedia/experimental
12:06 mobrovac@tin: Finished deploy [restbase/deploy@9314cf6]: Parsoid: Switch all but WPs to use the next-generation storage - T179417 (duration: 08m 43s)
11:57 mobrovac@tin: Started deploy [restbase/deploy@9314cf6]: Parsoid: Switch all but WPs to use the next-generation storage - T179417
11:43 elukey: drop table log.PageContentSaveComplete_5588433 from db1046,db1047,db1108,dbstore1002, archived on hdfs - T177101
11:31 moritzm: remove obsolete packages from stretch-wikimedia/experimental, now empty
11:29 mobrovac@tin: Finished deploy [restbase/deploy@f6c4e2d]: Parsoid module: use the Cassandra 2 tables as fallback when needed - T179417 (duration: 08m 15s)
11:26 elukey: drop log.MediaViewer_10867062_15423246 T168303
11:21 mobrovac@tin: Started deploy [restbase/deploy@f6c4e2d]: Parsoid module: use the Cassandra 2 tables as fallback when needed - T179417
10:54 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=logstash1003.eqiad.wmnet
10:53 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=logstash1002.eqiad.wmnet
10:53 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=logstash1001.eqiad.wmnet
10:53 gehel: depooling logstash100[123] in preparation for decommission - T175830
10:38 gehel: rolling restart of wdqs nodes for GC tuning - T175919
10:13 marostegui: Deploy alter table on db2019 - T174569
09:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1091 original weight - T161088 (duration: 00m 50s)
09:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1091 weight after maintenance - T161088 (duration: 00m 50s)
08:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1091 weight after maintenance - T161088 (duration: 00m 50s)
08:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1091 with low weight after maintenance - T161088 (duration: 00m 50s)
08:15 moritzm: restarting app server canaries to pick up curl security update
08:04 moritzm: installing curl security updates
07:59 marostegui: Stop mysql event scheduler on labsdb1001 - T179464
06:54 marostegui: Stop s3 instance on db2092 to copy it to db2085 - T178359
06:33 legoktm: done on mwdebug1002
06:31 marostegui: Stop MySQL on db1091 and db1103 to migrate db1091 to file per table
06:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1091 - T161088 (duration: 01m 07s)
06:27 marostegui: Deploy optimize table wikidatawiki.pagelinks on s5 master (db1063) - T174509
06:02 legoktm: live-hacking mwdebug1002
05:43 TimStarling: on puppetmaster2001: disk full due to logs, so I'm truncating /var/log/debug which is apparently copied to daemon.log anyway
03:29 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Nov 2 03:29:35 UTC 2017 (duration 7m 12s)
03:22 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.6) (duration: 15m 37s)
02:43 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.5) (duration: 16m 06s)
00:52 ebernhardson@tin: Synchronized php-1.31.0-wmf.5/extensions/WikimediaEvents/extension.json: SWAT followup: Update SearchSatisfaction eventlogging schema revision id (duration: 00m 50s)
00:51 ebernhardson@tin: Synchronized php-1.31.0-wmf.6/extensions/WikimediaEvents/extension.json: (no justification provided) (duration: 00m 50s)
00:32 bblack: cp4022 - varnish backend restart, mailbox
00:28 bblack: lvs1001-3: re-enable ethernet flowcontrol (short ethernet blip), with pybal stopped to failover to backups during...
23:37 thcipriani@tin: Synchronized php-1.31.0-wmf.5/extensions/WikimediaEvents: SWAT: Turn on Cirrus AB test for DBN group sizing (duration: 00m 50s)
23:35 thcipriani@tin: Synchronized php-1.31.0-wmf.6/extensions/WikimediaEvents: SWAT: Turn on Cirrus AB test for DBN group sizing (duration: 00m 51s)
23:29 thcipriani@tin: Synchronized php-1.31.0-wmf.6/extensions/ParserMigration/includes/ApiParserMigration.php: SWAT: API: Fix WikiPage namespace (duration: 00m 52s)
23:25 thcipriani@tin: Synchronized php-1.31.0-wmf.6/extensions/Wikibase/repo/Wikibase.hooks.php: SWAT: Allow turning Cirrus usage off from query T179428 (duration: 00m 49s)
23:23 thcipriani@tin: Synchronized php-1.31.0-wmf.6/extensions/Wikidata/extensions/Wikibase/repo/Wikibase.hooks.php: SWAT: Allow turning Cirrus usage off from query T179428 (duration: 00m 51s)
23:09 thcipriani@tin: Synchronized wmf-config/Wikibase-production.php: SWAT: Revert "Revert "Add negative weight to disambig entities"" T148411 (duration: 00m 51s)
22:49 ejegg: disabled fundraising unsubscribe import processing
22:33 Krinkle: group2(all-wikidata) wikis to wmf.5 from 24 hours ago seems to have caused a 60% drop in navigation timing metric report count (100/min => 40/min)
22:19 eileen1: process-control updated to 9c8b4bb
22:17 demon@tin: Synchronized wmf-config/CommonSettings.php: rel1_28 is dead (duration: 00m 50s)
22:06 ejegg: re-enabled fundraising jobs
21:48 ejegg: fundraising jobs disabled
20:10 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.6
20:05 demon@tin: Synchronized php: symlink swap (duration: 00m 49s)
19:57 krinkle@tin: Synchronized php-1.31.0-wmf.6/resources/src/startup.js: T178943 (duration: 00m 51s)
19:53 demon@tin: Synchronized php-1.31.0-wmf.5/extensions/CentralNotice: fix weird git rebasing issue (duration: 00m 53s)
19:41 demon@tin: Pruned MediaWiki: 1.31.0-wmf.4 [keeping static files] (duration: 01m 52s)
19:31 awight@tin: Synchronized php-1.31.0-wmf.5/extensions/ORES: Fix for API=oresscores, T179430 (duration: 00m 50s)
19:28 awight@tin: Synchronized php-1.31.0-wmf.6/extensions/ORES: Fix for API=oresscores, T179430 (duration: 00m 52s)
19:14 bblack: all hosts: manual cumin+sed removal of ethernet autoneg params from /e/n/i to match https://gerrit.wikimedia.org/r/#/c/387849/
19:08 otto@tin: Finished deploy [analytics/refinery@6d11d67]: Deploying refinery-source artifacts for 0.0.54 for JsonRefine job, T162610 (duration: 03m 45s)
19:05 otto@tin: Started deploy [analytics/refinery@6d11d67]: Deploying refinery-source artifacts for 0.0.54 for JsonRefine job, T162610
18:39 demon@tin: Synchronized wmf-config/: Dropping squid.php (hang on to your pants folks, this could be fun) (duration: 00m 51s)
18:36 demon@tin: Synchronized wmf-config/CommonSettings.php: use reverse-proxy.php no more squid.php (duration: 00m 50s)
18:35 demon@tin: Synchronized docroot/: dropping squid.php (duration: 00m 52s)
18:13 demon@tin: Synchronized static/images/project-logos/sewikimedia.png: new logo (duration: 00m 50s)
18:02 moritzm: installing openjpeg2 security updates
17:39 demon@tin: Synchronized php-1.31.0-wmf.6/includes/specials/pagers/ContribsPager.php: fix missing page_is_new error (duration: 00m 51s)
17:10 demon@tin: Synchronized wmf-config/: dropped squid-labs, no-op in prod (duration: 00m 52s)
17:09 demon@tin: Synchronized docroot/noc/: dropped squid-labs.php (duration: 00m 51s)
16:01 bblack: lvs1003 - puppet disabled, testing experimental ethtool ringbuffer change
15:15 moritzm: repo reorg: moved ftpsync from thirdparty to main and docker-engine from thirdparty to thirdparty/k8s
15:12 awight@tin: Finished deploy [ores/deploy@9f361d2]: revscoring 2 -> ores1002 (non-production) (duration: 02m 25s)
15:10 awight@tin: Started deploy [ores/deploy@9f361d2]: revscoring 2 -> ores1002 (non-production)
15:09 awight@tin: Finished deploy [ores/deploy@9f361d2]: revscoring 2 -> ores* (non-production) (duration: 01m 57s)
15:07 awight@tin: Started deploy [ores/deploy@9f361d2]: revscoring 2 -> ores* (non-production)
15:07 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: Update CirrusSearch MLR model on enwiki (duration: 00m 51s)
15:01 mobrovac: restbase: removing wikimedia-lvs-realserver from staging hosts T179494
14:55 bblack: strongswan experiment done, cp* back to puppet-agent-enabled
14:41 moritzm: installing poppler security updates
14:09 bblack: cp*: disabling puppet to test strongswan change...
13:49 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable NewUserMessage on fawikiquote - T179442 (duration: 00m 50s)
13:46 hashar@tin: Synchronized wmf-config/abusefilter.php: Enable blocking feature of abuse filter in fawikiquote - T178227 (duration: 00m 50s)
13:45 moritzm: installing quagga security updates
13:43 hashar@tin: Synchronized wmf-config/CommonSettings.php: Properly check for cluster existence prior setting TTM mirrors - T179270 (duration: 01m 05s)
13:41 ppchelko@tin: Finished deploy [restbase/deploy@2321c4c]: Update hyperswitch dependency. Take 2 (duration: 05m 50s)
13:35 ppchelko@tin: Started deploy [restbase/deploy@2321c4c]: Update hyperswitch dependency. Take 2
13:33 ppchelko@tin: Finished deploy [restbase/deploy@2321c4c]: Update hyperswitch dependency (duration: 03m 50s)
13:29 ppchelko@tin: Started deploy [restbase/deploy@2321c4c]: Update hyperswitch dependency
12:48 moritzm: installing libav security updates
12:44 moritzm: installinng libdatetime-timezone-perl stable updates on Debian
12:10 kartik@tin: Finished deploy [cxserver/deploy@10651e2]: Update cxserver to 0227acb
12:07 kartik@tin: Started deploy [cxserver/deploy@10651e2]: Update cxserver to 0227acb
08:35 elukey: forced umount/mount for /mnt/hdfs on stat1005 (not working after repeated oom kill actions)
03:22 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Nov 1 03:22:02 UTC 2017 (duration 7m 17s)
03:14 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.6) (duration: 15m 21s)
02:36 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.5) (duration: 09m 46s)
23:56 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: Adjust cirrussearch interleave configuration for AB test (duration: 00m 50s)
23:48 demon@tin: Synchronized scap/plugins/prep.py: no-op (duration: 00m 50s)
23:21 ebernhardson@tin: Synchronized wmf-config/CirrusSearch-common.php: Setup CirrusSearch AB test on dbn group sizing (duration: 00m 50s)
23:19 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: Setup CirrusSearch AB test on dbn group sizing (duration: 00m 53s)
21:38 awight@tin: Started deploy [ores/deploy@9f361d2]: revscoring 2 -> ores* (non-production)
21:37 awight@tin: Started deploy [ores/deploy@9f361d2]: revscoring 2 -> ores* (non-production)
21:16 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/386553 https://gerrit.wikimedia.org/r/386710 (duration: 00m 52s)
21:00 XioNoX: removing old AMS-IX IPv6 - T167840
20:57 mepps: rollback payments-wiki from 5303a8e b88b805
20:54 mepps: updated payments-wiki from b88b805 5303a8e
20:45 mepps: rolledback payments-wiki from 5303a8e b88b805
20:43 mepps: updated payments-wiki from b88b805 5303a8e
19:18 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.6
18:53 awight@tin: Finished deploy [ores/deploy@437fcf5]: revscoring 2 -> ores* (non-production) (duration: 10m 29s)
18:42 awight@tin: Started deploy [ores/deploy@437fcf5]: revscoring 2 -> ores* (non-production)
18:38 awight@tin: Finished deploy [ores/deploy@437fcf5]: revscoring 2 -> ores1002 (non-production) (duration: 02m 24s)
18:36 awight@tin: Started deploy [ores/deploy@437fcf5]: revscoring 2 -> ores1002 (non-production)
18:32 demon@tin: Synchronized php-1.31.0-wmf.6/extensions/CentralNotice: deployyyyy (duration: 00m 52s)
18:27 ejegg: updated CiviCRM from a34a9c6 415fde1
17:23 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: wikidata back to wmf.5
16:50 demon@tin: Finished scap: bootstrap wmf.6 (duration: 47m 39s)
16:41 ppchelko@tin: Finished deploy [restbase/deploy@7631ea7]: Enable new parsoid storage on test wikipedias (duration: 09m 07s)
16:32 ppchelko@tin: Started deploy [restbase/deploy@7631ea7]: Enable new parsoid storage on test wikipedias
16:09 ariel@tin: Finished deploy [dumps/dumps@fa0583b]: adapt dumpadmin script to use override section in config (duration: 00m 02s)
16:09 ariel@tin: Started deploy [dumps/dumps@fa0583b]: adapt dumpadmin script to use override section in config
16:02 demon@tin: Started scap: bootstrap wmf.6
15:43 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: all but wikidata.org -> wmf.5
15:42 demon@tin: Synchronized php: Symlink bump (duration: 00m 48s)
15:41 demon@tin: Pruned MediaWiki: 1.31.0-wmf.1 (duration: 03m 15s)
14:18 bblack: authdns (baham, radon, eeden): upgrade base packages (curl, git, libc-dev, etc)
14:13 bblack: lvsNNNN: upgrade base packages (curl, git, libc-dev, etc)
14:11 bblack: cpNNNN: upgrade base packages (curl, git, libc-dev, etc)
13:57 bblack: caches@eqiad - upgrade nginx to 1.13.6-2+wmf1~jessie1
13:54 bblack: caches@esams - upgrade nginx to 1.13.6-2+wmf1~jessie1
13:22 bblack: caches@codfw - upgrade nginx to 1.13.6-2+wmf1~jessie1
13:18 zeljkof: EU SWAT finished
13:17 zfilipin@tin: Synchronized php-1.31.0-wmf.4/extensions/EventBus/EventBus.php: SWAT: Dont log full request to service in case of delivery error (T179280) (duration: 00m 51s)
13:14 bblack: caches@ulsfo - upgrade nginx to 1.13.6-2+wmf1~jessie1
13:05 mobrovac@tin: Finished deploy [restbase/deploy@c0f9dc4]: Cassandra 3 Parsoid stash tables (duration: 02m 58s)
13:02 mobrovac@tin: Started deploy [restbase/deploy@c0f9dc4]: Cassandra 3 Parsoid stash tables
13:01 mobrovac@tin: Finished deploy [restbase/deploy@c0f9dc4]: Cassandra 3 Parsoid stash tables (duration: 15m 19s)
12:45 mobrovac@tin: Started deploy [restbase/deploy@c0f9dc4]: Cassandra 3 Parsoid stash tables
12:39 bblack: eqiad primary lvses (1001-3): disable pause [LRO not possible] on all ethernet interfaces (under pybal stopped briefly)
12:36 bblack: codfw primary lvses (2003-6): disable LRO,pause on all ethernet interfaces (under pybal stopped briefly)
12:24 bblack: cp4025: restart varnish-be for mailbox lag
12:23 bblack: cp4023: restart varnish-be for mailbox lag
12:21 bblack: esams primary lvses (3001-2): disable LRO,pause on eth0 (under pybal stopped briefly)
12:19 bblack: ulsfo primary lvses (4001-2): disable LRO,pause on eth0 (under pybal stopped briefly)
12:12 bblack: esams+ulsfo backup lvses (3003-4,4003-4): disable LRO,pause on eth0
12:10 bblack: core DC backup lvses (1004-6,2004-6): disable LRO,pause on all ethernet interfaces
11:05 oblivian@tin: Finished deploy [docker-pkg/deploy@20908e8]: Actually deploy the fix (duration: 00m 04s)
11:05 oblivian@tin: Started deploy [docker-pkg/deploy@20908e8]: Actually deploy the fix
11:02 oblivian@tin: Finished deploy [docker-pkg/deploy@576c80f]: Fix image_tag filter (duration: 00m 02s)
11:02 oblivian@tin: Started deploy [docker-pkg/deploy@576c80f]: Fix image_tag filter
10:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1084 original weight - T161088 (duration: 00m 48s)
10:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1084 weight - T161088 (duration: 00m 50s)
09:12 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2084 as multi-instance core host T178553 T178359 (duration: 00m 49s)
08:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1084 weight - T161088 (duration: 00m 50s)
08:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1084 with low weight - T161088 (duration: 00m 50s)
08:15 marostegui: Stop MySQL on db2086 to copy s5 to db2089 - T178359
07:09 marostegui: Stop MySQL on db2087.s6 to copy its content to db2089 - T178359
06:37 marostegui: Stop MySQL on db1103 and db1084 - T161088
06:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1084 - T161088 (duration: 00m 49s)
06:17 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2039 (duration: 01m 12s)
03:08 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Oct 31 03:08:48 UTC 2017 (duration 7m 18s)
03:01 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.5) (duration: 10m 05s)
02:34 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.4) (duration: 10m 33s)
23:28 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: Update cirrussearch MLR model for enwiki (duration: 00m 50s)
23:22 ebernhardson@tin: Synchronized php-1.31.0-wmf.4/extensions/CirrusSearch/: Fix highlight tags inside section anchors of search results (duration: 01m 01s)
23:19 ebernhardson@tin: Synchronized php-1.31.0-wmf.5/extensions/CirrusSearch/: Fix highlight tags inside section anchors of search results (duration: 01m 03s)
22:03 eileen1: update process-control to revision is 2392ddf
21:55 awight@tin: Finished deploy [ores/deploy@a0f7d5c]: Fun with deployments (non-production) T179336 (duration: 02m 03s)
21:53 awight@tin: Started deploy [ores/deploy@a0f7d5c]: Fun with deployments (non-production) T179336
21:48 awight@tin: Finished deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production) (duration: 01m 42s)
21:47 chasemp: restart nova-fullstack to force a run
21:46 awight@tin: Started deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production)
21:36 eileen1: update CiviCRM from 25c7a33 a34a9c6
21:02 urandom: T179083 : Creating commons_T_parsoid_stash_wikitextf0PBY8UXqY8UuiDv.{meta,data} (30 sec delay)
20:56 urandom: T179083 : Creating commons_T_parsoid_stash_htmlmXxc_uDhgnFQAdM8PPlH.{data,meta} (25 sec delay)
20:46 urandom: T179083 : Creating enwiki_T_parsoid_stash_section2ACMDK1DRzacK9nUB3.{data,meta} (15sec delay)
20:36 urandom: T179083 : Creating enwiki_T_parsoid_stash_dataWH8IDUS9SGI6LPpsJsLOQ.data
20:35 arlolra: Updated Parsoid to 292633c4 T133334 )
20:34 urandom: T179083 : Creating enwiki_T_parsoid_stash_dataWH8IDUS9SGI6LPpsJsLOQ.meta
20:33 urandom: T179083 : Creating enwiki_T_parsoid_stash_dataWH8IDUS9SGI6LPpsJsLOQ
20:28 urandom: T179083 : Creating table enwiki_T_parsoid_stash_wikitextf0PBY8UXqY8UuiDv1.data
20:27 urandom: T179083 : Creating table enwiki_T_parsoid_stash_wikitextf0PBY8UXqY8UuiDv1.meta
20:26 arlolra@tin: Finished deploy [parsoid/deploy@8a2c0ce]: Updating Parsoid to 292633c4
20:25 urandom: T179083 : Creating keyspace enwiki_T_parsoid_stash_wikitextf0PBY8UXqY8UuiDv1
20:20 urandom: T179083 : Creating table enwiki_T_parsoid_stash_htmlmXxc_uDhgnFQAdM8PPlH5.data
20:18 arlolra@tin: Started deploy [parsoid/deploy@8a2c0ce]: Updating Parsoid to 292633c4
20:14 urandom: T179083 : Creating table enwiki_T_parsoid_stash_htmlmXxc_uDhgnFQAdM8PPlH5.meta
20:09 awight@tin: Finished deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production) (duration: 14m 11s)
20:08 urandom: T179083 : Creating keyspace enwiki_T_parsoid_stash_htmlmXxc_uDhgnFQAdM8PPlH5
19:55 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create Appendix NS on Burmese Wiktionary T178545 (duration: 00m 55s)
19:55 awight@tin: Started deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production)
19:07 awight@tin: Finished deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production) (duration: 06m 16s)
19:01 awight@tin: Started deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production)
19:01 awight@tin: Finished deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production) (duration: 19m 53s)
18:41 awight@tin: Started deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production)
18:40 awight@tin: Started deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production)
18:22 awight@tin: Started deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production)
18:16 thcipriani@tin: Synchronized portals: SWAT: Bumping portals to master T128546 (duration: 00m 50s)
18:15 thcipriani@tin: Synchronized portals/prod/wikipedia.org/assets: SWAT: Bumping portals to master T128546 (duration: 00m 50s)
18:08 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add Timeless skin to wikitech T154371 (duration: 00m 51s)
17:08 madhuvishy: Revert dns switchover for c1 shards to c3 post labsdb1001 reboot T168584
17:07 gehel@tin: Finished deploy [wdqs/wdqs@fdb9100]: GUI update (duration: 02m 17s)
17:05 gehel@tin: Started deploy [wdqs/wdqs@fdb9100]: GUI update
16:45 oblivian@tin: Finished deploy [docker-pkg/deploy@576c80f]: New version with small fixes (duration: 00m 04s)
16:44 oblivian@tin: Started deploy [docker-pkg/deploy@576c80f]: New version with small fixes
15:20 XioNoX: re-enabling Zayo transit in eqiad
14:36 marostegui: Reboot labsdb1001 - T168584
14:30 marostegui: Stop replication and MySQL on labsdb1001 - T168584
14:09 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Revert "UBN! disbale ores for wikidata" (T179107 ) (duration: 00m 50s)
14:07 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Revert "UBN! disbale ores for wikidata" (T179107 ) (duration: 00m 49s)
14:00 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Change the af.wiktionary logo, part II (T178824 ) (duration: 00m 50s)
13:59 ladsgroup@tin: Synchronized static/images/project-logos: Change the af.wiktionary logo, part I (T178824 ) (duration: 00m 50s)
13:40 madhuvishy: Switched dns over for c1 shards to c3 in preparation of labsdb1001 reboot (Merged https://gerrit.wikimedia.org/r/#/c/386660/ , Ran puppet on labservice1001|2)
13:21 marostegui: Set innodb_max_dirty_pages_pct = 10 on labsdb1001 so it powers off a bit faster - T168584
13:19 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable the Autopatrolled User Rights & Ext:SandboxLink on hiwikiversity (T179251 T179252) (duration: 00m 50s)
12:58 reedy@tin: Synchronized wmf-config/db-labs.php: beta labs stuffs (duration: 00m 50s)
12:49 oblivian@tin: Finished deploy [docker-pkg/deploy@e9b9146]: Proper deploy of first version (duration: 00m 07s)
12:49 oblivian@tin: Started deploy [docker-pkg/deploy@e9b9146]: Proper deploy of first version
12:02 hoo: Manually started dumpwikidatajson on snapshot1007 (cron run failed tonight)
12:02 oblivian@tin: Finished deploy [docker-pkg/deploy@549c157]: (no justification provided) (duration: 00m 11s)
12:01 oblivian@tin: Started deploy [docker-pkg/deploy@549c157]: (no justification provided)
12:00 oblivian@tin: Finished deploy [docker-pkg/deploy@549c157]: (no justification provided) (duration: 00m 19s)
11:59 oblivian@tin: Started deploy [docker-pkg/deploy@549c157]: (no justification provided)
11:57 oblivian@tin: Finished deploy [docker-pkg/deploy@549c157]: (no justification provided) (duration: 00m 11s)
11:57 oblivian@tin: Started deploy [docker-pkg/deploy@549c157]: (no justification provided)
11:55 oblivian@tin: Started deploy [docker-pkg/deploy@549c157]: Test deploy for virtualenv creation
11:48 hoo: Ran rebuildPropertyInfo for wikidatawiki and manually cleared the property cache (mc)
11:44 hoo@tin: Synchronized wmf-config/Wikibase.php: Re-add property for RDF mapping of external identifiers for Wikidata (T179156 , T178180 ) (duration: 00m 49s)
11:33 hoo@tin: Synchronized wmf-config/Wikibase-production.php: Re-enable constraints check with SPARQL (T179156 ) (duration: 00m 50s)
11:24 marostegui: Compress table cebwiki.externallinks on db2092 - T178359
11:15 hoo@tin: rebuilt wikiversions.php and synchronized wikiversions files: Wikidatawiki back to wmf.5 (T179156 )
10:58 mobrovac@tin: Finished deploy [restbase/deploy@2b5889b]: Double-process all summaries and include the parsoid no-op switch, take #gazillion (duration: 08m 33s)
10:52 akosiaris: poweroff copper T176957
10:50 mobrovac@tin: Started deploy [restbase/deploy@2b5889b]: Double-process all summaries and include the parsoid no-op switch, take #gazillion
10:31 marostegui: Stop MySQL on db2039 to copy its data to db2087.s6 - T178359
10:27 mobrovac@tin: Finished deploy [restbase/deploy@2b5889b]: Double-process all summaries and include the parsoid no-op switch (duration: 04m 41s)
10:22 mobrovac@tin: Started deploy [restbase/deploy@2b5889b]: Double-process all summaries and include the parsoid no-op switch
10:03 mobrovac@tin: Finished deploy [restbase/deploy@2b5889b]: Double-process all summaries and include the parsoid no-op switch (duration: 12m 12s)
09:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1097 original weight - T161088 (duration: 00m 52s)
09:50 mobrovac@tin: Started deploy [restbase/deploy@2b5889b]: Double-process all summaries and include the parsoid no-op switch
09:22 marostegui: Stop replication in sync on db2039 and db2046 to reimport tables
09:08 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1097 api traffic - T161088 (duration: 00m 50s)
09:08 marostegui: Drop wb_entity_per_page page from s3 and s5 - T177601
08:51 ema: cp4022: restart varnish-be for mbox lag
08:42 elukey: raised priority of refreshlink and htmlcacheupdate job execution on jobrunners (https://gerrit.wikimedia.org/r/#/c/386636/ ) - T173710
08:39 marostegui: Stop replication on db2039 to reimport and compress tables
08:38 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1097 with low weight - T161088 (duration: 00m 50s)
08:33 moritzm: installing wget security updates
08:32 gehel: rolling restart of wdqs for config reload - T175919
08:31 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2039 (duration: 00m 50s)
08:28 ariel@tin: Finished deploy [dumps/dumps@c204c72]: fix args initialization issue in getconfigvals script (duration: 00m 02s)
08:28 ariel@tin: Started deploy [dumps/dumps@c204c72]: fix args initialization issue in getconfigvals script
08:23 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2040 - T178359 (duration: 00m 50s)
08:11 marostegui: Stop MySQL on db2086 to transfer s7 to db2087 - T178359
06:51 marostegui: Stop MySQL on db1097 to clone db1103 - T161088
06:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1097 - T161088 (duration: 00m 50s)
06:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Move db1103 from s3 to s4 - T161088 (duration: 00m 49s)
06:40 marostegui: Stop MySQL on db1103 to reclone it - T161088
06:24 marostegui: Optimize dewiki.pagelinks and dewiki.templatelinks on db1063 (s5 master) - T174509
06:12 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2040 - T178359 (duration: 00m 51s)
06:08 marostegui: Stop MySQL on db2040 to populate s7 on db2086 - T178359
05:58 marostegui: Deploy alter table on db1075 (s3 primary master) - T174509
03:11 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Oct 30 03:11:37 UTC 2017 (duration 7m 11s)
03:04 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.5) (duration: 11m 38s)
02:35 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.4) (duration: 11m 18s)
23:49 ema: powercycle cp4024
22:31 ariel@tin: Finished deploy [dumps/dumps@2aa2275]: fix keep setting to work with overrides (duration: 00m 02s)
22:31 ariel@tin: Started deploy [dumps/dumps@2aa2275]: fix keep setting to work with overrides
17:55 ariel@tin: Finished deploy [dumps/dumps@d8978ce]: add overrides section processing to config file (duration: 00m 04s)
17:55 ariel@tin: Started deploy [dumps/dumps@d8978ce]: add overrides section processing to config file
17:23 ariel@tin: Finished deploy [dumps/dumps@d426cf7]: batch 7z jobs, multistream job fixup (duration: 00m 02s)
17:23 ariel@tin: Started deploy [dumps/dumps@d426cf7]: batch 7z jobs, multistream job fixup
12:54 ema: cp4026: restart varnish-be for mbox lag
21:03 bblack: cp1067 (current target cache): disabling the relatively-new VCL that sets do_stream=false if !CL on applayer fetches...
19:39 hoo@tin: Synchronized wmf-config/CommonSettings.php: Half the Flow -> Parsoid timeout (100s -> 50s) (T179156 ) (duration: 00m 51s)
19:39 bblack: backend restart on cp1065
18:39 bblack: restarting varnish backend on cp1053 to move the lag/503 issues to another box and buy more time to debug
18:28 bblack: cp4025 - restart backend for mailbox lag (upload@ulsfo, unrelated to text-cluster issues)
18:21 bblack: cp1053 - manual VCL change, backends appservers+api_appservers, reduce connect/firstbyte/betweenbytes timeoues from 5/180/60 to 3/20/10
16:51 elukey: restart varnish backend on cp1055 - mailbox lag + T179156
12:14 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1313.eqiad.wmnet
12:10 elukey: manually killed (SIGTERM) hhvm on mw1313 - high load, hhvm-dump-debug not responsive
12:01 elukey@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw1313.eqiad.wmnet
11:53 elukey: restart hhvm on mw1285 - hhvm-dump-debug in /tmp/hhvm.17700.bt
11:24 hoo@tin: Synchronized wmf-config/Wikibase-labs.php: Consistency sync (duration: 00m 50s)
10:52 volans: restarted pdfrender on scb1001, was stuck since 2d with AssertionError: display is not set!
20:54 MaxSem: running migratePreferences.php on group2 wikis
19:09 hoo: Ran scap pull on mwdebug1001
18:20 awight@tin: Finished deploy [ores/deploy@185170f]: Test pip-9 scap trick on ores1002 (non-production) (duration: 02m 17s)
18:18 awight@tin: Started deploy [ores/deploy@185170f]: Test pip-9 scap trick on ores1002 (non-production)
17:54 hoo: Taking mwdebug1001 to do tests regarding T179156
16:38 gehel: re-enabling wdqs-updater
16:16 bblack: cp1054 varnish backend restarted (was 503s / bad-conns target of ongoing issues)
16:16 gehel: wdqs updater is now stopped for real
16:10 XioNoX: deactivating BGP sessions to Zayo in eqiad (flapping)
15:58 gehel: disabling wdqs updater on all nodes
15:50 hoo@tin: Synchronized wmf-config/Wikibase-production.php: Disable constraints check with SPARQL for now (T179156 ) (duration: 00m 50s)
15:48 marostegui: Compress InnoDB on db2039 (s6) - T178359
15:46 bblack: restart varnish-backend on cp4022 (upload@ulsfo) - mailbox
14:49 bblack: turn on cp4024 port on asw-ulsfo
13:52 bblack: reboot cp4021 to clean up oom messes
13:49 bblack: restarting nginx on cp4021, without NUMA memory constraints
12:10 marostegui: Optimize commonswiki.templatelinks on dbstore1001 - T162789
12:03 elukey: execute systemctl reset-failed kafka-mirror-main-eqiad_to_jumbo-eqiad.service on kafka-jumbo hosts (old unit not deployed anymore)
11:41 mutante: gerrit back - maintenance over
11:39 mutante: gerrit restart to apply gerrit:386793 is imminent
11:36 ema: cp4023: varnish-backend-restart for lag
11:06 mobrovac@tin: Finished deploy [citoid/deploy@ff63420]: Update dependencies (duration: 03m 18s)
11:04 marostegui: Optimize cebwiki.templatelinks on db1103 - T174509
11:03 mobrovac@tin: Started deploy [citoid/deploy@ff63420]: Update dependencies
08:56 marostegui: Optimize table commonswiki.recentchanges on dbstore1001 - T162789
08:11 marostegui: Drop redundant indexes from pagelinks and templatelinks on s3 wikis only for dbstore2001 and dbstore2002 - T174509
05:59 marostegui: Stop MySQL on db2084 to copy s5 to db2086 - T178359
05:55 marostegui: dbstore1001: convert itwiki.pagelinks to TokuDB - T162789
05:53 mutante: uploaded parsoid_0.8.0 to releases.wikimedia.org (for Subbu - T179134 )
05:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1060 - T174509 (duration: 00m 50s)
03:02 bblack: cp1067, cp4026 - backend restarts, mailbox lag
22:47 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: wikidata to wmf.4
22:39 bblack: restarting varnish-backend on cp1053 (mailbox lag from ongoing issues elsewhere?)
21:40 bblack: raising backend max_connections for api.svc.eqiad.wmnet + appservers.svc.eqiad.wmnet from 1K to 10K on cp1053.eqiad.wmnet (current funnel for the bulk of the 503s)
21:32 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Temporary disable remex html (T178632 ) (duration: 00m 50s)
21:30 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Temporary disable remex html (T178632 ) (duration: 00m 50s)
21:00 hoo: Fully revert all changes related to T178180
20:59 hoo@tin: Synchronized wmf-config/Wikibase.php: Revert "Add property for RDF mapping of external identifiers for Wikidata" (T178180 ) (duration: 00m 50s)
20:02 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: UBN! disbale ores for wikidata (T179107 ) (duration: 00m 50s)
20:00 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: UBN! disbale ores for wikidata (T179107 ) (duration: 00m 50s)
19:41 awight@tin: Finished deploy [ores/deploy@0adae70]: Increase extractor wikidata API timeout to 15s, T179107 (duration: 07m 25s)
19:36 aaron@tin: Started restart [jobrunner/jobrunner@a20d043]: (no justification provided)
19:34 awight@tin: Started deploy [ores/deploy@0adae70]: Increase extractor wikidata API timeout to 15s, T179107
19:33 awight@tin: Finished deploy [ores/deploy@0adae70]: Increase extractor wikidata API timeout to 15s, T179107 (duration: 00m 10s)
19:33 awight@tin: Started deploy [ores/deploy@0adae70]: Increase extractor wikidata API timeout to 15s, T179107
19:27 demon@tin: Synchronized php-1.31.0-wmf.5/includes/libs/filebackend/SwiftFileBackend.php: Better error grouping (duration: 00m 50s)
19:14 ladsgroup@tin: Synchronized php-1.31.0-wmf.5/extensions/WikibaseQualityConstraints/includes/ConstraintCheck/DelegatingConstraintChecker.php: Fix sorting of NullResults (T179038 ) (duration: 00m 49s)
19:12 ladsgroup@tin: Synchronized php-1.31.0-wmf.5/extensions/WikibaseQualityConstraints/tests/phpunit/DelegatingConstraintCheckerTest.php: Fix sorting of NullResults (T179038 ) (duration: 00m 50s)
18:52 ladsgroup@tin: Synchronized php-1.31.0-wmf.5/extensions/Wikidata/extensions/Constraints/tests/phpunit/DelegatingConstraintCheckerTest.php: Fix sorting of NullResults (T179038 ) (duration: 00m 49s)
18:51 ladsgroup@tin: Synchronized php-1.31.0-wmf.5/extensions/Wikidata/extensions/Constraints/includes/ConstraintCheck/DelegatingConstraintChecker.php: Fix sorting of NullResults (T179038 ) (duration: 01m 04s)
18:41 bblack: eqiad cp servers: rolling quick depool -> repool around ethtool parameter changes for -lro,-pause
18:30 urandom: Dropping and recreating keyspace enwiki_T_page__summary (restbase-ng cluster)
18:27 bblack: esams cp servers: rolling quick depool -> repool around ethtool parameter changes for -lro,-pause
18:26 urandom: T179105 : Altering "enwiki_T_page__summary".data to use LZ4Compressor
17:59 bblack: codfw cp servers: rolling quick depool -> repool around ethtool parameter changes for -lro,-pause
17:57 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4024.ulsfo.wmnet
17:49 bblack: ulsfo cp servers: rolling quick depool -> repool around ethtool parameter changes for -lro,-pause
17:47 awight@tin: Finished deploy [ores/deploy@f5deb7f]: Revscoring 2 on ores1002 (non-production) (duration: 01m 45s)
17:45 awight@tin: Started deploy [ores/deploy@f5deb7f]: Revscoring 2 on ores1002 (non-production)
17:36 arlolra: Updated Parsoid to 8e99708a T176728 )
17:30 arlolra@tin: Finished deploy [parsoid/deploy@4882c59]: Updating Parsoid to 8e99708a
17:21 arlolra@tin: Started deploy [parsoid/deploy@4882c59]: Updating Parsoid to 8e99708a
17:17 awight@tin: Finished deploy [ores/deploy@971be22]: Rolling back scb1002, T175180 T178441 (duration: 00m 32s)
17:16 awight@tin: Started deploy [ores/deploy@971be22]: Rolling back scb1002, T175180 T178441
17:14 awight@tin: Finished deploy [ores/deploy@971be22]: ORES w/ revscoring 2 and Celery 4, T175180 T178441 (duration: 03m 20s)
17:11 awight@tin: Started deploy [ores/deploy@971be22]: ORES w/ revscoring 2 and Celery 4, T175180 T178441
17:09 marostegui: Optimize ruwiki.recentchanges on dbstore1001 - T162789
17:08 dcausse: [logstash] reimporting today logs
17:08 dcausse: [logstash] deleting today logs to reapply mapping template
16:58 ottomata: now mirroring main Kafka cluster topics to jumbo Kafka cluster, with MirrorMaker instances running on analytics-eqiad broker nodes. https://phabricator.wikimedia.org/T177216
16:44 awight@tin: Finished deploy [ores/deploy@971be22]: Push ORES w/ Celery 4 support to new cluster (try to rebuild venv), T178441 (duration: 01m 02s)
16:43 awight@tin: Started deploy [ores/deploy@971be22]: Push ORES w/ Celery 4 support to new cluster (try to rebuild venv), T178441
16:33 awight@tin: Finished deploy [ores/deploy@971be22]: Push ORES w/ Celery 4 support to new cluster (take 3), T178441 (duration: 15m 33s)
16:21 urandom: restarted cassandra, restbase2001-b
16:18 awight@tin: Started deploy [ores/deploy@971be22]: Push ORES w/ Celery 4 support to new cluster (take 3), T178441
16:15 urandom: truncating cassandra hints in restbase-ng cluster
16:12 mobrovac@tin: Finished deploy [restbase/deploy@860cbfe]: Revert all summaries, back to all but WP (duration: 07m 57s)
16:04 mobrovac@tin: Started deploy [restbase/deploy@860cbfe]: Revert all summaries, back to all but WP
16:04 mobrovac@tin: Started deploy [restbase/deploy@860cbfe]: Revert all summaries, back to all but WP
15:50 mobrovac@tin: Finished deploy [restbase/deploy@522321a]: Double-process all summaries (duration: 09m 09s)
15:45 urandom: restarting cassandra, restbase2001-b
15:41 andrewbogott: upgrading dnsmasq on labnet1001 and labnet1002
15:41 mobrovac@tin: Started deploy [restbase/deploy@522321a]: Double-process all summaries
15:36 dcausse: reindexing logs in logstash
15:34 dcausse: deleting and reimporting logs in logstash (today data, might lose some logs in the process)
14:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1101 - T178383 (duration: 00m 49s)
14:04 zeljkof: EU SWAT finished
14:03 zfilipin@tin: Synchronized php-1.31.0-wmf.5/extensions/Wikibase: SWAT: Make search for titles be always uppercase (T179045) (duration: 01m 36s)
13:52 zfilipin@tin: Synchronized php-1.31.0-wmf.5/extensions/Wikidata: SWAT: Make search for titles be always uppercase (T179045) (duration: 02m 10s)
13:46 Amir1: ladsgroup@terbium:~$ mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildPropertyInfo.php --wiki=wikidatawiki --rebuild-all --force (T178180 )
13:41 godog: delete old CF cassandra metrics - T173436
13:32 godog: force delete old graphite cassandra metrics - T179057
13:28 moritzm: installing icu security update on trusty
13:28 zfilipin@tin: Synchronized wmf-config/Wikibase.php: SWAT: Add property for RDF mapping of external identifiers for Wikidata (T178180) (duration: 00m 50s)
13:21 marostegui: Compress InnoDB on db2040 - T178359
13:21 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable signature button in Projekt namespace on sewikimedia (T175363) (duration: 00m 50s)
13:13 zfilipin@tin: Synchronized wmf-config/InterwikiSortOrders.php: SWAT: Fix interwiki sort order for Northern Sami (T178965) (duration: 00m 50s)
12:57 marostegui@tin: Synchronized wmf-config/db-codfw.php: Reorganize future rc multi-instance hosts - T178359 (duration: 00m 50s)
12:30 bblack: powercycle cp4024 (depooled)
12:29 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4024.ulsfo.wmnet
10:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Clean up some comments in s3 (duration: 00m 50s)
09:01 moritzm: upgrade script runners to wikidiff2 1.5.1
08:58 ema: cache_text/upload varnish-be: set rutime parameter timeout_idle to 120s
08:34 moritzm: uploaded php-wikidiff2/hhvm-wikidiff2 1.5.1 for stretch-wikimedia
08:22 godog: add 100G to graphite1003's /var/lib/carbon
08:14 moritzm: upgrade deployment servers to wikidiff2 1.5.1
07:57 marostegui: Drop databases in s1 and s2 from db1047 and unconfigure replication - T177405
07:43 marostegui: Stop MySQL on db1047 to copy data over db1108 - T177405 T156844
07:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1077 - T164488 (duration: 00m 50s)
07:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1077 - T164488 (duration: 00m 50s)
07:12 marostegui: Stop replication in sync on db1103 and db1077 to fix data drifts - T164488
07:10 moritzm: upgrade mw1319-mw1328 to wikidiff2 1.5.1
06:57 marostegui: Stop MySQL on db2088 and db2084 to copy s2 and s4 to db2091 - T178359
06:57 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2038 (duration: 00m 50s)
06:55 marostegui: Optimize pagelinks and templatelinks on db1095 s1 - T174509
06:51 marostegui: Optimize recentchanges on s4 and s6 on labsdb1010 - T177772
06:15 moritzm: upgrade mw1299-mw1311 to wikidiff2 1.5.1
05:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1060 - T174509 (duration: 00m 50s)
05:14 marostegui: Optimize recentchanges on s4 and s6 on labsdb1009 - T177772
05:12 marostegui: Optimize pagelinks and templatelinks on db1060 - T174509
03:01 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Oct 26 03:01:19 UTC 2017 (duration 6m 48s)
02:54 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.5) (duration: 09m 30s)
02:31 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.4) (duration: 10m 34s)
23:48 legoktm@tin: Synchronized php-1.31.0-wmf.5/extensions/VisualEditor/modules/ve-mw/ui/styles/contextitems/ve.ui.MWInternalLinkContextItem.css: MWInternalLinkContextItem: increase specificity to override OOUI changes - T178933 (duration: 00m 50s)
23:39 legoktm@tin: Synchronized wmf-config/abusefilter.php: Change threshold for slow AbuseFilter logging to 800ms - T179039 (duration: 00m 50s)
22:42 MaxSem: running migratePreferences.php on group1 wikis
22:01 mobrovac@tin: Finished deploy [restbase/deploy@860cbfe]: Revert double-processing of all summaries to all except WPs, take #2 (duration: 09m 09s)
21:52 mobrovac@tin: Started deploy [restbase/deploy@860cbfe]: Revert double-processing of all summaries to all except WPs, take #2
21:48 mobrovac@tin: Started deploy [restbase/deploy@60ce036]: Revert double-processing of all summaries to all except WPs
21:33 bblack: experimental - disabled LRO on lvs4001+lvs4002
20:34 hasharAway: restarted zuul to flush a long queue in experimental pipeline (no other changes enqueued)
20:21 demon@tin: Synchronized php-1.31.0-wmf.5/extensions/ProofreadPage/: unbreak multiline text input (duration: 00m 53s)
19:28 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.5
18:45 bblack: s/1\.13\.16/1.13.6/
18:45 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable fine grained usage tracking on statement usage wikis T172914 (duration: 00m 50s)
18:44 bblack: reprepro: uploaded nginx-1.13.16-2+wmf1 packages to stretch and jessie
18:36 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Make disabled usage aspect use the new config T172914 (duration: 00m 50s)
18:33 niharika29@tin: Synchronized php-1.31.0-wmf.4/extensions/Wikidata: Allow specifying a replacement usage type in disabledUsageAspects T178153 (duration: 02m 10s)
18:25 niharika29@tin: Synchronized wmf-config/Wikibase.php: Make using CirrusSearch engine default for wbsearchentities T175741 (duration: 00m 48s)
18:23 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Add AbuseFilterSlow channel to monolog T178853 (duration: 00m 50s)
18:20 niharika29@tin: Synchronized php-1.31.0-wmf.5/extensions/ORES/: Update ApiHooks to conform with v3 responses T178962 (duration: 00m 50s)
18:19 niharika29@tin: Synchronized php-1.31.0-wmf.4/extensions/ORES/: Update ApiHooks to conform with v3 responses T178962 (duration: 00m 51s)
18:11 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable Special:EmailUser User Prohibit on All Wikis [mediawiki-config] - https://gerrit.wikimedia.org/r/386231 (duration: 00m 51s)
17:17 moritzm: upgrade mw2224-mw2242 to wikidiff2 1.5.1
16:59 moritzm: upgrade mw2190-mw2199 to wikidiff2 1.5.1
16:38 no_justification: gerrit: purging last remnants of debian package and running puppet. service may briefly restart
16:05 moritzm: upgrade mw2180-mw2189 to wikidiff2 1.5.1
16:03 moritzm: copied arcconf, hp-health, hpacucli, hpssa, hpssacli, hpssaducli, lsiutil, megacli, megaclisas-status to stretch-wikimedia/thirdparty/hwraid (thirdparty will be dropped at a later point)
15:39 marostegui: Optimize pagelinks and templatelinks on labsdb1011 - T174509
15:31 marostegui: Power off db1101 for HW maintenance - T178383
15:21 godog: bounce rsyslog on wezen
15:09 mattflaschen@tin: Synchronized wmf-config/InitialiseSettings.php: T177836 : Deploy Compact Language Links on the German Wikipedia (duration: 00m 49s)
14:52 gehel: applying new template to elasticsearch / logstash - T178530
14:51 ebernhardson: update logstash template on logstash elsaticsearch cluster for T178530
14:47 mattflaschen@tin: Synchronized wmf-config/InitialiseSettings.php: T166759 : Set $wgAutoloadAttemptLowercase to false (duration: 00m 50s)
14:37 mattflaschen@tin: Synchronized php-1.31.0-wmf.5/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T177956 : [cirrus] Turn off recall A/B test on enwiki (duration: 00m 49s)
14:36 mattflaschen@tin: Synchronized php-1.31.0-wmf.4/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T177956 : [cirrus] Turn off recall A/B test on enwiki (duration: 00m 49s)
14:25 moritzm: moved jmxtrans and prometheus-jmx-exporter from thirdparty to main as part of repo reorg for stretch-wikimedia
14:11 matt_flaschen: 'Deployed T178933 : MWInternalLinkContextItem: increase specificity to override OOUI changes'
14:10 mattflaschen@tin: Synchronized php-1.31.0-wmf.4/extensions/VisualEditor/modules/ve-mw/ui/styles/contextitems/ve.ui.MWInternalLinkContextItem.css: (no justification provided) (duration: 00m 50s)
13:36 mattflaschen@tin: Synchronized wmf-config/InitialiseSettings.php: T168886 : Updating wikis with consolidate editing feedback (duration: 00m 51s)
13:30 godog: deploy thumbor 1.7 - T178974
13:30 elukey: restart yarn nodemanager and hdfs datanode on analytics1030 to apply new JVM settings - T178876
13:15 chasemp: merge arturo's key as part of ops
13:04 matt_flaschen: About to run ULSCompactLinksDisablePref.php: Setting user preferences for Compact Language Links on dewiki, before removing Beta Feature and changing to regular preference. T177836
12:36 moritzm: upgrade mw2163-mw2179 to wikidiff2 1.5.1
12:17 mobrovac@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Activate warning logging for JobExecutor (duration: 00m 50s)
12:16 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: Activate warning logging for JobExecutor (duration: 00m 50s)
12:12 moritzm: upgrade mw1266-mw1275 to wikidiff2 1.5.1
11:30 moritzm: upgrade mw1280-mw1290, mw1312-mw1318 to wikidiff2 1.5.1
11:23 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2038 - T178359 (duration: 00m 49s)
10:49 mobrovac@tin: Finished scap: wmf-config/jobqueue-labs.php beta-only change for CP4JQ (duration: 04m 43s)
10:45 mobrovac@tin: Started scap: wmf-config/jobqueue-labs.php beta-only change for CP4JQ
10:41 mobrovac@tin: Finished deploy [restbase/deploy@2ae5b0c]: Remove /title/{title}/ and double-process all summaries - T158100 (duration: 09m 41s)
10:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1077 - T164488 (duration: 00m 50s)
10:32 moritzm: upgrade remaining video scalers in eqiad to wikidiff2 1.5.1
10:31 mobrovac@tin: Started deploy [restbase/deploy@2ae5b0c]: Remove /title/{title}/ and double-process all summaries - T158100
09:31 moritzm: upgrade mw1221-mw1235 to wikidiff2 (API servers)
09:07 gehel: rolling restart of all wdqs nodes for GC config change - T175919
08:12 moritzm: upgrade mw1238-mw1258 to wikidiff2
08:02 _joe_: decommissioning mw1168/69 as videoscalers, stopped jobrunner/jobchron, will stop hhvm/nginx/apache once current processing is done
07:42 gehel@tin: Finished deploy [wdqs/wdqs@0bb2b5c]: wdqs-updater upgrade for jolokia - T167842 (duration: 01m 44s)
07:40 gehel@tin: Started deploy [wdqs/wdqs@0bb2b5c]: wdqs-updater upgrade for jolokia - T167842
07:05 marostegui: Optimize pagelinks and templatelinks on db1021 - T174509
06:15 marostegui: Stop MySQL on db2038 to copy its data to db2084 - T178359
06:08 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2035 - T178359 (duration: 00m 50s)
06:02 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1065 - T174509 (duration: 00m 50s)
05:44 marostegui: Stop replication in sync on db1103 and db1077 to checksum data - T164488
05:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1077 - T164488 (duration: 01m 00s)
03:22 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Oct 25 03:22:51 UTC 2017 (duration 7m 20s)
03:15 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.5) (duration: 16m 46s)
02:35 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.4) (duration: 08m 59s)
22:54 ejegg: updated civicrm from from 0546446 25c7a33
21:24 ejegg: updated civicrm from 18edce1 0546446
21:21 mepps: updated payments-wiki from 1011434 b88b805
20:47 arlolra@tin: Finished deploy [parsoid/deploy@UNKNOWN]: (no justification provided) (duration: 00m 13s)
20:47 arlolra@tin: Started deploy [parsoid/deploy@UNKNOWN]: (no justification provided)
20:29 mutante: restarting gerrit to make gerrit-ssh listen on all IPs again
19:49 arlolra@tin: Finished deploy [parsoid/deploy@UNKNOWN]: Enabling logging for skwiki posts (duration: 00m 35s)
19:48 arlolra@tin: Started deploy [parsoid/deploy@UNKNOWN]: Enabling logging for skwiki posts
19:46 mutante: restarting gerrit on cobalt to apply change 354074
19:13 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.5
18:57 awight@tin: Finished deploy [ores/deploy@fb55ab8]: Rollback ORES to Revscoring 1.0, T175180 (duration: 10m 13s)
18:47 awight@tin: Started deploy [ores/deploy@fb55ab8]: Rollback ORES to Revscoring 1.0, T175180
18:47 ejegg: updated civicrm from b9d2067 18edce1
18:26 awight@tin: Finished deploy [ores/deploy@fb55ab8]: Update ORES to revscoring 2.x, T175180 (take 2) (duration: 10m 03s)
18:16 awight@tin: Started deploy [ores/deploy@fb55ab8]: Update ORES to revscoring 2.x, T175180 (take 2)
18:05 arlolra: Updated Parsoid to a3be9cfc T96923 , T177784 , T176568 , T25467 )
17:57 arlolra@tin: Finished deploy [parsoid/deploy@af55c4b]: Updating Parsoid to a3be9cfc
17:48 arlolra@tin: Started deploy [parsoid/deploy@af55c4b]: Updating Parsoid to a3be9cfc
17:28 demon@tin: Finished scap: bootstrap wmf.5 (duration: 48m 34s)
17:18 urandom: T178845 : Rolling Cassandra restart, eqiad
17:07 awight@tin: Started deploy [ores/deploy@fb55ab8]: Update ORES to revscoring 2.x, T175180
16:50 herron: upgrading puppet packages on codfw puppetmaster T177254
16:39 demon@tin: Started scap: bootstrap wmf.5
16:10 demon@tin: Pruned MediaWiki: 1.31.0-wmf.3 [keeping static files] (duration: 01m 20s)
15:58 elukey: drop MediaViewer_10867062_15423246 15423246 T168303
15:53 urandom: T178845 : Rolling Cassandra restart, codfw, row b
15:48 elukey: drop table Edit_13457736_15423246
15:39 elukey: set net.netfilter.nf_conntrack_tcp_timeout_time_wait=65 to mw[1308-1311] - T136094
15:09 gehel: restarting blazegraph on all wdqs nodes for GC config change - T175919
14:48 _joe_: also stopping HHVM, nginx, apache2 , and disabling all the services
14:46 _joe_: stopping the jobrunner service on mw1161-67
14:39 marostegui: Optimize pagelinks templatelinks and recentchanges on db1030 - T174509 https://phabricator.wikimedia.org/T177772
14:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1065 - T174509 (duration: 00m 45s)
14:33 marostegui: Optimize ores_classification on enwiki db1065 - T159753
14:32 marostegui: Optimize pagelinks and templatelinks on db1065 - T174509
14:24 gehel: restarting blazegraph on wdqs2001 for GC config change - T175919
14:20 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1106 weight (duration: 00m 45s)
13:42 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2035 - T178359 (duration: 00m 45s)
13:41 marostegui: Stop MySQL on db2035 to copy its data to db2088 - T178359
13:39 zeljkof: EU SWAT finished
13:39 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add $wgNamespaceRobotPolicies Config for hewikisource (T178775) (duration: 00m 46s)
10:50 dcausse: cirrus/elasticsearch: reindexing of 167 small wikis done (T177871 )
09:22 marostegui: Stop s1 on db2092 to copy its data to db2088 - T178359
08:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1082 original weight - T178460 (duration: 00m 45s)
07:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1082 weight - T178460 (duration: 00m 45s)
07:46 marostegui: Stop replication in sync on db1103 and db2018 to fix data drifts - T164488
07:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1078 - T164488 (duration: 00m 46s)
07:28 marostegui: Stop replication in sync on db1078 and db1103 to fix data drifts - https://phabricator.wikimedia.org/T164488
06:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1078 - T164488 (duration: 00m 45s)
06:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1082 with low weight - T178460 (duration: 00m 47s)
02:32 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Oct 24 02:32:25 UTC 2017 (duration 6m 33s)
02:25 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.4) (duration: 08m 13s)
23:09 demon@tin: Synchronized wmf-config/throttle.php: Working Class Movement Library (Salford) throttle rule (duration: 00m 46s)
22:59 bblack: cp4024: varnish-backend-restart for lag
21:40 XioNoX: increasing MTU on Zayo transit interfaces
20:34 arlolra: Updated Parsoid to 1cefef12 T178217 )
20:25 arlolra@tin: Finished deploy [parsoid/deploy@0759bd1]: Updating Parsoid to 1cefef12
20:14 arlolra@tin: Started deploy [parsoid/deploy@0759bd1]: Updating Parsoid to 1cefef12
19:37 bsitzmann@tin: Finished deploy [mobileapps/deploy@a654ce1]: Update mobileapps to 3628105 T178828 ) (duration: 06m 59s)
19:30 bsitzmann@tin: Started deploy [mobileapps/deploy@a654ce1]: Update mobileapps to 3628105 T178828 )
19:00 joal@tin: Finished deploy [analytics/aqs/deploy@ad22e0c]: Upgrade AQS node modules to restbase-latests - Redeploy after config fix (duration: 02m 34s)
18:58 joal@tin: Started deploy [analytics/aqs/deploy@ad22e0c]: Upgrade AQS node modules to restbase-latests - Redeploy after config fix
18:58 joal@tin: Finished deploy [analytics/aqs/deploy@ad22e0c]: Upgrade AQS node modules to restbase-latests - Redeploy after config fix (duration: 05m 07s)
18:52 joal@tin: Started deploy [analytics/aqs/deploy@ad22e0c]: Upgrade AQS node modules to restbase-latests - Redeploy after config fix
18:52 joal@tin: Finished deploy [analytics/aqs/deploy@ad22e0c]: Upgrade AQS node modules to restbase-latests (duration: 321m 51s)
17:56 gehel@tin: Finished deploy [wdqs/wdqs@b84592f]: WDQS GUI update (duration: 01m 58s)
17:54 gehel@tin: Started deploy [wdqs/wdqs@b84592f]: WDQS GUI update
17:48 XenoRyet: updated smashpig from d056a13 5262d53
17:32 demon@tin: Synchronized scap/plugins/prep.py: Fixes and improvements of various sorts (duration: 00m 45s)
17:26 gehel@tin: Finished deploy [wdqs/wdqs@4626acb]: deploying latest WDQS version, blazegraph + GUI updates (duration: 02m 07s)
17:24 gehel@tin: Started deploy [wdqs/wdqs@4626acb]: deploying latest WDQS version, blazegraph + GUI updates
16:53 demon@tin: Pruned MediaWiki: 1.30.0-wmf.19 (duration: 03m 09s)
15:59 elukey: forced BBU learn cycle on db1046 - T166141
15:00 herron: wiped resolver cache of records puppet.cofdw.wmnet and puppet.ulsfo.wmnet (T177254 )
14:46 herron: depooling codfw puppetmaster (via dns)
14:31 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1078 - T164488 (duration: 00m 46s)
14:05 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2037 - T178359 (duration: 00m 46s)
13:48 marostegui: Stop mysql and poweroff db1082 for HW maintenance - T178460
13:30 joal@tin: Started deploy [analytics/aqs/deploy@ad22e0c]: Upgrade AQS node modules to restbase-latests
13:19 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Add aleph500.biblacad.ro to $wgCopyUploadsDomains - T178753 (duration: 00m 47s)
13:19 gehel: rolling restart of wdqs for GC tuning - T175919
13:18 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Remove hooserv.net from wgCopyUploadsDomains (duration: 00m 47s)
13:08 zeljkof: EU SWAT finished, no deploys
13:00 bblack@neodymium: conftool action : set/pooled=no; selector: dc=ulsfo,cluster=cache_text,name=cp4018.ulsfo.wmnet
12:59 bblack@neodymium: conftool action : set/pooled=yes; selector: dc=ulsfo,cluster=cache_text,name=cp4032.ulsfo.wmnet
12:53 bblack@neodymium: conftool action : set/pooled=no; selector: dc=ulsfo,cluster=cache_text,name=cp4017.ulsfo.wmnet
12:47 bblack@neodymium: conftool action : set/pooled=yes; selector: dc=ulsfo,cluster=cache_text,name=cp4031.ulsfo.wmnet
12:46 moritzm: upgrading hhvm-wikdiff2 on remaining job runners in codfw
12:39 moritzm: upgrading hhvm-wikdiff2 on remaining video scalers in codfw
12:32 bblack@neodymium: conftool action : set/pooled=no; selector: dc=ulsfo,cluster=cache_text,name=cp4010.ulsfo.wmnet
12:29 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Remove hooserv.net from wgCopyUploadsDomains (duration: 00m 46s)
12:26 moritzm: upgrading hhvm-wikdiff2 on mw2153-mw2162 (job runners)
12:21 moritzm: upgrading hhvm-wikdiff2 on mw2254-mw2258 (app servers)
12:20 bblack@neodymium: conftool action : set/pooled=yes; selector: dc=ulsfo,cluster=cache_text,name=cp4030.ulsfo.wmnet
12:15 bblack@neodymium: conftool action : set/pooled=no; selector: dc=ulsfo,cluster=cache_text,name=cp4009.ulsfo.wmnet
12:12 bblack@neodymium: conftool action : set/pooled=yes; selector: dc=ulsfo,cluster=cache_text,name=cp4029.ulsfo.wmnet
12:03 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable description usage tracking on further wikis (T178515 ) (duration: 00m 47s)
11:11 moritzm: upgrading hhvm-wikdiff2 on mw2097-mw2114
11:10 marostegui: Compress innodb on db2038 and db2084 - T178359
10:56 moritzm: upgrading hhvm-wikdiff2 on remaining API servers in codfw
10:46 moritzm: upgrading hhvm-wikdiff2 on mw1161-mw1167 (job runners)
10:28 moritzm: upgrading hhvm-wikdiff2 on mw1180-mw1188 (app servers)
10:28 marostegui: Remove /srv from db1015 as it has been stopped for weeks now and will be decommissioned (and it is alerting low on disk space) - T173570
10:09 dcausse: elasticsearch/cirrus reindexing 167 wikis from terbium (T177871 )
10:08 gehel: restart tilerator / karthotherian for config change on all maps clusters - T160215
10:01 ema: restart varnish-be on cp4021 (mbox lag)
09:58 moritzm: upgrading hhvm-wikidiff2 on mw1293-mw1298 (scalers)
09:09 moritzm: upgrading hhvm-wikidiff2 on mw1189-mw1208 (API servers)
08:09 marostegui: Rename wb_entity_per_page table on s5 db1092 and s3 db1077 - T177601
08:07 moritzm: upgrading hhvm-wikdiff2 on mw1209-mw1220 (app servers)
07:23 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2037 and db2038 - T178359 (duration: 00m 46s)
06:56 moritzm: installing expat security updates on trusty
06:51 marostegui: Stop replication in sync on db1078 and db1103 to checksum data - T164488
06:50 moritzm: installing poppler security updates on trusty
06:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1078 - T164488 (duration: 00m 47s)
06:30 marostegui: Stop replication in sync on db1103 and db2018 to fix data drifts - T164488
06:09 marostegui: Stop replication on db2092 to re-import and compress linter and watchlist table
02:41 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Oct 23 02:41:04 UTC 2017 (duration 6m 55s)
02:34 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.4) (duration: 08m 45s)
08:16 hashar: Mass code-review+2 changes made by LibraryUpdater that already had Code-Review:+2 and NOT verified=-1 ( ping legoktm )
08:08 hashar: Stopping Zuul to flush its queue
23:46 demon@tin: Finished deploy [gerrit/gerrit@21b7332]: pushing deleteproject plugin (duration: 00m 09s)
23:46 demon@tin: Started deploy [gerrit/gerrit@21b7332]: pushing deleteproject plugin
19:19 hasharAway: restarting Jenkins - T178608
18:52 bblack: vhtcpd upgrade + queue delay puppetization deploy ( https://gerrit.wikimedia.org/r/385415 ) done on cp* fleet - T133821
18:28 bblack: upgrading vhtcpd to 0.1.1-1 on the cp* fleet
17:38 mutante: removing Apache and HHVM Ganglia stats from all appservers, part of retiring Ganglia (T177225 ) - some transient puppet issues while purging package and remnants - use grafana dashboard at https://grafana.wikimedia.org/dashboard/db/prometheus-apache-hhvm-dc-stats
16:39 aaron@tin: Synchronized php-1.31.0-wmf.4/extensions/TwoColConflict: 5c3224980fe
16:37 aaron@tin: Synchronized php-1.31.0-wmf.4/extensions/OATHAuth: c16a94e17b9981
16:05 krinkle@tin: Synchronized php-1.31.0-wmf.4/extensions/WikimediaMaintenance/getJobQueueLengths.php: I8737795e6
16:05 XioNoX: removing "policy-options policy-statement LVS_import term secondary" on cr routers
14:39 cmjohnson1: updating power firmware analytics1037
10:14 moritzm: upgrading hhvm-wikidiff2 to 1.5.1 on mw2150/mw2151/mw2244/mw2245
09:17 moritzm: upgrading hhvm-wikidiff2 to 1.5.1 on mw2200-mw2223
08:05 Amir1: mwscript createAndPromote.php --wiki=techconductwiki --sysop --bureaucrat 'Ladsgroup' --force
07:39 jynus: stopping db2033, too
07:36 jynus: stopping dbstore1002 and dbstore2002 x1 replication in sync
07:18 _joe_: uploading updated versions of docker base images (wikimedia-jessie, wikimedia-stretch) to docker-registry.wikimedia.org
06:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1078 - T164488 (duration: 00m 46s)
06:17 marostegui: Stop replication in sync on db1103 and db1078 - T164488
06:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1078 - T164488 (duration: 00m 45s)
06:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1072 - T164488 (duration: 00m 46s)
05:52 marostegui: Stop replication in sync on db1103 and db1072 - T164488
05:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1051 - T174509 (duration: 00m 47s)
02:19 bblack: upgrading vhtcpd to 0.1.1-1 on: cp1008, cp3030, cp3034, cp4029-32
02:17 bblack: uploaded vhtcpd-0.1.1-1 to reprepro
22:21 addshore: stop running extra change dispatchers on terbium for wikidatawiki
21:58 addshore: running extra change dispatchers on terbium for wikidatawiki for a short while
21:24 andrewbogott: restarting nginx on nitrogen
21:22 andrewbogott: restarting puppetdb on nitrogen
20:00 mforns@tin: Finished deploy [analytics/refinery@0c9ba04]: deploying refinery to fix banner activity false alarms (duration: 00m 14s)
19:59 mforns@tin: Started deploy [analytics/refinery@0c9ba04]: deploying refinery to fix banner activity false alarms
19:53 bblack: reboot cp4029-32 (initial reboot post-puppetization)
19:12 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.4
18:36 thcipriani@tin: Synchronized portals: SWAT: Bumping portals to master T128546 (duration: 00m 51s)
18:35 thcipriani@tin: Synchronized portals/prod/wikipedia.org/assets: SWAT: Bumping portals to master T128546 (duration: 00m 50s)
18:28 thcipriani@tin: Synchronized wmf-config/abusefilter.php: SWAT: Enable $wgAbuseFilterProfile on ptwiki T177641 (duration: 00m 50s)
18:20 thcipriani@tin: Synchronized php-1.31.0-wmf.4/extensions/Echo/modules/styles/mw.echo.ui.PageNotificationsOptionWidget.less: SWAT: mw.echo.ui.PageNotificationsOptionWidget: Fix CSS after changes in OOjs UI T178439 (duration: 00m 51s)
16:50 hashar: contint1001: installed python3 (pending https://gerrit.wikimedia.org/r/385210 )
15:14 marostegui: Stop replication in sync on db1103 and db1072 - https://phabricator.wikimedia.org/T164488
15:02 mforns@tin: Finished deploy [analytics/refinery@0c9ba04]: (no justification provided) (duration: 00m 07s)
15:02 mforns@tin: Started deploy [analytics/refinery@0c9ba04]: (no justification provided)
14:36 mforns@tin: Finished deploy [analytics/refinery@0c9ba04]: (no justification provided) (duration: 03m 29s)
14:33 mforns@tin: Started deploy [analytics/refinery@0c9ba04]: (no justification provided)
13:20 moritzm: upgrading mw2120-mw2147 to wikidiff2 1.5.1
12:35 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2034 and db2036 - T178359 (duration: 00m 50s)
11:40 akosiaris: T177196 upload prometheus-postgres-exporter_0.2.0+ds-2 to apt.wikimedia.org/stretch-wikimedia/main and copied over to apt.wikimedia.org/jessie-wikimedia/main
11:10 moritzm: upgrading mw1276-mw1279 (canary servers) to wikidiff2 1.5.1
10:14 godog: upgrade prometheus-hhvm-exporter to 0.4-1
10:07 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: Electron hanging - T174916
09:54 ppchelko@tin: Finished deploy [restbase/deploy@2001a66]: Enable summary on Cass 3 for all but wikipedia (duration: 07m 32s)
09:50 ariel@tin: Finished deploy [dumps/dumps@2b9326b]: improved job batches for rev content 7z recompression (duration: 00m 02s)
09:50 ariel@tin: Started deploy [dumps/dumps@2b9326b]: improved job batches for rev content 7z recompression
09:48 moritzm: installing pyjwt security updates
09:47 ppchelko@tin: Started deploy [restbase/deploy@2001a66]: Enable summary on Cass 3 for all but wikipedia
08:13 marostegui: Stop replication in sync on db1103 and db1072 - T164488
08:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1072 - T164488 (duration: 00m 50s)
07:31 marostegui: Stop MySQL on db2034 and db2036 to clone db2092
06:36 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2034 and db2036 - T178359 (duration: 01m 08s)
06:02 marostegui: Compress revision table across db1044 databases (s3)
05:46 marostegui: Optimize recentchanges, pagelinks and templatelinks on db1102 for s6 - T174509 T177772
05:45 marostegui: Optimize recentchanges, pagelinks and templatelinks on db1064 - T174509 T177772
05:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1051 - T174509 (duration: 00m 50s)
05:41 marostegui: Optimize pagelinks and templatelinks on db1051 - T174509
05:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1055 - T174509 (duration: 00m 50s)
05:31 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1056 - T174509 (duration: 00m 50s)
05:29 awight@tin: Started deploy [ores/deploy@19b1851]: Push ORES w/ Celery 4 support (take 2), T178441
05:08 awight@tin: Finished deploy [ores/deploy@19b1851]: Push ORES w/ Celery 4 support, T178441 (duration: 00m 18s)
05:08 awight@tin: Started deploy [ores/deploy@19b1851]: Push ORES w/ Celery 4 support, T178441
03:38 bblack: cp3034 - upgraded vhtcpd to 0.1.0 for testing
03:38 bblack: cp3030 - upgraded vhtcpd to 0.1.0 for testing
03:01 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Oct 19 03:01:54 UTC 2017 (duration 7m 2s)
02:54 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.4) (duration: 09m 28s)
02:29 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.3) (duration: 09m 40s)
00:01 maxsem@tin: Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/385106/2 (duration: 00m 50s)
23:51 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/385094/2 (duration: 00m 50s)
23:49 maxsem@tin: Synchronized static/images/project-logos/: https://gerrit.wikimedia.org/r/#/c/385094/2 (duration: 00m 50s)
23:45 maxsem@tin: Synchronized php-1.31.0-wmf.4/extensions/Collection/: https://gerrit.wikimedia.org/r/#/c/384903/ (duration: 00m 50s)
23:44 maxsem@tin: Synchronized php-1.31.0-wmf.3/extensions/Collection/: https://gerrit.wikimedia.org/r/#/c/385005/ (duration: 00m 51s)
23:34 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/385007/ (duration: 00m 51s)
23:05 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.4
23:04 demon@tin: Synchronized php: symlink swap (duration: 00m 49s)
22:55 tgr@tin: Synchronized private/: Remove the llama (T150554 ) (duration: 00m 50s)
22:27 tgr@tin: Synchronized wmf-config/: T174651 / gerrit 384908 - deploy ReadingLists to beta - noop for production (duration: 00m 53s)
20:49 ejegg: updated CiviCRM from 7c26791 b9d2067
20:16 XioNoX: Changing MTUs on interfaces to NTT
19:19 bblack: uploaded vhtcpd-0.1.0-1 to jessie-wikimedia, testing on cp1008 only for now
19:17 andrewbogott: restarting nodepool
19:15 andrewbogott: stopping nodepool temporarily to give rabbit a chance to catch up
19:01 MaxSem: ran LoginNotify/maintenance/migratePreferences.php on test and test2
18:57 andrewbogott: restarting nova-network on labnet1001
18:42 andrewbogott: restarting rabbitmq-server on labcontrol1001; too many timeouts
16:29 elukey: stop ircecho on einstenium for puppet shower
16:27 herron: restarted apache on puppetmasters
14:23 moritzm: uploaded hhvm 3.18.5+dfsg-1+wmf1+deb9u1 to apt.wikimedia.org/stretch-wikimedia
14:03 chasemp: add static ip to cloud mwstake project per T178012
14:03 chasemp: bump up quota on cyberpower cloud project per T178332
14:03 chasemp: create reading-lists cloud project
13:28 zeljkof: EU SWAT finished!
13:22 zfilipin@tin: Synchronized php-1.31.0-wmf.3/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: [cirrus] Turn on recall A/B test on enwiki (T177502) (duration: 00m 50s)
13:21 gehel: repooling wdqs1003 now that it has catched up on updates
13:15 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: pagePreviews: Restart A/B test on enwiki and dewiki (T176469) (duration: 00m 51s)
13:05 moritzm: uploaded wikidiff2 1.5.1 to apt.wikimedia.org
12:11 mobrovac@tin: Finished deploy [restbase/deploy@3c7abf6]: Bug fix: Remove the space in the summary table name (duration: 07m 13s)
12:04 mobrovac@tin: Started deploy [restbase/deploy@3c7abf6]: Bug fix: Remove the space in the summary table name
11:54 mobrovac@tin: Finished deploy [restbase/deploy@99052c1]: Use Cass3 storage for ruwiki summaries (duration: 08m 59s)
11:47 jynus@tin: Synchronized wmf-config/db-eqiad.php: Add db1105 to help db1071 because db1082 has crashed (duration: 00m 50s)
11:45 mobrovac@tin: Started deploy [restbase/deploy@99052c1]: Use Cass3 storage for ruwiki summaries
11:31 moritzm: upgrading mw1262-mw1265 (canaries) to wikidiff2 1.5.1
11:26 marostegui: Optimize pagelinks and templatelinks on db1102 s7 - T174509
11:21 moritzm: upgrading mw1261 to wikidiff2 1.5.1
11:11 mobrovac@tin: Finished deploy [restbase/deploy@15619e0]: Introduce the page/summary proxy (no-op for now), part #2 (duration: 08m 53s)
11:03 mobrovac@tin: Started deploy [restbase/deploy@15619e0]: Introduce the page/summary proxy (no-op for now), part #2
11:02 mobrovac@tin: Finished deploy [restbase/deploy@15619e0]: Introduce the page/summary proxy (no-op for now) (duration: 05m 23s)
10:57 mobrovac@tin: Started deploy [restbase/deploy@15619e0]: Introduce the page/summary proxy (no-op for now)
10:50 moritzm: rebooting multatuli for kernel update
10:21 moritzm: imported linux 4.9.30-2+deb9u5~bpo8+1 for jessie-wikimedia
09:51 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1098, depool db1082 (crashed) (duration: 00m 50s)
09:39 moritzm: installing xserver/xvfb security updates
09:36 jynus: reloading dbproxy1010's haproxy configuration
09:30 elukey: drop MobileWikiAppToCInteraction_10375484_15423246 T177960
08:41 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Re-enable Statement usage tracking on cawiki (T151717 ) (duration: 00m 50s)
08:25 marostegui: Stop MySQL on db2019 to copy its data to db2084 - T178359
08:23 mobrovac@tin: Finished deploy [changeprop/deploy@065a06e]: Bug fix: Apply ignore_errors on a per-request basis (duration: 01m 17s)
08:22 mobrovac@tin: Started deploy [changeprop/deploy@065a06e]: Bug fix: Apply ignore_errors on a per-request basis
08:09 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable description usage tracking on a few test wikis (T177155 ) (duration: 00m 50s)
07:49 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1056 - T174509 (duration: 00m 49s)
07:46 marostegui: Optimize pagelinks and templatelinks on db1056 - T174509
07:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1034 - T174509 (duration: 00m 59s)
07:14 gehel: depooling wdqs1003 until it catches up on updates
07:02 marostegui: Stop MySQL on db2079 to copy its data to db2084 - T178359
06:46 _joe_: powercycling wdqs1003
05:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1072 - T164488 (duration: 00m 49s)
05:43 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1055 - T174509 (duration: 00m 50s)
05:40 marostegui: Optimize pagelinks and templatelinks on db1055 - T174509
05:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1098 - T174509 (duration: 00m 49s)
05:36 marostegui: Optimize pagelinks and templatelinks on db1098 - T174509
05:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1067 - T174509 (duration: 00m 50s)
05:25 marostegui: Optimize pagelinks and templatelinks on db1102 for s2 - T174509
03:17 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Oct 18 03:17:01 UTC 2017 (duration 7m 3s)
03:09 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.4) (duration: 14m 58s)
02:33 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.3) (duration: 08m 21s)
23:31 thcipriani@tin: Synchronized php-1.31.0-wmf.4/extensions/Citoid/modules/ve.ui.CiteFromIdInspector.css: SWAT: ve.ui.CiteFromIdInspector: Fix CSS for context menus after changes in OOjs UI T178324 (duration: 00m 50s)
23:29 thcipriani@tin: Synchronized php-1.31.0-wmf.4/extensions/VisualEditor/modules/ve-mw/ui/styles/widgets/ve.ui.MWMediaInfoFieldWidget.css: SWAT: ve.ui.MWMediaInfoFieldWidget: Fix positioning of icons T178415 (duration: 00m 50s)
23:22 thcipriani@tin: Synchronized php-1.31.0-wmf.3/includes/shell/Command.php: SWAT: Shell\Command: Better walltime fallback T178314 (duration: 00m 51s)
23:15 thcipriani@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Remove setting no longer in MediaWiki (duration: 00m 50s)
23:06 mutante: disabling Icinga notifications for service recommendation_api on scb hosts - please remember to re-enable once ticket is resolved (T178445 ) (https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=recommendation_api%20endpoints )
22:28 ejegg: updated CiviCRM from fb513cc 7c26791
21:55 foks: Removed 2FA from User:Amakuru
21:39 robh: cp4026 being returned to service post memory replacement. puppet runs fine and all icinga checks are green. repooled.
21:23 robh: cp4026 coming down for memory swap, no point in maint moding the system since other cp checks will alert anyhow
20:14 mutante: gerrit2001 - re-enabled puppet, attempting restart with --slave after gerrit:384758
19:54 mutante: cobalt - re-enabled puppet, ran puppet, systemd says gerrit is active(running)
19:51 mutante: cobalt - stopping gerrit service, confirming pid file gets removed, move /etc/init.d/gerrit to /root
19:50 mutante: short gerrit downtime for systemd change coming up
19:23 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.4
19:06 mutante: gerrit2001 - rebooting to verify systemd change and service comes back properly
18:53 mutante: gerrit2001: stopping the gerrit service
18:52 mutante: cobalt: temp disable puppet | gerrit2001: stop puppet
18:22 paravoid: rebooting sodium for kernel/distro upgrade
18:14 paravoid: dist-upgrading sodium to stretch
18:12 demon@tin: Finished scap: bootstrap wmf.4 (duration: 48m 41s)
17:23 demon@tin: Started scap: bootstrap wmf.4
16:49 demon@tin: Pruned MediaWiki: 1.31.0-wmf.2 [keeping static files] (duration: 01m 24s)
16:35 gehel: restart tilerator / tileratorui on all maps cluster after deployment of latest fix - T177389
16:34 gehel@tin: Finished deploy [tilerator/deploy@f3b26f3]: fix tileshell not exiting - T177389 (duration: 01m 42s)
16:32 gehel@tin: Started deploy [tilerator/deploy@f3b26f3]: fix tileshell not exiting - T177389
15:46 _joe_: killed a stuck gerrit process on cobalt
14:47 bd808: wikitech: Cleaned up 'importers' user_group entries (T171682 )
14:25 Amir1: end of cleaning up ores_classification table in plwiki, start of ruwiki (T159753 )
14:09 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1081 - T174509 T177772 (duration: 00m 45s)
14:05 Amir1: start of cleaning up ores_classification table in plwiki (T159753 )
14:00 zeljkof: EU SWAT finished
13:59 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: [cirrus] Fix typo in UserTesting config var (take 2) (T177502) (duration: 00m 45s)
13:47 zeljkof: EU SWAT continues
13:31 zeljkof: stopping EU SWAT
13:29 jynus: killing commonswiki script on tin
13:17 zfilipin@tin: Synchronized php-1.31.0-wmf.3/extensions/UniversalLanguageSelector/maintenance/ULSCompactLinksDisablePref.php: SWAT: Remove the 20 edits threshold from ULSCompactLinksDisablePref.php (duration: 00m 46s)
12:53 volans: upgrading cumin masters to cumin 1.2.2-1
12:40 moritzm: upgrading mwdebug* to hhvm-wikidiff 1.5.1
11:47 marostegui: Optimize recentchanges on db1102 for s4 - T174509
11:47 marostegui: Optimize templatelinks and pagelinks on db1102 for s4 - T174509
11:45 Amir1: end of cleaning up ores_classification table in hewiki, start of nlwiki (T159753 )
11:39 Amir1: end of cleaning up ores_classification table in fiwiki, start of hewiki (T159753 )
11:32 jynus: test-installing proxysql on wasat T175672
11:24 Amir1: end of cleaning up ores_classification table in fawiki, start of fiwiki (T159753 )
11:12 gehel: restarting wdqs-updater on wdqs1004 - T175919
10:40 Amir1: end of cleaning up ores_classification table in etwiki, start of fawiki (T159753 )
10:36 Amir1: end of cleaning up ores_classification table in cswiki, start of etwiki (T159753 )
10:26 Amir1: start of cleaning up ores_classification in cswiki (T159753 )
10:16 akosiaris: removing akosiaris's yubikey from network devices
09:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1085 - T174509 T177772 (duration: 00m 46s)
09:23 marostegui: Stop MySQL on db2084 for multi-instance testing
09:16 mobrovac@tin: Finished deploy [restbase/deploy@7f228d4]: Remove /page/revision and add history metrics end points, part #3 (duration: 07m 34s)
09:08 mobrovac@tin: Started deploy [restbase/deploy@7f228d4]: Remove /page/revision and add history metrics end points, part #3
09:08 mobrovac@tin: Finished deploy [restbase/deploy@3bcd1b7]: Remove /page/revision and add history metrics end points, part #2 (duration: 25m 57s)
08:45 gehel: restarting blazegraph on wdqs1004 for GC tuning (adding -XX:+G1PrintRegionLivenessInfo) - T175919
08:42 mobrovac@tin: Started deploy [restbase/deploy@3bcd1b7]: Remove /page/revision and add history metrics end points, part #2
08:40 mobrovac@tin: Finished deploy [restbase/deploy@3bcd1b7]: Remove /page/revision and add history metrics end points - T158100 T175805 (duration: 01m 05s)
08:39 mobrovac@tin: Started deploy [restbase/deploy@3bcd1b7]: Remove /page/revision and add history metrics end points - T158100 T175805
07:54 marostegui: Stop MySQL and reboot db2081 to see if it works fine - T178140
07:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2086 to the config - T170662 (duration: 00m 45s)
07:52 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2086 to the config - T170662 (duration: 00m 45s)
07:51 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2086 to the config - T170662 (duration: 00m 45s)
07:42 marostegui: Stop MySQL on db2079 to clone db2086 - T170662
07:36 moritzm: rebooting boron for kernel update
07:31 ema: upgrade misc_eqiad to varnish 5 T177233
07:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1067 - T174509 (duration: 00m 45s)
07:11 marostegui: Optimize templatelinks and pagelinks on db1067 - T174509
07:10 marostegui: Optimize enwiki.ores_classification on db1067 - T159753
06:55 moritzm: installing libffi security updates on trusty (Debian already fixed)
06:07 marostegui: Stop replication in sync on db1072 and db1103 for data drift fixing - T164488
05:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1034 - T174509 (duration: 00m 45s)
05:54 marostegui: Optimize pagelinks and templatelinks on db1034 - T174509
05:50 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1085 - T174509 T177772 (duration: 00m 45s)
05:46 marostegui: Optimize pagelinks, templatelinks and recentchanges on db1085 T177772 T174509
05:42 marostegui: Optimize pagelinks, templatelinks and ores_classification on db1095 - T174509 T159753
05:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1081 - T174509 (duration: 00m 45s)
05:38 marostegui: Optimize recentchanges on db1081 - T177772
05:37 marostegui: Optimize pagelinks and templatelinks on db1081 - T174509
05:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1090 - T174509 (duration: 00m 45s)
05:32 marostegui: Optimize pagelinks and templatelinks on db1090 - T174509
05:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1073 - T174509 (duration: 00m 46s)
02:32 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Oct 17 02:32:30 UTC 2017 (duration 6m 57s)
02:25 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.3) (duration: 08m 23s)
00:34 ejegg: updated CiviCRM from 824c2d8 fb513cc
23:49 thcipriani@tin: Synchronized php-1.31.0-wmf.3/extensions/CheckUser/specials/SpecialCheckUser.php: SWAT: Revert "Revert "Restore checkuser-userlinks when exist"" T170507 (duration: 00m 46s)
23:46 ebernhardson: increase elasticsearch log levels for transport to debug to help track down T167410
23:39 ejegg: updated civicrm from b953064 824c2d8
23:25 thcipriani@tin: Synchronized php-1.31.0-wmf.3/extensions/Echo/includes/ContainmentSet.php: SWAT: ContainmentSet: Use strict comparison for array_search() T177825 (duration: 01m 45s)
20:35 chasemp: disable puppet around cloud things for a refactor merge
20:20 demon@tin: Pruned MediaWiki: 1.30.0-wmf.18 (duration: 02m 59s)
19:59 demon@tin: Synchronized wmf-config/CommonSettings.php: Ief0ea01f
19:58 jynus: sudo apt install wmf-mariadb101-client on terbium for debugging an issue
19:57 demon@tin: Synchronized wmf-config/InitialiseSettings.php: Ief0ea01f
19:56 demon@tin: Synchronized tests/cirrusTest.php: Ief0ea01f
18:18 thcipriani@tin: Synchronized wmf-config/Wikibase.php: SWAT: Add configuration for statement indexing for Wikidata T175199 (duration: 00m 47s)
17:35 ariel@tin: Finished deploy [dumps/dumps@2b9326b]: do the minimum for latestlinks job, no updating other files (duration: 00m 03s)
17:35 ariel@tin: Started deploy [dumps/dumps@2b9326b]: do the minimum for latestlinks job, no updating other files
17:08 gehel@tin: Finished deploy [wdqs/wdqs@b42eb0d]: WDQS GUI uipdates (duration: 02m 20s)
17:06 gehel@tin: Started deploy [wdqs/wdqs@b42eb0d]: WDQS GUI uipdates
16:56 ottomata: deploying LVS for druid-public-broker https://phabricator.wikimedia.org/T176223
16:39 joal@tin: Finished deploy [analytics/aqs/deploy@339674c]: Deploy new mediawiki-history endpoints (alpha version) after monitorin fix (duration: 04m 42s)
16:35 joal@tin: Started deploy [analytics/aqs/deploy@339674c]: Deploy new mediawiki-history endpoints (alpha version) after monitorin fix
16:23 joal@tin: Finished deploy [analytics/aqs/deploy@aa7ef0d]: Deploy new mediawiki-history endpoints (alpha version) after node-modules fix (duration: 06m 56s)
16:16 joal@tin: Started deploy [analytics/aqs/deploy@aa7ef0d]: Deploy new mediawiki-history endpoints (alpha version) after node-modules fix
16:09 ema: upgrade misc_codfw to varnish 5 T177233
16:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1084 - T174509 T177772 (duration: 00m 46s)
15:51 gehel: killing stuck tileshell process on maps-test2001 - T177389
15:51 gehel@tin: Finished deploy [tilerator/deploy@f3b26f3]: testing tileshell fix - T177389 (duration: 00m 20s)
15:50 gehel@tin: Started deploy [tilerator/deploy@f3b26f3]: testing tileshell fix - T177389
15:45 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1094 - T174509 (duration: 00m 46s)
15:38 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1088 - T174509 T177772 (duration: 00m 47s)
14:34 twentyafterfour: deploying hotfix for T178068 and T178107
14:30 moritzm: uploaded wikidiff 1.5.1 to apt.wikimedia.org for jessie/experimental
14:08 hashar: European swat is complete
14:08 hashar@tin: Synchronized wmf-config/abusefilter.php: (T177760 ) Enable AbuseFilter profiler | (T177761 ) Allow AbuseFilter to block & grant relevant permissions to permit administrators to manage the restricted option. (duration: 00m 47s)
13:50 moritzm: upgrading deployment-mediawiki0[56] to wikidiff2 1.5.1
13:45 hashar@tin: Synchronized wmf-config/: [cirrus] Prepare profiles for the recall A/B test on enwiki - T177502 (duration: 00m 48s)
13:44 moritzm: upgrading deployment-mediawiki04 to wikidiff2 1.5.1
13:42 hashar@tin: Synchronized wmf-config: [cirrus] Disable A/B test of MLR on testing on 18 wikis with > 1% of search traffic - T177490 (duration: 00m 48s)
13:38 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Add additional namespaces to search results for bnwikisource - T178041 (duration: 00m 49s)
12:59 gehel: restart cassandra on maps1001
12:38 ema: upgrading eqiad LVSs to pybal 1.14.2 (T178149 , T177815 )
11:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1074 - T174509 (duration: 00m 46s)
11:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1070 - T174509 (duration: 00m 46s)
11:10 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: Electron hanging - T174916
10:48 Amir1: cleaning up ores_classification table in wikidatawiki (T159753 )
10:03 ema: upgrading codfw LVSs to pybal 1.14.2 (T178149 , T177815 )
09:54 ema: upgrading esams LVSs to pybal 1.14.2 (T178149 , T177815 )
09:44 ema: upgrading ulsfo LVSs to pybal 1.14.2 (T178149 , T177815 )
09:43 ema: pybal 1.14.2 uploaded to apt.w.o
09:40 moritzm: restarting cassandra on restbase1007 to pick up NSS security update
09:39 moritzm: restarting cassandra on cerium/xenon/praseodymium to pick up NSS security update
09:37 gehel: restarting elasticsearch on elastic1017
09:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1088 - T174509 T177772 (duration: 00m 46s)
09:07 marostegui: Optimize recentchanges, pagelinks and templatelinks on db1088 - https://phabricator.wikimedia.org/T174509 https://phabricator.wikimedia.org/T177772
09:02 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1093 - T174509 T177772 (duration: 00m 46s)
08:53 marostegui: Stop MySQL on db1050 as it will be decommissioned - T178162
08:46 marostegui: Remove db1050 from tendril - T178162
08:31 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1050 from config - T178162 (duration: 00m 46s)
08:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1050 from config - T178162 (duration: 00m 46s)
07:56 marostegui: Stop db1103 and db1038 at the same position to fix data drifts - T164488
07:42 moritzm: installing NSS security updates
07:30 marostegui: Stop db1103 and db1072 at the same position to fix data drifts - T164488
07:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1072 - T164488 (duration: 00m 46s)
07:19 marostegui: Optimize table ores_classification on db1073 - T159753
07:14 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2084 to the config - T170662 (duration: 00m 46s)
07:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2084 to the config - T170662 (duration: 00m 47s)
06:25 marostegui: Stop MySQL on db2079 to clone db2084 from it - T170662
06:12 marostegui: Optimize pagelinks and templatelinks on db1094 - T174509
06:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1093 - T174509 T177772 (duration: 00m 46s)
06:04 marostegui: Optimize recentchanges, pagelinks and templatelinks on db1093 - T174509 T177772
06:00 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1070 - T174509 (duration: 00m 46s)
05:59 marostegui: Optimize recentchanges, pagelinks and templatelinks on db1070 - T174509
05:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1084 - T174509 T177772 (duration: 00m 46s)
05:53 marostegui: Optimize recentchanges, pagelinks and templatelinks on db1084 - T174509 T177772
05:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1074 - T174509 (duration: 00m 46s)
05:47 marostegui: Optimize pagelinks and templatelinks on db1074 - T174509
05:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1073 - T174509 (duration: 00m 47s)
05:37 marostegui: Optimize templatelinks and pagelinks on db1073 - T174509
05:36 marostegui: Optimize templatelinks and pagelinks on db1073 - 174509
02:35 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Oct 16 02:35:34 UTC 2017 (duration 6m 36s)
02:28 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.3) (duration: 07m 44s)
11:15 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: Electron hanging - T174916
03:52 paravoid: manually cleaning up pdns_metric_cron from all dnsrecursors (hydrogen/chromium/acamar/achernar/maerlant/nescio)
23:04 mutante: scb1001 - systemctl restart pdfrender (Icinga said: pdfrender Connection refused)
20:36 mutante: gerrit2001 - removing hhvm package (which is now possible without also removing scap, thanks thcipriani), apt-get autoremove to remove more unused packages (T178039 )
18:52 joal@tin: Finished deploy [analytics/aqs/deploy@0375c34]: Deploy new mediawiki-history endpoints-alpha version (duration: 15m 02s)
18:40 joal@tin: (no justification provided)
18:37 joal@tin: Started deploy [analytics/aqs/deploy@0375c34]: Deploy new mediawiki-history endpoints-alpha version
16:54 otto@tin: Finished deploy [analytics/refinery@28db253]: (no justification provided) (duration: 05m 50s)
16:48 otto@tin: Started deploy [analytics/refinery@28db253]: (no justification provided)
15:31 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1056 (duration: 00m 46s)
14:59 mutante: added cparle to wmf LDAP group
14:38 jynus: stopping slave on db1056 and optimize recentchanges
14:25 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1100, depool db1056, fix db1098 weight (duration: 00m 46s)
14:06 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1053 (duration: 00m 46s)
13:07 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1053 (duration: 00m 47s)
12:57 godog: upload scap 3.7.1-1 - T127762
11:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2082 to the config - T170662 (duration: 00m 46s)
11:26 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2082 to the config - T170662 (duration: 00m 46s)
10:53 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1098 (duration: 00m 47s)
10:22 godog: roll-restart pybal to apply https://gerrit.wikimedia.org/r/#/c/383997/ - T178149
10:04 jynus: stopping slave on db1098 and optimize recentchanges
09:54 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1098 (duration: 00m 46s)
09:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2082 to the config - T170662 (duration: 00m 46s)
09:22 Amir1: a small clean up of ores_classification table in wikidatawiki (T159753 )
09:21 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2082 to the config - T170662 (duration: 00m 47s)
09:04 godog: restart pybal on lvs1003 to test logstash depool - T178078
08:57 marostegui: Stop db1103 and db1038 in sync for more checksumming - T164488
08:34 ema: upgrade pybal on lvs1003 to 1.14.1 T177815
08:18 ema: upgrade pybal on lvs1006 to 1.14.1 T177815
08:15 jynus: restarting commons wiki recentchanges purge of wb entries T177772
08:11 ema@neodymium: conftool action : set/pooled=yes; selector: name=logstash1001.eqiad.wmnet,service=logstash-gelf,dc=eqiad
08:11 ema@neodymium: conftool action : set/pooled=no; selector: name=logstash1001.eqiad.wmnet,service=logstash-gelf,dc=eqiad
08:10 ema: depool/repool logstash1001 (service=logstash-gelf) T178078
07:35 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2081 to the config - T170662 (duration: 00m 46s)
07:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2081 to the config - T170662 (duration: 00m 47s)
06:23 marostegui: Stop MySQL on db2080 to copy its data to db2081 - T170662
00:22 mutante: authdns servers: deleted /etc/ganglia/conf.d/gndsd.pyconf & /usr/lib/ganglia/python_modules_gdnsd.py - removing ganglia stats
23:58 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Restore thanks limit on pl.wikipedia (T169268 ) (duration: 00m 47s)
23:50 cwd: re-enabled omnimail jobs
23:35 eileen1: update from 2d6668b b953064
23:25 dereckson@tin: Synchronized wmf-config/CommonSettings.php: Add logging for email blocks (T175419 ) (duration: 00m 46s)
23:13 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Switch test wikis to HTML5 fragment mode in links (T152540 ) (duration: 00m 47s)
22:44 demon@tin: Synchronized php-1.31.0-wmf.3/includes/specialpage/ChangesListSpecialPage.php: silence notices (duration: 00m 47s)
21:36 cwd: disabled omnimail jobs
19:47 ejegg: updated CiviCRM from 47de976 2d6668b
19:46 chasemp: disable puppet for lab*services* for merge of 383896
19:18 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.3
18:28 mutante: added Pablo (pgrass) to LDAP group 'wmde' (T177599 )
18:15 demon@tin: Synchronized php-1.31.0-wmf.3/extensions/ORES/includes/Hooks.php: Make watchlist queries suck less (duration: 00m 50s)
18:01 demon@tin: Synchronized php-1.31.0-wmf.3/extensions/JsonConfig/includes/JCCache.php: Silence junk warnings (duration: 00m 50s)
15:17 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: Electron hanging - T174916
15:11 mutante: gerrit2001 - rebooting for kernel upgrade
14:57 mutante: netmon1002 - arming keyholder for rancid deploy
14:52 mutante: netmon1002 (librenms, rancid) - rebooting for kernel upgrade
14:47 moritzm: rebooting kubernetes workers to new 4.9.51 kernel
14:41 moritzm: rebooting kubernetes staging servers to new 4.9.51 kernel
14:29 moritzm: rebooting kubernetes masters to new 4.9.51 kernel
13:54 moritzm: updating image scalers to librsvg 2.40.18
13:48 elukey: deployed the new Analytics Public Druid cluster - T176223
13:45 moritzm: updating remaining thumbor servers to librsvg 2.40.18
13:40 ladsgroup@tin: Synchronized wmf-config/Wikibase.php: SWAT: Remove deprecated config variable for Wikibase (T129475 ) (duration: 01m 02s)
13:33 moritzm: updating thumbor1001 to librsvg 2.40.18
13:13 moritzm: rolling restart of logstash to pick up thumbor logs (T150734 )
13:07 zeljkof: EU SWAT finished
12:56 moritzm: uploaded librsvg 2.40.18 for jessie-wikimedia to apt.wikimedia.org
11:46 ema: upgrading cp3010 to Varnish 5 T177233
11:27 Amir1: finished cleanup of ores_classification in wikidatawiki, still needs more work (T159753 )
10:03 ema: lvs1005: upgrade pybal to 1.14.1 T177815
10:02 ema: pybal 1.14.1 uploaded to apt.w.o
09:40 ema: upgrading cp3008 to Varnish 5 T177233
09:06 moritzm: installing nettle update from stretch 9.2 point release
09:01 moritzm: installing ncurses security updates
08:25 moritzm: installing gnutls28 update from stretch 9.2 point release
08:02 Amir1: starting cleanup of ores_classification in wikidatawiki (T159753 )
07:51 gehel: reinitialize cassandra on maps-test2004 to test vector tiles - T153282
07:40 moritzm: installing at-spi2-core update from stretch 9.2 point release
07:24 moritzm: installing gnupg2 update from stretch 9.2 point release
07:18 legoktm: purging html5-misnesting linter entries from database on terbium (T178040 )
03:25 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Oct 12 03:25:17 UTC 2017 (duration 7m 17s)
03:18 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.3) (duration: 15m 57s)
02:41 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 15m 39s)
00:30 twentyafterfour: phabricator update finished (a while ago, I forgot to log it)
00:08 twentyafterfour: upgrading phabricator to #phab-2017-10-11
00:05 mutante: netmon2001 arm keyholder for rancid deploy
00:01 twentyafterfour: Preparing to take phabricator offline for a hopefully momentary upgrade.
23:55 mutante: netmon2001 - smokeping back up - the reboot, as a side-effect, also resolved the systemd-icinga-alert that was pending in Icinga for a while with comments
23:52 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Vector print logo and print styles on all wikis T169732 PART II (duration: 00m 50s)
23:50 thcipriani@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Enable Vector print logo and print styles on all wikis T169732 PART I (duration: 00m 50s)
23:47 mutante: netmon2001 (smokeping) - kernel upgrade, schedule downtime, reboot
23:40 thcipriani@tin: Synchronized php-1.31.0-wmf.2/skins/Vector/ResourceLoaderLessModule.php: SWAT: Print logo should use an absolute URI T177800 (duration: 00m 49s)
23:38 mutante: netmon1003 - servermon back up - apt-get autoremove to drop unrequired python packages
23:37 thcipriani@tin: Synchronized php-1.31.0-wmf.3/skins/Vector/ResourceLoaderLessModule.php: SWAT: Print logo should use an absolute URI T177800 (duration: 00m 50s)
23:35 mutante: netmon1003 - cant seem to reboot normally, attempting via gnt-instance command
23:31 mutante: netmon1003 (servermon) - upgrading kernel (jessie), schedule icinga downtime, rebooting, short downtime for servermon itself
23:31 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Parser: Switch from Tidy to Remex on nowiki T177989 (duration: 00m 50s)
23:27 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Parser: Switch from Tidy to Remex on fawiki T176150 (duration: 00m 50s)
23:23 thcipriani@tin: Synchronized php-1.31.0-wmf.3/extensions/VisualEditor/modules/ve-mw/init/styles/ve.init.MWVESwitchConfirmDialog.css: SWAT: Fix WikiEditor mode switcher widget (duration: 00m 50s)
23:19 thcipriani@tin: Synchronized php-1.31.0-wmf.2/includes/specials/SpecialUnblock.php: SWAT: Fix unblocking autoblocks T177952 (duration: 00m 50s)
23:18 mutante: releases1001 upgrading kernel, installed linux-image-amd64, took out of cache::misc, upgrade libnss3, scheduled downtime, rebooting ...
23:17 thcipriani@tin: Synchronized php-1.31.0-wmf.3/includes/specials/SpecialUnblock.php: SWAT: Fix unblocking autoblocks T177952 (duration: 00m 51s)
22:01 volans: uploaded cumin_1.2.2-1_amd64.deb to apt.wikimedia.org jessie-wikimedia
21:44 robh: cp4009 shutdown accidental, will boot back up immediately
21:13 mutante: releases2001 - scheduled downtime, rebooting
21:07 mutante: running puppet on cache misc hosts to temp remove codfw backend of releases.wm.org, then rebooting releases2001 for kernel upgrade
20:22 subbu: updated Parsoid to ddf7b293 T138492 , T135667 , T177612 )
20:18 ssastry@tin: Finished deploy [parsoid/deploy@938e305]: Updating Parsoid to ddf7b293
20:17 hashar@tin: Finished deploy [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044 (duration: 02m 54s)
20:14 hashar@tin: Started deploy [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044
20:13 hashar@tin: Finished deploy [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044 (duration: 00m 02s)
20:13 hashar@tin: Started deploy [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044
20:11 hashar@tin: Started restart [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044
20:10 bsitzmann@tin: Finished deploy [mobileapps/deploy@15a52ba]: Update mobileapps to c045afc T177301 ) (duration: 05m 25s)
20:09 ssastry@tin: Started deploy [parsoid/deploy@938e305]: Updating Parsoid to ddf7b293
20:07 hashar@tin: Started restart [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044
20:05 bsitzmann@tin: Started deploy [mobileapps/deploy@15a52ba]: Update mobileapps to c045afc T177301 )
20:04 hashar@tin: Started restart [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044
20:01 chasemp: disable puppet around cloud things to roll refactor out slowly
19:29 mutante: releases1001 - same as 2001, upgraded jenkins to 2.73.2, kept existing config (T177962 )
19:27 mutante: releases2001 - upgraded jenkins to 2.73.2, kept existing config (vs overwriting with package config)
19:20 mutante: apt: reprepro copy stretch-wikimedia jessie-wikimedia jenkins
19:16 mutante: releases1001/releases2001 - apt-get autoremove to drop non-required python packages
19:03 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.3
19:02 demon@tin: Synchronized php: symlink swap (duration: 00m 49s)
18:47 gehel: killing stuck osm replication on maps-test2001
18:23 demon@tin: Synchronized wmf-config/Wikibase-labs.php: no-op, labs only (duration: 00m 59s)
18:19 demon@tin: Finished scap: wmf.3 bootstrap (duration: 44m 56s)
17:58 herron: ran /home/faidon/update-netboot-stretch.sh on puppetmaster1001
17:49 moritzm: uploaded jenkins 2.73.2 for jessie-wikimedia to apt.wikimedia.org
17:34 demon@tin: Started scap: wmf.3 bootstrap
17:32 demon@tin: Synchronized php: symlink swap to wmf.2, somehow got missed (duration: 00m 45s)
17:29 demon@tin: Synchronized scap/plugins/prep.py: No-op (duration: 01m 49s)
17:09 demon@tin: Pruned MediaWiki: 1.30.0-wmf.17 (duration: 02m 20s)
17:06 demon@tin: Pruned MediaWiki: 1.30.0-wmf.16 (duration: 02m 40s)
16:54 demon@tin: Pruned MediaWiki: 1.31.0-wmf.1 [keeping static files] (duration: 01m 44s)
16:24 hasharAway: Upgrade jenkins Maven integration plugin to 3.0 - T177962
16:15 godog: roll-restart thumbor to apply https://gerrit.wikimedia.org/r/380942 - T150734
16:12 hasharAway: Upgrading Jenkins CI T177962
16:09 gehel: decommission maps-test2004 from its cassandra cluster (free a node to test vector tiles) - T153282
16:09 XioNoX: resuming PDU upgrade
15:39 elukey: drop tables listed in https://phabricator.wikimedia.org/T171629#3674250 from db1046, db1047, dbstore1002
15:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1086 - T174509 (duration: 00m 47s)
15:21 _joe_: stopped redis on ocg* T177931
15:08 _joe_: ocg*: stop ocg; mv /srv/deployment /srv/stale, update-rc.d ocg disable, rm /etc/init/ocg.conf - T177931
14:55 _joe_: manually removing IPVS entries for ocg on eqiad LBs, T177931
14:47 _joe_: rolling restart of low-traffic pybals in eqiad for T177931
14:36 moritzm: installing dbus update from stretch 9.2 point release
13:40 hashar: European SWAT completed
13:39 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Remove compact language links dblist for simplicity (duration: 00m 46s)
13:38 hashar@tin: Synchronized docroot: Remove compact language links dblist for simplicity (duration: 00m 48s)
13:37 hashar@tin: Synchronized dblists: Remove compact language links dblist for simplicity (duration: 00m 47s)
13:36 hashar@tin: Synchronized wmf-config/CommonSettings.php: Remove compact language links dblist for simplicity (duration: 00m 47s)
13:29 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Configure wmgBabelMainCategory for the Dinka Wikipedia (duration: 00m 47s)
13:27 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable mapframe on eswiki - T177695 (duration: 00m 47s)
13:24 hashar@tin: Synchronized wmf-config/Wikibase-production.php: Enable constraint checks on qualifiers and references - T176863 (duration: 00m 47s)
13:21 moritzm: installing db5.3 security updates
13:03 godog: test graphite-web patch on labmon1001 - T177747
12:54 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Disable Statement usage tracking on cawiki (T151717 ) (duration: 00m 47s)
12:53 marostegui: Start recentchanges purge on s4 primary master T177772
12:48 marostegui: Kill recentchanges purge on s4 primary master - https://phabricator.wikimedia.org/T177772
12:44 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: (temp) Disable Statement usage tracking on cawiki (T151717 ) (duration: 00m 48s)
12:38 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable Statement usage tracking on cawiki and cewiki (T151717 ) (duration: 00m 47s)
12:32 hoo@tin: Synchronized wmf-config/: Move WB client "disabledUsageAspects" setting into $wmgWikibaseDisabledUsageAspects (duration: 00m 48s)
12:30 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Introduce $wmgWikibaseDisabledUsageAspects (duration: 00m 47s)
12:27 hoo@tin: scap aborted: wmf-config/InitialiseSettings.php Introduce $wmgWikibaseDisabledUsageAspects (duration: 02m 34s)
12:25 hoo@tin: Started scap: wmf-config/InitialiseSettings.php Introduce $wmgWikibaseDisabledUsageAspects
11:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1087 - T174509 (duration: 00m 50s)
11:49 ema: upgrading cp3007 to Varnish 5 T177233
11:34 moritzm: installing ruby1.9 security updates on trusty hosts
11:19 akosiaris: re-enable OTRS stats group T176221
11:11 jynus: starting purge of ruwiki.recentchanges T177772
10:55 akosiaris: OTRS upgrade done T176221
10:44 akosiaris: starting OTRS upgrade T176221
10:24 jynus: stopping dbstore2001:s3 for maintenance T177908
10:19 moritzm: installing curl security updates
09:32 moritzm: installing libxfont security updates
09:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1087 - T174509 (duration: 00m 46s)
09:27 marostegui: Optimize pagelinks and templatelinks on db1087 - T174509
09:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1082 - T174509 (duration: 00m 47s)
09:04 moritzm: installing curl security updates on app server canaries (along with HHVM restart)
08:50 moritzm: installing ffmpeg security updates
08:38 elukey: reboot kafka-jumbo hosts for kernel updates
08:37 marostegui: Stop replication in sync on db1103 and db1038 to checksum their data - T164488
08:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1103 - T164488 (duration: 00m 47s)
07:46 jynus: restart commonswiki recentchanges purging T177772
07:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1072 into the vslow and dump service for s3 - T172679 (duration: 00m 46s)
07:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1086 - T174509 (duration: 00m 47s)
07:02 marostegui: Optimize templatelinks and pagelinks on db1086 - T174509
06:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1082 - T174509 (duration: 00m 47s)
06:57 marostegui: Optimize templatelinks and pagelinks on db1082 - T174509
06:14 marostegui: Set db1068 s4 primary master to read-only OFF - T168661
06:14 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Set s4 to writable mode after maintenance - T168661 (duration: 00m 47s)
06:07 marostegui: Deploy alter table on s4 primary master - T168661
06:01 marostegui: Set read-only on db1068 s4 primary master - T168661
06:00 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Set s4 in read-only for maintenance - T168661 (duration: 00m 47s)
05:48 marostegui: Package update on db1068 s4 primary master - T168661
05:35 marostegui: Disable puppet on db1068 (s4 primary master) - T168661
05:35 marostegui: Disable puppet on db1068 (s4 primary master)
05:13 marostegui: Disable alerts on s4 for 2 hours - T168661
05:06 marostegui: Dump buffer pool on s4 primary master db1068 - T168661
05:03 kartik@tin: Finished deploy [cxserver/deploy@a79b38c]: Update cxserver to 273b515
05:00 kartik@tin: Started deploy [cxserver/deploy@a79b38c]: Update cxserver to 273b515
02:32 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Oct 11 02:32:46 UTC 2017 (duration 6m 39s)
02:26 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 08m 52s)
23:53 thcipriani@tin: Synchronized wmf-config: SWAT: Disable OCG services PART II T177795 (duration: 00m 48s)
23:52 thcipriani@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Disable OCG services PART I T177795 (duration: 00m 47s)
23:41 thcipriani@tin: Synchronized php-1.31.0-wmf.2/extensions/Collection/RenderingAPI.php: SWAT: Do not request render if renderer not configured T177795 (duration: 00m 47s)
23:39 XioNoX: upgrading ps1-a2-eqiad - T175341
23:35 thcipriani@tin: Synchronized wmf-config/CommonSettings.php: SWAT: revert Enable Vector print logo on test wiki (duration: 00m 47s)
23:21 thcipriani@tin: scap failed: average error rate on 3/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details)
23:12 thcipriani@tin: Synchronized wmf-config/CommonSettings.php: SWAT: wikitech: Align "contentadmin" and "sysop" permissions T171208 (duration: 00m 48s)
23:08 eileen1: ran the group clean up script (reduced smart groups)
22:28 eileen1: changed civicrm smart group cache timeout to 5 civicrm/admin/setting/search?reset=1
21:15 bblack: esams repooling - T167840
19:30 krinkle@tin: Synchronized docroot/noc/: Clean up noc base.css (duration: 00m 48s)
19:07 krinkle@tin: Synchronized docroot/noc/: Clean up noc/index CSS (duration: 00m 47s)
19:00 XioNoX: starting the work for T167840
18:41 mutante: logstash*: delete /usr/lib/ganglia/python_modules/elasticsearch_monitoring.py and /etc/ganglia/conf.d/elasticsearch.pyconf via cumin (T177225 )
17:32 XioNoX: depooling esams from DNS for T167840
17:26 gehel: shutting down and restarting elasticsearch on relforge1001 for testing - T170378
17:25 krinkle@tin: Synchronized php-1.31.0-wmf.2/resources/lib/jquery.ui/: I717f2580e3aae
16:19 milimetric@tin: Finished deploy [analytics/refinery@f4a0a33]: Deploying mostly for the new script in the bin folder (duration: 16m 01s)
16:03 milimetric@tin: Started deploy [analytics/refinery@f4a0a33]: Deploying mostly for the new script in the bin folder
15:48 jynus: starting purge of commonswiki.recentchanges T177772
15:39 thcipriani@tin: Synchronized README: noop deployment to test new logstash-checker.py host (duration: 00m 47s)
15:27 jynus: upgrading neodymium and sarin to mariadb-client 10.1.28
15:23 ema: cp1008 upgraded to varnish 5 T168529
15:00 godog: test stretch 9.2 upgrade on ms-fe2005 - T177739
14:59 cmjohnson1: lvs1007 swapping NIC card
14:53 godog: test stretch 9.2 upgrade on ms-be2013 - T177739
14:45 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1079 - T174509 (duration: 00m 47s)
14:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1080 - T174509 (duration: 00m 47s)
14:31 marostegui: Optimize table ores_classifications on db1080 - T159753
14:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1072 with low weight - T172679 (duration: 00m 47s)
14:24 aude@tin: Synchronized wmf-config/throttle.php: Add throttle rule T177835 (duration: 00m 47s)
14:22 elukey: add druid public cluster's IPs to analytics-in4 on cr1/cr2 - T177511
13:56 aude@tin: Synchronized wmf-config/throttle.php: Add throttle rule T177835 (duration: 02m 16s)
13:37 aude@tin: Synchronized wmf-config/Wikibase-production.php: Remove test wiki echo configs from Wikibase config (duration: 01m 31s)
13:20 aude@tin: Synchronized wmf-config/InitialiseSettings.php: Enable SandboxLink on gawiki (duration: 01m 50s)
13:11 gehel: restarting rsyslog on mw1180 - T177833
11:32 moritzm: downgrading HHVM on deployment-mediawiki04 to HHVM 3.18.2 temporarily (to further narrow down a problem with the new wikidiff package)
11:31 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1076 - T174509 (duration: 05m 02s)
11:31 dcausse: upgrading relforge to elastic 5.5.2
11:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1071 - T174509 (duration: 01m 21s)
10:33 akosiaris: T177511 switch druid100[456] to private1-x-eqiad VLANs
10:26 Amir1: start of cleaning up ores_classification in wikidatawiki (T159753 )
08:07 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2080 to the config - T170662 (duration: 00m 46s)
08:06 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2080 to the config - T170662 (duration: 00m 47s)
07:04 marostegui: Stop MySQL on db2079 to clone db2080 - https://phabricator.wikimedia.org/T170662
06:35 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s3 - T153033
06:27 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s1 - T153033
06:22 marostegui: Stop MySQL on db1038 to transfer its data to db1072 - T172679
06:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1038 - T172679 (duration: 00m 47s)
06:05 marostegui: Stop MySQL on db1072 to move it to s3 - T172679
05:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1079 - T174509 (duration: 00m 47s)
05:53 marostegui: Optimize templatelinks and pagelinks on db1079 - T174509
05:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1071 - T174509 (duration: 00m 48s)
05:45 marostegui: Optimize templatelinks and pagelinks on db1071 - T174509
05:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1076 - T174509 (duration: 00m 47s)
05:38 marostegui: Optimize templatelinks and pagelinks on db1076 - T174509
05:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1080 - T174509 (duration: 00m 48s)
05:32 marostegui: Optimize templatelinks and pagelinks on db1080 - T174509
02:35 eileen2: update civicrm from 00da4b0 47de976
02:32 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Oct 10 02:32:57 UTC 2017 (duration 6m 41s)
02:26 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 08m 49s)
01:21 eileen2: update civicrm from 276cbfd 00da4b0
22:18 hoo: Removed empty left-over xmldatadumps/public/other/wikibase/wikidatawiki/20171007 (https://dumps.wikimedia.org/wikidatawiki/entities/20171007/ )
19:53 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable new print styles on Vector in test wiki - T169732 (duration: 00m 47s)
19:51 hashar: morning swat completed
19:50 hashar@tin: Synchronized php-1.31.0-wmf.2/extensions/Echo/maintenance/updatePerUserBlacklist.php: Sync a live hack in Echo updatePerUserBlacklist https://gerrit.wikimedia.org/r/#/c/383182/ - T173475 (duration: 00m 47s)
19:48 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable new print styles on Vector in test wiki - T169732 (duration: 00m 47s)
19:22 hashar@tin: Synchronized wmf-config/CommonSettings.php: REVERT: Enable new print styles on Vector - T169732 (duration: 00m 49s)
19:19 hashar@tin: scap failed: average error rate on 8/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details)
19:17 hashar@tin: Synchronized php-1.31.0-wmf.2/extensions/Echo: Reapply "Use User Ids instead of User Names for Echo Mute"" - T173475 (duration: 00m 52s)
18:57 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable NewUserMessage for SUL accounts too on hi.wikiversity - T177690 (duration: 00m 47s)
18:52 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable NewUserMessage for SUL accounts too on dty.wiki - T177688 (duration: 00m 47s)
18:45 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Add autopatrolled user group to sd.wikipedia - T177141 (duration: 00m 47s)
18:00 ottomata: stopping druid services on druid1004 (not 1006)
17:59 ottomata: stopping druid services on druid1006
17:28 ottomata: stopping druid services on druid1005
16:34 ottomata: stopping druid services on druid1006
16:34 ottomata: beginning decom and reinstall process for druid1004-1006 -- T176223
15:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1091 - T174509 (duration: 00m 47s)
13:53 zeljkof: EU SWAT finished
13:53 zfilipin@tin: Synchronized php-1.31.0-wmf.2/extensions/Translate/: SWAT: Revert "[tech-debt] Remove usage of FuzzyLikeThis in favor of simple fuzzy match" (T177727) (duration: 00m 57s)
13:27 zfilipin@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Disallow most file types from upload to enwikivoyage (T176647) (duration: 00m 47s)
13:26 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Disallow most file types from upload to enwikivoyage (T176647) (duration: 00m 47s)
13:13 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix amwikimedia site name (T176042) (duration: 00m 47s)
13:10 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: New throttle rule (T177737) (duration: 00m 47s)
12:47 ema: restart pybal on eqiad load balancers to pick up bgp-med config change T165584
12:38 ema: restart pybal on codfw load balancers to pick up bgp-med config change T165584
12:31 ema: restart pybal on ulsfo load balancers to pick up bgp-med config change T165584
12:26 ema: restart pybal on esams load balancers to pick up bgp-med config change T165584
11:59 joal@tin: Finished deploy [analytics/refinery@c3812c2]: Analytics regular weekly deploy (duration: 10m 24s)
11:49 joal@tin: Started deploy [analytics/refinery@c3812c2]: Analytics regular weekly deploy
11:42 hashar: Upgrading Jenkins - T168644
11:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1083 - T174509 (duration: 00m 46s)
11:22 moritzm: installing apt updates from stretch 9.2 point release
11:22 marostegui: Optimize ores_classification table on db1083 - T159753
11:19 reedy@tin: Synchronized wmf-config/Wikibase-production.php: Actually disable injecting RC everywhere but commonswiki and ruwiki (duration: 00m 46s)
11:11 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1092 - T174509 (duration: 00m 47s)
11:08 moritzm: installing whois updates from stretch 9.2 point release
11:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1090 - T174509 (duration: 00m 46s)
10:54 moritzm: installing expect updates from stretch 9.2 point release
10:50 ladsgroup@tin: Synchronized wmf-config/Wikibase-production.php: Disable injecing RC records for commonswiki and ruwiki (T171027 ) (duration: 00m 47s)
10:38 ladsgroup@tin: Synchronized php-1.31.0-wmf.2/extensions/Wikidata/extensions/Wikibase/client: Re-instate $wgWBClientSettings[injectRecentChanges] (T171027 ) (duration: 00m 56s)
10:35 hoo: Started dumpwikidatajson.sh on snapshot1007 after T169680 disruptions
10:30 volans: rebooting snapshot1007 for NFS stuck - T169680
10:21 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s7 - T153033
10:21 volans: rebooting snapshot1006 for NFS stuck - T169680
10:15 elukey: rebooting snapshot1001 for NFS stuck - T169680
10:00 volans: rebooting snapshot1005 for NFS stuck - T169680
09:51 volans: rebooting dataset1001 for NFS stuck - T169680
09:51 ema: test setting bgp med on lvs3001/3003 T165584
09:50 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s5 - T153033
09:43 hashar: restarting Jenkins. Deadlock in SSHSlave plugin that causes memory to leak quite rapidly - T177749
09:34 moritzm: installing vim security updates
09:02 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2075 - T170662 (duration: 00m 47s)
08:56 volans: disabled puppet on 'stat100[5-6]*,snapshot100[1,5-7]*,dataset1001*' to manually recover from nfsd stuck - T169680
08:52 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s4 - T153033
08:51 marostegui: Optimize pagelinks and templatelinks on db1092 - T174509
08:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1092 - T174509 (duration: 00m 47s)
08:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1096 - T174509 (duration: 00m 47s)
08:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1066 on s1 to and move db1072 to s3 instead - T172679 (duration: 00m 47s)
08:12 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Move db1066 from s1 to s3 - T172679 (duration: 01m 25s)
07:33 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s2 - T153033
07:16 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2079 to the config - T170662 (duration: 00m 47s)
07:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2079 to the config - T170662 (duration: 00m 47s)
07:01 ema: cp4022 - backend restart, mailbox lag, cache_upload
06:56 marostegui: Stop MySQL on db2075 to use it to clone db2079 - T170662
06:56 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2075 - T170662 (duration: 00m 46s)
06:25 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s6 - T153033
06:14 marostegui: Optimize pagelinks and templatelinks on db1039 - T174509
06:11 marostegui: Optimize pagelinks and templatelinks on db1050 - T174509
06:08 marostegui: Optimize pagelinks and templatelinks on db1096 - T174509
06:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1096 - T174509 (duration: 00m 47s)
05:59 marostegui: Optimize pagelinks and templatelinks on db1091 - T174509
05:59 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1091 - T174509 (duration: 00m 47s)
05:53 marostegui: Optimize pagelinks and templatelinks on db1090 - T174509
05:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1090 - T174509 (duration: 00m 47s)
05:44 marostegui: Optimize pagelinks and templatelinks on db1083 - T174509
05:38 marostegui: Stop MySQL on db1083 to upgrade it
05:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1083 - T174509 (duration: 00m 47s)
02:48 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Oct 9 02:48:21 UTC 2017 (duration 6m 54s)
02:41 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 13m 49s)
21:40 hoo: Killed "dumpwikidatardf.sh truthy nt" (wikidata truthy dump) on snapshot1007, got stuck after T169680 .
11:28 volans: ack-ed puppet not running on stat100[5-6],snapshot100[1,5-7] due to NFSD stuck on dataset1001 - T169680
08:19 elukey: restart varnish backend on cp4026 to stop 503s
22:46 reedy@tin: Synchronized php-1.31.0-wmf.2/resources/: T177705 (duration: 00m 48s)
22:44 reedy@tin: Synchronized php-1.31.0-wmf.2/includes/specials/SpecialBlock.php: T177705 (duration: 00m 49s)
18:58 ejegg: updated CiviCRM from 197b22d 276cbfd
18:38 catrope@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/: Unbreak preference changes (T177524 ) (duration: 00m 47s)
18:23 arlolra@tin: Finished deploy [parsoid/deploy@e437c93]: Updating Parsoid to 772e11bf
18:10 arlolra@tin: Started deploy [parsoid/deploy@e437c93]: Updating Parsoid to 772e11bf
15:30 moritzm: repooling mw2145 (API server). it was previously depooled, but there's no tickets or SAL entries indicating hardware maintenance, so it was probably simply forgotten to repool it
15:25 moritzm: repooling mw2203 (API server). it was previously depooled, but there's no tickets or SAL entries indicating hardware maintenance, so it was probably simply forgotten to repool it
12:54 moritzm: repooling mw2248-mw2250 (job runners). they were previosly depooled, but there's no tickets or SAL entries indicating hardware maintenance, so they were probably simply forgotten to repool
11:46 bblack: cp1* (eqiad caches) - upgrade to nginx-1.13.5-1+wmf1~jessie1 to match all the other sites
11:34 moritzm: repooling mw2244/mw2245 (image scalers). they were previosly depooled, but there's no tickets or SAL entries indicating hardware maintenance, so they were probably simply forgotten to repool
11:21 moritzm: installing gdk-pixbuf security updates
11:10 elukey: rolling restart of all the druid daemons on druid100[1-6] to pick up new logging changes
11:03 moritzm: installing git security updates on trusty (Debian already fixed)
10:51 moritzm: upgrading HHVM on script runners to HHVM 3.18.5
10:47 moritzm: upgrade remaining video scalers in codfw to HHVM 3.18.5
10:43 elukey: restart replication (mysql+ eventlogging_sync) on dbstore1002 after mysql restart
10:17 moritzm: upgrade remaining job runners in codfw to HHVM 3.18.5
09:53 elukey: stop replication (eventlogging_sync + mysql) on dbstore1002 as prep step for mysql restart
09:17 hashar: Restarted the CI Jenkins for plugin upgrades
09:01 moritzm: upgrade remaining app servers in codfw to HHVM 3.18.5
08:40 moritzm: upgrade remaining image scalers in codfw to HHVM 3.18.5
08:09 ema: upgrade pybal to 1.14.0 on eqiad primary LVSs
07:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1076 original weight - T174054 (duration: 00m 48s)
07:23 ema: upgrade pybal to 1.14.0 on eqiad secondary LVSs
07:18 moritzm: upgrade remaining API servers in codfw to HHVM 3.18.5
07:11 legoktm: live hacking on mwdebug1002
07:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1076 weight - T174054 (duration: 00m 47s)
06:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1076 weight - T174054 (duration: 00m 47s)
06:35 ema: set max_connections=0 on cp3032 varnish-be
06:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1106 to the config - T172679 (duration: 00m 47s)
06:21 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1106 to the config - T172679 (duration: 00m 47s)
06:00 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1076 with low weight - T174054 (duration: 00m 46s)
05:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1053 - T174509 (duration: 00m 48s)
05:35 marostegui: Stop MySQL on db1076 for an upgrade
23:40 ejegg: updated CiviCRM from cca1a9d 197b22d
23:37 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/resources/lib/jquery/jquery.migrate.js: sync https://gerrit.wikimedia.org/r/#/c/382530 and https://gerrit.wikimedia.org/r/#/c/382529 (duration: 00m 47s)
23:34 XioNoX: upgrading ps1-d2-eqiad.mgmt.eqiad.wmnet (unused)
23:23 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch/includes/BaseInterwikiResolver.php: sync https://gerrit.wikimedia.org/r/#/c/382625/ (duration: 00m 47s)
23:21 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT https://gerrit.wikimedia.org/r/#/c/382611/ (duration: 00m 47s)
22:47 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch/includes/: Sync CirrusSearch extension again for good measure (duration: 00m 51s)
22:43 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch: Sync CirrusSearch extension again for good measure (duration: 00m 57s)
22:39 twentyafterfour@tin: Synchronized wmf-config/CirrusSearch-common.php: sync https://gerrit.wikimedia.org/r/#/c/382474/ (duration: 00m 47s)
22:37 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch/: deploy https://gerrit.wikimedia.org/r/#/c/382618/ (duration: 00m 58s)
22:13 awight@tin: Finished deploy [ores/deploy@42c5663]: Cause ORES service restart (duration: 00m 19s)
22:13 awight@tin: Started deploy [ores/deploy@42c5663]: Cause ORES service restart
22:04 bd808@tin: Finished deploy [striker/deploy@fbb9019]: Prevent tools from being named with invalid Kubernetes namespace labels (T176681 ) (duration: 00m 33s)
22:04 bd808@tin: Started deploy [striker/deploy@fbb9019]: Prevent tools from being named with invalid Kubernetes namespace labels (T176681 )
21:35 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.31.0-wmf.2 refs T174358
21:32 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch: sync CirrusSearch to deploy https://gerrit.wikimedia.org/r/#/c/382605/ refs T177535 (duration: 01m 06s)
21:22 ejegg: updated CiviCRM from 59e3e00 cca1a9d
21:13 twentyafterfour: Getting the train back on track: moving to wmf.2 as soon as https://gerrit.wikimedia.org/r/#/c/382605/1 merges
20:47 twentyafterfour: 1.31.0-wmf.2 is now blocked by T177535
20:28 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.31.0-wmf.1 refs T174358
20:26 twentyafterfour: rolling back to 1.31.0-wmf.1
20:11 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.31.0-wmf.2 refs T174358
20:10 twentyafterfour: Promote all wikis to 1.31.0-wmf.2 refs T174358 (currently there are no blockers and no significant logspam)
19:59 thcipriani@tin: Synchronized php-1.31.0-wmf.1/extensions/Echo: SWAT: revert "Use User Ids instead of User Names for Echo Mute" (duration: 00m 55s)
19:58 andrewbogott: upgrading wikitech-static to REL1_30
19:55 thcipriani@tin: Synchronized php-1.31.0-wmf.2/extensions/Echo: SWAT: revert "Use User Ids instead of User Names for Echo Mute" (duration: 00m 52s)
19:49 thcipriani@tin: Synchronized php-1.31.0-wmf.2/extensions/VisualEditor/extension.json: SWAT: Move 'parentheses' message to MWSaveDialog's modules T177446 (duration: 00m 50s)
19:45 thcipriani@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict: SWAT: Dont register simulate page when not enabled as a BF (duration: 00m 52s)
19:05 thcipriani@tin: Synchronized php-1.31.0-wmf.1/extensions/Echo: SWAT: Use User Ids instead of User Names for Echo Mute T173475 (duration: 00m 55s)
18:58 thcipriani@tin: Synchronized php-1.31.0-wmf.2/extensions/Echo: SWAT: Use User Ids instead of User Names for Echo Mute T173475 (duration: 00m 56s)
18:53 bblack: cp3* upgrade nginx to 1.13.5-1+wmf1~jessie1
18:38 urandom: restbase-ng: lowering compactionthroughput to 4 (current/5 jbod devices)
18:26 thcipriani@tin: Synchronized wmf-config: SWAT: Enable structured change filters by default on all wikis except FlaggedRevs wikis using non-protection modes T177445 (duration: 00m 54s)
18:13 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable jQuery 3 on all wikis T124742 (duration: 00m 52s)
17:27 cmjohnson1: disable puppet db1024
16:53 moritzm: upgrade remaining app servers in eqiad to HHVM 3.18.5
16:16 XioNoX: failure test for pfw3-codfw
15:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1078 - T174054 (duration: 00m 50s)
15:25 marostegui: Pulling out one disk from db1078 - T174054
15:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1078 - T174054 (duration: 00m 51s)
15:16 marostegui: Pulling out one disk from db1076 - T174054
14:53 hashar: All Jenkins infra is now on Java 8 - T162828
14:52 marostegui: Sanitize db1095 for amwikimedia - T176043
14:48 hashar: Restarting Jenkins
14:27 moritzm: upgrade remaining eqiad video scalers to HHVM 3.18.5
13:57 ladsgroup@tin: Synchronized wmf-config/Wikibase-production.php: Enable Statement usage tracking on kowiki and trwiki (T151717 ) (duration: 00m 50s)
13:51 ladsgroup@tin: Synchronized wmf-config/Wikibase-production.php: (no justification provided) (duration: 00m 50s)
13:49 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Document task reference for amwikimedia logo (T176042 ) (duration: 00m 54s)
13:43 Amir1: ladsgroup@terbium:~$ mwscript createAndPromote.php --wiki=amwikimedia --createAndPromote=sysop,bureaucrat 'Ladsgroup' "password removed obviously" --force (T176042 )
13:35 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Fix amwikimedia logo (T176042 ) (duration: 00m 50s)
13:26 ladsgroup@tin: Synchronized php-1.31.0-wmf.2/extensions/WikimediaMessages/i18n/wikimediaprojectnames: (no justification provided) (duration: 00m 49s)
13:15 godog: test upgrade node-exporter in thumbor/swift/services/puppet3-diffs/monitoring projects - T166561
13:11 ladsgroup@tin: Synchronized wmf-config/interwiki.php: Create amwikimedia (T176042 ) (duration: 00m 51s)
13:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1089 - T174509 (duration: 00m 50s)
12:58 ladsgroup@tin: Synchronized multiversion/MWMultiVersion.php: Create amwikimedia (T176042 ) (duration: 00m 50s)
12:53 ladsgroup@tin: Synchronized static/images/project-logos/: Create amwikimedia (T176042 ) (duration: 00m 50s)
12:52 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Create amwikimedia (T176042 ) (duration: 00m 50s)
12:50 ladsgroup@tin: rebuilt wikiversions.php and synchronized wikiversions files: Create amwikimedia (T176042 )
12:49 ladsgroup@tin: Synchronized dblists: Create amwikimedia (T176042 ) (duration: 00m 50s)
12:36 ladsgroup@tin: Synchronized dblists: (no justification provided) (duration: 00m 51s)
12:22 moritzm: installing poppler security updates on trusty
11:57 moritzm: upgrade remaining eqiad video scalers to HHVM 3.18.5
11:29 jynus: rebuilding globalimagelinks table on dbstore1001:s4
10:35 hashar: contint2001: switched java to version 8 as well as all the jre/jdk utilities managed by alternatives - T162828
10:26 moritzm: upgrade remaining eqiad API servers to HHVM 3.18.5
10:23 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/includes/TwoColConflictHooks.php: REVERT: Dont register simulate page when not enabled as a BF PT1/3 (duration: 00m 50s)
10:22 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/extension.json: REVERT: Dont register simulate page when not enabled as a BF PT2/3 (duration: 00m 50s)
10:17 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/extension.json: Dont register simulate page when not enabled as a BF PT2/3 (duration: 00m 50s)
10:16 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/includes/TwoColConflictHooks.php: Dont register simulate page when not enabled as a BF PT1/3 (duration: 00m 51s)
10:14 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/includes/SpecialConflictTestPage/SpecialConflictTestPage.php: Fix SimulateTwoColEditConflict when using a page with ns (duration: 00m 50s)
10:09 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/WikimediaEvents/WikimediaEventsHooks.php: 31-wmf.2 WikimediaEvents: WMDE: update $campaignStartTimestamp (duration: 00m 50s)
10:07 addshore@tin: Synchronized php-1.31.0-wmf.1/extensions/WikimediaEvents/WikimediaEventsHooks.php: 31-wmf.1 WikimediaEvents: WMDE: update $campaignStartTimestamp (duration: 00m 52s)
10:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1076 - T174054 (duration: 00m 50s)
09:52 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1099 - T174509 (duration: 00m 50s)
08:54 marostegui: Stop MySQL on db1104 to clone db1106 - T172679
08:53 gehel: rolling restart of wdqs to pick up new GC options - T175919
08:41 moritzm: upgrade remaining eqiad job runners to HHVM 3.18.5
08:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1105 to the config - T172679 (duration: 00m 50s)
08:39 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1105 to the config - T172679 (duration: 00m 52s)
08:29 marostegui: Optimize pagelinks and templatelinks on s5 - db1105 - T174509
08:21 moritzm: upgrade remaining eqiad image scalers to HHVM 3.18.5
08:12 marostegui: Optimize templatelinks and pagelinks on s1 s2 s4 s5 s6 and s7 on labsdb1009 - T174509
08:03 jynus: putting labsdb1009 under maintenance
07:54 moritzm: upgrade mw1221-mw1235 (API servers) to HHVM 3.18.5
07:26 legoktm: running batched query of "DELETE FROM linter WHERE linter_cat=12" on all wikis to clear out mostly bogus html5-misnesting category
07:11 ema: update pybal to 1.14.0 on esams primary LVSs
07:06 ema: update pybal to 1.14.0 on esams secondary LVSs
07:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1099 - T174509 (duration: 00m 52s)
07:01 marostegui: Optimize templatelinks and pagelinks on db1099 - T174509
06:56 moritzm: upgrade mw1238-mw1258 (app servers) to HHVM 3.18.5
06:40 marostegui: Stop MySQL on db1104 to clone db1105 from it - T172679
06:32 marostegui: Optimize templatelinks and pagelinks on db1069 - T174509
06:31 marostegui: Optimize templatelinks and pagelinks on db1053 - T174509
06:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1053 - T174509 (duration: 00m 50s)
06:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1089 - T174509 (duration: 00m 49s)
06:24 marostegui: Optimize templatelinks and pagelinks on db1089 - T174509
06:17 marostegui: Optimize s2.templatelinks and enwiki.pagelinks on codfw master db2017 (without replication) - T174509
06:13 marostegui: Optimize enwiki.templatelinks and enwiki.pagelinks on codfw master db2048 (without replication) - T174509
06:01 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2035 and db2056 - T174509 (duration: 00m 52s)
05:57 elukey: restart varnish backend on cp3040
05:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1097 - T174509 (duration: 01m 09s)
05:42 elukey: restart varnish backend on cp3030
02:59 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Oct 5 02:59:18 UTC 2017 (duration 7m 23s)
02:51 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 09m 01s)
02:29 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 09m 51s)
00:06 catrope@tin: Synchronized php-1.31.0-wmf.2/resources/src/mediawiki.rcfilters/mw.rcfilters.init.js: T176652 (duration: 00m 49s)
00:04 catrope@tin: Synchronized php-1.31.0-wmf.2/extensions/WikimediaEvents/modules/ext.wikimediaEvents.recentChangesClicks.js: T176652 (duration: 00m 50s)
23:55 eileen: update process_control to 10161f8
23:53 catrope@tin: Synchronized php-1.31.0-wmf.2/extensions/TimedMediaHandler/MwEmbedModules/EmbedPlayer/resources/jquery.embedPlayer.js: Fix jqmigrate warning (T169385 ) (duration: 00m 50s)
23:51 catrope@tin: Synchronized php-1.31.0-wmf.2/extensions/ORES/includes/Hooks.php: ORES highlights for RCFilters (T172757 ) (duration: 00m 50s)
23:49 catrope@tin: Synchronized php-1.31.0-wmf.2/resources/src/: ORES highlights for RCFilters (T172757 ) (duration: 00m 50s)
23:36 catrope@tin: Synchronized php-1.31.0-wmf.2/includes/changes/ChangesListFilter.php: ORES highlights for RCFilters (T172757 ) (duration: 00m 50s)
23:32 catrope@tin: Synchronized php-1.31.0-wmf.2/includes/changes/ChangesListFilterGroup.php: Fix PHP fatal when transcluding Special:Recentchanges (T176236 ) (duration: 00m 50s)
23:32 Krinkle: Upgrading xhgui on tungsten (operations/software/xhgui.git) from 0.5.2 to 0.8.1 (matching version used in operations/mediawiki-config vendor) - new feature: flamegraphs
23:29 catrope@tin: Synchronized wmf-config/throttle.php: Remove expired rules (duration: 00m 50s)
23:27 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable AbuseFilter runtime profile on ptwiki (T177336 ) (duration: 00m 51s)
23:19 catrope@tin: Synchronized static/images/mobile/copyright/wikipedia-wordmark-en.svg: Remove unnecessary id attributes (T175670 ) (duration: 00m 50s)
23:18 catrope@tin: Synchronized php-1.31.0-wmf.2/skins/MinervaNeue/: Correct feature phone threshold detection (T176286 ) (duration: 00m 53s)
22:35 ejegg: changed DonationInterface cURL timeout for all processors to 12 sec
21:09 arlolra: Updated Parsoid to 477942c3 T177115 )
21:07 hoo: Updated the Wikidata property suggester with data from Monday's JSON dump and applied the T132839 workarounds
21:03 arlolra@tin: Finished deploy [parsoid/deploy@617cdb3]: Updating Parsoid to 477942c3
20:53 arlolra@tin: Started deploy [parsoid/deploy@617cdb3]: Updating Parsoid to 477942c3
20:33 arlolra@tin: Finished deploy [parsoid/deploy@5b80254]: Updating Parsoid to 477942c3
20:32 arlolra@tin: Started deploy [parsoid/deploy@5b80254]: Updating Parsoid to 477942c3
20:18 pnorman: Start regenerating map tiles on codfw for z13-z14 - T176252
19:55 twentyafterfour@tin: Synchronized php: group1 wikis to 1.31.0-wmf.2 refs T174358 (duration: 00m 48s)
19:55 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.31.0-wmf.2 refs T174358
19:38 pnorman: Start regenerating map tiles on eqiad for z13-z14 - T176252
19:25 twentyafterfour: Deploying MediaWiki 1.31.0-wmf.2 to Group 1 wikis refs T174358
19:06 bblack: cp2* (all codfw caches): upgrade nginx to 1.13.5-1+wmf1~jessie1
18:39 ottomata: reverting roll out of LVS service druid-analytics.svc.eqiad.wmnet - this wont' work with hosts inside of Analytics VLAN
18:23 niharika29@tin: Synchronized php-1.31.0-wmf.2/skins/Vector/: Do not special-case ULS and Not logged in in RTL in personal bar T48947 , T177312 (duration: 00m 50s)
18:22 niharika29@tin: Synchronized php-1.31.0-wmf.1/skins/Vector/: Do not special-case ULS and Not logged in in RTL in personal bar T48947 , T177312 (duration: 00m 51s)
18:09 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable jQuery 3 on most group1 wikis (T124742 ) (duration: 00m 51s)
17:37 elukey: enabled basic ACLs on the Kafka Jumbo cluster - T173493
17:32 bblack: text@ulsfo - upgrade nginx to 1.13.5-1+wmf1~jessie1
17:32 ottomata: deploying new LVS service for druid-analytics-broker
17:16 moritzm: uploaded icu52 to stretch-wikimedia (co-installable forward port of jessie's ICU for stretch to be used for HHVM)
17:06 bblack: cp4022-26 (now all of upload@ulsfo) - upgrade nginx to 1.13.5-1+wmf1~jessie1
16:58 krinkle@tin: Synchronized php-1.31.0-wmf.2/extensions/TimedMediaHandler/resources/mw.MediaWikiPlayer.loader.js: T175506 (duration: 00m 52s)
16:29 bblack: lvs1001-3: base package upgrades, autoremove, +python-requests
16:24 bblack: recdns: upgrade base packages
16:23 volans: deleted 'R:File = /usr/local/bin/wmf-reimage' on matching hosts T176955
16:20 bblack: maelant: upgrade base packages
16:11 bblack: lvs1004-6: base package upgrades, autoremove, +python-requests
16:06 jynus: performing schema change on labsdb1009/10/11
16:01 ejegg: disabled CiviCRM dedupe jobs
15:40 bblack: acamar (recdns@codfw) - base package upgrades
15:39 bblack: lvs1007-12: base package upgrades, autoremove, +python-requests
15:37 bblack: hydrogen (recdns@eqiad) - base package upgrades
15:35 marostegui: Power off db2010 to decommission it - T175685
15:35 bblack: recdns: apt autoremove -> install python-requests
15:32 bblack: caches: apt autoremove -> install python-requests
15:28 marostegui: Disable puppet on db2010 - it will be decommissioned - T175685
15:24 andrewbogott: rebuilding labvirt1016
14:57 ema: cp4025: restart varnish backend
14:47 XioNoX: replacing faulty VC cable on fasw-c-eqiad
14:45 bblack: base package upgrades (+autoremove) on lvs3001-2
14:43 bblack: base package upgrades (+autoremove) on lvs3004
14:41 bblack: base package upgrades (+autoremove) on lvs3003
14:24 bblack: upgrading nginx to 1.13.5-1+wmf1~jessie1 on cp4021 (first live cache, upload@ulsfo)
14:18 bblack: base package upgrades (+autoremove) on lvs2*
14:10 bblack: base package upgrades (+autoremove) on lvs4*
13:47 zeljkof: EU SWAT finished
13:46 zfilipin@tin: Synchronized php-1.31.0-wmf.2/extensions/Translate/tag/PageTranslationHooks.php: SWAT: Revert "Use Language type hint in hooks" (T177352) (duration: 00m 50s)
13:45 zfilipin@tin: Synchronized php-1.31.0-wmf.2/extensions/Translate/TranslateHooks.php: SWAT: Revert "Use Language type hint in hooks" (T177352) (duration: 00m 50s)
13:41 zfilipin@tin: Synchronized php-1.31.0-wmf.2/extensions/Translate/TranslateHooks.php: php-1.31.0-wmf.2/extensions/Translate/tag/PageTranslationHooks.php SWAT: Revert "Use Language type hint in hooks" (T177352) (duration: 00m 52s)
13:26 zeljkof: one more commit for EU SWAT
13:26 marostegui: Optimize table ores_classification on enwiki codfw master db2048 with replication - might generate lag - T159753
13:22 marostegui: Optimize enwiki.ores_classification on db2069
13:06 zeljkof: EU SWAT finished
13:05 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: New throttle rule (T177370) (duration: 00m 53s)
12:59 jynus: upgrade and reboot labsdb1009
12:56 marostegui: Optimize templatelinks and pagelinks tables on s2 and s7 - labsdb1010 - T174509
12:38 moritzm: upgrading app servers in codfw to HHVM 3.18.5
12:21 moritzm: upgrading image scalers mw1293-mw1295 to HHVM 3.18.5
12:12 moritzm: upgrading HHVM on deployment servers to 3.18.5
12:10 elukey: added two new mediawiki videoscalers - mw1307/1318
12:09 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1318.eqiad.wmnet
12:09 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1307.eqiad.wmnet
12:00 gehel: mediawiki now uses the LVS endpoint for logstash - T175242
11:49 moritzm: upgrading job runners mw1161-mw1167 to HHVM 3.18.5
11:34 moritzm: installing ghostscript security updates
11:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1087 original weight (duration: 00m 47s)
10:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1087 weight (duration: 00m 51s)
10:29 jynus: upgrade and reboot labsdb1011
10:22 jynus: draining labsdb1011 connections
10:18 akosiaris: T177374 , fully depool wtp1001-wtp1024
10:18 akosiaris@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=wtp10([01][0-9]|2[0-4]).eqiad.wmnet
10:03 moritzm: install libidn security updates
10:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1087 with low weight (duration: 00m 50s)
09:54 _joe_: restarting hhvm on canaries (both appservers and canaries) T107128
09:47 moritzm: upgrading API app servers mw1189-mw1208 to HHVM 3.18.5
09:45 elukey: rolling restart of aqs nodes to pick up the new logstash lvs config
09:39 hoo: Killed remaining dumpRdf runners on snapshot1007 after two crashed due to the db1087 depool. Auto-restart will do the rest.
09:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1104 to the config - T172679 (duration: 00m 50s)
09:32 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1104 to the config - T172679 (duration: 00m 51s)
09:25 moritzm: installing ocaml security updates on trusty
09:16 marostegui: Reboot db1087 to pick up new kernel
09:09 _joe_: restarting hhvm on mwdebug*, T107128
09:07 marostegui: Optimize templatelinks and pagelinks tables on db1097 (s4) - T174509
09:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1097 - T174509 (duration: 00m 50s)
08:41 akosiaris@puppetmaster1001: conftool action : set/pooled=no; selector: name=wtp10([01][0-9]|2[0-4]).eqiad.wmnet
08:40 mobrovac@tin: Finished deploy [restbase/deploy@8eb758a]: Switch the mobile feeds to the new storage schema and Cassandra 3 (duration: 10m 05s)
08:36 moritzm: upgrading app servers mw1180-mw1188, mw1209-mw1220 to HHVM 3.18.5
08:30 mobrovac@tin: Started deploy [restbase/deploy@8eb758a]: Switch the mobile feeds to the new storage schema and Cassandra 3
08:28 marostegui: Stop MySQL on db2010 as it will be decommissioned - T175685
07:30 jynus: stopping s5 replication on dbstore1002 and converting wikidatawiki.wb_terms into TokuDB
07:11 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db2010 from config as it will be decommissioned - T175685 (duration: 00m 48s)
07:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db2010 from config as it will be decommissioned - T175685 (duration: 00m 48s)
06:53 marostegui: Stop MySQL on db1087 to clone db1104 from it - T172679
06:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1087 - T172679 (duration: 00m 50s)
06:30 marostegui: Optimize templatelinks and pagelinks tables on db1066 - T174509
06:27 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2035 and db2056 - T174509 (duration: 00m 50s)
06:21 elukey: restart varnish backend on cp3043
06:19 marostegui: Optimize pagelinks and templatelinks on dbstore2002 s1 - T174509
06:16 elukey: restart varnish backend on cp3041
06:13 marostegui: Optimize pagelinks and templatelinks tables on s7 codfw master (db2029) this will generate lag - T174509
05:58 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2034, db2055 db2062 - T174509 (duration: 00m 51s)
03:09 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Oct 4 03:09:34 UTC 2017 (duration 7m 15s)
03:02 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 15m 20s)
02:25 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 07m 44s)
00:33 ejegg: re-enabled fundraising queue consumers
00:29 ejegg: re-enabled CiviCRM de-dupe jobs
00:24 ejegg: re-enabled civicrm cron and ingenico jobs
00:20 ejegg: re-enabled fundraising stats and export jobs
00:16 ejegg: re-enabled omnimail import jobs
00:10 ejegg: re-enabled fundraising audit jobs
00:07 ejegg: re-enabled pending queue consumer
23:52 ejegg: stopped all jobs for process-control upgrade
23:46 pnorman: Start regenerating map tiles on eqiad for z0-z12 - T176252
23:20 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T175971 T176088 Enable RemexHTML on wikitech and eswikiversity (duration: 00m 51s)
23:11 ebernhardson@tin: Synchronized wmf-config/CirrusSearch-common.php: Pre-configure cirrussearch for 5.5.x upgrade (duration: 00m 51s)
22:26 bsitzmann@tin: Finished deploy [mobileapps/deploy@82aa7d6]: Update mobileapps to 5dc0c02 T175762 T177001 T176525 T176517 T176519 ) (duration: 06m 14s)
22:20 bsitzmann@tin: Started deploy [mobileapps/deploy@82aa7d6]: Update mobileapps to 5dc0c02 T175762 T177001 T176525 T176517 T176519 )
20:29 pnorman: Start regenerating map tiles on codfw for z0-z12 - T176252
20:17 demon@tin: Synchronized php-1.31.0-wmf.2/extensions/GlobalUserPage/includes/Hooks.php: fix broken thing (duration: 00m 51s)
19:58 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.2
19:33 demon@tin: Synchronized scap/plugins/prep.py: no-op (duration: 00m 51s)
19:17 demon@tin: Finished scap: bootstrap wmf.2 (duration: 33m 50s)
19:01 twentyafterfour: Starting MediaWiki train for 1.30.1-wmf.2
18:43 demon@tin: Started scap: bootstrap wmf.2
18:25 chasemp: change to private vlan for labweb1001 on asw-b-eqiad
18:24 chasemp: change to private vlan for labweb1002 on asw2-d-eqiad
18:24 chasemp: auth-dnsupdate on bahan for 382027
16:51 ejegg: updated payments-wiki from c2a3d43 b9859f6
15:57 bblack: upgrading basic packages on role::cache::text
15:50 bblack: upgrading basic packages on role::cache::upload
15:42 bblack: upgrading basic packages on role::cache::misc
15:29 mobrovac@tin: Finished deploy [restbase/deploy@02acc45]: (no justification provided) (duration: 06m 27s)
15:24 _joe_: uploading blubber to jessie-wikimedia T175296
15:23 mobrovac@tin: Started deploy [restbase/deploy@02acc45]: (no justification provided)
15:04 mobrovac@tin: Finished deploy [restbase/deploy@02acc45]: (no justification provided) (duration: 05m 14s)
14:59 mobrovac@tin: Started deploy [restbase/deploy@02acc45]: (no justification provided)
14:56 mobrovac@tin: Finished deploy [restbase/deploy@02acc45]: (no justification provided) (duration: 04m 25s)
14:56 bblack: upgrading basic packages on cp1065
14:52 mobrovac@tin: Started deploy [restbase/deploy@02acc45]: (no justification provided)
13:56 zeljkof: EU SWAT finished
13:55 zfilipin@tin: Synchronized php-1.31.0-wmf.1/extensions/Translate/webservices/CxserverWebService.php: SWAT: Make CxserverWebService forward compatible (duration: 00m 45s)
13:46 godog: upgrade grafana to 4.5.2 on krypton - T175980
13:41 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Adjust wgNamespacesToBeSearchedDefault for enwikibooks, fawikibooks and hewikisource (T176906 T176908 T176907) (duration: 00m 46s)
13:34 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Limit thanks for new users at pl.wikipedia to 3 per day" (T176174 T169268) (duration: 00m 46s)
13:26 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ShortUrl Extension on hiwikiversity (T177187) (duration: 00m 46s)
13:25 marostegui: Optimize templatelinks and pagelinks tables on s1, s4 and s6 on labsdb1010 - T174509
13:13 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable NewUserMessage Extension on hiwikiversity (T177188) (duration: 00m 46s)
13:04 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2044 - T174764 (duration: 00m 47s)
10:50 mobrovac@tin: Finished deploy [restbase/deploy@1cc530b]: (no justification provided) (duration: 08m 38s)
10:41 mobrovac@tin: Started deploy [restbase/deploy@1cc530b]: (no justification provided)
08:10 Amir1: cleaning up ores_classification tables (T159753 )
08:05 marostegui: Deplo alter table on s3 codfw master (db2018) to add PK to the main tables - T163912
08:02 mobrovac@tin: Finished deploy [restbase/deploy@65f519b]: Create new buckets for feed end points (duration: 09m 57s)
07:52 mobrovac@tin: Started deploy [restbase/deploy@65f519b]: Create new buckets for feed end points
07:20 gehel: killing stuck tileshell and cleaning up /srv/osm_expire on maps-test2001 - T175123
07:05 elukey: restart varnish backend on cp3031 (503s)
07:03 marostegui: Optimize pagelinks and templatelinks tables on labsdb1010 for s5 - T174509
07:01 marostegui: Optimize templatelinks and pagelinks tables on s6 codfw master db2028 (this might generate lag) - T174509
06:46 marostegui: Drop now redundant indexes from pagelinks and templatelinks on s7 - T174509
06:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1066 - T172679 (duration: 00m 47s)
05:54 marostegui: Optimize templatelinks and pagelinks tables on s5 master codfw (db2023): this will generate lag - T174509
05:51 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2049 and db2041 - T174509 (duration: 00m 47s)
03:23 demon@tin: Pruned MediaWiki: 1.30.0-wmf.19 [keeping static files] (duration: 01m 30s)
02:30 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Oct 3 02:30:30 UTC 2017 (duration 6m 39s)
02:23 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 07m 22s)
23:38 catrope@tin: Synchronized php-1.31.0-wmf.1/extensions/VisualEditor/: T177250 (duration: 00m 47s)
23:36 catrope@tin: Synchronized php-1.31.0-wmf.1/resources/src/mediawiki.rcfilters/dm/mw.rcfilters.dm.FiltersViewModel.js: T177107 (duration: 00m 47s)
23:33 mutante: releases2001 - kill and restart jenkins with systemctl
23:19 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Update WMF mailing address (T173684 ) and enable jQuery 3 on Wiktionaries (T124742 ) (duration: 00m 48s)
23:00 mutante: releases1001 - remove and reinstall jenkins
22:56 Reedy: created newsletter tables on officewiki T176199
19:36 ejegg: turned on CiviMail record creation
18:48 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Change the Turkish Wiktionary logo (T176008 ) (duration: 00m 46s)
18:46 niharika29@tin: Synchronized static/images/project-logos/: Change the Turkish Wiktionary logo (T176008 ) (duration: 00m 48s)
17:46 gehel@tin: Finished deploy [wdqs/wdqs@f20b726]: latest wdqs GUI + minor fixes (duration: 01m 48s)
17:44 gehel@tin: Started deploy [wdqs/wdqs@f20b726]: latest wdqs GUI + minor fixes
14:08 zeljkof: EU SWAT finished
14:07 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Sandbox menu to Javanese Wikipedia (T176308) (duration: 00m 46s)
14:01 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Extension:Newsletter on officewiki (T176199) (duration: 00m 46s)
13:59 zeljkof: extending EU SWAT for 5-10 minutes
13:51 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Change the zh-classicalwiki logo (T177165) (duration: 00m 45s)
13:50 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Change the zh-classicalwiki logo (T177165) (duration: 00m 46s)
13:44 zfilipin@tin: Synchronized static/images/project-logos/zh_classicalwiki-1.5x.png: static/images/project-logos/zh_classicalwiki-2x.png static/images/project-logos/zh_classicalwiki.png wmf-config/InitialiseSettings.php SWAT: Change the zh-classicalwiki logo (T177165) (duration: 00m 46s)
13:42 godog: roll restart thumbor to apply https://gerrit.wikimedia.org/r/#/c/381045/ - T166699
13:38 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable the Extension:SandboxLink on the nlwikinews (T177170) (duration: 00m 46s)
13:32 zfilipin@tin: Synchronized wmf-config/Wikibase-production.php: SWAT: Wikidata: Dont persist description usages (yet) (T177153) (duration: 00m 45s)
13:27 zfilipin@tin: Synchronized wmf-config/Wikibase-production.php: SWAT: Wikidata: Dont persist description usages (yet) (T177153) (duration: 00m 46s)
13:20 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add eliminator as a priviliged account (T176554) (duration: 00m 46s)
13:17 zfilipin@tin: Synchronized wmf-config/Wikibase-labs.php: wmf-config/Wikibase.php wmf-config/Wikibase-production.php SWAT: labs: Use redis lock manager for dispatching changes of Wikibase (T175109) (duration: 00m 46s)
13:09 zfilipin@tin: Synchronized wmf-config/Wikibase-labs.php: wmf-config/Wikibase.php wmf-config/Wikibase-production.php SWAT: labs: Use redis lock manager for dispatching changes of Wikibase (T175109) (duration: 00m 47s)
12:56 elukey: added two new mediawiki jobrunners - mw131[01]
12:50 marostegui: Optimize tables pagelinks and templatelinks on s5: dbstore2001
11:51 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2039 - T174509 (duration: 00m 46s)
10:06 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1311.eqiad.wmnet
10:06 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1310.eqiad.wmnet
10:04 mobrovac: restbase restarting on restbase1007 to start logging error bodies for T176728
09:55 jynus: stopping and upgrading labsdb1010
09:27 Amir1: finished the cleanup of ores_classification table in enwiki (T159753 )
09:12 godog: roll-restart thumbor to upgrade to 1.6 - T172556
09:06 elukey: forced remount of /mnt/hdfs on notebook1002
08:47 gehel: deploying latest tilerator on maps-test only, fix for T175123
08:46 gehel@tin: Finished deploy [tilerator/deploy@b02bd9a]: (no justification provided) (duration: 00m 22s)
08:46 gehel@tin: Started deploy [tilerator/deploy@b02bd9a]: (no justification provided)
08:46 marostegui: Poweroff db2044 for HW maintenance - T174764
08:40 marostegui: Stop MySQL on db2044 to get it ready to replace its mainboard - T174764
08:29 Amir1: finished the cleanup of ores_classification table in wikidatawiki and starting the enwiki one (T159753 )
08:24 jynus: stopping dbstore1001 for maintenance
08:12 marostegui: Optimize tables pagelinks and templatelinks on s5: db1100 (already depooled) - T174509
07:59 marostegui: Drop now redundant indexes from pagelinks and templatelinks from s5 - T174509
07:57 Amir1: starting a round of cleanup in ores_classification table in wikidatawiki
07:41 marostegui: Stop MySQL on db1036 as it is going to be decommissioned - https://phabricator.wikimedia.org/T176311
07:32 marostegui: Optimize table pagelinks and templatelinks on s6: dbstore2001 - T174509
07:21 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2039 - T174509 (duration: 00m 46s)
07:20 marostegui: Optimize table pagelinks and templatelinks on s6: db2039 - T174509
07:10 marostegui: Optimize table pagelinks and templatelinks on s1: db2034, db2055 db2062 - T174509
07:06 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2034, db2055 db2062 - T174509 (duration: 00m 46s)
06:55 _joe_: restarting a few API appservers with high cpu load
06:19 elukey: restart varnish backend on cp3040
06:10 elukey: restart varnish backend on cp3033
06:08 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1036 from config files as it will be decommissioned - T176311 (duration: 00m 46s)
06:07 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1036 from config files as it will be decommissioned - T176311 (duration: 00m 48s)
02:43 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Oct 2 02:43:33 UTC 2017 (duration 6m 39s)
02:36 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 08m 42s)
21:30 madhuvishy: powercycling labvirt1015
18:58 hashar: restarted Jenkins. Was stuck somehow
13:50 elukey: restart hhvm on mw1167 (jobrunner) - hhvm stuck, dump-debug in /tmp/hhvm.9624.bt.
20:54 catrope@tin: Synchronized wmf-config/CommonSettings.php: Bump $wgResourceLoaderStorageVersion (T176884 ) (duration: 00m 47s)
00:37 ejegg: increased PayPal Express Checkout cURL timeout to 12 seconds
22:12 ejegg: updated CiviCRM from 784756b 59e3e00
21:19 mutante: labstore2003/2004: closed idle screen sessions
20:27 mutante: mwlog1001, oxygen - closing unused screen session
20:20 mutante: install1002/install2002 - closing unused screen sessions (except a tail -f)
20:11 mutante: restbase2001 - closing screen sessions that weren't running anything anymore
19:23 ebernhardson: completed running cirrussearch category moves on commonswiki
18:19 catrope@tin: Synchronized php-1.31.0-wmf.1/includes/libs/CSSMin.php: T176884 (duration: 00m 47s)
18:08 chasemp: remove 'ToolLabs webservice - lighttpd on precise' catchpoint check for Catchpoint
18:07 chasemp: remove labsdb1002 checks on catchpoint (Toolforge)
18:06 chasemp: remove Labs puppetmaster eqiad catchpoint check (for Toolforge)
18:06 chasemp: remove grid-start-precise catchpoint check
18:04 chasemp: update catchpoing test for stream.wikimedia.org to watch https://stream.wikimedia.org/?doc
17:00 ebernhardson: kicking off extra job runners to process jobs moving commonswiki Category pages from general to content serach index
16:37 andrewbogott: re-imaging labvirt1017, 1018 in order to get it on the standard 4.4.0-81-generic kernel
15:52 andrewbogott: re-imaging labvirt1015 in order to get it on the standard 4.4.0-81-generic kernel
13:21 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2070, db2069 and db2042 - T174509 (duration: 00m 47s)
13:18 Amir1: starting a round of cleanup in ores_classification table in enwiki (T159753 )
13:09 elukey: depool mw1265 (hhvm 3.18.5) - disk filled up
12:43 bblack: removed numa=isolate sys entries manually (runtime + future boots) from cp4021
11:53 bblack: cp4021: reboot for numa mode switch
11:44 elukey: re-enable job runner daemons on mw130[8,9]
11:15 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2053 db2046 - T174509 (duration: 00m 48s)
11:05 elukey: precautionary stop of jobrunner/jobchron on the new mw130[8,9] - job queue size rapid increase investigation
09:55 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1309.eqiad.wmnet
09:33 ema: upgrade pybal to 1.14.0 on codfw primaries
08:20 ema: upgrade pybal to 1.14.0 on codfw secondaries
05:53 ema: cp4024 repooled T174891
05:21 ema: powerup cp4024 T174891
00:26 ejegg: updated fundraising tools from 69a9627 6d4b6f3
00:16 ejegg: updated CiviCRM 2138869 784756b
00:07 mutante: releases1001 - starting Jenkins setup wizard with generated admin pass
00:05 mutante: releases1001 - stopped puppet, manually fixing --prefix=/ci setting for jenkins process, killing it, removing init.d file, starting with systemd, jenkins now up T164030
23:46 awight@tin: Finished deploy [ores/deploy@42c5663]: Cause ORES service restart (duration: 00m 20s)
23:45 awight@tin: Started deploy [ores/deploy@42c5663]: Cause ORES service restart
22:18 MaxSem: Deployed fix for T176247
21:30 ejegg: updated CiviCRM from 3f48cb0 2138869
21:23 XenoRyet: Updated SmashPig from dc62203 d056a13
19:36 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.1
19:34 legoktm: mysql:wikiadmin@db1052 [enwiki]> delete from user_groups where ug_user =17629530 limit 1; (T176798 )
19:31 bblack: cp4026 - depool->reboot for bios updates, isolcpus fix
19:28 legoktm: mysql:wikiadmin@db1062 [centralauth]> UPDATE localuser SET lu_attached_method="new" WHERE lu_attached_method is NULL AND lu_name="Rua";
19:14 ejegg: updated fundraising tools 5708cf1
19:14 robh: cp4024 initial tests done (takes about 2 hours or so) then prompted for more in depth testing (hit yes, no failures so far) T174891
19:00 bblack: cp4025 - depool->reboot for bios updates, isolcpus fix
18:54 thcipriani@tin: Synchronized php-1.31.0-wmf.1/skins/MinervaNeue/includes/skins/SkinMinerva.php: SWAT: Revision::newFromTitle may return null T176882 (duration: 00m 50s)
18:42 bblack: cp4023 - depool->reboot for bios updates, isolcpus fix
18:37 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create rollbacker group at viwiktionary T176979 (duration: 00m 50s)
18:26 thcipriani@tin: Synchronized docroot/noc: SWAT: copy squid.php->reverse-proxy.php, squid-labs->reverse-proxy-staging PART II T104148 (duration: 00m 49s)
18:25 thcipriani@tin: Synchronized wmf-config: SWAT: copy squid.php->reverse-proxy.php, squid-labs->reverse-proxy-staging T104148 (duration: 00m 51s)
17:58 arlolra: Updated Parsoid to 2f4b9a8c T176363 , T176151 , T170832 )
17:52 mutante: librenms - renewed LE TLS cert ..via puppet, after Icinga alerted and accidentally still had "do_acme: false" in Hiera
17:46 arlolra@tin: Finished deploy [parsoid/deploy@84342fe]: Updating Parsoid to 2f4b9a8c
17:43 bblack: cp4028 - depool->reboot for bios updates, isolcpus fix
17:40 moritzm: installing apache updates on cobalt
17:32 bblack: cp4022 - depool->reboot for bios updates, isolcpus fix
17:30 arlolra@tin: Started deploy [parsoid/deploy@84342fe]: Updating Parsoid to 2f4b9a8c
17:25 marostegui: Run fixStuckGlobalRename.php --wiki=enwiktionary --logwiki=metawiki "CodeCat" "Rua" on terbium - T176985
17:20 bblack: cp4027 - depool->reboot for bios updates, isolcpus fix
17:08 bblack: cp4021 - depool->reboot for bios update
17:05 robh: bios firmware uploaded and awaiting reboot for application on cp402[1235678
16:48 bd808: Created centralauth.{analytics,web}.db.svc.eqiad.wmflabs Designate CNAME (T176978 )
16:26 robh: cp4024 has had ilom and bios firmware updaed per epsa error checking, now re-running epsa utility T174891
16:09 robh: cp4024 bios update in progress (has been for last 5 minutes), letting it alone for a solid 30 to let it finish
16:01 marostegui: Global rename of CodeCat → Rua - T176985
15:22 hoo: Ran scap pull on mwdebug1001 after experiments with T176844
14:54 dcausse: restarting elastic on relforge servers to reinstall prod ltr plugin
14:51 marostegui: Stop MySQL on db1035 as server is going to be decommissioned - T176931
14:45 moritzm: installing apache updates on puppet mastes (will trigger a few failing puppet runs, but ircecho disabled temporarily)
14:42 marostegui: Remove db1035 from tendril - T176931
14:31 moritzm: upgrading mw1276 (API canary) to HHVM 3.18.5
14:28 moritzm: upgrading mwdebug* to HHVM 3.18.5
13:56 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1035 from config as it will be decommissioned - T176931 (duration: 00m 49s)
13:55 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1035 from config as it will be decommissioned - T176931 (duration: 00m 50s)
13:40 zeljkof: EU SWAT finished
13:38 zfilipin@tin: Synchronized php-1.31.0-wmf.1/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: SWAT: ve.init.mw.DesktopArticleTarget: Remove hack for reversed tabs in RTL in Vector (T50017) (duration: 00m 50s)
13:37 moritzm: upgrading mw1262 to mw1265 (canary app servers) to HHVM 3.18.5
13:35 moritzm: uploaded HHVM 3.18.5 for jessie-wikimedia to apt.wikimedia.org
13:26 marostegui: Optimize table on s4 codfw master (db2051), with replication (lag might be generated) - T174509
13:13 ema: bounce pybal on ulsfo primaries to pick up bgp-local-ips config change T103882
13:12 marostegui: Drop redundant indexes from pagelinks and templatelinks on s4 - T174509
13:10 moritzm: upgrading mw1261 to HHVM 3.18.5
13:08 ema: bounce pybal on ulsfo secondaries to pick up bgp-local-ips config change T103882
13:02 marostegui: Optimize template links and pagelinks on db2053 and db2046 - T174509
13:00 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2053 db2046 - T174509 (duration: 00m 49s)
12:48 dcausse: restarting elastic on relforge servers to test a new version of the ltr plugin
12:27 ema: reboot cp402[2-6] to disable numa isolation
10:30 gehel: restart elasticsearch on relforge to validate new logging config - T175242
10:09 elukey: incrementally add mw1312-17 api-appservers to serve live traffic (weights will be raised incrementally)
10:00 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1308.eqiad.wmnet
09:59 elukey: added new mediawiki jobrunner - mw1308
09:52 ema: upgrade pybal to 1.14.0 on lvs1009-10
09:49 aaron@tin: Synchronized wmf-config/CommonSettings.php: Enable logging of post-send DB updates (duration: 00m 55s)
09:42 ema: upgrade pybal to 1.14.0 on lvs400[13]
08:00 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1288.eqiad.wmnet
07:27 moritzm: upgrading app servers in deployment-prep to HHVM 3.18.5
07:14 moritzm: installing apache updates on silver/wikitech.wikimedia.org
06:59 moritzm: installing apache updates on graphite* hosts
06:28 elukey: restart varnish backend on cp3032
05:54 marostegui: Optimize tables pagelinks and templatelinks on dbstore2001 (s2) - T174509
05:53 marostegui: Optimize tables pagelinks and templatelinks on dbstore2002 (s2) - T174509
05:43 marostegui: Run maintain-views on labsdb1001, labsdb1003, labsdb1009, labsdb1010, labsdb1011 for hiwikivoyage - T173027
05:38 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2049 and db2041 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 47s)
05:29 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2064 db2063 - T174509 (duration: 00m 48s)
05:21 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2076 db2067 db2060 - T174509 (duration: 00m 48s)
05:19 tstarling@tin: Finished scap: updating Vector skin for gerrit 381153 (duration: 20m 39s)
04:59 tstarling@tin: Started scap: updating Vector skin for gerrit 381153
03:57 bblack: ulsfo dns-depooled at ~03:51
03:26 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Sep 28 03:26:18 UTC 2017 (duration 6m 10s)
03:20 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 15m 44s)
02:42 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 15m 24s)
00:36 twentyafterfour: upgrading phabricator, expect minimal downtime
23:39 robh: cp4024 is running hardware testing, leave it alone T174891
23:17 catrope@tin: Synchronized wmf-config/Wikibase.php: Make CirrusSearch default for wbsearchentities on testwikidatawiki T175741 (duration: 00m 48s)
22:34 maxsem@tin: Synchronized php-1.31.0-wmf.1/includes/EditPage.php: https://gerrit.wikimedia.org/r/#/c/381139/1 (duration: 00m 49s)
19:14 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 -> wmf.1
19:13 demon@tin: Synchronized php: symlink bump (duration: 00m 47s)
19:10 demon@tin: Synchronized php-1.31.0-wmf.1/maintenance/backup.inc: unbreak dumps (duration: 00m 49s)
18:46 catrope@tin: Finished scap: SWAT: T176762 , T175358 , T172023 , T174719 , and add html5-misnesting to Linter (duration: 22m 55s)
18:39 bblack: re-pooling upload@ulsfo (text@ulsfo remains pooled)
18:23 catrope@tin: Started scap: SWAT: T176762 , T175358 , T172023 , T174719 , and add html5-misnesting to Linter
18:21 ejegg: updated payments-wiki from 6104e3d c2a3d43
17:40 ejegg: updated SmashPig Adyen settings
17:29 bblack: uploaded nginx-1.13.5-1+wmf1 to stretch-wikimedia, and matching nginx-1.13.5-1+wmf1~jessie1 to jessie-wikimedia
16:27 XioNoX: setting local-as to selected transit BGP sessions - T167840
16:00 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4026.ulsfo.wmnet
16:00 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4025.ulsfo.wmnet
15:59 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4023.ulsfo.wmnet
15:58 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4022.ulsfo.wmnet
15:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1103 with low weight - T172679 (duration: 00m 47s)
15:51 ema: lvs4002: upgrade pybal to 1.14.0
15:49 ema: lvs4004: upgrade pybal to 1.14.0
15:28 _joe_: uploaded prometheus-statsd-exporter to stretch-wikimedia T175539
15:24 bblack: rebooting cp402[2356] (bnx2x oops)
15:15 moritzm: revoking salt certs for WMCS
15:12 mlitn@tin: Synchronized wmf-config/CommonSettings-labs.php: Remove 3D beta config (duration: 00m 47s)
15:11 mlitn@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Remove 3D beta config (duration: 00m 48s)
15:10 mlitn@tin: Synchronized wmf-config/CommonSettings.php: Load 3D extension (duration: 00m 49s)
15:09 mlitn@tin: Synchronized wmf-config/InitialiseSettings.php: Config 3D to be loaded on test, test2 (duration: 00m 48s)
15:00 mlitn@tin: Finished scap: Enable 3D extension (duration: 37m 09s)
14:58 ema: lvs1007: upgrade pybal to 1.14.0
14:45 elukey: rolling restart of all the Yarn nodemanager daemons on analytics1028-1068
14:43 bblack: cp1008/pinkunicorn upgraded to test build of nginx-1.13.5-1+wmf1
14:28 volans: uploaded cumin_1.2.1-1_amd64.deb to apt.wikimedia.org jessie-wikimedia
14:24 akosiaris: change wtp1001 to wtp1024 weights to 5 from 15 in preparation for deprecation
14:24 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1024.eqiad.wmnet
14:24 mlitn@tin: Started scap: Enable 3D extension
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1023.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1022.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1021.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1020.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1019.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1018.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1017.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1016.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1015.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1014.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1013.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1012.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1011.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1010.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1009.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1008.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1007.eqiad.wmnet
14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1006.eqiad.wmnet
14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1005.eqiad.wmnet
14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1004.eqiad.wmnet
14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1003.eqiad.wmnet
14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1002.eqiad.wmnet
14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1001.eqiad.wmnet
14:17 akosiaris: T175242 restart eventstreams across the fleet to pick up the change in a rolling restart manner with a batch size of 2
14:16 akosiaris: T175242 restart parsoid across the fleet to pick up the change in a rolling restart manner with a batch size of 5
14:03 akosiaris: T175242 restart tilerator, tileratorui, restbase across the fleet to pick up the change in a rolling restart manner with a batch size of 2
13:43 akosiaris: T175242 re-enable puppet across aqs kafka maps maps-test ores restbase restbase-dev sca scb wtp clusters for merging https://gerrit.wikimedia.org/r/#/c/376500/ . Run puppet as well in a batched execution
13:42 anomie@tin: Synchronized php-1.31.0-wmf.1/includes/specials/SpecialWatchlist.php: SWAT: {{gerrit|380968}} Fix watchlist "in the last X hours" display (duration: 00m 48s)
13:42 elukey: raised Hadoop HDFS namenode master daemon max heap size to 6G (prev 4G) on analytics100[12]
13:31 akosiaris: T175242 eventstreams requires manual restart
13:25 akosiaris: T175242 parsoid requires manual restart
13:15 akosiaris: T175242 restbase requires manual restart
13:11 addshore: SWAT done
13:11 addshore@tin: Synchronized php-1.31.0-wmf.1/extensions/TwoColConflict/modules/ext.TwoColConflict.BaseVersionSelector.css: SWAT: Address changes in the label style in OOUI (duration: 00m 46s)
13:09 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add autopatrolled user group to dty.wikipedia (T176709) (duration: 00m 47s)
13:01 akosiaris: T175242 tilerator and tileratorui need manually restart
12:54 akosiaris: T175242 enabled puppet in aqs kafka maps maps-test selected hosts and ran puppet manually.
12:47 akosiaris: T175242 disable puppet across aqs kafka maps maps-test ores restbase restbase-dev sca scb wtp clusters for merging https://gerrit.wikimedia.org/r/#/c/376500/
12:20 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1103 to the array of hosts (duration: 00m 48s)
12:19 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1103 to the array of hosts (duration: 00m 49s)
12:14 moritzm: installing apache updates on einsteinium/tegmen
12:06 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on bnwiki (T176771 ) (duration: 00m 49s)
12:05 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on bnwiki (T176771 ) (duration: 01m 00s)
09:45 marostegui: Stop mysql on db1035 to copy its data to db1103 - T172679
09:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1035 to transfer its data to db1103 - T172679 (duration: 00m 48s)
08:55 mobrovac@tin: Finished deploy [changeprop/deploy@7a3dc66]: Use the /page/title/{title}/{revision} end point for revision-visibility-change - T158100 (duration: 01m 26s)
08:53 mobrovac@tin: Started deploy [changeprop/deploy@7a3dc66]: Use the /page/title/{title}/{revision} end point for revision-visibility-change - T158100
08:43 mobrovac@tin: Finished deploy [restbase/deploy@1cc530b]: Introduce the /page/title/{title}/{revision} end point - T158100 (duration: 10m 01s)
08:34 elukey: raise traffic weights to 30 for mw13[19-28] incrementally - T165519
08:33 mobrovac@tin: Started deploy [restbase/deploy@1cc530b]: Introduce the /page/title/{title}/{revision} end point - T158100
08:16 marostegui: Optimize tables pagelinks templatelinks on db2064 and db2063 - T174509
08:16 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2064 db2063 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 48s)
07:55 marostegui: Optimize pagelinks and template links tables on db2076 db2067 db2060 - T174509
07:52 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2076 db2067 db2060 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 48s)
07:37 marostegui: Optimize pagelinks and template links tables on db2070 db2069 db2042 db2016 - T174509
07:27 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2070, db2069 and db2042 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 48s)
07:17 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2071 and db2072 - T174509 (duration: 00m 47s)
06:57 marostegui: Deploy alter table on s6 codfw - T174509
06:55 marostegui: Deploy alter table on db1052 (enwiki master) - T174509
06:37 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add task number to db2044 line (duration: 00m 48s)
06:37 ema: restart varnish-be on cp3040
06:36 moritzm: installing git security updates (our config is not affected, but still fixing the underlying issue)
06:33 ema: restart varnish-be on cp3041
06:29 marosteg1i: Reboot db2044 after storage failure - T174764
06:15 elukey: restart varnish backend on cp3043
05:41 madhuvishy: service nova-fullstack restart on labnet1001
03:17 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Sep 27 03:17:22 UTC 2017 (duration 6m 22s)
03:11 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 15m 55s)
02:32 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 08m 42s)
23:25 thcipriani@tin: Synchronized php-1.31.0-wmf.1/extensions/VisualEditor/modules/ve-mw/ui/tools/ve.ui.MWSignatureTool.js: SWAT: Revert MWSignatureTool case (duration: 00m 48s)
23:20 thcipriani@tin: Synchronized php-1.30.0-wmf.19/resources/src/mediawiki.rcfilters/mw.rcfilters.init.js: SWAT: RCFilters: Log performance data T176652 (duration: 00m 48s)
23:19 thcipriani@tin: Synchronized php-1.31.0-wmf.1/resources/src/mediawiki.rcfilters/mw.rcfilters.init.js: SWAT: RCFilters: Log performance data T176652 (duration: 00m 51s)
22:03 no_justification: gerrit master restart, back momentarily
20:32 mutante: phab1001 - restarted apache after upgrade
20:30 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2044 - T174764 (duration: 00m 50s)
20:20 mutante: phab1001 (prod phab) - upgrade Apache package
20:15 mutante: phab2001 - install Apache and MariaDB package upgrades
19:16 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.31.0-wmf.1
18:25 mepps: Updated crm from 89b8007 3f48cb0
18:00 demon@tin: Finished scap: 1.31.0-wmf.1 bootstrap + testwiki (duration: 47m 56s)
17:12 demon@tin: Started scap: 1.31.0-wmf.1 bootstrap + testwiki
15:50 volans: powecycled cp2012, no ping or ssh console stuck
15:18 XioNoX: starting eqiad frack switch to new infra - T174218
15:06 gehel: restarting blazegraph on all wdqs nodes for heap resize - T175919
14:47 mobrovac@tin: Finished deploy [restbase/deploy@f989dd9]: Remove the old mobileapps module completely - T169940 (duration: 20m 35s)
14:36 marostegui: Optimize tables pagelinks and templatelinks on db2071 and db2072 from s1 - T174509
14:27 mobrovac@tin: Started deploy [restbase/deploy@f989dd9]: Remove the old mobileapps module completely - T169940
14:26 hoo: mwscript refreshLinks.php --wiki elwiki --namespace 0 on terbium has finished (T151717 )
14:12 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2071 and db2072 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 44s)
13:39 marostegui: Deploy alter table on s6 master (with replication enable, so lag will be generated) db2028 - T163979
13:25 hashar@tin: Synchronized wmf-config/filebackend.php: Expose Thumbor swift username - T144479 (duration: 00m 44s)
13:24 hashar@tin: Synchronized wmf-config/CommonSettings.php: Enable asia-specific Navigation Timing metric - T169522 (duration: 00m 47s)
13:22 godog: upgrade grafana to 4.5.2 on labmon1001 - T175980
13:15 hashar@tin: Synchronized wmf-config/CommonSettings.php: Disable RelatedArticles instrumentation on all wikis - T174944 (duration: 00m 45s)
13:02 marostegui: Drop redundant indexes from s6 - T174509
13:00 elukey: add mw132[7,8] to live traffic (new appservers) - weights will be increased incrementally from 5 to 20
12:44 gehel: restarting blazegraph on wdqs1004 / wdqs2001 for heap resive - T175919
12:27 marostegui: Drop redundant indexes from s2 - T174509
12:12 marostegui: Drop redundant indexes from enwiki on eqiad - T174509
12:03 kartik@tin: Finished deploy [cxserver/deploy@aa232ee]: Update cxserver, registry reconstruction (duration: 03m 03s)
12:00 kartik@tin: Started deploy [cxserver/deploy@aa232ee]: Update cxserver, registry reconstruction
11:58 moritzm: uploaded prometheus-apache-exporter 0.3-1+deb9u1 to apt.wikimedia.org/stretch-wikimedia
11:57 marostegui: Drop redundant indexes from enwiki on codfw - T174509
11:49 moritzm: uploaded prometheus-hhvm-exporter 0.3.1+deb9u1 to apt.wikimedia.org/stretch-wikimedia
11:30 kartik@tin: (no justification provided)
11:29 kartik@tin: Finished deploy [cxserver/deploy@f1d4851]: Update cxserver, registry reconstruction (duration: 00m 52s)
11:28 kartik@tin: Started deploy [cxserver/deploy@f1d4851]: Update cxserver, registry reconstruction
10:52 volans: uploaded cumin_1.1.1-1_amd64.deb to apt.wikimedia.org jessie-wikimedia
10:34 joal@tin: Finished deploy [analytics/refinery@5d806c6]: Regular analytics deploy (duration: 08m 00s)
10:26 joal@tin: Started deploy [analytics/refinery@5d806c6]: Regular analytics deploy
10:20 _joe_: rebooting copper (again)
09:12 elukey: add mw132[4,5,6] to live traffic (new appservers) - weights will be increased incrementally from 5 to 20
09:10 _joe_: rebooting copper to verify docker setup is correct
08:46 moritzm: installing apache updates on mw* app servers
08:44 godog: run puppet on ms-fe* to reduce conntrack generated by swift internal clients towards backends - T173731
08:24 godog: roll-restart thumbor to upgrade to 1.5 - T173804
08:09 godog: roll-restart swift-proxy to apply https://gerrit.wikimedia.org/r/#/c/380483/ - T161535
08:08 marostegui: Drop old temporary tar.gz files from dbstore1001 to get back some disk space
08:07 marostegui: Drop table trackbacks from s3 - T175051
07:48 mattflaschen@tin: Synchronized wmf-config/CommonSettings-labs.php: RCFilters: Beta Cluster only (duration: 00m 46s)
07:46 marostegui: Drop trackbacks table from s7 - T175051
07:28 marostegui: Drop trackbacks table from s6 - T175051
07:16 moritzm: installing perl security updates
06:45 _joe_: restarting varnish backend on cp3033, throwing 503s
06:29 mobrovac@tin: Finished deploy [cpjobqueue/deploy@17c3833]: (no justification provided) (duration: 00m 28s)
06:28 mobrovac@tin: Started deploy [cpjobqueue/deploy@17c3833]: (no justification provided)
06:18 marostegui: Drop table trackbacks from s4 - T175051
06:06 marostegui: Drop table trackbacks from s5 (only exists on dewiki) - T175051
05:48 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2047 - T176573 (duration: 00m 44s)
05:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove old db1055 comments (duration: 00m 46s)
02:29 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Sep 26 02:29:31 UTC 2017 (duration 5m 52s)
02:23 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 07m 26s)
23:58 ejegg: updated payments-wiki from d2ce697 6104e3d
23:15 mutante: gerrit2001 reinstalled with stretch, revoked old puppet cert, accepted new puppet cert, initial run that will do base and all the gerrit things at once.. (T168562 )
22:53 mutante: gerrit2001 - PXE booting
22:34 mutante: gerrit2001 - schedule downtime for reinstall with strech, reboot imminent
20:57 hoo: Manually pruned the sites and site_identifiers table on hiwikivoyage, than ran populateSitesTable.php. (T173030 )
20:37 arlolra: Updated Parsoid to 4230c27b T176425 , T173029 )
20:30 arlolra@tin: Finished deploy [parsoid/deploy@376faad]: Updating Parsoid to 4230c27b
20:13 arlolra@tin: Started deploy [parsoid/deploy@376faad]: Updating Parsoid to 4230c27b
19:58 XioNoX: renumbering ams BGP communities - T167840
19:25 robh@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp4021.ulsfo.wmnet
19:02 robh: cp4021 coming down for memory swap
19:02 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp4021.ulsfo.wmnet
18:54 MaxSem: started populating ip_changes on group2 wikis
18:38 demon@tin: Synchronized scap/plugins/clean.py: no-op/consistency (duration: 00m 45s)
18:22 ejegg: updated CiviCRM from 45c56cf 89b8007
17:58 demon@tin: Pruned MediaWiki: 1.30.0-wmf.15 (duration: 02m 46s)
17:07 gehel@tin: Finished deploy [wdqs/wdqs@3eec185]: Deploy new Blazegraph binary & GUI update (duration: 01m 51s)
17:05 gehel@tin: Started deploy [wdqs/wdqs@3eec185]: Deploy new Blazegraph binary & GUI update
17:02 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: Disable the updateBetaFeaturesUserCounts logging channel - T175637 (duration: 00m 46s)
16:37 demon@tin: Synchronized php-1.30.0-wmf.19/maintenance/getConfiguration.php: getConfiguration: Don't bail when a valid variable is set null (duration: 00m 47s)
15:57 urandom: T169940 : Converting restbase-ng data tables to size-tiered compaction
14:39 gehel: dropping badly named apifeature index from cirrus elasticsearch codfw/eqiad - T176430
14:31 gehel: rolling restart of logstash for config change - T176430
14:13 gehel: rolling restart of logstash for config change - T176430
13:46 zeljkof: EU SWAT finished
13:45 zfilipin@tin: Synchronized static/images/project-logos/hiwiki.png: SWAT: Make hiwiki logo a little bit bigger (T175721) (duration: 00m 46s)
13:37 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Import sources on hr.wikibooks (T176320) (duration: 00m 46s)
13:30 addshore@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/WikimediaEventsHooks.php: SWAT: gerrit:379749 onBeforeInitializeWMDECampaign, fix early bail condition T174948 (duration: 00m 46s)
13:18 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add several rights to eliminators in fawiki (T176553) (duration: 00m 46s)
12:52 moritzm: removed salt master from neodymium
12:38 moritzm: revoked all salt keys on neodymium
11:58 godog: powercycle ms-be2020 - blk_update_request: I/O error, dev sdb, sector in consol
11:54 moritzm: removing salt-minion/salt-common from production
11:47 Dereckson: Repopulate sites tables for Wikidata clients (T173013 )
11:47 Dereckson: Create new database and tables for hi.wikivoyage (T173013 )
11:45 dereckson@tin: Synchronized wmf-config/interwiki.php: Interwiki map update (+hiwikivoyage and other changes previous month, T176572 ) (duration: 00m 46s)
11:39 marostegui: Run redact_sanitarium for hiwikivoyage on db1095 - T173027
11:39 moritzm: upgrading tor to latest stable release (0.3.1.7)
11:37 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Initial configuration for hi.wikivoyage.org (T173013 ) (duration: 00m 46s)
11:37 dereckson@tin: Synchronized static/images/project-logos/: Logos for hi.wikivoyage.org (T173013 ) (duration: 00m 45s)
11:34 dereckson@tin: rebuilt wikiversions.php and synchronized wikiversions files: +hiwikivoyage (T173013 )
11:32 dereckson@tin: Synchronized dblists: Add hiwikivoyage to the set (T173013 ) (duration: 00m 46s)
10:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1055 weight (duration: 00m 46s)
10:14 elukey: add mw1323 to live traffic (mw appserver) - traffic weights will go from 5 to 20 incrementally
09:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1055 weight and remove main traffic from special slaves (duration: 00m 46s)
09:07 marostegui: Add 50GB to db1036 /srv partition - T176311
09:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1036 and full repool db1101 - T176311 (duration: 00m 46s)
08:49 moritzm: uploaded php-luasandbox 2.0.14, hhvm-tidy 0.1.3 and php-wikidiff2 1.4.1 for stretch-wikimedia to apt.wikimedia.org
08:36 mobrovac@tin: Finished deploy [restbase/deploy@ab24f70]: Switch mobile-sections to next-gen storage and stop storing the current-day feed in Cassandra - T169940 T176233 (duration: 10m 03s)
08:26 mobrovac@tin: Started deploy [restbase/deploy@ab24f70]: Switch mobile-sections to next-gen storage and stop storing the current-day feed in Cassandra - T169940 T176233
08:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1055 weight (duration: 00m 45s)
07:56 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1101 - T176311 (duration: 00m 46s)
07:42 moritzm: uploaded hhvm 3.18.2+dfsg-1+wmf5+deb9u1 for stretch-wikimedia to apt.wikimedia.org
07:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1055 weight (duration: 00m 47s)
07:37 elukey: add mw132[0,2] to live traffic (mw appservers) - traffic weights will go from 5 to 20 incrementally
06:38 moritzm: installing apache security updates on logstash*
02:36 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Sep 25 02:36:48 UTC 2017 (duration 6m 42s)
02:30 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 07m 17s)
14:23 jynus: restarting db2047 for kernel upgrade
14:16 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2047 (duration: 00m 48s)
off: restarted ircecho on einsteinium
23:36 mutante: gerrit2001 - rebooting .. wave a dead chicken http://www.catb.org/jargon/html/W/wave-a-dead-chicken.html
22:56 mutante: gerrit2001 - trying to manually start gerrit again, as opposed to puppet doing it.. debugging gerrit-ssh issue there, cobalt still untouched
22:34 mutante: gerrit2001 - stopping gerrit with gerrit.sh stop, letting puppet start it again
22:29 mutante: gerrit2001 - systemd says gerrit.service is failed. gerrit.sh start says "Already Running!!" :p - cobalt is fine
18:26 demon@tin: Pruned MediaWiki: 1.30.0-wmf.18 [keeping static files] (duration: 01m 29s)
15:21 hashar: Restarted Jenkins. Out of memory)
14:49 jynus@tin: Synchronized wmf-config/db-eqiad.php: repool db1101 with low weight (duration: 00m 47s)
14:43 moritzm: uploaded php-luasandbox build for src:php5.5 (required for CI tests on jessie)
14:42 moritzm: updated tor packages to 0.3.1.7 (new stable series)
13:49 elukey: mw1321 (new appserver) serving traffic (going to increase its weight up to 20)
12:02 hashar: apt-get upgrade on contint1001 / contint2001
09:42 elukey: mw1319 (new appserver) serving traffic (going to increase its weight up to 20)
09:14 jynus: stop mariadb at db1055 for upgrade and maintenance
06:32 moritzm: installing emacs security updates on trusty (Debian already fixed)
23:56 ejegg: updated payments-wiki from 7ed7243 d2ce697
23:23 ebernhardson@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/extension.json: T175047 : Stop delivering search relevance survey javascript (duration: 00m 46s)
23:18 ebernhardson@tin: Synchronized wmf-config: T175047 : Turn off human search relevance survey (duration: 00m 48s)
23:16 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T175047 : Turn off human search relevance survey (duration: 00m 46s)
22:59 jynus@tin: Synchronized wmf-config/db-eqiad.php: depool db1055 (duration: 00m 46s)
22:12 volans: uploaded cumin_1.1.0-1_amd64.deb to apt.wikimedia.org jessie-wikimedia
22:10 ejegg: updated payments-wiki from 9da8482 7ed7243
21:07 demon@tin: Synchronized wmf-config/CommonSettings.php: extdist settings (duration: 00m 46s)
21:04 ejegg: updated CiviCRM from fc56901 45c56cf
19:32 demon@tin: Synchronized wmf-config/InitialiseSettings.php: adjust logging on csp reports (duration: 00m 46s)
19:20 demon@tin: Synchronized php-1.30.0-wmf.19/extensions/VisualEditor/ApiVisualEditor.php: fix user/wgUser mixup (duration: 00m 46s)
19:16 gehel: upgrade to nodejs 6.11 on maps servers (including restart of tilerator / kartotherian) - T171707
19:16 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4026.*
19:04 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.19
18:56 urandom: T171772 : Applying locally hacked Prometheus exporter config to RESTBase dev Cassandra instances
18:44 thcipriani@tin: Synchronized php-1.30.0-wmf.19/resources/src/mediawiki.rcfilters: SWAT: WLFilters: Do not hide .watchlistDetails while loading T176300 RCFilters: Make the interface not jump around while loading (duration: 00m 49s)
18:41 urandom: T171772 : Restarting Cassandra restbase-dev1004-a to apply locally hacked Prometheus exporter config
18:34 thcipriani@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Turning off Explore Similar AB test T175649 (duration: 00m 49s)
18:31 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Print instrumentation T176341 (duration: 00m 52s)
14:56 moritzm: uploaded php-defaults and php-redis built against src:php5.5 to component/ci for jessie-wikimedia
13:45 gehel: upgrade to nodejs 6.11 on the full maps-test cluster - T171707
13:12 zeljkof: EU SWAT finished
13:09 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Switch elasticsearch active cluster to codfw" (duration: 00m 48s)
13:09 zfilipin@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Revert "Switch elasticsearch active cluster to codfw" (duration: 00m 48s)
13:07 zfilipin@tin: Synchronized tests/cirrusTest.php: SWAT: Revert "Switch elasticsearch active cluster to codfw" (duration: 00m 49s)
12:55 _joe_: stopped long-running runjobs for refreshlinks on commons.
12:50 mobrovac@tin: Finished deploy [restbase/deploy@bc02191]: produce a diff for the mobile proxy (duration: 18m 34s)
12:32 mobrovac@tin: Started deploy [restbase/deploy@bc02191]: produce a diff for the mobile proxy
11:59 moritzm: upgrading apache on deployment servers and script runners
11:40 moritzm: upgrading apache on canary app servers
11:17 moritzm: installing apache security updates (our configurations are not affected, the upload of 2.4.10-10+deb8u11+wmf1 is mostly for the benefit of whatever people are doing in Cloud VPS, but we should still fix the underlying bug in production as well)
11:14 dcausse: reindexing all hebrew wikis to pickup the new hebmorph analyzer
11:09 dcausse: reindexing ilwikimedia in codfw to pickup and test new hebmorph analyzer
10:26 akosiaris: upload docker-ce_17.06.2~ce-0~debian_amd64.deb to apt.wikimedia.org jessie-wikimedia/thirdparty/ci T175293
09:59 dcausse: reindexing group1 wikis T176397
09:54 dcausse: reindexing group0 wikis T176397
08:35 moritzm: unbreak deployment-puppetmaster02 in deployment-prep (broken by unattended-upgrades update of apache T159254 )
08:35 akosiaris: pool all of wtp1025 to wtp1048 T165520
08:35 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1044.eqiad.wmnet
08:35 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1043.eqiad.wmnet
08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1042.eqiad.wmnet
08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1041.eqiad.wmnet
08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1040.eqiad.wmnet
08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1039.eqiad.wmnet
08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1038.eqiad.wmnet
08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1037.eqiad.wmnet
08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1036.eqiad.wmnet
08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1035.eqiad.wmnet
08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1034.eqiad.wmnet
08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1033.eqiad.wmnet
08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1032.eqiad.wmnet
08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1031.eqiad.wmnet
08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1030.eqiad.wmnet
08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1029.eqiad.wmnet
08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1028.eqiad.wmnet
08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1027.eqiad.wmnet
08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1026.eqiad.wmnet
08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
08:30 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
08:30 akosiaris@puppetmaster1001: conftool action : set/weight=10; selector: name=wtp1025.eqiad.wmnet
08:25 mobrovac@tin: Finished deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests (duration: 03m 18s)
08:22 mobrovac@tin: Started deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests
08:21 mobrovac@tin: Finished deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests (duration: 00m 58s)
08:20 mobrovac@tin: Started deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests
08:19 mobrovac@tin: Finished deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests (duration: 05m 22s)
08:14 mobrovac@tin: Started deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests
08:08 mobrovac@tin: Finished deploy [restbase/deploy@3e9bd9f]: New storage schema for mobile-sections - T169940 (duration: 10m 28s)
07:57 mobrovac@tin: Started deploy [restbase/deploy@3e9bd9f]: New storage schema for mobile-sections - T169940
07:31 _joe_: restarted aphlict on phab1001
07:27 mobrovac@tin: Finished deploy [restbase/deploy@3e9bd9f]: New storage schema for mobile-sections, canary deploy for schema creation, take 2 - T169940 (duration: 00m 30s)
07:27 mobrovac@tin: Started deploy [restbase/deploy@3e9bd9f]: New storage schema for mobile-sections, canary deploy for schema creation, take 2 - T169940
07:09 ema: bounce pybal on lvs1003 to clear stale alert T176388
06:59 ema: bounce pybal on lvs1006 to clear stale alert
06:55 ema: bounce pybal on lvs1009 to clear stale alert
03:24 bblack: depooled cp4026
03:12 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Sep 21 03:12:41 UTC 2017 (duration 6m 36s)
03:06 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 14m 44s)
02:29 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 08m 50s)
00:41 eileen: update civicrm from da833ac fc56901
00:23 ebernhardson@tin: Synchronized php-1.30.0-wmf.18/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: Fix schema name for search satisfaction error logging (duration: 00m 53s)
00:18 ebernhardson@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: Fix schema name for search satisfaction error logging (duration: 00m 49s)
00:09 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable jQuery 3 on commons (T124742 ) (duration: 00m 49s)
00:02 catrope@tin: Synchronized php-1.30.0-wmf.19/resources/src/mediawiki.rcfilters/: Lazy-load the RCFilters menu (T176250 ) (duration: 00m 48s)
23:49 catrope@tin: Synchronized php-1.30.0-wmf.18/extensions/WikimediaEvents/: Log JS errors during search satisfaction tests (duration: 00m 48s)
23:47 catrope@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/: Log JS errors during search satisfaction tests (duration: 00m 49s)
23:40 mutante: phabricator: aphlict (notification service) now running on prod server
23:36 mutante: phab1001 - re-enable puppet, run after follow-up fix, adding notification server config
23:31 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Add throttle rule for John Michael Kohler Art Center (T176287 ) (duration: 00m 48s)
23:29 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable user email blacklist on meta (T174694 ) (duration: 00m 48s)
23:21 catrope@tin: Synchronized dblists/categories-rdf.dblist: Add mlwiki to categories-rdf.dblist (duration: 00m 48s)
23:14 mutante: phab2001 - deluser aphlict, delgroup aphlict, try to let puppet create it again, still fails, just won't create the group
23:13 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Add patroller group on metawiki (T176079 ) (duration: 00m 48s)
23:11 mutante: phab2001 - groupadd aphlict,temp adduser aphlict to debug user creation issue
22:22 ebernhardson: elasticsearch eqiad completely recovery, all green. Returning recovery max_bytes_per_sec to 20mb
21:54 mutante: phab: disable puppet on phab1001 for a minute, apply notification server change on phab2001
21:45 eileen: update civicrm from 0fb4676 da833ac
21:09 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4015.ulsfo.wmnet
20:58 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4007.ulsfo.wmnet
20:40 bsitzmann@tin: Finished deploy [mobileapps/deploy@e23bf66]: Update mobileapps to d46860e T175759 T176263 ) (duration: 06m 05s)
20:36 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4014.ulsfo.wmnet
lunch: increase max recovery bytes/s to 80mb in eqiad elasitcsearch cluster
20:34 bsitzmann@tin: Started deploy [mobileapps/deploy@e23bf66]: Update mobileapps to d46860e T175759 T176263 )
20:33 demon@tin: Synchronized php-1.30.0-wmf.19/includes/diff/DifferenceEngine.php: fix some warnings (duration: 00m 51s)
19:55 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4016.ulsfo.wmnet
19:54 eevans@tin: Finished deploy [cassandra/metrics-collector@df909a1] (dev-cluster): Deploy 4.1.0 artifact to dev cluster (duration: 00m 10s)
19:54 eevans@tin: Started deploy [cassandra/metrics-collector@df909a1] (dev-cluster): Deploy 4.1.0 artifact to dev cluster
19:53 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4006.ulsfo.wmnet
19:48 eevans@tin: Finished deploy [cassandra/metrics-collector@df909a1]: Pushing out 4.1.0 release (duration: 00m 47s)
19:47 eevans@tin: Started deploy [cassandra/metrics-collector@df909a1]: Pushing out 4.1.0 release
19:39 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4008.ulsfo.wmnet
19:26 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.19
19:24 demon@tin: Synchronized php: symlink swap for wmf.19 (duration: 00m 49s)
19:02 catrope@tin: Finished scap: Patches for T176302 , T175962 , T173533 (duration: 21m 29s)
18:40 catrope@tin: Started scap: Patches for T176302 , T175962 , T173533
17:29 hoo: Started mwscript refreshLinks.php --wiki elwiki --namespace 0 in a screen on terbium. Can be KILLed at any time, if needed.
17:29 hoo: Started mwscript refreshLinks.php --wiki elwiki --namespace 0 in a screen on elwiki. Can be KILLed at any time, if needed.
17:13 hoo: Ran mwscript refreshLinks.php --wiki elwiki --namespace 0 -e 30 for testing purposes
17:06 ejegg: removed limit from donation queue consumer
17:05 otto@tin: Finished deploy [eventstreams/deploy@e62ab64]: (no justification provided) (duration: 04m 44s)
17:01 otto@tin: Started deploy [eventstreams/deploy@e62ab64]: (no justification provided)
16:53 madhuvishy: Upgraded and restarted apache2 on labs-puppetmaster.wikimedia.org
16:45 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,cluster=elasticsearch
16:42 gehel: rolling restart of logstash ingesters after cirrus eqiad recovery
16:34 gehel: elasticsearch eqiad is finally recovering, after the restart of elastic1051
16:34 gehel: cleanup jar hell and restart elastic1051
16:19 gehel: starting elasticsearch on all nodes on eqiad
16:10 gehel: restarting elasticsearch masters on eqiad (elastic1030, 36 and 40)
16:04 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: dc=eqiad,cluster=elasticsearch
16:03 gehel: depooling all nodes in elasticsearch eqiad to investigate failed cluster restart (traffic is already directed to codfw)
15:46 ejegg: limited donations queue consumer to 100 per run
15:36 ejegg: updated payments-wiki from 896f16a 9da8482
15:36 gehel: upgrading elasticsearch plugins on elasticsearch eqiad, including cold restart of the cluster - T173231
15:33 ejegg: limited donations queue consumer to 200 per run
14:03 akosiaris@tin: Synchronized wmf-config/InitialiseSettings.php: Whitelist wtp10[25-48] for Linter (duration: 00m 49s)
13:42 otto@tin: Synchronized /srv/mediawiki-staging/wmf-config/CommonSettings-labs.php: (no justification provided) (duration: 00m 49s)
13:24 ema: begin rolling codfw varnish-be restarts, 20m spacing
13:20 dereckson@tin: Synchronized wmf-config/CommonSettings.php: Switch elasticsearch active cluster to codfw (Gerrit:378850 , 3/3) (duration: 00m 48s)
13:18 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Switch elasticsearch active cluster to codfw (Gerrit:378850 , 2/3) (duration: 00m 49s)
13:17 dereckson@tin: Synchronized tests/cirrusTest.php: Switch elasticsearch active cluster to codfw (Gerrit:378850 , 1/3, no-op in prod) (duration: 00m 49s)
12:51 ema: restarted varnish-be on cp4028
12:28 gehel: upgrading nodejs to 6.11 on maps-test2003 for testing - T171707
12:19 ema: restarted varnish-be on cp4027
12:18 moritzm: uploaded apache 2.4.10-10+deb8u11+wmf1 to jessie-wikimedia (rebuild of latest security update with our local patches on top)
12:08 hoo@tin: Synchronized wmf-config/Wikibase-production.php: Enable statement usage tracking on elwiki (T151717 ) (duration: 00m 49s)
11:58 mobrovac@tin: Finished deploy [restbase/deploy@dea2b41]: New storage schema for mobile-sections, canary deploy for schema creation - T169940 (duration: 07m 20s)
11:56 ema: restarted varnish-be on cp4018
11:51 mobrovac@tin: Started deploy [restbase/deploy@dea2b41]: New storage schema for mobile-sections, canary deploy for schema creation - T169940
11:49 jynus: stopping replication on db1101 for faster repartition work T176311
11:45 jynus@tin: Synchronized wmf-config/db-eqiad.php: depool db1101 (duration: 00m 58s)
11:34 ema: restarted varnish-be on cp4010
11:25 mobrovac@tin: Finished deploy [restbase/deploy@dea2b41]: (no justification provided) (duration: 00m 02s)
11:25 mobrovac@tin: Started deploy [restbase/deploy@dea2b41]: (no justification provided)
11:03 moritzm: rolling out new debdeploy release across the fleet
11:02 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 19s)
11:02 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided)
11:02 ema: restarted varnish-be on cp4008
11:02 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 04s)
11:01 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided)
11:01 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 03s)
11:01 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided)
11:00 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 04s)
11:00 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided)
09:55 bblack: restarted varnish-be on cp3042
09:29 bblack: restarted varnish-be on cp1052
09:23 ema: restarted varnish-be on cp3043
09:21 ema: restarted varnish-be on cp3041
09:19 ema: restarted varnish-be on cp3032
09:14 ema: restarted varnish-be on cp3040
08:42 bblack: restarted varnish-be on cp1053
08:41 bblack: restarted varnish-be on cp1068
08:40 bblack: restarted varnish-be on cp1054
08:40 bblack: restarted varnish-be on cp1065
07:11 ema: cp3030 varnish-be restart due to 503s
03:16 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Sep 20 03:16:25 UTC 2017 (duration 7m 31s)
03:08 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 15m 57s)
02:31 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 08m 06s)
00:55 bblack: cp1066 - backend restart, mailbox lag, text
00:54 bblack: cp4016 - backend restart, mailbox lag, text
23:57 thcipriani@tin: Synchronized php-1.30.0-wmf.18/extensions/VisualEditor/ApiVisualEditor.php: SWAT: ApiVisualEditor: Fix checkbox label message handling with Message objects T176249 (duration: 00m 49s)
23:56 thcipriani@tin: Synchronized php-1.30.0-wmf.19/extensions/VisualEditor/ApiVisualEditor.php: SWAT: ApiVisualEditor: Fix checkbox label message handling with Message objects T176249 (duration: 00m 48s)
23:46 thcipriani@tin: Synchronized php-1.30.0-wmf.18/extensions/WikimediaEvents/modules/ext.wikimediaEvents.humanSearchRelevance.js: SWAT: Javascript timestamps are in ms, not s T174106 (duration: 00m 48s)
23:45 thcipriani@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/modules/ext.wikimediaEvents.humanSearchRelevance.js: SWAT: Javascript timestamps are in ms, not s T174106 (duration: 00m 50s)
23:39 thcipriani@tin: Synchronized php-1.30.0-wmf.18/includes/specials/SpecialRecentchangeslinked.php: SWAT: SpecialRecentchangeslinked: Unconditionally join on the page table T176228 (duration: 00m 49s)
23:36 thcipriani@tin: Synchronized php-1.30.0-wmf.19/includes/specials/SpecialRecentchangeslinked.php: SWAT: SpecialRecentchangeslinked: Unconditionally join on the page table T176228 (duration: 00m 49s)
23:29 thcipriani@tin: Synchronized php-1.30.0-wmf.18/includes/page/PageArchive.php: SWAT: Do not attempt to copy to ip_changes in PageArchive class (duration: 00m 49s)
23:27 thcipriani@tin: Synchronized php-1.30.0-wmf.19/includes/page/PageArchive.php: SWAT: Do not attempt to copy to ip_changes in PageArchive class (duration: 00m 50s)
23:17 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable jQuery 3 on meta.wikimedia.org T124742 (duration: 00m 50s)
22:58 eileen: manually running Omnirecipient.load from 2017-09-18 with accidentally low throttle setting
22:22 mutante: gerrit2001 - stopped gerrit with init.d script. moved init.d script to /root, started with systemctl
22:21 eileen: update civicrm from 9677cf7 0fb4676
22:15 mutante: gerrit2001 /bin/sh /var/lib/gerrit2/review_site/bin/gerrit.sh start
22:02 mutante: gerrit2001 - systemctl start gerrit
21:51 volans: disabled puppet on labpuppetmaster1001, testing cumin T175712
21:27 MaxSem: on terbium: mwscript populateIpChanges.php --wiki=enwikivoyage --force
21:23 bblack@neodymium: conftool action : set/pooled=yes; selector: cluster=cache_text,name=cp4008.ulsfo.wmnet
21:23 bblack@neodymium: conftool action : set/pooled=yes; selector: cluster=cache_text,name=cp4016.ulsfo.wmnet
21:22 bblack@neodymium: conftool action : set/pooled=yes; selector: cluster=cache_text,dc=eqiad,name=cp1066.*
21:18 MaxSem: re-ran populateIpChanges.php on group0 wikis
21:15 maxsem@tin: Synchronized php-1.30.0-wmf.19/maintenance/populateIpChanges.php: https://gerrit.wikimedia.org/r/#/c/379057/1 (duration: 00m 49s)
21:06 ema: cp4009: varnish-be restart, 503 spike
20:04 bblack@neodymium: conftool action : set/pooled=no; selector: cluster=cache_upload,name=cp4013.ulsfo.wmnet
20:03 bblack@neodymium: conftool action : set/pooled=no; selector: cluster=cache_text,name=cp4016.ulsfo.wmnet
19:50 bblack@neodymium: conftool action : set/pooled=no; selector: cluster=cache_upload,name=cp4005.ulsfo.wmnet
19:49 bblack@neodymium: conftool action : set/pooled=no; selector: cluster=cache_text,name=cp4008.ulsfo.wmnet
19:17 ejegg: updated payments-wiki from 26b16ea 896f16a
19:08 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.19
18:52 gehel: upgrading nodejs to 6.11 on maps-test2004 for testing - T171707
18:35 urandom: T160570 , T169940 : Upgrade restbase-ng environment to Cassandra 3.11.0-wmf5
18:26 arlolra: Updated Parsoid to 05a0965 T122965 , T175101 , T173384 , T173061 , T172896 , T174977 , T159894 )
18:17 ladsgroup@tin: Synchronized php-1.30.0-wmf.18/extensions/Wikidata/extensions/Wikibase/repo/tests/phpunit/includes/Content/EntityContentTest.php: Fix undoing merge operations that turned Items into a redirects, part II (T175984 ) (duration: 00m 48s)
18:15 ladsgroup@tin: Synchronized php-1.30.0-wmf.18/extensions/Wikidata/extensions/Wikibase/repo/includes/Content/EntityContent.php: Fix undoing merge operations that turned Items into a redirects, part I (T175984 ) (duration: 00m 49s)
18:08 ladsgroup@tin: Synchronized php-1.30.0-wmf.19/extensions/Wikidata/extensions/Wikibase/repo/tests/phpunit/includes/Content/EntityContentTest.php: Fix undoing merge operations that turned Items into a redirects, part II (T175984 ) (duration: 00m 50s)
18:07 ladsgroup@tin: Synchronized php-1.30.0-wmf.19/extensions/Wikidata/extensions/Wikibase/repo/includes/Content/EntityContent.php: Fix undoing merge operations that turned Items into a redirects, part I (T175984 ) (duration: 00m 50s)
18:00 arlolra@tin: Finished deploy [parsoid/deploy@a01064d]: Updating Parsoid to 05a0965
17:54 ebernhardson: T173464 start cirrussearch reindex of all wikis starting with zh
17:48 arlolra@tin: Started deploy [parsoid/deploy@a01064d]: Updating Parsoid to 05a0965
17:43 jynus: stopping db2010 and cloning to db2078
17:16 demon@tin: Finished scap: bootstrap wmf.19 (duration: 31m 04s)
16:45 demon@tin: Started scap: bootstrap wmf.19
16:44 demon@tin: scap aborted: bootstrap wmf.19 (duration: 01m 52s)
16:42 demon@tin: Started scap: bootstrap wmf.19
16:38 jynus@tin: Synchronized wmf-config/db-codfw.php: decom db1018 (duration: 00m 51s)
16:37 jynus@tin: Synchronized wmf-config/db-eqiad.php: decom db1018 (duration: 01m 01s)
16:34 demon@tin: Pruned MediaWiki: 1.30.0-wmf.17 [keeping static files] (duration: 01m 18s)
16:31 demon@tin: Pruned MediaWiki: 1.30.0-wmf.14 (duration: 03m 15s)
16:29 jynus: disable puppet on db1009
16:27 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Trying that again, got an error the first time (duration: 00m 52s)
16:26 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable RCFilters by default on cawiki, frwiki and hewiki (T157642 ) (duration: 00m 59s)
16:19 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable structured filters on watchlist on all wikis (duration: 00m 45s)
16:11 ema: pybal 1.14.0 uploaded to apt.w.o
15:54 catrope@tin: Finished scap: core, GuidedTour and WikimediaMessages patches for T176191 , T167262 and T175765 (duration: 20m 32s)
15:33 catrope@tin: Started scap: core, GuidedTour and WikimediaMessages patches for T176191 , T167262 and T175765
15:22 jynus: enabling firewall on db1020 (m2-master)
15:01 ema: restart varnish-be on cp1055, 503 spike
14:55 jynus: deploying firewall to m5 hosts
14:39 jynus: disabling puppet on all misc active mysql hosts
13:35 zeljkof: EU SWAT finished
13:32 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Rename Wikisaurus namespace on Wiktionary to "Thesaurus" (T174264) (duration: 00m 45s)
13:26 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Rename Wikisaurus namespace on Wiktionary to "Thesaurus" (T174264) (duration: 02m 55s)
13:19 gehel: upgrading elasticsearch plugins on relforge, including cold restart of the cluster - T173231
13:17 gehel: upgrading elasticsearch plugins on elasticsearch codfw, including cold restart of the cluster - T173231
13:15 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Rename Wikisaurus namespace on Wiktionary to "Thesaurus" (T174264) (duration: 02m 56s)
13:04 gehel: upgrading elasticsearch plugins on elastic2001 - T173231
12:55 moritzm: rebooting kubestage1001
11:25 moritzm: uploaded debdeploy 0.99.1 to apt.wikimedia.org (for trusty, jessie and stretch)
11:23 kartik@tin: Finished deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7
11:20 kartik@tin: Started deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7
10:53 volans: restarted nagios-nrpe-server on stat1005
10:14 jynus: shutting down mysql on db1018
09:42 moritzm: installing libxml2 security updates on trusty (Debian already fixed)
09:40 elukey: powercycle analytics1062 - no ssh, console com2 frozen
09:34 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: Enable the updateBetaFeaturesUserCounts logging channel - T175637 (duration: 00m 50s)
09:30 ema: cp1062 - varnish-backend-restart, mbox lag
08:40 mobrovac@tin: Finished deploy [cxserver/deploy@03c1eb1]: (no justification provided) (duration: 00m 52s)
08:39 mobrovac@tin: Started deploy [cxserver/deploy@03c1eb1]: (no justification provided)
08:09 mobrovac@tin: Finished deploy [cxserver/deploy@03c1eb1]: (no justification provided) (duration: 00m 13s)
08:09 mobrovac@tin: Started deploy [cxserver/deploy@03c1eb1]: (no justification provided)
07:58 kartik@tin: Finished deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7
07:58 kartik@tin: Started deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7
07:50 moritzm: installing bind9 regression update on trusty servers
07:35 kartik@tin: (no justification provided)
07:35 kartik@tin: Finished deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7
07:34 kartik@tin: Started deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7
02:28 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Sep 19 02:28:56 UTC 2017 (duration 7m 6s)
02:21 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 07m 04s)
01:52 demon@tin: Synchronized scap/plugins/updatewikiversions.py: Minor newline nitpick. I promise that's it (duration: 00m 46s)
01:41 demon@tin: Synchronized scap/plugins/updatewikiversions.py: Minor spacing nitpick, that's it for today (duration: 00m 45s)
01:17 demon@tin: Synchronized scap/plugins/updatewikiversions.py: add new hotness updatewikiversions.py (duration: 00m 45s)
01:15 demon@tin: Synchronized multiversion/: rm old and busted updateWikiversions (duration: 01m 07s)
00:12 foks: Remove 2FA from users Alan and Magicknight94 (retroactive log)
00:10 eileen: update civicrm from 187238f b6f7dc3
23:28 tstarling@tin: Synchronized php-1.30.0-wmf.18/includes/filerepo/file/LocalFile.php: (no justification provided) (duration: 00m 46s)
23:13 thcipriani@tin: Synchronized php-1.30.0-wmf.18/extensions/ArticleCreationWorkflow: SWAT: Only intercept users who can potentially create articles otherwise T176100 (duration: 00m 46s)
22:32 demon@tin: Synchronized wmf-config/CommonSettings.php: Use PrivateSettings.php directly (duration: 00m 46s)
21:57 bawolff: deployed patch T175900
21:18 krinkle@tin: Synchronized php-1.30.0-wmf.18/extensions/NavigationTiming/modules/ext.navigationTiming.js: Unbreak - T176105 (duration: 00m 48s)
21:16 mutante: repeating import of gerrit_2.13.8+git1-wmf.7 on install1002, followed by reprepro copy stretch-wikimedia jessie-wikimedia gerrit to make it available on stretch
21:06 mutante: uploaded gerrit 2.13.8+git1-wmf.7 for jessie-wikimedia on APT.wm.org
20:42 krinkle@tin: Finished deploy [jobrunner/jobrunner@57f5f47]: No-op sync - first time scap3 - T129148 (duration: 03m 23s)
20:38 krinkle@tin: Started deploy [jobrunner/jobrunner@57f5f47]: No-op sync - first time scap3 - T129148
20:17 arlolra@tin: (no justification provided)
20:15 arlolra@tin: Finished deploy [parsoid/deploy@a01064d]: Updating Parsoid to 05a0965
20:13 arlolra@tin: Started deploy [parsoid/deploy@a01064d]: Updating Parsoid to 05a0965
19:29 XioNoX: restarting pfw3-codfw for software upgrade
19:10 urandom: T160570 : Upgrading restbase-dev100[5-6].eqiad.wmnet to Cassandra 3.11.0-wmf5
19:07 smalyshev@tin: Finished deploy [wdqs/wdqs@2523836]: Deploy correct prefixes conf (duration: 01m 20s)
19:05 smalyshev@tin: Started deploy [wdqs/wdqs@2523836]: Deploy correct prefixes conf
19:03 smalyshev@tin: Finished deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS, take 4 (duration: 02m 36s)
19:02 no_justification: gerrit: restarting service, back in a moment
19:00 smalyshev@tin: Started deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS, take 4
18:56 urandom: T160570 : Upgrading restbase-dev1004.eqiad.wmnet to Cassandra 3.11.0-wmf5 (canary)
18:46 smalyshev@tin: Finished deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS, take 3 (duration: 02m 25s)
18:44 smalyshev@tin: Started deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS, take 3
18:37 smalyshev@tin: (no justification provided)
18:36 smalyshev@tin: Finished deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS (duration: 02m 26s)
18:34 smalyshev@tin: Started deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS
18:20 maxsem@tin: Synchronized wmf-config/: https://gerrit.wikimedia.org/r/#/c/378357/ https://gerrit.wikimedia.org/r/#/c/376791/ (duration: 00m 48s)
17:42 smalyshev@tin: Finished deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS (duration: 00m 21s)
17:42 smalyshev@tin: Started deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS
17:26 demon@tin: Synchronized wmf-config/ProductionServices.php: no-op/comment fix (duration: 00m 45s)
16:46 jynus: shuting down db1100 T175973
16:40 reedy@tin: Synchronized php-1.30.0-wmf.18/maintenance/populateIpChanges.php: perf improvements T175962 (duration: 00m 46s)
16:05 bblack: cp1063 - backend restart, mailbox lag
15:38 bblack: cp1072 - varnish backend restart, mailbox lag
15:28 bblack: cp1074 - backend restart for mailbox lag
15:25 bblack: cp1050 - backend restart for mailbox lahg
14:27 reedy@tin: Synchronized php-1.30.0-wmf.18/includes/filerepo/file/LocalFile.php: T175444 (duration: 00m 47s)
13:29 ppchelko@tin: Finished deploy [cpjobqueue/deploy@2c422f5]: Annotate the job event with pipeline property (duration: 00m 29s)
13:28 ppchelko@tin: Started deploy [cpjobqueue/deploy@2c422f5]: Annotate the job event with pipeline property
13:25 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Meta(Talk)Namespace configuration for be.wiktionary - T175950 (duration: 00m 46s)
13:18 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: pagePreviews: Stop A/B test on enwiki and dewiki - T176068 (duration: 00m 45s)
13:10 hashar@tin: Synchronized php-1.30.0-wmf.18/extensions/BetaFeatures: Temporary log all the executions of the update job. - T175637 (duration: 00m 46s)
13:09 hashar@tin: Synchronized wmf-config: New 'abusefilter-helper' configuration for en.wikipedia - T175684 (duration: 00m 49s)
12:59 moritzm: installing libxen updates
12:36 moritzm: installing tomcat8 security updates
10:40 ema: cp1099 - backend restart, mailbox lag
06:42 moritzm: installing freexl security updates
06:18 TimStarling: testing EtcdConfig in beta cluster
05:40 tstarling@tin: Synchronized wmf-config: just for consistency, should have no effect (gerrit 375108) (duration: 00m 49s)
02:36 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Sep 18 02:36:50 UTC 2017 (duration 7m 6s)
02:29 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 08m 04s)
12:49 _joe_: taking a full debug dump of mw1288 after depooling it
12:40 _joe_: restarting hhvm on some api appservers, to ease the cpu overlock
09:46 elukey: restart varnish backend on cp1052 - recurrent mailbox lag
08:11 elukey: restart varnish backend on cp1053 - recurrent mailbox lag
22:05 godog: compress older otrs directories to reclaim inodes - T171490
16:53 elukey: restart varnish-backend on cp1073 (cache upload) for mailbox lag
15:40 andrewbogott: rebooting labvirt1017
15:34 andrewbogott: rebooting labvirt1015 for T176044
21:50 mutante: contint2001 - systemctl start zuul
21:47 mutante: contint1001 - tmp disable puppet - contint2001 - test zuul unit file (gerrit 359016)
21:43 demon@tin: Finished deploy [gerrit/gerrit@48ad332]: 2.13.9 -- not being used yet though (duration: 00m 08s)
21:43 demon@tin: Started deploy [gerrit/gerrit@48ad332]: 2.13.9 -- not being used yet though
15:46 bblack: cp1049 - backend restart, mailbox lag
15:40 bblack: cp1099 - backend restart, mailbox lag
15:38 bblack: cp1072 - backend restart, mailbox lag
15:21 _joe_: uploaded td-agent-bit (fluentd-bit) to wikimedia-stretch T175527
13:13 moritzm: pruned hhvm bytecode cache on mw2256, led to log spam for an undefined variable after being out of service for hardware maintenance
13:10 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw2256.codfw.wmnet
13:10 addshore@tin: Synchronized wmf-config/CommonSettings-labs.php: beta: $wgWikiDiff2MovedParagraphDetectionCutoff = 25 BETA ONLY 2/2 (duration: 00m 45s)
13:08 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: beta: $wgWikiDiff2MovedParagraphDetectionCutoff = 25 BETA ONLY 1/2 (duration: 00m 47s)
11:22 jmm@puppetmaster1001: conftool action : set/pooled=no; selector: mw2256.codfw.wmnet
09:10 moritzm: installing tcpdump security updates on trusty hosts (Debian systems already fixed)
08:54 moritzm: installing libbluetooth updates
08:03 jynus: dropping mariadb analytics users on db1031, db1029
07:49 Amir1: running maintenance/updateCollation.php --force on Persian (fa) wikis (T173601 )
07:39 gehel: shutting down and masking elasticsearch on elastic1020 - T175951
07:35 gehel: depooling elastic1020 - T175951
07:28 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw2256.codfw.wmnet
00:46 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1100 (duration: 00m 46s)
23:51 mutante: phabricator - tested gerrit 354247 before merging using apache-fast-test from tin -restarted apache
23:37 catrope@tin: Synchronized wmf-config/: VRS config for Electron (T175868 ) (duration: 00m 47s)
23:34 catrope@tin: Synchronized php-1.30.0-wmf.18/extensions/VisualEditor/lib/ve/src/ui/styles/ve.ui.TableLineContext.css: T169389 (duration: 00m 45s)
23:33 catrope@tin: Synchronized php-1.30.0-wmf.18/extensions/ContentTranslation/modules/widgets/translator/ext.cx.translator.js: Fix JS error in CX (duration: 00m 46s)
23:29 catrope@tin: Synchronized wmf-config/CommonSettings.php: Give sysops the flow-create-board right on Flow wikis (T175934 ) (duration: 00m 45s)
23:27 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Give sysops the flow-create-board right on Flow wikis (T175934 ) (duration: 00m 45s)
23:22 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Use RemexHtml on mediawikiwiki and testwiki (T175095 ) (duration: 00m 46s)
23:19 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=acamar.wikimedia.org
23:18 mutante: acamar is back up and running - no failed units
23:13 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Allow bureaucrats to remove accountcreator on mediawikiwiki (T175903 ) (duration: 00m 46s)
23:12 mutante: acamar - staged BIOS upgrade in progress
23:09 mutante: acamar - done with upgrade - rebooting - it's depooled and removed from resolv.conf - T162850
22:56 mutante: depooled acamar (codfw dns recursor) for BIOS upgrade - installing firmware
22:56 dzahn@neodymium: conftool action : set/pooled=no; selector: name=acamar.*
22:55 maxsem@tin: Synchronized wmf-config/: Labs only: https://gerrit.wikimedia.org/r/378184 https://gerrit.wikimedia.org/r/378187 (duration: 00m 47s)
22:15 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: ACTRIAL going live: https://gerrit.wikimedia.org/r/#/c/378104/ (duration: 00m 46s)
21:22 maxsem@tin: Synchronized php-1.30.0-wmf.18/extensions/ArticleCreationWorkflow/: https://gerrit.wikimedia.org/r/#/c/378125/ (duration: 00m 47s)
20:34 demon@tin: Synchronized php-1.30.0-wmf.18/skins/MinervaNeue/resources/skins.minerva.userpage.icons/userpage.svg: I95d76339
20:18 demon@tin: Synchronized php-1.30.0-wmf.18/includes/api/ApiFeedWatchlist.php: Ibea5bd88
20:04 demon@tin: Synchronized php-1.30.0-wmf.18/extensions/OAuth/backend/MWOAuthDAO.php: I785230df
20:02 ppchelko@tin: Finished deploy [cpjobqueue/deploy@af9b590]: Emit broker statistics more frequentl (duration: 00m 30s)
20:02 ppchelko@tin: Started deploy [cpjobqueue/deploy@af9b590]: Emit broker statistics more frequentl
19:58 dcausse: banning elastic1020 to see if T175951 is caused by mixed versions of the ltr plugin
19:36 demon@tin: Synchronized php-1.30.0-wmf.18/skins/MinervaNeue/resources/skins.minerva.icons.images.scripts/userNormal.svg: fix svg syntax (duration: 00m 46s)
19:31 demon@tin: Synchronized php-1.30.0-wmf.17/skins/MinervaNeue/resources/skins.minerva.icons.images.scripts/userNormal.svg: fix svg syntax (duration: 00m 45s)
19:07 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.18
18:58 tgr@tin: Synchronized wmf-config/PrivateSettings.php: set Electron secret for T175868 (duration: 00m 48s)
18:57 tgr@tin: Synchronized private/PrivateSettings.php: set Electron secret for T175868 (duration: 00m 49s)
18:29 twentyafterfour: Phabricator: Deploying hotfix for T175942
18:10 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Configure enwiki to use CirrusSearch MLR by default T175772 (duration: 00m 50s)
18:03 Jamesofur: disabled oauth validation on metawiki/SUL for Teles after verification
17:57 bblack: cp2001 backend restart, mailbox lag
17:56 bblack: cp1074 backend restart, mailbox lag
17:18 eevans@tin: Started restart [electron-render/deploy@8dd5f13]: (no justification provided)
17:16 gehel@tin: Finished deploy [wikimedia/discovery/analytics@ab5d5c1]: moving discovery/analytics to scap3 (duration: 00m 04s)
17:16 gehel@tin: Started deploy [wikimedia/discovery/analytics@ab5d5c1]: moving discovery/analytics to scap3
16:59 krinkle@tin: Synchronized php-1.30.0-wmf.17/extensions/NavigationTiming: T104902 (duration: 01m 00s)
15:51 bblack: repool cp1063 (after wiping varnish storage + restarting all 3x main service daemons)
15:51 bblack: cp1064 backend restart, mailbox lag
15:46 bblack: depool cp1063 (upload eqiad) - seems to be having some iowait issues?
14:07 dereckson@tin: Synchronized php-1.30.0-wmf.18/extensions/CheckUser/specials/SpecialCheckUser.php: Fix Special cu for ip ranges (temp) (T175898 ) (duration: 00m 49s)
13:34 dcausse@tin: Synchronized wmf-config/CirrusSearch-production.php: Setup Cirrus MLR models for top 20 language AB test (duration: 00m 49s)
13:33 dcausse@tin: Synchronized wmf-config/CirrusSearch-common.php: Setup Cirrus MLR models for top 20 language AB test (duration: 00m 48s)
13:31 dcausse@tin: Synchronized wmf-config/InitialiseSettings.php: Setup Cirrus MLR models for top 20 language AB test (duration: 00m 50s)
11:14 jynus@tin: Synchronized wmf-config/db-eqiad.php: Remove all references to db1049, pool db1100 (duration: 00m 49s)
11:12 akosiaris: upload apertium-crh-tur_0.2.0~r81866-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
11:12 jynus@tin: Synchronized wmf-config/db-codfw.php: Remove all references to db1049 (duration: 00m 50s)
09:51 akosiaris@puppetmaster1001: conftool action : set/pooled=no; selector: name=wtp1025.eqiad.wmnet
09:43 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
09:43 akosiaris@puppetmaster1001: conftool action : set/weight=1; selector: name=wtp1025.eqiad.wmnet
09:41 gehel: re-enabling puppet on all cassandra nodes - T171704
09:41 akosiaris@tin: Finished deploy [parsoid/deploy@cec7d17]: test deploy using dsh groups. T165520 (duration: 09m 30s)
09:32 akosiaris@tin: Started deploy [parsoid/deploy@cec7d17]: test deploy using dsh groups. T165520
09:02 akosiaris@tin: Finished deploy [parsoid/deploy@cec7d17]: test deploy using dsh groups. T165520 (duration: 02m 25s)
09:00 akosiaris@tin: Started deploy [parsoid/deploy@cec7d17]: test deploy using dsh groups. T165520
08:13 gehel: merging cassandra refactoring for puppet - T171704
02:55 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Sep 14 02:55:10 UTC 2017 (duration 7m 28s)
02:47 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 07m 58s)
02:27 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 09m 27s)
00:46 MaxSem: restarted populateIpChanges to use updated code
00:26 twentyafterfour: everything appears to be up and running. Phabricator upgrade complete.
00:23 twentyafterfour: running puppet on phab1001 to restore files which get lost by scap deploy
00:21 twentyafterfour@tin: Finished deploy [phabricator/deployment@7e3a0a8]: (no justification provided) (duration: 00m 01s)
00:21 twentyafterfour@tin: Started deploy [phabricator/deployment@7e3a0a8]: (no justification provided)
00:21 twentyafterfour@tin: Finished deploy [phabricator/deployment@7e3a0a8]: (no justification provided) (duration: 00m 25s)
00:20 twentyafterfour@tin: Started deploy [phabricator/deployment@7e3a0a8]: (no justification provided)
00:18 twentyafterfour: Begin phabricator upgrade #phab-2017-09-13
23:48 dereckson@tin: Synchronized php-1.30.0-wmf.18/maintenance/populateIpChanges.php: Improve ip_changes update script to insert rows in batches (Gerrit:377918 ) (duration: 00m 49s)
23:29 dereckson@tin: Synchronized php-1.30.0-wmf.18/extensions/Wikidata/extensions/Wikibase/client: Split page set before constructing InjectRCRecordsJob (T175316 ) (duration: 00m 57s)
23:28 dereckson@tin: Synchronized php-1.30.0-wmf.18/extensions/Echo/modules/api/mw.echo.api.APIHandler.js: (no justification provided) (duration: 00m 48s)
23:26 dereckson@tin: Synchronized wmf-config/CommonSettings.php: Set $wgScoreSafeMode to false (T174413 ) (duration: 00m 50s)
22:06 maxsem@tin: Synchronized php-1.30.0-wmf.18/extensions/EventBus/: https://gerrit.wikimedia.org/r/#/c/377900/1 (duration: 00m 49s)
22:03 maxsem@tin: Synchronized php-1.30.0-wmf.18/extensions/ArticleCreationWorkflow/: https://gerrit.wikimedia.org/r/#/c/377900/1 (duration: 00m 49s)
21:55 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/377874/ (duration: 00m 47s)
21:23 ejegg: updated payments-wiki from ed2e481 26b16ea
grrrr: populating ip_changes on group1 wikis
20:43 bblack: re-routed ulsfo->eqiad around possible codfw damage as well - https://gerrit.wikimedia.org/r/#/c/377871/
20:34 bblack: deployed https://gerrit.wikimedia.org/r/#/c/377845/ (no gerrit bot, jenkins lint failing, etc) due to possible codfw routing issues
19:39 andrewbogott: rebooting labtestvirt2001
19:35 ppchelko@tin: Finished deploy [changeprop/deploy@86e8103]: Separate kafka metrics by rule (duration: 01m 18s)
19:34 ppchelko@tin: Started deploy [changeprop/deploy@86e8103]: Separate kafka metrics by rule
19:21 urandom: T172384 : Upgrading restbase-dev100[5-6] to Cassandra 3.11.0-wmf4
19:19 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.18
19:13 demon@tin: Synchronized php: symlink thingie (duration: 00m 49s)
19:11 urandom: T172384 : Upgrading restbase-dev1004 to Cassandra 3.11.0-wmf4 (canary)
18:34 niharika29@tin: Synchronized static/images/project-logos/: Change logo for huwiktionary T175483 (duration: 00m 49s)
18:27 mutante: gerrit2002 - re-enabled puppet - had forgotten to enable it again after recently tryning to test the gerrit->logstash change
18:26 niharika29@tin: Synchronized php-1.30.0-wmf.18/extensions/VisualEditor/: Make ve.dm.Change part of core module T175828 (duration: 00m 50s)
18:25 niharika29@tin: Synchronized docroot/mediawiki/ontology/: Add setup for https://www.mediawiki.org/ontology T171807 (duration: 00m 49s)
18:12 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable Timeless on frwiki T154371 (duration: 00m 50s)
17:55 gwicke: rolling restart of pdfrender service in equiad after hang T174916 T172815
16:42 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=wdqs1001.eqiad.wmnet
16:42 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=wdqs1002.eqiad.wmnet
16:28 gehel: starting decommissioning of wdqs100[12] - T175595
16:25 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs1002.eqiad.wmnet
16:25 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs1001.eqiad.wmnet
16:22 andrewbogott: rebooting labtestvirt2002
16:08 ottomata: restarting druid-brokers with increase in query cache size
15:23 demon@tin: Synchronized php-1.30.0-wmf.18/includes/filerepo/file/LocalFile.php: I2ecb6030
15:22 demon@tin: Synchronized php-1.30.0-wmf.18/includes/diff/DifferenceEngine.php: I2ecb6030
15:21 demon@tin: Synchronized php-1.30.0-wmf.17/includes/filerepo/file/LocalFile.php: I2ecb6030
15:20 demon@tin: Synchronized php-1.30.0-wmf.17/includes/diff/DifferenceEngine.php: I2ecb6030
15:09 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/AbuseFilter/: backporting I13051038
14:31 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,cluster=logstash
14:31 gehel: pooling new logstash servers (logstash100[7-9]) - T175045
14:23 mobrovac@tin: Finished deploy [cpjobqueue/deploy@60d0a78]: Start using the EventBus infrastructure for the updateBetaFeaturesUserCounts job - T175210 (duration: 00m 33s)
14:22 mobrovac@tin: Started deploy [cpjobqueue/deploy@60d0a78]: Start using the EventBus infrastructure for the updateBetaFeaturesUserCounts job - T175210
13:45 akosiaris: upload apertium-crh_0.1.0~r81872-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
13:30 urandom: T169936 : Converting RESTBase dev environment to size-tiered compaction
13:29 elukey: update puppet compiler's facts via ./modules/puppet_compiler/files/compiler-update-facts
13:23 gehel: initial installation of logstash100[7-9] - T175045
13:21 zeljkof: EU SWAT finished
13:13 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: Temporary lift account creation limits for WM United Kingdom workshop (T175700) (duration: 00m 49s)
12:14 moritzm: rebooting mc1015 (spare) for some tests
12:07 jynus: stopping wikidata rebuild maintenance script on terbium
12:07 bblack: cp1049 - backend restart, mailbox lag
11:57 jynus@tin: Synchronized wmf-config/db-eqiad.php: Set db1097 as the main api server for s4, Remove regular load from db1070 (duration: 00m 56s)
11:39 jynus: disabling semi-sync replication on the largest s5 database servers
11:03 ariel@tin: Finished deploy [dumps/dumps@3f119ab]: monitor will cd to dumps repo before each run (duration: 00m 02s)
11:03 ariel@tin: Started deploy [dumps/dumps@3f119ab]: monitor will cd to dumps repo before each run
11:02 joal@tin: Finished deploy [analytics/refinery@b2e8852]: Regular analytics weekly deploy (new jars, oozie patches) (duration: 03m 12s)
10:58 joal@tin: Started deploy [analytics/refinery@b2e8852]: Regular analytics weekly deploy (new jars, oozie patches)
10:52 volans: reboot lvs3001
10:06 akosiaris: upload apertium-cat-srd_0.9.0~r82238-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
10:06 akosiaris: upload apertium-srd-ita_0.9.5~r82237-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
09:36 moritzm: installing bind updates from Debian stable/oldstable SUA update (client tools only)
09:25 akosiaris: upload apertium-srd_0.10.0~r82237-1+wmf1_amd64.changes to apt.wikimedia.org/jessie-wikimedia/main T174988
09:24 akosiaris: upload apertium-tur_0.1.0~r81882-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
09:23 akosiaris: upload apertium-ita_0.10.0~r82237-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
09:23 akosiaris: upload apertium-cat_2.3.0~r82237-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
09:05 akosiaris: upload apertium_3.4.2~r68466-2+wmf2 to apt.wikimedia.org/jessie-wikimedia
08:55 elukey: restart varnish backend on cp1072 (upload) - mailbox expiry lag
08:24 moritzm: installing emacs security updates
07:56 akosiaris: bounce varnish on cp1062, mailbox lag
07:47 moritzm: rolling out hhvm-luasandbox 2.0.14 to the remaining hosts in eqiad (along with HHVM restarts) (T173705 )
07:08 moritzm: installing tcpdump security updates (fixing 90 CVE IDs...)
05:18 kartik@tin: Finished deploy [cxserver/deploy@ee0081d]: Update cxserver to 3baaa36
05:15 kartik@tin: Started deploy [cxserver/deploy@ee0081d]: Update cxserver to 3baaa36
03:12 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Sep 13 03:12:46 UTC 2017 (duration 7m 30s)
03:05 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 16m 28s)
03:03 jynus: dropping old users from phabricator database
03:00 eileen: update civicrm from ee7dda3 187238f
02:41 demon@tin: Synchronized wmf-config/mobile.php: rm unused config var (duration: 00m 46s)
02:27 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 08m 31s)
01:19 jynus: stopping mariadb @ db1048 to clone it to db1059
00:11 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Special:PageLanguage on mul.wikisource Fix correct wiki name for wgPageLanguageUseDB on www.wikisource.org T175622 (duration: 00m 49s)
23:53 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Do not deploy Timeless on fr.wiktionary for now" (duration: 00m 49s)
23:35 thcipriani@tin: Synchronized wmf-config: SWAT: Enable Flow on Newsletter_talk on mediawiki.org T175502 (duration: 00m 52s)
22:17 chasemp: labvirt1018:~# /sbin/reboot
22:00 bsitzmann@tin: Finished deploy [mobileapps/deploy@b11b75c]: Update mobileapps to 297b048 T174707 T175305 ) (duration: 04m 28s)
21:55 bsitzmann@tin: Started deploy [mobileapps/deploy@b11b75c]: Update mobileapps to 297b048 T174707 T175305 )
21:49 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/377527/2 T175725 (duration: 00m 49s)
21:37 MaxSem: Populating ip_changes in group0 wikis
21:33 ejegg: changed contact de-duplication duty cycle to 25 minutes
21:05 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.18
20:33 demon@tin: Finished scap: bootstrap wmf.18 (for real this time) (duration: 26m 23s)
20:06 demon@tin: Started scap: bootstrap wmf.18 (for real this time)
19:53 demon@tin: Finished scap: bootstrap wmf.18 (duration: 08m 14s)
19:45 demon@tin: Started scap: bootstrap wmf.18
19:05 papaul: maintenance complete on mw2256
17:51 bblack: cp3034 - reset fetch_maxchunksize to 0.25G (default) for fe+be
17:40 cwd: turned on dedupe
16:55 niharika29@tin: Finished deploy [iegreview/iegreview@69c4c3f]: Deploying iegreview with scap3 T129154 (duration: 00m 01s)
16:55 niharika29@tin: Started deploy [iegreview/iegreview@69c4c3f]: Deploying iegreview with scap3 T129154
16:40 ejegg: updated CiviCRM from 63288ae ee7dda3
16:35 niharika29@tin: Finished deploy [scholarships/scholarships@004635d]: Deploying scholarships with scap3 T129134 (duration: 00m 03s)
16:35 niharika29@tin: Started deploy [scholarships/scholarships@004635d]: Deploying scholarships with scap3 T129134
16:26 volans@sarin: conftool action : set/pooled=yes; selector: name=mw2100.codfw.wmnet
16:18 ejegg: updated SmashPig from dce4f0a dc62203910
15:43 _joe_: restarted ircecho on einsteinium
15:35 gehel: restart elasticsearch on relforge (last check of new plugin deployment) - T158560
15:22 bblack: manually setting fetch_maxchunksize=16MB via varnishadm on cp3034 fe+be instances
15:06 chasemp: disable puppet for lab* things for designate refactor rolleout (delayed a few minutes for a meeting)
15:04 moritzm: uploaded php5.5 5.5.38 to component/ci of jessie-wikimedia (for use in CI)
14:53 papaul: shutting down mw2256 for maintenance
14:43 elukey: add kafka-jumbo IPs to the kafka term of the analytics-in4 filter on cr1/cr2 eqiad
14:42 ema: cp1074 - backend restart, mailbox lag
14:39 bblack: manually setting fetch_maxchunksize=64MB via varnishadm on cp3034 fe+be instances
14:19 jynus@tin: Synchronized wmf-config/db-eqiad.php: Remove all references to db1059 (duration: 00m 44s)
14:18 jynus@tin: Synchronized wmf-config/db-codfw.php: Remove all references to db1059 (duration: 00m 45s)
14:10 gehel: restarting elasticsearch on elastic1020 to validate new plugin deployment - T158560
14:07 gehel: switching elasticsearch plugin deployment to .deb instead of scap on cirrus/eqiad - T158560
14:06 volans: testing reimage of mw2100 - T166300
14:06 gehel: restarting elasticsearch on elastic2001 to validate new plugin deployment - T158560
14:04 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1059, pool db1097 with low load (duration: 00m 45s)
14:02 gehel: switching elasticsearch plugin deployment to .deb instead of scap on cirrus/codfw - T158560
13:37 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs1005.eqiad.wmnet
13:37 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs1004.eqiad.wmnet
13:37 gehel: pooling wdqs100[45] - T171210
13:31 hashar@tin: Synchronized php-1.30.0-wmf.17/extensions/CentralAuth/includes/api/ApiSetGlobalAccountStatus.php: API: Unbreak setglobalaccountstatus locked=lock - T175462 (duration: 00m 44s)
13:26 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable WikidataPageBanner for Russian Wikimedia chapter wiki - T175356 (duration: 00m 45s)
13:25 hashar@tin: Synchronized php-1.30.0-wmf.17/includes/changes/ChangesListBooleanFilter.php: WLFilters: Respect default values - T174725 (duration: 00m 45s)
13:21 hashar@tin: Synchronized wmf-config/CommonSettings.php: Add Extension:Newsletter permissions to CommonSettings (duration: 00m 45s)
13:18 hashar@tin: Synchronized wmf-config/throttle.php: Lift account creation restrictions for WM Taiwan 10th anniversary - T175534 (duration: 00m 45s)
13:15 hashar@tin: Synchronized php-1.30.0-wmf.17/extensions/Wikidata/extensions/Wikibase/view/resources/jquery/wikibase/themes/default/jquery.wikibase.entityselector.css: Remove red highlighting on all entity selectors - T175525 (duration: 00m 45s)
13:12 hashar@tin: Synchronized wmf-config/Wikibase-production.php: Reduce wikiPageUpdaterDbBatchSize to 20 - T173710 (duration: 00m 45s)
09:33 moritzm: upgrading deployment servers to HHVM-Luasandbox 2.0.14
09:32 moritzm: upgrading mw2* to HHVM-Luasandbox 2.0.14
09:25 moritzm: upgrading mw1293-mw1295 (image scalers) to HHVM-Luasandbox 2.0.14 (along with HHVM restarts)
09:24 moritzm: upgrading mw1293-mw1205 (image scalers) to HHVM-Luasandbox 2.0.14 (along with HHVM restarts)
08:38 gehel: deploying elasticsearch plugins on relforge - T158560 (wrong ticket number before)
08:34 gehel: deploying elasticsearch plugins on relforge - T175159
08:28 moritzm: upgrading mw1189-mw1208 (API servers) to HHVM-Luasandbox 2.0.14 (along with HHVM restarts)
07:35 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool es1019 with full weight (duration: 00m 45s)
07:15 moritzm: upgrading mw1180-mw1188, mw1209-1220 (app servers) to HHVM-Luasandbox 2.0.14 (along with HHVM restarts)
02:29 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Sep 12 02:29:20 UTC 2017 (duration 6m 45s)
02:22 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 07m 31s)
02:10 bblack: cp1050 - backend restart, mailbox lag
02:08 bblack: cp1099 - backend restart, mailbox lagh
00:48 awight@tin: Finished deploy [ores/deploy@42c5663]: Cause ORES service restart (duration: 00m 18s)
00:48 awight@tin: Started deploy [ores/deploy@42c5663]: Cause ORES service restart
23:06 mutante: jenkins-bot says: java.lang.OutofMemoryError: PermGen space
22:30 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4026.*
22:30 dereckson@tin: Synchronized wmf-config/CommonSettings-labs.php: Fix MediaWiki\Auth\AbstractPreAuthenticationProvider case (Gerrit:377281 , no-op in prod) (duration: 00m 45s)
21:45 awight@tin: Finished deploy [ores/deploy@42c5663]: Try ORES filehandle fix on new cluster (duration: 13m 47s)
21:41 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4025.*
21:31 awight@tin: Started deploy [ores/deploy@42c5663]: Try ORES filehandle fix on new cluster
21:03 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4023.*
20:13 demon@tin: Pruned MediaWiki: 1.30.0-wmf.13 (duration: 02m 49s)
19:11 chasemp: adjust nodepool rate to 10
19:08 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool es1019 with low load (duration: 00m 46s)
19:01 mutante: restbase-dev1004:/srv# mv /home/eevans/cassandra-1502141526-pid13410.hprof /srv/eevans/ to free disk space
18:52 mutante: restbase - java[12418]: Error: Could not find or load main class org.wikimedia.cassandra.metrics.service.Service
18:40 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable responsive reference columns on wiktionaries, wikivoyages and enwiki https://gerrit.wikimedia.org/r/#/c/376573/ , https://gerrit.wikimedia.org/r/#/c/371630/ (duration: 00m 46s)
18:33 robh: firmware update on scs-a1-codfw and scs-c1-codfw has been complete and tested good T174475
18:29 chasemp: bump nodepool rate to 15 and max-servers to 18 (no restart as it picks up config periodically)
18:29 robh: firmware update on scs-a1-codfw and scs-c1-codfw has been started, awaiting completion T174475
18:24 robh: updating firmware on scs-a1-codfw and scs-c1-codfw
18:10 robh: scs-ulsfo has successfully updated firmware T174475
18:03 robh: upgrading firmware on scs-ulsfo
17:09 gehel@tin: Finished deploy [wdqs/wdqs@177d20a]: (no justification provided) (duration: 02m 39s)
17:07 gehel: wdqs weekly deployment (logging and throttling fixes)
17:06 gehel@tin: Started deploy [wdqs/wdqs@177d20a]: (no justification provided)
15:20 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4022.*
15:20 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4028.*
14:40 godog: roll-restart thumbor to apply https://gerrit.wikimedia.org/r/#/c/377264/ and upgrade to 1.4 - T173580 T174997
14:24 hashar: tin.eqiad.wmnet: reverted patch "WLFilters: Respect default values" that got merged but left undeployed
14:23 hashar: tin.eqiad.wmnet: reverted all four mediawiki-config that got merged but left undeployed and rebased the workspace
14:05 hashar: European SWAT cancelled due to deploy infrastructure issue
13:55 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 27s)
13:54 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 27s)
13:50 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 28s)
13:43 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 28s)
13:40 hashar@tin: Synchronized static/images/project-logos/huwiktionary.png: Change logo for huwiktonary - T175483 (duration: 00m 28s)
11:43 moritzm: installing file/libmagic security updates (stretch-specific, does not affect older distros)
10:30 jynus: restarting mysql @ labsdb1001
09:40 godog: roll-restart thumbor to apply changes for gif max animated area - T173580
09:39 mobrovac@tin: Finished deploy [cpjobqueue/deploy@e73a10f]: Initial deploy of cpjobqueue - T175281 (duration: 00m 20s)
09:39 mobrovac@tin: Started deploy [cpjobqueue/deploy@e73a10f]: Initial deploy of cpjobqueue - T175281
08:44 mobrovac@tin: Finished deploy [zotero/translators@a0c41c3]: Update translators to upstream e03695273 T174992 (duration: 00m 07s)
08:44 mobrovac@tin: Started deploy [zotero/translators@a0c41c3]: Update translators to upstream e03695273 T174992
08:16 volans: reimage script tests continue on mc100[1-2] T166300
07:59 moritzm: installing systemd updates from stretch point update
07:41 moritzm: installing remaining libonig security updates
02:41 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Sep 11 02:41:43 UTC 2017 (duration 7m 0s)
02:34 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 08m 10s)
21:00 bawolff: disabled Jforrester's gerrit account and flush sessions
13:41 elukey: restart Varnish backend on cp1073 (cache::upload) for mailbox expiry lag
13:13 elukey: restart cp1053's varnish backend for mailbox expiry lag and 503s - T175473
00:37 chasemp: labnodepool1001:~# service nodepool restart
00:36 bd808: wmcs: purged rabbitmq ceilometer_notifications.* queues
23:42 chasemp: labnodepool1001:~# service nodepool start
23:06 chasemp: freeswap && labcontrol1001:/home/rush# service rabbitmq-server restart
22:59 chasemp: restart nova-api and nova-network
22:49 chasemp: labnodepool1001:~# service nodepool stop
22:44 madhuvishy: Running service rabbitmq-server restart on labcontrol1001
20:41 legoktm@tin: Synchronized php-1.30.0-wmf.17/includes/filerepo/file/LocalFile.php: Fix issues related to comments and files in UI and API - T175443 T175444 (duration: 00m 46s)
01:12 ebernhardson@tin: Synchronized php-1.30.0-wmf.17/extensions/WikimediaEvents/modules/ext.wikimediaEvents.humanSearchRelevance.js: T171740 : Reduce annoyance of survey by enforcing minimum 2 days between showing survey to same browser (duration: 00m 46s)
23:46 ebernhardson@tin: Synchronized wmf-config/CirrusSearch-rel-survey.php: T171740 : Fix inverted sampling rates for human relevance survey (duration: 00m 47s)
23:38 mutante: netmon1002 - terminated unused screen session
23:09 mutante: phab1001, phab2001 - terminating 3 unused screen processes that were from iridium migration / data sync
21:54 mutante: gerrit2001 - restarting gerrit to test logstash config change (while not touching "live" gerrit)
20:24 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/Flow/includes/: bugs and stuff (duration: 00m 54s)
20:14 mutante: heze - rebooting for firmware upgrade
20:10 mutante: heze (bacula storage) - installing BIOS upgrade (T162850 )
20:10 demon@tin: Synchronized scap/scap.cfg: drop git_repo for now, T175041 (duration: 00m 46s)
19:43 mutante: zosma.codfw.wmnet - delete salt key, puppet node clean, puppet node deactivate, remove from Icinga,... (T138650 )
19:37 mutante: removing ganeti instance 'zosma' on ganeti2001 (T138650 )
18:17 demon@tin: Synchronized wmf-config/FeaturedFeedsWMF.php: no-op (duration: 00m 46s)
16:04 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/AbuseFilter/includes/Views/AbuseFilterViewExamine.php: T175338 , followup (duration: 00m 46s)
16:03 demon@tin: Synchronized php-1.30.0-wmf.17/includes/revisiondelete/RevDelLogList.php: typofix (duration: 00m 46s)
15:53 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4027.*
15:47 ejegg: increased cURL timeout for PayPal IPN confirmation to 14 sec
15:07 akosiaris@tin: Finished deploy [librenms/librenms@5554213]: Testing new scap keyholder_key config (duration: 00m 04s)
15:06 akosiaris@tin: Started deploy [librenms/librenms@5554213]: Testing new scap keyholder_key config
15:02 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4021.*
14:58 volans: testing wmf-auto-reimage also on mc1002 T166300 T164341
12:36 _joe_: restarting a check of the md2 devices on conf200* T162013 after reducing sync speed
12:36 _joe_: reduced raid resync max speed of raid devices on conf200* to 20000
10:35 hashar@tin: Synchronized php-1.30.0-wmf.17/extensions/ProofreadPage: Restore page status buttons - T175304 (duration: 00m 50s)
10:12 _joe_: stopped all md devices syncs on conf200* machines
10:09 _joe_: etcd cluster lost consensus T162013
10:07 volans: testing wmf-auto-reimage on mc1001 T166300 T164341
09:58 _joe_: running the same command on conf2002 too
09:48 _joe_: running `/usr/share/mdadm/checkarray --cron --all --idle --quiet` on conf2001 and conf2003 trying to reproduce consensus issues in T162013
09:25 moritzm: restarting apache on tegmen/einsteinium to pick up security update
09:09 moritzm: installing libonig security updates
09:08 mobrovac@tin: Finished deploy [restbase/deploy@0d39acf] (dev-cluster): Use double writing for mobileapps in the dev cluster, take #2 - T169940 (duration: 01m 01s)
09:07 mobrovac@tin: Started deploy [restbase/deploy@0d39acf] (dev-cluster): Use double writing for mobileapps in the dev cluster, take #2 - T169940
09:01 moritzm: installing remaining gnutls updates from jessie point release
08:13 moritzm: installing libgd2 security updates
08:11 mobrovac@tin: Started deploy [restbase/deploy@0d39acf] (dev-cluster): Use double writing for mobileapps in the dev cluster - T169940
08:11 mobrovac: restbase enabled back puppet in the dev cluster - T169940
07:53 moritzm: installing ruby2.3 security updates
07:32 mobrovac@tin: Finished deploy [restbase/deploy@0d39acf] (dev-cluster): (no justification provided) (duration: 07m 02s)
07:25 mobrovac@tin: Started deploy [restbase/deploy@0d39acf] (dev-cluster): (no justification provided)
07:21 mobrovac@tin: Finished deploy [restbase/deploy@e6aeeeb] (dev-cluster): (no justification provided) (duration: 00m 11s)
07:21 mobrovac@tin: Started deploy [restbase/deploy@e6aeeeb] (dev-cluster): (no justification provided)
00:42 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/AbuseFilter/includes/Views/AbuseFilterViewExamine.php: fix comment stuff (duration: 00m 46s)
23:42 bblack: cp1066 - depooled all traffic services, seems to have some bug causing 503 spikes
23:19 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.17
23:15 XenoRyet: Updated Smashpig from 8eb98c1 dce4f0a
22:41 niharika29@tin: Synchronized php-1.30.0-wmf.17/extensions/ArticleCreationWorkflow/: Bump extension version (duration: 00m 49s)
22:03 niharika29@tin: Finished scap: Deploying ArticleCreation Workflow on test2 wiki TT175302 (duration: 23m 00s)
21:40 niharika29@tin: Started scap: Deploying ArticleCreation Workflow on test2 wiki TT175302
21:23 demon@tin: Synchronized php-1.30.0-wmf.17/includes/api/ApiQueryRecentChanges.php: T175307 (duration: 00m 49s)
21:01 urandom: T169940 : Disabling puppet and restbase service in RESTBase dev
21:00 urandom: T169940 : Disabling changeprop in RESTBase dev environment
19:05 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/MobileFrontend/includes/specials/SpecialMobileHistory.php: T175161 (duration: 00m 49s)
18:48 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.17
18:47 demon@tin: Synchronized php: symlink update -> wmf.17 (duration: 00m 48s)
18:39 legoktm: restarted script to reparse all pages in parsoid for Linter (python3 parsoid_reparse.py http://parsoid.discovery.wmnet:8000 --sitematrix --linter-only --skip-closed https://aa.wikipedia.org/w/api.php ) - T161556
17:55 ejegg: disabled queue consumers for data cleaning job
17:52 ejegg: updated CiviCRM from c1ece1e 63288ae
17:35 otto@tin: Finished deploy [analytics/refinery@6b60f2c]: (no justification provided) (duration: 03m 23s)
17:32 otto@tin: Started deploy [analytics/refinery@6b60f2c]: (no justification provided)
17:31 otto@tin: Finished deploy [analytics/refinery@bdf6754]: (no justification provided) (duration: 00m 23s)
17:31 otto@tin: Started deploy [analytics/refinery@bdf6754]: (no justification provided)
17:23 otto@tin: Finished deploy [analytics/refinery@bdf6754]: (no justification provided) (duration: 03m 25s)
17:20 otto@tin: Started deploy [analytics/refinery@bdf6754]: (no justification provided)
16:13 demon@tin: Synchronized wmf-config/InitialiseSettings.php: Remove Extension:RelatedSites from zhwikivoyage (duration: 00m 50s)
15:32 mobrovac@tin: Finished deploy [restbase/deploy@77961d0]: Revert using MCS for the summary end point - T168848 (duration: 06m 18s)
15:25 mobrovac@tin: Started deploy [restbase/deploy@77961d0]: Revert using MCS for the summary end point - T168848
14:45 _joe_: manually running /usr/share/mdadm/checkarray --cron --all --idle --quiet on conf2001, trying to reproduce consensus issues in T162013
14:40 jynus: rebooting and upgrading es1019
13:54 godog: powercycle ms-be2023 - load through the roof and no login possible
13:54 oblivian@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=jobrunner,name=eqiad
13:14 zeljkof: EU SWAT finished
13:13 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add English Wiktionary as a client of Wikidata (T159316) (duration: 00m 49s)
13:12 zfilipin@tin: Synchronized dblists/wikidataclient.dblist: SWAT: Add English Wiktionary as a client of Wikidata (T159316) (duration: 00m 49s)
11:24 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1100 with 0 weight - T172679 (duration: 00m 49s)
09:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1100 depooled to s5 array - T172679 (duration: 00m 49s)
08:59 ariel@tin: Finished deploy [dumps/dumps@da05e9f]: add general info to siteinfo dumps (duration: 00m 02s)
08:58 ariel@tin: Started deploy [dumps/dumps@da05e9f]: add general info to siteinfo dumps
08:44 elukey: restart varnish backend on cp1063 - mailbox expiry lag
07:57 elukey: force re-mount of /mnt/hdfs on stat1005
07:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1100 - T172679 (duration: 00m 48s)
07:53 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1100 - T172679 (duration: 00m 49s)
07:45 mobrovac@tin: Finished deploy [restbase/deploy@b51c58c]: Use MCS for the summary endpoint - T168848 (duration: 06m 36s)
07:39 mobrovac@tin: Started deploy [restbase/deploy@b51c58c]: Use MCS for the summary endpoint - T168848
07:35 mobrovac@tin: Finished deploy [restbase/deploy@b51c58c] (staging): Use MCS for the summary endpoint (duration: 01m 43s)
07:33 mobrovac@tin: Started deploy [restbase/deploy@b51c58c] (staging): Use MCS for the summary endpoint
07:20 oblivian@tin: Synchronized rpc/RunSingleJob.php: Adding the RunSingleJob.php rpc endpoint for jobrunners (duration: 00m 56s)
06:07 eileen: civicrm updated from 73a24b4 c1ece1e
03:13 eileen: seconds behind=78
03:08 eileen: seconds behind 55
02:52 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Sep 7 02:52:05 UTC 2017 (duration 7m 15s)
02:44 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 06m 15s)
02:43 eileen: restarting groupmember.load after hacking out cache delete calls
02:40 eileen: update civicrm from 764bfe1 73a24b4
02:24 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 07m 23s)
02:05 eileen: replag 217 behind
01:56 eileen: replag 197 seconds
01:47 eileen: replag 23 seconds
01:29 eileen: replag 0
01:22 eileen: replag 550 sec behind master
01:13 eileen: replag delay 422
01:12 eileen: job killed
00:49 eileen: 8389 contacts now
00:49 eileen: 7971 contacts so far
00:46 eileen: started Omnigroupmember.load
00:24 twentyafterfour: phabricator update deployed
00:12 twentyafterfour@tin: Finished deploy [phabricator/deployment@c265c1a]: (no justification provided) (duration: 00m 24s)
00:11 twentyafterfour@tin: Started deploy [phabricator/deployment@c265c1a]: (no justification provided)
00:07 twentyafterfour: deploying phabricator update 2017-09-06 https://phabricator.wikimedia.org/project/view/2980/
23:41 twentyafterfour@tin: Synchronized php-1.30.0-wmf.16/extensions/WikimediaEvents/: Sync Change-Id: I7ae522 T174387 (duration: 00m 49s)
23:28 twentyafterfour@tin: Synchronized php-1.30.0-wmf.17/extensions/WikimediaEvents/: Sync Change-Id: I7ae522 T174387 (duration: 00m 49s)
23:23 twentyafterfour@tin: Synchronized wmf-config/CirrusSearch-common.php: CirrusSearch-common.php goes last (duration: 00m 48s)
23:21 twentyafterfour@tin: Synchronized wmf-config/InitialiseSettings.php: Now sync initializesettings (duration: 00m 49s)
23:19 twentyafterfour@tin: Synchronized wmf-config/CirrusSearch-rel-survey.php: sync the added file first (duration: 00m 49s)
23:17 twentyafterfour@tin: scap failed: average error rate on 9/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details)
23:12 twentyafterfour@tin: Synchronized wmf-config/: roll-back due to huge error rate spike (duration: 00m 51s)
23:10 twentyafterfour@tin: Synchronized wmf-config/: deploy config for CirrusSearch human relevancy survey. Change-Id: I272c69 T174106 (duration: 00m 52s)
22:18 no_justification: prior deploy was no-op
22:18 demon@tin: Finished deploy [gerrit/gerrit@d4f9a77]: (no justification provided) (duration: 00m 07s)
22:18 demon@tin: Started deploy [gerrit/gerrit@d4f9a77]: (no justification provided)
22:11 bsitzmann@tin: Finished deploy [mobileapps/deploy@507a479]: Update mobileapps to 2cb6281 T168848 T169277 T169274 T162179 T164033 T167921 T174698 T168848 T174808 ) (duration: 04m 53s)
22:06 bsitzmann@tin: Started deploy [mobileapps/deploy@507a479]: Update mobileapps to 2cb6281 T168848 T169277 T169274 T162179 T164033 T167921 T174698 T168848 T174808 )
21:39 niharika29@tin: Synchronized wmf-config/CommonSettings-labs.php: Config for ArticleCreationWorkflow T175054 (duration: 00m 50s)
21:28 ejegg: updated payments-wiki from bf97cd2 ed2e481
21:24 ejegg: updated CiviCRM from f9c373e 764bfe1
21:17 ottomata: rebooting kafka-jumbo1004
20:34 arlolra: Updated Parsoid to f9d367ea T169342 )
20:27 arlolra@tin: Finished deploy [parsoid/deploy@f07ac8c]: Updating Parsoid to f9d367ea
20:18 arlolra@tin: Started deploy [parsoid/deploy@f07ac8c]: Updating Parsoid to f9d367ea
19:53 ottomata: reimaging kafka-jumbo* with stretch
19:48 chasemp: reboot labvirt1018
18:50 maxsem@tin: Synchronized wmf-config/: just to make sure... (duration: 00m 50s)
18:48 maxsem@tin: Synchronized wmf-config/CommonSettings.php: revert (duration: 00m 49s)
18:33 maxsem@tin: Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/370292/3 (duration: 00m 49s)
18:25 maxsem@tin: Synchronized wmf-config/CommonSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/370291/2 (duration: 00m 49s)
18:19 maxsem@tin: Synchronized wmf-config/Wikibase.php: https://gerrit.wikimedia.org/r/#/c/376317/2 (duration: 00m 48s)
18:18 maxsem@tin: Synchronized wmf-config/Wikibase-production.php: https://gerrit.wikimedia.org/r/#/c/376317/2 (duration: 00m 48s)
18:09 maxsem@tin: Synchronized portals: (no justification provided) (duration: 00m 49s)
18:09 ejegg: rolled payments-wiki back to bf97cd2
18:08 maxsem@tin: Synchronized portals/prod/wikipedia.org/assets: (no justification provided) (duration: 00m 50s)
18:05 ejegg: updated payments-wiki from bf97cd2 6911158
17:53 bblack: cp1048 - varnish backend restart for mailbox lag...
17:23 demon@tin: Synchronized wmf-config/FeaturedFeedsWMF.php: code cleanup (duration: 00m 49s)
17:20 mutante: added LDAP user schoenbaechler to WMF group (T175156 )
17:16 mutante: modify-ldap-group on terbium is broken
17:02 ejegg: disabled creating CiviMail records for Thank You emails
16:51 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/Flow/includes/: I284b5aa
16:41 chasemp: disable puppet for cloud things to test changes
16:40 bblack: cp1072 - varnish backend restart (mailbox lag)
16:40 bblack: cp1062 - varnish backend restart (mailbox lag)
15:55 ejegg: re-enabled IPN notification at PayPal console (48 minutes ago)
15:53 akosiaris: powercycling lvs3001
15:28 papaul: firmware upgrade on mw2256
15:11 bblack: cp1049 - restart varnish backend (mailbox lag)
15:07 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool es1019 (duration: 00m 49s)
15:06 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2040 (duration: 00m 49s)
14:17 gehel: wdqs1005 is coming up after a few reimaging issues, expect some icinga noise...
13:19 jynus: restarting and upgrading db2040
13:13 reedy@tin: Synchronized wmf-config/Wikibase-production.php: Move some wikidata config T174962 (duration: 00m 48s)
13:12 reedy@tin: Synchronized wmf-config/Wikibase-labs.php: Move some wikidata config T174962 (duration: 00m 49s)
13:07 reedy@tin: Synchronized wmf-config/throttle.php: Throttle exception T175113 (duration: 00m 49s)
12:08 moritzm: installing libapache2-mod-perl update from jessie 8.9 update
11:41 moritzm: installing gtk+2.0 update from jessie 8.9 update
11:24 elukey: temporarily raise kafka log4j authorizer verbosity to DEBUG on kafka1012 - T173493
11:21 moritzm: installing gnutls update from jessie 8.9 and stretch 9.1 point updates
11:20 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2040 (duration: 00m 49s)
11:20 godog: reimage restbase1008 with cassandra 3 - T169939
11:15 marostegui: Disable puppet on db1100 for mydumper/myloader
10:26 moritzm: installing perl update from stretch 9.1 point release
10:25 moritzm: installing perl update from jessie 8.9 point release
10:12 godog: reimage restbase1010 with cassandra 3 - T169939
09:39 moritzm: installing libgd security updates
09:28 moritzm: installing libonig security uodates#
09:05 jynus: disabling puppet on most db hosts to merge firewall changes safely
08:53 godog: reimage restbase1009 with cassandra 3 - T169939
08:49 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T174948 Add WMDE log channel (duration: 00m 49s)
08:43 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T110170 Enable Newsletter on mediawikiwiki (duration: 00m 51s)
07:15 marostegui: Stop MySQL on db1049 to copy its content to db1100 - https://phabricator.wikimedia.org/T172679
06:46 moritzm: installing php-luasandbox 2.0.14 on API canaries along with HHVM restart (T173705 )
06:37 marostegui: Truncate l10n_cache table across production - T150306
04:08 demon@tin: Synchronized wmf-config/throttle.php: no-op (duration: 00m 49s)
03:14 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Sep 6 03:14:16 UTC 2017 (duration 7m 6s)
03:07 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 14m 49s)
02:31 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 08m 18s)
away: disabled IPN messages at the PayPal console
00:14 Reedy: that was T174747 Adjust language icon color
00:14 reedy@tin: Synchronized php-1.30.0-wmf.16/skins/MinervaNeue: (no justification provided) (duration: 00m 49s)
00:10 reedy@tin: Synchronized php-1.30.0-wmf.16/extensions/Scribunto/tests/phpunit/: Fix broken test (duration: 00m 49s)
00:09 reedy@tin: Synchronized php-1.30.0-wmf.17/extensions/Scribunto/tests/phpunit/: Fix broken test (duration: 00m 50s)
23:58 reedy@tin: Synchronized wmf-config/: Test ArticleCreationWorkflow extension on the Beta Cluster (duration: 00m 51s)
23:39 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: Enable AbuseFilter runtime profile T161059 (duration: 00m 49s)
23:33 foks: 2FA disabled for Omshivaprakash (T175075 )
22:17 robh: rebooting mw1163 as its serial console is locked up
22:07 demon@tin: Synchronized php-1.30.0-wmf.16/extensions/FlaggedRevs/frontend/specialpages/reports/: kill some warnings and stuff (duration: 02m 55s)
22:06 demon@tin: Synchronized php-1.30.0-wmf.16/extensions/FlaggedRevs/frontend/specialpages/reports/: (no justification provided) (duration: 03m 13s)
20:48 ejegg: re-enabled donation queue consumer and thank you mailer
20:40 ejegg: updated civicrm from 820bd98 f9c373e
20:37 ejegg: disabled Donation queue consumer and thank-you mail sender
19:38 gilles@tin: Synchronized private/PrivateSettings.php: Add Thumbor username to Swift configuration (duration: 00m 48s)
19:33 jynus: rebuilding querycache table on tewiktionary.dbstore1002, it had crashed
19:28 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: wmf.17 on group0
19:05 Reedy: deleted querycache rows where qc_type = '' on all wikis T174513
18:11 ejegg: updated fundraising tools from 9554229 8d39806
18:11 demon@tin: Finished scap: wmf.17 bootstrap (duration: 47m 00s)
17:38 bblack: cp1074 - restart varnish backend, mailbox lag
17:24 demon@tin: Started scap: wmf.17 bootstrap
17:14 demon@tin: Pruned MediaWiki: 1.30.0-wmf.12 (duration: 02m 32s)
16:59 andrewbogott: reducing the number of concurrent nova-conductor workers. May cause hiccups as services restart
16:58 elukey: ran authdns-update from ns1.w.o after https://gerrit.wikimedia.org/r/374385 to create electcom.wikimedia.org
16:51 jynus: reloading dbproxy1005 to repoing to db1009 again- things seem stable right now
16:45 elukey: add new virtualhost electcom.wikimedia.org to the appservers apache config - https://gerrit.wikimedia.org/r/374389 (implies apache config reload)
16:33 mattflaschen@tin: Synchronized php-1.30.0-wmf.16/: Prepare to enable RCFilters (WLFilters) on Watchlist, but without i18n changes (reverted) (duration: 12m 41s)
16:14 addshore: addshore@terbium:~$ mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki mediawikiwiki --extension newsletter
16:08 akosiaris: reboot all kubernetes boxes T170119
15:43 matt_flaschen: 'Scap sync failed on i18n, so I'll deploy just the non-i18n ones'
15:36 mattflaschen@tin: scap aborted: Prepare to enable RCFilters (WLFilters) on Watchlist (duration: 08m 42s)
15:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1083 - T174509 (duration: 00m 47s)
15:27 mattflaschen@tin: Started scap: Prepare to enable RCFilters (WLFilters) on Watchlist
14:30 reedy@tin: Synchronized wmf-config/: phpcs (duration: 00m 47s)
14:28 reedy@tin: Synchronized tests/: phpcs (duration: 00m 46s)
14:26 reedy@tin: Synchronized errorpages/404.php: phpcs (duration: 00m 45s)
14:25 reedy@tin: Synchronized docroot/noc/conf/highlight.php: Fix links for dblists in highlight.php (duration: 00m 44s)
14:15 zeljkof: EU SWAT finished
14:13 zfilipin@tin: Synchronized php-1.30.0-wmf.16/includes/EditPage.php: SWAT: Re add wpScrolltop id in EditPage (T174723) (duration: 00m 45s)
13:47 zfilipin@tin: Synchronized php-1.30.0-wmf.16/extensions/ContentTranslation/: SWAT: encodeURIComponent title to escape / properly (T174792) (duration: 00m 48s)
13:30 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add abusefilter-view-private to rollbackers in zhwiki (T174978) (duration: 00m 46s)
13:28 godog: reimage restbase2005 - T169939
13:17 marostegui: Drop pr_index tables from where ProofreadPage isn't enabled - T174782
13:14 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Update logo for sr.wikibooks.org (T172284) (duration: 00m 46s)
12:24 hashar: ci: switched mediawiki/core mediawiki/vendor php5.5 jobs from trusty to jessie - T161882
12:15 ladsgroup@tin: Synchronized wmf-config/Wikibase-production.php: Enable sendEchoNotification for enwiki, dewiki, frwiki (T142102 ) (duration: 00m 46s)
11:54 marostegui: Drop reader_feedback, reader_feedback_history, reader_feedback_pages tables - T174586
11:07 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: (Actually) Enable the ArticlePlaceholder on sqwiki (T174335 ) (duration: 00m 45s)
11:05 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on sqwiki (T174335 ) (duration: 00m 46s)
10:38 hashar: Nodepool / CI are fully backup and processing queued jobs
10:25 hashar: Restarting Nodepool. I have managed to spawn instances manually for contintcloud tenant
10:18 _joe_: stopping puppet, nginx on mw2249 for experimentation
10:11 ema: reboot cp4024 T174891
10:08 godog: copy python-logstash from jessie-wikimedia to stretch-wikimedia
09:59 hashar: Stopped Nodepool on labnodepool1001.eqiad.wmnet
09:52 akosiaris: reimage kubernetes[12]00[1-4] T170119
08:49 moritzm: upgrading canary app servers to hhvm-luasandbox 2.0.14 (T173705 )
08:47 moritzm: uploaded php-luasandbox/hhvm-luasandbox 2.0.14 to apt.wikimedia.org (T173705 )
07:14 moritzm: installing libgd security updates on canary app servers (along with hhvm restart)
07:03 _joe_: launching manually 3 workers for refreshLinks jobs on commons, T173710
06:32 marostegui: Deploy alter table on db1083 - T174509
06:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1083 - T174509 (duration: 00m 46s)
06:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1089 - T174509 (duration: 00m 46s)
06:09 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1089 - T174509 (duration: 00m 55s)
02:30 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Sep 5 02:30:03 UTC 2017 (duration 6m 37s)
02:23 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 07m 28s)
23:32 Dereckson: Set e-mail to Hexabot bot account to allow account recovery (T174973 )
22:07 legoktm: starting script to reparse all pages in parsoid for Linter (python2 parsoid-reparse.py http://parsoid.discovery.wmnet:8000 --sitematrix --linter-only --skip-closed https://aa.wikipedia.org/w/api.php ) - T161556
17:50 moritzm: uploaded debdeploy 0.0.99-2 to apt.wikimedia.org
17:19 gehel@tin: Finished deploy [wdqs/wdqs@1caaa30]: weekly WDQS deployment (duration: 02m 46s)
17:17 gehel@tin: Started deploy [wdqs/wdqs@1caaa30]: weekly WDQS deployment
16:29 mobrovac@tin: Finished deploy [restbase/deploy@8c9c436]: (no justification provided) (duration: 06m 23s)
16:23 mobrovac@tin: Started deploy [restbase/deploy@8c9c436]: (no justification provided)
16:10 mobrovac: restbase depooled restbase10(0[89]|10) and restbase2005 for T169939
16:07 _joe_: rolling restart of pybal in codfw/eqiad low-traffic pools for picking up the new jobrunner LVS endpoint
15:42 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,service=nginx
14:51 godog: reimage restbase2003 for use with cassandra 3 / jbod - T169939
14:27 moritzm: enabled LDAP authentication for ganglia.wikimedia.org
13:37 gehel: rolling restart of elasticsearch/logstash to cleanup unused plugins - T174933
13:34 gehel: delete leftover grafana-dashboards index on elasticsearch logstash cluster
13:22 gehel: restart elasticsearch on logstash1006 to test plugin removal - T174933
12:26 moritzm: uploaded debdeploy 0.0.99-1+trusty/trusty-wikimedia to apt.wikimedia.org
12:18 marostegui: Stop MySQL on db1037 as it is going to be decommissioned - T174902
12:10 elukey: rolling restart of zookeeper on conf100[123] for jvm security updates
11:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1037 as it will be decommissioned - T174902 (duration: 00m 46s)
11:52 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1037 as it will be decommissioned - T174902 (duration: 00m 46s)
11:45 godog: restart varnish on cp1099 - mailbox lag
10:22 ariel@tin: Finished deploy [dumps/dumps@0b32eef]: fix abstract dumps variants regression (duration: 00m 02s)
10:22 ariel@tin: Started deploy [dumps/dumps@0b32eef]: fix abstract dumps variants regression
09:51 moritzm: uploaded debdeploy 0.0.99-1+jessie/jessie-wikimedia to apt.wikimedia.org
09:41 mobrovac@tin: Started restart [mathoid/deploy@44ea6d8]: Restart for libxml update
09:36 moritzm: uploaded debdeploy 0.0.99-1/stretch-wikimedia to apt.wikimedia.org
08:52 elukey: restart zookeeper on conf2003 for jvm security updates
08:48 marostegui: Stop MySQL on db1045 as it will be decommissioned - T174806
08:47 moritzm: installing mercurial security updates
08:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1045 as it will be decommissioned - T174806 (duration: 00m 46s)
08:39 moritzm: installing libgcrypt20 security updates
08:39 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1045 as it will be decommissioned - T174806 (duration: 00m 48s)
08:33 godog: upload scap 3.7.0-1 to apt.w.o - T127762
07:39 moritzm: installing gnupg security updates
07:03 marostegui: Deploy alter table on s3 codfw master with replication on kpbwiki - T168661
07:02 _joe_: starting additional runJobs instance for htmlcacheupdate on commons T173710
04:52 marostegui: Deploy alter table on db1052 - T168661
02:44 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Sep 4 02:44:15 UTC 2017 (duration 6m 52s)
02:37 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 08m 17s)
17:52 elukey: depooled cp4024 (ulsfo upload) due to kernel errors in dmesg
15:57 ema: restart varnish backend on cp1073 (mailbox lag)
09:41 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4024.ulsfo.wmnet,service=varnish-fe
09:41 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4024.ulsfo.wmnet,service=nginx
09:25 ema: cp4024 is down, powercycled
09:13 _joe_: manually fixed value of /conftool/v1/pools/eqiad/api_appserver/apache2/mw1208.eqiad.wmnet in codfw to allow replication to restart
00:59 legoktm: legoktm@terbium:~$ mwscript extensions/WikimediaMaintenance/filebackend/setZoneAccess.php --wiki=hiwikiversity --backend=local-multiwrite # T174859
