Server Admin Log/Archive 33

2017-12-30

18:22 ejegg: disabled failing silverpop data fetch jobs

2017-12-29

18:31 mutante: gerrit: restart service to apply change from Dec 20th, avoid logspam due to unapplied config change
15:38 ariel@tin: Finished deploy [dumps/dumps@51d4fd6]: enable pageslogging dump in parallel jobs (duration: 00m 02s)
15:38 ariel@tin: Started deploy [dumps/dumps@51d4fd6]: enable pageslogging dump in parallel jobs
14:46 apergos: restarted puppetdb on nitrogen, puppetdb refusing to hand out facts again
08:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1110 (duration: 00m 52s)
08:19 marostegui: Stop replication in sync on dbstore1002.s5 and db1110
08:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1110 to reimport dewiki.imagelinks on dbstore1002 (duration: 00m 52s)
07:43 marostegui: Fixing dbstore1002 dewiki.imagelinks

2017-12-28

14:18 marostegui: Power cycle pc2005 as it is down

2017-12-27

19:54 godog: power reset ms-be1033
19:47 godog: reboot ms-be1033 - T183724
17:00 twentyafterfour: restarting apache on phab1001
15:56 mobrovac@tin: Started restart [electron-render/deploy@94d27d7]: Bounce Electron, stuck - T174916

2017-12-26

16:33 mobrovac@tin: Finished deploy [mathoid/deploy@63b2ddc]: Bring back Mathoid in codfw to v0.6.5 in sync with eqiad - T179419 T172767 (duration: 01m 44s)
16:31 mobrovac@tin: Started deploy [mathoid/deploy@63b2ddc]: Bring back Mathoid in codfw to v0.6.5 in sync with eqiad - T179419 T172767
16:27 mutante: re-generated DNS zones to add new language lfn (Lingua Franca Nova) (T183561)
12:00 Amir1: ladsgroup@terbium:~$ mwscript extensions/Cognate/maintenance/populateCognatePages.php --wiki hifwiktionary (T180785)
11:59 Amir1: ladsgroup@terbium:~$ mwscript extensions/Cognate/maintenance/populateCognateSites.php --wiki aawiktionary --site-group wiktionary (T180785)
11:05 ariel@tin: Finished deploy [dumps/dumps@8c44d0a]: re-enable file size reporting for dump files still being written (duration: 00m 02s)
11:05 ariel@tin: Started deploy [dumps/dumps@8c44d0a]: re-enable file size reporting for dump files still being written
08:34 akosiaris: migrate logstash1007 off ganeti1006 T181121
08:21 akosiaris: migrate logstash1007 off ganeti1006
03:52 madhuvishy: restart pdfrender on scb1002
03:50 madhuvishy: restart pdfrender on scb1004
03:32 madhuvishy: restart pdfrender on scb1003 following icinga page
03:28 madhuvishy: restart pdfrender on scb1001 following icinga page

2017-12-24

16:10 _joe_: restarted pdfrenderer on scb1002

2017-12-23

12:30 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: BETA ONLY T183500 Re Enable AdvancedSearch on deployment.wikimedia.beta (duration: 00m 51s)
12:20 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: BETA ONLY T183499 Fix wgContentLanguage on deployment.wikimedia.beta (duration: 00m 52s)

2017-12-22

18:31 mobrovac@tin: Finished deploy [mathoid/deploy@7c5f8e2]: Better handling of chem rules (codfw only) - T183557 (duration: 02m 45s)
18:28 mobrovac@tin: Started deploy [mathoid/deploy@7c5f8e2]: Better handling of chem rules (codfw only) - T183557
16:42 demon@tin: Pruned MediaWiki: 1.31.0-wmf.11 [keeping static files] (duration: 01m 16s)
16:05 demon@tin: Synchronized README: forcing co-master sync (duration: 00m 51s)
15:56 demon@tin: Synchronized scap/plugins/clean.py: no-op (duration: 00m 51s)
15:52 demon@tin: Pruned MediaWiki: 1.31.0-wmf.10 [keeping static files] (duration: 01m 18s)
15:50 demon@tin: Pruned MediaWiki: 1.31.0-wmf.10 [keeping static files] (duration: 01m 47s)
14:48 bblack: esams TLS cert switch from digicert-2016 to digicert-2017
14:18 mobrovac@tin: Finished deploy [mathoid/deploy@6c29c09]: Update Mathoid to v0.7.0 in CODFW only to prefill storage, take 2 - T179419 T172767 (duration: 04m 23s)
14:13 mobrovac@tin: Started deploy [mathoid/deploy@6c29c09]: Update Mathoid to v0.7.0 in CODFW only to prefill storage, take 2 - T179419 T172767
14:05 mobrovac@tin: Finished deploy [mathoid/deploy@7f39b71]: Update Mathoid to v0.7.0 in CODFW only to prefill storage - T179419 T172767 (duration: 02m 57s)
14:02 mobrovac@tin: Started deploy [mathoid/deploy@7f39b71]: Update Mathoid to v0.7.0 in CODFW only to prefill storage - T179419 T172767
13:37 mobrovac: restbase depool restbase2008 for T179419
13:17 bblack: repooling cp4032 - T183176
06:44 jynus: start manual backup of db1029 onto db1056

2017-12-21

19:25 robh: cp4032 going offline for memory swap
17:24 jynus: starting manual backup of x1-master onto db1055
17:01 volans: debugging Icinga notes_url (no side effect expected but logging it in case there will be) T170353
16:56 volans: restarted ircecho to pick up gerrit/399658
15:43 joal@tin: Finished deploy [analytics/refinery@92f9318]: Deploying for new pageview_top_bycountry job (duration: 04m 40s)
15:39 joal@tin: Started deploy [analytics/refinery@92f9318]: Deploying for new pageview_top_bycountry job
12:39 elukey: restart druid historical/broker to apply new jvm settings to druid public workers (druid100[456] - https://gerrit.wikimedia.org/r/399617)
12:07 elukey@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw1333.*.eqiad.wmnet
11:43 _joe_: refreshing puppet facts on the puppet compilers
10:54 jynus@tin: Synchronized wmf-config/db-eqiad.php: Remove db1055 & db1056 (duration: 00m 51s)
10:52 jynus@tin: Synchronized wmf-config/db-codfw.php: Remove db1055 & db1056 (duration: 00m 51s)
10:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1082 and db1100 original weight - T161294 (duration: 00m 51s)
10:15 elukey: rolling restart of eventbus on kafka* for openssl security updates
10:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1100 and start restoring original weight for db1082 - T161294 (duration: 00m 48s)
09:50 elukey: restart eventbus on kafka2001 for openssl updates
09:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1100 - T161294 (duration: 00m 51s)
09:30 elukey: restart zookeeper on conf100[2,3] for jvm updates - T179943
09:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1100 - T161294 (duration: 00m 51s)
09:20 elukey: restart zookeeper on conf1001 for jvm updates - T179943
08:51 elukey: run kafka preferred-replica-election after maintenance of kafka1023 (fully bootstrapped now)
08:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1100 - T161294 (duration: 00m 51s)
08:34 marostegui: Stop replication in sync on db1100 and dbstore1002 - T161294
08:23 elukey: repool mw1277 after investigation
07:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Slowly repool db1100- T161294 (duration: 00m 52s)
07:21 marostegui: Upgrade MariaDB on db1100
07:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1109 and db1096:3315 - T161294 (duration: 00m 51s)
06:55 _joe_: restarting hhvm on mw1231 and mw1208
06:50 marostegui: Stop replication in sync on dbstore1002 - db1100 - T161294
06:42 marostegui: Stop replication in sync on db1100 - db2052 - T161294
06:36 marostegui: Remove some old files in dbstore1001:/srv/tmp to address the WARNING alert
06:35 marostegui: Stop replication in sync db1100 and db1071 - T161294
06:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1109 - T161294 (duration: 00m 51s)
03:59 mutante: mw1315 kill and restart hhvm | mw1312 stop and start hhvm
03:55 mutante: mw1290 kill and restart hhvm | mw1230 stop and start hhvm
02:20 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.12) (duration: 05m 31s)
01:16 eileen: update CiviCRM civicrm revision changed from 9d6ba74f57 to ffa9d7fc7a, config revision is da6fa4cce9

2017-12-20

23:26 twentyafterfour: restarting apache on phab1001 to deploy a hotfix for T144184
21:42 mepps: updated dash from 114131713e to b01458b260
21:20 otto@tin: Finished deploy [analytics/refinery@548dad7]: deploying refinery v0.0.56 with JsonRefine fixes to allow Popups schema to be refined. This is a no-op for everything else (duration: 04m 51s)
21:15 otto@tin: Started deploy [analytics/refinery@548dad7]: deploying refinery v0.0.56 with JsonRefine fixes to allow Popups schema to be refined. This is a no-op for everything else
20:31 ebernhardson: T183053 update elasticsearch settings for wikidatawiki_content on codfw to use: index.refresh_interval=5s
20:18 XioNoX: remove local-as from cr2-esams IX4
19:46 XioNoX: remove local-as from cr2-esams IX6
17:57 _joe_: depooling mw1277 for further investigation
17:51 demon@tin: Synchronized wmf-config/InitialiseSettings.php: T172875 (second try, forgot to pull first (duration: 00m 51s)
17:48 elukey: new mw jobrunner in production (mw1334) - T165519
17:47 demon@tin: Synchronized wmf-config/InitialiseSettings.php: T172875 (duration: 00m 51s)
17:46 demon@tin: Synchronized wmf-config/Wikibase-production.php: T180614 (duration: 00m 51s)
17:43 elukey: restart zookeeper on conf2003 for jvm updates - T179943
17:42 elukey: restart hhvm on mw1316
17:38 demon@tin: Synchronized wmf-config/InitialiseSettings.php: sandbox link on Atikamekw 'pedia (duration: 00m 52s)
17:38 elukey: restart hhvm on mw1234
17:32 elukey: restart zookeeper on conf2002 for jvm updates - T179943
17:20 demon@tin: Synchronized wmf-config/CommonSettings.php: more n0-0ps (duration: 00m 52s)
17:16 demon@tin: Synchronized multiversion/submodules.json: no-op (duration: 00m 51s)
17:09 demon@tin: Synchronized README: noop (duration: 00m 51s)
16:27 ema: lvs1003: stop pybal, clean ipvs services, start pybal
16:25 ema: lvs1006: stop pybal, clean ipvs services, start pybal
15:54 moritzm: rolling restart of scb* hosts in eqiad to pick up openssl update
15:47 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1334.eqiad.wmnet
15:38 moritzm: rolling restart of remaining scb* hosts in codfw to pick up openssl update
14:26 ema: start pybal on lvs2003
14:24 ema: stop pybal on lvs2003, clean IPVS table after traffic failover to get rid of trendingedits `TCP 10.2.1.9:6699 wrr`
14:14 moritzm: upgrading openssl on wdqs (along with system service restarts)
14:13 ema: bounce pybal on lvs2006 and clean IPVS table
14:06 elukey: temporarily shutdown kafka on kafka1023 to move some topic partitions on different disk partition (disk space usage alerts)
13:49 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1276.eqiad.wmnet
13:38 elukey@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw1276.eqiad.wmnet
13:38 moritzm: upgrading openssl on wdqs (along with system service restarts)
13:35 moritzm: rolling restart of aqs to pick up openssl security update
13:20 moritzm: upgrading openssl on aqs/druid clusters (along with system service restarts)
13:11 jynus: restart dbproxy1008 to test workaround is working on cold restart
13:01 moritzm: upgrading openssl on hadoop cluster (along with system service restarts)
12:34 marostegui: Enable notifications for db1100 - T161294
12:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 and db1109 - T161294 (duration: 00m 51s)
12:18 moritzm: installing iproute2 bugfix update from stretch point relesae
11:50 godog: create k8s-staging LVs in prometheus/eqiad - T163692
11:37 moritzm: upgrading openssl on elastic* (along with system service restarts)
11:27 akosiaris: repool ganeti1006, rebalance row_A ganeti nodegroup. T181121
11:18 jynus: restart dbproxy1005 to test cold service start
11:11 marostegui: Stop replication in sync on db1100 and db1071 - T161294
10:57 arturo: remove old kernel packages from silver.wikimedia.org to free space
09:56 jynus: restart dbproxy1001 to test cold service start
09:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1105:3311 - T174569 (duration: 00m 51s)
09:24 jynus: disable puppet on dbproxies for gerrit:399359 deployment
08:44 moritzm: powercycling ganeti1005
08:30 moritzm: installing rsync security updates
08:24 jynus: starting reimage of dbproxy1008
08:11 marostegui: Stop replication in sync on db1100 and dbstore1002 - T161294
07:56 jynus: starting reimage of dbproxy1007
07:35 jynus: starting reimage of dbproxy1005
07:24 jynus: upgrading and restarting dbproxy1004
07:09 marostegui: Stop replication in sync on db1096:3315 and db1100 - T161294
07:08 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1096:3315 - T161294 (duration: 00m 51s)
06:53 marostegui: Stop replication in sync on db1101:3318 and db1109 - T161294
06:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1099:3318 depool db1101:3318 - T161294 (duration: 00m 51s)
06:19 marostegui: Deploy schema change on db1105:3311 - T174569
06:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1105:3311 - T174569 (duration: 00m 52s)
06:06 Jamesofur: reset mailman password for tawikisource T183329
02:20 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.12) (duration: 05m 45s)

2017-12-19

21:51 XioNoX: removing local-as AS43821 from ams transits - T167840
19:49 mutante: deleted ganglia.wikimedia.org from DNS - webserver was already down since yesterday - not used anymore (T177225)
19:23 mutante: webperf1001/webperf2001 - rebooting for kernel upgrades (not used yet)
19:18 mutante: gerrit2001 - reboot for kernel upgrade
18:14 moritzm: installing zsh update from stretch point release
17:11 jynus: purging ferm from dbproxy1002, 3, 6, 9, 10 and 11
17:00 moritzm: installing libxv security updates on jessie
16:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1097:3315 - T161294 (duration: 00m 51s)
16:45 marostegui: Defragment s7 databases on db1102 - https://phabricator.wikimedia.org/T172169
16:38 awight@tin: Synchronized wmf-config/CirrusSearch-labs.php: wmf-config/CommonSettings-labs.php wmf-config/db-labs.php wmf-config/InitialiseSettings-labs.php wmf-config/interwiki-labs.php wmf-config/jobqueue-labs.php wmf-config/mc-labs.php wmf-config/mobile-labs.php wmf-config/Wikibase-labs.php Sync out labs config changes (duration: 00m 51s)
16:34 elukey: manually started eventlogging cleaner on db1107 to purge/sanitize data up to 90 days ago (tmux is running for user eventlogcleaner) - T108850
16:22 moritzm: installing ncurses updates from jessie point release
15:26 jgleeson: switched back on donations queue consumer and thank you mailer service
15:25 chasemp: labvirt10[19|20] aptitude install linux-image-4.4.0-81-generic linux-image-extra-4.4.0-81-generic; sudo update-grub; /sbin/reboot T172538
15:10 marostegui: Stop replication in sync on db1109 and db1099:3318 - https://phabricator.wikimedia.org/T161294
15:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1099:3318 and db1109 - T161294 (duration: 00m 51s)
15:09 jgleeson: updated civicrm from e0ee2d189c to 9d6ba74f57
15:04 jgleeson: switched off donations queue consumer and thank you mailer service in preparation for new release
15:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1106 - T161294 (duration: 00m 52s)
14:49 moritzm: restarting hhvm on canary app servers to pick up security updates for openssl, icu and libx11
14:44 moritzm: installing request-tracker4 update from jessie point release on ununpentium
14:28 jynus: disabling puppet on dbproxies for 399164 deploy
13:57 moritzm: upgrading pdns-recursor on achernar/acamar to 4.0.4+deb9u3~bpo8+1 (security fix)
13:13 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1317.eqiad.wmnet
13:05 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1318.eqiad.wmnet
13:05 elukey@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw133[0-1].eqiad.wmnet
12:12 hashar: CI: switching mwgate-composer-php70 job from Nodepool to Docker | https://gerrit.wikimedia.org/r/#/c/398921/
11:59 hashar: CI: switching composer-php55 / composer-package-php55 jobs from Nodepool to Docker | https://gerrit.wikimedia.org/r/#/c/398920/
11:29 moritzm: upgrading pdns-recursor on maerlant to 4.0.4+deb9u3~bpo8+1 (security fix)
11:28 hashar@tin: Finished deploy [docker-pkg/deploy@09087ad]: Bumping Jinja2 2.9.6..2.10 (duration: 00m 30s)
11:28 hashar@tin: Started deploy [docker-pkg/deploy@09087ad]: Bumping Jinja2 2.9.6..2.10
11:16 moritzm: uploaded pdns-recursor 4.0.4+deb9u3~bpo8+1 to apt.wikimedia.org
10:54 ariel@tin: Finished deploy [dumps/dumps@2bafffe]: allow dump runs in specified wiki list order, rather than by longest to wait (duration: 00m 02s)
10:54 ariel@tin: Started deploy [dumps/dumps@2bafffe]: allow dump runs in specified wiki list order, rather than by longest to wait
10:52 moritzm: upgrading pdns-recursor on nescio to 4.0.4+deb9u3~bpo8+1 (security fix)
10:47 elukey: restart zookeeper on conf2001 for jvm updates - T179943
10:45 jynus: disabling puppet on dbproxies for 398450 deploy
10:38 godog: rollout updated version of prometheus-nutcracker-exporter
09:12 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1097:3315 - T161294 (duration: 00m 51s)
08:56 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1106 - T161294 (duration: 00m 51s)
08:35 moritzm: reimaging mw1317 (video scaler) to stretch
08:28 marostegui: Stop replication in sync on db2045 and db1109 - T161294
08:21 moritzm: installing openssl security updates
08:05 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw2246.codfw.wmnet
08:05 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw2119.codfw.wmnet
07:00 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1106 - T161294 (duration: 00m 51s)
06:53 mobrovac@tin: Finished deploy [restbase/deploy@2b75a64]: Bug fix: Add the time_to_live config option to the Parsoid module (duration: 04m 26s)
06:51 marostegui: Stop replication in sync on db1106 and db2052 - T161294
06:49 mobrovac@tin: Started deploy [restbase/deploy@2b75a64]: Bug fix: Add the time_to_live config option to the Parsoid module
06:40 marostegui: Stop replication in sync on db1106 and dbstore1002 s5 - T161294
06:29 marostegui: Stop replication in sync on db1100 and db1106 - T161294
06:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1106 - T161294 (duration: 00m 53s)
06:09 marostegui: Deploy schema change on db1065 (s1 sanitarium master) with replication, so some lag will be generated on labs - T174569
05:18 andrewbogott: restarting slapd on seaborgium (in response to ldap complaints on the grid master)
02:24 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.12) (duration: 05m 22s)
00:44 mutante: einsteinium: sudo systemctl restrart ircecho (alias kick-icinga-wm)

2017-12-18

22:34 ejegg: updated payments-wiki from f594dfa763 to e91db27108
21:00 mutante: uranium - apt-get remove ganglia-webfrontend, apache2
20:53 mutante: ganglia.wikimedia.org shut down just now after a deprecation period - service is out of commission - T177225
20:53 chasemp: reboot labtestvirt2003
20:49 mutante: install1002/2002 - killing all ganglia processes, decoming aggregators
20:48 bawolff@tin: Synchronized php-1.31.0-wmf.12/extensions/TemplateData/TemplateDataBlob.php: T118682 (duration: 00m 52s)
19:49 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp4032.ulsfo.wmnet
18:42 moritzm: installing xml2 updates from stretch point release
18:28 moritzm: installing libxkbcommon updates from stretch point release
18:11 moritzm: installing python updates from stretch point release
17:51 elukey: run kafka preferred-replica-election on the analytics cluster to allow kafka1023 (new node) to become a partition leader
16:23 demon@tin: Pruned MediaWiki: 1.31.0-wmf.11 [keeping static files] (duration: 01m 18s)
16:17 demon@tin: Pruned MediaWiki: 1.31.0-wmf.8 (duration: 04m 58s)
16:15 elukey@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=mw133[0-7].eqiad.wmnet
16:09 thcipriani@tin: Synchronized README: noop sync to test scap 3.7.4-3 (duration: 03m 02s)
16:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 - T174569 (duration: 03m 03s)
15:47 marostegui: Stop MySQL on db1111 to copy its content to db1112 - T180788
15:45 jynus: stop and upgrade db1107 T183123
15:37 marostegui: Stop db1100 and dbstore1002 in sync - T161294
15:28 elukey@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw1329.eqiad.wmnet
15:23 moritzm: uploaded prometheus-blazegraph-exporter, prometheus-wdqs-updater-exporter and prometheus-pdns-exporter to apt.wikimedia.org
15:16 chasemp: reboot labtestvirt2003
14:41 ema: upgrade pinkunicorn to latest jessie point release (8.10) T182656
14:13 elukey: temporarily stopped mysql consumers on eventlog1001 to ease a mysql backup on db1107 - T183123
13:58 jynus: starting one-time backup of eventlogging database on db1107:/srv/backups T183123
13:29 marostegui: Stop replicaiton in sync on db1109 and db2045 - T161294
13:25 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1307.eqiad.wmnet
13:25 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1313.eqiad.wmnet
13:21 jmm@puppetmaster1001: conftool action : set/pooled=no; selector: mw1313.eqiad.wmnet
13:20 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1260.eqiad.wmnet
13:20 jmm@puppetmaster1001: conftool action : set/pooled=no; selector: mw1260.eqiad.wmnet
13:18 marostegui: Stop replication in sync on db1100 and db2052 - T161294
13:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1106 - T161294 (duration: 03m 06s)
13:00 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1106 - T161294 (duration: 03m 03s)
12:58 marostegui: Stop replication in sync on db1106 and db1100 - T161294
12:35 marostegui: Deploy schema change on db1099:331 and db1067 - T174569
12:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1099:3311 - T174569 (duration: 03m 05s)
12:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1066 - T174569 (duration: 03m 06s)
12:20 akosiaris: build scap 3.7.4-3 and upload to jessie-wikimedia, stretch-wikimedia, trusty-wikimedia. T183046, T182347
11:37 hashar: restarting Jenkins CI to upgrade the monitoring plugin
11:24 _joe_: ran cleanup script on scb* T180384
11:20 mobrovac: stopping the trending edits service - T180384
11:18 moritzm: reimaging mw1307 (video scaler) to stretch
11:08 _joe_: rolling restart of pybal on the low-traffic balancers
11:04 _joe_: disabled notifications for trendingedits.svc T180384
10:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1109 - T161294 (duration: 00m 56s)
10:30 marostegui: Stop replication on db1109 and db2045 in sync - T161294
10:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1109 - T161294 (duration: 00m 56s)
09:47 marostegui: Deploy schema change on db1066 - T174569
09:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1066 - T174569 (duration: 00m 56s)
09:43 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1073 - T174569 (duration: 00m 57s)
09:40 moritzm: installing openssl security updates
09:39 marostegui: Deploy schema change on db1055 (already depooled) - T174569
09:28 gehel: removing initial import datafiles from maps[12]001
08:58 marostegui: Stop replication in sync on db1100 and db2052 - T161294
08:57 elukey: rolling restart of the Yarn nodemanagers (hadoop) on analytics10[456]* to pick up new settings - T182276
08:30 Jamesofur: insert decryption key for 2017 Arb elections
08:08 volans: powercycling ganeti1005
06:57 _joe_: reeanbling puppet across servers with scap
06:44 marostegui: Defragment s2 databases on db1102 - T172169
06:43 _joe_: restarted hhvm on mw1283, still the same kind of lockups
06:33 marostegui: Deploy schema change on db1073 - T174569
06:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1073 - T174569 (duration: 00m 57s)
06:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1106 - T161294 (duration: 00m 56s)
06:17 marostegui: Stop replication in sync on db1106 and db1100 - T161294
06:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1106 - T161294 (duration: 00m 57s)
03:02 eileen: re-enable multiqueue_consumer process-control config revision is 1ae3778278
03:00 eileen: civicrm revision changed from 43d3f4d739 to e0ee2d189c
02:32 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.12) (duration: 05m 57s)
01:17 cwd: disabled fredge multiqueue consumer
00:49 eileen: civicrm revision changed from 798e24671b to 43d3f4d739

2017-12-16

17:27 no_justification: gerrit: Back, might see a few transient puppet failures if git pulls happened during the d/t, but should all recover
17:21 no_justification: gerrit: halting service momentarily for account reindexing
15:00 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1100 (duration: 00m 58s)
12:29 ariel@tin: Finished deploy [dumps/dumps@95dbfe6]: revert previous deploy (duration: 00m 02s)
12:29 ariel@tin: Started deploy [dumps/dumps@95dbfe6]: revert previous deploy
12:16 ariel@tin: Finished deploy [dumps/dumps@faf7de8]: use cat to recombine gzipped files (duration: 00m 02s)
12:16 ariel@tin: Started deploy [dumps/dumps@faf7de8]: use cat to recombine gzipped files
02:04 mutante: labweb1002 - manually downgrade to scap 3.7.4-1 (disabled puppet)
01:49 mutante: reimported scap 3.7.4-1 into APT (jessie-wikimedia) after fixing md5/sha sums in .dsc and .changes files to match orig.tar.gz | copied it from jessie-wikimedia to trusty and stretch-wikimedia. all distributions downgraded to 3.7.4-1 (T183046)
01:42 mutante: reimported scap 3.7.4-1 into APT (jessie-wikimedia) after fixing md5/sha sums in .dsc and .changes files to match orig.tar.gz
01:15 mutante: reprepro removing scap 3.7.4-2 package, attempting to reimport 3.7.4-1 package
01:11 mutante: reprepro remove trusty-wikimedia scap
00:44 demon@tin: Synchronized php-1.31.0-wmf.12/extensions/LoginNotify/includes/LoginNotify.php: T182867 (duration: 00m 57s)
00:40 demon@tin: Synchronized README: Testing again, this time with feeling (duration: 00m 56s)
00:28 mutante: re-disabled puppet on labweb1001/labweb1002 (as it was before)
00:26 mutante: re-enabling puppet on scap hosts
00:25 demon@tin: Synchronized README: Testing (duration: 00m 57s)
00:10 mutante: no more scap 3.7.4-2 found across 'R:Package = scap' (T183046)

2017-12-15

23:55 mutante: downgrading scap from 3.7.4-2 to 3.7.4-1 where it is installed - cumin -b 10 -s 5 'R:Package = scap' 'if dpkg -l scap | grep "3.7.4.2" && file /var/cache/apt/archives/scap_3.7.4-1_all.deb; then puppet agent --disable; apt-get remove --yes -q scap ; dpkg -i /var/cache/apt/archives/scap_3.7.4-1_all.deb ; fi' targeting 478 hosts (T183046)
23:37 mutante: aqs1004, analytics1003, downgraded scap to 3.7.4-1
22:55 mutante: tin - apt-get remove scap ; dpkg -i /var/cache/apt/archives/scap_3.7.4-1_all.deb
19:39 chasemp: reboot labtestvirt2003
18:15 jgleeson: rolled back to civicrm to 798e2467 to investigate prometheus bug
17:32 jynus: stop, upgrade and reboot labsdb1009
17:18 jynus: reloading dbproxy1010
16:44 chasemp: labtestvirt2003:~# /sbin/reboot to pickup new kernel
16:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1084 original weight (duration: 00m 57s)
16:10 elukey: re-enable piwik on bohrium after mysql backup restore
15:11 chasemp: reimage labtestvirt2003.codfw.wmnet
13:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1084 and restore original weight for db1081 (duration: 00m 56s)
13:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1084 and db1081 (duration: 00m 56s)
13:20 chasemp: disable puppet across eqiad lab* things to land a bit of code gracefully
13:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1084 (duration: 00m 56s)
12:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1084 with low weight - T180788 (duration: 00m 57s)
10:59 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1080 - T174569 (duration: 00m 56s)
10:59 jynus: reloading dbproxy1011
10:50 marostegui: Stop MySQL on db1084 to clone db1111 - T180788
10:49 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1084 to clone db1111 - T180788 (duration: 00m 56s)
10:31 elukey: rolling restart of yarn nodemanagers on an103* to apply new config - T182276
10:14 moritzm: uploaded prometheus-dns-rec-exporter 0.3 to apt.wikimedia.org
09:50 elukey: restore piwik database on bohrium after mysql corruption - piwik disabled
09:26 marostegui: Deploy schema change on db1080 - T174569
09:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1080 - T174569 (duration: 00m 56s)
09:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1083 - T174569 (duration: 00m 56s)
08:50 moritzm: powercycling ganeti1005
08:38 marostegui: Stop MySQL on db1034 to decommission it - T182556
08:26 marostegui: Remove db1034 from tendril - T182556
06:50 marostegui: Deploy schema change on db1083 - T174569
06:49 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1083 - T174569 (duration: 00m 56s)
06:43 marostegui: Deploy schema change on dbstore1001 (s1) - T174569
06:29 marostegui: Fix dbstore1001 s5 replication
06:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 and db1109 - T161294 (duration: 01m 16s)
03:01 mutante: db2016 thru db2019 - had to manually kill gmond process to decom ganglia, other db codfw hosts: didnt need it | running puppet on db205* and others in codfw to remove all ganglia (T177225)
01:23 catrope@tin: Synchronized php-1.31.0-wmf.12/extensions/ORES: Fix broken join conditions (T182936) (duration: 00m 57s)
01:22 hoo: Updated the Wikidata property suggester with data from Monday's JSON dump and applied the T132839 workarounds
01:19 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Don't discount file searches on commonswiki (duration: 00m 57s)
01:11 catrope@tin: Synchronized php-1.31.0-wmf.11: Pulling in today's cherry-picks into wmf.11 too (duration: 10m 49s)
00:44 catrope@tin: Synchronized php-1.31.0-wmf.12/resources/src/mediawiki.rcfilters/ui/mw.rcfilters.ui.MenuSelectWidget.js: T182711 (duration: 00m 56s)
00:33 catrope@tin: Synchronized php-1.31.0-wmf.12/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.ArticleTargetEvents.js: Track editor mode on save events (T182610) (duration: 00m 56s)
00:25 catrope@tin: Synchronized php-1.31.0-wmf.12/resources/lib/oojs-ui/oojs-ui-core.js: Backport OOjs UI fix for T182359, T182395 (duration: 00m 57s)
00:14 RoanKattouw: updateCollation.php finished on sewiki (T181503)
00:13 RoanKattouw: Running updateCollation.php on sewiki
00:13 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Set category collation for sewiki to uppercase-se (duration: 00m 57s)

2017-12-14

22:58 demon@tin: rebuilt and synchronized wikiversions files: group2 to wmf.12 (#2)
22:47 demon@tin: Synchronized php-1.31.0-wmf.12/extensions/ORES/includes/Hooks/ApiHooksHandler.php: fix undefined property (duration: 01m 08s)
22:17 demon@tin: rebuilt and synchronized wikiversions files: rollback, ORES breaking stuff on enwiki
22:08 demon@tin: rebuilt and synchronized wikiversions files: group2 to wmf.12
21:53 ejegg: updated CiviCRM from 1086291bb3 to 41a23f9fc3
20:33 awight: stress testing ores*
20:30 demon@tin: Pruned MediaWiki: 1.31.0-wmf.10 [keeping static files] (duration: 03m 32s)
20:28 jgleeson: turned back on donations queue consume service and thank you mailer service
20:21 awight@tin: Started restart [ores/deploy@b67bba7]: (non-production) Restart ORES services on ores*
20:19 thcipriani@tin: Finished scap: SWAT: Update en/i18n message for multiple-unclosed-formatting-tags (duration: 30m 55s)
20:02 jgleeson: updated civicrm from 798e24671b to 1086291bb3
{{safesubst:SAL entry|1=20:02 urandom: lowering cassandra compaction throughput to 5MB/s, restbase101{2,4}-{a,b,c}}}
19:58 jgleeson: Temporarily disabled donations queue consume service and thank you mailer service
19:48 thcipriani@tin: Started scap: SWAT: Update en/i18n message for multiple-unclosed-formatting-tags
19:43 thcipriani@tin: Synchronized wmf-config/throttle.php: SWAT: Define new throttle rule and cleaning expired rules T182889 (duration: 01m 08s)
19:29 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wgNamespaceRobotPolicies for wikidata T181525 (duration: 01m 04s)
19:20 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove single editor tab for plwiki T181045 (duration: 01m 09s)
19:02 awight@tin: Finished deploy [ores/deploy@b67bba7]: Redeploy ORES to scb1001 (duration: 01m 04s)
19:01 awight@tin: Started deploy [ores/deploy@b67bba7]: Redeploy ORES to scb1001
19:01 bsitzmann@tin: Finished deploy [mobileapps/deploy@bf85a55]: Update mobileapps to ff74bb1 (T182868 T182774) (duration: 12m 26s)
18:54 awight@tin: Started restart [ores/deploy@b67bba7]: Restart ORES services
18:51 arlolra: Updated Parsoid to ca20680 (T182793, T182774)
18:48 bsitzmann@tin: Started deploy [mobileapps/deploy@bf85a55]: Update mobileapps to ff74bb1 (T182868 T182774)
18:41 arlolra@tin: Finished deploy [parsoid/deploy@13b5cb5]: (no justification provided) (duration: 09m 19s)
18:32 arlolra@tin: Started deploy [parsoid/deploy@13b5cb5]: (no justification provided)
18:24 elukey: replace kafka1018 with kafka1023 (Analytics Kafka cluster)
18:11 awight@tin: Started deploy [ores/deploy@b67bba7]: Update ORES service to b67bba77acb
16:34 bd808: Scholarships: Set application start and close dates via web UI (T181072)
16:25 niharika29@tin: Finished deploy [scholarships/scholarships@872381d]: Deploy wikimania scholarships app for 2018 T181072 (duration: 00m 02s)
16:25 niharika29@tin: Started deploy [scholarships/scholarships@872381d]: Deploy wikimania scholarships app for 2018 T181072
16:07 bd808: Scholarships: updated database schema with 20171212-add-scholarship-orgs-field.sql (T181072)
15:48 marostegui: Deploy schema change on dbstore1002 (s1) - T174569
15:25 jynus: stop, upgrade and restart labsdb1011
15:07 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1089 - T174569 (duration: 01m 08s)
14:21 hashar@tin: Synchronized php-1.31.0-wmf.12/resources/src/mediawiki.rcfilters: Swat for RCFilters https://gerrit.wikimedia.org/r/#/c/398242/ https://gerrit.wikimedia.org/r/#/c/398239/ (duration: 01m 09s)
13:47 chasemp: I'm reverting https://gerrit.wikimedia.org/r/#/c/394541/ as it broke the database puppet for labservices (used for powerdns backend)
13:42 godog: upgrade grafana to 4.6.3 - T182294
13:41 elukey: update facts for puppet compiler to pick up new hosts
13:32 mobrovac@tin: Finished deploy [restbase/deploy@187d8ba]: Remove Trending Edits end point and stop storing feed results in Cassandra - T180384 T179412 (duration: 05m 37s)
13:26 mobrovac@tin: Started deploy [restbase/deploy@187d8ba]: Remove Trending Edits end point and stop storing feed results in Cassandra - T180384 T179412
13:10 marostegui: Deploy schema change on db1089 (s1) - T174569
13:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1089 - T174569 (duration: 01m 07s)
12:56 awight: stress testing on ores1*.eqiad.wmnet cluster, T182249
12:42 awight@tin: Finished deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster (duration: 00m 59s)
12:41 awight@tin: Started deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster
12:40 awight@tin: Finished deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster (duration: 00m 45s)
12:39 awight@tin: Started deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster
12:18 gehel: re-initialize cassandra on maps-test2001 - T182583
12:11 jynus: disable puppet on all databases to deploy safely https://gerrit.wikimedia.org/r/398246
12:00 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1091 - T174569 (duration: 01m 08s)
11:27 awight@tin: Finished deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster (duration: 00m 20s)
11:27 awight@tin: Started deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster
11:24 awight@tin: Finished deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster (duration: 00m 05s)
11:23 awight@tin: Started deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster
11:23 awight@tin: Finished deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster (duration: 00m 11s)
11:23 awight@tin: Started deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster
11:16 akosiaris@tin: Finished deploy [ores/deploy@b67bba7]: T181661 (duration: 00m 03s)
11:16 akosiaris@tin: Started deploy [ores/deploy@b67bba7]: T181661
11:14 akosiaris@tin: Finished deploy [ores/deploy@b67bba7]: T181661 (duration: 00m 03s)
11:14 akosiaris@tin: Started deploy [ores/deploy@b67bba7]: T181661
11:13 akosiaris@tin: Finished deploy [ores/deploy@b67bba7]: T181661 (duration: 00m 55s)
11:12 akosiaris@tin: Started deploy [ores/deploy@b67bba7]: T181661
09:27 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1067 (duration: 01m 08s)
08:39 marostegui: Reload dbproxy1011 config
07:18 marostegui: Stop replication and set read-only on labsdb1003 - T142807
07:08 marostegui: Stop replication in sync on db1109 and db1101:3318 - T161294
07:08 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1101:3318 and db1109 - T161294 (duration: 01m 07s)
07:02 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 and db1109 - T161294 (duration: 01m 08s)
06:55 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1101:3318 and db1109 - T161294 (duration: 01m 09s)
06:35 marostegui: Deploy schema change on db1091 (s4) - T174569
06:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1091 - T174569 (duration: 01m 08s)
02:26 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.11) (duration: 08m 05s)
01:46 twentyafterfour: no phabricator deployment tonight, not enough time to prepare and test the update due to a short outage earlier this evening.
00:12 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: T182534 (duration: 01m 08s)

2017-12-13

23:54 twentyafterfour: restarted phd on phab1001 (for good measure)
23:23 reedy@tin: Synchronized php-1.31.0-wmf.12/extensions/Flow/Hooks.php: unbreak CheckUser (duration: 01m 08s)
23:22 foks: deleted 22 illegal images from server
22:51 volans: restarting apache2 on phab1001, phabricator timing out
22:23 ppchelko@tin: Finished deploy [cpjobqueue/deploy@044cd23]: Fix sha1-based deduplication (duration: 00m 34s)
22:23 ppchelko@tin: Started deploy [cpjobqueue/deploy@044cd23]: Fix sha1-based deduplication
22:03 mholloway-shell@tin: Finished deploy [mobileapps/deploy@e62d8e3]: Update mobileapps to ddddebb (duration: 05m 24s)
21:58 mholloway-shell@tin: Started deploy [mobileapps/deploy@e62d8e3]: Update mobileapps to ddddebb
21:54 ejegg: updated civicrm from 85a8526eb8 to 798e24671b
21:37 mholloway-shell@tin: Finished deploy [mobileapps/deploy@b8082da]: Update mobileapps to bf67c97 (duration: 04m 19s)
21:33 mholloway-shell@tin: Started deploy [mobileapps/deploy@b8082da]: Update mobileapps to bf67c97
21:24 ppchelko@tin: Finished deploy [restbase/deploy@a993556]: Do not fallback if the revision is not specified T182770 (duration: 04m 04s)
21:20 ppchelko@tin: Started deploy [restbase/deploy@a993556]: Do not fallback if the revision is not specified T182770
20:27 demon@tin: Synchronized php-1.31.0-wmf.12/extensions/WikimediaEvents/extension.json: James_F made me do it (duration: 01m 08s)
20:14 demon@tin: rebuilt and synchronized wikiversions files: group1 to wmf.12
20:10 demon@tin: Synchronized php: symlink bump for wmf.12 (duration: 01m 07s)
19:37 ebernhardson@tin: Synchronized php-1.31.0-wmf.12/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.trackSubscriber.js: SWAT: VE trackSubscriber: Add timing data for 'loaded' state (duration: 01m 07s)
19:35 ppchelko@tin: Finished deploy [restbase/deploy@3f4bedc]: Remove references to Cassandra 2 from Parsoid storage T179417 (duration: 04m 43s)
19:34 ebernhardson@tin: Synchronized php-1.31.0-wmf.12/resources/src/mediawiki.rcfilters/mw.rcfilters.UriProcessor.js: SWAT: T182734: RCLFilters: support target page with a subpage (duration: 01m 07s)
19:30 ppchelko@tin: Started deploy [restbase/deploy@3f4bedc]: Remove references to Cassandra 2 from Parsoid storage T179417
19:25 ebernhardson@tin: Synchronized php-1.31.0-wmf.12/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.trackSubscriber.js: SWAT: VE trackSubscriber: data isn't required (duration: 01m 08s)
19:14 ebernhardson@tin: Synchronized php-1.31.0-wmf.12/resources/src/mediawiki.rcfilters/: SWAT: T182788: RCFilters: Fix live update (duration: 01m 08s)
19:07 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Turn on cirrus MLR for 4 more wikis (duration: 01m 09s)
19:05 mutante: phab[12]001,mx[12]001,mendelevium,fermium: rm /usr/local/bin/exim-to-gmetric and remove root's crontab lines to follow-up gerrit:382916
18:31 mutante: releases2001: /srv/mediawiki# rm -rf extensions/ skins/ vendor/ | clean up removed repos, let puppet clone, to match releases1001 and fix puppet run
17:33 awight@tin: Finished deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster (duration: 00m 59s)
17:32 awight@tin: Started deploy [ores/deploy@b67bba7]: (non-production) Update ORES on new cluster
17:30 moritzm: installing wireshark security updates
16:30 marostegui: Deploy schema change on s4 on dbstore1001 - T174569
16:19 godog: try again deleting obsolete cassandra metrics from graphite2002 - T181964
15:50 akosiaris@tin: Finished deploy [ores/deploy@b4f2b02]: (no justification provided) (duration: 02m 22s)
15:48 akosiaris@tin: Started deploy [ores/deploy@b4f2b02]: (no justification provided)
15:47 akosiaris@tin: Finished deploy [ores/deploy@b4f2b02]: (no justification provided) (duration: 01m 30s)
15:46 akosiaris@tin: Started deploy [ores/deploy@b4f2b02]: (no justification provided)
15:43 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1034 from config - T182556 (duration: 01m 07s)
15:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1034 from config - T182556 (duration: 01m 09s)
15:39 akosiaris@tin: Finished deploy [ores/deploy@b4f2b02]: (no justification provided) (duration: 04m 46s)
15:34 akosiaris@tin: Started deploy [ores/deploy@b4f2b02]: (no justification provided)
15:21 chasemp: remove and purge vblade-persist and runit from labstore1004 T182781
15:02 jynus: upgrade and restart db1067
15:02 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1067 (duration: 01m 07s)
14:43 zeljkof: EU SWAT finished
14:38 zfilipin@tin: Synchronized portals: SWAT: Bumping portals to master (T128546) (duration: 01m 09s)
14:37 zfilipin@tin: Synchronized portals/prod/wikipedia.org/assets: SWAT: Bumping portals to master (T128546) (duration: 01m 08s)
14:29 dcausse@tin: Synchronized php-1.31.0-wmf.12/extensions/Wikibase: T182293 Extract names of search fields as constants (duration: 02m 05s)
14:22 dcausse@tin: Synchronized wmf-config/InitialiseSettings.php: T182293 [cirrus] tune wikidata similarity configuration 2/2 (duration: 01m 07s)
14:20 dcausse@tin: Synchronized wmf-config/Wikibase.php: T182293 [cirrus] tune wikidata similarity configuration 1/2 (duration: 01m 12s)
14:03 akosiaris: gnt-node remove ganeti1006 T181121
14:01 elukey: restart Yarn nodemanagers on analytics102[8,9] to apply new settings - T182276
13:21 moritzm: uploaded prometheus-pdns-rec-exporter 0.2-1 to apt.wikimedia.org
13:10 marostegui: Deploy schema change on s4 - dbstore1002 - T174569
12:53 marostegui: Deploy alter table on s4 db1064 (sanitarium master) with replication, this will generate lag on labs replicas - T174569
12:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1103:3314 - T174569 (duration: 01m 36s)
12:19 moritzm: uploaded prometheus-ircd-exporter and prometheus-pdns-rec-exporter to apt.wikimedia.org
11:59 elukey: forced remount of /mnt/hdfs after OOM event on stat1005
11:41 akosiaris: empty ganeti1006 for reimage after ltpstress and bios upgrades T181121
11:16 mobrovac@tin: Finished deploy [graphoid/deploy@7979a40]: Update to service-template-node v0.5.4 (duration: 03m 56s)
11:12 mobrovac@tin: Started deploy [graphoid/deploy@7979a40]: Update to service-template-node v0.5.4
10:19 mobrovac@tin: Finished deploy [recommendation-api/deploy@ac66089]: Update to service-template-node v0.5.4 (duration: 02m 20s)
10:17 mobrovac@tin: Started deploy [recommendation-api/deploy@ac66089]: Update to service-template-node v0.5.4
09:41 godog: upload prometheus-elasticsearch-exporter to jessie-wikimedia - T181627
08:38 jynus: migrate away from ganeti1008
06:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1103:3314 - T174569 (duration: 01m 11s)
06:25 marostegui: Deploy schema change on db1103:3314 - T174569
06:12 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1097:3314 - T174569 (duration: 01m 08s)
02:26 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.11) (duration: 07m 02s)
00:49 ebernhardson@tin: Synchronized php-1.31.0-wmf.11/extensions/WikimediaEvents/: SWAT: T182616: turn on second mlr ab test for hewiki (duration: 01m 08s)
00:44 ebernhardson@tin: Synchronized php-1.31.0-wmf.12/extensions/WikimediaEvents/: SWAT: T182616: turn on second mlr ab test for hewiki (duration: 01m 08s)
00:40 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Turn on cirrus MLR for most wikis with >1% of search traffic (duration: 01m 08s)
00:35 mholloway-shell@tin: Finished deploy [mobileapps/deploy@5832a8c]: Update mobileapps to bfc3588 (duration: 04m 48s)
00:32 brion: restarted requeueTranscodes.php on terbium for mp3 audio generation backfill (had dropped DB connection)
00:31 mholloway-shell@tin: Started deploy [mobileapps/deploy@5832a8c]: Update mobileapps to bfc3588
00:05 ebernhardson@tin: Synchronized wmf-config/: SWAT: T182616: Setup MLR AB test for hewiki (duration: 01m 10s)

2017-12-12

22:30 mholloway-shell@tin: Finished deploy [mobileapps/deploy@ea8f05d]: Update mobileapps to 94f267b (duration: 05m 10s)
22:24 mholloway-shell@tin: Started deploy [mobileapps/deploy@ea8f05d]: Update mobileapps to 94f267b
22:02 urandom: setting compaction throughput to 5 MB/s, restbase1010
21:31 mutante: releases1001 - rm mediawiki core repo and let puppet try to recreate it (follow-up issue after gerrit:397891)
21:28 mobrovac@tin: Finished deploy [restbase/deploy@dceab2e]: Switch to Parsoid content v1.6.0 and switch to Cassandra 3 storage - T179417 (duration: 04m 16s)
21:27 demon@tin: Synchronized php-1.31.0-wmf.12/extensions/GlobalBlocking/: (no justification provided) (duration: 01m 07s)
21:26 mholloway-shell@tin: Finished deploy [mobileapps/deploy@28bfda3]: Update mobileapps to d0ee651 (duration: 01m 45s)
21:24 demon@tin: Synchronized php-1.31.0-wmf.11/extensions/GlobalBlocking/: (no justification provided) (duration: 01m 08s)
21:24 mholloway-shell@tin: Started deploy [mobileapps/deploy@28bfda3]: Update mobileapps to d0ee651
21:24 mholloway-shell@tin: Finished deploy [mobileapps/deploy@b2d5b8e]: Update mobileapps to 172abc7 (duration: 00m 30s)
21:24 mholloway-shell@tin: Started deploy [mobileapps/deploy@b2d5b8e]: Update mobileapps to 172abc7
21:23 mobrovac@tin: Started deploy [restbase/deploy@dceab2e]: Switch to Parsoid content v1.6.0 and switch to Cassandra 3 storage - T179417
21:21 ppchelko@tin: Finished deploy [restbase/deploy@dceab2e]: Bump expected parsoid version, but do not switch summaries, take 4 (duration: 02m 45s)
21:18 ppchelko@tin: Started deploy [restbase/deploy@dceab2e]: Bump expected parsoid version, but do not switch summaries, take 4
21:15 ppchelko@tin: Finished deploy [restbase/deploy@dceab2e]: Bump expected parsoid version, but do not switch summaries, take 3 (duration: 02m 04s)
21:13 ppchelko@tin: Started deploy [restbase/deploy@dceab2e]: Bump expected parsoid version, but do not switch summaries, take 3
21:13 ppchelko@tin: Finished deploy [restbase/deploy@dceab2e]: Bump expected parsoid version, but do not switch summaries, take 2 after failed content rerender (duration: 05m 05s)
21:12 mobrovac@tin: Started restart [electron-render/deploy@94d27d7]: Electron hanging - T174916
21:08 ppchelko@tin: Started deploy [restbase/deploy@dceab2e]: Bump expected parsoid version, but do not switch summaries, take 2 after failed content rerender
21:08 ppchelko@tin: Finished deploy [restbase/deploy@dceab2e]: Bump expected parsoid version, but do not switch summaries (duration: 05m 51s)
21:02 ppchelko@tin: Started deploy [restbase/deploy@dceab2e]: Bump expected parsoid version, but do not switch summaries
20:55 ppchelko@tin: Finished deploy [restbase/deploy@d3ca789]: Revert deployment for using MCS for summaries (duration: 01m 36s)
20:53 ppchelko@tin: Started deploy [restbase/deploy@d3ca789]: Revert deployment for using MCS for summaries
20:53 demon@tin: rebuilt and synchronized wikiversions files: group0 to wmf.12
20:51 ppchelko@tin: Finished deploy [restbase/deploy@506047c]: Update expected Parsoid version, switched summary to MCS (duration: 03m 30s)
20:48 ppchelko@tin: Started deploy [restbase/deploy@506047c]: Update expected Parsoid version, switched summary to MCS
20:44 mholloway-shell@tin: Finished deploy [mobileapps/deploy@b2d5b8e]: Update mobileapps to 172abc7 (duration: 04m 48s)
20:39 mholloway-shell@tin: Started deploy [mobileapps/deploy@b2d5b8e]: Update mobileapps to 172abc7
20:35 mholloway-shell@tin: Finished deploy [mobileapps/deploy@0a9d635]: Update mobileapps to 035608d (duration: 02m 32s)
20:32 mholloway-shell@tin: Started deploy [mobileapps/deploy@0a9d635]: Update mobileapps to 035608d
20:31 mholloway-shell@tin: Finished deploy [mobileapps/deploy@2690678]: Update mobileapps to 5b8796d (duration: 40m 04s)
19:57 arlolra: Updated Parsoid to 741fc5d (T114072, T181226, T21910, T152540, T103714, T97093, T118520, T181229, T182338, T182170, T169006)
19:51 mholloway-shell@tin: Started deploy [mobileapps/deploy@2690678]: Update mobileapps to 5b8796d
19:43 arlolra@tin: Finished deploy [parsoid/deploy@98139cb]: (no justification provided) (duration: 14m 57s)
19:35 anomie: Running cleanupUsersWithNoId.php on all wikis (this will take a while), see T181731
19:12 arlolra: Parsoid deploy aborted and rolled back to 01c1fc3 while RESTBase fixes an issue
19:06 arlolra@tin: Finished deploy [parsoid/deploy@98139cb]: Updating Parsoid to 741fc5d (duration: 05m 33s)
19:00 arlolra@tin: Started deploy [parsoid/deploy@98139cb]: Updating Parsoid to 741fc5d
18:48 aaron@tin: Synchronized php-1.31.0-wmf.12/includes/Setup.php: 058c17e702eb0 (duration: 01m 09s)
18:21 moritzm: uploaded prometheus-ircd-exporter to apt.wikimedia.org
17:52 demon@tin: Finished scap: bootstrap wmf.12 (duration: 29m 35s)
17:22 demon@tin: Started scap: bootstrap wmf.12
17:15 demon@tin: Pruned MediaWiki: 1.31.0-wmf.7 (duration: 08m 45s)
17:12 akosiaris@tin: Finished deploy [ores/deploy@b4f2b02]: T181661 (duration: 00m 03s)
17:12 akosiaris@tin: Started deploy [ores/deploy@b4f2b02]: T181661
17:11 akosiaris@tin: Finished deploy [ores/deploy@b4f2b02]: T181661 (duration: 00m 36s)
17:11 akosiaris@tin: Started deploy [ores/deploy@b4f2b02]: T181661
17:03 akosiaris@tin: Started deploy [ores/deploy@b4f2b02]: T181661
16:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1084 and db1081 original weight (duration: 00m 56s)
16:40 jynus: restart and upgrade db1059 (phabricator passive db)
16:35 gehel@tin: Finished deploy [kartotherian/deploy@6e223df]: new kartotherian packaging on maps-test2001 (duration: 00m 19s)
16:35 gehel@tin: Started deploy [kartotherian/deploy@6e223df]: new kartotherian packaging on maps-test2001
16:34 gehel@tin: Finished deploy [tilerator/deploy@29d633e]: new tilerator packaging on maps-test2001 (duration: 00m 20s)
16:34 gehel@tin: Started deploy [tilerator/deploy@29d633e]: new tilerator packaging on maps-test2001
16:30 gehel@tin: Finished deploy [tilerator/deploy@29d633e]: new tilerator packaging on maps-test2002 (duration: 00m 20s)
16:30 gehel@tin: Started deploy [tilerator/deploy@29d633e]: new tilerator packaging on maps-test2002
16:28 gehel@tin: Finished deploy [kartotherian/deploy@6e223df]: new kartotherian packaging on maps-test2002 (duration: 00m 18s)
16:28 gehel@tin: Started deploy [kartotherian/deploy@6e223df]: new kartotherian packaging on maps-test2002
16:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1084 (duration: 00m 53s)
16:24 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2072 (duration: 00m 55s)
16:22 akosiaris@tin: Started deploy [ores/deploy@b4f2b02]: T181661
16:21 akosiaris: failover boron to ganeti1008
16:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1084 (duration: 00m 56s)
15:59 akosiaris@tin: Finished deploy [ores/deploy@b4f2b02]: T181661 (duration: 00m 04s)
15:59 akosiaris@tin: Started deploy [ores/deploy@b4f2b02]: T181661
15:59 akosiaris@tin: Finished deploy [ores/deploy@b4f2b02]: T181661 (duration: 00m 09s)
15:58 akosiaris@tin: Started deploy [ores/deploy@b4f2b02]: T181661
15:46 jynus: stop, upgrade and reboot db2072
15:34 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2072 (duration: 00m 56s)
15:31 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1097:3314 - T174569 (duration: 01m 01s)
15:31 marostegui: Deploy schema change on s4 db1097:3314 - T174569
15:25 gehel: powercycling maps-test2003
15:24 elukey: rename notebook1002 -> kafka1023 - step 3, replace notebook1002 with kafka1023 in the puppet config
15:20 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1084 with low weight (duration: 01m 18s)
15:06 marostegui: Upgrade MySQl on db1084
15:02 elukey: clear recdns records related to notebook1002/kafka1023 (rec_control wipe-cache kafka1023.eqiad.wmnet kafka1023.mgmt.eqiad.wmnet notebook1002.eqiad.wmnet 14.5.64.10.in-addr.arpa 104.3.65.10.in-addr.arpa) - T181518
14:58 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1260.eqiad.wmnet
14:46 elukey: start rename notebook1002 -> kafka1023 - step 2, dns config (host already shutdown) - T181518
14:39 hashar: Rebuild operations-puppet-tests-docker image based on c76d892 | T178620 and /cache being owned by root
14:37 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2071 (duration: 00m 56s)
14:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore original weight for db1086 (duration: 00m 56s)
14:20 zeljkof: EU SWAT finished
14:18 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add NS aliases for zh_wikiquote (T181374) (duration: 00m 56s)
14:09 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: Lift account registration on en.wiki (T182665) (duration: 00m 56s)
14:02 herron: upgrading codfw puppet agents
13:59 jynus: stop, upgrade and reboot db2071
13:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1086 (duration: 00m 56s)
13:56 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2071 (duration: 00m 57s)
13:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1086 (duration: 00m 57s)
12:44 akosiaris@puppetmaster1001: conftool action : set/weight=10; selector: scb1004.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
12:10 akosiaris@puppetmaster1001: conftool action : set/weight=10; selector: scb1004.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
12:10 akosiaris@puppetmaster1001: conftool action : set/weight=10; selector: scb1003.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
12:06 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1092 (duration: 00m 55s)
11:56 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1086 with low weight - T163190 (duration: 00m 55s)
11:27 jynus: stop, upgrade and reboot db1092
11:26 marostegui: Upgrade MySQL and kernel on db1086
11:26 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1092 (duration: 00m 56s)
11:12 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2066 (duration: 00m 57s)
11:11 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1084 - T174569 (duration: 00m 57s)
11:11 marostegui: Deploy schema change on db1084 (s4) - T174569
11:05 marostegui: Deploy schema change on db1056 (s4) already depooled - T174569
11:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1081 - T174569 (duration: 00m 56s)
10:46 gehel@tin: Finished deploy [kartotherian/deploy@6e223df]: new kartotherian packaging on maps-test2003 (duration: 00m 10s)
10:46 gehel@tin: Started deploy [kartotherian/deploy@6e223df]: new kartotherian packaging on maps-test2003
10:26 jynus: stop, upgrade and reboot db2066
10:24 gehel@tin: Finished deploy [kartotherian/deploy@6e223df]: new kartotherian packaging on maps-test2004 (duration: 00m 18s)
10:24 gehel@tin: Started deploy [kartotherian/deploy@6e223df]: new kartotherian packaging on maps-test2004
10:23 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2066 (duration: 00m 56s)
10:14 moritzm: reimaging mw1260 (video scaler) to stretch
10:04 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2059 (duration: 00m 56s)
09:51 moritzm: updated stretch installer netboot image after stretch 9.3 point release
09:51 moritzm: updated stretch installer netboot image after jessie 8.10 point release
09:44 marostegui: Stop db1039 and db1086 in sync - T163190
09:43 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1086 - T163190 (duration: 00m 56s)
09:38 gehel: reduce replication factor for cassandra on maps-test cluster and reset cassandra on maps-test2001 to work around limited disk space - T182583
09:34 marostegui: Stop replication in sync on db1034 and db1039 for data consistency check - T163190
09:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1096:3316 after InnoDB there - T178359 (duration: 00m 56s)
09:00 jynus: stop, upgrade and reboot db2059
08:57 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2059 (duration: 00m 56s)
08:47 moritzm: updated jessie installer netboot image after jessie 8.10 point release
07:15 marostegui: Deploy schema change on db1081 - https://phabricator.wikimedia.org/T174569
07:14 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1081 - T174569 (duration: 00m 56s)
06:56 marostegui: stop MySQL on db1055 - T182653
06:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1055 - T178359 T182653 (duration: 00m 56s)
02:27 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.11) (duration: 06m 21s)
00:53 legoktm@tin: Synchronized wmf-config/CommonSettings.php: Remove manual firejailing of Score binaries (T181535) (duration: 00m 56s)
00:47 legoktm@tin: Synchronized wmf-config/CommonSettings.php: Have ExtensionDistributor treat REL1_30 as stable (duration: 00m 56s)
00:39 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Re-enable ORES on fawiki (T182354) (duration: 00m 56s)
00:25 ejegg: updated CiviCRM from 2db9c76 to 85a8526
00:23 catrope@tin: Synchronized wmf-config/CommonSettings.php: Fix default for rcOresDamagingPref (duration: 00m 56s)
00:22 ejegg: disabled nightly Ingenico WX audit parsing job

2017-12-11

23:37 smalyshev@tin: Finished deploy [wdqs/wdqs@f6b110f]: updater fix and GUI update (duration: 06m 10s)
23:31 smalyshev@tin: Started deploy [wdqs/wdqs@f6b110f]: updater fix and GUI update
23:28 mholloway-shell@tin: Finished deploy [mobileapps/deploy@07293bc]: Update mobileapps to e290b17 (duration: 06m 16s)
23:22 mholloway-shell@tin: Started deploy [mobileapps/deploy@07293bc]: Update mobileapps to e290b17
21:11 mholloway-shell@tin: Finished deploy [mobileapps/deploy@6347d62]: Update mobileapps to 61ca333 (duration: 07m 56s)
21:04 mholloway-shell@tin: Started deploy [mobileapps/deploy@6347d62]: Update mobileapps to 61ca333
19:47 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: T181493: Enable Page Previews EventLogging instrumentation (duration: 00m 56s)
19:47 mobrovac@tin: Finished deploy [restbase/deploy@be7d72f]: Expose the Reading Lists end points, take #3 - T181107 (duration: 06m 19s)
19:46 ppchelko@tin: Finished deploy [cpjobqueue/deploy@b1beaf1]: Revert dedupe based on sha1 as well as on event ID (duration: 00m 29s)
19:46 ppchelko@tin: Started deploy [cpjobqueue/deploy@b1beaf1]: Revert dedupe based on sha1 as well as on event ID
{{safesubst:SAL entry|1=19:43 urandom: lower compaction throughput to 2 MB/s, restbase1010-{a,b,c} - T178177}}
19:40 mobrovac@tin: Started deploy [restbase/deploy@be7d72f]: Expose the Reading Lists end points, take #3 - T181107
19:34 mobrovac@tin: Finished deploy [restbase/deploy@bce2885]: Expose the Reading Lists end points, take #2 - T181107 (duration: 05m 33s)
19:28 mobrovac@tin: Started deploy [restbase/deploy@bce2885]: Expose the Reading Lists end points, take #2 - T181107
19:28 mobrovac@tin: Finished deploy [restbase/deploy@bce2885]: Expose the Reading Lists end points - T181107 (duration: 01m 26s)
19:26 mobrovac@tin: Started deploy [restbase/deploy@bce2885]: Expose the Reading Lists end points - T181107
19:24 ebernhardson@tin: Synchronized wmf-config/throttle.php: SWAT: T182613 Update throttle rule for McGill University Library (duration: 00m 56s)
19:21 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: T131132: Switch submit button from save to publish on enwiki (duration: 02m 43s)
19:19 awight@tin: Finished deploy [ores/deploy@1c0ede0]: (non-production) Testing parallel ORES deployment, T181661 (duration: 01m 12s)
19:17 awight@tin: Started deploy [ores/deploy@1c0ede0]: (non-production) Testing parallel ORES deployment, T181661
19:16 ebernhardson@tin: Synchronized php-1.31.0-wmf.11/extensions/CirrusSearch/includes/Search/RescoreBuilders.php: SWAT: Add query string for running Cirrus MLR pre-deploy checks (duration: 00m 57s)
18:56 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: Switch cebwiki, ruwiki and small projects to Kafka for htmlCacheUpdate - T182023 (duration: 00m 57s)
18:55 ppchelko@tin: Finished deploy [cpjobqueue/deploy@e1075af]: Enable htmlCacheUpdate for ceb and ru wiki and small projects T182023 (duration: 00m 34s)
18:54 ppchelko@tin: Started deploy [cpjobqueue/deploy@e1075af]: Enable htmlCacheUpdate for ceb and ru wiki and small projects T182023
18:03 gehel@tin: Finished deploy [wdqs/wdqs@353b3cb]: wdqs: GUI and updater updates (duration: 01m 14s)
18:02 gehel@tin: Started deploy [wdqs/wdqs@353b3cb]: wdqs: GUI and updater updates
16:42 hashar: Restarting Jenkins
15:36 marostegui: Deploy schema change on s2 master (db1054) - T174569
14:51 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Disable the wgTranslateNumerals at hiwikiversity T182584 (duration: 00m 56s)
14:46 niharika29@tin: Synchronized php-1.31.0-wmf.11/includes/specials/SpecialUndelete.php: Revert replacing textarea in Special:Undelete with OOUI T182398 (duration: 00m 57s)
14:34 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Bureaucrats to grant and remove translationadmin rights; sysops to add and remove the same from themselves - T182492 (duration: 00m 56s)
14:32 niharika29@tin: Synchronized wmf-config/interwiki.php: Update the Interwiki map - T182506 (duration: 00m 56s)
14:25 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Create NS_PROJECT and NS_PROJECT_TALK alias for kowikisource (T182487) (duration: 00m 56s)
13:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1060 - T174569 (duration: 04m 42s)
12:39 _joe_: trying to get a full core dump from hhvm on mw1200
11:12 _joe_: depooling mw1200 for investigation instead
11:11 _joe_: restarting hhvm on mw1200, stuck in a kernel task
11:08 jdrewniak@tin: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 45s)
11:07 jdrewniak@tin: Synchronized portals/prod/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 44s)
10:49 ema: cp4021: restart varnish-be due to mbox lag
10:04 godog: upgrade grafana to 4.6.2 on labmon1001 - T182294
10:00 jynus: stopping dbstore2001:s5 and dbstore1002 (s5) mysql replication in sync
09:28 akosiaris: upload scap_3.7.4-1 to apt.wikimedia.org/jessie-wikimedia/main
09:16 gehel: cleaning old cassandra dumps on maps-test2001 servers
09:15 gehel: cleaning up old postgres logs on maps-test2001
09:05 elukey: set notebook1002 as role::spare as prep step to reimage it to kafka1023
09:03 jynus: dropping multiple leftover files from db1102
08:52 marostegui: Stop replication in sync on db1034 and db1039 - T163190
08:12 elukey: powercycle ganeti1008 - all vms stuck, console com2 showed a ton of printks without a clear indicator of the root cause
07:49 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1034 - T182556 (duration: 00m 45s)
07:44 _joe_: restarting hhvm on mw1189,mw1229,mw1235,mw1282,mw1285,mw1315,mw1316, all stuck with a kernel hang
06:59 _joe_: restarted hhvm, nginx on mw1280, hanging kernel operations
06:45 marostegui: Deploy schema change on s2 db1060 with replication enabled, this will generate some lag on s2 on labs - T174569
06:45 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1060 - T174569 (duration: 00m 44s)
06:22 marostegui: Compress s6 on db1096 - T178359
06:21 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1096:3316 to compress InnoDB there - T178359 (duration: 00m 45s)
02:43 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.11) (duration: 09m 21s)

2017-12-10

20:33 elukey: execute restart-hhvm on mw1312 - hhvm stuck multiple times queueing requests
20:01 elukey: ran kafka preferred-replica-election for the kafka analytics cluster (1012->1022) to re-add kafka1012 to the kafka brokers acting as partition leaders (will spread the load in a better way)

2017-12-09

17:00 apergos: restarted hhvm on mw1276, the same old hang with the same old symptoms
16:10 awight@tin: Finished deploy [ores/deploy@1c0ede0]: Reducing ORES Celery log verbosity (take 4\!) (duration: 03m 01s)
16:07 awight@tin: Started deploy [ores/deploy@1c0ede0]: Reducing ORES Celery log verbosity (take 4\!)
16:02 awight@tin: Finished deploy [ores/deploy@1c0ede0]: Reducing ORES Celery log verbosity (duration: 05m 58s)
15:56 awight@tin: Started deploy [ores/deploy@1c0ede0]: Reducing ORES Celery log verbosity
15:55 awight@tin: Finished deploy [ores/deploy@1c0ede0]: Reducing ORES Celery log verbosity (duration: 00m 17s)
15:55 awight@tin: Started deploy [ores/deploy@1c0ede0]: Reducing ORES Celery log verbosity
15:53 awight@tin: Finished deploy [ores/deploy@1c0ede0]: Reducing ORES Celery log verbosity (duration: 00m 31s)
15:53 awight@tin: Started deploy [ores/deploy@1c0ede0]: Reducing ORES Celery log verbosity
15:53 apergos: did same on scb1002,3,4
15:48 awight: Making an emergency deployment to ORES logging config to reduce verbosity.
15:45 apergos: on scb1001 moved daemon.log out of the way, did "service rsyslog rotate", saved the last 5000 entries for use by ores team, removed the log
11:44 apergos: that server list: mw1278, 1277, 1226, 1234, 1230
11:42 apergos: restarted hhvm on api servers after lockup
11:19 legoktm@tin: Synchronized wmf-config/InitialiseSettings.php: Disable ORES in fawiki - T182354 (duration: 00m 45s)
00:11 Jamesofur: removed 2FA from EVinente after verification T182373

2017-12-08

23:23 hashar: force ran puppet on contint2001
22:15 madhuvishy: Kicked off rsync of /data/xmldatadumps/public to labstore1006 & 7
22:05 smalyshev@tin: Finished deploy [wdqs/wdqs@353b3cb]: temporary fix for T182464, better fix coming soon (duration: 05m 55s)
21:59 smalyshev@tin: Started deploy [wdqs/wdqs@353b3cb]: temporary fix for T182464, better fix coming soon
20:22 aaron@tin: Synchronized php-1.31.0-wmf.11/includes/Setup.php: a319c3e7ab61 - disable cpPosTime injection (duration: 00m 45s)
18:00 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: Disable GlobalBlocking on fishbowl wikis (duration: 00m 45s)
16:23 urandom: starting cassandra, restbase1010 - T178177
16:22 urandom: disabling smart path, restbase1010, arrays 'b'...'e' - T178177
16:20 urandom: disabling smart path, restbase1010, array 'a' (canary) - T178177
16:15 urandom: shutting down cassandra, restbase1010 - T178177
15:35 marostegui: Fix dbstore1002 s5 replication
15:28 gehel@tin: Finished deploy [tilerator/deploy@29d633e]: testing new tilerator packaging on maps-test2003 (duration: 00m 03s)
15:28 gehel@tin: Started deploy [tilerator/deploy@29d633e]: testing new tilerator packaging on maps-test2003
15:08 gehel@tin: Finished deploy [tilerator/deploy@29d633e]: testing new tilerator packaging on maps-test2003 (duration: 02m 08s)
15:06 gehel@tin: Started deploy [tilerator/deploy@29d633e]: testing new tilerator packaging on maps-test2003
15:05 gehel@tin: Finished deploy [tilerator/deploy@29d633e]: testing new tilerator packaging on maps-test2003 (duration: 00m 42s)
15:05 gehel@tin: Started deploy [tilerator/deploy@29d633e]: testing new tilerator packaging on maps-test2003
14:39 gehel@tin: Finished deploy [tilerator/deploy@e52ea1d]: testing new tilerator packaging on maps-test2003 (duration: 02m 34s)
14:36 gehel@tin: Started deploy [tilerator/deploy@e52ea1d]: testing new tilerator packaging on maps-test2003
11:45 elukey: updated prometheus-druid-exporter on druid* to 0.6
11:39 elukey: upload prometheus-druid-exporter 0.6 to stretch/jessie wikimedia
06:52 marostegui: Fix labsdb1004 replication broken
06:43 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1099:3311 - T178359 (duration: 00m 55s)
04:56 bd808: scholarships: updated db schema for 2018 cycle (T181072)
00:04 bblack: cp4026 - restart varnish backend, mailbox lag

2017-12-07

23:59 tgr@tin: Synchronized wmf-config/InitialiseSettings.php: T181107 enable ReadingLists on all wikis (duration: 00m 46s)
23:54 tgr: ran mwscript extensions/ReadingLists/maintenance/populateProjectsFromSiteMatrix.php --wiki=testwiki
23:47 tgr@tin: Finished scap: T181107 deploy ReadingLists to testwiki (duration: 24m 44s)
23:22 tgr@tin: Started scap: T181107 deploy ReadingLists to testwiki
23:04 tgr: ran mwscript ../../../home/tgr/sql.php --wiki=mediawikiwiki --cluster extension1 --wikidb wikishared /srv/mediawiki-staging/php-1.31.0-wmf.11/extensions/ReadingLists/sql/readinglists.sql
22:33 tgr@tin: Synchronized php-1.31.0-wmf.11/extensions/ReadingLists/: ReadingLists/wmf.11: catching up with master (duration: 00m 45s)
22:29 tgr@tin: Synchronized php-1.31.0-wmf.10/extensions/ReadingLists/: ReadingLists/wmf.10: catching up with master (duration: 00m 46s)
20:39 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.11
20:35 elukey: restart hhvm on mw1235 - hhvm-dump-debug hanging out, not stacktrace available
20:31 elukey: restart hhvm on mw1281 - hhvm stuck (hhvm-dump-debug timing out)
20:22 joal@tin: Finished deploy [analytics/refinery@53bd630]: Regular analytics deploy - Long time no see, deployment :) - post-patch-2 (hopefully last for tonight) (duration: 05m 21s)
20:18 herron: re-pooling eqiad puppet 4 masters via dns puppet.eqiad.wmnet puppet.wikimedia.org
20:17 joal@tin: Started deploy [analytics/refinery@53bd630]: Regular analytics deploy - Long time no see, deployment :) - post-patch-2 (hopefully last for tonight)
20:14 joal@tin: Finished deploy [analytics/refinery@3e52903]: Regular analytics deploy - Long time no see, deployment :) - post-patch (duration: 01m 20s)
20:12 joal@tin: Started deploy [analytics/refinery@3e52903]: Regular analytics deploy - Long time no see, deployment :) - post-patch
19:52 joal@tin: Finished deploy [analytics/refinery@bd9c6cc]: Regualr analytics deploy - Long time no see, deployment :) (duration: 12m 15s)
19:40 joal@tin: Started deploy [analytics/refinery@bd9c6cc]: Regualr analytics deploy - Long time no see, deployment :)
19:35 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Start description usage tracking for commonswiki T106287 (duration: 00m 48s)
19:29 thcipriani@tin: Synchronized php-1.31.0-wmf.11/includes/specialpage/ChangesListSpecialPage.php: SWAT: WLFilters: Correctly check if RCFilters should be enabled on WL T182318 (duration: 00m 48s)
19:24 thcipriani@tin: Synchronized wmf-config/Wikibase-production.php: SWAT: Remove obsolete WikibaseQualityConstraints settings (duration: 00m 48s)
19:14 ppchelko@tin: Finished deploy [changeprop/deploy@3c4f51d]: Long awaited deploy: generic optimizations, gc metric, delay reporting (duration: 01m 15s)
19:13 ppchelko@tin: Started deploy [changeprop/deploy@3c4f51d]: Long awaited deploy: generic optimizations, gc metric, delay reporting
19:12 thcipriani@tin: Synchronized wmf-config/throttle.php: SWAT: Rm all past throttle overrides in throttle.php (duration: 00m 48s)
18:06 mholloway-shell@tin: Finished deploy [mobileapps/deploy@2fa32ed]: Update mobileapps to 71f581c (duration: 05m 36s)
18:00 mholloway-shell@tin: Started deploy [mobileapps/deploy@2fa32ed]: Update mobileapps to 71f581c
18:00 herron: re-pooling eqiad puppet 4 masters as puppet.ulsfo.wnet
17:33 demon@tin: Synchronized php-1.31.0-wmf.11/extensions/VisualEditor/lib/ve: Ief480487, Deskana made me do it (duration: 00m 49s)
17:25 milimetric@tin: Finished deploy [analytics/aqs/deploy@4ec13b4]: (no justification provided) (duration: 07m 28s)
17:25 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1314.eqiad.wmnet
17:25 herron: puppetmaster1001 upgraded to puppet 4. re-enabling puppet agents across the fleet
17:18 milimetric@tin: Started deploy [analytics/aqs/deploy@4ec13b4]: (no justification provided)
17:13 herron: upgrading puppetmaster1001 to puppet 4
17:08 herron: temporarily disabling all puppet agents during puppetmaster1001 (puppet ca) upgrade to puppet 4
16:14 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1099:3311 weight - T178359 (duration: 00m 48s)
16:13 herron: upgrading puppetmaster1002 to puppet 4
16:03 moritzm: uploaded openssl 1.0.2n for jessie-wikimedia to apt.wikimedia.org
16:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1093 back as main traffic in s6 - T178359 (duration: 00m 48s)
15:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1099:3311 weight - T178359 (duration: 00m 48s)
15:42 elukey: hhvm-dump-debug for mw1314 saved to /tmp/hhvm.17991.bt.
15:30 elukey@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw1314.eqiad.wmnet
15:24 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1099:3311 weight - T178359 (duration: 00m 48s)
15:19 otto@tin: Finished deploy [analytics/superset/deploy@f0f5adf]: initial deployment (duration: 00m 02s)
15:18 otto@tin: Started deploy [analytics/superset/deploy@f0f5adf]: initial deployment
14:55 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1099:3311 with low weight - T178359 (duration: 00m 47s)
14:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully repool db1074 (duration: 00m 47s)
14:09 zeljkof: EU SWAT finished
14:08 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: alswiki: Set wgRestrictDisplayTitle = false (T182154) (duration: 00m 49s)
13:39 moritzm: reimaging mw2246 to stretch
12:54 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw2152.codfw.wmnet
12:32 addshore@tin: Synchronized wmf-config/Wikibase.php: Move Wikibase dispatchingLockManager to InitialiseSettings PT 2/2 (duration: 00m 48s)
12:31 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Move Wikibase dispatchingLockManager to InitialiseSettings PT 1/2 (duration: 00m 48s)
11:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase API traffic for db1074 (duration: 00m 48s)
11:41 moritzm: reimaging mw2152 to stretch
11:35 marostegui: Compress s8 on db1099 - T178359
11:21 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase API traffic for db1074 (duration: 00m 48s)
11:11 mobrovac@tin: Finished deploy [restbase/deploy@097ba7d]: Add CORS headers to erroneous responses as well - T182103 (duration: 05m 24s)
11:05 mobrovac@tin: Started deploy [restbase/deploy@097ba7d]: Add CORS headers to erroneous responses as well - T182103
10:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1074 with low weight (duration: 00m 48s)
10:50 elukey: powercycle analytics1003 - no serial console, ssh stuck in System is booting up. See pam_nologin(8)
10:28 _joe_: depooling mw1283 for further investigation
10:14 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1098:3316 db1098:3317 - T178359 (duration: 00m 51s)
10:12 elukey: reboot analytics1003 for kernel+jvm updates - T179943
10:01 gehel: upgrade of ELK stack on logstash100* completed - Kibana was unavailable for longer than expected - T178412
09:48 hashar: CI: removed Wikidata from configuration, replaced by Wikibase. wmf/* and REL branches are going to be broken though | https://gerrit.wikimedia.org/r/395704 | T181838
09:46 akosiaris: silence ganeti1006 on icinga T181121
09:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1098:3316 db1098:3317 - T178359 (duration: 00m 52s)
09:28 marostegui: Upgrade MySQL and kernel on db1074
09:22 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=logstash1009.eqiad.wmnet
09:19 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=logstash1009.eqiad.wmnet
09:18 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=logstash1008.eqiad.wmnet
09:14 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=logstash1008.eqiad.wmnet
09:13 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=logstash1007.eqiad.wmnet
09:08 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=logstash1007.eqiad.wmnet
09:05 gehel@tin: Finished deploy [logstash/plugins@b13d2fa]: (no justification provided) (duration: 00m 02s)
09:05 gehel@tin: Started deploy [logstash/plugins@b13d2fa]: (no justification provided)
09:00 gehel: upgrading ELK stack on logstash100* - some log messages might be lost during the upgrade - T178412
08:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1098:3316 db1098:3317 - T178359 (duration: 00m 48s)
08:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1098:3316 db1098:3317 - T178359 (duration: 00m 48s)
08:28 elukey: install prometheus-druid-exporter 0.5 on druid*
08:26 elukey: upload prometheus-druid-exporter 0.5-1 to jessie/stretch-wikimedia
07:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Slowly pool db1098:3316 db1098:3317 - T178359 (duration: 00m 47s)
07:18 marostegui@tin: Synchronized wmf-config/db-codfw.php: Slowly pool db1098:3316 db1098:3317 - T178359 (duration: 00m 48s)
06:45 marostegui: Stop replication on db1099:3311 to reimport: change_tag, tag_summary, user and watchlist tables and recompress again - T178359
06:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1074 - T174569 (duration: 00m 48s)
06:40 marostegui: Deploy schema change on db1074 (s2) - T174569
03:00 catrope@tin: Synchronized php-1.31.0-wmf.11/resources/src/mediawiki.rcfilters/: T182268 (duration: 00m 57s)
02:29 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.10) (duration: 08m 57s)

2017-12-06

23:45 eileen: update CiviCRM from 85263f6 to 2db9c76 (improve org merges)
23:12 ppchelko@tin: Started restart [changeprop/deploy@065a06e]: (no justification provided)
23:04 eileen: update process-control to b1cd515 (reenable bounce processing, enable dedupe on orgs)
22:58 ppchelko@tin: Finished deploy [cpjobqueue/deploy@761524f]: Separate delay and totaldelay metrics T182216 (duration: 00m 31s)
22:57 ppchelko@tin: Started deploy [cpjobqueue/deploy@761524f]: Separate delay and totaldelay metrics T182216
22:55 mutante: stat1003 - re-enabled puppet after putting role::spare on it (T175150)
22:48 mutante: stat1003 - this host was kind of invisible (not in site, not in icinga) but still up, re-enabling puppet after re-adding it to site
22:03 demon@tin: Synchronized php-1.31.0-wmf.11/extensions/WikidataPageBanner/includes/WikidataPageBanner.hooks.php: unbreak (duration: 00m 48s)
22:00 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.11, again
21:51 hoo@tin: Synchronized php-1.31.0-wmf.10/extensions/Wikibase/lib/includes/Changes/EntityChange.php: Make EntityChange truly forward compatible with compact diffs (T182243) (duration: 00m 48s)
21:43 arlolra: Updated Parsoid to 01c1fc3 (T178253, T61840, T180930, T179259)
21:34 arlolra@tin: Finished deploy [parsoid/deploy@dfcc622]: Updating Parsoid to 01c1fc3 (duration: 09m 45s)
21:24 arlolra@tin: Started deploy [parsoid/deploy@dfcc622]: Updating Parsoid to 01c1fc3
20:45 marostegui: Add 400G to labsdb1003 /srv partition
20:15 hoo: Ran "scap pull" on snapshot1001 after T177486 related tests
20:10 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: partial rollback -- wikidata errors
20:07 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.11
20:05 demon@tin: Synchronized php: symlink bump (duration: 00m 47s)
19:32 hoo@tin: Synchronized php-1.31.0-wmf.11/extensions/Wikibase/repo/maintenance/dispatchChanges.php: dispatch: track how long client selecting takes (duration: 00m 48s)
19:24 ppchelko@tin: Finished deploy [cpjobqueue/deploy@8c66189]: Deduplicate based on event sha1 as well as on id, take 2 (duration: 00m 38s)
19:24 ppchelko@tin: Started deploy [cpjobqueue/deploy@8c66189]: Deduplicate based on event sha1 as well as on id, take 2
19:23 hoo@tin: Synchronized php-1.31.0-wmf.10/extensions/Wikibase/repo/maintenance/dispatchChanges.php: dispatch: track how long client selecting takes (duration: 00m 48s)
19:22 ppchelko@tin: Started deploy [cpjobqueue/deploy@8c66189]: Deduplicate based on event sha1 as well as on id, take 2
19:22 ppchelko@tin: Finished deploy [cpjobqueue/deploy@8c66189]: Deduplicate based on event sha1 as well as on id (duration: 00m 11s)
19:22 ppchelko@tin: Started deploy [cpjobqueue/deploy@8c66189]: Deduplicate based on event sha1 as well as on id
19:20 ppchelko@tin: Finished deploy [cpjobqueue/deploy@df72b34]: Deduplicate based on event sha1 as well as on id (duration: 00m 09s)
19:19 ppchelko@tin: Started deploy [cpjobqueue/deploy@df72b34]: Deduplicate based on event sha1 as well as on id
18:44 volans: stopped irc echo on tegmen, re-enabled puppet and run it (it was disabled by a run-no-puppet sync_icinga_state)
18:32 mobrovac@tin: Finished deploy [zotero/translators@3044b3a]: Update translators to 092c7bc - T178596 (duration: 00m 07s)
18:32 mobrovac@tin: Started deploy [zotero/translators@3044b3a]: Update translators to 092c7bc - T178596
18:16 herron: upgrading rhodium to puppet 4
18:13 mobrovac@tin: Synchronized wmf-config/jobqueue.php: Switch htmlCacheUpdate jobs for wiktionaries to EventBus, file 2/2 - T182023 (duration: 00m 48s)
18:12 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: Switch htmlCacheUpdate jobs for wiktionaries to EventBus, file 1/2 - T182023 (duration: 00m 48s)
18:05 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add https://studiezaal.nijmegen.nl to $wgCopyUploadsDomains (T181713) (duration: 00m 49s)
17:51 ppchelko@tin: Finished deploy [cpjobqueue/deploy@df72b34]: Switch htmlCacheUpdate for wiktionaries, attempt 2 T182023 (duration: 00m 32s)
17:50 ppchelko@tin: Started deploy [cpjobqueue/deploy@df72b34]: Switch htmlCacheUpdate for wiktionaries, attempt 2 T182023
17:44 ppchelko@tin: Finished deploy [cpjobqueue/deploy@3281df1]: Switch htmlCacheUpdate for wiktionaries T182023 (duration: 02m 57s)
17:42 ppchelko@tin: Started deploy [cpjobqueue/deploy@3281df1]: Switch htmlCacheUpdate for wiktionaries T182023
17:36 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1260.eqiad.wmnet
17:05 hoo: Started Wikidata RDF dumps (sudo -b -u datasets bash -c 'dumpwikidatardf.sh all ttl; dumpwikidatardf.sh truthy nt') on snapshot1007
16:33 anomie@terbium: Finished cleanupUsersWithNoId.php on mediawikiwiki
16:29 anomie@terbium: Running cleanupUsersWithNoId.php for mediawikiwiki, see T181731
16:28 anomie@terbium: Finished cleanupUsersWithNoId.php on testwikidatawiki
16:27 anomie@terbium: Running cleanupUsersWithNoId.php for testwikidatawiki, see T181731
16:26 anomie@terbium: Finished cleanupUsersWithNoId.php on test2wiki
16:14 anomie@terbium: Running cleanupUsersWithNoId.php for test2wiki, see T181731
16:11 anomie@terbium: Finished cleanupUsersWithNoId.php on testwiki
16:03 anomie@terbium: Running cleanupUsersWithNoId.php for testwiki, see T181731
15:13 akosiaris: upload kubernetes_1.7.10-1_amd64 on apt.wikimedia.org/stretch-wikimedia/main T181489
14:59 addshore: swat done
14:57 addshore@tin: Synchronized php-1.31.0-wmf.11/extensions/Wikibase/repo/includes/ChangeDispatcher.php: SWAT Tracking within ChangeDispatcher::getPendingChanges (duration: 00m 49s)
14:53 addshore@tin: Synchronized php-1.31.0-wmf.10/extensions/Wikibase/repo/includes/ChangeDispatcher.php: SWAT Tracking within ChangeDispatcher::getPendingChanges (duration: 00m 49s)
14:33 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add https://studiezaal.nijmegen.nl to $wgCopyUploadsDomains (T181713) (duration: 00m 47s)
14:22 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add Portal namespace for mwl.wikipedia (T180052) (duration: 00m 48s)
14:13 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT T180476 wmgUseNewWikiDiff2Extension true for dewiki (duration: 00m 48s)
14:08 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT T181498 T181329 Enable AdvancedSearch on fawiki and huwiki (duration: 00m 50s)
13:40 moritzm: reimaging mw1260 to stretch
10:55 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw2118.codfw.wmnet
10:55 moritzm: reimaging mw2119 to stretch
10:45 mobrovac@tin: Finished deploy [restbase/deploy@b1d7c82]: Use Cass3 for revisions, deprecate trending-edits, fix CX end point - T179421 T180384 T173801 (duration: 06m 02s)
10:39 mobrovac@tin: Started deploy [restbase/deploy@b1d7c82]: Use Cass3 for revisions, deprecate trending-edits, fix CX end point - T179421 T180384 T173801
09:34 gehel: shuttting down logstash / elasticsearch on logstash100[123] in preparation for decommission -T175830
08:05 moritzm: reimaging mw2118 to stretch (now for real, yesterday's reimage logged to SAL was interrupted)
07:33 bblack: cp4024 - backend restart, mailbox lag
02:39 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.10) (duration: 08m 53s)
01:01 eileen: update process-control to 55529ea - threshold for $500+ dedupe was too low
00:36 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable description usage tracking for all wikis except commons T106287 (duration: 00m 48s)
00:27 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable RemexHTML on wikis with zero high priority linter errors T182042 (duration: 00m 51s)

2017-12-05

23:07 eileen: update civicrm from d2c70d2 to 85263f6 (Major gifts address - choose latest)
22:29 eileen: update process_control to 3522186 renable jobs to fetch recipient data
22:13 mutante: restbase1010 failed at reboot with P6431 , after a cold start (power off, power on) it came back though :) (T178177 T141756)
22:00 mutante: restbase1010 - rebooting for firmware upgrade
21:58 mutante: restbase1010 - upgraded HP firmware (Flashing Smart Array P440ar in Slot 0 [ 3.56 -> 6.06 ]) T141756 T178177
21:56 urandom: draining cassandra instances, restbase1010 - T178177
21:52 mutante: restbase1010 - upgrading firmware - Flashing Smart Array P440ar in Slot 0 [ 3.56 -> 6.06 ]
21:31 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 shenanigans / wmf.11
21:12 demon@tin: Finished scap: bootstrap wmf.11 (duration: 51m 01s)
20:50 twentyafterfour: phabricator: running `sudo bin/garbage collect --collector search.ferret.ngram`
20:29 twentyafterfour: phabricator: running `sudo bin/search ngrams --threshold 0.2`
20:21 demon@tin: Started scap: bootstrap wmf.11
19:50 mutante: forcing puppet ron on ores eqiad to reduce number of celery workers used for stress test
19:37 ppchelko@tin: Finished deploy [eventlogging/eventbus@6ca0372]: Make the kafka async deliver callback thread-safe. T180017 (duration: 01m 29s)
19:36 ppchelko@tin: Started deploy [eventlogging/eventbus@6ca0372]: Make the kafka async deliver callback thread-safe. T180017
19:20 gehel: moving eventlogging collection by logstash from logstash1003 to logstash1007, no messages **should** be lost - T175830
18:24 ppchelko@tin: Finished deploy [eventlogging/eventbus@6ca0372]: Make the kafka async deliver callback thread-safe. Limited to kafka1001. T180017 (duration: 00m 14s)
18:24 ppchelko@tin: Started deploy [eventlogging/eventbus@6ca0372]: Make the kafka async deliver callback thread-safe. Limited to kafka1001. T180017
17:27 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw2118.codfw.wmnet
15:56 awight: beginning stress test on ores* (non-production)
15:54 awight@tin: Finished deploy [ores/deploy@6baed71]: (non-production) Test ORES deployment to ores100[1-2] (duration: 01m 01s)
15:53 awight@tin: Started deploy [ores/deploy@6baed71]: (non-production) Test ORES deployment to ores100[1-2]
15:15 ottomata: restarrting kafka-jumbo brokers, applying SSL (downtime scheduled)
15:03 urandom: bootstrapping cassandra, restbase1014-c - T179422
14:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully repool db1076 (duration: 00m 43s)
14:41 zeljkof: EU SWAT finished
14:40 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Add category collation for sewiki" (T181503) (duration: 00m 44s)
14:15 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable RemexHTML on itwiki and dewiki (T181188 T181190) (duration: 00m 43s)
13:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1076 weight (duration: 00m 44s)
13:14 moritzm: reimaging mw2118/mw2119 (video scalers) to stretch
12:21 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully repool db1034 (duration: 00m 43s)
12:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1076 with low weight (duration: 00m 44s)
11:58 marostegui: Upgrade MariaDB and kernel on db1076
11:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1034 - T178359! (duration: 00m 43s)
10:53 moritzm: rebooting meitnerium/archiva.wikimedia.org for update to 4.9.51
10:46 moritzm: rebooting einsteinium (icinga.wikimedia.org) for update to 4.9,51
10:45 elukey: reboot druid1003 for kernel+jvm updates - T179943
10:33 addshore@tin: Synchronized wmf-config/extension-list: extension-list extension.json entrypoint for ArticlePlaceholder (duration: 00m 43s)
10:33 moritzm: rebooting tegmen for update to 4.9,51
10:07 jynus: stopping dbstore1002 (s5) and dbstore2001 (s5) for maintenance
10:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1034 - T178359! (duration: 00m 43s)
10:00 ema: cp4021: restart varnish-be due to mbox lag/fetch failures
09:49 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1034 with low weight - T178359! (duration: 00m 43s)
09:42 elukey: reboot analytics100[12] for kernel+jvm updates (Hadoop Master nodes) - T179943
09:39 godog: bootstrap restbase1014-b - T179422
09:20 marostegui: Optimize s7 on db1098 - T178359
09:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1076 - T174569 (duration: 00m 43s)
09:01 marostegui: Deploy schema change on db1076 (s2) - T174569
08:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1090 - T174569 (duration: 00m 44s)
08:54 moritzm: enabling test production traffic for mw1259 (stretch-based video scaler)
08:24 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1259.eqiad.wmnet
08:08 moritzm: installing python updates on trusty
07:27 kartik@tin: Finished deploy [cxserver/deploy@4b74f03]: Update cxserver to 1693bcf (duration: 03m 22s)
07:23 kartik@tin: Started deploy [cxserver/deploy@4b74f03]: Update cxserver to 1693bcf
06:58 marostegui: Stop MySQL on db1034 to clone db1098:3317 - T178359
06:55 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1034 - T178359! (duration: 00m 43s)
06:48 marostegui: Fix dbstore1002 replication
06:20 marostegui: Deploy schema change on db1090 (s2) - T174569
06:20 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1090 - T174569 (duration: 00m 43s)
06:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1103:3312 - T174569 (duration: 00m 44s)
02:29 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.10) (duration: 06m 36s)
00:29 mutante: bast4001 - removing ganglia aggregators, package, config...

2017-12-04

23:44 mutante: bast3002 - killall -u ganglia to kill all aggregator procs, apt-get remove --purge ganglia-monitor, rm -rf /etc/ganglia, rm -rf /usr/lib/ganglia, apt-get autoremove
23:28 anomie@tin: Synchronized wmf-config/InitialiseSettings.php: Revert wgCommentTableSchemaMigrationStage change, breaks too much stuff (duration: 00m 44s)
22:08 brion: brion running requeueTranscodes.php on terbium to batch-run .mp3 output on Commons (T181749)
21:27 ppchelko@tin: Finished deploy [cpjobqueue/deploy@c8bea2e]: (no justification provided) (duration: 00m 42s)
21:26 ppchelko@tin: Started deploy [cpjobqueue/deploy@c8bea2e]: (no justification provided)
21:26 awight@tin: Finished deploy [ores/deploy@6baed71]: Update ORES to 6baed71 (duration: 12m 54s)
21:25 eileen: re-enable dedupe job process control commit 461f482
21:13 awight@tin: Started deploy [ores/deploy@6baed71]: Update ORES to 6baed71
21:07 gehel@tin: Finished deploy [kartotherian/deploy@6e223df]: testing new kartotherian packaging on maps-test2003 (duration: 00m 20s)
21:07 gehel@tin: Started deploy [kartotherian/deploy@6e223df]: testing new kartotherian packaging on maps-test2003
20:07 anomie@tin: Synchronized wmf-config/InitialiseSettings.php: Revert wgCommentTableSchemaMigrationStage change, breaks too much stuff (duration: 00m 43s)
19:31 legoktm@tin: Synchronized wmf-config/InitialiseSettings.php: Set wgCommentTableSchemaMigrationStage = MIGRATION_WRITE_BOTH on test wikis (T166733) (duration: 00m 43s)
19:18 gehel@tin: Finished deploy [kartotherian/deploy@e166d87]: testing new kartotherian packaging on maps-test2003 (duration: 00m 03s)
19:18 gehel@tin: Started deploy [kartotherian/deploy@e166d87]: testing new kartotherian packaging on maps-test2003
19:17 legoktm@tin: Synchronized wmf-config/InitialiseSettings.php: Set $wgRestrictionMethod = 'firejail'; everywhere (T173370) (duration: 00m 43s)
19:11 legoktm@tin: Synchronized static/images/mobile/copyright/: Add converted copyright svg images as png files - https://gerrit.wikimedia.org/r/#/c/394820/ (duration: 00m 43s)
19:06 legoktm@tin: Synchronized wmf-config/InitialiseSettings-labs.php: beta: Add ORES filter thresholds for simplewiki (duration: 00m 43s)
18:46 hoo: Started dumpwikidatajson.sh on snapshot1007 (T181385)
18:40 hoo: Ran scap pull on snapshot1001 (T181385)
18:35 hoo@tin: Synchronized php-1.31.0-wmf.10/includes/objectcache/ObjectCache.php: Only send statsd data for WAN cache in non-CLI mode (T181385) (duration: 00m 44s)
18:28 godog: bootstrap restbase1014-a - T179422
18:04 gehel@tin: Finished deploy [wdqs/wdqs@2873745]: wdqs GUI update (duration: 02m 10s)
18:02 gehel@tin: Started deploy [wdqs/wdqs@2873745]: wdqs GUI update
17:41 demon@tin: Synchronized php-1.31.0-wmf.10/extensions/CirrusSearch/maintenance/: fix some deprecated spam (duration: 00m 44s)
17:40 ejegg: disabled CiviCRM bounce processing again
17:12 demon@tin: Pruned MediaWiki: 1.31.0-wmf.6 (duration: 02m 29s)
17:08 demon@tin: Pruned MediaWiki: 1.31.0-wmf.5 (duration: 02m 58s)
16:54 demon@tin: Synchronized docroot/noc/conf/dblists: double checking symlink move (duration: 00m 44s)
16:44 ejegg: re-enabled CiviCRM bounced mail processing
16:41 marostegui: Deploy schema change on db1053 (s2) - T174569
16:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1103:3312 - T174569 (duration: 00m 45s)
16:39 demon@tin: Finished scap: docroot/noc/conf/ Adding new dblist symlink (duration: 02m 36s)
16:39 marostegui: Deploy schema change on db1103:3312 - T174569
16:37 demon@tin: Started scap: docroot/noc/conf/ Adding new dblist symlink
16:37 demon@tin: scap aborted: docroot/noc/conf/dblists (duration: 00m 02s)
16:37 demon@tin: Started scap: docroot/noc/conf/dblists
16:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1105:3312 - T174569 (duration: 00m 45s)
16:32 demon@tin: Finished scap: docroot/noc/conf/ drop some dangling symlinks (duration: 03m 53s)
16:28 demon@tin: Started scap: docroot/noc/conf/ drop some dangling symlinks
16:11 herron: re-enabling puppet agents in eqiad
16:07 demon@tin: Finished scap: Revert "Special:Preferences: Use OOjs UI" and follow-ups (duration: 21m 19s)
16:05 herron: re-enabling puppet agents in codfw
15:46 demon@tin: Started scap: Revert "Special:Preferences: Use OOjs UI" and follow-ups
15:31 herron: temporarily disabling puppet agents in eqiad and codfw while puppetdb catches up with command queue
15:29 herron: disabling ircecho temporarily
15:24 _joe_: restarting puppetdb on nihal, will cause puppet failures
15:23 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1259.eqiad.wmnet
15:18 demon@tin: Synchronized wmf-config/CommonSettings-labs.php: no-op (duration: 00m 45s)
15:01 godog: reimage restbase1014 - T179422
14:51 herron: cutting over all production puppet agents to codfw puppet 4 masters via dns
14:50 Amir1: deployed backward compatibility of entity compact diff transmit T113468
14:49 ladsgroup@tin: Synchronized php-1.31.0-wmf.10/extensions/Wikibase: (no justification provided) (duration: 01m 41s)
14:37 gehel@tin: Finished deploy [kartotherian/deploy@e166d87]: dummy kartotherian deployment to test udp2log config change - T175242 (duration: 00m 03s)
14:37 gehel@tin: Started deploy [kartotherian/deploy@e166d87]: dummy kartotherian deployment to test udp2log config change - T175242
14:30 elukey: reboot druid100[23] for kernel updates
14:28 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Localize sitename and meta NS for wawiktionary (T181782) (duration: 00m 46s)
14:01 elukey: reboot analytics106* (hadoop worker nodes) for kernel+jvm updates - T179943
13:24 paravoid: upgrading bast2001 to stretch
13:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1105:3312 - T174569 (duration: 00m 45s)
13:20 marostegui: Deploy schema change on db1105:3312 (s2) - T174569
13:05 marostegui: Compress s6 on db1098 - T178359
12:44 gehel@tin: Finished deploy [kartotherian/deploy@e166d87]: testing new kartotherian packaging on maps-test2003 (duration: 00m 22s)
12:43 gehel@tin: Started deploy [kartotherian/deploy@e166d87]: testing new kartotherian packaging on maps-test2003
11:06 jdrewniak@tin: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 45s)
11:05 jdrewniak@tin: Synchronized portals/prod/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 45s)
10:37 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2085 (duration: 00m 43s)
10:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1096:3316 - T178359wq! (duration: 00m 45s)
09:55 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1099:3318 from s5 (duration: 00m 44s)
09:51 godog: bootstrap restbase1012-c - T179422
09:32 godog: clear erroneous table metrics from graphite1003 / graphite2002 - T181689
09:24 elukey: reboot analytics105* (hadoop worker nodes) for kernel+jvm updates - T179943
09:19 jynus: rebooting mariadb at labsdb1005
09:12 moritzm: reimaging mw1259 (video scaler) to stretch, will be kept disabled initially (some controlled live tests following)
08:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1096:3315 and 3316 - T178359 (duration: 00m 45s)
08:45 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1096:3316 - T178359 (duration: 00m 45s)
08:44 moritzm: updating tor on radium to 0.3.1.9
08:41 moritzm: updating tor packages to 0.3.1.9
08:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1096:3315 and pool db1096:3316 - T178359 (duration: 00m 45s)
08:12 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db1096:3315 - T178359 (duration: 00m 44s)
08:11 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1096:3315 - T178359 (duration: 00m 45s)
07:53 moritzm: installing curl security updates
07:17 marostegui: Compress s1 on db1099 - T178359
07:08 marostegui: Stop MySQL on db1044 as it will be decommissioned - T181696
07:05 _joe_: playing with puppetdb status for ores2003 (deactivating/reactivating node)
06:40 marostegui: Stop MySQL on db1098 to clone db1096.s6 - T178359
06:39 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1044 from config as it will be decommissioned - T181696 (duration: 00m 45s)
06:38 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1044 from config as it will be decommissioned - T181696 (duration: 00m 45s)
06:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1098 - T178359 (duration: 00m 46s)
06:21 marostegui: Deploy alter table on s3 master (db1075) without replication - T174569
02:32 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.10) (duration: 06m 28s)

2017-12-03

15:33 ejegg: disabled CiviCRM bounce processing job
12:17 akosiaris: empty ganeti1006, it had issues this morning per T181121
12:06 marostegui: Fix dbstore1002 replication
07:44 akosiaris: ran puppet on conf2002, etcdmirror-conftool-eqiad-wmnet got started again
05:11 andrewbogott: deleting files on labsdb1003 /srv/tmp older than 30 days
03:57 no_justification: gerrit2001: icinga is flapping on the gerrit process/systemd check, but this is kind of known (not sure why it's doing this all of a sudden). It's not letting me acknowledge it, but it's fine/harmless. Cf T176532

2017-12-02

17:55 marostegui: Reboot db1096.s5 to pick up the correct innodb_buffer_pool size after finishing compressing s5 - T178359
03:51 hoo: Ran "scap pull" on snapshot1001, after final T181385 tests
00:03 mutante: tried one more time on db2028,db2029, both trusty. on db2028: gmond was running as user ganglia-monitor, failed, had to manually kill the process, run puppet again then ok. on db2029, gmond was running as "499" but puppet just ran and removed it without manual intervention. (T177225)

2017-12-01

23:15 urandom: starting cassandra bootstrap, restbase1012-b - T179422
21:49 mutante: db2029 - removing ganglia-monitor, testing to kill gmond, running puppet to figure out how to cleanly remove it on trusty
21:12 mutante: db2023 killed gmond (ganglia-monitor) process manually which was still running even though ganglia-monitor package was removed and caused puppet breakage (it seems only on trusty). after that puppet run is clean again and ganglia removed. (T177225) (https://gerrit.wikimedia.org/r/#/c/394647/1)
20:18 awight@tin: Started deploy [ores/deploy@9afbf14]: (non-production) Test ORES deployment to ores100*
20:17 awight@tin: Finished deploy [ores/deploy@9afbf14]: (non-production) Test ORES deployment to ores1001 (duration: 02m 31s)
20:15 awight@tin: Started deploy [ores/deploy@9afbf14]: (non-production) Test ORES deployment to ores1001
20:03 aaron@tin: Synchronized php-1.31.0-wmf.10/includes/libs/objectcache/WANObjectCache.php: f096d0b465b75d - temp logging for statsd spam (duration: 00m 45s)
18:59 demon@tin: Synchronized wmf-config/CommonSettings-labs.php: no-op (duration: 00m 46s)
18:22 mutante: Phabricator: restarting Apache for php-curl update
18:21 _joe_: restarting apache2 on the codfw puppetmasters
18:06 marktraceur@tin: Synchronized php-1.31.0-wmf.10/extensions/UploadWizard/resources/controller/uw.controller.Deed.js: (no justification provided) (duration: 00m 46s)
17:49 mutante: phab2001 - restarted apache
17:33 herron: stopped ircecho on einsteinium
17:00 awight@tin: Unlocked for deployment [ores/deploy]: Don't deploy while we're messing with git-lfs (duration: 00m 14s)
17:00 awight@tin: Locking from deployment [ores/deploy]: Don't deploy while we're messing with git-lfs (planned duration: 16666666666m 39s)
17:00 awight@tin: Locking from deployment [ores/deploy]: Don't deploy while we're messing with git-lfs (planned duration: -1m 59s)
16:59 awight@tin: Unlocked for deployment [ores/deploy]: Don't deploy while we're messing with git-lfs (duration: 00m 07s)
16:59 awight@tin: Locking from deployment [ores/deploy]: Don't deploy while we're messing with git-lfs (planned duration: 60m 00s)
16:34 jynus: stopping db2092 to clone s1 to db2085
16:24 urandom: starting cassandra bootstrap, restbase1012-a -- T179422
15:27 godog: bounce uwsgi on labmon1001 - stuck
15:21 moritzm: installing nspr security updates on trusty
15:17 moritzm: installing ffmpeg security updates
15:14 gehel@tin: Finished deploy [kartotherian/deploy@df7ebff]: testing new kartotherian packaging on maps-test2003 (duration: 00m 20s)
15:14 jynus@tin: Synchronized wmf-config/db-codfw.php: Undeploy db2092, use db2085 for s1 (duration: 00m 45s)
15:14 gehel@tin: Started deploy [kartotherian/deploy@df7ebff]: testing new kartotherian packaging on maps-test2003
15:14 moritzm: installing libxcursor security updates on trusty
15:09 jynus@tin: Synchronized wmf-config/db-eqiad.php: Undeploy db2092, use db2085 for s1 (duration: 00m 45s)
14:24 awight@tin: Finished deploy [ores/deploy@532bd0b]: (non-production) Update ORES on new cluster (duration: 02m 06s)
14:22 awight@tin: Started deploy [ores/deploy@532bd0b]: (non-production) Update ORES on new cluster
14:22 akosiaris: upload apertium-crh-tur_0.3.0~r83159-1+wmf1 to apt.wikimedia.org/jessie-wikimedia component main. T181465
14:10 herron: cutting all puppet service records over to codfw puppet 4 masters
12:44 elukey: reboot druid1001 for kernel+jvm updates - T179943
12:11 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2055 (duration: 00m 45s)
11:59 jynus: restarting and upgrading mysql on labsdb1004
11:28 jynus: upgrading and restarting dbstore2001
11:14 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2055 (duration: 00m 46s)
11:08 marostegui: Stop MySQL on db2055 for testing
11:04 marostegui: Update MySQL on db1039 for testing
10:57 elukey: reboot analytics1028 for kernel + jvm updates (Hadoop HDFS journalnode) - T179943
10:56 godog: delete docker diskspace metrics from labs - T181476
10:21 godog: initial purge of old table metrics from graphite2002 - T181689
10:18 moritzm: reimaging mw1259 (video scaler) to stretch, will be kept disabled initially (with some live tests starting next week)
09:55 akosiaris: run memtester 61G on ganeti1008 T181121
09:23 elukey: reboot analytics104* for kernel+jvm updates - T179943
09:23 akosiaris: powercycle ganeti1008 T181121, it's largely unresponsive
08:41 marostegui: Remove db1046 and db1047 from tendril - T156844
08:40 elukey: reboot the remaining analytics103* hadoop workers to pick up kernel+jvm updates - T179943
08:28 akosiaris: empty ganeti1008, move VMs to ganeti1006 T181121
08:20 akosiaris: repool ganeti1005, ganeti1006 to empty ganeti1008 T181121
07:51 akosiaris: upload apertium-tur_0.2.0~r83161-1+wmf1, apertium-crh_0.2.0~r83161-1+wmf1 to apt.wikimedia.org/jessie-wikimedia component main. T181465
07:14 marostegui: Logging retroactively for the record, restarting MySQL on db1039
05:14 legoktm@tin: Synchronized wmf-config/InitialiseSettings.php: touch (duration: 00m 44s)
01:46 hoo: Ran scap pull on mwdebug1001 after T181385 testing
00:13 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: HTML5 sections be upon us! (duration: 00m 45s)
00:04 hoo: Killed all remaining Wikidata JSON/RDF dumpers, due to T181385. This means no dumps this week!

2017-11-30

23:54 demon@tin: Synchronized wmf-config/extension-list-labs: no-op (duration: 00m 45s)
23:39 addshore@tin: Synchronized wmf-config: wdbuild: T173818 T177060 Add wikidata extensions to extension-list (duration: 00m 46s)
23:35 addshore@tin: Synchronized wmf-config: wdbuild: T173818 Remove Wikibase-buildentry.php config file (empty) (duration: 00m 46s)
23:34 addshore@tin: Synchronized wmf-config/Wikibase.php: wdbuild: T173818 Remove Wikibase-buildentry.php config file (empty) (duration: 00m 45s)
23:30 addshore@tin: Synchronized wmf-config: wdbuild: T173818 Remove wmgUseWikidataBuild (duration: 00m 46s)
23:29 addshore@tin: Synchronized wmf-config/Wikibase-buildentry.php: wdbuild: T173818 Remove wmgUseWikidataBuild (duration: 00m 45s)
23:21 addshore@tin: Synchronized wmf-config: wdbuild: T173818 wdbuild: Stop loading from build on ALL WIKIS The build is dead! Mwahahaaa (duration: 00m 47s)
23:18 bsitzmann@tin: Finished deploy [mobileapps/deploy@4305d96]: Update mobileapps to 4317ea5 (T181743) (duration: 04m 50s)
23:15 addshore@tin: Synchronized wmf-config: wdbuild: wdbuild: Stop loading from build on all wikis (except the one i really dont want to break) (duration: 00m 46s)
23:14 bsitzmann@tin: Started deploy [mobileapps/deploy@4305d96]: Update mobileapps to 4317ea5 (T181743)
23:06 addshore@tin: Synchronized wmf-config: wdbuild: wdbuild: Stop loading from build on group1 (duration: 00m 46s)
22:55 addshore@tin: Synchronized wmf-config: wdbuild: wdbuild: Stop loading from build on group0 (duration: 00m 46s)
22:42 addshore: BETA ONLY was a lie on that last one ...
22:41 addshore@tin: Synchronized wmf-config: wdbuild: BETA ONLY wdbuild: Stop loading from build on test and testwikidata (duration: 00m 47s)
22:40 demon@tin: Pruned MediaWiki: 1.31.0-wmf.8 [keeping static files] (duration: 02m 33s)
22:27 addshore@tin: Synchronized wmf-config: wdbuild: BETA ONLY gerrit:394411 (duration: 00m 47s)
21:35 addshore@tin: Synchronized wmf-config: wdbuild: BETA ONLY (duration: 00m 47s)
21:31 jynus: upgrading labsdb1010 and restarting mariadb
21:23 mutante: powercycling kafka1018 (was down in Icinga and saw in SAL: reboot kafka10[12-22] for kernel + jvm updates - T179943)
21:18 jynus: upgrade and restart db2078
21:14 addshore@tin: Synchronized wmf-config: wdbuild: BETA ONLY (duration: 00m 47s)
21:01 addshore@tin: Synchronized wmf-config: wdbuild: T173818: add switch to ease killing (again) (duration: 00m 46s)
20:59 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: wdbuild: T173818: add switch to ease killing (again) (duration: 00m 45s)
20:49 XenoRyet: Updated tools from 6e604fd to 626fe02
20:40 bd808: Sending Toolforge survey final reminder emails from silver for T177126
20:32 addshore@tin: Synchronized wmf-config: REVERT wdbuild: T173818: add switch to ease killing (duration: 00m 47s)
20:30 addshore@tin: Synchronized wmf-config: wdbuild: T173818: add switch to ease killing (duration: 00m 47s)
20:13 ejegg: re-adjusted Civi job timings. QC every odd min for 105 sec, TY every even min for 70 sec after 45 sec delay
20:12 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.10
19:55 ejegg: 105 seconds for each job
19:55 ejegg: adjusted timings of Civi jobs to let TY and QC run concurrently
19:37 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Temporary disable remex html" T178632 (duration: 00m 49s)
19:20 thcipriani@tin: Synchronized php-1.31.0-wmf.10/includes/htmlform/fields/HTMLMultiSelectField.php: SWAT: HTMLMultiSelectField: Allow formatting in section headings in OOUI mode T181698 (duration: 00m 49s)
19:12 thcipriani@tin: Synchronized wmf-config: SWAT: Enable MP3 uploads on Commons T120288 (duration: 00m 51s)
17:57 andrewbogott: upgrading labtestpuppetmaster2001 to puppet 4.8
17:47 moritzm: uploaded prometheus-openldap-exporter 0+git20171128-1 for jessie-wikimedia (T181511)
17:06 herron: beginning cut over of esams to codfw puppet 4 masters
16:44 urandom: drop (erroneous) legacy tables from -ng cassandra cluster - T181689
16:12 elukey: drain and reboot analytics1031->39 to pick up jvm+kernel updates - T179943
15:59 jynus: setup prometheus with unix_socket on new server db1107 and db1108
15:19 gehel: restart blazegraph on wdqs1004
15:16 jynus@tin: Synchronized wmf-config/db-eqiad.php: Increase db1110 load (duration: 00m 48s)
15:11 herron: beginning cut over of ulsfo to codfw puppet 4 masters
14:50 zeljkof: EU SWAT finished
14:46 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable WikiLove Extension on pa.wiki (T178919) (duration: 00m 49s)
14:28 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add import sources to de.wiki (T181695) (duration: 00m 49s)
14:20 zfilipin@tin: Synchronized php-1.31.0-wmf.10/extensions/ContentTranslation/modules/dashboard/ext.cx.dashboard.js: SWAT: Set default languages after fetching valid languages (duration: 00m 49s)
14:19 marostegui: Deploy schema change on s3 - db1078 - T174569
13:23 addshore@tin: Synchronized php-1.31.0-wmf.10/extensions/AdvancedSearch/modules/ext.advancedSearch.init.js: pre-swat: T181644 Force search profile advanced in AdvancedSearch gerrit:394297 (duration: 00m 48s)
13:22 addshore@tin: Synchronized php-1.31.0-wmf.8/extensions/AdvancedSearch/modules/ext.advancedSearch.init.js: pre-swat: T181644 Force search profile advanced in AdvancedSearch gerrit:394299 (duration: 00m 50s)
13:18 herron: cutting codfw puppet agents over to puppetmaster2001.codfw.wmnet
12:07 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Tackle s4 DB weights to make them more equal (duration: 00m 48s)
11:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1081 and reduce traffic for db1091 (duration: 00m 50s)
10:33 jynus: stopping replication on db1044 and db1095 (s3)
09:16 moritzm: installing exim security updates on stretch (jessie/trusty not affected)
09:14 elukey: drain and reboot analytics1029/1030 for jvm+kernel updates (Hadoop worker canaries)
09:05 godog: add 200G of space to graphite2002 carbon lv
08:25 moritzm: rolling restart of mw canaries to pick up curl security update
08:21 marostegui: Enable GTID on s8 eqiad hosts that do not have it enabled (db1109, db1104, db1101, db1092, db1087, db1063) - T177208
07:48 moritzm: installing curl security updates
07:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1056 (duration: 01m 18s)
06:48 marostegui: Deploy schema change on dbstore1001 s3 - T174569
06:26 marostegui: Deploy schema change on s3 db1077 - T174569
02:48 ebernhardson@tin: Finished deploy [search/mjolnir/deploy@7aa39b7]: (no justification provided) (duration: 03m 15s)
02:45 ebernhardson@tin: Started deploy [search/mjolnir/deploy@7aa39b7]: (no justification provided)
02:29 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.8) (duration: 08m 43s)
01:17 ejegg: updated fundraising tools from 6d4b6f3 to 6e604fd
01:06 twentyafterfour: finished phabricator upgrade, service is online and appears to be functioning normally
01:03 twentyafterfour: Starting phabricator upgrade & maintenance. Service will be offline for less than 5 minutes.
01:02 bsitzmann@tin: Finished deploy [mobileapps/deploy@fa2a877]: Update mobileapps to dcea7d3 (T181004) (duration: 06m 08s)
00:56 bsitzmann@tin: Started deploy [mobileapps/deploy@fa2a877]: Update mobileapps to dcea7d3 (T181004)
00:53 maxsem@tin: Finished scap: Message updates for https://gerrit.wikimedia.org/r/#/c/394155/ (duration: 26m 38s)
00:48 ejegg: re-enabled CiviCRM jobs
00:42 ejegg: updated CiviCRM from 0f95f3e to e81228f
00:40 ejegg: disabled CiviCRM jobs
00:31 ejegg: updated fundraising dashboard from 6ee6567 to 1141317
00:27 maxsem@tin: Started scap: Message updates for https://gerrit.wikimedia.org/r/#/c/394155/
00:24 maxsem@tin: Synchronized php-1.31.0-wmf.10/extensions/ContentTranslation/: https://gerrit.wikimedia.org/r/#/c/394206/ (duration: 00m 52s)
00:21 ejegg: added weekly Ingenico audit processing job in makemissing mode

2017-11-29

23:20 eileen: update civicrm from a1022cf to 0f95f3e (Benevity import fix)
22:23 hashar: Nodepool had some troubles spawning new instances from 21:09 to 21:36, and took a while to recover. Issue similar to T170492#3581822
21:51 SMalyshev: starting wikidata reindex (T181426)
21:23 no_justification: docroot sync was for If1afa59a
21:21 demon@tin: Synchronized docroot/: (no justification provided) (duration: 00m 49s)
20:29 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.10
20:26 demon@tin: Synchronized php: symlink bump (duration: 00m 48s)
20:05 urandom: restarting cassandra bootstrap of restbase1012-a (T179422)
19:55 mutante: restarting gerrit to apply config change and set gitBasicAuth to true to unblock T171758 (gerrit:391865)
19:50 herron: beginning rolling cut over of eqiad scb hosts to codfw puppet 4 masters
19:15 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/394059/3 (duration: 00m 49s)
18:57 herron: beginning cutover of codfw lvs2* systems to codfw puppet 4 masters. first standby nodes, then active nodes
18:12 marostegui: Compress s5 on db1096 - T178359
18:10 jynus@tin: Synchronized wmf-config/db-eqiad.php: Increase db1110 load (duration: 00m 53s)
18:05 awight@tin: Finished deploy [ores/deploy@532bd0b]: (non-production) Update ORES on new cluster (duration: 01m 11s)
18:04 awight@tin: Started deploy [ores/deploy@532bd0b]: (non-production) Update ORES on new cluster
18:03 herron: beginning cut over of codfw cp2* servers to codfw puppet 4 masters
17:40 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool db1110 with low load (duration: 00m 48s)
17:40 godog: bootstrapping restbase1012-a - T179422
17:19 herron: beginning cut over of cp200[12] to codfw puppet 4 masters
17:16 ejegg: increased time limit for donation queue consumer from 60 to 70 seconds, reduced thank you job from 55 to 45 seconds
17:03 ejegg: re-enabled Civi jobs
16:58 ejegg: disabled Civi jobs for Deadlock-retry update
16:46 jynus@tin: Synchronized wmf-config/db-codfw.php: Update db1110 ip (duration: 00m 50s)
16:43 jynus@tin: Synchronized wmf-config/db-eqiad.php: Update db1110 ip (duration: 00m 49s)
15:02 chasemp: purge old qcow2 images from under /home on labtestcontrol2001
15:00 awight: Restarting ORES celery workers manually, T181538
14:39 _joe_: reloading apache on puppetmasters to pick up the configuration changes
14:37 awight@tin: Started restart [ores/deploy@e58bfbf]: Restart ORES services (take 2), T181538
14:36 elukey: reboot druid100[456] for jvm+kernel updates - T179943
14:35 awight@tin: Finished deploy [ores/deploy@e58bfbf]: Restart ORES services, T181538 (duration: 00m 16s)
14:35 awight@tin: Started deploy [ores/deploy@e58bfbf]: Restart ORES services, T181538
14:25 zeljkof: EU SWAT finished
14:18 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: Define throttle rule (T181367) (duration: 00m 49s)
14:14 herron: beginning cut over of cache::misc and codfw cache::canary cp servers to codfw puppet4 masters
14:13 moritzm: rebooting neodymium for kernel update to 4.9.51
14:12 godog: reimage restbase1012 - T179422
14:10 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT T180291 T180128 AdvancedSearch for arwiki (duration: 00m 49s)
14:07 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT T180128 AdvancedSearch for dewiki (duration: 00m 50s)
13:34 awight@tin: Finished deploy [ores/deploy@e58bfbf]: (non-production) Update ORES on new cluster (take 3) (duration: 16m 49s)
13:18 elukey: reboot kafka100[23] for jvm+kernel updates - T179943
13:17 awight@tin: Started deploy [ores/deploy@e58bfbf]: (non-production) Update ORES on new cluster (take 3)
12:34 jynus@tin: Synchronized wmf-config/db-eqiad.php: Setup s8 on eqiad, with no wikis (duration: 00m 48s)
12:24 jynus: scap pull on mwdebug1001
12:23 jynus@tin: Synchronized wmf-config/db-codfw.php: Pool db1096:3315 (duration: 00m 48s)
12:01 marostegui: Deploy schema change on db1072 (sanitarium master) on s3 with replication enabled to replicate to labs - T174569
11:30 elukey: reboot kafka1001 for kernel + jvm updates - T179943
10:15 akosiaris: disable puppet on oresrdb* for merging https://gerrit.wikimedia.org/r/#/c/394022/. T181563
09:50 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1110 (duration: 00m 48s)
09:46 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1110 (duration: 00m 49s)
09:39 godog: upload cassandra-tools-wmf 1.0.2-1 - T181438
09:28 mobrovac@tin: Finished deploy [electron-render/deploy@94d27d7]: Update to electron v1.7.9 and start using the Charter font - T181200 (duration: 04m 03s)
09:24 mobrovac@tin: Started deploy [electron-render/deploy@94d27d7]: Update to electron v1.7.9 and start using the Charter font - T181200
09:04 godog: bootstrap restbase1007-c - T179422
08:41 moritzm: uploaded prometheus-openldap-exporter 0+git20171128-1 for jessie-wikimedia (T181511)
08:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1096 - T178359 (duration: 00m 48s)
07:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1099:3318 and db1055 - T178359 (duration: 00m 48s)
07:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1099:3318 traffic - T178359 (duration: 00m 45s)
07:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1099:3318 with low weight - T178359 (duration: 00m 45s)
06:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1055 - T178359 (duration: 00m 48s)
06:42 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1099:3311 and db1099:3318 to the config (depooled) T178359 (duration: 00m 48s)
06:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1099:3311 and db1099:3318 to the config (depooled) T178359 (duration: 00m 49s)
06:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1055 with lower weight - T178359 (duration: 00m 50s)
06:17 ebernhardson@tin: Finished deploy [search/mjolnir/deploy@7aa39b7]: (no justification provided) (duration: 04m 27s)
06:12 ebernhardson@tin: Started deploy [search/mjolnir/deploy@7aa39b7]: (no justification provided)
03:39 ejegg: reduced CiviMail sample rate to 0.35
03:09 ejegg: reduced donation queue consumer time limit from 70 to 60 seconds, increased ty mail batch time limit from 45 to 55 seconds
02:50 ejegg: reduced donation queue consumer time limit from 75 to 70 seconds, increased ty mail batch time limit from 40 to 45 seconds
02:44 ejegg: reduced CiviMail record creation rate from 100% to 50%
02:24 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.8) (duration: 06m 08s)
01:25 mutante: snapshot1001 - closed idle screen session
01:22 mutante: analytics1003 - closed idle screen session
01:21 mutante: mw1276 - run "scap pull" to get in sync after hardware issue, then pooled again (T181397)
01:09 mutante: restarting ircecho - it stopped talking
01:07 mutante: forcing puppet run on all labvirt* machines to clean out Icinga alerts
00:44 awight@tin: Synchronized php-1.31.0-wmf.10/extensions/ORES: Hotfix to mitigate cache stampeding, T181567 (duration: 00m 49s)
00:33 reedy@tin: Synchronized php-1.31.0-wmf.10/includes/logging/LogPager.php: Fix fatal on Special:Log T181565 (duration: 00m 48s)
00:32 awight@tin: Synchronized php-1.31.0-wmf.8/extensions/ORES: Hotfix to mitigate cache stampeding, T181567 (duration: 00m 50s)
00:18 Jamesofur: deleted 6 archived files from servers for legal compliance
00:08 ejegg: updated fundrasing dashboard from df94248 to 6ee6567

2017-11-28

23:58 hoo: Ran scap pull on mwdebug1001 after T181385 related testing
23:07 akosiaris@tin: Synchronized wmf-config/CommonSettings.php: T181538 (duration: 00m 49s)
22:31 akosiaris@tin: Synchronized wmf-config/CommonSettings.php: (no justification provided) (duration: 00m 49s)
22:31 akosiaris: deploy wmf-config/CommonSettings.php for ORES internal discovery URL, https://gerrit.wikimedia.org/r/#/c/393924/ T181538
21:41 akosiaris: disable ORES queue redis persistency by config set appendonly no on oresrdb1001
21:24 hoo: Manually killed all remaining Wikidata TTL (RDF) dumpers on snapshot1007. Some shards failed due to the db1110 depool.
21:23 hoo: Manually killed all remaining Wikidata JSON dumpers on snapshot1007. Some shards failed due to the db1110 depool.
20:56 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: removing aawiki from group0
20:42 gehel: repooling elastic2004 after RAID controller maintenance - T181412
20:42 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.10
20:28 mutante: forcing puppet run on cache misc to revert "failover ORES to codfw"
20:19 demon@tin: Synchronized scap/plugins/prep.py: no-op (duration: 00m 48s)
20:16 demon@tin: Synchronized dblists/group0.dblist: adding some new wikis (duration: 00m 48s)
18:57 demon@tin: Finished scap: bootstrap wmf.10 (duration: 35m 20s)
18:51 demon@tin: Finished deploy [gerrit/gerrit@571cf4c]: deploying 2.15+ polygerrit style changes (duration: 00m 09s)
18:51 demon@tin: Started deploy [gerrit/gerrit@571cf4c]: deploying 2.15+ polygerrit style changes
18:40 akosiaris: revert weight changes for scb1001, scb1002 T181835
18:39 akosiaris@puppetmaster1001: conftool action : set/weight=10; selector: scb1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
18:39 akosiaris@puppetmaster1001: conftool action : set/weight=10; selector: scb1001.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
18:35 awight@tin: Finished deploy [ores/deploy@e58bfbf]: (non-production) Update ORES on new cluster (take 2) (duration: 04m 30s)
18:32 akosiaris: force puppet run on cache::misc boxes T181538
18:30 awight@tin: Started deploy [ores/deploy@e58bfbf]: (non-production) Update ORES on new cluster (take 2)
18:28 urandom: (re)bootstrapping cassandra, restbase1007-b - T179422
18:22 demon@tin: Started scap: bootstrap wmf.10
18:18 awight@tin: Finished deploy [ores/deploy@e58bfbf]: (non-production) Update ORES on new cluster (duration: 01m 41s)
18:17 awight@tin: Started deploy [ores/deploy@e58bfbf]: (non-production) Update ORES on new cluster
18:00 akosiaris: force stop celery-ores-worker on scb1001
17:58 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: scb1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
17:58 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: scb1001.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
17:46 urandom: decommissioning cassandra, restbase1007-b - T179422
17:42 urandom: restart cassandra, restbase1007, to pickup logstash java deps - T179422
16:45 ejegg: updated fundraising dashboard from d8c86e7 to df94248
16:25 bblack: mw1329 boot to PXE (should come up with new .66 IP)
15:39 jynus@tin: Synchronized wmf-config/db-eqiad.php: depool db1110 (duration: 00m 44s)
15:38 bblack: powered off mw1329
15:06 marostegui: Compress s5 on db1099
15:04 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Revert "Comply wikidata with new ores thresholds" (duration: 00m 45s)
14:56 marostegui: Deploy schema change on s3 on dbstore1002 - T174569
14:45 gehel@tin: Finished deploy [kartotherian/deploy@cb9b1ef]: testing new kartotherian packaging on maps-test2003 (duration: 00m 02s)
14:45 gehel@tin: Started deploy [kartotherian/deploy@cb9b1ef]: testing new kartotherian packaging on maps-test2003
14:41 otto@tin: Finished deploy [eventlogging/analytics@c464b8c]: Fixing bug where userAgent set by client producer was not used T178440 (duration: 00m 02s)
14:40 otto@tin: Started deploy [eventlogging/analytics@c464b8c]: Fixing bug where userAgent set by client producer was not used T178440
14:40 otto@tin: Started deploy [eventlogging/analytics@c464b8c]: Fixing bug where userAgent set by client producer was not used
14:39 gehel@tin: Finished deploy [kartotherian/deploy@cb9b1ef]: testing new kartotherian packaging on maps-test2003 (duration: 00m 18s)
14:39 gehel@tin: Started deploy [kartotherian/deploy@cb9b1ef]: testing new kartotherian packaging on maps-test2003
14:37 marostegui: Stop MySQL on db1055 to clone db1099:3311 - T178359
14:34 gehel@tin: Finished deploy [kartotherian/deploy@55c5da4]: (no justification provided) (duration: 00m 11s)
14:34 gehel@tin: Started deploy [kartotherian/deploy@55c5da4]: (no justification provided)
14:21 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1055 - T178359 (duration: 00m 44s)
14:17 elukey: reboot kafka10[12-22] for kernel + jvm updates - T179943
14:03 elukey: reboot kafka200[123] for kernel + jvm updates - T179943
12:25 godog: cleanup wanobjectcache metrics with hashes - T178531
12:14 godog: bounce carbon-frontend-relay after https://gerrit.wikimedia.org/r/393749
11:25 hoo: Manually re-started Wikidata JSON dumps on snapshot1007, got stuck after db1082 went down.
10:55 jynus: stop db1044 replication for db1095 master switchover
10:49 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1082. Depool db1044 (duration: 00m 44s)
10:44 mobrovac@tin: Synchronized wmf-config/jobqueue.php: Process wikibase-addUsagesForPage only via EventBus - T175212 (duration: 00m 44s)
10:08 ppchelko@tin: Finished deploy [cpjobqueue/deploy@c4b9e16]: Enable wikibase-addUsagesForPage with low concurrency (duration: 00m 29s)
10:07 ppchelko@tin: Started deploy [cpjobqueue/deploy@c4b9e16]: Enable wikibase-addUsagesForPage with low concurrency
09:35 jynus: restart db1087 for maintenance
09:22 jynus: restart db1082
09:19 godog: unmask and restart restbase1007-b - T179422
09:15 moritzm: installibg libxml-libxml-perl security updates on trusty (Debian already fixed)
09:11 ppchelko@tin: Finished deploy [cpjobqueue/deploy@b0b1793]: Remove double-processing created for consumer group renaming (duration: 00m 28s)
09:10 ppchelko@tin: Started deploy [cpjobqueue/deploy@b0b1793]: Remove double-processing created for consumer group renaming
09:04 marostegui: Drop database log from dbstore1002 - T156844
08:57 ppchelko@tin: Finished deploy [cpjobqueue/deploy@2212086]: Move enabled jobs config to vars.yaml (duration: 00m 39s)
08:56 ppchelko@tin: Started deploy [cpjobqueue/deploy@2212086]: Move enabled jobs config to vars.yaml
08:38 jynus@tin: Synchronized wmf-config/db-eqiad.php: depool db1082 (duration: 00m 44s)
08:02 mobrovac: bootstrap restbase1007-b - T179422
06:39 marostegui: Stop MySQL on db1099 to copy its content to dbstore1001 and reimage it as multi-instance - T178359
05:23 mutante: labtestpuppetmaster2001 - manually purging ganglia things since puppet is disabled
05:05 mutante: labweb1001/labweb1002: manually purging ganglia package/config/service/unit files because puppet is disabled there (T177225)
04:08 eileen: update process-control to b40eaec, disabling 3 jobs to reduce load for Big english starting: dedupe_civicrm_contacts, omnimail_recipient_load, omnimail_recipient_load_backfill
03:58 demon@tin: Pruned MediaWiki: 1.31.0-wmf.4 (duration: 03m 13s)
03:31 mutante: labservices1001/1002,labtestservices2001 - remove pdns_gmetric cronjobs causing cron spam after ganglia decom from lab*
02:20 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.8) (duration: 05m 23s)

2017-11-27

22:29 mutante: labtestnet2001 - re-enabled puppet to decom ganglia, no errors
22:22 mutante: decom'ing Ganglia from all labtest* hosts - packages removed by puppet etc, might cause some false positives in Icinga but it's ok (T177225)
22:21 otto@tin: Finished deploy [eventlogging/eventbus@e024af3]: deploy revert checked out at 3df06ab (we caught one). T180017 (duration: 00m 10s)
22:21 otto@tin: Started deploy [eventlogging/eventbus@e024af3]: deploy revert checked out at 3df06ab (we caught one). T180017
21:59 awight: clearing ORES threshold caches for ruwiki, frwiki, wikidatawiki.
21:59 awight@tin: Synchronized wmf-config/InitialiseSettings.php: Reenable ORES on frwiki, ruwiki, and wikidata; T181006 (duration: 00m 45s)
21:53 awight@tin: Finished deploy [ores/deploy@e58bfbf]: (codfw) Update ORES for impossible threshold handling and new wikidata editquality model, T179711 T180686 T180450 (duration: 05m 06s)
21:47 awight@tin: Started deploy [ores/deploy@e58bfbf]: (codfw) Update ORES for impossible threshold handling and new wikidata editquality model, T179711 T180686 T180450
21:46 awight@tin: Finished deploy [ores/deploy@e58bfbf]: Update ORES for impossible threshold handling and new wikidata editquality model, T179711 T180686 T180450 (duration: 12m 55s)
21:33 awight@tin: Started deploy [ores/deploy@e58bfbf]: Update ORES for impossible threshold handling and new wikidata editquality model, T179711 T180686 T180450
21:23 awight@tin: Synchronized php-1.31.0-wmf.8/extensions/ORES: ORES error handling for bad thresholds, T181191 (duration: 00m 46s)
21:11 awight@tin: Synchronized wmf-config/InitialiseSettings.php: Temporarily disable ORES on wikidata (duration: 00m 45s)
21:10 awight: Previous “rollback ORES” was from a stale screen session, no actions were taken today.
21:08 awight@tin: Finished deploy [ores/deploy@82a13ae]: Rollback ORES (take 3); 181006 (duration: 9942m 42s)
20:21 demon@tin: i lied, that was for wgStyleVersion removal
20:20 demon@tin: Synchronized wmf-config/CommonSettings.php: cli sapi fixes (duration: 00m 45s)
20:17 ejegg: updated payments-wiki from fea6b37 to f594dfa
20:03 kaldari@tin: Synchronized wmf-config/InitialiseSettings-labs.php: (no justification provided) (duration: 00m 45s)
20:02 kaldari@tin: Synchronized wmf-config/CommonSettings.php: (no justification provided) (duration: 00m 45s)
20:01 kaldari@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 46s)
19:54 bblack: crN-ulsfo: remove lvs400[1-4] from PyBal BGP neighbors list
19:42 catrope@tin: Synchronized php-1.31.0-wmf.8/resources/src/mediawiki.rcfilters/mw.rcfilters.UriProcessor.js: T181100 (duration: 00m 45s)
19:41 demon@tin: Synchronized wmf-config/InitialiseSettings.php: babel officewiki thing (duration: 00m 45s)
19:39 catrope@tin: Synchronized php-1.31.0-wmf.8/includes/specials/SpecialRecentchangeslinked.php: T181100 (duration: 00m 45s)
19:34 otto@tin: Finished deploy [eventlogging/eventbus@3df06ab]: Temp deploy to kafka1001 only to catch bug: T180017 (duration: 00m 12s)
19:34 otto@tin: Started deploy [eventlogging/eventbus@3df06ab]: Temp deploy to kafka1001 only to catch bug: T180017
19:21 catrope@tin: Synchronized wmf-config/Wikibase.php: T176903 (duration: 00m 45s)
19:17 AaronSchulz: Removed more bogus md5 wanobjecache/ metric from graphite[12]001
19:11 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: T179241 (duration: 00m 45s)
19:02 volans: restarted ircecho
18:51 ejegg: updated payments-wiki from 6b3019b to fea6b37
18:50 dcausse: elastic/cirrus: reindexing english group2 wikis: T179945
18:45 volans: stopped ircecho temporary to avoid spam
18:14 XioNoX: pushing v6 gre firewall filter
18:12 gehel@tin: Finished deploy [wdqs/wdqs@7ad20f3]: (no justification provided) (duration: 02m 10s)
18:11 gehel: deploying blazegraph + GUI updates
18:10 gehel@tin: Started deploy [wdqs/wdqs@7ad20f3]: (no justification provided)
18:08 hoo: Killed truthy nt dumpers stuck at 100% CPU (from last week) on snapshot1007 (T181385)
18:06 akosiaris: poweroff ganeti1005, ganeti1006 T181121
18:05 bd808: Sending Toolforge survey 1 week reminder emails from silver for T177126
16:07 bblack: lvs400x - puppet disabled for https://gerrit.wikimedia.org/r/#/c/393610
15:52 herron: beginning canary cutover/deployment of codfw prometheus servers to codfw puppet 4 puppetmasters
15:19 chasemp: disable puppet across cloud things for cleanup
15:11 herron: beginning canary cutover/deployment of codfw elasticsearch servers to codfw puppet 4 puppetmasters
14:22 zeljkof: EU SWAT finished
14:19 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: IP cap lift for Semaine contributive 2017-2018 (T181360) (duration: 00m 45s)
14:10 addshore@tin: Synchronized php-1.31.0-wmf.8/extensions/AdvancedSearch: SWAT AdvancedSearch T181175 T181222, adjust links in beta section & fix usability of mobile search (duration: 00m 46s)
13:31 moritzm: installing nspr security updates on trusty
13:22 elukey: remove eventlogging replication support (log database) from dbstore1002 - T156844
13:17 moritzm: updating nginx on francium
13:01 Pchelolo: stop cpjobqueue in eqiad for backlog accumulation
12:44 Pchelolo: started parsoid linter script to generate load on cpjobqueue (python3 parsoid_reparse.py http://parsoid.discovery.wmnet:8000 --sitematrix --linter-only --skip-closed https://commons.wikimedia.org/w/api.php)
12:44 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: gerrit:393587 BETA wmgMonologChannels FileImporter => debug (duration: 01m 01s)
12:43 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: mw1276.eqiad.wmnet
12:35 moritzm: powercycling mw1276
12:21 addshore@tin: Synchronized wmf-config/CommonSettings-labs.php: gerrit:393582 LABS ONLY Actually call wfLoadExtension for FileExporter & Importer on beta BETA (T181383) (duration: 00m 44s)
12:14 marostegui: Stop replication on db1109 to test table rename for s5/s8 failover
12:09 ppchelko@tin: Finished deploy [cpjobqueue/deploy@47d27dc]: Enable keep-alive T181007 (duration: 01m 03s)
12:08 ppchelko@tin: Started deploy [cpjobqueue/deploy@47d27dc]: Enable keep-alive T181007
11:47 addshore@tin: Synchronized wmf-config/CommonSettings-labs.php: gerrit:393577 LABS ONLY Enable FileImporter & FileExporter on BETA PT2/2 (T181383) (duration: 00m 45s)
11:46 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: gerrit:393577 LABS ONLY Enable FileImporter & FileExporter on BETA PT1/2 (T181383) (duration: 00m 45s)
11:44 addshore@tin: Synchronized wmf-config/extension-list-labs: gerrit:393576 BETA ONLY Add FileExporter & FileImporter to extension-list-labs (duration: 00m 45s)
11:27 hashar: contint1001 enable Icinga "Disks space" notification again. It is no more complaing about Docker partitions | ping mutante | T178454
11:17 godog: bootstrap restbase1007-a - T179422
11:05 jdrewniak@tin: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 45s)
11:05 jdrewniak@tin: Synchronized portals/prod/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 46s)
10:53 moritzm: installing postgresql-common security updates
10:08 ppchelko@tin: Finished deploy [cpjobqueue/deploy@e35aa05]: Revert using keep-alive (duration: 00m 22s)
10:08 ppchelko@tin: Started deploy [cpjobqueue/deploy@e35aa05]: Revert using keep-alive
09:42 ppchelko@tin: Finished deploy [cpjobqueue/deploy@b570d4e]: Make http agent use keep-alive (duration: 00m 48s)
09:41 ppchelko@tin: Started deploy [cpjobqueue/deploy@b570d4e]: Make http agent use keep-alive
09:14 godog: reimage restbase1007 - T179422
08:55 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1097:3314 - T178359 (duration: 00m 43s)
08:40 moritzm: installing openjdk security updates on hadoop, druid and kafka clusters
08:27 marostegui: Deploy schema change on dbstore1002 and dbstore1001 - T174569
08:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1097:3314 weight - T178359 (duration: 00m 45s)
07:20 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1097:3314 weight - T178359 (duration: 00m 45s)
07:10 marostegui: Stop MySQL on db1021 as it will be decommissioned - T181378
06:53 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1021 from the config as it will be decommissioned - T181378 (duration: 00m 44s)
06:52 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1021 from the config as it will be decommissioned - T181378 (duration: 00m 45s)
06:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1097:3314 with low weight - T178359 (duration: 00m 46s)
02:22 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.8) (duration: 05m 44s)

2017-11-25

19:31 marostegui: Set 32:3 disk to offline on db1051
16:09 kartik@tin: Finished deploy [cxserver/deploy@11aecc9]: Update cxserver to 0c242c0, Pin service-runner to 2.4.2 (duration: 03m 29s)
16:05 kartik@tin: Started deploy [cxserver/deploy@11aecc9]: Update cxserver to 0c242c0, Pin service-runner to 2.4.2
16:05 godog: unban statsd traffic from scb on graphite1001 - T181333
15:45 ppchelko@tin: Finished deploy [cpjobqueue/deploy@e35aa05]: Rollback. Disable GC metric reporting T181333 (duration: 00m 31s)
15:45 ppchelko@tin: Started deploy [cpjobqueue/deploy@e35aa05]: Rollback. Disable GC metric reporting T181333
15:37 volans: restarted statsd-proxy on graphite1001 (died during investigation) T181333
14:34 godog: rolling restart of cxserver to alleviate metrics leak - T181333
14:26 godog: restart cxserver on scb100[34] - T181333
14:10 godog: roll-restart cpjobqueue to alleviate metrics leak - T181333
13:40 godog: drop incoming statsd from scb to graphite1001 temporarily - T181333
08:32 ariel@tin: Finished deploy [dumps/dumps@ec21673]: fix abstracts recombine job (duration: 00m 02s)
08:32 ariel@tin: Started deploy [dumps/dumps@ec21673]: fix abstracts recombine job

2017-11-24

13:52 moritzm: removing git packages from jessie-wikimedia/experimental (replaced by component/git)
13:24 moritzm: installing openjpeg2 updates (original security already got installed after initial release, but there was a binNMU for amd64)
13:17 marostegui: Stop replication on db1097 to reimport and recompress commonswiki.watchlist
12:54 jynus: reenabling puppet on db1071
12:50 jynus: resetting replication on es1011 for consistency with other replica sets
12:40 jynus: setting up s8 topology on eqiad
12:38 jynus: disable puppet on db1071 and stop local s5 heartbeat there
12:32 reedy@tin: Synchronized docroot/mediawiki/keys/: Fixup keys (duration: 00m 45s)
12:13 marostegui: Enable GTID on es2018 - T181293
11:57 marostegui: Disable puppet on es2018 - T181293
11:50 jynus@tin: Synchronized wmf-config/db-codfw.php: depool es2018 T181293 (duration: 00m 45s)
11:48 marostegui: Reboot es2018 after full-upgrade - T181293
11:25 marostegui: Restart mysql on es2018
11:25 jynus@tin: Synchronized wmf-config/db-eqiad.php: db2085:3318, db2086:3318 (duration: 00m 43s)
11:24 jynus@tin: Synchronized wmf-config/db-codfw.php: Pool db2038, db2085:3318, db2086:3318 (duration: 00m 45s)
10:55 marostegui: Restart MySQL on db2086 to move s5 to s8
10:37 jynus: cancelling db2085 restart, only doing mysql:s5
10:35 jynus: restarting db2085 (including both s5 and s3 instances)
10:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool all future s8 slaves for a topology change - T177208 (duration: 00m 45s)
10:22 moritzm: installing ca-cerfificates updates on trusty hosts
09:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1097:3315 and db1092 (duration: 00m 45s)
09:50 jynus: restarting db2045
09:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1101:3318 db1097:3315 abd db1092 (duration: 00m 45s)
08:49 marostegui: Stop MySQL on db1092
08:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1101:3318 in s5 to warm it up and depool db1092 - T178359 T177208 (duration: 00m 45s)
08:40 moritzm: installing java security updates on notebook* hosts
08:38 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Slowly pool db1097:3315 - T178359 (duration: 00m 45s)
08:37 marostegui@tin: Synchronized wmf-config/db-codfw.php: Slowly pool db1097:3315 - T178359 (duration: 00m 45s)
08:20 moritzm: installing java security updates on meitnerium
08:15 moritzm: installing java security updates on stat1004
08:14 hashar: restarting jenkins on contint1001 for a java update
08:07 elukey: re-enabling piwik on bohrium (only VM running on ganeti1006 atm) after mysql tables restore completed
06:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Slowly pool db1101:3318 in s5 to warm it up - T178359 (duration: 00m 45s)
06:46 marostegui@tin: Synchronized wmf-config/db-codfw.php: Slowly pool db1101:3318 in s5 to warm it up - T178359 (duration: 00m 49s)
00:12 reedy@tin: Synchronized docroot/mediawiki/keys/: Add Brian Wolff's key (duration: 00m 45s)

2017-11-23

23:42 reedy@tin: Synchronized wmf-config/: Simplify variable assignment (duration: 00m 47s)
23:25 reedy@tin: Synchronized composer.lock: (no justification provided) (duration: 00m 44s)
23:24 reedy@tin: Synchronized composer.json: (no justification provided) (duration: 00m 45s)
23:23 reedy@tin: Synchronized phpcs.xml: (no justification provided) (duration: 00m 45s)
22:39 demon@tin: Synchronized wmf-config/: style fixes, no-op (duration: 00m 47s)
22:19 demon@tin: Synchronized wmf-config/CommonSettings.php: no-op, moving stuff around (duration: 00m 47s)
21:34 ariel@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2251.codfw.wmnet
21:15 apergos: rebooted mw2251 after unresponsive on mgmt console and no ping
20:49 demon@tin: Synchronized static/images/project-logos/nowikimedia.png: logo update (duration: 02m 51s)
17:29 akosiaris: set ganeti1006 as drained. ganeti1005 was already set. That will prevent scheduling VMs on those. T181121
17:01 akosiaris: force powercycle on ganeti1006 T181121
16:30 akosiaris: gnt-node failover ganeti1006
15:50 jynus: disable puppet on db2023, db2038 for deployment whle transfer is ongoing
15:23 moritzm: rebooting mwdebug* servers for update to 4.9.51
14:10 marostegui: Stop Mysql on db1101.s5 to clone db1097 - T178359
14:09 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1053 in s2 as vslow to replace db1021 - T134476 (duration: 00m 45s)
13:13 jynus: sutting down db2023 and db2038 for cloning
12:47 reedy@tin: Synchronized wmf-config/CommonSettings.php: GWToolset migratory config T87928 (duration: 00m 46s)
12:40 moritzm: migrating instances off ganeti2001 / kernel reboot to 4.9.51
12:10 moritzm: migrating instances off ganeti2002 / kernel reboot to 4.9.51
11:52 moritzm: migrating instances off ganeti2003 / kernel reboot to 4.9.51
11:37 kartik@tin: Finished deploy [cxserver/deploy@51b78ce]: T181209 Update cxserver to e8fe3f0 to fix Youdao MT (duration: 03m 01s)
11:35 moritzm: migrating instances off ganeti2004 / kernel reboot to 4.9.51
11:35 moritzm: migrating instances off ganeti200 / kernel reboot to 4.9.51
11:34 kartik@tin: Started deploy [cxserver/deploy@51b78ce]: T181209 Update cxserver to e8fe3f0 to fix Youdao MT
11:23 moritzm: migrating instances off ganeti2005 / kernel reboot to 4.9.51
11:15 moritzm: migrating instances off ganeti2006 / kernel reboot to 4.9.51
11:06 moritzm: migrating instances off ganeti2007 / kernel reboot to 4.9.51
10:57 moritzm: migrating instances off ganeti2008 / kernel reboot to 4.9.51
10:07 moritzm: rebooting sarin for update to 4.9.51
09:16 jynus@tin: Synchronized wmf-config/db-codfw.php: mariadb: Switchover s5 codfw master (duration: 00m 45s)
08:56 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1072 (duration: 00m 44s)
08:44 jynus: stopping and restarting db2052
08:31 moritzm: installing bdb security updates on trusty
08:29 jynus: starting switchover of db2023 to db2052
08:21 marostegui: Stop MySQL on db1072 to MySQL upgrade and kernel upgrade
08:21 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1072 (duration: 00m 45s)
07:28 marostegui: Stop MySQL on db1021 and db1053 to clone db1053
07:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1021 - T134476 (duration: 00m 45s)
07:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Move db1053 to s2 to replace db1021 as vslow, dump slave - T134476 (duration: 00m 45s)
06:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1089 - T180045 (duration: 00m 45s)
06:55 marostegui: Drop index on ores_classification on s1 - T180045
06:53 marostegui: Drop index on ores_classification on s2 - T180045
06:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1089 - T180045 (duration: 00m 45s)
06:44 legoktm: legoktm@tin:~$ echo "https://www.mediawiki.org/keys/keys.html" | mwscript purgeList.php
06:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1051 and db1063 in vslow service for s5 to warm them up for the s8 split - T177208 (duration: 00m 46s)
05:58 kartik@tin: Finished deploy [cxserver/deploy@5b35ed5]: Update cxserver to e8fe3f0 (duration: 04m 13s)
05:54 kartik@tin: Started deploy [cxserver/deploy@5b35ed5]: Update cxserver to e8fe3f0
02:23 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.8) (duration: 05m 27s)
01:36 demon@tin: Synchronized wmf-config/InitialiseSettings.php: dropping education program from cswiki (duration: 00m 45s)
01:29 demon@tin: Synchronized wmf-config/CommonSettings.php: Set $wgCentralAuthGlobalBlockInterwikiPrefix (duration: 00m 45s)
01:26 demon@tin: Synchronized wmf-config/InitialiseSettings-labs.php: no-op (duration: 00m 44s)
01:24 demon@tin: Synchronized wmf-config/CommonSettings.php: abusefilter stuff (duration: 00m 45s)
01:21 demon@tin: Synchronized wmf-config/InitialiseSettings.php: abusefilter stuff (duration: 00m 45s)
01:19 demon@tin: Synchronized wmf-config/CommonSettings.php: abusefilter stuff (duration: 00m 45s)
01:05 demon@tin: Synchronized docroot/mediawiki/keys/: update formatting, docs, etc (duration: 00m 46s)

2017-11-22

21:15 mutante: running puppet on logstash hosts to apply config change 392897 (add log4j filter)
21:11 demon@tin: Synchronized docroot/noc/conf/index.php: favicon fix (duration: 00m 46s)
20:35 demon@tin: Synchronized dblists/: now with more alphabeticalizedness (duration: 00m 45s)
20:25 demon@tin: Synchronized dblists/Makefile: no-op (duration: 00m 45s)
20:23 demon@tin: Synchronized tests/noc-conf/NOCDblistTest.php: no-op (duration: 00m 45s)
20:19 demon@tin: Synchronized wmf-config/LabsServices.php: no-op (duration: 00m 45s)
20:14 demon@tin: Synchronized wmf-config/InitialiseSettings.php: adding national library of Israel to copy domains (duration: 00m 45s)
20:13 otto@tin: Finished deploy [eventlogging/analytics@57234e7]: no-op: removing now unneeded code that might accidentally serialize userAgent to json string: T179625 (duration: 00m 04s)
20:13 otto@tin: Started deploy [eventlogging/analytics@57234e7]: no-op: removing now unneeded code that might accidentally serialize userAgent to json string: T179625
20:09 demon@tin: Synchronized robots.txt: block a nasty bot 💔 (duration: 00m 44s)
20:02 demon@tin: Synchronized multiversion/bin/expanddblist: fix param warning (duration: 00m 45s)
20:00 mepps: updated civicrm from a16e566 to a1022cf
19:58 demon@tin: Synchronized wmf-config/InitialiseSettings.php: comments (duration: 00m 45s)
19:56 demon@tin: Synchronized multiversion/MWScript.php: assume aawiki for purgeUrls (duration: 00m 45s)
19:51 demon@tin: Synchronized docroot/: removing old foundation docroot (duration: 00m 46s)
19:49 urandom: starting cassandra cleanups, restbase-200{1,3,5}-a - T179422
19:41 demon@tin: Synchronized w/extract2.php: removing old portal support (duration: 00m 45s)
18:47 demon@tin: Synchronized dblists/closed.dblist: closed transitionteamwiki (duration: 00m 45s)
18:34 demon@tin: Synchronized wmf-config/InitialiseSettings-labs.php: no-op (duration: 00m 45s)
18:32 demon@tin: Synchronized scap/plugins/updatewikiversions.py: minor fix (duration: 00m 45s)
18:30 demon@tin: Pruned MediaWiki: 1.31.0-wmf.7 [keeping static files] (duration: 01m 46s)
18:12 herron: re-enabling puppet agents after puppetdb postgres security updates
18:09 moritzm: installing postgres security updates on nitrogen/puppetdb
18:05 moritzm: installing postgres security updates on nihal/puppetdb
18:02 herron: disabling puppet agents for puppetdb postgres security update
17:30 demon@tin: Synchronized php-1.31.0-wmf.8/extensions/AdvancedSearch/: fixing layout issues in timeless (duration: 00m 46s)
16:54 mepps: updated payments-wiki from 1ca91b1 to 6b3019b
16:45 chasemp: disable puppet accross labtest things
16:34 marostegui: Compress s4 on db1097 - T178359
16:28 herron: starting canary deploy/cutover of codfw scb hosts to codfw puppet 4 masters
16:16 elukey: restart druid broker,coordinator,historical daemons on druid100[123] to pick up new logging settings
15:41 jynus: starting manually pt-heartbeat for s8 on db1071
15:22 herron: beginning cut over of codfw db servers (^db2.*) to codfw puppet 4 masters
14:49 jynus@tin: Synchronized wmf-config/db-codfw.php: mariadb: Setup s8 replica set on codfw (duration: 00m 45s)
14:27 moritzm: installing libxml-libxml-perl security updates
14:21 jynus: starting database topology changes for s8 on codfw T177208
14:11 urandom: bootstrapping cassandra, restbase2004-c.codfw.wmnet - T179422
13:43 apergos: one more round of labstore1006 <-- ms1001 rsync catchup
13:37 moritzm: installing imagemagick security updates
12:39 marostegui: Stop MySQL on db1053 to clone db1097.s4 - T178359
12:38 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1053 - T178359 (duration: 00m 45s)
12:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Start adapting the config to move db1097 to s4 and s5 as multi-instance rc slave T178359 (duration: 00m 45s)
12:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1101.s7 - T178359 (duration: 00m 45s)
11:51 jynus: starting dropping incorrectly created database on s7 amwikimedia (not to be confused with production wiki s3 amwikimedia)
11:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1101.s7 - T178359 (duration: 00m 45s)
11:19 akosiaris: gnt-node evacuate -s -f ganeti1005. T181121
11:02 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1101.s7 - T178359 (duration: 00m 45s)
10:54 akosiaris: gnt-node migrate -f ganeti1005. T181121
10:51 marostegui: Drop index from ores_classification on s5 - T180045
10:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1051 and db1063 in vslow service for s5 to warm them up for the s8 split - T177208 (duration: 00m 45s)
10:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1101 to s5 and s7 as recentchanges multi-instance slave - T178359 (duration: 00m 45s)
10:04 moritzm: running "scap pull" on mw1191, it's depooled and marked as "inactive", but health checks are triggering db errors
09:35 bblack: cr[12]-ulsfo - switch static fallback LVS routes from lvs400[12] to lvs400[56]
09:27 bblack: lvs@ulsfo - done switching primaries (host MED config) - lvs400[56] now primary for text/upload traffic
09:11 akosiaris@tin: Finished deploy [parsoid/deploy@b150764]: T180211 (duration: 05m 05s)
09:08 bblack: puppet disabled on lvs400[1256] for switching primaries
09:06 akosiaris@tin: Started deploy [parsoid/deploy@b150764]: T180211
09:04 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp2017.codfw.wmnet
09:00 bblack: lvs4005 - reboot to clear experimental stuff
08:16 bblack: backend restart on cp4024 (upload@ulsfo) - mailbox lag
07:56 marostegui: Drop index from ores_classification on s3 - T180045
07:50 marostegui: Drop index from ores_classification on s6 - T180045
07:48 marostegui: Drop index from ores_classification on s7 - T180045
07:29 _joe_: stopping the additional workers for htmlCacheUpdate (commons and ruwiki), adding one additional runner for refreshLinks on ruwiki
06:43 marostegui: Stop MySQL on db1063 and db1051 (which is going to be recloned) - T177208
06:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Move db1051 from s1 to s5 - T177208 (duration: 00m 53s)
06:28 ejegg: updated payments-wiki from f871160 to 1ca91b1
03:36 mutante: powercycled mw2251 which had gone down without further comment
02:35 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.8) (duration: 11m 34s)
01:09 hoo: Cleaned out remaining T180934 related log blow up on snapshot1007 (dumpwikidatajson-wikidata-20171120-all-0.log)
00:43 mutante: gerrit restarting service to apply config change
00:40 mutante: gerrit - re-enabling puppet to apply logstash change on cobalt, gerrit restart incoming (T141324)
00:39 maxsem@tin: Synchronized php-1.31.0-wmf.8/extensions/CentralNotice/: https://gerrit.wikimedia.org/r/#/c/392754/ (duration: 00m 47s)
00:11 maxsem@tin: Synchronized php-1.31.0-wmf.8/extensions/InputBox/: https://gerrit.wikimedia.org/r/#/c/392745/ (duration: 00m 45s)
00:10 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/392576/ (duration: 00m 46s)
00:07 bblack@neodymium: conftool action : set/pooled=yes; selector: name=nescio.wikimedia.org
00:04 bblack@neodymium: conftool action : set/pooled=no; selector: name=nescio.wikimedia.org
00:02 bblack@neodymium: conftool action : set/pooled=yes; selector: name=maerlant.wikimedia.org
00:01 bblack@neodymium: conftool action : set/pooled=no; selector: name=maerlant.wikimedia.org
00:01 bblack@neodymium: conftool action : set/pooled=yes; selector: name=maerlant.wikimedia.org
00:00 bblack@neodymium: conftool action : set/pooled=yes; selector: name=chromium.wikimedia.org

2017-11-21

23:55 bblack@neodymium: conftool action : set/pooled=no; selector: name=chromium.wikimedia.org
23:30 mholloway-shell@tin: Finished deploy [mobileapps/deploy@fc01242]: Update mobileapps to 52d6a83 (duration: 04m 36s)
23:26 mholloway-shell@tin: Started deploy [mobileapps/deploy@fc01242]: Update mobileapps to 52d6a83
22:45 bblack@neodymium: conftool action : set/pooled=yes; selector: name=achernar.wikimedia.org
22:38 bblack@neodymium: conftool action : set/pooled=no; selector: name=achernar.wikimedia.org
22:36 bblack@neodymium: conftool action : set/pooled=yes; selector: name=hydrogen.wikimedia.org
22:29 urandom: Bootstrapping Cassandra, restbase2004-b.codfw.wmnet (T179422)
22:12 mutante: gerrit - temp disable puppet on cobalt (prod gerrit), test switching gerrit logging to logstash on gerrit2001 - gerrit:392079 gerrit:392083 T141324
22:05 bblack@neodymium: conftool action : set/pooled=no; selector: name=hydrogen.wikimedia.org
22:03 bblack@neodymium: conftool action : set/pooled=yes; selector: name=acamar.wikimedia.org
22:02 bblack: repooling acamar for recdns
21:26 ariel@tin: Finished deploy [dumps/dumps@16f92d6]: take 2: gzip namespace and abstract dumps; remove last configfile existence checks (duration: 00m 02s)
21:26 ariel@tin: Started deploy [dumps/dumps@16f92d6]: take 2: gzip namespace and abstract dumps; remove last configfile existence checks
20:45 bblack: recdns: puppet disabled on all, acamar depooled, careful deploys going on for anycast+recdns stuff
20:42 ayounsi@neodymium: conftool action : set/pooled=no; selector: name=acamar.wikimedia.org
20:26 ariel@tin: Finished deploy [dumps/dumps@16f92d6]: gzip namespace and abstract dumps; remove last configfile existence checks (duration: 00m 16s)
20:26 ariel@tin: Started deploy [dumps/dumps@16f92d6]: gzip namespace and abstract dumps; remove last configfile existence checks
20:22 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to 1.31.0-wmf.8
20:00 thcipriani: finishing wmf.8 rollout, starting group2 to wmf.8
19:21 addshore: SWAT done!
19:20 addshore@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Add images.collection.cooperhewitt.org & *.dimu.org to wgCopyUploadsDomains. T180791 T180241 (duration: 00m 48s)
19:11 addshore@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Enable AdvancedSearch on group0 PT2/2 (duration: 00m 49s)
19:10 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable AdvancedSearch on group0 PT1/2 (duration: 00m 50s)
18:40 mholloway-shell@tin: Finished deploy [mobileapps/deploy@dd41387]: Update mobileapps to 9d1602d (duration: 05m 09s)
18:35 mholloway-shell@tin: Started deploy [mobileapps/deploy@dd41387]: Update mobileapps to 9d1602d
18:28 smalyshev@tin: Finished deploy [wdqs/wdqs@c69c739]: Restore categories vocabulary to V003 (duration: 01m 55s)
18:26 smalyshev@tin: Started deploy [wdqs/wdqs@c69c739]: Restore categories vocabulary to V003
18:25 smalyshev@tin: Finished deploy [wdqs/wdqs@7d951d2]: Restore categories vocabulary to V003 (duration: 00m 11s)
18:25 smalyshev@tin: Started deploy [wdqs/wdqs@7d951d2]: Restore categories vocabulary to V003
16:45 marostegui: Compress s3 on db2085 - T178359
16:35 papaul: powering down wtp2017 for disk replacement
16:19 papaul: updating firmware on db2068
15:31 _joe_: rolling restart of pybal on low-traffic in codfw, eqiad for the new depool thresholds for MW
15:00 ppchelko@tin: Finished deploy [cpjobqueue/deploy@3e948de]: Revert: Temporarily disable deduplication (duration: 00m 26s)
15:00 ppchelko@tin: Started deploy [cpjobqueue/deploy@3e948de]: Revert: Temporarily disable deduplication
14:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore original weight for db1096 (duration: 00m 48s)
14:39 ppchelko@tin: Finished deploy [cpjobqueue/deploy@aac3201]: Temporarily disable deduplication (duration: 00m 29s)
14:39 ppchelko@tin: Started deploy [cpjobqueue/deploy@aac3201]: Temporarily disable deduplication
14:29 zeljkof: EU SWAT finished
14:25 bblack: cr[12]-ulsfo: allow lvs400[567] as PyBal neighbors for BGP
14:18 zfilipin@tin: Synchronized php-1.31.0-wmf.8/extensions/TimedMediaHandler/MwEmbedModules/EmbedPlayer/resources/mw.EmbedPlayerOgvJs.js: SWAT: Disable wasm, use asm.js codec modules for Safari/Edge (T181022) (duration: 00m 49s)
14:12 jdrewniak@tin: Synchronized portals: SWAT: Bumping portals to master (T128546) (duration: 00m 49s)
14:11 jdrewniak@tin: Synchronized portals/prod/wikipedia.org/assets: SWAT: Bumping portals to master (T128546) (duration: 00m 50s)
14:11 ppchelko@tin: Started restart [cpjobqueue/deploy@5341d94]: Restart to try increased maxSockets
14:11 Pchelolo: restart cpjobqueue to try increasing maxSockets
14:08 bblack: not-abnormally-quick reboot on lvs4005
14:08 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: electron stuck - T174916
13:45 ppchelko@tin: Finished deploy [cpjobqueue/deploy@5341d94]: Enable GC metrics reporting (duration: 00m 36s)
13:44 ppchelko@tin: Started deploy [cpjobqueue/deploy@5341d94]: Enable GC metrics reporting
13:39 _joe_: starting 2 manual runners for htmlcacheupdate on commons, 1 for htmlcacheupdate and 1 for refreshlinks on ruwiki, on terbium
13:39 bblack: quick reboot on lvs4005
12:02 kartik@tin: Finished deploy [cxserver/deploy@b87a27a]: Update cxserver to 4301987 (duration: 03m 24s)
11:58 kartik@tin: Started deploy [cxserver/deploy@b87a27a]: Update cxserver to 4301987
11:04 akosiaris: reboot ununpentium for serial tty change
11:03 akosiaris: reboot oresrdb2001 for serial tty change
11:02 akosiaris: reboot netmon1003 for serial tty change
11:01 akosiaris: reboot install2002 for serial tty change
10:57 akosiaris: reboot install1002 for serial tty change
10:35 godog: bootstrap cassandra restbase2004-a - T179422
10:11 ppchelko@tin: Finished deploy [cpjobqueue/deploy@e35aa05]: Set consumer_batch_size to 10 T181007 (duration: 00m 31s)
10:10 ppchelko@tin: Started deploy [cpjobqueue/deploy@e35aa05]: Set consumer_batch_size to 10 T181007
09:39 elukey: upload prometheus-druid-exporter 0.4 to jessie/stretch-wikimedia
09:28 marostegui: Shutdown db2068 for maintenance - T180927
09:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1096 (duration: 00m 49s)
09:02 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1096 (duration: 00m 49s)
08:39 marostegui: Drop index on db1089 enwiki.ores_classification - T180045
08:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1096 with low weight - T178359 (duration: 00m 48s)
08:27 marostegui: Reboot db1096 for kernel and MariaDB upgrade
08:19 marostegui: Compress s5 on db1101 - T178359
08:11 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1109 and db1110 - T180700 (duration: 00m 48s)
07:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1109 and db1110 - T180700 (duration: 00m 49s)
07:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1109 and db1110 in s5 with small weight - T180700 (duration: 00m 48s)
06:50 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1087 - T179106 (duration: 00m 48s)
06:47 marostegui: Remove index wb_terms_language from db1087 - https://phabricator.wikimedia.org/T179106
06:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1087 - T179106 (duration: 00m 48s)
06:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1096 - T178359 (duration: 00m 48s)
06:27 marostegui: Stop MySQL on db1096 to clone db1101.s5 - T178359
06:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1082 original weight - T177208 (duration: 00m 49s)
02:58 mutante: phabricator back up
02:55 mutante: phab1001 (phabricator prod) reboot for kernel upgrade
02:48 krinkle@tin: Synchronized docroot/noc: clean up (duration: 00m 49s)
02:23 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.7) (duration: 05m 42s)
02:03 smalyshev@tin: Started deploy [wdqs/wdqs@7d951d2]: Restore categories vocabulary to V003
00:19 catrope@tin: Synchronized php-1.31.0-wmf.8/resources/src/mediawiki.rcfilters/ui/mw.rcfilters.ui.ItemMenuOptionWidget.js: T180863 (duration: 00m 49s)

2017-11-20

23:46 mutante: phab2001 - reboot for kernel upgrade
23:35 legoktm@tin: Synchronized wmf-config/InitialiseSettings.php: emergency disable ORES on frwp/ruwp T181006 (duration: 00m 49s)
23:35 awight: purge cache keys for ORES thresholds on frwiki and ruwiki
23:25 awight@tin: Started deploy [ores/deploy@82a13ae]: Rollback ORES (take 3); 181006
23:19 awight: aborted ORES rollback
23:18 awight@tin: Started deploy [ores/deploy@95cd523]: Rollback ORES (take 2); 181006
23:14 smalyshev@tin: Finished deploy [wdqs/wdqs@7d951d2]: Rollback categories vocabulary version due to a bug (duration: 08m 11s)
23:11 awight: purged memcache key 'ruwiki:ORES:threshold_statistics:goodfaith:1’, T181006
23:06 smalyshev@tin: Started deploy [wdqs/wdqs@7d951d2]: Rollback categories vocabulary version due to a bug
22:55 awight@tin: Finished deploy [ores/deploy@5084251]: Rollback ORES; T179711 (duration: 01m 05s)
22:55 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: no wmf.8 for group2. i hate my life
22:55 awight: rolling back ORES to fix T181006
22:54 eileen: update civicrm from 272580a to a16e566
22:54 awight@tin: Started deploy [ores/deploy@5084251]: Rollback ORES; T179711
22:50 Krinkle: MW HTTP 500 spike tracked as https://phabricator.wikimedia.org/T181006
22:49 Krinkle: Sharp rise in HTTP 500 errors as of 22:05 (45 minutes ago)
22:29 aaron@tin: Synchronized php-1.31.0-wmf.7/includes/libs/objectcache/WANObjectCache.php: fix cache key namespace (duration: 00m 49s)
22:27 awight@tin: Finished deploy [ores/deploy@5084251]: Updating ORES to revscoring 2.0.10, T179711 (duration: 49m 54s)
22:26 aaron@tin: Synchronized php-1.31.0-wmf.7/extensions/Collection: cache key name fix (duration: 00m 49s)
22:11 aaron@tin: Synchronized php-1.31.0-wmf.8/includes/libs/objectcache/WANObjectCache.php: 7e74b49: namespace WAN cache variant keys (duration: 00m 48s)
22:04 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.8
21:52 bblack: ns2 back on eeden
21:52 aaron@tin: Synchronized php-1.31.0-wmf.8/extensions/Collection: 3baebf4a: cache key name fix (duration: 00m 51s)
21:44 bblack: rebooting eeden
21:37 awight@tin: Started deploy [ores/deploy@5084251]: Updating ORES to revscoring 2.0.10, T179711
21:33 bblack: routing ns2.wikimedia.org to radon for eeden reboot
21:19 bblack: routing ns0.wikimedia.org back to radon post-reboot
21:13 bblack: rebooting radon
21:07 bblack: re-routing ns0.wikimedia.org traffic to baham for radon reboot
20:17 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Remove overlapping userrights (T101983) (duration: 19m 41s)
20:16 bblack: ns1 dns traffic back to normal on baham
20:16 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Revert "Enable Translate extension in amwikimedia (T180879)" (duration: 00m 48s)
20:11 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Enable Translate extension in amwikimedia (T180879) (duration: 00m 49s)
20:11 hashar: CI docker jobs were all broken due to a mistake. Should be back now. T177684
20:07 ladsgroup@tin: Synchronized php-1.31.0-wmf.8/resources/src/mediawiki.rcfilters/ui/mw.rcfilters.ui.ItemMenuOptionWidget.js: RCFilters: Only apply excluded label to namespace items (T180863) (duration: 00m 49s)
20:06 bblack: rebooting baham
19:58 bblack: re-routing ns1.wikimedia.org traffic to radon for baham reboot
19:52 chasemp: updating phab mail handler
19:51 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Remove overlapping userrights (T101983) (duration: 00m 49s)
19:47 ottomata: restarted coal with fixes for eventcapsule changes in T179625
19:42 ladsgroup@tin: Synchronized wmf-config/throttle.php: Adjust throttle.php for dewiki workshop (T180046) (duration: 00m 49s)
19:40 ladsgroup@tin: Synchronized wmf-config/throttle.php: Adjust throttle.php for dewiki workshop (T180046) (duration: 00m 48s)
19:36 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Allow admins to remove users from MP3 uploaders user group (T180002) (duration: 00m 49s)
19:31 ladsgroup@tin: Synchronized portals: SWAT: Bumping portals to master (T128546) (duration: 00m 51s)
19:30 ladsgroup@tin: Synchronized portals/prod/wikipedia.org/assets: SWAT: Bumping portals to master (T128546) (duration: 00m 49s)
19:22 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Switch submit button from 'save' to 'publish' on dewiki (duration: 00m 50s)
18:51 chasemp: disable puppet across cloud things to rollout https://gerrit.wikimedia.org/r/#/c/392168/ slowly
18:25 smalyshev@tin: Finished deploy [wdqs/wdqs@2e39b69]: Blazegraph and GUI update (duration: 01m 42s)
18:23 smalyshev@tin: Started deploy [wdqs/wdqs@2e39b69]: Blazegraph and GUI update
17:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1082 with low weight - T177208 (duration: 00m 49s)
17:19 bd808: Sending Toolforge survey emails from silver for T177126
17:00 moritzm: uploaded git 2.11-3+deb9u2+bpo8+wmf1 for component/git to apt.wikimedia.org/jessie-wikimedia
16:43 marostegui: Reboot db1082 for kernel upgrade and MariaDB upgrade to 10.0.33
16:39 ppchelko@tin: Finished deploy [cpjobqueue/deploy@174420f]: Revert: Temporary set consumer_batch_size to 50, forgot -f, checkout prev rev (duration: 03m 33s)
16:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1109 and db1110 to the config depooled - T180700 (duration: 00m 48s)
16:38 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1109 and db1110 to the config depooled - T180700 (duration: 00m 48s)
16:36 ppchelko@tin: Started deploy [cpjobqueue/deploy@174420f]: Revert: Temporary set consumer_batch_size to 50, forgot -f, checkout prev rev
16:32 ppchelko@tin: Finished deploy [cpjobqueue/deploy@f0610d3]: Revert: Temporary set consumer_batch_size to 50, forgot -f (duration: 00m 28s)
16:32 ppchelko@tin: Started deploy [cpjobqueue/deploy@f0610d3]: Revert: Temporary set consumer_batch_size to 50, forgot -f
16:24 marostegui: Add db1109 and db1110 to tendril - T180700
16:06 ppchelko@tin: Finished deploy [cpjobqueue/deploy@f0610d3]: Revert: Temporary set consumer_batch_size to 50 (duration: 00m 17s)
16:06 ppchelko@tin: Started deploy [cpjobqueue/deploy@f0610d3]: Revert: Temporary set consumer_batch_size to 50
15:59 urandom: Use lz4 compression instead of deflate (T180804)
15:45 jynus: shutting down db2068 for maintenance after depool T180927
15:42 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2068 (duration: 00m 49s)
15:34 chasemp: disable puppet for a merge across cloud things
15:31 ppchelko@tin: Finished deploy [cpjobqueue/deploy@f0610d3]: Temporary set consumer_batch_size to 50 (duration: 00m 30s)
15:30 ppchelko@tin: Started deploy [cpjobqueue/deploy@f0610d3]: Temporary set consumer_batch_size to 50
15:03 herron: pointing codfw mw servers at codfw puppet 4 masters via puppetmaster2001
14:39 zeljkof: EU SWAT finished
14:34 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Single edit tab in Catalan Wikipedia (T180660) (duration: 00m 49s)
14:25 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable wgNamespacesWithSubpages for hiwikiversity (T180913) (duration: 00m 48s)
14:22 jynus: enable semi-sync replication on s5
14:14 dcausse@tin: Synchronized wmf-config/CirrusSearch-common.php: T180805: Revert [cirrus] disable token count router (duration: 00m 49s)
14:05 elukey: upload prometheus-druid-exporter 0.3 to jessie-wikimedia
13:53 Pchelolo: disable puppet on scb100x and stop cpjobqueue to accumulate some backlog
13:39 ppchelko@tin: Finished deploy [cpjobqueue/deploy@174420f]: Optimise committed offset calculation (duration: 00m 28s)
13:39 ppchelko@tin: Started deploy [cpjobqueue/deploy@174420f]: Optimise committed offset calculation
13:30 elukey: upload prometheus-druid-exporter 0.3 to stretch-wikimedia
12:40 marostegui: Optimize db1063 wikidatawiki.wb_terms
12:17 moritzm: uploaded hhvm 3.18.5+dfsg-1+wmf1+icu57 to apt.wikimedia.org (jessie-wikimedia/component/icu57) (HHVM build linked against a co-installable backport of icu57)
11:46 moritzm: rebooting tungsten for update to 4.9.51
11:41 moritzm: rebooting etherpad1001 (etherpad.wikimedia.org) for update to 4.9.51
11:36 moritzm: rebooting pybal-test for update to 4.9.51
11:27 moritzm: rebooting restbase-test cluster for update to 4.9.51
11:23 ppchelko@tin: Finished deploy [cpjobqueue/deploy@bdcef23]: Various performance improvements in committing (duration: 00m 30s)
11:22 ppchelko@tin: Started deploy [cpjobqueue/deploy@bdcef23]: Various performance improvements in committing
11:13 moritzm: rebooting ruthenium for update to 4.9.51
10:57 godog: reimage restbase2004 - T179422
10:51 moritzm: rebooting cerium/praseodymium/xenon for update to 4.9.51
10:42 moritzm: rebooting mwlog1001 for update to 4.9.51
10:39 moritzm: rebooting mwlog2001 for update to 4.9.51
10:23 hoo: Manually re-started the Wikidata entity JSON dump on snapshot1007 (T180934)
10:19 dcausse: elastic/cirrus: reindexing english group0 and group1 wikis: T179945
10:08 hashar: contint1001: sudo systemctl start jenkins
10:05 moritzm: rebooting contint1001 for update to 4.9.51
09:22 moritzm: rebooting hafnium for update to 4.9.51
08:46 marostegui: Run mydumper for db1047.staging - T156844
07:58 moritzm: installing procmail security updates
07:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1097 - T178359 (duration: 00m 49s)
06:41 marostegui: Stop MySQL on db1082 to clone db1109 and db1110 - T180700
06:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1082 - T177208 (duration: 00m 48s)
06:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore original weights for db1100 and db1071 - T180917 (duration: 00m 49s)
06:15 marostegui: Reboot db2068 - T180927
02:28 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.7) (duration: 06m 26s)

2017-11-19

23:20 Jamesofur: removed 2FA for Ask21 T180889
20:28 krinkle@tin: Synchronized docroot/noc/index.html: noc: Link to Grafana (duration: 00m 49s)
19:50 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1100 (duration: 00m 49s)
07:09 mobrovac@tin: Started restart [zotero/translators@a0c41c3]: Zotero eating up memory
07:01 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: electron stuck - T174916

2017-11-18

20:06 gehel: upgrade of elasticsearch eqiad complete - T178411
18:03 reedy@tin: Synchronized multiversion/vendor/: bump (duration: 01m 14s)
15:53 marostegui: Compress s5 on db2085 - T178359
14:48 marostegui: Compress s3 on db2092 - T178359
00:30 urandom: Bootstrapping restbase2006-b (T179422)

2017-11-17

20:35 mutante: netmon1002 - rsync smokeping data back from local backup to show measurements made from eqiad as requested on T180812
20:05 mutante: running puppet on all cache misc to switch smokeping web to eqiad
20:02 mutante: T180812 copying smokeping data from 2001 to 1002 - netmon1002: /usr/bin/rsync -avp rsync://netmon2001.wikimedia.org/var-lib-smokeping /var/lib/smokeping/ | switching backend from codfw to eqiad
19:37 mutante: T180812 - @netmon1002:# rsync -avp /var/lib/smokeping/ /root/backup/netmon1002/201711717/var/lib/smokeping/@netmon2001:/# rsync -avp /var/lib/smokeping/ /root/backup/netmon2001/201711717/var/lib/smokeping/
19:36 mutante: @netmon1002:/var/lib/smokeping# rsync -avp /var/lib/smokeping/ /root/backup/netmon1002/201711717/var/lib/smokeping/
18:37 anomie@tin: Synchronized wmf-config/InitialiseSettings.php: Setting wgCommentTableSchemaMigrationStage = MIGRATION_WRITE_BOTH on Beta Cluster, no prod change (duration: 00m 48s)
18:36 anomie@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Setting wgCommentTableSchemaMigrationStage = MIGRATION_WRITE_BOTH on Beta Cluster, no prod change (duration: 00m 49s)
17:43 demon@tin: Synchronized php-1.31.0-wmf.8/extensions/Wikidata/extensions/Wikibase/client/WikibaseClient.php: fix client dependencies (duration: 00m 49s)
17:41 demon@tin: Synchronized php-1.31.0-wmf.8/extensions/Wikibase/client/WikibaseClient.php: fix client dependencies (duration: 00m 50s)
17:21 bblack: enabling new normalization code for all upload@esams (done)
17:19 bblack: enabling new normalization code for all upload@eqiad
17:14 marostegui: Revert schema change on dbstore1001 - T180714
17:12 marostegui: Move dbstore1001 under db1070 - T180714
17:12 bblack: enabling new normalization code for all upload@codfw
16:52 bblack: enabling new normalization code for all upload@ulsfo
16:24 bblack: disabling puppet on all cp* (testing encoding patch)
16:19 moritzm: uploaded boost 1.55.0+dfsg-3+wmf2+icu57 to apt.wikimedia.org for jessie-wikimedia/component/icu57 (needed for HHVM build linked against ICU 57)
16:04 dcausse@tin: Synchronized wmf-config/CirrusSearch-common.php: T180795 [cirrus] disable token count router (duration: 00m 49s)
15:43 chasemp: labservices1001:~# mv /var/zones/tools.eqiad.wmflabs /home/rush T180797
15:26 urandom: Starting restbase2006-c w/ -Dcassandra.replace_address=10.192.48.51 (T179422)
14:06 jynus: deploy master events to db1070
14:04 chasemp: labstore1003 service nfs-kernel-server restart && service rsync start
13:55 moritzm: uploaded boost 1.55.0+dfsg-3+wmf1+icu57 to apt.wikimedia.org for jessie-wikimedia/component/icu57 (needed for HHVM build linked against ICU 57)
12:15 moritzm: installing openssl updates on poolcounters
12:07 moritzm: installing openssl updates on graphite* hosts
11:46 moritzm: installing openssl updates on etcd* hosts
11:43 moritzm: installing openssl updates on dbproxy hosts
11:33 moritzm: installing openssl updates on puppetmasters
11:01 akosiaris: create webperf1001, webperf2001 in ganeti T179036
10:50 akosiaris@tin: Synchronized wmf-config/db-eqiad.php: (no justification provided) (duration: 00m 49s)
10:49 akosiaris: sync wmf-config/db-eqiad.php for T180724
10:48 akosiaris@tin: scap aborted: (no justification provided) (duration: 00m 02s)
10:48 akosiaris@tin: Started scap: (no justification provided)
10:47 akosiaris: pool mw2251 T180724
10:47 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw2251.codfw.wmnet
10:01 moritzm: rebooting darmstadtium (docker registry) for update to 4.9.51
09:59 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore original traffic for the s5 eqiad hosts that were out for hours due to schema change revert after s5 master crash (duration: 00m 48s)
09:54 moritzm: rebooting labnodepool* for update to 4.9.51
09:38 ppchelko@tin: Finished deploy [cpjobqueue/deploy@2141162]: Bump concurrency even more (duration: 00m 29s)
09:38 ppchelko@tin: Started deploy [cpjobqueue/deploy@2141162]: Bump concurrency even more
09:20 godog: start restbase2006-c instead, restbase2006-b failed and -c shows as "down" - T179422
09:13 jynus: changing master of db1071 (63 -> 70)
09:10 gehel: cleanup leftover titlesuggest indices on elasticsearch eqiad (jawiki, frwiki, ptwiki)
09:05 ppchelko@tin: Finished deploy [cpjobqueue/deploy@cbd25d3]: Bump overall concurrency to get rid of RecordLintJob backlog (duration: 00m 35s)
09:04 ppchelko@tin: Started deploy [cpjobqueue/deploy@cbd25d3]: Bump overall concurrency to get rid of RecordLintJob backlog
08:54 godog: bootstrap restbase2006-b - T179422
08:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for the s5 eqiad hosts that were out for hours due to schema change revert after s5 master crash (duration: 00m 50s)
08:04 elukey: reboot stat100[456] for kernel updates
07:51 marostegui: Revert schema change on dbstore1002 - T180714
07:50 marostegui: Move dbstore1002 under db1070 - T180714
07:33 marostegui: Enable GTID on all the eqiad up-to-date hosts, only pending db1071 - T180714
07:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool s5 eqiad hosts after reverting schema change - T180714 (duration: 00m 50s)
06:34 marostegui: Revert schema changes on s5 codfw master with replication enabled, lag will be generated on codfw s5 - T180714
04:43 krinkle@tin: Synchronized wmf-config/CommonSettings-labs.php: no-op for labs (duration: 00m 51s)
01:32 ejegg: re-enabled donation queue consumer
01:28 ejegg: updated CiviCRM from 8454e06 to 272580a
01:07 ejegg: updated CiviCRM from 0b8ceea to 8454e06
food: disabled donations queue consumer for thank you subject update
00:41 krinkle@tin: Synchronized wmf-config/PhpAutoPrepend.php: I60cce0 (duration: 00m 48s)
00:40 krinkle@tin: Synchronized wmf-config/StartProfiler.php: I60cce0 (duration: 00m 49s)
00:37 krinkle@tin: Synchronized wmf-config/profiler.php: I60cce0 (duration: 00m 48s)
00:23 urandom: Decommissioning Cassandra, restbase1014-c.eqiad.wmnet (T179422)
00:22 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/390131/3 (duration: 00m 49s)

2017-11-16

23:50 mutante: wezen - systemctl restart rsyslog
21:52 mepps: updated civicrm from 22f6532 to 0b8ceea
21:05 mepps: updated payments-wiki from 210cb37 to f871160
20:51 apergos: one more catchup rsync from ms1001 to labstore1006 kicking off
20:15 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool only 'good' servers - third try (duration: 00m 49s)
20:11 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool only 'good' servers - second try (duration: 00m 49s)
20:10 demon@tin: Unlocked for deployment [operations/mediawiki-config]: No deploys, recovering from downtime (more narrow locking) (duration: 06m 49s)
20:03 demon@tin: Locking from deployment [operations/mediawiki-config]: No deploys, recovering from downtime (more narrow locking) (planned duration: 360m 00s)
20:02 twentyafterfour: deploy a9613b4 to hotfix T180706
19:56 demon@tin: Unlocked for deployment [ALL REPOSITORIES]: No deploys, recovering from downtime (duration: 50m 28s)
19:54 ayounsi@tin: Finished deploy [netbox/deploy@19f4f65]: (no justification provided) (duration: 00m 04s)
19:54 ayounsi@tin: Started deploy [netbox/deploy@19f4f65]: (no justification provided)
19:49 akosiaris: schedule extra downtime for s5 slaves
19:46 urandom: Bootstapping Cassandra restbase2006-a (T179422)
19:13 jynus: reset slave all on db1070 @ db1063-bin.001382:234464548
19:05 demon@tin: Locking from deployment [ALL REPOSITORIES]: No deploys, recovering from downtime (planned duration: 360m 00s)
19:05 demon@tin: Unlocked for deployment [ALL REPOSITORIES]: Dealing with outage, no deploys for now (duration: 01m 22s)
19:03 demon@tin: Locking from deployment [ALL REPOSITORIES]: Dealing with outage, no deploys for now (planned duration: 60m 00s)
18:48 marostegui: dewikipedia and wikidata currently back to writable
18:44 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool only 'good' servers (duration: 00m 48s)
18:41 mutante: dewikipedia and wikidata currently back to readonly-mode while wikidata is being worked on
18:41 marostegui: Set s5 master read_only
18:25 akosiaris: set mw2251 to inactive. T180724
18:24 akosiaris@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=mw2251.codfw.wmnet
17:57 gehel: silencing wdqs
17:55 jynus@tin: Synchronized wmf-config/db-eqiad.php: Failover db1063 to db1070 (duration: 00m 46s)
17:44 XioNoX: merging netbox CR
17:41 bblack: disable port ge-5/0/39 on asw-c-eqiad (db1063)
17:40 urandom: Decommissioning Cassandra, restbase1014-b.eqiad.wmnet (T179422)
17:36 jynus: stopping slave on db1070
17:28 urandom: Converting 'enwiki parsoid' to size-tiered compaction (T179422)
17:19 demon@tin: Finished scap: consistency (duration: 25m 12s)
17:18 bblack: Temporary GeoDNS routing changes (eqsin traffic simulation using ulsfo) - https://gerrit.wikimedia.org/r/#/c/391357/ - expecting ~24h, West Asia latencies will probably increase, spike in cache misses, etc...
17:18 urandom: Converting 'wikipedia parsoid' to size-tiered compaction (T179422)
17:16 godog: reimage restbase2006 - T179422
17:15 urandom: Converting 'commons mobile' to size-tiered compaction (T179422)
17:05 urandom: Converting 'others mobile' to size-tiered compaction (T179422)
16:54 demon@tin: Started scap: consistency
16:54 godog: reimage restbase2006 - T179422
16:48 jynus: stop and restart db1071 for upgrade and reconfiguration
16:48 herron: beginning gradual cutover of codfw mw systems to puppet 4 master puppetmaster2001
16:48 godog: upgrade hpsa firmware to 6.06 on restbase2006 - T141756
16:43 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1034 and db1079 original weight (duration: 00m 48s)
16:29 jynus@tin: Synchronized wmf-config/db-eqiad.php: mariadb: Depool db1071, pool db1104 as api (duration: 00m 49s)
16:21 moritzm: restarting apache on labs puppet masters to pick up openssl updates
16:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1034 weight (duration: 00m 48s)
15:47 moritzm: rebooting ores1* for update to 4.9.51
15:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1034 weight (duration: 00m 49s)
15:33 moritzm: rebooting seaborgium (slapd) for update to 4.9.51
15:22 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2038 - T178359 (duration: 00m 49s)
15:16 milimetric@tin: Finished deploy [analytics/refinery@4ef15d3]: Mainly deploying the interlanguage navigation dataset (duration: 14m 29s)
15:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1034 weight (duration: 00m 48s)
15:10 godog: upgrade prometheus-redis-exporter to 0.13-1 - T148637
15:06 moritzm: installing postgres security updates on labsdb1006/1007
15:02 milimetric@tin: Started deploy [analytics/refinery@4ef15d3]: Mainly deploying the interlanguage navigation dataset
15:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1051 - T177208 (duration: 00m 48s)
15:00 herron: beginning cut over of codfw canary appservers to puppet 4 master puppetmaster2001
14:55 moritzm: rebooting serpens (slapd) for update to 4.9.51
14:50 elukey: updating puppet compiler's facts (following https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource:Puppet3-diffs#FAQ)
14:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1034 weight - T178359 (duration: 00m 49s)
14:26 dcausse: EU swat done
14:23 moritzm: rebooting kubetcd* to 4.9.51
14:18 dcausse@tin: Synchronized wmf-config/CirrusSearch-common.php: T177913: [cirrus] Add overridden iw prefix for svwiki (2/2) (duration: 00m 48s)
14:16 dcausse@tin: Synchronized wmf-config/InitialiseSettings.php: T177913: [cirrus] Add overridden iw prefix for svwiki (1/2) (duration: 00m 50s)
14:02 gehel: starting upgrade of elasticsearch eqiad - T178411
13:46 moritzm: rebooting bohrium (piwik host) for update to 4.9.51
13:33 moritzm: installing openssl updates on es* hosts
13:24 moritzm: rebooting dubnium/pollux (openldap corp mirror) for update to 4.9.51
13:12 moritzm: installing openssl updates on pc* hosts
13:07 elukey: restart aqs on aqs100[5-9] to apply localQuorum (https://gerrit.wikimedia.org/r/391765) - T164348
13:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1034 weight - T178359 (duration: 00m 49s)
12:54 moritzm: rebooting radium (tor relay) for update to 4.9.51
12:34 moritzm: installing openssl updates on memcached/redis clusters
12:27 hoo: Updated operations/dumps/dcat (ea4e75..7734e04) on snapshot1007
12:09 moritzm: rebooting dbmonitor* hosts for update to 4.9.51
12:03 addshore@tin: Synchronized php-1.31.0-wmf.8/extensions/Wikidata/extensions/Constraints: T180665 Manually applied: Fix SparqlHelper::getCacheMaxAge() (duration: 00m 52s)
11:58 addshore@tin: Synchronized php-1.31.0-wmf.8/extensions/WikibaseQualityConstraints: T180665 Fix SparqlHelper::getCacheMaxAge() (duration: 00m 53s)
11:42 moritzm: installing openssl updates on conf* clusters
11:37 addshore@tin: Synchronized wmf-config/Wikibase-production.php: testwikidata only, T180665, WBQC configuration for testwikidatawiki (duration: 00m 49s)
11:32 addshore: addshore@terbium:~$ mwscript extensions/WikibaseQualityConstraints/maintenance/ImportConstraintStatements.php --wiki=testwikidatawiki > importConstraintStatements.log
11:21 godog: upgrade prometheus to 1.8.1+ds+k8s-1 in ulsfo/esams/eqiad - T177395
11:09 moritzm: rebooting debug proxies (hassium/hassaleh) for update to 4.9.51
11:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1034 weight - T178359 (duration: 00m 48s)
11:01 jynus: shutting down labsdb1010 to clone to labsdb1009
10:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1034 with low weight - T178359 (duration: 00m 48s)
10:26 addshore: Kill the build deploy slot done!
10:24 addshore@tin: Synchronized php-1.31.0-wmf.8/extensions/Wikibase: T180634 Bring Wikibase up to date with .8 branch of Wikidata build extension (duration: 01m 46s)
10:15 addshore@tin: Synchronized php-1.31.0-wmf.8/extensions/WikibaseQualityConstraints: T180634 Bring WikibaseQualityConstraints up to date with .8 branch of Wikidata build extension (duration: 00m 53s)
09:59 moritzm: rebooting prometheus servers in eqiad for update to 4.9.51
09:44 elukey: restart aqs on aqs1004 to apply localQuorum (https://gerrit.wikimedia.org/r/391765) - T164348
09:38 moritzm: rebooting prometheus servers in codfw for update to 4.9.51
09:30 moritzm: uploaded icu57.1-6+wmf2 to jessie-wikimedia/component/icu57
09:23 godog: bootstrap restbase2002-c - T179422
09:19 godog: upgrade grafana to 4.6.1 on https://grafana.wikimedia.org/ - T180428
08:19 marostegui: Deploy schema change on db1092 - T174569
08:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1092 - T174569 (duration: 00m 49s)
07:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1106 (duration: 00m 49s)
07:40 marostegui: Stop MySQL on db1034 to copy its content to db1101.s7 - T178359
07:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1034 - T178359 (duration: 00m 49s)
07:17 ppchelko@tin: Finished deploy [eventlogging/eventbus@872cfb3]: Revert gerrit 302372 due to AssertionError T180017 (duration: 00m 14s)
07:17 ppchelko@tin: Started deploy [eventlogging/eventbus@872cfb3]: Revert gerrit 302372 due to AssertionError T180017
06:54 marostegui: Deploy alter table on db1096 - T174569
06:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1096 - T174569 (duration: 00m 48s)
06:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1099 and db1071 - T174569 (duration: 00m 49s)
06:15 smalyshev@tin: Finished deploy [wdqs/wdqs@b44cf27]: data reload/T176593 (duration: 00m 28s)
06:14 smalyshev@tin: Started deploy [wdqs/wdqs@b44cf27]: data reload/T176593
02:26 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.7) (duration: 07m 54s)
02:09 twentyafterfour: phabricator database migrations complete, service is back online
01:28 twentyafterfour: Phabricator will be offline for a couple of minutes while I apply database migrations.
01:22 twentyafterfour: updating phabricator (belatedly)
01:05 demon@tin: Synchronized scap/plugins/prep.py: no-op, co-master sync (duration: 00m 55s)
00:59 tzatziki: Removing 2FA from AuburnPilot (T180654)
00:59 tzatziki: Removing 2FA from Twotwo2019 (T180438)
00:42 eileen: update civicrm from b99a9cf to 22f6532

2017-11-15

22:48 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: wikidatawiki back to wmf.7
22:21 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.8
22:11 urandom: Setting keyspaces erroneously configured for leveled compaction, to use size-tiered (T180568)
22:05 madhuvishy: Kicking off re-enabling puppet and puppet runs across Cloud VPS instances
21:33 demon@tin: Synchronized php: symlink swap (duration: 00m 49s)
21:21 madhuvishy: Disabling puppet across cloud VPS through cumin on labpuppetmaster1001
21:01 ottomata: restarting kafka-jumbo brokers to update to Kafka 0.11.0.1
20:52 urandom: Restarting restbase2002-a.codfw.wmnet (T180568)
20:40 mutante: restarting gerrit to enable 'large file support'-plugin gerrit:391635
20:38 addshore@tin: Finished scap: Update extensions/Wikidata to new wmf/1.31.0-wmf.8 branch (again) T180539 (duration: 21m 48s)
20:17 addshore@tin: Started scap: Update extensions/Wikidata to new wmf/1.31.0-wmf.8 branch (again) T180539
20:11 otto@tin: Finished deploy [eventlogging/eventbus@872cfb3]: deploying kafka-futures change to kafka1001 only, will apply async with https://gerrit.wikimedia.org/r/#/c/391634/ (duration: 00m 20s)
20:11 otto@tin: Started deploy [eventlogging/eventbus@872cfb3]: deploying kafka-futures change to kafka1001 only, will apply async with https://gerrit.wikimedia.org/r/#/c/391634/
19:53 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Add BP and WP aliases for project namespace on mwlwiki (T180052) (duration: 00m 48s)
19:48 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable SandboxLink on mwlwiki (T180052) (duration: 00m 48s)
19:41 demon@tin: Finished deploy [gerrit/gerrit@bce982f]: adding lfs plugin @ 2.13.9 (duration: 00m 08s)
19:41 demon@tin: Started deploy [gerrit/gerrit@bce982f]: adding lfs plugin @ 2.13.9
19:40 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Disable EventLogging for Popups (T178500) (duration: 00m 49s)
19:37 gehel: upgrade of elasticsearch codfw completed, cluster still recovering - T178411
19:34 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable Minerva download icon on all wikis (T179914) (duration: 00m 48s)
19:30 chasemp: labstore1004:~# service nfs-kernel-server restart
19:19 catrope@tin: Synchronized php-1.31.0-wmf.8/resources/src/mediawiki.rcfilters/mw.rcfilters.UriProcessor.js: T180577 (duration: 00m 49s)
19:16 catrope@tin: Synchronized php-1.31.0-wmf.7/skins/MinervaNeue/resources/skins.minerva.scripts/init.js: Limit download button to Google Chrome (T179529, T179914) (duration: 00m 49s)
19:14 catrope@tin: Synchronized wmf-config/flaggedrevs.php: Enable RCFilters on all remaining wikis (T177445) (duration: 00m 49s)
19:01 urandom: Restarting Cassandra instances on restbase2001.codfw.wmnet (T180568)
18:38 Jamesofur: removed 2FA from User:Einsbor after verification for votewiki, stewardwiki and SUL
18:20 moritzm: rebooting auth* servers for update to 4.9.51
18:20 moritzm: rebooting auth
17:57 demon@tin: Synchronized docroot/search.wikimedia.org/index.php: removing support for non-wikipedias (duration: 00m 49s)
16:03 addshore@tin: Finished scap: Revert partial scap: Update extensions/Wikidata to new wmf/1.31.0-wmf.8 branch, second try (T180539) (duration: 23m 48s)
15:56 oblivian@tin: Finished deploy [docker-pkg/deploy@9b319d2]: Adding do extension to jinja2, needed for contint images (duration: 00m 18s)
15:56 oblivian@tin: Started deploy [docker-pkg/deploy@9b319d2]: Adding do extension to jinja2, needed for contint images
15:55 ema: restart pybal on lvs[12]00[36] for config change https://gerrit.wikimedia.org/r/#/c/389964/
15:43 jynus: shutting down labsdb1009 for maintenance T179244
15:40 addshore@tin: Started scap: Revert partial scap: Update extensions/Wikidata to new wmf/1.31.0-wmf.8 branch, second try (T180539)
15:29 moritzm: installing perl security updates on trusty (Debian hosts fixed two months ago)
15:14 ladsgroup@tin: scap aborted: Update extensions/Wikidata to new wmf/1.31.0-wmf.8 branch, second try (T180539) (duration: 03m 48s)
15:13 moritzm: removing unused kernels from prometheus*
15:11 ladsgroup@tin: Started scap: Update extensions/Wikidata to new wmf/1.31.0-wmf.8 branch, second try (T180539)
15:05 moritzm: rebooting kubestagetcd* for update to 4.9.51
14:57 ladsgroup@tin: scap aborted: Update extensions/Wikidata to new wmf/1.31.0-wmf.8 branch (T180539) (duration: 00m 31s)
14:57 ladsgroup@tin: Started scap: Update extensions/Wikidata to new wmf/1.31.0-wmf.8 branch (T180539)
14:42 Amir1: deployed ORES change for wikidata (gerrit:391197, phab:T180450)
14:41 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 49s)
14:37 mobrovac: restbase creating Cassandra 3 revision tables on restbase1009 - T179421
14:30 zfilipin@tin: Synchronized wmf-config/: SWAT: Remove wgContentTranslationEnableSuggestions (duration: 00m 51s)
14:27 ema: cache_upload: upgrade varnish to 4.1.8-1wm2
14:22 zfilipin@tin: Synchronized wmf-config/CommonSettings-labs.php: SWAT: Beta: Explicitly set cookieDomain for ContentTranslationSiteTemplates (T149879) (duration: 00m 49s)
14:18 ema: repool cp4024 T174891
14:17 mutante: wtp2017 - systemctl start ferm (ferm wasnt running due to failed DNS lookup for prometheus2003 sometime in the past)
14:12 zfilipin@tin: Synchronized php-1.31.0-wmf.8/extensions/EventBus/EventBus.php: SWAT: Increase request timeout to match kafka produce timeout (T180017) (duration: 00m 50s)
14:11 godog: upgrade hpsa firmware to 6.06 on restbase2004 - T180562 T141756
14:10 ema: upgrade varnish to 4.1.8-1wm2 on cp4024 (cache_upload, depooled)
14:04 jynus: reload haproxy on dbproxy1010
14:00 moritzm: installing openssl updates on kafka and hadoop clusters
13:34 ema: cache_text: upgrade varnish to 4.1.8-1wm2
13:31 ppchelko@tin: Started restart [changeprop/deploy@065a06e]: Restart to rebalance all rules T179684
13:14 moritzm: rebooting video scalers in eqiad for update to 4.9.51 (and to pick up openssl update)
13:07 ema: upgrade varnish to 4.1.8-1wm2 on cp3030 (cache_text)
12:41 elukey: re-enable eventlogging after maintenance
12:24 jynus: deploying dns change for m4-master
12:19 ema: cache_misc: upgrade varnish to 5.1.3-1wm3
12:09 elukey: executed sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=65 on all jobrunners
12:01 ema: upgrade varnish to 5.1.3-1wm3 on cp3007 (cache_misc)
11:58 ema: varnish 4.1.8-1wm2 uploaded to apt.w.o (main)
11:50 ema: varnish 5.1.3-1wm3 uploaded to apt.w.o (experimental)
11:45 jynus: restart haproxy on dbproxy1009
11:32 marostegui: Stop MySQL on db1046
10:58 moritzm: rebooting job runners in eqiad for update to 4.9.51 (and to pick up openssl update)
10:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1106, optimizing wikidatawiki.wb_terms (duration: 00m 49s)
10:09 ppchelko@tin: Finished deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x attempt 4 T179786 (duration: 04m 51s)
10:04 ppchelko@tin: Started deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x attempt 4 T179786
10:04 ppchelko@tin: Finished deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x attempt 3, force T179786 (duration: 04m 44s)
09:59 ppchelko@tin: Started deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x attempt 3, force T179786
09:59 ppchelko@tin: Finished deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x attempt 2 T179786 (duration: 03m 32s)
09:58 ppchelko@tin: (no justification provided)
09:55 ppchelko@tin: Started deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x attempt 2 T179786
09:55 ppchelko@tin: Finished deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x T179786 (duration: 02m 44s)
09:53 godog: reboot restbase2004 - T180562
09:52 ppchelko@tin: Started deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x T179786
09:52 mobrovac@tin: Started restart [restbase/deploy@c76a665]: Pick up the new seeds definition - T179422
09:52 ppchelko@tin: Started deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x T179786
09:32 godog: restart cassandra on restbase2002-b - T180568
09:30 moritzm: updating openssl on database hosts
09:19 godog: restart cassandra on restbase2001-c - T180568
09:15 marostegui: Stop mysql on db1046 to transfer its content to db1107 - T177405
09:08 elukey: stop eventlogging on eventlog1001, eventlogging replication on db1108/db1047/dbstore1002 as preparation steps to migrate the log db from db1046 to db1107
09:02 jynus: rebooting labsdb1010 for kernel upgrade
08:51 elukey: reboot thorium (hosting all analytics websites) for kernel updates
08:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1101 - going to convert it to mult-instance T178359 (duration: 00m 48s)
07:49 marostegui: Deploy schema change on db1071 and db1099 (s5) - T174569
07:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1099 and db1071 - T174569 (duration: 00m 49s)
07:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1106 and db1100 - T174569 (duration: 00m 49s)
05:33 subbu: subbu@terbium (followup to the linter-reparse.py script log entry) it is safe to kill -9 the script on terbium anytime if there are any problems because of it
05:27 subbu: subbu@terbium running linter-reparse.py script to initialize baseline linter categories for all wikis (12 hours in so far .. expected to run for ~2 weeks). hits parsoid eqiad cluster.
04:06 demon@tin: Synchronized docroot/search.wikimedia.org/robots.txt: go away robots / kill some 404s (duration: 00m 50s)
02:29 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.7) (duration: 05m 57s)
00:59 ejegg: updated payments-wiki from d150287 to 210cb37
00:53 Reedy: going to restart zuul as it's got backed up
00:33 maxsem@tin: Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/391220/ (duration: 00m 49s)
00:18 ejegg: updated SmashPig payments listener from 5262d53 to 45aa626
00:12 ejegg: updated CiviCRM from f571c67 to b99a9cf

2017-11-14

23:29 mutante: releases1001 - chmod g+w /srv/org/wikimedia/releases/mediawiki/1.*
23:01 urandom: Decommissioning Cassandra, restbase1014-a.eqiad.wmnet (T179422)
22:57 demon@tin: Synchronized w/: Removing mobilelanding from global w/, few sites actually need it (duration: 00m 49s)
22:55 demon@tin: Synchronized docroot/m.wikipedia.org/w/mobilelanding.php: symlink -> real file (duration: 00m 50s)
21:42 no_justification: gerrit: restarted services on master/cobalt, things will flap for a second
21:38 mutante: gerrit2001 - restarted gerrit
21:17 ebernhardson@tin: Finished deploy [search/mjolnir/deploy@494e0c6]: redeploy mjolnir submodule bump to master (duration: 02m 11s)
21:15 ebernhardson@tin: Started deploy [search/mjolnir/deploy@494e0c6]: redeploy mjolnir submodule bump to master
21:03 ebernhardson@tin: Finished deploy [search/mjolnir/deploy@77310bc]: update mjolnir to master (duration: 02m 05s)
21:01 ebernhardson@tin: Started deploy [search/mjolnir/deploy@77310bc]: update mjolnir to master
20:37 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.8
20:15 bblack: reboot cp4024
19:58 ebernhardson@tin: Finished deploy [search/mjolnir/deploy@b20f0da]: test mjolnir deployment (duration: 02m 12s)
19:57 urandom: Bootstrapping restbase2002-b.codfw.wmnet (T179422)
19:56 ebernhardson@tin: Started deploy [search/mjolnir/deploy@b20f0da]: test mjolnir deployment
19:47 demon@tin: Finished scap: bootstrap wmf.8 (duration: 44m 37s)
19:32 chasemp: clean up instances in error state in testlabs project
19:10 jynus: restart db2034 for mariadb upgrade
19:09 chasemp: for i in `OS_TENANT_NAME=testlabs openstack server list | grep stress | awk '{print $2}'`; do echo $i; OS_TENANT_NAME=testlabs openstack server delete $i; sleep 30; done T171473
19:02 demon@tin: Started scap: bootstrap wmf.8
18:32 arlolra: Updated Parsoid to e71937d0 (T178253)
18:31 mholloway-shell@tin: Finished deploy [mobileapps/deploy@9b10959]: Redeploying: Update mobileapps to c002862 (duration: 29m 12s)
18:28 moritzm: upgraded nginx on notebook* to 1.13.6
18:26 madhuvishy: Upgraded notebook1001 and 1002 to kernel version 4.9.51-1~bpo8+1
18:22 arlolra@tin: Finished deploy [parsoid/deploy@b150764]: Updating Parsoid to e71937d0 (duration: 09m 13s)
18:21 ebernhardson@tin: Finished deploy [search/mjolnir/deploy@e6905f4]: test mjolnir deployment (duration: 01m 13s)
18:19 ebernhardson@tin: Started deploy [search/mjolnir/deploy@e6905f4]: test mjolnir deployment
18:13 arlolra@tin: Started deploy [parsoid/deploy@b150764]: Updating Parsoid to e71937d0
18:08 ebernhardson@tin: Finished deploy [search/mjolnir/deploy@cd6ddda]: (no justification provided) (duration: 04m 28s)
18:07 moritzm: rebooting notebook* hosts for update to 4.9.51
18:04 ebernhardson@tin: Started deploy [search/mjolnir/deploy@cd6ddda]: (no justification provided)
18:02 mholloway-shell@tin: Started deploy [mobileapps/deploy@9b10959]: Redeploying: Update mobileapps to c002862
17:58 ebernhardson@tin: Finished deploy [search/mjolnir/deploy@ceb5c2f]: (no justification provided) (duration: 00m 20s)
17:57 ebernhardson@tin: Started deploy [search/mjolnir/deploy@ceb5c2f]: (no justification provided)
17:56 ebernhardson@tin: Finished deploy [search/MjoLniR/deploy@607adfb]: (no justification provided) (duration: 00m 14s)
17:56 ebernhardson@tin: Started deploy [search/MjoLniR/deploy@607adfb]: (no justification provided)
17:44 herron: restarted puppetdb on nitrogen and nihal to pick up jre updates
17:42 smalyshev@tin: Finished deploy [wdqs/wdqs@b44cf27]: data reload/T176593 (duration: 00m 16s)
17:42 smalyshev@tin: Started deploy [wdqs/wdqs@b44cf27]: data reload/T176593
17:27 demon@tin: Pruned MediaWiki: 1.31.0-wmf.3 (duration: 07m 58s)
17:12 demon@tin: Synchronized wmf-config/CommonSettings.php: Beta-only, no-op (duration: 00m 43s)
17:11 demon@tin: Synchronized wmf-config/extension-list-labs: No-op (duration: 00m 44s)
17:10 demon@tin: Pruned MediaWiki: 1.31.0-wmf.6 [keeping static files] (duration: 02m 48s)
16:32 twentyafterfour: Restarting apache2 on phab1001 (deploy phabricator hotfix: D876 )
16:23 jynus: stop labsdb1010 mariadb to clone it later to labsdb1009 T179244
15:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1105 in s1 and s2 - T178359 (duration: 00m 45s)
15:00 thcipriani@tin: testing IRC logging
15:00 urandom: Decommissioning Cassandra, restbase1012-c.eqiad.wmnet (T179422)
14:37 moritzm: installing postgres security updates on maps*
14:29 zeljkof: EU SWAT finished
14:07 moritzm: installing postgres security updates on labsdb1004
13:11 moritzm: installing openssl updates
13:10 akosiaris: shutdown install1002 for disk resize
12:57 godog: upgrade grafana to 4.6.1 on https://grafana-labs.wikimedia.org/ - T180428
12:03 ema: restart varnish-be on cp3007, requests failing with 'no backend connection'
11:49 moritzm: rebooting remaining app servers in eqiad for update to Linux 4.9.51 (and to pick up OpenSSL updates)
11:22 moritzm: rebooting remaining API servers in eqiad for update to Linux 4.9.51 (and to pick up OpenSSL updates)
11:04 godog: reimage restbase2002 - T179422
10:50 marostegui: Deploy alter table on s5: db1104 db1100 db1106 - T174569
10:32 elukey: removed old target configs from /srv/prometheus/analytics/targets on prometheus100[34] after https://gerrit.wikimedia.org/r/391179
10:23 moritzm: rebooting image scalers in eqiad for update to Linux 4.9.51 (and to pick up OpenSSL updates)
10:07 godog: upload scap 3.7.2-1 - T127762
09:59 mobrovac@tin: Synchronized wmf-config/jobqueue.php: JobQueue: Migrate RecordLintJob to EventBus - T175212 (duration: 00m 46s)
09:56 marostegui: Deploy alter table on s5 - dbstore1001 - T174569
09:49 foks: Disabled 2FA for Jean-Frédéric
09:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1105 as multi-instance host for s1 and s2 - T178359 (duration: 00m 46s)
09:45 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db1105 as multi-instance host for s1 and s2 - T178359 (duration: 00m 47s)
09:43 godog: upgrade prometheus to 1.8.1 with k8s on prometheus2004 - T177395
09:27 ppchelko@tin: Finished deploy [cpjobqueue/deploy@df5fca9]: Enable RecordLintJob processing (duration: 00m 40s)
09:27 ppchelko@tin: Started deploy [cpjobqueue/deploy@df5fca9]: Enable RecordLintJob processing
09:07 moritzm: rebooting remaining app servers in codfw for update to Linux 4.9.51 (and to pick up OpenSSL updates)
09:05 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw2108.codfw.wmnet
09:05 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: wtp2018.codfw.wmnet
08:31 marostegui: Deploy alter table on s5 - dbstore1002 - T174569
08:22 addshore@tin: Synchronized wmf-config/extension-list-labs: Add AdvancedSearch to extension-list T180147 PT 2/2 LABS ONLY (duration: 00m 46s)
08:21 addshore@tin: Synchronized wmf-config/extension-list: Add AdvancedSearch to extension-list T180147 PT 1/2 (duration: 00m 47s)
08:13 moritzm: rebooting job runners in codfw for update to Linux 4.9.51 (and to pick up OpenSSL updates)
06:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103 on s2 and s4 - T178359 (duration: 00m 47s)
06:18 marostegui: Deploy alter table on s6 primary master (db1061) - T174569
02:30 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Nov 14 02:30:59 UTC 2017 (duration 6m 38s)
02:24 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.7) (duration: 07m 28s)
01:35 mdholloway: mobileapps finished rolling back to 11/8 deployment
01:34 mholloway-shell@tin: Finished deploy [mobileapps/deploy@00e60b2]: (no justification provided) (duration: 02m 27s)
01:31 mholloway-shell@tin: Started deploy [mobileapps/deploy@00e60b2]: (no justification provided)
01:30 mdholloway: mobileapps rolling back today's deployment due to a significant increase in errors.

2017-11-13

22:49 bawolff: deployed patch T119158 (Will affect language converter. -{}- no longer allowed in link urls)
22:42 bawolff: deployed patch T124404
22:02 mholloway-shell@tin: Finished deploy [mobileapps/deploy@9b10959]: Update mobileapps to c002862 (duration: 05m 31s)
21:57 mholloway-shell@tin: Started deploy [mobileapps/deploy@9b10959]: Update mobileapps to c002862
21:40 urandom: Decommissioning Cassandra, restbase1012-b.eqiad.wmnet (T179422)
20:20 ejegg: updated CiviCRM from ddc3881 to f571c67
19:20 smalyshev@tin: Finished deploy [wdqs/wdqs@b44cf27]: data reload/T176593 (duration: 00m 20s)
19:20 smalyshev@tin: Started deploy [wdqs/wdqs@b44cf27]: data reload/T176593
19:18 thcipriani@tin: Synchronized wmf-config/abusefilter.php: SWAT: Enable per-filter profiling on enwiki T179323 (duration: 00m 45s)
19:16 thcipriani@tin: Synchronized docroot/search.wikimedia.org/index.php: search.wikimedia.org: Clean up result returning logic (duration: 00m 47s)
19:15 madhuvishy: Kick of second dumps rsync from ms1001 to labstore1006
18:28 gehel: deploy latest blazegraph + GUI on wdqs200[23] to switch vocabulary - T176593
18:20 elukey: drain + shutdown analytics1029 as prep step to replace the BBU - T178742
18:09 hoo: Ran "scap pull" on mwdebug1001/snapshot1001 after (further) tests re T177486
18:02 urandom: Restarting Cassandra, restbase-dev1004.eqiad.wmnet (testing new `c-foreach-restart`)
18:00 demon@tin: Synchronized docroot/search.wikimedia.org/index.php: minor cleanup (duration: 00m 47s)
17:28 hoo: Ran "scap pull" on mwdebug1001 after tests re T177486
17:12 mobrovac: restbase depooling restbase1007, restbase1012, restbase1014, restbase2002, restbase2004, restbase2006 for T179422
16:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103 on s2 and s4 - T178359 (duration: 00m 47s)
16:45 mobrovac@tin: Finished deploy [mathoid/deploy@63b2ddc]: Update to service-template-node v0.5.3 - T151396 (duration: 03m 45s)
16:42 mobrovac@tin: Started deploy [mathoid/deploy@63b2ddc]: Update to service-template-node v0.5.3 - T151396
16:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1085 - T174569 (duration: 00m 46s)
15:58 akosiaris@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=wtp2017.codfw.wmnet
15:58 akosiaris@puppetmaster1001: conftool action : set/pooled=no; selector: name=wtp2017.codfw.wmnet
15:22 otto@tin: Finished deploy [eventlogging/analytics@e024af3]: T179625 (duration: 00m 02s)
15:22 otto@tin: Started deploy [eventlogging/analytics@e024af3]: T179625
15:09 otto@tin: Finished deploy [eventlogging/analytics@03285e4]: Reverting, got an error: userAgent is a <type unicode>. (duration: 00m 02s)
15:09 otto@tin: Started deploy [eventlogging/analytics@03285e4]: Reverting, got an error: userAgent is a <type unicode>.
15:08 urandom: Decommissioning Cassandra, restbase1012-a.eqiad.wmnet (T179422)
15:06 otto@tin: Finished deploy [eventlogging/analytics@5796c27]: T179625 (duration: 00m 04s)
15:06 otto@tin: Started deploy [eventlogging/analytics@5796c27]: T179625
14:58 herron: upgrading puppetmaster2002 to puppet 4
14:15 zeljkof: EU SWAT finished
14:10 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Whitelist jenkins in test wiki (T167432) (duration: 00m 47s)
14:08 marostegui: Deploy alter table on db1102.s6 (with replication - sanitarium master) - T174569
13:49 marostegui: Stop replication on labsdb1010 to copy cebwiki.geo_tags table
13:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103 on s2 and s4 - T178359 (duration: 00m 46s)
13:36 marostegui@tin: Synchronized wmf-config/db-codfw.php: Increase weight for db1103 on s2 and s4 - T178359 (duration: 00m 46s)
13:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1083 - T174569 (duration: 00m 46s)
13:09 marostegui: Deploy schema change on db1083 - T174569
12:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1088 - T174569 (duration: 00m 47s)
12:14 ema: cache_misc: upgrade varnish to 5.1.3-1wm2
12:08 ema: cp3008: upgrade varnish to 5.1.3-1wm2
12:02 ema: cp4021: restart varnish-be (mbox lag)
11:56 moritzm: installing imagemagick security updates
11:48 moritzm: installing irssi security updates
11:18 elukey: restart of all the druid daemons on druid100[1-6] to apply the new prometheus jmx jvm exporters - T177459
10:57 gehel: upgrade elasticsearch on cirrus / codfw - T178411
10:49 marostegui: Deploy schema change on db1088 - T174569
10:49 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1088 - T174569 (duration: 00m 54s)
10:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1093 - T174569 (duration: 00m 46s)
10:28 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db1103 as multi-instance host for s2 and s4 - T178359 (duration: 00m 46s)
10:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1103 as multi-instance host for s2 and s4 - T178359 (duration: 00m 47s)
10:12 moritzm: rebooting remaining Parsoid hosts in eqiad for update to Linux 4.9.51 (and to pick up OpenSSL update)
09:44 godog: test upgrade of prometheus 1.8.1 with k8s on prometheus2003 - T177395
09:29 moritzm: rebooting mw1221-mw1235 for update to Linux 4.9.51 (and to pick up OpenSSL update)
09:02 elukey: restart of druid brokers on druid100[1-6] to apply https://gerrit.wikimedia.org/r/390419 - T177459
08:41 marostegui: Deploy alter table on db1093 - T174569
08:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1093 - T174569 (duration: 00m 46s)
08:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1098 after alter table - T174569 (duration: 00m 47s)
08:33 moritzm: rebooting mw1238-mw1258 for update to Linux 4.9.51 (and to pick up OpenSSL update)
07:48 moritzm: installing ruby2.3 security updates
07:38 marostegui: Deploy alter table to db1104 - T179106
07:33 marostegui: Optimize wb_terms table on db2052 - T179106
06:44 marostegui: Deploy alter table directly on codfw s5 master (db2023), this will generate lag on codfw - T179793
06:27 marostegui: Deploy alter table on db1098 - T174569
06:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1098 - T174569 (duration: 00m 49s)
06:18 marostegui: Deploy alter table db2086 - T179106
04:01 urandom: Decommissioning Cassandra, restbase1007-c.eqiad.wmnet (T179422)
02:38 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Nov 13 02:38:38 UTC 2017 (duration 6m 42s)
02:31 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.7) (duration: 08m 18s)

2017-11-12

19:39 urandom: Decommissioning Cassandra, restbase1007-b.eqiad.wmnet (T179422)
16:19 bblack: cp4026 - restart backend (mailbox lag)
01:29 urandom: Decommissioning Cassandra, restbase1007-a.eqiad.wmnet (T179422)

2017-11-11

13:52 urandom: Decommissioning Cassandra, restbase2006-c.codfw.wmnet (T179422)
00:48 urandom: Decommissioning Cassandra, restbase2006-b.codfw.wmnet (T179422)

2017-11-10

18:08 smalyshev@tin: Finished deploy [wdqs/wdqs@ccab8ce]: data reload/T176593 (duration: 00m 34s)
18:08 smalyshev@tin: Started deploy [wdqs/wdqs@ccab8ce]: data reload/T176593
17:20 moritzm: uploaded icu 57.1-6+wmf1 for jessie-wikimedia/component/icu57 (co-installable build for ICU migration)
17:11 moritzm: freed some disk space on install1002
16:47 marostegui: Deploy alter table on s6, dbstore1001, dbstore1002 abd db1030 - T174569
15:03 moritzm: rebooting remaing API servers in codfw to 4.9.51 (and to pick up OpenSSL updates)
14:57 urandom: Decommissioning Cassandra, restbase2006-a.codfw.wmnet (T179422)
14:17 moritzm: rebooting image scalers in codfw to 4.9.51 (and to pick up OpenSSL updates)
13:10 marostegui: truncate /var/log/nginx/error.log.1 on install1002 as it is filling up
13:09 moritzm: powercycling mw2118, stuck after reboot
13:05 ema: cp4021: restart varnish-be due to mbox lag
12:48 moritzm: rebooting video scalers in codfw to 4.9.51 (and to pick up OpenSSL updates)
11:46 _joe_: restarted all services and repooled scb1001
11:38 moritzm: rebooting mw2163-2199 to 4.9.51 (and to pick up OpenSSL updates)
11:32 _joe_: stopping mobileapps as well on scb1001
11:27 moritzm: rebooting wtp1025 to 4.9.51
11:22 _joe_: stopping changeprop, celery-ores, cpjobqueue on scb1001
11:21 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Disable AdvancedSearch on deployment.beta BETA ONLY T180201 (duration: 00m 46s)
11:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove old comment about db1080 (duration: 00m 46s)
11:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1055 original weight - T178359 (duration: 00m 46s)
10:59 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: wtp2017.codfw.wmnet
10:55 _joe_: depooling scb1001 from all services while it becomes healthy again
10:52 _joe_: restarting ores on scb1001, causing memory exhaustion
10:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1055 weight - T178359 (duration: 00m 47s)
10:49 moritzm: powercycling wtp2017, stuck after reboot
10:24 marostegui: Deploy schema change on db2089 - T179106
10:11 moritzm: rebooting Parsoid servers in codfw to 4.9.51 (and to pick up OpenSSL updates)
10:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1055 weight - T178359 (duration: 00m 58s)
09:54 marostegui: Compress enwiki on db1105.s1 - T178359
09:45 moritzm: powercycling mw2108, stuck after reboot
09:30 hashar: Upgrading operations-puppet-tests-docker jenkins job to stop passing docker --tty and thus have signals forwarded from 'docker run' - T176747
09:23 moritzm: rebooting mw2097-mw2117 to 4.9.51 (and to pick up OpenSSL updates)
09:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1055 with low weight - T178359 (duration: 00m 47s)
09:13 moritzm: powercycling mw2213, stuck after reboot
08:43 moritzm: rebooting mw2200-mw2223 to 4.9.51 (and to pick up OpenSSL updates)
07:50 smalyshev@tin: Finished deploy [wdqs/wdqs@213f864]: (no justification provided) (duration: 00m 33s)
07:49 smalyshev@tin: Started deploy [wdqs/wdqs@213f864]: (no justification provided)
07:27 _joe_: restarting apache on phab1001
07:20 marostegui: Deploy alter table on s3.codfw master (db2018) with replication, this will generate lag on codfw - T174569
06:50 marostegui: Deploy alter table on s5 eqiad master (db1063) - T172207
06:41 marostegui: Stop MySQL on db1055 to copy its content to db1105 - T178359
06:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1055 - T178359 (duration: 00m 49s)
06:39 marostegui: Force a BBU relearn on db1046 - T166141
01:21 aaron@tin: Synchronized php-1.31.0-wmf.7/includes/db: Use the main stash for LBFactory "memStash" parameter (duration: 00m 47s)
01:18 demon@tin: Synchronized docroot/search.wikimedia.org/index.php: minor cleanups, less 500s (duration: 00m 47s)

2017-11-09

22:28 urandom: Decommissioning Cassandra, restbase2004-c.codfw.wmnet (T179422)
20:28 demon@tin: Synchronized php-1.31.0-wmf.7/includes/libs/objectcache/WANObjectCache.php: less spammy error logs (duration: 00m 47s)
20:09 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.7
19:06 ebernhardson@tin: Synchronized php-1.31.0-wmf.7/extensions/WikimediaEvents/modules/all/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Turn off DBN sizing AB test (duration: 00m 51s)
19:00 bblack: cp3030 - end experimentation, puppetizing back to normal config
18:46 bblack: cp3030 - round 2 of ssl_do_wait_shutdown test
18:41 arlolra: Updated Parsoid to 2887b5ad (T178253, T173643, T176728, T180010, T171381, T179757)
18:28 arlolra@tin: Finished deploy [parsoid/deploy@d1c7386]: Updating Parsoid to 2887b5ad (duration: 12m 20s)
18:22 bblack: cp3030: puppet-disabled + manual nginx ssl_do_wait_shutdown config
18:16 arlolra@tin: Started deploy [parsoid/deploy@d1c7386]: Updating Parsoid to 2887b5ad
18:14 moritzm: not rebooting parsoid hosts due to Services deployment window, instead rolling restart of mw2120-mw2139 for kernel update to 4.9.51
18:04 moritzm: rolling restart of parsoid servers in codfw for 4.9.51 kernel update
17:14 urandom: Restarting Cassandra, restbase2005-b.codfw.wmnet (T179419)
16:33 urandom: Restarting Cassandra, restbase2005-a.codfw.wmnet (T179419)
15:12 urandom: Creating mathoid schema (T179419)
15:05 addshore@tin: Synchronized wmf-config/CommonSettings-labs.php: Enable AdvancedSearch on beta LABS / BETA ONLY PT2/2 (duration: 00m 49s)
15:04 addshore: last sync was actually "Enable AdvancedSearch on beta"
15:04 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Add AdvancedSearch to extension-list-labs LABS / BETA ONLY PT1/2 (duration: 00m 50s)
14:57 addshore@tin: Synchronized wmf-config/extension-list-labs: Add AdvancedSearch to extension-list-labs LABS / BETA ONLY (duration: 00m 50s)
14:42 zeljkof: EU SWAT finished
14:40 zfilipin@tin: Synchronized wmf-config/StartProfiler.php: SWAT: xenon: encode the request method as a virtual stack frame (duration: 00m 50s)
14:35 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Use a threshold that ores in frwiki can stand (T180115) (duration: 00m 50s)
14:30 zfilipin@tin: Synchronized php-1.31.0-wmf.7/extensions/ContentTranslation/modules/: SWAT: Bring back the overlay support for a specific screen region (T179997) (duration: 00m 50s)
14:25 zfilipin@tin: Synchronized php-1.31.0-wmf.7/extensions/EventBus/EventBus.php: SWAT: Logging improvements Rename logged field to fix logstash mapping (duration: 00m 54s)
14:09 urandom: Decommissioning Cassandra, restbase2004-b.codfw.wmnet (T179422)
13:04 moritzm: rebooting mw1209-mw1220 (app servers) to 4.9.51 (also to pick up new OpenSSL)
12:38 moritzm: rebooting mw1189-mw1208 (API servers) to 4.9.5 (also to pick up new OpenSSL)
12:37 Amir1: ladsgroup@terbium:/srv/mediawiki-staging/php-1.31.0-wmf.6$ mwscript extensions/ORES/maintenance/CheckModelVersions.php --wiki=frwiki (T180115)
11:51 moritzm: rebooting mw1180-mw1188 (app servers) to 4.9.5 (also to pick up new OpenSSL)
11:45 _joe_: removed all local hacks from puppetmaster1001, now it uses rhodium again
11:37 _joe_: cleaning up spurious directories /var/lib/puppet/server/ssl/ca from eqiad's puppetmaster backends, generated due to some error on 8/11/2017
11:10 ema: powercycle cp2008, stuck rebooting
11:03 moritzm: rebooting mw1276-mw1279 (API canaries) to 4.9.5 (also to pick up new OpenSSL)
09:50 moritzm: rolling reboot of scb in eqiad for kernel update (also to pick up openssl updates)
07:56 ema: cp1074 failed rebooting, power-cycled
07:46 _joe_: restarting apache on rhodium after setting --profile --trace in the puppet settings
07:04 legoktm@tin: Synchronized php-1.31.0-wmf.7/resources/: Restore jquery.badge and jquery.placeholder modules (duration: 00m 53s)
02:58 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Nov 9 02:58:30 UTC 2017 (duration 6m 59s)
02:51 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.7) (duration: 09m 09s)
02:29 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.6) (duration: 09m 18s)
01:02 twentyafterfour: Not deploying any phabricator updates this week.

2017-11-08

23:45 urandom: Decommissioning Cassandra, restbase2004.codfw.wmnet (T179422)
22:50 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.7 try #2
22:43 ebernhardson@tin: Synchronized php-1.31.0-wmf.6/extensions/CirrusSearch/: Backport cirrus rescore profile refactor to wmf.6 (duration: 01m 02s)
22:21 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 back to wmf.6
22:09 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.7
21:44 bsitzmann@tin: Finished deploy [mobileapps/deploy@00e60b2]: Update mobileapps to 8e82983 (T178706 T178708 T178333 T170692) (duration: 07m 12s)
21:37 bsitzmann@tin: Started deploy [mobileapps/deploy@00e60b2]: Update mobileapps to 8e82983 (T178706 T178708 T178333 T170692)
21:34 ejegg: re-started donations queue consumer with new thank you letter
21:09 ejegg: updated CiviCRM from b11c591 to ddc3881
20:52 ejegg: turned off donations consumer for ty letter update
20:36 ejegg: updated payments-wiki from a539d27 to d150287
20:33 aaron@tin: Synchronized php-1.31.0-wmf.6/extensions/CentralAuth: Use the proper cache key method in loadFromCache() (duration: 00m 54s)
20:22 apergos: rsync from ms1001 to labstore1006 of dumps, 17T so expect it to take several days
20:10 herron: depooled rhodium via puppetmaster1001 apache config
20:09 demon@tin: Synchronized php: symlink (duration: 00m 49s)
19:57 demon@tin: Synchronized php-1.31.0-wmf.7/extensions/CentralAuth/includes/CentralAuthUser.php: (no justification provided) (duration: 00m 51s)
19:55 urandom: Creating mathoid schema (T179419)
19:31 niharika29@tin: Synchronized php-1.31.0-wmf.6/extensions/ORES/: Store stats of accessing ores service for getting thresholds T179862 (duration: 00m 51s)
19:31 urandom: Creating page restrictions schema (T179421)
19:30 niharika29@tin: Synchronized php-1.31.0-wmf.7/extensions/ORES/: Store stats of accessing ores service for getting thresholds T179862 (duration: 00m 51s)
19:24 niharika29@tin: Synchronized php-1.31.0-wmf.7/extensions/CirrusSearch/: Revert Improve handling of 5xx responses to elasticsearch requests (duration: 01m 02s)
19:14 smalyshev@tin: Finished deploy [wdqs/wdqs@b330bc8]: Update service whitelist (duration: 02m 58s)
19:11 smalyshev@tin: Started deploy [wdqs/wdqs@b330bc8]: Update service whitelist
18:27 demon@tin: Synchronized wmf-config/: Dropping old PrivateSettings symlink (ducks and covers) (duration: 00m 52s)
18:19 demon@tin: Synchronized phpcs.xml: no-op (duration: 00m 50s)
17:58 urandom: Restarting Cassandra, restbase2005-[abc]
17:51 urandom: Clearing snapshots in RESTBase legacy Cassandra cluster (T179417)
17:47 marostegui: Deploy alter table on s1 codfw primary master (db2048) with replication, this will generate lag on codfw - T174569
17:43 mobrovac: restbase truncate the default parsoid storage group's tables for T179417
17:41 urandom: Restarting Cassandra, restbase2001-[abc]
17:19 urandom: Restarting Cassandra, restbase2003-[abc]
17:09 herron: regenerated rhodium puppet certificate
17:08 urandom: Restarting Cassandra, restbase1009-[abc]
16:58 urandom: Restarting Cassandra, restbase1008-[abc]
16:49 awight@tin: Finished deploy [ores/deploy@82a13ae]: Roll back scb1002 (duration: 02m 37s)
16:47 awight@tin: Started deploy [ores/deploy@82a13ae]: Roll back scb1002
16:45 awight@tin: Finished deploy [ores/deploy@1b0e59f]: Try to purge specter of revscoring 1 (duration: 05m 45s)
16:40 awight@tin: Started deploy [ores/deploy@1b0e59f]: Try to purge specter of revscoring 1
16:37 urandom: Restarting Cassandra, restbase1010-[abc]
16:37 godog: disregard message about thumbor rolling-restart, upgrade already done and only thumbor1001 rebooted now
16:34 awight@tin: Finished deploy [ores/deploy@82a13ae]: Fix ORES on scb1002 (duration: 00m 03s)
16:34 awight@tin: Started deploy [ores/deploy@82a13ae]: Fix ORES on scb1002
16:29 godog: roll-restart thumbor in eqiad for kernel upgrade
16:11 mobrovac@tin: Finished deploy [restbase/deploy@c5dd1e2]: Switch wiktionary definitions to use the next-gen storage, take 2b - T179420 (duration: 07m 22s)
16:04 mobrovac@tin: Started deploy [restbase/deploy@c5dd1e2]: Switch wiktionary definitions to use the next-gen storage, take 2b - T179420
16:02 mobrovac@tin: Finished deploy [restbase/deploy@c5dd1e2]: Switch wiktionary definitions to use the next-gen storage, take 2 - T179420 (duration: 00m 13s)
16:02 mobrovac@tin: Started deploy [restbase/deploy@c5dd1e2]: Switch wiktionary definitions to use the next-gen storage, take 2 - T179420
16:01 otto@tin: Finished deploy [eventlogging/analytics@03285e4]: Reverting EvenCapsule update and fixes, processes got restarted too early (duration: 00m 02s)
16:01 otto@tin: Started deploy [eventlogging/analytics@03285e4]: Reverting EvenCapsule update and fixes, processes got restarted too early
15:33 urandom: Decommissioning restbase2001-c.codfw.wmnet (T179422)
15:23 _joe_: testing changes on rhodium regarding hostprivkey,hostcert
15:21 ema: eqiad lvs reboots: upgrading kernel to 4.9.51, libssl to 1.0.2m
15:17 otto@tin: Finished deploy [eventlogging/analytics@02c5a6b]: EventCapsule update and fixes, this is no-op as is. T179625 (duration: 00m 04s)
15:17 otto@tin: Started deploy [eventlogging/analytics@02c5a6b]: EventCapsule update and fixes, this is no-op as is. T179625
15:01 otto@tin: Started restart [eventlogging/eventbus@41e3418]: Bumping worker processes to 16 on all targets: T180017
14:55 otto@tin: Started restart [eventlogging/eventbus@41e3418]: Bumping worker processes to 16: T180017
14:54 ema: esams lvs reboots: upgrading kernel to 4.9.51, libssl to 1.0.2m
14:54 otto@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 12s)
14:54 otto@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
14:41 ema: powercycle cp1050 (failed reboot)
14:29 addshore@tin: Synchronized php-1.31.0-wmf.7/docs/uidesign/mediawiki.diff.html: SWAT Add render moved paragraphs marker in diff view PT 2/2 DOCS ONLY (duration: 00m 50s)
14:28 addshore@tin: Synchronized php-1.31.0-wmf.7/resources/src/mediawiki/mediawiki.diff.styles.css: SWAT Add render moved paragraphs marker in diff view PT 1/2 (duration: 00m 51s)
14:16 volans: upgrading cumin to v1.3.0 on prod and WMCS cumin masters
14:10 zfilipin@tin: Synchronized php-1.31.0-wmf.6/extensions/EventBus/EventBus.php: SWAT: Logging improvements. (duration: 00m 52s)
14:08 ema: codfw lvs reboots: upgrading kernel to 4.9.51, libssl to 1.0.2m
13:03 hasharAway: Upgrading jenkins on contint1001/contint2001
12:54 bblack@puppetmaster1001: conftool action : set/pooled=yes; selector: name=phab2001-vcs.codfw.wmnet
12:53 bblack: restart pybal on lvs2002 for git-ssh.codfw deploy
12:51 bblack: restart pybal on lvs2005 for git-ssh.codfw deploy
12:46 mutante: osmium - re-enabling puppet - temp test is over and will be decom'ed
12:41 mutante: krypton (misc PHP apps, scholarships.wm, iegreview.wm, grafana, racktables, burrow) rebooting for kernel upgrade
12:35 mutante: bromine (misc static tistes, annual/transparency/static-bz) - rebooting for kernel upgrade
12:07 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1051 original weight after maintenance - T178359 (duration: 00m 50s)
12:03 mutante: alcyone (url-downloader) rebooting for kernel upgrade
11:53 mutante: netmon1003 (servermon) - rebooting for kernel upgrade
11:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1051 weight after maintenance - T178359 (duration: 00m 50s)
11:48 moritzm: installed openjdk-8/openssl updates and new kernels on restbase*
11:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1051 weight after maintenance - T178359 (duration: 00m 50s)
11:12 moritzm: installing jenkins security update on releases*
11:10 moritzm: imported jenkins 2.73.3 to apt.wikimedia.org
10:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1051 with low weight after maintenance - T178359 (duration: 01m 01s)
10:57 ema: ulsfo lvs reboots: upgrading kernel to 4.9.51, libssl to 1.0.2m
10:42 mutante: actinium - reboot for kernel upgrade (url-downloader)
10:40 ema: varnish 5.1.3-1wm2 built and uploaded to apt.w.o (experimental)
10:39 mutante: aluminium - reboot for kernel upgrade (url-downloader)
10:28 elukey@puppetmaster1001: conftool action : set/pooled=no; selector: name=aqs1005.eqiad.wmnet
10:18 elukey@puppetmaster1001: conftool action : set/pooled=no; selector: name=aqs1004.eqiad.wmnet
10:07 elukey: reboot aqs100[4-9] for jvm and kernel updates
09:37 mutante: planet1001, alsafi (url-downloader) - apt autoremove; reboot for kernel upgrade
09:32 mutante: restarting ircecho (icinga-wm)
09:26 mutante: ununpentium (rt.wikimedia.org): apt-get autoremove; reboot for kernel upgrade
09:23 mutante: rutherforidum (people.wikimedia.org) : apt-get autoremove ; reboot for kernel upgrade
09:22 Pchelolo: restart cassandra-a on restbase1010
09:09 Pchelolo: restart restbase on 1013 and 1015
08:58 mutante: planet2001 - apt autoremove; reboot for kernel upgrade
08:40 ema: resume cache_text/upload rolling reboots: upgrading kernel to 4.9.51, libssl to 1.0.2m and 1.1.0g
07:55 marostegui: Deploy alter table on s1 - on codfw master (db2048) with replication enabled - T172207
07:46 marostegui: Deploy alter table on s2 - on codfw master (db2017) with replication enabled - T172207
07:21 marostegui: Deploy alter table on s3 - on codfw master (db2018) with replication enabled - T172207
07:13 marostegui: Deploy alter table on s7 - on codfw master (db2029) with replication enabled - T172207
07:13 krinkle@tin: Synchronized docroot/noc/conf/: I2e51e783a (duration: 01m 06s)
06:57 marostegui: Deploy alter table on s6 - on codfw master (db2028) with replication enabled - T172207
06:26 marostegui: Add 330G to db2023 partition to make sure the alter over logging table runs fine - T174569
06:16 Krinkle: Restarted uwsgi-graphite-web service on graphite2001
06:16 Krinkle: Restarted uwsgi-graphite-web service on graphite1001
06:15 marostegui: Stop MySQL on db1051 to copy its content to db1105.s1 - T178359
06:14 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1051 - T178359 (duration: 00m 51s)
05:45 Krinkle: rm -rf /var/lib/carbon/whisper/MediaWiki/wanobjectcache/centralauth_user_* on graphite1001 and graphite2001 for T179999
03:17 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Nov 8 03:17:53 UTC 2017 (duration 7m 13s)
03:10 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.7) (duration: 15m 16s)
02:44 aaron@tin: Synchronized php-1.31.0-wmf.7/tests: Deploy 087f2d579a9f which reverts 4432e898be0 due to statsd spam (duration: 01m 14s)
02:42 aaron@tin: Synchronized php-1.31.0-wmf.7/includes: Deploy 087f2d579a9f which reverts 4432e898be0 due to statsd spam (duration: 01m 40s)
02:33 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.6) (duration: 07m 38s)

2017-11-07

23:57 urandom: Decommissioning restbase2001-b.codfw.wmnet (T179422)
22:50 ejegg: re-enabled thank you mailer
22:49 ejegg: updated CiviCRM from 85e89c6 to dc1b279
22:39 hoo: Reset a global account's email, per T179950
22:28 ejegg: updated CiviCRM from 122ba65 to 85e89c6
21:56 ejegg: updated CiviCRM from dc9d054 to 122ba65
21:20 addshore@tin: Synchronized wmf-config/Wikibase-buildentry.php: T176948 Remove Shared Cache settings from Wikibase-buildentry (duration: 00m 50s)
21:16 addshore@tin: Synchronized wmf-config/Wikibase.php: T176948 Stop using wgWikibaseSharedCacheKeyPrefix from Wikidata build (duration: 00m 49s)
21:09 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: T176948 Remove wmgWikibaseUseConfigFromWikidataBuild PT 3/3 (LABS) (duration: 00m 50s)
21:07 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T176948 Remove wmgWikibaseUseConfigFromWikidataBuild PT 2/3 (duration: 00m 52s)
21:06 addshore@tin: Synchronized wmf-config/Wikibase.php: T176948 Remove wmgWikibaseUseConfigFromWikidataBuild PT 1/3 (duration: 00m 50s)
21:02 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T176948 wmgWikibaseUseConfigFromWikidataBuild flase for all of PROD (duration: 00m 49s)
21:01 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T176948 Load wikibase build from mediawiki-config for wikidataclient (duration: 00m 50s)
20:59 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T176948 Load wikibase build from mediawiki-config for group0 & group1 (duration: 00m 50s)
20:54 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T176948 Load wikibase build from mediawiki-config for hewiki (duration: 00m 49s)
20:51 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T176948 Load wikibase build from mediawiki-config for wikidatawiki (duration: 00m 50s)
20:48 herron: puppet issue cleared after reverting 386666. restarting ircecho on einsteinium
20:45 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: T176948 #1 #2 #3 Load wikibase build from mediawiki-config for BETA ONLY (duration: 00m 50s)
20:28 addshore@tin: Synchronized wmf-config/Wikibase.php: Add loading of wikibase extensions from build PT 3/3 (duration: 00m 50s)
20:27 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Add loading of wikibase extensions from build PT 2/3 (duration: 00m 50s)
20:25 addshore@tin: Synchronized wmf-config/Wikibase-buildentry.php: Add loading of wikibase extensions from build PT 1/3 (duration: 00m 49s)
20:20 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Remove unused wmgUseWikibasePropertySuggester PT 2/2 (duration: 00m 50s)
20:19 addshore@tin: Synchronized wmf-config/CommonSettings.php: Remove unused wmgUseWikibasePropertySuggester PT 1/2 (duration: 00m 50s)
20:06 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.7
19:44 volans: restarted apache2 on rhodium (puppet master failing)
19:31 herron: restart apache2 service on rhodium
19:12 demon@tin: Finished scap: wmf.7 bootstrap (duration: 48m 15s)
19:03 awight: begin stress test on ores*
18:58 awight@tin: Finished deploy [ores/deploy@29905e5]: test deployment to repair ores1009 (non-production) (duration: 00m 31s)
18:57 awight@tin: Started deploy [ores/deploy@29905e5]: test deployment to repair ores1009 (non-production)
18:56 awight@tin: Finished deploy [ores/deploy@29905e5]: test deployment to repair ores1008 (non-production) (duration: 00m 32s)
18:56 awight@tin: Started deploy [ores/deploy@29905e5]: test deployment to repair ores1008 (non-production)
18:50 awight@tin: Finished deploy [ores/deploy@29905e5]: test deployment to repair ores1002 (non-production) (duration: 00m 33s)
18:49 awight@tin: Started deploy [ores/deploy@29905e5]: test deployment to repair ores1002 (non-production)
18:39 volans: slowly running puppet on failed hosts with cumin (concurrency=5)
18:38 awight@tin: Finished deploy [ores/deploy@29905e5]: test deployment to ores* (non-production) (duration: 00m 20s)
18:38 awight@tin: Started deploy [ores/deploy@29905e5]: test deployment to ores* (non-production)
18:36 _joe_: restarting apache2 on rhodium
18:24 demon@tin: Started scap: wmf.7 bootstrap
18:16 awight@tin: Finished deploy [ores/deploy@29905e5]: test deployment to ores* (non-production) (duration: 01m 04s)
18:15 awight@tin: Started deploy [ores/deploy@29905e5]: test deployment to ores* (non-production)
18:06 herron: restarting puppetdb service on nitrogen
17:56 urandom: Decommissioning restbase2001-a.codfw.wmnet (T179422)
17:55 elukey: stop ircecho on einstenium (puppet shower from nitrogen)
17:40 ema: stop cache_text/upload rolling reboots, resuming tomorrow
{{safesubst:SAL entry|1=17:38 urandom: Restart Cassandra, restbase1010-{a,b,c}.eqiad.wmnet (T178177)}}
17:33 urandom: Clearing Cassandra snapshots (T179422)
17:32 moritzm: rolling reboot of scb in codfw to pick up new kernel (and openssl updates)
{{safesubst:SAL entry|1=17:02 urandom: Restarting Cassandra, restbase2001-{a,b,c} to apply OpenJDK upgrade}}
16:47 demon@tin: Synchronized multiversion/submodules.json: no op (duration: 00m 47s)
16:12 ema: start cache_text/upload rolling reboots: upgrading kernel to 4.9.51, libssl to 1.0.2m and 1.1.0g
15:46 gehel: rolling restart of cassandra maps-test for logging change
15:43 godog: roll-restart thumbor in eqiad for kernel upgrade
15:37 urandom: T179420: recreating wiktionary definition schemas
15:29 mobrovac@tin: Finished deploy [restbase/deploy@eab2948]: revert definition switch, wrong schema - T179420 (duration: 06m 46s)
15:22 mobrovac@tin: Started deploy [restbase/deploy@eab2948]: revert definition switch, wrong schema - T179420
15:12 _joe_: added a runner for htmlCacheUpdate on cewiki too
15:10 mobrovac@tin: Finished deploy [restbase/deploy@c5dd1e2]: Switch wiktionary definitions to use the next-gen storage - T179420 (duration: 07m 52s)
15:02 mobrovac@tin: Started deploy [restbase/deploy@c5dd1e2]: Switch wiktionary definitions to use the next-gen storage - T179420
14:52 godog: roll-restart thumbor for kernel upgrades
14:38 mobrovac: restbase creating wiktionary definition schemas for T179420
14:36 godog: reboot wezen for kernel upgrade
14:33 godog: reboot lithium for kernel upgrade
14:26 elukey: rolling restart of kafka on kafka-jumbo* for jvm security updates
14:20 moritzm: rebooting mw canaries to 4.9.51 kernel (also picking up openssl/openssl1.1 updates)
14:19 moritzm: rearming keyholder on naos
14:14 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable ShortUrl on pa.wiki - T178919 (duration: 00m 46s)
14:11 hashar: mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki=pawiki ShortUrl - T178919
14:08 moritzm: rebooting tureis for kernel update
14:07 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: add _ in appendix talk namespace at mywiktionary - T179907 (duration: 00m 45s)
14:02 moritzm: actually holding mw canary reboots until SWAT is over
14:01 moritzm: rebooting mw canaries to 4.9.51 kernel (also picking up openssl/openssl1.1 updates)
13:50 herron: rebooted fermium (lists) for kernel update
13:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add comment about db1105 current status (duration: 00m 47s)
13:43 herron: starting rolling mx reboots for kernel update
13:31 marostegui: Deploy schema change on s5 codfw master (db2023) with replication, this will generate lag on codfw - T174569
13:19 moritzm: rebooting naos for kernel update
13:17 gehel: reboot maps eqiad cluster for upgrades
13:10 moritzm: rebooting wasat for kernel update
12:50 mutante: hafnium, tungsten: groupdel perf-roots to go with gerrit:389663 (T179728)
12:00 mobrovac@tin: Finished deploy [restbase/deploy@eab2948]: Use the new storage for wikidata.org - T179417 (duration: 08m 14s)
11:52 mobrovac@tin: Started deploy [restbase/deploy@eab2948]: Use the new storage for wikidata.org - T179417
11:43 mobrovac: restbase truncating cassandra 2 non-WP tables for T179417
11:42 mobrovac: restbase truncating cassandra 2 non-WP tables for T179420
11:29 moritzm: installing java security updates/restarting cassandra on restbase2001 (cassandra3 node)
11:07 ema: cache_misc rolling reboots: upgrading kernel to 4.9.51, libssl to 1.0.2m and 1.1.0g
10:59 ema: reboot cp3007: upgrading kernel to 4.9.51, libssl to 1.0.2m and 1.1.0g
10:59 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db2092 as recentchanges multi-instance host on s1 and s3 T178359 (duration: 00m 45s)
10:58 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2092 as recentchanges multi-instance host on s1 and s3 T178359 (duration: 00m 45s)
10:41 moritzm: restarting ntpd on dns recursors to pick up openssl update
10:28 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2091 as recentchanges multi-instance host on s2 and s4 T178359 (duration: 00m 45s)
10:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db2091 as recentchanges multi-instance host on s2 and s4 T178359 (duration: 00m 46s)
10:24 elukey: create staging database on db1108 (researchers scratch pad) - T177405
10:19 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2086 as recentchanges multi-instance host on s5 and s7 T178359 (duration: 00m 45s)
10:09 gehel: reboot of maps codfw cluster for upgrades
09:59 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db2086 as recentchanges multi-instance host on s5 and s7 T178359 (duration: 00m 45s)
09:58 paravoid: updating certspotter to 0.5 in apt and tegmen/einsteinium
09:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db2089 as recentchanges multi-instance host on s6 and s5(s8) T178359 (duration: 00m 46s)
09:39 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2089 as recentchanges multi-instance host on s6 and s5(s8) T178359 (duration: 00m 45s)
09:13 marostegui: Stop MySQL on db1038 - host to be decommissioned - T177911
09:11 moritzm: installing java security updates/restarting cassandra on restbase2002
09:02 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1038 from config - T177911 (duration: 00m 45s)
09:02 ppchelko@tin: Finished deploy [cpjobqueue/deploy@3db0cc4]: Do not set Host header for requests to jobrunner (duration: 00m 49s)
09:02 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1038 from config - T177911 (duration: 00m 47s)
09:01 ppchelko@tin: Started deploy [cpjobqueue/deploy@3db0cc4]: Do not set Host header for requests to jobrunner
08:49 marostegui: Run redact_sanitarium for hifwiktionary on db1095 (sanitarium) - T173647
08:48 marostegui: Optimize pagelinks and templatelinks on s7 master - db1062 - T174509
07:18 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: Electron stuck, restarting - T174916
07:11 ema: reboot pinkunicorn for kernel (4.9.51) and openssl (1.0.2m) upgrades
06:19 marostegui: Deploy alter table on s7 codfw master (db2029) with replication, this will cause lag in codfw - T174569
02:29 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Nov 7 02:29:54 UTC 2017 (duration 6m 39s)
02:23 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.6) (duration: 07m 19s)

2017-11-06

22:34 bawolff: Deploy patch for T178451
21:48 arlolra: Updated Parsoid to 6cb77104 (T179579, T175792)
21:42 arlolra@tin: Finished deploy [parsoid/deploy@d1df3c3]: (no justification provided) (duration: 08m 46s)
21:35 awight: Begin stress testing ores* (non-production)
21:33 arlolra@tin: Started deploy [parsoid/deploy@d1df3c3]: (no justification provided)
21:20 XioNoX: re-activating BGP session to telia in ulsfo
21:16 ladsgroup@tin: Finished deploy [ores/deploy@97a1d80]: Deploying early November (T179837) (duration: 10m 47s)
21:05 ladsgroup@tin: Started deploy [ores/deploy@97a1d80]: Deploying early November (T179837)
20:56 reedy@tin: Synchronized wmf-config/interwiki.php: Update IW Map again (duration: 00m 46s)
20:50 marostegui: Deploy alter table on s2 codfw master (db2017) with replication, this will generate lag in codfw - T174569
20:36 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: hifwiktionary T173643 (duration: 00m 45s)
20:34 reedy@tin: rebuilt wikiversions.php and synchronized wikiversions files: hifwiktionary T173643
20:33 reedy@tin: Synchronized dblists/: hifwiktionary T173643 (duration: 00m 47s)
20:04 reedy@tin: Synchronized wmf-config/interwiki.php: Update IW map (duration: 00m 45s)
19:57 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: electcomwiki T174370 (duration: 00m 45s)
19:56 reedy@tin: rebuilt wikiversions.php and synchronized wikiversions files: electcomwiki T174370
19:55 reedy@tin: Synchronized dblists/: electcomwiki T174370 (duration: 00m 44s)
19:52 demon@tin: Pruned MediaWiki: 1.31.0-wmf.2 (duration: 02m 49s)
19:47 demon@tin: Pruned MediaWiki: 1.31.0-wmf.5 [keeping static files] (duration: 01m 38s)
19:22 Amir1: ladsgroup@terbium:/srv/mediawiki-staging/php-1.31.0-wmf.4$ mwscript extensions/ORES/maintenance/CheckModelVersions.php --wiki=enwiki (T179596)
19:21 kaldari@tin: Synchronized wmf-config/InitialiseSettings.php: Updating InitialiseSettings for draftquality data (duration: 00m 48s)
18:15 XioNoX: deactivating BGP session to telia in ulsfo
18:09 gehel@tin: Finished deploy [wdqs/wdqs@fa8bb12]: WDQS GUI and Blazegraph update (duration: 01m 53s)
18:07 gehel@tin: Started deploy [wdqs/wdqs@fa8bb12]: WDQS GUI and Blazegraph update
17:50 awight@tin: Finished deploy [ores/deploy@29905e5]: restart services on ores* (non-production) (duration: 17m 18s)
17:48 urandom: T179083: Restarting Cassandra instances on restbase2001.codfw.wmnet
17:32 awight@tin: Started deploy [ores/deploy@29905e5]: restart services on ores* (non-production)
16:51 marostegui: Optimize pagelinks and templatelinks on s2 master - db1054 - T174509
16:26 bblack: Switching unified TLS certificate in the US
16:25 awight@tin: Finished deploy [ores/deploy@82a13ae]: Redeploy ORES master to scb* (duration: 16m 10s)
16:09 awight@tin: Started deploy [ores/deploy@82a13ae]: Redeploy ORES master to scb*
15:50 akosiaris@tin: Started deploy [ores/deploy@29905e5]: testing deploy
15:49 akosiaris@tin: Started deploy [ores/deploy@29905e5]: testing deploy
15:43 awight@tin: Finished deploy [ores/deploy@29905e5]: restart services on ores* (non-production) (duration: 12m 45s)
15:31 awight@tin: Started deploy [ores/deploy@29905e5]: restart services on ores* (non-production)
15:26 gehel: reboot of maps-test cluster for jvm and kernel upgrades
15:13 ppchelko@tin: Finished deploy [cpjobqueue/deploy@96f55c6]: USe correct regex for job selection (duration: 00m 29s)
15:12 ppchelko@tin: Started deploy [cpjobqueue/deploy@96f55c6]: USe correct regex for job selection
15:03 addshore@tin: Synchronized wmf-config/CommonSettings-labs.php: SWAT Set wgWikiDiff2MovedParagraphDetectionCutoff for group0 PT 3/3 (duration: 00m 46s)
15:01 addshore@tin: Synchronized wmf-config/CommonSettings.php: SWAT Set wgWikiDiff2MovedParagraphDetectionCutoff for group0 PT 2/3 (duration: 00m 46s)
15:00 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT Set wgWikiDiff2MovedParagraphDetectionCutoff for group0 PT 1/3 (duration: 00m 47s)
14:57 gehel: reboot of relforge for kernel + jvm upgrade
14:50 mobrovac@tin: Synchronized wmf-config/jobqueue.php: Switch MessageIndexRebuildJob, flaggedrevs_CacheUpdate and deleteLinks jobs to the EventBus infrastructure - T175210 (duration: 00m 46s)
14:50 ppchelko@tin: Finished deploy [cpjobqueue/deploy@e93feba]: Start processing all 'hearted' jobs T175210 (duration: 00m 44s)
14:49 ppchelko@tin: Started deploy [cpjobqueue/deploy@e93feba]: Start processing all 'hearted' jobs T175210
14:48 ppchelko@tin: Started deploy [cpjobqueue/deploy@e93feba]: (no justification provided)
14:45 awight@tin: Finished deploy [ores/deploy@29905e5]: Force deployment on ores* (non-production) (duration: 05m 43s)
14:45 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs2003.codfw.wmnet
14:45 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs2002.codfw.wmnet
14:40 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs2002.codfw.wmnet
14:40 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs2001.codfw.wmnet
14:39 awight@tin: Started deploy [ores/deploy@29905e5]: Force deployment on ores* (non-production)
14:35 awight@tin: Finished deploy [ores/deploy@29905e5]: restart services on ores* (non-production) (duration: 04m 48s)
14:34 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs2001.codfw.wmnet
14:33 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs1005.eqiad.wmnet
14:30 awight@tin: Started deploy [ores/deploy@29905e5]: restart services on ores* (non-production)
14:27 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs1005.eqiad.wmnet
14:27 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs1004.eqiad.wmnet
14:21 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs1004.eqiad.wmnet
14:21 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs1003.eqiad.wmnet
14:20 awight@tin: Finished deploy [ores/deploy@29905e5]: celery 4 -> ores* (non-production) (duration: 19m 22s)
14:18 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: EventBus: Enable logging - T150106 (duration: 00m 47s)
14:14 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs1003.eqiad.wmnet
14:14 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=wdqs1003.eqiad.wmnet
14:13 mobrovac@tin: Synchronized php-1.31.0-wmf.6/extensions/EventBus/EventBus.php: EventBus - Improve logging for T150106 (duration: 00m 48s)
14:01 gehel: full restart of wdqs eqiad / codfw for multiple upgrades
14:01 awight@tin: Started deploy [ores/deploy@29905e5]: celery 4 -> ores* (non-production)
13:03 elukey: rolling restart of druid historical daemons on druid100[1-6] to apply https://gerrit.wikimedia.org/r/#/c/389429
12:58 moritzm: installing openssl security updates
12:55 moritzm: restarting apache on netmon to pick up openssl update
12:43 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: Electron stuck, restrting - T174916
12:29 moritzm: installing imagemagick security updates
11:45 oblivian@tin: Started deploy [jobrunner/jobrunner@a20d043]: (no justification provided)
11:44 marostegui: Deploy alter table on s6 codfw master (with replication, so this will generate lag on codfw) - T174569
11:19 marostegui: Compress InnoDB on db1103.s2 - T178359
11:16 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2087 as recentchanges multi-instance host on s6 and s7 T178359 (duration: 00m 47s)
11:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db2087 as recentchanges multi-instance host on s6 and s7 T178359 (duration: 00m 48s)
10:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1101 original weight after maintenance - T178359 (duration: 00m 46s)
10:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give db1101 more weight after maintenance - T178359 (duration: 00m 46s)
10:05 moritzm: uploading linux 4.9.51-1~bpo8+1 for jessie-wikimedia to apt.wikimedia.org
09:57 godog: roll-upgrade swift to 2.10.2 in codfw and roll-restart for kernel upgrade - T177739
09:56 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db2088 as recentchanges multi-instance host on s1 and s2 T178359 (duration: 00m 46s)
09:55 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2088 as recentchanges multi-instance host on s1 and s2 T178359 (duration: 00m 46s)
09:37 _joe_: manually running htmlCacheUpdate for commonswiki and ruwiki on terbium, T173710
09:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db2085 as recentchanges multi-instance host on s3 and s5 T178553 T178359 (duration: 00m 46s)
09:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2085 as multi-instance core host on eqiad file T178553 T178359 (duration: 00m 46s)
09:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give db1101 more weight after maintenance - T178359 (duration: 00m 46s)
09:07 moritzm: removed monitoring/RAID packages from stretch-wikimedia/thirdparty, now in thirdparty/hwraid
08:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1101 with low weight after maintenance - T178359 (duration: 00m 47s)
06:46 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2084 as multi-instance core host on eqiad file T178553 T178359 (duration: 00m 46s)
06:38 marostegui: Stop MySQL on db1101 to copy its content to db1103.s4 - T178359
06:31 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1101 - T178359 (duration: 00m 48s)
06:19 marostegui: Optimize pagelinks and templatelinks on s6 master (db1061) - T174509
06:16 marostegui: Deploy alter table on s4 codfw master (with no replication) - T174569
02:44 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Nov 6 02:44:41 UTC 2017 (duration 6m 55s)
02:37 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.6) (duration: 07m 17s)

2017-11-04

13:34 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=scb1002.eqiad.wmnet
13:19 elukey@puppetmaster1001: conftool action : set/pooled=no; selector: name=scb1002.eqiad.wmnet
12:36 mobrovac: trending-edits depool it from scb1002 to investigate
00:18 Krinkle: Finished whisper-mass-resize for frontend.navtiming on graphite2001 (T179622)

2017-11-03

20:28 marostegui: Deploy alter table db2058 - T174569
18:52 Krinkle: Starting whisper-mass-resize for frontend.navtiming on graphite2001 (T179622)
18:51 Krinkle: whisper-mass-resize completed for graphite1001.eqiad.wmnet:/var/lib/carbon/whisper/frontend/navtiming (at 01:34 UTC actually)
17:56 demon@tin: Synchronized php-1.31.0-wmf.6/includes/libs/rdbms/database/DatabaseMysqlBase.php: logging (duration: 00m 47s)
16:17 marostegui: Deploy alter table on db2044 - T174569
16:03 demon@tin: Synchronized php-1.31.0-wmf.6/extensions/ContentTranslation/api/ApiQueryTranslatorStats.php: fix warnings about bad min() calls (duration: 00m 47s)
15:50 cmjohnson1: powering off labvirt1015 to swap CPU (again)
15:49 akosiaris: T177393 enable RBAC for kubernetes in production and staging
15:35 volans: uploaded cumin_1.3.0-1_amd64.deb to apt.wikimedia.org jessie-wikimedia
14:19 moritzm: installing java sec updates / cassandra restarts on xenon/praseodymium (cerium already done earlier)
14:16 hashar: restarting Jenkins on contint1001
14:12 hashar: Upgrading packages on contint1001 / contint2001
13:50 moritzm: installing heimdal security updates on trusty
13:35 ppchelko@tin: Started restart [electron-render/deploy@8dd5f13]: Restarting, service is stuck
13:33 Pchelolo: restart electron render service
12:41 moritzm: uploaded openjdk 8u151-b12 to apt.wikimedia.org/jessie-wikimedia
11:45 moritzm: restarting jenkins on releases* to pick up openjdk security update
11:41 mutante: gerrit - restart Apache - apply gerrit:354078 - only listen on service IP
11:12 moritzm: uploaded openssl 1.0.2m-1 for to apt.wikimedia.org
10:46 marostegui: Compress InnoDB on db1103.s4 - T178359
10:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add comment to db1103 status (duration: 00m 47s)
10:42 mutante: cobalt: (gerrit) temp disable puppet - gerrit2001: restart apache, apply gerrit:354078 stop listening on all interfaces
10:39 oblivian@tin: Synchronized wmf-config/CommonSettings.php: Increase concurrency of htmlCacheUpdate jobs T173710 (duration: 00m 48s)
09:09 mobrovac@tin: Finished deploy [restbase/deploy@a67b4a7]: Use only the new storage for summaries - T179418 (duration: 10m 40s)
08:58 mobrovac@tin: Started deploy [restbase/deploy@a67b4a7]: Use only the new storage for summaries - T179418
08:46 ema: pool cp4021
08:16 elukey: drop CommandInvocation_15243810 and CommandInvocation_15243810_15423246 from analytics dbs (db1046/db1047/db1108/dbstore1002) - data archived on HDFS - T166712
08:12 moritzm: installing openjdk-8 security updates on stretch
07:57 marostegui: Deploy alter table on s4.db2037 - T174569
07:29 elukey: depooled mw1191 and powercycle - host down with 'CPU 1 check errors' in racadm getsel
06:39 marostegui: Stop MySQL on db2089.s5 to copy it to db2085 - T178359
06:25 marostegui: Stop MySQL on db1103 to copy it to dbstore1001 and reimage it - T178359

2017-11-02

23:36 krinkle@tin: Synchronized php-1.31.0-wmf.6/extensions/NavigationTiming/modules/ext.navigationTiming.js: Fix zero values - T178479 (duration: 00m 47s)
23:10 herron: disabled puppet master debug logging on puppetmaster2001 via /usr/share/puppet/rack/puppet-master/config.ru
22:48 volans: cleaned daemon.log from puppet spam logs on puppetmaster2001 (15GB) - cc herron
22:39 halfak@tin: Started deploy [ores/deploy@82a13ae]: T179621
22:02 Krinkle: Mass-resizing Graphite/Whisper files on graphite1001 and graphite2001 for T179622 (frontend.* namespace)
21:53 ayounsi@tin: Finished deploy [librenms/librenms@ad48b4d]: (no justification provided) (duration: 00m 05s)
21:52 ayounsi@tin: Started deploy [librenms/librenms@ad48b4d]: (no justification provided)
21:50 XioNoX: upgrading librenms
21:38 hoo: Updated the Wikidata property suggester with data from Monday's JSON dump and applied the T132839 workarounds
20:56 madhuvishy: Powercycling labvirt1015
20:18 chasemp: simulate load for labvirt1015 'sudo cumin "name:labvirt1015stresstest*" "stress-ng --cpu 1 --io 2 --vm 1 --vm-bytes 1G &"'
19:45 mepps: updated payments-wiki from b88b805 to a539d27
19:28 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.6
18:20 gehel: restarting blazegraph on wdqs1003
18:13 gehel: restarting stuck updater on wdqs1003
17:55 demon@tin: Synchronized php-1.31.0-wmf.6/includes/Message.php: logging fixes (duration: 00m 48s)
17:53 demon@tin: Synchronized php-1.31.0-wmf.6/includes/libs/rdbms: logging improvements (duration: 00m 54s)
17:47 moritzm: uploaded openssl 1.1.0g-1+wmf1 to jessie-wikimedia
17:42 awight@tin: Finished deploy [ores/deploy@0e54a3c]: revscoring 2, T175180 (duration: 33m 50s)
17:08 awight@tin: Started deploy [ores/deploy@0e54a3c]: revscoring 2, T175180
16:14 ema: restart varnish-be on cp4026 due to mbox lag
16:04 mobrovac@tin: Synchronized wmf-config/jobqueue.php: Use only EventBus for processing updateBetaFeatureUserCount - T175210 (duration: 00m 51s)
13:46 chasemp: ugrade dnsmasq-base on labcontrol*
13:36 zeljkof: EU SWAT finished
13:35 zfilipin@tin: Synchronized php-1.31.0-wmf.6/extensions/ContentTranslation/modules/publish/ext.cx.publish.js: SWAT: CX1: Check for template adaptation failures before publishing (T154116) (duration: 00m 50s)
13:11 zfilipin@tin: Synchronized wmf-config/ProductionServices.php: SWAT: use the logstash LVS endpoint (T175242) (duration: 00m 51s)
12:09 moritzm: removed obsolete packages from jessie-wikimedia/experimental
12:06 mobrovac@tin: Finished deploy [restbase/deploy@9314cf6]: Parsoid: Switch all but WPs to use the next-generation storage - T179417 (duration: 08m 43s)
11:57 mobrovac@tin: Started deploy [restbase/deploy@9314cf6]: Parsoid: Switch all but WPs to use the next-generation storage - T179417
11:43 elukey: drop table log.PageContentSaveComplete_5588433 from db1046,db1047,db1108,dbstore1002, archived on hdfs - T177101
11:31 moritzm: remove obsolete packages from stretch-wikimedia/experimental, now empty
11:29 mobrovac@tin: Finished deploy [restbase/deploy@f6c4e2d]: Parsoid module: use the Cassandra 2 tables as fallback when needed - T179417 (duration: 08m 15s)
11:26 elukey: drop log.MediaViewer_10867062_15423246 from db1047,db1108 since already archived in hdfs - T168303
11:21 mobrovac@tin: Started deploy [restbase/deploy@f6c4e2d]: Parsoid module: use the Cassandra 2 tables as fallback when needed - T179417
10:54 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=logstash1003.eqiad.wmnet
10:53 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=logstash1002.eqiad.wmnet
10:53 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=logstash1001.eqiad.wmnet
10:53 gehel: depooling logstash100[123] in preparation for decommission - T175830
10:38 gehel: rolling restart of wdqs nodes for GC tuning - T175919
10:13 marostegui: Deploy alter table on db2019 - T174569
09:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1091 original weight - T161088 (duration: 00m 50s)
09:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1091 weight after maintenance - T161088 (duration: 00m 50s)
08:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1091 weight after maintenance - T161088 (duration: 00m 50s)
08:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1091 with low weight after maintenance - T161088 (duration: 00m 50s)
08:15 moritzm: restarting app server canaries to pick up curl security update
08:04 moritzm: installing curl security updates
07:59 marostegui: Stop mysql event scheduler on labsdb1001 - T179464
06:54 marostegui: Stop s3 instance on db2092 to copy it to db2085 - T178359
06:33 legoktm: done on mwdebug1002
06:31 marostegui: Stop MySQL on db1091 and db1103 to migrate db1091 to file per table
06:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1091 - T161088 (duration: 01m 07s)
06:27 marostegui: Deploy optimize table wikidatawiki.pagelinks on s5 master (db1063) - T174509
06:02 legoktm: live-hacking mwdebug1002
05:43 TimStarling: on puppetmaster2001: disk full due to logs, so I'm truncating /var/log/debug which is apparently copied to daemon.log anyway
03:29 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Nov 2 03:29:35 UTC 2017 (duration 7m 12s)
03:22 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.6) (duration: 15m 37s)
02:43 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.5) (duration: 16m 06s)
00:52 ebernhardson@tin: Synchronized php-1.31.0-wmf.5/extensions/WikimediaEvents/extension.json: SWAT followup: Update SearchSatisfaction eventlogging schema revision id (duration: 00m 50s)
00:51 ebernhardson@tin: Synchronized php-1.31.0-wmf.6/extensions/WikimediaEvents/extension.json: (no justification provided) (duration: 00m 50s)
00:32 bblack: cp4022 - varnish backend restart, mailbox
00:28 bblack: lvs1001-3: re-enable ethernet flowcontrol (short ethernet blip), with pybal stopped to failover to backups during...

2017-11-01

23:37 thcipriani@tin: Synchronized php-1.31.0-wmf.5/extensions/WikimediaEvents: SWAT: Turn on Cirrus AB test for DBN group sizing (duration: 00m 50s)
23:35 thcipriani@tin: Synchronized php-1.31.0-wmf.6/extensions/WikimediaEvents: SWAT: Turn on Cirrus AB test for DBN group sizing (duration: 00m 51s)
23:29 thcipriani@tin: Synchronized php-1.31.0-wmf.6/extensions/ParserMigration/includes/ApiParserMigration.php: SWAT: API: Fix WikiPage namespace (duration: 00m 52s)
23:25 thcipriani@tin: Synchronized php-1.31.0-wmf.6/extensions/Wikibase/repo/Wikibase.hooks.php: SWAT: Allow turning Cirrus usage off from query T179428 (duration: 00m 49s)
23:23 thcipriani@tin: Synchronized php-1.31.0-wmf.6/extensions/Wikidata/extensions/Wikibase/repo/Wikibase.hooks.php: SWAT: Allow turning Cirrus usage off from query T179428 (duration: 00m 51s)
23:09 thcipriani@tin: Synchronized wmf-config/Wikibase-production.php: SWAT: Revert "Revert "Add negative weight to disambig entities"" T148411 (duration: 00m 51s)
22:49 ejegg: disabled fundraising unsubscribe import processing
22:33 Krinkle: group2(all-wikidata) wikis to wmf.5 from 24 hours ago seems to have caused a 60% drop in navigation timing metric report count (100/min => 40/min)
22:19 eileen1: process-control updated to 9c8b4bb
22:17 demon@tin: Synchronized wmf-config/CommonSettings.php: rel1_28 is dead (duration: 00m 50s)
22:06 ejegg: re-enabled fundraising jobs
21:48 ejegg: fundraising jobs disabled
20:10 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.6
20:05 demon@tin: Synchronized php: symlink swap (duration: 00m 49s)
19:57 krinkle@tin: Synchronized php-1.31.0-wmf.6/resources/src/startup.js: T178943 (duration: 00m 51s)
19:53 demon@tin: Synchronized php-1.31.0-wmf.5/extensions/CentralNotice: fix weird git rebasing issue (duration: 00m 53s)
19:41 demon@tin: Pruned MediaWiki: 1.31.0-wmf.4 [keeping static files] (duration: 01m 52s)
19:31 awight@tin: Synchronized php-1.31.0-wmf.5/extensions/ORES: Fix for API=oresscores, T179430 (duration: 00m 50s)
19:28 awight@tin: Synchronized php-1.31.0-wmf.6/extensions/ORES: Fix for API=oresscores, T179430 (duration: 00m 52s)
19:14 bblack: all hosts: manual cumin+sed removal of ethernet autoneg params from /e/n/i to match https://gerrit.wikimedia.org/r/#/c/387849/
19:08 otto@tin: Finished deploy [analytics/refinery@6d11d67]: Deploying refinery-source artifacts for 0.0.54 for JsonRefine job, T162610 (duration: 03m 45s)
19:05 otto@tin: Started deploy [analytics/refinery@6d11d67]: Deploying refinery-source artifacts for 0.0.54 for JsonRefine job, T162610
18:39 demon@tin: Synchronized wmf-config/: Dropping squid.php (hang on to your pants folks, this could be fun) (duration: 00m 51s)
18:36 demon@tin: Synchronized wmf-config/CommonSettings.php: use reverse-proxy.php no more squid.php (duration: 00m 50s)
18:35 demon@tin: Synchronized docroot/: dropping squid.php (duration: 00m 52s)
18:13 demon@tin: Synchronized static/images/project-logos/sewikimedia.png: new logo (duration: 00m 50s)
18:02 moritzm: installing openjpeg2 security updates
17:39 demon@tin: Synchronized php-1.31.0-wmf.6/includes/specials/pagers/ContribsPager.php: fix missing page_is_new error (duration: 00m 51s)
17:10 demon@tin: Synchronized wmf-config/: dropped squid-labs, no-op in prod (duration: 00m 52s)
17:09 demon@tin: Synchronized docroot/noc/: dropped squid-labs.php (duration: 00m 51s)
16:01 bblack: lvs1003 - puppet disabled, testing experimental ethtool ringbuffer change
15:15 moritzm: repo reorg: moved ftpsync from thirdparty to main and docker-engine from thirdparty to thirdparty/k8s
15:12 awight@tin: Finished deploy [ores/deploy@9f361d2]: revscoring 2 -> ores1002 (non-production) (duration: 02m 25s)
15:10 awight@tin: Started deploy [ores/deploy@9f361d2]: revscoring 2 -> ores1002 (non-production)
15:09 awight@tin: Finished deploy [ores/deploy@9f361d2]: revscoring 2 -> ores* (non-production) (duration: 01m 57s)
15:07 awight@tin: Started deploy [ores/deploy@9f361d2]: revscoring 2 -> ores* (non-production)
15:07 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: Update CirrusSearch MLR model on enwiki (duration: 00m 51s)
15:01 mobrovac: restbase: removing wikimedia-lvs-realserver from staging hosts T179494
14:55 bblack: strongswan experiment done, cp* back to puppet-agent-enabled
14:41 moritzm: installing poppler security updates
14:09 bblack: cp*: disabling puppet to test strongswan change...
13:49 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable NewUserMessage on fawikiquote - T179442 (duration: 00m 50s)
13:46 hashar@tin: Synchronized wmf-config/abusefilter.php: Enable blocking feature of abuse filter in fawikiquote - T178227 (duration: 00m 50s)
13:45 moritzm: installing quagga security updates
13:43 hashar@tin: Synchronized wmf-config/CommonSettings.php: Properly check for cluster existence prior setting TTM mirrors - T179270 (duration: 01m 05s)
13:41 ppchelko@tin: Finished deploy [restbase/deploy@2321c4c]: Update hyperswitch dependency. Take 2 (duration: 05m 50s)
13:35 ppchelko@tin: Started deploy [restbase/deploy@2321c4c]: Update hyperswitch dependency. Take 2
13:33 ppchelko@tin: Finished deploy [restbase/deploy@2321c4c]: Update hyperswitch dependency (duration: 03m 50s)
13:29 ppchelko@tin: Started deploy [restbase/deploy@2321c4c]: Update hyperswitch dependency
12:48 moritzm: installing libav security updates
12:44 moritzm: installinng libdatetime-timezone-perl stable updates on Debian
12:10 kartik@tin: Finished deploy [cxserver/deploy@10651e2]: Update cxserver to 0227acb (duration: 03m 07s)
12:07 kartik@tin: Started deploy [cxserver/deploy@10651e2]: Update cxserver to 0227acb
08:35 elukey: forced umount/mount for /mnt/hdfs on stat1005 (not working after repeated oom kill actions)
03:22 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Nov 1 03:22:02 UTC 2017 (duration 7m 17s)
03:14 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.6) (duration: 15m 21s)
02:36 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.5) (duration: 09m 46s)

2017-10-31

23:56 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: Adjust cirrussearch interleave configuration for AB test (duration: 00m 50s)
23:48 demon@tin: Synchronized scap/plugins/prep.py: no-op (duration: 00m 50s)
23:21 ebernhardson@tin: Synchronized wmf-config/CirrusSearch-common.php: Setup CirrusSearch AB test on dbn group sizing (duration: 00m 50s)
23:19 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: Setup CirrusSearch AB test on dbn group sizing (duration: 00m 53s)
21:38 awight@tin: Started deploy [ores/deploy@9f361d2]: revscoring 2 -> ores* (non-production)
21:37 awight@tin: Started deploy [ores/deploy@9f361d2]: revscoring 2 -> ores* (non-production)
21:16 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/386553 https://gerrit.wikimedia.org/r/386710 (duration: 00m 52s)
21:00 XioNoX: removing old AMS-IX IPv6 - T167840
20:57 mepps: rollback payments-wiki from 5303a8e to b88b805
20:54 mepps: updated payments-wiki from b88b805 to 5303a8e
20:45 mepps: rolledback payments-wiki from 5303a8e to b88b805
20:43 mepps: updated payments-wiki from b88b805 to 5303a8e
19:18 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.6
18:53 awight@tin: Finished deploy [ores/deploy@437fcf5]: revscoring 2 -> ores* (non-production) (duration: 10m 29s)
18:42 awight@tin: Started deploy [ores/deploy@437fcf5]: revscoring 2 -> ores* (non-production)
18:38 awight@tin: Finished deploy [ores/deploy@437fcf5]: revscoring 2 -> ores1002 (non-production) (duration: 02m 24s)
18:36 awight@tin: Started deploy [ores/deploy@437fcf5]: revscoring 2 -> ores1002 (non-production)
18:32 demon@tin: Synchronized php-1.31.0-wmf.6/extensions/CentralNotice: deployyyyy (duration: 00m 52s)
18:27 ejegg: updated CiviCRM from a34a9c6 to 415fde1
17:23 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: wikidata back to wmf.5
16:50 demon@tin: Finished scap: bootstrap wmf.6 (duration: 47m 39s)
16:41 ppchelko@tin: Finished deploy [restbase/deploy@7631ea7]: Enable new parsoid storage on test wikipedias (duration: 09m 07s)
16:32 ppchelko@tin: Started deploy [restbase/deploy@7631ea7]: Enable new parsoid storage on test wikipedias
16:09 ariel@tin: Finished deploy [dumps/dumps@fa0583b]: adapt dumpadmin script to use override section in config (duration: 00m 02s)
16:09 ariel@tin: Started deploy [dumps/dumps@fa0583b]: adapt dumpadmin script to use override section in config
16:02 demon@tin: Started scap: bootstrap wmf.6
15:43 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: all but wikidata.org -> wmf.5
15:42 demon@tin: Synchronized php: Symlink bump (duration: 00m 48s)
15:41 demon@tin: Pruned MediaWiki: 1.31.0-wmf.1 (duration: 03m 15s)
14:18 bblack: authdns (baham, radon, eeden): upgrade base packages (curl, git, libc-dev, etc)
14:13 bblack: lvsNNNN: upgrade base packages (curl, git, libc-dev, etc)
14:11 bblack: cpNNNN: upgrade base packages (curl, git, libc-dev, etc)
13:57 bblack: caches@eqiad - upgrade nginx to 1.13.6-2+wmf1~jessie1
13:54 bblack: caches@esams - upgrade nginx to 1.13.6-2+wmf1~jessie1
13:22 bblack: caches@codfw - upgrade nginx to 1.13.6-2+wmf1~jessie1
13:18 zeljkof: EU SWAT finished
13:17 zfilipin@tin: Synchronized php-1.31.0-wmf.4/extensions/EventBus/EventBus.php: SWAT: Dont log full request to service in case of delivery error (T179280) (duration: 00m 51s)
13:14 bblack: caches@ulsfo - upgrade nginx to 1.13.6-2+wmf1~jessie1
13:05 mobrovac@tin: Finished deploy [restbase/deploy@c0f9dc4]: Cassandra 3 Parsoid stash tables (duration: 02m 58s)
13:02 mobrovac@tin: Started deploy [restbase/deploy@c0f9dc4]: Cassandra 3 Parsoid stash tables
13:01 mobrovac@tin: Finished deploy [restbase/deploy@c0f9dc4]: Cassandra 3 Parsoid stash tables (duration: 15m 19s)
12:45 mobrovac@tin: Started deploy [restbase/deploy@c0f9dc4]: Cassandra 3 Parsoid stash tables
12:39 bblack: eqiad primary lvses (1001-3): disable pause [LRO not possible] on all ethernet interfaces (under pybal stopped briefly)
12:36 bblack: codfw primary lvses (2003-6): disable LRO,pause on all ethernet interfaces (under pybal stopped briefly)
12:24 bblack: cp4025: restart varnish-be for mailbox lag
12:23 bblack: cp4023: restart varnish-be for mailbox lag
12:21 bblack: esams primary lvses (3001-2): disable LRO,pause on eth0 (under pybal stopped briefly)
12:19 bblack: ulsfo primary lvses (4001-2): disable LRO,pause on eth0 (under pybal stopped briefly)
12:12 bblack: esams+ulsfo backup lvses (3003-4,4003-4): disable LRO,pause on eth0
12:10 bblack: core DC backup lvses (1004-6,2004-6): disable LRO,pause on all ethernet interfaces
11:05 oblivian@tin: Finished deploy [docker-pkg/deploy@20908e8]: Actually deploy the fix (duration: 00m 04s)
11:05 oblivian@tin: Started deploy [docker-pkg/deploy@20908e8]: Actually deploy the fix
11:02 oblivian@tin: Finished deploy [docker-pkg/deploy@576c80f]: Fix image_tag filter (duration: 00m 02s)
11:02 oblivian@tin: Started deploy [docker-pkg/deploy@576c80f]: Fix image_tag filter
10:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1084 original weight - T161088 (duration: 00m 48s)
10:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1084 weight - T161088 (duration: 00m 50s)
09:12 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2084 as multi-instance core host T178553 T178359 (duration: 00m 49s)
08:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1084 weight - T161088 (duration: 00m 50s)
08:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1084 with low weight - T161088 (duration: 00m 50s)
08:15 marostegui: Stop MySQL on db2086 to copy s5 to db2089 - T178359
07:09 marostegui: Stop MySQL on db2087.s6 to copy its content to db2089 - T178359
06:37 marostegui: Stop MySQL on db1103 and db1084 - T161088
06:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1084 - T161088 (duration: 00m 49s)
06:17 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2039 (duration: 01m 12s)
03:08 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Oct 31 03:08:48 UTC 2017 (duration 7m 18s)
03:01 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.5) (duration: 10m 05s)
02:34 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.4) (duration: 10m 33s)

2017-10-30

23:28 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: Update cirrussearch MLR model for enwiki (duration: 00m 50s)
23:22 ebernhardson@tin: Synchronized php-1.31.0-wmf.4/extensions/CirrusSearch/: Fix highlight tags inside section anchors of search results (duration: 01m 01s)
23:19 ebernhardson@tin: Synchronized php-1.31.0-wmf.5/extensions/CirrusSearch/: Fix highlight tags inside section anchors of search results (duration: 01m 03s)
22:03 eileen1: update process-control to revision is 2392ddf
21:55 awight@tin: Finished deploy [ores/deploy@a0f7d5c]: Fun with deployments (non-production) T179336 (duration: 02m 03s)
21:53 awight@tin: Started deploy [ores/deploy@a0f7d5c]: Fun with deployments (non-production) T179336
21:48 awight@tin: Finished deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production) (duration: 01m 42s)
21:47 chasemp: restart nova-fullstack to force a run
21:46 awight@tin: Started deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production)
21:36 eileen1: update CiviCRM from 25c7a33 to a34a9c6
21:02 urandom: T179083: Creating commons_T_parsoid_stash_wikitextf0PBY8UXqY8UuiDv.{meta,data} (30 sec delay)
20:56 urandom: T179083: Creating commons_T_parsoid_stash_htmlmXxc_uDhgnFQAdM8PPlH.{data,meta} (25 sec delay)
20:46 urandom: T179083: Creating enwiki_T_parsoid_stash_section2ACMDK1DRzacK9nUB3.{data,meta} (15sec delay)
20:36 urandom: T179083: Creating enwiki_T_parsoid_stash_dataWH8IDUS9SGI6LPpsJsLOQ.data
20:35 arlolra: Updated Parsoid to 292633c4 (T133334)
20:34 urandom: T179083: Creating enwiki_T_parsoid_stash_dataWH8IDUS9SGI6LPpsJsLOQ.meta
20:33 urandom: T179083: Creating enwiki_T_parsoid_stash_dataWH8IDUS9SGI6LPpsJsLOQ
20:28 urandom: T179083: Creating table enwiki_T_parsoid_stash_wikitextf0PBY8UXqY8UuiDv1.data
20:27 urandom: T179083: Creating table enwiki_T_parsoid_stash_wikitextf0PBY8UXqY8UuiDv1.meta
20:26 arlolra@tin: Finished deploy [parsoid/deploy@8a2c0ce]: Updating Parsoid to 292633c4 (duration: 07m 33s)
20:25 urandom: T179083: Creating keyspace enwiki_T_parsoid_stash_wikitextf0PBY8UXqY8UuiDv1
20:20 urandom: T179083: Creating table enwiki_T_parsoid_stash_htmlmXxc_uDhgnFQAdM8PPlH5.data
20:18 arlolra@tin: Started deploy [parsoid/deploy@8a2c0ce]: Updating Parsoid to 292633c4
20:14 urandom: T179083: Creating table enwiki_T_parsoid_stash_htmlmXxc_uDhgnFQAdM8PPlH5.meta
20:09 awight@tin: Finished deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production) (duration: 14m 11s)
20:08 urandom: T179083: Creating keyspace enwiki_T_parsoid_stash_htmlmXxc_uDhgnFQAdM8PPlH5
19:55 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create Appendix NS on Burmese Wiktionary T178545 (duration: 00m 55s)
19:55 awight@tin: Started deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production)
19:07 awight@tin: Finished deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production) (duration: 06m 16s)
19:01 awight@tin: Started deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production)
19:01 awight@tin: Finished deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production) (duration: 19m 53s)
18:41 awight@tin: Started deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production)
18:40 awight@tin: Started deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production)
18:22 awight@tin: Started deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production)
18:16 thcipriani@tin: Synchronized portals: SWAT: Bumping portals to master T128546 (duration: 00m 50s)
18:15 thcipriani@tin: Synchronized portals/prod/wikipedia.org/assets: SWAT: Bumping portals to master T128546 (duration: 00m 50s)
18:08 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add Timeless skin to wikitech T154371 (duration: 00m 51s)
17:08 madhuvishy: Revert dns switchover for c1 shards to c3 post labsdb1001 reboot T168584
17:07 gehel@tin: Finished deploy [wdqs/wdqs@fdb9100]: GUI update (duration: 02m 17s)
17:05 gehel@tin: Started deploy [wdqs/wdqs@fdb9100]: GUI update
16:45 oblivian@tin: Finished deploy [docker-pkg/deploy@576c80f]: New version with small fixes (duration: 00m 04s)
16:44 oblivian@tin: Started deploy [docker-pkg/deploy@576c80f]: New version with small fixes
15:20 XioNoX: re-enabling Zayo transit in eqiad
14:36 marostegui: Reboot labsdb1001 - T168584
14:30 marostegui: Stop replication and MySQL on labsdb1001 - T168584
14:09 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Revert "UBN! disbale ores for wikidata" (T179107) (duration: 00m 50s)
14:07 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Revert "UBN! disbale ores for wikidata" (T179107) (duration: 00m 49s)
14:00 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Change the af.wiktionary logo, part II (T178824) (duration: 00m 50s)
13:59 ladsgroup@tin: Synchronized static/images/project-logos: Change the af.wiktionary logo, part I (T178824) (duration: 00m 50s)
13:40 madhuvishy: Switched dns over for c1 shards to c3 in preparation of labsdb1001 reboot (Merged https://gerrit.wikimedia.org/r/#/c/386660/, Ran puppet on labservice1001|2)
13:21 marostegui: Set innodb_max_dirty_pages_pct = 10 on labsdb1001 so it powers off a bit faster - T168584
13:19 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable the Autopatrolled User Rights & Ext:SandboxLink on hiwikiversity (T179251 T179252) (duration: 00m 50s)
12:58 reedy@tin: Synchronized wmf-config/db-labs.php: beta labs stuffs (duration: 00m 50s)
12:49 oblivian@tin: Finished deploy [docker-pkg/deploy@e9b9146]: Proper deploy of first version (duration: 00m 07s)
12:49 oblivian@tin: Started deploy [docker-pkg/deploy@e9b9146]: Proper deploy of first version
12:02 hoo: Manually started dumpwikidatajson on snapshot1007 (cron run failed tonight)
12:02 oblivian@tin: Finished deploy [docker-pkg/deploy@549c157]: (no justification provided) (duration: 00m 11s)
12:01 oblivian@tin: Started deploy [docker-pkg/deploy@549c157]: (no justification provided)
12:00 oblivian@tin: Finished deploy [docker-pkg/deploy@549c157]: (no justification provided) (duration: 00m 19s)
11:59 oblivian@tin: Started deploy [docker-pkg/deploy@549c157]: (no justification provided)
11:57 oblivian@tin: Finished deploy [docker-pkg/deploy@549c157]: (no justification provided) (duration: 00m 11s)
11:57 oblivian@tin: Started deploy [docker-pkg/deploy@549c157]: (no justification provided)
11:55 oblivian@tin: Started deploy [docker-pkg/deploy@549c157]: Test deploy for virtualenv creation
11:48 hoo: Ran rebuildPropertyInfo for wikidatawiki and manually cleared the property cache (mc)
11:44 hoo@tin: Synchronized wmf-config/Wikibase.php: Re-add property for RDF mapping of external identifiers for Wikidata (T179156, T178180) (duration: 00m 49s)
11:33 hoo@tin: Synchronized wmf-config/Wikibase-production.php: Re-enable constraints check with SPARQL (T179156) (duration: 00m 50s)
11:24 marostegui: Compress table cebwiki.externallinks on db2092 - T178359
11:15 hoo@tin: rebuilt wikiversions.php and synchronized wikiversions files: Wikidatawiki back to wmf.5 (T179156)
10:58 mobrovac@tin: Finished deploy [restbase/deploy@2b5889b]: Double-process all summaries and include the parsoid no-op switch, take #gazillion (duration: 08m 33s)
10:52 akosiaris: poweroff copper T176957
10:50 mobrovac@tin: Started deploy [restbase/deploy@2b5889b]: Double-process all summaries and include the parsoid no-op switch, take #gazillion
10:31 marostegui: Stop MySQL on db2039 to copy its data to db2087.s6 - T178359
10:27 mobrovac@tin: Finished deploy [restbase/deploy@2b5889b]: Double-process all summaries and include the parsoid no-op switch (duration: 04m 41s)
10:22 mobrovac@tin: Started deploy [restbase/deploy@2b5889b]: Double-process all summaries and include the parsoid no-op switch
10:03 mobrovac@tin: Finished deploy [restbase/deploy@2b5889b]: Double-process all summaries and include the parsoid no-op switch (duration: 12m 12s)
09:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1097 original weight - T161088 (duration: 00m 52s)
09:50 mobrovac@tin: Started deploy [restbase/deploy@2b5889b]: Double-process all summaries and include the parsoid no-op switch
09:22 marostegui: Stop replication in sync on db2039 and db2046 to reimport tables
09:08 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1097 api traffic - T161088 (duration: 00m 50s)
09:08 marostegui: Drop wb_entity_per_page page from s3 and s5 - T177601
08:51 ema: cp4022: restart varnish-be for mbox lag
08:42 elukey: raised priority of refreshlink and htmlcacheupdate job execution on jobrunners (https://gerrit.wikimedia.org/r/#/c/386636/) - T173710
08:39 marostegui: Stop replication on db2039 to reimport and compress tables
08:38 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1097 with low weight - T161088 (duration: 00m 50s)
08:33 moritzm: installing wget security updates
08:32 gehel: rolling restart of wdqs for config reload - T175919
08:31 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2039 (duration: 00m 50s)
08:28 ariel@tin: Finished deploy [dumps/dumps@c204c72]: fix args initialization issue in getconfigvals script (duration: 00m 02s)
08:28 ariel@tin: Started deploy [dumps/dumps@c204c72]: fix args initialization issue in getconfigvals script
08:23 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2040 - T178359 (duration: 00m 50s)
08:11 marostegui: Stop MySQL on db2086 to transfer s7 to db2087 - T178359
06:51 marostegui: Stop MySQL on db1097 to clone db1103 - T161088
06:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1097 - T161088 (duration: 00m 50s)
06:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Move db1103 from s3 to s4 - T161088 (duration: 00m 49s)
06:40 marostegui: Stop MySQL on db1103 to reclone it - T161088
06:24 marostegui: Optimize dewiki.pagelinks and dewiki.templatelinks on db1063 (s5 master) - T174509
06:12 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2040 - T178359 (duration: 00m 51s)
06:08 marostegui: Stop MySQL on db2040 to populate s7 on db2086 - T178359
05:58 marostegui: Deploy alter table on db1075 (s3 primary master) - T174509
03:11 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Oct 30 03:11:37 UTC 2017 (duration 7m 11s)
03:04 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.5) (duration: 11m 38s)
02:35 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.4) (duration: 11m 18s)

2017-10-29

23:49 ema: powercycle cp4024
22:31 ariel@tin: Finished deploy [dumps/dumps@2aa2275]: fix keep setting to work with overrides (duration: 00m 02s)
22:31 ariel@tin: Started deploy [dumps/dumps@2aa2275]: fix keep setting to work with overrides
17:55 ariel@tin: Finished deploy [dumps/dumps@d8978ce]: add overrides section processing to config file (duration: 00m 04s)
17:55 ariel@tin: Started deploy [dumps/dumps@d8978ce]: add overrides section processing to config file
17:23 ariel@tin: Finished deploy [dumps/dumps@d426cf7]: batch 7z jobs, multistream job fixup (duration: 00m 02s)
17:23 ariel@tin: Started deploy [dumps/dumps@d426cf7]: batch 7z jobs, multistream job fixup
12:54 ema: cp4026: restart varnish-be for mbox lag

2017-10-28

21:03 bblack: cp1067 (current target cache): disabling the relatively-new VCL that sets do_stream=false if !CL on applayer fetches...
19:39 hoo@tin: Synchronized wmf-config/CommonSettings.php: Half the Flow -> Parsoid timeout (100s -> 50s) (T179156) (duration: 00m 51s)
19:39 bblack: backend restart on cp1065
18:39 bblack: restarting varnish backend on cp1053 to move the lag/503 issues to another box and buy more time to debug
18:28 bblack: cp4025 - restart backend for mailbox lag (upload@ulsfo, unrelated to text-cluster issues)
18:21 bblack: cp1053 - manual VCL change, backends appservers+api_appservers, reduce connect/firstbyte/betweenbytes timeoues from 5/180/60 to 3/20/10
16:51 elukey: restart varnish backend on cp1055 - mailbox lag + T179156
12:14 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1313.eqiad.wmnet
12:10 elukey: manually killed (SIGTERM) hhvm on mw1313 - high load, hhvm-dump-debug not responsive
12:01 elukey@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw1313.eqiad.wmnet
11:53 elukey: restart hhvm on mw1285 - hhvm-dump-debug in /tmp/hhvm.17700.bt
11:24 hoo@tin: Synchronized wmf-config/Wikibase-labs.php: Consistency sync (duration: 00m 50s)
10:52 volans: restarted pdfrender on scb1001, was stuck since 2d with AssertionError: display is not set!

2017-10-27

20:54 MaxSem: running migratePreferences.php on group2 wikis
19:09 hoo: Ran scap pull on mwdebug1001
18:20 awight@tin: Finished deploy [ores/deploy@185170f]: Test pip-9 scap trick on ores1002 (non-production) (duration: 02m 17s)
18:18 awight@tin: Started deploy [ores/deploy@185170f]: Test pip-9 scap trick on ores1002 (non-production)
17:54 hoo: Taking mwdebug1001 to do tests regarding T179156
16:38 gehel: re-enabling wdqs-updater
16:16 bblack: cp1054 varnish backend restarted (was 503s / bad-conns target of ongoing issues)
16:16 gehel: wdqs updater is now stopped for real
16:10 XioNoX: deactivating BGP sessions to Zayo in eqiad (flapping)
15:58 gehel: disabling wdqs updater on all nodes
15:50 hoo@tin: Synchronized wmf-config/Wikibase-production.php: Disable constraints check with SPARQL for now (T179156) (duration: 00m 50s)
15:48 marostegui: Compress InnoDB on db2039 (s6) - T178359
15:46 bblack: restart varnish-backend on cp4022 (upload@ulsfo) - mailbox
14:49 bblack: turn on cp4024 port on asw-ulsfo
13:52 bblack: reboot cp4021 to clean up oom messes
13:49 bblack: restarting nginx on cp4021, without NUMA memory constraints
12:10 marostegui: Optimize commonswiki.templatelinks on dbstore1001 - T162789
12:03 elukey: execute systemctl reset-failed kafka-mirror-main-eqiad_to_jumbo-eqiad.service on kafka-jumbo hosts (old unit not deployed anymore)
11:41 mutante: gerrit back - maintenance over
11:39 mutante: gerrit restart to apply gerrit:386793 is imminent
11:36 ema: cp4023: varnish-backend-restart for lag
11:06 mobrovac@tin: Finished deploy [citoid/deploy@ff63420]: Update dependencies (duration: 03m 18s)
11:04 marostegui: Optimize cebwiki.templatelinks on db1103 - T174509
11:03 mobrovac@tin: Started deploy [citoid/deploy@ff63420]: Update dependencies
08:56 marostegui: Optimize table commonswiki.recentchanges on dbstore1001 - T162789
08:11 marostegui: Drop redundant indexes from pagelinks and templatelinks on s3 wikis only for dbstore2001 and dbstore2002 - T174509
05:59 marostegui: Stop MySQL on db2084 to copy s5 to db2086 - T178359
05:55 marostegui: dbstore1001: convert itwiki.pagelinks to TokuDB - T162789
05:53 mutante: uploaded parsoid_0.8.0 to releases.wikimedia.org (for Subbu - T179134)
05:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1060 - T174509 (duration: 00m 50s)
03:02 bblack: cp1067, cp4026 - backend restarts, mailbox lag

2017-10-26

22:47 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: wikidata to wmf.4
22:39 bblack: restarting varnish-backend on cp1053 (mailbox lag from ongoing issues elsewhere?)
21:40 bblack: raising backend max_connections for api.svc.eqiad.wmnet + appservers.svc.eqiad.wmnet from 1K to 10K on cp1053.eqiad.wmnet (current funnel for the bulk of the 503s)
21:32 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Temporary disable remex html (T178632) (duration: 00m 50s)
21:30 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Temporary disable remex html (T178632) (duration: 00m 50s)
21:00 hoo: Fully revert all changes related to T178180
20:59 hoo@tin: Synchronized wmf-config/Wikibase.php: Revert "Add property for RDF mapping of external identifiers for Wikidata" (T178180) (duration: 00m 50s)
20:02 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: UBN! disbale ores for wikidata (T179107) (duration: 00m 50s)
20:00 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: UBN! disbale ores for wikidata (T179107) (duration: 00m 50s)
19:41 awight@tin: Finished deploy [ores/deploy@0adae70]: Increase extractor wikidata API timeout to 15s, T179107 (duration: 07m 25s)
19:36 aaron@tin: Started restart [jobrunner/jobrunner@a20d043]: (no justification provided)
19:34 awight@tin: Started deploy [ores/deploy@0adae70]: Increase extractor wikidata API timeout to 15s, T179107
19:33 awight@tin: Finished deploy [ores/deploy@0adae70]: Increase extractor wikidata API timeout to 15s, T179107 (duration: 00m 10s)
19:33 awight@tin: Started deploy [ores/deploy@0adae70]: Increase extractor wikidata API timeout to 15s, T179107
19:27 demon@tin: Synchronized php-1.31.0-wmf.5/includes/libs/filebackend/SwiftFileBackend.php: Better error grouping (duration: 00m 50s)
19:14 ladsgroup@tin: Synchronized php-1.31.0-wmf.5/extensions/WikibaseQualityConstraints/includes/ConstraintCheck/DelegatingConstraintChecker.php: Fix sorting of NullResults (T179038) (duration: 00m 49s)
19:12 ladsgroup@tin: Synchronized php-1.31.0-wmf.5/extensions/WikibaseQualityConstraints/tests/phpunit/DelegatingConstraintCheckerTest.php: Fix sorting of NullResults (T179038) (duration: 00m 50s)
18:52 ladsgroup@tin: Synchronized php-1.31.0-wmf.5/extensions/Wikidata/extensions/Constraints/tests/phpunit/DelegatingConstraintCheckerTest.php: Fix sorting of NullResults (T179038) (duration: 00m 49s)
18:51 ladsgroup@tin: Synchronized php-1.31.0-wmf.5/extensions/Wikidata/extensions/Constraints/includes/ConstraintCheck/DelegatingConstraintChecker.php: Fix sorting of NullResults (T179038) (duration: 01m 04s)
18:41 bblack: eqiad cp servers: rolling quick depool -> repool around ethtool parameter changes for -lro,-pause
18:30 urandom: Dropping and recreating keyspace enwiki_T_page__summary (restbase-ng cluster)
18:27 bblack: esams cp servers: rolling quick depool -> repool around ethtool parameter changes for -lro,-pause
18:26 urandom: T179105: Altering "enwiki_T_page__summary".data to use LZ4Compressor
17:59 bblack: codfw cp servers: rolling quick depool -> repool around ethtool parameter changes for -lro,-pause
17:57 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4024.ulsfo.wmnet
17:49 bblack: ulsfo cp servers: rolling quick depool -> repool around ethtool parameter changes for -lro,-pause
17:47 awight@tin: Finished deploy [ores/deploy@f5deb7f]: Revscoring 2 on ores1002 (non-production) (duration: 01m 45s)
17:45 awight@tin: Started deploy [ores/deploy@f5deb7f]: Revscoring 2 on ores1002 (non-production)
17:36 arlolra: Updated Parsoid to 8e99708a (T176728)
17:30 arlolra@tin: Finished deploy [parsoid/deploy@4882c59]: Updating Parsoid to 8e99708a (duration: 08m 40s)
17:21 arlolra@tin: Started deploy [parsoid/deploy@4882c59]: Updating Parsoid to 8e99708a
17:17 awight@tin: Finished deploy [ores/deploy@971be22]: Rolling back scb1002, T175180 T178441 (duration: 00m 32s)
17:16 awight@tin: Started deploy [ores/deploy@971be22]: Rolling back scb1002, T175180 T178441
17:14 awight@tin: Finished deploy [ores/deploy@971be22]: ORES w/ revscoring 2 and Celery 4, T175180 T178441 (duration: 03m 20s)
17:11 awight@tin: Started deploy [ores/deploy@971be22]: ORES w/ revscoring 2 and Celery 4, T175180 T178441
17:09 marostegui: Optimize ruwiki.recentchanges on dbstore1001 - T162789
17:08 dcausse: [logstash] reimporting today logs
17:08 dcausse: [logstash] deleting today logs to reapply mapping template
16:58 ottomata: now mirroring main Kafka cluster topics to jumbo Kafka cluster, with MirrorMaker instances running on analytics-eqiad broker nodes. https://phabricator.wikimedia.org/T177216
16:44 awight@tin: Finished deploy [ores/deploy@971be22]: Push ORES w/ Celery 4 support to new cluster (try to rebuild venv), T178441 (duration: 01m 02s)
16:43 awight@tin: Started deploy [ores/deploy@971be22]: Push ORES w/ Celery 4 support to new cluster (try to rebuild venv), T178441
16:33 awight@tin: Finished deploy [ores/deploy@971be22]: Push ORES w/ Celery 4 support to new cluster (take 3), T178441 (duration: 15m 33s)
16:21 urandom: restarted cassandra, restbase2001-b
16:18 awight@tin: Started deploy [ores/deploy@971be22]: Push ORES w/ Celery 4 support to new cluster (take 3), T178441
16:15 urandom: truncating cassandra hints in restbase-ng cluster
16:12 mobrovac@tin: Finished deploy [restbase/deploy@860cbfe]: Revert all summaries, back to all but WP (duration: 07m 57s)
16:04 mobrovac@tin: Started deploy [restbase/deploy@860cbfe]: Revert all summaries, back to all but WP
16:04 mobrovac@tin: Started deploy [restbase/deploy@860cbfe]: Revert all summaries, back to all but WP
15:50 mobrovac@tin: Finished deploy [restbase/deploy@522321a]: Double-process all summaries (duration: 09m 09s)
15:45 urandom: restarting cassandra, restbase2001-b
15:41 andrewbogott: upgrading dnsmasq on labnet1001 and labnet1002
15:41 mobrovac@tin: Started deploy [restbase/deploy@522321a]: Double-process all summaries
15:36 dcausse: reindexing logs in logstash
15:34 dcausse: deleting and reimporting logs in logstash (today data, might lose some logs in the process)
14:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1101 - T178383 (duration: 00m 49s)
14:04 zeljkof: EU SWAT finished
14:03 zfilipin@tin: Synchronized php-1.31.0-wmf.5/extensions/Wikibase: SWAT: Make search for titles be always uppercase (T179045) (duration: 01m 36s)
13:52 zfilipin@tin: Synchronized php-1.31.0-wmf.5/extensions/Wikidata: SWAT: Make search for titles be always uppercase (T179045) (duration: 02m 10s)
13:46 Amir1: ladsgroup@terbium:~$ mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildPropertyInfo.php --wiki=wikidatawiki --rebuild-all --force (T178180)
13:41 godog: delete old CF cassandra metrics - T173436
13:32 godog: force delete old graphite cassandra metrics - T179057
13:28 moritzm: installing icu security update on trusty
13:28 zfilipin@tin: Synchronized wmf-config/Wikibase.php: SWAT: Add property for RDF mapping of external identifiers for Wikidata (T178180) (duration: 00m 50s)
13:21 marostegui: Compress InnoDB on db2040 - T178359
13:21 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable signature button in Projekt namespace on sewikimedia (T175363) (duration: 00m 50s)
13:13 zfilipin@tin: Synchronized wmf-config/InterwikiSortOrders.php: SWAT: Fix interwiki sort order for Northern Sami (T178965) (duration: 00m 50s)
12:57 marostegui@tin: Synchronized wmf-config/db-codfw.php: Reorganize future rc multi-instance hosts - T178359 (duration: 00m 50s)
12:30 bblack: powercycle cp4024 (depooled)
12:29 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4024.ulsfo.wmnet
10:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Clean up some comments in s3 (duration: 00m 50s)
09:01 moritzm: upgrade script runners to wikidiff2 1.5.1
08:58 ema: cache_text/upload varnish-be: set rutime parameter timeout_idle to 120s
08:34 moritzm: uploaded php-wikidiff2/hhvm-wikidiff2 1.5.1 for stretch-wikimedia
08:22 godog: add 100G to graphite1003's /var/lib/carbon
08:14 moritzm: upgrade deployment servers to wikidiff2 1.5.1
07:57 marostegui: Drop databases in s1 and s2 from db1047 and unconfigure replication - T177405
07:43 marostegui: Stop MySQL on db1047 to copy data over db1108 - T177405 T156844
07:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1077 - T164488 (duration: 00m 50s)
07:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1077 - T164488 (duration: 00m 50s)
07:12 marostegui: Stop replication in sync on db1103 and db1077 to fix data drifts - T164488
07:10 moritzm: upgrade mw1319-mw1328 to wikidiff2 1.5.1
06:57 marostegui: Stop MySQL on db2088 and db2084 to copy s2 and s4 to db2091 - T178359
06:57 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2038 (duration: 00m 50s)
06:55 marostegui: Optimize pagelinks and templatelinks on db1095 s1 - T174509
06:51 marostegui: Optimize recentchanges on s4 and s6 on labsdb1010 - T177772
06:15 moritzm: upgrade mw1299-mw1311 to wikidiff2 1.5.1
05:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1060 - T174509 (duration: 00m 50s)
05:14 marostegui: Optimize recentchanges on s4 and s6 on labsdb1009 - T177772
05:12 marostegui: Optimize pagelinks and templatelinks on db1060 - T174509
03:01 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Oct 26 03:01:19 UTC 2017 (duration 6m 48s)
02:54 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.5) (duration: 09m 30s)
02:31 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.4) (duration: 10m 34s)

2017-10-25

23:48 legoktm@tin: Synchronized php-1.31.0-wmf.5/extensions/VisualEditor/modules/ve-mw/ui/styles/contextitems/ve.ui.MWInternalLinkContextItem.css: MWInternalLinkContextItem: increase specificity to override OOUI changes - T178933 (duration: 00m 50s)
23:39 legoktm@tin: Synchronized wmf-config/abusefilter.php: Change threshold for slow AbuseFilter logging to 800ms - T179039 (duration: 00m 50s)
22:42 MaxSem: running migratePreferences.php on group1 wikis
22:01 mobrovac@tin: Finished deploy [restbase/deploy@860cbfe]: Revert double-processing of all summaries to all except WPs, take #2 (duration: 09m 09s)
21:52 mobrovac@tin: Started deploy [restbase/deploy@860cbfe]: Revert double-processing of all summaries to all except WPs, take #2
21:48 mobrovac@tin: Started deploy [restbase/deploy@60ce036]: Revert double-processing of all summaries to all except WPs
21:33 bblack: experimental - disabled LRO on lvs4001+lvs4002
20:34 hasharAway: restarted zuul to flush a long queue in experimental pipeline (no other changes enqueued)
20:21 demon@tin: Synchronized php-1.31.0-wmf.5/extensions/ProofreadPage/: unbreak multiline text input (duration: 00m 53s)
19:28 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.5
18:45 bblack: s/1\.13\.16/1.13.6/
18:45 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable fine grained usage tracking on statement usage wikis T172914 (duration: 00m 50s)
18:44 bblack: reprepro: uploaded nginx-1.13.16-2+wmf1 packages to stretch and jessie
18:36 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Make disabled usage aspect use the new config T172914 (duration: 00m 50s)
18:33 niharika29@tin: Synchronized php-1.31.0-wmf.4/extensions/Wikidata: Allow specifying a replacement usage type in disabledUsageAspects T178153 (duration: 02m 10s)
18:25 niharika29@tin: Synchronized wmf-config/Wikibase.php: Make using CirrusSearch engine default for wbsearchentities T175741 (duration: 00m 48s)
18:23 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Add AbuseFilterSlow channel to monolog T178853 (duration: 00m 50s)
18:20 niharika29@tin: Synchronized php-1.31.0-wmf.5/extensions/ORES/: Update ApiHooks to conform with v3 responses T178962 (duration: 00m 50s)
18:19 niharika29@tin: Synchronized php-1.31.0-wmf.4/extensions/ORES/: Update ApiHooks to conform with v3 responses T178962 (duration: 00m 51s)
18:11 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable Special:EmailUser User Prohibit on All Wikis [mediawiki-config] - https://gerrit.wikimedia.org/r/386231 (duration: 00m 51s)
17:17 moritzm: upgrade mw2224-mw2242 to wikidiff2 1.5.1
16:59 moritzm: upgrade mw2190-mw2199 to wikidiff2 1.5.1
16:38 no_justification: gerrit: purging last remnants of debian package and running puppet. service may briefly restart
16:05 moritzm: upgrade mw2180-mw2189 to wikidiff2 1.5.1
16:03 moritzm: copied arcconf, hp-health, hpacucli, hpssa, hpssacli, hpssaducli, lsiutil, megacli, megaclisas-status to stretch-wikimedia/thirdparty/hwraid (thirdparty will be dropped at a later point)
15:39 marostegui: Optimize pagelinks and templatelinks on labsdb1011 - T174509
15:31 marostegui: Power off db1101 for HW maintenance - T178383
15:21 godog: bounce rsyslog on wezen
15:09 mattflaschen@tin: Synchronized wmf-config/InitialiseSettings.php: T177836: Deploy Compact Language Links on the German Wikipedia (duration: 00m 49s)
14:52 gehel: applying new template to elasticsearch / logstash - T178530
14:51 ebernhardson: update logstash template on logstash elsaticsearch cluster for T178530
14:47 mattflaschen@tin: Synchronized wmf-config/InitialiseSettings.php: T166759: Set $wgAutoloadAttemptLowercase to false (duration: 00m 50s)
14:37 mattflaschen@tin: Synchronized php-1.31.0-wmf.5/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T177956: [cirrus] Turn off recall A/B test on enwiki (duration: 00m 49s)
14:36 mattflaschen@tin: Synchronized php-1.31.0-wmf.4/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T177956: [cirrus] Turn off recall A/B test on enwiki (duration: 00m 49s)
14:25 moritzm: moved jmxtrans and prometheus-jmx-exporter from thirdparty to main as part of repo reorg for stretch-wikimedia
14:11 matt_flaschen: 'Deployed T178933: MWInternalLinkContextItem: increase specificity to override OOUI changes'
14:10 mattflaschen@tin: Synchronized php-1.31.0-wmf.4/extensions/VisualEditor/modules/ve-mw/ui/styles/contextitems/ve.ui.MWInternalLinkContextItem.css: (no justification provided) (duration: 00m 50s)
13:36 mattflaschen@tin: Synchronized wmf-config/InitialiseSettings.php: T168886: Updating wikis with consolidate editing feedback (duration: 00m 51s)
13:30 godog: deploy thumbor 1.7 - T178974
13:30 elukey: restart yarn nodemanager and hdfs datanode on analytics1030 to apply new JVM settings - T178876
13:15 chasemp: merge arturo's key as part of ops
13:04 matt_flaschen: About to run ULSCompactLinksDisablePref.php: Setting user preferences for Compact Language Links on dewiki, before removing Beta Feature and changing to regular preference. T177836
12:36 moritzm: upgrade mw2163-mw2179 to wikidiff2 1.5.1
12:17 mobrovac@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Activate warning logging for JobExecutor (duration: 00m 50s)
12:16 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: Activate warning logging for JobExecutor (duration: 00m 50s)
12:12 moritzm: upgrade mw1266-mw1275 to wikidiff2 1.5.1
11:30 moritzm: upgrade mw1280-mw1290, mw1312-mw1318 to wikidiff2 1.5.1
11:23 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2038 - T178359 (duration: 00m 49s)
10:49 mobrovac@tin: Finished scap: wmf-config/jobqueue-labs.php beta-only change for CP4JQ (duration: 04m 43s)
10:45 mobrovac@tin: Started scap: wmf-config/jobqueue-labs.php beta-only change for CP4JQ
10:41 mobrovac@tin: Finished deploy [restbase/deploy@2ae5b0c]: Remove /title/{title}/ and double-process all summaries - T158100 (duration: 09m 41s)
10:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1077 - T164488 (duration: 00m 50s)
10:32 moritzm: upgrade remaining video scalers in eqiad to wikidiff2 1.5.1
10:31 mobrovac@tin: Started deploy [restbase/deploy@2ae5b0c]: Remove /title/{title}/ and double-process all summaries - T158100
09:31 moritzm: upgrade mw1221-mw1235 to wikidiff2 (API servers)
09:07 gehel: rolling restart of all wdqs nodes for GC config change - T175919
08:12 moritzm: upgrade mw1238-mw1258 to wikidiff2
08:02 _joe_: decommissioning mw1168/69 as videoscalers, stopped jobrunner/jobchron, will stop hhvm/nginx/apache once current processing is done
07:42 gehel@tin: Finished deploy [wdqs/wdqs@0bb2b5c]: wdqs-updater upgrade for jolokia - T167842 (duration: 01m 44s)
07:40 gehel@tin: Started deploy [wdqs/wdqs@0bb2b5c]: wdqs-updater upgrade for jolokia - T167842
07:05 marostegui: Optimize pagelinks and templatelinks on db1021 - T174509
06:15 marostegui: Stop MySQL on db2038 to copy its data to db2084 - T178359
06:08 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2035 - T178359 (duration: 00m 50s)
06:02 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1065 - T174509 (duration: 00m 50s)
05:44 marostegui: Stop replication in sync on db1103 and db1077 to checksum data - T164488
05:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1077 - T164488 (duration: 01m 00s)
03:22 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Oct 25 03:22:51 UTC 2017 (duration 7m 20s)
03:15 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.5) (duration: 16m 46s)
02:35 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.4) (duration: 08m 59s)

2017-10-24

22:54 ejegg: updated civicrm from from 0546446 to 25c7a33
21:24 ejegg: updated civicrm from 18edce1 to 0546446
21:21 mepps: updated payments-wiki from 1011434 to b88b805
20:47 arlolra@tin: Finished deploy [parsoid/deploy@UNKNOWN]: (no justification provided) (duration: 00m 13s)
20:47 arlolra@tin: Started deploy [parsoid/deploy@UNKNOWN]: (no justification provided)
20:29 mutante: restarting gerrit to make gerrit-ssh listen on all IPs again
19:49 arlolra@tin: Finished deploy [parsoid/deploy@UNKNOWN]: Enabling logging for skwiki posts (duration: 00m 35s)
19:48 arlolra@tin: Started deploy [parsoid/deploy@UNKNOWN]: Enabling logging for skwiki posts
19:46 mutante: restarting gerrit on cobalt to apply change 354074
19:13 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.5
18:57 awight@tin: Finished deploy [ores/deploy@fb55ab8]: Rollback ORES to Revscoring 1.0, T175180 (duration: 10m 13s)
18:47 awight@tin: Started deploy [ores/deploy@fb55ab8]: Rollback ORES to Revscoring 1.0, T175180
18:47 ejegg: updated civicrm from b9d2067 to 18edce1
18:26 awight@tin: Finished deploy [ores/deploy@fb55ab8]: Update ORES to revscoring 2.x, T175180 (take 2) (duration: 10m 03s)
18:16 awight@tin: Started deploy [ores/deploy@fb55ab8]: Update ORES to revscoring 2.x, T175180 (take 2)
18:05 arlolra: Updated Parsoid to a3be9cfc (T96923, T177784, T176568, T25467)
17:57 arlolra@tin: Finished deploy [parsoid/deploy@af55c4b]: Updating Parsoid to a3be9cfc (duration: 09m 16s)
17:48 arlolra@tin: Started deploy [parsoid/deploy@af55c4b]: Updating Parsoid to a3be9cfc
17:28 demon@tin: Finished scap: bootstrap wmf.5 (duration: 48m 34s)
17:18 urandom: T178845: Rolling Cassandra restart, eqiad
17:07 awight@tin: Started deploy [ores/deploy@fb55ab8]: Update ORES to revscoring 2.x, T175180
16:50 herron: upgrading puppet packages on codfw puppetmaster T177254
16:39 demon@tin: Started scap: bootstrap wmf.5
16:10 demon@tin: Pruned MediaWiki: 1.31.0-wmf.3 [keeping static files] (duration: 01m 20s)
15:58 elukey: drop MediaViewer_10867062_15423246 and MobileWebUIClickTracking_10742159_15423246 from the log database on db1046 (archived on hadoop) - T168303
15:53 urandom: T178845: Rolling Cassandra restart, codfw, row b
15:48 elukey: drop table Edit_13457736_15423246 from the log database (Eventlogging) on db104[6,7], dbstore1002
15:39 elukey: set net.netfilter.nf_conntrack_tcp_timeout_time_wait=65 to mw[1308-1311] - T136094
15:09 gehel: restarting blazegraph on all wdqs nodes for GC config change - T175919
14:48 _joe_: also stopping HHVM, nginx, apache2 , and disabling all the services
14:46 _joe_: stopping the jobrunner service on mw1161-67
14:39 marostegui: Optimize pagelinks templatelinks and recentchanges on db1030 - T174509 https://phabricator.wikimedia.org/T177772
14:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1065 - T174509 (duration: 00m 45s)
14:33 marostegui: Optimize ores_classification on enwiki db1065 - T159753
14:32 marostegui: Optimize pagelinks and templatelinks on db1065 - T174509
14:24 gehel: restarting blazegraph on wdqs2001 for GC config change - T175919
14:20 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1106 weight (duration: 00m 45s)
13:42 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2035 - T178359 (duration: 00m 45s)
13:41 marostegui: Stop MySQL on db2035 to copy its data to db2088 - T178359
13:39 zeljkof: EU SWAT finished
13:39 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add $wgNamespaceRobotPolicies Config for hewikisource (T178775) (duration: 00m 46s)
10:50 dcausse: cirrus/elasticsearch: reindexing of 167 small wikis done (T177871)
09:22 marostegui: Stop s1 on db2092 to copy its data to db2088 - T178359
08:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1082 original weight - T178460 (duration: 00m 45s)
07:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1082 weight - T178460 (duration: 00m 45s)
07:46 marostegui: Stop replication in sync on db1103 and db2018 to fix data drifts - T164488
07:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1078 - T164488 (duration: 00m 46s)
07:28 marostegui: Stop replication in sync on db1078 and db1103 to fix data drifts - https://phabricator.wikimedia.org/T164488
06:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1078 - T164488 (duration: 00m 45s)
06:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1082 with low weight - T178460 (duration: 00m 47s)
02:32 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Oct 24 02:32:25 UTC 2017 (duration 6m 33s)
02:25 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.4) (duration: 08m 13s)

2017-10-23

23:09 demon@tin: Synchronized wmf-config/throttle.php: Working Class Movement Library (Salford) throttle rule (duration: 00m 46s)
22:59 bblack: cp4024: varnish-backend-restart for lag
21:40 XioNoX: increasing MTU on Zayo transit interfaces
20:34 arlolra: Updated Parsoid to 1cefef12 (T178217)
20:25 arlolra@tin: Finished deploy [parsoid/deploy@0759bd1]: Updating Parsoid to 1cefef12 (duration: 11m 14s)
20:14 arlolra@tin: Started deploy [parsoid/deploy@0759bd1]: Updating Parsoid to 1cefef12
19:37 bsitzmann@tin: Finished deploy [mobileapps/deploy@a654ce1]: Update mobileapps to 3628105 (T178828) (duration: 06m 59s)
19:30 bsitzmann@tin: Started deploy [mobileapps/deploy@a654ce1]: Update mobileapps to 3628105 (T178828)
19:00 joal@tin: Finished deploy [analytics/aqs/deploy@ad22e0c]: Upgrade AQS node modules to restbase-latests - Redeploy after config fix (duration: 02m 34s)
18:58 joal@tin: Started deploy [analytics/aqs/deploy@ad22e0c]: Upgrade AQS node modules to restbase-latests - Redeploy after config fix
18:58 joal@tin: Finished deploy [analytics/aqs/deploy@ad22e0c]: Upgrade AQS node modules to restbase-latests - Redeploy after config fix (duration: 05m 07s)
18:52 joal@tin: Started deploy [analytics/aqs/deploy@ad22e0c]: Upgrade AQS node modules to restbase-latests - Redeploy after config fix
18:52 joal@tin: Finished deploy [analytics/aqs/deploy@ad22e0c]: Upgrade AQS node modules to restbase-latests (duration: 321m 51s)
17:56 gehel@tin: Finished deploy [wdqs/wdqs@b84592f]: WDQS GUI update (duration: 01m 58s)
17:54 gehel@tin: Started deploy [wdqs/wdqs@b84592f]: WDQS GUI update
17:48 XenoRyet: updated smashpig from d056a13 to 5262d53
17:32 demon@tin: Synchronized scap/plugins/prep.py: Fixes and improvements of various sorts (duration: 00m 45s)
17:26 gehel@tin: Finished deploy [wdqs/wdqs@4626acb]: deploying latest WDQS version, blazegraph + GUI updates (duration: 02m 07s)
17:24 gehel@tin: Started deploy [wdqs/wdqs@4626acb]: deploying latest WDQS version, blazegraph + GUI updates
16:53 demon@tin: Pruned MediaWiki: 1.30.0-wmf.19 (duration: 03m 09s)
15:59 elukey: forced BBU learn cycle on db1046 - T166141
15:00 herron: wiped resolver cache of records puppet.cofdw.wmnet and puppet.ulsfo.wmnet (T177254)
14:46 herron: depooling codfw puppetmaster (via dns)
14:31 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1078 - T164488 (duration: 00m 46s)
14:05 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2037 - T178359 (duration: 00m 46s)
13:48 marostegui: Stop mysql and poweroff db1082 for HW maintenance - T178460
13:30 joal@tin: Started deploy [analytics/aqs/deploy@ad22e0c]: Upgrade AQS node modules to restbase-latests
13:19 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Add aleph500.biblacad.ro to $wgCopyUploadsDomains - T178753 (duration: 00m 47s)
13:19 gehel: rolling restart of wdqs for GC tuning - T175919
13:18 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Remove hooserv.net from wgCopyUploadsDomains (duration: 00m 47s)
13:08 zeljkof: EU SWAT finished, no deploys
13:00 bblack@neodymium: conftool action : set/pooled=no; selector: dc=ulsfo,cluster=cache_text,name=cp4018.ulsfo.wmnet
12:59 bblack@neodymium: conftool action : set/pooled=yes; selector: dc=ulsfo,cluster=cache_text,name=cp4032.ulsfo.wmnet
12:53 bblack@neodymium: conftool action : set/pooled=no; selector: dc=ulsfo,cluster=cache_text,name=cp4017.ulsfo.wmnet
12:47 bblack@neodymium: conftool action : set/pooled=yes; selector: dc=ulsfo,cluster=cache_text,name=cp4031.ulsfo.wmnet
12:46 moritzm: upgrading hhvm-wikdiff2 on remaining job runners in codfw
12:39 moritzm: upgrading hhvm-wikdiff2 on remaining video scalers in codfw
12:32 bblack@neodymium: conftool action : set/pooled=no; selector: dc=ulsfo,cluster=cache_text,name=cp4010.ulsfo.wmnet
12:29 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Remove hooserv.net from wgCopyUploadsDomains (duration: 00m 46s)
12:26 moritzm: upgrading hhvm-wikdiff2 on mw2153-mw2162 (job runners)
12:21 moritzm: upgrading hhvm-wikdiff2 on mw2254-mw2258 (app servers)
12:20 bblack@neodymium: conftool action : set/pooled=yes; selector: dc=ulsfo,cluster=cache_text,name=cp4030.ulsfo.wmnet
12:15 bblack@neodymium: conftool action : set/pooled=no; selector: dc=ulsfo,cluster=cache_text,name=cp4009.ulsfo.wmnet
12:12 bblack@neodymium: conftool action : set/pooled=yes; selector: dc=ulsfo,cluster=cache_text,name=cp4029.ulsfo.wmnet
12:03 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable description usage tracking on further wikis (T178515) (duration: 00m 47s)
11:11 moritzm: upgrading hhvm-wikdiff2 on mw2097-mw2114
11:10 marostegui: Compress innodb on db2038 and db2084 - T178359
10:56 moritzm: upgrading hhvm-wikdiff2 on remaining API servers in codfw
10:46 moritzm: upgrading hhvm-wikdiff2 on mw1161-mw1167 (job runners)
10:28 moritzm: upgrading hhvm-wikdiff2 on mw1180-mw1188 (app servers)
10:28 marostegui: Remove /srv from db1015 as it has been stopped for weeks now and will be decommissioned (and it is alerting low on disk space) - T173570
10:09 dcausse: elasticsearch/cirrus reindexing 167 wikis from terbium (T177871)
10:08 gehel: restart tilerator / karthotherian for config change on all maps clusters - T160215
10:01 ema: restart varnish-be on cp4021 (mbox lag)
09:58 moritzm: upgrading hhvm-wikidiff2 on mw1293-mw1298 (scalers)
09:09 moritzm: upgrading hhvm-wikidiff2 on mw1189-mw1208 (API servers)
08:09 marostegui: Rename wb_entity_per_page table on s5 db1092 and s3 db1077 - T177601
08:07 moritzm: upgrading hhvm-wikdiff2 on mw1209-mw1220 (app servers)
07:23 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2037 and db2038 - T178359 (duration: 00m 46s)
06:56 moritzm: installing expat security updates on trusty
06:51 marostegui: Stop replication in sync on db1078 and db1103 to checksum data - T164488
06:50 moritzm: installing poppler security updates on trusty
06:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1078 - T164488 (duration: 00m 47s)
06:30 marostegui: Stop replication in sync on db1103 and db2018 to fix data drifts - T164488
06:09 marostegui: Stop replication on db2092 to re-import and compress linter and watchlist table
02:41 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Oct 23 02:41:04 UTC 2017 (duration 6m 55s)
02:34 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.4) (duration: 08m 45s)

2017-10-21

08:16 hashar: Mass code-review+2 changes made by LibraryUpdater that already had Code-Review:+2 and NOT verified=-1 ( ping legoktm )
08:08 hashar: Stopping Zuul to flush its queue

2017-10-20

23:46 demon@tin: Finished deploy [gerrit/gerrit@21b7332]: pushing deleteproject plugin (duration: 00m 09s)
23:46 demon@tin: Started deploy [gerrit/gerrit@21b7332]: pushing deleteproject plugin
19:19 hasharAway: restarting Jenkins - T178608
18:52 bblack: vhtcpd upgrade + queue delay puppetization deploy ( https://gerrit.wikimedia.org/r/385415 ) done on cp* fleet - T133821
18:28 bblack: upgrading vhtcpd to 0.1.1-1 on the cp* fleet
17:38 mutante: removing Apache and HHVM Ganglia stats from all appservers, part of retiring Ganglia (T177225) - some transient puppet issues while purging package and remnants - use grafana dashboard at https://grafana.wikimedia.org/dashboard/db/prometheus-apache-hhvm-dc-stats
16:39 aaron@tin: Synchronized php-1.31.0-wmf.4/extensions/TwoColConflict: 5c3224980fe (duration: 00m 46s)
16:37 aaron@tin: Synchronized php-1.31.0-wmf.4/extensions/OATHAuth: c16a94e17b9981 (duration: 00m 47s)
16:05 krinkle@tin: Synchronized php-1.31.0-wmf.4/extensions/WikimediaMaintenance/getJobQueueLengths.php: I8737795e6 (duration: 00m 47s)
16:05 XioNoX: removing "policy-options policy-statement LVS_import term secondary" on cr routers
14:39 cmjohnson1: updating power firmware analytics1037
10:14 moritzm: upgrading hhvm-wikidiff2 to 1.5.1 on mw2150/mw2151/mw2244/mw2245
09:17 moritzm: upgrading hhvm-wikidiff2 to 1.5.1 on mw2200-mw2223
08:05 Amir1: mwscript createAndPromote.php --wiki=techconductwiki --sysop --bureaucrat 'Ladsgroup' --force
07:39 jynus: stopping db2033, too
07:36 jynus: stopping dbstore1002 and dbstore2002 x1 replication in sync
07:18 _joe_: uploading updated versions of docker base images (wikimedia-jessie, wikimedia-stretch) to docker-registry.wikimedia.org
06:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1078 - T164488 (duration: 00m 46s)
06:17 marostegui: Stop replication in sync on db1103 and db1078 - T164488
06:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1078 - T164488 (duration: 00m 45s)
06:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1072 - T164488 (duration: 00m 46s)
05:52 marostegui: Stop replication in sync on db1103 and db1072 - T164488
05:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1051 - T174509 (duration: 00m 47s)
02:19 bblack: upgrading vhtcpd to 0.1.1-1 on: cp1008, cp3030, cp3034, cp4029-32
02:17 bblack: uploaded vhtcpd-0.1.1-1 to reprepro

2017-10-19

22:21 addshore: stop running extra change dispatchers on terbium for wikidatawiki
21:58 addshore: running extra change dispatchers on terbium for wikidatawiki for a short while
21:24 andrewbogott: restarting nginx on nitrogen
21:22 andrewbogott: restarting puppetdb on nitrogen
20:00 mforns@tin: Finished deploy [analytics/refinery@0c9ba04]: deploying refinery to fix banner activity false alarms (duration: 00m 14s)
19:59 mforns@tin: Started deploy [analytics/refinery@0c9ba04]: deploying refinery to fix banner activity false alarms
19:53 bblack: reboot cp4029-32 (initial reboot post-puppetization)
19:12 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.4
18:36 thcipriani@tin: Synchronized portals: SWAT: Bumping portals to master T128546 (duration: 00m 51s)
18:35 thcipriani@tin: Synchronized portals/prod/wikipedia.org/assets: SWAT: Bumping portals to master T128546 (duration: 00m 50s)
18:28 thcipriani@tin: Synchronized wmf-config/abusefilter.php: SWAT: Enable $wgAbuseFilterProfile on ptwiki T177641 (duration: 00m 50s)
18:20 thcipriani@tin: Synchronized php-1.31.0-wmf.4/extensions/Echo/modules/styles/mw.echo.ui.PageNotificationsOptionWidget.less: SWAT: mw.echo.ui.PageNotificationsOptionWidget: Fix CSS after changes in OOjs UI T178439 (duration: 00m 51s)
16:50 hashar: contint1001: installed python3 (pending https://gerrit.wikimedia.org/r/385210 )
15:14 marostegui: Stop replication in sync on db1103 and db1072 - https://phabricator.wikimedia.org/T164488
15:02 mforns@tin: Finished deploy [analytics/refinery@0c9ba04]: (no justification provided) (duration: 00m 07s)
15:02 mforns@tin: Started deploy [analytics/refinery@0c9ba04]: (no justification provided)
14:36 mforns@tin: Finished deploy [analytics/refinery@0c9ba04]: (no justification provided) (duration: 03m 29s)
14:33 mforns@tin: Started deploy [analytics/refinery@0c9ba04]: (no justification provided)
13:20 moritzm: upgrading mw2120-mw2147 to wikidiff2 1.5.1
12:35 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2034 and db2036 - T178359 (duration: 00m 50s)
11:40 akosiaris: T177196 upload prometheus-postgres-exporter_0.2.0+ds-2 to apt.wikimedia.org/stretch-wikimedia/main and copied over to apt.wikimedia.org/jessie-wikimedia/main
11:10 moritzm: upgrading mw1276-mw1279 (canary servers) to wikidiff2 1.5.1
10:14 godog: upgrade prometheus-hhvm-exporter to 0.4-1
10:07 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: Electron hanging - T174916
09:54 ppchelko@tin: Finished deploy [restbase/deploy@2001a66]: Enable summary on Cass 3 for all but wikipedia (duration: 07m 32s)
09:50 ariel@tin: Finished deploy [dumps/dumps@2b9326b]: improved job batches for rev content 7z recompression (duration: 00m 02s)
09:50 ariel@tin: Started deploy [dumps/dumps@2b9326b]: improved job batches for rev content 7z recompression
09:48 moritzm: installing pyjwt security updates
09:47 ppchelko@tin: Started deploy [restbase/deploy@2001a66]: Enable summary on Cass 3 for all but wikipedia
08:13 marostegui: Stop replication in sync on db1103 and db1072 - T164488
08:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1072 - T164488 (duration: 00m 50s)
07:31 marostegui: Stop MySQL on db2034 and db2036 to clone db2092
06:36 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2034 and db2036 - T178359 (duration: 01m 08s)
06:02 marostegui: Compress revision table across db1044 databases (s3)
05:46 marostegui: Optimize recentchanges, pagelinks and templatelinks on db1102 for s6 - T174509 T177772
05:45 marostegui: Optimize recentchanges, pagelinks and templatelinks on db1064 - T174509 T177772
05:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1051 - T174509 (duration: 00m 50s)
05:41 marostegui: Optimize pagelinks and templatelinks on db1051 - T174509
05:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1055 - T174509 (duration: 00m 50s)
05:31 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1056 - T174509 (duration: 00m 50s)
05:29 awight@tin: Started deploy [ores/deploy@19b1851]: Push ORES w/ Celery 4 support (take 2), T178441
05:08 awight@tin: Finished deploy [ores/deploy@19b1851]: Push ORES w/ Celery 4 support, T178441 (duration: 00m 18s)
05:08 awight@tin: Started deploy [ores/deploy@19b1851]: Push ORES w/ Celery 4 support, T178441
03:38 bblack: cp3034 - upgraded vhtcpd to 0.1.0 for testing
03:38 bblack: cp3030 - upgraded vhtcpd to 0.1.0 for testing
03:01 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Oct 19 03:01:54 UTC 2017 (duration 7m 2s)
02:54 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.4) (duration: 09m 28s)
02:29 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.3) (duration: 09m 40s)
00:01 maxsem@tin: Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/385106/2 (duration: 00m 50s)

2017-10-18

23:51 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/385094/2 (duration: 00m 50s)
23:49 maxsem@tin: Synchronized static/images/project-logos/: https://gerrit.wikimedia.org/r/#/c/385094/2 (duration: 00m 50s)
23:45 maxsem@tin: Synchronized php-1.31.0-wmf.4/extensions/Collection/: https://gerrit.wikimedia.org/r/#/c/384903/ (duration: 00m 50s)
23:44 maxsem@tin: Synchronized php-1.31.0-wmf.3/extensions/Collection/: https://gerrit.wikimedia.org/r/#/c/385005/ (duration: 00m 51s)
23:34 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/385007/ (duration: 00m 51s)
23:05 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.4
23:04 demon@tin: Synchronized php: symlink swap (duration: 00m 49s)
22:55 tgr@tin: Synchronized private/: Remove the llama (T150554) (duration: 00m 50s)
22:27 tgr@tin: Synchronized wmf-config/: T174651 / gerrit 384908 - deploy ReadingLists to beta - noop for production (duration: 00m 53s)
20:49 ejegg: updated CiviCRM from 7c26791 to b9d2067
20:16 XioNoX: Changing MTUs on interfaces to NTT
19:19 bblack: uploaded vhtcpd-0.1.0-1 to jessie-wikimedia, testing on cp1008 only for now
19:17 andrewbogott: restarting nodepool
19:15 andrewbogott: stopping nodepool temporarily to give rabbit a chance to catch up
19:01 MaxSem: ran LoginNotify/maintenance/migratePreferences.php on test and test2
18:57 andrewbogott: restarting nova-network on labnet1001
18:42 andrewbogott: restarting rabbitmq-server on labcontrol1001; too many timeouts
16:29 elukey: stop ircecho on einstenium for puppet shower
16:27 herron: restarted apache on puppetmasters
14:23 moritzm: uploaded hhvm 3.18.5+dfsg-1+wmf1+deb9u1 to apt.wikimedia.org/stretch-wikimedia
14:03 chasemp: add static ip to cloud mwstake project per T178012
14:03 chasemp: bump up quota on cyberpower cloud project per T178332
14:03 chasemp: create reading-lists cloud project
13:28 zeljkof: EU SWAT finished!
13:22 zfilipin@tin: Synchronized php-1.31.0-wmf.3/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: [cirrus] Turn on recall A/B test on enwiki (T177502) (duration: 00m 50s)
13:21 gehel: repooling wdqs1003 now that it has catched up on updates
13:15 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: pagePreviews: Restart A/B test on enwiki and dewiki (T176469) (duration: 00m 51s)
13:05 moritzm: uploaded wikidiff2 1.5.1 to apt.wikimedia.org
12:11 mobrovac@tin: Finished deploy [restbase/deploy@3c7abf6]: Bug fix: Remove the space in the summary table name (duration: 07m 13s)
12:04 mobrovac@tin: Started deploy [restbase/deploy@3c7abf6]: Bug fix: Remove the space in the summary table name
11:54 mobrovac@tin: Finished deploy [restbase/deploy@99052c1]: Use Cass3 storage for ruwiki summaries (duration: 08m 59s)
11:47 jynus@tin: Synchronized wmf-config/db-eqiad.php: Add db1105 to help db1071 because db1082 has crashed (duration: 00m 50s)
11:45 mobrovac@tin: Started deploy [restbase/deploy@99052c1]: Use Cass3 storage for ruwiki summaries
11:31 moritzm: upgrading mw1262-mw1265 (canaries) to wikidiff2 1.5.1
11:26 marostegui: Optimize pagelinks and templatelinks on db1102 s7 - T174509
11:21 moritzm: upgrading mw1261 to wikidiff2 1.5.1
11:11 mobrovac@tin: Finished deploy [restbase/deploy@15619e0]: Introduce the page/summary proxy (no-op for now), part #2 (duration: 08m 53s)
11:03 mobrovac@tin: Started deploy [restbase/deploy@15619e0]: Introduce the page/summary proxy (no-op for now), part #2
11:02 mobrovac@tin: Finished deploy [restbase/deploy@15619e0]: Introduce the page/summary proxy (no-op for now) (duration: 05m 23s)
10:57 mobrovac@tin: Started deploy [restbase/deploy@15619e0]: Introduce the page/summary proxy (no-op for now)
10:50 moritzm: rebooting multatuli for kernel update
10:21 moritzm: imported linux 4.9.30-2+deb9u5~bpo8+1 for jessie-wikimedia
09:51 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1098, depool db1082 (crashed) (duration: 00m 50s)
09:39 moritzm: installing xserver/xvfb security updates
09:36 jynus: reloading dbproxy1010's haproxy configuration
09:30 elukey: drop MobileWikiAppToCInteraction_10375484_15423246 from the log database on dbstore1002,db1047,db1046 - T177960
08:41 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Re-enable Statement usage tracking on cawiki (T151717) (duration: 00m 50s)
08:25 marostegui: Stop MySQL on db2019 to copy its data to db2084 - T178359
08:23 mobrovac@tin: Finished deploy [changeprop/deploy@065a06e]: Bug fix: Apply ignore_errors on a per-request basis (duration: 01m 17s)
08:22 mobrovac@tin: Started deploy [changeprop/deploy@065a06e]: Bug fix: Apply ignore_errors on a per-request basis
08:09 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable description usage tracking on a few test wikis (T177155) (duration: 00m 50s)
07:49 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1056 - T174509 (duration: 00m 49s)
07:46 marostegui: Optimize pagelinks and templatelinks on db1056 - T174509
07:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1034 - T174509 (duration: 00m 59s)
07:14 gehel: depooling wdqs1003 until it catches up on updates
07:02 marostegui: Stop MySQL on db2079 to copy its data to db2084 - T178359
06:46 _joe_: powercycling wdqs1003
05:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1072 - T164488 (duration: 00m 49s)
05:43 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1055 - T174509 (duration: 00m 50s)
05:40 marostegui: Optimize pagelinks and templatelinks on db1055 - T174509
05:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1098 - T174509 (duration: 00m 49s)
05:36 marostegui: Optimize pagelinks and templatelinks on db1098 - T174509
05:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1067 - T174509 (duration: 00m 50s)
05:25 marostegui: Optimize pagelinks and templatelinks on db1102 for s2 - T174509
03:17 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Oct 18 03:17:01 UTC 2017 (duration 7m 3s)
03:09 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.4) (duration: 14m 58s)
02:33 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.3) (duration: 08m 21s)

2017-10-17

23:31 thcipriani@tin: Synchronized php-1.31.0-wmf.4/extensions/Citoid/modules/ve.ui.CiteFromIdInspector.css: SWAT: ve.ui.CiteFromIdInspector: Fix CSS for context menus after changes in OOjs UI T178324 (duration: 00m 50s)
23:29 thcipriani@tin: Synchronized php-1.31.0-wmf.4/extensions/VisualEditor/modules/ve-mw/ui/styles/widgets/ve.ui.MWMediaInfoFieldWidget.css: SWAT: ve.ui.MWMediaInfoFieldWidget: Fix positioning of icons T178415 (duration: 00m 50s)
23:22 thcipriani@tin: Synchronized php-1.31.0-wmf.3/includes/shell/Command.php: SWAT: Shell\Command: Better walltime fallback T178314 (duration: 00m 51s)
23:15 thcipriani@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Remove setting no longer in MediaWiki (duration: 00m 50s)
23:06 mutante: disabling Icinga notifications for service recommendation_api on scb hosts - please remember to re-enable once ticket is resolved (T178445) (https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=recommendation_api%20endpoints)
22:28 ejegg: updated CiviCRM from fb513cc to 7c26791
21:55 foks: Removed 2FA from User:Amakuru
21:39 robh: cp4026 being returned to service post memory replacement. puppet runs fine and all icinga checks are green. repooled.
21:23 robh: cp4026 coming down for memory swap, no point in maint moding the system since other cp checks will alert anyhow
20:14 mutante: gerrit2001 - re-enabled puppet, attempting restart with --slave after gerrit:384758
19:54 mutante: cobalt - re-enabled puppet, ran puppet, systemd says gerrit is active(running)
19:51 mutante: cobalt - stopping gerrit service, confirming pid file gets removed, move /etc/init.d/gerrit to /root
19:50 mutante: short gerrit downtime for systemd change coming up
19:23 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.4
19:06 mutante: gerrit2001 - rebooting to verify systemd change and service comes back properly
18:53 mutante: gerrit2001: stopping the gerrit service
18:52 mutante: cobalt: temp disable puppet | gerrit2001: stop puppet
18:22 paravoid: rebooting sodium for kernel/distro upgrade
18:14 paravoid: dist-upgrading sodium to stretch
18:12 demon@tin: Finished scap: bootstrap wmf.4 (duration: 48m 41s)
17:23 demon@tin: Started scap: bootstrap wmf.4
16:49 demon@tin: Pruned MediaWiki: 1.31.0-wmf.2 [keeping static files] (duration: 01m 24s)
16:35 gehel: restart tilerator / tileratorui on all maps cluster after deployment of latest fix - T177389
16:34 gehel@tin: Finished deploy [tilerator/deploy@f3b26f3]: fix tileshell not exiting - T177389 (duration: 01m 42s)
16:32 gehel@tin: Started deploy [tilerator/deploy@f3b26f3]: fix tileshell not exiting - T177389
15:46 _joe_: killed a stuck gerrit process on cobalt
14:47 bd808: wikitech: Cleaned up 'importers' user_group entries (T171682)
14:25 Amir1: end of cleaning up ores_classification table in plwiki, start of ruwiki (T159753)
14:09 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1081 - T174509 T177772 (duration: 00m 45s)
14:05 Amir1: start of cleaning up ores_classification table in plwiki (T159753)
14:00 zeljkof: EU SWAT finished
13:59 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: [cirrus] Fix typo in UserTesting config var (take 2) (T177502) (duration: 00m 45s)
13:47 zeljkof: EU SWAT continues
13:31 zeljkof: stopping EU SWAT
13:29 jynus: killing commonswiki script on tin
13:17 zfilipin@tin: Synchronized php-1.31.0-wmf.3/extensions/UniversalLanguageSelector/maintenance/ULSCompactLinksDisablePref.php: SWAT: Remove the 20 edits threshold from ULSCompactLinksDisablePref.php (duration: 00m 46s)
12:53 volans: upgrading cumin masters to cumin 1.2.2-1
12:40 moritzm: upgrading mwdebug* to hhvm-wikidiff 1.5.1
11:47 marostegui: Optimize recentchanges on db1102 for s4 - T174509
11:47 marostegui: Optimize templatelinks and pagelinks on db1102 for s4 - T174509
11:45 Amir1: end of cleaning up ores_classification table in hewiki, start of nlwiki (T159753)
11:39 Amir1: end of cleaning up ores_classification table in fiwiki, start of hewiki (T159753)
11:32 jynus: test-installing proxysql on wasat T175672
11:24 Amir1: end of cleaning up ores_classification table in fawiki, start of fiwiki (T159753)
11:12 gehel: restarting wdqs-updater on wdqs1004 - T175919
10:40 Amir1: end of cleaning up ores_classification table in etwiki, start of fawiki (T159753)
10:36 Amir1: end of cleaning up ores_classification table in cswiki, start of etwiki (T159753)
10:26 Amir1: start of cleaning up ores_classification in cswiki (T159753)
10:16 akosiaris: removing akosiaris's yubikey from network devices
09:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1085 - T174509 T177772 (duration: 00m 46s)
09:23 marostegui: Stop MySQL on db2084 for multi-instance testing
09:16 mobrovac@tin: Finished deploy [restbase/deploy@7f228d4]: Remove /page/revision and add history metrics end points, part #3 (duration: 07m 34s)
09:08 mobrovac@tin: Started deploy [restbase/deploy@7f228d4]: Remove /page/revision and add history metrics end points, part #3
09:08 mobrovac@tin: Finished deploy [restbase/deploy@3bcd1b7]: Remove /page/revision and add history metrics end points, part #2 (duration: 25m 57s)
08:45 gehel: restarting blazegraph on wdqs1004 for GC tuning (adding -XX:+G1PrintRegionLivenessInfo) - T175919
08:42 mobrovac@tin: Started deploy [restbase/deploy@3bcd1b7]: Remove /page/revision and add history metrics end points, part #2
08:40 mobrovac@tin: Finished deploy [restbase/deploy@3bcd1b7]: Remove /page/revision and add history metrics end points - T158100 T175805 (duration: 01m 05s)
08:39 mobrovac@tin: Started deploy [restbase/deploy@3bcd1b7]: Remove /page/revision and add history metrics end points - T158100 T175805
07:54 marostegui: Stop MySQL and reboot db2081 to see if it works fine - T178140
07:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2086 to the config - T170662 (duration: 00m 45s)
07:52 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2086 to the config - T170662 (duration: 00m 45s)
07:51 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2086 to the config - T170662 (duration: 00m 45s)
07:42 marostegui: Stop MySQL on db2079 to clone db2086 - T170662
07:36 moritzm: rebooting boron for kernel update
07:31 ema: upgrade misc_eqiad to varnish 5 T177233
07:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1067 - T174509 (duration: 00m 45s)
07:11 marostegui: Optimize templatelinks and pagelinks on db1067 - T174509
07:10 marostegui: Optimize enwiki.ores_classification on db1067 - T159753
06:55 moritzm: installing libffi security updates on trusty (Debian already fixed)
06:07 marostegui: Stop replication in sync on db1072 and db1103 for data drift fixing - T164488
05:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1034 - T174509 (duration: 00m 45s)
05:54 marostegui: Optimize pagelinks and templatelinks on db1034 - T174509
05:50 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1085 - T174509 T177772 (duration: 00m 45s)
05:46 marostegui: Optimize pagelinks, templatelinks and recentchanges on db1085 T177772 T174509
05:42 marostegui: Optimize pagelinks, templatelinks and ores_classification on db1095 - T174509 T159753
05:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1081 - T174509 (duration: 00m 45s)
05:38 marostegui: Optimize recentchanges on db1081 - T177772
05:37 marostegui: Optimize pagelinks and templatelinks on db1081 - T174509
05:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1090 - T174509 (duration: 00m 45s)
05:32 marostegui: Optimize pagelinks and templatelinks on db1090 - T174509
05:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1073 - T174509 (duration: 00m 46s)
02:32 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Oct 17 02:32:30 UTC 2017 (duration 6m 57s)
02:25 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.3) (duration: 08m 23s)
00:34 ejegg: updated CiviCRM from 824c2d8 to fb513cc

2017-10-16

23:49 thcipriani@tin: Synchronized php-1.31.0-wmf.3/extensions/CheckUser/specials/SpecialCheckUser.php: SWAT: Revert "Revert "Restore checkuser-userlinks when exist"" T170507 (duration: 00m 46s)
23:46 ebernhardson: increase elasticsearch log levels for transport to debug to help track down T167410
23:39 ejegg: updated civicrm from b953064 to 824c2d8
23:25 thcipriani@tin: Synchronized php-1.31.0-wmf.3/extensions/Echo/includes/ContainmentSet.php: SWAT: ContainmentSet: Use strict comparison for array_search() T177825 (duration: 01m 45s)
20:35 chasemp: disable puppet around cloud things for a refactor merge
20:20 demon@tin: Pruned MediaWiki: 1.30.0-wmf.18 (duration: 02m 59s)
19:59 demon@tin: Synchronized wmf-config/CommonSettings.php: Ief0ea01f (duration: 00m 46s)
19:58 jynus: sudo apt install wmf-mariadb101-client on terbium for debugging an issue
19:57 demon@tin: Synchronized wmf-config/InitialiseSettings.php: Ief0ea01f (duration: 00m 46s)
19:56 demon@tin: Synchronized tests/cirrusTest.php: Ief0ea01f (duration: 00m 47s)
18:18 thcipriani@tin: Synchronized wmf-config/Wikibase.php: SWAT: Add configuration for statement indexing for Wikidata T175199 (duration: 00m 47s)
17:35 ariel@tin: Finished deploy [dumps/dumps@2b9326b]: do the minimum for latestlinks job, no updating other files (duration: 00m 03s)
17:35 ariel@tin: Started deploy [dumps/dumps@2b9326b]: do the minimum for latestlinks job, no updating other files
17:08 gehel@tin: Finished deploy [wdqs/wdqs@b42eb0d]: WDQS GUI uipdates (duration: 02m 20s)
17:06 gehel@tin: Started deploy [wdqs/wdqs@b42eb0d]: WDQS GUI uipdates
16:56 ottomata: deploying LVS for druid-public-broker https://phabricator.wikimedia.org/T176223
16:39 joal@tin: Finished deploy [analytics/aqs/deploy@339674c]: Deploy new mediawiki-history endpoints (alpha version) after monitorin fix (duration: 04m 42s)
16:35 joal@tin: Started deploy [analytics/aqs/deploy@339674c]: Deploy new mediawiki-history endpoints (alpha version) after monitorin fix
16:23 joal@tin: Finished deploy [analytics/aqs/deploy@aa7ef0d]: Deploy new mediawiki-history endpoints (alpha version) after node-modules fix (duration: 06m 56s)
16:16 joal@tin: Started deploy [analytics/aqs/deploy@aa7ef0d]: Deploy new mediawiki-history endpoints (alpha version) after node-modules fix
16:09 ema: upgrade misc_codfw to varnish 5 T177233
16:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1084 - T174509 T177772 (duration: 00m 46s)
15:51 gehel: killing stuck tileshell process on maps-test2001 - T177389
15:51 gehel@tin: Finished deploy [tilerator/deploy@f3b26f3]: testing tileshell fix - T177389 (duration: 00m 20s)
15:50 gehel@tin: Started deploy [tilerator/deploy@f3b26f3]: testing tileshell fix - T177389
15:45 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1094 - T174509 (duration: 00m 46s)
15:38 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1088 - T174509 T177772 (duration: 00m 47s)
14:34 twentyafterfour: deploying hotfix for T178068 and T178107
14:30 moritzm: uploaded wikidiff 1.5.1 to apt.wikimedia.org for jessie/experimental
14:08 hashar: European swat is complete
14:08 hashar@tin: Synchronized wmf-config/abusefilter.php: (T177760) Enable AbuseFilter profiler | (T177761) Allow AbuseFilter to block & grant relevant permissions to permit administrators to manage the restricted option. (duration: 00m 47s)
13:50 moritzm: upgrading deployment-mediawiki0[56] to wikidiff2 1.5.1
13:45 hashar@tin: Synchronized wmf-config/: [cirrus] Prepare profiles for the recall A/B test on enwiki - T177502 (duration: 00m 48s)
13:44 moritzm: upgrading deployment-mediawiki04 to wikidiff2 1.5.1
13:42 hashar@tin: Synchronized wmf-config: [cirrus] Disable A/B test of MLR on testing on 18 wikis with > 1% of search traffic - T177490 (duration: 00m 48s)
13:38 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Add additional namespaces to search results for bnwikisource - T178041 (duration: 00m 49s)
12:59 gehel: restart cassandra on maps1001
12:38 ema: upgrading eqiad LVSs to pybal 1.14.2 (T178149, T177815)
11:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1074 - T174509 (duration: 00m 46s)
11:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1070 - T174509 (duration: 00m 46s)
11:10 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: Electron hanging - T174916
10:48 Amir1: cleaning up ores_classification table in wikidatawiki (T159753)
10:03 ema: upgrading codfw LVSs to pybal 1.14.2 (T178149, T177815)
09:54 ema: upgrading esams LVSs to pybal 1.14.2 (T178149, T177815)
09:44 ema: upgrading ulsfo LVSs to pybal 1.14.2 (T178149, T177815)
09:43 ema: pybal 1.14.2 uploaded to apt.w.o
09:40 moritzm: restarting cassandra on restbase1007 to pick up NSS security update
09:39 moritzm: restarting cassandra on cerium/xenon/praseodymium to pick up NSS security update
09:37 gehel: restarting elasticsearch on elastic1017
09:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1088 - T174509 T177772 (duration: 00m 46s)
09:07 marostegui: Optimize recentchanges, pagelinks and templatelinks on db1088 - https://phabricator.wikimedia.org/T174509 https://phabricator.wikimedia.org/T177772
09:02 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1093 - T174509 T177772 (duration: 00m 46s)
08:53 marostegui: Stop MySQL on db1050 as it will be decommissioned - T178162
08:46 marostegui: Remove db1050 from tendril - T178162
08:31 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1050 from config - T178162 (duration: 00m 46s)
08:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1050 from config - T178162 (duration: 00m 46s)
07:56 marostegui: Stop db1103 and db1038 at the same position to fix data drifts - T164488
07:42 moritzm: installing NSS security updates
07:30 marostegui: Stop db1103 and db1072 at the same position to fix data drifts - T164488
07:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1072 - T164488 (duration: 00m 46s)
07:19 marostegui: Optimize table ores_classification on db1073 - T159753
07:14 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2084 to the config - T170662 (duration: 00m 46s)
07:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2084 to the config - T170662 (duration: 00m 47s)
06:25 marostegui: Stop MySQL on db2079 to clone db2084 from it - T170662
06:12 marostegui: Optimize pagelinks and templatelinks on db1094 - T174509
06:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1093 - T174509 T177772 (duration: 00m 46s)
06:04 marostegui: Optimize recentchanges, pagelinks and templatelinks on db1093 - T174509 T177772
06:00 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1070 - T174509 (duration: 00m 46s)
05:59 marostegui: Optimize recentchanges, pagelinks and templatelinks on db1070 - T174509
05:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1084 - T174509 T177772 (duration: 00m 46s)
05:53 marostegui: Optimize recentchanges, pagelinks and templatelinks on db1084 - T174509 T177772
05:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1074 - T174509 (duration: 00m 46s)
05:47 marostegui: Optimize pagelinks and templatelinks on db1074 - T174509
05:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1073 - T174509 (duration: 00m 47s)
05:37 marostegui: Optimize templatelinks and pagelinks on db1073 - T174509
05:36 marostegui: Optimize templatelinks and pagelinks on db1073 - 174509
02:35 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Oct 16 02:35:34 UTC 2017 (duration 6m 36s)
02:28 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.3) (duration: 07m 44s)

2017-10-15

11:15 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: Electron hanging - T174916

2017-10-14

03:52 paravoid: manually cleaning up pdns_metric_cron from all dnsrecursors (hydrogen/chromium/acamar/achernar/maerlant/nescio)

2017-10-13

23:04 mutante: scb1001 - systemctl restart pdfrender (Icinga said: pdfrender Connection refused)
20:36 mutante: gerrit2001 - removing hhvm package (which is now possible without also removing scap, thanks thcipriani), apt-get autoremove to remove more unused packages (T178039)
18:52 joal@tin: Finished deploy [analytics/aqs/deploy@0375c34]: Deploy new mediawiki-history endpoints-alpha version (duration: 15m 02s)
18:40 joal@tin: (no justification provided)
18:37 joal@tin: Started deploy [analytics/aqs/deploy@0375c34]: Deploy new mediawiki-history endpoints-alpha version
16:54 otto@tin: Finished deploy [analytics/refinery@28db253]: (no justification provided) (duration: 05m 50s)
16:48 otto@tin: Started deploy [analytics/refinery@28db253]: (no justification provided)
15:31 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1056 (duration: 00m 46s)
14:59 mutante: added cparle to wmf LDAP group
14:38 jynus: stopping slave on db1056 and optimize recentchanges
14:25 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1100, depool db1056, fix db1098 weight (duration: 00m 46s)
14:06 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1053 (duration: 00m 46s)
13:07 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1053 (duration: 00m 47s)
12:57 godog: upload scap 3.7.1-1 - T127762
11:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2082 to the config - T170662 (duration: 00m 46s)
11:26 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2082 to the config - T170662 (duration: 00m 46s)
10:53 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1098 (duration: 00m 47s)
10:22 godog: roll-restart pybal to apply https://gerrit.wikimedia.org/r/#/c/383997/ - T178149
10:04 jynus: stopping slave on db1098 and optimize recentchanges
09:54 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1098 (duration: 00m 46s)
09:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2082 to the config - T170662 (duration: 00m 46s)
09:22 Amir1: a small clean up of ores_classification table in wikidatawiki (T159753)
09:21 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2082 to the config - T170662 (duration: 00m 47s)
09:04 godog: restart pybal on lvs1003 to test logstash depool - T178078
08:57 marostegui: Stop db1103 and db1038 in sync for more checksumming - T164488
08:34 ema: upgrade pybal on lvs1003 to 1.14.1 T177815
08:18 ema: upgrade pybal on lvs1006 to 1.14.1 T177815
08:15 jynus: restarting commons wiki recentchanges purge of wb entries T177772
08:11 ema@neodymium: conftool action : set/pooled=yes; selector: name=logstash1001.eqiad.wmnet,service=logstash-gelf,dc=eqiad
08:11 ema@neodymium: conftool action : set/pooled=no; selector: name=logstash1001.eqiad.wmnet,service=logstash-gelf,dc=eqiad
08:10 ema: depool/repool logstash1001 (service=logstash-gelf) T178078
07:35 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2081 to the config - T170662 (duration: 00m 46s)
07:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2081 to the config - T170662 (duration: 00m 47s)
06:23 marostegui: Stop MySQL on db2080 to copy its data to db2081 - T170662
00:22 mutante: authdns servers: deleted /etc/ganglia/conf.d/gndsd.pyconf & /usr/lib/ganglia/python_modules_gdnsd.py - removing ganglia stats

2017-10-12

23:58 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Restore thanks limit on pl.wikipedia (T169268) (duration: 00m 47s)
23:50 cwd: re-enabled omnimail jobs
23:35 eileen1: update from 2d6668b to b953064
23:25 dereckson@tin: Synchronized wmf-config/CommonSettings.php: Add logging for email blocks (T175419) (duration: 00m 46s)
23:13 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Switch test wikis to HTML5 fragment mode in links (T152540) (duration: 00m 47s)
22:44 demon@tin: Synchronized php-1.31.0-wmf.3/includes/specialpage/ChangesListSpecialPage.php: silence notices (duration: 00m 47s)
21:36 cwd: disabled omnimail jobs
19:47 ejegg: updated CiviCRM from 47de976 to 2d6668b
19:46 chasemp: disable puppet for lab*services* for merge of 383896
19:18 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.3
18:28 mutante: added Pablo (pgrass) to LDAP group 'wmde' (T177599)
18:15 demon@tin: Synchronized php-1.31.0-wmf.3/extensions/ORES/includes/Hooks.php: Make watchlist queries suck less (duration: 00m 50s)
18:01 demon@tin: Synchronized php-1.31.0-wmf.3/extensions/JsonConfig/includes/JCCache.php: Silence junk warnings (duration: 00m 50s)
15:17 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: Electron hanging - T174916
15:11 mutante: gerrit2001 - rebooting for kernel upgrade
14:57 mutante: netmon1002 - arming keyholder for rancid deploy
14:52 mutante: netmon1002 (librenms, rancid) - rebooting for kernel upgrade
14:47 moritzm: rebooting kubernetes workers to new 4.9.51 kernel
14:41 moritzm: rebooting kubernetes staging servers to new 4.9.51 kernel
14:29 moritzm: rebooting kubernetes masters to new 4.9.51 kernel
13:54 moritzm: updating image scalers to librsvg 2.40.18
13:48 elukey: deployed the new Analytics Public Druid cluster - T176223
13:45 moritzm: updating remaining thumbor servers to librsvg 2.40.18
13:40 ladsgroup@tin: Synchronized wmf-config/Wikibase.php: SWAT: Remove deprecated config variable for Wikibase (T129475) (duration: 01m 02s)
13:33 moritzm: updating thumbor1001 to librsvg 2.40.18
13:13 moritzm: rolling restart of logstash to pick up thumbor logs (T150734)
13:07 zeljkof: EU SWAT finished
12:56 moritzm: uploaded librsvg 2.40.18 for jessie-wikimedia to apt.wikimedia.org
11:46 ema: upgrading cp3010 to Varnish 5 T177233
11:27 Amir1: finished cleanup of ores_classification in wikidatawiki, still needs more work (T159753)
10:03 ema: lvs1005: upgrade pybal to 1.14.1 T177815
10:02 ema: pybal 1.14.1 uploaded to apt.w.o
09:40 ema: upgrading cp3008 to Varnish 5 T177233
09:06 moritzm: installing nettle update from stretch 9.2 point release
09:01 moritzm: installing ncurses security updates
08:25 moritzm: installing gnutls28 update from stretch 9.2 point release
08:02 Amir1: starting cleanup of ores_classification in wikidatawiki (T159753)
07:51 gehel: reinitialize cassandra on maps-test2004 to test vector tiles - T153282
07:40 moritzm: installing at-spi2-core update from stretch 9.2 point release
07:24 moritzm: installing gnupg2 update from stretch 9.2 point release
07:18 legoktm: purging html5-misnesting linter entries from database on terbium (T178040)
03:25 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Oct 12 03:25:17 UTC 2017 (duration 7m 17s)
03:18 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.3) (duration: 15m 57s)
02:41 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 15m 39s)
00:30 twentyafterfour: phabricator update finished (a while ago, I forgot to log it)
00:08 twentyafterfour: upgrading phabricator to #phab-2017-10-11
00:05 mutante: netmon2001 arm keyholder for rancid deploy
00:01 twentyafterfour: Preparing to take phabricator offline for a hopefully momentary upgrade.

2017-10-11

23:55 mutante: netmon2001 - smokeping back up - the reboot, as a side-effect, also resolved the systemd-icinga-alert that was pending in Icinga for a while with comments
23:52 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Vector print logo and print styles on all wikis T169732 PART II (duration: 00m 50s)
23:50 thcipriani@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Enable Vector print logo and print styles on all wikis T169732 PART I (duration: 00m 50s)
23:47 mutante: netmon2001 (smokeping) - kernel upgrade, schedule downtime, reboot
23:40 thcipriani@tin: Synchronized php-1.31.0-wmf.2/skins/Vector/ResourceLoaderLessModule.php: SWAT: Print logo should use an absolute URI T177800 (duration: 00m 49s)
23:38 mutante: netmon1003 - servermon back up - apt-get autoremove to drop unrequired python packages
23:37 thcipriani@tin: Synchronized php-1.31.0-wmf.3/skins/Vector/ResourceLoaderLessModule.php: SWAT: Print logo should use an absolute URI T177800 (duration: 00m 50s)
23:35 mutante: netmon1003 - cant seem to reboot normally, attempting via gnt-instance command
23:31 mutante: netmon1003 (servermon) - upgrading kernel (jessie), schedule icinga downtime, rebooting, short downtime for servermon itself
23:31 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Parser: Switch from Tidy to Remex on nowiki T177989 (duration: 00m 50s)
23:27 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Parser: Switch from Tidy to Remex on fawiki T176150 (duration: 00m 50s)
23:23 thcipriani@tin: Synchronized php-1.31.0-wmf.3/extensions/VisualEditor/modules/ve-mw/init/styles/ve.init.MWVESwitchConfirmDialog.css: SWAT: Fix WikiEditor mode switcher widget (duration: 00m 50s)
23:19 thcipriani@tin: Synchronized php-1.31.0-wmf.2/includes/specials/SpecialUnblock.php: SWAT: Fix unblocking autoblocks T177952 (duration: 00m 50s)
23:18 mutante: releases1001 upgrading kernel, installed linux-image-amd64, took out of cache::misc, upgrade libnss3, scheduled downtime, rebooting ...
23:17 thcipriani@tin: Synchronized php-1.31.0-wmf.3/includes/specials/SpecialUnblock.php: SWAT: Fix unblocking autoblocks T177952 (duration: 00m 51s)
22:01 volans: uploaded cumin_1.2.2-1_amd64.deb to apt.wikimedia.org jessie-wikimedia
21:44 robh: cp4009 shutdown accidental, will boot back up immediately
21:13 mutante: releases2001 - scheduled downtime, rebooting
21:07 mutante: running puppet on cache misc hosts to temp remove codfw backend of releases.wm.org, then rebooting releases2001 for kernel upgrade
20:22 subbu: updated Parsoid to ddf7b293 (T138492, T135667, T177612)
20:18 ssastry@tin: Finished deploy [parsoid/deploy@938e305]: Updating Parsoid to ddf7b293 (duration: 08m 42s)
20:17 hashar@tin: Finished deploy [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044 (duration: 02m 54s)
20:14 hashar@tin: Started deploy [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044
20:13 hashar@tin: Finished deploy [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044 (duration: 00m 02s)
20:13 hashar@tin: Started deploy [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044
20:11 hashar@tin: Started restart [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044
20:10 bsitzmann@tin: Finished deploy [mobileapps/deploy@15a52ba]: Update mobileapps to c045afc (T177301) (duration: 05m 25s)
20:09 ssastry@tin: Started deploy [parsoid/deploy@938e305]: Updating Parsoid to ddf7b293
20:07 hashar@tin: Started restart [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044
20:05 bsitzmann@tin: Started deploy [mobileapps/deploy@15a52ba]: Update mobileapps to c045afc (T177301)
20:04 hashar@tin: Started restart [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044
20:01 chasemp: disable puppet around cloud things to roll refactor out slowly
19:29 mutante: releases1001 - same as 2001, upgraded jenkins to 2.73.2, kept existing config (T177962)
19:27 mutante: releases2001 - upgraded jenkins to 2.73.2, kept existing config (vs overwriting with package config)
19:20 mutante: apt: reprepro copy stretch-wikimedia jessie-wikimedia jenkins
19:16 mutante: releases1001/releases2001 - apt-get autoremove to drop non-required python packages
19:03 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.3
19:02 demon@tin: Synchronized php: symlink swap (duration: 00m 49s)
18:47 gehel: killing stuck osm replication on maps-test2001
18:23 demon@tin: Synchronized wmf-config/Wikibase-labs.php: no-op, labs only (duration: 00m 59s)
18:19 demon@tin: Finished scap: wmf.3 bootstrap (duration: 44m 56s)
17:58 herron: ran /home/faidon/update-netboot-stretch.sh on puppetmaster1001
17:49 moritzm: uploaded jenkins 2.73.2 for jessie-wikimedia to apt.wikimedia.org
17:34 demon@tin: Started scap: wmf.3 bootstrap
17:32 demon@tin: Synchronized php: symlink swap to wmf.2, somehow got missed (duration: 00m 45s)
17:29 demon@tin: Synchronized scap/plugins/prep.py: No-op (duration: 01m 49s)
17:09 demon@tin: Pruned MediaWiki: 1.30.0-wmf.17 (duration: 02m 20s)
17:06 demon@tin: Pruned MediaWiki: 1.30.0-wmf.16 (duration: 02m 40s)
16:54 demon@tin: Pruned MediaWiki: 1.31.0-wmf.1 [keeping static files] (duration: 01m 44s)
16:24 hasharAway: Upgrade jenkins Maven integration plugin to 3.0 - T177962
16:15 godog: roll-restart thumbor to apply https://gerrit.wikimedia.org/r/380942 - T150734
16:12 hasharAway: Upgrading Jenkins CI T177962
16:09 gehel: decommission maps-test2004 from its cassandra cluster (free a node to test vector tiles) - T153282
16:09 XioNoX: resuming PDU upgrade
15:39 elukey: drop tables listed in https://phabricator.wikimedia.org/T171629#3674250 from db1046, db1047, dbstore1002
15:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1086 - T174509 (duration: 00m 47s)
15:21 _joe_: stopped redis on ocg* T177931
15:08 _joe_: ocg*: stop ocg; mv /srv/deployment /srv/stale, update-rc.d ocg disable, rm /etc/init/ocg.conf - T177931
14:55 _joe_: manually removing IPVS entries for ocg on eqiad LBs, T177931
14:47 _joe_: rolling restart of low-traffic pybals in eqiad for T177931
14:36 moritzm: installing dbus update from stretch 9.2 point release
13:40 hashar: European SWAT completed
13:39 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Remove compact language links dblist for simplicity (duration: 00m 46s)
13:38 hashar@tin: Synchronized docroot: Remove compact language links dblist for simplicity (duration: 00m 48s)
13:37 hashar@tin: Synchronized dblists: Remove compact language links dblist for simplicity (duration: 00m 47s)
13:36 hashar@tin: Synchronized wmf-config/CommonSettings.php: Remove compact language links dblist for simplicity (duration: 00m 47s)
13:29 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Configure wmgBabelMainCategory for the Dinka Wikipedia (duration: 00m 47s)
13:27 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable mapframe on eswiki - T177695 (duration: 00m 47s)
13:24 hashar@tin: Synchronized wmf-config/Wikibase-production.php: Enable constraint checks on qualifiers and references - T176863 (duration: 00m 47s)
13:21 moritzm: installing db5.3 security updates
13:03 godog: test graphite-web patch on labmon1001 - T177747
12:54 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Disable Statement usage tracking on cawiki (T151717) (duration: 00m 47s)
12:53 marostegui: Start recentchanges purge on s4 primary master T177772
12:48 marostegui: Kill recentchanges purge on s4 primary master - https://phabricator.wikimedia.org/T177772
12:44 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: (temp) Disable Statement usage tracking on cawiki (T151717) (duration: 00m 48s)
12:38 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable Statement usage tracking on cawiki and cewiki (T151717) (duration: 00m 47s)
12:32 hoo@tin: Synchronized wmf-config/: Move WB client "disabledUsageAspects" setting into $wmgWikibaseDisabledUsageAspects (duration: 00m 48s)
12:30 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Introduce $wmgWikibaseDisabledUsageAspects (duration: 00m 47s)
12:27 hoo@tin: scap aborted: wmf-config/InitialiseSettings.php Introduce $wmgWikibaseDisabledUsageAspects (duration: 02m 34s)
12:25 hoo@tin: Started scap: wmf-config/InitialiseSettings.php Introduce $wmgWikibaseDisabledUsageAspects
11:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1087 - T174509 (duration: 00m 50s)
11:49 ema: upgrading cp3007 to Varnish 5 T177233
11:34 moritzm: installing ruby1.9 security updates on trusty hosts
11:19 akosiaris: re-enable OTRS stats group T176221
11:11 jynus: starting purge of ruwiki.recentchanges T177772
10:55 akosiaris: OTRS upgrade done T176221
10:44 akosiaris: starting OTRS upgrade T176221
10:24 jynus: stopping dbstore2001:s3 for maintenance T177908
10:19 moritzm: installing curl security updates
09:32 moritzm: installing libxfont security updates
09:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1087 - T174509 (duration: 00m 46s)
09:27 marostegui: Optimize pagelinks and templatelinks on db1087 - T174509
09:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1082 - T174509 (duration: 00m 47s)
09:04 moritzm: installing curl security updates on app server canaries (along with HHVM restart)
08:50 moritzm: installing ffmpeg security updates
08:38 elukey: reboot kafka-jumbo hosts for kernel updates
08:37 marostegui: Stop replication in sync on db1103 and db1038 to checksum their data - T164488
08:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1103 - T164488 (duration: 00m 47s)
07:46 jynus: restart commonswiki recentchanges purging T177772
07:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1072 into the vslow and dump service for s3 - T172679 (duration: 00m 46s)
07:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1086 - T174509 (duration: 00m 47s)
07:02 marostegui: Optimize templatelinks and pagelinks on db1086 - T174509
06:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1082 - T174509 (duration: 00m 47s)
06:57 marostegui: Optimize templatelinks and pagelinks on db1082 - T174509
06:14 marostegui: Set db1068 s4 primary master to read-only OFF - T168661
06:14 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Set s4 to writable mode after maintenance - T168661 (duration: 00m 47s)
06:07 marostegui: Deploy alter table on s4 primary master - T168661
06:01 marostegui: Set read-only on db1068 s4 primary master - T168661
06:00 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Set s4 in read-only for maintenance - T168661 (duration: 00m 47s)
05:48 marostegui: Package update on db1068 s4 primary master - T168661
05:35 marostegui: Disable puppet on db1068 (s4 primary master) - T168661
05:35 marostegui: Disable puppet on db1068 (s4 primary master)
05:13 marostegui: Disable alerts on s4 for 2 hours - T168661
05:06 marostegui: Dump buffer pool on s4 primary master db1068 - T168661
05:03 kartik@tin: Finished deploy [cxserver/deploy@a79b38c]: Update cxserver to 273b515 (duration: 02m 57s)
05:00 kartik@tin: Started deploy [cxserver/deploy@a79b38c]: Update cxserver to 273b515
02:32 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Oct 11 02:32:46 UTC 2017 (duration 6m 39s)
02:26 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 08m 52s)

2017-10-10

23:53 thcipriani@tin: Synchronized wmf-config: SWAT: Disable OCG services PART II T177795 (duration: 00m 48s)
23:52 thcipriani@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Disable OCG services PART I T177795 (duration: 00m 47s)
23:41 thcipriani@tin: Synchronized php-1.31.0-wmf.2/extensions/Collection/RenderingAPI.php: SWAT: Do not request render if renderer not configured T177795 (duration: 00m 47s)
23:39 XioNoX: upgrading ps1-a2-eqiad - T175341
23:35 thcipriani@tin: Synchronized wmf-config/CommonSettings.php: SWAT: revert Enable Vector print logo on test wiki (duration: 00m 47s)
23:21 thcipriani@tin: scap failed: average error rate on 3/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details)
23:12 thcipriani@tin: Synchronized wmf-config/CommonSettings.php: SWAT: wikitech: Align "contentadmin" and "sysop" permissions T171208 (duration: 00m 48s)
23:08 eileen1: ran the group clean up script (reduced smart groups)
22:28 eileen1: changed civicrm smart group cache timeout to 5 civicrm/admin/setting/search?reset=1
21:15 bblack: esams repooling - T167840
19:30 krinkle@tin: Synchronized docroot/noc/: Clean up noc base.css (duration: 00m 48s)
19:07 krinkle@tin: Synchronized docroot/noc/: Clean up noc/index CSS (duration: 00m 47s)
19:00 XioNoX: starting the work for T167840
18:41 mutante: logstash*: delete /usr/lib/ganglia/python_modules/elasticsearch_monitoring.py and /etc/ganglia/conf.d/elasticsearch.pyconf via cumin (T177225)
17:32 XioNoX: depooling esams from DNS for T167840
17:26 gehel: shutting down and restarting elasticsearch on relforge1001 for testing - T170378
17:25 krinkle@tin: Synchronized php-1.31.0-wmf.2/resources/lib/jquery.ui/: I717f2580e3aae (duration: 00m 47s)
16:19 milimetric@tin: Finished deploy [analytics/refinery@f4a0a33]: Deploying mostly for the new script in the bin folder (duration: 16m 01s)
16:03 milimetric@tin: Started deploy [analytics/refinery@f4a0a33]: Deploying mostly for the new script in the bin folder
15:48 jynus: starting purge of commonswiki.recentchanges T177772
15:39 thcipriani@tin: Synchronized README: noop deployment to test new logstash-checker.py host (duration: 00m 47s)
15:27 jynus: upgrading neodymium and sarin to mariadb-client 10.1.28
15:23 ema: cp1008 upgraded to varnish 5 T168529
15:00 godog: test stretch 9.2 upgrade on ms-fe2005 - T177739
14:59 cmjohnson1: lvs1007 swapping NIC card
14:53 godog: test stretch 9.2 upgrade on ms-be2013 - T177739
14:45 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1079 - T174509 (duration: 00m 47s)
14:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1080 - T174509 (duration: 00m 47s)
14:31 marostegui: Optimize table ores_classifications on db1080 - T159753
14:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1072 with low weight - T172679 (duration: 00m 47s)
14:24 aude@tin: Synchronized wmf-config/throttle.php: Add throttle rule T177835 (duration: 00m 47s)
14:22 elukey: add druid public cluster's IPs to analytics-in4 on cr1/cr2 - T177511
13:56 aude@tin: Synchronized wmf-config/throttle.php: Add throttle rule T177835 (duration: 02m 16s)
13:37 aude@tin: Synchronized wmf-config/Wikibase-production.php: Remove test wiki echo configs from Wikibase config (duration: 01m 31s)
13:20 aude@tin: Synchronized wmf-config/InitialiseSettings.php: Enable SandboxLink on gawiki (duration: 01m 50s)
13:11 gehel: restarting rsyslog on mw1180 - T177833
11:32 moritzm: downgrading HHVM on deployment-mediawiki04 to HHVM 3.18.2 temporarily (to further narrow down a problem with the new wikidiff package)
11:31 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1076 - T174509 (duration: 05m 02s)
11:31 dcausse: upgrading relforge to elastic 5.5.2
11:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1071 - T174509 (duration: 01m 21s)
10:33 akosiaris: T177511 switch druid100[456] to private1-x-eqiad VLANs
10:26 Amir1: start of cleaning up ores_classification in wikidatawiki (T159753)
08:07 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2080 to the config - T170662 (duration: 00m 46s)
08:06 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2080 to the config - T170662 (duration: 00m 47s)
07:04 marostegui: Stop MySQL on db2079 to clone db2080 - https://phabricator.wikimedia.org/T170662
06:35 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s3 - T153033
06:27 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s1 - T153033
06:22 marostegui: Stop MySQL on db1038 to transfer its data to db1072 - T172679
06:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1038 - T172679 (duration: 00m 47s)
06:05 marostegui: Stop MySQL on db1072 to move it to s3 - T172679
05:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1079 - T174509 (duration: 00m 47s)
05:53 marostegui: Optimize templatelinks and pagelinks on db1079 - T174509
05:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1071 - T174509 (duration: 00m 48s)
05:45 marostegui: Optimize templatelinks and pagelinks on db1071 - T174509
05:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1076 - T174509 (duration: 00m 47s)
05:38 marostegui: Optimize templatelinks and pagelinks on db1076 - T174509
05:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1080 - T174509 (duration: 00m 48s)
05:32 marostegui: Optimize templatelinks and pagelinks on db1080 - T174509
02:35 eileen2: update civicrm from 00da4b0 to 47de976
02:32 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Oct 10 02:32:57 UTC 2017 (duration 6m 41s)
02:26 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 08m 49s)
01:21 eileen2: update civicrm from 276cbfd to 00da4b0

2017-10-09

22:18 hoo: Removed empty left-over xmldatadumps/public/other/wikibase/wikidatawiki/20171007 (https://dumps.wikimedia.org/wikidatawiki/entities/20171007/)
19:53 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable new print styles on Vector in test wiki - T169732 (duration: 00m 47s)
19:51 hashar: morning swat completed
19:50 hashar@tin: Synchronized php-1.31.0-wmf.2/extensions/Echo/maintenance/updatePerUserBlacklist.php: Sync a live hack in Echo updatePerUserBlacklist https://gerrit.wikimedia.org/r/#/c/383182/ - T173475 (duration: 00m 47s)
19:48 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable new print styles on Vector in test wiki - T169732 (duration: 00m 47s)
19:22 hashar@tin: Synchronized wmf-config/CommonSettings.php: REVERT: Enable new print styles on Vector - T169732 (duration: 00m 49s)
19:19 hashar@tin: scap failed: average error rate on 8/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details)
19:17 hashar@tin: Synchronized php-1.31.0-wmf.2/extensions/Echo: Reapply "Use User Ids instead of User Names for Echo Mute"" - T173475 (duration: 00m 52s)
18:57 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable NewUserMessage for SUL accounts too on hi.wikiversity - T177690 (duration: 00m 47s)
18:52 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable NewUserMessage for SUL accounts too on dty.wiki - T177688 (duration: 00m 47s)
18:45 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Add autopatrolled user group to sd.wikipedia - T177141 (duration: 00m 47s)
18:00 ottomata: stopping druid services on druid1004 (not 1006)
17:59 ottomata: stopping druid services on druid1006
17:28 ottomata: stopping druid services on druid1005
16:34 ottomata: stopping druid services on druid1006
16:34 ottomata: beginning decom and reinstall process for druid1004-1006 -- T176223
15:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1091 - T174509 (duration: 00m 47s)
13:53 zeljkof: EU SWAT finished
13:53 zfilipin@tin: Synchronized php-1.31.0-wmf.2/extensions/Translate/: SWAT: Revert "[tech-debt] Remove usage of FuzzyLikeThis in favor of simple fuzzy match" (T177727) (duration: 00m 57s)
13:27 zfilipin@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Disallow most file types from upload to enwikivoyage (T176647) (duration: 00m 47s)
13:26 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Disallow most file types from upload to enwikivoyage (T176647) (duration: 00m 47s)
13:13 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix amwikimedia site name (T176042) (duration: 00m 47s)
13:10 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: New throttle rule (T177737) (duration: 00m 47s)
12:47 ema: restart pybal on eqiad load balancers to pick up bgp-med config change T165584
12:38 ema: restart pybal on codfw load balancers to pick up bgp-med config change T165584
12:31 ema: restart pybal on ulsfo load balancers to pick up bgp-med config change T165584
12:26 ema: restart pybal on esams load balancers to pick up bgp-med config change T165584
11:59 joal@tin: Finished deploy [analytics/refinery@c3812c2]: Analytics regular weekly deploy (duration: 10m 24s)
11:49 joal@tin: Started deploy [analytics/refinery@c3812c2]: Analytics regular weekly deploy
11:42 hashar: Upgrading Jenkins - T168644
11:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1083 - T174509 (duration: 00m 46s)
11:22 moritzm: installing apt updates from stretch 9.2 point release
11:22 marostegui: Optimize ores_classification table on db1083 - T159753
11:19 reedy@tin: Synchronized wmf-config/Wikibase-production.php: Actually disable injecting RC everywhere but commonswiki and ruwiki (duration: 00m 46s)
11:11 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1092 - T174509 (duration: 00m 47s)
11:08 moritzm: installing whois updates from stretch 9.2 point release
11:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1090 - T174509 (duration: 00m 46s)
10:54 moritzm: installing expect updates from stretch 9.2 point release
10:50 ladsgroup@tin: Synchronized wmf-config/Wikibase-production.php: Disable injecing RC records for commonswiki and ruwiki (T171027) (duration: 00m 47s)
10:38 ladsgroup@tin: Synchronized php-1.31.0-wmf.2/extensions/Wikidata/extensions/Wikibase/client: Re-instate $wgWBClientSettings[injectRecentChanges] (T171027) (duration: 00m 56s)
10:35 hoo: Started dumpwikidatajson.sh on snapshot1007 after T169680 disruptions
10:30 volans: rebooting snapshot1007 for NFS stuck - T169680
10:21 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s7 - T153033
10:21 volans: rebooting snapshot1006 for NFS stuck - T169680
10:15 elukey: rebooting snapshot1001 for NFS stuck - T169680
10:00 volans: rebooting snapshot1005 for NFS stuck - T169680
09:51 volans: rebooting dataset1001 for NFS stuck - T169680
09:51 ema: test setting bgp med on lvs3001/3003 T165584
09:50 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s5 - T153033
09:43 hashar: restarting Jenkins. Deadlock in SSHSlave plugin that causes memory to leak quite rapidly - T177749
09:34 moritzm: installing vim security updates
09:02 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2075 - T170662 (duration: 00m 47s)
08:56 volans: disabled puppet on 'stat100[5-6]*,snapshot100[1,5-7]*,dataset1001*' to manually recover from nfsd stuck - T169680
08:52 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s4 - T153033
08:51 marostegui: Optimize pagelinks and templatelinks on db1092 - T174509
08:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1092 - T174509 (duration: 00m 47s)
08:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1096 - T174509 (duration: 00m 47s)
08:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1066 on s1 to and move db1072 to s3 instead - T172679 (duration: 00m 47s)
08:12 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Move db1066 from s1 to s3 - T172679 (duration: 01m 25s)
07:33 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s2 - T153033
07:16 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2079 to the config - T170662 (duration: 00m 47s)
07:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2079 to the config - T170662 (duration: 00m 47s)
07:01 ema: cp4022 - backend restart, mailbox lag, cache_upload
06:56 marostegui: Stop MySQL on db2075 to use it to clone db2079 - T170662
06:56 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2075 - T170662 (duration: 00m 46s)
06:25 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s6 - T153033
06:14 marostegui: Optimize pagelinks and templatelinks on db1039 - T174509
06:11 marostegui: Optimize pagelinks and templatelinks on db1050 - T174509
06:08 marostegui: Optimize pagelinks and templatelinks on db1096 - T174509
06:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1096 - T174509 (duration: 00m 47s)
05:59 marostegui: Optimize pagelinks and templatelinks on db1091 - T174509
05:59 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1091 - T174509 (duration: 00m 47s)
05:53 marostegui: Optimize pagelinks and templatelinks on db1090 - T174509
05:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1090 - T174509 (duration: 00m 47s)
05:44 marostegui: Optimize pagelinks and templatelinks on db1083 - T174509
05:38 marostegui: Stop MySQL on db1083 to upgrade it
05:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1083 - T174509 (duration: 00m 47s)
02:48 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Oct 9 02:48:21 UTC 2017 (duration 6m 54s)
02:41 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 13m 49s)

2017-10-08

21:40 hoo: Killed "dumpwikidatardf.sh truthy nt" (wikidata truthy dump) on snapshot1007, got stuck after T169680.
11:28 volans: ack-ed puppet not running on stat100[5-6],snapshot100[1,5-7] due to NFSD stuck on dataset1001 - T169680
08:19 elukey: restart varnish backend on cp4026 to stop 503s

2017-10-07

22:46 reedy@tin: Synchronized php-1.31.0-wmf.2/resources/: T177705 (duration: 00m 48s)
22:44 reedy@tin: Synchronized php-1.31.0-wmf.2/includes/specials/SpecialBlock.php: T177705 (duration: 00m 49s)

2017-10-06

18:58 ejegg: updated CiviCRM from 197b22d to 276cbfd
18:38 catrope@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/: Unbreak preference changes (T177524) (duration: 00m 47s)
18:23 arlolra@tin: Finished deploy [parsoid/deploy@e437c93]: Updating Parsoid to 772e11bf (duration: 12m 41s)
18:10 arlolra@tin: Started deploy [parsoid/deploy@e437c93]: Updating Parsoid to 772e11bf
15:30 moritzm: repooling mw2145 (API server). it was previously depooled, but there's no tickets or SAL entries indicating hardware maintenance, so it was probably simply forgotten to repool it
15:25 moritzm: repooling mw2203 (API server). it was previously depooled, but there's no tickets or SAL entries indicating hardware maintenance, so it was probably simply forgotten to repool it
12:54 moritzm: repooling mw2248-mw2250 (job runners). they were previosly depooled, but there's no tickets or SAL entries indicating hardware maintenance, so they were probably simply forgotten to repool
11:46 bblack: cp1* (eqiad caches) - upgrade to nginx-1.13.5-1+wmf1~jessie1 to match all the other sites
11:34 moritzm: repooling mw2244/mw2245 (image scalers). they were previosly depooled, but there's no tickets or SAL entries indicating hardware maintenance, so they were probably simply forgotten to repool
11:21 moritzm: installing gdk-pixbuf security updates
11:10 elukey: rolling restart of all the druid daemons on druid100[1-6] to pick up new logging changes
11:03 moritzm: installing git security updates on trusty (Debian already fixed)
10:51 moritzm: upgrading HHVM on script runners to HHVM 3.18.5
10:47 moritzm: upgrade remaining video scalers in codfw to HHVM 3.18.5
10:43 elukey: restart replication (mysql+ eventlogging_sync) on dbstore1002 after mysql restart
10:17 moritzm: upgrade remaining job runners in codfw to HHVM 3.18.5
09:53 elukey: stop replication (eventlogging_sync + mysql) on dbstore1002 as prep step for mysql restart
09:17 hashar: Restarted the CI Jenkins for plugin upgrades
09:01 moritzm: upgrade remaining app servers in codfw to HHVM 3.18.5
08:40 moritzm: upgrade remaining image scalers in codfw to HHVM 3.18.5
08:09 ema: upgrade pybal to 1.14.0 on eqiad primary LVSs
07:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1076 original weight - T174054 (duration: 00m 48s)
07:23 ema: upgrade pybal to 1.14.0 on eqiad secondary LVSs
07:18 moritzm: upgrade remaining API servers in codfw to HHVM 3.18.5
07:11 legoktm: live hacking on mwdebug1002
07:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1076 weight - T174054 (duration: 00m 47s)
06:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1076 weight - T174054 (duration: 00m 47s)
06:35 ema: set max_connections=0 on cp3032 varnish-be
06:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1106 to the config - T172679 (duration: 00m 47s)
06:21 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1106 to the config - T172679 (duration: 00m 47s)
06:00 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1076 with low weight - T174054 (duration: 00m 46s)
05:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1053 - T174509 (duration: 00m 48s)
05:35 marostegui: Stop MySQL on db1076 for an upgrade

2017-10-05

23:40 ejegg: updated CiviCRM from cca1a9d to 197b22d
23:37 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/resources/lib/jquery/jquery.migrate.js: sync https://gerrit.wikimedia.org/r/#/c/382530 and https://gerrit.wikimedia.org/r/#/c/382529 (duration: 00m 47s)
23:34 XioNoX: upgrading ps1-d2-eqiad.mgmt.eqiad.wmnet (unused)
23:23 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch/includes/BaseInterwikiResolver.php: sync https://gerrit.wikimedia.org/r/#/c/382625/ (duration: 00m 47s)
23:21 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT https://gerrit.wikimedia.org/r/#/c/382611/ (duration: 00m 47s)
22:47 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch/includes/: Sync CirrusSearch extension again for good measure (duration: 00m 51s)
22:43 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch: Sync CirrusSearch extension again for good measure (duration: 00m 57s)
22:39 twentyafterfour@tin: Synchronized wmf-config/CirrusSearch-common.php: sync https://gerrit.wikimedia.org/r/#/c/382474/ (duration: 00m 47s)
22:37 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch/: deploy https://gerrit.wikimedia.org/r/#/c/382618/ (duration: 00m 58s)
22:13 awight@tin: Finished deploy [ores/deploy@42c5663]: Cause ORES service restart (duration: 00m 19s)
22:13 awight@tin: Started deploy [ores/deploy@42c5663]: Cause ORES service restart
22:04 bd808@tin: Finished deploy [striker/deploy@fbb9019]: Prevent tools from being named with invalid Kubernetes namespace labels (T176681) (duration: 00m 33s)
22:04 bd808@tin: Started deploy [striker/deploy@fbb9019]: Prevent tools from being named with invalid Kubernetes namespace labels (T176681)
21:35 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.31.0-wmf.2 refs T174358
21:32 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch: sync CirrusSearch to deploy https://gerrit.wikimedia.org/r/#/c/382605/ refs T177535 (duration: 01m 06s)
21:22 ejegg: updated CiviCRM from 59e3e00 to cca1a9d
21:13 twentyafterfour: Getting the train back on track: moving to wmf.2 as soon as https://gerrit.wikimedia.org/r/#/c/382605/1 merges
20:47 twentyafterfour: 1.31.0-wmf.2 is now blocked by T177535
20:28 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.31.0-wmf.1 refs T174358
20:26 twentyafterfour: rolling back to 1.31.0-wmf.1
20:11 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.31.0-wmf.2 refs T174358
20:10 twentyafterfour: Promote all wikis to 1.31.0-wmf.2 refs T174358 (currently there are no blockers and no significant logspam)
19:59 thcipriani@tin: Synchronized php-1.31.0-wmf.1/extensions/Echo: SWAT: revert "Use User Ids instead of User Names for Echo Mute" (duration: 00m 55s)
19:58 andrewbogott: upgrading wikitech-static to REL1_30
19:55 thcipriani@tin: Synchronized php-1.31.0-wmf.2/extensions/Echo: SWAT: revert "Use User Ids instead of User Names for Echo Mute" (duration: 00m 52s)
19:49 thcipriani@tin: Synchronized php-1.31.0-wmf.2/extensions/VisualEditor/extension.json: SWAT: Move 'parentheses' message to MWSaveDialog's modules T177446 (duration: 00m 50s)
19:45 thcipriani@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict: SWAT: Dont register simulate page when not enabled as a BF (duration: 00m 52s)
19:05 thcipriani@tin: Synchronized php-1.31.0-wmf.1/extensions/Echo: SWAT: Use User Ids instead of User Names for Echo Mute T173475 (duration: 00m 55s)
18:58 thcipriani@tin: Synchronized php-1.31.0-wmf.2/extensions/Echo: SWAT: Use User Ids instead of User Names for Echo Mute T173475 (duration: 00m 56s)
18:53 bblack: cp3* upgrade nginx to 1.13.5-1+wmf1~jessie1
18:38 urandom: restbase-ng: lowering compactionthroughput to 4 (current/5 jbod devices)
18:26 thcipriani@tin: Synchronized wmf-config: SWAT: Enable structured change filters by default on all wikis except FlaggedRevs wikis using non-protection modes T177445 (duration: 00m 54s)
18:13 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable jQuery 3 on all wikis T124742 (duration: 00m 52s)
17:27 cmjohnson1: disable puppet db1024
16:53 moritzm: upgrade remaining app servers in eqiad to HHVM 3.18.5
16:16 XioNoX: failure test for pfw3-codfw
15:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1078 - T174054 (duration: 00m 50s)
15:25 marostegui: Pulling out one disk from db1078 - T174054
15:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1078 - T174054 (duration: 00m 51s)
15:16 marostegui: Pulling out one disk from db1076 - T174054
14:53 hashar: All Jenkins infra is now on Java 8 - T162828
14:52 marostegui: Sanitize db1095 for amwikimedia - T176043
14:48 hashar: Restarting Jenkins
14:27 moritzm: upgrade remaining eqiad video scalers to HHVM 3.18.5
13:57 ladsgroup@tin: Synchronized wmf-config/Wikibase-production.php: Enable Statement usage tracking on kowiki and trwiki (T151717) (duration: 00m 50s)
13:51 ladsgroup@tin: Synchronized wmf-config/Wikibase-production.php: (no justification provided) (duration: 00m 50s)
13:49 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Document task reference for amwikimedia logo (T176042) (duration: 00m 54s)
13:43 Amir1: ladsgroup@terbium:~$ mwscript createAndPromote.php --wiki=amwikimedia --createAndPromote=sysop,bureaucrat 'Ladsgroup' "password removed obviously" --force (T176042)
13:35 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Fix amwikimedia logo (T176042) (duration: 00m 50s)
13:26 ladsgroup@tin: Synchronized php-1.31.0-wmf.2/extensions/WikimediaMessages/i18n/wikimediaprojectnames: (no justification provided) (duration: 00m 49s)
13:15 godog: test upgrade node-exporter in thumbor/swift/services/puppet3-diffs/monitoring projects - T166561
13:11 ladsgroup@tin: Synchronized wmf-config/interwiki.php: Create amwikimedia (T176042) (duration: 00m 51s)
13:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1089 - T174509 (duration: 00m 50s)
12:58 ladsgroup@tin: Synchronized multiversion/MWMultiVersion.php: Create amwikimedia (T176042) (duration: 00m 50s)
12:53 ladsgroup@tin: Synchronized static/images/project-logos/: Create amwikimedia (T176042) (duration: 00m 50s)
12:52 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Create amwikimedia (T176042) (duration: 00m 50s)
12:50 ladsgroup@tin: rebuilt wikiversions.php and synchronized wikiversions files: Create amwikimedia (T176042)
12:49 ladsgroup@tin: Synchronized dblists: Create amwikimedia (T176042) (duration: 00m 50s)
12:36 ladsgroup@tin: Synchronized dblists: (no justification provided) (duration: 00m 51s)
12:22 moritzm: installing poppler security updates on trusty
11:57 moritzm: upgrade remaining eqiad video scalers to HHVM 3.18.5
11:29 jynus: rebuilding globalimagelinks table on dbstore1001:s4
10:35 hashar: contint2001: switched java to version 8 as well as all the jre/jdk utilities managed by alternatives - T162828
10:26 moritzm: upgrade remaining eqiad API servers to HHVM 3.18.5
10:23 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/includes/TwoColConflictHooks.php: REVERT: Dont register simulate page when not enabled as a BF PT1/3 (duration: 00m 50s)
10:22 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/extension.json: REVERT: Dont register simulate page when not enabled as a BF PT2/3 (duration: 00m 50s)
10:17 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/extension.json: Dont register simulate page when not enabled as a BF PT2/3 (duration: 00m 50s)
10:16 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/includes/TwoColConflictHooks.php: Dont register simulate page when not enabled as a BF PT1/3 (duration: 00m 51s)
10:14 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/includes/SpecialConflictTestPage/SpecialConflictTestPage.php: Fix SimulateTwoColEditConflict when using a page with ns (duration: 00m 50s)
10:09 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/WikimediaEvents/WikimediaEventsHooks.php: 31-wmf.2 WikimediaEvents: WMDE: update $campaignStartTimestamp (duration: 00m 50s)
10:07 addshore@tin: Synchronized php-1.31.0-wmf.1/extensions/WikimediaEvents/WikimediaEventsHooks.php: 31-wmf.1 WikimediaEvents: WMDE: update $campaignStartTimestamp (duration: 00m 52s)
10:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1076 - T174054 (duration: 00m 50s)
09:52 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1099 - T174509 (duration: 00m 50s)
08:54 marostegui: Stop MySQL on db1104 to clone db1106 - T172679
08:53 gehel: rolling restart of wdqs to pick up new GC options - T175919
08:41 moritzm: upgrade remaining eqiad job runners to HHVM 3.18.5
08:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1105 to the config - T172679 (duration: 00m 50s)
08:39 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1105 to the config - T172679 (duration: 00m 52s)
08:29 marostegui: Optimize pagelinks and templatelinks on s5 - db1105 - T174509
08:21 moritzm: upgrade remaining eqiad image scalers to HHVM 3.18.5
08:12 marostegui: Optimize templatelinks and pagelinks on s1 s2 s4 s5 s6 and s7 on labsdb1009 - T174509
08:03 jynus: putting labsdb1009 under maintenance
07:54 moritzm: upgrade mw1221-mw1235 (API servers) to HHVM 3.18.5
07:26 legoktm: running batched query of "DELETE FROM linter WHERE linter_cat=12" on all wikis to clear out mostly bogus html5-misnesting category
07:11 ema: update pybal to 1.14.0 on esams primary LVSs
07:06 ema: update pybal to 1.14.0 on esams secondary LVSs
07:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1099 - T174509 (duration: 00m 52s)
07:01 marostegui: Optimize templatelinks and pagelinks on db1099 - T174509
06:56 moritzm: upgrade mw1238-mw1258 (app servers) to HHVM 3.18.5
06:40 marostegui: Stop MySQL on db1104 to clone db1105 from it - T172679
06:32 marostegui: Optimize templatelinks and pagelinks on db1069 - T174509
06:31 marostegui: Optimize templatelinks and pagelinks on db1053 - T174509
06:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1053 - T174509 (duration: 00m 50s)
06:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1089 - T174509 (duration: 00m 49s)
06:24 marostegui: Optimize templatelinks and pagelinks on db1089 - T174509
06:17 marostegui: Optimize s2.templatelinks and enwiki.pagelinks on codfw master db2017 (without replication) - T174509
06:13 marostegui: Optimize enwiki.templatelinks and enwiki.pagelinks on codfw master db2048 (without replication) - T174509
06:01 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2035 and db2056 - T174509 (duration: 00m 52s)
05:57 elukey: restart varnish backend on cp3040
05:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1097 - T174509 (duration: 01m 09s)
05:42 elukey: restart varnish backend on cp3030
02:59 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Oct 5 02:59:18 UTC 2017 (duration 7m 23s)
02:51 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 09m 01s)
02:29 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 09m 51s)
00:06 catrope@tin: Synchronized php-1.31.0-wmf.2/resources/src/mediawiki.rcfilters/mw.rcfilters.init.js: T176652 (duration: 00m 49s)
00:04 catrope@tin: Synchronized php-1.31.0-wmf.2/extensions/WikimediaEvents/modules/ext.wikimediaEvents.recentChangesClicks.js: T176652 (duration: 00m 50s)

2017-10-04

23:55 eileen: update process_control to 10161f8
23:53 catrope@tin: Synchronized php-1.31.0-wmf.2/extensions/TimedMediaHandler/MwEmbedModules/EmbedPlayer/resources/jquery.embedPlayer.js: Fix jqmigrate warning (T169385) (duration: 00m 50s)
23:51 catrope@tin: Synchronized php-1.31.0-wmf.2/extensions/ORES/includes/Hooks.php: ORES highlights for RCFilters (T172757) (duration: 00m 50s)
23:49 catrope@tin: Synchronized php-1.31.0-wmf.2/resources/src/: ORES highlights for RCFilters (T172757) (duration: 00m 50s)
23:36 catrope@tin: Synchronized php-1.31.0-wmf.2/includes/changes/ChangesListFilter.php: ORES highlights for RCFilters (T172757) (duration: 00m 50s)
23:32 catrope@tin: Synchronized php-1.31.0-wmf.2/includes/changes/ChangesListFilterGroup.php: Fix PHP fatal when transcluding Special:Recentchanges (T176236) (duration: 00m 50s)
23:32 Krinkle: Upgrading xhgui on tungsten (operations/software/xhgui.git) from 0.5.2 to 0.8.1 (matching version used in operations/mediawiki-config vendor) - new feature: flamegraphs
23:29 catrope@tin: Synchronized wmf-config/throttle.php: Remove expired rules (duration: 00m 50s)
23:27 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable AbuseFilter runtime profile on ptwiki (T177336) (duration: 00m 51s)
23:19 catrope@tin: Synchronized static/images/mobile/copyright/wikipedia-wordmark-en.svg: Remove unnecessary id attributes (T175670) (duration: 00m 50s)
23:18 catrope@tin: Synchronized php-1.31.0-wmf.2/skins/MinervaNeue/: Correct feature phone threshold detection (T176286) (duration: 00m 53s)
22:35 ejegg: changed DonationInterface cURL timeout for all processors to 12 sec
21:09 arlolra: Updated Parsoid to 477942c3 (T177115)
21:07 hoo: Updated the Wikidata property suggester with data from Monday's JSON dump and applied the T132839 workarounds
21:03 arlolra@tin: Finished deploy [parsoid/deploy@617cdb3]: Updating Parsoid to 477942c3 (duration: 10m 01s)
20:53 arlolra@tin: Started deploy [parsoid/deploy@617cdb3]: Updating Parsoid to 477942c3
20:33 arlolra@tin: Finished deploy [parsoid/deploy@5b80254]: Updating Parsoid to 477942c3 (duration: 00m 58s)
20:32 arlolra@tin: Started deploy [parsoid/deploy@5b80254]: Updating Parsoid to 477942c3
20:18 pnorman: Start regenerating map tiles on codfw for z13-z14 - T176252
19:55 twentyafterfour@tin: Synchronized php: group1 wikis to 1.31.0-wmf.2 refs T174358 (duration: 00m 48s)
19:55 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.31.0-wmf.2 refs T174358
19:38 pnorman: Start regenerating map tiles on eqiad for z13-z14 - T176252
19:25 twentyafterfour: Deploying MediaWiki 1.31.0-wmf.2 to Group 1 wikis refs T174358
19:06 bblack: cp2* (all codfw caches): upgrade nginx to 1.13.5-1+wmf1~jessie1
18:39 ottomata: reverting roll out of LVS service druid-analytics.svc.eqiad.wmnet - this wont' work with hosts inside of Analytics VLAN
18:23 niharika29@tin: Synchronized php-1.31.0-wmf.2/skins/Vector/: Do not special-case ULS and Not logged in in RTL in personal bar T48947, T177312 (duration: 00m 50s)
18:22 niharika29@tin: Synchronized php-1.31.0-wmf.1/skins/Vector/: Do not special-case ULS and Not logged in in RTL in personal bar T48947, T177312 (duration: 00m 51s)
18:09 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable jQuery 3 on most group1 wikis (T124742) (duration: 00m 51s)
17:37 elukey: enabled basic ACLs on the Kafka Jumbo cluster - T173493
17:32 bblack: text@ulsfo - upgrade nginx to 1.13.5-1+wmf1~jessie1
17:32 ottomata: deploying new LVS service for druid-analytics-broker
17:16 moritzm: uploaded icu52 to stretch-wikimedia (co-installable forward port of jessie's ICU for stretch to be used for HHVM)
17:06 bblack: cp4022-26 (now all of upload@ulsfo) - upgrade nginx to 1.13.5-1+wmf1~jessie1
16:58 krinkle@tin: Synchronized php-1.31.0-wmf.2/extensions/TimedMediaHandler/resources/mw.MediaWikiPlayer.loader.js: T175506 (duration: 00m 52s)
16:29 bblack: lvs1001-3: base package upgrades, autoremove, +python-requests
16:24 bblack: recdns: upgrade base packages
16:23 volans: deleted 'R:File = /usr/local/bin/wmf-reimage' on matching hosts T176955
16:20 bblack: maelant: upgrade base packages
16:11 bblack: lvs1004-6: base package upgrades, autoremove, +python-requests
16:06 jynus: performing schema change on labsdb1009/10/11
16:01 ejegg: disabled CiviCRM dedupe jobs
15:40 bblack: acamar (recdns@codfw) - base package upgrades
15:39 bblack: lvs1007-12: base package upgrades, autoremove, +python-requests
15:37 bblack: hydrogen (recdns@eqiad) - base package upgrades
15:35 marostegui: Power off db2010 to decommission it - T175685
15:35 bblack: recdns: apt autoremove -> install python-requests
15:32 bblack: caches: apt autoremove -> install python-requests
15:28 marostegui: Disable puppet on db2010 - it will be decommissioned - T175685
15:24 andrewbogott: rebuilding labvirt1016
14:57 ema: cp4025: restart varnish backend
14:47 XioNoX: replacing faulty VC cable on fasw-c-eqiad
14:45 bblack: base package upgrades (+autoremove) on lvs3001-2
14:43 bblack: base package upgrades (+autoremove) on lvs3004
14:41 bblack: base package upgrades (+autoremove) on lvs3003
14:24 bblack: upgrading nginx to 1.13.5-1+wmf1~jessie1 on cp4021 (first live cache, upload@ulsfo)
14:18 bblack: base package upgrades (+autoremove) on lvs2*
14:10 bblack: base package upgrades (+autoremove) on lvs4*
13:47 zeljkof: EU SWAT finished
13:46 zfilipin@tin: Synchronized php-1.31.0-wmf.2/extensions/Translate/tag/PageTranslationHooks.php: SWAT: Revert "Use Language type hint in hooks" (T177352) (duration: 00m 50s)
13:45 zfilipin@tin: Synchronized php-1.31.0-wmf.2/extensions/Translate/TranslateHooks.php: SWAT: Revert "Use Language type hint in hooks" (T177352) (duration: 00m 50s)
13:41 zfilipin@tin: Synchronized php-1.31.0-wmf.2/extensions/Translate/TranslateHooks.php: php-1.31.0-wmf.2/extensions/Translate/tag/PageTranslationHooks.php SWAT: Revert "Use Language type hint in hooks" (T177352) (duration: 00m 52s)
13:26 zeljkof: one more commit for EU SWAT
13:26 marostegui: Optimize table ores_classification on enwiki codfw master db2048 with replication - might generate lag - T159753
13:22 marostegui: Optimize enwiki.ores_classification on db2069
13:06 zeljkof: EU SWAT finished
13:05 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: New throttle rule (T177370) (duration: 00m 53s)
12:59 jynus: upgrade and reboot labsdb1009
12:56 marostegui: Optimize templatelinks and pagelinks tables on s2 and s7 - labsdb1010 - T174509
12:38 moritzm: upgrading app servers in codfw to HHVM 3.18.5
12:21 moritzm: upgrading image scalers mw1293-mw1295 to HHVM 3.18.5
12:12 moritzm: upgrading HHVM on deployment servers to 3.18.5
12:10 elukey: added two new mediawiki videoscalers - mw1307/1318
12:09 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1318.eqiad.wmnet
12:09 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1307.eqiad.wmnet
12:00 gehel: mediawiki now uses the LVS endpoint for logstash - T175242
11:49 moritzm: upgrading job runners mw1161-mw1167 to HHVM 3.18.5
11:34 moritzm: installing ghostscript security updates
11:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1087 original weight (duration: 00m 47s)
10:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1087 weight (duration: 00m 51s)
10:29 jynus: upgrade and reboot labsdb1011
10:22 jynus: draining labsdb1011 connections
10:18 akosiaris: T177374, fully depool wtp1001-wtp1024
10:18 akosiaris@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=wtp10([01][0-9]|2[0-4]).eqiad.wmnet
10:03 moritzm: install libidn security updates
10:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1087 with low weight (duration: 00m 50s)
09:54 _joe_: restarting hhvm on canaries (both appservers and canaries) T107128
09:47 moritzm: upgrading API app servers mw1189-mw1208 to HHVM 3.18.5
09:45 elukey: rolling restart of aqs nodes to pick up the new logstash lvs config
09:39 hoo: Killed remaining dumpRdf runners on snapshot1007 after two crashed due to the db1087 depool. Auto-restart will do the rest.
09:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1104 to the config - T172679 (duration: 00m 50s)
09:32 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1104 to the config - T172679 (duration: 00m 51s)
09:25 moritzm: installing ocaml security updates on trusty
09:16 marostegui: Reboot db1087 to pick up new kernel
09:09 _joe_: restarting hhvm on mwdebug*, T107128
09:07 marostegui: Optimize templatelinks and pagelinks tables on db1097 (s4) - T174509
09:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1097 - T174509 (duration: 00m 50s)
08:41 akosiaris@puppetmaster1001: conftool action : set/pooled=no; selector: name=wtp10([01][0-9]|2[0-4]).eqiad.wmnet
08:40 mobrovac@tin: Finished deploy [restbase/deploy@8eb758a]: Switch the mobile feeds to the new storage schema and Cassandra 3 (duration: 10m 05s)
08:36 moritzm: upgrading app servers mw1180-mw1188, mw1209-mw1220 to HHVM 3.18.5
08:30 mobrovac@tin: Started deploy [restbase/deploy@8eb758a]: Switch the mobile feeds to the new storage schema and Cassandra 3
08:28 marostegui: Stop MySQL on db2010 as it will be decommissioned - T175685
07:30 jynus: stopping s5 replication on dbstore1002 and converting wikidatawiki.wb_terms into TokuDB
07:11 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db2010 from config as it will be decommissioned - T175685 (duration: 00m 48s)
07:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db2010 from config as it will be decommissioned - T175685 (duration: 00m 48s)
06:53 marostegui: Stop MySQL on db1087 to clone db1104 from it - T172679
06:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1087 - T172679 (duration: 00m 50s)
06:30 marostegui: Optimize templatelinks and pagelinks tables on db1066 - T174509
06:27 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2035 and db2056 - T174509 (duration: 00m 50s)
06:21 elukey: restart varnish backend on cp3043
06:19 marostegui: Optimize pagelinks and templatelinks on dbstore2002 s1 - T174509
06:16 elukey: restart varnish backend on cp3041
06:13 marostegui: Optimize pagelinks and templatelinks tables on s7 codfw master (db2029) this will generate lag - T174509
05:58 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2034, db2055 db2062 - T174509 (duration: 00m 51s)
03:09 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Oct 4 03:09:34 UTC 2017 (duration 7m 15s)
03:02 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 15m 20s)
02:25 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 07m 44s)
00:33 ejegg: re-enabled fundraising queue consumers
00:29 ejegg: re-enabled CiviCRM de-dupe jobs
00:24 ejegg: re-enabled civicrm cron and ingenico jobs
00:20 ejegg: re-enabled fundraising stats and export jobs
00:16 ejegg: re-enabled omnimail import jobs
00:10 ejegg: re-enabled fundraising audit jobs
00:07 ejegg: re-enabled pending queue consumer

2017-10-03

23:52 ejegg: stopped all jobs for process-control upgrade
23:46 pnorman: Start regenerating map tiles on eqiad for z0-z12 - T176252
23:20 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T175971 T176088 Enable RemexHTML on wikitech and eswikiversity (duration: 00m 51s)
23:11 ebernhardson@tin: Synchronized wmf-config/CirrusSearch-common.php: Pre-configure cirrussearch for 5.5.x upgrade (duration: 00m 51s)
22:26 bsitzmann@tin: Finished deploy [mobileapps/deploy@82aa7d6]: Update mobileapps to 5dc0c02 (T175762 T177001 T176525 T176517 T176519) (duration: 06m 14s)
22:20 bsitzmann@tin: Started deploy [mobileapps/deploy@82aa7d6]: Update mobileapps to 5dc0c02 (T175762 T177001 T176525 T176517 T176519)
20:29 pnorman: Start regenerating map tiles on codfw for z0-z12 - T176252
20:17 demon@tin: Synchronized php-1.31.0-wmf.2/extensions/GlobalUserPage/includes/Hooks.php: fix broken thing (duration: 00m 51s)
19:58 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.2
19:33 demon@tin: Synchronized scap/plugins/prep.py: no-op (duration: 00m 51s)
19:17 demon@tin: Finished scap: bootstrap wmf.2 (duration: 33m 50s)
19:01 twentyafterfour: Starting MediaWiki train for 1.30.1-wmf.2
18:43 demon@tin: Started scap: bootstrap wmf.2
18:25 chasemp: change to private vlan for labweb1001 on asw-b-eqiad
18:24 chasemp: change to private vlan for labweb1002 on asw2-d-eqiad
18:24 chasemp: auth-dnsupdate on bahan for 382027
16:51 ejegg: updated payments-wiki from c2a3d43 to b9859f6
15:57 bblack: upgrading basic packages on role::cache::text
15:50 bblack: upgrading basic packages on role::cache::upload
15:42 bblack: upgrading basic packages on role::cache::misc
15:29 mobrovac@tin: Finished deploy [restbase/deploy@02acc45]: (no justification provided) (duration: 06m 27s)
15:24 _joe_: uploading blubber to jessie-wikimedia T175296
15:23 mobrovac@tin: Started deploy [restbase/deploy@02acc45]: (no justification provided)
15:04 mobrovac@tin: Finished deploy [restbase/deploy@02acc45]: (no justification provided) (duration: 05m 14s)
14:59 mobrovac@tin: Started deploy [restbase/deploy@02acc45]: (no justification provided)
14:56 mobrovac@tin: Finished deploy [restbase/deploy@02acc45]: (no justification provided) (duration: 04m 25s)
14:56 bblack: upgrading basic packages on cp1065
14:52 mobrovac@tin: Started deploy [restbase/deploy@02acc45]: (no justification provided)
13:56 zeljkof: EU SWAT finished
13:55 zfilipin@tin: Synchronized php-1.31.0-wmf.1/extensions/Translate/webservices/CxserverWebService.php: SWAT: Make CxserverWebService forward compatible (duration: 00m 45s)
13:46 godog: upgrade grafana to 4.5.2 on krypton - T175980
13:41 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Adjust wgNamespacesToBeSearchedDefault for enwikibooks, fawikibooks and hewikisource (T176906 T176908 T176907) (duration: 00m 46s)
13:34 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Limit thanks for new users at pl.wikipedia to 3 per day" (T176174 T169268) (duration: 00m 46s)
13:26 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ShortUrl Extension on hiwikiversity (T177187) (duration: 00m 46s)
13:25 marostegui: Optimize templatelinks and pagelinks tables on s1, s4 and s6 on labsdb1010 - T174509
13:13 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable NewUserMessage Extension on hiwikiversity (T177188) (duration: 00m 46s)
13:04 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2044 - T174764 (duration: 00m 47s)
10:50 mobrovac@tin: Finished deploy [restbase/deploy@1cc530b]: (no justification provided) (duration: 08m 38s)
10:41 mobrovac@tin: Started deploy [restbase/deploy@1cc530b]: (no justification provided)
08:10 Amir1: cleaning up ores_classification tables (T159753)
08:05 marostegui: Deplo alter table on s3 codfw master (db2018) to add PK to the main tables - T163912
08:02 mobrovac@tin: Finished deploy [restbase/deploy@65f519b]: Create new buckets for feed end points (duration: 09m 57s)
07:52 mobrovac@tin: Started deploy [restbase/deploy@65f519b]: Create new buckets for feed end points
07:20 gehel: killing stuck tileshell and cleaning up /srv/osm_expire on maps-test2001 - T175123
07:05 elukey: restart varnish backend on cp3031 (503s)
07:03 marostegui: Optimize pagelinks and templatelinks tables on labsdb1010 for s5 - T174509
07:01 marostegui: Optimize templatelinks and pagelinks tables on s6 codfw master db2028 (this might generate lag) - T174509
06:46 marostegui: Drop now redundant indexes from pagelinks and templatelinks on s7 - T174509
06:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1066 - T172679 (duration: 00m 47s)
05:54 marostegui: Optimize templatelinks and pagelinks tables on s5 master codfw (db2023): this will generate lag - T174509
05:51 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2049 and db2041 - T174509 (duration: 00m 47s)
03:23 demon@tin: Pruned MediaWiki: 1.30.0-wmf.19 [keeping static files] (duration: 01m 30s)
02:30 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Oct 3 02:30:30 UTC 2017 (duration 6m 39s)
02:23 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 07m 22s)

2017-10-02

23:38 catrope@tin: Synchronized php-1.31.0-wmf.1/extensions/VisualEditor/: T177250 (duration: 00m 47s)
23:36 catrope@tin: Synchronized php-1.31.0-wmf.1/resources/src/mediawiki.rcfilters/dm/mw.rcfilters.dm.FiltersViewModel.js: T177107 (duration: 00m 47s)
23:33 mutante: releases2001 - kill and restart jenkins with systemctl
23:19 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Update WMF mailing address (T173684) and enable jQuery 3 on Wiktionaries (T124742) (duration: 00m 48s)
23:00 mutante: releases1001 - remove and reinstall jenkins
22:56 Reedy: created newsletter tables on officewiki T176199
19:36 ejegg: turned on CiviMail record creation
18:48 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Change the Turkish Wiktionary logo (T176008) (duration: 00m 46s)
18:46 niharika29@tin: Synchronized static/images/project-logos/: Change the Turkish Wiktionary logo (T176008) (duration: 00m 48s)
17:46 gehel@tin: Finished deploy [wdqs/wdqs@f20b726]: latest wdqs GUI + minor fixes (duration: 01m 48s)
17:44 gehel@tin: Started deploy [wdqs/wdqs@f20b726]: latest wdqs GUI + minor fixes
14:08 zeljkof: EU SWAT finished
14:07 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Sandbox menu to Javanese Wikipedia (T176308) (duration: 00m 46s)
14:01 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Extension:Newsletter on officewiki (T176199) (duration: 00m 46s)
13:59 zeljkof: extending EU SWAT for 5-10 minutes
13:51 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Change the zh-classicalwiki logo (T177165) (duration: 00m 45s)
13:50 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Change the zh-classicalwiki logo (T177165) (duration: 00m 46s)
13:44 zfilipin@tin: Synchronized static/images/project-logos/zh_classicalwiki-1.5x.png: static/images/project-logos/zh_classicalwiki-2x.png static/images/project-logos/zh_classicalwiki.png wmf-config/InitialiseSettings.php SWAT: Change the zh-classicalwiki logo (T177165) (duration: 00m 46s)
13:42 godog: roll restart thumbor to apply https://gerrit.wikimedia.org/r/#/c/381045/ - T166699
13:38 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable the Extension:SandboxLink on the nlwikinews (T177170) (duration: 00m 46s)
13:32 zfilipin@tin: Synchronized wmf-config/Wikibase-production.php: SWAT: Wikidata: Dont persist description usages (yet) (T177153) (duration: 00m 45s)
13:27 zfilipin@tin: Synchronized wmf-config/Wikibase-production.php: SWAT: Wikidata: Dont persist description usages (yet) (T177153) (duration: 00m 46s)
13:20 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add eliminator as a priviliged account (T176554) (duration: 00m 46s)
13:17 zfilipin@tin: Synchronized wmf-config/Wikibase-labs.php: wmf-config/Wikibase.php wmf-config/Wikibase-production.php SWAT: labs: Use redis lock manager for dispatching changes of Wikibase (T175109) (duration: 00m 46s)
13:09 zfilipin@tin: Synchronized wmf-config/Wikibase-labs.php: wmf-config/Wikibase.php wmf-config/Wikibase-production.php SWAT: labs: Use redis lock manager for dispatching changes of Wikibase (T175109) (duration: 00m 47s)
12:56 elukey: added two new mediawiki jobrunners - mw131[01]
12:50 marostegui: Optimize tables pagelinks and templatelinks on s5: dbstore2001
11:51 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2039 - T174509 (duration: 00m 46s)
10:06 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1311.eqiad.wmnet
10:06 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1310.eqiad.wmnet
10:04 mobrovac: restbase restarting on restbase1007 to start logging error bodies for T176728
09:55 jynus: stopping and upgrading labsdb1010
09:27 Amir1: finished the cleanup of ores_classification table in enwiki (T159753)
09:12 godog: roll-restart thumbor to upgrade to 1.6 - T172556
09:06 elukey: forced remount of /mnt/hdfs on notebook1002
08:47 gehel: deploying latest tilerator on maps-test only, fix for T175123
08:46 gehel@tin: Finished deploy [tilerator/deploy@b02bd9a]: (no justification provided) (duration: 00m 22s)
08:46 gehel@tin: Started deploy [tilerator/deploy@b02bd9a]: (no justification provided)
08:46 marostegui: Poweroff db2044 for HW maintenance - T174764
08:40 marostegui: Stop MySQL on db2044 to get it ready to replace its mainboard - T174764
08:29 Amir1: finished the cleanup of ores_classification table in wikidatawiki and starting the enwiki one (T159753)
08:24 jynus: stopping dbstore1001 for maintenance
08:12 marostegui: Optimize tables pagelinks and templatelinks on s5: db1100 (already depooled) - T174509
07:59 marostegui: Drop now redundant indexes from pagelinks and templatelinks from s5 - T174509
07:57 Amir1: starting a round of cleanup in ores_classification table in wikidatawiki
07:41 marostegui: Stop MySQL on db1036 as it is going to be decommissioned - https://phabricator.wikimedia.org/T176311
07:32 marostegui: Optimize table pagelinks and templatelinks on s6: dbstore2001 - T174509
07:21 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2039 - T174509 (duration: 00m 46s)
07:20 marostegui: Optimize table pagelinks and templatelinks on s6: db2039 - T174509
07:10 marostegui: Optimize table pagelinks and templatelinks on s1: db2034, db2055 db2062 - T174509
07:06 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2034, db2055 db2062 - T174509 (duration: 00m 46s)
06:55 _joe_: restarting a few API appservers with high cpu load
06:19 elukey: restart varnish backend on cp3040
06:10 elukey: restart varnish backend on cp3033
06:08 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1036 from config files as it will be decommissioned - T176311 (duration: 00m 46s)
06:07 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1036 from config files as it will be decommissioned - T176311 (duration: 00m 48s)
02:43 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Oct 2 02:43:33 UTC 2017 (duration 6m 39s)
02:36 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 08m 42s)

2017-10-01

21:30 madhuvishy: powercycling labvirt1015
18:58 hashar: restarted Jenkins. Was stuck somehow
13:50 elukey: restart hhvm on mw1167 (jobrunner) - hhvm stuck, dump-debug in /tmp/hhvm.9624.bt.

2017-09-30

20:54 catrope@tin: Synchronized wmf-config/CommonSettings.php: Bump $wgResourceLoaderStorageVersion (T176884) (duration: 00m 47s)
00:37 ejegg: increased PayPal Express Checkout cURL timeout to 12 seconds

2017-09-29

22:12 ejegg: updated CiviCRM from 784756b to 59e3e00
21:19 mutante: labstore2003/2004: closed idle screen sessions
20:27 mutante: mwlog1001, oxygen - closing unused screen session
20:20 mutante: install1002/install2002 - closing unused screen sessions (except a tail -f)
20:11 mutante: restbase2001 - closing screen sessions that weren't running anything anymore
19:23 ebernhardson: completed running cirrussearch category moves on commonswiki
18:19 catrope@tin: Synchronized php-1.31.0-wmf.1/includes/libs/CSSMin.php: T176884 (duration: 00m 47s)
18:08 chasemp: remove 'ToolLabs webservice - lighttpd on precise' catchpoint check for Catchpoint
18:07 chasemp: remove labsdb1002 checks on catchpoint (Toolforge)
18:06 chasemp: remove Labs puppetmaster eqiad catchpoint check (for Toolforge)
18:06 chasemp: remove grid-start-precise catchpoint check
18:04 chasemp: update catchpoing test for stream.wikimedia.org to watch https://stream.wikimedia.org/?doc
17:00 ebernhardson: kicking off extra job runners to process jobs moving commonswiki Category pages from general to content serach index
16:37 andrewbogott: re-imaging labvirt1017, 1018 in order to get it on the standard 4.4.0-81-generic kernel
15:52 andrewbogott: re-imaging labvirt1015 in order to get it on the standard 4.4.0-81-generic kernel
13:21 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2070, db2069 and db2042 - T174509 (duration: 00m 47s)
13:18 Amir1: starting a round of cleanup in ores_classification table in enwiki (T159753)
13:09 elukey: depool mw1265 (hhvm 3.18.5) - disk filled up
12:43 bblack: removed numa=isolate sys entries manually (runtime + future boots) from cp4021
11:53 bblack: cp4021: reboot for numa mode switch
11:44 elukey: re-enable job runner daemons on mw130[8,9]
11:15 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2053 db2046 - T174509 (duration: 00m 48s)
11:05 elukey: precautionary stop of jobrunner/jobchron on the new mw130[8,9] - job queue size rapid increase investigation
09:55 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1309.eqiad.wmnet
09:33 ema: upgrade pybal to 1.14.0 on codfw primaries
08:20 ema: upgrade pybal to 1.14.0 on codfw secondaries
05:53 ema: cp4024 repooled T174891
05:21 ema: powerup cp4024 T174891
00:26 ejegg: updated fundraising tools from 69a9627 to 6d4b6f3
00:16 ejegg: updated CiviCRM 2138869 to 784756b
00:07 mutante: releases1001 - starting Jenkins setup wizard with generated admin pass
00:05 mutante: releases1001 - stopped puppet, manually fixing --prefix=/ci setting for jenkins process, killing it, removing init.d file, starting with systemd, jenkins now up T164030

2017-09-28

23:46 awight@tin: Finished deploy [ores/deploy@42c5663]: Cause ORES service restart (duration: 00m 20s)
23:45 awight@tin: Started deploy [ores/deploy@42c5663]: Cause ORES service restart
22:18 MaxSem: Deployed fix for T176247
21:30 ejegg: updated CiviCRM from 3f48cb0 to 2138869
21:23 XenoRyet: Updated SmashPig from dc62203 to d056a13
19:36 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.1
19:34 legoktm: mysql:wikiadmin@db1052 [enwiki]> delete from user_groups where ug_user =17629530 limit 1; (T176798)
19:31 bblack: cp4026 - depool->reboot for bios updates, isolcpus fix
19:28 legoktm: mysql:wikiadmin@db1062 [centralauth]> UPDATE localuser SET lu_attached_method="new" WHERE lu_attached_method is NULL AND lu_name="Rua";
19:14 ejegg: updated fundraising tools 5708cf1 Add 'first_donation_date' column to Silverpop export
19:14 robh: cp4024 initial tests done (takes about 2 hours or so) then prompted for more in depth testing (hit yes, no failures so far) T174891
19:00 bblack: cp4025 - depool->reboot for bios updates, isolcpus fix
18:54 thcipriani@tin: Synchronized php-1.31.0-wmf.1/skins/MinervaNeue/includes/skins/SkinMinerva.php: SWAT: Revision::newFromTitle may return null T176882 (duration: 00m 50s)
18:42 bblack: cp4023 - depool->reboot for bios updates, isolcpus fix
18:37 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create rollbacker group at viwiktionary T176979 (duration: 00m 50s)
18:26 thcipriani@tin: Synchronized docroot/noc: SWAT: copy squid.php->reverse-proxy.php, squid-labs->reverse-proxy-staging PART II T104148 (duration: 00m 49s)
18:25 thcipriani@tin: Synchronized wmf-config: SWAT: copy squid.php->reverse-proxy.php, squid-labs->reverse-proxy-staging T104148 (duration: 00m 51s)
17:58 arlolra: Updated Parsoid to 2f4b9a8c (T176363, T176151, T170832)
17:52 mutante: librenms - renewed LE TLS cert ..via puppet, after Icinga alerted and accidentally still had "do_acme: false" in Hiera
17:46 arlolra@tin: Finished deploy [parsoid/deploy@84342fe]: Updating Parsoid to 2f4b9a8c (duration: 15m 25s)
17:43 bblack: cp4028 - depool->reboot for bios updates, isolcpus fix
17:40 moritzm: installing apache updates on cobalt
17:32 bblack: cp4022 - depool->reboot for bios updates, isolcpus fix
17:30 arlolra@tin: Started deploy [parsoid/deploy@84342fe]: Updating Parsoid to 2f4b9a8c
17:25 marostegui: Run fixStuckGlobalRename.php --wiki=enwiktionary --logwiki=metawiki "CodeCat" "Rua" on terbium - T176985
17:20 bblack: cp4027 - depool->reboot for bios updates, isolcpus fix
17:08 bblack: cp4021 - depool->reboot for bios update
17:05 robh: bios firmware uploaded and awaiting reboot for application on cp402[1235678]
16:48 bd808: Created centralauth.{analytics,web}.db.svc.eqiad.wmflabs Designate CNAME (T176978)
16:26 robh: cp4024 has had ilom and bios firmware updaed per epsa error checking, now re-running epsa utility T174891
16:09 robh: cp4024 bios update in progress (has been for last 5 minutes), letting it alone for a solid 30 to let it finish
16:01 marostegui: Global rename of CodeCat → Rua - T176985
15:22 hoo: Ran scap pull on mwdebug1001 after experiments with T176844
14:54 dcausse: restarting elastic on relforge servers to reinstall prod ltr plugin
14:51 marostegui: Stop MySQL on db1035 as server is going to be decommissioned - T176931
14:45 moritzm: installing apache updates on puppet mastes (will trigger a few failing puppet runs, but ircecho disabled temporarily)
14:42 marostegui: Remove db1035 from tendril - T176931
14:31 moritzm: upgrading mw1276 (API canary) to HHVM 3.18.5
14:28 moritzm: upgrading mwdebug* to HHVM 3.18.5
13:56 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1035 from config as it will be decommissioned - T176931 (duration: 00m 49s)
13:55 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1035 from config as it will be decommissioned - T176931 (duration: 00m 50s)
13:40 zeljkof: EU SWAT finished
13:38 zfilipin@tin: Synchronized php-1.31.0-wmf.1/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: SWAT: ve.init.mw.DesktopArticleTarget: Remove hack for reversed tabs in RTL in Vector (T50017) (duration: 00m 50s)
13:37 moritzm: upgrading mw1262 to mw1265 (canary app servers) to HHVM 3.18.5
13:35 moritzm: uploaded HHVM 3.18.5 for jessie-wikimedia to apt.wikimedia.org
13:26 marostegui: Optimize table on s4 codfw master (db2051), with replication (lag might be generated) - T174509
13:13 ema: bounce pybal on ulsfo primaries to pick up bgp-local-ips config change T103882
13:12 marostegui: Drop redundant indexes from pagelinks and templatelinks on s4 - T174509
13:10 moritzm: upgrading mw1261 to HHVM 3.18.5
13:08 ema: bounce pybal on ulsfo secondaries to pick up bgp-local-ips config change T103882
13:02 marostegui: Optimize template links and pagelinks on db2053 and db2046 - T174509
13:00 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2053 db2046 - T174509 (duration: 00m 49s)
12:48 dcausse: restarting elastic on relforge servers to test a new version of the ltr plugin
12:27 ema: reboot cp402[2-6] to disable numa isolation
10:30 gehel: restart elasticsearch on relforge to validate new logging config - T175242
10:09 elukey: incrementally add mw1312-17 api-appservers to serve live traffic (weights will be raised incrementally)
10:00 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1308.eqiad.wmnet
09:59 elukey: added new mediawiki jobrunner - mw1308
09:52 ema: upgrade pybal to 1.14.0 on lvs1009-10
09:49 aaron@tin: Synchronized wmf-config/CommonSettings.php: Enable logging of post-send DB updates (duration: 00m 55s)
09:42 ema: upgrade pybal to 1.14.0 on lvs400[13]
08:00 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1288.eqiad.wmnet
07:27 moritzm: upgrading app servers in deployment-prep to HHVM 3.18.5
07:14 moritzm: installing apache updates on silver/wikitech.wikimedia.org
06:59 moritzm: installing apache updates on graphite* hosts
06:28 elukey: restart varnish backend on cp3032
05:54 marostegui: Optimize tables pagelinks and templatelinks on dbstore2001 (s2) - T174509
05:53 marostegui: Optimize tables pagelinks and templatelinks on dbstore2002 (s2) - T174509
05:43 marostegui: Run maintain-views on labsdb1001, labsdb1003, labsdb1009, labsdb1010, labsdb1011 for hiwikivoyage - T173027
05:38 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2049 and db2041 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 47s)
05:29 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2064 db2063 - T174509 (duration: 00m 48s)
05:21 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2076 db2067 db2060 - T174509 (duration: 00m 48s)
05:19 tstarling@tin: Finished scap: updating Vector skin for gerrit 381153 (duration: 20m 39s)
04:59 tstarling@tin: Started scap: updating Vector skin for gerrit 381153
03:57 bblack: ulsfo dns-depooled at ~03:51
03:26 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Sep 28 03:26:18 UTC 2017 (duration 6m 10s)
03:20 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 15m 44s)
02:42 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 15m 24s)
00:36 twentyafterfour: upgrading phabricator, expect minimal downtime

2017-09-27

23:39 robh: cp4024 is running hardware testing, leave it alone T174891
23:17 catrope@tin: Synchronized wmf-config/Wikibase.php: Make CirrusSearch default for wbsearchentities on testwikidatawiki T175741 (duration: 00m 48s)
22:34 maxsem@tin: Synchronized php-1.31.0-wmf.1/includes/EditPage.php: https://gerrit.wikimedia.org/r/#/c/381139/1 (duration: 00m 49s)
19:14 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 -> wmf.1
19:13 demon@tin: Synchronized php: symlink bump (duration: 00m 47s)
19:10 demon@tin: Synchronized php-1.31.0-wmf.1/maintenance/backup.inc: unbreak dumps (duration: 00m 49s)
18:46 catrope@tin: Finished scap: SWAT: T176762, T175358, T172023, T174719, and add html5-misnesting to Linter (duration: 22m 55s)
18:39 bblack: re-pooling upload@ulsfo (text@ulsfo remains pooled)
18:23 catrope@tin: Started scap: SWAT: T176762, T175358, T172023, T174719, and add html5-misnesting to Linter
18:21 ejegg: updated payments-wiki from 6104e3d to c2a3d43
17:40 ejegg: updated SmashPig Adyen settings
17:29 bblack: uploaded nginx-1.13.5-1+wmf1 to stretch-wikimedia, and matching nginx-1.13.5-1+wmf1~jessie1 to jessie-wikimedia
16:27 XioNoX: setting local-as to selected transit BGP sessions - T167840
16:00 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4026.ulsfo.wmnet
16:00 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4025.ulsfo.wmnet
15:59 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4023.ulsfo.wmnet
15:58 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4022.ulsfo.wmnet
15:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1103 with low weight - T172679 (duration: 00m 47s)
15:51 ema: lvs4002: upgrade pybal to 1.14.0
15:49 ema: lvs4004: upgrade pybal to 1.14.0
15:28 _joe_: uploaded prometheus-statsd-exporter to stretch-wikimedia T175539
15:24 bblack: rebooting cp402[2356] (bnx2x oops)
15:15 moritzm: revoking salt certs for WMCS
15:12 mlitn@tin: Synchronized wmf-config/CommonSettings-labs.php: Remove 3D beta config (duration: 00m 47s)
15:11 mlitn@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Remove 3D beta config (duration: 00m 48s)
15:10 mlitn@tin: Synchronized wmf-config/CommonSettings.php: Load 3D extension (duration: 00m 49s)
15:09 mlitn@tin: Synchronized wmf-config/InitialiseSettings.php: Config 3D to be loaded on test, test2 (duration: 00m 48s)
15:00 mlitn@tin: Finished scap: Enable 3D extension (duration: 37m 09s)
14:58 ema: lvs1007: upgrade pybal to 1.14.0
14:45 elukey: rolling restart of all the Yarn nodemanager daemons on analytics1028-1068
14:43 bblack: cp1008/pinkunicorn upgraded to test build of nginx-1.13.5-1+wmf1
14:28 volans: uploaded cumin_1.2.1-1_amd64.deb to apt.wikimedia.org jessie-wikimedia
14:24 akosiaris: change wtp1001 to wtp1024 weights to 5 from 15 in preparation for deprecation
14:24 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1024.eqiad.wmnet
14:24 mlitn@tin: Started scap: Enable 3D extension
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1023.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1022.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1021.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1020.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1019.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1018.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1017.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1016.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1015.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1014.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1013.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1012.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1011.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1010.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1009.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1008.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1007.eqiad.wmnet
14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1006.eqiad.wmnet
14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1005.eqiad.wmnet
14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1004.eqiad.wmnet
14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1003.eqiad.wmnet
14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1002.eqiad.wmnet
14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1001.eqiad.wmnet
14:17 akosiaris: T175242 restart eventstreams across the fleet to pick up the change in a rolling restart manner with a batch size of 2
14:16 akosiaris: T175242 restart parsoid across the fleet to pick up the change in a rolling restart manner with a batch size of 5
14:03 akosiaris: T175242 restart tilerator, tileratorui, restbase across the fleet to pick up the change in a rolling restart manner with a batch size of 2
13:43 akosiaris: T175242 re-enable puppet across aqs kafka maps maps-test ores restbase restbase-dev sca scb wtp clusters for merging https://gerrit.wikimedia.org/r/#/c/376500/. Run puppet as well in a batched execution
13:42 anomie@tin: Synchronized php-1.31.0-wmf.1/includes/specials/SpecialWatchlist.php: SWAT: {{gerrit|380968}} Fix watchlist "in the last X hours" display (duration: 00m 48s)
13:42 elukey: raised Hadoop HDFS namenode master daemon max heap size to 6G (prev 4G) on analytics100[12]
13:31 akosiaris: T175242 eventstreams requires manual restart
13:25 akosiaris: T175242 parsoid requires manual restart
13:15 akosiaris: T175242 restbase requires manual restart
13:11 addshore: SWAT done
13:11 addshore@tin: Synchronized php-1.31.0-wmf.1/extensions/TwoColConflict/modules/ext.TwoColConflict.BaseVersionSelector.css: SWAT: Address changes in the label style in OOUI (duration: 00m 46s)
13:09 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add autopatrolled user group to dty.wikipedia (T176709) (duration: 00m 47s)
13:01 akosiaris: T175242 tilerator and tileratorui need manually restart
12:54 akosiaris: T175242 enabled puppet in aqs kafka maps maps-test selected hosts and ran puppet manually.
12:47 akosiaris: T175242 disable puppet across aqs kafka maps maps-test ores restbase restbase-dev sca scb wtp clusters for merging https://gerrit.wikimedia.org/r/#/c/376500/
12:20 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1103 to the array of hosts (duration: 00m 48s)
12:19 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1103 to the array of hosts (duration: 00m 49s)
12:14 moritzm: installing apache updates on einsteinium/tegmen
12:06 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on bnwiki (T176771) (duration: 00m 49s)
12:05 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on bnwiki (T176771) (duration: 01m 00s)
09:45 marostegui: Stop mysql on db1035 to copy its data to db1103 - T172679
09:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1035 to transfer its data to db1103 - T172679 (duration: 00m 48s)
08:55 mobrovac@tin: Finished deploy [changeprop/deploy@7a3dc66]: Use the /page/title/{title}/{revision} end point for revision-visibility-change - T158100 (duration: 01m 26s)
08:53 mobrovac@tin: Started deploy [changeprop/deploy@7a3dc66]: Use the /page/title/{title}/{revision} end point for revision-visibility-change - T158100
08:43 mobrovac@tin: Finished deploy [restbase/deploy@1cc530b]: Introduce the /page/title/{title}/{revision} end point - T158100 (duration: 10m 01s)
08:34 elukey: raise traffic weights to 30 for mw13[19-28] incrementally - T165519
08:33 mobrovac@tin: Started deploy [restbase/deploy@1cc530b]: Introduce the /page/title/{title}/{revision} end point - T158100
08:16 marostegui: Optimize tables pagelinks templatelinks on db2064 and db2063 - T174509
08:16 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2064 db2063 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 48s)
07:55 marostegui: Optimize pagelinks and template links tables on db2076 db2067 db2060 - T174509
07:52 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2076 db2067 db2060 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 48s)
07:37 marostegui: Optimize pagelinks and template links tables on db2070 db2069 db2042 db2016 - T174509
07:27 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2070, db2069 and db2042 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 48s)
07:17 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2071 and db2072 - T174509 (duration: 00m 47s)
06:57 marostegui: Deploy alter table on s6 codfw - T174509
06:55 marostegui: Deploy alter table on db1052 (enwiki master) - T174509
06:37 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add task number to db2044 line (duration: 00m 48s)
06:37 ema: restart varnish-be on cp3040
06:36 moritzm: installing git security updates (our config is not affected, but still fixing the underlying issue)
06:33 ema: restart varnish-be on cp3041
06:29 marosteg1i: Reboot db2044 after storage failure - T174764
06:15 elukey: restart varnish backend on cp3043
05:41 madhuvishy: service nova-fullstack restart on labnet1001
03:17 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Sep 27 03:17:22 UTC 2017 (duration 6m 22s)
03:11 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 15m 55s)
02:32 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 08m 42s)

2017-09-26

23:25 thcipriani@tin: Synchronized php-1.31.0-wmf.1/extensions/VisualEditor/modules/ve-mw/ui/tools/ve.ui.MWSignatureTool.js: SWAT: Revert MWSignatureTool case (duration: 00m 48s)
23:20 thcipriani@tin: Synchronized php-1.30.0-wmf.19/resources/src/mediawiki.rcfilters/mw.rcfilters.init.js: SWAT: RCFilters: Log performance data T176652 (duration: 00m 48s)
23:19 thcipriani@tin: Synchronized php-1.31.0-wmf.1/resources/src/mediawiki.rcfilters/mw.rcfilters.init.js: SWAT: RCFilters: Log performance data T176652 (duration: 00m 51s)
22:03 no_justification: gerrit master restart, back momentarily
20:32 mutante: phab1001 - restarted apache after upgrade
20:30 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2044 - T174764 (duration: 00m 50s)
20:20 mutante: phab1001 (prod phab) - upgrade Apache package
20:15 mutante: phab2001 - install Apache and MariaDB package upgrades
19:16 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.31.0-wmf.1
18:25 mepps: Updated crm from 89b8007 to 3f48cb0
18:00 demon@tin: Finished scap: 1.31.0-wmf.1 bootstrap + testwiki (duration: 47m 56s)
17:12 demon@tin: Started scap: 1.31.0-wmf.1 bootstrap + testwiki
15:50 volans: powecycled cp2012, no ping or ssh console stuck
15:18 XioNoX: starting eqiad frack switch to new infra - T174218
15:06 gehel: restarting blazegraph on all wdqs nodes for heap resize - T175919
14:47 mobrovac@tin: Finished deploy [restbase/deploy@f989dd9]: Remove the old mobileapps module completely - T169940 (duration: 20m 35s)
14:36 marostegui: Optimize tables pagelinks and templatelinks on db2071 and db2072 from s1 - T174509
14:27 mobrovac@tin: Started deploy [restbase/deploy@f989dd9]: Remove the old mobileapps module completely - T169940
14:26 hoo: mwscript refreshLinks.php --wiki elwiki --namespace 0 on terbium has finished (T151717)
14:12 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2071 and db2072 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 44s)
13:39 marostegui: Deploy alter table on s6 master (with replication enable, so lag will be generated) db2028 - T163979
13:25 hashar@tin: Synchronized wmf-config/filebackend.php: Expose Thumbor swift username - T144479 (duration: 00m 44s)
13:24 hashar@tin: Synchronized wmf-config/CommonSettings.php: Enable asia-specific Navigation Timing metric - T169522 (duration: 00m 47s)
13:22 godog: upgrade grafana to 4.5.2 on labmon1001 - T175980
13:15 hashar@tin: Synchronized wmf-config/CommonSettings.php: Disable RelatedArticles instrumentation on all wikis - T174944 (duration: 00m 45s)
13:02 marostegui: Drop redundant indexes from s6 - T174509
13:00 elukey: add mw132[7,8] to live traffic (new appservers) - weights will be increased incrementally from 5 to 20
12:44 gehel: restarting blazegraph on wdqs1004 / wdqs2001 for heap resive - T175919
12:27 marostegui: Drop redundant indexes from s2 - T174509
12:12 marostegui: Drop redundant indexes from enwiki on eqiad - T174509
12:03 kartik@tin: Finished deploy [cxserver/deploy@aa232ee]: Update cxserver, registry reconstruction (duration: 03m 03s)
12:00 kartik@tin: Started deploy [cxserver/deploy@aa232ee]: Update cxserver, registry reconstruction
11:58 moritzm: uploaded prometheus-apache-exporter 0.3-1+deb9u1 to apt.wikimedia.org/stretch-wikimedia
11:57 marostegui: Drop redundant indexes from enwiki on codfw - T174509
11:49 moritzm: uploaded prometheus-hhvm-exporter 0.3.1+deb9u1 to apt.wikimedia.org/stretch-wikimedia
11:30 kartik@tin: (no justification provided)
11:29 kartik@tin: Finished deploy [cxserver/deploy@f1d4851]: Update cxserver, registry reconstruction (duration: 00m 52s)
11:28 kartik@tin: Started deploy [cxserver/deploy@f1d4851]: Update cxserver, registry reconstruction
10:52 volans: uploaded cumin_1.1.1-1_amd64.deb to apt.wikimedia.org jessie-wikimedia
10:34 joal@tin: Finished deploy [analytics/refinery@5d806c6]: Regular analytics deploy (duration: 08m 00s)
10:26 joal@tin: Started deploy [analytics/refinery@5d806c6]: Regular analytics deploy
10:20 _joe_: rebooting copper (again)
09:12 elukey: add mw132[4,5,6] to live traffic (new appservers) - weights will be increased incrementally from 5 to 20
09:10 _joe_: rebooting copper to verify docker setup is correct
08:46 moritzm: installing apache updates on mw* app servers
08:44 godog: run puppet on ms-fe* to reduce conntrack generated by swift internal clients towards backends - T173731
08:24 godog: roll-restart thumbor to upgrade to 1.5 - T173804
08:09 godog: roll-restart swift-proxy to apply https://gerrit.wikimedia.org/r/#/c/380483/ - T161535
08:08 marostegui: Drop old temporary tar.gz files from dbstore1001 to get back some disk space
08:07 marostegui: Drop table trackbacks from s3 - T175051
07:48 mattflaschen@tin: Synchronized wmf-config/CommonSettings-labs.php: RCFilters: Beta Cluster only (duration: 00m 46s)
07:46 marostegui: Drop trackbacks table from s7 - T175051
07:28 marostegui: Drop trackbacks table from s6 - T175051
07:16 moritzm: installing perl security updates
06:45 _joe_: restarting varnish backend on cp3033, throwing 503s
06:29 mobrovac@tin: Finished deploy [cpjobqueue/deploy@17c3833]: (no justification provided) (duration: 00m 28s)
06:28 mobrovac@tin: Started deploy [cpjobqueue/deploy@17c3833]: (no justification provided)
06:18 marostegui: Drop table trackbacks from s4 - T175051
06:06 marostegui: Drop table trackbacks from s5 (only exists on dewiki) - T175051
05:48 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2047 - T176573 (duration: 00m 44s)
05:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove old db1055 comments (duration: 00m 46s)
02:29 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Sep 26 02:29:31 UTC 2017 (duration 5m 52s)
02:23 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 07m 26s)

2017-09-25

23:58 ejegg: updated payments-wiki from d2ce697 to 6104e3d
23:15 mutante: gerrit2001 reinstalled with stretch, revoked old puppet cert, accepted new puppet cert, initial run that will do base and all the gerrit things at once.. (T168562)
22:53 mutante: gerrit2001 - PXE booting
22:34 mutante: gerrit2001 - schedule downtime for reinstall with strech, reboot imminent
20:57 hoo: Manually pruned the sites and site_identifiers table on hiwikivoyage, than ran populateSitesTable.php. (T173030)
20:37 arlolra: Updated Parsoid to 4230c27b (T176425, T173029)
20:30 arlolra@tin: Finished deploy [parsoid/deploy@376faad]: Updating Parsoid to 4230c27b (duration: 16m 52s)
20:13 arlolra@tin: Started deploy [parsoid/deploy@376faad]: Updating Parsoid to 4230c27b
19:58 XioNoX: renumbering ams BGP communities - T167840
19:25 robh@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp4021.ulsfo.wmnet
19:02 robh: cp4021 coming down for memory swap
19:02 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp4021.ulsfo.wmnet
18:54 MaxSem: started populating ip_changes on group2 wikis
18:38 demon@tin: Synchronized scap/plugins/clean.py: no-op/consistency (duration: 00m 45s)
18:22 ejegg: updated CiviCRM from 45c56cf to 89b8007
17:58 demon@tin: Pruned MediaWiki: 1.30.0-wmf.15 (duration: 02m 46s)
17:07 gehel@tin: Finished deploy [wdqs/wdqs@3eec185]: Deploy new Blazegraph binary & GUI update (duration: 01m 51s)
17:05 gehel@tin: Started deploy [wdqs/wdqs@3eec185]: Deploy new Blazegraph binary & GUI update
17:02 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: Disable the updateBetaFeaturesUserCounts logging channel - T175637 (duration: 00m 46s)
16:37 demon@tin: Synchronized php-1.30.0-wmf.19/maintenance/getConfiguration.php: getConfiguration: Don't bail when a valid variable is set null (duration: 00m 47s)
15:57 urandom: T169940: Converting restbase-ng data tables to size-tiered compaction
14:39 gehel: dropping badly named apifeature index from cirrus elasticsearch codfw/eqiad - T176430
14:31 gehel: rolling restart of logstash for config change - T176430
14:13 gehel: rolling restart of logstash for config change - T176430
13:46 zeljkof: EU SWAT finished
13:45 zfilipin@tin: Synchronized static/images/project-logos/hiwiki.png: SWAT: Make hiwiki logo a little bit bigger (T175721) (duration: 00m 46s)
13:37 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Import sources on hr.wikibooks (T176320) (duration: 00m 46s)
13:30 addshore@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/WikimediaEventsHooks.php: SWAT: gerrit:379749 onBeforeInitializeWMDECampaign, fix early bail condition T174948 (duration: 00m 46s)
13:18 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add several rights to eliminators in fawiki (T176553) (duration: 00m 46s)
12:52 moritzm: removed salt master from neodymium
12:38 moritzm: revoked all salt keys on neodymium
11:58 godog: powercycle ms-be2020 - blk_update_request: I/O error, dev sdb, sector in consol
11:54 moritzm: removing salt-minion/salt-common from production
11:47 Dereckson: Repopulate sites tables for Wikidata clients (T173013)
11:47 Dereckson: Create new database and tables for hi.wikivoyage (T173013)
11:45 dereckson@tin: Synchronized wmf-config/interwiki.php: Interwiki map update (+hiwikivoyage and other changes previous month, T176572) (duration: 00m 46s)
11:39 marostegui: Run redact_sanitarium for hiwikivoyage on db1095 - T173027
11:39 moritzm: upgrading tor to latest stable release (0.3.1.7)
11:37 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Initial configuration for hi.wikivoyage.org (T173013) (duration: 00m 46s)
11:37 dereckson@tin: Synchronized static/images/project-logos/: Logos for hi.wikivoyage.org (T173013) (duration: 00m 45s)
11:34 dereckson@tin: rebuilt wikiversions.php and synchronized wikiversions files: +hiwikivoyage (T173013)
11:32 dereckson@tin: Synchronized dblists: Add hiwikivoyage to the set (T173013) (duration: 00m 46s)
10:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1055 weight (duration: 00m 46s)
10:14 elukey: add mw1323 to live traffic (mw appserver) - traffic weights will go from 5 to 20 incrementally
09:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1055 weight and remove main traffic from special slaves (duration: 00m 46s)
09:07 marostegui: Add 50GB to db1036 /srv partition - T176311
09:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1036 and full repool db1101 - T176311 (duration: 00m 46s)
08:49 moritzm: uploaded php-luasandbox 2.0.14, hhvm-tidy 0.1.3 and php-wikidiff2 1.4.1 for stretch-wikimedia to apt.wikimedia.org
08:36 mobrovac@tin: Finished deploy [restbase/deploy@ab24f70]: Switch mobile-sections to next-gen storage and stop storing the current-day feed in Cassandra - T169940 T176233 (duration: 10m 03s)
08:26 mobrovac@tin: Started deploy [restbase/deploy@ab24f70]: Switch mobile-sections to next-gen storage and stop storing the current-day feed in Cassandra - T169940 T176233
08:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1055 weight (duration: 00m 45s)
07:56 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1101 - T176311 (duration: 00m 46s)
07:42 moritzm: uploaded hhvm 3.18.2+dfsg-1+wmf5+deb9u1 for stretch-wikimedia to apt.wikimedia.org
07:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1055 weight (duration: 00m 47s)
07:37 elukey: add mw132[0,2] to live traffic (mw appservers) - traffic weights will go from 5 to 20 incrementally
06:38 moritzm: installing apache security updates on logstash*
02:36 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Sep 25 02:36:48 UTC 2017 (duration 6m 42s)
02:30 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 07m 17s)

2017-09-24

14:23 jynus: restarting db2047 for kernel upgrade
14:16 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2047 (duration: 00m 48s)
off: restarted ircecho on einsteinium

2017-09-23

21:12 hoo: Ran scap pull on mwdebug1001 to undo changes to profile https://github.com/wmde/WikibaseDataModel/pull/761/files

2017-09-22

23:36 mutante: gerrit2001 - rebooting .. wave a dead chicken http://www.catb.org/jargon/html/W/wave-a-dead-chicken.html
22:56 mutante: gerrit2001 - trying to manually start gerrit again, as opposed to puppet doing it.. debugging gerrit-ssh issue there, cobalt still untouched
22:34 mutante: gerrit2001 - stopping gerrit with gerrit.sh stop, letting puppet start it again
22:29 mutante: gerrit2001 - systemd says gerrit.service is failed. gerrit.sh start says "Already Running!!" :p - cobalt is fine
18:26 demon@tin: Pruned MediaWiki: 1.30.0-wmf.18 [keeping static files] (duration: 01m 29s)
15:21 hashar: Restarted Jenkins. Out of memory)
14:49 jynus@tin: Synchronized wmf-config/db-eqiad.php: repool db1101 with low weight (duration: 00m 47s)
14:43 moritzm: uploaded php-luasandbox build for src:php5.5 (required for CI tests on jessie)
14:42 moritzm: updated tor packages to 0.3.1.7 (new stable series)
13:49 elukey: mw1321 (new appserver) serving traffic (going to increase its weight up to 20)
12:02 hashar: apt-get upgrade on contint1001 / contint2001
09:42 elukey: mw1319 (new appserver) serving traffic (going to increase its weight up to 20)
09:14 jynus: stop mariadb at db1055 for upgrade and maintenance
06:32 moritzm: installing emacs security updates on trusty (Debian already fixed)

2017-09-21

23:56 ejegg: updated payments-wiki from 7ed7243 to d2ce697
23:23 ebernhardson@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/extension.json: T175047: Stop delivering search relevance survey javascript (duration: 00m 46s)
23:18 ebernhardson@tin: Synchronized wmf-config: T175047: Turn off human search relevance survey (duration: 00m 48s)
23:16 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T175047: Turn off human search relevance survey (duration: 00m 46s)
22:59 jynus@tin: Synchronized wmf-config/db-eqiad.php: depool db1055 (duration: 00m 46s)
22:12 volans: uploaded cumin_1.1.0-1_amd64.deb to apt.wikimedia.org jessie-wikimedia
22:10 ejegg: updated payments-wiki from 9da8482 to 7ed7243
21:07 demon@tin: Synchronized wmf-config/CommonSettings.php: extdist settings (duration: 00m 46s)
21:04 ejegg: updated CiviCRM from fc56901 to 45c56cf
19:32 demon@tin: Synchronized wmf-config/InitialiseSettings.php: adjust logging on csp reports (duration: 00m 46s)
19:20 demon@tin: Synchronized php-1.30.0-wmf.19/extensions/VisualEditor/ApiVisualEditor.php: fix user/wgUser mixup (duration: 00m 46s)
19:16 gehel: upgrade to nodejs 6.11 on maps servers (including restart of tilerator / kartotherian) - T171707
19:16 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4026.*
19:04 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.19
18:56 urandom: T171772: Applying locally hacked Prometheus exporter config to RESTBase dev Cassandra instances
18:44 thcipriani@tin: Synchronized php-1.30.0-wmf.19/resources/src/mediawiki.rcfilters: SWAT: WLFilters: Do not hide .watchlistDetails while loading T176300 RCFilters: Make the interface not jump around while loading (duration: 00m 49s)
18:41 urandom: T171772: Restarting Cassandra restbase-dev1004-a to apply locally hacked Prometheus exporter config
18:34 thcipriani@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Turning off Explore Similar AB test T175649 (duration: 00m 49s)
18:31 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Print instrumentation T176341 (duration: 00m 52s)
14:56 moritzm: uploaded php-defaults and php-redis built against src:php5.5 to component/ci for jessie-wikimedia
13:45 gehel: upgrade to nodejs 6.11 on the full maps-test cluster - T171707
13:12 zeljkof: EU SWAT finished
13:09 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Switch elasticsearch active cluster to codfw" (duration: 00m 48s)
13:09 zfilipin@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Revert "Switch elasticsearch active cluster to codfw" (duration: 00m 48s)
13:07 zfilipin@tin: Synchronized tests/cirrusTest.php: SWAT: Revert "Switch elasticsearch active cluster to codfw" (duration: 00m 49s)
12:55 _joe_: stopped long-running runjobs for refreshlinks on commons.
12:50 mobrovac@tin: Finished deploy [restbase/deploy@bc02191]: produce a diff for the mobile proxy (duration: 18m 34s)
12:32 mobrovac@tin: Started deploy [restbase/deploy@bc02191]: produce a diff for the mobile proxy
11:59 moritzm: upgrading apache on deployment servers and script runners
11:40 moritzm: upgrading apache on canary app servers
11:17 moritzm: installing apache security updates (our configurations are not affected, the upload of 2.4.10-10+deb8u11+wmf1 is mostly for the benefit of whatever people are doing in Cloud VPS, but we should still fix the underlying bug in production as well)
11:14 dcausse: reindexing all hebrew wikis to pickup the new hebmorph analyzer
11:09 dcausse: reindexing ilwikimedia in codfw to pickup and test new hebmorph analyzer
10:26 akosiaris: upload docker-ce_17.06.2~ce-0~debian_amd64.deb to apt.wikimedia.org jessie-wikimedia/thirdparty/ci T175293
09:59 dcausse: reindexing group1 wikis T176397
09:54 dcausse: reindexing group0 wikis T176397
08:35 moritzm: unbreak deployment-puppetmaster02 in deployment-prep (broken by unattended-upgrades update of apache T159254)
08:35 akosiaris: pool all of wtp1025 to wtp1048 T165520
08:35 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1044.eqiad.wmnet
08:35 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1043.eqiad.wmnet
08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1042.eqiad.wmnet
08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1041.eqiad.wmnet
08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1040.eqiad.wmnet
08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1039.eqiad.wmnet
08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1038.eqiad.wmnet
08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1037.eqiad.wmnet
08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1036.eqiad.wmnet
08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1035.eqiad.wmnet
08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1034.eqiad.wmnet
08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1033.eqiad.wmnet
08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1032.eqiad.wmnet
08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1031.eqiad.wmnet
08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1030.eqiad.wmnet
08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1029.eqiad.wmnet
08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1028.eqiad.wmnet
08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1027.eqiad.wmnet
08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1026.eqiad.wmnet
08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
08:30 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
08:30 akosiaris@puppetmaster1001: conftool action : set/weight=10; selector: name=wtp1025.eqiad.wmnet
08:25 mobrovac@tin: Finished deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests (duration: 03m 18s)
08:22 mobrovac@tin: Started deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests
08:21 mobrovac@tin: Finished deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests (duration: 00m 58s)
08:20 mobrovac@tin: Started deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests
08:19 mobrovac@tin: Finished deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests (duration: 05m 22s)
08:14 mobrovac@tin: Started deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests
08:08 mobrovac@tin: Finished deploy [restbase/deploy@3e9bd9f]: New storage schema for mobile-sections - T169940 (duration: 10m 28s)
07:57 mobrovac@tin: Started deploy [restbase/deploy@3e9bd9f]: New storage schema for mobile-sections - T169940
07:31 _joe_: restarted aphlict on phab1001
07:27 mobrovac@tin: Finished deploy [restbase/deploy@3e9bd9f]: New storage schema for mobile-sections, canary deploy for schema creation, take 2 - T169940 (duration: 00m 30s)
07:27 mobrovac@tin: Started deploy [restbase/deploy@3e9bd9f]: New storage schema for mobile-sections, canary deploy for schema creation, take 2 - T169940
07:09 ema: bounce pybal on lvs1003 to clear stale alert T176388
06:59 ema: bounce pybal on lvs1006 to clear stale alert
06:55 ema: bounce pybal on lvs1009 to clear stale alert
03:24 bblack: depooled cp4026
03:12 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Sep 21 03:12:41 UTC 2017 (duration 6m 36s)
03:06 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 14m 44s)
02:29 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 08m 50s)
00:41 eileen: update civicrm from da833ac to fc56901
00:23 ebernhardson@tin: Synchronized php-1.30.0-wmf.18/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: Fix schema name for search satisfaction error logging (duration: 00m 53s)
00:18 ebernhardson@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: Fix schema name for search satisfaction error logging (duration: 00m 49s)
00:09 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable jQuery 3 on commons (T124742) (duration: 00m 49s)
00:02 catrope@tin: Synchronized php-1.30.0-wmf.19/resources/src/mediawiki.rcfilters/: Lazy-load the RCFilters menu (T176250) (duration: 00m 48s)

2017-09-20

23:49 catrope@tin: Synchronized php-1.30.0-wmf.18/extensions/WikimediaEvents/: Log JS errors during search satisfaction tests (duration: 00m 48s)
23:47 catrope@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/: Log JS errors during search satisfaction tests (duration: 00m 49s)
23:40 mutante: phabricator: aphlict (notification service) now running on prod server
23:36 mutante: phab1001 - re-enable puppet, run after follow-up fix, adding notification server config
23:31 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Add throttle rule for John Michael Kohler Art Center (T176287) (duration: 00m 48s)
23:29 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable user email blacklist on meta (T174694) (duration: 00m 48s)
23:21 catrope@tin: Synchronized dblists/categories-rdf.dblist: Add mlwiki to categories-rdf.dblist (duration: 00m 48s)
23:14 mutante: phab2001 - deluser aphlict, delgroup aphlict, try to let puppet create it again, still fails, just won't create the group
23:13 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Add patroller group on metawiki (T176079) (duration: 00m 48s)
23:11 mutante: phab2001 - groupadd aphlict,temp adduser aphlict to debug user creation issue
22:22 ebernhardson: elasticsearch eqiad completely recovery, all green. Returning recovery max_bytes_per_sec to 20mb
21:54 mutante: phab: disable puppet on phab1001 for a minute, apply notification server change on phab2001
21:45 eileen: update civicrm from 0fb4676 to da833ac
21:09 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4015.ulsfo.wmnet
20:58 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4007.ulsfo.wmnet
20:40 bsitzmann@tin: Finished deploy [mobileapps/deploy@e23bf66]: Update mobileapps to d46860e (T175759 T176263) (duration: 06m 05s)
20:36 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4014.ulsfo.wmnet
lunch: increase max recovery bytes/s to 80mb in eqiad elasitcsearch cluster
20:34 bsitzmann@tin: Started deploy [mobileapps/deploy@e23bf66]: Update mobileapps to d46860e (T175759 T176263)
20:33 demon@tin: Synchronized php-1.30.0-wmf.19/includes/diff/DifferenceEngine.php: fix some warnings (duration: 00m 51s)
19:55 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4016.ulsfo.wmnet
19:54 eevans@tin: Finished deploy [cassandra/metrics-collector@df909a1] (dev-cluster): Deploy 4.1.0 artifact to dev cluster (duration: 00m 10s)
19:54 eevans@tin: Started deploy [cassandra/metrics-collector@df909a1] (dev-cluster): Deploy 4.1.0 artifact to dev cluster
19:53 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4006.ulsfo.wmnet
19:48 eevans@tin: Finished deploy [cassandra/metrics-collector@df909a1]: Pushing out 4.1.0 release (duration: 00m 47s)
19:47 eevans@tin: Started deploy [cassandra/metrics-collector@df909a1]: Pushing out 4.1.0 release
19:39 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4008.ulsfo.wmnet
19:26 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.19
19:24 demon@tin: Synchronized php: symlink swap for wmf.19 (duration: 00m 49s)
19:02 catrope@tin: Finished scap: Patches for T176302, T175962, T173533 (duration: 21m 29s)
18:40 catrope@tin: Started scap: Patches for T176302, T175962, T173533
17:29 hoo: Started mwscript refreshLinks.php --wiki elwiki --namespace 0 in a screen on terbium. Can be KILLed at any time, if needed.
17:29 hoo: Started mwscript refreshLinks.php --wiki elwiki --namespace 0 in a screen on elwiki. Can be KILLed at any time, if needed.
17:13 hoo: Ran mwscript refreshLinks.php --wiki elwiki --namespace 0 -e 30 for testing purposes
17:06 ejegg: removed limit from donation queue consumer
17:05 otto@tin: Finished deploy [eventstreams/deploy@e62ab64]: (no justification provided) (duration: 04m 44s)
17:01 otto@tin: Started deploy [eventstreams/deploy@e62ab64]: (no justification provided)
16:53 madhuvishy: Upgraded and restarted apache2 on labs-puppetmaster.wikimedia.org
16:45 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,cluster=elasticsearch
16:42 gehel: rolling restart of logstash ingesters after cirrus eqiad recovery
16:34 gehel: elasticsearch eqiad is finally recovering, after the restart of elastic1051
16:34 gehel: cleanup jar hell and restart elastic1051
16:19 gehel: starting elasticsearch on all nodes on eqiad
16:10 gehel: restarting elasticsearch masters on eqiad (elastic1030, 36 and 40)
16:04 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: dc=eqiad,cluster=elasticsearch
16:03 gehel: depooling all nodes in elasticsearch eqiad to investigate failed cluster restart (traffic is already directed to codfw)
15:46 ejegg: limited donations queue consumer to 100 per run
15:36 ejegg: updated payments-wiki from 896f16a to 9da8482
15:36 gehel: upgrading elasticsearch plugins on elasticsearch eqiad, including cold restart of the cluster - T173231
15:33 ejegg: limited donations queue consumer to 200 per run
14:03 akosiaris@tin: Synchronized wmf-config/InitialiseSettings.php: Whitelist wtp10[25-48] for Linter (duration: 00m 49s)
13:42 otto@tin: Synchronized /srv/mediawiki-staging/wmf-config/CommonSettings-labs.php: (no justification provided) (duration: 00m 49s)
13:24 ema: begin rolling codfw varnish-be restarts, 20m spacing
13:20 dereckson@tin: Synchronized wmf-config/CommonSettings.php: Switch elasticsearch active cluster to codfw (Gerrit:378850, 3/3) (duration: 00m 48s)
13:18 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Switch elasticsearch active cluster to codfw (Gerrit:378850, 2/3) (duration: 00m 49s)
13:17 dereckson@tin: Synchronized tests/cirrusTest.php: Switch elasticsearch active cluster to codfw (Gerrit:378850, 1/3, no-op in prod) (duration: 00m 49s)
12:51 ema: restarted varnish-be on cp4028
12:28 gehel: upgrading nodejs to 6.11 on maps-test2003 for testing - T171707
12:19 ema: restarted varnish-be on cp4027
12:18 moritzm: uploaded apache 2.4.10-10+deb8u11+wmf1 to jessie-wikimedia (rebuild of latest security update with our local patches on top)
12:08 hoo@tin: Synchronized wmf-config/Wikibase-production.php: Enable statement usage tracking on elwiki (T151717) (duration: 00m 49s)
11:58 mobrovac@tin: Finished deploy [restbase/deploy@dea2b41]: New storage schema for mobile-sections, canary deploy for schema creation - T169940 (duration: 07m 20s)
11:56 ema: restarted varnish-be on cp4018
11:51 mobrovac@tin: Started deploy [restbase/deploy@dea2b41]: New storage schema for mobile-sections, canary deploy for schema creation - T169940
11:49 jynus: stopping replication on db1101 for faster repartition work T176311
11:45 jynus@tin: Synchronized wmf-config/db-eqiad.php: depool db1101 (duration: 00m 58s)
11:34 ema: restarted varnish-be on cp4010
11:25 mobrovac@tin: Finished deploy [restbase/deploy@dea2b41]: (no justification provided) (duration: 00m 02s)
11:25 mobrovac@tin: Started deploy [restbase/deploy@dea2b41]: (no justification provided)
11:03 moritzm: rolling out new debdeploy release across the fleet
11:02 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 19s)
11:02 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided)
11:02 ema: restarted varnish-be on cp4008
11:02 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 04s)
11:01 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided)
11:01 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 03s)
11:01 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided)
11:00 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 04s)
11:00 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided)
09:55 bblack: restarted varnish-be on cp3042
09:29 bblack: restarted varnish-be on cp1052
09:23 ema: restarted varnish-be on cp3043
09:21 ema: restarted varnish-be on cp3041
09:19 ema: restarted varnish-be on cp3032
09:14 ema: restarted varnish-be on cp3040
08:42 bblack: restarted varnish-be on cp1053
08:41 bblack: restarted varnish-be on cp1068
08:40 bblack: restarted varnish-be on cp1054
08:40 bblack: restarted varnish-be on cp1065
07:11 ema: cp3030 varnish-be restart due to 503s
03:16 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Sep 20 03:16:25 UTC 2017 (duration 7m 31s)
03:08 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 15m 57s)
02:31 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 08m 06s)
00:55 bblack: cp1066 - backend restart, mailbox lag, text
00:54 bblack: cp4016 - backend restart, mailbox lag, text

2017-09-19

23:57 thcipriani@tin: Synchronized php-1.30.0-wmf.18/extensions/VisualEditor/ApiVisualEditor.php: SWAT: ApiVisualEditor: Fix checkbox label message handling with Message objects T176249 (duration: 00m 49s)
23:56 thcipriani@tin: Synchronized php-1.30.0-wmf.19/extensions/VisualEditor/ApiVisualEditor.php: SWAT: ApiVisualEditor: Fix checkbox label message handling with Message objects T176249 (duration: 00m 48s)
23:46 thcipriani@tin: Synchronized php-1.30.0-wmf.18/extensions/WikimediaEvents/modules/ext.wikimediaEvents.humanSearchRelevance.js: SWAT: Javascript timestamps are in ms, not s T174106 (duration: 00m 48s)
23:45 thcipriani@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/modules/ext.wikimediaEvents.humanSearchRelevance.js: SWAT: Javascript timestamps are in ms, not s T174106 (duration: 00m 50s)
23:39 thcipriani@tin: Synchronized php-1.30.0-wmf.18/includes/specials/SpecialRecentchangeslinked.php: SWAT: SpecialRecentchangeslinked: Unconditionally join on the page table T176228 (duration: 00m 49s)
23:36 thcipriani@tin: Synchronized php-1.30.0-wmf.19/includes/specials/SpecialRecentchangeslinked.php: SWAT: SpecialRecentchangeslinked: Unconditionally join on the page table T176228 (duration: 00m 49s)
23:29 thcipriani@tin: Synchronized php-1.30.0-wmf.18/includes/page/PageArchive.php: SWAT: Do not attempt to copy to ip_changes in PageArchive class (duration: 00m 49s)
23:27 thcipriani@tin: Synchronized php-1.30.0-wmf.19/includes/page/PageArchive.php: SWAT: Do not attempt to copy to ip_changes in PageArchive class (duration: 00m 50s)
23:17 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable jQuery 3 on meta.wikimedia.org T124742 (duration: 00m 50s)
22:58 eileen: manually running Omnirecipient.load from 2017-09-18 with accidentally low throttle setting
22:22 mutante: gerrit2001 - stopped gerrit with init.d script. moved init.d script to /root, started with systemctl
22:21 eileen: update civicrm from 9677cf7 to 0fb4676
22:15 mutante: gerrit2001 /bin/sh /var/lib/gerrit2/review_site/bin/gerrit.sh start
22:02 mutante: gerrit2001 - systemctl start gerrit
21:51 volans: disabled puppet on labpuppetmaster1001, testing cumin T175712
21:27 MaxSem: on terbium: mwscript populateIpChanges.php --wiki=enwikivoyage --force
21:23 bblack@neodymium: conftool action : set/pooled=yes; selector: cluster=cache_text,name=cp4008.ulsfo.wmnet
21:23 bblack@neodymium: conftool action : set/pooled=yes; selector: cluster=cache_text,name=cp4016.ulsfo.wmnet
21:22 bblack@neodymium: conftool action : set/pooled=yes; selector: cluster=cache_text,dc=eqiad,name=cp1066.*
21:18 MaxSem: re-ran populateIpChanges.php on group0 wikis
21:15 maxsem@tin: Synchronized php-1.30.0-wmf.19/maintenance/populateIpChanges.php: https://gerrit.wikimedia.org/r/#/c/379057/1 (duration: 00m 49s)
21:06 ema: cp4009: varnish-be restart, 503 spike
20:04 bblack@neodymium: conftool action : set/pooled=no; selector: cluster=cache_upload,name=cp4013.ulsfo.wmnet
20:03 bblack@neodymium: conftool action : set/pooled=no; selector: cluster=cache_text,name=cp4016.ulsfo.wmnet
19:50 bblack@neodymium: conftool action : set/pooled=no; selector: cluster=cache_upload,name=cp4005.ulsfo.wmnet
19:49 bblack@neodymium: conftool action : set/pooled=no; selector: cluster=cache_text,name=cp4008.ulsfo.wmnet
19:17 ejegg: updated payments-wiki from 26b16ea to 896f16a
19:08 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.19
18:52 gehel: upgrading nodejs to 6.11 on maps-test2004 for testing - T171707
18:35 urandom: T160570, T169940: Upgrade restbase-ng environment to Cassandra 3.11.0-wmf5
18:26 arlolra: Updated Parsoid to 05a0965 (T122965, T175101, T173384, T173061, T172896, T174977, T159894)
18:17 ladsgroup@tin: Synchronized php-1.30.0-wmf.18/extensions/Wikidata/extensions/Wikibase/repo/tests/phpunit/includes/Content/EntityContentTest.php: Fix undoing merge operations that turned Items into a redirects, part II (T175984) (duration: 00m 48s)
18:15 ladsgroup@tin: Synchronized php-1.30.0-wmf.18/extensions/Wikidata/extensions/Wikibase/repo/includes/Content/EntityContent.php: Fix undoing merge operations that turned Items into a redirects, part I (T175984) (duration: 00m 49s)
18:08 ladsgroup@tin: Synchronized php-1.30.0-wmf.19/extensions/Wikidata/extensions/Wikibase/repo/tests/phpunit/includes/Content/EntityContentTest.php: Fix undoing merge operations that turned Items into a redirects, part II (T175984) (duration: 00m 50s)
18:07 ladsgroup@tin: Synchronized php-1.30.0-wmf.19/extensions/Wikidata/extensions/Wikibase/repo/includes/Content/EntityContent.php: Fix undoing merge operations that turned Items into a redirects, part I (T175984) (duration: 00m 50s)
18:00 arlolra@tin: Finished deploy [parsoid/deploy@a01064d]: Updating Parsoid to 05a0965 (duration: 12m 26s)
17:54 ebernhardson: T173464 start cirrussearch reindex of all wikis starting with zh
17:48 arlolra@tin: Started deploy [parsoid/deploy@a01064d]: Updating Parsoid to 05a0965
17:43 jynus: stopping db2010 and cloning to db2078
17:16 demon@tin: Finished scap: bootstrap wmf.19 (duration: 31m 04s)
16:45 demon@tin: Started scap: bootstrap wmf.19
16:44 demon@tin: scap aborted: bootstrap wmf.19 (duration: 01m 52s)
16:42 demon@tin: Started scap: bootstrap wmf.19
16:38 jynus@tin: Synchronized wmf-config/db-codfw.php: decom db1018 (duration: 00m 51s)
16:37 jynus@tin: Synchronized wmf-config/db-eqiad.php: decom db1018 (duration: 01m 01s)
16:34 demon@tin: Pruned MediaWiki: 1.30.0-wmf.17 [keeping static files] (duration: 01m 18s)
16:31 demon@tin: Pruned MediaWiki: 1.30.0-wmf.14 (duration: 03m 15s)
16:29 jynus: disable puppet on db1009
16:27 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Trying that again, got an error the first time (duration: 00m 52s)
16:26 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable RCFilters by default on cawiki, frwiki and hewiki (T157642) (duration: 00m 59s)
16:19 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable structured filters on watchlist on all wikis (duration: 00m 45s)
16:11 ema: pybal 1.14.0 uploaded to apt.w.o
15:54 catrope@tin: Finished scap: core, GuidedTour and WikimediaMessages patches for T176191, T167262 and T175765 (duration: 20m 32s)
15:33 catrope@tin: Started scap: core, GuidedTour and WikimediaMessages patches for T176191, T167262 and T175765
15:22 jynus: enabling firewall on db1020 (m2-master)
15:01 ema: restart varnish-be on cp1055, 503 spike
14:55 jynus: deploying firewall to m5 hosts
14:39 jynus: disabling puppet on all misc active mysql hosts
13:35 zeljkof: EU SWAT finished
13:32 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Rename Wikisaurus namespace on Wiktionary to "Thesaurus" (T174264) (duration: 00m 45s)
13:26 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Rename Wikisaurus namespace on Wiktionary to "Thesaurus" (T174264) (duration: 02m 55s)
13:19 gehel: upgrading elasticsearch plugins on relforge, including cold restart of the cluster - T173231
13:17 gehel: upgrading elasticsearch plugins on elasticsearch codfw, including cold restart of the cluster - T173231
13:15 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Rename Wikisaurus namespace on Wiktionary to "Thesaurus" (T174264) (duration: 02m 56s)
13:04 gehel: upgrading elasticsearch plugins on elastic2001 - T173231
12:55 moritzm: rebooting kubestage1001
11:25 moritzm: uploaded debdeploy 0.99.1 to apt.wikimedia.org (for trusty, jessie and stretch)
11:23 kartik@tin: Finished deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7 (duration: 02m 42s)
11:20 kartik@tin: Started deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7
10:53 volans: restarted nagios-nrpe-server on stat1005
10:14 jynus: shutting down mysql on db1018
09:42 moritzm: installing libxml2 security updates on trusty (Debian already fixed)
09:40 elukey: powercycle analytics1062 - no ssh, console com2 frozen
09:34 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: Enable the updateBetaFeaturesUserCounts logging channel - T175637 (duration: 00m 50s)
09:30 ema: cp1062 - varnish-backend-restart, mbox lag
08:40 mobrovac@tin: Finished deploy [cxserver/deploy@03c1eb1]: (no justification provided) (duration: 00m 52s)
08:39 mobrovac@tin: Started deploy [cxserver/deploy@03c1eb1]: (no justification provided)
08:09 mobrovac@tin: Finished deploy [cxserver/deploy@03c1eb1]: (no justification provided) (duration: 00m 13s)
08:09 mobrovac@tin: Started deploy [cxserver/deploy@03c1eb1]: (no justification provided)
07:58 kartik@tin: Finished deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7 (duration: 00m 37s)
07:58 kartik@tin: Started deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7
07:50 moritzm: installing bind9 regression update on trusty servers
07:35 kartik@tin: (no justification provided)
07:35 kartik@tin: Finished deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7 (duration: 01m 04s)
07:34 kartik@tin: Started deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7
02:28 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Sep 19 02:28:56 UTC 2017 (duration 7m 6s)
02:21 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 07m 04s)
01:52 demon@tin: Synchronized scap/plugins/updatewikiversions.py: Minor newline nitpick. I promise that's it (duration: 00m 46s)
01:41 demon@tin: Synchronized scap/plugins/updatewikiversions.py: Minor spacing nitpick, that's it for today (duration: 00m 45s)
01:17 demon@tin: Synchronized scap/plugins/updatewikiversions.py: add new hotness updatewikiversions.py (duration: 00m 45s)
01:15 demon@tin: Synchronized multiversion/: rm old and busted updateWikiversions (duration: 01m 07s)
00:12 foks: Remove 2FA from users Alan and Magicknight94 (retroactive log)
00:10 eileen: update civicrm from 187238f to b6f7dc3

2017-09-18

23:28 tstarling@tin: Synchronized php-1.30.0-wmf.18/includes/filerepo/file/LocalFile.php: (no justification provided) (duration: 00m 46s)
23:13 thcipriani@tin: Synchronized php-1.30.0-wmf.18/extensions/ArticleCreationWorkflow: SWAT: Only intercept users who can potentially create articles otherwise T176100 (duration: 00m 46s)
22:32 demon@tin: Synchronized wmf-config/CommonSettings.php: Use PrivateSettings.php directly (duration: 00m 46s)
21:57 bawolff: deployed patch T175900
21:18 krinkle@tin: Synchronized php-1.30.0-wmf.18/extensions/NavigationTiming/modules/ext.navigationTiming.js: Unbreak - T176105 (duration: 00m 48s)
21:16 mutante: repeating import of gerrit_2.13.8+git1-wmf.7 on install1002, followed by reprepro copy stretch-wikimedia jessie-wikimedia gerrit to make it available on stretch
21:06 mutante: uploaded gerrit 2.13.8+git1-wmf.7 for jessie-wikimedia on APT.wm.org
20:42 krinkle@tin: Finished deploy [jobrunner/jobrunner@57f5f47]: No-op sync - first time scap3 - T129148 (duration: 03m 23s)
20:38 krinkle@tin: Started deploy [jobrunner/jobrunner@57f5f47]: No-op sync - first time scap3 - T129148
20:17 arlolra@tin: (no justification provided)
20:15 arlolra@tin: Finished deploy [parsoid/deploy@a01064d]: Updating Parsoid to 05a0965 (duration: 01m 53s)
20:13 arlolra@tin: Started deploy [parsoid/deploy@a01064d]: Updating Parsoid to 05a0965
19:29 XioNoX: restarting pfw3-codfw for software upgrade
19:10 urandom: T160570: Upgrading restbase-dev100[5-6].eqiad.wmnet to Cassandra 3.11.0-wmf5
19:07 smalyshev@tin: Finished deploy [wdqs/wdqs@2523836]: Deploy correct prefixes conf (duration: 01m 20s)
19:05 smalyshev@tin: Started deploy [wdqs/wdqs@2523836]: Deploy correct prefixes conf
19:03 smalyshev@tin: Finished deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS, take 4 (duration: 02m 36s)
19:02 no_justification: gerrit: restarting service, back in a moment
19:00 smalyshev@tin: Started deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS, take 4
18:56 urandom: T160570: Upgrading restbase-dev1004.eqiad.wmnet to Cassandra 3.11.0-wmf5 (canary)
18:46 smalyshev@tin: Finished deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS, take 3 (duration: 02m 25s)
18:44 smalyshev@tin: Started deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS, take 3
18:37 smalyshev@tin: (no justification provided)
18:36 smalyshev@tin: Finished deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS (duration: 02m 26s)
18:34 smalyshev@tin: Started deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS
18:20 maxsem@tin: Synchronized wmf-config/: https://gerrit.wikimedia.org/r/#/c/378357/ https://gerrit.wikimedia.org/r/#/c/376791/ (duration: 00m 48s)
17:42 smalyshev@tin: Finished deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS (duration: 00m 21s)
17:42 smalyshev@tin: Started deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS
17:26 demon@tin: Synchronized wmf-config/ProductionServices.php: no-op/comment fix (duration: 00m 45s)
16:46 jynus: shuting down db1100 T175973
16:40 reedy@tin: Synchronized php-1.30.0-wmf.18/maintenance/populateIpChanges.php: perf improvements T175962 (duration: 00m 46s)
16:05 bblack: cp1063 - backend restart, mailbox lag
15:38 bblack: cp1072 - varnish backend restart, mailbox lag
15:28 bblack: cp1074 - backend restart for mailbox lag
15:25 bblack: cp1050 - backend restart for mailbox lahg
14:27 reedy@tin: Synchronized php-1.30.0-wmf.18/includes/filerepo/file/LocalFile.php: T175444 (duration: 00m 47s)
13:29 ppchelko@tin: Finished deploy [cpjobqueue/deploy@2c422f5]: Annotate the job event with pipeline property (duration: 00m 29s)
13:28 ppchelko@tin: Started deploy [cpjobqueue/deploy@2c422f5]: Annotate the job event with pipeline property
13:25 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Meta(Talk)Namespace configuration for be.wiktionary - T175950 (duration: 00m 46s)
13:18 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: pagePreviews: Stop A/B test on enwiki and dewiki - T176068 (duration: 00m 45s)
13:10 hashar@tin: Synchronized php-1.30.0-wmf.18/extensions/BetaFeatures: Temporary log all the executions of the update job. - T175637 (duration: 00m 46s)
13:09 hashar@tin: Synchronized wmf-config: New 'abusefilter-helper' configuration for en.wikipedia - T175684 (duration: 00m 49s)
12:59 moritzm: installing libxen updates
12:36 moritzm: installing tomcat8 security updates
10:40 ema: cp1099 - backend restart, mailbox lag
06:42 moritzm: installing freexl security updates
06:18 TimStarling: testing EtcdConfig in beta cluster
05:40 tstarling@tin: Synchronized wmf-config: just for consistency, should have no effect (gerrit 375108) (duration: 00m 49s)
02:36 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Sep 18 02:36:50 UTC 2017 (duration 7m 6s)
02:29 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 08m 04s)

2017-09-17

12:49 _joe_: taking a full debug dump of mw1288 after depooling it
12:40 _joe_: restarting hhvm on some api appservers, to ease the cpu overlock
09:46 elukey: restart varnish backend on cp1052 - recurrent mailbox lag
08:11 elukey: restart varnish backend on cp1053 - recurrent mailbox lag

2017-09-16

22:05 godog: compress older otrs directories to reclaim inodes - T171490
16:53 elukey: restart varnish-backend on cp1073 (cache upload) for mailbox lag
15:40 andrewbogott: rebooting labvirt1017
15:34 andrewbogott: rebooting labvirt1015 for T176044

2017-09-15

21:50 mutante: contint2001 - systemctl start zuul
21:47 mutante: contint1001 - tmp disable puppet - contint2001 - test zuul unit file (gerrit 359016)
21:43 demon@tin: Finished deploy [gerrit/gerrit@48ad332]: 2.13.9 -- not being used yet though (duration: 00m 08s)
21:43 demon@tin: Started deploy [gerrit/gerrit@48ad332]: 2.13.9 -- not being used yet though
15:46 bblack: cp1049 - backend restart, mailbox lag
15:40 bblack: cp1099 - backend restart, mailbox lag
15:38 bblack: cp1072 - backend restart, mailbox lag
15:21 _joe_: uploaded td-agent-bit (fluentd-bit) to wikimedia-stretch T175527
13:13 moritzm: pruned hhvm bytecode cache on mw2256, led to log spam for an undefined variable after being out of service for hardware maintenance
13:10 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw2256.codfw.wmnet
13:10 addshore@tin: Synchronized wmf-config/CommonSettings-labs.php: beta: $wgWikiDiff2MovedParagraphDetectionCutoff = 25 BETA ONLY 2/2 (duration: 00m 45s)
13:08 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: beta: $wgWikiDiff2MovedParagraphDetectionCutoff = 25 BETA ONLY 1/2 (duration: 00m 47s)
11:22 jmm@puppetmaster1001: conftool action : set/pooled=no; selector: mw2256.codfw.wmnet
09:10 moritzm: installing tcpdump security updates on trusty hosts (Debian systems already fixed)
08:54 moritzm: installing libbluetooth updates
08:03 jynus: dropping mariadb analytics users on db1031, db1029
07:49 Amir1: running maintenance/updateCollation.php --force on Persian (fa) wikis (T173601)
07:39 gehel: shutting down and masking elasticsearch on elastic1020 - T175951
07:35 gehel: depooling elastic1020 - T175951
07:28 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw2256.codfw.wmnet
00:46 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1100 (duration: 00m 46s)

2017-09-14

23:51 mutante: phabricator - tested gerrit 354247 before merging using apache-fast-test from tin -restarted apache
23:37 catrope@tin: Synchronized wmf-config/: VRS config for Electron (T175868) (duration: 00m 47s)
23:34 catrope@tin: Synchronized php-1.30.0-wmf.18/extensions/VisualEditor/lib/ve/src/ui/styles/ve.ui.TableLineContext.css: T169389 (duration: 00m 45s)
23:33 catrope@tin: Synchronized php-1.30.0-wmf.18/extensions/ContentTranslation/modules/widgets/translator/ext.cx.translator.js: Fix JS error in CX (duration: 00m 46s)
23:29 catrope@tin: Synchronized wmf-config/CommonSettings.php: Give sysops the flow-create-board right on Flow wikis (T175934) (duration: 00m 45s)
23:27 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Give sysops the flow-create-board right on Flow wikis (T175934) (duration: 00m 45s)
23:22 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Use RemexHtml on mediawikiwiki and testwiki (T175095) (duration: 00m 46s)
23:19 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=acamar.wikimedia.org
23:18 mutante: acamar is back up and running - no failed units
23:13 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Allow bureaucrats to remove accountcreator on mediawikiwiki (T175903) (duration: 00m 46s)
23:12 mutante: acamar - staged BIOS upgrade in progress
23:09 mutante: acamar - done with upgrade - rebooting - it's depooled and removed from resolv.conf - T162850
22:56 mutante: depooled acamar (codfw dns recursor) for BIOS upgrade - installing firmware
22:56 dzahn@neodymium: conftool action : set/pooled=no; selector: name=acamar.*
22:55 maxsem@tin: Synchronized wmf-config/: Labs only: https://gerrit.wikimedia.org/r/378184 https://gerrit.wikimedia.org/r/378187 (duration: 00m 47s)
22:15 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: ACTRIAL going live: https://gerrit.wikimedia.org/r/#/c/378104/ (duration: 00m 46s)
21:22 maxsem@tin: Synchronized php-1.30.0-wmf.18/extensions/ArticleCreationWorkflow/: https://gerrit.wikimedia.org/r/#/c/378125/ (duration: 00m 47s)
20:34 demon@tin: Synchronized php-1.30.0-wmf.18/skins/MinervaNeue/resources/skins.minerva.userpage.icons/userpage.svg: I95d76339 (duration: 00m 45s)
20:18 demon@tin: Synchronized php-1.30.0-wmf.18/includes/api/ApiFeedWatchlist.php: Ibea5bd88 (duration: 00m 46s)
20:04 demon@tin: Synchronized php-1.30.0-wmf.18/extensions/OAuth/backend/MWOAuthDAO.php: I785230df (duration: 00m 46s)
20:02 ppchelko@tin: Finished deploy [cpjobqueue/deploy@af9b590]: Emit broker statistics more frequentl (duration: 00m 30s)
20:02 ppchelko@tin: Started deploy [cpjobqueue/deploy@af9b590]: Emit broker statistics more frequentl
19:58 dcausse: banning elastic1020 to see if T175951 is caused by mixed versions of the ltr plugin
19:36 demon@tin: Synchronized php-1.30.0-wmf.18/skins/MinervaNeue/resources/skins.minerva.icons.images.scripts/userNormal.svg: fix svg syntax (duration: 00m 46s)
19:31 demon@tin: Synchronized php-1.30.0-wmf.17/skins/MinervaNeue/resources/skins.minerva.icons.images.scripts/userNormal.svg: fix svg syntax (duration: 00m 45s)
19:07 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.18
18:58 tgr@tin: Synchronized wmf-config/PrivateSettings.php: set Electron secret for T175868 (duration: 00m 48s)
18:57 tgr@tin: Synchronized private/PrivateSettings.php: set Electron secret for T175868 (duration: 00m 49s)
18:29 twentyafterfour: Phabricator: Deploying hotfix for T175942
18:10 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Configure enwiki to use CirrusSearch MLR by default T175772 (duration: 00m 50s)
18:03 Jamesofur: disabled oauth validation on metawiki/SUL for Teles after verification
17:57 bblack: cp2001 backend restart, mailbox lag
17:56 bblack: cp1074 backend restart, mailbox lag
17:18 eevans@tin: Started restart [electron-render/deploy@8dd5f13]: (no justification provided)
17:16 gehel@tin: Finished deploy [wikimedia/discovery/analytics@ab5d5c1]: moving discovery/analytics to scap3 (duration: 00m 04s)
17:16 gehel@tin: Started deploy [wikimedia/discovery/analytics@ab5d5c1]: moving discovery/analytics to scap3
16:59 krinkle@tin: Synchronized php-1.30.0-wmf.17/extensions/NavigationTiming: T104902 (duration: 01m 00s)
15:51 bblack: repool cp1063 (after wiping varnish storage + restarting all 3x main service daemons)
15:51 bblack: cp1064 backend restart, mailbox lag
15:46 bblack: depool cp1063 (upload eqiad) - seems to be having some iowait issues?
14:07 dereckson@tin: Synchronized php-1.30.0-wmf.18/extensions/CheckUser/specials/SpecialCheckUser.php: Fix Special cu for ip ranges (temp) (T175898) (duration: 00m 49s)
13:34 dcausse@tin: Synchronized wmf-config/CirrusSearch-production.php: Setup Cirrus MLR models for top 20 language AB test (duration: 00m 49s)
13:33 dcausse@tin: Synchronized wmf-config/CirrusSearch-common.php: Setup Cirrus MLR models for top 20 language AB test (duration: 00m 48s)
13:31 dcausse@tin: Synchronized wmf-config/InitialiseSettings.php: Setup Cirrus MLR models for top 20 language AB test (duration: 00m 50s)
11:14 jynus@tin: Synchronized wmf-config/db-eqiad.php: Remove all references to db1049, pool db1100 (duration: 00m 49s)
11:12 akosiaris: upload apertium-crh-tur_0.2.0~r81866-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
11:12 jynus@tin: Synchronized wmf-config/db-codfw.php: Remove all references to db1049 (duration: 00m 50s)
09:51 akosiaris@puppetmaster1001: conftool action : set/pooled=no; selector: name=wtp1025.eqiad.wmnet
09:43 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
09:43 akosiaris@puppetmaster1001: conftool action : set/weight=1; selector: name=wtp1025.eqiad.wmnet
09:41 gehel: re-enabling puppet on all cassandra nodes - T171704
09:41 akosiaris@tin: Finished deploy [parsoid/deploy@cec7d17]: test deploy using dsh groups. T165520 (duration: 09m 30s)
09:32 akosiaris@tin: Started deploy [parsoid/deploy@cec7d17]: test deploy using dsh groups. T165520
09:02 akosiaris@tin: Finished deploy [parsoid/deploy@cec7d17]: test deploy using dsh groups. T165520 (duration: 02m 25s)
09:00 akosiaris@tin: Started deploy [parsoid/deploy@cec7d17]: test deploy using dsh groups. T165520
08:13 gehel: merging cassandra refactoring for puppet - T171704
02:55 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Sep 14 02:55:10 UTC 2017 (duration 7m 28s)
02:47 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 07m 58s)
02:27 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 09m 27s)
00:46 MaxSem: restarted populateIpChanges to use updated code
00:26 twentyafterfour: everything appears to be up and running. Phabricator upgrade complete.
00:23 twentyafterfour: running puppet on phab1001 to restore files which get lost by scap deploy
00:21 twentyafterfour@tin: Finished deploy [phabricator/deployment@7e3a0a8]: (no justification provided) (duration: 00m 01s)
00:21 twentyafterfour@tin: Started deploy [phabricator/deployment@7e3a0a8]: (no justification provided)
00:21 twentyafterfour@tin: Finished deploy [phabricator/deployment@7e3a0a8]: (no justification provided) (duration: 00m 25s)
00:20 twentyafterfour@tin: Started deploy [phabricator/deployment@7e3a0a8]: (no justification provided)
00:18 twentyafterfour: Begin phabricator upgrade #phab-2017-09-13

2017-09-13

23:48 dereckson@tin: Synchronized php-1.30.0-wmf.18/maintenance/populateIpChanges.php: Improve ip_changes update script to insert rows in batches (Gerrit:377918) (duration: 00m 49s)
23:29 dereckson@tin: Synchronized php-1.30.0-wmf.18/extensions/Wikidata/extensions/Wikibase/client: Split page set before constructing InjectRCRecordsJob (T175316) (duration: 00m 57s)
23:28 dereckson@tin: Synchronized php-1.30.0-wmf.18/extensions/Echo/modules/api/mw.echo.api.APIHandler.js: (no justification provided) (duration: 00m 48s)
23:26 dereckson@tin: Synchronized wmf-config/CommonSettings.php: Set $wgScoreSafeMode to false (T174413) (duration: 00m 50s)
22:06 maxsem@tin: Synchronized php-1.30.0-wmf.18/extensions/EventBus/: https://gerrit.wikimedia.org/r/#/c/377900/1 (duration: 00m 49s)
22:03 maxsem@tin: Synchronized php-1.30.0-wmf.18/extensions/ArticleCreationWorkflow/: https://gerrit.wikimedia.org/r/#/c/377900/1 (duration: 00m 49s)
21:55 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/377874/ (duration: 00m 47s)
21:23 ejegg: updated payments-wiki from ed2e481 to 26b16ea
grrrr: populating ip_changes on group1 wikis
20:43 bblack: re-routed ulsfo->eqiad around possible codfw damage as well - https://gerrit.wikimedia.org/r/#/c/377871/
20:34 bblack: deployed https://gerrit.wikimedia.org/r/#/c/377845/ (no gerrit bot, jenkins lint failing, etc) due to possible codfw routing issues
19:39 andrewbogott: rebooting labtestvirt2001
19:35 ppchelko@tin: Finished deploy [changeprop/deploy@86e8103]: Separate kafka metrics by rule (duration: 01m 18s)
19:34 ppchelko@tin: Started deploy [changeprop/deploy@86e8103]: Separate kafka metrics by rule
19:21 urandom: T172384: Upgrading restbase-dev100[5-6] to Cassandra 3.11.0-wmf4
19:19 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.18
19:13 demon@tin: Synchronized php: symlink thingie (duration: 00m 49s)
19:11 urandom: T172384: Upgrading restbase-dev1004 to Cassandra 3.11.0-wmf4 (canary)
18:34 niharika29@tin: Synchronized static/images/project-logos/: Change logo for huwiktionary T175483 (duration: 00m 49s)
18:27 mutante: gerrit2002 - re-enabled puppet - had forgotten to enable it again after recently tryning to test the gerrit->logstash change
18:26 niharika29@tin: Synchronized php-1.30.0-wmf.18/extensions/VisualEditor/: Make ve.dm.Change part of core module T175828 (duration: 00m 50s)
18:25 niharika29@tin: Synchronized docroot/mediawiki/ontology/: Add setup for https://www.mediawiki.org/ontology T171807 (duration: 00m 49s)
18:12 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable Timeless on frwiki T154371 (duration: 00m 50s)
17:55 gwicke: rolling restart of pdfrender service in equiad after hang T174916 T172815
16:42 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=wdqs1001.eqiad.wmnet
16:42 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=wdqs1002.eqiad.wmnet
16:28 gehel: starting decommissioning of wdqs100[12] - T175595
16:25 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs1002.eqiad.wmnet
16:25 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs1001.eqiad.wmnet
16:22 andrewbogott: rebooting labtestvirt2002
16:08 ottomata: restarting druid-brokers with increase in query cache size
15:23 demon@tin: Synchronized php-1.30.0-wmf.18/includes/filerepo/file/LocalFile.php: I2ecb6030 (duration: 00m 49s)
15:22 demon@tin: Synchronized php-1.30.0-wmf.18/includes/diff/DifferenceEngine.php: I2ecb6030 (duration: 00m 49s)
15:21 demon@tin: Synchronized php-1.30.0-wmf.17/includes/filerepo/file/LocalFile.php: I2ecb6030 (duration: 00m 49s)
15:20 demon@tin: Synchronized php-1.30.0-wmf.17/includes/diff/DifferenceEngine.php: I2ecb6030 (duration: 00m 49s)
15:09 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/AbuseFilter/: backporting I13051038 (duration: 00m 52s)
14:31 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,cluster=logstash
14:31 gehel: pooling new logstash servers (logstash100[7-9]) - T175045
14:23 mobrovac@tin: Finished deploy [cpjobqueue/deploy@60d0a78]: Start using the EventBus infrastructure for the updateBetaFeaturesUserCounts job - T175210 (duration: 00m 33s)
14:22 mobrovac@tin: Started deploy [cpjobqueue/deploy@60d0a78]: Start using the EventBus infrastructure for the updateBetaFeaturesUserCounts job - T175210
13:45 akosiaris: upload apertium-crh_0.1.0~r81872-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
13:30 urandom: T169936: Converting RESTBase dev environment to size-tiered compaction
13:29 elukey: update puppet compiler's facts via ./modules/puppet_compiler/files/compiler-update-facts
13:23 gehel: initial installation of logstash100[7-9] - T175045
13:21 zeljkof: EU SWAT finished
13:13 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: Temporary lift account creation limits for WM United Kingdom workshop (T175700) (duration: 00m 49s)
12:14 moritzm: rebooting mc1015 (spare) for some tests
12:07 jynus: stopping wikidata rebuild maintenance script on terbium
12:07 bblack: cp1049 - backend restart, mailbox lag
11:57 jynus@tin: Synchronized wmf-config/db-eqiad.php: Set db1097 as the main api server for s4, Remove regular load from db1070 (duration: 00m 56s)
11:39 jynus: disabling semi-sync replication on the largest s5 database servers
11:03 ariel@tin: Finished deploy [dumps/dumps@3f119ab]: monitor will cd to dumps repo before each run (duration: 00m 02s)
11:03 ariel@tin: Started deploy [dumps/dumps@3f119ab]: monitor will cd to dumps repo before each run
11:02 joal@tin: Finished deploy [analytics/refinery@b2e8852]: Regular analytics weekly deploy (new jars, oozie patches) (duration: 03m 12s)
10:58 joal@tin: Started deploy [analytics/refinery@b2e8852]: Regular analytics weekly deploy (new jars, oozie patches)
10:52 volans: reboot lvs3001
10:06 akosiaris: upload apertium-cat-srd_0.9.0~r82238-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
10:06 akosiaris: upload apertium-srd-ita_0.9.5~r82237-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
09:36 moritzm: installing bind updates from Debian stable/oldstable SUA update (client tools only)
09:25 akosiaris: upload apertium-srd_0.10.0~r82237-1+wmf1_amd64.changes to apt.wikimedia.org/jessie-wikimedia/main T174988
09:24 akosiaris: upload apertium-tur_0.1.0~r81882-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
09:23 akosiaris: upload apertium-ita_0.10.0~r82237-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
09:23 akosiaris: upload apertium-cat_2.3.0~r82237-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
09:05 akosiaris: upload apertium_3.4.2~r68466-2+wmf2 to apt.wikimedia.org/jessie-wikimedia
08:55 elukey: restart varnish backend on cp1072 (upload) - mailbox expiry lag
08:24 moritzm: installing emacs security updates
07:56 akosiaris: bounce varnish on cp1062, mailbox lag
07:47 moritzm: rolling out hhvm-luasandbox 2.0.14 to the remaining hosts in eqiad (along with HHVM restarts) (T173705)
07:08 moritzm: installing tcpdump security updates (fixing 90 CVE IDs...)
05:18 kartik@tin: Finished deploy [cxserver/deploy@ee0081d]: Update cxserver to 3baaa36 (duration: 02m 56s)
05:15 kartik@tin: Started deploy [cxserver/deploy@ee0081d]: Update cxserver to 3baaa36
03:12 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Sep 13 03:12:46 UTC 2017 (duration 7m 30s)
03:05 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 16m 28s)
03:03 jynus: dropping old users from phabricator database
03:00 eileen: update civicrm from ee7dda3 to 187238f
02:41 demon@tin: Synchronized wmf-config/mobile.php: rm unused config var (duration: 00m 46s)
02:27 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 08m 31s)
01:19 jynus: stopping mariadb @ db1048 to clone it to db1059
00:11 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Special:PageLanguage on mul.wikisource Fix correct wiki name for wgPageLanguageUseDB on www.wikisource.org T175622 (duration: 00m 49s)

2017-09-12

23:53 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Do not deploy Timeless on fr.wiktionary for now" (duration: 00m 49s)
23:35 thcipriani@tin: Synchronized wmf-config: SWAT: Enable Flow on Newsletter_talk on mediawiki.org T175502 (duration: 00m 52s)
22:17 chasemp: labvirt1018:~# /sbin/reboot
22:00 bsitzmann@tin: Finished deploy [mobileapps/deploy@b11b75c]: Update mobileapps to 297b048 (T174707 T175305) (duration: 04m 28s)
21:55 bsitzmann@tin: Started deploy [mobileapps/deploy@b11b75c]: Update mobileapps to 297b048 (T174707 T175305)
21:49 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/377527/2 T175725 (duration: 00m 49s)
21:37 MaxSem: Populating ip_changes in group0 wikis
21:33 ejegg: changed contact de-duplication duty cycle to 25 minutes
21:05 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.18
20:33 demon@tin: Finished scap: bootstrap wmf.18 (for real this time) (duration: 26m 23s)
20:06 demon@tin: Started scap: bootstrap wmf.18 (for real this time)
19:53 demon@tin: Finished scap: bootstrap wmf.18 (duration: 08m 14s)
19:45 demon@tin: Started scap: bootstrap wmf.18
19:05 papaul: maintenance complete on mw2256
17:51 bblack: cp3034 - reset fetch_maxchunksize to 0.25G (default) for fe+be
17:40 cwd: turned on dedupe
16:55 niharika29@tin: Finished deploy [iegreview/iegreview@69c4c3f]: Deploying iegreview with scap3 T129154 (duration: 00m 01s)
16:55 niharika29@tin: Started deploy [iegreview/iegreview@69c4c3f]: Deploying iegreview with scap3 T129154
16:40 ejegg: updated CiviCRM from 63288ae to ee7dda3
16:35 niharika29@tin: Finished deploy [scholarships/scholarships@004635d]: Deploying scholarships with scap3 T129134 (duration: 00m 03s)
16:35 niharika29@tin: Started deploy [scholarships/scholarships@004635d]: Deploying scholarships with scap3 T129134
16:26 volans@sarin: conftool action : set/pooled=yes; selector: name=mw2100.codfw.wmnet
16:18 ejegg: updated SmashPig from dce4f0a to dc62203910
15:43 _joe_: restarted ircecho on einsteinium
15:35 gehel: restart elasticsearch on relforge (last check of new plugin deployment) - T158560
15:22 bblack: manually setting fetch_maxchunksize=16MB via varnishadm on cp3034 fe+be instances
15:06 chasemp: disable puppet for lab* things for designate refactor rolleout (delayed a few minutes for a meeting)
15:04 moritzm: uploaded php5.5 5.5.38 to component/ci of jessie-wikimedia (for use in CI)
14:53 papaul: shutting down mw2256 for maintenance
14:43 elukey: add kafka-jumbo IPs to the kafka term of the analytics-in4 filter on cr1/cr2 eqiad
14:42 ema: cp1074 - backend restart, mailbox lag
14:39 bblack: manually setting fetch_maxchunksize=64MB via varnishadm on cp3034 fe+be instances
14:19 jynus@tin: Synchronized wmf-config/db-eqiad.php: Remove all references to db1059 (duration: 00m 44s)
14:18 jynus@tin: Synchronized wmf-config/db-codfw.php: Remove all references to db1059 (duration: 00m 45s)
14:10 gehel: restarting elasticsearch on elastic1020 to validate new plugin deployment - T158560
14:07 gehel: switching elasticsearch plugin deployment to .deb instead of scap on cirrus/eqiad - T158560
14:06 volans: testing reimage of mw2100 - T166300
14:06 gehel: restarting elasticsearch on elastic2001 to validate new plugin deployment - T158560
14:04 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1059, pool db1097 with low load (duration: 00m 45s)
14:02 gehel: switching elasticsearch plugin deployment to .deb instead of scap on cirrus/codfw - T158560
13:37 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs1005.eqiad.wmnet
13:37 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs1004.eqiad.wmnet
13:37 gehel: pooling wdqs100[45] - T171210
13:31 hashar@tin: Synchronized php-1.30.0-wmf.17/extensions/CentralAuth/includes/api/ApiSetGlobalAccountStatus.php: API: Unbreak setglobalaccountstatus locked=lock - T175462 (duration: 00m 44s)
13:26 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable WikidataPageBanner for Russian Wikimedia chapter wiki - T175356 (duration: 00m 45s)
13:25 hashar@tin: Synchronized php-1.30.0-wmf.17/includes/changes/ChangesListBooleanFilter.php: WLFilters: Respect default values - T174725 (duration: 00m 45s)
13:21 hashar@tin: Synchronized wmf-config/CommonSettings.php: Add Extension:Newsletter permissions to CommonSettings (duration: 00m 45s)
13:18 hashar@tin: Synchronized wmf-config/throttle.php: Lift account creation restrictions for WM Taiwan 10th anniversary - T175534 (duration: 00m 45s)
13:15 hashar@tin: Synchronized php-1.30.0-wmf.17/extensions/Wikidata/extensions/Wikibase/view/resources/jquery/wikibase/themes/default/jquery.wikibase.entityselector.css: Remove red highlighting on all entity selectors - T175525 (duration: 00m 45s)
13:12 hashar@tin: Synchronized wmf-config/Wikibase-production.php: Reduce wikiPageUpdaterDbBatchSize to 20 - T173710 (duration: 00m 45s)
09:33 moritzm: upgrading deployment servers to HHVM-Luasandbox 2.0.14
09:32 moritzm: upgrading mw2* to HHVM-Luasandbox 2.0.14
09:25 moritzm: upgrading mw1293-mw1295 (image scalers) to HHVM-Luasandbox 2.0.14 (along with HHVM restarts)
09:24 moritzm: upgrading mw1293-mw1205 (image scalers) to HHVM-Luasandbox 2.0.14 (along with HHVM restarts)
08:38 gehel: deploying elasticsearch plugins on relforge - T158560 (wrong ticket number before)
08:34 gehel: deploying elasticsearch plugins on relforge - T175159
08:28 moritzm: upgrading mw1189-mw1208 (API servers) to HHVM-Luasandbox 2.0.14 (along with HHVM restarts)
07:35 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool es1019 with full weight (duration: 00m 45s)
07:15 moritzm: upgrading mw1180-mw1188, mw1209-1220 (app servers) to HHVM-Luasandbox 2.0.14 (along with HHVM restarts)
02:29 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Sep 12 02:29:20 UTC 2017 (duration 6m 45s)
02:22 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 07m 31s)
02:10 bblack: cp1050 - backend restart, mailbox lag
02:08 bblack: cp1099 - backend restart, mailbox lagh
00:48 awight@tin: Finished deploy [ores/deploy@42c5663]: Cause ORES service restart (duration: 00m 18s)
00:48 awight@tin: Started deploy [ores/deploy@42c5663]: Cause ORES service restart

2017-09-11

23:06 mutante: jenkins-bot says: java.lang.OutofMemoryError: PermGen space
22:30 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4026.*
22:30 dereckson@tin: Synchronized wmf-config/CommonSettings-labs.php: Fix MediaWiki\Auth\AbstractPreAuthenticationProvider case (Gerrit:377281, no-op in prod) (duration: 00m 45s)
21:45 awight@tin: Finished deploy [ores/deploy@42c5663]: Try ORES filehandle fix on new cluster (duration: 13m 47s)
21:41 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4025.*
21:31 awight@tin: Started deploy [ores/deploy@42c5663]: Try ORES filehandle fix on new cluster
21:03 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4023.*
20:13 demon@tin: Pruned MediaWiki: 1.30.0-wmf.13 (duration: 02m 49s)
19:11 chasemp: adjust nodepool rate to 10
19:08 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool es1019 with low load (duration: 00m 46s)
19:01 mutante: restbase-dev1004:/srv# mv /home/eevans/cassandra-1502141526-pid13410.hprof /srv/eevans/ to free disk space
18:52 mutante: restbase - java[12418]: Error: Could not find or load main class org.wikimedia.cassandra.metrics.service.Service
18:40 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable responsive reference columns on wiktionaries, wikivoyages and enwiki https://gerrit.wikimedia.org/r/#/c/376573/, https://gerrit.wikimedia.org/r/#/c/371630/ (duration: 00m 46s)
18:33 robh: firmware update on scs-a1-codfw and scs-c1-codfw has been complete and tested good T174475
18:29 chasemp: bump nodepool rate to 15 and max-servers to 18 (no restart as it picks up config periodically)
18:29 robh: firmware update on scs-a1-codfw and scs-c1-codfw has been started, awaiting completion T174475
18:24 robh: updating firmware on scs-a1-codfw and scs-c1-codfw
18:10 robh: scs-ulsfo has successfully updated firmware T174475
18:03 robh: upgrading firmware on scs-ulsfo
17:09 gehel@tin: Finished deploy [wdqs/wdqs@177d20a]: (no justification provided) (duration: 02m 39s)
17:07 gehel: wdqs weekly deployment (logging and throttling fixes)
17:06 gehel@tin: Started deploy [wdqs/wdqs@177d20a]: (no justification provided)
15:20 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4022.*
15:20 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4028.*
14:40 godog: roll-restart thumbor to apply https://gerrit.wikimedia.org/r/#/c/377264/ and upgrade to 1.4 - T173580 T174997
14:24 hashar: tin.eqiad.wmnet: reverted patch "WLFilters: Respect default values" that got merged but left undeployed
14:23 hashar: tin.eqiad.wmnet: reverted all four mediawiki-config that got merged but left undeployed and rebased the workspace
14:05 hashar: European SWAT cancelled due to deploy infrastructure issue
13:55 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 27s)
13:54 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 27s)
13:50 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 28s)
13:43 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 28s)
13:40 hashar@tin: Synchronized static/images/project-logos/huwiktionary.png: Change logo for huwiktonary - T175483 (duration: 00m 28s)
11:43 moritzm: installing file/libmagic security updates (stretch-specific, does not affect older distros)
10:30 jynus: restarting mysql @ labsdb1001
09:40 godog: roll-restart thumbor to apply changes for gif max animated area - T173580
09:39 mobrovac@tin: Finished deploy [cpjobqueue/deploy@e73a10f]: Initial deploy of cpjobqueue - T175281 (duration: 00m 20s)
09:39 mobrovac@tin: Started deploy [cpjobqueue/deploy@e73a10f]: Initial deploy of cpjobqueue - T175281
08:44 mobrovac@tin: Finished deploy [zotero/translators@a0c41c3]: Update translators to upstream e03695273 - T174992 (duration: 00m 07s)
08:44 mobrovac@tin: Started deploy [zotero/translators@a0c41c3]: Update translators to upstream e03695273 - T174992
08:16 volans: reimage script tests continue on mc100[1-2] T166300
07:59 moritzm: installing systemd updates from stretch point update
07:41 moritzm: installing remaining libonig security updates
02:41 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Sep 11 02:41:43 UTC 2017 (duration 7m 0s)
02:34 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 08m 10s)

2017-09-10

21:00 bawolff: disabled Jforrester's gerrit account and flush sessions
13:41 elukey: restart Varnish backend on cp1073 (cache::upload) for mailbox expiry lag
13:13 elukey: restart cp1053's varnish backend for mailbox expiry lag and 503s - T175473
00:37 chasemp: labnodepool1001:~# service nodepool restart
00:36 bd808: wmcs: purged rabbitmq ceilometer_notifications.* queues

2017-09-09

23:42 chasemp: labnodepool1001:~# service nodepool start
23:06 chasemp: freeswap && labcontrol1001:/home/rush# service rabbitmq-server restart
22:59 chasemp: restart nova-api and nova-network
22:49 chasemp: labnodepool1001:~# service nodepool stop
22:44 madhuvishy: Running service rabbitmq-server restart on labcontrol1001
20:41 legoktm@tin: Synchronized php-1.30.0-wmf.17/includes/filerepo/file/LocalFile.php: Fix issues related to comments and files in UI and API - T175443 T175444 (duration: 00m 46s)
01:12 ebernhardson@tin: Synchronized php-1.30.0-wmf.17/extensions/WikimediaEvents/modules/ext.wikimediaEvents.humanSearchRelevance.js: T171740: Reduce annoyance of survey by enforcing minimum 2 days between showing survey to same browser (duration: 00m 46s)

2017-09-08

23:46 ebernhardson@tin: Synchronized wmf-config/CirrusSearch-rel-survey.php: T171740: Fix inverted sampling rates for human relevance survey (duration: 00m 47s)
23:38 mutante: netmon1002 - terminated unused screen session
23:09 mutante: phab1001, phab2001 - terminating 3 unused screen processes that were from iridium migration / data sync
21:54 mutante: gerrit2001 - restarting gerrit to test logstash config change (while not touching "live" gerrit)
20:24 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/Flow/includes/: bugs and stuff (duration: 00m 54s)
20:14 mutante: heze - rebooting for firmware upgrade
20:10 mutante: heze (bacula storage) - installing BIOS upgrade (T162850)
20:10 demon@tin: Synchronized scap/scap.cfg: drop git_repo for now, T175041 (duration: 00m 46s)
19:43 mutante: zosma.codfw.wmnet - delete salt key, puppet node clean, puppet node deactivate, remove from Icinga,... (T138650)
19:37 mutante: removing ganeti instance 'zosma' on ganeti2001 (T138650)
18:17 demon@tin: Synchronized wmf-config/FeaturedFeedsWMF.php: no-op (duration: 00m 46s)
16:04 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/AbuseFilter/includes/Views/AbuseFilterViewExamine.php: T175338, followup (duration: 00m 46s)
16:03 demon@tin: Synchronized php-1.30.0-wmf.17/includes/revisiondelete/RevDelLogList.php: typofix (duration: 00m 46s)
15:53 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4027.*
15:47 ejegg: increased cURL timeout for PayPal IPN confirmation to 14 sec
15:07 akosiaris@tin: Finished deploy [librenms/librenms@5554213]: Testing new scap keyholder_key config (duration: 00m 04s)
15:06 akosiaris@tin: Started deploy [librenms/librenms@5554213]: Testing new scap keyholder_key config
15:02 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4021.*
14:58 volans: testing wmf-auto-reimage also on mc1002 T166300 T164341
12:36 _joe_: restarting a check of the md2 devices on conf200* T162013 after reducing sync speed
12:36 _joe_: reduced raid resync max speed of raid devices on conf200* to 20000
10:35 hashar@tin: Synchronized php-1.30.0-wmf.17/extensions/ProofreadPage: Restore page status buttons - T175304 (duration: 00m 50s)
10:12 _joe_: stopped all md devices syncs on conf200* machines
10:09 _joe_: etcd cluster lost consensus T162013
10:07 volans: testing wmf-auto-reimage on mc1001 T166300 T164341
09:58 _joe_: running the same command on conf2002 too
09:48 _joe_: running `/usr/share/mdadm/checkarray --cron --all --idle --quiet` on conf2001 and conf2003 trying to reproduce consensus issues in T162013
09:25 moritzm: restarting apache on tegmen/einsteinium to pick up security update
09:09 moritzm: installing libonig security updates
09:08 mobrovac@tin: Finished deploy [restbase/deploy@0d39acf] (dev-cluster): Use double writing for mobileapps in the dev cluster, take #2 - T169940 (duration: 01m 01s)
09:07 mobrovac@tin: Started deploy [restbase/deploy@0d39acf] (dev-cluster): Use double writing for mobileapps in the dev cluster, take #2 - T169940
09:01 moritzm: installing remaining gnutls updates from jessie point release
08:13 moritzm: installing libgd2 security updates
08:11 mobrovac@tin: Started deploy [restbase/deploy@0d39acf] (dev-cluster): Use double writing for mobileapps in the dev cluster - T169940
08:11 mobrovac: restbase enabled back puppet in the dev cluster - T169940
07:53 moritzm: installing ruby2.3 security updates
07:32 mobrovac@tin: Finished deploy [restbase/deploy@0d39acf] (dev-cluster): (no justification provided) (duration: 07m 02s)
07:25 mobrovac@tin: Started deploy [restbase/deploy@0d39acf] (dev-cluster): (no justification provided)
07:21 mobrovac@tin: Finished deploy [restbase/deploy@e6aeeeb] (dev-cluster): (no justification provided) (duration: 00m 11s)
07:21 mobrovac@tin: Started deploy [restbase/deploy@e6aeeeb] (dev-cluster): (no justification provided)
00:42 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/AbuseFilter/includes/Views/AbuseFilterViewExamine.php: fix comment stuff (duration: 00m 46s)

2017-09-07

23:42 bblack: cp1066 - depooled all traffic services, seems to have some bug causing 503 spikes
23:19 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.17
23:15 XenoRyet: Updated Smashpig from 8eb98c1 to dce4f0a
22:41 niharika29@tin: Synchronized php-1.30.0-wmf.17/extensions/ArticleCreationWorkflow/: Bump extension version (duration: 00m 49s)
22:03 niharika29@tin: Finished scap: Deploying ArticleCreation Workflow on test2 wiki TT175302 (duration: 23m 00s)
21:40 niharika29@tin: Started scap: Deploying ArticleCreation Workflow on test2 wiki TT175302
21:23 demon@tin: Synchronized php-1.30.0-wmf.17/includes/api/ApiQueryRecentChanges.php: T175307 (duration: 00m 49s)
21:01 urandom: T169940: Disabling puppet and restbase service in RESTBase dev
21:00 urandom: T169940: Disabling changeprop in RESTBase dev environment
19:05 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/MobileFrontend/includes/specials/SpecialMobileHistory.php: T175161 (duration: 00m 49s)
18:48 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.17
18:47 demon@tin: Synchronized php: symlink update -> wmf.17 (duration: 00m 48s)
18:39 legoktm: restarted script to reparse all pages in parsoid for Linter (python3 parsoid_reparse.py http://parsoid.discovery.wmnet:8000 --sitematrix --linter-only --skip-closed https://aa.wikipedia.org/w/api.php) - T161556
17:55 ejegg: disabled queue consumers for data cleaning job
17:52 ejegg: updated CiviCRM from c1ece1e to 63288ae
17:35 otto@tin: Finished deploy [analytics/refinery@6b60f2c]: (no justification provided) (duration: 03m 23s)
17:32 otto@tin: Started deploy [analytics/refinery@6b60f2c]: (no justification provided)
17:31 otto@tin: Finished deploy [analytics/refinery@bdf6754]: (no justification provided) (duration: 00m 23s)
17:31 otto@tin: Started deploy [analytics/refinery@bdf6754]: (no justification provided)
17:23 otto@tin: Finished deploy [analytics/refinery@bdf6754]: (no justification provided) (duration: 03m 25s)
17:20 otto@tin: Started deploy [analytics/refinery@bdf6754]: (no justification provided)
16:13 demon@tin: Synchronized wmf-config/InitialiseSettings.php: Remove Extension:RelatedSites from zhwikivoyage (duration: 00m 50s)
15:32 mobrovac@tin: Finished deploy [restbase/deploy@77961d0]: Revert using MCS for the summary end point - T168848 (duration: 06m 18s)
15:25 mobrovac@tin: Started deploy [restbase/deploy@77961d0]: Revert using MCS for the summary end point - T168848
14:45 _joe_: manually running /usr/share/mdadm/checkarray --cron --all --idle --quiet on conf2001, trying to reproduce consensus issues in T162013
14:40 jynus: rebooting and upgrading es1019
13:54 godog: powercycle ms-be2023 - load through the roof and no login possible
13:54 oblivian@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=jobrunner,name=eqiad
13:14 zeljkof: EU SWAT finished
13:13 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add English Wiktionary as a client of Wikidata (T159316) (duration: 00m 49s)
13:12 zfilipin@tin: Synchronized dblists/wikidataclient.dblist: SWAT: Add English Wiktionary as a client of Wikidata (T159316) (duration: 00m 49s)
11:24 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1100 with 0 weight - T172679 (duration: 00m 49s)
09:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1100 depooled to s5 array - T172679 (duration: 00m 49s)
08:59 ariel@tin: Finished deploy [dumps/dumps@da05e9f]: add general info to siteinfo dumps (duration: 00m 02s)
08:58 ariel@tin: Started deploy [dumps/dumps@da05e9f]: add general info to siteinfo dumps
08:44 elukey: restart varnish backend on cp1063 - mailbox expiry lag
07:57 elukey: force re-mount of /mnt/hdfs on stat1005
07:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1100 - T172679 (duration: 00m 48s)
07:53 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1100 - T172679 (duration: 00m 49s)
07:45 mobrovac@tin: Finished deploy [restbase/deploy@b51c58c]: Use MCS for the summary endpoint - T168848 (duration: 06m 36s)
07:39 mobrovac@tin: Started deploy [restbase/deploy@b51c58c]: Use MCS for the summary endpoint - T168848
07:35 mobrovac@tin: Finished deploy [restbase/deploy@b51c58c] (staging): Use MCS for the summary endpoint (duration: 01m 43s)
07:33 mobrovac@tin: Started deploy [restbase/deploy@b51c58c] (staging): Use MCS for the summary endpoint
07:20 oblivian@tin: Synchronized rpc/RunSingleJob.php: Adding the RunSingleJob.php rpc endpoint for jobrunners (duration: 00m 56s)
06:07 eileen: civicrm updated from 73a24b4 to c1ece1e
03:13 eileen: seconds behind=78
03:08 eileen: seconds behind 55
02:52 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Sep 7 02:52:05 UTC 2017 (duration 7m 15s)
02:44 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 06m 15s)
02:43 eileen: restarting groupmember.load after hacking out cache delete calls
02:40 eileen: update civicrm from 764bfe1 to 73a24b4
02:24 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 07m 23s)
02:05 eileen: replag 217 behind
01:56 eileen: replag 197 seconds
01:47 eileen: replag 23 seconds
01:29 eileen: replag 0
01:22 eileen: replag 550 sec behind master
01:13 eileen: replag delay 422
01:12 eileen: job killed
00:49 eileen: 8389 contacts now
00:49 eileen: 7971 contacts so far
00:46 eileen: started Omnigroupmember.load
00:24 twentyafterfour: phabricator update deployed
00:12 twentyafterfour@tin: Finished deploy [phabricator/deployment@c265c1a]: (no justification provided) (duration: 00m 24s)
00:11 twentyafterfour@tin: Started deploy [phabricator/deployment@c265c1a]: (no justification provided)
00:07 twentyafterfour: deploying phabricator update 2017-09-06 https://phabricator.wikimedia.org/project/view/2980/

2017-09-06

23:41 twentyafterfour@tin: Synchronized php-1.30.0-wmf.16/extensions/WikimediaEvents/: Sync Change-Id: I7ae522 to all webservers refs T174387 (duration: 00m 49s)
23:28 twentyafterfour@tin: Synchronized php-1.30.0-wmf.17/extensions/WikimediaEvents/: Sync Change-Id: I7ae522 to wmf.17 refs T174387 (duration: 00m 49s)
23:23 twentyafterfour@tin: Synchronized wmf-config/CirrusSearch-common.php: CirrusSearch-common.php goes last (duration: 00m 48s)
23:21 twentyafterfour@tin: Synchronized wmf-config/InitialiseSettings.php: Now sync initializesettings (duration: 00m 49s)
23:19 twentyafterfour@tin: Synchronized wmf-config/CirrusSearch-rel-survey.php: sync the added file first (duration: 00m 49s)
23:17 twentyafterfour@tin: scap failed: average error rate on 9/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details)
23:12 twentyafterfour@tin: Synchronized wmf-config/: roll-back due to huge error rate spike (duration: 00m 51s)
23:10 twentyafterfour@tin: Synchronized wmf-config/: deploy config for CirrusSearch human relevancy survey. Change-Id: I272c69 Bug: T174106 (duration: 00m 52s)
22:18 no_justification: prior deploy was no-op
22:18 demon@tin: Finished deploy [gerrit/gerrit@d4f9a77]: (no justification provided) (duration: 00m 07s)
22:18 demon@tin: Started deploy [gerrit/gerrit@d4f9a77]: (no justification provided)
22:11 bsitzmann@tin: Finished deploy [mobileapps/deploy@507a479]: Update mobileapps to 2cb6281 (T168848 T169277 T169274 T162179 T164033 T167921 T174698 T168848 T174808) (duration: 04m 53s)
22:06 bsitzmann@tin: Started deploy [mobileapps/deploy@507a479]: Update mobileapps to 2cb6281 (T168848 T169277 T169274 T162179 T164033 T167921 T174698 T168848 T174808)
21:39 niharika29@tin: Synchronized wmf-config/CommonSettings-labs.php: Config for ArticleCreationWorkflow T175054 (duration: 00m 50s)
21:28 ejegg: updated payments-wiki from bf97cd2 to ed2e481
21:24 ejegg: updated CiviCRM from f9c373e to 764bfe1
21:17 ottomata: rebooting kafka-jumbo1004
20:34 arlolra: Updated Parsoid to f9d367ea (T169342)
20:27 arlolra@tin: Finished deploy [parsoid/deploy@f07ac8c]: Updating Parsoid to f9d367ea (duration: 08m 27s)
20:18 arlolra@tin: Started deploy [parsoid/deploy@f07ac8c]: Updating Parsoid to f9d367ea
19:53 ottomata: reimaging kafka-jumbo* with stretch
19:48 chasemp: reboot labvirt1018
18:50 maxsem@tin: Synchronized wmf-config/: just to make sure... (duration: 00m 50s)
18:48 maxsem@tin: Synchronized wmf-config/CommonSettings.php: revert (duration: 00m 49s)
18:33 maxsem@tin: Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/370292/3 (duration: 00m 49s)
18:25 maxsem@tin: Synchronized wmf-config/CommonSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/370291/2 (duration: 00m 49s)
18:19 maxsem@tin: Synchronized wmf-config/Wikibase.php: https://gerrit.wikimedia.org/r/#/c/376317/2 (duration: 00m 48s)
18:18 maxsem@tin: Synchronized wmf-config/Wikibase-production.php: https://gerrit.wikimedia.org/r/#/c/376317/2 (duration: 00m 48s)
18:09 maxsem@tin: Synchronized portals: (no justification provided) (duration: 00m 49s)
18:09 ejegg: rolled payments-wiki back to bf97cd2
18:08 maxsem@tin: Synchronized portals/prod/wikipedia.org/assets: (no justification provided) (duration: 00m 50s)
18:05 ejegg: updated payments-wiki from bf97cd2 to 6911158
17:53 bblack: cp1048 - varnish backend restart for mailbox lag...
17:23 demon@tin: Synchronized wmf-config/FeaturedFeedsWMF.php: code cleanup (duration: 00m 49s)
17:20 mutante: added LDAP user schoenbaechler to WMF group (T175156)
17:16 mutante: modify-ldap-group on terbium is broken
17:02 ejegg: disabled creating CiviMail records for Thank You emails
16:51 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/Flow/includes/: I284b5aa (duration: 01m 01s)
16:41 chasemp: disable puppet for cloud things to test changes
16:40 bblack: cp1072 - varnish backend restart (mailbox lag)
16:40 bblack: cp1062 - varnish backend restart (mailbox lag)
15:55 ejegg: re-enabled IPN notification at PayPal console (48 minutes ago)
15:53 akosiaris: powercycling lvs3001
15:28 papaul: firmware upgrade on mw2256
15:11 bblack: cp1049 - restart varnish backend (mailbox lag)
15:07 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool es1019 (duration: 00m 49s)
15:06 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2040 (duration: 00m 49s)
14:17 gehel: wdqs1005 is coming up after a few reimaging issues, expect some icinga noise...
13:19 jynus: restarting and upgrading db2040
13:13 reedy@tin: Synchronized wmf-config/Wikibase-production.php: Move some wikidata config T174962 (duration: 00m 48s)
13:12 reedy@tin: Synchronized wmf-config/Wikibase-labs.php: Move some wikidata config T174962 (duration: 00m 49s)
13:07 reedy@tin: Synchronized wmf-config/throttle.php: Throttle exception T175113 (duration: 00m 49s)
12:08 moritzm: installing libapache2-mod-perl update from jessie 8.9 update
11:41 moritzm: installing gtk+2.0 update from jessie 8.9 update
11:24 elukey: temporarily raise kafka log4j authorizer verbosity to DEBUG on kafka1012 - T173493
11:21 moritzm: installing gnutls update from jessie 8.9 and stretch 9.1 point updates
11:20 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2040 (duration: 00m 49s)
11:20 godog: reimage restbase1008 with cassandra 3 - T169939
11:15 marostegui: Disable puppet on db1100 for mydumper/myloader
10:26 moritzm: installing perl update from stretch 9.1 point release
10:25 moritzm: installing perl update from jessie 8.9 point release
10:12 godog: reimage restbase1010 with cassandra 3 - T169939
09:39 moritzm: installing libgd security updates
09:28 moritzm: installing libonig security uodates#
09:05 jynus: disabling puppet on most db hosts to merge firewall changes safely
08:53 godog: reimage restbase1009 with cassandra 3 - T169939
08:49 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T174948 Add WMDE log channel (duration: 00m 49s)
08:43 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T110170 Enable Newsletter on mediawikiwiki (duration: 00m 51s)
07:15 marostegui: Stop MySQL on db1049 to copy its content to db1100 - https://phabricator.wikimedia.org/T172679
06:46 moritzm: installing php-luasandbox 2.0.14 on API canaries along with HHVM restart (T173705)
06:37 marostegui: Truncate l10n_cache table across production - T150306
04:08 demon@tin: Synchronized wmf-config/throttle.php: no-op (duration: 00m 49s)
03:14 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Sep 6 03:14:16 UTC 2017 (duration 7m 6s)
03:07 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 14m 49s)
02:31 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 08m 18s)
away: disabled IPN messages at the PayPal console
00:14 Reedy: that was T174747 Adjust language icon color
00:14 reedy@tin: Synchronized php-1.30.0-wmf.16/skins/MinervaNeue: (no justification provided) (duration: 00m 49s)
00:10 reedy@tin: Synchronized php-1.30.0-wmf.16/extensions/Scribunto/tests/phpunit/: Fix broken test (duration: 00m 49s)
00:09 reedy@tin: Synchronized php-1.30.0-wmf.17/extensions/Scribunto/tests/phpunit/: Fix broken test (duration: 00m 50s)

2017-09-05

23:58 reedy@tin: Synchronized wmf-config/: Test ArticleCreationWorkflow extension on the Beta Cluster (duration: 00m 51s)
23:39 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: Enable AbuseFilter runtime profile T161059 (duration: 00m 49s)
23:33 foks: 2FA disabled for Omshivaprakash (T175075)
22:17 robh: rebooting mw1163 as its serial console is locked up
22:07 demon@tin: Synchronized php-1.30.0-wmf.16/extensions/FlaggedRevs/frontend/specialpages/reports/: kill some warnings and stuff (duration: 02m 55s)
22:06 demon@tin: Synchronized php-1.30.0-wmf.16/extensions/FlaggedRevs/frontend/specialpages/reports/: (no justification provided) (duration: 03m 13s)
20:48 ejegg: re-enabled donation queue consumer and thank you mailer
20:40 ejegg: updated civicrm from 820bd98 to f9c373e
20:37 ejegg: disabled Donation queue consumer and thank-you mail sender
19:38 gilles@tin: Synchronized private/PrivateSettings.php: Add Thumbor username to Swift configuration (duration: 00m 48s)
19:33 jynus: rebuilding querycache table on tewiktionary.dbstore1002, it had crashed
19:28 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: wmf.17 on group0
19:05 Reedy: deleted querycache rows where qc_type = '' on all wikis T174513
18:11 ejegg: updated fundraising tools from 9554229 to 8d39806
18:11 demon@tin: Finished scap: wmf.17 bootstrap (duration: 47m 00s)
17:38 bblack: cp1074 - restart varnish backend, mailbox lag
17:24 demon@tin: Started scap: wmf.17 bootstrap
17:14 demon@tin: Pruned MediaWiki: 1.30.0-wmf.12 (duration: 02m 32s)
16:59 andrewbogott: reducing the number of concurrent nova-conductor workers. May cause hiccups as services restart
16:58 elukey: ran authdns-update from ns1.w.o after https://gerrit.wikimedia.org/r/374385 to create electcom.wikimedia.org
16:51 jynus: reloading dbproxy1005 to repoing to db1009 again- things seem stable right now
16:45 elukey: add new virtualhost electcom.wikimedia.org to the appservers apache config - https://gerrit.wikimedia.org/r/374389 (implies apache config reload)
16:33 mattflaschen@tin: Synchronized php-1.30.0-wmf.16/: Prepare to enable RCFilters (WLFilters) on Watchlist, but without i18n changes (reverted) (duration: 12m 41s)
16:14 addshore: addshore@terbium:~$ mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki mediawikiwiki --extension newsletter
16:08 akosiaris: reboot all kubernetes boxes T170119
15:43 matt_flaschen: 'Scap sync failed on i18n, so I'll deploy just the non-i18n ones'
15:36 mattflaschen@tin: scap aborted: Prepare to enable RCFilters (WLFilters) on Watchlist (duration: 08m 42s)
15:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1083 - T174509 (duration: 00m 47s)
15:27 mattflaschen@tin: Started scap: Prepare to enable RCFilters (WLFilters) on Watchlist
14:30 reedy@tin: Synchronized wmf-config/: phpcs (duration: 00m 47s)
14:28 reedy@tin: Synchronized tests/: phpcs (duration: 00m 46s)
14:26 reedy@tin: Synchronized errorpages/404.php: phpcs (duration: 00m 45s)
14:25 reedy@tin: Synchronized docroot/noc/conf/highlight.php: Fix links for dblists in highlight.php (duration: 00m 44s)
14:15 zeljkof: EU SWAT finished
14:13 zfilipin@tin: Synchronized php-1.30.0-wmf.16/includes/EditPage.php: SWAT: Re add wpScrolltop id in EditPage (T174723) (duration: 00m 45s)
13:47 zfilipin@tin: Synchronized php-1.30.0-wmf.16/extensions/ContentTranslation/: SWAT: encodeURIComponent title to escape / properly (T174792) (duration: 00m 48s)
13:30 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add abusefilter-view-private to rollbackers in zhwiki (T174978) (duration: 00m 46s)
13:28 godog: reimage restbase2005 - T169939
13:17 marostegui: Drop pr_index tables from where ProofreadPage isn't enabled - T174782
13:14 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Update logo for sr.wikibooks.org (T172284) (duration: 00m 46s)
12:24 hashar: ci: switched mediawiki/core mediawiki/vendor php5.5 jobs from trusty to jessie - T161882
12:15 ladsgroup@tin: Synchronized wmf-config/Wikibase-production.php: Enable sendEchoNotification for enwiki, dewiki, frwiki (T142102) (duration: 00m 46s)
11:54 marostegui: Drop reader_feedback, reader_feedback_history, reader_feedback_pages tables - T174586
11:07 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: (Actually) Enable the ArticlePlaceholder on sqwiki (T174335) (duration: 00m 45s)
11:05 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on sqwiki (T174335) (duration: 00m 46s)
10:38 hashar: Nodepool / CI are fully backup and processing queued jobs
10:25 hashar: Restarting Nodepool. I have managed to spawn instances manually for contintcloud tenant
10:18 _joe_: stopping puppet, nginx on mw2249 for experimentation
10:11 ema: reboot cp4024 T174891
10:08 godog: copy python-logstash from jessie-wikimedia to stretch-wikimedia
09:59 hashar: Stopped Nodepool on labnodepool1001.eqiad.wmnet
09:52 akosiaris: reimage kubernetes[12]00[1-4] T170119
08:49 moritzm: upgrading canary app servers to hhvm-luasandbox 2.0.14 (T173705)
08:47 moritzm: uploaded php-luasandbox/hhvm-luasandbox 2.0.14 to apt.wikimedia.org (T173705)
07:14 moritzm: installing libgd security updates on canary app servers (along with hhvm restart)
07:03 _joe_: launching manually 3 workers for refreshLinks jobs on commons, T173710
06:32 marostegui: Deploy alter table on db1083 - T174509
06:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1083 - T174509 (duration: 00m 46s)
06:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1089 - T174509 (duration: 00m 46s)
06:09 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1089 - T174509 (duration: 00m 55s)
02:30 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Sep 5 02:30:03 UTC 2017 (duration 6m 37s)
02:23 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 07m 28s)

2017-09-04

23:32 Dereckson: Set e-mail to Hexabot bot account to allow account recovery (T174973)
22:07 legoktm: starting script to reparse all pages in parsoid for Linter (python2 parsoid-reparse.py http://parsoid.discovery.wmnet:8000 --sitematrix --linter-only --skip-closed https://aa.wikipedia.org/w/api.php) - T161556
17:50 moritzm: uploaded debdeploy 0.0.99-2 to apt.wikimedia.org
17:19 gehel@tin: Finished deploy [wdqs/wdqs@1caaa30]: weekly WDQS deployment (duration: 02m 46s)
17:17 gehel@tin: Started deploy [wdqs/wdqs@1caaa30]: weekly WDQS deployment
16:29 mobrovac@tin: Finished deploy [restbase/deploy@8c9c436]: (no justification provided) (duration: 06m 23s)
16:23 mobrovac@tin: Started deploy [restbase/deploy@8c9c436]: (no justification provided)
16:10 mobrovac: restbase depooled restbase10(0[89]|10) and restbase2005 for T169939
16:07 _joe_: rolling restart of pybal in codfw/eqiad low-traffic pools for picking up the new jobrunner LVS endpoint
15:42 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,service=nginx
14:51 godog: reimage restbase2003 for use with cassandra 3 / jbod - T169939
14:27 moritzm: enabled LDAP authentication for ganglia.wikimedia.org
13:37 gehel: rolling restart of elasticsearch/logstash to cleanup unused plugins - T174933
13:34 gehel: delete leftover grafana-dashboards index on elasticsearch logstash cluster
13:22 gehel: restart elasticsearch on logstash1006 to test plugin removal - T174933
12:26 moritzm: uploaded debdeploy 0.0.99-1+trusty/trusty-wikimedia to apt.wikimedia.org
12:18 marostegui: Stop MySQL on db1037 as it is going to be decommissioned - T174902
12:10 elukey: rolling restart of zookeeper on conf100[123] for jvm security updates
11:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1037 as it will be decommissioned - T174902 (duration: 00m 46s)
11:52 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1037 as it will be decommissioned - T174902 (duration: 00m 46s)
11:45 godog: restart varnish on cp1099 - mailbox lag
10:22 ariel@tin: Finished deploy [dumps/dumps@0b32eef]: fix abstract dumps variants regression (duration: 00m 02s)
10:22 ariel@tin: Started deploy [dumps/dumps@0b32eef]: fix abstract dumps variants regression
09:51 moritzm: uploaded debdeploy 0.0.99-1+jessie/jessie-wikimedia to apt.wikimedia.org
09:41 mobrovac@tin: Started restart [mathoid/deploy@44ea6d8]: Restart for libxml update
09:36 moritzm: uploaded debdeploy 0.0.99-1/stretch-wikimedia to apt.wikimedia.org
08:52 elukey: restart zookeeper on conf2003 for jvm security updates
08:48 marostegui: Stop MySQL on db1045 as it will be decommissioned - T174806
08:47 moritzm: installing mercurial security updates
08:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1045 as it will be decommissioned - T174806 (duration: 00m 46s)
08:39 moritzm: installing libgcrypt20 security updates
08:39 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1045 as it will be decommissioned - T174806 (duration: 00m 48s)
08:33 godog: upload scap 3.7.0-1 to apt.w.o - T127762
07:39 moritzm: installing gnupg security updates
07:03 marostegui: Deploy alter table on s3 codfw master with replication on kpbwiki - T168661
07:02 _joe_: starting additional runJobs instance for htmlcacheupdate on commons T173710
04:52 marostegui: Deploy alter table on db1052 - T168661
02:44 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Sep 4 02:44:15 UTC 2017 (duration 6m 52s)
02:37 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 08m 17s)

2017-09-03

17:52 elukey: depooled cp4024 (ulsfo upload) due to kernel errors in dmesg
15:57 ema: restart varnish backend on cp1073 (mailbox lag)
09:41 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4024.ulsfo.wmnet,service=varnish-fe
09:41 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4024.ulsfo.wmnet,service=nginx
09:25 ema: cp4024 is down, powercycled
09:13 _joe_: manually fixed value of /conftool/v1/pools/eqiad/api_appserver/apache2/mw1208.eqiad.wmnet in codfw to allow replication to restart
00:59 legoktm: legoktm@terbium:~$ mwscript extensions/WikimediaMaintenance/filebackend/setZoneAccess.php --wiki=hiwikiversity --backend=local-multiwrite # T174859

2017-09-01

16:18 bblack: restart varnish backend on cp1074 (mailbox lag)
{{safesubst:SAL entry|1=15:15 urandom: Restarting Cassandra: restbase-dev100[5-6]-{a,b}}}
{{safesubst:SAL entry|1=15:06 urandom: Restarting Cassandra: restbase-dev1004-{a,b}}}
15:00 reedy@tin: Synchronized php-1.30.0-wmf.16/extensions/Popups/: T174724 (duration: 00m 46s)
14:28 marostegui: Add 150G to /srv partition on labsdb1001
13:42 marostegui: Rename table pr_index on enwiki on db1089 - T174782
13:10 marostegui: Stop MySQL on db1026 as it will be decommissioned - T174763
12:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1081 original weight - T168661 (duration: 00m 43s)
12:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1026 as it will be decommissioned - T174763 (duration: 00m 43s)
12:17 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1026 as it will be decommissioned - T174763 (duration: 00m 43s)
11:50 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1081 weight - T168661 (duration: 00m 43s)
11:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1081 with low weight - T168661 (duration: 00m 44s)
10:35 elukey: stop puppet on thorium and disable root rsyncs - T174756
10:31 marostegui: Upgrade MariaDB to 10.0.32 on db1081 - T168661
10:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1081 - T168661 (duration: 00m 43s)
09:10 godog: bounce graphite-web on graphite1001, problematic query? using all CPU
09:04 ema: lvs1007 upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882
08:59 godog: depool restbase200[135] before reimage - T169939
07:22 elukey: restart apache2 and hue on thorium, Analytics sites down, investigating
07:06 marostegui: Power reset db2044 as it is unresponsive - T174764
03:54 krinkle@tin: Synchronized wmf-config/InitialiseSettings.php: I4dfc33f66c3 - Enable jQuery 3 on nlwiki sister projects (duration: 00m 43s)
00:57 tstarling@tin: Synchronized php-1.30.0-wmf.16/includes/parser/Parser.php: (no justification provided) (duration: 00m 44s)
00:13 RainbowSprinkles: gerrit: flushed all non-login caches, things might be sluggish for the next ~15mins or so

2017-08-31

20:03 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: (no justification provided)
19:27 mattflaschen@tin: Finished scap: Watchlist filters: Convert edit watchlist button to new UX and fix server-side tag filtering. T172030 (duration: 21m 23s)
19:17 cmjohnson1: powercycled mw1294
19:05 mattflaschen@tin: Started scap: Watchlist filters: Convert edit watchlist button to new UX and fix server-side tag filtering. T172030
18:53 mattflaschen@tin: Synchronized docroot/noc/conf/interwiki-labs.php.txt: Interwiki links on Beta Cluster: T69931 (duration: 00m 47s)
18:51 mattflaschen@tin: Synchronized wmf-config: Interwiki links on Beta Cluster: T69931 (duration: 00m 49s)
17:50 demon@tin: Synchronized php-1.30.0-wmf.16/skins/MinervaNeue/resources/skins.minerva.icons.images.scripts/watch.svg: sorta impromptu swat thing (duration: 00m 47s)
16:35 ejegg: re-enabled donation queue consumer and thank you mail sender
15:25 ejegg: updated civicrm from f00611d to 820bd98
15:20 ejegg: updated SmashPig from 8f4388d to 8eb98c1
15:19 ejegg: turned off donation queue consumer and thank you job
15:18 elukey: restart zookeeper on conf200[2,3] for jvm security updates
13:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1084 original weight - T168661 (duration: 00m 47s)
13:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1084 weight - T168661 (duration: 00m 47s)
13:07 elukey: restart zookeeper on conf2001 for security updates (canary node)
13:00 moritzm: installing augeas security updates
12:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1084 with low weight - T168661 (duration: 00m 47s)
12:14 marostegui: Upgrade MariaDB on db1084 - T168661
12:11 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1084 - T168661 (duration: 00m 47s)
12:05 akosiaris: reimage chlorine and kubernetes1004 T170119
12:04 akosiaris: reimage chlorine and kubernetes1004 T10119
11:58 moritzm: restarting nginx on dataset1001/ms1001 to pick up libxml security update
11:21 elukey: restart apache2 on bohrium for libxml + gnutls security updates
11:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1056 original weight - T168661 (duration: 00m 47s)
11:17 moritzm: restarting apache on dbmonitor* to pick up libxml security update
10:27 moritzm: restarting nginx on prometheus* to pick up libxml security update
10:22 moritzm: restarting apache on einsteinium/tegmen (Icinga hosts) to pick up libxml security update
10:14 moritzm: restarting nginx on meitnerium/archiva.wikimedia.org to pick up libxml security update
10:13 moritzm: restarting nginx on debug proxies to pick up libxml security update
10:10 moritzm: restarting apache on hafnium to pick up libxml security update
09:57 addshore: extensions/Cognate/maintenance/recalculateCognateNormalizedHashes.php run done, 6326 hashes recalculated, T172987
09:50 addshore: addshore@terbium:~$ mwscript extensions/Cognate/maintenance/recalculateCognateNormalizedHashes.php --wiki enwiktionary --batch-size 1000000 # Should upsert 6326 rows T172987
09:44 addshore: addshore@terbium:~$ mwscript extensions/Cognate/maintenance/recalculateCognateNormalizedHashes.php --wiki enwiktionary --batch-size 1000000 --dry-run # T172987
09:43 moritzm: restarting apache on auth* to pick up libxml security update
09:40 moritzm: installing libxml2 security updates
09:10 moritzm: installing experimental hhvm-luasandbox package on mw1261 (canary app server) with backported patch for T171392 / T173705
09:04 hashar: contint1001 upgrading git, restarting git-daemon and zuul-merger - T161086
09:01 godog: test reimage of restbase2001 - T169939
08:57 hashar: contint2001 upgrading git, restarting git-daemon and zuul-merger - T161086
08:45 godog: remove /srv/deployment/grafana on naos and tin, unused - T170881
08:41 gehel: restart postgresql / kartotherian / tilerator / tileratorui on maps@eqiad
08:40 addshore@tin: Synchronized php-1.30.0-wmf.16/extensions/Cognate/maintenance/recalculateCognateNormalizedHashes.php: T172987 Add waitForReplication to RecalculateCognateNormalizedHashes script (duration: 00m 47s)
08:20 gehel: restart postgresql / kartotherian / tilerator / tileratorui on maps@codfw
08:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic on db1056 - T168661 (duration: 00m 47s)
08:01 marostegui: Rename reader_feedback, reader_feedback_history, reader_feedback_pages tables on dewiki on db1092 - T174586
07:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic on db1056 - T168661 (duration: 00m 47s)
06:45 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1056 with low weight - T168661 (duration: 00m 46s)
06:19 marostegui: Upgrade MariaDB to 10.0.32 on db1056 - T168661
06:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1056 - T168661 (duration: 00m 47s)
06:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1056 - T168661 (duration: 00m 47s)
02:55 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Aug 31 02:55:12 UTC 2017 (duration 7m 9s)
02:48 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 06m 40s)
02:30 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 10m 02s)

2017-08-30

23:10 demon@tin: Synchronized php-1.30.0-wmf.16/extensions/Newsletter/includes/specials/SpecialNewsletter.php: T174604 (duration: 00m 47s)
23:09 demon@tin: Synchronized php-1.30.0-wmf.15/extensions/Newsletter/includes/specials/SpecialNewsletter.php: T174604 (duration: 00m 48s)
23:05 demon@tin: Synchronized wmf-config/unitConversionConfig.json: swat (duration: 00m 47s)
21:05 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 back to 1.30.0-wmf.16
21:03 thcipriani@tin: Synchronized php-1.30.0-wmf.16/extensions/ProofreadPage/ProofreadPage.body.php: ProofreadPage: Avoids a stack overflow T173520 (duration: 00m 47s)
20:45 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 back to 1.30.0-wmf.15
20:22 ppchelko@tin: Finished deploy [changeprop/deploy@8d5fa29]: Start rejecting deduplicated messages (duration: 01m 05s)
20:21 ppchelko@tin: Started deploy [changeprop/deploy@8d5fa29]: Start rejecting deduplicated messages
20:07 urandom: T169939: Decommission Cassandra: restbase1009-c.eqiad.wmnet
19:09 ppchelko@tin: Finished deploy [changeprop/deploy@ff93ea8]: Don't deduplicate retry messages (duration: 01m 17s)
19:07 ppchelko@tin: Started deploy [changeprop/deploy@ff93ea8]: Don't deduplicate retry messages
19:06 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.16
19:04 demon@tin: Synchronized php: Symlink swap for group1/wmf.16 (duration: 00m 46s)
18:11 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/374399/4 (duration: 00m 47s)
17:29 urandom: T169939: Decommission Cassandra: restbase1009-b.eqiad.wmnet
17:27 hoo: Updated the Wikidata property suggester with data from Monday's JSON dump and applied the T132839 workarounds
17:12 ejegg: disabled creating CiviMail records for Thank You emails
16:49 XioNoX: rolling back vrrp priority change for the cr2<->asw-d-codfw link - T174366
16:34 XioNoX: start work on the cr2<->asw-d-codfw link - T174366
16:13 XioNoX: lowering vrrp priority for the cr2<->asw-d-codfw link - T174366
15:56 elukey: re-added analytics1055 among the hdfs/yarn worker after maintenance
15:49 bblack: cp1049 - restart varnish backend, mailbox lag
15:43 urandom: T169939: Decommission Cassandra: restbase1009-a.eqiad.wmnet
15:02 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1294.eqiad.wmnet
14:31 moritzm: restarting nginx on sodium to pick up libxml security update
14:07 elukey: restart java daemons on druid100[456] for jvm security updates
14:04 moritzm: restarting squid on install* servers to pick up libxml security update
14:04 moritzm: restarting squid on installurl downloaders to pick up libxml security update
14:02 moritzm: restarting apache on krypton to pick up libxml security update
13:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1053 original weight - T168661 (duration: 00m 46s)
13:55 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: pagePreviews: Scale A/B test bucket sizes by 10 - T172291 (duration: 00m 46s)
13:50 moritzm: restarting squid on url downloaders to pick up libxml security update
13:34 moritzm: installing libxml2 security updates
13:23 moritzm: installing ffmpeg security upadtes
13:21 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1053 weight - T168661 (duration: 00m 46s)
13:19 hashar: European SWAT completed
13:16 gehel: restarting wdqs-blazegraph and wdqs-updater on all wdqs nodes for logback config change - T172710
13:11 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable SandboxLink on cywiki - T173054 (duration: 00m 48s)
12:55 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1053 weight - T168661 (duration: 00m 45s)
12:51 gehel: restart wdqs-updater on wdqs2001 for config change
12:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1053 weight - T168661 (duration: 00m 47s)
12:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1053 with low weight - T168661 (duration: 00m 46s)
12:05 marostegui: Upgrade MariaDB to 10.0.32 on db1053 - T168661
12:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1053 - T168661 (duration: 00m 47s)
11:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1055 original weight - T174265 (duration: 00m 46s)
10:24 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give db1055 more traffic - T174265 (duration: 00m 47s)
09:49 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give db1055 more traffic - T174265 (duration: 00m 47s)
09:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1059 - T168661 (duration: 00m 47s)
09:07 elukey: restart all jvm daemons on druid100[123] for security updates
08:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give db1055 more traffic - T174265 (duration: 00m 48s)
08:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give db1055 more traffic - T174265 (duration: 00m 47s)
07:47 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1228.eqiad.wmnet
07:47 marostegui: Upgrade MariaDB on db1059 to 10.0.32 - T168661
07:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1059 for a MariaDB upgrade - T168661 (duration: 00m 53s)
07:14 moritzm: powering down mw1294 for hardware diagnostics (T1674069
07:12 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: mw1294.eqiad.wmnet
07:12 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Slowly repool db1055 - T174265 (duration: 00m 52s)
03:16 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Aug 30 03:16:09 UTC 2017 (duration 7m 19s)
03:08 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 15m 42s)
02:32 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 08m 28s)
02:17 demon@tin: Finished deploy [gerrit/gerrit@f33c63f]: Testing, no-op (duration: 00m 07s)
02:17 demon@tin: Started deploy [gerrit/gerrit@f33c63f]: Testing, no-op
01:27 demon@tin: Synchronized wmf-config/InitialiseSettings.php: remove education program from legalteamwiki (duration: 00m 47s)
00:53 demon@tin: Synchronized wmf-config/flaggedrevs.php: killing FR on testwiki (duration: 00m 46s)
00:52 demon@tin: Synchronized dblists/flaggedrevs.dblist: killing FR on testwiki (duration: 00m 47s)
00:05 cwd: turned dedupe off

2017-08-29

23:50 cmjohnson1: reinstalling mw1228
23:34 demon@tin: Synchronized php-1.30.0-wmf.15/extensions/Popups/: swat (duration: 00m 47s)
23:33 demon@tin: Synchronized php-1.30.0-wmf.16/extensions/Popups/: swat (duration: 00m 50s)
21:33 urandom: T169939: Decommission Cassandra: restbase1008-c.eqiad.wmnet
21:16 andrewbogott: restarting designate-sink on labservices1001
21:01 ejegg: updated DjangoBannerStats from 5963e7c to ce95eab
20:56 ejegg: updated payments-wiki from 58d33b8 to bf97cd2
20:49 godog: bounce varnish on cp1074 / cp1099 / cp1072 - mailbox lag
20:38 ppchelko@tin: Finished deploy [changeprop/deploy@a57c79d]: Release a redis-based deduplicator in test mode. Attempt 2 (duration: 01m 17s)
20:37 ppchelko@tin: Started deploy [changeprop/deploy@a57c79d]: Release a redis-based deduplicator in test mode. Attempt 2
20:36 ppchelko@tin: Finished deploy [changeprop/deploy@ed0fadc]: Release a redis-based deduplicator in test mode (duration: 04m 28s)
20:31 ppchelko@tin: Started deploy [changeprop/deploy@ed0fadc]: Release a redis-based deduplicator in test mode
19:47 urandom: T169939: Decommission Cassandra: restbase1008-b.eqiad.wmnet
19:34 ottomata: restarting analytics kafka brokers and all mirror maker processes to apply https://gerrit.wikimedia.org/r/#/c/374610/
19:12 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.16
19:08 ppchelko@tin: Finished deploy [restbase/deploy@7f2e55f]: Update CXServer endpoints config (duration: 06m 48s)
19:02 ottomata: restarting main-eqiad -> analytics kafka mirror maker processes on analytics kafka brokers, something is not working...
19:01 XenoRyet: updated Smashpig from 98c5516 to 8f4388d
19:01 ppchelko@tin: Started deploy [restbase/deploy@7f2e55f]: Update CXServer endpoints config
18:19 robh: attempting firmware upgrade on scs-a8-eqiad
18:17 Pchelolo: depooling restbase1008,1009,1010,2003,2005 while cluster reshaping is going on
18:15 demon@tin: Finished scap: bootstrap wmf.16 (duration: 45m 27s)
18:15 urandom: T169939: Decommission Cassandra: restbase1008-a.eqiad.wmnet
17:30 demon@tin: Started scap: bootstrap wmf.16
17:23 ejegg: restarted donation queue consumer and thank you mailer
17:20 Reedy: Disabled oathauth for KartikMistry on wikitech
16:33 ejegg: updated civicrm from 7dfea0d to f00611d
16:32 ejegg: turned off donation queue consumer and thank you mailer
16:04 ejegg: restarted ingenico recurring charge job
15:54 urandom: T169939: Decommission Cassandra: restbase2005-c.codfw.wmnet
14:51 elukey: drop log.MobileWebUIClickTracking_10742159_15423246 from db1047 (archived on HDFS) - T172322
14:33 marostegui: Shutdown db1055 to replace its BBU - T174265
13:12 zeljkof: EU SWAT finished
13:10 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add several HD logos (T150618) (duration: 00m 43s)
13:09 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Add several HD logos (T150618) (duration: 00m 43s)
12:37 elukey: restart kafka daemons on kafka1014 for jvm security updates
12:31 godog: bounce pybal on lvs1003 to pick up config changes
11:49 elukey: restart kafka daemons on kafka1013 for jvm security updates
11:43 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1168.eqiad.wmnet
11:30 elukey: restart java daemons on analytics100[1,2] (Hadoop Master nodes) for jvm updates
11:08 marostegui: Stop MariaDB on db1097 to migrate it to file per table - T161088
10:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1097 - T161088 (duration: 00m 43s)
10:48 moritzm: installing libxml2 security updates
10:31 moritzm: reimaging mw1168 (video scaler) to jessie
10:16 jynus@tin: Synchronized wmf-config/db-eqiad.php: decom db1028 (duration: 00m 42s)
10:15 jynus@tin: Synchronized wmf-config/db-codfw.php: decom db1028 (duration: 02m 48s)
09:59 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1064 - T168661 (duration: 00m 43s)
09:39 marostegui: Update MariaDB on db1064 to 10.0.32 - T168661
09:38 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1064 for a mariadb upgrade - T168661 (duration: 00m 43s)
09:33 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1169.eqiad.wmnet
09:29 elukey: re-installed pmacct/librdkafka1/kafkacat on rhenium with stretch versions - T173489
09:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1091 original weight - T168661 (duration: 00m 43s)
08:53 ema: upgrading cache_text to varnish 4.1.8-1wm1
08:45 akosiaris: reprepro copy calico, calico-cni from jessie-wikimedia to stretch-wikimedia (apt.wikimedia.org) T170119
08:43 hashar: Restarting Jenkins for openjdk update
08:42 elukey: restart yarn/hdfs daemons for openjdk security updates
08:42 akosiaris: upload kubernetes_1.7.4-1 to apt.wikimedia.org/stretch-wikimedia/main T170119
08:42 moritzm: restarting archiva to pick up openjdk security update
08:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1091 weight - T168661 (duration: 00m 43s)
08:35 gehel: restart wdqs-updater on wdqs2001
08:32 hashar: apt-get upgrade on contint1001 and contint2001
08:20 moritzm: reimaging mw1169 (video scaler) to jessie
08:09 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1091 with low weight - T168661 (duration: 00m 43s)
08:06 elukey: drop log.MobileWebUIClickTracking_10742159_15423246 from dbstore1002 to free space (table archived on HDFS) - T172322 T168303
07:41 marostegui: Upgrade MariaDB on db1091 to 10.0.32 - T168661
07:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1097 - T168661 (duration: 00m 42s)
06:56 moritzm: installing ghostscript security updates on trusty (Debian already fixed)
06:09 demon@tin: Synchronized scap/plugins/clean.py: no-op, for consistency (duration: 00m 43s)
06:05 marostegui: Restart MariaDB on db1102 and db1095 to pick up new replication filters - T174385
05:49 demon@tin: Pruned MediaWiki: 1.30.0-wmf.11 (duration: 02m 50s)
02:34 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Aug 29 02:34:09 UTC 2017 (duration 6m 44s)
02:27 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 08m 00s)
00:25 urandom: T169939: Decommissioning restbase1010-a.eqiad.wmnet

2017-08-28

23:37 reedy@tin: Synchronized php-1.30.0-wmf.15/extensions/Score/: consistency (duration: 00m 43s)
23:31 reedy@tin: Synchronized php-1.30.0-wmf.15/extensions/Score/Score.body.php: output (duration: 00m 43s)
23:27 reedy@tin: Synchronized php-1.30.0-wmf.15/extensions/Score/Score.body.php: output (duration: 00m 42s)
23:23 reedy@tin: Synchronized wmf-config/CommonSettings.php: abc2ly in firejail (duration: 00m 43s)
23:22 reedy@tin: Synchronized php-1.30.0-wmf.15/extensions/Score/Score.body.php: Fix escaping (duration: 00m 43s)
23:13 cwd: re-enabled dedupe jobs
23:13 ebernhardson@tin: Synchronized php-1.30.0-wmf.15/extensions/WikimediaEvents/: Turn off two cirrus AB tests T171742 T171214 (duration: 00m 44s)
23:08 ebernhardson@tin: Synchronized docroot/noc/conf/categories-rdf.dblist: T173892: Add list for wikis that would have categories dumped into RDF (duration: 00m 43s)
23:07 ebernhardson@tin: Synchronized dblists/categories-rdf.dblist: T173892: Add list for wikis that would have categories dumped into RDF (duration: 00m 43s)
22:56 ejegg: turned fundraising orphan rectifier back on
22:46 urandom: T169939: Decommissioning restbase1010-b.eqiad.wmnet
21:48 krinkle@tin: Synchronized wmf-config/InitialiseSettings.php: I1c9c9c0e - Enable jQuery 3 on nlwiki, svwiki, plwiki (duration: 00m 44s)
21:37 reedy@tin: Synchronized wmf-config/CommonSettings.php: Disable wgScoreAbc2Ly firejail T172582 (duration: 00m 44s)
21:06 ejegg: switched nightly Ingenico audit from wr1 to wx file parsing
21:06 reedy@tin: Synchronized wmf-config/CommonSettings.php: Run Score binaries in firejail T172582 (duration: 00m 43s)
21:01 reedy@tin: Synchronized wmf-config/CommonSettings.php: Move email blacklist to meta:Email_blacklist (duration: 00m 45s)
20:53 XioNoX: pushing "aggregate defaults discard" to all the cr* routers - T174364
20:52 urandom: T169939: Decommissioning restbase1010-c.eqiad.wmnet
20:16 ayounsi@tin: Finished deploy [librenms/librenms@5fea59c]: (no justification provided) (duration: 00m 05s)
20:16 ayounsi@tin: Started deploy [librenms/librenms@5fea59c]: (no justification provided)
20:16 XioNoX: upgrading librenms to 1.31.01
20:08 bd808@tin: Finished deploy [striker/deploy@47b5e8a]: Deploying 47b5e8a (various bug fixes) (duration: 00m 26s)
20:07 bd808@tin: Started deploy [striker/deploy@47b5e8a]: Deploying 47b5e8a (various bug fixes)
19:24 ejegg: updated CiviCRM from 200f34b to 7dfea0d
19:24 ottomata: restarting eventbus in eqiad to apply https://gerrit.wikimedia.org/r/#/c/372179/
19:23 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Grant arbcomers abusefilter-view-private and abusefilter-log-private at cswiki T174357; Enable popups on en and de wiki for A/B test T172291 (duration: 00m 43s)
19:15 herron: scb1001: restarted pdfrender service
19:14 ottomata: restarting eventbus in codfw to verify https://gerrit.wikimedia.org/r/#/c/372179/
19:09 ottomata: rolling restart and rebalances of main-* kafka clusters for https://gerrit.wikimedia.org/r/#/c/372179/
19:03 niharika29@tin: Synchronized php-1.30.0-wmf.15/extensions/CodeMirror/: Fix exception on some combination of quotes T174060 (duration: 00m 44s)
18:48 gwicke: restarted pdfrender instances in eqiad (T159922)
18:48 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: pagePreviews: remove invalidated popup sampling rate variables https://gerrit.wikimedia.org/r/#/c/373171/7 (duration: 00m 43s)
18:47 ayounsi@tin: Finished deploy [librenms/librenms@8c9da11]: (no justification provided) (duration: 00m 02s)
18:47 ayounsi@tin: Started deploy [librenms/librenms@8c9da11]: (no justification provided)
18:43 aaron@tin: Synchronized php-1.30.0-wmf.15/includes/jobqueue/jobs/HTMLCacheUpdateJob.php: 2d83569397 - Fix old regression in HTMLCacheUpdate de-duplication (duration: 00m 44s)
18:37 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Fix incorrect Special:UserLogin in Popups blacklist https://gerrit.wikimedia.org/r/#/c/373696/ (duration: 00m 43s)
18:33 XioNoX: upgrading librenms to 1.31
18:32 ayounsi@tin: Finished deploy [librenms/librenms@8c9da11]: (no justification provided) (duration: 00m 05s)
18:32 ayounsi@tin: Started deploy [librenms/librenms@8c9da11]: (no justification provided)
18:30 ejegg: updated CiviCRM from c30ec13 to 200f34b
18:21 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable wgEchoPerUserBlacklist at all wikis T173838 (duration: 00m 43s)
18:15 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Upload wikipedia wordmark zh https://gerrit.wikimedia.org/r/#/c/373948 (duration: 00m 45s)
18:14 niharika29@tin: Synchronized static/images/mobile/copyright/: Upload wikipedia wordmark zh https://gerrit.wikimedia.org/r/#/c/373948 (duration: 00m 44s)
17:59 XioNoX: pushing "aggregate defaults discard" to *ams - T174364
17:56 urandom: T169939: Decommission Cassandra: restbase2005-b.codfw.wmnet
17:31 gehel: restarting wdqs-blazegraph and wdqs-updater for config change
17:19 gehel: restarting wdqs-blazegraph and wdqs-updater for config change
17:11 XioNoX: pushing "aggregate defaults discard" to cr2-knams - T174364
17:09 gehel@tin: Finished deploy [wdqs/wdqs@90f4e2d]: (no justification provided) (duration: 02m 27s)
17:07 gehel@tin: Started deploy [wdqs/wdqs@90f4e2d]: (no justification provided)
17:07 gehel@tin: Started deploy [wdqs/wdqs@90f4e2d]: (no justification provided)
16:46 demon@tin: Pruned MediaWiki: 1.30.0-wmf.10 (duration: 02m 41s)
16:39 demon@tin: Pruned MediaWiki: 1.30.0-wmf.14 [keeping static files] (duration: 02m 00s)
14:39 elukey: restart eventlogging_sync on dbstore1002 - issue after drop of old table
14:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1099 with normal weight on s5 - T172679 (duration: 00m 44s)
14:31 elukey: drop PageContentSaveComplete_5588433_15423246 from the log database on db1046 (m4-master) - T170720
14:28 zeljkof: EU SWAT finished
14:28 zfilipin@tin: Synchronized php-1.30.0-wmf.15/includes/jobqueue/jobs/HTMLCacheUpdateJob.php: SWAT: Disable rebound CDN purges for backlinks in HTMLCacheUpdateJob (T173710) (duration: 00m 45s)
14:01 zeljkof: extending EU SWAT for 10 or so minutes to deploy two more patches
13:55 elukey: restart kafka* daemons on kafka1012 for openjdk security updates (canary)
13:52 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Log WikibaseQualityConstraints (T171281) (duration: 00m 44s)
13:42 zfilipin@tin: Synchronized wmf-config/throttle-analyze.php: SWAT: Automatically include commons and wikidata in $wmgThrottlingExceptions (T163872) (duration: 00m 44s)
13:36 zfilipin@tin: Synchronized wmf-config/CommonSettings.php: SWAT: throttle.php: Separate the throttling definitions from the exception values itself (T167040) (duration: 00m 44s)
13:34 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: throttle.php: Separate the throttling definitions from the exception values itself (T167040) (duration: 00m 44s)
13:33 zfilipin@tin: Synchronized wmf-config/throttle-analyze.php: SWAT: throttle.php: Separate the throttling definitions from the exception values itself (T167040) (duration: 00m 44s)
13:27 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Allow sysops to grant/remove transwiki user group in dtywiki (T174226) (duration: 00m 44s)
13:21 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add 4 URLs to $wgCopyUploadsDomain whitelist (T174152) (duration: 00m 45s)
13:09 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Remove non-transparent background from dty.wiki logos (T174098) (duration: 00m 45s)
12:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add more weight to db1099 on s5 - T172679 (duration: 00m 44s)
12:43 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: mw1259.eqiad.wmnet
12:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add more weight to db1099 on s5 - T172679 (duration: 00m 44s)
12:07 gehel: restarting nginx for upgrade on wdqs*
12:05 gehel: restarting nginx for upgrade on elastic* / relforge*
12:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1099 pooled with low weight to s5 - T172679 (duration: 00m 45s)
11:05 moritzm: installing libxml2 security updates
08:41 moritzm: restarting squid3 on install1002
08:16 marostegui: Ugprade MariaDB on s4 codfw master - db2051 to 10.0.32 - T168661
08:10 marostegui: Stop MySQL on db1045 - T172679
08:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1045 to clone db1099 from it - T172679 (duration: 00m 46s)
07:51 moritzm: reimaging mw1259 to jessie (T145742)
07:22 marostegui: Force re-learn cycle on db1055 - https://phabricator.wikimedia.org/T174265
07:12 moritzm: installing openjdk security updates on notebook* hosts
07:05 moritzm: installing openjdk security updates on meitnerium
02:36 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Aug 28 02:36:23 UTC 2017 (duration 6m 54s)
02:29 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 09m 36s)

2017-08-27

20:35 ariel@tin: Finished deploy [dumps/dumps@39f9b52]: write output files to temp location and move into place when complete (duration: 00m 02s)
20:34 ariel@tin: Started deploy [dumps/dumps@39f9b52]: write output files to temp location and move into place when complete
15:46 akosiaris: upload kubernetes_1.4.6-7 to apt.wikimedia.org/jessie-wikimedia/main T170346
05:30 marostegui: Force BBU relearn on db1055 - T174265
00:45 jynus: correction last log s/db1051/db1055/
00:42 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1051, hw issues, may get lag (duration: 00m 44s)

2017-08-26

15:46 godog: bounce varnish on cp1073 - mailbox lag

2017-08-25

19:09 gehel: actually not killing the "stuck discovery report-updater process on stat1005", it is already gone - T174110
19:07 gehel: kill stuck discovery report-updater process on stat1005 - T174110
16:03 moritzm: created cn=ciadmin group in LDAP to be used for privileged Jenkins access, initial members in ticket (T169557)
14:29 godog: bounce varnish on cp1049 / cp1072 - mailbox lag
14:16 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db1097 - T168661 (duration: 00m 44s)
13:53 marostegui: Upgrade MariaDB on db1097 to 10.0.32 - T168661
13:53 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db1097 - T168661 (duration: 00m 44s)
13:40 marostegui: Upgrade MariaDB to 10.0.32 on db2019 - T168661
13:37 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2037 - T168661 (duration: 00m 45s)
13:24 marostegui: Upgrade MariaDB on db2037 to 10.0.32 - T168661
13:02 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2037 - T168661 (duration: 00m 44s)
12:51 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2044 - T168661 (duration: 00m 44s)
12:29 marostegui: Upgrade MariaDB on db2044 - T168661
12:29 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2044 - T168661 (duration: 00m 43s)
12:19 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2058 - T168661 (duration: 00m 48s)
11:48 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1028 (duration: 00m 47s)
09:19 moritzm: restart thumbor to pick up libxml2 security updates
09:12 godog: reimage restbase2001 to test new partman recipe - T169939
08:41 moritzm: restarting apache/hhvm on canary app servers to pick up libxml security update
08:00 volans: upgrading cumin to v1.0.0 on sarin/neodymium
07:55 gehel: reimaging logstash1006 after change of failed disk - T173679
07:49 jynus@tin: Synchronized wmf-config/db-eqiad.php: pool db1069 with more weight (duration: 00m 44s)
07:00 jynus@tin: Synchronized wmf-config/db-eqiad.php: decom db1033, pool db1069 with low weight (duration: 00m 44s)
06:55 jynus@tin: Synchronized wmf-config/db-codfw.php: decom db1033 (duration: 00m 43s)
06:30 jynus@tin: Synchronized wmf-config/db-eqiad.php: formatting fixes (duration: 00m 44s)
06:11 jynus@tin: Synchronized wmf-config/db-codfw.php: repool es2013, formatting fixes (duration: 00m 44s)
06:01 marostegui: Upgrade MariaDB to 10.0.32 on db2058 - T168661
06:00 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2058 - T168661 (duration: 00m 43s)
05:30 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2065 - T168661 (duration: 00m 44s)
05:18 moritzm: re-enable mw1260 (video scaler) for active job processing
05:16 marostegui: Reboot db2065 to pick up new kernel
05:08 marostegui: Upgrade MariaDB on db2065 - T168661
05:03 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2065 for a MariaDB upgrade - T168661 (duration: 00m 44s)
01:12 mattflaschen@tin: Synchronized php-1.30.0-wmf.15/extensions/Flow/modules/engine/components/board/features/flow-board-loadmore.js: T173807: Fix Flow infinite scroll (duration: 00m 45s)

2017-08-24

23:17 thcipriani@tin: Synchronized php-1.30.0-wmf.15/extensions/TimedMediaHandler/TimedMediaHandler.php: SWAT: Disable Ogg Theora video transcodes in default config T172445 (duration: 00m 45s)
22:45 demon@tin: Pruned MediaWiki: 1.30.0-wmf.13 [keeping static files] (duration: 02m 01s)
22:24 ejegg: updated civicrm from fa882f2 to c30ec13 (pt-br and ja thank you letters)
21:59 gwicke: restarted pdfrender service instances in eqiad / T159922
21:54 maxsem@tin: Synchronized php-1.30.0-wmf.15/extensions/LoginNotify/: https://gerrit.wikimedia.org/r/#/c/373701/1 (duration: 00m 45s)
21:13 maxsem@tin: Synchronized php-1.30.0-wmf.15/extensions/LoginNotify/: Logging: https://gerrit.wikimedia.org/r/#/c/373691/ (duration: 00m 44s)
20:07 andrewbogott: stopping, starting, restarting etc. nodepool
19:58 urandom: T169939: Decommission Cassandra: restbase2003-c.codfw.wmnet
19:24 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.30.0-wmf.15
19:14 andrewbogott: restarting rabbitmq-server on labcontrol1001
18:38 ejegg: updated payments-wiki from bd9f730 to 58d33b8
18:38 bblack: varnish backend restart - cp1074 - mailbox lag
18:24 urandom: T169939: Decommission Cassandra: restbase2003-b.codfw.wmnet
17:33 arlolra@tin: Finished deploy [parsoid/deploy@bd12f8a]: Updating Parsoid to 538dad7f (duration: 12m 24s)
17:21 arlolra@tin: Started deploy [parsoid/deploy@bd12f8a]: Updating Parsoid to 538dad7f
17:01 jynus: stopping and cloning db1033 to db1069
16:52 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give normal weight to db1096 - T172679 (duration: 00m 47s)
16:43 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give some more weight to db1096 - T172679 (duration: 00m 47s)
16:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give some more weight to db1096 - T172679 (duration: 00m 47s)
16:34 urandom: T169939: Decommission Cassandra: restbase2003-a.codfw.wmnet
16:24 andrewbogott: apt-get upgrade and updated to MW 1.29.1 on wikitech-static. Rebooting to pick up kernel updates.
15:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give some weight to db1096 - T172679 (duration: 00m 47s)
15:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1096 depooled to s5 - T172679 (duration: 00m 47s)
14:44 ladsgroup@tin: Synchronized php-1.30.0-wmf.15/extensions/Wikidata/extensions/Wikibase/client/includes/Changes/WikiPageUpdater.php: Reduce batch size in WikiPageUpdater (T173710) (duration: 00m 48s)
14:14 phuedx: refreshLinks on bawiki (T173994)
14:13 volans: updated cumin package in our APT to version cumin_1.0.0-1
13:19 phuedx@tin: Synchronized wmf-config/InitialiseSettings-labs.php: T171853: Re-enable Page Previews for enwiki and dewiki on the Beta Cluster (duration: 00m 47s)
13:13 phuedx@tin: Synchronized portals: (no justification provided) (duration: 00m 49s)
13:12 phuedx@tin: Synchronized portals/prod/wikipedia.org/assets: (no justification provided) (duration: 00m 49s)
12:45 gehel: killing discovery report updater on stats1005 (stuck since Aug 15) - T173333
12:30 marostegui: Stop MySQL on db1026 to clone db1096 from it - T172679
12:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1026 to clone db1096 from it T172679 (duration: 00m 47s)
11:36 jynus: disable puppet on db1069 in preparation for reimage
11:08 marostegui: Global rename Papa1234 → Karl-Heinz Jansen - T173859
09:56 jynus@tin: Synchronized wmf-config/db-eqiad.php: repool db1078 with full weight (duration: 00m 46s)
09:33 volans: executing https://wikitech.wikimedia.org/wiki/Cumin#Run_Puppet_only_if_last_run_failed to get rid of failed ones
08:54 jynus: 16 minute test run of rebuildTermSqlIndex.php on terbium
08:52 marostegui: Drop tables article_assessment, article_assessment_pages, article_assessment_ratings tables from testwiki - T173590
07:24 Amir1: starting the run for rebuildTermIndex (T171460)
02:50 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Aug 24 02:50:40 UTC 2017 (duration 7m 4s)
02:43 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 05m 56s)
02:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 09m 50s)
01:46 aaron@tin: Synchronized php-1.30.0-wmf.15/includes/jobqueue: Deploy 752580637 (T173710) (duration: 00m 49s)

2017-08-23

23:49 maxsem@tin: Synchronized php-1.30.0-wmf.14/extensions/LoginNotify/: SWAT fixes (duration: 00m 47s)
23:47 maxsem@tin: Synchronized php-1.30.0-wmf.15/extensions/LoginNotify/: SWAT fixes (duration: 00m 47s)
23:42 maxsem@tin: Synchronized php-1.30.0-wmf.14/extensions/CodeMirror/: https://gerrit.wikimedia.org/r/#/c/373324/ (duration: 00m 47s)
23:41 maxsem@tin: Synchronized php-1.30.0-wmf.15/extensions/CodeMirror/: https://gerrit.wikimedia.org/r/#/c/373324/ (duration: 00m 49s)
23:28 maxsem@tin: Synchronized php-1.30.0-wmf.15/resources/: https://gerrit.wikimedia.org/r/#/c/373391/ (duration: 00m 48s)
23:27 maxsem@tin: Synchronized wmf-config/MetaContactPages.php: https://gerrit.wikimedia.org/r/#/c/373369/ (duration: 00m 48s)
20:59 urandom: T169939: Truncating MCS tables
20:26 thcipriani@tin: Synchronized php: group1 wikis to 1.30.0-wmf.15 (duration: 00m 46s)
20:25 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.30.0-wmf.15
20:13 jynus@tin: Synchronized wmf-config/db-eqiad.php: repool db1078 with low weight (duration: 00m 47s)
20:12 thcipriani@tin: Finished scap: Revert ProofreadPage to db7507246665e69384c1d92af2aedc62263a5116 for wmf.15 T173520 (duration: 04m 11s)
20:08 thcipriani@tin: Started scap: Revert ProofreadPage to db7507246665e69384c1d92af2aedc62263a5116 for wmf.15 T173520
19:49 thcipriani@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Beta-only change: pagePreviews: Enable A/B test (BC-only (duration: 00m 47s)
19:45 joal@tin: Finished deploy [analytics/refinery@f467ce1]: Bug fixing deploy (duration: 03m 16s)
19:44 thcipriani@tin: Synchronized php-1.30.0-wmf.14/extensions/MobileFrontend: Verify the existence of `url` key when parsing lang objects T172316 (duration: 00m 56s)
19:42 joal@tin: Started deploy [analytics/refinery@f467ce1]: Bug fixing deploy
19:36 thcipriani@tin: Synchronized php-1.30.0-wmf.15/extensions/TimedMediaHandler: Enable WebM playback via ogv.js T172444 (duration: 00m 50s)
19:22 kaldari: foreachwiki sql.php /srv/mediawiki/php/maintenance/archives/patch-ip_changes.sql
19:19 kaldari: mwscript sql.php --wiki=testwiki /srv/mediawiki/php/maintenance/archives/patch-ip_changes.sql
19:15 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Remove CitThisPage from blacklist for page previews T173865 (duration: 00m 47s)
19:13 niharika29@tin: Synchronized php-1.30.0-wmf.15/extensions/TimedMediaHandler/: Enable WebM playback via ogv.js https://gerrit.wikimedia.org/r/#/c/373126/ (duration: 00m 49s)
19:12 niharika29@tin: Synchronized php-1.30.0-wmf.14/extensions/TimedMediaHandler/: Enable WebM playback via ogv.js https://gerrit.wikimedia.org/r/#/c/373126/ (duration: 00m 50s)
19:00 niharika29@tin: Synchronized php-1.30.0-wmf.14/extensions/MobileFrontend/: Verify the existence of key when parsing lang objects https://gerrit.wikimedia.org/r/#/c/373292/ (duration: 00m 49s)
18:57 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Log channel for LoginNotify T173888 (duration: 00m 47s)
18:55 ppchelko@tin: Finished deploy [changeprop/deploy@1998c10]: [Config] Disable mobile rerenders for non-wikipedia domains (duration: 01m 33s)
18:54 ppchelko@tin: Started deploy [changeprop/deploy@1998c10]: [Config] Disable mobile rerenders for non-wikipedia domains
18:52 niharika29@tin: Synchronized php-1.30.0-wmf.15/extensions/LoginNotify/: Log everything T173888 (duration: 00m 48s)
18:43 gehel: manually running report updater for discovery golden data on stat1005 - T173333
18:40 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Reinforce LoginNotify settings https://gerrit.wikimedia.org/r/#/c/372555/ (duration: 00m 47s)
18:40 gehel: resetting permissions on stat1005:/srv/published-datasets/discovery - T173333
18:39 niharika29@tin: Synchronized wmf-config/CommonSettings.php: Reinforce LoginNotify settings https://gerrit.wikimedia.org/r/#/c/372555/ (duration: 00m 47s)
18:32 thcipriani: jenkins restarted, zuul should pick-up queue
18:26 thcipriani: jenkins appears to be having some issues, trying to dump threads now and then restart to see if it helps
18:13 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Allow crats to add people to accountcreator group on mw.or https://gerrit.wikimedia.org/r/#/c/373317/ (duration: 00m 49s)
17:40 bblack: reboot cp4021
16:38 cwd: disabled all dedupe
16:26 demon@tin: Pruned MediaWiki: 1.30.0-wmf.11 [keeping static files] (duration: 01m 32s)
16:24 bd808@tin: Finished deploy [striker/deploy@2f2dd7c]: Deploying 2f2dd7c "Tool account creation and more" (T128400, T149458, T159044, T164847, T167931, T168480, T173845) (duration: 00m 39s)
16:23 bd808@tin: Started deploy [striker/deploy@2f2dd7c]: Deploying 2f2dd7c "Tool account creation and more" (T128400, T149458, T159044, T164847, T167931, T168480, T173845)
16:22 demon@tin: Pruned MediaWiki: 1.30.0-wmf.9 (duration: 03m 44s)
15:35 legoktm: added addshore to "integration" gerrit group (T173233)
15:10 gehel: shutdown logstash1006 for disk replacement - T173679
14:49 moritzm: installing texlive security updates on trusty (Debian already fixed)
14:36 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4027.ulsfo.wmnet
14:34 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4027.ulsfo.wmnet
13:31 marostegui: Stop MySQL on db1041 to get it ready for decommission - T173915
13:13 zeljkof: EU SWAT finished
13:10 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Flow as a Beta feature on wawiki and wawikionary (T172947) (duration: 00m 48s)
12:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1041 to decommission it - T173915 (duration: 00m 48s)
12:15 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1041 to decommission it - T173915 (duration: 00m 48s)
10:39 jynus: stopping and deleting s1 and s4 from dbstore2001
10:06 joal@tin: Finished deploy [analytics/refinery@84d6ee4]: Regular deploy (1 month since last) (duration: 03m 30s)
10:03 moritzm: upgrading remaining image scalers to luasandbox 2.0.13
10:02 joal@tin: Started deploy [analytics/refinery@84d6ee4]: Regular deploy (1 month since last)
09:36 godog: kill older and running tasks from gerrit queue, it was stuck
09:10 moritzm: upgrading remaining API servers in to luasandbox 2.0.13
09:08 godog: upload prometheus-blackbox-exporter 0.7.0+ds1-1~wmf1 to jessie-wikimedia, backported - T169860
08:35 marostegui: Stop Upgrade MySQL on db2073 to 10.0.32 - T168661
08:22 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2073 - T168661 (duration: 00m 59s)
08:18 moritzm: upgrading remaining job runners to luasandbox 2.0.13
07:12 marostegui: Global rename: Opdire657 → Sakiv - T173834
06:55 moritzm: upgrading remaining app servers in to luasandbox 2.0.13
03:15 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Aug 23 03:15:36 UTC 2017 (duration 7m 6s)
03:08 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 15m 11s)
02:32 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 08m 03s)

2017-08-22

23:39 thcipriani@tin: Synchronized php-1.30.0-wmf.15/extensions/Linter/includes/LintErrorsPager.php: SWAT: Fix up 11f4a97ba6bcd0c1de (duration: 00m 49s)
23:35 thcipriani@tin: Finished scap: SWAT: Fix typo in the notification message (duration: 21m 48s)
23:13 thcipriani@tin: Started scap: SWAT: Fix typo in the notification message
23:10 thcipriani@tin: Synchronized php-1.30.0-wmf.14/extensions/Popups/includes/PopupsHooks.php: SWAT: Remove aborting of BeforePageDisplay hook T173411 (duration: 00m 49s)
21:18 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Retry CodeMirror deployment T170966 (duration: 00m 49s)
20:41 ejegg: updated CiviCRM from 4aa177b to fa882f2
20:25 mutante: restbase-dev* - puppet runs fail due to E: Version '3.11.0' for 'cassandra' was not found
20:00 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.30.0-wmf.15
19:56 urandom: starting cassandra restbase2004-a and restbase2006-c, OOMs
19:46 thcipriani@tin: Finished scap: testwiki to php-1.30.0-wmf.15 and rebuild l10n cache (duration: 45m 55s)
19:00 thcipriani@tin: Started scap: testwiki to php-1.30.0-wmf.15 and rebuild l10n cache
18:50 robh: labservies1002 update completed (not 1003, typo)
18:50 robh: labservies1003 update completed
18:40 robh: firmware update of labservices1002 in progress
17:54 andrewbogott: removing obsolete apache2 and puppetmaster packages from labcontrol boxes for https://phabricator.wikimedia.org/T171786
17:07 thcipriani: starting branch cut for 1.30.0-wmf.15
13:58 godog: bounce varnish on cp1074 / cp1049 / cp1073 - mailbox problems
13:48 zeljkof: EU SWAT finished
13:45 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: relatedArticles: Tidy up config (T165991) (duration: 00m 44s)
13:44 zfilipin@tin: Synchronized wmf-config/CommonSettings.php: SWAT: relatedArticles: Tidy up config (T165991) (duration: 00m 44s)
13:29 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wmgUseWikimediaShopLink to true for ptwiki (T173768) (duration: 00m 45s)
10:41 moritzm: upgrading hhvm-luasandbox on mw1293-mw1295 (image scalers)
10:21 Amir1: another run of rebuildTermSqlIndex (T171460)
09:54 moritzm: upgrading hhvm-luasandbox on mw1189-mw1208 (API servers)
09:42 marostegui: Renaming user Darwinius → DarwIn - T173159
09:35 moritzm: upgrading hhvm-luasandbox on mw1180-1188 and mw1209-mw1220 (app servers)
09:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1064 - T172996 (duration: 00m 44s)
08:59 akosiaris: upload apertium-bel-rus_0.2.0~r81186-1+wmf to apt.wikimedia.org/jessie-wikimedia/main
08:55 moritzm: upgrading hhvm-luasandbox on mw1161-mw1167 (job runners)
08:43 akosiaris: upload apertium-rus_0.1.0~r81184-1+wmf to apt.wikimedia.org/jessie-wikimedia/main
08:43 akosiaris: upload apertium-bel_0.1.0~r81357-1+wmf to apt.wikimedia.org/jessie-wikimedia/main
08:35 godog: bounce varnish on cp1072 - mailbox problems
08:33 godog: bounce varnish on cp4015 - mailbox problems
08:24 marostegui: Drop s4 from db1095 with NO replication - T172996
08:23 moritzm: upgrading hhvm-luasandbox on deployment servers / script runners
07:34 marostegui: Stop replication on db1064 sanitarium2 and sanitarium3 master to move labsdb1009,10 and 11 s4 from db1095 to db1102 - https://phabricator.wikimedia.org/T172996
07:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1064 - T172996 (duration: 00m 52s)
07:28 moritzm: installing ruby security updates on trusty (Debian already fixed)
07:11 moritzm: installing c-ares security updates on trusty (Debian already fixed)
06:54 moritzm: installing graphite2 (the image library, not the metrics tool) security updates on trusty (Debian already fixed)
02:30 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Aug 22 02:30:47 UTC 2017 (duration 6m 55s)
02:23 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 07m 11s)

2017-08-21

22:50 robh: rebooting tin for firmware update since its idle
22:20 mutante: praseodymium - installing BIOS upgrade, reboot
22:11 mutante: xenon - installing BIOS upgrade
21:58 mutante: cerium (cassandra test) - rebooting for firmware upgrade
21:54 mutante: cerium - installing Dell BIOS upgrade (T162850)
21:26 mutante: bast2001 - running Dell BIOS firmware upgrade (T162850)
21:20 arlolra@tin: Finished deploy [parsoid/deploy@2210a38]: Updating Parsoid to 28a9a22b (duration: 10m 59s)
21:09 arlolra@tin: Started deploy [parsoid/deploy@2210a38]: Updating Parsoid to 28a9a22b
18:40 niharika29@tin: Synchronized dblists/pp_stage1.dblist: Roll page previews out to all wikis except en and de wiki T162672 (duration: 00m 44s)
18:34 niharika29@tin: Synchronized wmf-config/CommonSettings.php: Enable OOjs UI EditPage on all wikis https://gerrit.wikimedia.org/r/#/c/366868/ (duration: 00m 44s)
18:33 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable OOjs UI EditPage on all wikis https://gerrit.wikimedia.org/r/#/c/366868/ (duration: 00m 44s)
18:30 niharika29@tin: Synchronized wmf-config/Wikibase.php: Wikibase on deployment-prep: Exclude non-existent wikis from clientDbList T173571 (duration: 00m 44s)
18:25 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Increase AbuseFilter autodisable thresholds for Meta-Wiki T173633 (duration: 00m 44s)
18:22 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Add CookBook and Cookbook Talk NS on hiwikibooks T173398 (duration: 00m 45s)
18:16 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Administrators to add/remove 'transwiki' at nowiktionary T172365 (duration: 00m 45s)
18:12 niharika29@tin: Synchronized static/images/: Set project logo for wikimania2018wiki T173042 (duration: 00m 44s)
18:11 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Set project logo for wikimania2018wiki T173042 (duration: 00m 44s)
17:22 mobrovac@tin: Finished deploy [cxserver/deploy@f43ef96]: Bring back cxserver on scb2001 to a stable state - T173038 (duration: 00m 14s)
17:22 mobrovac@tin: Started deploy [cxserver/deploy@f43ef96]: Bring back cxserver on scb2001 to a stable state - T173038
17:12 gehel@tin: Finished deploy [wdqs/wdqs@a1c4f1f]: (no justification provided) (duration: 02m 27s)
17:10 gehel@tin: Started deploy [wdqs/wdqs@a1c4f1f]: (no justification provided)
15:40 bblack: cp1099 - varnish backend restart for mailbox lag
14:55 mobrovac: cxserver depool scb2001 to debug failed checks - T173038
14:54 mobrovac@tin: Finished deploy [cxserver/deploy@1065ffe]: Deploy 1065ffe2 to canary scb2001 for debugging - T173038 (duration: 00m 18s)
14:53 mobrovac@tin: Started deploy [cxserver/deploy@1065ffe]: Deploy 1065ffe2 to canary scb2001 for debugging - T173038
14:52 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1079 - T163190 (duration: 00m 44s)
14:04 zeljkof: EU SWAT finished
14:04 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add HD logos for srwikisource, update them too (T172268) (duration: 00m 44s)
14:02 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Add HD logos for srwikisource, update them too (T172268) (duration: 00m 44s)
13:53 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Update logos for srwikinews, add HD version for them (T172255) (duration: 00m 49s)
13:52 jynus: stop dbstore2001 mariadb@s4, start mariadb@s7
13:52 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Update logos for srwikinews, add HD version for them (T172255) (duration: 00m 44s)
13:46 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set X-Frame-Options: SAMEORIGIN if UploadWizard enabled (T173631) (duration: 00m 44s)
13:46 ottomata: adding index on (database, rev_timestamp) on mediawiki_page_create_2 table on db1047: T170990
13:37 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Update logos for srwiktionary, add HD logos for srwiktionary (T172245) (duration: 00m 44s)
13:36 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Update logos for srwiktionary, add HD logos for srwiktionary (T172245) (duration: 00m 45s)
13:27 jynus: stop dbstore2001 mariadb@s7
13:27 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add new logo for the Baskhir Wikibooks (T173471) (duration: 00m 44s)
13:26 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Add new logo for the Baskhir Wikibooks (T173471) (duration: 00m 44s)
13:26 ottomata: adding index on (database, rev_timestamp) on mediawiki_page_create_2 table on dbstore1002: T170990
13:11 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Reopen bawikibooks (T173471) (duration: 00m 44s)
13:10 zfilipin@tin: Synchronized dblists/closed.dblist: SWAT: Reopen bawikibooks (T173471) (duration: 00m 44s)
12:52 marostegui: Stop replication on db1079 and db1041 to compare their data - T163190
12:52 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1079 - T163190 (duration: 00m 45s)
12:35 marostegui: Stop MySQL on db1015 to decommission it - T173570
09:50 jynus: restart dbstore2001 mariadb@s1
09:40 jynus: restart dbstore2001 mariadb@x1
09:29 moritzm: upgrading hhvm-luasandbox on mw1262-1265
09:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1015 - T173570 (duration: 00m 44s)
09:15 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1015 - T173570 (duration: 00m 44s)
09:01 moritzm: upgrading hhvm-luasandbox on mw1261
08:08 marostegui: Rename tables article_assessment, article_assessment_pages, article_assessment_ratings tables from testwiki on db1078 - T173590
07:42 marostegui: Drop cx_drafts table from x1 - T172364
07:13 marostegui: Stop s4 replication thread on db1095 - T172996
07:09 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2047 (duration: 00m 45s)
02:36 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Aug 21 02:36:58 UTC 2017 (duration 7m 3s)
02:29 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 08m 57s)

2017-08-19

13:17 Amir1: another run: (Hotwc3 → HotWC3) (Lamia Bahy → Albedo11) (Monóxido de carbono → Roquetero) (PaulMichaels → PaulBenario) (Rodrigo.dst → RodrigoTavares) (Sadia Tasnim (Moyna) → মুহাম্মদ সুমন মাহমুদ) (Syou 18331322 → Ms3102) (TzvetelinaOOD1 → Tzveti1) (World Para Taekwondo → TKD at World Para Taekwondo) (Yaellerner → Ya1levy777) (平井俊光 → Toshimit) (T173419)
13:00 Amir1: running the script for ("Clopper228" "CGminded") and ("Gregory.lussier" "StevenSmith83473") (T173419)
12:47 Amir1: ladsgroup@terbium:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=metawiki --logwiki=metawiki "Bolsée" "Kathmandu2017" (T173419)
03:21 krinkle@tin: Synchronized wmf-config/InitialiseSettings.php: Enable jQuery 3 on test.wikidata and mediawiki.org - Id4d42d8c53 (duration: 00m 45s)

2017-08-18

21:05 mutante: gerrit2001 - restarted gerrit again after reverting gerrit:372426, using systemctl commands, not 'service' or init.d
20:55 mutante: restarting gerrit on gerrit2001
20:48 demon@tin: Synchronized scap/plugins/clean.py: Completeness (duration: 00m 44s)
20:43 RainbowSprinkles: prior thing was a no-op, testing
20:42 demon@tin: Pruned MediaWiki: 1.30.0-wmf.9 [keeping static files] (duration: 01m 43s)
19:03 bblack: cp1074 - varnish backend restart (mailbox lag)
19:01 bblack: reboot cp4021-8
18:21 bblack: puppeting cp4021-8 - expect possibility of ipsec alerts, etc...
13:32 Amir1: another run of /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=item --from-id $(tail -100 /tmp/rebuildTermSqlIndex.log | grep -E "Processed up to page (\d+?)" | sed -E "s/Processed up to page //; s/ \(Q.+?//" | tail -1) >>/tmp/rebuildTermSqlIndex.log 2>&1
13:26 Amir1: ladsgroup@terbium:~$ mwscript namespaceDupes.php --wiki=hiwikiversity (T172977)
12:44 Amir1: start of ladsgroup@terbium:~$ time /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=item
11:36 jynus: change topology of dbstore2001:x1 and dbstore2002:x1
11:00 elukey: reboot dbstore1002 for kernel updates
10:34 elukey: restart mysql on dbstore1002 - attempt to reclaim space after big table drop (stop slaves and el_sync, check running queries, stop mysql, check process, start mysql)
09:55 Amir1: one small pass of ladsgroup@terbium:~$ time /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=entity (T171460)
09:46 Amir1: ladsgroup@terbium:~$ time /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=property (T171460)
09:42 moritzm: installing libmspack security updates
09:16 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2077 - T168409 (duration: 00m 44s)
07:28 marostegui: Stop MySQL on db2077 to copy it to dbstore2001 - T168409
07:26 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2077 - T168409 (duration: 00m 44s)
07:20 moritzm: installing openjdk-7 security updates on trusty
06:58 jynus: and nginx
06:55 jynus: systemctl restart squid3.service on install2002, apt seem stuck on some servers
06:24 jynus: stopping and upgrading db2033
05:43 jynus: upgrading and restarting all mariadb instances on dbstore2002
afk: updated fundraising dash from 696a3ff to 7dfc969
00:48 ebernhardson@tin: Synchronized php-1.30.0-wmf.14/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T171213: Increase sampling rate of cirrus satisfaction schema (again) to 1k per bucket per day (duration: 00m 44s)
00:18 ebernhardson@tin: Synchronized php-1.30.0-wmf.14/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T171213: Increase sampling rate of cirrus satisfaction schema (duration: 00m 44s)

2017-08-17

23:31 thcipriani@tin: Synchronized php-1.30.0-wmf.14/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Revert "Disable cirrus MLR ab test" (duration: 00m 44s)
23:21 thcipriani@tin: Synchronized php-1.30.0-wmf.14/resources/src/mediawiki.rcfilters/dm/mw.rcfilters.dm.FilterGroup.js: SWAT: RCFilters: Fix validation for single_option groups T173303 (duration: 00m 44s)
23:14 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Reapply "Remove temporary wgStructuredChangeFiltersEnableExperimentalViews setting" (duration: 00m 45s)
22:10 ppchelko@tin: Finished deploy [changeprop/deploy@2c553a6]: Lower the concurrecy for transcludes to decrease cassandra load during cluster reshaping (duration: 01m 14s)
22:09 thcipriani@tin: Synchronized php-1.30.0-wmf.14/includes/specials/SpecialNewpages.php: Restore the newFromId() approach in SpecialNewpages::feedItemDesc T173541 (duration: 00m 46s)
22:09 ppchelko@tin: Started deploy [changeprop/deploy@2c553a6]: Lower the concurrecy for transcludes to decrease cassandra load during cluster reshaping
22:07 gwicke: restarting pdfrender on scb100* nodes
21:38 ejegg: updated dash from bec0077 to 696a3ff
21:28 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable LoginNotify on sites without Echo too. :|https://gerrit.wikimedia.org/r/#/c/372456/ (duration: 00m 44s)
21:19 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable LoginNotify finally! Yippeeeeee https://gerrit.wikimedia.org/r/#/c/372450/ (duration: 00m 45s)
21:17 bblack: cp1063: varnish backend restart (mailbox lag)
20:07 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.30.0-wmf.14
19:28 chasemp: disable puppet for cloud things for some careful refactor merging
19:14 thcipriani@tin: Synchronized php: group1 wikis to wmf.14 (duration: 00m 46s)
19:12 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to wmf.14
19:07 thcipriani@tin: Finished scap: ProofReadPage Revert to db7507246665e69384c1d92af2aedc62263a5116 T173520 (duration: 06m 13s)
19:01 thcipriani@tin: Started scap: ProofReadPage Revert to db7507246665e69384c1d92af2aedc62263a5116 T173520
18:28 ebernhardson: restart logstash on logstash1003 to see why its not reading EventError messages from kafka
18:24 niharika29@tin: Synchronized php-1.30.0-wmf.14/extensions/RevisionSlider/: Revert Reintroduce hover and bar clicking - https://gerrit.wikimedia.org/r/#/c/372384/ (duration: 00m 48s)
18:16 jynus: upgrading and restarting all mariadb instances on dbstore2001
18:12 niharika29@tin: Synchronized php-1.30.0-wmf.14/extensions/LoginNotify/: Log usage statistics https://gerrit.wikimedia.org/r/#/c/372214/ (duration: 00m 51s)
18:03 thcipriani@tin: Synchronized php-1.30.0-wmf.14/extensions/UploadWizard/UploadWizard.config.php: SWAT: Preserve array keys (language keys) when sorting the language dropdown T173522 (duration: 00m 51s)
17:53 jynus: restarting mariadb (dbstore2001:x1) to test new buffer pool configuration
17:20 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis back to wmf.13 now T173520
17:10 RainbowSprinkles: gerrit: cpu spikes, lots of large gc logs. Looking into it. (nb: things might be a little slow, but it is /up/)
16:54 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis back to wmf.14 now for T164173
16:51 thcipriani@tin: Synchronized php-1.30.0-wmf.14/includes/jobqueue/jobs/RefreshLinksJob.php: Avoid lock acquisition errors for multi-title refreshlinks jobs T173462 (duration: 00m 51s)
16:45 ejegg: updated SmashPig from c501f53 to 98c5516
16:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db2077 - T170662 (duration: 00m 49s)
16:24 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2077 - T170662 (duration: 00m 51s)
16:16 urandom: T169939: RESTBase: converting (33) new keyspaces to time-windowed compaction
16:11 urandom: T169939: Lower eqiad compaction throughput from 20MB/s to 15MB/s
15:46 godog: reimage restbase2001 - T169939
15:42 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2001.codfw.wmnet
14:32 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp3036.*
13:30 zeljkof: EU SWAT finished
13:24 zfilipin@tin: Synchronized static/favicon/wikiversity.ico: SWAT: Update wikiversity favicon (T160491) (duration: 00m 50s)
13:15 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Change $wgArticleCountMethod to any for srwikiquote (T172974) (duration: 00m 51s)
13:08 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: Add one throttling exception (T173444) (duration: 00m 51s)
12:38 moritzm: installing libgd/libsoup security updates
12:08 urandom: T169939: Decommissioning Cassandra/restbase2001-c.codfw.wmnet
12:07 urandom: T169939: Decommissioning Cassandra/restbase2001-b.codfw.wmnet
10:29 gilles: Thumbor stress test finished
10:20 gilles: Load testing Thumbor
10:13 godog: roll-restart nginx on thumbor to apply https://gerrit.wikimedia.org/r/372199
08:36 godog: roll-restart thumbor to apply webp support change
08:30 godog: restart varnish on cp1062 to fix mailbox lag
08:05 godog: reboot ms-be2021, unreachable on network
06:24 marostegui: Stop slave on db2047 to fix duplicate keys - T151029
05:55 marostegui: Stop replication on db2077 to fix duplicate entries - T151029
05:48 marostegui: Stop replication in sync on db1078 and db1015 - T164488
03:11 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Aug 17 03:11:13 UTC 2017 (duration 7m 18s)
03:03 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 03m 58s)
02:50 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.13) (duration: 08m 33s)
02:27 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 52s)
00:45 urandom: T169939: Decommissioning Cassandra/restbase2001-b.codfw.wmnet

2017-08-16

22:30 urandom: T169939: Decommissioning Cassandra/restbase2001-a.codfw.wmnet
22:14 urandom: T169939: Cleaning up wikipedia parsoid snapshots
22:13 urandom: T169939: Rolling restart of Cassandra complete
21:38 bawolff: deleting private info from securepoll_votes that the script missed due to ref-integ issues for old elections (T173393)
21:37 thcipriani: train is on hold pending resolution of T173462
21:33 bawolff: deleting private info in enwiki arbcom1_vote table (T173393)
21:30 urandom: T169939: Rolling restart of Cassandra instances, eqiad, rack d
21:22 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: revert group1 wikis to 1.30.0-wmf.14 for T173462
21:21 thcipriani@tin: Synchronized php: revert group1 wikis to 1.30.0-wmf.14 for T173462 (duration: 00m 47s)
20:46 urandom: T169939: Rolling restart of Cassandra instances, eqiad, rack b
20:45 arlolra@tin: Finished deploy [parsoid/deploy@a9dc803]: Updating Parsoid to 1832a78e (duration: 08m 52s)
20:37 arlolra@tin: Started deploy [parsoid/deploy@a9dc803]: Updating Parsoid to 1832a78e
19:57 urandom: T169939: Rolling restart of Cassandra instances, eqiad, rack a
19:52 MaxSem: Manually cleaning up PI on enwiki (T173393)
19:37 thcipriani@tin: Synchronized php: group1 wikis to 1.30.0-wmf.14 (duration: 00m 46s)
19:35 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.30.0-wmf.14
19:06 urandom: T169939: Rolling restart of Cassandra instances, codfw, rack d
18:57 thcipriani@tin: Synchronized wmf-config/CirrusSearch-common.php: SWAT: cirrus Tune ordering of crossproject search results on enwiki T171803 PART II (duration: 00m 50s)
18:56 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: cirrus Tune ordering of crossproject search results on enwiki T171803 PART I (duration: 00m 51s)
18:41 thcipriani@tin: Synchronized wmf-config: SWAT: Apply token count limits to phrase queries on all wikis T172653 (duration: 00m 53s)
18:30 thcipriani@tin: Synchronized php-1.30.0-wmf.13/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Disable cirrus MLR ab test T171214 (duration: 00m 50s)
18:29 thcipriani@tin: Synchronized php-1.30.0-wmf.14/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Disable cirrus MLR ab test T171214 (duration: 00m 51s)
18:20 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: JobQueueEventBus: Enable group1 T163380 (duration: 00m 54s)
18:18 urandom: T169939: Rolling restart of Cassandra instances, codfw, rack c
17:31 urandom: T169939: Rolling restart of Cassandra instances, codfw, rack b
17:07 demon@tin: Synchronized php-1.30.0-wmf.14/extensions/AntiSpoof/SpoofUser.php: T173394 (duration: 00m 51s)
15:54 marostegui: Stop MySQL and shutdown db1078 for HW checks - T173365
15:43 elukey: drop PageContentSaveComplete_5588433_15423246 from db1047 and dbstore1002 (analytics-slaves)
15:41 godog: delete outdated CFs cassandra metrics from graphite2002 and graphite1003
15:35 marostegui: Rename cx_drafts table on db1029 - T172364
15:09 godog: restart varnish on cp1072 to clear mailbox lag
14:56 godog: restart varnish on cp1099 to clear mailbox lag
14:48 godog: restart varnish on cp1074 to clear mailbox lag
14:44 godog: restart varnish on cp1049 to clear mailbox lag
14:38 gilles: Thumbor stress test finished
14:29 gilles: Stress testing Thumbor from single IP
13:57 zeljkof: EU SWAT finished
13:35 zfilipin@tin: Synchronized dblists/pp_stage1.dblist: SWAT: pagePreviews: Deploy to next 100 stage 1 wikis (T162672) (duration: 00m 50s)
13:09 zfilipin@tin: Synchronized portals: (no justification provided) (duration: 00m 52s)
13:08 zfilipin@tin: Synchronized portals/prod/wikipedia.org/assets: (no justification provided) (duration: 00m 52s)
12:25 Amir1: ladsgroup@terbium:~$ time /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki testwikidatawiki --entity-type=property (T172776, T171460)
12:25 moritzm: powercycling cp3036
12:24 marostegui: Compressing InnoDB on db2077 - T168409
12:22 jmm@puppetmaster1001: conftool action : set/pooled=no; selector: cp3036.esams.wmnet
10:46 marostegui: Stop replication in sync on db1015 and db1078 - T164488
10:42 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2076 - T170662 (duration: 00m 51s)
10:03 godog: copy ubuntu-font-family-sources to stretch-wikimedia - T170817
08:37 marostegui: Drop wikigrok tables from s1, s3 and s5 - T172020
08:12 marostegui: Stop MySQL on db2047 to copy its content to db2077 - T170662
08:10 moritzm: bounced pdfrender on scb1004 (T159922)
08:07 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2047 - T170662 (duration: 01m 06s)
07:25 marostegui: Stop MySQL on db2076 to copy its content to dbstore2001 - T168409
07:08 elukey: executed sudo find -type f -mtime +30 -exec rm {} \; in /var/log/carbon to free some space
06:40 marostegui: Run pt-table-checksum on s3 for revision table - T164488
06:01 marostegui: Stop replication on db2076 to fix duplicate entries - T151029
03:35 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Aug 16 03:35:43 UTC 2017 (duration 7m 21s)
03:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 16m 32s)
02:51 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.13) (duration: 08m 08s)
02:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 51s)
00:43 dereckson@tin: Synchronized php-1.30.0-wmf.12/extensions/Wikidata/Wikidata.php: Explicitly load badges extension (Gerrit:372088) (duration: 00m 51s)
00:42 dereckson@tin: Synchronized php-1.30.0-wmf.14/extensions/Wikidata/Wikidata.php: Explicitly load badges extension (Gerrit:372051) (duration: 00m 51s)
00:38 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Enable Timeless on three French wikis (T154371) + Fixes for Wikidata: Remove wbq_evaluation logging, Update Wikidata property blacklist (Gerrit:367913 and Gerrit:370846) (duration: 00m 53s)
00:02 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Enable WikidataPageBanner on test wikis (T173388) (duration: 00m 51s)

2017-08-15

23:58 dereckson@tin: Synchronized php-1.30.0-wmf.14/skins/Timeless/resources/screen-common.less: Fix messed up recent changes/watchlist legends (T173151) (duration: 00m 50s)
23:55 dereckson@tin: Synchronized php-1.30.0-wmf.13/skins/Timeless/resources/screen-common.less: Fix messed up recent changes/watchlist legends (T173151) (duration: 00m 54s)
23:14 Dereckson: Ran updateArticleCount.php on sr.wikiquote (T172974)
23:12 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Create a few of namespace aliases for hiwikiversity (T172977) (duration: 00m 53s)
22:50 ppchelko@tin: Started restart [changeprop/deploy@444223d]: (no justification provided)
22:45 Pchelolo: scb - codfw: enabling puppet and starting changeprop back T169939
21:40 mobrovac: scb - codfw: disabling puppet and stopping changeprop to release load on Cassandra while dropping old tables - T169939
21:26 urandom: Starting Cassandra restbase2005-b
19:24 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.14
19:19 mobrovac@tin: Finished deploy [restbase/deploy@1139d00]: Use only new parsoid tables (before the removal of old ones), part #2 - T169939 (duration: 02m 50s)
19:16 mobrovac@tin: Started deploy [restbase/deploy@1139d00]: Use only new parsoid tables (before the removal of old ones), part #2 - T169939
19:14 demon@tin: Finished scap: bootstrap wmf.14 v2.0 (duration: 30m 56s)
19:04 mobrovac@tin: Started deploy [restbase/deploy@1139d00]: Use only new parsoid tables (before the removal of old ones) - T169939
19:03 mobrovac@tin: Finished deploy [restbase/deploy@1139d00] (staging): Use only new parsoid tables (before the removal of old ones) - T169939 (duration: 04m 39s)
18:58 mobrovac@tin: Started deploy [restbase/deploy@1139d00] (staging): Use only new parsoid tables (before the removal of old ones) - T169939
18:43 demon@tin: Started scap: bootstrap wmf.14 v2.0
18:39 demon@tin: scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_3982832257" --threads=10 --lang en --quiet' returned non-zero exit status 255 (duration: 03m 05s)
18:35 demon@tin: Started scap: bootstrap wmf.14
18:18 demon@tin: Pruned MediaWiki: 1.30.0-wmf.12 [keeping static files] (duration: 02m 47s)
17:59 bsitzmann@tin: Finished deploy [mobileapps/deploy@34a1304]: Update mobileapps to 33b80dd (T172829 T152441 T172021 T103362) (duration: 04m 00s)
17:55 bsitzmann@tin: Started deploy [mobileapps/deploy@34a1304]: Update mobileapps to 33b80dd (T172829 T152441 T172021 T103362)
15:21 mobrovac: restarting pdfrender on scb1001, added some debug messages to help us diagnose T159922
13:42 gehel: cleanup of old leftover logs on deployment-logstash2:/var/log/logstash
13:21 moritzm: bounced pdfrender on scb1001 (T159922)
11:32 moritzm: installing Linux updates on stretch systems (no reboots yet)
10:45 moritzm: installing PHP security updates on trusty
10:09 jynus: rebooting db1078
09:39 jynus@tin: Synchronized wmf-config/db-eqiad.php: depool db1078 (duration: 00m 49s)
07:56 moritzm: installing cvs security updates
07:18 elukey: restart pdfrender on scb1003
02:56 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Aug 15 02:56:14 UTC 2017 (duration 6m 42s)
02:49 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.13) (duration: 07m 53s)
02:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 58s)

2017-08-14

22:16 Dereckson: Previous log entry is related to Gerrit:371960 and Gerrit:371961.
22:15 dereckson@tin: Synchronized php-1.30.0-wmf.11/extensions/EventBus/JobQueueEventBus.php: JobQueueEventBus: Populate the database field + not set properties are accessed (duration: 00m 48s)
20:44 ppchelko@tin: Finished deploy [restbase/deploy@4d6c706]: Temporary fallback to the new storage buckets before truncation (duration: 09m 35s)
20:35 ppchelko@tin: Started deploy [restbase/deploy@4d6c706]: Temporary fallback to the new storage buckets before truncation
17:09 gehel: resetting mode on stat1005:/srv/published-datasets/discovery recursively
16:29 elukey: execute sudo find -type f -mtime +60 -exec rm {} \; in /var/log/carbon on graphite2001 to free some space in /
13:32 elukey: Execute systemctl mask nfacctd on rhenium.wikimedia.org for T172681
13:08 zeljkof: nothing for EU SWAT
12:21 jynus: stopping replication on all instances of dbstore2001 T169516
11:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2076 - T170662 T151029 (duration: 00m 47s)
11:00 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2076 - T170662 T151029 (duration: 01m 02s)
10:34 marostegui: Restart MySQL on db1069 to pick up new replcation filters
10:27 marostegui: Restart MySQL on db1095 to pick up new replication filters
09:16 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: service=thumbor,name=thumbor2001.codfw.wmnet
09:07 _joe_: stopping thumbor on thumbor2001 after depooling it for testing
09:05 oblivian@puppetmaster1001: conftool action : set/pooled=no; selector: service=thumbor,name=thumbor2001.codfw.wmnet
03:15 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Aug 14 03:15:31 UTC 2017 (duration 7m 6s)
03:08 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.13) (duration: 06m 47s)
02:39 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 29s)

2017-08-13

22:45 ebernhardson: restart elsaticsearch on elastic1017 after setting md2 readahead to 256 to match md2 on 1032-152
19:36 godog: upload python-thumbor-wikimedia 1.2 - T161719
19:01 godog: bounce pdfrender on scb1001 and scb1003 - T159922
18:46 godog: bounce carbon and uwsgi on graphite1003
17:53 reedy@tin: Synchronized wmf-config/db-eqiad.php: fix comment (duration: 00m 47s)
17:52 reedy@tin: Synchronized wmf-config/db-codfw.php: fix comment (duration: 00m 47s)
17:51 reedy@tin: Synchronized refresh-dblist: phpcs (duration: 00m 48s)

2017-08-12

20:00 krinkle@tin: Synchronized php-1.30.0-wmf.13/includes/jobqueue/JobQueueGroup.php: T171371 - Log job pushes to bogus wikis (duration: 00m 53s)
16:09 Reedy: Deleted some bogus user languages from commonswiki.user_properties
15:25 elukey: powercycle mw2256 (able to use com2 but not to login as root, regular ssh hanging) - T163346

2017-08-11

21:47 legoktm@tin: Finished scap: (no justification provided) (duration: 07m 09s)
21:41 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: mw2256.codfw.wmnet
21:40 legoktm@tin: Started scap: (no justification provided)
21:40 legoktm@tin: scap failed: LockFailedError Failed to acquire lock "/var/lock/scap.unknown-but-probably-mediawiki.lock"; owner is "legoktm"; reason is "Deploying Timeless (try 2) - T154371" (duration: 00m 00s)
21:02 ebernhardson: unban elastic1017 from elasticsearch cluster
20:52 bblack: varnish backend restart on cp1099
20:46 legoktm@tin: Started scap: Deploying Timeless (try 2) - T154371
20:42 legoktm@tin: scap aborted: Deploying Timeless - T154371 (duration: 05m 10s)
20:36 legoktm@tin: Started scap: Deploying Timeless - T154371
20:19 bblack: varnish backend restart on cp1049 + cp1074 (mailbox lag)
19:59 ebernhardson: ban elastic1017 from eqiad search cluster
18:28 moritzm: installing subversion security updates
17:19 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2075 (duration: 00m 47s)
17:05 twentyafterfour: Deploying phabricator security update
16:21 jynus_: stop db1069:s6 replication and dropping frwiki, jawiki, ruwiki
15:41 jynus_: stopping db2075 to clone it to dbstore2001
15:39 moritzm: installing git security updates on trusty (jessie/stretch already fixed)
15:38 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2075 (duration: 00m 48s)
13:51 elukey: moved the eventbus scap deployment dirs on kafka[12]00[123] to deploy-service:deploy-service to allow scap to depool/pool - T171506
13:49 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 27s)
13:49 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
13:42 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 13s)
13:42 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
13:25 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 34s)
13:24 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
13:16 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 12s)
13:15 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
13:00 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 17s)
13:00 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
12:51 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 04s)
12:51 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
12:44 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 05s)
12:44 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
12:05 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2046 - T151029 (duration: 00m 48s)
11:25 jynus: stopping and upgrading labsdb1010
11:19 marostegui: Stop replication on db2046 to fix duplicate entries - T151029
09:28 jynus: stopping and restarting es2013 for upgrade
09:07 jynus: stopping and restarting db2046 for upgrade
07:35 elukey: restart pdfrender on scb1004

2017-08-10

23:43 maxsem@tin: Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/371196/2 (duration: 00m 47s)
23:26 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/370310/ (duration: 00m 47s)
23:25 maxsem@tin: Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/370310/ (duration: 00m 46s)
22:31 twentyafterfour@tin: Synchronized php-1.30.0-wmf.13/includes/specials/SpecialDoubleRedirects.php: Hopefully fix T173045 (duration: 00m 48s)
22:17 moritzm: installing git security-updates
20:51 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment, take #2.5 - T137371 (duration: 00m 21s)
20:51 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment, take #2.5 - T137371
20:50 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment, take #2 - T137371 (duration: 00m 09s)
20:50 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment, take #2 - T137371
20:35 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment - T137371 (duration: 00m 05s)
20:34 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment - T137371
20:00 mobrovac@tin: Finished deploy [cassandra/metrics-collector@5db1a43] (staging): First Scap3 deployment - T137371 (duration: 18m 17s)
19:42 mobrovac@tin: Started deploy [cassandra/metrics-collector@5db1a43] (staging): First Scap3 deployment - T137371
19:35 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: touch (duration: 00m 47s)
19:33 reedy@tin: Synchronized wmf-config/: Newsletter on testwiki (duration: 00m 49s)
19:31 mobrovac@tin: Finished deploy [cassandra/logstash-logback-encoder@d085ffa] (aqs): first Scap3 deployment - T116340 (duration: 01m 08s)
19:30 mobrovac@tin: Started deploy [cassandra/logstash-logback-encoder@d085ffa] (aqs): first Scap3 deployment - T116340
19:30 mobrovac@tin: Finished deploy [cassandra/logstash-logback-encoder@d085ffa]: first Scap3 deployment - T116340 (duration: 00m 34s)
19:29 mobrovac@tin: Started deploy [cassandra/logstash-logback-encoder@d085ffa]: first Scap3 deployment - T116340
19:28 mobrovac@tin: Finished deploy [cassandra/logstash-logback-encoder@d085ffa] (staging): first Scap3 deployment (rest of the nodes) - T116340 (duration: 00m 12s)
19:28 mobrovac@tin: Started deploy [cassandra/logstash-logback-encoder@d085ffa] (staging): first Scap3 deployment (rest of the nodes) - T116340
19:27 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: All wikis (except wikidata) to 1.30.0-wmf.13 refs T170631
19:16 godog: restart cassandra instances on xenon to test logstash-logback-encoder deploy
19:13 mobrovac@tin: Finished deploy [cassandra/logstash-logback-encoder@d085ffa] (staging): first Scap3 deployment - T116340 (duration: 00m 03s)
19:13 mobrovac@tin: Started deploy [cassandra/logstash-logback-encoder@d085ffa] (staging): first Scap3 deployment - T116340
18:56 Reedy: created newsletter tables on testwiki
18:55 reedy@tin: Synchronized php-1.30.0-wmf.13/extensions/Newsletter/: sql file updates (duration: 00m 52s)
18:53 reedy@tin: Synchronized php-1.30.0-wmf.13/extensions/WikimediaMaintenance/createExtensionTables.php: newsletter (duration: 00m 52s)
18:21 demon@tin: Synchronized wmf-config/InitialiseSettings.php: Fix commons/commonswiki snafu (duration: 00m 52s)
17:53 kartik@tin: Finished deploy [cxserver/deploy@f43ef96]: (no justification provided) (duration: 00m 44s)
17:52 kartik@tin: Started deploy [cxserver/deploy@f43ef96]: (no justification provided)
17:42 kartik@tin: Finished deploy [cxserver/deploy@1065ffe]: Update cxserver to 686f4f3 (duration: 00m 40s)
17:42 kartik@tin: Started deploy [cxserver/deploy@1065ffe]: Update cxserver to 686f4f3
16:23 ema: cp1072: restart varnish backend
15:48 XioNoX: troubleshoting interface errors between pfw3-codfw and fasw-codfw
15:09 marostegui: Stop replication on db2046 to fix duplicate entries - T151029
15:05 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2046 - T151029 (duration: 00m 50s)
15:03 marostegui: Poweroff es2013 for maintenance - T172265
14:45 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2075 - T170662 (duration: 00m 51s)
14:32 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2045 - T151029 (duration: 00m 51s)
14:11 elukey: restart kafka1012 temporary with some logs to TRACE to debug T172681
away: updated civicrm from 200abc2 to 4aa177b
13:43 marostegui: Compress cebwiki on db1095 - T153058
13:32 zeljkof: EU SWAT finished
13:26 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wgArticleCountMethod to any on srwikisource (T172974) (duration: 00m 53s)
13:18 zeljkof: EU SWAT, part two
13:17 marostegui: Drop m3 databases from dbstore1002 - T156758
13:16 zeljkof: EU SWAT finished
13:15 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable NewUserMessage on knwiki (T172894) (duration: 00m 52s)
12:56 marostegui: Stop MySQL on db2045 to upgrade socket location - T148507
12:07 elukey: restored varnishakafka on cp3032
11:17 elukey: disabled puppet on cp3032 and restarted varnishkafka with debug logging
09:58 jynus: continuing cloning of dbstore2002 to dbstore2001
09:30 gehel: repooling wdqs2001, long after data reload completed
09:30 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs2001.codfw.wmnet
09:27 gehel@tin: Finished deploy [wdqs/wdqs@c186e3e]: (no justification provided) (duration: 01m 31s)
09:25 gehel@tin: Started deploy [wdqs/wdqs@c186e3e]: (no justification provided)
09:17 jynus: disabling lag notification on all s4 replicas
08:59 elukey: update librdkafka1 to 0.9.4.1 on eventlog1001
08:15 elukey: add 50G to carbon lv on graphite1003 and 100G on graphite2002
07:32 jynus: enabling semisymc master replication on db1068
07:29 jynus: disabling semisync slave replication on all SSD hosts on s4
06:45 elukey: powercycle mw2256 - T163346
06:38 elukey: restart pdfrender on scb1004
06:04 Dereckson: Removed 2FA for GoldRingChip account (T172878)
03:23 bblack: cp1008, restbase-dev100[456] have puppet disabled, manual "rm /etc/ssh/userkeys/dzahn"
03:20 bblack: batched cumin puppet agent run on all hosts (not forced)
02:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 09m 12s)

2017-08-09

22:06 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: Get baaack you broken CodeMirror! (duration: 00m 51s)
21:20 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: Deploy CodeMirror https://gerrit.wikimedia.org/r/#/c/370943/ (duration: 00m 50s)
21:11 maxsem@tin: Synchronized php-1.30.0-wmf.12/extensions/CodeMirror: https://gerrit.wikimedia.org/r/#/c/370904/ (duration: 00m 51s)
21:05 ppchelko@tin: Finished deploy [restbase/deploy@f97beeb]: Rollback on canary (duration: 07m 30s)
21:01 godog: add 100G to carbon lv on graphite1003 and graphite2002
20:58 ppchelko@tin: Started deploy [restbase/deploy@f97beeb]: Rollback on canary
20:54 ppchelko@tin: Finished deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. All tables created, finish deploy (duration: 07m 12s)
20:51 marostegui: Remove m3 replication from dbstore1002 - T156758
20:47 ppchelko@tin: Started deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. All tables created, finish deploy
20:43 ppchelko@tin: Finished deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. Attempt 3 (duration: 08m 05s)
20:37 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2046 - T170662 (duration: 00m 49s)
20:35 ppchelko@tin: Started deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. Attempt 3
20:22 twentyafterfour: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: Actually sync all group1 wikis (except wikidata) to 1.30.0-wmf.13 refs T170631
18:57 ppchelko@tin: Started deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. Attempt 2
18:39 marostegui: Stop replication on db2045 to fix duplicate keys - T151029
18:36 urandom: Restarting Cassandra, restbase2001-b.codfw.wmnet (schema jankiness)
18:27 urandom: Restarting Cassandra, restbase2001-a.codfw.wmnet (schema jankiness)
18:10 ebernhardson: restart es on elastic1018 to test interleaved numa
17:50 marostegui: Add db2076 to tendril - T170662
17:41 marostegui: Compress innodb on s6 on db2076 - T170662
17:31 legoktm: changed mediawiki/vendor to fast forward only
17:24 ppchelko@tin: Finished deploy [restbase/deploy@f97beeb]: Rollback on canary (duration: 01m 27s)
17:23 ppchelko@tin: Started deploy [restbase/deploy@f97beeb]: Rollback on canary
16:35 ppchelko@tin: Finished deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation (duration: 04m 46s)
16:30 ppchelko@tin: Started deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation
16:29 urandom: T172384: Upgrading Cassandra in RESTBase dev to 3.11.0-wmf2 (patched to disable use of FastThreadLocal)
16:15 marostegui: Stop mysql on db2046 to copy its content to db2076 - T170662
16:14 elukey: rolling restart of eventstream on scb hosts to deploy https://gerrit.wikimedia.org/r/370793
16:11 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2046 - T170662 (duration: 00m 50s)
16:08 gehel: config update on logstash, a few logs might be lost during restart - T172713
15:19 XioNoX: removing old pfw related config from cr1-codfw - T171970
14:08 mutante: purging req.urls with "^/resources" from varnish cluster, to fix redirect with cached 404
14:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2075 to s5 - T170662 (duration: 00m 50s)
14:03 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2075 to s5 - T170662 (duration: 00m 51s)
13:55 marostegui: Reboot db2076 for maintenance - T170662
13:21 marostegui: Stop replication on db2075 for maintenance - T170662
13:18 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Allow bureaucrats remove confirmed user group (T101983) (duration: 00m 51s)
13:10 gehel@tin: Finished deploy [wdqs/wdqs@6620e0f]: (no justification provided) (duration: 01m 34s)
13:09 gehel@tin: Started deploy [wdqs/wdqs@6620e0f]: (no justification provided)
12:41 gehel: restarting updater on wdqs1001 (real fix coming up soon
10:20 jynus: stopping dbstore2002's mysqls and cloning them to dbstore2001
08:39 jynus: disable puppet on dbstore2001, about to be reimaged
07:47 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1065, db1064, db1070 (duration: 01m 04s)
07:12 jynus: stopping replication in sync between db1069 and db1065, db1044, db1064, db1070
07:07 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1065, db1064, db1070 (duration: 00m 47s)
06:26 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1079 (duration: 00m 51s)
06:11 jynus: stopping db1069 (s7) and db1079 replication in sync
06:03 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1085, depool db1079 (duration: 00m 51s)
05:42 jynus: stopping db1069 (s6) and db1085 replication in sync
05:32 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1060, depool db1085 (duration: 00m 50s)
04:56 jynus: stopping db1069 (s2) and db1060 replication in sync
04:54 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1060 (duration: 00m 58s)
04:48 jynus: restarting pdfrender on scb1001, unresponsive
04:09 jynus: powercycling mw2256
03:23 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.13) (duration: 07m 33s)
02:56 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.12) (duration: 07m 45s)
02:38 ebernhardson: restart elasticsearch1017 with niofs store
02:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 38s)
00:27 ebernhardson: test vm.zone_reclaim_mode=0 on elastic1017

2017-08-08

23:23 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Allow bureaucrats on WMF wikis to grant and remove 'confirmed' (T101983) (duration: 00m 51s)
21:25 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 wikis to 1.30.0-wmf.13 refs T170631
21:08 ppchelko@tin: Finished deploy [restbase/deploy@c16fb6b]: Update summary and licensing information for pageviews API (duration: 08m 31s)
21:05 twentyafterfour@tin: Finished scap: again: deploy 1.30.0-wmf.13 to testwikis and rebuild l10n refs T170631 (duration: 42m 55s)
20:59 ppchelko@tin: Started deploy [restbase/deploy@c16fb6b]: Update summary and licensing information for pageviews API
20:32 twentyafterfour: updated mediawiki.org changelog for 1.30.0-wmf.13
20:22 twentyafterfour@tin: Started scap: again: deploy 1.30.0-wmf.13 to testwikis and rebuild l10n refs T170631
19:46 ema: restart varnish backend on cp1074
19:45 godog: bounce thumbor-instances on thumbor1003 to make sure all memory limits are applied
19:44 twentyafterfour@tin: scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="test2wiki" --outdir="/tmp/scap_l10n_224168097" --threads=10 --lang en --quiet' returned non-zero exit status 255 (duration: 07m 33s)
19:36 twentyafterfour@tin: Started scap: deploy 1.30.0-wmf.13 to testwikis and rebuild l10n refs T170631
18:28 XioNoX: frack-codfw moved to new infrastructure
18:27 elukey: re-enabled irc-echo after the puppet shower
18:11 elukey: stop ircecho to avoid puppet shower
16:59 twentyafterfour: Branching mediawiki/master to mediawiki/wmf/1.30.0-wmf.13 refs T170631
16:04 elukey: rolling restart of varnishkafka-webrequest to apply https://gerrit.wikimedia.org/r/#/c/370659/ (puppet automatically restarts)
15:03 XioNoX: starting pfw-codfw migration - T171970
14:19 elukey: restart of all the varnishkafka statsv/eventlogging instances on caching hosts to pick up https://gerrit.wikimedia.org/r/370644 (puppet automatic restarts)
14:16 elukey: set mw2256 pooled=inactive + downtime to allow BIOS upgrade - T163346
14:07 ladsgroup@tin: Synchronized wmf-config/Wikibase-production.php: SWAT: Add copyright info for Wikidata API (T112606) (duration: 00m 47s)
13:56 gehel: dump of task API for elasticsearch eqiad - T169498
13:54 ladsgroup@tin: Synchronized static/images/project-logos: SWAT: Fix srwiki logos (T150618) (duration: 00m 48s)
13:00 elukey: restart varnishkafka-webrequest with kafka.broker.version.fallback=0.9.0.1 + kafka.api.version.request=false on cp3032 (local test, to rollback remove the lines from /etc/varnishkafka/webrequest.conf)
12:49 Amir1: start of ladsgroup@terbium:~$ /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=property --rebuild-all-terms (T172776)
12:32 elukey: restart pdfrender on scb1002
12:20 Amir1: start of ladsgroup@terbium:~$ /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=property (T172776)
12:15 gehel: restarting wdqs-updater on wdqs1001
12:14 elukey: stop eventlogging on eventlog1001 to test kafka consumer failures
11:22 Amir1: start of ladsgroup@terbium:~$ timeout 3500s /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=item >>/tmp/rebuildTermSqlIndex.log 2>&1 (T171460)
10:12 elukey: update librdkafka1* on notebook100[12] and stat1003
09:09 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool db1098 as the main recentchanges/watchlist s6 role (duration: 00m 47s)
08:46 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool db1098 at 50% load (duration: 00m 46s)
08:28 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool db1098 with limited load (duration: 00m 46s)
07:53 jynus@tin: Synchronized wmf-config/db-codfw.php: Add db1098 (duration: 00m 47s)
07:41 elukey: stop puppet on cp3032 (cache::text) to set varnishkafka-webrequest logging to debug
07:34 Amir1: stopped the script and re-running without --deduplicate-terms (T171460)
07:21 Amir1: start of ladsgroup@terbium:~$ time mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki=wikidatawiki --entity-type=property --deduplicate-terms (T171460)
06:12 elukey: alert users with big home directories for stat1005 disk alarms (will erase data later on only if they don't answer)
06:12 elukey: restart pdfrender on scb1003
03:55 mutante: phab1001 /usr/local/bin/community_metrics.sh | /usr/local/bin/project_changes.sh creating stats mails to admins (which failed before) (T163938)
03:25 mutante: phab1001 /srv/phab/tools/public_task_dump.py to create dump, was failed cron due to missing /srv/dumps/
02:53 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Aug 8 02:53:13 UTC 2017 (duration 6m 53s)
02:46 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.12) (duration: 06m 59s)
02:36 ejegg: enabled donation queue consumer
02:26 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 25s)
00:46 ejegg: disabled donation queue consumer

2017-08-07

21:28 urandom: T172384: Upgrading Cassandra to 3.11.0-wmf1 in dev environment (build patched to disable in-built heap dumping)
19:51 ejegg: updated CiviCRM from f24ba78 to 200abc2
19:05 ebernhardson@tin: Synchronized php-1.30.0-wmf.12/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T171212 - Turn on CirrusSearch MLR AB test (duration: 00m 46s)
19:00 ebernhardson@tin: Synchronized wmf-config/CirrusSearch-common.php: T169498 - Enable max token count for phrase rescore on zh lang wikis (step 2) (duration: 00m 46s)
18:59 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T169498 - Enable max token count for phrase rescore on zh lang wikis (step 1) (duration: 00m 46s)
18:53 ebernhardson@tin: Synchronized wmf-config/CirrusSearch-production.php: T171212 - Update CirrusSearch AB test rescore profiles (duration: 00m 46s)
18:44 ebernhardson@tin: Synchronized wmf-config/Wikibase-labs.php: T112606 - beta only - Add copyright info for Wikidata API (duration: 00m 46s)
18:43 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T172630 - Enable wgMinervaEnableSiteNotice for kowiki (duration: 00m 46s)
18:38 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T170687 - Exclude files from Special:ShortPages on commons (duration: 00m 46s)
18:23 ebernhardson@tin: Synchronized php-1.30.0-wmf.12/extensions/CirrusSearch/: T169498 limit phrase token count, T172464 constant boost ltr queries (duration: 00m 58s)
18:10 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T172594 - Translate sitename for nl.wikinews (duration: 00m 47s)
18:05 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: Grant autopatrol to editor in en.wikibooks - T172561 (duration: 00m 47s)
17:57 mutante: phab2001 - re-enabling puppet, but closing firewall for 80/443
17:56 jynus: stopping slave and reparitioning db1098
17:10 marostegui: Restart s7 instance on db1102 to pick up new replication filters - T172693
17:05 gehel@tin: Finished deploy [wdqs/wdqs@da33919]: (no justification provided) (duration: 02m 28s)
17:02 gehel@tin: Started deploy [wdqs/wdqs@da33919]: (no justification provided)
16:44 marostegui: Restart s7 instance on db1069 to pick up new replication filters - T172693
16:37 XioNoX: manually restarted varnish on cp1099
15:10 thcipriani: restarting jenkins for plugin upgrade
14:51 gehel: reducing elasticsearch eqiad concurrent rebalance to 4 (from 8)
14:38 elukey: updated librdkafka1 and ++1 to 0.9.4.1 on hafnium
14:32 mutante: phab2001 - stopping Apache,schedule downtime for http and puppet
14:22 herron: mx[1,2]001, fermium: Installed libmail-dkim-perl and restarted spamassassin service - T172689
13:15 jynus: reboot db1098
12:39 _joe_: restarting pdfrender on scb1001, T159922
12:39 elukey: restart kafka on kafka1018 to force it out of the kafka topic leaders - T172681
12:26 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2074 - T171321 (duration: 00m 45s)
12:08 gehel: deploying https://gerrit.wikimedia.org/r/#/c/299825/ - some logs will be lost during logstash restart
10:02 marostegui: Add dbstore2002:3313 to tendril - T171321
09:47 jynus: stopping db1050's mysql and cloning it to db1089
09:06 elukey: set net.netfilter.nf_conntrack_tcp_timeout_time_wait=65 (was 120) on all the analytics kafka brokers - T136094
09:03 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2065 after fixing: linter, page and watchlist tables (duration: 00m 47s)
08:12 marostegui: Force BBU re-learn on db1016 - T166344
07:02 marostegui: Stop replication on db2065 to reimport: page, linter and watchlist tables
07:02 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2065 to reimport: page, linter and watchlist tables (duration: 00m 47s)
06:38 marostegui: Stop MySQL on db2074 - T171321
06:37 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2074 - T171321 (duration: 00m 46s)
06:33 marostegui: Stop replication on db2075 - T170662
06:27 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2073 - T171321 (duration: 00m 47s)
06:20 marostegui: Force BBU re-learn on db1016 - T166344
02:57 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Aug 7 02:57:42 UTC 2017 (duration 6m 42s)
02:51 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.12) (duration: 07m 56s)
02:30 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 10m 16s)

2017-08-06

13:17 elukey: powercycle mw2256 - com2 frozen - T163346
13:13 elukey: restart pdfrender on scb1002
06:18 ebernhardson@tin: Synchronized wmf-config/PoolCounterSettings.php: T169498: Reduce cirrus search pool counter to 200 parallel requests cluster wide (duration: 02m 54s)
01:28 chasemp: conf2002:~# service etcdmirror-conftool-eqiad-wmnet restart (not sure what else to do the service failed)

2017-08-05

14:40 Reedy: created oauth tables on foundationwiki T172591
14:13 reedy@tin: Synchronized php-1.30.0-wmf.12/extensions/WikimediaMaintenance/createExtensionTables.php: add oauth (duration: 00m 48s)

2017-08-04

23:51 mutante: phab2001 - removed outdated /etc/hosts entries, that fixed rsync, syncing /srv/repos/ from phab1001
23:35 mutante: phab2001 rebooting
23:35 mutante: phab2001 - installing various package upgrades, apt-get autoremove old kernel images
23:12 mutante: "reserved" UID 498 for phd on https://wikitech.wikimedia.org/wiki/UID | phab2001: find -exec chown to fix all the files , restart cron
23:04 mutante: phab2001 - changing UID/GID for phd user from 997:997 to 498:498 to make it match phab1001, to fix rsync breaking permissions. (rsync forces --numeric-ids when fetching from and rsyncd configured with chroot=yes). chown -R phw:www-data /srv/repos/
22:37 ejegg: restarted donations and refund queue consumers
21:44 ejegg: stopped donations and refund queue consumers
21:24 urandom: T172384: Disabling Puppet in dev environment to prevent unattended Cassandra restarts
20:19 mutante: renewing SSL cert for status.wm.org (just like wikitech-static, but that one didnt have monitoring?)
20:02 mutante: wikitech-static-ord - apt-get install certbot
19:41 ejegg: updated CiviCRM from f1fd7f0 to f24ba78
19:38 mutante: renaming graphite varnish director/fixing config, running puppet on cache misc, tested on cp1045
18:17 andrewbogott: switched most cloud instance to new puppetmasters, as per https://phabricator.wikimedia.org/T171786
11:46 marostegui: Deploy schema change directly on s3 master for maiwikimedia - T172485
11:30 marostegui: Deploy schema change directly on s3 master for kbpwiki - T172485
11:14 marostegui: Deploy schema change directly on s3 master for dinwiki - T172485
10:14 marostegui: Deploy schema change directly on s3 master for atjwiki - T172485
10:05 marostegui: Stop replication on db2073 for maintenance
09:22 marostegui: Add dbstore2002 to tendril - T171321
09:19 marostegui: Deploy schema change directly on s3 master for techconductwiki - T172485
08:35 marostegui: Deploy schema change directly on s3 master for hiwikiversity - T172485
08:19 marostegui: Deploy schema change directly on s3 master for wikimania2018wiki - T172485
08:04 marostegui: Sanitize wikimania2018wiki on sanitarium and sanitarium2 - T155041
07:47 marostegui: Stop MySQL on db2073 to copy its data to dbstore2002 - https://phabricator.wikimedia.org/T171321
07:47 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2073 - T171321 (duration: 00m 47s)
07:07 moritzm: installing imagemagick regression security updates on trusty
06:47 marostegui: Sanitize hiwikiversity on sanitarium and sanitarium2 - T171829
05:23 mutante: phab1001 sudo ip addr del 10.64.32.186/32 dev eth0 (T172478)
02:28 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=phab1001-vcs.eqiad.wmnet
02:28 dzahn@neodymium: conftool action : set/pooled=no; selector: name=phab1001-vcs.eqiad.wmnet
02:15 mutante: phab1001 can't talk to mx servers via IPv6, but works via IPv4. iridium and other mailservers can also talk IPv6 to it. why? it did not change even when stopping ferm on client and on server it allows from anywhere. workaround for now was to hardcode IPv4 IP in phab config. (T163938)
02:13 twentyafterfour: outgoing phab mail is working again
01:16 twentyafterfour: twentyafterfour@phab1001:/srv/repos$ sudo chown -R phd:www-data /srv/repos
01:16 twentyafterfour: twentyafterfour@phab1001:/srv/repos$ sudo chmod -x /usr/local/sbin/sync-srv-repos
01:12 mutante: phab1001 - stopped and started exim, which is now running with same options as iridium
00:53 twentyafterfour: starting phd to shut up icinga
00:40 ebernhardson@tin: Synchronized php-1.30.0-wmf.12/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: Turn CirrusSearch MLR test back off (duration: 00m 47s)
00:35 twentyafterfour: phabricator web service is also up
00:34 mutante: phab1001 - service IPs switched - puppet ran - ssh-phab service up
00:30 twentyafterfour: phab1001 chown -R phd:phd /srv/repos
00:28 twentyafterfour: testing phd on phab1001
00:24 mutante: phab1001 sudo ip addr add 2620:0:861:103:10:64:32:186/128 dev eth0
00:24 mutante: phab1001 sudo ip addr add 10.64.32.186/32 dev eth0
00:23 mutante: iridium sudo ip addr del 2620:0:861:ed1a::3:16/128 dev lo
00:23 mutante: iridiium sudo ip addr del 208.80.154.250/32 dev lo
00:23 mutante: iridium sudo ip addr del 2620:0:861:103:10:64:32:186/128 dev eth0
00:23 mutante: iridium sudo ip addr del 10.64.32.186/32 dev eth0
00:07 twentyafterfour: stoped phd and apache on iridium
00:07 twentyafterfour: stopped ssh-phab on iridium
00:03 mutante: phab1001 - /usr/bin/rsync -av rsync://iridium.eqiad.wmnet/srv-repos /srv/repos/
00:02 twentyafterfour: Taking phabricator down for maintenance / migration to a new server: phab1001

2017-08-03

23:43 ebernhardson@tin: Synchronized php-1.30.0-wmf.12/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T171212: Turn on CirrusSearch MLR AB test (duration: 00m 46s)
23:36 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/370117/2 (duration: 00m 47s)
23:14 ebernhardson@tin: Synchronized php-1.30.0-wmf.12/extensions/CodeMirror/resources/ext.CodeMirror.js: Only show popup if CodeMirror button exists (duration: 00m 46s)
23:12 ebernhardson@tin: Synchronized php-1.30.0-wmf.12/extensions/WikimediaEvents/extension.json: (no justification provided) (duration: 00m 47s)
23:08 ebernhardson@tin: Synchronized wmf-config/CirrusSearch-production.php: CirrusSearch config for MLR test (step 2) (duration: 00m 46s)
23:07 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: CirrusSearch config for MLR test (step 1) (duration: 00m 47s)
22:06 ejegg: restarted donation and refund queue consumers
21:51 maxsem@tin: Synchronized wmf-config/CommonSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/370096/ (duration: 00m 47s)
21:51 ejegg: enabled ingenico_audit drupal module
21:45 ejegg: updated civicrm from 5c741b1 to f1fd7f0
21:43 ejegg: disabled donation and refund queue consumers
21:31 chasemp: clear out nova-fullstack project so we can monitor fresh on new puppetmaster
21:30 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: Enable CodeMirror https://gerrit.wikimedia.org/r/#/c/370072/ (duration: 00m 47s)
21:20 maxsem@tin: Synchronized php-1.30.0-wmf.12/extensions/CodeMirror: https://gerrit.wikimedia.org/r/#/c/370066/ https://gerrit.wikimedia.org/r/#/c/370074/ (duration: 00m 47s)
21:04 herron: copper /var/cache/pbuilder 95% full - grew /dev/copper-vg/pbuilder fs by +5G and tune2fs -m 0. now at 85% full
20:40 milimetric@tin: Finished deploy [analytics/refinery@cc40bf2]: Fix sqoop script with updated scap config (duration: 11m 51s)
20:29 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis except wikidata to 1.30.0-wmf.12, leaving wikidata behind due to T172320
20:28 milimetric@tin: Started deploy [analytics/refinery@cc40bf2]: Fix sqoop script with updated scap config
20:24 milimetric@tin: Finished deploy [analytics/refinery@cc40bf2]: Fix sqoop script (duration: 33m 17s)
19:51 milimetric@tin: Started deploy [analytics/refinery@cc40bf2]: Fix sqoop script
19:21 twentyafterfour@tin: Synchronized php-1.30.0-wmf.12/extensions/Wikidata/extensions/Wikibase/client/includes/Changes: be sure https://gerrit.wikimedia.org/r/#/c/369847/ is sync'd (duration: 00m 46s)
19:12 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.30.0-wmf.12 refs T168053
19:08 twentyafterfour: deploying 1.30.0-wmf.12 to group1, will proceed to group2 after verifying the branch is stable.
18:59 arlolra: Updated Parsoid to 6e1a20d5 (T168765, T155038, T165977)
18:53 demon@tin: Synchronized wmf-config/InitialiseSettings.php: Final gerrit 369960: "pagePreviews: Deploy to first 50 of stage 1 wikis" (duration: 00m 46s)
18:52 arlolra@tin: Finished deploy [parsoid/deploy@48be65b]: Updating Parsoid to 6e1a20d5 (duration: 06m 25s)
18:51 demon@tin: Synchronized wmf-config/CommonSettings.php: new dblist for gerrit 369960 (duration: 00m 46s)
18:46 arlolra@tin: Started deploy [parsoid/deploy@48be65b]: Updating Parsoid to 6e1a20d5
18:41 demon@tin: Synchronized docroot/: gerrit 369960 (duration: 00m 47s)
18:39 demon@tin: Synchronized dblists/: gerrit 369960 (duration: 00m 47s)
18:29 demon@tin: Synchronized wmf-config/InitialiseSettings.php: last part of gerrit 351287 (duration: 00m 47s)
18:23 mobrovac@tin: Finished deploy [restbase/deploy@65af18d]: Expose the recommendation API publicly and activate hiwikiversity - T170877 T168765 (duration: 08m 33s)
18:17 demon@tin: Synchronized wmf-config/CommonSettings.php: Add new dblist for page preview stuff, basically no-op (duration: 00m 47s)
18:15 demon@tin: Synchronized docroot/noc/conf/pp_stage0.dblist: tidy up pp config (duration: 00m 46s)
18:15 mobrovac@tin: Started deploy [restbase/deploy@65af18d]: Expose the recommendation API publicly and activate hiwikiversity - T170877 T168765
18:14 demon@tin: Synchronized dblists/pp_stage0.dblist: clean up config (duration: 00m 47s)
18:11 mobrovac@tin: Finished deploy [restbase/deploy@65af18d] (staging): (no justification provided) (duration: 01m 57s)
18:09 mobrovac@tin: Started deploy [restbase/deploy@65af18d] (staging): (no justification provided)
18:03 arlolra: Updated Parsoid to 651f12c2 (T119802, T73386, T170289)
17:55 andrewbogott: restarting rabbitmq-server on labcontrol1001
17:50 arlolra@tin: Finished deploy [parsoid/deploy@612e711]: Updating Parsoid to 651f12c2 (duration: 11m 17s)
17:39 arlolra@tin: Started deploy [parsoid/deploy@612e711]: Updating Parsoid to 651f12c2
16:15 jynus: repointing dbproxy1008 back to db1043
16:06 demon@tin: Finished deploy [gerrit/gerrit@15f1544]: This is a test, disregard (duration: 00m 03s)
16:05 demon@tin: Started deploy [gerrit/gerrit@15f1544]: This is a test, disregard
16:03 demon@tin: Synchronized wmf-config/: doc fixes, no-op (duration: 00m 48s)
16:02 demon@tin: Synchronized scap/plugins/prep.py: doc fixes, no-op (duration: 00m 47s)
16:00 godog: silence cassandra-related alerts on restbase-dev cluster, known OOMs
15:44 urandom: T172384: lower tombstone failure threshold in RESTBase dev to 1000
15:39 reedy@tin: Synchronized wmf-config/interwiki.php: Update for 3 new wikis (duration: 00m 46s)
15:31 reedy@tin: rebuilt wikiversions.php and synchronized wikiversions files: techconductwiki
15:27 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: techconductwiki (duration: 00m 46s)
15:25 reedy@tin: Synchronized static/images/project-logos/: techconductwiki (duration: 00m 44s)
15:24 reedy@tin: Synchronized dblists/: techconductwiki (duration: 00m 47s)
15:03 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic1028.eqiad.wmnet
15:02 gehel: unbanning and repooling elastic1028 - T168816
14:58 reedy@tin: Synchronized static/images/project-logos/: hiwikiversity (duration: 00m 46s)
14:57 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: hiwikiversity (duration: 00m 48s)
14:55 reedy@tin: rebuilt wikiversions.php and synchronized wikiversions files: hiwikiversity
14:55 reedy@tin: Synchronized dblists: hiwikiversity (duration: 00m 49s)
14:53 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic10(29|30|31|32).eqiad.wmnet
14:53 Dereckson: mwscript initSiteStats.php --wiki srwikiquote --update (T172241)
14:52 gehel: unbanning and repooling elastic10(29|30|31|32) - T168816
14:29 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: wikimania2018wiki (duration: 00m 47s)
14:28 reedy@tin: rebuilt wikiversions.php and synchronized wikiversions files: (no justification provided)
14:27 reedy@tin: Synchronized dblists: wikimania2018wiki (duration: 00m 47s)
14:21 marostegui: Restart MySQL on db2019 to get the new socket location updated
14:19 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2045 - T170662 (duration: 00m 46s)
14:06 marostegui: Compress s5 on db2075 - T170662
14:02 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=elastic10(28|29|30|31|32).eqiad.wmnet
14:01 gehel: depooling and shutting down elastic10(28|29|30|31|32) - T168816
14:00 marostegui: Add db2075 to tendril - T170662
13:44 marostegui: Enable gtid back on codfw s4 slaves - T170351
13:35 marostegui@tin: Synchronized wmf-config/db-codfw.php: Promote db2051 as s4 codfw master - T170351 (duration: 00m 46s)
13:28 elukey: drop CookieBlock backup tables for T171883
13:26 marostegui: Starting the actual s4 codfw failover db2019 -> db2051 - T170351
13:22 gehel: banning elastic1032 - T168816
13:19 chasemp: disabling puppet acros wmcs things for testing
13:14 marostegui: Start topology change for s4 in codfw, slaves will be moved under db2051 - T170351
13:10 aude@tin: Synchronized php-1.30.0-wmf.12/extensions/Wikidata: Fix bug in InjectRCRecordsJob (duration: 02m 12s)
13:01 marostegui: Disable gtid on s4 codfw slaves to get ready for the topology change - T170351
12:15 marostegui: Restart MySQL on db2051 - T170351
10:21 elukey: removed /run/pacct_shadow.d on stat1005 to allow atopacct.service restart
10:01 moritzm: installing poppler security updates on trusty
09:57 _joe_: systemctl reset-failed puppetmaster.service on labpuppetmaster1002
09:52 moritzm: installing mysql 5.5 security updates (package as shipped by Debian jessie, not our internal wmf-mariadb package)
09:32 gehel: banning elastic103[01] - T168816
08:56 ema: upgrading cache_upload to varnish 4.1.8-1wm1
08:36 gehel: banning elastic102[89] - T168816
08:35 marostegui: Stop MySQL on db2045 to copy its data to db2075 - T170662
08:34 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2045 - T170662 (duration: 00m 47s)
08:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db2045 - T170662 (duration: 00m 47s)
07:58 marostegui: Add db2073 and db2074 to tendril - T170662
06:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2074 to s3 - T170662 (duration: 00m 46s)
06:18 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2074 to s3 - T170662 (duration: 00m 46s)
06:01 marostegui: Compress s7 on db1102 - T172169
05:57 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2051 - T170351 (duration: 00m 54s)
05:27 marostegui: Deploy alter table on enwiki - labsdb1010 - T166204
05:17 marostegui: Stop replication on labsdb1011 for maintenance - T153743
03:09 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Aug 3 03:09:22 UTC 2017 (duration 7m 18s)
03:02 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.12) (duration: 05m 47s)
02:33 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 07m 53s)
01:40 twentyafterfour: repositories synced for phabricator migration.
00:28 reedy@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Simply ores (duration: 00m 47s)
00:26 twentyafterfour: scheduled downtime for phabricator migration
00:12 mutante: phab1001 starting ferm service
00:11 mutante: iridium restarted ferm - rsync fragment was in config but not applied, breaking data rsync to phab1001
00:05 mutante: rsyncing /srv/repos from iridium to phab1001 (T163938)

2017-08-02

23:53 ebernhardson@tin: Finished scap: dblists/rtl.dblist T172305: Adding RTL database list for project with default RTL languages (duration: 36m 21s)
23:17 ebernhardson@tin: Started scap: dblists/rtl.dblist T172305: Adding RTL database list for project with default RTL languages
21:48 ejegg: updated SmashPig from f4ca53c to c501f53
20:24 bsitzmann@tin: Finished deploy [mobileapps/deploy@3b61ced]: Update mobileapps to 2d8e8f6 (T170325) (duration: 05m 38s)
20:19 bsitzmann@tin: Started deploy [mobileapps/deploy@3b61ced]: Update mobileapps to 2d8e8f6 (T170325)
19:50 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.30.0-wmf.11 refs T168053 - rollback due to T172320
19:24 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.30.0-wmf.12 refs refs T168053
19:23 twentyafterfour: group1 wikis to 1.30.0-wmf.12 refs T168053
18:41 thcipriani@tin: Synchronized php-1.30.0-wmf.12/includes/specials/SpecialRecentchanges.php: SWAT: Follow-up 31be7d0: send tags list if experimental mode is disabled (duration: 00m 47s)
18:21 thcipriani@tin: Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: beta-only change Enable HTML5 sections in betalabs (duration: 00m 46s)
18:20 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic10(24|25|26|27).eqiad.wmnet
18:18 thcipriani@tin: Synchronized php-1.30.0-wmf.12/includes/specials/SpecialUndelete.php: SWAT: Fix Special:Undelete search - use variable and not request param (duration: 00m 46s)
18:18 gehel: un-banning and repooling elastic102[4567] - T168816
18:13 thcipriani@tin: Synchronized wmf-config/jobqueue.php: SWAT: JobQueueEventBus: Enable job events in group0 wikis T163380 Part II (duration: 00m 47s)
18:11 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: JobQueueEventBus: Enable job events in group0 wikis T163380 Part I (duration: 00m 47s)
17:56 ottomata: restart kafka1012 broker with listeners=PLAINTEXT://:9092 to verify https://gerrit.wikimedia.org/r/#/c/356232/ before merge. This should be a functional no-op
17:44 mobrovac@tin: Finished deploy [cxserver/deploy@f43ef96]: Revert canary scb2001 to f43ef963, take #2 (duration: 00m 16s)
17:44 mobrovac@tin: Started deploy [cxserver/deploy@f43ef96]: Revert canary scb2001 to f43ef963, take #2
17:44 mobrovac@tin: Finished deploy [cxserver/deploy@f43ef96]: Revert canary scb2001 to f43ef963 (duration: 00m 29s)
17:43 mobrovac@tin: Started deploy [cxserver/deploy@f43ef96]: Revert canary scb2001 to f43ef963
17:26 mobrovac@tin: Finished deploy [recommendation-api/deploy@8dfae34]: (no justification provided) (duration: 01m 36s)
17:25 mobrovac@tin: Started deploy [recommendation-api/deploy@8dfae34]: (no justification provided)
17:18 herron: installed libmail-spf-perl and restarted spamassassin service on mx[1,2]001 - T172299
16:45 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=elastic10(24|25|26|27).eqiad.wmnet
16:43 gehel: depooling and shutting down elastic102[4567] - T168816
15:34 mobrovac@tin: Finished deploy [cxserver/deploy@cf3e280]: (no justification provided) (duration: 00m 22s)
15:34 mobrovac@tin: Started deploy [cxserver/deploy@cf3e280]: (no justification provided)
15:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Change db2051 IP - T169501 (duration: 00m 46s)
15:17 marostegui@tin: Synchronized wmf-config/db-codfw.php: Change db2051 IP - T169501 (duration: 00m 46s)
15:17 nschaaf@tin: Finished deploy [recommendation-api/deploy@baa11c0]: source parameter validation (duration: 03m 22s)
15:14 nschaaf@tin: Started deploy [recommendation-api/deploy@baa11c0]: source parameter validation
15:06 marostegui: Poweroff db2051 to get it move to another rack - T169501
14:46 ema: upgrading cache_misc to varnish 4.1.8-1wm1
14:31 ema: varnish 4.1.8-1wm1 (fixes VSV00001, DSA 3924-1) built and uploaded to apt.w.o
14:12 marostegui: Stop MySQL on db2051 in order to get it ready to move to another rack - T170351
13:16 hashar@tin: Synchronized wmf-config/Wikibase-production.php: Write to term_full_entity_id column in wb_terms table in prod too - T167229 (duration: 00m 46s)
13:07 hashar@tin: Synchronized php-1.30.0-wmf.12/extensions/Wikidata: fix constraint type checks - T169326 (duration: 02m 13s)
12:59 gehel: banning elastic102[67] - T168816
12:41 gehel: banning elastic102[45] - T168816
11:42 jynus: disabling autolearn on newest db1* hosts
11:39 jynus: disabling autolearn on newest db2* hosts
11:33 marostegui: Stop MySQL on db2051 for maintenance - T170351
11:29 jynus: setting all es hosts with disabled auto-learn bbu properties
11:22 kartik@tin: (no justification provided)
11:21 kartik@tin: (no justification provided)
11:19 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2051 - T170351 (duration: 00m 46s)
11:18 kartik@tin: Finished deploy [cxserver/deploy@cf3e280]: Update cxserver to fe03ad7 (duration: 00m 56s)
11:17 kartik@tin: Started deploy [cxserver/deploy@cf3e280]: Update cxserver to fe03ad7
11:12 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool es2013 (duration: 00m 47s)
11:02 hashar: Upgraded tox on CI to 2.5.0
10:49 marostegui: Force a re-learn cycle on es2013
10:18 marostegui: Rename wikigrok tables on enwiki on db1089 - T172020
09:03 marostegui: Drop table click_tracking_user_properties wherever it exists - T115982
09:02 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2057 - T170662 (duration: 00m 46s)
08:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2073 to s4 - T170662 (duration: 00m 47s)
08:50 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2073 to s4 - T170662 (duration: 00m 57s)
08:32 marostegui: Drop table click_tracking wherever it exists - T115982
08:15 godog: bounce pdfrender on scb1001, stuck
07:09 marostegui: Disable BBU auto-learn on es2013
06:58 marostegui: Stop MySQL on db2057 for maintenance - T148507
06:50 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1055 - T166204 (duration: 00m 45s)
05:47 demon@tin: Synchronized dblists/: No-op (duration: 00m 47s)
02:59 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Aug 2 02:59:33 UTC 2017 (duration 7m 3s)
02:52 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.12) (duration: 05m 54s)
02:36 eileen: update process-control to 3d3978a
02:27 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 07m 35s)

2017-08-01

23:47 reedy@tin: Synchronized wmf-config/CommonSettings.php: Make Babel use databasey stuff T145366 (duration: 00m 46s)
23:46 eileen: update process control to33a0262d87aa1512888c82ac6b4ed91de3d3cac5
23:44 reedy@tin: Synchronized wmf-config/CommonSettings.php: wfLoadExtension Scribunto (duration: 00m 46s)
23:43 reedy@tin: Synchronized wmf-config/extension-list: json! (duration: 00m 46s)
23:41 reedy@tin: Synchronized wmf-config/CommonSettings.php: Unbreak ContactPage T172199 (duration: 00m 46s)
23:39 reedy@tin: Synchronized wmf-config/MetaContactPages.php: Unbreak ContactPage T172199 (duration: 00m 46s)
23:36 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: T167071 (duration: 00m 47s)
23:30 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: wikinews - T172211 (duration: 00m 46s)
23:26 reedy@tin: Synchronized php-1.30.0-wmf.12/resources/src/mediawiki.rcfilters/mw.rcfilters.Controller.js: T172156 T171514 (duration: 00m 46s)
23:24 reedy@tin: Synchronized static/images/project-logos/: optimise pngs (duration: 00m 46s)
23:23 eileen: update process_control to ecb6669
23:23 reedy@tin: Synchronized static/apple-touch/: optimise pngs (duration: 00m 46s)
23:21 reedy@tin: Synchronized docroot/noc/css/images/: optimise pngs (duration: 00m 47s)
23:13 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: Disable popups on Special pages T170893 (duration: 00m 47s)
23:09 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: New wordmarks T168203 T171769 (duration: 00m 47s)
23:08 reedy@tin: Synchronized static/images/mobile/copyright/: 2 new workmarks (duration: 00m 48s)
22:57 chasemp: push procs off swap for labcontrol1001 w/ swapoff -a & swapon -a
22:20 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/369434/3 (duration: 00m 47s)
22:20 eileen: tools updated from 58bcbf3 to 9554229
22:17 eileen: update process_control to c198b7f
21:48 demon@tin: Started deploy [gerrit/gerrit@15f1544]: Initial deploy x2
21:47 demon@tin: Finished deploy [gerrit/gerrit@15f1544]: Initial deploy (duration: 00m 40s)
21:47 demon@tin: Started deploy [gerrit/gerrit@15f1544]: Initial deploy
21:46 demon@tin: Started deploy [gerrit/gerrit@15f1544]: (no justification provided)
21:45 demon@tin: Finished deploy [gerrit/gerrit@15f1544]: (no justification provided) (duration: 00m 06s)
21:45 demon@tin: Started deploy [gerrit/gerrit@15f1544]: (no justification provided)
21:44 demon@tin: Started deploy [gerrit/gerrit@15f1544]: (no justification provided)
20:40 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 wikis to 1.30.0-wmf.12 refs refs T168053
19:33 twentyafterfour@tin: Finished scap: Sync 1.30.0-wmf.12 and build l10n refs T168053 (duration: 29m 58s)
19:19 urandom: restarting Cassandra, restbase1004-a.eqiad.wmnet, aberrant read latency
19:03 twentyafterfour@tin: Started scap: Sync 1.30.0-wmf.12 and build l10n refs T168053
18:49 twentyafterfour@tin: Finished deploy [phabricator/deployment@3d728e1]: testing phab1001 deployment (duration: 00m 02s)
18:49 twentyafterfour@tin: Started deploy [phabricator/deployment@3d728e1]: testing phab1001 deployment
18:47 twentyafterfour@tin: Finished deploy [phabricator/deployment@3d728e1]: testing phab1001 deployment (duration: 00m 16s)
18:47 twentyafterfour@tin: Started deploy [phabricator/deployment@3d728e1]: testing phab1001 deployment
18:43 twentyafterfour@tin: Finished deploy [phabricator/deployment@3d728e1]: (no justification provided) (duration: 00m 48s)
18:42 twentyafterfour@tin: Started deploy [phabricator/deployment@3d728e1]: (no justification provided)
17:23 twentyafterfour: MediaWiki train for 1.30.0-wmf.12 - finished `scap prep` & `scap patch` refs T168053
16:41 ejegg: updated CiviCRM from 23f2bbf to 5c741b1
16:25 twentyafterfour: MediaWiki Train: Creating new branch wmf/1.30.0-wmf.12 from master. See T168053 for deployment blockers.
16:17 dcausse: restarting elastic on relforge100x servers to pick up new version of the plugins
15:54 bblack: varnish backend restart on cp1072 (mailbox lag)
15:43 bblack: rebooting lvs1002
15:42 marostegui: db1069: Migrate trwiktionary.page from TokuDB to InnoDB
15:40 bblack: stopping pybal on lvs1002 for impending reboot
15:39 bblack: stopping pybal on 1002 for impending reboot
15:33 marostegui: Stop s3 on db1069 - replication stuck
15:17 marostegui: Stop MySQL on db1055 for maintenance - https://phabricator.wikimedia.org/T148507
14:56 andrewbogott: rebooting labvirt1016
14:46 marostegui: Deploy InnoDB compression on s3 - db2074 for the following tables (revision, pagelinks and templatelinks) - T170662
14:45 ema: lvs1001-1003 (eqiad primaries): upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882
14:28 ema: lvs1004-1006 (eqiad secondaries): upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882
14:27 bblack: restart varnish backend on cp1049 (mailbox lag)
14:12 bblack: restart varnish backend on cp1074 (mailbox lag)
13:53 ema@neodymium: conftool action : set/pooled=yes; selector: name=achernar.wikimedia.org,service=pdns_recursor
13:26 ema@neodymium: conftool action : set/pooled=no; selector: name=achernar.wikimedia.org,service=pdns_recursor
13:10 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable OOjs UI EditPage on all wikis except Commons (duration: 00m 44s)
13:06 marostegui: Compress s2 on db1102 - T172169
12:49 elukey: restart hive daemons on analytics1003 to pick up new jvm settings (bigger Xmx, JMX ports)
12:06 dcausse: 100% cpu spike on elastic1023 caused percentiles to jump for a short period of time (T169498)
12:04 elukey: stop eventlogging_sync on analytics-slaves && rename all CookieBlock* tables (log db) to CookieBlock*_backup - T171883
11:52 marostegui: Stop MySQL on db2057 to copy its data to db2074 - T170662
11:36 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2057 - T170662 (duration: 00m 43s)
11:10 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2065 - T170662 (duration: 00m 43s)
10:31 hashar: Enabling Zuul/CI again and reenabling puppet on contint1001
10:24 hashar: contint1001 stopped puppet agent to prevent Zuul server to come back up
10:12 hashar: Stopped Zuul / CI for mass mediawiki extension changes
08:55 ema: lvs2001-2003 (codfw primaries): upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882
08:32 ema: lvs2004-2006 (codfw secondaries): upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882
08:03 ema: lvs3*: upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882
07:40 ema: lvs4001, lvs4002 (ulsfo primaries): upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882
07:35 ema: lvs4003, lvs4004 (ulsfo secondaries): upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882
07:33 ema: pybal 1.13.11 uploaded to apt.w.o T103882
06:20 marostegui: Stop MySQL on db2065 to copy its data to db2073 - T170662
06:13 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2065 - T170662 (duration: 00m 43s)
05:21 marostegui: Restart MySQL on labsdb1003 as it is totally stuck
03:50 mobrovac@tin: Finished deploy [restbase/deploy@0d12138]: Add nl.wikinews - T171897 (duration: 07m 57s)
03:42 mobrovac@tin: Started deploy [restbase/deploy@0d12138]: Add nl.wikinews - T171897
02:32 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Aug 1 02:32:17 UTC 2017 (duration 6m 36s)
02:25 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 07m 46s)

2000s

Archive 1: 2004 Jun - 2004 Sep
Archive 2: 2004 Oct - 2004 Nov
Archive 3: 2004 Dec - 2005 Mar
Archive 4: 2005 Apr - 2005 Jul
Archive 5: 2005 Aug - 2005 Oct, with revision history 2004-06-23 to 2005-11-25
Archive 6: 2005 Nov - 2006 Feb
Archive 7: 2006 Mar - 2006 Jun
Archive 8: 2006 Jul - 2006 Sep
Archive 9: 2006 Oct - 2007 Jan, with revision history 2005-11-25 to 2007-02-21
Archive 10: 2007 Feb - 2007 Jun
Archive 11: 2007 Jul - 2007 Dec
Archive 12: 2008 Jan - 2008 Jul
Archive 12a: 2008 Aug
Archive 12b: 2008 Sept
Archive 13: 2008 Oct - 2009 Jun
Archive 14: 2009 Jun - 2009 Dec

2010s

2020s